I am currently doing an internship in England. Therefore, I keep alternating between French and English in my different emails and other forms of communication on the Internet. I have been surprised to see that some websites are able to recognize when I use French or when I use English. For example, Facebook automatically proposes me to translate. I was really amazed by this ability, how can a computer know what language I am using? Especially, I use a QWERTY keyboard, which means without accent. Therefore, I don’t write in a proper French and I never use accents.

I remembered some courses of data analysis and of computer science where the issue was to determine the group whom an individual belonged to.


The problem:

If I choose a text, how can I make my computer know if it’s an English text or a French text without accent? Here we try to differentiate without accent since it would be a very (too much) easy way to do it. In particular, how can I do if I can’t have access to French and English dictionaries?

The model:

We assume we have a sample of French and English texts which will be used as benchmarks. We call frenchText(i) the i-th French text we have, and englishText(j) the j-th English text we have. The aim is to determine if the text called TEXT is French or English.

For every text, we count the proportion of every letter. We only take into account the 26 alphabet letters in order to keep the program as simple as possible. Then we compute the average for all the French texts and for all the English texts. In such a way we obtain the normal proportion of each letter in an “average” French (or English) text.

Then, we count the proportion of occurrence of the letters in TEXT and we compute the Euclidean distance between TEXT and the average French text (we call this distance d(TEXT, averageF)) and between TEXT and the average English text (we call this distance d(TEXT, averageE)). If d(TEXT, averageF) is greater than d(TEXT, averageE) we consider that TEXT is more similar to the English texts and therefore is certainly English. Respectively, if d(TEXT, averageF) < d(TEXT, averageE), we consider the text to be French.

The Euclidean distance between two vectors p and q of dimension n.

What is interesting is that, once we have considered a first TEXT, we can, according to the decision we made, take into account TEXT in the calculation of the average letter occurrences in the French texts (respectively English texts), if it has been considered as a French text (resp. as an English text). And then we can use this more accurate average to determine the language of the new text. This is why it is called statistical learning. The more text we compute, the more likely the decision is right.

It would be interesting to use a weighted average in order to take into account the probability of the event [TEXT is French] (resp. [TEXT is English]).

The results:

For the initialization, I have used two French texts and two English texts. Each text is small (about 100 words).

Then I have tested the program on different French and English texts. The results are relevant with the languages of the texts.

If you want to run this program, you will need the files .txt. You can find them on my Google page : https://sites.google.com/site/probaperceptionstock/, the file is a ZIP called textPost5. You can unzip and save this file on your computer in any folder you want.

Warnings:

This method must be use carefully, if the first tests on the first texts are wrong, then, the results are likely to be inconsistent. Indeed, a wrong proportion of letters has consequences on the determination of a text.

Besides, we use the Euclidean distance in order to make a clean and simple presentation. However, it is far from being the best distance for such a problem. Data analysis methods, may well be more relevant. For example, as I explain in a previous post (http://probaperception.blogspot.co.uk/2012/09/v-behaviorurldefaultvmlo.html) the Mahalanobis distance could be interesting in this context.

The code (R):


setwd("U:/Blog/Post5")
#you will have to change this directory according to your own folder

countLetter = function(lowerCase,upperCase, myTable){
            if(is.na(myTable[lowerCase]+myTable[upperCase])){
                        if(is.na(myTable[lowerCase])){return(myTable[upperCase])}
                        else{return(myTable[lowerCase])}
            }
            else{return(myTable[lowerCase] + myTable[upperCase])}
}


proportion = function(myText){

myTextSplit = strsplit(myText,NULL)
table = table(myTextSplit)

a = countLetter("a", "A", table)
b = countLetter("b", "B", table)
c = countLetter("c", "C", table)
d = countLetter("d", "D", table)
e = countLetter("e", "E", table)
f = countLetter("f", "F", table)
g = countLetter("g", "G", table)
h = countLetter("h", "H", table)
i = countLetter("i", "I", table)
j = countLetter("j", "J", table)
k = countLetter("k", "K", table)
l = countLetter("l", "L", table)
m = countLetter("m", "M", table)
n = countLetter("n", "N", table)
o = countLetter("o", "O", table)
p = countLetter("p", "P", table)
q = countLetter("q", "Q", table)
r = countLetter("r", "R", table)
s = countLetter("s", "S", table)
t = countLetter("t", "T", table)
u = countLetter("u", "U", table)
v = countLetter("v", "V", table)
w = countLetter("w", "W", table)
x = countLetter("x", "X", table)
y = countLetter("y", "Y", table)
z = countLetter("z", "Z", table)
total = sum(c(a, b , c, d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z), na.rm = T)

list = c(a, b , c, d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z)/total
list[which(is.na(list))] = 0
return(list)
}


distance = function(myVector, myGroup){
            mean = apply(myGroup,2,mean)
            dist = 0
            for (i in 1:length(myVector)){
                        dist = dist + (mean[i]-myVector[i])**2
            }
            return(sqrt(dist))
}

choice = function(myVector, mypropF, mypropE){
            if(distance(myVector, mypropF) > 1.1*distance(myVector, mypropE)){
                        cat("the text is certainly English")
                        mypropE = rbind(mypropE, myVector)
            }
            else if(distance(myVector, mypropE) > 1.1*distance(myVector, mypropF)){
                        cat("the text is certainly French")
                        mypropF = rbind(mypropF, myVector)
            }
            x = vector("list", 2)
            x[[1]] = mypropF
            x[[2]] = mypropE
           
            return(x)
}

fileName = 'tf1.txt'
tf1 = readChar(fileName, file.info(fileName)$size)
fileName = 'tf2.txt'
tf2 = readChar(fileName, file.info(fileName)$size)
fileName = 'te1.txt'
te1 = readChar(fileName, file.info(fileName)$size)
fileName = 'te2.txt'
te2 = readChar(fileName, file.info(fileName)$size)

fileName = 'tm1.txt'
tm1 = readChar(fileName, file.info(fileName)$size)
propM1 = proportion(tm1)
#French Text


fileName = 'tm2.txt'
tm2 = readChar(fileName, file.info(fileName)$size)
propM2 = proportion(tm2)
#French Text


fileName = 'tm3.txt'
tm3 = readChar(fileName, file.info(fileName)$size)
propM3 = proportion(tm3)
#French Text


fileName = 'tm4.txt'
tm4 = readChar(fileName, file.info(fileName)$size)
propM4 = proportion(tm4)
#English Text


fileName = 'tm5.txt'
tm5 = readChar(fileName, file.info(fileName)$size)
propM5 = proportion(tm5)
#English Text


fileName = 'tm6.txt'
tm6 = readChar(fileName, file.info(fileName)$size)
propM6 = proportion(tm6)
#English Text

list = choice(propM1, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM2, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM3, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM4, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM5, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM6, propF, propE)
propF = list[[1]]
propE = list[[2]]

Posted by
Location: University of Cambridge, Cambridge, Cambridgeshire CB2 9LH, UK
Labels: language statistical learning translation
0

Add a comment

The financial market is not only made of stock options. Other financial products enable market actors to target specific aims. For example, an oil buyer like a flight company may want to cover the risk of increase in the price of oil.

I found a golden website. The blog of Esteban Moro. He uses R to work on networks. In particular he has done a really nice code to make some great videos of networks. This post is purely a copy of his code. I just changed a few arguments to change colors and to do my own network.

3

As you have certainly seen now, I like working on artificial neural networks. I have written a few posts about models with neural networks (Models to generate networks, Want to win to Guess Who and Study of spatial segregation).

1

I already talked about networks a few times in this blog. In particular, I had this approach to explain spatial segregation in a city or to solve the Guess Who? problem. However, one of the question is how to generate a good network.

The function apply() is certainly one of the most useful function. I was scared of it during a while and refused to use it. But it makes the code so much faster to write and so efficient that we can't afford not using it.

1

Have you ever played the board game "Guess who?".

If you want to choose randomly your next holidays destination, you are likely to process in a way which is certainly biased. Especially if you choose randomly the latitude and the longitude.

4

My previous post is about a method to simulate a Brownian motion. A friend of mine emailed me yesterday to tell me that this is useless if we do not know how to simulate a normally distributed variable.

The Brownian motion is certainly the most famous stochastic process (a random variable evolving in the time). It has been the first way to model a stock option price (Louis Bachelier's thesis in 1900).

1

The merge of two insurance companies enables to curb the probability of ruin by sharing the risk and the capital of the two companies.

For example, we can consider two insurance companies, A and B.

How to estimate PI when we only have R and the formula for the surface of a circle (Surface = PI * r * r)?

The estimation of this number has been one of the greatest challenge in the history of mathematics. PI is the ratio between a circle's circumference and diameter.

I was in a party last night and a guy was totally drunk. Not just the guy who had a few drinks and speaks a bit too loud, but the one who is not very likely to remember what he has done during his night, but who is rather very likely to suffer from a huge headache today.

I'm very glad if you use or at least read sometimes my blog. However, I'd like to give you a list of wonderful blogs which are useful if you are interested in applied mathematics and programming for simulation.

My favorite one is written by Arthur Charpentier, a French mathematician.

I am currently doing an internship in England. Therefore, I keep alternating between French and English in my different emails and other forms of communication on the Internet. I have been surprised to see that some websites are able to recognize when I use French or when I use English.

The VIX (volatility index) is a financial index which measures the expectation of the volatility of the stock market index S&P 500 (SPX). The higher is the value of the VIX the higher are the expectations of important variations in the S&P 500 during the next month.

The Olympic Games have finished a couple of days ago. Two entire weeks of complete devotion for sport. Unfortunately I hadn’t got any ticket but I didn’t fail to watch many games on TV and internet.

Hello (New World!), 

My name is Edwin, I’m a 22 year-old French student in applied mathematics. In particular, I study probability, statistics and risk theory.

Blog Archive
Blog Archive
Translate
Translate
Loading