I am currently doing an internship in England. Therefore, I keep alternating between French and English in my different emails and other forms of communication on the Internet. I have been surprised to see that some websites are able to recognize when I use French or when I use English. For example, Facebook automatically proposes me to translate. I was really amazed by this ability, how can a computer know what language I am using? Especially, I use a QWERTY keyboard, which means without accent. Therefore, I don’t write in a proper French and I never use accents.

I remembered some courses of data analysis and of computer science where the issue was to determine the group whom an individual belonged to.


The problem:

If I choose a text, how can I make my computer know if it’s an English text or a French text without accent? Here we try to differentiate without accent since it would be a very (too much) easy way to do it. In particular, how can I do if I can’t have access to French and English dictionaries?

The model:

We assume we have a sample of French and English texts which will be used as benchmarks. We call frenchText(i) the i-th French text we have, and englishText(j) the j-th English text we have. The aim is to determine if the text called TEXT is French or English.

For every text, we count the proportion of every letter. We only take into account the 26 alphabet letters in order to keep the program as simple as possible. Then we compute the average for all the French texts and for all the English texts. In such a way we obtain the normal proportion of each letter in an “average” French (or English) text.

Then, we count the proportion of occurrence of the letters in TEXT and we compute the Euclidean distance between TEXT and the average French text (we call this distance d(TEXT, averageF)) and between TEXT and the average English text (we call this distance d(TEXT, averageE)). If d(TEXT, averageF) is greater than d(TEXT, averageE) we consider that TEXT is more similar to the English texts and therefore is certainly English. Respectively, if d(TEXT, averageF) < d(TEXT, averageE), we consider the text to be French.

The Euclidean distance between two vectors p and q of dimension n.

What is interesting is that, once we have considered a first TEXT, we can, according to the decision we made, take into account TEXT in the calculation of the average letter occurrences in the French texts (respectively English texts), if it has been considered as a French text (resp. as an English text). And then we can use this more accurate average to determine the language of the new text. This is why it is called statistical learning. The more text we compute, the more likely the decision is right.

It would be interesting to use a weighted average in order to take into account the probability of the event [TEXT is French] (resp. [TEXT is English]).

The results:

For the initialization, I have used two French texts and two English texts. Each text is small (about 100 words).

Then I have tested the program on different French and English texts. The results are relevant with the languages of the texts.

If you want to run this program, you will need the files .txt. You can find them on my Google page : https://sites.google.com/site/probaperceptionstock/, the file is a ZIP called textPost5. You can unzip and save this file on your computer in any folder you want.

Warnings:

This method must be use carefully, if the first tests on the first texts are wrong, then, the results are likely to be inconsistent. Indeed, a wrong proportion of letters has consequences on the determination of a text.

Besides, we use the Euclidean distance in order to make a clean and simple presentation. However, it is far from being the best distance for such a problem. Data analysis methods, may well be more relevant. For example, as I explain in a previous post (http://probaperception.blogspot.co.uk/2012/09/v-behaviorurldefaultvmlo.html) the Mahalanobis distance could be interesting in this context.

The code (R):


setwd("U:/Blog/Post5")
#you will have to change this directory according to your own folder

countLetter = function(lowerCase,upperCase, myTable){
            if(is.na(myTable[lowerCase]+myTable[upperCase])){
                        if(is.na(myTable[lowerCase])){return(myTable[upperCase])}
                        else{return(myTable[lowerCase])}
            }
            else{return(myTable[lowerCase] + myTable[upperCase])}
}


proportion = function(myText){

myTextSplit = strsplit(myText,NULL)
table = table(myTextSplit)

a = countLetter("a", "A", table)
b = countLetter("b", "B", table)
c = countLetter("c", "C", table)
d = countLetter("d", "D", table)
e = countLetter("e", "E", table)
f = countLetter("f", "F", table)
g = countLetter("g", "G", table)
h = countLetter("h", "H", table)
i = countLetter("i", "I", table)
j = countLetter("j", "J", table)
k = countLetter("k", "K", table)
l = countLetter("l", "L", table)
m = countLetter("m", "M", table)
n = countLetter("n", "N", table)
o = countLetter("o", "O", table)
p = countLetter("p", "P", table)
q = countLetter("q", "Q", table)
r = countLetter("r", "R", table)
s = countLetter("s", "S", table)
t = countLetter("t", "T", table)
u = countLetter("u", "U", table)
v = countLetter("v", "V", table)
w = countLetter("w", "W", table)
x = countLetter("x", "X", table)
y = countLetter("y", "Y", table)
z = countLetter("z", "Z", table)
total = sum(c(a, b , c, d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z), na.rm = T)

list = c(a, b , c, d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z)/total
list[which(is.na(list))] = 0
return(list)
}


distance = function(myVector, myGroup){
            mean = apply(myGroup,2,mean)
            dist = 0
            for (i in 1:length(myVector)){
                        dist = dist + (mean[i]-myVector[i])**2
            }
            return(sqrt(dist))
}

choice = function(myVector, mypropF, mypropE){
            if(distance(myVector, mypropF) > 1.1*distance(myVector, mypropE)){
                        cat("the text is certainly English")
                        mypropE = rbind(mypropE, myVector)
            }
            else if(distance(myVector, mypropE) > 1.1*distance(myVector, mypropF)){
                        cat("the text is certainly French")
                        mypropF = rbind(mypropF, myVector)
            }
            x = vector("list", 2)
            x[[1]] = mypropF
            x[[2]] = mypropE
           
            return(x)
}

fileName = 'tf1.txt'
tf1 = readChar(fileName, file.info(fileName)$size)
fileName = 'tf2.txt'
tf2 = readChar(fileName, file.info(fileName)$size)
fileName = 'te1.txt'
te1 = readChar(fileName, file.info(fileName)$size)
fileName = 'te2.txt'
te2 = readChar(fileName, file.info(fileName)$size)

fileName = 'tm1.txt'
tm1 = readChar(fileName, file.info(fileName)$size)
propM1 = proportion(tm1)
#French Text


fileName = 'tm2.txt'
tm2 = readChar(fileName, file.info(fileName)$size)
propM2 = proportion(tm2)
#French Text


fileName = 'tm3.txt'
tm3 = readChar(fileName, file.info(fileName)$size)
propM3 = proportion(tm3)
#French Text


fileName = 'tm4.txt'
tm4 = readChar(fileName, file.info(fileName)$size)
propM4 = proportion(tm4)
#English Text


fileName = 'tm5.txt'
tm5 = readChar(fileName, file.info(fileName)$size)
propM5 = proportion(tm5)
#English Text


fileName = 'tm6.txt'
tm6 = readChar(fileName, file.info(fileName)$size)
propM6 = proportion(tm6)
#English Text

list = choice(propM1, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM2, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM3, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM4, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM5, propF, propE)
propF = list[[1]]
propE = list[[2]]

list = choice(propM6, propF, propE)
propF = list[[1]]
propE = list[[2]]

0

Add a comment

The financial market is not only made of stock options. Other financial products enable market actors to target specific aims. For example, an oil buyer like a flight company may want to cover the risk of increase in the price of oil. In this case it is possible to buy on the financial market what is known as a "Call" or a "Call Option".

A Call Option is a contract between two counterparties (the flight company and a financial actor). The buyer of the Call has the opportunity but not the obligation to buy a certain  quantity of a certain product (called the underlying) at a certain date (the maturity) for a certain price (the strike).

If the flight company doesn't want to suffer from the risk of an increase of oil on the market within the next year, a Call Option of maturity 1 year, with a certain strike K may be bought. In this case, if the oil price on the market is over the strike, the flight company will use the Call option to buy oil for the price of the strike. If the price of the oil is lower than the strike, then the flight will not use the option and will buy oil on the market.

As you can see, this contract is not symmetric  The flight company has an option (to buy or not to buy) while the second counterparty just follows the decision of the flight company. Therefore, the flight company will have to pay something to the second counterparty. One of the great question is how much should it be charged.

We will therefore use a pricing algorithm to estimate the price of this Call Option. We first build an algorithm to simulate the value of an asset in the model of Black-Scholes. Then we simulate the historical value of this asset in order to simulate the final value of the Call Option.

Indeed, if we know the value of the asset at the maturity (A(T)), we get directly the value of the Call Option of strike K at maturity : max(A(T) - K , 0).

Our asset is randomly distributed as:
A(t+dt) = A(t) (1 + r dt + v(B(t+dt) -B(t))

Where r is the interest rate, v the volatility of the asset and B a Brownian process (for more detail you can look at the post on Brownian motion).

After repeating the process many times, we estimate the mean of the Call Option for different strike in order to estimate the price of the Call Option.


As we can see the price of the Call decreases with the strike. Actually it converges towards 0. Indeed, the higher the strike is, the lower the chance are such that the asset goes over the strike.

What is done here can be done for many, many, many other financial products in order to price them by Monte Carlo.



The code (R):


sample.size <- 365
mu <- 0.1
sigma <- 0.2

a0 <- 1
Asset <- function(sample.size = 365, mu = 0.1, sigma = 0.2, a0 = 1){
  dt <- 1/sample.size
  sdt =sigma*sqrt(dt)
  gauss <- rnorm(sample.size)
  asset <- NULL
  asset[1] <- a0 + mu + sigma * gauss[1]
  test.default <- FALSE
  for(i in 2:365){
    if (!test.default){
      asset[i] <- asset[i-1] * (1 + dt*mu + sdt * gauss[i])
    }
    else{
      asset[i] <- 0
    }
    if (asset[i] <= 0){
      asset[i] <- 0
      test.default <- TRUE
    }
  }
  return(asset)
}


PriceEstimation <- function(t, tf = 365, r = 0.1, strike = 1, n = 1000, mu = 0.1, sigma = 0.2, a0 = 1){
  mean <- 0
  for(i1 in 1:n){
    mean <- mean + exp(-r*(tf-t)/tf) * max(0, (Asset(tf, mu, sigma, a0)-strike))
  }
  mean <- mean/n
  return(mean)
}

res <- NULL
for(i in 1:length(seq(0, 3, 0.05))){
  res[i] <- PriceEstimation(t = 0, tf = 365, r = 0.1, strike = seq(0, 3, 0.05)[i], n = 1000, mu = 0.1, sigma = 0.2, a0 = 1)
}

plot(seq(0,3, 0.05), res,type = 'l', xlab = "Strike Value",  ylab = "Price of the Call")



0

Add a comment

Blog Archive
Translate
Translate
Loading