Ask HN: Sentiment Analysis – how to handle biased word list lengths?

1. Insert each negative example six times into your training set (or weight negative examples accordingly, ie use #positive matches - 6 * #negative matches / (2 * positive word count) as your score

2. Take your distribution of sentiment scores as calculated over held out data (or the training set itself, but be warned that this will skew your results), and calculate the mean and standard deviation. Normalize your results by subtracting the mean and dividing by the standard deviation. You can then say that positive sentiment is > 0 and negative sentiment < 0, with the absolute value being the strength of the classification.

Or you could simply count negative words as 0.44 rather than 1 (800/1800 if those numbers are correct).

Is "not" a negative word? This might be causing problems with things like "This cake is not bad" which has positive sentiment even if it has 2 negative words

Also, consider using bigrams (i.e. word-pairs) to do sentiment matching which will make matching more precise.

2. Create a validation by manually identifying each review as positive or negative. Each time you modify your algorithm run it through your validation set and note the results in a spreadsheet. If you don't do that, you'll never know if and how you've improved the results. The bigger the validation set the better. Similarly, you can use part of your validation set as a training set into a classifier.

3. Find a scale that works to bias your score. For example, I would try to bias your negative score using a log scale. The fewer negative words you have the more they are worth, the more you have the less they are worth.

In the simple case your hyperparameter can be α in

sentiment = (α * #positive_matches - #negative_matches) / (document_word_count)

You can heuristically tune the weights of your lexicon to fit your intuition, but evidence is necessary to progress adequately.

In case you find unbalanced amount of examples, apply an unbalanced effectiveness score like the F-measure to obtain a fair performance of your system.

(There are a lot of suggestions here, so it would be nice if at least you could choose the "best" one)

The way I see it, (and i may very well be slightly off point) you have a corpus of 2000 docs 2 lists -> [Wpos] & [Wneg] with count[Wneg] a factor more than count[Wpos]

if you compute a [0-1] normalized tf-idf score for each term in the set [Wpos] & [Wneg] and sum them up for all words in each of those two sets, you get a score proportional to the count of positive words & negative words. Normalized here would mean using relative frequencies, rather than absolute freq. [I prefer calling the latter term counts]

This puts document_word_count based normalization out of picture and makes it implicit in the tf-idf step.

Now you have Two numbers, Sum(Positive normalized TF-IDFs) and Sum(Negative Normalized TF-IDFs) which you can individually normalize for your list sizes, and then use the two scores for sentiment classification. A dirty hack, and somewhat inefficient if you don't maintain a reverse index.

Second approach could be this. Use your Word List, both positive and negative, to do a Okapi BM25 scoring against your docs using the list as the query set. So you would get a BM25score for your docs. and you can use that to define sentiments.

Corpus - D Di = Document in the corpus you want to classify Query1 = {set of positive words} Query2 = {set of negative words}

PositiveScore = BM25(Query1, Di ) NegativeScore = BM25(Query2, Di )

Some Combination to do classification. if Positive > Negative Score : call it positive!

Just a thought. BM25 has some flexibility in tuning it for length normalizations. Check footnote.

PS: There is the British National Corpus too for word frequencies :)

[1]BM25 and normalizations. http://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-...

My intuition tells me that you can't really do both: Either use only those words in your dictionary and get the behaviour right for single sentences, or extrapolate as if both sets where the same size (weighted average, would be the easiest). By extrapolating you may assume, that for each positive match that you get, you'll miss other positive matches. That means you generally underestimate positive matches, compared to negative matches. This only works on large datasets.

But really, how bad is it to get a sentiment of -0.17 for a single sentence? It tells you that it was a negative sentence but that you have a high chance that there was a positive word in there that you missed, which is what you need to implement to get neutral sentiment for large neutral documents.

(2) the right way to do this is to train a probability estimator on your scores, that is, put +/- labels on some of your documents, then apply logistic regression.

http://en.wikipedia.org/wiki/Logistic_regression

A lot of machine learning people think this is harder than it is and worry more about regularization, overfitting and such, but in the case of turning a score into a probability estimator you are (a) fitting a small number of variables and (b) if you have a lot of data and make a histogram you will ALWAYS get a logistic curve for any reasonable score, I think it has something to do with the central limit theorem.

This seems to be one of the best kept secrets in machine learning. I used to be the bagman who supplied data to people at the Cornell CS department and we ran into a problem where there was an inbalance in the positive and negative set and in that case the 0 threshold for the SVM is not in the right place because it gets the wrong idea about the prior distribution and T Joachims told us to do the logistic regression trick.

Also if you read the papers about IBM Watson they tried just about everything to fit probability estimators and wound up concluding that logistic regression "just works" almost all the time.