Sentiment Analysis

In short, Sentiment Analysis is the process of detecting the contextual polarity of text. In other words, it determines whether a piece of writing is positive, negative or neutral.

An alternative term is opinion mining, as it derives the opinion, or the attitude of a speaker. A common use case for this technology is to discover how people feel about a particular topic.

For example, do people on Twitter think that Chinese food in San Francisco is good or bad?

Analyzing tweets for sentiment will answer this question for you. You can also learn why people think the food is good or bad, by extracting the exact word indicating why people did or didn't like the food. Example: "too salty"

This is the kind of insight one hopes to find when conducting market research. Now you know whether to expand your Chinese food empire to San Francisco or to keep it in Las Angeles.

Sentiment Analysis can be used to determine sentiment on a variety of levels. It will score the entire document as positive or negative, and it will also score the sentiment of individual words or phrases in the document.

For example, if someone writes a Facebook comment that reads:

"I love the summer in New York, but I hate the winter."

The individual scores would show "love the summer" as positive and "hate the winter" as negative. However, the sentiment for the entire comment would be neutral, because the positive sentiment for the word love would cancel out the negative sentiment for the word hate.

Because Sentiment Analysis can track a particular topic, many companies use it to track or monitor their products, services or reputation in general. For example, if someone is attacking your brand on social media, sentiment analysis will score the post as extremely negative, and you can create alerts for posts with hyper-negative sentiment scores.

Measuring sentiment accuracy

The accuracy of Sentiment Analysis can be measured in many ways, but the most common way is to score accuracy in comparison to a human. A study from the University of Pittsburgh shows that humans can only agree on whether or not a sentence has the correct sentiment, 80% of the time. So any natural language processing engine that can score around 80% is doing a great job with accuracy.

There are a few major challenges for an engine analyzing text for sentiment. One of the biggest issues is that it has trouble understanding irony. Even humans have trouble with someone who is being sarcastic. It is one of the most common mistakes a text analytics engine makes when trying to analyze text for sentiment.

Even humans have trouble, as they can analyze with 80% accuracy. Other problems are when words have multiple definitions. There are a few engines that use deep learning to help them understand context. For example, if someone is talking about excel, are they talking about the data software, the chewing gum, or the verb that describes how someone can be extremely good at something?