Text Analysis: Sentiment Analysis

This is part 5 of our series on Text Analysis, previous segments are available in our archives.

The past few days, I’ve talked about figuring out which are the most important words in customer feedback, and how to visualize them. This can be extremely helpful, but what if you want to know the answer to a more abstract question – how do customers feel in their responses?

Sentiment Analysis

The field of sentiment analysis does just this – it uses natural language processing and statistical methods to translate text to a subjective determination of feeling. One of the more basic approaches to sentiment analysis is to understand the polarity of of text – is the feeling positive, negative, or neutral. Advanced techniques can go into more depth by categorizing the feelings in to emotions of happy, sad, angry, etc.

As one can imagine, determining the sentiment from text can be difficult, particularly because the same sentence can be interpreted in different ways and people use colloquial / sarcastic language in informal settings. For example, “this is just like the previous model” could be interpreted a number of ways:

  • Happy: I had the old model and it is no longer made so I was very happy to find something that fits my needs at a comparable price.
  • Neutral: I am simply stating a fact that the differences between the two models are minimal.
  • Disappointed: I had high hopes for the new model’s launch but am disappointed to see that it has no new features but is more expensive – I’m going to keep my old version.

Or if a dessert shop had a review that said, “I need these cookies in a bad way”, most humans would interpret that as a positive sentiment, but a simple algorithm that simply looks for negative words like “bad” might score this as a negative sentiment.

Because of this ambiguity, there is a lot of room for interpretation. One study found that humans classifying sentiments only agreed with each other 79% of the time.1 So, computer algorithms don’t need to be perfect to still do a good job. They can do a first pass to help you narrow your focus when you have too many responses to read manually. Thankfully, there are a number of tools available for you to use if you want to try using automated sentiment analysis, so you don’t have to build it yourself.

Questions? Send any questions on data analytics or pricing strategy to doug@outlier.ai and I’ll answer them in future issues!


[1] http://mashable.com/2010/04/19/sentiment-analysis/#HpiQcruCkkqJ