Sentiment Analysis

Sentiment analysis (also known as opinion mining or emotion AI) refers to the use of Natural Language Processing (NLP), text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to the voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine.

It is a process of detecting positive or negative sentiment in text. It is often used by businesses to detect sentiment in social data, gauge brand reputation, and understand customers.

Since customers express their thoughts and feelings more openly than ever before, sentiment analysis is becoming an essential tool to monitor and understand that sentiment. Automatically analyzing customer feedback, such as opinions in survey responses and social media conversations, allows brands to learn what makes customers happy or frustrated so that they can tailor products and services to meet their customer's needs.

For example, using sentiment analysis to automatically analyze 4,000+ reviews about your product could help you discover if customers are happy about your pricing plans and customer service.

Maybe you want to gauge brand sentiment on social media, in real-time and over time, so you can detect disgruntled customers immediately and respond as soon as possible.

Types of Sentiment Analysis

Sentiment Analysis models focus on polarity (positive, negative, neutral) but also on feelings and emotions (angry, happy, sad, etc), urgency (urgent, not urgent), and even intentions (interested v. not interested).

Depending on how you want to interpret customer feedback and queries, you can define and tailor your categories to meet your sentiment analysis needs. In the meantime, here are some of the most popular types of sentiment analysis:

Fine-grained Sentiment Analysis

If polarity precision is important to your business, you might consider expanding your polarity categories to include:

• Very Positive

• Positive

• Neutral

• Negative

• Very Negative

This is usually referred to as fine-grained sentiment analysis and could be used to interpret 5-star ratings in a review, for example:

• Very Positive = 5 stars

• Very Negative = 1 star

Emotion Detection

This type of sentiment analysis aims to detect emotions, like happiness, frustration, anger, sadness, and so on. Many emotion systems use lexicons (i.e., lists of words and the emotions they convey) or complex machine learning algorithms.

One of the downsides of using lexicons is that people express emotions in different ways. Some words that typically express anger, like bad or kill (e.g., your product is so bad or your customer support is killing me) might also express happiness (e.g., this is a badass or you are killing it).

Aspect-based Sentiment Analysis

Usually, when analyzing sentiments of texts, let's say product reviews, you will want to know which particular aspect or features people are mentioning in a positive, neutral, or negative way. That is where aspect-based sentiment analysis can help, for example in this text: “The battery life of this camera is too short”, an aspect-based classifier would be able to determine that the sentence expresses a negative opinion about the feature battery life.

Multilingual sentiment analysis

Multilingual sentiment analysis can be difficult. It involves a lot of preprocessing and resources. Most of these resources are available online (e.g., sentiment lexicons), while others need to be created (e.g., translated corpora or noise detection algorithms), but you will need to know how to code to use them.

Why is Sentiment Analysis Important?

Sentiment Analysis is extremely important because it helps businesses quickly understand the overall opinions of their customers. By automatically sorting the sentiment behind reviews, social media conversations, and more, you can make faster and more accurate decisions.

It is estimated that 90% of the world's data is unstructured, in other words, it is unorganized. Huge volumes of unstructured business data are created every day: emails, support tickets, chats, social media conversations, surveys, articles, documents, etc). But it is hard to analyze for sentiment in a timely and efficient manner.

The overall benefits of Sentiment Analysis include

• Sorting Data at Scale: Can you imagine manually sorting through thousands of tweets, customer support conversations, or surveys? There is just too much business data to process manually. Sentiment analysis helps businesses process huge amounts of data in an efficient and cost-effective way.

• Real-Time Analysis: Sentiment Analysis can identify critical issues in real-time, for example, is a PR crisis on social media escalating? Is an angry customer about to churn? Sentiment analysis models can help you immediately identify these kinds of situations, so you can take action right away.

• Consistent Criteria: It is estimated that people only agree around 60-65% of the time when determining the sentiment of a particular text. Tagging text by sentiment is highly subjective, influenced by personal experiences, thoughts, and beliefs. By using a centralized sentiment analysis system, companies can apply the same criteria to all of their data, helping them improve accuracy and gain better insights.

How does Sentiment Analysis work?

Sentiment Analysis, otherwise known as opinion mining, works thanks to natural language processing (NLP) and machine learning algorithms, to automatically determine the emotional tone behind online conversations.

There are different algorithms you can implement in sentiment analysis models, depending on how much data you need to analyze, and how accurate you need your model to be.

Sentiment analysis algorithms fall into one of three buckets:

• Rule-based: these systems automatically perform sentiment analysis based on a set of manually crafted rules.

• Automatic: systems rely on machine learning techniques to learn from data.

• Hybrid: systems combine both rule-based and automatic approaches.

Sentiment Analysis Challenges

Sentiment Analysis is one of the hardest tasks in natural language processing because even humans struggle to analyze sentiments accurately.

Data scientists are getting better at creating more accurate sentiment classifiers, but there is still a long way to go. Let's take a closer look at some of the main challenges of machine-based sentiment analysis:

Subjectivity and Tone

There are two types of text: subjective and objective. Objective texts do not contain explicit sentiments, whereas subjective do. Say, for example, you intend to analyze the sentiment of the following two texts:

The package is nice. The package is red.

Most people would say that sentiment is positive for the first one and natural for the second one, right? All predicates (adjectives, verbs, and some nouns) should not be treated the same with respect to how they create a sentiment. In the examples above, nice is more subjective than red.

Context and Polarity

All utterances are uttered at some point in time, in some place, by and to some people, you get the point. All utterances are uttered in context. Analyzing sentiment without context gets pretty difficult. However, machines cannot learn about contexts if they are not mentioned explicitly. One of the problems that arise from context is changes in polarity. Look at the following responses to a survey:

Everything of it. Absolutely nothing!

Imagine the responses above come from answers to the question what did you like about the event? The first response would be positive and the second one would be negative, right? Now, imagine the responses come from answers to the question what did you dislike about the event? The negative in the question will make sentiment analysis change altogether.

A good deal of pre-processing or post-processing will be needed if we are to take into account at least part of the context in which texts were produced. However, how to pre-process or post-process data in order to capture the bits of context that will help analyze sentiment is not straightforward.

Irony and Sarcasm

When it comes to irony and sarcasm, people express their negative sentiments using positive words, which can be difficult for machines to detect without having a thorough understanding of the context of the situation in which a feeling was expressed.

For example, look at some possible answers to the question, Did you enjoy your shopping experience with us?

Yeah, sure. So smooth! Not one, but many!

What sentiment would you assign to the responses above? The first response with an exclamation mark could be negative, right? The problem is there is no textual cue that will help a machine learn or at least question that sentiment since yeah and sure often belong to positive or neutral texts.

How about the second response? In this context, the sentiment is positive, but we are sure you can come up with many different contexts in which the same response can express a negative sentiment.

Comparisons

How to treat comparisons in sentiment analysis is another challenge worth tackling. Look at the texts below:

This product is second to none. This is better than older tools. This is better than nothing.

The first comparison doesn't need any contextual clues to be classified correctly. It is clear that it is positive.

The second and third texts are a little more difficult to classify, though. Would you classify them as neutral, positive, or even negative? Once again, context can make a difference. For example, if the 'older tools' in the second text were considered useless, then the second text is pretty similar to the third text.

Emojis

There are two types of emojis according to Guibon et al.. Western emojis (e.g. :D) is encoded in only one or two characters, whereas Eastern emojis (e.g. ¯ \ (ツ) / ¯) is a longer combination of characters of a vertical nature. Emojis play an important role in the sentiment of texts, particularly in tweets.

You’ll need to pay special attention to character-level, as well as word-level when performing sentiment analysis on tweets. A lot of preprocessing might also be needed. For example, you might want to preprocess social media content and transform both Western and Eastern emojis into tokens and whitelist them (i.e. always take them as a feature for classification purposes) in order to help improve sentiment analysis performance.

Defining Neutral

Defining what we mean by neutral is another challenge to tackle in order to perform accurate sentiment analysis. As in all classification problems, defining your categories -and, in this case, the neutral tag- is one of the most important parts of the problem. What you mean by neutral, positive, or negative does matter when you train sentiment analysis models. Since tagging data requires that tagging criteria be consistent, a good definition of the problem is a must.

Sentiment Analysis Use Cases and Applications

The applications of sentiment analysis are endless and can be applied to any industry, from finance and retail to hospitality and technology. Below, we’ve listed some of the most popular ways that sentiment analysis is being used in business:

1. Social Media Monitoring
2. Brand Monitoring
3. Voice of Customer (VoC)
4. Customer Service
5. Market Research