Sentiment coding in free text responses – harder than it sounds?

“What is the customer sentiment for our brand?”

Stakeholders want metrics – especially ones that they understand – and when it comes to text analysis, it seems as though a simple output for your open-ended questions is a positive/negative sentiment score. But what are the challenges with using sentiment as a KPI?

1. Tracking sentiment in context

One challenge with sentiment analysis is context. What is positive in one context is negative in another – sentiment is not an objective metric but a subjective one.

We tested this problem and gave 10 people 100 responses to code as positive/negative, and no two people coded the responses the same way. Depending on the industry, the question asked, and what would be considered positive/negative in that specific context, generating a generic sentiment score can result in a misleading metric.

2. Language is not binary

A simple positive/negative sentiment score negates much of the subtleties of language. A response could be coded as negative, but is the respondent slightly annoyed or absolutely furious?

A positive response could be someone who is simply adequately satisfied, or someone who has had the greatest experience of their life. It’s important to explore these subtleties and not simply put them in two very broad buckets.

3. Considering sentiment in complex responses

It’s rare for a customer’s feedback to be nothing but positive or nothing but negative. There’s likely elements of both in their feedback – so do you code the entire thing as positive/negative off a single keyword? It’s important to be able to segment positive and negative feedback within the same response.

So what’s the answer? 

Metrics that mean something to you

Here at Relative Insight, we understand that your business has specific needs and concerns.

Using our Custom Themes feature, informed by our NLP engine and AI powered analysis, you are able to construct language models that actually give you meaningful metrics that are specific to your objectives, and you can then track those changes in these over time.

This allows you to identify specific causes of both positive and negative changes in key metrics such as NPS and CSAT scores.

Do your stakeholders still want a sentiment score? Never fear – we still offer that metric too! But hopefully as you can see, Relative Insight allows you to go further than just positive/negative sentiment reporting.