No more word clouds: what word frequency analysis tools are hiding

The words contained in survey open-ends, social media conversations, online reviews and customer service transcripts can be a powerful source of business intelligence – if you have the right tools to analyze them! 

Basic word frequency analysis tools like those embedded in social media listening and survey platforms tend to focus on word cloud generation, visually pleasing outputs that have a bad habit of keeping the most important insights hidden. 

What is word frequency analysis?

Word frequency analysis is a common method of qualitative text data analysis that involves counting up mentions of a particular word or phrase as a means of understanding the dominant topics in a particular data set.

The results of the analysis are presented in word frequency analysis visualizations that often take the form of word clouds or frequency tables. In the case of word clouds, the bigger the word, the more frequently it occurs in the analyzed text data. 

There’s no doubt that word clouds are visually appealing and easy to understand. Because of this, they’ve become a popular analytical output. But when it comes to insights, are word clouds telling you anything you don’t already know?

The problem with word frequency analysis tools 

Let’s consider a word cloud analysis example whereby we want to analyze customer survey responses or social media conversations about a retailer. 

The word cloud will almost certainly include words like ‘shopping’, ‘store’, ‘order’, ‘check out’ and ‘price’. But the truth is, none of these findings are particularly insightful because the words surfaced in the word cloud represent the general vocabulary of going shopping. A word cloud from last month, last quarter, last year or a competing retailer would likely feature the same words.

All this to say that word clouds tend to surface the words you would EXPECT people to be using when talking about a particular topic. Telling your stakeholders that this is what your audiences are talking about is not going to impress them. At the end of the day, the best insights are the ones that surprise you and word cloud data visualizations rarely do that. 

The importance of context

Another downside of word frequency analysis tools is that they look at datasets in isolation and in doing so struggle to capture context. A relevant basis of comparison or reference point is required to judge the significance of a particular finding. The best insights can be found by understanding the differences between comparable data sets. 

Considering our retail example, knowing that the phrase ‘delivery issues’ occurred in 2% of survey responses tells us very little. After all, a retailer of any considerable size is destined to have delivery issues from time to time, regardless of how well oiled their logistics processes are. What’s more insightful is understanding how this compares to the competition. If the industry average is 5%, you can probably say you’re doing alright. Conversely, if you’re receiving 5x more delivery complaints than your top competitor, it’s probably an issue that warrants urgent attention.

Word clouds only communicate the headlines about what’s happening in your data but to really understand your customers and target audiences you need to dive deeper into your data to get the next level of insight.

Surface actionable insights with linguistic text analysis

Thankfully, there are alternatives to word frequency analysis tools. Purpose-built text analytics tools like Relative Insight offer a broad range of solutions for analyzing large volumes of unstructured text data. Unlike other BI solutions, these platforms are solely focused on finding insights in text. 

Beyond simply counting the words present in your data set, text analysis platforms leverage computational linguistics, natural language processing, AI and data science to help you efficiently and objectively find actionable insights in your text data. This involves reporting not just on the words present in the data but also on the topics, emotions, grammar features and phrases. Good text analysis tools will also be able to efficiently spot the differences and similarities between data sets and produce visual outputs that allow you to understand what people are talking about at a glance.  

Tools like Relative Insight help you go beyond rudimentary word clouds and get to the heart of what your customers, target audiences and competitors are saying so you can make better business decisions. 

Interested in learning more about text analytics? Download our beginner’s guide or book a no-commitment discovery call with one of our experts to have all your questions answered.