Which AI tool is better for text analysis: Relative Insight or ChatGPT?

ChatGPT has already revolutionized the way many people work. The AI tool has proven itself capable of many tasks, from idea generation to writing assistance. However, how does it fare when doing something as innovative as text analysis?

To find out, we put it up against Relative Insight’s specialist text analytics platform. This titanic clash of AI-powered software utilizing large language models (LLMs), a battle between natural language understanding (NLU) and natural language generation (NLG), would decide whether you could add text analytics to the list of things ChatGPT does better than existing technology.

In the pre-match build up, Ryan Callihan, Head of AI and Data Science at Relative Insight, gave his take on all things ChatGPT. Unsurprisingly, he predicted a win for Relative Insight in terms of text analytics, yet highlighted ways in which the company is looking to augment its offering through ChatGPT.

With the stakes so high, Relative Insight needed its very best representative in this colossal contest. Enter Flick Edwards, Relative Insight’s Customer Experience Director. Flick is a text analysis master who makes her keyboard sing with the sound of actionable insights. Her very presence dispelled the nerves of any Relative Insight staff nervous that ChatGPT would render their work obsolete.

Her opponent? A mysterious figure. Some say they’re a physical manifestation of ChatGPT itself, ready to lead the machines in their inevitable uprising. Others, that they’re a direct descendent of Queen Elizabeth II looking to reclaim the throne. All we knew was… they were in a morph suit.

The competition looked at five aspects of text analytics:

Time
Collaboration
Metrics
Accuracy
Creativity

Rather than determining the victor through complex algorithms, as requested by ChatGPT, the winner would be the AI tool which won the most rounds. Let the contest begin!

Does Relative Insight or ChatGPT offer faster time to insight?

The first element of the battle was time to insight. Getting actionable findings rapidly is critical for time-poor insights professionals. Each participant was tasked with uncovering information from one of Relative Insight’s employee onboarding feedback surveys. This loose definition ensured that neither of these AI tools had an unfair advantage.

Flick immediately uploaded the survey data to the Relative Insight platform through Smart Upload, taking a matter of seconds. This instantly recognized meta data points within the survey. In turn, this saved Flick time on the next step of her analysis, as she split responses between long-serving and short-serving employees. This comparison quickly surfaced that less-tenured staff were infinitely more likely to discuss onboarding software Wonderway. She speedily deduced that it would benefit longer-serving employees to go over elements of the onboarding process using this software. Flick took three minutes and 48 seconds to produce her first actionable insight.

Unfortunately for Flick, ChatGPT had clearly entered an exceptional operator into this contest. While the text generation tool works rapidly, it’s limited by the prompt that is input by users. The mystery morph figure, fingers typing faster than the speed of light, took just over three minutes to get their prompt right, incorporating data for the AI tool to investigate.

Once the prompt was in, ChatGPT generated its response within seconds. While the insights NLG software surfaced weren’t anywhere near as actionable as those found in Relative Insight, the purpose of this round was speed. Therefore, finding that respondents had used the word ‘onboarding’ 128 times within three minutes and 37 seconds was enough for ChatGPT to take round one. However, despite her defeat, Flick was still proud of uncovering an insight in under four minutes.

Score: Relative Insight 0-1 ChatGPT

Sharing visualized insights with stakeholders

With both participants having rapidly produced insights, their next challenge was to share them. Their task was simple – communicate their findings visually to Relative Insight’s people team.

Flick jumped straight to work. Thanks to Relative Insight’s insight cards, all she had to do was tag ‘Wonderway’ to this visualization, as well as adding verbatim examples of respondents using the word and stating actions the team should take. She exported the card out of the platform, attached it to an email, sent it, then put her feet up. This took her all of 90 seconds.

ChatGPT’s operator didn’t have this easy visualization at their disposal. They quickly asked the tool to produce the wording for a presentation about the insights in question. While it generated this text efficiently, there was no way of visualizing it without manually entering it into a presentation.

At this point, the mystery morph suited competitor conceded the round to Flick and Relative Insight, simply typing: “Computer says no.”

Score: Relative Insight 1-1 ChatGPT

Whose text analysis metrics mattered most?

With the scores tied, round three would be pivotal in deciding which competitor took the win. This part of the contest focused on metrics. Without backing up research with hard metrics, insights will be ignored by stakeholders.

Some say that comparison is the thief of joy, but at Relative Insight comparing data sets is vital to quantifying qualitative data. This methodology creates Relative difference, a metrics which illustrates how likely a linguistic feature (topics, grammar, words, phrases and emotions) is to appear in one data set over another. This number is a metric that you’re able to take to your stakeholders – from line managers through to the c-suite – to clearly communicate insights in a way that resonates.

In this example, Flick’s email to Relative Insight’s people team generated an instant policy change. By highlighting that newer hires talked about ‘Wonderway’ infinitely more (meaning this word didn’t appear in long-serving employees’ responses), Flick’s stakeholder said they would immediately plan to enrol and train long-tenured staff on the Wonderway platform.

To contrast this, most text analysis tools focus on frequency analysis – counting words. When ChatGPT’s representative was finally coaxed into sending an email to the people team, they simply repeated that the word ‘onboarding’ was used 128 times in responses. Unsurprisingly, the people team didn’t even bother to respond to this email – much to the mystery morph suited competitor’s frustration.

This made for a resounding win for Relative Insight. Indeed, in the time it took for ChatGPT to wait for a response from stakeholders, Flick built three further insight cards utilizing Relative difference and exported them to the people team.

Score: Relative Insight 2-1 ChatGPT

Which AI tool produces the most accurate text analytics?

The next part of the competition focused on the accuracy of each AI tool. When using text analytics to drive change, you need to know that the insights you’re uncovering reflect what’s actually happening, making accuracy vital.

With questions already being raised about the accuracy of information provided by ChatGPT, its operator knew they had to proceed with caution here. They changed the prompt to do more than count words, instead asking ChatGPT to summarize a selection of the responses from the survey, which it did rapidly. However, all this effectively did was neaten up the wording within responses, rather than draw any conclusions from them.

The Relative Insight platform uses natural language processing to break down and quantify text through five linguistic features – topics, words, phrases, grammar and emotions. This makes it easy for insights professionals to identify what matters most to their organization. The software only displays linguistic features which mean a 95% statistical significance threshold – meaning here is an extremely high degree of confidence that all findings are not happening by chance.

With accuracy a given, Flick felt assured that the insight cards she built reflected the responses from the onboarding survey.

While it’s arguable that both platforms can be accurate, ties were forbidden in this colossal clash. Relative Insight checks that its output accurately represents data uploaded to the platform. ChatGPT simply does what’s asked of it without checking – it can even surface false information if the source it chooses is incorrect. This makes Relative Insight the more accurate AI tool.

Score: Relative Insight 3-1 ChatGPT

Harnessing human creativity

Flick and Relative Insight had an unassailable lead heading into the final round. Billed pre-match as the crucial decider, the creativity element simply turned into a Flick victory lap.

While some people associate discovery with things like scurvy and dangerous travel, the beauty of Relative Insight is that it surfaces everything you need to know in an unbiased way. It does the legwork for insights professionals, allowing them to pick and choose the features most pertinent to their organization. It enables you to discover things you didn’t even think to look for – removing the need for hypothesis-driven research.

ChatGPT does what it’s told. You issue a prompt, it gives a response – no more, no less. It’s an efficient way of generating text and ideas, however, it also removes human ingenuity. The creative process is limited to what you can write in the prompt.

This was illustrated in the final command do both contestants: Do creative things with your insights. While Flick immediately got to work, ChatGPT’s operator was flummoxed. Eventually, they resorted to attempting to put Flick off, such is the lack of creativity involved in using ChatGPT.

Meanwhile, Flick was using Relative Insight’s software to discover and visualize insights in many ways, including creating a Heatmap.

Another resounding victory for Flick and Relative Insight made it 4-1 to the specialist text analytics platform.

Analyzing, quantifying and visualizing text data using an insights tool

In a competition judging AI tools’ text analysis capabilities, it’s unsurprising that the specialist software is the victor. While ChatGPT can support with some elements of text analytics, it doesn’t produce actionable insights which offer value to your organization.

While OpenAI’s software may be slightly faster if you’re an expert in writing prompts and equally accurate if you’re manually copying text data for it to analyze, the surface-level insights it generates aren’t sufficient to move the needle with stakeholders.

Relative Insight’s quantified and visualized insights grab stakeholders’ attention. The platform brings structure to unstructured data, allowing you to quickly build insights that drive business change. See for yourself how each of the five components from the competition work within the platform by signing up for a free trial now.