case study

Amplifying Insights: How AI-Driven Probing Transforms Survey Responses

overview

Socratic set out to demonstrate the power and effectiveness of the use of AI as part of a recent research on research examination. Using several tools to capture similar information from participants, Socratic measured the quality and depth of responses when using an AI chat bot to interact with participants.

By questioning over 600 different consumers about their auto insurance and randomly offering different methods of capturing feedback, Socratic examined the impact of using an advanced AI approach within a research project to improve the insights for a client.

Solution

Socratic partnered with Reveal AI to design and implement a survey-based AI chat bot prompt to gauge and measure the differences in responses against those respondents who were provided a traditional open-end feedback option.

The use of AI within surveys is intended to improve the efficiency of the instrument’s ability to gain more information from the respondent as well as create a path towards deeper and richer information about the survey topic.

approach

600 consumer respondents were divided into 4 groups and were provided with a different method to respond to a set of questions.

Traditional Open-End

Respondents were provided with an open-end response text box for a typed response.

Enhanced Open-End

These respondents had a quality meter underneath their text box that scored the response in real-time on the amount and quality of input.

AI Chat Bot with 2 Probes

The AI chat bot provided respondents with a conversational probe question for the open-ended question. It would include two follow-up questions; each probe from the chat bot laddered on the survey participant’s previous response.

AI Chat Bot with 4 Probes

This group received the AI chat bot with the same conversational probes but allowed for up to four follow-up questions.

Outcomes

There was significant improvement in the responses when using the AI chat bot to instigate more comprehensive responses from the respondent. Additionally, there were also advantages to limiting the number of probes provided by the AI chat bot. The use of the AI chat bot as part of the survey design resulted in much more powerful data to fuel business decision-making than the control method of a traditional open text response, even with a quality meter score attached to the text box.

The respondents who were provided with the AI chat bot as part of their survey produced better feedback than traditional open-ended questions. The chat bot with two additional probe questions produced 2.5x more feedback, and the one with four probe questions produced 5x more feedback.

The responses from each of the four additional probe questions were compared using the Flesch Kincaid Grade Level formula and resulted in elevated grade levels by at least one full grade level versus traditional open-ended methods. This indicates a more detailed and thoughtful response to the question asked in the survey.

When asked, the survey participants who responded using the AI chat bot said they were able to share everything they wanted to share (62% vs 44% agree) and that they found it easy to share ideas this way (61% vs 49% agree).

Using AI tools as part of a research design can dramatically improve the quality and depth of the research insights compared to traditional methods. As AI in research is a recommendation to consider not every project and not every open-end requires nuanced data that conversational probing provides.

Ready for Exceptional Insights?

Connect with us today to uncover the powerful possibilities for your marketing strategies.

Please contact [email protected] to connect with us.

We’ll reach out promptly to explore your research needs.

Setting Insights Higher

Socratic Technologies is a research-based consultancy that drives strategy and lifts brands. We believe in delivering high-touch client experiences as a cornerstone to providing irreplaceable value.