The most consistent complaint since the onset of online Marketing Research has been sample quality. Keeping up with lazy or outright fraudulent survey respondents has been an ongoing struggle.
One of the core challenges is that there is no financial incentive for the panel providers to weed out poor-quality respondents. Every person they send through that completes and isn't removed for quality reasons is of value to them. There is no incentive for them not to send that person through as many times as they can get away with.
Socratic has been on the leading edge of sample quality since 1994 when we were founded. Early on, we had less of a problem with fraud than we did with simple lazy responders. Our efforts then were focused on engagement, gamification, and building quality data by connecting with respondents. As web-based research continued to grow and panel companies began monetizing their access to audiences, our focus shifted to ensuring that respondents are whom they say they are and meet the target qualifications.
Technical improvements - In the beginning, it was the use of simple Cookies, hard to detect Flash-Cookies until Flash was retired. The introduction of CAPTHA – reCAPTCHA – and now reCAPTCHA v3, and digital fingerprinting based on data available about the system and browser are now utilized.
Programmatic steps – In all surveys developed by Socratic, the inclusion of different approaches such as logic traps, red-herrings, convergent/divergent validity checks, and straight-lining help identify bad actors and suspicious activity.
Manual reviews - we take steps to review every record for open-end articulation and completeness. These reviews include searching the web for matches to the text entered to determine if people were copying and pasting from google searches.
These steps will only continue into the foreseeable future. While they take a great deal of time and money, it is a required endeavor to deliver high-quality data for our clients and to know that the research we are conducting is accurate and actionable.
Engagement of the participant remains a concern, but preventing fraud is by far the primary concern today. Even with all the technical and programmatic approaches to weed out fraudulent survey takers, we still find many fraudulent records that are only identifiable once you review the records manually.
Socratic's ongoing effort to constantly improve our technical controls has introduced the Fraud Risk Score™ (FRS). The FRS is a measurement of each survey respondent's potential of being fraudulent. The FRS is comprised of the following:
The result is an individual FRS rating for each respondent. The FRS ranges from 0-100. In general, we categorize the values:
0 - 74 = acceptable
75+ = suspicious
85+ = risky
90+ = high-risk
We then determine a level at which we will auto-terminate an individual or flag them for further review.
Presently, Socratic removes an average of 23% of records supplied by panel companies. In a typical consumer study. 21% are removed from the automated data checks, while an additional 2% are from the manual data checks. The Fraud Risk Score identifies an additional 4% of high-risk respondents from future data sets. It also reviews all respondents who previously passed our technical checks but were removed during manual audits. Socratic will always conduct manual checks, but the FRS helps reduce the manual checking efforts considerably and allows us to target records with higher scores that may not be beyond the quality cut point.
The Fraud Risk Score delivers more reliable data for better decision-making for Socratic's clients and their research efforts on each and every project.
Socratic is announcing the creation of the Panel Fraud Index™ (PFI™).
The Socratic PFI™ aggregates the Fraud Risk Score across respondents from each panel partner and represents an overall quality index that indicates how poorly or how well that panel performs based on the people they attempt to send to our studies. Ultimately, the higher quality of particpants they send, the lower their PFI™ score.
High-risk panel providers are flagged with high PFI™, meaning a high percentage of panelists are high risk based on their FRS and taken out of our pool of panels in use. The purpose of the PFI™ is for Socratic to help the panel partners improve the participants while bringing forward a tool that our clients and partners where panel quality can be accessed. Because the index is updated each week, the movement of scores and inclusion of panel providers will change. Giving panel companies the incentive to clean up panels of poor-quality people and collaborate toward providing higher-quality panels for research partners only helps the research industry efforts in removing bad actors at each step of the research process. The Panel Fraud Index™ can be found at https://www.sotech.com/panel-fraud-index.
