On a previous post we told you about the Online Panels Mystery Shopping exercise done by our team. Let’s do a recap: our team created two fictional panelists identities and then joined 10 Online Panels, including Netquest’s, with the objective of comparing the differences in panel management by experiencing it through the panelist's eyes.
This investigation analyzed different characteristics, such as the registration process, the demographic information collected, the panels’ privacy policies and their incentivization system, among others. Today we want to talk to you about a no minor aspect: Surveys’ quality controls.
In the past, we’ve talked about the importance behind the collection of data used in the market research industry and said that this data is not only used when it comes to making a decision in product testing but it can also be used for political opinion purposes.
Now, let’s look at the seven survey quality controls compared by our team among the online panels, and let us explain why it is important to always take them into account:
1. Double opt-in
It’s the double confirmation and authorization of a person to receive certain communications from a company. The difference here, is that in the case of the double opt-in a person does not only check a box, like in the regular opt-in, but will also receive an email to confirm his email address. Its importance lies in the fact that through a two easy steps process, a person gives his full consent to be included in a mailing list.
In this category, we detected that 7 out of the 10 panels reviewed do have a double opt-in process, including the Netquest panel.
2. Identity verification
Imagine you meet someone online, and that in the communication you maintain with that person, he or she sends you pictures of himself or herself, and when you both finally agree to meet in person, someone completely different shows up; surely you would be very confused.
So, in order to prevent a different person from supplanting the panelist and we end up delivering poor quality data, the Netquest panel asks its panelists two questions, regarding their identity, at the beginning of each survey, if the answers turn out to be different from the ones registered in the panelist profile, the supplanter won't be able to answer the survey.
Only Netquest and another panel have an identity verification process. In the other 8 panels, we were able to check that 3 of them do not have this process; and as to the other 5, we were not able to verify it.
3. Re-captcha
Yes, it is a double captcha process. By using this test you can verify that people answering surveys are actually humans, and although it may seem too obvious, it makes sure the respondent is not a bot interacting with a website.
The difference between a captcha and a re-captcha is that the second one, besides asking a person to click on the “I’m not a robot” box, is usually accompanied with some other process that verifies that the web navigation looks like the one a human would do, for example, with the IP address and the use of cookies.
Six out of the ten panels we checked have a re-captcha process, including Netquest, meanwhile, we were not able to identify the use of re-captcha in any of the other four panels.
4. “Red herring” questions
Have you ever had to answer a surprise quiz? Many teachers use this type of test to make sure their students have been learning, and more importantly, that their students have been paying attention.
If this was a survey, we would ask you “Without going back to the beginning of this post, could you name and explain the first point mentioned?”
“Red herring” or trick questions have the intention to detect if a panelist is or isn't paying attention. Even though there are different types of trick questions, they all can be summed up as a control inserted in the survey, questions that require a specific response on the side of the respondent to allow him or her to move forward.
We were able to identify their usage in just 2 out of the 10 evaluated panels. Of course, Netquest panel counts with a system of trick questions in its surveys to make sure panelists are answering consciously.
5. IP Control
Surely you´ve heard about the IP address, but do you know what it is? The IP address is a numeric code used to identify a device when it is online. These numbers composing the code indicate a series of properties about the device, for example, the country from where the device is connecting to the web.
With an IP control, we can make sure that panelists really live in the country they´ve indicated in their registration, reducing the probabilities of us delivering information or data with poor quality.
Regarding this quality control, we identified that only one panel besides Netquest has an IP Control. For the rest of the panels, it was not possible to identify it.
6. Answer consistency
Have you noticed that during your visits to the doctor, he or she asks you the same thing more than once, in different ways? For example: Where do you feel pain or pressure? a couple of questions later, Does it hurt when I apply pressure on the abdomen? Well, doctors do this to make sure that your symptoms match the diagnosis they’re about to give you.
In this same way, surveys must be consistent in their flux with the answers given by the panelists; this makes sure that every answer is honest and the results obtained with them are going to provide quality data. Out of the 10 reviewed panels, we were only able to identify that the Netquest panel applies answer consistency controls.
7. Speeder
If someone says to you, it took him only 3 hours to read the complete saga of Lord of the Rings, would you believe him?
Although surveys are not as extensive or complex as Tolkien’s work, it is still necessary to measure the time it takes a panelist to answer, and in case we detect that the answering time is way below the estimated, that panelist gets filtered out or disqualified.
In a fast reading, it is very likely that a panelist omits details, that he or she is not paying enough attention or even choosing the answers randomly. This speed control, or speeder, was not detected in eight panels, while in the remaining 2 in which the speed control was detected, one of them is the Netquest panel.
And- in any case you want to know the number- it takes 26 hours to read the Lord of the Rings saga, much more than what it took you to read up to here, so now that we’ve reviewed these seven basic quality controls we want to invite you to think: for your next research, how many of this quality controls does the panel you are thinking on has? Does that panel care about survey quality controls?
Remember “you get what you pay for”, and avoid to commit the integrity of the information you are going to use to make a decision or to generate that insight you so badly want to share with the rest of the world.
If you want to know more about our panel, about how we fulfill the quality requirements to get ISO norm certification for Online Panels, and the care we invest in survey quality, don’t hesitate to contact one of our representatives.
