The ISO26362 standard for access panels –thus applicable to online panels- was born with the purpose of providing research clients more clarity on how said data sources are created and managed. This is a key goal in a digitalized environment where clients can hardly see with their own eyes how the data for their research are obtained.
We, the companies who have certified our panels, know that this standard does not impose certain quality criteria, probably because it has not been proven which methods for recruiting, retaining and researching panelists offer the best results to the client. This standard does not state whether the better option is to have an open or closed panel to voluntary registration. It does not set a minimum incentive level for participants in research studies, nor does it specify whether it is necessary to pay for filtered participations. This standard only requires certified panels to explain how they solve each of these questions, among many others.
However, given that the standard’s very purpose must continue to be transparency, we strongly believe that it is necessary to apply a very simple change that would greatly boost this purpose: redefining the way in which panels calculate their size. Now is the time to make this change, when the ISO committee in charge has decided to merge standards 26362 and 20525 and review their content.
Size does matter
The size declared by most online panels has always been subject of discussion. We are stunned witnesses of the publication of colossal sizes, several hundreds of thousands of people. However, when these panels are requested to use a sample of 5,000 people, the response is not any less surprising: “this project is not viable.” How can a panel of several hundreds of thousands of people not be able to complete a survey of 5,000 participants? The problem lies in how size is calculated.
Standard 26362 attempted to bring order to this criteria disparity. In order to do so, it defines an active panelist, one who can be considered in calculating the size of a panel, as one who “has participated in at least one survey in the past year.” So if a panelist is recruited on 1 January and we get them to do a survey, they are considered panelists until 31 December even if they do not take part in any other survey.
The current definition does not offer a real vision of a panel’s size and, consequently, goes against the general purpose of the standard. This can be easily understandable with a simple example.
Let us suppose that we have two panels (A and B) with over 50,000 unique panelists to complete surveys. That is, if a client asks to survey the whole panel on one single survey, each panel is able to provide 50,000 responses by different people.
Let us suppose that panel A’s monthly churn rate is 1% while that of panel B is 50%. Let us also suppose that both panels intend to keep a production capacity of 50,000 respondents, so they must replace their monthly dropouts.
In this case, panels A and B will have to recruit the following number of panelists per year:
A – 50,000 x 1% x 12 months = 6,000 panelists
B – 50,000 x 50% x 12 months = 350,000 panelists
Given that both panelists which are already part of the pool and panelists recruited to replace dropouts have participated in at least one survey during the past year, the ISO size of both panels ends up being:
A – 50,000 + 6,000 = 56,000
B – 50,000 + 350,000 = 400,000
Therefore, we have two panels that objectively show the same capacity. However, one of them declares having a size –according to the norm- almost 8 times bigger than the other one.
A new definition for panel size
Here at Netquest we propose a new definition closer to the reality of being an active panelist, that is, someone who will take part in the next survey to which they are invited. With this criterion, the size of the panel becomes “the largest sample” that a panel can offer to a client (maximum delivery of the panel.)
There are various alternatives as to how a panel must calculate this maximum delivery. A simple and effective way to do so is to consider as active “every panelist that has responded to the last survey to which they were invited, without considering invitations sent out in the past 15 days, which the panelist might have not had time to respond yet, but not due to lack of will.” This method calculates the latest largest sample the panel was able to offer as the best indicator of its current maximum delivery.
Benefits of change
These are the main reasons why we request the support of the industry to change this key aspect of the standard.
- The current definition is just wrong. Our proposal suits exactly what clients understand by the size of a panel. It is difficult to explain to clients why a panel of a supposed size of 400,000 people can only deliver 50,000 responses for a particular survey.
- The current definition paradoxically favors panels that are worse at retaining participants, thus making it difficult to differentiate true online panels from river sampling systems (occasional survey participants.) The new definition would render a clear explanation to clients regarding the kind of panels they are hiring.
- The new definition would force the panel to be more transparent regarding its processes, a goal already present in the ISO standard spirit. The current definition makes publishing unrealistically high panel sizes easier, thus allowing the panel to cash in projects that surpass their real production capacity, outsourcing the remainder of the sample to third parties in a rather not transparent manner for the client.
- This change not only promotes transparency but also quality. Knowing that a panel’s real size is small shows that they are probably over surveying panelists (generating many surveys with the same respondents.) It also shows a high panelist rotation, and therefore their quality cannot be validated by a checking method that requires comparing their behavior throughout time (as the current standard states.)
- The new definition would allow to precisely measure a panel company’s real assets, thus providing more clarity to shareholders and facilitating any future purchase sale company transactions.
For all these reasons, we request the industry’s support to change this aspect of the standard. This small change could boost and improve our sector’s standards, transcending the group of certified companies and creating a fair and simple scale with which one can calculate panels’ real capacity.
