The survey Academy Series: 4th Chapter
Most of the questions used in a survey can only be answered by a closed set of options (aka “closed questions”). However, it is also possible to ask questions that allow the respondent to answer anything, without a list of possible alternatives (aka “open ended questions”).
There are two main types of open ended questions (Couper et al, 2011): ones that lead to very short answers (e.g., place of birth, or preferred beer brand) and ones that require narrative responses in the form of more elaborate and detailed responses (e.g., what do you think will be the next challenges for you in the future?).
Open ended questions to collect narrative responses have some pros and cons. On one hand, answers cannot directly be used for quantitative analysis, as coding needs to be done to be able to work with these responses. This coding can be time consuming and quite complicated if the answers given by the respondents are not directly in line with what the researcher wanted to measure (see our previous post on questionnaire design). On the other hand, they allow us to obtain a much wider range of responses, with more detailed content and information. Additionally, respondents are not guided in any particular direction, which allows them to answer in a more natural way that resembles a dialogue, giving greater freedom to the respondents to give accurate answers. Hence, these types of questions tend to better suit the researcher’s needs than closed questions.
In face-to-face or phone surveys, the respondent can give an answer of the length of their choice. But in an online survey, the researcher needs to decide the best way of presenting the open ended question. Particularly, two decisions have to be made: the text box size (the visual box in which the respondent writes the answer) and whether or not to include a counter indicating how many characters are available.
Results from different research projects on the impact of the size of the text box (Christian and Dillman, 2004; Israel, 2010; Smyth et al. 2009) agree that a bigger text box produces longer responses. There is also scientific evidence showing that the use of a counter can lead to an increase in the average number of characters in the responses (Emde y Fuchs, 2012).
Moreover, Emde and Fuchs (2013) studied the interaction between the text box size and the number of characters of the counter. The results are quite intuitive: a counter with more characters than the text box size shown at the beginning produces longer answers, whereas a counter with fewer characters leads to shorter responses. Therefore, it seems that respondents take the counter into account more (when there is only one) than the reference provided by the size of the text box.
Furthermore, Emde and Fuchs (2013) also conclude that both the text box size (150 characters or 300 or 600) and the counter presence do not affect the nonresponse rate, and neither the number average of mentioned topics. Therefore, what really changes is the level of detail in the provided answers.
In conclusion, to collect narrative responses with open ended questions, it seems advisable to use a text box in combination with a counter which indicates more characters than initially visible (up to 33% more). It is preferable to use a relatively large text box, without being too excessive, since combining it with a counter with more characters means answers will be longer and, moreover, it will not change the number of mentioned topics.
Bibliographic references:
- Christian, L. M., and Dillman, D. A. (2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68(1), 57-80.
- Couper, M. P., Kennedy, C., Conrad, F. G., y Tourangeau, R. (2011). Designing input fields for non-narrative open-ended responses in web surveys. Journal of Official Statistics, 27(1), 65-85
- Emde, M., and Fuchs, M. (2012). Using Adaptive Questionnaire Design in Open-Ended Questions: A Field Experiment. In JSM Proceedings, Statistical Computing Section. Alexandria, VA: American Statistical Association
- Emde, M., and M. Fuchs (2013). “Using interactive feedback to enhance response quality in Web surveys: The case of open-ended questions”. Presentación GOR conferencia 2013 (Mannheim): https://www.conftool.net/en/index.html
- Israel, G. D. (2010). Effects of answer space size on responses to open-ended questions in mail surveys. Journal of Official Statistics, 26(2), 271-285.
- Smyth, J. D., Dillman, D. A., Christian, L. M., y Mcbride, M. (2009). Open-ended questions in web surveys. Can increasing the size of answer boxes and providing extra verbal instructions improve response quality? Public Opinion Quarterly, 73(2), 325-337.
