Can we use AI chatbots to assist respondents with question clarification in web surveys? Revisiting an old LFS project with new tools.
Katharina Rossbach, Dag F. Gravem
Statistics Norway, Oslo
Abstract
In sample surveys, different efforts have been made mimicking dialogue features typical of interviewer administered modes, in web modes. This is partly inspired by grounding theory, which stresses that a common understanding of key concepts is a premise for a functioning conversation. An interview is regarded as a conversational and collaborative effort rather than as a rigidly standardized script.
Observation and tests show that interviewers regularly engage in such conversational interviewing, providing different types of clarifying information and calculations to obtain valid responses. This can occur both when the respondent asks for further clarification or help, or when the interviewer perceives that the respondent is unsure or has misinterpreted a question.
About a decade ago, Statistics Norway experimented with introducing conversational dialogue features in web questionnaires, specifically the Labour Force Survey (LFS). (Gravem 2013.) A "Don't know" response would trigger a probe asking what made the question difficult to answer, followed by help targeting that problem. The project was abandoned partly due to the gargantuan effort required to script questions and answers for every conceivable problem in all steps of the response process, and for all the questions of the complex LFS.
With the emergence of large language models and virtual assistants such as ChatGPT, the question presents itself of whether these new tools can do the job that was previously deemed too demanding. Both the use of chatbots in businesses and the overall research regarding how to create efficient and engaging chatbots has also grown in recent years. In this paper we want to explore and present what the current knowledge and tools can offer to conduct a hybrid web survey, illustrating it on LFS. We will explore what kind of steps must be taken and which considerations must be made when creating a large language model for a social survey such as the LFS. Although the technical challenges might be drastically reduced compared to 10 years ago, it still will take time to develop a functional chatbot. Moreover, challenges, concerns, and limitations regarding the use of chatbots have to be addressed. We will examine legal and ethical issues, respondent motivation, potential problems and limitations when integrating chatbots in web surveys.
Gravem, Dag F. (2013): Error Prevention through Interviewer Emulation? Introducing questionnaire dialogues in the Norwegian LFS questionnaire. Paper presented at the European Survey Research Conference, Ljubljana 2013.