Understanding uncertainty with a new take on questionnaires

Researchers from the University of Nottingham have been working to address uncertainty in questionnaires across a variety of problems with support from the UK EPSRC and the National Cyber Security Centre (NCSC).  They have developed an interval-valued response method as an alternative to the commonly used questionnaire approaches. Their research has been published in Behavior Research Methods.

Filling out surveys, questionnaires and reviews is a part of everyday life. These are used as a way of collecting feedback and data for a range of applications, but current models are inflexible in the responses they can collect, often using scales of 1-5, likely to unlikely, or agree to disagree, where the respondent can only choose one score. 

Christian Wagner, Professor of Computer Science, has been leading the research. He explains that "current methods for collecting quantitative or qualitative data each have their own limitations and we are exploring the viability of alternatives designed to maximize data capture for effort invested. Specifically, we have focused on an issue we believe has received relatively little attention given its potential significance—the efficient capture of uncertainty as perceived by an individual respondent. We have all been faced with a conventional scale and been unsure which option to choose, whether the question is badly worded or ambiguous, or whether we don't know the exact answer or indeed feel that multiple options apply. Our research offers an alternative response system that takes this uncertainty into account, providing better quality data for little to no added effort. This has a lot of benefits, and it could even be used to improve data privacy in the future." 

The new method uses ellipses to allow respondents to choose a range when answering a question, in an intuitive and accessible way. So instead of choosing one discrete point along the scale, the respondent can draw an interval around all options that apply. The team have tested these interval-valued questionnaires in a variety of areas; from vulnerability assessments in cyber security, to consumer perceptions of products, patient understanding in healthcare, and environmental management. 

Zack Ellerby, lead author of the paper elaborates that "data is used to inform decision making across a wide range of areas, and our research shows that there is an efficient way to capture better quality data without requiring much more time or money – and that people can effectively use this approach to communicate their uncertainty."  

The research is led by the Lab for Uncertainty in Data and Decision Making (LUCID), at the University of Nottingham's School of Computer Science, and is an inter-disciplinary collaboration with partners including Carnegie Mellon University's School of Social and Decision Sciences. Christian Wagner, who leads the lab, add that "this most recent output really highlights the potential of cross-disciplinary research, linking both Computer Science and Psychology in this case. In parallel to ongoing research focussing on the analysis of the resulting data, we are already working with stakeholders in other sectors, such as consumer goods manufacturers to apply this to their business and we are keen to expand into other areas to explore how beneficial this method can be."