Study: Addressing 'mischievous responders' would increase validity of adolescent research

"Mischievous responders" play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these "jokester youths," according to research published online today in Educational Researcher, a peer-reviewed journal of the American Educational Research Association (AERA).

"Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions", by Joseph P. Robinson-Cimpian of the University of Illinois at Urbana-Champaign, suggests that mischievous responders are present in commonly used data sets and that their presence may severely alter impressions of relative risk between groups of adolescents. The article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders often introduce in research on adolescents.

Mischievous responders are youths who provide extreme and potentially untruthful responses to multiple questions on self-administered questionnaires (SAQs). By providing misleading responses that they think are funny, these responders, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

In his article, Robinson-Cimpian demonstrates that even a very small proportion of mischievous responders can lead to exaggerated risk estimates for lesbian, gay, bisexual, and questioning (LGBQ), transgender, and disabled youths, with regard to drug usage, suicidal thoughts, and school disengagement. For example, potentially mischievous responders caused LGBQ–heterosexual disparities in recent cocaine/crack use to inflate from 4 percentage points to 12 percentage points.

This video is not supported by your browser at this time.
Author Joseph P. Robinson-Cimpian discusses key findings.

To identify mischievous responders, Robinson-Cimpian's method relies on detecting patterns of unusual answers provided by survey respondents.

"If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid," said Robinson-Cimpian. "Just like these jokester youths think it's funny to say that they are gay and blind, they also think it's funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk."

SAQs are often used to estimate the psychological and health disparities between groups, including adoptees and non-adoptees, sexual minorities and non-minorities, racial/ethnic minorities and non-minorities, and individuals with and without disabilities, among others. The anonymity often provided by SAQs makes them an important tool in adolescent research, but Robinson-Cimpian cautions that researchers must take steps to help ensure the data are valid.

"Within the past decade, much research has been criticized for failure to replicate and for exaggerated results," wrote Robinson-Cimpian in his article. He also said, "If we want sound research and policy, we need to have sound data. The procedures introduced here have broad relevance to research and can be widely, and easily, implemented."

The proposed approach does not require additional data collection, which makes it more practical, cost-effective, and applicable to already collected data. Robinson-Cimpian's technique is unique in that it can be used in situations where it is impossible or unethical to verify information by comparing adolescents' responses with those of their parents or by conducting in-person follow-up interviews with the respondents.

The four-step sensitivity analysis proposed by Robinson-Cimpian requires first identifying youths who provide high numbers of low-frequency responses and then comparing estimated disparities when including and excluding these youths.

While it is possible that some respondents will be mistakenly identified as mischievous, or that some mischievous respondents will not provide enough low-frequency responses to be identified as mischievous, there is great potential for both scientific and practical damage when no analysis is performed to assess data validity and ensure robust results, according to Robinson-Cimpian. This includes perpetuated stigma against minority groups.

For instance, this type of sensitivity analysis may have prevented well-intentioned research from being used by groups that claim LGBTQ identification leads to substance abuse and other risky behaviors, and thus advocate against using school resources to help LGBTQ youths, wrote Robinson-Cimpian in his article.

Provided by American Educational Research Association

5 /5 (2 votes)
add to favorites email to friend print save as pdf

Related Stories

Study explores why gay, lesbian teens binge drink

May 03, 2014

Higher rates of binge drinking by lesbian and gay adolescents compared to their heterosexual peers may be due to chronic stress caused by difficult social situations, according to a study to be presented Saturday, May 3, ...

LGBT youth face greater cancer risks

Feb 19, 2014

A new study led by City College of New York psychologist Margaret Rosario found that youths of same-sex orientation are more likely to engage in behaviors associated with cancer risk than heterosexuals. The peer-reviewed ...

Recommended for you

Controlling childbirth pain tied to lower depression risk

1 hour ago

Controlling pain during childbirth and post delivery may reduce the risk of postpartum depression, writes Katherine Wisner, M.D., a Northwestern Medicine® perinatal psychiatrist, in a July 23 editorial in Anesthesia & An ...

How children categorize living things

10 hours ago

How would a child respond to this question? Would his or her list be full of relatives, animals from movies and books, or perhaps neighborhood pets? Would the poppies blooming on the front steps make the list or the oak tree ...

Preschoolers can reflect on what they don't know

11 hours ago

Contrary to previous assumptions, researchers find that preschoolers are able to gauge the strength of their memories and make decisions based on their self-assessments. The study findings are published in ...

User comments