Ratings rise over time because they feel easier to make

March 12, 2018, Association for Psychological Science
Ratings rise over time because they feel easier to make
Credit: Association for Psychological Science

Tasks often feel easier to perform as we gain experience with them, which can have unintended consequences when the task involves rating a series of items, according to findings published in Psychological Science, a journal of the Association for Psychological Science. The findings show that people tend to attribute the increasing ease of making ratings to the items themselves rather than to the ratings process, resulting in rating inflation over time.

"We find that increased experience makes the evaluation process easier which, in many instances, leads to an upward trend in judges' evaluations," says study author Kieran O'Connor of the McIntire School of Commerce at the University of Virginia. "This effect emerged with judges on a dance show, with teachers who give higher grades the longer they teach a course, and in the lab where we have people evaluate photos or short stories over successive days."

Given that sequential ratings are a part of many everyday activities, this research has potentially wide-reaching implications.

"These findings suggest that people new to an evaluation task may be more critical than those who have been doing the evaluation task for longer," O'Connor notes. "This can bias decisions about hiring, employee evaluations, grading, and scoring - simply based on the experience of the evaluator."

In one study, O'Connor and coauthor Amar Cheema examined judges' ratings from the TV show Dancing With the Stars. The researchers specifically looked at 5,511 scores made by the three core judges over 20 seasons of the show. They found that the judges' average scores increased season by season over the 20-season run. Additional analyses indicate that the rise in ratings was not due to professional dance partners' increasing experience with the show, or increased dance ability, over successive seasons.

Ratings inflation also emerged in a domain with broad relevance: Student grades. The researchers analyzed data from 991 course sections offered at a large US university over successive spring and fall semesters from 2000 to 2009. They found that successive sections offered by the same instructor had higher grades over time, a phenomenon that couldn't be explained by instructor ability or student performance improving over time, or even by calendar year (i.e., general grade inflation over time).

Studying the phenomenon in a controlled experimental setting, O'Connor and Cheema also found the rating inflation effect in students' ratings of short stories. Students evaluated 1 story per day over a 2-week period, evaluating a total of 10 stories presented in random order. Analyses of 1,572 observations from 168 participants showed that participants tended to rate each successive story more positively than the previous one. Participants also reported that the rating process became easier, quicker, and more enjoyable over time, but did not believe that their ratings became more favorable.

Another online study produced a similar result, showing that participants perceived the ratings process as more fluent—that is, easier and quicker—as they evaluated more stories, which boosted their ratings over time. Despite this, participants remained unaware of the study aims.

The inflation effect may be small, the researchers note, but it is consistent across different contexts and can influence outcomes in meaningful ways. In the college grades study, for example, course GPA increased on average from a B+ to an A- over approximately 20 successive offerings.

Typical approaches to ensuring objectivity, such as randomizing the order of items, will not address the inflation effect. As such, it is important to understand the factors that could actually mitigate this bias:

"Making people aware of this pattern, and perhaps providing them an explicit standard that they should stick to when evaluating, may be the way to go," O'Connor says.

Explore further: Study says some nursing homes gaming the system to improve their Medicare star ratings

More information: Kieran O'Connor et al, Do Evaluations Rise With Experience?, Psychological Science (2018). DOI: 10.1177/0956797617744517

Related Stories

Study says some nursing homes gaming the system to improve their Medicare star ratings

January 17, 2018
For families faced with the difficult decision of placing a loved one in a nursing home, a government rating system is often the only source of information to determine which facilities are the best. However, a new study ...

Women beat expectations when playing chess against men, according to new research

January 30, 2018
Data from 160,000 ranked chess players and more than five million chess matches suggests that women playing against men perform better than expected based on their official chess ratings, according to a new study by the University ...

Online students give instructors higher marks if they think instructors are men

December 9, 2014
A new study shows that college students in online courses give better evaluations to instructors they think are men - even when the instructor is actually a woman.

Consumers' trust in online user ratings misplaced, study says

April 28, 2016
The belief that online user ratings are good indicators of product quality is largely an illusion, according to a new University of Colorado Boulder study.

Our psychological biases mean order matters when we judge items in sequence

January 25, 2017
We often need to make decisions about sequences of things or people rather than just a single item in isolation. For instance, in an everyday setting, we might choose which smartphone to buy after trying out several. There ...

Online ratings not aligned with objective quality measures

October 17, 2017
(HealthDay)—Online consumer ratings of specialist physicians do not predict objective measures of quality of care or peer assessment of clinical performance, according to a study published online Sept. 8 in the Journal ...

Recommended for you

New brain research suggests that schizophrenia is an extreme version of a common personality type

September 18, 2018
Researchers have found that the signals in people's brains differ depending on a particular aspect of an individual's personality, termed Schizotypy, a discovery that could improve the way schizophrenia is characterised and ...

Why the brain struggles to get off the sofa

September 18, 2018
About 30% of adults and 80% of teenagers today do not meet the minimum levels of daily physical activity for staying healthy, as recommended by the World Health Organisation (WHO). Previous studies have already demonstrated ...

New era in virtual reality therapy for common phobias

September 18, 2018
Dick Tracey didn't have to visit a tall building to get over his fear of heights. He put on a virtual reality headset.

Do we trust people who speak with an accent?

September 18, 2018
You are in a strange neighbourhood, your cell phone's dead, and you desperately need to find the closest garage. A couple of people on the street chime in, each sending you in opposite directions. One person sounds like a ...

We are predisposed to forgive, new research suggests

September 17, 2018
When assessing the moral character of others, people cling to good impressions but readily adjust their opinions about those who have behaved badly, according to new research.

Being forgotten by acquaintances can affect self-esteem in the same way as being rejected

September 17, 2018
Psychologists at The University of Aberdeen looking into the experience of being forgotten have discovered that memory lapses can damage relationships.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.