News

Online User Ratings Are Bullshit

Every day, consumers make innumerable decisions based on those little gold stars on sites like Yelp, AllRecipes.com or Amazon. But that doesn't make the information they convey true. In fact, your trust in those astral ratings is misguided...

Amazon screenshot

Every day, consumers make innumerable decisions based on those little gold stars on sites like Yelp, AllRecipes.com or Amazon. But that doesn't make the information they convey true. In fact, your trust in those astral ratings is simply misguided.

A new study from the University of Colorado Boulder published in the Journal of Consumer Research examined 1,272 products in 120 product categories to prove that user reviews are just a bunch of hot ham water. The results showed little correlation between average user ratings online and product ratings based on objective testing in consumer reports. Basically, people on the internet know exactly as much as Jon Snow.

"The likelihood that an item with a higher user rating performs objectively better than an item with a lower user rating is only 57%," said Bart de Langhe, author of the study. "User ratings provide very little insight about objective product performance."

While a lot of shoppers reference consumer reports for objective analysis, those product reviews tend to be about industry standards rather than the way people live with products, and that's part of what may have started to drive people to user reviews. The chile pepper handle on your quesadilla maker doesn't objectively make your quesadillas taste better, but that doesn't stop it from being your favorite part of the device and bumping up the rating by a star.

Adding another dimension to the research, de Langhe and cohorts write that user reviews do a poor job of predicting the resale value of used products, another way of determining the quality.

"Products with better reliability and performance retain more of their value over time," said de Langhe. "If average user ratings reflect objective quality, they should correlate positively with resale values. The fact that they don't casts more doubt on the validity of user ratings."

In essence, nothing stops your friend who puts ketchup on macaroni and cheese from adding Yelp ratings daily to complain about how a restaurant's mac and cheese doesn't have ketchup on it. Those ratings, over time, influence other consumers' decisions.

"This is a mistake," said de Langhe. "Oftentimes, there are just not enough ratings for a product or there is too much disagreement among reviewers. In this case, consumers should not trust the average very much, but they do nonetheless."

The study's authors say you should be more cautious when using user reviews to make purchasing decisions. Looking deeper into the number of reviews that construct an average review or finding a reviewer you trust can go a long way to improving results. However, if you're choosing restaurants based on a five-star review from a guy who also extolls the virtues of Hitch, it may be too late for you already. 


Sign up here for our daily Thrillist email and subscribe here for our YouTube channel to get your fix of the best in food/drink/fun.

Dustin Nelson is a Senior Writer on the news team at Thrillist. Follow him @dlukenelson.