Yelp's Top 100 Places to Eat Is Wrong & Kind of Crazy: An Over-Analysis
Earlier this week, Yelp’s Official Blog posted a story listing its Top 100 Places to Eat in the US for 2016. I saw it on television. Our news team covered it. Regional sites wrote quick stories about local restaurants that were named. And Twitter, a semi-private club exclusively used by businesses, self-congratulatory journalists, and PR interns working for celebrities, filled up with back-slapping tweets.
Also, it is not a great list.
Though listicle backlash is in serious bloom right now, and bad listicles speak to everything the Internet does wrong -- lazy journalism, poaching off of other people’s work, just creating a thing because people can’t resist looking at it -- Yelp, to their credit, are not doing that (if anything, many of the worst listicle offenders likely poach ideas in regions they’ve never been to, from positive Yelp reviews).
They have tons of original data from hundreds of thousands of people who feel very compelled to offer up ratings and tell you about their club sandwiches and Yelp attempted somewhat admirably to parse this data and these opinions into some sort of a national list. The way they’ve framed it is “businesses that rank so highly in the Yelp community’s opinion that they have earned the status of ‘must try within this lifetime.’” And that is all well and good, until you actually read through the list.
The first and craziest of issues: 45 of the 100 places included are in California. Quite incredibly, they’re saying that nearly half of the spots you need to try in America are in one state. In fact, only 19 states are represented in this list, and 66 of the 100 restaurants mentioned are on the West Coast. One tenth of the list comes from the tiny island state of Hawaii. By this logic, 31 states do not, according to the Yelp community’s opinion, have any restaurants or sandwich shops or delis or pancake houses or rib joints in the top 100 worth trying this lifetime. None of the Jucy Lucy spots in Minnesota. Or the Christmas enchilada joints in New Mexico. Or the lobster roll shacks in Maine. Or whatever they call the places that sell chislic in South Dakota.
(I reached out to Yelp to ask if they were worried that a list purporting to represent all of America skews so hard towards one state and generally to the West, and here is their reply from Rachel Walker, who posted the list: "Yes, that's an accurate observation that there are lots of California businesses. The list represents opinions of the Yelp community, which is strongest in some of our oldest communities and may contribute to why you see many California businesses represented. Our data science team used statistical analysis to compare businesses across the country, but population was not specifically a factor")
Look, I live in San Francisco where Yelp is headquartered, so even if I could suspend my disbelief that it’s a coincidence that nearly half of all must-try restaurants happen to be located in the state with Yelp’s main office (and 12 of those in greater San Francisco/Napa), I still feel like — just sticking to the places I know personally — it’s basically a list version of Hunger Games tribute selection: a bunch of seemingly random names pulled out to represent different districts. Gary Danko, at #6 on the list, is likely the fourth best very fancy restaurant in San Francisco. Lou’s Cafe (#53) is a pretty good sandwich spot in the Richmond District (Roxie Food Center, at #95 is better).
Gramercy Tavern, a classic Danny Meyer restaurant that opened in 1994, is the first NYC restaurant included, and it’s #32. Quiet sushi delis by my dad’s house in San Diego sit above Eleven Madison Park in NY. The human brain demands order and reason to help parse information but the list just feels so hopelessly arbitrary, it’s making me both nervous and existential.
Yelp is saying that nearly half of the spots you need to try are in one single state.
And when I tried to dig in even further, it just created more confusion/nervousness/philosophical dread: At the bottom, there is a note breaking down the Yelp Data Science team’s methodology. It mentions “businesses from across the country were compared using a ranking that looks at both the ratings and the number of reviews while accounting for quality, popularity, and statistical fluctuations.” Who is accounting for quality and popularity? The Yelp Data Science Team? (Response from Yelp: "Yep! Our data science team created an algorithm to allow comparison of businesses across the nation with the factors included in the methodology").
But let’s take a deeper look into the “ratings and the number of reviews” section. Most of the places on the list average between 1500 and 4000 reviews. On the high end, Phil’s BBQ in San Diego has around 10,000 (and is on our best BBQ list, and delicious). But the low end is where it gets interesting and shows an even more absurd West Coast bias. At #22, Halls Chophouse in Charleston, SC is the number one rated place on the East Coast. It only has 640 reviews, which is well below the average, but it has a perfect five star rating, so you can kind of see how that might prop it up. BUT, sitting one spot higher, is a coffee and tea spot in Vegas with only 732 reviews and a 4.5 star rating.
The absurdity is further confirmed by looking at the top ten. Here is a breakdown of the top, with number of reviews followed by the star rating.
1. 5199 reviews - 4.5 stars
2. 4002 - 4.5
3. 4182 - 4.5
4. 1872 - 4.5
5. 668 - 5
6. 4384 - 4.5
7. 1892 - 4.5
8. 3454 - 4.5
9. 2674 - 4.5
10. 4181 - 4.5
That one five star outlier with a drastically smaller amount of reviews? TKB Bakery in Indio, CA. And this is where the quality and popularity line gets really confusing. What makes TKB better than Halls, which sits at 21? They both have perfect ratings (highest quality), and TKB only has 28 more reviews (same level of popularity). The only difference I can see is that only one is in that beautiful water starved state where famous people practice science-fiction based religions.
Further parsing the methodology, you run into another strange phrase: “Businesses must be primarily a restaurant or place to eat a meal to be included on the list and must abide by Yelp’s Terms of Service and Content guidelines.” Now I understand Yelp’s Terms of Service for users (you can read it here), but how does this apply to restaurants? They must’ve signed off on something to be considered here? Rachel from Yelp responded to this too, saying that "Our TOS and Content guidelines apply to all users, consumer and business side. There's a specific business section at the bottom of our content guidelines too."
All in all, I would not care about any of this, or waste an absurd amount of time picking it apart and driving myself mad if I didn’t feel like this matters. As you know, we, too, create lists of restaurants. And bars. And lots of other things. I specifically have been in charge of writing the national lists of best restaurants for four years. And when we first put out these lists years back, I definitely made embarrassing mistakes and omissions.
But once I started tapping into our huge network of staffers and trusted freelancers around the country and having say our Alabama-bred New Orleans-based editor pick the South, we started getting it right. And I came to realize that this stuff is extremely important to get right, and not just for the service journalism principle of it. It’s important because the restaurant industry is fucking hard and making a living in it is fucking hard and if you’re doing cool delicious things that can be celebrated and publicized, you deserve to get a nod because you likely need all the help you can get. So being on a highly searchable, highly publicized list of the Top 100 Places to Eat in the US is probably pretty huge for a small mom and pop sandwich shop or rib joint. It’d just be cool if they weren’t all in California.
Sign up here for our daily Thrillist email, and get your fix of the best in food/drink/fun.