And when I tried to dig in even further, it just created more confusion/nervousness/philosophical dread: At the bottom, there is a note breaking down the Yelp Data Science team’s methodology. It mentions “businesses from across the country were compared using a ranking that looks at both the ratings and the number of reviews while accounting for quality, popularity, and statistical fluctuations.” Who is accounting for quality and popularity? The Yelp Data Science Team? (Response from Yelp: "Yep! Our data science team created an algorithm to allow comparison of businesses across the nation with the factors included in the methodology").
But let’s take a deeper look into the “ratings and the number of reviews” section. Most of the places on the list average between 1500 and 4000 reviews. On the high end, Phil’s BBQ in San Diego has around 10,000 (and is on our best BBQ list, and delicious). But the low end is where it gets interesting and shows an even more absurd West Coast bias. At #22, Halls Chophouse in Charleston, SC is the number one rated place on the East Coast. It only has 640 reviews, which is well below the average, but it has a perfect five star rating, so you can kind of see how that might prop it up. BUT, sitting one spot higher, is a coffee and tea spot in Vegas with only 732 reviews and a 4.5 star rating.
The absurdity is further confirmed by looking at the top ten. Here is a breakdown of the top, with number of reviews followed by the star rating.
1. 5199 reviews - 4.5 stars
2. 4002 - 4.5
3. 4182 - 4.5
4. 1872 - 4.5
5. 668 - 5
6. 4384 - 4.5
7. 1892 - 4.5
8. 3454 - 4.5
9. 2674 - 4.5
10. 4181 - 4.5
That one five star outlier with a drastically smaller amount of reviews? TKB Bakery in Indio, CA. And this is where the quality and popularity line gets really confusing. What makes TKB better than Halls, which sits at 21? They both have perfect ratings (highest quality), and TKB only has 28 more reviews (same level of popularity). The only difference I can see is that only one is in that beautiful water starved state where famous people practice science-fiction based religions.
Further parsing the methodology, you run into another strange phrase: “Businesses must be primarily a restaurant or place to eat a meal to be included on the list and must abide by Yelp’s Terms of Service and Content guidelines.” Now I understand Yelp’s Terms of Service for users (you can read it here), but how does this apply to restaurants? They must’ve signed off on something to be considered here? Rachel from Yelp responded to this too, saying that "Our TOS and Content guidelines apply to all users, consumer and business side. There's a specific business section at the bottom of our content guidelines too."
All in all, I would not care about any of this, or waste an absurd amount of time picking it apart and driving myself mad if I didn’t feel like this matters. As you know, we, too, create lists of restaurants. And bars. And lots of other things. I specifically have been in charge of writing the national lists of best restaurants for four years. And when we first put out these lists years back, I definitely made embarrassing mistakes and omissions.
But once I started tapping into our huge network of staffers and trusted freelancers around the country and having say our Alabama-bred New Orleans-based editor pick the South, we started getting it right. And I came to realize that this stuff is extremely important to get right, and not just for the service journalism principle of it. It’s important because the restaurant industry is fucking hard and making a living in it is fucking hard and if you’re doing cool delicious things that can be celebrated and publicized, you deserve to get a nod because you likely need all the help you can get. So being on a highly searchable, highly publicized list of the Top 100 Places to Eat in the US is probably pretty huge for a small mom and pop sandwich shop or rib joint. It’d just be cool if they weren’t all in California.
Sign up here for our daily Thrillist email, and get your fix of the best in food/drink/fun.
Kevin Alexander is Thrillist’s National Writer-at-Large but kind of wants to join the Yelp Data Science Team. Follow his attempts to make LinkedIn connections @KAlexander03.