The Trouble With Zagat

The Zagat Guide is a wonderful restaurant directory. It allows you to search on a wide variety of criteria (neighborhood, cuisine, etc.), and it provides just about all of the basic information you need (address, phone number, hours, map, price range). The comments provided, although brief, are often witty and scathingly accurate.

But the one area where Zagat falls down is the statistic most often quoted, and for which Zagat is best known: the numeric ratings of each restaurant. Zagat separately rates Food, Décor and Service on a 1-to-30 scale. If properly used, this scale would provide sufficient amplitude to distinguish the neighborhood taco stand from Alain Ducasse and Per Se. In practice, it does nothing of the kind. This is ironic, given that restaurants love to post their so-called “Zagat rating,” and some will say that they’re “Zagat rated.” What is this so-called “rating”?

For starters, Zagat is a raw popularity contest, with very little guidance given to the voters. Someone who thinks Olive Garden is a pretty good restaurant is going to rate all of Little Italy off-the-charts, while an experienced high-end diner will pooh-pooh anyplace that lacks a chef’s tasting menu. The upshot is you have a hot dog stand like Gray’s Papaya carrying a Zagat food rating of 20 out of 30, which (according to Zagat’s own definitions) is supposed to mean “very good to excellent,” when the highest rating in New York is just 28.

Zagat’s own voting mechanism is largely at fault. Individual voters are allowed to vote on a 0-to-3 scale. Zagat says that “1” is supposed to mean “good,” but psychologically a “1” vote feels like “below average.” People will realize that “3” must be pretty damned good, so there’s a tendency for almost everything to get rated “2”. For the final rating, Zagat multiplies the average by 10 and rounds off, resulting in the familiar 1-to-30 scale.

A look at the details shows that this is a serious problem. Of the 1,454 restaurants in Zagat’s 2003 New York guide, 74% of them carry a food rating between 18 and 23. What’s more, 97% of them carry a food rating of 16 or higher, and none carry a food rating worse than 9. The upshot is that what’s claimed to be a 1–30 scale is, for all practical purposes, a 16–28 scale. You can safely say that any restaurant with a Zagat rating of 25 or higher is very good. But ratings below 25, which is almost all of them, are in an undifferentiated scrum, and aren’t statistically significant.

Oddly, voters are considerably more discriminating in their Décor ratings: just 62% of New York restaurants have a Décor rating 16 or higher, and just 36% are clustered in the 18–23 range. When it comes to Service, Zagat voters rate about 80% of restaurants 16 or higher, and 54% are in the 18–23 range. So the Zagat Service ratings are nearly, but not quite, as useless as the Food ratings, while the Décor ratings actually do seem to mean something.

The pernicious tendency of the ratings to cluster around 20 is shown in the following graph:

I am not sure why voters are least discriminating about the one thing that should matter most at a restuarant - the food - but perhaps it’s because the qualities that make food great are awfully difficult to describe. Yet, everyone knows an ugly room when they see it.

I think Zagat would be considerably more reliable if they collected votes on the same scale they report, from 0 to 30. Voters would then tend to rate an average restaurant “15”, instead of “2”. The higher Zagat ratings would be harder to get, and the scale overall would be a lot more meaningful.

[Update: After I posted this, a colleague on eGullet observed that the Zagat food ratings are almost a proper bell curve, if you consider “average” to be 20 rather than 15. The problem is that the standard deviation is only about 2, which means that the scale simply fails to offer a meaningful spread between the best and the worst.]

Of course, there’s no chance of Tim and Nina actually changing anything, so the Zagat ratings will continue to be the least useful part of what is otherwise a very useful service.