There’s been a lot of discussion over the last month or so about the game reviewing system. Stephen Totilo at MTV did a whole week of coverage on the subject, which I suggest you go browse. The spark for this discussion was the overwhelmingly glowing reviews of GTA 4, and has continued with arguments over Metal Gear Solid 4. Reviews are on the industry’s mind, and everyone thinks they know what’s wrong with them. So being egotistical and analytical, I thought I’d conclusively answer this.

The basic problem with game reviews is that no one agrees on what they’re supposed to be. Some people think they should be an objective analysis of the value of a game, in the vein of a Consumer Reports review. Other people think they should be commentary on a game’s artistic and cultural value in the vein of a book review. Should they have a score? Should the score be objective or subjective? Should they be exhaustive, or only focus on the particularly important (good or bad) parts of a game. Should they be written for a general or specific audience? Should they have a byline or be from “The Magazine”?  Should they include vague, poorly described categories like “Fun Factor”?

I think some of these extremes are clearly wrong. Consumer-report style value-only reviews are not what most reviewers or readers want. Games are not toasters, and players read reviews as much after the fact as before. Everyone wants at least some commentary and analysis. However, you can go too far, as I think book reviews (traditional newspaper ones) do. Every time I read a book review in a newspaper I come out with a clear picture of the reviewer’s innermost feelings and desires, but not a damn clue about if the book is any good. At some point professional book reviewers went down the path of full subjectivity, and I find amazon book reviews to be way more useful (and fun to read) than professional ones. Whenever a magazine tries to take out scores, everyone hates it, so we need some sort of score. I love the build up of context that comes from a well written review, but at the end of the day I want to get what I (indirectly) pay experts for in the first place: an opinion. Oh, and you know what sucks about institutional reviews? You can’t tell if you actually have anything in common with the reviewer. In fact, smaller genres often get underrated because they’re not correctly identifying the audience of the game (sometimes the publisher is at fault here).

So, we want a review that has a good mix of objective analysis, subjective opinion, and commentary. There should be some sort of clearly defined rating system, and enough information to help us decide if the review is relevant to our interests. Does this remind you of any other type of review? Yup, this is exactly what a well written movie review is. When Roger Ebert writes a review for the Sun Times, it is completely obvious that his star rating is his completely subjective opinion and not some sort of “composite” number that is derived from Sound and Fun Factor. No one calls up the Sun Times to complain that their movie got a bad review, they either complain to Roger (ineffectually) or chalk it up to difference of opinion. But, fanboys and developers across the world ask, why does one reviewer’s subjective opinion matter? Can’t game reviewers somehow come up with a provably “correct” rating for a game? And if they can’t why should we care about the aggregate at all?

The answer is that experts tend to be really good at rating things. Half of Blink by Malcolm Gladwell discusses how the gut instincts (ie, subjective score) of experts tend to be very good predictors of enjoyment by non experts. However, what experts (and humans in general) almost universally suck at is justifying that gut instinct. When we try to figure out where our rating comes from, we attempt to craft an explanation of the specific way we cleverly came up with that score. This is the process that generates the text of most game reviews (“The game is great, but the sound effects could use more punch”). Once you write a bunch of explanatory text, it then makes sense to adjust the score to be more “correct”, which generally pushes it away from accuracy. My feeling on this is that the more emotional a game experience is (there’s no way those 10 scores for GTA 4 were objective, the game has many obvious flaws), the more subjective and useful a game score is. However, any text justifying that score is going to basically be a waste of time, and should be better spent on commentary.

So if movie review scores (and theoretically game review scores) are just subjective opinion that may not apply to you, why is an aggregate score useful? Well, if a consumer shares sufficient traits with the “average” game reviewer, the aggregate score is a very useful approximation of their own estimated rating for a game.  In general, the hardcore gamers who are the initial adopters, the free advertisers, and the fanboys share a lot in common with the average game reviewer. If a game is well reviewed, it is very likely that it will generate positive gamer goodwill towards the developer and publisher and build a reputation of quality. Given that review scores have a direct effect on the future sales of related games, I think it absolutely makes sense to use metacritic scores for things like royalty payments. If you, as a game developer, can’t make a well reviewed game (thusly hurting the future sales of related games targeted at hardcore gamers), you either need to stop trying to sell games to hardcore gamers, adopt a publisher-free funding model, or stop making shitty games. As for magazine publishers? You should just make your reviews as much like good movie reviews as possible, and ignore all of us whiny developers.