Double Buffered

A Programmer’s View of Game Design, Development, and Culture

Posts Tagged ‘reviews’

“How We Decide” If a Game is a 9.5

Posted by Ben Zeigler on June 8, 2009

A few weeks ago I was listening to episode 4 of Out of the Game, and they started talking about “How We Decide” by Jonah Lehrer. At this point I realized I had been given a copy of it as a present, and given that it’s a book on psychology endorsed by Shawn Elliott I put it at the top of my reading stack. I’m glad I did, because I quite enjoyed it. A quick summary is that it’s a more complete, better written version of “Blink” by Malcolm Gladwell. It combines a few really good anecdotes about quick decision making  (apparently when caught in a quick moving forest fire the right solution is to light a SMALLER fire directly in front of you) with an overview of current research into both conscious and unconscious decision making. I recommend it to anyone with an interest in psychology.

The book builds what seems to be a really solid framework for making good decisions in a wide variety of contexts. There are two main processes we go through for making decisions: emotional and logical. They’re good at solving different kinds of problems, and are very complementary. The unconscious brain is very good at pattern matching and evaluating statistical models, and presents the results as input into your conscious brain as “gut feelings”. However, it is bad at dealing with falsehoods or irrelevant information. The conscious brain is good at simulating outcomes to solve problems and at regulating emotion, but it is very capable of thinking too much and incorrectly overriding emotional inputs. The general formula is to use your conscious brain to filter information and monitor emotional state (the best decisions get made while you are in a moderately excited state, as opposed to entirely dispassionate or enraged), and then let your unconscious brain think about it for a bit. The one that “feels right” will more often than not be the right decision.

I wrote last year about psychology and Game Reviews, and there’s a study in How We Decide that directly supports my thoughts. in 1990 Timothy Wilson put together a study comparing the ability of college students to rate jams to the ability of jam experts from Consumer Reports. When simply asked to rate the jams, the college students showed a correlation coefficient of .55 with the experts, which is reasonably high and shows that the expert’s choices matched fairly well with the average college student. Then, Wilson asked a different group of students to analyze why they preferred certain jams using elaborate questionnaires and a wide variety of categories. This group of college students showed a correlation coefficient of .11, which is essentially meaningless. After further study it turns out what happens is that the jam “reviewers” would try to describe individual components, such as “spreadibility”, that didn’t really affect their overall enjoyment of the jam. Then, as they evaluated all of these categories they tended to revise their preferences to match with what they had written in the review. The reviewers had overthought the problem and in the process had modified their initial preferences to match their specific analysis, as opposed to analyzing their preferences.

A similar study was performed by Wilson with paintings (a Monet, a Van Gogh, and 3 humorous cast posters). One group of women was just asked to choose their favorite painting, and 95% chose the Monet or Van Gogh. The second group was asked to explain why they liked the poster they chose, and that group was split 50% between the fine art and cat posters, because the cat posters had more content available for explanation (the subjects were not trained artists).  The book goes through a litany of other studies, all showing that when you try to carefully think through a complicated decision you end up making poor choices.

Let’s see, these studies are all about situations with a complicated decision and the need to generate explanatory content. In all of them, people who explained their decision before making it made worse decisions. Yeah, that sounds a lot like game reviews. If we assume that game reviewers are trained experts (most are) who have consciously trained themselves to be a good judge of games, and are not influenced by extremely strong emotions at the time of rating, their initial gut assessment of a review score is likely to correlate very strongly to the actual enjoyment of a game. However, if a reviewer rationally dissects a game into components they will be likely to rate a game higher or lower then it actually warrants, because it has “bad replay value” or something. So, game reviewers, stop thinking so much about the game and just pay attention to your emotional state while you’re reviewing a game. You’ll make better decisions.

Posted in Game Development, Random | Tagged: , , , | 1 Comment »

Reviews Symposium

Posted by Ben Zeigler on December 19, 2008

I’ve posted a bit about game reviews in the past, and it’s a topic that I still find very interesting. On top of that, I tend to find things involving Shawn Elliott interesting as well. Luckilly, these two topics have now intersected! Over at Shawn‘s blog, he is posting what he calls an “Online Reviews Symposium”. It sounds pretentious (and I think any use of the word Symposium automatically makes you pretentious and faux-academic, sorry), but it’s actually just an extended online conversation about reviews, featuring a wide variety of viewpoints.

Anyone who is interested in game reviews, or analysis of games journalism in general, should absolutely go check out Part 1: Review Scores. The opinions range the gamut, from people like N’Gai Croal who generally think review scores are pointless, to more practical views such as Robert Ashley’s, who discusses their actual utility to readers. As I’ve said before, I definitely think review scores have value as shorthand, and the aggregate sites such as MetaCritic provide an important service. However, it is possible to serve many of the same goals without an explicit score, which I think the kotaku reviews generally do quite well.

I’m definitely going to be checking this whole thing out as it develops, and I’m interested to see if the format Shawn has chosen works out (I’m a bit skeptical).

Posted in Game Culture | Tagged: , , | Comments Off on Reviews Symposium

The “Problem” With Game Reviews

Posted by Ben Zeigler on June 19, 2008

There’s been a lot of discussion over the last month or so about the game reviewing system. Stephen Totilo at MTV did a whole week of coverage on the subject, which I suggest you go browse. The spark for this discussion was the overwhelmingly glowing reviews of GTA 4, and has continued with arguments over Metal Gear Solid 4. Reviews are on the industry’s mind, and everyone thinks they know what’s wrong with them. So being egotistical and analytical, I thought I’d conclusively answer this.

The basic problem with game reviews is that no one agrees on what they’re supposed to be. Some people think they should be an objective analysis of the value of a game, in the vein of a Consumer Reports review. Other people think they should be commentary on a game’s artistic and cultural value in the vein of a book review. Should they have a score? Should the score be objective or subjective? Should they be exhaustive, or only focus on the particularly important (good or bad) parts of a game. Should they be written for a general or specific audience? Should they have a byline or be from “The Magazine”?  Should they include vague, poorly described categories like “Fun Factor”?

I think some of these extremes are clearly wrong. Consumer-report style value-only reviews are not what most reviewers or readers want. Games are not toasters, and players read reviews as much after the fact as before. Everyone wants at least some commentary and analysis. However, you can go too far, as I think book reviews (traditional newspaper ones) do. Every time I read a book review in a newspaper I come out with a clear picture of the reviewer’s innermost feelings and desires, but not a damn clue about if the book is any good. At some point professional book reviewers went down the path of full subjectivity, and I find amazon book reviews to be way more useful (and fun to read) than professional ones. Whenever a magazine tries to take out scores, everyone hates it, so we need some sort of score. I love the build up of context that comes from a well written review, but at the end of the day I want to get what I (indirectly) pay experts for in the first place: an opinion. Oh, and you know what sucks about institutional reviews? You can’t tell if you actually have anything in common with the reviewer. In fact, smaller genres often get underrated because they’re not correctly identifying the audience of the game (sometimes the publisher is at fault here).

So, we want a review that has a good mix of objective analysis, subjective opinion, and commentary. There should be some sort of clearly defined rating system, and enough information to help us decide if the review is relevant to our interests. Does this remind you of any other type of review? Yup, this is exactly what a well written movie review is. When Roger Ebert writes a review for the Sun Times, it is completely obvious that his star rating is his completely subjective opinion and not some sort of “composite” number that is derived from Sound and Fun Factor. No one calls up the Sun Times to complain that their movie got a bad review, they either complain to Roger (ineffectually) or chalk it up to difference of opinion. But, fanboys and developers across the world ask, why does one reviewer’s subjective opinion matter? Can’t game reviewers somehow come up with a provably “correct” rating for a game? And if they can’t why should we care about the aggregate at all?

The answer is that experts tend to be really good at rating things. Half of Blink by Malcolm Gladwell discusses how the gut instincts (ie, subjective score) of experts tend to be very good predictors of enjoyment by non experts. However, what experts (and humans in general) almost universally suck at is justifying that gut instinct. When we try to figure out where our rating comes from, we attempt to craft an explanation of the specific way we cleverly came up with that score. This is the process that generates the text of most game reviews (“The game is great, but the sound effects could use more punch”). Once you write a bunch of explanatory text, it then makes sense to adjust the score to be more “correct”, which generally pushes it away from accuracy. My feeling on this is that the more emotional a game experience is (there’s no way those 10 scores for GTA 4 were objective, the game has many obvious flaws), the more subjective and useful a game score is. However, any text justifying that score is going to basically be a waste of time, and should be better spent on commentary.

So if movie review scores (and theoretically game review scores) are just subjective opinion that may not apply to you, why is an aggregate score useful? Well, if a consumer shares sufficient traits with the “average” game reviewer, the aggregate score is a very useful approximation of their own estimated rating for a game.  In general, the hardcore gamers who are the initial adopters, the free advertisers, and the fanboys share a lot in common with the average game reviewer. If a game is well reviewed, it is very likely that it will generate positive gamer goodwill towards the developer and publisher and build a reputation of quality. Given that review scores have a direct effect on the future sales of related games, I think it absolutely makes sense to use metacritic scores for things like royalty payments. If you, as a game developer, can’t make a well reviewed game (thusly hurting the future sales of related games targeted at hardcore gamers), you either need to stop trying to sell games to hardcore gamers, adopt a publisher-free funding model, or stop making shitty games. As for magazine publishers? You should just make your reviews as much like good movie reviews as possible, and ignore all of us whiny developers.

Posted in Game Development | Tagged: , , , , | 3 Comments »