Double Buffered

A Programmer’s View of Game Design, Development, and Culture

The Players Are Wrong, But Listen Anyway

Posted by Ben Zeigler on June 16, 2009

When you’re deep in Beta, or you’re just taking the unusual step of actively seeking out what people think about your game, you’re going to interact with direct user feedback. Now, unlike feedback that has been filtered through OCR (which sometimes improves it and sometimes obscures it), direct user feedback tends to be either VERY FORCEFUL or just a bit confused. Based on my observations of fellow employees it seems to be standard policy for a new developer to seek out feedback, realize that some of it is hateful or confused, and then conflate the rest of your player base with the minority (well, or majority depending). The human brain really likes to make those kind of assumptions, so it takes conscious effort to get over the impression that not everyone who disagrees with you is a worthless human being. It turns out that most of the direct feedback you get from users DOES have value, if you just know how to mine it. Anyway, here’s my informal guide to Actually Learning From Users, broken down into helpfully pedantic steps:

1. Note Context

The context of the feedback should have a large impact on your interpretation of it. For the MMO-playing audience this can be broken down into Official Forums and Email (except for those bizarre companies that think they’re a waste of time), MMO Forums (F13, Fires of Heaven, Massively), and General Forums (Kotaku, Joystiq, etc). Feedback on the official forums tends to come from either low post-count users with specific issues or high post-count users who are part of the forum community. MMO forums tends to be full of cynical posters with a high level of knowledge and distrust, but with the occasional great suggestion. General forums are full of people who sort of heard something about your game and are a good way to get a feel for how well your marketing (formal and word of mouth) is working.

2. Ascertain Motive

Once you know the context of a posting you can make a quick guess as to the motive of the poster. The first motive is the always exciting Trolling. From my personal experience this is actually pretty rare on official boards, but this happens a lot on the general boards. If the goal of a poster is to instigate conversation or argument without actually contributing anything, go ahead and ignore them. A related but much less evil motive is Conversation. These posters will have very high post counts and are an integral part of the community, but most are useless for feedback purposes because message board posting, much like Blogging, is all about volume instead of content. A small number of high-volume posters graduate past Conversation into Aggregation. These users (I was one at one point) actively enjoy collating feedback for developers, and some are very good at this. During City of Villains Beta I trusted the bug-aggregate threads far more then I trusted our Q/A department, sad to say. Past those, another motive is Confusion. These users are often new and have a lack of knowledge. Because of this they will often be ostracized by the other members of a community, but to a developer they’re invaluable. If a number of posters are confused by some part of the game, there are no plans to make it better, and your game isn’t Darkfall, you Have A Problem. Either the system isn’t being explained well enough, or more likely your system is just too damn complicated in the first place. If players don’t at least have the illusion of understanding a system, they’re going to be constantly second guessing their choices and be too worried to have any fun.

The last motive is Advocacy. These users have a very specific goal, which is to get something about your game changed. These are the trickiest to deal with, because the first instinct is to ignore people who want something as being selfish and self-serving. But this is only part of it. Users who constantly gripe about their class will always exist no matter how well balanced your game is. The usefulness of Advocacy feedback comes in specificity. If a player feels that his class is gimped overall but doesn’t provide detail, that post is largely worthless. However, if a player specifies that a specific ability seems underpowered, and non-regulars agree (regular complainers will agree that everything is broken), it’s worth taking a look at. The odds are good there’s either a design bug or misleading player feedback. Oh, and as for those players who REALLY care about the way that one CERTAIN cape looks and it NEEDS TO BE RED or else they’ll QUIT FOR REAL, those players should be humored but largely ignored. MMOs are so much about personal identity that it’s inevitable you will highly annoy detail oriented people who can’t have exactly what they want. The only people who threaten to quit are those who are highly invested in your game: you should worry about the ones who don’t give a shit either way.

3. Extract Value

Any feedback that passes the Motive check (specific advocacy, content aggregation, and confusion) contains some value to a game developer, and I refer to this as Valid Feedback. Now, I use Valid instead of Good for a reason. The posts contain useful information but there’s absolutely no guarantee that it’s correct information. For instance, a confused user will often make weird assumptions about the cause of a bug because of their lack of background. The useful kind will still report what confused them, but if you see completely illogical feature requests or feedback it’s probably from someone who is confused but doesn’t want to admit it (because of the social ramifications of admitting to being confused). In these situations you have to ask “Why would they say this?” instead of “What did they say?”. Let’s say a user angrily posts that you’re evil for removing his favorite power. Instead of ignoring him because the power still exists in the source data, you can check to see if something weird broke in the runtime to hide it from the UI.

This same principal needs to be applied to Advocates. When someone discovers something about a game that makes them unhappy, the average player will not do a very good job of figuring out why they’re unhappy. Instead, they will grasp around a bit until they find a vaguely plausible cause, and then flog that horse until it’s dead. This often results in a petition of some sort, many of which won’t make any sense. As a developer you can either ignore them for being incorrect, or you can embrace their opinion as being valid but misdirected. For instance let’s say a large number of players complain about the fact that you nerfed a particular self-healing power which was obviously broken and overpowered. You can say the players are selfish exploiters, or you can take a deeper look and wonder if the players are only upset because that broken self heal was the only way to compensate for the occasional player mistake during unforgiving combat. Players choose to post out of genuine frustration, but instead of reasoning why they tend to just rationalize.

4. Repeat

These rules are a bit verbose, but after reading a few hundred posts you can start to filter them really quickly. Most feedback will end up not being useful, which should give you enough time to think about what’s actually important. Basically, if enough sincere people post asking you do something, then there is very likely a real problem. However, the solution to that problem is unlikely to be what they originally asked for (although specific posters with deep knowledge and analytical skills are better at this then your Q/A department. Learn who they are). Extracting out the Valid feedback from the chaff is not a skill that comes naturally, but I think it’s invaluable for making a compelling long-term product. Directly following user requests and completely ignoring them are two different paths to ruin for an MMO, and the solution is to seek the middle path of interprating that feedback without being slave to it.

7 Responses to “The Players Are Wrong, But Listen Anyway”

  1. […] Sin embargo, le he dedicado algún tiempo a algunas  lecturas  ultimamente: ¿qué es un desarrollador independiente?, tobold sobre skills (habilidades), y sobre las reacciones de los jugadores en un beta. […]

  2. Sounds about right. A lot of what I really like about Dungeon Runners, and what I’m really proud of, I have to chalk up at least partially to having participated in the forums and playing the game. You can’t take everything said at face value, but there is a lot of hidden value if you look at what is causing people to say what they say, feel the way they feel.

  3. […] MMORPG 1 y 2, Cómo mejorar el combate en los MMORPG, Es un mundo pequeño,  Wow PVP, Los jugadores están equivocados, y un largo […]

  4. A great list, and frighteningly apt. I’m guilty of loving to gab about games, and in all my forum time it’s become incredibly apparent that there are people who love to complain and yet have no idea what their actual issue is.

    My least favorite type is still the vocal “hardcore” majority; they’ll defend a game they like to the death, the good and bad, and actually openly say they’d prefer it if fewer people played the game, not caring for whether or not other playstyles enjoy it. They usually obfuscate the line between “things I like about this game” and “things that are necessary to this game being good that only smart people like me would like”.

    But still, there’s value in all manner of feedback. It just takes smart devs to see through the crap.

  5. […] level play, but someone somewhere will always make these sorts of statements, period.  As noted in an article I read recently, for the developer it’s all a matter of discerning intent in user feedback: how many players […]

  6. […] The Players are Wrong but Listen Anyway This article by a game developer goes into detail as to what type of feedback to gather from the users during the game-making process. This frankly doesn’t rise to the level of a legal duty, but more of a moral duty… that is from a game developer with a sincere wish to make as fair and as entertaining of a game as possible. If you are strictly looking for a legal viewpoint, maybe you could use the forums and feedbacks as a sort of “testing ground” to determine if the beta testers feel the game is working out as a fair and even playing field, depending upon the user’s choice of character, etc. But the developer does a great job of noting these opinions must be taken in the context that most of these early users have an “axe to grind” or may be overly skeptical types that over-critique a supposedly fun experience. A great read for developers or anyone. […]

  7. […] their reasoning as they do NOT have the same mental context you do as the designer. (Note: I 100% agree with and endorse this feedback […]

Sorry, the comment form is closed at this time.

%d bloggers like this: