The Players Are Wrong, But Listen Anyway
Posted by Ben Zeigler on June 16, 2009
When you’re deep in Beta, or you’re just taking the unusual step of actively seeking out what people think about your game, you’re going to interact with direct user feedback. Now, unlike feedback that has been filtered through OCR (which sometimes improves it and sometimes obscures it), direct user feedback tends to be either VERY FORCEFUL or just a bit confused. Based on my observations of fellow employees it seems to be standard policy for a new developer to seek out feedback, realize that some of it is hateful or confused, and then conflate the rest of your player base with the minority (well, or majority depending). The human brain really likes to make those kind of assumptions, so it takes conscious effort to get over the impression that not everyone who disagrees with you is a worthless human being. It turns out that most of the direct feedback you get from users DOES have value, if you just know how to mine it. Anyway, here’s my informal guide to Actually Learning From Users, broken down into helpfully pedantic steps:
1. Note Context
The context of the feedback should have a large impact on your interpretation of it. For the MMO-playing audience this can be broken down into Official Forums and Email (except for those bizarre companies that think they’re a waste of time), MMO Forums (F13, Fires of Heaven, Massively), and General Forums (Kotaku, Joystiq, etc). Feedback on the official forums tends to come from either low post-count users with specific issues or high post-count users who are part of the forum community. MMO forums tends to be full of cynical posters with a high level of knowledge and distrust, but with the occasional great suggestion. General forums are full of people who sort of heard something about your game and are a good way to get a feel for how well your marketing (formal and word of mouth) is working.
2. Ascertain Motive
Once you know the context of a posting you can make a quick guess as to the motive of the poster. The first motive is the always exciting Trolling. From my personal experience this is actually pretty rare on official boards, but this happens a lot on the general boards. If the goal of a poster is to instigate conversation or argument without actually contributing anything, go ahead and ignore them. A related but much less evil motive is Conversation. These posters will have very high post counts and are an integral part of the community, but most are useless for feedback purposes because message board posting, much like Blogging, is all about volume instead of content. A small number of high-volume posters graduate past Conversation into Aggregation. These users (I was one at one point) actively enjoy collating feedback for developers, and some are very good at this. During City of Villains Beta I trusted the bug-aggregate threads far more then I trusted our Q/A department, sad to say. Past those, another motive is Confusion. These users are often new and have a lack of knowledge. Because of this they will often be ostracized by the other members of a community, but to a developer they’re invaluable. If a number of posters are confused by some part of the game, there are no plans to make it better, and your game isn’t Darkfall, you Have A Problem. Either the system isn’t being explained well enough, or more likely your system is just too damn complicated in the first place. If players don’t at least have the illusion of understanding a system, they’re going to be constantly second guessing their choices and be too worried to have any fun.
The last motive is Advocacy. These users have a very specific goal, which is to get something about your game changed. These are the trickiest to deal with, because the first instinct is to ignore people who want something as being selfish and self-serving. But this is only part of it. Users who constantly gripe about their class will always exist no matter how well balanced your game is. The usefulness of Advocacy feedback comes in specificity. If a player feels that his class is gimped overall but doesn’t provide detail, that post is largely worthless. However, if a player specifies that a specific ability seems underpowered, and non-regulars agree (regular complainers will agree that everything is broken), it’s worth taking a look at. The odds are good there’s either a design bug or misleading player feedback. Oh, and as for those players who REALLY care about the way that one CERTAIN cape looks and it NEEDS TO BE RED or else they’ll QUIT FOR REAL, those players should be humored but largely ignored. MMOs are so much about personal identity that it’s inevitable you will highly annoy detail oriented people who can’t have exactly what they want. The only people who threaten to quit are those who are highly invested in your game: you should worry about the ones who don’t give a shit either way.
3. Extract Value
Any feedback that passes the Motive check (specific advocacy, content aggregation, and confusion) contains some value to a game developer, and I refer to this as Valid Feedback. Now, I use Valid instead of Good for a reason. The posts contain useful information but there’s absolutely no guarantee that it’s correct information. For instance, a confused user will often make weird assumptions about the cause of a bug because of their lack of background. The useful kind will still report what confused them, but if you see completely illogical feature requests or feedback it’s probably from someone who is confused but doesn’t want to admit it (because of the social ramifications of admitting to being confused). In these situations you have to ask “Why would they say this?” instead of “What did they say?”. Let’s say a user angrily posts that you’re evil for removing his favorite power. Instead of ignoring him because the power still exists in the source data, you can check to see if something weird broke in the runtime to hide it from the UI.
This same principal needs to be applied to Advocates. When someone discovers something about a game that makes them unhappy, the average player will not do a very good job of figuring out why they’re unhappy. Instead, they will grasp around a bit until they find a vaguely plausible cause, and then flog that horse until it’s dead. This often results in a petition of some sort, many of which won’t make any sense. As a developer you can either ignore them for being incorrect, or you can embrace their opinion as being valid but misdirected. For instance let’s say a large number of players complain about the fact that you nerfed a particular self-healing power which was obviously broken and overpowered. You can say the players are selfish exploiters, or you can take a deeper look and wonder if the players are only upset because that broken self heal was the only way to compensate for the occasional player mistake during unforgiving combat. Players choose to post out of genuine frustration, but instead of reasoning why they tend to just rationalize.
These rules are a bit verbose, but after reading a few hundred posts you can start to filter them really quickly. Most feedback will end up not being useful, which should give you enough time to think about what’s actually important. Basically, if enough sincere people post asking you do something, then there is very likely a real problem. However, the solution to that problem is unlikely to be what they originally asked for (although specific posters with deep knowledge and analytical skills are better at this then your Q/A department. Learn who they are). Extracting out the Valid feedback from the chaff is not a skill that comes naturally, but I think it’s invaluable for making a compelling long-term product. Directly following user requests and completely ignoring them are two different paths to ruin for an MMO, and the solution is to seek the middle path of interprating that feedback without being slave to it.