Double Buffered

A Programmer’s View of Game Design, Development, and Culture

What is Beta Testing Good For?

Posted by Ben Zeigler on April 21, 2009

A few days ago, TenTonHammer posted a comprehensive article about what Beta Testing means to MMOs. I got it off Scott Hartsman’s twitter, and it has a lot of quotes from both developers and users, regarding what Beta really means these days. To summarize, both developers and players generally agree that Beta is less about gathering player feedback, and more about marketing. I will 100% agree that OPEN Beta (or pre-order Beta, whatever) is entirely about marketing and load balance testing. A month out from release is too late to really do any significant changes to the game. However, at least at companies that are open to feedback, a fairly large Closed Beta is required to make a high-quality product. Other than the obvious benefit of having players do free Q/A, here’s why Beta is vitally important:

  1. Gets your Community up and running. You can’t start a community from nothing, so if you want a solid and supportive community at launch, you need to put in a lot of work before launch. A big part of this is getting together the right community team, and there’s really no way to figure out what will work without just TRYING it. So, during Beta both the OCR team and Devs can directly interact with the players and set up a rapport that will hopefully carry through launch. Both the CoH and CoV betas had good developer-community communication, and I definitely think it followed through past launch. For this reason, I think using different community tools (*cough*Warhammer*cough*) for Beta and Live is a stupid idea.
  2. Helps resolve developer choices. The development of an MMO involves a LOT of people, and reasonable people can disagree on various features and design decisions. Sometimes it works out that it’s fairly easy to implement two versions of a feature, but everyone knows that a choice has to be made before launch, for either player confusion or implementation issues. If the developers structure it correctly, it can be very helpful to use Closed Beta as an extremely large focus test. So, you can give players a set of options and see what the reaction is, both via data mining and direct feedback. In cases like this, the players in Closed Beta will have a direct effect on the final product, and I know that’s happened before at Cryptic.
  3. Tracks down compatibility issues. This is kind of boring but absolutely crucial. MMOs are the most complex type of video game, and PCs are stupidly complicated. This means that you’re guaranteed to have a shitload of obscure technical conflicts, and you’re going to have to spend incredibly tedious hours resolving 90% of them (you never fix all weird technical conflicts, you just have to live with that). It’s important to make sure your Beta includes a very wide variety of computer setups, and inevitably it means that some percentage of you users just won’t get to play for a few weeks. I have to say I was mildly irritated by some of the user comments in the article on this subject. Very likely the reason the client doesn’t run on your computer (if it does run on most other people’s) is that the PC platform is stupidly complex and prone to failure.
  4. Drives your post-launch content. By the time you get to large scale Closed Beta, it’s generally too late to make dramatic content changes in time for launch. However, this does NOT mean that user feedback during Beta is useless. Remember, an MMO tends to change a lot in the first big content release after launch. This is because the feedback from Beta users ends up directly affecting the design decisions and content additions that go into content that comes out 2-3 months after launch. For instance, a LOT of the content in the 40-50 level range in CoH (and CoV) was directly built in reaction to shortcomings that became obvious during Beta testing. Again, this is an area that Beta players had a direct impact on the future of the product.

If you don’t run your Closed Beta with an eye for gathering feedback from testers, you’re going to end up with a game that has a disconnected and isolated community, unpopular design decisions that the developers dont particularly love, a cavalcade of horrible compatibility issues, and no plan for post-release content updates. If you run your Beta as an advertising campaign with incredibly strict moderation and no developer feedback you’re taking a huge risk. I could name a few recent MMOs that have had large initial launches and exactly those kind of post-launch problems.


7 Responses to “What is Beta Testing Good For?”

  1. This is actually what my company does (helping companies manage real beta tests), and I couldn’t agree more. Beta testing has been bastardized a bit in recent years, especially in the gaming realm – but also on the web in general. Chances are if you’re hearing about a beta test from more than a couple people or web-sites, then realistically it’s a preview/demo, and your written/spoken feedback will have absolutely no effect on that product, so take that for what it’s worth (on the other hand your actual playtime will generate very valuable data). That said, just about all of these companies DO have internal betas, or at least “friends and family” alphas – the public just obviously doesn’t know about them, and they do generally benefit a great deal from those. Luke @ Centercode

  2. JZig said

    Luke, I understand that in other areas it may not be the case that written feedback during Beta is ignored, but I feel that it is NECESSARY for developers to listen to feedback from MMO Betas, for the reasons above. I know from personal experience that I have used written feedback from our large-scale closed betas (the ones with around 10k players, as opposed to F&F Alpha) to directly improve the quality of our products. I feel there is a strong correlation between MMO developers who ignore written beta feedback, and MMO developers who ignore ALL user feedback. And ignoring user feedback means that your game will have a large initial spike and no legs, which has happened a lot recently.

  3. JZig said

    Err, I mean “Luke, I understand that in other areas it may be the case that written feedback during Beta is ignored”

  4. Well to clarify one thing – I believe that companies should always be listening to user feedback, and beta is no exception to that rule. Of course I’m biased, but if I didn’t believe I wouldn’t live it.

    Anyway, beta feedback in a game, especially in a public beta, and even more-so in an MMO is an extremely complex and tricky subject. It’s not something that can really be trivialized to something you should or shouldn’t do. I personally believe that it takes a great deal of skill on the role of the designer to understand what to listen to and more importantly what not to. The biggest challenge is the fact that user opinions are often so heated, quite often very self-interested, and generally fueled by a number of overlapping and clashing often extremely vocal minorities. Honestly not a job I want. (:

    On top of that there’s the logistical nightmare of deciphering 50,000+ individual pieces of feedback, and figuring out what bucket each of them fall into. (:

    I think Blizzard is a good example of a company that’s earned a reputation amongst many hardcore gamers (at least in regards to WoW) for not listening to user feedback, and yet nobody can hold a candle to their success – so obviously they’re doing something extremely right. I’m curious what your thoughts are on that?

  5. And to comment on your correction post – from my experience (we’ve run over 300 betas, and our software used for many hundred more) – it’s generally not common for product managers and developers to ignore beta feedback. Most companies take their beta process pretty seriously and invest in it for a reason – it produces a higher quality product that makes them substantially more money.

    Unfortunately gaming is one of the most complex areas to make good use of user feedback in, as the customers are just so blatantly passionate about it, combined with the fact that it’s a young industry full of just as passionate people who look at game design as more of an art form than a business. In other words, they’re much more likely to “trust their gut” – which sometimes pays off, and sometimes doesn’t.

  6. […] Subscribe  Subscribe in a reader « What is Beta Testing Good For? […]

  7. […] by JZig on June 16, 2009 When you’re deep in Beta, or you’re just taking the unusual step of actively seeking out what people think about your […]

Sorry, the comment form is closed at this time.

%d bloggers like this: