What is Beta Testing Good For?
Posted by Ben Zeigler on April 21, 2009
A few days ago, TenTonHammer posted a comprehensive article about what Beta Testing means to MMOs. I got it off Scott Hartsman’s twitter, and it has a lot of quotes from both developers and users, regarding what Beta really means these days. To summarize, both developers and players generally agree that Beta is less about gathering player feedback, and more about marketing. I will 100% agree that OPEN Beta (or pre-order Beta, whatever) is entirely about marketing and load balance testing. A month out from release is too late to really do any significant changes to the game. However, at least at companies that are open to feedback, a fairly large Closed Beta is required to make a high-quality product. Other than the obvious benefit of having players do free Q/A, here’s why Beta is vitally important:
- Gets your Community up and running. You can’t start a community from nothing, so if you want a solid and supportive community at launch, you need to put in a lot of work before launch. A big part of this is getting together the right community team, and there’s really no way to figure out what will work without just TRYING it. So, during Beta both the OCR team and Devs can directly interact with the players and set up a rapport that will hopefully carry through launch. Both the CoH and CoV betas had good developer-community communication, and I definitely think it followed through past launch. For this reason, I think using different community tools (*cough*Warhammer*cough*) for Beta and Live is a stupid idea.
- Helps resolve developer choices. The development of an MMO involves a LOT of people, and reasonable people can disagree on various features and design decisions. Sometimes it works out that it’s fairly easy to implement two versions of a feature, but everyone knows that a choice has to be made before launch, for either player confusion or implementation issues. If the developers structure it correctly, it can be very helpful to use Closed Beta as an extremely large focus test. So, you can give players a set of options and see what the reaction is, both via data mining and direct feedback. In cases like this, the players in Closed Beta will have a direct effect on the final product, and I know that’s happened before at Cryptic.
- Tracks down compatibility issues. This is kind of boring but absolutely crucial. MMOs are the most complex type of video game, and PCs are stupidly complicated. This means that you’re guaranteed to have a shitload of obscure technical conflicts, and you’re going to have to spend incredibly tedious hours resolving 90% of them (you never fix all weird technical conflicts, you just have to live with that). It’s important to make sure your Beta includes a very wide variety of computer setups, and inevitably it means that some percentage of you users just won’t get to play for a few weeks. I have to say I was mildly irritated by some of the user comments in the article on this subject. Very likely the reason the client doesn’t run on your computer (if it does run on most other people’s) is that the PC platform is stupidly complex and prone to failure.
- Drives your post-launch content. By the time you get to large scale Closed Beta, it’s generally too late to make dramatic content changes in time for launch. However, this does NOT mean that user feedback during Beta is useless. Remember, an MMO tends to change a lot in the first big content release after launch. This is because the feedback from Beta users ends up directly affecting the design decisions and content additions that go into content that comes out 2-3 months after launch. For instance, a LOT of the content in the 40-50 level range in CoH (and CoV) was directly built in reaction to shortcomings that became obvious during Beta testing. Again, this is an area that Beta players had a direct impact on the future of the product.
If you don’t run your Closed Beta with an eye for gathering feedback from testers, you’re going to end up with a game that has a disconnected and isolated community, unpopular design decisions that the developers dont particularly love, a cavalcade of horrible compatibility issues, and no plan for post-release content updates. If you run your Beta as an advertising campaign with incredibly strict moderation and no developer feedback you’re taking a huge risk. I could name a few recent MMOs that have had large initial launches and exactly those kind of post-launch problems.
7 Responses to “What is Beta Testing Good For?”
Sorry, the comment form is closed at this time.