Double Buffered

A Programmer’s View of Game Design, Development, and Culture

Archive for April, 2009

What is Beta Testing Good For?

Posted by Ben Zeigler on April 21, 2009

A few days ago, TenTonHammer posted a comprehensive article about what Beta Testing means to MMOs. I got it off Scott Hartsman’s twitter, and it has a lot of quotes from both developers and users, regarding what Beta really means these days. To summarize, both developers and players generally agree that Beta is less about gathering player feedback, and more about marketing. I will 100% agree that OPEN Beta (or pre-order Beta, whatever) is entirely about marketing and load balance testing. A month out from release is too late to really do any significant changes to the game. However, at least at companies that are open to feedback, a fairly large Closed Beta is required to make a high-quality product. Other than the obvious benefit of having players do free Q/A, here’s why Beta is vitally important:

  1. Gets your Community up and running. You can’t start a community from nothing, so if you want a solid and supportive community at launch, you need to put in a lot of work before launch. A big part of this is getting together the right community team, and there’s really no way to figure out what will work without just TRYING it. So, during Beta both the OCR team and Devs can directly interact with the players and set up a rapport that will hopefully carry through launch. Both the CoH and CoV betas had good developer-community communication, and I definitely think it followed through past launch. For this reason, I think using different community tools (*cough*Warhammer*cough*) for Beta and Live is a stupid idea.
  2. Helps resolve developer choices. The development of an MMO involves a LOT of people, and reasonable people can disagree on various features and design decisions. Sometimes it works out that it’s fairly easy to implement two versions of a feature, but everyone knows that a choice has to be made before launch, for either player confusion or implementation issues. If the developers structure it correctly, it can be very helpful to use Closed Beta as an extremely large focus test. So, you can give players a set of options and see what the reaction is, both via data mining and direct feedback. In cases like this, the players in Closed Beta will have a direct effect on the final product, and I know that’s happened before at Cryptic.
  3. Tracks down compatibility issues. This is kind of boring but absolutely crucial. MMOs are the most complex type of video game, and PCs are stupidly complicated. This means that you’re guaranteed to have a shitload of obscure technical conflicts, and you’re going to have to spend incredibly tedious hours resolving 90% of them (you never fix all weird technical conflicts, you just have to live with that). It’s important to make sure your Beta includes a very wide variety of computer setups, and inevitably it means that some percentage of you users just won’t get to play for a few weeks. I have to say I was mildly irritated by some of the user comments in the article on this subject. Very likely the reason the client doesn’t run on your computer (if it does run on most other people’s) is that the PC platform is stupidly complex and prone to failure.
  4. Drives your post-launch content. By the time you get to large scale Closed Beta, it’s generally too late to make dramatic content changes in time for launch. However, this does NOT mean that user feedback during Beta is useless. Remember, an MMO tends to change a lot in the first big content release after launch. This is because the feedback from Beta users ends up directly affecting the design decisions and content additions that go into content that comes out 2-3 months after launch. For instance, a LOT of the content in the 40-50 level range in CoH (and CoV) was directly built in reaction to shortcomings that became obvious during Beta testing. Again, this is an area that Beta players had a direct impact on the future of the product.

If you don’t run your Closed Beta with an eye for gathering feedback from testers, you’re going to end up with a game that has a disconnected and isolated community, unpopular design decisions that the developers dont particularly love, a cavalcade of horrible compatibility issues, and no plan for post-release content updates. If you run your Beta as an advertising campaign with incredibly strict moderation and no developer feedback you’re taking a huge risk. I could name a few recent MMOs that have had large initial launches and exactly those kind of post-launch problems.

Posted in Game Development, MMO Design | Tagged: , , | 7 Comments »

2008 Game Developer Salary Survey

Posted by Ben Zeigler on April 10, 2009

I opened up this month’s Game Developer, and it’s salary survey time again! I thought I would take a look at the numbers and compare them to last year’s survey. First of all, the tone of the piece was a bit different. Jill Duffy, who wrote the article and was a senior contributing editor, mentioned that she was being laid off in the intro to the piece. Despite that depressing start, salary numbers lag a bit behind the economy, so wages were actually up this year. Next year is the one where we’ll really see the fallout. Most of the trends are basically the same as last year, so no point in going over the general trends. One thing worth noting is that the US sample size was only 1,879 this year (down 600 from last year) so statistically it’s a bit more suspect.

One area I wanted to revisit was trends in gender pay equality. Here’s the 2008 gender pay equality chart:

Field Percentage of Females Female Compensation Difference
Programming 3% $10,249 less (12%)
Art 10% $8,456 less (12%)
Design 6% $18,324 less (26%)
Production 21% $10,079 less (12%)
Business 14% $28,704 less (26%)
Q/A 14% $7,500 more (+16%)

Right, so, except for Q/A (either the demographics of Q/A changed dramatically in the last year, or something weird is going on statistically) and to less extent Business, this is pretty bad news. The gender disparity in Programming, Art, and Production rose by 3% which is disappointing. The gender disparity in design rose by a ridiculous 11%. Female game designers now make $18 thousand less a year than male game designers, and game designers don’t really make that much on average. All of the trends I mentioned last year about females having a hard time getting top-level design jobs seems to be even worse this year. I find that more than a bit depressing.

Posted in Game Development | Tagged: , , | Leave a Comment »

Why OnLive Won’t Work

Posted by Ben Zeigler on April 5, 2009

So the big topic of general interest at last week’s GDC was OnLive. Lots of people, both gamers and developers, were excited about the possibility of playing PC games without a PC. It’s supposed to solve piracy and system requirements issues in one go. Well, I tried it out in person at GDC, and I can say conclusively that OnLive’s plans will never work. But, it’s not for the reason you may think. The technology is viable, it’s just the economics that make no sense.

There was a bit of a line at the OnLive booth, but I waited patiently before I got to play BioShock over it for a few minutes. The experience was actually better than I had expected. The image quality was perfectly fine in my opinion, although it got worse when you moved the camera rapidly. There was noticeable lag, but it was still playable. It felt almost exactly like playing an FPS on too-high settings. Basically it was like playing BioShock on high quality settings at about 15 frames per second. It’s not an experience I would consider paying money for, but I can see how it would be acceptable for slightly less picky gamers. So, OnLive can successfully play games remotely 50 miles away from San Francisco to Santa Clara (where the servers were), using a dedicated line, and to a quality level acceptable to an average user but not to the hardcore. That’s actually a pretty impressive accomplishment.

But it isn’t enough to actually make money. Time to do some math. According to what they’ve said publicly about their business plan, they plan to stream games over the general internet, as long as you are within 1000 miles of a server (1000 miles would be significantly worse lag then I experienced), and have a 1.5 Megabit sustained connection (5 Megabit for HD, which we’ll just ignore because it’s even sillier). Ignoring the issue with ISP bandwidth caps (which could be a serious problem) this is mostly reasonable. However, it leaves out a big part of the equation: the servers and bandwidth on OnLive’s end are not free. In fact, they are very expensive.

Thick-client Western MMOs are a space I am reasonably familiar with. Out of your $15 subscription fee, somewhere around $5 goes directly to hosting and bandwith (the rest towards development, support, and licensing). MMO bandwidth is split between patching (let’s say 1 Gigabyte a month per player on average) and runtime (usually pretty light, 100 kilabit/s MAX). At 50 hours of play a month (which is about average) that ends up being around 3 Gigabytes of bandwith per month. On the other hand, 50 hours a month of 1.5 megabit streaming is 30 Gigabytes per month. So, bandwidth costs for OnLive are 10x what they are for MMO hosting. The other part of the cost of running an MMO is purchasing and powering servers. An average MMO server can host a minimum of 50 concurrent players (a lot more in the case of games like WoW which are light on server use). These are fairly buff machines (but not special in any way), so let’s say that either you could buy 5 cheap desktops for that price, or host 5 client instances of OnLive games, giving a maximum concurrent OnLive users per server at about 5. This means that OnLive will need to buy and run 10x as many computers as an MMO.

The two components of hosting are both about 10X worse for OnLive than they are for MMOs (and that’s a very generous number). Multiplied by the wholesale hosting cost of $5 a month for MMOs, this gives an estimated wholesale cost per user of OnLive to $50/month. Remember, this is wholesale and before they make ANY money off of this. So assuming a minimum margin of around $10/month, anyone who wants to use OnLive is going to be paying at least $60/month for it. This is why they haven’t announced pricing yet, because they know what the reaction would be. So, the audience for OnLive is people willing to pay $60/month for pc gaming, but not hardcore pc gamers who care about latency.

This business plan doesn’t make any sense. On the high end, it fails miserably compared to Steam, which is a lot cheaper and more satisfying for the hardcore. On the casual end, why pay $60/month when you don’t care about graphics and are happy playing browser games? There are only two ways to make this work: either contract directly with ISPs (which would negate the bandwidth costs) and get it subsidized through already-expensive cable bills, or sell the company for lots of money up front. Considering that OnLive was created by Steve Perlman, who is the same guy who sold the largely useless thinclient technology WebTV to Microsoft for $425 million, I think it’s obvious what the corporate strategy is: Get bought before you have to launch the service and hope no one digs too deeply into the realities of the business plan.

Posted in Game Development | Tagged: , , | 6 Comments »