My Letter to Brian About A Public Alpha/Beta: Your Thoughts...

[Shining1]

Hmmm. Basically, it sounds good okay - BUT there are issues. That's not to dismiss the idea - but to be realistic about the effort manifest in this idea.

Numbers in a public beta: the last one they did had 25. That's enough of a minor management crisis to contend with. Having 100's of people in a logicistal nightmare.

Quality: Apparently, the game was improved ENORMOUSLY - graphically, gameplay, the whole works. People criticise SMAC now - I agree, incidentally - but by the sounds of it the current version is an angel compared to the dog we might have received.

But most of the problems concerned the infrequent crashes and the end game bugs. Having more people involved doesn't help these things much - bugs can creep into a gold candidate as easily as they can a beta build. And it took me about a month to get properly into the endgame of CivII. Unless you make a concerted effort to get people to test the whole thing, you end up with a lopsided product with a flawless beginning and a nominal endgame.

Greasy letter: I think you should start collecting signatures now. This kind of this requires a major groundswell of support, rather than a couple of greasy letters (although I quietly suspect they're planning one anyway - so you might score a major coup if you get your timing right ).

The money thing: I dunno. If I'm going to test a game, I don't want to pay. I want to be paid - at least with a free game. Beta testing is hard work, and often not fun.

"Hi Brian,

CivIII crashed again. I was 5 turns into a polynesian game. Here's the save game file and the windows error message. Hope this helps!

Cheers,
Ned."

Not the best way to spend a weekend.

Never the less, if we can assemble a list of ideas, we're qualified to test the game too. Add my name to the list.

Shining1

[the Octopus]

I'll chime in again about the actual content of the letter: The basic premise is "you screwed up SMAC bigtime, don't screw up again, or else!". You catch more flies with honey...

I truly doubt that having a large number of people as alpha or beta testers is realistic. Firaxis is a small company, with a few programmers on a project. The kind of thing you want them to organize is a logistical nightmare, and of questionable utility from a bug-finding perspective. There are better ways to produce a quality product. If you simply want them to give you a vehicle so you can criticize the game engine, etc., then ask for a demo, not an alpha version.


------------------
CIV3-THE MASTER LIST-TECHNOLOGY "THREAD MASTER"

[yin26]

The idea of a limited beta does sound better. Of course, the importance of our list should be stressed in such a scenario: This list will be one of (if not THE) best indications for Firaxis what we would like to see BEFORE they get so far along that it's too hard to turn back and 'fix' it.

Thanks for all the good replies.

Yin

------------------
CIV3 DEVELOPMENT LIST COORDINATOR

**(un)Officially Making Lists for Firaxis Since SMAC Enhancement 3!**

[Blade Runner]

Shining1,

I agree with the problem to manage 1500 people is a nightmare. But my idea was the next:
1. Early beta to everybody -> we can send feedback about the CONCEPTS, INTERFACE of the game.
2. Closed beta to rid off the bugs.

Firaxis can send a kind of compatibility list (let say pentium proc, matrox G200,..., EIDE or SCSI HD ...) which parts are OK to run this early beta. If your machine is compatible with the game you can use it, if not, you need to wait for the end product.

Blade Runner

[Sieve Too]

Blade Runner: I'm still not convinced. I guess I've seen too many, "I don't like the sound" type comments to believe that a large group will be disciplined in their criticisms. Again, you open yourself up to a backlash when you don't respond the way people want you to. And even if you had a mind reader to figure out everyone's ramblings, some ideas will contradict. In a large group there is no way to get consensus. Someone will inevitably get pissed off and tell their friends not to bother buying the game since those "arrogant programmers" don't listen to good ideas. Once a negative buzz gets on the net, you might as well cancel the project.

[kmj]

Of course, they could organize some sort of forum for people to post their complaints/bugs/etc. They could have different sections for each different type of thing. This way they could weed out the more important posts. If it was organiized at all, instead of "send all comments to Brian Reynolds' desk", it shouldn't be that bad. And the fact is, the larger the pool of testers, the more likely you are to find problems. I refer you to Rong's earlier post about the Cathedral and the Bazaar.

I like the idea, but that may just be because I want to get my hands on it.

[Brother Greg]

A public beta test will never work. One reason? A beta test usually goes through multiple stages. SMAC I believe went through about ten. So, that's 10 CDs to 25 beta testers. 250 CDs with some simple math.

And guess what - they were only for playbalancing, not for bug finding - this was a mistake.

Multiply that by about, um, one hundred thousand to a million or more for an open beta (we are talking about CIV here). And then try and make money from it. You'd have to charge $200+ for the price of being a beta tester.

Plus, do any of you have any idea of the amount of crap that is actually involved in testing? It ain't just "play this game through 100 times, and tell us what you think". It is stuff like when you discover a bug, document everything you did in the last turn, if you could recreate it, and if you could, if you could avoid it, and how. Testing isn't fun, and a lot of people here seem to think that it is.

And as for anybody wading through all of the replies, trying to correlate them, and so on and so on, forget it. It would take hundreds of people.

Nice idea, but it will never happen. My reasons are just some of the more obvious.

[Travathian]

Brother Greg, you're ignorance is appalling.

Do you think any software maker would do a full public beta? Not a chance. Maybe a few hundred people, but nothing like a hundred thousand to a million. And as far as all of your cds. Know how much it costs to make a cd? Less than a buck. And if the update was small, they'd just put it on a secure site for downloading, not just send out a new disk. Granted they might do that for major revisions, but not every little thing.

Now I will agree about the playtesting for game balance only is a shame on their part.
Beta testing should be for bug fixing, playtesting, stability, etc.

[Dominique]

Please forget about that Beta thing, tempting as it might seem.

Reasons:

a) AFAIK no game company has done a PUBLIC beta until now - why should this change now?

b) Let's a ssume a public beta is out - what could happen?
- the beta is bad, and Firaxis gets negative feedback... baaaaaaaad marketing
- the beta is good, and people enjoy it so much that the tension (aka "motivation to buy") is lessened - after all, why pay $40 for some slight improvements when you could play the core game for $10?

c) Activision showed us how to successfully sell a game (sadly so): Make a big hype, sell a public beta as release, and THEN patch it (after the people have paid full price).

So?

[Shining1]

Personally, I think that firaxis will handle the beta the same way they handled SMAC - a small committed group of americans(?) to do the hard yards, and then a demo release to generate widespread public feedback.

Okay, it didn't work for SMAC. My reasoning - the game wasn't really up to scratch when it was released to the testers. Thus you had the 'improvement' process problem - the game did get a hell of a lot better while being tested, thus making everyone very happy and hence missing the fact that it wasn't yet 100%.

To get around this, I suggest a two stage public beta - 25 people to start with and then another 25 a month or so later. Hence you don't get the 'added value' problem with the second round of players, who see a markedly improved version of CivIII as the starting point, and not as a mountain of hard work and argument.

And you get brian to include the sea borders, under threat of physical violence, if nothing else works.

Travathian: I don't get it. Didn't BG just say the same thing? How is then appallingly ignorant?

The 'flame BG sh*tless over nothing' tactic seemed to work for Yin - I guess it's worth a try .

Shining1