chantz explains regression testing

[Asmodean]

Full story here: http://www.ina-community.com/forums/...hreadid=242812

***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***

Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?

Okay. So MoO3 was going to be like Civ3, in that they were going release an unfinished game. I, for one, am extremely happy that they didn't. I am a strong believer in finished products.

Asmodean

[moomin]

And now inquiring minds would love to find out just what those reasons that Chantz "cannot and will not go into" are.

Chantz, my pal, feel free to mail the full disclosure to Firaxis, at least. I'm sure they'd love to know how to convince IG to allow them to ship useable products...

[Proteus_MST]

But maybe by shipping unfinished Products Firaxis just made it possible for Quicksilver to ship a useable product

[MarkG]

full copy for the time the server is busy

A bit of detail about testing MOO3

Folks.

Some people are confused about how regression testing works. I'm going to take minute to hopefully give you some insight into why it takes so long to test moo3. Note that companies have varying methodology but the end result is the *same*; shake out all the major bugs.

1: MOO3 is huge. HUGE. Did i say how huge it is? There are so many permutations and so many ways to "skin a cat" that testing them all takes a team of 20 people working roughly 16 hours a day 3 straight weeks to slog through it. That's with people assigned to specific areas of focus. Did i say how huge it is? IT'S HUGE.

2: Re: Regression testing. See the above statement? Every time we get a new build, we start from scratch and test every known permutation. Bugs come up, they always do. The important thing to note here is that *are the bugs that come up during a regression test show stoppers and the ones that we're marked as showstoppers in the previous build fixed?*

3: What's a showstopper? Many, many things. A repeatable crash. Crashing after 2-4-6 hours of gameplay. Running out of memory and crashing. Save game corruption. Multiplayer hangs. Missing feature or a feature not working properly. Missing art; the list goes on and on. We have an enormous checklist we go through. Every time we fix a bug, we take a chance that we might break something else. To mitigate this, especially during the final weeks of testing, we enter into extreme paranoia mode. We call this Code Review. Every time a programmer makes a change, it is reviewed by at least 2 other programmers who share some knowledge of that particular piece of code. A line-by-line review is done, questions are asked, flags are raised (or not, as the case may be). That bug fix then gets tested for at least a few hours before it gets "checked in" to the main source tree. It gets compiled into the build (along with any other fixes) and is then sent off for proper (see above) testing.

4: Source code control. Know it, love it, it is your best friend. When a programmer makes a change to a source or header file (or creates a new one), it is done outside the main "tree", which resides on a SERVER that all team members have access to. In other words, it is done locally on that programmer’s machine. So, if i create a file or modify a file called spies.cpp (c++ extension), i'd do it locally to ensure that i'm not stomping on code in the main tree and causing dumb errors and stopping other people from working. A file can only be checked out by one programmer (or designer or artist for that matter) at a time, for obvious reasons. This is really the condensed version, as there is much more to it, but this sums it up in a nutshell.

5: Sometimes we know right away that a build doesn't cut the mustard (a crash 20 turns into it), but we are *now* at the point where most problems are occurring very LATE GAME in MP on Large maps with many races; in other words, we are exercising and stress testing areas of the game we couldn't 6 weeks ago because we couldn't GET there due to crashes and errors. Game balance also figured into this. Heck, there was a time when you could *not* win against the new orions no matter how many ships you could throw at them (turn after turn). That means no Sole Superpower Victory. That's a Showstopper because you cannot win the game - at all. No fun.

***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***

Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?

Anyway, hopefully this gives you some insight into how regression testing (at least in my little world) works. No questions please, just enjoy (or not, as the case may be).

Thanks,
Constantine

[Main_Brain]

mhh extensive Testing..
I conclude that there are Cheats in the Game :P

[Gooberman32]

I was right then bugs are late game

[H Tower]

hard to get rid of those

[Skanky Burns]

Running out of memory and crashing.

Memory leaks are evil. At least that is one advantage of using java over, say, c++.

[moomin]

Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

Here's a newsflash. Software gets written using methods like these all the time. Some of it orders of magnitude more complex than any PC game. And while it would be unfair to say many software projects deliver on time, Moo3 current troubles are somewhat exeptional - but bad project management seems to be the standard in this particular industry. Still, that is what we're talking about. Bad project management. In fact, from the outside, it seems like atrocious project management.

[Unspeakable Horror]

Catch 22 eh?

What does it mean?

[TCO]

A self-defeating requirement. For instance if the only place to complain about internet connectivity problems is a website. Or if you have to have experience to get a job...but the only way to get experience is to have a job.

[Unspeakable Horror]

Thanks

[habadacus]

It's nice we got bit of information, it really is, although it doesn't include a release date, and even if it did I'd be reluctant to put any faith in it, but it doesn't really seem to shed any light. I mean, yeah when you code something, you have to fix bugs. Knowing how you fix the bugs doesn't help me. That it's a long arduous process isn't news to anybody, that's why programmers and developers get paychecks. I would symathize so much more with the developer's plight if this game hadn't been in development FOREVER. YEARS EVEN!!!
He says how HUGE the game is, well thats nice, and that it takes a team of 20 people working 16 hours for three weeks straight to test it. Um, am I mising something here? I'm no Human Resourses Administrator, but maybe you should get ANOTHER 20 GUYS. How hard is it to hire game testers? I know from being one at EA they don't exactly make 100k year, in fact I got paid salary, so the all the extra hours in the crunch time didn't cost them anything more.
My final nitpicking point is he says it was a mistake to say the release date would be Nov 26th, but that was a a command from on high, and under mysterious circumstances that directive changed.
OK well, they knew the game was HUGE, they knew they only had 20 OVERWORKED GUYS , they knew they had to do SOURCE CODE CONTROL and other exciting and arcane sounding things and that the game had bugs and missing art and everything else. So given all that, and the huge sigh of relief everyone must have enjoyed after the Borg or whatever said they didn't have to go gold on the 11/26, then what the hell was up with the Dec 4th release date? They were so glad to to have extra time and be released from the demands of thier demon masters. Who got the bright idea that 8 days was gonna be the ticket they needed to release what by now ought to be the BEST GAME EVER?
Well, anyway, I feel better after my little rant. Although I whine and moan, it's only because I'm DYING to play this game already. I took a week off from work starting the 18th becasue I was hoping the game would be out by then, and again it looks like I will be disappointed. (and bored... Hegemonia sucks) And this is the best they can do by way of explaination? Felgercarb! If they miss December altogether, and I recant and actually do buy this game it had better transform my computer into and actual ALIEN BATTLECRUISER. Anything else will seem anti-climactic.

[Asmodean]

@habadacus

You very nearly turned me around with your post. But then I realized that QS and IG have promised us a flawless game. And that's what makes the waiting time durable for me. OK...so they've made mistakes, what with the release dates and all. So what. People make mistakes. Deal with it.

I, for one, would much rather wait untill january or february, and then play a bug-free game, than playing a game that crashes in turn 258 in my christmas holiday.

Asmodean

[Zed-F]

Habadacus, adding more people at this point will not solve the problem, it will worsen it. Adding more people 6 months or a year ago when they might have had time to train them might have helped, but not now, and 6 months or a year ago they either (a) didn't realize that they would need more people or (b) far more likely, realized that they needed more people but couldn't afford to hire any more. So, they had to make do with what they had.

Moreover, there are natural limits to how much you can effectively subdivide any given set of work. I don't know if the regression team has reached that point, but throwing more people at a problem does not always help even if they have lots of time to train before crunch time. Nine women cannot make a baby in one month.