chantz explains regression testing

[Grumbold]

Momin the nice thing is that they are even using rigorous QA methods and sticking to them. Many othr developers either don't employ them or allow the publisher to publish and be damned despite the results. I'm sure MoO3 will still have an issues list when it ships, but hopefully nothing that should have us tearing our hair out waiting for the patch to arrive...

[Xentax]

Memory leaks can be hard to track down, especially if you don't have the right tools (like Purify); there are also various ways to program defensively to reduce the chance of them cropping up in the first place (auto pointers, making objects reference counted, etc.).

Usually, they end up being a contractual thing -- block of code A calls block of code B, object(s) created, B thinks A will be responsible for cleaning it/them up, A thinks B will, and voila, memory leakage.

Still, they're usually fairly easy to fix once located, unlike, say, some sort of unanticipated threading issue.

Moomin: While software engineering (and the subset, testing) practices might not be a "newsflash" to everyone, an overview of what's involved probably IS new information for a majority of the people reading about MOO3 (even moreso on the IG boards than here, perhaps).

Lots of people that SHOULD know better don't, even -- for example, the next pointy-haired type that says "It's just software" in my presence won't live long enough to make another such baseless assumption...

[moomin]

Originally posted by Xentax
Moomin: While software engineering (and the subset, testing) practices might not be a "newsflash" to everyone, an overview of what's involved probably IS new information for a majority of the people reading about MOO3 (even moreso on the IG boards than here, perhaps).

Perhaps. Given the amount of impatient whining about release dates all over the net, some explanation was clearly in order. It jsut rubbed me the wrong way. I get the impression Chantz thinks this is something terribly unique being done. And it ain't. Except, perhaps, in the gaming industry (or, PC gaming indutry - I bet the console shops do their programming and testing by the book).

Lots of people that SHOULD know better don't, even -- for example, the next pointy-haired type that says "It's just software" in my presence won't live long enough to make another such baseless assumption...

As a former IT slave who worked the trenches and propelled myself into pointy-haried-hood by means of MBA, I feel your pain. I just hope you aren't implying that there's plenty of these guys around at QS?

[Ozymandous]

Originally posted by moomin
Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

Here's a newsflash. Software gets written using methods like these all the time. Some of it orders of magnitude more complex than any PC game. And while it would be unfair to say many software projects deliver on time, Moo3 current troubles are somewhat exeptional - but bad project management seems to be the standard in this particular industry. Still, that is what we're talking about. Bad project management. In fact, from the outside, it seems like atrocious project management.

Funny, I don't seem to remember people making fan sites and/or frothing at the mouth on message boards daily anticipating the next installment of the MS Office suite do you? Which means that MS can put no release date what-so-ever and people will just pick it up when they see it. This is SELDOM the case with any game, people want it NOW and usually post negatively when they can't have their instant gratification.

Easy to throw stones when you have *absolutely* no idea what sort of PM QSI and IG have isn't it? Tell you what, since you seem to have all the answers, why don't you start your own company and develop the perfect game since you obviously know how to do everything perfectly.

Better yet, try to be positive for a change, heaven forbid you aren't absolutely pissing your pants with joy when MOO comes out or I guess we'll have to call you " " and you'll have to go back to the "perfect" Civ2.

[Ozymandous]

Originally posted by moomin


Perhaps. Given the amount of impatient whining about release dates all over the net, some explanation was clearly in order. It jsut rubbed me the wrong way. I get the impression Chantz thinks this is something terribly unique being done. And it ain't. Except, perhaps, in the gaming industry (or, PC gaming indutry - I bet the console shops do their programming and testing by the book).

Ah, comparing console to PC game, like apples and oranges... Consoles are *so* hard to develop for when you know exactly what software & hardware you're writing for right off the bat.

Console games = kit cars, you know where each part goes because they are all the same.
PC games = hand crafted and built-to-order concept car where you have an idea of what you want but have to figure out just *how* to make this spiffy piece fit around the gas tank, between the shocks, axel and drive shaft.

They are a tad different (not better or worse, just moe to developing a PC game.)

[Craig P.]

Originally posted by moomin
Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

I think you are completely missing the context for chantz' post. IMO, it's a necessary response to the whinery that has been a substantial portion of the postage on the official boards.

Note that in a vacuum, or in the context of this forum, I think your response is reasonable.

[moomin]

Originally posted by Ozymandous
Easy to throw stones when you have *absolutely* no idea what sort of PM QSI and IG have isn't it?

My, such hostility. Can't be good for the blood pressure, you know. Now, I have a pretty good idea of the measures QS employs - after all, everybody who isn't illiterate is perfectly able to read about them in some detail in Chantz's posting!

Tell you what, since you seem to have all the answers, why don't you start your own company and develop the perfect game since you obviously know how to do everything perfectly.

I already head a development shop. True, we don't do games, we do medical software, but so far we've never missed a delivery date, and believe me, our QA procedures leave anything in the gaming industry in the dust, if only for liability reasons. Now, mr Friendly, what exactely are your qualifications, except for ranting?

[moomin]

Originally posted by Craig P.
I think you are completely missing the context for chantz' post. IMO, it's a necessary response to the whinery that has been a substantial portion of the postage on the official boards.

That's quite possible. I haven't frequented those. But I still can't help but to think that some boilder-plate CB would have done just as well as an explanation and left a lot of stakeholders happier.

[Xentax]

Oh, I wasn't referring to pointy-hairs at QSI, rather the group at one of our (former) customers.

As I've said before, the PC gaming industry seems to lack the techniques a lot of the rest of the software industry uses to plan projects that get done on time and on budget; personally, I think it's a response to short-cycle pressure that's become a real trend, hopefully one that will be reversed soon.

I mean, the short cycle is really a *myth*, nowadays. Companies keep promising games out in a year or 18 months, and end up being a year late. If they'd given themselves the 2 or 3 years they needed, up front, they'd end up with a BETTER product at the SAME end date, because MOST (not necessarily all) people, including engineers, work better when they're not laboring against an impossible deadline. When you're trying to do the impossible, corners are invariably cut, shortcuts are taken, and the product suffers as a result.

[Gooberman32]

Im not sure about recent advances in console technology, but can console games be patched?

[moomin]

Originally posted by Xentax
I mean, the short cycle is really a *myth*, nowadays. Companies keep promising games out in a year or 18 months, and end up being a year late. If they'd given themselves the 2 or 3 years they needed, up front, they'd end up with a BETTER product at the SAME end date, because MOST (not necessarily all) people, including engineers, work better when they're not laboring against an impossible deadline. When you're trying to do the impossible, corners are invariably cut, shortcuts are taken, and the product suffers as a result.

This seems pretty obvious, yes. Given that you are something of an industy insider, would you care to guess about what the reasons behind this seemingly self-destructive behaviour is? Is it because the customers accept it? Is it because it has become a norm noone seeking funding dare deviate from? It's hard to understand it from the outside.

[Grumbold]

The only positive side to it that I can see is that it forces a serious assessment of the product 40-50% of the way into development. All too often companies are making claims about what it can or cannot do at 95% of the wya to release that suddenly get cut or massively revised before release.

It would be a lot more honest to write the original project plan with a crunch point at 50%, but that just doesn't happen in most cases. Its easy to keep putting it off, thinking features are still achievable given a bit more work. There also may be an issue of Producers not wanting to take on new projects knowing that it has a minimum 3 year life cycle when many of them are financially hard pressed. I doubt many execs get promotions and bonuses for telling it like it is rather than shooting for ambitious targets then chasing the developers hard if they miss.

[LDiCesare]

One thing strikes me about the regression testing techniques chantz talks about.
What about automated non-regression tests?
Surely if he talks about code control, he should speak about that, but does not. Don't they use any? That would save them time. Of course that doesn't replace manual testing, but still, seeing no mention of it is pretty frightening.
And the definitions of huge are not the same for everyone. 20 testers? My, we have many more where I work. And we don't delay our products that much. But, sometimes, for the sake of quality, maybe we should...

Plus I wonder whether code reviews are as efficient as pair programming. Anyone has opinions/experience about that?

[Zed-F]

Automated testing works best in an environment with limited churn. Otherwise the (non-trivial) cost of writing and rewriting the testcases as things change exceeds the value you get out of them. We typically only consider introducing automated testing after the first release of a product, once there is a relatively solid base from which to work and once the time pressure has abated. MOO3 is not yet in that position, but I would think that automated testing is something that they would consider implementing as they start working on the first patches, to help ensure the patches don't break existing functionality.

[Straybow]

Moomin, is that medical database or equipment control, or maybe something else entirely? What kind of AI does that use? Does it have cool white blood cell battle graphics?