chantz explains regression testing

[Capt Dizle]

Originally posted by HugoHillbilly


As I recall NWN was in developement for 5 years. Enough said.

We may be able to say the same about Moo3 when and if it is released.

[Xentax]

Originally posted by kalbear
More often than not, a low-level design doc ends up looking eerily like code; my argument would be to write self-documenting code, and allow it to explain the low-level architecture.

Design first is often an excellent strategy, but it's not the only one that works. Ever read any XP/Agile programming books, Xentax? I would argue that many developers think in terms of coding as the way they explain design, and so long as you approach coding early on as rough drafts that will be modified or thrown away in the future, it can be just as stable, useful, and reliable as fully specced, fully designed engineering projects. Often moreso, because they are able to go through many iterations, and iterations often produce better code.

If I said, or implied, that a complete low level design should be done before implementation starts, I apologize as that was not my intent. And, as I said, a *complete* low-level design is often not necessary or even appropriate. Depending on the situation, it's quite possible to start coding on one area or module before completing the low-level design of another area. We usually try to get all interfaces (and, for C++, all headers) done first, then we're free to fill in the rest, even if "the rest" involves some additional low-level design before coding. By "interfaces", I mean any module-to-module communication, not just "interface" in the OO sense of the word.

However, a complete *high level* design, BEFORE implementation, is a MUST, at least IMHO.

I have done some XP, and I've read about XP/Agile development. Paired programming can be a WONDERFUL experience in terms of code quality, and it's a great way to "learn the ropes", pairing an experienced developer with a lesser-experienced one (I've been in both positions now, though I'm still more likely to be the lesser-experienced one in most situations).

But, as others have said, you HAVE to spend time up front in design, because any potential oversights or gotchas ("Sleeping Dragons" as I've heard them called) are MUCH cheaper to resolve before you start coding.

The most important thing for everyone out there to understand is that the specs are the key. Period. Bad specs == bad code. Incomplete specs == incomplete code. If one expects to write anything of quality when specifications aren't understood, they are wrong.

That's clearly one of the bigger flaws of MOO3, and we all know it. The specs called for IFP, which is a high-level way of controlling the entire game. Removing it changes a LOT of the way that the code was organized, prepared, and planned on. It effectively could touch everything. That's something of a biggie. Every time features get added or removed, again, incomplete specs. These are things that are almost impossible to avoid in the gaming environment without massive beta testing (like Blizzard does, for instance), and in general are hard to deal with when releasing a commercial retail level product. The less customers a product has, the easier it is to get specs right the first time. Retail has the most customers. QED.

I don't know how much game concepts like IFP affected game design in terms of code. I'm sure removing IFP had a non-trivial impact on the code, but it's hard to estimate just how big a change that required. I mean, they could have left a lot of the mechanics alone and just kept a hidden, non-depleting pool of IFP. Obviously, certain techs and actions had to be removed since they were tied to IFPs directly, but techs at least are pretty much data rather than code, so removing them isn't too big a deal. They may leave behind game mechanics that are never used, though.

In more general terms, we at least *try* to keep specs up to date when situations change -- so that, at the end of the project, the design DOES match the code. Of course, we're usually short of the ideal, since revising design docs after or during the fact takes time (and we all know time = money).

All of this tends to be problems of scale. On a small project, it's both cheap and easy to wiggle as needed. On a larger project (especially if the team is large), you pretty much have to keep the specs current or you'll have inconsistencies in implementation, as some people will know about changes but others won't.

[kalbear]

We usually try to get all interfaces (and, for C++, all headers) done first, then we're free to fill in the rest, even if "the rest" involves some additional low-level design before coding. By "interfaces", I mean any module-to-module communication, not just "interface" in the OO sense of the word.

Agreed, though often the interfaces dictate well enough the behavior of the modules and the interactions so that full-fledged documentation is not necessary.

A complete high-level design for a retail product is a must, also agreed. This is especially true for the UI framework; the underlying mechanics don't need as much of this, as it's basically one large spreadsheet with various things acting on it (some of which are players, some of which are AIs).

But, as others have said, you HAVE to spend time up front in design, because any potential oversights or gotchas ("Sleeping Dragons" as I've heard them called) are MUCH cheaper to resolve before you start coding.

Depends on the problem at hand, the scope of the problem, and how expensive it is to refactor that code out of there. I brought up the example of IFPs as a tough one to remove, and the reason is that it touches everything. All the graphics need to be changed, all the ways you interact with those graphics need to be modified. It's a pretty large interface change on the macro and micro level, and those kinds of high-level design changes usually are expensive.

Except that, if you have everything else in place, it's not so bad. If you have all the unit tests, if you have the graphics frameworks and the storyboard, it's just a matter of making a lot of propagating changes all over the place. It's work you have to do again, but it's not hugely costly. That's at least one of the ideas behind XP - that future design changes won't be horrendously expensive in time and manpower.

I think that can be very true for a small group of people working on a retail product that doesn't involve fabricating hardware - much like, say, almost all games published. That isn't to say you should just start coding and be done with it; it's to say that design should be considered and reconsidered every time you touch the code (refactoring, another rule of XP), and you shouldn't be afraid to do so early.

All of this tends to be problems of scale. On a small project, it's both cheap and easy to wiggle as needed. On a larger project (especially if the team is large), you pretty much have to keep the specs current or you'll have inconsistencies in implementation, as some people will know about changes but others won't.

True, though with things like pair programming and a large body of unit tests, the specs need not be updated nearly as much.

Games, I feel, really can benefit from the XP process - at least for devs. Having massive unit testing done early is a godsend, after all. Being able to up-front change design without serious expense, because you've already budgeted that in and plan on doing so multiple times, seems like the way to go. Having a working model as early as possible also seems like a good thing. Imagine how many problems in, say, Everquest, would have been solved this way?

[ShuShu]

Moomin, I don't know of many applications...

...that have the user base, platform diversity, and reliability requirements of computer games.

I spent 8 years working for a securities exchange which had a parallel processing hot backup, and 16 redundant databases because each minute of downtime was pegged at 1 million dollars. Even so, that was a cakewalk compared to game testing. And, appropriately, we didn't have nearly the regression staff chantz described.

I also worked for a credit history firm, building an API between the applications and the database. We had the fortune and misfortune to be able to completely automate the regression test, so that the test took 5 hours. ( Of course, getting a trillion test cases down to 0 bugs took a few years. )

MOO III might not have a trillion test cases, but testing cannot be automated very much (because of the custom graphical interface).

Bottom line is that a bug free product is impossible, and most people are ignorant of that fact. Chantz's reply tried to explain why. It also tried to go further and say that the product [B]could[B] be released now, but was not, and that represented a commitment to quality. I would like to add: a commitment that is sorely lacking in today's computer game industry.

Your response to Chantz's post reminds me of the director who almost fired me on the spot when I told him that every release of software we ever loaded into his system had bugs.

Pop Quiz: What's the biggest problem with doing a thorough job of looking for bugs?

Answer: Finding them.

Paraphrased: Ignorance is bliss...

[ShuShu]

Okay design freaks...

Paralysis by Analysis... I am a systems architect, so believe me when I say, I love designs, but you have got to start building to test it out. Too much time is wasted arguing whether something will work, too little spent on arguing about what is needed.

Building a program is not like building a bridge. All the bridges in the world have the same set of rules. Bridge stand, Bridge bear load, Bridge not fall down. Sir Isaac Newton is the primary subject-matter expert, now go do your max/min projects and be done with it. Every program has different requirements and different SMEs.

Even with bridge building, every design has build and test cycles incorporated on a scaled down models.

The gaming industry has an even tougher row to hoe. Even if the code satisfies the design flawlessly, it still has to be fun. Fun is not as easy to measure as, Bridge still standing or field not null. The only way to test "fun" is to build it first...

I see the problem as just the opposite. Too much of the game developement cycle is now dedicated to pre-build work, and too little dedicated to post-build adjustments.

PS I believe the reason for this is economics 101 stuff. Why pay for a QA staff when you can get a QA staff to pay you for the privilege? The only risk comes when you are too far off the mark and cannot rework your way into a quality product (HOMMIV). This approach only works for established franchises. New franchises tend to be more solid because the free QA staff does not yet exist...

[Xentax]

Originally posted by kalbear


Agreed, though often the interfaces dictate well enough the behavior of the modules and the interactions so that full-fledged documentation is not necessary.

A complete high-level design for a retail product is a must, also agreed. This is especially true for the UI framework; the underlying mechanics don't need as much of this, as it's basically one large spreadsheet with various things acting on it (some of which are players, some of which are AIs).



Depends on the problem at hand, the scope of the problem, and how expensive it is to refactor that code out of there. I brought up the example of IFPs as a tough one to remove, and the reason is that it touches everything. All the graphics need to be changed, all the ways you interact with those graphics need to be modified. It's a pretty large interface change on the macro and micro level, and those kinds of high-level design changes usually are expensive.

Except that, if you have everything else in place, it's not so bad. If you have all the unit tests, if you have the graphics frameworks and the storyboard, it's just a matter of making a lot of propagating changes all over the place. It's work you have to do again, but it's not hugely costly. That's at least one of the ideas behind XP - that future design changes won't be horrendously expensive in time and manpower.

I think that can be very true for a small group of people working on a retail product that doesn't involve fabricating hardware - much like, say, almost all games published. That isn't to say you should just start coding and be done with it; it's to say that design should be considered and reconsidered every time you touch the code (refactoring, another rule of XP), and you shouldn't be afraid to do so early.



True, though with things like pair programming and a large body of unit tests, the specs need not be updated nearly as much.

Games, I feel, really can benefit from the XP process - at least for devs. Having massive unit testing done early is a godsend, after all. Being able to up-front change design without serious expense, because you've already budgeted that in and plan on doing so multiple times, seems like the way to go. Having a working model as early as possible also seems like a good thing. Imagine how many problems in, say, Everquest, would have been solved this way?

It's that budgeting for changes, in terms of both cost and time, that is the great failing of so many software projects. It's so much more common to have less than 100% of the time you'd need to do it in UTOPIAN conditions, and we all know that it NEVER goes 100% according to plan.

Maybe that's one reason the project manager we had with a Marine background was SOOO capable (far and away the best manager I've ever worked under) -- he *expected* "the waste material to interface with the air oscillation unit" at least once per project, and planned appropriately.

[Asmodean]

Originally posted by Xentax


he *expected* "the waste material to interface with the air oscillation unit" at least once per project

HeHe...Nice one, Xentax.

Asmodean

[Gooberman32]

well dont you have to get console games perfect the first time because you cant patch a console game? so you kind of have a car that you cant fix, so you would spend more time getting the console game perfect or something close to it.

[Harry Seldon]

Originally posted by Gooberman32
well dont you have to get console games perfect the first time because you cant patch a console game? so you kind of have a car that you cant fix, so you would spend more time getting the console game perfect or something close to it.

Enjoy this while it lasts. I have an Xbox and it already has games that could use a patch. As online console gaming becomes more prevalent, it offers a convenient way to patch a broken game. Console players like Sony and Microsoft won't complain because it forces consumers to participate(buy) in their online experience. Look for more shoddily produced games to hit the market as more people get online; it's not going to be any different than the PC market

[Xentax]

*If* the console-online-experience catches on, anyway -- XBox seems to have a small but rabid following so far, and the requirement that the X-Box Live games all support voice and game-matching service, etc. is an interesting approach.

Considering that by the time you get the console, the modem or broadband adapter, and a keyboard, and whatever else, it really does become little different than a PC (except that the target platform is much more uniform).

I dunno, I suppose it'll happen, if good enough and unique enough games are console only (I'm still royally pissed that Halo hasn't come out on the PC yet!).

But, AFAIK, only the X-Box has the internal hard disk that a patch really needs to be viable on the console -- I can't see the PS2 and Gamecube trying to use a memory card to store a patch...

So, Gooberman, yes you DO need to get much closer to "perfection" for console games, since patches aren't a possibility yet, and probably won't be for some time. However, since you have a consistent target platform and a purpose-built bare-bones OS (rather than a multi-purpose does-all one like Windows, MacOS, etc.), it tends to be a somewhat simpler problem to solve, though it will always require very vigorous testing.

And you usually *do* find a few glitches in any console game, but they're rarely if ever major ones. For example, I noticed that the "You are here" marker in GTA:Vice City is slightly offset from your actual position, at least sometimes. Annoying, to be sure, but also not surprisingly something that escaped the testers' attention.

[Harry Seldon]

The PS2 has an attachable hard drive that's required for some games, FFXI being the most notable. I believe the EverQuest port requires the hard drive as well.

[Asmodean]

Hmmm...maybe it's time to "unsticky" this thread. No replies since january 6th.

Asmodean

[Kc7mxo]

Six stickies is a bit much.