chantz explains regression testing

[Asmodean]

Now bear in mind that I know next to nothing about these things. But it struck me that in the days just after Civ3 was released, Soren Johnson - the AI programmer for that game - said that one of the ways he tested the AI was to let the game play an entire night with no human players (something which in that game can only be done in debug mode). Surely QSI have a similar method

Asmodean

[LDiCesare]

Playing the AI against itself will have little to do with regression testing. Regression testing expects the results of the run to be always the same so that they can be compared (it's a bit more complicated, e.g. if you want to test performances when you have the right to be better...).

You want to track inconsistencies and crashed, it is not the same as launching the AI do stuff all alone. You want to track selections, actions from the player when you do regression testing.

Some regression tests could be automated by playing the game against itself, but it would be very limited, and you would have to develop some code to "read" the output (for instance if you make a scenario where you want to test a warrior vs. 3 tanks fight, you have to check the warrior is dead after the fight, which Soren Johnson or you will see by casting a glance at the screen, but the computer has no idea how to report that.

When you want to simulate thousands of different combinations (think each unit vs unit fight in Civ - now think of the variety of units in Moo, spice up with races special abilities and tactical comnbat): thousands of fight reports should be looked at to see if everything went on as expected.

If you want to test these, you'd better write some actual testing software. Believe me, it is much more complicated than launching the game (against itself or not). This could be partly automated, however.
(edited: added paragraphs)

[Corentor]

Originally posted by Asmodean
Now bear in mind that I know next to nothing about these things. But it struck me that in the days just after Civ3 was released, Soren Johnson - the AI programmer for that game - said that one of the ways he tested the AI was to let the game play an entire night with no human players (something which in that game can only be done in debug mode). Surely QSI have a similar method

Asmodean

Sure, they probably do but that will only show the more obvious bugs.

Most of the bugs that are found through regression testing occur because the tests are designed to run through boundary cases of situations or are designed to run every single line of code possible.

The AI will do 90% of the things humans will do but its the 10% that they don't do that they need to check for. But they can't tell exactly what the AI has done or hasn't done so everything still needs to be tested.

[Xentax]

Originally posted by moomin


This seems pretty obvious, yes. Given that you are something of an industy insider, would you care to guess about what the reasons behind this seemingly self-destructive behaviour is? Is it because the customers accept it? Is it because it has become a norm noone seeking funding dare deviate from? It's hard to understand it from the outside.

The problem, IMHO, is that too few people understand that software really IS engineering, as much as building a bridge or fabricating a circuit board.

There are several reasons for this. One, software is intangible -- you run the binary, and things happen, but only someone who understands the source code can appreciate the complexity of what is making those things happen. It's much much easier to be impressed by a towering bridge or an intricate circuit board.

Second, a lot of people seem to think that, since seemingly anyone can do fairly basic stuff -- create an HTML page, or a simple VB GUI, that the rest of it can't be "all that hard". Of course, these same people don't apply the same logic to building a sand castle vs. building a REAL castle.

Third, since software CAN be modified and improved after the fact, there's a perception that getting it as right as possible the first time isn't AS important as it is for, say, building construction. Again, a flawed assumption -- good software is a product of up front analysis and design, like any engineering effort. People that just jump into writing the code are DOOMED to fail, and don't let them tell you otherwise.

Finally, the ultimate problem is that the people *making the decisions* are the ones who aren't waking up to these facts -- that software development IS an engineering process, and one that needs the same level of attention to detail, preparation, review, and so on.

When the right people understand these issues, and give them due attention, you get software done on time, on budget, feature complete, and stable. The very first project I worked on for a paying customer, less than 6 months out of college, *doubled* its performance requirement, on hardware that cost half as much as the originally proposed spec machine, and *does not crash* -- I mean it is almost literally impossible to crash the software (you have to have some other app sabotaging the OS, or have the OS or the hardware fail). That was a 6 month project, of which only 1.5 was spent writing the code -- almost a month was spent first nailing down the requirements, another month doing (and reviewing!) the design, then implementation, then testing, performance tuning, etc. Just writing code from day one, I'm sure the project would have taken twice as long, or more, because the decisions and changes that became necessary during requirements and design can be made MUCH faster if code doesn't have to be changed as a result.

[Bird-O-Prey]

"When the right people understand these issues, and give them due attention, you get software done on time, on budget, feature complete, and stable."

Xentax,
For one who is a BT for a game with so many missed release dates, that is one helluva statement. I applaud you for it.

[Straybow]

Yep. When I took structural engr programming we spent most of our time creating the design (not really a traditional flowchart, but something close). We only started coding when the design was 100% complete. Then each subroutine was created as a null-return stub, and the code was fleshed in and tested out incrementally.

Even with that precaution I had an error in one parameter (an array dimension—we were using good ol' FORTRAN) that took two or three weeks to track down. As soon as I discovered the errant digit the whole program worked flawlessly because the testing still worked right on the other subroutines.

Obviously, with a game in which functions or features or the entire objective may change with the stroke of a pen, the ability to complete the design before coding may be hampered. That can't be fun for a disciplined programmer…

[Blade Runner]

Hmmm... I think the complexity of this program is somewhere in the 100 thousands lines of source code. It is impossible to produce a complex desing document, which has a complete description of the whole program. Probably they have a kinda big picture desing doc and many smaller design and programmer docs for each aspect/part of the game.

[Xentax]

It's not impossible to produce a low-level design document. It's time consuming, and depending on the size/stability of the dev team, it may be *overkill*, but it's not impossible.

At a minimum, module interfaces -- anywhere one major area of the code is talking to another major area -- need to be completely designed and reviewed. Anything one person is designing and another implementing should also see as complete a design as possible, because the designer's set of assumptions won't necessarily match up to the actual implementor's.

While a given dev may not need to get every last detail designed for the parts of the project that are "internal" to his area of responsibility, he still SHOULD at least cover all the bases. Ideally, he should still generate as complete a design as possible, because there IS always a chance he'll get hit by a bus halfway through.

[Zed-F]

Even more importantly than getting hit by a bus:
- Designing first (and documenting it!) means you have a chance to think about all the questions you need to answer, so that you are less likely to get halfway through implementation and suddenly realize you forgot something completely
- Having a review of written documentation will give other designers working on related components or components you need to interface with a chance to see what you're doing, make suggestions, and accomodate what they're doing to fit what you're doing (more than just at the interface level -- for instance, common look and feel, similar debugging tools, etc.)

[Satyre]

Heh... Moomin, you have provoked a lot of discussion...

Funnily enough you compared MOO3 with Neverwinter Nights... which had to be one of the most shoddily developed games in the history of the gaming indusry.

I am truly saddened that bioware managed to make such a mess of a game with so much promise. Honestly, their quality control process has fans living in fear of what the next patch will break, a sad thiing really.

-Perhaps they were rushed into releasing the game, rather than simply putting off the release date until the game was wel and truly finished.



Personally I have enough other interests and enough to do that I am not going to chew off my fingernails waiting for a game, no matter how eagerly I might await it.

....in the end I sit back and enjoy the excitement of the expectation.... ahhhh sweeet anticipation.

[kalbear]

More often than not, a low-level design doc ends up looking eerily like code; my argument would be to write self-documenting code, and allow it to explain the low-level architecture.

Design first is often an excellent strategy, but it's not the only one that works. Ever read any XP/Agile programming books, Xentax? I would argue that many developers think in terms of coding as the way they explain design, and so long as you approach coding early on as rough drafts that will be modified or thrown away in the future, it can be just as stable, useful, and reliable as fully specced, fully designed engineering projects. Often moreso, because they are able to go through many iterations, and iterations often produce better code.

The most important thing for everyone out there to understand is that the specs are the key. Period. Bad specs == bad code. Incomplete specs == incomplete code. If one expects to write anything of quality when specifications aren't understood, they are wrong.

That's clearly one of the bigger flaws of MOO3, and we all know it. The specs called for IFP, which is a high-level way of controlling the entire game. Removing it changes a LOT of the way that the code was organized, prepared, and planned on. It effectively could touch everything. That's something of a biggie. Every time features get added or removed, again, incomplete specs. These are things that are almost impossible to avoid in the gaming environment without massive beta testing (like Blizzard does, for instance), and in general are hard to deal with when releasing a commercial retail level product. The less customers a product has, the easier it is to get specs right the first time. Retail has the most customers. QED.

[moomin]

Originally posted by Xentax
The problem, IMHO, is that too few people understand that software really IS engineering, as much as building a bridge or fabricating a circuit board.

Hmm. But this is true for the entire software industry - people have much more respect for hardware engineering than for software. Nevertheless, it seems downright ludicrous practices with regard to time planning and project requirements management isn't so much an exeption as a rule in the PC gaming indutry, and given the countless failures of the model currently employed you'd think the backers would finally start doing their due diligence and demand better practices. I still don't understand why this isn't happening.

[moomin]

Originally posted by Satyre
Funnily enough you compared MOO3 with Neverwinter Nights... which had to be one of the most shoddily developed games in the history of the gaming indusry.

Well, I've played NWN every weekend online with my old collage pals since it was released, and while it took a patch or two until it networked reliably, I haven't had it crash on me yet. Not a single time. Neither has any of the people I play with. So I'm pretty impressed with the game, given how incredibly flexible and open-ended the editor is.

But that's neither here nor there. My point wasn't that NWN is much better quality than Moo3 will be (although I will be very, very pleased if it runs half as smooth and stable for me as NWN had been doing). The point is that it is an undertaking that very much more complicted than Moo3, because of the versatility of the toolset, which Moo3 won't even be close to match.

[Grumbold]

I'd second that opinion of NWN. It has been a very good piece of software imo. Some people had issues with the nature of the 1 player campaign that came with the box, but the whole package was excellent and stable as far as I am concerned.

[HugoHillbilly]

Originally posted by moomin


Well, I've played NWN every weekend online with my old collage pals since it was released, and while it took a patch or two until it networked reliably, I haven't had it crash on me yet. Not a single time. Neither has any of the people I play with. So I'm pretty impressed with the game, given how incredibly flexible and open-ended the editor is.

But that's neither here nor there. My point wasn't that NWN is much better quality than Moo3 will be (although I will be very, very pleased if it runs half as smooth and stable for me as NWN had been doing). The point is that it is an undertaking that very much more complicted than Moo3, because of the versatility of the toolset, which Moo3 won't even be close to match.

As I recall NWN was in developement for 5 years. Enough said.