Saving and restoring games

[Gary Thomas]

I can't find any earlier references to this, though I remember a considerable discussion.

Having offered to code this, I propose to implement a system I have had in mind for some time, using the well known principle of coder's privilege.

The System:

1. Saving is automatic at the end of every turn.

2. A game may be saved part way through the current turn only - at the end of the turn the part way save is removed.

3. The data is saved in an incremental fashion, with a complete copy every so often (the interval to be derived by experiment), though truly unchanging details will be saved once only.

4. Each model will have its own save class, separate from other models.

5. The save file will be zipped.

6. If an earlier game (other than the last move or the current move) is reloaded, any subsequrent saves are lost. However, the player will be given an option to branch the saved game, thus producing another, and distinct, save file.

The advantages:

1. There is no silly micromanagement of save files as happens in Civ.

2. A complete playback of the game is possible.

3. Waste of space by duplicating static data, move after move, is much reduced. In particular, the basic map is saved once only.

4. It exploits the particular assets of Java.

5. I expect to code it so that loading a game is essentially instantaneous. I say this with some confidence, after recoding a program that manipulated an 8Mb Access database into Java. It turned out that the entire information in the database can be stored (zipped) in about 500K, and can be read in in about 1.5 seconds.

Cheers

[Mark_Everson]

Hey Gary:

The previous discussion you remember is probably in Planning for Demos 7 & 8.

I repeat my criticism of your approach. Essentially I believe that the save files will be prohibitively large. Here's what I said before:

I just think that So Much is going to change over even a few turns that you will end up saving virtually the entire game every time anyway. The unchanging parts are: The map outlines, and numbers of sites per map square will change infrequently. Just about everyting else will change over relatively small numbers of turns. Everything in the economy changes gradually, Same for much of the Social model. These are the largest memory-users in the game.

However, you can do the estimates of the save file size better than I. I've said my piece, and won't gripe any further! But I will say I told you so when the save files start approaching 100MB. (I'm not talking about for the current scenarios, but rather full-map games. There is no detailed calculation to back up my contentions. . .)

Cya,

Mark

[Gary Thomas]

If your answer is that the save files will be too large, then all we have to do is try it out. If the save files are too large, then you are right. On the other hand, separate save files (which appears to be the alternative that you prefer) will necessarily be even larger. So you reject one method in favour of a worse one?

Cheers

[Mark_Everson]

Hi Gary:

If you save all the game info then you only need a Single (or a few) large files rather than one that contains 100x as much info (100x is an order of magnitude estimate due to saving state changes every 10 turns for a 1000 turn game.) Seems straigtforward. I agree you lose the game-evolution info this way.

But anyway, as you say, the empirical approach will soon show what the relevant numbers are.

[Gary Thomas]

If you save all the game info then you only need a Single (or a few) large files

Every time you save, which, if Civ3 is any indication, is every turn.

Cheers

[LDiCesare]

5. I expect to code it so that loading a game is essentially instantaneous. I say this with some confidence, after recoding a program that manipulated an 8Mb Access database into Java. It turned out that the entire information in the database can be stored (zipped) in about 500K, and can be read in in about 1.5 seconds.

I know the stuff. I prototyped things last year where I could gain a factor of 100 in file size because of far too many information duplications in the file format.
I agree with Gary's approach. It should be tried. If size is an issue, we will see when this happens, and it should be relatively easy to lose the previous turns if that's what you want. Keep in mind that just zipping the file often gives very good results.

[Gary Thomas]

Thanks Laurent.

It is because of work experience that I am convinced that the results will be startlingly positive.

Cheers

[Gary Thomas]

I have in mind an extension of the system used to input scenario data. There will be simple interface (read, write, add, subtract) to process character data in xml format. Each save file will contain the whole initial scenario at the start. My first cut will just contain incremental data (apart from the initial scenario). It will start with a list of dates, allowing a player to load other than the most recent version, from a menu.

There will also be a restore option, which immediately loads the most recent game at its most recent point (an option missing from Civ, though present in other games).

Cheers

[Gary Thomas]

I have had a temporary rethink!

There is now some pressure to get something available soon, so we can produce version 8.

Also, I have realized that the code is closely related to the code for reading in a scenario - I can save a lot of effort (and reduce the amount of code) by using that code to restore a game.

As well, since my next project is a scenario editor, I can use the save game code there as save scenario code.

The consequence of all this is that, at least as a first cut, the save/restore system will essentially be a save/restore scenario system.

Once that is working I will write the scenario editor.

And once that is working I will work on the incremental save option, since, by that time, I expect that even Mark will realize that the files are getting large.

Cheers

[Mark_Everson]

Originally posted by Gary Thomas
As well, since my next project is a scenario editor, I can use the save game code there as save scenario code.

The consequence of all this is that, at least as a first cut, the save/restore system will essentially be a save/restore scenario system.

Once that is working I will write the scenario editor.

Sounds Outstanding!

And once that is working I will work on the incremental save option, since, by that time, I expect that even Mark will realize that the files are getting large.

Personally, I think it'll take Mark longer than that to come around, but you may be right. . .

[Gary Thomas]

OK, apparently everything is on track except my ability to convince Mark that I know what I am doing.

Cheers

[LDiCesare]

That's not really a problem, since you'll be doing it anyway.

[Gary Thomas]

I have encountered a roadblock in my save/restore coding.

There is a structural problem which I have discussed previously (about two years ago).

The basic game was designed to read in a whole lot of files - civilizations, military, social, economic, religion, and so forth, overlaid by specific scenarios.

Frankly this is a big unmanageable mess, and clashes horribly with the save/restore mechanism.

It is also something that I have been against from the beginning.

My conception and preference is that each scenario definition file should be completely self-contained, so only a single file is read in for each play of the game.

The objection to this was that it meant that scenario designers would have to repeat large amounts of data, from one scenario to another. This applies particularly to the military data.

Having given the matter considerable thought recently, I realized that we have been on a slightly wrong track.

Yes, there should be comprehensive files on the topics I listed at the start of this post. On the other hand, a scenario should be a single file and the program should not need to read in a large number of files containing irrelevant data.

The solution, now that I have been studying the data saving system in some detail, is a properly designed scenario editor which will take the necessary parts of the standard reference files and produce a single scenario file.

As well as being tidier than the present system, it will add one very large benefit. It will mean that the global data will be included in the scenario file. So, in order to test the effect of these global parameters one will be able to produce a test scenario in which only a global parameter is changed. This could them be downloaded and tested by people other than the coders.

The effect of this is that I am more or less forced to exchange my top priorities and work on the scenario editor now.

This in turn means that it will not be done this week, as I had hoped.

Cheers

[Gary Thomas]

I put this in a separate post to avoid diluting the impact of the previous post. This relates to names of things in general (though not to language, Jack).

One of the problems we have had in scenario design is that the same civilization has different incarnations in different scenarios. For example, the Roman civilization of Delenda is quite different, and many centuries removed, from the Roman civilization in Attila. There is no provision in the data file system to differentiate these, apart from giving them different names.

My proposal, to get around this difficulty, is to introduce a two-level naming system (like the genus - species system of Linnaeus). Within a game references would be to Rome. However, between scenarios we would have Rome - Early Republic, Rome - Marian, Rome - Late Republic, Rome - Caesar and so forth. The scenario editor would then allow the scenario designer to select the appropriate one.

Another simple refinement for the scenario editor, will be to specify an era, and restrict all options to that era. This is easy to do, provided each of the XML objects allow the specification of one or more eras.

Cheers

[Mark_Everson]

Thanks for the description of your issues and proposed solution, Gary. Your proposed change of direction sounds right to me given the circumstances. It's a shame about slowing the short-term progress, but I think the scenario editor will so accelerate mid-term progress that it might be the best thing to do even not considering the issues you've raised. Hopefully, Laurent won't have any big problems with the change in direction.

On the naming thing, that sounds fine to me.