Saving and restoring games

[Gary Thomas]

I can't find any earlier references to this, though I remember a considerable discussion.

Having offered to code this, I propose to implement a system I have had in mind for some time, using the well known principle of coder's privilege.

The System:

1. Saving is automatic at the end of every turn.

2. A game may be saved part way through the current turn only - at the end of the turn the part way save is removed.

3. The data is saved in an incremental fashion, with a complete copy every so often (the interval to be derived by experiment), though truly unchanging details will be saved once only.

4. Each model will have its own save class, separate from other models.

5. The save file will be zipped.

6. If an earlier game (other than the last move or the current move) is reloaded, any subsequrent saves are lost. However, the player will be given an option to branch the saved game, thus producing another, and distinct, save file.

The advantages:

1. There is no silly micromanagement of save files as happens in Civ.

2. A complete playback of the game is possible.

3. Waste of space by duplicating static data, move after move, is much reduced. In particular, the basic map is saved once only.

4. It exploits the particular assets of Java.

5. I expect to code it so that loading a game is essentially instantaneous. I say this with some confidence, after recoding a program that manipulated an 8Mb Access database into Java. It turned out that the entire information in the database can be stored (zipped) in about 500K, and can be read in in about 1.5 seconds.

Cheers

[Mark_Everson]

Hey Gary:

The previous discussion you remember is probably in Planning for Demos 7 & 8.

I repeat my criticism of your approach. Essentially I believe that the save files will be prohibitively large. Here's what I said before:

I just think that So Much is going to change over even a few turns that you will end up saving virtually the entire game every time anyway. The unchanging parts are: The map outlines, and numbers of sites per map square will change infrequently. Just about everyting else will change over relatively small numbers of turns. Everything in the economy changes gradually, Same for much of the Social model. These are the largest memory-users in the game.

However, you can do the estimates of the save file size better than I. I've said my piece, and won't gripe any further! But I will say I told you so when the save files start approaching 100MB. (I'm not talking about for the current scenarios, but rather full-map games. There is no detailed calculation to back up my contentions. . .)

Cya,

Mark

[Gary Thomas]

If your answer is that the save files will be too large, then all we have to do is try it out. If the save files are too large, then you are right. On the other hand, separate save files (which appears to be the alternative that you prefer) will necessarily be even larger. So you reject one method in favour of a worse one?

Cheers

[Mark_Everson]

Hi Gary:

If you save all the game info then you only need a Single (or a few) large files rather than one that contains 100x as much info (100x is an order of magnitude estimate due to saving state changes every 10 turns for a 1000 turn game.) Seems straigtforward. I agree you lose the game-evolution info this way.

But anyway, as you say, the empirical approach will soon show what the relevant numbers are.

[Gary Thomas]

If you save all the game info then you only need a Single (or a few) large files

Every time you save, which, if Civ3 is any indication, is every turn.

Cheers

[LDiCesare]

5. I expect to code it so that loading a game is essentially instantaneous. I say this with some confidence, after recoding a program that manipulated an 8Mb Access database into Java. It turned out that the entire information in the database can be stored (zipped) in about 500K, and can be read in in about 1.5 seconds.

I know the stuff. I prototyped things last year where I could gain a factor of 100 in file size because of far too many information duplications in the file format.
I agree with Gary's approach. It should be tried. If size is an issue, we will see when this happens, and it should be relatively easy to lose the previous turns if that's what you want. Keep in mind that just zipping the file often gives very good results.

[Gary Thomas]

Thanks Laurent.

It is because of work experience that I am convinced that the results will be startlingly positive.

Cheers

[Gary Thomas]

I have in mind an extension of the system used to input scenario data. There will be simple interface (read, write, add, subtract) to process character data in xml format. Each save file will contain the whole initial scenario at the start. My first cut will just contain incremental data (apart from the initial scenario). It will start with a list of dates, allowing a player to load other than the most recent version, from a menu.

There will also be a restore option, which immediately loads the most recent game at its most recent point (an option missing from Civ, though present in other games).

Cheers

[Gary Thomas]

I have had a temporary rethink!

There is now some pressure to get something available soon, so we can produce version 8.

Also, I have realized that the code is closely related to the code for reading in a scenario - I can save a lot of effort (and reduce the amount of code) by using that code to restore a game.

As well, since my next project is a scenario editor, I can use the save game code there as save scenario code.

The consequence of all this is that, at least as a first cut, the save/restore system will essentially be a save/restore scenario system.

Once that is working I will write the scenario editor.

And once that is working I will work on the incremental save option, since, by that time, I expect that even Mark will realize that the files are getting large.

Cheers

[Mark_Everson]

Originally posted by Gary Thomas
As well, since my next project is a scenario editor, I can use the save game code there as save scenario code.

The consequence of all this is that, at least as a first cut, the save/restore system will essentially be a save/restore scenario system.

Once that is working I will write the scenario editor.

Sounds Outstanding!

And once that is working I will work on the incremental save option, since, by that time, I expect that even Mark will realize that the files are getting large.

Personally, I think it'll take Mark longer than that to come around, but you may be right. . .

[Gary Thomas]

OK, apparently everything is on track except my ability to convince Mark that I know what I am doing.

Cheers

[LDiCesare]

That's not really a problem, since you'll be doing it anyway.

[Gary Thomas]

I have encountered a roadblock in my save/restore coding.

There is a structural problem which I have discussed previously (about two years ago).

The basic game was designed to read in a whole lot of files - civilizations, military, social, economic, religion, and so forth, overlaid by specific scenarios.

Frankly this is a big unmanageable mess, and clashes horribly with the save/restore mechanism.

It is also something that I have been against from the beginning.

My conception and preference is that each scenario definition file should be completely self-contained, so only a single file is read in for each play of the game.

The objection to this was that it meant that scenario designers would have to repeat large amounts of data, from one scenario to another. This applies particularly to the military data.

Having given the matter considerable thought recently, I realized that we have been on a slightly wrong track.

Yes, there should be comprehensive files on the topics I listed at the start of this post. On the other hand, a scenario should be a single file and the program should not need to read in a large number of files containing irrelevant data.

The solution, now that I have been studying the data saving system in some detail, is a properly designed scenario editor which will take the necessary parts of the standard reference files and produce a single scenario file.

As well as being tidier than the present system, it will add one very large benefit. It will mean that the global data will be included in the scenario file. So, in order to test the effect of these global parameters one will be able to produce a test scenario in which only a global parameter is changed. This could them be downloaded and tested by people other than the coders.

The effect of this is that I am more or less forced to exchange my top priorities and work on the scenario editor now.

This in turn means that it will not be done this week, as I had hoped.

Cheers

[Gary Thomas]

I put this in a separate post to avoid diluting the impact of the previous post. This relates to names of things in general (though not to language, Jack).

One of the problems we have had in scenario design is that the same civilization has different incarnations in different scenarios. For example, the Roman civilization of Delenda is quite different, and many centuries removed, from the Roman civilization in Attila. There is no provision in the data file system to differentiate these, apart from giving them different names.

My proposal, to get around this difficulty, is to introduce a two-level naming system (like the genus - species system of Linnaeus). Within a game references would be to Rome. However, between scenarios we would have Rome - Early Republic, Rome - Marian, Rome - Late Republic, Rome - Caesar and so forth. The scenario editor would then allow the scenario designer to select the appropriate one.

Another simple refinement for the scenario editor, will be to specify an era, and restrict all options to that era. This is easy to do, provided each of the XML objects allow the specification of one or more eras.

Cheers

[Mark_Everson]

Thanks for the description of your issues and proposed solution, Gary. Your proposed change of direction sounds right to me given the circumstances. It's a shame about slowing the short-term progress, but I think the scenario editor will so accelerate mid-term progress that it might be the best thing to do even not considering the issues you've raised. Hopefully, Laurent won't have any big problems with the change in direction.

On the naming thing, that sounds fine to me.

[LDiCesare]

I don't really understand why several files is a bad thing. Given the current code, where each file is read in a separate thread, I understand it can cause problems, but it's just an ordering problem. I'd much rather have a #include-like directive than have a single file, because I like to edit things by hand. Now having bigger files won't prevent me from doing this, and I can always start with the scenario editor if needs be. I also don't get why having several files from the start hurts. You can always merge them together into a single 'scenario' file? But hey, now that I've ranted a bit, I must confess that Gary has the very bad habit of being right when it comes to coding, so I think that's probably the thing to do.
Now the next question is: Do we wait for this to be completed to get a demo 8 out or not?

[Gary Thomas]

The problem is that we are making the game play the part of a scenario editor. The present program reads in a great deal of scattered information and discards most of it. Under that system, when I write the save file it will be quite different from the scenario file. If someone starts a game and immediately saves it, for a great many reasons, mostly concerned with debugging and code safety, I would like the saved file to be, as nearly as possible, exactly like the scenario file.

What I am really aiming for is a system that is simple.

The argument that the game should be structured to make it easy to hand edit scenario files is not a good one. You are, I think, viewing scenario files as a kind of programming code. For the real purposes of a scenario file, an include statement is very wrong, and leads, inevitably, to dependencies. I believe that I have expressed my opinion of the deeply evil nature of dependencies in the past, though, it seems, not sufficiently eloquently.

I am not interested in making it easier for people who want to hand edit things, at a cost of making it more difficult for the people who play the game.

When I devise a brilliant new scenario, I want to be able to send someone a scenario file and have it run, without having to also send a military file, a social file, a religion file, a technology file, a riots file, a weather file, and whatever else. Once the scenario is set, it does not need any of these things. They should be incorporated when the scenario itself is built. For practical reasons, and only in the short run, I will accept a separate map file. However, because the map file and the scenario file are indissolubly connected, I would prefer that they be incorporated in a single file.

The fact that I went a lot of trouble to make the data files human readable and modifiable, in the form of XML files, should not be considered as meaning that that is the way scenarios should be designed and built. It is, essentially, a debugging tool only.

Because, in recent years, a scenario editor, in most strategy games, has become almost as important as the game itself, I think that such an editor should be provided (and I have started doing it). And it should be to the same standard as the game itself. People who want to hand edit the result can, of course do so, but using cut and paste. It has fewer dependencies.

And I think, with some regret, that the next version should be issued without waiting for the save/restore.

Cheers

[LDiCesare]

Don't worry, Gary, I know the evil of dependencies. But I also know the evil of copy paste (when you change a bit of code and find that is has been copied in ten files by different people who could have reused your service...).
How will you go for the scenario editor? How will you say when you create a new scenario that you reuse the tech tree, etc.? Will you keep resource files to build the scenario file from? I mean resource files are just used to speed up the scneario editor to lead to a single scenario file, but never used by the game itself.

[Gary Thomas]

In one sense the scenario editor will do what the game itself does. The fileinput class reads a considerable number of files and, essentially, combines them into one file. What I propose to do (amongst other things!) is to move that functionality to the scenario editor.

I would hope that it will be easier to make minor changes in the scenario editor, rather than by cutting and pasting.

As to general functionality, the scenario editor will work on two levels. It will enable editing of the standard data - changing global values, or editing what I call the reference files, that is, the files which contain lists of unit archetypes, civilizations and so forth.

Those reference files will be read in every time the scenario editor is run.

The person using the editor will have the option when designing a scenario of selecting any of these components from menus, and modifiying them if desired, or of creating new objects specific to that scenario.

Because the save/restore and scenario editor will be closely related, it will also be possible to edit a saved game, as though it were a scenario. This adds the feature that I have always wanted, of changing the player civilization during the game.

Cheers

[Gary Thomas]

There are a couple of matters I overlooked in my earlier post.

Because much of my interest is in the simulation aspect of the game, rather than the competitive aspect, I like the idea of running a game and switching among the civilizations to get a better grasp of their problems and challenges at various points in the game. That possibility really interests me.

The other thing is that the non-map parts of the scenario editor are very much easier to code than the map part. I will address them first. The first version of the editor will not have map editing included.

Cheers