Civilization III BIC file format (2nd thread)

[pdescobar]

Hmm. I'm guessing by the name that's a commercial library and so I'm probably not gonna be able to decompress such files on my own in perl am I?

BTW, for those unfamiliar with it, the way to properly invoke the command Gramphos posted from a DOS shell is:

savexpnd.exe "compressed_filename" "uncompressed_filename"

The quotes are pretty much mandatory and I sometimes have trouble with filenames longer than 8.3; now if only I could figure out the proper syntax to call this from inside my cygwin-bash shell...

[pdescobar]

Grr. Double-post. Sorry 'bout that.

[Gramphos]

the PKWARE library is commercial. However, you can get the decompression function I use from the CPT sourcecode if you want to complie something for linux.

[roudoudou77]

pkware decompression

thanks Gramphos for your imput

I have also found a source code somewhere in the Net
to decompress pkware compression format. i have checked some SAV files with SAVExpnd to verify it produce the same output correctly

for those interested, the link is :

http://perso.wanadoo.fr/the-home-of-roudoudou/civ3/blast.zip

[roudoudou77]

SAV format

Gramphos,


I have made a simple SAV of a game under CIV3 PTW v1.14f and I have obtained some strange data : there is 3 "GAME" section in the file and the last one of the "GAME" section does even not follow the rule :
- section name
- number of sub-section
- lenght of sub-section

then I see many TILE sections , not just 1 like in BIC files ..

that makes me crazy - and my program too - !!!

( I have uploaded the compressed SAV to :

http://perso.wanadoo.fr/the-home-of-roudoudou/civ3/julo6.SAV

)

could you help me and explain me the difference there is between the BIC format and the uncompressed SAV format ?

[Gramphos]

The SAV format is less space efficient than the BIC format.
For a section in the BIC format there are
4 char NAME
4 long Number
for each section
   4 long length of section n
   Section Data


For the saves the build up is
4 char NAME
4 long size of section header (many sections have variable length data blocks after the header)
The first section of the SAVE is the "BIC " section, which stores the full BIC/BIX file used with it
header data
data

The First section of the saves is "BIC " In the first public saveversion (14 (1.07f)) the "BIC " header were four bytes telling the length of the BIC data.
In later versions it will also contain info on the source of the data.

The two first GAME sections you referre too is (as I see it a bug, only one of them is needed (and it's only with later varsions it started to show up), but I've not taken myself time to see which one of the two the game actually used (they are identical))

The third on is the GAME section for the Ssave which contains overall data as for instance the number of cities, units and colonies (in header) and research progress for all civs and great wonder data (in the data area)

[roudoudou77]

SAV format

OK, now I understand that there is a BIC file "embedded" inside the SAV file , and that the 2 first GAME section belong to this "embedded" BIC .


Next questions :

1) the GAME section that follow the "embedded" BIC is quite strange : it's length is not correctly indicated ?

2) then there is a repetition of CNSL,LEAD,CULT,ESPN sections, it seems to be like garbage ??? I can't figure what those Fireaxis programmers tried to do with that :-(

3) near the end of the file I found many TILE section (it seems there is 1 section for each tile of the game), each TILE section is 180 bytes ; how should these data be interpreted ?

[pdescobar]

Re: pkware decompression

Originally posted by roudoudou77
I have also found a source code somewhere in the Net to decompress pkware compression format. i ave checked some SAV files with SAVExpnd to verify it produce the same output correctly

for those interested, the link is :
http://perso.wanadoo.fr/the-home-of-...civ3/blast.zip

Yes, I believe this is the code I actually wound up using myself. After looking at Gramphos' CPT source I located a webpage on the compression format and wound up compiling and using Mark Adler's "Blast" package to do the decompression which looks to be what roudoudou77 used for that executable above.

[Gramphos]

Re: SAV format

Originally posted by roudoudou77
1) the GAME section that follow the "embedded" BIC is quite strange : it's length is not correctly indicated ?

The length indication is the length of the GAME header, and the values in the header, and the rules (number of stuff) makes the game to understand how long the data is. I don't know exactly what controls the length, and am using a search to find the CNSL section (which is only a fixed length thing, that I've no idea of what it controls).

2) then there is a repetition of CNSL,LEAD,CULT,ESPN sections, it seems to be like garbage ??? I can't figure what those Fireaxis programmers tried to do with that :-(

LEAD, CULT, ESPN, ESPN is the build up for one player. there shall be 32 sets of them in the save. Again the LEAD starts with a header followed by variable sized data. (If you are to work with this one, it's the section that I know most of.)

3) near the end of the file I found many TILE section (it seems there is 1 section for each tile of the game), each TILE section is 180 bytes ; how should these data be interpreted ?

IIRC there are 2xTILE for each tile. one short and one long. The shor stores most of the map data, and are quite similar to the tile sections of the BICs, and the longer stores extra data as FOW and city working on a tile.

[BlueWlvrn]

BIX TILE format

It looks like in the BIX 11.18 version in the TILE section, there are now 4+29 bytes for each tile.

Can anyone tell me where the extra 6 bytes were added with respect to the BIC format posted here?

It looks like they maybe added in between and not appended to the end.

Thanks for the help.

-BlueWlvrn

[Gramphos]

The TILE format in BIX 11.18. I can tell what 2 of the bytes are. I can also tell a little more detailed on the other parts. As I've found out that some of it is incorrect.

4	char		"TILE"

4	long		number of tiles (= width/2 * height)



  For each tile:

    4	long		length of tile data

    1	byte		river connection info  (binary)

					00000000

			river in north	.......1

			river in west	......1.

			river in east	.....1..

			river in south	....1...

    1	byte		border

    4	long		resource

    1	byte		image

    1	byte		file:	0 = xtgc.pcx, 1 = xgpc.pcx, 2 = xdgc.pcx,

				3 = xdpc.pcx, 4 = xdgp.pcx, 5 = xggc.pcx,

				6 = wcso.pcx, 7 = wsss.pcx, 8 = wooo.pcx

    2	short		??? (0)

    1	byte		overlays (binary):

					00000000

			road		.......1

			railroad	......1.

			mine		.....1..

			irrigation	....1...

			fortress	...1....

			goody huts	..1.....

			pollution	.1......

			barbarian camp	1.......

    1	byte		terrain

    1	byte		bonuses (binary):

					00000000

			bonus grassland .......1

			player start    ....1...

			snow-capped mts ...1....

			pine forest     ..1.....

    1	byte		river crossing data (binarry):

					00000000

			crossing N	.......1

			crossing NE	......1.

			crossing E	.....1..

			crossing SE	....1...

			crossing S	...1....

			crossing SW	..1.....

			crossing W	.1......

			crossing NW	1.......



    2	short		??? (-1 = nothing, 75 = barbarian camp)

    4	long		??? (-1)

    2	short		continent

    1	byte		???

    2	short		VictoryLocation (VL if 0 otherwise -1)

    4	long		??? 

[BlueWlvrn]

Thanks Gramphose for the additional information.

Do you know how the image number in the TILE section maps to the terrain PCX files?

For example, image 50 from the xtgc file, should be a solid grassland, but I don't seem to be able to figure out the decode sequence to get it to that image square in the file.

Thanks again for any help.

-BlueWlvrn

[BlueWlvrn]

Ok, I think I figured out the pcx files.

These files are actually a half off of the map grid.

So the TILE section defines the terrain type, but the image applies to the shifted grid square that coveres the upper corner.

   /\
  /  \
  \/\/
  /\/\
  \  /
   \/

The top diamond is the IMAGE and the bottom diamond is the actual terrain type as seen by the map grid

[pdescobar]

Some more on the differences between BIC (4.01) and BIX (11.18) which I have found; confirmations and corrections appreciated:

'DIFF' section:

• There is a 4-byte long value added to the end of the section corresponding to the "Num. citizens quelled by military" setting on the Difficulty Levels tab in the editor.

'RULE' section:

• There is a 4-byte long value added to the end of the section corresponding to the "Flag Unit" setting on the General Settings tab in the editor. (index into unit array)

'PRTO' section:

• They went nuts on this one with an extra 40 bytes of data for each unit as well as changes to some of the existing data. This section is the same as that posted up to and including the 4-byte long index for the 3rd required resource.
• The next entry is still a 4-byte long section of flags for unit abilities. However, the "UNUSED (was mounted)" ability in the second byte is now the "Flag" ability (Princess in standard rules). Additionally, 3 more abilities have been added to the end of the list in byte 4. They are "Lethal Land Bombardment" (bit 3), "Lethal Sea Bombardment" (bit 4) and "King" (bit 5).
• Again, the next entry is the same with some additions. It is still a 4-byte long section of flags for AI strategies. However, 2 more strategies have been added to the end of the list in byte 3. These are "Flag" (bit 2) and "King" (bit 3).
• The availability 4-byte long entry remains essentially the same.
• The next 2 4-byte entries are no longer the conglomeration of orders and actions flags they were in v4.01; I believe they are now essentially garbage values as I couldn't come up with any discernible pattern to them.
• The next 3 4-byte sections (class, "other strategy," and HP bonus) are the same
• A new 4-byte section for "Standard Orders" comes next. These are flags and only the first byte is currently defined:

  
  
   00000000 (first byte)
  
   .......1 Skip Turn
  
   ......1. Wait
  
   .....1.. Fortify
  
   ....1... Disband
  
   ...1.... Goto
  
   ..1..... Explore
  
   .1...... Sentry
  
   1....... (unused)

• Then comes a new 4-byte section for "Special Actions." The first byte and a quarter of the second are all that are currently defined.:

  
  
   00000000 (first byte)
  
   .......1 Load
  
   ......1. Unload
  
   .....1.. Airlift
  
   ....1... Pillage
  
   ...1.... Bombard
  
   ..1..... Airdrop
  
   .1...... Build Army
  
   1....... Finish Improvements
  
  
  
   00000000 (second byte)
  
   .......1 Upgrade Unit
  
   ......1. Capture
  
            (rest unused)

• Next is a new 4-byte section for "Worker/Engineer Actions." Only the first 2 bytes are currently defined.

  
  
   00000000 (first byte)
  
   .......1 Build Colony
  
   ......1. Build City
  
   .....1.. Build Road
  
   ....1... Build Railroad
  
   ...1.... Build Fortress
  
   ..1..... Build Mine
  
   .1...... Irrigate
  
   1....... Clear Forest
  
  
  
   00000000 (second byte)
  
   .......1 Clear Jungle
  
   ......1. Plant Forest
  
   .....1.. Clear Pollution
  
   ....1... Automate
  
   ...1.... Join City
  
   ..1..... Build Airfield
  
   .1...... Build Radar Tower
  
   1....... Build Outpost

• Then comes a 4-byte long section for "Air Missions." Only part of the first byte is currently defined.

  
  
   00000000 (first byte)
  
   .......1 Bombing
  
   ......1. Recon
  
   .....1.. Interception
  
   ....1... Re-base
  
   ...1.... Precision Bombing
  
            (rest unused)

• Next is a strange section I don't fully understand. It seems to be a 4-byte long section full of flags which are duplicates of those covered previously. They cover the first two bytes plus one bit in a third..

  
  
   00000000 (first byte)
  
   .......1 Sentry (Standard Orders)
  
   ......1. Bombard (Special Actions)
  
   .....1.. Build Colony OR Build Road (both Worker Actions)
  
   ....1... Build Road (Worker Actions)
  
   ...1.... Build Railroad (Worker Actions)
  
   ..1..... Build Road (Worker Actions)
  
   .1...... Irrigate (Worker Actions)
  
   1....... Clear Forest (Worker Actions)
  
  (Note how Build Road sets 3 bits on its own and 
  
   that bit 2 is set if either Build Road or Build Colony is on.)
  
  
  
   00000000 (second byte)
  
   .......1 Clear Jungle (Worker Actions)
  
   ......1. Clear Pollution (Worker Actions)
  
   .....1.. Automate (Worker Actions)
  
   ....1... Automate (Worker Actions)
  
   ...1.... Bombing (Air Missions)
  
   ..1..... Precision Bombing (Air Missions)
  
   .1...... Automate (Worker Actions)
  
   1....... Goto (Standard Orders) OR Re-base (Air Missions
  
  (Note how Automate sets 3 bits on its own and 
  
   that bit 7 is set if either Goto or Re-base is on.)
  
  
  
   00000000 (third byte)
  
   .......1 Always ON (At least, I can't figure out how to turn it off)
  
            (rest unused)

• Next is a 4-byte long value which is 1 if "Bombard Effects" is checked in the "Unit Statistics" section and 0 if it is unchecked.
• The next 12 bytes are essentially flags for the "Ignore Movement Cost" entries. Each byte is 1 if the corresponding terrain type is selected and 0 if it is not.

  
  
  Byte Terrain
  
   00  Desert
  
   01  Plains
  
   02  Grassland
  
   03  Tundra
  
  
  
   04  Flood Plains
  
   05  Hills
  
   06  Mountains
  
   07  Forest
  
  
  
   08  Jungle
  
   09  Coast
  
   10  Sea
  
   11  Ocean

• The last entry is a 4-byte long value corresponding to the "Req. Support" checkbox on the "Unit Statistics" section. It is 1 if that box is checked and 0 if not.

[Gramphos]

It seems to be what I've found out. I currently don't have time to see if anything differs from whjaty I know, or to update the first post/start a new thread.