Privateer 2 editing and extracting

[HCl]

Hey that's great! Having those palette files as GIFs turns out to be pretty handy

The most interesting thing to me is that those Palette GIFs are 256x64. An 8-bit palette is 256 colors, so this suggests that there are 64 sub-palettes, one being each row of the GIF, used for gamma correction by the game.

Seems like a reasonable assumption. Lets go with that!

While I don't have my P2 files with me at the moment, I took the Xavier pic from this thread along with game.gif and made a quick-and-dirty Matlab script for testing this. Both files are 8-bit, so i can easily work with those (jpeg encoding, on the other hand, discards all indexing information...).

The results are on the attached pic!

- Figure 1 is game.gif itself. Nothing new here, just making sure i'm loading the data correctly
- Figure 2 is the Xavier pic using the GIF palette directly.
- Figures 3 and 4 are the Xavier pic using selected subsections of the palette. One of them is therefore lighter and the other darker.

So this confirms that the same WC3/StarLancer RLE algorithm works fine for P2 as well Next step then is to dig further into the header. When we understand it better, we'll be able to parse the data more robustly.

 

[Bandit LOAF]

As a certified non-genius, I am completely lost. Have you guys had any luck extracting text from the game? (BBS entries, mission briefings, e-mails, etc.)

(I Facebook'd this thread to Paul Hughes, thinking he'd get a kick out of seeing the kind of work y'all are doing. Privateer 2 really deserves this kind of attention, at long last!)

 

[HCl]

I had some success in the past extracting text from the game. From what I recall, it is compressed with yet another algorithm... However, that's pretty much all I remember at the moment, the last time I've done some serious digging into the P2 EXE was probably around 1998-99.

However, since this is a problem I encountered in the past, chances are good that I still have something on my backup CDs or notes. I'll look into it and let you know if I find anything!

 

[Bandit LOAF]

Paul Hughes says:

It's been a while but it's coming back to me. IIRC all the text was huffman compressed and then bit indexed. The sprites are in VFX shape format (run length encoded). I'd forgotten about the Zlib stuff - I could've sworn we removed the deflate stuff as it slowed the load down - for the most part it used a much simpler decompressor for speed.

 

[AD]

So aparently P2 likes Zzzzzzs. I took a look at the end of Dark.exe and it seems to be either a list of included C files or the actual files themselves. Some examples are

zbsetup.c 1.18 1995/08/31
zbrendr.c 1.36 1995/08/31
surface.c 1.11 1995/03/29
angles.c 1.12 1995/02/22
prepmesh.c 1.35 1995/08/31
pixelmap.c 1.17 1995/08/31
def_mdl.c 1.13 1995/03/29
def_mat.c 1.4 1995/02/22
zbmesh.c 1.67 1995/08/31
zbmatl.c 1.4 1995/03/01
prepmatl.c 1.6 1995/02/22
zbmeshp.c 1.20 1995/08/31
zbmeshe.c 1.19 1995/08/31
envmap.c 1.11 1995/03/01
zbclip.c 1.4 1995/08/31

Something to do with Z buffers? Zlib? A number of those obviously have to do with the renderer, and at least two of the ones I listed probably are to do with the deflate functions used by the game.

Also there's some copyright info regarding P2s inflate function atributed to Mark Adler which kind of is redundant to talk about since we already know that Zlib is used in the files...

 

[HCl]

Great thread we have here, lets try to keep it running!

Apparently, the game might always use the same palette. Of the various GIFs AD posted that I checked, GAME.GIF and the various CV-xx-yy GIFs seem to have exactly the same 256-color palette, the only thing that varies is how those 256 colors are used. I speculate that GAME.GIF defines a lookup table for gamma levels, while the CV GIFs define conversion tables in order to stylize the textures according to a given criteria (eg. pirate clans... Chirichan might use the Yellow conversion table, while the Kindred may use the Red one, and so on).

I made a small script for playing with this color conversion mechanism a bit, using textures from spacetex.iff (the textures are already unpacked once you run inflate on them, no need to run them through the RLE unpacker). It seems to work out Pic attached on this thread, in case you're curious.

Regarding text compression: Thanks LOAF! That is actually very useful information and will allow me to try a few things out before diving into the EXE.

AD, I suspect those C file references might be related to the BRender engine. It was a really nice engine, I wish I could take a peek at it some day.

 

[HCl]

This is probably of limited interest at this point, but here's a quick update:

Although i haven't located anything interesting on my notes, I managed to remember a few things about the geometry format used by P2. Made a quick script to test a few things and i'm starting to see some progress. (pic attached, although Matlab is not the best software package for 3D rendering, so things look a bit off)

As expected, VERS is a vertex list, FACS is a face list and FMTS is a rather wasteful material list (one 32-char string per triangle, apparently!). All three chunks are deflated. There are still a few fields I am not taking into account in VERS and FACS, so there are still a few models that end up looking rather strange... but I hope to work these issues out soon.

Then there is the chunks AD mentioned, which I haven't really looked at very closely, but I assume define what CLUTs to apply to the various textures...

Anyway, that's what I have for now. If anyone has more info on any aspect of P2 file formats, i suggest posting it here, let's keep this thread going! On my part, i'll try to start typing some specs as well, about the various P2 things i've been playing with.

 

[Dundradal]

This is awesome stuff! I just have absolutely no idea what it means but it's very cool to see how a WC game comes together.

 

[AD]

This is probably of limited interest at this point, but here's a quick update:

Although i haven't located anything interesting on my notes, I managed to remember a few things about the geometry format used by P2. Made a quick script to test a few things and i'm starting to see some progress. (pic attached, although Matlab is not the best software package for 3D rendering, so things look a bit off)

As expected, VERS is a vertex list, FACS is a face list and FMTS is a rather wasteful material list (one 32-char string per triangle, apparently!). All three chunks are deflated. There are still a few fields I am not taking into account in VERS and FACS, so there are still a few models that end up looking rather strange... but I hope to work these issues out soon.

Then there is the chunks AD mentioned, which I haven't really looked at very closely, but I assume define what CLUTs to apply to the various textures...

Anyway, that's what I have for now. If anyone has more info on any aspect of P2 file formats, i suggest posting it here, let's keep this thread going! On my part, i'll try to start typing some specs as well, about the various P2 things i've been playing with.

Awesome! I pretty much figures that VERS meant vertex or vertices. I just didn't have a clue as to how to read the deflated chunk or translate it into useful info. Ultimately it would be great to be able to import new ships into P2 or replace existing ones... A standoff style P2 mod would be awesome.

I did have a theory that I haven't tested about the textures. You mentioned that they are an odd size. It may be that the extra info in the texture references in the model file define lighting conditions. Since it's a software renderer, they may have cheated the lighting effects by adjusting which part of the palette table to use (like the gamma function you referenced). So instead of complicated shader routines it essentially adjusts a slider up and down the color table. It's just a thought though and I more than likely have no clue what I"m talking about.

 

[HCl]

Paging Lt. Sosa...

A very quick story... well, mostly as an excuse for bringing this thread back from the dead

Today I travelled a bit in order to enjoy easter holiday. As a matter of principle, i did not want to take my PC with me (holidays and all)... but I couldn't resist taking a couple printouts from a few Privateer 2 files, just in case. I enjoy both WC and puzzles, so I figured I could look into it during the trip and see if something occurred to me about that Huffman-based text compression algorithm. I made some progress earlier on how the Huffman tree was encoded, so maybe I could review what I had done, make sure I didn't do any mistake and, if I did, try to figure out another approach.

Things seemed to make sense, which is always good. Since i had some time, I tried to hand decode (pen and paper, very low tech) a few bytes from the printouts I had. Luckily I had the header of the TXT2 chunk and what seemed to be the start of a valid data stream with me, so I went ahead with the attempt.

Long story shot: things went well! I cannot tell you how satisfying it was to eventually be able to read "Visit Angus Santana at Interplanetary Aid" I'm including a photo, in case everyone is interested.

Anyway, I'll make a decent unpacker during the weekend!

 

[PopsiclePete]

Oh my God ...


and by God I mean HCl...


That's amazing human computing !

 

[Iceblade]

Oh my God ...


and by God I mean HCl...


That's amazing human computing !

This stuff is a light snack for someone who eats assembler for lunch.

hmm....just had a thought, maybe HCl isn't human, but an android from the future who has come to give us Wing Commander fans a helping hand.

 

[Wedge009]

I know a little of Huffman encoding, but the stuff that HCl goes through is mostly beyond me (at least, without taking the effort to go through a few textbooks or three).

Thanks for your hard work, Mario!

 

[Madman]

And that ladies and gents, is why everyone should know their ascii tables!
Seriously though, incredible stuff!

 

[AD]

Since i had some time, I tried to hand decode (pen and paper, very low tech) a few bytes from the printouts I had. Luckily I had the header of the TXT2 chunk and what seemed to be the start of a valid data stream with me, so I went ahead with the attempt.

Long story short: things went well! I cannot tell you how satisfying it was to eventually be able to read "Visit Angus Santana at Interplanetary Aid" I'm including a photo, in case everyone is interested.

That's awesome! I really wish I knew more about programming. It will be great to be able to go through the P2 text. Amazing work here, HCl.

 

[HCl]

Hehe thanks for the positive feedback guys, glad to know you find this low-level stuff interesting!

I was playing a bit with Matlab today, implementing a couple of details I figured out on paper, and ran the script on a couple of chunks from gameflow.iff and booth.iff. Seems to work fine, from the booth chunk i seem to get the full set of news, which makes for some interesting reading given how long it's been since I played P2.

Gotta love those brutal sports on the Tri-System:

D.A. issue new guidelines <255>
Deathball correspondent Viman Vigga reports <255>
The Deathball Association this afternoon revealed plans to add four new rules to the rulebook, a decision which will make that slim journal three times larger. The resolution was deemed necessary in light of a few recent matches - notably the Treyk-Barzon Incident, and the Fewley-Trenzik Massacre. From now on, use of nuclear, biological or chemical weapons is strictly banned. Any team using them immediately forfeit the match, be banned for a year, and have its manager executed. Also proscribed is the use of military vehicles and hardware - no more napalm strikes, unfortunately. Oh well, it's still a great game. <255>

Anyway, i'm attaching a text file with these two unpacked chunks (note the special markers surrounded by <> characters). Hopefully this will be of some use to the WCPedia Project!

I'll be converting the code to C# and releasing a command-line tool (and its source) soon.

 

[Lin Kuei Ominae]

Oh wow. You sir, are a genius.

With your text unpacker c# code, i can also put another part to my P2-FileEditor. I hope i can soon find enough time to finish it for a first beta release.

 

[HCl]

And here it is, a simple command-line uncompressor for P2 text, with C# source code. As often happens with my tools, this tool is meant for the more technically inclined people, as you have to supply the offset of the Huffman-compressed chunk (which may not be too obvious to everyone).

Of course, feel free to improve on this in any way you like!

Lin Kuei, feel free to copy / paste this code into your editor. Start by looking at the main method in Main.cs and see how the code is used. Any question, feel free to ask! Have fun

 

[HCl]

A couple more details regarding text compression:

It seems that not only the TXT2 chunks contain compressed text. A few DATx chunks in the Booth IFF seem to contain a lot of data as well:

- DAT1 contains Bios.
- DAT2 contains Companies.
- DAT3 contains the ship registry.
- and DAT4 contains planetary information.

I extracted this info and put it on a file for easy access (attached on this post). I'm not sure how useful it will be though, since almost all this information is already on this article on the CIC, good job putting it together by the way! Only one ship seems to be missing from the CIC database:

Z376_267G <255>
Universal Passenger Liners <255>
Previa Class <255>
Passenger Liner, 3 years service. Licensed to carry 150 passengers from Bex to Hades only. <255>

- The "TEXT" chunk on Booth.iff seems to be compressed with... yet another algorithm. At a glance it suggests dictionary-based compression (eg. Lempel Ziv). Hopefully i'll be able to sink my teeth into this soon, so i'll start by testing this hypothesis first.

 

[Lin Kuei Ominae]

I really hope you get the booth uncompressed as well, as i have the guess that this one contains all the ships info like shields, hull, speed, steering, bounty fee etc. And being able to edit these was always the holy grail for my p2-editor.

Oh and thank you very much for the source code and all the awesome work you put in my beloved P2D.