Privateer 2 editing and extracting

[Bandit LOAF]

Incredible! Thank you so much. It will certainly be a big help, since the version at WCNews was typed up by hand from screenshots by me and is full of typos.

Please let me know if you come across...

- The mission text (e-mails/briefings/etc.)
- The variables that can slot into the booth entries. I really want to know what all the possible names for reporters are.

Amazing, amazing stuff... but I'm sure I've told you that before.

 

[Lin Kuei Ominae]

I really want to know what all the possible names for reporters are.

With the p2-editor you can find them out too. (or look in the boothinfo.ini of the editor, where i collected already many different strings/names)

[NEWSREPORTER]
00=Anthony Stockton
01=Amanda Webster
02=Mike Cottam
03=Phil Meller
04=Nick Elms
05=Chris Battson
06=John Miles
07=Mark Goldsworthy
08=Paul Chapman
09=Richard Curtis
10=Paul Hughes
11=Tony Knox
12=Jerwyn Gooson
13=Julia Derrek
14=Nick Goldsworthy
15=Jennie Sanders
16=Bev Hayes
17=Tracey Elms
18=Elspeth Caldwell
19=Alex Camilleri
20=Robin Green
21=Matthew Miles-Griffiths
22=Brian Hussey
23=Ann Williams
24=Erin Roberts
25=Joss Ellis
26=Andrew Bramley Hill
27=Brian Turnell
28=Adam Medhurst
29=Andrea Griffiths
30=NAME:

entry 30 and up seem to be not used anymore as reporternames ingame as these represent strings for other things.

 

[Bandit LOAF]

AWESOME, thanks!

It's mostly Privateer 2 developers!

 

[HCl]

Please let me know if you come across...

- The mission text (e-mails/briefings/etc.)

There you go! Those were actually scattered across several files. I'm not completely sure why at this point, but i'd guess it could be due to the game's mission selection mechanism (in order to give you a set of missions at various stages of the plot progression).

Anyway, that should be all mission-related text. Hope it is useful!

 

[HCl]

Managed to make a bit more progress on text extraction. Here's a few more bits (attached):

- Ran into the Wingmen descriptions chunk. It was huffman compressed, but I don't think I posted it before, so here it is, in case it is useful.

- Also, I finally managed to crack that dictionary-based compression algorithm I mentioned a while back! This gives us access to the Equipment descriptions, as well as a lot of compressed SHP chunks. I'm guessing this was adopted as a lighter alternative to deflate at some point during development.

A couple references from this chunk might correspond to some ships that appear on various missions (eg. Police Ship Sweeney, S.S. Amelia, ...)

- Finally, a couple of deflate-compressed chunks with miscellaneous data. These are not very interesting by themselves, but could be of some use later on (cross referencing some entries with other text files may be useful at some point, figuring indexes, and so on).

Anyway fun stuff!

 

[Bandit LOAF]

A couple references from this chunk might correspond to some ships that appear on various missions (eg. Police Ship Sweeney, S.S. Amelia, ...)

I think you may have discovered something extra-special here!

What I'm betting it is is some earlier index for the database. Check out the list (below)--you've got planets, businesses, characters and what seem to be names of ships... but they don't match up to those lists in the game itself. Many that are in the game aren't listed while others we've never heard of are (Mehran Anderson? Simon Blade?)... and the ships all have names instead of registry numbers.

The ships don't match missions... it seems to be a collection of fighter types from the game and entirely new names... plus a few of the flavor text names that wound up as classes in the finished database (Salvia, Yackard).

Anhur
Bex
Crius
Hades
Hephaestus
Hermes
Janus IV
Karatikus
Blessed Aquawine Bex
Blessed Brew Bar
CIS
Crius Hospital
Crius Net
Crius Off Planet Control
Galactic Gourmet Hotel
Holy Museum Of Hom
Interplanetary Aid
Kappa Labs
Kindred
New Crius Examiner
Rampant Roboot Bar
Shadow Play Bar
Shernikov Medical Equipment
Sinners Inn
Surgeons Blunder Bar
Taffin Reclamation (Crius)
Ames, Senator Veklyn
Ames, Temessa Shuna
Anderson, Mehran
Arris, Lev
Avignoni, Dimitri
Banks, Melissa Kathryn
Barlow, Bernice
Blade, Simon
Bray, Reginald Arthur
Bray, Ronald Henry
Carmichael, Hugo
Douglas, Pemtur
Farquharson, Errendi ('Fark')
Fossa, Alberto Rinaldo
Frevel, Dr. Janna
Gabriel, Aunt Maria Filippa
Gould, Dr. Graham
Hassan, Commander David
Kane, Joe
Karsht, Harold
Kashumai, 'Uncle' Veldon
'Kronos'
Loomis, Dr. Yvan
Maddox, William
'Malakai'
McCloud, Ralph
Mitorr, Jan
Nabokov, Sheila
Phillips, Louisa Elizabeth
Porter, Pietr
Proctor, Chief Ronald
Ramirez, Nelson James
Regis, Larn
'Rhineheart'
Ricaud, Vell
Ricaud, Sar
Ricaud, Vell II
Santana, Angus
Shondi, Xavier
Skintight, Zakariah
Taffin, Hal
Vonx, Lord Michael
Adrilak
Airhead
Alpha
Aswan
Aurora
Baldesh
Bigfoot
Bikila
Blackdog
Blackfire
Blask
Borrealis
Brave
Canberra
Chaos
Cloc
Cygnus
Danrik
Defector
Dionysus
Dogstar
Doppelganger
Drakkar
Dreamstate
Duress
Eclipse
Eradicator
Excalibur
Eye Of The Storm
Faldari
Famagusta
Flipside
Freij
Funkbuster
Garios
Gazar
Geek
Geo
Glutino
Gunslinger
Headfirst
Helter Skelter
Heretic
Herima
Hyppolita
Icarus
Ilia
Iron horse
Isaoko
Jendevi
Jinter
Judar
Kabaka
Kalrechi
Kalthike
Karnene

Kassar
Ketra
Kitrum
Lady Of The Lake
Lazarus
Liberty
Lima
Lionheart
Lucitania
Masrada
Miguez
Ogan
Olura
Olympus
Omega
Papagos
Patriot
Police Ship Sweeney
Presto
Red Dragon
S.S. Amelia
S.S. Ballistic
S.S. Caroline
S.S. Manchester
S.S. Manhattan
Salvia
Scarecrow
Shadowcaster
Shaman
Skeces
Sledgehammer
Sundog
Tempest
Titania
Trigo
Trojan Horse
Vacuum Oasis
Varjner
Veldor
Velecia
Vetrece
Victor
Vindicator
Violator
White Lightning
Yabar
Yackard
Zeus
Zion

 

[HCl]

Good catch, you may actually be right! I initially suspected there could be a few markers at the Mission text used for inserting a random ship name, so i dismissed the names I did not recognize due to simply not remembering them (not having played the game for some time).

Turns out that ship names on the missions are actually hardcoded and, as you said, a few of the names do not seem to appear anywhere else, which means that the strings on this list may be from a previous version indeed. Exciting stuff!

I wonder if other such artifacts are in the game files somewhere, waiting to be found... I'll keep looking!

 

[HCl]

It turns out that the actual ship names and IDs in each mission are deflated in a DATA chunk in the MISSIONx.IFF files. I'm attaching these, in case it is useful. Indeed, you will find a few familiar names there:

ROMPER ONE
RAMIUS SHAFT
AROON STREBOR
BRAD STROKER
SIMON CHARGER
(...)
CIS KAITLIN
CIS PHOENIX
CIS ICE BREAKER
(...)
PAPOGOD FORCE 4
PAPOGOD FORCE 76
PAPOGOD UNIT 69
PAPOGOD UNIT 54
PAPOGOD FORCE 75
PAPOGOD UNIT 6A
PAPOGOD UNIT 3B
PAPOGOD UNIT 2

Can't say i recall these Papogod references, but i'm sure they appear in some of the Papogod-related missions. Anyway, hopefully these will be useful references for the WCPedia!

Haven't run into any more text in the meantime. I made a script to scan the files for more chunks compressed with the newly-discovered dictionary-based compression algorithm, but there only seems to be one text chunk compressed with that algorithm. Everything else seems to be sprite data, 1.40 mostly.

I think that should be most, if not all, of the text. Let me know if I missed something though!

 

[Dundradal]

HCl you continue to absolutely astound me good buddy!

All this stuff I know is going to be put to good use in Lin's editor and in WCPedia!

 

[arfgh]

Hey guys.

I am a Privateer 2: The Darkening enthusiast since its releasing in 1996.

I played my version tons of times, but i can say that always was a big problem for me, because the game crash and crash and always crash when certain situation happends. This situation is when some enemy speak when we are shooting him, just when he die.

With the time i have located the problem that is 100% sure into the file 'SPEECH.BIG'. My file contains audio spanish cause my version is this. If i remove this file from the installation, or just change it for the english or german version, the game never crash.

In my opinion, the problem happends because the game dont stop to play the sound at its end, because sometimes i can hear a piece of a next audio communication, or the computer voice just repeat itself. The crash never happend in this moment, but the end of the mission will always finish with a crash. This happend with all the versions of the game, including the Windows deluxe version. In the past, when played the dos version, the whole pc crash, but now with the windows deluxe version, the game just stop and we are rejected to desktop with the dark.exe process in memory.

I want to request to all, and specially to Mario Brito, the examination of my 'SPEECH.BIG' to know what is the problem. My objective is to reconstruct this file, with audio files that dont cause the problem. I know that Mario made a tool to extract the files into the .big, but i dont know how to import again files to a new 'SPEECH.BIG'.

Download the famous 'SPEECH.BIG' --> here! it comes directly from my original CDs


Plesase help me to fix this problem for one of my favourite games ever.

 

[thp]

Really useful thread, thanks I think it was never mentioned what the algorithm for decompressing the non-Huffman, non-Deflate compressed data is. This is the data that starts with the bytes 0x10, 0xfb, and it looks like this algorithm is documented here (have yet to test it myself on the Privateer 2 data yet, but the data looks very much like it fits that description): http://simswiki.info/wiki.php?title=Sims_3:DBPF/Compression

 

[thp]

Yep, this decompression algorithm works indeed. Not sure if it was mentioned already in this thread, but for image data ("1.40" signature), the shp2bmp code works, but if the first sub-image size is 3 * 256 (768 bytes), then this has to be interpreted as a 24-bit RGB palette for the other images (there might be some info in the header to determine whether it's a palette, but the 768 heuristic worked fine so far). Not all images contain a palette, for those that don't (e.g. pilot comm images), indexing from GAME.GIF should work.

 

[thp]

Some progress on extracting models; the SPACEGEO.IFF file refers to files in SPACETEX.IFF (obviously). The VERS chunk contains 10 floats per vertex, the first 3 are the (x, y, z) coordinates, the next 2 are the (u, v) coordinate. The last three are likely normals (the vector length is 1.0), but haven't gotten them to work correctly, so the normals are not yet used. The FACS chunk contains 18 uint16_t values per face, the first 3 are indices into the vertex table. Also the "_d1", "_d2", "_d3" suffixes specify the LOD levels for the models (with _d1 being the most detailed, and _d3 being the least detailed). Here's some random models imported into Blender.

 

[AD]

Hi thp! It's pretty cool to see you taking a crack at this and resurrecting this thread.

 

[thp]

Cleaned up the 3D model extraction code a bit, and fixed a UV coordinate calculation bug that caused artifacts with the planet textures.

Here's a quick'n'dirty WebGL-based Model Viewer for the extracted models, which were converted to Wavefront OBJ, and colormaps to PNG:
http://originalthp.solsector.net/p2models/

 

[AD]

Cleaned up the 3D model extraction code a bit, and fixed a UV coordinate calculation bug that caused artifacts with the planet textures.

Here's a quick'n'dirty WebGL-based Model Viewer for the extracted models, which were converted to Wavefront OBJ, and colormaps to PNG:
http://originalthp.solsector.net/p2models/

Awesome work! HCl and I (mostly HCl really) did some work making a viewer for the CIC website for the WC3/4/armada ships that uses webGL. We had talked about expanding it to include the formats used in some of the other games at some point. I have no idea if it's actually linked on the CIC front page but the viewer can be found here https://www.wcnews.com/shipviewer/ and the update here https://www.wcnews.com/news/2014/08/17/hcl-s-online-ship-viewer

I know we had been looking into the P2 format a bit at one point but I don't really remember how far HCl got with extracting the actual ship models. However, it might be useful for comparison purposes, but Paul Hughes sent in a batch of source 3DS models for P2 at one point https://www.wcnews.com/news/2009/08/11/privateer-2-in-3d

EDIT: Quick comment on your model viewer... great work. Pretty much every model has three LOD versions, which is neat. Also, I know that it's a "quick and dirty" version, it does look like some of the palettes aren't being displayed correctly on some objects and ships. Possibly other transparency information isn't working correctly either... I could be wrong, but thinks like the sheild hit tiles look funny. With the WC3 and 4 stuff, there were also issues with objects that had pretty much zero thickness and showing correctly since sometimes they needed to be told to display on both sides since they would be invisible if you were looking at the back of the object.

 

[thp]

I know we had been looking into the P2 format a bit at one point but I don't really remember how far HCl got with extracting the actual ship models. However, it might be useful for comparison purposes, but Paul Hughes sent in a batch of source 3DS models for P2 at one point https://www.wcnews.com/news/2009/08/11/privateer-2-in-3d

https://www.wcnews.com/news/2009/08/11/privateer-2-in-3dYou can extract the files yourself with the tool below (point it to SPACETEX.IFF and SPACEGEO.IFF). I tried the .3DS files in Blender, but didn't get textured models (but that might be a limitation of the Blender 3DS importer).

Here's a link to the tool I've been working on in the last few days based on this thread, it can extract quite a bit of Privateer 2 data, but there are still some known bugs (e.g. it doesn't apply the palette contained in the BASE image to the animation images in SETS.IFF, so the palette is off there -- but the animation images work great with the BASE palette; the same is true for some images in BOOTH.IFF -- the palette is available inside the file, just not applied everywhere):

http://originalthp.solsector.net/priv2dump/

I have tested it with the German and English version of Privateer 2; if anybody could test it with other language versions, that'd be great.

Random observation: If you dump BOOTH.IFF, you'll notice that the pre-rendered 3D rotation images of ships and equipment only contain half a rotation -- I first wondered why that is, but then noticed that all ships and equipments are symmetrical, so a full rotation can be done by doing the half rotation, and then doing the other half of the rotation by mirroring the image horizontally and playing it back in reverse.

Also, I know that it's a "quick and dirty" version, it does look like some of the palettes aren't being displayed correctly on some objects and ships. Possibly other transparency information isn't working correctly either... I could be wrong, but thinks like the sheild hit tiles look funny. With the WC3 and 4 stuff, there were also issues with objects that had pretty much zero thickness and showing correctly since sometimes they needed to be told to display on both sides since they would be invisible if you were looking at the back of the object.

There are texture maps that convert certain colors in the palette to some other colors, these probably need to be applied properly depending on ship type (maybe some ship models are also re-used and just applied with different colors). Right now, all the colormaps from SPACETEX.IFF are written using the default (GAME.GIF-based) palette - this might explain some wrong textures you see. Transparency is easy to fix, right now the tool doesn't apply any transparency, but it's possible to make palette index zero transparent (which seems to be the way the game handles transparency also in other images), which fixes the transparency (but some images such as BASE images don't have transparency, so I didn't want to apply that unconditionally).

The "zero thickness" issue (e.g. seen with the missiles) is just backface culling being turned on in Three.js by default, if one would turn that off, it would look correct (possibly somewhere in the face definition it would tell us if it's a two-sided triangle, or we just turn it off unconditionally).

 

[AD]

This is an awesome tool, thanks! I had no trouble with the unpacker at all. I haven't spent a ton of time playing with it yet but I did find one fun thing... looks like the menu screen for some kind of debug/programming tool

 

[thp]

I did find one fun thing... looks like the menu screen for some kind of debug/programming tool

Interesting, I didn't find this in my version of Privateer 2 -- which version and language are you using / in which file did you find that image?

 

[AD]

I have the English DOS CD version. I'm fairly certain it's the US version. I believe the file was TESTBED.SHP and was on CD1.