Wing Commander Toolbox

The following is an example for extracting the cockpit images. Since the game data is divided among multiples files, and is contained in files within files, a multi step process is used.

We start by extracting the files from a tree file:

WCArmToolsCmd.exe untree DATA.TRE​

Next we convert a palette file into an image:

WCArmToolsCmd.exe unpalette DATA\PALETTE\SPACE.PAL​

Next we unmake an interchange file since it contains the shapes we need:

WCArmToolsCmd.exe unmake DATA\COCKPITS\ARROW.IFF /t:Interchange​

Finally, we convert the shapes into images:

WCArmToolsCmd.exe unshape ARROW???.SHP /p:SPACE.PAL.PNG​


ARROW000.SHP-Shape000.PNG
ARROW001.SHP-Shape000.PNG


ARROW002.SHP-Shape000.PNG
ARROW003.SHP-Shape000.PNG
 
Although there is still work to be done for Armada, I have decided to skip ahead to WC3. I have been looking forward for a while now to start picking this game apart and it will probably involve plenty of work since everything changed in WC3: the tree archive format is different and so is the pack file format; a new compression scheme was introduced (big thanks to @Pedro for pointing me to functioning code); the shapes use a new image encoding and even the palettes are also encoded differently; and of course the addition of video.

Since it is easier to unmake a file than to make a new one, I will focus on undoing the game data. We have to start somewhere and the best place to start is with the tree archives. It has been commented in the past that the tree archives seem unnecessarily complicated, and I agree. As usual, the documentation provided by @HCl has been very helpful.

In Privateer 1 (and Armada), the tree archives use a simple directory listing all the files by name and an offset to the data. For WC3, they implemented a mechanism to remove the file names from the archive directory, which I think was done to make the life of fans more difficult; although maybe I am being unfair and this was done for performance reason.

So if the file names are not present, how does the game find a file? It uses a hash of the file name instead. Simply put, the hash value is a fixed sized value (number) generated from a file name of variable length; for example:

0x7C073737 ..\..\DATA\PROFILE\KCRUISER.IFF​
0x2364EEC4 ..\..\DATA\MOVIES\INFLIGHT\SC_28.MVE​

But the hash function used for WC3 is problematic; it generates lots of collisions meaning different file names can easily produce the same hash value, and these values must be unique within a tree archive and even across all the actively loaded archives. Which is why you will see some file names included in an archive to compensate for these collisions:

0xAFDC274D ..\..\DATA\MISSIONS\TSIM008.IFF​
0xAFDC274D ..\..\DATA\MISSIONS\TSIM010.IFF​
​
0xD92B954E ..\..\DATA\OBJECTS\SHUD.IFF​
0xD92B954E ..\..\DATA\OBJECTS\TALL.IFF​

For a little while now, I have been hacking away at these hashes to reverse them to their original file names. I have managed to reverse all of them and I am 98-99% confident on the names. Although in some cases I may not be able to determine the original name, the name that I use is correct in that it will produce the same hash value. Included in the download is a directory (.\external\wc3-hashes\) with text files containing a mapping of hashes to file names.

So this update to the toolbox is small: tree archive extraction with reversed file names; but one that will greatly facilitate working with the large amount of data. As usual, see the first post for an updated download.

When extracting the files within a tree archive using the extree command and specifying one of the included text file to use as the hash-to-name list, it will create the proper directory structure, same as is done with Privateer 1 and Armada.

WC3ToolsCmd.exe extree CD1MISS.TRE /hashes:external\wc3-hashes\CD1MISS.TRE.txt​

Code: 
DATA\PROFILE\VICA1.IFF
DATA\PROFILE\VICA2.IFF
DATA\PROFILE\VICA3.IFF
DATA\PROFILE\VICA4.IFF
DATA\PROFILE\VICB1.IFF
DATA\PROFILE\VICB2.IFF
DATA\PROFILE\VICB2NE.IFF
DATA\PROFILE\VICC2.IFF
DATA\PROFILE\VICC2NE.IFF
DATA\PROFILE\VICD1.IFF
DATA\PROFILE\VICD1NF.IFF
DATA\PROFILE\VICD2.IFF
DATA\PROFILE\VICD3-D2.IFF
DATA\PROFILE\VICD3.IFF
DATA\PROFILE\FIRECLAW.IFF
DATA\PROFILE\SIMFLASH.IFF
...

WC3ToolsCmd.exe extree CD3MISS.TRE /hashes:external\wc3-hashes\CD3MISS.TRE.txt​

Code: 
DATA\PROFILE\VICJ2.IFF
DATA\PROFILE\VICJ3.IFF
DATA\PROFILE\VICK1.IFF
DATA\PROFILE\VICK2.IFF
DATA\PROFILE\VICK3.IFF
DATA\PROFILE\VICK4.IFF
DATA\PROFILE\VICL1.IFF
DATA\PROFILE\VICL1D.IFF
DATA\PROFILE\VICL2.IFF
DATA\PROFILE\VICL3.IFF
DATA\PROFILE\VICL4.IFF
DATA\PROFILE\HOBBESB.IFF
DATA\PROFILE\PLAYERJ.IFF
DATA\PROFILE\THRAKATB.IFF
DATA\PROFILE\THRAKATH.IFF
DATA\PROFILE\BEHEMOTH.IFF
DATA\PROFILE\SKYMASTR.IFF
...
 
Last edited:
Thanks for this. As you press on would you be willing to share source and/or collaborate?
I don’t have much more than I last told you but I have a bit relating to game flow, audio and mission definitions
 
Added more extraction support for WC3; this time something we can actually see. This update includes support for extracting an image representation of a palette; this is actually a little trickier than it might seem. The colors in WC3 are not simply presented as they are encoded in a palette. Instead, gamma correction is applied to brighten or even darken a scene if necessary.

In DOS the options are more limited. Either gamma correction is disable which creates a very dark scene, or it is set to a low or high increase in brightness.

The flight deck without any gamma correction:

flight-deck.png

The palettes for the flight deck - with no gamma correction, and with low and high increases:

flight-deck-palette-dos-off.png flight-deck-palette-dos-low.png flight-deck-palette-dos-high.png

In Windows, a default gamma shift is set, but you can easily brighten or darken a scene in graduated steps. This is done by using a seed value to generate a mapping table which is used to shift the individual color components. When extracting a palette, I use the same default setting as in Windows, but you can specify a different value if you wish. You can also disable gamma shifting and produce the same effect as in DOS.

From the command line, the first example uses the default seed value of 80, the second example increases the brightness, and the third example disables any gamma shifting:

WC3ToolsCmd.exe expalette FILENAME.PAL​
WC3ToolsCmd.exe expalette FILENAME.PAL /gamma:90​
WC3ToolsCmd.exe expalette FILENAME.PAL /gamma:-1​

The other bit of functionality is to make use of the palette and extract images from shape files. The use of an external palette is needed since shapes contain no color values.

Since the game data is divided among multiples files, and is contained in files within files, a multi-step process is used.


Let's begin by extracting the files from a tree archive (also making use the hash-to-name mapping file):

WC3ToolsCmd.exe extree GAMEFLOW.TRE /hashes:external\wc3-hashes\GAMEFLOW.TRE.txt​

Afterwards we extract the packets:

WC3ToolsCmd.exe expack DATA\GAMEFLOW\SHAPES.PAK​

Most of the extracted packets are shape files, but the ones which are 768 bytes are palettes. Let's rename a few files for convenience:

rename SHAPES.PAK-Packet036.PCK *.PAL​
rename SHAPES.PAK-Packet037.PCK *.SHP​

Next we convert a palette file into an image:

WC3ToolsCmd.exe expalette SHAPES.PAK-Packet036.PAL​

Finally, we convert the shapes into images:

WC3ToolsCmd.exe exshape SHAPES.PAK-Packet037.SHP /palette:SHAPES.PAK-Packet036.PAL.PNG​

SHAPES.PAK-Packet037.SHP-Shape000.PNG


Here are some additional images:

SHAPES.PAK-Packet044.SHP-Shape000.PNG
SHAPES.PAK-Packet045.SHP-Shape000.PNG
SHAPES.PAK-Packet046.SHP-Shape000.PNG


Surprisingly, some of the rooms are composited:

SHAPES.PAK-Packet057.SHP-Shape000.PNG SHAPES.PAK-Packet058.SHP-Shape000.PNG SHAPES.PAK-Packet075.SHP-Shape000.PNG SHAPES.PAK-Packet076.SHP-Shape000.PNG
 
UnnamedCharacter said:
I would happy to share the knowledge I am gaining from WC3. Obviously I am focusing on the toolbox, but any assistance I can provide.
Click to expand...

Missions would be a big one, specifically scripting register usage. I've mapped out a lot of the instructions, especially those to do with gameflow (argh my disappointment when I realized they have two entirely separate scripting systems), much less on the variable front.
I also haven't done the video decoding yet (just the DVD version of WCIV); although it's well enough documented if you have working code that would save me time for the comms videos (audio is already working). I would *love* to be able to support color PSX/3DO video if you ever tackle port formats.
Also save game file formats I haven't looked at yet, I'd love to make it possible to share save games between the remake and original.
 
UnnamedCharacter said:
Added more extraction support for WC3; this time something we can actually see. This update includes support for extracting an image representation of a palette; this is actually a little trickier than it might seem. The colors in WC3 are not simply presented as they are encoded in a palette. Instead, gamma correction is applied to brighten or even darken a scene if necessary.

In DOS the options are more limited. Either gamma correction is disable which creates a very dark scene, or it is set to a low or high increase in brightness.

The flight deck without any gamma correction:

View attachment 11216

The palettes for the flight deck - with no gamma correction, and with low and high increases:

View attachment 11219 View attachment 11218 View attachment 11217

In Windows, a default gamma shift is set, but you can easily brighten or darken a scene in graduated steps. This is done by using a seed value to generate a mapping table which is used to shift the individual color components. When extracting a palette, I use the same default setting as in Windows, but you can specify a different value if you wish. You can also disable gamma shifting and produce the same effect as in DOS.

From the command line, the first example uses the default seed value of 80, the second example increases the brightness, and the third example disables any gamma shifting:

WC3ToolsCmd.exe expalette FILENAME.PAL​
WC3ToolsCmd.exe expalette FILENAME.PAL /gamma:90​
WC3ToolsCmd.exe expalette FILENAME.PAL /gamma:-1​

The other bit of functionality is to make use of the palette and extract images from shape files. The use of an external palette is needed since shapes contain no color values.

Since the game data is divided among multiples files, and is contained in files within files, a multi-step process is used.


Let's begin by extracting the files from a tree archive (also making use the hash-to-name mapping file):

WC3ToolsCmd.exe extree GAMEFLOW.TRE /hashes:external\wc3-hashes\GAMEFLOW.TRE.txt​

Afterwards we extract the packets:

WC3ToolsCmd.exe expack DATA\GAMEFLOW\SHAPES.PAK​

Most of the extracted packets are shape files, but the ones which are 768 bytes are palettes. Let's rename a few files for convenience:

rename SHAPES.PAK-Packet036.PCK *.PAL​
rename SHAPES.PAK-Packet037.PCK *.SHP​

Next we convert a palette file into an image:

WC3ToolsCmd.exe expalette SHAPES.PAK-Packet036.PAL​

Finally, we convert the shapes into images:

WC3ToolsCmd.exe exshape SHAPES.PAK-Packet037.SHP /palette:SHAPES.PAK-Packet036.PAL.PNG​

View attachment 11220

Here are some additional images:

View attachment 11221 View attachment 11222 View attachment 11223

Surprisingly, some of the rooms are composited:

View attachment 11224 View attachment 11225 View attachment 11226 View attachment 11227
Click to expand...
something that i noticed while playing with reshade filters using the windows version

the 3d backgrounds really depends on dithering to remove the artifacts and keeping the image a bit more defined.
resident evil and other 3d rendered background games uses this trick a lot.
1569391709420.gif
 
@Pedro

I am working on the videos right now. It is going well, being made considerably easier thanks the to code from @HCl; in fact, I am using it practically verbatim.

Not sure what I will work on next, probably on the interchange files (IFF); if I dig deep enough on these this might be the most useful for you.
 
UnnamedCharacter said:
@Pedro

I am working on the videos right now. It is going well, being made considerably easier thanks the to code from @HCl; in fact, I am using it practically verbatim.

Not sure what I will work on next, probably on the interchange files (IFF); if I dig deep enough on these this might be the most useful for you.
Click to expand...

Thanks, we have the dev cd iff file reader, but it's buggy on modern systems and not very flexible. Since it's my primary way of reverse engineering most of the files and improved reader would be super helpful (particularly as you can't specify how to view chunks (float, int, string etc)). If you could save out chunk definitions so when you re-load it you keep that view it'd shave months off development (and fans could contribute new definitions as they reverse engineer each iff).
 
ShadowArm said:
the 3d backgrounds really depends on dithering to remove the artifacts and keeping the image a bit more defined.
Click to expand...
It seems to me that both versions are using the dithering method, where one is more fine grained than the other one (regarding your example image).
 
Hi, I was wondering if the wctoolbox code is available. Is it on github or something? I would love to see if I could get it to run in Linux.

thanks!
 
Those are pretty amazing! And to think that those have been locked up on the discs everyone owns right under our noses for more than 25 years.
 
What's great about these is they really help bridge the gap from Wing Commander 2's art style to Wing Commander 3. It looks like Wing 3 started out by extending WC2's visual style and then they've made a decision to turn down the lighting and make the ship look darker. I also like how the Hangar shots reveal why the Wing Commander 3 3DO Hangar outside texture looks different to the PC edition.
1586124777653.png
1586124819437.png
 
Darkmage said:
What's great about these is they really help bridge the gap from Wing Commander 2's art style to Wing Commander 3. It looks like Wing 3 started out by extending WC2's visual style and then they've made a decision to turn down the lighting and make the ship look darker. I also like how the Hangar shots reveal why the Wing Commander 3 3DO Hangar outside texture looks different to the PC edition. View attachment 11296View attachment 11297
Click to expand...
i prefer more 3do darker space than navy blue one from PSX and PC

i hope that the toolbox also helps in discovering a way to replace or even retexture models on the KS version
 
You must log in or register to reply here.
Back
Top