Confederation [Work in Progress]

[AllTinker]

Hey all - I thought I'd post something about a project I've been chipping away at for quite a while now, which I've been calling Confederation.

Confederation is essentially a custom cross-platform engine which reads original data files from the Wing Commander (and Privateer) series - based on some of my own ("clean room") reverse engineering, but also of course the work of various members of the community over the years. I don't think there's any format left to figure out at this point, at least as far as this project is concerned...

My immediate goal (which may well still take a couple more years) is to fully re-implement Wing Commander 1 & 2, favouring making subtle improvements here and there over perfect accuracy. It's not the only motivating factor, but the main one is honestly the lack of speed limiting in DOS WC1/2... I can't deal with it anymore. I've tried modifying the original machine code in various ways, but without the source code I've given up on achieving what I want with the original engine. I've tried Kilrathi Saga, but that doesn't do it for me either.

In concert with rebuilding WC1/2, I've been working on a mission editor, not only to view/debug existing missions, but to create new ones; and new campaigns, including new cutscenes/dialogue and so on. The whole thing is built around a scripting system, so even before I open source it (which I plan to do), it should be possible to make very complex addons.

I'm also trying to make sure it can easily support new and/or replacement art - one of the many things I've been slowly working on are widescreen extensions of the WC1 art (supporting up to 21:9 currently). Keep in mind these are very much work-in-progress / in lieu of something better by a more skilled artist:

I've also been experimenting with building "wrap-around" versions of the cockpits, so they can give more of an impression of your head actually turning, rather than just snapping from view to view:

And I'm not sure if this is going anywhere or worth showing, but speaking of adding new stuff here's a rough sketch of a potential Firekkan Confed pilot (with original art reference on the left). I'm not sure if they'd wear something so similar to regular uniforms though (maybe?):

Here are some quick shots of the inbuilt content browser, just showing some ships from WC1-4 and Armada respectively:



I'm not quite ready to show actual gameplay, but for the sake of showing something in motion, here's a short vid of the OriginFX intro running in my engine (using inbuilt MT-32 emulation for the music).

Anywho, that's probably enough for now. Life and motivation-willing, I hope to have some gameplay to show "soon". Maybe some other things in the meantime. I dunno, if anyone cares please let me know!

 

[wcnut]

What do you mean you arn't a skilled artist? That is some damn fine work! If you had told me these were hidden assets from the first game, I'd have believed you.

 

[Bandit LOAF]

Super cool! I'm fascinated and there's a lot that could help other projects for sure!

Have you been able to read data from any of the console ports? Getting the 3DO, Playstation, SNES, etc. versions of artwork and other assets is a big blocker for me right now!

 

[Pedro]

This is really neat; it's actually the same approach we've been taking the WC4 remake (although we've obviously been focused on the WC3/4 formats).

Whilst we've reverse engineered pretty much everything on PC from missions to comms as LOAF says if you've done any work with the PSX comm videos that'd be huge for us as otherwise we are stuck with the monochrome PC versions which cause issues with AI upscaling.

Love the 21:9 artwork, it really looks like it belongs.
Would you be interested in collaborating? Our SFX man who also handles the tools was just about to start work on a WC3/4 mission editor but it seems foolish to not co-ordinate.

Come say hi on Discord if you like!

Discord - Group Chat That’s All Fun & Games

Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.

discord.gg

 

[AllTinker]

If you had told me these were hidden assets from the first game, I'd have believed you.

Thanks! Definitely what I was going for, so that's nice to hear.

Have you been able to read data from any of the console ports? Getting the 3DO, Playstation, SNES, etc. versions of artwork and other assets is a big blocker for me right now!

So far I've mainly looked into the FM Towns versions, the Sega CD version, and 3DO/Mac Super Wing Commander. My intent is to allow switching/mixing of different datasets, for instance audio-wise for WC1/2 to toggle some combination of (emulated) OPL FM or MT-32, and/or different sets of digital sound effects/music/voices, either from non-DOS ports or new custom sets.

First of all I might as well mention a few more things I've figured out/been working on for the original DOS/Windows versions:

One thing I've done with the DOS version is figure out the MIDI format. No regular MIDI player I tried copes with the data. I eventually worked out that this is because Wing Commander uses a second/non-standard "meta message" type (0xFE) to store data; which the engine can then use to drive the timing of various things; presumably primarily internal to the music system itself. I'm not sure if that's a new finding, but I couldn't find any info on it... It does also open up the possibility of more directly porting the music to General MIDI (to better drive non-MT-32 devices / soundfonts), but I haven't actually done anything on that (yet).

I've also been (painstakingly) extracting the OPL FM information for the OPL (Adlib) sounds and music, so they can be emulated in realtime, rather than simply playing back a digital capture. I believe I have all of the effects and music for WC1, and it does sound accurate, but there's room for technical improvement on how I'm handling the music.

As for the later games - the WC4 Remastered crew may already be well aware of this, if it's relevant, but - when I was looking into WC3/4, I did notice people were using code pulled straight from the original executable to extract some of the compressed data. Because I want to release Confederation with zero legal issues, I couldn't go down that route, so I figured out the actual algorithm WC3/4 uses - QFS, which is used in addition to the older/more common LZWX. Prophecy uses LZSS, which was much easier to figure out and I'm fairly sure was already known.

The source code release for BRender has helped correct some things here and there with Privateer 2, although I'd already figured out the bulk of the formats prior to that. Of course it's very much wishful thinking that I will get around to Privateer 1/2... But who knows! There's so much to explore just in WC1. While I'm daydreaming, one thing I'm very keen to add is multiplayer/cooperative support - I've been writing the code with it in mind. Here's a quick hypothetical generic pilot comms portrait I put together:

On to the ports:

I've extracted the digital audio from the FM Towns versions of WC1/2 - I'm not sure if that's new or not. I've attached a ZIP containing the WAVs I extracted from their FMP format; it also includes the code I wrote to extract them. There are a few joke sound effects in there...

EDIT: Too big to attach I guess - I've uploaded it here instead.

I believe everything else was virtually identical to the DOS version beyond the Japanese localisation, although I did find a couple of extra images (which I posted quite a while back - first one clearly an employee easter egg "hidden" in a very obscure image format):


For the Sega CD version, all I've done is extract the voice samples, which others have also done. Likewise I have all the Amiga audio/music extracted. The SNES version I've barely looked at - as far as Confederation goes I may try to extract the audio/music, which is usually relatively easy on SNES given how the sound chip works. It's definitely possible to pull the graphics etc., but from memory they were pretty seriously crunched down versions, so it's not on my radar to do myself at this stage...

Super Wing Commander is the big one, but I've definitely been treating is as a distant secondary objective. I'd love to have it playable in Confederation eventually - I've figured out some of the audio/visual formats but there's still work to be done there. Despite both versions being identical(?) at runtime, there seems to be quite significant differences data-wise between the 3DO/Mac versions. So far I've had more luck extracting stuff from the 3DO version, but I've definitely cast an eye over the Mac files - some are potentially interesting/helpful for pulling out a complete dataset.

Whilst we've reverse engineered pretty much everything on PC from missions to comms as LOAF says if you've done any work with the PSX comm videos that'd be huge for us as otherwise we are stuck with the monochrome PC versions which cause issues with AI upscaling.

Hmmn, I have actually looked a bit at PSX WC3/4 and definitely extracted some stuff, but it was a few years ago and I'll need to dig up my notes for them, which have apparently gone walkabout. Either way I'm keen to have a (re-)look into it!

Would you be interested in collaborating? Our SFX man who also handles the tools was just about to start work on a WC3/4 mission editor but it seems foolish to not co-ordinate.

Definitely up for collaborating, even if it's just (hopefully) helping with the aforementioned PSX stuff if I can. The mission/campaign editor I'm working on is pretty tightly integrated with my engine and scripting (C++ / AngelScript respectively); all the tools are integrated rather than separate, so it might be tricky to consolidate it directly. Not sure! Anyway I'll jump on your Discord at some point soon.

 

[Pedro]

As for the later games - the WC4 Remastered crew may already be well aware of this, if it's relevant, but - when I was looking into WC3/4, I did notice people were using code pulled straight from the original executable to extract some of the compressed data. Because I want to release Confederation with zero legal issues, I couldn't go down that route, so I figured out the actual algorithm WC3/4 uses - QFS, which is used in addition to the older/more common LZWX. Prophecy uses LZSS, which was much easier to figure out and I'm fairly sure was already known.

I knew it as Refpack; the source that everyone references for it was actually released here on the CIC on one of the dev CDs.

Definitely up for collaborating, even if it's just (hopefully) helping with the aforementioned PSX stuff if I can. The mission/campaign editor I'm working on is pretty tightly integrated with my engine and scripting (C++ / AngelScript respectively); all the tools are integrated rather than separate, so it might be tricky to consolidate it directly. Not sure! Anyway I'll jump on your Discord at some point soon.

Probably not a lot of cross over if you're converting to a scripting language; we are using C++ and the engine has support for scripting languages, but that's not the path we've been taking. We've been reading in the original script files as is rather than converting to an intermediate format (although we do spit out the code in a semi human readable format to sanity check it) so we were thinking of an editor that could make missions which would work with the original if people so chose.
Still I think most WC projects are C++, what we can share may be limited to a set of library functions and classes for reading the original data into easier to parse intermediate structures, but that would still have value. It's always a shame to have to reinvent the wheel everytime; I know a lot of what I had to do HCl had done before, and some of the toolbox we'd already done but hadn't shared it.

Anyway hope to see you stop by soon.

 

[FekLeyrTarg]

This sounds and looks very much awesome.

Do you intend to support the Kilrathi Saga Port of WC1 and 2 as well?
If yes, I'll be happy to contribute my Oldziey/KS-style arrangement of the OriginFX fanfare and the SO1 track "Thrakhath's New Plans" which I created last year.

 

[AllTinker]

Those sound awesome!

In a way you could see Confederation as more of an alternative to Kilrathi Saga. Both are essentially alternative executables for WC1/2 using the same game data; but although people are doing amazing stuff with KS, a big part of the idea for Confederation is that dropping in new content (like your music) should be much easier by design. Think more in the vein of adding new packages/WADs to a Doom source port with new levels or replacement art/music/gameplay/etc. In the end though KS/DOSBox will remain the most authentic ways to play WC1/2 on modern hardware, so they'll still have their place pretty much regardless of how accurate Confederation becomes.

Probably not a lot of cross over if you're converting to a scripting language; we are using C++ and the engine has support for scripting languages, but that's not the path we've been taking. We've been reading in the original script files as is rather than converting to an intermediate format (although we do spit out the code in a semi human readable format to sanity check it) so we were thinking of an editor that could make missions which would work with the original if people so chose.

To be clearer, the scripting is more for the game logic and some of the tooling, for modability and (in theory) development speed. I'm reading everything I can from the original data, but a lot of stuff for WC1/2 is buried away in the executable; that's mainly the stuff I'm rebuilding in script. Apart from just the intellectual enjoyment of pulling apart the formats I honestly haven't gone that deep on the other games' intricacies yet - I'd be surprised if I'm ahead of you folks on anything for WC4. Hopefully I can help with the PSX VDU comms videos though; I'm looking into it.

Writing out content compatible with the original games would be pretty cool - that's not something I'm currently planning on doing with Confederation. Perhaps via a restricted compatibility mode for specific games, eventually? Maybe? The core issue there is my focus on things like allowing mixing of content from different games (e.g. having Privateer content in a WC2 mission), and adding new content generally; unrestricted by the original engine(s).

Still I think most WC projects are C++, what we can share may be limited to a set of library functions and classes for reading the original data into easier to parse intermediate structures, but that would still have value. It's always a shame to have to reinvent the wheel everytime; I know a lot of what I had to do HCl had done before, and some of the toolbox we'd already done but hadn't shared it.

You're right, it's something I've been thinking about too. Some kind of hypothetical separate C++ open-source base library for reading/writing the content seems to make the most sense to me. The Confederation code as it stands is built on my own base library and fully indulges in my own coding preferences - but the code could absolutely be refactored; e.g. to use STL or some other set of base containers, custom or otherwise...

 

[Vidmaster]

The fixed framerate SWC is the game of my dreams, no matter how distant it might be. Good luck with this project

 

[Wedge009]

Am I right in understanding the extensions to the background images to support wide aspect ratios are not from the original data? At first glance I thought they were...

 

[AllTinker]

Am I right in understanding the extensions to the background images to support wide aspect ratios are not from the original data? At first glance I thought they were...

Happy to hear it. I just tried to copy the original art style as closely as I could manage; I reused bits and pieces where possible, but a lot of it was done from scratch.

Here's another one - I'm planning to make some parts of these dynamic, e.g. perhaps it won't always be a Hornet in the bay:

A quick update on the project: I've recently started work on splitting out the code for reading the original data formats into a separate C++ base library which will be open sourced as soon as it's stable API-wise.

The idea is that it will be shared between Confederation and WC4 Remake; thus we can more easily pool our efforts - and those of anyone else who'd like to contribute. The library will be usable by any projects wanting to read (and perhaps write) formats used by the Wing Commander & Privateer series, and perhaps some other Origin games too (e.g. I've done some reverse engineering of BioForge previously, and Strike Commander has been mostly(?) reverse engineered already).

 

[AllTinker]

In case anyone is curious about my process, in the case of the landing screen it looks something like this (with original art layer hidden, and yellow reference line layer visible):

Just tracing out important lines to capture the perspective, and then doing what I can to match the original colours and style.

I try to match it with other existing in-game references, like the shot of the "runway" during landing (which I also have to finish extending...) which led me to adding the 7 from 07.

I have several others in various stages of completion for cutscenes and such; I've been focussing on the primary gameplay ones for now, which I think are all either complete or pretty close to complete. WC2 I haven't even looked at yet for widescreen art because it's much scarier. I'd still like to get to it eventually though.

 

[Howard Day]

AMAZING! I shall watch this with great interest.
You're doing wonderful work here. Any plans to add rotational angles to the ships?

 

[Klavs81]

This is awesome work! Excited to see where this goes!

 

[AllTinker]

Thanks guys!

Any plans to add rotational angles to the ships?

I have given some vague thought to that, and I think the only way I'd be able to manage it personally is to voxelise the sprites. Something like the Voxel Doom project, but (hopefully) through less labour-intensive methods - either way, at least the ships aren't animated... It may be a dead-end, but I think it's worth experimenting with at some point.

The other option would be having someone who actually knows what they're doing producing new renders - like your amazing work on your Wing Leader project. On that note, please do let me know if any kind of collaboration would be on the cards. No stress if not, though! I'm trying to make sure Confederation can support that kind of visual overhaul, at least for the older 2.5D games.

If and when it reaches the full 3D games, I don't intend to go much beyond the original experience, just modern resolutions/widescreen and other minor upgrades. I'll gratefully leave full-on remastering of those to WC4 Remake. If someone wants to contribute 3D ship models for WC1/2 to Confederation some day, I think it would be cool to have them match closely with the WC3/4 lowpoly style - the hope being that that way the different eras of ships could coexist nicely visually. Will be cool to see if we ever get there I guess.

 

[Howard Day]

Oooh, I'd be delighted to chat. I don't want to over promise, I already have way too many side projects, but I would still love to talk!

 

[Pedro]

If and when it reaches the full 3D games, I don't intend to go much beyond the original experience, just modern resolutions/widescreen and other minor upgrades. I'll gratefully leave full-on remastering of those to WC4 Remake. If someone wants to contribute 3D ship models for WC1/2 to Confederation some day, I think it would be cool to have them match closely with the WC3/4 lowpoly style - the hope being that that way the different eras of ships could coexist nicely visually. Will be cool to see if we ever get there I guess.

Just FYI we are adding a "classic" mode with support for the original artwork. It's basic at the moment only replacing the ship models, but long term it should look pretty indistinguishable from the original. We don't have any designs on supporting the old momentum free flight model that only WC3/4 had, but otherwise should be a classic experience.
Heck we've even added support for the PSX missions incase your nostalgia is for a reduced mission set and no ground missions 🤷‍♂️
We've also been adding WC3 support as we go so are *reasonably* confident that if we finish 4 then 3 should be pretty smooth sailing.

Once upon a time we considered doing the other titles but realistically we'd have just open sourced and left that to others. I don't want to say I underestimated how much work this would be as I always knew it'd be a lot - but maybe I overestimated how much time I'd have.

It's a shame that we are repeating work of creating flight models; AI etc, but we tech programmers are attached to our own engines. Still it seems foolish to redo the work specific to each title when the Privateers need love too


Super excited about a potential collaboration between you and Howie. We were all really looking forward to Wing Leader.

 

[AllTinker]

Oh I know, I tried the classic mode in the latest demo. To reiterate, my focus is on the 2.5D games for the foreseeable future, and that includes Privateer. That said, I don't believe that having multiple overlapping projects is a bad thing at all, but of course I do see where you're coming from. Hopefully this open base library project will help mitigate at least some low-level reinvention/rediscovery, and I hope we'll continue to collaborate, share and encourage in general. There's still plenty I'm keen to assist with for WC4 Remake (including WC3 stuff), even if it isn't directly writing game/engine code necessarily.

This will no doubt sound very familiar (or even stating the obvious), but the motivation for using my own code is grounded in some level of practicality, i.e. the core code drives my other existing and future projects (including commercial ones). Even game-specific code slowly improves and tests my core code. A flimsy justification for what's really just fun hobby indulgence, but it's a non-zero baseline.

Oooh, I'd be delighted to chat. I don't want to over promise, I already have way too many side projects, but I would still love to talk!

Awesome, and no stress; I'm quite familiar with that predicament.

 

[Wedge009]

The idea is that [the code] will be shared between Confederation and WC4 Remake; thus we can more easily pool our efforts...

That said, I don't believe that having multiple overlapping projects is a bad thing at all, but of course I do see where you're coming from. Hopefully this open base library project will help mitigate at least some low-level reinvention/rediscovery, and I hope we'll continue to collaborate, share and encourage in general.

I recall in the early days of our community, HCl and others did a lot of work in reverse-engineering and extracting information from the game files across the series. However, I recall those efforts were largely distributed in closed binaries (though I do remember seeing a bit of documentation as well). In our current context it does make sense to have FOSS libraries to share and pool efforts - did your current work use any of those past efforts or did you do everything from scratch?

 

[AllTinker]

I'm absolutely building on the work of the community - even the closed-source stuff has been tremendously helpful, by showing me what the extracted data is supposed to look like. The Wing Commander Toolbox has been especially useful in that regard, and I've looked at (I'm pretty sure) all of HCl's tools and write-ups. Doing things that way also helps achieve "clean-room" reverse engineering; while it's probably unnecessarily paranoid, that does help ensure that there are no legal issues present in the Confederation code in case EA goes full Nintendo at some point. So while not having source code (to the original game or extraction tools) is unfortunate, it does have at least one upside.

Wing Commander Toolbox also does some things Confederation won't, e.g. extracting values from the original executable. If I was working in .NET rather than C++ I'd probably just be using the WCT API...