I would recommend a mission format that is based on a text file (preferably UTF-8, for us Europeans, but I'm fine with ASCII as well), just like the FS2 mission files. Hell, I'm not a big fan of XML, but you could use that kind of stuff.
Or perhaps a JSON-based format. It is smaller than XML and IMO it even is easier to read for a human or a program. There are open source XML and JSON parsers available in many programming languages.
You could even consider making the mission file editor open source.
I fully understand you want the game itself closed source, but think about it: That way other skilled people can help create the editor, and add features. FRED2 has become really much better since it is open source.
But to be honest: I do quite some things in a text editor instead of FRED. If you know what you are doing it is actually less work in many cases.
EDIT: More technical stuff:
If you look at the FS2 mission file format you can see they have divided it in parts. The briefing is in there, the groups list, ship list, the events, mission parameters like backgrounds and stuff, and the messages. (from the top of my head, I'm at the wrong PC at the moment
) All of those are objects, and the ships and messages and so on are referenced by the events.
I think it is pretty well done. It has some weak aspects (Asteroid fields for example. Those are not cool), but overall I think it does its job pretty well.
BTW: Are your mission map sizes fixed? And can there be multiple maps in one mission, or would big mission with jumps in them actually be several missions stitched together, like in WC2 and WC3?
@wcnut
Sorry for the confusion about FS2, I was referring to the file
format, which is indeed text (open a FS2 mission with a text editor, you can read all that stuff!
)
Of course I know FRED does exist, and yes, it is a great tool. But I could live without a graphical mission editor, if there is at least an ASCII file format that is well documented.