Integrating alternate model formats

Integrating alternate model formats

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Post Reply
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Integrating alternate model formats

Post by SpliFF »

UPDATE (Dec 2010). First release. See this post for details.


I know the topic of a new model format comes up a lot but I was wondering about something. An engine I was planning to use before I found Spring Apocalyx, has model loaders for MD2, MD3, MDL, Cal3D, and MS3D (Most include animation support, some even skeletal).

Now given Apocalyx is pretty modular and I'm 99% sure it's GPL I could possibly integrate those model loaders into Spring. Presumably the models would then end up in some kind of generic model + animations data structure so I was wondering, generally speaking, what additional information or steps would Spring need to to use these models for units, terrain or features?

Obviously it's hard to say without seeing the Apocalyx code, and I'm not that familiar with the Spring code either, but I'm still interested in knowing what kind of issues I might need to deal with.

The only issues I know for sure would be dealing with Spring-specific data like teamcolor, model height and radius. Automatic calculation of these would be difficult for models with long pointy bits. I was thinking though that the unit FBI might be able to specify these like it does for footprint. The other missing link would be knowing the units 'base' piece though I guess you could set that via FBI as well.

example unitname.lua:

Code: Select all

-- Set the radius offset from actual model center
centerX = 0.0,
centerY = -5.0,
centerZ = 0.0,
-- unit radius
radius = 6.0,
-- unit parent piece name
basePiece = 'body',
-- scale model pieces
scale = 0.5,
So, other than telling me it's going to be really hard (i know that), any useful comments on the actual requirements?
Last edited by SpliFF on 18 Dec 2010, 00:01, edited 1 time in total.
daan 79

Re: Integrating alternate model formats

Post by daan 79 »

Image

compatible model
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

Thanks BD. Just to be clear is there anything I need other than a valid S3DModel to support Spring's current features (ignoring skeletal animation for the minute) ?

BTW, I like the look of assimp too. Probably the right way to go with this. It might also be a good idea to cache converted models similar to the path and archive caches. This could significantly speed up loading as long as the cache is flushed when the model or Spring version changes.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Integrating alternate model formats

Post by hoijui »

what if we choose only the most potent model format to be suppoerted by spring, and have a separate application, or descriptions for many programms, how to convert fro many given format to this one? That would make spring simpler (which is alwyas a good thing in software).
....Is there something like an all potent model format?
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

Well the main purpose behind it was to take existing free models or your favorite model editor and basically use them "as-is". Having an intermediate application can make things harder. As a real world example it took me about 30 "manhours" of work to get some models out of Warzone2100. The reason is the process looks like:

1.) Extract PIE models from WDG archive
2.) Install Blender plugin to read PIE files
3.) Install Python
4.) Read model file
5.) Export 3DS from blender
6.) Import 3DS to Upspring
7.) Modify in Upspring
8.) Save as S3O
9.) Pray it all worked

At each stage there were always issues. Missing dependencies, wrong versions, missing or inadequate document, bugs and so on. I've just got my first complete model ingame after starting the process 6 months ago.

In short, pulling some steps out of the process is a huge bonus for content developers. On the flip-side it adds complexity to Spring. On the other hand the assimp library offers a way out by encapsulating the added complexity in a library which can be extended and bug-fixed outside of Spring, like GLUT, Lua, OpenAL, etc.

I guess it's a tradeoff between content developers and the Spring devs, however I think in the long run it will make things easier all round, especially if existing s3o/3do content can be easily ported to a more common format (meaning all model code can be virtualised through assimp rather than dealing with formats on an individual basis).

Now to answer your actually question, yes it isn't a bad idea to have an external converter (convert models once rather than on each load) but that does require more work on the part of content makers and opens up more things that can go wrong. With the conversion internal to Spring then you only need a working copy of Spring and the assimp library.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Integrating alternate model formats

Post by hoijui »

ok, i fully accept this as beeing better :-)
..its not like i had any idea bout all this, i just wondered if possibly.. it was simpler. but the abstracting through this library sounds good!
Thanks for the explanation!
daan 79

Re: Integrating alternate model formats

Post by daan 79 »

I went fromsome tiff or something in to blender and than to max and from there to upspring and i was lucky to have half of the textures preserved for the mod battlestar galactica.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

I've looked at assimp and I _really_ like what I see. I've decided to put further development on my mod on hold for a while to experiment with setting this up. My plan is to focus on enabling this only for features and possibly terrain. That will take many complexities like COB interaction out of the equation. If things go well then I'll evaluate the potential for implementing a generic unit model loader.
User avatar
Agon
Posts: 527
Joined: 16 May 2007, 18:33

Re: Integrating alternate model formats

Post by Agon »

Assimp loads different model formats into a general abstraction layer which is flat based like MD5, I'm right?

You would need to integrate assimp into Spring and convert the "assimp"-model to the tree based 3DModel of Spring.
Or write another 3DModel where you can convert the "assimp"-model much easier.

I think the first one is kinda a hack and 3DModel does not support very much. A Collada file would blow 3DModel simply away.
Ether extending or writing another 3DModel would be the best, thought.

Warning I don't know much about this, better ask Kloot or jK.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

It's more sophisticated than a flat file. It generates a scene graph as a class instance maintaining node names, node relationships, bones, meshes, lighting, texture and animation. It also maintains a pointer to the root node (or 'base' in Spring parlance).
The Node Hierarchy
Nodes are little named entities in the scene that have a place and orientation relative to their parents. Starting from the scene's root node all nodes can have 0 to x child nodes, thus forming a hierarchy. They form the base on which the scene is built on: a node can refer to 0..x meshes, can be referred to by a bone of a mesh or can be animated by a key sequence of an animation. DirectX calls them "frames", others call them "objects", we call them aiNode.

A node can potentially refer to single or multiple meshes. The meshes are not stored inside the node, but instead in an array of aiMesh inside the aiScene. A node only refers to them by their array index. This also means that multiple nodes can refer to the same mesh, which provides a simple form of instancing. A mesh referred to by this way lives in the node's local coordinate system. If you want the mesh's orientation in global space, you'd have to concatenate the transformations from the referring node and all of its parents.

Most of the file formats don't really support complex scenes, though, but a single model only. But there are more complex formats such as .3ds, .x or .collada scenes which may contain an arbitrary complex hierarchy of nodes and meshes.
In practice not all supported formats provide complete data at this stage. There will definitely be bugs as well, it's only at RC1 stage of development (post-beta, pre-release). I wouldn't expect miracles but I would be suprised if node names and relationships weren't preserved for the formats that actually support them.

Doco:
This is the scene class. All imported models should have these members:
http://assimp.sourceforge.net/lib_html/ ... scene.html
Even more detail:
http://assimp.sourceforge.net/lib_html/data.html
aiScene forms the root of the data, from here you gain access to all the nodes, meshes, materials, animations or textures that were read from the imported file. The aiScene is returned from a successful call to Assimp::Importer::ReadFile(), aiImportFile() or aiImportFileEx()
The main difficulty I see from this statement is that it expects a file. Spring's VFS has no "files", since everything external is virtualised data from compressed archives. A workaround for this should be reasonably straight forward though.

At any rate the resulting object is far more detailed than Springs 3DModel/S3DModel classes. Of course putting that data into use is an entirely different beast. I am certainly not committing myself to building an animation system or model deformations. As I said before, my plan is to implement static models from this library and leave the really hard stuff to people better qualified.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Integrating alternate model formats

Post by jK »

SpliFF wrote:
aiScene forms the root of the data, from here you gain access to all the nodes, meshes, materials, animations or textures that were read from the imported file. The aiScene is returned from a successful call to Assimp::Importer::ReadFile(), aiImportFile() or aiImportFileEx()
The main difficulty I see from this statement is that it expects a file. Spring's VFS has no "files", since everything external is virtualised data from compressed archives.
check the definition of aiImportFileEx ;)

PS: Joints/Bones are very similar to Spring's model pieces -> it is just that joints/bones don't save vertices themselves (also you can't hide them).

PPS: aiScene (which holds the meshes,...) is easy to integrate into the S3DModel class.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

Thanks jK, I was going to look into that but I figured the Ex extension meant it was more of a 'smart loader' (ie, guesses the model format)

One more thing about using an external app. I have three specific issues with it:

1.) External apps are generally poorly maintained relative to the core application. As a case in point ArchiveMover (despite being part of the Spring source) has issues which have been neglected for a long time. They are quite simple problems (like not recognising the new modinfo.lua format) but the energy to fix them seems disproportionate to the main Spring application. Upspring isn't even being maintained any more!

2.) Related to point 1., external apps has a disappointing habit of being implemented in horrible, os-centric, outdated languages like VB and Delphi. The reasons are irrelevant but the trend is there. Spring is committed to being cross-platform and already has the cross-platform display, logging and utility code that a model converter would require.

3.) Assimp has no export facilities, it is geared entirely to model import. An external anything->s3o writer would need a full s3o export implementation. That's not too bad but what if Spring changes it's preferred format in the future?

All in all I don't consider Springs load time to be an area of concern. Even really fast games (played to conclusion) take a minimum of 10 minutes to play out. If Spring took another 30 seconds to load due a more complex model loader I wouldn't be too upset. Supreme Commander takes much longer to load a comparable map and that hasn't hurt its popularity much.
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Integrating alternate model formats

Post by Tobi »

If load time is a problem just add some caching :-)

But indeed, for real games I wouldn't consider it an issue. For developing it may be a bit frustrating, though, but I suppose if it's frustrating enough someone would indeed add caching ;-)
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

jK wrote:PPS: aiScene (which holds the meshes,...) is easy to integrate into the S3DModel class.
Perhaps, but if this thing works out I then I can see mangling/dismantling the aiScene to fit the constraints of 3DModel being a real head-scratcher for future devs. It's like breaking down your ferrari to fix your Datsun. I mean no disrespect to anyone who helped develop the 3DModel/S3DModel classes, it's just that they're dated in comparison to the shiny new "ass" format.

I'd rather go the other way and build an aiScene from the s3o data. I know this means changing more code overall but it's a cleaner way to go forward. In my mind aiScene is the new 3DModel, not an intermediate.

Of course I might just change my mind when I realise how much work is involved. :(
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Integrating alternate model formats

Post by jK »

It is never good to use classes/structures of third party products in your code, cos:

1. you can't extend those by your needs
2. they can change with new versions, and break your whole program
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

You're right. I was about to take that back but you beat me too it. Yes, the whole point of C++ is customisation and extension. I will extend the 3DModel interface.

Apart from that aiScene doesn't gel well with Spring's naming convention and I hate inconsistent naming above all other sins.
User avatar
Pressure Line
Posts: 2283
Joined: 21 May 2007, 02:09

Re: Integrating alternate model formats

Post by Pressure Line »

SpliFF wrote:The reason is the process looks like:

...list of steps stuff...

At each stage there were always issues. Missing dependencies, wrong versions, missing or inadequate document, bugs and so on. I've just got my first complete model ingame after starting the process 6 months ago.
well tbh you're kinda doin it wrong (if you want to make a Spring game you are going to have to become proficient in the use of at least one 3d modeling suite) and model conversion (depending on the format) can be quite tricky and involve multiple steps and/or using programs that you are not familiar with to get them into a usable state. this is the price you pay for not creating the content yourself.

as a side note: some of the models for what im working on are in *.ase format. Blender has an .ase import plugin availible, so does 3dsmax (iirc 3dsmax actually has native ase import/export capability) but both programs are disporportionately hard (for me) to use. luckily i found a program called 3d Object Converter, which can import and export hundreds of model formats (including more than a few game model formats!) which has simplified the process for me. its all a matter of having the right tools and knowing how to use them

Im not opposed to moar model formats being supported btw, esp not if we can include better animation support.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Integrating alternate model formats

Post by SpliFF »

I like 3DConverter, I even got the developer to support Supreme Commnder models - but asking modders to buy tools isn't really practical given the high percentage of modellers who are still at school.

Anyway, I wasn't doing any wrong, what I did was save myself literally hundreds of hours of modelling and texturing work (we're talking about roughly 40 models here). The conversion time was only a fraction of the time it would take me to start from scratch. That's time I can spend on things I'm better at.
User avatar
Pressure Line
Posts: 2283
Joined: 21 May 2007, 02:09

Re: Integrating alternate model formats

Post by Pressure Line »

true enough that... i didnt exactly buy it though >_> i needed it because blender was messing up the uvs on the .ase models i was converting (the mc2 stuff) and it has a neat batch converting tool.

virtually all of the current/in public development (ie i know about them) are either a) TA based or b) scratch built so for most content devs its not an issue.

but thats beside the point. either completing the md5 model support (bones plz?) and/or making s3o better (multiple textures per unit plz?) would probably be time better spent imo. but thats just me :)
Post Reply

Return to “Engine”