Python S3O module and optimizer

Python S3O module and optimizer

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Python S3O module and optimizer

Post by Muon »

I wrote a Python S3O model parsing/serialization module, and a simple model optimizer based on it!

https://github.com/Muon/s3o-tools

The optimizer removes redundant data and reindexes the mesh. It does miracles on poorly exported models with bad index tables (looking at you, Kernel Panic!). I've tested it on KP's model set: it reduced the size of the model data by about half - from 737409 bytes to 430561 bytes. There should also be a performance boost because of the indexing but I was too lazy to do any real measurement :-) I'll probably add a tri-stripper later.
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

Damn nice stuff, way cleaner and better than the one I wrote. Just so you know, upspring doesn't handle triangle stripped pieces, while the engine theoretically does (I haven't tested it)

Upspring has a command called 'optimize', I think it does the same thing as your code - it removed redundant vertices. The performance gain from this is noticeable.
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

One of the S44 devs just ran it on the mod's models, which were already Upspring-optimized and it cut them down to size nicely. I ran it myself too, and the reduction is significant (http://pastebin.com/GVyQAq3C). Hmm, Upspring doesn't sound nice. And that fact I couldn't compile it didn't help its case.
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6240
Joined: 29 Apr 2005, 01:14

Re: Python S3O module and optimizer

Post by FLOZi »

* Can't guarantee that the S44 models have had Upspring's optimize run on them, I was commenting on the software's general lack of redeeming qualities. :P

FWIW Behe, your Python s3o parser is much nicer than mine. :wink:


Next up Muon, write an ASSIMP-loading model assembling tool to replace Upspring that is not Blender.
Last edited by FLOZi on 23 Jan 2012, 23:49, edited 2 times in total.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Python S3O module and optimizer

Post by jK »

:oops:
I think I know why you reduced even already optimized meshes:
You forgot the normals & texcoords! Vertices can have the same vertexcoords but different normals & texcoords, if so they need to be added separately to the vertexbuffer.
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

I love upspring, its one of the best tools around - sure it has UI oddities, but what software doesnt? Besides that, its reliable and fast to work in.
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6240
Joined: 29 Apr 2005, 01:14

Re: Python S3O module and optimizer

Post by FLOZi »

In what quantum reality is Upspring reliable? :shock:
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

jK wrote::oops:
I think I know why you reduced even already optimized meshes:
You forgot the normals & texcoords! Vertices can have the same vertexcoords but different normals & texcoords, if so they need to be added separately to the vertexbuffer.
No, I do full vertex comparisons - position, normals, and texcoords. It's the most conservative optimizer on Earth :P

Also, updated: forgot to rename optimize_piece() to recursively_optimize_pieces() when I cleaned up the code so the optimizer only went 2 levels deep (root and immediate children).
Last edited by Muon on 24 Jan 2012, 14:27, edited 3 times in total.
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

Ive put over 100 units ingame with upspring, didnt crash often, its use was deterministic. Sure I've run into quirks, but they were extremely minor.
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

Okay! So, with the latest fix, the optimizer reduces Market Garden's objects3d from 57540187 bytes to 36098843 bytes - a whopping 20.4 MB reduction. And everything works!
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6240
Joined: 29 Apr 2005, 01:14

Re: Python S3O module and optimizer

Post by FLOZi »

Beherith wrote:Ive put over 100 units ingame with upspring, didnt crash often, its use was deterministic. Sure I've run into quirks, but they were extremely minor.
Maybe i was spoilt by the luxury of 3do Builder 2.02 (not 2.1, that was pants too)

Muon; Perhaps a message at the end of processing several files with the total change in Mb?
Last edited by FLOZi on 24 Jan 2012, 13:55, edited 1 time in total.
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

It is DONE!

Also, fun fact: I just noticed my remapping algorithm also improves the spatial locality of the vertex data in exactly the same way as NvTriStrip's RemapIndices() function.
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

Wonderful! Ill test how this works with BAR. Thanks for your effort!

Did you do a performance test? Try it with /cheat then /give 500 gertiger and compare FPS.
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

No, not a really real one - don't have time today. But /cheat and /give all was so much smoother after I ran my optimizer on everything. It didn't stutter at all at normal zoom! (As opposed to much stuttering pre-optimization.)

EDIT: Went and did it anyway. On 500 tanks it went from 15 to 16 FPS (Linux, Sandy Bridge graphics, lowest settings) :| Maybe it's more important with more graphical bling? Maybe it just removed a ton of dead vertex data :?
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

Ill try it on my trusty 8800GT. Maybe NVTriStrip was made for nvidia cache architecture?
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6240
Joined: 29 Apr 2005, 01:14

Re: Python S3O module and optimizer

Post by FLOZi »

Stuttering on /give all has other contributing factors like texture loading
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

Beherith wrote:Ill try it on my trusty 8800GT. Maybe NVTriStrip was made for nvidia cache architecture?
The remapping algorithm in there is not nVidia-specific. I didn't even know my remapping algorithm was the same as NVTriStrip's until today!
FLOZi wrote:Stuttering on /give all has other contributing factors like texture loading
I measured the framerate after loading completely completed. (Went from 12 to about 30 FPS.)
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6240
Joined: 29 Apr 2005, 01:14

Re: Python S3O module and optimizer

Post by FLOZi »

Regarding tri-stripping, you've probably already seen it, but:

http://users.telenet.be/tfautre/softdev ... istrip.htm
User avatar
Beherith
Posts: 5145
Joined: 26 Oct 2007, 16:21

Re: Python S3O module and optimizer

Post by Beherith »

Just a side note on triangle stripping: 99% of spring's s3o's are using triangle only primitives. This is because they dont use quads if there is a single non quad present, and because upspring doesnt make or support triangle strips.

This can result in troubles, namely if one optimizes to triangle strips, then later has to edit the model - he must reconvert to an un-triangle-stripped version to edit it in upspring (for such things as changing radius, pieces, empty pieces, changing texture; so this can happen a lot in a mod under development)
Muon
Posts: 24
Joined: 23 Jan 2012, 21:42

Re: Python S3O module and optimizer

Post by Muon »

Well, as I said, I don't do triangle stripping yet. The models will be 100% Upspring compatible after optimization. I looked at SS3OPiece::DrawForList() and seems that it's only able to restart triangle strips if the NV_primitive_restart extension is present, so I don't think that Spring would properly render strips without explicit degenerate triangles at all!
Post Reply

Return to “Engine”