Python S3O module and optimizer

[Muon]

I wrote a Python S3O model parsing/serialization module, and a simple model optimizer based on it!

https://github.com/Muon/s3o-tools

The optimizer removes redundant data and reindexes the mesh. It does miracles on poorly exported models with bad index tables (looking at you, Kernel Panic!). I've tested it on KP's model set: it reduced the size of the model data by about half - from 737409 bytes to 430561 bytes. There should also be a performance boost because of the indexing but I was too lazy to do any real measurement I'll probably add a tri-stripper later.

[Beherith]

Damn nice stuff, way cleaner and better than the one I wrote. Just so you know, upspring doesn't handle triangle stripped pieces, while the engine theoretically does (I haven't tested it)

Upspring has a command called 'optimize', I think it does the same thing as your code - it removed redundant vertices. The performance gain from this is noticeable.

[Muon]

One of the S44 devs just ran it on the mod's models, which were already Upspring-optimized and it cut them down to size nicely. I ran it myself too, and the reduction is significant (http://pastebin.com/GVyQAq3C). Hmm, Upspring doesn't sound nice. And that fact I couldn't compile it didn't help its case.

[FLOZi]

* Can't guarantee that the S44 models have had Upspring's optimize run on them, I was commenting on the software's general lack of redeeming qualities.

FWIW Behe, your Python s3o parser is much nicer than mine.


Next up Muon, write an ASSIMP-loading model assembling tool to replace Upspring that is not Blender.

[jK]

I think I know why you reduced even already optimized meshes:
You forgot the normals & texcoords! Vertices can have the same vertexcoords but different normals & texcoords, if so they need to be added separately to the vertexbuffer.

[Beherith]

I love upspring, its one of the best tools around - sure it has UI oddities, but what software doesnt? Besides that, its reliable and fast to work in.

[FLOZi]

In what quantum reality is Upspring reliable?

[Muon]

I think I know why you reduced even already optimized meshes:
You forgot the normals & texcoords! Vertices can have the same vertexcoords but different normals & texcoords, if so they need to be added separately to the vertexbuffer.

No, I do full vertex comparisons - position, normals, and texcoords. It's the most conservative optimizer on Earth

Also, updated: forgot to rename optimize_piece() to recursively_optimize_pieces() when I cleaned up the code so the optimizer only went 2 levels deep (root and immediate children).

[Beherith]

Ive put over 100 units ingame with upspring, didnt crash often, its use was deterministic. Sure I've run into quirks, but they were extremely minor.

[Muon]

Okay! So, with the latest fix, the optimizer reduces Market Garden's objects3d from 57540187 bytes to 36098843 bytes - a whopping 20.4 MB reduction. And everything works!

[FLOZi]

Ive put over 100 units ingame with upspring, didnt crash often, its use was deterministic. Sure I've run into quirks, but they were extremely minor.

Maybe i was spoilt by the luxury of 3do Builder 2.02 (not 2.1, that was pants too)

Muon; Perhaps a message at the end of processing several files with the total change in Mb?

[Muon]

It is DONE!

Also, fun fact: I just noticed my remapping algorithm also improves the spatial locality of the vertex data in exactly the same way as NvTriStrip's RemapIndices() function.

[Beherith]

Wonderful! Ill test how this works with BAR. Thanks for your effort!

Did you do a performance test? Try it with /cheat then /give 500 gertiger and compare FPS.

[Muon]

No, not a really real one - don't have time today. But /cheat and /give all was so much smoother after I ran my optimizer on everything. It didn't stutter at all at normal zoom! (As opposed to much stuttering pre-optimization.)

EDIT: Went and did it anyway. On 500 tanks it went from 15 to 16 FPS (Linux, Sandy Bridge graphics, lowest settings) Maybe it's more important with more graphical bling? Maybe it just removed a ton of dead vertex data

[Beherith]

Ill try it on my trusty 8800GT. Maybe NVTriStrip was made for nvidia cache architecture?

[FLOZi]

Stuttering on /give all has other contributing factors like texture loading

[Muon]

Ill try it on my trusty 8800GT. Maybe NVTriStrip was made for nvidia cache architecture?

The remapping algorithm in there is not nVidia-specific. I didn't even know my remapping algorithm was the same as NVTriStrip's until today!

Stuttering on /give all has other contributing factors like texture loading

I measured the framerate after loading completely completed. (Went from 12 to about 30 FPS.)

[FLOZi]

Regarding tri-stripping, you've probably already seen it, but:

http://users.telenet.be/tfautre/softdev ... istrip.htm

[Beherith]

Just a side note on triangle stripping: 99% of spring's s3o's are using triangle only primitives. This is because they dont use quads if there is a single non quad present, and because upspring doesnt make or support triangle strips.

This can result in troubles, namely if one optimizes to triangle strips, then later has to edit the model - he must reconvert to an un-triangle-stripped version to edit it in upspring (for such things as changing radius, pieces, empty pieces, changing texture; so this can happen a lot in a mod under development)

[Muon]

Well, as I said, I don't do triangle stripping yet. The models will be 100% Upspring compatible after optimization. I looked at SS3OPiece::DrawForList() and seems that it's only able to restart triangle strips if the NV_primitive_restart extension is present, so I don't think that Spring would properly render strips without explicit degenerate triangles at all!