Planned rendering engine rewrite

[smoth]

Then stop posting about pops if you have no intention of doing anything meaningful with it.

[Argh]

I am doing meaningful things with it. I listed several.

I am explaining how what it does and how it operates can be applied to many different tasks.

[AF]

I am not making a patch. I do not know C++, I don't have time to drop everything and learn it, and it's more useful to just keep pushing forward with Lua that can be translated back again fairly easily.

This threads is about C++ work not lua work

I mean, hello, for the most part, Lua is just calling C++ stuff that is already in Spring.

The lua spring OGL API and the C++ OGL API are not 1:1 like for like.

Try the demos, maybe try reading the code. You guys seem to think there's a massive difference between Lua OpenGL and C++ OpenGL. There isn't, when we aren't talking about UnitRendering, and the shader and logic are both portable.

Argh, we know C++, and although I may not be an expert on the engine internals,I know enough about open gl coding in C++, and spliff and jk know a lot more and are actively neck deep in engine code, and we're all telling you that the things your talking about, are not only wrong, but utterly irrelevant to the subject at hand!

Most of us here are aware that we should unsync as much of the process as possible, and move as many rendering related tasks to the gpu. You don't need to tell us about this.

You admit yourself you haven't the C++ knowledge to even start doing any of this yourself,

shhhh don't bother with all that, I have no C++ knowledge and I don't want to bother learning it, but these are just display lists and basic geometrical transformations, here's POPS, it uses FBOs and GLSL shaders, read the documentation and if you don't agree with me, then you mustn't have read it

Ironically you have yet to offer us something to form an opinion of design wise other than the buzzwords GLSL FBO VBO, and a general truism that we should shift stuff off the cpu to the gpu.

You need to post a design that can actually be implemented, that relies on details, rather than references to existing projects or designs, so no using the word POPS, no 'display lists this', no general truisms and verbosity like 'we should shift to gpu.

Your description should be lightweight and concise. Big words and verbose sentences will not help you, and I know your a fan of taking ages to say something so you dont leave out details, but here it is actually working heavily against you, and introducing ambiguity.

[AF]

I am doing meaningful things with it. I listed several.

I am explaining how what it does and how it operates can be applied to many different tasks.

Sadly this is not what this thread is about, which si why a new thread is demanded.

[Argh]

Ironically you have yet to offer us something to form an opinion of design wise other than the buzzwords GLSL FBO VBO, and a general truism that we should shift stuff off the cpu to the gpu.

No, I told you, "hey, there's source that exists that implements a lot of this".

The lack of 1:1 between Lua OGL and C++ OGL is really not that big of a deal for the stuff in P.O.P.S.- this is not UnitRendering... see attached. I am not arguing with people who haven't actually read it, or get it.

Anyhow, that's it, I guess. I really would have been willing to help with the GLSL side if this hadn't turned into such a completely asinine experience, but I'm not interested in helping this, and will not be willing to test it.

[Kloot]

Playing devil's advocate here for a moment, from my personal point of view (ie. that of a dev with reasonable knowledge of the engine, C++, OGL, shaders, rendering techniques, etc.):

It's just that it seems to me that you can't review the whole project of a new modelrenderer, so your engine experience is limited and your OpenGL experience, too.

I think this is the central issue. You (SpliFF) yourself have said a few times you're not the most qualified person for this task. Yet you ask a question like:

One thing every project I've worked on has taught me is to be cautious about ambitious ideas unless you know how committed your teammates are to the implementation. I fully accept your Gl-foo and C-foo is superior to my hamster-style but if you've really been planning this thing for a year and you haven't done it yet then where does that leave me? With ideas I really can't implement.

Whether it leaves you with unimplementable (wrt. your level of expertise) ideas is not really relevant, this rather concerns the future of Spring. As you acknowledged, jK realizes the full scope of what a rewrite would entail better than you do (and from his post you can tell that there is more to it than creating some abstractions for model / texture / shader objects), and so presumably has good reasons to delay his plans. Also, the statements you've made over the past few days are IMHO anything but "cautious" (announcing to rewrite half of UpSpring, a format converter, the unit renderer, multi-threading said renderer from scratch, animation logic, ...), which -- together with the "how does this engine feature work" topics you've started -- to me just convey the impression you don't have the experience to judge the amount and kind of work involved objectively. Even without having seen code examples to measure your true skills by, I can tell you that

All of my planning so far involves relatively straight-forward changes to specific parts of the engine I've had time to study

is very probably an under-estimation.

It's unlikely I could really implement what you've described in a way you would be happy with.

This of course doesn't mean you need to give up on your own "v1.1" branch at all, just that you may end up spending a lot of effort for nothing if you proceed with all your plans anyway. Only you can decide if that's worth the investment you're committing yourself to.

[AF]

Ah argh washes his hands, pops example still doesnt prove the points, it just shows how far removed lua GL is from the task at hand

[zwzsg]

argh if pops is so good

It's the same problem as World Builder - you keep raving about this awesome tool you've got,

Then there's also history and experience, that taught us repeatedly that Argh sounds competent, when he has no clue what he's talking about. That he's dithyrambic about his own tools, and when you actually look at them you see a clumsy wheel reimplementation.

Hmm, sorry if that sounds mean, big kudos for nevertheless getting your stuff holding together, and for having made the Spring game that came closest to a commercial level of polish.

But please leave Spring main branch to people who are actually competent.

[SpliFF]

All of my planning so far involves relatively straight-forward changes to specific parts of the engine I've had time to study

is very probably an under-estimation.

Probably, but "relatively" is a fairly unspecific word. I am allowing myself at least 6 months to spend on this (allowing for RL and all). If I don't finish, well the code is always going to be there on github for anyone who wants to try again.

It's unlikely I could really implement what you've described in a way you would be happy with.

This of course doesn't mean you need to give up on your own "v1.1" branch at all, just that you may end up spending a lot of effort for nothing if you proceed with all your plans anyway. Only you can decide if that's worth the investment you're committing yourself to.

Yeah I get that, but then why would I complain if someone came along after my efforts and made Spring better? Really, I'm not that precious about credit here, ultimately I'll learn a lot whatever happens. Maybe my effort will encourage others.

I didn't know any Lua when I started modding DOW and Supcom either but before I left the SC community my mods were some of the most advanced (not the most finished or the most popular, just advanced in terms of code). Most of what I learnt from those projects is just as applicable to Spring. I like learning.

So yeah, if this all gets redone don't expect me to wuaaagh about it unless I have specific and legitimate concerns. By that time I'll probably know enough for those concerns to be meaningful.


On a related note any concerns about me breaking Spring are overlooking one very important fact: I don't have write access to Spring's official source code. What that means is simply I cannot break anything unless a core Spring developer adds my code to Spring. So really concerns like that are really concerns that experienced Spring developers would commit unreviewed code from an unknown developer and push it straight into a release without going through the usual internal testing, beta testing and RC testing. So ask yourself, do you really have that little faith in Spring's QA? If you do there isn't much I can do about it and my code is the least of your problems.

[zwzsg]

Writing converted models back to the mod archive wouldn't be hard but BD warns me that repacking a mod client-side, even if every client does it the same way, could cause desyncs. Writing a batch converter is MUCH easier than an assimp plugin though since it only involves a small rewrite of UpSpring to take commandline flags and read/write lua.

What? Did you geniunely considered Writing converted models back to the mod archive, repacking a mod client-side?

That's very wrong, and I am not refering to desynch. Spring has no right to automatically mess up with my mod/maps files, no matter how good its intention are.

It's okay to make a batch too to automatically process the archive to make it compliant with new format but either:
- Leave the original intact, make the copy in RAM, or in some cache folder, or with the new name.
- Make the batching an act that has to be explicitetly done by the mod author, with proper warning and disclaimer.

But either way, do not make Spring.exe autosilently repack my mod/maps archive after fiddling with them!!!

No matter the whys and hows, that is just Bad, from a moral standpoint.

[bobthedinosaur]

Agreed with Z. Lets the modders do it.

[AF]

It also poses a problem for content distribution, we would have to reupload all the files again else if host has a mod with has AFLSNDFLKN the mod on the filesite/torrent will not match despite having the same resulting content, because its been repacked and now has different files and filesizes inside.

[aegis]

new file extension for new format, allow repacking to the new file extension

[zwzsg]

new file extension for new format

Are you talking about the mod, or the model?

Because there is no need of new file extension for the mod, you can just rename the mod while keeping the file extension, like for instance turning S44Lyuban_v107.sdz to S44Lyuban_v107_Spliffed.sdz (don't forget the modinfo.lua inside it too).

And for the models, well, it's obvious they won't be called .s3o anymore since they'll instead be *.dae *.3ds *.ase *.obj *.ply *.dxf *.lwo *.stl *.ac *.mdl *.md2 *.md3 *.md5.

[AF]

=z I dont like the archive and game autorenaming either

What happened to implementing a 3do and s3o plugin for ASSIMP?

[zwzsg]

=z I dont like the archive and game autorenaming either

Would be a much, much lesser evil that auto-changing-and-repacking-under-same-name.

[SpliFF]

=z I dont like the archive and game autorenaming either

What happened to implementing a 3do and s3o plugin for ASSIMP?

I said it was a possibility, not a good idea. Its main failing, aside from being difficult, is that these plugins would quickly fall behind the main assimp release and probably not be maintained.

In addition it would mean keeping alive a format that has no purpose except to support a single game that no longer needs it. Other than UpSpring there aren't any 3D software that supports these formats and there probably never will be.

This proposal isn't much different to writing plugins for 3DS Max and Blender. They solve a short term need until the author loses interest then they become incompatible and unmaintained (assuming they ever worked well in the first place).

Converting the assets to new formats is more viable in the long term than providing half-arsed support for these formats now.

As for the "moral implications" of repacking an archive I'm afraid I just don't get it, however I'm going with Argh's suggestion of a model cache anyway (shock horror!) so it's irrelevant now anyway. If authors repack their own mods then the model cache is irrelevant too.

[SpliFF]

Well from what I've seen Spring on low-end hardware (without shader/GL2.0 support) appears to primarily be something devs do while waiting for public transport or pretending to listen in class.

When Spring does run at all it looks like you're lucky to get FPS in the double digits. Still, rather than drop support entirely this seems like a good argument for making Spring more playable on old PCs, laptops, netbooks and PDAs just with new options to make it even faster.

So as far as my plans for an engine rewrite I'm going set the bar pretty low. Rendering will be possible using:

• no unit/feature rendering (RenderUnits=0, RenderFeatures=0)
• units and features as sprites (SpriteUnits=1, SpriteFeatures=1)
• wireframe (WireframeUnits=1, WireframeFeatures=1)
• flat grey shading (UnitTextures=0, FeatureTextures=0)
• flat teamcolor shading (UnitTextures=0 and UnitTeamColor=1)
• basic diffuse textures (UnitShaders=0, FeatureShaders=0)

Not shown in the list is the option to use 'Model' in place of Unit/Feature in most options to enable/disable all usage. This would affect not just units/features but any use of a model (via lua, for UI purposes, etc...)

Deliberately missing from the list is advanced texturing using teamcolor, glow, alpha and specular textures for non-GLSL cards. This just means that if your hardware doesn't have shaders you won't get pretty effects from multiple texture passes. All the multitexturing stuff will be moved to shaders for better performance on current gen hardware. Maintaining the current FFP for multi-texturing is just too low a priority given it's just adding eye-crack to hardware that's already struggling for decent FPS.

None of this is particularly difficult, most of it is already in the engine. We already have a sprite generator for fartextures so this is just an extension of that to ensure they are always used and that they support features. So in short this proposal just boils down to making sure these options are maintained in a future rewrite and are easy to find and use.

PS. The RenderUnits=0 option may seem silly at first glance but what it really means is you're playing using unit icons only (for games that support them). I found when playing Supreme Commander that 90% of the time you're doing this anyway (strategic zoom). If you want to know what the thing under the cursor is exactly you can see it in the unit detail box when you hover the mouse over its icon.

[Argh]

Deliberately missing from the list is advanced texturing using teamcolor, glow, alpha and specular textures for non-GLSL cards. This just means that if your hardware doesn't have shaders you won't get pretty effects from multiple texture passes. All the multitexturing stuff will be moved to shaders for better performance on current gen hardware. Maintaining the current FFP for multi-texturing is just too low a priority given it's just adding eye-crack to hardware that's already struggling for decent FPS.

Better plan: don't even bother writing that at all; don't write any default rendering systems at all. Let Lua do it, from the start and by design, and something like jK's UnitRendering framework (see the shark stuff; the shark is really beside the point) could be used universally, with some sort of defaults duplicating what's currently being done with ARB in base/.

There's absolutely no point in including fixed-function stuff to be "safe" for really old hardware. It's extremely slow, and even the "safest" code in Spring's 3DO system is using ARB shaders, so really you're doing people a massive un-favor if you write slow fixed-function code as "safe" instead of just saying, "hey, we're really sorry, but your piece-of-crap Intel chipset just can't run this modern video game engine". Furthermore, if somebody absolutely wants a "safe" rendering system... if a Lua framework was used from the beginning, it'd be a cinch to add that in.

[SpliFF]

That's a pretty good suggestion. Spring's base archives can do a default implementation and mods can override as they see fit. I don't see any performance issues here since it's basically a set and forget system using displaylists, VBOs and shaders.

The rendering options will be implemented as classes so it isn't a case of rendering loops being if/elseif/elseif/else. It will just use the appropriate class based on your options so multiple renderers shouldn't have any noticeable performance implications.

Code: Select all

ModelRenderer
    |__ NullModelRenderer
    |__ WireframeModelRenderer
    |__ SpriteModelRenderer
    |__ FlatModelRenderer
    |__ LuaModelRenderer
    ...
|__ UnitRenderer
    |__ WireframeUnitRenderer
    |__ SpriteUnitRenderer
    |__ FlatUnitRenderer
    ...
|__ FeatureRenderer
    |__ WireframeFeatureRenderer
    |__ SpriteFeatureRenderer
    |__ FlatFeatureRenderer
    ...

Something like that. Lua won't run everything because the engine will need access to these classes as well but I see no reason why Lua shouldn't be able to override when the end-user allows it. At the end of the day the end-user needs full control, not mod authors. You shouldn't dictate minimum hardware from your game.