Multithreading

[MidKnight]

Double-posting because of a change in topic:

Would it be a bad idea to, instead of changing the API fundamentally, expand it with a new set of "MT approved" functions?

That way, we could have a system that could handle multithreading effectively without sacrificing backward compatibility, and even check widgets and gadgets for compatibility, and display a "THIS PROBABLY WON'T WORK" warning if they fail!

[Cheesecan]

It's difficult to solve and there are competing ideas on the best solution. Ultimately the problem is twofold:

1.) It is difficult to multithread opengl calls. There is a translation layer developed by zerver but it appears to have high overhead under certain conditions - hence the issue with poor framerates in certain widgets.

These days most graphics computations are done on the GPU running shaders. These are massively parallel already..

I've done some tests in my own heightmapping engine which resembles spring a bit, and maybe 99% of pure opengl drawing occurred when calling gl_swap_buffer which is a gpu-side operation.

The most expensive calls are copying data from RAM to GPU vertex buffers. These won't really be speeded up by multithreading because the bus or gpu bandwidth will be saturated when in use so there's no room for more.

Multi-threading has it uses but for opengl calls? I am skeptical..

[phas]

Double-posting because of a change in topic:

Would it be a bad idea to, instead of changing the API fundamentally, expand it with a new set of "MT approved" functions?

That way, we could have a system that could handle multithreading effectively without sacrificing backward compatibility, and even check widgets and gadgets for compatibility, and display a "THIS PROBABLY WON'T WORK" warning if they fail!

Yeah, that's the point.
Now, being so easy they would have done it so probably you can't.
But maybe it's possible two offer two separate set of LUA api. "Old" and "New", which is mt-friendly.

This will of course require some extra work from engine developers, but since new scripts will be written for the new system, essentially you don't have to backports new features to the one. On the other hand, scripts and mods could be ported one by one and maybe even piece by piece inside the same mod, on a opportunity base.

At the end when there's no used piece of lua code using the old system, you can remove the old framework and live happy.
If i'm correct, something like that was done for maps format.

[smoth]

you really need to not speculate and go actually do research. Not in any *A project. You need to look at the one's who internals function strongly based on lua. The part that concerns me the most about you is the reckless attitude that you have taken up. No programmer worth his salt says something like that with no research. It is a terrible attitude.

[phas]

I'm just describing one of the various common techniques that are used in similar cases, when you have to break backward compatibility but you have a large codebase that could be painful to port to the new system.

It may not be feasible in this case, but must i to read the whole spring engine code to be morally justified by you to describe a common solution?

[smoth]

one of the various common techniques that are used in similar cases, when you have to break backward compatibility but you have a large codebase that could be painful to port to the new system.

Again, this is something that you are confused on due to lack of knowledge. The codebase of spring as an engine is polluted with a lot of hardcoded inflexible functionality that we are trying to break out and put into lua. Such functionality is many times more detrimental to project development. That is what the codebase of spring is concerned with, rendering, connectivity and GAME LOGIC. Which imo much of that should be lua-fied. loading units, maps etc all handle via lua. Yet the resources that the *A use are engine internal.

many games in this engine have taken lua to use for their feature additions and even for some rendering things like normal mapping is handled in lua for some games. we have whole ui frameworks that have been in development for years etc.

I just don't think you have really looked into what is going on and are blindly making suggestions.

It may not be feasible in this case, but must i to read the whole spring engine code to be morally justified by you to describe a common solution?

I am saying you are wasting your time until you actually start understanding what we are talking about. Spring is atypical, you cannot apply a typical solution.

don't bring a cannonball when you need a flyswatter.

*edits* just in case you felt that was me trying to attack you or something similar. I am not, I just think you should really look into the way things work before making suggestions.

[zerver]

I think this rant is superfluous because there will be no need to break Lua compatibility or provide "MT approved" functions. Most problems WRT Lua rendering will go away in 0.83, with no change needed. If there is any breaking of backwards compatibility, it will be minor.

There will be various Lua threading modes (optional) for advanced programmers who want to utilize multiple Lua states for some added performance, at the cost of higher complexity.

Like now, the Lua programmer will have to think a little in terms of MT to make the code fully stable.

[FLOZi]

I think this rant is superfluous because there will be no need to break Lua compatibility or provide "MT approved" functions. Most problems WRT Lua rendering will go away in 0.83, with no change needed. If there is any breaking of backwards compatibility, it will be minor.

There will be various Lua threading modes (optional) for advanced programmers who want to utilize multiple Lua states for some added performance, at the cost of higher complexity.

Good stuff

Like now, the Lua programmer will have to think a little in terms of MT to make the code fully stable.

Clear guidelines would be useful.

[smoth]

I think this rant is superfluous because there will be no need to break Lua compatibility or provide "MT approved" functions. Most problems WRT Lua rendering will go away in 0.83, with no change needed. If there is any breaking of backwards compatibility, it will be minor.

I figured as much that is why I pointed out way earlier that you are still working on it.

[SpliFF]

There will be various Lua threading modes (optional) for advanced programmers who want to utilize multiple Lua states for some added performance, at the cost of higher complexity.

Fantastic. The ability to spawn and join threads is one of the last things Supreme Commanders' API provided that Spring does not. I have a widget in development that hits the CPU hard and would benefit greatly from a threaded implementation.

[MidKnight]

Wonderful, zerver! You're a hero!

[zerver]

Fantastic. The ability to spawn and join threads is one of the last things Supreme Commanders' API provided that Spring does not. I have a widget in development that hits the CPU hard and would benefit greatly from a threaded implementation.

Well, what I meant was the ability to have one or two states for widgets and/or gadgets. This means you can have parallel execution, but only with two threads (sim/draw). But I suppose it could be expanded to allow additional threads as well.

[jK]

Fantastic. The ability to spawn and join threads is one of the last things Supreme Commanders' API provided that Spring does not. I have a widget in development that hits the CPU hard and would benefit greatly from a threaded implementation.

Well, what I meant was the ability to have one or two states for widgets and/or gadgets. This means you can have parallel execution, but only with two threads (sim/draw). But I suppose it could be expanded to allow additional threads as well.

Please don't recycle your idea, if you want parallel execution in Lua then add LuaLanes.

[SpliFF]

+1 for LuaLanes. It looks like a good model for threading and has the platform support needed. It may need to be limited to unsynced code though because nobody wants to have to debug threaded desyncs.

[zerver]

+/- 0 until I actually have tested lanes. I'm suspecting it is not ideal for this specific application so for now I'm more pro a custom made interface. I'm suspecting the overhead of starting the lanes is way too big. The docs say "1000's of lanes a second" (without libs) which is not impressive and would require that lanes can be kept alive and assigned new tasks dynamically to be useful.

Threading should be available for synced and unsynced. It is ideal for making lua pathfinders etc.