Where's MT Going?

[zerver]

If mod devs can test with master branch, they can do so with any other branch.

They can, but they won't.

It is a bad idea to merge something that could mean compatibility issues into master if there is an imminent release. Better do it asap.

[SirMaverick]

If mod devs can test with master branch, they can do so with any other branch.

They can, but they won't.

Any idea why? Many already don't just test with current release (CL) but with master (M). Why won't they test with another branch (OB) if you add a new (bigger) feature there?
I don't understand why they do test CR vs. M, but won't do CL/M vs. OB.

[Tobi]

As for dual lua states, as far as I know all comments in the source about it are from trepan, who himself wanted to split the states but didn't do this before he left. I don't know what reasons for it he had.

Personally I think the state split might give slightly more consistent code - all of synced LuaRules, unsynced LuaRules and LuaUI could keep the call-ins in the global table (now LuaRules keeps it in the "register", IIRC). Besides, it might make multithreading easier. One piece of advise though, if you try to do this, complete and test the state split first, and only start making it multithreaded afterwards. Splitting the Lua states is complex enough, no reason to mix it with multithreading things right from the start.


As for where development is happening: if you change things that break / have a good chance of breaking current mods in non-trivial manner or about which discussion is still happening, put it in a branch.

When all devs are convinced it's good enough to merge, merge.

If everyone uses #ifdef..#endif instead of branches the code becomes a maintenance nightmare very soon (we have already way too much #ifdef..#endif IMO), and in the end, instead of resolving some merge conflicts, you need to manually go and clean up the code to "perform" the merge, instead of simply letting a tool do the merge.

The main problem with putting such experimental/breaking stuff in master is that it blocks everyone else from doing development, because they first need to figure out why all kinds of things broke. That definitely applies if it's broken by default, instead of compatible by default. Though even if it's compatible by default, the maintenance argument still applies.

Master should always compile and always run the latest stable of a number of major games without problems. There can be exceptions of course (like there was for the modinfo.lua change), but they should remain exceptions, as that kind of thing really hinders other development. Anything that violates this belongs in a branch.

As for the argument that modders test master and not another branch; that may be true but it does not outweigh the disadvantages above. If the experimental feature is good enough, you should be able to interest some modders to test the branch.


PS: zerver, if you want a username/password to trigger buildbot builds [of other branches] please PM me

[zerver]

I agree with that. However, since MT has so many compatibility issues already, I still don't see any reason not to apply this state split to MT.

Breaking single thread is a totally different story, and from the beginning I invited to discussion about disabling the dual states by default. Seems it didn't work though.

[jK]

Breaking single thread is a totally different story, and from the beginning I invited to discussion about disabling the dual states by default. Seems it didn't work though.

It is a bad idea to merge something that could mean compatibility issues into master if there is an imminent release. Better do it asap.

These 2 quotes are a bit contrary, aren't they?
So do you think now it is a very long term project (post 83) or a short term project?

This should be in master mainly so that mod devs who want to can see what is happening to the MT build, and fix their LUA stuff. To answer the topic of this thread, in essence.

I'd like to hear Tobi's standpoint on the dual lua states. Reading the source, it is apparent to me that it the lua state split has been suggested by other devs, for other reasons.

http://github.com/spring/spring/blob/ma ... dle.h#L251

Maybe to keep synced and unsynced totally separated? It sure has advantages.

That's exactly the reason why I don't think you are the best man for this change, you neither now how the current Spring's Lua Handles work, nor what flaws it has and trepan wanted to fix in his never commited 500kB patch.
So you totally misunderstood trepan's comment. Current LuaHandleSynced (used in LuaRules & LuaGaia) uses one LuaState for synced & unsynced. Trepan did it for easier data sharing via SYNCED & SendToUnsynced, but in the end it just made things worse, so he wanted to give the unsynced parts of those handles their own LuaStates.
The comment has nothing to do with Sim & Rendering split at all!

And the point is that when this split has made (you didn't so in your commit!) ONLY the SYNCED LuaStates in LuaHandleSynced instances would need to lock the sim thread. ALL other LuaStates (LuaUI, unsynced LuaRules, unsynced LuaGaia) could use a work queue for their events and so run parallel to the sim thread! (and those are also the most time intensive ones because they do the rendering)

[zerver]

That's exactly the reason why I don't think you are the best man for this change, you neither now how the current Spring's Lua Handles work, nor what flaws it has and trepan wanted to fix in his never commited 500kB patch.
So you totally misunderstood trepan's comment. Current LuaHandleSynced (used in LuaRules & LuaGaia) uses one LuaState for synced & unsynced. Trepan did it for easier data sharing via SYNCED & SendToUnsynced, but in the end it just made things worse, so he wanted to give the unsynced parts of those handles their own LuaStates.
The comment has nothing to do with Sim & Rendering split at all!

Ehm, that is exacly what I wrote:

Maybe to keep synced and unsynced totally separated? It sure has advantages.

Trepans mistake was probably trying to share/copy data between the states. I will do no such thing. And if his patch was 500kb, then at least I'm a better person to do this than Trepan

And the point is that when this split has made (you didn't so in your commit!) ONLY the SYNCED LuaStates in LuaHandleSynced instances would need to lock the sim thread. ALL other LuaStates (LuaUI, unsynced LuaRules, unsynced LuaGaia) could use a work queue for their events and so run parallel to the sim thread! (and those are also the most time intensive ones because they do the rendering)

My split is based on thread, so if we assume the synced lua state is only touched by the sim thread, I did in fact split into synced and unsynced. The synced states never need to lock anything, as they are completely executed in the Sim thread already.

I'm pushing a new split states version soon that allows widgets to use SendToUnsynced for communication purposes. It breaks some stuff, but only in MT so no worries.

[jK]

This discussion is just dull ...

I bring up valid arguments ... you say you know it better.
I give you alternatives ... you say there is just your way.
I try to explain you how current Lua works ... you say "don't worry".

If you continue with this, then spring-mt died for me and it just becomes a "zerver's 2-threaded BA-(without any Lua)-only spring-like engine".
And I need to start a new project from the beginning to multithread spring ...


Sorry, but it had to be said ...

[smoth]

seems to me there is too much disagreement on MT, really why is it in the main branch? would it really be too hard to do it in a branch or are you just unsure of that? Appeal to authority(I know because I am me) is not a good justification for anything and will result in a weaker product.

Personally I think it really needs to be brought up in the meeting and I would like to see tobis thoughts on it.

[zerver]

zerver's 2-threaded BA-(without any Lua)-only spring-like engine"

ROFL. Actually with twice as much Lua as the single thread version.

Moar is better.

[Tobi]

I'm pushing a new split states version soon that allows widgets to use SendToUnsynced for communication purposes. It breaks some stuff, but only in MT so no worries.

widgets?

[hoijui]

as i see it:
in the long run, there will not be an ST and an MT spring, but only one MT, that also works fine on single core/CPU machines. at least that seems to be the most sane way to go to me. therefore, doing something only in the MT version (which btw, in practice means again doing a branch with #ifdefs) is not a good way to go, if you think on the two things coming together again.. it is the same like including in right into master/all of spring.
so.. it just has to be in a branch, doing it just for MT is no suitable workaround, if there is no agreement of it being the right way. what jk said is also possible of course, but i think that would create much more problems then just doing a branch (maintenance wise mostly).
using branches is easy with git!!! (yes, really!!1111)
the only "problem" you have with a branch, are merge conflicts. but you would have the same problems otherwise too, just would they happen more spread out (every time you want to push/rebase on master), instead of when you merge (you can merge master back in at any time, to spread it as much as you want). there is really no added work, if not someone works heavily on the same part of the engine like you do in the branch, in the master branch directly. and if you do it in a branch, we make sure that all the devs know, and if they start doing heavy changes there, directly in master, it will be undone for the reason of disrespect. they would have to use a branch too.

[zerver]

widgets?

Yes, widgets.

Code: Select all

function widget:GameFrame()
  if (Spring.GetGameFrame() > 0) then
    SendToUnsynced("game_frame")
  end
end

For the sake of consistency, I prefer to treat it as synced also for widgets, even though it could be batched like I previously did http://github.com/spring/spring/commit/ ... 5997f55cfb

Initial testing results for CPU load, BA 7.19, 2x"give all", all widgets and LuaShaders on:
Single Lua state: 50% load
Dual Lua state 10% load

[KDR_11k]

I don't like that, currently SendToUnsynced is pretty much a secure connection, letting widgets send arbitrary data there would increase the risk of people cheating in order to get information they shouldn't get.

If you want to do something more useful in that direction allow unsynced gadget code to declare actions (you know, the stuff that can be bound to keys) if that's not already in.

[Tobi]

So with your changes widgets run also in 2 states? One unsynced and one unsynced?

And the first unsynced has to send stuff to the other unsynced using SendToUnsynced, because the first unsynced has the sim call-ins, but can not render, and the second unsynced doesn't have the sim call-ins, but can render?

[zerver]

Tobi: Yes

KDR: Please elaborate. Is a widget allowed to read more secret stuff in GameFrame compared to other places? If so, you could just Spring.Echo it to the console anyway, so I don't see the problem.

[Argh]

No, I think the issue is RecvFromSynced().

If you have the name of a Gadget function that sends, say, every unit's position, then you can write a Widget to spy on all players in about 10 minutes.

Perhaps what needs to happen is that RecvFromSynced needs to be divided into RecvFromGadget / RecvFromWidget.

[SpliFF]

Having just purchased a 6-core Phenon I'll be watching the progress of MT very closely. From my early impressions of this tech I can tell you that the technology is really well suited to running many small threads rather than 2-3 large ones. I'll be looking into ways Spring can further exploit this such as loading individual models in seperate threads, implementing threaded pathfinding, moving individual AIs and widgets to their own threads, ecetera.

Not trying to imply this will be easy or should happen tomorrow but I'm certain it's the mindset serious developers should be adopting towards all future projects.

With CPU speeds maxing out around 4Ghz I'm convinced we'll be seeing desktop systems with up to 100 cores within 10 years. There are already 64-core processors in the server/hpc world.

Unless of course someone breaks the "silicon barrier" or develops something entirely new and cheap in the next year of so and speeds start rising again.

I know someone heavily involved in quantum computing and I'm quite certain from the things he talks about that we won't be seeing quantum computers on our desks any time soon. Same for optical computers, not sure what happened to that but I've never even heard of one being built let alone in mass production. Organic computers could in theory run as fast as our brains but again we're not even seeing systems made up of a few neurons, let alone anything as powerful as even a simple calculator. Even then researchers are fairly certain out brains are massively parallel so organic systems would simply increase the need for parallel software rather than diminish it.

It might be worth mentioning at this point the challenges for Spring. Most devs understand this stuff but I'm writing it for those who are new to the concepts:

* Lua doesn't run threaded. It has 'coroutines' but they share a thread. You can run run multiple Lua states in seperate threads but they won't be able to share data directly and will need some sort of inter-thread communication and synchronisation.

* Other libraries are similar. OpenGL for example and probably OpenAL (audio) and several others (I don't really know off-hand).

* Threaded software becomes extremely unpredictable. The sequence of events and results can change based on factors like the load a given core is under and OS scheduling in general. Synced code can't have ANY uncertainty or the game will desync. What that means in practice is the Sim thread will be extremely difficult to make parallel.

* Some types of logic require a constant feedback of previous results (recursion). This sort of thing cannot be done in parallel, at least not without introducing unacceptable overhead.

* Similarly some operations work better with access to data from a local cache. Pushing code and data to other cores can be a serious overhead in many cases worse than maxing out a single core.

* Threaded applications are notoriously difficult to debug. You not only have to know what's happening in time, you often have to know what's happening in space (ie, in different locations at the same time). Many traditional debugging tools and methods (even logging) are often not up to the task of presenting this information clearly).

Hope that's a pretty accurate assessment and that it clarifies a few things for those new to the concepts.

[Pxtl]

In C# there are some new functions that do one obvious thing for hyperthreading:

A "ForEach" loop that doesn't loop sequentially, instead it takes all the objects in the loop and divvies them up the requested operation on them between multiple threads. Obv. this only works in cases where the cost of computing is less than the cost of making the needed data available to every thread.

Now, obv. you're not doing C#, but it's a good straightforward way to get some threading into your app.

[Argh]

Thanks for that breakdown of MT issues.

My question on this is related to data access, which seems to be a crux of all these problems.

Why can't sim just store everything twice, so that all non-sim stuff is accessing a different data pool that is binary-identical?

[SpliFF]

Because:

a.) It's expensive to move data around.
b.) You waste precious cache.
c.) If you change the data it's not identical anymore.
d.) You need more than 2 copies, you need 1 copy per thread so these problems grow with the number of processors.

The upshot of all that is you usually want to copy only what you need. That's easier said than done because separating what's needed from what isn't has it's own complexities and performance impacts. The best solution for any given problem is often going to be unique to the actual problem and not something that can be solved or discussed in such generalities.