Where's MT Going?

[Argh]

Just thought I'd ask about the general state of things. Right now, MT doesn't allow synced Gadgets to perform OpenGL stuff, so it's not useful for what I'm doing.

Had an idea about this: instead of Lua, maybe move the map drawing over to MT, and run a silent sim that only deals with map-altering events?

Might be a better use for a second processor than Widgets, and less hassle than dealing with the Widget / Gadget divide. If map deformation wasn't a sacred cow, it wouldn't even need to do that much.

[SeanHeron]

To answer the threads topic, you'll want to read these dev minutes (specifically "The big refactor"). It's discussed now and then in later minutes as well.

No comment otherwise.

[Argh]

Ah, I see! They're thinking of building a shared memory state for both processors, so sim on P0 can write, P1, P2 etc. can read.

That would solve a whole bunch of issues

OK, well I'll just shut up about it then and hope it gets done

[Argh]

Actually, one thought did occur to me:

If you build the memory-write procedure to be accessible by multiple processors from the start, instead of aiming for just two, then you can make use of four or more. Then processes can have a method that puts them in the right thread if the machine has X processors. I know I'm probably mangling terminology here, but hopefully you guys get it.

So, in the future, sim on P0, sound on P1, unit transforms on P2, Widget OpenGL on P3, etc.

This will eat a lot of RAM, and I know that copying won't be free, but the savings of offloading all the other stuff should make up the difference.

IOW, plan it for being able to expand now. Think generically, even if that's inefficient for the 2-processor test case. Few machines sold now have < 4 processors.

Planning for 2 is fine; planning to make it expand to whatever the machine will bear up to the limits of what can be handed off, though- that's golden.

[zerver]

(1) Triple buffering of selected rendering-related unit/model data, stored in a contiguous, flat, pointerless (i was about to write pointless ) structure
(2) Separate LUA execution environments for Render/Sim
(3) Draw call-ins only allowed to access the data selected in (1)

(2) is done already (bugs are likely). This cripples much of the current Lua rendering code, since Draw and Sim can no longer access each others variables. SendToUnsynced is the only allowed communication method between the two lua environments. Communication is delayed, so more strict error handling will be needed, since sending a unit ID or such to unsynced means the unit ID may be invalid already once it arrives on the unsynced side.

Master now also allows OpenGL in Sim thread, thanks to jK.

Implementing (1) and (3) means lots of work.

[Argh]

KK, so basically, we're all on the same page here, and that explains why it's borked (from my POV, not that I mean that it's borked for everybody, since I presume that there are cases where it's a significant speedup- I'm using unitRendering, though, so I am hosed atm).

Anyhow, don't mean to intrude, I just wanted to know what is up and stuff since I haven't been around and my test seemed to indicate I won't be using it for awhile yet.

[KDR_11k]

Right now, MT doesn't allow synced Gadgets to perform OpenGL stuff, so it's not useful for what I'm doing.

Wrong. It causes a performance hit but it's still possible. For the mods I've tried the performance gain from having all three cores active more than makes up for the MT draw overhead (framerate went from ~30 to ~80 with 10% draw overhead).

[zerver]

Right now, MT doesn't allow synced Gadgets to perform OpenGL stuff, so it's not useful for what I'm doing.

• You can do it in an unsynced context in 0.82, with a performance hit depending on how much time is spent inside the draw call-in.
• In 0.83 you will be able to do it in a synced context as well, and the performance hit may be history.

[jK]

In 0.83 you will be able to do it in a synced context as well, and the performance hit may be history.

You still don't understand how Lua works in Spring ...
I really doubt that you will ever be able to finish/fix this thing, so it is really better to move these things into a branch because else it will be the hell to undo it afterwards ...

[Argh]

Wrong. It causes a performance hit but it's still possible.

Doesn't work over here. POPS is all Gadget-side, and doesn't do anything under MT, works fine in single-thread. The bigger issue is that unitRendering doesn't work at all, i.e. all my fancy Unit shaders went out window and nothing draws

In short... unless I accept a very loose definition of "working", it's not working atm

[hoijui]

..., so it is really better to move these things into a branch because else it will be the hell to undo it afterwards ...

i totally agree with this. sadly, we did not talk about this in the meeting, but ... this is really needed! stop working on this in master please!
as we already have master_fixed, i will cherry pick stuff on there up to where we are now (it excludes the Lua handle split stuff).
That it possibly be some work to merge it in later is no valid argument to not do it. That is the case with every branch, still branches are good to have, and this is a prime example for when to use a branch. using macros instead is not a valid alternative either.
I developed a branch for over a year, and i had to solve a lot of merge conflicts. it is a price we have to pay, but it is necessary.

[zerver]

Why couldn't you just set DUAL_LUA_STATES to 0 instead?

[hoijui]

you are abusing pre-processor macro stuff as a kind of branching technique. you should be using a branch. if the macro way would be a good way of doing it, no C/C++ app would have to use branches.

differences:

• branch all the work rests on the shoulders of the branch maintainer
• branch there is more uncertainty of whether it will ever be included
• branch you can adapt to changes of others when you want (merging master back in)
• branch separates two things which should be separate
• branch is the de-facto standard for doing exactly this kind of things
• allows for much easier comparison: git diff --stat master dualLuaState

if someone else would like to do an alternative way of solving the same problem (assuming it would be incompatible with your way), he could do that in an other topic branch, parallel to master. so his work is simpler, you two do not directly cause merge hell for each other, and it is easier to compare the two approaches.

[zerver]

Please tell me who will be making this alternative solution. Otherwise I will unrevert and make MT use the dual states. You can have a single state in single thread if you want.

[hoijui]

do it in a branch:

Code: Select all

git branch dualLuaStates
git checkout dualLuaStates
git revert ca83a6b4fe28331455c6bfa5b87d6330fb60ee93

then you can remove your macro stuff from that branch too. and you can push that branch to the main repo, and thus also build it with the build-bot.

it does not need an alternative solution for a branch to be right way to do it.

[zerver]

This should be in master mainly so that mod devs who want to can see what is happening to the MT build, and fix their LUA stuff. To answer the topic of this thread, in essence.

I'd like to hear Tobi's standpoint on the dual lua states. Reading the source, it is apparent to me that it the lua state split has been suggested by other devs, for other reasons.

http://github.com/spring/spring/blob/ma ... dle.h#L251

Maybe to keep synced and unsynced totally separated? It sure has advantages.

[hoijui]

thing is, we other devs want to work too, and we don't want to wait with that, and be unable to test until you are done, and all the mod devs have adjusted their mods. as i said, if you develop in a branch, you can use the buildbot too, and mod devs can see too what is going on, and test with the branch. of course it is a bit more work for you, and more work for mod devs if they want/have to maintain a separate branch of their mod (do they have to?), but the same was the case for pureint and AI devs... it is still the right way to go, cause if you do all this in master, it hinders other development.

[zerver]

Thing is, if I had made this a MT-only thing there is no way you would have reverted it. Don't use the sledgehammer when you have a shiny pocket knife.

Anyway, I'm waiting for tobi/kloot to comment on this.

[KDR_11k]

Wrong. It causes a performance hit but it's still possible.

Doesn't work over here. POPS is all Gadget-side, and doesn't do anything under MT, works fine in single-thread. The bigger issue is that unitRendering doesn't work at all, i.e. all my fancy Unit shaders went out window and nothing draws

In short... unless I accept a very loose definition of "working", it's not working atm

Fibre works perfectly. Micro Modules is only missing the jump out effect. Air Raid is perfect. Thermonuclear PONG is perfect. Kernel Panic is perfect. Cuberor is perfect.

This should be in master mainly so that mod devs who want to can see what is happening to the MT build, and fix their LUA stuff. To answer the topic of this thread, in essence.

I'd like to hear Tobi's standpoint on the dual lua states. Reading the source, it is apparent to me that it the lua state split has been suggested by other devs, for other reasons.

http://github.com/spring/spring/blob/ma ... dle.h#L251

Maybe to keep synced and unsynced totally separated? It sure has advantages.

Git makes branching really easy, git-based projects will constantly branch and merge as new features are being implemented. There's no reason not to branch, it can be merged into master at any time if that's desired.

[SirMaverick]

This should be in master mainly so that mod devs who want to can see what is happening to the MT build, and fix their LUA stuff. To answer the topic of this thread, in essence.

If mod devs can test with master branch, they can do so with any other branch.

as i said, if you develop in a branch, you can use the buildbot too, and mod devs can see too what is going on, and test with the branch. of course it is a bit more work for you, and more work for mod devs if they want/have to maintain a separate branch of their mod (do they have to?)

It's easier to maintain two mod branches than making one mod branch work with two engine branches. It only leads to more clutter as you have to catch cases all over the place.
Branches are cleaner for development.