Dev meeting minutes 2010-10-03

[zerver]

Then, it feels sound to write something like _G.EXPORT.MyData instead of just _G.MyData when a gadget want to transmit MyData from synced to unsynced, and, as long you make a clear sticky about it, shouldn't cause thaaat much trouble.

Nah, you'd write EXPORT.MyData, so it's even clearer than before.

It seems this "EXPORT" thing is indeed necessary to have any chance of success at all. I'm writing the code to copy _G --> SYNCED now and _G does not only contain enormous amounts of "crap" but also cyclical tables that make my stupid copy routines hang.

If you're going to rewrite it anyway, question is if pure SendToUnsynced communication is better anyway, because that will be much more efficient.

[Argh]

SendToUnsynced would be fine, if we could send tables. Honestly, I'd prefer it that way, with GG still allowed but not preferred. It would be a lot cleaner to simply create ghost copies of synced data we need on the other side that way.

[jK]

SYNCED is easier to use because you just need to set it up once.
Yeah, it can be even faster in spring-mt, because in most cases you don't need SendToUnsynced at all. This either are reduces the locking times between those LuaStates or in the way zerver implemented his SendToUnsynced it saves a lot memory.

[Argh]

I'm cool with that, too. I just want it to be less messy than atm, where (just as one example) I was forced to use SendLuaUIMsg simply because synced Gadgets can't write to WG

But I'm concerned about the size of the table- potentially if that's used for everything, wouldn't it get really large, and won't Lua's performance start to drag simply because of making alterations to a huge table?

[jK]

*lag*

[jK]

I'm cool with that, too. I just want it to be less messy than atm, where (just as one example) I was forced to use SendLuaUIMsg simply because synced Gadgets can't write to WG

Sure you can LuaRules(synced)->SendToUnsynced->LuaRules(unsynced)->Script.LuaUI.FOOBAR->LuaUI

But I'm concerned about the size of the table- potentially if that's used for everything, wouldn't it get really large, and won't Lua's performance start to drag simply because of making alterations to a huge table?

It already does so. Atm it does this a bit on-demand, but that's not a huge diff imo.

[Argh]

LuaRules(synced)->SendToUnsynced->LuaRules(unsynced)->Script.LuaUI.FOOBAR->LuaUI

That is weird, but it's a lot less ugly than what I was doing.

But I just had a head-smacking thought anyhow. I can move the thing that I moved out to Widget-land to a separate Gadget instead, and make it fail without repercussions other than an infolog error <facepalm>

Oh wait, no, I can't do that either, I forgot why that was impossible, it's relying on other stuff. But I'll look at this again.

[Forboding Angel]

AF, what the hell does Argh's PURE site have to do with this site? Stop sidetracking! You too smoth, whether PURE's gadgetry works or not is absolutely irrelevant to this discussion.

[zwzsg]

Both were relevant replies to Argh wishes to manage Spring marketing.

LuaRules(synced)->SendToUnsynced->LuaRules(unsynced)->Script.LuaUI.FOOBAR->LuaUI

For my "dump" command, I've had to use LuaUI -> Spring.SendCommands -> LuaRules (synced) -> SendToUnsynced -> LuaRules (unsynced) -> Script.LuaUI -> LuaUI

And yeah, SYNCED has a use too. For event things, it's easier to use SendToUnsynced, but for data things, it's easier to use _G. -> SYNCED. Sure, if SYNCED wasn't there, you could workaround by using SendToUnsynced, but you would have to write a little function to receive and store data, making things even less straighforward than now. Would be ugly to burden half the gadgets with half a page of code just to transmit data from synced to unsynced. And it's a common enough problem that it deserves a solution at the engine level, well written and optimised, instead of forcing every gadget to maitain his own.

[Argh]

So... why not do it this way:

Gadgets are allowed to read and write to UNSYNCED, just as Widgets currently write to WG. Synced Gadgets may only read/write to UNSYNCED during GameFrame(). Unsynced Gadgets may read/write it whenever they operate.

Only Gadgets may read / write to SYNCED. Synced Gadget code may only read/write to SYNCED during GameFrame().

That would tidy up a lot of messes. It would give us clear security rules, get rid of the need for extraneous messaging, allow for transfer of tables, let gadgets be in charge of anything they need to be without major problems and it would give synced LuaRules a clear set of rules it would have to follow that should be MT-friendly.

It would also break a lot of code, but in a way that wouldn't be a pain in the butt to fix. Instead of SendToUnsynced-->get msg-->blah, you'd just move that code up to synced and put it within GameFrame(), and other than adding local blah = SYNCED.blah on the unsynced side ... for the vast majority of Gadgets, that would be all that would be required.

For those of us doing various baroque stuff because of the current mess, we could get rid of the vast majority at a stroke, which I don't think anybody will be crying about.

[zwzsg]

You suggested synced should be able to read what unsynced wrote.

That makes me want to insult you for being yourself. Because, either you still have no clue what synced mean, either you deceitfully avoided mentionning the awful mess of messaging that would need to be done engine wise for that to work. That you think only letting gadgets read during GameFrame would be of any help, and that you compared the situation to WG, suggest that indeed, you're doing the Argh thing again.

I mean, if you suggest the engine performing black magic to let synced read what unsynced wrote would be clean, you might as well proclaim: Let's merge the synced and unsynced parts of gadgets, woohoo no more problem!

But I heard it's against the forum rules to be disrepectful, no matter against who or why. So maybe I can try to suppress my urge to flame you for being articulate in the writing yet so wrong in the concepts, pose as a polite gentlemen interested in your proposal, and go:

Please elaborate some more on what would the engine need to do to let synced read what unsynced wrote.

For exemple, try to explain what would happen when such a gadget is run on a DSD 8v8 autohost:

Code: Select all

if (gadgetHandler:IsSyncedCode()) then

	gadget:GameFrame()
		Spring.Echo("Key:",SYNCED.kp)
	end

else

	gadget:KeyPress(key)
		SYNCED.kp=key
	end

end

Show the keypresses from the host only? Use the last written value, no matter which host? Only work when every player presses the exact same keys in every same 1/30 of second? Crash and burn into sync errors? Something else?

How intuitive such a behavior would that be to a noob gadget coder?

But then I risk you writing a essay full of impressive words but void of technical truth, and I'd have just given you credibility instead of trapping and exposing you. Yeah, better flame instead! :D

[AF]

Both were relevant replies to Argh wishes to manage Spring marketing.

And a startling insight into the reading comprehension or lack of in that he responded to mine and smoths marketing and site posts with talk on multithreading showing he didn't even bother to read them yet still managed to pull up a response to the imaginary alternate universe posts.

Anyways enough of that

[Argh]

We already have the problem of letting unsynced communicate with synced. You make it sound like I don't know what that means, but I've actually dealt with it. It certainly did involve some pretty ugly messaging made hugely inefficient by the lack of engine support.

The fact is that it's unavoidable and yet it's desireable, so why not just design it to work that way?

There aren't a lot of situations that demand this, to be sure, but there are a few. Let me enlighten you:

1. GetUnitsInView, for example, is a callout that only can operate during unsynced operations atm, because it's dealing with OpenGL issues (frustrum detection). In theory, that one operation could be opened to synced, but there are other examples as well.

2. Sometime in the future, we're probably going to want to perform operations on things like FBOs and we're going to want to inform synced Lua of the results. I know you haven't played around with computing stuff on GPUs yet, but there are reasons for wanting to be able to do things like that.

In short... you're incorrectly presuming a lot of things here.

Using GameFrame() means that it occurs when sim has mutex lock and has access to the single data source, since my proposal that sim simply write more than one place for data seems to not be the preferred solution to certain issues.

I still don't agree with that; the reading I've done on how multi-threading is supposed to work strongly imply that unless sim's workload drops a very considerable amount, then the supposed "speed benefits" of MT will be largely illusionary, as sim will still be locking up the data a lot, and unsynced will be sitting there doing nothing a lot, because it can't get access. But that's not my decision, and if we're going with the single data source model where threads all compete, then we may as well deal with communications in a more realistic way.

IOW, it may not be perfect, but it's preferable to the terrible muddle that now exists. One-way data transfer has not been terribly satisfactory, and giving synced the ability to run certain types of operations so that it's not necessary will be massively inefficient.

Lastly, because I know all too well that once you've started one of these silly arguments you won't let it drop until you've tried to attack every point, yes, there would have to be arbitration involved, where the host would be the determining actor; if we're doing GPU calcs on one device and synced doesn't agree everywhere, the host probably has to be the one whose version of reality is used.

The engine doesn't deal with this at all atm; it presumes that everybody stays in total lockstep on the synced side and unsynced can be going to hell in a handbasket. I don't think that's going to be supportable forever, largely because people will want to do stuff with shaders and then have gamecode reflect the results.

How that should work is entirely beyond my limited understanding of how Spring's communications work and the implications in terms of bandwidth, but that's a bridge that will probably need to be crossed.

I'm not envisioning a system where Gadgets ask Widgets for the time of day on a regular basis; but being able to poll where everybody's camera is and then, when that data's available, causing certain things to happen, is just one example of the stuff we can't do elegantly atm and need to have a solution for.

But for the most part, communications would remain a one-way road like it is now, simply because of speed problems if nothing else.

[zwzsg]

We already have the problem of letting unsynced communicate with synced.

I know. I'd been doing a little bit of modding on the side, you know.

I've actually dealt with it. It certainly did involve some pretty ugly messaging made hugely inefficient by the lack of engine support.

Didn't I say just two posts go I've dealt with a funnily convoluted path: For my "dump" command, I've had to use LuaUI -> Spring.SendCommands -> LuaRules (synced) -> SendToUnsynced -> LuaRules (unsynced) -> Script.LuaUI -> LuaUI ?

Oh yes, yes I did say it. So why do you presume I never needed such?

There aren't a lot of situations that demand this, to be sure, but there are a few. Let me enlighten you

1. ...

2. ...

No one ever contested we need synced <-> unsynced communication. Everybody here operate under the idea of: We need it, how do we do it? So, strawman much?

The big problem is that your solution amounts to: Use magik!

But, if you had bothered to think about the quoted part of my previous post, you'd see that either:
- Magik is sure desirable, but not really achievable.
- Any system appearing easy as magik to the user is actually powered by unintelligible super advanced technology.

If you prefer, to make synced<->unsynced Lua gadget comm as simple as a shared table, you will need an awful mess in the engine. And, worse, you can't avoid side-effect breaking the shared table illusion about, oh, roughly everytime it's used maybe. Again, refer to the quote of my previous post:

On the surface, that exemple gadget would looks simple: unsynced get the pressed key, pass it to synced. Until you realise that in multiplayer, all synced state must the same, all unsynced state can be different, which mean that no matter how you want the engine to handle it, what is read in synced isn't what is written in unsynched.
- Player A press key 1, so SYNCED.kp=1
- Player B press key 2, so SYNCED.kp=2
- The Player A & B computer's synced states are identical, so 1=2
Oops your sytem requires a breach of logic. Well, no big deal, uh?
After all if it's desirable for all your FBO and mutex and threadlock and MT and OpenGL, then let it be!
By wishful thinking of Argh, 1 should from now on be declared as equal to 2.
And indeed, if we could have 1=2, then we'd have resolved any and every problem!
So that is the solution! Why don't dev make it happen?

Or, we get serious and realise that it wouldn't really be a shared table, but some bulky system in the engine that collect all the values from the unsynced side of every computer, pick one (which? Lastest? Host?), make sure every client pick the same, then send the chosen value back to every computer accross the nework. So, an unclean, slow, and deceptive system.


So:
- The ugliness of the black magik needed engine-sided would counterbalance how clean it'd appear on the Lua side.
- Presenting as a shared table what conceptually can't even be would be dangerous, the user would think it's table, while it's a fake, an illusionary bridge over a fiery pit. Better expose the peril clearly.

Of course, this answer operate on the premises you understand what synced/unsynced means to Spring. Which I'm pretty sure you don't, at least not completly, not with its implications. Answer the quote of my previous post if you do.

Using GameFrame() means that it occurs when sim has mutex lock and has access to the single data source

Typical Argh sentence: sounds impressive, until you get to its meaning, and realise it's senseless, just a grammatically correct sentence formed with words randomly picked up from dev threads.

[Argh]

The problem with your example is that it's not what I foresee where we'd want to bother.

Please examine what I've said, not what you're presuming.

And go look up mutex locks. It's just thread scheduling for MT.

[zwzsg]

Lastly, ...

Damn, you edited while I was typing my reply.

One-way data transfer has not been terribly satisfactory

We actually already have two ways data transfer.

You said GPU of the host. Didn't you realise that's exactly why I said "autohost" in my exemple? (Assuming autohost runs headless).

The engine doesn't deal with this at all atm; it presumes that everybody stays in total lockstep on the synced side and unsynced can be going to hell in a handbasket.

Cool you do have basic understanding of what's Spring sync.

yes, there would have to be arbitration involved

That's where we disagree: You think that's a minor point, while I say it is what will turn you concept in a bigger mess than what we have now.

Oh, synchronisation between players, and synchronisation along time, too.

I don't think that's going to be supportable forever, largely because people will want to do stuff with shaders and then have gamecode reflect the results.

Sure, let's rewrite Spring from the ground up then.

The problem with your example is that it's not what I foresee where we'd want to bother.

Please examine what I've said, not what you're presuming.

Ok, all you want is to use the graphic card computing power for game altering events. Doesn't mean you can escape examining my case.

[Tobi]

1. Why are we still showing advertisements for IW on the front page? Their vaporware status as a product is why I've been unwilling to put up promotional materials for my free stuff or participate, and it still makes me angry every time I see it. They haven't even updated in 5 months; isn't it time to finally acknowledge that Spring's front-page PR has been an awful failure and is managed by the wrong people?

2. In light of 1... isn't about time we got somebody competent to handle marketing? I have proven experience in this area, and I'd be happy to do it, if I don't have to deal with a committee second-guessing me for the first month (after that, we can judge by results).

If you don't like that choice, second choices would be Forb, if he's willing to give you the time, or AF, ditto. Having Moderators who don't actually make or market games, don't have strong visual-arts skills and have no track record is just bad policy. I would like to see this reviewed. I'm not talking about rebuilding the front-page, btw- I think it's fine, structurally. Just dealing with the content and presentation.

The website isn't maintained at all at the moment. (Hasn't been for a quite long time if you ignore a few small fixes here and there.)

Personally I just try to do the minimum required to keep it up, but I have no plans to do any changes other than that.

Same applies to PR and marketing, there is simply no one who is doing this as far as I know. If you want to do it, go for it, I suppose I can give you a forum rank if you want.

As for the website, the actual code is in git: http://github.com/spring/spring-website
Instructions on how to set it up locally are included.
Content (i.e. banner and screenshots) are mostly in the forum, so if you just feel the urge to modify that then there is no need to set up a development copy of the website.

[Argh]

I'd need access rights.

I don't want to do anything major to the website's look; I think roflcopter's design is fine. I would make changes to the content and do some things to get modders re-involved with the moribund front page space and do other things to revive the marketing. If you want to hear a detailed plan, I can type up my thoughts and we can talk about it via PM.

@zxswg:

all you want is to use the graphic card computing power for game altering events. Doesn't mean you can escape examining my case

Well... it's not that solid of a presumption on my part, to be perfectly honest. I'm just looking forward here.

I think it's inevitable that people will want to do things like agent systems using a GPU in RTS games, for example, and I think that if we aren't talking about this now, and trying to resolve the synced / unsynced communications issues now, then it's going to remain one of the many things we can't do. If you've got a solution, let's hear it; my main point here is that we probably need to think about this stuff now, while people are tearing up Lua and the OpenGL state and are redefining what all this stuff is.

It'd be pretty dumb to tear everything up and break every game on the engine (for the third time in a year, no less) and make pretty heavy changes to how sim deals with the graphical world, and then not deal with this problem. It's one of those things that may bite us in the ass later.

[lurker]

Your proposal, Argh, would need the synced state of the host (or blessed user when an autohost is involved) to be mirrored to everyone before each game frame, but after the previous rendered frame. This means:
1. You can't sim and render at the same time, killing MT.
2. You have to transfer between kilobytesand many megabytes each second.
3. You can't sim and render at the same time, plus network lag. Since synced has to get to unsynced data *after* it's made consistent, 20ms ping leads to 60% or more of the time being waiting for network data instead of playing.
4. If you want to let synced run ahead while the data is wrong, to fix 3, then you have to make the entire capable of realtime rewinding somehow. At which point I will implement achron.

[smoth]

I do not want argh controlling the site. I do not for one second believe he will not abuse such power.