Meeting minutes 2010-09-19

[zerver]

Present: abma, kloot, tobi, zerver


<[RoX]Tobi> suppose we can finish quite fast this time :)



==Release plan==
<[RoX]Tobi> needs jK probably, to know about the unitsync bug
<zerver> yeah



== UML colab. tool (test Inkscape, Dia, ArgoUML till next week) ==
<[RoX]Tobi> personally I didn't get around to anything Spring related last week, also didn't test UML tools
<[RoX]Tobi> did you try any?
<zerver> no
<Kloot> I just looked at dia again and found that it does not have a .uml export option
<zerver> i have some experience with uml, but none of these tools
<Kloot> dunno if that's a crucial point :)
<zerver> question is if we would actually use it...
<Kloot> maybe if multiple people are going to work on the same refactor
<[RoX]Tobi> it's useful sometimes to explain some overview
<zerver> if it needs uml to begin with, maybe the design is too complex :)
<[RoX]Tobi> I don't think it will work as long term documentation thing
<[RoX]Tobi> because that would just get out of date pretty soon
<zerver> indeed, i suggest try it if you have issues working together
<[RoX]Tobi> nah, uml is just a handy standard so at least everyone draws sort of the same shapes when he needs to explain a class diagram (or other UML-able things)
<[RoX]Tobi> yup
<zerver> what i meant is, if you need to draw at all, maybe the design is too complex
<zerver> in a larger team, i understand it is 100% necessary though
<Kloot> still visually parsing some boxes and arrows is faster than scanning through code to get an idea of the structure
<zerver> yes
<[RoX]Tobi> yeah
<[RoX]Tobi> anyway, I guess only when more of us are present we can finalise this, so we can leave it till next week too
<zerver> but to make any kind of uml as docs is fail like tobi said
<Kloot> and hopefully the diagrams for springwill always fit on a single screen
<zerver> it gets outdated and confusing
<zerver> so we try it when the need arises and not before?
<Kloot> yeah outdated docs are always a problem ofc
<[RoX]Tobi> yeah I agree with that too
<Kloot> +1 too, no point creating a need

Conclusions:
* UML as docs is major fail since it gets outdated quickly
* Testing UML collab before there is a real need seems like a waste of time



== dram/sim separation ==
<zerver> just want to hear if u had any progress with this
<zerver> i mean the triple buffered render data and stuff
<zerver> i've redesigned LuaHandle so it has two lua_States now
<zerver> it picks state at entry depending on which thread it is
<zerver> and i implemented some hacks to make sure single threaded does the same, even though it has only one thread
<[RoX]Tobi> the prototype?
<zerver> yeah
<zerver> i guess that is quite big job
<zerver> but if you have done anything... or i can give it a try
<[RoX]Tobi> not much I think, I recall I spent some time on it after last meeting before my holidays and before going on holidays, but definitely didn't finish anything yet
<[RoX]Tobi> didn't really think it was still needed to continue on it
<zerver> what was needed?
<[RoX]Tobi> to continue on the prototype
<zerver> sry im kinda lost, what do you mean
<[RoX]Tobi> I just thought the prototype wasn't needed anymore, since I didn't hear much about this separation in last meetings
<zerver> oh
<zerver> this means i can take over
<zerver> i will aim for that flat pointerless structure though, unless you have found other magical solution
<[RoX]Tobi> I was still working on that :)
<[RoX]Tobi> basically I kept the multiple buffers for each object
<zerver> ok, that could work
<[RoX]Tobi> so that things could still have pointers and normal OO
<[RoX]Tobi> idea was to make getters or so that somehow "guess" from which buffer they need to return data
<zerver> buffers must be entirely separated from the object itself though, so it can be deleted but the buffers remain
<[RoX]Tobi> that could be implemented by just not deleting the object immediately, but only marking it as deleted
<[RoX]Tobi> and really deleting it later on when it's sure no threads could still need it
<zerver> ok, that means deletion must be unsynced
<zerver> currently it is very synced
<[RoX]Tobi> yeah
<[RoX]Tobi> but that's just because synced things are in the dtor
<zerver> ya
<[RoX]Tobi> if those are moved in a Destroy() function or so, then the semantics of that could be that after Destroy(), the object "becomes" unsynced, and no synced object shall still have a reference to the object
<[RoX]Tobi> but the memory would be kept alive by some unsynced component that handles delay-deleting it
<zerver> yeah, if you done any coding related to this already, give me
<[RoX]Tobi> I'll look
<zerver> otherwise i start work on this after LuaHandle is finished
<[RoX]Tobi> though iirc it was still really *really* in prototyping phase
<zerver> np, i probably should do it anyway since threading is my field
<[RoX]Tobi> I pasted it in the etherpad new structure document
<zerver> ah, will have a look

Conclusions:
* LuaHandle has been split, but it is still a prototype
* zerver takes over work on draw/sim separation possibly based on tobi's structure document



== communication enhancement with mod-devs ==
<[RoX]Tobi> I just want to remember everyone (including myself ) to write up some example questions in the etherpad
<zerver> yeah, should those questions have answers also?
<[RoX]Tobi> not (yet) needed
<Kloot> ah right
<[RoX]Tobi> might be nice to answer them after they have been posted on the forum

Conclusions:
* example questions are not ready to be published yet



== improve unitsync ==
<[AG]abma> to unitsync: i talked about making some downloading stuff as an extra lib
<[AG]abma> i wrote a prototype that implemented the current download-systems (plasma + rapid), currently it's only a console app
<[AG]abma> and i found, that these systems have some limitations
<[RoX]Tobi> oh, a c++ implementation of rapid?
<[AG]abma> plasma for example can't find all files on springfiles
<[AG]abma> yep
<[AG]abma> but its a prototype..quick and dirty
<[AG]abma> but it seems to work
<[RoX]Tobi> do you know about my python implementation?
<[AG]abma> yep
<[RoX]Tobi> ok
<[AG]abma> i contacted jj for adding a search-function, but i get no response
<[AG]abma> useing the normal search function + parse the html-stuff makes no sense i think
<[AG]abma> ehm... i got the response, that this could be added and he will send me some source, so that i can write that
<[RoX]Tobi> :)
<[AG]abma> i think, this would be best, because this would be a veryvery simple download-system... and it would allow for simple downloading mods + maps for example in the installer
<[AG]abma> plasma has some deps, that would be bad for including it into spring
<[AG]abma> thats the current status there
<[RoX]Tobi> ok
<[AG]abma> to improve unitsync in general, i would like to keep the current functions, but make it easier to use... for example remove the init function and do a lazy init
<[RoX]Tobi> I wonder if it won't stiffle innovation if breaking upgrades of those systems become coupled to spring releases
<[AG]abma> that's the plan
<[RoX]Tobi> redesign of the api for current functionality could be good
<[AG]abma> i tought about making something like the ai-interface... but to do that, a cleanup of the current api is necessary i think
<[RoX]Tobi> like the ai interface? as in, core C interface and include generated interfaces for more languages?
<[RoX]Tobi> or some other aspect of the ai interface?
<[AG]abma> c interface + maybe more languages
<[AG]abma> but in this point i'm really unsure what would be best
<[AG]abma> first comes the cleanup :)
<[AG]abma> is there currently a console app for the unitsync?
<[RoX]Tobi> unitsync interface at this moment is simple enough to access it from many languages using the language' native primitives for talking to C code
<[AG]abma> this would make testing easier
<[RoX]Tobi> just tools/unitsync/test/test.cpp
<[RoX]Tobi> but that might be a little outdated
<[AG]abma> ah, ok
<[RoX]Tobi> and definitely doesn't exercise all of it
<[AG]abma> i think i'll talk to braindamage, because he should know, what is needed there
<[AG]abma> with there, i mean the api
<[RoX]Tobi> ok
<[AG]abma> the delphi-app in unitsync/test can be removed?
<[AG]abma> last-update was 2006...
<[RoX]Tobi> yeah
<[AG]abma> ok
<[AG]abma> ok.. that was my "anything else" :)
<[AG]abma> hmm.. one thing: did someone think about regression-tests in spring?
<[RoX]Tobi> think, yes
<[RoX]Tobi> but not much more then that :)
<[AG]abma> ok
<[RoX]Tobi> it seems very hard to make good regression tests that don't require (much) manual intervention and don't fail everytime synced stuff gives slightly different results or some internal AI is made a bit smarter
<[RoX]Tobi> i.e. it's hard to define whether a change in behaviour from version X to version X+1 was intended and an improvement, or unintended and a bug
<[RoX]Tobi> maybe simpler things could be tested though, but those also break less
<[RoX]Tobi> otoh I have no experience with regression tests in any project.. (only unit tests and a bit integration/functional tests)
<[AG]abma> me to..
<[AG]abma> i thought about some simple semi-automatic-tests like: cheat 100 units into map, let walk them from a to b to c... if units don't reach destination in about a specific time, test failed
<[AG]abma> this thing could be added to the default scripts
<[AG]abma> just an idea
<[RoX]Tobi> yeah that might work, it's also one approach I thought about sometime
<[RoX]Tobi> in particular now with Lua making the tests could be quite easy
<[AG]abma> yep...
<[RoX]Tobi> but you need a huge amount of them to really test a significant portion of the features
<[RoX]Tobi> and I fear a lot of specific features will be very hard to test
<[RoX]Tobi> just one problem is testing the combinatorial explosion of possible unitdef field combinations... :)
<[AG]abma>
<[AG]abma> maybe mod-devs could help here too
<[RoX]Tobi> and e.g. testing walk from a to b to c is easy to implement, but I fear that is the exception. Something like testing turnradius would already become pretty messy I think...
<[AG]abma> yep, to write tests for everything is impossible
<[RoX]Tobi> though obviously fact that half of the features are not really testable isn't a good reason to not test the other half :)
<[AG]abma> hmm

Conclusions:
* abma has made a rapid implementation prototype in C++
* regression testing is interesting, has potential to reduce number of failed releases, but is difficult to realize



== next meeting ==
<zerver> next week as usual?
<[RoX]Tobi> possibly I can not be here next week same time, although I might still shuffle around my plans for that weekend
<Kloot> I don't know yet either
<zerver> should be fine here, unless i forget it
<[RoX]Tobi> ok

[Beherith]

Im wondering if this would count as some form of regression testing:
I run latest git spring (while my pc is idle) with some stable bots in repeat, on a number of maps.
If it crashes, I generate a report, and run again.

Would there be interest or would it be a valid idea to perform this kind of testing?

[aegis]

if we setup something automatic to do this with headless spring and allowed people to opt in, devs might appreciate it

[zwzsg]

You could also use the "Fusion Farm" mission from my BASP mutator: It plays a fairly large battle, with most of BA unit types, for hours, without human interaction.

[Licho]

Note, plasma is not supposed to find all files on springfiles.
File must first be registered using register webservice method, only springdownloader does it.

Plasma server needs data like piece hashes, minimap, modoptions, dependencies to be of any use. Also it critically needs spring hash of the file which still changes every spring version

For info how it works contanct me directly. I wonder how did you get info about those systems, because AFAIK its not properly documented. Like streamer for rapid or what is metadata format in Plasma.

Also in plasma server sources you can find how it looks for SF mirrors.

http://code.google.com/p/summerspring/s ... rovider.cs

Oh and I still believe its waste of time to code something no existing lobby will likely use, because they all have downloader implemented already.

[AF]

I disagree, implementing downloader would still be useful for the following reasons:

• standalone tools
• New Lobbies
• Experiments and Prototyping
• Lua Lobby
• Autohosts and bots

[Licho]

On the other hand, extremely slow development cycle + existing problems with unitsync don't give me much faith in such solution.

I dislike having anything hardcoded in engine which changes 1x in half a year. In last 1 year we changed rapid specification like 5-6x. Its just extremely unflexible to wait for engine release each time we want to change something.

Also autohosts don't need content files..

[AF]

I agree there also. Id like to see a project with this purpose, but if it can be kept outside the spring release cycle, that would be helpful. A generic libplasma anybody?

[Zydox]

Im wondering if this would count as some form of regression testing:
I run latest git spring (while my pc is idle) with some stable bots in repeat, on a number of maps.
If it crashes, I generate a report, and run again.

Would there be interest or would it be a valid idea to perform this kind of testing?

Interested in trying to build some kind of automated AI battle system?

Or perhaps bring the AI ladder to life?

[hoijui]

i see two viable solutions:

• a native library with a C API
• a Python app with command-line interface

the second would likely be an extension of rapid, and probably easier to make and maintain, while the first one would probably allow for more control by the application using the library.

in any way, it had to be cross-platform, and very easy to install. for example, it would be bad if:
"if you want to have Downloader support in SpringLobby, you have to install Python 2.7+ and toolXX, and then start a command line and write install ABC"
instead, SL team should be able to pack a few files with their installer.
(not to say that SL would use it; just used as an example).

an other important part would be, that the lobby devs and other devs that would possibly use the lib can actively take part in the enhancement of the basic library and the extension of the DL protocols. it helps little if we have a library that could be used by everyone, but is used by no one, cause they have no control over it, and have to rely on someone else fixing it when their lobby fails downloading. same if they want to add a feature (eg, add more meta data to certain file types), but some lone protocol master guy says no.

[Licho]

Well SD code contains "PlasmaDownloader.dll" - fully multi-platform library which has benefit of being constantly updated and fixed and implements all features of the system.

[hoijui]

apart form requiring mono (how would a native linux app use that dll?), it still would leave the whole thing totally out of control of other devs. it has nothing of what is required except so-so cross-platform. practice proves me right in that it does not serve to solve what is trying to be solved here.

[AF]

There is a large chunk of the build process for that file which is hidden inside the mysterious block/step called 'acquire mono'

[Licho]

Native linux app could run it as executable if needed.
Out of control of other devs? Its all open source repositories, are you claiming nobody except me can handle a bit of C#?

It has nothing of what is required? What you mean here? :) It has full specification.

Question is what is the goal.
All lobbies have downloader already. If its for experiment and tools then its fine for them to use mono. If its for lua lobby - well it will be the most advanced part of it for some time i guess.

[hoijui]

it misses:
* real cross-platform usability
* an abstract API that allows to use rapid and plasma + possibly other systems

there is simply no point in arguing about this. we all know that nobody would ever use that, except someone writing a C# lobby. an other reason not to argue is, that a C/C++ lib can still be done of course.
the problem there is only: what will happen if the author of/a dev using the C++ lib would want to change one of the protocols?
then again, in the worst case, other master servers (supporting the different protocol) could be used. so no limitations! :D
hero-in

[AF]

All major lobby projects have support, also remember, lobbies are not the only kind of program on the lobby server scene.

edited for clarification

[hoijui]

hae?
no idea what you want to say there.

[Licho]

* Real abstract api? Have you actually looked into code? It has abstract api. It supports any kind of download systems, and basic interface is incredibly simple. You just tell it type (can be "unknown") and spring name of resource to get and it gets it (any way it sees fit, including dependencies, checking local stuff etc).
It also has 2 different interfaces to monitor progress in real time or order new downloads, one is incredibly simple text file based.

* real cross-platform usability? The lib works on linux all the time. Springies are using it for example. If you suggest python, .NET is same but does not need any special installation on recent windows.

Meh ok, so I guess that means you just don't want to cooperate with existing systems and you don't want to reuse existing code.

[AF]

No he just doesn't want to use Mono, mono instils a slew of limitations which you would naturally say are irrelevant because they are indeed irrelevant to the current mainstream lobbies, but at the same time are not irrelevant to various projects and ideas that could arise

What you could do instead is educate. How would I include a mono library in my projects? Or what if I do not want to add Mono as a dependency? What if I do not know a shred of C# and want to change things?

I for one would very much like it if I avoided the DLL boundary altogether if possible and integrated the library directly into my projects, but that isnt possible unless I'm running on .Net/Mono

Of course I could just release an objective C version and use all the same arguements and you would all be up in arms about it.

[Licho]

For mono stuff to run on linux you need mono installed in system. But that should not be a problem, ubuntu for example comes with it installed already.

There are several ways to use mono more directly:

* COM interface - this technology allows instantiation and use of classes/interfaces across different disparate technologies. Objects run in their own "space" so you don't have to depend directly on mono/.net to use it. Im not sure if its implemented in linux, because its windows technology.
* Simply running assembly through mono and talking to it using other interface. Mono itself has no problem executing native code and linking to native libs or sharing unmanaged structures.
* Mono wrapper - you can include native libs which allow you to run and use mono code directly. This is non-trivial option.

It does not sound simple, though actually COM is very simple if it works on linux. But making properly portable C++ library which builds easily isn't very simple either - especially if you depend on lots of other libs. Mono at least has benefit that all you need is monodevelop and it compiles windows based project without issues.