AILadder and SpringGrid

[gajop]

Ok, so here's a small update regarding the projects AILadder and SpringGrid. For those of you who don't know what those projects are, you can take a look in these two threads:
http://springrts.com/phpbb/viewtopic.php?f=15&t=20830
http://springrts.com/phpbb/viewtopic.php?f=15&t=20358
Basically, their initial goal was to provide an automated, multiuser system to run headless spring instances of AI matches, in order to best get an idea of how good AIs are.

I've forked the two projects and they can be seen on:
http://github.com/gajop/springgrid
http://github.com/gajop/ailadder/tree/gridclient (current branch of interest is gridclient, will merge it into master when i get it cleaned up)

Currently, I've done the following:
1. Fixed some random bugs here and there.
2. Added multithreading ability to botrunner and some other minor features.
3. Added a league view ability to ailadder (you can now see leagues). The current league system is the usual football league system (or so i was told, it's been more than 10 years since i looked at those...)
4. Probably introduced some new bugs.

So... this is just a post of notice, while you could use it now to get some sort of leagues going, it's probably best to wait a week or so until you can expect a certain level of stability and usability in it. (One of the few bugs that happen when I run a couple of spring instances are crashes on all java/python AIs and AAI, amongst others).

Also, and this will have to be dealt eventually, is to get someone to host springgrid and ailadder, preferably on the same host (especially with the current ailadder <-> springgrid architecture design). I could probably set it up on amazon or something similar temporarily, but I'm not sure if I'm willing to fund it permanently (should be 20$ or so?).

The easy part is running the games, and when we do get a default springgrid site, it should be as easy as running a virtual box image - for your own sake I do not suggest running it outside of a virtual environment (then again, I haven't made an image yet...). Although I implemented a killswitch for those AAIs :) that try to use too much memory and eventually end up swapping your PC into oblivion, there are still a couple of smart AIs that get in your way (Hugh AI likes to pop GUI interfaces and error messages while spring is ran headlessly in the background).

All in all, it should be fully usable and polished in a month or so (but no promises on that exact deadline since uni starts in a week), but if you really want to try it now:
1. download them from git (git clone ...)
2. setup both ailadder and springgrid into two dirs, using
python setupwebsite.py (use mysql... i had some issues with sqlite)
3. setup a virtual dir for both ailadder and springgrid in apache... there's a howto somewhere in those two projects
4. make SURE springgrid can write in replays (i always need to do chmod a+w for the replays dir), running web diagnostics on springgrid site will tell you if it can
5. run botrunner.py from springgrid/botrunner directory, you might need to set some configs first, such as springgrid destination or whatever
6. create a league in ailadder (make sure ailadder is configured to use your springgrid)
7. queue matches in ailadder (can be done from "View league results")
8. wait ...

Oh and, if you have any requests about pretty much anything let me know, although I do have a lot of things that need to be done, it would be best if I can direct my work to what's needed.

[koshi]

Way back, BrainDamage and I talked to Hugh about interfacing the AIladder with our ladder system http://ladder.springlobby.info/. I have no recollection whatsoever what the result was, but if you want catch either of us on lobby.

[hoijui]

i would be willing to help solving problems with AAI and HughAI. i remember that HughAI supports one or more AI options that allow disabling the GUI, or maybe it was in a config file...
anyway, i have commit access to HughAI and AAI.
i'd do everything from rewriting HughAI stuff to better with with headless, over valgrind-ing AAI to simple pushing your changes.

i do not have any server, but for others that might be interested in hosting ai-ladder and the grid... how much CPU, Ram and bandwidth would it use, and how much/what kind of access would you need?

[gajop]

i would be willing to help solving problems with AAI and HughAI. i remember that HughAI supports one or more AI options that allow disabling the GUI, or maybe it was in a config file...
anyway, i have commit access to HughAI and AAI.
i'd do everything from rewriting HughAI stuff to better with with headless, over valgrind-ing AAI to simple pushing your changes.

i do not have any server, but for others that might be interested in hosting ai-ladder and the grid... how much CPU, Ram and bandwidth would it use, and how much/what kind of access would you need?

As expected from you =)
Anyhow, I haven't yet given bug reports on them since I'm not sure what's causing the bugs exactly (especially the problem with java and python AIs, including null AIs which just shouldn't have any problems).

As for CPU/Ram/bandwidth on ailadder and springgrid, while I don't have any numbers yet (something still needs to be fixed before I'll do any estimates), I can still guess that those requirements will be very low, probably negligible as both ailadder and springgrid don't have persistent processes running, they just serve requests (ailadder serves user requests and sometimes relays them to springgrid, like scheduling matches or retrieving match info; springgrid also serves botrunner requests and pings - whose use depends on the refresh time people set... but it's not something to be overly concerned with, or so i hope)

All in all I'd say the biggest cost from hosting springgrid and ailadder is to get a server that's willing and capable (mod_python enabled apache) to host it.

[gajop]

Way back, BrainDamage and I talked to Hugh about interfacing the AIladder with our ladder system http://ladder.springlobby.info/. I have no recollection whatsoever what the result was, but if you want catch either of us on lobby.

so you're the ones responsible for springlobby
well, sure, we can talk (what are you nicks or channel you're using? my nick is gajop), but as far as i see it, the only reason for interfacing it with your ladder system would be to display information from ailadder on your site - ailadder/springgrid/botrunner are not meant to be used for games with humans

[koshi]

Maybe there's a chance for you to save work by having a ladderbot present in the games you run on the grid? You could save having to keep tabs on it yourself (scoring, ranking and all that). Running your own instance of the ladder site code inside the virtual env would also be possible I guess. Then again I know nothing of the current abilities of the grid/ailadder.
Nicks would be BrainDamage or _koshi_ to be found easily on #springlobby.

[Beherith]

I have a 512 core linux cluster with 2TB ram at work, which sits idle most of the time...
If anyone would want to do some neural or other learning AI projects on that, I could be of assistance.

[slogic]

Beherith, learn AAI then and post here its brains wich can be bundled with AI :)

[hoijui]

i don't think that if you put spring and AAI as they are now on a cluster, it would have much benefit to running it an a single/dual core machine. you would need to run lots of simultaneous games and using something like GA to combine learn files after the games.

[AF]

Interesting, one of the major problems with the spring grid and AI league projects was that their aim was not to provide an AI league and spring grid, but were instead a means to an end to make hughperkins more employable.

Hence the rampant restructuring, dependencies, and feature additions, rather than the most basic primitive features that are essential for a ladder to be a ladder. Hence why he got so much criticism from me.

Having an AI ladder that actually has a ladder would be a very welcome improvement.

[gajop]

I have a 512 core linux cluster with 2TB ram at work, which sits idle most of the time...
If anyone would want to do some neural or other learning AI projects on that, I could be of assistance.

sounds fun, i guess i could have it set up to have low process usage with niceness; it already kills the game if it uses a memory greater than a set amount, which is 512MB per game by default (although there's a timeout of about 1 second so it's not a hard limit, but given that info you have about 4GB per core so it shouldn't be that much of a strain)...

i remember reading hugh's thread about those ubuntu images, i guess i could eventually set the same thing, although i fear they will still take at least a couple hundred MBs unless i work a while on reducing it

so until then i would hate to see you lose your job because some AIs decided to create subprocesses and wreak havoc, or if botrunner failed to restrain them :)

i don't think that if you put spring and AAI as they are now on a cluster, it would have much benefit to running it an a single/dual core machine. you would need to run lots of simultaneous games and using something like GA to combine learn files after the games.

don't know about AAI, but i think it might be possible in the future to get statistical information about matches from replays generated by AIs from springgrid;
the first thing that comes to mind is creating bots that at random choose build orders (to a bigger or lesser degree, either choosing things like plant types/unit types based on time or something more complicated like build orders and positioning)
with future ailadder versions it shouldn't be too hard to schedule matches for a couple of thousand matches on different maps.
each league contains numbots ^ 2 * nummaps * nummatches
let the processing begins =)

[gajop]

Interesting, one of the major problems with the spring grid and AI league projects was that their aim was not to provide an AI league and spring grid, but were instead a means to an end to make hughperkins more employable.

Hence the rampant restructuring, dependencies, and feature additions, rather than the most basic primitive features that are essential for a ladder to be a ladder. Hence why he got so much criticism from me.

Having an AI ladder that actually has a ladder would be a very welcome improvement.

well, while i did find it somewhat odd that the ladder, even a simple one like, was not implemented, it really didn't take me too many commits to get it done, although ill probably need at least a week or so before i make it user friendly

[gajop]

Oh, and a couple of things I'd like to ask for general oppinions. Assuming league involves each member of the league playing the same amount of matches against each other:
1. What kind of league scoring do you want? And should there be multiple ones, and you get to choose which one you want on demand? For now I'm using a customized football league* (3 win, 1 draw, 0 loss).
2. Should we even have real ladders like in most other PC games, games such as chess, etc? Most of those games are played by humans and so they cannot rely on having everyone play the same amount of matches. Often, as a way to solve that problem, they use things like ratings, which change based on winning or losing against other rated players. I just do not see a benefit of using that here.
3. Would there be an interest in using a virtualbox image that can just be started and expected to run the botrunner until it's turned off?
4. Anything else you'd want to be implemented? Currently, the things on my mind are:
have ailadder cache info from springgrid as to decrease latency and reduce bandwidth strain
allow easier scheduling of matches/site navigation
perhaps allow for multiple springgrids for ailadders
allow for fast league creation based on templates like "CanPlaySpring" (test if you can defeat NullAIs), inter version compare for AIs
possibly have open leagues which can have AIs added at a later time to see improvements

So all in all, not that many critical stuff, it's either a good sign that it's nearing a useful state or I'm missing something.


* Football leagues normally don't include crashes and gametimeouts, for now I'm awarding points as draw (I cannot determine who caused the crash yet).

[koshi]

SpringLadder already has flexible ranking implementations. You can change the ranking algo for a given ladder at any time. Currently implemented are a stupid example ranking algo and one based on Glicko2. LadderBots automatically score a match they witness based on events from that match. Disconnects, kicks and whatnot are taken into account.
If you plan to do same-sized team matches, the 'football league' ranking fails. Glicko is kinda ok. Best, and also allowing arbitrary sized teams, would be Trueskill.

[gajop]

I'm not so sure about that. I'll probably provide you with a reasonably easy API for xmlrpcs to query completely played leagues, and we can see how Glicko or Trueskill does on it.
One of the key things that bothers me with human-based systems is having a perception of time, learning and similar. AIs don't really learn, and especially those that start in a clean environment after each game. In fact, they are far more deterministic when it comes to outcomes between AIs than human players are.

Regarding same-sized teams, when it comes to bots, n * First_AI vs n * Second_AI would probably still work with the football system, problems might arise when mixing AIs of different teams and obviously from uneven games.

[koshi]

To be clear, I don't actually want to be able to query from SpringLadder. I was just pointing out that some code is already available (Python, btw). If you store replays, parsing matches from those thru LadderBots (something we already can do iirc) would be easier than creating an api in case anyone wants to 'replay' a league on SpringLadder.

[hoijui]

a VirtualBox image for a botrunner .. i would like that. a problem i see there though, is how to maintain it painlessly. redistributing a 1GB image each time a new spring version comes out is not a nice thing of course. same with new AI versions.

for what you mentioned about time not being important for AIs.
wouldn't it be possible to use an algorithm that does not care for time?
koshi, does your system support something like meta leagues? so for example, we would have a BA_1vs1_AI_League, BA_3vs3_AI_League, ... and all of them combined under the AI_League? would that make sense at all? :D

as i get it, the SL ladder system just collects basic data, and the way it is represented/evaluated can be chosen/changed any time later; right?

[koshi]

I don't really see how a meta-league would make sense, the individual ones having different restrictions and whatnot.
And yes, it's just match data that is stored really. You can change the ranking/match scoring at any time you like.

[AF]

Could you allow an export feature that exported a lua XML or son format with basic stats? Nothing too much, just basic probability stats, e.g. Shard 0.4 versus 0.5 what is the probability of 0.4 winning drawing or loosing, and the 3 totals, no per battle details, that can be for another API. That way I can use the data either in a Lua AI, a website, via JavaScript, or in an app

[Beherith]

Our system uses the Condor job distribution system, and as such, is quite flexible and robust.
Plus I enjoy throwing retarded amounts of computing power at silly things.