SpringGrid

[hughperkins]

This is a factorization of the AILadder thread, since the project has factorized into two parts:
- SpringGrid: manages botrunners, contains the match request queue, stores the results
- AILadder: a client of SpringGrid's web service, that manages bot matches for the purposes of ranking the AIs, and providing a ladder

This thread is intended to contain updates and discussions for SpringGrid. Updates and discussions for AILadder are intended to stay in the original thread ( http://springrts.com/phpbb/viewtopic.php?f=15&t=20358 )

SpringGrid consists of:
- a website, that is lightweight in terms of cpu and memory, and can be hosted on a shard web host
- botrunners, which run Spring matches. They will use lots of cpu and memory, because they run Spring. They can be behind a firewall and a NAT, since they connect to the website and do a pull. There can be lots of botrunners, and they can be started in a fairly ad-hoc fashion, since they register themselves with the central web site

The website consists of:
- a webservice, botrunner_webservice, to which the botrunners talk
- a webservice, api_webservice, to which clients can talk, ie other websites, or other applications. The api_webservice provides functionality such as posting new match requests and reading the results
- web pages, for use directly through the website
- a database, anything compatibel with SQLAlchemy

Technologies used:
- python
- Jinja2 templates
- SQLAlchemy ORM
- xmlrpc

Staging website:

http://manageddreams.com/springgridstag ... runners.py

Sourcecode:

http://github.com/hughperkins/springgrid

Instructions to run a botrunner:

- download from http://github.com/hughperkins/springgri ... ves/master
- decompress, cd into the botrunner directory
- python botrunner.py
- enter 'http://manageddreams.com/springgridstaging' as the name of the website when requested
- enter a name to identify the botrunner on the website, and anything you like as the sharedsecret, but make sure you remember it

[hughperkins]

Lots of updates in git today:

Visible changes:
- supports downloading AIs!
-- for now this means AIs that don't need platform specific compiling
-- could be AIs in any language, but the only AIs that I think meet this requirement currently are the Java ones or Jython

Implications for using botrunner:
- when botrunner first runs, without an existing config.py or config.pyc file, it will ask you in addition:
-- is it ok to download ais? (with a big warning)
-- what is the path to the java executable? (in order to set JAVA_HOME)

How to package your AI:
Well, the website and botrunners arent updated yet, but basically
- create a directory with your AI's name
- create a subdirectory with your AI's version
- copy the AI files into this subdirectory. If it's java, it will need to be compiled to bytecode etc
- tar.bz2 the whole thing
- the downloader looks for this directory structure, and uses it to figure out which files to copy.

Changes under the hood:
- migrated to use the SQLAlchemy syntax for secondary/association tables, which makes things a lot more intuitive (no more option.option.option_name ...)

[hoijui]

nice! :D
its hard to keep up with your changes!
how about the botrunners of aegis, are they in use already, or are there still problems?

[hughperkins]

They work great. Initial tests with RAI, KAIK and AAI went through in ... well I think 35 matches or so took easily less than 10 minutes or so. I'm not sure the exact timing.

They're down at the moment because the server was rebooted, and I was figuring out the download code.

There are two immediate ways forward for getting more AIs into the system:
- get the download code written (done! today), and install Java on the netboot systems (I think aegis is looking at this)
- get the download code written, and add in compilation code.

Related to the second option, and also augmenting security considerably, I'm considering to what extent we could run spring in a vm, eg VirtualBox or similar. The security advantage is obvious. In addition:
- it would standardize the environment, so devs could build their own AIs for that environment

VMs are not notoriously fast, but generally I feel that the rate-limiting step in VMs that makes them slow is graphics. In our case we're headless so it doesn't really apply.

I haven't started to look at just how much memory, download footprint and so on would be associated with a VM instance.

I'm kind of imagining something like seti-at-home:
- when the computer has been idle for more than 90 minutes (for example, and configurable), then botrunner wakes up
- wakes up the VM, starts pumping matches through
- if someone starts using cpu etc, botrunner immediately kills the vm, and the computer becomes usable interactively again.

In the case that such a procedure is not insanely hard to do, it might encourage people to install botrunners since:
- wouldn't interfere with their normal use of the computer
- security threat of running third-party code would be much reduced

[hoijui]

yeahh :D :D cool idea.
though... when i imagine .. running spring as kind of.. a screen saver: i get back at my computer.. killing a VM is not a very cheap process, so the HD will work for an other 5 mins till the PC really gets usable again. but if you think its plausible... nothing against trying it!

Your post made me think...
no matter whether the botrunner compiles the AI or the AI dev uploads the finished one: once this whole thing is kind of stable, it would need a reference spring implementation. would you tie it to the current release version of spring?
How would you manage botrunners.. ie, what happens wiht those that connect to the grid and have an other version?
would it be an option when scheduling battles, eg. a check box: only use current spring release version, or a version selection box, or a text field to defien a regex for matching version

[hughperkins]

Yes, also, for matches that are already run, do we need to rerun them when spring version changes?

Mostly changes to spring make little difference to ai performance? The main difference is sometimes (often?), AIs do not survive the version n-1 to version n transition at all.

Without considering the issue with what to do with old data, data from old versions of spring, I guess my gut feeling is that the spring that is running should be the same as whatever the lobby is asking people to run?

It might be appropriate to store the version of spring used along with the results of each match, then later on, those using the results can decide whether or not to use the old data.

Edit: for the waking up, could also maybe just give it an alarm? ie it shuts down the vm at 7am each morning for example. Maybe a range of times for which to start the vm?

Edit2: oh I see what youre saying: as soon as the version of spring changes, all the AIs that are not Java (and maybe even the Java ones, who knows?) will break. I guess that.... maybe the AIs that are uploaded are recorded as being for whatever is the current version of spring at the time.

When a new version of Spring comes out, the slate of eligible AIs is wiped clean, until:
- the AIs that came with the new version of Spring are added back in automatically
- AI devs upload new versions of their AIs
- tentatively, I feel maybe the bytecode and interpreted AIs might run even with the new version, so maybe keep those active across spring versions.

[aegis]

killing a VM is not a very cheap process

yes it is, my netboot vm shuts down in seconds.

[aegis]

They're down at the moment because the server was rebooted, and I was figuring out the download code.

sorry, power outage... the servers automatically restart, but my computer doesn't, and it's not on a ups.

install Java on the netboot systems (I think aegis is looking at this)

I did that when you originally asked

- if someone starts using cpu etc, botrunner immediately kills the vm, and the computer becomes usable interactively again.

In the case that such a procedure is not insanely hard to do, it might encourage people to install botrunners since:
- wouldn't interfere with their normal use of the computer
- security threat of running third-party code would be much reduced

I'd support this and could possibly provide disk images and premade appliances for a variety of vm hosts.
giving the instance ultra-low process priority would probably actually work fairly well, allowing any other processes to continue uninterrupted.

edit: timed my netbooted vm, it takes approx 15 seconds to turn on and 3 seconds to turn off.

[hughperkins]

install Java on the netboot systems (I think aegis is looking at this)

I did that when you originally asked

Yes, both java and javac are available now. Awesome!

edit: timed my netbooted vm, it takes approx 15 seconds to turn on and 3 seconds to turn off.

:-DDDDD

Still, I think most desktop pcs are just a smidgeon less powerful than your "I can run a complete bot match in 20 seconds" computational neutron stars!

[hughperkins]

Update: note that I've changed the admin password on http://manageddreams.com/springgridstaging . Anyone who needs an account please pm me, with desire username and initial password, and I'll look at setting you up.

I've done this since botrunners are now capable of downloading native code, so I'd like adding AIs to need authentication from now on.

The 'reloaddb.cgi' script no longer exists, so it is no longer possible to reset the admin pass back to admin by simply calling this script

[aegis]

edit: timed my netbooted vm, it takes approx 15 seconds to turn on and 3 seconds to turn off.

Still, I think most desktop pcs are just a smidgeon less powerful than your "I can run a complete bot match in 20 seconds" computational neutron stars!

this was a virtual machine running on my computer, even if the other host had 1/3 the computing power, it'd take three to ten seconds to turn off (I'm sure it's i/o bound since it's network-booted over 100mbit)

[hughperkins]

Updates today, to git and staging:
- AIInfo.lua is checked before installing an AI, to make sure there is no mismatch between the directory names and the name/version in the AIInfo.lua, to avoid having versioning conflicts and so on

A few updates behind the scenes, but nothing that anyone interested can't read from the git log

Guys, I'm thinking of getting a job, so, rather than just disappearing one day what things might be appropriate to do / document as a priority before then?

That doesn't mean I wont maintain it once I have a job, but I'm unlikely to devote hours daily on it at that time

I mean, I might not get a job, but there's a chance

[hughperkins]

Update: there is an issue with the jvm opening a debugging port on 7777 at the moment, which doesn't necessarily close if spring doesn't shut down cleanly (which is fairly often ), and therefore conflicts fatally (for Spring) at the next spring startup.

I'm just checking with hoijui how to shut this off (I think it is some changes in jvm.properties and interface.properties?), and with aegis to apply these changes onto his cluster.

[hughperkins]

Started to play with VirtualBox. Its awesome! Really easy to control from the commandline. It can run headlessly (I havent tested, but it has a VBoxHeadless command, so I'm going to guess it runs ok also aegis says he's used it headlessly and it works great).

My plan is:
- create a minimal system that is capable of running spring headlessly
- create a host share called 'spring' in which we will dump spring headless, and maps, and mods (so they don't have to be in the image)
- create a host share called 'exec' into which we will dump botrunner

Then, when the vm starts up, the ubuntu insiide the vm executes exec/bootstrap.py, which will then do a git fetch on the botrunner, then run the botrunner (something like that).

Since the maps and mods are outside of the guest vm, they will not have to be in the vm image, and also, it makes it easy to share maps and mods between the guest vm for the botrunner, and the normal spring in normal usage.

[hoijui]

cool idea!

[hughperkins]

Created working VirtualBox image

Ok, I started playing with using VirtualBox. And I have now in my possession a tar.bz2 file that is 197megs that contains a ubuntu virtualbox image:
- ubuntu 9.04 minimal
- plus git-core (so we can use git)
- plus binutils, gcc, make
- plus virtualbox guest additions (so we can use virtualbox shared folders)
- plus libboost-[thread|system|program_options|signals|regexe]1.37.0, libglew1.5, liddevil1c2, libxcursor1, libfreetype6, libvorbisfile3, libogg0, libopenal1 (dependencies of spring-headlessstubs)
- plus openjdk-6-jre-headless (so we can run Java AIs)

spring-headlessstubs has run successfully from this image.

Instructions to make spring-headlessstubs run:

Code: Select all

# create vm and configure it:
VBoxManager createvm botrunner --register
VBoxManager modifyvm botrunner --memory 512MB
VBoxManager modifyvm botrunner --nic1 nat

# add hard drive image:
# [register the hard-drive image file somehow, haven't tried doing that yet ;-)], then:
VBoxManager modifyvm botrunner -hda [harddrive image registration uuid]

# add shared folders:
VBoxManager sharedfolder add botrunner --name exec --hostpath [path to your botrunner directory]
VBoxManager sharedfolder add botrunner --name spring --hostpath [path to directory with spring-headlessstubs and libunitsync.so in it]
VBoxManager sharedfolder add botrunner --name springdata --hostpath [path to directory containing maps and mods diretories]

# start vm
VBoxManager -startvm botrunner

In the VM that starts, you can do:

Code: Select all

(login as username user password springrts)
./domounts.sh

... which should mount /media/exec, /media/spring and /media/springdata , without needing a password, since it's included in /etc/visudo

/media/springdata is already included in the vm's .springrc as the spring data directory.

So, to run a game in spring-headlessstubs, in the vm do:

cd /media/spring
./spring-headleessstubs nameofsomespringstartscript.txt

You'll need the image, which I haven't uploaded yet I probably can't upload it until tonight, or maybe rsync can handle uploading it in chunks during the day?

You'll also need spring-headlessstubs, compiled for Ubuntu, which can be provided separately, and is therefore easy to update. It's just a normal Ubuntu Jaunty 32-bit build, nothing special about it. I did not rebuild it inside this image, just used the same build as on my machine.

When I built spring-headlessstubs, I used ccmake . and changed the prefix, the datadirectory, the game directrory, and basically every output directory to a 'game' subdirectory of the git spring directory, and built into that directory, and provided that 'game' subdirectory as the /media/spring directory that is used by the vm.

At this point, the VM does not automatically run stuff at start up, but I feel one of the hardest bits has been achieved here. One way to control the VM perhaps could be to install openssh-server into the image, and just control the image from ssh? The vm is behind a nat on the local machine, so its basically protected from the outside world, so opening an ssh port on it should be fine.

By keeping the maps, mods, spring executable and botrunner outside of the image, we:
- make the image easy to maintain, since we basically don't need to maintain it!
- make it easy to share maps and mods between the local computer and the vm
- make it easy to update spring-headlessstubs to the newest version
- make it easy to update botrunner as needed

These directories can be mounted as read-only, and are so by default if you follow the instructions above.

The VM starts *really quickly*, faster than spring startup I feel, so we can simply make the VM itself run in some read-only mode, and reset the VM between games.

Appendix: procedure to re-create the same image

The procedure to make the image was:
- download ubuntu minimal (about 12 megs...)
- created a vm, with 512 megs of ram, nic1 set to 'nat' and a vdi hard drive of 2 gigabytes
- installed ubuntu minimal to the vm, username user, password springrts

install git
- install git-core, by doing 'sudo apt-get install git-core'

install guest additions
- install the following packages (again 'sudo apt-get install [packagename]'):
-- binutils
-- make
-- linux-headers-generic
-- gcc
- install guest additions, by selecting this from the virtualbox menu, then 'mkdir /tmp/mnt; sudo mount /dev/cdrom /tmp/mnt; cd /tmp/mnt; ./[virtualbox run for 86, cant remember the name]; cd ; sudo umount /tmp/mnt'
- 'sudo apt-get purge linux-headers-generic'
- 'sudo apt-get clean'

install spring-headlessstubs dependencies

- libopenal1
- libogg0
- libvorbisfile3 (implies libvorbis0)
- libfreetype6
- libxcursor1 (246kb)
- libboost-regex1.37.0 (17MB)
- libboost-thread1.37.0 (1.5MB)
- libboost-program-options1.37.0(1.7MB)
- libboost-system1.37.0 (1.5MB)
- libboost-signals1.37.0 (1.5MB)
- libglew1.5 (2.8MB)
- libdevil1c2 (6.2MB) (implies libsdl, lots of sound, image....)

-> at this point, spring-headlessstubs will run far enough to complain about lack of maps
(209MB increase in host disk usage for this)

Then I set up the three shares describe above:
- /media/exec -> botrunner directory of springgrid git
- /media/spring -> game directory of springheadless git build, containing spring-headlessstubs and libunitsync.so
- /media/springdata -> directory containing maps and mods that you want to use in the vm

changed datadir in .springrc to point to /media/springdata

-> at thhis point, springheadlessstubs will run a game between NullAI fine

install java

- install openjdk-6-jre-headless (91MB)

at this point, running a game between nulloojavaai crashed at start (assertion something about about static quad < basequads.size() )

- to fix this, I copied the .springrc from my machine, which has lots of settings set to minimal, and this error went away.
- .springrc from my machine here http://pastebin.com/f51f89842

-> at this point spring-headlessstubs could load and run a game between nulloojavaais just fine!
- also, python is already installed

Reduce image size
- I did an 'apt-get clean' to remove excess files
- I deleted the files in /var/lib/apt/lists which were about 50 megs and not needed in the image
- then, I used sfill from secure-delete to fill the slack-space with zeros so it would compress well, but I feel it would be sufficient to use dd if=/dev/zero of=/somezerofile , and stop it once its created a /somezerofile of about 500megs, then delete /somezerofile

then shut down ('sudo init 0'), and tar.bz2'd the hard drive image.

The internal size of the image, as seen by doing 'df -h' inside the vm, and looking at the '/' mount is about 600MB, which is not bad!

[hughperkins]

Thoughts on networking, communications with the vm.

Some architectures possible on the botrunner hosts so that the botrunner script runs, downloads requests, and uploads them to the website effectively are:
- vm runs NAT'd or bridged, runs the botrunner script itself
- vm runs with no network. Use a file share as a dropbox to communicate with the vm. botrunner host chats with the website, drops requests in the dropbox, botrunner vm processes the request, and drops the result back in the dropbox, for the botrunner host to upload to the website

There are various other permutations on the theme, but basically:
- in virtualbox, it seems that the host doesnt have access to nat'd vms, unless it enables port-forwarding ... which will forward the port for other hosts nearby trying to communicate too, which means the vm is not very protected. Maybe it doesnt matter?
- if the virtualbox run NAT'd, all communications appear to come from the host. if the virtualbox vm catches a spam bot, or a spam relay, the spam will appear to come from the host. not really great.
- if the virtualbox runs in bridged mode, it will need an appropriate dhcp server. not sure how well this will work over wifi and stuff
-- also, no way of blocking spam bots and stuff in the virtual box configuration; the host would need to run an effective firewall for this, and thats more work than most people will likely wish to do.

I feel that:
- in a world where I am a maintainer, the no-network and dropbox solution is superior, because it really locks down the vm: no spam bots, no spam relays, no network intrusions....
- in a world where other people may maintain the project(which will hopefully be fairly soon, though you never know...), I feel that it's an extra layer to maintain, too complicated, and the chances of the AIs having spam bots and relays inside is fairly small, since the circle of AI devs is quite small and well-known.

So, since I'm not sure, I thought I'd throw the question out and find out your opinions on these matters? Opinions as a potential maintainer? Opinions as a potential botrunner host?

[hughperkins]

I've thought a little about how the vm and the host fit in with each other.

I feel that simplest is best: we just run the botrunner in the vm. It can be started at boot via /etc/init.d most likely using a su -u, I guess?

What about security? I had been assuming that the entire vm is suspect, and should be considered to be compromised, but actually, I feel that perhaps we could treat it more like a dmz.

If we run spring as an unpriveleged user, and make sure there are no obvious exploits like the visudo script above , then spring and its ais are limited in what they can do to the vm.

Next, we can run iptables and disable all outgoing connections to 192.168.0.0/16 , 10.0.0.0/8 and so on, maybe disable all outgoing ports except 80, 443, 53, maybe a couple of others. Getting rid of 25 goes quite a long way

Sicne spring runs as an unprivileged user in the vm, it is difficult for it to modify iptables, in the absence of an escalation exploit.

We could always throw in an apparmor profile or two for good measure, but probably just making sure that spring/the ais cant become root, and that iptables locks things down quite a lot, is probably pretty good security? Good enough most likely?

[hoijui]

sweet! :D
maybe we can get this VM with the release version on the download page.also.. i did not know ther eis a headless JRE :D cool thing!

[hughperkins]

sweet! :D
maybe we can get this VM with the release version on the download page.

Ah... because no need to sign a license agreement you mean?