SpringGrid

[hughperkins]

I've uploaded the VirtualBox image. It's here:

http://manageddreams.com/ailadder/downl ... tu.tar.bz2

[hoijui]

umm.. i dont know.. did not at all think about licences...
just though that this is much more comfortable then setting it all up manually, so ppl might actually run botrunners if thye can just install VirtualBox, download this image and be ready, compared to setting it all up manually, so.. it would be better to have this then a headless spring installer or packages, or at least in addition to that.
what about license you think this coudl be a problem/solves a problem?

[hughperkins]

Ah, I thought you were talking about the openjdk-6-jre-headless package, which is not sun-licensed, but is normal GPL opensource, and doesnt show a EULA disclaimer during installation.

The normal Sun java package (sun-java6-jre) pops up a huge EULA.

The Java AI interface you wrote works with either of them, so probably the openjdk one has a big advantage here.

Looking at virtualbox, whilst the image itself can contain all sorts of things, the situation is I feel probably more complicated than a simple botrunner since:
- a simple botrunner is fairly, well, simple just needs python basically, and a couple of paths, like, where is spring? which website do you want to talk to?
- installing virtualbox is not entirely trivial: its a large, platform-specific install, involving device drivers. On linux, you need to add the user to a group, which means the user has to logout and in again I think?
- something, some script, presumably has to manage starting/stopping/setting up the vm
- and lastly, one still needs to provide the same paths as before to the whole process: where is spring's data directory? where is spring-headless directory? which website shall we connect to?

The main advantages of using virtualbox I feel are:
- security
- means that an AI can be compiled once, on Ubuntu, and work across all botrunners
- it's kind of fun to play with

[hughperkins]

contents

1. probable bug in image
2. code signing. code signing libraries for python?
3. network + ssh configuration in vm

1. probable bug in image

Note: I just discovered a rogue snapshot on my system, so the image above may be non-functional :-/ I'm playing with the Export functionality which looks like it will produce a nice well-formed complete machine ("appliance" in the manual's jargon), already configured, so then an installation script would basically need to just take care of:
- configuring the shared folders: point them at spring, botrunner code etc,
- configuring the name and shared secret of the botrunner, which website to point to (as for now)
- install spring-headlessstubs
- (and, maybe, install virtualbox itself, but since many users will be Windows ,and I'm on linux, that's not something that I can package easily. At least: I can package it, but I can't test that the package works, so it probably won't....)

2. code signing. code signing libraries for python?

By the way, looking at security, I was reading through debian's faq (because eeebuntu is switching from ubuntu to debian unstable, and I dont use eeebuntu myself, but it was enough to spur my interest to find out what is debian unstable), and I found the following interesting snippet:

http://en.wikipedia.org/wiki/Debian

"Package uploads are automatically processed to ensure that they are well-formed (all the requisite files are in place) and that the package is digitally signed by a Debian developer using OpenPGP-compatible software. All Debian developers have public keys.[41] Packages are signed to be able to reject uploads from hostile outsiders to the project, and to permit accountability in the event that a package contains a serious bug, a violation of policy, or malicious code." (highlighting added by me).

Sounds like what we need basically, and then the virtualbox just becomes an extra layer of protection. I think with all these things in place, its probably fairly secure? :
- signing of packages by ai-devs, to give additional accountability
- virtualbox, so ais can't access the host system's fs directly, at least, except for the explicitly shard folders
- internal guest os firewall preventing most outgoing connections (eg port 25 will be unlikely to be amongst the whitelist... (port 25 is email))
- botrunner inside the guest os will run as an unprivileged user

A big advantage of using signing is that then the website no longer needs to be quite so locked down, "only" the key signing/distribution process needs to be secured. In any case, the attack surface is much reduced.

Also, perhaps, as I write this, maybe botrunner hosts could be the ones to choose which devs' keys they accept? Rather than having some central controller, which tends to work not that great in an opensource environment?

I guess a key blocking point for using key signing is that python doesnt natively support it AFAIK? So, needs another library. Any suggestions on this?

3. network + ssh configuration in vm

By the way, for the network for virtualbox, at the moment I've kind of settled on activating two network connections:
- 'nat' -> provides outgoing network for the botrunner inside the vm. unfortunately this blocks all incoming connections, even from the host, so we add also:
- 'host-only' -> this allows incoming ssh connections from the host

I've also installed an ssh server, so the host can ssh into the vm. I've created a public/private key pair so its easy to connect to the vm without needing to type in passwords and stuff. (of course, everyone gets the same keypair, but since incoming network connections are blocked except from the host...)

(PS, no job yet. If anyone happens to know any vacancies in Hong Kong for a C#/Java/Python/Erlang/Haskell developer/scripter with some knowledge of equity derivatives / finance, and lots of knowledge of systems administration, please get in touch asap! cv here: http://manageddreams.com/hughperkins2009nova.pdf )

(PPS from the cv: "ÔÇ£Habitation loyer moder├®ÔÇØ. Website to deploy vmware containers to the workstations on our backup-sites, which are then provided to the Grid teams." Sounds familiar )

[aegis]

virtualbox has a "bridge" mode which makes it act like another client on the network

[BrainDamage]

Also, perhaps, as I write this, maybe botrunner hosts could be the ones to choose which devs' keys they accept? Rather than having some central controller, which tends to work not that great in an opensource environment?

use the gpg key signing infrastructure

I guess a key blocking point for using key signing is that python doesnt natively support it AFAIK? So, needs another library. Any suggestions on this?

python-pyme or python-gnupginterface

[hughperkins]

Yeah gpg looks like the standard. It's a bit disappointing that there is no native python package to handle it: they seem to be wrappers through swig and stuff. I'm tempted to just run it from the commandline, like the subprocess.Popen equivalent of:

Code: Select all

gpg --import someaidevkey
gpg --verify somefile.tar.bz2.sig somefile.tar.bz2

Edit: also, it would be nice to do the checking on the host, but I think it will be easier to install things if the checking is done on the guest vm. Thoughts?

[hughperkins]

Current short-term plans:

signing
- add a column 'key reference' (eg 'E4236F7B') to the AIs
-- this will be used to make sure that an AI download request will only be sent to botrunners that have authorized the corresponding public key
- botrunners will upload a list of supported public keys from ... the gpg keyring? the config.py?
- install gpg to the vm image
- get botrunner.py to call out to gpg to verify the signature of downloaded ais

virtualbox installation

- create a script to import the vbox appliance image, and customize the shared folders
- possibly, download and install vbox if it isn't already installed
- upload a .tar.bz2 of a spring-headlessstubs build
- in the installation script, download spring-headlessstubs, decompress it, into the appropriate shared folder location
- I suppose botrunner should be installed as a snapshot of git, rather than being git cloned, for security reasons for example
- ask the user where the directory is with the maps and mods that he/she wishes to share with the vm

virtualbox image

- create a /etc/init.d script that su -u's into a different user, then runs /media/exec/bootstrap.py

[aegis]

event.d is easier to setup than init.d

[hughperkins]

event.d is easier to setup than init.d

Ah! Excellent info!

[hughperkins]

New VirtualBox 'Appliance' image uploaded:

http://manageddreams.com/ailadder/download/test.ovf (5KB)
http://manageddreams.com/ailadder/download/ubuntu.vmdk (256MB)

What it does:
- when it starts up, it will mount the spring, springdata, springwritabledat, and exec shared folders
- it will run 'python exec/bootstrap.py' as the user 'user'
- there is an ssh server, that you can connect to on 192.168.56.101 (username user password springrts)
- it contains all the dependencies to run spring-headlessstubs, and it contains a jre)

Expected shared folder assignments:
- the image expects the following shared folder contents:
-- exec: contains 'botrunner' directory, with a new file 'bootstrap.py' (which is now present in the springgrid git repository) (for now I feel the best way to set this up is: 'git clone git://github.com/hughperkins/springgrid.git' )
-- exec/config_vbox.py should be a copy of the normal config.py, configured for vbox as follows: http://pastebin.com/f562ca4f4 (a planned installer will create this automatically in theory in the future, but doesn't exist yet) This folder should be mounted read-only.
-- spring: contains spring-headlessstubs, libunitsync.so, compiled for ubuntu jaunty, and anything else that one might consider to go with the spring executables. This folder should be mounted read-only (Edit: here is a spring-headless build that you can use as the contents of this folder: http://manageddreams.com/ailadder/downl ... ty.tar.bz2 )
-- springwritabledat : this will be written to by the vm. It will download AIs to here. You might consider not using this for anything except for VM AIs, to keep a separation of the two machines. This directory could be initially empty I think, maybe just create an empty 'AI' subdirectory
-- springdata: maps and mods directories should probably be here. This folder should be mounted read-only.

To see what is going on on the botrunner script running in the machine at startup:
- the bootstrap script is run using the 'screen' command (aegis uses this on his machines, the screen command, and it works really well, so going to copy this method )
- you can attach to the script's display from any session, ssh or console, by doing simply 'screen -r'
-- you can detach again by doing 'ctrl-a' and then 'd'

Changes under the hood:
On the vm:
- deleted domounts.sh
- moved the mounts to /etc/fstab
- removed domounts.sh from sudoers
- created /etc/init.d/botrunner_bootstrap.sh
- created /etc/event.d/botrunner_bootstrap
- installed 'screen'
- instaled 'openssh-server'
- configured nic2 as 'host-only'
- added eth1 to /etc/network/interfaces

In springgrid git master:
- created botrunner/bootstrap.py
- added commandline option '--configpath=<configpath>' to botrunner

Edit: updated link to springheadless build for build against spring 0.80.5.1.

[hughperkins]

Added a couple of bash examples of calling the api_webservice , http://github.com/hughperkins/springgri ... service.py :

http://github.com/hughperkins/springgri ... rvice/bash

The goal here is to present the 'worst case' of calling the api_webservice from another language: ie in any language with access to tcp/ip it is trivial - though tedious - to code the xmlrpc comms by hand.

The two examples presented are:

- calling 'getais' , which takes no parameters, and returns a list of {'ai_name':ai_name,'ai_version': ai_version} dictionaries
- calling 'schedulematchv1', which takes in a map_name string, a mod_name string, a list of two {'ai_name': ai_name, 'ai_version': ai_version} dictionaries (surprisingly easy to write by hand in xmlrpc), and an empty list; schedules a match, and returns a tuple of (1,'') if it succeeds or (0,errormessage) otherwise.

Edited to correct that ais are sent/received as dictionaries, not tuples.

[hughperkins]

Use OpenID for accounts?

Had a quick play and wrote a short prototype, not at all connected to the springgrid code, except it uses jinjahelper, which successfully reported it had authenticated me:

Code: Select all

#!/usr/bin/python

import os
import cgitb; cgitb.enable()
import cgi
import urlparse

from openid.consumer import consumer
from openid.store import filestore

from utils import *

def go():
   openidurl = formhelper.getValue('openid')
   if openidurl == None and formhelper.getValue('openid.claimed_id') == None:
      jinjahelper.rendertemplate('testopenid.html')
      return
   mystore = filestore.FileOpenIDStore( 'openidstore' )

   sessiondata = {}
   myconsumer = consumer.Consumer( sessiondata, mystore)

   if formhelper.getValue('openid.claimed_id') == None:
      myrequest = myconsumer.begin( openidurl, anonymous = True )

      print "Content-type: text/html"
      print "Location: " + myrequest.redirectURL('http://localhost/openid','http://localhost/openid/testopenid.py')
      print '\n\n'
      return

   _querystringdict = urlparse.parse_qs(os.getenv("QUERY_STRING"))
   querystringdict = {}
   for key in _querystringdict.keys():
      querystringdict[key] = _querystringdict[key][0]

   result = myconsumer.complete(querystringdict, 'http://localhost/openid/testopenid.py')

   jinjahelper.message('done ' + str(result) )

go()

[AF]

hmm, I have a presentation due early december, and Id like to have something I can show through this, yet from what I can see you really need to start thinking from the perspective of 'how the hell do I use this interface' rather than treating it as a technical implementation exercise.

edit:

• No leagues listings, was this not the whole point of the project?
• The css styling I gave you keeps 'vanishing' and getting replaced with the fugly old styling, what gives?
• What is the point of a login if its not apparent what you need to login for, or how to register?
• Details on the test login have been lost under hundreds fo forum posts. Searching a 17 page thread ftl
• Still not obvious how exactly a new AI is to be added

So far you have been treating this as a technical implementation exercise. It's time focus shifted from 'how do we implement this?' to 'how do you use this?'

[hughperkins]

Hi AF,

What used to be AILadder is fissioned into two separate projects:
- SpringGrid: only handles running botrunners, storing results, queueing requests, dispatching those requests to the botrunners, keeping track of the status of each botrunner
- AILadder: only handles leagues, leaguegroups, figuring out which AIs need to play who, and requesting SpringGrid to run them.

SpringGrid is discussed in this thread, the one your are reading now.

AILadder is in this thread, the original AILadder thread:
http://springrts.com/phpbb/viewtopic.ph ... &start=320

SpringGrid's code is at:
http://github.com/hughperkins/springgrid

AILadder's code is at:
http://github.com/hughperkins/ailadder

AILadder chats to SpringGrid using an xmlrpc api.

The architecture is summarized here:

http://manageddreams.com/ailaddergrid/architecture.py

You can find the leagues you are looking for for example on this page:

http://manageddreams.com/ailaddergrid/s ... ircount.py

Edit: PS logins:

- login to springgridstaging is by request, the default passwords have been changed. There is no way of anonymously resetting them or reloading the database.
- login to ailaddergrid site: password is: admin / admin . If it's been changed, you can reset it by reloading the database by calling: http://manageddreams.ccom/ailaddergrid/reloaddb.cgi

Edit: PPS The short answer is: what you want is AILadder, specifically http://manageddreams.ccom/ailaddergrid , which is discussed in http://springrts.com/phpbb/viewtopic.ph ... 21#p387421

[hughperkins]

Upgraded springgrid to openid:
- when you login, you can provide an openid instead of a username and password
- if you don't have an account yet, it will create one
-- username will be set to be the same as the openid
-- the account will initially have zero privileges, ie it's basically a 'guest' by default
- the old guest account has been removed

http://manageddreams.com/springgridstaging/loginform.py

Anticipated usage:
- a user logs in once with their openid account
- an accountadmin can then assign appropriate privileges to that account (botrunneradmin, aiadmin, accountadmin, ... as appropriate)

Tech notes / implementation:
- under the hood, an account can have a password, or not, and any number of openids assigned
- in the future, we could extend front-end forms to manipulate these more extensively if we wished to

[hughperkins]

Started playing around with Amazon EC2. Someone suggested a few weeks ago using EC2 for botrunners, but at the time, in my head, I was envisaging a system where the botrunners were always running, and when I thought about EC2 I imagined a system where the website might start them automatically when there were things in the queue, which sounded equally expensive...

But it just occurred to me: who wants the botrunners to run more than other people? Arguably people who write the AIs, AI devs, certainly it is what is motivating me and with Amazon EC2 you can use other people's os images, so, do you know where I am going with this? Idea:

- AI dev pumps some games into the match queue
- starts some EC2 instances using his/her EC2 account
-- they can either be an official botrunner instance, shared between everyone
-- or it could be his/her own instance of a specific OS type (most likely Ubuntu jaunty/karmic)
--- and launching/installing the botrunner by simply curl'ing/wget'ing a static url, and running the downloaded script
- the instances run until the queue is empty, or after a certain predermined (by the AI dev) time has elapsed, or until the AI dev terminates it, whichever comes sooner
- and Ploof! the ai dev can read the results in the ladder website!

Advantages of this:

Compared to just running botrunners locally on one's own machines directly:
- no security issues
- can run just fine 'on an eeepc', or rather 'launched from an eeepc'
- doesn't use air-conditioning, power, fan, make noises, take up space, ...
- doesn't interfere with watching movies etc...

Compared to running botrunners locally in a virtualbox:
- far simpler to set up and distribute. Creating a setup/installation script to install virtualbox, and download a vm and so on is really a *lot* of work, and not maintainable at all I feel
- security probably equivalent
- same advantages for running locally directly in terms of not needing air-condigitioning, power, fan, or disturbing a video, game and so on

Compared to running botrunners on someone else's grid farm:
- lots more control
- not reliant on a single person to have a grid
- generally more flexible, scalable

Thoughts?

[hoijui]

just to see if i get the basic ide aof that thing right (amazon EC2):

you make a virtual image, and upload it to Amazon. thne you pay them per GHz, and they run this image on their machines. you configure details through a we interface (VM options or the like).
yeah?

[hughperkins]

just to see if i get the basic ide aof that thing right (amazon EC2):

you make a virtual image, and upload it to Amazon. thne you pay them per GHz, and they run this image on their machines. you configure details through a we interface (VM options or the like).
yeah?

I think your question is: what is Amazon EC2? Yes, they have an api you can call from your local machine's commandline, and they also have a web interface, as you say.

There are a bunch of pre-existing virtual images available, various types of Windows (yes, I know, I could use this for Windows testing!), Fedora, Ubuntu, all sorts of stuff...

You can also make your own.

Cost is, roughly:
- about 10p per hour per machine instance
- about 15p per gigabyte storage per month (so if you have an image that takes up 600meg, that's about 8p a month )
- something trivial for bandwidth, it works out very small for me, but if one had a huge Amazon-type website, I'm sure that would become non-negligible, or if one wanted to stream videos I guess

[hughperkins]

Thinking about how to build spring-headlessstubs for this...

There are a few options...

- grab the latest spring release tag, eg 0.80.5.2 is the current one I think?, and merge springheadless onto that, to give a new branch 0.80.5.2_headless in the springheadless repo, and build that

- just build springheadless head. The issue with that I see is that some of the AIs (E323AI springs to mind) may not build on a non-release build

(and conversely the issue with doing a merge onto the latest tag is that: it's a lot of work! so it simply won't happen unless someone feels like maintaining that, and I for one am not volunteering for that...)

- grab the 0.80.n.0 tag, and build against that
-- the issue with that is that there could be critical bugs
-- and the advantage, why we might do that, is it reduces the amount of effort to keep things up to date

An automatic way of building the latest tag would be nice, but until springheadless is merged into spring proper, I feel that that seems unlikely to happen, not automatically. Playing around with it just now, in various sequences of merging /rebasing, generally about 4 files conflicted each time, not necessarily the same ones, depending on sequencing.

Building springheadless head sounds the easiest, but that has it's own issue: it's not a stable target, so how do we decide what to build? Maybe we just make another tag, separate from the spring release tags, but that has its own issues...

I feel the simplest solution would be if we could merge springheadless into spring soonish, and then a lot of these issues go away, since there would no longer be a need for a merge.

Or if there is a way of making the merge automated?

Thoughts? (I don't really have a good solution at the moment, though admittedly its 4am in the morning at the moment... )