Spring RTS Engine

Posted: **23 Nov 2009, 02:55**

I dont have 3 days to spare, I barely know python syntax nm enough to implement this, and I dont have the time to learn. I don't have knowledge of amazon EC2, nor the cash to spare for amazon services.

And I'm not insulting you, if I wanted to insult you it would be blatant expletives.

You also have meltraxes ladder to refer to for a ranking algorithm.

Posted: **23 Nov 2009, 02:57**

AF wrote:I have the machines available to set up as bot runners, after all I can leave this machine on for a day and run through a tonne

Ok, great! But they're never up... but what you could do is set up a bunch of matches, and pump the results out.

but at the end of it, what do I get out of it, my AI isn't even listed in the site, and I have no idea how to register on the site nm login to set them up.

To add AIs:
- install Spring on a machine
- install your AI on that machine
- run botrunner on that machine
- Ploof! by magic your AI will appear on the website

As far as logging in, on SpringGrid, you can use an OpenID. On AILadder, you can use username admin password admin.

Posted: **23 Nov 2009, 06:03**

In the meantime, going back to EC2 and stuff

What we want is:

- a way of making it possible for AIs to provide scripted/bytecode AIs that we can run -> this is easy

- a way of making it possible for AIs to provide compileable/compiled AIs that we can compile/run -> this is a lot more challenging

This post is going to focus on that second case: how to make it possible for AI devs to provide AIs in a format that is easily runnable in an EC2 instance? Much of this will apply to running it in other botrunners, but I will reduce the scope of this post by only considering the EC2 environment.

Plan for compilation:
- we could select a single point on a git branch as a tag, and build spring-hl from that, and the ai interfaces
- then, the AI devs can build their AI against the same point
- an issue: what if we need to make changes to spring-hl itself to get things to work?
- we could simply move the single point to a new point
- ... or we could just always use springheadless HEAD ...

- or .... we could have a rolling compile. An instance that handles compiling against head, and by starting it, it will compile the latest head and interfaces, and upload those to S3, or similar
- then, when we start it, it downloads the previous compile, and rebuilds

- or, we could just have a compilation instance that simply downloads everything, including the source, and builds the latest spring-hl, and tar.bz2s it and uploads that to s3, or similar
- and then shuts itself down
- we could also have a compilation instance whose job is to compile an AI
- it would need the spring source
- it might also need to have built spring
- and then it would upload the result to s3, or similar

- or, each botrunner instance could be a compilation isntance
- maybe this is the simplest to start with?
- maybe depends on cost of bandwidth and so on to get to fully compiled state?

Posted: **23 Nov 2009, 06:20**

Created a script at:
http://manageddreams.com/springgridstag ... ompiler.py (it's stored in git too), which you can use on EC2 ami ami-1515f67c like this:

- start instance of ami-1515f67c
- ssh into running instance
- inside instance:
-- wget http://manageddreams.com/springgridstag ... ompiler.py
-- bash bootstrap_cloudcompiler.py

This will:
- install all dependencies (java, git, boost, gcc, ...)
- download springheadless head
- build everything

The actual build takes I think about 15 minutes, but anyway times from the other steps just now:

13h11 start script
13h13 script finished installing packages
13h13 script finished git checkout of springheadless
13h14 script finished cmake
... and I don't have a time for finishing the build yet since it's still running

Package installation is insanely fast, though I think there is a slight bandwidth cost to that (5 pence?).

There is also a script at http://manageddreams.com/springgridstag ... trunner.py , but this depends on there being a precompiled spring-hl in a suitable tar.gz file, and also this won't be sufficient to be able to compile AIs.

Edit, maybe what we need is for the compilation process to provide as an output, stored on s3 or similar:
- the spring-hl build, along with unitsync, the base files and so on, so that botrunner can run
- the spring headers, and interfaces correspdongin to that, so taht we can build AI sourcecode

.. then the botrunner can download these things from s3, and be in a pretty decent position to run things, and compile them.

Decision to make as far as instances, compiation:

options: one instance/script type:
- compiler and botrunner are the same thing
-- easy
-- instances take a looonngnggg time to start
-- considerable waste of instance time

option: two instance/script types:
- compiler: builds spring and uploads it to s3
- botrunner: builds ais (so needs compilation dependencies), and runs botrunner

Advantages: reasonably fast
Disadvantage: each botrunner will install all compilation dependencies, which uses maybe 500meg of bandwidth (so about 7.5p or so)

option: three instances/script types:
- compiler: builds spring and uploads it to s3
- ai compiler: downloads built spring from s3, compiles one or more ais, then uploads those to s3
- botrunner: runs pre-compiled ais and spring

In theory this has the potential to be more efficient but:
- if we run the instances in parallel on s3, we are charged for multiple instance-hours
- seems more complicated

... so I might go for the two instance option:
- one instance/script to compile spring-hl, the ai interfaces, and upload this to s3
- one instance/script which runs a botrunner, and a botrunner is a full compiler instance, but uses the precompiled spring-hl from the compiler instance.

Posted: **23 Nov 2009, 08:36**

Update:
- script at http://manageddreams.com/bootstrap_cloudcompiler.py should be fairly operational now, though it might need a little bit of nursing the first through runs through
- a build is available, for karmic, of spring-hl at:

http://hughperkins.s3.amazonaws.com/spr ... ss.tar.bz2

- the cloudcompiler script does the following:
-- ask for s3 connection credentials
-- asks for s3 bucket name (corresponds to the 'hughperkins' bit in the download url above)
-- asks whether to shut down the machine after compiling and uploading has finished
-- downloads all dependencies, java, s3fs, gcc, boost, ...
-- downloads springheadless from git, and checks out the origin/springheadless branch
-- configures and makes
-- tar.bz2's the entire source tree, including build tree and install dir (result is about 185MB)
-- uploads to s3
-- shuts down, or not, depending on earlier user preference

The goal is that this is run ... occasionally. When a new spring release comes out perhaps, or if there is a showstopper bug in spring-hl which we had to fix.

Next up:
- refining the http://manageddreams.com/springgridstag ... trunner.py script to:
- download build dependencies
- download springheadless.tar.bz2 from the compile script
- ... and thus be capable of compiling C/C++ AIs on the fly

Posted: **23 Nov 2009, 09:48**

Update:
- http://manageddreams.com/springgridstag ... trunner.py starts a botrunner instance now, on a fresh Karmic instance (ie on an EC2 instance of ami-1515f67c for example)
- to use, on a karmic instance in EC2 or virtual box or similar, run:

Code: Select all

wget manageddreams.com/springgridstaging/bootstrap_cloudbotrunner.py && bash bootstrap_cloudbotrunner.py

- this will:
-- install spring dependencies (boost, java, ...)
-- download springheadless.tar.bz2 from s3
-- download the latest version of springgrid botrunner from git
-- configure and start botrunner

Next up:
- botrunner should be able to download maps and mods itself
-- which implies the ability to add map and mod download urls to the website
-- and a configuration option in the botrunner to turn this functionality on
-- and the ability of the botrunner_webservice, on the website, to tell a botrunner to download a specific map or mod

Posted: **23 Nov 2009, 11:29**

maybe it would be easier to use Lichos content downloader service, then to download from websites, as.. they sem to change syntax of download URLs from time to time, and also seem to require special download mechanisms at times (eg, you have to click osmething on a page, instead of just downloading a URL.. i know too little about that).

AF, you know, hugh has a Netbook only, and what he tried to tell you, but you seemed to not care ofr, is:
if you want the system to be practically usable, and expect him to do this, then someone should donate a machine (may be virtual) or an account on a linux server at least, which runs 24/7.
that is what he was trying to get since .. maybe 2 months now. while aegis did something... he did it on his own, and the machines are not always up, and hugh can not directly operate on them. this owuld be ok if it were all ready yet, but as there are still things to be done, and hugh has to change stuff on the server often, this is a practical nightmare.

i dont have a suitable machine.

Posted: **23 Nov 2009, 16:24**

I cannot leaves these machines on 24/7, and I'm not sure I could provide remote shell access if that's required.

Posted: **23 Nov 2009, 17:07**

Updates to SpringGrid:
- botrunner now capable of downloading maps (beta, not very tested) and mods (beta, not very tested)
- botrunner now capable of downloading and compiling C/C++ AIs, by copying them into the spring source directory AI/Skirmish, and calling 'make install' (beta, not very tested)
- website provides possibiity to register, modify and delete maps and mods

It's midnight, and I don't think it's worth starting an EC2 instance just for 15 minutes, since the billing is per hour, although admittedly an hour would only cost about 6pence

Next up:
- either create a new concept like sessions but different, or
- modify the session concept entirely. Basically:
-- each cloud instance has its own set of maps, ais, and mods
-- so in the current system, it needs to have a unique botrunner name
-- ... since maps, mods and ais are registered per botrunner
-- ... but ...
-- we want to be able probably to manage all cloud instances in the website as a single unit, eg to set options like 'operationalai'
-- we probably don't want to have to set options by hand for every new instance of a cloud botrunner...
-- sessions could map appropriately to cloud instances
-- ... but ...
-- ... sessions do not contain maps, mods or ais, merely a sessionid, and a record of what match request is being processed at the moment (if any)
-- ... so, we might need to move the maps, mods and ai registration onto a per-'session' basis, possibly renaming 'session' to 'instance', and possibly providing a per-botrunner option to decide whether spring data is global to the botrunner, or local to each instance
-- ... which sounds like a lot of work basically

Posted: **23 Nov 2009, 17:13**

hoijui wrote:
maybe it would be easier to use Lichos content downloader service, then to download from websites, as.. they sem to change syntax of download URLs from time to time, and also seem to require special download mechanisms at times (eg, you have to click osmething on a page, instead of just downloading a URL.. i know too little about that).

Hmmm, sounds interesting. Might be a little slow though? Since the maps are downloaded each time an instance is started, and each instance downloads the maps afresh.

One possibility could be to store the maps in s3, which costs about 0.04pence per map per month for storage space, assuming each map is 4meg.

AF, you know, hugh has a Netbook only, and what he tried to tell you, but you seemed to not care ofr, is:
if you want the system to be practically usable, and expect him to do this, then someone should donate a machine (may be virtual) or an account on a linux server at least, which runs 24/7.
that is what he was trying to get since .. maybe 2 months now. while aegis did something... he did it on his own, and the machines are not always up, and hugh can not directly operate on them. this owuld be ok if it were all ready yet, but as there are still things to be done, and hugh has to change stuff on the server often, this is a practical nightmare.

Thank-you hoijui

Posted: **24 Nov 2009, 04:33**

Update:
- ai sourcecode downloading + compiling tested, corrected, and works now
- map and mod downloading tested and works ok
- created a webpage at 'Runner', 'Start botrunner instance...' that describes how to start a botrunner:

http://manageddreams.com/springgridstag ... trunner.py

Submitting AI source code

Note that the compiler unit, erroneously I think, checks for a VERSION file in the AI. I think it should check for a data/AIInfo.lua file instead, and sanity-check the values there, but for now, if there is a VERSION file, even empty, in the AI it should work.

The AI sourcecode will be copied into the spring build tree, next to E323AI and so on, deleting any existing folder with the same name as the ai, then the compiler unit runs:
- cmake ..
- make <ainame>
- cmake -P AI/Skirmish/<ainame>/cmake_install.cmake

This all works fine, and then the game is launched using spring-hl.

Major issues

Error to fix:
- when spring-hl runs, it gives an error:

Code: Select all

Starting GameServer: 6 ms
Content error: MapInfo: could not open file: maphelper/mapinfo.lua
Warning: Incorrect/Missing content:
  MapInfo: could not open file: maphelper/mapinfo.lua

I suspect this is a well-known newbie build error, so I'll check the wiki, and maybe irc, but if anyone has any ideas?

Next up for dev:

So, next up:
- fix the bug with mapinfo.lua
- possibly, look at implementing a botrunner option 'dataisinstancelocal' so that maps, mods and so on are reigsterd per session rather than per-botrunner

Edit: maybe the mapinfo.lua error is related to 'Fixed crash on startup on Ubuntu Karmic' in 0.80.5.2 release? ie, we may not be using the 0.80.5.2 code, but something fractionally older and this is the bug in question?

Edit2: I dont get the same error on my springheadless jaunty build, so either there is something missing in my karmic build environment, or something I'm doing wrong in it, or there is something different about karmic

I tried merging in latest version of spring into a local copy of springheadless, and building that, and that gave the same error.

I suppose I need to download spring source itself, build that, and show that it gives the same error and then check in the dev forums (I checked in irc, but despite some ideas the issue is still unresolved)

Edit3: built spring git on karmic, and installed x, but apparently running glx applications on X is a little ambitiuos

.

So my options seem to be:
- wait for someone to figure this out
- switch to using jaunty ec2 images
- install karmic on my eeepc

by the way, there is a bug in the springheadless.tar.bz2 archive: BINDIR points to 'bin' rather than '/home/ubuntu/git/springheadless/game', but moving the executables by hand into the game directory doesnt fix the problem....

Edit4: tried in jaunty ec2 image, and spring-hl ran fine, but I discovered a bunch of bugs in my compiler script, so fixed those, and re-trying on karmic...

Posted: **25 Nov 2009, 05:09**

Update:
- lots of bugs fixed in cloudcompiler for karmic
- cloudcompiler may work on jaunty now (may be one or two bugs)
- springheadless .tar.bz2 created by cloudcompiler is now distribution specific, ie 'springheadless_jaunty.tar.bz2' vs 'springheadless_karmic.tar.bz2'

Progress on maphelper/mapinfo.lua bug:
- running spring-hl on a karmic cloudcompiler instance works just fine
- running spring-hl from the downloaded springheadless_karmic.tar.bz2 does NOT work

Ideas for maphelper/mapinfo.lua issue:
- file date/times are wrong in downloaded springheadless_karmic.tar.bz2, eg maybe the data files are older than the spring-hl so spring-hl refuses to load it?
- there is some dependency that is missing on the cloudbotrunner instances
- there is something that is being installed to a global directory, and hence is missing from the springheadless_xxx.tar.bz2 archive (but: base/maphelper.sdz is there, and unitsync is correctly looking in the right directory => weird!)

Posted: **25 Nov 2009, 07:21**

Found the reason for the mapinfo error! I'd named the smalldivide map as 'SmallDivide.smf' instead of as 'SmallDivide.sd7', and that caused the error :-O

A lot of work for diagnosing that... not a very explicit error message

.

I'll fix botrunner to download the map as .sd7 or .sdz, and that should fix that hopefully.

Posted: **25 Nov 2009, 08:10**

cloud botrunner is working!

Cloud botrunner runs botrunner on a base install of Karmic server, or anything compatible.

- it runs requests in the queue according to the maps, mods and ais it has installed
- when necessary to run a request, it will download the missing map, mod or ai, and install it
- it is capable of compiling, installing and running C/C++ AIs

How to start a botrunner instance

Please see http://manageddreams.com/springgridstag ... trunner.py

Should I use VirtualBox, my own pc or Amazon EC2?

Yes

Well, it's not really recommended to use your own pc because it's a bit of a security risk.

The easiest way I feel is to run an Amazon EC2 instance, because it's easy to do, and avoids disturbing your own PC, using air-conditioning etc.

If you have your own spare pc lying around, you can use those. Just install ubuntu karmic to it, or install virtualbox, and install ubuntu karmic to that.

What's this 'Amazon EC2' thing?

You can rent server instances on Amazon EC2, or any similar cloud service for that matter, but I've only tried EC2, and it worked well for me.

The instances are billed per hour, at about 8 pence an hour or so for a linux box, which is ... affordable I feel. A whole day costs about 64p, or about a quarter a pint of beer in a student bar...

You do need a credit card unfortunately, or get your parents to use theirs I guess.

Gotchas:
- once an instance is started, it will run until you shut it off. I've created a cron job on my web hosting to turn mine off if I forget http://hughperkins.com/techblog/?p=58
- remember you're billed per instance, so don't go starting 19 of them at once

At least, unless you want to be billed over a pound an hour for them...

Can't I just use VirtualBox?

Yes.

Or my own pc?

Yes, if you're fine with running arbitrary code on it...

If you're running ubuntu, you might look into AppArmor.

If you're running Windows ... do you really want to enable arbitrary code execution? But the scripts are tested and optimized for ubuntu anyway, so I feel you will find it muuuuch easier just to install virtualbox, and run a botrunner inside that.

Posted: **26 Nov 2009, 01:49**

One thing I could do with is one of those widgets that accelerates the game from 3x to, well, quite a lot faster. If anyone happens to have one, or knows how to make one, could they post a link here, along with instructions on how to install it?

Posted: **26 Nov 2009, 01:51**

huh, thought it was merged into the headless git repo

Code: Select all

function widget:GetInfo()
	return {
		name = "HeadlessSetup",
		desc = "Setup for headless botrunner games.",
		author = "aegis",
		date = "October 18, 2009",
		license = "Public Domain",
		layer = 0,
		enabled = true,
	}
end

local startingSpeed = 120
local timer
local headless

function widget:Initialize()
	headless = (Spring.GetConfigInt('Headless', 0) ~= 0)
	if (headless) then
		Spring.Echo('Prepping for headless...')
		Spring.SendCommands(
			string.format('setmaxspeed %i', startingSpeed),
			string.format('setminspeed %i', startingSpeed),
			'hideinterface'
			)
	end
end

function widget:GameStart()
	Spring.Echo('Game started... starting timer.')
	timer = Spring.GetTimer()
end

function widget:GameOver()
	local time = Spring.DiffTimers(Spring.GetTimer(), timer)
	Spring.Echo(string.format('Game over, realtime: %i seconds, gametime: %i seconds', time, Spring.GetGameSeconds()))
end

LuaUI/Widgets/headless_setup.lua

Posted: **26 Nov 2009, 02:41**

Ah sweet! So I just copy that into game/LuaAI/Widgets, and it will automatically get taken into account at runtime? Is there anything that needs to be done to activate it at runtime?

Posted: **26 Nov 2009, 03:35**

Right, we tried chatting in irc, but my being disconnected after 30 seconds and taking 60 seconds to reconnect made it fairly unworkable...

Further to our chat, I did the following:
- created a file LuaUI/Widgets/headless_setup.lua with the contents from the forums
- added 'LuaAutoEnableUserWidgets=1' to the start of $HOME/.springrc
- deleted LuaUI/Config/BA.lua
- retried ./spring-hl myscript.txt

... but so far it still seems to be running at 3x speed. I don't think the widget is being taken into account.

I can give you ssh access if you want (you'll need to grab a copy of my private key, or maybe send me your own public key I guess could make more sense now that I think about it

).

Posted: **26 Nov 2009, 03:43**

Ah, I've figured it out, and now I agree with an earlier comment by Auswaschbar

,and have a better idea than my earlier proposal on config files. Anyway, in spring-hl we are overriding the config file:

Code: Select all

#ifdef HEADLESS
   ...
	configHandler->SetOverride("LuaAutoEnableUserWidgets", 0); // not sure about this one?
   ...
#endif

Not sure how you managed to overcome that on your system, but I think that maybe a better solution than my original proposal/implementation on overriding the config inside spring-hl could be to provide an option on the commandline to specify the path to a specific configfile, eg we could do:

Code: Select all

./spring-hl --config-path headless.config

For now, maybe I'll back the code out that hard-codes the config, and we'll just shift all that stuff into the global config. hoijui, this post is kind of posing the question to you too for validation.

Posted: **26 Nov 2009, 08:07**

Update: cloudbotrunner now runs at x120 speed
- migrated springheadless to use aegis' widget, see http://springrts.com/phpbb/viewtopic.ph ... 97#p393897
- rebuilt springheadless_karmic.tar.bz2
- updated cloudbotrunner to copy config-headless.txt into $HOME/.springrc , to initialize the spring configuration appropriately for spring headless

I've also extended the StartBotrunner page somewhat with details on how to recompile springheadless_karmic.tar.bz2 if needed.

By the way, it occurred to me that this should probably run ok also on a netboot environment, as long as each node has enough memory. One slight optimization that could be done in such an environment would be to implement a large internet cache, which would mean that the package installation on each instance startup would be very fast.

Spring RTS Engine

SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid

Re: SpringGrid