AI Ladder, 0.5

[aegis]

I'd also finish flash dotwars and support it on the ladder, maybe even for live games.

[Error323]

Yeah this is definitly a good step for AI's in spring. Also I'm trying to work out an AI system that has the ability to learn and adapt with Kloot. Still brainwaving though, not at all sure if this will work. But we've got some interesting ideas and it would definitly need a system like this to learn on.

[AF]

there reallly isnt much need for a ladder as it is, the released AIs rank as follows:

E323AI
KAI
RAI
AAI

which incidently is also the order of which requires the most memory

when hughAI is released fully configed then there will be leeway for some games, and I dont know if NTAI is likely to make a return in the near future but the playing ability of NTAI is hugely dependant on the quality of its configs, it can potentially beat most other AIs simply because it can be given build orders that it executes perfectly (although last release its combat behavior was admittedly very sloppy)

NTai will support playing averagely at least one mod by christmas, if onyl as part of a feasability study.

Since the last release, the sections of NTai managing task lists have under gone huge architectural redesign and refactors, and more is set to come, in all areas. So hold no assumptions. I have a major change of tact with regards to how I approach AI. There will only be one more major release that natively supports the current configuration format.

[hughperkins]

when hughAI is released fully configed

HughAI runs as ARM under BA. I do not have any plans to extend that, though obviously if the ladder decided the standard was CORE under OTA or something then I would tweak the config slightly to use that instead.

What do you mean by "fully configured"? Is there some issue that you've come across running HughAI as ARM under BA?

[AF]

Any ladder involving BA would have to do equal amounts of Core and Arm for a fair test, which would show up in the BA ladder league, as hughAI (core) being at the bottom of the list, and HughAI(Arm) being much higher

[1v0ry_k1ng]

when hughAI is released fully configed

HughAI runs as ARM under BA. I do not have any plans to extend that, though obviously if the ladder decided the standard was CORE under OTA or something then I would tweak the config slightly to use that instead.

What do you mean by "fully configured"? Is there some issue that you've come across running HughAI as ARM under BA?

by fully figured I mean it can play the mod as every side. its hardly complete if it cant. also what AF said

[JohannesH]

Is a human player required to play both sides in a ladder?

[hughperkins]

by fully figured I mean it can play the mod as every side. its hardly complete if it cant. also what AF said

Fine... I'll look into it. Grrrr! :-D

[hughperkins]

Hi 1vory_k1ng, there is a new release of HughAI now, which contains configuration files both for ARM and CORE, running on BA.

http://springrts.com/phpbb/viewtopic.ph ... 93#p380393

It's also possible to create configuration files for other mods, though the other mods better be quite similar in game style to BA, otherwise the config will not really be flexible enough to cope

[hughperkins]

Ladder would be epic to be able to track progress and as a means of regression testing.

Definitely. First hand experience of that just now, having released a version that could handle CORE, and then realizing I had broken ARM :-D Fixed now, but still the point is, tryign to test by hand on different maps, mods, sides, maphack vs no maphack etc => lose.

Automated testing of this via ladder => win.

[hughperkins]

Thoughts on architecture, putting the 'who won?' issue to one side for one moment:

- we have one central web-site, fairly lightweight, where devs can register their AIs etc
- we have a bunch of ad-hoc calc engines that can say 'hello' to the website, and request a game to run, downloading any appropriate maps and so on as needed, or telling the webserver 'hey, I have these maps and mods, so feed me games that use these'.
- public/private key cryptography to authenticate the calc engines to the website server
- whoever is managing the website is responsible for approving, or not, servers to connect to it, and for adding each server's private key to an allowed server whitelist

This way, the website stays very lightweight, can be run on dreamhost or whatever, the server can be run on any machine anywhere, without needing some dyndns or some such trickery, or opening ports on one's firewall or anything like that, and lots of people can volunteer their servers/ machines to run games if they want.

Obviously, anyone running such a server needs to be somewhat trusted not to tweak the results, hence the cryptography authentication, and also the results would, it would be better, say which server they ran on (name, and optionally that server's public key).

How does that sound?

[hughperkins]

One other thought: it could be useful to have a standardized source of entropy, random numbers, that can journal itself into a file, and let anyone rerun a match in order to verify that it was run exactly as said.

in aicallback:

aicallback.getRandomInt(); // returns a random number from a per-AI source of entropy.

Either the source of entropy has a single initialization number, that can be pseudo-randomly selected at game initailization, and recorded somewhere; or every random number sent to the AI can be recorded in a journal.

The first option would probably be the easiest.

Then, there is little incentive for anyone to 'tweak' their server slightly since it's easy enough for others to detect this.

Of course, AIs would need to be written to use the aicallback's entropy function, but that's not really that hard, and that's entirely at the developer's discretion anyway. If the AI developer doesn't do that, well, tough

Of course, someone can still introduce some kind of sampling / deskdrawer bias by running ten games and only submitting the ones where their own pet AI won, and I don't think there is any way around that, but still, that at least requires that their own pet ai is somewhat capable of winning, much more work than just starting a game and clicking an 'i win!' button! or somehow hacking the damage calculations.

... oh! here's how to get around the sampling bias issue: the seed to the entropy generator's is created by the webserver! So, now the calc engine can run the game as many times as it likes without submitting, but with the same source of entropy, every game will have an identical outcome!

So...

Summary for providing high level of verification / trustworthiness of the results of calc engines without needing to trust the calc engines themselves very strongly

- add an aicallback.getRandomInt() method to AI callback
- this gets a random number based off of a seed provided by the webserver
- the seeds are recorded, and made available, along with each game result on the website, so it's easy for other people to verify the results, and then, generally, there is zero incentive to proactively tweak such results

[hughperkins]

Determining win/loss

Right, let's keep it simple... we run only games against BA for now, we can consider other mods later.

We launch Spring, for two AIs, one against one, and check infolog.txt for the following string:

"] Team"

... once we find that string, the other AI is assumed to have won, and we kill spring and log the result.

If the game lasts more than I dunno 30 minutes? without either side winning we report a draw.

Both the string and the maximum-game-time-for-draw can be held in a config file 'BA.xml', on the webserver.

How does that sound?

[hughperkins]

Oh, one further thought: dev / AI authentication.

The AIs are running native code natively on the calc engines...

There's no technical way of determining whether the code is safe or not prior to running it (boils down to 'halting problem').

So, we need to be able to trust the code.

Two ways of doing this:
- AI devs provide a URL for their git repository, or similar, and just bump the server occasionally to pull the latest updates, compile, and rerun the games, or
- AI devs provide a public key to the website admin, who registers the AI dev on the website
- - code can be provided directly by devs, in a tar.bz2 or similar, and signed with the AI devs private key

In the second case, since we want to keep hte webserver light-weight, the webserver only stores a URL that points to the signed code to be run, and it is the calc engine that downloads and runs the signed code.

[hughperkins]

Made a very rough and ready calcengine here:

http://github.com/hughperkins/ailadder

GPLv2, feel free to fork or whatever, just keep my copyright notice etc, yada yada...

What this does is, it' a short python script that:
- downloads an xml file from the server
- - in the future, the xml file would be provided by some script on the webserver
- - for now, it's just a static xml file on my own dreamhost server
- fills out an appropriate script.txt
- launches spring
- waits for someone to die, by monitoring infolog.txt
- and ... that's it for now,

... which seems to cover a fair few of the basic things.

To go further we would most likely want to start thinking about the website side of things.

Also, it would be nice to have some sort of authentication and so on, but that can wait for a while, not high priority just yet, perhaps?

[hughperkins]

I suppose next, on the webserver we need:

- a queue of games that need to be played
- - this can be a table in MySql or similar, or even just an xml file, but MySql is readily available, and easy to use from Python, seems as good as anything

- a place to store the results
- - presumably another table

- a web page (python script) that hands out a game from the game queue, and adds a reference to that game to an 'inprogress' queue, along with a reference to the server, and the date/time it was assigned
- - if the game is assigned for too long (more than an hour say), it is revoked from that server, and can be reassigned to another, or to the same one, but anyway, the webserver will no longer wait for that server to produce a result

- a web page to receive the results
- - adds the result to the results table
- - marks that game as finished in the queue, or maybe just removes it from the queue (marking it is finished gives a better audit trail, but needs more space, and needs some purge process later)

We'd also want some simple web page to show the results, maybe just a table with every match played, and the result? I mean, for a first draft rough-cut proof-of-concept.

I might take a look at at least writing some .sql to create the tables, since that sounds relatively easy as a first step...

[1v0ry_k1ng]

Hi 1vory_k1ng, there is a new release of HughAI now, which contains configuration files both for ARM and CORE, running on BA.

http://springrts.com/phpbb/viewtopic.ph ... 93#p380393

It's also possible to create configuration files for other mods, though the other mods better be quite similar in game style to BA, otherwise the config will not really be flexible enough to cope

sweet, ok testing vs the other ais commencing. ill use it with maphack on.

[hughperkins]

Random thought: it could be interesting if not only AI devs could submit AIs, but anyone can submit workflows / config files. AIs that are sufficiently flexible in their configuration can be massively influenced by their configuration.

One way of doing this could be that AIDevs can nominate for their AI certain files that are 'configuration'. Then anyone can register and upload their own configurations. Or maybe there is a small filter on who can register, since it is possible to write exploit code in configuration files... but it's much easier to validate a simple xml configuration file or tdf file than 500Ko of object code...

Then, there can be battles between:
- AIDevs, trying to write the strongest AIs
- non-devs, trying to write the strongest configs
- AIDevs, trying to write the best engine that is most used to create the strongest configs

[aegis]

my servers are net-booted, so I'm not worried about binary exploits... (they'd also probably be compiling the AIs from source anyway)

we could start out just using them and add support for third-party servers along with better security later

[hughperkins]

my servers are net-booted, so I'm not worried about binary exploits... (they'd also probably be compiling the AIs from source anyway)

Wow, sweet That sounds cool

Quick question: what OSs can you run on these servers? I'm targeting a linux environment at the moment since my own hosting and my desktop are both linux. Targeting windows as well is far from impossible, but personally I have no access to a windows environment, so there'd be loads of things that would need tweaking (paths and stuff) before they worked ok.

By the way, got as far as creating some sql scripts to build some tables on a database:

http://github.com/hughperkins/ailadder/ ... server/db/
http://github.com/hughperkins/ailadder/ ... db/tables/