AI Ladder, 0.5

[hughperkins]

Update to staging:
- page added to register new accounts
- page added to register new calcengines
- - the sharedsecret only shows up if you are logged in with the same account used to create the calcengine

For now, there is no way of deleting anything, aside from wiping the database . Once that becomes an issue, it should be fairly easy to add a button to each form to delete things.

[hughperkins]

Added the possibility of specifying an AI url download location to ais to staging + git.

Unsure to what extent it is better to download the AIs ready-built (faster, easier, riskier) or build from source (sounds really a pain...).

If the AIs are downloaded from a url, they'd either better be heavily authenticated somehow, or be running on a decently firewalled machine that can be reimaged easily and regularly.

For now, nothing is actually done with the urls, they're just stored in the database, and displayed in the list, and they're not mandatory when registering an ai, for now.

[hughperkins]

Updated staging + git:
- added leagues and leaguegroups
- organized the menu on the left a bit better

http://manageddreams.com/ailadderstaging/

A league is a specific combination of:
- a single map
- a single mod
- a specific set of otpions, eg cheating enabled

Options are not yet implemented so a league at the moment is one map and one mod combined.

A leaguegroup is an arbitrary combination of leagues. for example, one could group together all leagues that are over water, or all that have cheating switched on.

A league can be in more than one leaguegroups, or in no leaguegroups at all. It's quite flexible.

The idea is that the results can be calculated for each league and each leaguegroup.

It's not done yet, but I guess somethign like:

- each AI can be assigned to play in one or more leagues (maybe also to one or more leaguegroups)

- a scheduler makes sure that every ai assigned to each league plays against every other ai assigned to that league
- - a configurable minimum number of matches, but seeing as we don't have any calcengines yet and we'll probably just be using my eeepc 4g, with 512meg of ram and a 600megahertz processor, probably 3 games per ai combination per league is enough :-D

- a results compiler goes through each league, for each pair of ais, and determines which of the two ais won, for thoese two ais, for that league
- a ladder for each league is determined by counting the number of ais each ai can beat in that league (or maybe just the number of matches won overal? but then we have bias depending on how many matches each ai played, and against who... so we'd have to tweak the calculatiosn to take that into account => harder :-D )
- for each league group, the results of each member league are ... amalgamted together ... somehow :-D ... to form a ladder for each leaguegroup

The result will be that we can have for example:

- watermaps league group: 1st ... 2nd ... 3rd...
- cheating maps league group: 1st ... 2nd .... 3rd ...
- ...

As for when the calculations are run, I'm still tossing up between a scheduled job, or every time a new result arrives, or manually. for now it will probably be manually (clicking on a webpage link), because it's easiest, lowest risk, easiest to test.

[AF]

It might be best to take the menu out of a frame, and shove it at the top of the page so that you can increase the chances of a shiny AF CSS graphics styling patch.

Finishing my php version just to get a shiny User Interface isn't such an appetizing idea when you've got a working prototype.

[hughperkins]

Can you show me an example? Preferably in Python but it's the layout that is important, so anything really, maybe an example on your own hosting?

There are about 20 menu items now, so I'm not quite sure how that could be put at the top of the page? If you've got an easy solution, I can switch to that. If it's not so easy, well, I probably won't

[hughperkins]

Updated http://manageddreams.com/ailadder to match staging.

Going to try running a calcengine on my eeepc overnight, see what happens... probalby it will just repeatedly crash :-D Anyway, we shall see.

Note: just corrected a couple of bugs that prevented the calcengine running just now:
- on dreamhost, with a slightly older version of python, datetime.strptime doesn't exist, and I was using that, but no longer am
- the version of HughAI in static data was set to 1.0, instead of 0.1, causing a seg fault when the game started

[hughperkins]

First game finished ok and the replay uploaded ok, it looks like!

HughAI has a crash at the end, which I think is fixed now, hopefully.

E323AI won.

Hopefully, the other 9 matches won't be E323AI winning every time, though I'm not holding my breath on that....

Results pane:
http://manageddreams.com/ailadder/viewresults.py

Website:
http://manageddreams.com/ailadder

[hoijui]

... mm nice!

[hughperkins]

Ok, so just woke up.. it seems that four more games went through ok, with E323AI slaughtering HughAI at every step , but anyway, that is out of scope itself as far as AILadder :-DDDD , until match 8 when HughAI found a new way of getting an exception, and hanging the game.

So:
- the calcengine works ok when everything goes ok
- it really does need to implement the game timeout functionality (it's not hard to do, and most of the logic to do this is already in place, just need to test for this...)
- it might be useful to upload the logfile from each AI along with the replay. For example, each AI could specify the path to its logfile, relative to its AI directory, for example "team%TEAMNUMBER%.log".

And, for the AI itself, it would probably be useful if there is an option to turn on or off batch/calc mode. For example, in HughAI, currently, when running interactively, the exception is shoved in your face, making it really obvious, but stopping the game. In batch mode, this could just be quietly logged and then ignored, and could be seen in the uploaded logfile later.

Edit: I will probably turn off the 'calc engine' now, since I need it to do stuff on it, and there's not really space to run eclipse, firefox and spring all at once on an eeepc :-D

[AF]

At the side of the websit eyou have a frame webpage contianing at most only a handful of items, lay them out horizontally or in a box inside a div and use a vertical column. There is no excuse for using frames for basic menus in this day and age.

Google, 2 column css layout, you should find readymade examples you can use.

[hughperkins]

Updated git and staging:

Feature updates:
- roles added: accountadmin and aiadmin
- only accountadmins can add/remove accounts, and add/remove roles
- only aiadmins can add new ais (since I'm thinking of starting to use the downloadurl, so I want that to be fairly secure)
- you can now remove accounts
- and you can remove roles

New staticdata updates:
- new default account 'guest', 'guest'
- added three dummy AIs (I'll probably populate the static data with dummy results at some point)

Technical updates:
- moved modules to subdirectories, a little cleaner

[hughperkins]

@AF, I googled as you said, but it wasn't immediately obvious. If you can come up with an example, or ideally provide the html and css for the current AILadder, that would speed up such a modification a lot!

At the moment, appearance is quite near the bottom of my priorities. Somewhat higher up are these things:

todo:
- add subscription to each league for each ai
- add scheduler, to queue up matchrequests for each league
- add results calculator
- add gameendstring to per-mod config
- add cheating / no-cheating option, and enforce in engine
- some way of passing options through to ai (via script.txt?)
- store passwords as salted hashes

@All:

By the way, as another request, if someone wants to make me a python module, or link to an existing operational one, with two functions, something like:

# takes a clear text password, returns something that can be stored
# in a passwords database table
def clearTextPasswordToSaltedHash( clearTextPassword ):
return saltedhash

# check clearTextpassword, from user, against saltedhash,
# from database passwords table, and return True if
# the password from user is correct, otherwise False
def checkPassword( clearTextPassword, saltedhash ):
return passwordok

Note that if you're going to do that PLEASE state you're going to do that first, otherwise I might work on it at the same time...

That goes for just about anything else you want to change in AILadder really: just email me first, or post here, and then I'll know not to touch that bit :-D

[hoijui]

for both the users and this ladder, it would be good if we introduced some standards. what i can think of right now:
logging & cheating
eg, instead of AIs having to define where their log file is, we could probably just define that log files have to go to the subfolder log, and that on subsequent runs of the AI, old logs are getting moved to an archive subdir, so only logs of the lat game are directly under log.
with cheating, we could adhere to a common standard as well:
in AIOptions.lua, we could have a property that defines modes supported by the AI, for example:

Code: Select all

	{
		key="cheating",
		name="Supported Cheating Modes",
		desc="What cheats shall be used by this AI",
		type="list",
		def="default",
		section="difficulty",
		items={
			{key="none",      name="None",           desc="Do not use any form of cheating"},
			{key="all",       name="All Supported",  desc="Use all cheats supported by this AI"},
			{key="default",   name="Default",        desc="Use the default set of cheats, as defined by the dev(s) of this AI"},
			{key="events",    name="Cheat-Events",   desc="Perfect intelligence agency: AI knows about events it can not see"},
			{key="resources", name="Resource-Hacks", desc="The AI can give resources to its self, for free"},
			{key="map",       name="Map-Hacks",      desc="AI can always see the whole map"},
		}
	},

this could be the default, while AIs would be free to define only a subset of these, or additional ones.

Maybe there is more then logging and cheating, and my proposal is only meant as an example, the real format should be discussed among active AI devs. just want to say that both users and this ladder would benefit from standards.

[hughperkins]

@hoijui,

Yeah, your ideas sound good.

By coincidence, I started looking at adding options today:

@all:

git and staging updated:
- options added to ais, leagues, and matchrequests
- can change account password
- mapadmin, modadmin and leagueadmin roles added
- can add a download url to maps and mods (maybe that was done earlier? can't remember...)

http://manageddreams.com/ailadderstaging/

How the options work

For now, they don't actually do anything, they're just assigned to leagues, ais and matches.

The vision however is that:
- a set of options are assigned to each league, for example 'cheatingallowed' or 'cheatingequalslose'
- any ai with at least those options assigned to it as compatible options will play in that league

Certain options may be hard-coded to do stuff, for example, the cheatingequalslose option is likely to result in a check being carried out by the calcengine at runtime: any ai that activates cheating automatically loses.

The reason I added options is that there are only two cheating AIs available at the moment: E323AI and HughAI, AFAIK; so to add more AIs to the list, I need to make it possible to have at least two leagues: cheating and non-cheating.

The reason I added the ability to change the password is that I will probably upgrade the main http://manageddreams.com/ailadder website soon, and change the admin password, and then hand out indviidual accounts to anyone who wants one, with roles as appropriate; and then it will be possible to get the calcengines to trust the ai urls and so on, and automatically download ais, maps, mods...

So, the next steps most likely are:
- enforce the cheatingequalslose option on the calcengine
- add AAI to the static data
- add a matchrequest scheduler
- setup some matches between AAI and HughAI (I think HughAI can win those... but anyway, that's not the ponit it's just to get some games out there).

... and then, at some point after that, figure out how to crunch results, and display them appropriately.

[hughperkins]

Right, having added startscript option reading to my own AI, it looks like it is important that options can have a value, not just a name/key.

Whether all options should be like that, or whether only options that will be provided to an AI should be like that is an open question in my mind.

Advantages of them all having values:
- uniformity

Advatnages of leagues and so on only having a list of keys:
- simpler for the leagues and so on

[hughperkins]

Migrated to use a CSS menu. That was a *LOT* of work, AF. So I hope you feel it was worth it (git + staging updated).

[hughperkins]

Update to git + staging:
- passwords are now stored as salted hashes

Yay!

[hughperkins]

Improved the css a little:

[hughperkins]

You know what, could do with some engines really Kind of hard to process results without engines pumping results through

It's fairly easy to run an engine. Just download latest git, go into calcengine directory, tweak config.py to point to ailadder website, and run! It will assume you have SmallDivide installed and BA 6.96 installed, but anyway, if there's at least one person running a calcengine, I'll look at making sure that the engine downloads the appropriate mods and maps and AIs automatically.

[hoijui]

ok i tried it. my calcengine is called hoijui, but it does not show up here:
http://manageddreams.com/ailadderstagin ... engines.py

btw.. calcengine is not a good name :D
ie.. now that i tried this all, and foudn out what it is, it kind of makes sense, but ... the name should rather make it clear beforehand already.
Maybe something like AI Battle Executer ...
best would be if you had a smal graphic, a use case diagram, eg:
user makes a request for a battle on the webserver, the webserver thess the calc engines to run the battle, when done, the results are shown.. with some arrows and stuff.
not too important... but might give you more ppl that do it.

also, to set it up, i had to consult many places. eg i had to see the first post of this thread for the homepage to set in the config file.
the howtouse.txt did not mention to change the website, and your previous post did not mention some other things to change in config.py. also.. spring can be spread over many dirs, and usually is so in linux.. and also begins to be so in windows ...
springgamedir has to be the writeable data-dir? or the dir wiht the spring binary? or any data dir? (i assumed the writeable data-dir, but you should make it more clear.)
i set the website to this:

Code: Select all

websiteurl = "http://manageddreams.com/ailadderstaging/"

is this correct?
you could have this in the template already, i would say.