SpringGrid

[aegis]

merge to spring proper imho

[hoijui]

merge into spring master seems the way to go for me too.
sorry, my main attention lies on the ai refractor (which is going very slow), but yeah.. sorry for neglecting this so much.
well. we should merge to latest master, and do some tests: copile on gcc and MinGW, and run some test games. with we, i mean as many as possible.
one thing that was not yet done i think, is sync testing with a normal build. it should not be a problem, but cant hurt to do.
don not hesitate to push further for it
testers needed!

[aegis]

doesn't compile on mingw with cmake due to process creation length limit

[hughperkins]

Started trying to set up a Windows compilation environment. What a hastle!

Compare:

linux:
- just run a script like:

Code: Select all

springheadlesspath='http://manageddreams.com/ailadder/download/springheadlessstubs08051_ubuntujaunty.tar.bz2'
gitclonepath='git://github.com/hughperkins/springgrid.git'
export JAVA_HOME='/usr/lib/jvm/java-6-openjdk/jre'

sudo apt-get update
sudo apt-get --yes install git-core
sudo apt-get --yes install openjdk-6-jre-headless
sudo apt-get --yes install libopenal1 libogg0 libvorbisfile3 libfreetype6 libxcursor1 libboost-regex1.40.0 libboost-thread1.40.0 libboost-program-options1.40.0 libboost-system1.40.0 libboost-signals1.40.0 libglew1.5  libdevil1c2
sudo apt-get --yes install cmake-curses-gui build-essential p7zip-full
sudo apt-get --yes install zlib1g-dev libfreetype6-dev libsdl1.2-dev libopenal-dev libglew-dev zip libvorbis-dev libxcursor-dev libdevil-dev libboost-regex1.40-dev libboost-thread1.40-dev libboost-program-options1.40-dev libboost-system1.40-dev libboost-signals1.40-dev
sudo apt-get --yes install openjdk-6-jdk

git clone $gitclonepath
cd springheadless
git checkout -b springheadless origin/springheadless

mkdir build
mkdir game
cd build
cmake -D AIDIR:STRING=$HOME/git/springheadless/game -D AI_DATA_DIR:STRING=$HOME/git/springheadless/game -D AI_LIBS_DIR:STRING=$HOME/git/springheadless/game -D CMAKE_INSTALL_PREFIX:STRING=$HOME/git/springheadless/game -D DATADIR:STRING=$HOME/git/springheadless/game -D LIBDIR:STRING=$HOME/git/springheadless/game ..
make

... Ploof! Done! Fast. Easy. No need for some graphical terminal stuff.

Windows:
- open rdesktop connection. Sllloooowwww
- open internet explorer. Ok, I have nothing against internet explorer... yet
- paste http://www.freesshd.com/freeSSHd.exe into the address bar, and click enter
- it won't let me download it! because of security settings
- fiddle around with security settings, now it lets me download the file
- more warnings and stuff...
- run it
- click 'next', 'next', 'next', 'yes', 'yes', 'i agree to buy you a car', 'please consider this eula accepted', 'sure I'll buy you two cars', ... starting to dislike this
- setup ssh (graphically)
- connect from ssh
- realize that, there is diddly I can do from ssh, since there is no curl, wget, apt-get, and any installer is graphical anyway :-OOOOO

I am not a linux zealot. Windows has lots of advantages I feel, but administering servers remotely and scripting stuff does not seem to be one of them...

It's taken me one hour so far and I don't even have gcc installed

[hughperkins]

Right, I'm kind of done with trying to figure out how to install mingw on Windows. It's like water and oil ....

aegis, do you know how we could fix the issues with the mingw build in springheadless?

Alternatively, if someone feels like playing with Amazon EC2 and creating a Spring build environment in Windows, then I can grab that image and have a look at the springheadless build using that.

For anyone that wants to have a look at doing this, here are the instructions I made so far:

Code: Select all

Locally:
ec2-run-instances ami-df20c3b6 --key mygpgkey --instance-type m1.small

In the instance:
- open internet explorer
- tools, properties, security, custom level... enable downloading, ok, ok ...
- download and install 
http://www.freesshd.com/freeSSHd.exe  (next, next, next, etc... bleurgh!)
- launch sshd from the start menu
- it will appear in the systray
- right-click, 'settings....'
- add a user, using sha1 hash password for example

locally:
- ec2-authorize default -p 22   (if not already done for linux images)
- connect to instance using ssh, yay!

back in instance rdesktop (bleurghhh... :-/ no way to install things from commandline without repackaging them first...)
- download http://msysgit.googlecode.com/files/Git-1.6.5.1-preview20091022.exe, and run it
- open a cmd prompt
- cd c:\
- git clone git://github.com/hughperkins/springgrid.git

I don't think the ssh is strictly necessary for compiling spring but it will be nicer than using the graphical console I feel once mingw and so on are installed.

[hughperkins]

In case anyone cares (unlikely ), here are instructions for going from a base Windows 2003 server install on EC2 to getting spring building:

Code: Select all

# Locally:
ec2-run-instances ami-df20c3b6 --key mygpgkey --instance-type m1.small
ec2-authorize default -p 22   (if not already done for linux images)

# In the instance:
- open internet explorer
- tools, properties, security, custom level... enable downloading, ok, ok ...
- download and install http://www.freesshd.com/freeSSHd.exe  (next, next, next, etc... bleurgh!)
- download and run http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-setup.exe
- launch sshd from the start menu
- it will appear in the systray
- right-click, 'settings....'
- add a user, using sha1 hash password for example

# locally:
- connect to instance using ssh, yay!

# in ssh connection:
mkdir c:\tools
copy c:\program files\gnuwin32\bin\* c:\tools
set PATH=%PATH%;c:\tools
mkdir c:\MingW32
cd c:\mingw32
wget http://sourceforge.net/projects/mingw/files/GCC%20Version%204/Current%20Release_%20gcc-4.4.0/gcc-full-4.4.0-mingw32-bin-2.tar.lzma/download
wget http://sourceforge.net/projects/mingw/files/MinGW%20Runtime/mingwrt-3.15/mingwrt-3.15-mingw32-dev.tar.gz/download
wget http://sourceforge.net/projects/mingw/files/MinGW%20Runtime/mingwrt-3.15/mingwrt-3.15-mingw32-dll.tar.gz/download
wget http://sourceforge.net/projects/mingw/files/MinGW%20API%20for%20MS-Windows/w32api-3.14/w32api-3.14-mingw32-dev.tar.gz/download
wget http://sourceforge.net/projects/mingw/files/GNU%20Binutils/Current%20Release_%20GNU%20binutils-2.19.1/binutils-2.19.1-mingw32-bin.tar.gz/download
wget http://sourceforge.net/projects/mingw/files/MinGW%20make/mingw32-make-3.81-20090910/mingw32-make-3.81-20090910.tar.gz/download
wget http://www.cmake.org/files/v2.6/cmake-2.6.2-win32-x86.exe
wget http://cloud.github.com/downloads/spring/spring/mingwlibs_v19.3.7z
wget http://downloads.sourceforge.net/sevenzip/7z465.exe
wget http://msysgit.googlecode.com/files/Git-1.6.5.1-preview20091022.exe

# (ideally we'd also download a jdk and install that too)

# back in instance window:
# run 7z465.exe from c:\mingw32, "next", "next", "next", ...
# run git-1.6.5.1-preview ... exe, 'next', 'next', 'next', ...
# run cmake-2.6.2-win32-x86.exe, 'next', ...

# back in ssh:
# use '\program files\7-zip\7z.exe x' to extract each file in c:\mingw32:
set PATH=%PATH%;c:\program files\7-zip
7z x gcc-full-4.4.0-mingw32-bin-2.tar.lzma
7z x mingwrt-3.15-mingw32-dev.tar.gz
7z x mingwrt-3.15-mingw32-dll.tar.gz
7z x w32api-3.14-mingw32-dev.tar.gz
7z x mingw32-make-3.81-20090910.tar.gz
7z x binutils-2.19.1-mingw32-bin.tar.gz
7z x mingwlibs_v19.3.7z

# sleep or something then:
7z x gcc-full-4.4.0-mingw32-bin-2.tar
7z x mingwrt-3.15-mingw32-dev.tar
7z x mingwrt-3.15-mingw32-dll.tar
7z x w32api-3.14-mingw32-dev.tar
7z x mingw32-make-3.81-20090910.tar
7z x binutils-2.19.1-mingw32-bin.tar 

cd \
c:\program files\git\bin\git clone git://github.com/hughperkins/springheadless.git
cd \springheadless
"c:\program files\git\bin\git" checkout -b springheadless origin/springheadless

move \mingw32\mingwlibs_v19.3 \springheadless\mingwlibs

mkdir c:\springheadless\build
cd \springheadless\build
set MINGDIR=C:/MinGW32
set PATH=%PATH%;c:\tools;%MINGDIR%\bin;C:\Program Files\CMake 2.6\bin
set GAMEDIR=c:\springheadless\game

# doing cmake from commandline doesn't work for me.  I tried:
# cmake -G "MinGW Makefiles" -D AIDIR:STRING=%GAMEDIR% -D AI_DATA_DIR:STRING=%GAMEDIR% -D AI_LIBS_DIR:STRING=%GAMEDIR% -D CMAKE_INSTALL_PREFIX:STRING=%GAMEDIR% -D DATADIR:STRING=%GAMEDIR% -D LIBDIR:STRING=%GAMEDIR% ..
# cmake -D AIDIR:STRING=%GAMEDIR% -D AI_DATA_DIR:STRING=%GAMEDIR% -D AI_LIBS_DIR:STRING=%GAMEDIR% -D CMAKE_INSTALL_PREFIX:STRING=%GAMEDIR% -D DATADIR:STRING=%GAMEDIR% -D LIBDIR:STRING=%GAMEDIR% ..

# so instead, in instance:
set PATH=%PATH%;c:\tools;%MINGDIR%\bin;C:\Program Files\CMake 2.6\bin
cmake-gui
set source path and build path, then configure
change AIDIR, BINDIR, CMAKE_INSTALL_PREFIX, DATADIR and LIBDIR to c:/springheadless/game
configure again, then generate

# back in ssh session:
# from build directory:
make

It's a mixture of scriptable stuff, and stuff where you have to point and click, but it's probably a lot quicker to follow than to do it all by hand the first time.... I guess?

[hughperkins]

doesn't compile on mingw with cmake due to process creation length limit

I guess this is the error 'the parameter is incorrect'?

The link command for mingw32 seems to be in the file [builddir]/tools/HeadlessSpring/CMakeFiles/spring-headlessstubs.dir/link.txt

It looks like this command is evaluated at the command prompt, since it starts with:

Code: Select all

C:\MinGW32\bin\g++.exe 

One difference between the spring-headless link command and the link command for actual spring could be that each object file is prefixed with 'CMakeFiles\spring-headlessstubs.dir\__\__\' whereas for the actual spring link I guess this will be something quite a lot shorter like 'CMakeFiles\spring.dir\'

I guess the link failing is a bug in cmake in a way, since ideally it would take care of this stuff for us...

In the meantime, I guess three ideas:
- shorten the name from spring-headlessstubs to just spring-hl
- move it up in the directory path to remove one or two ..\
- break the link into two, by building a shared object first, then linking with that (linking statically won't work I think, I already tried that a few weeks ago: at link time, only things necessary to satisfy immediate link dependencies end up in the final statically-linked exe, and certain derived classes seemed to be missing; I'm guessing if it was a dynamically-linked dll, the link step would be at runtime and this issue would'nt arise?)
- something else...?

The current link.txt for spring-headlessstubs is 35786 bytes for mingw32 I think. The one for spring is 25155 characters. The exe with the limitation seems to be g++, or maybe the windows command shell.

[me goes off and googles command-line length windows ]

Hmmmm, that's odd. The maximum command-line length in Windows is only 8191 ( characters http://support.microsoft.com/kb/830473 ). The maximum length of an environment variable is 32,767 characters, which corresponds to the lengths we're seeing: 35786 is just longer than 32767, and 25155 is not. I guess that mingw32-make, or maybe cmake, somehow cleverly dumps link.txt into an environment variable to pass it into g++?

Soo....

Replacing 'spring-headlessstubs' by 'spring-hl' gets link.txt down to 30935 characters (doing a simple search and replace in gedit), so I'll try renaming the target to spring-hl and rebuilding, and maybe that will work?

[hughperkins]

Ok, it built, albeit with a weird warning message:

Code: Select all

Linking CXX executable ..\..\spring-hl.exe                                      
Creating library file: libspring-hl.dll.a                                       
Warning: resolving _wglGetProcAddress@4 by linking to _wglGetProcAddress        
Use --enable-stdcall-fixup to disable these warnings                            
Use --disable-stdcall-fixup to disable these fixups                             
[100%] Built target spring-hl  

Anyone know if this is normal or not in mingw32 and if it's an issue or not and anyway to get rid of the warning, by fixing it or suppressing it?

[hughperkins]

Also, nothing to do with spring-headless, but when I did a 'mingw32-make install' it just builds everything, or tries to, and this came up:

Code: Select all

[ 50%] Building C object AI/Interfaces/C/CMakeFiles/C-AIInterface.dir/__/__/Wrap
pers/CUtils/SharedLibrary.c.obj                                                 
In file included from C:\springheadless\AI\Wrappers\CUtils\SharedLibrary.c:21:  
c:\mingw32\bin\../lib/gcc/mingw32/4.4.0/../../../../include/stdio.h: In function
 'fprintf':                                                                     
c:\mingw32\bin\../lib/gcc/mingw32/4.4.0/../../../../include/stdio.h:245: error: 
conflicting types for '__p___argv'                                              
c:\mingw32\bin\../lib/gcc/mingw32/4.4.0/../../../../include/stdlib.h:78: note: p
revious declaration of '__p___argv' was here                                    
mingw32-make[2]: *** [AI/Interfaces/C/CMakeFiles/C-AIInterface.dir/__/__/Wrapper
s/CUtils/SharedLibrary.c.obj] Error 1                                           
mingw32-make[1]: *** [AI/Interfaces/C/CMakeFiles/C-AIInterface.dir/all] Error 2 
mingw32-make: *** [all] Error 2  

Edit: hmmm, I guess this is a problem with my mingw installation rather than with the code? Seems to be some conflicting include files?

Edit2: problem goes away by including <stdio.h> before other stuff:

Code: Select all

#include <stdio.h>

#include "SharedLibrary.h"

I also had to do the same thing for ... hmmm that didn't work, for Util.c , so I did a quick hello world to check my installation:

Code: Select all

#include <stdlib.h>
#include <stdio.h>

int main( int argc, char *argv[] ) {
   printf("Hello world\n");
   return 0;
}

saved as 'test.c', compiled using gcc -> worked ok. Reversed the order of stdio and stdlib, and compiled and worked fine ok too.

Edit3: fixed Util.c, by including stdio.h first:

Code: Select all

#include <stdio.h>       // fgets()
#include <string.h>      // strcpy(), str...()
#include <stdlib.h>      // malloc(), calloc(), free()
#include <stdarg.h>      // var-args

Seems very hacky, maybe there is somethign wrong with my mingw installation? The whole installation process of mingw itself doesn't feel very 'controlled', not like using apt-get for example, or having a single mingw installer that handles everything. Plus, maybe not many people use C, so there is a bug somewhere in mingw?

[Tobi]

Hmmmm, that's odd. The maximum command-line length in Windows is only 8191 ( characters http://support.microsoft.com/kb/830473 ). The maximum length of an environment variable is 32,767 characters, which corresponds to the lengths we're seeing: 35786 is just longer than 32767, and 25155 is not. I guess that mingw32-make, or maybe cmake, somehow cleverly dumps link.txt into an environment variable to pass it into g++?

Isn't this just when you actually use the command-line, or something equivalent like system() libc call?

scons and cmake (probably) both use the CreateProcess API, therefore bypassing the command-line. The CreateProcess API has a maximum limit to the command line arguments of 32766 characters.

(For scons, I actually had to implement this hack to make it use the CreateProcess API.)

I've no clue how to get command-lines even longer than that, I think then you need to implement some hack in the called process, like a single command-line argument that specifies a file with other command-line arguments.

[hughperkins]

scons and cmake (probably) both use the CreateProcess API, therefore bypassing the command-line. The CreateProcess API has a maximum limit to the command line arguments of 32766 characters.

Ah ok, that makes sense!

I've no clue how to get command-lines even longer than that, I think then you need to implement some hack in the called process, like a single command-line argument that specifies a file with other command-line arguments.

I feel that a 'clean' solution could be that g++ et al provided the option to give the names of the files in a file, and then just pass that single file's filename in the commandline, but that's going waaaayyy out of the scope of Spring Edit: Ooops that's what you said

[hughperkins]

In relation to getting mingw32 to build spring-hl, it built to completion ok with the new name, so I've checked in a modified CMakeLists.txt with the new name.

I also checked in tweaks to the include order to CUtils/Cutil.c and SharedLibrary.c, which should probably be checked in directly to spring git itself probably, rather than to springheadless, but I felt it was as good a place to put the changes as any....

[aegis]

it failed with a different error when I tried manually passing link.txt, and yeah, the error was the 32kb CreateProcess limit.

[AF]

I still think your jumping the gun. Trying to jump ahead and secure a system when it doesn't even implement the basic premise that launched the project to begin with seems a bit distorted.....

In my opinion your in very real danger of this still not having fulfilled its original purpose at a basic level 6 months from now, despite rampant development and code work.

[hughperkins]

I still think your jumping the gun. Trying to jump ahead and secure a system when it doesn't even implement the basic premise that launched the project to begin with seems a bit distorted.....

Hi AF, yes your December deadline is looming quickly!

I think the functionalities you are missing are the league statistics and analysis? As you say these are not implemented at this time, nor am I actively working on this stuff, since I am looking for a job, in theory

You could rewrite the whole AILadder/SpringGrid infrastructure in php, but... so let's go over the infrastructure quickly. We have three parts:

- botrunners
- SpringGrid: botrunner manager
- AILadder: league manager

SpringGrid is by far the most complex and biggest chunk of code of the three. It is a lot of code, it is fairly complete, and I feel it works quite well. It handles a queue of matches, which are distributed to the botrunners, and it collects the results from the botrunners. It includes the botrunner code itself, to run the matches.

AILadder is a very thin veneer over the top. It chats with SpringGrid in order to run matches, using an xmlrpc layer. To be honest, you could probably write your own version of AILadder in php if you are pressed for time and don't want to tweak the existing version. I feel that's not a bad idea to be honest.

Botrunners... well, I struggle to find botrunners that work when I want, and without needing to spend 2 hours connecting to irc to beg people to start some for me but for you they are easy I think since you can just use your university infrastructure? The code for the botrunners belongs to the springgrid project now, and it works ok I feel. Certainly to write your own would be a fairly big job I think, in order to match the same level of completeness and robustness.

So, to recap:

- botrunners. You can run these in your university infrastructure, not a problem for you
- SpringGrid, ie the botrunner manager. This is fairly complete. I feel you might consider using this for your project.
- AILadder, ie the league manager. This is a pretty thin veneer at the moment. I feel you could write your own in php quite quickly, and if you need it working quickly I feel this is not a bad option.

[hughperkins]

it failed with a different error when I tried manually passing link.txt, and yeah, the error was the 32kb CreateProcess limit.

Does the latest git version of springheadless, ie the branch 'origin/headless' in git://github.com/hughperkins/springheadless.git , fix the issue for you?

[AF]

I cannot use my university infrastructure actually, I cannot even launch msn without resorting to the decrepit windows messenger.

What I do have are a macbook and a desktop with decent specs with Vista and windows 7 running on them.

I am not sure how I would setup spring grid or AI Ladder on my own hosts and I'm afraid I have already ran out of time by a long shot if I take the prospect of a php ladder site in.

I abandoned my php site in September under the understanding that the most basic functionality aka the one that was the whole point of the project was round the corner, and its been 3 months and no progress has been made on it despite large development work.

If you want a job out of this, glowing praise and actual use would be more valuable than an endless development dialogue, its the difference between actually delivering, and technical filibustering.

[hughperkins]

I cannot use my university infrastructure actually, I cannot even launch msn without resorting to the decrepit windows messenger.

Ok, so you're in the same situation as me, so you either have to persuade people to run botrunners (maybe you can?), or you can take advantage of EC2, which is what I'm working on at the moment...

By the way, all of this stuff will run at the moment, without me working on the EC2 stuff: all I'm really looking at is packaging, but it's quite possible to run stuff without easy-to-use packaging.

As for the statistics, without match results, there is little incentive to write a statistics package. Having easy-to-use, readily available botrunner machines is a pre-requisite for writing the statistics package. At least: it's not a prequisite for writing it, but it's a pre-requisite for wanting to write it, which in an opensource environment comes to the same thing.

[AF]

I have the machines available to set up as bot runners, after all I can leave this machine on for a day and run through a tonne, but at the end of it, what do I get out of it, my AI isn't even listed in the site, and I have no idea how to register on the site nm login to set them up. There isn't even a ladder on the ladder site.

What I want is basically a league listing that's just #1 AAI #2 BAI #3 CAI #4 DAI etc etc etc Just like a player ladder

[hughperkins]

If you want a job out of this, glowing praise and actual use would be more valuable than an endless development dialogue, its the difference between actually delivering, and technical filibustering.

True... but in fact my experience so far is that no-one cares about my opensource experience. They just ask about my professional experience. Maybe if I'd just left college, but I haven't. So I put it on there anyway, but all it does is make my cv look a little non-standard and maverick, but anyway, I'm just doing this for fun, not like someone who is getting a masters out of it

Anyway, if you want this stuff working for your masters, there are a few choices:
- keep insulting me, best choice I reckons
- write your own from scratch entirely
- rewrite just AILadder
- add the missing functionalities to AILadder... It's Python code, it's not that scary. You're actually getting a thesis out of this, and the functionality you want is probably about three days work. Have a look at http://en.wikipedia.org/wiki/Chess_rating_systems , or Google TrueSkill and so on
- get a grid running, eg in EC2, with a bunch of league results on different maps and stuff, awaiting simply someone to come along and write a statistics analysis page...