Separating/Modularizing the View

[hughperkins]

Right, next question: if we're going to consider stubbing out SDL_video stuff, how much stuff would that actually mean stubbing out?

The following command should give all the usages of SDL_video functions, with a few false positives thrown in (input and stuff):

user@701:/data/taspring/git/spring/rts$ find . -name "*.cpp" -print | xargs -l1 grep SDL_ | grep -v "case SDL" | grep -v SDL_Event | grep -v "#include" | grep -v SDL_GetTicks | grep -v SDL_BUTTON | grep -v SDL_MOUSE| grep -v SDL_RELEASED | grep -v SDL_PRESSED >/tmp/sdlrefs.txt:

The result is here:

http://pastebin.com/fdfd4cca

... so basically, 134 non-unique function calls, including false positives. The actual functions called are pretty much mostly ones already stubbed in the prototype:

- sdl_gl_setvideomode
- sdl_gl_getattribute
- sdl_gl_setattribute
- sdl_gl_swapbuffers

A few others, not already stubbed out, but shouldn't be terribly difficult I think?:
- sdl_wm_seticon
- sdl_wm_setcaption
- sdl_getwminfo

Open question in my mind: to what extent do we have to stub out also the input and sound stuff?

[hughperkins]

Hey! Random thought!

If we're thinking of possibly using special linking to create a headless client... really quick hack to make spring headless:
- make our own opengl_stub library
- make our own sdl_stub library

Both libraries basically just do nothing, but implement all the function calls we're using.

So, if you want a normal head... full(?) client, then you link normally:

Code: Select all

g++ -o spring *.o -lGL -lSDL ...

... and if you want a headless client, you link with the stub libraries:

Code: Select all

g++ -o spring *.o -lGLstub -lSDLstub ...

It'd be quick and dirty, but it might well work really easily?

[hughperkins]

Woot! Making stub libraries seems to work!

Prototype here:

http://manageddreams.com/ailadder/downl ... l2.tar.bz2

It's using the same original code as before. However... big difference ... didn't actually touch the code! I mean, I did: to add some printfs to check the thing is actually running, but... that was it! Didn't change any of the code, and compiled it against the normal SDL.h header file.

Then... created two stub libraries, one of which I thought would surely cause a segfault because the function declaraataions are totally wrong (glstub.c), but anyway, seem to work just fine!

sdlstub.c:

http://pastebin.com/f3fc343ae

glstub.c:

http://pastebin.com/f50381302

Then, built and linked like this:

Code: Select all

gcc -c -I/usr/include -I/usr/include/SDL sdlstub.c
ar rcs libSDLstub.a sdlstub.o

gcc -c -I/usr/include -I/usr/include/SDL glstub.c
ar rcs libGLstub.a glstub.o

gcc -c -I/usr/include -I/usr/include/SDL sdl_ogl.c
gcc -o sdl_ogl_headless -L. sdl_ogl.o -lGLstub -lSDLstub

And ... ploof! it runs on a console! Without X! Plus, it runs really fast since even SDL_Delay is stubbed out :-D

Obviously, I can't screenshot that... oh... I can take an awesome quality picture with my rubbish mobile phone... let me do thaat.. one second....



Edit: another shot, with the X server totally stopped this time:

[hughperkins]

Right, got somewhat far:

- created a big part of glstub.c
- created glewstub.c
- created pretty much all of sdlstub.c

I think we probably need to stub libX11 and libX11cursors too, but didn't do that yet. There's jsut a few function calls anyway, to these libraries.

running within X, but with SDL stubbed, so any non-stubbed gl call casus a seg fault, gets this far:

http://pastebin.com/f3f2a7de6

gf gets back from work soon. She'll kill me if I don't stop doing this and have a shower about half an hour ago

[hughperkins]

Created a git repository for the stubs here:

http://github.com/hughperkins/headlessopenglstubs

Edit: renamed repository

[momfreeek]

well played sir . win the debate by implementing the simplest solution before anyone has a chance to say its a stupid idea. nintuplet (?) posting is forgiven. I guess doing it the 'proper' way still has its advantages.

[hughperkins]

I guess doing it the 'proper' way still has its advantages.

Oh yeah for sure.

Basically:

Stubbing out hte libraries: quick, hacky solution. Will be discarded within a month, or never used.
Modularizing the view: decent, well-architected solution, that will last, and has lots of other advantages besdies.

So, what I figure is: create a hacky solution for today, to get it working, and then behind the scenes hoijui can carry on on the decent well-architected solution which will replace the quick hacky solution later on.

Sounds reasonable?

[imbaczek]

yes, as long as the hackish solution is indeed much easier to do. i'm afraid that the gl stub will have to be more than a simple stub due to gl being a state machine, but my gl-fu is too weak to be sure about it.

[Kloot]

For AI batch processing and a base for further prototyping, you may also be interested in http://github.com/Kloot/spring. It's a compilable and runnable fork of the engine with almost all unsynced code (rendering, sound, input, interface; the only dependencies besides boost* are SDL_timer and IL) removed, so you can't do much with the client except process scripts or demos from the command line, but it will work on headless systems.

[hughperkins]

Hi Kloot,

Wow! So, what's the downside? Why are we not using that already?

[Kloot]

Because it didn't exist until quite recently. The main downside is that the client can't be controlled anymore (eg. to set the game speed), but a simple CL interface would fix that.

[Tobi]

If you just want to force a higher then 1 gamespeed you can set both min and max gameSpeed in the startscript to a higher value. If I recall correctly this sets the initial gamespeed to the higher value too.

[hughperkins]

Well that sounds like just what we need!

What happens about versioning? I imagine it must be quite a challenge to keep it up-to-date wrt Spring master?

[hughperkins]

Right, the library stubs for Spring are completed. Spring can now run headlessly using them.

Tested with x windows shutdown: got all the way through initialization, loaded up the java AIs, and carried on executing.

The code for the headless stubs is at:

http://github.com/hughperkins/headlessopenglstubs

There is a spring executable, the one I built, for Ubuntu jaunty here:

http://manageddreams.com/ailadder/downl ... untujaunty

To build your own:

- download spring source
- I tested it with the 0.80.4 label on Spring git master
- build it to completion
- copy the files from the headlessspring git repository above into the 'rts' directory
- cd into the rts directory
- run './build.sh'
- copy the resulting 'springheadless' file into your favorite game directory
- change into your game directory, and do './springheadless script.txt' (you'll need an appropriate script.txt that has two AIs playing head to head against each other, and fixed start positions).

Note that it's tested/optimized for linux/Ubuntu.

Example infolog.txt, running two null java oo ais:

http://pastebin.com/f7c5c19bb

Edit:

Photos of the console:

Launch "./foo scriptoojava.txt"


Starting up:




Java AI initializing:


Edit: moved repository

[hoijui]

umm...
i am a bit tired, plus having my days..
why not making all this stuff available in a git fork/branch of spring?

one thing i did not understand:
you want to have a single executable that can run headless or with graphics? One for each is fine, and doing both in one would mean you had to dynamically load all graphic libs, as these should not have to be available on systems that want ot run headless. eg, for a headless build, you have to link without GL. i dont know about SDL... as part of it we use is headless (SDL_Ticks for timestamps (if i am not wrong)), and other stuff is graphics related (mouse handling?).
so in short:
the headless build would be something like the deidacted build. you may have a look at the CMake build file for it, or have a look at the headless buildfile in the headless branch. Both are under tools/.
If you need live help/hints, se me in the lobby.

edit:
forgot.. zerver did something comparable to what you described with the replacing of opengl cals for the multithreaded build of spring, if i was informed right.

and... good luck! :D
i am willing to help, but trying to put all together into a commitable form, while you know it all already...

[hughperkins]

Hi hoijui,

Ok, just wrote a lot, and then re-read your post, and realized you are not pushing for a single executable, but that you think that two is just fine

Ok, so ...

Just to look at your technical points:

The stubs are totally independent of the rest of Spring. They have no Spring includes or dependencies. They could be used in any opengl/glew/glu project. The only provisos:
- they do not implement the entire opengl API, so other opengl projects would need to add the missing methods (mind you, Spring uses a lot of methods, so most opengl projects wishing to use the stubs to be headless would probably only be missing one or two, but still)
- the glew stub is heavily tweaked to return only the extensions that Spring absolutely insists on, and also the glew stub is massively incomplete, we only stub the tiniest number of methods out of those available, and actually link to the real GLEW library just after the stub library; but this could be made programmatic, ie we could programatically control which stubs GLEW says it handles

These stubs could be in a library under lib perhaps? Or maybe in tools? Probably rts/lib could be ok though?

Looking at SDL, SDL is entirely stubbed out, and the real SDL library is not linked with at all. The stub returns sensible values for SDL_GetTicks(), using boost::xtime , and also implements SDL_Delay. PollEvent simply never returns any event.


Moving on to CMake stuff...

CMake seems to be massively better than scons! Still, I'm not an expert in CMake, not like you, but maybe I'll have a look at doing this.

The only bit of the headless spring that is spring-dependent at all is the link step, which at the moment looks like this:

Code: Select all

# points on link order:
# glewstub MUST be before glew
# streflop should be after lua
g++ -g -o springheadless -L. -Llib -Llib/streflop $(find . -name "*.o" -print |grep -v "testmain.o"  | grep -v "^./lib") -lSDLstub -lGLstub -lboost_system-mt -lboost_thread-mt -lboost_signals-mt -lopenal -logg -lboost_filesystem-mt -lboost_regex-mt -lboost_serialization-mt -lz -lboost_program_options-mt -lfreetype -lIL -lILU -lvorbis -lX11 -lXcursor -lvorbisenc -lvorbisfile -l7zip -lhpiutil2 -loscpack -lminizip -lglewstub -lglustub -lGLEW -llua -lstreflop

I'm not really sure how to integrate this into CMakeLists, though it's clearly the place where it belongs :-D I'll take a look and see how I go, and then hassle you in irc

[hughperkins]

Ok, so forked Spring, and added changes to CMakeLists.txt, here:

http://github.com/hughperkins/springhea ... its/master

- "headlessstubs" is a library under rts/lib
- tools/HeadlessSpring builds 'spring-headlessstubs' (I felt better not to have a name conflict with your own output )

Edited to truncate a little for concision.

[hughperkins]

A few tweaks:

rts/CMakeLists.txt exports its list of files to rts/../CMakeLists.txt like this:

Code: Select all

SET( allspringfiles ${gamefiles} ${luafiles} ${mapfiles} ${fsfiles} ${renderfiles} ${simfiles} ${sysfiles} ${aifiles} ${nedmalloc_obj} )
SET( allspringfiles ${allspringfiles} PARENT_SCOPE )

Each AUX_SOURCE_DIRECTORY now specifies the absolute path, rather than the relative path, like this:

Instead of:

Code: Select all

AUX_SOURCE_DIRECTORY(System/Script sysfiles)

we have:

Code: Select all

AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR}/System/Script sysfiles)

Then, it is easy to build from tools/HeadlessSpring, by simply doing:

Code: Select all

# need to add this so stuff compiles:
INCLUDE_DIRECTORIES(BEFORE ${SPRINGRTSDIR})
INCLUDE_DIRECTORIES(BEFORE ${SPRINGRTSDIR}/System)

ADD_EXECUTABLE(spring-headlessstubs ${allspringfiles})
TARGET_LINK_LIBRARIES(spring-headlessstubs ${spring_headlessstubs_libraries} )

.... where ${allspringfiles} comes directly from rts/MakeFiles.txt, so that's pretty easy.

Just waiting it to build to see what happens.

Edit:

Right, it built and ran ok:



From the console:





Normal spring built and ran ok too.

Could you check that the commit looks reasonable, and then push that onto spring master?

http://github.com/hughperkins/springheadless

[hoijui]

cool! :D
i did not have a look at the commits, but just reading the posts.. it sounds good :D
now i am gone for 2 or 3 hours, then gonna look at it and try to compile on my headless machine and.. cool! :D

[hoijui]

Code: Select all

git clone git://github.com/hughperkins/springheadless.git
Initialized empty Git repository in springheadless/.git/
fatal: The remote end hung up unexpectedly

something seems to be borked about your fork.
i tried to clone spring master on the same machine, and it worked, so it can not be my local setup.

there was once was an issue wiht the main repo, where it was unfetchable.. do not if it showed the same symptoms though.
i heard that github did move to a new server.. maybe this is related.

just tested, same thing happens on my other machine.
maybe somoene else shoudl try too, and if it is realy borked, maybe delete the repo and create a new fork and force push your local repo on there again.