Separating/Modularizing the View - Page 2

Separating/Modularizing the View

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 07:06

Right, next question: if we're going to consider stubbing out SDL_video stuff, how much stuff would that actually mean stubbing out?

The following command should give all the usages of SDL_video functions, with a few false positives thrown in (input and stuff):

user@701:/data/taspring/git/spring/rts$ find . -name "*.cpp" -print | xargs -l1 grep SDL_ | grep -v "case SDL" | grep -v SDL_Event | grep -v "#include" | grep -v SDL_GetTicks | grep -v SDL_BUTTON | grep -v SDL_MOUSE| grep -v SDL_RELEASED | grep -v SDL_PRESSED >/tmp/sdlrefs.txt:

The result is here:

http://pastebin.com/fdfd4cca

... so basically, 134 non-unique function calls, including false positives. The actual functions called are pretty much mostly ones already stubbed in the prototype:

- sdl_gl_setvideomode
- sdl_gl_getattribute
- sdl_gl_setattribute
- sdl_gl_swapbuffers

A few others, not already stubbed out, but shouldn't be terribly difficult I think?:
- sdl_wm_seticon
- sdl_wm_setcaption
- sdl_getwminfo

Open question in my mind: to what extent do we have to stub out also the input and sound stuff? :?:
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 07:37

Hey! Random thought!

If we're thinking of possibly using special linking to create a headless client... really quick hack to make spring headless:
- make our own opengl_stub library
- make our own sdl_stub library

Both libraries basically just do nothing, but implement all the function calls we're using.

So, if you want a normal head... full(?) client, then you link normally:

Code: Select all

g++ -o spring *.o -lGL -lSDL ...


... and if you want a headless client, you link with the stub libraries:

Code: Select all

g++ -o spring *.o -lGLstub -lSDLstub ...


It'd be quick and dirty, but it might well work really easily?
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 09:04

Woot! Making stub libraries seems to work!

Prototype here:

http://manageddreams.com/ailadder/downl ... l2.tar.bz2

It's using the same original code as before. However... big difference ... didn't actually touch the code! I mean, I did: to add some printfs to check the thing is actually running, but... that was it! Didn't change any of the code, and compiled it against the normal SDL.h header file.

Then... created two stub libraries, one of which I thought would surely cause a segfault because the function declaraataions are totally wrong (glstub.c), but anyway, seem to work just fine!

sdlstub.c:

http://pastebin.com/f3fc343ae

glstub.c:

http://pastebin.com/f50381302

Then, built and linked like this:

Code: Select all

gcc -c -I/usr/include -I/usr/include/SDL sdlstub.c
ar rcs libSDLstub.a sdlstub.o

gcc -c -I/usr/include -I/usr/include/SDL glstub.c
ar rcs libGLstub.a glstub.o

gcc -c -I/usr/include -I/usr/include/SDL sdl_ogl.c
gcc -o sdl_ogl_headless -L. sdl_ogl.o -lGLstub -lSDLstub


And ... ploof! it runs on a console! Without X! Plus, it runs really fast since even SDL_Delay is stubbed out :-D

Obviously, I can't screenshot that... oh... I can take an awesome quality picture with my rubbish mobile phone... let me do thaat.. one second....

Image

Edit: another shot, with the X server totally stopped this time:

Image
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 12:43

Right, got somewhat far:

- created a big part of glstub.c
- created glewstub.c
- created pretty much all of sdlstub.c

I think we probably need to stub libX11 and libX11cursors too, but didn't do that yet. There's jsut a few function calls anyway, to these libraries.

running within X, but with SDL stubbed, so any non-stubbed gl call casus a seg fault, gets this far:

http://pastebin.com/f3f2a7de6

gf gets back from work soon. She'll kill me if I don't stop doing this and have a shower about half an hour ago :mrgreen:
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 13:03

Created a git repository for the stubs here:

http://github.com/hughperkins/headlessopenglstubs

Edit: renamed repository
Last edited by hughperkins on 29 Sep 2009, 04:46, edited 1 time in total.
0 x

User avatar
momfreeek
Posts: 625
Joined: 29 Apr 2008, 16:50

Re: Separating/Modularizing the View

Postby momfreeek » 28 Sep 2009, 13:23

well played sir :-). win the debate by implementing the simplest solution before anyone has a chance to say its a stupid idea. nintuplet (?) posting is forgiven. I guess doing it the 'proper' way still has its advantages.
Last edited by momfreeek on 28 Sep 2009, 13:28, edited 1 time in total.
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 13:27

I guess doing it the 'proper' way still has its advantages.


Oh yeah for sure.

Basically:

Stubbing out hte libraries: quick, hacky solution. Will be discarded within a month, or never used.
Modularizing the view: decent, well-architected solution, that will last, and has lots of other advantages besdies.

So, what I figure is: create a hacky solution for today, to get it working, and then behind the scenes hoijui can carry on on the decent well-architected solution which will replace the quick hacky solution later on.

Sounds reasonable?
0 x

imbaczek
Posts: 3629
Joined: 22 Aug 2006, 16:19

Re: Separating/Modularizing the View

Postby imbaczek » 28 Sep 2009, 13:37

yes, as long as the hackish solution is indeed much easier to do. i'm afraid that the gl stub will have to be more than a simple stub due to gl being a state machine, but my gl-fu is too weak to be sure about it.
0 x

Kloot
Spring Developer
Posts: 1811
Joined: 08 Oct 2006, 16:58
x 83

Re: Separating/Modularizing the View

Postby Kloot » 28 Sep 2009, 16:46

For AI batch processing and a base for further prototyping, you may also be interested in http://github.com/Kloot/spring. It's a compilable and runnable fork of the engine with almost all unsynced code (rendering, sound, input, interface; the only dependencies besides boost* are SDL_timer and IL) removed, so you can't do much with the client except process scripts or demos from the command line, but it will work on headless systems.
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 16:48

Hi Kloot,

Wow! So, what's the downside? Why are we not using that already?
0 x

Kloot
Spring Developer
Posts: 1811
Joined: 08 Oct 2006, 16:58
x 83

Re: Separating/Modularizing the View

Postby Kloot » 28 Sep 2009, 17:47

Because it didn't exist until quite recently. ;) The main downside is that the client can't be controlled anymore (eg. to set the game speed), but a simple CL interface would fix that.
0 x

Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Separating/Modularizing the View

Postby Tobi » 28 Sep 2009, 17:50

If you just want to force a higher then 1 gamespeed you can set both min and max gameSpeed in the startscript to a higher value. If I recall correctly this sets the initial gamespeed to the higher value too.
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 18:07

Well that sounds like just what we need!

What happens about versioning? I imagine it must be quite a challenge to keep it up-to-date wrt Spring master?
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 28 Sep 2009, 21:09

Right, the library stubs for Spring are completed. Spring can now run headlessly using them.

Tested with x windows shutdown: got all the way through initialization, loaded up the java AIs, and carried on executing.

The code for the headless stubs is at:

http://github.com/hughperkins/headlessopenglstubs

There is a spring executable, the one I built, for Ubuntu jaunty here:

http://manageddreams.com/ailadder/downl ... untujaunty

To build your own:

- download spring source
- I tested it with the 0.80.4 label on Spring git master
- build it to completion
- copy the files from the headlessspring git repository above into the 'rts' directory
- cd into the rts directory
- run './build.sh'
- copy the resulting 'springheadless' file into your favorite game directory
- change into your game directory, and do './springheadless script.txt' (you'll need an appropriate script.txt that has two AIs playing head to head against each other, and fixed start positions).

Note that it's tested/optimized for linux/Ubuntu.

Example infolog.txt, running two null java oo ais:

http://pastebin.com/f7c5c19bb

Edit:

Photos of the console:

Launch "./foo scriptoojava.txt"
Image

Starting up:
Image

Image

Java AI initializing:
Image

Edit: moved repository
Last edited by hughperkins on 29 Sep 2009, 04:46, edited 1 time in total.
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Separating/Modularizing the View

Postby hoijui » 28 Sep 2009, 22:16

umm...
i am a bit tired, plus having my days.. ;-)
why not making all this stuff available in a git fork/branch of spring?

one thing i did not understand:
you want to have a single executable that can run headless or with graphics? One for each is fine, and doing both in one would mean you had to dynamically load all graphic libs, as these should not have to be available on systems that want ot run headless. eg, for a headless build, you have to link without GL. i dont know about SDL... as part of it we use is headless (SDL_Ticks for timestamps (if i am not wrong)), and other stuff is graphics related (mouse handling?).
so in short:
the headless build would be something like the deidacted build. you may have a look at the CMake build file for it, or have a look at the headless buildfile in the headless branch. Both are under tools/.
If you need live help/hints, se me in the lobby.

edit:
forgot.. zerver did something comparable to what you described with the replacing of opengl cals for the multithreaded build of spring, if i was informed right.

and... good luck! :D
i am willing to help, but trying to put all together into a commitable form, while you know it all already...
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 29 Sep 2009, 04:20

Hi hoijui,

Ok, just wrote a lot, and then re-read your post, and realized you are not pushing for a single executable, but that you think that two is just fine :-)

Ok, so ...

Just to look at your technical points:

The stubs are totally independent of the rest of Spring. They have no Spring includes or dependencies. They could be used in any opengl/glew/glu project. The only provisos:
- they do not implement the entire opengl API, so other opengl projects would need to add the missing methods (mind you, Spring uses a lot of methods, so most opengl projects wishing to use the stubs to be headless would probably only be missing one or two, but still)
- the glew stub is heavily tweaked to return only the extensions that Spring absolutely insists on, and also the glew stub is massively incomplete, we only stub the tiniest number of methods out of those available, and actually link to the real GLEW library just after the stub library; but this could be made programmatic, ie we could programatically control which stubs GLEW says it handles

These stubs could be in a library under lib perhaps? Or maybe in tools? Probably rts/lib could be ok though?

Looking at SDL, SDL is entirely stubbed out, and the real SDL library is not linked with at all. The stub returns sensible values for SDL_GetTicks(), using boost::xtime , and also implements SDL_Delay. PollEvent simply never returns any event.


Moving on to CMake stuff...

CMake seems to be massively better than scons! Still, I'm not an expert in CMake, not like you, but maybe I'll have a look at doing this.

The only bit of the headless spring that is spring-dependent at all is the link step, which at the moment looks like this:

Code: Select all

# points on link order:
# glewstub MUST be before glew
# streflop should be after lua
g++ -g -o springheadless -L. -Llib -Llib/streflop $(find . -name "*.o" -print |grep -v "testmain.o"  | grep -v "^./lib") -lSDLstub -lGLstub -lboost_system-mt -lboost_thread-mt -lboost_signals-mt -lopenal -logg -lboost_filesystem-mt -lboost_regex-mt -lboost_serialization-mt -lz -lboost_program_options-mt -lfreetype -lIL -lILU -lvorbis -lX11 -lXcursor -lvorbisenc -lvorbisfile -l7zip -lhpiutil2 -loscpack -lminizip -lglewstub -lglustub -lGLEW -llua -lstreflop


I'm not really sure how to integrate this into CMakeLists, though it's clearly the place where it belongs :-D I'll take a look and see how I go, and then hassle you in irc 8)
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 29 Sep 2009, 08:34

Ok, so forked Spring, and added changes to CMakeLists.txt, here:

http://github.com/hughperkins/springhea ... its/master

- "headlessstubs" is a library under rts/lib
- tools/HeadlessSpring builds 'spring-headlessstubs' (I felt better not to have a name conflict with your own output )

Edited to truncate a little for concision.
Last edited by hughperkins on 29 Sep 2009, 11:21, edited 2 times in total.
0 x

User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Separating/Modularizing the View

Postby hughperkins » 29 Sep 2009, 11:17

A few tweaks:

rts/CMakeLists.txt exports its list of files to rts/../CMakeLists.txt like this:

Code: Select all

SET( allspringfiles ${gamefiles} ${luafiles} ${mapfiles} ${fsfiles} ${renderfiles} ${simfiles} ${sysfiles} ${aifiles} ${nedmalloc_obj} )
SET( allspringfiles ${allspringfiles} PARENT_SCOPE )


Each AUX_SOURCE_DIRECTORY now specifies the absolute path, rather than the relative path, like this:

Instead of:

Code: Select all

AUX_SOURCE_DIRECTORY(System/Script sysfiles)


we have:

Code: Select all

AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR}/System/Script sysfiles)


Then, it is easy to build from tools/HeadlessSpring, by simply doing:

Code: Select all

# need to add this so stuff compiles:
INCLUDE_DIRECTORIES(BEFORE ${SPRINGRTSDIR})
INCLUDE_DIRECTORIES(BEFORE ${SPRINGRTSDIR}/System)

ADD_EXECUTABLE(spring-headlessstubs ${allspringfiles})
TARGET_LINK_LIBRARIES(spring-headlessstubs ${spring_headlessstubs_libraries} )


.... where ${allspringfiles} comes directly from rts/MakeFiles.txt, so that's pretty easy.

Just waiting it to build to see what happens.

Edit:

Right, it built and ran ok:

Image

From the console:

Image

Image

Normal spring built and ran ok too.

Could you check that the commit looks reasonable, and then push that onto spring master?

http://github.com/hughperkins/springheadless
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Separating/Modularizing the View

Postby hoijui » 29 Sep 2009, 12:59

cool! :D
i did not have a look at the commits, but just reading the posts.. it sounds good :D
now i am gone for 2 or 3 hours, then gonna look at it and try to compile on my headless machine and.. cool! :D
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Separating/Modularizing the View

Postby hoijui » 29 Sep 2009, 16:21

Code: Select all

git clone git://github.com/hughperkins/springheadless.git
Initialized empty Git repository in springheadless/.git/
fatal: The remote end hung up unexpectedly


something seems to be borked about your fork.
i tried to clone spring master on the same machine, and it worked, so it can not be my local setup.

there was once was an issue wiht the main repo, where it was unfetchable.. do not if it showed the same symptoms though.
i heard that github did move to a new server.. maybe this is related.

just tested, same thing happens on my other machine.
maybe somoene else shoudl try too, and if it is realy borked, maybe delete the repo and create a new fork and force push your local repo on there again.
0 x


Return to “Engine”