Statically linked Spring

Statically linked Spring

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Post Reply
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14673
Joined: 17 Nov 2005, 02:43

Statically linked Spring

Post by Forboding Angel »

Ok so I have heard reports that JK accomplished a "Statically linked as possible" build of spring.

Would it not be possible to distribute that via the package managers and just have it connect to the stuff that isn't set statically, on the user's machine?

In this way, couldn't we do multiple engine support and 'near' portable builds fairly easily on linux?
User avatar
koshi
Lobby Developer
Posts: 1059
Joined: 14 Aug 2007, 16:15

Re: Statically linked Spring

Post by koshi »

We're not quite there yet by my latest knowledge of his progress. Unitsync with no runtime dependencies is still a huge pita and would require still more effort to prepare a fitting build setup.
Once/if that is done the only thing I see missing is repeatability. As in building such a mostly-static package should be integrated in some automated build process.
Only then I'd say it would be ready for wide-spread use.
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14673
Joined: 17 Nov 2005, 02:43

Re: Statically linked Spring

Post by Forboding Angel »

So theoretically possible, but basically unitsync would need some loving?

We should get on that! If we could accomplish on linux what we currently can in windows, it would be a HUGE boon for everything and everyone involved.
User avatar
koshi
Lobby Developer
Posts: 1059
Joined: 14 Aug 2007, 16:15

Re: Statically linked Spring

Post by koshi »

Not unitsync itself needs loving, but it needs a build env where every lib in its closure of dependencies is compiled with a certain flag. That may or may not already be the case for any given lib, but it certainly not for a some that jk could confirm.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Statically linked Spring

Post by Licho »

It could be useful even without unitsync if engine itself can be built self contained.

Lobby could work without unitsync. Plasma service provides a simple query itnerface that for md5 tells you what it is (map,mod, minimap, heightmap, modoptions etc..). Using this, lobby can function without unitsync (thats how zkl does multi engine atm).

Another option is using just one unitsync because nothing usually changes in unitsync returned data between two different unitsync/engine versions (except content spring hash which is now optional).
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Statically linked Spring

Post by jK »

DIY TUTORIAL

tools you need:
* linux (possible under mingw too, but ways more complicated)
* cmake-gui (CMAKE_VERBOSE_MAKEFILE=TRUE)
* editor of your favored choice (my is `nano`)
* ldd (`ldd spring` and `ldd -v spring`) shows dependencies of a ELF binary
* pkg-config (`pkg-config --list-all | grep -i %ideaOfPkgName` + `pkg-config --static --libs %realPkgName`)

gentoo related
* equery & emerge

So here what I have done:
1. I set CMAKE_VERBOSE_MAKEFILE=TRUE
2. c&p the last final spring binary linking cmd into a `compile_static.sh`
3. reorganized it for readability
4. added "-static-libstdc++ -static-libgcc"
5. run the script and checked the binary's linked .so's (`ldd spring`) -> nothing changed final binary was still linking to the shared objects of those. reason for that is that other linked .so depend on those, so we need to first get rid of the other shared object (SDL, DevIL, ...)
6. run `ldd -v spring` to get the .so dependencies in a tree like form (first try to get rid of those library with the most less dependencies themselves)
7. started moving to move single packages from shared linking (-Wl,-Bdynamic) to static linked (-Wl,-Bstatic) -> you either get these case:
7.i an unresolved linker error occurs: means don't have the static .a version of the library installed. On gentoo this is mostly easy to solve:
7.i.a use `equery b /usr/lib/libXYZ.so` to find out to which ebuild the .so belongs
7.i.b recompile the package with `USE="static-libs" emerge %packagename
7.ii it compiled fine, but `ldd` shows the package is still linked dynamically
7.ii.a find pkgname with pkg-config and then check the dependencies of the pkg (use the 2 cmds from above)
7.ii.b find ebuild of those libaries, note `-lXft` will link to `/usr/lib/libXft.so` so check its ebuild and recompile it the same way as in (7.i)
8. repeat (5.) - (7.)

Gentoo is missing some "static-libs" for a few packages (OpenAL & SDL) also there is no libGLU.a available (but there is a way to compile it yourself). OpenAL can static linked AFAIK, but SDL & some other GCC libs can't.
AFAIK the remaining dependencies can be ignored cause their ABI is stable (GCC, OpenAL, SDL, GLU), so it is enough to depend on that those libraries are installed on the client system (version doesn't matter!).

So you end with a final minimal binary that has like ~2/3 less dependencies.
Last edited by jK on 20 Mar 2012, 11:40, edited 1 time in total.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Statically linked Spring

Post by jK »

So this is obviously no solution for real releases. Your final script e.g. will fail when you add new cpp files or change buildprocess via cmake-gui (compile with DEBUG2 instead of RELWITHDEBINFO etc.).

Also one goal was/is it to remove any dependencies of `unitsync.dll` cause a lobby had conflicts with its own dependencies & the unitsync ones. So it would be nice to still link spring.exe dynamically, but unitsync.dll static.

So the wish is to integrate this process into cmake. Problem is CMAKE is a pain in the ass ... it's very inflexible and will always prefer dynamic linked objects and there is no native support to change this.
Still you can trick it with

Code: Select all

SET(CMAKE_FIND_LIBRARY_SUFFIXES .a ${CMAKE_FIND_LIBRARY_SUFFIXES})
But this way it will always prefer static linked objects (so you have the other extreme)
Neither can you use it on a global level (you can only enclose Find() call with it) cause as said not all libraries work when linked static.

If you got now the idea to add a

Code: Select all

code for spring binary
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a")
Find("freetype");
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so")

code for unitsync binary
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so")
Find("freetype");
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a")
it won't work cause CMAKE caches the results of Find() and so the 2nd call to it will always return the same results as the first call.

So CMAKE_FIND_LIBRARY_SUFFIXES won't allow you to link a library dyn. & statically in the same project.

Now there is a 2nd way, with:

Code: Select all

TARGET_LINK_LIBRARIES(foobar "/usr/lib/libdevil.a")
you can force CMAKE to use the static lib. You only need to transform all your ".so" strings into ".a" ones.
So I wrote a PREFERSTATIC(DEVIL_IL_LIBRARY_STATIC ${DEVIL_IL_LIBRARY}) macro, it searches for a static version of ${DEVIL_IL_LIBRARY} and saves it's location to ${DEVIL_IL_LIBRARY_STATIC} or if it didn't found a corresponding ".a" it will output the ".so" one.
This way you can link the same library differently in `spring` & `unitsync`.
Last edited by jK on 20 Mar 2012, 11:42, edited 1 time in total.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Statically linked Spring

Post by jK »

So you think everything is fine now? ZONK

Now we hit the problem koshi noted. You cannot just take your .a's and link them to your shared object (unitsync). For a binary this works fine, but shared objects need special .a's. Any .a linked in a shared object MUST be compiled with -fPIC, gentoo's ebuilds don't do so by default. Neither do I know if you want so (there must be some cons of -fPIC, else it would be default enabled).

unsolved:
So the aim is now to find an easy way to compile all linked libraries with -fPIC.

unsolved #2:
Find an easy way to cross-compile all linked libraries in gentoo. (in theory ebuilds are TARGET independent and you can cross-compile them for windows, but a few ebuilds just fail & or need to get adjusted, so it's not that simple).
wolas
Posts: 112
Joined: 30 Jul 2010, 20:40

Re: Statically linked Spring

Post by wolas »

http://www.stanford.edu/~pgbovine/cde.html

I taken amarok.1.4 from debian etch and run it on squeeze it does work :) I think some magick tricks can be learnt from it.
wolas
Posts: 112
Joined: 30 Jul 2010, 20:40

Re: Statically linked Spring

Post by wolas »

Wohoo I have kinda chrooted spring now. With *all* needed and not really needed shit to run it and even more I think. It even has my /etc/groups file dunno if it really relies on it...

And unpacked it weights only less than 200MB (spring is stripped) I think it should be possible to decrease size much more.

http://springfiles.com/sites/default/fi ... ick.tar.gz in case this link are not permanent pls notify me I will upload somewhere else now tested from 3 ip adress it does work.

This is self compiled spring on machine debian testing 64bit and NVIDIA blob drivers it is kinda buggy in fact it often doesnt exit after game end(not related to chooted package but just some bug)

So if you have 64bit machine + nvidia it really should run I guess and if this thing really would work why not to use it? I guess we need to remove all audio libraries and some others, but I m not guru here :)

http://www.stanford.edu/~pgbovine/cde/manual/ configuration is easy.

So pls 64 bit guys try to test it all you need is to do ./cde-package/spring.cde or if you want to create your build and paste here to test ./cde201209 spring with copied cde.options in same directory (by default it would package whole .spring directory and those are big :)

By the way it seems it doest use our .spring config but I guess thats easily fixable.
User avatar
Vadi
Posts: 446
Joined: 03 Jan 2008, 14:51

Re: Statically linked Spring

Post by Vadi »

Hey, I'm on 64bit nvidia too, so I tested it out per request.

'/home/wolas/' seems specific to your machine, and it tries to load the .springrc from there.
wolas
Posts: 112
Joined: 30 Jul 2010, 20:40

Re: Statically linked Spring

Post by wolas »

Paths are easily changed in that, but I used /home/ to prevent that dunno why it still searchs it here also dunno about performace impact. I m lazy to test it more :) Nobody really interested in this anyway for proper test just few virtualbox images or few free partitions and it is done.
User avatar
Vadi
Posts: 446
Joined: 03 Jan 2008, 14:51

Re: Statically linked Spring

Post by Vadi »

You replied too quick, I was hoping to upgrade my driver and post again. Anyway, I'm not sure what is the message from the nVidia driver about - this is a clean installation of 11.10, and the latest one available to me is 280. Regardless, all 3D games work, so I don't think it's an issue.

Launching spring w/o any mods or maps installed = OK
Trying out the Lobby Connect (WIP) = missing libgtk3:

vadi@gooseberry:~$ '/home/vadi/Desktop/spring_magick/cde-package/spring.cde'
Using configuration source: "/home/wolas/.springrc"
Error: FileConfigSource: Error: Could not write to config file "/home/wolas/.springrc"
[CMyMath::Init] CPU SSE mask: 127, flags:
SSE 1.0: 1, SSE 2.0: 1
SSE 3.0: 1, SSSE 3.0: 1
SSE 4.1: 1, SSE 4.2: 1
SSE 4.0A: 0, SSE 5.0A: 0
using streflop SSE FP-math mode, CPU supports SSE instructions
OS: Linux 3.0.0-17-generic #30-Ubuntu SMP Thu Mar 8 20:45:39 UTC 2012 x86_64
OS: 64bit native mode
[Watchdog] Installed (HangTimeout: 10sec)
Available log sections: Font, VFS, Piece, KeyBindings, Sound, CSMFGroundTextures, RoamMeshDrawer, BumpWater, DynWater, SkyBox, FarTextureHandler, ModelDrawer, OBJParser, WorldObjectModelRenderer, Texture, CollisionVolume, GroundMoveType, UnitScript, AutohostInterface, ArchiveScanner, Path, Model
Enabled log sections: Sound(LOG_LEVEL_INFO)
Enable or disable log sections using the LogSections configuration key
or the SPRING_LOG_SECTIONS environment variable (both comma separated).
Use "none" to disable the default log sections.
LogOutput initialized.
Spring 87.0
Build date/time: Mar 11 2012 18:41:20
Build environment: boost-104800, GNU libstdc++ version 20120301
Compiler: gcc-4.6.3
Using read-write data directory: /usr/local/bin/
Using read-only data directory: /usr/local/share/games/spring/
Scanning: /usr/local/share/games/spring/base
Supported Video modes: 192640x480, 640x400, 640x350, 576x432, 512x384, 416x312, 400x300, 360x200, 320x240, 320x200, 320x175
Error: API mismatch: the NVIDIA kernel module has version 280.13,
but this NVIDIA driver component has version 295.20. Please make
sure that the kernel module and all NVIDIA driver components
have the same version.
Video mode set to 1920x1080/32bit
[f=0000000] SDL version: 1.2.15
[f=0000000] GL version: 2.1.2 NVIDIA 280.13
[f=0000000] GL vendor: NVIDIA Corporation
[f=0000000] GL renderer: GeForce GTX 560/PCI/SSE2
[f=0000000] GLSL version: (null)
[f=0000000] GLEW version: 1.5.4
[f=0000000] Video RAM: unknown
[f=0000000] GL info:
haveARB: 1, haveGLSL: 0, ATI hacks: 0
FBO support: 1, NPOT-texture support: 1, 24bit Z-buffer support: 1
maximum texture size: 16384, compress MIP-map textures: 0
AL lib: pulseaudio.c:612: Context did not connect: Access denied
[f=0000000] [Sound] OpenAL info:
[f=0000000] [Sound] Available Devices:
AL lib: pulseaudio.c:612: Context did not connect: Access denied
[f=0000000] Joysticks found: 0
[f=0000000] Warning: Joystick 0 not found
[f=0000000] CPU Cores: 4
[f=0000000] CPU affinity not set
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pcm_dmix.c:957:(snd_pcm_dmix_open) The dmix plugin supports only playback stream
Cannot connect to server socket err = No such file or directory
Cannot connect to server socket
jack server is not running or cannot be started
[f=0000000] [Sound] ALSA Default
[f=0000000] [Sound] PortAudio Default
[f=0000000] [Sound] No Output
[f=0000000] [Sound] Device: ALSA Default
[f=0000000] [Sound] Vendor: OpenAL Community
[f=0000000] [Sound] Version: 1.1 ALSOFT 1.13
[f=0000000] [Sound] Renderer: OpenAL Soft
[f=0000000] [Sound] AL Extensions: AL_EXT_DOUBLE AL_EXT_EXPONENT_DISTANCE AL_EXT_FLOAT32 AL_EXT_IMA4 AL_EXT_LINEAR_DISTANCE AL_EXT_MCFORMATS AL_EXT_MULAW AL_EXT_MULAW_MCFORMATS AL_EXT_OFFSET AL_EXT_source_distance_model AL_LOKI_quadriphonic AL_SOFT_buffer_sub_data AL_SOFT_loop_points
[f=0000000] [Sound] ALC Extensions: ALC_ENUMERATE_ALL_EXT ALC_ENUMERATION_EXT ALC_EXT_CAPTURE ALC_EXT_disconnect ALC_EXT_EFX ALC_EXT_thread_local_context
[f=0000000] [Sound] Warning: EFX Supported: no
[f=0000000] [Sound] Max Sounds: 128
[f=0000000] [ArchiveScanner] Warning: No game selected checksum: not found (0)
[f=0000000] [ArchiveScanner] Warning: No map selected checksum: not found (0)
[f=0000000] Error: Segmentation fault (SIGSEGV) in spring 87.0
[f=0000000] Error: Stacktrace:
sh: addr2line: not found
[f=0000000] Error: <0> /lib/x86_64-linux-gnu/libpthread.so.0(+0xf030) [0x7fbcead7d030]
[f=0000000] Error: <1> /usr/local/bin/spring() [0x7e4fc0]
[f=0000000] Error: <2> /usr/local/bin/spring() [0x7d6a67]
[f=0000000] Error: <3> /usr/local/bin/spring() [0xa833d4]
[f=0000000] Error: <4> /usr/local/bin/spring() [0x7d6b13]
[f=0000000] Error: <5> /usr/local/bin/spring() [0x7d92ad]
[f=0000000] Error: <6> /usr/local/bin/spring() [0x7d1e57]
[f=0000000] Error: <7> /usr/local/bin/spring() [0x7d11c2]
[f=0000000] Error: <8> /usr/local/bin/spring() [0x7d29d0]
[f=0000000] Error: <9> /usr/local/bin/spring() [0x7d2a04]
Error: Spring crashed Spring has crashed:
Segmentation fault (SIGSEGV).

A stacktrace has been written to:
/usr/local/bin/infolog.txt
zenity: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
Error: Spring crashed
Spring has crashed:
Segmentation fault (SIGSEGV).

A stacktrace has been written to:
/usr/local/bin/infolog.txt
vadi@gooseberry:~$ cat /usr/local/bin/infolog.txt
cat: /usr/local/bin/infolog.txt: No such file or directory
vadi@gooseberry:~$
I'll grab some mods & maps into my .spring and test more.

edit: "AL lib: pulseaudio.c:612: Context did not connect: Access denied" might be a bit of a warning sign there for missing sound later on... I certainly didn't have any sound in the menu, and my audio is OK.
wolas
Posts: 112
Joined: 30 Jul 2010, 20:40

Re: Statically linked Spring

Post by wolas »

You see this thing is very experimental. I could have removed all libs related to nvidia and gfx, but well I was unsure how it would react. With sound I predicted that this could happen(pulseaudio configures per user or something and it is prone to fail) thats why etc folder is here, but I see it didint helped at all maybe it should be removed as well.

In fact I m not linux guru, but with a bit more fidling and removing bad libraries such as nvidia ones and maybe sound so spring probably would detect them in new system this would work.

And anyway this should be done in a way that you could choose spring version. Maybe wrong thing was that there is "search paths" and not specified exactly each lib and file which is required to run spring thats why it fails. Maybe if I have time and patience I will install some feodora or whatever different from debian architecture and see how this would work.

By the way that infolog should be in magick-spring folder :)
Post Reply

Return to “Engine”