faster (re-)compiles

faster (re-)compiles

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Post Reply
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

faster (re-)compiles

Post by abma »

it seems ninja becomes stable:

in ubuntu 12.10 add this ppa:
https://launchpad.net/~jpakkane/+archive/ppa

type

Code: Select all

cmake -G Ninja . 
and then

Code: Select all

ninja
when recompiling its awesome fast in comparison to GNU-make.


a "null" compile:

Code: Select all

ninja
[2/2] Running utility command for generateVersionFiles
Spring engine version: 91.0.1-358-g782f47f develop ()
[2/2] cd /home/abma/dev/spring/develop/cont/base && sh ./make_gamedata_arch.sh /home/abma/dev/spring/develop/base /usr/bin/7z
Using build directory: /home/abma/dev/spring/develop/base/ ...
Creating spring/bitmaps.sdz
Creating springcontent.sdz
Creating maphelper.sdz
Creating cursors.sdz

real	0m0.573s
user	0m1.112s
sys	0m0.132s
Last edited by abma on 12 Nov 2012, 05:05, edited 2 times in total.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: faster (re-)compiles

Post by SpliFF »

Awesome, will look into this.

One of my pet peeves is how ridiculously inefficient the whole make/cmake/autotools etc process can be. I've tried using tools like ccache and distcc but haven't been impressed with the results (esp when they break builds). Will see if Ninja can do better.

On the topic of compile times I'd like to propose we disable compile-time support for about 80% of the ASSIMP formats. The only *useful* formats IMHO are OBJ, MD*, 3DS, DXF, X, LWO, Milkshape, Collada and Blend. The rest are legacy, fringe, buggy (eg, Irrlich) or too limited in features to be useful (tools exist to convert these anyway).

Should just require modifying the CMake file? Also, the ASSIMP library now supports *export*. It might end up being more practical to only support Collada, OBJ and S3O/3DO in RELEASE builds and ship a small model converter tool for everything else.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: faster (re-)compiles

Post by abma »

as assimp has few / no dependencies in spring, you mostly will compile it once... this only will really speed up compile if you do a clean compile...

also, ccache only does help, if you increase cache size to somethink like >10GB as afaik a single spring compile uses ~3GB (which means, ccache already starts to discard files, when it isn't finished the first compile ...)

yeah, i would guess disabling formats would only require to comment out some lines here: https://github.com/spring/spring/blob/d ... s.txt#L599 (but i don't see really improvements in speedup compile, as its only compiled once). it would help much more to make stuff in spring more modular, so it is compiled only once... i'm currently trying to do so with parts of the filesystem (Archives).
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: faster (re-)compiles

Post by SpliFF »

Incidently it looks like ASSIMP is now a stable system package in ubuntu and fedora and available on Gentoo via zugaina overlay. Provided the library is capable of producing consistent results across systems/versions without our tweaks then we might not be far away from retiring the static version entirely and rolling it into mingwlibs as a DLL. Not sure the status on MacOS though.

Anyway, sorry for derail... Go Ninja!..
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: faster (re-)compiles

Post by hoijui »

for spring devs, the main problem i see in general, is the thing i discussed in the past already:
when switching from one git branch to an other, and back, without doing anything else, and using a separate build dir for each branch, you still have to re-compile all the files that differ between the two branches now. this leads to one of two things:
1. devs will either not want to switch branches, which makes them less agile, and which blocks them from effectively using one of the big advantages of git
2. or they have to maintain a separate local dir for each branch they are compiling, which allows them to work on different branches with a little less hassle. but it is still far away from what git would allow, and needs lots of scripting work to be used somewhat agile.
i had an idea for how to solve this, and started implementing a script, but never finished it. the idea would have been about like this:
1. before switching to an other branch with git, the script would ask make, what files it would compile if it were to be run right now. this list of files would be written to a special file related to this branch.
2. then the git branch switch is performed
3. somewhen later a switch back occurs
4. right after that switch back to the original branch, make is told to treat all files as "do-not-require-compiling"
5. then we perform a touch on all the files enlisted in the file created in step 1.

i guess that ninja does not solve that problem, does it?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: faster (re-)compiles

Post by abma »

with (big) ccache & ninja i guess you get a nice speed when switching branches... and no, i don't think ninja knows about git or branches, as it needs for example CMake to generate the build files.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: faster (re-)compiles

Post by hoijui »

yeah.. it surely does not know about branches/git, but it would work if it would check for file changes instead of timestamps, but as it is trimmed on speed, it surely does not do that either.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: faster (re-)compiles

Post by jK »

Ninja doesn't produce working debugsymbols here :<

Edit: gdb works, spring internal one doesn't work. Also it doesn't include the PWD to the datadirs o_O

edit2: strange, after a `rm CMakeCache.txt` & cmake regeneration it works now.
zerver
Spring Developer
Posts: 1358
Joined: 16 Dec 2006, 20:59

Re: faster (re-)compiles

Post by zerver »

I am amazed multithreading is still not a mainstream feature among all compilers and build systems.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: faster (re-)compiles

Post by abma »

Note: ninja works on win32, a .exe also exists: https://github.com/downloads/martine/ni ... 15-win.zip

it automaticly uses all cores... so compiling is much faster on windows, too...
Post Reply

Return to “Engine”