Building the official Spring 95.0 version for Debian

Building the official Spring 95.0 version for Debian

Discuss everything related to running Spring on your chosen distribution of Linux.

Moderator: Moderators

Post Reply
apoleon
Posts: 8
Joined: 12 Dec 2013, 21:19

Building the official Spring 95.0 version for Debian

Post by apoleon »

Hi all,

I'm currently in the process of adopting the Debian Spring package. I have successfully built 95.0 but encountered three issues. I'd be glad for some pointers here.

tl;dr

1. Tests don't work
2. Cannot pass custom CPPFLAGS or CXXFLAGS to CMake
3. Embedded libraries

1. Tests don't work

I would like to run the test suite but most tests fail like this one
Could not find executable /tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/test_SyncedPrimitive
Looked in the following places:
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Release/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Release/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Debug/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Debug/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/MinSizeRel/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/MinSizeRel/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/RelWithDebInfo/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/RelWithDebInfo/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Deployment/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Deployment/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Development/test_SyncedPrimitive
/tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Development/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Release/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Release/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Debug/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Debug/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/MinSizeRel/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/MinSizeRel/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/RelWithDebInfo/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/RelWithDebInfo/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Deployment/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Deployment/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Development/test_SyncedPrimitive
tmp/buildd/spring-95.0+dfsg/obj-x86_64-linux-gnu/test/Development/test_SyncedPrimitive
It seems those executables aren't built and

Code: Select all

make tests
is not a valid CMake target, contrary to the description in test/README.md.

How can I use the test suite?

2. Debian usually passes some additional CPPFLAGS and CXXFLAGS to the gcc compiler for hardening reasons. https://wiki.debian.org/Hardening

I haven't had any luck so far with adding additional flags to the build, e.g with -DCMAKE_CXX_FLAGS. I think it would make the whole engine more secure if options like -D_FORTIFY_SOURCE=2 or -fstack-protector --param ssp-buffer-size=4 were added.

The question is do you support hardening build flags (I read this could cause online sync problems) or what do you think in general about it?

3. Spring embeds a couple of well-known libraries such as lua and luasocket or assimp. Using embedded libraries is frowned upon in Debian and normally they are replaced by system libraries to make it easier to maintain the software in question and to fix security issues only once.

I did this for the assimp library but it seems your lua and luasocket versions are more difficult to replace. Is there a chance to get your changes into the upstream version of lua and luasocket?

Thanks for reading
Kloot
Spring Developer
Posts: 1867
Joined: 08 Oct 2006, 16:58

Re: Building the official Spring 95.0 version for Debian

Post by Kloot »

2) Passing those flags would be a good idea but none are proven to sync (neither in isolation nor in combination) and we must be as paranoid about that (even moreso in fact, since hopefully nobody runs Spring as root) as about system security.

3) Before you accidentally create packages that are useless for online play, please read http://springrts.com/phpbb/viewtopic.php?f=12&t=24813 because it should answer most of your questions.

(I do wish package maintainers would stop assuming that Spring embeds libraries by choice...)
Last edited by Kloot on 13 Dec 2013, 16:38, edited 1 time in total.
User avatar
SinbadEV
Posts: 6475
Joined: 02 May 2005, 03:56

Re: Building the official Spring 95.0 version for Debian

Post by SinbadEV »

apoleon wrote:3. Spring embeds a couple of well-known libraries such as lua and luasocket or assimp. Using embedded libraries is frowned upon in Debian and normally they are replaced by system libraries to make it easier to maintain the software in question and to fix security issues only once.

I did this for the assimp library but it seems your lua and luasocket versions are more difficult to replace. Is there a chance to get your changes into the upstream version of lua and luasocket?
I have a vague recollection that in the case of the Lua stuff it is embedded for the sake of cross-platform compatibility of synced code.
User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: Building the official Spring 95.0 version for Debian

Post by Anarchid »

This might be a good time to discuss whether it should be possible to install multiple versions of spring on a single machine. I.e:

- `spring` is a metapackage that depends on latest spring
- actual engine is packaged into e.g `spring-91', `spring-95`.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Building the official Spring 95.0 version for Debian

Post by jK »

1.) builddir must not be the srcdir
2.) there are CUSTOM_CFLAGS & CUSTOM_COMPILER_DEFINITIONS cmake vars, but as kloot already noted you must check each single flag you put in there. Else you desync and so breaks online gaming
3.) as kloot already linked, you must not replace anything in spring, Spring already uses systemlibs wherever possible, all included libs are sync-relevant and have modifications in their float math, to ensure all computations are bit-equal between computers
apoleon
Posts: 8
Joined: 12 Dec 2013, 21:19

Re: Building the official Spring 95.0 version for Debian

Post by apoleon »

Kloot wrote:2) Passing those flags would be a good idea but none are proven to sync (neither in isolation nor in combination) and we must be as paranoid about that (even moreso in fact, since hopefully nobody runs Spring as root) as about system security.
I see. Then I won't enable these specific hardening flags for spring. How do you communicate security issues or important problems for Spring? Is there a mailing list or some kind of newsletter I can subscribe to?
Kloot wrote: 3) Before you accidentally create packages that are useless for online play, please read http://springrts.com/phpbb/viewtopic.php?f=12&t=24813 because it should answer most of your questions.

(I do wish package maintainers would stop assuming that Spring embeds libraries by choice...)
jK wrote:3.) as kloot already linked, you must not replace anything in spring, Spring already uses systemlibs wherever possible, all included libs are sync-relevant and have modifications in their float math, to ensure all computations are bit-equal between computers
Thanks for the link. Indeed it seems the Fedora maintainer already voiced my concerns in the past but I'd like to ask a few more questions. First of all I want to counter the prejudice that package maintainers arbitrarily mutilate source packages. I think nobody is really interested in broken software. But I need to understand your decisions, so that I can reliably maintain spring in the future and if necessary defend your decisions.

I define my role as a maintainer as someone who is the link between upstream developers and end-users, someone who integrates software and makes it painless to install and use. Since I'm a user myself I want software to be stable and reliable, secure and free. As a package maintainer I also have to take various other factors into account. When I take on a job I'm dedicated to see it through but without your help this is hardly achievable. So here is my understanding of the current embedded library situation. Please correct me if I'm wrong.

* lua and luasocket
Your changes are not upstreamable. Replacing these libraries will break online gaming.

* include folder
Spring already uses system libraries and uses the include folder only as a fallback. From my point of view these files are just convenience copies and shouldn't have any impact on syncing. But using system libraries currently doesn't work properly. See http://springrts.com/mantis/view.php?id=4208 for more information.

* minizip
Replacing minizip is currently blocked by Debian bug http://bugs.debian.org/574798
It is safe to replace minizip ones the Debian bug is resolved because you already support minizip as a shared library.

* 7z
You discussed replacing 7z with xz. What is the current status of these considerations?

* squish

Couldn't find anything about this library. Debian ships a similar texture-compression library called http://packages.qa.debian.org/s/s2tc.html
Have you ever considered using this one instead?

* assimp

Regarding this post
http://springrts.com/phpbb/viewtopic.ph ... 20#p502423

using assimp as a shared library should be fairly safe but not completely. I saw that Fedora already replaced the embedded assimp lib and I'm currently running some online gaming tests to check whether there are sync problems or not.
Your recommendation is to use the embedded lib. Could you elaborate why this is still strictly needed today? Are you in contact with the developers of assimp and are your changes upstreamable?

Some other notes:

Currently you ship two binary java blobs, vecmath.jar and jna.jar in your source tarball without corresponding sources. These files have been replaced by Debian's version a long time ago and this seems to work. There are also dll files in installer/nsis_plugins which are superfluous on a Linux system. They also come without source.

jK wrote:1.) builddir must not be the srcdir
Hmm, I'm fairly certain that build dir and source dir are different. Are there more information available how to build the test suite? What commands do you use for building it? The strange thing is that 3 tests pass but the majority fails.

Anarchid wrote:This might be a good time to discuss whether it should be possible to install multiple versions of spring on a single machine. I.e:

- `spring` is a metapackage that depends on latest spring
- actual engine is packaged into e.g `spring-91', `spring-95`.
Not sure if I understand you correctly. Packaging each spring release in a different source package isn't feasible. Think about code duplication and increased maintenance work.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Building the official Spring 95.0 version for Debian

Post by jK »

apoleon wrote:I see. Then I won't enable these specific hardening flags for spring. How do you communicate security issues or important problems for Spring? Is there a mailing list or some kind of newsletter I can subscribe to?
We are low on devs currently, we fix security bugs when they happen/are exploited :/
Mantis is main platform for discussions, we also got an etherpad.
There is a mailinglist but we don't use it ourself.
apoleon wrote:* minizip
Replacing minizip is currently blocked by Debian bug http://bugs.debian.org/574798
It is safe to replace minizip ones the Debian bug is resolved because you already support minizip as a shared library.
On Gentoo Spring already uses system minizip.
apoleon wrote:* 7z
You discussed replacing 7z with xz. What is the current status of these considerations?
One of the devs (abma) did the heroic work of updating 7z. After that none dev did anything in the direction of implementing libarchive support.
apoleon wrote:* squish
Couldn't find anything about this library. Debian ships a similar texture-compression library called http://packages.qa.debian.org/s/s2tc.html
Have you ever considered using this one instead?
S2TC is a compression lib. We use libSquish to decompress our terrain textures (which are stored in S3TC in the mapfiles) to recompress them to ETC1/2 (a new texcompression format), when S3TC is not available (i.e. ati opensource drivers, still they don't suppport ETC yet).
It's a workaround for our white-terrain problem with many mesa-based drivers.
Still LibSquish might cause patent issues and can be disabled via USE_LIBSQUISH.
apoleon wrote:* assimp
Regarding this post
http://springrts.com/phpbb/viewtopic.ph ... 20#p502423
using assimp as a shared library should be fairly safe but not completely. I saw that Fedora already replaced the embedded assimp lib and I'm currently running some online gaming tests to check whether there are sync problems or not.
Your recommendation is to use the embedded lib. Could you elaborate why this is still strictly needed today? Are you in contact with the developers of assimp and are your changes upstreamable?
No assimp can never be linked shared, the one said didn't knew how complicated it is to supply sync-ness. After his pull-request another dev needed to do a lot modifications to assimp before it was sync-safe. Updating assimp these days is one of the worst maintain jobs in spring >_<, it can only simplified when assimp would use a header file we all used math funcs are defined (like lua does), so we can override them with streflop ones.
apoleon wrote:Currently you ship two binary java blobs, vecmath.jar and jna.jar in your source tarball without corresponding sources. These files have been replaced by Debian's version a long time ago and this seems to work. There are also dll files in installer/nsis_plugins which are superfluous on a Linux system. They also come without source.
That's AI no idea from that, the one who implemented it is unactive atm.
apoleon wrote:
jK wrote:1.) builddir must not be the srcdir
Hmm, I'm fairly certain that build dir and source dir are different. Are there more information available how to build the test suite? What commands do you use for building it? The strange thing is that 3 tests pass but the majority fails.
`make tests` or `ninja tests`
All test pass fine atm, see http://buildbot.springrts.com/waterfall
e.g. http://buildbot.springrts.com/builders/ ... logs/stdio
Last edited by jK on 14 Dec 2013, 17:27, edited 1 time in total.
User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: Building the official Spring 95.0 version for Debian

Post by Anarchid »

Not sure if I understand you correctly. Packaging each spring release in a different source package isn't feasible. Think about code duplication and increased maintenance work.
I meant things like python2.7 and python3 being capable of co-existance on the same system, same with lua libraries, etc.

However, most of those projects are larger, and probably have more people willing to do the packaging work; doing this for spring might indeed be too much.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Building the official Spring 95.0 version for Debian

Post by jK »

Anarchid wrote:
Not sure if I understand you correctly. Packaging each spring release in a different source package isn't feasible. Think about code duplication and increased maintenance work.
I meant things like python2.7 and python3 being capable of co-existance on the same system, same with lua libraries, etc.

However, most of those projects are larger, and probably have more people willing to do the packaging work; doing this for spring might indeed be too much.
That's no maintainer question atm, Spring still uses unversioned dirs & binaries.
apoleon
Posts: 8
Joined: 12 Dec 2013, 21:19

Re: Building the official Spring 95.0 version for Debian

Post by apoleon »

jK wrote:No assimp can never be linked shared, the one said didn't knew how complicated it is to supply sync-ness. After his pull-request another dev needed to do a lot modifications to assimp before it was sync-safe. Updating assimp these days is one of the worst maintain jobs in spring >_<, it can only simplified when assimp would use a header file we all used math funcs are defined (like lua does), so we can override them with streflop ones.
Did you report this issue to the upstream maintainer of assimp? Could you provide a link to your request? Maybe this just needs some kind of friendly reminder.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Building the official Spring 95.0 version for Debian

Post by jK »

apoleon wrote:Did you report this issue to the upstream maintainer of assimp? Could you provide a link to your request? Maybe this just needs some kind of friendly reminder.
Another dev had IRC contact with the assimp crew. I just know that he mentioned this solution duno if it ever was officially reported into their system.
apoleon
Posts: 8
Joined: 12 Dec 2013, 21:19

Re: Building the official Spring 95.0 version for Debian

Post by apoleon »

jK wrote:Another dev had IRC contact with the assimp crew. I just know that he mentioned this solution duno if it ever was officially reported into their system.
Hi,

I gladly open a ticket with assimp upstream, if you provide your suggested changes, a patch and some additional information.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Building the official Spring 95.0 version for Debian

Post by abma »

apoleon wrote:I gladly open a ticket with assimp upstream, if you provide your suggested changes, a patch and some additional information.
this is what is patched atm:
https://github.com/spring/assimp/commit ... 49037f86f3

to fix it upstream, this has to be made configureable.
also big fat note: the patch isn't up to date, but at least it gives an impression whats needed to be changed.


about libarchive: the problem is, that libarchive is made to work with streams. spring mostly uses archives with random access, so a wrapper has to be written first, so libarchive can be used. i guess easiest would be to uncompress whole archive to memory but this would heavily increase mem-usage...
apoleon
Posts: 8
Joined: 12 Dec 2013, 21:19

Re: Building the official Spring 95.0 version for Debian

Post by apoleon »

apoleon wrote:this is what is patched atm:
https://github.com/spring/assimp/commit ... 49037f86f3


Hello,

I have started a topic upstream at

https://sourceforge.net/p/assimp/discus ... /b7c98598/

and asked the developers of assimp for a clarification.
Post Reply

Return to “Linux”