Static libs - can they be removed?

Static libs - can they be removed?

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Static libs - can they be removed?

Post by gilboa »

Hello all,

I've recently became the Fedora (Linux distribution) spring and springlobby package maintainer.
However, Fedora has a strict policy against including static version of system libraries due to the obvious possible security implication of using unsupported / obsolete version of these libraries.
In-order to get exempt from this policy I require a detailed explanation that will submitted to the Fedora governing board.

As far as I could see, springs source tarball includes a copy of lua, md5 and 7zip (have I missed anything? Keep in mind that I've yet to check if springs actually links against these libraries).

Before I start diving into the code, can any of these libraries be replaced with the system version?
If not, are you using particular patches that require the use of static libraries? Did you attempt to get these patches included in the upstream version?

Thanks for a great game!
- Gilboa
User avatar
BrainDamage
Lobby Developer
Posts: 1164
Joined: 25 Sep 2006, 13:56

Re: Static libs - can they be removed?

Post by BrainDamage »

Hi
gilboa wrote: As far as I could see, springs source tarball includes a copy of lua, md5 and 7zip (have I missed anything? Keep in mind that I've yet to check if springs actually links against these libraries).
I'd have to check myself, at the moment I'm writing this post on top of my mind
gilboa wrote:Before I start diving into the code, can any of these libraries be replaced with the system version?
no, as far as I remember, all those libraries link to streflop ( see below )
gilboa wrote:If not, are you using particular patches that require the use of static libraries?
In order to ensure that floating point calculations provide consistent results on different operative systems and different architectures, in certain critical parts of the code and relative libraries used there, Spring makes use of the streflop library, which replaces some mathematical functions in C/C++ code with it's own version.
if I recall correctly, all the libraries you mentioned contain such special linking.
Removing streflop link would result in broken networking and broken replay watching with other clients
gilboa wrote:Did you attempt to get these patches included in the upstream version?
I am not aware of any attempts for such, and I think given the very specific use case, it would be unnecessary.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Static libs - can they be removed?

Post by lurker »

I can't think of why md5 and 7zip would require that?

Lua has that and other customizations, and updates are pretty rare for it. Also we don't have a fully secured lua environment in the first place.
User avatar
SpliFF
Posts: 1224
Joined: 28 Jul 2008, 06:51

Re: Static libs - can they be removed?

Post by SpliFF »

7zip was originally a windows tool that came across to linux later. It's likely it's static simply because it wasn't originally part of most distros. I can't see why it wouldn't return identical data cross platorm since an archive format that corrupts data is a broken (or lossy) format. Same with md5 (it should always return the same value or it wouldn't be doing its job).

I know lua has some custom stuff so it will need to remain static.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Static libs - can they be removed?

Post by hoijui »

will discuss this in the next meeting.
gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Re: Static libs - can they be removed?

Post by gilboa »

First,

Thanks for the prompt replay.
I will wait for the results of the next dev IRC meeting to see which libraries can be replaced by the system default and for which libraries I must get Fedora governing board authorization.

- Gilboa
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Static libs - can they be removed?

Post by Tobi »

MD5 and 7-zip are probably only included for pragmatic reasons (i.e. it not being included in most distros / it simply being there since Springs Windows only times / etc.)

Lua has patches applied, must link to streflop, and is configured different from stock Lua (most importantly we need lua_Number to be float and not double). Lua is particularly important because parts of the game code may be written in it, which must yield exactly identical results (also floating point operations!) on all platforms.
User avatar
SinbadEV
Posts: 6475
Joined: 02 May 2005, 03:56

Re: Static libs - can they be removed?

Post by SinbadEV »

Tobi wrote:MD5 and 7-zip are probably only included for pragmatic reasons (i.e. it not being included in most distros / it simply being there since Springs Windows only times / etc.)

Lua has patches applied, must link to streflop, and is configured different from stock Lua (most importantly we need lua_Number to be float and not double). Lua is particularly important because parts of the game code may be written in it, which must yield exactly identical results (also floating point operations!) on all platforms.
I'm not a dev and not a linux user but... Wouldn't the fact that the source-code for our modifications is open source mean that we could just include the modified source in our repository and have it built compile time instead of the static version?... or whatever (replacing my stupid words with the right ones that mean what I'm trying to say of course)
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Static libs - can they be removed?

Post by hoijui »

what you think we should do is what we are already doing. ;-)
static is a type of library, it does not mean that we do not build the library at spring compile time.

maybe this will help:
http://stackoverflow.com/questions/1400 ... -libraries
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Static libs - can they be removed?

Post by Tobi »

The confusion may be because the actual issue is not so much whether these libraries are statically linked or dynamically linked, but that the libraries are vendorized, which is jargon for "we copied the sources of the libraries to our repository and included them into our own build, instead of using the system's copy of the libraries."

In general Linux distributions do not like this as it causes an explosion of work when e.g. a security fix needs to be applied to such a library. (Someone once told me there was an incident like that a long time ago with zlib when many programs in Linux had zlib vendorized.)

In Windows-land you could compare it with every application you install that embeds a browser (which are many), shipping it's own copy of Internet Explorer, instead of using the system-wide installed one. You can imagine it would then be nearly impossible for Microsoft to keep all those copies up to date with the latest security patches.
gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Re: Static libs - can they be removed?

Post by gilboa »

Tobi wrote:In general Linux distributions do not like this as it causes an explosion of work when e.g. a security fix needs to be applied to such a library. (Someone once told me there was an incident like that a long time ago with zlib when many programs in Linux had zlib vendorized.)
/+1.
Hence Fedora more-or-less prohibits using statically linked libraries without written approval.

I looked at the dev-irc-log, and it seems that there was an agreement to start removing statically linked libraries that weren't strictly required.

How can I get an official comment from one of the devs that can be attached to an official request for a temporary static-libs approval?

Beyond that, I see that according the the irc chat log, I can already rip 7za and md5 with no adverse effect. Am I correct?

- Gilboa
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Static libs - can they be removed?

Post by hoijui »

we agreed that 7z, md5 and minizip could be used as shared libs instead, but i found out that they are not available as (shared) libs in packages (Gentoo 64bit), therefore... we will not change anything.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: Static libs - can they be removed?

Post by AF »

Why is that an issue? Plenty of people compile spring and other projects from the git repo using mingwlibs and vclibs, why not gentoo?
gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Re: Static libs - can they be removed?

Post by gilboa »

hoijui wrote:we agreed that 7z, md5 and minizip could be used as shared libs instead, but i found out that they are not available as (shared) libs in packages (Gentoo 64bit), therefore... we will not change anything.
This seems backward to me. It's the distribution's job to supply the prerequisites, not the end-user-app. Just consider how KDE or GNOME would look like, if they had to carry a copy of all the shared libraries that they used inside their tarball. Ugh.
Never the less that's your prerogative.

May I suggest that you add compile time detection to you build-system; using the distribution supplied libraries whenever possible? It'll save me the need to patch your tarball and build-system in-order to remain compliant with the Fedora guidelines.

- Gilboa
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Static libs - can they be removed?

Post by lurker »

To an extent it's the distribution's job, but it'll take a bit to convince me that something as simple as md5 is better off as a library. Zip libraries yes.
gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Re: Static libs - can they be removed?

Post by gilboa »

lurker wrote:To an extent it's the distribution's job, but it'll take a bit to convince me that something as simple as md5 is better off as a library. Zip libraries yes.
No problem.
Simply find a GPL implementation of a MD5 checksum function and place it inside springs (w/ proper acknowledgment) - this should even reduce your compilation footprint.
However, once you use static libraries you make life harder to distribution maintainers.

- Gilboa
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Static libs - can they be removed?

Post by hoijui »

aehm... if you tell me that there are distributions that have them as shared libs, i will do it. the spring for debian maintainers did what they wanted them-selfs, plus they have a much larger user-base (as it also includes ubuntu). so do you have an 7z lib on fedora? if not, go get that first, maybe.
Last edited by hoijui on 28 Dec 2010, 15:28, edited 1 time in total.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Static libs - can they be removed?

Post by lurker »

Md5's implemented under /lib/ right now, but so is lua lobby, a piece of spring. All the other directories under /lib/ have a handful of files with notable functionality. Md5 is one .c with three functions, implementing an algorithm simple enough to be given in full as code on its wikipedia page. It doesn't even compile into its own object file; it's stuck with an even smaller file implementing base64.
gilboa
Posts: 41
Joined: 29 Apr 2010, 01:20

Re: Static libs - can they be removed?

Post by gilboa »

hoijui wrote:aehm... if you tell me that there are distributions that have them as shared libs, i will do it. the spring for debian maintainers did what they wanted them-selfs, plus they have a much larger user-base (as it also includes ubuntu). so do you have a 7z lib on fedora? if not, go get that first, maybe.
OK, let me reiterate, I wasn't trying to tell you how to do your job; sorry it was implied.

I require two things:
- Make shared libraries compile time optional so maintainers / distribution who opt to use non-static version of these libraries can do so. Let -me- do the hard work of getting the missing bits include into Fedora.
- In-order to get permission from Fedora to use static libraries (such as, say, Lua) I require a short explanation from the devs.

- Gilboa
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Static libs - can they be removed?

Post by hoijui »

so it is enough if i do in our configure/build script:

Code: Select all

if (systemHasSharedLib(7z))
   linkToShared();
else
   compileStatic();
   linkToStatic();
?

For the other libs (not 7z. md5. minizip), the reason is that we require changes to these libs to be able to keep synced (and in the case of Lua, for optimization/threading related reasons). these changes would not make sense for other users of these libraries.
Post Reply

Return to “Engine”