View topic - Static libs - can they be removed?


All times are UTC + 1 hour


Post new topic Reply to topic  [ 41 posts ]  Go to page 1, 2, 3  Next
Author Message
PostPosted: 16 Dec 2010, 18:02 

Joined: 29 Apr 2010, 00:20
Hello all,

I've recently became the Fedora (Linux distribution) spring and springlobby package maintainer.
However, Fedora has a strict policy against including static version of system libraries due to the obvious possible security implication of using unsupported / obsolete version of these libraries.
In-order to get exempt from this policy I require a detailed explanation that will submitted to the Fedora governing board.

As far as I could see, springs source tarball includes a copy of lua, md5 and 7zip (have I missed anything? Keep in mind that I've yet to check if springs actually links against these libraries).

Before I start diving into the code, can any of these libraries be replaced with the system version?
If not, are you using particular patches that require the use of static libraries? Did you attempt to get these patches included in the upstream version?

Thanks for a great game!
- Gilboa
Top
 Offline Profile  
 
PostPosted: 16 Dec 2010, 18:50 
Lobby Developer
User avatar

Joined: 25 Sep 2006, 12:56
Hi
gilboa wrote:
As far as I could see, springs source tarball includes a copy of lua, md5 and 7zip (have I missed anything? Keep in mind that I've yet to check if springs actually links against these libraries).

I'd have to check myself, at the moment I'm writing this post on top of my mind
gilboa wrote:
Before I start diving into the code, can any of these libraries be replaced with the system version?

no, as far as I remember, all those libraries link to streflop ( see below )
gilboa wrote:
If not, are you using particular patches that require the use of static libraries?

In order to ensure that floating point calculations provide consistent results on different operative systems and different architectures, in certain critical parts of the code and relative libraries used there, Spring makes use of the streflop library, which replaces some mathematical functions in C/C++ code with it's own version.
if I recall correctly, all the libraries you mentioned contain such special linking.
Removing streflop link would result in broken networking and broken replay watching with other clients
gilboa wrote:
Did you attempt to get these patches included in the upstream version?

I am not aware of any attempts for such, and I think given the very specific use case, it would be unnecessary.
Top
 Offline Profile  
 
PostPosted: 17 Dec 2010, 09:38 
Redacted
User avatar

Joined: 08 Jan 2007, 06:13
Location: Don't be silly. If there's no machine heaven, where do all the toasters go?
I can't think of why md5 and 7zip would require that?

Lua has that and other customizations, and updates are pretty rare for it. Also we don't have a fully secured lua environment in the first place.
Top
 Offline Profile  
 
PostPosted: 17 Dec 2010, 09:54 
User avatar

Joined: 28 Jul 2008, 05:51
Location: Australia
7zip was originally a windows tool that came across to linux later. It's likely it's static simply because it wasn't originally part of most distros. I can't see why it wouldn't return identical data cross platorm since an archive format that corrupts data is a broken (or lossy) format. Same with md5 (it should always return the same value or it wouldn't be doing its job).

I know lua has some custom stuff so it will need to remain static.
Top
 Offline Profile  
 
PostPosted: 17 Dec 2010, 11:36 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
will discuss this in the next meeting.
Top
 Offline Profile  
 
PostPosted: 17 Dec 2010, 17:44 

Joined: 29 Apr 2010, 00:20
First,

Thanks for the prompt replay.
I will wait for the results of the next dev IRC meeting to see which libraries can be replaced by the system default and for which libraries I must get Fedora governing board authorization.

- Gilboa
Top
 Offline Profile  
 
PostPosted: 20 Dec 2010, 20:33 
Spring Developer

Joined: 01 Jun 2005, 10:36
Location: The Netherlands
MD5 and 7-zip are probably only included for pragmatic reasons (i.e. it not being included in most distros / it simply being there since Springs Windows only times / etc.)

Lua has patches applied, must link to streflop, and is configured different from stock Lua (most importantly we need lua_Number to be float and not double). Lua is particularly important because parts of the game code may be written in it, which must yield exactly identical results (also floating point operations!) on all platforms.
Top
 Offline Profile  
 
PostPosted: 21 Dec 2010, 00:43 
User avatar

Joined: 02 May 2005, 02:56
Location: Canada
Tobi wrote:
MD5 and 7-zip are probably only included for pragmatic reasons (i.e. it not being included in most distros / it simply being there since Springs Windows only times / etc.)

Lua has patches applied, must link to streflop, and is configured different from stock Lua (most importantly we need lua_Number to be float and not double). Lua is particularly important because parts of the game code may be written in it, which must yield exactly identical results (also floating point operations!) on all platforms.


I'm not a dev and not a linux user but... Wouldn't the fact that the source-code for our modifications is open source mean that we could just include the modified source in our repository and have it built compile time instead of the static version?... or whatever (replacing my stupid words with the right ones that mean what I'm trying to say of course)
Top
 Offline Profile  
 
PostPosted: 21 Dec 2010, 09:36 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
what you think we should do is what we are already doing. ;-)
static is a type of library, it does not mean that we do not build the library at spring compile time.

maybe this will help:
http://stackoverflow.com/questions/1400 ... -libraries
Top
 Offline Profile  
 
PostPosted: 21 Dec 2010, 13:44 
Spring Developer

Joined: 01 Jun 2005, 10:36
Location: The Netherlands
The confusion may be because the actual issue is not so much whether these libraries are statically linked or dynamically linked, but that the libraries are vendorized, which is jargon for "we copied the sources of the libraries to our repository and included them into our own build, instead of using the system's copy of the libraries."

In general Linux distributions do not like this as it causes an explosion of work when e.g. a security fix needs to be applied to such a library. (Someone once told me there was an incident like that a long time ago with zlib when many programs in Linux had zlib vendorized.)

In Windows-land you could compare it with every application you install that embeds a browser (which are many), shipping it's own copy of Internet Explorer, instead of using the system-wide installed one. You can imagine it would then be nearly impossible for Microsoft to keep all those copies up to date with the latest security patches.
Top
 Offline Profile  
 
PostPosted: 27 Dec 2010, 11:35 

Joined: 29 Apr 2010, 00:20
Tobi wrote:
In general Linux distributions do not like this as it causes an explosion of work when e.g. a security fix needs to be applied to such a library. (Someone once told me there was an incident like that a long time ago with zlib when many programs in Linux had zlib vendorized.)


/+1.
Hence Fedora more-or-less prohibits using statically linked libraries without written approval.

I looked at the dev-irc-log, and it seems that there was an agreement to start removing statically linked libraries that weren't strictly required.

How can I get an official comment from one of the devs that can be attached to an official request for a temporary static-libs approval?

Beyond that, I see that according the the irc chat log, I can already rip 7za and md5 with no adverse effect. Am I correct?

- Gilboa
Top
 Offline Profile  
 
PostPosted: 27 Dec 2010, 13:51 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
we agreed that 7z, md5 and minizip could be used as shared libs instead, but i found out that they are not available as (shared) libs in packages (Gentoo 64bit), therefore... we will not change anything.
Top
 Offline Profile  
 
PostPosted: 27 Dec 2010, 21:21 
AI Coder
User avatar

Joined: 14 Sep 2004, 10:32
Location: Cookieland
Why is that an issue? Plenty of people compile spring and other projects from the git repo using mingwlibs and vclibs, why not gentoo?
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 05:47 

Joined: 29 Apr 2010, 00:20
hoijui wrote:
we agreed that 7z, md5 and minizip could be used as shared libs instead, but i found out that they are not available as (shared) libs in packages (Gentoo 64bit), therefore... we will not change anything.


This seems backward to me. It's the distribution's job to supply the prerequisites, not the end-user-app. Just consider how KDE or GNOME would look like, if they had to carry a copy of all the shared libraries that they used inside their tarball. Ugh.
Never the less that's your prerogative.

May I suggest that you add compile time detection to you build-system; using the distribution supplied libraries whenever possible? It'll save me the need to patch your tarball and build-system in-order to remain compliant with the Fedora guidelines.

- Gilboa
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 09:12 
Redacted
User avatar

Joined: 08 Jan 2007, 06:13
Location: Don't be silly. If there's no machine heaven, where do all the toasters go?
To an extent it's the distribution's job, but it'll take a bit to convince me that something as simple as md5 is better off as a library. Zip libraries yes.
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 09:52 

Joined: 29 Apr 2010, 00:20
lurker wrote:
To an extent it's the distribution's job, but it'll take a bit to convince me that something as simple as md5 is better off as a library. Zip libraries yes.


No problem.
Simply find a GPL implementation of a MD5 checksum function and place it inside springs (w/ proper acknowledgment) - this should even reduce your compilation footprint.
However, once you use static libraries you make life harder to distribution maintainers.

- Gilboa
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 10:22 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
aehm... if you tell me that there are distributions that have them as shared libs, i will do it. the spring for debian maintainers did what they wanted them-selfs, plus they have a much larger user-base (as it also includes ubuntu). so do you have an 7z lib on fedora? if not, go get that first, maybe.


Last edited by hoijui on 28 Dec 2010, 15:28, edited 1 time in total.
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 14:32 
Redacted
User avatar

Joined: 08 Jan 2007, 06:13
Location: Don't be silly. If there's no machine heaven, where do all the toasters go?
Md5's implemented under /lib/ right now, but so is lua lobby, a piece of spring. All the other directories under /lib/ have a handful of files with notable functionality. Md5 is one .c with three functions, implementing an algorithm simple enough to be given in full as code on its wikipedia page. It doesn't even compile into its own object file; it's stuck with an even smaller file implementing base64.
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 15:29 

Joined: 29 Apr 2010, 00:20
hoijui wrote:
aehm... if you tell me that there are distributions that have them as shared libs, i will do it. the spring for debian maintainers did what they wanted them-selfs, plus they have a much larger user-base (as it also includes ubuntu). so do you have a 7z lib on fedora? if not, go get that first, maybe.


OK, let me reiterate, I wasn't trying to tell you how to do your job; sorry it was implied.

I require two things:
- Make shared libraries compile time optional so maintainers / distribution who opt to use non-static version of these libraries can do so. Let -me- do the hard work of getting the missing bits include into Fedora.
- In-order to get permission from Fedora to use static libraries (such as, say, Lua) I require a short explanation from the devs.

- Gilboa
Top
 Offline Profile  
 
PostPosted: 28 Dec 2010, 16:45 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
so it is enough if i do in our configure/build script:
Code:
if (systemHasSharedLib(7z))
   linkToShared();
else
   compileStatic();
   linkToStatic();

?

For the other libs (not 7z. md5. minizip), the reason is that we require changes to these libs to be able to keep synced (and in the case of Lua, for optimization/threading related reasons). these changes would not make sense for other users of these libraries.
Top
 Offline Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 41 posts ]  Go to page 1, 2, 3  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: Google [Bot] and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group
Site layout created by Roflcopter et al.