Spring segfaults on load

Spring RTS Engine

Open Source Realtime Strategy Game Engine

Skip to content

Spring segfaults on load

Discuss everything related to running Spring on your chosen distribution of Linux.

Moderator: Moderators

Post Reply
HAARP
Posts: 182
Joined: 06 Apr 2006, 07:18

Spring segfaults on load

Post by HAARP »

It doesn't matter which start script, map or mod I use, my newly-compiled Spring always segfaults while loading.
Recompiled with debugging support-> Looks like it's related to OpenAL, but that's all I can figure out...

Code: Select all

GNU gdb 6.6
This GDB was configured as "i686-pc-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run
Starting program: /usr/games/bin/spring 
OpenAL: ALC_EXT_capture AL_EXT_capture AL_EXT_vorbis AL_EXT_MP3 AL_LOKI_quadriphonic AL_LOKI_play_position AL_LOKI_WAVE_format AL_LOKI_IMA_ADPCM_format AL_LOKI_buffer_data_callback ALC_LOKI_audio_channel 

Program received signal SIGSEGV, Segmentation fault.
0xb7ea2fea in ?? () from /usr/lib/libopenal.so.0
(gdb) bt
#0  0xb7e02fea in ?? () from /usr/lib/libopenal.so.0
#1  0x000a6180 in ?? ()
#2  0x0971c168 in ?? ()
#3  0xbf8fb938 in ?? ()
#4  0x0000001e in ?? ()
#5  0x00008010 in ?? ()
#6  0xfffffffb in ?? ()
#7  0x007f0c7f in ?? ()
#8  0xb7e0cff4 in ?? () from /usr/lib/libopenal.so.0
#9  0x00000000 in ?? ()
(gdb) continue
Continuing.
[New LWP 13320]
[LWP 13320 exited]
[New LWP 13317]
ptrace: No such process.
(gdb) q
It appears that the symbol table got trashed somewhere after linking, even tho I used the nostrip feature

infolog.txt

Code: Select all

Using read-write data directory: /home/haarp/.spring/
Using read-only  data directory: /usr/share/games/spring/
Using read-only  data directory: /usr/share/games/spring/
Using read-only  data directory: /usr/games/lib/spring/
Using script Commanders
Map: SmallDivide.smf
Mod: "Expand and Exterminate v 0.173" from Expand_and_Exterminate_v_0.173.sd7
Created server on port 8452
Created local client with number 0
Listening to local client on connection 0
TransportHover: 0 
Opening map file
Loading Map
Loading detail textures
Creating overhead texture
Creating ground shading
Loading tile file
Reading tiles
Reading tile map
Creating projectile texture
Number of damage types: 15
Loading units and weapons
Top
HAARP
Posts: 182
Joined: 06 Apr 2006, 07:18

Post by HAARP »

Disabling sound makes it not segfault -> problems with OpenAL. I investigate.
Top
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Spring segfaults on load

Post by Tobi »

HAARP wrote: It appears that the symbol table got trashed somewhere after linking, even tho I used the nostrip feature
Or it's just because it crashes in OpenAL's sound thread which lives in libopenal.so, which probably has been stripped from debugging symbols.
Top
HAARP
Posts: 182
Joined: 06 Apr 2006, 07:18

Re: Spring segfaults on load

Post by HAARP »

Tobi wrote:
HAARP wrote: It appears that the symbol table got trashed somewhere after linking, even tho I used the nostrip feature
Or it's just because it crashes in OpenAL's sound thread which lives in libopenal.so, which probably has been stripped from debugging symbols.
Of course! I'm dumb. Recompiled OpenAL with nostrip:

Code: Select all

$ gdb spring
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run
Starting program: /usr/games/bin/spring 
CONFIG  [al_config.c:675] define speaker-num
CONFIG  [al_config.c:675] define display-banner
CONFIG  [al_config.c:675] define source-gain
CONFIG  [alc/alc_device.c:43] Couldn't parse config file.
CONFIG  [al_config.c:550] could not resolve devices
CONFIG  [al_config.c:550] could not resolve direction
CONFIG  [al_config.c:550] could not resolve sampling-rate
CONFIG  [al_rcvar.c:367] rc_foreach fail type = ALRC_INVALID
CONFIG  [al_config.c:884] defining speaker-num
CONFIG  [al_config.c:550] could not resolve direction
CONFIG  [al_config.c:550] could not resolve devices
CONFIG  [al_config.c:550] could not resolve sampling-rate
CONFIG  [al_config.c:550] could not resolve devices
CONFIG  [al_config.c:550] could not resolve lin-dsp-path
CONFIG  [al_config.c:550] could not resolve native-use-select
grab_native: (path /dev/sound/dsp fd 20)
CONFIG  [arch/i386/x86_cpu_caps_prk.c:136] mmx found 1  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:138] sse found 1  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:140] sse2 found 0  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:142] sse3 found 0  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:144] amd_3dnow found 1  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:146] amd_3dnowext found 1  use 1
CONFIG  [arch/i386/x86_cpu_caps_prk.c:148] amd_sse_mmx found 1  use 1
EXT     [al_ext.c:422] registered alBufferAppendData_LOKI at 0xb7e4cb01
EXT     [al_ext.c:422] registered alBufferWriteData_LOKI at 0xb7e4db9b
EXT     [al_ext.c:422] registered alBufferAppendWriteData_LOKI at 0xb7e4d2d1
EXT     [al_ext.c:422] registered alReverbScale_LOKI at 0xb7e4e12b
EXT     [al_ext.c:422] registered alReverbDelay_LOKI at 0xb7e4df9a
EXT     [al_ext.c:422] registered alBombOnError_LOKI at 0xb7e4c770
EXT     [al_ext.c:422] registered alBufferi_LOKI at 0xb7e4c89d
EXT     [al_ext.c:422] registered alBufferDataWithCallback_LOKI at 0xb7e4df62
EXT     [al_ext.c:422] registered alGenStreamingBuffers_LOKI at 0xb7e4c787
EXT     [al_ext.c:422] registered alcGetAudioChannel_LOKI at 0xb7e4e2ef
EXT     [al_ext.c:422] registered alcSetAudioChannel_LOKI at 0xb7e4e267
EXT     [al_ext.c:422] registered alutLoadMP3_LOKI at 0xb7e4bbfd
EXT     [al_ext.c:422] registered alutLoadVorbis_LOKI at 0xb7e4b0ce
EXT     [al_ext.c:422] registered alCaptureInit_EXT at 0xb7e4c674
EXT     [al_ext.c:422] registered alCaptureStart_EXT at 0xb7e4c3dc
EXT     [al_ext.c:422] registered alCaptureStop_EXT at 0xb7e4c3e2
EXT     [al_ext.c:422] registered alCaptureGetData_EXT at 0xb7e4c3e8
EXT     [al_ext.c:422] registered alCaptureDestroy_EXT at 0xb7e4c5b7
CONTEXT [alc/alc_speaker.c:150] _alcSpeakerInit: ( sdis 1.000000 )
MATH    [alc/alc_speaker.c:96] SpAdj: l/r [-1.000000|0.000000|0.000000] [1.000000|0.000000|0.000000]
CONTEXT [alc/alc_context.c:698] new bufsiz = 4096
set_fd in: bufsiz 4096 fmt 0x10 speed 48000 channels 2
set_fd out: bufsiz 1024 fmt 0x10 speed 48000 channels 2
CONVERT [alc/alc_device.c:217] after set_audiodevice, f|s|b 0x1103|48000|1024
CONVERT [al_mixer.c:695] _alSetMixer f|c|s [0x1101|2|48000] -> [0x1103|2|48000]
OpenAL: ALC_EXT_capture AL_EXT_capture AL_EXT_vorbis AL_EXT_MP3 AL_LOKI_quadriphonic AL_LOKI_play_position AL_LOKI_WAVE_format AL_LOKI_IMA_ADPCM_format AL_LOKI_buffer_data_callback ALC_LOKI_audio_channel 
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [4609|1|11025]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|11025|4]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|8|2]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1100|1|11025] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [4609|73744]
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [21500|1|11025]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|11025|4]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|8|2]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1100|1|11025] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [21500|344000]
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [21024|1|11025]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|11025|4]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|8|2]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1100|1|11025] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [21024|336384]
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [74191|1|22050]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|22050|2]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|8|2]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1100|1|22050] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [74191|593528]
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [9665|1|11025]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|11025|4]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|8|2]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1100|1|11025] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [9665|154640]
CONVERT [al_buffer.c:1046] _alConvert [f_size|f_channels|f_freq] [170078|1|22050]
CONVERT [al_buffer.c:1051] _alConvert [t_channels|f_channels|t/f] [1|1|1]
CONVERT [al_buffer.c:1060] _alConvert [t_freq|f_freq|t/f] [48000|22050|2]
CONVERT [al_buffer.c:1066] _alConvert [t_bits|f_bits|t/f] [16|16|1]
CONVERT [al_buffer.c:1073] _alConvert f|c|s [0x1101|1|22050] -> [0x1101|1|48000]
CONVERT [al_buffer.c:1100] _alConvert [len|newlen] [170078|680312]

Program received signal SIGSEGV, Segmentation fault.
0xb7e49a87 in acFreqSLOW (cvt=0xbfcbddb4, format=<value optimized out>) at audioconvert/ac_freq.c:253
253     audioconvert/ac_freq.c: No such file or directory.
        in audioconvert/ac_freq.c
Current language:  auto; currently c
(gdb) bt
#0  0xb7e49a87 in acFreqSLOW (cvt=0xbfcbddb4, format=<value optimized out>) at audioconvert/ac_freq.c:253
#1  0xb7e49888 in acFreqMUL2 (cvt=0xbfcbddb4, format=32784) at audioconvert/ac_freq.c:151
#2  0xb7e49f69 in acConvertAudio (cvt=0xb37ac008) at audioconvert/ac_helper.c:244
#3  0xb7e24468 in _alConvert (data=0x999d9d4, f_format=4353, f_size=170078, f_freq=22050, t_format=4353, t_freq=48000, retsize=0xbfcbde78, 
    should_use_passed_data=0) at al_buffer.c:1126
#4  0xb7e251c9 in alBufferData (bid=32774, format=4353, data=0x999d9d4, size=170078, freq=22050) at al_buffer.c:674
#5  0x08385591 in COpenALSound::ReadWAV ()
#6  0x08385719 in COpenALSound::LoadALBuffer ()
#7  0x08385846 in COpenALSound::GetWaveId ()
#8  0x085a310f in CWeaponDefHandler::LoadSound ()
#9  0x085ae56a in CWeaponDefHandler::ParseTAWeapon ()
#10 0x085aee20 in CWeaponDefHandler ()
#11 0x085d82f6 in CUnitDefHandler ()
#12 0x0841b69a in CGame ()
#13 0x0842b0c3 in CPreGame::Update ()
#14 0x08348745 in SpringApp::Update ()
#15 0x0834c707 in SpringApp::Run ()
#16 0x0834c873 in Run ()
#17 0x0834ca6a in main ()
(gdb) continue
Continuing.
[New LWP 10736]
[LWP 10736 exited]
[New LWP 10733]
ptrace: No such process.
(gdb) q
Hope someone can use this
I think that audioconvert/ac_freq.c was somehow missing while compiling openal or something. Any ideas?
Top
HAARP
Posts: 182
Joined: 06 Apr 2006, 07:18

Post by HAARP »

Ok, I tried to investigate this problem for a few days now with no luck. I blame it on lack of C skills ;)
Anyone here who could have an idea? Tobi?
Top
thomasa88
Posts: 5
Joined: 25 Dec 2007, 23:47

Re: Spring segfaults on load

Post by thomasa88 »

Hi, spring stopped segfaulting when I removed this patch to openal source.

mrt@barbara ~ $ cat /usr/portage/media-libs/openal/files/0.0.8/005_all_alsa_default_freq.patch

Code: Select all

diff -urN openal-0.0.8.orig/src/alc/alc_context.h openal-0.0.8/src/alc/alc_context.h
--- openal-0.0.8.orig/src/alc/alc_context.h     2005-11-18 05:31:47.000000000 -0800
+++ openal-0.0.8/src/alc/alc_context.h  2007-09-20 13:16:22.000000000 -0700
@@ -18,7 +18,7 @@
 /*
  * _ALC_DEF_FREQ is the default internal mixing frequency.
  */
-#define _ALC_DEF_FREQ        44100
+#define _ALC_DEF_FREQ        48000

 /*
  * _ALC_DEF_BUFSIZ is the default length of chunks mixed and written to the
Top
Kloot
Spring Developer
Posts: 1867
Joined: 08 Oct 2006, 16:58

Re: Spring segfaults on load

Post by Kloot »

#5 0x08385591 in COpenALSound::ReadWAV ()
#6 0x08385719 in COpenALSound::LoadALBuffer ()
#7 0x08385846 in COpenALSound::GetWaveId ()
#8 0x085a310f in CWeaponDefHandler::LoadSound ()
#9 0x085ae56a in CWeaponDefHandler::ParseTAWeapon ()
#10 0x085aee20 in CWeaponDefHandler ()
#11 0x085d82f6 in CUnitDefHandler ()
#12 0x0841b69a in CGame ()
#13 0x0842b0c3 in CPreGame::Update ()
#14 0x08348745 in SpringApp::Update ()
#15 0x0834c707 in SpringApp::Run ()
#16 0x0834c873 in Run ()
#17 0x0834ca6a in main ()
I got this trace when loading XTA 9.2 the other day, it
seems only triggered by very specific .wav's (files that
contain a data-length field in their header whose value
exceeds their actual size in bytes, ionbeam.wav in XTA's
case). Spring guards against such files now, but there's
probably a relation to your version of OpenAL as well (EE
0.173 has never crashed for me despite also bundling a
few of these oddball wavs).
Top
thomasa88
Posts: 5
Joined: 25 Dec 2007, 23:47

Re: Spring segfaults on load

Post by thomasa88 »

Kloot wrote:
#5 0x08385591 in COpenALSound::ReadWAV ()
#6 0x08385719 in COpenALSound::LoadALBuffer ()
#7 0x08385846 in COpenALSound::GetWaveId ()
#8 0x085a310f in CWeaponDefHandler::LoadSound ()
#9 0x085ae56a in CWeaponDefHandler::ParseTAWeapon ()
#10 0x085aee20 in CWeaponDefHandler ()
#11 0x085d82f6 in CUnitDefHandler ()
#12 0x0841b69a in CGame ()
#13 0x0842b0c3 in CPreGame::Update ()
#14 0x08348745 in SpringApp::Update ()
#15 0x0834c707 in SpringApp::Run ()
#16 0x0834c873 in Run ()
#17 0x0834ca6a in main ()
I got this trace when loading XTA 9.2 the other day, it
seems only triggered by very specific .wav's (files that
contain a data-length field in their header whose value
exceeds their actual size in bytes, ionbeam.wav in XTA's
case). Spring guards against such files now, but there's
probably a relation to your version of OpenAL as well (EE
0.173 has never crashed for me despite also bundling a
few of these oddball wavs).
Hi, which version of Spring? I'm running 0.75b2 with openal 0.0.8, maybe I should try latest svn out..
Top
Kloot
Spring Developer
Posts: 1867
Joined: 08 Oct 2006, 16:58

Re: Spring segfaults on load

Post by Kloot »

A custom SVN build, but it also crashed in 75b2. The
file filter is in SVN revision 5105 and above, be sure to
check infolog.txt for OpenAL messages after testing a
mod (any wavs with this problem will have their header
data dumped there).

PS. for comparison, a test run of XTA 9.2 produces the
following output:

Code: Select all

OpenAL: sounds/ionbeam.wav has data length 1252024692 greater than actual data length 215714
OpenAL: size 215758
OpenAL: sizeof(WAVHeader) 44
OpenAL: format_tag 1
OpenAL: channels 2
OpenAL: BlockAlign 4
OpenAL: BitsPerSample 16
OpenAL: totalLength 215750
OpenAL: length 18
OpenAL: SamplesPerSec 16000
OpenAL: AvgBytesPerSec 64000

OpenAL: sounds/rockhit.wav has data length 291939 greater than actual data length 19514
OpenAL: size 19558
OpenAL: sizeof(WAVHeader) 44
OpenAL: format_tag 1
OpenAL: channels 1
OpenAL: BlockAlign 1
OpenAL: BitsPerSample 8
OpenAL: totalLength 19550
OpenAL: length 18
OpenAL: SamplesPerSec 11025
OpenAL: AvgBytesPerSec 11025
Top
thomasa88
Posts: 5
Joined: 25 Dec 2007, 23:47

Re: Spring segfaults on load

Post by thomasa88 »

Tried svn rev 5111 and with untouched openal it segfaults with absolute annihilation, and no errors in the infolog about sounds (did not turn on debug though, so I don't know if it is the same problem). :(

I will compile everything as debug tomorrow and see what it gives.
Top
HAARP
Posts: 182
Joined: 06 Apr 2006, 07:18

Re: Spring segfaults on load

Post by HAARP »

thomasa88 wrote:Hi, spring stopped segfaulting when I removed this patch to openal source.

mrt@barbara ~ $ cat /usr/portage/media-libs/openal/files/0.0.8/005_all_alsa_default_freq.patch

Code: Select all

diff -urN openal-0.0.8.orig/src/alc/alc_context.h openal-0.0.8/src/alc/alc_context.h
--- openal-0.0.8.orig/src/alc/alc_context.h     2005-11-18 05:31:47.000000000 -0800
+++ openal-0.0.8/src/alc/alc_context.h  2007-09-20 13:16:22.000000000 -0700
@@ -18,7 +18,7 @@
 /*
  * _ALC_DEF_FREQ is the default internal mixing frequency.
  */
-#define _ALC_DEF_FREQ        44100
+#define _ALC_DEF_FREQ        48000

 /*
  * _ALC_DEF_BUFSIZ is the default length of chunks mixed and written to the
As I already said on the Gentoo Forums, removing this patch from the OpenAL source fixed the issue completely for me. Thanks again!
Top
thomasa88
Posts: 5
Joined: 25 Dec 2007, 23:47

Re: Spring segfaults on load

Post by thomasa88 »

HAARP wrote: As I already said on the Gentoo Forums, removing this patch from the OpenAL source fixed the issue completely for me. Thanks again!
I didn't see that it was the same username lol :P

Hmm, maybe I'm having problems with openal giving any sound now, any of you know if there's a program to test it?

(compiling debug versions of both right now, funny that the ebuild svn always takes the absolute latest version =p)

edit.

Found this bug http://bugs.gentoo.org/show_bug.cgi?id=193147 but instead of changing half of my sound system I ended up compiling openal without the last alsa patch (as posted before) and using oss, as openal alsa support seems really messy with an intel card.

Found out I still need to configure alsa to 44100 Hz so I have created a special user for openal, using the following configs:

~/.openalrc

Code: Select all

#try to make games using 44100 Hz sounds work

#(define devices '(alsa))
#(define devices '(alsa esd arts sdl null))
#(define devices '(null))
(define devices '(oss))

# Sound Output
(define alsa-out-device "default")
#(define alsa-out-device "hw:0,0")

# Sound Input i.e microphone
(define alsa-in-device "default")
#(define alsa-in-device "hw:0,0")

# 2 Speaker setup
(define speaker-num 2)

(define sampling-rate 44100)
~/.asoundrc (from gentoo bug report)

Code: Select all

#####
# For use with Intel HDA.
# - Provides full-duplex for the default device via dmix / dsnoop and asym.
# - Forces 44.1kHz sampling rate (instead of 48kHz) for OpenAL compatibility.
#   Note: For this to work "(define sampling-rate 44100)" must be specified in ~/.openalrc also.
#         See http://bugs.gentoo.org/show_bug.cgi?id=193147 for details.
#
# REMEMBER TO STOP ALL APPS USING ALSA AND RUN "/etc/init.d/alsasound restart" AS ROOT!
# Otherwise changes seem not to be picked up correctly sometimes (very confusing ;-)
#####

pcm.!default {
    type plug
    slave.pcm "duplex"
}

ctl.!default {
    type hw
    card 0
}

pcm.duplex {
    type asym
    playback.pcm "dmixer"
    capture.pcm "dsnooper"
}

pcm.dmixer {
    type dmix
    ipc_key 1024
    slave {
        pcm "hw:0,0"
        channels 2
        period_time 0
        period_size 2048
        buffer_size 4096
        rate 44100
        }

    bindings {
        0 0
        1 1
        }
}

pcm.dsnooper {
    type dsnoop
    ipc_key 1025
    slave {
        pcm "hw:0,0"
        channels 2
        period_time 0
        period_size 2048
        buffer_size 4096
        rate 44100
        }

    bindings {
        0 0
        1 1
        }
}
edit.

aoss seems to work if the apps are run as the same user (probably has to do with the alsa config locking somehow). I have made a spring script that just puts aoss in front and then I've put it in a dir in the beginning of the PATH and set in it springlobby =)
Top
OldMcDonald
Posts: 5
Joined: 14 Sep 2008, 20:08

Re: Spring segfaults on load

Post by OldMcDonald »

All this seems to work with openal-1.5, and without touching any other configuration file...


Best,
OMD
Top
Post Reply

Return to “Linux”

Powered by phpBB® Forum Software © phpBB Limited
 

 

Site created by the community