Spring 0.80.5.2 MT Build Hangs

[Delta_]

I was just playing in a game and all of a sudden it just stopped; the screen went black, and I wasn't able to do anything. I know it crashed right after spamming "[ 50973] GML error: Sim thread called gmlGenLists" in the console, so I'm guessing that has something to do with it. I could be wrong though.

This log is incomplete because I had to manually kill the game: infolog.txt

I know I can choose not to use the MT build, but I rather the game run faster, as opposed to it lagging really badly. And anyways, it still crashes on and off using the single threaded build, so I don't know what's better.

Most of my system stats are in the log, but I'll list them here anyways:
CPU: AMD Opteron 165 @ 2.8GHz
MB: Asus A8N5X
Ram: OCZ Platinum 2x1GB DDR-400
Video: GeForce 7600 GT 256MB
OS: Ubuntu 9.10 (Karmic Koala)

Hopefully this bug report doesn't just get discarded, and if you know of the issue already do whatever you want. Anyways, thanks for your help and time!

[Delta_]

I just tried playing another game; maybe 20-30 minutes into the game it crashed, this time it with a segmentation fault. From what I've seen the SMP version of Spring has had over a year of development--well at least dating back that far. I know you guys don't work 8 hours a day on Spring, but I was under the impression that the MT build was stable enough for use, guess not. So, from this moment on I'm going back to the regular single threaded build, hopefully it won't crash as much.

log: infolog.txt

Code: Select all

[  40992] Segmentation fault (SIGSEGV) in spring 0.80.5.2
Stacktrace:
/usr/local/bin/spring [0xd2fcdf]
/lib/libc.so.6 [0x7f4c9ac72530]
/usr/local/bin/spring [0x8f3c26]
/usr/local/bin/spring [0xe1ef5c]
/usr/local/bin/spring [0xe2503c]
/usr/local/bin/spring [0xe1f230]
/usr/local/bin/spring [0xe27eec]
/usr/local/bin/spring [0xe1e29d]
/usr/local/bin/spring [0xe1f64c]
/usr/local/bin/spring [0xe27f97]
/usr/local/bin/spring [0xe36dae]
/usr/local/bin/spring [0xe1ef5c]
/usr/local/bin/spring [0xe2503c]
/usr/local/bin/spring [0xe1f230]
/usr/local/bin/spring [0xe27eec]
/usr/local/bin/spring [0xe1e29d]
/usr/local/bin/spring [0xe1f64c]
/usr/local/bin/spring [0xe27f97]
/usr/local/bin/spring [0x9416e2]
/usr/local/bin/spring [0x9417ef]
/usr/local/bin/spring [0x94190c]
/usr/local/bin/spring [0x94ea0a]
/usr/local/bin/spring [0x94698d]
/usr/local/bin/spring [0xd231e6]
/usr/local/bin/spring [0x73b8e3]
/usr/local/bin/spring [0xcdc316]
/usr/local/bin/spring [0xcdc5ec]
/usr/local/bin/spring [0xcedbd6]
/usr/local/bin/spring [0xceded9]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7f4c9ac5dabd]
/usr/local/bin/spring [0x6dc769]
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0

[zerver]

Hi!

You can make a debug MT build by adding compiler flag USE_GML_DEBUG. This will tell you which widgets that cause the "GML error". These widgets must be disabled! (The error is there for a reason, it means the game will probably hang sooner or later.)

Regarding the crash, you will have to translate those stacktrace addresses yourself since you are on Linux.

FYI the windows build is stable enough imo, you just have to choose your widgets very carefully.

[Delta_]

I would have never guessed a Widget could cause that, wow! Also, I'll look into getting the stacktrace translated, not exactly sure how though. I'll post back later on with more information if I have any.

Thanks for your help!

[Delta_]

I disabled the widget that was causing the errors, and all seemed to be going good. I played a 40 minute game with no crashes. But when I started the second game, maybe 10 minutes into it, the game crashed with another segmentation fault. It looks like the same error to me, but I'm not sure.

I don't know what's wrong, there's nothing in the Spring logs to indicate a problem. So this is all I have to go on for now.

Is there some program I can use on Linux to translate this?

Code: Select all

[   9715] Segmentation fault (SIGSEGV) in spring 0.80.5.2
Stacktrace:
/usr/local/bin/spring [0xd38c7b]
/lib/libc.so.6 [0x7fb3ab567530]
/usr/local/bin/spring [0x8f691a]
/usr/local/bin/spring [0xe27ef8]
/usr/local/bin/spring [0xe2dfd8]
/usr/local/bin/spring [0xe281cc]
/usr/local/bin/spring [0xe30e88]
/usr/local/bin/spring [0xe27239]
/usr/local/bin/spring [0xe285e8]
/usr/local/bin/spring [0xe30f33]
/usr/local/bin/spring [0xe3fd4a]
/usr/local/bin/spring [0xe27ef8]
/usr/local/bin/spring [0xe2dfd8]
/usr/local/bin/spring [0xe281cc]
/usr/local/bin/spring [0xe30e88]
/usr/local/bin/spring [0xe27239]
/usr/local/bin/spring [0xe285e8]
/usr/local/bin/spring [0xe30f33]
/usr/local/bin/spring [0x946556]
/usr/local/bin/spring [0x946663]
/usr/local/bin/spring [0x946780]
/usr/local/bin/spring [0x954526]
/usr/local/bin/spring [0x94d0e2]
/usr/local/bin/spring [0xd2c4ea]
/usr/local/bin/spring [0x73b148]
/usr/local/bin/spring [0x73c0f0]
/usr/local/bin/spring [0xce523e]
/usr/local/bin/spring [0xce5514]
/usr/local/bin/spring [0xcf6b72]
/usr/local/bin/spring [0xcf6e75]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7fb3ab552abd]
/usr/local/bin/spring [0x6dc769]
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0

[Beherith]

I think you can translate stack traces by posting the whole infolog.txt to pastebin.ca, and joining #buildserv in the lobby, and saying:

!translate file=pastebin_url

[Delta_]

It doesn't appear to be working. Is there something I'm doing wrong?

Code: Select all

[23:56:52] <Delta> !translate file=http://pastebin.ca/1699695
[23:56:53] * BuildServ * ----- Start of translation process ( file=http://pastebin.ca/1699695 auto=yes links=yes ) -----
[23:56:53] * BuildServ * Downloading stacktrace...
[23:56:55] * BuildServ *     [OK] Downloaded at 93.3 KB/s
[23:56:55] * BuildServ * Detecting build parameters...
[23:56:55] * BuildServ *     [WARNING] Unable to find detailed version in infolog, autodetection disabled
[23:56:55] * BuildServ *     [WARNING] Using latest debug symbols available, this is probably NOT what you want!
[23:56:55] * BuildServ * Checking debug data availability...
[23:56:55] * BuildServ *     [OK]
[23:56:55] * BuildServ * Parsing stacktrace...
[23:56:55] * BuildServ *     [ERROR] Unable to find translatable stacktrace in file
[23:56:55] * BuildServ * ----- End of translation process (profile=default file=http://pastebin.ca/1699695 buildsys=cmake toolchain=mingw32 auto=yes vcs=git rev=0.79.0-699-gc06ccba links=yes ) -----

[hoijui]

stack trace translation on buildbot only works for windows builds.
on linux, you have to compile a debug version yourself, see instructions here:
http://springrts.com/wiki/Building_Spring_on_Linux
instead of cmake ., use this line:

Code: Select all

cmake -G "Unix Makefiles" -DCMAKE_BUILT_TYPE=DEBUG3 -DUSE_GML_DEBUG=1

[Delta_]

Alright, I compiled a debug version of Spring. I played a couple of games since then and still it haven't crashed, which is a good sign.

Another thing I've noticed is that the MT build reports way more CPU load than what, I think, is necessary according to my frames per second.

Here in this screen shot I'm holding 30-60 fps, but still zerver kicked me for lagging the game, when I clearly wasn't.

Full Size Image Link


Can someone please tell me why this is still so laggy. The game is using 130% (It might be using more, but that's what I've obsearved) out of a possible 200% of both CPU cores. Isn't that enough?

[hoijui]

FPS (graphics only) are not directly cuppled with lag (network or CPU restricted). i dont know the MT build in practise, but what i have heard, for some it is faster, and for others it is way slower, and it seems you are one of them, so maybe stay with the single threaded build. if it is not that, then it was your network connection, but from what info i got so far, this seems unlikely.

[zerver]

> 100% CPU makes me think there is some kind of bug...

If you start a single player test game, /cheat, /give all a few times, and then crank up the speed to 20, how high does your CPU load go?

Edit: Oh, btw in your screenshot your ping is also 4000 - a strong indication that your machine really has trouble keeping up with the speed.

[Delta_]

hoijui, it might have been my Internet connection lagging some, since I'm on a wireless ISP and my average latency is 130ms. Also At this point I think the MT build provides me with more frames per second than the single threaded build. Just higher overall CPU usage.

zerver, I ran the game with the cheats and set the game speed to 20 like you said. I Did a "/give all" two times, and my CPU usage, according to the game, was 98-102% and I was holding 50-70 fps.

I was watching the CPU usage on my system as well, and the game never seemed to use over 130% hanging around 100-120%.

At this point, I think, I might try reverting back to version 0.80.5.1. It seemed to report a lower CPU usage in game when I last tried it.

EDIT:

I just tried it again with a max of 1000 units. I don't know what happened to v-sync, but I was getting 450 fps and the simulation was extremely choppy: things would move forwards then bounce back. All why my CPU was sitting at a nice 100-102% usage.

[zerver]

Can you compare the CPU usage for MT vs non-MT to confirm that MT is higher?

Just use cheats like before and compare the CPU load for identical scenarios...

Try to use a scenario where CPU is ~50%.

[Jazcash]

Yeh, I used to have 100% CPU use with 400 or so FPS and 0 lag quite a while ago and then it stopped for some reason. I know a few other people who've had that too... Fake information is fake...

Could someone tell me how I can hide/block any messages containing the phrase "GML" from appearing in the console?

[hoijui]

mmm a regex filter would be nice! i miss it so much in all kind of chats already. guess that would be easy to do wiht a widget.

[zerver]

JAZ, actually 100% CPU + 400 FPS is feasible with MT, although I have never encountered it. The CPU value is for Sim, and the rendering is more or less independent of the Sim.

If you want to get rid of those GML errors, use a widget as hoi suggested, build your own executable or disable the bad widgets...

[Jazcash]

I'd love too but I don't know Lua atm. I'll probably learn it later in my life when I've covered the major languages
I think a built in "chat filter" system would be nice for everybody to use though.

Something like "If x is in line then hide line", or "hide x word", "highlight x" options with a nice smexy database with players inputting x.

I'd love to hide so so many annoying messages in the console.

[Delta_]

I was unable to reproduce the same CPU load for some reason. I did, however, see a 101% every now and then. I never saw anything above 99% with the single thread build.

Spring 0.80.5.2:
729 units with game speed set to 1: CPU Load 14-20% 100-130 fps
729 units with game speed set to 20: CPU Load 98-99% 1 fps

Spring 0.80.5.2 MT (2 Threads):
729 units with game speed set to 1: CPU Load 20-30% 90-130 fps
729 units with game speed set to 20: CPU Load 98-99% 140-218 fps

I only ran these tests once per build, so the numbers might be off some.

[zerver]

Okay, obviously some issues with the linux build. My guess is some synchronization issues leading to the Sim thread waiting (livelock).

Last thing you could try before we give up and wait for a linux developer to solve this:
Locate file gml.h
Find this line: #define GML_MUTEX_PROFILER 0 // enables profiler
Change to: #define GML_MUTEX_PROFILER 1 // enables profiler

Compile the MT build.

Run the game like you did before with 729 units.
Let it run for a minute or so.
Press B.
Check the list of mutexes if any one appears to use a large %.
Would be good if u also make a screen shot and post here.

[Delta_]

Alright, recompiled with "#define GML_MUTEX_PROFILER 1" and here are the results:

Screen Shot #1
Screen Shot #2
Screen Shot #3

Looks like the "luaMutex" is using the most of whatever that would be. But I really don't know what I'm looking at.

Edit:

Apparently the game doesn't like being minimized. Infolog.txt

It also spammed "spring: /usr/include/boost/thread/pthread/mutex.hpp:50: void boost::mutex::lock(): Assertion `!pthread_mutex_lock(&m)' failed." in the console a whole lot of times.