View Issue Details

IDProjectCategoryView StatusLast Update
0005254Spring engineGeneralpublic2016-05-27 14:59
ReporterDeinFreund Assigned Toabma  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionno change required 
Product Version101.0+git 
Summary0005254: Sudden crashes
DescriptionThe latest spring dev crashes in the first few minutes of each game, without any apparent reason. The games are AI only.

I'll attach a infologs of different crashes, demos are not being written.

http://hastebin.com/ekucugahog.xml
http://hastebin.com/ogagotetun.xml
http://hastebin.com/ubesirisix.xml
http://hastebin.com/ojagaxesuw.xml
TagsNo tags attached.
Checked infolog.txt for Errors

Activities

DeinFreund

2016-05-26 16:10

reporter   ~0016351

Last edited: 2016-05-26 16:15

Is there any way I can get those stack traces to show files/line numbers?

Or some other way to get more verbose logging regarding the cause of crash?

user744

2016-05-26 16:13

  ~0016352

https://springrts.com/wiki/Development:Getting_Started#Buildbot_and_Stacktrace_translator

DeinFreund

2016-05-26 16:19

reporter   ~0016353

Last edited: 2016-05-26 16:20

Ok, here are the translated stacktraces: http://hastebin.com/zarebojilo.sm

rts/lib/lua/include/LuaUser.cpp:191 is on two of them.

DeinFreund

2016-05-26 17:12

reporter   ~0016355

Sometimes there are units teleporting around before the game crashes, maybe some memory glitch caused by too much pointer juggling?

Trace + Infolog: http://paste.springfiles.com/view/281523d3

hokomoko

2016-05-26 17:39

developer   ~0016356

are you sure you're not just out of memory?

DeinFreund

2016-05-26 18:07

reporter   ~0016357

https://i.imgur.com/HcL0mQG.png

The drop is Spring crashing. I don't think memory is limiting here.

hokomoko

2016-05-26 18:15

developer   ~0016358

weird, maybe your AI has memory issues?

DeinFreund

2016-05-26 18:42

reporter   ~0016359

Ok, the issue is related to the java interface. It doesn't happen when my AI isn't playing.

abma

2016-05-26 19:03

administrator   ~0016360

Last edited: 2016-05-26 19:04

AFAIK spring 32 bit is limited to <4GB Memory.

-> memory usage of spring is relevant, not the available memory.

hokomoko

2016-05-26 19:22

developer   ~0016361

the bump in the image seems <4 GB

abma

2016-05-26 19:54

administrator   ~0016362

from infolog.txt:

<SkirmishAI: CSI (team 2)>: Initialized thread 1

are you sure the spring api isn't accessed outside of the spring-mainthread?

the ai interface isn't threadsafe! (!!!)

abma

2016-05-26 19:55

administrator   ~0016363

also: can i/we reproduce this issue somehow?

where is the sourcecode?

abma

2016-05-26 19:57

administrator   ~0016364

an other paste from your provided infolog.txt:

[f=0001410] Error: [ErrorMessageBox][1] msg="not enough memory" IsMainThread()=1 fromMain=0

DeinFreund

2016-05-26 21:10

reporter   ~0016365

I've been replacing/turning off all the multithreaded parts now but the crashes still happen. I'll check memory consumption on next run to see if maybe the 4 GB limit causes problems.

DeinFreund

2016-05-26 21:17

reporter   ~0016366

Another crash, but Spring never went above 1.2 GB. I couldn't see any big spikes in other runs either.

Is there maybe a way to log all JavaInterface calls?

DeinFreund

2016-05-26 21:43

reporter   ~0016367

Last edited: 2016-05-26 21:47

If you want to try to reproduce, I'm playing CSI vs CAI on Comet Catcher Redux, latest Zero-K Stable.

AI build: http://www.mediafire.com/download/44fbnguc2t1648n/zkcbai.zip

AI source: http://www.mediafire.com/download/cgouaoj3qkv0x4c/src.zip

The pathfinder precalculation is still being done multithreaded but that doesn't cause the crashing.

The crash can take up to 10 minutes to occur.

Originally multithreaded classes are PathPrecalculaterThread.java, BackgroundPathfinder.java, BuilderHandler.java.

abma

2016-05-27 07:58

administrator   ~0016368

Last edited: 2016-05-27 08:09

AI crashes first:

http://paste.springfiles.com/view/raw/af56dea8

then spring hangs and then spring crashes in frame -1 with "failed to allocate memory".

(64 Bit Ubuntu / 16GB RAM)

DeinFreund

2016-05-27 08:37

reporter   ~0016369

First exception is expected, it has to create that file on first run.
The 2nd exception is a bit odd, but neither of those should actually crash the AI. I.e. return non-zero.

I assume the AI GUI didn't load for you. I can check if it's the GUI that's causing some issues, because AWT runs in it's own thread by default.

DeinFreund

2016-05-27 08:40

reporter   ~0016370

Just noticed one of my newer optimizations actually causes the GUI to access the Interface :/

Will check that now.

DeinFreund

2016-05-27 09:28

reporter   ~0016371

Last edited: 2016-05-27 09:28

Ok, I can confirm that the issue was caused by the GUI thread accessing the interface. Thanks for your help in finding this.

If you think it's acceptable for Spring to crash here then the issue is resolved, otherwise I'd like to see Spring throw an exception when accessed from outside the main thread.

hokomoko

2016-05-27 09:33

developer   ~0016372

Checking the current thread is much more demanding than one would think. Even assertions can make the debug version barely playable.

DeinFreund

2016-05-27 09:50

reporter   ~0016373

A simple mutex to check whether the call happens outside update/other callins would have already helped here. (Because my own debug output showed that there were interface calls during those times)

This shouldn't cost more than a handful of cycles per callin.

hokomoko

2016-05-27 09:54

developer   ~0016374

It will.
Maybe not on linux, but definitely on windows.

Issue History

Date Modified Username Field Change
2016-05-26 16:09 DeinFreund New Issue
2016-05-26 16:10 DeinFreund Note Added: 0016351
2016-05-26 16:13 user744 Note Added: 0016352
2016-05-26 16:15 DeinFreund Note Edited: 0016351
2016-05-26 16:19 DeinFreund Note Added: 0016353
2016-05-26 16:20 DeinFreund Note Edited: 0016353
2016-05-26 17:12 DeinFreund Note Added: 0016355
2016-05-26 17:39 hokomoko Note Added: 0016356
2016-05-26 17:39 hokomoko Note View State: 0016353: private
2016-05-26 17:39 hokomoko Note View State: 0016353: public
2016-05-26 18:07 DeinFreund Note Added: 0016357
2016-05-26 18:15 hokomoko Note Added: 0016358
2016-05-26 18:42 DeinFreund Note Added: 0016359
2016-05-26 19:03 abma Note Added: 0016360
2016-05-26 19:04 abma Note Edited: 0016360
2016-05-26 19:22 hokomoko Note Added: 0016361
2016-05-26 19:54 abma Note Added: 0016362
2016-05-26 19:55 abma Note Added: 0016363
2016-05-26 19:57 abma Note Added: 0016364
2016-05-26 21:10 DeinFreund Note Added: 0016365
2016-05-26 21:17 DeinFreund Note Added: 0016366
2016-05-26 21:43 DeinFreund Note Added: 0016367
2016-05-26 21:47 DeinFreund Note Edited: 0016367
2016-05-27 07:58 abma Note Added: 0016368
2016-05-27 08:05 abma Note Edited: 0016368
2016-05-27 08:09 abma Note Edited: 0016368
2016-05-27 08:37 DeinFreund Note Added: 0016369
2016-05-27 08:40 DeinFreund Note Added: 0016370
2016-05-27 09:28 DeinFreund Note Added: 0016371
2016-05-27 09:28 DeinFreund Note Edited: 0016371
2016-05-27 09:33 hokomoko Note Added: 0016372
2016-05-27 09:50 DeinFreund Note Added: 0016373
2016-05-27 09:54 hokomoko Note Added: 0016374
2016-05-27 14:59 abma Status new => closed
2016-05-27 14:59 abma Assigned To => abma
2016-05-27 14:59 abma Resolution open => no change required