View Issue Details

IDProjectCategoryView StatusLast Update
0005754Spring engineGeneralpublic2017-09-13 19:39
ReporterGoogle_Frog Assigned Tohokomoko  
PrioritynormalSeveritycrashReproducibilityhave not tried
Status resolvedResolutionfixed 
Product Version103.0 +git 
Fixed in Version103.0 +git 
Summary0005754: 103.0.1-1336 Load crash with only half a stacktrace
DescriptionThe bottom of the stacktrace is missing and we see a few of these: https://github.com/ZeroK-RTS/CrashReports/issues/1986

Is this enough? Can something be done to get the rest?
TagsNo tags attached.
Checked infolog.txt for Errors

Activities

Kloot

2017-09-08 03:02

developer   ~0018355

Last edited: 2017-09-08 03:02

it could be the (rather arcane) stacktracer is simply bugged.

another possibility (assuming similar timeouts) is that users are allowing Windows to kill Spring before it has a chance to write the trace to infolog, although I would expect more partial dumps in that case.

Google_Frog

2017-09-08 03:56

reporter   ~0018356

I have seen many reports with this particular cutoff in the infolog so, unless there is a long pause between writing the two parts of the stacktrace, I think it is unlikely that these are due to killing Windows.

abma

2017-09-08 12:47

administrator   ~0018357

Spring exit code is: -805306369,

i'm pretty sure this is due killing...

abma

2017-09-08 12:49

administrator   ~0018358

https://springrts.com/mantis/view.php?id=5504#c17396

-805306369 -> 0xCFFFFFFF ->

https://msdn.microsoft.com/en-us/library/windows/hardware/ff558784%28v=vs.85%29.aspx

STATUS_APPLICATION_HANG

Kloot

2017-09-08 17:18

developer   ~0018359

please update to c98e00e5

Kloot

2017-09-09 23:54

developer   ~0018364

considering this resolved until I see a counter-example.

Google_Frog

2017-09-10 04:25

reporter   ~0018365

Here is a load crash with half a stacktrace with 103.0.1-1380-g56d4e44: https://github.com/ZeroK-RTS/CrashReports/issues/2016

Google_Frog

2017-09-10 10:45

reporter   ~0018366

This too: https://github.com/ZeroK-RTS/CrashReports/issues/2020

Kloot

2017-09-12 22:58

developer   ~0018374

Last edited: 2017-09-12 23:00

builds after 9e9dbfc4 will generate more debug output.

overzealous anti-virus software may also be triggering many of these hangs (e.g. https://github.com/ZeroK-RTS/CrashReports/issues/2034 references atcuf32.dll aka Bitdefender) in which case there is no engine-side solution.

hokomoko

2017-09-13 14:21

developer   ~0018376

in local tests, the engine overwritten the stacktraces when logging lines after it

hokomoko

2017-09-13 14:43

developer   ~0018377

you may consider reverting the alternative crashhandler logging codepath after https://github.com/spring/spring/commit/e72b969d151d12594ad31f59cf7060c8558faa71 (assuming it works).

Kloot

2017-09-13 15:14

developer   ~0018378

I'd be happy to, but why would spring::thread behave differently than std::thread? it's just a typedef, https://github.com/spring/spring/blob/develop/rts/System/Threading/SpringThreading.h#L53

hokomoko

2017-09-13 15:19

developer   ~0018379

you are correct, I think I've now found the actual reason and am working on it

hokomoko

2017-09-13 17:50

developer   ~0018380

hmm, I'm still not at the source of the matter, but when I revert pre https://github.com/spring/spring/commit/6a552f66f2a965afd15b7e432b50d57460c6553d
I don't have the same hangs on startup.

hokomoko

2017-09-13 18:14

developer   ~0018381

after the above commit, the watchdog hangs on:
https://github.com/spring/spring/blob/6a552f66f2a965afd15b7e432b50d57460c6553d/rts/System/Platform/Win/CrashHandler.cpp#L275

hokomoko

2017-09-13 18:21

developer   ~0018382

Last edited: 2017-09-13 18:22

If I comment it out, it hangs on SNPRINTF calls.
I think there's some internal windows lock that is required for prints/string ops and when we suspend the thread we get stuck.
I guess the best solution may be to delay all printing until the thread is resumed.
I'll try and implement this.

Kloot

2017-09-13 18:53

developer   ~0018383

Last edited: 2017-09-13 19:13

Seems like a reasonable explanation given how much crap is written to infolog during loading, the suspended thread could easily be holding some OS lock already at the moment hang detection triggers (which it only does once per second) although I wouldn't expect sprintf to require any.

If we can't even write to a buffer, I guess a simple list (or custom spring_sprintf) will have to do.

hokomoko

2017-09-13 19:39

developer   ~0018384

Fix 508a1973d21530d8fb8f2ebacea140486e2bad4d committed to develop branch: fix 0005754, repo: spring changeset id: 8761

Issue History

Date Modified Username Field Change
2017-09-08 02:30 Google_Frog New Issue
2017-09-08 03:02 Kloot Note Added: 0018355
2017-09-08 03:02 Kloot Note Edited: 0018355
2017-09-08 03:02 Kloot Note Edited: 0018355
2017-09-08 03:56 Google_Frog Note Added: 0018356
2017-09-08 12:47 abma Note Added: 0018357
2017-09-08 12:49 abma Note Added: 0018358
2017-09-08 14:45 Kloot Assigned To => Kloot
2017-09-08 14:45 Kloot Status new => assigned
2017-09-08 17:18 Kloot Status assigned => feedback
2017-09-08 17:18 Kloot Note Added: 0018359
2017-09-09 23:54 Kloot Status feedback => resolved
2017-09-09 23:54 Kloot Resolution open => fixed
2017-09-09 23:54 Kloot Fixed in Version => 103.0 +git
2017-09-09 23:54 Kloot Note Added: 0018364
2017-09-10 04:25 Google_Frog Status resolved => feedback
2017-09-10 04:25 Google_Frog Resolution fixed => reopened
2017-09-10 04:25 Google_Frog Note Added: 0018365
2017-09-10 10:45 Google_Frog Note Added: 0018366
2017-09-10 10:45 Google_Frog Status feedback => assigned
2017-09-12 22:58 Kloot Note Added: 0018374
2017-09-12 22:59 Kloot Assigned To Kloot =>
2017-09-12 22:59 Kloot Status assigned => feedback
2017-09-12 23:00 Kloot Note Edited: 0018374
2017-09-13 14:21 hokomoko Note Added: 0018376
2017-09-13 14:43 hokomoko Note Added: 0018377
2017-09-13 15:14 Kloot Note Added: 0018378
2017-09-13 15:19 hokomoko Note Added: 0018379
2017-09-13 17:50 hokomoko Note Added: 0018380
2017-09-13 18:14 hokomoko Note Added: 0018381
2017-09-13 18:21 hokomoko Note Added: 0018382
2017-09-13 18:22 hokomoko Note Edited: 0018382
2017-09-13 18:53 Kloot Note Added: 0018383
2017-09-13 18:55 Kloot Note Edited: 0018383
2017-09-13 19:13 Kloot Note Edited: 0018383
2017-09-13 19:39 hokomoko Changeset attached => spring develop 508a1973
2017-09-13 19:39 hokomoko Note Added: 0018384
2017-09-13 19:39 hokomoko Assigned To => hokomoko
2017-09-13 19:39 hokomoko Status feedback => resolved
2017-09-13 19:39 hokomoko Resolution reopened => fixed