Remove hang detection

Remove hang detection

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Post Reply
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14599
Joined: 17 Nov 2005, 02:43

Remove hang detection

Post by Forboding Angel » 24 Dec 2018, 19:41

Can we please have hang detection completely removed (actual off switch that defaults to being off, not advocating for actual code removal) from production versions of spring? I understand that it is valuable for debugging, but in practice what happens is that people with low end processors get stuck in a hang detection loop where hang detection triggers repeatedly.

Frustratingly, hang detection will also trigger for other unrelated reasons as well. Occasionally, it will even trigger for me on load. I believe this can happen if you have some intensive lua stuff going on at load (in my case, it would generally be printing hundreds/thousands of debug lines to infolog when testing some things).

But in practice, we end up with some very unhappy users that can actually run the games fine, but essentially take so long to load due to recurring hang detection that the host times them out. It's very frustrating from a gamedev standpoint. We end up not having any real control. Even when using hangtimeout=-1 in springsettings, it will still trigger! I keep having this come up, and every time I deal with it, but it gets more and more frustrating each time.

There have been requests in the past for hang detection's removal. For that matter, from it's inception it received a cold reception because of the issues that it caused, but most of us understand it's value. That said, it's value goes way down when it interferes with "normal" operation.
Today's Struggle wrote:Dio BrandoToday at 9:16 AM
Hi, somebody said they could help my friend with his issue. He keeps crashing just before he loads into a match.(edited)

Scary le PooToday at 9:16 AM
Is there an error that pops up?

Dio BrandoToday at 9:17 AM
No it just says the programs stops responding.

Scary le PooToday at 9:18 AM
That's a new one
Ok
Do this
From the main lobby menu, click settings, then click pastebin infolog and put the link here

Dio BrandoToday at 9:19 AM
Alright he is doing it now.

Scary le PooToday at 9:20 AM
Hardware?
Cpu/ram/vidcard

Dio BrandoToday at 9:21 AM
I don't know I'll ask him

ZoptoToday at 9:23 AM
rip raaar

Dio BrandoToday at 9:27 AM
@Scary le Poo He sent me this
http://paste.springfiles.com/view/30ab50b5

Scary le PooToday at 9:30 AM
Tell him to in steam library
Right click Evo, properties, local files tab, browse local files, navigate to evo_data folder, upload infolog.txt (the entire thing) to somewhere. Pastebin.com is a good place

Dio BrandoToday at 9:33 AM
Alright I viewed his shared screen as well he isn't randomly crashing he is just stuck in a infinite 'loading please wait' state

Scary le PooToday at 9:33 AM
Ohhhhh
It's hang detection
Easy fix
Sec

Dio BrandoToday at 9:35 AM
I told him to be patient and he finally loaded in however it says I've left when I'm in the game waiting for him
Scary le PooToday at 9:36 AM
Yeah the server gave up waiting
Ok so the fix is
In the Evo data folder
There is a file called springsettings.cfg
Add a new line to the bottom of it in notepad and put this
hangtimeout=-10
That should tell hang detection to fuck off

Dio BrandoToday at 9:37 AM
kk
like this?
https://gyazo.com/ef28b303fd2b4f177e5e2e0c20327efb
Gyazo
Gyazo

Scary le PooToday at 9:39 AM
It's cause his processor is being slow to load. The engine is writing stack traces because it thinks he is going to crash
Yep

Dio BrandoToday at 9:39 AM
kk thanks I'll see if this fixes it for him

Scary le PooToday at 9:40 AM
Good deal :-)

Dio BrandoToday at 9:47 AM
Thanks it works now!
Nvm, he timed out..

Scary le PooToday at 10:08 AM
Shit
@Dio Brando ok I'm at an actual computer now
can you get me his infolog.txt?

Dio BrandoToday at 10:09 AM
Hes went offline
Idk when he will be back

Scary le PooToday at 10:13 AM
ok, sorry
His hardware seems to be under spec
is it a laptop perchance?
What is triggering hang detection is slow loading and I believe cpu deadlock. As a result the engine thinks he is going to crash so it writes a stack trace to infolog real quick before the process can die, but then his cpu starts responding again, and then stops, which triggers another stack trace. Rinse and repeat 20 times.

There is a movement to get hang detection removed from the engine entirely, as many of the gamedevs find it irritating. But the engine devs value it for debugging purposes where otherwise they would get no stacktrace information.

So it's a bit of an ongoing battle
It only happens with really low end cpus generally
@Dio Brando try setting hangtimeout value to -1 instead of -10 and see if there is an improvement
in theory the values should achieve an identical result, so this shouldn't effect anything, but just for the sake of being thorough
If should be noted that my descriptions to Dio of what hang detection does and what it's for are intentionally simplistic. There is nothing to be gained by giving him a full technical rundown, many aspects of which I wouldn't even necessarily fully grasp.
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14599
Joined: 17 Nov 2005, 02:43

Re: Remove hang detection

Post by Forboding Angel » 25 Dec 2018, 02:25

In viewing threads from 2012 I found out that HangTimeout may actually be case sensitive which would be a bit mind boggling, but it should definitely bear some testing.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Remove hang detection

Post by gajop » 25 Dec 2018, 02:39

No, I want hang timeout to be on by default, anything that helps detect issues is good.

That said, if you want, you can disable it completely, or simply disable it in specific blocks of code with the new clearwatchdog parameters.
hangtimeout=-1 in springsettings, it will still trigger!
This seems like a bug that should be properly reported (full infolog please)
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14599
Joined: 17 Nov 2005, 02:43

Re: Remove hang detection

Post by Forboding Angel » 25 Dec 2018, 05:00

It only helps when people are posting those infologs to mantis. People from steam are not coming here for help (nor should they be expected to). People from zk are not coming here to post on mantis.

HangTimeout drives away players, because there are many with low end machines that can run the games just fine, but get stuck in the HangTimeout loop. Look at that pastebin above. It's just a continual hang detection loop that is completely unnecessary because while is computer loads the game more slowly, it can very much run the game. How are you not seeing this?
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Remove hang detection

Post by gajop » 25 Dec 2018, 07:19

How do they get 'stuck' in the HangTimeout loop? Is it happening too frequently and somehow causing spring not to load? I don't understand what you're saying exactly.
The above pastebin is incomplete, it's hard to make any guesses from it (such as what their springsettings were). Generally it's useful and can help find bugs such as https://springrts.com/mantis/view.php?id=6105

I think you can just disable it for Evo if you don't want it. I disabled it for a bit for SB, until this was implemented ( https://springrts.com/mantis/view.php?id=6102 ), because I have some very long operations (like texture export).
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14599
Joined: 17 Nov 2005, 02:43

Re: Remove hang detection

Post by Forboding Angel » 25 Dec 2018, 12:37

The pastebin is incomplete because it is too long (it's an issue with weblobby's automatic pastebin thing... it's finicky), but if you look at it, hang detection is triggered 5 times in succession in just this little sliver of infolog.
Is it happening too frequently and somehow causing spring not to load?
YES! And it's been a problem for YEARS!!!! Do a forum search for hang detection and hang timeout. You'll find a myriad of posts with people reporting the exact same behavior over and over and over AND OVER AND OVER! Since its inception it has been causing issues in the engine; this is literally undeniable by any reasonable person. It's not like I'm inventing scenarios here. It has been reported on these forums for over 6 years repeatedly. Pardon me for getting a bit worked up about it when something that is a known and obvious issue is ignored because it makes troubleshooting more convenient for engine devs. There are better ways to implement something like this.
Someone wrote:Well just include a springsettings with steam/itch/whatever that disables it.
You know, that would be a great idea. But what about when steam or itch verifies files, either automatically or at user request. Or when I push an update and that verification process overwrites the file with the default. That is a significant issue.
I think you can just disable it for Evo if you don't want it.
I cannot do this in the latest released engine. I am hoping that in the next engine release, maybe if I whine and moan enough, we can get some control over it.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Remove hang detection

Post by gajop » 25 Dec 2018, 14:39

Seems like your real issue is overriding engine default settings.
This is something you should look into, as you can't really expect default settings to fit your every need.
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14599
Joined: 17 Nov 2005, 02:43

Re: Remove hang detection

Post by Forboding Angel » 26 Dec 2018, 07:07

I suppose I'll have to just write a littler helper program to generate the springsettings.cfg for me if one doesn't already exist.

This should not be an issue and having the off switch for hang detection reside in springsettings which the engine will delete any value that matches default (which in and of itself is nonsensical) is just a lack of foresight imo.
0 x

Post Reply

Return to “Engine”