View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004568Spring engineGeneralpublic2014-09-30 20:042014-10-03 00:26
ReporterJools Assigned TojK  
PrioritynormalSeveritymajorReproducibilitysometimes
Status resolvedResolutionfixed 
Product Version97.0.1+git 
Summary0004568: Lua Garbage-Collector sometimes doesn't catch up fast enough when to many allocations are made / frame
DescriptionI have done some testing with several engine versions to see which ones cause increase in lua allocated memory. All tests are done with two KAIK 0.13 AI:s playing against each other on Coyote Creek V3, and with the game XTA test-785.

They are furthermore played with /luaui disabled and with /nocost on.

Lua-allocated memory is reported after 12:00 min of game time, along with a screenshot and infolog for that engine.

In spring 96.0 xta uses 4 lua states, in engine 97+ it uses only 3. With luaui disabled there are 2 states remaining in 97+. In 96.0.1, it also uses 2 states with luaui disabled.
Additional InformationEngine: Lua-allocated memory at 12:00:
--------------------------------------

97.0.1-450: 223.5 MB
97.0.1-350: 79.7 MB
97.0.1-250:~ 100 MB (screenshot missing)
97.0.1-210: 86.1 MB
97.0.1-208: 22.7 MB
97.0.1-208: 62.2 MB (second run)
97.0: 138.5 MB
96.0.1-408: 97.8 MB
96.0: 15.9 MB (with luaui)
96.0.1: 4.3 MB

It seems that the memory starts to increase after some event, because in 250 and 350 the memory stayed at about 30 Mb until maybe 9 minutes, then it started increasing. Once this happens it increases exponentially I seems. In 208 run 2 the memory stayed at 23 Mb until 11 mins, then it increased in 1 minute to 62.2 Mb.

At 12 minutes, the AI:s operating under no-cost have about 700 - 1000 units and are fighting a lot, although they may differ from game to game. They usually get about 700 units at the 8 minute mark or so.
TagsNo tags attached.
Attached Files
memorytest_infologs.zip (Attachment missing)
demos.zip (Attachment missing)
demos2.zip (Attachment missing)
screen00397_96.0.png (Attachment missing)
screen00402_208.png (Attachment missing)
screen00405_450.png (Attachment missing)
screen00406_350.png (Attachment missing)
screen00407_210.png (Attachment missing)
screen00408_97.0.png (Attachment missing)
screen00409_96-408.png (Attachment missing)
screen00410_208_2.png (Attachment missing)
screen00411_96.0.1.png (Attachment missing)
springsettings.cfg (Attachment missing)
infolog_ba_test3296.txt (Attachment missing)
20141001_200615_Coyote_CreekV4_97.0.1-450-g502a946 develop_ba_test_3296.sdf (Attachment missing)
20141001_200130_Coyote_CreekV4_97.0.1-450-g502a946 develop_bota_1.7.11.sdf (Attachment missing)
20141001_203230_Coyote_CreekV4_97.0.1-450-g502a946 develop_xta_barebone.sdf (Attachment missing)
20141003_004121_Four_Lakes_V2_97.0.1-452-g6150c70 develop.sdf (Attachment missing)
Checked infolog.txt for Errors

Relationships

related to 0004564 resolvedjK Crash due to memory allocation 

Activities

Jools

2014-09-30 20:10

reporter   ~0013683

Correction: 97.0.1-250: about 100 MB (screenshot missing)

Jools

2014-09-30 20:30

reporter   ~0013684

Last edited: 2014-09-30 20:31

 Possibly related to http://springrts.com/mantis/view.php?id=4564

abma

2014-09-30 21:19

administrator   ~0013685

the test was to run KAIK vs KAIK games to compare memory usage?

also please give the full version string, for example 97.0.1-250 is missing the hash, correct would be 97.0.1-447-ga9ae331 for example.

abma

2014-09-30 21:22

administrator   ~0013686

correct map name is "Coyote_Creekv3"

abma

2014-09-30 21:29

administrator   ~0013687

Last edited: 2014-09-30 21:36

 /luarules disable seems to free most lua memory used. so memleak very likely its synced lua code.

very likely the gadget that uses most cpu time is also the gadget memleaking.

for me "Cloak lose target" & "game_zombiemode" uses most cpu.

please try to disable these gadgets and see if still memleaks!

Jools

2014-09-30 22:06

reporter   ~0013688

Yes, it was run KAIK vs KAIK (v 0.13). Would it be better to test with replays? And with human people with give all?

I noticed the same with cloak and lose target. Zombies were just enabled to get more battle, will try again. However, there was still a difference between 96.0 and dev versions, but maybe they work differently.

Was trying to bisect where changes happened, but could not find any versions between 96.0.1 and 96.0.1-406

abma

2014-09-30 22:33

administrator   ~0013689

imo comparing versions doesn't make much sense as behaviour of the engine changed / features are added, etcetc. either you bisect or its mostly useless.

it would be great if some simple example would lead to the memleak, so testing as human makes more sense, you mostly know what you are doing and you can only do one thing at once. :)


so sth. like /give all or /give 100 someunit & /give 100 someunit 1

abma

2014-10-01 00:45

administrator   ~0013690

Last edited: 2014-10-01 00:48

 in 96.0.1-234-gae8db54 the garbage-collector was changed:
https://github.com/spring/spring/commit/ae8db547c93bba13e5fe2475ad648f1c93df283f

not sure if related.

it can be reproduced when:

1. game heavily lags (1fps or so)
2. lua memory increase is pretty fast.

just do some /cheat + /give all /give all 1

for me it looks atm as the garbage collector doesn't catch up fast enough, the other posibility is that XTA just uses to many memory. with BA i can't trigger this bug this way, but BA doesn't use less memory!


atm i'm not sure if this is a game or engine bug. no matter of that the memory usage of xta imo should be decreased!

abma

2014-10-01 01:57

administrator   ~0013691

Last edited: 2014-10-01 02:00

 increasing MaxLuaGarbageCollectionTime seems to fix it for me.

-> xta should reduce memory usage/allocations!

Jools

2014-10-01 01:59

reporter   ~0013692

Last edited: 2014-10-01 02:01

 I agree with that conclusion, I would have tested this on something light weight such as BOTA, but didn't find any AI that works for it.

Jools

2014-10-01 19:18

reporter   ~0013693

I tested again with spring_{develop}97.0.1-450-g502a946_minimal-portable and managed to reproduce the memory increase with other mods too. Memory increase seems to be proportional to the number of callin calls in the mod, with /luaui disabled and a minimum of essential gadgets in xta, there is no memory increase, but on the other hand, nothing is really happening either in lua. I attach such a barebone xta version.

I also managed to reproduce the memory increase with BOTA and with Balanced Annihilation test-3296 (infolog attached). It is reproducible by just giving 500 stumpies to each team and let them fight each other. It is interesting to note that even after the battle has ended, memory keeps increasing. I sped up the game and at 10 minutes ba had used over 100 Mb of memory.

BOTA has many gadgets from xta so it's maybe not a good test, but OTOH it has other effects and models. But ofc also ba and xta share some gadgets.

Another possibility is that the memory leak could be somehow tied to features/wrecks, since it happened also after battle was completed.

Jools

2014-10-01 19:33

reporter   ~0013694

Last edited: 2014-10-01 19:45

 It doesn't happen in xta with minimal gadgets (even with luaui), ie no callins done during gametime, so maybe it's related to gadgethandler?

Jools

2014-10-01 20:02

reporter   ~0013695

Last edited: 2014-10-01 20:07

 Link to xta-barebone: https://drive.google.com/uc?id=0B4WyZ8625I2LUzk4X0xvU0p3Rms&export=download

Link to bota 1.7.11: https://drive.google.com/uc?id=0B4WyZ8625I2LYlAyT3dxeklHMzQ&export=download

Jools

2014-10-02 23:45

reporter   ~0013698

I managed to run into the memory limit of about 750 Mb in 4 minutes or so of gametime, here's a link to the replay and the infolog.

Most memory was probaly caused by luaui, I didn't have profiler enabled but the fps was fine most of the time so the profiler is not really relevant anyway.

This time I tried with 97.0.1-452-g6150c70 develop and the game was XTA test-793 (rapid tag is xta:test)

Alt-b shows that the garbage collectors total time is about 1.7 seconds.

Issue History

Date Modified Username Field Change
2014-09-30 20:04 Jools New Issue
2014-09-30 20:04 Jools File Added: memorytest_infologs.zip
2014-09-30 20:10 Jools Note Added: 0013683
2014-09-30 20:14 Jools File Added: demos.zip
2014-09-30 20:15 Jools File Added: demos2.zip
2014-09-30 20:18 Jools File Added: screen00397_96.0.png
2014-09-30 20:19 Jools File Added: screen00402_208.png
2014-09-30 20:22 Jools File Added: screen00405_450.png
2014-09-30 20:22 Jools File Added: screen00406_350.png
2014-09-30 20:23 Jools File Added: screen00407_210.png
2014-09-30 20:24 Jools File Added: screen00408_97.0.png
2014-09-30 20:27 Jools File Added: screen00409_96-408.png
2014-09-30 20:28 Jools File Added: screen00410_208_2.png
2014-09-30 20:29 Jools File Added: screen00411_96.0.1.png
2014-09-30 20:29 Jools File Added: springsettings.cfg
2014-09-30 20:30 Jools Note Added: 0013684
2014-09-30 20:31 Jools Note Edited: 0013684
2014-09-30 21:16 abma Additional Information Updated
2014-09-30 21:19 abma Note Added: 0013685
2014-09-30 21:19 abma Status new => feedback
2014-09-30 21:22 abma Note Added: 0013686
2014-09-30 21:29 abma Note Added: 0013687
2014-09-30 21:32 abma Note Edited: 0013687
2014-09-30 21:36 abma Note Edited: 0013687
2014-09-30 21:37 abma Relationship added related to 0004564
2014-09-30 22:06 Jools Note Added: 0013688
2014-09-30 22:06 Jools Status feedback => new
2014-09-30 22:33 abma Note Added: 0013689
2014-10-01 00:45 abma Note Added: 0013690
2014-10-01 00:46 abma Note Edited: 0013690
2014-10-01 00:47 abma Note Edited: 0013690
2014-10-01 00:48 abma Note Edited: 0013690
2014-10-01 00:48 abma Note Edited: 0013690
2014-10-01 01:57 abma Note Added: 0013691
2014-10-01 01:59 Jools Note Added: 0013692
2014-10-01 02:00 abma Note Edited: 0013691
2014-10-01 02:01 Jools Note Edited: 0013692
2014-10-01 02:05 abma Summary Increase in Lua-allocated memory => Lua Garbage-Collector sometimes doesn't catch up fast enough when to many allocations are made / frame
2014-10-01 19:18 Jools Note Added: 0013693
2014-10-01 19:18 Jools File Added: infolog_ba_test3296.txt
2014-10-01 19:20 Jools File Added: 20141001_200615_Coyote_CreekV4_97.0.1-450-g502a946 develop_ba_test_3296.sdf
2014-10-01 19:22 Jools File Added: 20141001_200130_Coyote_CreekV4_97.0.1-450-g502a946 develop_bota_1.7.11.sdf
2014-10-01 19:33 Jools Note Added: 0013694
2014-10-01 19:33 Jools File Added: 20141001_203230_Coyote_CreekV4_97.0.1-450-g502a946 develop_xta_barebone.sdf
2014-10-01 19:45 Jools Note Edited: 0013694
2014-10-01 20:02 Jools Note Added: 0013695
2014-10-01 20:07 Jools Note Edited: 0013695
2014-10-02 23:45 Jools Note Added: 0013698
2014-10-02 23:46 Jools File Added: 20141003_004121_Four_Lakes_V2_97.0.1-452-g6150c70 develop.sdf
2014-10-03 00:26 jK Changeset attached => spring develop 3d2568f6
2014-10-03 00:26 jK Assigned To => jK
2014-10-03 00:26 jK Status new => resolved
2014-10-03 00:26 jK Resolution open => fixed