| View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||
| ID | Project | Category | View Status | Date Submitted | Last Update | ||||
|---|---|---|---|---|---|---|---|---|---|
| 0001415 | Spring engine | Linux | public | 2009-05-26 16:48 | 2009-05-27 09:59 | ||||
| Reporter | tvo | ||||||||
| Assigned To | tvo | ||||||||
| Priority | normal | Severity | crash | Reproducibility | always | ||||
| Status | resolved | Resolution | fixed | ||||||
| Product Version | 0.79.0.2 | ||||||||
| Target Version | Fixed in Version | ||||||||
| Summary | 0001415: infinite memory usage in CBFGroundDrawer | ||||||||
| Description | Spring tries to allocate a near-infinite amount of memory just after finalizing. Together with det I tracked this down to CBFGroundDrawer, will add log of the gdb session later. See also: - http://springrts.com/phpbb/viewtopic.php?p=353761#p353761 - http://springrts.com/phpbb/viewtopic.php?f=11&t=18969 People with this issue: - det - [RoX]Bashar - [oxnull]Lupus - billyboy747 Everyone with this issue is using 64 bit Linux and an ATI card, might be related. | ||||||||
| Additional Information | #0 0x00007f94e9a173e5 in memcpy () from /lib/libc.so.6 #1 0x0000000000a0e771 in CVertexArray::EnlargeDrawArray (this=0x3626af0) at ~/projects/spring/rts/Rendering/GL/VertexArray.cpp:321 #2 0x00000000007f8202 in CVertexArray::EnlargeArrays (this=0x3626af0, vertexes=4294967140, strips=4294967255, stripsize=3) at ~/projects/spring/rts/Rendering/GL/VertexArray.inl:22 0000003 0x000000000096b5a0 in CBFGroundDrawer::DoDrawGroundRow (this=0x3e486a0, bty=1) at ~/projects/spring/rts/Map/SMF/BFGroundDrawer.cpp:364 0000004 0x0000000000966184 in CBFGroundDrawer::Draw (this=0x3e486a0, drawWaterReflection=false, drawUnitReflection=true, VP=0) at ~/projects/spring/rts/Map/SMF/BFGroundDrawer.cpp:716 0000005 0x0000000000a2e207 in CUnitDrawer::CreateReflectionFace (this=0x61c4f10, gltype=34069, camdir= {static binder = {class_ = 0x327e4a0, base = 0x0, flags = creg::CF_None, memberRegistrator = 0x12500c8, name = 0xe1163c "float3", size = 12, constructor = 0xc90f36 <float3::_ConstructInstance(void*)>, destructor = 0xc90f66 <float3::_DestructInstance(void*)>, nextBinder = 0x1250080}, static memberRegistrator = 0x1250160, static maxxpos = 5119, static maxzpos = 5119, x = -2.39488166e+32, y = 4.59163468e-41, z = 0}) at ~/projects/spring/rts/Rendering/UnitModels/UnitDrawer.cpp:1411 #6 0x0000000000a2e3b7 in CUnitDrawer::UpdateReflectTex (this=0x61c4f10) at ~/projects/spring/rts/Rendering/UnitModels/UnitDrawer.cpp:1356 #7 0x00000000006b80a3 in CGame::Draw (this=0x39d7300) at ~/projects/spring/rts/Game/Game.cpp:2830 #8 0x0000000000caeafd in SpringApp::Update (this=0x7ffff53cf670) at ~/projects/spring/rts/System/SpringApp.cpp:894 #9 0x0000000000cb6a82 in SpringApp::Run (this=0x7ffff53cf670, argc=2, argv=0x7ffff53cf7a8) at ~/projects/spring/rts/System/SpringApp.cpp:1105 0000010 0x0000000000cb6fd6 in Run (argc=2, argv=0x7ffff53cf7a8) at ~/projects/spring/rts/System/Main.cpp:95 #11 0x0000000000cb702b in main (argc=2, argv=0x7ffff53cf7a8) at ~/projects/spring/rts/System/Main.cpp:111 < tvo> det: could you go to EnlargeDrawArray function (using up/down commands) < tvo> and run p pos, p oldsize, and p newsize < det> #1 0x0000000000a0e771 in CVertexArray::EnlargeDrawArray (this=0x45e69b0) < det> at ~/projects/spring/rts/Rendering/GL/VertexArray.cpp:321 < det> 321IImemcpy(tempArray,drawArray,oldsize*sizeof(float)); < det> (gdb) p pos < det> $1 = 1650 < det> (gdb) p oldsize < det> $2 = 524288000 < det> (gdb) p newsize < det> $3 = 1048576000 < tvo> humm that seems fair bit too big < tvo> det: maybe go one up < tvo> and p vertexes, p strips and p stripsize < det> (gdb) p vertexes < det> $4 = 4294967140 < det> (gdb) p strips < det> $5 = 4294967255 < det> (gdb) p stripsize < det> $6 = 3 < tvo> det: one up again < tvo> det: could you p nloop? < det> gdb) up < det> 0000003 0x000000000096b5a0 in CBFGroundDrawer::DoDrawGroundRow (this=0x53c1180, < det> bty=1) at ~/projects/spring/rts/Map/SMF/BFGroundDrawer.cpp:364 < det> 364IIIIIma->EnlargeArrays(52*nloop, 14*nloop+1); //! includes one extra for final endstrip < det> (gdb) p nloop < det> $7 = -3 < tvo> ah! < tvo> could you p xe and p xs < det> (gdb) p xe < det> $8 = 256 < det> (gdb) p xs < det> $9 = 264 < tvo> so x end is bigger then x start < tvo> so it allocates negative number of vertices < tvo> which cause of unsigned fail becomes 4G of vertices < det> yet it only seems to happen in first time map loads :x maybe coincidence < tvo> det: suppose this is enough for now, this code is too complex to quickly understand while your PC is sitting with 4G of RAM allocated ;) < tvo> det: thanks for the help, I think this will help a lot tracing down the real cause already < det> No, thank you for the help :-) | ||||||||
| Tags | No tags attached. | ||||||||
| Checked infolog.txt for Errors | |||||||||
| Attached Files |
| ||||||||
 Relationships |
||||||
|
||||||
 Relationships |
| Notes |
|
|
Lupus (reporter) 2009-05-26 18:14 |
I'm running gentoo 64 bit with MT spring, not Windows |
|
tvo (reporter) 2009-05-26 19:38 |
Ah right, updated issue. Seems the pattern is clearly 64 bit Linux then ;-) |
|
Kloot (developer) 2009-05-26 20:09 |
What could be the 64-bit connection though? EnlargeArrays() takes unsigned ints, which have the same width in either case, and nloop is a plain int everywhere it appears. |
|
Kloot (developer) 2009-05-26 20:15 Last edited: 2009-05-26 20:29 |
FYI, I just added an assert(nloop >= 0); in CBFGroundDrawer::DoDrawGroundRow after the first "int nloop", and it failed on my 32-bit machine too (loading SSB, which normally works fine). Backtrace: 0000004 0x083b65de in CBFGroundDrawer::DoDrawGroundRow (this=0xae470d0, bty=1) at rts/Map/SMF/BFGroundDrawer.cpp:364 0000005 0x083b21c7 in CBFGroundDrawer::Draw (this=0xae470d0, drawWaterReflection=false, drawUnitReflection=true, VP=0) at rts/Map/SMF/BFGroundDrawer.cpp:717 #6 0x08335463 in CUnitDrawer::CreateReflectionFace (this=0xc117f68, gltype=34069, camdir= {static binder = {class_ = 0xa4d8aa0, base = 0x0, flags = creg::CF_None, memberRegistrator = 0x86dda64, name = 0x85f318a "float3", size = 12, constructor = 0x81068a0 <float3::_ConstructInstance(void*)>, destructor = 0x81068c0 <float3::_DestructInstance(void*)>, nextBinder = 0x86dda40}, static memberRegistrator = 0x86ddad4, static maxxpos = 8191, static maxzpos = 8191, x = -1.11327457, y = -2.85530205e-05, z = -1.11326933}) at rts/Rendering/UnitModels/UnitDrawer.cpp:1411 #7 0x083355b6 in CUnitDrawer::UpdateReflectTex (this=0xc117f68) at rts/Rendering/UnitModels/UnitDrawer.cpp:1356 #8 0x084b8355 in CGame::Draw (this=0xa8e3100) at rts/Game/Game.cpp:2830 #9 0x080f2b7d in SpringApp::Update (this=0xbf8e84d0) at rts/System/SpringApp.cpp:894 0000010 0x080f8e0c in SpringApp::Run (this=0xbf8e84d0, argc=1, argv=0xbf8e85c4) at rts/System/SpringApp.cpp:1105 Variable details: (gdb) print nloop // (xe - xs) / lod + 1 $1 = -1 (gdb) print xe // xe = xend $2 = 256 (gdb) print xs // xs = xstart $3 = 272 (gdb) print lod $4=8 (gdb) print neededLod $11 = 142 (gdb) print bty $5 = 1 (gdb) print minlx // xstart = max(minlx, mintx) $6 = 152 (gdb) print mintx $7 = 128 (gdb) print maxlx // xend = min(maxlx, maxtx) No symbol "maxlx" in current context. // ?? (gdb) print maxtx $8 = 256 (gdb) print viewRadius $9 = 14 (gdb) frame 6 (gdb) print camera->pos $13 = {static binder = {class_ = 0xa4d8aa0, base = 0x0, flags = creg::CF_None, memberRegistrator = 0x86dda64, name = 0x85f318a "float3", size = 12, constructor = 0x81068a0 <float3::_ConstructInstance(void*)>, destructor = 0x81068c0 <float3::_DestructInstance(void*)>, nextBinder = 0x86dda40}, static memberRegistrator = 0x86ddad4, static maxxpos = 8191, static maxzpos = 8191, x = 2000, y = 50, z = 2023.57043} (gdb) print readmap->currMinHeight $15 = -95 (gdb) print readmap->currMaxHeight $16 = 347.059875 |
|
tvo (reporter) 2009-05-26 20:23 Last edited: 2009-05-26 20:28 |
Asked det to perform a "bt full" -> http://pastebin.ca/1435226 Also, <det@IRC> I reverted e7dc30f1d09a503e0de7319a436bad1127d41430 <det@IRC> And now the problem is gone <det@IRC> Ok, I am confident that reverting that commit fixes it <det@IRC> I can reproduce 100% for non-revert spring by rm -rf paths and dropping vm cache <det@IRC> and I tried like 10x with reverted spring binary on various maps <det@IRC> I take it back :< <det@IRC> happening again |
|
tvo (reporter) 2009-05-26 20:54 |
The while((char *)drawArrayPos>(char *)drawArraySize-stripsize*sizeof(float)*vertexes) (VertexArray.inl:21) suppresses the symptoms on 32 bit cause on 32 bit the char* minus huge unsigned integer wraps around and ends up being just a bit larger then the char*, so the condition is never true if nloop underflows and it doesn't allocate anything. On 64 bit the minus will be done using 64 bits however so it doesn't wrap around and ends up being lower then the char*, so it will actually perform the 4G allocation. |
|
jK (developer) 2009-05-26 21:01 Last edited: 2009-05-26 21:07 |
yup, Kloot is right and the corresponding commit is: 4f939d82fb8eb323e3ff12b2a55474151d3bd331 and the problem is not to see in the int -> unsigned int change, the bug is that nloop can be negative, so either it needs to be inverted or it is just the wrong input for EnlargeArrays(), and with wrong args to EnlargeArrays it could overwrite random data in AddVertexQ which doesn't check the alloc'ed array size. |
|
Kloot (developer) 2009-05-26 21:12 Last edited: 2009-05-26 21:13 |
Yeah, the ultimate cause seems to be the values in left and right themselves, which leads to FindRange() setting xs > xe for certain y: i: 0, y: 152, flil->base: 219.426651, flil->dir: -0.000000 i: 1, y: 152, flil->base: 209.186462, flil->dir: 0.000000 i: 2, y: 152, flil->base: 332.048615, flil->dir: -0.324372 i: 3, y: 152, flil->base: 167.951385, flil->dir: 0.324372 i: 0, y: 152, flir->base: 1298.223511, flir->dir: -0.000000 |
|
tvo (reporter) 2009-05-26 21:15 |
I've committed a (temp?) fix in 660bc902d0abaae9cc61452c4d005aef54236cfa by skipping the row if xe < xs. From what I understood from the code it makes sense to not enlarge vertex array at all if the body of the loop from xs..xe won't execute a single time, and I don't see any change in the rendering. |
|
Lupus (reporter) 2009-05-26 21:21 |
gonna try it right now |
|
Lupus (reporter) 2009-05-26 23:07 |
Yeah, it works now. GJ |
|
tvo (reporter) 2009-05-26 23:42 |
There was still another case, fixed in a70a9af32f235ac644f44720cfd460b7c700ba57. (Asserted that end > start or nloop > 0 in every place where it is used.) |
|
tvo (reporter) 2009-05-27 09:59 |
det reported this last commit fixed it for him |
| Notes |
| Issue History |
|||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2009-05-26 16:48 | tvo | New Issue | |
| 2009-05-26 16:49 | tvo | Status | new => assigned |
| 2009-05-26 16:49 | tvo | Assigned To | => tvo |
| 2009-05-26 18:14 | Lupus | Note Added: 0003486 | |
| 2009-05-26 19:37 | tvo | Additional Information Updated | |
| 2009-05-26 19:38 | tvo | Note Added: 0003488 | |
| 2009-05-26 19:38 | tvo | Description Updated | |
| 2009-05-26 19:40 | tvo | Relationship added | has duplicate 0001417 |
| 2009-05-26 19:53 | tvo | Summary | infinite memory usage after finalizing => infinite memory usage in CBFGroundDrawer |
| 2009-05-26 19:53 | tvo | Description Updated | |
| 2009-05-26 20:08 | tvo | Description Updated | |
| 2009-05-26 20:09 | Kloot | Note Added: 0003491 | |
| 2009-05-26 20:15 | Kloot | Note Added: 0003492 | |
| 2009-05-26 20:21 | Kloot | Note Edited: 0003492 | |
| 2009-05-26 20:23 | Kloot | Note Edited: 0003492 | |
| 2009-05-26 20:23 | tvo | Note Added: 0003493 | |
| 2009-05-26 20:24 | tvo | Note Edited: 0003493 | |
| 2009-05-26 20:25 | tvo | Note Edited: 0003493 | |
| 2009-05-26 20:26 | Kloot | Note Edited: 0003492 | |
| 2009-05-26 20:28 | tvo | Note Edited: 0003493 | |
| 2009-05-26 20:29 | Kloot | Note Edited: 0003492 | |
| 2009-05-26 20:54 | tvo | Note Added: 0003494 | |
| 2009-05-26 21:01 | jK | Note Added: 0003495 | |
| 2009-05-26 21:02 | jK | Note Edited: 0003495 | |
| 2009-05-26 21:07 | jK | Note Edited: 0003495 | |
| 2009-05-26 21:12 | Kloot | Note Added: 0003496 | |
| 2009-05-26 21:12 | Kloot | Note Edited: 0003496 | |
| 2009-05-26 21:13 | Kloot | Note Edited: 0003496 | |
| 2009-05-26 21:15 | tvo | Note Added: 0003497 | |
| 2009-05-26 21:21 | Lupus | Note Added: 0003498 | |
| 2009-05-26 23:07 | Lupus | Note Added: 0003499 | |
| 2009-05-26 23:42 | tvo | Note Added: 0003501 | |
| 2009-05-27 09:59 | tvo | Note Added: 0003503 | |
| 2009-05-27 09:59 | tvo | Status | assigned => resolved |
| 2009-05-27 09:59 | tvo | Fixed in Version | => master |
| 2009-05-27 09:59 | tvo | Resolution | open => fixed |
| Issue History |