2020-04-05 08:17 CEST

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0006340Spring engineGeneralpublic2020-03-28 23:58
ReporterKrogoth 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusfeedbackResolutionopen 
Product Version104.0 +git 
Target Version105.0Fixed in Version 
Summary0006340: Significant reduction in FPS with scroll/zoom on maps with rough terrain
DescriptionSuspecting some changes in ROAM, that appeared shortly after version 1250 (not sure which version exactly).
While on powerful PC it might not be very noticeable, on a laptop it renders game unbearable.
Steps To ReproduceSelect a map with significant roughness, e.g. Stronghold.
Scroll/zoom.
Additional InformationSame issue was addressed by Zero-K players.
TagsNo tags attached.
Checked infolog.txt for ErrorsIrrelevant
Attached Files

-Relationships
has duplicate 0006348closed There's camera stuttering on larger maps 
+Relationships

-Notes

~0020220

Krogoth (reporter)

To avoid misunderstanding:
v 1250 works fine, no lag.

~0020221

Anarchid (reporter)

There was some investigation WRT this around ZK.

This is quite definitely ROAM.

Moving the camera while viewing the cliffy borders on the map Onyx Cauldron with the ROAM mesh drawer ON results in a massive FPS drop.

Panning with the legacy mesh drawer results in a very slightly lower FPS when camera is static, but completely smooth panning - an overall massive increase in quality.

The ZK investigation has not resulted in an engine issue because there's a user-side workaround in the form of the "/mapmeshdrawer" command. If the issues are truly unbearable, a game may include a widget to automatically switch to the legacy setting when on a map that is known to perform badly with ROAM.

~0020222

Krogoth (reporter)

Hmm...
From my experience, all maps perform badly with new ROAM.
Switching to legacy could be a last resort, but I really like how ROAM looks, more than legacy.
I didn't investigate the difference between new and old, but the old one worked well, correct?

~0020223

Krogoth (reporter)

*between new and old ENGINE code

~0020224

Anarchid (reporter)

I don't know if 1250 works correct.

These statements are making me confused:

> v 1250 works fine, no lag.
> I didn't investigate the difference between new and old, but the old one worked well, correct?

~0020225

Krogoth (reporter)

it does.
meant difference in code.
bug maybe anywhere, theoretically not in ROAM code, so I can't 100% say that ROAM code is bad.

~0020226

Kloot (developer)

I got tired of seeing 0006220 pop up over and over again and disabled threaded mesh tessellation in ROAM to get rid of it.

Anyone here is free to volunteer to fix the actual bug and restore performance.

~0020227

Krogoth (reporter)

Ill try

~0020240

Krogoth (reporter)

While trying to reproduce the crash..
How normal is "cannot recreate persistent storage buffer", VBO error?

~0020241

Krogoth (reporter)

nvm

~0020251

Krogoth (reporter)

Found it!
------
|X|X|
------
Look at the middle diamond (right quarter of left patch, left quarter of right patch)
If left triangle of it is getting split and when it tries to split base neighbour and goes out of nodes, it leaves childs hanging with unset neighbours to the right side.
Then, if in the next iteration right triangle (of the center diamond) goes out of nodes right away, it doesn't connect hanging pointers.
Finally, if tesselation from bottom or top causes left triangle to split (via multiple splits such that ultimately cause Patch:line 240 executed), then splitting left triangle's child nodes will use unset neighbours,
thus access violation.
Obviously, it doesn't happen like this if single thread.

~0020252

Krogoth (reporter)

Principally: the problem is leaving pointers hanging between splits of triangle and its base neighbour.

~0020253

IceXuick (reporter)

Great!

~0020254

Kloot (developer)

thanks for the legwork, I'll see what I can do soon.

~0020263

IceXuick (reporter)

Any news on this bug and or possible fix?

~0020264

Krogoth (reporter)

Just in case if Kloot was busy and to ease things up for him,
position in code:
https://github.com/spring/spring/blob/a17a11a58b06da8bc8510d1eb3cd92b559ea694c/rts/Map/SMF/ROAM/Patch.cpp#L293
Setting tlc->RightNeighbour and trc->RightNeighbour only when branch, thus leaving them hanging if both branch this and next iter go out of nodes.
One of solutions could be eliminating childs of empty base neighbour (in addition to own), but there can be many solutions.

~0020265

Krogoth (reporter)

*trc->LEftNeighbour

~0020266

Krogoth (reporter)

*not empty base neighbour, but one with not valid childs
Where is My EDIT Button? :X

~0020272

IceXuick (reporter)

Latest 1455 release made this problem even worse...

~0020306

Google_Frog (reporter)

I don't know what caused it, but there is a performance regression between 104.0.1-1435-g79d77ca and 104.0.1-1455-g39f8fbd. My benchmark shows a difference of 19% between the average update time. The benchmark does not move the camera, however there is some terrain deformation. I'll bisect.

~0020310

IceXuick (reporter)

Awesome! I hope you can find it.
I'm currently working on a new map and i'm already with-holding myself from adding too much detail to increase the chance it doesn't stutter.

which.. could be both good and bad :)
(good = more optimized cleaner maps / bad = i'm constrained in use of creative/detailed maps)

~0020311

IceXuick (reporter)

I've tried to pin down what might be the exact cause in my TMA20 map. I've been reducing (micro)details, smoothened rough areas, removed almost 90% of rocky/bumpy area's and rounded hard cliff edges.

I've done this in 7 steps, everytime removing more and more.
Despite it having effect, even after the 7th lowering of details, the stuttering is still not gone, though like 75%-ish less.

I would like to put this as a good test case, where you can see/test both maps (high quality one vs very tuned down version) - but both still have the stuttering. I think it's a great loss that this stuttering happens, because it severely limits mappers.

You can only get decent framerates/no stutter if you have very simple maps. When you want some more organic stuff, and more microdetails (which greatly enhance the more closeup battles - which i think Spring definately could use) then you quickly go into stutter-fest-mode.

Currently rounding up my 7th revision into a separate map that i'll upload as an 'optimized' version. But i hope you can really look into this, because this is just not something you'd want.

~0020312

IceXuick (reporter)

PS. you can test the 'optimized' TMA20XR 2.1 here: https://springfiles.com/spring/spring-maps/tma-20-xr

And also my map DSDR (Delta Siege Dry Remake) does have some stutter issues as well. Is it possible that ROAM has problems with erosion (thinner deeper lines in the surface)?

Download lastest DSDR 3.5 here: https://springfiles.com/spring/spring-maps/dsdr

I'll try a test where i blur/smooth out only these erosion-lines and see what happens...

~0020313

IceXuick (reporter)

Update: Smoothing out the erosion-lines and overall smoothing edges did improve fps/stutter issues, but this varies between systems, and still is noticeable, especially on lower-end systems.

So (if the above from Krogoth is correct) ROAM has some issues.

There's another issue about cratereffect (the dynamic cratering/terraindeforming) that results in large decreases of fps/smoothness, especially noticeable in fast succession like mortar volleys, missile barrages or D-guns.

Might these perf-issues also be related to ROAM not performing optimal? I can remember slightly that these deforms worked pretty well and fast/smooth in older Spring RTS.

I hope someone of you can dive into this and try to fix this, because i think it's a pretty fundamental (and unique) game-aspect/feature that should work as good as possible.

~0020318

IceXuick (reporter)

I can confirm v 1250 to be practically flawless and stutter free.
All maps mentioned above work very well with the 1250 release in both w32 and w64 versions.

@Kloot i heard you are the one who can most probably fix this? Could you please look into this?

~0020319

Kloot (developer)

Last edited: 2020-02-03 11:50

View 2 revisions

I don't actively dev Spring much anymore, but try the following:

  download the last available engine build (1463-gb19d6f5)
  enter "/maptesselusethreads 0 0" into the console; compare performance against both 1250 and 1435-g79d77ca
  enter "/maptesselusethreads 1 1" into the console and repeat

~0020320

IceXuick (reporter)

in latest 1463 and 1250 these things don't make any difference.
Is that possible?

~0020321

Kloot (developer)

Last edited: 2020-02-03 17:40

View 2 revisions

only if you included the ""'s when entering /maptesselusethreads 0 0; there should be a very detectable change in CPU usage afterward.

fyi this command does not exist in 1250, I added it to 1455+ when fixing 0006220.

~0020323

IceXuick (reporter)

I'll ask Floris. I've tried every combination wiht/without "" and with 0, 0 0, with ;, with cheats, but i don't see any difference/thing happen.

~0020324

Floris (reporter)

i didnt notice anythign either,
but i made the game now also set /mapmeshdrawer 1 for best performance in the meantime

~0020326

IceXuick (reporter)

Downside is that without cam-movement - fps is like 40% lower with meshdrawer 1 (basic mesh) set on quality bias 5 (comparable with grounddetail of 130 with meshdrawer 2 = ROAM)

~0020327

IceXuick (reporter)

And this only is an issue with larger maps, or maps that have more height variation - so setting globally to meshdrawer 1 would affect everyone that plays other or smaller maps too much.

~0020328

IceXuick (reporter)

@Kloot - did you read Korogths notes? He said he found the problem, probably with the multithreading. Can't you use this for a solution? Or perhaps tell us how to exactly get this fixed in BAR?

~0020329

Floris (reporter)

btw, even when using the lowest grounddetail of 4 on roam, the fps drop when moving camera is nearly the same as with highest detail

~0020330

IceXuick (reporter)

yeah - but in 1250 this is no issue at all - so something went wrong/broke between then and now.

~0020333

Krogoth (reporter)

all is known, just needs someone's time investment to write actual fix
i am busy myself for now
as for maptesselfhleawufhie - no visual difference, confirm

~0020334

Krogoth (reporter)

*noticeable at least

~0020335

IceXuick (reporter)

The speeds on 1250 are pretty insanely good compared to 1453-55-63... @Krogoth do you think you could bring this back with this fix?

~0020337

Kloot (developer)

Last edited: 2020-02-04 22:14

View 3 revisions

1) 0006220 should be fixed in 1455

2) the regression between 104.0.1-1435-g79d77ca and 104.0.1-1455-g39f8fbd (0006362) should be fixed in 1463

3) "even when using the lowest grounddetail of 4 on roam, the fps drop when moving camera is nearly the same as with highest detail" and "As for the ROAM/Basic map mesh discussion - both are severely impacted by this. So as well for ROAM as for the other Mapmesh techniques this decrease happens." (from 0006362) imply there is a separate issue unrelated to ROAM which still exists in 1463

4) I have no time to investigate this soon

~0020338

IceXuick (reporter)

Hi @Kloot,

Thanks for your time and investigations & fixes.

I hope you can find time somewhere because i think the map-mesh situation isn't good.

~0020346

Google_Frog (reporter)

Here is a video comparing 1463 and 1453: https://www.youtube.com/watch?v=pkOmFqXgdkE

maptesselusethreads does not appear to have any effect, and 1453 is noticeably worse than 1463 when moving the camera.

~0020348

Google_Frog (reporter)

104.0.1-1454-g3c0e4e1 has good performance and has bad performance 104.0.1-1455-g39f8fbd.

~0020349

Google_Frog (reporter)

Nevermind, I was mistaken. I ran some benchmarks on various engine versions. This benchmark zooms the map in and out rapidly: https://www.youtube.com/watch?v=lNhcRp13lto

~0020350

Google_Frog (reporter)

I compared the interpretation of the terrain detail setting with the changes. Both versions appear to result in the same level of detail, so this is not a cause of the issue.

~0020351

Google_Frog (reporter)

I made PRs https://github.com/spring/spring/pull/494 https://github.com/spring/spring/pull/495

~0020367

abma (administrator)

@Google_Frog:

1. do you agree that https://github.com/spring/spring/pull/504 makes https://github.com/spring/spring/pull/495 obsolete? (for me it looked so...)


2. if so: can the same change applied to maintenance?

(also, sorry, i should have squashed the pull request to a single commit)

~0020371

Google_Frog (reporter)

They are probably obsolete but I won't be able to do the proper test until the same change is allied to maintenance.

~0020372

Google_Frog (reporter)

*applied. And obsolescence is looking very likely from what I have seen on Discord.

~0020373

Google_Frog (reporter)

I've run the benchmark. https://github.com/spring/spring/commit/69b06d03411fd97f0042a348d839e88d16bd6cb1 is an improvement over the worst of this bug, but it is still worse than 1435 and https://github.com/spring/spring/pull/497. The "1474 maptes00" run is 1474 with the addition of Spring.SendCommands("maptesselusethreads 0 0").

~0020374

Google_Frog (reporter)

I have closed https://github.com/spring/spring/pull/497 because it conflicts with the changes.

https://github.com/spring/spring/pull/494 and https://github.com/spring/spring/pull/495 do not conflict and provide the fastest version of ROAM. Beherith has shown interest in modifying current ROAM so I think the best approach is to pull the single thread ROAM PRs. This makes it much easier to compare future work (including ivand's potential tessellation attempts) to a known performant version.

Even now there is the possibility that a performance regression in a different part of Spring snuck in between 1463 and 1474l. This seems unlikely from the commit log, but holding off on this will only make comparing performance harder.
+Notes

-Issue History
Date Modified Username Field Change
2019-11-15 15:09 Krogoth New Issue
2019-11-15 15:13 Krogoth Note Added: 0020220
2019-11-15 15:17 Anarchid Note Added: 0020221
2019-11-15 16:04 Krogoth Note Added: 0020222
2019-11-15 16:05 Krogoth Note Added: 0020223
2019-11-15 16:09 Anarchid Note Added: 0020224
2019-11-15 16:12 Krogoth Note Added: 0020225
2019-11-15 16:36 Kloot Note Added: 0020226
2019-11-15 16:51 Krogoth Note Added: 0020227
2019-11-18 11:04 Krogoth Note Added: 0020240
2019-11-18 11:15 Krogoth Note Added: 0020241
2019-11-20 08:48 Kloot Relationship added has duplicate 0006348
2019-11-20 15:19 Krogoth Note Added: 0020251
2019-11-20 15:20 Krogoth Note Added: 0020252
2019-11-20 17:32 IceXuick Note Added: 0020253
2019-11-20 22:51 Kloot Note Added: 0020254
2019-11-28 15:23 IceXuick Note Added: 0020263
2019-11-29 06:29 Krogoth Note Added: 0020264
2019-11-29 06:30 Krogoth Note Added: 0020265
2019-11-29 06:37 Krogoth Note Added: 0020266
2019-12-03 22:54 IceXuick Note Added: 0020272
2019-12-30 14:48 abma Target Version => 105.0
2019-12-30 14:48 abma Checked infolog.txt for Errors Irrelevant => Irrelevant
2020-01-21 03:45 Google_Frog File Added: 1435_1455.zip
2020-01-21 03:45 Google_Frog Note Added: 0020306
2020-01-21 10:43 IceXuick Note Added: 0020310
2020-01-27 10:30 IceXuick Note Added: 0020311
2020-01-27 23:20 IceXuick Note Added: 0020312
2020-01-28 16:07 IceXuick Note Added: 0020313
2020-02-03 08:02 IceXuick Note Added: 0020318
2020-02-03 11:50 Kloot Note Added: 0020319
2020-02-03 11:50 Kloot Note Edited: 0020319 View Revisions
2020-02-03 15:59 IceXuick Note Added: 0020320
2020-02-03 17:19 Kloot Note Added: 0020321
2020-02-03 17:40 Kloot Note Edited: 0020321 View Revisions
2020-02-04 10:43 IceXuick Note Added: 0020323
2020-02-04 13:42 Floris Note Added: 0020324
2020-02-04 14:13 IceXuick Note Added: 0020326
2020-02-04 14:14 IceXuick Note Added: 0020327
2020-02-04 14:16 IceXuick Note Added: 0020328
2020-02-04 14:36 Floris Note Added: 0020329
2020-02-04 14:48 IceXuick Note Added: 0020330
2020-02-04 20:04 Krogoth Note Added: 0020333
2020-02-04 20:05 Krogoth Note Added: 0020334
2020-02-04 21:40 IceXuick Note Added: 0020335
2020-02-04 22:09 Kloot Note Added: 0020337
2020-02-04 22:12 Kloot Note Edited: 0020337 View Revisions
2020-02-04 22:14 Kloot Note Edited: 0020337 View Revisions
2020-02-05 01:06 IceXuick Note Added: 0020338
2020-02-11 12:44 Google_Frog Note Added: 0020346
2020-02-17 02:58 Google_Frog Note Added: 0020348
2020-02-17 05:23 Google_Frog File Added: 20200217_033223_dataFile.csv
2020-02-17 05:23 Google_Frog Note Added: 0020349
2020-02-17 05:33 Google_Frog File Added: terrainDetail.jpg
2020-02-17 05:33 Google_Frog Note Added: 0020350
2020-02-18 12:48 Google_Frog Note Added: 0020351
2020-03-26 22:28 abma Note Added: 0020367
2020-03-26 22:29 abma Status new => feedback
2020-03-27 03:12 Google_Frog Note Added: 0020371
2020-03-27 07:26 Google_Frog Note Added: 0020372
2020-03-28 23:41 Google_Frog File Added: 20200328_123752_dataFile.csv
2020-03-28 23:41 Google_Frog Note Added: 0020373
2020-03-28 23:58 Google_Frog Note Added: 0020374
+Issue History