| View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||
| ID | Project | Category | View Status | Date Submitted | Last Update | ||||
|---|---|---|---|---|---|---|---|---|---|
| 0004459 | Spring engine | General | public | 2014-06-27 15:41 | 2014-06-28 21:20 | ||||
| Reporter | msafwan | ||||||||
| Assigned To | abma | ||||||||
| Priority | low | Severity | major | Reproducibility | always | ||||
| Status | resolved | Resolution | fixed | ||||||
| Product Version | 97.0.1+git | ||||||||
| Target Version | Fixed in Version | ||||||||
| Summary | 0004459: ArchiveScanner search (*.smf, *.sm3) file in Mods cost alot of time | ||||||||
| Description | I always have to wait several minutes to start Spring first time. I tried adding "clock()" function in ArchiveScanner.cpp and found: https://github.com/spring/spring/blob/develop/rts/System/FileSystem/ArchiveScanner.cpp#L544 use most of the time. It want to find *.smf and *.sm3 but all Mods don't have this file and Mods have thousand of file. | ||||||||
| Steps To Reproduce | To reproduce: -First delete archive cache file. (Spring/cache/ArchiveCache.lua) -Then make sure you have thousand of Mods (Licho's server have more!) -Then Start any Spring version (old or new is same). | ||||||||
| Additional Information | Here is some of echo: Warning: SCAN ARCHIVE TIME (STAT): 0 Warning: SCAN ARCHIVE TIME (CONTENT): 41 Warning: SCAN ARCHIVE TIME (CONTENT FILECOUNT): 7062 Warning: SCAN ARCHIVE TIME: 98 Warning: SCAN ARCHIVE TIME (STAT): 0 Warning: SCAN ARCHIVE TIME (CONTENT): 40 Warning: SCAN ARCHIVE TIME (CONTENT FILECOUNT): 7037 Warning: SCAN ARCHIVE TIME: 104 Warning: SCAN ARCHIVE TIME (STAT): 0 Warning: SCAN ARCHIVE TIME (CONTENT): 42 Warning: SCAN ARCHIVE TIME (CONTENT FILECOUNT): 7068 Warning: SCAN ARCHIVE TIME: 245 I haven't put clock() in all places yet, if I found more I will report. (this is really important, now Licho can't turn on Autoregistrator.exe anymore because it took too much time) | ||||||||
| Tags | efficiency, first time load, unitsync | ||||||||
| Checked infolog.txt for Errors | |||||||||
| Attached Files |
| ||||||||
 Relationships |
|
 Relationships |
| Notes |
|
|
msafwan (reporter) 2014-06-27 16:13 Last edited: 2014-06-27 18:02 |
I added extra echo, and the file that have 7000file is this: FILECOUNT: 7062 NAME: C:\Users\User\Documents\My Games\Spring\packages\ea2115c01ba050d82b726d825c9dca7a.sdp FILECOUNT: 7003 NAME: C:\Users\User\Documents\My Games\Spring\packages\e9f3057e28b02d784ec362afee835d0b.sdp This files was downloaded by Rapid. It just around ~200KB. Its an incremental-changes/patch over previous downloaded Mods (I'm not sure how it contain 7000 files). This might be ZK-test version but Stable version also download this file. Current filecount for ZK is 7487 files and ~200MB for zipped and ~500MB for unzipped (http://packages.springrts.com/builds/ ). Each loop to find *.smf & *.sm3 cost around ~40ms x 2000mods. It should be more efficient because all this Mods don't have map file (I think). |
|
Jools (reporter) 2014-06-27 19:42 |
Well, the obvious resolution is to delete some mods until you have, say, tens of mods instead of thousands :) |
|
abma (administrator) 2014-06-28 05:28 |
"Autoregistrator" should use a dedicated folder which contains only the file, which has to be scanned. You could also use (or improve if needed) upq which is used on springfiles. .Sdp contain the full game! |
|
msafwan (reporter) 2014-06-28 13:11 Last edited: 2014-06-28 13:25 |
That loop in ArchiveScanner.cpp loop over thousand of file and do nothing with its information, especially when the Archives contain no maps. For sake of efficiency, someone should design a better escape condition for this loop. IMO modder should include a file that notify ArchiveScanner.cpp that this Mod contain a map. It reduce CPU consumption by 70%. IMO other part of ArchiveScanner.cpp might also be more efficient if there's a better standard that reduce file searching. Some 5 minute scan can be reduced to 20 second, such is what happen to ZKL's (local)Autoregistrator ( https://github.com/ZeroK-RTS/Zero-K-Infrastructure/commit/4478d61b485ebb17d3977dcf969ce8ae3494452a )(which also did too much loop which information then discarded after finished). 300 second to 20 second run-time is phenomenal optimization. P/s: I can probably delete all my mods now, but if someone fix this code now it will more awesome in the future! |
|
abma (administrator) 2014-06-28 18:32 Last edited: 2014-06-28 18:37 |
> if there's a better standard that reduce file searching. thats the problem. enforcing this (mapinfo.lua / modinfo.lua) would break A LOT of maps/games. many maps don't contain mapinfo.lua afaik ?! also full-scan is done to check if archive is solid or not. solid archives slows stuff down. |
|
abma (administrator) 2014-06-28 20:24 Last edited: 2014-06-28 20:36 |
i've tried to speed it up, but when doing this i found this case: when a file is fully scanned, any error reading a file will be detected! if its not fully scanned, you can start games with broken files which can cause desyncs/errors/... bad things! with this check broken maps/games aren't listed. imo safety first! (so no change required...) |
|
abma (administrator) 2014-06-28 21:13 |
note: > if its not fully scanned, you can start games with broken file thats not fully true, in most cases files weren't full checked / errors ignored so give this speedup a try. |
|
abma (administrator) 2014-06-28 21:20 |
@msafwan: just for the curious: does my change speed it up? i didn't measure it. but now it doesn't fully scan rapid + sdz any more. |
| Notes |
| Issue History |
|||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2014-06-27 15:41 | msafwan | New Issue | |
| 2014-06-27 15:42 | msafwan | Tag Attached: efficiency | |
| 2014-06-27 15:43 | msafwan | Tag Attached: unitsync | |
| 2014-06-27 15:43 | msafwan | Tag Attached: first time load | |
| 2014-06-27 16:13 | msafwan | Note Added: 0013370 | |
| 2014-06-27 16:17 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 16:23 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 16:24 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 16:24 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 16:25 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 16:26 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 17:06 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 17:07 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 18:01 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 18:01 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 18:02 | msafwan | Note Edited: 0013370 | View Revisions |
| 2014-06-27 19:42 | Jools | Note Added: 0013371 | |
| 2014-06-28 05:28 | abma | Note Added: 0013372 | |
| 2014-06-28 13:11 | msafwan | Note Added: 0013373 | |
| 2014-06-28 13:15 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:17 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:18 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:19 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:21 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:24 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 13:25 | msafwan | Note Edited: 0013373 | View Revisions |
| 2014-06-28 18:32 | abma | Note Added: 0013374 | |
| 2014-06-28 18:33 | abma | Note Edited: 0013374 | View Revisions |
| 2014-06-28 18:34 | abma | Note Edited: 0013374 | View Revisions |
| 2014-06-28 18:37 | abma | Note Edited: 0013374 | View Revisions |
| 2014-06-28 20:24 | abma | Note Added: 0013375 | |
| 2014-06-28 20:36 | abma | Note Edited: 0013375 | View Revisions |
| 2014-06-28 21:12 | abma | Changeset attached | => spring develop a97b0e11 |
| 2014-06-28 21:12 | abma | Assigned To | => abma |
| 2014-06-28 21:12 | abma | Status | new => resolved |
| 2014-06-28 21:12 | abma | Resolution | open => fixed |
| 2014-06-28 21:13 | abma | Note Added: 0013376 | |
| 2014-06-28 21:20 | abma | Note Added: 0013377 | |
| Issue History |