View Issue Details

IDProjectCategoryView StatusLast Update
0006163Spring engineLuapublic2019-06-30 10:47
ReporterGoogle_Frog Assigned ToKloot  
PrioritynormalSeveritycrashReproducibilityhave not tried
Status resolvedResolutionfixed 
Product Version104.0 +git 
Fixed in Version104.0 +git 
Summary0006163: 104.0.1-1060-g7f3541f maintenance VFS seemed to misplace files inside a sdz during a game
DescriptionI received some desync reports caused by a missing file. The surprising thing is that this missing file should cause the game to desync immediately, yet it did not. katastrophe had the missing file and, luckily, sent a report: https://github.com/ZeroK-RTS/CrashReports/issues/20580

Some sort of buffering occurs during the game. It initially fails to find some files:

[f=0029249] [~CBufferedArchive][name=C:\Users\Katastrophe\Documents\My Games\Zero-K\games\c6384de84018c64982268a7a399db121.sdz] 294516711 bytes cached in 3697 files
[f=0029415] <snd_noises.lua>: Error file Sounds/reply/windmill.WAV doesn't exist.
[f=0029475] <snd_noises.lua>: Error file Sounds/reply/hovercraft_select.WAV doesn't exist.

Later this occurs:

[f=0030045] Error: [LuaRules::RunCallInTraceback] error=2 (LUA_ERRRUN) callin=UnitCreated trace=[Internal Lua error: Call failure] Include() could not load 'LuaRules/Configs/customcmds.h.lua'
stack traceback:
    [C]: in function 'sp_CallAsUnit'
    [string "LuaRules/Gadgets/unit_script.lua"]:311: in function 'CallAsUnitNoReturn'
    [string "LuaRules/Gadgets/unit_script.lua"]:737: in function 'UnitCreated'
    [string "LuaRules/gadgets.lua"]:1432: in function <[string "LuaRules/gadgets.lua"]:1427>
    (tail call): ?

The game would be unplayable if 'LuaRules/Configs/customcmds.h.lua' was missing for the entire game, so VFS seemed to lose the file during the game.

Here are reports from some other players in the game, gathered because the missing file caused desync:
 - https://github.com/ZeroK-RTS/CrashReports/issues/20572
 - https://github.com/ZeroK-RTS/CrashReports/issues/20574
 - https://github.com/ZeroK-RTS/CrashReports/issues/20575
 - https://github.com/ZeroK-RTS/CrashReports/issues/20577
 - https://github.com/ZeroK-RTS/CrashReports/issues/20587

The same thing seemed to occur to Godde in a separate game, but I don't have his report. All I have is this desync report from another player in the game: https://github.com/ZeroK-RTS/CrashReports/issues/20559
TagsNo tags attached.
Checked infolog.txt for ErrorsYes

Activities

Kloot

2019-03-13 14:14

developer   ~0019837

There is a config option (VFSCacheArchiveFiles) to turn off buffering, set that to 0 when you push the next ZK update. This may explain many past desyncs.

Google_Frog

2019-03-13 14:55

reporter   ~0019838

https://github.com/ZeroK-RTS/Chobby/commit/02a62484d172cc485273639946167f7b450dd496

VFSCacheArchiveFiles is not in https://springrts.com/wiki/Springsettings.cfg, but I found it in the sources as a springsetting, so that is how I have set it.

Google_Frog

2019-03-16 11:05

reporter   ~0019860

Here is the issue on 104.0.1-1085-gfc6d80e, but I am unsure whether VFSCacheArchiveFiles requires a restart to apply.
 * https://github.com/ZeroK-RTS/CrashReports/issues/20672
 * https://github.com/ZeroK-RTS/CrashReports/issues/20674

Kloot

2019-03-16 12:30

developer   ~0019861

It does, but 2067{2,4} seem to be duplicates of 0006157.

Google_Frog

2019-03-17 08:10

reporter   ~0019865

Here is another, but it occurred 14 hours ago so there may have been insufficient restarts for the settings update.
 * https://github.com/ZeroK-RTS/CrashReports/issues/20686 - "Just as I observed this happening, the unit commands UI panel also started rendering white."
 * https://github.com/ZeroK-RTS/CrashReports/issues/20687
 * https://github.com/ZeroK-RTS/CrashReports/issues/20688
The first one is from the player who had the VFS issue, and commented on the ticket that UI panel textures were lost as well.

Kloot

2019-03-17 21:19

developer   ~0019870

Last edited: 2019-03-17 21:19

from the ~CBufferedArchive lines in darloth's log (20686) we can infer the settings change hadn't percolated yet.

textures vanishing is a sign of memory corruption.

Google_Frog

2019-03-31 00:48

reporter   ~0019901

I have seen two recent instances of the issue, so it is possibly not resolved.

The first occurrence has someone lose customcmds, which would definitely have crashed the game earlier if it was missing from the start of the game. The affected player didn't submit a report though.
 * https://github.com/ZeroK-RTS/CrashReports/issues/21081
 * https://github.com/ZeroK-RTS/CrashReports/issues/21082
 * https://github.com/ZeroK-RTS/CrashReports/issues/21084
[f=0014943] [Game::ClientReadNet][LOGMSG] sender="Hellaciouss" string="[Internal Lua error: Call failure] Include() could not load 'LuaRules/Configs/customcmds.h.lua'
stack traceback:
    [C]: in function 'sp_CallAsUnit'
    [string "LuaRules/Gadgets/unit_script.lua"]:317: in function 'CallAsUnitNoReturn'
    [string "LuaRules/Gadgets/unit_script.lua"]:756: in function 'UnitCreated'
    [string "LuaRules/gadgets.lua"]:1398: in function <[string "LuaRules/gadgets.lua"]:1393>
    (tail call): ?"
[f=0014943] Sync error for Hellaciouss in frame 14925 (got 5ff22da1, correct is e508c608)

The second occurrence may be unrelated as the player is unable to load the model for which the error later occurs.
 * https://github.com/ZeroK-RTS/CrashReports/issues/21091 - Report for the affected player (Reapo), has VFSCacheArchiveFiles = 0.
[f=0021312] game_message: Metal Extractor is under attack
[f=0021500] Error: [LuaRules::RunCallInTraceback] error=2 (LUA_ERRRUN) callin=UnitCreated trace=[Internal Lua error: Call failure] [string "scripts/turretgauss.lua"]:4: piece not found: Concrete
stack traceback:
    [C]: in function 'sp_CallAsUnit'
    [string "LuaRules/Gadgets/unit_script.lua"]:317: in function 'CallAsUnitNoReturn'
    [string "LuaRules/Gadgets/unit_script.lua"]:756: in function 'UnitCreated'
    [string "LuaRules/gadgets.lua"]:1398: in function <[string "LuaRules/gadgets.lua"]:1393>
    (tail call): ?
[f=0021511] [Game::ClientReadNet][LOGMSG] sender="Reapo" string="[Internal Lua error: Call failure] [string "scripts/turretgauss.lua"]:4: piece not found: Concrete
stack traceback:
    [C]: in function 'sp_CallAsUnit'
    [string "LuaRules/Gadgets/unit_script.lua"]:317: in function 'CallAsUnitNoReturn'
    [string "LuaRules/Gadgets/unit_script.lua"]:756: in function 'UnitCreated'
    [string "LuaRules/gadgets.lua"]:1398: in function <[string "LuaRules/gadgets.lua"]:1393>
    (tail call): ?"
[f=0021521] Sync error for Reapo in frame 21500 (got d84785b3, correct is 9c68bbdc)

Kloot

2019-03-31 03:12

developer   ~0019904

The customcmds.h.lua error always originates from unit_script which performs some include caching trickery not typically seen in other gadgets, unlikely to be a coincidence. Knowing under what exact conditions it triggers would help.

Kloot

2019-04-14 12:20

developer   ~0019939

I need debug output from 1209-gcef61f0 or later.

Kloot

2019-04-27 10:43

developer   ~0019962

No further response and a version rollback, UTR.

Google_Frog

2019-04-27 11:28

reporter   ~0019963

There will be a response once we manage to update. When I last tried it became apparent that the ZK infrastructure would update the engine for some players, but not the default game version. I found out because the changes to GetTeamInfo broke compatibility, resulting in too many fatal crashes for my liking.

Kloot

2019-06-30 10:47

developer   ~0020013

This is either fixed or traceable to the root cause in 1273-ge1c1143, updating would be appreciated.

Issue History

Date Modified Username Field Change
2019-03-13 05:31 Google_Frog New Issue
2019-03-13 14:14 Kloot Note Added: 0019837
2019-03-13 14:14 Kloot Status new => feedback
2019-03-13 14:55 Google_Frog Note Added: 0019838
2019-03-13 14:55 Google_Frog Status feedback => new
2019-03-16 11:05 Google_Frog Note Added: 0019860
2019-03-16 12:30 Kloot Note Added: 0019861
2019-03-17 08:10 Google_Frog Note Added: 0019865
2019-03-17 21:19 Kloot Note Added: 0019870
2019-03-17 21:19 Kloot Note Edited: 0019870
2019-03-31 00:48 Google_Frog Note Added: 0019901
2019-03-31 03:12 Kloot Note Added: 0019904
2019-04-14 12:20 Kloot Note Added: 0019939
2019-04-14 12:20 Kloot Status new => feedback
2019-04-27 10:43 Kloot Status feedback => closed
2019-04-27 10:43 Kloot Resolution open => unable to reproduce
2019-04-27 10:43 Kloot Note Added: 0019962
2019-04-27 11:28 Google_Frog Status closed => feedback
2019-04-27 11:28 Google_Frog Resolution unable to reproduce => reopened
2019-04-27 11:28 Google_Frog Note Added: 0019963
2019-06-30 10:47 Kloot Assigned To => Kloot
2019-06-30 10:47 Kloot Status feedback => resolved
2019-06-30 10:47 Kloot Resolution reopened => fixed
2019-06-30 10:47 Kloot Fixed in Version => 104.0 +git
2019-06-30 10:47 Kloot Note Added: 0020013