Allow reloading archives while developing

Allow reloading archives while developing

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Post Reply
gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Allow reloading archives while developing

Post by gajop » 24 Oct 2018, 01:35

I think it would be nice to allow reloading archives while developing, as right now we need to restart each time we add or remove a file.
I would like to provide a way of reloading the entire archive, and collecting the list of files from disk at the time of the reload.

Couple of ways to do it:
1) reload the entire archive each time /luaui reload or /luarules reload is invoked, and Lua devMode is true.
Pros: Pretty intuitive and doesn't require active thinking.
Cons: Might be expensive to go to disk each time we reload. Wouldn't help if we just toggle on/off a single addon that tries to load a newly created file.

2) Provide a new /reloadarchives or just /reload command that can be used to manually reload all loaded archives. Doable only when Lua devMode is true. It could also do /luaui reload, /luarules reload and so on, as necessary
Pros: Doesn't make /lua* reload commands any more expensive, and it could even be automated with external fs monitoring tools, like springmon
Cons: Requires devs to know about this other command, and use it when necessary.

Right now I'm mostly interested in addon files, but I think it would be cool to allow reloading defs, map files, and artwork, with some constraints.

I think this is very important as restarting and setting up the same situation can be very time consuming when developing new games (LD). Also it's probably confusing to new devs.
PS: I'd like to work on this, just got a bit of a cold now so I figured I'd just write this out first.

Thoughts?
2 x

User avatar
Silentwings
Moderator
Posts: 3584
Joined: 25 Oct 2008, 00:23

Re: Allow reloading archives while developing

Post by Silentwings » 24 Oct 2018, 08:07

1) reload the entire archive each time /luaui reload or /luarules reload is invoked, and Lua devMode is true.
Pros: Pretty intuitive and doesn't require active thinking.
I don't think this is intuitive - a reload xyz should only reload xyz. Also, it could make it very difficult to test widgets that had luarules interaction, or reacted to specific sim/luarules states (e.g. when the luarules state communicated to the widget typically depends on luarules having saved info from events in the past).
Provide a new /reloadarchives
If it's possible, as a clean sweep, sounds great.

Imo a /reloaddefs command, which was even possible in the past, and these days would also have to re-run _post, would be useful too. I don't think reloadmap is needed.
1 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14585
Joined: 17 Nov 2005, 02:43

Re: Allow reloading archives while developing

Post by Forboding Angel » 24 Oct 2018, 19:33

Well, spring already trips over large .git histories in .sdd. The idea of forcing a reload worries me a bit. However, it could be easily argued that reloading content would not be the same as spring doing it's cache thing when spring.exe is run.

Honestly, I would much rather that the cache issue be sorted out. Can't spring be made to just completely ignore .directories inside game .sdd s? I hesitate to call them "hidden" directories, because lolwindows.

If spring didn't take 5 minutes to load on utterly random occasions, I would be so much happier. Granted, the long loading times generally happens directly after a push/pull (this is why I blame spring reading .git). If this wasn't an issue, then I would be all for reloading content via a command.

Edit: Holy shit... Is spring recreating cache because of crlf->lf (because windows and git) line endings?
Last edited by Forboding Angel on 25 Oct 2018, 13:02, edited 1 time in total.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Allow reloading archives while developing

Post by gajop » 25 Oct 2018, 00:45

Silentwings wrote:
24 Oct 2018, 08:07
1) reload the entire archive each time /luaui reload or /luarules reload is invoked, and Lua devMode is true.
Pros: Pretty intuitive and doesn't require active thinking.
I don't think this is intuitive - a reload xyz should only reload xyz. Also, it could make it very difficult to test widgets that had luarules interaction, or reacted to specific sim/luarules states (e.g. when the luarules state communicated to the widget typically depends on luarules having saved info from events in the past).
To put it another way, would including new files be appropriate when reloading files used by LuaUI/LuaRules with /luaui reload and /luarules reload?

It is kinda hard to define what those files are, as they might appropriately be put outside of the LuaUI/LuaRules dirs (often the case for config files, libs, etc.), so effectively updating the entire archive file list might be necessary
Silentwings wrote:
24 Oct 2018, 08:07
Provide a new /reloadarchives
If it's possible, as a clean sweep, sounds great.

Imo a /reloaddefs command, which was even possible in the past, and these days would also have to re-run _post, would be useful too. I don't think reloadmap is needed.
I am starting to think /reload might be the best choice.
Is there a need to have a separate /reload* for all different types of things? (Lua, defs, etc.) Might as well just reload everything that has changed, as we have to go to disk and retrieve files.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Allow reloading archives while developing

Post by gajop » 25 Oct 2018, 00:48

Forboding Angel wrote:
24 Oct 2018, 19:33
Well, spring already trips over large .git histories in .sdd.
I've seen you mention this before and I'm going to investigate what the issue might be. We really shouldn't be loading that by default, and you already claim to have it in your springignore.txt
0 x

Kloot
Spring Developer
Posts: 1865
Joined: 08 Oct 2006, 16:58

Re: Allow reloading archives while developing

Post by Kloot » 25 Oct 2018, 02:18

This comment should clue you in why using .sdd's as repository roots is a bad idea:

Code: Select all

    // If we are here, we could have invalid info in the cache
    // Force a reread if it is a directory archive (.sdd), as
    // st_mtime only reflects changes to the directory itself,
    // not the contents.

OT: implementing separate /reload*'s (/reloadscripts, /reloadsounds, etc) would be manageable and precedented, but a monolithic command or reloading entire archives severely complicates things.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Allow reloading archives while developing

Post by gajop » 25 Oct 2018, 02:51

Kloot wrote:
25 Oct 2018, 02:18
This comment should clue you in why using .sdd's as repository roots is a bad idea:
Are you talking about development or playing? Since I don't know of an alternative to .sdd for development.
Kloot wrote:
25 Oct 2018, 02:18
OT: implementing separate /reload*'s (/reloadscripts, /reloadsounds, etc) is manageable and precedented, but a monolithic command or reloading entire archives will severely complicate things.
Hmm I suppose you're right, a full reload might be better done with Spring.Reload, but I do think we need to refresh the archive file list so we can load new files through VFS. That should be done in one command.
It might be a bit of a PITA if that's all the command did, as users would have to write additional /*reload* commands after that. That's why I suggested that maybe /luaui reload should also reload the archive (maybe just update the file list?)

Think we're nearing a stage where it would be good to start with implementations and see how it behaves
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14585
Joined: 17 Nov 2005, 02:43

Re: Allow reloading archives while developing

Post by Forboding Angel » 25 Oct 2018, 11:53

Kloot wrote:
25 Oct 2018, 02:18
This comment should clue you in why using .sdd's as repository roots is a bad idea
IDD, Unfortunately, it must be this way in order to utilize rapid, or so I was instructed back when I was setting up git for use with rapid. So, Unless I'm mistaken, anyone who wants to utilize rapid simply doesn't really have a choice. Much in the same way that I actually wanted a release branch and to have rapid use that. However, afaik, the way rapid is set up can only look in the master branch.

It would appear that our hands are tied.
0 x

User avatar
Silentwings
Moderator
Posts: 3584
Joined: 25 Oct 2008, 00:23

Re: Allow reloading archives while developing

Post by Silentwings » 25 Oct 2018, 11:55

I thought the .springignore file was meant to fix these issues? I wonder if maybe there is some confusion here about what was meant.
1 x

Kloot
Spring Developer
Posts: 1865
Joined: 08 Oct 2006, 16:58

Re: Allow reloading archives while developing

Post by Kloot » 25 Oct 2018, 12:13

Archive entries listed in springignore are only exempt from checksum calculations (obviously unwanted on .git and other sync-breaking data), but the main issue with sdd-as-repo usage is that when you update any file inside an .sdd the OS has to throw out part of its disk cache, and since VCS'es like git tend to touch many files at once just for bookkeeping purposes you can get massive loading time spikes seemingly at random when the engine determines (by examining the root st_mtime stamp supplied by the OS) an archive to have been modified and is forced to rescan all of it. This will keep happening even if your springignore.txt contains a '*'.
gajop wrote:I do think we need to refresh the archive file list so we can load new files through VFS
Quite literally no part of Spring is designed for archive contents to be added (or deleted) at game run-time, and even plain changes to a majority of file types (say models) won't be picked up easily. These are the proverbial good intentions paving the road to hell.
2 x

Post Reply

Return to “Engine”