Dev Meeting Minutes (06-12-2010) - Page 2

Dev Meeting Minutes (06-12-2010)

Minutes of the meetings between Spring developers are archived here.
Satirik
Lobby Developer
Posts: 1688
Joined: 16 Mar 2007, 18:27

Re: Dev Meeting Minutes (06-12-2010)

Post by Satirik »

can't delete my own posts :roll:
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

i have the source ready now. it checks the cost for extracting a meta file, and if it is too high (easily customizable function), it marks the archive as broken. As i have it now:

Code: Select all

bool costTooHigh = (to-be-unpacked-data-size > 32KB) && (to-be-unpacked-data-size > file-size);
which files are meta files?
modinfo.lua and mapinfo.lua are a sure thing, but are there other files that should be easily available without having to decompress the whole archive? Maybe *.lua, *.smf or *.sm3

In all my mods & maps, there are currently two ModInfo.lua files that have a too high cost, and 74 *.smf files in archives that would be marked broken. Checking for modinfo.lua, mapinfo.lua, *.smf and *.sm3.
(i guess the smf do not make sense, do they? what stuff is required to generate a minimap?)
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Dev Meeting Minutes (06-12-2010)

Post by zwzsg »

If the archive is entirely non-solid, am I allowed to have a meta file over 32kb? Or does this size limit apply regardless?

As I already had trouble with the 32k limit on startscript, I am worried that one day I'll make an epicly complicated configurable scripted mega map project and weep when it gets rejected by Spring.

Why do you count .smf as a meta file, and not .smd? I would have thought that if there was only one important meta info file to let accessible, it would be the .smd!

Smoth maps use a rather complex set of meta configuration file, might want to check with him or his maps (from 1.26release.zip).
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

the only criteria for a file to be meta in this context, is that this file might be of interest separately/in combination with other meta-files, but independent of non-meta files in the same archive.
And yes, as long as the meta file is in a separate/isolated packing block ("folder" in 7zip talk), which is the case for all files in completely non solid blocks, or when using the compression method shown by ipavlov, it does not matter how large it is; the archive will not be marked as broken.

I did not know.. but looking at the source, it seems like the SMF is used to create the mini-map, therefore it could make sense to have it as meta file. if the SMF usually makes up 90% of the archive content anyway, it is not so clear anymore though.
Last edited by hoijui on 10 Dec 2010, 14:39, edited 1 time in total.
User avatar
quantum
Posts: 590
Joined: 19 Sep 2006, 22:48

Re: Dev Meeting Minutes (06-12-2010)

Post by quantum »

For mods, I can think of:
  • modinfo.lua
  • luaai.lua for Lua AIs
  • sidedata.tdf/sidedata.lua for sides
  • side pictures
  • all the defs.lua dependencies, including unit definition files, for the disabled units list.
  • modoptions.lua
  • engineoptions.lua
  • utilities used by unitsync, such as gamedata/system.lua
  • additional files used by modders with VFS.Include
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

ahh nice quantum, that is waht i need :-)
what i ultimately need, is a function like that:

Code: Select all

bool IsMetaFile(std::string filePath);
So i can not check what files are included by VFS .include in other files for example.

optimum for me, is a list like this:

Code: Select all

/mod*.lua
/luaai.lua
/sidepics/*
/units/*.fbi
/gamedata/sidedata.tdf
/gamedata/sidedata.lua
/ stand for the archive root. i don't know the location of all the files you mentioned (eg, is engineoptions.lua in root or gamedata?) and which files might likely be included by the meta-files, are they all in one sub-dir maybe?
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker »

1. Does this mean that m--info.lua, even though it's in the root, doesn't generally get added first and become easy to decompress?

2. The things quantum listed are generally part of loading a single map/game to interact with; I don't see much need for them to be individually fast. Caching should be capable of handling it.

3. I suggest a simpler check: bool costTooHigh = (to-be-unpacked-data-size - file-size > 32KB);
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Dev Meeting Minutes (06-12-2010)

Post by zwzsg »

Beherith wrote:I did some testing with small and large maps and mods, there is less than 5% diff between solid and non solid. The only thing in favor of not removing solid is the fact that some legacy (very few) are in solid.
I did some testing with solid and non solid.
spring 0.82.6.1 portable, 2 mods, 12 maps. So a very minimalistic setting.

Load time with all archives solid: 2min 15s
Load time with none of the archives solid: <5s
User avatar
knorke
Posts: 7971
Joined: 22 Feb 2006, 01:02

Re: Dev Meeting Minutes (06-12-2010)

Post by knorke »

why does it have to check all archives anyway?
if i am playing modX.sd7 on mapY.sd7, why would it need to scan other files?
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Dev Meeting Minutes (06-12-2010)

Post by zwzsg »

Actually, I'm talking about the first time Spring is run, and build the cache.

Once the cache is built, subsquent runs are faster.
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14673
Joined: 17 Nov 2005, 02:43

Re: Dev Meeting Minutes (06-12-2010)

Post by Forboding Angel »

All this over solid archives? Fuck that, just disallow solid archives, issue solved.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker »

It would help if you read the thread and saw how a way to detect solid archives was only now worked out.
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Dev Meeting Minutes (06-12-2010)

Post by Tobi »

knorke wrote:why does it have to check all archives anyway?
if i am playing modX.sd7 on mapY.sd7, why would it need to scan other files?
It needs to build a cache of the metadata (game/map name, dependencies, etc.) so that lobbies can access that metadata quickly.

After an archive is read and the metadata is cached Spring will only re-read the archive if the modification time changed.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

lurker wrote:1. Does this mean that m--info.lua, even though it's in the root, doesn't generally get added first and become easy to decompress?

2. The things quantum listed are generally part of loading a single map/game to interact with; I don't see much need for them to be individually fast. Caching should be capable of handling it.

3. I suggest a simpler check: bool costTooHigh = (to-be-unpacked-data-size - file-size > 32KB);
1. If you want a file from a solid block, you have to decompress the whole solid block. There is no order of files that would matter there. If there are no solid blocks, or they are small, then you have to decompress little, and here order does not matter either. I do not know the default settings of 7zip, but i know that Ultra settings include (full-)solid -> the whole archive one solid block. my guess is, that the default is solid blocks of x KB, or something like that.

2. Yeah true, as it is now in spring world. though i could imagine some tools being interested in these files when scanning through. For example, some auto-generated Game info site, listing all the units, LuaAIs and so on, for each uploaded archive. but yeah, to mark the mod as broken to speed-up such a hypothetical thing is probably not good :D. i may put these in a warn-only list, if i see that many of the current mod archives would be marked as broken because of these.

3. will use :-)

so as it is now, the main (always error) things on the list are still only mod- & mapinfo.lua plus *.smf *.sm3.
User avatar
quantum
Posts: 590
Joined: 19 Sep 2006, 22:48

Re: Dev Meeting Minutes (06-12-2010)

Post by quantum »

Here are the paths:

Code: Select all

/armor.txt
/engineoptions.lua
/luaai.lua
/modinfo.lua
/modoptions.lua
/springignore.txt
/validmaps.lua
/features/*
/gamedata/*
/sidepics/*
/units/*
/weapons/*
edit: changed luaui.lua to luaai.lua
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker »

hoijui wrote:1. If you want a file from a solid block, you have to decompress the whole solid block. There is no order of files that would matter there. If there are no solid blocks, or they are small, then you have to decompress little, and here order does not matter either. I do not know the default settings of 7zip, but i know that Ultra settings include (full-)solid -> the whole archive one solid block. my guess is, that the default is solid blocks of x KB, or something like that..
You're sure you have to decompress the whole thing? This is definitely an implementation issue. There is absolutely no reason to decompress an entire block to get data at the start of it. I wonder if it can be overcome.

7zip has a default solid block size of 128MB on fast and 2GB on normal.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

@quantum: thanks :-)
but why the weapons and features?

@lurker
yes i am sure, and yes it is a matter of how solid blocks work in LZMA. There is no such thing like the file at the start of the solid block. It is called solid, cause you have to read and decompress all of it, to get to any byte of uncompressed data. to overcome the downside of this is the essence of this discussion.
if it could be overcome in a way you think about it, the creator of 7zip would probably have told me when i asked for a solution, but he advised to do it this way.
Limitation of the solid block size usually decreases compression ratio but gives the following advantages:
* Decreases losses in case of future archive damage.
* Decreases extraction time of a group of files (or just one file), so long as the group doesn't contain the entire archive.
from the 7zip Manual at:
http://bugaco.com/7zip/MANUAL/switches/method.htm#Solid
User avatar
quantum
Posts: 590
Joined: 19 Sep 2006, 22:48

Re: Dev Meeting Minutes (06-12-2010)

Post by quantum »

hoijui wrote: but why the weapons and features?
Lobbies get the unit list from unitsync. Unitsync gets it by executing gamedata/defs.lua, which loads units, features, weapons, move types and armors (that's why armor.txt is in the list).
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker »

LZMA doesn't work that way. There's no lookahead in decompression; you run through a stream linearly, keeping around the last X decompressed bytes so it can easily output repeated data. 7-zip's GUI reflects this. The 'memory usage for decompressing' is always the window size + 2MB, no matter how large the solid block size is. Did you specify that you needed a file at the *start* of the archive (is this even the case; has anyone checked?)?
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui »

quantum, thanks again :-)

lurker
the LZMA wiki does explain the algorithm, but does not even contain the words "solid" or "block":
http://en.wikipedia.org/wiki/LZMA

my guess is, that this is kind of an extension by 7zip, which crams multiple files into one, or something like that. or where did you read about how it works?
Post Reply

Return to “Meeting Minutes”