Dev Meeting Minutes (06-12-2010) - Page 3

Dev Meeting Minutes (06-12-2010)

Minutes of the meetings between Spring developers are archived here.
abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Dev Meeting Minutes (06-12-2010)

Post by abma » 14 Dec 2010, 01:21

afaik, "solid" is stuff of the container-format, in this case 7z.
lzma is only the compression.

i think you've to look into the 7z-sdk to find the word solid.
0 x

User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker » 14 Dec 2010, 02:27

It doesn't seem to be anything complicated. Instead of compressing each file with LZMA, it compresses each X megabytes with LZMA. So it should be possible to easily extract items at the start of a block.

The order in which it compresses files, however, is still an unknown.
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Dev Meeting Minutes (06-12-2010)

Post by hoijui » 14 Dec 2010, 13:22

no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.
0 x

id10terror
Posts: 35
Joined: 08 Nov 2009, 06:39

Re: Dev Meeting Minutes (06-12-2010)

Post by id10terror » 14 Dec 2010, 16:58

The files are compressed (yes in a container format), solid refers to the fact that there is no index to the archive, meaning to get a single file out the whole container file must be 'read'(usually this also means decompressed) to find the start of the particular file you want.

Such archivers could create a dynamic in memory cache of such locations provided the files they were processing werent to large, but this also begs the question; why not simply cache config files as loaded from archives?
0 x

User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker » 15 Dec 2010, 01:22

id10terror wrote:solid refers to the fact that there is no index to the archive
Wrong. All the metadata is stored in its own location, you can see it without extracting, and in fact the entire point of a solid block size is that you only need to extract the blocks you know the file is in, not the whole archive.

hoijui wrote:no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.
How is this at all possible with LZMA compression on each block?

Edit: Okay, I just did a very simple test. 300MB solid archive with a lot of files, tell it to extract the entire thing. Lo and behold, it starts outputting files immediately, even though it's only a fraction of a percent through decompressing the block.
0 x

id10terror
Posts: 35
Joined: 08 Nov 2009, 06:39

Re: Dev Meeting Minutes (06-12-2010)

Post by id10terror » 15 Dec 2010, 02:38

Aww man, extract one file from the 'middle' of the archive and you will notice that the decompressor goes thru all the files in the archive to extract the one you want. if the archive is non solid(or contain an index) the file will be extracted without processing all the compressed files. Now you can do your own research here ->
http://en.wikipedia.org/wiki/Solid_compressionSpecifically the parts
In computing, solid compression refers to a method for data compression of multiple files, wherein all the compressed files are concatenated and treated as a single data block.
On the other hand, getting a single file out of a solid archive requires processing all the files before it, so modifying solid archives can be slow and inconvenient
Dont waste anymore of my time.
Apologies to lurker for the above comment, i was mad, in reality my time is pretty worthless. I'll leave the comment there tho so as not to cover up who i am.
0 x

User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Dev Meeting Minutes (06-12-2010)

Post by lurker » 15 Dec 2010, 21:12

Thank you, id10terror, for making my point. You only have to extract all the files before it if you want a file from the middle of a block. m??info.lua is very likely to be at the beginning of a block.
0 x

id10terror
Posts: 35
Joined: 08 Nov 2009, 06:39

Re: Dev Meeting Minutes (06-12-2010)

Post by id10terror » 16 Dec 2010, 03:21

Ahhhh, it gets better. I feel empowered now, being both wrong and right at exactly the same time. Oh the power! You guys better watch out or i'm going to go quantum on your asses. ;)
0 x

User avatar
smoth
Posts: 22300
Joined: 13 Jan 2005, 00:46

Re: Dev Meeting Minutes (06-12-2010)

Post by smoth » 16 Dec 2010, 05:02

so the difference is between sequential and indexed? Is there any question as to what is faster? I don't get this whole thread.
0 x

id10terror
Posts: 35
Joined: 08 Nov 2009, 06:39

Re: Dev Meeting Minutes (06-12-2010)

Post by id10terror » 16 Dec 2010, 09:34

In Sequential: If the content being read is not at the beginning of a solid archive all contents before it must be read. If you have 'ordered' content this isn't a problem really. Most people don't 'order' the content as this requires adding files 'one at a time' to an archive.

In Indexed: Doesn't matter where the content is located in the archive a read to a 'far' content should be as fast as a read to 'close' content.

Some people create solid(sequential) archives to squeeze that little bit extra compression, this has a larger effect if the archive contains a lot of small files.
=== un-support solid archives? ===
<zerver> what is that?
<hoijui> i got told, that in the past, we did not suport solid archives, but now we do
<hoijui> soid archives are quite bad for mods and maps, aren't they?
<hoijui> like.. when you want to look for ModInfo.lua only
<hoijui> so i suggest, removign suport for solid archives, or at least show some big ugly warning
Apparently they want to stop supporting solid archives. Another user has already pointed out the caveat that goes with this.
<Tobi> if it can be disabled using the new library..
The end decision in regard to solid archives:
bool costTooHigh = (to-be-unpacked-data-size > 32KB) && (to-be-unpacked-data-size > file-size);
I may have this wrong but: the check is only run if the archive is solid. So if you use a non solid archive you will not have to worry.
1.gif
7Zip Add to archive screenshot
(52.58 KiB) Downloaded 2 times
0 x

Post Reply

Return to “Meeting Minutes”

cron