Dev Meeting Minutes (06-12-2010)

[abma]

afaik, "solid" is stuff of the container-format, in this case 7z.
lzma is only the compression.

i think you've to look into the 7z-sdk to find the word solid.

[lurker]

It doesn't seem to be anything complicated. Instead of compressing each file with LZMA, it compresses each X megabytes with LZMA. So it should be possible to easily extract items at the start of a block.

The order in which it compresses files, however, is still an unknown.

[hoijui]

no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.

[id10terror]

The files are compressed (yes in a container format), solid refers to the fact that there is no index to the archive, meaning to get a single file out the whole container file must be 'read'(usually this also means decompressed) to find the start of the particular file you want.

Such archivers could create a dynamic in memory cache of such locations provided the files they were processing werent to large, but this also begs the question; why not simply cache config files as loaded from archives?

[lurker]

solid refers to the fact that there is no index to the archive

Wrong. All the metadata is stored in its own location, you can see it without extracting, and in fact the entire point of a solid block size is that you only need to extract the blocks you know the file is in, not the whole archive.

no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.

How is this at all possible with LZMA compression on each block?

Edit: Okay, I just did a very simple test. 300MB solid archive with a lot of files, tell it to extract the entire thing. Lo and behold, it starts outputting files immediately, even though it's only a fraction of a percent through decompressing the block.

[id10terror]

Aww man, extract one file from the 'middle' of the archive and you will notice that the decompressor goes thru all the files in the archive to extract the one you want. if the archive is non solid(or contain an index) the file will be extracted without processing all the compressed files. Now you can do your own research here ->
http://en.wikipedia.org/wiki/Solid_compressionSpecifically the parts

In computing, solid compression refers to a method for data compression of multiple files, wherein all the compressed files are concatenated and treated as a single data block.

On the other hand, getting a single file out of a solid archive requires processing all the files before it, so modifying solid archives can be slow and inconvenient

Dont waste anymore of my time.
Apologies to lurker for the above comment, i was mad, in reality my time is pretty worthless. I'll leave the comment there tho so as not to cover up who i am.

[lurker]

Thank you, id10terror, for making my point. You only have to extract all the files before it if you want a file from the middle of a block. m??info.lua is very likely to be at the beginning of a block.

[id10terror]

Ahhhh, it gets better. I feel empowered now, being both wrong and right at exactly the same time. Oh the power! You guys better watch out or i'm going to go quantum on your asses.

[smoth]

so the difference is between sequential and indexed? Is there any question as to what is faster? I don't get this whole thread.

[id10terror]

In Sequential: If the content being read is not at the beginning of a solid archive all contents before it must be read. If you have 'ordered' content this isn't a problem really. Most people don't 'order' the content as this requires adding files 'one at a time' to an archive.

In Indexed: Doesn't matter where the content is located in the archive a read to a 'far' content should be as fast as a read to 'close' content.

Some people create solid(sequential) archives to squeeze that little bit extra compression, this has a larger effect if the archive contains a lot of small files.

=== un-support solid archives? ===
<zerver> what is that?
<hoijui> i got told, that in the past, we did not suport solid archives, but now we do
<hoijui> soid archives are quite bad for mods and maps, aren't they?
<hoijui> like.. when you want to look for ModInfo.lua only
<hoijui> so i suggest, removign suport for solid archives, or at least show some big ugly warning

Apparently they want to stop supporting solid archives. Another user has already pointed out the caveat that goes with this.

<Tobi> if it can be disabled using the new library..

The end decision in regard to solid archives:

bool costTooHigh = (to-be-unpacked-data-size > 32KB) && (to-be-unpacked-data-size > file-size);

I may have this wrong but: the check is only run if the archive is solid. So if you use a non solid archive you will not have to worry.

1.gif

7Zip Add to archive screenshot

(52.58 KiB) Downloaded 2 times