afaik, "solid" is stuff of the container-format, in this case 7z.
lzma is only the compression.
i think you've to look into the 7z-sdk to find the word solid.
Dev Meeting Minutes (06-12-2010)
Re: Dev Meeting Minutes (06-12-2010)
It doesn't seem to be anything complicated. Instead of compressing each file with LZMA, it compresses each X megabytes with LZMA. So it should be possible to easily extract items at the start of a block.
The order in which it compresses files, however, is still an unknown.
The order in which it compresses files, however, is still an unknown.
Re: Dev Meeting Minutes (06-12-2010)
no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.
-
- Posts: 35
- Joined: 08 Nov 2009, 06:39
Re: Dev Meeting Minutes (06-12-2010)
The files are compressed (yes in a container format), solid refers to the fact that there is no index to the archive, meaning to get a single file out the whole container file must be 'read'(usually this also means decompressed) to find the start of the particular file you want.
Such archivers could create a dynamic in memory cache of such locations provided the files they were processing werent to large, but this also begs the question; why not simply cache config files as loaded from archives?
Such archivers could create a dynamic in memory cache of such locations provided the files they were processing werent to large, but this also begs the question; why not simply cache config files as loaded from archives?
Re: Dev Meeting Minutes (06-12-2010)
Wrong. All the metadata is stored in its own location, you can see it without extracting, and in fact the entire point of a solid block size is that you only need to extract the blocks you know the file is in, not the whole archive.id10terror wrote:solid refers to the fact that there is no index to the archive
How is this at all possible with LZMA compression on each block?hoijui wrote:no, you always have to read a whole solid block from disc, and extract it, to get to any data inside of it. if there is any well defined order, it does not matter.
Edit: Okay, I just did a very simple test. 300MB solid archive with a lot of files, tell it to extract the entire thing. Lo and behold, it starts outputting files immediately, even though it's only a fraction of a percent through decompressing the block.
-
- Posts: 35
- Joined: 08 Nov 2009, 06:39
Re: Dev Meeting Minutes (06-12-2010)
Aww man, extract one file from the 'middle' of the archive and you will notice that the decompressor goes thru all the files in the archive to extract the one you want. if the archive is non solid(or contain an index) the file will be extracted without processing all the compressed files. Now you can do your own research here ->
http://en.wikipedia.org/wiki/Solid_compressionSpecifically the parts
Apologies to lurker for the above comment, i was mad, in reality my time is pretty worthless. I'll leave the comment there tho so as not to cover up who i am.
http://en.wikipedia.org/wiki/Solid_compressionSpecifically the parts
In computing, solid compression refers to a method for data compression of multiple files, wherein all the compressed files are concatenated and treated as a single data block.
Dont waste anymore of my time.On the other hand, getting a single file out of a solid archive requires processing all the files before it, so modifying solid archives can be slow and inconvenient
Apologies to lurker for the above comment, i was mad, in reality my time is pretty worthless. I'll leave the comment there tho so as not to cover up who i am.
Re: Dev Meeting Minutes (06-12-2010)
Thank you, id10terror, for making my point. You only have to extract all the files before it if you want a file from the middle of a block. m??info.lua is very likely to be at the beginning of a block.
-
- Posts: 35
- Joined: 08 Nov 2009, 06:39
Re: Dev Meeting Minutes (06-12-2010)
Ahhhh, it gets better. I feel empowered now, being both wrong and right at exactly the same time. Oh the power! You guys better watch out or i'm going to go quantum on your asses.
Re: Dev Meeting Minutes (06-12-2010)
so the difference is between sequential and indexed? Is there any question as to what is faster? I don't get this whole thread.
-
- Posts: 35
- Joined: 08 Nov 2009, 06:39
Re: Dev Meeting Minutes (06-12-2010)
In Sequential: If the content being read is not at the beginning of a solid archive all contents before it must be read. If you have 'ordered' content this isn't a problem really. Most people don't 'order' the content as this requires adding files 'one at a time' to an archive.
In Indexed: Doesn't matter where the content is located in the archive a read to a 'far' content should be as fast as a read to 'close' content.
Some people create solid(sequential) archives to squeeze that little bit extra compression, this has a larger effect if the archive contains a lot of small files.
In Indexed: Doesn't matter where the content is located in the archive a read to a 'far' content should be as fast as a read to 'close' content.
Some people create solid(sequential) archives to squeeze that little bit extra compression, this has a larger effect if the archive contains a lot of small files.
Apparently they want to stop supporting solid archives. Another user has already pointed out the caveat that goes with this.=== un-support solid archives? ===
<zerver> what is that?
<hoijui> i got told, that in the past, we did not suport solid archives, but now we do
<hoijui> soid archives are quite bad for mods and maps, aren't they?
<hoijui> like.. when you want to look for ModInfo.lua only
<hoijui> so i suggest, removign suport for solid archives, or at least show some big ugly warning
The end decision in regard to solid archives:<Tobi> if it can be disabled using the new library..
I may have this wrong but: the check is only run if the archive is solid. So if you use a non solid archive you will not have to worry.bool costTooHigh = (to-be-unpacked-data-size > 32KB) && (to-be-unpacked-data-size > file-size);