SD7 vs SDZ: read time
Moderator: Moderators
SD7 vs SDZ: read time
Can one of these be accessed by Spring faster? Is there any reason to use SDZ instead of SD7? I'm sort of curious why so many things, like XTA for instance, are still SDZ when superior compression is available.
-
Forboding Angel
- Evolution RTS Developer
- Posts: 14673
- Joined: 17 Nov 2005, 02:43
Sdz is accessed faster. At least on my machine this is the way it works.
See when it says "Opening map file" when loading up? That is accessed faster for example. I don't use sd7 simply because I don't like it. The compression is so huge that it makes minimaps load slower, makes loading the game slower when starting the match, etc.
I have done somewhat extensive testing on this and found the above to be true.
For example, delete your pathing info on altored earth. then put the map in a zip file on max compression and rename to sdz. Then take the same maps folder and compress it with 7z on ultra setting.
You will notice a big difference in loading times (probably... on my machine I notice it). The tradeoff is filesize. Personally I don't care about filesize. Other ppl don't agree with me thats fine. I'm not forcing them to download anything I post.
See when it says "Opening map file" when loading up? That is accessed faster for example. I don't use sd7 simply because I don't like it. The compression is so huge that it makes minimaps load slower, makes loading the game slower when starting the match, etc.
I have done somewhat extensive testing on this and found the above to be true.
For example, delete your pathing info on altored earth. then put the map in a zip file on max compression and rename to sdz. Then take the same maps folder and compress it with 7z on ultra setting.
You will notice a big difference in loading times (probably... on my machine I notice it). The tradeoff is filesize. Personally I don't care about filesize. Other ppl don't agree with me thats fine. I'm not forcing them to download anything I post.
-
Tim Blokdijk
- Posts: 1242
- Joined: 29 May 2005, 11:18
What I'm really concerned about is quickening AA's operation. I use the "fast" compression method with 7zip because it requires less time and memory to decompress and its savings compared to Ultra are almost the same.
Something I'm curious about though, is the periodic "hiccup" that happens when you start construction on new things, especially construction units and factories. Is this the game accessing the SD7 file and caching the buildpics? Or is it completely unrelated?
What got me thinking about this was when I converted all my BPs to DDS and noticed that the effect *seemed* to be a little less noticable. So it seemed to logically follow that it might be possible to negate the hiccup entirely if I used the right combination of low compression on the archive and high compression on the buildpics. Or somesuch similar...
Then I remembered that XTA has always been sdz. Maybe they too noticed this phenomenon and found it was not as bad with zip compression? There's no other logical reason they'd stick with crappier compression when the savings could be as much as a few megabytes.
I guess what I really need to know is, when spring initializes, does it decompress the sd7 entirely, or just the parts that are needed at any given time? If the entire thing is decompressed, the compression obviously couldn't be to blame.
Something I'm curious about though, is the periodic "hiccup" that happens when you start construction on new things, especially construction units and factories. Is this the game accessing the SD7 file and caching the buildpics? Or is it completely unrelated?
What got me thinking about this was when I converted all my BPs to DDS and noticed that the effect *seemed* to be a little less noticable. So it seemed to logically follow that it might be possible to negate the hiccup entirely if I used the right combination of low compression on the archive and high compression on the buildpics. Or somesuch similar...
Then I remembered that XTA has always been sdz. Maybe they too noticed this phenomenon and found it was not as bad with zip compression? There's no other logical reason they'd stick with crappier compression when the savings could be as much as a few megabytes.
I guess what I really need to know is, when spring initializes, does it decompress the sd7 entirely, or just the parts that are needed at any given time? If the entire thing is decompressed, the compression obviously couldn't be to blame.
no its not just units that hickup its the (builder) units that do it... and buildings too... anything with build treeGnome wrote:XTA hiccups for me as much as any SD7 compressed mod.
I think it's texture loading on the model, btw, since it only does it once per unit type. Build one Krogoth and it hiccups, but it doesn't on the second or after.
Something completly illogical? Did you add more stuff? That would seem illogical to me, as you are trying to reduce the filesize.
Or, you changed to a different compression option, that usually makes stuff larger.
Or you found a magic thing that made an uber compression program for you, that is some how compatible with spring?
Or your computer exploded, in the process ejecting a floppy disk that somehow had the new AA thing on it?
Or aliens visited you in your sleep, and in exchange for letting them carry out experiments, they game you the new AA thing?
Tell us!
Or, you changed to a different compression option, that usually makes stuff larger.
Or you found a magic thing that made an uber compression program for you, that is some how compatible with spring?
Or your computer exploded, in the process ejecting a floppy disk that somehow had the new AA thing on it?
Or aliens visited you in your sleep, and in exchange for letting them carry out experiments, they game you the new AA thing?
Tell us!
Well, 1.42 is out of the bag now, so I might as well divulge my accidental discovery.
If you compress your main mod file(s) as a sdz (ZIP rather than 7zip), with NO compression - as in, "Store" setting, you can then compress the resulting zip file much smaller when you put it in the distribution ultra-compressed zip.
So, for instance, inside the AA ZIP file (which is compressed to the maximum), you will find my readme, changelog, license, a few pimping links, plus the 5 AA data files:
AASpring142.sdz (20 mb)
AASpring142S.sdz (25 kb)
AASpring142F.sdz (35 kb)
AASpring142B.sdz (25kb)
AASpring142LT.sdz (25kb)
These last 5 files are completely free from compression - meaning that they are 1) quite large (20.5 mb in all) and 2) accessed by the game faster than any other possible compression.
If I compress each of the above 5 files as sd7 (7zip), the mod is 9.6 mb. If I compress them as normal-compression sdz (zip), the mod is 10.2 mb. If I do not compress them at all, the mod is 8.6 mb.
So, basically, the file itself is smaller, yet the game can access it faster. This almost certainly doesn't apply to map files, but I haven't tested it. Maybe it's something AA-specific, but I don't see why it would be. It's completely illogical, yet for some reason the mod is smaller when you don't compress it.
The only person who loses is the guy who can't spare some extra megabytes in his spring directory. IE, nobody, not even these low-polygon, low-detail-demanding freaks.
If you compress your main mod file(s) as a sdz (ZIP rather than 7zip), with NO compression - as in, "Store" setting, you can then compress the resulting zip file much smaller when you put it in the distribution ultra-compressed zip.
So, for instance, inside the AA ZIP file (which is compressed to the maximum), you will find my readme, changelog, license, a few pimping links, plus the 5 AA data files:
AASpring142.sdz (20 mb)
AASpring142S.sdz (25 kb)
AASpring142F.sdz (35 kb)
AASpring142B.sdz (25kb)
AASpring142LT.sdz (25kb)
These last 5 files are completely free from compression - meaning that they are 1) quite large (20.5 mb in all) and 2) accessed by the game faster than any other possible compression.
If I compress each of the above 5 files as sd7 (7zip), the mod is 9.6 mb. If I compress them as normal-compression sdz (zip), the mod is 10.2 mb. If I do not compress them at all, the mod is 8.6 mb.
So, basically, the file itself is smaller, yet the game can access it faster. This almost certainly doesn't apply to map files, but I haven't tested it. Maybe it's something AA-specific, but I don't see why it would be. It's completely illogical, yet for some reason the mod is smaller when you don't compress it.
The only person who loses is the guy who can't spare some extra megabytes in his spring directory. IE, nobody, not even these low-polygon, low-detail-demanding freaks.
Wow.
Oh just wow.
You just noticed the wonders of solid compression. You could enable it just by setting the checkbox in 7zip directly
Welcome to 1995.
Have you ever seen a .tar.gz file? Wonder why they use "tar" to stick files together before they are zipped? Just the very same effect.
Here is the reason why this works that way and why its not even remotely unlogical:
When all files are storred together in a long block, intrafile redundency will also be eleminated. Also, it doesnt need a header segment for each file with the corresponding data. To understand this, remember that the default compression window for 7-zip, for example, is 4 or 8 Mbyte. Those 30kbyte pictures already end before the algorithms have "come to speed".
Also, if you already have those files lzw encoded, this process results in the entropy to skyrocket, thus resulting in 7-zip not being able to use the uniform features of the data (like those series of 96*96 bitmap blocks).
And here is the reason why you _shouldnt_ do it:
With solid compression (or your fake approach), you cannot decompress one file from the archive without decopressing the whole archive from the beginning to the point of the requested file. This doesnt hurt for sequenciel stuff, but for random access it can easily make it 50 times slower.
Oh just wow.
You just noticed the wonders of solid compression. You could enable it just by setting the checkbox in 7zip directly
Welcome to 1995.
Have you ever seen a .tar.gz file? Wonder why they use "tar" to stick files together before they are zipped? Just the very same effect.
Here is the reason why this works that way and why its not even remotely unlogical:
When all files are storred together in a long block, intrafile redundency will also be eleminated. Also, it doesnt need a header segment for each file with the corresponding data. To understand this, remember that the default compression window for 7-zip, for example, is 4 or 8 Mbyte. Those 30kbyte pictures already end before the algorithms have "come to speed".
Also, if you already have those files lzw encoded, this process results in the entropy to skyrocket, thus resulting in 7-zip not being able to use the uniform features of the data (like those series of 96*96 bitmap blocks).
And here is the reason why you _shouldnt_ do it:
With solid compression (or your fake approach), you cannot decompress one file from the archive without decopressing the whole archive from the beginning to the point of the requested file. This doesnt hurt for sequenciel stuff, but for random access it can easily make it 50 times slower.