Spring RTS Engine

Following post cover TAS Maps and Objects3D compression test to show difference between 7-Zip , RAR, ZIP and once ACE compression ratios:

Changelog:
- updated to match latest GOW build v2
- fixed little bug (left two files generated from TAS in redhaven_01)
- added compression result of HAPI content
- added metalheck/metalheck2 and prime core maps
- added ACE selfextracting value of altored_mars (bad result anyway :)

Summary:
Unbeatable winner for TAS maps and Objects3D is Opensource 7-zip http://www.7-zip.org
Compressing times (approx 3 times longer than RAR) don't matter as for distributing files is priority smallest possible size and fast decompression time ... and these key elements 7-zip meets.

Suggestion to map makers, mod makers and so on please use it!
You lower overall bandwidth demand at servers and notable reduce times needed to download Your work!.

Test system:
AMD 3800+, 1GB DDR466, SATA strip raid

Settings:
For RAR and ZIP was used WinRAR set to maximal compression, no recovery record, solid archive type, self extracting
For 7-zip was used 7z format , LZMA compression, dictionary size 16MB , word size 96 , solid archive type, selfextracting

Results:
sizes are in bytes

Complete set of all custom maps released till now:

27404649 maps.7z
34071273 maps.rar
50379714 maps.zip


Difference in sizes is huge 7-zip offers 1.84 times better compression than ZIP and 1.24 times better compression than RAR

Complete set of all updated 3DO (combined of what posted zw* and from evolva):

394896 Objects3D.7z
472536 Objects3D.rar
524337 Objects3D.zip

7-zip still offers 1.33 times better compression than ZIP and 1.2 times better compression than RAR

HAPI content of TAS:
08803363 7Z
08912204 7Z and only 1MB dictionary !
09272961 RAR
13181220 ZIP
12838180 TAS CCX

HAPI content of OTA:
088696 7z
132997 RAR
469648 ZIP
420768 OTA UFO

You can clearly see here

Each map standalone (waste of time test zip there, you can see it on originals from authors :) :

5037224 altored_mars.7z
6098071 altored_mars.rar
6287897 altored_mars.ace ( by Caydr )
3250821 comet_catcher_09.7z
3742065 comet_catcher_09.rar
1455105 core_prime_industrial_area_01.7z
2016659 core_prime_industrial_area_01.rar
0228437 crosspaths.7z
0269910 crosspaths.rar
4336235 gods_of_war_v2.7z
5159163 gods_of_war_v2.rar
4996962 metalheck.7z
5811897 metalheck.rar
6891561 metal_heck_2(spring_map).7z
9425531 metal_heck_2(spring_map).rar
1813844 redhaven_01.7z
2333095 redhaven_01.rar

Each 7-zip archive can be downloaded from following mirror
You can extract them directly to TAS main directory or rename to *.exe and use self extracting feature.

3Dobjects:
http://3web.dkm.cz/myie2cz/tas/Objects3D.7z

Each map standalone:
http://3web.dkm.cz/myie2cz/tas/altored_mars.7z
http://3web.dkm.cz/myie2cz/tas/comet_catcher_09.7z
http://3web.dkm.cz/myie2cz/tas/core_pri ... area_01.7z
http://3web.dkm.cz/myie2cz/tas/crosspaths.7z
http://3web.dkm.cz/myie2cz/tas/gods_of_war_v2.7z
http://3web.dkm.cz/myie2cz/tas/metalheck.7z
http://3web.dkm.cz/myie2cz/tas/metal_he ... ng_map).7z
http://3web.dkm.cz/myie2cz/tas/redhaven_01.7z

OTA settings ufo file (this one is not self extracting because of size):
http://3web.dkm.cz/myie2cz/tas/OTA_sett ... g_patch.7z

7 zip seems like a nice format, if someone want to integrate it into the CFilehandler so that it get supported the same way as the hpi formats it would be nice.

little update to match latest maps ...

once more small update

How does Ace stand up? My 16 mb map was reduced to 6 mb.

Caydr wrote:How does Ace stand up? My 16 mb map was reduced to 6 mb.

quite bad, see results 1MB worse ...

what about tar.bz2 and tar.gz?

Tangaroa wrote:what about tar.bz2 and tar.gz?

you seriously joking right ? :) ... bz2 will be fine but why when 7-zip is already ported to linux ... so why bother ? :)

So you prolly own stocks in 7 Zip i guess then spamming a topic like this :)

Redfish wrote:So you prolly own stocks in 7 Zip i guess then spamming a topic like this :)

you know i wish 7-zip was on stock exchange :) it will be worthy investment :) lol

Yes, 7 zip probably gives better compression than zip and rar. Does that mean we should abandon at once .zip? I don't think so. Compression ratio isn't the only think to consider when realeasing an archive for everybody to download.

Nowadays, every (or let's say 99%) PC that is able to run Spring can handle zip. It's even been integrated in some newest Microsoft Windows Operatation system file manager (not that I like MS or that particular feature). So everyone know what a .zip is and know how to handle them. .rar aren't that common, but it's also years old and was used by many, so it's not as user-friendly as .zip but still not that bad.

7z on the other hand I think I never saw. And I'm probably not the only one. So if see an archive ending in 7s and my usual file archive can't handle it, I'll probably wonder what is it, then leave for later (later as in never).

Ok, the renaming into .exe to get a self extracting sorta beat me, but I can still counter that argument saying that I would have to know it before taking advantage of that property (most people who will one day or later download a spring map won't have read your thread first), and also by claiming I hate self extractible zip because I'm paranoid about .exe

So, the point I was trying to make, was that althought formats with better compression exist, it makes sense to use not the one with the best compression rate but the one most widespread instead.

Also, I'll tell you to compare not only the size of each compression format for a given map, but also the size of each map for a given archive format.: Map filesizes varies from 1 to 25. So incomparably more space can be saved by asking the map makers to do good job than by annoying the end-user with exotic archive formats.

bah i was going to say something on 7-zip defence but ... i let this to decide others ... if you want use zip, rar or ace ... Your choice ...

p.s. about that detail "any" PC can run zip ... well 7-zip too ...

@zwzsg People who don't do edditing, normal user don't have to open 7zip.
And I hope every model, map make have sufficent knowledge to install a small compression program.

I'm for 7-zip too because I really fear to have a 5go Spring folder...
If Dwarden test is correct (http://taspring.clan-sy.com/phpbb/viewtopic.php?t=843) I would really apreciate to have a 40% better compression on my map folder.

I am thinking seriously about implimenting the 7z compression into a new HPI file structure. We don't need to use the entire 7z file system to use the 7z compression.

Gimme a couple weeks, I'll have it all nailed down.

There's also the matter of 7zip using a _LOT_ of system memory. According to my setup compiler, the LZMA method that 7zip uses has the following approximate memory usage:

Decompression Compression
fast 3 MB 3 MB
normal 4 MB 27 MB
max 10 MB 84 MB
ultra 34 MB 369 MB

(sorry about the formatting...)

Caydr wrote:There's also the matter of 7zip using a _LOT_ of system memory. According to my setup compiler, the LZMA method that 7zip uses has the following approximate memory usage:

Decompression Compression
fast 3 MB 3 MB
normal 4 MB 27 MB
max 10 MB 84 MB
ultra 34 MB 369 MB

(sorry about the formatting...)

memory usage in 7-zip is affected by compression method and mainly dictionary size!!!

optimal values are use dictionary 1MB and 96 Word ... 7z format, Ultra compression level and LZMA method ...

using these values mean You will need only 75MB for compression and 3MB for decompression ...

using higher dictionary size needs more ram both for Compression and decompression !

and it will only change to compression is maximally by 1-3% ...
using 2-16MB dictionary can come handy when you compressing 100k + files ... but again this need bit of testing as it vary by archived file content used ...


word size, that could vary but usually 96 produced best result for compression ratio ... using lower valeus was resulting into noticeably worse ratio and higher value was from 1% better to 1% worse result ... so i used for test 96 ...

i think following table in 7zhelp can give tips and help understanding how choose best one as it contains memory needed, speeds and so on (you need scroll 1 page down) ...
http://3web.dkm.cz/myie2cz/tas/7ziphelp.htm