Units, weapons...hpi's OH MY

[Buggi]

OY!

Half of the stuff I'm trying to get loaded in the game won't load because the id's are all fubar. All the current utilities shoot down the units that go beyond the scope of OTA... >_<

Hence my need to redo the HPI, or Happy, format scripts.

I was looking into a 7zip format and that's just crazy. It's going to take several weeks to begin that transition so for the short term I'm going to try and load HPI's.

One thing that's needed is the ability to change the location on the menu of units.

Some days I hate being a programmer, I know how much work that's going to be

[Dave]

Actually, after checking out 7zip some more, Ace07 and myself are a little concerned about it's portability. However, there are several components to the whole thing.

Which components of 7zip were you specifically intending on using?

As a side note, wouldn't it be better to go for a more widely supported/known format, like zip? Many games and game engines use zip (or some camouflaged form thereof), and I believe that open source implementations are legion, in any language we could conceivably need.

[Ace07]

Yes, it depends on which pieces you want to impliment.

However, the real question is: how much do we want to compress the files?

Do we need the smallest of the smallest for our game files? I doubt that the install will ever be bigger than 200 megs, so do we need to use 7zip at all?

I admit that it would be cool if we could impliment it with effortless ease (because 7Zip is even better than RAR, which was all the rave a couple of years ago). But, like Dave said above, we are concerned with the portability of 7Zip code. Implimenting 7Zip into the game's engine will definitely be more work than going with a well-known format like Zip.

Not only will it be easier for you, it will also be easier for us (the porters).

[Dave]

And just as another quick addition, the zzip library offers a simple and portable SDK for accessing zip files. It also has a slightly better compression ratio than standard (ie. winzip) tools. It is used by Ogre for their resource management system, where models/textures/material scripts can be stored in zip files.

[Buggi]

Frankly I am not all that concerned with compression, unless we're dealing with a TON of new textures or extremely large maps.

I have 320 total gigs of space on this system, space isn't an issue. :D

However, if a big map is 40 megs, I can understand the attempt to find the better compression method.

[Dave]

I'm not really concerned about compression either. I'm more interested in the archiving notion that the compression brings in, which'd avoid us having hundreds of individual files for a player who doesn't care.

But the very real problem with 7zip is that it would probably require us to port that to linux along with TAspring. Well, actually I'm not certain that a linux port doesn't exist, but the 7zip site is very confusing on this point, and from what we were able to work out, some important parts of the whole thing would need porting.
And, quite frankly, we have enough on our plate with TAspring's code, so if it were possible right now to avoid bringing in extra porting work from outside, I'd appreciate it :D

That's why I suggested zzip, because I know it'll work on linux and windows (given that Ogre3d uses it). It also means users probably won't have to install any other software to access the archive's content (compressed folder support in windows, and zzip is pretty much standard stuff in linux), which will avoid them depending on TAspring's tools the same way we now depend on outdated HPI utilities :)

However, a downside of zzip is that, as far as we could see, there are no C# bindings or C# implementations for it, so you'd have to use some other library that enables access to zip files. There should be a fair few floating around, given zip's popularity.

[Dwarden]

Dave, i'm quite sure you know it , but if all of sudden not ...
here is that quick posix (unix, linux, freeBSD, MacOS etc) port of 7-zip cmdline
http://sourceforge.net/projects/p7zip/

[el_muchacho]

7zip is very slow too.

Maybe zlib (gzip) or LZO compression (used in the UPX tool) would be better. LZO has the advantage of being extremely fast, much faster than most other decompression algorithms, while still achieving a good ratio. In fact, reading large data compressed with LZO might very well prove as fast or even faster than reading the same data in uncompressed format.

http://www.oberhumer.com/opensource/lzo/

[Buggi]

The problem with ANY new implimentation of any new format is not only would it have to be done from scratch with tools like what I'm developing, is it would also have to be supported by the spring engine, so the base source would have to change. In addition to ALL THAT Dave would have to port the new implimentation to Linux.

BUT! (big but) Nothing is saying we can't use a new compression/decompression schema in the current HPI format. All we'd need to do is change the header a little. Bend it to our will so to speak.

I have the source compression/decompression schema's of 7zip. It wouldn't be too much of a stretch to just impliment the new compression in the format. It's rated as 8meg/s decompression... I think... I still need to figure out the FUBAR recursive way that the HAPI directory's work.

[PauloMorfeo]

... pretty much standard stuff in linux ...

I think the best is to look for a tool from Linux that suits the needs. A tool in Linux will most certainly have Win32 decompiler versions and stuff. From Linux tools, one can even have high probability of finding the code for what you are trying to do.

The other way around, has a great probability (like it seems to have happened right now) that it will not have a Linux port, not be easily Linux portable or that it won't integrate into Linux without much work.

Standard compressed files in Linux are .tar.gz .
I am almost sure that the .gz format compresses at least almost as good as .rar (WinRAR opens .tar.gz) and i think that .gz2 has a higher compression than .rar (WinRAR also opens .gz2, i think).

But this is a thing to be decided by the developers!?

I am very eager to see this game on Linux. It will blow sick high anything else i have ever seen for Linux!!!!!

[Ace07]

But this is a thing to be decided by the developers!?

I think we need better data before we can decide. And the SY's would probably be the ones to decide since it is their project.

[el_muchacho]

Standard compressed files in Linux are .tar.gz .
I am almost sure that the .gz format compresses at least almost as good as .rar (WinRAR opens .tar.gz) and i think that .gz2 has a higher compression than .rar (WinRAR also opens .gz2, i think).
!

You probably mean bz2 ? bzip2 achieves a good ratio but is really slow. To gain a few %, the effort seems exponential and is not worth the wait (for a game at least).
The standard zlib implements the algorithm used in gzip, I believe.

[Dwarden]

Standard compressed files in Linux are .tar.gz .
I am almost sure that the .gz format compresses at least almost as good as .rar (WinRAR opens .tar.gz) and i think that .gz2 has a higher compression than .rar (WinRAR also opens .gz2, i think).
!

You probably mean bz2 ? bzip2 achieves a good ratio but is really slow. To gain a few %, the effort seems exponential and is not worth the wait (for a game at least).
The standard zlib implements the algorithm used in gzip, I believe.

7-zip also support bzip2 for zip ...
but bzip2 is more than 2 times slower for decompressing than using LZMA (7z)
zip/gzip is approx twice as fast as LZMA for decopression

i think following table in 7zhelp can give tips and help understanding how choose best one as it contains memory needed, speeds and so on (you need scroll 1 page down) ...
http://3web.dkm.cz/myie2cz/tas/7ziphelp.htm

[PauloMorfeo]

... .gz2 ...

You probably mean bz2 ? ...

Probably
Something was sounding wrong to me...
Yes, maybe more suited to the Internet where maximum compression is the goal.

Dwarden:
No matter how good 7-zip might be, if it proves to be too dificult to implement, we should look for something else. But this is something for people developing to devide.

[Xon]

How about an opensource Zip, GZip, Tar and BZip2 library implementation for C#
linky?

It is under a slightly modified GPL licensed too:

The library is released under the GPL with the following exception:

Linking this library statically or dynamically with other modules is making a combined work based on this library. Thus, the terms and conditions of the GNU General Public License cover the whole combination.

As a special exception, the copyright holders of this library give you permission to link this library with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you also meet, for each linked independent module, the terms and conditions of the license of that module. An independent module is a module which is not derived from or based on this library. If you modify this library, you may extend this exception to your version of the library, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version.

[Dwarden]

http://oleg.wl500g.info/
http://oleg.wl500g.info/lzma/

this is one of used implementation of 7z SDK and LZMA on linux ...
maybe some of You will find this usefull ...