HPI version 3 diagram - Page 2

HPI version 3 diagram

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

jouninkomiko
Posts: 436
Joined: 26 Aug 2004, 08:11

Post by jouninkomiko »

oh if you're talking about a front end to the program, don't do it in c++ :) i was mostly referring to the backend, but if your program does the trick... all that really needs to be done is add the functionality to CHpiHandler.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

Glad your on my side :-)

Now, any chance I could get dev access to the CVS source? >_<
jouninkomiko
Posts: 436
Joined: 26 Aug 2004, 08:11

Post by jouninkomiko »

better ask SJ or Fnord about that... right now though, you can submit patch files which then upon testing can be applied to the tree
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

Hi. I heard about this project on linuxfr (http://linuxfr.org/2005/05/09/18885.html, in french) and I think I can help - at least coding, but maybe also porting it to linux.

I wonder why you need to design some kind of filesystem (the hpi file) when there are already filesystem on every computer. Wouldn't it be beter to use the already existing directory structure of every filesystem than building another (less bugs, less things to think of, better protability in case of different endianness, faster on filesystems with b-tree structures,...) ?
To make the tree easier to share, there are already existing tools like unix tar or any archiver you like that knows how to put all the files together and compress the bundle.

I might also help with anti-cheat system, as I coded one for GhostRecon a few years ago - I might post a subject about that.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

Okay, bottom line. A decision had to be made, it was made, and that's the decision going forward. The new format will allow multiple compression schema's in one file, TAR is good for text, SevenZip is GREAT for images, and Zip is just the defacto standard.

You can't code with so much indecision out there, so I made one.

See my Spring Construction Set Update post for more info.
User avatar
PauloMorfeo
Posts: 2004
Joined: 15 Dec 2004, 20:53

Post by PauloMorfeo »

Subdino wrote:...
I wonder why you need to design some kind of filesystem (the hpi file) when there are already filesystem on every computer.
...
Are you talking of «filesystem» as in EXT3 / ReiserFS / NTFS / Fat32?

I think that that is not what's beeing discussed here!? They are planning on a new format of file that stores the game data.
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

Buggi wrote:You can't code with so much indecision out there, so I made one.
Right. But I think it's never too late to talk about such decision.
More than replacing the HPI3 format, I wa more thinking of adding the "host filesystem" I talk about to give it a try. I don't think it requires much work, and wouldn't bloat the code.

BTW, why is a compression needed ? Are the files that big that they can't fit in today's that-much-gigabytes hard disk drives ? Of course, for distribution, compression is a good thing. But when it comes to day-to-day file use, doesn't it increase the time needed to load a file and increase the ram needed to keep the inflated copy - or maybe even swap/temp space, which slows down things a bit more ?
Buggi wrote:See my Spring Construction Set Update post for more info.
Nice work.
The only work of that kind that would be required using the "host filesystem" method would be to start user's favorite file browser and start editing files with user's favorite png/pcx/gaf/text/mesh stand-alone editor. The "convert" button would be changed to an "extract from HPI" button, only knowing about the old HPI format (and maybe knowing the new HPI too).

I'm more and more adept of the "productive laziness", the "rule of reuse", and of the unix point of view about computer tools (small tools that just do their job right). It seems that you do the same way, if I'm right thinking the mesh viewer is distinct from the HPI editor.
Last edited by Subdino on 15 May 2005, 17:55, edited 1 time in total.
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

PauloMorfeo wrote:Are you talking of «filesystem» as in EXT3 / ReiserFS / NTFS / Fat32?
I think that that is not what's beeing discussed here!? They are planning on a new format of file that stores the game data.
Isn't it a filesystem ?
Folder/subfolder structure, file separation, file names, file attributes...
That is what I call a filesystem.

When you put files in a partition, formated with a filesystem, you make multiple elements fit in one bigger. When you put multiple elements in a file, that's the exact same thing. And when it is overkill to store - say - each line of a text in a different file, I think it is not to store each... file... in a different file.
Databases are also filesystems (huge tablespaces containing individual tuples).

And because designing a filesystem is far from easy, and could have bnefits of what is done for databases, I think handling file bundles should be done by the filesystem. When the filesystem is improved, Springs gets its part of the improvement without having to be rebuilt.

The file system described here is good for handling read-only files, but what if a file gets changed ? A whole HPI needs to be rebuilt. That wouldn't happen with individual files.

[EDIT]Removed the ReiserFS things, wasn't satisfying.[/EDIT]
Last edited by Subdino on 15 May 2005, 21:02, edited 2 times in total.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

You've got to be kidding me. Why is compression needed?

Not everyone has multi gig's free. Just like not everyone has 2 gigs of ram, not even me :)

Oh, as for "productive laziness". Not my style. I do things the right way not the easy way. All tools will be merged eventually into the construction set. Including the map editor.

The convert button will be a nice feature to convert old Version 1 HPI's into HPI3 files. Hence the new extensions. .sdu and .sd3 are the only two file extensions it'll convert too. SD3 is strictly HPI3 (patent pending :P )
"Spring Definition Version 3" basically.

As such, the files embeded inside that format will also be unique. I'll be posting a outline of the directory structure soon. I also have to include something inside the file for my webservice to read so it can archive the files and create a database list of the files, their specs, pics, and info.

Welcome to the 21st century. :D

-Buggi
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

Buggi wrote:Not everyone has multi gig's free. Just like not everyone has 2 gigs of ram, not even me :)
So you should be kidding too.
Do you read/parse files when they are compressed ? No. So where are they uncompresed ? In ram, or stored on disk temporarily. So they take more space, because there can be found somewhere compressed and somewhere else inflated - not for a long time, but it is at run time, when resources are more precious than any other time.
Buggi wrote:Oh, as for "productive laziness". Not my style. I do things the right way not the easy way.
You should think you code better than Mrs Kerninghan & Ritchie [EDIT]not to mention all the linux, bsd, hurd contributors and all the guys who worked on earlier unix OSes[/EDIT], so. For memories, those guys invented the language you use to code. [EDIT]BTW, Ritchie also invented unix, for which C was invented[/EDIT] Come on, have a look at "The art of Unix Programming", you'll learn tons of things that can apply everywhere, even under windows. I haven't finished it yet, and I'm convinced they do not only the right thing, but they do the thing right.
Buggi wrote:All tools will be merged eventually into the construction set. Including the map editor.
Merged ? You mean they'll all fit in one fat .exe file ? I hope not, that the most horrible thing that could happen to an open-source project to have bloated source code. I hope you mean having many .exe in one package, each one beeing independant from the other. Or I think you'll run to the catastrophe if you wan some help from hackers (in the right meaning of the term, not in the fashion one).
Buggi wrote:SD3 is strictly HPI3 (patent pending :P ) "Spring Definition Version 3" basically.
There are some problems with such file formats when it comes to portability, like byte order, value lengths... Wouldn't it be good to have a universal format (which is, each file by itself) ?
Buggi wrote:As such, the files embeded inside that format will also be unique. I'll be posting a outline of the directory structure soon. I also have to include something inside the file for my webservice to read so it can archive the files and create a database list of the files, their specs, pics, and info.
Same remark.
[EDIT]erm, almost the same remark. I think you know what you are doing inside the files. What I think is not the best way is the way you pack them.[/EDIT]
Buggi wrote:Welcome to the 21st century. :D
Welcome to the good ol' computer world where every coder think they do better than the guy that came before them - if he even know he existed. And doing so he does always the same thing : inventing the wheel, making it square to be easier to pile up.</cynic>
Last edited by Subdino on 15 May 2005, 21:05, edited 3 times in total.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

Lets see some code man.

I'm anxious to learn from the best.
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

I'm not the best. I'm really, really, far from that.
What I do is that I recognise my masters, which are all those contributors I mentioned earlier (in an EDIT) and in some people (more precisely R. M. Stallmann, Y. Okuji, L. Torvalds, to name a really few of them). And I believe - and understand - what they told me, which is : use what exists, don't bother yourself writing things that mimic the existing - if it is not to give a direct alternative. Use modularity where possible (the most common example is paging programs like "more" and "less" : don't write your own pager in your own program, provide output to the existing pager so the user can choose which one to use).

The parallel I see in HPI is : use the OS filesystem to glue files together, that will save tons of work designing tools to read & write the filesystem - and once [not] written, it will save time maintaining the code and porting it to other platforms. Use separate programs to edit files, don't write your own image/text editor. Give the user the choice of those tools.

I'm not here to get any personal consideration, I'm here to try to give some help.
I can't help if I have to put sugar in my sentences.
So I tell sincerely where I think there are problems, because it's the first step to a discution.
And as I have some experience with code portability (from x86 to usparc, so from little endian 32 bits to big endian 64 bits) I think I can help putting big flashy warnings where I have already had problems.

[EDIT]BTW : "The Art of Unix Programming". If you have a palm or something like that, you can also read it with Plucker[/EDIT]
Dwarden
Posts: 278
Joined: 25 Feb 2005, 03:21

Post by Dwarden »

example:
so i have 300 maps, each got after complete extraction 200MB that gives like 58.6GB of space used ...

now ... these 300maps, each compressed into "such" HPI3 file is only 10MB size big ... that gives 2.9GB of space used ...

so after i start game with some i get like 3GB used for maps including temporary extracted map ...

that gives me around 55GB saved space ...

and did i mentioned there are some thousands TA maps?

,,,,

now subdino ... can you please explain more Your last post ? :)
interrupt
Posts: 5
Joined: 10 May 2005, 06:13

Post by interrupt »

Only when I have a 1 gigabyte/sec fiber line to the internet plugged directly into my house will I say that compression is not needed. But at that time, files will be so huge that we'll still need it :)
jouninkomiko
Posts: 436
Joined: 26 Aug 2004, 08:11

Post by jouninkomiko »

dino, while your point is valid, itll be in memory either way be it in compressed or uncompressed form. the difference is how much is taken overall... decompression is a very fast process, and if the archive is not solid, you can simply decompress the bit you need, sparing a ton of hard drive space until the information is sorely needed. if the decompression used more resource and hurt efficiency to a noticable degree, i'd have to agree with you
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

Dwarden wrote:example:
so i have 300 maps, each got after complete extraction 200MB that gives like 58.6GB of space used ...
That's what I didn't knew, some exact numbers. Of course, if the compression ratio is that high, and if there are few things that needs to be inflated at runtime, it is interesting to use compression.
interrupt wrote:Only when I have a 1 gigabyte/sec fiber
Read again, I said that for data exchange compression is needed. Not only to save bandwith, but also on any archive medium.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

Compression ratio's are INSANE now..

Example...

When creating my Asia Pacific map I was using 3 BMP's each at 512 megs, the texture, height, and metal maps. Now that's 1.5 gig's of BMP all over my HD.

I wanted to test SevenZip (7z) and while it took 45 seconds to compress the 3 files, they compressed to 16 meg.

1.5 gig -> 16 meg.

Do the math for that. that's over 99.9% compression!!

I wish I could pack cloths that well. I wouldn't need a suitcase, just a ziplock bag :D

-Buggi
renrutal
Posts: 84
Joined: 28 Apr 2005, 16:45

Post by renrutal »

dino, just because we are defining some packaging standard for Spring, it doesn't mean it will be the only one to be used. I'm sure we will be able put the files directly into the folders, w/o any compression, and Spring will load them. It's a great option when you are developing and debugging the data to be used in game.

I'm completely with Buggi when it comes to get the work done in any exotic languages than not getting anything done. Later we will solve the merging problem. It's easier to write(or rewrite) the code with the general idea already in mind.
Subdino
Posts: 25
Joined: 15 May 2005, 08:44

Post by Subdino »

renrutal wrote:put the files directly into the folders, w/o any compression
I think I should have separated more explicitly the 2 things :
-Is it pertinent to build a new file format ?
-Is it pertinent to compress files ?

The second answer is, as far as I can see, fixed to "yes". (BTW, Buggi, wouldn't PNG get an even better level of compression than 7zip ? The big advantage of PNG on JPG is that it doesn't loose any data information, and on GIF is that it isn't limited to a 255 colors palette)

If the first answer is "there is a new file format, but we can also use multiples backends, such as a bare filesystem use", it will be perfect to me.
User avatar
PauloMorfeo
Posts: 2004
Joined: 15 Dec 2004, 20:53

Post by PauloMorfeo »

Subdino wrote:
PauloMorfeo wrote:Are you talking of «filesystem» as in EXT3 / ReiserFS / NTFS / Fat32?
I think that that is not what's beeing discussed here!? They are planning on a new format of file that stores the game data.
Isn't it a filesystem ?
Folder/subfolder structure, file separation, file names, file attributes...
That is what I call a filesystem.
...
:-) Maybe. I never knew what an OS file system really is and how it is diferent from accessing a compressed file.

Subdino:
About compression, the units set shipped with TA::Spring, XTA, is stored in the file taenheter.ccx.
It has 3712 files. They only have 23,1 MBs of size but, at least on my Fat32 file system, it takes 119MBs out of my HDD. In, for example ReiserFS which takes extremely good care of small files, this wouldn't be so much diferent but in most file systems, it is. While compressed, it takes 12,2 MBs.
Post Reply

Return to “Engine”