Suggestions: New Map Format, Mod Management

Suggestions: New Map Format, Mod Management

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

Gnomre
Imperial Winter Developer
Posts: 1754
Joined: 06 Feb 2005, 13:42

Suggestions: New Map Format, Mod Management

Post by Gnomre »

(Let me first note: Sorry if this is in the wrong forum, move it if necessary. This one just seemed most logical to me.)

1. New map format
Right now, Spring uses a horrible horrible map format. Giant texture maps are NOT the way to go in a full 3D game, which is why you don't see that method in any other modern games. Right now I have a mere 8 maps in my Spring maps folder, and they are using 160 MB of space. THIS IS MORE THAN ALL OF THE MAPS RELEASED IN THE CC EXPANSION COMBINED. Obviously something needs to be done. Now, I'm nowhere near good enough to code anything remotely close to what I am going to suggest and I understand the difficulty of rewriting that entire part of the engine, but I think it should be done now before even more maps are made.

Basically, I say the metal and heightmap should be kept. Those aren't harming much, and they compress well enough to not make a difference. However, instead of having a massive texture map to go along with them, which has tons of redundant data (especially between maps that share similar textures), I think a more modern approach should be taken. Have sets of textures, such as a set of ice textures, a set of green, etc. Some of these could be distributed with the game itself (as the maps it ships with need textures as well), but it'd also be possible for mapmakers to create their own.

The mapmaker would then go into his .smd file (or a future map editor which doesn't exist yet) and do something like this:

[TexturePackageName.xxx\TextureName]
Zone1=1023 113 8402 2753 55 218;

Now, let me explain the numbers. The first and second numbers would be coordinates on the map at the locations to start the texture. The third and fourth numbers would be the end of the texture, forming a box on the map where the texture is applied. The final two numbers are the height constraints, so the texture would only be applied inside the box between the heights of 55 and 218. Of course, more detail could be given in this format: A zone's prevelance over other zones (maybe give each zone a "ranking" between 0-255, the higher the number the more it will show, so when two zones occupy the same coordinates the one with the higher prevelance would be shown). You could also make height exceptions, such as in the example above, have a tag that makes it so the texture will *not* show between 60 and 100. That's just giving the mapper more power.

Basically, this way you would just have a few moderately large texture packages. Yes, they'd be a pain to download, but that's the beauty: They would only need downloaded once! If Spring shipped with a greenworld texture package, any third party maps using those textures would only need to have their heightmap, metal map, and smd file distributed! It'd drastically cut filesize, bandwidth bills, hard drive space used, and wait times for those will slower connections in the long run. This should also easily accomodate hosts sending maps to clients in the battleroom, assuming everyone already has the texture packages.

The only downsides to this I can think of right now are:
-Load times. I imagine it'd take longer to load and apply the textures like that than just loading a premade texture map.
-Gathering coordinates. It'd be a pain to do without a proper visual map editor.

2. Mod Management
I actually posted this before the forum hack, but I'll go ahead and do so again. For this, I basically suggest taking a UT2004 style approach: each mod gets its own sub directory in the Spring folder. ALL of the mod's files will be stored in its directory. Each mod will ship with an ini file. This ini file will tell spring the mod name and the paths to look for files, such as this:

Code: Select all

[Mod]
ModName=Uberhack;
ModDescription=Lemmings ahoy!;
ModAuthor=Brave Sir Robin;
ModSite=http://www.planetta.com/bsr/;
[Paths]
Maps=../Maps/; //Look in the main Spring map folder, this wouldn't be required, so mods could easily ship with their own maps
Maps=./Maps/; //Look in the Uberhack/Maps/ folder
Data1=../; //Load the stuff from the main spring dir first so it doesn't have to be redistributed
Data2=./; //Load from the mod's folder now, which has the relevant files that make Uberhack Uberhack
DataFileExtension1=.hpi;
DataFileExtension2=.ufo;
DataFileExtension3=.ccx;
DataFileExtension4=.gp3; //This would allow the mod maker to define his own file extensions, however they would still use the HPI format. The higher the number the filetype is, the greater prevelance it will take, much like in TA. however, the mod maker could define as many types as he wanted in any order he wanted.
Of course, many more tags could be added here, but that's the general idea.

Inside of Spring.exe, and in the BattleRoom client, the user would be given the option to choose a mod from a simple drop down menu or something. In the case of the battleroom, there would be a column to tell users looking to join which mod it is. A user could also make a shortcut to the mod, in the format:
C:\Games\Spring\spring.exe -mod=Uberhack
Which would immediately load the game with Uberhack rather than having to choose it in the menu. The code for the -mod line would simply be the filename of the ini file for the mod.

This would make mod installation and uninstallation a breeze; to install or uninstall, you simply extract to the spring folder or delete the mod's folder respectively. There would be no loose files anywhere else, each mod would only get its one folder, with the ability to reference data from the main Spring folders.

That's pretty much it. Hope you guys like the ideas.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

First I'd like to see wayyyyyyy more detail and thought put into your map format.

The reason they are so large right now is they're not in a HPI file and not compressed. Once I get the new HPI format up and running, and can work with the devs a little closer on it, this should help a lot.

If you find a way to cut out some image data while still maintaining sweetness, go for it. But you need to think about it awhile and plan it out. I'd like to see an actual example of what your talking about.

TA used "tiles" in their maps and it *severely* limited creativity, so if you can find a way around this, great!

Also, the engine is still in version 0.41b... :-)
Gnomre
Imperial Winter Developer
Posts: 1754
Joined: 06 Feb 2005, 13:42

Post by Gnomre »

Ever used Unreal Editor? Think about Terrain Editing there. Yes, UED has a fancy GUI and stuff that lets you paint textures onto the the terrain and shape it in a similar fashion, but it is in essence what I described: A simple heightmap with a list (stored within their map format) of coordinates to apply textures. Same idea here. It's kind of hard to explain, it's something you have to use yourself to fully grasp, I think... but once you do, you realize it is a FAR better system.

Oh, and btw, HPI won't make a difference for compression. You know that hoth map Storm released recently? Well, when I sent it to him, it was a 192 MB, 8192x8192 bitmap file. I tried compressing it with HPI (CC compression) to send it to him, 56 MB. I then did rar, 47 MB. As you can see for yourself, the final spring copy of the map is around 8 MB. Besides, the highest level of compression HPI uses is basically just zlib, which isn't all that great...
User avatar
Ace07
Posts: 348
Joined: 21 Apr 2005, 20:46

Post by Ace07 »

I agree with Gnome. Texture packs and then ways to mesh the textures together to create a sense of realism would be best. What I would do though:

Have various textures that can tile VERY well, and do it this way.

For each texture, make a greyscale PNG/BMP that describes where the texture goes. This file would be almost exactly the same as the heightmap and such. Then you could use your favorite image program to splash the texture across the map. Spring could then load the tiled textures and all the greyscale maps associated with them. I don't know how feasible this is however. It seams a lot more seemless then the block by block idea though. :P

(With image editors, you can create cool gradients and stuff)


EDIT: And yes, the whole mod thing needs a lot of work. This project needs better resource management. :P
User avatar
sp2danny72
Posts: 60
Joined: 09 Jan 2005, 04:52

tiles

Post by sp2danny72 »

It does not limit anything, if the map has 3 tilesets to choose from:
the 'spring default' the 'mod default' and the 'map specific', and if
tiles could be aritbrary sized and applied on the map.

Also, the ./Mod/<modname> idea is good, i've been thinking along
those lines too... This will allow for better consistency in multiplayer,
if the enginge checksum the mod used...
User avatar
[K.B.] Napalm Cobra
Posts: 1222
Joined: 16 Aug 2004, 06:15

Post by [K.B.] Napalm Cobra »

I dunno about the map system, it could increase loading times horrendeously for a bit of saved hd. That is if I'm interpretting what you're saying correctly, is it similar to what terrtexgen does?
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

I plan on attempting to impliment 7zip into the compression options. While it takes a good 30 seconds to compress a 20 meg file, it compresses to 8 meg and decompresses EXCEPTIONALLY fast, roughly 11meg/second.

Just one of the zip options that is going to be available in the next format.
User avatar
Ace07
Posts: 348
Joined: 21 Apr 2005, 20:46

Post by Ace07 »

Buggi wrote:I plan on attempting to impliment 7zip into the compression options. While it takes a good 30 seconds to compress a 20 meg file, it compresses to 8 meg and decompresses EXCEPTIONALLY fast, roughly 11meg/second.

Just one of the zip options that is going to be available in the next format.
What he was pointing out was that although 7zip is the best lossless compression mechanism, it still wasn't significantly better than Rar or the other top notch compression systems. 192 mb -> 56mb is fairly good actually. But good luck getting it any smaller than 40.

The differences between 7zip and bz2 + gz is a few percentage points. So, he was pointing out that a new architecture needs to be built so that maps aren't so flipping huge. I mean...even FPS's don't have 40 mb map files! A new compression system won't solve the problem that the map files are just too big currently.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

Yes, I realize that :P

I think I'll include Tar as well @.@;
Gnomre
Imperial Winter Developer
Posts: 1754
Joined: 06 Feb 2005, 13:42

Post by Gnomre »

[K.B.] Napalm Cobra wrote:I dunno about the map system, it could increase loading times horrendeously for a bit of saved hd. That is if I'm interpretting what you're saying correctly, is it similar to what terrtexgen does?
More along the lines of any terrain you see in FPS games, or the terrain in Dawn of War... games of that nature. They manage it well enough, I think Spring could in time as well. Yes, the load times would be longer for larger maps, but the same can be said in *any* game. Anyway, it's not something we'd need to see immediately; the current map format will suffice for a while. It does need changed before too much time and effort on the part of the mappers is wasted, though. But, as mentioned, we're not to 1.0 yet so there's still plenty of time :)
Sean Mirrsen
Posts: 578
Joined: 19 Aug 2004, 17:38

Post by Sean Mirrsen »

There's this Perimeter game (developed somewhere in the former USSR), that uses a different system of mapping for deformable landscape. I have to say, that system clearly isn't the best, but the things it does to terrain...
In short, it has a high-resolution heightmap, with a same-resolution texturemap. Relating to the current system, it could make like... quadruple heightmap size, and half the texturemap size. Since the heightmap is easily compressed (being greyscale), its size increase will bring less weight than reducing the texturemap will take away. In addition, the use of detail textures on the landscape, and, more importantly, custom detail textures, will mask the somewhat lower resolution of the texture.
On tiles: I think the terrain could have a more advanced system. Firstly, it would have a base color. Then, there would be an archive (or archives) of images that could be used by merely positioning them on the landscape. Using transparency on the images, you could create anything from spots on the ground (largeish spots :P), to some specific rock or cliff textures. And, finally, the terrain would have a 'damaged' texture. Or several textures to reflect how much damage an area suffered. These textures would be applied to disturbed soil, and some more could be applied at different depth of holes made in the landscape.

Wow, long post. I hope I didn't confuse anyone.
User avatar
Redfish
Posts: 289
Joined: 27 Feb 2005, 16:12

Post by Redfish »

I could make a smd editor. But i think it would be nicer to have these options set in the file format itself instead of a seperate file, and maybe it's nice to program stuff into the maps themselves, so we don't have to be dependent on what the Sy's put into it.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Post by AF »

Why dont you do what i originaly suggested a few weeks ago. Instead of a hgue bitmap we could ahve a second type fo map format that uses the same sort fo thgina s terragen. Afterrall a terragen world and heightmap isnt 100MB but it can stil provide a similair if nto better approximation of terrain in some cases. Of course this doesnt fit inw ith every type of world so the other types fo map discussed here would be needed.
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Post by zwzsg »

When making my CPIA port I noted that:
- If I resize the texture map to 8192x8192, the .sm2 weight 45MB and mapconv.exe spends an awfully long time creating about 50 tiles.
- If I crop the texture map to 8192x8192, the .sm2 weight 8MB and mapconv is much quicker and creates much less tiles.
- If I use what would be the .bmp of an empty flat TA metal map, the .sm2 weight 5MB and the compilation is hyperfast and mapconv tells me it creates only one tile.

I think that proves Spring map use tiles, and that the tiles have the same size or a multiple of TA tiles size. That mapmakers choose to use rendered style map, or to simply ignore any tiling and waste 18Mb on a simple metal map, is entirely another problem. Blame the mapmakers, note the S.Y. (Ok, the S.Y. could have told us there was a tiling and how to efficiently take advantage of it). For info, in TA you can also have 18Mb 10x10 rendered maps that look like crap, and 0.6 Mb 63x63 handmade maps that look beautiful.


As for mod management, I stick to my idea. Basically don't go create a complex system with lots of ini and custom format and new var all around the place, that would only work for one or two total conversion, but simply have a window in the lobby to disable or enable files, and force every other player in the lobby to use the same files, or download them if they don't have the same.

I mean, let's say I simply wants to add one new unit in your system, what do I do? I go create a new folder where I put all TA units + mine, then waste 15minutes trying to understand the syntax of the .ini file to create a new mod, then upload the whole as a 13Mb zip, then tell everyone to install it?

While in my system, I would simply drop the ufo in spring folder, and everyone would transparently get it. And transparently gets updated version as I make them.
Gnomre
Imperial Winter Developer
Posts: 1754
Joined: 06 Feb 2005, 13:42

Post by Gnomre »

Units and maps would just be drag and drop to the spring folder like you suggest. Only full mods (Uberhack, SWTA, AA, etc) would have their own folders and inis and stuff. The mod maker would have the duty of creating the ini and directory structure and everything for the end user to use. The end user would merely extract the mod's zip to the spring folder (or run a mod installer which extracts itself to the spring folder, or whatever), and BAM, he'd have that mod in his list of mods to play. And if he wanted to uninstall it, it's as simple as deleting one directory. Bam, gone, no mess, no compatibility issues, nothing, the mod is gone.
User avatar
sp2danny72
Posts: 60
Joined: 09 Jan 2005, 04:52

Post by sp2danny72 »

zwzsg wrote: that would only work for one or two total conversion
there wont be one or two total conversions, not if we do this right.
zwzsg wrote: I mean, let's say I simply wants to add one new unit in your system
, what do I do? I go create a new folder where I put all TA units + mine, then waste 15minutes trying to understand the syntax of the .ini file to create a new mod, then upload the whole as a 13Mb zip, then tell everyone to install it?
No, you put the unit-file in an othewhise emty folder. Anything spring
does not find in the /mod/xxx folder it loks for in /mod/default


this way we can have good support for both dropin addition units
AND complete mods
Last edited by sp2danny72 on 08 May 2005, 01:38, edited 1 time in total.
User avatar
Buggi
Posts: 875
Joined: 29 Apr 2005, 07:46

Post by Buggi »

I could integrate mod management into my Construction Set.

I just need a final or, nearly final, concept on how to do it.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Post by AF »

wtf sp2danny72, I never said that infact you've quoted more text from em in that post than I've said words in this entire thread.

In future blame zwzsg for zwzsg's posts and me for mine
User avatar
sp2danny72
Posts: 60
Joined: 09 Jan 2005, 04:52

oooops

Post by sp2danny72 »

ooops, sorry. my mistake. now corrected.
Gnomre
Imperial Winter Developer
Posts: 1754
Joined: 06 Feb 2005, 13:42

Post by Gnomre »

Buggi wrote:I could integrate mod management into my Construction Set.

I just need a final or, nearly final, concept on how to do it.
Mod Management

1. Directory Structure
See image. It's fairly self explanetory.

Image

2. INI File
The INI filename should be simple with as few characters as possible and no spaces in its name for use in shortcuts (see below). These are the tags I have thought to add. Read their comments.

ModName - A simple string, giving the name of the mod in the in-game menus. Spaces are allowed here, so you could put "Absolute Annihilation" or "Star Wars TA".
ModDescription - A description of what the mod is or does. For use in in game menus.
ModAuthor - An author name or mod team name or whatever
ModSite - A URL to the mod's site. Doesn't necessarily need to be clickable, just there
Maps1 - Where to look for maps to use for this mod. "./Maps/" is the Maps folder in the Mod's directory, "../Maps/" is the main Spring maps folder.
Maps2 - Used to define another place, in case you want to have a mod specific map folder as well as the ability to use main Spring maps. Possibly let it keep going as far as the modmaker wants, but I only see a need for two tags here.
Data1 - The path to load data from. Data1 will be read before Data2, so any settings in Data2 will override Data1. This would primarily be used for rebalancing mods such as Uberhack, which use most of the content that comes with spring as a base, but changes various little settings like weapon damage and whatnot. The path would be "../" for the default Spring folder, and "./Data/" for the mod's data files.
Data2= - Most likely "./Data/", unless the mod doesn't use *any* data that ships with spring at all.
DataFileExtension1 - Here's the fun part. The modmakers could define their own file extensions to read. Let's say you want your mod to use .uhd (uberhack data) or whatever by default. You can have it do that. The .uhd file would still be an HPI (or whatever compression method spring uses in the end), just with a different extension. Much like the Data1 and 2 sections above, these are read in order: DataFileExtensions2 settings will override DataFileExtensions1. However, a DataFileExtensions1 file in the Data2 directory will override a DataFileExtensions2 file in the Data1 directory. Feel free to reread that until it makes sense :)
DataFileExtension2
DataFileExtension3
DataFileExtension4 - And so on. I say give the modmakers an unlimited depth, or at least 8 or 16 extensions deep to work with. The more the better, though, since it would make it incredibly simple to release a patch. You would only need to distribute the patch's data file and the new INI to read it last.

Those are all the tags I can think of. More could be added, of course.

3. Shortcuts
This is simple. A user can bypass the menu to activate a mod by creating his own shortcut to it:

Image

As it shows, the user simply adds the commandline tag -mod= to his path, and he puts in the filename of the desired mod's INI file (minus the .ini extension? up to you guys there). He'd be on his own for an icon, but mod makers could distribute one of those as well if they wanted.
Post Reply

Return to “Engine”