Page 1 of 1
Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 15:27
by AF
Currently we have a tonne of stuff that goes into a program and out comes a complete map.
Instead what if we had a compiler that partially compiled a map, so we'd have a blank black map completely level that e 'inserted' or compiled portions into it? This could allow tweaks of existing maps or for huge maps built from lots of little textures, and would allow extensions added to the edges of maps without decompiling, resizing and recompiling.
This kind of compiler would also allow the completion of a full map compiler that was parallelized, aka distributed/multithreaded, allowing much faster compilation times.
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 16:49
by zwzsg
I lay more hope into lurker's promise of adding Lua hook to pick tiles by index, than to your idea, because it would allow ingame redoing of map.
But ultimatly, both idea are pointless since 95% of mappers just feed mapconv with a huge rendered pictures, disregarding the tile system entirely, following the lines of though that hard drive are cheap enough that releasing another 70Mb file for a new DSD tweak is considered the way to go.
Have you realised that for instance since 60b1 (four years ago), the .smt can be shared between maps, and as far as I know this ability was used only once!
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 17:28
by AF
Thats because ti mismatches with what techniques are used, but such a compiler could parallelize compilation of a single image by spawning subprocesses for individual sections.
It would also allow mappers to break images that are too large into smaller images and do them bit by bit. Not to mention the possibility of pause and resume.
It also means for those hand drawing maps that if they want to change a small section of the map they only have to rebuild a small sectionr atehr than feeding in a giant file thats just been modified, they feed in just the bit that needs changing.
Also while lurkers tile lua is useful it has an overhead which would be huge for manually setting individual tiles for an entire 60x60 map.
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 19:13
by lurker
AF wrote:overhead
Lalala, punch some numbers, 16 tiles per 512 unit, 960 tiles each way, a million calls to spring, grab some comparisons, maybe one minute max to change every tile on the entire map? And that's with a poorly-designed call.
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 21:02
by AF
Loading times could increase by as much as a minute?!
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 22:04
by Forboding Angel
zwzsg wrote:I lay more hope into lurker's promise of adding Lua hook to pick tiles by index, than to your idea, because it would allow ingame redoing of map.
But ultimatly, both idea are pointless since 95% of mappers just feed mapconv with a huge rendered pictures, disregarding the tile system entirely, following the lines of though that hard drive are cheap enough that releasing another 70Mb file for a new DSD tweak is considered the way to go.
Have you realised that for instance since 60b1 (four years ago), the .smt can be shared between maps, and as far as I know this ability was used only once!
I used the smt sharing extensively until the spring devs broke multiple maps in a single archive, which makes smt sharing worthless, because the lobby won't parse the maps anymore.
Furthermore, sm3 was our big hope for splattering remember? Smoth, lathan, argh and I went to some significant trouble to prove that it was viable and to point out the issues with it. JC decided not to fix it and that was that. Z I would love a splatting system, but we don't have it at the moment. Yes our mapping system is ridiculous, but people are trying to make the best of it.
And as pointed out before, the tiling system in mapconv doesn't really work the way you think it does, well not really anyway. You could easily tiles stuff in photoshop for a map, but it would looks pretty bad in most cases, not all, but most, especially for organic terrain.
When I was playing around with small divide if you remember I tried exactly that, tiling images in photoshop, the synopsis was that it kinda worked, but it was also so bland that it couldn't really be taken seriously.
Re: Request: Incremental Compiler and merger
Posted: 20 Apr 2009, 23:18
by TradeMark
I'm making some map compiler which shouldnt take more memory than what you want it to take, and wouldnt need to recompile the whole map if you changed one thing in it (metalmap, waterlevel, etc).
At the moment i cant even make bigger maps than 22x22, which why this new compiler is needed.
I also hate how slow the current map compiler is o_o
Whats the point first splitting the big BMP into smaller BMP parts, then saving to HDD, then converting those BMP files into DDS, saving to HDD and removing the BMP's, then resizing the DDS'es and saving the data into SMT >_>
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 01:35
by prodigy
AF wrote:Currently we have a tonne of stuff that goes into a program and out comes a complete map.
Instead what if we had a compiler that partially compiled a map, so we'd have a blank black map completely level that e 'inserted' or compiled portions into it? This could allow tweaks of existing maps or for huge maps built from lots of little textures, and would allow extensions added to the edges of maps without decompiling, resizing and recompiling.
This kind of compiler would also allow the completion of a full map compiler that was parallelized, aka distributed/multithreaded, allowing much faster compilation times.
I haven't got the map data that you are compiling (maybe send it to me?). But this is what I have found so far when it comes to mapconv and performance.
In my mapconv tests, the time consuming part of map compilation is when the texture is compressed using the DXT1 algorithm. Basically the big texture is split into 32x32 pixel tiles which are then compressed by DXT1, into 4 different sublevels, 32x32, 16x16, 8x8, 4x4. DXT1 decompression is fast, but the compression is slow.
SSE2 makes a big speed improvement when the DXT1 compression algorithm can use this. libsquish can be compiled with SSE2 extensions, and this reduces the time by several orders of magnitude for me. Unfortunately I only have working Linux SSE2 builds for my compiler so far. The MingW cross compiler I use produces compression code that faults on windows (and wine) when SSE2 is enabled(edit: I did a build on a windows VM and it works fine in v0.2). I don't know if the nvdxt.exe program uses SSE2 (edit: in fact Nvidias texture tools are based on Simon Browns Squish library).
There is another aspect to compression in a SMF/SMT, which is useful to know about, because it is adjustable, and is probably the time bottleneck for large maps. It is basically duplicate tile removal. If there are two (or more) tiles on the map that have identical or similar pixels, only one 32x32 tile will be put in the SMT, and this single tile referenced two (or more) times by the SMF. This is where the compression variable used by mapconv comes into play, it affects how tiles are compared to determine if they are identical or similar, by scaling the mean color values in the comparision, and the border area of the tiles to compare. The more tiles to compare, the slower the process.
Here's an example of the duplicate tile removal, this map is imaginary, if you have one tile of water surrounded by tiles of land:
Code: Select all
SMT (the 32x32 pixel tiles, each one 680 bytes):
1 2
+-------+------+
| water | land |
+-------+------+
SMF (just numbers referencing the SMT, each number 4 bytes)
+---+---+---+
| 2 | 2 | 2 |
+---+---+---+
| 2 | 1 | 2 |
+---+---+---+
| 2 | 2 | 2 |
+---+---+---+
In this imaginary case, without the duplicate tile removal, the SMT would be 9x680=6120 bytes. With duplicate tile removal, the SMT would be 2x680=1360 bytes.
Back to DXT1: it would be possible to send some of the tiles to different cpus to be compressed by DXT1.
However what seems to crucial to your idea is that to perform the tile comparison to remove the duplicates, all of the tiles need to be in memory at the same time.
I can try experimenting with threading the DXT1 compression phase, using two threads, and see how it goes.
As an aside, It would be straightforward to add an option to edit an existing SMF file, say for example to replace just the height map or metal map without touching the texture.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 01:39
by lurker
AF wrote:Loading times could increase by as much as a minute?!
D00d, I'm talking about rebuilding a map when
devving it.
But that's with a horrible algorithm. Just have lua pass in a table that gets stored in the tile mapping and you can do this in a fraction of a second.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 01:44
by zwzsg
AF wrote:Loading times could increase by as much as a minute?!
Look, I made a map that redo its entire heightmap at startup, and the heightmap is 4x4 more detailed than the tileindexmap (assuming 32x32 tiles). You can try that map, the loading time increase isn't even felt.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 13:49
by AF
D00d, I'm talking about rebuilding a map when devving it.
But that's with a horrible algorithm. Just have lua pass in a table that gets stored in the tile mapping and you can do this in a fraction of a second.
hmm interesting
Look, I made a map that redo its entire heightmap at startup, and the heightmap is 4x4 more detailed than the tileindexmap (assuming 32x32 tiles). You can try that map, the loading time increase isn't even felt.
Heightmap is not the same, it is an unfair comparison
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 14:02
by zwzsg
Unfair in the sense tile-index-map would be way faster, yes.
There are 16 time less tiles index to be changed than height points, and there is no need to repathing, apply damage, etc, since it's purely visual.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 14:58
by Beherith
Hacking sm2 is so much worse than what we could have with sm3
Prodigy:
Your compiler takes more than 20x more time than mapconv for me.
And still the nvdxt is the slow part.
Who cares about memory use for compiling? I never ever found it to be an issue with even just 1gb ram.
Trademark: mapconv has -q, although it seems to be broken.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 15:13
by TradeMark
Beherith wrote:Who cares about memory use for compiling? I never ever found it to be an issue with even just 1gb ram.
Trademark: mapconv has -q, although it seems to be broken.
i had 3 gigs of memory, but mapconv failed in opening my 700 meg BMP, iDk why. just said not enough memory or something.
i will still make my own compiler now when i am familiar with the map format, it will become much more faster too, since it wont write on the HDD anything, except the output SMT in the end.
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 15:17
by AF
zwzsg wrote:Unfair in the sense tile-index-map would be way faster, yes.
There are 16 time less tiles index to be changed than height points, and there is no need to repathing, apply damage, etc, since it's purely visual.
I was actually referring to mismatching mini-maps in the lobby
Re: Request: Incremental Compiler and merger
Posted: 21 Apr 2009, 15:45
by Beherith
Btw, the new CUDA nvcompress does an 8*8 map in 9 seconds for me, too bad they dropped proper command line and diff input file format support.
Trade:
known bug, save the 700mb file as a max quality jpeg.
Re: Request: Incremental Compiler and merger
Posted: 22 Apr 2009, 02:44
by Argh
There is another aspect to compression in a SMF/SMT, which is useful to know about, because it is adjustable, and is probably the time bottleneck for large maps. It is basically duplicate tile removal. If there are two (or more) tiles on the map that have identical or similar pixels, only one 32x32 tile will be put in the SMT, and this single tile referenced two (or more) times by the SMF. This is where the compression variable used by mapconv comes into play, it affects how tiles are compared to determine if they are identical or similar, by scaling the mean color values in the comparision, and the border area of the tiles to compare. The more tiles to compare, the slower the process.
Meh... if the tile-comparision algorithm worked better, maps could be a lot smaller and probably also run a lot faster. Compile times would go up, but it's irrelevant so long as the final results are OK. The preview you can see in HeightMapTweak or World Machine is scaled almost exactly the same as what you're going to see in Spring.
Except that
won't happen, because really decent maps need baked-in shading (either done in the bitmaps themselves, like SM2, or during pre-game, like the most modern engines are doing), which throws all this tile shit out the window. I can't see making any maps that don't have extensive use of baking, so long as I continue to use SM2. It just looks completely unprofessional without it.
Other than minor manipulation like doing an "animation" for a dynamic heightmap toy, like walls that move when objects approach them, or something, this is all pretty much just shuffling deck-chairs on the Titanic, guys.
Basically, I think that SM2 is a complete dead-end, and I wish that somebody would sit down with what I proposed as SM4 and see if they can get it to a prototype stage, since there was some general agreement that it might actually work.
Re: Request: Incremental Compiler and merger
Posted: 22 Apr 2009, 05:52
by prodigy
Argh wrote:I wish that somebody would sit down with what I proposed as SM4 and see if they can get it to a prototype stage, since there was some general agreement that it might actually work.
Link to your SM4 spec please?
Re: Request: Incremental Compiler and merger
Posted: 22 Apr 2009, 05:58
by Argh
http://springrts.com/phpbb/viewtopic.php?f=12&t=18421
Read all of it, we hashed out many things, so the early part, where I was just presenting the idea, is pretty rough. I think that the basic idea is solid, though, and the shader programs for doing the splats already exists in Spring, or could be ported to GLSL fairly easily (normalmaps would be harder, baked lighting even harder, but that's all getting well ahead of things- the first thing is to see whether the basic concept works well enough to bother).