mapconv - fixing for linux, and other things

[Beherith]

Awesome :)

[enetheru]

Ok, so while I was working on a new remake, of an old map. I really really want to be able to create an smt file using multiple 1024x1024 images rather than one gigantic one.

Because I render 1024 tiles in blender and stitch them together after the fact...

so I was thinking to myself why stitch them? its a lot easier to deal with smaller 1024x1024 images.

But working on the smt stuff in mapconv started to annoy me, so I separated out the smt stuff into a seperate exe, tileconv.

At the moment I only have decompling of the smt working. since compiling works ok(sort of) in mapconv I didnt want to duplicate functionality right away. I can copy paste that soon.

So, the way it works is that it will create a file with dimensions x=y=ceil(sqrt(numTiles)) and chuck all the tiles into that.

Or you can provide a tile index and it will spit out whatever the tileindex represents.

As always, I welcome any issue reporting, feature requests, whatever
https://github.com/enetheru/MapConv

[enetheru]

I have been pretty much working solidly on this for the past few weeks.

No new functionality to report, I've just been re-arranging the source to use a helper class for the functions. it makes things a little easier to chop and change.

for instance, decompiling only one part of the smf, or replacing a part of the smf.

its still a little messy atm so I haven't yet committed the changes.

[enetheru]

Still working on this.

I keep discovering new things, or changing my mind on design so I go back and re-code a bunch of it over and over lol. It's fun so whatevs.

I feel though that I'm starting to stabilise around a set of methods that do what I want.

OpenImageIO is a godsend
https://sites.google.com/site/openimageio/home

I'll have more updates later.

[enetheru]

anyway, so this post is because I have to export from my brain to paper and bounce off of the process. so ignore it if you want or whatever.

I'm trying to think about how I can incorporate the use cases I want to generate the tilefile.

So far the current status quo is to use one large image. I'm not happy with that. I want to use multiple smaller images.

I want to take into consideration that the tiles it generates may not need to be compared to any others for optimisation, for instance if you have a tile file already as an image with a tilemap and dont need to optimise.

I want to be able to paste/burn decals into the tiles, for metal, gas, features etc. it will make things a little easier for less complex setups.

I've been wracking my brain on how to achieve the various use cases optimally, for instance, if you bake decals, what happens if it falls on a tile boundary, or an image boundary. blah blah blah..

so I've come to the conclusion that I will pre-process the inputs to become one super large image, perform my baking/painting on that, and then use that to generate the tile files.

image boundaries dissapear, decal painting becomes easy.
future modification becomes easy.

Also. I've given up on any sort of windows support. frankly I don't care to spend my brain cycles on thinking about it.. I already spent hours today looking at setting up a cross compile toolchain.
That being said i would like it to run on windows, just that I don't want to spend any of my own resources on making it happen. patches welcome.

@behe, as of the current version of openimageio it supports image hash, numerical comparisons, and also feature comparisons.. so the options to make good optimization are at my disposal. :D

[enetheru]

I have inputting multiple source images as large tiles working correctly now.

so I can render my super large images as 2048x2048 tiles and feed them into tileconv to spit out the smt.

I'm working on decal painting now, but running into small problems I will manage.

Then I will work on tile matching.

I've got three four options to work with.
* no compression
* hash generation for exact tile detection
* numerical comparison
* Hector Yee's visual perception metric http://pdiff.sourceforge.net/metric.html

[enetheru]

ok, so i really dont know if anyone reads this anymore, but.

I implemented the most naive tile matching usig numeric comparison of pixel difference values. and lets just say it will take forever to match every tile against every tile.

so.. any computer scientists in the house who would like to point me to a nice way to reduce the search space?

my naive thinking is to get the average color, ie the last mip, and use it as buckets.. still means we have 65536 buckets to choose from.

or to dynamically split the buckets as more nodes get added, or whatever?

I'm looking for a direction to head.

[Beherith]

be greedy: dont do a full similarity matrix, just compare 1 to all, and merge it with its closest tile (if applicable), update distances for replaced tile.

[enetheru]

be greedy: dont do a full similarity matrix, just compare 1 to all, and merge it with its closest tile (if applicable), update distances for replaced tile.

not sure what you mean. "just compare 1 to all" ?

[gajop]

I'm not sure what you're doing here.
You could use Gaussian blur/filter if you want to reduce detail.
If you are doing some sort of tile grouping, you could try using a hashmap: value -> list structure.

[enetheru]

I'm not sure what you're doing here.
You could use Gaussian blur/filter if you want to reduce detail.
If you are doing some sort of tile grouping, you could try using a hashmap: value -> list structure.

for exact comparisons I use a hash list and compare, that doesn't take much time really. I've done a 28x28 map using this method and its fine.

its just that comparing the contents of the image with another image takes more time, so I cant compare each tile against all previous tiles, it just takes too long. for a 28x28 map there are 200 704 tiles.. that's 40 282 095 616 comparisons for a worst case scenario.. and remember i want to be able to build maps that are 256x256(for lulz, honestly they don't work in game, the heightmap and tile map are just too large.) but 16x256 is doable.

So I'm leaning towards using a method that splits the search area into bins. 65536 bins, one for each colour.

getting the average pixel value for the tile is rather easy. and that would determine which bin it goes into.

when I do a compare, I check the bin that it belongs in for tiles and compare against only them, and perhaps +-n bins.

The +-n values would also be the maximum allowed variation in pixel colour when doing the actual comparison.

I'm working on tile comparison because the image pasting code is waiting on upstream to fix something.

[enetheru]

because I suck at programming I had to scrap my big plans..

it compares numerically the 32 previous tiles for a match, and if a single pixel difference exceeds the threshhold(defined on the cmd line) then it is considered unique.

its fast and it works.

I do plan on giving more tweaking options soon. I can do total errors threshold, max single error threshhold, average error threshhold , per channel etc(not that I think that's useful).

I have to wait for OpenImageIO 3.0 to be released to use the perceptual metric comparison functions. it also gives me the image pasting

only image pasting to go before feature parity with existing mapconv is met.

I have a bunch of technical debt I can clean up too, so I'll do that in the meantime.

and of course documentation. wiki etc etc.

[knorke]

ok, so i really dont know if anyone reads this anymore, but.

I read it but don't have anything to contribute..
Just cool that somebody is working on mapconv, think for a while it was one of those "yes nice but nobody is going to do it" things. so good luck :)

[enetheru]

ok, so i really dont know if anyone reads this anymore, but.

I read it but don't have anything to contribute..
Just cool that somebody is working on mapconv, think for a while it was one of those "yes nice but nobody is going to do it" things. so good luck :)

Thanks.

So I've done a lot of work recently, broken many things, refactored the code a ton hence the breakage. the simple case works again but i'm concentrating on more ambitious goals so the state of the code is in flux.

for the last whole three days Ive been attempting to get a windows toolchain working with msys variants, and today I finally had a breakthrough although its only most of the way there.

wrote up instructions here:
https://github.com/enetheru/MapConv/wik ... with-msys2

to overcome the problems I'm probably going to replace nvtt with squish

After messing about with this for a while I've come to the conclusion that smf/smt is a complete waste of everyone's time and should be fazed out of existance in favour of using dds/tiff/exr images directly.

32x32 pixel tiles are completely bollocks, the engine doesn't even use them it stitches them together into 1024x1024 tiles anyway so lets just load lots of 1024x1024 dds images.
Tiff supports UINT16 and is loadable/savable by many applications.
exr supports UINT for the tilemap.

but with 1024x1024 tiles, which relates to a 2x2 map, a 64x64 map is only 1024 tiles large, you could express the tilemap in a text file without much effort.

I would like to find out how the texture sizes are mapped to the ROAM patches, whether its 1:1 or possibly some other method. I would like to make the tilesizes that spring generates/loads as small as possible because I see a future for tile based terrain. I still want to see maps that are super massive, and this is only a small part of the whole that I am touching, I realise there are many many other factors that go into this.

[jK]

I would like to find out how the texture sizes are mapped to the ROAM patches, whether its 1:1 or possibly some other method.

ROAM renders exactly as legacy renderer, each 1x1 map square is a single mesh with its own texture bound.

[enetheru]

I would like to find out how the texture sizes are mapped to the ROAM patches, whether its 1:1 or possibly some other method.

ROAM renders exactly as legacy renderer, each 1x1 map square is a single mesh with its own texture bound.

Ok, so that means that for each 1024x1024 image generated from the map files, 4 roam patches reference it.

512x512 is much better for my purposes anyway :D

[Anarchid]

After messing about with this for a while I've come to the conclusion that smf/smt is a complete waste of everyone's time and should be fazed out of existance in favour of using dds/tiff/exr images directly.

There was talk about this in #sy earlier, replacing smd/smt with direct images to remove dependency on ill-maintained and hard-to use compilers.

The time is ripe, it seems

Also: if you get rid of 16x16 pixel tiles, you will be my hero forever, because it will probably unbork VoidGround.

[enetheru]

Cancelling my windows plans for now, I cant get nvtt compiled and linking on windows, and I cant get libsquish to play nice with openimageio, a week hitting my head against a brick wall is definitely enough.

[Jools]

After messing about with this for a while I've come to the conclusion that smf/smt is a complete waste of everyone's time and should be fazed out of existance in favour of using dds/tiff/exr images directly.

There was talk about this in #sy earlier, replacing smd/smt with direct images to remove dependency on ill-maintained and hard-to use compilers.

The time is ripe, it seems

Also: if you get rid of 16x16 pixel tiles, you will be my hero forever, because it will probably unbork VoidGround.

I don't think that time will ever be ripe for something that will render most of the maps we play useless. Development has to be backwards compatible.

[enetheru]

I don't think that time will ever be ripe for something that will render most of the maps we play useless. Development has to be backwards compatible.

I'm aware that any changes need to be backwards compatible for any chance of success of inclusion.