smf_tools, alpha work

[enetheru]

Blabla Spring doesn't differ at all from your code: it takes the tiles, uses squish to decompress them and then uses rgetc to recompress them.

The difference:
* Spring takes all the tiles in the tiles vector(contiguous data) and recompresses it all in one go.

* smf_tools takes tiles of varying sizes(as defined by a tilemap, or generates one), splits them up or joins pieces of them together as needed to form tiles of the desired size. its not a straight forward process.

its designed to be flexible with input and output, makes the artists life easier even if it sacrifices some performance.

Checked your code, don't see any threading ...
Compression is ideal for threading & squish works fine with it. Nor does it increase memusage at all, neither does it make the code more complicated.

There isn't any that I've made. Have some compassion for a beginner coder.

[jK]

* smf_tools takes tiles of varying sizes(as defined by a tilemap, or generates one), splits them up or joins pieces of them together as needed to form tiles of the desired size. its not a straight forward process.

your code should consist of stages:
* check input and bring it in the wanted form (resize images etc.)
* split the groundtex into tiles & check for duplicates -> gen tilemap
* compress tiles
* assembly

steps (1) & (2) are easily threadable (cause they operate per tile and tiles don't share data that would need to be mutex locked)
and you should hard divide your code in such stages (don't mix them), cause of flexibility, KISS & maintability (as you said: sacrifice a bit performance for it).

[enetheru]

For what I'm doing that would take a lot longer.

The idea involved is pretty easy.
1. input a series of images, or smt's as a set of tiles in a cache(tilecache).
2. input a csv which we can use as a tilemap, or generate one with consecutive numbering.
2. define a tiledimage type that uses the tilecache and tilemap.
3. query the tiledimage for pixel regions
4. scale those pixel regions as necessary
5. output to either an image or an smt

some of my implementation is a bit messy due to me learning new concepts as i go. but the basic gist of it is fairly straight forward. in fact a lot of image types are tiled internally anyway.

[jK]

So you aren't checking for duplicates at all?
This destroys the whole idea behind smt ...
1. increases filesize _a lot_ 2. loadtimes are increased a lot 3. runtime performance decreases a lot

-> nobody should ever release such a created map

[enetheru]

So you aren't checking for duplicates at all?
This destroys the whole idea behind smt ...
1. increases filesize _a lot_ 2. loadtimes are increased a lot 3. runtime performance decreases a lot

-> nobody should ever release such a created map

clearly there is some misunderstanding here.

The idea of feeding in fully formed maps and duplicating data in the smt from duplicate tile references is very stupid as you have pointed out, i'm surprised you would attribute me of being this dense.

The design of smf_tools is to allow creating smt's of different tile size(read: more artist friendly tilesize). you would build the smt directly from unique tiles.

You use those tiles in a separate tool to create the map like tiled(previously linked) its a completely different workflow.

When I bring back duplicate tile detection then it will be useful for giant renders, but thats not the focus right this minute, after all we already have tools to do that, whats the point in writing a new one?

This is about tile based terrain, actual tile based terrain, not sub-optimal compression of giant renders.

I really wish i could either chat in person, or something. its kind of hard to explain the beauty of the whole situation, and how its fully backwards compatible, and opens up more options to explore.

if you have the time browse this video, its sort of where i'm headed with this:
https://www.youtube.com/watch?v=9cCbXJsWe5o

imagine fully ssmf ready tiles, which gives you a base to work from, full shading, height, typemap whatever.

load that up in gajop's editor or blender or whatever and customise the heightmap, or typemap or texture, and only save the differences in an additional smt and map unique portions.

you have a base tileset that acts as a module like springfeatures does, then you only keep the differences in your own map.

you would define tilesets as "worlds", and maps as unique places on that world..

its kind of a large project, but its totally doable, and doesnt break compatibility with any existing stuff, and lots of low hanging fruit in pieces that can be done incrementally. which is good for a guy like me with average coding skills. just takes time.

[enetheru]

I can totally see how its not clear based on the descriptions of whats going on and such.

input big render = smt and tilemap, with duplicate detection still to be written
input tiles = smt and discard the tilemap it outputs

input big render = split into smaller images and tilemap, with duplicate detection still to be written
input tiles and smt = reconstruct image, or re-cut image into tiles of different size.

all the operations shared so much in common that they can all be summed up with a single process.

[jK]

then you should emphasize that this is not a replacement for mapconvg and that people shouldn't release maps made by it.

[enetheru]

then you should emphasize that this is not a replacement for mapconvg and that people shouldn't release maps made by it.

Well it could be with a little more work, and depending on how you build your map it may already make maps with smaller file sizes, so I'm not going to tell anyone what to do, right tool for the right job and all that

[Anarchid]

then you should emphasize that this is not a replacement for mapconvg and that people shouldn't release maps made by it.

Except mapconvng fucks up alpha.

if you have the time browse this video, its sort of where i'm headed with this:
https://www.youtube.com/watch?v=9cCbXJsWe5o

I wonder if there's an easy path to integrate geometry cliff placement into this.

[enetheru]

if you have the time browse this video, its sort of where i'm headed with this:
https://www.youtube.com/watch?v=9cCbXJsWe5o

I wonder if there's an easy path to integrate geometry cliff placement into this.

thank you I can see that you understand where im headed.

[enetheru]

I've cleaned up the progress bar so it isn't so shitty. only updates every second now instead of every tile.
and doesn't flash anymore.

Also added image hash checking to not save duplicate tiles.
I might add a pixel difference threshold, or a perceptual difference threshold matcher in the future but for now its not a major goal.

the progress bar fix should avoid some work so might make it a slight bit faster but don't expect anything big.

[enetheru]

ok so..

i'm still working on making this better. I started messing about with profiling with valgrind and kcachegrind to see where all the cycles were being spent and the majority of it was on a function from one of my dependencies OpenImageIO::resize

so i figured out that i was calling way more often that needed, so i re-ordered some of the functions so that duplicate tile detection is called before scaling. which wont help most people since its rare to scale a map up or down.

and i'm using a variant OpenImageIO::resample which is a lot faster but only works when scaling down because scaling up causes black borders to appear

and so now I have a 14k^2 diffuse texture being converted to smt in ~1.5 minutes

Code: Select all

real	1m23.130s
user	1m22.748s
sys	0m2.719s

[enetheru]

I just noticed that core platform's smt is 50mb
using smt_convert it gets down to 6mb

compressed into the 7z archive there isnt much difference 20.3mb vs 19.8mb

do the currently being used archives get unzipped into ram and stick around?

[qray]

Just for completeness: I noticed the big file size, too. If compiled with windows mapconv the smt is very big, with linux mapconvng much smaller, but I don't remember how big exactly. Had to use windows implementation due to voidground.
After zipping, both results have basically same size.

[enetheru]

Update :

Because I'm trying to make a comprehensive tile merge/split program, I needed to solve the edge case where tiles share a border. think height map.


so I made that work.
Couple that with the fact that i made smt_convert handle high bit depth pixels and I can now 'tilify' height map images.

Caveat: Currently you cannot split an image to tiles with shared borders only join. I dont really expect that to change any time soon.

Also fixed some bugs to do with converting pixel formats.

[enetheru]

Not that it matters, but I've found myself hacking away at this again.
most mention-able thing is a pretty helper script called makemap.sh that compiles a map using a typical workflow, has an interactive menu, and will auto load the game to confirm that it works.

I really should do other things, but I find it cathartic to work on this, and it's still teaching me.

https://github.com/enetheru/smf_tools/tree/v0.1.1