smf_tools, alpha work

[Anarchid]

I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.

smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse

[enetheru]

I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.

Thanks.

Having people actually want to test things I make is new to me, I wish i knew how to ask the right questions to find out why its so slow.

I'm assuming the cmd you are using is along the lines of

Code: Select all

smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg

Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).

what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg

Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.

the estimation of time only properly kicks in after the first 100 tiles are copied, I build that thing is a super rush. I should fix that.

smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse

hells yeah, blender rocks.

[enetheru]

ok this is rough, but here are some 32bit windows binaries..
when I say rough i mean really really rough.

https://www.dropbox.com/s/wlth5n4qgacyh ... s.zip?dl=0

[Anarchid]

I'm assuming the cmd you are using is along the lines of
Code:
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg

Currently it's more like this:

Code: Select all

smt_convert -v -smt ./mosaic/*

what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg

"Whatever blender spits out", in this case a png.

I didn't run a mapconv test yesterday. Too many irons in the fire.

[enetheru]

If anyone was interested; this page sort of gives the direction I'm heading with this stuff, feel free to edit it or add to it etc.

say for instance, you could add a section for "reasons why not to do this" and write some shit down.

https://springrts.com/wiki/Enetheru:SMF-SMT_Update

[jK]

Add ETC1 compression as an option for smt tiles

AFAIK none gpu supports them in hardware yet (only smartphones gpus do)

[enetheru]

ok so here is a screenshot of a map loaded in spring with an smt tilesize of 1024x1024.
I'm testing other sizes at the moment to confirm correct operation and will submit a pull request shortly.

[enetheru]

So for a test i went and created a map using tiled
with art from Lost Garden
and this is preliminary results.

Each tile is 128x128 pixels.


The internal format of the tiled maps can be chosen as csv, so thats what i'm importing into my smf.

For those with less imagination, visualise that the tiles include information for all the shading textures and height map. fully compatible with the existing style of huge images. could also mix and match approaches.

[Jools]

Can we have larger screenshots?

[enetheru]

Can we have larger screenshots?

image is hosted on imgur at 1080p, spring forum displays it small but its still 1080p, just open image in new tab

[Jools]

Ok, nice, ty.

But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.

[enetheru]

Ok, nice, ty.

But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.

umm, ive forked the code and am developing this, this is running on my pc, from source I compiled myself. pretty sure the versions are tied to the commit history somehow

[Anarchid]

Code: Select all

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ls -l map_final.png
-rw-r--r-- 1 anarchid anarchid 411713972 Feb  3 23:08 map_final.png

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ cat compile.sh 
#echo "converting heightmap to png..." &&
#convert heightmap.exr -depth 16 -size 769x769 -type Grayscale heightmap.png &&
echo "compiling map from png..." &&
springMapConvNG -t map_final.png -m metalmap.png -v vegetation.png -h heightmap.png -maxh 412 -minh -100 -o coagulation -ct 0 -th 0 -minimap minimap.png

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ./compile.sh 
compiling map from png...
Range : 0 -> 47208
Warning: Metal map has wrong size , rescaling! (384,384) instead of (448,448)
Loading minimap minimap.png
Warning: Vegetation map has wrong size , rescaling!
Compressing   749200/   50176      -  50152 tiles                    
Compress done , ratio: 100

This took ~ 20 seconds.

[enetheru]

~20 seconds wow.

Ive done some testing on my side and it appears that squish takes most of the time, just removing the squish call dropped the time it takes by 80% for me.

libsquish gives you three options for compression quality, so i changed it to the lowest one and now the only slowdown will be in my own code. 8k texture takes ~2 minutes instead of ~10.

I also changed minor othe things, pls test again and let me know results. :D

[smoth]

Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.

[enetheru]

Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.

Performance increase in what respect?
And thanks for the offer. I can generate huge images if needed.

[Anarchid]

The new one took 11 minutes 42 seconds. This is acceptable.

(also this is more than 80% time shaven off, more like 90%, a ten-fold speedup!)

The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done

I'll check if it scratches my voidground itches later this week.

[jK]

tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot

also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.

[enetheru]

The new one took 11 minutes 42 seconds. This is acceptable.

Phew, Thats encouraging, still too long IMHO, but better.
I don't know much about optimisation yet so completely out of my depth when thinking about it.

The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done

Its a bit of a hack anyway, I should re-work it.

tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot

I had in my previous version before the complete re-write; tile hash checking, and some pixel difference tolerance checking. and wanted to add perceptual difference checking. but I removed all of that to sort out the change in structure, it will make a comeback, later.

also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.

on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.

you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.

[jK]

also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.

on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.

Blabla Spring doesn't differ at all from your code: it takes the tiles, uses squish to decompress them and then uses rgetc to recompress them.

you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.

Checked your code, don't see any threading ...
Compression is ideal for threading & squish works fine with it. Nor does it increase memusage at all, neither does it make the code more complicated.
You should really use it, see https://github.com/spring/spring/blob/d ... s.cpp#L247