Page 3 of 4
Re: smf_tools, alpha work
Posted: 10 Feb 2015, 13:41
by Anarchid
I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.
smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse

Re: smf_tools, alpha work
Posted: 10 Feb 2015, 23:18
by enetheru
Anarchid wrote:I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.
Thanks.
Having people actually want to test things I make is new to me, I wish i knew how to ask the right questions to find out why its so slow.
I'm assuming the cmd you are using is along the lines of
Code: Select all
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
Anarchid wrote:Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).
what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg
Anarchid wrote:Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.
the estimation of time only properly kicks in after the first 100 tiles are copied, I build that thing is a super rush. I should fix that.
Anarchid wrote:smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse
hells yeah, blender rocks.
Re: smf_tools, alpha work
Posted: 11 Feb 2015, 10:38
by enetheru
ok this is rough, but here are some 32bit windows binaries..
when I say rough i mean really really rough.
https://www.dropbox.com/s/wlth5n4qgacyh ... s.zip?dl=0
Re: smf_tools, alpha work
Posted: 11 Feb 2015, 11:44
by Anarchid
I'm assuming the cmd you are using is along the lines of
Code:
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
Currently it's more like this:
what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg
"Whatever blender spits out", in this case a png.
I didn't run a mapconv test yesterday. Too many irons in the fire.
Re: smf_tools, alpha work
Posted: 13 Feb 2015, 01:25
by enetheru
If anyone was interested; this page sort of gives the direction I'm heading with this stuff, feel free to edit it or add to it etc.
say for instance, you could add a section for "reasons why not to do this" and write some shit down.
https://springrts.com/wiki/Enetheru:SMF-SMT_Update
Re: smf_tools, alpha work
Posted: 13 Feb 2015, 03:03
by jK
enetheru wrote:Add ETC1 compression as an option for smt tiles
AFAIK none gpu supports them in hardware yet (only smartphones gpus do)
Re: smf_tools, alpha work
Posted: 13 Feb 2015, 12:04
by enetheru
ok so here is a screenshot of a map loaded in spring with an smt tilesize of 1024x1024.
I'm testing other sizes at the moment to confirm correct operation and will submit a pull request shortly.

Re: smf_tools, alpha work
Posted: 14 Feb 2015, 10:30
by enetheru
So for a test i went and created a map using
tiled
with art from
Lost Garden
and this is preliminary results.
Each tile is 128x128 pixels.
The internal format of the tiled maps can be chosen as csv, so thats what i'm importing into my smf.
For those with less imagination, visualise that the tiles include information for all the shading textures and height map. fully compatible with the existing style of huge images. could also mix and match approaches.
Re: smf_tools, alpha work
Posted: 14 Feb 2015, 11:23
by Jools
Can we have larger screenshots?
Re: smf_tools, alpha work
Posted: 14 Feb 2015, 11:34
by enetheru
Jools wrote:Can we have larger screenshots?
image is hosted on imgur at 1080p, spring forum displays it small but its still 1080p, just open image in new tab
Re: smf_tools, alpha work
Posted: 14 Feb 2015, 12:03
by Jools
Ok, nice, ty.
But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
Re: smf_tools, alpha work
Posted: 14 Feb 2015, 12:10
by enetheru
Jools wrote:Ok, nice, ty.
But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
umm, ive forked the code and am developing this, this is running on my pc, from source I compiled myself. pretty sure the versions are tied to the commit history somehow
Re: smf_tools, alpha work
Posted: 14 Feb 2015, 19:54
by Anarchid
Code: Select all
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ls -l map_final.png
-rw-r--r-- 1 anarchid anarchid 411713972 Feb 3 23:08 map_final.png
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ cat compile.sh
#echo "converting heightmap to png..." &&
#convert heightmap.exr -depth 16 -size 769x769 -type Grayscale heightmap.png &&
echo "compiling map from png..." &&
springMapConvNG -t map_final.png -m metalmap.png -v vegetation.png -h heightmap.png -maxh 412 -minh -100 -o coagulation -ct 0 -th 0 -minimap minimap.png
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ./compile.sh
compiling map from png...
Range : 0 -> 47208
Warning: Metal map has wrong size , rescaling! (384,384) instead of (448,448)
Loading minimap minimap.png
Warning: Vegetation map has wrong size , rescaling!
Compressing 749200/ 50176 - 50152 tiles
Compress done , ratio: 100
This took ~ 20 seconds.
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 01:09
by enetheru
~20 seconds wow.
Ive done some testing on my side and it appears that squish takes most of the time, just removing the squish call dropped the time it takes by 80% for me.
libsquish gives you three options for compression quality, so i changed it to the lowest one and now the only slowdown will be in my own code. 8k texture takes ~2 minutes instead of ~10.
I also changed minor othe things, pls test again and let me know results. :D
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 02:12
by smoth
Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 05:02
by enetheru
smoth wrote:Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
Performance increase in what respect?
And thanks for the offer. I can generate huge images if needed.
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 14:48
by Anarchid
The new one took 11 minutes 42 seconds. This is acceptable.
(also this is more than 80% time shaven off, more like 90%, a ten-fold speedup!)
The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
I'll check if it scratches my voidground itches later this week.
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 16:46
by jK
tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot
also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
Re: smf_tools, alpha work
Posted: 15 Feb 2015, 23:46
by enetheru
Anarchid wrote:The new one took 11 minutes 42 seconds. This is acceptable.
Phew, Thats encouraging, still too long IMHO, but better.
I don't know much about optimisation yet so completely out of my depth when thinking about it.
Anarchid wrote:The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
Its a bit of a hack anyway, I should re-work it.
jK wrote:tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot
I had in my previous version before the complete re-write; tile hash checking, and some pixel difference tolerance checking. and wanted to add perceptual difference checking. but I removed all of that to sort out the change in structure, it will make a comeback, later.
jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.
you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
Re: smf_tools, alpha work
Posted: 16 Feb 2015, 02:57
by jK
enetheru wrote:jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.
Blabla Spring doesn't differ at all from your code: it takes the tiles, uses squish to decompress them and then uses rgetc to recompress them.
enetheru wrote:you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
Checked your code, don't see any threading ...
Compression is ideal for threading & squish works fine with it. Nor does it increase memusage at all, neither does it make the code more complicated.
You should really use it, see
https://github.com/spring/spring/blob/d ... s.cpp#L247