smf_tools, alpha work
Moderator: Moderators
Re: smf_tools, alpha work
I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.
smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse
smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse
Re: smf_tools, alpha work
Thanks.Anarchid wrote:I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.
Having people actually want to test things I make is new to me, I wish i knew how to ask the right questions to find out why its so slow.
I'm assuming the cmd you are using is along the lines of
Code: Select all
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpegAnarchid wrote:Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).
the estimation of time only properly kicks in after the first 100 tiles are copied, I build that thing is a super rush. I should fix that.Anarchid wrote:Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.
hells yeah, blender rocks.Anarchid wrote:smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse
Re: smf_tools, alpha work
ok this is rough, but here are some 32bit windows binaries..
when I say rough i mean really really rough.
https://www.dropbox.com/s/wlth5n4qgacyh ... s.zip?dl=0
when I say rough i mean really really rough.
https://www.dropbox.com/s/wlth5n4qgacyh ... s.zip?dl=0
Re: smf_tools, alpha work
Currently it's more like this:I'm assuming the cmd you are using is along the lines of
Code:
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
Code: Select all
smt_convert -v -smt ./mosaic/*
"Whatever blender spits out", in this case a png.what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg
I didn't run a mapconv test yesterday. Too many irons in the fire.
Re: smf_tools, alpha work
If anyone was interested; this page sort of gives the direction I'm heading with this stuff, feel free to edit it or add to it etc.
say for instance, you could add a section for "reasons why not to do this" and write some shit down.
https://springrts.com/wiki/Enetheru:SMF-SMT_Update
say for instance, you could add a section for "reasons why not to do this" and write some shit down.
https://springrts.com/wiki/Enetheru:SMF-SMT_Update
Re: smf_tools, alpha work
AFAIK none gpu supports them in hardware yet (only smartphones gpus do)enetheru wrote:Add ETC1 compression as an option for smt tiles
Re: smf_tools, alpha work
ok so here is a screenshot of a map loaded in spring with an smt tilesize of 1024x1024.
I'm testing other sizes at the moment to confirm correct operation and will submit a pull request shortly.
I'm testing other sizes at the moment to confirm correct operation and will submit a pull request shortly.
Re: smf_tools, alpha work
So for a test i went and created a map using tiled
with art from Lost Garden
and this is preliminary results.
Each tile is 128x128 pixels.
The internal format of the tiled maps can be chosen as csv, so thats what i'm importing into my smf.
For those with less imagination, visualise that the tiles include information for all the shading textures and height map. fully compatible with the existing style of huge images. could also mix and match approaches.
with art from Lost Garden
and this is preliminary results.
Each tile is 128x128 pixels.
The internal format of the tiled maps can be chosen as csv, so thats what i'm importing into my smf.
For those with less imagination, visualise that the tiles include information for all the shading textures and height map. fully compatible with the existing style of huge images. could also mix and match approaches.
Re: smf_tools, alpha work
Can we have larger screenshots?
Re: smf_tools, alpha work
image is hosted on imgur at 1080p, spring forum displays it small but its still 1080p, just open image in new tabJools wrote:Can we have larger screenshots?
Re: smf_tools, alpha work
Ok, nice, ty.
But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
Re: smf_tools, alpha work
umm, ive forked the code and am developing this, this is running on my pc, from source I compiled myself. pretty sure the versions are tied to the commit history somehowJools wrote:Ok, nice, ty.
But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
Re: smf_tools, alpha work
Code: Select all
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ls -l map_final.png
-rw-r--r-- 1 anarchid anarchid 411713972 Feb 3 23:08 map_final.png
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ cat compile.sh
#echo "converting heightmap to png..." &&
#convert heightmap.exr -depth 16 -size 769x769 -type Grayscale heightmap.png &&
echo "compiling map from png..." &&
springMapConvNG -t map_final.png -m metalmap.png -v vegetation.png -h heightmap.png -maxh 412 -minh -100 -o coagulation -ct 0 -th 0 -minimap minimap.png
anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ./compile.sh
compiling map from png...
Range : 0 -> 47208
Warning: Metal map has wrong size , rescaling! (384,384) instead of (448,448)
Loading minimap minimap.png
Warning: Vegetation map has wrong size , rescaling!
Compressing 749200/ 50176 - 50152 tiles
Compress done , ratio: 100
Re: smf_tools, alpha work
~20 seconds wow.
Ive done some testing on my side and it appears that squish takes most of the time, just removing the squish call dropped the time it takes by 80% for me.
libsquish gives you three options for compression quality, so i changed it to the lowest one and now the only slowdown will be in my own code. 8k texture takes ~2 minutes instead of ~10.
I also changed minor othe things, pls test again and let me know results. :D
Ive done some testing on my side and it appears that squish takes most of the time, just removing the squish call dropped the time it takes by 80% for me.
libsquish gives you three options for compression quality, so i changed it to the lowest one and now the only slowdown will be in my own code. 8k texture takes ~2 minutes instead of ~10.
I also changed minor othe things, pls test again and let me know results. :D
Re: smf_tools, alpha work
Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
Re: smf_tools, alpha work
Performance increase in what respect?smoth wrote:Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
And thanks for the offer. I can generate huge images if needed.
Re: smf_tools, alpha work
The new one took 11 minutes 42 seconds. This is acceptable.
(also this is more than 80% time shaven off, more like 90%, a ten-fold speedup!)
The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
I'll check if it scratches my voidground itches later this week.
(also this is more than 80% time shaven off, more like 90%, a ten-fold speedup!)
The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
I'll check if it scratches my voidground itches later this week.
Re: smf_tools, alpha work
tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot
also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
Re: smf_tools, alpha work
Phew, Thats encouraging, still too long IMHO, but better.Anarchid wrote:The new one took 11 minutes 42 seconds. This is acceptable.
I don't know much about optimisation yet so completely out of my depth when thinking about it.
Its a bit of a hack anyway, I should re-work it.Anarchid wrote:The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
I had in my previous version before the complete re-write; tile hash checking, and some pixel difference tolerance checking. and wanted to add perceptual difference checking. but I removed all of that to sort out the change in structure, it will make a comeback, later.jK wrote:tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot
on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
Re: smf_tools, alpha work
Blabla Spring doesn't differ at all from your code: it takes the tiles, uses squish to decompress them and then uses rgetc to recompress them.enetheru wrote:on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
Checked your code, don't see any threading ...enetheru wrote:you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
Compression is ideal for threading & squish works fine with it. Nor does it increase memusage at all, neither does it make the code more complicated.
You should really use it, see https://github.com/spring/spring/blob/d ... s.cpp#L247