smf_tools, alpha work - Page 3

smf_tools, alpha work

Discuss maps & map creation - from concept to execution to the ever elusive release.

Moderator: Moderators

User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: smf_tools, alpha work

Post by Anarchid »

I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.

smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse :P
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

Anarchid wrote:I've an i3 mobile with 6gb ram i think. I'll test mapconv-ng's speed when i get home, the smaller 6k image took some minutes formerly.
Thanks.

Having people actually want to test things I make is new to me, I wish i knew how to ask the right questions to find out why its so slow.

I'm assuming the cmd you are using is along the lines of

Code: Select all

smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
Anarchid wrote:Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).
what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg
Anarchid wrote:Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.
the estimation of time only properly kicks in after the first 100 tiles are copied, I build that thing is a super rush. I should fix that.
Anarchid wrote:smf_tools currently takes more time to compile the map for me than it takes for Blender to generate the diffuse
hells yeah, blender rocks.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

ok this is rough, but here are some 32bit windows binaries..
when I say rough i mean really really rough.

https://www.dropbox.com/s/wlth5n4qgacyh ... s.zip?dl=0
User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: smf_tools, alpha work

Post by Anarchid »

I'm assuming the cmd you are using is along the lines of
Code:
smt_convert -v --smt --tilesize 32x32 -o output.smt 7ktex.jpg
Currently it's more like this:

Code: Select all

smt_convert -v -smt ./mosaic/*
what file format is that? i have a 8k texture for charlie and its only 17mb saved as jpeg
"Whatever blender spits out", in this case a png.

I didn't run a mapconv test yesterday. Too many irons in the fire.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

If anyone was interested; this page sort of gives the direction I'm heading with this stuff, feel free to edit it or add to it etc.

say for instance, you could add a section for "reasons why not to do this" and write some shit down. :P

https://springrts.com/wiki/Enetheru:SMF-SMT_Update
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: smf_tools, alpha work

Post by jK »

enetheru wrote:Add ETC1 compression as an option for smt tiles
AFAIK none gpu supports them in hardware yet (only smartphones gpus do)
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

ok so here is a screenshot of a map loaded in spring with an smt tilesize of 1024x1024.
I'm testing other sizes at the moment to confirm correct operation and will submit a pull request shortly.
Image
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

So for a test i went and created a map using tiled
with art from Lost Garden
and this is preliminary results.

Each tile is 128x128 pixels.
Image

The internal format of the tiled maps can be chosen as csv, so thats what i'm importing into my smf.

For those with less imagination, visualise that the tiles include information for all the shading textures and height map. fully compatible with the existing style of huge images. could also mix and match approaches.
User avatar
Jools
XTA Developer
Posts: 2816
Joined: 23 Feb 2009, 16:29

Re: smf_tools, alpha work

Post by Jools »

Can we have larger screenshots?
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

Jools wrote:Can we have larger screenshots?
image is hosted on imgur at 1080p, spring forum displays it small but its still 1080p, just open image in new tab
User avatar
Jools
XTA Developer
Posts: 2816
Joined: 23 Feb 2009, 16:29

Re: smf_tools, alpha work

Post by Jools »

Ok, nice, ty.

But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

Jools wrote:Ok, nice, ty.

But spring 98.0.1-380 ? That's not even built yet? The latest shows -376.
umm, ive forked the code and am developing this, this is running on my pc, from source I compiled myself. pretty sure the versions are tied to the commit history somehow
User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: smf_tools, alpha work

Post by Anarchid »

Code: Select all

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ls -l map_final.png
-rw-r--r-- 1 anarchid anarchid 411713972 Feb  3 23:08 map_final.png

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ cat compile.sh 
#echo "converting heightmap to png..." &&
#convert heightmap.exr -depth 16 -size 769x769 -type Grayscale heightmap.png &&
echo "compiling map from png..." &&
springMapConvNG -t map_final.png -m metalmap.png -v vegetation.png -h heightmap.png -maxh 412 -minh -100 -o coagulation -ct 0 -th 0 -minimap minimap.png

anarchid@wormbox:~/.spring/maps/coagulation.sdd/maps$ ./compile.sh 
compiling map from png...
Range : 0 -> 47208
Warning: Metal map has wrong size , rescaling! (384,384) instead of (448,448)
Loading minimap minimap.png
Warning: Vegetation map has wrong size , rescaling!
Compressing   749200/   50176      -  50152 tiles                    
Compress done , ratio: 100
This took ~ 20 seconds.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

~20 seconds wow.

Ive done some testing on my side and it appears that squish takes most of the time, just removing the squish call dropped the time it takes by 80% for me.

libsquish gives you three options for compression quality, so i changed it to the lowest one and now the only slowdown will be in my own code. 8k texture takes ~2 minutes instead of ~10.

I also changed minor othe things, pls test again and let me know results. :D
User avatar
smoth
Posts: 22309
Joined: 13 Jan 2005, 00:46

Re: smf_tools, alpha work

Post by smoth »

Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

smoth wrote:Apologies in advance if this is a dumb question but do you see much performance increase? would you like one my larger more complex textures to play with? I can upload any map you need to help you out.
Performance increase in what respect?
And thanks for the offer. I can generate huge images if needed.
User avatar
Anarchid
Posts: 1384
Joined: 30 Nov 2008, 04:31

Re: smf_tools, alpha work

Post by Anarchid »

The new one took 11 minutes 42 seconds. This is acceptable.

(also this is more than 80% time shaven off, more like 90%, a ten-fold speedup!)

The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done :P

I'll check if it scratches my voidground itches later this week.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: smf_tools, alpha work

Post by jK »

tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot

also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
User avatar
enetheru
Posts: 627
Joined: 11 Jun 2010, 07:32

Re: smf_tools, alpha work

Post by enetheru »

Anarchid wrote:The new one took 11 minutes 42 seconds. This is acceptable.
Phew, Thats encouraging, still too long IMHO, but better.
I don't know much about optimisation yet so completely out of my depth when thinking about it.
Anarchid wrote:The progress bar doesn't appear if i remove -v, and with -v it only appears when it's almost done
Its a bit of a hack anyway, I should re-work it.
jK wrote:tip: compare the uncompressed & the dxt tile, and run squish in high quality mode on those where it differs a lot
I had in my previous version before the complete re-write; tile hash checking, and some pixel difference tolerance checking. and wanted to add perceptual difference checking. but I removed all of that to sort out the change in structure, it will make a comeback, later.
jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.

you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: smf_tools, alpha work

Post by jK »

enetheru wrote:
jK wrote:also do you run it threaded? 10mins for 8k^2 sounds really long, i.e. Spring converts dxt1 to etc1 on the fly in under <1min.
on my pc it only takes 2 minutes. If it was as simple as re-encoding tiles I can see why its so fast, but my utility takes any tile based image and converts it into any other tile based image.
Blabla Spring doesn't differ at all from your code: it takes the tiles, uses squish to decompress them and then uses rgetc to recompress them.
enetheru wrote:you can feed it any resolution tiles, and ask it to spit out any number of other tiles at any size(with minor limitations like multiple of 4 dimensions) and it doesn't construct a giant image to do so. I wanted to process ridiculously sized images without filling up my ram.
Checked your code, don't see any threading ...
Compression is ideal for threading & squish works fine with it. Nor does it increase memusage at all, neither does it make the code more complicated.
You should really use it, see https://github.com/spring/spring/blob/d ... s.cpp#L247
Post Reply

Return to “Map Creation”