smf_tools, alpha work

[Anarchid]

Pulled and compiled latest commits; it still doesn't seem to be very fast at compiling that 7k texture.

Will see if mosaic will help.

[enetheru]

Pulled and compiled latest commits; it still doesn't seem to be very fast at compiling that 7k texture.

Will see if mosaic will help.

it takes my pc about 3-4 minutes to convert the 7k texture, whats your rough timing?

[Anarchid]

1.5 hours. 0_0

That's after i split it into chunks of 1024.

But i'm still using old image library...

[abma]

debug vs release build?

[gajop]

Talked about with Anarchid and this seems cool.
Could you describe what this does, so a layman (preferably one without much knowledge about the Spring ecosystem) could understand?
What are the goals of the project, what it produces and what's required by it? etc. etc

[Forboding Angel]

As far as I can tell, it does the exact same thing that grout does for l3dt textures.

[enetheru]

I'm working atm and have stuff after work to do. so ill respond in about 9 hours time.

[enetheru]

Talked about with Anarchid and this seems cool.
Could you describe what this does, so a layman (preferably one without much knowledge about the Spring ecosystem) could understand?
What are the goals of the project, what it produces and what's required by it? etc. etc

it started out as a re-write of mapconv, but then it morphed as i learned more about smf and smt and how they were used in the spring engine.

smt_info - spits out smt information
smt_convert: takes a tile based image and spits out another tile based image.

smf_info - provides information about smt files
smf_decc - decompile smf into its components
smf_cc - constructs smf files based on inputs.

some of the interesting points it has is that:
* almost any source image size, it will scale shrink to fit your specifications.
* generic source tiles can be any size, say you have a mozaic of 1024x tiles to make up a 32768x huge image, thats ok. input as source and spit out any* size tiles you want.
* can use a tilemap to stitch together tiles and output either more tiles or a huge image.

why?
* because 32x32 pixel tiles are too small.. and i plan to make spring accept tiles of upto 1024x initially and perhaps single images for the diffuse(much like the ssmf works).
* because i believe that tile based images can be really powerful and i want to work on making it easier for artists. have a bunch of 256x256 pixel tiles and want to build a map using it? no problem images + csv map.

I have loads of ideas for shrinking the filesize of smf/smt while providing more flexibility and greater artistic control, and most importantly is backwards compatible with the way things are done already.. its basically a win win for spring.

[jK]

* because 32x32 pixel tiles are too small..

because of what?

[enetheru]

* because 32x32 pixel tiles are too small..

because of what?

justification is useful.

Because typical texture sizes for artists to work with are much larger, 128x128, 256x256, 512x512
being able to generate SMT files that contain these tiles directly makes decompiling them, appending to them, replacing specific tiles in the smt much more sane.
if you have to split all source images into 32x32 pixel tiles pulling them back out again to work with or ammending them is really painful.

because spring actually uses 1024x1024 tiles and stitches together the 32x32 tiles in the load phase. we can cut this out completely by using 1024x1024 tiles in the smt directly.

because the uint32 tilemap dimensions are (mapsize * 512) / tilesize. which can be reduced in resolution by increasing the tile size, hence decreasing the size of map data.

So i'm anticipating a reduced complexity, reduced load time, reduced filesize.

[enetheru]

So i'm anticipating a reduced complexity, reduced load time, reduced filesize.

Complexity in this case is from the perspective of content creators, the customers of the engine.

[jK]

* because 32x32 pixel tiles are too small..

because of what?

because the uint32 tilemap dimensions are (mapsize * 512) / tilesize. which can be reduced in resolution by increasing the tile size, hence decreasing the size of map data.

So i'm anticipating a reduced complexity, reduced load time, reduced filesize.

Spring uses 1024^2 textures for a 2x2 mapsize.
Also each 1024^2 texture is sliced into tiles of 32x32 -> (1024x1024)/(32x32)=(32x32)=1024 tiles

So for a 16x16 map, this means:
(16x16)/(2x2) = 8*8 = 64 textures of size 1024^2
64*1024tiles = 65k tiles

In the tileindexmap each tile is represented by a uint32 pointer/index.
So for a 16x16 map the tileindexmap is in total 65k * 4byte = 256kB.
Nothing worth to reduce further, esp. cause it compresses well (only a minimal subset of the uint32 range is used).

So where is the reduction in filesize & loadtime?

[PicassoCT]

In detecting tiles which are rotated, and store the 45° direction as 3bit info dir, thus compressing even further down, by reducing tiles, which are just turned duplicates..

Haha, whack the idea guy in reverse...

[jK]

iirc that was someone else's idea (argh? tr..?)
even code was supplied (only for rotating the dxt tile, not a full engine patch)

before implementing such first someone has to benchmark how well it performs

[enetheru]

In detecting tiles which are rotated, and store the 45° direction as 3bit info dir, thus compressing even further down, by reducing tiles, which are just turned duplicates..

Haha, whack the idea guy in reverse...

Yeah i had though of this, its on my list.

before implementing such first someone has to benchmark how well it performs

Of course.

Spring uses 1024^2 textures for a 2x2 mapsize.
Also each 1024^2 texture is sliced into tiles of 32x32 -> (1024x1024)/(32x32)=(32x32)=1024 tiles

So for a 16x16 map, this means:
(16x16)/(2x2) = 8*8 = 64 textures of size 1024^2
64*1024tiles = 65k tiles

but what if you want a 64x64 sized map?
(64x64)/(2x2) = 32*32 = 1024 textures of size 1024^2
~4mb per compressed 1024^2
That's a lot of data, so having re-usable tiles here is kinda necessary
1024*1024 = ~1million tiles.

tilemap
(32*32)^2 = 1024^2 = ~33mb uncompressed

But I want to go bigger

(128x128)/(2x2) = 64*64 = 4096 textures of size 1024^2
~4mb per compressed 1024^2
4096*1024 = ~4million tiles.

tilemap
(32*64)^2 = 2096^2 = ~134mb uncompressed

I'm not too sure I have my math 100% correct. but certainly the lower to regular sized maps don't see much improvement.

[jK]

I'm not too sure I have my math 100% correct. but certainly the lower to regular sized maps don't see much improvement.

it's not

but what if you want a 64x64 sized map?
...
(32*32)*1024 = ~1million tiles.

tilemap
(32*32)^2 = 1024^2 = ~33mb uncompressed

-> 1M tiles each tile needs a uint32 of 4byte -> 4MB

Even on a 128x128 map it's just 16MB. Much less than the texture data and it still compress well, so it's just a fraction of the final mapsize.

[enetheru]

Ahh yes, I was mixing up my bits and bytes.. silly me.

Tilemap savings will be minimal, but they still exist, it may not be a reason in and of itself to make modifications, but the fact that we get to save a little is nice.

Until I come up with some code that works to test this, discussion is pointless. I'll report back when I have more to show.

[Anarchid]

I've recompiled it today with what they say is the stable version of oiio, and it still takes more than an hour.

Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.

Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).

[enetheru]

I've recompiled it today with what they say is the stable version of oiio, and it still takes more than an hour.

Is there anything i could do to help identify the issue? Verbose mode seems to suggest it's just taking a significant while to copy each mosaic tile.

Maybe it's the data? Should i upload my 7k tex (wow that's 500mb).

its not openimageio thats for sure, those guys are pro's(sonyimageworks, and all the other big GC factories).

So the problem would be the way the data flows through my program, the way the tiles are requested.
It was really slow for me before because it would load up the big image every time it requested a tile. thats why a mosaic'd image was better.what a waste of time that was. so I kept around a pointer to the previous request and just never free'd it unless the source tile was different.

https://github.com/enetheru/smf_tools/b ... e.cpp#L126
https://github.com/enetheru/smf_tools/b ... e.cpp#L159

I realise I'm never freeing the pointer completely, but i kinda just wanted it to work for now and the program exits and cleans that up anyway(for now). I will make a more robust method with perhaps a faster path for single image workflow.

How long does it take for mapconv to process that image?
whats your computer specs?
I have an i7gen4 with 16gig ram

[enetheru]

I also really appreciate the time you are taking to test my project :D
Internet hugs for you!!