mapconv, using libsquish for dxt1 compression

[prodigy]

I am working on my own changes to mapconv. Using libsquish for dxt1 compression, so there is no need to use the external nvdxt.exe. edit: The only other big difference is instead of command line arguments it takes a lua file.

edit: It is basically exactly the same code etc as the mapconv from the spring trunk about version 0.69???

The image code has been changed, currently only BMP files are supported. edit: Raw heightmaps same as original mapconv.

The exe takes a filename of a lua script, which is basically just a table that contains all the info that used to be fed in the command line of mapconv. I use lua to create the SMD file from this table as well.

To try it out on the command line: mapc.exe mapname.map

For now I have put windows and linux builds (and source code) here:

http://code.google.com/p/mapc/

Here are the contents of an example map file.

Code: Select all

--[[
   An example Spring map description file.
--]]

Map = {
  id = 878212, -- a unique number that identifies the map
  name = "chasmofchaos1",
  description = [[Chasm of Chaos Version 1]],
  max_height = 400,
  min_height = 100,
  metal_map = "metalmap.bmp",
  height_map = "heightmap.bmp",
  invert = true, -- normally true when height map is a BMP
  compress = 0.7,
  lowpass = true,
  type_map = "",
  feature_map = "",
  texture_map = "texture.bmp",
  smd = {
    MAP = {
      Description="A lot of metal in the chasm, but beware the chaos",
      TidalString=20,
      Gravity=130,
      MaxMetal=1.0,
      ExtractorRadius=100,
      MapHardness=100,
      ATMOSPHERE = {
        FogColour="0.7 0.7 0.8",
        FogStart=0.2,
        CloudColor="0.9 0.9 0.9",
        SkyColor="0.1 0.15 0.7",
        SunColor="1.0 1.0 1.0",
        CloudDensity="0.55",
        MinWind=5,
        MaxWind=20
      },
      WATER = {
		WaterBaseColor="0.4 0.6 0.8", --color at water surface
		WaterAbsorb="0.004 0.004 0.002", --how fast different colors are absorbed by the water
		WaterMinColor="0.1 0.1 0.3", --the min value the water colors will go down to
		WaterPlaneColor="0.1 0.1 0.3"
	  },
	  LIGHT = {
		SunDir="0 1 2", -- direction of sun (spring will normalize it later)(y component is upward)

		GroundAmbientColor="0.4 0.4 0.4", -- ambient (non sun lit) color of ground (and grass trees etc)
		GroundSunColor="0.7 0.7 0.7",	-- color of ground where fully sun lit (added to ambient)
		GroundShadowDensity="0.8",		-- how far from the non shadowed to the ambient color stuff in shadow will go

		UnitAmbientColor="0.3 0.3 0.3",	-- ambient (non sun lit) color of units (and wreckage etc)
		UnitSunColor="0.8 0.8 0.8",		-- color of units where fully sun lit (added to ambient)
		UnitShadowDensity="0.8"			-- how far from the non shadowed to the ambient color stuff in shadow will go
	  },
	  TERRAINTYPE0 = {	 -- which terrain type goes where on the map is determined by the typemap
		name="default", -- human visible identifier of the terrain type
		hardness=1, -- multiplier of the global maphardness value (default value=1)
		tankmovespeed=1,	-- multiplier of tank units speed after slope mods etc has been applied (default value=1)
		kbotmovespeed=1,    -- multiplier of kbot units speed after slope mods etc has been applied (default value=1)
		hovermovespeed=1,	-- multiplier of hover units speed after slope mods etc has been applied (default value=1)
		shipmovespeed=1     --multiplier of ship units speed after slope mods etc has been applied (default value=1)
	  },
          TEAM0 = {
		StartPosX=128,
		StartPosZ=512
	  },
	  TEAM1 = {
		StartPosX=256,
		StartPosZ=512
	  },
	  TEAM2 = {
		StartPosX=384,
		StartPosZ=512
	  },
	  TEAM3 = {
		StartPosX=3968,
		StartPosZ=512
	  },
	  TEAM4 = {
		StartPosX=3840,
		StartPosZ=512
	  },
	  TEAM5 = {
		StartPosX=3712,
		StartPosZ=512
	  }
    }
  }
}

if Map.smd then
  dofile("smd.lua")
  write_smd(Map)
end

The smd.lua file that serializes the SMD is included in the zip file.

If anyone can try compiling some maps with this compiler and let me know how they turn out, that would be great. All you have to do is change the value of height_map, texture_map, metal_map.

The advantage of these mapconv changes are:

• * libsquish is open source, and is builtin so there is no cost of executing an external process. I cannot find the source code for nvdxt.exe
  * using lua allows putting in hooks and including other lua files to do some customized processing before writing the SMF and SMT. Also it is easy to serialize the SMD from a lua table.

[smoth]

if it doesn't support raw it is useless.

[Beherith]

3 hours for a 32*32 map is painful

[BrainDamage]

it's not "ta spring" since ages, please fix your tags & description

also, source please? the svn repo is empty

[BaNa]

why thank you for working on a better working map compressor, we are all very happy to see new ways emerging.

[Beherith]

Hmm, the compression setting seems to do nothing.
Also, it doesnt clean up after itself.

[AF]

I would think improvements in compilation of this format would be better of through the use of multiple cores n such rather than through revised image decompression/compression methods

[prodigy]

Here is a new build: http://mapc.googlecode.com/files/mapcwin-0.2.zip

This is an SSE2 build, so the DXT1 squishing should be much faster than 0.1.

Source code is here: http://code.google.com/p/mapc/source/br ... nk/mapconv

This is essentially the same mapconv code as always, with the following differences:

• * It takes a single lua file instead of many command line arguments. The arguments are taken from the lua file.
  * It uses libsquish to do the DXT1 compress instead of nvdxt.exe.

Thanks for your comments guys.

@smoth: yes it supports raw height maps like the original mapconv
@Beherith: try v0.2 which was compiled with SSE2. v 0.2 cleans up after itself now. The compression should work the same as the original mapconv, are you seeing a difference in behavior between my build and the old mapconv?
@Brain Damage: source is there now, and tags updated
@BaNa: thanks for your positive comment
@AF: we can't really change the compression algorithms. The map format depends on the algorithms used. It may be possible to parallelize the tiling though.

[jK]

you should perhaps use nvidia's texture tools instead of libsquish.
http://developer.nvidia.com/object/texture_tools.html

[prodigy]

you should perhaps use nvidia's texture tools instead of libsquish.
http://developer.nvidia.com/object/texture_tools.html

Interestingly on that web site, if you click on the "complete source code" link it takes you to http://code.google.com/p/nvidia-texture-tools/ where it says that the DXT1 compression is based on Simon Browns squish library.

So by directly using Squish we're able to bypass all the Nvidia wrappers, and use essentially the exact same DXT1 code.

[jK]

The library also contains an alternative GPU-accelerated compressor that uses CUDA and is one order of magnitude faster.

So the CUDA acceleration wouldn't affect all ppl, but those with a nv series>=8, would benefit a lot of it.

[prodigy]

@jK: I have no interest in integrating mapconv with CUDA, as I do not have a Nvidia graphics card, and CUDA does not appear to have an open source style license. But you are welcome to join the mapc google project if you want to add CUDA support.

[jK]

I don't own a series >=8 either, but a speedup of 5-15x is worth to think about it imo.
Also you shouldn't bother with CUDA at all, that's why there is the library.
And CUDA is an API implementation, and most API implementations you use aren't open source (most opengl drivers, etc.).

[prodigy]

@jK: I looked at the nvidia-texture-tools and looks reasonable to use this instead of Squish directly. But I probably won't do it anytime soon, since Squish alone works well enough for me. Anyone who wants to join the project and contribute to the code is welcome.

[clericvash]

@jK: I looked at the nvidia-texture-tools and looks reasonable to use this instead of Squish directly. But I probably won't do it anytime soon, since Squish alone works well enough for me. Anyone who wants to join the project and contribute to the code is welcome.

Could you not put in a wrapper to detect if people have nv 8+ series to then use the better stuff for them?

[lurker]

I think it does that mostly by itself. The issue is the time to put the better stuff in.

[Beherith]

@jK: I looked at the nvidia-texture-tools and looks reasonable to use this instead of Squish directly. But I probably won't do it anytime soon, since Squish alone works well enough for me. Anyone who wants to join the project and contribute to the code is welcome.

Your current version is unusably slow on windows

[prodigy]

@Beherith:

One step that my mapconv doesn't do is the dither. I found that if I dither the texture using GIMP and then use that dithered texture, my mapconv runs a lot faster. Can you dither your texture to say 256 colors and then use that dithered texture?

In Gimp I dithered by clicking Image->Mode->Indexed, then Generate Optimum plalette 256 colors, Dithering: Floyd-steinberg. OK.

After that I went Image->Mode->RGB to get the image back to RGB format, then saved as 24-bit BMP.

If I dither it down to 16 colors, squish runs even faster.

So it is up to you, if you want squish to run faster you can do the dither the texture before running mapc. The texture may look slightly worse in the game though?

My Spring build is broken right now so I haven't been able to check the results of different dithering in game.

Also I have added another setting to the squish compression that makes it run a little faster as well. To get still more speed you can try version 0.3 from here:

http://mapc.googlecode.com/files/mapcwin-0.3.zip