Physically Based Rendering

[ivand]

Hi folks!

I've put together an implementation of Physically Based Rendering for spring models.
It generally works ok it seems, but I kinda stuck with Image Based Lighting. I've tried to leverage built-in $specular and $reflection cubemaps, but I don't get how to use these textures to do IBL.

I've sampled both, but neither of them look like an environment reflection:



Can someone shed some light on the topic?

Some PBR teaser without IBL (IBL is replaced with white color): https://drive.google.com/file/d/1KkY_kg ... fQHV_/view

[Kloot]

Both screenshots look correct to me.

The specular cubemap (top image) basically just stores C * pow(dot(N, L), e), where C is the sun's model specular color, L the sun direction and e the specular exponent. This is going to be black for most N's.

Ravaged's skybox contains green near the horizon and blue-grey at its zenith, which are reflected by the model's vertical and horizontal surfaces respectively, although those two terrain reflection "arches" on its barrel don't seem 100% accurate for sky-facing geometry.

Increase CubeTexSizeReflection in springsettings.cfg to make reflections less downsampled.

[ivand]

Thx @Kloot.
So it's really $reflection tex I should use for IBL.

I've changed the PBR shader and now it looks so much better:
PBR shader on Coagulation Marsh

[ivand]

One more question to @Kloot.

I noticed that $reflection texture has GL_LINEAR minification filter. From the PBR effects perspective it's much better to have that equal to GL_LINEAR_MIPMAP_LINEAR instead. Can this be changed?

Generally I noticed that several textures have filters that don't fit my shaders.
One another example is heightmap "nearest" sampling. I either need to copy $heightmap to another texture with the filtering I want or introduce sampler on the shader side (where bilinear is easy to do, but mipmap emulation is hard or even impossible).

Unfortunately the current Lua API only allows to select sampling in two cases (AFAIK). When:
1. Texture is created from some file. E.g. gl.Texture(":iac:/LuaUI/images/image.png")
2. During the texture creation, i.e. gl.CreateTexture parameters.

Is it possible to add a Lua API that would allow for changing the texture filtering after the texture have been created?

[MaDDoX]

I've been collaborating with Ivand to get this working, mostly from a technical artist point of view - model & textures also provided, they're for a WIP new Spring game. Amazing job so far Ivand, I can't tell you how happy I am with what you accomplished! :) This is the #1 graphical breakthrough Spring needs for its graphics to be on par with most of the top engines out there. I hope we can get it working with Terrain and features, it'd be killer!

That said, there's one major thing left to get this looking perfect as we get in Substance, Sketchfab, Marmoset etc: HDR IBL (high dynamic range, image-based lighting). Many HDR images use a 16-bit per channel format, which Spring wouldn't be able to read and also would take too much disk + ram space. *But* there's the Radiance image format (.hdr), which manages to encode the HDR image's full dynamic range in a simple 32-bit image - meaning it can be stored as a DXT5 .DDS file, for fast reads and smaller file sizes. Link:
https://en.wikipedia.org/wiki/RGBE_image_format

Tell me if you want a couple samples of an HDR texture in .DDS format, I'll be glad to provide it to you.

So, this kind of image (optimally panoramic/equirectangular) could be read from an entry in mapinfo.lua and, if not provided, fallback to use the $reflection texture. Talking of $reflection, I'm not sure if the "e" (specular exponent) is being generated with floating point precision, the result you've shown looks a lot like SDR to me - but maybe it's just a poor cubemap defined in the map. Anyways, for the PBR shader you could try applying a curve remap to its strength, so the closer to the upper range of luminance you are, the stronger the actually applied lighting intensity would be.

[ivand]

Thanks for the praise, @MaDDox. I wouldn't have started PBR implementation, if no one had provided me a model and textures for PBR. Fortunately you did, so we've got a semi-complete implementation for now.

Features could be incorporated easily. As far as PBR map shader, It's likely doable with recent spring versions, but it requires completely separate Lua gadgetry and slight changes to the shader code. I might look into this later.

Yesterday I found and tested a piece of code that allows to sample the environment from the equirectangular image. https://www.shadertoy.com/view/4lycz3. See the vec3 getEnvironment(vec3 rayDirection) function.

It's also not hard to map an RGBE data to HDR linear(?) RGB in the shader.
If you could author an RGBE map image, it won't take me long to make it usable instead of $reflection.

[ivand]

Big thanks to @Kloot. He managed to resolve all issues/feature requests I raised in the previous post.

In case someone was wondering what kind of effect environment mipmaping has on reflections, see here

After I've managed to solve a couple of hidden issues (like the fact that normals stored in sRGB) I've come to the following result. Note sky is pitch black here, so model is also a bit dark.

[PicassoCT]

Fuck, every map- retune the sky for this

[ivand]

I was thinking about skybox problem and came up with several ideas:

1) MaDDox prefers usage of the custom HDR environment maps. Since Spring OpenGL API can't into cubemaps, they are going to be supplied in equirectangular projection. Pros: if I happen to handle them right, they should look gorgeous. Cons: bye-bye dynamic reflection of terrain.
2) There could be a way to combine static HDR environment map with $reflection. I don't yet know how, but might figure something out. This is essentially extension of the current implementation + (1) + some math to put them together.
3) Color hack: I can sample skydome in a few places and if average luminosity of the pixels is too low, I can tone map it to something brighter. The main issue here is same as in (2). I've got to find the way to tall apart sampling of terrain from sampling of the sky.

As far as PBR progress, it has slowed down a bit. Yesterday I was able to incorporate parallax mapping. Unfortunately it didn't look any good on the model I was using (jeffy). MaDDox told me that parallax should work better for the cases like terrain rendering. Today I've cleaned out bugs from my pet WebGL project and looks like MaDDox was right. In the certain scenarios it looks indeed as good as it's hard to guess displacements have no geometry behind them. See here

[Kloot]

2) extend gl.CreateTexture with cubemap support (easy-ish), then dynamically replace the skybox by your HDR environment cubemap via Spring.SetSkyBoxTexture (currently deadweight)

[ivand]

Hi!

I'm more or less done with draft version of PBR unit shader.

You can find sources here: https://github.com/lhog/Zero-K/tree/pbrCUS_v1
And see changes against vanilla ZK here: https://github.com/lhog/Zero-K/compare/ ... :pbrCUS_v1

Generally the shader can be used in any spring game/mod. It's essential though that the game has some implementation of Custom Unit Shaders (CUS), that looks into the "Materials" directory or something like that.

What you need for integration:
1 ) Model. I only played with Collada/DAE models, others might work, but I've never tested that.
2 ) Model description file named <model.dae>.lua. My implementation uses same file as mentioned here: https://springrts.com/wiki/Assimp except it extends it with "pbr" section. This section defines PBR textures and parameters to be used to render this model. See https://github.com/lhog/Zero-K/blob/pbr ... ua#L10-L78 it's very well commented file.
3 ) PBR textures. These go into "unittextures" dir and should be referenced and described in <model.dae>.lua file. Note that if you use DDS, it might be a good idea to generate mip maps. It's especially important for normal map file.
4 ) brdflutTex.png. Don't bother what it is, just put it into "unittextures" dir from here https://github.com/lhog/Zero-K/blob/pbr ... lutTex.png
5 ) Shader definition file. https://github.com/lhog/Zero-K/blob/pbr ... rs/pbr.lua Put it in the same dir as in repository
6 ) Material parser file. https://github.com/lhog/Zero-K/blob/pbr ... 10_pbr.lua Put it in the same dir as in repository
7 ) If you want your model to recieve the custom image based lighting (IBL). Define these two textures: https://github.com/lhog/Zero-K/blob/pbr ... ua#L74-L76 If SPECULARMAP and/or IRRADIANCEMAP are existing files in "unittextures" dir then renderer expects them to be in equirectangular projection and in RGBE format. If SPECULARMAP and/or IRRADIANCEMAP are not defined, renderer uses standard reflection cubemap for both.
8 ) Some parameters (like final gamma correction or tonemapping or environment maps) are possible to define either in model file or in map file. Although latter exists in the code, I've never tested it so far. Should work, but you will never know until you test.

Ping me if I missed something or some point needs better description.

[ivand]

Small update: I seem to have fixed parallax occlusion mapping:

It should look cool on bumpy areas.

One question left up in the air is how to specify material transparency/cutoffs within the current CUS framework. Currently alpha value is silently ignored down-to around 0.5, after that the pixels just turn black. This is generally not what required by artists. They want, for example, trees with cut off pixel areas.

I tried to ask in #sy or play with materials myself, but didn't get any positive results.
@Kloot, can you suggest if transparent materials in CUS are possible (and how?)

Thanks,

[Kloot]

They are (https://springrts.com/wiki/Lua_UnitRend ... erialTypes), but only during the forward pass. See these framework lines for pointers:

https://github.com/lhog/Zero-K/blob/pbr ... s.lua#L287
https://github.com/lhog/Zero-K/blob/pbr ... s.lua#L446

The one downside here is that the DrawUnit callin is slow in large numbers, which may have to be solved soon to drive wide adoption by games.

[ivand]

@Kloot, I tried to change exactly those values to something else, for example to "alpha" (which is presumably a material with support of transparency). This change effectively knocks out CUS rendering as it falls back to engine rendering:

"opaque":


"alpha"

[Kloot]

Alpha materials are only used when the engine renders cloaked units (TA legacy cruft), just add canCloak=true to your test-object defs or call Spring.SetUnitCloak.

[PicassoCT]

If there was a time machine, and Kloot had acess to it, Hitler would live and some of the sys would have a really bad time

[ivand]

Adding canCloak = true to UnitDef and switching to "alpha" didn't do the trick for me, unfortunately.

Though I semi-randomly came to the following blending state: https://github.com/lhog/Zero-K/blob/797 ... br.lua#L13

Whenever I'm dealing with blending state I feel lost, so it can be very wrong, but seems to work well nevertheless:

[ivand]

I noticed that for some models, normal mapping looked wrong.

Specifically right/left and up/down direction of normals looked flipped.
Here is how mapped normals(world space) should look like:

While up until my recent commit it looked like this:


Notice how red and blue look flipped.

I'm not really sure why it happens on some models and doesn't happen on others. Probably has something to do with winding order or smth.

Anyways I've added flipBitangent option to model definition file, so that that kind of flip can be corrected. Now with flipBitangent = true, rendering is correct:


I'm eager to see people start testing my implementation, so small glitches like this are discovered and sorted out.

[Kloot]

For S3O's Spring calculates TBN handedness per vertex such that mirrored UV's can be used on symmetric models, which obviously requires a consistent winding order to work. If your models are in .dae format, then the tangents are calculated by assimp which may internally do the same thing.

wrt alpha materials: in addition to canCloak = true, did you actually give your "units" a cloak command while testing...?

[ivand]

I'm not sure tangent/bitangent flip has straight connection to model file type. In example I have had two dae models:
1) Horizontal plane
2) A car

By default (with no flips) the normal mapping for them looks very different:



biTangent flipping is certainly required for the plane, but I'm not sure if car needs it (flipping doesn't change much the way it looks).

Disregard the above.
Now I'm thinking maybe flipping should have been happening by default...

The way I had it originally was B = cross(N,T), but then I found out that B calculated this way was not the same as the one taken from gl_MultiTexCoord6.xyz, so I quickly changed the order, so now it's B = cross(T, N). This way they both gave the same result:
https://github.com/lhog/Zero-K/blob/pbr ... rt#L51-L53
https://github.com/lhog/Zero-K/blob/pbr ... r.vert#L59

Also it gave consistent result with the piece of code that calculated TBN in fragment shader (I did that being unaware that tangent/bitangent was always supplied):
https://github.com/lhog/Zero-K/blob/pbr ... #L283-L313
Anyway, now if I found out that everything needs to be flipped, this means that B = cross(N,T) was right and B = cross(T, N), which gives the same result as engine's bitangent is wrong. I'm confused a.t.m, so will have a thorough look and report back.

As far as alpha. What I did was:
1) Added canCloak = true to car model.
2) Temporarily changed opaque to alpha in one, other or both places here:
https://github.com/lhog/Zero-K/blob/pbr ... s.lua#L287 and
https://github.com/lhog/Zero-K/blob/pbr ... s.lua#L446
3) Looked at visuals when unit was visible and when it was cloaked (pressed cloak button). As far as I recall if I cloaked unit it didn't get any screen representation at all (nothing was drawn).