Splatoon in Unity

Infinite Splatoon Style Splatting In Unity

Download the 2018.3 repo: Splatoonity on GitHub

The basic idea

This works a little bit like deferred decals meets light maps.  With deferred decals, the world position is figured out from the depth buffer and then is transformed into decal space.  Then the decal can be sampled and applied.  But this won't work with areas that are not on the screen, it also doesn't save the decals.  You would have to draw every single decal, every frame, and that would start to slow you frame rate after a while.  Plus you would have a hard time figuring out how much area each color was taking up because decals can go on top of other decals and you can't really check how much actual space a decal is covering.

What we need to do is figure out a way to consolidate all the splats and then draw them all at once on the world.  Kinda like how a light map works, but for decals.

Get the world Position

First we need to have the world position, since you can't draw decals without knowing where to draw them.  To do that we draw the model to a ARGBFloat render texture, outputting it's world position in the pixel shader.  But drawing the model as is won't do, we need to draw it as if it's second uv channel were its position.

When a model gets rendered the vertex shader takes the vertex positions and maps them to the screen based the camera with this little bit of code:

o.pos = UnityObjectToClipPos(v.vertex);

But you don't have to use the vertex position of the model, you can use whatever you want.  In this case we take the second uv channel and using that as the position.

float3 uvWorldPos = float3( v.texcoord1.xy * 2.0 - 1.0, 0.5 );
o.pos = mul( UNITY_MATRIX_VP, float4( uvWorldPos, 1.0 ) );
o.worldPos = mul(unity_ObjectToWorld, v.vertex).xyz;

It looks a little bit different but uvWorldPos is basically unwrapping the model and putting it in front of an orthographic camera.  The camera that draws this will need to be at the center of the world and pointing in the correct direction in order to see the model.  The actual world position is passed down and is written out in the pixel shader.

World position texture

Why render out a world position texture?  Well because once it's rendered we don't need to worry about the model anymore.  There could be lots of models, who knows.  And more models = more draw calls.  As long as it doesn't move we don't need to bother with it.  A tangent map and binormal map are also generated in the same way and stored for later use when generating a normal map for the edges of the splats. Use ddx and ddy of the world position to generate the tangents and binormals for the second uv set.  This is optional though as there are other ways of getting world normals without pre-computed tangents.  I'll talk about that later.

float3 worldTangent = normalize( ddx( i.worldPos ) ) * 0.5 + 0.5;

float3 worldBinormal = normalize( ddy( i.worldPos ) ) * 0.5 + 0.5;

Assemble the decals
Just as if you were drawing deferred decals, you need to collect them all in one decal manager and then tell each one to render.  We use a static instance splat manager and whatever wants to draw a splat adds it's splat to the splat manager.  The biggest difference with a decal manager is that we only need to add the splat once, not every frame.

Once all the splats are assembled they can be blit to a splat texture, referencing the world texture just like deferred decals reference the depth buffer.

The splats get drawn to alternating textures (ping pong buffers) so that new splats can be custom blended with old splats.  The world position is sampled from the baked texture and is multiplied by each splat transform matrix.  Each splat color needs to remove any previous splat colors to keep track of score or eventually everything would be covered with every color.

float4 currentSplat = tex2D(_LastSplatTex, i.uv);
float4 wpos = tex2D(_WorldPosTex, i.uv);

for( int i = 0; i < _TotalSplats; i++ ){
 float3 opos = mul(_SplatMatrix[i], float4(wpos.xyz,1)).xyz;

 // skip if outside of projection volume
 if( opos.x > -0.5 && opos.x < 0.5 && opos.y > -0.5 && opos.y < 0.5 && opos.z > -0.5 && opos.z < 0.5 ){
  // generate splat uvs
  float2 uv = saturate( opos.xz + 0.5 );
  uv *= _SplatScaleBias[i].xy;
  uv += _SplatScaleBias[i].zw;
    
  // sample the texture
  float newSplatTex = tex2D( _MainTex, uv ).x;
  newSplatTex = saturate( newSplatTex - abs( opos.y ) * abs( opos.y ) );
  currentSplat = min( currentSplat, 1.0 - newSplatTex * ( 1.0 - _SplatChannelMask[i] ) );
  currentSplat = max( currentSplat, newSplatTex * _SplatChannelMask[i]);
 }

}

// mask based on world coverage
// needed for accurate score calculation
return currentSplat * wpos.w;

Just like light maps this splat map is pretty low resolution and not detailed enough to look good on it's own.  Thankfully we just need smooth edges and not per pixel details, something that distance field textures are good at.  Below is the atlas of distance field textures for the splat decals and the multi channel distance field for the final splat map.

Splat distance field decal textures

Splat decals applied to splat map

Updating the score
To update the score we downsample the splat map first to a 256x256 texture with generated mip maps using a shader that steps the distance field at 0.5 to ensure that the score will mimic what is scene in came, and then again to a 4x4 texture.  Then we sample the colors, average them together and set the score based on the brightness of each channel.

This is done in a co-routine that spreads out the work over multiple frames since we don't need it to be super responsive.  The co-routine updates the score continually once every second.


Drawing the splats (surface shader)
Now that we have a distance field splat map we can easily sample the splats in the material shader.  We sample the splat texture using the second uv set which we can get by putting uv2 in front of the splat texture sample name in the input struct:

struct Input {
 float2 uv_MainTex;
 float2 uv2_SplatTex;
 float3 worldNormal;
 float3 worldTangent;
 float3 worldBinormal;
 float3 worldPos;
 INTERNAL_DATA
};

We sample the splat texture and also one texel up and one texel over to create a normal offset map

// Sample splat map texture with offsets
float4 splatSDF = tex2D (_SplatTex, IN.uv2_SplatTex);
float4 splatSDFx = tex2D (_SplatTex, IN.uv2_SplatTex + float2(_SplatTex_TexelSize.x,0) );
float4 splatSDFy = tex2D (_SplatTex, IN.uv2_SplatTex + float2(0,_SplatTex_TexelSize.y) );

Because the distance field edge is created in the shader, when viewed at harsh angles or from a far distance the edges can become smaller than one pixel which aliasess and doesn't look very good.  This code tries to create an edge width that will not alias.  This is similar to signed distance field text rendering.  It's not perfect and doesn't help with specular aliasing.

// Use ddx ddy to figure out a max clip amount to keep edge aliasing at bay when viewing from extreme angles or distances
half splatDDX = length( ddx(IN.uv2_SplatTex * _SplatTex_TexelSize.zw) );
half splatDDY = length( ddy(IN.uv2_SplatTex * _SplatTex_TexelSize.zw) );
half clipDist = sqrt( splatDDX * splatDDX + splatDDY * splatDDY );
half clipDistHard = max( clipDist * 0.01, 0.01 );
half clipDistSoft = 0.01 * _SplatEdgeBumpWidth;

We smoothstep the splat distance field to create a crisp but soft edge for each channel.  Each channel must bleed over itself just a little bit to ensure there are no holes when splats of different colors meet.  A second smooth step is done to create a mask for the splat edges.

// Smoothstep to make a soft mask for the splats
float4 splatMask = smoothstep( ( _Clip - 0.01 ) - clipDistHard, ( _Clip - 0.01 ) + clipDistHard, splatSDF );
float splatMaskTotal = max( max( splatMask.x, splatMask.y ), max( splatMask.z, splatMask.w ) );

// Smoothstep to make the edge bump mask for the splats
float4 splatMaskInside = smoothstep( _Clip - clipDistSoft, _Clip + clipDistSoft, splatSDF );
splatMaskInside = max( max( splatMaskInside.x, splatMaskInside.y ), max( splatMaskInside.z, splatMaskInside.w ) );

Now we create a normal offset for each channel of the splat map and combine them all into a single normal offset.  Also we can sample a tiling normal map to give the splatted areas some texture.  Note that the _SplatTileNormalTex is uncompressed just because I think it looks better with glossy surfaces.  This normal offset is in the tangent space of the second uv channel and we need to get it into the tangent space of the first uv channel to combine with the regular material's bump map.

// Create normal offset for each splat channel
float4 offsetSplatX = splatSDF - splatSDFx;
float4 offsetSplatY = splatSDF - splatSDFy;

// Combine all normal offsets into single offset
float2 offsetSplat = lerp( float2(offsetSplatX.x,offsetSplatY.x), float2(offsetSplatX.y,offsetSplatY.y), splatMask.y );
offsetSplat = lerp( offsetSplat, float2(offsetSplatX.z,offsetSplatY.z), splatMask.z );
offsetSplat = lerp( offsetSplat, float2(offsetSplatX.w,offsetSplatY.w), splatMask.w );
offsetSplat = normalize( float3( offsetSplat, 0.0001) ).xy; // Normalize to ensure parity between texture sizes
offsetSplat = offsetSplat * ( 1.0 - splatMaskInside ) * _SplatEdgeBump;

// Add some extra bump over the splat areas
float2 splatTileNormalTex = tex2D( _SplatTileNormalTex, IN.uv2_SplatTex * 10.0 ).xy;
offsetSplat += ( splatTileNormalTex.xy - 0.5 ) * _SplatTileBump  * 0.2;

First we need to get the splat edge normal into world space.  There's two ways of going about this.  The first is to generate and store the tangents elsewhere, which is what I originally did and is included in the package.  The second is to compute the the world normal without tangents which is also included in the package but is commented out.  Depending on what your bottlenecks are (memory vs instructions) you can pick which technique to use.  They both produce similar results.

The tangent-less normals were implemented from the example in this blog post:

Followup: Normal Mapping Without Precomputed Tangents

// Create the world normal of the splats
#if 0
 // Use tangentless technique to get world normals
 float3 worldNormal = WorldNormalVector (IN, float3(0,0,1) );
 float3 offsetSplatLocal2 = normalize( float3( offsetSplat, sqrt( 1.0 - saturate( dot( offsetSplat, offsetSplat ) ) ) ) );
 float3 offsetSplatWorld = perturb_normal( offsetSplatLocal2, worldNormal, normalize( IN.worldPos - _WorldSpaceCameraPos ), IN.uv2_SplatTex );
#else
 // Sample the world tangent and binormal textures for texcoord1 (the second uv channel)
 // you could skip the binormal texture and cross the vertex normal with the tangent texture to get the bitangent
 float3 worldTangentTex = tex2D ( _WorldTangentTex, IN.uv2_SplatTex ).xyz * 2.0 - 1.0;
 float3 worldBinormalTex = tex2D ( _WorldBinormalTex, IN.uv2_SplatTex ).xyz * 2.0 - 1.0;

 // Create the world normal of the splats
 float3 offsetSplatWorld = offsetSplat.x * worldTangentTex + offsetSplat.y * worldBinormalTex;
#endif

Now that the splat edge normal is in world space we need to get it into the original tangent space.

// Get the tangent and binormal for the texcoord0 (this is just the actual tangent and binormal that comes in from the vertex shader)
float3 worldTangent = WorldNormalVector (IN, float3(1,0,0) );
float3 worldBinormal = WorldNormalVector (IN, float3(0,1,0) );

// Convert the splat world normal to tangent normal for texcood0
float3 offsetSplatLocal = 0.0001;
offsetSplatLocal.x = dot( worldTangent, offsetSplatWorld );
offsetSplatLocal.y = dot( worldBinormal, offsetSplatWorld );
offsetSplatLocal = normalize( offsetSplatLocal );

Talk about a roundabout solution. Now we can sample the main material normal and combine it with the splat normal.

// sample the normal map for the main material
float4 normalMap = tex2D( _BumpTex, IN.uv_MainTex );
normalMap.xyz = UnpackNormal( normalMap );
float3 tanNormal = normalMap.xyz;

// Add the splat normal to the tangent normal
tanNormal.xy += offsetSplatLocal * splatMaskTotal;
tanNormal = normalize( tanNormal );

Sample the albedo texture and lerp it with the 4 splat colors using the splat mask

// Albedo comes from a texture tinted by color
float4 MainTex = tex2D (_MainTex, IN.uv_MainTex );
fixed4 c = MainTex * _Color;

// Lerp the color with the splat colors based on the splat mask channels
c.xyz = lerp( c.xyz, float3(1.0,0.5,0.0), splatMask.x );
c.xyz = lerp( c.xyz, float3(1.0,0.0,0.0), splatMask.y );
c.xyz = lerp( c.xyz, float3(0.0,1.0,0.0), splatMask.z );
c.xyz = lerp( c.xyz, float3(0.0,0.0,1.0), splatMask.w );

All that's left is to output the surface values.

o.Albedo = c.rgb;
o.Normal = tanNormal;
o.Metallic = _Metallic;
o.Smoothness = lerp( _Glossiness, 0.7, splatMaskTotal );
o.Alpha = c.a;

Final result

And that's all there is to it!