Saturday, April 15, 2017

Splatoon in Unity

Infinite Splatoon Style Splatting In Unity

Download the 2018.3 repo: Splatoonity on GitHub

The basic idea
This works a little bit like deferred decals meets light maps.  With deferred decals, the world position is figured out from the depth buffer and then is transformed into decal space.  Then the decal can be sampled and applied.  But this won't work with areas that are not on the screen, it also doesn't save the decals.  You would have to draw every single decal, every frame, and that would start to slow you frame rate after a while.  Plus you would have a hard time figuring out how much area each color was taking up because decals can go on top of other decals and you can't really check how much actual space a decal is covering.

What we need to do is figure out a way to consolidate all the splats and then draw them all at once on the world.  Kinda like how a light map works, but for decals.


Get the world Position
First we need to have the world position, since you can't draw decals without knowing where to draw them.  To do that we draw the model to a ARGBFloat render texture, outputting it's world position in the pixel shader.  But drawing the model as is won't do, we need to draw it as if it's second uv channel were its position.

When a model gets rendered the vertex shader takes the vertex positions and maps them to the screen based the camera with this little bit of code:

o.pos = UnityObjectToClipPos(v.vertex);
But you don't have to use the vertex position of the model, you can use whatever you want.  In this case we take the second uv channel and using that as the position.

float3 uvWorldPos = float3( v.texcoord1.xy * 2.0 - 1.0, 0.5 );
o.pos = mul( UNITY_MATRIX_VP, float4( uvWorldPos, 1.0 ) );
o.worldPos = mul(unity_ObjectToWorld, v.vertex).xyz;
It looks a little bit different but uvWorldPos is basically unwrapping the model and putting it in front of an orthographic camera.  The camera that draws this will need to be at the center of the world and pointing in the correct direction in order to see the model.  The actual world position is passed down and is written out in the pixel shader.

World position texture

Why render out a world position texture?  Well because once it's rendered we don't need to worry about the model anymore.  There could be lots of models, who knows.  And more models = more draw calls.  As long as it doesn't move we don't need to bother with it.  A tangent map and binormal map are also generated in the same way and stored for later use when generating a normal map for the edges of the splats. Use ddx and ddy of the world position to generate the tangents and binormals for the second uv set.  This is optional though as there are other ways of getting world normals without pre-computed tangents.  I'll talk about that later.

float3 worldTangent = normalize( ddx( i.worldPos ) ) * 0.5 + 0.5;
float3 worldBinormal = normalize( ddy( i.worldPos ) ) * 0.5 + 0.5;

Assemble the decals
Just as if you were drawing deferred decals, you need to collect them all in one decal manager and then tell each one to render.  We use a static instance splat manager and whatever wants to draw a splat adds it's splat to the splat manager.  The biggest difference with a decal manager is that we only need to add the splat once, not every frame.

Once all the splats are assembled they can be blit to a splat texture, referencing the world texture just like deferred decals reference the depth buffer.

The splats get drawn to alternating textures (ping pong buffers) so that new splats can be custom blended with old splats.  The world position is sampled from the baked texture and is multiplied by each splat transform matrix.  Each splat color needs to remove any previous splat colors to keep track of score or eventually everything would be covered with every color.

float4 currentSplat = tex2D(_LastSplatTex, i.uv);
float4 wpos = tex2D(_WorldPosTex, i.uv);

for( int i = 0; i < _TotalSplats; i++ ){
 float3 opos = mul(_SplatMatrix[i], float4(wpos.xyz,1)).xyz;

 // skip if outside of projection volume
 if( opos.x > -0.5 && opos.x < 0.5 && opos.y > -0.5 && opos.y < 0.5 && opos.z > -0.5 && opos.z < 0.5 ){
  // generate splat uvs
  float2 uv = saturate( opos.xz + 0.5 );
  uv *= _SplatScaleBias[i].xy;
  uv += _SplatScaleBias[i].zw;
    
  // sample the texture
  float newSplatTex = tex2D( _MainTex, uv ).x;
  newSplatTex = saturate( newSplatTex - abs( opos.y ) * abs( opos.y ) );
  currentSplat = min( currentSplat, 1.0 - newSplatTex * ( 1.0 - _SplatChannelMask[i] ) );
  currentSplat = max( currentSplat, newSplatTex * _SplatChannelMask[i]);
 }

}

// mask based on world coverage
// needed for accurate score calculation
return currentSplat * wpos.w;
Just like light maps this splat map is pretty low resolution and not detailed enough to look good on it's own.  Thankfully we just need smooth edges and not per pixel details, something that distance field textures are good at.  Below is the atlas of distance field textures for the splat decals and the multi channel distance field for the final splat map.
Splat distance field decal textures

Splat decals applied to splat map

Updating the score
To update the score we downsample the splat map first to a 256x256 texture with generated mip maps using a shader that steps the distance field at 0.5 to ensure that the score will mimic what is scene in came, and then again to a 4x4 texture.  Then we sample the colors, average them together and set the score based on the brightness of each channel.

This is done in a co-routine that spreads out the work over multiple frames since we don't need it to be super responsive.  The co-routine updates the score continually once every second.


Drawing the splats (surface shader)
Now that we have a distance field splat map we can easily sample the splats in the material shader.  We sample the splat texture using the second uv set which we can get by putting uv2 in front of the splat texture sample name in the input struct:

struct Input {
 float2 uv_MainTex;
 float2 uv2_SplatTex;
 float3 worldNormal;
 float3 worldTangent;
 float3 worldBinormal;
 float3 worldPos;
 INTERNAL_DATA
};
We sample the splat texture and also one texel up and one texel over to create a normal offset map

// Sample splat map texture with offsets
float4 splatSDF = tex2D (_SplatTex, IN.uv2_SplatTex);
float4 splatSDFx = tex2D (_SplatTex, IN.uv2_SplatTex + float2(_SplatTex_TexelSize.x,0) );
float4 splatSDFy = tex2D (_SplatTex, IN.uv2_SplatTex + float2(0,_SplatTex_TexelSize.y) );
Because the distance field edge is created in the shader, when viewed at harsh angles or from a far distance the edges can become smaller than one pixel which aliasess and doesn't look very good.  This code tries to create an edge width that will not alias.  This is similar to signed distance field text rendering.  It's not perfect and doesn't help with specular aliasing.

// Use ddx ddy to figure out a max clip amount to keep edge aliasing at bay when viewing from extreme angles or distances
half splatDDX = length( ddx(IN.uv2_SplatTex * _SplatTex_TexelSize.zw) );
half splatDDY = length( ddy(IN.uv2_SplatTex * _SplatTex_TexelSize.zw) );
half clipDist = sqrt( splatDDX * splatDDX + splatDDY * splatDDY );
half clipDistHard = max( clipDist * 0.01, 0.01 );
half clipDistSoft = 0.01 * _SplatEdgeBumpWidth;
We smoothstep the splat distance field to create a crisp but soft edge for each channel.  Each channel must bleed over itself just a little bit to ensure there are no holes when splats of different colors meet.  A second smooth step is done to create a mask for the splat edges.

// Smoothstep to make a soft mask for the splats
float4 splatMask = smoothstep( ( _Clip - 0.01 ) - clipDistHard, ( _Clip - 0.01 ) + clipDistHard, splatSDF );
float splatMaskTotal = max( max( splatMask.x, splatMask.y ), max( splatMask.z, splatMask.w ) );

// Smoothstep to make the edge bump mask for the splats
float4 splatMaskInside = smoothstep( _Clip - clipDistSoft, _Clip + clipDistSoft, splatSDF );
splatMaskInside = max( max( splatMaskInside.x, splatMaskInside.y ), max( splatMaskInside.z, splatMaskInside.w ) );
Now we create a normal offset for each channel of the splat map and combine them all into a single normal offset.  Also we can sample a tiling normal map to give the splatted areas some texture.  Note that the _SplatTileNormalTex is uncompressed just because I think it looks better with glossy surfaces.  This normal offset is in the tangent space of the second uv channel and we need to get it into the tangent space of the first uv channel to combine with the regular material's bump map.

// Create normal offset for each splat channel
float4 offsetSplatX = splatSDF - splatSDFx;
float4 offsetSplatY = splatSDF - splatSDFy;

// Combine all normal offsets into single offset
float2 offsetSplat = lerp( float2(offsetSplatX.x,offsetSplatY.x), float2(offsetSplatX.y,offsetSplatY.y), splatMask.y );
offsetSplat = lerp( offsetSplat, float2(offsetSplatX.z,offsetSplatY.z), splatMask.z );
offsetSplat = lerp( offsetSplat, float2(offsetSplatX.w,offsetSplatY.w), splatMask.w );
offsetSplat = normalize( float3( offsetSplat, 0.0001) ).xy; // Normalize to ensure parity between texture sizes
offsetSplat = offsetSplat * ( 1.0 - splatMaskInside ) * _SplatEdgeBump;

// Add some extra bump over the splat areas
float2 splatTileNormalTex = tex2D( _SplatTileNormalTex, IN.uv2_SplatTex * 10.0 ).xy;
offsetSplat += ( splatTileNormalTex.xy - 0.5 ) * _SplatTileBump  * 0.2;
First we need to get the splat edge normal into world space.  There's two ways of going about this.  The first is to generate and store the tangents elsewhere, which is what I originally did and is included in the package.  The second is to compute the the world normal without tangents which is also included in the package but is commented out.  Depending on what your bottlenecks are (memory vs instructions) you can pick which technique to use.  They both produce similar results.

The tangent-less normals were implemented from the example in this blog post:

// Create the world normal of the splats
#if 0
 // Use tangentless technique to get world normals
 float3 worldNormal = WorldNormalVector (IN, float3(0,0,1) );
 float3 offsetSplatLocal2 = normalize( float3( offsetSplat, sqrt( 1.0 - saturate( dot( offsetSplat, offsetSplat ) ) ) ) );
 float3 offsetSplatWorld = perturb_normal( offsetSplatLocal2, worldNormal, normalize( IN.worldPos - _WorldSpaceCameraPos ), IN.uv2_SplatTex );
#else
 // Sample the world tangent and binormal textures for texcoord1 (the second uv channel)
 // you could skip the binormal texture and cross the vertex normal with the tangent texture to get the bitangent
 float3 worldTangentTex = tex2D ( _WorldTangentTex, IN.uv2_SplatTex ).xyz * 2.0 - 1.0;
 float3 worldBinormalTex = tex2D ( _WorldBinormalTex, IN.uv2_SplatTex ).xyz * 2.0 - 1.0;

 // Create the world normal of the splats
 float3 offsetSplatWorld = offsetSplat.x * worldTangentTex + offsetSplat.y * worldBinormalTex;
#endif
Now that the splat edge normal is in world space we need to get it into the original tangent space.

// Get the tangent and binormal for the texcoord0 (this is just the actual tangent and binormal that comes in from the vertex shader)
float3 worldTangent = WorldNormalVector (IN, float3(1,0,0) );
float3 worldBinormal = WorldNormalVector (IN, float3(0,1,0) );

// Convert the splat world normal to tangent normal for texcood0
float3 offsetSplatLocal = 0.0001;
offsetSplatLocal.x = dot( worldTangent, offsetSplatWorld );
offsetSplatLocal.y = dot( worldBinormal, offsetSplatWorld );
offsetSplatLocal = normalize( offsetSplatLocal );
Talk about a roundabout solution. Now we can sample the main material normal and combine it with the splat normal.

// sample the normal map for the main material
float4 normalMap = tex2D( _BumpTex, IN.uv_MainTex );
normalMap.xyz = UnpackNormal( normalMap );
float3 tanNormal = normalMap.xyz;

// Add the splat normal to the tangent normal
tanNormal.xy += offsetSplatLocal * splatMaskTotal;
tanNormal = normalize( tanNormal );
Sample the albedo texture and lerp it with the 4 splat colors using the splat mask

// Albedo comes from a texture tinted by color
float4 MainTex = tex2D (_MainTex, IN.uv_MainTex );
fixed4 c = MainTex * _Color;

// Lerp the color with the splat colors based on the splat mask channels
c.xyz = lerp( c.xyz, float3(1.0,0.5,0.0), splatMask.x );
c.xyz = lerp( c.xyz, float3(1.0,0.0,0.0), splatMask.y );
c.xyz = lerp( c.xyz, float3(0.0,1.0,0.0), splatMask.z );
c.xyz = lerp( c.xyz, float3(0.0,0.0,1.0), splatMask.w );
All that's left is to output the surface values.

o.Albedo = c.rgb;
o.Normal = tanNormal;
o.Metallic = _Metallic;
o.Smoothness = lerp( _Glossiness, 0.7, splatMaskTotal );
o.Alpha = c.a;
Final result
And that's all there is to it!

50 comments:

  1. How to clear all splats at once ?

    ReplyDelete
    Replies
    1. Just blit a black texture to the splat render target to clear all splats at once.

      Delete
  2. Great job!!
    This is what I'm searching for a while...
    Thank you for sharing it

    ReplyDelete
  3. Can I have multiple 3D objects with different materials and splat over them?

    ReplyDelete
    Replies
    1. I tried to duplicate the SplatMaterial and assign it to another GameObject but it doesn't work.
      Can I use the only one SplatManager class for every 3D objects?
      It would be very useful if you show us an example.
      Thank you in advance.

      Delete
    2. The provided example is a proof of concept, not a complete product. You will have to expand upon it to suit the specific needs of your project.

      Delete
  4. How do you sync those splats in multiplayer(Photon)?

    ReplyDelete
    Replies
    1. Each client has to maintain its own splat map. You would have to tell each client to draw the same splats, not send splat map information over the network. You could then have each client calculate the score and average them and throw out any outliers in case the results are different.

      Delete
  5. Very nicely :D
    How i can save it?

    ReplyDelete
  6. This is so neat! Do you know what might be causing splats to be cut in half sometimes? like here: https://ibb.co/e8fv6m

    ReplyDelete
    Replies
    1. I wouldn't know from that one picture. Double check the uvs.

      Delete
    2. This comment has been removed by the author.

      Delete
  7. My research led me to conclusion that the cut off happens everytime this if does not trigger:

    if (leftVec.magnitude > 0.001f)
    {
    newSplatObject.transform.rotation = Quaternion.LookRotation(leftVec, hit.normal);
    }
    does this rings any bell?

    ReplyDelete
    Replies
    1. That's a check to make sure that the Quaternion.LookRotation will work and not throw an error, try replacing it with:

      if( leftVec.magnitude > 0.001f ){
      newSplatObject.transform.rotation = Quaternion.LookRotation( leftVec, hit.normal );
      }else{
      newSplatObject.transform.rotation = Quaternion.LookRotation( Vector3.left, Vector3.up );
      }

      Delete
  8. Can this possibly run smoothly on mobile devices in general?

    ReplyDelete
  9. Hello, how can I make it work in any 3d model? I tried everything but I could not, I must have done something wrong!

    ReplyDelete
  10. How to use it on another GameObject, what are the prerequisites?

    ReplyDelete
  11. Am super amateur with shaders to the point that i don't know where to put all of this code other than in the v2f vert function thing. Is there a place where I can just download the shader itself?

    ReplyDelete
    Replies
    1. He's included a unity package at the very beginning, just under the image.

      Delete
    2. I am greatly inspired by your example. I am eager to learn the project you provided. Unfortunately, the unitypackage you provided cannot be opened. Could you please provide a github or send it to my email? Thanks again for sharing

      Delete
  12. Failed to import package with error: Couldn't decompress package
    Unity2017.4 Unity5.3 can not open this exzample....

    ReplyDelete
  13. Is there a way to splat paint of other materials? (i'm using a cel-shaded material)

    ReplyDelete
  14. How do you splat paint on different materials/objects?

    ReplyDelete
  15. Hey Michael, the download links 404 for me

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. Am I correct in the assumption that you issue a draw call covering the entire splat map, which loops through all accumilated decals? That sounds very inefficient to me. It would especially become a problem if your mesh is big and you have to bump the resolution of the render target.
    Is there a way to limit the number of pixels you render?

    ReplyDelete
    Replies
    1. It does issue a draw call for any frame that a splat is added, it will add up to 10 splats per draw call and will just run through any backlog of splats in the next frames.

      You could figure out which meshes are getting splats and render them to the splat map instead of using a world position texture but then you would have to figure something out to bleed the texture. For bleeding you could use the alpha as coverage with only 3 splat channels, or have a separate coverage texture, and do the bleeding right in the surface shader.

      There's lots of improvements that can be made, this is just the simplest implementation for the sake of example.

      Delete
    2. Thanks for the clarification.

      I was asking here about a more efficient implementation, because I couldn't find any in-depth information on how the technique works in the first place.

      Can you elaborate on what you mean with "bleed the texture"? I saw you mentioned in the article, however don't fully understand it.

      Delete
  18. Anyway we can get a mirror to the unitypackage? The link posted does not work anymore
    Thanks!

    ReplyDelete
  19. Hi Michael,

    If you could upload the unitypackage again, it would be really awesome!
    Thank you for your help

    ReplyDelete
  20. Hey Michael,
    Big thanks for uploading this. My friends and I used this for our game and we couldn't have done it without you
    Thanks,
    VGDA CSULB

    ReplyDelete
  21. Replies
    1. Link is at the top:
      https://github.com/SquirrelyJones/Splatoonity

      Delete
    2. This comment has been removed by the author.

      Delete
  22. This comment has been removed by the author.

    ReplyDelete
  23. Minus the messed up lighting, would you know why this is happening?

    https://gyazo.com/00273db8e62d45fb32792716fb06d7ae

    I fully filled in where I could splat at on my texture. Maybe its unwrapping poorly?

    ReplyDelete
    Replies
    1. You can remove this comments or keep them up.

      So the problem was I still had a test prefab which used the same wrapping. So when I colored the other prefab, it filled in the rest of the missing areas in my example. When removing the test prefab it works correctly now.

      To everyone wondering why it isnt working on their own 3d models, it is because you have to regenerate the lighting. Use his test scene to look at the lighting details ( In Unity: Window> Rendering> Lighting settings), then generate, then enable auto-generate. Whenever you have an index error in the console you have to manually regenerate the lighting. Else use OP's shader. make sure theres a receiver on what you want splat on. Make sure you have a splatmanager just on like a gamecontroller object, and also on the camera, the splat maker example script. Also on the splat manager script, make sure the splat texture is: splats

      Delete
  24. Would you mind explain about the "BleedTexture()" method, I dont really understand why you bleed the world position out 2 pixels.

    ReplyDelete
  25. Would it be possible to convert this shader to URP?

    ReplyDelete
  26. Great stuff! Id like to also know if this can be easily converted to be compatible with URP -

    ReplyDelete
  27. I can't seem to figure out whats going on here:
    float3 uvWorldPos = float3( v.texcoord1.xy * 2.0 - 1.0, 0.5 );
    Why are you doing that math to the UV coordinates to turn it into "world pos".

    ReplyDelete
  28. How to Play Pai Gow Poker | BetRivers Casino - Wolverione
    Pai Gow Poker 1xbet app is an online version of 토토사이트 a worrione traditional table game in which players place bets in the background. Pai Gow Poker uses only the symbols หารายได้เสริม from a www.jtmhub.com

    ReplyDelete