Cel Shader with Outline Effect in Unity

finished dog shader

Woohoo, our first ever development blog post! Today, I want to talk about how I wrote the cel shader that we’ll use for most of the materials in the game, and how I added the outline/ highlight effect for when the player clicks on a dog.

If you like this post, be sure to follow me on Twitter 🙂

If you’ve never written shader code before, or used Unity, you still may get plenty out of this post, because the algorithms used to write the cel shader and the outline are applicable to any shader. If you do write shader code in Unity, you’ll find the rest of the techniques extra helpful 🙂

If you have any questions or critiques, feel free to comment. This is my first time writing shader code EVER, so I’m happy to receive constructive criticism.

This is the end result, applied to an adorable Boston Terrier model made by our lead artist Kytana Le:

finished dog shader


The High Level

The cel shader was written in Unity’s shader language, which is almost exactly like HLSL, but with some preprocesser tags and formatting that let you access Unity features.

The shader has two passes: one that applies the texture and lighting, and one that applies the outline. The shader’s input, which are configured in the Unity editor, are the main texture, the ramp texture (used for lighting), and the outline size and color.

In short, the outline is achieved by drawing a scaled version of the mesh “behind” the original mesh. We do two drawings of the mesh by using two passes: one for the regular lighting pass, and one for the outline pass.

The Regular Lighting Pass

Please excuse my messily combined c++ ish/ hlsl ish pseudocode.

Lighting Algorithm

float lightingScalar = dot(input.normal, lightDirection)
float3 lightingColor = rampTexture->getColorAtXY(lightingScalar, 0.5)
float3 outputColor = input.color * lightingColor;

The texture/lighting pass is nothing advanced. To calculate lighting, it takes the dot product of the surface normal and light direction to result in a normalized (0-1) scaler, which it then uses to sample a position on the ramp shader on the x-axis.

The ramp shader is a 2d texture that only has two colors on it: dark blue on the left, and white on the right. The position on the x-axis of the ramp shader then corresponds to either of those colors. Areas on the model which are less exposed to light (in shadow) thus get the dark blue color, and areas on the model which are more exposed to the light get the white color.

This color is multiplied with the input texture’s albedo. Boom! You’ve got lighting applied. And since lighting is one of two colors (dark blue or white), you get a cel-shaded effect with sharp edges on the shadows and no blending.

The Outline Pass

Basically, the z-buffer is used to place objects visually in front of or behind each other in a simulated 3D space, and the stencil buffer is used to do special effects to override the z-buffer.

The outline is achieved by drawing a scaled version of the original mesh after the original mesh, and using the stencil buffer to not draw the outline where the original has already been drawn. This gives the effect of the outline being “behind” the regular color & lighting pass.

First, let’s look at how to scale the mesh:

Outline Algorithm

float4 extrudedVertex = input.vertex + (input.normal * outlineSize)

Achieving the scaled version of the mesh is simple: scale each vertex along its normal direction by outlineSize.

Scaling along the normals assures that the outline mesh will scale evenly around the entire object mesh, so even concave parts appear covered by an even thickness outline. The larger outlineSize is, the thicker the outline will be.

Now, let’s draw the outline “behind” the regular lighting pass. Let’s look at the settings for the z-buffer and stencil buffer for each pass:

Lighting/ Texture Pass
stencilRef: 4
compare: always
pass: replace
zfail: keep


Outline Pass
culling: off
zwrite: off
ztest: on
stencilRef: 4
compare: notequal
fail: keep
pass: replace

The first pass (the lighting and texturing pass) writes to the stencil buffer. (4 is an arbitrary reference number.)

The second pass (the outline pass) reads the stencil buffer, and, where it sees the same reference already written, keeps the original pixel in place. (In our case, the pixel written by the regular lighting pass.) This achieves the effect of making the scaled mesh look like it’s “behind”- or only contouring- the original mesh. I also make sure to turn culling off, so that we don’t discard any vertexes of the outline mesh.

Finally, each pixel of the outline mesh is filled in with the same solid color. We don’t really need lighting for a cartoonish outline. Ta-daa!!! A beautiful outline.

Mistakes LEARNED from

Mistake #1: Blended cel shading. This is clearly a rookie mistake from a first-time shader writer. My first try at the cel shader had some blended edges on the shadows, which is not what we want for a hard cel look. The mistake I made was doing the lighting calculation in the vertex shader step. Because the vertex shader is per-vertex, every pixel in between is linearly interpolated, which means you end up with a blended effect for some pixels if you do lighting in the vertex shader. Moving the lighting calculation to the fragment shader, which is calculated per-pixel, fixed this immediately. Funnily enough, not ALL of the faces on the model were blended, which turned out to be because the model was non-manifold, which has to do with mistake #3.

first pass at dog shader

Mitake #2: Uneven outline. You can see in the image above that the outline isn’t smooth and evenly contouring the mesh. The original algorithm I had for scaling the mesh didn’t include the normals- it was just multiplying the vertex by outlineSize. This doesn’t work for convex models, which really any complicated model that isn’t a pure cube or sphere is probably going to be. Scaling along the normals made a nice, even outline.

Mistake #3: Broken pieces of the extruded mesh. When using the low-poly shibe above, and applying the extrusion technique for scaling, there were gaps between each face. This is because the low-poly shibe has some faces of the mesh that aren’t really connected on the edges that they meet at, making it a non-manifold mesh. Then, when the vertexes are extruded, the faces don’t remain visually connected. I had to use a manifold mesh for anything using the outline shader, which is really almost any mesh that isn’t low-poly.

Mistake #4: Missing pieces of the outline. This was because I originally didn’t have the cull:off tag on the outline pass. We don’t want to discard any parts of the outline pass, even if they’re technically “inside” the mesh.

Code

Shader "Custom/CelEffects"
  { 

  Properties
  {
    _MainTex("Texture", 2D) = "white" {}
    _RampTex("Ramp", 2D) = "white" {}
    _Color("Color", Color) = (1, 1, 1, 1)
    _OutlineExtrusion("Outline Extrusion", float) = 0
    _OutlineColor("Outline Color", Color) = (0, 0, 0, 1)
  }

  SubShader
  {

    // Regular color & lighting pass
    Pass
    {

      // Write to Stencil buffer (so that outline pass can read)
      Stencil
      {
        Ref 4
        Comp always
        Pass replace
        ZFail keep
      }

      CGPROGRAM
      #pragma vertex vert
      #pragma fragment frag

      // Properties
      sampler2D _MainTex;
      sampler2D _RampTex;
      float4 _Color;
      float4 _LightColor0; // provided by Unity

      struct vertexInput
      {
        float4 vertex : POSITION;
        float3 normal : NORMAL;
        float3 texCoord : TEXCOORD0;
      };

      struct vertexOutput
      {
        float4 pos : SV_POSITION;
        float3 normal : NORMAL;
        float3 texCoord : TEXCOORD0;
      };

      vertexOutput vert(vertexInput input)
      {
        vertexOutput output;

        // convert input to world space
        output.pos = UnityObjectToClipPos(input.vertex);
        // need float4 to mult with 4x4 matrix
        float4 normal4 = float4(input.normal, 0.0);
        output.normal = normalize(mul(normal4, unity_WorldToObject).xyz);
        output.texCoord = input.texCoord;
        return output;
      }

      float4 frag(vertexOutput input) : COLOR
      {
        // convert light direction to world space & normalize
        // _WorldSpaceLightPos0 provided by Unity
        float3 lightDir = normalize(_WorldSpaceLightPos0.xyz);

        // finds location on ramp texture that we should sample
        // based on angle between surface normal and light direction
        // sample texture for color
        float ramp = clamp(dot(input.normal, lightDir), 0, 1.0);
        float3 lighting = tex2D(_RampTex, float2(ramp, 0.5)).rgb;
        float4 albedo = tex2D(_MainTex, input.texCoord.xy);
        // _LightColor0 provided by Unity
        float3 rgb = albedo.rgb * _LightColor0.rgb * lighting * _Color.rgb; 

        return float4(rgb, 1.0);
      }

      ENDCG
    }

    // Outline pass
    Pass
    {

      Cull OFF
      ZWrite OFF
      ZTest ON

      // Won't draw where it sees ref value 4
      Stencil
      {
        Ref 4
        Comp notequal
        Fail keep
        Pass replace
      }

      CGPROGRAM
      #pragma vertex vert
      #pragma fragment frag

      // Properties
      uniform float4 _OutlineColor;
      uniform float _OutlineSize;
      uniform float _OutlineExtrusion;

      struct vertexInput
      {
        float4 vertex : POSITION;
        float3 normal : NORMAL;
      };

      struct vertexOutput
      {
        float4 pos : SV_POSITION;
        float4 color : COLOR;
      };

      vertexOutput vert(vertexInput input)
      {
        vertexOutput output;
        float4 newPos = input.vertex;

        // normal extrusion technique
        float3 normal = normalize(input.normal); newPos += float4(normal, 0.0) *
            _OutlineExtrusion;

        // convert to world space
        output.pos = UnityObjectToClipPos(newPos);
        output.color = _OutlineColor;

        return output;
      }

      float4 frag(vertexOutput input) : COLOR
      {
        return input.color;
      }

      ENDCG
    }
  }
}

If y’all have any questions about writing shaders in Unity, I’m happy to share as much as I know. I’m not an expert, but I’m always willing to help other indie devs 🙂

Good luck,

Lindsey Reid

@thelindseyreid / @sogoodgames

PS, here’s the Unity graphics settings for this tutorial.

Author: Linden Reid

Game developer and tutorial writer :D

3 thoughts on “Cel Shader with Outline Effect in Unity”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s