Glass Shader with Opaque Edge Effect in Unity

Today, I’m going to explain how I created the shader for glass objects in our game. Glass and other translucent objects are often slightly more opaque at the edges, as demonstrated with this beer glass. It gives a feeling of thickness and solid…ness to an otherwise frail object. And it just looks cool! You could probably use it for a bunch of different effects: jellyfish, jello, light bulbs… really anything translucent.

glass

This shader recreates this effect using basic shader concepts. I created it in Unity’s shader language, but the algorithm could apply to any shader.

If you like this post, be sure to follow us on Twitter @sogoodgames 🙂

Here’s the final product, applied to a Boston terrier modeled by our artist Kytana Le:

glassrotate

THE HIGH LEVEL

As with the cel outline shader from our last post, this shader was written in Unity’s shader language, which is almost exactly like HLSL, but with some preprocesser tags and formatting that let you access Unity features. You need to understand what “surface normal” means to understand this algorithm, but not Unity :p

This shader has one pass and takes in four properties as input. We apply a texture (which is totally optional and not used in the example pictures), a base color, and an edge color with a tunable dropoff rate. The dropoff rate describes how thick the edge appears, or how much of it bleeds from the edge into the center of the model.

Properties
{
_MainTex("Texture", 2D) = "white" {}
_Color("Color", Color) = (1, 1, 1, 1)
_EdgeColor("Edge Color", Color) = (1, 1, 1, 1)
_EdgeThickness("Silouette Dropoff Rate", float) = 1.0
}

The ~brilliant~ algorithm is below. Firstly, we normalize the view direction vector and the surface normal vector. Then, the factor determining how opaque the edge is at this point on the texture is equal to the dot product of the view direction and the surface normal. The closer to being perpendicular that the angle of the view direction and the surface normal are, the lower edgeFactor will be, which we’ll use to make the pixel more opaque. We take the absolute value of the dot product, since the dot product returns a value from -1 to 1, and we don’t want any negative values.

float edgeFactor = abs( dot(viewDirection, surfaceNormal) );

Why does this work? In the image below, the arrows represent the surface normals. If you can imagine a vector representing viewDirection going directly into the screen, you’ll see that arrows closer to the edge of the object are also closer to a 90 degree angle to the viewDirection vector.

surfacenormal.png

We apply this edgeFactor to the pixel’s opacity by dividing the input opacity by the edgeFactor, then raising it to the power of the dropoff rate. Since the edgeFactor is a value from 0 to 1, dividing the input opacity by the edgeFactor has the effect of lowering the opacity value, at lower and lower amount until we get to the absolute edge of the surface where the normals are (almost) perpendicular to the view direction and the edgeFactor is (almost) 0. (But never actually 0, I hope, because then I might get a division-by-0-error.) Raising this value to the power of edgeDropoffRate has the very nice effect of making the edge opacity very low for most of the model, then increase dramatically in opacity at the edge.

float opacity = min( 1.0, color.alpha / edgeFactor );
opacity = pow( opacity, edgeDropoffRate );

Note that we also clamp the opacity to a 0-1 value, to prevent small errors.

You could apply this same algorithm to the edge color, but I found it prettier to linearly combine the albedo color and edge color based on the edge factor.

float oneMinusEdge = 1.0 - edgeFactor;
float3 rgb = (_Color.rgb * edgeFactor) + (_EdgeColor * oneMinusEdge);

Ta-daa, you did it!!

Here’s what the shader can look like with a little tuning to the albedo color, edge color, and edge thickness properties. I think it’s starting to look a little jelly-like, but that’s a post for another time :V

purpledoggo.png

Now I’m getting excited about the possibility of writing a jiggly jello shader with vertex wiggling :p

 

MISTAKES LEARNED FROM

forgotzwriteoff

Mistake #1: Chunks of material missing / inconsistent. You need to make sure ZWrite is Off. We don’t want pixels on the same translucent model to compete for the same spot on the ZBuffer, so don’t allow them to write to it. We do still want them to be able to read, so that other objects can occlude this one. (PS, I may have botched this explanation- kindly write me a correcting comment if I did? :P)

forgotculloff

Mistake #2: Parts of model not showing / overlapping. You also need to turn Culling Off. We want to draw both the front and back faces of the model. Normally, for optimization purposes, culling is set to Cull Back by default, which will not draw the back faces of the model. The back faces are usually the inside of the model, so normally the player doesn’t want to see them. However, we do want see the inside surfaces of this model, since it’s completely translucent! :-0

 

CODE

Shader "Custom/glass"
{
	Properties
	{
		_MainTex("Texture", 2D) = "white" {}
		_Color("Color", Color) = (1, 1, 1, 1)
		_EdgeColor("Edge Color", Color) = (1, 1, 1, 1)
		_EdgeThickness("Silouette Dropoff Rate", float) = 1.0
	}

	SubShader
	{
		Tags
		{
			"Queue" = "Transparent"
		}

		Pass
		{
			Cull Off
			ZWrite Off
			Blend SrcAlpha OneMinusSrcAlpha // standard alpha blending

			CGPROGRAM

			#pragma vertex vert
			#pragma fragment frag

			// Properties
			sampler2D		_MainTex;
			uniform float4	_Color;
			uniform float4	_EdgeColor;
			uniform float   _EdgeThickness;

			struct vertexInput
			{
				float4 vertex : POSITION;
				float3 normal : NORMAL;
				float3 texCoord : TEXCOORD0;
			};

			struct vertexOutput
			{
				float4 pos : SV_POSITION;
				float3 normal : NORMAL;
				float3 texCoord : TEXCOORD0;
				float3 viewDir : TEXCOORD1;
			};

			vertexOutput vert(vertexInput input)
			{
				vertexOutput output;

				// convert input to world space
				output.pos = UnityObjectToClipPos(input.vertex);
				float4 normal4 = float4(input.normal, 0.0);
				output.normal = normalize(mul(normal4, unity_WorldToObject).xyz);
				output.viewDir = normalize(_WorldSpaceCameraPos - mul(unity_ObjectToWorld, input.vertex).xyz);

				output.texCoord = input.texCoord;

				return output;
			}

			float4 frag(vertexOutput input) : COLOR
			{
				// sample texture for color
				float4 texColor = tex2D(_MainTex, input.texCoord.xy);

				// apply silouette equation
				// based on how close normal is to being orthogonal to view vector
				// dot product is smaller the smaller the angle bw the vectors is
				// close to edge = closer to 0
				// far from edge = closer to 1
				float edgeFactor = abs(dot(input.viewDir, input.normal));

				// apply edgeFactor to Albedo color & EdgeColor
				float oneMinusEdge = 1.0 - edgeFactor;
				float3 rgb = (_Color.rgb * edgeFactor) + (_EdgeColor * oneMinusEdge);
				rgb = min(float3(1, 1, 1), rgb); // clamp to real color vals
				rgb = rgb * texColor.rgb;

				// apply edgeFactor to Albedo transparency & EdgeColor transparency
				// close to edge = more opaque EdgeColor & more transparent Albedo
				float opacity = min(1.0, _Color.a / edgeFactor);

				// opacity^thickness means the edge color will be near 0 away from the edges
				// and escalate quickly in opacity towards the edges
				opacity = pow(opacity, _EdgeThickness);
				opacity = opacity * texColor.a;

				float4 output = float4(rgb, opacity);
				return output;
			}

			ENDCG
		}
	}

}

 

As always, if y’all have any questions, feel free to leave them in the comments or tweet me! 😀

Good luck,

Lindsey Reid

@lindseywaslike

@sogoodgames

Cel Shader with Outline Effect in Unity

Woohoo, our first ever development blog post! Today, I want to talk about how I wrote the cel shader that we’ll use for most of the materials in the game, and how I added the outline/ highlight effect for when the player clicks on a dog.

If you like this post, be sure to follow us on Twitter @sogoodgames 🙂

If you’ve never written shader code before, or used Unity, you still may get plenty out of this post, because the algorithms used to write the cel shader and the outline are applicable to any shader. If you do write shader code in Unity, you’ll find the rest of the techniques extra helpful 🙂

If you have any questions or critiques, feel free to comment. This is my first time writing shader code EVER, so I’m happy to receive constructive criticism.

This is the end result, applied to an adorable Boston Terrier model made by our lead artist Kytana Le:

finished dog shader

The High Level

The cel shader was written in Unity’s shader language, which is almost exactly like HLSL, but with some preprocesser tags and formatting that let you access Unity features.

The shader has two passes: one that applies the texture and lighting, and one that applies the outline. The shader’s input, which are configured in the Unity editor, are the main texture, the ramp texture (used for lighting), and the outline size and color.

Please excuse my messily combined c++ ish/ hlsl ish pseudocode.

Lighting Algorithm

float lightingScalar = dot(input.normal, lightDirection)
float3 lightingColor = rampTexture->getColorAtXY(lightingScalar, 0.5)
float3 outputColor = input.color * lightingColor;

The texture/lighting pass is nothing advanced. To calculate lighting, it takes the dot product of the surface normal and light direction to result in a normalized (0-1) scaler, which it then uses to sample a position on the ramp shader on the x-axis. The ramp shader is a 2d texture that only has two colors on it: dark blue on the left, and white on the right. The position on the x-axis of the ramp shader then corresponds to either of those colors. Areas on the model which are less exposed to light (in shadow) thus get the dark blue color, and areas on the model which are more exposed to the light get the white color. This color is multiplied with the input texture’s albedo. Boom! You’ve got lighting applied. And since lighting is one of two colors (dark blue or white), you get a cel-shaded effect with sharp edges on the shadows and no blending.

The outline pass is a satisfying little trick. To do it, I had to learn about how the z-buffer and stencil buffer work. To put it quite basically, the z-buffer is used to place objects visually in front of or behind each other in a simulated 3D space, and the stencil buffer is used to do special effects to override the z-buffer. The outline is achieved by drawing a scaled version of the original mesh after the original mesh, and using the stencil buffer to not draw the outline where the original has already been drawn.

Outline Algorithm

float4 extrudedVertex = input.vertex + (input.normal * outlineSize)

Achieving the scaled version of the mesh is simple: I scale each vertex along its normal direction by outlineSize. Scaling along the normals assures that the outline mesh will scale evenly around the entire object mesh, so even concave parts appear covered by an even thickness outline. The larger outlineSize is, the thicker the outline will be.

Lighting/ Texture Pass
stencilRef: 4
compare: always
pass: replace
zfail: keep


Outline Pass
culling: off
zwrite: off
ztest: on
stencilRef: 4
compare: notequal
fail: keep
pass: replace

Using the stencil buffer is the other important half of this trick. The first pass (the lighting and texturing pass) writes to the stencil buffer. (4 is an arbitrary reference number.) The second pass (the outline pass) reads the stencil buffer, and, where it sees the same reference already written, keeps the original pixel in place. This achieves the effect of making the scaled mesh look like it’s behind- or only contouring- the original mesh. I also make sure to turn culling off, so that we don’t discard any vertexes of the outline mesh.

Finally, each pixel of the outline mesh is filled in with the same solid color. We don’t really need lighting for a cartoonish outline. Ta-daa!!! A beautiful outline.

Mistakes LEARNED from

Mistake #1: Blended cel shading. This is clearly a rookie mistake from a first-time shader writer. My first try at the cel shader had some blended edges on the shadows, which is not what we want for a hard cel look. The mistake I made was doing the lighting calculation in the vertex shader step. Because the vertex shader is per-vertex, every pixel in between is linearly interpolated, which means you end up with a blended effect for some pixels if you do lighting in the vertex shader. Moving the lighting calculation to the fragment shader, which is calculated per-pixel, fixed this immediately. Funnily enough, not ALL of the faces on the model were blended, which turned out to be because the model was non-manifold, which has to do with mistake #3.

first pass at dog shader

Mitake #2: Uneven outline. You can see in the image above that the outline isn’t smooth and evenly contouring the mesh. The original algorithm I had for scaling the mesh didn’t include the normals- it was just multiplying the vertex by outlineSize. This doesn’t work for convex models, which really any complicated model that isn’t a pure cube or sphere is probably going to be. Scaling along the normals made a nice, even outline.

Mistake #3: Broken pieces of the extruded mesh. When using the low-poly shibe above, and applying the extrusion technique for scaling, there were gaps between each face. This is because the low-poly shibe has some faces of the mesh that aren’t really connected on the edges that they meet at, making it a non-manifold mesh. Then, when the vertexes are extruded, the faces don’t remain visually connected. I had to use a manifold mesh for anything using the outline shader, which is really almost any mesh that isn’t low-poly.

Mistake #4: Missing pieces of the outline. This was because I originally didn’t have the cull:off tag on the outline pass. We don’t want to discard any parts of the outline pass, even if they’re technically “inside” the mesh.

Code

Shader "Custom/CelEffects"
  { 

  Properties
  { 
    _MainTex("Texture", 2D) = "white" {} 
    _RampTex("Ramp", 2D) = "white" {} 
    _Color("Color", Color) = (1, 1, 1, 1) 
    _OutlineExtrusion("Outline Extrusion", float) = 0
    _OutlineColor("Outline Color", Color) = (0, 0, 0, 1)
  }

  SubShader
  {

    // Regular color & lighting pass
    Pass 
    {

      // Write to Stencil buffer (so that outline pass can read) 
      Stencil 
      { 
        Ref 4 
        Comp always 
        Pass replace 
        ZFail keep 
      }

      CGPROGRAM
      #pragma vertex vert
      #pragma fragment frag

      // Properties
      sampler2D _MainTex;
      sampler2D _RampTex;
      float4 _Color;
      float4 _LightColor0; // provided by Unity

      struct vertexInput
      {
        float4 vertex : POSITION;
        float3 normal : NORMAL;
        float3 texCoord : TEXCOORD0;
      };

      struct vertexOutput
      { 
        float4 pos : SV_POSITION;
        float3 normal : NORMAL;
        float3 texCoord : TEXCOORD0;
      };

      vertexOutput vert(vertexInput input)
      {
        vertexOutput output;

        // convert input to world space
        output.pos = UnityObjectToClipPos(input.vertex);
        // need float4 to mult with 4x4 matrix
        float4 normal4 = float4(input.normal, 0.0);
        output.normal = normalize(mul(normal4, unity_WorldToObject).xyz);
        output.texCoord = input.texCoord;
        return output;
      }

      float4 frag(vertexOutput input) : COLOR
      {
        // convert light direction to world space & normalize
        // _WorldSpaceLightPos0 provided by Unity
        float3 lightDir = normalize(_WorldSpaceLightPos0.xyz);

        // finds location on ramp texture that we should sample
        // based on angle between surface normal and light direction
        // sample texture for color
        float ramp = clamp(dot(input.normal, lightDir), 0, 1.0);
        float3 lighting = tex2D(_RampTex, float2(ramp, 0.5)).rgb;
        float4 albedo = tex2D(_MainTex, input.texCoord.xy);
        // _LightColor0 provided by Unity
        float3 rgb = albedo.rgb * _LightColor0.rgb * lighting * _Color.rgb; 

        return float4(rgb, 1.0);
      }

      ENDCG
    }

    // Outline pass
    Pass
    {

      Cull OFF
      ZWrite OFF
      ZTest ON

      // Won't draw where it sees ref value 4
      Stencil
      {
        Ref 4
        Comp notequal
        Fail keep
        Pass replace
      }

      CGPROGRAM
      #pragma vertex vert
      #pragma fragment frag

      // Properties
      uniform float4 _OutlineColor;
      uniform float _OutlineSize;
      uniform float _OutlineExtrusion;

      struct vertexInput
      {
        float4 vertex : POSITION;
        float3 normal : NORMAL;
      };

      struct vertexOutput
      {
        float4 pos : SV_POSITION;
        float4 color : COLOR;
      };

      vertexOutput vert(vertexInput input)
      {
        vertexOutput output;
        float4 newPos = input.vertex;
      
        // normal extrusion technique
        float3 normal = normalize(input.normal); newPos += float4(normal, 0.0) *                                 
            _OutlineExtrusion;

        // convert to world space
        output.pos = UnityObjectToClipPos(newPos);
        output.color = _OutlineColor;

        return output;
      }

      float4 frag(vertexOutput input) : COLOR
      {
        return input.color;
      }

      ENDCG
    }
  }
}

If y’all have any questions about writing shaders in Unity, I’m happy to share as much as I know. I’m not an expert, but I’m always willing to help other indie devs 🙂

Good luck,

Lindsey Reid

@lindseywaslike

@sogoodgames