Monthly Archives: December 2016

Using a Pixel Shader for Facial Expressions

I’ve been playing around with shaders and I thought about making a quick work-in-progress video to show the visual effects that I have been working with. In a nutshell, I have enhanced the human expression engine that I have implemented for my 3D models. The assumption here is that all characters will always be at a certain distance, and there will never be any kind of close-up. Granted, the video is a close-up itself, but that is only for the purpose of this article.

Here is the video:

The effects shown were implemented within the Pixel Shader. That’s a rather big gamble because a complex Pixel Shader may cause a catastrophic drop in performance. The advantage that I have is that my choice of design is towards a cartoon style, which makes pretty much everything much easier and cheaper than what would have been if I had chosen a realistic approach. That allows me to get away with a simplistic algorithm: The simpler the better.

In Layman’s terms, a pixel shader is a program that “paints” a tridimensional model according to a given image called “texture”. Models are made of triangles. Each triangle has three corners or “vertices”, each corner has a coordinate that points to a pixel in our texture, and all three pixels delimit a triangle in this texture. A pixel shader draws this very same triangle on the screen by extrapolating these coordinates, pixel by pixel, sampling the color to display from our texture.

Now, the features shown in the video work by pretty much messing with these sampling coordinates. Not all the texture is always shown and, instead, the pixel shader input specifies which section of the texture should be drawn.

I. Eye movement

When the shader draws the upper half of the face, if the color of the texture sample is white then it’s drawing the eyes. Then, sample again, but this time add an offset that points to an area on our texture where the irises are located. Messing up with this offset gives the effect of eye movement. This variation is a parameter, member of the pixel shader input. Granted, the irises are not a perfect circle, but then again, they are round enough at a distance.

II. Blinking

The texture has three pairs of eye lids in it: open, half closed, and closed. A shader input field specifies which of these pairs is drawn at a given time. My animation engine computes which one should be shown (the less operations at a shader level the better), following the pattern “open – half closed – closed – half closed – open”.

A special condition applies: the animation engine half closes the eyelids when the character is looking down. That makes the overall effect a little bit more natural.

III. Smile and Frown

The tips of the mouth can be crooked up or down. Originally, on the texture, the mouth is flat. Crooking the tips of the mouth can create a smile or a frown. Only the tip of the mouth is crooked in order to give the effect of full body lips.

This one is a little bit of a gamble. DirectX experts may argue here that this effect could have a better performance if done within the vertex shader. They would be right if it were not for the fact that I’m running very close to the 39 bone limit imposed by the size of the memory buffer in the average video adapter for PC.

VI. Eyelashes

Two thirds of the eyelashes can be tilted up and down, expressing the state of emotion of the character: sad, angry, evil mischief. The secret is to know at which point the tilt starts. The fact that this is a cartoon character makes this quite easy.