Skip to content

Stencil and Render Order Guide

This section covers NiloToon's stencil features, eyebrow/eyelash render order, and 2D shadow visibility settings.

Using Stencil Shader on Characters

You can use GenericStencilUnlit.shader as an example shader. This is useful when you want to apply stencil effects that use drawn character pixels as a stencil mask, or when you need a reference shader.

Extra Thick Outline and Stencil

When "Extra Thick Outline" is enabled, the leftmost bit of the stencil value will be set to 1 by NiloToon's pass. For example:

  • Material's Stencil Ref = 3 (00000011)
  • Actual Stencil Ref after the special stencil pass = 10000011

Stencil Shader example

Eyebrow on Top of Hair (Opaque)

If you only need to simply draw eyebrows/eyelashes on top of hair (without semi-transparent blending), the simplest and quickest method is to use the Z Offset (eyebrow) group in the eyebrow/eyelash material. No extra material, stencil, or NiloToonRendererRedrawer is needed -- this step alone will work.

Z Offset Method

Before editing the eyebrow material:

Before Z Offset 1 Before Z Offset 2

After editing the eyebrow material (Z Offset enabled):

After Z Offset 1 After Z Offset 2

Recommended Values

-0.02 / -0.03 is usually a good starting point. This means pushing the eyebrow 2cm / 3cm towards the camera.

Caution

Do not set values too large! (e.g., -0.1 means pushing 10cm toward the camera, which is typically too much.) Setting it too large may cause eyebrows to appear in front of unwanted objects like hands or arms.

Z Offset Mask

If the eyebrow/eyelash is not an isolated material (e.g., there is only a single face material that contains the entire face including eyebrows), use the Z Offset (eyebrow) group's Mask Map to ensure the ZOffset only applies to the eyebrow/eyelash area.

Z Offset Mask setting

Stencil Method

If you prefer not to use ZOffset and the eyebrow material is an isolated material, you can use the traditional stencil method.

  1. The eyebrow material writes to stencil first, using the Stencil Buffer group in the material
  2. The hair uses the stencil to reject rendering for eyebrow pixels - the hair draws as usual but leaves a hole for the eyebrow pixels
  3. You get a result similar to the ZOffset method, but you may see wrong occlusion issues (e.g., the eyebrow may be visible from the back side of the hair if the hair material does not split front hair into an isolated material)

Eyebrow on Top of Hair (Semi-Transparent)

Before:

Before semi-transparent eyebrow

After (note the difference in the eyebrow area):

After semi-transparent eyebrow

You can choose from three methods (Method A / B / C). If the eyebrow is an isolated material, Method C is recommended.

Method A (Semi-Transparent Hair)

The easiest method: make the front hair semi-transparent so the color underneath is visible. No Stencil / ZOffset / NiloToonRendererRedrawer is needed.

If the front hair material's Base Map already has alpha for front hair:

  • Surface Type = Opaque(Alpha)/Outline

If the front hair material's Base Map does not have alpha for front hair:

  • Surface Type = Opaque(Alpha)/Outline
  • Enable BaseMap Alpha Modify group, assign a texture to Alpha Modify Tex, set Mode = Multiply, and select the RGBA channel that contains the alpha
  • (Optional) Set Is front hair? = 1 - the BaseMap Alpha Modify effect will fade out when the character is not facing the camera
  • Render Queue: Ensure the hair is drawn after the face/eyebrow

Method B (ZOffset Eyebrow Redraw)

Uses NiloToonRendererRedrawer and a Material Variant to redraw a semi-transparent eyebrow on top of hair.

  1. Create a Material Variant of the original eyebrow material and set Surface Type = Transparent/No Outline
  2. Add the NiloToonRendererRedrawer script to the eyebrow's renderer
  3. Click the + button on NiloToonRendererRedrawer to add a redraw request, then assign the Material Variant
  4. Apply the following settings to the Material Variant:
    • Surface Type = Transparent/No Outline
    • BaseColor Alpha < 1 (e.g., 0.5)
    • ZOffset group = enabled, preset set to 3cm or larger

Method C (Stencil + Render Queue Hair Redraw)

This method produces a visually perfect result but is much more complex to set up.

Requirements

This method only works with Forward / Forward+ (because Stencil is required).

Steps Overview:

  1. Render the face (excluding eyebrow) material normally without stencil settings (Render Queue 2000-2450)
  2. Render the eye/eyebrow material with Stencil Writer settings - Preset 1st>Writer (eyebrow), Comp = Always, Pass = Replace, with a special Ref (e.g., 77) (Render Queue 2451)
  3. Render the front hair material with Stencil Reader settings - Preset 2nd>Reader (Original Hair), Comp = NotEqual, Pass = Keep, with the same Ref (e.g., 77) (Render Queue 2452)
  4. Create a Material Variant of step 3's hair material and redraw it using NiloToonRendererRedrawer - Preset 3rd>Reader-Invert (Redraw Hair), Comp = Equal, Pass = Keep, with the same Ref (Render Queue 2452)
  5. Apply semi-transparent settings to the Material Variant:
    • Surface Type = Opaque(Alpha)/Outline or Opaque(Alpha)/No Outline
    • Enable BaseMap Alpha Modify, assign the PureBlack4x4 texture from the NiloToonURP folder
    • Strength = desired value (e.g., 0.25 to 0.5)
    • Is front hair? = 1

Ref ID

Ref ID 77 is an arbitrary number. It is recommended to be unique between characters.

Render Order

You must control the correct draw order of 1 > 2 > (3+4) using each material's Render Queue.

2D Shadow Visibility Threshold

2D Shadow Threshold setting

If the 2D shadow disappears when the shadow caster is very close to the shadow-receiving surface, adjust the Threshold Offset and Fadeout Range in DepthTex RimLight&Shadow > Shadow Area.