Anti-Aliasing (AA) and DLSS Guide¶
This section provides an AA selection guide for NiloToon and explains how to set up DLSS.
Summary - Choosing the Best AA¶
| Condition | Recommended AA |
|---|---|
| RenderScale <= 1 + NVIDIA RTX GPU | DLSS |
| RenderScale <= 1 + Non-NVIDIA GPU (AMD / Console) | FSR3 |
| RenderScale > 1 + TAA Ghosting/Blur acceptable | TAA |
| RenderScale > 1 + TAA Ghosting/Blur not acceptable | MSAA* |
Note
Unity6 STP and Intel XeSS2 are excluded from the list above. This is not because they are bad (they may actually be similar to or better than FSR3), but because they have not been tested enough to draw a conclusion. MSAA* means MSAA + FXAA/SMAA/Off.
Overall Graphics Quality AA Ranking¶
| RenderScale | Ranking |
|---|---|
| <= 1 | [Best] DLSS >> FSR3 > TAA >>> MSAA* [Worst] |
| > 1 | [Best] TAA >>> MSAA* [Worst] |
Classic Outline AA Ranking¶
| RenderScale | Ranking |
|---|---|
| <= 1 | [Sharpest] MSAA >> DLSS > FSR3 >>>>> TAA* [Blurriest] |
| > 1 | [Sharpest] MSAA >>>>>>>> TAA* [Blurriest] |
Limitations
DLSS and FSR3 cannot be used when RenderScale > 1.
Recommended AA Selection Process¶
In real projects (primarily VTuber 3D Live/concert PC projects) focused on visual quality and targeting 60fps on high-end PCs, AA is selected in the following order:
Step 1: MSAA (Default Choice)¶
MSAA = 4x, RenderScale = 2, Camera AA = Off
MSAA never produces ghosting or blurriness, and does not require all opaque shaders to have correct depth and motion vector passes. Optionally enabling FXAA/SMAA can improve 2D Rim Light AA.
Step 2: TAA (When Shading Aliasing Occurs)¶
If character Rim Light, Screen Space Outline, PBR Specular, or small objects have shading aliasing, or if VRAM is insufficient due to MSAA, switch to TAA.
MSAA = Off, RenderScale = 2, Camera AA = TAA
This removes almost all kinds of aliasing while improving performance, but introduces ghosting and blur due to the inherent nature of TAA.
Step 3: DLSS (When Performance Is Insufficient)¶
If the above methods do not provide sufficient performance (fps) at 4K/8K with RenderScale = 2, switch to DLSS.
DLSS NativeAA (MSAA = Off, RenderScale = 1, Camera AA = Off)
This provides much better performance since RenderScale is <= 1, but prevents using RenderScale = 2 for improved quality/sharpness.
Settings That Affect AA¶
To improve the quality and AA of NiloToon shader features, you can increase the following settings, though doing so will slow GPU rendering performance:
| Setting | Affects |
|---|---|
| Game resolution | Outline and Rim Light quality |
| Render Scale | Outline and Rim Light quality |
| Camera AA (FXAA/SMAA/TAA) | Outline and Rim Light quality |
| DLSS / FSR3 quality | Outline and Rim Light quality |
| MSAA | Classic Outline quality only (does not improve Screen Space Outline / Rim Light / Specular aliasing) |
| NiloToon Self Shadow Map resolution and Soft Shadow quality | NiloToon Self Shadow Map quality only |
MSAA + FXAA/SMAA Combinations¶
In Unity URP, FXAA/SMAA can be used regardless of whether MSAA is on or off, so combinations are possible.
| AA Type | Characteristics |
|---|---|
| MSAA | Best for geometry edges (Classic Outline AA). No overall image blur. However, extremely GPU-intensive at high resolution. Does not improve Rim Light, Screen Space Outline, or Specular Highlight |
| FXAA | Fastest AA. Overall blurrier than SMAA/MSAA |
| SMAA | Slower than FXAA but sharper. Not very helpful for Outline AA |
Classic Outline AA
To primarily improve Classic Outline AA, among MSAA/FXAA/SMAA, only MSAA is useful.
TAA vs MSAA¶
In Unity URP, TAA can only be used when MSAA is off.
| Comparison | TAA | MSAA |
|---|---|---|
| Overall image AA | Better (handles all aliasing) | Geometry edges only |
| Classic Outline AA | Fails in fast motion | Best |
| Screen Space Outline AA | Better | Cannot improve |
| Rim Light AA | Better | Cannot improve |
| Specular AA | Better | Cannot improve |
| Performance | Better | - |
| Sharpness | Causes blur | No blur |
| Ghosting | May occur in fast motion | None |
| Shadow AA | Slightly improved | Cannot improve |
| Shader requirements | All opaque materials need Depth/Motion Vector Pass | Not required |
| Editor version | Unity 2022+ | All URP versions |
How to Use TAA¶
- Select the Camera in the Scene view or Hierarchy and view it in the Inspector
- Navigate to Rendering > Anti-aliasing and select Temporal Anti-aliasing (TAA)
- Navigate to Output > MSAA and select Off
Features Incompatible with TAA
- Multisample Anti-aliasing (MSAA)
- Camera Stacking
- Dynamic Resolution
TAA Ghosting Issues¶
Even when all opaque shaders have correct depth and motion vector passes, TAA ghosting can still exist.
Example: When the background displays high-frequency detail with dense brightness changes (e.g., a big TV screen in a 3D Live concert with many black circle mask patterns on a bright screen), and the front character is moving fast, TAA ghosting will appear.
AA/Quality Presets¶
Editor Recorder (Maximum Quality)¶
When using Recorder to produce video (offline recording MV, trailer, etc.):
| Setting | (1) 4K (TAA) | (2) 8K (TAA) | (3) 4K (DLSS) | (4) 8K (DLSS) |
|---|---|---|---|---|
| Game Window resolution | 2160p(4K) | 4320p(8K) | 4320p(8K) | 4320p(8K) |
| RenderScale | 2 | 1 | 1(DLSS locked) | 1(DLSS locked) |
| Camera AA | TAA(Very High) | TAA(Very High) | None | None |
| MSAA | None | None | None | None |
| Self Shadow resolution | >= 8192 | >= 8192 | >= 8192 | >= 8192 |
| Soft Shadow quality | High | High | High | High |
| DLSS quality | N/A | N/A | NativeAA | NativeAA |
Note
Setting (1) is used most frequently. It requires the least SSD space and is very simple to set up. All settings assume a GPU with at least 16GB VRAM (e.g., RTX 4070 Ti or better).
PC/Console (Very High)¶
- Game Window resolution = 1440p-2160p(4K)
- RenderScale = 1-2
- Camera AA = TAA(Very High)
- MSAA = Off
- Self Shadow resolution = 4096-8192
- Soft Shadow quality = Medium / High
PC/Console (High)¶
- Game Window = 1080p-1440p
- RenderScale = 1
- Camera AA = SMAA
- MSAA = 4x
- Self Shadow resolution = 4096
- Soft Shadow quality = Medium
Mobile (Very High)¶
- Game Window = 1080p (using Screen.SetResolution)
- RenderScale = 0.8-0.9 (actual 3D rendering 864p-972p)
- Camera AA = None
- MSAA = 2x-4x
Mobile (High)¶
- Game Window = 1080p
- RenderScale = 0.7-0.8 (actual 3D rendering 756p-864p)
- Camera AA = None
- MSAA = 2x-4x
Mobile (Medium)¶
- Game Window = 960p
- RenderScale = 0.68-0.78 (actual 3D rendering 648p-756p)
- Camera AA = None
- MSAA = 2x-4x
Mobile (Low)¶
- Game Window = 840p
- RenderScale = 0.64-0.77 (actual 3D rendering 540p-648p)
- Camera AA = None
- MSAA = 2x-4x
Recommended Maximum Mobile Resolution
Most mobile games use 720p-864p as the maximum actual 3D rendering resolution even at the highest graphics setting (e.g., Genshin Impact / Honkai: Star Rail / Wuthering Waves / Infinity Nikki). We strongly recommend using 720p-864p as the maximum 3D rendering resolution for mobile.
What to Keep Even at Minimum Settings¶
In a game's graphics settings UI, the following two features have a major impact on character rendering if disabled:
- AA (MSAA / TAA / DLSS / FSR3 / XeSS2 / SGSR2)
- Bloom (URP Bloom / NiloToon Bloom)
Important
We strongly recommend keeping at least AA and Bloom enabled even at the lowest graphics quality preset. Keeping AA and Bloom preserves character rendering quality (Classic Outline and overall color).
Color Banding¶
If you see Color Banding in areas with color gradients (e.g., the color fadeout of volumetric light beams):
- Enable the camera's Dither
- Do not use FXAA
DLSS (Best AA and FPS)¶
Why Use DLSS?¶
DLSS 4.5 is the best Anti-Aliasing (AA) and Upscaling tool, and can replace URP's TAA/STP when RenderScale <= 1 on NVIDIA RTX GPUs.
DLSS is useful in the following situations:
- GPU Bound due to high resolution or high RenderScale
- When you want to replace TAA's ghosting and blurriness with the best AA tool when RenderScale <= 1
DLSS 4.5 can be purchased and used from the Unity Asset Store.
Version Warning
Do not use the old DLSS3 asset. Only use DLSS 4.5. Only DLSS 4.5 supports the Transformer Model (M). For RTX 40xx or higher, Preset M is recommended; otherwise use Preset K.
In the image above, you need to zoom in to see the AA difference. Focus on the Outline and 2D Rim Light aliasing to observe the AA quality difference.
DLSS vs TAA/STP¶
DLAA (DLSS's NativeAA) is "DLSS without upscaling" -- it is the best AA method at RenderScale = 1, far superior to URP's TAA/STP.
At 4K with RenderScale of 1:
| Aspect | DLAA vs TAA/STP |
|---|---|
| Sharpness | DLAA is much sharper |
| Ghosting | DLAA has far less |
| Pixel Jitter | DLAA is almost invisible; TAA/STP is easily visible |
| Specular Aliasing handling | DLAA is more stable |
| Overall AA result | DLAA is the best |
| GPU speed | DLAA is slower than TAA/STP |
| RenderScale limitation | DLAA is forced to <= 1; TAA can use > 1 (max 2) for Super Sampling |
When to Use DLSS¶
Generally used when targeting a 4K Game Window or above. For 1080p/1440p, try RenderScale = 2 + TAA first, then switch to DLSS NativeAA if the GPU load is heavy.
Minimum Quality - Resolution¶
For good DLSS results, the internal resolution should be at least >= 1440p.
| Game Window Resolution | Minimum Quality Preset |
|---|---|
| 8K (4320p) | Ultra Performance (RenderScale = 0.333) (= Internal 1440p) |
| 4K (2160p) | Quality (RenderScale = 0.666) (= Internal 1440p) |
| 1440p | Native AA (RenderScale = 1) (= Internal 1440p) |
| 1080p | Native AA (RenderScale = 1) (= Internal 1080p)* |
1080p Limitation
1080p may be too low for DLSS to produce good results even with Native AA. Classic Outline aliasing may be visible.
DLSS Setup and Usage¶
- Purchase DLSS4 from the Asset Store
- Import the package from Package Manager > My Assets
- Attach the TND Upscaler script to your 3D Main Camera:
- Upscalers = DLSS4.5
- Quality Mode = Native AA (lower it if GPU bound)
- Set all Render Presets to Preset M (for RTX 20xx/30xx, use Preset K if GPU performance is important)
- Turn off the Main Camera's Camera AA (FXAA/SMAA/TAA)
- Fix all errors and warnings displayed inside the TND Upscaler script
- Enter Play Mode
Required
When using DLSS, you must turn off all Camera AA (FXAA/SMAA/TAA).
When Not to Use DLSS¶
At 1080p/1440p, your GPU can often handle RenderScale = 2 + TAA. This setting generally produces better image quality than RenderScale = 1 + DLAA, since it is brute-force Super Sampling AA (closer to ground truth quality).
For offline video production using Recorder, RenderScale = 2 + TAA is generally the best option. FPS does not matter, only visual quality matters.
DLSS Settings for Games¶
When using DLSS (and other upscalers like FSR4/FSR3/XeSS2) for a PC game, since players have different GPUs, it is best to use all upscalers with the following fallback order:
- DLSS 4.5
- FSR 4
- XeSS 2
- FSR 3
- SGSR 2 (optional: safe fallback for non-PC platforms)
Expose the upscaler quality settings in your game's graphics settings page.
| Quality Mode | Scaling | RenderScale |
|---|---|---|
| Native AA | 1.0x | 1 |
| Ultra Quality | 1.2x | 0.833 |
| Quality | 1.5x | 0.666 |
| Balanced | 1.7x | 0.588 |
| Performance | 2.0x | 0.5 |
| Ultra Performance | 3.0x | 0.333 |
VRAM Cost¶
RTX 30/20 Series
Preset L/M incur a much higher VRAM cost on RTX 30/20 series. If VRAM usage is important, always use Preset K for RTX 30/20 series.
Performance Cost¶
Preset Selection Guide
- RTX 30/20 series: If GPU performance is important, always use Preset K
- RTX 40/50 series: Preset M is recommended for better visual quality with a relatively low additional performance cost