Skip to content

Video Compression Guide

Compressing to a Smaller Video

If a ProRes mov file is too large for uploading, sharing, storage, or video editing (e.g., a 4K60fps 7-minute ProRes 422 HQ video is ~100GB), you can download HandBrake to reduce file size without visible quality loss.

TL;DR - General Solution

  1. Open HandBrake
  2. Import the large video
  3. Select the Production Standard preset Production Standard Preset
  4. Change Video Encoder from H.264 (x264) to H.265 10-bit (x265) (if H.265 is available) Select H.265 10-bit
  5. Start Encode. The resulting video will be significantly smaller (e.g., 100GB -> 7GB), and you will not notice any quality difference unless you pause and compare frame by frame

Background Knowledge

Encoder Differences

Encoder Quality Size Compatibility
H.264 Average Large Universally supported
H.265 Good Medium Good
AV1 Very Good Smallest Limited
ProRes Visually Lossless Extremely Large Good
  • H.264: Universally supported. Quality is average and file size is large
  • H.265: Has good compatibility. Quality is good and file size is medium
  • AV1: Compatibility is limited, but quality is very good and file size is the smallest. Not suitable for video editing software. Due to AV1's open-source, royalty-free nature and backing by major tech companies, AV1 is expected to gradually replace H.264/H.265/H.266
  • ProRes: The result of Unity Recorder's ProRes recording. Visually lossless quality with fast decoding, making it great for video editing, but file size is extremely large

VMAF Score

VMAF is a 0~100 score developed by Netflix, where 100 means visually lossless when comparing the compression result to the original source video. If you want near visually lossless compression, aim for VMAF = 99.9~100.

RF/CRF (Rate Factor)

Similar to a quality level in reverse. The lower the RF, the better the quality but the larger the file size. Recommended starting RF values for achieving VMAF 99.9~100 (visually lossless):

Encoder Recommended RF
H.264 (x264) RF = 10
H.265 (x265) RF = 8
H.265 10-bit (x265) RF = 10
AV1 10-bit (SVT) RF = 14

HandBrake Guide

Prerequisites

  1. Do not use GPU encoders (e.g., NVEnc). GPU encoders are designed to sacrifice quality for encoding speed, primarily for real-time streaming. This guide focuses on CPU encoding only, where we care about the final quality and file size, not encoding speed. Do Not Use GPU Encoders

  2. In HandBrake's Filter tab, always turn off all filters. Since the input source is ProRes recording from Unity, no filters are needed. Enabling filters like Interlace Detection or Deinterlace will produce weird high-contrast dots/aliasing/pixel displacement around eyes, mouth, and outlines. All Filters Off

Important

Always set all filters = Off in the Filters tab!

Choosing the Best Encoder

Criteria Priority
Maximum Quality AV1 > H.265 > H.264
Smallest File Size AV1 > H.265 > H.264
Maximum Compatibility H.264 > H.265 > AV1
Video Editing Software Decode Performance H.264 > H.265 > AV1

AV1 Encoding

To produce the highest quality VMAF ~= 100 AV1 video:

  1. Select the Super HQ 2160p60 4K AV1 preset AV1 Preset Selection
  2. Disable all filters in the Filter tab Disable Filters
  3. Set RF to 14 or lower to maintain VMAF ~= 100 RF 14 Setting
  4. (Optional) If you don't need visually lossless results and prefer a smaller file (VMAF = 99.9~100), increase RF to 15~22
  5. (Optional) For a much smaller file (VMAF < 99.9), increase RF to 23~30

AV1 RF Reference Table

Input video: 3384x1440, H.264, SDR, ~900Mbps bitrate, CRF0, 60fps, 4 minutes 9 seconds, NiloToon concert MV.

AV1 RF Chart

Compression Setting VMAF (closer to 100 is better) File Size (smaller is better)
Uncompressed input N/A 25.42GB
RF=10 100 2.46GB
RF=12 100 1.96GB
RF=14 100 1.65GB
RF=15 99.99 1.54GB
RF=16 99.99 1.43GB
RF=18 99.97 1.25GB
RF=20 99.95 1.12GB
RF=22 99.91 1.01GB
RF=25 99.82 0.86GB (860MB)
RF=30 99.39 0.63GB (630MB)
RF=63 71.07 0.07GB (70MB)

AV1 Summary

If you don't care about compatibility and video editing software use, and just want the smallest file with very good quality for playback/archive, AV1 + no filters + RF=14 is a great starting point.

AV1 10-bit (NVEnc)

This is the GPU AV1 encoder for NVIDIA RTX 40xx cards. Encoding speed is very fast, but it may produce incorrect results on some frames, making it unreliable. Not recommended.

H.265 Encoding

If AV1 is not an option due to compatibility issues, the next best choice is H.265 10-bit.

To produce a VMAF ~= 100 H.265 video:

  1. Select the Production Standard preset Production Standard
  2. Click the Video tab
  3. Change encoder from H.264 (x264) -> H.265 10-bit (x265)
  4. Keep RF = 10 H.265 Settings
  5. (Optional) Change RF from 10 to 11~18 according to your acceptable quality loss

H.265 RF Reference Table

Production Standard, H.265 10-bit (x265), No Filters:

Compression Setting VMAF File Size
Uncompressed input N/A 25.42GB
RF=10 100 2.64GB
RF=11 99.99 2.34GB
RF=14 99.92 1.92GB
RF=18 99.33 1.15GB
RF=20 98.56 0.89GB
RF=22 97.33 0.69GB

H.264 Encoding

Typically, H.264 is only used when you need to improve decoding speed in video editing software.

For a VMAF ~= 100 H.264 video:

  1. Select the Production Standard preset and do not change any settings
  2. Keep RF = 10 H.264 Settings

Production Standard Setup

The Production Standard preset is not 100% truly lossless, but it is visually lossless. In practice, the result is always sufficient for any video editing (default CRF=10).

For a 7-minute 3840x2160 60fps ProRes 422 HQ Recorder recording (~100GB), Production Standard preset converts to:

  • ~15GB (H.264), VMAF ~= 100
  • ~7GB (H.265), VMAF ~= 99.98

RF/CRF (Rate Factor) Guide

CRF Value Preset Description
2 Production Max Maximum quality, nearly true lossless, but often overkill
10 Production Standard Best quality, visually lossless. Recommended for production
15 - High quality, compression artifacts start to appear
20 Super HQ 2160p60 4K HEVC Average quality, visible compression artifacts
22 Creator 2160p60 4K Low quality, easily visible compression artifacts
24 Fast 2160p60 4K HEVC Poor quality, aggressive compression
26+ - Too aggressive compression, do not use!

RF Adjustment Guide

  • To improve quality, use lower RF (10~15)
  • To reduce file size, use higher RF (16~22)

Checking VMAF Score

To check VMAF scores locally after compressing a video, use VMAF-GUI. Extract it to your fastest SSD to improve score calculation speed.

VMAF GUI Tool

Score Meaning
100 Visually lossless, suitable for video editing software
99.9~100 Sufficient for video editing software
99.0~99.9 Sufficient as a final result video for playback
<99.0 Compression artifacts visible

Video Compare Tool

Tools for checking compression differences between videos:

8K AV1 Upload

TL;DR

To achieve the highest encoding quality when uploading, ensure the uploaded video meets these conditions:

  • 8K (width >= 7680 and height >= 4320)
  • 16:9 aspect ratio
  • ProRes422HQ .mov format

Other resolutions/aspects/formats may prevent 8K. The above settings (8K, 16:9, ProRes422HQ) generally trigger the use of the AV1 encoder (AV1 is the best encoder).

When successful, typically within 1-2 days after upload, the AV1 (av01) Codec will be used and the 8K option will be available.

AV1 8K Success

If 8K upload is not possible, at minimum upload as 4K 16:9:

  • width >= 3840 and height >= 2160
  • 16:9 aspect ratio

1080p/1440p Uploads Not Recommended

To ensure encoding result quality, uploading at 1080p/1440p is no longer recommended.

Introduction

When uploading a video, no matter how good the encoding quality of your local file is, uploading at less than 8K (7680x4320) (e.g., 1080p/1440p) will result in lower displayed quality than your local file. This is because VP9 1080p/1440p encoding intentionally uses lower quality.

Solution

  • Upscale to 8K (7680x4320) and 16:9 before uploading to trigger the highest quality AV1 (av01) encoder instead of VP9 (vp09).

Step-by-Step Guide

  1. Prepare the video file for upload
  2. Open HandBrake and select the video as input
  3. Select the Production Max preset
  4. Click the Dimensions tab
  5. Disable Optimal Size
  6. Enable Allow Upscaling
  7. If the input video is 16:9 aspect ratio, set Resolution and Scale > Scaled Size = 7680x4320
  8. If the input video is wider than 16:9, set width to 7680 and use Border (Top & Bottom) to fill empty space until Final Dimension is 7680x4320
  9. If the input video is narrower than 16:9, set height to 4320 and use Border (Left & Right) to fill empty space until Final Dimension is 7680x4320
  10. If all steps are correct, Final Dimensions = 7680 x 4320 (8K 16:9) will be displayed 8K Dimensions Settings
  11. Click Start Encode. The resulting video may be very large (~150GB for 4 minutes at 60fps)
  12. Upload the result directly

Verifying Success

  • Success: After hours to days following upload, the Codec will be re-encoded to AV1 (av01) and the 8K option will be available to viewers AV1 Success Verification
  • Failure: After upload, the VP9 (vp09) Codec remains permanently, and the 8K option will not be available unless the video becomes popular VP9 Failure

Upload Resolution Reference

To trigger 8K AV1, 8K 16:9 is generally required:

Upload Resolution Result
3384x1440 VP9, 4K (no 8K option)
5076x2160 VP9, 4K (no 8K option)
6768x2880 VP9, 4K (no 8K option)
10152x4320 VP9, 4K (not 16:9, so no 8K option)
7680x4320 AV1, 8K (16:9 aspect ratio, success)

8K AI Upscale Tools

The HandBrake upscale method above only increases resolution without adding new details. If you need sharp 8K results, use an AI upscaler:

However, for upload purposes, since videos are heavily compressed, the sharpness/detail improvements from AI upscalers will likely be visually removed.