Image for post
Image for post
80x80 sample cutouts taken from the “Dinner Table” video — grabbed at timecode 00:00:12, pixel position x=1374, y=756 of the original HD video vs every contestant’s 2x reconstruction — the downsampled 40x40 source area is overlaid on each reconstruction

Comparing Video Upscaling Methods on Stock Footage

Upsampling Algorithm Shootout

The Challenge

Image for post
Image for post
Photo credit: Zain Saleem

Meet the Contestants

PIXOP — PIXOP Platform (SaaS)

Image for post
Image for post
Pixop Super Resolution “Clarity boost” setting (i.e. the default “Low”) used for all videos
Image for post
Image for post
Example: Single frame of PIXOP Super Resolution x3 upscaling extracted from the “Dinner Table” video at 00:00:03

FFmpeg — libswscale (5.3.100)

Image for post
Image for post
Example: Single frame of Lanczos interpolation x3 upscaling extracted from the “Dinner Table” video at 00:00:03

Grass Valley — Alchemist XF (free trial, 4.0.1.2)

Image for post
Image for post
Alchemist profile for 29.97 fps videos — for 30 fps video the “[Default] 1080 30p ProRes” profile was used instead
Image for post
Image for post
Example: Single frame of Alchemist Ph.C. x3 upscaling extracted from the “Dinner Table” video at 00:00:03

Blackmagic Design — DaVinci Resolve 16 (free trial, 16.0.0.060)

Image for post
Image for post
2x upscaling settings used for DaVinci Resolve — for 3x upscaling Super Scale was set to “3x”, the other two parameters were the same
Image for post
Image for post
Example: Single frame of DaVinci Resolve Super Scale x3 upscaling extracted from the “Dinner Table” video at 00:00:03

Adobe — After Effects (16.1.13)

Image for post
Image for post
3x upscaling settings used in Adobe After Effects — for 2x upscaling Scale was set to “200%”, all other parameters were the same
Image for post
Image for post
Example: Single frame of Adobe After Effect’s Detail-preserving Upscale x3 upscaling extracted from the “Dinner Table” video at 00:00:03

Boris FX — Continuum (free trial, 12.5.2)

Image for post
Image for post
2x upscaling settings used for BCC UpRez in Final Cut Pro X 10.4.6 — for 3x upscaling Scale X and Scale Y was set to “300.0”, all other parameters were the same
Image for post
Image for post
Example: Single frame of BCC UpRez x3 upscaling extracted from the “Dinner Table” video at 00:00:03

Test Methodology

VMAF — the quality metric of choice

Source Videos

ffmpeg -i [original_mov] -vf "scale=iw/2:ih/2" -sws_flags area -c:v prores [half_size_mov]

Reconstructed Videos

Covering Watermarks

Image for post
Image for post
Mask used to remove watermarks/nag screens — it effectively hides 16.8% of all pixels when computing the VMAF scores

How VMAF scores are computed

ffmpeg -nostats -i reference.mov -i [reconstructed_mov] -i mask.png -lavfi "[2:v]split[overl1][overl2];[0:v][overl1]overlay[v1];[1:v][overl2]overlay[v2];[v1][v2]libvmaf" -f null –
[libvmaf @ 0x7f806c4505c0] VMAF score: 92.419584

Test Protocol

Test 1: Dinner Table

Transcoded version of the original “Dinner Time” video in 1080p HD

Metadata

Notes

2x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=672, y=384 of the original HD video vs every contestant’s 2x reconstruction — grabbed from the “Dinner Table” video at 00:00:03 — the downsampled 33x33 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 2x downsampled “Dinner Table” 1080p HD Shutterstock video (157 MiB) for every tested algorithm

3x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=672, y=384 of the original HD video vs every contestant’s 3x reconstruction — grabbed from the “Dinner Table” video at 00:00:03 — the downsampled 22x22 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 3x downsampled “Dinner Table” 1080p HD Shutterstock video (78 MiB) for every tested algorithm

Test 2: Turtle

Transcoded version of the original “Turtle” video in 1080p HD

Metadata

Notes

2x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=696, y=480 of the original HD video vs every contestant’s 2x reconstruction — grabbed from the “Turtle” video at 00:00:24 — the downsampled 33x33 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 2x downsampled “Turtle” 1080p HD Shutterstock video (151 MiB) for every tested algorithm

3x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=696, y=480 of the original HD video vs every contestant’s 3x reconstruction — grabbed from the “Turtle” video at 00:00:24 — the downsampled 22x22 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 3x downsampled “Turtle” 1080p HD Shutterstock video (78 MiB) for every tested algorithm

Test 3: Color Explosion

Transcoded version of the original “Color Explosion” video in 1080p HD

Metadata

Notes

2x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=912, y=620 of the original HD video vs every contestant’s 2x reconstruction — grabbed from the “Explosion” video at 00:00:03 — the downsampled 33x33 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 2x downsampled “Color Explosion” 1080p HD Shutterstock video (150 MiB) for every tested algorithm

3x reconstruction

Image for post
Image for post
66x66 cutouts at pixel position x=912, y=620 of the original HD video vs every contestant’s 3x reconstruction — grabbed from the “Explosion” video at 00:00:03 — the downsampled 22x22 source area is overlaid on each reconstruction
Image for post
Image for post
VMAF scores of reconstructing 3x downsampled “Color Explosion” 1080p HD Shutterstock video (68 MiB) for every tested algorithm

Noteworthy Observations

Future Work

Image for post
Image for post
Photo credit: Drew Beamer

Summary

Software engineer, pixel junkie & CTO of PIXOP — a startup using machine learning for enhancing the perceived quality of all images

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store