├── LICENSE ├── README.md └── images └── common_workflow.png /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2020, Leandro Moreira 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # VMAF 2 | 3 | [VMAF](https://github.com/Netflix/vmaf) is a perceptual video quality assessment algorithm developed by **Netflix**. It seeks to reflect the viewer’s perception of the streaming quality. 4 | 5 | It's a valuable quality metric that can be used to computerize the common media workflow quality analysis instead of relying on people's eyes. 6 | 7 | Its common usages, best practices, tips are explained here. Most of the information here is based on the [main repository](https://github.com/Netflix/vmaf) and the [latest post from Netflix](https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12). 8 | 9 | # TLDR; how to use 10 | 11 | ```bash 12 | # all commands were performed using 13 | # MacOS 10.15.4 - 2.3 GHz Quad-Core Intel Core i5 14 | # Docker Desktop 2.3.0 / 4 logical cores / 2 GB RAM 15 | 16 | # PLEASE PROVIDE two video files for comparison 17 | # ref.mp4 - the reference video file 18 | # distor.mp4 - the distorted videoo file (transcoded) 19 | # both files must be at the root where you ran these commands 20 | 21 | # analyzing all frames (from 1m 1080p video) 22 | time docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 23 | ffmpeg -i /files/ref.mp4 -i /files/distor.mp4 \ 24 | -lavfi libvmaf=log_fmt=json -f null - 25 | 26 | Exec FPS: 14.318552 27 | VMAF score = 93.130018 28 | 0.08s user 0.16s system 0% cpu 4:13.55 total 29 | 30 | # (QUICKER) analyzing 1 out of 5 frames (from 1m 1080p video) 31 | time docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 32 | ffmpeg -i /files/ref.mp4 -i /files/distor.mp4 \ 33 | -lavfi libvmaf=log_fmt=json:n_subsample=5 -f null - 34 | 35 | Exec FPS: 58.999069 36 | VMAF score = 93.130807 37 | 0.05s user 0.12s system 0% cpu 1:02.25 total 38 | # we make it faster (by using subsample) but the VMAF score is almost the same 39 | 40 | 41 | # analyzing different renditions 42 | docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 43 | ffmpeg -i /files/ref.mp4 -i /files/distor_lower_resolution.mp4 \ 44 | -filter_complex \ 45 | "[1:v]scale=1920x1080:flags=bicubic[main];[main][0:v]libvmaf=log_fmt=json:n_subsample=5" -f null - 46 | 47 | 48 | ``` 49 | 50 | 51 | ## Common workflow 52 | 53 | VMAF compares a reference video file against a targe one, usually, the following steps are taken: 54 | 55 | 1. **fetch** the reference video file 56 | 1. **apply transformations** to target video file 57 | * mostly: 58 | * transcode (changing codec), 59 | * transize (changing resolution), 60 | * and transrating (changing bitrate). 61 | 1. run the vmaf **comparison** 62 | 1. check the **score** (>= 70 good/fair) 63 | 1. if it's not good, go back to step 2, change the transformation's properties 64 | 65 | ![workflow](/images/common_workflow.png) 66 | 67 | ## Toolset 68 | 69 | Although VMAF can be used as standalone software, it's easier and faster to use it as a dockerized FFmpeg filter. 70 | 71 | * It's faster because it takes advantage of the FFmpeg's intermediate steps (decoding/scaling) to calc VMAF, avoiding multiple passes and unnecessary storage. 72 | * It's easier because it's a single step/tool process. 73 | 74 | The docker image we're going to use for FFmpeg/VMAF is [five82/ffmpeg-git](https://github.com/five82/ffmpeg-git) and for FFmpeg/media transformation is [jrottenberg/ffmpeg](https://github.com/jrottenberg/ffmpeg). 75 | 76 | ## Preping enviroment 77 | 78 | To practice the proposed workflow, we need to have a video file, for that matter let's download the big buck bunny. 79 | 80 | ```bash 81 | # download the file we'll use throught the examples 82 | # HUGE download ahead 330MB 83 | wget http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4 84 | 85 | # using "standard" naming 86 | # ____. 87 | mv bbb_sunflower_1080p_60fps_normal.mp4 bunny_634s_60fps_1080p_4487kb.mp4 88 | ``` 89 | 90 | Working with a 10m video long, for learning purpose, might be slow/boring, let's generate an 1m short clip. 91 | 92 | ```bash 93 | # generating a random 1 minute short clip 94 | # not changing codec though 95 | docker run --rm -v $(pwd):/files jrottenberg/ffmpeg \ 96 | -i /files/bunny_634s_60fps_1080p_4487kb.mp4 \ 97 | -ss 00:01:24 -t 00:01:00 -c copy \ 98 | /files/bunny_60s_60fps_1080p_4487kb.mp4 99 | 100 | ``` 101 | 102 | ## Getting the reference 103 | 104 | The reference file `bunny_60s_60fps_1080p_4487kb.mp4` has the following properties. 105 | 106 | | Property | Value | 107 | |----------|:-------------:| 108 | | codec | h264 | 109 | | bitrate | 4487 Kb | 110 | | resolution | 1080p | 111 | | container | mp4 | 112 | | duration | 10s | 113 | 114 | 115 | ## Applying transformations 116 | 117 | Toward to deliver video for all devices in diverse network conditions, the media needs to pass through some transformations, mostly related to the number of bits required and the resolution. 118 | 119 | #### Transrating 120 | 121 | ```bash 122 | # same resolution but different bit rate 123 | docker run --rm -v $(pwd):/files jrottenberg/ffmpeg \ 124 | -i /files/bunny_60s_60fps_1080p_4487kb.mp4 \ 125 | -c:v libx264 -crf 23 -maxrate 2M -bufsize 4M \ 126 | /files/bunny_60s_60fps_1080p_2000kb.mp4 127 | ``` 128 | | Property | Value | 129 | |----------|:-------------:| 130 | | codec | h264 | 131 | | bitrate | 2000 Kb | 132 | | resolution | 1080p | 133 | | container | mp4 | 134 | | duration | 10s | 135 | 136 | 137 | #### Transizing 138 | 139 | ```bash 140 | # changing resolution 141 | docker run --rm -v $(pwd):/files jrottenberg/ffmpeg \ 142 | -i /files/bunny_60s_60fps_1080p_4487kb.mp4 \ 143 | -vf scale=640:-1 -sws_flags lanczos \ 144 | -c:v libx264 -crf 23 -maxrate 600k -bufsize 1200k \ 145 | /files/bunny_60s_60fps_640p_600kb.mp4 146 | ``` 147 | 148 | | Property | Value | 149 | |----------|:-------------:| 150 | | codec | h264 | 151 | | bitrate | 600 Kb | 152 | | resolution | 640p | 153 | | container | mp4 | 154 | | duration | 10s | 155 | 156 | 157 | ## Run VMAF 158 | 159 | ### All frames - slower 160 | ```bash 161 | time docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 162 | ffmpeg -i /files/bunny_60s_60fps_1080p_4487kb.mp4 \ 163 | -i /files/bunny_60s_60fps_1080p_2000kb.mp4 \ 164 | -lavfi libvmaf=log_fmt=json -f null - 165 | 166 | Exec FPS: 14.318552 167 | VMAF score = 93.130018 168 | 0.08s user 0.16s system 0% cpu 4:13.55 total 169 | ``` 170 | ### Subsampling frames - faster (1 out of 5) 171 | ```bash 172 | time docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 173 | ffmpeg -i /files/bunny_60s_60fps_1080p_4487kb.mp4 \ 174 | -i /files/bunny_60s_60fps_1080p_2000kb.mp4 \ 175 | -lavfi libvmaf=log_fmt=json:n_subsample=5 -f null - 176 | 177 | Exec FPS: 58.999069 178 | VMAF score = 93.130807 179 | 0.05s user 0.12s system 0% cpu 1:02.25 total 180 | ``` 181 | 182 | Faster (almost 4x) and still kept the close VMAF score. 183 | 184 | ### Different resolution 185 | 186 | ```bash 187 | time docker run --rm -v $(pwd):/files five82/ffmpeg-git \ 188 | ffmpeg -i /files/bunny_60s_60fps_1080p_4487kb.mp4 \ 189 | -i /files/bunny_60s_60fps_640p_600kb.mp4 \ 190 | -filter_complex \ 191 | "[1:v]scale=1920x1080:flags=bicubic[main];[main][0:v]libvmaf=log_fmt=json:n_subsample=5" -f null - 192 | 193 | ``` 194 | 195 | ### How about different container check? 196 | You need to conform the timing information. 197 | 198 | ```bash 199 | docker run --rm -v $(pwd):/files jrottenberg/ffmpeg -w /files \ 200 | -i main.mpg -i ref.mkv \ 201 | -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=psnr=1:log_fmt=json" -f null - 202 | ``` 203 | Example taken from [FFmpeg's site](https://ffmpeg.org/ffmpeg-all.html#Examples-115). 204 | ### Does it work for mobile, 4k view port? 205 | 206 | For [4k, you can check FAQ](https://github.com/Netflix/vmaf/blob/master/FAQ.md#q-will-vmaf-work-on-4k-videos) and for mobile you can [check the available models](https://github.com/Netflix/vmaf/blob/master/resource/doc/models.md#predict-quality-on-a-cellular-phone-screen). 207 | -------------------------------------------------------------------------------- /images/common_workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leandromoreira/docker-ffmpeg-vmaf/a61a9342b5f243b56ee538ff388fb5bd7f611157/images/common_workflow.png --------------------------------------------------------------------------------