├── .gitignore ├── CONTRIBUTING ├── LICENSE ├── README.md ├── example-frame-ssim-y.png ├── example-ssim-y.png ├── generate_data.py ├── generate_graphs.py ├── setup.sh ├── setup_aom.sh ├── setup_openh264.sh ├── setup_vmaf.sh └── setup_yami.sh /.gitignore: -------------------------------------------------------------------------------- 1 | aom/ 2 | libvpx/ 3 | openh264/ 4 | out/ 5 | vmaf/ 6 | yami/ 7 | -------------------------------------------------------------------------------- /CONTRIBUTING: -------------------------------------------------------------------------------- 1 | Want to contribute? Great! First, read this page (including the small print at the end). 2 | 3 | ### Before you contribute 4 | Before we can use your code, you must sign the 5 | [Google Individual Contributor License Agreement] 6 | (https://cla.developers.google.com/about/google-individual) 7 | (CLA), which you can do online. The CLA is necessary mainly because you own the 8 | copyright to your changes, even after your contribution becomes part of our 9 | codebase, so we need your permission to use and distribute your code. We also 10 | need to be sure of various other things—for instance that you'll tell us if you 11 | know that your code infringes on other people's patents. You don't have to sign 12 | the CLA until after you've submitted your code for review and a member has 13 | approved it, but you must do it before we can put your code into our codebase. 14 | Before you start working on a larger contribution, you should get in touch with 15 | us first through the issue tracker with your idea so that we can help out and 16 | possibly guide you. Coordinating up front makes it much easier to avoid 17 | frustration later on. 18 | 19 | ### Code reviews 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. 22 | 23 | ### The small print 24 | Contributions made by corporations are covered by a different agreement than 25 | the one above, the 26 | [Software Grant and Corporate Contributor License Agreement] 27 | (https://cla.developers.google.com/about/google-corporate). 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Measuring Video Codec Performance 2 | 3 | _This is not an official Google product._ 4 | 5 | ![Example graph of SSIM-Y over multiple bitrates](example-ssim-y.png) 6 | ![Example graph of per-frame SSIM-Y inside a single clip](example-frame-ssim-y.png) 7 | 8 | This project contains a couple of scripts that can be used to generate quality 9 | metrics and graphs for different video codecs, encoders and settings. 10 | 11 | Quality metrics can be generated for `.y4m` as well as `.yuv` raw I420 video 12 | files. `.yuv` files require the special format `clip.WIDTH_HEIGHT.yuv:FPS` since 13 | width, height and fps metadata are not available in this containerless format. 14 | 15 | A set of industry-standard clips that can be used are available at 16 | [Xiph.org Video Test Media](https://media.xiph.org/video/derf/), aka. "derf's 17 | collection". 18 | 19 | 20 | ## Dependencies 21 | 22 | To build pinned versions of dependencies, comparison tools and libvpx run: 23 | 24 | $ ./setup.sh 25 | 26 | This requires `git` and build dependencies for libvpx that are not listed here. 27 | See build instructions for libvpx for build dependencies. 28 | 29 | To use `.y4m` files as input (instead of `.yuv`), `mediainfo` and `ffmpeg` are 30 | both required (to extract metadata and convert to `.yuv`). They can either be 31 | built and installed from source or likely by running (or similar depending on 32 | distribution): 33 | 34 | $ sudo apt-get install ffmpeg mediainfo 35 | 36 | 37 | ## Encoders 38 | 39 | After building dependencies with `./setup.sh` libvpx encoders are available. 40 | Additional encoders have to be fetched and built by using their corresponding 41 | setup scripts. 42 | 43 | `libvpx-rt:vp8` and `libvpx-rt:vp9` use libvpx encoders with settings as close 44 | as possible to settings used by Chromium's [WebRTC](https://code.webrtc.org) 45 | implementation. 46 | 47 | _TODO(pbos): Add reasonable non-realtime settings for `--good` and `--best` 48 | settings as `libvpx-good` and `libvpx-best` encoders for comparison with 49 | `aom-good`._ 50 | 51 | ### libyami 52 | 53 | To build pinned versions of libyami, VA-API and required utils run: 54 | 55 | $ ./setup_yami.sh 56 | 57 | Using libyami encoders (`yami:vp8`, `yami:vp9`) requires VA-API hardware 58 | encoding support that's at least available on newer Intel chipsets. Hardware 59 | encoding support can be probed for with `vainfo`. 60 | 61 | ### aomedia 62 | 63 | To build pinned versions of [aomedia](http://aomedia.org/) utils run: 64 | 65 | $ ./setup_aom.sh 66 | 67 | This permits encoding and evaluating quality for the AV1 video codec by running 68 | the encoder pair `aom-good:av1`. This runs a runs `aomenc` with `--good` 69 | configured as a 2-pass non-realtime encoding. This is significantly slower than 70 | realtime targets but provides better quality. 71 | 72 | _There's currently no realtime target for AV1 encoding as the codec isn't 73 | considered realtime ready at the point of writing. When it is, `aom-rt` should 74 | be added and runs could then be reasonably compared to other realtime encoders 75 | and codecs._ 76 | 77 | ### OpenH264 78 | 79 | To build pinned versions of OpenH264, run: 80 | 81 | $ ./setup_openh264.sh 82 | 83 | OpenH264 is a single-pass encoder used in WebRTC both in Chrome and Firefox. 84 | This adds the `openh264:h264` which runs `h264enc` with settings that are 85 | intended to be close to WebRTC's implementation. 86 | 87 | ## Generating Data 88 | 89 | To generate graph data (after building and installing dependencies), see: 90 | 91 | $ ./generate_data.py --help 92 | 93 | Example usage: 94 | 95 | $ ./generate_data.py --out=libvpx-rt.txt --encoders=libvpx-rt:vp8,libvpx-rt:vp9 clip1.320_240.yuv:30 clip2.320_180.yuv:30 clip3.y4m 96 | 97 | This will generate `libvpx-rt.txt` with an array of Python dictionaries with 98 | metrics used later to build graphs. This part takes a long time (may take hours 99 | or even days depending on clips, encoders and configurations) as multiple clips 100 | are encoded using various settings. Make sure to back up this file after running 101 | or risk running the whole thing all over again. 102 | 103 | To preserve encoded files, supply the `--encoded-file-dir` argument. 104 | 105 | ### VMAF 106 | 107 | Graph data can be optionally supplemented with 108 | [VMAF](https://github.com/Netflix/vmaf) metrics. To build a pinned version of 109 | VMAF, run: 110 | 111 | $ ./setup_vmaf.sh 112 | 113 | This requires several additional dependencies that are not listed here. 114 | See build instructions for VMAF for build dependencies. 115 | 116 | To enable the creation of VMAF metrics, supply the `--enable-vmaf` argument to 117 | `generate_data.py`. 118 | 119 | ### System Binaries 120 | 121 | To use system versions of binaries (either installed or otherwise available in 122 | your `PATH` variable), supply `--use-system-path` to `generate_data.py`. This 123 | will fall back to locally-compiled binaries (but warn) if the encoder commands 124 | are not available in `PATH`. 125 | 126 | 127 | ## Dumping Encoder Commands 128 | 129 | For debugging and reproducing (if you're working on encoders) it can be useful 130 | to know which encoder command produced a certain data point. 131 | 132 | To dump the commands used to generate data instead of running them, supply 133 | `--dump-commands` to `generate_data.py`. 134 | 135 | 136 | ## Generating Graphs 137 | 138 | To generate graphs from existing graph data run: 139 | 140 | $ generate_graphs.py --out-dir OUT_DIR graph_file.txt [graph_file.txt ...] 141 | 142 | This will generate several graph image files under `OUT_DIR` from data files 143 | generated using `generate_data.py`, where each clip and temporal/spatial 144 | configuration are grouped together to generate graphs comparing different 145 | encoders and layer performances for separate `SSIM`, `AvgPSNR` and `GlbPSNR` 146 | metrics. Multiple encoders and codecs are placed in the same graphs to enable a 147 | comparison between them. 148 | 149 | The script also generates graphs for encode time used. For speed tests it's 150 | recommended to use a SSD or similar, along with a single worker instance to 151 | minimize the impact that competing processes and disk/network drive performance 152 | has on time spent encoding. 153 | 154 | _The scripts make heavy use of temporary filespace. Every worker instance uses 155 | disk space roughly equal to a few copies of the original raw video file that is 156 | usually huge to begin with. To solve or mitigate issues where disk space runs 157 | out during graph-data generation, either reduce the amount of workers used with 158 | `--workers` or use another temporary directory (with more space available) by 159 | changing the `TMPDIR` environment variable._ 160 | 161 | 162 | ## Adding or Updating Encoder Implementations 163 | 164 | Adding support for additional encoders are encouraged. This requires adding an 165 | entry under `generate_data.py` which handles the new encoder, optionally 166 | including support for spatial/temporal configurations. 167 | 168 | Any improvements upstream to encoder implementations have to be pulled in by 169 | updating pinned revision hashes in corresponding setup/build scripts. 170 | -------------------------------------------------------------------------------- /example-frame-ssim-y.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google/rtc-video-quality/6a0ca781eb7563584281675e7af7ca2300c20672/example-frame-ssim-y.png -------------------------------------------------------------------------------- /example-ssim-y.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google/rtc-video-quality/6a0ca781eb7563584281675e7af7ca2300c20672/example-ssim-y.png -------------------------------------------------------------------------------- /generate_data.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # Copyright 2016 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import argparse 17 | import csv 18 | import json 19 | import multiprocessing 20 | import os 21 | import pprint 22 | import re 23 | import shutil 24 | import subprocess 25 | import sys 26 | import tempfile 27 | import threading 28 | import time 29 | 30 | libvpx_threads = 4 31 | 32 | binary_absolute_paths = {} 33 | 34 | def find_absolute_path(use_system_path, binary): 35 | global binary_absolute_paths 36 | if binary in binary_absolute_paths: 37 | return binary_absolute_paths[binary] 38 | 39 | if use_system_path: 40 | for path in os.environ["PATH"].split(os.pathsep): 41 | target = os.path.join(path.strip('"'), os.path.basename(binary)) 42 | if os.path.isfile(target) and os.access(target, os.X_OK): 43 | binary_absolute_paths[binary] = target 44 | return target 45 | target = os.path.join(os.path.dirname(os.path.abspath(__file__)), binary) 46 | if os.path.isfile(target) and os.access(target, os.X_OK): 47 | if use_system_path: 48 | print "WARNING: '%s' not in PATH (using --use-system-path), falling back on locally-compiled binary." % os.path.basename(binary) 49 | binary_absolute_paths[binary] = target 50 | return target 51 | 52 | sys.exit("ERROR: '%s' missing, did you run the corresponding setup script?" % (os.path.basename(binary) if use_system_path else target)) 53 | 54 | def aom_command(job, temp_dir): 55 | assert job['num_spatial_layers'] == 1 56 | assert job['num_temporal_layers'] == 1 57 | assert job['codec'] == 'av1' 58 | # TODO(pbos): Add realtime config (aom-rt) when AV1 is realtime ready. 59 | assert job['encoder'] == 'aom-good' 60 | 61 | (fd, first_pass_file) = tempfile.mkstemp(dir=temp_dir, suffix=".fpf") 62 | os.close(fd) 63 | 64 | (fd, encoded_filename) = tempfile.mkstemp(dir=temp_dir, suffix=".webm") 65 | os.close(fd) 66 | 67 | clip = job['clip'] 68 | fps = int(clip['fps'] + 0.5) 69 | command = [ 70 | "aom/aomenc", 71 | "--codec=av1", 72 | "-p", "2", 73 | "--fpf=%s" % first_pass_file, 74 | "--good", 75 | "--cpu-used=0", 76 | "--target-bitrate=%d" % job['target_bitrates_kbps'][0], 77 | '--fps=%d/1' % fps, 78 | "--lag-in-frames=25", 79 | "--min-q=0", 80 | "--max-q=63", 81 | "--auto-alt-ref=1", 82 | "--kf-max-dist=150", 83 | "--kf-min-dist=0", 84 | "--drop-frame=0", 85 | "--static-thresh=0", 86 | "--bias-pct=50", 87 | "--minsection-pct=0", 88 | "--maxsection-pct=2000", 89 | "--arnr-maxframes=7", 90 | "--arnr-strength=5", 91 | "--sharpness=0", 92 | "--undershoot-pct=100", 93 | "--overshoot-pct=100", 94 | "--frame-parallel=0", 95 | "--tile-columns=0", 96 | "--profile=0", 97 | '--width=%d' % clip['width'], 98 | '--height=%d' % clip['height'], 99 | '--output=%s' % encoded_filename, 100 | clip['yuv_file'], 101 | ] 102 | encoded_files = [{'spatial-layer': 0, 'temporal-layer': 0, 'filename': encoded_filename}] 103 | return (command, encoded_files) 104 | 105 | def libvpx_tl_command(job, temp_dir): 106 | # Parameters are intended to be as close as possible to realtime settings used 107 | # in WebRTC. 108 | assert job['num_temporal_layers'] <= 3 109 | # TODO(pbos): Account for low resolution CPU levels (see below). 110 | codec_cpu = 6 if job['codec'] == 'vp8' else 7 111 | layer_strategy = 8 if job['num_temporal_layers'] == 2 else 10 112 | outfile_prefix = '%s/out' % temp_dir 113 | clip = job['clip'] 114 | fps = int(clip['fps'] + 0.5) 115 | 116 | command = [ 117 | 'libvpx/examples/vpx_temporal_svc_encoder', 118 | clip['yuv_file'], 119 | outfile_prefix, 120 | job['codec'], 121 | clip['width'], 122 | clip['height'], 123 | '1', 124 | fps, 125 | codec_cpu, 126 | '0', 127 | libvpx_threads, 128 | layer_strategy 129 | ] + job['target_bitrates_kbps'] 130 | command = [str(i) for i in command] 131 | encoded_files = [{'spatial-layer': 0, 'temporal-layer': i, 'filename': "%s_%d.ivf" % (outfile_prefix, i)} for i in range(job['num_temporal_layers'])] 132 | 133 | return ([str(i) for i in command], encoded_files) 134 | 135 | def libvpx_command(job, temp_dir): 136 | # Parameters are intended to be as close as possible to realtime settings used 137 | # in WebRTC. 138 | if (job['num_temporal_layers'] > 1): 139 | return libvpx_tl_command(job, temp_dir) 140 | assert job['num_spatial_layers'] == 1 141 | # TODO(pbos): Account for low resolutions (use -4 and 5 for CPU levels). 142 | common_params = [ 143 | "--lag-in-frames=0", 144 | "--error-resilient=1", 145 | "--kf-min-dist=3000", 146 | "--kf-max-dist=3000", 147 | "--static-thresh=1", 148 | "--end-usage=cbr", 149 | "--undershoot-pct=100", 150 | "--overshoot-pct=15", 151 | "--buf-sz=1000", 152 | "--buf-initial-sz=500", 153 | "--buf-optimal-sz=600", 154 | "--max-intra-rate=900", 155 | "--resize-allowed=0", 156 | "--drop-frame=0", 157 | "--passes=1", 158 | "--rt", 159 | "--noise-sensitivity=0", 160 | "--threads=%d" % libvpx_threads, 161 | ] 162 | if job['codec'] == 'vp8': 163 | codec_params = [ 164 | "--codec=vp8", 165 | "--cpu-used=-6", 166 | "--min-q=2", 167 | "--max-q=56", 168 | "--screen-content-mode=0", 169 | ] 170 | elif job['codec'] == 'vp9': 171 | codec_params = [ 172 | "--codec=vp9", 173 | "--cpu-used=7", 174 | "--min-q=2", 175 | "--max-q=52", 176 | "--aq-mode=3", 177 | ] 178 | 179 | (fd, encoded_filename) = tempfile.mkstemp(dir=temp_dir, suffix=".webm") 180 | os.close(fd) 181 | 182 | clip = job['clip'] 183 | # Round FPS. For quality comparisons it's likely close enough to not be 184 | # misrepresentative. From a quality perspective there's no point to fully 185 | # respecting NTSC or other non-integer FPS formats here. 186 | fps = int(clip['fps'] + 0.5) 187 | 188 | command = ['libvpx/vpxenc'] + codec_params + common_params + [ 189 | '--fps=%d/1' % fps, 190 | '--target-bitrate=%d' % job['target_bitrates_kbps'][0], 191 | '--width=%d' % clip['width'], 192 | '--height=%d' % clip['height'], 193 | '--output=%s' % encoded_filename, 194 | clip['yuv_file'] 195 | ] 196 | encoded_files = [{'spatial-layer': 0, 'temporal-layer': 0, 'filename': encoded_filename}] 197 | return (command, encoded_files) 198 | 199 | 200 | def openh264_command(job, temp_dir): 201 | assert job['codec'] == 'h264' 202 | # TODO(pbos): Consider AVC support. 203 | assert job['num_spatial_layers'] == 1 204 | # TODO(pbos): Add temporal-layer support (-numtl). 205 | assert job['num_temporal_layers'] == 1 206 | 207 | (fd, encoded_filename) = tempfile.mkstemp(dir=temp_dir, suffix=".264") 208 | os.close(fd) 209 | 210 | clip = job['clip'] 211 | 212 | command = [ 213 | 'openh264/h264enc', 214 | '-rc', 1, 215 | '-denois', 0, 216 | '-scene', 0, 217 | '-bgd', 0, 218 | '-fs', 0, 219 | '-tarb', job['target_bitrates_kbps'][0], 220 | '-sw', clip['width'], 221 | '-sh', clip['height'], 222 | '-frin', clip['fps'], 223 | '-org', clip['yuv_file'], 224 | '-bf', encoded_filename, 225 | '-numl', 1, 226 | '-dw', 0, clip['width'], 227 | '-dh', 0, clip['height'], 228 | '-frout', 0, clip['fps'], 229 | '-ltarb', 0, job['target_bitrates_kbps'][0], 230 | ] 231 | encoded_files = [{'spatial-layer': 0, 'temporal-layer': 0, 'filename': encoded_filename}] 232 | return ([str(i) for i in command], encoded_files) 233 | 234 | 235 | def yami_command(job, temp_dir): 236 | assert job['num_spatial_layers'] == 1 237 | assert job['num_temporal_layers'] == 1 238 | 239 | (fd, encoded_filename) = tempfile.mkstemp(dir=temp_dir, suffix=".ivf") 240 | os.close(fd) 241 | 242 | clip = job['clip'] 243 | # Round FPS. For quality comparisons it's likely close enough to not be 244 | # misrepresentative. From a quality perspective there's no point to fully 245 | # respecting NTSC or other non-integer FPS formats here. 246 | fps = int(clip['fps'] + 0.5) 247 | 248 | command = [ 249 | 'yami/libyami/bin/yamiencode', 250 | '--rcmode', 'CBR', 251 | '--ipperiod', 1, 252 | '--intraperiod', 3000, 253 | '-c', job['codec'].upper(), 254 | '-i', clip['yuv_file'], 255 | '-W', clip['width'], 256 | '-H', clip['height'], 257 | '-f', fps, 258 | '-o', encoded_filename, 259 | '-b', job['target_bitrates_kbps'][0], 260 | ] 261 | encoded_files = [{'spatial-layer': 0, 'temporal-layer': 0, 'filename': encoded_filename}] 262 | return ([str(i) for i in command], encoded_files) 263 | 264 | encoder_commands = { 265 | 'aom-good' : aom_command, 266 | 'openh264' : openh264_command, 267 | 'libvpx-rt' : libvpx_command, 268 | 'yami' : yami_command, 269 | } 270 | 271 | 272 | yuv_clip_pattern = re.compile(r"^(.*[\._](\d+)_(\d+).yuv):(\d+)$") 273 | def clip_arg(clip): 274 | (file_root, file_ext) = os.path.splitext(clip) 275 | if file_ext == '.y4m': 276 | width = int(subprocess.check_output(["mediainfo", "--Inform=Video;%Width%", clip])) 277 | height = int(subprocess.check_output(["mediainfo", "--Inform=Video;%Height%", clip])) 278 | fps = float(subprocess.check_output(["mediainfo", "--Inform=Video;%FrameRate%", clip])) 279 | return {'input_file': clip, 'height': height, 'width': width, 'fps': fps, 'file_type': 'y4m'} 280 | 281 | # Make sure YUV files are correctly formatted + look readable before actually 282 | # running the script on them. 283 | clip_match = yuv_clip_pattern.match(clip) 284 | if not clip_match: 285 | raise argparse.ArgumentTypeError("Argument '%s' doesn't match input format.\n" % clip) 286 | input_file = clip_match.group(1) 287 | if not os.path.isfile(input_file) or not os.access(input_file, os.R_OK): 288 | raise argparse.ArgumentTypeError("'%s' is either not a file or cannot be opened for reading.\n" % input_file) 289 | return {'input_file': clip_match.group(1), 'width': int(clip_match.group(2)), 'height': int(clip_match.group(3)), 'fps' : float(clip_match.group(4)), 'file_type': 'yuv'} 290 | 291 | 292 | def encoder_pairs(string): 293 | pair_pattern = re.compile(r"^([\w\-]+):(\w+)$") 294 | encoders = [] 295 | for pair in string.split(','): 296 | pair_match = pair_pattern.match(pair) 297 | if not pair_match: 298 | raise argparse.ArgumentTypeError("Argument '%s' of '%s' doesn't match input format.\n" % (pair, string)) 299 | if not pair_match.group(1) in encoder_commands: 300 | raise argparse.ArgumentTypeError("Unknown encoder: '%s' in pair '%s'\n" % (pair_match.group(1), pair)) 301 | encoders.append((pair_match.group(1), pair_match.group(2))) 302 | return encoders 303 | 304 | 305 | def writable_dir(directory): 306 | if not os.path.isdir(directory) or not os.access(directory, os.W_OK): 307 | raise argparse.ArgumentTypeError("'%s' is either not a directory or cannot be opened for writing.\n" % directory) 308 | return directory 309 | 310 | 311 | def positive_int(num): 312 | num_int = int(num) 313 | if num_int <= 0: 314 | raise argparse.ArgumentTypeError("'%d' is not a positive integer.\n" % num) 315 | return num_int 316 | 317 | 318 | parser = argparse.ArgumentParser(description='Generate graph data for video-quality comparison.') 319 | parser.add_argument('clips', nargs='+', metavar='clip_WIDTH_HEIGHT.yuv:FPS|clip.y4m', type=clip_arg) 320 | parser.add_argument('--dump-commands', action='store_true') 321 | parser.add_argument('--enable-vmaf', action='store_true') 322 | parser.add_argument('--encoded-file-dir', default=None, type=writable_dir) 323 | parser.add_argument('--encoders', required=True, metavar='encoder:codec,encoder:codec...', type=encoder_pairs) 324 | parser.add_argument('--frame-offset', default=0, type=positive_int) 325 | parser.add_argument('--num-frames', default=-1, type=positive_int) 326 | # TODO(pbos): Add support for multiple spatial layers. 327 | parser.add_argument('--num-spatial-layers', type=int, default=1, choices=[1]) 328 | parser.add_argument('--num-temporal-layers', type=int, default=1, choices=[1,2,3]) 329 | parser.add_argument('--out', required=True, metavar='output.txt', type=argparse.FileType('w')) 330 | parser.add_argument('--use-system-path', action='store_true') 331 | parser.add_argument('--workers', type=int, default=multiprocessing.cpu_count()) 332 | 333 | 334 | def prepare_clips(args, temp_dir): 335 | clips = args.clips 336 | y4m_clips = [clip for clip in clips if clip['file_type'] == 'y4m'] 337 | if y4m_clips: 338 | print "Converting %d .y4m clip%s..." % (len(y4m_clips), "" if len(y4m_clips) == 1 else "s") 339 | for clip in y4m_clips: 340 | (fd, yuv_file) = tempfile.mkstemp(dir=temp_dir, suffix=".%d_%d.yuv" % (clip['width'], clip['height'])) 341 | os.close(fd) 342 | with open(os.devnull, 'w') as devnull: 343 | subprocess.check_call(['ffmpeg', '-y', '-i', clip['input_file'], yuv_file], stdout=devnull, stderr=devnull) 344 | clip['yuv_file'] = yuv_file 345 | for clip in clips: 346 | clip['sha1sum'] = subprocess.check_output(['sha1sum', clip['input_file']]).split(' ', 1)[0] 347 | if 'yuv_file' not in clip: 348 | clip['yuv_file'] = clip['input_file'] 349 | frame_size = 6 * clip['width'] * clip['height'] / 4 350 | input_yuv_filesize = os.path.getsize(clip['yuv_file']) 351 | clip['input_total_frames'] = input_yuv_filesize / frame_size 352 | # Truncate file if necessary. 353 | if args.frame_offset > 0 or args.num_frames > 0: 354 | (fd, truncated_filename) = tempfile.mkstemp(dir=temp_dir, suffix=".yuv") 355 | blocksize = 2048 * 1024 356 | total_filesize = args.num_frames * frame_size 357 | with os.fdopen(fd, 'wb', blocksize) as truncated_file: 358 | with open(clip['yuv_file'], 'rb') as original_file: 359 | original_file.seek(args.frame_offset * frame_size) 360 | while total_filesize > 0: 361 | data = original_file.read(blocksize if blocksize < total_filesize else total_filesize) 362 | truncated_file.write(data) 363 | total_filesize -= blocksize 364 | clip['yuv_file'] = truncated_filename 365 | 366 | 367 | def decode_file(job, temp_dir, encoded_file): 368 | (fd, decoded_file) = tempfile.mkstemp(dir=temp_dir, suffix=".yuv") 369 | os.close(fd) 370 | (fd, framestats_file) = tempfile.mkstemp(dir=temp_dir, suffix=".csv") 371 | os.close(fd) 372 | with open(os.devnull, 'w') as devnull: 373 | if job['codec'] in ['av1', 'vp8', 'vp9']: 374 | decoder = 'aom/aomdec' if job['codec'] == 'av1' else 'libvpx/vpxdec' 375 | subprocess.check_call([decoder, '--i420', '--codec=%s' % job['codec'], '-o', decoded_file, encoded_file, '--framestats=%s' % framestats_file], stdout=devnull, stderr=devnull) 376 | elif job['codec'] == 'h264': 377 | subprocess.check_call(['openh264/h264dec', encoded_file, decoded_file], stdout=devnull, stderr=devnull) 378 | # TODO(pbos): Generate H264 framestats. 379 | framestats_file = None 380 | return (decoded_file, framestats_file) 381 | 382 | 383 | def add_framestats(results_dict, framestats_file, statstype): 384 | with open(framestats_file) as csvfile: 385 | reader = csv.DictReader(csvfile) 386 | for row in reader: 387 | for (metric, value) in row.items(): 388 | metric_key = 'frame-%s' % metric 389 | if metric_key not in results_dict: 390 | results_dict[metric_key] = [] 391 | results_dict[metric_key].append(statstype(value)) 392 | 393 | 394 | def generate_metrics(results_dict, job, temp_dir, encoded_file): 395 | (decoded_file, decoder_framestats) = decode_file(job, temp_dir, encoded_file['filename']) 396 | clip = job['clip'] 397 | temporal_divide = 2 ** (job['num_temporal_layers'] - 1 - encoded_file['temporal-layer']) 398 | temporal_skip = temporal_divide - 1 399 | # TODO(pbos): Perform SSIM on downscaled .yuv files for spatial layers. 400 | (fd, metrics_framestats) = tempfile.mkstemp(dir=temp_dir, suffix=".csv") 401 | os.close(fd) 402 | ssim_results = subprocess.check_output(['libvpx/tools/tiny_ssim', clip['yuv_file'], decoded_file, "%dx%d" % (results_dict['width'], results_dict['height']), str(temporal_skip), metrics_framestats]).splitlines() 403 | metric_map = { 404 | 'AvgPSNR': 'avg-psnr', 405 | 'AvgPSNR-Y': 'avg-psnr-y', 406 | 'AvgPSNR-U': 'avg-psnr-u', 407 | 'AvgPSNR-V': 'avg-psnr-v', 408 | 'GlbPSNR': 'glb-psnr', 409 | 'GlbPSNR-Y': 'glb-psnr-y', 410 | 'GlbPSNR-U': 'glb-psnr-u', 411 | 'GlbPSNR-V': 'glb-psnr-v', 412 | 'SSIM': 'ssim', 413 | 'SSIM-Y': 'ssim-y', 414 | 'SSIM-U': 'ssim-u', 415 | 'SSIM-V': 'ssim-v', 416 | 'VpxSSIM': 'vpx-ssim', 417 | } 418 | for line in ssim_results: 419 | if not line: 420 | continue 421 | (metric, value) = line.split(': ') 422 | if metric in metric_map: 423 | results_dict[metric_map[metric]] = float(value) 424 | elif metric == 'Nframes': 425 | layer_frames = int(value) 426 | results_dict['frame-count'] = layer_frames 427 | 428 | if decoder_framestats: 429 | add_framestats(results_dict, decoder_framestats, int) 430 | add_framestats(results_dict, metrics_framestats, float) 431 | 432 | if args.enable_vmaf: 433 | vmaf_results = subprocess.check_output(['vmaf/run_vmaf', 'yuv420p', str(results_dict['width']), str(results_dict['height']), clip['yuv_file'], decoded_file, '--out-fmt', 'json']) 434 | vmaf_obj = json.loads(vmaf_results) 435 | results_dict['vmaf'] = float(vmaf_obj['aggregate']['VMAF_score']) 436 | 437 | results_dict['frame-vmaf'] = [] 438 | for frame in vmaf_obj['frames']: 439 | results_dict['frame-vmaf'].append(frame['VMAF_score']) 440 | 441 | layer_fps = clip['fps'] / temporal_divide 442 | results_dict['layer-fps'] = layer_fps 443 | 444 | spatial_divide = 2 ** (job['num_spatial_layers'] - 1 - encoded_file['spatial-layer']) 445 | results_dict['layer-width'] = results_dict['width'] // spatial_divide 446 | results_dict['layer-height'] = results_dict['height'] // spatial_divide 447 | 448 | target_bitrate_bps = job['target_bitrates_kbps'][encoded_file['temporal-layer']] * 1000 449 | bitrate_used_bps = os.path.getsize(encoded_file['filename']) * 8 * layer_fps / layer_frames 450 | results_dict['target-bitrate-bps'] = target_bitrate_bps 451 | results_dict['actual-bitrate-bps'] = bitrate_used_bps 452 | results_dict['bitrate-utilization'] = float(bitrate_used_bps) / target_bitrate_bps 453 | 454 | 455 | def run_command(job, (command, encoded_files), job_temp_dir, encoded_file_dir): 456 | clip = job['clip'] 457 | start_time = time.time() 458 | try: 459 | process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 460 | except OSError as e: 461 | return (None, "> %s\n%s" % (" ".join(command), e)) 462 | (output, _) = process.communicate() 463 | actual_encode_ms = (time.time() - start_time) * 1000 464 | input_yuv_filesize = os.path.getsize(clip['yuv_file']) 465 | input_num_frames = int(input_yuv_filesize / (6 * clip['width'] * clip['height'] / 4)) 466 | target_encode_ms = float(input_num_frames) * 1000 / clip['fps'] 467 | if process.returncode != 0: 468 | return (None, "> %s\n%s" % (" ".join(command), output)) 469 | results = [{} for i in range(len(encoded_files))] 470 | for i in range(len(results)): 471 | results_dict = results[i] 472 | results_dict['input-file'] = os.path.basename(clip['input_file']) 473 | results_dict['input-file-sha1sum'] = clip['sha1sum'] 474 | results_dict['input-total-frames'] = clip['input_total_frames'] 475 | results_dict['frame-offset'] = args.frame_offset 476 | results_dict['bitrate-config-kbps'] = job['target_bitrates_kbps'] 477 | results_dict['layer-pattern'] = "%dsl%dtl" % (job['num_spatial_layers'], job['num_temporal_layers']) 478 | results_dict['encoder'] = job['encoder'] 479 | results_dict['codec'] = job['codec'] 480 | results_dict['height'] = clip['height'] 481 | results_dict['width'] = clip['width'] 482 | results_dict['fps'] = clip['fps'] 483 | results_dict['actual-encode-time-ms'] = actual_encode_ms 484 | results_dict['target-encode-time-ms'] = target_encode_ms 485 | results_dict['encode-time-utilization'] = actual_encode_ms / target_encode_ms 486 | layer = encoded_files[i] 487 | 488 | results_dict['temporal-layer'] = layer['temporal-layer'] 489 | results_dict['spatial-layer'] = layer['spatial-layer'] 490 | 491 | generate_metrics(results_dict, job, job_temp_dir, layer) 492 | if encoded_file_dir: 493 | encoded_file_pattern = "%s-%s-%s-%dsl%dtl-%d-sl%d-tl%d%s" % (os.path.splitext(os.path.basename(clip['input_file']))[0], job['encoder'], job['codec'], job['num_spatial_layers'], job['num_temporal_layers'], job['target_bitrates_kbps'][-1], layer['spatial-layer'], layer['temporal-layer'], os.path.splitext(layer['filename'])[1]) 494 | shutil.move(layer['filename'], os.path.join(encoded_file_dir, encoded_file_pattern)) 495 | else: 496 | os.remove(layer['filename']) 497 | 498 | shutil.rmtree(job_temp_dir) 499 | 500 | return (results, output) 501 | 502 | 503 | def find_bitrates(width, height): 504 | # Do multiples of 100, because grouping based on bitrate splits in 505 | # generate_graphs.py doesn't round properly. 506 | 507 | # TODO(pbos): Propagate the bitrate split in the data instead of inferring it 508 | # from the job to avoid rounding errors. 509 | 510 | # Significantly lower than exact value, so 800p still counts as 720p for 511 | # instance. 512 | pixel_bound = width * height / 1.5 513 | if pixel_bound <= 320 * 240: 514 | return [100, 200, 400, 600, 800, 1200] 515 | if pixel_bound <= 640 * 480: 516 | return [200, 300, 500, 800, 1200, 2000] 517 | if pixel_bound <= 1280 * 720: 518 | return [400, 800, 1200, 1600, 2500, 5000] 519 | if pixel_bound <= 1920 * 1080: 520 | return [800, 1200, 2000, 3000, 5000, 10000] 521 | return [1200, 1800, 3000, 6000, 10000, 15000] 522 | 523 | 524 | layer_bitrates = [[1], [0.6, 1], [0.45, 0.65, 1]] 525 | def split_temporal_bitrates_kbps(target_bitrate_kbps, num_temporal_layers): 526 | bitrates_kbps = [] 527 | for i in range(num_temporal_layers): 528 | layer_bitrate_kbps = int(layer_bitrates[num_temporal_layers - 1][i] * target_bitrate_kbps) 529 | bitrates_kbps.append(layer_bitrate_kbps) 530 | return bitrates_kbps 531 | 532 | 533 | def generate_jobs(args, temp_dir): 534 | jobs = [] 535 | for clip in args.clips: 536 | bitrates = find_bitrates(clip['width'], clip['height']) 537 | for bitrate_kbps in bitrates: 538 | for (encoder, codec) in args.encoders: 539 | job = { 540 | 'encoder': encoder, 541 | 'codec': codec, 542 | 'clip': clip, 543 | 'target_bitrates_kbps': split_temporal_bitrates_kbps(bitrate_kbps, args.num_temporal_layers), 544 | 'num_spatial_layers': args.num_spatial_layers, 545 | 'num_temporal_layers': args.num_temporal_layers, 546 | } 547 | job_temp_dir = tempfile.mkdtemp(dir=temp_dir) 548 | (command, encoded_files) = encoder_commands[job['encoder']](job, job_temp_dir) 549 | command[0] = find_absolute_path(args.use_system_path, command[0]) 550 | jobs.append((job, (command, encoded_files), job_temp_dir)) 551 | return jobs 552 | 553 | def start_daemon(func): 554 | t = threading.Thread(target=func) 555 | t.daemon = True 556 | t.start() 557 | return t 558 | 559 | def job_to_string(job): 560 | return "%s:%s %dsl%dtl %s %s" % (job['encoder'], job['codec'], job['num_spatial_layers'], job['num_temporal_layers'], ":".join(str(i) for i in job['target_bitrates_kbps']), os.path.basename(job['clip']['input_file'])) 561 | 562 | def worker(): 563 | global args 564 | global jobs 565 | global current_job 566 | global has_errored 567 | global total_jobs 568 | pp = pprint.PrettyPrinter(indent=2) 569 | while True: 570 | with thread_lock: 571 | if not jobs: 572 | return 573 | (job, command, job_temp_dir) = jobs.pop() 574 | 575 | (results, error) = run_command(job, command, job_temp_dir, args.encoded_file_dir) 576 | 577 | job_str = job_to_string(job) 578 | 579 | with thread_lock: 580 | current_job += 1 581 | run_ok = results is not None 582 | print "[%d/%d] %s (%s)" % (current_job, total_jobs, job_str, "OK" if run_ok else "ERROR") 583 | if not run_ok: 584 | has_errored = True 585 | print error 586 | else: 587 | for result in results: 588 | args.out.write(pp.pformat(result)) 589 | args.out.write(',\n') 590 | args.out.flush() 591 | 592 | 593 | thread_lock = threading.Lock() 594 | 595 | def main(): 596 | global args 597 | global jobs 598 | global total_jobs 599 | global current_job 600 | global has_errored 601 | 602 | temp_dir = tempfile.mkdtemp() 603 | 604 | args = parser.parse_args() 605 | prepare_clips(args, temp_dir) 606 | jobs = generate_jobs(args, temp_dir) 607 | total_jobs = len(jobs) 608 | current_job = 0 609 | has_errored = False 610 | 611 | if args.dump_commands: 612 | for (job, (command, encoded_files), job_temp_dir) in jobs: 613 | current_job += 1 614 | print "[%d/%d] %s" % (current_job, total_jobs, job_to_string(job)) 615 | print "> %s" % " ".join(command) 616 | print 617 | 618 | shutil.rmtree(temp_dir) 619 | return 0 620 | 621 | # Make sure commands for quality metrics are present. 622 | find_absolute_path(False, 'libvpx/tools/tiny_ssim') 623 | for (encoder, codec) in args.encoders: 624 | if codec in ['vp8', 'vp9']: 625 | find_absolute_path(False, 'libvpx/vpxdec') 626 | elif codec == 'av1': 627 | find_absolute_path(False, 'aom/aomdec') 628 | elif codec == 'h264': 629 | find_absolute_path(False, 'openh264/h264dec') 630 | if args.enable_vmaf: 631 | find_absolute_path(False, 'vmaf/run_vmaf') 632 | 633 | print "[0/%d] Running jobs..." % total_jobs 634 | 635 | args.out.write('[') 636 | 637 | workers = [start_daemon(worker) for i in range(args.workers)] 638 | [t.join() for t in workers] 639 | 640 | args.out.write(']\n') 641 | 642 | shutil.rmtree(temp_dir) 643 | return 1 if has_errored else 0 644 | 645 | if __name__ == '__main__': 646 | sys.exit(main()) 647 | -------------------------------------------------------------------------------- /generate_graphs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # Copyright 2016 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import argparse 17 | import ast 18 | import matplotlib.pyplot as plt 19 | import os 20 | import re 21 | 22 | layer_regex_pattern = re.compile(r"^(\d)sl(\d)tl$") 23 | def writable_dir(directory): 24 | if not os.path.isdir(directory) or not os.access(directory, os.W_OK): 25 | raise argparse.ArgumentTypeError("'%s' is either not a directory or cannot be opened for writing.\n" % directory) 26 | return directory 27 | 28 | def formats(formats_list): 29 | formats = formats_list.split(',') 30 | for extension in formats: 31 | if extension not in ['png', 'svg']: 32 | raise argparse.ArgumentTypeError("'%s' is not a valid file format.\n" % extension) 33 | return formats 34 | 35 | 36 | parser = argparse.ArgumentParser(description='Generate graphs from data files.') 37 | parser.add_argument('graph_files', nargs='+', metavar='graph_file.txt', type=argparse.FileType('r')) 38 | parser.add_argument('--out-dir', required=True, type=writable_dir) 39 | parser.add_argument('--formats', type=formats, metavar='png,svg', help='comma-separated list of output formats', default=['png', 'svg']) 40 | 41 | def split_data(graph_data, attribute): 42 | groups = {} 43 | for element in graph_data: 44 | value = element[attribute] 45 | if value not in groups: 46 | groups[value] = [] 47 | groups[value].append(element) 48 | return groups.values() 49 | 50 | def normalize_bitrate_config_string(config): 51 | return ":".join([str(int(x * 100.0 / config[-1])) for x in config]) 52 | 53 | 54 | def generate_graphs(output_dict, graph_data, target_metric, bitrate_config_string): 55 | lines = {} 56 | for encoder in split_data(graph_data, 'encoder'): 57 | for codec in split_data(encoder, 'codec'): 58 | for layer in split_data(codec, 'temporal-layer'): 59 | metric_data = [] 60 | for data in layer: 61 | if target_metric not in data: 62 | return 63 | metric_data.append((data['target-bitrate-bps']/1000, data[target_metric], data['bitrate-utilization'])) 64 | line_name = '%s:%s (tl%d)' % (layer[0]['encoder'], layer[0]['codec'], layer[0]['temporal-layer']) 65 | # Sort points on target bitrate. 66 | lines[line_name] = sorted(metric_data, key=lambda point: point[0]) 67 | 68 | graph_name = "%s-%s-%s:%s" % (graph_data[0]['input-file'], graph_data[0]['layer-pattern'], bitrate_config_string, target_metric) 69 | output_dict[('', graph_name)] = lines 70 | 71 | def main(): 72 | args = parser.parse_args() 73 | graph_data = [] 74 | for f in args.graph_files: 75 | graph_data += ast.literal_eval(f.read()) 76 | 77 | graph_dict = {} 78 | for input_files in split_data(graph_data, 'input-file'): 79 | for layer_pattern in split_data(input_files, 'layer-pattern'): 80 | for data in layer_pattern: 81 | metrics = [ 82 | 'vpx-ssim', 83 | 'ssim', 84 | 'ssim-y', 85 | 'ssim-u', 86 | 'ssim-v', 87 | 'avg-psnr', 88 | 'avg-psnr-y', 89 | 'avg-psnr-u', 90 | 'avg-psnr-v', 91 | 'glb-psnr', 92 | 'glb-psnr-y', 93 | 'glb-psnr-u', 94 | 'glb-psnr-v', 95 | 'encode-time-utilization', 96 | 'vmaf' 97 | ] 98 | for metric in metrics: 99 | generate_graphs(graph_dict, layer_pattern, metric, normalize_bitrate_config_string(data['bitrate-config-kbps'])) 100 | 101 | for point in graph_data: 102 | pattern_match = layer_regex_pattern.match(point['layer-pattern']) 103 | num_temporal_layers = int(pattern_match.group(2)) 104 | temporal_divide = 2 ** (num_temporal_layers - 1 - point['temporal-layer']) 105 | frame_metrics = [ 106 | 'frame-ssim', 107 | 'frame-ssim-y', 108 | 'frame-ssim-u', 109 | 'frame-ssim-v', 110 | 'frame-psnr', 111 | 'frame-psnr-y', 112 | 'frame-psnr-u', 113 | 'frame-psnr-v', 114 | 'frame-qp', 115 | 'frame-bytes', 116 | 'frame-vmaf' 117 | ] 118 | for target_metric in frame_metrics: 119 | if target_metric not in point: 120 | continue 121 | 122 | split_on_codecs = target_metric == 'frame-qp' 123 | 124 | if split_on_codecs: 125 | graph_name = "%s-%s-%s-%dkbps-tl%d-%s:%s" % (point['input-file'], point['layer-pattern'], normalize_bitrate_config_string(point['bitrate-config-kbps']), point['bitrate-config-kbps'][-1], point['temporal-layer'], point['codec'], target_metric) 126 | line_name = '%s' % point['encoder'] 127 | else: 128 | graph_name = "%s-%s-%s-%dkbps-tl%d:%s" % (point['input-file'], point['layer-pattern'], normalize_bitrate_config_string(point['bitrate-config-kbps']), point['bitrate-config-kbps'][-1], point['temporal-layer'], target_metric) 129 | line_name = '%s:%s' % (point['encoder'], point['codec']) 130 | graph_info = ('frame-data-%s/' % point['input-file'], graph_name) 131 | if not graph_info in graph_dict: 132 | graph_dict[graph_info] = {} 133 | line = [] 134 | for idx, val in enumerate(point[target_metric]): 135 | frame_size = point['frame-bytes'][idx] if 'frame-bytes' in point else -1 136 | line.append((point['frame-offset'] + temporal_divide * idx + 1, val, frame_size)) 137 | graph_dict[graph_info][line_name] = line 138 | 139 | current_graph = 1 140 | total_graphs = len(graph_dict) 141 | for (subdir, graph_name), lines in graph_dict.iteritems(): 142 | print "[%d/%d] %s" % (current_graph, total_graphs, graph_name) 143 | current_graph += 1 144 | metric = graph_name.split(':')[-1] 145 | fig, ax = plt.subplots() 146 | ax.set_title(graph_name) 147 | frame_data = 'frame-' in metric 148 | ax2 = None 149 | ax2_bitrate_utilization = False 150 | linestyle = 'o--' 151 | ax2_linestyle = 'x-' 152 | 153 | if frame_data: 154 | ax.set_xlabel('Frame') 155 | linestyle = '-' 156 | if metric == 'frame-bytes': 157 | ax.set_ylabel('Frame Size (bytes / frame)') 158 | else: 159 | ax.set_ylabel(metric.replace('frame-', '').upper()) 160 | ax2 = ax.twinx() 161 | ax2.set_ylabel('Frame Size (bytes / frame)') 162 | ax2_linestyle = '-' 163 | elif metric == 'encode-time-utilization': 164 | ax.set_xlabel('Layer Target Bitrate (kbps)') 165 | ax.set_ylabel('Encode Time (fraction)') 166 | # Draw a reference line for realtime. 167 | ax.axhline(1.0, color='k', alpha=0.2, linestyle='--') 168 | else: 169 | ax.set_xlabel('Layer Target Bitrate (kbps)') 170 | ax.set_ylabel(metric.upper()) 171 | ax2 = ax.twinx() 172 | ax2.set_ylabel('Bitrate Utilization (actual / target)') 173 | ax2_bitrate_utilization = True 174 | 175 | for title in sorted(lines.keys()): 176 | points = lines[title] 177 | x = [] 178 | y = [] 179 | y2 = [] 180 | for bitrate_kbps, value, utilization in points: 181 | x.append(bitrate_kbps) 182 | y.append(value) 183 | y2.append(utilization) 184 | ax.plot(x, y, linestyle, linewidth=1, label=title) 185 | if ax2: 186 | ax2.plot(x, y2, ax2_linestyle, alpha=0.2) 187 | ax.legend(loc='best', fancybox=True, framealpha=0.5) 188 | 189 | if metric == 'encode-time-utilization': 190 | # Make sure the horizontal reference line at 1.0 can be seen. 191 | (lower, upper) = ax.get_ylim() 192 | if upper < 1.10: 193 | ax.set_ylim(top=1.10) 194 | 195 | # TODO(pbos): Read 'input-total-frames' from input and set as graph xlim. 196 | if frame_data: 197 | ax.set_xlim(left=0) 198 | 199 | if ax2_bitrate_utilization: 200 | # Set bitrate limit axes to +/- 20%. 201 | ax2.set_ylim(bottom=0.80, top=1.20) 202 | 203 | for extension in args.formats: 204 | graph_dir = os.path.join(args.out_dir, extension, subdir) 205 | if not os.path.exists(graph_dir): 206 | os.makedirs(graph_dir) 207 | plt.savefig(os.path.join(graph_dir, "%s.%s" % (graph_name.replace(":", "-"), extension))) 208 | plt.close() 209 | 210 | if __name__ == '__main__': 211 | main() 212 | 213 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2016 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | set -x 17 | 18 | # Download libvpx if not available. 19 | if [ ! -d libvpx ]; then 20 | git clone https://chromium.googlesource.com/webm/libvpx 21 | fi 22 | 23 | # Check out the pinned libvpx version. 24 | pushd libvpx 25 | git fetch 26 | git checkout --detach e758f9d45704ea0247c1b654f8602c967fa44199 27 | 28 | # Build libvpx 29 | ./configure --enable-pic --enable-experimental --enable-spatial-svc --enable-multi-res-encoding 30 | make -j32 31 | -------------------------------------------------------------------------------- /setup_aom.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2017 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | set -x 17 | 18 | # Download aom if not available. 19 | if [ ! -d aom ]; then 20 | git clone https://aomedia.googlesource.com/aom 21 | fi 22 | 23 | # Check out the pinned aom version. 24 | pushd aom 25 | git fetch 26 | git checkout --detach 09c0a5bcf57c58963516289ff2173576e3fa8378 27 | 28 | # Build aom 29 | ./configure --enable-pic 30 | make -j32 31 | -------------------------------------------------------------------------------- /setup_openh264.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2017 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | set -e 17 | set -x 18 | 19 | # Download libvpx if not available. 20 | if [ ! -d openh264 ]; then 21 | git clone https://github.com/cisco/openh264 22 | fi 23 | 24 | # Check out the pinned OpenH264 version. 25 | pushd openh264 26 | git fetch 27 | git checkout --detach 02218e2dbda258993cb91bc3468fe9ed508e16ca 28 | 29 | # Build OpenH264 30 | make -j32 31 | -------------------------------------------------------------------------------- /setup_vmaf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2017 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | set -x 17 | 18 | # Download vmaf if not available. 19 | if [ ! -d vmaf ]; then 20 | git clone https://github.com/Netflix/vmaf.git 21 | fi 22 | 23 | # Check out the pinned vmaf version. 24 | pushd vmaf 25 | git fetch 26 | git checkout --detach 45c57f7b67cebc301d85715669b9126063903ac2 27 | 28 | # Build vmaf 29 | make -j32 30 | -------------------------------------------------------------------------------- /setup_yami.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2016 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | # This script is based on the libyami build instructions at: 17 | # https://github.com/01org/libyami/wiki/Build 18 | 19 | export YAMI_ROOT_DIR="`pwd`/yami" 20 | export VAAPI_PREFIX="${YAMI_ROOT_DIR}/vaapi" 21 | export LIBYAMI_PREFIX="${YAMI_ROOT_DIR}/libyami" 22 | ADD_PKG_CONFIG_PATH="${VAAPI_PREFIX}/lib/pkgconfig/:${LIBYAMI_PREFIX}/lib/pkgconfig/" 23 | ADD_LD_LIBRARY_PATH="${VAAPI_PREFIX}/lib/:${LIBYAMI_PREFIX}/lib/" 24 | ADD_PATH="${VAAPI_PREFIX}/bin/:${LIBYAMI_PREFIX}/bin/" 25 | 26 | PLATFORM_ARCH_64=`uname -a | grep x86_64` 27 | if [ -n "$PKG_CONFIG_PATH" ]; then 28 | export PKG_CONFIG_PATH="${ADD_PKG_CONFIG_PATH}:$PKG_CONFIG_PATH" 29 | elif [ -n "$PLATFORM_ARCH_64" ]; then 30 | export PKG_CONFIG_PATH="${ADD_PKG_CONFIG_PATH}:/usr/lib/pkgconfig/:/usr/lib/i386-linux-gnu/pkgconfig/" 31 | else 32 | export PKG_CONFIG_PATH="${ADD_PKG_CONFIG_PATH}:/usr/lib/pkgconfig/:/usr/lib/x86_64-linux-gnu/pkgconfig/" 33 | fi 34 | 35 | export LD_LIBRARY_PATH="${ADD_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH" 36 | 37 | export PATH="${ADD_PATH}:$PATH" 38 | 39 | set -e 40 | set -x 41 | 42 | mkdir -p "$YAMI_ROOT_DIR/build" 43 | 44 | pushd "$YAMI_ROOT_DIR/build" 45 | 46 | if [ ! -d libva ]; then 47 | git clone https://github.com/01org/libva.git 48 | fi 49 | 50 | if [ ! -d intel-vaapi-driver ]; then 51 | git clone https://github.com/01org/intel-vaapi-driver.git 52 | fi 53 | 54 | if [ ! -d libyami ]; then 55 | git clone -b apache https://github.com/01org/libyami.git 56 | fi 57 | 58 | if [ ! -d libyami-utils ]; then 59 | git clone https://github.com/01org/libyami-utils.git 60 | fi 61 | 62 | pushd libva 63 | git fetch 64 | git checkout --detach c8d523bcc1e8cfbc432002908dc1e37de002ce78 65 | ./autogen.sh "--prefix=$VAAPI_PREFIX" && make -j32 && make install 66 | popd 67 | 68 | pushd intel-vaapi-driver 69 | git fetch 70 | git checkout --detach 8ccf612d70e333491b1f496ec8542582286a296c 71 | ./autogen.sh "--prefix=$VAAPI_PREFIX" && make -j32 && make install 72 | popd 73 | 74 | pushd libyami 75 | git fetch 76 | git checkout --detach 125f35d8412252aa67efcb7f13737746a1299f1e 77 | ./autogen.sh --enable-vp8enc --enable-vp9enc --disable-x11 "--prefix=$LIBYAMI_PREFIX" && make -j32 && make install 78 | popd 79 | 80 | pushd libyami-utils 81 | git fetch 82 | git checkout --detach 0b024ad25c8f9972dd8970642aa5c71fd70ad1c6 83 | ./autogen.sh --disable-v4l2 --disable-tests-gles --disable-md5 --disable-x11 "--prefix=$LIBYAMI_PREFIX" && make -j32 && make install 84 | popd 85 | --------------------------------------------------------------------------------