├── .gitignore ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── README.md ├── __init__.py ├── __main__.py ├── bits.py ├── cfgutil.py ├── docs └── transperf-screenshot.png ├── executor.py ├── gen.py ├── iface_cfg └── sample.py ├── js.py ├── launch.py ├── log.py ├── metric.py ├── nodeinit.py ├── orch.py ├── outparser.py ├── path.py ├── recv.py ├── regress.py ├── sample_config ├── 4bbr2_50M_40ms_BDP.py └── sample.py ├── send.py ├── shell.py ├── tcp.py ├── templates.py ├── third_party ├── dygraphs │ ├── LICENSE │ ├── dygraph.css │ ├── dygraph.min.js │ ├── shapes.js │ └── synchronizer.js └── jquery │ ├── LICENSE │ └── jquery-2.1.3.min.js ├── virtcleanup.py └── virtsetup.py /.gitignore: -------------------------------------------------------------------------------- 1 | ### Python ### 2 | # Binaries. 3 | /tc 4 | /netperf 5 | /netserver 6 | /nsenter 7 | /brctl 8 | 9 | # Python .zip files. 10 | transperf.zip 11 | 12 | # Generic generated files. 13 | Module.symvers 14 | 15 | # Results. 16 | LATEST 17 | 18 | # Byte-compiled / optimized / DLL files 19 | __pycache__/ 20 | *.py[cod] 21 | *$py.class 22 | 23 | # Environments 24 | .env 25 | .venv 26 | env/ 27 | venv/ 28 | ENV/ 29 | env.bak/ 30 | venv.bak/ 31 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 5 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 6 | 7 | ## [Unreleased] 8 | 9 | ## [0.1.0] - 2019-09-19 10 | Initial open-source release of transperf. 11 | 12 | [Unreleased]: https://github.com/google/transperf/compare/v0.1.0...master 13 | [0.1.0]: https://github.com/google/transperf/releases/tag/v0.1.0 14 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows 28 | [Google's Open Source Community Guidelines](https://opensource.google.com/conduct/). 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | SRC=$(wildcard *.py) 16 | 17 | # To create a Python-runnable zip file we need to add __main__.py in 18 | # the root and add all other python files inside the transperf folder 19 | # to preserve the package structure. For that, we need a structure like: 20 | # 21 | # transperf.zip / 22 | # __main__.py 23 | # transperf / 24 | # third_party / 25 | # launch.py 26 | # recv.py 27 | # ... 28 | # The following command first creates a zip file that has __main__.py 29 | # in its root, and then add all other python files inside transperf 30 | # directory. 31 | # 32 | # Please note that the zip commands, used here, preserve the contents 33 | # of the zipfile and only appends the new files. 34 | zip: $(SRC) 35 | rm -f transperf.zip 36 | zip -r transperf.zip __main__.py && \ 37 | cd ../ && \ 38 | zip -r transperf/transperf.zip transperf/*.py transperf/third_party 39 | 40 | clean: 41 | rm transperf.zip 42 | 43 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Transperf 2 | [![License](https://img.shields.io/github/license/google/transperf)](https://github.com/google/transperf/blob/master/LICENSE) 3 | [![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/google/transperf?sort=semver)](https://github.com/google/transperf/releases) 4 | 5 | Transperf is a tool for testing transport protocol performance. Its key features 6 | include: 7 | * Testing TCP performance over a variety of emulated network scenarios (using 8 | [netem](https://wiki.linuxfoundation.org/networking/netem)), including RTT, 9 | bottleneck bandwidth, and policed rate that can change over time. 10 | * Testing multiple flows, on multiple machines. 11 | * Testing multiple congestion control algorithms in a single test. 12 | * Using a Python-based language for specifying test configurations and making 13 | assertions about the expected results. 14 | * Visualizing results using a web GUI, along with archiving the configuration, 15 | logs, pcap traces, tcptrace xplot files, and outputs of a test. 16 | 17 | [![A screenshot of a sample run with transperf](docs/transperf-screenshot.png)](docs/transperf-screenshot.png) 18 | 19 | **Note that this is not an officially supported Google product**. 20 | 21 | 22 | ## CHANGELOG 23 | Please see the [CHANGELOG](CHANGELOG.md) for a release history. 24 | 25 | 26 | ## Documentation quick links 27 | * [Installation](#installation) 28 | * [User Guide](#user-guide) 29 | * [Testing Metrics](#testing-metrics) 30 | * [Examples](#examples) 31 | * [Frequently Asked Questions](#faqs) 32 | 33 | 34 | ## Installation 35 | Because transperf is launched from your local machine and run on a remote host, 36 | there are steps that need to be taken on both your local machine and the remote 37 | host. 38 | 39 | ### Setup Remote Hosts 40 | The following instructions are specifically for running transperf using [GCE 41 | VMs][GCE] using Debian 9, and may need to be adjusted depending on your testing 42 | environment. The number of instances you need to create depends on the types of 43 | tests you want to run. At minimum, you need two: one receiver and one sender 44 | instance. 45 | 46 | 1. For each instance, first ensure that each remote host is up-to-date and has 47 | these dependencies installed. 48 | 49 | On the remote hosts: 50 | 51 | ``` 52 | $ sudo apt update && sudo apt upgrade 53 | $ sudo apt install rsync psmisc ifenslave pkg-config bison flex git make \ 54 | bridge-utils util-linux tcpdump 55 | ``` 56 | 57 | Some distributions have an out-of-date version of iproute2, so you should 58 | install the latest version from source. 59 | 60 | ``` 61 | $ git clone git://git.kernel.org/pub/scm/network/iproute2/iproute2.git 62 | $ cd iproute2/ 63 | $ ./configure 64 | $ sudo make install 65 | ``` 66 | 67 | 2. Then, if you want to test bbrv2, follow the instructions on the [bbr 68 | repository](https://github.com/google/bbr/tree/v2alpha#obtaining-kernel-sources-with-tcp-bbr-v2-alpha) 69 | to install a kernel with bbrv2 on each of the remote hosts. 70 | 71 | 3. Configure rsyslog on remote hosts. By default, rsyslog on Google Cloud 72 | collects logs too aggressively, which causes issues in collecting kernel log 73 | output needed by transperf. On each remote machine, edit the following file 74 | 75 | ``` 76 | $ sudo vim /etc/rsyslog.d/90-google.conf 77 | ``` 78 | 79 | by commenting out the final line, as shown below. 80 | 81 | ```cfg 82 | # Google Compute Engine default console logging. 83 | # 84 | # daemon: logging from Google provided daemons. 85 | # kern: logging information in case of an unexpected crash during boot. 86 | # 87 | # daemon,kern.* /dev/console 88 | ``` 89 | 90 | Then, restart rsyslog: 91 | 92 | ``` 93 | $ sudo service rsyslog restart 94 | ``` 95 | 96 | 4. Enable root login. Transperf needs to execute as root on all remote 97 | machines, so root login must be enabled by editing the following file 98 | 99 | ``` 100 | $ sudo vim /etc/ssh/sshd_config 101 | ``` 102 | 103 | by finding the line containing `PermitRootLogin` and setting it to `yes`: 104 | 105 | ```cfg 106 | ... 107 | # Authentication: 108 | 109 | #LoginGraceTime 2m 110 | PermitRootLogin yes 111 | #StrictModes yes 112 | #MaxAuthTries 6 113 | #MaxSessions 10 114 | ... 115 | ``` 116 | 117 | Then, restart sshd: 118 | 119 | ``` 120 | $ sudo service sshd restart 121 | ``` 122 | 123 | Verify that this worked by ensuring you can ssh into the remote machine as 124 | root: 125 | 126 | ``` 127 | $ ssh root@$HOST 128 | ``` 129 | 130 | Note that you may need to [use gcloud to establish SSH keys 131 | once](https://cloud.google.com/compute/docs/instances/connecting-advanced#root). 132 | However, after they keys are exchanged, you should be able to ssh directly. 133 | 134 | 5. Setup a bonding device on the receiver. Whichever remote host you want to use 135 | as your receiver must be set up with a bonding device named `eth0`, and the 136 | physical NIC named `eth1`. To do this, rename the physical device from the 137 | default value (e.g. `eth0` or `ens4`) to `eth1` by adding the following line 138 | to the specified file. 139 | 140 | /etc/udev/rules.d/70-persistent-net.rules: 141 | ``` 142 | SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="", NAME="eth1" 143 | ``` 144 | 145 | Then, create a bonding device named `eth0`, and have `eth1` set as a slave. 146 | 147 | /etc/network/interfaces: 148 | ``` 149 | # interfaces(5) file used by ifup(8) and ifdown(8) 150 | 151 | auto eth0 152 | 153 | iface eth0 inet dhcp 154 | bond-mode active-backup 155 | slaves eth1 156 | ``` 157 | 158 | You will then need to reboot the VM for these changes to take effect. You 159 | can verify that it worked correctly by running 160 | 161 | ``` 162 | $ ip address 163 | ``` 164 | 165 | and checking that eth0 is a master device, and eth1 is a slave. 166 | 167 | On the senders, you will need to make sure the physical NIC is named `eth0`. 168 | If it is already named `eth0` (e.g., on Debian) you do not need to do 169 | anything. Senders do not need a bonding device. 170 | 171 | /etc/udev/rules.d/70-persistent-net.rules: 172 | ``` 173 | SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="", NAME="eth0" 174 | ``` 175 | 176 | 6. (If using GCE) Ensure that your firewall rules will not block tcp/udp 177 | traffic. Transperf uses ports 6200 and 6300 explicitly, and also uses 178 | randomly generated ports to transfer data between senders and receivers. 179 | Consequently, you need to make sure your [firewall rules in 180 | GCP](https://cloud.google.com/vpc/docs/firewalls) are set to allow tcp/udp 181 | traffic between your VMs. That is, if you run 182 | 183 | ``` 184 | gcloud compute firewall-rules list 185 | ``` 186 | 187 | you should see a line like 188 | 189 | ``` 190 | NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED 191 | ... 192 | default-xg4al7wofiqhpghuvl4aw6so default INGRESS 1000 tcp,udp False 193 | ... 194 | ``` 195 | 196 | if you have configured your firewall rules correctly. 197 | 198 | ### Setup local machine 199 | 200 | Your local machine must be the same architecture as your remote hosts (e.g., 201 | `x86_64`) because you will be compiling up-to-date binaries on your local 202 | machine, which will then be copied and run on the remote hosts. Your local 203 | machine is also responsible for processing all of the data retrieved from 204 | running the transperf experiment. 205 | 206 | 1. Install dependencies. 207 | 208 | ``` 209 | $ sudo apt update 210 | $ sudo apt install python-scapy wireshark tcptrace autoconf texinfo \ 211 | git build-essential rsync 212 | ``` 213 | 214 | 2. Build netperf from source to include burst and interval features. 215 | 216 | ``` 217 | $ git clone -b netperf-2.7.0 https://github.com/HewlettPackard/netperf 218 | $ cd netperf 219 | $ ./autogen.sh 220 | $ ./configure --enable-burst --enable-intervals 221 | $ sudo make install 222 | ``` 223 | 224 | 225 | ## User Guide 226 | ### Build 227 | Unlike a [typical Python 228 | package](https://packaging.python.org/tutorials/packaging-projects/), transperf 229 | is run [directly as a ZIP file](https://bugs.python.org/issue1739468). To 230 | prepare the ZIP from the root of the repository, just run: 231 | 232 | ``` 233 | $ make 234 | ``` 235 | 236 | ### Quick Start: Multi-server Mode 237 | Most transperf experiments are run in multi-server mode. To run all the test 238 | configurations in the `sample_config` directory on `host1`, `host2`, and `host3` 239 | by launching from your local machine, simply run: 240 | 241 | ``` 242 | $ python transperf.zip -v -x -c sample_config root@host1 root@host2 root@host3 243 | ``` 244 | 245 | If you're using [GCE VMs][GCE], you will need to provide both their external IPs 246 | and internal IPs, like this: 247 | 248 | ``` 249 | $ python transperf.zip -v -x -c sample_config root@ext_ip1/int_ip1 root@ext_ip2/int_ip2 root@ext_ip3/int_ip3 250 | ``` 251 | 252 | Note that the first host provided as a command line argument is used as the 253 | receiver. 254 | 255 | ### Quick Start: Single-server Mode 256 | Transperf also has basic support for running tests using a single remote 257 | machine, rather than the normal workflow of requiring >= 2 remote machines. This 258 | is accomplished using Linux network containers (via 259 | [netns](https://lwn.net/Articles/580893/)). Note that TCP parameters are not 260 | virtualized. Consequently, tests in single-server mode are limited to using only 261 | a single set of parameters per congestion control algorithm. 262 | 263 | To run a test configuration on the remote `host1`: 264 | 265 | ``` 266 | python transperf.zip -v -x -c sample_config/4bbr_50M_40ms_BDP.py --virtcleanup --ssrv host1 root@rcv root@snd1 root@snd2 267 | ``` 268 | 269 | ## Testing Metrics 270 | Transperf includes the ability to set expectations for a variety of metrics and 271 | check the results of an experimental run against those expectations. This allows 272 | transperf to be used for high-level testing of transport protocol behavior. 273 | 274 | These checks can be customized by including a dictionary of values, as shown 275 | below. 276 | 277 | ```py 278 | scores = { 279 | "tput": 90, # use X% of total bw as goodput 280 | "rtt_med": 25, # 100 * minRTT/rtt_i 281 | "lock_on_bw": 10, # should exit STARTUP at bw at least x% of bottleneck 282 | "loss_avoid": 95, # non-loss percentage 283 | "fairness": 75, # jain fairness index * 100 over lifetime 284 | "convergence_fairness": 85, # jain fairness for each second after conv 285 | "convergence_sec": 1.1, # how long before asserting convergence 286 | } 287 | check = check_with_scores(scores) 288 | ``` 289 | 290 | Each of these values corresponds to one of the following metrics. 291 | 292 | #### Goodput (`tput`) 293 | Throughput represents the bits transmitted per second measured during the test 294 | for all the connections, while goodput represents transmission excluding 295 | retransmission and header overhead. The `tput` metric reports the total goodput. 296 | We normalize `tput` into range [0,100] by dividing the total goodput, `b`, by 297 | the bottleneck bandwidth, `BW`, and multiplying by 100. 298 | 299 | ``` 300 | tput = 100 * b / BW 301 | ``` 302 | 303 | #### RTT Reduction (`rtt_med`) 304 | Round-trip time (RTT) represents the two-way latency measured by the congestion 305 | control module. To assign a score that is higher with lower RTT, we use the term 306 | *RTT reduction*. We normalize the RTT reduction as follows: 307 | 308 | ``` 309 | rtt_med = 100 * minRTT_i / medRTT_i 310 | ``` 311 | 312 | where minRTT and medRTT are the minimum and median measured RTTs for a flow, 313 | `i`. 314 | 315 | #### Lock on Bandwidth (`lock_on_bw`) 316 | This metric represents the ratio of the bandwidth at which BBR exists STARTUP. 317 | That is, it checks that 318 | 319 | ``` 320 | measured_lock_on_bw_i >= lock_on_target 321 | ``` 322 | 323 | where 324 | 325 | ``` 326 | lock_on_target = (lock_on_bw / 100) * btl_bw / num_conns 327 | ``` 328 | 329 | and subscript `i` represents that this condition is checked for each flow `i`. 330 | 331 | #### Loss Avoidance (`loss_avoid`) 332 | Given a loss rate, `l`, ranging from 0 to 1, the normalized loss avoidance score 333 | is the non-loss percent: 334 | 335 | ``` 336 | loss_avoid = 100 * (1 - l) 337 | ``` 338 | 339 | This gives lower loss rate a higher score. 340 | 341 | #### Fairness (`fairness`) 342 | We use [Jain's fairness index](https://en.wikipedia.org/wiki/Fairness_measure) 343 | to represent how fair the goodput is allocated between connections during the 344 | test. We adjust the Jain's fairness index from (0,1] to (0,100] as follows. 345 | 346 | ``` 347 | f(b_1, b_2, ..., b_n) = 100 * (sum(b_i))^2 / (n * sum(b_i^2)) 348 | ``` 349 | 350 | where `n` is the number of connections and `b_i` is the goodput for connection 351 | `i`. 352 | 353 | #### Convergence Fairness (`convergence_fairness`) 354 | Unlike Fairness, which is the pass/fail threshold for the Jain fairness index 355 | for each flow over its entire lifetime, convergence fairness is the minimum 356 | Jain's fairness index that the flow throughputs must have in *every* one-second 357 | time buck after `t = convergence_sec`. 358 | 359 | 360 | ## Examples 361 | Example configurations for transperf tests can be found in 362 | [`sample_config`](sample_config). 363 | 364 | [GCE]: https://cloud.google.com/compute/ 365 | 366 | 367 | ## FAQs 368 | **I keep getting Permission denied (publickey) even though PermitRootLogin is 369 | yes** 370 | 371 | Try running ssh with verbose output (`ssh -v`). If you haven't connected to root 372 | using gcloud, do that once so that SSH keys can be created and transferred. 373 | 374 | You may need to specify the identity file created by gcloud (e.g. 375 | `~/.ssh/google_compute_engine`) on your local machine. We recommend just adding 376 | that key using `ssh-agent`: 377 | 378 | ``` 379 | $ eval `ssh-agent` 380 | $ ssh-add ~/.ssh/google_compute_engine 381 | ``` 382 | -------------------------------------------------------------------------------- /__main__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Main entry file for transperf.""" 16 | 17 | import sys 18 | 19 | from transperf import launch 20 | 21 | if __name__ == '__main__': 22 | sys.exit(launch.main()) 23 | -------------------------------------------------------------------------------- /bits.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """A few bit hacks (from https://graphics.stanford.edu/~seander/bithacks.html). 16 | """ 17 | 18 | def next_power_of_two(i): 19 | """Returns the next highest power of two. 20 | 21 | Args: 22 | i: Is the number and should be a 32 bit integer. 23 | 24 | Returns: 25 | The next highest power of two. 26 | """ 27 | i -= 1 28 | i |= i >> 1 29 | i |= i >> 2 30 | i |= i >> 4 31 | i |= i >> 8 32 | i |= i >> 16 33 | i += 1 34 | return i 35 | 36 | 37 | def num_bits(i): 38 | """Returns the number of bits in an unsigned integer.""" 39 | n = 0 40 | while i: 41 | n += 1 42 | i &= i - 1 43 | return n 44 | 45 | 46 | def trailing_zero_bits(i): 47 | """Returns the number of trailing zero bits.""" 48 | n = 0 49 | while not i & 0x1: 50 | i >>= 1 51 | n += 1 52 | return n 53 | 54 | -------------------------------------------------------------------------------- /cfgutil.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Utilities to parse configuration files.""" 16 | 17 | import ast 18 | import os 19 | 20 | import transperf 21 | 22 | 23 | def config_from_script(script): 24 | """Creates a transperf.Config object from the script. 25 | 26 | Args: 27 | script: The content of the configuration script. 28 | 29 | Returns: 30 | A new transperf.Config object. 31 | """ 32 | ns = {} 33 | code = compile('from transperf import *', '', 'exec') 34 | exec code in ns 35 | # These are the symbols that transperf creates. 36 | transperf_ns = dict(ns) 37 | 38 | # Find the order in which the parameters are assigned. 39 | mod = ast.parse(script) 40 | assign_order = [] 41 | for node in mod.body: 42 | if not isinstance(node, ast.Assign): 43 | continue 44 | for t in node.targets: 45 | if isinstance(t, ast.Tuple): 46 | for e in t.elts: 47 | if isinstance(e, ast.Name): 48 | assign_order.append(node.id) 49 | elif isinstance(t, ast.Name): 50 | assign_order.append(t.id) 51 | 52 | code = compile(script, '