├── .gitmodules ├── LICENSE ├── README.md ├── README_old.md ├── extern └── instrumentations │ └── client.c ├── kits ├── mvfst │ ├── README.md │ ├── client.sh │ ├── config.sh │ ├── multi_client.sh │ ├── server.sh │ └── tperf.cpp ├── picoquic │ ├── README.md │ ├── client.sh │ ├── config.sh │ ├── multi_client.sh │ ├── picoquicdemo.c │ └── server.sh ├── quant │ ├── README.md │ ├── client.c │ ├── client_netmap.sh │ ├── config.sh │ └── server_netmap.sh └── quicly │ ├── README.md │ ├── cli.c │ ├── client.sh │ ├── config.sh │ ├── multi_clients.sh │ └── server.sh └── unfinished ├── data_process.py └── main.py /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "extern"] 2 | path = extern 3 | url = https://github.com/NTAP/quant.git 4 | [submodule "extern/quant"] 5 | path = extern/quant 6 | url = https://github.com/NTAP/quant.git 7 | [submodule "extern/picoquic"] 8 | path = extern/picoquic 9 | url = https://github.com/private-octopus/picoquic.git 10 | [submodule "extern/quicly"] 11 | path = extern/quicly 12 | url = https://github.com/h2o/quicly.git 13 | [submodule "extern/mvfst"] 14 | path = extern/mvfst 15 | url = https://github.com/facebookincubator/mvfst.git 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Winters123 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # QUIC-measurement-kit 2 | This repo is the tools & scripts used for conducting the QUIC measurement described in [1] and the generated results. 3 | 4 | ### What are these files? 5 | 6 | Being said that these are a bunch of toolkits for protocol's implementation performance profiling, its hard to make them well-organized in a repo based on the fact that we need to adapt them for various opensource implementations. However, I did try the best to make them easy to read and reuse. 7 | 8 | In general, the subdir `kits` contains all the scripts used for the measurements. 9 | Each implementation subdir contains **4** files and users need to know: 10 | 1. `server.sh`: to fire up the QUIC server; 11 | 2. `client.sh`: to fire up a single connection client and request a fixed amount of data block from the server; 12 | 3. `multi_client.sh`: to fire up multiple connections as clients and request fixed amount of data blocks from the server; 13 | 4. `config.sh` to move the instrumented source files into the orignal project. 14 | 15 | 16 | ### How to run them & collect data from them? 17 | 18 | Users need to follow the `README.md` in each subdir to conduct the measurements. 19 | Modifications should be made on those scripts (follow the inside comments) to enable/disable `perf`. 20 | To better visualize the `perf` result, [flamegraph tools](https://github.com/brendangregg/FlameGraph) are highly recommended for analyses & calculation. 21 | 22 | For throughput profiling, after the file is generated, users can use `auto_proc.py` to analyse the result (sum, average, etc.). Again, customizing the scripts based on your own needs is high recommended. 23 | 24 | ### How do they work? 25 | 26 | For CPU profiling, `perf` is the core tool that is called in the scripts. As for throughput, the process is a bit more complex: instrumentations need to be added in the source-code to obtain the real-time throughput. This is the main reason why this kit only support the QUIC implementations I adapted. 27 | 28 | ### How to contribute to them? 29 | 30 | Any helpful contribution to the scripts is welcome: 31 | 32 | 1. Optimize the scripts for better formed results. 33 | 2. Change the ways to obtain the throughput so that we don't have to add instrumentations every time. 34 | 3. create a high-level script to run all the tasks (CPU profiling, throughput test, etc.) automatically. 35 | 4. support the test for latency. 36 | 37 | ### Reference 38 | 39 | 1. Dissecting QUIC Implementation Performance, Xiangrui et al. ACM EuroSys2020 Poster, Greece. 40 | 2. Making QUIC quicker with NIC offload, Xiangrui et al. ACM SIGCOMM EPIQ, USA. 41 | -------------------------------------------------------------------------------- /README_old.md: -------------------------------------------------------------------------------- 1 | # QUIC-measurement-kit 2 | This repo is the tools & scripts used for conducting the QUIC measurement described in [1] and the generated results. 3 | 4 | ### What are these files? 5 | 6 | Being said that these are a bunch of toolkits for protocol's implementation performance profiling, its hard to make them well-organized in a repo based on the fact that we need to adapt them for various opensource implementations. However, I did try the best to make them easy to read and reuse. 7 | 8 | In general, this repo contains 2 branches: `single_conn` and `multi_conn` and a adapted `TLEM` toolkit. `single_conn` is responsible for measurement based on a single QUIC connection, similar speak to `multi_conn`. The adapted `TLEM` is based on the built-in network simulator `TLEM` in `netmap`. I modified its source code to be able to introduce network interference only to specific packets (e.g., QUIC ACKs). 9 | 10 | In each branch, there are 2 sub-branches: `CPU profileing` and `throughput profiling`. In each of the branch, there is a `main.py` script which is able to conduct the measurement automatically. In order to successfully launch the measurement, some parameters are needed. use `main.py -help` for the details of these parameters. Bare in mind that these script now can only be used for the adapted `quant`, `picoquic`, `quicly` and `mvfst`. In order to use these scripts for other QUIC implementations, modifications of the source code are needed. It is highly recommended users modify these scripts based on their requirements. 11 | 12 | ### How to run them & collect data from them? 13 | 14 | Users should first copy the files in each branch to the root directory of the implementation they want to measure. Then, run the scripts `main.py` (with **correct parameters**) to launch the measurement. For CPU profiling, the results are stores as `server.perf` and `client.perf`. To better visualize the result, [flamegraph tools](https://github.com/brendangregg/FlameGraph) are highly recommended for analyses & calculation. 15 | 16 | For throughput profiling, the real-time throughput results are stored in `throughput.txt`. After the file is generated, users can use `auto_proc.py` to analyse the result (sum, average, etc.). Again, customizing the scripts based on your own needs is high recommended. 17 | 18 | ### How do they work? 19 | 20 | For CPU profiling, `perf` is the core tool that is called in the scripts. As for throughput, the process is a bit more complex: instrumentations need to be added in the source-code to obtain the real-time throughput. This is the main reason why this kit only support the QUIC implementations I adapted. 21 | 22 | ### How to contribute to them? 23 | 24 | Any helpful contribution to the scripts is welcome: 25 | 26 | 1. Optimize the scripts for better formed results. 27 | 2. Change the ways to obtain the throughput so that we don't have to add instrumentations every time. 28 | 3. create a high-level script to run all the tasks (CPU profiling, throughput test, etc.) automatically. 29 | 4. support the test for latency. 30 | 31 | ### Reference 32 | 33 | 1. Dissecting QUIC Implementation Performance, Xiangrui et al. EuroSys2020 Poster, Greece. -------------------------------------------------------------------------------- /extern/instrumentations/client.c: -------------------------------------------------------------------------------- 1 | // SPDX-License-Identifier: BSD-2-Clause 2 | // 3 | // Copyright (c) 2016-2020, NetApp, Inc. 4 | // All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are met: 8 | // 9 | // 1. Redistributions of source code must retain the above copyright notice, 10 | // this list of conditions and the following disclaimer. 11 | // 12 | // 2. Redistributions in binary form must reproduce the above copyright notice, 13 | // this list of conditions and the following disclaimer in the documentation 14 | // and/or other materials provided with the distribution. 15 | // 16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 19 | // ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 20 | // LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 21 | // CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 22 | // SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 23 | // INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 24 | // CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 25 | // ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 26 | // POSSIBILITY OF SUCH DAMAGE. 27 | 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include 40 | #include 41 | #include // IWYU pragma: no_forward_declare sockaddr 42 | #include 43 | #include 44 | #include 45 | #include 46 | #include 47 | 48 | #ifdef __FreeBSD__ 49 | #include 50 | #endif 51 | 52 | #define klib_unused 53 | 54 | #include 55 | #include 56 | #include 57 | 58 | 59 | #define timespec_to_double(diff) \ 60 | ((double)(diff).tv_sec + (double)(diff).tv_nsec / NS_PER_S) 61 | 62 | 63 | #define bps(bytes, secs) \ 64 | __extension__({ \ 65 | static char _str[32]; \ 66 | const double _bps = ((secs) > 1e-9) ? (double)(bytes)*8 / (secs) : 0; \ 67 | if (_bps > NS_PER_S) \ 68 | snprintf(_str, sizeof(_str), "%.3f Gb/s", _bps / NS_PER_S); \ 69 | else if (_bps > US_PER_S) \ 70 | snprintf(_str, sizeof(_str), "%.3f Mb/s", _bps / US_PER_S); \ 71 | else if (_bps > MS_PER_S) \ 72 | snprintf(_str, sizeof(_str), "%.3f Kb/s", _bps / MS_PER_S); \ 73 | else \ 74 | snprintf(_str, sizeof(_str), "%.3f b/s", _bps); \ 75 | _str; \ 76 | }) 77 | 78 | #define ins_bps(bytes, secs) \ 79 | __extension__({ \ 80 | static char _str[32]; \ 81 | const double _bps = ((secs) > 1e-9) ? (double)(bytes)*8 / (secs) : 0; \ 82 | snprintf(_str, sizeof(_str), "%.3f", _bps / US_PER_S); \ 83 | _str; \ 84 | }) 85 | 86 | 87 | struct conn_cache_entry { 88 | struct q_conn * c; 89 | struct addrinfo * migr_peer; 90 | bool migrated; 91 | uint8_t _unused[7]; 92 | }; 93 | 94 | 95 | KHASH_MAP_INIT_INT64(conn_cache, struct conn_cache_entry *) 96 | 97 | 98 | static uint32_t vers = 0; 99 | static uint32_t timeout = 10; 100 | static uint32_t num_bufs = 100000; 101 | static uint32_t reps = 1; 102 | static bool do_h3 = false; 103 | static bool do_v6 = false; 104 | static bool do_chacha = false; 105 | static bool flip_keys = false; 106 | static bool zlen_cids = false; 107 | static bool write_files = false; 108 | static bool test_qr = false; 109 | #ifndef NO_MIGRATION 110 | static bool rebind = false; 111 | static bool switch_ip = false; 112 | #endif 113 | 114 | 115 | struct stream_entry { 116 | sl_entry(stream_entry) next; 117 | struct conn_cache_entry * cce; 118 | struct q_stream * s; 119 | char * url; 120 | struct timespec req_t; 121 | struct timespec rep_t; 122 | struct w_iov_sq req; 123 | struct w_iov_sq rep; 124 | }; 125 | 126 | 127 | static sl_head(stream_list, stream_entry) sl = sl_head_initializer(sl); 128 | 129 | 130 | static inline uint64_t __attribute__((nonnull)) 131 | conn_cache_key(const struct sockaddr * const sock) 132 | { 133 | const struct sockaddr_in * const sock4 = 134 | (const struct sockaddr_in *)(const void *)sock; 135 | 136 | return ((uint64_t)sock4->sin_addr.s_addr 137 | << sizeof(sock4->sin_addr.s_addr) * 8) | 138 | (uint64_t)sock4->sin_port; 139 | } 140 | 141 | 142 | static void __attribute__((noreturn, nonnull)) 143 | usage(const char * const name, 144 | const char * const ifname, 145 | const char * const cache, 146 | const char * const tls_log, 147 | const char * const qlog_dir, 148 | const bool verify_certs) 149 | { 150 | printf("%s [options] URL [URL...]\n", name); 151 | printf("\t[-3]\t\tsend a static H3 request; default %s\n", 152 | do_h3 ? "true" : "false"); 153 | printf("\t[-6]\t\tuse IPv6; default %s\n", do_v6 ? "true" : "false"); 154 | printf("\t[-a]\t\tforce Chacha20; default %s\n", 155 | do_chacha ? "true" : "false"); 156 | printf("\t[-b bufs]\tnumber of network buffers to allocate; default %u\n", 157 | num_bufs); 158 | printf("\t[-c]\t\tverify TLS certificates; default %s\n", 159 | verify_certs ? "true" : "false"); 160 | printf("\t[-e version]\tQUIC version to use; default 0x%08x\n", vers); 161 | printf("\t[-i interface]\tinterface to run over; default %s\n", ifname); 162 | printf("\t[-l log]\tlog file for TLS keys; default %s\n", 163 | *tls_log ? tls_log : "false"); 164 | printf("\t[-m]\t\ttest multi-pkt initial (\"quantum-readiness\"); default " 165 | "%s\n", 166 | test_qr ? "true" : "false"); 167 | #ifndef NO_MIGRATION 168 | printf("\t[-n]\t\tsimulate NAT rebind (use twice for \"real\" migration); " 169 | "default %s\n", 170 | rebind ? "true" : "false"); 171 | #endif 172 | printf("\t[-q log]\twrite qlog events to directory; default %s\n", 173 | *qlog_dir ? qlog_dir : "false"); 174 | printf("\t[-r reps]\trepetitions for all URLs; default %u\n", reps); 175 | printf("\t[-s cache]\tTLS 0-RTT state cache; default %s\n", cache); 176 | printf("\t[-t timeout]\tidle timeout in seconds; default %u\n", timeout); 177 | printf("\t[-u]\t\tupdate TLS keys; default %s\n", 178 | flip_keys ? "true" : "false"); 179 | #ifndef NDEBUG 180 | printf("\t[-v verbosity]\tverbosity level (0-%d, default %d)\n", DLEVEL, 181 | util_dlevel); 182 | #endif 183 | printf("\t[-w]\t\twrite retrieved objects to disk; default %s\n", 184 | write_files ? "true" : "false"); 185 | printf("\t[-z]\t\tuse zero-length source connection IDs; default %s\n", 186 | zlen_cids ? "true" : "false"); 187 | exit(0); 188 | } 189 | 190 | 191 | static void __attribute__((nonnull)) 192 | set_from_url(char * const var, 193 | const size_t len, 194 | const char * const url, 195 | const struct http_parser_url * const u, 196 | const enum http_parser_url_fields f, 197 | const char * const def) 198 | { 199 | if ((u->field_set & (1 << f)) == 0) { 200 | strncpy(var, def, len); 201 | var[len - 1] = 0; 202 | } else { 203 | const uint16_t l = f == UF_PATH 204 | ? (uint16_t)strlen(url) - u->field_data[f].off 205 | : u->field_data[f].len; 206 | strncpy(var, &url[u->field_data[f].off], l); 207 | var[l] = 0; 208 | } 209 | } 210 | 211 | 212 | #ifndef NO_MIGRATION 213 | static void __attribute__((nonnull(1))) 214 | try_migrate(struct conn_cache_entry * const cce) 215 | { 216 | if (rebind && cce->migrated == false) 217 | cce->migrated = q_migrate(cce->c, switch_ip, 218 | cce->migr_peer ? cce->migr_peer->ai_addr : 0); 219 | } 220 | #else 221 | #define try_migrate(...) 222 | #endif 223 | 224 | 225 | static struct addrinfo * __attribute__((nonnull)) 226 | get_addr(const int af, const char * const dest, const char * const port) 227 | { 228 | struct addrinfo * peer = 0; 229 | const int err = getaddrinfo( 230 | dest, port, &(const struct addrinfo){.ai_family = af}, &peer); 231 | if (err == 0) 232 | return peer; 233 | 234 | if (err != EAI_FAMILY && err != EAI_NONAME) 235 | // when looking up migr_peer, these errors are OK to occur 236 | warn(ERR, "getaddrinfo: %s", gai_strerror(err)); 237 | if (peer) 238 | freeaddrinfo(peer); 239 | return 0; 240 | } 241 | 242 | 243 | static struct q_conn * __attribute__((nonnull)) 244 | get(char * const url, struct w_engine * const w, khash_t(conn_cache) * cc) 245 | { 246 | // parse and verify the URIs passed on the command line 247 | struct http_parser_url u = {0}; 248 | if (http_parser_parse_url(url, strlen(url), 0, &u)) { 249 | warn(ERR, "http_parser_parse_url: %s", 250 | http_errno_description((enum http_errno)errno)); 251 | return 0; 252 | } 253 | 254 | ensure((u.field_set & (1 << UF_USERINFO)) == 0, 255 | "userinfo unsupported in URL"); 256 | 257 | // extract relevant info from URL 258 | char dest[1024]; 259 | char port[64]; 260 | char path[8192]; 261 | set_from_url(dest, sizeof(dest), url, &u, UF_HOST, "localhost"); 262 | set_from_url(port, sizeof(port), url, &u, UF_PORT, "4433"); 263 | set_from_url(path, sizeof(path), url, &u, UF_PATH, "/index.html"); 264 | 265 | struct addrinfo * const peer = 266 | get_addr(do_v6 ? AF_INET6 : AF_INET, dest, port); 267 | struct addrinfo * const migr_peer = 268 | get_addr(do_v6 ? AF_INET : AF_INET6, dest, port); 269 | if (peer == 0) 270 | goto fail; 271 | 272 | // do we have a connection open to this peer? 273 | khiter_t k = kh_get(conn_cache, cc, conn_cache_key(peer->ai_addr)); 274 | struct conn_cache_entry * cce = (k == kh_end(cc) ? 0 : kh_val(cc, k)); 275 | 276 | // add to stream list 277 | struct stream_entry * se = calloc(1, sizeof(*se)); 278 | ensure(se, "calloc failed"); 279 | sq_init(&se->rep); 280 | sl_insert_head(&sl, se, next); 281 | 282 | sq_init(&se->req); 283 | if (do_h3) { 284 | q_alloc(w, &se->req, 0, peer->ai_family, 1024); 285 | struct w_iov * const v = sq_first(&se->req); 286 | const uint16_t len = 287 | (uint16_t)(h3zero_create_request_header_frame( 288 | &v->buf[3], v->buf + v->len - 3, (uint8_t *)path, 289 | strlen(path), dest) - 290 | &v->buf[3]); 291 | 292 | v->buf[0] = h3zero_frame_header; 293 | if (len < 64) { 294 | v->buf[1] = (uint8_t)len; 295 | memmove(&v->buf[2], &v->buf[3], len); 296 | v->len = 2 + len; 297 | } else { 298 | v->buf[1] = (uint8_t)((len >> 8) | 0x40); 299 | v->buf[2] = (uint8_t)(len & 0xff); 300 | v->len = 3 + len; 301 | } 302 | 303 | } else { 304 | // assemble an HTTP/0.9 request 305 | char req_str[sizeof(path) + 6]; 306 | const int req_str_len = 307 | snprintf(req_str, sizeof(req_str), "GET %s\r\n", path); 308 | q_chunk_str(w, cce ? cce->c : 0, peer->ai_family, req_str, 309 | (uint32_t)req_str_len, &se->req); 310 | } 311 | 312 | const bool opened_new = cce == 0; 313 | if (cce == 0) { 314 | clock_gettime(CLOCK_MONOTONIC, &se->req_t); 315 | // no, open a new connection 316 | struct q_conn * const c = q_connect( 317 | w, peer->ai_addr, dest, &se->req, &se->s, true, 318 | do_h3 ? "h3-" DRAFT_VERSION_STRING : "hq-" DRAFT_VERSION_STRING, 0); 319 | if (c == 0) 320 | goto fail; 321 | 322 | if (do_h3) { 323 | // we need to open a uni stream for an empty H/3 SETTINGS frame 324 | struct q_stream * const ss = q_rsv_stream(c, false); 325 | if (ss == 0) 326 | return 0; 327 | static const uint8_t h3_empty_settings[] = { 328 | 0x00, h3zero_frame_settings, 0x00}; 329 | // XXX lsquic doesn't like a FIN on this stream 330 | q_write_str(w, ss, (const char *)h3_empty_settings, 331 | sizeof(h3_empty_settings), false); 332 | } 333 | 334 | cce = calloc(1, sizeof(*cce)); 335 | ensure(cce, "calloc failed"); 336 | cce->c = c; 337 | cce->migr_peer = migr_peer; 338 | 339 | // insert into connection cache 340 | int ret; 341 | k = kh_put(conn_cache, cc, conn_cache_key(peer->ai_addr), &ret); 342 | ensure(ret >= 1, "inserted returned %d", ret); 343 | kh_val(cc, k) = cce; 344 | } 345 | 346 | if (opened_new == false) { 347 | se->s = q_rsv_stream(cce->c, true); 348 | if (se->s) { 349 | clock_gettime(CLOCK_MONOTONIC, &se->req_t); 350 | q_write(se->s, &se->req, true); 351 | } 352 | } 353 | try_migrate(cce); 354 | 355 | se->cce = cce; 356 | se->url = url; 357 | freeaddrinfo(peer); 358 | return cce->c; 359 | 360 | fail: 361 | freeaddrinfo(peer); 362 | freeaddrinfo(migr_peer); 363 | return 0; 364 | } 365 | 366 | 367 | static void __attribute__((nonnull)) free_cc(khash_t(conn_cache) * cc) 368 | { 369 | struct conn_cache_entry * cce; 370 | kh_foreach_value(cc, cce, { 371 | freeaddrinfo(cce->migr_peer); 372 | free(cce); 373 | }); 374 | kh_release(conn_cache, cc); 375 | } 376 | 377 | 378 | static void free_se(struct stream_entry * const se) 379 | { 380 | q_free(&se->req); 381 | q_free(&se->rep); 382 | free(se); 383 | } 384 | 385 | 386 | static void free_sl_head(void) 387 | { 388 | struct stream_entry * const se = sl_first(&sl); 389 | sl_remove_head(&sl, next); 390 | free_se(se); 391 | } 392 | 393 | 394 | static void free_sl(void) 395 | { 396 | while (sl_empty(&sl) == false) 397 | free_sl_head(); 398 | } 399 | 400 | 401 | static void __attribute__((nonnull)) 402 | write_object(struct stream_entry * const se) 403 | { 404 | char * const slash = strrchr(se->url, '/'); 405 | if (slash && *(slash + 1) == 0) 406 | // this URL ends in a slash, so strip that to name the file 407 | *slash = 0; 408 | 409 | const int fd = 410 | open(*basename(se->url) == 0 ? "index.html" : basename(se->url), 411 | O_CREAT | O_WRONLY | O_CLOEXEC, 412 | S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH); 413 | ensure(fd != -1, "cannot open %s", basename(se->url)); 414 | 415 | struct iovec vec[IOV_MAX]; 416 | struct w_iov * v = sq_first(&se->rep); 417 | int i = 0; 418 | while (v) { 419 | vec[i].iov_base = v->buf; 420 | vec[i].iov_len = v->len; 421 | if (++i == IOV_MAX || sq_next(v, next) == 0) { 422 | ensure(writev(fd, vec, i) != -1, "cannot writev"); 423 | i = 0; 424 | } 425 | v = sq_next(v, next); 426 | } 427 | close(fd); 428 | } 429 | 430 | 431 | int main(int argc, char * argv[]) 432 | { 433 | #ifndef NDEBUG 434 | util_dlevel = DLEVEL; // default to maximum compiled-in verbosity 435 | #endif 436 | char ifname[IFNAMSIZ] = "lo" 437 | #ifndef __linux__ 438 | "0" 439 | #endif 440 | ; 441 | int ch; 442 | char cache[MAXPATHLEN] = "/tmp/" QUANT "-session"; 443 | char tls_log[MAXPATHLEN] = ""; 444 | char qlog_dir[MAXPATHLEN] = ""; 445 | bool verify_certs = false; 446 | int ret = -1; 447 | 448 | // set default TLS log file from environment 449 | const char * const keylog = getenv("SSLKEYLOGFILE"); 450 | if (keylog) { 451 | strncpy(tls_log, keylog, MAXPATHLEN); 452 | tls_log[MAXPATHLEN - 1] = 0; 453 | } 454 | 455 | while ((ch = getopt(argc, argv, 456 | "hi:v:s:t:l:cu36azb:wr:q:me:" 457 | #ifndef NO_MIGRATION 458 | "n" 459 | #endif 460 | )) != -1) { 461 | switch (ch) { 462 | case 'i': 463 | strncpy(ifname, optarg, sizeof(ifname) - 1); 464 | break; 465 | case 's': 466 | strncpy(cache, optarg, sizeof(cache) - 1); 467 | break; 468 | case 'q': 469 | strncpy(qlog_dir, optarg, sizeof(qlog_dir) - 1); 470 | break; 471 | case 't': 472 | timeout = (uint32_t)MIN(600, strtoul(optarg, 0, 10)); // 10 min 473 | break; 474 | case 'b': 475 | num_bufs = (uint32_t)MIN(strtoul(optarg, 0, 10), UINT32_MAX); 476 | break; 477 | case 'r': 478 | reps = (uint32_t)MAX(1, MIN(strtoul(optarg, 0, 10), UINT32_MAX)); 479 | break; 480 | case 'e': 481 | vers = (uint32_t)MAX(1, MIN(strtoul(optarg, 0, 16), UINT32_MAX)); 482 | break; 483 | case 'l': 484 | strncpy(tls_log, optarg, sizeof(tls_log) - 1); 485 | break; 486 | case 'c': 487 | verify_certs = true; 488 | break; 489 | case 'u': 490 | flip_keys = true; 491 | break; 492 | case '3': 493 | do_h3 = true; 494 | break; 495 | case '6': 496 | do_v6 = true; 497 | break; 498 | case 'a': 499 | do_chacha = true; 500 | break; 501 | case 'z': 502 | zlen_cids = true; 503 | break; 504 | case 'w': 505 | write_files = true; 506 | break; 507 | case 'm': 508 | test_qr = true; 509 | break; 510 | #ifndef NO_MIGRATION 511 | case 'n': 512 | if (rebind) 513 | switch_ip = true; 514 | rebind = true; 515 | break; 516 | #endif 517 | case 'v': 518 | #ifndef NDEBUG 519 | util_dlevel = (short)MIN(DLEVEL, strtoul(optarg, 0, 10)); 520 | #endif 521 | break; 522 | case 'h': 523 | case '?': 524 | default: 525 | usage(basename(argv[0]), ifname, cache, tls_log, qlog_dir, 526 | verify_certs); 527 | } 528 | } 529 | 530 | struct w_engine * const w = q_init( 531 | ifname, 532 | &(const struct q_conf){ 533 | .conn_conf = 534 | &(struct q_conn_conf){.enable_tls_key_updates = flip_keys, 535 | .enable_spinbit = true, 536 | .enable_udp_zero_checksums = true, 537 | .idle_timeout = timeout, 538 | .version = vers, 539 | .enable_quantum_readiness_test = test_qr}, 540 | .qlog_dir = *qlog_dir ? qlog_dir : 0, 541 | .force_chacha20 = do_chacha, 542 | .num_bufs = num_bufs, 543 | .ticket_store = cache, 544 | .tls_log = *tls_log ? tls_log : 0, 545 | .client_cid_len = zlen_cids ? 0 : 4, 546 | .enable_tls_cert_verify = verify_certs}); 547 | khash_t(conn_cache) cc = {0}; 548 | FILE *ins_fd; 549 | ins_fd = fopen("bps_ins.txt", "a"); 550 | if (reps > 1){ 551 | puts("size\ttime\t\tbps\t\turl"); 552 | } 553 | double sum_len = 0; 554 | double sum_elapsed = 0; 555 | for (uint64_t r = 1; r <= reps; r++) { 556 | int url_idx = optind; 557 | while (url_idx < argc) { 558 | // open a new connection, or get an open one 559 | warn(INF, "%s retrieving %s", basename(argv[0]), argv[url_idx]); 560 | get(argv[url_idx++], w, &cc); 561 | } 562 | 563 | // collect the replies 564 | bool all_closed; 565 | do { 566 | all_closed = true; 567 | bool rxed_new = false; 568 | struct stream_entry * se = 0; //Is this entry the stream entry of the client? 569 | struct stream_entry * tmp = 0; 570 | sl_foreach_safe (se, &sl, next, tmp) { 571 | if (se->cce == 0 || se->cce->c == 0 || se->s == 0) { 572 | sl_remove(&sl, se, stream_entry, next); 573 | free_se(se); 574 | continue; 575 | } 576 | try_migrate(se->cce); 577 | rxed_new |= q_read_stream(se->s, &se->rep, false); 578 | 579 | const bool is_closed = q_peer_closed_stream(se->s); 580 | all_closed &= is_closed; 581 | if (is_closed) 582 | clock_gettime(CLOCK_MONOTONIC, &se->rep_t); 583 | } 584 | 585 | if (rxed_new == false && all_closed == false) { 586 | struct q_conn * c; 587 | q_ready(w, timeout * NS_PER_S, &c); 588 | if (c == 0) 589 | break; 590 | if (q_is_conn_closed(c)) 591 | break; 592 | } 593 | 594 | } while (all_closed == false); 595 | 596 | // print/save the replies 597 | while (sl_empty(&sl) == false) { 598 | struct stream_entry * const se = sl_first(&sl); 599 | if (ret == -1) 600 | ret = w_iov_sq_cnt(&se->rep) == 0; 601 | else 602 | ret |= w_iov_sq_cnt(&se->rep) == 0; 603 | 604 | struct timespec diff; 605 | timespec_sub(&se->rep_t, &se->req_t, &diff); 606 | const double elapsed = timespec_to_double(diff); 607 | const uint_t rep_len = w_iov_sq_len(&se->rep); 608 | sum_len += (double)rep_len; 609 | sum_elapsed += elapsed; 610 | if (reps > 1) 611 | printf("%" PRIu "\t%f\t\"%s\"\t%s\n", rep_len, elapsed, 612 | bps(rep_len, elapsed), se->url); 613 | fprintf(ins_fd, "%s\n", ins_bps(rep_len, elapsed)); 614 | 615 | #ifndef NDEBUG 616 | char cid_str[64]; 617 | q_cid_str(se->cce->c, cid_str, sizeof(cid_str)); 618 | warn(WRN, 619 | "read %" PRIu 620 | " byte%s in %.3f sec (%s) on conn %s strm %" PRIu, 621 | rep_len, plural(rep_len), elapsed < 0 ? 0 : elapsed, 622 | bps(rep_len, elapsed), cid_str, q_sid(se->s)); 623 | #endif 624 | 625 | // retrieve the TX'ed request 626 | q_stream_get_written(se->s, &se->req); 627 | 628 | if (write_files) 629 | write_object(se); 630 | 631 | // save the object, and print its first three packets to stdout 632 | struct w_iov * v; 633 | uint32_t n = 0; 634 | sq_foreach (v, &se->rep, next) { 635 | // cppcheck-suppress nullPointer 636 | const bool is_last = v == sq_last(&se->rep, w_iov, next); 637 | if (w_iov_sq_cnt(&se->rep) > 100 || reps > 1) 638 | // don't print large responses, or repeated ones 639 | continue; 640 | 641 | // XXX the strnlen() test is super-hacky 642 | if (do_h3 && n == 0 && 643 | (v->buf[0] != 0x01 && v->buf[0] != 0xff && 644 | strnlen((char *)v->buf, v->len) == v->len)) 645 | warn(WRN, "no h3 payload"); 646 | if (n < 4 || is_last) { 647 | if (do_h3) { 648 | #ifndef NDEBUG 649 | if (util_dlevel == DBG) 650 | hexdump(v->buf, v->len); 651 | #endif 652 | } else { 653 | // don't print newlines to console log 654 | for (uint16_t p = 0; p < v->len; p++) 655 | if (v->buf[p] == '\n' || v->buf[p] == '\r') 656 | v->buf[p] = ' '; 657 | printf("%.*s%s", v->len, v->buf, is_last ? "\n" : ""); 658 | if (is_last) 659 | fflush(stdout); 660 | } 661 | } else 662 | printf("."); 663 | n++; 664 | } 665 | 666 | q_free_stream(se->s); 667 | free_sl_head(); 668 | } 669 | } 670 | 671 | if (reps > 1) 672 | fclose(ins_fd); 673 | printf("TOTAL: %s\n", bps(sum_len, sum_elapsed)); 674 | 675 | free_cc(&cc); 676 | free_sl(); 677 | q_cleanup(w); 678 | warn(DBG, "%s exiting with %d", basename(argv[0]), ret); 679 | return ret; 680 | } 681 | -------------------------------------------------------------------------------- /kits/mvfst/README.md: -------------------------------------------------------------------------------- 1 | ### README 2 | 3 | 1. Firstly you need to setup the connection between two interfaces (`ens3f0` `ens3f1` by default); 4 | 2. `TLEM` can be used for this purpose. For example if you want to add 500us latency between the two interfaces, you can use cmds below. More options can be found [here](https://github.com/luigirizzo/netmap/tree/master/apps/tlem). 5 | > ./tlem -D 500us -i netmap:ens4f0 -i netmap:ens4f1 6 | 3. Run `config.sh` and re-compile `picoquic`; 7 | 4. Copy `server.sh`, `client.sh` and `multi_client.sh` into `extern/mvfst/mvfst/quic/tools/tperf/`; 8 | 5. Run the quant server first by executing `server.sh`; 9 | 6. Run the quant client by executing `client.sh` (or `multi_client.sh` for multi-connection test); 10 | 7. Results (Throughput) is extracted to `throughput.txt`. -------------------------------------------------------------------------------- /kits/mvfst/client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/ bash 2 | #a stands for total repetitions of test (duration) 3 | 4 | echo "+++{$a}" >> throughput.txt 5 | #taskset 128 ./tperf -max_log_size 1 -congestion cubic -duration {$a} -host 192.168.0.1 -mode client -p 4433 6 | #use the cmd below to fire up perf. 7 | perf record --call-graph lbr -- taskset 128 ~/yuecheng/QUIC-measurement-kit/extern/mvfst/_build/build/quic/tools/tperf/./tperf -max_log_size 1 -congestion cubic -duration $1 -host 192.168.0.1 -mode client -port 4433 8 | sleep 1 9 | echo "---{$1}" >> throughput.txt 10 | -------------------------------------------------------------------------------- /kits/mvfst/config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | mv ../../extern/mvfst/quic/tools/tperf/tperf.cpp ../../extern/mvfst/quic/tools/tperf/tperf.cpp.bak 3 | cp tperf.cpp ../../extern/mvfst/quic/tools/tperf/ -------------------------------------------------------------------------------- /kits/mvfst/multi_client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # a stands for overall number of connections 3 | a=$1 4 | # b stands for tags (packet loss rate, etc.) 5 | b=$2 6 | ifconfig ens3f1 192.168.0.4 7 | echo "+++${b}" >>throughput.txt 8 | for i in $(eval echo {0..${a}}) 9 | do 10 | myval=$(echo "2^${i}"|bc) 11 | #use the cmd below to fire up perf. 12 | #perf record --call-graph lbr -- taskset 64 ./tperf -max_log_size 1 -congestion cubic -duration {$a} -host 192.168.0.1 -mode client -p 4433 & 13 | taskset $myval ./tperf -max_log_size 1 -congestion cubic -duration {$a} -host 192.168.0.1 -mode client -p 4433 & 14 | 15 | done 16 | sleep 15 17 | echo "---${b}" >>throughput.txt 18 | -------------------------------------------------------------------------------- /kits/mvfst/server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/ bash 2 | 3 | #ip netns exec blue taskset 128 ~/yuecheng/QUIC-measurement-kit/extern/mvfst/_build/build/quic/tools/tperf/./tperf -max_log_size 1 -congestion cubic -host 192.168.0.2 -mode server -port 4433 4 | #use the cmd below to fire up perf. 5 | perf record --call-graph lbr ip netns exec blue taskset 128 ~/yuecheng/QUIC-measurement-kit/extern/mvfst/_build/build/quic/tools/tperf/./tperf -max_log_size 1 -congestion cubic -host 192.168.0.1 -mode server -port 4434 6 | -------------------------------------------------------------------------------- /kits/mvfst/tperf.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) Facebook, Inc. and its affiliates. 3 | * 4 | * This source code is licensed under the MIT license found in the 5 | * LICENSE file in the root directory of this source tree. 6 | * 7 | */ 8 | 9 | #include 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | 27 | DEFINE_string(host, "::1", "TPerf server hostname/IP"); 28 | DEFINE_int32(port, 6666, "TPerf server port"); 29 | DEFINE_string(mode, "server", "Mode to run in: 'client' or 'server'"); 30 | DEFINE_int32(duration, 10, "Duration of test in seconds"); 31 | DEFINE_uint64( 32 | block_size, 33 | 4096, 34 | "Amount of data written to stream each iteration"); 35 | DEFINE_uint64(writes_per_loop, 5, "Amount of socket writes per event loop"); 36 | DEFINE_uint64(window, 64 * 1024, "Flow control window size"); 37 | DEFINE_string(congestion, "newreno", "newreno/cubic/bbr/none"); 38 | DEFINE_bool(pacing, false, "Enable pacing"); 39 | DEFINE_bool(gso, false, "Enable GSO writes to the socket"); 40 | DEFINE_uint32( 41 | client_transport_timer_resolution_ms, 42 | 1, 43 | "Timer resolution for Ack and Loss tiemout in client transport"); 44 | DEFINE_string( 45 | server_qlogger_path, 46 | "", 47 | "Path to the directory where qlog files will be written. File will be named" 48 | " as .qlog where CID is the DCID from client's perspective."); 49 | DEFINE_uint32( 50 | max_cwnd_mss, 51 | quic::kLargeMaxCwndInMss, 52 | "Max cwnd in the unit of mss"); 53 | DEFINE_uint32(num_streams, 1, "Number of streams to send on simultaneously"); 54 | DEFINE_uint64( 55 | bytes_per_stream, 56 | 0, 57 | "Maximum number of bytes per stream. " 58 | "0 (the default) means the stream lives for the whole duration of the test."); 59 | DEFINE_string( 60 | pacing_observer, 61 | "none", 62 | "none/time/rtt/ack: Pacing observer bucket type: per 3ms, per rtt or per ack"); 63 | DEFINE_uint32( 64 | max_receive_packet_size, 65 | std::max( 66 | quic::kDefaultV4UDPSendPacketLen, 67 | quic::kDefaultV6UDPSendPacketLen), 68 | "Maximum packet size to advertise to the peer."); 69 | 70 | FILE *f; 71 | 72 | namespace quic { 73 | namespace tperf { 74 | 75 | class ServerStreamHandler : public quic::QuicSocket::ConnectionCallback, 76 | public quic::QuicSocket::ReadCallback, 77 | public quic::QuicSocket::WriteCallback { 78 | public: 79 | explicit ServerStreamHandler( 80 | folly::EventBase* evbIn, 81 | uint64_t blockSize, 82 | uint32_t numStreams, 83 | uint64_t maxBytesPerStream) 84 | : evb_(evbIn), 85 | blockSize_(blockSize), 86 | numStreams_(numStreams), 87 | maxBytesPerStream_(maxBytesPerStream) {} 88 | 89 | void setQuicSocket(std::shared_ptr socket) { 90 | sock_ = socket; 91 | } 92 | 93 | void onNewBidirectionalStream(quic::StreamId id) noexcept override { 94 | LOG(INFO) << "Got bidirectional stream id=" << id; 95 | sock_->setReadCallback(id, this); 96 | } 97 | 98 | void onNewUnidirectionalStream(quic::StreamId id) noexcept override { 99 | LOG(INFO) << "Got unidirectional stream id=" << id; 100 | sock_->setReadCallback(id, this); 101 | } 102 | 103 | void onStopSending( 104 | quic::StreamId id, 105 | quic::ApplicationErrorCode error) noexcept override { 106 | LOG(INFO) << "Got StopSending stream id=" << id << " error=" << error; 107 | } 108 | 109 | void onConnectionEnd() noexcept override { 110 | LOG(INFO) << "Socket closed"; 111 | sock_.reset(); 112 | } 113 | 114 | void onConnectionError( 115 | std::pair error) noexcept override { 116 | LOG(ERROR) << "Socket error=" << toString(error.first); 117 | } 118 | 119 | void onTransportReady() noexcept override { 120 | LOG(INFO) << "Starting sends to client."; 121 | for (uint32_t i = 0; i < numStreams_; i++) { 122 | createNewStream(); 123 | } 124 | } 125 | 126 | void createNewStream() noexcept { 127 | if (!sock_) { 128 | VLOG(4) << __func__ << ": socket is closed."; 129 | return; 130 | } 131 | auto stream = sock_->createUnidirectionalStream(); 132 | VLOG(5) << "New Stream with id = " << stream.value(); 133 | CHECK(stream.hasValue()); 134 | bytesPerStream_[stream.value()] = 0; 135 | notifyDataForStream(stream.value()); 136 | } 137 | 138 | void notifyDataForStream(quic::StreamId id) { 139 | evb_->runInEventBaseThread([&, id]() { 140 | if (!sock_) { 141 | VLOG(5) << "notifyDataForStream(" << id << "): socket is closed."; 142 | return; 143 | } 144 | auto res = sock_->notifyPendingWriteOnStream(id, this); 145 | if (res.hasError()) { 146 | LOG(FATAL) << quic::toString(res.error()); 147 | } 148 | }); 149 | } 150 | 151 | void readAvailable(quic::StreamId id) noexcept override { 152 | LOG(INFO) << "read available for stream id=" << id; 153 | } 154 | 155 | void readError( 156 | quic::StreamId id, 157 | std::pair> 158 | error) noexcept override { 159 | LOG(ERROR) << "Got read error on stream=" << id 160 | << " error=" << toString(error); 161 | // A read error only terminates the ingress portion of the stream state. 162 | // Your application should probably terminate the egress portion via 163 | // resetStream 164 | } 165 | 166 | void onStreamWriteReady( 167 | quic::StreamId id, 168 | uint64_t maxToSend) noexcept override { 169 | bool eof = false; 170 | uint64_t toSend = maxToSend; 171 | if (maxBytesPerStream_ > 0) { 172 | toSend = 173 | std::min(toSend, maxBytesPerStream_ - bytesPerStream_[id]); 174 | bytesPerStream_[id] += toSend; 175 | if (bytesPerStream_[id] >= maxBytesPerStream_) { 176 | eof = true; 177 | } 178 | } 179 | auto buf = folly::IOBuf::createChain(toSend, blockSize_); 180 | auto curBuf = buf.get(); 181 | do { 182 | curBuf->append(curBuf->capacity()); 183 | curBuf = curBuf->next(); 184 | } while (curBuf != buf.get()); 185 | auto res = sock_->writeChain(id, std::move(buf), eof, true, nullptr); 186 | if (res.hasError()) { 187 | LOG(FATAL) << "Got error on write: " << quic::toString(res.error()); 188 | } 189 | if (!eof) { 190 | notifyDataForStream(id); 191 | } else { 192 | bytesPerStream_.erase(id); 193 | createNewStream(); 194 | } 195 | } 196 | 197 | void onStreamWriteError( 198 | quic::StreamId id, 199 | std::pair> 200 | error) noexcept override { 201 | LOG(ERROR) << "write error with stream=" << id 202 | << " error=" << toString(error); 203 | } 204 | 205 | folly::EventBase* getEventBase() { 206 | return evb_; 207 | } 208 | 209 | private: 210 | std::shared_ptr sock_; 211 | folly::EventBase* evb_; 212 | uint64_t blockSize_; 213 | uint32_t numStreams_; 214 | uint64_t maxBytesPerStream_; 215 | std::unordered_map bytesPerStream_; 216 | }; 217 | 218 | class TPerfServerTransportFactory : public quic::QuicServerTransportFactory { 219 | public: 220 | ~TPerfServerTransportFactory() override = default; 221 | 222 | explicit TPerfServerTransportFactory( 223 | uint64_t blockSize, 224 | uint32_t numStreams, 225 | uint64_t maxBytesPerStream) 226 | : blockSize_(blockSize), 227 | numStreams_(numStreams), 228 | maxBytesPerStream_(maxBytesPerStream) {} 229 | 230 | quic::QuicServerTransport::Ptr make( 231 | folly::EventBase* evb, 232 | std::unique_ptr sock, 233 | const folly::SocketAddress&, 234 | std::shared_ptr 235 | ctx) noexcept override { 236 | CHECK_EQ(evb, sock->getEventBase()); 237 | auto serverHandler = std::make_unique( 238 | evb, blockSize_, numStreams_, maxBytesPerStream_); 239 | auto transport = quic::QuicServerTransport::make( 240 | evb, std::move(sock), *serverHandler, ctx); 241 | if (!FLAGS_server_qlogger_path.empty()) { 242 | auto qlogger = std::make_shared( 243 | VantagePoint::Server, FLAGS_server_qlogger_path); 244 | setPacingObserver(qlogger, transport.get(), FLAGS_pacing_observer); 245 | transport->setQLogger(std::move(qlogger)); 246 | } 247 | serverHandler->setQuicSocket(transport); 248 | handlers_.push_back(std::move(serverHandler)); 249 | return transport; 250 | } 251 | 252 | private: 253 | void setPacingObserver( 254 | std::shared_ptr& qlogger, 255 | quic::QuicServerTransport* transport, 256 | const std::string& pacingObserverType) { 257 | if (pacingObserverType == "time") { 258 | qlogger->setPacingObserver( 259 | std::make_unique(qlogger, 3ms)); 260 | } else if (pacingObserverType == "rtt") { 261 | qlogger->setPacingObserver(std::make_unique( 262 | qlogger, *transport->getState())); 263 | } else if (pacingObserverType == "ack") { 264 | qlogger->setPacingObserver(std::make_unique(qlogger)); 265 | } 266 | } 267 | 268 | std::vector> handlers_; 269 | uint64_t blockSize_; 270 | uint32_t numStreams_; 271 | uint64_t maxBytesPerStream_; 272 | }; 273 | 274 | class Timer { 275 | bool clear = false; 276 | 277 | public: 278 | template 279 | void setTimeout(Function function, int delay); 280 | template 281 | void setInterval(Function function, int interval); 282 | void stop(); 283 | }; 284 | 285 | template 286 | void Timer::setTimeout(Function function, int delay) { 287 | this->clear = false; 288 | std::thread t([=,this]() { 289 | if(this->clear) return; 290 | std::this_thread::sleep_for(std::chrono::milliseconds(delay)); 291 | if(this->clear) return; 292 | function(); 293 | }); 294 | t.detach(); 295 | } 296 | 297 | template 298 | void Timer::setInterval(Function function, int interval) { 299 | this->clear = false; 300 | std::thread t([=,this] { 301 | if(this->clear) return; 302 | while(1){ 303 | std::this_thread::sleep_for(std::chrono::milliseconds(interval)); 304 | if(this->clear) return; 305 | function(); 306 | } 307 | }); 308 | t.detach(); 309 | } 310 | 311 | void Timer::stop() { 312 | this->clear = true; 313 | } 314 | 315 | class TPerfServer { 316 | public: 317 | explicit TPerfServer( 318 | const std::string& host, 319 | uint16_t port, 320 | uint64_t blockSize, 321 | uint64_t writesPerLoop, 322 | quic::CongestionControlType congestionControlType, 323 | bool gso, 324 | uint32_t maxCwndInMss, 325 | bool pacing, 326 | uint32_t numStreams, 327 | uint64_t maxBytesPerStream, 328 | uint32_t maxReceivePacketSize) 329 | : host_(host), port_(port), server_(QuicServer::createQuicServer()) { 330 | eventBase_.setName("tperf_server"); 331 | server_->setQuicServerTransportFactory( 332 | std::make_unique( 333 | blockSize, numStreams, maxBytesPerStream)); 334 | auto serverCtx = quic::test::createServerCtx(); 335 | serverCtx->setClock(std::make_shared()); 336 | server_->setFizzContext(serverCtx); 337 | quic::TransportSettings settings; 338 | settings.maxCwndInMss = maxCwndInMss; 339 | settings.writeConnectionDataPacketsLimit = writesPerLoop; 340 | settings.defaultCongestionController = congestionControlType; 341 | settings.pacingEnabled = pacing; 342 | if (pacing) { 343 | settings.pacingTimerTickInterval = 200us; 344 | } 345 | if (gso) { 346 | settings.batchingMode = QuicBatchingMode::BATCHING_MODE_GSO; 347 | settings.maxBatchSize = 16; 348 | } 349 | settings.maxRecvPacketSize = maxReceivePacketSize; 350 | settings.canIgnorePathMTU = true; 351 | server_->setTransportSettings(settings); 352 | } 353 | 354 | void start() { 355 | // Create a SocketAddress and the default or passed in host. 356 | folly::SocketAddress addr1(host_.c_str(), port_); 357 | addr1.setFromHostPort(host_, port_); 358 | server_->start(addr1, 0); 359 | LOG(INFO) << "tperf server started at: " << addr1.describe(); 360 | eventBase_.loopForever(); 361 | } 362 | 363 | private: 364 | std::string host_; 365 | uint16_t port_; 366 | folly::EventBase eventBase_; 367 | std::shared_ptr server_; 368 | }; 369 | 370 | class TPerfClient : public quic::QuicSocket::ConnectionCallback, 371 | public quic::QuicSocket::ReadCallback, 372 | public quic::QuicSocket::WriteCallback, 373 | public folly::HHWheelTimer::Callback { 374 | public: 375 | TPerfClient( 376 | const std::string& host, 377 | uint16_t port, 378 | std::chrono::milliseconds transportTimerResolution, 379 | int32_t duration, 380 | uint64_t window, 381 | bool gso, 382 | quic::CongestionControlType congestionControlType, 383 | uint32_t maxReceivePacketSize) 384 | : host_(host), 385 | port_(port), 386 | eventBase_(transportTimerResolution), 387 | duration_(duration), 388 | window_(window), 389 | gso_(gso), 390 | congestionControlType_(congestionControlType), 391 | maxReceivePacketSize_(maxReceivePacketSize) { 392 | eventBase_.setName("tperf_client"); 393 | } 394 | 395 | void averageThroughput(){ 396 | constexpr double bytesPerMegabit = 131072; 397 | printf("Received %lu bytes in %lu sec\n", receivedBytes_, 398 | duration_.count()); 399 | printf("Average throughput: %lf Mb/s\n", 400 | (receivedBytes_ / bytesPerMegabit) / duration_.count()); 401 | 402 | fprintf(f, "%lf\n", 403 | (receivedBytes_ / bytesPerMegabit) / duration_.count()); 404 | } 405 | 406 | void timeoutExpired() noexcept override { 407 | quicClient_->closeNow(folly::none); 408 | constexpr double bytesPerMegabit = 131072; 409 | LOG(INFO) << "Received " << receivedBytes_ << " bytes in " 410 | << duration_.count() << " seconds."; 411 | LOG(INFO) << "Overall throughput: " 412 | << (receivedBytes_ / bytesPerMegabit) / duration_.count() 413 | << "Mb/s"; 414 | // Per Stream Stats 415 | LOG(INFO) << "Average per Stream throughput: " 416 | << ((receivedBytes_ / receivedStreams_) / bytesPerMegabit) / 417 | duration_.count() 418 | << "Mb/s over " << receivedStreams_ << " streams"; 419 | if (receivedStreams_ != 1) { 420 | LOG(INFO) << "Histogram per Stream bytes: " << std::endl; 421 | LOG(INFO) << "Lo\tHi\tNum\tSum"; 422 | for (const auto bytes : bytesPerStream_) { 423 | bytesPerStreamHistogram_.addValue(bytes.second); 424 | } 425 | std::ostringstream os; 426 | bytesPerStreamHistogram_.toTSV(os); 427 | std::vector lines; 428 | folly::split("\n", os.str(), lines); 429 | for (const auto& line : lines) { 430 | LOG(INFO) << line; 431 | } 432 | } 433 | } 434 | 435 | virtual void callbackCanceled() noexcept override {} 436 | 437 | void readAvailable(quic::StreamId streamId) noexcept override { 438 | auto readData = quicClient_->read(streamId, 0); 439 | if (readData.hasError()) { 440 | LOG(FATAL) << "TPerfClient failed read from stream=" << streamId 441 | << ", error=" << (uint32_t)readData.error(); 442 | } 443 | 444 | auto readBytes = readData->first->computeChainDataLength(); 445 | receivedBytes_ += readBytes; 446 | bytesPerStream_[streamId] += readBytes; 447 | if (readData.value().second) { 448 | bytesPerStreamHistogram_.addValue(bytesPerStream_[streamId]); 449 | bytesPerStream_.erase(streamId); 450 | } 451 | } 452 | 453 | void readError( 454 | quic::StreamId /*streamId*/, 455 | std::pair> 456 | /*error*/) noexcept override { 457 | // A read error only terminates the ingress portion of the stream state. 458 | // Your application should probably terminate the egress portion via 459 | // resetStream 460 | } 461 | 462 | void onNewBidirectionalStream(quic::StreamId id) noexcept override { 463 | LOG(INFO) << "TPerfClient: new bidirectional stream=" << id; 464 | quicClient_->setReadCallback(id, this); 465 | } 466 | 467 | void onNewUnidirectionalStream(quic::StreamId id) noexcept override { 468 | VLOG(5) << "TPerfClient: new unidirectional stream=" << id; 469 | if (!timerScheduled_) { 470 | timerScheduled_ = true; 471 | eventBase_.timer().scheduleTimeout(this, duration_); 472 | } 473 | quicClient_->setReadCallback(id, this); 474 | receivedStreams_++; 475 | } 476 | 477 | void onTransportReady() noexcept override { 478 | LOG(INFO) << "TPerfClient: onTransportReady"; 479 | } 480 | 481 | void onStopSending( 482 | quic::StreamId id, 483 | quic::ApplicationErrorCode /*error*/) noexcept override { 484 | VLOG(10) << "TPerfClient got StopSending stream id=" << id; 485 | } 486 | 487 | void onConnectionEnd() noexcept override { 488 | LOG(INFO) << "TPerfClient connection end"; 489 | 490 | eventBase_.terminateLoopSoon(); 491 | } 492 | 493 | void onConnectionError( 494 | std::pair error) noexcept override { 495 | LOG(ERROR) << "TPerfClient error: " << toString(error.first); 496 | eventBase_.terminateLoopSoon(); 497 | } 498 | 499 | void onStreamWriteReady( 500 | quic::StreamId id, 501 | uint64_t maxToSend) noexcept override { 502 | LOG(INFO) << "TPerfClient stream" << id 503 | << " is write ready with maxToSend=" << maxToSend; 504 | } 505 | 506 | void onStreamWriteError( 507 | quic::StreamId id, 508 | std::pair> 509 | error) noexcept override { 510 | LOG(ERROR) << "TPerfClient write error with stream=" << id 511 | << " error=" << toString(error); 512 | } 513 | 514 | void rtThroughput(){ 515 | constexpr double bytesPerMegabit = 131072; 516 | uint64_t receivedBytes_bf = 0; 517 | uint64_t receivedBytes_af, deltaReceivedBytes; 518 | //update RT-throughput every second 519 | Timer t = Timer(); 520 | t.setInterval([&](){ 521 | receivedBytes_af = receivedBytes_; 522 | deltaReceivedBytes = receivedBytes_af - receivedBytes_bf; 523 | receivedBytes_bf = receivedBytes_af; 524 | printf("Average throughput: %lf Mb/s\n" 525 | , (deltaReceivedBytes / bytesPerMegabit) / 1.0); 526 | fprintf(f, "%lf\n" 527 | , (deltaReceivedBytes / bytesPerMegabit) / 1.0); 528 | }, 1000); 529 | } 530 | 531 | void start() { 532 | folly::SocketAddress addr(host_.c_str(), port_); 533 | 534 | auto sock = std::make_unique(&eventBase_); 535 | auto fizzClientContext = 536 | FizzClientQuicHandshakeContext::Builder() 537 | .setCertificateVerifier(test::createTestCertificateVerifier()) 538 | .build(); 539 | quicClient_ = std::make_shared( 540 | &eventBase_, std::move(sock), std::move(fizzClientContext)); 541 | quicClient_->setHostname("tperf"); 542 | quicClient_->addNewPeerAddress(addr); 543 | quicClient_->setCongestionControllerFactory( 544 | std::make_shared()); 545 | auto settings = quicClient_->getTransportSettings(); 546 | settings.advertisedInitialUniStreamWindowSize = window_; 547 | // TODO figure out what actually to do with conn flow control and not sent 548 | // limit. 549 | settings.advertisedInitialConnectionWindowSize = 550 | std::numeric_limits::max(); 551 | settings.connectUDP = true; 552 | settings.shouldRecvBatch = true; 553 | settings.defaultCongestionController = congestionControlType_; 554 | if (congestionControlType_ == quic::CongestionControlType::BBR) { 555 | settings.pacingEnabled = true; 556 | settings.pacingTimerTickInterval = 200us; 557 | } 558 | if (gso_) { 559 | settings.batchingMode = QuicBatchingMode::BATCHING_MODE_GSO; 560 | settings.maxBatchSize = 16; 561 | } 562 | settings.maxRecvPacketSize = maxReceivePacketSize_; 563 | settings.canIgnorePathMTU = true; 564 | quicClient_->setTransportSettings(settings); 565 | 566 | LOG(INFO) << "TPerfClient connecting to " << addr.describe(); 567 | quicClient_->start(this); 568 | eventBase_.loopForever(); 569 | } 570 | 571 | ~TPerfClient() override = default; 572 | 573 | private: 574 | bool timerScheduled_{false}; 575 | std::string host_; 576 | uint16_t port_; 577 | std::shared_ptr quicClient_; 578 | folly::EventBase eventBase_; 579 | uint64_t receivedBytes_{0}; 580 | uint64_t receivedStreams_{0}; 581 | std::map bytesPerStream_; 582 | folly::Histogram bytesPerStreamHistogram_{1024, 583 | 0, 584 | 1024 * 1024 * 1024}; 585 | std::chrono::seconds duration_; 586 | uint64_t window_; 587 | bool gso_; 588 | quic::CongestionControlType congestionControlType_; 589 | uint32_t maxReceivePacketSize_; 590 | }; 591 | 592 | } // namespace tperf 593 | } // namespace quic 594 | 595 | using namespace quic::tperf; 596 | 597 | quic::CongestionControlType flagsToCongestionControlType( 598 | const std::string& congestionControlType) { 599 | if (congestionControlType == "cubic") { 600 | return quic::CongestionControlType::Cubic; 601 | } else if (congestionControlType == "newreno") { 602 | return quic::CongestionControlType::NewReno; 603 | } else if (congestionControlType == "bbr") { 604 | return quic::CongestionControlType::BBR; 605 | } else if (congestionControlType == "copa") { 606 | return quic::CongestionControlType::Copa; 607 | } else if (congestionControlType == "none") { 608 | return quic::CongestionControlType::None; 609 | } 610 | throw std::invalid_argument(folly::to( 611 | "Unknown congestion controller ", congestionControlType)); 612 | } 613 | 614 | int main(int argc, char* argv[]) { 615 | #if FOLLY_HAVE_LIBGFLAGS 616 | // Enable glog logging to stderr by default. 617 | gflags::SetCommandLineOptionWithMode( 618 | "logtostderr", "1", gflags::SET_FLAGS_DEFAULT); 619 | #endif 620 | gflags::ParseCommandLineFlags(&argc, &argv, false); 621 | folly::Init init(&argc, &argv); 622 | fizz::CryptoUtils::init(); 623 | 624 | if (FLAGS_mode == "server") { 625 | TPerfServer server( 626 | FLAGS_host, 627 | FLAGS_port, 628 | FLAGS_block_size, 629 | FLAGS_writes_per_loop, 630 | flagsToCongestionControlType(FLAGS_congestion), 631 | FLAGS_gso, 632 | FLAGS_max_cwnd_mss, 633 | FLAGS_pacing, 634 | FLAGS_num_streams, 635 | FLAGS_bytes_per_stream, 636 | FLAGS_max_receive_packet_size); 637 | server.start(); 638 | } else if (FLAGS_mode == "client") { 639 | //open "throughput.txt" for recording 640 | f = fopen("throughput.txt", "a"); 641 | if (FLAGS_num_streams != 1) { 642 | LOG(ERROR) << "num_streams option is server only"; 643 | return 1; 644 | } 645 | if (FLAGS_bytes_per_stream != 0) { 646 | LOG(ERROR) << "bytes_per_stream option is server only"; 647 | return 1; 648 | } 649 | TPerfClient client( 650 | FLAGS_host, 651 | FLAGS_port, 652 | std::chrono::milliseconds(FLAGS_client_transport_timer_resolution_ms), 653 | FLAGS_duration, 654 | FLAGS_window, 655 | FLAGS_gso, 656 | flagsToCongestionControlType(FLAGS_congestion), 657 | FLAGS_max_receive_packet_size); 658 | //open up a thread to print real-time throughput every second. 659 | //client.rtThroughput(); 660 | client.start(); 661 | client.averageThroughput(); 662 | fclose(f); 663 | } 664 | return 0; 665 | } 666 | -------------------------------------------------------------------------------- /kits/picoquic/README.md: -------------------------------------------------------------------------------- 1 | ### README 2 | 3 | 1. Firstly you need to setup the connection between two interfaces (`ens3f0` `ens3f1` by default); 4 | 2. `TLEM` can be used for this purpose. For example if you want to add 500us latency between the two interfaces, you can use cmds below. More options can be found [here](https://github.com/luigirizzo/netmap/tree/master/apps/tlem). 5 | > ./tlem -D 500us -i netmap:ens4f0 -i netmap:ens4f1 6 | 3. Run `config.sh` and re-compile `picoquic`; 7 | 4. Copy `server.sh`, `client.sh` and `multi_client.sh` into `extern/picoquic/`; 8 | 5. Run the quant server first by executing `server.sh`; 9 | 6. Run the quant client by executing `client.sh`; 10 | 7. Results (Throughput) is extracted to `Throughput.txt`. -------------------------------------------------------------------------------- /kits/picoquic/client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # a stands for overrall connections you would ues. 3 | a=$1 4 | # b stands for repetitions of the test. 5 | b=$2 6 | ifconfig ens3f1 192.168.0.4 7 | echo "+++${b}" >>Throughput.txt 8 | for i in $(eval echo {0..${a}}) 9 | do 10 | #perf record --call-graph lbr -- taskset 64 ./picoquicdemo -l n -D 192.168.0.1 4433 11 | taskset 128 ./picoquicdemo -l n -D 192.168.0.1 4433 12 | done 13 | sleep 15 14 | echo "---${b}" >>Throughput.txt 15 | -------------------------------------------------------------------------------- /kits/picoquic/config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #put the instrumented source file into the proj 3 | mv ../../extern/picoquic/picoquicfirst/picoquicdemo.c ../../extern/picoquic/picoquicfirst/picoquicdemo.c.bak 4 | cp picoquicdemo.c ../../extern/picoquic/picoquicfirst/ -------------------------------------------------------------------------------- /kits/picoquic/multi_client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # a stands for overall number of connections 3 | a=$1 4 | # b stands for tags (packet loss rate, etc.) 5 | b=$2 6 | ifconfig ens3f1 192.168.0.4 7 | echo "+++${b}" >>Throughput.txt 8 | for i in $(eval echo {0..${a}}) 9 | do 10 | #perf record --call-graph lbr -- taskset 64 ./picoquicdemo -l n -D 192.168.0.1 4433 11 | myval=$(echo "2^${i}"|bc) 12 | taskset $myval ./picoquicdemo -l n -D 192.168.0.1 4433 & 13 | 14 | done 15 | sleep 15 16 | echo "---${b}" >>Throughput.txt 17 | -------------------------------------------------------------------------------- /kits/picoquic/picoquicdemo.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Author: Christian Huitema 3 | * Copyright (c) 2017, Private Octopus, Inc. 4 | * All rights reserved. 5 | * 6 | * Permission to use, copy, modify, and distribute this software for any 7 | * purpose with or without fee is hereby granted, provided that the above 8 | * copyright notice and this permission notice appear in all copies. 9 | * 10 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 11 | * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 12 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 13 | * DISCLAIMED. IN NO EVENT SHALL Private Octopus, Inc. BE LIABLE FOR ANY 14 | * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 15 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 16 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 17 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 18 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 19 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 20 | */ 21 | 22 | #ifdef _WINDOWS 23 | #define WIN32_LEAN_AND_MEAN 24 | #include "getopt.h" 25 | #include 26 | #include 27 | #include 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | 34 | #ifndef SOCKET_TYPE 35 | #define SOCKET_TYPE SOCKET 36 | #endif 37 | #ifndef SOCKET_CLOSE 38 | #define SOCKET_CLOSE(x) closesocket(x) 39 | #endif 40 | #ifndef WSA_LAST_ERROR 41 | #define WSA_LAST_ERROR(x) WSAGetLastError() 42 | #endif 43 | #ifndef socklen_t 44 | #define socklen_t int 45 | #endif 46 | 47 | #define SERVER_CERT_FILE "certs\\cert.pem" 48 | #define SERVER_KEY_FILE "certs\\key.pem" 49 | 50 | #else /* Linux */ 51 | 52 | #include 53 | #include 54 | #include 55 | #include 56 | #include 57 | #include 58 | #include 59 | 60 | #ifndef __USE_XOPEN2K 61 | #define __USE_XOPEN2K 62 | #endif 63 | #ifndef __USE_POSIX 64 | #define __USE_POSIX 65 | #endif 66 | #include 67 | #include 68 | #include 69 | #include 70 | #include 71 | 72 | #ifndef SOCKET_TYPE 73 | #define SOCKET_TYPE int 74 | #endif 75 | #ifndef INVALID_SOCKET 76 | #define INVALID_SOCKET -1 77 | #endif 78 | #ifndef SOCKET_CLOSE 79 | #define SOCKET_CLOSE(x) close(x) 80 | #endif 81 | #ifndef WSA_LAST_ERROR 82 | #define WSA_LAST_ERROR(x) ((long)(x)) 83 | #endif 84 | 85 | #define SERVER_CERT_FILE "certs/cert.pem" 86 | #define SERVER_KEY_FILE "certs/key.pem" 87 | 88 | #endif 89 | 90 | static const int default_server_port = 4443; 91 | static const char* default_server_name = "::"; 92 | static const char* ticket_store_filename = "demo_ticket_store.bin"; 93 | static const char* token_store_filename = "demo_token_store.bin"; 94 | 95 | #include "picoquic.h" 96 | #include "picoquic_internal.h" 97 | #include "picosocks.h" 98 | #include "picoquic_utils.h" 99 | #include "h3zero.c" 100 | #include "democlient.h" 101 | #include "demoserver.h" 102 | #include "siduck.h" 103 | 104 | /* 105 | * SIDUCK datagram demo call back. 106 | */ 107 | int siduck_callback(picoquic_cnx_t* cnx, 108 | uint64_t stream_id, uint8_t* bytes, size_t length, 109 | picoquic_call_back_event_t fin_or_event, void* callback_ctx, void* v_stream_ctx); 110 | 111 | void print_address(FILE* F_log, struct sockaddr* address, char* label, picoquic_connection_id_t cnx_id) 112 | { 113 | char hostname[256]; 114 | 115 | const char* x = inet_ntop(address->sa_family, 116 | (address->sa_family == AF_INET) ? (void*)&(((struct sockaddr_in*)address)->sin_addr) : (void*)&(((struct sockaddr_in6*)address)->sin6_addr), 117 | hostname, sizeof(hostname)); 118 | 119 | fprintf(F_log, "%016llx : ", (unsigned long long)picoquic_val64_connection_id(cnx_id)); 120 | 121 | if (x != NULL) { 122 | fprintf(F_log, "%s %s, port %d\n", label, x, 123 | (address->sa_family == AF_INET) ? ((struct sockaddr_in*)address)->sin_port : ((struct sockaddr_in6*)address)->sin6_port); 124 | } else { 125 | fprintf(F_log, "%s: inet_ntop failed with error # %ld\n", label, WSA_LAST_ERROR(errno)); 126 | } 127 | } 128 | 129 | static void picoquic_set_key_log_file_from_env(picoquic_quic_t* quic) 130 | { 131 | char * keylog_filename = NULL; 132 | FILE* F = NULL; 133 | 134 | #ifdef _WINDOWS 135 | size_t len; 136 | errno_t err = _dupenv_s(&keylog_filename, &len, "SSLKEYLOGFILE"); 137 | 138 | if (keylog_filename == NULL) { 139 | return; 140 | } 141 | 142 | if (err == 0) { 143 | err = fopen_s(&F, keylog_filename, "a"); 144 | 145 | free(keylog_filename); 146 | 147 | if (err != 0 || F == NULL) { 148 | return; 149 | } 150 | } 151 | #else 152 | keylog_filename = getenv("SSLKEYLOGFILE"); 153 | if (keylog_filename == NULL) { 154 | return; 155 | } 156 | F = fopen(keylog_filename, "a"); 157 | if (F == NULL) { 158 | return; 159 | } 160 | #endif 161 | 162 | picoquic_set_key_log_file(quic, F); 163 | } 164 | 165 | int quic_server(const char* server_name, int server_port, 166 | const char* pem_cert, const char* pem_key, 167 | int just_once, int do_hrr, picoquic_connection_id_cb_fn cnx_id_callback, 168 | void* cnx_id_callback_ctx, uint8_t reset_seed[PICOQUIC_RESET_SECRET_SIZE], 169 | int dest_if, int mtu_max, uint32_t proposed_version, 170 | const char * esni_key_file_name, const char * esni_rr_file_name, 171 | FILE * F_log, char const* bin_file, char const * cc_log_dir, int use_long_log, 172 | picoquic_congestion_algorithm_t const * cc_algorithm, char const * web_folder) 173 | { 174 | /* Start: start the QUIC process with cert and key files */ 175 | int ret = 0; 176 | picoquic_quic_t* qserver = NULL; 177 | picoquic_server_sockets_t server_sockets; 178 | struct sockaddr_storage addr_from; 179 | struct sockaddr_storage addr_to; 180 | unsigned long if_index_to; 181 | socklen_t from_length; 182 | socklen_t to_length; 183 | uint8_t buffer[1536]; 184 | uint8_t send_buffer[1536]; 185 | size_t send_length = 0; 186 | int bytes_recv; 187 | uint64_t current_time = 0; 188 | int64_t delay_max = 10000000; 189 | int connection_done = 0; 190 | picohttp_server_parameters_t picoquic_file_param; 191 | uint64_t loop_count_time = 0; 192 | int nb_loops = 0; 193 | int first_connection_seen = 0; 194 | 195 | memset(&picoquic_file_param, 0, sizeof(picohttp_server_parameters_t)); 196 | picoquic_file_param.web_folder = web_folder; 197 | 198 | // picoquic_set_default_callback(test_ctx->qserver, server_callback_fn, server_param); 199 | 200 | /* Open a UDP socket */ 201 | ret = picoquic_open_server_sockets(&server_sockets, server_port); 202 | 203 | /* Wait for packets and process them */ 204 | if (ret == 0) { 205 | current_time = picoquic_current_time(); 206 | loop_count_time = current_time; 207 | /* Create QUIC context */ 208 | qserver = picoquic_create(8, pem_cert, pem_key, NULL, NULL, 209 | picoquic_demo_server_callback, &picoquic_file_param, 210 | cnx_id_callback, cnx_id_callback_ctx, reset_seed, current_time, NULL, NULL, NULL, 0); 211 | 212 | if (qserver == NULL) { 213 | printf("Could not create server context\n"); 214 | ret = -1; 215 | } else { 216 | picoquic_set_alpn_select_fn(qserver, picoquic_demo_server_callback_select_alpn); 217 | if (do_hrr != 0) { 218 | picoquic_set_cookie_mode(qserver, 1); 219 | } 220 | qserver->mtu_max = mtu_max; 221 | 222 | if (cc_algorithm == NULL) { 223 | cc_algorithm = picoquic_bbr_algorithm; 224 | } 225 | picoquic_set_default_congestion_algorithm(qserver, cc_algorithm); 226 | 227 | PICOQUIC_SET_LOG(qserver, F_log); 228 | 229 | if (use_long_log) { 230 | qserver->use_long_log = 1; 231 | } 232 | 233 | picoquic_set_key_log_file_from_env(qserver); 234 | 235 | if (cc_log_dir != NULL) { 236 | picoquic_set_cc_log(qserver, cc_log_dir); 237 | } 238 | 239 | if (bin_file != NULL) { 240 | picoquic_set_binlog(qserver, bin_file); 241 | } 242 | 243 | if (esni_key_file_name != NULL && esni_rr_file_name != NULL) { 244 | ret = picoquic_esni_load_key(qserver, esni_key_file_name); 245 | if (ret == 0) { 246 | ret = picoquic_esni_server_setup(qserver, esni_rr_file_name); 247 | } 248 | } 249 | } 250 | } 251 | 252 | /* Wait for packets */ 253 | while (ret == 0 && (!just_once || !connection_done)) { 254 | int64_t delta_t = picoquic_get_next_wake_delay(qserver, current_time, delay_max); 255 | unsigned char received_ecn; 256 | 257 | from_length = to_length = sizeof(struct sockaddr_storage); 258 | if_index_to = 0; 259 | 260 | bytes_recv = picoquic_select(server_sockets.s_socket, PICOQUIC_NB_SERVER_SOCKETS, 261 | &addr_from, &from_length, 262 | &addr_to, &to_length, &if_index_to, &received_ecn, 263 | buffer, sizeof(buffer), 264 | delta_t, ¤t_time); 265 | 266 | nb_loops++; 267 | if (nb_loops >= 10000) { 268 | FILE* loop_out = (F_log == NULL) ? stdout : F_log; 269 | uint64_t loop_delta = current_time - loop_count_time; 270 | loop_count_time = current_time; 271 | 272 | fprintf(loop_out, "Looped %d times in %llu microsec, file: %d, line: %d\n", 273 | nb_loops, (unsigned long long) loop_delta, qserver->wake_file, qserver->wake_line); 274 | fflush(loop_out); 275 | nb_loops = 0; 276 | } 277 | 278 | if (bytes_recv < 0) { 279 | ret = -1; 280 | } else { 281 | uint64_t loop_time; 282 | 283 | if (bytes_recv > 0) { 284 | /* Submit the packet to the server */ 285 | (void)picoquic_incoming_packet(qserver, buffer, 286 | (size_t)bytes_recv, (struct sockaddr*)&addr_from, 287 | (struct sockaddr*)&addr_to, if_index_to, received_ecn, 288 | current_time); 289 | 290 | if (just_once && !first_connection_seen && picoquic_get_first_cnx(qserver) != NULL) { 291 | first_connection_seen = 1; 292 | fprintf(stdout, "First connection noticed.\n"); 293 | } 294 | } 295 | loop_time = current_time; 296 | 297 | do { 298 | int peer_addr_len = 0; 299 | struct sockaddr_storage peer_addr; 300 | int local_addr_len = 0; 301 | struct sockaddr_storage local_addr; 302 | int if_index = dest_if; 303 | 304 | 305 | ret = picoquic_prepare_next_packet(qserver, loop_time, 306 | send_buffer, sizeof(send_buffer), &send_length, 307 | &peer_addr, &peer_addr_len, &local_addr, &local_addr_len, 308 | &if_index); 309 | 310 | if (ret == 0 && send_length > 0) { 311 | loop_count_time = current_time; 312 | nb_loops = 0; 313 | (void)picoquic_send_through_server_sockets(&server_sockets, 314 | (struct sockaddr*) & peer_addr, peer_addr_len, (struct sockaddr*) & local_addr, local_addr_len, if_index, 315 | (const char*)send_buffer, (int)send_length); 316 | } 317 | 318 | } while (ret == 0 && send_length > 0); 319 | 320 | if (just_once && first_connection_seen && picoquic_get_first_cnx(qserver) == NULL) { 321 | fprintf(stdout, "No more active connections.\n"); 322 | connection_done = 1; 323 | } 324 | } 325 | } 326 | 327 | printf("Server exit, ret = %d\n", ret); 328 | if (F_log != NULL) { 329 | fprintf(F_log, "Server exit, ret = %d\n", ret); 330 | fflush(F_log); 331 | if (F_log != stdout) { 332 | (void)picoquic_file_close(F_log); 333 | } 334 | } 335 | 336 | /* Clean up */ 337 | if (qserver != NULL) { 338 | picoquic_free(qserver); 339 | } 340 | 341 | picoquic_close_server_sockets(&server_sockets); 342 | 343 | return ret; 344 | } 345 | 346 | static const char * test_scenario_default = "0:/30000000;4:test.html;8:/12345670;12:main.jpg;16:war-and-peace.txt;20:en/latest/;24:/file-123K"; 347 | 348 | #define PICOQUIC_DEMO_CLIENT_MAX_RECEIVE_BATCH 4 349 | 350 | /* Client client migration to a new port number: 351 | * - close the current socket. 352 | * - open another socket at a randomly picked port number. 353 | * - call the create probe API. 354 | * This is a bit tricky because the probe API requires passing the new address, 355 | * but in many cases the client will be behind a NAT, so it will not know its 356 | * actual IP address. 357 | */ 358 | int quic_client_migrate(picoquic_cnx_t * cnx, SOCKET_TYPE * fd, struct sockaddr * server_address, struct sockaddr* client_address, 359 | int * address_updated, int force_migration, FILE * F_log) 360 | { 361 | int ret = 0; 362 | 363 | if (server_address == NULL) { 364 | server_address = (struct sockaddr*) & cnx->path[0]->peer_addr; 365 | } 366 | 367 | if (force_migration != 2) { 368 | SOCKET_TYPE fd_m; 369 | 370 | 371 | fd_m = picoquic_open_client_socket(server_address->sa_family); 372 | if (fd_m == INVALID_SOCKET) { 373 | fprintf(stdout, "Could not open new socket.\n"); 374 | if (F_log != stdout && F_log != stderr && F_log != NULL) 375 | { 376 | fprintf(stdout, "Could not open new socket.\n"); 377 | } 378 | ret = -1; 379 | } 380 | else { 381 | if (force_migration == 3) { 382 | uint16_t port = (client_address->sa_family == AF_INET) ? 383 | ((struct sockaddr_in*)client_address)->sin_port : 384 | ((struct sockaddr_in6*)client_address)->sin6_port; 385 | 386 | for (int trial = 0; trial < 4; trial++) { 387 | port++; 388 | ret = picoquic_bind_to_port(fd_m, client_address->sa_family, port); 389 | if (ret == 0) { 390 | if (client_address->sa_family == AF_INET) { 391 | ((struct sockaddr_in*)client_address)->sin_port = port; 392 | } 393 | else { 394 | ((struct sockaddr_in6*)client_address)->sin6_port = port; 395 | } 396 | break; 397 | } 398 | } 399 | if (ret != 0) { 400 | DBG_PRINTF("Could not bind new socket to port %d", port); 401 | } 402 | } 403 | if (ret == 0) { 404 | SOCKET_CLOSE(*fd); 405 | *fd = fd_m; 406 | } 407 | else { 408 | SOCKET_CLOSE(fd_m); 409 | } 410 | } 411 | } 412 | 413 | if (ret == 0) { 414 | if (force_migration == 1) { 415 | fprintf(stdout, "Switch to new port. Will test NAT rebinding support.\n"); 416 | if (F_log != stdout && F_log != stderr && F_log != NULL) 417 | { 418 | fprintf(F_log, "Switch to new port. Will test NAT rebinding support.\n"); 419 | } 420 | *address_updated = 0; 421 | } 422 | else if (force_migration == 2) { 423 | ret = picoquic_renew_connection_id(cnx, 0); 424 | if (ret != 0) { 425 | if (ret == PICOQUIC_ERROR_MIGRATION_DISABLED) { 426 | fprintf(stdout, "Migration disabled, cannot test CNXID renewal.\n"); 427 | if (F_log != stdout && F_log != stderr && F_log != NULL) 428 | { 429 | fprintf(stdout, "Migration disabled, cannot test CNXID renewal.\n"); 430 | } 431 | } 432 | else { 433 | fprintf(stdout, "Renew CNXID failed, error: %x.\n", ret); 434 | if (F_log != stdout && F_log != stderr && F_log != NULL) 435 | { 436 | fprintf(F_log, "Create Probe failed, error: %x.\n", ret); 437 | } 438 | } 439 | } 440 | else { 441 | fprintf(stdout, "Switching to new CNXID.\n"); 442 | if (F_log != stdout && F_log != stderr && F_log != NULL) 443 | { 444 | fprintf(F_log, "Switching to new CNXID.\n"); 445 | } 446 | } 447 | } 448 | else { 449 | ret = picoquic_create_probe(cnx, server_address, client_address); 450 | if (ret != 0) { 451 | if (ret == PICOQUIC_ERROR_MIGRATION_DISABLED) { 452 | fprintf(stdout, "Migration disabled, will test NAT rebinding support.\n"); 453 | if (F_log != stdout && F_log != stderr && F_log != NULL) 454 | { 455 | fprintf(F_log, "Will test NAT rebinding support.\n"); 456 | } 457 | 458 | ret = 0; 459 | } 460 | else { 461 | fprintf(stdout, "Create Probe failed, error: %x.\n", ret); 462 | if (F_log != stdout && F_log != stderr && F_log != NULL) 463 | { 464 | fprintf(F_log, "Create Probe failed, error: %x.\n", ret); 465 | } 466 | } 467 | } 468 | else { 469 | *address_updated = 1; 470 | fprintf(stdout, "Switch to new port, sending probe.\n"); 471 | if (F_log != stdout && F_log != stderr && F_log != NULL) 472 | { 473 | fprintf(F_log, "Switch to new port, sending probe.\n"); 474 | } 475 | } 476 | } 477 | } 478 | 479 | return ret; 480 | } 481 | 482 | /* Quic Client */ 483 | int quic_client(const char* ip_address_text, int server_port, 484 | const char * sni, const char * esni_rr_file, 485 | const char * alpn, const char * root_crt, 486 | uint32_t proposed_version, int force_zero_share, int force_migration, 487 | int nb_packets_before_key_update, int mtu_max, FILE* F_log, char const* bin_file, 488 | int client_cnx_id_length, char const * client_scenario_text, char const * cc_log_dir, 489 | int no_disk, int use_long_log, picoquic_congestion_algorithm_t const* cc_algorithm, 490 | int large_client_hello) 491 | { 492 | /* Start: start the QUIC process with cert and key files */ 493 | int ret = 0; 494 | picoquic_quic_t* qclient = NULL; 495 | picoquic_cnx_t* cnx_client = NULL; 496 | picoquic_demo_callback_ctx_t callback_ctx; 497 | SOCKET_TYPE fd = INVALID_SOCKET; 498 | struct sockaddr_storage server_address; 499 | struct sockaddr_storage client_address; 500 | struct sockaddr_storage packet_from; 501 | struct sockaddr_storage packet_to; 502 | unsigned long if_index_to; 503 | socklen_t from_length; 504 | socklen_t to_length; 505 | int server_addr_length = 0; 506 | uint8_t buffer[1536]; 507 | uint8_t send_buffer[1536]; 508 | size_t send_length = 0; 509 | uint64_t key_update_done = 0; 510 | int bytes_recv; 511 | int bytes_sent; 512 | uint64_t current_time = 0; 513 | int client_ready_loop = 0; 514 | int client_receive_loop = 0; 515 | int established = 0; 516 | int is_name = 0; 517 | int migration_started = 0; 518 | int address_updated = 0; 519 | int64_t delay_max = 10000000; 520 | int64_t delta_t = 0; 521 | int notified_ready = 0; 522 | int zero_rtt_available = 0; 523 | size_t client_sc_nb = 0; 524 | picoquic_demo_stream_desc_t * client_sc = NULL; 525 | int is_siduck = 0; 526 | siduck_ctx_t* siduck_ctx = NULL; 527 | char const* saved_alpn = NULL; 528 | 529 | FILE *fd_tho; 530 | 531 | fd_tho = fopen("Throughput.txt", "a"); 532 | 533 | if (alpn != NULL && strcmp(alpn, "siduck") == 0) { 534 | /* Set a siduck client */ 535 | is_siduck = 1; 536 | siduck_ctx = siduck_create_ctx(stdout); 537 | if (siduck_ctx == NULL) { 538 | fprintf(stdout, "Could not get ready to quack\n"); 539 | return -1; 540 | } 541 | fprintf(stdout, "Getting ready to quack\n"); 542 | } 543 | else { 544 | 545 | if (no_disk) { 546 | fprintf(stdout, "Files not saved to disk (-D, no_disk)\n"); 547 | } 548 | 549 | if (client_scenario_text == NULL) { 550 | client_scenario_text = test_scenario_default; 551 | } 552 | 553 | fprintf(stdout, "Testing scenario: <%s>\n", client_scenario_text); 554 | ret = demo_client_parse_scenario_desc(client_scenario_text, &client_sc_nb, &client_sc); 555 | if (ret != 0) { 556 | fprintf(stdout, "Cannot parse the specified scenario.\n"); 557 | return -1; 558 | } 559 | else { 560 | ret = picoquic_demo_client_initialize_context(&callback_ctx, client_sc, client_sc_nb, alpn, no_disk, 0); 561 | } 562 | } 563 | 564 | if (ret == 0) { 565 | ret = picoquic_get_server_address(ip_address_text, server_port, &server_address, &server_addr_length, &is_name); 566 | if (sni == NULL && is_name != 0) { 567 | sni = ip_address_text; 568 | } 569 | } 570 | 571 | /* Open a UDP socket */ 572 | 573 | if (ret == 0) { 574 | fd = picoquic_open_client_socket(server_address.ss_family); 575 | if (fd == INVALID_SOCKET) { 576 | ret = -1; 577 | } 578 | } 579 | 580 | /* Create QUIC context */ 581 | current_time = picoquic_current_time(); 582 | callback_ctx.last_interaction_time = current_time; 583 | 584 | if (ret == 0) { 585 | qclient = picoquic_create(8, NULL, NULL, root_crt, alpn, NULL, NULL, NULL, NULL, NULL, current_time, NULL, ticket_store_filename, NULL, 0); 586 | 587 | if (qclient == NULL) { 588 | ret = -1; 589 | } else { 590 | if (cc_algorithm == NULL) { 591 | cc_algorithm = picoquic_bbr_algorithm; 592 | } 593 | picoquic_set_default_congestion_algorithm(qclient, cc_algorithm); 594 | 595 | if (picoquic_load_tokens(&qclient->p_first_token, current_time, token_store_filename) != 0) { 596 | fprintf(stderr, "Could not load tokens from <%s>.\n", token_store_filename); 597 | } 598 | 599 | if (force_zero_share) { 600 | qclient->flags |= picoquic_context_client_zero_share; 601 | } 602 | qclient->mtu_max = mtu_max; 603 | 604 | (void)picoquic_set_default_connection_id_length(qclient, (uint8_t)client_cnx_id_length); 605 | 606 | PICOQUIC_SET_LOG(qclient, F_log); 607 | if (use_long_log) { 608 | qclient->use_long_log = 1; 609 | } 610 | 611 | picoquic_set_key_log_file_from_env(qclient); 612 | 613 | if (cc_log_dir != NULL) { 614 | picoquic_set_cc_log(qclient, cc_log_dir); 615 | } 616 | 617 | if (bin_file != NULL) { 618 | picoquic_set_binlog(qclient, bin_file); 619 | } 620 | 621 | if (sni == NULL) { 622 | /* Standard verifier would crash */ 623 | fprintf(stdout, "No server name specified, certificate will not be verified.\n"); 624 | if (F_log != stdout && F_log != stderr && F_log != NULL) 625 | { 626 | fprintf(F_log, "No server name specified, certificate will not be verified.\n"); 627 | } 628 | picoquic_set_null_verifier(qclient); 629 | } 630 | else if (root_crt == NULL) { 631 | /* Standard verifier would crash */ 632 | fprintf(stdout, "No root crt list specified, certificate will not be verified.\n"); 633 | if (F_log != stdout && F_log != stderr && F_log != NULL) 634 | { 635 | fprintf(F_log, "No root crt list specified, certificate will not be verified.\n"); 636 | } 637 | picoquic_set_null_verifier(qclient); 638 | } 639 | 640 | } 641 | } 642 | 643 | /* Create the client connection */ 644 | if (ret == 0) { 645 | /* Create a client connection */ 646 | cnx_client = picoquic_create_cnx(qclient, picoquic_null_connection_id, picoquic_null_connection_id, 647 | (struct sockaddr*)&server_address, current_time, 648 | proposed_version, sni, alpn, 1); 649 | 650 | if (cnx_client == NULL) { 651 | ret = -1; 652 | } 653 | else { 654 | if (is_siduck) { 655 | picoquic_set_callback(cnx_client, siduck_callback, siduck_ctx); 656 | cnx_client->local_parameters.max_datagram_size = 128; 657 | } 658 | else { 659 | picoquic_set_callback(cnx_client, picoquic_demo_client_callback, &callback_ctx); 660 | 661 | if (cnx_client->alpn == NULL) { 662 | picoquic_demo_client_set_alpn_from_tickets(cnx_client, &callback_ctx, current_time); 663 | if (cnx_client->alpn != NULL) { 664 | fprintf(stdout, "Set ALPN to %s based on stored ticket\n", cnx_client->alpn); 665 | } 666 | } 667 | 668 | /* Requires TP grease, for interop tests */ 669 | cnx_client->grease_transport_parameters = 1; 670 | cnx_client->local_parameters.enable_one_way_delay = 1; 671 | 672 | if (callback_ctx.tp != NULL) { 673 | picoquic_set_transport_parameters(cnx_client, callback_ctx.tp); 674 | } 675 | } 676 | 677 | if (large_client_hello) { 678 | cnx_client->test_large_chello = 1; 679 | } 680 | 681 | if (esni_rr_file != NULL) { 682 | ret = picoquic_esni_client_from_file(cnx_client, esni_rr_file); 683 | } 684 | 685 | if (ret == 0) { 686 | ret = picoquic_start_client_cnx(cnx_client); 687 | } 688 | 689 | if (ret == 0 && !is_siduck) { 690 | if (picoquic_is_0rtt_available(cnx_client) && (proposed_version & 0x0a0a0a0a) != 0x0a0a0a0a) { 691 | zero_rtt_available = 1; 692 | 693 | /* Queue a simple frame to perform 0-RTT test */ 694 | /* Start the download scenario */ 695 | 696 | ret = picoquic_demo_client_start_streams(cnx_client, &callback_ctx, PICOQUIC_DEMO_STREAM_ID_INITIAL); 697 | } 698 | } 699 | 700 | if (ret == 0) { 701 | ret = picoquic_prepare_packet(cnx_client, current_time, 702 | send_buffer, sizeof(send_buffer), &send_length, NULL, NULL, NULL, NULL); 703 | 704 | if (ret == 0 && send_length > 0) { 705 | bytes_sent = sendto(fd, (const char*)send_buffer, (int)send_length, 0, 706 | (struct sockaddr*)&server_address, server_addr_length); 707 | 708 | if (F_log != NULL) { 709 | if (bytes_sent <= 0) 710 | { 711 | fprintf(F_log, "Cannot send first packet to server, returns %d\n", bytes_sent); 712 | ret = -1; 713 | } 714 | } 715 | } 716 | } 717 | } 718 | } 719 | 720 | /* Wait for packets */ 721 | while (ret == 0 && picoquic_get_cnx_state(cnx_client) != picoquic_state_disconnected) { 722 | unsigned char received_ecn; 723 | 724 | from_length = to_length = sizeof(struct sockaddr_storage); 725 | 726 | bytes_recv = picoquic_select(&fd, 1, &packet_from, &from_length, 727 | &packet_to, &to_length, &if_index_to, &received_ecn, 728 | buffer, sizeof(buffer), 729 | delta_t, 730 | ¤t_time); 731 | 732 | if (bytes_recv != 0 && F_log != NULL && 733 | (cnx_client == NULL || cnx_client->pkt_ctx[picoquic_packet_context_application].send_sequence < PICOQUIC_LOG_PACKET_MAX_SEQUENCE || qclient->use_long_log)){ 734 | fprintf(F_log, "Select returns %d, from length %d\n", bytes_recv, from_length); 735 | } 736 | 737 | if (bytes_recv != 0 && to_length != 0) { 738 | /* Keeping track of the addresses and ports, as we 739 | * need them to verify the migration behavior */ 740 | if (!address_updated) { 741 | struct sockaddr_storage local_address; 742 | if (picoquic_get_local_address(fd, &local_address) != 0) { 743 | memset(&local_address, 0, sizeof(struct sockaddr_storage)); 744 | fprintf(stderr, "Could not read local address.\n"); 745 | } 746 | 747 | address_updated = 1; 748 | picoquic_store_addr(&client_address, (struct sockaddr *)&packet_to); 749 | if (client_address.ss_family == AF_INET) { 750 | ((struct sockaddr_in *)&client_address)->sin_port = 751 | ((struct sockaddr_in *)&local_address)->sin_port; 752 | fprintf(stdout, "IPv4 port: %d.\n", ((struct sockaddr_in*)& client_address)->sin_port); 753 | } 754 | else { 755 | ((struct sockaddr_in6 *)&client_address)->sin6_port = 756 | ((struct sockaddr_in6 *)&local_address)->sin6_port; 757 | fprintf(stdout, "IPv6 port: %d.\n", ((struct sockaddr_in6*)& client_address)->sin6_port); 758 | } 759 | if (F_log != stdout && F_log != stderr && F_log != NULL) 760 | { 761 | fprintf(F_log, "Client port (AF=%d): %d.\n", 762 | client_address.ss_family, 763 | (client_address.ss_family == AF_INET)? 764 | ((struct sockaddr_in*) & client_address)->sin_port: 765 | ((struct sockaddr_in6*) & client_address)->sin6_port 766 | ); 767 | } 768 | if (F_log != NULL) { 769 | fprintf(F_log, "Local address updated\n"); 770 | } 771 | } 772 | 773 | if (client_address.ss_family == AF_INET) { 774 | ((struct sockaddr_in *)&packet_to)->sin_port = 775 | ((struct sockaddr_in *)&client_address)->sin_port; 776 | } 777 | else { 778 | ((struct sockaddr_in6 *)&packet_to)->sin6_port = 779 | ((struct sockaddr_in6 *)&client_address)->sin6_port; 780 | } 781 | } 782 | 783 | if (bytes_recv < 0) { 784 | ret = -1; 785 | } else { 786 | if (bytes_recv > 0) { 787 | /* Submit the packet to the client */ 788 | ret = picoquic_incoming_packet(qclient, buffer, 789 | (size_t)bytes_recv, (struct sockaddr*)&packet_from, 790 | (struct sockaddr*)&packet_to, if_index_to, received_ecn, 791 | current_time); 792 | client_receive_loop++; 793 | 794 | if (picoquic_get_cnx_state(cnx_client) == picoquic_state_client_almost_ready && notified_ready == 0) { 795 | if (picoquic_tls_is_psk_handshake(cnx_client)) { 796 | fprintf(stdout, "The session was properly resumed!\n"); 797 | if (F_log != stdout && F_log != stderr && F_log != NULL) { 798 | fprintf(F_log, "The session was properly resumed!\n"); 799 | } 800 | } 801 | 802 | if (cnx_client->zero_rtt_data_accepted) { 803 | fprintf(stdout, "Zero RTT data is accepted!\n"); 804 | } 805 | 806 | if (cnx_client->alpn != NULL) { 807 | fprintf(stdout, "Negotiated ALPN: %s\n", cnx_client->alpn); 808 | saved_alpn = picoquic_string_duplicate(cnx_client->alpn); 809 | } 810 | fprintf(stdout, "Almost ready!\n\n"); 811 | notified_ready = 1; 812 | } 813 | 814 | if (ret != 0 && F_log != NULL) { 815 | picoquic_log_error_packet(F_log, buffer, (size_t)bytes_recv, ret); 816 | } 817 | 818 | delta_t = 0; 819 | } 820 | 821 | /* In normal circumstances, the code waits until all packets in the receive 822 | * queue have been processed before sending new packets. However, if the server 823 | * is sending lots and lots of data this can lead to the client not getting 824 | * the occasion to send acknowledgements. The server will start retransmissions, 825 | * and may eventually drop the connection for lack of acks. So we limit 826 | * the number of packets that can be received before sending responses. */ 827 | 828 | if (bytes_recv == 0 || (ret == 0 && client_receive_loop > PICOQUIC_DEMO_CLIENT_MAX_RECEIVE_BATCH)) { 829 | client_receive_loop = 0; 830 | 831 | if (ret == 0 && (picoquic_get_cnx_state(cnx_client) == picoquic_state_ready || 832 | picoquic_get_cnx_state(cnx_client) == picoquic_state_client_ready_start)) { 833 | if (established == 0) { 834 | if (F_log != NULL) { 835 | picoquic_log_transport_ids(F_log, cnx_client, 0); 836 | } 837 | printf("Connection established. Version = %x, I-CID: %llx\n", 838 | picoquic_supported_versions[cnx_client->version_index].version, 839 | (unsigned long long)picoquic_val64_connection_id(picoquic_get_logging_cnxid(cnx_client))); 840 | established = 1; 841 | 842 | if (zero_rtt_available == 0 && !is_siduck) { 843 | /* Start the download scenario */ 844 | 845 | picoquic_demo_client_start_streams(cnx_client, &callback_ctx, PICOQUIC_DEMO_STREAM_ID_INITIAL); 846 | } 847 | } 848 | 849 | client_ready_loop++; 850 | 851 | if (force_migration && migration_started == 0 && address_updated && 852 | picoquic_get_cnx_state(cnx_client) == picoquic_state_ready && cnx_client->probe_first == NULL && 853 | (cnx_client->cnxid_stash_first != NULL || force_migration == 1) 854 | && picoquic_get_cnx_state(cnx_client) == picoquic_state_ready) { 855 | if (force_migration == 3 && picoquic_compare_addr( 856 | (struct sockaddr*) & server_address, (struct sockaddr*) & cnx_client->path[0]->peer_addr) != 0) { 857 | fprintf(stdout, "Migration already accumplished to server preferred address!\n"); 858 | migration_started = -1; 859 | } 860 | else { 861 | int mig_ret = quic_client_migrate(cnx_client, &fd, NULL, (struct sockaddr*) & client_address, &address_updated, force_migration, F_log); 862 | 863 | migration_started = 1; 864 | 865 | if (mig_ret != 0) { 866 | fprintf(stdout, "Will not test migration.\n"); 867 | migration_started = -1; 868 | } 869 | } 870 | } 871 | 872 | if (nb_packets_before_key_update > 0 && 873 | !key_update_done && 874 | cnx_client->pkt_ctx[picoquic_packet_context_application].first_sack_item.end_of_sack_range > (uint64_t)nb_packets_before_key_update) { 875 | int key_rot_ret = picoquic_start_key_rotation(cnx_client); 876 | if (key_rot_ret != 0) { 877 | fprintf(stdout, "Will not test key rotation.\n"); 878 | key_update_done = (uint64_t)-1; 879 | } 880 | else { 881 | fprintf(stdout, "Key rotation started.\n"); 882 | key_update_done = 1; 883 | } 884 | } 885 | 886 | if (bytes_recv == 0 || client_ready_loop > 4) { 887 | if (!is_siduck && callback_ctx.nb_open_streams == 0) { 888 | if (cnx_client->nb_zero_rtt_sent != 0) { 889 | fprintf(stdout, "Out of %d zero RTT packets, %d were acked by the server.\n", 890 | cnx_client->nb_zero_rtt_sent, cnx_client->nb_zero_rtt_acked); 891 | if (F_log != stdout && F_log != stderr && F_log != NULL) 892 | { 893 | fprintf(F_log, "Out of %d zero RTT packets, %d were acked by the server.\n", 894 | cnx_client->nb_zero_rtt_sent, cnx_client->nb_zero_rtt_acked); 895 | } 896 | } 897 | fprintf(stdout, "All done, Closing the connection.\n"); 898 | if (F_log != stdout && F_log != stderr && F_log != NULL) 899 | { 900 | fprintf(F_log, "All done, Closing the connection.\n"); 901 | } 902 | if (picoquic_get_data_received(cnx_client) > 0) { 903 | double duration_usec = (double)(current_time - picoquic_get_cnx_start_time(cnx_client)); 904 | 905 | if (duration_usec > 0) { 906 | double receive_rate_mbps = 8.0*((double)picoquic_get_data_received(cnx_client)) / duration_usec; 907 | fprintf(stdout, "Received %llu bytes in %f seconds, %f Mbps.\n", 908 | (unsigned long long)picoquic_get_data_received(cnx_client), 909 | duration_usec/1000000.0, receive_rate_mbps); 910 | fprintf(fd_tho, "%f\n", receive_rate_mbps); 911 | if (F_log != stdout && F_log != stderr && F_log != NULL) 912 | { 913 | fprintf(F_log, "Received %llu bytes in %f seconds, %f Mbps.\n", 914 | (unsigned long long)picoquic_get_data_received(cnx_client), 915 | duration_usec / 1000000.0, receive_rate_mbps); 916 | } 917 | } 918 | } 919 | 920 | ret = picoquic_close(cnx_client, 0); 921 | } 922 | else if ( 923 | current_time > callback_ctx.last_interaction_time && current_time - callback_ctx.last_interaction_time > 10000000ull 924 | && picoquic_is_cnx_backlog_empty(cnx_client)) { 925 | fprintf(stdout, "No progress for 10 seconds. Closing. \n"); 926 | if (F_log != stdout && F_log != stderr && F_log != NULL) 927 | { 928 | fprintf(F_log, "No progress for 10 seconds. Closing. \n"); 929 | } 930 | ret = picoquic_close(cnx_client, 0); 931 | } 932 | } 933 | } 934 | 935 | if (ret == 0) { 936 | struct sockaddr_storage x_to; 937 | int x_to_length; 938 | struct sockaddr_storage x_from; 939 | int x_from_length; 940 | 941 | send_length = PICOQUIC_MAX_PACKET_SIZE; 942 | 943 | current_time = picoquic_get_quic_time(qclient); 944 | 945 | ret = picoquic_prepare_packet(cnx_client, current_time, 946 | send_buffer, sizeof(send_buffer), &send_length, &x_to, &x_to_length, &x_from, &x_from_length); 947 | 948 | if (migration_started && force_migration == 3 && send_length > 0 && address_updated) { 949 | if (picoquic_compare_addr((struct sockaddr*) & x_from, (struct sockaddr*) & client_address) != 0) { 950 | if (F_log != NULL) { 951 | fprintf(F_log, "Dropping packet sent from wrong address, port: %d\n", 952 | (client_address.ss_family == AF_INET) ? 953 | ((struct sockaddr_in*) & x_from)->sin_port : 954 | ((struct sockaddr_in6*) & x_from)->sin6_port); 955 | } 956 | send_length = 0; 957 | } 958 | } 959 | 960 | if (ret == 0 && send_length > 0) { 961 | bytes_sent = sendto(fd, (const char*)send_buffer, (int)send_length, 0, 962 | (struct sockaddr*)&x_to, x_to_length); 963 | 964 | if (bytes_sent <= 0) 965 | { 966 | fprintf(stdout, "Cannot send packet to server, returns %d\n", bytes_sent); 967 | 968 | if (F_log != stdout && F_log != stderr && F_log != NULL) 969 | { 970 | fprintf(F_log, "Cannot send packet to server, returns %d\n", bytes_sent); 971 | } 972 | } 973 | } 974 | } 975 | 976 | delta_t = picoquic_get_next_wake_delay(qclient, current_time, delay_max); 977 | 978 | if (delta_t > 10000 && (is_siduck || callback_ctx.nb_open_streams == 0) && 979 | picoquic_is_cnx_backlog_empty(cnx_client)) { 980 | delta_t = 10000; 981 | } 982 | } 983 | } 984 | } 985 | 986 | /* Clean up */ 987 | if (is_siduck) { 988 | free(siduck_ctx); 989 | } else { 990 | picoquic_demo_client_delete_context(&callback_ctx); 991 | } 992 | 993 | if (qclient != NULL) { 994 | uint8_t* ticket; 995 | uint16_t ticket_length; 996 | 997 | if (sni != NULL && saved_alpn != NULL && 0 == picoquic_get_ticket(qclient->p_first_ticket, current_time, sni, (uint16_t)strlen(sni), saved_alpn, 998 | (uint16_t)strlen(saved_alpn), &ticket, &ticket_length, NULL, 0)) { 999 | FILE * F = (F_log != NULL) ? F_log : stdout; 1000 | fprintf(F, "Received ticket from %s (%s):\n", sni, saved_alpn); 1001 | picoquic_log_picotls_ticket(F, picoquic_null_connection_id, ticket, ticket_length); 1002 | } 1003 | 1004 | if (picoquic_save_tickets(qclient->p_first_ticket, current_time, ticket_store_filename) != 0) { 1005 | fprintf(stderr, "Could not store the saved session tickets.\n"); 1006 | } 1007 | 1008 | 1009 | if (picoquic_save_tokens(qclient->p_first_token, current_time, token_store_filename) != 0) { 1010 | fprintf(stderr, "Could not save tokens to <%s>.\n", token_store_filename); 1011 | } 1012 | 1013 | picoquic_free(qclient); 1014 | } 1015 | 1016 | if (fd != INVALID_SOCKET) { 1017 | SOCKET_CLOSE(fd); 1018 | } 1019 | 1020 | if (saved_alpn != NULL) { 1021 | free((void *)saved_alpn); 1022 | saved_alpn = NULL; 1023 | } 1024 | 1025 | if (client_scenario_text != NULL && client_sc != NULL) { 1026 | demo_client_delete_scenario_desc(client_sc_nb, client_sc); 1027 | client_sc = NULL; 1028 | } 1029 | fclose(fd_tho); 1030 | return ret; 1031 | } 1032 | 1033 | uint32_t parse_target_version(char const* v_arg) 1034 | { 1035 | /* Expect the version to be encoded in base 16 */ 1036 | uint32_t v = 0; 1037 | char const* x = v_arg; 1038 | 1039 | while (*x != 0) { 1040 | int c = *x; 1041 | 1042 | if (c >= '0' && c <= '9') { 1043 | c -= '0'; 1044 | } else if (c >= 'a' && c <= 'f') { 1045 | c -= 'a'; 1046 | c += 10; 1047 | } else if (c >= 'A' && c <= 'F') { 1048 | c -= 'A'; 1049 | c += 10; 1050 | } else { 1051 | v = 0; 1052 | break; 1053 | } 1054 | v *= 16; 1055 | v += c; 1056 | x++; 1057 | } 1058 | 1059 | return v; 1060 | } 1061 | 1062 | void usage() 1063 | { 1064 | fprintf(stderr, "PicoQUIC demo client and server\n"); 1065 | fprintf(stderr, "Usage: picoquicdemo [server_name [port [scenario]]] \n"); 1066 | fprintf(stderr, " For the client mode, specify server_name and port.\n"); 1067 | fprintf(stderr, " For the server mode, use -p to specify the port.\n"); 1068 | fprintf(stderr, "Options:\n"); 1069 | fprintf(stderr, " -c file cert file (default: %s)\n", SERVER_CERT_FILE); 1070 | fprintf(stderr, " -e if Send on interface (default: -1)\n"); 1071 | fprintf(stderr, " -1: receiving interface\n"); 1072 | fprintf(stderr, " 0: routing lookup\n"); 1073 | fprintf(stderr, " n: ifindex\n"); 1074 | fprintf(stderr, " -f migration_mode Force client to migrate to start migration:\n"); 1075 | fprintf(stderr, " -f 1 test NAT rebinding,\n"); 1076 | fprintf(stderr, " -f 2 test CNXID renewal,\n"); 1077 | fprintf(stderr, " -f 3 test migration to new address.\n"); 1078 | fprintf(stderr, " -h This help message\n"); 1079 | fprintf(stderr, " -i Connection ID modification: (src & ~mask) || val\n"); 1080 | fprintf(stderr, " Implies unconditional server cnx_id xmit\n"); 1081 | fprintf(stderr, " where is int:\n"); 1082 | fprintf(stderr, " 0: picoquic_cnx_id_random\n"); 1083 | fprintf(stderr, " 1: picoquic_cnx_id_remote (client)\n"); 1084 | fprintf(stderr, " 2: same as 0, plus encryption of unmasked data\n"); 1085 | fprintf(stderr, " 3: same as 0, plus encryption of all data\n"); 1086 | fprintf(stderr, " val and mask must be hex strings of same length, 4 to 18\n"); 1087 | fprintf(stderr, " -k file key file (default: %s)\n", SERVER_KEY_FILE); 1088 | fprintf(stderr, " -K file ESNI private key file (default: don't use ESNI)\n"); 1089 | fprintf(stderr, " -E file ESNI RR file (default: don't use ESNI)\n"); 1090 | fprintf(stderr, " -w folder Folder containing web pages served by server\n"); 1091 | fprintf(stderr, " -l file Log file, Log to stdout if file = \"n\". No logging if absent.\n"); 1092 | fprintf(stderr, " -L Log all packets. If absent, log stops after 100 packets.\n"); 1093 | fprintf(stderr, " -p port server port (default: %d)\n", default_server_port); 1094 | fprintf(stderr, " -m mtu_max Largest mtu value that can be tried for discovery\n"); 1095 | fprintf(stderr, " -n sni sni (default: server name)\n"); 1096 | fprintf(stderr, " -a alpn alpn (default function of version)\n"); 1097 | fprintf(stderr, " -r Do Reset Request\n"); 1098 | fprintf(stderr, " -s <64b 64b> Reset seed\n"); 1099 | fprintf(stderr, " -t file root trust file\n"); 1100 | fprintf(stderr, " -u nb trigger key update after receiving packets on client\n"); 1101 | fprintf(stderr, " -v version Version proposed by client, e.g. -v ff000012\n"); 1102 | fprintf(stderr, " -z Set TLS zero share behavior on client, to force HRR.\n"); 1103 | fprintf(stderr, " -1 Once: close the server after processing 1 connection.\n"); 1104 | fprintf(stderr, " -S solution_dir Set the path to the source files to find the default files\n"); 1105 | fprintf(stderr, " -I length Length of CNX_ID used by the client, default=8\n"); 1106 | fprintf(stderr, " -g cc_log_dir log congestion control traces in specified dir\n"); 1107 | fprintf(stderr, " -G cc_algorithm Use the specified congestion control algorithm:\n"); 1108 | fprintf(stderr, " reno, cubic or fast. Defaults to cubic.\n"); 1109 | fprintf(stderr, " -D no disk: do not save received files on disk.\n"); 1110 | fprintf(stderr, " -Q send a large client hello in order to test post quantum\n"); 1111 | fprintf(stderr, " readiness.\n"); 1112 | 1113 | fprintf(stderr, "\nThe scenario argument specifies the set of files that should be retrieved,\n"); 1114 | fprintf(stderr, "and their order. The syntax is:\n"); 1115 | fprintf(stderr, " *{[':'[':'[:]]]path;}\n"); 1116 | fprintf(stderr, "where:\n"); 1117 | fprintf(stderr, " : The numeric ID of the QUIC stream, e.g. 4. By default, the\n"); 1118 | fprintf(stderr, " next stream in the logical QUIC order, 0, 4, 8, etc."); 1119 | fprintf(stderr, " : The numeric ID of the previous stream. The GET command will\n"); 1120 | fprintf(stderr, " be issued after that stream's transfer finishes. By default,\n"); 1121 | fprintf(stderr, " previous stream in this scenario.\n"); 1122 | fprintf(stderr, " : Whether the received file should be written to disc as\n"); 1123 | fprintf(stderr, " binary(b) or text(t). Defaults to text.\n"); 1124 | fprintf(stderr, " : The name of the document that should be retrieved\n"); 1125 | fprintf(stderr, "If no scenario is specified, the client executes the default scenario.\n"); 1126 | 1127 | exit(1); 1128 | } 1129 | 1130 | int main(int argc, char** argv) 1131 | { 1132 | const char * solution_dir = NULL; 1133 | const char * server_name = default_server_name; 1134 | const char * server_cert_file = NULL; 1135 | const char * server_key_file = NULL; 1136 | const char * esni_key_file = NULL; 1137 | const char * esni_rr_file = NULL; 1138 | const char * log_file = NULL; 1139 | const char * bin_file = NULL; 1140 | const char * sni = NULL; 1141 | const char * alpn = NULL; 1142 | const char * cc_log_dir = NULL; 1143 | const char* www_dir = NULL; 1144 | picoquic_congestion_algorithm_t const* cc_algorithm = NULL; 1145 | int server_port = default_server_port; 1146 | const char* root_trust_file = NULL; 1147 | uint32_t proposed_version = 0; 1148 | int is_client = 0; 1149 | int just_once = 0; 1150 | int do_hrr = 0; 1151 | int force_zero_share = 0; 1152 | int force_migration = 0; 1153 | int large_client_hello = 0; 1154 | int nb_packets_before_update = 0; 1155 | int client_cnx_id_length = 8; 1156 | int no_disk = 0; 1157 | int use_long_log = 0; 1158 | picoquic_connection_id_callback_ctx_t * cnx_id_cbdata = NULL; 1159 | uint64_t* reset_seed = NULL; 1160 | uint64_t reset_seed_x[2]; 1161 | int dest_if = -1; 1162 | int mtu_max = 0; 1163 | char default_server_cert_file[512]; 1164 | char default_server_key_file[512]; 1165 | char * client_scenario = NULL; 1166 | FILE* F_log = NULL; 1167 | int ret = 0; 1168 | 1169 | #ifdef _WINDOWS 1170 | WSADATA wsaData = { 0 }; 1171 | (void)WSA_START(MAKEWORD(2, 2), &wsaData); 1172 | #endif 1173 | 1174 | /* Get the parameters */ 1175 | int opt; 1176 | while ((opt = getopt(argc, argv, "c:k:K:p:u:v:w:f:i:s:e:E:l:b:m:n:a:t:S:I:g:G:1rhzDLQ")) != -1) { 1177 | switch (opt) { 1178 | case 'c': 1179 | server_cert_file = optarg; 1180 | break; 1181 | case 'k': 1182 | server_key_file = optarg; 1183 | break; 1184 | case 'K': 1185 | esni_key_file = optarg; 1186 | break; 1187 | case 'p': 1188 | if ((server_port = atoi(optarg)) <= 0) { 1189 | fprintf(stderr, "Invalid port: %s\n", optarg); 1190 | usage(); 1191 | } 1192 | break; 1193 | case 'u': 1194 | if ((nb_packets_before_update = atoi(optarg)) <= 0) { 1195 | fprintf(stderr, "Invalid number of packets: %s\n", optarg); 1196 | usage(); 1197 | } 1198 | break; 1199 | case 'v': 1200 | if ((proposed_version = parse_target_version(optarg)) <= 0) { 1201 | fprintf(stderr, "Invalid version: %s\n", optarg); 1202 | usage(); 1203 | } 1204 | break; 1205 | case 'w': 1206 | www_dir = optarg; 1207 | break; 1208 | case '1': 1209 | just_once = 1; 1210 | break; 1211 | case 'r': 1212 | do_hrr = 1; 1213 | break; 1214 | case 's': 1215 | if (optind + 1 > argc) { 1216 | fprintf(stderr, "option requires more arguments -- s\n"); 1217 | usage(); 1218 | } 1219 | reset_seed = reset_seed_x; /* replacing the original alloca, which is not supported in Windows or BSD */ 1220 | reset_seed[1] = strtoul(optarg, NULL, 0); 1221 | reset_seed[0] = strtoul(argv[optind++], NULL, 0); 1222 | break; 1223 | case 'S': 1224 | solution_dir = optarg; 1225 | break; 1226 | case 'g': 1227 | cc_log_dir = optarg; 1228 | break; 1229 | case 'G': 1230 | cc_algorithm = picoquic_get_congestion_algorithm(optarg); 1231 | if (cc_algorithm == NULL) { 1232 | fprintf(stderr, "Unsupported congestion control algorithm: %s\n", optarg); 1233 | usage(); 1234 | } 1235 | break; 1236 | case 'e': 1237 | dest_if = atoi(optarg); 1238 | break; 1239 | case 'E': 1240 | esni_rr_file = optarg; 1241 | break; 1242 | case 'i': 1243 | if (optind + 2 > argc) { 1244 | fprintf(stderr, "option requires more arguments -- i\n"); 1245 | usage(); 1246 | } 1247 | cnx_id_cbdata = picoquic_connection_id_callback_create_ctx(optarg, argv[optind], argv[optind + 1]); 1248 | if (cnx_id_cbdata == NULL) { 1249 | fprintf(stderr, "could not create callback context (%s, %s, %s)\n", optarg, argv[optind], argv[optind + 1]); 1250 | usage(); 1251 | } 1252 | optind += 2; 1253 | break; 1254 | case 'l': 1255 | log_file = optarg; 1256 | break; 1257 | case 'b': 1258 | bin_file = optarg; 1259 | break; 1260 | case 'L': 1261 | use_long_log = 1; 1262 | break; 1263 | case 'm': 1264 | mtu_max = atoi(optarg); 1265 | if (mtu_max <= 0 || mtu_max > PICOQUIC_MAX_PACKET_SIZE) { 1266 | fprintf(stderr, "Invalid max mtu: %s\n", optarg); 1267 | usage(); 1268 | } 1269 | break; 1270 | case 'n': 1271 | sni = optarg; 1272 | break; 1273 | case 'a': 1274 | alpn = optarg; 1275 | break; 1276 | case 't': 1277 | root_trust_file = optarg; 1278 | break; 1279 | case 'z': 1280 | force_zero_share = 1; 1281 | break; 1282 | case 'f': 1283 | force_migration = atoi(optarg); 1284 | if (force_migration <= 0 || force_migration > 3) { 1285 | fprintf(stderr, "Invalid migration mode: %s\n", optarg); 1286 | usage(); 1287 | } 1288 | break; 1289 | case 'I': 1290 | client_cnx_id_length = atoi(optarg); 1291 | if (client_cnx_id_length < 0 || client_cnx_id_length > PICOQUIC_CONNECTION_ID_MAX_SIZE){ 1292 | fprintf(stderr, "Invalid connection id length: %s\n", optarg); 1293 | usage(); 1294 | } 1295 | break; 1296 | case 'D': 1297 | no_disk = 1; 1298 | break; 1299 | case 'Q': 1300 | large_client_hello = 1; 1301 | break; 1302 | case 'h': 1303 | usage(); 1304 | break; 1305 | default: 1306 | usage(); 1307 | break; 1308 | } 1309 | } 1310 | 1311 | /* Simplified style params */ 1312 | if (optind < argc) { 1313 | server_name = argv[optind++]; 1314 | is_client = 1; 1315 | } 1316 | 1317 | if (optind < argc) { 1318 | if ((server_port = atoi(argv[optind++])) <= 0) { 1319 | fprintf(stderr, "Invalid port: %s\n", optarg); 1320 | usage(); 1321 | } 1322 | } 1323 | 1324 | if (optind < argc) { 1325 | client_scenario = argv[optind++]; 1326 | } 1327 | 1328 | if (optind < argc) { 1329 | usage(); 1330 | } 1331 | 1332 | if (log_file != NULL) { 1333 | if (strcmp(log_file, "-") == 0) { 1334 | F_log = stdout; 1335 | } 1336 | else 1337 | { 1338 | if ((F_log = picoquic_file_open(log_file, "w")) == NULL) { 1339 | fprintf(stderr, "Could not open the log file <%s>\n", log_file); 1340 | } 1341 | } 1342 | } 1343 | 1344 | if (F_log != NULL) { 1345 | debug_printf_push_stream(F_log); 1346 | } 1347 | 1348 | if (is_client == 0) { 1349 | 1350 | if (server_cert_file == NULL && 1351 | picoquic_get_input_path(default_server_cert_file, sizeof(default_server_cert_file), solution_dir, SERVER_CERT_FILE) == 0) { 1352 | server_cert_file = default_server_cert_file; 1353 | } 1354 | 1355 | if (server_key_file == NULL && 1356 | picoquic_get_input_path(default_server_key_file, sizeof(default_server_key_file), solution_dir, SERVER_KEY_FILE) == 0) { 1357 | server_key_file = default_server_key_file; 1358 | } 1359 | 1360 | /* Run as server */ 1361 | printf("Starting Picoquic server (v%s) on port %d, server name = %s, just_once = %d, hrr= %d\n", 1362 | PICOQUIC_VERSION, server_port, server_name, just_once, do_hrr); 1363 | ret = quic_server(server_name, server_port, 1364 | server_cert_file, server_key_file, just_once, do_hrr, 1365 | (cnx_id_cbdata == NULL) ? NULL : picoquic_connection_id_callback, 1366 | (cnx_id_cbdata == NULL) ? NULL : (void*)cnx_id_cbdata, 1367 | (uint8_t*)reset_seed, dest_if, mtu_max, proposed_version, 1368 | esni_key_file, esni_rr_file, 1369 | F_log, bin_file, cc_log_dir, use_long_log, cc_algorithm, www_dir); 1370 | printf("Server exit with code = %d\n", ret); 1371 | } else { 1372 | /* Run as client */ 1373 | printf("Starting Picoquic (v%s) connection to server = %s, port = %d\n", PICOQUIC_VERSION, server_name, server_port); 1374 | ret = quic_client(server_name, server_port, sni, esni_rr_file, alpn, root_trust_file, proposed_version, force_zero_share, 1375 | force_migration, nb_packets_before_update, mtu_max, F_log, bin_file, client_cnx_id_length, client_scenario, 1376 | cc_log_dir, no_disk, use_long_log, cc_algorithm, large_client_hello); 1377 | 1378 | printf("Client exit with code = %d\n", ret); 1379 | } 1380 | 1381 | if (F_log != stdout) { 1382 | (void)picoquic_file_close(F_log); 1383 | } 1384 | 1385 | if (cnx_id_cbdata != NULL) { 1386 | picoquic_connection_id_callback_free_ctx(cnx_id_cbdata); 1387 | } 1388 | } 1389 | -------------------------------------------------------------------------------- /kits/picoquic/server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ip netns exec blue taskset 256 ./picoquicdemo -p 4433 4 | 5 | #default cc is cubic, use -G to change cc algorithms. 6 | 7 | #use the cmd below to fire up perf. 8 | #perf record --call-graph lbr -- ip netns exec blue taskset 256 ./picoquicdemo -p 4433 -------------------------------------------------------------------------------- /kits/quant/README.md: -------------------------------------------------------------------------------- 1 | ### README 2 | 3 | 1. Firstly you need to setup the connection between two interfaces (`ens3f0` `ens3f1` by default); 4 | 2. `TLEM` can be used for this purpose. For example if you want to add 500us latency between the two interfaces, you can use cmds below. More options can be found [here](https://github.com/luigirizzo/netmap/tree/master/apps/tlem). 5 | > ./tlem -D 500us -i netmap:ens4f0 -i netmap:ens4f1 6 | 3. Run `config.sh` and re-compile `quant`; 7 | 4. Copy `server.sh`, `client.sh` and `multi_client.sh` into `extern/quant/Release/bin/` (or `extern/quant/Debug/bin/`); 8 | 5. Run the quant server first by executing `server_netmap.sh`; 9 | 6. Run the quant client by executing `client_netmap.sh`; 10 | 7. Results (Throughput) is extracted to `bps_ins.txt`. -------------------------------------------------------------------------------- /kits/quant/client.c: -------------------------------------------------------------------------------- 1 | // SPDX-License-Identifier: BSD-2-Clause 2 | // 3 | // Copyright (c) 2016-2020, NetApp, Inc. 4 | // All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are met: 8 | // 9 | // 1. Redistributions of source code must retain the above copyright notice, 10 | // this list of conditions and the following disclaimer. 11 | // 12 | // 2. Redistributions in binary form must reproduce the above copyright notice, 13 | // this list of conditions and the following disclaimer in the documentation 14 | // and/or other materials provided with the distribution. 15 | // 16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 19 | // ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 20 | // LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 21 | // CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 22 | // SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 23 | // INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 24 | // CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 25 | // ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 26 | // POSSIBILITY OF SUCH DAMAGE. 27 | 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include 40 | #include 41 | #include // IWYU pragma: no_forward_declare sockaddr 42 | #include 43 | #include 44 | #include 45 | #include 46 | #include 47 | 48 | #ifdef __FreeBSD__ 49 | #include 50 | #endif 51 | 52 | #define klib_unused 53 | 54 | #include 55 | #include 56 | #include 57 | 58 | 59 | #define timespec_to_double(diff) \ 60 | ((double)(diff).tv_sec + (double)(diff).tv_nsec / NS_PER_S) 61 | 62 | 63 | #define bps(bytes, secs) \ 64 | __extension__({ \ 65 | static char _str[32]; \ 66 | const double _bps = ((secs) > 1e-9) ? (double)(bytes)*8 / (secs) : 0; \ 67 | if (_bps > NS_PER_S) \ 68 | snprintf(_str, sizeof(_str), "%.3f Gb/s", _bps / NS_PER_S); \ 69 | else if (_bps > US_PER_S) \ 70 | snprintf(_str, sizeof(_str), "%.3f Mb/s", _bps / US_PER_S); \ 71 | else if (_bps > MS_PER_S) \ 72 | snprintf(_str, sizeof(_str), "%.3f Kb/s", _bps / MS_PER_S); \ 73 | else \ 74 | snprintf(_str, sizeof(_str), "%.3f b/s", _bps); \ 75 | _str; \ 76 | }) 77 | 78 | #define ins_bps(bytes, secs) \ 79 | __extension__({ \ 80 | static char _str[32]; \ 81 | const double _bps = ((secs) > 1e-9) ? (double)(bytes)*8 / (secs) : 0; \ 82 | snprintf(_str, sizeof(_str), "%.3f", _bps / US_PER_S); \ 83 | _str; \ 84 | }) 85 | 86 | 87 | struct conn_cache_entry { 88 | struct q_conn * c; 89 | struct addrinfo * migr_peer; 90 | bool migrated; 91 | uint8_t _unused[7]; 92 | }; 93 | 94 | 95 | KHASH_MAP_INIT_INT64(conn_cache, struct conn_cache_entry *) 96 | 97 | 98 | static uint32_t vers = 0; 99 | static uint32_t timeout = 10; 100 | static uint32_t num_bufs = 100000; 101 | static uint32_t reps = 1; 102 | static bool do_h3 = false; 103 | static bool do_v6 = false; 104 | static bool do_chacha = false; 105 | static bool flip_keys = false; 106 | static bool zlen_cids = false; 107 | static bool write_files = false; 108 | static bool test_qr = false; 109 | #ifndef NO_MIGRATION 110 | static bool rebind = false; 111 | static bool switch_ip = false; 112 | #endif 113 | 114 | 115 | struct stream_entry { 116 | sl_entry(stream_entry) next; 117 | struct conn_cache_entry * cce; 118 | struct q_stream * s; 119 | char * url; 120 | struct timespec req_t; 121 | struct timespec rep_t; 122 | struct w_iov_sq req; 123 | struct w_iov_sq rep; 124 | }; 125 | 126 | 127 | static sl_head(stream_list, stream_entry) sl = sl_head_initializer(sl); 128 | 129 | 130 | static inline uint64_t __attribute__((nonnull)) 131 | conn_cache_key(const struct sockaddr * const sock) 132 | { 133 | const struct sockaddr_in * const sock4 = 134 | (const struct sockaddr_in *)(const void *)sock; 135 | 136 | return ((uint64_t)sock4->sin_addr.s_addr 137 | << sizeof(sock4->sin_addr.s_addr) * 8) | 138 | (uint64_t)sock4->sin_port; 139 | } 140 | 141 | 142 | static void __attribute__((noreturn, nonnull)) 143 | usage(const char * const name, 144 | const char * const ifname, 145 | const char * const cache, 146 | const char * const tls_log, 147 | const char * const qlog_dir, 148 | const bool verify_certs) 149 | { 150 | printf("%s [options] URL [URL...]\n", name); 151 | printf("\t[-3]\t\tsend a static H3 request; default %s\n", 152 | do_h3 ? "true" : "false"); 153 | printf("\t[-6]\t\tuse IPv6; default %s\n", do_v6 ? "true" : "false"); 154 | printf("\t[-a]\t\tforce Chacha20; default %s\n", 155 | do_chacha ? "true" : "false"); 156 | printf("\t[-b bufs]\tnumber of network buffers to allocate; default %u\n", 157 | num_bufs); 158 | printf("\t[-c]\t\tverify TLS certificates; default %s\n", 159 | verify_certs ? "true" : "false"); 160 | printf("\t[-e version]\tQUIC version to use; default 0x%08x\n", vers); 161 | printf("\t[-i interface]\tinterface to run over; default %s\n", ifname); 162 | printf("\t[-l log]\tlog file for TLS keys; default %s\n", 163 | *tls_log ? tls_log : "false"); 164 | printf("\t[-m]\t\ttest multi-pkt initial (\"quantum-readiness\"); default " 165 | "%s\n", 166 | test_qr ? "true" : "false"); 167 | #ifndef NO_MIGRATION 168 | printf("\t[-n]\t\tsimulate NAT rebind (use twice for \"real\" migration); " 169 | "default %s\n", 170 | rebind ? "true" : "false"); 171 | #endif 172 | printf("\t[-q log]\twrite qlog events to directory; default %s\n", 173 | *qlog_dir ? qlog_dir : "false"); 174 | printf("\t[-r reps]\trepetitions for all URLs; default %u\n", reps); 175 | printf("\t[-s cache]\tTLS 0-RTT state cache; default %s\n", cache); 176 | printf("\t[-t timeout]\tidle timeout in seconds; default %u\n", timeout); 177 | printf("\t[-u]\t\tupdate TLS keys; default %s\n", 178 | flip_keys ? "true" : "false"); 179 | #ifndef NDEBUG 180 | printf("\t[-v verbosity]\tverbosity level (0-%d, default %d)\n", DLEVEL, 181 | util_dlevel); 182 | #endif 183 | printf("\t[-w]\t\twrite retrieved objects to disk; default %s\n", 184 | write_files ? "true" : "false"); 185 | printf("\t[-z]\t\tuse zero-length source connection IDs; default %s\n", 186 | zlen_cids ? "true" : "false"); 187 | exit(0); 188 | } 189 | 190 | 191 | static void __attribute__((nonnull)) 192 | set_from_url(char * const var, 193 | const size_t len, 194 | const char * const url, 195 | const struct http_parser_url * const u, 196 | const enum http_parser_url_fields f, 197 | const char * const def) 198 | { 199 | if ((u->field_set & (1 << f)) == 0) { 200 | strncpy(var, def, len); 201 | var[len - 1] = 0; 202 | } else { 203 | const uint16_t l = f == UF_PATH 204 | ? (uint16_t)strlen(url) - u->field_data[f].off 205 | : u->field_data[f].len; 206 | strncpy(var, &url[u->field_data[f].off], l); 207 | var[l] = 0; 208 | } 209 | } 210 | 211 | 212 | #ifndef NO_MIGRATION 213 | static void __attribute__((nonnull(1))) 214 | try_migrate(struct conn_cache_entry * const cce) 215 | { 216 | if (rebind && cce->migrated == false) 217 | cce->migrated = q_migrate(cce->c, switch_ip, 218 | cce->migr_peer ? cce->migr_peer->ai_addr : 0); 219 | } 220 | #else 221 | #define try_migrate(...) 222 | #endif 223 | 224 | 225 | static struct addrinfo * __attribute__((nonnull)) 226 | get_addr(const int af, const char * const dest, const char * const port) 227 | { 228 | struct addrinfo * peer = 0; 229 | const int err = getaddrinfo( 230 | dest, port, &(const struct addrinfo){.ai_family = af}, &peer); 231 | if (err == 0) 232 | return peer; 233 | 234 | if (err != EAI_FAMILY && err != EAI_NONAME) 235 | // when looking up migr_peer, these errors are OK to occur 236 | warn(ERR, "getaddrinfo: %s", gai_strerror(err)); 237 | if (peer) 238 | freeaddrinfo(peer); 239 | return 0; 240 | } 241 | 242 | 243 | static struct q_conn * __attribute__((nonnull)) 244 | get(char * const url, struct w_engine * const w, khash_t(conn_cache) * cc) 245 | { 246 | // parse and verify the URIs passed on the command line 247 | struct http_parser_url u = {0}; 248 | if (http_parser_parse_url(url, strlen(url), 0, &u)) { 249 | warn(ERR, "http_parser_parse_url: %s", 250 | http_errno_description((enum http_errno)errno)); 251 | return 0; 252 | } 253 | 254 | ensure((u.field_set & (1 << UF_USERINFO)) == 0, 255 | "userinfo unsupported in URL"); 256 | 257 | // extract relevant info from URL 258 | char dest[1024]; 259 | char port[64]; 260 | char path[8192]; 261 | set_from_url(dest, sizeof(dest), url, &u, UF_HOST, "localhost"); 262 | set_from_url(port, sizeof(port), url, &u, UF_PORT, "4433"); 263 | set_from_url(path, sizeof(path), url, &u, UF_PATH, "/index.html"); 264 | 265 | struct addrinfo * const peer = 266 | get_addr(do_v6 ? AF_INET6 : AF_INET, dest, port); 267 | struct addrinfo * const migr_peer = 268 | get_addr(do_v6 ? AF_INET : AF_INET6, dest, port); 269 | if (peer == 0) 270 | goto fail; 271 | 272 | // do we have a connection open to this peer? 273 | khiter_t k = kh_get(conn_cache, cc, conn_cache_key(peer->ai_addr)); 274 | struct conn_cache_entry * cce = (k == kh_end(cc) ? 0 : kh_val(cc, k)); 275 | 276 | // add to stream list 277 | struct stream_entry * se = calloc(1, sizeof(*se)); 278 | ensure(se, "calloc failed"); 279 | sq_init(&se->rep); 280 | sl_insert_head(&sl, se, next); 281 | 282 | sq_init(&se->req); 283 | if (do_h3) { 284 | q_alloc(w, &se->req, 0, peer->ai_family, 1024); 285 | struct w_iov * const v = sq_first(&se->req); 286 | const uint16_t len = 287 | (uint16_t)(h3zero_create_request_header_frame( 288 | &v->buf[3], v->buf + v->len - 3, (uint8_t *)path, 289 | strlen(path), dest) - 290 | &v->buf[3]); 291 | 292 | v->buf[0] = h3zero_frame_header; 293 | if (len < 64) { 294 | v->buf[1] = (uint8_t)len; 295 | memmove(&v->buf[2], &v->buf[3], len); 296 | v->len = 2 + len; 297 | } else { 298 | v->buf[1] = (uint8_t)((len >> 8) | 0x40); 299 | v->buf[2] = (uint8_t)(len & 0xff); 300 | v->len = 3 + len; 301 | } 302 | 303 | } else { 304 | // assemble an HTTP/0.9 request 305 | char req_str[sizeof(path) + 6]; 306 | const int req_str_len = 307 | snprintf(req_str, sizeof(req_str), "GET %s\r\n", path); 308 | q_chunk_str(w, cce ? cce->c : 0, peer->ai_family, req_str, 309 | (uint32_t)req_str_len, &se->req); 310 | } 311 | 312 | const bool opened_new = cce == 0; 313 | if (cce == 0) { 314 | clock_gettime(CLOCK_MONOTONIC, &se->req_t); 315 | // no, open a new connection 316 | struct q_conn * const c = q_connect( 317 | w, peer->ai_addr, dest, &se->req, &se->s, true, 318 | do_h3 ? "h3-" DRAFT_VERSION_STRING : "hq-" DRAFT_VERSION_STRING, 0); 319 | if (c == 0) 320 | goto fail; 321 | 322 | if (do_h3) { 323 | // we need to open a uni stream for an empty H/3 SETTINGS frame 324 | struct q_stream * const ss = q_rsv_stream(c, false); 325 | if (ss == 0) 326 | return 0; 327 | static const uint8_t h3_empty_settings[] = { 328 | 0x00, h3zero_frame_settings, 0x00}; 329 | // XXX lsquic doesn't like a FIN on this stream 330 | q_write_str(w, ss, (const char *)h3_empty_settings, 331 | sizeof(h3_empty_settings), false); 332 | } 333 | 334 | cce = calloc(1, sizeof(*cce)); 335 | ensure(cce, "calloc failed"); 336 | cce->c = c; 337 | cce->migr_peer = migr_peer; 338 | 339 | // insert into connection cache 340 | int ret; 341 | k = kh_put(conn_cache, cc, conn_cache_key(peer->ai_addr), &ret); 342 | ensure(ret >= 1, "inserted returned %d", ret); 343 | kh_val(cc, k) = cce; 344 | } 345 | 346 | if (opened_new == false) { 347 | se->s = q_rsv_stream(cce->c, true); 348 | if (se->s) { 349 | clock_gettime(CLOCK_MONOTONIC, &se->req_t); 350 | q_write(se->s, &se->req, true); 351 | } 352 | } 353 | try_migrate(cce); 354 | 355 | se->cce = cce; 356 | se->url = url; 357 | freeaddrinfo(peer); 358 | return cce->c; 359 | 360 | fail: 361 | freeaddrinfo(peer); 362 | freeaddrinfo(migr_peer); 363 | return 0; 364 | } 365 | 366 | 367 | static void __attribute__((nonnull)) free_cc(khash_t(conn_cache) * cc) 368 | { 369 | struct conn_cache_entry * cce; 370 | kh_foreach_value(cc, cce, { 371 | freeaddrinfo(cce->migr_peer); 372 | free(cce); 373 | }); 374 | kh_release(conn_cache, cc); 375 | } 376 | 377 | 378 | static void free_se(struct stream_entry * const se) 379 | { 380 | q_free(&se->req); 381 | q_free(&se->rep); 382 | free(se); 383 | } 384 | 385 | 386 | static void free_sl_head(void) 387 | { 388 | struct stream_entry * const se = sl_first(&sl); 389 | sl_remove_head(&sl, next); 390 | free_se(se); 391 | } 392 | 393 | 394 | static void free_sl(void) 395 | { 396 | while (sl_empty(&sl) == false) 397 | free_sl_head(); 398 | } 399 | 400 | 401 | static void __attribute__((nonnull)) 402 | write_object(struct stream_entry * const se) 403 | { 404 | char * const slash = strrchr(se->url, '/'); 405 | if (slash && *(slash + 1) == 0) 406 | // this URL ends in a slash, so strip that to name the file 407 | *slash = 0; 408 | 409 | const int fd = 410 | open(*basename(se->url) == 0 ? "index.html" : basename(se->url), 411 | O_CREAT | O_WRONLY | O_CLOEXEC, 412 | S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH); 413 | ensure(fd != -1, "cannot open %s", basename(se->url)); 414 | 415 | struct iovec vec[IOV_MAX]; 416 | struct w_iov * v = sq_first(&se->rep); 417 | int i = 0; 418 | while (v) { 419 | vec[i].iov_base = v->buf; 420 | vec[i].iov_len = v->len; 421 | if (++i == IOV_MAX || sq_next(v, next) == 0) { 422 | ensure(writev(fd, vec, i) != -1, "cannot writev"); 423 | i = 0; 424 | } 425 | v = sq_next(v, next); 426 | } 427 | close(fd); 428 | } 429 | 430 | 431 | int main(int argc, char * argv[]) 432 | { 433 | #ifndef NDEBUG 434 | util_dlevel = DLEVEL; // default to maximum compiled-in verbosity 435 | #endif 436 | char ifname[IFNAMSIZ] = "lo" 437 | #ifndef __linux__ 438 | "0" 439 | #endif 440 | ; 441 | int ch; 442 | char cache[MAXPATHLEN] = "/tmp/" QUANT "-session"; 443 | char tls_log[MAXPATHLEN] = ""; 444 | char qlog_dir[MAXPATHLEN] = ""; 445 | bool verify_certs = false; 446 | int ret = -1; 447 | 448 | // set default TLS log file from environment 449 | const char * const keylog = getenv("SSLKEYLOGFILE"); 450 | if (keylog) { 451 | strncpy(tls_log, keylog, MAXPATHLEN); 452 | tls_log[MAXPATHLEN - 1] = 0; 453 | } 454 | 455 | while ((ch = getopt(argc, argv, 456 | "hi:v:s:t:l:cu36azb:wr:q:me:" 457 | #ifndef NO_MIGRATION 458 | "n" 459 | #endif 460 | )) != -1) { 461 | switch (ch) { 462 | case 'i': 463 | strncpy(ifname, optarg, sizeof(ifname) - 1); 464 | break; 465 | case 's': 466 | strncpy(cache, optarg, sizeof(cache) - 1); 467 | break; 468 | case 'q': 469 | strncpy(qlog_dir, optarg, sizeof(qlog_dir) - 1); 470 | break; 471 | case 't': 472 | timeout = (uint32_t)MIN(600, strtoul(optarg, 0, 10)); // 10 min 473 | break; 474 | case 'b': 475 | num_bufs = (uint32_t)MIN(strtoul(optarg, 0, 10), UINT32_MAX); 476 | break; 477 | case 'r': 478 | reps = (uint32_t)MAX(1, MIN(strtoul(optarg, 0, 10), UINT32_MAX)); 479 | break; 480 | case 'e': 481 | vers = (uint32_t)MAX(1, MIN(strtoul(optarg, 0, 16), UINT32_MAX)); 482 | break; 483 | case 'l': 484 | strncpy(tls_log, optarg, sizeof(tls_log) - 1); 485 | break; 486 | case 'c': 487 | verify_certs = true; 488 | break; 489 | case 'u': 490 | flip_keys = true; 491 | break; 492 | case '3': 493 | do_h3 = true; 494 | break; 495 | case '6': 496 | do_v6 = true; 497 | break; 498 | case 'a': 499 | do_chacha = true; 500 | break; 501 | case 'z': 502 | zlen_cids = true; 503 | break; 504 | case 'w': 505 | write_files = true; 506 | break; 507 | case 'm': 508 | test_qr = true; 509 | break; 510 | #ifndef NO_MIGRATION 511 | case 'n': 512 | if (rebind) 513 | switch_ip = true; 514 | rebind = true; 515 | break; 516 | #endif 517 | case 'v': 518 | #ifndef NDEBUG 519 | util_dlevel = (short)MIN(DLEVEL, strtoul(optarg, 0, 10)); 520 | #endif 521 | break; 522 | case 'h': 523 | case '?': 524 | default: 525 | usage(basename(argv[0]), ifname, cache, tls_log, qlog_dir, 526 | verify_certs); 527 | } 528 | } 529 | 530 | struct w_engine * const w = q_init( 531 | ifname, 532 | &(const struct q_conf){ 533 | .conn_conf = 534 | &(struct q_conn_conf){.enable_tls_key_updates = flip_keys, 535 | .enable_spinbit = true, 536 | .enable_udp_zero_checksums = true, 537 | .idle_timeout = timeout, 538 | .version = vers, 539 | .enable_quantum_readiness_test = test_qr}, 540 | .qlog_dir = *qlog_dir ? qlog_dir : 0, 541 | .force_chacha20 = do_chacha, 542 | .num_bufs = num_bufs, 543 | .ticket_store = cache, 544 | .tls_log = *tls_log ? tls_log : 0, 545 | .client_cid_len = zlen_cids ? 0 : 4, 546 | .enable_tls_cert_verify = verify_certs}); 547 | khash_t(conn_cache) cc = {0}; 548 | FILE *ins_fd; 549 | ins_fd = fopen("bps_ins.txt", "a"); 550 | if (reps > 1){ 551 | puts("size\ttime\t\tbps\t\turl"); 552 | } 553 | double sum_len = 0; 554 | double sum_elapsed = 0; 555 | for (uint64_t r = 1; r <= reps; r++) { 556 | int url_idx = optind; 557 | while (url_idx < argc) { 558 | // open a new connection, or get an open one 559 | warn(INF, "%s retrieving %s", basename(argv[0]), argv[url_idx]); 560 | get(argv[url_idx++], w, &cc); 561 | } 562 | 563 | // collect the replies 564 | bool all_closed; 565 | do { 566 | all_closed = true; 567 | bool rxed_new = false; 568 | struct stream_entry * se = 0; //Is this entry the stream entry of the client? 569 | struct stream_entry * tmp = 0; 570 | sl_foreach_safe (se, &sl, next, tmp) { 571 | if (se->cce == 0 || se->cce->c == 0 || se->s == 0) { 572 | sl_remove(&sl, se, stream_entry, next); 573 | free_se(se); 574 | continue; 575 | } 576 | try_migrate(se->cce); 577 | rxed_new |= q_read_stream(se->s, &se->rep, false); 578 | 579 | const bool is_closed = q_peer_closed_stream(se->s); 580 | all_closed &= is_closed; 581 | if (is_closed) 582 | clock_gettime(CLOCK_MONOTONIC, &se->rep_t); 583 | } 584 | 585 | if (rxed_new == false && all_closed == false) { 586 | struct q_conn * c; 587 | q_ready(w, timeout * NS_PER_S, &c); 588 | if (c == 0) 589 | break; 590 | if (q_is_conn_closed(c)) 591 | break; 592 | } 593 | 594 | } while (all_closed == false); 595 | 596 | // print/save the replies 597 | while (sl_empty(&sl) == false) { 598 | struct stream_entry * const se = sl_first(&sl); 599 | if (ret == -1) 600 | ret = w_iov_sq_cnt(&se->rep) == 0; 601 | else 602 | ret |= w_iov_sq_cnt(&se->rep) == 0; 603 | 604 | struct timespec diff; 605 | timespec_sub(&se->rep_t, &se->req_t, &diff); 606 | const double elapsed = timespec_to_double(diff); 607 | const uint_t rep_len = w_iov_sq_len(&se->rep); 608 | sum_len += (double)rep_len; 609 | sum_elapsed += elapsed; 610 | if (reps > 1) 611 | printf("%" PRIu "\t%f\t\"%s\"\t%s\n", rep_len, elapsed, 612 | bps(rep_len, elapsed), se->url); 613 | fprintf(ins_fd, "%s\n", ins_bps(rep_len, elapsed)); 614 | 615 | #ifndef NDEBUG 616 | char cid_str[64]; 617 | q_cid_str(se->cce->c, cid_str, sizeof(cid_str)); 618 | warn(WRN, 619 | "read %" PRIu 620 | " byte%s in %.3f sec (%s) on conn %s strm %" PRIu, 621 | rep_len, plural(rep_len), elapsed < 0 ? 0 : elapsed, 622 | bps(rep_len, elapsed), cid_str, q_sid(se->s)); 623 | #endif 624 | 625 | // retrieve the TX'ed request 626 | q_stream_get_written(se->s, &se->req); 627 | 628 | if (write_files) 629 | write_object(se); 630 | 631 | // save the object, and print its first three packets to stdout 632 | struct w_iov * v; 633 | uint32_t n = 0; 634 | sq_foreach (v, &se->rep, next) { 635 | // cppcheck-suppress nullPointer 636 | const bool is_last = v == sq_last(&se->rep, w_iov, next); 637 | if (w_iov_sq_cnt(&se->rep) > 100 || reps > 1) 638 | // don't print large responses, or repeated ones 639 | continue; 640 | 641 | // XXX the strnlen() test is super-hacky 642 | if (do_h3 && n == 0 && 643 | (v->buf[0] != 0x01 && v->buf[0] != 0xff && 644 | strnlen((char *)v->buf, v->len) == v->len)) 645 | warn(WRN, "no h3 payload"); 646 | if (n < 4 || is_last) { 647 | if (do_h3) { 648 | #ifndef NDEBUG 649 | if (util_dlevel == DBG) 650 | hexdump(v->buf, v->len); 651 | #endif 652 | } else { 653 | // don't print newlines to console log 654 | for (uint16_t p = 0; p < v->len; p++) 655 | if (v->buf[p] == '\n' || v->buf[p] == '\r') 656 | v->buf[p] = ' '; 657 | printf("%.*s%s", v->len, v->buf, is_last ? "\n" : ""); 658 | if (is_last) 659 | fflush(stdout); 660 | } 661 | } else 662 | printf("."); 663 | n++; 664 | } 665 | 666 | q_free_stream(se->s); 667 | free_sl_head(); 668 | } 669 | } 670 | 671 | if (reps > 1) 672 | fclose(ins_fd); 673 | printf("TOTAL: %s\n", bps(sum_len, sum_elapsed)); 674 | 675 | free_cc(&cc); 676 | free_sl(); 677 | q_cleanup(w); 678 | warn(DBG, "%s exiting with %d", basename(argv[0]), ret); 679 | return ret; 680 | } 681 | 682 | -------------------------------------------------------------------------------- /kits/quant/client_netmap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/ bash 2 | # $a is the test tag (can be used to identify loss rate, etc.) 3 | cp client_netmap.sh ../../extern/quant/Release . 4 | cd ../../extern/quant/Release 5 | 6 | a=$1 7 | echo "+++{$a}" >> bps_ins.txt 8 | #change client-warp to client to replace netmap with socket. 9 | taskset 64 ./bin/client-warp -b 50000 -r 20 -i ens3f1 http://192.168.0.1:4433/50000000 10 | sleep 1 11 | echo "---{$a}" >> bps_ins.txt 12 | -------------------------------------------------------------------------------- /kits/quant/config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #put the instrumented source file into the proj 3 | mv ../../extern/quant/bin/client.c ../../extern/quant/bin/client.c.bak 4 | cp client.c ../../extern/quant/bin -------------------------------------------------------------------------------- /kits/quant/server_netmap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/ bash 2 | 3 | taskset 128 ip netns exec blue ./bin/server-warp -b 50000 -i ens3f0 -p 4433 4 | -------------------------------------------------------------------------------- /kits/quicly/README.md: -------------------------------------------------------------------------------- 1 | ### README 2 | 3 | 1. Firstly you need to setup the connection between two interfaces (`ens3f0` `ens3f1` by default); 4 | 2. `TLEM` can be used for this purpose. For example if you want to add 500us latency between the two interfaces, you can use cmds below. More options can be found [here](https://github.com/luigirizzo/netmap/tree/master/apps/tlem). 5 | > ./tlem -D 500us -i netmap:ens4f0 -i netmap:ens4f1 6 | 3. Run `config.sh` and re-compile `quicly`; 7 | 4. Copy `server.sh`, `client.sh` and `multi_client.sh` into `extern/quicly/`; 8 | 5. Run the quant server first by executing `server.sh`; 9 | 6. Run the quant client by executing `client.sh` (with correct parameters); 10 | 7. Results (Throughput) is extracted to `flow_completion_time.txt`. -------------------------------------------------------------------------------- /kits/quicly/cli.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2017 Fastly, Kazuho Oku 3 | * 4 | * Permission is hereby granted, free of charge, to any person obtaining a copy 5 | * of this software and associated documentation files (the "Software"), to 6 | * deal in the Software without restriction, including without limitation the 7 | * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 8 | * sell copies of the Software, and to permit persons to whom the Software is 9 | * furnished to do so, subject to the following conditions: 10 | * 11 | * The above copyright notice and this permission notice shall be included in 12 | * all copies or substantial portions of the Software. 13 | * 14 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 19 | * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 20 | * IN THE SOFTWARE. 21 | */ 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include "quicly.h" 34 | #include "quicly/defaults.h" 35 | #include "quicly/streambuf.h" 36 | #include "../deps/picotls/t/util.h" 37 | 38 | extern int errno; 39 | FILE *quicly_trace_fp = NULL; 40 | static unsigned verbosity = 0; 41 | static int64_t enqueue_requests_at = 0, request_interval = 0; 42 | static int64_t fct_switch = 0; 43 | 44 | static void hexdump(const char *title, const uint8_t *p, size_t l) 45 | { 46 | fprintf(stderr, "%s (%zu bytes):\n", title, l); 47 | 48 | while (l != 0) { 49 | int i; 50 | fputs(" ", stderr); 51 | for (i = 0; i < 16; ++i) { 52 | fprintf(stderr, " %02x", *p++); 53 | if (--l == 0) 54 | break; 55 | } 56 | fputc('\n', stderr); 57 | } 58 | } 59 | 60 | static int save_session_ticket_cb(ptls_save_ticket_t *_self, ptls_t *tls, ptls_iovec_t src); 61 | static int on_client_hello_cb(ptls_on_client_hello_t *_self, ptls_t *tls, ptls_on_client_hello_parameters_t *params); 62 | 63 | static const char *session_file = NULL; 64 | static ptls_handshake_properties_t hs_properties; 65 | static quicly_transport_parameters_t resumed_transport_params; 66 | static ptls_iovec_t resumption_token; 67 | static quicly_context_t ctx; 68 | static quicly_cid_plaintext_t next_cid; 69 | static struct { 70 | ptls_aead_context_t *enc, *dec; 71 | } address_token_aead; 72 | static ptls_save_ticket_t save_session_ticket = {save_session_ticket_cb}; 73 | static ptls_on_client_hello_t on_client_hello = {on_client_hello_cb}; 74 | static int enforce_retry; 75 | 76 | ptls_key_exchange_algorithm_t *key_exchanges[128]; 77 | static ptls_context_t tlsctx = {.random_bytes = ptls_openssl_random_bytes, 78 | .get_time = &ptls_get_time, 79 | .key_exchanges = key_exchanges, 80 | .cipher_suites = ptls_openssl_cipher_suites, 81 | .require_dhe_on_psk = 1, 82 | .save_ticket = &save_session_ticket, 83 | .on_client_hello = &on_client_hello}; 84 | static struct { 85 | ptls_iovec_t list[16]; 86 | size_t count; 87 | } negotiated_protocols; 88 | 89 | /** 90 | * list of requests to be processed, terminated by reqs[N].path == NULL 91 | */ 92 | struct { 93 | const char *path; 94 | int to_file; 95 | } * reqs; 96 | 97 | struct st_stream_data_t { 98 | quicly_streambuf_t streambuf; 99 | FILE *outfp; 100 | }; 101 | 102 | static void on_stop_sending(quicly_stream_t *stream, int err); 103 | static void on_receive_reset(quicly_stream_t *stream, int err); 104 | static void server_on_receive(quicly_stream_t *stream, size_t off, const void *src, size_t len); 105 | static void client_on_receive(quicly_stream_t *stream, size_t off, const void *src, size_t len); 106 | 107 | static const quicly_stream_callbacks_t server_stream_callbacks = {quicly_streambuf_destroy, 108 | quicly_streambuf_egress_shift, 109 | quicly_streambuf_egress_emit, 110 | on_stop_sending, 111 | server_on_receive, 112 | on_receive_reset}, 113 | client_stream_callbacks = {quicly_streambuf_destroy, 114 | quicly_streambuf_egress_shift, 115 | quicly_streambuf_egress_emit, 116 | on_stop_sending, 117 | client_on_receive, 118 | on_receive_reset}; 119 | 120 | static void dump_stats(FILE *fp, quicly_conn_t *conn) 121 | { 122 | quicly_stats_t stats; 123 | 124 | quicly_get_stats(conn, &stats); 125 | fprintf(fp, 126 | "packets-received: %" PRIu64 ", packets-decryption-failed: %" PRIu64 ", packets-sent: %" PRIu64 127 | ", packets-lost: %" PRIu64 ", ack-received: %" PRIu64 ", bytes-received: %" PRIu64 ", bytes-sent: %" PRIu64 128 | ", srtt: %" PRIu32 "\n", 129 | stats.num_packets.received, stats.num_packets.decryption_failed, stats.num_packets.sent, stats.num_packets.lost, 130 | stats.num_packets.ack_received, stats.num_bytes.received, stats.num_bytes.sent, stats.rtt.smoothed); 131 | } 132 | 133 | static int validate_path(const char *path) 134 | { 135 | if (path[0] != '/') 136 | return 0; 137 | /* TODO avoid false positives on the client-side */ 138 | if (strstr(path, "/.") != NULL) 139 | return 0; 140 | return 1; 141 | } 142 | 143 | static int parse_request(ptls_iovec_t input, char **path, int *is_http1) 144 | { 145 | size_t off = 0, path_start; 146 | 147 | for (off = 0; off != input.len; ++off) 148 | if (input.base[off] == ' ') 149 | goto EndOfMethod; 150 | return 0; 151 | 152 | EndOfMethod: 153 | ++off; 154 | path_start = off; 155 | for (; off != input.len; ++off) 156 | if (input.base[off] == ' ' || input.base[off] == '\r' || input.base[off] == '\n') 157 | goto EndOfPath; 158 | return 0; 159 | 160 | EndOfPath: 161 | *path = (char *)(input.base + path_start); 162 | *is_http1 = input.base[off] == ' '; 163 | input.base[off] = '\0'; 164 | return 1; 165 | } 166 | 167 | static void send_str(quicly_stream_t *stream, const char *s) 168 | { 169 | quicly_streambuf_egress_write(stream, s, strlen(s)); 170 | } 171 | 172 | static void send_header(quicly_stream_t *stream, int is_http1, int status, const char *mime_type) 173 | { 174 | char buf[256]; 175 | 176 | if (!is_http1) 177 | return; 178 | 179 | sprintf(buf, "HTTP/1.1 %03d OK\r\nConnection: close\r\nContent-Type: %s\r\n\r\n", status, mime_type); 180 | send_str(stream, buf); 181 | } 182 | 183 | static int flatten_file_vec(quicly_sendbuf_vec_t *vec, void *dst, size_t off, size_t len) 184 | { 185 | int fd = (intptr_t)vec->cbdata; 186 | ssize_t rret; 187 | 188 | /* FIXME handle partial read */ 189 | while ((rret = pread(fd, dst, len, off)) == -1 && errno == EINTR) 190 | ; 191 | 192 | return rret == len ? 0 : QUICLY_TRANSPORT_ERROR_INTERNAL; /* should return application-level error */ 193 | } 194 | 195 | static void discard_file_vec(quicly_sendbuf_vec_t *vec) 196 | { 197 | int fd = (intptr_t)vec->cbdata; 198 | close(fd); 199 | } 200 | 201 | static int send_file(quicly_stream_t *stream, int is_http1, const char *fn, const char *mime_type) 202 | { 203 | static const quicly_streambuf_sendvec_callbacks_t send_file_callbacks = {flatten_file_vec, discard_file_vec}; 204 | int fd; 205 | struct stat st; 206 | int errnum; 207 | if ((fd = open(fn, O_RDONLY)) == -1){ 208 | errnum = errno; 209 | fprintf(stderr, "Error opening file: %s\n", strerror(errnum)); 210 | printf("cannot open()\n"); 211 | return 0; 212 | } 213 | 214 | if (fstat(fd, &st) != 0 || S_ISDIR(st.st_mode)) { 215 | close(fd); 216 | return 0; 217 | } 218 | 219 | send_header(stream, is_http1, 200, mime_type); 220 | quicly_sendbuf_vec_t vec = {&send_file_callbacks, (size_t)st.st_size, (void *)(intptr_t)fd}; 221 | quicly_streambuf_egress_write_vec(stream, &vec); 222 | //close(fd); 223 | return 1; 224 | } 225 | 226 | /** 227 | * This function is an implementation of the quicly_sendbuf_flatten_vec_cb callback. Refer to the doc-comments of the callback type 228 | * for the API. 229 | */ 230 | static int flatten_sized_text(quicly_sendbuf_vec_t *vec, void *dst, size_t off, size_t len) 231 | { 232 | static const char pattern[] = 233 | "hello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 234 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 235 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 236 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 237 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 238 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 239 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 240 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 241 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 242 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 243 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 244 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 245 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 246 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 247 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello world\nhello " 248 | "world\nhello world\nhello world\nhello world\nhello world\nhello world\n"; 249 | 250 | const char *src = pattern + off % 12; 251 | assert(src + len - pattern <= sizeof(pattern) - 1); /* pattern is bigger than MTU size */ 252 | memcpy(dst, src, len); 253 | return 0; 254 | 255 | #undef PATTERN 256 | } 257 | 258 | static int send_sized_text(quicly_stream_t *stream, const char *path, int is_http1) 259 | { 260 | size_t size; 261 | int lastpos; 262 | 263 | if (sscanf(path, "/%zu%n", &size, &lastpos) != 1) 264 | return 0; 265 | if (lastpos != strlen(path)) 266 | return 0; 267 | 268 | send_header(stream, is_http1, 200, "text/plain; charset=utf-8"); 269 | static const quicly_streambuf_sendvec_callbacks_t callbacks = {flatten_sized_text}; 270 | quicly_sendbuf_vec_t vec = {&callbacks, size, NULL}; 271 | quicly_streambuf_egress_write_vec(stream, &vec); 272 | return 1; 273 | } 274 | 275 | static void on_stop_sending(quicly_stream_t *stream, int err) 276 | { 277 | assert(QUICLY_ERROR_IS_QUIC_APPLICATION(err)); 278 | fprintf(stderr, "received STOP_SENDING: %" PRIu16 "\n", QUICLY_ERROR_GET_ERROR_CODE(err)); 279 | } 280 | 281 | static void on_receive_reset(quicly_stream_t *stream, int err) 282 | { 283 | assert(QUICLY_ERROR_IS_QUIC_APPLICATION(err)); 284 | fprintf(stderr, "received RESET_STREAM: %" PRIu16 "\n", QUICLY_ERROR_GET_ERROR_CODE(err)); 285 | } 286 | 287 | static void server_on_receive(quicly_stream_t *stream, size_t off, const void *src, size_t len) 288 | { 289 | char *path; 290 | int is_http1; 291 | 292 | if (!quicly_sendstate_is_open(&stream->sendstate)) 293 | return; 294 | 295 | if (quicly_streambuf_ingress_receive(stream, off, src, len) != 0) 296 | return; 297 | 298 | if (!parse_request(quicly_streambuf_ingress_get(stream), &path, &is_http1)) { 299 | if (!quicly_recvstate_transfer_complete(&stream->recvstate)) 300 | return; 301 | /* failed to parse request */ 302 | send_header(stream, 1, 500, "text/plain; charset=utf-8"); 303 | send_str(stream, "failed to parse HTTP request\n"); 304 | goto Sent; 305 | } 306 | if (!quicly_recvstate_transfer_complete(&stream->recvstate)) 307 | quicly_request_stop(stream, QUICLY_ERROR_FROM_APPLICATION_ERROR_CODE(0)); 308 | 309 | if (strcmp(path, "/logo.jpg") == 0 && send_file(stream, is_http1, "assets/logo.jpg", "image/jpeg")) 310 | goto Sent; 311 | if (strcmp(path, "/main.jpg") == 0 && send_file(stream, is_http1, "assets/main.jpg", "image/jpeg")) 312 | goto Sent; 313 | if (send_sized_text(stream, path, is_http1)){ 314 | printf("debug 1\n"); 315 | goto Sent; 316 | } 317 | printf("the path is: %s\n", path); 318 | int validate_ret = validate_path(path); 319 | int sent_file_ret = send_file(stream, is_http1, path+1, "text/plain"); 320 | printf("validate ret: %d;\tsent_file_ret: %d\n", validate_ret, sent_file_ret); 321 | if (validate_ret && sent_file_ret){ 322 | printf("debug 2\n"); 323 | goto Sent; 324 | } 325 | send_header(stream, is_http1, 404, "text/plain; charset=utf-8"); 326 | send_str(stream, "not found\n"); 327 | Sent: 328 | quicly_streambuf_egress_shutdown(stream); 329 | quicly_streambuf_ingress_shift(stream, len); 330 | } 331 | 332 | static void client_on_receive(quicly_stream_t *stream, size_t off, const void *src, size_t len) 333 | { 334 | struct st_stream_data_t *stream_data = stream->data; 335 | ptls_iovec_t input; 336 | 337 | if (quicly_streambuf_ingress_receive(stream, off, src, len) != 0) 338 | return; 339 | 340 | if ((input = quicly_streambuf_ingress_get(stream)).len != 0) { 341 | //FILE *out = (stream_data->outfp == NULL) ? stdout : stream_data->outfp; 342 | //fwrite(input.base, 1, input.len, out); 343 | //fflush(out); 344 | quicly_streambuf_ingress_shift(stream, input.len); 345 | } 346 | 347 | if (quicly_recvstate_transfer_complete(&stream->recvstate)) { 348 | if (stream_data->outfp != NULL) 349 | printf("transfer complete\n"); 350 | //fclose(stream_data->outfp); 351 | static size_t num_resp_received; 352 | ++num_resp_received; 353 | if (reqs[num_resp_received].path == NULL) { 354 | if (request_interval != 0) { 355 | enqueue_requests_at = ctx.now->cb(ctx.now) + request_interval; 356 | } else { 357 | dump_stats(stderr, stream->conn); 358 | quicly_close(stream->conn, 0, ""); 359 | } 360 | } 361 | } 362 | } 363 | 364 | static int on_stream_open(quicly_stream_open_t *self, quicly_stream_t *stream) 365 | { 366 | int ret; 367 | 368 | if ((ret = quicly_streambuf_create(stream, sizeof(struct st_stream_data_t))) != 0) 369 | return ret; 370 | stream->callbacks = ctx.tls->certificates.count != 0 ? &server_stream_callbacks : &client_stream_callbacks; 371 | return 0; 372 | } 373 | 374 | static quicly_stream_open_t stream_open = {&on_stream_open}; 375 | 376 | static void on_closed_by_peer(quicly_closed_by_peer_t *self, quicly_conn_t *conn, int err, uint64_t frame_type, const char *reason, 377 | size_t reason_len) 378 | { 379 | if (QUICLY_ERROR_IS_QUIC_TRANSPORT(err)) { 380 | fprintf(stderr, "transport close:code=0x%" PRIx16 ";frame=%" PRIu64 ";reason=%.*s\n", QUICLY_ERROR_GET_ERROR_CODE(err), 381 | frame_type, (int)reason_len, reason); 382 | } else if (QUICLY_ERROR_IS_QUIC_APPLICATION(err)) { 383 | fprintf(stderr, "application close:code=0x%" PRIx16 ";reason=%.*s\n", QUICLY_ERROR_GET_ERROR_CODE(err), (int)reason_len, 384 | reason); 385 | } else if (err == QUICLY_ERROR_RECEIVED_STATELESS_RESET) { 386 | fprintf(stderr, "stateless reset\n"); 387 | } else { 388 | fprintf(stderr, "unexpected close:code=%d\n", err); 389 | } 390 | } 391 | 392 | static quicly_closed_by_peer_t closed_by_peer = {&on_closed_by_peer}; 393 | 394 | static int on_generate_resumption_token(quicly_generate_resumption_token_t *self, quicly_conn_t *conn, ptls_buffer_t *buf, 395 | quicly_address_token_plaintext_t *token) 396 | { 397 | return quicly_encrypt_address_token(tlsctx.random_bytes, address_token_aead.enc, buf, buf->off, token); 398 | } 399 | 400 | static quicly_generate_resumption_token_t generate_resumption_token = {&on_generate_resumption_token}; 401 | 402 | static int send_one(int fd, quicly_datagram_t *p) 403 | { 404 | int ret; 405 | struct msghdr mess; 406 | struct iovec vec; 407 | memset(&mess, 0, sizeof(mess)); 408 | mess.msg_name = &p->dest.sa; 409 | mess.msg_namelen = quicly_get_socklen(&p->dest.sa); 410 | vec.iov_base = p->data.base; 411 | vec.iov_len = p->data.len; 412 | mess.msg_iov = &vec; 413 | mess.msg_iovlen = 1; 414 | if (verbosity >= 2) 415 | hexdump("sendmsg", vec.iov_base, vec.iov_len); 416 | while ((ret = (int)sendmsg(fd, &mess, 0)) == -1 && errno == EINTR) 417 | ; 418 | return ret; 419 | } 420 | 421 | static int send_pending(int fd, quicly_conn_t *conn) 422 | { 423 | quicly_datagram_t *packets[16]; 424 | size_t num_packets, i; 425 | int ret; 426 | 427 | do { 428 | num_packets = sizeof(packets) / sizeof(packets[0]); 429 | if ((ret = quicly_send(conn, packets, &num_packets)) == 0) { 430 | for (i = 0; i != num_packets; ++i) { 431 | if ((ret = send_one(fd, packets[i])) == -1) 432 | perror("sendmsg failed"); 433 | ret = 0; 434 | quicly_packet_allocator_t *pa = quicly_get_context(conn)->packet_allocator; 435 | pa->free_packet(pa, packets[i]); 436 | } 437 | } 438 | } while (ret == 0 && num_packets == sizeof(packets) / sizeof(packets[0])); 439 | 440 | return ret; 441 | } 442 | 443 | static void enqueue_requests(quicly_conn_t *conn) 444 | { 445 | size_t i; 446 | int ret; 447 | 448 | for (i = 0; reqs[i].path != NULL; ++i) { 449 | char req[1024], destfile[1024]; 450 | quicly_stream_t *stream; 451 | ret = quicly_open_stream(conn, &stream, 0); 452 | assert(ret == 0); 453 | sprintf(req, "GET %s\r\n", reqs[i].path); 454 | send_str(stream, req); 455 | quicly_streambuf_egress_shutdown(stream); 456 | 457 | if (reqs[i].to_file) { 458 | struct st_stream_data_t *stream_data = stream->data; 459 | sprintf(destfile, "%s.downloaded", strrchr(reqs[i].path, '/') + 1); 460 | stream_data->outfp = fopen(destfile, "w"); 461 | if (stream_data->outfp == NULL) { 462 | fprintf(stderr, "failed to open destination file:%s:%s\n", reqs[i].path, strerror(errno)); 463 | exit(1); 464 | } 465 | } 466 | } 467 | enqueue_requests_at = INT64_MAX; 468 | } 469 | 470 | static int run_client(struct sockaddr *sa, const char *host) 471 | { 472 | int fd, ret; 473 | struct sockaddr_in local; 474 | quicly_conn_t *conn = NULL; 475 | 476 | FILE *fd_fct; 477 | fd_fct = fopen("flow_completion_time.txt","a"); 478 | 479 | struct timeval fct_tv_start, *fct_tv_start_ptr, fct_tv_end, *fct_tv_end_ptr; 480 | fct_tv_start_ptr = &fct_tv_start; 481 | fct_tv_end_ptr = &fct_tv_end; 482 | 483 | if ((fd = socket(sa->sa_family, SOCK_DGRAM, IPPROTO_UDP)) == -1) { 484 | perror("socket(2) failed"); 485 | return 1; 486 | } 487 | memset(&local, 0, sizeof(local)); 488 | local.sin_family = AF_INET; 489 | if (bind(fd, (void *)&local, sizeof(local)) != 0) { 490 | perror("bind(2) failed"); 491 | return 1; 492 | } 493 | 494 | 495 | if(fct_switch == 1){ 496 | if(gettimeofday(fct_tv_start_ptr, NULL)!=0) 497 | return 1; 498 | } 499 | 500 | ret = quicly_connect(&conn, &ctx, host, sa, NULL, &next_cid, resumption_token, &hs_properties, &resumed_transport_params); 501 | assert(ret == 0); 502 | ++next_cid.master_id; 503 | enqueue_requests(conn); 504 | send_pending(fd, conn); 505 | 506 | while (1) { 507 | fd_set readfds; 508 | struct timeval *tv, tvbuf; 509 | do { 510 | int64_t timeout_at = conn != NULL ? quicly_get_first_timeout(conn) : INT64_MAX; 511 | if (enqueue_requests_at < timeout_at) 512 | timeout_at = enqueue_requests_at; 513 | if (timeout_at != INT64_MAX) { 514 | quicly_context_t *ctx = quicly_get_context(conn); 515 | int64_t delta = timeout_at - ctx->now->cb(ctx->now); 516 | if (delta > 0) { 517 | tvbuf.tv_sec = delta / 1000; 518 | tvbuf.tv_usec = (delta % 1000) * 1000; 519 | } else { 520 | tvbuf.tv_sec = 0; 521 | tvbuf.tv_usec = 0; 522 | } 523 | tv = &tvbuf; 524 | } else { 525 | tv = NULL; 526 | } 527 | FD_ZERO(&readfds); 528 | FD_SET(fd, &readfds); 529 | } while (select(fd + 1, &readfds, NULL, NULL, tv) == -1 && errno == EINTR); 530 | if (enqueue_requests_at <= ctx.now->cb(ctx.now)) 531 | enqueue_requests(conn); 532 | if (FD_ISSET(fd, &readfds)) { 533 | uint8_t buf[4096]; 534 | struct msghdr mess; 535 | struct sockaddr sa; 536 | struct iovec vec; 537 | memset(&mess, 0, sizeof(mess)); 538 | mess.msg_name = &sa; 539 | mess.msg_namelen = sizeof(sa); 540 | vec.iov_base = buf; 541 | vec.iov_len = sizeof(buf); 542 | mess.msg_iov = &vec; 543 | mess.msg_iovlen = 1; 544 | ssize_t rret; 545 | while ((rret = recvmsg(fd, &mess, 0)) <= 0) 546 | ; 547 | if (verbosity >= 2) 548 | hexdump("recvmsg", buf, rret); 549 | size_t off = 0; 550 | while (off != rret) { 551 | quicly_decoded_packet_t packet; 552 | size_t plen = quicly_decode_packet(&ctx, &packet, buf + off, rret - off); 553 | if (plen == SIZE_MAX) 554 | break; 555 | quicly_receive(conn, NULL, &sa, &packet); 556 | off += plen; 557 | } 558 | } 559 | if (conn != NULL) { 560 | ret = send_pending(fd, conn); 561 | if (ret != 0) { 562 | quicly_free(conn); 563 | conn = NULL; 564 | if (ret == QUICLY_ERROR_FREE_CONNECTION) { 565 | if(fct_switch == 1){ 566 | if(gettimeofday(fct_tv_end_ptr, NULL)!=0) 567 | return 1; 568 | unsigned long flow_com_t = fct_tv_end_ptr->tv_sec*1000000+fct_tv_end_ptr->tv_usec - fct_tv_start_ptr->tv_sec*1000000 - fct_tv_start_ptr->tv_usec; 569 | printf("flow completion time: %luus\n", flow_com_t); 570 | fprintf(fd_fct, "%.3lf\n", (double)(192.0/(flow_com_t/1000000.0))); 571 | fclose(fd_fct); 572 | } 573 | return 0; 574 | } else { 575 | fprintf(stderr, "quicly_send returned %d\n", ret); 576 | return 1; 577 | } 578 | } 579 | } 580 | } 581 | } 582 | 583 | static quicly_conn_t **conns; 584 | static size_t num_conns = 0; 585 | 586 | static void on_signal(int signo) 587 | { 588 | size_t i; 589 | for (i = 0; i != num_conns; ++i) { 590 | const quicly_cid_plaintext_t *master_id = quicly_get_master_id(conns[i]); 591 | fprintf(stderr, "conn:%08" PRIu32 ": ", master_id->master_id); 592 | dump_stats(stderr, conns[i]); 593 | } 594 | if (signo == SIGINT) 595 | _exit(0); 596 | } 597 | 598 | static int validate_token(struct sockaddr *remote, ptls_iovec_t client_cid, ptls_iovec_t server_cid, 599 | quicly_address_token_plaintext_t *token) 600 | { 601 | int64_t age; 602 | int port_is_equal; 603 | 604 | if ((age = ctx.now->cb(ctx.now) - token->issued_at) < 0) 605 | age = 0; 606 | if (remote->sa_family != token->remote.sa.sa_family) 607 | return 0; 608 | switch (remote->sa_family) { 609 | case AF_INET: { 610 | struct sockaddr_in *sin = (struct sockaddr_in *)remote; 611 | if (sin->sin_addr.s_addr != token->remote.sin.sin_addr.s_addr) 612 | return 0; 613 | port_is_equal = sin->sin_port == token->remote.sin.sin_port; 614 | } break; 615 | case AF_INET6: { 616 | struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)remote; 617 | if (memcmp(&sin6->sin6_addr, &token->remote.sin6.sin6_addr, sizeof(sin6->sin6_addr)) != 0) 618 | return 0; 619 | port_is_equal = sin6->sin6_port == token->remote.sin6.sin6_port; 620 | } break; 621 | default: 622 | return 0; 623 | } 624 | if (token->is_retry) { 625 | if (age > 30000) 626 | return 0; 627 | if (!port_is_equal) 628 | return 0; 629 | uint64_t cidhash_actual; 630 | if (quicly_retry_calc_cidpair_hash(&ptls_openssl_sha256, client_cid, server_cid, &cidhash_actual) != 0) 631 | return 0; 632 | if (token->retry.cidpair_hash != cidhash_actual) 633 | return 0; 634 | } else { 635 | if (age > 10 * 60 * 1000) 636 | return 0; 637 | } 638 | return 1; 639 | } 640 | 641 | static int run_server(struct sockaddr *sa, socklen_t salen) 642 | { 643 | int fd; 644 | 645 | signal(SIGINT, on_signal); 646 | signal(SIGHUP, on_signal); 647 | 648 | if ((fd = socket(sa->sa_family, SOCK_DGRAM, IPPROTO_UDP)) == -1) { 649 | perror("socket(2) failed"); 650 | return 1; 651 | } 652 | int on = 1; 653 | if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) != 0) { 654 | perror("setsockopt(SO_REUSEADDR) failed"); 655 | return 1; 656 | } 657 | if (bind(fd, sa, salen) != 0) { 658 | perror("bind(2) failed"); 659 | return 1; 660 | } 661 | 662 | while (1) { 663 | fd_set readfds; 664 | struct timeval *tv, tvbuf; 665 | do { 666 | int64_t timeout_at = INT64_MAX; 667 | size_t i; 668 | for (i = 0; i != num_conns; ++i) { 669 | int64_t conn_to = quicly_get_first_timeout(conns[i]); 670 | if (conn_to < timeout_at) 671 | timeout_at = conn_to; 672 | } 673 | if (timeout_at != INT64_MAX) { 674 | int64_t delta = timeout_at - ctx.now->cb(ctx.now); 675 | if (delta > 0) { 676 | tvbuf.tv_sec = delta / 1000; 677 | tvbuf.tv_usec = (delta % 1000) * 1000; 678 | } else { 679 | tvbuf.tv_sec = 0; 680 | tvbuf.tv_usec = 0; 681 | } 682 | tv = &tvbuf; 683 | } else { 684 | tv = NULL; 685 | } 686 | FD_ZERO(&readfds); 687 | FD_SET(fd, &readfds); 688 | } while (select(fd + 1, &readfds, NULL, NULL, tv) == -1 && errno == EINTR); 689 | if (FD_ISSET(fd, &readfds)) { 690 | uint8_t buf[4096]; 691 | struct msghdr mess; 692 | struct sockaddr sa; 693 | struct iovec vec; 694 | memset(&mess, 0, sizeof(mess)); 695 | mess.msg_name = &sa; 696 | mess.msg_namelen = sizeof(sa); 697 | vec.iov_base = buf; 698 | vec.iov_len = sizeof(buf); 699 | mess.msg_iov = &vec; 700 | mess.msg_iovlen = 1; 701 | ssize_t rret; 702 | while ((rret = recvmsg(fd, &mess, 0)) <= 0) 703 | ; 704 | if (verbosity >= 2) 705 | hexdump("recvmsg", buf, rret); 706 | size_t off = 0; 707 | while (off != rret) { 708 | quicly_decoded_packet_t packet; 709 | size_t plen = quicly_decode_packet(&ctx, &packet, buf + off, rret - off); 710 | if (plen == SIZE_MAX) 711 | break; 712 | if (QUICLY_PACKET_IS_LONG_HEADER(packet.octets.base[0])) { 713 | if (packet.version != QUICLY_PROTOCOL_VERSION) { 714 | quicly_datagram_t *rp = 715 | quicly_send_version_negotiation(&ctx, &sa, packet.cid.src, NULL, packet.cid.dest.encrypted); 716 | assert(rp != NULL); 717 | if (send_one(fd, rp) == -1) 718 | perror("sendmsg failed"); 719 | break; 720 | } 721 | /* there is no way to send response to these v1 packets */ 722 | if (packet.cid.dest.encrypted.len > QUICLY_MAX_CID_LEN_V1 || packet.cid.src.len > QUICLY_MAX_CID_LEN_V1) 723 | break; 724 | } 725 | 726 | quicly_conn_t *conn = NULL; 727 | size_t i; 728 | for (i = 0; i != num_conns; ++i) { 729 | if (quicly_is_destination(conns[i], NULL, &sa, &packet)) { 730 | conn = conns[i]; 731 | break; 732 | } 733 | } 734 | if (conn != NULL) { 735 | /* existing connection */ 736 | quicly_receive(conn, NULL, &sa, &packet); 737 | } else if (QUICLY_PACKET_IS_INITIAL(packet.octets.base[0])) { 738 | /* long header packet; potentially a new connection */ 739 | quicly_address_token_plaintext_t *token = NULL, token_buf; 740 | if (packet.token.len != 0 && 741 | quicly_decrypt_address_token(address_token_aead.dec, &token_buf, packet.token.base, packet.token.len, 0) == 742 | 0 && 743 | validate_token(&sa, packet.cid.src, packet.cid.dest.encrypted, &token_buf)) 744 | token = &token_buf; 745 | if (enforce_retry && token == NULL && packet.cid.dest.encrypted.len >= 8) { 746 | /* unbound connection; send a retry token unless the client has supplied the correct one, but not too many 747 | */ 748 | uint8_t new_server_cid[8]; 749 | memcpy(new_server_cid, packet.cid.dest.encrypted.base, sizeof(new_server_cid)); 750 | new_server_cid[0] ^= 0xff; 751 | quicly_datagram_t *rp = 752 | quicly_send_retry(&ctx, address_token_aead.enc, &sa, packet.cid.src, NULL, 753 | ptls_iovec_init(new_server_cid, sizeof(new_server_cid)), packet.cid.dest.encrypted, 754 | ptls_iovec_init(NULL, 0), ptls_iovec_init(NULL, 0), NULL); 755 | assert(rp != NULL); 756 | if (send_one(fd, rp) == -1) 757 | perror("sendmsg failed"); 758 | break; 759 | } else { 760 | /* new connection */ 761 | int ret = quicly_accept(&conn, &ctx, NULL, &sa, &packet, token, &next_cid, NULL); 762 | if (ret == 0) { 763 | assert(conn != NULL); 764 | ++next_cid.master_id; 765 | conns = realloc(conns, sizeof(*conns) * (num_conns + 1)); 766 | assert(conns != NULL); 767 | conns[num_conns++] = conn; 768 | } else { 769 | assert(conn == NULL); 770 | } 771 | } 772 | } else if (!QUICLY_PACKET_IS_LONG_HEADER(packet.octets.base[0])) { 773 | /* short header packet; potentially a dead connection. No need to check the length of the incoming packet, 774 | * because loop is prevented by authenticating the CID (by checking node_id and thread_id). If the peer is also 775 | * sending a reset, then the next CID is highly likely to contain a non-authenticating CID, ... */ 776 | if (packet.cid.dest.plaintext.node_id == 0 && packet.cid.dest.plaintext.thread_id == 0) { 777 | quicly_datagram_t *dgram = quicly_send_stateless_reset(&ctx, &sa, NULL, packet.cid.dest.encrypted.base); 778 | if (send_one(fd, dgram) == -1) 779 | perror("sendmsg failed"); 780 | } 781 | } 782 | off += plen; 783 | } 784 | } 785 | { 786 | size_t i; 787 | for (i = 0; i != num_conns; ++i) { 788 | if (quicly_get_first_timeout(conns[i]) <= ctx.now->cb(ctx.now)) { 789 | if (send_pending(fd, conns[i]) != 0) { 790 | dump_stats(stderr, conns[i]); 791 | quicly_free(conns[i]); 792 | memmove(conns + i, conns + i + 1, (num_conns - i - 1) * sizeof(*conns)); 793 | --i; 794 | --num_conns; 795 | } 796 | } 797 | } 798 | } 799 | } 800 | } 801 | 802 | static void load_session(void) 803 | { 804 | static uint8_t buf[65536]; 805 | size_t len; 806 | int ret; 807 | 808 | { 809 | FILE *fp; 810 | if ((fp = fopen(session_file, "rb")) == NULL) 811 | return; 812 | len = fread(buf, 1, sizeof(buf), fp); 813 | if (len == 0 || !feof(fp)) { 814 | fprintf(stderr, "failed to load ticket from file:%s\n", session_file); 815 | exit(1); 816 | } 817 | fclose(fp); 818 | } 819 | 820 | { 821 | const uint8_t *src = buf, *end = buf + len; 822 | ptls_iovec_t ticket; 823 | ptls_decode_open_block(src, end, 2, { 824 | ticket = ptls_iovec_init(src, end - src); 825 | src = end; 826 | }); 827 | ptls_decode_open_block(src, end, 2, { 828 | if ((ret = quicly_decode_transport_parameter_list(&resumed_transport_params, NULL, NULL, 1, src, end)) != 0) 829 | goto Exit; 830 | src = end; 831 | }); 832 | ptls_decode_open_block(src, end, 2, { 833 | if ((resumption_token.len = end - src) != 0) { 834 | resumption_token.base = malloc(resumption_token.len); 835 | memcpy(resumption_token.base, src, resumption_token.len); 836 | } 837 | src = end; 838 | }); 839 | hs_properties.client.session_ticket = ticket; 840 | } 841 | 842 | Exit:; 843 | } 844 | 845 | static struct { 846 | ptls_iovec_t tls_ticket; 847 | ptls_iovec_t address_token; 848 | } session_info; 849 | 850 | int save_session(const quicly_transport_parameters_t *transport_params) 851 | { 852 | ptls_buffer_t buf; 853 | FILE *fp = NULL; 854 | int ret; 855 | 856 | if (session_file == NULL) 857 | return 0; 858 | 859 | ptls_buffer_init(&buf, "", 0); 860 | 861 | /* build data (session ticket and transport parameters) */ 862 | ptls_buffer_push_block(&buf, 2, { ptls_buffer_pushv(&buf, session_info.tls_ticket.base, session_info.tls_ticket.len); }); 863 | ptls_buffer_push_block(&buf, 2, { 864 | if ((ret = quicly_encode_transport_parameter_list(&buf, 1, transport_params, NULL, NULL, 0)) != 0) 865 | goto Exit; 866 | }); 867 | ptls_buffer_push_block(&buf, 2, { ptls_buffer_pushv(&buf, session_info.address_token.base, session_info.address_token.len); }); 868 | 869 | /* write file */ 870 | if ((fp = fopen(session_file, "wb")) == NULL) { 871 | fprintf(stderr, "failed to open file:%s:%s\n", session_file, strerror(errno)); 872 | ret = PTLS_ERROR_LIBRARY; 873 | goto Exit; 874 | } 875 | fwrite(buf.base, 1, buf.off, fp); 876 | 877 | ret = 0; 878 | Exit: 879 | if (fp != NULL) 880 | fclose(fp); 881 | ptls_buffer_dispose(&buf); 882 | return 0; 883 | } 884 | 885 | int save_session_ticket_cb(ptls_save_ticket_t *_self, ptls_t *tls, ptls_iovec_t src) 886 | { 887 | free(session_info.tls_ticket.base); 888 | session_info.tls_ticket = ptls_iovec_init(malloc(src.len), src.len); 889 | memcpy(session_info.tls_ticket.base, src.base, src.len); 890 | 891 | quicly_conn_t *conn = *ptls_get_data_ptr(tls); 892 | return save_session(quicly_get_peer_transport_parameters(conn)); 893 | } 894 | 895 | static int save_resumption_token_cb(quicly_save_resumption_token_t *_self, quicly_conn_t *conn, ptls_iovec_t token) 896 | { 897 | free(session_info.address_token.base); 898 | session_info.address_token = ptls_iovec_init(malloc(token.len), token.len); 899 | memcpy(session_info.address_token.base, token.base, token.len); 900 | 901 | return save_session(quicly_get_peer_transport_parameters(conn)); 902 | } 903 | 904 | static quicly_save_resumption_token_t save_resumption_token = {save_resumption_token_cb}; 905 | 906 | static int on_client_hello_cb(ptls_on_client_hello_t *_self, ptls_t *tls, ptls_on_client_hello_parameters_t *params) 907 | { 908 | int ret; 909 | 910 | if (negotiated_protocols.count != 0) { 911 | size_t i, j; 912 | const ptls_iovec_t *x, *y; 913 | for (i = 0; i != negotiated_protocols.count; ++i) { 914 | x = negotiated_protocols.list + i; 915 | for (j = 0; j != params->negotiated_protocols.count; ++j) { 916 | y = params->negotiated_protocols.list + j; 917 | if (x->len == y->len && memcmp(x->base, y->base, x->len) == 0) 918 | goto ALPN_Found; 919 | } 920 | } 921 | return PTLS_ALERT_NO_APPLICATION_PROTOCOL; 922 | ALPN_Found: 923 | if ((ret = ptls_set_negotiated_protocol(tls, (const char *)x->base, x->len)) != 0) 924 | return ret; 925 | } 926 | 927 | return 0; 928 | } 929 | 930 | static void usage(const char *cmd) 931 | { 932 | printf("Usage: %s [options] host port\n" 933 | "\n" 934 | "Options:\n" 935 | " -a ALPN identifier; repeat the option to set multiple\n" 936 | " candidates\n" 937 | " -C CID encryption key (server-only). Randomly generated\n" 938 | " if omitted.\n" 939 | " -c certificate-file\n" 940 | " -k key-file specifies the credentials to be used for running the\n" 941 | " server. If omitted, the command runs as a client.\n" 942 | " -K num-packets perform key update every num-packets packets\n" 943 | " -e event-log-file file to log events\n" 944 | " -E expand Client Hello (sends multiple client Initials)\n" 945 | " -i interval interval to reissue requests (in milliseconds)\n" 946 | " -I timeout idle timeout (in milliseconds; default: 600,000)\n" 947 | " -l log-file file to log traffic secrets\n" 948 | " -M max stream data (in bytes; default: 1MB)\n" 949 | " -m max data (in bytes; default: 16MB)\n" 950 | " -N enforce HelloRetryRequest (client-only)\n" 951 | " -n enforce version negotiation (client-only)\n" 952 | " -p path path to request (can be set multiple times)\n" 953 | " -P path path to request, store response to file (can be set multiple times)\n" 954 | " -R require Retry (server only)\n" 955 | " -r [initial-pto] initial PTO (in milliseconds)\n" 956 | " -S [num-speculative-ptos] number of speculative PTOs\n" 957 | " -s session-file file to load / store the session ticket\n" 958 | " -V verify peer using the default certificates\n" 959 | " -v verbose mode (-vv emits packet dumps as well)\n" 960 | " -x named-group named group to be used (default: secp256r1)\n" 961 | " -X max bidirectional stream count (default: 100)\n" 962 | " -h print this help\n" 963 | "\n", 964 | cmd); 965 | } 966 | 967 | static void push_req(const char *path, int to_file) 968 | { 969 | size_t i; 970 | for (i = 0; reqs[i].path != NULL; ++i) 971 | ; 972 | reqs = realloc(reqs, sizeof(*reqs) * (i + 2)); 973 | reqs[i].path = path; 974 | reqs[i].to_file = to_file; 975 | memset(reqs + i + 1, 0, sizeof(*reqs)); 976 | } 977 | 978 | int main(int argc, char **argv) 979 | { 980 | const char *host, *port, *cid_key = NULL; 981 | struct sockaddr_storage sa; 982 | socklen_t salen; 983 | int ch; 984 | 985 | reqs = malloc(sizeof(*reqs)); 986 | memset(reqs, 0, sizeof(*reqs)); 987 | ctx = quicly_spec_context; 988 | ctx.tls = &tlsctx; 989 | ctx.stream_open = &stream_open; 990 | ctx.closed_by_peer = &closed_by_peer; 991 | ctx.save_resumption_token = &save_resumption_token; 992 | ctx.generate_resumption_token = &generate_resumption_token; 993 | 994 | setup_session_cache(ctx.tls); 995 | quicly_amend_ptls_context(ctx.tls); 996 | 997 | { 998 | uint8_t secret[PTLS_MAX_DIGEST_SIZE]; 999 | ctx.tls->random_bytes(secret, ptls_openssl_sha256.digest_size); 1000 | address_token_aead.enc = ptls_aead_new(&ptls_openssl_aes128gcm, &ptls_openssl_sha256, 1, secret, ""); 1001 | address_token_aead.dec = ptls_aead_new(&ptls_openssl_aes128gcm, &ptls_openssl_sha256, 0, secret, ""); 1002 | } 1003 | 1004 | while ((ch = getopt(argc, argv, "a:C:c:k:K:Ee:i:I:l:M:m:Nnp:P:Rr:S:s:Vvox:X:h")) != -1) { 1005 | switch (ch) { 1006 | case 'a': 1007 | assert(negotiated_protocols.count < sizeof(negotiated_protocols.list) / sizeof(negotiated_protocols.list[0])); 1008 | negotiated_protocols.list[negotiated_protocols.count++] = ptls_iovec_init(optarg, strlen(optarg)); 1009 | break; 1010 | case 'C': 1011 | cid_key = optarg; 1012 | break; 1013 | case 'c': 1014 | load_certificate_chain(ctx.tls, optarg); 1015 | break; 1016 | case 'k': 1017 | load_private_key(ctx.tls, optarg); 1018 | break; 1019 | case 'K': 1020 | if (sscanf(optarg, "%" PRIu64, &ctx.max_packets_per_key) != 1) { 1021 | fprintf(stderr, "failed to parse key update interval: %s\n", optarg); 1022 | exit(1); 1023 | } 1024 | break; 1025 | case 'E': 1026 | ctx.expand_client_hello = 1; 1027 | break; 1028 | case 'e': 1029 | if ((quicly_trace_fp = fopen(optarg, "w")) == NULL) { 1030 | fprintf(stderr, "failed to open file:%s:%s\n", optarg, strerror(errno)); 1031 | exit(1); 1032 | } 1033 | setvbuf(quicly_trace_fp, NULL, _IONBF, 0); 1034 | break; 1035 | case 'i': 1036 | if (sscanf(optarg, "%" SCNd64, &request_interval) != 1) { 1037 | fprintf(stderr, "failed to parse request interval: %s\n", optarg); 1038 | exit(1); 1039 | } 1040 | break; 1041 | case 'I': 1042 | if (sscanf(optarg, "%" SCNd64, &ctx.transport_params.max_idle_timeout) != 1) { 1043 | fprintf(stderr, "failed to parse idle timeout: %s\n", optarg); 1044 | exit(1); 1045 | } 1046 | case 'l': 1047 | setup_log_event(ctx.tls, optarg); 1048 | break; 1049 | case 'M': { 1050 | uint64_t v; 1051 | if (sscanf(optarg, "%" SCNu64, &v) != 1) { 1052 | fprintf(stderr, "failed to parse max stream data:%s\n", optarg); 1053 | exit(1); 1054 | } 1055 | ctx.transport_params.max_stream_data.bidi_local = v; 1056 | ctx.transport_params.max_stream_data.bidi_remote = v; 1057 | ctx.transport_params.max_stream_data.uni = v; 1058 | } break; 1059 | case 'm': 1060 | if (sscanf(optarg, "%" SCNu64, &ctx.transport_params.max_data) != 1) { 1061 | fprintf(stderr, "failed to parse max data:%s\n", optarg); 1062 | exit(1); 1063 | } 1064 | break; 1065 | case 'N': 1066 | hs_properties.client.negotiate_before_key_exchange = 1; 1067 | break; 1068 | case 'n': 1069 | ctx.enforce_version_negotiation = 1; 1070 | break; 1071 | case 'p': 1072 | case 'P': { 1073 | if (!validate_path(optarg)) { 1074 | fprintf(stderr, "invalid path:%s\n", optarg); 1075 | exit(1); 1076 | } 1077 | push_req(optarg, ch == 'P'); 1078 | } break; 1079 | case 'R': 1080 | enforce_retry = 1; 1081 | break; 1082 | case 'r': 1083 | if (sscanf(optarg, "%" SCNu32, &ctx.loss.default_initial_rtt) != 1) { 1084 | fprintf(stderr, "invalid argument passed to `-r`\n"); 1085 | exit(1); 1086 | } 1087 | break; 1088 | case 'S': 1089 | if (sscanf(optarg, "%" SCNu8, &ctx.loss.num_speculative_ptos) != 1) { 1090 | fprintf(stderr, "invalid argument passed to `-S`\n"); 1091 | exit(1); 1092 | } 1093 | break; 1094 | case 's': 1095 | session_file = optarg; 1096 | break; 1097 | case 'V': 1098 | setup_verify_certificate(ctx.tls); 1099 | break; 1100 | case 'v': 1101 | ++verbosity; 1102 | break; 1103 | case 'o': 1104 | fct_switch = 1; 1105 | break; 1106 | case 'x': { 1107 | size_t i; 1108 | for (i = 0; key_exchanges[i] != NULL; ++i) 1109 | ; 1110 | #define MATCH(name) \ 1111 | if (key_exchanges[i] == NULL && strcasecmp(optarg, #name) == 0) \ 1112 | key_exchanges[i] = &ptls_openssl_##name 1113 | MATCH(secp256r1); 1114 | #if PTLS_OPENSSL_HAVE_SECP384R1 1115 | MATCH(secp384r1); 1116 | #endif 1117 | #if PTLS_OPENSSL_HAVE_SECP521R1 1118 | MATCH(secp521r1); 1119 | #endif 1120 | #if PTLS_OPENSSL_HAVE_X25519 1121 | MATCH(x25519); 1122 | #endif 1123 | #undef MATCH 1124 | if (key_exchanges[i] == NULL) { 1125 | fprintf(stderr, "unknown key exchange: %s\n", optarg); 1126 | exit(1); 1127 | } 1128 | } break; 1129 | case 'X': 1130 | if (sscanf(optarg, "%" SCNu64, &ctx.transport_params.max_streams_bidi) != 1) { 1131 | fprintf(stderr, "failed to parse max streams count: %s\n", optarg); 1132 | exit(1); 1133 | } 1134 | break; 1135 | default: 1136 | usage(argv[0]); 1137 | exit(1); 1138 | } 1139 | } 1140 | argc -= optind; 1141 | argv += optind; 1142 | 1143 | if (reqs[0].path == NULL) 1144 | push_req("/", 0); 1145 | 1146 | if (key_exchanges[0] == NULL) 1147 | key_exchanges[0] = &ptls_openssl_secp256r1; 1148 | 1149 | if (ctx.tls->certificates.count != 0 || ctx.tls->sign_certificate != NULL) { 1150 | /* server */ 1151 | if (ctx.tls->certificates.count == 0 || ctx.tls->sign_certificate == NULL) { 1152 | fprintf(stderr, "-c and -k options must be used together\n"); 1153 | exit(1); 1154 | } 1155 | if (cid_key == NULL) { 1156 | static char random_key[17]; 1157 | tlsctx.random_bytes(random_key, sizeof(random_key) - 1); 1158 | cid_key = random_key; 1159 | } 1160 | ctx.cid_encryptor = quicly_new_default_cid_encryptor(&ptls_openssl_bfecb, &ptls_openssl_aes128ecb, &ptls_openssl_sha256, 1161 | ptls_iovec_init(cid_key, strlen(cid_key))); 1162 | } else { 1163 | /* client */ 1164 | hs_properties.client.negotiated_protocols.list = negotiated_protocols.list; 1165 | hs_properties.client.negotiated_protocols.count = negotiated_protocols.count; 1166 | if (session_file != NULL) 1167 | load_session(); 1168 | } 1169 | if (argc != 2) { 1170 | fprintf(stderr, "missing host and port\n"); 1171 | exit(1); 1172 | } 1173 | host = (--argc, *argv++); 1174 | port = (--argc, *argv++); 1175 | 1176 | if (resolve_address((void *)&sa, &salen, host, port, AF_INET, SOCK_DGRAM, IPPROTO_UDP) != 0) 1177 | exit(1); 1178 | 1179 | return ctx.tls->certificates.count != 0 ? run_server((void *)&sa, salen) : run_client((void *)&sa, host); 1180 | } 1181 | -------------------------------------------------------------------------------- /kits/quicly/client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #a stands for the repetitions 3 | a=$1 4 | #b stands for the tags (loss rate, etc.) 5 | b=$2 6 | #generate a fixed size output.dat file. 7 | dd if /dev/zero of=output.dat bs=50M count=1 8 | echo "+++${b}" >> flow_completion_time.txt 9 | for i in $(eval echo {0..${a}}) 10 | do 11 | #perf record --call-graph lbr -- taskset 16 ./cli 192.168.0.1 4433 -p /output.dat -o 12 | taskset 128 ./cli 192.168.0.1 4433 -p /output.dat -o 13 | myval=$(echo "2^$i"|bc) 14 | #taskset $myval ./cli 192.168.0.1 4433 -p /output.dat -o & 15 | #sleep 0.05 16 | done 17 | sleep 5 18 | echo "---${b}" >> flow_completion_time.txt 19 | -------------------------------------------------------------------------------- /kits/quicly/config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #put the instrumented source file into the proj 3 | mv ../../extern/quicly/src/cli.c ../../extern/quicly/src/cli.c.bak 4 | cp cli.c ../../extern/quicly/src -------------------------------------------------------------------------------- /kits/quicly/multi_clients.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # a stands for the overall connections during the test. 4 | a=$1 5 | ifconfig ens3f1 192.168.0.4 6 | for i in {1..$a} 7 | do 8 | #perf record --call-graph lbr -- taskset 16 ./cli 192.168.0.1 4433 -p /output.dat -o 9 | #taskset 16 ./cli 192.168.0.1 4433 -p /output.dat -o 10 | myval=$(echo "2^$i"|bc) 11 | taskset $myval ./cli 192.168.0.1 4433 -p /output.dat -o & 12 | #sleep 0.05 13 | done 14 | echo "---${a}" >> flow_completion_time.txt 15 | -------------------------------------------------------------------------------- /kits/quicly/server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #replace `cert` and `key` with the actual ones. 3 | ip netns exec blue taskset 128 ./cli -c cert -k key ./cli 192.168.0.1 4433 -------------------------------------------------------------------------------- /unfinished/data_process.py: -------------------------------------------------------------------------------- 1 | import sys 2 | global aver 3 | global sume 4 | def calc(rnd): 5 | print(rnd) 6 | f = open("Throughput.txt", "r") 7 | items = [] 8 | start = 0 9 | end = 0 10 | for line in f: 11 | if("+++" in line): 12 | if int(line[3:]) == rnd: 13 | start = 1 14 | end = 0 15 | else: 16 | start = 0 17 | continue 18 | elif start == 1: 19 | if (line[0:2] in "---"): 20 | if int(line[3:])==rnd: 21 | end = 1 22 | start = 0 23 | if end == 0: 24 | item = float(line) 25 | items.append(item) 26 | else: 27 | continue 28 | aver.append(sum(items)/len(items)) 29 | sume.append(sum(items)) 30 | f.close() 31 | 32 | if __name__ == "__main__": 33 | aver = [] 34 | sume = [] 35 | t_c = [5,10,20,30,40,50,60,70] 36 | for i in range(0,11): 37 | calc(i) 38 | print("average:", str(aver)) 39 | print("sum:", str(sume)) 40 | print("finished!") 41 | -------------------------------------------------------------------------------- /unfinished/main.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import sys 4 | from pathlib import Path 5 | import shutil 6 | 7 | # #demos for shutil here: 8 | # # $ mv src dest 9 | # shutil.move('src', 'dest') 10 | # # $ cp src dest 11 | # shutil.copy2('src', 'dest') 12 | # # $ cp -r src dest 13 | # shutil.copytree('src', 'dest') 14 | # # $ rm a_file 15 | # os.remove('a_file') 16 | # # $ rm -r a_dir 17 | # shutil.rmtree('a_dir') 18 | # # $ tar caf 'my_archive.tar.gz' 'my_folder' 19 | # shutil.make_archive('my_archive.tar.gz', 'gztar', 'my_folder') 20 | # # $ tar xaf 'my_archive.tar.gz' 21 | # shutil.unpack_archive('my_archive.tar.gz') 22 | # # chown user:ninjaaron a_file.txt 23 | # shutil.chown('a_file.txt', 'ninjaaron', 'user') 24 | # # info about disk usage, a bit like `df`, but not exactly. 25 | # shutil.disk_usage('.') 26 | # # which vi 27 | # shutil.which('vi') 28 | 29 | p = Path() 30 | if(len(sys.argv) == 1): 31 | print("need to specify one implementation.") 32 | exit 33 | elif(len(sys.argv)>2): 34 | print("too many arguments.") 35 | exit 36 | else: 37 | if(sys.argv[1] == 'quant'): 38 | # mv the modified to make the quant client/server binary file. 39 | shutil.move(p/'extern/instrumentations/quant_client.c', p/'extern/instrumentations/client.c') 40 | shutil.copy2(p/'extern/instrumentations/client.c', p/'quant/bin/') 41 | # Makefile 42 | elif(sys.argv[1] == 'quicly'): 43 | pass 44 | 45 | ''' 46 | p = Path() 47 | #p = p.absolute() 48 | print(p.stat()) 49 | print(oct(p.stat().st_mode)) 50 | readu = p/'hah.txt' 51 | with readu.open(mode='w') as fd: 52 | fd.writelines('just a test') 53 | readu.chmod(0o777) 54 | ''' 55 | 56 | # argv[1]: choose the quic implementation 57 | 58 | #choose the quic implementation 59 | 60 | #select the instrumented file and replace the original file 61 | 62 | #execute the makefile process and generate the binary file 63 | 64 | # probably this step should be done by bash instead of Python according to: 65 | # https://github.com/ninjaaron/replacing-bash-scripting-with-python#command-line-interfaces 66 | 67 | #remind the user on how to run the measurement 68 | 69 | #lots of text on how to get the result --------------------------------------------------------------------------------