├── CMakeLists.txt ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── FlowCache.h ├── FlowCacheHealthCheck.cpp ├── GeneveHandler.cpp ├── GeneveHandler.h ├── GenevePacket.cpp ├── GenevePacket.h ├── HISTORY.md ├── HealthCheck.h ├── LICENSE ├── Logger.cpp ├── Logger.h ├── NOTICE ├── PacketHeaderV4.cpp ├── PacketHeaderV4.h ├── PacketHeaderV6.cpp ├── PacketHeaderV6.h ├── README.md ├── TunInterface.cpp ├── TunInterface.h ├── UDPPacketReceiver.cpp ├── UDPPacketReceiver.h ├── example-scripts ├── create-nat-overlapping.sh ├── create-nat.sh ├── create-overlapping-v4.sh ├── create-passthrough.sh ├── create-route.sh ├── example-topology-one-arm.template └── example-topology-two-arm.template ├── json.hpp ├── main.cpp ├── utils.cpp └── utils.h /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.13) 2 | project(gwlbtun) 3 | set(CMAKE_CXX_STANDARD 17) 4 | find_package(Threads) 5 | 6 | set(Boost_INCLUDE_DIR /home/ec2-user/boost) 7 | find_package(Boost 1.83 REQUIRED) 8 | if(Boost_FOUND) 9 | include_directories(${Boost_INCLUDE_DIRS}) 10 | endif() 11 | 12 | set(CMAKE_CXX_FLAGS "-Wall -Wextra") 13 | set(CMAKE_CXX_FLAGS_RELEASE "-Ofast") 14 | 15 | set(CMAKE_CXX_FLAGS_DEBUG "-fstack-protector-all -fsanitize=address -pg -O0 -g") 16 | set(CMAKE_EXE_LINKER_FLAGS_DEBUG "-lasan") 17 | 18 | add_compile_options("") 19 | 20 | add_executable(gwlbtun main.cpp UDPPacketReceiver.cpp UDPPacketReceiver.h GenevePacket.cpp GenevePacket.h TunInterface.cpp TunInterface.h utils.cpp utils.h PacketHeaderV4.cpp PacketHeaderV4.h PacketHeaderV6.cpp PacketHeaderV6.h GeneveHandler.cpp GeneveHandler.h FlowCache.h FlowCacheHealthCheck.cpp Logger.cpp Logger.h 21 | HealthCheck.h 22 | FlowCacheHealthCheck.cpp) 23 | target_link_libraries (gwlbtun ${CMAKE_THREAD_LIBS_INIT}) 24 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /FlowCache.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * Templated class to handle both our IPv4 and IPv6 flow caches. This is broken out to make experimenting with different 6 | * data structures in terms of performance much easier. Used by GeneveHandler. 7 | */ 8 | 9 | #ifndef GWLBTUN_FLOWCACHE_H 10 | #define GWLBTUN_FLOWCACHE_H 11 | 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include "HealthCheck.h" 19 | 20 | class FlowCacheHealthCheck : public HealthCheck { 21 | public: 22 | FlowCacheHealthCheck(std::string, int, long unsigned int, long unsigned int); 23 | std::string output_str() ; 24 | json output_json(); 25 | 26 | private: 27 | std::string cacheName; 28 | int cacheTimeout; 29 | long unsigned int size; 30 | long unsigned int timedOut; 31 | }; 32 | 33 | /** 34 | * Cache entry format 35 | */ 36 | template class FlowCacheEntry { 37 | public: 38 | FlowCacheEntry(V entrydata); 39 | 40 | time_t last; 41 | uint32_t useCount; 42 | V data; 43 | }; 44 | 45 | /** 46 | * Cache entry functions 47 | */ 48 | template FlowCacheEntry::FlowCacheEntry(V entrydata) : 49 | last(time(NULL)), useCount(1), data(entrydata) 50 | { 51 | } 52 | 53 | /** 54 | * FlowCache itself 55 | * @tparam K Key type 56 | * @tparam V Value type 57 | */ 58 | template class FlowCache { 59 | public: 60 | FlowCache(std::string cacheName, int cacheTimeout); 61 | V lookup(K key); 62 | V emplace_or_lookup(K key, V value); 63 | FlowCacheHealthCheck check(); 64 | private: 65 | const int cacheTimeout; 66 | const std::string cacheName; 67 | boost::concurrent_flat_map> cache; 68 | }; 69 | 70 | /** 71 | * Initializer. 72 | * @param cacheName Human name of this cache, used for diagnostic outputs. 73 | */ 74 | template 75 | FlowCache::FlowCache(std::string cacheName, int cacheTimeout) : 76 | cacheTimeout(cacheTimeout), cacheName(std::move(cacheName)) 77 | { 78 | } 79 | 80 | /** 81 | * Look up a value for key K in our cache. Raises invalid_argument exception if key not present. 82 | * 83 | * @param key Key to lookup 84 | * @return Value if present, raises invalid_argument exception if key not present. 85 | */ 86 | templateV FlowCache::lookup(K key) 87 | { 88 | V ret; 89 | if(cache.visit(key, [&](auto& fce) { fce.second.last = time(NULL); fce.second.useCount ++; ret = fce.second.data; })) 90 | return ret; 91 | else 92 | throw std::invalid_argument("Key not found in FlowCache."); 93 | } 94 | 95 | /** 96 | * Looks up a value for key K in our cache. If not present, insert with value V. 97 | * 98 | * @param K Key to lookup 99 | * @param V Value to insert if K is not present. 100 | * @return Value (either the one looked up, or the inserted data, as appropriate). 101 | */ 102 | templateV FlowCache::emplace_or_lookup(K key, V value) 103 | { 104 | V ret; 105 | if(!(cache.emplace_or_visit(key, value, [&](auto& fce) { fce.second.last = time(NULL); fce.second.useCount ++; ret = fce.second.data; }))) 106 | return ret; 107 | return value; 108 | } 109 | 110 | /** 111 | * Return diagnostic status of our cache. Needs to be called occasionally as it does cleanup as a side effect. 112 | * 113 | * @return String of diagnostic text. 114 | */ 115 | template FlowCacheHealthCheck FlowCache::check() 116 | { 117 | long unsigned int timedOut; 118 | time_t expireTime = time(NULL) - cacheTimeout; 119 | 120 | timedOut = cache.erase_if([expireTime](auto& fce) { return fce.second.last < expireTime; }); 121 | 122 | return { cacheName, cacheTimeout, cache.size(), timedOut }; 123 | } 124 | 125 | #endif //GWLBTUN_FLOWCACHE_H 126 | -------------------------------------------------------------------------------- /FlowCacheHealthCheck.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #include "FlowCache.h" 5 | #include "HealthCheck.h" 6 | 7 | FlowCacheHealthCheck::FlowCacheHealthCheck(std::string cacheName, int cacheTimeout, long unsigned int size, long unsigned int timedOut) : 8 | cacheName(cacheName), cacheTimeout(cacheTimeout), size(size), timedOut(timedOut) 9 | { 10 | } 11 | 12 | std::string FlowCacheHealthCheck::output_str() 13 | { 14 | return cacheName + " : Contains " + std::to_string(size) + " elements, of which " + std::to_string(timedOut) + " were just timed out (idle timeout is " + std::to_string(cacheTimeout) + " seconds).\n"; 15 | } 16 | 17 | json FlowCacheHealthCheck::output_json() 18 | { 19 | return {{"cacheName", cacheName}, {"size", size}, {"timedOut", timedOut}, "idleTimeoutSecs", cacheTimeout }; 20 | } 21 | -------------------------------------------------------------------------------- /GeneveHandler.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * Handles all of our Geneve tunnel functions: 6 | * - Launches a UDPPacketReceiver to receive packets on port 6081 7 | * - For each VNI received, it starts a new Tun interface named gwi-, and does a callback 8 | * - For each packet received over the UDPPacketReceiver, decode, store the resulting flowCookie, and send to the gwi- tunnel interface to the OS 9 | * - For each packet received via a Tun interface, encode if possible, and send to GWLB. 10 | * 11 | * Also provides the GwlbData class, which stores PacketHeaders with their matching GenevePacket so that the GENEVE 12 | * options can be reapplied to matching traffic. 13 | */ 14 | 15 | #include "GeneveHandler.h" 16 | #include "utils.h" 17 | #include 18 | #include 19 | #include 20 | #include "Logger.h" 21 | 22 | using namespace std::string_literals; 23 | 24 | #define GWLB_MTU 8500 // MTU of customer payload packets that can be processed 25 | #define GENEVE_PORT 6081 // UDP port number that GENEVE uses by standard 26 | 27 | /** 28 | * Empty GwlbData initializer. Needed as we move-assign on occasion. 29 | */ 30 | GwlbData::GwlbData() {} 31 | 32 | /** 33 | * Build a GwlbData structure. Stores the data, and sets the lastSeen timer to now. 34 | * 35 | * @param gp GenevePacket to store 36 | * @param srcAddr Source address of the GENEVE packet 37 | * @param srcPort Source port of the GENEVE packet 38 | * @param dstAddr Destination address of the GENEVE packet 39 | * @param dstPort Destination port of the GENEVE packet 40 | */ 41 | GwlbData::GwlbData(GenevePacket &gp, struct in_addr *srcAddr, uint16_t srcPort, struct in_addr *dstAddr, uint16_t dstPort) : 42 | gp(gp), srcAddr(*srcAddr), srcPort(srcPort), dstAddr(*dstAddr), dstPort(dstPort) 43 | { 44 | } 45 | 46 | /** 47 | * Starts the GeneveHandler. Builds a UDPPacketReceiver on port 6081 with callbacks in this class to handle packets 48 | * as they come in. 49 | * 50 | * @param createCallback Function to call when a new endpoint is seen. 51 | * @param destroyCallback Function to call when an endpoint has gone away and we need to clean up. 52 | * @param destroyTimeout How long to wait for an endpoint to be idle before calling destroyCallback. 53 | */ 54 | GeneveHandler::GeneveHandler(ghCallback createCallback, ghCallback destroyCallback, int destroyTimeout, int cacheTimeout, ThreadConfig udpThreads, ThreadConfig tunThreads) 55 | : healthy(true), 56 | createCallback(std::move(createCallback)), destroyCallback(std::move(destroyCallback)), eniDestroyTimeout(destroyTimeout), cacheTimeout(cacheTimeout), 57 | tunThreadConfig(std::move(tunThreads)) 58 | { 59 | // Set up UDP receiver threads. 60 | udpRcvr.setup(udpThreads, GENEVE_PORT, std::bind(&GeneveHandler::udpReceiverCallback, this, std::placeholders::_1, std::placeholders::_2, std::placeholders::_3, std::placeholders::_4, std::placeholders::_5, std::placeholders::_6)); 61 | } 62 | 63 | /** 64 | * Perform a health check of the GeneveHandler and all components it is using. 65 | * 66 | * @return A human-readable string of the health status. 67 | */ 68 | GeneveHandlerHealthCheck::GeneveHandlerHealthCheck(bool healthy, UDPPacketReceiverHealthCheck udp, std::list enis) : 69 | healthy(healthy), udp(std::move(udp)), enis(std::move(enis)) 70 | { 71 | } 72 | 73 | std::string GeneveHandlerHealthCheck::output_str() 74 | { 75 | std::string ret; 76 | ret += udp.output_str(); 77 | 78 | for(auto &eni : enis) 79 | ret += eni.output_str(); 80 | 81 | return ret; 82 | } 83 | 84 | json GeneveHandlerHealthCheck::output_json() 85 | { 86 | json ret; 87 | 88 | ret = { {"udp", udp.output_json()}, {"enis", json::array()} }; 89 | 90 | for(auto &eni : enis) 91 | ret["enis"].push_back(eni.output_json()); 92 | 93 | return ret; 94 | } 95 | 96 | GeneveHandlerHealthCheck GeneveHandler::check() 97 | { 98 | LOG(LS_HEALTHCHECK, LL_DEBUG, "Health check starting"); 99 | 100 | std::list enis; 101 | 102 | // Clean up any ENI handlers that have apparently gone idle, if we're not keeping them around forever. 103 | if(eniDestroyTimeout > 0) 104 | eniHandlers.erase_if([&](auto& eniHandler) { return (*eniHandler.second.ptr).hasGoneIdle(eniDestroyTimeout); }); 105 | 106 | // Check remaining handlers. 107 | eniHandlers.visit_all([&enis](auto& eniHandler) { enis.push_back( (*eniHandler.second.ptr).check() ); ; }); 108 | 109 | return { udpRcvr.healthCheck(), udpRcvr.status(), enis }; 110 | } 111 | 112 | /** 113 | * Callback function passed to UDPPacketReceiver to handle GenevePackets. Detemrine which GeneveHandlerENI this packet 114 | * is for (creating a new one if needed), and pass to that class. 115 | * 116 | * @param pkt The packet received. 117 | * @param pktlen Length of packet received. 118 | * @param srcAddr Source address the packet came from. 119 | * @param srcPort Source port the packet came from. 120 | * @param dstAddr Destination address the packet was sent to. 121 | * @param dstPort Destination port the packet was sent to. 122 | */ 123 | void GeneveHandler::udpReceiverCallback(unsigned char *pkt, ssize_t pktlen, struct in_addr *srcAddr, uint16_t srcPort, struct in_addr *dstAddr, uint16_t dstPort) 124 | { 125 | if(IS_LOGGING(LS_GENEVE, LL_DEBUG)) 126 | { 127 | LOG(LS_GENEVE, LL_DEBUG, "Received a packet of "s + ts(pktlen) + " bytes from " + inet_ntoa(*srcAddr) + " port " + ts(srcPort) + " sent to " + inet_ntoa(*dstAddr) + " port " + ts(dstPort)); 128 | LOGHEXDUMP(LS_GENEVE, LL_DEBUGDETAIL, "GWLB Packet", pkt, pktlen); 129 | } 130 | try { 131 | auto gp = GenevePacket(pkt, pktlen); 132 | auto gd = GwlbData(gp, srcAddr, srcPort, dstAddr, dstPort); 133 | 134 | // The GenevePacket class does sanity checks to ensure this was a Geneve packet. Verify the result of those checks. 135 | if(gp.status != GP_STATUS_OK) 136 | { 137 | LOG(LS_GENEVE, LL_DEBUG, "Geneve Header not OK"); 138 | return; 139 | } 140 | 141 | if(!gp.gwlbeEniIdValid) 142 | { 143 | LOG(LS_GENEVE, LL_DEBUG, "GWLBe ENI ID not valid"); 144 | return; 145 | } 146 | 147 | // Figure out which GeneveHandlerENI this needs to dispatch to, creating if necessary 148 | if(eniHandlers.try_emplace_or_cvisit(gp.gwlbeEniId, gp.gwlbeEniId, cacheTimeout, tunThreadConfig, createCallback, destroyCallback, [&](const auto& eniHandler) 149 | { 150 | (*eniHandler.second.ptr).udpReceiverCallback(gd, pkt, pktlen); 151 | })) 152 | { 153 | // We did a create - redo the visit to do the call. 154 | eniHandlers.cvisit(gp.gwlbeEniId, [&](const auto& eniHandler) 155 | { 156 | (*eniHandler.second.ptr).udpReceiverCallback(gd, pkt, pktlen); 157 | }); 158 | } 159 | } 160 | catch (std::exception& e) { 161 | LOG(LS_TUNNEL, LL_CRITICAL, "Tunnel or ENI creation failed:"s + e.what()); 162 | } 163 | } 164 | 165 | 166 | /** 167 | * GeneveHandlerENI handles all aspects of handling for a given ENI. It is separated out this way to make dealing with 168 | * keeping all the resources needed on a per ENI basis easier. 169 | */ 170 | GeneveHandlerENI::GeneveHandlerENI(eniid_t eni, int cacheTimeout, ThreadConfig& tunThreadConfig, ghCallback createCallback, ghCallback destroyCallback) : 171 | eni(eni), eniStr(MakeENIStr(eni)), cacheTimeout(cacheTimeout), 172 | devInName(devname_make(eni, true)), 173 | #ifndef NO_RETURN_TRAFFIC 174 | devOutName(devname_make(eni, false)), 175 | gwlbV4Cookies("IPv4 Flow Cache for ENI " + eniStr, cacheTimeout), gwlbV6Cookies("IPv6 Flow Cache for ENI " + eniStr, cacheTimeout), 176 | #else 177 | devOutName("none"s), 178 | #endif 179 | createCallback(std::move(createCallback)), destroyCallback(std::move(destroyCallback)) 180 | { 181 | // Set up a socket we use for sending traffic out for ENI. 182 | tunnelIn = std::make_unique(devInName, GWLB_MTU, tunThreadConfig, std::bind(&GeneveHandlerENI::tunReceiverCallback, this, std::placeholders::_1, std::placeholders::_2)); 183 | #ifndef NO_RETURN_TRAFFIC 184 | tunnelOut = std::make_unique(devOutName, GWLB_MTU, tunThreadConfig, std::bind(&GeneveHandlerENI::tunReceiverCallback, this, std::placeholders::_1, std::placeholders::_2)); 185 | sendingSock = socket(AF_INET,SOCK_RAW, IPPROTO_RAW); 186 | if(sendingSock == -1) 187 | throw std::runtime_error("Unable to allocate a socket for sending UDP traffic."); 188 | #endif 189 | this->createCallback(devInName, devOutName, this->eni); 190 | } 191 | 192 | GeneveHandlerENI::~GeneveHandlerENI() 193 | { 194 | this->destroyCallback(devInName, devOutName, this->eni); 195 | } 196 | 197 | /** 198 | * Callback function passed to TunInterface to handle packets coming back in from the OS to either the gwi- or the 199 | * gwo- interface. Attempts to match the packet header to a seen flow (outptus a message and returns if none is found) 200 | * and then sends the packet correctly formed back to GWLB. 201 | * 202 | * @param eniId The ENI ID of the TUN interface 203 | * @param pkt The packet received. 204 | * @param pktlen Length of packet received. 205 | */ 206 | void GeneveHandlerENI::tunReceiverCallback(unsigned char *pktbuf, ssize_t pktlen) 207 | { 208 | LOG(LS_TUNNEL, LL_DEBUG, "Received a packet of " + ts(pktlen) + " bytes for ENI Id:" + eniStr); 209 | LOGHEXDUMP(LS_TUNNEL, LL_DEBUGDETAIL, "Tun Packet", pktbuf, pktlen); 210 | 211 | #ifdef NO_RETURN_TRAFFIC 212 | LOG(LS_TUNNEL, LL_DEBUG, "Received a packet, but NO_RETURN_TRAFFIC is defined. Discarding."); 213 | return; 214 | #else 215 | // Ignore packets that are not IPv4 or IPv6, or aren't at least long enough to have those sized headers. 216 | if( pktlen < 20 ) 217 | { 218 | LOG(LS_TUNNEL, LL_DEBUG, "Received a packet that is not long enough to have an IP header, ignoring."); 219 | return; 220 | } 221 | try 222 | { 223 | GwlbData gd; 224 | 225 | switch( (pktbuf[0] & 0xF0) >> 4) 226 | { 227 | case 4: 228 | { 229 | auto ph = PacketHeaderV4(pktbuf, pktlen); 230 | try { 231 | gd = gwlbV4Cookies.lookup(ph); 232 | } 233 | catch (std::invalid_argument &e) { 234 | LOG(LS_TUNNEL, LL_DEBUG, "Flow " + ph.text() + " has not been seen coming in from GWLB - dropping. (Remember - GWLB is for inline inspection only - you cannot source new flows from this device into it.)"); 235 | return; 236 | } 237 | LOG(LS_TUNNEL, LL_DEBUGDETAIL, "Resolved packet header " + ph.text() + " to options " + gd.gp.text()); 238 | break; 239 | } 240 | case 6: 241 | { 242 | auto ph = PacketHeaderV6(pktbuf, pktlen); 243 | try { 244 | gd = gwlbV6Cookies.lookup(ph); 245 | } 246 | catch (std::invalid_argument &e) { 247 | LOG(LS_TUNNEL, LL_DEBUG, "Flow " + ph.text() + " has not been seen coming in from GWLB - dropping. (Remember - GWLB is for inline inspection only - you cannot source new flows from this device into it.)"); 248 | return; 249 | } 250 | LOG(LS_TUNNEL, LL_DEBUGDETAIL, "Resolved packet header " + ph.text() + " to options " + gd.gp.text()); 251 | break; 252 | } 253 | default: 254 | { 255 | LOG(LS_TUNNEL, LL_DEBUG, "Received a packet that wasn't IPv4 or IPv6. Ignoring."); 256 | return; 257 | } 258 | } 259 | 260 | // Build the packet to send back to GWLB. 261 | // Following as per https://aws.amazon.com/blogs/networking-and-content-delivery/integrate-your-custom-logic-or-appliance-with-aws-gateway-load-balancer/ 262 | unsigned char *genevePkt = new unsigned char[pktlen + gd.gp.headerLen]; 263 | // Encapsulate this packet with the original Geneve header 264 | memcpy(genevePkt, &gd.gp.header.front(), gd.gp.headerLen); 265 | // Copy the packet in after the Geneve header. 266 | memcpy(genevePkt + gd.gp.headerLen, pktbuf, pktlen); 267 | // Swap source and destination IP addresses, but preserve ports, and send back to GWLB. 268 | sendUdp(sendingSock, gd.dstAddr, gd.srcPort, gd.srcAddr, gd.dstPort, genevePkt, pktlen + gd.gp.headerLen); 269 | delete[] genevePkt; 270 | } catch(std::invalid_argument& err) { 271 | LOG(LS_TUNNEL, LL_DEBUG, "Packet processor has a malformed packet: "s + err.what()); 272 | return; 273 | } 274 | #endif 275 | } 276 | 277 | /** 278 | * Callback function passed to UDPPacketReceiver to handle GenevePackets. Called by GeneveHandler once the ENI 279 | * has been determined (creating this class if needed) 280 | * 281 | * @param gd The GwlbData (it was processed by GeneveHandler to do its work) 282 | * @param pkt The packet received. 283 | * @param pktlen Length of packet received. 284 | */ 285 | void GeneveHandlerENI::udpReceiverCallback(const GwlbData &gd, unsigned char *pkt, ssize_t pktlen) 286 | { 287 | try { 288 | if( (pktlen - gd.gp.headerLen) > (ssize_t)sizeof(struct ip) ) 289 | { 290 | struct ip *iph = (struct ip *)(pkt + gd.gp.headerLen); 291 | if(iph->ip_v == (unsigned int)4) 292 | { 293 | #ifndef NO_RETURN_TRAFFIC 294 | auto ph = PacketHeaderV4(pkt + gd.gp.headerLen, pktlen - gd.gp.headerLen); 295 | // Ensure flow is in flow cache. 296 | auto gdret = gwlbV4Cookies.emplace_or_lookup(ph, gd); 297 | #endif 298 | // Route the decap'ed packet to our tun interface. 299 | (*tunnelIn).writePacket(pkt + gd.gp.headerLen, pktlen - gd.gp.headerLen); 300 | } else if(iph->ip_v == (unsigned int)6) { 301 | #ifndef NO_RETURN_TRAFFIC 302 | auto ph = PacketHeaderV6(pkt + gd.gp.headerLen, pktlen - gd.gp.headerLen); 303 | // Ensure flow is in flow cache. 304 | auto gdret = gwlbV6Cookies.emplace_or_lookup(ph, gd); 305 | #endif 306 | // Route the decap'ed packet to our tun interface. 307 | (*tunnelIn).writePacket(pkt + gd.gp.headerLen, pktlen - gd.gp.headerLen); 308 | } else { 309 | LOG(LS_UDP, LL_DEBUG, "Got a strange IP protocol version - "s + ts(iph->ip_v) + " at offset " + ts(gd.gp.headerLen) + ". Dropping packet."); 310 | } 311 | } 312 | } catch(std::invalid_argument& err) { 313 | LOG(LS_UDP, LL_DEBUG, "Packet processor has a malformed packet: "s + err.what()); 314 | return; 315 | } 316 | } 317 | 318 | /** 319 | * Perform a health check on this ENI, and return some information. 320 | * @return 321 | */ 322 | #ifndef NO_RETURN_TRAFFIC 323 | GeneveHandlerENIHealthCheck::GeneveHandlerENIHealthCheck(std::string eniStr, TunInterfaceHealthCheck tunnelIn, TunInterfaceHealthCheck tunnelOut, FlowCacheHealthCheck v4FlowCache, FlowCacheHealthCheck v6FlowCache) : 324 | eniStr(eniStr), tunnelIn(std::move(tunnelIn)), tunnelOut(std::move(tunnelOut)), v4FlowCache(std::move(v4FlowCache)), v6FlowCache(std::move(v6FlowCache)) 325 | { 326 | } 327 | #else 328 | GeneveHandlerENIHealthCheck::GeneveHandlerENIHealthCheck(std::string eniStr, TunInterfaceHealthCheck tunnelIn) : 329 | eniStr(eniStr), tunnelIn(std::move(tunnelIn)) 330 | { 331 | } 332 | #endif 333 | 334 | std::string GeneveHandlerENIHealthCheck::output_str() 335 | { 336 | std::stringstream ret; 337 | 338 | ret << "Handler for ENI " << eniStr << std::endl; 339 | 340 | ret << tunnelIn.output_str(); 341 | #ifndef NO_RETURN_TRAFFIC 342 | ret << tunnelOut.output_str(); 343 | ret << v4FlowCache.output_str(); 344 | ret << v6FlowCache.output_str(); 345 | #endif 346 | 347 | return ret.str(); 348 | } 349 | 350 | json GeneveHandlerENIHealthCheck::output_json() 351 | { 352 | #ifndef NO_RETURN_TRAFFIC 353 | return {{"eniStr", eniStr}, {"tunnelIn", tunnelIn.output_json()}, {"tunnelOut", tunnelOut.output_json()}, {"v4FlowCache", v4FlowCache.output_json()}, {"v6FlowCache", v6FlowCache.output_json()}}; 354 | #else 355 | return {{"eniStr", eniStr}, {"tunnelIn", tunnelIn.output_json()}}; 356 | #endif 357 | // TODO. 358 | } 359 | 360 | GeneveHandlerENIHealthCheck GeneveHandlerENI::check() 361 | { 362 | #ifndef NO_RETURN_TRAFFIC 363 | return { eniStr, tunnelIn->status(), tunnelOut->status(), gwlbV4Cookies.check(), gwlbV6Cookies.check()}; 364 | #else 365 | return { eniStr, tunnelIn->status() }; 366 | #endif 367 | } 368 | 369 | /** 370 | * Check to see if we haven't seen traffic in timeout seconds. 371 | * 372 | * @return True if we haven't seen a packet in timeout seconds, false otherwise. 373 | */ 374 | bool GeneveHandlerENI::hasGoneIdle(int timeout) 375 | { 376 | std::chrono::steady_clock::time_point expireTime = std::chrono::steady_clock::now() - std::chrono::seconds(timeout); 377 | 378 | if(tunnelIn->lastPacketTime() > expireTime) return false; 379 | #ifndef NO_RETURN_TRAFFIC 380 | if(tunnelOut->lastPacketTime() > expireTime) return false; 381 | #endif 382 | return true; 383 | } 384 | 385 | /** 386 | * GeneveHandlerENI unique_ptr wrapper class 387 | */ 388 | GeneveHandlerENIPtr::GeneveHandlerENIPtr(eniid_t eni, int cacheTimeout, ThreadConfig &tunThreadConfig, ghCallback createCallback, ghCallback destroyCallback) 389 | { 390 | ptr = std::make_unique(eni, cacheTimeout, tunThreadConfig, createCallback, destroyCallback); 391 | } 392 | 393 | 394 | std::string devname_make(eniid_t eni, bool inbound) { 395 | if(inbound) 396 | return "gwi-"s + toBase60(eni); 397 | else 398 | return "gwo-"s + toBase60(eni); 399 | } 400 | 401 | 402 | -------------------------------------------------------------------------------- /GeneveHandler.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWLBTUN_GENEVEHANDLER_H 5 | #define GWLBTUN_GENEVEHANDLER_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include "UDPPacketReceiver.h" 13 | #include "TunInterface.h" 14 | #include "GenevePacket.h" 15 | #include "PacketHeaderV4.h" 16 | #include "PacketHeaderV6.h" 17 | #include "FlowCache.h" 18 | #include "utils.h" 19 | #include // Needed for IFNAMSIZ define 20 | #include 21 | #include "HealthCheck.h" 22 | 23 | typedef std::function ghCallback; 24 | 25 | // Data we need to send with the packet back to GWLB, including the Geneve header and outer UDP header information. 26 | class GwlbData { 27 | public: 28 | GwlbData(); 29 | GwlbData(GenevePacket &gp, struct in_addr *srcAddr, uint16_t srcPort, struct in_addr *dstAddr, uint16_t dstPort); 30 | 31 | // Elements are arranged so that when doing sorting/searching, we get entropy early. This gives a slight 32 | // improvement to the lookup time. 33 | GenevePacket gp; 34 | struct in_addr srcAddr; 35 | uint16_t srcPort; 36 | struct in_addr dstAddr; 37 | uint16_t dstPort; 38 | }; 39 | 40 | /** 41 | * For each ENI (GWLBe) that is detected, a copy of GeneveHandlerENI is created. 42 | */ 43 | 44 | class GeneveHandlerENIHealthCheck : public HealthCheck { 45 | public: 46 | #ifndef NO_RETURN_TRAFFIC 47 | GeneveHandlerENIHealthCheck(std::string, TunInterfaceHealthCheck, TunInterfaceHealthCheck, FlowCacheHealthCheck, FlowCacheHealthCheck); 48 | #else 49 | GeneveHandlerENIHealthCheck(std::string, TunInterfaceHealthCheck); 50 | #endif 51 | std::string output_str() ; 52 | json output_json(); 53 | 54 | private: 55 | std::string eniStr; 56 | TunInterfaceHealthCheck tunnelIn; 57 | #ifndef NO_RETURN_TRAFFIC 58 | TunInterfaceHealthCheck tunnelOut; 59 | FlowCacheHealthCheck v4FlowCache; 60 | FlowCacheHealthCheck v6FlowCache; 61 | #endif 62 | }; 63 | 64 | class GeneveHandlerENI { 65 | public: 66 | GeneveHandlerENI(eniid_t eni, int cacheTimeout, ThreadConfig& tunThreadConfig, ghCallback createCallback, ghCallback destroyCallback); 67 | ~GeneveHandlerENI(); 68 | void udpReceiverCallback(const GwlbData &gd, unsigned char *pkt, ssize_t pktlen); 69 | void tunReceiverCallback(unsigned char *pktbuf, ssize_t pktlen); 70 | GeneveHandlerENIHealthCheck check(); 71 | bool hasGoneIdle(int timeout); 72 | 73 | private: 74 | const eniid_t eni; 75 | const std::string eniStr; 76 | int cacheTimeout; 77 | 78 | const std::string devInName; 79 | const std::string devOutName; 80 | 81 | std::unique_ptr tunnelIn; 82 | #ifndef NO_RETURN_TRAFFIC 83 | std::unique_ptr tunnelOut; 84 | 85 | FlowCache gwlbV4Cookies; 86 | FlowCache gwlbV6Cookies; 87 | #endif 88 | // Socket used by all threads for sending 89 | int sendingSock; 90 | const ghCallback createCallback; 91 | const ghCallback destroyCallback; 92 | }; 93 | 94 | /** 95 | * Simple class wrapper for GeneveHandlerENI that leverages unique_ptr to keep things intact. Class is needed 96 | * to prevent unnecessary early construction/destruction in the concurrent_flat_map try_emplace calls. 97 | */ 98 | class GeneveHandlerENIPtr { 99 | public: 100 | GeneveHandlerENIPtr(eniid_t eni, int idleTimeout, ThreadConfig& tunThreadConfig, ghCallback createCallback, ghCallback destroyCallback); 101 | std::unique_ptr ptr; 102 | }; 103 | 104 | class GeneveHandlerHealthCheck : public HealthCheck { 105 | public: 106 | GeneveHandlerHealthCheck(bool, UDPPacketReceiverHealthCheck, std::list); 107 | std::string output_str() ; 108 | json output_json(); 109 | 110 | private: 111 | bool healthy; 112 | UDPPacketReceiverHealthCheck udp; 113 | std::list enis; 114 | }; 115 | 116 | class GeneveHandler { 117 | public: 118 | GeneveHandler(ghCallback createCallback, ghCallback destroyCallback, int destroyTimeout, int cacheTimeout, ThreadConfig udpThreads, ThreadConfig tunThreads); 119 | void udpReceiverCallback(unsigned char *pkt, ssize_t pktlen, struct in_addr *srcAddr, uint16_t srcPort, struct in_addr *dstAddr, uint16_t dstPort); 120 | GeneveHandlerHealthCheck check(); 121 | bool healthy; // Updated by check() 122 | 123 | private: 124 | boost::concurrent_flat_map eniHandlers; 125 | ghCallback createCallback; 126 | ghCallback destroyCallback; 127 | int eniDestroyTimeout; 128 | int cacheTimeout; 129 | ThreadConfig tunThreadConfig; 130 | UDPPacketReceiver udpRcvr; 131 | 132 | }; 133 | 134 | 135 | 136 | 137 | std::string devname_make(eniid_t eni, bool inbound); 138 | 139 | #endif //GWLBTUN_GENEVEHANDLER_H 140 | -------------------------------------------------------------------------------- /GenevePacket.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * GenevePacket class takes in a raw packet buffer and attempts to interpret it as a Geneve-encapsulated packet. 6 | * The status member variable will be GP_STATUS_OK if this is successful and everything passes sanity checks, 7 | * otherwise will be one of the other codes defined in GeneveHeader.h. 8 | * 9 | * This code is based on the version of Geneve as defined in RFC 8926. 10 | */ 11 | 12 | #include "GenevePacket.h" 13 | #include 14 | #include "utils.h" 15 | 16 | /** 17 | * Empty initializer 18 | */ 19 | GenevePacket::GenevePacket() : status(GP_STATUS_EMPTY) { } 20 | 21 | /** 22 | * Interpret the packet as a GENEVE packet. Does sanity checks to ensure the packet is correct. Callers should 23 | * verify the class's status member is GP_STATUS_OK before using - otherwise handle the error code provided in 24 | * that variable. 25 | * 26 | * @param pktBuf Pointer to the UDP packet payload received. 27 | * @param pktLen Length of pktBuf 28 | */ 29 | GenevePacket::GenevePacket(unsigned char *pktBuf, ssize_t pktLen) 30 | : status(GP_STATUS_EMPTY) 31 | { 32 | // 1) Process the Geneve Header in the passed in raw packet buffer. Since we're receiving via a UDP socket, 33 | // the outer IPv4 header has been removed for us by the OS, so our first byte is the outer Geneve Reader. 34 | // Geneve Header: 35 | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 36 | // |Ver| Opt Len |O|C| Rsvd. | Protocol Type | 37 | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 38 | // | Virtual Network Identifier (VNI) | Reserved | 39 | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 40 | // | | 41 | // ~ Variable-Length Options ~ 42 | // | | 43 | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 44 | unsigned char workBuf[8]; 45 | unsigned char *pktPtr; // Pointer to where the packet is being processed. 46 | 47 | if(pktLen < 8) 48 | { 49 | status = GP_STATUS_TOO_SHORT; 50 | return; 51 | } 52 | // Version check. This must be 0, by the current RFC. 53 | if( (pktBuf[0] & 0xc0) != 0) 54 | { 55 | status = GP_STATUS_BAD_VER; 56 | return; 57 | } 58 | 59 | // This application has no use for the O or C bits. Reserved must be ignored. 60 | 61 | // GWLB uses L3 IPv4 encapsulation - ethertype 0x0800, or L3 IPv6 encapsulation - 0x86dd. Make sure that's what is here. 62 | if( (be16toh(*(uint16_t *)&pktBuf[2]) != ETH_P_IP) && (be16toh(*(uint16_t *)&pktBuf[2]) != ETH_P_IPV6)) 63 | { 64 | status = GP_STATUS_BAD_ETHERTYPE; 65 | return; 66 | } 67 | bzero(workBuf, 8); 68 | // This next copy is deliberately dest off by 1 to allow using be32toh which expects 32 bits. The off-by-one 69 | // and the bzero just before ensure the first 8 bits are 0. 70 | memcpy(&workBuf[1], &pktBuf[4], 3); 71 | geneveVni = be32toh(*(uint32_t *)&workBuf); 72 | 73 | // Geneve options processing. The options length is expressed in 4-byte multiples (RFC 8926 section 3.4). 74 | int optLen = (pktBuf[0] & 0x3f) * 4; 75 | if( optLen > pktLen - 8) 76 | { 77 | status = GP_STATUS_BAD_OPTLEN; 78 | return; 79 | } 80 | 81 | // Work through the packet buffer, option-by-option, moving pktPtr as needed. We're expecting 3, so reserve that. 82 | geneveOptions.reserve(3); 83 | pktPtr = &pktBuf[8]; 84 | struct GeneveOption go; 85 | while(pktPtr < &pktBuf[8 + optLen]) 86 | { 87 | // Convert option to host format and in the struct. See RFC 8926 section 3.5. 88 | bzero(&go, sizeof(go)); 89 | go.optClass = be16toh(*(uint16_t *)&pktPtr[0]); 90 | go.optType = (uint8_t)pktPtr[2]; 91 | go.r = (unsigned char)(pktPtr[3] & 0xe0 >> 5); 92 | go.optLen = (pktPtr[3] & 0x1f) * 4; 93 | if(go.optLen > 0) go.optData = &pktPtr[4]; 94 | geneveOptions.push_back(go); 95 | 96 | // check for AWS specific options for GWLB. 97 | if(go.optClass == 0x108 && go.optType == 1 && go.optLen == 8) 98 | { 99 | gwlbeEniIdValid = true; 100 | gwlbeEniId = be64toh(*(uint64_t *)go.optData); 101 | } 102 | else if(go.optClass == 0x108 && go.optType == 2 && go.optLen == 8) 103 | { 104 | attachmentIdValid = true; 105 | attachmentId = be64toh(*(uint64_t *)go.optData); 106 | } 107 | else if(go.optClass == 0x108 && go.optType == 3 && go.optLen == 4) 108 | { 109 | flowCookieValid = true; 110 | flowCookie = be32toh(*(uint32_t *)go.optData); 111 | } 112 | pktPtr += 4 + go.optLen; 113 | } 114 | 115 | header = std::vector(pktBuf, pktBuf + 8 + optLen); 116 | headerLen = 8 + optLen; 117 | 118 | // If the three mandatory options for GWLB weren't seen, this can't be a valid packet from it. 119 | if(!gwlbeEniIdValid || !attachmentIdValid || !flowCookieValid) 120 | status = GP_STATUS_MISSING_GWLB_OPTIONS; 121 | else 122 | status = GP_STATUS_OK; 123 | } 124 | 125 | std::string GenevePacket::text() 126 | { 127 | std::ostringstream ss; 128 | ss << " Status: " << (status) << std::hex << " GWLBe ENI ID: " << (MakeENIStr(gwlbeEniIdValid?gwlbeEniId:0)) << " Attachment ID: " << (attachmentIdValid?attachmentId:0) << " Flow Cookie: " << (flowCookieValid?flowCookie:0) << std::dec; 129 | return ss.str(); 130 | } 131 | 132 | /** 133 | * Output a human-readable string of the contents of this GenevePacket. 134 | * 135 | * @param os Output stream to write to. 136 | * @param m The GenevePacket to write. 137 | * @return The same output stream, with the text added. 138 | */ 139 | auto operator<<(std::ostream& os, GenevePacket const& m) -> std::ostream& 140 | { 141 | return os << " Status: " << (m.status) << std::hex << " GWLBe ENI ID: " << (MakeENIStr(m.gwlbeEniIdValid?m.gwlbeEniId:0)) << " Attachment ID: " << (m.attachmentIdValid?m.attachmentId:0) << " Flow Cookie: " << (m.flowCookieValid?m.flowCookie:0) << std::dec; 142 | } 143 | 144 | -------------------------------------------------------------------------------- /GenevePacket.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWAPPLIANCE_GENEVEPACKET_H 5 | #define GWAPPLIANCE_GENEVEPACKET_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | typedef uint64_t eniid_t; 14 | 15 | // Storage for options seen in a Geneve header. 16 | struct GeneveOption { 17 | uint16_t optClass; // Option class, converted to host format 18 | uint8_t optType; // Type, converted to host format 19 | unsigned char r : 3; // R bits 20 | uint8_t optLen; // Option length, in bytes (converted from on-wire 4-byte multiples value) 21 | unsigned char *optData; // Pointer to the option data. May be nullptr, if optLen is 0. 22 | }; 23 | 24 | class GenevePacket { 25 | public: 26 | GenevePacket(); 27 | GenevePacket(unsigned char *pktBuf, ssize_t pktLen); // pktBuf points to the start of the Geneve header (i.e. after the outer UDP header) 28 | 29 | int status; 30 | int headerLen; // Total length of the Geneve header parsed. 31 | std::vector geneveOptions; // Parsed options from the Geneve header. 32 | uint32_t geneveVni; // The outer VNI identifier from the Geneve header. 33 | bool gwlbeEniIdValid, attachmentIdValid, flowCookieValid; // False if the options weren't found (and the below values MUST NOT be used), or true if they were. 34 | eniid_t gwlbeEniId; // The GWLBE ENI ID option, if it was found (check via the valid boolean) 35 | uint64_t attachmentId; // The attachment ID option, if it was found 36 | uint32_t flowCookie; // The flow cookie, if it was found 37 | std::vectorheader; // Copy of the Geneve header to put back on packets 38 | 39 | std::string text(); 40 | friend auto operator<<(std::ostream& os, GenevePacket const& m) -> std::ostream&; 41 | }; 42 | 43 | // GenevePacket status codes 44 | enum { 45 | GP_STATUS_EMPTY = 0, // Nothing done yet 46 | GP_STATUS_OK, // Everything looks good, packet is valid. 47 | GP_STATUS_TOO_SHORT, // Packet is too short to carry a Geneve header 48 | GP_STATUS_BAD_VER, // Version wasn't 0 49 | GP_STATUS_BAD_OPTLEN, // Options length exceeded packet size 50 | GP_STATUS_BAD_ETHERTYPE, // Geneve Ethertype wasn't 0x8000 (L3 IPv4) 51 | GP_STATUS_MISSING_GWLB_OPTIONS, // Required options for GWLB are missing. 52 | }; 53 | 54 | #endif //GWAPPLIANCE_GENEVEPACKET_H 55 | -------------------------------------------------------------------------------- /HISTORY.md: -------------------------------------------------------------------------------- 1 | ## 2024.09.01 Release (v3.0): 2 | - Support GWLB variable timeouts for the flow cache. 3 | 4 | ## 2024.08.07 Release (v2.5): 5 | - Add in JSON health check output support. 6 | - Fix a bug that occasionally caused a crash on startup if the logger thread didn't quite initialize quick enough 7 | - Fix a second related bug that occasionally caused a crash on shutdown if the logger thread didn't terminate in the right order. 8 | 9 | ## 2023.10.03 Release (v2.4): 10 | - Replace flow caches with version based on Boost's concurrent_flat_map, improving performance and reducing CPU usage due to time spent waiting for locks. 11 | - Standardized logging into its own class and thread, with improved configuration options 12 | - General code cleanup - broke out per ENI handling to its own class (GeneveHandlerENI) which simplified code a fair bit. 13 | 14 | ## 2023.06.07 Release: 15 | - Add NO_RETURN_TRAFFIC define by request - this strips out some of the internal tracking and removes the ability to send packets back to GWLB, but increases performance on incoming packet handling. 16 | 17 | ## 2022.11.17 Release: 18 | - **Update to support IPv6 payloads from GWLB** 19 | - Updated CMakeLists file to cleanly separate Debug and Release build options 20 | - Rearrange initializers to cleanup some harmless ```-Wreorder``` warnings, and reorder a couple fields to optimize memory access 21 | - Update packet hashing algorithm to be in one place (utils.h, defined inline), along with adding stats as to how the hash algorithm is performing to the status webpage when the debug flag is on. Thusfar in testing, the simple add-all-fields algorithm does well with avoiding collisions and is fast. 22 | - Updated help on script parameters, as suggested by liujunhui74 23 | - Added recognizing the flow in both directions when seen the first time, as suggested by liujunhui74 24 | - Cleaned up debugging output to be consistent, and added milliseconds to the timestamp 25 | - Update shutdown process to have all threads stop processing, then shutdown. In high PPS testing, there would occasionally be a race condition in shutting down that would result in a use-after-free error which this resolves. 26 | - Updated the hashing for PacketHeader classes to extend std::hash instead of providing their own additional classes to std::unordered_map for cleaner operation. 27 | - Reserve 3 GENEVE header options by default when processing a GENEVE packet, avoiding an unnecessary realloc due to dynamic vector expansion. 28 | - Fixed an issue where in high PPS testing, the UDP thread would take a long time to shutdown (due to not checking for the shutdown requested flag in the inner packet processing loop) 29 | - Replaced some ```sprintf```s with ```snprintf```s in ```hexDump()``` for safety 30 | 31 | ## 2022.05.13 Release: 32 | - Initial version 33 | -------------------------------------------------------------------------------- /HealthCheck.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | // Base class for health check objects 5 | #ifndef GWLBTUN_HEALTHCHECK_H 6 | #define GWLBTUN_HEALTHCHECK_H 7 | 8 | #include "json.hpp" 9 | using json = nlohmann::json; 10 | 11 | class HealthCheck { 12 | public: 13 | virtual std::string output_str() = 0; 14 | virtual json output_json() = 0; 15 | }; 16 | 17 | #endif //GWLBTUN_HEALTHCHECK_H 18 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /Logger.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #include "Logger.h" 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | using namespace std::string_literals; 12 | 13 | static const std::array loggingSections = { 14 | "core"s, "udp"s, "geneve"s, "tunnel"s, "healthcheck"s, "all"s 15 | }; 16 | 17 | static const std::array loggingLevels = { 18 | "critical"s, "important"s, "info"s, "debug"s, "debugdetail"s 19 | }; 20 | 21 | Logger::Logger(std::string loggingOptions) : 22 | cfg(optionsParse(loggingOptions)), shouldTerminate(false), thread(&Logger::threadFunc, this) 23 | { 24 | 25 | } 26 | 27 | Logger::~Logger() 28 | { 29 | shouldTerminate = true; 30 | queue_condvar.notify_one(); 31 | } 32 | 33 | std::string Logger::help() 34 | { 35 | std::stringstream ss; 36 | 37 | ss << "The logging configuration can be set by passing a string to the --logging option. That string is a series of
=, comma separated and case insensitive." << std::endl << "The available sections are: "; 38 | for(auto& it : loggingSections) ss << it << " "; 39 | ss << std::endl; 40 | ss << "The logging levels available for each are: "; 41 | for(auto& it : loggingLevels) ss << it << " "; 42 | ss << std::endl; 43 | ss << "The default level for all sections is 'important'."; 44 | ss << std::endl; 45 | 46 | return ss.str(); 47 | } 48 | 49 | LoggingConfiguration Logger::optionsParse(std::string loggingOptions) 50 | { 51 | LoggingConfiguration lc; 52 | LogSection i; 53 | 54 | // Set default levels 55 | for(i = LS_CORE; i < LS_COUNT; i = LogSection(i+1)) 56 | lc.ll[i] = LL_INFO; 57 | 58 | // Parse string 59 | std::istringstream ss(loggingOptions); 60 | std::string token; 61 | while(std::getline(ss, token, ',')) 62 | { 63 | size_t pos = token.find('='); 64 | if(pos != std::string::npos) 65 | { 66 | std::string key = token.substr(0, pos); 67 | std::string value = token.substr(pos + 1); 68 | 69 | int valN = FindIndexOf(loggingLevels, value); 70 | if(valN == -1) 71 | { 72 | fprintf(stderr, "Unrecognized logging level '%s' caused %s to be ignored.", value.c_str(), token.c_str()); 73 | } else { 74 | int keyN = FindIndexOf(loggingSections, key); 75 | if(keyN == -1) 76 | { 77 | fprintf(stderr, "Unrecognized logging section '%s' caused %s to be ignored.", key.c_str(), token.c_str()); 78 | } else if(keyN == LS_COUNT) { // aka 'all' 79 | for(i = LS_CORE; i < LS_COUNT; i = LogSection(i+1)) 80 | lc.ll[i] = (LogLevel)valN; 81 | } else { 82 | lc.ll[(LogSection)keyN] = (LogLevel)valN; 83 | } 84 | } 85 | } 86 | } 87 | return lc; 88 | } 89 | 90 | /** 91 | * Logging thread. Receives messages from the Log function, and prints them out, splitting lines on newlines 92 | * as needed. 93 | */ 94 | void Logger::threadFunc() 95 | { 96 | struct tm localtm; 97 | struct LoggingMessage lm; 98 | 99 | pthread_setname_np(pthread_self(), "gwlbtun Logger"); 100 | std::stringstream ss; 101 | ss << "Logging thread started. Configuration:"; 102 | for(auto i = LS_CORE; i < LS_COUNT; i = LogSection(i+1)) 103 | ss << loggingSections[i] << "=" << loggingLevels[cfg.ll[i]].c_str() << " "; 104 | 105 | LOG(LS_CORE, LL_IMPORTANT, ss.str()); 106 | 107 | while(!shouldTerminate) 108 | { 109 | std::unique_lock lk(queue_mutex); 110 | 111 | while(!queue.empty()) 112 | { 113 | std::stringstream header; 114 | lm = queue.front(); 115 | queue.pop(); 116 | auto millis = std::chrono::duration_cast(lm.ts.time_since_epoch()).count() % 1000; 117 | std::time_t cur_time = std::chrono::system_clock::to_time_t(lm.ts); 118 | localtime_r(&cur_time, &localtm); 119 | header << std::put_time(&localtm, "%F %T") << "."s << std::setw(3) << std::setfill('0') << millis << ": " << lm.thread << "(" << loggingSections[lm.ls] << "," << loggingLevels[lm.ll] << ") : "; 120 | std::string line; 121 | std::stringstream is(lm.msg); 122 | while(std::getline(is, line)) 123 | { 124 | std::cout << header.str() << ": " << line << std::endl; 125 | } 126 | } 127 | 128 | queue_condvar.wait(lk); 129 | } 130 | } 131 | 132 | /** 133 | * Log a message. Sends the message to the logging thread, which will split lines if needed. 134 | * 135 | * @param ls Logging section (LS_ defines) 136 | * @param ll Logging level (LL_ defines) 137 | * @param fmt_str String, possibly including printf-type formatting arguments 138 | * @param ... Arguments for the formatting in fmt_str 139 | */ 140 | void Logger::Log(LogSection ls, LogLevel ll, const std::string& fmt_str, ...) 141 | { 142 | va_list ap; 143 | 144 | // This check also occurs the LOG() macro, but let's verify 145 | if(cfg.ll[ls] >= ll) 146 | { 147 | struct LoggingMessage lm; 148 | va_start(ap, &fmt_str); 149 | lm = (struct LoggingMessage){.ls = ls, .ll = ll, .msg = stringFormat(fmt_str, ap), .ts = std::chrono::system_clock::now(), .thread = "" }; 150 | va_end(ap); 151 | pthread_getname_np(pthread_self(), lm.thread, 16); 152 | { 153 | std::lock_guard lock(queue_mutex); 154 | queue.emplace(std::move(lm)); 155 | } 156 | queue_condvar.notify_one(); 157 | } 158 | } 159 | 160 | /** 161 | * Log a hex dump. Formats the incoming buffer into multiple lines, then sends to the logging thread. 162 | * The thread will split the lines for us, ensuring they all output at the same time. 163 | * 164 | * @param ls Logging section (LS_ defines) 165 | * @param ll Logging level (LL_ defines) 166 | * @param buffer Buffer the hex dump 167 | * @param bufsize Length of buffer 168 | */ 169 | void Logger::LogHexDump(LogSection ls, LogLevel ll, const void *buffer, std::size_t bufsize) 170 | { 171 | if(cfg.ll[ls] >= ll) 172 | { 173 | std::stringstream ss; 174 | char hexBuf[140]; 175 | char printBuf[36]; 176 | unsigned char *cBuffer = (unsigned char *)buffer; 177 | std::size_t offset; 178 | const bool showPrintableChars = true; 179 | 180 | for(offset = 0; offset < bufsize; offset ++) 181 | { 182 | if(offset % 32 == 0) 183 | { 184 | if(offset > 0) ss << hexBuf << printBuf << std::endl; 185 | bzero(hexBuf, 140); 186 | bzero(printBuf, 36); 187 | if(showPrintableChars) strcpy(printBuf, " | "); 188 | snprintf(hexBuf, 7, "%04zx: ", offset); 189 | } 190 | snprintf(&hexBuf[5+((offset % 32)*3)], 4, " %02x", cBuffer[offset]); 191 | if(showPrintableChars) 192 | printBuf[3 + (offset % 32)] = (isprint(cBuffer[offset]))?cBuffer[offset]:'.'; 193 | } 194 | // Add in enough padding to make sure our lines line up. 195 | for(offset = 5+((offset % 32) * 3); offset < 101; offset ++) hexBuf[offset] = ' '; 196 | if(offset > 0) ss << hexBuf << printBuf << std::endl; 197 | 198 | this->Log(ls, ll, ss.str()); 199 | } 200 | } 201 | -------------------------------------------------------------------------------- /Logger.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWLBTUN_LOGGER_H 5 | #define GWLBTUN_LOGGER_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include "utils.h" 16 | 17 | // Log Section (LS_) defines 18 | typedef enum { 19 | LS_CORE, // Core messaging, startup, etc. 20 | LS_UDP, // UDP receiver 21 | LS_GENEVE, // GENEVE packet details 22 | LS_TUNNEL, // Tunnel processing 23 | LS_HEALTHCHECK, // Health check reporting 24 | LS_COUNT 25 | } LogSection; 26 | 27 | // Log level (LL_) defines 28 | typedef enum { 29 | LL_CRITICAL, // Critical alerts 30 | LL_IMPORTANT, // Important messages 31 | LL_INFO, // Informational messages 32 | LL_DEBUG, // Debug messages 33 | LL_DEBUGDETAIL, // Debug detail 34 | LL_COUNT 35 | } LogLevel; 36 | 37 | struct LoggingConfiguration { 38 | LogLevel ll[LS_COUNT]; 39 | }; 40 | 41 | struct LoggingMessage { 42 | LogSection ls; 43 | LogLevel ll; 44 | std::string msg; 45 | std::chrono::system_clock::time_point ts; 46 | char thread[16]; 47 | }; 48 | 49 | extern class Logger *logger; 50 | 51 | #define LOG(s,l,msg,...) { if(logger->cfg.ll[s] >= l) logger->Log(s, l, msg, ##__VA_ARGS__); }; 52 | #define LOGHEXDUMP(s,l,hdr,buf,buflen) { if(logger->cfg.ll[s] >= l) logger->LogHexDump(s, l, buf, buflen); }; 53 | #define IS_LOGGING(s,l) (logger->cfg.ll[s] <= l) 54 | #define ts(s) std::to_string(s) 55 | 56 | class Logger { 57 | public: 58 | Logger(std::string loggingOptions); 59 | ~Logger(); 60 | std::string help(); // Return help text for configuring the logger. 61 | const LoggingConfiguration cfg; 62 | void Log(LogSection ls, LogLevel ll, const std::string& fmt_str, ...); 63 | void LogHexDump(LogSection ls, LogLevel ll, const void *buf, std::size_t bufLen); 64 | 65 | private: 66 | bool shouldTerminate; 67 | std::thread thread; 68 | 69 | LoggingConfiguration optionsParse(std::string loggingOptions); 70 | void threadFunc(); 71 | 72 | // Logging queue 73 | std::mutex queue_mutex; 74 | std::condition_variable queue_condvar; 75 | std::queue queue; 76 | }; 77 | 78 | #endif //GWLBTUN_LOGGER_H 79 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | aws-gateway-load-balancer-tunnel-handler 2 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /PacketHeaderV4.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * PacketHeaderV4 class serves to interpret and provide a hashing function for an IPv4 header, looking at similiar fields 6 | * to what GWLB does when producing a flow cookie. 7 | */ 8 | 9 | #include "PacketHeaderV4.h" 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | #include "utils.h" 19 | 20 | /** 21 | * Parse an IP packet header. 22 | * 23 | * @param pktbuf IP packet header to parse 24 | * @param pktlen Length of the packet. 25 | */ 26 | PacketHeaderV4::PacketHeaderV4(unsigned char *pktbuf, ssize_t pktlen) 27 | { 28 | struct ip *iph = (struct ip *)pktbuf; 29 | 30 | if(pktlen < (ssize_t)sizeof(struct ip)) 31 | throw std::invalid_argument("PacketHeaderV4 provided a packet too small to be an IPv4 packet."); 32 | 33 | if(iph->ip_v != 4) 34 | throw std::invalid_argument("PacketHeaderV4 provided a packet that isn't IPv4."); 35 | 36 | prot = iph->ip_p; 37 | src = be32toh(*(uint32_t *)&iph->ip_src); 38 | dst = be32toh(*(uint32_t *)&iph->ip_dst); 39 | switch(prot) 40 | { 41 | case IPPROTO_UDP: 42 | { 43 | if(pktlen < (ssize_t)(sizeof(struct ip) + sizeof(struct udphdr))) 44 | throw std::invalid_argument("PacketHeaderV4 provided a packet with protocol=UDP, but too small to carry UDP information."); 45 | struct udphdr *udp = (struct udphdr *)(pktbuf + sizeof(struct ip)); 46 | srcpt = be16toh(udp->uh_sport); 47 | dstpt = be16toh(udp->uh_dport); 48 | break; 49 | } 50 | case IPPROTO_TCP: 51 | { 52 | if(pktlen < (ssize_t)(sizeof(struct ip) + sizeof(struct tcphdr))) 53 | throw std::invalid_argument("PacketHeaderV4 provided a packet with protocol=TCP, but too small to carry UDP information."); 54 | struct tcphdr *tcp = (struct tcphdr *)(pktbuf + sizeof(struct ip)); 55 | srcpt = be16toh(tcp->th_sport); 56 | dstpt = be16toh(tcp->th_dport); 57 | break; 58 | } 59 | default: 60 | { 61 | srcpt = 0; 62 | dstpt = 0; 63 | break; 64 | } 65 | } 66 | } 67 | 68 | /** 69 | * Compare if the headers in one packet match another. Meant to be used for the various sort and search functions. 70 | * 71 | * @param ph PacketHeaderV4 to compare againgst. 72 | * @return true if PacketHeaders are the same, false otherwise. 73 | */ 74 | bool PacketHeaderV4::operator==(const PacketHeaderV4 &ph) const 75 | { 76 | #ifdef HASH_IS_SYMMETRICAL 77 | return prot == ph.prot && ((src == ph.src && dst == ph.dst && srcpt == ph.srcpt && dstpt == ph.dstpt) || 78 | (src == ph.dst && dst == ph.src && srcpt == ph.dstpt && dstpt == ph.srcpt)); 79 | #else 80 | return prot == ph.prot && src == ph.src && dst == ph.dst && srcpt == ph.srcpt && dstpt == ph.dstpt; 81 | #endif 82 | } 83 | 84 | /** 85 | * Returns a hash value based on the protocol data. 86 | * 87 | * @return Hash value. 88 | */ 89 | std::size_t PacketHeaderV4::hash() const 90 | { 91 | return hashFunc(prot, (void *)&src, (void *)&dst, 4, srcpt, dstpt); 92 | } 93 | 94 | /** 95 | * Returns a string with the contents of the PacketHeaderV4 for human consumption. 96 | * 97 | * @return The string. 98 | */ 99 | std::string PacketHeaderV4::text() const 100 | { 101 | std::string s = "PacketHeaderV4: "; 102 | 103 | s = s + stringFormat("SrcIP: %d.%d.%d.%d, DstIP: %d.%d.%d.%d", 104 | (src & 0xFF000000) >> 24, (src & 0xFF0000) >> 16, (src & 0xFF00) >> 8, src & 0xFF, 105 | (dst & 0xFF000000) >> 24, (dst & 0xFF0000) >> 16, (dst & 0xFF00) >> 8, dst & 0xFF); 106 | 107 | if(prot == IPPROTO_UDP) 108 | s = s + stringFormat(" Prot UDP, src pt %d, dst pt %d", srcpt, dstpt); 109 | else if(prot == IPPROTO_TCP) 110 | s = s + stringFormat(" Prot TCP, src pt %d, dst pt %d", srcpt, dstpt); 111 | else 112 | s = s + " Prot #" + std::to_string(prot); 113 | 114 | //s = s + stringFormat(", hash %x", hash()); 115 | 116 | return s; 117 | } 118 | 119 | /** 120 | * Output the human-readable content to an output stream. 121 | * @param os Output stream to write to. 122 | * @param m The PacketHeaderV4 to output 123 | * @return The same output stream. 124 | */ 125 | std::ostream &operator<<(std::ostream &os, PacketHeaderV4 const &m) 126 | { 127 | return os << m.text(); 128 | } 129 | 130 | /** 131 | * Extend std::hash for PacketHeaderV4 132 | */ 133 | std::size_t std::hash::operator()(const PacketHeaderV4& t) const 134 | { 135 | return t.hash(); 136 | }; 137 | 138 | std::size_t hash_value(const PacketHeaderV4& t) 139 | { 140 | return t.hash(); 141 | }; 142 | -------------------------------------------------------------------------------- /PacketHeaderV4.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | // 4 | // Quick class to generate hashes of IPv4 packets for use in std::unordered_map 5 | // 6 | 7 | #ifndef GWLBTUN_PACKETHEADERV4_H 8 | #define GWLBTUN_PACKETHEADERV4_H 9 | 10 | #include 11 | #include 12 | #include 13 | #include 14 | 15 | class PacketHeaderV4 { 16 | public: 17 | PacketHeaderV4(unsigned char *pktbuf, ssize_t pktlen); // pktbuf points to the start of the IP header. 18 | bool operator==(const PacketHeaderV4 &) const; 19 | std::string text() const; 20 | std::size_t hash() const; 21 | 22 | private: 23 | uint32_t src; 24 | uint32_t dst; 25 | uint16_t srcpt; 26 | uint16_t dstpt; 27 | uint8_t prot; 28 | }; 29 | 30 | std::ostream &operator<<(std::ostream &os, PacketHeaderV4 const &m); 31 | template<> struct std::hash:unary_function { std::size_t operator()(const PacketHeaderV4& t) const; }; 32 | std::size_t hash_value(const PacketHeaderV4& t); 33 | 34 | #endif //GWLBTUN_PACKETHEADERV4_H 35 | -------------------------------------------------------------------------------- /PacketHeaderV6.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * PacketHeaderV4 class serves to interpret and provide a hashing function for an IPv6 header, looking at similiar fields 6 | * to what GWLB does when producing a flow cookie. 7 | */ 8 | 9 | #include "PacketHeaderV6.h" 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | 19 | #include "utils.h" 20 | using namespace std::string_literals; 21 | 22 | /** 23 | * Parse an IP packet header. 24 | * 25 | * @param pktbuf IP packet header to parse 26 | * @param pktlen Length of the packet. 27 | */ 28 | PacketHeaderV6::PacketHeaderV6(unsigned char *pktbuf, ssize_t pktlen) 29 | { 30 | struct ip6_hdr *iph6 = (struct ip6_hdr *)pktbuf; 31 | 32 | if(pktlen < (ssize_t)sizeof(struct ip6_hdr)) 33 | throw std::invalid_argument("PacketHeaderV6 provided a packet too small to be an IPv6 packet."); 34 | 35 | if( ((iph6->ip6_ctlun.ip6_un2_vfc & 0xF0) >> 4) != 6) 36 | 37 | throw std::invalid_argument("PacketHeaderV6 provided a packet that isn't IPv6 : "s + std::to_string(iph6->ip6_ctlun.ip6_un2_vfc) + " -- "s + std::to_string(iph6->ip6_ctlun.ip6_un2_vfc >> 4) + " at offset "s + std::to_string( 38 | offsetof(struct ip6_hdr, ip6_ctlun.ip6_un2_vfc))); 39 | 40 | // The first 4 bits are version, next 8 are traffic class. The last 20 (0xFFFFF) are the flow label. 41 | flow = be32toh(iph6->ip6_ctlun.ip6_un1.ip6_un1_flow) & 0xFFFFF; 42 | memcpy(&src, &iph6->ip6_src, sizeof(struct in6_addr)); 43 | memcpy(&dst, &iph6->ip6_dst, sizeof(struct in6_addr)); 44 | prot = iph6->ip6_ctlun.ip6_un1.ip6_un1_nxt; 45 | switch(prot) 46 | { 47 | case IPPROTO_UDP: 48 | { 49 | if(pktlen < (ssize_t)(sizeof(struct ip6_hdr) + sizeof(struct udphdr))) 50 | throw std::invalid_argument("PacketHeaderV6 provided a packet with protocol=UDP, but too small to carry UDP information."); 51 | struct udphdr *udp = (struct udphdr *)(pktbuf + sizeof(struct ip6_hdr)); 52 | srcpt = be16toh(udp->uh_sport); 53 | dstpt = be16toh(udp->uh_dport); 54 | break; 55 | } 56 | case IPPROTO_TCP: 57 | { 58 | if(pktlen < (ssize_t)(sizeof(struct ip6_hdr) + sizeof(struct tcphdr))) 59 | throw std::invalid_argument("PacketHeaderV6 provided a packet with protocol=TCP, but too small to carry UDP information."); 60 | struct tcphdr *tcp = (struct tcphdr *)(pktbuf + sizeof(struct ip6_hdr)); 61 | srcpt = be16toh(tcp->th_sport); 62 | dstpt = be16toh(tcp->th_dport); 63 | break; 64 | } 65 | default: 66 | srcpt = 0; 67 | dstpt = 0; 68 | break; 69 | } 70 | } 71 | 72 | /** 73 | * Compare if the headers in one packet match another. Meant to be used for the various sort and search functions. 74 | * 75 | * @param ph PacketHeaderV4 to compare againgst. 76 | * @return true if PacketHeaders are the same, false otherwise. 77 | */ 78 | bool PacketHeaderV6::operator==(const PacketHeaderV6 &ph) const 79 | { 80 | #ifdef HASH_IS_SYMMETRICAL 81 | return prot == ph.prot && 82 | ((srcpt == ph.srcpt && dstpt == ph.dstpt && !memcmp(&src, &ph.src, sizeof(struct in6_addr)) && !memcmp(&dst, &ph.dst, sizeof(struct in6_addr))) || 83 | (srcpt == ph.dstpt && dstpt == ph.srcpt && !memcmp(&src, &ph.dst, sizeof(struct in6_addr)) && !memcmp(&dst, &ph.src, sizeof(struct in6_addr)))); 84 | #else 85 | return prot == ph.prot && srcpt == ph.srcpt && dstpt == ph.dstpt && 86 | !memcmp(&src, &ph.src, sizeof(struct in6_addr)) && !memcmp(&dst, &ph.dst, sizeof(struct in6_addr)); 87 | #endif 88 | } 89 | 90 | /** 91 | * Returns a hash value based on the protocol data. 92 | * 93 | * @return Hash value. 94 | */ 95 | std::size_t PacketHeaderV6::hash() const 96 | { 97 | return hashFunc(prot, (void *)&src, (void *)&dst, 16, srcpt, dstpt); 98 | } 99 | 100 | /** 101 | * Returns a string with the contents of the PacketHeaderV4 for human consumption. 102 | * 103 | * @return The string. 104 | */ 105 | std::string PacketHeaderV6::text() const 106 | { 107 | std::string s = "PacketHeaderV6: "; 108 | 109 | char srcip[INET6_ADDRSTRLEN], dstip[INET6_ADDRSTRLEN]; 110 | 111 | inet_ntop(AF_INET6, &src, srcip, INET6_ADDRSTRLEN); 112 | inet_ntop(AF_INET6, &dst, dstip, INET6_ADDRSTRLEN); 113 | 114 | s = s + stringFormat("SrcIP: %s, DstIP: %s, Flow: %x", srcip, dstip, flow); 115 | 116 | if(prot == IPPROTO_UDP) 117 | s = s + stringFormat(" Prot UDP, src pt %d, dst pt %d", srcpt, dstpt); 118 | else if(prot == IPPROTO_TCP) 119 | s = s + stringFormat(" Prot TCP, src pt %d, dst pt %d", srcpt, dstpt); 120 | else 121 | s = s + " Prot #" + std::to_string(prot); 122 | 123 | //s = s + stringFormat(", hash %x", hash()); 124 | 125 | return s; 126 | } 127 | 128 | /** 129 | * Output the human-readable content to an output stream. 130 | * @param os Output stream to write to. 131 | * @param m The PacketHeaderV4 to output 132 | * @return The same output stream. 133 | */ 134 | std::ostream &operator<<(std::ostream &os, PacketHeaderV6 const &m) 135 | { 136 | return os << m.text(); 137 | } 138 | 139 | /** 140 | * Extend std::hash for PacketHeaderV6 141 | */ 142 | std::size_t std::hash::operator()(const PacketHeaderV6& t) const 143 | { 144 | return t.hash(); 145 | }; 146 | 147 | std::size_t hash_value(const PacketHeaderV6& t) 148 | { 149 | return t.hash(); 150 | }; -------------------------------------------------------------------------------- /PacketHeaderV6.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | // 4 | // Quick class to generate hashes of IPv6 packets for use in std::unordered_map 5 | // 6 | 7 | #ifndef GWLBTUN_PACKETHEADERV6_H 8 | #define GWLBTUN_PACKETHEADERV6_H 9 | 10 | #include 11 | #include 12 | #include 13 | #include 14 | 15 | class PacketHeaderV6 { 16 | public: 17 | PacketHeaderV6(unsigned char *pktbuf, ssize_t pktlen); // pktbuf points to the start of the IP header. 18 | bool operator==(const PacketHeaderV6 &) const; 19 | std::string text() const; 20 | std::size_t hash() const; 21 | 22 | private: 23 | struct in6_addr src; 24 | struct in6_addr dst; 25 | uint32_t flow; 26 | uint16_t srcpt; 27 | uint16_t dstpt; 28 | uint8_t prot; 29 | }; 30 | 31 | std::ostream &operator<<(std::ostream &os, PacketHeaderV6 const &m); 32 | template<> struct std::hash:unary_function { std::size_t operator()(const PacketHeaderV6& t) const; }; 33 | std::size_t hash_value(const PacketHeaderV6& t); 34 | 35 | #endif //GWLBTUN_PACKETHEADERV6_H 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # aws-gateway-load-balancer-tunnel-handler 2 | This software supports using the Gateway Load Balancer AWS service. It is designed to be ran on a GWLB target, takes in the Geneve encapsulated data and creates Linux tun (layer 3) interfaces per endpoint. This allows standard Linux tools (iptables, etc.) to work with GWLB. 3 | 4 | See the 'example-scripts' folder for some of the options that can be used as create scripts for this software. 5 | 6 | ## To Compile 7 | On an Amazon Linux 2 or AL2023 host, copy this code down, and install dependencies: 8 | 9 | ``` 10 | sudo yum groupinstall "Development Tools" 11 | sudo yum install cmake3 12 | ``` 13 | 14 | In the directory with the source code, do ```cmake3 .; make``` to build. This code works with both Intel and Graviton-based architectures. 15 | 16 | **This version requires the Boost libraries, version 1.83.0 or greater.** This tends to be a newer version than available on distributions (for example, at time of writing, 1.75 is available in AL2023). You may need to go to https://www.boost.org/, download, and install a newer version than what's available in the repositories 17 | 18 | ## Usage 19 | For Linux, the application requires CAP_NET_ADMIN capability to create the tunnel interfaces along with the example helper scripts. 20 | ``` 21 | Tunnel Handler for AWS Gateway Load Balancer 22 | Usage: ./gwlbtun [options] 23 | Example: ./gwlbtun 24 | 25 | -h Print this help 26 | -c FILE Command to execute when a new tunnel has been built. See below for arguments passed. 27 | -r FILE Command to execute when a tunnel times out and is about to be destroyed. See below for arguments passed. 28 | -t TIME Minimum time in seconds between last packet seen and to consider the tunnel timed out. Set to 0 (the default) to never time out tunnels. 29 | Note the actual time between last packet and the destroy call may be longer than this time. 30 | -p PORT Listen to TCP port PORT and provide a health status report on it. 31 | -j For health check detailed statistics, output as JSON instead of text. 32 | -s Only return simple health check status (only the HTTP response code), instead of detailed statistics. 33 | -d Enable debugging output. Short version of --logging all=debug. 34 | 35 | Threading options: 36 | --udpthreads NUM Generate NUM threads for the UDP receiver. 37 | --udpaffinity AFFIN Generate threads for the UDP receiver, pinned to the cores listed. Takes precedence over udptreads. 38 | --tunthreads NUM Generate NUM threads for each tunnel processor. 39 | --tunaffinity AFFIN Generate threads for each tunnel processor, pinned to the cores listed. Takes precedence over tunthreads. 40 | 41 | AFFIN arguments take a comma separated list of cores or range of cores, e.g. 1-2,4,7-8. 42 | It is recommended to have the same number of UDP threads as tunnel processor threads, in one-arm operation. 43 | If unspecified, --udpthreads and --tunthreads will be assumed as a default, based on the number of cores present. 44 | 45 | Logging options: 46 | --logging CONFIG Set the logging configuration, as described below. 47 | --------------------------------------------------------------------------------------------------------- 48 | Hook scripts arguments: 49 | These arguments are provided when gwlbtun calls the hook scripts (the -c and/or -r command options). 50 | On gwlbtun startup, it will automatically create gwi- and gwo- interfaces upon seeing the first packet from a specific GWLBE, and the hook scripts are invoked when interfaces are created or destroyed. You should at least disable rpf_filter for the gwi- tunnel interface with the hook scripts. 51 | The hook scripts will be called with the following arguments: 52 | 1: The string 'CREATE' or 'DESTROY', depending on which operation is occurring. 53 | 2: The interface name of the ingress interface (gwi-). 54 | 3: The interface name of the egress interface (gwo-). Packets can be sent out via in the ingress 55 | as well, but having two different interfaces makes routing and iptables easier. 56 | 4: The GWLBE ENI ID in base 16 (e.g. '2b8ee1d4db0c51c4') associated with this tunnel. 57 | 58 | The in the interface name is replaced with the base 60 encoded ENI ID (to fit inside the 15 character 59 | device name limit). 60 | --------------------------------------------------------------------------------------------------------- 61 | The logging configuration can be set by passing a string to the --logging option. That string is a series of
=, comma separated and case insensitive. 62 | The available sections are: core udp geneve tunnel healthcheck all 63 | The logging levels available for each are: critical important info debug debugdetail 64 | The default level for all secions is 'important'. 65 | ``` 66 | 67 | ## Source code layout 68 | main.cpp contains the start of the code, but primarily interfaces with GeneveHandler, defined in GeneveHandler.cpp. 69 | That class launches the multithreaded UDP receiver, and then creates GeneveHandlerENI class instances per GWLB ENI detected. 70 | The GeneveHandlerENI class instantiates the TunInterfaces as needed, and generally manages the entire packet handling flow for that ENI. 71 | GenevePacket and PacketHeader handle parsing and validating GENEVE packets and IP packets respectively, and are called by GeneveHandler as needed. 72 | Logger handles processing logging messages from all threads, ensuring they get output correctly to terminal, and filtering against the logging configuration provided. 73 | 74 | ## Multithreading 75 | gwlbtun supports multithreading, and doing so is recommended on multicore systems. You can specify either the number or threads, or a specific affinity for CPU cores, for both the UDP receiver and the tunnel handler threads. You should test to see which set of options work best for your workload, especially if you have additional processes doing processing on the device. By default, gwlbtun will create one UDP receive thread and one tunnel processing thread per core. 76 | 77 | gwlbtun labels its threads with its name (gwlbtun), and either Uxxx for the UDP threads option which is simply an index, or UAxxx for the UDP affinity option, with the number being the core that thread is set for. The tunnel threads are labeled the same, except with a T instead of a U. 78 | 79 | ## Kernel sysctls 80 | Because most usages of gwlbtun have it sitting in the middle of a communications path (bump in the wire), none of the traffic is directly destined for it. Thus, in most cases, you should disable the reverse path filter (rp_filter) on associated GWI interfaces, in order for the kernel to allow the traffic through. The hook scripts are a good place to do this (the input interface is passed as $2) and the examples in example-scripts show different ways. One option: 81 | ``` 82 | sysctl net.ipv4.conf.$2.rp_filter=0 83 | ``` 84 | 85 | Additionally, if you're doing NAT or other forwarding operations, you need to enable IP forwarding for IPv4 and IPv6 as appropriate: 86 | ``` 87 | sysctl net.ipv4.ip_forward=1 88 | sysctl net.ipv6.conf.all.forwarding=1 89 | ``` 90 | 91 | ## Advanced usages 92 | 93 | ### No return mode 94 | If you are only interested in the ability to receive traffic to an L3 tunnel interface, and will never send traffic back to GWLB, you can #define NO_RETURN_TRAFFIC in utils.h. This removes the gwo interfaces and all cookie flow tracking, which saves on time used to synchronize that flow tracking table. Note that this puts your appliance in a two-arm mode with GWLB, and also may result in asymmetric traffic routing, which may have performance implications elsewhere. 95 | 96 | ### Handling overlapping CIDRs 97 | See the example-scripts/create-nat-overlapping.sh script for an example of handling overlapping CIDRs in different GWLB endpoints in two-arm mode. This script leverages conntrack and marking to accomplish this. 98 | 99 | ### Supporting very high packet rates 100 | If your deployment is supporting high packet rates (greater than 1M pps typically), you may need to tweak some kernel settings to handle microbursts in traffic well. In testing in extremely high PPS scenarios (a fleet of iperf-based senders, all going through one c6in.32xlarge instance), you may want to consider settings akin to this (if memory allows): 101 | ``` 102 | sysctl -w net.core.rmem_max=50000000 103 | sysctl -w net.core.rmem_default=50000000 104 | ``` 105 | 106 | You can see if this problem is occurring by monitoring for UDP receive buffer errors (RcvbufErrors) with commands similar to: 107 | ``` 108 | # cat /proc/net/snmp | grep Udp: 109 | Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti MemErrors 110 | Udp: 2985556669902 428 0 666162 0 0 0 0 0 111 | ``` 112 | 113 | If RcvbufErrors is incrementing steadily, you should increase the rmem values as described above. 114 | 115 | ### Health Check output 116 | If you do not provide the -s flag, the health check port produces human-readable statistics about the traffic gwlbtun is processing. If you add in the -j flag, this output is formatted as JSON for consumption by outside monitoring processes. 117 | 118 | ## Security 119 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 120 | 121 | ## License 122 | This tool is licensed under the MIT-0 License. See the LICENSE file. 123 | 124 | ### json.hpp 125 | The class is licensed under the MIT License: 126 | 127 | Copyright © 2013-2022 Niels Lohmann 128 | 129 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 130 | 131 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 132 | 133 | THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 134 | -------------------------------------------------------------------------------- /TunInterface.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * TunInterface handles creating and processing traffic received on TUN interfaces. This class: 6 | * - Creates the requested Tun interface 7 | * - Launches threads to service that interface 8 | * - Takes a callback function (recvDispatcher) which is called for each packet received by that interface. 9 | * - Provides a status() function that returns the packet counters and checks that the thread is still alive. 10 | */ 11 | 12 | #include "TunInterface.h" 13 | 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include "utils.h" 25 | #include "Logger.h" 26 | 27 | using namespace std::string_literals; 28 | 29 | /** 30 | * Constructor. Build a TUN interface and start listening for packets on it. 31 | * 32 | * @param devname The name of the TUN itnerface to build. 33 | * @param mtu MTU to set the interface to. 34 | * @param recvDispatcher Function the thread should callback to on packets received. 35 | */ 36 | TunInterface::TunInterface(std::string devname, int mtu, ThreadConfig threadConfig, tunCallback recvDispatcherParam) 37 | : lastPacket(std::chrono::steady_clock::now()),pktsOut(0),bytesOut(0) 38 | { 39 | LOG(LS_TUNNEL, LL_DEBUG, "TunInterface creating for "s + devname); 40 | this->devname = devname; 41 | 42 | // Set up our threads as per threadConfig 43 | int tIndex = 0; 44 | for(int core : threadConfig.cfg) 45 | { 46 | threads[tIndex].setup(tIndex, core, allocateHandle(), recvDispatcherParam); 47 | tIndex ++; 48 | } 49 | 50 | // Mark the tun device link up. We need a dummy socket to do this call. 51 | struct ifreq ifr; 52 | bzero(&ifr, sizeof(ifr)); 53 | strncpy(ifr.ifr_name, devname.c_str(), IFNAMSIZ); 54 | ifr.ifr_flags = IFF_TUN | IFF_NO_PI; 55 | 56 | int dummy = socket(PF_INET, SOCK_DGRAM, 0); 57 | if(ioctl(dummy, SIOCGIFFLAGS, (void *)&ifr) < 0) 58 | throw std::system_error(errno, std::generic_category(), "Unable to get device flags"); 59 | 60 | ifr.ifr_flags |= IFF_UP; // Set interface is up 61 | ifr.ifr_flags |= IFF_RUNNING; // Interface is running 62 | ifr.ifr_flags &= ~IFF_POINTOPOINT; 63 | if(ioctl(dummy, SIOCSIFFLAGS, (void *)&ifr) < 0) 64 | throw std::system_error(errno, std::generic_category(), "Unable to set device flags"); 65 | 66 | // Set the MTU to the GWLB standard (8500) 67 | ifr.ifr_mtu = mtu; 68 | if(ioctl(dummy, SIOCSIFMTU, (void *)&ifr) < 0) 69 | throw std::system_error(errno, std::generic_category(), "Unable to set MTU"); 70 | close(dummy); 71 | } 72 | 73 | /** 74 | * Destructor. Signals the thread to stop, waits for it to shut down, destroys the TUN interface, and returns. 75 | */ 76 | TunInterface::~TunInterface() 77 | { 78 | shutdown(); 79 | } 80 | 81 | void TunInterface::shutdown() 82 | { 83 | LOG(LS_TUNNEL, LL_DEBUG, "TunInterface destroying for "s + devname); 84 | 85 | // Signal all threads to shutdown down, then wait for all acks. 86 | for(auto &thread : threads) 87 | { 88 | thread.shutdown(); 89 | } 90 | 91 | bool allgood = false; 92 | while(!allgood) 93 | { 94 | allgood = true; 95 | for(auto &thread : threads) 96 | { 97 | if(thread.setupCalled) 98 | { 99 | if(thread.healthCheck()) 100 | allgood = false; 101 | } 102 | } 103 | if(!allgood) 104 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); 105 | } 106 | } 107 | 108 | /** 109 | * Send a packet out the TUN interface. Updates internal counters as well. 110 | * 111 | * @param pkt Buffer pointing to the packet to send 112 | * @param pktlen Packet length 113 | */ 114 | void TunInterface::writePacket(unsigned char *pkt, ssize_t pktlen) 115 | { 116 | if(!writerHandles.visit(pthread_self(), [&](auto& targetfd) { write(targetfd.second, (void *)pkt, pktlen); })) 117 | { 118 | // Key wasn't found - create and send. 119 | int targetfd = allocateHandle(); 120 | writerHandles.emplace(pthread_self(), targetfd); 121 | write(targetfd, (void *)pkt, pktlen); 122 | } 123 | lastPacket = std::chrono::steady_clock::now(); 124 | pktsOut ++; bytesOut += pktlen; 125 | } 126 | 127 | /** 128 | * Human-readable status check of the module. 129 | * 130 | * @return A HealthCheck class 131 | */ 132 | TunInterfaceHealthCheck TunInterface::status() 133 | { 134 | std::list thcs; 135 | for(auto &t : threads) 136 | thcs.push_back(t.status()); 137 | 138 | return { devname, pktsOut, bytesOut, lastPacket, thcs }; 139 | } 140 | 141 | /** 142 | * Return the last time any of our threads saw a packet. 143 | * @return 144 | */ 145 | std::chrono::steady_clock::time_point TunInterface::lastPacketTime() 146 | { 147 | std::chrono::steady_clock::time_point ret; 148 | 149 | for(auto &t : threads) 150 | { 151 | auto r = t.lastPacketTime(); 152 | if(r > ret) ret = r; 153 | } 154 | return ret; 155 | } 156 | 157 | /** 158 | * TunInterfaceThread class handles an individual thread assigned to processing packets coming in from the OS via the gwo interfaces. 159 | * Packets 160 | * - Launches a thread to service that interface 161 | * - Takes a callback function (recvDispatcher) which is called for each packet received by that interface. 162 | * - Provides a status() function that returns the packet counters and checks that the thread is still alive. 163 | */ 164 | 165 | TunInterfaceThread::TunInterfaceThread() 166 | : setupCalled(false),lastPacket(std::chrono::steady_clock::now()),pktsIn(0),pktsOut(0),bytesIn(0),bytesOut(0),shutdownRequested(false) 167 | { 168 | 169 | } 170 | 171 | TunInterfaceThread::~TunInterfaceThread() noexcept 172 | { 173 | shutdownRequested = true; 174 | // If this thread has been setup and is running, signal shutdown and wait for it to complete. 175 | if(thread.valid()) 176 | { 177 | auto status = thread.wait_for(std::chrono::seconds(2)); 178 | while(status == std::future_status::timeout) 179 | { 180 | LOG(LS_TUNNEL, LL_INFO, "Tunnel thread "s + ts(threadNumber) + " has not yet shutdown - waiting more."s); 181 | status = thread.wait_for(std::chrono::seconds(1)); 182 | } 183 | } 184 | } 185 | 186 | /** 187 | * Set up the tunnel handling thread and start it. 188 | * @param threadNum 189 | * @param coreNum 190 | * @param fd 191 | * @param recvDispatcher 192 | */ 193 | void TunInterfaceThread::setup(int threadNumberParam, int coreNumberParam, int fdParam, tunCallback recvDispatcherParam) 194 | { 195 | threadNumber = threadNumberParam; 196 | coreNumber = coreNumberParam; 197 | recvDispatcher = recvDispatcherParam; 198 | fd = fdParam; 199 | setupCalled = true; 200 | thread = std::async(&TunInterfaceThread::threadFunction, this); 201 | } 202 | 203 | /** 204 | * Thread for the tunnel handler. Opens a new fd, waits for packets to come in, then calls recvDispatch. 205 | * 206 | * @return Never returns until termination signal is sent. 207 | */ 208 | int TunInterfaceThread::threadFunction() 209 | { 210 | char threadName[16]; 211 | threadId = gettid(); 212 | snprintf(threadName, 15, "gwlbtun T%03d", threadNumber); 213 | pthread_setname_np(pthread_self(), threadName); 214 | LOG(LS_TUNNEL, LL_DEBUG, "Thread starting"); 215 | 216 | // If a specific core was requested, attempt to set affinity. 217 | if(coreNumber != -1) 218 | { 219 | cpu_set_t cpuset; 220 | CPU_ZERO(&cpuset); 221 | CPU_SET(coreNumber, &cpuset); 222 | int s = pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset); 223 | if(s != 0) 224 | { 225 | LOG(LS_TUNNEL, LL_INFO, "Unable to set TUN thread CPU affinity to core "s + ts(coreNumber) + ": "s + std::error_code{errno, std::generic_category()}.message() + ". Thread continuing to run with affinity unset."s); 226 | } else { 227 | snprintf(threadName, 15, "gwlbtun TA%03d", coreNumber); 228 | pthread_setname_np(pthread_self(), threadName); 229 | } 230 | } 231 | 232 | unsigned char *pktbuf; 233 | // Static packet processing buffer. 234 | pktbuf = new unsigned char[65535]; 235 | 236 | // Receive packets and dispatch them. Additionally, ensure a check at least every second to make sure a 237 | // shutdown hasn't been requested. 238 | ssize_t msgLen; 239 | struct timeval tv; 240 | fd_set readfds; 241 | while(!shutdownRequested) 242 | { 243 | tv.tv_sec = 1; tv.tv_usec = 0; 244 | FD_ZERO(&readfds); 245 | FD_SET(fd, &readfds); 246 | select(fd + 1, &readfds, nullptr, nullptr, &tv); 247 | if(FD_ISSET(fd, &readfds)) 248 | { 249 | // The tun interface has received packets. Drain all, dispatching each. 250 | msgLen = read(fd, pktbuf, 65534); // Remember: TUN devices always return 1 and only 1 packet on read() 251 | lastPacket = std::chrono::steady_clock::now(); 252 | try { 253 | recvDispatcher(pktbuf, msgLen); 254 | } 255 | catch (std::exception& e) { 256 | LOG(LS_TUNNEL, LL_IMPORTANT, "Packet dispatch function failed: "s + e.what()); 257 | } 258 | pktsIn ++; bytesIn += msgLen; 259 | } 260 | } 261 | LOG(LS_TUNNEL, LL_DEBUG, "Thread stopping by request"); 262 | delete [] pktbuf; 263 | return(0); 264 | } 265 | 266 | 267 | bool TunInterfaceThread::healthCheck() 268 | { 269 | if(thread.valid()) 270 | { 271 | auto status = thread.wait_for(std::chrono::seconds(0)); 272 | if(status != std::future_status::timeout) 273 | { 274 | return false; 275 | } 276 | return true; 277 | } 278 | return false; 279 | } 280 | 281 | TunInterfaceThreadHealthCheck TunInterfaceThread::status() 282 | { 283 | if(thread.valid()) 284 | return {true, healthCheck(), threadNumber, threadId, pktsIn, bytesIn, lastPacket }; 285 | else 286 | return { false, false, 0, 0, 0, 0, std::chrono::steady_clock::now() }; 287 | } 288 | 289 | void TunInterfaceThread::shutdown() 290 | { 291 | shutdownRequested = true; 292 | } 293 | 294 | std::chrono::steady_clock::time_point TunInterfaceThread::lastPacketTime() 295 | { 296 | return lastPacket.load(); 297 | } 298 | 299 | /* 300 | * Allocate a new fd for our tun device. May throw exceptions. 301 | * 302 | * @return The new file descriptor. 303 | */ 304 | int TunInterface::allocateHandle() 305 | { 306 | int fd; 307 | 308 | // Set up a new multiqueue file handler to process our packets 309 | struct ifreq ifr; 310 | bzero(&ifr, sizeof(ifr)); 311 | 312 | // Code adapted from Linux Documentation/networking/tuntap.txt to create the tun device. 313 | if((fd = open("/dev/net/tun", O_RDWR)) < 0) 314 | { 315 | LOG(LS_TUNNEL, LL_CRITICAL, "Unable to open /dev/net/tun "s + std::error_code{errno, std::generic_category()}.message()); 316 | throw std::system_error(errno, std::generic_category(), "Unable to open /dev/net/tun"); 317 | } 318 | 319 | ifr.ifr_flags = IFF_TUN | IFF_NO_PI | IFF_MULTI_QUEUE; 320 | strncpy(ifr.ifr_name, devname.c_str(), IFNAMSIZ); 321 | 322 | if(ioctl(fd, TUNSETIFF, (void *)&ifr) < 0) 323 | { 324 | LOG(LS_TUNNEL, LL_CRITICAL, "Unable to create TUN device " + devname + " (does this process have CAP_NET_ADMIN capability?) " + std::error_code{errno, std::generic_category()}.message()); 325 | throw std::system_error(errno, std::generic_category(), "Unable to create TUN device (does this process have CAP_NET_ADMIN capability?)"); 326 | } 327 | 328 | return fd; 329 | } 330 | 331 | TunInterfaceThreadHealthCheck::TunInterfaceThreadHealthCheck(bool threadValid, bool healthy, int threadNumber, 332 | int threadId, uint64_t pktsIn, uint64_t bytesIn, 333 | std::chrono::steady_clock::time_point lastPacket) : 334 | threadValid(threadValid), healthy(healthy), threadNumber(threadNumber), threadId(threadId), pktsIn(pktsIn), bytesIn(bytesIn), lastPacket(lastPacket) { 335 | } 336 | 337 | std::string TunInterfaceThreadHealthCheck::output_str() 338 | { 339 | std::string ret; 340 | if(threadValid) 341 | { 342 | ret = "Tunnel handler thread "s + std::to_string(threadNumber) + " (ID "s + std::to_string(threadId) + ")"s; 343 | 344 | if(healthy) 345 | ret += ": Healthy, "s; 346 | else { 347 | ret += ": NOT healthy, "s; 348 | } 349 | ret += std::to_string(pktsIn) + " packets in from OS, "s + std::to_string(bytesIn) + " bytes in from OS, "s; 350 | ret += timepointDeltaString(std::chrono::steady_clock::now(), lastPacket) + " since last packet.\n"; 351 | } 352 | return ret; 353 | } 354 | 355 | json TunInterfaceThreadHealthCheck::output_json() 356 | { 357 | return { {"valid", threadValid}, {"threadNumber", threadNumber}, {"threadId", threadId}, 358 | {"healthy", healthy}, {"pktsIn", pktsIn}, {"bytesIn", bytesIn}, {"secsSincelastPacket", timepointDeltaDouble(std::chrono::steady_clock::now(), lastPacket)} }; 359 | } 360 | 361 | TunInterfaceHealthCheck::TunInterfaceHealthCheck(std::string devname, uint64_t pktsOut, uint64_t bytesOut, std::chrono::steady_clock::time_point lastPacket, std::list thcs) : 362 | devname(devname), pktsOut(pktsOut), bytesOut(bytesOut), lastPacket(lastPacket), thcs(std::move(thcs)) 363 | {} 364 | 365 | std::string TunInterfaceHealthCheck::output_str() 366 | { 367 | std::string ret; 368 | 369 | ret += "Interface "s + devname + ":\n"s; 370 | 371 | ret += std::to_string(pktsOut) + " packets out to OS, "s + std::to_string(bytesOut) + " bytes out to OS, "s; 372 | ret += timepointDeltaString(std::chrono::steady_clock::now(), lastPacket) + " since last packet.\n"; 373 | 374 | for(auto &t : thcs) 375 | { 376 | ret += t.output_str(); 377 | } 378 | 379 | ret += "\n"s; 380 | 381 | return ret; 382 | } 383 | 384 | json TunInterfaceHealthCheck::output_json() 385 | { 386 | json ret; 387 | 388 | ret = { {"devname", devname}, {"pktsOut", pktsOut}, {"bytesOut", bytesOut}, {"secsSincelastPacket", timepointDeltaDouble(std::chrono::steady_clock::now(), lastPacket)} , 389 | {"threads", json::array()} }; 390 | 391 | for(auto &t : thcs) 392 | { 393 | auto js = t.output_json(); 394 | if(js["valid"] == true) 395 | ret["threads"].push_back(js); 396 | } 397 | 398 | return ret; 399 | } -------------------------------------------------------------------------------- /TunInterface.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWAPPLIANCE_TUNINTERFACE_H 5 | #define GWAPPLIANCE_TUNINTERFACE_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include "utils.h" 15 | #include "HealthCheck.h" 16 | 17 | 18 | typedef std::function tunCallback; 19 | 20 | class TunInterfaceThreadHealthCheck : public HealthCheck { 21 | public: 22 | TunInterfaceThreadHealthCheck(bool threadValid, bool healthy, int threadNumber, int threadId, uint64_t pktsIn, uint64_t bytesIn, std::chrono::steady_clock::time_point lastPacket); 23 | std::string output_str(); 24 | json output_json(); 25 | 26 | private: 27 | bool threadValid, healthy; 28 | int threadNumber, threadId; 29 | uint64_t pktsIn, bytesIn; 30 | std::chrono::steady_clock::time_point lastPacket; 31 | }; 32 | 33 | class TunInterfaceThread { 34 | public: 35 | TunInterfaceThread(); 36 | ~TunInterfaceThread(); 37 | 38 | void setup(int threadNum, int coreNum, int fd, tunCallback recvDispatcher); 39 | bool healthCheck(); 40 | TunInterfaceThreadHealthCheck status(); 41 | void shutdown(); 42 | bool setupCalled; 43 | std::chrono::steady_clock::time_point lastPacketTime(); 44 | 45 | private: 46 | std::atomic lastPacket; 47 | std::atomic pktsIn, pktsOut, bytesIn, bytesOut; 48 | tunCallback recvDispatcher; 49 | bool shutdownRequested; 50 | int fd; 51 | int threadNumber; 52 | int coreNumber; 53 | pid_t threadId; 54 | std::future thread; 55 | int threadFunction(); 56 | }; 57 | 58 | class TunInterfaceHealthCheck : public HealthCheck { 59 | public: 60 | TunInterfaceHealthCheck(std::string devname, uint64_t pktsOut, uint64_t bytesOut, std::chrono::steady_clock::time_point lastPacket, std::list thcs); 61 | std::string output_str(); 62 | json output_json(); 63 | 64 | private: 65 | std::string devname; 66 | uint64_t pktsOut, bytesOut; 67 | std::chrono::steady_clock::time_point lastPacket; 68 | std::list thcs; 69 | }; 70 | 71 | class TunInterface { 72 | public: 73 | TunInterface(std::string devname, int mtu, ThreadConfig threadConfig, tunCallback recvDispatcher); 74 | ~TunInterface(); 75 | 76 | void writePacket(unsigned char *pkt, ssize_t pktlen); 77 | TunInterfaceHealthCheck status(); 78 | void shutdown(); 79 | 80 | std::string devname; 81 | std::chrono::steady_clock::time_point lastPacketTime(); 82 | 83 | private: 84 | std::atomic lastPacket; 85 | std::atomic pktsOut, bytesOut; 86 | std::array threads; 87 | boost::concurrent_flat_map writerHandles; 88 | int allocateHandle(); 89 | }; 90 | 91 | #endif //GWAPPLIANCE_TUNINTERFACE_H 92 | -------------------------------------------------------------------------------- /UDPPacketReceiver.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * UDPPacketReceiver class handles monitoring a UDP port (6081 for GENEVE). It performs the following functions: 6 | * - Start a listener for the port 7 | * - Launch threads to listen for packets in on that port 8 | * - Call recvDispatcher for each packet received. 9 | * - Provides a status() function that returns the packet counters and checks that the thread is still alive. 10 | */ 11 | 12 | #include 13 | #include 14 | #include "UDPPacketReceiver.h" 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include "utils.h" 20 | #include "Logger.h" 21 | 22 | using namespace std::string_literals; 23 | 24 | /** 25 | * Default constructor for array initialization. 26 | */ 27 | UDPPacketReceiver::UDPPacketReceiver() 28 | : portNumber(0) 29 | { 30 | // Empty init. Need to set port and receive function with a setup() call. 31 | } 32 | 33 | /** 34 | * Destructor. Signals all threads to shut down, waits for that to finish. 35 | */ 36 | UDPPacketReceiver::~UDPPacketReceiver() 37 | { 38 | shutdown(); 39 | } 40 | 41 | void UDPPacketReceiver::shutdown() 42 | { 43 | // Signal all threads to shutdown, then wait for all acks. 44 | for(auto &thread : threads) 45 | { 46 | thread.shutdown(); 47 | } 48 | 49 | bool allgood = false; 50 | while(!allgood) 51 | { 52 | allgood = true; 53 | for(auto &thread : threads) 54 | { 55 | if(thread.setupCalled) 56 | { 57 | if(thread.healthCheck()) 58 | allgood = false; 59 | } 60 | } 61 | if(!allgood) 62 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); 63 | } 64 | } 65 | 66 | /** 67 | * Setup the receiver. Open a UDP receiver and starts the thread for it. 68 | * 69 | * @param portNumber UDP port number to listen to. 70 | * @param recvDispatcher Function to callback to on each packet received. 71 | */ 72 | void UDPPacketReceiver::setup(ThreadConfig threadConfig, uint16_t portNumberParam, udpCallback recvDispatcherParam) 73 | { 74 | LOG(LS_UDP, LL_DEBUG, "UDP receiver setting up on port "s + ts(portNumberParam)); 75 | portNumber = portNumberParam; 76 | 77 | // Set up our threads as per threadConfig 78 | int tIndex = 0; 79 | for(int core : threadConfig.cfg) 80 | { 81 | threads[tIndex].setup(tIndex, core, portNumberParam, recvDispatcherParam); 82 | tIndex ++; 83 | } 84 | } 85 | 86 | /** 87 | * Check on the status of the UDP receiver thread. 88 | * 89 | * @return true if the thread is still alive, false otherwise. 90 | */ 91 | bool UDPPacketReceiver::healthCheck() 92 | { 93 | bool status = true; 94 | 95 | for(auto &t : threads) 96 | { 97 | if(t.setupCalled) 98 | { 99 | if(!t.healthCheck()) 100 | status = false; 101 | } 102 | } 103 | 104 | return status; 105 | } 106 | 107 | /** 108 | * Human-readable status check of the module. 109 | * 110 | * @return A string containing thread status and packet counters. 111 | */ 112 | UDPPacketReceiverHealthCheck UDPPacketReceiver::status() { 113 | std::list thcs; 114 | 115 | for(auto &t : threads) 116 | { 117 | thcs.push_back(t.status()); 118 | } 119 | 120 | return {portNumber, thcs}; 121 | } 122 | 123 | /** 124 | * UDPPacketReceiverThread coordinates and holds the thread for one individual thread of UDP receiving. 125 | */ 126 | UDPPacketReceiverThread::UDPPacketReceiverThread() 127 | : setupCalled(false), sock(0), portNumber(0), threadNumber(0), coreNumber(0), shutdownRequested(false), lastPacket(std::chrono::steady_clock::now()), pktsIn(0), bytesIn(0) 128 | { 129 | // Empty init, until setup is called. 130 | } 131 | 132 | UDPPacketReceiverThread::~UDPPacketReceiverThread() 133 | { 134 | shutdownRequested = true; 135 | // If this thread has been setup and is running, signal shutdown and wait for it to complete. 136 | if(thread.valid()) 137 | { 138 | auto status = thread.wait_for(std::chrono::seconds(2)); 139 | while(status == std::future_status::timeout) 140 | { 141 | LOG(LS_UDP, LL_DEBUG, "UDP receiver thread "s + ts(threadNumber) + " has not yet shutdown - waiting more."s); 142 | status = thread.wait_for(std::chrono::seconds(1)); 143 | } 144 | } 145 | } 146 | 147 | void UDPPacketReceiverThread::setup(int threadNumberParam, int coreNumberParam, uint16_t portNumberParam, 148 | udpCallback recvDispatcherParam) 149 | { 150 | int yes = 1; 151 | struct sockaddr_in address{}; 152 | 153 | threadNumber = threadNumberParam; 154 | coreNumber = coreNumberParam; 155 | recvDispatcher = std::move(recvDispatcherParam); 156 | portNumber = portNumberParam; 157 | setupCalled = true; 158 | 159 | if ((sock = socket(AF_INET, SOCK_DGRAM, 0)) == 0) 160 | throw std::system_error(errno, std::generic_category(), "Socket creation failed"); 161 | 162 | if(setsockopt(sock, SOL_SOCKET, SO_REUSEADDR | SO_REUSEPORT, &yes, sizeof(yes))) 163 | throw std::system_error(errno, std::generic_category(), "Socket setsockopt() for port reuse failed"); 164 | 165 | address.sin_family = AF_INET; 166 | address.sin_addr.s_addr = INADDR_ANY; 167 | address.sin_port = htons(portNumber); 168 | 169 | // Set up to receive the packet info 170 | yes = 1; 171 | if(setsockopt(sock, IPPROTO_IP, IP_PKTINFO, &yes, sizeof(yes))) 172 | throw std::system_error(errno, std::generic_category(), "Socket setsockopt() for IP_PKTINFO failed"); 173 | 174 | if(bind(sock, (const struct sockaddr *)&address, (socklen_t)sizeof(address)) < 0) 175 | throw std::system_error(errno, std::generic_category(), "Port binding failed"); 176 | 177 | thread = std::async(&UDPPacketReceiverThread::threadFunction, this); 178 | } 179 | 180 | /** 181 | * Thread for the UDP receiver. Waits for packets to come in, then calls our dispatcher. 182 | * 183 | * @return Never returns, until told to shutdown. 184 | */ 185 | int UDPPacketReceiverThread::threadFunction() 186 | { 187 | struct sockaddr_storage src_addr{}; 188 | struct sockaddr_in *src_addr4; 189 | struct msghdr mh{}; 190 | struct cmsghdr *cmhdr; 191 | struct iovec iov[1]; 192 | struct in_pktinfo *ipi; 193 | unsigned char *pktbuf, *control; 194 | char threadName[16]; 195 | 196 | // Static packet processing buffers. 197 | pktbuf = new unsigned char[65535]; 198 | control = new unsigned char[2048]; 199 | threadId = gettid(); 200 | snprintf(threadName, 15, "gwlbtun U%03d", threadNumber); 201 | pthread_setname_np(pthread_self(), threadName); 202 | 203 | // If a specific core was requested, attempt to set affinity. 204 | if(coreNumber != -1) 205 | { 206 | cpu_set_t cpuset; 207 | CPU_ZERO(&cpuset); 208 | CPU_SET(coreNumber, &cpuset); 209 | int s = pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset); 210 | if(s != 0) 211 | { 212 | LOG(LS_UDP, LL_IMPORTANT, "Unable to set thread CPU affinity to core "s + ts(coreNumber) + ": "s + std::error_code{errno, std::generic_category()}.message() + ". Thread continuing to run with affinity unset."s); 213 | } else { 214 | snprintf(threadName, 15, "gwlbtun UA%03d", coreNumber); 215 | pthread_setname_np(pthread_self(), threadName); 216 | } 217 | } 218 | 219 | iov[0].iov_base = pktbuf; 220 | iov[0].iov_len = 65534; 221 | mh.msg_name = &src_addr; 222 | mh.msg_namelen = sizeof(src_addr); 223 | mh.msg_iov = iov; 224 | mh.msg_iovlen = 1; 225 | mh.msg_control = control; 226 | mh.msg_controllen = 2048; 227 | 228 | // Receive packets and dispatch them. check every second to make sure a shutdown hasn't been requested. 229 | // printf("UDP receive thread active.\n"); 230 | ssize_t msgLen; 231 | struct timeval tv{}; 232 | fd_set readfds; 233 | while(!shutdownRequested) 234 | { 235 | // printf("tick\n"); 236 | tv.tv_sec = 1; tv.tv_usec = 0; 237 | FD_ZERO(&readfds); 238 | FD_SET(sock, &readfds); 239 | select(sock + 1, &readfds, nullptr, nullptr, &tv); 240 | if(FD_ISSET(sock, &readfds)) 241 | { 242 | msgLen = recvmsg(sock, &mh, MSG_DONTWAIT); 243 | while(msgLen > 0 && !shutdownRequested) { 244 | if(src_addr.ss_family == AF_INET) 245 | { 246 | // Cycle through the control data to get the IP address this was sent to, then call dispatch. 247 | cmhdr = CMSG_FIRSTHDR(&mh); 248 | while(cmhdr) 249 | { 250 | if(cmhdr->cmsg_level == IPPROTO_IP && cmhdr->cmsg_type == IP_PKTINFO) 251 | { 252 | ipi = (struct in_pktinfo *)CMSG_DATA(cmhdr); 253 | src_addr4 = (struct sockaddr_in *)&src_addr; 254 | lastPacket = std::chrono::steady_clock::now(); 255 | pktsIn ++; 256 | bytesIn += msgLen; 257 | try { 258 | recvDispatcher(pktbuf, msgLen, &src_addr4->sin_addr, be16toh(src_addr4->sin_port), &ipi->ipi_spec_dst, portNumber); 259 | } 260 | catch (std::exception& e) { 261 | LOG(LS_UDP, LL_IMPORTANT, "UDP packet dispatch function failed: "s + e.what()); 262 | } 263 | } 264 | cmhdr = CMSG_NXTHDR(&mh, cmhdr); 265 | } 266 | } 267 | msgLen = recvmsg(sock, &mh, MSG_DONTWAIT); 268 | } 269 | } 270 | } 271 | // printf("UDP receive thread shutdown.\n"); 272 | delete [] pktbuf; 273 | delete [] control; 274 | return(0); 275 | } 276 | 277 | bool UDPPacketReceiverThread::healthCheck() 278 | { 279 | if(thread.valid()) 280 | { 281 | auto status = thread.wait_for(std::chrono::seconds(0)); 282 | if(status != std::future_status::timeout) 283 | { 284 | return false; 285 | } 286 | return true; 287 | } 288 | return false; 289 | } 290 | 291 | UDPPacketReceiverThreadHealthCheck UDPPacketReceiverThread::status() 292 | { 293 | if(thread.valid()) 294 | return {true, healthCheck(), threadNumber, threadId, pktsIn, bytesIn, lastPacket}; 295 | else 296 | return {false, false, 0, 0, 0, 0, std::chrono::steady_clock::now()}; 297 | } 298 | 299 | void UDPPacketReceiverThread::shutdown() 300 | { 301 | shutdownRequested = true; 302 | // The std::async threads will see that boolean change within 1 second, then exit. 303 | } 304 | 305 | 306 | UDPPacketReceiverThreadHealthCheck::UDPPacketReceiverThreadHealthCheck(bool threadValid, bool healthy, int threadNumber, int threadId, uint64_t pktsIn, uint64_t bytesIn, std::chrono::steady_clock::time_point lastPacket) : 307 | threadValid(threadValid), healthy(healthy), threadNumber(threadNumber), threadId(threadId), pktsIn(pktsIn), bytesIn(bytesIn), lastPacket(lastPacket) 308 | { 309 | } 310 | 311 | std::string UDPPacketReceiverThreadHealthCheck::output_str() 312 | { 313 | std::string ret; 314 | 315 | if(!threadValid) 316 | return ""; 317 | 318 | ret += "UDP receiver thread "s + std::to_string(threadNumber) + " (ID "s + std::to_string(threadId) + ")"s; 319 | if(healthy) 320 | ret += ": Healthy, "s; 321 | else { 322 | ret += ": NOT healthy, "s; 323 | } 324 | ret += std::to_string(pktsIn) + " packets in, "s + std::to_string(bytesIn) + " bytes in, "s; 325 | ret += timepointDeltaString(std::chrono::steady_clock::now(), lastPacket) + " since last packet.\n"s; 326 | 327 | return ret; 328 | } 329 | 330 | json UDPPacketReceiverThreadHealthCheck::output_json() 331 | { 332 | return { {"valid", threadValid}, {"healthy", healthy}, {"threadNumber", threadNumber}, {"threadId", threadId}, 333 | {"pktsIn", pktsIn}, {"bytesIn", bytesIn}, {"secsSinceLastPacket", timepointDeltaDouble(std::chrono::steady_clock::now(), lastPacket)} }; 334 | } 335 | 336 | UDPPacketReceiverHealthCheck::UDPPacketReceiverHealthCheck(uint16_t portNumber, std::list threadHealthChecks) : 337 | portNumber(portNumber), threadHealthChecks(std::move(threadHealthChecks)) 338 | { 339 | } 340 | 341 | std::string UDPPacketReceiverHealthCheck::output_str() 342 | { 343 | std::string ret; 344 | 345 | ret += "UDP receiver threads for port number "s + std::to_string(portNumber) + ":\n"; 346 | for(auto &t : threadHealthChecks) 347 | ret += t.output_str(); 348 | 349 | ret += "\n"; 350 | 351 | return ret; 352 | } 353 | 354 | json UDPPacketReceiverHealthCheck::output_json() 355 | { 356 | json ret; 357 | 358 | ret["UDPPacketReceiver"] = { {"portNumber", portNumber}, {"threads", json::array()} }; 359 | 360 | for(auto &t : threadHealthChecks) 361 | { 362 | auto js = t.output_json(); 363 | if (js["valid"] == true) 364 | ret["UDPPacketReceiver"]["threads"].push_back(js); 365 | } 366 | 367 | return ret; 368 | } -------------------------------------------------------------------------------- /UDPPacketReceiver.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWAPPLIANCE_UDPPACKETRECEIVER_H 5 | #define GWAPPLIANCE_UDPPACKETRECEIVER_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include "utils.h" 12 | #include "HealthCheck.h" 13 | 14 | typedef std::function udpCallback; 15 | 16 | 17 | class UDPPacketReceiverThreadHealthCheck : public HealthCheck { 18 | public: 19 | UDPPacketReceiverThreadHealthCheck(bool threadValid, bool healthy, int threadNumber, int threadId, uint64_t pktsIn, uint64_t bytesIn, std::chrono::steady_clock::time_point lastPacket); 20 | std::string output_str(); 21 | json output_json(); 22 | 23 | private: 24 | bool threadValid, healthy; 25 | int threadNumber, threadId; 26 | uint64_t pktsIn, bytesIn; 27 | std::chrono::steady_clock::time_point lastPacket; 28 | }; 29 | 30 | class UDPPacketReceiverThread { 31 | public: 32 | UDPPacketReceiverThread(); 33 | ~UDPPacketReceiverThread(); 34 | 35 | void setup(int threadNumberParam, int coreNumberParam, uint16_t portNumberParam, udpCallback recvDispatcherParam); 36 | bool healthCheck(); 37 | UDPPacketReceiverThreadHealthCheck status(); 38 | void shutdown(); 39 | bool setupCalled; 40 | 41 | private: 42 | int sock; 43 | uint16_t portNumber; 44 | int threadNumber; 45 | int coreNumber; 46 | bool shutdownRequested; 47 | pid_t threadId; 48 | std::future thread; 49 | udpCallback recvDispatcher; 50 | int threadFunction(); 51 | std::atomic lastPacket; 52 | std::atomic pktsIn, bytesIn; 53 | }; 54 | 55 | class UDPPacketReceiverHealthCheck : public HealthCheck { 56 | public: 57 | UDPPacketReceiverHealthCheck(uint16_t portNumber, std::list threadHealthChecks); 58 | std::string output_str() ; 59 | json output_json(); 60 | 61 | private: 62 | uint16_t portNumber; 63 | std::list threadHealthChecks; 64 | }; 65 | 66 | class UDPPacketReceiver { 67 | public: 68 | UDPPacketReceiver(); 69 | ~UDPPacketReceiver(); 70 | 71 | void setup(ThreadConfig threadConfig, uint16_t portNumberParam, udpCallback recvDispatcherParam); 72 | bool healthCheck(); 73 | UDPPacketReceiverHealthCheck status(); 74 | void shutdown(); 75 | 76 | private: 77 | uint16_t portNumber; 78 | std::array threads; 79 | }; 80 | 81 | 82 | #endif //GWAPPLIANCE_UDPPACKETRECEIVER_H 83 | -------------------------------------------------------------------------------- /example-scripts/create-nat-overlapping.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # This version of the create-nat script deals with multiple endpoints that may have overlapping IP CIDR ranges. 4 | 5 | # Note: This requires this instance to have Source/Dest check disabled. 6 | # aws ec2 modify-instance-attribute --instance-id= --no-source-dest-check 7 | 8 | echo Mode is $1, In Int is $2, Out Int is $3, ENI is $4 9 | 10 | if [ "$1" = "CREATE" ]; then 11 | echo "==> Setting up two-armed NAT" 12 | 13 | # Make sure conntrack mod is installed 14 | modprobe ip_conntrack 15 | 16 | # Ensure there is a route table named for this ENI 17 | grep $4 /etc/iproute2/rt_tables > /dev/null 18 | if [ $? -ne 0 ]; then 19 | LASTTAB=`grep -v "#" /etc/iproute2/rt_tables | cut -f 1 | tail -n 1` 20 | NEWTAB=$((LASTTAB+1)) 21 | echo -e "$NEWTAB\t$4" >> /etc/iproute2/rt_tables 22 | fi 23 | 24 | # Since in stock Linux, our new tables will start with an ID of 1 and go up, we will re-use that number as 25 | # the eth0 alias for numbering 26 | TABLENUM=`grep $4 /etc/iproute2/rt_tables | cut -f 1` 27 | 28 | # Flush the existing route table 29 | ip route flush table $4 30 | # Set output routes. Get the VPC assigned IPv4 and IPv6 blocks for the VPC this ENI ends at 31 | VPCID=`aws ec2 describe-vpc-endpoints --filters Name=vpc-endpoint-id,Values=vpce-$4 --region us-west-2 | jq -r .VpcEndpoints[].VpcId` 32 | IPV4=`aws ec2 describe-vpcs --filters Name=vpc-id,Values=$VPCID --region us-west-2 | jq -r .Vpcs[].CidrBlockAssociationSet[].CidrBlock` 33 | IPV6=`aws ec2 describe-vpcs --filters Name=vpc-id,Values=$VPCID --region us-west-2 | jq -r .Vpcs[].Ipv6CidrBlockAssociationSet[].Ipv6CidrBlock` 34 | echo "- Routing far-side VPC blocks of $IPV4 and $IPV6 via $3 on table $4" 35 | ip route add $IPV4 dev $3 table $4 36 | ip -6 route add $IPV6 dev $3 table $4 37 | 38 | # Add default routes 39 | ip route add 0.0.0.0/0 via 10.10.0.1 table $4 40 | 41 | # Get the ::1 address of eth0 42 | OURV6=`ifconfig eth0 | grep "prefixlen 128" | cut -f 10 -d ' '` 43 | DRV6=`python3 -c "import ipaddress as ipa; a=ipa.ip_interface('$OURV6/64'); print(a.network.network_address + 1);"` 44 | ip -6 route add ::/0 via $DRV6 table $4 45 | 46 | # Add interfaces to that route table for outbound 47 | ip rule add iif $2 table $4 48 | ip -6 rule add iif $2 table $4 49 | 50 | # Set up IPTables to do the NAT. We do a mark on ingress to make it easy to match on egress. 51 | # We save and restore the mark using CONNTRACK to do our connection tracking 52 | 53 | # Traffic outbound - set a mark on ingress and save it on egress to CONNMARK for restoring on inbound 54 | iptables -t mangle -A PREROUTING -i $2 -j MARK --set-mark $TABLENUM 55 | iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark 56 | iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE 57 | ip6tables -t mangle -A PREROUTING -i $2 -j MARK --set-mark $TABLENUM 58 | ip6tables -t mangle -A POSTROUTING -j CONNMARK --save-mark 59 | ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE 60 | 61 | # Traffic inbound - restore the mark from CONNMARK 62 | iptables -t mangle -A PREROUTING -i eth0 -j CONNMARK --restore-mark 63 | ip6tables -t mangle -A PREROUTING -i eth0 -j CONNMARK --restore-mark 64 | # Use that mark to go to the correct routing table 65 | ip rule add fwmark $TABLENUM table $4 66 | ip -6 rule add fwmark $TABLENUM table $4 67 | 68 | echo 1 > /proc/sys/net/ipv4/ip_forward 69 | echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter 70 | echo 0 > /proc/sys/net/ipv4/conf/$2/rp_filter 71 | echo 1 > /proc/sys/net/ipv6/conf/all/forwarding 72 | else 73 | echo "==> Destroying up two-armed NAT" 74 | 75 | # Delete our rules we created 76 | ip rule del table $4 77 | ip rule del table $4 78 | ip -6 rule del table $4 79 | ip -6 rule del table $4 80 | 81 | fi 82 | -------------------------------------------------------------------------------- /example-scripts/create-nat.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: This requires this instance to have Source/Dest check disabled. 4 | # aws ec2 modify-instance-attribute --instance-id= --no-source-dest-check 5 | 6 | echo Mode is $1, In Int is $2, Out Int is $3, ENI is $4 7 | 8 | # Get our output interface (eth0, ens5, etc) 9 | OUTINT=`ip route show default | cut -f 5 -d ' '` 10 | 11 | if [ "$1" = "CREATE" ]; then 12 | echo "==> Setting up two-armed NAT" 13 | 14 | iptables -t nat -A POSTROUTING -o $OUTINT -j MASQUERADE 15 | iptables -A FORWARD -i $2 -o $OUTINT -j ACCEPT 16 | ip6tables -t nat -A POSTROUTING -o $OUTINT -j MASQUERADE 17 | ip6tables -A FORWARD -i $2 -o $OUTINT -j ACCEPT 18 | 19 | echo 1 > /proc/sys/net/ipv4/ip_forward 20 | echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter 21 | echo 0 > /proc/sys/net/ipv4/conf/$2/rp_filter 22 | echo 1 > /proc/sys/net/ipv6/conf/all/forwarding 23 | else 24 | echo "==> Destroying up two-armed NAT" 25 | 26 | iptables -t nat -D POSTROUTING -o $OUTINT -j MASQUERADE 27 | iptables -D FORWARD -i $2 -o $OUTINT -j ACCEPT 28 | ip6tables -t nat -D POSTROUTING -o $OUTINT -j MASQUERADE 29 | ip6tables -D FORWARD -i $2 -o $OUTINT -j ACCEPT 30 | fi 31 | -------------------------------------------------------------------------------- /example-scripts/create-overlapping-v4.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # This version of the create-nat script deals with multiple endpoints that may have overlapping IP CIDR ranges. 4 | 5 | # Note: This requires this instance to have Source/Dest check disabled. 6 | # aws ec2 modify-instance-attribute --instance-id= --no-source-dest-check 7 | 8 | echo Mode is $1, In Int is $2, Out Int is $3, ENI is $4 9 | 10 | if [ "$1" = "CREATE" ]; then 11 | echo "==> Setting up two-armed NAT" 12 | 13 | # Make sure conntrack mod is installed 14 | modprobe ip_conntrack 15 | 16 | # Get what interface are we routing traffic out to (eth0, enX0, etc) 17 | OUTINT=`ip route show default | cut -f 5 -d ' '` 18 | 19 | # Have we set our global rules yet? If not, set them up. 20 | iptables -t mangle -L INPUT -n | grep -q "CONNMARK save" 21 | if [ $? -eq 1 ]; then 22 | # Need to set global rules. 23 | echo "- First tunnel detected. Setting global networking configuration." 24 | # Save connmark on egress 25 | iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark 26 | # Enable NAT to our outbound interface 27 | iptables -t nat -A POSTROUTING -o $OUTINT -j MASQUERADE 28 | # Ensure traffic go to us locally (INPUT) and coming from us locally (OUTPUT) are marked and restored. 29 | iptables -t mangle -A INPUT -j CONNMARK --save-mark 30 | iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark 31 | # Traffic inbound from NATting- restore the mark from CONNMARK 32 | iptables -t mangle -A PREROUTING -i $OUTINT -j CONNMARK --restore-mark 33 | # Enable forwarding. 34 | echo 1 > /proc/sys/net/ipv4/ip_forward 35 | echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter 36 | fi 37 | 38 | # Ensure there is a route table named for this ENI 39 | grep $4 /etc/iproute2/rt_tables > /dev/null 40 | if [ $? -ne 0 ]; then 41 | LASTTAB=`grep -v "#" /etc/iproute2/rt_tables | cut -f 1 | tail -n 1` 42 | NEWTAB=$((LASTTAB+1)) 43 | echo -e "$NEWTAB\t$4" >> /etc/iproute2/rt_tables 44 | fi 45 | TABLENUM=`grep $4 /etc/iproute2/rt_tables | cut -f 1` 46 | 47 | # Flush the existing route table 48 | ip route flush table $4 49 | # Set output routes. Get the VPC assigned IPv4 and IPv6 blocks for the VPC this ENI ends at 50 | VPCID=`aws ec2 describe-vpc-endpoints --filters Name=vpc-endpoint-id,Values=vpce-$4 | jq -r .VpcEndpoints[].VpcId` 51 | IPV4=`aws ec2 describe-vpcs --filters Name=vpc-id,Values=$VPCID | jq -r .Vpcs[].CidrBlockAssociationSet[].CidrBlock` 52 | echo "- Routing far-side VPC blocks of $IPV4 via $3 on table $4" 53 | ip route add $IPV4 dev $3 table $4 54 | 55 | echo "- Routing all other traffic received from $2 to be NATed out $OUTINT" 56 | # Add default routes to send traffic out for NAT'ing 57 | ip route add 0.0.0.0/0 dev $OUTINT table $4 58 | 59 | # Add interfaces to that route table for outbound 60 | ip rule add iif $2 table $4 61 | 62 | # Traffic outbound - set a mark on ingress and save it on egress to CONNMARK for restoring on inbound 63 | iptables -t mangle -A PREROUTING -i $2 -j MARK --set-mark $TABLENUM 64 | # Use those marks to go to the correct routing table 65 | ip rule add fwmark $TABLENUM table $4 66 | # Ignore reverse path filter for this interface 67 | echo 0 > /proc/sys/net/ipv4/conf/$2/rp_filter 68 | 69 | else 70 | echo "==> Destroying up two-armed NAT" 71 | 72 | # Delete our rules we created 73 | ip rule del fwmark $TABLENUM table $4 74 | ip rule del table $4 75 | ip rule del table $4 76 | 77 | # Delete table entries create 78 | iptables -t mangle -D PREROUTING -i $2 -j MARK --set-mark $TABLENUM 79 | fi 80 | -------------------------------------------------------------------------------- /example-scripts/create-passthrough.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "==> Setting up simple passthrough" 4 | echo Mode is $1, In Int is $2, Out Int is $3, ENI is $4 5 | 6 | tc qdisc add dev $2 ingress 7 | tc filter add dev $2 parent ffff: protocol all prio 2 u32 match u32 0 0 flowid 1:1 action mirred egress mirror dev $3 8 | -------------------------------------------------------------------------------- /example-scripts/create-route.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "==> Setting up to simply route incoming packets back out" 4 | echo Mode is $1, In Int is $2, Out Int is $3, ENI is $4 5 | 6 | sysctl net.ipv4.ip_forward=1 7 | sysctl net.ipv4.conf.$2.rp_filter=0 8 | sysctl net.ipv6.conf.all.forwarding=1 9 | -------------------------------------------------------------------------------- /example-scripts/example-topology-one-arm.template: -------------------------------------------------------------------------------- 1 | --- 2 | AWSTemplateFormatVersion: "2010-09-09" 3 | Description: Set up a demonstration topology for using gwlbtun in one-arm mode 4 | Parameters: 5 | LinuxAmiId: 6 | Type: AWS::SSM::Parameter::Value 7 | Description: AMI ID to use for GWLB appliances 8 | Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2' 9 | GWLBInstanceType: 10 | Type: String 11 | Description: Instance type code to use for the GWLB appliances. The default is fine for light testing, for heavier workloads you should consider a c5n instance. 12 | Default: t2.micro 13 | ApplicationInstanceType: 14 | Type: String 15 | Description: Instance type code to use for the application. 16 | Default: t2.micro 17 | 18 | Resources: 19 | EC2Role: 20 | Type: AWS::IAM::Role 21 | Properties: 22 | RoleName: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 23 | Description: Role for all EC2 instances to run under to allow SSM and S3 24 | AssumeRolePolicyDocument: 25 | Version: "2012-10-17" 26 | Statement: 27 | - Effect: Allow 28 | Principal: 29 | Service: 30 | - ec2.amazonaws.com 31 | Action: 32 | - 'sts:AssumeRole' 33 | Policies: 34 | - PolicyName: AllowIPAssign 35 | PolicyDocument: 36 | Version: "2012-10-17" 37 | Statement: 38 | - Effect: Allow 39 | Action: 40 | - "ec2:AssignPrivateIpAddresses" 41 | - "ec2:UnassignPrivateIpAddresses" 42 | - "ec2:AssignIpv6Addresses" 43 | - "ec2:UnassignIpv6Addresses" 44 | Resource: !Sub "arn:aws:ec2:*:${AWS::AccountId}:network-interface/*" 45 | ManagedPolicyArns: 46 | - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore 47 | - arn:aws:iam::aws:policy/AmazonSSMPatchAssociation 48 | - arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess 49 | - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess 50 | EC2RoleInstanceProfile: 51 | Type: AWS::IAM::InstanceProfile 52 | Properties: 53 | InstanceProfileName: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 54 | Path: / 55 | Roles: 56 | - !Ref EC2Role 57 | 58 | GWLBTunVPC: 59 | Type: AWS::EC2::VPC 60 | Properties: 61 | CidrBlock: 10.10.0.0/16 62 | EnableDnsHostnames: true 63 | EnableDnsSupport: true 64 | Tags: 65 | - Key: Name 66 | Value: !Join [ "-", [ !Ref AWS::StackName, "gwlbtun" ] ] 67 | 68 | GWLBTunVPCv6: 69 | Type: AWS::EC2::VPCCidrBlock 70 | Properties: 71 | VpcId: !Ref GWLBTunVPC 72 | AmazonProvidedIpv6CidrBlock: true 73 | 74 | GWLBTunPrivateSubnet: 75 | Type: AWS::EC2::Subnet 76 | DependsOn: GWLBTunVPCv6 77 | Properties: 78 | AvailabilityZone: !Select [0, !GetAZs '' ] 79 | CidrBlock: !Select [0, !Cidr [!GetAtt GWLBTunVPC.CidrBlock, 8, 8]] 80 | Ipv6CidrBlock: !Select [0, !Cidr [!Select [0, !GetAtt GWLBTunVPC.Ipv6CidrBlocks], 8, 64]] 81 | VpcId: !Ref GWLBTunVPC 82 | Tags: 83 | - Key: Name 84 | Value: !Join [ "-", [ !Ref AWS::StackName, "gwlbtun private" ] ] 85 | 86 | GWLBTunPrivateRT: 87 | Type: AWS::EC2::RouteTable 88 | Properties: 89 | VpcId: !Ref GWLBTunVPC 90 | Tags: 91 | - Key: Name 92 | Value: !Join [ "-", [ !Ref AWS::StackName, "gwlbtun private" ] ] 93 | 94 | GWLBTunPrivateRTA: 95 | Type: AWS::EC2::SubnetRouteTableAssociation 96 | Properties: 97 | SubnetId: !Ref GWLBTunPrivateSubnet 98 | RouteTableId: !Ref GWLBTunPrivateRT 99 | 100 | # Set up the GWLB 101 | GWLB: 102 | Type: AWS::ElasticLoadBalancingV2::LoadBalancer 103 | Properties: 104 | Name: !Join [ "-", [ !Ref AWS::StackName, "GWLB" ] ] 105 | Type: gateway 106 | IpAddressType: dualstack 107 | Subnets: 108 | - !Ref GWLBTunPrivateSubnet 109 | 110 | GWLBTargetGroup: 111 | Type: AWS::ElasticLoadBalancingV2::TargetGroup 112 | Properties: 113 | Name: !Join [ "-", [ !Ref AWS::StackName, "Instances" ] ] 114 | VpcId: !Ref GWLBTunVPC 115 | Protocol: GENEVE 116 | Port: 6081 117 | HealthCheckProtocol: HTTP 118 | HealthCheckPort: 80 119 | HealthCheckPath: / 120 | HealthCheckEnabled: true 121 | TargetType: ip # TargetType IP allows specifying the specific NIC on the GWLBTun instance to target 122 | Targets: 123 | - Id: !GetAtt GWLBTunInstanceOneArm.PrivateIp 124 | 125 | GWLBListener: 126 | Type: AWS::ElasticLoadBalancingV2::Listener 127 | Properties: 128 | LoadBalancerArn: !Ref GWLB 129 | DefaultActions: 130 | - Type: forward 131 | TargetGroupArn: !Ref GWLBTargetGroup 132 | 133 | GWLBEndpointService: 134 | Type: AWS::EC2::VPCEndpointService 135 | Properties: 136 | GatewayLoadBalancerArns: 137 | - !Ref GWLB 138 | AcceptanceRequired: false 139 | 140 | GWLBe: 141 | Type: AWS::EC2::VPCEndpoint 142 | DependsOn: 143 | - GWLBEndpointService 144 | - GWLBTunInstanceOneArm 145 | Properties: 146 | VpcId: !Ref ApplicationVPC 147 | ServiceName: !Sub "com.amazonaws.vpce.${AWS::Region}.${GWLBEndpointService}" 148 | VpcEndpointType: GatewayLoadBalancer 149 | SubnetIds: 150 | - !Ref ApplicationGWLBeSubnet 151 | 152 | # Build a GWLBTun instance. 153 | # The yum installs tc for AL2, iproute-tc for AL2022. One or the other will always fail - this is expected. 154 | GWLBTunInstanceOneArm: 155 | Type: AWS::EC2::Instance 156 | Properties: 157 | ImageId: !Ref LinuxAmiId 158 | InstanceType: !Ref GWLBInstanceType 159 | IamInstanceProfile: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 160 | BlockDeviceMappings: 161 | - DeviceName: /dev/xvda 162 | Ebs: 163 | VolumeSize: 9 164 | NetworkInterfaces: 165 | - DeviceIndex: 0 166 | AssociatePublicIpAddress: True 167 | SubnetId: !Ref GWLBTunPrivateSubnet 168 | GroupSet: 169 | - Ref: GWLBTunSG 170 | UserData: 171 | Fn::Base64: | 172 | #!/bin/bash -ex 173 | yum -y groupinstall "Development Tools" 174 | yum -y install cmake3 175 | yum -y install tc || true 176 | yum -y install iproute-tc || true 177 | cd /root 178 | git clone https://github.com/aws-samples/aws-gateway-load-balancer-tunnel-handler.git 179 | cd aws-gateway-load-balancer-tunnel-handler 180 | cmake3 . 181 | make 182 | 183 | echo "[Unit]" > /usr/lib/systemd/system/gwlbtun.service 184 | echo "Description=AWS GWLB Tunnel Handler" >> /usr/lib/systemd/system/gwlbtun.service 185 | echo "" >> /usr/lib/systemd/system/gwlbtun.service 186 | echo "[Service]" >> /usr/lib/systemd/system/gwlbtun.service 187 | echo "ExecStart=/root/aws-gateway-load-balancer-tunnel-handler/gwlbtun -c /root/aws-gateway-load-balancer-tunnel-handler/example-scripts/create-passthrough.sh -p 80" >> /usr/lib/systemd/system/gwlbtun.service 188 | echo "Restart=always" >> /usr/lib/systemd/system/gwlbtun.service 189 | echo "RestartSec=5s" >> /usr/lib/systemd/system/gwlbtun.service 190 | 191 | systemctl daemon-reload 192 | systemctl enable --now --no-block gwlbtun.service 193 | systemctl start gwlbtun.service 194 | echo 195 | Tags: 196 | - Key: Name 197 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun" ] ] 198 | 199 | GWLBTunSG: 200 | Type: AWS::EC2::SecurityGroup 201 | Properties: 202 | VpcId: !Ref GWLBTunVPC 203 | GroupName: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun Security" ] ] 204 | GroupDescription: GWLBTun Security 205 | SecurityGroupIngress: 206 | - CidrIp: !GetAtt GWLBTunVPC.CidrBlock 207 | IpProtocol: -1 208 | 209 | # Make an Internet Gateway in our GWLB VPC so it can download our source code and some packages. 210 | GWLBTunIG: 211 | Type: AWS::EC2::InternetGateway 212 | Properties: 213 | Tags: 214 | - Key: Name 215 | Value: !Join [ "-", [ !Ref AWS::StackName, "gwlbtun" ] ] 216 | 217 | GWLBTunIGA: 218 | Type: AWS::EC2::VPCGatewayAttachment 219 | Properties: 220 | InternetGatewayId: !Ref GWLBTunIG 221 | VpcId: !Ref GWLBTunVPC 222 | 223 | GWLBTunIGRoute: 224 | Type: AWS::EC2::Route 225 | DependsOn: GWLBTunIGA 226 | Properties: 227 | RouteTableId: !Ref GWLBTunPrivateRT 228 | DestinationCidrBlock: 0.0.0.0/0 229 | GatewayId: !Ref GWLBTunIG 230 | 231 | # Set up the Application VPC 232 | ApplicationVPC: 233 | Type: AWS::EC2::VPC 234 | Properties: 235 | CidrBlock: 10.20.0.0/16 236 | EnableDnsHostnames: true 237 | EnableDnsSupport: true 238 | Tags: 239 | - Key: Name 240 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application VPC" ] ] 241 | 242 | ApplicationVPCv6: 243 | Type: AWS::EC2::VPCCidrBlock 244 | Properties: 245 | VpcId: !Ref ApplicationVPC 246 | AmazonProvidedIpv6CidrBlock: true 247 | 248 | ApplicationPrivateSubnet: 249 | Type: AWS::EC2::Subnet 250 | DependsOn: ApplicationVPCv6 251 | Properties: 252 | AvailabilityZone: !Select [0, !GetAZs '' ] 253 | CidrBlock: !Select [0, !Cidr [!GetAtt ApplicationVPC.CidrBlock, 8, 8]] 254 | Ipv6CidrBlock: !Select [0, !Cidr [!Select [0, !GetAtt ApplicationVPC.Ipv6CidrBlocks], 8, 64]] 255 | VpcId: !Ref ApplicationVPC 256 | Tags: 257 | - Key: Name 258 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application private" ] ] 259 | 260 | ApplicationPrivateRT: 261 | Type: AWS::EC2::RouteTable 262 | Properties: 263 | VpcId: !Ref ApplicationVPC 264 | Tags: 265 | - Key: Name 266 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application private" ] ] 267 | 268 | ApplicationPrivateRTA: 269 | Type: AWS::EC2::SubnetRouteTableAssociation 270 | Properties: 271 | SubnetId: !Ref ApplicationPrivateSubnet 272 | RouteTableId: !Ref ApplicationPrivateRT 273 | 274 | ApplicationGWLBeSubnet: 275 | Type: AWS::EC2::Subnet 276 | DependsOn: ApplicationVPCv6 277 | Properties: 278 | AvailabilityZone: !Select [0, !GetAZs '' ] 279 | CidrBlock: !Select [ 1, !Cidr [ !GetAtt ApplicationVPC.CidrBlock, 8, 8]] 280 | Ipv6CidrBlock: !Select [ 1, !Cidr [ !Select [ 0, !GetAtt ApplicationVPC.Ipv6CidrBlocks ], 8, 64 ] ] 281 | VpcId: !Ref ApplicationVPC 282 | Tags: 283 | - Key: Name 284 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application GWLBe" ] ] 285 | 286 | ApplicationGWLBeRT: 287 | Type: AWS::EC2::RouteTable 288 | Properties: 289 | VpcId: !Ref ApplicationVPC 290 | Tags: 291 | - Key: Name 292 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application GWLBe" ] ] 293 | 294 | ApplicationGWLBeRTA: 295 | Type: AWS::EC2::SubnetRouteTableAssociation 296 | Properties: 297 | SubnetId: !Ref ApplicationGWLBeSubnet 298 | RouteTableId: !Ref ApplicationGWLBeRT 299 | 300 | ApplicationNATSubnet: 301 | Type: AWS::EC2::Subnet 302 | DependsOn: ApplicationVPCv6 303 | Properties: 304 | AvailabilityZone: !Select [0, !GetAZs '' ] 305 | CidrBlock: !Select [ 2, !Cidr [ !GetAtt ApplicationVPC.CidrBlock, 8, 8]] 306 | Ipv6CidrBlock: !Select [ 2, !Cidr [ !Select [ 0, !GetAtt ApplicationVPC.Ipv6CidrBlocks ], 8, 64 ] ] 307 | VpcId: !Ref ApplicationVPC 308 | Tags: 309 | - Key: Name 310 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application NAT" ] ] 311 | 312 | ApplicationNATRT: 313 | Type: AWS::EC2::RouteTable 314 | Properties: 315 | VpcId: !Ref ApplicationVPC 316 | Tags: 317 | - Key: Name 318 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application NAT" ] ] 319 | 320 | ApplicationNATRTA: 321 | Type: AWS::EC2::SubnetRouteTableAssociation 322 | Properties: 323 | SubnetId: !Ref ApplicationNATSubnet 324 | RouteTableId: !Ref ApplicationNATRT 325 | 326 | # Make a NAT Gateway in our application VPC for traffic leaving GWLB to go to 327 | ApplicationNGEIP: 328 | Type: AWS::EC2::EIP 329 | Properties: 330 | Domain: vpc 331 | 332 | ApplicationNG: 333 | Type: AWS::EC2::NatGateway 334 | Properties: 335 | ConnectivityType: public 336 | AllocationId: !GetAtt ApplicationNGEIP.AllocationId 337 | SubnetId: !Ref ApplicationNATSubnet 338 | 339 | # Make an Internet Gateway in our application VPC and route to it from the public subnet. 340 | ApplicationIG: 341 | Type: AWS::EC2::InternetGateway 342 | Properties: 343 | Tags: 344 | - Key: Name 345 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application" ] ] 346 | 347 | ApplicationIGA: 348 | Type: AWS::EC2::VPCGatewayAttachment 349 | Properties: 350 | InternetGatewayId: !Ref ApplicationIG 351 | VpcId: !Ref ApplicationVPC 352 | 353 | ApplicationIGRT: 354 | Type: AWS::EC2::RouteTable 355 | Properties: 356 | VpcId: !Ref ApplicationVPC 357 | Tags: 358 | - Key: Name 359 | Value: !Join [ "-", [ !Ref AWS::StackName, "Internet Gateway" ] ] 360 | 361 | ApplicationIGRTA: 362 | Type: AWS::EC2::GatewayRouteTableAssociation 363 | Properties: 364 | GatewayId: !Ref ApplicationIG 365 | RouteTableId: !Ref ApplicationIGRT 366 | 367 | # Route traffic from our applications to the GWLB endpoint 368 | ApplicationPrivateGWLBRouteOutv4: 369 | Type: AWS::EC2::Route 370 | Properties: 371 | RouteTableId: !Ref ApplicationPrivateRT 372 | VpcEndpointId: !Ref GWLBe 373 | DestinationCidrBlock: 0.0.0.0/0 374 | 375 | ApplicationPrivateGWLBRouteOutv6: 376 | Type: AWS::EC2::Route 377 | Properties: 378 | RouteTableId: !Ref ApplicationPrivateRT 379 | VpcEndpointId: !Ref GWLBe 380 | DestinationIpv6CidrBlock: ::/0 381 | 382 | # Route the return from GWLB to the NAT Gateway for non-local v4 and the IG for non-local v6 383 | ApplicationGWLBtoNATRouteOutv4: 384 | Type: AWS::EC2::Route 385 | DependsOn: ApplicationIGA 386 | Properties: 387 | RouteTableId: !Ref ApplicationGWLBeRT 388 | DestinationCidrBlock: 0.0.0.0/0 389 | NatGatewayId: !Ref ApplicationNG 390 | 391 | ApplicationGWLBtoNATRouteOutv6: 392 | Type: AWS::EC2::Route 393 | DependsOn: ApplicationIGA 394 | Properties: 395 | RouteTableId: !Ref ApplicationGWLBeRT 396 | DestinationIpv6CidrBlock: ::/0 397 | GatewayId: !Ref ApplicationIG 398 | 399 | # Route traffic out from the NAT GW to the IGW 400 | ApplicationNATtoIGWRouteOut: 401 | Type: AWS::EC2::Route 402 | DependsOn: ApplicationIGA 403 | Properties: 404 | RouteTableId: !Ref ApplicationNATRT 405 | DestinationCidrBlock: 0.0.0.0/0 406 | GatewayId: !Ref ApplicationIG 407 | 408 | # Route traffic coming in back to our GWLBe 409 | ApplicationNATtoGWLBeRouteInv4: 410 | Type: AWS::EC2::Route 411 | DependsOn: ApplicationIGA 412 | Properties: 413 | RouteTableId: !Ref ApplicationNATRT 414 | DestinationCidrBlock: !GetAtt ApplicationPrivateSubnet.CidrBlock 415 | VpcEndpointId: !Ref GWLBe 416 | 417 | ApplicationNATtoGWLBeRouteInv6: 418 | Type: AWS::EC2::Route 419 | DependsOn: ApplicationIGRTA 420 | Properties: 421 | RouteTableId: !Ref ApplicationIGRT 422 | DestinationIpv6CidrBlock: !Select [0, !GetAtt ApplicationPrivateSubnet.Ipv6CidrBlocks] 423 | VpcEndpointId: !Ref GWLBe 424 | 425 | # Finally, create a test application instance 426 | ApplicationInstance: 427 | Type: AWS::EC2::Instance 428 | Properties: 429 | ImageId: !Ref LinuxAmiId 430 | InstanceType: !Ref ApplicationInstanceType 431 | IamInstanceProfile: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 432 | BlockDeviceMappings: 433 | - DeviceName: /dev/xvda 434 | Ebs: 435 | VolumeSize: 8 436 | NetworkInterfaces: 437 | - DeviceIndex: 0 438 | SubnetId: !Ref ApplicationPrivateSubnet 439 | Ipv6AddressCount: 1 440 | Tags: 441 | - Key: Name 442 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application" ] ] 443 | 444 | # Do the fixes to our endpoint to allow IPv6 to work, until IPv6 support is added to CF. 445 | LambdaGWLBeIPv6FixRole: 446 | Type: AWS::IAM::Role 447 | Properties: 448 | RoleName: !Sub "${AWS::StackName}-LambdaGWLBeIPv6Fix" 449 | Description: Role for Cloudformation lambda script GWLBeIPv6Fix 450 | AssumeRolePolicyDocument: 451 | Version: '2012-10-17' 452 | Statement: 453 | - Effect: Allow 454 | Principal: { Service: [ lambda.amazonaws.com ] } 455 | Action: [ 'sts:AssumeRole' ] 456 | Policies: 457 | - PolicyName: AllowModifyVPCEndpoint 458 | PolicyDocument: 459 | Version: "2012-10-17" 460 | Statement: 461 | - Effect: "Allow" 462 | Action: 463 | - ec2:ModifyVpcEndpoint 464 | Resource: !Sub "arn:aws:ec2:*:${AWS::AccountId}:vpc-endpoint/*" 465 | Path: "/" 466 | ManagedPolicyArns: 467 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 468 | 469 | # Pass in the endpoint ID as GWLBeId. The version of botocore in lambda is too old to recognize the IpAddressType 470 | # parameter, thus the install of a newer version at the start. 471 | LambdaGWLBeIPv6Fix: 472 | Type: AWS::Lambda::Function 473 | Properties: 474 | FunctionName: !Sub "${AWS::StackName}-GWLBeIPv6Fix" 475 | Handler: index.handler 476 | Role: !GetAtt LambdaGWLBeIPv6FixRole.Arn 477 | Timeout: 30 478 | Code: 479 | ZipFile: | 480 | import sys 481 | from pip._internal import main 482 | 483 | main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check']) 484 | sys.path.insert(0,'/tmp/') 485 | 486 | import boto3 487 | import cfnresponse 488 | import threading 489 | import logging 490 | import botocore 491 | import time 492 | 493 | def timeout(event, context): 494 | logging.error('Execution is about to time out, sending failure response to CloudFormation') 495 | cfnresponse.send(event, context, cfnresponse.FAILED, {}, None, reason="Execution timeout") 496 | 497 | def handler(event, context): 498 | timer = threading.Timer((context.get_remaining_time_in_millis() / 1000.00) - 0.5, timeout, args=[event, context]) 499 | timer.start() 500 | try: 501 | if event['RequestType'] == 'Create' or event['RequestType'] == 'Update': 502 | ec2 = boto3.client('ec2') 503 | res = ec2.modify_vpc_endpoint(VpcEndpointId=event['ResourceProperties']['GWLBeId'], IpAddressType='dualstack') 504 | if 'Return' in res and res['Return'] == True: 505 | cfnresponse.send(event, context, cfnresponse.SUCCESS, res, event['ResourceProperties']['GWLBeId']) 506 | else: 507 | logging.error(f"modify_vpc_endpoint returned something other than True: {res}") 508 | cfnresponse.send(event, context, cfnresponse.FAILED, res) 509 | else: 510 | cfnresponse.send(event, context, cfnresponse.SUCCESS, {}) 511 | except Exception as e: 512 | logging.error('Exception: %s' % e, exc_info=True) 513 | logging.error(f"Data from request:\nevent={event}\ncontext={context}") 514 | cfnresponse.send(event, context, cfnresponse.FAILED, {"e": f'{e}'}) 515 | raise e 516 | Runtime: python3.9 517 | 518 | GWLBeIPv6: 519 | Type: Custom::GWLBIPV6 520 | DependsOn: 521 | - GWLBESIPv6 522 | Properties: 523 | ServiceToken: !GetAtt LambdaGWLBeIPv6Fix.Arn 524 | GWLBeId: !Ref GWLBe 525 | 526 | LambdaGWLBESIPv6FixRole: 527 | Type: AWS::IAM::Role 528 | Properties: 529 | RoleName: !Sub "${AWS::StackName}-LambdaGWLBESIPv6Fix" 530 | Description: Role for Cloudformation lambda script GWLBESIPv6Fix 531 | AssumeRolePolicyDocument: 532 | Version: '2012-10-17' 533 | Statement: 534 | - Effect: Allow 535 | Principal: { Service: [ lambda.amazonaws.com ] } 536 | Action: [ 'sts:AssumeRole' ] 537 | Policies: 538 | - PolicyName: AllowModifyVPCEndpoint 539 | PolicyDocument: 540 | Version: "2012-10-17" 541 | Statement: 542 | - Effect: "Allow" 543 | Action: 544 | - ec2:ModifyVpcEndpointServiceConfiguration 545 | Resource: !Sub "arn:aws:ec2:*:${AWS::AccountId}:vpc-endpoint-service/*" 546 | Path: "/" 547 | ManagedPolicyArns: 548 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 549 | 550 | LambdaGWLBESIPv6Fix: 551 | Type: AWS::Lambda::Function 552 | Properties: 553 | FunctionName: !Sub "${AWS::StackName}-GWLBESIPv6Fix" 554 | Handler: index.handler 555 | Role: !GetAtt LambdaGWLBESIPv6FixRole.Arn 556 | Timeout: 30 557 | Code: 558 | ZipFile: | 559 | import sys 560 | from pip._internal import main 561 | 562 | main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check']) 563 | sys.path.insert(0,'/tmp/') 564 | 565 | import boto3 566 | import cfnresponse 567 | import threading 568 | import logging 569 | import botocore 570 | import time 571 | 572 | def timeout(event, context): 573 | logging.error('Execution is about to time out, sending failure response to CloudFormation') 574 | cfnresponse.send(event, context, cfnresponse.FAILED, {}, None, reason="Execution timeout") 575 | 576 | def handler(event, context): 577 | timer = threading.Timer((context.get_remaining_time_in_millis() / 1000.00) - 0.5, timeout, args=[event, context]) 578 | timer.start() 579 | try: 580 | if event['RequestType'] == 'Create' or event['RequestType'] == 'Update': 581 | ec2 = boto3.client('ec2') 582 | res = ec2.modify_vpc_endpoint_service_configuration(ServiceId=event['ResourceProperties']['GWLBES'], AddSupportedIpAddressTypes=['ipv6']) 583 | if 'Return' in res and res['Return'] == True: 584 | cfnresponse.send(event, context, cfnresponse.SUCCESS, res, event['ResourceProperties']['GWLBES']) 585 | else: 586 | logging.error(f"modify_vpc_endpoint returned something other than True: {res}") 587 | cfnresponse.send(event, context, cfnresponse.FAILED, res) 588 | else: 589 | cfnresponse.send(event, context, cfnresponse.SUCCESS, {}) 590 | except Exception as e: 591 | logging.error('Exception: %s' % e, exc_info=True) 592 | logging.error(f"Data from request:\nevent={event}\ncontext={context}") 593 | cfnresponse.send(event, context, cfnresponse.FAILED, {"e": f'{e}'}) 594 | raise e 595 | Runtime: python3.9 596 | 597 | GWLBESIPv6: 598 | Type: Custom::GWLBIPV6 599 | Properties: 600 | ServiceToken: !GetAtt LambdaGWLBESIPv6Fix.Arn 601 | GWLBES: !Ref GWLBEndpointService 602 | 603 | Outputs: 604 | GWLBES: 605 | Description: ID of the GWLB endpoint service 606 | Value: !Sub "com.amazonaws.vpce.${AWS::Region}.${GWLBEndpointService}" 607 | Export: 608 | Name: !Sub "${AWS::StackName}-EndpointService" 609 | -------------------------------------------------------------------------------- /example-scripts/example-topology-two-arm.template: -------------------------------------------------------------------------------- 1 | --- 2 | AWSTemplateFormatVersion: "2010-09-09" 3 | Description: Set up a demonstration topology for using gwlbtun in two-arm mode 4 | Parameters: 5 | LinuxAmiId: 6 | Type: AWS::SSM::Parameter::Value 7 | Description: AMI ID to use for GWLB appliances 8 | Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2' 9 | GWLBInstanceType: 10 | Type: String 11 | Description: Instance type code to use for the GWLB appliances. The default is fine for light testing, for heavier workloads you should consider a c5n instance. 12 | Default: t2.micro 13 | 14 | Resources: 15 | EC2Role: 16 | Type: AWS::IAM::Role 17 | Properties: 18 | RoleName: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 19 | Description: Role for all EC2 instances to run under to allow SSM and S3 20 | AssumeRolePolicyDocument: 21 | Version: "2012-10-17" 22 | Statement: 23 | - Effect: Allow 24 | Principal: 25 | Service: 26 | - ec2.amazonaws.com 27 | Action: 28 | - 'sts:AssumeRole' 29 | ManagedPolicyArns: 30 | - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore 31 | - arn:aws:iam::aws:policy/AmazonSSMPatchAssociation 32 | - arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess 33 | - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess 34 | EC2RoleInstanceProfile: 35 | Type: AWS::IAM::InstanceProfile 36 | Properties: 37 | InstanceProfileName: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 38 | Path: / 39 | Roles: 40 | - !Ref EC2Role 41 | 42 | GWLBTunVPC: 43 | Type: AWS::EC2::VPC 44 | Properties: 45 | CidrBlock: 10.10.0.0/16 46 | EnableDnsHostnames: true 47 | EnableDnsSupport: true 48 | Tags: 49 | - Key: Name 50 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun" ] ] 51 | 52 | GWLBTunPublicSubnet: 53 | Type: AWS::EC2::Subnet 54 | Properties: 55 | AvailabilityZone: !Select [0, !GetAZs '' ] 56 | CidrBlock: 10.10.2.0/24 57 | VpcId: !Ref GWLBTunVPC 58 | Tags: 59 | - Key: Name 60 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun public" ] ] 61 | 62 | GWLBTunPublicRT: 63 | Type: AWS::EC2::RouteTable 64 | Properties: 65 | VpcId: !Ref GWLBTunVPC 66 | Tags: 67 | - Key: Name 68 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun public" ] ] 69 | 70 | GWLBTunPublicRTA: 71 | Type: AWS::EC2::SubnetRouteTableAssociation 72 | Properties: 73 | SubnetId: !Ref GWLBTunPublicSubnet 74 | RouteTableId: !Ref GWLBTunPublicRT 75 | 76 | # Set up the GWLB 77 | GWLB: 78 | Type: AWS::ElasticLoadBalancingV2::LoadBalancer 79 | Properties: 80 | Name: !Join [ "-", [ !Ref AWS::StackName, "GWLB" ] ] 81 | Type: gateway 82 | Subnets: 83 | - !Ref GWLBTunPublicSubnet 84 | 85 | GWLBTargetGroup: 86 | Type: AWS::ElasticLoadBalancingV2::TargetGroup 87 | Properties: 88 | Name: !Join [ "-", [ !Ref AWS::StackName, "Instances" ] ] 89 | VpcId: !Ref GWLBTunVPC 90 | Protocol: GENEVE 91 | Port: 6081 92 | HealthCheckProtocol: HTTP 93 | HealthCheckPort: 80 94 | HealthCheckPath: / 95 | HealthCheckEnabled: true 96 | TargetType: ip # TargetType IP allows specifying the specific NIC on the GWLBTun instance to target 97 | Targets: 98 | - Id: !GetAtt GWLBTunInstanceTwoArm.PrivateIp 99 | 100 | GWLBListener: 101 | Type: AWS::ElasticLoadBalancingV2::Listener 102 | Properties: 103 | LoadBalancerArn: !Ref GWLB 104 | DefaultActions: 105 | - Type: forward 106 | TargetGroupArn: !Ref GWLBTargetGroup 107 | 108 | GWLBEndpointService: 109 | Type: AWS::EC2::VPCEndpointService 110 | Properties: 111 | GatewayLoadBalancerArns: 112 | - !Ref GWLB 113 | AcceptanceRequired: false 114 | 115 | GWLBe: 116 | Type: AWS::EC2::VPCEndpoint 117 | DependsOn: GWLBEndpointService 118 | Properties: 119 | VpcId: !Ref ApplicationVPC 120 | ServiceName: !Sub "com.amazonaws.vpce.${AWS::Region}.${GWLBEndpointService}" 121 | VpcEndpointType: GatewayLoadBalancer 122 | SubnetIds: 123 | - !Ref ApplicationPrivateSubnet 124 | 125 | # Build a GWLBTun instance 126 | GWLBTunInstanceTwoArm: 127 | Type: AWS::EC2::Instance 128 | Properties: 129 | ImageId: !Ref LinuxAmiId 130 | InstanceType: !Ref GWLBInstanceType 131 | IamInstanceProfile: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 132 | SourceDestCheck: false 133 | BlockDeviceMappings: 134 | - DeviceName: /dev/xvda 135 | Ebs: 136 | VolumeSize: 8 137 | NetworkInterfaces: 138 | - DeviceIndex: 0 139 | AssociatePublicIpAddress: True 140 | SubnetId: !Ref GWLBTunPublicSubnet 141 | GroupSet: 142 | - Ref: GWLBTunSG 143 | UserData: 144 | Fn::Base64: | 145 | #!/bin/bash -ex 146 | yum -y groupinstall "Development Tools" 147 | yum -y install cmake3 148 | cd /root 149 | git clone https://github.com/aws-samples/aws-gateway-load-balancer-tunnel-handler.git 150 | cd aws-gateway-load-balancer-tunnel-handler 151 | cmake3 . 152 | make 153 | 154 | echo "[Unit]" > /usr/lib/systemd/system/gwlbtun.service 155 | echo "Description=AWS GWLB Tunnel Handler" >> /usr/lib/systemd/system/gwlbtun.service 156 | echo "" >> /usr/lib/systemd/system/gwlbtun.service 157 | echo "[Service]" >> /usr/lib/systemd/system/gwlbtun.service 158 | echo "ExecStart=/root/aws-gateway-load-balancer-tunnel-handler/gwlbtun -c /root/aws-gateway-load-balancer-tunnel-handler/example-scripts/create-nat.sh -p 80" >> /usr/lib/systemd/system/gwlbtun.service 159 | echo "Restart=always" >> /usr/lib/systemd/system/gwlbtun.service 160 | echo "RestartSec=5s" >> /usr/lib/systemd/system/gwlbtun.service 161 | 162 | systemctl daemon-reload 163 | systemctl enable --now --no-block gwlbtun.service 164 | systemctl start gwlbtun.service 165 | echo 166 | Tags: 167 | - Key: Name 168 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun" ] ] 169 | 170 | GWLBTunSG: 171 | Type: AWS::EC2::SecurityGroup 172 | Properties: 173 | VpcId: !Ref GWLBTunVPC 174 | GroupName: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun Security" ] ] 175 | GroupDescription: GWLBTun Security 176 | SecurityGroupIngress: 177 | - CidrIp: !GetAtt GWLBTunVPC.CidrBlock 178 | IpProtocol: -1 179 | 180 | # Make an Internet Gateway in our GWLB VPC and route to it from the public subnet. 181 | GWLBTunIG: 182 | Type: AWS::EC2::InternetGateway 183 | Properties: 184 | Tags: 185 | - Key: Name 186 | Value: !Join [ "-", [ !Ref AWS::StackName, "GWLBTun" ] ] 187 | 188 | GWLBTunIGA: 189 | Type: AWS::EC2::VPCGatewayAttachment 190 | Properties: 191 | InternetGatewayId: !Ref GWLBTunIG 192 | VpcId: !Ref GWLBTunVPC 193 | 194 | GWLBTunIGRoute: 195 | Type: AWS::EC2::Route 196 | DependsOn: GWLBTunIGA 197 | Properties: 198 | RouteTableId: !Ref GWLBTunPublicRT 199 | DestinationCidrBlock: 0.0.0.0/0 200 | GatewayId: !Ref GWLBTunIG 201 | 202 | # Set up the Application VPC 203 | ApplicationVPC: 204 | Type: AWS::EC2::VPC 205 | Properties: 206 | CidrBlock: 10.20.0.0/16 207 | EnableDnsHostnames: true 208 | EnableDnsSupport: true 209 | Tags: 210 | - Key: Name 211 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application" ] ] 212 | 213 | ApplicationPrivateSubnet: 214 | Type: AWS::EC2::Subnet 215 | Properties: 216 | AvailabilityZone: !Select [0, !GetAZs '' ] 217 | CidrBlock: 10.20.0.0/24 218 | VpcId: !Ref ApplicationVPC 219 | Tags: 220 | - Key: Name 221 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application private" ] ] 222 | 223 | ApplicationPrivateRT: 224 | Type: AWS::EC2::RouteTable 225 | Properties: 226 | VpcId: !Ref ApplicationVPC 227 | Tags: 228 | - Key: Name 229 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application private" ] ] 230 | 231 | ApplicationPrivateRTA: 232 | Type: AWS::EC2::SubnetRouteTableAssociation 233 | Properties: 234 | SubnetId: !Ref ApplicationPrivateSubnet 235 | RouteTableId: !Ref ApplicationPrivateRT 236 | 237 | # Route traffic from our applications to the GWLB endpoint 238 | ApplicationPrivateGWLBRoute: 239 | Type: AWS::EC2::Route 240 | Properties: 241 | RouteTableId: !Ref ApplicationPrivateRT 242 | VpcEndpointId: !Ref GWLBe 243 | DestinationCidrBlock: 0.0.0.0/0 244 | 245 | # Create a test application instance 246 | ApplicationInstance: 247 | Type: AWS::EC2::Instance 248 | Properties: 249 | ImageId: !Ref LinuxAmiId 250 | InstanceType: t2.micro 251 | IamInstanceProfile: !Join [ "-", [ !Ref AWS::StackName, "EC2Role" ] ] 252 | BlockDeviceMappings: 253 | - DeviceName: /dev/xvda 254 | Ebs: 255 | VolumeSize: 8 256 | NetworkInterfaces: 257 | - DeviceIndex: 0 258 | SubnetId: !Ref ApplicationPrivateSubnet 259 | Tags: 260 | - Key: Name 261 | Value: !Join [ "-", [ !Ref AWS::StackName, "Application" ] ] 262 | 263 | # Do the fixes to our endpoint to allow IPv6 to work, until IPv6 support is added to CF. 264 | LambdaGWLBeIPv6FixRole: 265 | Type: AWS::IAM::Role 266 | Properties: 267 | RoleName: !Sub "${AWS::StackName}-LambdaGWLBeIPv6Fix" 268 | Description: Role for Cloudformation lambda script GWLBeIPv6Fix 269 | AssumeRolePolicyDocument: 270 | Version: '2012-10-17' 271 | Statement: 272 | - Effect: Allow 273 | Principal: { Service: [ lambda.amazonaws.com ] } 274 | Action: [ 'sts:AssumeRole' ] 275 | Policies: 276 | - PolicyName: AllowModifyVPCEndpoint 277 | PolicyDocument: 278 | Version: "2012-10-17" 279 | Statement: 280 | - Effect: "Allow" 281 | Action: 282 | - ec2:ModifyVpcEndpoint 283 | Resource: !Sub "arn:aws:ec2:*:${AWS::AccountId}:vpc-endpoint/*" 284 | Path: "/" 285 | ManagedPolicyArns: 286 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 287 | 288 | # Pass in the endpoint ID as GWLBeId. The version of botocore in lambda is too old to recognize the IpAddressType 289 | # parameter, thus the install of a newer version at the start. 290 | LambdaGWLBeIPv6Fix: 291 | Type: AWS::Lambda::Function 292 | Properties: 293 | FunctionName: !Sub "${AWS::StackName}-GWLBeIPv6Fix" 294 | Handler: index.handler 295 | Role: !GetAtt LambdaGWLBeIPv6FixRole.Arn 296 | Timeout: 30 297 | Code: 298 | ZipFile: | 299 | import sys 300 | from pip._internal import main 301 | 302 | main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check']) 303 | sys.path.insert(0,'/tmp/') 304 | 305 | import boto3 306 | import cfnresponse 307 | import threading 308 | import logging 309 | import botocore 310 | import time 311 | 312 | def timeout(event, context): 313 | logging.error('Execution is about to time out, sending failure response to CloudFormation') 314 | cfnresponse.send(event, context, cfnresponse.FAILED, {}, None, reason="Execution timeout") 315 | 316 | def handler(event, context): 317 | timer = threading.Timer((context.get_remaining_time_in_millis() / 1000.00) - 0.5, timeout, args=[event, context]) 318 | timer.start() 319 | try: 320 | if event['RequestType'] == 'Create' or event['RequestType'] == 'Update': 321 | ec2 = boto3.client('ec2') 322 | res = ec2.modify_vpc_endpoint(VpcEndpointId=event['ResourceProperties']['GWLBeId'], IpAddressType='dualstack') 323 | if 'Return' in res and res['Return'] == True: 324 | cfnresponse.send(event, context, cfnresponse.SUCCESS, res, event['ResourceProperties']['GWLBeId']) 325 | else: 326 | logging.error(f"modify_vpc_endpoint returned something other than True: {res}") 327 | cfnresponse.send(event, context, cfnresponse.FAILED, res) 328 | else: 329 | cfnresponse.send(event, context, cfnresponse.SUCCESS, {}) 330 | except Exception as e: 331 | logging.error('Exception: %s' % e, exc_info=True) 332 | logging.error(f"Data from request:\nevent={event}\ncontext={context}") 333 | cfnresponse.send(event, context, cfnresponse.FAILED, {"e": f'{e}'}) 334 | raise e 335 | Runtime: python3.9 336 | 337 | GWLBeIPv6: 338 | Type: Custom::GWLBIPV6 339 | DependsOn: 340 | - GWLBESIPv6 341 | Properties: 342 | ServiceToken: !GetAtt LambdaGWLBeIPv6Fix.Arn 343 | GWLBeId: !Ref GWLBe 344 | 345 | LambdaGWLBESIPv6FixRole: 346 | Type: AWS::IAM::Role 347 | Properties: 348 | RoleName: !Sub "${AWS::StackName}-LambdaGWLBESIPv6Fix" 349 | Description: Role for Cloudformation lambda script GWLBESIPv6Fix 350 | AssumeRolePolicyDocument: 351 | Version: '2012-10-17' 352 | Statement: 353 | - Effect: Allow 354 | Principal: { Service: [ lambda.amazonaws.com ] } 355 | Action: [ 'sts:AssumeRole' ] 356 | Policies: 357 | - PolicyName: AllowModifyVPCEndpoint 358 | PolicyDocument: 359 | Version: "2012-10-17" 360 | Statement: 361 | - Effect: "Allow" 362 | Action: 363 | - ec2:ModifyVpcEndpointServiceConfiguration 364 | Resource: !Sub "arn:aws:ec2:*:${AWS::AccountId}:vpc-endpoint-service/*" 365 | Path: "/" 366 | ManagedPolicyArns: 367 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 368 | 369 | LambdaGWLBESIPv6Fix: 370 | Type: AWS::Lambda::Function 371 | Properties: 372 | FunctionName: !Sub "${AWS::StackName}-GWLBESIPv6Fix" 373 | Handler: index.handler 374 | Role: !GetAtt LambdaGWLBESIPv6FixRole.Arn 375 | Timeout: 30 376 | Code: 377 | ZipFile: | 378 | import sys 379 | from pip._internal import main 380 | 381 | main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check']) 382 | sys.path.insert(0,'/tmp/') 383 | 384 | import boto3 385 | import cfnresponse 386 | import threading 387 | import logging 388 | import botocore 389 | import time 390 | 391 | def timeout(event, context): 392 | logging.error('Execution is about to time out, sending failure response to CloudFormation') 393 | cfnresponse.send(event, context, cfnresponse.FAILED, {}, None, reason="Execution timeout") 394 | 395 | def handler(event, context): 396 | timer = threading.Timer((context.get_remaining_time_in_millis() / 1000.00) - 0.5, timeout, args=[event, context]) 397 | timer.start() 398 | try: 399 | if event['RequestType'] == 'Create' or event['RequestType'] == 'Update': 400 | ec2 = boto3.client('ec2') 401 | res = ec2.modify_vpc_endpoint_service_configuration(ServiceId=event['ResourceProperties']['GWLBES'], AddSupportedIpAddressTypes=['ipv6']) 402 | if 'Return' in res and res['Return'] == True: 403 | cfnresponse.send(event, context, cfnresponse.SUCCESS, res, event['ResourceProperties']['GWLBES']) 404 | else: 405 | logging.error(f"modify_vpc_endpoint returned something other than True: {res}") 406 | cfnresponse.send(event, context, cfnresponse.FAILED, res) 407 | else: 408 | cfnresponse.send(event, context, cfnresponse.SUCCESS, {}) 409 | except Exception as e: 410 | logging.error('Exception: %s' % e, exc_info=True) 411 | logging.error(f"Data from request:\nevent={event}\ncontext={context}") 412 | cfnresponse.send(event, context, cfnresponse.FAILED, {"e": f'{e}'}) 413 | raise e 414 | Runtime: python3.9 415 | 416 | GWLBESIPv6: 417 | Type: Custom::GWLBIPV6 418 | Properties: 419 | ServiceToken: !GetAtt LambdaGWLBESIPv6Fix.Arn 420 | GWLBES: !Ref GWLBEndpointService 421 | 422 | Outputs: 423 | GWLBES: 424 | Description: ID of the GWLB endpoint service 425 | Value: !Sub "com.amazonaws.vpce.${AWS::Region}.${GWLBEndpointService}" 426 | Export: 427 | Name: !Sub "${AWS::StackName}-EndpointService" 428 | -------------------------------------------------------------------------------- /main.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | // GWLB Tunnel Handling user-space program. See the README.md for details on usage. 5 | 6 | #include 7 | #include 8 | #include 9 | #include "GeneveHandler.h" 10 | #include 11 | #include 12 | #include 13 | #include "utils.h" 14 | #include 15 | #include 16 | #include "Logger.h" 17 | 18 | using namespace std::string_literals; 19 | 20 | std::string newCmd = ""; 21 | std::string delCmd = ""; 22 | volatile sig_atomic_t keepRunning = 1; 23 | 24 | /** 25 | * Callback function for when a new GWLB endpoint has been detected by GeneveHandler. Prints a message and calls the create script. 26 | * 27 | * @param ingressInt New ingress interface. 28 | * @param egressInt New egress interface. 29 | * @param eniId ENI ID of the new endpoint. 30 | */ 31 | void newInterfaceCallback(std::string ingressInt, const std::string egressInt, eniid_t eniId) 32 | { 33 | LOG(LS_CORE, LL_IMPORTANT, "New interface "s + ingressInt + " and "s + egressInt + " for ENI ID "s + MakeENIStr(eniId) + " created."s); 34 | if(newCmd.length() > 0) 35 | { 36 | std::stringstream ss; 37 | ss << newCmd << " CREATE " << ingressInt << " " << egressInt << " " << MakeENIStr(eniId); 38 | system(ss.str().c_str()); 39 | } 40 | } 41 | 42 | /** 43 | * Callback function for when GeneveHandler deems an endpoint has disappeared. Prints a message and calls the delete script. 44 | * @param ingressInt Old ingress interface. 45 | * @param egressInt Old egress interface. 46 | * @param eniId Old ENI ID. 47 | */ 48 | void deleteInterfaceCallback(std::string ingressInt, const std::string egressInt, eniid_t eniId) 49 | { 50 | LOG(LS_CORE, LL_IMPORTANT, "Removing interface "s + ingressInt + " and "s + egressInt + " for ENI ID "s + MakeENIStr(eniId) + "."s); 51 | if(delCmd.length() > 0) 52 | { 53 | std::stringstream ss; 54 | ss << delCmd << " DESTROY " << ingressInt << " " << egressInt << " " << MakeENIStr(eniId); 55 | system(ss.str().c_str()); 56 | } 57 | } 58 | 59 | /** 60 | * Performs a health check of the GeneveHandler and sends an HTTP/1.1 string conveying that status. The HTTP header 61 | * has a 200 code if everything is well, a 503 if not. 62 | * 63 | * @param details true to return packet counters, false to just return the status code. 64 | * @param gh The GeneveHandler to return the status for. 65 | * @param s The socket to send the health check to 66 | * @param json Whether to output as human text (false) or json (true) 67 | */ 68 | void performHealthCheck(bool details, GeneveHandler *gh, int s, bool json) 69 | { 70 | GeneveHandlerHealthCheck ghhc = gh->check(); 71 | 72 | std::string response = "HTTP/1.1 "s + (gh->healthy ? "200 OK"s: "503 Failed"s) + "\n"s + 73 | "Cache-Control: max-age=0, no-cache\n"; 74 | 75 | if(json) 76 | response += "Content-Type: application/json\n\n"; 77 | else 78 | response += "Content-Type: text/html\n\n\n\nHealth check"s; 79 | 80 | if(details) 81 | { 82 | if (json) 83 | response += ghhc.output_json().dump(); 84 | else 85 | response += ghhc.output_str() + "\n"; 86 | } 87 | 88 | send(s, response.c_str(), response.length(), 0); 89 | } 90 | 91 | /** 92 | * Returns the number of cores available to this process. 93 | * @return Integer core count. 94 | */ 95 | int numCores() 96 | { 97 | cpu_set_t cpuset; 98 | sched_getaffinity(0, sizeof(cpuset), &cpuset); 99 | return CPU_COUNT(&cpuset); 100 | } 101 | 102 | /** 103 | * Prints command help. 104 | * 105 | * @param progname 106 | */ 107 | void printHelp(char *progname) 108 | { 109 | fprintf(stderr, 110 | "AWS Gateway Load Balancer Tunnel Handler v%d.%d\n" 111 | "Usage: %s [options]\n" 112 | "Example: %s\n" 113 | "\n" 114 | " -h Print this help\n" 115 | " -c FILE Command to execute when a new tunnel has been built. See below for arguments passed.\n" 116 | " -r FILE Command to execute when a tunnel times out and is about to be destroyed. See below for arguments passed.\n" 117 | " -t TIME Minimum time in seconds between last packet seen and to consider the tunnel timed out. Set to 0 (the default) to never time out tunnels.\n" 118 | " Note the actual time between last packet and the destroy call may be longer than this time.\n" 119 | #ifndef NO_RETURN_TRAFFIC 120 | " -i TIME Idle timeout to use for the flow caches. Set this to match what GWLB is configured for. Defaults to 350 seconds.\n" 121 | #endif 122 | " -p PORT Listen to TCP port PORT and provide a health status report on it.\n" 123 | " -j For health check detailed statistics, output as JSON instead of text.\n" 124 | " -s Only return simple health check status (only the HTTP response code), instead of detailed statistics.\n" 125 | " -d Enable debugging output. Short version of --logging all=debug.\n" 126 | "\n" 127 | "Threading options:\n" 128 | " --udpthreads NUM Generate NUM threads for the UDP receiver.\n" 129 | " --udpaffinity AFFIN Generate threads for the UDP receiver, pinned to the cores listed. Takes precedence over udptreads.\n" 130 | " --tunthreads NUM Generate NUM threads for each tunnel processor.\n" 131 | " --tunaffinity AFFIN Generate threads for each tunnel processor, pinned to the cores listed. Takes precedence over tunthreads.\n" 132 | "\n" 133 | "AFFIN arguments take a comma separated list of cores or range of cores, e.g. 1-2,4,7-8.\n" 134 | "It is recommended to have the same number of UDP threads as tunnel processor threads, in one-arm operation.\n" 135 | "If unspecified, --udpthreads %d and --tunthreads %d will be assumed as a default, based on the number of cores present.\n" 136 | "\n" 137 | "Logging options:\n" 138 | " --logging CONFIG Set the logging configuration, as described below.\n" 139 | #ifdef NO_RETURN_TRAFFIC 140 | "This version of GWLBTun has been compiled with NO_RETURN_TRAFFIC defined.\n" 141 | #endif 142 | "---------------------------------------------------------------------------------------------------------\n" 143 | "Hook scripts arguments:\n" 144 | "These arguments are provided when gwlbtun calls the hook scripts (the -c and/or -r command options).\n" 145 | "On gwlbtun startup, it will automatically create gwi- and gwo- interfaces upon seeing the first packet from a specific GWLBE, and the hook scripts are invoked when interfaces are created or destroyed. You should at least disable rpf_filter for the gwi- tunnel interface with the hook scripts.\n" 146 | "The hook scripts will be called with the following arguments:\n" 147 | "1: The string 'CREATE' or 'DESTROY', depending on which operation is occurring.\n" 148 | "2: The interface name of the ingress interface (gwi-).\n" 149 | "3: The interface name of the egress interface (gwo-). Packets can be sent out via in the ingress\n" 150 | " as well, but having two different interfaces makes routing and iptables easier.\n" 151 | "4: The GWLBE ENI ID in base 16 (e.g. '2b8ee1d4db0c51c4') associated with this tunnel.\n" 152 | "\n" 153 | "The in the interface name is replaced with the base 60 encoded ENI ID (to fit inside the 15 character\n" 154 | "device name limit).\n" 155 | "---------------------------------------------------------------------------------------------------------\n" 156 | , VERSION_MAJOR, VERSION_MINOR, progname, progname, numCores(), numCores()); 157 | fprintf(stderr, logger->help().c_str()); 158 | } 159 | 160 | /** 161 | * Handler for when Ctrl-C is received. Sets a global flag so everything can start shutting down. 162 | * 163 | * @param sig 164 | */ 165 | void shutdownHandler(int sig) 166 | { 167 | keepRunning = 0; 168 | } 169 | 170 | class Logger *logger; 171 | 172 | int main(int argc, char *argv[]) 173 | { 174 | int c; 175 | int healthCheck = 0, healthSocket; 176 | int tunnelTimeout = 0, cacheTimeout = 350; 177 | int udpthreads = numCores(), tunthreads = numCores(); 178 | std::string udpaffinity, tunaffinity, logoptions; 179 | bool detailedHealth = true, printHelpFlag = false, jsonHealth = false; 180 | 181 | static struct option long_options[] = { 182 | {"cmdnew", required_argument, NULL, 'c'}, 183 | {"cmddel", required_argument, NULL, 'r'}, 184 | {"timeout", required_argument, NULL, 't'}, 185 | {"port", required_argument, NULL, 'p'}, 186 | {"debug", no_argument, NULL, 'd'}, 187 | {"help", no_argument, NULL, 'h'}, 188 | {"help", no_argument, NULL, '?'}, 189 | {"udpthreads", required_argument, NULL, 0}, // optind 7 190 | {"udpaffinity", required_argument, NULL, 0}, // optind 8 191 | {"tunthreads", required_argument, NULL, 0}, // optind 9 192 | {"tunaffinity", required_argument, NULL, 0}, // optind 10 193 | {"logging", required_argument, NULL, 0}, // optind 11 194 | {"json", no_argument, NULL, 'j'}, // optind 12 195 | {"idle", required_argument, NULL, 'i'}, // optind 13 196 | {0, 0, 0, 0} 197 | }; 198 | 199 | // Argument parsing 200 | int optind; 201 | while ((c = getopt_long (argc, argv, "h?djxc:r:t:p:si:", long_options, &optind)) != -1) 202 | { 203 | switch(c) 204 | { 205 | case 0: 206 | // Long option 207 | switch(optind) { 208 | case 7: 209 | udpthreads = atoi(optarg); 210 | break; 211 | case 8: 212 | udpaffinity = std::string(optarg); 213 | break; 214 | case 9: 215 | tunthreads = atoi(optarg); 216 | break; 217 | case 10: 218 | tunaffinity = std::string(optarg); 219 | break; 220 | case 11: 221 | logoptions = std::string(optarg); 222 | break; 223 | } 224 | break; 225 | case 'c': 226 | newCmd = std::string(optarg); 227 | break; 228 | case 'r': 229 | delCmd = std::string(optarg); 230 | break; 231 | case 't': 232 | tunnelTimeout = atoi(optarg); 233 | break; 234 | case 'p': 235 | healthCheck = atoi(optarg); 236 | break; 237 | case 's': 238 | detailedHealth = false; 239 | break; 240 | case 'd': 241 | logoptions = "all=debug"; 242 | break; 243 | case 'j': 244 | jsonHealth = true; 245 | break; 246 | case 'i': 247 | cacheTimeout = atoi(optarg); 248 | break; 249 | case '?': 250 | case 'h': 251 | default: 252 | printHelpFlag = true; 253 | break; 254 | } 255 | } 256 | 257 | if(printHelpFlag) 258 | { 259 | printHelp(argv[0]); 260 | exit(EXIT_FAILURE); 261 | } 262 | 263 | logger = new Logger(logoptions); 264 | 265 | // Set up for health check reporting, if requested. We listen on both IPv4 and IPv6 for completeness, although 266 | // GWLB only supports IPv4. 267 | if(healthCheck > 0) 268 | { 269 | if((healthSocket = socket(AF_INET6, SOCK_STREAM, 0)) == 0) 270 | { 271 | LOG(LS_CORE, LL_CRITICAL, "Creating health check socket failed: "s + std::strerror(errno)); 272 | exit(EXIT_FAILURE); 273 | } 274 | 275 | struct sockaddr_in6 addr; 276 | bzero(&addr, sizeof(addr)); 277 | 278 | addr.sin6_family = AF_INET6; 279 | addr.sin6_port = htons(healthCheck); 280 | addr.sin6_addr = in6addr_any; 281 | if(bind(healthSocket, (struct sockaddr *)&addr, sizeof(addr)) < 0) 282 | { 283 | LOG(LS_CORE, LL_CRITICAL, "Unable to listen to health status port: "s + std::strerror(errno)); 284 | exit(EXIT_FAILURE); 285 | } 286 | listen(healthSocket, 3); 287 | } 288 | 289 | signal(SIGINT, shutdownHandler); 290 | 291 | ThreadConfig udp, tun; 292 | ParseThreadConfiguration(udpthreads, udpaffinity, &udp); 293 | ParseThreadConfiguration(tunthreads, tunaffinity, &tun); 294 | 295 | auto gh = new GeneveHandler(&newInterfaceCallback, &deleteInterfaceCallback, tunnelTimeout, cacheTimeout, udp, tun); 296 | struct timespec timeout; 297 | timeout.tv_sec = 1; timeout.tv_nsec = 0; 298 | fd_set fds; 299 | int ready; 300 | int ticksSinceCheck = 60; 301 | LOG(LS_CORE, LL_IMPORTANT, "AWS Gateway Load Balancer Tunnel Handler v%d.%d started.", VERSION_MAJOR, VERSION_MINOR); 302 | while(keepRunning) 303 | { 304 | FD_ZERO(&fds); 305 | if(healthCheck > 0) 306 | { 307 | FD_SET(healthSocket, &fds); 308 | } 309 | 310 | ready = pselect(healthSocket + 1, &fds, nullptr, nullptr, &timeout, nullptr); 311 | if(ready > 0 && healthCheck > 0 && FD_ISSET(healthSocket, &fds)) 312 | { 313 | // Process a health check client 314 | int hsClient; 315 | struct sockaddr_in6 from; 316 | socklen_t fromlen = sizeof(from); 317 | hsClient = accept(healthSocket, (struct sockaddr *)&from, &fromlen); 318 | LOG(LS_HEALTHCHECK, LL_DEBUG, "Processing a health check client for " + sockaddrToName((struct sockaddr *)&from)); 319 | performHealthCheck(detailedHealth, gh, hsClient, jsonHealth); 320 | close(hsClient); 321 | ticksSinceCheck = 60; 322 | } 323 | 324 | ticksSinceCheck --; 325 | if(ticksSinceCheck < 0) 326 | { 327 | GeneveHandlerHealthCheck ghhc = gh->check(); 328 | LOG(LS_HEALTHCHECK, LL_DEBUG, ghhc.output_str()); 329 | ticksSinceCheck = 60; 330 | } 331 | } 332 | 333 | // The loop was interrupted (most likely by Ctrl-C or likewise). Clean up a few things. 334 | LOG(LS_CORE, LL_IMPORTANT, "Shutting down."); 335 | delete(gh); 336 | if(healthCheck > 0) close(healthSocket); 337 | LOG(LS_CORE, LL_IMPORTANT, "Shutdown complete."); 338 | 339 | return 0; 340 | } 341 | -------------------------------------------------------------------------------- /utils.cpp: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | /** 5 | * Miscellaneous handy utilities. 6 | */ 7 | 8 | #include "utils.h" 9 | 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include "Logger.h" 19 | 20 | using namespace std::string_literals; 21 | 22 | /** 23 | * Perform a printf-like function, but on a std::string and returning a std::string 24 | * 25 | * @param fmt_str printf-like formatting string 26 | * @param ... Parameters for the fmt_str 27 | * @return A std::string of the formatted output 28 | */ 29 | std::string stringFormat(const std::string& fmt_str, ...) { 30 | va_list ap; 31 | 32 | va_start(ap, &fmt_str); 33 | std::string ret = stringFormat(fmt_str, ap); 34 | va_end(ap); 35 | 36 | return ret; 37 | } 38 | 39 | std::string stringFormat(const std::string& fmt_str, va_list ap) 40 | { 41 | int final_n, n = ((int)fmt_str.size()) * 2; /* Reserve two times as much as the length of the fmt_str */ 42 | std::unique_ptr formatted; 43 | while(true) { 44 | formatted.reset(new char[n]); /* Wrap the plain char array into the unique_ptr */ 45 | strcpy(&formatted[0], fmt_str.c_str()); 46 | final_n = vsnprintf(&formatted[0], n, fmt_str.c_str(), ap); 47 | if (final_n < 0 || final_n >= n) 48 | n += abs(final_n - n + 1); 49 | else 50 | break; 51 | } 52 | return std::string(formatted.get()); 53 | } 54 | 55 | /** 56 | * Send a UDP packet, with control over the source and destination. We have to use a RAW socket for specifying the 57 | * source port and it changing constantly. 58 | * 59 | * @param sock Existing socket(AF_INET,SOCK_RAW, IPPROTO_RAW) to use for sending 60 | * @param from_addr IP address to send the packet from. 61 | * @param from_port UDP source port to use 62 | * @param to_addr IP address to send the packet to. 63 | * @param to_port UDP destination port 64 | * @param pktBuf Payload buffer pointer 65 | * @param pktLen Payload buffer length 66 | * @return true if packet was sent successfully, false otherwise. 67 | */ 68 | bool sendUdp(int sock, struct in_addr from_addr, uint16_t from_port, struct in_addr to_addr, uint16_t to_port, unsigned char *pktBuf, ssize_t pktLen) 69 | { 70 | // Build the IP header 71 | uint8_t packet_buffer[16000]; 72 | struct iphdr *iph; 73 | struct udphdr *udph; 74 | iph = (struct iphdr *)&packet_buffer[0]; 75 | udph = (struct udphdr *)&packet_buffer[sizeof(struct iphdr)]; 76 | iph->version = 4; 77 | iph->ihl = 5; 78 | iph->tos = 0; 79 | iph->tot_len = htons(sizeof(struct iphdr) + sizeof(struct udphdr) + pktLen); 80 | iph->id = 0; 81 | iph->frag_off = 0; 82 | iph->ttl = 2; 83 | iph->protocol = IPPROTO_UDP; 84 | iph->check = 0; 85 | iph->saddr = from_addr.s_addr; 86 | iph->daddr = to_addr.s_addr; 87 | 88 | udph->source = htons(from_port); 89 | udph->dest = htons(to_port); 90 | udph->len = htons(sizeof(struct udphdr) + pktLen); 91 | 92 | memcpy(&packet_buffer[sizeof(struct iphdr) + sizeof(struct udphdr)], pktBuf, pktLen); 93 | 94 | // Linux will return an EINVAL if we have an addr with a non-zero sin_port. 95 | struct sockaddr_in to_zero_port; 96 | to_zero_port.sin_family = AF_INET; 97 | to_zero_port.sin_port = 0; 98 | to_zero_port.sin_addr.s_addr = to_addr.s_addr; 99 | 100 | if(sendto(sock, packet_buffer, sizeof(struct iphdr) + sizeof(struct udphdr) + pktLen, 0, (struct sockaddr *)&to_zero_port, sizeof(to_zero_port)) < 0) 101 | { 102 | LOG(LS_UDP, LL_IMPORTANT, "Unable to send UDP packet. Parameters were %d, %p, %d, %d, %d, %d", sock, packet_buffer, sizeof(struct iphdr) + sizeof(struct udphdr) + pktLen, 0, (struct sockaddr *)&to_addr, sizeof(to_addr)); 103 | LOGHEXDUMP(LS_UDP, LL_IMPORTANT, "UDP packet buffer", packet_buffer, sizeof(struct iphdr) + sizeof(struct udphdr) + pktLen); 104 | return false; 105 | } 106 | 107 | return true; 108 | } 109 | 110 | const std::string base60 = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; 111 | /** 112 | * Convert a 64 bit number to base60 notation. Note: With a uint64_t, the maximum return from this is "uur953OEv0f". 113 | * 114 | * @param val The value to convert 115 | * @return The value represented as base60 116 | */ 117 | std::string toBase60(uint64_t val) 118 | { 119 | std::string ret; 120 | while(val >= 60) 121 | { 122 | ret.insert(0, 1, base60[val % 60]); 123 | val = val / 60; 124 | } 125 | ret = base60[val] + ret; 126 | return ret; 127 | } 128 | 129 | /** 130 | * Convert the time between two time_points to a human-readable short form duration. 131 | * 132 | * @param t1 The earlier timepoint 133 | * @param t2 The later timepoint 134 | * @return Human-readable (hours, minutes, seconds, etc) duration between the two time_points. 135 | */ 136 | std::string timepointDeltaString(std::chrono::steady_clock::time_point t1, std::chrono::steady_clock::time_point t2) 137 | { 138 | char tbuf[32]; 139 | long delta = std::chrono::duration_cast(t1 - t2).count(); 140 | std::string ret; 141 | 142 | if (delta > 86400000) { ret += std::to_string((long)delta / 86400000) + "d "s; delta = delta % 86400000; } 143 | if (delta > 3600000) { ret += std::to_string((long)delta / 3600000) + "h"s; delta = delta % 3600000; } 144 | if (delta > 60000) { ret += std::to_string((long)delta / 60000) + "m"s; delta = delta % 60000; } 145 | snprintf(tbuf, 31, "%.3f", float(delta / 1000.0)); 146 | ret += tbuf + "s"s; 147 | 148 | return ret; 149 | } 150 | 151 | double timepointDeltaDouble(std::chrono::steady_clock::time_point t1, std::chrono::steady_clock::time_point t2) 152 | { 153 | return (double)(std::chrono::duration_cast(t1 - t2).count()) / 1000.0; 154 | } 155 | 156 | /** 157 | * Convert a sockaddr to a human-readable value. Supports both IPv4 and IPv6. 158 | * 159 | * @param sa The sockaddr structure to parse 160 | * @return The human-readable IP address the sockaddr represents. 161 | */ 162 | std::string sockaddrToName(struct sockaddr *sa) 163 | { 164 | char tbuf[256]; 165 | if(sa->sa_family == AF_INET) 166 | inet_ntop(AF_INET, &((struct sockaddr_in *)sa)->sin_addr, tbuf, 256); 167 | else if(sa->sa_family == AF_INET6) 168 | inet_ntop(AF_INET6, &((struct sockaddr_in6 *)sa)->sin6_addr, tbuf, 256); 169 | else 170 | strcpy(tbuf, "Unknown address family"); 171 | 172 | return std::string(tbuf); 173 | } 174 | 175 | /** 176 | * Parse input parameters of thread count and core configuration, and populate the ThreadConfig. A thread configuration 177 | * string is a comma-separated list of cores or ranges, i.e. 1,2,4-6,8. 178 | * 179 | * @param threadcount Number of threads requested on command line, if provided, 0 if not. Defaults to 1, if needed. 180 | * @param affinity The thread affinity string requested on command line. 181 | * @param dest 182 | * @return None, but raises exceptions on error (generally a threadcfg that doesn't make sense) 183 | */ 184 | void ParseThreadConfiguration(int threadcount, std::string& affinity, ThreadConfig *dest) 185 | { 186 | // Handle case when the detailed thread configuration wasn't passed, and we just want some threads. 187 | if(affinity.empty()) 188 | { 189 | if(threadcount == 0) 190 | dest->cfg.resize(1, -1); 191 | else 192 | dest->cfg.resize(threadcount, -1); 193 | } else { 194 | // Parse the threadcfg string. 195 | dest->cfg.resize(0); 196 | size_t pos = 0; 197 | while(pos < affinity.length()) 198 | { 199 | // Find next comma if any, pull out substring. 200 | size_t comma_pos = affinity.find(",",pos); 201 | if(comma_pos == std::string::npos) 202 | comma_pos = affinity.length(); 203 | std::string range = affinity.substr(pos, comma_pos - pos); 204 | // Is this a range? 205 | size_t dash_pos = range.find("-"); 206 | if(dash_pos == std::string::npos) 207 | { 208 | // Nope, just a bare number. 209 | dest->cfg.push_back(std::stoi(range)); 210 | } else { 211 | int start = std::stoi(range.substr(0, dash_pos)); 212 | int end = std::stoi(range.substr(dash_pos + 1)); 213 | for(int i = start ; i <= end ; i ++) 214 | { 215 | dest->cfg.push_back(i); 216 | } 217 | } 218 | // Move up. 219 | pos = comma_pos + 1; 220 | } 221 | } 222 | // Do some checks. 223 | if(dest->cfg.size() > MAX_THREADS) 224 | throw std::length_error("The number of threads specified ("s + std::to_string(dest->cfg.size()) + ") exceeds the maximum allowed ("s + std::to_string(MAX_THREADS) + "). Recompile code and increase MAX_THREADS in utils.h if you need more."s); 225 | 226 | // We will check the ability to set affinity at set time. 227 | } 228 | 229 | /** 230 | * Convert an eniid_t to a hex string 231 | * @param eni 232 | * @return 233 | */ 234 | std::string MakeENIStr(eniid_t eni) 235 | { 236 | std::stringstream ss; 237 | 238 | ss << std::hex << std::setw(17) << std::setfill('0') << eni << std::dec; 239 | return ss.str(); 240 | } 241 | 242 | -------------------------------------------------------------------------------- /utils.h: -------------------------------------------------------------------------------- 1 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // SPDX-License-Identifier: MIT-0 3 | 4 | #ifndef GWLBTUN_UTILS_H 5 | #define GWLBTUN_UTILS_H 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include // For va_start, etc. 14 | #include 15 | #include 16 | #include 17 | #include "GenevePacket.h" // For eniid_t 18 | 19 | using namespace std::string_literals; 20 | 21 | #define VERSION_MAJOR 3 22 | #define VERSION_MINOR 0 23 | 24 | // If only decapsulation is required, i.e. you will never send traffic back to GWLB via the local interfaces, 25 | // you can define the following symbol to improve performance (GWLBTun no longer needs to track flow cookies, etc.) 26 | //#define NO_RETURN_TRAFFIC 27 | 28 | // Thread configuration parser and data struct 29 | #define MAX_THREADS 128 30 | typedef struct ThreadConfigStruct { 31 | std::vector cfg; 32 | } ThreadConfig; 33 | 34 | std::string stringFormat(const std::string& fmt_str, ...); 35 | std::string stringFormat(const std::string& fmt_str, va_list ap); 36 | bool sendUdp(int sock, struct in_addr from_addr, uint16_t from_port, struct in_addr to_addr, uint16_t to_port, unsigned char *pktBuf, ssize_t pktLen); 37 | std::string toBase60(uint64_t val); 38 | std::string timepointDeltaString(std::chrono::steady_clock::time_point t1, std::chrono::steady_clock::time_point t2); 39 | double timepointDeltaDouble(std::chrono::steady_clock::time_point t1, std::chrono::steady_clock::time_point t2); 40 | std::string sockaddrToName(struct sockaddr *sa); 41 | void ParseThreadConfiguration(int threadcount, std::string& affinity, ThreadConfig *dest); 42 | std::string MakeENIStr(eniid_t eni); 43 | int FindIndexOf(std::vector vector, std::string search); 44 | 45 | // If hashFunc is a function that does not result in the same hash for both flow directions, 46 | // #undef the next line so that GeneveHandler and PacketHeader changes their logic appropriately. 47 | #define HASH_IS_SYMMETRICAL 48 | /** 49 | * Simple, basic, but very fast hash function. Returns same hash in both directions, so leave HASH_IS_SYMMETRICAL 50 | * defined. 51 | * @param prot Protocol number 52 | * @param srcip Pointer to source IP data 53 | * @param dstip Pointer to destination IP data 54 | * @param ipsize Size of IP data (4 for IPv4, 16 for IPv6) 55 | * @param srcpt Source Port Number 56 | * @param dstpt Destination Port NUmber 57 | * @return 58 | */ 59 | inline size_t hashFunc(uint8_t prot, void *srcip, void *dstip, int ipsize, uint16_t srcpt, uint16_t dstpt) 60 | { 61 | uint32_t *srciplongs = (uint32_t *)srcip; 62 | uint32_t *dstiplongs = (uint32_t *)dstip; 63 | if(ipsize == 4) 64 | { 65 | return prot + srciplongs[0] + dstiplongs[0] + srcpt + dstpt; 66 | } else { 67 | return prot + srciplongs[0] + srciplongs[1] + srciplongs[2] + srciplongs[3] + 68 | dstiplongs[0] + dstiplongs[1] + dstiplongs[2] + dstiplongs[3] + srcpt + dstpt; 69 | } 70 | } 71 | 72 | /** 73 | * Case-insensitive iterable thing search 74 | * 75 | * @param iter Iterable to search through 76 | * @param search Search string 77 | * @return Index in vector of string case-insensitive, or -1 if not found. 78 | */ 79 | template int FindIndexOf(X iter, std::string search) 80 | { 81 | int ret = 0; 82 | 83 | std::string searchLower = search; 84 | std::transform(searchLower.begin(), searchLower.end(), searchLower.begin(), [](unsigned char c){return std::tolower(c); }); 85 | 86 | for(auto it = iter.begin(); it != iter.end(); it++, ret++) 87 | if(*it == searchLower) 88 | return ret; 89 | 90 | return -1; 91 | } 92 | 93 | #endif //GWLBTUN_UTILS_H 94 | --------------------------------------------------------------------------------