├── CONTRIBUTING ├── LICENSE ├── Makefile ├── README.md ├── asn_map.cc ├── asn_map.h ├── asn_map_test.cc ├── clerk.cc ├── flow.cc ├── flow.h ├── flow_test.cc ├── geolite_asns.py ├── headers.cc ├── headers.h ├── headers_test.cc ├── ipfix.cc ├── ipfix.h ├── send.cc ├── send.h ├── send_test.cc ├── stringpiece.h ├── test_main.cc ├── testimony.cc ├── testimony.h ├── util.cc └── util.h /CONTRIBUTING: -------------------------------------------------------------------------------- 1 | Want to contribute? Great! First, read this page (including the small print at the end). 2 | 3 | ### Before you contribute 4 | Before we can use your code, you must sign the 5 | [Google Individual Contributor License Agreement](https://cla.developers.google.com/about/google-individual) 6 | (CLA), which you can do online. The CLA is necessary mainly because you own the 7 | copyright to your changes, even after your contribution becomes part of our 8 | codebase, so we need your permission to use and distribute your code. We also 9 | need to be sure of various other things—for instance that you'll tell us if you 10 | know that your code infringes on other people's patents. You don't have to sign 11 | the CLA until after you've submitted your code for review and a member has 12 | approved it, but you must do it before we can put your code into our codebase. 13 | Before you start working on a larger contribution, you should get in touch with 14 | us first through the issue tracker with your idea so that we can help out and 15 | possibly guide you. Coordinating up front makes it much easier to avoid 16 | frustration later on. 17 | 18 | ### Code reviews 19 | All submissions, including submissions by project members, require review. We 20 | use Github pull requests for this purpose. 21 | 22 | ### The small print 23 | Contributions made by corporations are covered by a different agreement than 24 | the one above, the 25 | [Software Grant and Corporate Contributor License Agreement](https://cla.developers.google.com/about/google-corporate). -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Google Inc. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | DEPS=Makefile *.h 16 | 17 | CC=g++ 18 | CFLAGS=-std=c++0x -g -Wall -fno-strict-aliasing -fPIC -fPIE -pie -fstack-protector -D_FORTIFY_SOURCE=2 -rdynamic -O2 -Wno-narrowing 19 | LDFLAGS=-Wl,-z,now -Wl,-z,relro 20 | SHARED_LIBS=-ltestimony -lglog -lgflags -lcityhash -lpthread 21 | TEST_LIBS=-lgtest 22 | STATIC_LIBS=/usr/lib/x86_64-linux-gnu/libglog.a /usr/lib/libtestimony.a /usr/local/lib/libcityhash.a /usr/lib/x86_64-linux-gnu/libgflags.a 23 | 24 | OBJECTS=flow.o headers.o ipfix.o send.o testimony.o util.o asn_map.o 25 | TESTS=flow_test.o headers_test.o send_test.o asn_map_test.o 26 | 27 | all: clerk 28 | 29 | clean: 30 | rm -f *.o clerk core 31 | 32 | 33 | ### Building clerk, either in normal (g++) or sanitization (clang) modes ### 34 | 35 | # Generate g++ object files. 36 | %.o: %.cc $(DEPS) 37 | $(CC) $(CFLAGS) -c -o $@ $< 38 | 39 | # Generate the clerk binary itself. You mostly want this :) 40 | clerk: $(OBJECTS) 41 | $(CC) $(CFLAGS) -o $@ clerk.cc $^ $(LDFLAGS) $(SHARED_LIBS) 42 | 43 | .PHONY: test 44 | test: $(OBJECTS) $(TESTS) 45 | $(CC) $(CFLAGS) -o $@ test_main.cc $^ $(LDFLAGS) $(SHARED_LIBS) $(TEST_LIBS) && ./test 46 | 47 | clerk_static: $(OBJECTS) 48 | $(CC) $(CFLAGS) -o $@ clerk.cc $^ $(LDFLAGS) $(STATIC_LIBS) 49 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Clerk 2 | 3 | Clerk is a passive netflow/IPFIX generator designed for high-throughput and 4 | testimony-based packet sharing. 5 | 6 | ## Architecture 7 | 8 | `clerk` uses https://github.com/google/testimony to get packets across N threads. 9 | 10 | 1 Packet hits NIC 11 | 1 Kernel places packet in `AF_PACKET` mmap region 12 | 1 `testimonyd` hands mmap region to `clerk` packet thread 13 | 1 `clerk` thread looks up and updates flow info 14 | * creates a key based on identifiers (src/dst IP/port, protocol, qos, etc) 15 | * looks up current stats, creating empty statistics if necessary 16 | * updates stats with new bytes/packets/tcp flags/etc. 17 | 1 every minute, `clerk` main thread sends IPFIX 18 | * gathers flows from each of N packet threads 19 | * combines flows 20 | * generates IPFIX packets based on combined flow from all threads 21 | * sends out UDP socket 22 | 23 | ## Flow Information 24 | 25 | Currently, `clerk` uses a fixed template (actually 2, one for IPv4, the other 26 | for IPv6): 27 | 28 | * `IPV4_SRC_ADDR` (4 bytes) or `IPV6_SRC_ADDR` (16 bytes) 29 | * `IPV4_DST_ADDR` (4 bytes) or `IPV6_DST_ADDR` (16 bytes) 30 | * `L4_SRC_PORT` (2 bytes) 31 | * `L4_DST_PORT` (2 bytes) 32 | * `PROTOCOL` (1 byte) 33 | * `TCP_FLAGS` (1 byte) 34 | * `ICMP_TYPE` (2 bytes) 35 | * `BGP_SOURCE_AS_NUMBER` (4 bytes) 36 | * `BGP_DESTINATION_AS_NUMBER` (4 bytes) 37 | * `IN_BYTES` (8 bytes) 38 | * `IN_PKTS` (8 bytes) 39 | * `FLOW_START_NANOSECONDS` (8 bytes) 40 | * `FLOW_END_NANOSECONDS` (8 bytes) 41 | * `IP_CLASS_OF_SERVICE` (1 byte) 42 | * `FLOW_END_REASON` (1 byte) 43 | * `VLAN_ID` (2 bytes) 44 | 45 | It's probably possible to expand this further in the future, but for now this 46 | solves most of our internal needs quite nicely. 47 | 48 | ## Disclaimer 49 | 50 | This is not an official Google product. 51 | -------------------------------------------------------------------------------- /asn_map.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "asn_map.h" 16 | 17 | #include 18 | 19 | #include 20 | 21 | #include 22 | 23 | namespace clerk { 24 | 25 | namespace { 26 | 27 | // Useful for printf debugging :D 28 | std::string IPAsString(const uint8_t* ip) { 29 | const uint8_t* u = reinterpret_cast(ip); 30 | char buf[40]; 31 | return std::string( 32 | buf, snprintf(buf, sizeof(buf), 33 | "%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%" 34 | "02x%02x:%02x%02x", 35 | u[0], u[1], u[2], u[3], u[4], u[5], u[6], u[7], u[8], u[9], 36 | u[10], u[11], u[12], u[13], u[14], u[15])); 37 | } 38 | 39 | } // namespace 40 | 41 | const uint32_t ASNMap::NoASN = 0; 42 | 43 | ASNMap::ASNMap() {} 44 | 45 | ASNMap::~ASNMap() {} 46 | 47 | void ASNMap::Add(const uint8_t* from, const uint8_t* to, uint32_t asn) { 48 | // from must be <= to. 49 | CHECK_LE(Compare(from, to), 0); 50 | CHECK_NE(asn, NoASN); 51 | // [from, to] should not intersect with any current range. 52 | auto found = set_.lower_bound(Range(to)); 53 | if (found != set_.end()) { 54 | CHECK_LT(Compare(to, found->from), 0); 55 | } 56 | if (found != set_.begin()) { 57 | --found; 58 | CHECK_LT(Compare(found->to, from), 0); 59 | } 60 | Range r(from, to, asn); 61 | VLOG(1) << "Mapping range " << IPAsString(from) << " - " << IPAsString(to) 62 | << " to ASN " << asn; 63 | set_.emplace(r); 64 | } 65 | 66 | ASNMap::Range::Range(const uint8_t* a, const uint8_t* b, uint32_t n) : asn(n) { 67 | memcpy(from, a, 16); 68 | memcpy(to, b, 16); 69 | } 70 | 71 | bool ASNMap::Range::Contains(const uint8_t* addr) const { 72 | return Compare(from, addr) <= 0 && Compare(addr, to) <= 0; 73 | } 74 | 75 | uint32_t ASNMap::ASN(const uint8_t* addr) const { 76 | const auto& found = set_.lower_bound(Range(addr)); 77 | if (found != set_.end() && found->Contains(addr)) { 78 | VLOG(2) << "Mapped " << IPAsString(addr) << " to ASN " << found->asn; 79 | return found->asn; 80 | } 81 | VLOG(2) << "Mapped " << IPAsString(addr) << " to NoASN (0)"; 82 | return NoASN; 83 | } 84 | 85 | namespace internal { 86 | 87 | // Pull out a CSV value from a line pointed to by *val. Returns a 88 | // null-terminated value string, and points *val past it so NextCSVValue may be 89 | // called on it again. Returns nullptr if unable to pull out a value. 90 | char* NextCSVValue(char** val) { 91 | char* limit = *val + strlen(*val); 92 | if (*val == limit) { 93 | return nullptr; 94 | } 95 | char* out = *val; 96 | char* comma = strchr(*val, ','); 97 | if (comma) { 98 | *comma = '\0'; 99 | *val = comma + 1; 100 | } else { 101 | *val = limit; 102 | } 103 | return out; 104 | } 105 | 106 | } // namespace internal 107 | 108 | void LoadFromCSV(ASNMap* to, FILE* f) { 109 | char line[1024]; 110 | int lines = 0; 111 | while (fgets(line, sizeof(line), f) != nullptr) { 112 | lines++; 113 | char* next = line; 114 | char* startip = CHECK_NOTNULL(internal::NextCSVValue(&next)); 115 | char* limitip = CHECK_NOTNULL(internal::NextCSVValue(&next)); 116 | char* asn = CHECK_NOTNULL(internal::NextCSVValue(&next)); 117 | uint8_t startaddr[16]; 118 | uint8_t limitaddr[16]; 119 | memset(startaddr, 0, sizeof(startaddr)); 120 | memset(limitaddr, 0, sizeof(limitaddr)); 121 | PCHECK(1 == inet_pton(AF_INET6, startip, startaddr)); 122 | PCHECK(1 == inet_pton(AF_INET6, limitip, limitaddr)); 123 | to->Add(startaddr, limitaddr, atoll(asn)); 124 | } 125 | LOG(INFO) << "Read " << lines << " entries from ASN CSV"; 126 | } 127 | 128 | } // namespace clerk 129 | -------------------------------------------------------------------------------- /asn_map.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_ASN_MAP_H_ 16 | #define CLERK_ASN_MAP_H_ 17 | 18 | #include 19 | #include 20 | #include 21 | 22 | #include 23 | #include 24 | 25 | namespace clerk { 26 | 27 | class ASNMap { 28 | public: 29 | ASNMap(); 30 | ~ASNMap(); 31 | // from and to must point to 16-byte IP addresses. IPv4 addresses must be 32 | // IPv4-mapped IPv6 addresses in the lowest-order bytes (e.g. ::192.168.1.1). 33 | void Add(const uint8_t* from, const uint8_t* to, uint32_t asn); 34 | 35 | // addr must point to a 16-byte IP address. IPv4 addresses must be 36 | // IPv4-mapped IPv6 addresses in the lowest-order bytes (e.g. ::192.168.1.1). 37 | // Returns NoASN if not found. 38 | uint32_t ASN(const uint8_t* addr) const; 39 | 40 | // Clear removes all current mapping from this map. 41 | void Clear() { set_.clear(); } 42 | 43 | static const uint32_t NoASN; // == 0 44 | 45 | private: 46 | struct Range { 47 | Range() { memset(this, 0, sizeof(Range)); } 48 | Range(const uint8_t* a, const uint8_t* b, uint32_t n); 49 | Range(const uint8_t* b) : Range() { memcpy(to, b, 16); } // for finds 50 | bool Contains(const uint8_t* addr) const; 51 | bool operator<(const Range& r) const { 52 | // We order based on the 'to' address (the higher of from and to), so that 53 | // set::lower_bound will immediately return the only range candidate which 54 | // might contain an address. 55 | return ASNMap::Compare(to, r.to) < 0; 56 | } 57 | 58 | uint8_t from[16]; 59 | uint8_t to[16]; 60 | uint32_t asn; 61 | }; 62 | 63 | static inline int Compare(const uint8_t* a, const uint8_t* b) { 64 | return memcmp(a, b, 16); 65 | } 66 | 67 | std::set set_; 68 | }; 69 | 70 | // Load CSV of IP ranges and ASNs. Example file lines: 71 | // ::,::ffff,1234 72 | // ::1:0,2001::,4567 73 | // Each line contains a start and limit IP address, and an ASN. 74 | // IPs are mapped to ASNs using these (non-overlapping, inclusive) ranges. 75 | // IPv4 addresses are mapped in the range ::0000:0000 - ::FFFF:FFFF. 76 | void LoadFromCSV(ASNMap* to, FILE* f); 77 | 78 | namespace internal { // exposed just for testing. 79 | 80 | char* NextCSVValue(char** val); 81 | 82 | } // internal 83 | 84 | } // namespace clerk 85 | 86 | #endif // CLERK_ASN_MAP_H_ 87 | -------------------------------------------------------------------------------- /asn_map_test.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | 17 | #include 18 | #include "asn_map.h" 19 | 20 | namespace clerk { 21 | 22 | class CSVTest : public ::testing::Test {}; 23 | 24 | TEST_F(CSVTest, TestNext) { 25 | char input[] = "ABC,DEF,GHI,JKL,MNOP,QRS"; 26 | char* val = input; 27 | const char* want[] = { 28 | "ABC", "DEF", "GHI", "JKL", "MNOP", "QRS", 29 | }; 30 | int want_size = sizeof(want) / sizeof(*want); 31 | for (int i = 0; i < want_size; i++) { 32 | char* got = internal::NextCSVValue(&val); 33 | EXPECT_EQ(std::string(want[i]), std::string(got)); 34 | } 35 | EXPECT_EQ(internal::NextCSVValue(&val), nullptr); 36 | } 37 | 38 | class ASNMapTest : public ::testing::Test {}; 39 | 40 | TEST_F(ASNMapTest, TestBasic) { 41 | uint8_t ipA[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; 42 | uint8_t ipAB[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3}; 43 | uint8_t ipB[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff}; 44 | uint8_t ipC[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0}; 45 | uint8_t ipCD[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3}; 46 | uint8_t ipD[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0xff, 0xff}; 47 | uint8_t ipDE[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0xff, 0xff}; 48 | uint8_t ipE[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0}; 49 | uint8_t ipEF[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 3}; 50 | uint8_t ipF[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0xff, 0xff, 0xff}; 51 | uint8_t ipG[] = {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0xff, 0xff, 0xff}; 52 | 53 | ASNMap m; 54 | m.Add(ipE, ipF, 3); 55 | m.Add(ipA, ipB, 1); 56 | m.Add(ipG, ipG, 4); 57 | m.Add(ipC, ipD, 2); 58 | 59 | // Check in-between values 60 | EXPECT_EQ(m.ASN(ipAB), 1); 61 | EXPECT_EQ(m.ASN(ipCD), 2); 62 | EXPECT_EQ(m.ASN(ipDE), ASNMap::NoASN); 63 | EXPECT_EQ(m.ASN(ipEF), 3); 64 | 65 | // Check boundaries 66 | EXPECT_EQ(m.ASN(ipA), 1); 67 | EXPECT_EQ(m.ASN(ipB), 1); 68 | EXPECT_EQ(m.ASN(ipC), 2); 69 | EXPECT_EQ(m.ASN(ipD), 2); 70 | EXPECT_EQ(m.ASN(ipE), 3); 71 | EXPECT_EQ(m.ASN(ipF), 3); 72 | EXPECT_EQ(m.ASN(ipG), 4); 73 | } 74 | 75 | } // namespace clerk 76 | -------------------------------------------------------------------------------- /clerk.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | #include 17 | 18 | #include 19 | #include 20 | 21 | #include 22 | #include "asn_map.h" 23 | #include "ipfix.h" 24 | #include "testimony.h" 25 | 26 | #include "util.h" 27 | 28 | using google::ParseCommandLineFlags; 29 | 30 | DEFINE_string(testimony, "", "Name of testimony socket"); 31 | DEFINE_string(collector, "127.0.0.1:6555", "Socket address of collector"); 32 | DEFINE_double(upload_every_secs, 60, "Upload IPFIX to collector once every X"); 33 | DEFINE_double(flow_timeout_secs, 60 * 5, "Time out flows after X"); 34 | DEFINE_string(asns_csv, "", 35 | "Filename of ASN CSV file. See *_asns.py for ways to get ASN " 36 | "data readable by clerk."); 37 | DEFINE_double(asns_reread_every_secs, 86400, 38 | "Reread ASN CSV file once every X seconds"); 39 | 40 | // CombineGather parallelizes the process of combining multiple IPFIX states 41 | // together, by synchronously combining half of them with the other half, until 42 | // there's only one left. 43 | void CombineGather(std::vector>* states) { 44 | while (states->size() > 1) { 45 | std::vector threads; 46 | // New size is 1/2 the old size, rounded up. 47 | int new_size = states->size() / 2 + states->size() % 2; 48 | LOG(INFO) << "Combining " << states->size() << " states into " << new_size; 49 | for (int i = 0; i < new_size; i++) { 50 | size_t other = i + new_size; 51 | if (other < states->size()) { // not always true, if states.size() is odd 52 | threads.emplace_back(std::thread([states, i, other]() { 53 | // Actual combination implemented with += operator. 54 | *reinterpret_cast((*states)[i].get()) += 55 | *reinterpret_cast((*states)[other].get()); 56 | })); 57 | } 58 | } 59 | for (size_t i = 0; i < threads.size(); i++) { 60 | threads[i].join(); 61 | } 62 | // Now throw away the second half, since it was combined with the first 63 | // and is now redundant. 64 | states->resize(new_size); 65 | } 66 | } 67 | 68 | void AddASNsTo(clerk::flow::Table* t, const clerk::ASNMap& asns) { 69 | LOG(INFO) << "Adding ASNs to flows"; 70 | for (auto iter = t->begin(); iter != t->end(); ++iter) { 71 | iter->second.src_asn = asns.ASN(iter->first.src_ip); 72 | iter->second.dst_asn = asns.ASN(iter->first.dst_ip); 73 | } 74 | } 75 | 76 | // Convert a socket address to a sockaddr_storage. 77 | // This is quick and dirty, and could definitely use some work. 78 | // Right now, it supports 2 formats: 79 | // 192.168.1.2:3333 (IPv4:Port) 80 | // [2001::0123]:4444 ([IPv6]:Port) 81 | void StringToSocketStorage(const std::string& addr, struct sockaddr_storage* ss, 82 | socklen_t* size) { 83 | memset(ss, 0, sizeof(*ss)); 84 | string mutable_addr = addr; 85 | CHECK_GT(mutable_addr.size(), 1); 86 | auto colon = mutable_addr.find_last_of(':'); 87 | CHECK_NE(colon, std::string::npos); 88 | mutable_addr[colon] = '\0'; // make string before colon null-terminated 89 | int port = atoi(mutable_addr.data() + colon + 1); 90 | if (mutable_addr[0] == '[') { // v6 91 | auto ip6end = mutable_addr.find_first_of(']'); 92 | CHECK_NE(ip6end, std::string::npos); 93 | mutable_addr[ip6end] = '\0'; 94 | CHECK_EQ(ip6end + 1, colon); 95 | auto in6 = reinterpret_cast(ss); 96 | in6->sin6_port = htons(port); 97 | in6->sin6_family = AF_INET6; 98 | // We add 1 to data() to skip over initial '[' char. 99 | CHECK_EQ(1, inet_pton(AF_INET6, mutable_addr.data() + 1, &in6->sin6_addr)); 100 | *size = sizeof(struct sockaddr_in6); 101 | } else { // v4 102 | auto in4 = reinterpret_cast(ss); 103 | in4->sin_port = htons(port); 104 | in4->sin_family = AF_INET; 105 | CHECK_EQ(1, inet_pton(AF_INET, mutable_addr.data(), &in4->sin_addr)); 106 | *size = sizeof(struct sockaddr_in); 107 | } 108 | } 109 | 110 | void ReadASNs(clerk::ASNMap* map) { 111 | if (!FLAGS_asns_csv.empty()) { 112 | LOG(INFO) << "Reading ASNs from " << FLAGS_asns_csv; 113 | auto f = fopen(FLAGS_asns_csv.c_str(), "r"); 114 | PCHECK(f != nullptr) << "Failed to open " << FLAGS_asns_csv; 115 | map->Clear(); 116 | clerk::LoadFromCSV(map, f); 117 | fclose(f); 118 | } 119 | } 120 | 121 | int main(int argc, char** argv) { 122 | ParseCommandLineFlags(&argc, &argv, true); 123 | clerk::ASNMap asns; 124 | ReadASNs(&asns); 125 | double last_asn_read_secs = GetCurrentTimeSeconds(); 126 | 127 | clerk::IPFIXFactory factory; 128 | 129 | std::unique_ptr sender; 130 | if (FLAGS_collector == "stdout") { 131 | sender.reset(new clerk::FileSender(stdout, &factory)); 132 | } else { 133 | struct sockaddr_storage ss; 134 | socklen_t ss_size; 135 | StringToSocketStorage(FLAGS_collector, &ss, &ss_size); 136 | int fd = socket(ss.ss_family, SOCK_DGRAM, 0); 137 | PCHECK(connect(fd, reinterpret_cast(&ss), ss_size) >= 0) 138 | << "Connect to " << FLAGS_collector << " failed"; 139 | sender.reset(new clerk::PacketSender(fd, &factory)); 140 | } 141 | 142 | clerk::TestimonyProcessor processor(FLAGS_testimony, &factory); 143 | double last_upload_secs = GetCurrentTimeSeconds(); 144 | processor.StartThreads(); 145 | while (1) { 146 | SleepForSeconds(last_upload_secs + FLAGS_upload_every_secs - 147 | GetCurrentTimeSeconds()); 148 | last_upload_secs = GetCurrentTimeSeconds(); 149 | factory.SetCutoffNanos((last_upload_secs - FLAGS_flow_timeout_secs) * 150 | kNumNanosPerSecond); 151 | std::vector> states; 152 | processor.Gather(&states, false); 153 | CombineGather(&states); 154 | clerk::IPFIX* first = reinterpret_cast(states[0].get()); 155 | clerk::flow::Table f; 156 | first->SwapFlows(&f); 157 | AddASNsTo(&f, asns); 158 | sender->Send(f); 159 | if (last_upload_secs - last_asn_read_secs > FLAGS_asns_reread_every_secs) { 160 | last_asn_read_secs = last_upload_secs; 161 | ReadASNs(&asns); 162 | } 163 | } 164 | } 165 | -------------------------------------------------------------------------------- /flow.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | #include "flow.h" 15 | 16 | #include 17 | #include 18 | 19 | 20 | namespace clerk { 21 | namespace flow { 22 | 23 | Key::Key() { memset(this, 0, sizeof(*this)); } 24 | 25 | bool Key::operator==(const Key& other) const { 26 | return memcmp(this, &other, sizeof(Key)) == 0; 27 | } 28 | 29 | size_t Key::hash() const { 30 | return CityHash64(reinterpret_cast(this), sizeof(*this)); 31 | } 32 | 33 | Stats::Stats() { memset(this, 0, sizeof(*this)); } 34 | 35 | Stats::Stats(uint64_t b, uint64_t p, uint64_t ts_ns) 36 | : bytes(b), packets(p), tcp_flags(0), first_ns(ts_ns), last_ns(ts_ns) {} 37 | 38 | const Stats& Stats::operator+=(const Stats& f) { 39 | bytes += f.bytes; 40 | packets += f.packets; 41 | tcp_flags |= f.tcp_flags; 42 | if (!first_ns || first_ns > f.first_ns) first_ns = f.first_ns; 43 | if (!last_ns || last_ns < f.last_ns) last_ns = f.last_ns; 44 | return *this; 45 | } 46 | 47 | const Stats& AddToTable(Table* t, const Key& key, const Stats& stats) { 48 | auto finder = t->find(key); 49 | if (finder == t->end()) { 50 | auto emplaced = t->emplace(key, stats); 51 | return emplaced.first->second; 52 | } 53 | finder->second += stats; 54 | return finder->second; 55 | } 56 | 57 | void CombineTable(Table* dst, const Table& src) { 58 | for (const auto& iter : src) { 59 | AddToTable(dst, iter.first, iter.second); 60 | } 61 | } 62 | 63 | } // namespace flow 64 | } // namespace clerk 65 | -------------------------------------------------------------------------------- /flow.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_FLOW_H_ 16 | #define CLERK_FLOW_H_ 17 | 18 | #include 19 | 20 | #include 21 | 22 | namespace clerk { 23 | namespace flow { 24 | 25 | struct Key { 26 | Key(); 27 | 28 | uint8_t src_ip[16]; 29 | uint8_t dst_ip[16]; 30 | uint16_t src_port; 31 | uint16_t dst_port; 32 | uint16_t vlan; 33 | uint8_t icmp_type; 34 | uint8_t icmp_code; 35 | uint8_t network; // 0 = unknown, 4 = ipv4, 6 = ipv6 36 | uint8_t protocol; 37 | uint8_t tos; // IPv4 TOS, IPv6 traffic class 38 | 39 | bool operator==(const Key& b) const; 40 | inline bool operator!=(const Key& b) const { return !operator==(b); } 41 | size_t hash() const; 42 | void set_src_ip4(uint32_t ip4); 43 | void set_dst_ip4(uint32_t ip4); 44 | void set_network(uint8_t net); 45 | uint32_t get_src_ip4() const; 46 | uint32_t get_dst_ip4() const; 47 | void set_src_ip6(const char* ip6); 48 | void set_dst_ip6(const char* ip6); 49 | }; 50 | 51 | inline void Key::set_network(uint8_t net) { 52 | if (__builtin_expect((network == 6 && net == 4), false)) { 53 | // If we switch from v6 to v4, we need to clear the bits of the IPs. 54 | memset(src_ip, 0, sizeof(src_ip)); 55 | memset(dst_ip, 0, sizeof(dst_ip)); 56 | } 57 | network = net; 58 | } 59 | inline void Key::set_src_ip4(uint32_t ip4) { 60 | set_network(4); 61 | src_ip[12] = ip4 >> 24; 62 | src_ip[13] = ip4 >> 16; 63 | src_ip[14] = ip4 >> 8; 64 | src_ip[15] = ip4; 65 | } 66 | inline void Key::set_dst_ip4(uint32_t ip4) { 67 | set_network(4); 68 | dst_ip[12] = ip4 >> 24; 69 | dst_ip[13] = ip4 >> 16; 70 | dst_ip[14] = ip4 >> 8; 71 | dst_ip[15] = ip4; 72 | } 73 | #define LEFT_SHIFT_32(x, by) (((uint32_t)(x)) << (by)) 74 | inline uint32_t Key::get_src_ip4() const { 75 | CHECK_EQ(network, 4); 76 | return LEFT_SHIFT_32(src_ip[12], 24) | 77 | LEFT_SHIFT_32(src_ip[13], 16) | 78 | LEFT_SHIFT_32(src_ip[14], 8) | 79 | LEFT_SHIFT_32(src_ip[15], 0); 80 | } 81 | inline uint32_t Key::get_dst_ip4() const { 82 | CHECK_EQ(network, 4); 83 | return LEFT_SHIFT_32(dst_ip[12], 24) | 84 | LEFT_SHIFT_32(dst_ip[13], 16) | 85 | LEFT_SHIFT_32(dst_ip[14], 8) | 86 | LEFT_SHIFT_32(dst_ip[15], 0); 87 | } 88 | #undef LEFT_SHIFT_32 89 | inline void Key::set_src_ip6(const char* ip6) { 90 | set_network(6); 91 | memcpy(src_ip, ip6, sizeof(src_ip)); 92 | } 93 | inline void Key::set_dst_ip6(const char* ip6) { 94 | set_network(6); 95 | memcpy(dst_ip, ip6, sizeof(dst_ip)); 96 | } 97 | 98 | struct Stats { 99 | // From http://www.iana.org/assignments/ipfix/ipfix.xhtml 100 | enum FinishedType { 101 | IDLE_TIMEOUT = 102 | 1, // The Flow was terminated because it was considered to be idle. 103 | ACTIVE_TIMEOUT = 2, // The Flow was terminated for reporting purposes while 104 | // it was still active, for example, after the maximum 105 | // lifetime of unreported Flows was reached. 106 | END_DETECTED = 3, // The Flow was terminated because the Metering Process 107 | // detected signals indicating the end of the Flow, for 108 | // example, the TCP FIN flag. 109 | FORCED_END = 4, // The Flow was terminated because of some external event, 110 | // for example, a shutdown of the Metering Process 111 | // initiated by a network management application. 112 | LACK_OF_RESOURCES = 5, // The Flow was terminated because of lack of 113 | // resources available to the Metering Process 114 | // and/or the Exporting Process. 115 | }; 116 | Stats(); 117 | Stats(uint64_t b, uint64_t p, uint64_t ts_ns); 118 | 119 | uint64_t bytes; 120 | uint64_t packets; 121 | uint8_t tcp_flags; 122 | uint64_t first_ns, last_ns; // nanos since epoch 123 | 124 | 125 | // Actually part of the key, but filled in later on, when the key is already 126 | // constant and unmodifiable. 127 | uint32_t src_asn; 128 | uint32_t dst_asn; 129 | 130 | const Stats& operator+=(const Stats& f); 131 | uint8_t Finished(uint64_t cutoff_ns) const { 132 | if (last_ns < cutoff_ns) { 133 | return IDLE_TIMEOUT; 134 | } 135 | if (tcp_flags & (0x01 /* FIN */ | 0x04 /* RST */)) { 136 | return END_DETECTED; 137 | } 138 | return ACTIVE_TIMEOUT; 139 | } 140 | }; 141 | 142 | } // namespace flow 143 | } // namespace clerk 144 | 145 | namespace std { 146 | 147 | template <> 148 | struct hash { 149 | size_t operator()(const clerk::flow::Key& k) const { return k.hash(); } 150 | }; 151 | 152 | } // namespace std 153 | 154 | namespace clerk { 155 | namespace flow { 156 | 157 | typedef std::unordered_map Table; 158 | const Stats& AddToTable(Table* t, const Key& key, const Stats& stats); 159 | void CombineTable(Table* dst, const Table& src); 160 | 161 | } // namespace flow 162 | } // namespace clerk 163 | 164 | #endif // CLERK_FLOW_H_ 165 | -------------------------------------------------------------------------------- /flow_test.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | 17 | #include "flow.h" 18 | 19 | namespace clerk { 20 | namespace flow { 21 | 22 | class KeyTest : public ::testing::Test {}; 23 | 24 | const char data[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 25 | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}; 26 | 27 | TEST_F(KeyTest, TestCombine) { 28 | Key a; 29 | a.set_src_ip4(1); 30 | a.set_dst_ip4(2); 31 | a.src_port = 3; 32 | a.dst_port = 4; 33 | a.protocol = 5; 34 | Key b; 35 | b.set_src_ip4(1); 36 | b.set_dst_ip4(2); 37 | b.src_port = 3; 38 | b.dst_port = 4; 39 | b.protocol = 5; 40 | EXPECT_EQ(a, b); 41 | #define EQMOD(field, val) \ 42 | do { \ 43 | auto old = b.field; \ 44 | b.field = val; \ 45 | EXPECT_NE(a, b); \ 46 | EXPECT_NE(a.hash(), b.hash()); \ 47 | b.field = old; \ 48 | EXPECT_EQ(a, b); \ 49 | EXPECT_EQ(a.hash(), b.hash()); \ 50 | } while (0) 51 | EQMOD(src_port, 9); 52 | EQMOD(dst_port, 9); 53 | EQMOD(protocol, 9); 54 | EQMOD(network, 9); 55 | EQMOD(tos, 9); 56 | EQMOD(icmp_type, 9); 57 | EQMOD(icmp_code, 9); 58 | #undef EQMOD 59 | 60 | a.set_src_ip4(0); 61 | a.set_dst_ip4(0); 62 | a.set_src_ip6(&data[0]); 63 | a.set_dst_ip6(&data[0]); 64 | EXPECT_NE(a, b); 65 | b.set_src_ip4(0); 66 | b.set_dst_ip4(0); 67 | b.set_src_ip6(&data[16]); 68 | b.set_dst_ip6(&data[16]); 69 | EXPECT_EQ(a, b); 70 | } 71 | 72 | class StatsTest : public ::testing::Test {}; 73 | 74 | TEST_F(StatsTest, TestIncrement) { 75 | Stats a(10, 1, 1000); 76 | EXPECT_EQ(a.bytes, 10); 77 | EXPECT_EQ(a.packets, 1); 78 | EXPECT_EQ(a.first_ns, 1000); 79 | EXPECT_EQ(a.last_ns, 1000); 80 | a += Stats(5, 2, 1500); 81 | EXPECT_EQ(a.bytes, 15); 82 | EXPECT_EQ(a.packets, 3); 83 | EXPECT_EQ(a.first_ns, 1000); 84 | EXPECT_EQ(a.last_ns, 1500); 85 | a += Stats(3, 4, 500); // backwards in time, shouldn't probably happen 86 | EXPECT_EQ(a.bytes, 18); 87 | EXPECT_EQ(a.packets, 7); 88 | EXPECT_EQ(a.first_ns, 500); 89 | EXPECT_EQ(a.last_ns, 1500); 90 | } 91 | 92 | class TableTest : public ::testing::Test {}; 93 | 94 | TEST_F(TableTest, TestAdd) { 95 | Table t; 96 | for (int i = 0; i < 100; i++) { 97 | for (int ip = 0; ip < 16; ip++) { 98 | Key a; 99 | a.set_src_ip6(&data[ip]); 100 | a.set_dst_ip6(&data[ip]); 101 | auto s = AddToTable(&t, a, Stats(i, i * 2, 1000)); 102 | EXPECT_EQ(s.bytes, i * (i + 1) / 2); 103 | EXPECT_EQ(s.packets, i * (i + 1)); 104 | } 105 | } 106 | } 107 | 108 | } // namespace flow 109 | } // namespace clerk 110 | -------------------------------------------------------------------------------- /geolite_asns.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # Copyright 2016 Google Inc. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Pulls down MaxMind GeoLite ASN lists into a single CSV expected by clerk. 16 | 17 | Pulls down IPv4 and IPv6 ASN maps provided at 18 | http://dev.maxmind.com/geoip/legacy/geolite/ and outputs them to STDOUT as a 19 | single CSV file that's usable by clerk. 20 | """ 21 | 22 | # This product uses GeoLite data created by MaxMind, available from 23 | # http://www.maxmind.com. 24 | 25 | import csv 26 | import StringIO 27 | import urllib 28 | import zipfile 29 | 30 | V4_URL = 'http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum2.zip' 31 | V6_URL = 'http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum2v6.zip' 32 | 33 | 34 | def CSVFileFromURL(url): 35 | zf = zipfile.ZipFile(StringIO.StringIO(urllib.urlopen(url).read())) 36 | return csv.reader(zf.open(zf.namelist()[0])) 37 | 38 | 39 | def V4IntToIPv6(v4): 40 | return '::' + '%04x:%04x' % (v4 >> 16, v4 & 0xffff) 41 | 42 | 43 | def ASNFromCSV(cell): 44 | return int(cell.split(' ')[0][2:]) 45 | 46 | 47 | def main(): 48 | """Download CSVs, output to single, combined CSV file.""" 49 | for row in CSVFileFromURL(V4_URL): 50 | # V4 format is IP,IP,AS### maybe other stuff 51 | # where IPs are ints (10.1.1.1 is '167837953'). 52 | ip_a_int, ip_b_int, asn = int(row[0]), int(row[1]), ASNFromCSV(row[2]) 53 | print '%s,%s,%d' % (V4IntToIPv6(ip_a_int), V4IntToIPv6(ip_b_int), asn) 54 | for row in CSVFileFromURL(V6_URL): 55 | # V6 format is AS### maybe other stuff,IP,IP,CIDR 56 | # where IPs are human-readable IPv6 addresses ('2001::1234'). 57 | print '%s,%s,%d' % (row[1], row[2], ASNFromCSV(row[0])) 58 | 59 | 60 | if __name__ == '__main__': 61 | main() 62 | -------------------------------------------------------------------------------- /headers.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "headers.h" 16 | 17 | #include // ntohs(), ntohl() 18 | #include 19 | 20 | #include "stringpiece.h" 21 | 22 | namespace clerk { 23 | 24 | namespace { 25 | 26 | // kTypeEthernet is NOT a valid ETH_P_ type. We use it to signify that the next 27 | // layer to decode is an ethernet header. 28 | const uint16_t kTypeEthernet = 0; 29 | const uint16_t kProtocolUnknown = 0; 30 | const uint32_t kMPLSBottomOfStack = 1 << 8; 31 | 32 | } // namespace 33 | 34 | Headers::Headers() { Reset(); } 35 | 36 | void Headers::Reset() { memset(this, 0, sizeof(*this)); } 37 | 38 | void Headers::Parse(StringPiece p) { 39 | Reset(); 40 | const char* start = p.data(); 41 | const char* limit = start + p.size(); 42 | uint16_t type = kTypeEthernet; 43 | uint16_t protocol = kProtocolUnknown; 44 | 45 | // We use a goto loop within this switch statement to strip all pre-IP-header 46 | // layers off of the given packet. 47 | pre_ip_encapsulation: 48 | switch (type) { 49 | case kTypeEthernet: { 50 | if (start + sizeof(struct ethhdr) > limit) { 51 | return; 52 | } 53 | eth = reinterpret_cast(start); 54 | start += sizeof(struct ethhdr); 55 | type = ntohs(eth->h_proto); 56 | goto pre_ip_encapsulation; 57 | } 58 | case ETH_P_8021Q: 59 | FALLTHROUGH_INTENDED; 60 | case ETH_P_8021AD: 61 | FALLTHROUGH_INTENDED; 62 | case ETH_P_QINQ1: 63 | FALLTHROUGH_INTENDED; 64 | case ETH_P_QINQ2: 65 | FALLTHROUGH_INTENDED; 66 | case ETH_P_QINQ3: { 67 | if (start + 4 > limit) { 68 | return; 69 | } 70 | // AddVLAN(ntohs(*reinterpret_cast(start)) & 0x0FFF, 71 | // packet_offset); 72 | type = ntohs(*reinterpret_cast(start + 2)); 73 | start += 4; 74 | goto pre_ip_encapsulation; 75 | } 76 | case ETH_P_MPLS_UC: 77 | FALLTHROUGH_INTENDED; 78 | case ETH_P_MPLS_MC: { 79 | uint32_t mpls_header = 0; 80 | do { 81 | // We check for 5 bytes, because we need to parse the first nibble after 82 | // the MPLS header to figure out the next layer type. 83 | if (start + 5 > limit) { 84 | return; 85 | } 86 | mpls_header = ntohl(*reinterpret_cast(start)); 87 | // AddMPLS(mpls_header >> 12, packet_offset); 88 | start += 4; 89 | } while (!(mpls_header & kMPLSBottomOfStack)); 90 | // Use the first nibble after the last MPLS layer to determine the 91 | // underlying packet type. 92 | switch (start[0] >> 4) { 93 | case 0: // RFC4385 94 | type = kTypeEthernet; 95 | start += 4; // Skip over PW ethernet control word. 96 | break; 97 | case 4: 98 | type = ETH_P_IP; 99 | break; 100 | case 6: 101 | type = ETH_P_IPV6; 102 | break; 103 | default: 104 | return; 105 | } 106 | goto pre_ip_encapsulation; 107 | } 108 | 109 | // All of the above use the pre_ip_encapsulation loop. 110 | // All of the below do not. 111 | 112 | case ETH_P_IP: { 113 | if (start + sizeof(struct iphdr) > limit) { 114 | return; 115 | } 116 | ip4 = reinterpret_cast(start); 117 | size_t len = ip4->ihl; 118 | len *= 4; 119 | if (len < 20) return; 120 | protocol = ip4->protocol; 121 | start += len; 122 | break; 123 | } 124 | case ETH_P_IPV6: { 125 | if (start + sizeof(struct ip6_hdr) > limit) { 126 | return; 127 | } 128 | ip6 = reinterpret_cast(start); 129 | protocol = ip6->ip6_ctlun.ip6_un1.ip6_un1_nxt; 130 | start += sizeof(struct ip6_hdr); 131 | 132 | // Here, we use another goto loop to strip off all IPv6 extensions. 133 | ip6_extensions: 134 | switch (protocol) { 135 | case IPPROTO_FRAGMENT: { 136 | if (start + sizeof(struct ip6_frag) > limit) { 137 | return; 138 | } 139 | ip6frag = reinterpret_cast(start); 140 | if (ntohs(ip6frag->ip6f_offlg) & 0xfff8) { 141 | // If we're not the first fragment, break out of the loop so we 142 | // can store the IPs we have but recognize in the protocol switch 143 | // later on that we don't know what this packet is. 144 | break; 145 | } 146 | // otherwise, fall through to treating this like any other 147 | // extension. 148 | FALLTHROUGH_INTENDED; 149 | } 150 | #ifdef IPPROTO_MH 151 | case IPPROTO_MH: 152 | FALLTHROUGH_INTENDED; 153 | #endif 154 | case IPPROTO_HOPOPTS: 155 | FALLTHROUGH_INTENDED; 156 | case IPPROTO_ROUTING: 157 | FALLTHROUGH_INTENDED; 158 | case IPPROTO_DSTOPTS: { 159 | if (start + sizeof(struct ip6_ext) > limit) { 160 | return; 161 | } 162 | auto ip6ext = reinterpret_cast(start); 163 | protocol = ip6ext->ip6e_nxt; 164 | start += (ip6ext->ip6e_len + 1) * 8; 165 | goto ip6_extensions; 166 | } 167 | } 168 | break; 169 | } 170 | default: 171 | return; 172 | } 173 | 174 | #define LAYER_4_PROTOCOL(typ, name, strct) \ 175 | case typ: { \ 176 | if (start + sizeof(struct strct) > limit) { \ 177 | return; \ 178 | } \ 179 | name = reinterpret_cast(start); \ 180 | start += sizeof(struct strct); \ 181 | break; \ 182 | } 183 | switch (protocol) { 184 | LAYER_4_PROTOCOL(IPPROTO_TCP, tcp, tcphdr) 185 | LAYER_4_PROTOCOL(IPPROTO_UDP, udp, udphdr) 186 | LAYER_4_PROTOCOL(IPPROTO_ICMP, icmp4, icmphdr) 187 | LAYER_4_PROTOCOL(IPPROTO_ICMPV6, icmp6, icmp6_hdr) 188 | default: 189 | return; 190 | } 191 | #undef LAYER_4_PROTOCOL 192 | } 193 | 194 | } // namespace clerk 195 | -------------------------------------------------------------------------------- /headers.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_HEADERS_H_ 16 | #define CLERK_HEADERS_H_ 17 | 18 | #include // ethhdr 19 | #define __FAVOR_BSD 20 | #include // tcphdr 21 | #undef __FAVOR_BSD 22 | #include // icmp6_hdr 23 | #include // iphdr 24 | #include // ip6_hdr 25 | #include // icmphdr 26 | #include // udphdr 27 | 28 | #include "util.h" 29 | #include "stringpiece.h" 30 | 31 | namespace clerk { 32 | 33 | struct Headers { 34 | Headers(); 35 | 36 | void Reset(); 37 | 38 | // Parse the given packet data, setting the found header pointers in this 39 | // struct. Note that initially we expect to find an ethernet header first, 40 | // other link types are not yet supported (though they could be later on). 41 | void Parse(StringPiece p); 42 | 43 | // These pointers are reset to NULL on every call to Parse, then those 44 | // associated with the packet are set. 45 | 46 | // Layer 2 47 | const struct ethhdr* eth; 48 | 49 | // Layer 3 50 | const struct iphdr* ip4; 51 | const struct ip6_hdr* ip6; 52 | 53 | // Layer 4 54 | const struct tcphdr* tcp; 55 | const struct udphdr* udp; 56 | const struct icmphdr* icmp4; 57 | const struct icmp6_hdr* icmp6; 58 | 59 | // Other metadata 60 | const struct ip6_frag* ip6frag; 61 | }; 62 | 63 | } // namespace clerk 64 | 65 | #endif // CLERK_HEADERS_H_ 66 | -------------------------------------------------------------------------------- /headers_test.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | 17 | #include "headers.h" 18 | #include "stringpiece.h" 19 | #include "util.h" 20 | 21 | namespace clerk { 22 | 23 | class HeadersTest : public ::testing::Test {}; 24 | 25 | const char ip6tcp[] = { 26 | 0x24, 0xbe, 0x05, 0x27, 0x0b, 0x17, 0x44, 0xf4, 0x77, 0x95, 0xd0, 27 | 0xc1, 0x86, 0xdd, 0x66, 0x00, 0x00, 0x00, 0x00, 0x20, 0x06, 0x3c, 28 | 0x26, 0x07, 0xf8, 0xb0, 0x40, 0x0f, 0x08, 0x02, 0x00, 0x00, 0x00, 29 | 0x00, 0x00, 0x00, 0x20, 0x05, 0x26, 0x20, 0x00, 0x00, 0x10, 0x05, 30 | 0x00, 0x12, 0xb5, 0x62, 0x47, 0xd0, 0x45, 0xdd, 0x9f, 0xf6, 0x01, 31 | 0xbb, 0xb5, 0xea, 0x97, 0xb3, 0xf0, 0xde, 0xed, 0xc2, 0xf5, 0x9e, 32 | 0x80, 0x10, 0x00, 0xec, 0x35, 0x11, 0x00, 0x00, 0x01, 0x01, 0x08, 33 | 0x0a, 0x19, 0xab, 0x61, 0x81, 0x23, 0x11, 0xde, 0xdc, 34 | }; 35 | 36 | const char ip4udp[] = { 37 | 0x00, 0x00, 0x5e, 0x00, 0x01, 0xca, 0x24, 0xbe, 0x05, 0x27, 0x0b, 0x17, 38 | 0x08, 0x00, 0x45, 0x00, 0x00, 0x41, 0x92, 0xa0, 0x40, 0x00, 0x40, 0x11, 39 | 0xce, 0x4b, 0x64, 0x64, 0xca, 0x49, 0xac, 0x10, 0xff, 0x01, 0x10, 0x12, 40 | 0x00, 0x35, 0x00, 0x2d, 0xd9, 0xfe, 0x14, 0x1a, 0x01, 0x00, 0x00, 0x01, 41 | 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x04, 0x78, 0x6b, 0x63, 0x64, 0x03, 42 | 0x63, 0x6f, 0x6d, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x29, 0x10, 43 | 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 44 | }; 45 | 46 | #define POINTERS_EQUAL(a, b) \ 47 | EXPECT_EQ(reinterpret_cast(a), reinterpret_cast(b)) 48 | 49 | TEST_F(HeadersTest, TestEquality) { 50 | Headers h; 51 | h.Parse(StringPiece(ip6tcp, sizeof(ip6tcp))); 52 | POINTERS_EQUAL(h.eth, ip6tcp + 0); 53 | POINTERS_EQUAL(h.ip4, nullptr); 54 | POINTERS_EQUAL(h.ip6, ip6tcp + 14); 55 | POINTERS_EQUAL(h.tcp, ip6tcp + 54); 56 | POINTERS_EQUAL(h.udp, nullptr); 57 | POINTERS_EQUAL(h.icmp4, nullptr); 58 | POINTERS_EQUAL(h.icmp6, nullptr); 59 | POINTERS_EQUAL(h.ip6frag, nullptr); 60 | h.Parse(StringPiece(ip4udp, sizeof(ip4udp))); 61 | POINTERS_EQUAL(h.eth, ip4udp + 0); 62 | POINTERS_EQUAL(h.ip4, ip4udp + 14); 63 | POINTERS_EQUAL(h.ip6, nullptr); 64 | POINTERS_EQUAL(h.tcp, nullptr); 65 | POINTERS_EQUAL(h.udp, ip4udp + 34); 66 | POINTERS_EQUAL(h.icmp4, nullptr); 67 | POINTERS_EQUAL(h.icmp6, nullptr); 68 | POINTERS_EQUAL(h.ip6frag, nullptr); 69 | h.Reset(); 70 | POINTERS_EQUAL(h.eth, nullptr); 71 | POINTERS_EQUAL(h.ip4, nullptr); 72 | POINTERS_EQUAL(h.ip6, nullptr); 73 | POINTERS_EQUAL(h.tcp, nullptr); 74 | POINTERS_EQUAL(h.udp, nullptr); 75 | POINTERS_EQUAL(h.icmp4, nullptr); 76 | POINTERS_EQUAL(h.icmp6, nullptr); 77 | POINTERS_EQUAL(h.ip6frag, nullptr); 78 | } 79 | 80 | TEST_F(HeadersTest, ICMP4) { 81 | const char ip4icmp[] = { 82 | 0x24, 0xbe, 0x05, 0x27, 0x0b, 0x17, 0x44, 0xf4, 0x77, 0x95, 0xd0, 83 | 0xc1, 0x08, 0x00, 0x45, 0x80, 0x00, 0x54, 0x00, 0x00, 0x00, 0x00, 84 | 0x34, 0x01, 0x47, 0x6c, 0x08, 0x08, 0x08, 0x08, 0x64, 0x64, 0xca, 85 | 0x49, 0x00, 0x00, 0x1e, 0x71, 0x30, 0x40, 0x00, 0x02, 0x37, 0x66, 86 | 0x32, 0x57, 0x00, 0x00, 0x00, 0x00, 0x85, 0xbc, 0x03, 0x00, 0x00, 87 | 0x00, 0x00, 0x00, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 88 | 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x21, 0x22, 89 | 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 90 | 0x2e, 0x2f, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 91 | }; 92 | Headers h; 93 | h.Parse(StringPiece(ip4icmp, sizeof(ip4icmp))); 94 | POINTERS_EQUAL(h.eth, ip4icmp + 0); 95 | POINTERS_EQUAL(h.ip4, ip4icmp + 14); 96 | POINTERS_EQUAL(h.ip6, nullptr); 97 | POINTERS_EQUAL(h.tcp, nullptr); 98 | POINTERS_EQUAL(h.udp, nullptr); 99 | POINTERS_EQUAL(h.icmp4, ip4icmp + 34); 100 | POINTERS_EQUAL(h.icmp6, nullptr); 101 | POINTERS_EQUAL(h.ip6frag, nullptr); 102 | } 103 | 104 | TEST_F(HeadersTest, ICMP6) { 105 | const char ip6icmp[] = { 106 | 0x00, 0x00, 0x86, 0x05, 0x80, 0xda, 0x00, 0x60, 0x97, 0x07, 0x69, 0xea, 107 | 0x86, 0xdd, 0x60, 0x00, 0x00, 0x00, 0x00, 0x44, 0x3a, 0x40, 0x3f, 0xfe, 108 | 0x05, 0x07, 0x00, 0x00, 0x00, 0x01, 0x02, 0x60, 0x97, 0xff, 0xfe, 0x07, 109 | 0x69, 0xea, 0x3f, 0xfe, 0x05, 0x07, 0x00, 0x00, 0x00, 0x01, 0x02, 0x00, 110 | 0x86, 0xff, 0xfe, 0x05, 0x80, 0xda, 0x03, 0x00, 0xf7, 0x52, 0x00, 0x00, 111 | 0x00, 0x00, 0x60, 0x00, 0x00, 0x00, 0x00, 0x14, 0x11, 0x01, 0x3f, 0xfe, 112 | 0x05, 0x07, 0x00, 0x00, 0x00, 0x01, 0x02, 0x00, 0x86, 0xff, 0xfe, 0x05, 113 | 0x80, 0xda, 0x3f, 0xfe, 0x05, 0x01, 0x04, 0x10, 0x00, 0x00, 0x02, 0xc0, 114 | 0xdf, 0xff, 0xfe, 0x47, 0x03, 0x3e, 0xa0, 0x75, 0x82, 0x9c, 0x00, 0x14, 115 | 0xec, 0x4b, 0x02, 0x01, 0x00, 0x00, 0xf9, 0xc8, 0xe7, 0x36, 0x8a, 0x2c, 116 | 0x09, 0x00, 117 | }; 118 | Headers h; 119 | h.Parse(StringPiece(ip6icmp, sizeof(ip6icmp))); 120 | POINTERS_EQUAL(h.eth, ip6icmp + 0); 121 | POINTERS_EQUAL(h.ip4, nullptr); 122 | POINTERS_EQUAL(h.ip6, ip6icmp + 14); 123 | POINTERS_EQUAL(h.tcp, nullptr); 124 | POINTERS_EQUAL(h.udp, nullptr); 125 | POINTERS_EQUAL(h.icmp4, nullptr); 126 | POINTERS_EQUAL(h.icmp6, ip6icmp + 54); 127 | POINTERS_EQUAL(h.ip6frag, nullptr); 128 | } 129 | 130 | } // namespace clerk 131 | -------------------------------------------------------------------------------- /ipfix.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include // inet_ntop 16 | #include // INET6_ADDRSTRLEN 17 | 18 | #include "ipfix.h" 19 | #include "flow.h" 20 | #include "send.h" 21 | #include "util.h" 22 | 23 | #include 24 | 25 | namespace clerk { 26 | 27 | IPFIX::IPFIX(const IPFIX* other, const IPFIXFactory* f) : factory_(f) { 28 | CHECK(f != nullptr); 29 | if (other) { 30 | flows_ = other->flows_; 31 | for (auto iter = flows_.begin(); iter != flows_.end(); ) { 32 | if (iter->second.Finished(factory_->CutoffNanos()) == 33 | flow::Stats::ACTIVE_TIMEOUT) { 34 | iter->second.packets = 0; 35 | iter->second.bytes = 0; 36 | iter->second.tcp_flags = 0; 37 | ++iter; 38 | } else { 39 | iter = flows_.erase(iter); 40 | } 41 | } 42 | // Should the number of flows we maintain shrink a lot, we'd like our memory 43 | // usage to decrease. However, we don't want to have to 44 | // shrink/grow/shrink/grow our maps constantly. So, we call 'reserve' on 45 | // our new map to shrink it to the size of our old map. If the flows we 46 | // need in memory decrease, this will decrease the map's bucket size (which 47 | // otherwise was copied over via the copy constructor and will _not_ 48 | // decrease ever). However, since it does it based on the old size, we have 49 | // a bit of a buffer for size decreases. 50 | flows_.reserve(other->flows_.size()); 51 | LOG(INFO) << "Retained " << flows_.size() << " from previous in " 52 | << flows_.bucket_count() << " buckets"; 53 | } 54 | } 55 | 56 | void IPFIX::Process(const Packet& p) { 57 | flow::Key key; 58 | flow::Stats stats(p.hdr()->tp_len, 1, p.ts_nanos()); 59 | 60 | // Layer 2-ish 61 | if (p.hdr()->tp_status & TP_STATUS_VLAN_VALID) { 62 | key.vlan = p.hdr()->hv1.tp_vlan_tci; 63 | } 64 | 65 | // Layer 3 66 | auto h = p.headers(); 67 | if (h.ip4) { 68 | key.set_src_ip4(ntohl(h.ip4->saddr)); 69 | key.set_dst_ip4(ntohl(h.ip4->daddr)); 70 | key.protocol = h.ip4->protocol; 71 | key.network = 4; 72 | key.tos = h.ip4->tos >> 2; 73 | } else if (h.ip6) { 74 | key.protocol = h.ip6->ip6_ctlun.ip6_un1.ip6_un1_nxt; 75 | key.set_src_ip6(reinterpret_cast(&h.ip6->ip6_src)); 76 | key.set_dst_ip6(reinterpret_cast(&h.ip6->ip6_dst)); 77 | key.network = 6; 78 | key.tos = (h.ip6->ip6_flow & 0x0FC00000) >> 22; 79 | } 80 | 81 | // Layer 4 82 | if (h.tcp) { 83 | key.src_port = ntohs(h.tcp->th_sport); 84 | key.dst_port = ntohs(h.tcp->th_dport); 85 | stats.tcp_flags = h.tcp->th_flags; 86 | } else if (h.udp) { 87 | key.src_port = ntohs(h.udp->source); 88 | key.dst_port = ntohs(h.udp->dest); 89 | } else if (h.icmp4) { 90 | key.icmp_type = h.icmp4->type; 91 | key.icmp_code = h.icmp4->code; 92 | } else if (h.icmp6) { 93 | key.icmp_type = h.icmp6->icmp6_type; 94 | key.icmp_code = h.icmp6->icmp6_code; 95 | } 96 | 97 | flow::AddToTable(&flows_, key, stats); 98 | return; 99 | } 100 | 101 | void PacketSender::Send(const flow::Table& flows) { 102 | uint32_t unix_secs = GetCurrentTimeNanos() / kNumNanosPerSecond; 103 | LOG(INFO) << "FLUSHING " << flows.size() << " to " << fd_; 104 | ipfix::IPFIXPacket pkt(unix_secs); 105 | 106 | // Write IPv4 template and packets. 107 | LOG(INFO) << "Writing IPv4 template"; 108 | pkt.Reset(ipfix::PT_TEMPLATE, seq_); 109 | pkt.WriteFlowSet(true); 110 | pkt.SendTo(fd_); 111 | 112 | pkt.Reset(ipfix::PT_V4, seq_); 113 | int ip4count = 0; 114 | for (auto iter : flows) { 115 | auto end_reason = iter.second.Finished(factory_->CutoffNanos()); 116 | if (iter.first.network == 4 && 117 | (iter.second.packets > 0 || 118 | end_reason != flow::Stats::ACTIVE_TIMEOUT)) { 119 | ip4count++; 120 | seq_++; 121 | if (pkt.AddToBuffer(iter.first, iter.second, end_reason)) { 122 | pkt.SendTo(fd_); 123 | pkt.Reset(ipfix::PT_V4, seq_); 124 | } 125 | } 126 | } 127 | if (pkt.count()) { 128 | pkt.SendTo(fd_); 129 | } 130 | LOG(INFO) << "Wrote IPv4: " << ip4count; 131 | 132 | // Write IPv6 template and packets. 133 | LOG(INFO) << "Writing IPv6 template"; 134 | pkt.Reset(ipfix::PT_TEMPLATE, seq_); 135 | pkt.WriteFlowSet(false); 136 | pkt.SendTo(fd_); 137 | 138 | pkt.Reset(ipfix::PT_V6, seq_); 139 | int ip6count = 0; 140 | for (auto iter : flows) { 141 | auto end_reason = iter.second.Finished(factory_->CutoffNanos()); 142 | if (iter.first.network == 6 && 143 | (iter.second.packets > 0 || 144 | end_reason != flow::Stats::ACTIVE_TIMEOUT)) { 145 | ip6count++; 146 | seq_++; 147 | if (pkt.AddToBuffer(iter.first, iter.second, end_reason)) { 148 | pkt.SendTo(fd_); 149 | pkt.Reset(ipfix::PT_V6, seq_); 150 | } 151 | } 152 | } 153 | if (pkt.count()) { 154 | pkt.SendTo(fd_); 155 | } 156 | LOG(INFO) << "Wrote IPv6: " << ip6count; 157 | } 158 | 159 | static void WriteIPToBuffer(char* buf, int n, const uint8_t* ip, bool v4) { 160 | inet_ntop(v4 ? AF_INET : AF_INET6, ip + (v4 ? 12 : 0), buf, n); 161 | } 162 | 163 | void FileSender::Send(const flow::Table& flows) { 164 | char src_ip_buf[INET6_ADDRSTRLEN]; 165 | char dst_ip_buf[INET6_ADDRSTRLEN]; 166 | fprintf(f_, 167 | "FlowStart,FlowEnd,SrcIP,DstIP,SrcPort,DstPort,VLAN,TOS,Protocol," 168 | "ICMPType,ICMPCode,Bytes,Packets\n"); 169 | for (const auto& iter : flows) { 170 | auto end_reason = iter.second.Finished(factory_->CutoffNanos()); 171 | auto key = iter.first; 172 | auto stats = iter.second; 173 | if (stats.packets > 0 || end_reason != flow::Stats::ACTIVE_TIMEOUT) { 174 | WriteIPToBuffer(src_ip_buf, sizeof(src_ip_buf), key.src_ip, 175 | key.network == 4); 176 | WriteIPToBuffer(dst_ip_buf, sizeof(src_ip_buf), key.dst_ip, 177 | key.network == 4); 178 | fprintf(f_, "%.9Lf,%.9Lf,%s,%s,%d,%d,%d,%d,%d,%d,%d,%lu,%lu,%d\n", 179 | stats.first_ns * 1.0L / kNumNanosPerSecond, 180 | stats.last_ns * 1.0L / kNumNanosPerSecond, src_ip_buf, dst_ip_buf, 181 | key.src_port, key.dst_port, key.vlan, key.tos, key.protocol, 182 | key.icmp_type, key.icmp_code, stats.bytes, stats.packets, 183 | end_reason); 184 | } 185 | } 186 | fflush(f_); 187 | } 188 | 189 | void IPFIX::operator+=(const IPFIX& other) { 190 | LOG(INFO) << "Adding " << other.flows_.size() << " flows into " 191 | << flows_.size(); 192 | flow::CombineTable(&flows_, other.flows_); 193 | } 194 | 195 | } // namespace clerk 196 | -------------------------------------------------------------------------------- /ipfix.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_IPFIX_H_ 16 | #define CLERK_IPFIX_H_ 17 | 18 | #include "flow.h" 19 | #include "testimony.h" 20 | 21 | namespace clerk { 22 | 23 | class IPFIXFactory; 24 | 25 | class Sender { 26 | public: 27 | Sender() {} 28 | virtual ~Sender() {} 29 | virtual void Send(const flow::Table& flows) = 0; 30 | }; 31 | 32 | class PacketSender : public Sender { 33 | public: 34 | PacketSender(int sock_fd, const IPFIXFactory* fact) 35 | : factory_(fact), fd_(sock_fd), seq_(0) {} 36 | ~PacketSender() override {} 37 | 38 | void Send(const flow::Table& flows) override; 39 | 40 | private: 41 | const IPFIXFactory* factory_; 42 | int fd_; 43 | uint32_t seq_; 44 | }; 45 | 46 | class FileSender : public Sender { 47 | public: 48 | explicit FileSender(FILE* f, const IPFIXFactory* fact) 49 | : factory_(fact), f_(f) {} 50 | ~FileSender() override {} 51 | 52 | void Send(const flow::Table& flows) override; 53 | 54 | private: 55 | const IPFIXFactory* factory_; 56 | FILE* f_; 57 | }; 58 | 59 | // IPFIX gathers IPFIX statistics about network flows, then provides a method 60 | // (SendTo) to send them via UDP over a network socket. 61 | class IPFIX : public State { 62 | public: 63 | // Create a new IPFIX. If 'old' is non-null, it contains the previous state 64 | // for this thread, which we aggregate into the new state. 65 | IPFIX(const IPFIX* old, const IPFIXFactory* f); 66 | ~IPFIX() override {} 67 | 68 | // Process implements clerk::State by updating our flow table. 69 | void Process(const Packet& p) override; 70 | // += aggregates multiple IPFIX states together. 71 | void operator+=(const IPFIX& other); 72 | 73 | void SwapFlows(flow::Table* f) { f->swap(flows_); } 74 | 75 | private: 76 | flow::Table flows_; 77 | const IPFIXFactory* factory_; 78 | 79 | DISALLOW_COPY_AND_ASSIGN(IPFIX); 80 | }; 81 | 82 | class IPFIXFactory : public StateFactory { 83 | public: 84 | IPFIXFactory() : flow_timeout_cutoff_ns_(0) {} 85 | ~IPFIXFactory() override {} 86 | 87 | std::unique_ptr New(const State* old) const override { 88 | return std::unique_ptr( 89 | new IPFIX(reinterpret_cast(old), this)); 90 | } 91 | void SetCutoffNanos(uint64_t ms) { flow_timeout_cutoff_ns_ = ms; } 92 | uint64_t CutoffNanos() const { return flow_timeout_cutoff_ns_; } 93 | 94 | private: 95 | uint64_t flow_timeout_cutoff_ns_; 96 | }; 97 | 98 | } // namespace clerk 99 | 100 | #endif // CLERK_IPFIX_H_ 101 | -------------------------------------------------------------------------------- /send.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "send.h" 16 | 17 | #include 18 | #include 19 | #include "stringpiece.h" 20 | 21 | #include "util.h" 22 | 23 | namespace clerk { 24 | namespace ipfix { 25 | 26 | static inline void WriteChars(char** buffer, uint8_t a, uint8_t b, uint8_t c, 27 | uint8_t d) { 28 | char* buf = *buffer; 29 | buf[0] = a; 30 | buf[1] = b; 31 | buf[2] = c; 32 | buf[3] = d; 33 | *buffer += 4; 34 | } 35 | 36 | static inline void WriteBE32(char** buffer, uint32_t data) { 37 | WriteChars(buffer, data >> 24, data >> 16, data >> 8, data); 38 | } 39 | 40 | static inline void WriteBE16s(char** buffer, uint16_t first, uint16_t second) { 41 | WriteChars(buffer, first >> 8, first, second >> 8, second); 42 | } 43 | 44 | static inline void WriteBE64(char** buffer, uint64_t v) { 45 | WriteBE32(buffer, v >> 32); 46 | WriteBE32(buffer, v & 0xFFFFFFFF); 47 | } 48 | 49 | IPFIXPacket::IPFIXPacket(uint32_t unix_secs) : unix_secs_(unix_secs) {} 50 | 51 | void IPFIXPacket::Reset(PacketType t, uint32_t seq) { 52 | count_ = 0; 53 | type_ = t; 54 | memset(buffer_, 0, kMaxPacketSize); 55 | start_ = &buffer_[0]; 56 | current_ = &buffer_[0]; 57 | limit_ = start_ + kMaxPacketSize; 58 | 59 | CHECK_LE(current_ + kHeaderSize, limit_); 60 | char* want = current_ + kHeaderSize; 61 | WriteBE16s(¤t_, 0xffff, 0xffff); // Will rewrite in SendTo. 62 | WriteBE32(¤t_, unix_secs_); 63 | WriteBE32(¤t_, seq); 64 | WriteBE32(¤t_, 12345); // Source ID 65 | record_buf_ = current_; 66 | WriteBE16s(¤t_, 0xffff, 0xffff); // Will rewrite in SendTo. 67 | CHECK_EQ(current_, want) << "diff: " << current_ - buffer_; 68 | } 69 | StringPiece IPFIXPacket::PacketData() { 70 | CHECK(record_buf_ != nullptr); 71 | WriteBE16s(&record_buf_, type_, current_ - record_buf_); 72 | char* first = start_; 73 | WriteBE16s(&first, 10, current_ - start_); 74 | return StringPiece(buffer_, current_ - buffer_); 75 | } 76 | 77 | void IPFIXPacket::SendTo(int sock_fd) { 78 | auto data = PacketData(); 79 | if (send(sock_fd, data.data(), data.size(), 0) < 0) { 80 | PLOG(ERROR) << "Sending packet to socket failed"; 81 | } 82 | } 83 | 84 | int IPFIXPacket::count() const { return count_; } 85 | 86 | bool IPFIXPacket::AddToBuffer(const flow::Key& k, const flow::Stats& f, 87 | uint8_t end_reason) { 88 | CHECK_LE(current_ + kSingleRecordSize, limit_); 89 | char* want = current_ + kSingleRecordSize; 90 | count_++; 91 | switch (type_) { 92 | case ipfix::PT_V4: 93 | CHECK_EQ(k.network, 4); 94 | WriteBE32(¤t_, k.get_src_ip4()); 95 | WriteBE32(¤t_, k.get_dst_ip4()); 96 | break; 97 | case ipfix::PT_V6: 98 | CHECK_EQ(k.network, 6); 99 | memcpy(current_, k.src_ip, 16); 100 | current_ += 16; 101 | memcpy(current_, k.dst_ip, 16); 102 | current_ += 16; 103 | break; 104 | case ipfix::PT_TEMPLATE: 105 | LOG(FATAL) << "Adding to template"; 106 | default: 107 | LOG(FATAL) << "Bad packet type " << type_; 108 | } 109 | WriteBE16s(¤t_, k.src_port, k.dst_port); 110 | WriteChars(¤t_, k.protocol, f.tcp_flags, k.icmp_type, k.icmp_code); 111 | WriteBE32(¤t_, f.src_asn); 112 | WriteBE32(¤t_, f.dst_asn); 113 | WriteBE64(¤t_, f.bytes); 114 | WriteBE64(¤t_, f.packets); 115 | // Note that even though we have nanoseconds, we write out milliseconds. This 116 | // is because IPFIX says that micros/nanos should be in stupid NTP format 117 | // (https://tools.ietf.org/html/rfc5905#section-6) and I'm too lazy to compute 118 | // it. 119 | WriteBE64(¤t_, f.first_ns / kNumNanosPerMilli); 120 | WriteBE64(¤t_, f.last_ns / kNumNanosPerMilli); 121 | WriteChars(¤t_, k.tos, end_reason, k.vlan >> 8, k.vlan); 122 | CHECK_LE(current_, want); 123 | return current_ + kSingleRecordSize >= limit_; 124 | } 125 | 126 | void IPFIXPacket::WriteFlowSet(bool v4) { 127 | count_++; 128 | CHECK_EQ(type_, ipfix::PT_TEMPLATE); 129 | CHECK_LE(current_ + kFlowSetSize, limit_); 130 | char* want = current_ + kFlowSetSize; 131 | WriteBE16s(¤t_, v4 ? ipfix::PT_V4 : ipfix::PT_V6, 132 | kFieldCount); // template ID, field count 133 | if (v4) { 134 | WriteBE16s(¤t_, IPV4_SRC_ADDR, 4); 135 | WriteBE16s(¤t_, IPV4_DST_ADDR, 4); 136 | } else { 137 | WriteBE16s(¤t_, IPV6_SRC_ADDR, 16); 138 | WriteBE16s(¤t_, IPV6_DST_ADDR, 16); 139 | } 140 | WriteBE16s(¤t_, L4_SRC_PORT, 2); 141 | WriteBE16s(¤t_, L4_DST_PORT, 2); 142 | WriteBE16s(¤t_, PROTOCOL, 1); 143 | WriteBE16s(¤t_, TCP_FLAGS, 1); 144 | WriteBE16s(¤t_, ICMP_TYPE, 2); 145 | WriteBE16s(¤t_, BGP_SOURCE_AS_NUMBER, 4); 146 | WriteBE16s(¤t_, BGP_DESTINATION_AS_NUMBER, 4); 147 | WriteBE16s(¤t_, IN_BYTES, 8); 148 | WriteBE16s(¤t_, IN_PKTS, 8); 149 | WriteBE16s(¤t_, FLOW_START_MILLISECONDS, 8); 150 | WriteBE16s(¤t_, FLOW_END_MILLISECONDS, 8); 151 | WriteBE16s(¤t_, IP_CLASS_OF_SERVICE, 1); 152 | WriteBE16s(¤t_, FLOW_END_REASON, 1); 153 | WriteBE16s(¤t_, VLAN_ID, 2); 154 | CHECK_EQ(current_, want); 155 | } 156 | 157 | } // namespace ipfix 158 | } // namespace clerk 159 | -------------------------------------------------------------------------------- /send.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_SEND_H_ 16 | #define CLERK_SEND_H_ 17 | 18 | #include // uint32_t, etc. 19 | #include // size_t 20 | 21 | #include "flow.h" 22 | #include "util.h" 23 | #include "stringpiece.h" 24 | 25 | namespace clerk { 26 | namespace ipfix { 27 | 28 | const size_t kMaxPacketSize = 1400; 29 | const size_t kSingleRecordSize = 30 | 16 + 16 + // IPv6 addresses for ipv6. IPv4 is 4+4, so this overestimates 31 | // in that case. 32 | 2 + 2 + // Ports 33 | 1 + // Protocol 34 | 1 + // TCP flags 35 | 2 + // ICMP code 36 | 4 + // Src AS 37 | 4 + // Dst AS 38 | 8 + // Bytes 39 | 8 + // Packets 40 | 8 + // First millis since epoch, uint64 41 | 8 + // Last millis since epoch, uint64 42 | 1 + // IP TOS 43 | 1 + // Flow end reason 44 | 2 + // VLAN ID 45 | 0; 46 | const uint16_t kFieldCount = 16; 47 | const size_t kHeaderSize = 20; 48 | const size_t kFlowSetSize = 2 * 2 + // Template ID, field count 49 | kFieldCount * 4; // fields' type/length 50 | 51 | // Pulled from http://www.ietf.org/rfc/rfc3954.txt 52 | enum IpfixTypes { 53 | IN_BYTES = 1, 54 | IN_PKTS = 2, 55 | PROTOCOL = 4, 56 | IP_CLASS_OF_SERVICE = 5, 57 | TCP_FLAGS = 6, 58 | L4_SRC_PORT = 7, 59 | IPV4_SRC_ADDR = 8, 60 | L4_DST_PORT = 11, 61 | IPV4_DST_ADDR = 12, 62 | BGP_SOURCE_AS_NUMBER = 16, 63 | BGP_DESTINATION_AS_NUMBER = 17, 64 | IPV6_SRC_ADDR = 27, 65 | IPV6_DST_ADDR = 28, 66 | ICMP_TYPE = 32, 67 | VLAN_ID = 58, 68 | FLOW_END_REASON = 136, 69 | FLOW_START_MILLISECONDS = 152, 70 | FLOW_END_MILLISECONDS = 153, 71 | }; 72 | 73 | enum PacketType { 74 | PT_V4 = 256, 75 | PT_V6 = 257, 76 | PT_TEMPLATE = 2, 77 | }; 78 | 79 | // IPFIXPacket is a helper to build an IPFIX (netflow v10) packet to send over 80 | // the network. It's a little tricky, so read all the fine print... or just use 81 | // IPFIX::Send instead ;) 82 | class IPFIXPacket { 83 | public: 84 | // Creates a new packet with the given uptime and current time. 85 | explicit IPFIXPacket(uint32_t unix_secs); 86 | 87 | // Reset this pcket to a packet type. If that packet type is PT_TEMPLATE, the 88 | // packet is immediately sendable, and AddToBuffer will CHECK-fail. 89 | // Otherwise, AddToBuffer must be called before SendTo. 90 | void Reset(PacketType t, uint32_t seq); 91 | // Get the packet data to send. 92 | StringPiece PacketData(); 93 | // Send packet data to socket. 94 | void SendTo(int sock_fd); 95 | // Number of entries added to the buffer. 96 | int count() const; 97 | // AddToBuffer adds the given key/flow to the packet. If the packet is full 98 | // and must be immediately sent, returns true. 99 | bool AddToBuffer(const flow::Key& k, const flow::Stats& f, 100 | uint8_t end_reason); 101 | 102 | // Writes the template for v4 or v6 to the packet. Should be called only once 103 | // on a single packet, packet type must be PT_TEMPLATE. 104 | void WriteFlowSet(bool v4); 105 | 106 | private: 107 | char buffer_[kMaxPacketSize]; 108 | char* start_; 109 | char* record_buf_; 110 | char* current_; 111 | char* limit_; 112 | uint16_t count_; 113 | PacketType type_; 114 | uint32_t unix_secs_; 115 | }; 116 | 117 | } // namespace ipfix 118 | } // namespace clerk 119 | 120 | #endif // CLERK_SEND_H_ 121 | -------------------------------------------------------------------------------- /send_test.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "send.h" 16 | #include "flow.h" 17 | #include "util.h" 18 | #include "stringpiece.h" 19 | 20 | #include 21 | #include 22 | 23 | namespace clerk { 24 | namespace ipfix { 25 | 26 | class SendTest : public ::testing::Test {}; 27 | 28 | void PrintPacket(StringPiece p) { 29 | if (VLOG_IS_ON(1)) { 30 | printf("const char packet[] = {"); 31 | for (size_t i = 0; i < p.size(); i++) { 32 | if (i % 8 == 0) { 33 | printf("\n\t"); 34 | } 35 | printf("0x%02X, ", uint8_t(p[i])); 36 | } 37 | printf("\n};\n"); 38 | } 39 | } 40 | 41 | TEST_F(SendTest, TemplateV4Packet) { 42 | const char want[] = { 43 | 0x00, 0x0A, 0x00, 0x58, 0x00, 0x00, 0x00, 0xDE, 0x00, 0x00, 0x00, 44 | 0x03, 0x00, 0x00, 0x30, 0x39, 0x00, 0x02, 0x00, 0x48, 0x01, 0x00, 45 | 0x00, 0x10, 0x00, 0x08, 0x00, 0x04, 0x00, 0x0C, 0x00, 0x04, 0x00, 46 | 0x07, 0x00, 0x02, 0x00, 0x0B, 0x00, 0x02, 0x00, 0x04, 0x00, 0x01, 47 | 0x00, 0x06, 0x00, 0x01, 0x00, 0x20, 0x00, 0x02, 0x00, 0x10, 0x00, 48 | 0x04, 0x00, 0x11, 0x00, 0x04, 0x00, 0x01, 0x00, 0x08, 0x00, 0x02, 49 | 0x00, 0x08, 0x00, 0x98, 0x00, 0x08, 0x00, 0x99, 0x00, 0x08, 0x00, 50 | 0x05, 0x00, 0x01, 0x00, 0x88, 0x00, 0x01, 0x00, 0x3A, 0x00, 0x02, 51 | }; 52 | IPFIXPacket p(222); 53 | p.Reset(PT_TEMPLATE, 3); 54 | p.WriteFlowSet(true); 55 | auto data = p.PacketData(); 56 | PrintPacket(data); 57 | ASSERT_EQ(data, StringPiece(want, sizeof(want))); 58 | } 59 | 60 | TEST_F(SendTest, TemplateV6Packet) { 61 | const char want[] = { 62 | 0x00, 0x0A, 0x00, 0x58, 0x00, 0x00, 0x00, 0xDE, 0x00, 0x00, 0x00, 63 | 0x03, 0x00, 0x00, 0x30, 0x39, 0x00, 0x02, 0x00, 0x48, 0x01, 0x01, 64 | 0x00, 0x10, 0x00, 0x1B, 0x00, 0x10, 0x00, 0x1C, 0x00, 0x10, 0x00, 65 | 0x07, 0x00, 0x02, 0x00, 0x0B, 0x00, 0x02, 0x00, 0x04, 0x00, 0x01, 66 | 0x00, 0x06, 0x00, 0x01, 0x00, 0x20, 0x00, 0x02, 0x00, 0x10, 0x00, 67 | 0x04, 0x00, 0x11, 0x00, 0x04, 0x00, 0x01, 0x00, 0x08, 0x00, 0x02, 68 | 0x00, 0x08, 0x00, 0x98, 0x00, 0x08, 0x00, 0x99, 0x00, 0x08, 0x00, 69 | 0x05, 0x00, 0x01, 0x00, 0x88, 0x00, 0x01, 0x00, 0x3A, 0x00, 0x02, 70 | }; 71 | IPFIXPacket p(222); 72 | p.Reset(PT_TEMPLATE, 3); 73 | p.WriteFlowSet(false); 74 | auto data = p.PacketData(); 75 | PrintPacket(data); 76 | ASSERT_EQ(data, StringPiece(want, sizeof(want))); 77 | } 78 | 79 | TEST_F(SendTest, DataV4Packet) { 80 | const char want[] = { 81 | // header 82 | 0x00, 0x0A, 0x00, 0x8C, 0x00, 0x00, 0x00, 0xDE, 0x00, 0x00, 0x00, 0x03, 83 | 0x00, 0x00, 0x30, 0x39, 84 | // record set 85 | 0x01, 0x00, 0x00, 0x7C, 86 | // record 1 87 | 0x11, 0x22, 0x33, 0x44, 0xAA, 0xBB, 0xCC, 0xDD, 0x99, 0x99, 0xAA, 0xAA, 88 | 0x00, 0x00, 0x55, 0x66, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 89 | 0x00, 0x00, 0x00, 0x77, 0x77, 0x77, 0x77, 0x77, 0x00, 0x00, 0x00, 0x88, 90 | 0x88, 0x88, 0x88, 0x88, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 91 | 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xEE, 0xAB, 0xFF, 0xFF, 92 | // record 2 93 | 0x11, 0x22, 0x33, 0x44, 0xAA, 0xBB, 0xCC, 0xDD, 0x99, 0x99, 0xAA, 0xAA, 94 | 0x00, 0x00, 0x55, 0x66, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 95 | 0x00, 0x00, 0x00, 0x77, 0x77, 0x77, 0x77, 0x77, 0x00, 0x00, 0x00, 0x88, 96 | 0x88, 0x88, 0x88, 0x88, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 97 | 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xEE, 0xCD, 0xFF, 0xFF, 98 | }; 99 | flow::Key k; 100 | flow::Stats s; 101 | k.set_src_ip4(0x11223344); 102 | k.set_dst_ip4(0xAABBCCDD); 103 | k.src_port = 0x9999; 104 | k.dst_port = 0xAAAA; 105 | k.tos = 0xEE; 106 | k.vlan = 0xFFFF; 107 | k.icmp_type = 0x55; 108 | k.icmp_code = 0x66; 109 | s.bytes = 0x7777777777LL; 110 | s.packets = 0x8888888888LL; 111 | IPFIXPacket p(222); 112 | p.Reset(PT_V4, 3); 113 | p.AddToBuffer(k, s, 0xAB); 114 | p.AddToBuffer(k, s, 0xCD); 115 | auto data = p.PacketData(); 116 | PrintPacket(data); 117 | ASSERT_EQ(data, StringPiece(want, sizeof(want))); 118 | } 119 | 120 | } // namespace ipfix 121 | } // namespace clerk 122 | -------------------------------------------------------------------------------- /stringpiece.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_STRINGPIECE_H_ 16 | #define CLERK_STRINGPIECE_H_ 17 | 18 | 19 | namespace clerk { 20 | 21 | class StringPiece { 22 | public: 23 | StringPiece() : data_(nullptr), size_(0) {} 24 | StringPiece(const char* data, size_t size) : data_(data), size_(size) {} 25 | const char* data() const { return data_; } 26 | size_t size() const { return size_; } 27 | char operator[](size_t i) const { return data_[i]; } 28 | bool operator==(const StringPiece& s) const { 29 | return s.size_ == size_ && memcmp(s.data_, data_, size_) == 0; 30 | } 31 | 32 | private: 33 | const char* data_; 34 | size_t size_; 35 | }; 36 | 37 | } // namespace clerk 38 | 39 | #endif // CLERK_STRINGPIECE_H_ 40 | -------------------------------------------------------------------------------- /test_main.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | #include 17 | 18 | using google::ParseCommandLineFlags; 19 | 20 | int main(int argc, char** argv) { 21 | ::testing::InitGoogleTest(&argc, argv); 22 | ParseCommandLineFlags(&argc, &argv, true); 23 | return RUN_ALL_TESTS(); 24 | } 25 | -------------------------------------------------------------------------------- /testimony.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | #include 17 | 18 | #include "stringpiece.h" 19 | #include "testimony.h" 20 | 21 | namespace clerk { 22 | 23 | Packet::Packet(const struct tpacket3_hdr* hdr) : hdr_(hdr) { 24 | headers_.Parse(data()); 25 | } 26 | 27 | StringPiece Packet::data() const { 28 | return StringPiece(reinterpret_cast(testimony_packet_data(hdr_)), 29 | hdr_->tp_snaplen); 30 | } 31 | 32 | int64_t Packet::ts_nanos() const { return testimony_packet_nanos(hdr_); } 33 | 34 | TestimonyProcessor::TestimonyProcessor(const string& socket, 35 | const StateFactory* states) 36 | : socket_(socket), states_(states) {} 37 | 38 | void TestimonyProcessor::StartThreads() { 39 | CHECK_EQ(0, threads_.size()); 40 | testimony t; 41 | LOG(INFO) << "Initial connection to testimony socket " << socket_; 42 | CHECK_EQ(0, testimony_connect(&t, socket_.c_str())); 43 | for (int i = 0; i < testimony_conn(t)->fanout_size; i++) { 44 | LOG(INFO) << "Starting testimony thread " << i; 45 | testimony thread_t; 46 | CHECK_EQ(0, testimony_connect(&thread_t, socket_.c_str())); 47 | testimony_conn(thread_t)->fanout_index = i; 48 | CHECK_EQ(0, testimony_init(thread_t)) << testimony_error(thread_t); 49 | threads_.emplace_back(std::unique_ptr( 50 | new TestimonyThread(thread_t, states_->New(nullptr), &last_))); 51 | } 52 | testimony_close(t); 53 | } 54 | 55 | void TestimonyProcessor::Gather(std::vector>* states, 56 | bool last) { 57 | CHECK_NE(0, threads_.size()); 58 | CHECK(!last_.HasBeenNotified()); 59 | if (last) { 60 | LOG(INFO) << "Final TestimonyProcessor gather, stopping threads"; 61 | last_.Notify(); 62 | for (size_t i = 0; i < threads_.size(); i++) { 63 | LOG(INFO) << "Waiting for thread " << i; 64 | threads_[i]->Join(); 65 | LOG(INFO) << "Thread " << i << " completed"; 66 | } 67 | } 68 | states->clear(); 69 | LOG(INFO) << "Gathering state from " << threads_.size() << " threads"; 70 | states->resize(threads_.size()); 71 | for (size_t i = 0; i < threads_.size(); i++) { 72 | (*states)[i] = threads_[i]->SwapState(states_); 73 | } 74 | } 75 | 76 | TestimonyProcessor::~TestimonyProcessor() { CHECK(last_.HasBeenNotified()); } 77 | 78 | std::unique_ptr TestimonyThread::SwapState(const StateFactory* states) { 79 | std::unique_lock ml(mu_); 80 | auto next = states->New(state_.get()); 81 | state_.swap(next); 82 | return next; 83 | } 84 | 85 | void TestimonyThread::Run() { 86 | testimony_iter iter; 87 | CHECK_EQ(0, testimony_iter_init(&iter)); 88 | while (!last_->HasBeenNotified()) { 89 | const struct tpacket_block_desc* block; 90 | CHECK_EQ(0, testimony_get_block(t_, 1000, &block)) << testimony_error(t_); 91 | if (!block) { 92 | VLOG(1) << "Timed out waiting for testimony block"; 93 | continue; 94 | } 95 | VLOG(1) << "Got testimony block"; 96 | CHECK_EQ(0, testimony_iter_reset(iter, block)); 97 | const struct tpacket3_hdr* hdr; 98 | while ((hdr = testimony_iter_next(iter)) != nullptr) { 99 | Packet p(hdr); 100 | std::unique_lock ml(mu_); 101 | state_->Process(p); 102 | } 103 | CHECK_EQ(0, testimony_return_block(t_, block)) << testimony_error(t_); 104 | } 105 | CHECK_EQ(0, testimony_iter_close(iter)); 106 | } 107 | 108 | TestimonyThread::TestimonyThread(testimony t, std::unique_ptr s, 109 | Notification* last) 110 | : state_(std::move(s)), t_(t), last_(last) { 111 | thread_.reset(new std::thread([this]() { Run(); })); 112 | } 113 | 114 | TestimonyThread::~TestimonyThread() { 115 | thread_->join(); 116 | CHECK_EQ(0, testimony_close(t_)); 117 | } 118 | 119 | } // namespace clerk 120 | -------------------------------------------------------------------------------- /testimony.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_TESTIMONY_H_ 16 | #define CLERK_TESTIMONY_H_ 17 | 18 | // Provides bindings to get packets from Testimony, and a method for gathering 19 | // state from multiple packets, then combining it. 20 | 21 | #include 22 | #include 23 | 24 | #include 25 | #include 26 | #include 27 | #include 28 | 29 | #include "headers.h" 30 | #include "util.h" 31 | #include "stringpiece.h" 32 | 33 | // TODO(user): Use namespace access::security::clerk. 34 | namespace clerk { 35 | 36 | class Packet; 37 | 38 | // State is a user-defined class for gathering state from a stream of packets. 39 | class State { 40 | public: 41 | State() {} 42 | virtual ~State() {} 43 | virtual void Process(const Packet& p) = 0; 44 | 45 | private: 46 | DISALLOW_COPY_AND_ASSIGN(State); 47 | }; 48 | 49 | // StateFactory creates new states. 50 | class StateFactory { 51 | public: 52 | virtual ~StateFactory() {} 53 | virtual std::unique_ptr New(const State* old) const = 0; 54 | }; 55 | 56 | template 57 | class EmptyConstructorFactory : public StateFactory { 58 | public: 59 | std::unique_ptr New(const State* old) const override { 60 | return std::unique_ptr(new T()); 61 | } 62 | }; 63 | 64 | template 65 | class SelfConstructorFactory : public StateFactory { 66 | public: 67 | std::unique_ptr New(const State* old) const override { 68 | return std::unique_ptr(new T(reinterpret_cast(old))); 69 | } 70 | }; 71 | 72 | class TestimonyThread; 73 | 74 | // TestimonyProcessor runs TestimonyThreads and gathers states from 75 | // them. 76 | class TestimonyProcessor { 77 | public: 78 | TestimonyProcessor(const string& socket, const StateFactory* states); 79 | virtual ~TestimonyProcessor(); 80 | 81 | void StartThreads(); 82 | // Gather all states currently in threads, replacing them with empty ones. 83 | // Note that Gather with last=true MUST be called before TestimonyProcessor is 84 | // destructed, to stop all threads and gather their final state. 85 | // Also, StartThreads must be called first, one time, before the first Gather 86 | // call. 87 | void Gather(std::vector>* states, bool last); 88 | 89 | private: 90 | const string socket_; 91 | const StateFactory* states_; 92 | std::vector> threads_; 93 | Notification last_; 94 | DISALLOW_COPY_AND_ASSIGN(TestimonyProcessor); 95 | }; 96 | 97 | // TestimonyThread is internal to TestimonyProcessor. It gathers state on a 98 | // single testimony stream. 99 | class TestimonyThread { 100 | public: 101 | TestimonyThread(testimony t, std::unique_ptr s, Notification* last); 102 | ~TestimonyThread(); 103 | std::unique_ptr SwapState(const StateFactory* states); 104 | void Join() { thread_->join(); } 105 | 106 | private: 107 | void Run(); 108 | 109 | std::mutex mu_; 110 | std::unique_ptr state_; 111 | testimony t_; 112 | Notification* last_; 113 | std::unique_ptr thread_; 114 | }; 115 | 116 | // Packet provides data on a single testimony packet. 117 | class Packet { 118 | public: 119 | virtual ~Packet() {} 120 | StringPiece data() const; 121 | int64_t ts_nanos() const; 122 | const struct tpacket3_hdr* hdr() const { return hdr_; } 123 | const Headers& headers() const { return headers_; } 124 | 125 | private: 126 | friend class TestimonyThread; 127 | explicit Packet(const struct tpacket3_hdr* hdr); 128 | const struct tpacket3_hdr* hdr_; 129 | Headers headers_; 130 | DISALLOW_COPY_AND_ASSIGN(Packet); 131 | }; 132 | 133 | } // namespace clerk 134 | 135 | #endif // CLERK_TESTIMONY_H_ 136 | -------------------------------------------------------------------------------- /util.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "util.h" 16 | 17 | struct timespec clerk_clock_realtime, clerk_clock_monotonic; 18 | clockid_t clerk_clock_mono_id = CLOCK_MONOTONIC; 19 | bool run_init_time = InitTime(); 20 | -------------------------------------------------------------------------------- /util.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All rights reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #ifndef CLERK_UTIL_H_ 16 | #define CLERK_UTIL_H_ 17 | 18 | #include 19 | #include 20 | 21 | using namespace std; 22 | 23 | class Notification { 24 | public: 25 | Notification() : done_(false) {} 26 | bool HasBeenNotified() { 27 | std::unique_lock ml(mu_); 28 | return done_; 29 | } 30 | void Notify() { 31 | std::unique_lock ml(mu_); 32 | done_ = true; 33 | } 34 | 35 | private: 36 | bool done_; 37 | std::mutex mu_; 38 | std::condition_variable cond_; 39 | }; 40 | 41 | #define DISALLOW_COPY_AND_ASSIGN(Type) \ 42 | Type(const Type&) {} \ 43 | Type(const Type&&) {} \ 44 | void operator=(const Type&) {} 45 | #define FALLTHROUGH_INTENDED \ 46 | do { \ 47 | } while (0) 48 | 49 | extern struct timespec clerk_clock_realtime, clerk_clock_monotonic; 50 | extern clockid_t clerk_clock_mono_id; 51 | inline bool InitTime() { 52 | clock_gettime(CLOCK_REALTIME, &clerk_clock_realtime); 53 | #ifdef CLOCK_MONOTONIC_RAW 54 | // If monotonic raw clock is supported and available, let's use that. 55 | if (!clock_gettime(CLOCK_MONOTONIC_RAW, &clerk_clock_monotonic)) { 56 | clerk_clock_mono_id = CLOCK_MONOTONIC_RAW; 57 | return true; 58 | } 59 | #endif 60 | clock_gettime(CLOCK_MONOTONIC, &clerk_clock_monotonic); 61 | return true; 62 | } 63 | extern bool clerk_run_init_time; 64 | 65 | const int64_t kNumMillisPerSecond = 1000LL; 66 | const int64_t kNumNanosPerMilli = 1000000LL; 67 | const int64_t kNumNanosPerSecond = 1000000000LL; 68 | 69 | inline int64_t GetCurrentTimeNanos() { 70 | struct timespec tv; 71 | clock_gettime(clerk_clock_mono_id, &tv); 72 | int64_t secs = 73 | clerk_clock_realtime.tv_sec - clerk_clock_monotonic.tv_sec + tv.tv_sec; 74 | int64_t nsecs = 75 | clerk_clock_realtime.tv_nsec - clerk_clock_monotonic.tv_nsec + tv.tv_nsec; 76 | return secs * kNumNanosPerSecond + nsecs; 77 | } 78 | inline double GetCurrentTimeSeconds() { 79 | return GetCurrentTimeNanos() * 1.0L / kNumNanosPerSecond; 80 | } 81 | 82 | inline void SleepForNanoseconds(int64_t nanos) { 83 | if (nanos <= 0) { 84 | return; 85 | } 86 | struct timespec tv; 87 | tv.tv_sec = nanos / kNumNanosPerSecond; 88 | tv.tv_nsec = nanos % kNumNanosPerSecond; 89 | while (EINTR == clock_nanosleep(CLOCK_MONOTONIC, 0, &tv, &tv)) { 90 | } 91 | } 92 | inline void SleepForSeconds(double seconds) { 93 | SleepForNanoseconds(seconds * kNumNanosPerSecond); 94 | } 95 | 96 | #endif // CLERK_UTIL_H_ 97 | --------------------------------------------------------------------------------