├── LICENSE ├── NetFlow.kibanadashboard ├── README.md ├── bulk-import.py ├── mapping.json └── tools └── split.bash /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. -------------------------------------------------------------------------------- /NetFlow.kibanadashboard: -------------------------------------------------------------------------------- 1 | { 2 | "title": "NetFlow", 3 | "services": { 4 | "query": { 5 | "list": { 6 | "2": { 7 | "id": 2, 8 | "color": "#6ED0E0", 9 | "alias": "", 10 | "pin": false, 11 | "type": "lucene", 12 | "enable": true, 13 | "query": "" 14 | } 15 | }, 16 | "ids": [ 17 | 2 18 | ] 19 | }, 20 | "filter": { 21 | "list": {}, 22 | "ids": [] 23 | } 24 | }, 25 | "rows": [ 26 | { 27 | "title": "Summary", 28 | "height": "350px", 29 | "editable": true, 30 | "collapse": false, 31 | "collapsable": true, 32 | "panels": [ 33 | { 34 | "error": false, 35 | "span": 3, 36 | "editable": true, 37 | "type": "terms", 38 | "loadingEditor": false, 39 | "field": "pr", 40 | "exclude": [], 41 | "missing": true, 42 | "other": true, 43 | "size": 10, 44 | "order": "count", 45 | "style": { 46 | "font-size": "10pt" 47 | }, 48 | "donut": true, 49 | "tilt": true, 50 | "labels": true, 51 | "arrangement": "horizontal", 52 | "chart": "table", 53 | "counter_pos": "above", 54 | "spyable": true, 55 | "queries": { 56 | "mode": "all", 57 | "ids": [ 58 | 2 59 | ] 60 | }, 61 | "tmode": "terms_stats", 62 | "tstat": "total", 63 | "valuefield": "byte", 64 | "title": "Protocols" 65 | }, 66 | { 67 | "error": false, 68 | "span": 3, 69 | "editable": true, 70 | "type": "terms", 71 | "loadingEditor": false, 72 | "field": "out", 73 | "exclude": [], 74 | "missing": true, 75 | "other": true, 76 | "size": 10, 77 | "order": "count", 78 | "style": { 79 | "font-size": "10pt" 80 | }, 81 | "donut": true, 82 | "tilt": true, 83 | "labels": true, 84 | "arrangement": "horizontal", 85 | "chart": "pie", 86 | "counter_pos": "above", 87 | "spyable": true, 88 | "queries": { 89 | "mode": "all", 90 | "ids": [ 91 | 2 92 | ] 93 | }, 94 | "tmode": "terms_stats", 95 | "tstat": "total", 96 | "valuefield": "byte", 97 | "title": "Interface" 98 | }, 99 | { 100 | "error": false, 101 | "span": 3, 102 | "editable": true, 103 | "type": "terms", 104 | "loadingEditor": false, 105 | "field": "_type", 106 | "exclude": [], 107 | "missing": false, 108 | "other": false, 109 | "size": 10, 110 | "order": "count", 111 | "style": { 112 | "font-size": "10pt" 113 | }, 114 | "donut": true, 115 | "tilt": true, 116 | "labels": true, 117 | "arrangement": "horizontal", 118 | "chart": "pie", 119 | "counter_pos": "above", 120 | "spyable": true, 121 | "queries": { 122 | "mode": "all", 123 | "ids": [ 124 | 2 125 | ] 126 | }, 127 | "tmode": "terms", 128 | "tstat": "total", 129 | "valuefield": "", 130 | "title": "Collectors" 131 | }, 132 | { 133 | "error": false, 134 | "span": 3, 135 | "editable": true, 136 | "type": "stats", 137 | "loadingEditor": false, 138 | "queries": { 139 | "mode": "all", 140 | "ids": [ 141 | 2 142 | ] 143 | }, 144 | "style": { 145 | "font-size": "24pt" 146 | }, 147 | "format": "bytes", 148 | "mode": "total", 149 | "display_breakdown": "yes", 150 | "sort_field": "", 151 | "sort_reverse": false, 152 | "label_name": "Query", 153 | "value_name": "Value", 154 | "spyable": true, 155 | "field": "byte", 156 | "unit": "Byte", 157 | "title": "Byte" 158 | } 159 | ], 160 | "notice": false 161 | }, 162 | { 163 | "title": "Flows", 164 | "height": "250px", 165 | "editable": true, 166 | "collapse": false, 167 | "collapsable": true, 168 | "panels": [ 169 | { 170 | "error": false, 171 | "span": 3, 172 | "editable": true, 173 | "type": "terms", 174 | "loadingEditor": false, 175 | "field": "sa", 176 | "exclude": [], 177 | "missing": true, 178 | "other": true, 179 | "size": 100, 180 | "order": "total", 181 | "style": { 182 | "font-size": "10pt" 183 | }, 184 | "donut": true, 185 | "tilt": true, 186 | "labels": true, 187 | "arrangement": "vertical", 188 | "chart": "pie", 189 | "counter_pos": "none", 190 | "spyable": true, 191 | "queries": { 192 | "mode": "all", 193 | "ids": [ 194 | 2 195 | ] 196 | }, 197 | "tmode": "terms_stats", 198 | "tstat": "total", 199 | "valuefield": "byte", 200 | "title": "Source Address" 201 | }, 202 | { 203 | "error": false, 204 | "span": 3, 205 | "editable": true, 206 | "type": "terms", 207 | "loadingEditor": false, 208 | "field": "sp", 209 | "exclude": [], 210 | "missing": true, 211 | "other": true, 212 | "size": 100, 213 | "order": "total", 214 | "style": { 215 | "font-size": "10pt" 216 | }, 217 | "donut": true, 218 | "tilt": true, 219 | "labels": true, 220 | "arrangement": "horizontal", 221 | "chart": "pie", 222 | "counter_pos": "none", 223 | "spyable": true, 224 | "queries": { 225 | "mode": "all", 226 | "ids": [ 227 | 2 228 | ] 229 | }, 230 | "tmode": "terms_stats", 231 | "tstat": "total", 232 | "valuefield": "byte", 233 | "title": "Source Port" 234 | }, 235 | { 236 | "error": false, 237 | "span": 3, 238 | "editable": true, 239 | "type": "terms", 240 | "loadingEditor": false, 241 | "field": "da", 242 | "exclude": [], 243 | "missing": true, 244 | "other": true, 245 | "size": 100, 246 | "order": "total", 247 | "style": { 248 | "font-size": "10pt" 249 | }, 250 | "donut": true, 251 | "tilt": true, 252 | "labels": true, 253 | "arrangement": "horizontal", 254 | "chart": "pie", 255 | "counter_pos": "none", 256 | "spyable": true, 257 | "queries": { 258 | "mode": "all", 259 | "ids": [ 260 | 2 261 | ] 262 | }, 263 | "tmode": "terms_stats", 264 | "tstat": "total", 265 | "valuefield": "byte", 266 | "title": "Destination Address" 267 | }, 268 | { 269 | "error": false, 270 | "span": 3, 271 | "editable": true, 272 | "type": "terms", 273 | "loadingEditor": false, 274 | "field": "dp", 275 | "exclude": [], 276 | "missing": true, 277 | "other": true, 278 | "size": 100, 279 | "order": "total", 280 | "style": { 281 | "font-size": "10pt" 282 | }, 283 | "donut": true, 284 | "tilt": true, 285 | "labels": true, 286 | "arrangement": "horizontal", 287 | "chart": "pie", 288 | "counter_pos": "none", 289 | "spyable": true, 290 | "queries": { 291 | "mode": "all", 292 | "ids": [ 293 | 2 294 | ] 295 | }, 296 | "tmode": "terms_stats", 297 | "tstat": "total", 298 | "valuefield": "byte", 299 | "title": "Destination Ports" 300 | } 301 | ], 302 | "notice": false 303 | }, 304 | { 305 | "title": "Time Series", 306 | "height": "150px", 307 | "editable": true, 308 | "collapse": false, 309 | "collapsable": true, 310 | "panels": [ 311 | { 312 | "error": false, 313 | "span": 9, 314 | "editable": true, 315 | "type": "column", 316 | "loadingEditor": false, 317 | "panels": [ 318 | { 319 | "type": "histogram", 320 | "mode": "total", 321 | "value_field": "byte", 322 | "time_field": "ts", 323 | "timezone": "utc", 324 | "height": "150", 325 | "x-axis": true, 326 | "y-axis": true, 327 | "scale": 1, 328 | "y_format": "bytes", 329 | "grid": { 330 | "max": null, 331 | "min": 0 332 | }, 333 | "queries": { 334 | "mode": "all", 335 | "ids": [ 336 | 2 337 | ] 338 | }, 339 | "annotate": { 340 | "enable": false, 341 | "query": "*", 342 | "size": 20, 343 | "field": "_type", 344 | "sort": [ 345 | "_score", 346 | "desc" 347 | ] 348 | }, 349 | "auto_int": false, 350 | "resolution": 100, 351 | "interval": "1m", 352 | "intervals": [ 353 | "auto", 354 | "1s", 355 | "1m", 356 | "5m", 357 | "10m", 358 | "30m", 359 | "1h", 360 | "3h", 361 | "12h", 362 | "1d", 363 | "1w", 364 | "1y" 365 | ], 366 | "lines": true, 367 | "fill": 0, 368 | "linewidth": 1, 369 | "points": false, 370 | "pointradius": 5, 371 | "bars": true, 372 | "stack": true, 373 | "spyable": true, 374 | "zoomlinks": true, 375 | "options": true, 376 | "legend": true, 377 | "show_query": true, 378 | "interactive": true, 379 | "legend_counts": true, 380 | "percentage": false, 381 | "zerofill": true, 382 | "derivative": false, 383 | "tooltip": { 384 | "value_type": "cumulative", 385 | "query_as_alias": true 386 | }, 387 | "title": "Traffic", 388 | "editable": true 389 | }, 390 | { 391 | "loading": false, 392 | "sizeable": false, 393 | "draggable": false, 394 | "removable": false, 395 | "span": 10, 396 | "height": "150px", 397 | "editable": true, 398 | "type": "histogram", 399 | "mode": "total", 400 | "value_field": "pkt", 401 | "time_field": "ts", 402 | "timezone": "utc", 403 | "x-axis": true, 404 | "y-axis": true, 405 | "scale": 1, 406 | "y_format": "short", 407 | "grid": { 408 | "max": null, 409 | "min": 0 410 | }, 411 | "queries": { 412 | "mode": "all", 413 | "ids": [ 414 | 2 415 | ] 416 | }, 417 | "annotate": { 418 | "enable": false, 419 | "query": "*", 420 | "size": 20, 421 | "field": "_type", 422 | "sort": [ 423 | "_score", 424 | "desc" 425 | ] 426 | }, 427 | "auto_int": false, 428 | "resolution": 100, 429 | "interval": "1m", 430 | "intervals": [ 431 | "auto", 432 | "1s", 433 | "1m", 434 | "5m", 435 | "10m", 436 | "30m", 437 | "1h", 438 | "3h", 439 | "12h", 440 | "1d", 441 | "1w", 442 | "1y" 443 | ], 444 | "lines": true, 445 | "fill": 0, 446 | "linewidth": 1, 447 | "points": false, 448 | "pointradius": 5, 449 | "bars": true, 450 | "stack": true, 451 | "spyable": true, 452 | "zoomlinks": true, 453 | "options": true, 454 | "legend": true, 455 | "show_query": true, 456 | "interactive": true, 457 | "legend_counts": true, 458 | "percentage": false, 459 | "zerofill": true, 460 | "derivative": false, 461 | "tooltip": { 462 | "value_type": "cumulative", 463 | "query_as_alias": true 464 | }, 465 | "title": "PPS" 466 | } 467 | ], 468 | "title": "Time Series" 469 | }, 470 | { 471 | "error": false, 472 | "span": 3, 473 | "editable": true, 474 | "type": "terms", 475 | "loadingEditor": false, 476 | "field": "sa", 477 | "exclude": [], 478 | "missing": true, 479 | "other": true, 480 | "size": 50, 481 | "order": "total", 482 | "style": { 483 | "font-size": "7pt" 484 | }, 485 | "donut": false, 486 | "tilt": false, 487 | "labels": true, 488 | "arrangement": "horizontal", 489 | "chart": "table", 490 | "counter_pos": "above", 491 | "spyable": true, 492 | "queries": { 493 | "mode": "all", 494 | "ids": [ 495 | 2 496 | ] 497 | }, 498 | "tmode": "terms_stats", 499 | "tstat": "total", 500 | "valuefield": "byte", 501 | "title": "IP" 502 | } 503 | ], 504 | "notice": false 505 | } 506 | ], 507 | "editable": true, 508 | "index": { 509 | "interval": "none", 510 | "pattern": "[logstash-]YYYY.MM.DD", 511 | "default": "tuned", 512 | "warm_fields": true 513 | }, 514 | "style": "light", 515 | "failover": false, 516 | "panel_hints": true, 517 | "loader": { 518 | "save_gist": false, 519 | "save_elasticsearch": true, 520 | "save_local": true, 521 | "save_default": true, 522 | "save_temp": true, 523 | "save_temp_ttl_enable": true, 524 | "save_temp_ttl": "30d", 525 | "load_gist": true, 526 | "load_elasticsearch": true, 527 | "load_elasticsearch_size": 20, 528 | "load_local": true, 529 | "hide": false 530 | }, 531 | "pulldowns": [ 532 | { 533 | "type": "query", 534 | "collapse": true, 535 | "notice": false, 536 | "query": "*", 537 | "pinned": true, 538 | "history": [ 539 | "_type:\"fnf1x\"", 540 | "_type:\"fnf3x\"", 541 | "_type:\"fnf2x\"", 542 | "_type: \"fnf2x\"", 543 | "_type : \"fnf2\"", 544 | "_type : \"fnf1\"", 545 | "", 546 | "_type : \"fnf3\"", 547 | "_type = \"fnf1\"", 548 | "*type=fnf2" 549 | ], 550 | "remember": 10, 551 | "enable": true 552 | }, 553 | { 554 | "type": "filtering", 555 | "collapse": true, 556 | "notice": false, 557 | "enable": true 558 | } 559 | ], 560 | "nav": [ 561 | { 562 | "type": "timepicker", 563 | "collapse": false, 564 | "notice": false, 565 | "status": "Stable", 566 | "time_options": [ 567 | "5m", 568 | "15m", 569 | "1h", 570 | "6h", 571 | "12h", 572 | "24h", 573 | "2d", 574 | "7d", 575 | "30d" 576 | ], 577 | "refresh_intervals": [ 578 | "5s", 579 | "10s", 580 | "30s", 581 | "1m", 582 | "5m", 583 | "15m", 584 | "30m", 585 | "1h", 586 | "2h", 587 | "1d" 588 | ], 589 | "timefield": "ts", 590 | "enable": true, 591 | "now": false 592 | } 593 | ], 594 | "refresh": false 595 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | networkanalysis 2 | =============== 3 | 4 | Network Analysis using ElasticSearch and Kibana 5 | -------------------------------------------------------------------------------- /bulk-import.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | import csv, sys, json, elasticsearch 3 | from elasticsearch import helpers 4 | 5 | es = elasticsearch.Elasticsearch() 6 | csvfile = '2014-03-12.csv' 7 | jdata = dict() 8 | actions = list() 9 | i = 0 10 | proto = {'0': '0', '1': 'ICMP', '2': 'IGMP', '6': 'TCP', '17': 'UDP', '112': 'VRRP', '3': 'GGP', '46': 'RSVP', '50': 'ESP'} 11 | 12 | with open(csvfile, 'rb') as file : 13 | line = csv.reader(file, delimiter = ',', skipinitialspace = True) 14 | for row in line : 15 | i += 1 16 | jdata = { 'ts': row[0][0:19], 'byte': int(row[5]), 'sa': row[1], 'sp': int(float(row[2])), 'da': row[3], 'dp': int(float(row[4])), 'pkt': int(row[6]), 'out': int(row[7]), 'pr': proto[row[8].strip()] } 17 | action = { '_index': 'tuned', '_type': 'fnf3x', '_id': i, '_source': json.dumps(jdata, separators=(',', ':'))} 18 | actions.append(action) 19 | if i % 100000 == 0: 20 | elasticsearch.helpers.bulk(es, actions) 21 | print "Indexed %d, working on next 100000" %(i) 22 | actions = list() 23 | elasticsearch.helpers.bulk(es, actions) 24 | print "Indexed %d, finishing." %(i) 25 | -------------------------------------------------------------------------------- /mapping.json: -------------------------------------------------------------------------------- 1 | { 2 | "tuned": { 3 | "mappings": { 4 | "fnf1x": { 5 | "_all": { 6 | "enabled": false 7 | }, 8 | "_source": { 9 | "enabled": false 10 | }, 11 | "properties": { 12 | "byte": { 13 | "type": "long" 14 | }, 15 | "da": { 16 | "type": "string", 17 | "index": "not_analyzed" 18 | }, 19 | "dp": { 20 | "type": "integer" 21 | }, 22 | "out": { 23 | "type": "integer" 24 | }, 25 | "pkt": { 26 | "type": "long" 27 | }, 28 | "pr": { 29 | "type": "string", 30 | "index": "not_analyzed" 31 | }, 32 | "sa": { 33 | "type": "string", 34 | "index": "not_analyzed" 35 | }, 36 | "sp": { 37 | "type": "integer" 38 | }, 39 | "ts": { 40 | "type": "date", 41 | "format": "YYYY-MM-dd HH:mm:ss" 42 | } 43 | } 44 | }, 45 | "fnf3x": { 46 | "_all": { 47 | "enabled": false 48 | }, 49 | "_source": { 50 | "enabled": false 51 | }, 52 | "properties": { 53 | "byte": { 54 | "type": "long" 55 | }, 56 | "da": { 57 | "type": "string", 58 | "index": "not_analyzed" 59 | }, 60 | "dp": { 61 | "type": "integer" 62 | }, 63 | "out": { 64 | "type": "integer" 65 | }, 66 | "pkt": { 67 | "type": "long" 68 | }, 69 | "pr": { 70 | "type": "string", 71 | "index": "not_analyzed" 72 | }, 73 | "sa": { 74 | "type": "string", 75 | "index": "not_analyzed" 76 | }, 77 | "sp": { 78 | "type": "integer" 79 | }, 80 | "ts": { 81 | "type": "date", 82 | "format": "YYYY-MM-dd HH:mm:ss" 83 | } 84 | } 85 | }, 86 | "fnf2x": { 87 | "_all": { 88 | "enabled": false 89 | }, 90 | "_source": { 91 | "enabled": false 92 | }, 93 | "properties": { 94 | "byte": { 95 | "type": "long" 96 | }, 97 | "da": { 98 | "type": "string", 99 | "index": "not_analyzed" 100 | }, 101 | "dp": { 102 | "type": "integer" 103 | }, 104 | "out": { 105 | "type": "integer" 106 | }, 107 | "pkt": { 108 | "type": "long" 109 | }, 110 | "pr": { 111 | "type": "string", 112 | "index": "not_analyzed" 113 | }, 114 | "sa": { 115 | "type": "string", 116 | "index": "not_analyzed" 117 | }, 118 | "sp": { 119 | "type": "integer" 120 | }, 121 | "ts": { 122 | "type": "date", 123 | "format": "YYYY-MM-dd HH:mm:ss" 124 | } 125 | } 126 | } 127 | } 128 | } 129 | } 130 | -------------------------------------------------------------------------------- /tools/split.bash: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # format nfcapd.201211062105 3 | #-t