├── LICENSE ├── README.md ├── grafana-docker ├── dashboards │ ├── dashboard.yml │ └── grafana-zeek-dashboard.json ├── datasources │ └── datasource.yml └── docker-compose.yml └── zeek-docker ├── Dockerfile ├── local_asn.zeek └── zeek-to-sqlite.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Hacker Target Pty Ltd. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Zeek & Grafana Integration for Network Monitoring 2 | 3 | This repository provides a quick way to get started using Zeek with a practical use case. The focus is to analyse a network pcap and enable easy visual analysis using Grafana Charts. 4 | The mini project consists of three parts. 5 | 6 | 1. Custom Zeek Docker build that generates zeek log files with GeoIP, ASN and JA3 / JA4 fingerprints. 7 | 8 | 2. Python Script to convert zeek log files to an SQLite database. 9 | 10 | 3. Custom Grafana Docker build with a pre-configured dashboard for analysing Zeek Data. 11 | 12 | Keeping this project simple and broken up into three parts should help both Zeek newcomers and those with more experience get up and running quickly. Working from these base images it would be an easy task to add other packages, and extend the dashboard to suit your own environment or use case. 13 | 14 | ![Grafana Dashboard](https://hackertarget.com/images/zeek-grafana-screenshot2.webp) 15 | 16 | A two part article is available that goes into more detail about the project and getting started. 17 | 18 | [Zeek with GeoIP, ASN & JA4 in 5 minutes](https://hackertarget.com/zeek-geoip-asn-ja4/) 19 | 20 | [Zeek Dashboard using Grafana](https://hackertarget.com/zeek-dashboard-grafana/) 21 | 22 | 23 | ## Overview 24 | 25 | The project is structured to use Docker containers for easy setup and portability. It includes a customized Zeek container for log generation and a Grafana container for data visualization. 26 | 27 | 28 | ### File Structure 29 | 30 | - **Dockerfile**: Located at `./pcap-did-what/zeek-docker/Dockerfile`, this file creates a Docker container based on the official Zeek image. It includes the installation of necessary packages for JA3 / JA4 fingerprinting and GeoIP, with a custom script for ASN enrichment. 31 | - **local_asn.zeek**: `./pcap-did-what/zeek-docker/local_asn.zeek`, a small zeek script to add ASN information to the conn.log. The script uses the builtin zeek function (lookup_autonomous_system). 32 | - **docker-compose.yml**: Found in `./pcap-did-what/grafana-docker/docker-compose.yml`, this Docker Compose file sets up the Grafana container, configuring it to use a custom SQLite datasource and including volumes for persistent storage and configuration. 33 | - **dashboard.yml**: Located at `./pcap-did-what/grafana-docker/dashboards/dashboard.yml`, this configuration file specifies the dashboard provider settings for Grafana. 34 | - **datasource.yml**: Found in `./pcap-did-what/grafana-docker/datasources/datasource.yml`, this file configures Grafana to use an SQLite database as the data source, pointing to the Zeek logs stored in SQLite format. 35 | - **GeoLite2-ASN.mmdb**: Place in `./pcap-did-what/zeek-docker/` - required file needed to be downloaded from MaxMind. 36 | - **GeoLite2-City.mmdb**: Another required file from MaxMind. Register Free and Download file to `zeek-docker`. 37 | 38 | ## Usage 39 | 40 | To get started with this setup, ensure you have Docker and Docker Compose installed on your system. 41 | 42 | **Requirement** - You will need the MaxMind GeoIP and ASN databases, these require registration (free) and a download of the two files from MaxMind. Place in `zeek-docker`. 43 | 44 | Follow these steps to deploy the environment: 45 | 46 | 1. **Clone this repository** 47 | 48 | ```git clone https://github.com/hackertarget/pcap-did-what.git``` 49 | 50 | 2. **Build Zeek Docker Image** 51 | 52 | From the ./pcap-did-what/ build zeek image. 53 | 54 | ``` 55 | cd zeek-docker 56 | sudo docker build -t zeek-custom . 57 | ``` 58 | 59 | Running the docker will drop you into bash. From bash it is possible to generate the zeek logs and convert them to sqlite db. 60 | 61 | ``` 62 | # zeek -C -r /data/test.pcap local 63 | # zeek-to-sqlite.py 64 | ``` 65 | 66 | 3. **Start the Grafana Container** 67 | 68 | Use docker-compose to start the Grafana container: 69 | 70 | ``` 71 | cd ../grafana-docker 72 | sudo docker-compose up -d 73 | ``` 74 | 75 | 76 | 4. **Accessing Grafana:** 77 | Once the containers are up and running, access the Grafana dashboard through your web browser: 78 | 79 | ```http://localhost:3000``` 80 | 81 | Use the default Grafana credentials (admin/admin) unless changed in the configuration. The dashboard should be loaded but there will be likely no data unless the zeek_log.db file has been generated and is in place. 82 | 83 | ## Customization and Notes 84 | 85 | - You can modify the `Dockerfile` to include additional Zeek scripts or alter the existing configuration to suit your network analysis requirements. 86 | - The Grafana dashboard and data source configurations can be adjusted in `dashboard.yml` and `datasource.yml` respectively, allowing for further customization of how data is displayed. 87 | - zeek-to-sqlite.py uses a hardcoded /data/ path to match the build. Modify this as required. 88 | - The local_asn.zeek script could be used to learn more about zeek scripting or extend it to populate other log files. 89 | 90 | 91 | ## Credits 92 | 93 | Thanks to Zeek and Grafana for making high quality open source software. 94 | 95 | Also, thanks to Christos @ [Threat Hunting Tails](https://threathuntingtails.com/zeek-asn-enrichment/) for the ASN information and asn.zeek script. 96 | 97 | -------------------------------------------------------------------------------- /grafana-docker/dashboards/dashboard.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | providers: 4 | - name: 'default' 5 | orgId: 1 6 | folder: '' 7 | type: file 8 | disableDeletion: false 9 | editable: true 10 | options: 11 | path: /etc/grafana/provisioning/dashboards 12 | 13 | -------------------------------------------------------------------------------- /grafana-docker/dashboards/grafana-zeek-dashboard.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": { 7 | "type": "grafana", 8 | "uid": "-- Grafana --" 9 | }, 10 | "enable": true, 11 | "hide": true, 12 | "iconColor": "rgba(0, 211, 255, 1)", 13 | "name": "Annotations & Alerts", 14 | "type": "dashboard" 15 | } 16 | ] 17 | }, 18 | "editable": true, 19 | "fiscalYearStartMonth": 0, 20 | "graphTooltip": 0, 21 | "id": 1, 22 | "links": [], 23 | "liveNow": false, 24 | "panels": [ 25 | { 26 | "datasource": { 27 | "type": "frser-sqlite-datasource", 28 | "uid": "SQLite" 29 | }, 30 | "fieldConfig": { 31 | "defaults": { 32 | "color": { 33 | "mode": "palette-classic" 34 | }, 35 | "custom": { 36 | "axisBorderShow": false, 37 | "axisCenteredZero": false, 38 | "axisColorMode": "text", 39 | "axisLabel": "", 40 | "axisPlacement": "auto", 41 | "fillOpacity": 89, 42 | "gradientMode": "opacity", 43 | "hideFrom": { 44 | "legend": false, 45 | "tooltip": false, 46 | "viz": false 47 | }, 48 | "lineWidth": 1, 49 | "scaleDistribution": { 50 | "type": "linear" 51 | }, 52 | "thresholdsStyle": { 53 | "mode": "off" 54 | } 55 | }, 56 | "mappings": [], 57 | "thresholds": { 58 | "mode": "absolute", 59 | "steps": [ 60 | { 61 | "color": "green", 62 | "value": null 63 | }, 64 | { 65 | "color": "red", 66 | "value": 80 67 | } 68 | ] 69 | }, 70 | "unitScale": true 71 | }, 72 | "overrides": [ 73 | { 74 | "matcher": { 75 | "id": "byName", 76 | "options": "count" 77 | }, 78 | "properties": [ 79 | { 80 | "id": "color", 81 | "value": { 82 | "fixedColor": "#4ecda7", 83 | "mode": "fixed" 84 | } 85 | } 86 | ] 87 | } 88 | ] 89 | }, 90 | "gridPos": { 91 | "h": 16, 92 | "w": 5, 93 | "x": 0, 94 | "y": 0 95 | }, 96 | "id": 7, 97 | "options": { 98 | "barRadius": 0.15, 99 | "barWidth": 0.75, 100 | "fullHighlight": false, 101 | "groupWidth": 0.7, 102 | "legend": { 103 | "calcs": [], 104 | "displayMode": "list", 105 | "placement": "bottom", 106 | "showLegend": true 107 | }, 108 | "orientation": "horizontal", 109 | "showValue": "auto", 110 | "stacking": "none", 111 | "tooltip": { 112 | "mode": "single", 113 | "sort": "none" 114 | }, 115 | "xTickLabelRotation": 0, 116 | "xTickLabelSpacing": 0 117 | }, 118 | "targets": [ 119 | { 120 | "datasource": { 121 | "type": "frser-sqlite-datasource", 122 | "uid": "SQLite" 123 | }, 124 | "queryText": "SELECT \n query, COUNT(*) as count\nFROM dns \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY query\nORDER BY count DESC\nLIMIT 30", 125 | "queryType": "table", 126 | "rawQueryText": "SELECT \n query, COUNT(*) as count\nFROM dns \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY query\nORDER BY count DESC\nLIMIT 30", 127 | "refId": "A", 128 | "timeColumns": [ 129 | "time", 130 | "ts" 131 | ] 132 | } 133 | ], 134 | "title": "DNS Queries", 135 | "type": "barchart" 136 | }, 137 | { 138 | "datasource": { 139 | "type": "frser-sqlite-datasource", 140 | "uid": "SQLite" 141 | }, 142 | "fieldConfig": { 143 | "defaults": { 144 | "color": { 145 | "mode": "palette-classic" 146 | }, 147 | "custom": { 148 | "axisBorderShow": false, 149 | "axisCenteredZero": false, 150 | "axisColorMode": "text", 151 | "axisLabel": "", 152 | "axisPlacement": "auto", 153 | "fillOpacity": 80, 154 | "gradientMode": "opacity", 155 | "hideFrom": { 156 | "legend": false, 157 | "tooltip": false, 158 | "viz": false 159 | }, 160 | "lineWidth": 1, 161 | "scaleDistribution": { 162 | "type": "linear" 163 | }, 164 | "thresholdsStyle": { 165 | "mode": "off" 166 | } 167 | }, 168 | "mappings": [], 169 | "thresholds": { 170 | "mode": "absolute", 171 | "steps": [ 172 | { 173 | "color": "green", 174 | "value": null 175 | }, 176 | { 177 | "color": "red", 178 | "value": 80 179 | } 180 | ] 181 | }, 182 | "unitScale": true 183 | }, 184 | "overrides": [ 185 | { 186 | "matcher": { 187 | "id": "byName", 188 | "options": "count" 189 | }, 190 | "properties": [ 191 | { 192 | "id": "color", 193 | "value": { 194 | "fixedColor": "#4ecda7", 195 | "mode": "fixed" 196 | } 197 | } 198 | ] 199 | } 200 | ] 201 | }, 202 | "gridPos": { 203 | "h": 16, 204 | "w": 5, 205 | "x": 5, 206 | "y": 0 207 | }, 208 | "id": 8, 209 | "options": { 210 | "barRadius": 0.1, 211 | "barWidth": 0.75, 212 | "fullHighlight": false, 213 | "groupWidth": 0.7, 214 | "legend": { 215 | "calcs": [], 216 | "displayMode": "list", 217 | "placement": "bottom", 218 | "showLegend": true 219 | }, 220 | "orientation": "horizontal", 221 | "showValue": "auto", 222 | "stacking": "none", 223 | "tooltip": { 224 | "mode": "single", 225 | "sort": "none" 226 | }, 227 | "xTickLabelRotation": 0, 228 | "xTickLabelSpacing": 0 229 | }, 230 | "targets": [ 231 | { 232 | "datasource": { 233 | "type": "frser-sqlite-datasource", 234 | "uid": "SQLite" 235 | }, 236 | "queryText": "SELECT \n host, COUNT(*) as count\nFROM http \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY host\nORDER BY count DESC\nLIMIT 30\n", 237 | "queryType": "table", 238 | "rawQueryText": "SELECT \n host, COUNT(*) as count\nFROM http \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY host\nORDER BY count DESC\nLIMIT 30\n", 239 | "refId": "A", 240 | "timeColumns": [ 241 | "time", 242 | "ts" 243 | ] 244 | } 245 | ], 246 | "title": "HTTP Hosts", 247 | "type": "barchart" 248 | }, 249 | { 250 | "datasource": { 251 | "type": "frser-sqlite-datasource", 252 | "uid": "SQLite" 253 | }, 254 | "fieldConfig": { 255 | "defaults": { 256 | "color": { 257 | "mode": "palette-classic" 258 | }, 259 | "custom": { 260 | "axisBorderShow": false, 261 | "axisCenteredZero": false, 262 | "axisColorMode": "text", 263 | "axisLabel": "", 264 | "axisPlacement": "auto", 265 | "fillOpacity": 80, 266 | "gradientMode": "opacity", 267 | "hideFrom": { 268 | "legend": false, 269 | "tooltip": false, 270 | "viz": false 271 | }, 272 | "lineWidth": 1, 273 | "scaleDistribution": { 274 | "type": "linear" 275 | }, 276 | "thresholdsStyle": { 277 | "mode": "off" 278 | } 279 | }, 280 | "mappings": [], 281 | "thresholds": { 282 | "mode": "absolute", 283 | "steps": [ 284 | { 285 | "color": "#73BF69", 286 | "value": null 287 | }, 288 | { 289 | "color": "red", 290 | "value": 80 291 | } 292 | ] 293 | }, 294 | "unitScale": true 295 | }, 296 | "overrides": [ 297 | { 298 | "matcher": { 299 | "id": "byName", 300 | "options": "count" 301 | }, 302 | "properties": [ 303 | { 304 | "id": "color", 305 | "value": { 306 | "fixedColor": "#4ecda7", 307 | "mode": "fixed" 308 | } 309 | } 310 | ] 311 | } 312 | ] 313 | }, 314 | "gridPos": { 315 | "h": 16, 316 | "w": 5, 317 | "x": 10, 318 | "y": 0 319 | }, 320 | "id": 9, 321 | "options": { 322 | "barRadius": 0.1, 323 | "barWidth": 0.75, 324 | "fullHighlight": false, 325 | "groupWidth": 0.7, 326 | "legend": { 327 | "calcs": [], 328 | "displayMode": "list", 329 | "placement": "bottom", 330 | "showLegend": true 331 | }, 332 | "orientation": "horizontal", 333 | "showValue": "auto", 334 | "stacking": "none", 335 | "tooltip": { 336 | "mode": "single", 337 | "sort": "none" 338 | }, 339 | "xTickLabelRotation": 0, 340 | "xTickLabelSpacing": 0 341 | }, 342 | "targets": [ 343 | { 344 | "datasource": { 345 | "type": "frser-sqlite-datasource", 346 | "uid": "SQLite" 347 | }, 348 | "queryText": "SELECT \n server_name, COUNT(*) as count\nFROM ssl \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY server_name\nORDER BY count DESC\nLIMIT 30", 349 | "queryType": "table", 350 | "rawQueryText": "SELECT \n server_name, COUNT(*) as count\nFROM ssl \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY server_name\nORDER BY count DESC\nLIMIT 30", 351 | "refId": "A", 352 | "timeColumns": [ 353 | "time", 354 | "ts" 355 | ] 356 | } 357 | ], 358 | "title": "SSL Server Name", 359 | "type": "barchart" 360 | }, 361 | { 362 | "datasource": { 363 | "type": "frser-sqlite-datasource", 364 | "uid": "SQLite" 365 | }, 366 | "fieldConfig": { 367 | "defaults": { 368 | "color": { 369 | "mode": "thresholds" 370 | }, 371 | "custom": { 372 | "hideFrom": { 373 | "legend": false, 374 | "tooltip": false, 375 | "viz": false 376 | } 377 | }, 378 | "mappings": [], 379 | "thresholds": { 380 | "mode": "absolute", 381 | "steps": [ 382 | { 383 | "color": "green", 384 | "value": null 385 | }, 386 | { 387 | "color": "red", 388 | "value": 80 389 | } 390 | ] 391 | }, 392 | "unitScale": true 393 | }, 394 | "overrides": [] 395 | }, 396 | "gridPos": { 397 | "h": 11, 398 | "w": 9, 399 | "x": 15, 400 | "y": 0 401 | }, 402 | "id": 1, 403 | "options": { 404 | "basemap": { 405 | "config": {}, 406 | "name": "Layer 0", 407 | "type": "default" 408 | }, 409 | "controls": { 410 | "mouseWheelZoom": true, 411 | "showAttribution": true, 412 | "showDebug": false, 413 | "showMeasure": false, 414 | "showScale": false, 415 | "showZoom": true 416 | }, 417 | "layers": [ 418 | { 419 | "config": { 420 | "showLegend": false, 421 | "style": { 422 | "color": { 423 | "fixed": "#4ecda7" 424 | }, 425 | "opacity": 0.6, 426 | "rotation": { 427 | "fixed": 0, 428 | "max": 360, 429 | "min": -360, 430 | "mode": "mod" 431 | }, 432 | "size": { 433 | "fixed": 5, 434 | "max": 15, 435 | "min": 2 436 | }, 437 | "symbol": { 438 | "fixed": "img/icons/marker/circle.svg", 439 | "mode": "fixed" 440 | }, 441 | "symbolAlign": { 442 | "horizontal": "center", 443 | "vertical": "center" 444 | }, 445 | "textConfig": { 446 | "fontSize": 12, 447 | "offsetX": 0, 448 | "offsetY": 0, 449 | "textAlign": "center", 450 | "textBaseline": "middle" 451 | } 452 | } 453 | }, 454 | "location": { 455 | "mode": "auto" 456 | }, 457 | "name": "Layer 1", 458 | "opacity": 0.3, 459 | "tooltip": true, 460 | "type": "markers" 461 | } 462 | ], 463 | "tooltip": { 464 | "mode": "details" 465 | }, 466 | "view": { 467 | "allLayers": true, 468 | "id": "zero", 469 | "lat": 0, 470 | "lon": 0, 471 | "zoom": 1 472 | } 473 | }, 474 | "pluginVersion": "10.3.3", 475 | "targets": [ 476 | { 477 | "datasource": { 478 | "type": "frser-sqlite-datasource", 479 | "uid": "SQLite" 480 | }, 481 | "queryText": "SELECT `geo_resp_latitude` AS latitude, `geo_resp_longitude` AS longitude, count(*) as count\nFROM conn\nGROUP BY `geo_resp_latitude`, `geo_resp_longitude`", 482 | "queryType": "table", 483 | "rawQueryText": "SELECT `geo_resp_latitude` AS latitude, `geo_resp_longitude` AS longitude, count(*) as count\nFROM conn\nGROUP BY `geo_resp_latitude`, `geo_resp_longitude`", 484 | "refId": "A", 485 | "timeColumns": [ 486 | "time", 487 | "ts" 488 | ] 489 | } 490 | ], 491 | "title": "Remote Connections", 492 | "type": "geomap" 493 | }, 494 | { 495 | "datasource": { 496 | "type": "frser-sqlite-datasource", 497 | "uid": "SQLite" 498 | }, 499 | "gridPos": { 500 | "h": 19, 501 | "w": 9, 502 | "x": 15, 503 | "y": 11 504 | }, 505 | "id": 4, 506 | "targets": [ 507 | { 508 | "datasource": { 509 | "type": "frser-sqlite-datasource", 510 | "uid": "SQLite" 511 | }, 512 | "queryText": "SELECT DISTINCT id_orig_h AS id, \nid_orig_h AS source, \nid_resp_h AS target \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000; ", 513 | "queryType": "table", 514 | "rawQueryText": "SELECT DISTINCT id_orig_h AS id, \nid_orig_h AS source, \nid_resp_h AS target \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000; ", 515 | "refId": "A", 516 | "timeColumns": [ 517 | "time", 518 | "ts" 519 | ] 520 | } 521 | ], 522 | "title": "Node Connections", 523 | "type": "nodeGraph" 524 | }, 525 | { 526 | "datasource": { 527 | "type": "frser-sqlite-datasource", 528 | "uid": "SQLite" 529 | }, 530 | "fieldConfig": { 531 | "defaults": { 532 | "color": { 533 | "mode": "palette-classic" 534 | }, 535 | "custom": { 536 | "axisBorderShow": false, 537 | "axisCenteredZero": false, 538 | "axisColorMode": "text", 539 | "axisLabel": "", 540 | "axisPlacement": "auto", 541 | "barAlignment": 0, 542 | "drawStyle": "line", 543 | "fillOpacity": 0, 544 | "gradientMode": "none", 545 | "hideFrom": { 546 | "legend": false, 547 | "tooltip": false, 548 | "viz": false 549 | }, 550 | "insertNulls": false, 551 | "lineInterpolation": "linear", 552 | "lineWidth": 1, 553 | "pointSize": 5, 554 | "scaleDistribution": { 555 | "type": "linear" 556 | }, 557 | "showPoints": "auto", 558 | "spanNulls": false, 559 | "stacking": { 560 | "group": "A", 561 | "mode": "none" 562 | }, 563 | "thresholdsStyle": { 564 | "mode": "off" 565 | } 566 | }, 567 | "mappings": [], 568 | "thresholds": { 569 | "mode": "absolute", 570 | "steps": [ 571 | { 572 | "color": "green", 573 | "value": null 574 | }, 575 | { 576 | "color": "red", 577 | "value": 80 578 | } 579 | ] 580 | }, 581 | "unitScale": true 582 | }, 583 | "overrides": [ 584 | { 585 | "matcher": { 586 | "id": "byName", 587 | "options": "value" 588 | }, 589 | "properties": [ 590 | { 591 | "id": "color", 592 | "value": { 593 | "fixedColor": "#4ecda7", 594 | "mode": "fixed" 595 | } 596 | } 597 | ] 598 | } 599 | ] 600 | }, 601 | "gridPos": { 602 | "h": 6, 603 | "w": 9, 604 | "x": 0, 605 | "y": 16 606 | }, 607 | "id": 3, 608 | "options": { 609 | "legend": { 610 | "calcs": [], 611 | "displayMode": "list", 612 | "placement": "bottom", 613 | "showLegend": true 614 | }, 615 | "tooltip": { 616 | "mode": "single", 617 | "sort": "none" 618 | } 619 | }, 620 | "pluginVersion": "10.3.3", 621 | "targets": [ 622 | { 623 | "datasource": { 624 | "type": "frser-sqlite-datasource", 625 | "uid": "SQLite" 626 | }, 627 | "queryText": "SELECT \n (ROUND(ts / 5) * 5) AS time, \n COUNT(*) AS \"value\"\nFROM \n conn\nWHERE \n ts >= $__from / 1000 AND \n ts < $__to / 1000\nGROUP BY \n time\nORDER BY \n time ASC\n", 628 | "queryType": "table", 629 | "rawQueryText": "SELECT \n (ROUND(ts / 5) * 5) AS time, \n COUNT(*) AS \"value\"\nFROM \n conn\nWHERE \n ts >= $__from / 1000 AND \n ts < $__to / 1000\nGROUP BY \n time\nORDER BY \n time ASC\n", 630 | "refId": "A", 631 | "timeColumns": [ 632 | "time", 633 | "ts" 634 | ] 635 | } 636 | ], 637 | "title": "Connections / 5 sec", 638 | "type": "timeseries" 639 | }, 640 | { 641 | "datasource": { 642 | "type": "frser-sqlite-datasource", 643 | "uid": "SQLite" 644 | }, 645 | "fieldConfig": { 646 | "defaults": { 647 | "color": { 648 | "fixedColor": "#4ecda7", 649 | "mode": "shades" 650 | }, 651 | "custom": { 652 | "hideFrom": { 653 | "legend": false, 654 | "tooltip": false, 655 | "viz": false 656 | } 657 | }, 658 | "mappings": [], 659 | "unit": "decbytes", 660 | "unitScale": true 661 | }, 662 | "overrides": [ 663 | { 664 | "matcher": { 665 | "id": "byName", 666 | "options": "10.128.2.24" 667 | }, 668 | "properties": [ 669 | { 670 | "id": "color", 671 | "value": { 672 | "fixedColor": "#4ecda7", 673 | "mode": "fixed" 674 | } 675 | } 676 | ] 677 | } 678 | ] 679 | }, 680 | "gridPos": { 681 | "h": 6, 682 | "w": 3, 683 | "x": 9, 684 | "y": 16 685 | }, 686 | "id": 5, 687 | "options": { 688 | "displayLabels": [], 689 | "legend": { 690 | "displayMode": "list", 691 | "placement": "bottom", 692 | "showLegend": false 693 | }, 694 | "pieType": "donut", 695 | "reduceOptions": { 696 | "calcs": [ 697 | "lastNotNull" 698 | ], 699 | "fields": "/.*/", 700 | "values": true 701 | }, 702 | "tooltip": { 703 | "mode": "single", 704 | "sort": "none" 705 | } 706 | }, 707 | "targets": [ 708 | { 709 | "datasource": { 710 | "type": "frser-sqlite-datasource", 711 | "uid": "SQLite" 712 | }, 713 | "queryText": "SELECT \n id_orig_h, \n sum(orig_bytes) AS total_orig_bytes \nFROM conn \nWHERE ts >= ($__from / 1000) AND ts < ($__to / 1000) \nGROUP BY id_orig_h \nORDER BY total_orig_bytes DESC \nLIMIT 20; ", 714 | "queryType": "table", 715 | "rawQueryText": "SELECT \n id_orig_h, \n sum(orig_bytes) AS total_orig_bytes \nFROM conn \nWHERE ts >= ($__from / 1000) AND ts < ($__to / 1000) \nGROUP BY id_orig_h \nORDER BY total_orig_bytes DESC \nLIMIT 20; ", 716 | "refId": "A", 717 | "timeColumns": [ 718 | "time", 719 | "ts" 720 | ] 721 | } 722 | ], 723 | "title": "Outgoing Bytes / host", 724 | "type": "piechart" 725 | }, 726 | { 727 | "datasource": { 728 | "type": "frser-sqlite-datasource", 729 | "uid": "SQLite" 730 | }, 731 | "fieldConfig": { 732 | "defaults": { 733 | "color": { 734 | "fixedColor": "#4ecda7", 735 | "mode": "shades" 736 | }, 737 | "custom": { 738 | "hideFrom": { 739 | "legend": false, 740 | "tooltip": false, 741 | "viz": false 742 | } 743 | }, 744 | "mappings": [], 745 | "unit": "decbytes", 746 | "unitScale": true 747 | }, 748 | "overrides": [] 749 | }, 750 | "gridPos": { 751 | "h": 6, 752 | "w": 3, 753 | "x": 12, 754 | "y": 16 755 | }, 756 | "id": 2, 757 | "options": { 758 | "displayLabels": [], 759 | "legend": { 760 | "displayMode": "table", 761 | "placement": "right", 762 | "showLegend": false, 763 | "values": [] 764 | }, 765 | "pieType": "donut", 766 | "reduceOptions": { 767 | "calcs": [ 768 | "lastNotNull" 769 | ], 770 | "fields": "", 771 | "values": true 772 | }, 773 | "tooltip": { 774 | "mode": "single", 775 | "sort": "asc" 776 | } 777 | }, 778 | "targets": [ 779 | { 780 | "datasource": { 781 | "type": "frser-sqlite-datasource", 782 | "uid": "SQLite" 783 | }, 784 | "queryText": "SELECT \n id_resp_h, \n sum(resp_bytes) AS total_resp_bytes \nFROM conn \nWHERE ts >= ($__from / 1000) AND ts < ($__to / 1000) \nGROUP BY id_resp_h \nORDER BY total_resp_bytes DESC \nLIMIT 20; ", 785 | "queryType": "table", 786 | "rawQueryText": "SELECT \n id_resp_h, \n sum(resp_bytes) AS total_resp_bytes \nFROM conn \nWHERE ts >= ($__from / 1000) AND ts < ($__to / 1000) \nGROUP BY id_resp_h \nORDER BY total_resp_bytes DESC \nLIMIT 20; ", 787 | "refId": "A", 788 | "timeColumns": [ 789 | "time", 790 | "ts" 791 | ] 792 | } 793 | ], 794 | "title": "Incoming Bytes / host", 795 | "type": "piechart" 796 | }, 797 | { 798 | "datasource": { 799 | "type": "frser-sqlite-datasource", 800 | "uid": "SQLite" 801 | }, 802 | "fieldConfig": { 803 | "defaults": { 804 | "color": { 805 | "mode": "thresholds" 806 | }, 807 | "custom": { 808 | "align": "auto", 809 | "cellOptions": { 810 | "type": "auto" 811 | }, 812 | "inspect": false 813 | }, 814 | "mappings": [], 815 | "thresholds": { 816 | "mode": "absolute", 817 | "steps": [ 818 | { 819 | "color": "green", 820 | "value": null 821 | }, 822 | { 823 | "color": "red", 824 | "value": 80 825 | } 826 | ] 827 | }, 828 | "unitScale": true 829 | }, 830 | "overrides": [ 831 | { 832 | "matcher": { 833 | "id": "byName", 834 | "options": "total_orig" 835 | }, 836 | "properties": [ 837 | { 838 | "id": "unit", 839 | "value": "decbytes" 840 | } 841 | ] 842 | }, 843 | { 844 | "matcher": { 845 | "id": "byName", 846 | "options": "total_resp" 847 | }, 848 | "properties": [ 849 | { 850 | "id": "unit", 851 | "value": "decbytes" 852 | } 853 | ] 854 | } 855 | ] 856 | }, 857 | "gridPos": { 858 | "h": 8, 859 | "w": 15, 860 | "x": 0, 861 | "y": 22 862 | }, 863 | "id": 10, 864 | "options": { 865 | "cellHeight": "sm", 866 | "footer": { 867 | "countRows": false, 868 | "fields": "", 869 | "reducer": [ 870 | "sum" 871 | ], 872 | "show": false 873 | }, 874 | "showHeader": true, 875 | "sortBy": [] 876 | }, 877 | "pluginVersion": "10.3.3", 878 | "targets": [ 879 | { 880 | "datasource": { 881 | "type": "frser-sqlite-datasource", 882 | "uid": "SQLite" 883 | }, 884 | "queryText": "SELECT id_orig_h, id_resp_h, id_resp_p, geo_resp_country_code, SUM(orig_bytes) as total_orig, SUM(resp_bytes) as total_resp \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000 \nGROUP BY id_orig_h, id_resp_h \nORDER BY total_resp DESC ", 885 | "queryType": "table", 886 | "rawQueryText": "SELECT id_orig_h, id_resp_h, id_resp_p, geo_resp_country_code, SUM(orig_bytes) as total_orig, SUM(resp_bytes) as total_resp \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000 \nGROUP BY id_orig_h, id_resp_h \nORDER BY total_resp DESC ", 887 | "refId": "A", 888 | "timeColumns": [ 889 | "time", 890 | "ts" 891 | ] 892 | } 893 | ], 894 | "title": "Connections / Port / Geo / Bytes", 895 | "type": "table" 896 | }, 897 | { 898 | "datasource": { 899 | "type": "frser-sqlite-datasource", 900 | "uid": "SQLite" 901 | }, 902 | "fieldConfig": { 903 | "defaults": { 904 | "color": { 905 | "mode": "palette-classic" 906 | }, 907 | "custom": { 908 | "axisBorderShow": false, 909 | "axisCenteredZero": false, 910 | "axisColorMode": "text", 911 | "axisLabel": "", 912 | "axisPlacement": "auto", 913 | "fillOpacity": 71, 914 | "gradientMode": "hue", 915 | "hideFrom": { 916 | "legend": false, 917 | "tooltip": false, 918 | "viz": false 919 | }, 920 | "lineWidth": 2, 921 | "scaleDistribution": { 922 | "type": "linear" 923 | }, 924 | "thresholdsStyle": { 925 | "mode": "off" 926 | } 927 | }, 928 | "mappings": [], 929 | "thresholds": { 930 | "mode": "absolute", 931 | "steps": [ 932 | { 933 | "color": "green", 934 | "value": null 935 | }, 936 | { 937 | "color": "red", 938 | "value": 80 939 | } 940 | ] 941 | }, 942 | "unit": "decbytes", 943 | "unitScale": true 944 | }, 945 | "overrides": [ 946 | { 947 | "matcher": { 948 | "id": "byName", 949 | "options": "total_resp" 950 | }, 951 | "properties": [ 952 | { 953 | "id": "color", 954 | "value": { 955 | "fixedColor": "#4ecda7", 956 | "mode": "fixed" 957 | } 958 | } 959 | ] 960 | }, 961 | { 962 | "matcher": { 963 | "id": "byName", 964 | "options": "total_orig" 965 | }, 966 | "properties": [ 967 | { 968 | "id": "color", 969 | "value": { 970 | "fixedColor": "#a74ecd", 971 | "mode": "fixed" 972 | } 973 | } 974 | ] 975 | } 976 | ] 977 | }, 978 | "gridPos": { 979 | "h": 16, 980 | "w": 6, 981 | "x": 0, 982 | "y": 30 983 | }, 984 | "id": 6, 985 | "options": { 986 | "barRadius": 0, 987 | "barWidth": 1, 988 | "fullHighlight": false, 989 | "groupWidth": 0.8, 990 | "legend": { 991 | "calcs": [], 992 | "displayMode": "list", 993 | "placement": "bottom", 994 | "showLegend": true 995 | }, 996 | "orientation": "horizontal", 997 | "showValue": "auto", 998 | "stacking": "none", 999 | "tooltip": { 1000 | "mode": "single", 1001 | "sort": "none" 1002 | }, 1003 | "xField": "id_resp_h", 1004 | "xTickLabelRotation": 0, 1005 | "xTickLabelSpacing": 0 1006 | }, 1007 | "targets": [ 1008 | { 1009 | "datasource": { 1010 | "type": "frser-sqlite-datasource", 1011 | "uid": "SQLite" 1012 | }, 1013 | "queryText": "SELECT id_orig_h, id_resp_h, SUM(orig_bytes) as total_orig, SUM(resp_bytes) as total_resp \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000 \nGROUP BY id_orig_h, id_resp_h \nORDER BY total_resp DESC\nLIMIT 30 ", 1014 | "queryType": "table", 1015 | "rawQueryText": "SELECT id_orig_h, id_resp_h, SUM(orig_bytes) as total_orig, SUM(resp_bytes) as total_resp \nFROM conn \nWHERE ts >= $__from / 1000 and ts < $__to / 1000 \nGROUP BY id_orig_h, id_resp_h \nORDER BY total_resp DESC\nLIMIT 30 ", 1016 | "refId": "A", 1017 | "timeColumns": [ 1018 | "time", 1019 | "ts" 1020 | ] 1021 | } 1022 | ], 1023 | "title": "Bytes Transfered / Host", 1024 | "type": "barchart" 1025 | }, 1026 | { 1027 | "datasource": { 1028 | "type": "frser-sqlite-datasource", 1029 | "uid": "SQLite" 1030 | }, 1031 | "fieldConfig": { 1032 | "defaults": { 1033 | "custom": { 1034 | "align": "auto", 1035 | "cellOptions": { 1036 | "type": "auto" 1037 | }, 1038 | "filterable": false, 1039 | "inspect": false 1040 | }, 1041 | "mappings": [], 1042 | "thresholds": { 1043 | "mode": "absolute", 1044 | "steps": [ 1045 | { 1046 | "color": "green", 1047 | "value": null 1048 | }, 1049 | { 1050 | "color": "red", 1051 | "value": 80 1052 | } 1053 | ] 1054 | }, 1055 | "unitScale": true 1056 | }, 1057 | "overrides": [ 1058 | { 1059 | "matcher": { 1060 | "id": "byName", 1061 | "options": "count" 1062 | }, 1063 | "properties": [ 1064 | { 1065 | "id": "custom.width", 1066 | "value": 662 1067 | }, 1068 | { 1069 | "id": "custom.cellOptions", 1070 | "value": { 1071 | "mode": "basic", 1072 | "type": "gauge", 1073 | "valueDisplayMode": "text" 1074 | } 1075 | } 1076 | ] 1077 | } 1078 | ] 1079 | }, 1080 | "gridPos": { 1081 | "h": 16, 1082 | "w": 18, 1083 | "x": 6, 1084 | "y": 30 1085 | }, 1086 | "id": 11, 1087 | "options": { 1088 | "cellHeight": "sm", 1089 | "footer": { 1090 | "countRows": false, 1091 | "fields": [], 1092 | "reducer": [ 1093 | "sum" 1094 | ], 1095 | "show": false 1096 | }, 1097 | "showHeader": true, 1098 | "sortBy": [] 1099 | }, 1100 | "pluginVersion": "10.3.3", 1101 | "targets": [ 1102 | { 1103 | "datasource": { 1104 | "type": "frser-sqlite-datasource", 1105 | "uid": "SQLite" 1106 | }, 1107 | "queryText": "SELECT \n certificate_subject, certificate_issuer, COUNT(*) as count\nFROM x509 \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY certificate_issuer\nORDER BY count DESC\n", 1108 | "queryType": "table", 1109 | "rawQueryText": "SELECT \n certificate_subject, certificate_issuer, COUNT(*) as count\nFROM x509 \nWHERE ts >= $__from / 1000 AND ts < $__to / 1000\nGROUP BY certificate_issuer\nORDER BY count DESC\n", 1110 | "refId": "A", 1111 | "timeColumns": [ 1112 | "time", 1113 | "ts" 1114 | ] 1115 | } 1116 | ], 1117 | "title": "SSL Subject / Issuer", 1118 | "type": "table" 1119 | } 1120 | ], 1121 | "refresh": false, 1122 | "schemaVersion": 39, 1123 | "tags": [], 1124 | "templating": { 1125 | "list": [] 1126 | }, 1127 | "time": { 1128 | "from": "2024-02-17T17:05:00.899Z", 1129 | "to": "2024-02-17T17:07:54.003Z" 1130 | }, 1131 | "timepicker": {}, 1132 | "timezone": "", 1133 | "title": "Zeek Dashboard", 1134 | "uid": "a345c401-504a-47e8-8bbe-47b08d724d8f", 1135 | "version": 59, 1136 | "weekStart": "" 1137 | } 1138 | -------------------------------------------------------------------------------- /grafana-docker/datasources/datasource.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | datasources: 4 | - name: SQLite 5 | type: frser-sqlite-datasource 6 | access: proxy 7 | isDefault: true 8 | jsonData: 9 | path: /data/zeek_logs.db 10 | 11 | -------------------------------------------------------------------------------- /grafana-docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | grafana: 4 | image: grafana/grafana 5 | container_name: grafana 6 | ports: 7 | - "3000:3000" 8 | volumes: 9 | - ../data:/data 10 | - ./grafana-storage:/var/lib/grafana 11 | - ./datasources:/etc/grafana/provisioning/datasources 12 | - ./dashboards:/etc/grafana/provisioning/dashboards 13 | environment: 14 | - GF_INSTALL_PLUGINS=frser-sqlite-datasource 15 | - GF_PATHS_DATA=/var/lib/grafana 16 | - GF_PATHS_LOGS=/var/log/grafana 17 | - GF_PATHS_PLUGINS=/var/lib/grafana/plugins 18 | - GF_PATHS_PROVISIONING=/etc/grafana/provisioning 19 | user: "0:472" 20 | -------------------------------------------------------------------------------- /zeek-docker/Dockerfile: -------------------------------------------------------------------------------- 1 | # Use the official Zeek image as a base 2 | FROM zeek/zeek 3 | 4 | # Install necessary tools for zkg (if needed) and the ja3 package 5 | RUN zkg refresh && \ 6 | zkg autoconfig --force 7 | 8 | RUN zkg install ja3 --force && \ 9 | zkg install zeek/foxio/ja4 --force && \ 10 | zkg install geoip-conn --force 11 | 12 | # Copy Maxmind GeoLite2 files (download from Maxmind with Free account) 13 | COPY GeoLite2-City.mmdb /usr/share/GeoIP/ 14 | COPY GeoLite2-ASN.mmdb /usr/share/GeoIP/ 15 | 16 | COPY local_asn.zeek /usr/local/zeek/share/zeek/site/local_asn.zeek 17 | 18 | # Python script to convert zeek logs to sqlite db 19 | COPY zeek-to-sqlite.py /usr/local/bin/ 20 | RUN chmod +x /usr/local/bin/zeek-to-sqlite.py 21 | 22 | # Add packages and local_asn script to local.zeek config 23 | RUN echo "@load ja3" >> /usr/local/zeek/share/zeek/site/local.zeek 24 | RUN echo "@load ja4" >> /usr/local/zeek/share/zeek/site/local.zeek 25 | RUN echo "@load geoip-conn" >> /usr/local/zeek/share/zeek/site/local.zeek 26 | RUN echo "@load local_asn" >> /usr/local/zeek/share/zeek/site/local.zeek 27 | 28 | ENTRYPOINT ["/bin/bash"] 29 | -------------------------------------------------------------------------------- /zeek-docker/local_asn.zeek: -------------------------------------------------------------------------------- 1 | # Script to enrich the conn.log with ASN number and ASN name 2 | 3 | export { 4 | redef record Conn::Info += { 5 | orig_h_asn: geo_autonomous_system &log &optional; 6 | resp_h_asn: geo_autonomous_system &log &optional; 7 | }; 8 | } 9 | 10 | event connection_state_remove(c: connection) &priority=0 11 | { 12 | if (!c?$conn || !c$conn?$id) return; 13 | 14 | local orig_h = c$conn$id$orig_h; 15 | local resp_h = c$conn$id$resp_h; 16 | 17 | if (!Site::is_private_addr(orig_h)) { 18 | c$conn$orig_h_asn = lookup_autonomous_system(orig_h); 19 | } 20 | 21 | if (!Site::is_private_addr(resp_h)) { 22 | c$conn$resp_h_asn = lookup_autonomous_system(resp_h); 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /zeek-docker/zeek-to-sqlite.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | import sqlite3 3 | import os 4 | import csv 5 | 6 | def create_table_from_log(cursor, table_name, columns): 7 | if not columns: # Check if the columns list is empty 8 | raise ValueError(f"No columns found for table {table_name}") 9 | column_definitions = ', '.join([f'"{col}" TEXT' for col in columns]) 10 | create_table_sql = f'CREATE TABLE IF NOT EXISTS "{table_name}" ({column_definitions});' 11 | cursor.execute(create_table_sql) 12 | 13 | def insert_data_from_log(cursor, table_name, columns, data): 14 | placeholders = ', '.join(['?' for _ in columns]) 15 | column_names = ', '.join([f'"{col}"' for col in columns]) 16 | insert_sql = f'INSERT INTO "{table_name}" ({column_names}) VALUES ({placeholders});' 17 | cursor.executemany(insert_sql, data) 18 | 19 | def convert_zeek_logs_to_sqlite(db_name, directory): 20 | conn = sqlite3.connect(db_name) 21 | cursor = conn.cursor() 22 | 23 | for filename in os.listdir(directory): 24 | if filename.endswith(".log"): 25 | table_name = os.path.splitext(filename)[0].replace('-', '_') # Ensure table names are valid SQLite identifiers 26 | log_path = os.path.join(directory, filename) 27 | 28 | with open(log_path, 'r') as file: 29 | reader = csv.reader(file, delimiter='\t') 30 | columns = [] 31 | for row in reader: 32 | if row[0].startswith('#fields'): 33 | columns = [col.replace('.', '_') for col in row[1:]] # Adjusted to handle '#fields' 34 | break # Stop after finding the columns 35 | 36 | if not columns: 37 | print(f"No columns extracted for {filename}.") 38 | continue 39 | 40 | create_table_from_log(cursor, table_name, columns) 41 | 42 | data = [row for row in reader if not row[0].startswith('#')] # Skip comment lines 43 | insert_data_from_log(cursor, table_name, columns, data) 44 | 45 | conn.commit() 46 | conn.close() 47 | 48 | # Example usage 49 | directory_path = '/data/' # Update this to the path of your Zeek log files 50 | db_name = 'zeek_logs.db' 51 | convert_zeek_logs_to_sqlite(db_name, directory_path) 52 | 53 | --------------------------------------------------------------------------------