├── .github └── workflows │ └── jekyll-gh-pages.yml ├── .gitignore ├── README.md ├── assets ├── 55f6ea5e-2c60-40e5-964f-47a8950d210f.png ├── 5677fc72-f46c-49f3-b5f3-7245ff32448e.png ├── 56a64d7a-63dc-4471-bce9-4accc25ed029.png ├── 56e02aba-9dd0-49ea-bd95-026e55fec118.png ├── 57573875-59fc-4667-b85c-7361d5d09a03.png ├── 580eebab-6048-42c1-945a-50200a09010e.png └── 5cf66e53-b5f8-43e7-be9a-49880a3b4631.png ├── clsReporting.py ├── clsReportingData.py ├── clsReportingStatistics.py ├── config.py.default ├── docs ├── misp-reporting-curation-1.png ├── misp-reporting-curation-2.png ├── misp-reporting-curation-3.png ├── misp-reporting-curation-4.png ├── misp-reporting-summary-1.png ├── misp-reporting-summary-2.png ├── misp-reporting-summary-3.png ├── misp-reporting-summary-4.png ├── misp-reporting-summary-5.png ├── misp-reporting-summary-6.png ├── misp-reporting-summary-7.png ├── misp-reporting-summary-8.png └── misp-reporting-summary-9.png ├── reporting.py ├── requirements.txt └── templates ├── style-blue.css ├── style-red.css ├── style.css ├── template.html ├── template_contributors.html ├── template_curation.html └── template_infrastructure.html /.github/workflows/jekyll-gh-pages.yml: -------------------------------------------------------------------------------- 1 | # Sample workflow for building and deploying a Jekyll site to GitHub Pages 2 | name: Deploy Jekyll with GitHub Pages dependencies preinstalled 3 | 4 | on: 5 | # Runs on pushes targeting the default branch 6 | push: 7 | branches: ["master"] 8 | 9 | # Allows you to run this workflow manually from the Actions tab 10 | workflow_dispatch: 11 | 12 | # Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages 13 | permissions: 14 | contents: read 15 | pages: write 16 | id-token: write 17 | 18 | # Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. 19 | # However, do NOT cancel in-progress runs as we want to allow these production deployments to complete. 20 | concurrency: 21 | group: "pages" 22 | cancel-in-progress: false 23 | 24 | jobs: 25 | # Build job 26 | build: 27 | runs-on: ubuntu-latest 28 | steps: 29 | - name: Checkout 30 | uses: actions/checkout@v4 31 | - name: Setup Pages 32 | uses: actions/configure-pages@v5 33 | - name: Build with Jekyll 34 | uses: actions/jekyll-build-pages@v1 35 | with: 36 | source: ./ 37 | destination: ./_site 38 | - name: Upload artifact 39 | uses: actions/upload-pages-artifact@v3 40 | 41 | # Deployment job 42 | deploy: 43 | environment: 44 | name: github-pages 45 | url: ${{ steps.deployment.outputs.page_url }} 46 | runs-on: ubuntu-latest 47 | needs: build 48 | steps: 49 | - name: Deploy to GitHub Pages 50 | id: deployment 51 | uses: actions/deploy-pages@v4 52 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | config.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MISP reporting 2 | 3 | **MISP reporting** lets you explore the threat information stored in your MISP instance. It provides an overview of  4 | - the trend in the volume of events and attributes 5 | - the distribution of attribute types 6 | - recent activity by key organisations 7 | - threat level and TLP classification of events 8 | - targeted geographies and sectors 9 | - TTPs used by threat actors 10 | - vulnerabilities and weaknesses referenced in threat events 11 | 12 | **MISP reporting** queries your instance, summarises the data, and outputs **HTML reports** together with a **JSON statistics** file. The generated **graphs** can also be embedded in other reporting tools. 13 | 14 | # Setup 15 | 16 | MISP reporting is written in Python and requires a MISP API key. 17 | 18 | Because the web server must read and write the files, ownership must be set to the web‑server user. On Ubuntu this user is `www-data`; on Red Hat it is `apache`. Run the following commands as the appropriate user (replace `www-data` with `apache` on Red Hat). 19 | 20 | **Step 1: Create the reporting directory and clone the repository** 21 | 22 | Custom MISP scripts are best placed in a dedicated directory such as `/var/www/MISP/misp-custom/`. Create it if necessary and clone the project. 23 | 24 | ```bash 25 | sudo -u www-data mkdir /var/www/MISP/misp-custom/ 26 | sudo -u www-data git clone https://github.com/cudeso/misp-reporting.git 27 | cd misp-reporting 28 | ``` 29 | 30 | **Step 2: Set up a Python virtual environment and install dependencies** 31 | 32 | To keep system packages untouched, create a dedicated virtual environment named `venv`. If `virtualenv` is unavailable you may use `python3 -m venv venv`. 33 | 34 | ```bash 35 | sudo -u www-data virtualenv venv 36 | sudo -u www-data venv/bin/pip install -r requirements.txt 37 | ``` 38 | 39 | **Step 3: Configure the application** 40 | 41 | Copy the default configuration and edit it to suit your environment. 42 | 43 | ```bash 44 | sudo -u www-data cp config.py.default config.py 45 | ``` 46 | 47 | At a minimum you must set 48 | 49 | - **misp_url** – the URL of your MISP instance 50 | - **misp_key** – an API key (ideally for an account with the *admin* role) 51 | - **output_dir** – where reports will be written 52 | - **install_dir** – where MISP reporting is installed 53 | 54 | **Step 4: Create the output directories** 55 | 56 | Reports are written to `output_dir`, usually `/var/www/MISP/app/webroot/misp-reporting`. The directory must also contain a copy of the *assets* folder, which holds the logos of key MISP organisations. 57 | 58 | ```bash 59 | sudo -u www-data mkdir /var/www/MISP/app/webroot/misp-reporting/ 60 | sudo -u www-data cp -r /var/www/MISP/misp-custom/misp-reporting/assets /var/www/MISP/app/webroot/misp-reporting/ 61 | chown -R www-data:www-data /var/www/MISP/app/webroot/misp-reporting/ 62 | ``` 63 | 64 | **Step 5: Add your organisation’s logo** 65 | 66 | Copy your logo (and those of other key organisations) into the `assets` directory: 67 | 68 | ```bash 69 | sudo -u www-data cp logo.png /var/www/MISP/app/webroot/misp-reporting/assets/ 70 | ``` 71 | 72 | **Step 6: Run the reporting script** 73 | 74 | Generate the reports: 75 | 76 | ```bash 77 | sudo -u www-data /var/www/MISP/misp-custom/misp-reporting/venv/bin/python /var/www/MISP/misp-custom/misp-reporting/reporting.py 78 | ``` 79 | 80 | The reports are now available at `https:///misp-reporting/`. 81 | 82 | # Cronjob 83 | 84 | To run the report daily, add a cronjob for the web‑server user: 85 | 86 | ```bash 87 | crontab -u www-data -e 88 | ``` 89 | 90 | Insert the following line (adjust the path if required): 91 | 92 | ```bash 93 | 00 6 * * * cd /var/www/MISP/misp-custom/misp-reporting/ ; /var/www/MISP/misp-custom/misp-reporting/venv/bin/python /var/www/MISP/misp-custom/misp-reporting/reporting.py 94 | ``` 95 | 96 | # Reports 97 | 98 | Three report types are planned. The **summary** and **curation overview** are complete; **infrastructure monitoring** is in progress. 99 | 100 | ## Summary report 101 | 102 | The summary report presents high‑level statistics for your MISP server. The header also links to the other reports and provides a button to log in to MISP. 103 | 104 | ![docs/misp-reporting-summary-1.png](docs/misp-reporting-summary-1.png) 105 | 106 | It begins with a summary of published events. Based on `reporting_period`, `reporting_trending_count`, and `reporting_filter`, you see the trend in published events and their attributes. 107 | 108 | ![docs/misp-reporting-summary-2.png](docs/misp-reporting-summary-2.png) 109 | 110 | The next section examines attribute details, showing which attribute types appeared during the last 24 hours and other periods. The breakdown is controlled by `attribute_summary`. 111 | 112 | ![docs/misp-reporting-summary-3.png](docs/misp-reporting-summary-3.png) 113 | 114 | A further section highlights key organisations (defined in `key_organisations`), listing event and attribute counts for the past 24 hours and the current reporting period. Direct links open the corresponding events in MISP. 115 | 116 | ![docs/misp-reporting-summary-4.png](docs/misp-reporting-summary-4.png) 117 | 118 | Threat level and Traffic Light Protocol (TLP) designations follow, enabling you to spot spikes in high‑severity or specific TLP‑tagged events at a glance. 119 | 120 | ![docs/misp-reporting-summary-5.png](docs/misp-reporting-summary-5.png) 121 | 122 | Two charts then show which countries or regions were targeted and which sectors were attacked, using galaxy tags set in `filter_geo` and `filter_sector`. 123 | 124 | ![docs/misp-reporting-summary-6.png](docs/misp-reporting-summary-6.png) 125 | 126 | ![docs/misp-reporting-summary-7.png](docs/misp-reporting-summary-7.png) 127 | 128 | Similarly, threat actors (`filter_ttp_actors`) and attack techniques (`filter_ttp_pattern`) are listed. 129 | 130 | ![docs/misp-reporting-summary-8.png](docs/misp-reporting-summary-8.png) 131 | 132 | Finally, the report summarises all CVEs referenced in events. Additional details, such as descriptions and CVSS scores, are retrieved from cvepremium.circl.lu (`cve_url`). CVEs above the threshold set in `reporting_cve_highlight` are emphasised. 133 | 134 | ![docs/misp-reporting-summary-9.png](docs/misp-reporting-summary-9.png) 135 | 136 | ## Curation report 137 | 138 | The curation report shows how many threat events have been curated—either automatically or manually. A typical **curation process** checks events against the MISP warninglists and the CIRCL Hashlookup service; matches lead to indicators being marked non‑actionable (the *to_ids* flag is cleared). Curation may also enrich events with contextual data. In this setup, curated events are placed in the local workflow state *complete*; uncurated events remain *incomplete*. 139 | 140 | The opening section lists how many events are awaiting curation, how many high‑threat events are waiting, and how many events from completely reliable sources are pending. Links take you straight to each set in MISP. 141 | 142 | ![docs/misp-reporting-curation-1.png](docs/misp-reporting-curation-1.png) 143 | 144 | Next, the report plots the dates of curated and uncurated events so you can see whether outstanding items are recent or historic. 145 | 146 | ![docs/misp-reporting-curation-2.png](docs/misp-reporting-curation-2.png) 147 | 148 | You can also view the volume of curated versus uncurated events per organisation. 149 | 150 | ![docs/misp-reporting-curation-3.png](docs/misp-reporting-curation-3.png) 151 | 152 | Finally, a table lists the individual events, highlighting those with high threat levels and those from fully reliable sources. This helps you prioritise urgent reviews. For automation ideas see the MISP playbook [Curate threat events](https://misp.github.io/misp-playbooks/misp-playbooks/pb_curate_misp_events-with_output.html). 153 | 154 | ![docs/misp-reporting-curation-4.png](docs/misp-reporting-curation-4.png) 155 | 156 | ## Statistics file 157 | 158 | Besides the HTML reports, misp‑reporting creates a JSON file, `statistics.json`, suitable for other tools: 159 | 160 | ``` 161 | { 162 | "misp_server": "https://misp", 163 | "report_date": "2025-04-16", 164 | "statistics": "{'event_count': 138584, 'attribute_count': 6877874, 'user_count': 2, 'org_count': 1124, 'local_org_count': 2}", 165 | "today_statistics": "{'today_event_count': 36598, 'today_attribute_count': 36598, 'today_attribute_ids_count': 29370}", 166 | "today_statistics_attributes": "{'hashes': 4017, 'url': 295, 'domain': 22422, 'Other': 607, 'email': 22, 'network': 2007}" 167 | } 168 | ``` 169 | 170 | ## Graphics 171 | 172 | All images are stored in the report directory and may be reused elsewhere: 173 | 174 | - `attributes_bar_chart.png` 175 | - `attributes_type_bar_chart.png` 176 | - `curated_events_bubble_chart.png` 177 | - `geo_targeting_map.png` 178 | - `threatlevel_bar_chart.png` 179 | - `threat_levels_bar.png` 180 | - `tlp_pie_chart.png` 181 | - `attributes_trending.png` 182 | - `attributes_type_daily_bar_chart.png` 183 | - `events_trending.png` 184 | - `threatlevel_pie_chart.png` 185 | - `threat_levels_pie.png` 186 | - `trending_graph.png` 187 | 188 | ## Prevent unauthenticated access 189 | 190 | By default, everything under `/var/www/MISP/app/webroot/misp-reporting` is publicly accessible. To restrict access, modify MISP core as explained in [github.com/MISP/MISP/issues/10272](https://github.com/MISP/MISP/issues/10272) by editing `/var/www/MISP/app/AttributesController.php`. 191 | 192 | # Configuration options 193 | 194 | The configuration is handled in `config.py`. 195 | 196 | Set the log file (`logfile`), logger name (`logname`), and whether to log uncurated events (`log_incomplete`): 197 | 198 | ``` 199 | config = { 200 | "logfile": "/var/www/MISP/app/tmp/logs/misp-reporting.log", 201 | "logname": "misp-reporting", 202 | "log_incomplete": False, 203 | ``` 204 | 205 | Define the length of the reporting period (`reporting_period`), how many periods to display (`reporting_trending_count`), and any event filter (`reporting_filter`). `reporting_eventdetails_onlykeyorgs` determines whether to print event details later (see `print_event_details`). `reporting_filter_timestamp` lets you trend on either *timestamp* or *published* dates. 206 | 207 | ``` 208 | "reporting_period": "30d", 209 | "reporting_filter": None, 210 | "reporting_eventdetails_onlykeyorgs": True, 211 | "reporting_trending_count": 6, 212 | "reporting_filter_attribute_type_ids": True, 213 | "reporting_filter_timestamp": "published", 214 | "reporting_filter_published": True, 215 | 216 | "print_event_details": False, 217 | ``` 218 | 219 | MISP connection settings follow. Provide the instance URL, whether to verify the certificate, and an API key. `misp_page_size` controls pagination. `cve_url` and `vulnerability_lookup_url` supply vulnerability information. `reporting_cve_highlight` sets the CVSS threshold for highlighting CVEs. 220 | 221 | ``` 222 | "misp_url": "", 223 | "misp_verifycert": False, 224 | "misp_key": "", 225 | "misp_page_size": 200, 226 | "cve_url": "https://cvepremium.circl.lu/api/cve/", 227 | "vulnerability_lookup_url": "https://vulnerability.circl.lu/vuln/", 228 | 229 | "reporting_cve_highlight": 7, 230 | ``` 231 | 232 | Filter options: 233 | 234 | ``` 235 | "filter_sector": "misp-galaxy:sector", 236 | "filter_sector_count": 10, 237 | "filter_geo": "misp-galaxy:target-information", 238 | "filter_geo_count": 10, 239 | "filter_ttp_actors": ["misp-galaxy:threat-actor", "misp-galaxy:mitre-intrusion-set"], 240 | "filter_ttp_actors_count": 15, 241 | "filter_ttp_pattern": ["misp-galaxy:mitre-attack-pattern"], 242 | "filter_ttp_pattern_count": 15, 243 | ``` 244 | 245 | Attribute groupings: 246 | 247 | ``` 248 | "attribute_summary": {"network": ["ip-src", "ip-dst", "ip-src|port", "ip-dst|port"], 249 | "hashes": ["sha1", "sha256", "md5", "filename|sha256", "filename|sha1", "filename|sha256", "filename|md5", "filename|sha256"], 250 | "domain": ["hostname", "domain", "domain|ip"], 251 | "url": ["url"], 252 | "email": ["email", "email-src", "email-dst"], 253 | "vulnerability": ["vulnerability", "cpe", "weakness"]}, 254 | "attribute_other": "Other", 255 | ``` 256 | 257 | Key organisations and their logos: 258 | 259 | ``` 260 | "key_organisations": {"5cf66e53-b5f8-43e7-be9a-49880a3b4631": {"logo": "5cf66e53-b5f8-43e7-be9a-49880a3b4631.png"}, 261 | "55f6ea5e-2c60-40e5-964f-47a8950d210f": {"logo": "55f6ea5e-2c60-40e5-964f-47a8950d210f.png"}, 262 | "56a64d7a-63dc-4471-bce9-4accc25ed029": {"logo": "56a64d7a-63dc-4471-bce9-4accc25ed029.png"}, 263 | "56e02aba-9dd0-49ea-bd95-026e55fec118": {"logo": "56e02aba-9dd0-49ea-bd95-026e55fec118.png"}, 264 | "5677fc72-f46c-49f3-b5f3-7245ff32448e": {"logo": "5677fc72-f46c-49f3-b5f3-7245ff32448e.png"}}, 265 | ``` 266 | 267 | Installation paths, templates, and appearance: 268 | 269 | ``` 270 | "output_assets": "assets", 271 | "output_dir": "/var/www/MISP/app/webroot/misp-reporting", 272 | "install_dir": "/var/www/MISP/misp-custom/misp-reporting", 273 | "template_css": "templates/style.css", 274 | "template_html": "templates/template.html", 275 | "template_curation_html": "templates/template_curation.html", 276 | "template_infrastructure_html": "templates/template_infrastructure.html", 277 | "template_contributors_html": "templates/template_contributors.html", 278 | "noimage_path": "no-image-32.png", 279 | "logo": "assets/cudeso.png", 280 | 281 | "distribution": ["Your organisation", "This community", "Connected communities", "All communities", "Sharing group"], 282 | "analysis": ["Initial", "Ongoing", "Completed"], 283 | "threatlevel_key_mapping": {'1': 'High', '2': 'Medium', '3': 'Low', '4': 'Undefined'}, 284 | "tlp_ignore_graph": ['tlp:unclear', 'tlp:ex:chr', 'tlp:amber+strict'], 285 | "workflow_complete": "workflow:state=\"complete\"", 286 | "workflow_incomplete": "workflow:state=\"incomplete\"", 287 | } 288 | ``` 289 | -------------------------------------------------------------------------------- /assets/55f6ea5e-2c60-40e5-964f-47a8950d210f.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/55f6ea5e-2c60-40e5-964f-47a8950d210f.png -------------------------------------------------------------------------------- /assets/5677fc72-f46c-49f3-b5f3-7245ff32448e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/5677fc72-f46c-49f3-b5f3-7245ff32448e.png -------------------------------------------------------------------------------- /assets/56a64d7a-63dc-4471-bce9-4accc25ed029.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/56a64d7a-63dc-4471-bce9-4accc25ed029.png -------------------------------------------------------------------------------- /assets/56e02aba-9dd0-49ea-bd95-026e55fec118.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/56e02aba-9dd0-49ea-bd95-026e55fec118.png -------------------------------------------------------------------------------- /assets/57573875-59fc-4667-b85c-7361d5d09a03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/57573875-59fc-4667-b85c-7361d5d09a03.png -------------------------------------------------------------------------------- /assets/580eebab-6048-42c1-945a-50200a09010e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/580eebab-6048-42c1-945a-50200a09010e.png -------------------------------------------------------------------------------- /assets/5cf66e53-b5f8-43e7-be9a-49880a3b4631.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/assets/5cf66e53-b5f8-43e7-be9a-49880a3b4631.png -------------------------------------------------------------------------------- /clsReporting.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | from matplotlib.ticker import MultipleLocator 14 | import numpy as np 15 | from collections import defaultdict 16 | 17 | 18 | class Reporting: 19 | def __init__(self, config, logger, data, output_dir="report_output"): 20 | self.config = config 21 | self.logger = logger 22 | self.data = data 23 | self.data_for_report = {} 24 | 25 | self.report_date = datetime.now().strftime("%Y-%m-%d") 26 | self.report_misp_server = self.config["misp_url"] 27 | self.reporting_period = self.config["reporting_period"] 28 | self.output_dir = self.config["output_dir"] 29 | self.template_css = "{}/{}".format(self.config["install_dir"], self.config["template_css"]) 30 | self.template_html = "{}/{}".format(self.config["install_dir"], self.config["template_html"]) 31 | self.template_curation_html = "{}/{}".format(self.config["install_dir"], self.config["template_curation_html"]) 32 | self.template_infrastructure_html = "{}/{}".format(self.config["install_dir"], self.config["template_infrastructure_html"]) 33 | self.template_contributors_html = "{}/{}".format(self.config["install_dir"], self.config["template_contributors_html"]) 34 | self.assets_dir = os.path.join(self.output_dir, self.config["output_assets"]) 35 | os.makedirs(self.assets_dir, exist_ok=True) 36 | 37 | self.events_trending_path = os.path.join(self.output_dir, "events_trending.png") 38 | self.attributes_trending_path = os.path.join(self.output_dir, "attributes_trending.png") 39 | self.attributes_type_bar_chart_path = os.path.join(self.output_dir, "attributes_type_bar_chart.png") 40 | self.attributes_type_daily_bar_chart_path = os.path.join(self.output_dir, "attributes_type_daily_bar_chart.png") 41 | self.threatlevel_bar_chart_path = os.path.join(self.output_dir, "threatlevel_bar_chart.png") 42 | self.tlp_pie_chart_path = os.path.join(self.output_dir, "tlp_pie_chart.png") 43 | self.geo_targeting_map_path = os.path.join(self.output_dir, "geo_targeting_map.png") 44 | self.sector_targeting_bar_chart_path = os.path.join(self.output_dir, "sector_targeting_bar_chart.png") 45 | self.curated_events_bubble_path = os.path.join(self.output_dir, "curated_events_bubble_chart.png") 46 | self.noimage_path = self.config["noimage_path"] 47 | 48 | self.threatlevel_key_mapping = self.config["threatlevel_key_mapping"] 49 | self.tlp_ignore_graph = self.config["tlp_ignore_graph"] 50 | self.attribute_summary = self.config["attribute_summary"] 51 | self.attribute_other = self.config["attribute_other"] 52 | self.key_organisations = self.config["key_organisations"] 53 | 54 | if self.config["misp_verifycert"] is False: 55 | import urllib3 56 | urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 57 | self.misp = PyMISP(self.config["misp_url"], self.config["misp_key"], self.config["misp_verifycert"]) 58 | 59 | def write_index(self): 60 | html_content = """ 61 | 62 | 63 | 64 | Redirecting... 65 | 66 | 67 | 68 | 69 | 70 |

If you are not automatically redirected, please click here.

71 | 72 | 73 | """ 74 | 75 | with open("{}/index.html".format(self.config["output_dir"]), 'w') as f: 76 | f.write(html_content) 77 | 78 | def render_infrastructure(self): 79 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 80 | 81 | template_css_file = self.template_css 82 | with open(template_css_file, "r") as f: 83 | css_content = f.read() 84 | 85 | template_file = self.template_infrastructure_html 86 | with open(template_file, "r") as f: 87 | html_template = f.read() 88 | 89 | # Render the HTML 90 | template = Template(html_template) 91 | html_content = template.render( 92 | css=css_content, 93 | title="MISP Infrastructure summary", 94 | logo=self.config["logo"], 95 | report_date=self.report_date, 96 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 97 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 98 | report_misp_server=self.report_misp_server, 99 | report_misp_results=self.data.get("infrastructure_misp", {}), 100 | 101 | ) 102 | 103 | # Save the HTML file 104 | output_html_path = os.path.join(self.output_dir, "misp_infrastructure.html") 105 | with open(output_html_path, "w") as f: 106 | f.write(html_content) 107 | return True 108 | 109 | def render_contributors(self): 110 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 111 | 112 | print_misp_filters = None 113 | if self.config["reporting_filter"] is not None: 114 | print_misp_filters = self.config["reporting_filter"] 115 | 116 | days = int(self.config["reporting_period"].strip("d")) 117 | current_date = datetime.now() 118 | past_date = current_date - timedelta(days=days) 119 | print_period = f"(until {past_date.strftime('%Y-%m-%d')})" 120 | 121 | # ############### Contributing organisations 122 | key = "contributor_orgs" 123 | if key in self.data: 124 | dataset = self.data[key] 125 | updated_dataset = {} 126 | for orgid in dataset: 127 | try: 128 | org = self.misp.get_organisation(orgid) 129 | if "Organisation" in org: 130 | org_name = org["Organisation"]["name"] 131 | uuid = org["Organisation"]["uuid"] 132 | logo = None 133 | if uuid in self.key_organisations: 134 | logo = self.key_organisations[uuid]["logo"] 135 | period_events = dataset[orgid]["reporting-period"] 136 | today_events = dataset[orgid]["today"] 137 | alltime_events = dataset[orgid]["all_time"] 138 | updated_dataset[org_name] = {"logo": logo, "org_uuid": uuid, "period_events": period_events, 139 | "today_events": today_events, "alltime_events": alltime_events} 140 | else: 141 | self.logger.error("Unable to get organisation info for {}".format(orgid)) 142 | except Exception as e: 143 | self.logger.error("Unable to get organisation info for {} - {}".format(orgid, e)) 144 | self.data_for_report[key] = updated_dataset 145 | self.logger.debug(" Created {}".format(key)) 146 | else: 147 | self.data_for_report[key] = {} 148 | self.logger.error(" Not found: {}".format(key)) 149 | 150 | reporting_filter = "" 151 | if self.config["reporting_filter"] is not None: 152 | reporting_filter = "MISP filters: {} ".format(self.config["reporting_filter"]) 153 | 154 | template_css_file = self.template_css 155 | with open(template_css_file, "r") as f: 156 | css_content = f.read() 157 | 158 | template_file = self.template_contributors_html 159 | with open(template_file, "r") as f: 160 | html_template = f.read() 161 | 162 | # Render the HTML 163 | template = Template(html_template) 164 | html_content = template.render( 165 | css=css_content, 166 | title="MISP Contributors summary", 167 | logo=self.config["logo"], 168 | report_date=self.report_date, 169 | report_misp_filters=print_misp_filters, 170 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 171 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 172 | reporting_period=self.config["reporting_period"], 173 | summary={"period": print_period}, 174 | report_misp_server=self.report_misp_server, 175 | contributors_org = self.data_for_report["contributor_orgs"], 176 | contributors_user = {} 177 | ) 178 | 179 | # Save the HTML file 180 | output_html_path = os.path.join(self.output_dir, "misp_contributors.html") 181 | with open(output_html_path, "w") as f: 182 | f.write(html_content) 183 | return True 184 | 185 | def render_curation_report(self): 186 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 187 | 188 | print_misp_filters = None 189 | # ############### General statistics 190 | key = "statistics" 191 | if key in self.data: 192 | dataset = self.data[key] 193 | updated_dataset = {} 194 | days = int(self.config["reporting_period"].strip("d")) 195 | current_date = datetime.now() 196 | past_date = current_date - timedelta(days=days) 197 | reporting_period = self.config["reporting_period"] 198 | updated_dataset["period"] = f"(until {past_date.strftime('%Y-%m-%d')})" 199 | if self.config["reporting_filter"] is not None: 200 | print_misp_filters = self.config["reporting_filter"] 201 | #updated_dataset["events"] = dataset["event_count"] 202 | #updated_dataset["attributes"] = dataset["attribute_count"] 203 | self.data_for_report[key] = updated_dataset 204 | self.logger.debug(" Created {}".format(key)) 205 | else: 206 | self.data_for_report[key] = {} 207 | self.logger.error(" Not found: {}".format(key)) 208 | 209 | # ############### Curation events 210 | key = "curation_complete" 211 | curation_complete_count = 0 212 | curation_complete_today_count = 0 213 | curation_incomplete_count = 0 214 | curation_incomplete_today_count = 0 215 | curation_complete_events = [] 216 | curation_incomplete_events = [] 217 | curation_incomplete_high_events = [] 218 | curation_incomplete_high_count = 0 219 | curation_incomplete_adm_high_events = [] 220 | curation_incomplete_adm_high_count = 0 221 | 222 | if key in self.data: 223 | dataset = self.data[key] 224 | curation_complete_count = len(dataset) 225 | curation_complete_events = dataset 226 | self.logger.debug(" Created {}".format(key)) 227 | else: 228 | self.data_for_report[key] = {} 229 | self.logger.error(" Not found: {}".format(key)) 230 | 231 | if "curation_complete_today" in self.data: 232 | dataset = self.data["curation_complete_today"] 233 | curation_complete_today_count = len(dataset) 234 | self.logger.debug(" Created {}".format("curation_complete_today")) 235 | else: 236 | self.data_for_report["curation_complete_today"] = {} 237 | self.logger.error(" Not found: {}".format("curation_complete_today")) 238 | 239 | key = "curation_incomplete" 240 | if key in self.data: 241 | dataset = self.data[key] 242 | curation_incomplete_count = len(dataset) 243 | curation_incomplete_events = dataset 244 | self.logger.debug(" Created {}".format(key)) 245 | else: 246 | self.data_for_report[key] = {} 247 | self.logger.error(" Not found: {}".format(key)) 248 | 249 | if "curation_incomplete_today" in self.data: 250 | dataset = self.data["curation_incomplete_today"] 251 | curation_incomplete_today_count = len(dataset) 252 | self.logger.debug(" Created {}".format("curation_incomplete_today")) 253 | else: 254 | self.data_for_report["curation_incomplete_today"] = {} 255 | self.logger.error(" Not found: {}".format("curation_incomplete_today")) 256 | 257 | if "curation_incomplete_high" in self.data: 258 | dataset = self.data["curation_incomplete_high"] 259 | curation_incomplete_high_count = len(dataset) 260 | curation_incomplete_high_events = dataset 261 | self.logger.debug(" Created {}".format("curation_incomplete_high")) 262 | else: 263 | self.data_for_report["curation_incomplete_high"] = {} 264 | self.logger.error(" Not found: {}".format("curation_incomplete_high")) 265 | 266 | if "curation_incomplete_adm_high" in self.data: 267 | dataset = self.data["curation_incomplete_adm_high"] 268 | curation_incomplete_adm_high_count = len(dataset) 269 | curation_incomplete_adm_high_events = dataset 270 | self.logger.debug(" Created {}".format("curation_incomplete_adm_high")) 271 | else: 272 | self.data_for_report["curation_incomplete_adm_high"] = {} 273 | self.logger.error(" Not found: {}".format("curation_incomplete_adm_high")) 274 | 275 | key1 = "curation_incomplete_date" 276 | key2 = "curation_complete_date" 277 | if key1 in self.data and key2 in self.data: 278 | self.data_for_report[key1] = self._aggregate_by_month(self.data[key1]) 279 | self.data_for_report[key2] = self._aggregate_by_month(self.data[key2]) 280 | 281 | all_months = set(self.data_for_report[key1].keys()) | set(self.data_for_report[key2].keys()) 282 | if not all_months: 283 | self.logger.error("Not all months for {} or {}".format(key1, key2)) 284 | else: 285 | sorted_months = sorted(all_months, key=lambda x: datetime.strptime(x, "%Y-%m")) 286 | earliest, latest = sorted_months[0], sorted_months[-1] 287 | 288 | full_months = self._month_range(earliest, latest) 289 | values1 = [self.data_for_report[key1].get(m, 0) for m in full_months] 290 | values2 = [self.data_for_report[key2].get(m, 0) for m in full_months] 291 | 292 | self.create_bubble_chart(values1, values2, full_months, self.curated_events_bubble_path, "Event dates", "Not curated", "Curated", True) 293 | self.logger.debug(" Created {} and {}".format(key1, key2)) 294 | else: 295 | self.data_for_report[key1] = {} 296 | self.data_for_report[key2] = {} 297 | self.logger.error(" Not found: {} or {}".format(key1, key2)) 298 | 299 | key1 = "curation_orgs_complete" 300 | key2 = "curation_orgs_incomplete" 301 | if key1 in self.data and key2 in self.data: 302 | dataset = self.data[key1] 303 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 304 | self.data_for_report[key1] = sorted_data 305 | 306 | dataset = self.data[key2] 307 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 308 | self.data_for_report[key2] = sorted_data 309 | else: 310 | self.data_for_report[key1] = {} 311 | self.data_for_report[key2] = {} 312 | self.logger.error(" Not found: {} or {}".format(key1, key2)) 313 | 314 | template_css_file = self.template_css 315 | with open(template_css_file, "r") as f: 316 | css_content = f.read() 317 | 318 | template_file = self.template_curation_html 319 | with open(template_file, "r") as f: 320 | html_template = f.read() 321 | 322 | # Render the HTML 323 | template = Template(html_template) 324 | html_content = template.render( 325 | css=css_content, 326 | title="MISP Curation summary", 327 | logo=self.config["logo"], 328 | report_date=self.report_date, 329 | report_misp_filters=print_misp_filters, 330 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 331 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 332 | reporting_period=self.config["reporting_period"], 333 | 334 | report_misp_server=self.report_misp_server, 335 | summary=self.data_for_report.get("statistics", {}), 336 | curation_incomplete_today_count=curation_incomplete_today_count, 337 | curation_complete_today_count=curation_complete_today_count, 338 | curation_complete_count=curation_complete_count, 339 | curation_incomplete_count=curation_incomplete_count, 340 | 341 | curation_complete=curation_complete_events, 342 | curation_incomplete=curation_incomplete_events, 343 | 344 | curation_incomplete_high=curation_incomplete_high_events, 345 | curation_incomplete_high_count=curation_incomplete_high_count, 346 | curation_incomplete_adm_high=curation_incomplete_adm_high_events, 347 | curation_incomplete_adm_high_count=curation_incomplete_adm_high_count, 348 | 349 | curation_complete_org=self.data_for_report["curation_orgs_complete"], 350 | curation_incomplete_org=self.data_for_report["curation_orgs_incomplete"], 351 | 352 | curated_events_bubble=os.path.basename(self.curated_events_bubble_path) 353 | ) 354 | 355 | # Save the HTML file 356 | output_html_path = os.path.join(self.output_dir, "misp_curation.html") 357 | with open(output_html_path, "w") as f: 358 | f.write(html_content) 359 | return True 360 | 361 | def _aggregate_by_month(self, data_dict): 362 | monthly_data = defaultdict(int) 363 | for date_str, count in data_dict.items(): 364 | year_month = date_str[:7] # YYYY-MM 365 | monthly_data[year_month] += count 366 | return dict(monthly_data) 367 | 368 | def _month_range(self, start_ym, end_ym): 369 | start = datetime.strptime(start_ym, "%Y-%m") 370 | end = datetime.strptime(end_ym, "%Y-%m") 371 | current = start 372 | result = [] 373 | while current <= end: 374 | result.append(current.strftime("%Y-%m")) 375 | year = current.year 376 | month = current.month 377 | if month == 12: 378 | year += 1 379 | month = 1 380 | else: 381 | month += 1 382 | current = datetime(year, month, 1) 383 | return result 384 | 385 | def create_bubble_chart(self, values1, values2, full_months, output_path, title, data1_label, data2_label, full_width=False): 386 | figsize = (8, 4) if full_width else (6, 4) 387 | plt.figure(figsize=figsize) 388 | x_positions = list(range(len(full_months))) 389 | sizes1 = [v * 50 for v in values1] 390 | sizes2 = [v * 50 for v in values2] 391 | plt.scatter(x_positions, [0]*len(full_months), s=sizes1, alpha=0.6, c="#D35400", edgecolors="black", label=data1_label) 392 | plt.scatter(x_positions, [1]*len(full_months), s=sizes2, alpha=0.6, c="#F39C12", edgecolors="black", label=data2_label) 393 | 394 | plt.title(title, fontsize=10) 395 | 396 | tick_positions = x_positions[::6] 397 | tick_labels = [full_months[i] for i in tick_positions] 398 | plt.xticks(tick_positions, tick_labels, rotation=45, ha="right", fontsize=8) 399 | 400 | plt.ylim(-0.5, 1.5) 401 | plt.yticks([0, 1], [data1_label, data2_label]) 402 | plt.xlabel(" ", fontsize=8) 403 | plt.tight_layout() 404 | plt.savefig(output_path, dpi=100) 405 | plt.close() 406 | 407 | def render_report(self): 408 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 409 | 410 | self.logger.debug("Removing older images") 411 | for img in [self.events_trending_path, 412 | self.attributes_trending_path, 413 | self.attributes_type_bar_chart_path, 414 | self.attributes_type_daily_bar_chart_path, 415 | self.threatlevel_bar_chart_path, 416 | self.tlp_pie_chart_path, 417 | self.geo_targeting_map_path, 418 | self.sector_targeting_bar_chart_path 419 | ]: 420 | if os.path.exists(img): 421 | os.remove(img) 422 | 423 | print_misp_filters = None 424 | # ############### Trending events 425 | key = "trending-events" 426 | if key in self.data: 427 | dataset = self.data[key] 428 | days = int(self.config["reporting_period"].strip("d")) 429 | sorted_keys = sorted(dataset.keys(), reverse=True) 430 | updated_dataset = {} 431 | highest_key = sorted_keys[0] 432 | updated_dataset[f"{highest_key + days}d-{highest_key}d"] = dataset[highest_key] 433 | for start, end in zip(sorted_keys[1:], sorted_keys): 434 | updated_dataset[f"{end}d-{start}d"] = dataset[start] 435 | self.data_for_report[key] = updated_dataset 436 | self.create_trending_graph(self.data_for_report[key], self.events_trending_path, "Trending events") 437 | self.logger.debug(" Created {}".format(self.events_trending_path)) 438 | else: 439 | self.events_trending_path = self.noimage_path 440 | self.data_for_report[key] = {} 441 | self.logger.error(" Not found: {}".format(key)) 442 | 443 | # ############### Trending attributes 444 | key = "trending-attributes" 445 | if key in self.data: 446 | dataset = self.data[key] 447 | dataset2 = self.data["trending-attributes_ids"] 448 | 449 | days = int(self.config["reporting_period"].strip("d")) 450 | sorted_keys = sorted(dataset.keys(), reverse=True) 451 | updated_dataset = {} 452 | highest_key = sorted_keys[0] 453 | updated_dataset[f"{highest_key + days}d-{highest_key}d"] = [dataset[highest_key], dataset2[highest_key]] 454 | for start, end in zip(sorted_keys[1:], sorted_keys): 455 | updated_dataset[f"{end}d-{start}d"] = [dataset[start], dataset2[start]] 456 | 457 | self.data_for_report[key] = updated_dataset 458 | self.create_trending_graph_double(self.data_for_report[key], self.attributes_trending_path, "Trending attributes", "Attributes", "With to_ids") 459 | self.logger.debug(" Created {}".format(self.attributes_trending_path)) 460 | else: 461 | self.attributes_trending_path = self.noimage_path 462 | self.data_for_report[key] = {} 463 | self.logger.error(" Not found: {}".format(key)) 464 | 465 | # ############### Statistics attributes 466 | key = "statistics-attributes" 467 | if key in self.data: 468 | dataset = self.data[key] 469 | self.data_for_report[key] = dataset 470 | self.create_bar_chart(self.data_for_report[key], self.attributes_type_bar_chart_path, "Attributes type distribution ({})".format(self.config["reporting_period"]), full_width=False, value_index=0) 471 | self.create_bar_chart(self.data_for_report[key], self.attributes_type_daily_bar_chart_path, "Attributes type distribution (24h)", full_width=False, value_index=1) 472 | self.data_for_report[key] = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 473 | self.logger.debug(" Created {}".format(self.attributes_type_bar_chart_path)) 474 | self.logger.debug(" Created {}".format(self.attributes_type_daily_bar_chart_path)) 475 | else: 476 | self.attributes_type_bar_chart_path = self.noimage_path 477 | self.attributes_type_daily_bar_chart_path = self.noimage_path 478 | self.data_for_report[key] = {} 479 | self.logger.error(" Not found: {}".format(key)) 480 | 481 | # ############### Statistics threatlevel 482 | key = "statistics-threatlevel" 483 | if key in self.data: 484 | dataset = self.data[key] 485 | updated_dataset = {self.threatlevel_key_mapping[key]: value for key, value in dataset.items()} 486 | self.data_for_report[key] = updated_dataset 487 | self.create_bar_chart(self.data_for_report[key], self.threatlevel_bar_chart_path, "Threat level", value_index=-1) 488 | self.logger.debug(" Created {}".format(self.threatlevel_bar_chart_path)) 489 | else: 490 | self.threatlevel_bar_chart_path = self.noimage_path 491 | self.data_for_report[key] = {} 492 | self.logger.error(" Not found: {}".format(key)) 493 | 494 | # ############### Statistics TLP 495 | key = "statistics-tlp" 496 | if key in self.data: 497 | dataset = self.data[key] 498 | updated_dataset = {} 499 | for delkey in dataset: 500 | if delkey not in self.tlp_ignore_graph: 501 | updated_dataset[delkey] = dataset[delkey] 502 | self.data_for_report[key] = dataset 503 | self.create_pie_chart(updated_dataset, self.tlp_pie_chart_path, "TLP", colors=["red", "orange", "green", "#d3d3d3", "#e8e6e6", "gray"]) 504 | self.logger.debug(" Created {}".format(self.tlp_pie_chart_path)) 505 | else: 506 | self.tlp_pie_chart_path = self.noimage_path 507 | self.data_for_report[key] = {} 508 | self.logger.error(" Not found: {}".format(key)) 509 | 510 | # ############### Statistics key organisations 511 | key = "statistics-keyorgs" 512 | if key in self.data: 513 | dataset = self.data[key] 514 | updated_dataset = {} 515 | for uuid in dataset: 516 | if uuid in self.key_organisations: 517 | try: 518 | org = self.misp.get_organisation(uuid) 519 | if "Organisation" in org: 520 | org_name = org["Organisation"]["name"] 521 | logo = self.key_organisations[uuid]["logo"] 522 | period_events = dataset[uuid]["reporting-period"]["events"] 523 | period_attributes = dataset[uuid]["reporting-period"]["attributes"] 524 | period_attributes_ids = dataset[uuid]["reporting-period"]["attributes_ids"] 525 | today_events = dataset[uuid]["today"]["events"] 526 | today_attributes = dataset[uuid]["today"]["attributes"] 527 | today_attributes_ids = dataset[uuid]["today"]["attributes_ids"] 528 | updated_dataset[org_name] = {"logo": f"{logo}", "org_uuid": f"{uuid}", "period_events": f"{period_events}", 529 | "period_attributes": f"{period_attributes}", 530 | "period_attributes_ids": f"{period_attributes_ids}", 531 | "today_events": f"{today_events}", 532 | "today_attributes": f"{today_attributes}", 533 | "today_attributes_ids": f"{today_attributes_ids}"} 534 | else: 535 | self.logger.error("Unable to get organisation info for {}".format(uuid)) 536 | except Exception as e: 537 | self.logger.error("Unable to get organisation info for {} - {}".format(uuid, e)) 538 | self.data_for_report[key] = updated_dataset 539 | self.logger.debug(" Created {}".format(key)) 540 | else: 541 | self.data_for_report[key] = {} 542 | self.logger.error(" Not found: {}".format(key)) 543 | 544 | # ############### General statistics 545 | key = "statistics" 546 | if key in self.data: 547 | dataset = self.data[key] 548 | updated_dataset = {} 549 | days = int(self.config["reporting_period"].strip("d")) 550 | current_date = datetime.now() 551 | past_date = current_date - timedelta(days=days) 552 | reporting_period = self.config["reporting_period"] 553 | updated_dataset["period"] = f"(until {past_date.strftime('%Y-%m-%d')})" 554 | if self.config["reporting_filter"] is not None: 555 | print_misp_filters = self.config["reporting_filter"] 556 | if "trending-events" in self.data: 557 | updated_dataset["period_events"] = self.data["trending-events"][0] 558 | updated_dataset["period_attributes"] = self.data["trending-attributes"][0] 559 | updated_dataset["period_attributes_ids"] = self.data["trending-attributes_ids"][0] 560 | else: 561 | updated_dataset["period_events"] = "No data" 562 | updated_dataset["period_attributes"] = "No data" 563 | updated_dataset["period_attributes_ids"] = "No data" 564 | if "today-events" in self.data: 565 | updated_dataset["today_events"] = self.data["today-events"] 566 | updated_dataset["today_attributes"] = self.data["today-attributes"] 567 | updated_dataset["today_attributes_ids"] = self.data["today-attributes_ids"] 568 | else: 569 | updated_dataset["today_events"] = "No data" 570 | updated_dataset["today_attributes"] = "No data" 571 | updated_dataset["today_attributes_ids"] = "No data" 572 | updated_dataset["events"] = dataset["event_count"] 573 | updated_dataset["attributes"] = dataset["attribute_count"] 574 | updated_dataset["correlations"] = dataset["correlation_count"] 575 | updated_dataset["organisations"] = dataset["org_count"] 576 | updated_dataset["local_organisations"] = dataset["local_org_count"] 577 | updated_dataset["users"] = dataset["user_count"] 578 | self.data_for_report[key] = updated_dataset 579 | self.logger.debug(" Created {}".format(key)) 580 | else: 581 | self.data_for_report[key] = {} 582 | self.logger.error(" Not found: {}".format(key)) 583 | 584 | # ############### Event details 585 | key = "detailed_events" 586 | if key in self.data: 587 | dataset = self.data[key] 588 | self.data_for_report[key] = dataset 589 | self.logger.debug(" Created {}".format(key)) 590 | else: 591 | self.data_for_report[key] = {} 592 | self.logger.error(" Not found: {}".format(key)) 593 | 594 | # ############### Targeting geo 595 | key = "targeting-geo" 596 | if key in self.data and len(self.data[key]) > 0: 597 | dataset = self.data[key] 598 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 599 | if self.config["filter_geo_count"] > 0: 600 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_geo_count"]]) 601 | else: 602 | self.data_for_report[key] = sorted_data 603 | self.create_geo_targeting_map(self.data_for_report[key], self.geo_targeting_map_path) 604 | self.logger.debug(" Created {}".format(self.events_trending_path)) 605 | else: 606 | self.geo_targeting_map_path = self.noimage_path 607 | self.data_for_report[key] = {} 608 | self.logger.error(" Not found: {}".format(key)) 609 | 610 | # ############### Targeting sector 611 | key = "targeting-sector" 612 | if key in self.data and len(self.data[key]) > 0: 613 | dataset = self.data[key] 614 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 615 | if self.config["filter_sector_count"] > 0: 616 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_sector_count"]]) 617 | else: 618 | self.data_for_report[key] = sorted_data 619 | self.create_horizontal_bar_chart(self.data_for_report[key], self.sector_targeting_bar_chart_path, "Sector targeting") 620 | self.logger.debug(" Created {}".format(self.events_trending_path)) 621 | else: 622 | self.sector_targeting_bar_chart_path = self.noimage_path 623 | self.data_for_report[key] = {} 624 | self.logger.error(" Not found: {}".format(key)) 625 | 626 | # ############### TTP pattern 627 | key = "ttp_pattern" 628 | if key in self.data and len(self.data[key]) > 0: 629 | dataset = self.data[key] 630 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 631 | if self.config["filter_ttp_pattern_count"] > 0: 632 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_ttp_pattern_count"]]) 633 | else: 634 | self.data_for_report[key] = sorted_data 635 | self.logger.debug(" Created {}".format(key)) 636 | else: 637 | self.data_for_report[key] = {} 638 | self.logger.error(" Not found: {}".format(key)) 639 | 640 | # ############### TTP actors 641 | key = "ttp_actors" 642 | if key in self.data and len(self.data[key]) > 0: 643 | dataset = self.data[key] 644 | self.data_for_report[key] = dataset 645 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 646 | if self.config["filter_ttp_actors_count"] > 0: 647 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_ttp_actors_count"]]) 648 | else: 649 | self.data_for_report[key] = sorted_data 650 | self.logger.debug(" Created {}".format(key)) 651 | else: 652 | self.data_for_report[key] = {} 653 | self.logger.error(" Not found: {}".format(key)) 654 | 655 | # ############### Vulnerabilities 656 | key = "vulnerabilities" 657 | if key in self.data and len(self.data[key]) > 0: 658 | dataset = self.data[key] 659 | self.data_for_report[key] = {} 660 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1]["count"], reverse=True)) 661 | self.data_for_report[key] = sorted_data 662 | self.logger.debug(" Created {}".format(key)) 663 | else: 664 | self.data_for_report[key] = {} 665 | self.logger.error(" Not found: {}".format(key)) 666 | 667 | # ############### Labels 668 | if self.config["reporting_filter_timestamp"] == "timestamp": 669 | reporting_filter_timestamp = "recently changed" 670 | else: 671 | reporting_filter_timestamp = "published" 672 | 673 | if self.config["reporting_filter_attribute_type_ids"] == True: 674 | attributes_with_ids_or_not = "(only attributes where to_ids is set to true)" 675 | else: 676 | attributes_with_ids_or_not = "(all attributes, regardless of the to_ids flag)" 677 | 678 | template_css_file = self.template_css 679 | with open(template_css_file, "r") as f: 680 | css_content = f.read() 681 | 682 | template_file = self.template_html 683 | with open(template_file, "r") as f: 684 | html_template = f.read() 685 | 686 | template = Template(html_template) 687 | html_content = template.render( 688 | css=css_content, 689 | title="MISP Summary", 690 | logo=self.config["logo"], 691 | report_date=self.report_date, 692 | report_misp_filters=print_misp_filters, 693 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 694 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 695 | reporting_period=self.config["reporting_period"], 696 | report_misp_server=self.report_misp_server, 697 | summary=self.data_for_report.get("statistics", {}), 698 | trending_events=self.data_for_report.get("trending-events", {}), 699 | trending_attributes=self.data_for_report.get("trending-attributes", {}), 700 | detailed_events=self.data_for_report["detailed_events"], 701 | print_event_details=self.config["print_event_details"], 702 | attributes_type=self.data_for_report["statistics-attributes"], 703 | threatlevel=self.data_for_report["statistics-threatlevel"], 704 | tlp=self.data_for_report["statistics-tlp"], 705 | keyorgs=self.data_for_report["statistics-keyorgs"], 706 | target_geo=self.data_for_report["targeting-geo"], 707 | target_sector=self.data_for_report["targeting-sector"], 708 | ttp_pattern=self.data_for_report["ttp_pattern"], 709 | ttp_actors=self.data_for_report["ttp_actors"], 710 | vulnerabilities=self.data_for_report["vulnerabilities"], 711 | 712 | events_trending_path=os.path.basename(self.events_trending_path), 713 | attributes_trending_path=os.path.basename(self.attributes_trending_path), 714 | attributes_type_bar_chart_path=os.path.basename(self.attributes_type_bar_chart_path), 715 | attributes_type_daily_bar_chart_path=os.path.basename(self.attributes_type_daily_bar_chart_path), 716 | threatlevel_bar_chart_path=os.path.basename(self.threatlevel_bar_chart_path), 717 | tlp_pie_chart_path=os.path.basename(self.tlp_pie_chart_path), 718 | geo_targeting_map_path=os.path.basename(self.geo_targeting_map_path), 719 | sector_targeting_bar_chart_path=os.path.basename(self.sector_targeting_bar_chart_path), 720 | 721 | reporting_filter_timestamp=reporting_filter_timestamp, 722 | vulnerability_lookup_url=self.config["vulnerability_lookup_url"], 723 | attributes_with_ids_or_not=attributes_with_ids_or_not, 724 | cve_highlight=self.config["reporting_cve_highlight"], 725 | ) 726 | 727 | output_html_path = os.path.join(self.output_dir, "misp_summary.html") 728 | with open(output_html_path, "w") as f: 729 | f.write(html_content) 730 | return True 731 | 732 | def create_geo_targeting_map(self, data, output_path): 733 | countries = list(data.keys()) 734 | counts = list(data.values()) 735 | fig = px.choropleth( 736 | locations=countries, 737 | locationmode="country names", 738 | color=counts, 739 | labels={"color": "Count"}, 740 | ) 741 | fig.write_image(output_path) 742 | 743 | def create_horizontal_bar_chart(self, data, output_path, title): 744 | labels = list(data.keys()) 745 | values = list(data.values()) 746 | 747 | if all(v == 0 for v in values): 748 | values = [0.1] * len(values) # Avoid fully empty chart 749 | 750 | cmap = plt.colormaps['Oranges'] 751 | colors = [cmap(i / len(labels)) for i in range(len(labels))] 752 | 753 | plt.figure(figsize=(6, 4)) 754 | plt.barh(labels, values, color=colors) 755 | plt.title(title) 756 | plt.xlabel("Count") 757 | 758 | # If all values are just dummy 0.1 759 | if all(v == 0.1 for v in values): 760 | plt.text( 761 | 0.5, 0.5, 762 | "No data available", 763 | fontsize=12, ha="center", 764 | transform=plt.gca().transAxes 765 | ) 766 | # Just keep a simple 0,1 scale 767 | plt.xticks([0, 1]) 768 | else: 769 | max_val = max(values) 770 | # We'll consider integer ticks up to at least ceil of max_val 771 | max_int = int(np.ceil(max_val)) 772 | 773 | if max_int > 5: 774 | # Generate exactly 5 ticks (0 to max_val) 775 | ticks = np.linspace(0, max_val, 5) 776 | # Round them to integers if desired 777 | tick_labels = [int(round(t)) for t in ticks] 778 | plt.xticks(ticks, tick_labels) 779 | else: 780 | # Less than or equal to 5, so just show integer ticks 781 | ticks = range(0, max_int + 1) 782 | plt.xticks(ticks, ticks) 783 | 784 | plt.tight_layout() 785 | plt.savefig(output_path, dpi=100) 786 | plt.close() 787 | 788 | def create_bar_chart(self, data, output_path, title, full_width=False, value_index=0): 789 | labels = list(data.keys()) 790 | if value_index == -1: 791 | values = list(data.values()) 792 | else: 793 | values = [v[value_index] for v in data.values()] 794 | 795 | if all(v == 0 for v in values): 796 | values = [0.1] * len(values) # Avoid fully empty chart 797 | 798 | figsize = (8, 4) if full_width else (4, 3) 799 | 800 | cmap = plt.colormaps['Oranges'] 801 | colors = [cmap(i / len(labels)) for i in range(len(labels))] 802 | 803 | plt.figure(figsize=figsize) 804 | plt.bar(labels, values, color=colors) 805 | plt.title(title, fontsize=10) 806 | plt.ylabel("Count", fontsize=8) 807 | 808 | ax = plt.gca() 809 | ax.xaxis.set_major_locator(MultipleLocator(1)) 810 | plt.xticks(fontsize=8, rotation=45) 811 | plt.yticks(fontsize=8) 812 | 813 | # If dummy values were used, indicate it 814 | if all(v == 0.1 for v in values): 815 | plt.text(0.5, 0.5, "No data available", fontsize=12, ha="center", transform=plt.gca().transAxes) 816 | 817 | plt.tight_layout() 818 | plt.savefig(output_path, dpi=100) 819 | plt.close() 820 | 821 | def create_pie_chart(self, data, output_path, title, colors): 822 | labels = list(data.keys()) 823 | sizes = list(data.values()) 824 | 825 | if sum(sizes) == 0: 826 | self.logger.info("No data to display in pie chart.") 827 | plt.figure(figsize=(4, 3)) 828 | plt.pie( 829 | [1], # Single value to create a full circle 830 | labels=["No data"], 831 | colors=["lightgrey"], 832 | startangle=90, 833 | wedgeprops={'width': 0.4} 834 | ) 835 | plt.title(title, fontsize=10) 836 | plt.axis("equal") 837 | plt.tight_layout() 838 | plt.savefig(output_path, dpi=100) 839 | plt.close() 840 | else: 841 | plt.figure(figsize=(4, 3)) 842 | plt.pie( 843 | sizes, 844 | labels=labels, 845 | autopct="%1.1f%%", 846 | startangle=90, 847 | colors=colors, 848 | wedgeprops={'width': 0.4} 849 | ) 850 | plt.title(title, fontsize=10) 851 | plt.axis("equal") 852 | plt.tight_layout() 853 | plt.savefig(output_path, dpi=100) 854 | plt.close() 855 | 856 | def create_trending_graph(self, data, output_path, title): 857 | months = list(data.keys()) 858 | values = list(data.values()) 859 | 860 | plt.figure(figsize=(4, 3)) 861 | plt.plot(months, values, marker="o", color="#FF6600") 862 | plt.title(title, fontsize=10) 863 | plt.ylabel("Count", fontsize=8) 864 | plt.xticks(fontsize=8, rotation=45) 865 | plt.yticks(fontsize=8) 866 | plt.grid(True, linestyle='--', alpha=0.7) 867 | plt.tight_layout() 868 | plt.savefig(output_path, dpi=100) 869 | plt.close() 870 | 871 | def create_trending_graph_double(self, data, output_path, title, label1, label2): 872 | months = list(data.keys()) 873 | pairs = list(data.values()) 874 | 875 | # Separate out the first/second data points in each pair 876 | first_values = [p[0] for p in pairs] 877 | second_values = [p[1] for p in pairs] 878 | 879 | plt.figure(figsize=(4, 3)) 880 | plt.plot(months, first_values, marker="o", color="#ffcc00", label=label1) 881 | plt.plot(months, second_values, marker="o", color="#ff1e00", label=label2) 882 | 883 | plt.title(title, fontsize=10) 884 | plt.ylabel("Count", fontsize=8) 885 | plt.xticks(fontsize=8, rotation=45) 886 | plt.yticks(fontsize=8) 887 | plt.grid(True, linestyle='--', alpha=0.7) 888 | plt.legend(fontsize=8) 889 | plt.tight_layout() 890 | plt.savefig(output_path, dpi=100) 891 | plt.close() 892 | -------------------------------------------------------------------------------- /clsReportingData.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta, timezone 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | import numpy as np 14 | 15 | 16 | class ReportingData(): 17 | def __init__(self, config, logger): 18 | self.filters = config["reporting_filter"] 19 | self.config = config 20 | self.logger = logger 21 | self.misp_headers = {"Authorization": self.config["misp_key"], "Content-Type": "application/json", "Accept": "application/json"} 22 | if self.config["misp_verifycert"] is False: 23 | import urllib3 24 | urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 25 | self.misp = PyMISP(self.config["misp_url"], self.config["misp_key"], self.config["misp_verifycert"]) 26 | self.data = {} 27 | self.data_for_reporting_period = False 28 | self.data_for_today = False 29 | 30 | self.attribute_summary = self.config["attribute_summary"] 31 | self.attribute_other = self.config["attribute_other"] 32 | self.key_organisations = self.config["key_organisations"] 33 | self.threatlevel_key_mapping = self.config["threatlevel_key_mapping"] 34 | 35 | self.workflow_complete = self.config["workflow_complete"] 36 | self.workflow_incomplete = self.config["workflow_incomplete"] 37 | 38 | self.filter_sector = self.config["filter_sector"] 39 | self.filter_geo = self.config["filter_geo"] 40 | 41 | self.filter_ttp_actors = self.config["filter_ttp_actors"] 42 | self.filter_ttp_pattern = self.config["filter_ttp_pattern"] 43 | 44 | self.misp_infrastructure_monitor = [{"This server": {"misp_url": self.config["misp_url"], 45 | "misp_verifycert": self.config["misp_verifycert"], 46 | "misp_key": self.config["misp_key"]}}] + self.config["misp_infrastructure_monitor"] 47 | 48 | def print(self): 49 | print(self.data) 50 | 51 | def get_statistics(self): 52 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 53 | 54 | statistics = self._request_get("/users/statistics") 55 | self.data["statistics"] = {"user_count": 0, "org_count": 0, "local_org_count": 0, "event_count": 0, "attribute_count": 0, 56 | "correlation_count": 0} 57 | if statistics and statistics.json().get("stats", False): 58 | json_statistics = statistics.json()["stats"] 59 | self.data["statistics"]["user_count"] = json_statistics["user_count"] 60 | self.data["statistics"]["org_count"] = json_statistics["org_count"] 61 | self.data["statistics"]["local_org_count"] = json_statistics["local_org_count"] 62 | self.data["statistics"]["event_count"] = json_statistics["event_count"] 63 | self.data["statistics"]["attribute_count"] = json_statistics["attribute_count"] 64 | self.data["statistics"]["correlation_count"] = json_statistics["correlation_count"] 65 | 66 | def get_today_events_attributes(self): 67 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 68 | 69 | self.data["today-events"] = {} 70 | self.data["today-attributes"] = {} 71 | self.data["today-attributes_ids"] = {} 72 | 73 | response = self._get_data_for_today() 74 | self.data["today-events"] = len(response) 75 | 76 | attributesqt = 0 77 | attributesqt_ids = 0 78 | for event in response: 79 | attributesqt += len(event["Event"]["Attribute"]) 80 | for attr in event["Event"]["Attribute"]: 81 | if attr["to_ids"] == 1: 82 | attributesqt_ids += 1 83 | for misp_object in event["Event"]["Object"]: 84 | attributesqt += len(misp_object["Attribute"]) 85 | for attr in misp_object["Attribute"]: 86 | if attr["to_ids"] == 1: 87 | attributesqt_ids += 1 88 | self.data["today-attributes"] = attributesqt 89 | self.data["today-attributes_ids"] = attributesqt_ids 90 | 91 | def get_trending_events_attributes(self): 92 | # Get the trends data 93 | # - First the data for the reporting period 94 | # - Then the data for the last reporting_trending_count (fe. 6) periods 95 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 96 | 97 | self.data["trending-events"] = {} 98 | self.data["trending-attributes"] = {} 99 | self.data["trending-attributes_ids"] = {} 100 | 101 | days = int(''.join(filter(str.isdigit, self.config["reporting_period"]))) 102 | self.logger.debug(" Get {}".format(days)) 103 | response = self._get_data_for_reporting_period() 104 | self.data["trending-events"][0] = len(response) 105 | 106 | attributesqt = 0 107 | attributesqt_ids = 0 108 | for event in response: 109 | attributesqt += len(event["Event"]["Attribute"]) 110 | for attr in event["Event"]["Attribute"]: 111 | if attr["to_ids"] == 1: 112 | attributesqt_ids += 1 113 | for misp_object in event["Event"]["Object"]: 114 | attributesqt += len(misp_object["Attribute"]) 115 | for attr in misp_object["Attribute"]: 116 | if attr["to_ids"] == 1: 117 | attributesqt_ids += 1 118 | self.data["trending-attributes"][0] = attributesqt 119 | self.data["trending-attributes_ids"][0] = attributesqt_ids 120 | 121 | count = 1 122 | while self.config["reporting_trending_count"] > count: 123 | start_period = days * count 124 | end_period = days * (count + 1) 125 | #timestamp_filter = ["{}d".format(start_period), "{}d".format(end_period)] 126 | self.logger.debug(" Get {} - {}".format(start_period, end_period)) 127 | 128 | current_page = 1 129 | tmp_len = 0 130 | response = [] 131 | while True: 132 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], ["{}d".format(start_period), "{}d".format(end_period)]) 133 | tmp_response = self.misp.search("events", **filter_params) 134 | if len(tmp_response) > 0: 135 | tmp_len = tmp_len + len(tmp_response) 136 | response += tmp_response 137 | else: 138 | break 139 | current_page += 1 140 | self.data["trending-events"][start_period] = tmp_len 141 | 142 | attributesqt = 0 143 | attributesqt_ids = 0 144 | for event in response: 145 | attributesqt += len(event["Event"]["Attribute"]) 146 | for attr in event["Event"]["Attribute"]: 147 | if attr["to_ids"] == 1: 148 | attributesqt_ids += 1 149 | for misp_object in event["Event"]["Object"]: 150 | attributesqt += len(misp_object["Attribute"]) 151 | for attr in misp_object["Attribute"]: 152 | if attr["to_ids"] == 1: 153 | attributesqt_ids += 1 154 | self.data["trending-attributes"][start_period] = attributesqt 155 | self.data["trending-attributes_ids"][start_period] = attributesqt_ids 156 | 157 | count += 1 158 | 159 | def get_statistics_attributes(self): 160 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 161 | self.data["statistics-attributes"] = {} 162 | 163 | response_reporting_period = self._get_data_for_reporting_period() 164 | self._process_attribute_counts(response_reporting_period, index=0) 165 | response_today = self._get_data_for_today() 166 | self._process_attribute_counts(response_today, index=1) 167 | 168 | def _process_attribute_counts(self, events, index): 169 | for event in events: 170 | for attribute in event["Event"]["Attribute"]: 171 | if self.config["reporting_filter_attribute_type_ids"] and attribute["to_ids"] == 0: 172 | continue 173 | 174 | attribute_type = self._convert_attribute_category(attribute["type"]) 175 | if attribute_type and attribute_type not in self.data["statistics-attributes"]: 176 | self.data["statistics-attributes"][attribute_type] = [0, 0] 177 | self.data["statistics-attributes"][attribute_type][index] += 1 178 | 179 | for misp_object in event["Event"]["Object"]: 180 | for attr in misp_object["Attribute"]: 181 | if self.config["reporting_filter_attribute_type_ids"] and attr["to_ids"] == 0: 182 | continue 183 | 184 | attribute_type = self._convert_attribute_category(attr["type"]) 185 | if attribute_type and attribute_type not in self.data["statistics-attributes"]: 186 | self.data["statistics-attributes"][attribute_type] = [0, 0] 187 | self.data["statistics-attributes"][attribute_type][index] += 1 188 | 189 | def get_statistics_keyorgs(self): 190 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 191 | self.data["statistics-keyorgs"] = {} 192 | 193 | org_uuid_list = list(self.key_organisations) 194 | if len(org_uuid_list) > 0: 195 | for orgc in org_uuid_list: 196 | self.data["statistics-keyorgs"][orgc] = {"reporting-period": {"events": 0, "attributes": 0, "attributes_ids": 0}, 197 | "today": {"events": 0, "attributes": 0, "attributes_ids": 0}} 198 | 199 | response = self._get_data_for_reporting_period() 200 | self._process_get_statistics_keyorgs(response, "reporting-period") 201 | response = self._get_data_for_today() 202 | self._process_get_statistics_keyorgs(response, "today") 203 | 204 | def get_threatlevel(self): 205 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 206 | self.data["statistics-threatlevel"] = {"1": 0, "2": 0, "3": 0, "4": 0} 207 | response = self._get_data_for_reporting_period() 208 | for event in response: 209 | threat_level_id = event["Event"]["threat_level_id"] 210 | self.data["statistics-threatlevel"][threat_level_id] += 1 211 | 212 | def get_tlplevel(self): 213 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 214 | self.data["statistics-tlp"] = {"tlp:red": 0, "tlp:amber": 0, "tlp:amber+strict": 0, "tlp:green": 0, "tlp:clear": 0, "tlp:ex:chr": 0, "tlp:unclear": 0} 215 | response = self._get_data_for_reporting_period() 216 | 217 | for event in response: 218 | tags = event["Event"].get("Tag", []) 219 | if len(tags) > 0: 220 | for tag in tags: 221 | if tag["name"].startswith("tlp:"): 222 | if tag["name"] == "tlp:white": 223 | tag_tlp = "tlp:clear" 224 | else: 225 | tag_tlp = tag["name"] 226 | self.data["statistics-tlp"][tag_tlp] += 1 227 | 228 | def get_eventdetails(self): 229 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 230 | self.data["detailed_events"] = [] 231 | response = self._get_data_for_reporting_period() 232 | self.distribution = self.config["distribution"] 233 | self.analysis_state = self.config["analysis"] 234 | 235 | for event in response: 236 | process_event = False 237 | if self.config["reporting_eventdetails_onlykeyorgs"]: 238 | if event["Event"]["Orgc"]["uuid"] in self.config["key_organisations"]: 239 | process_event = True 240 | else: 241 | process_event = True 242 | if process_event: 243 | tags = event["Event"].get("Tag", []) 244 | tag_tlp = "" 245 | if len(tags) > 0: 246 | for tag in tags: 247 | if tag["name"].startswith("tlp:"): 248 | if tag["name"] == "tlp:white": 249 | tag_tlp = "tlp:clear" 250 | else: 251 | tag_tlp = tag["name"] 252 | entry = {"date": event["Event"]["date"], 253 | "id": event["Event"]["id"], 254 | "org": event["Event"]["Orgc"]["name"], 255 | "distribution": self.distribution[int(event["Event"]["distribution"])], 256 | "analysis": self.analysis_state[int(event["Event"]["analysis"])], 257 | "published": event["Event"]["published"], 258 | "threat_level": self.threatlevel_key_mapping[event["Event"]["threat_level_id"]], 259 | "tlp": tag_tlp, 260 | "info": event["Event"]["info"][:30], 261 | "indicators": event["Event"]["attribute_count"]} 262 | self.data["detailed_events"].append(entry) 263 | 264 | def get_ttp(self): 265 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 266 | self.data["ttp_pattern"] = {} 267 | self.data["ttp_actors"] = {} 268 | response = self._get_data_for_reporting_period() 269 | for event in response: 270 | tags = event["Event"].get("Tag", []) 271 | if len(tags) > 0: 272 | for tag in tags: 273 | for ttp in self.filter_ttp_pattern: 274 | if ttp in tag["name"]: 275 | item = tag["name"].split("{}=".format(ttp))[1].replace("\"", "") 276 | if item in self.data["ttp_pattern"]: 277 | self.data["ttp_pattern"][item] += 1 278 | else: 279 | self.data["ttp_pattern"][item] = 1 280 | for ttp in self.filter_ttp_actors: 281 | if ttp in tag["name"]: 282 | item = tag["name"].split("{}=".format(ttp))[1].replace("\"", "") 283 | if item in self.data["ttp_actors"]: 284 | self.data["ttp_actors"][item] += 1 285 | else: 286 | self.data["ttp_actors"][item] = 1 287 | 288 | def get_target_sector(self): 289 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 290 | self.data["targeting-sector"] = {} 291 | response = self._get_data_for_reporting_period() 292 | for event in response: 293 | tags = event["Event"].get("Tag", []) 294 | if len(tags) > 0: 295 | for tag in tags: 296 | if self.filter_sector in tag["name"]: 297 | sector = tag["name"].split("{}=".format(self.filter_sector))[1].replace("\"", "") 298 | if sector in self.data["targeting-sector"]: 299 | self.data["targeting-sector"][sector] += 1 300 | else: 301 | self.data["targeting-sector"][sector] = 1 302 | 303 | def get_target_geo(self): 304 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 305 | self.data["targeting-geo"] = {} 306 | response = self._get_data_for_reporting_period() 307 | for event in response: 308 | tags = event["Event"].get("Tag", []) 309 | if len(tags) > 0: 310 | for tag in tags: 311 | if self.filter_geo in tag["name"]: 312 | geo = tag["name"].split("{}=".format(self.filter_geo))[1].replace("\"", "") 313 | if geo in self.data["targeting-geo"]: 314 | self.data["targeting-geo"][geo] += 1 315 | else: 316 | self.data["targeting-geo"][geo] = 1 317 | 318 | def get_vulnerabilities(self): 319 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 320 | self.data["vulnerabilities"] = {} 321 | tmp_data = {} 322 | response = self._get_data_for_reporting_period() 323 | for event in response: 324 | for attribute in event["Event"]["Attribute"]: 325 | if attribute["type"] == "vulnerability": 326 | if attribute["value"] in tmp_data: 327 | tmp_data[attribute["value"]] += 1 328 | else: 329 | tmp_data[attribute["value"]] = 1 330 | for misp_object in event["Event"]["Object"]: 331 | for attribute in misp_object["Attribute"]: 332 | if attribute["type"] == "vulnerability": 333 | if attribute["value"] in tmp_data: 334 | tmp_data[attribute["value"]] += 1 335 | else: 336 | tmp_data[attribute["value"]] = 1 337 | 338 | for cve in tmp_data: 339 | cve_url = self.config["cve_url"] 340 | cvss_base_score = "?" 341 | cve_summary = "" 342 | try: 343 | response = requests.get(f"{cve_url}/{cve}") 344 | cve_data = response.json() 345 | 346 | containers = cve_data.get("containers", {}) 347 | cna = containers.get("cna", False) # CVE Numbering Authorities 348 | cve_summary = cna["descriptions"][0]["value"] 349 | adp = containers.get("adp", False) # Authorized Data Publisher 350 | if cna and "metrics" in cna and "cvssV3_1" in cna["metrics"][0]: 351 | cvss_base_score = cna["metrics"][0]["cvssV3_1"]["baseScore"] 352 | elif adp: 353 | for section in adp: 354 | metrics = section.get("metrics", []) 355 | for metric in metrics: 356 | if "cvssV3_1" in metric: 357 | cvss_base_score = metric["cvssV3_1"]["baseScore"] 358 | break 359 | else: 360 | continue 361 | break 362 | except Exception as e: 363 | self.logger.debug("Unable to get CVE details for {} - {}".format(cve, e)) 364 | entry = {"count": tmp_data[cve], "summary": cve_summary, "cvss3": cvss_base_score} 365 | self.data["vulnerabilities"][cve] = entry 366 | 367 | def get_curation(self): 368 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 369 | self.data["curation_complete"] = [] 370 | self.data["curation_complete_today"] = [] 371 | self.data["curation_incomplete"] = [] 372 | self.data["curation_incomplete_today"] = [] 373 | self.data["curation_complete_date"] = {} 374 | self.data["curation_incomplete_date"] = {} 375 | self.data["curation_orgs_complete"] = {} 376 | self.data["curation_orgs_incomplete"] = {} 377 | self.data["curation_incomplete_high"] = [] 378 | self.data["curation_incomplete_adm_high"] = [] 379 | 380 | self.distribution = self.config["distribution"] 381 | self.analysis_state = self.config["analysis"] 382 | 383 | # Reset to get curation data (published, and not published) 384 | self.data_for_reporting_period = None 385 | response = self._get_data_for_reporting_period(published=None) 386 | 387 | for event in response: 388 | complete_event = False 389 | entry = {"date": event["Event"]["date"], 390 | "id": event["Event"]["id"], 391 | "org": event["Event"]["Orgc"]["name"], 392 | "info": event["Event"]["info"][:30], 393 | "indicators": event["Event"]["attribute_count"]} 394 | if event["Event"]["published"]: 395 | tags = event["Event"].get("Tag", []) 396 | if len(tags) > 0: 397 | for tag in tags: 398 | if tag["name"] == self.workflow_complete: 399 | publish_timestamp_str = event["Event"]["publish_timestamp"] 400 | publish_timestamp = int(publish_timestamp_str) 401 | publish_time = datetime.fromtimestamp(publish_timestamp, tz=timezone.utc) 402 | now = datetime.now(timezone.utc) 403 | if (now - publish_time) < timedelta(days=1): 404 | self.data["curation_complete_today"].append(entry) 405 | self.data["curation_complete"].append(entry) 406 | 407 | if event["Event"]["date"] in self.data["curation_complete_date"]: 408 | self.data["curation_complete_date"][event["Event"]["date"]] += 1 409 | else: 410 | self.data["curation_complete_date"][event["Event"]["date"]] = 1 411 | if event["Event"]["Orgc"]["name"] in self.data["curation_orgs_complete"]: 412 | self.data["curation_orgs_complete"][event["Event"]["Orgc"]["name"]] += 1 413 | else: 414 | self.data["curation_orgs_complete"][event["Event"]["Orgc"]["name"]] = 1 415 | complete_event = True 416 | break 417 | 418 | if not complete_event: 419 | if self.config["log_incomplete"]: 420 | self.logger.debug("Consider event {} {} as incomplete".format(event["Event"]["id"], event["Event"]["info"])) 421 | publish_timestamp_str = event["Event"]["publish_timestamp"] 422 | publish_timestamp = int(publish_timestamp_str) 423 | publish_time = datetime.fromtimestamp(publish_timestamp, tz=timezone.utc) 424 | now = datetime.now(timezone.utc) 425 | if (now - publish_time) < timedelta(days=1): 426 | self.data["curation_incomplete_today"].append(entry) 427 | 428 | # Additional reporting for curation of today 429 | if event["Event"]["threat_level_id"] == "1": 430 | self.data["curation_incomplete_high"].append(entry) 431 | tags = event["Event"].get("Tag", []) 432 | for tag in tags: 433 | if tag["name"] == "admiralty-scale:source-reliability=\"a\"": 434 | self.data["curation_incomplete_adm_high"].append(entry) 435 | break 436 | 437 | self.data["curation_incomplete"].append(entry) 438 | 439 | if event["Event"]["date"] in self.data["curation_incomplete_date"]: 440 | self.data["curation_incomplete_date"][event["Event"]["date"]] += 1 441 | else: 442 | self.data["curation_incomplete_date"][event["Event"]["date"]] = 1 443 | if event["Event"]["Orgc"]["name"] in self.data["curation_orgs_incomplete"]: 444 | self.data["curation_orgs_incomplete"][event["Event"]["Orgc"]["name"]] += 1 445 | else: 446 | self.data["curation_orgs_incomplete"][event["Event"]["Orgc"]["name"]] = 1 447 | 448 | def get_infrastructure(self): 449 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 450 | self.data["infrastructure_misp"] = False 451 | results = {} 452 | 453 | for entry in self.misp_infrastructure_monitor: 454 | for name, config in entry.items(): 455 | self.logger.info("Checking MISP server: {}".format(name)) 456 | results[name] = { 457 | "url": config.get("misp_url", False), 458 | "status": {}, 459 | "server": {}, 460 | "remote_servers": [] 461 | } 462 | try: 463 | misp_server = PyMISP(config["misp_url"], config["misp_key"], config["misp_verifycert"]) 464 | if not misp_server: 465 | self.logger.error("Unable to connect to MISP server: {}".format(name)) 466 | results[name]["status"] = "ERROR" 467 | results[name]["remote_servers"] = [] 468 | continue 469 | 470 | server_settings = misp_server.server_settings() 471 | if "errors" in server_settings: 472 | results[name]["status"] = "ERROR" 473 | results[name]["server"] = config["misp_url"] 474 | self.logger.error("Unable to access {}, {}".format(config["misp_url"], server_settings)) 475 | continue 476 | 477 | if server_settings: 478 | self.logger.info("MISP server {} is reachable".format(name)) 479 | workers = server_settings.get("workers", {}) 480 | workers_report = {"cache": "ERROR", "default": "ERROR"} 481 | redis_report = "ERROR" 482 | if "cache" in workers and workers["cache"].get("ok") is True: 483 | workers_report["cache"] = "OK" 484 | if "default" in workers and workers["default"].get("ok") is True: 485 | workers_report["default"] = "OK" 486 | if "redisInfo" in server_settings and server_settings["redisInfo"].get("connection") is True: 487 | redis_report = "OK" 488 | server = { 489 | "version": server_settings["version"]["current"], 490 | "zmqStatus": server_settings.get("zmqStatus"), 491 | "moduleStatus_enrichment": server_settings["moduleStatus"].get("Enrichment"), 492 | "moduleStatus_import": server_settings["moduleStatus"].get("Import"), 493 | "moduleStatus_export": server_settings["moduleStatus"].get("Export"), 494 | "workers": workers_report, 495 | "redis": redis_report 496 | } 497 | results[name]["status"] = "OK" 498 | results[name]["server"] = server 499 | 500 | remote_servers = misp_server.servers(pythonify=True) 501 | results_remote_servers = [] 502 | for server in remote_servers: 503 | self.logger.info("Checking remote MISP server: {}".format(server["name"])) 504 | try: 505 | server_status = misp_server.test_server(server) 506 | status_message = "" 507 | status_message_extra = "" 508 | status_color = None 509 | if "status" in server_status: 510 | status = server_status["status"] 511 | if status == 1: 512 | status_message = "OK" 513 | status_color = "green" 514 | elif status == 2: 515 | status_message = "Server unreachable" 516 | status_message_extra = "There seems to be a connection issue. Make sure that the entered URL is correct and that the certificates are in order." 517 | status_color = "red" 518 | elif status == 3: 519 | status_message = "Unexpected error" 520 | status_message_extra = "The server returned an unexpected result. Make sure that the provided URL (or certificate if it applies) are correct." 521 | status_color = "red" 522 | elif status == 4: 523 | status_message = "Authentication failed" 524 | status_message_extra = "Authentication failed due to incorrect authentication key or insufficient privileges on the remote instance." 525 | status_color = "red" 526 | elif status == 5: 527 | status_message = "Password change required" 528 | status_message_extra = "Authentication failed because the sync user is expected to change passwords. Log into the remote MISP to rectify this." 529 | status_color = "red" 530 | elif status == 6: 531 | status_message = "Terms not accepted" 532 | status_message_extra = "Authentication failed because the sync user on the remote has not accepted the terms of use. Log into the remote MISP to rectify this." 533 | status_color = "red" 534 | elif status == 7: 535 | status_message = "Remote user not a sync user, only pulling events is available" 536 | status_color = "orange" 537 | elif status == 8: 538 | status_message = "Remote user not a sync user, only pulling events is available. Pushing available for sightings only" 539 | status_color = "orange" 540 | results_remote_servers.append({ 541 | "name": server["name"], 542 | "url": server["url"], 543 | "status": status_message, 544 | "status_color": status_color, 545 | "status_extra": status_message_extra, 546 | "id": server["id"], 547 | }) 548 | except Exception as e: 549 | self.logger.error("Error checking remote server {}: {}".format(server["name"], e)) 550 | results_remote_servers.append({ 551 | "name": server["name"], 552 | "url": server["url"], 553 | "status": "ERROR", 554 | "status_color": "red", 555 | "status_extra": str(e), 556 | "id": server.get("id", None), 557 | }) 558 | results[name]["remote_servers"] = results_remote_servers 559 | 560 | except Exception as e: 561 | self.logger.error("Unable to connect to MISP server {}: {}".format(name, e)) 562 | results[name]["status"] = "ERROR" 563 | results[name]["remote_servers"] = [] 564 | 565 | self.data["infrastructure_misp"] = results 566 | 567 | def _process_get_statistics_keyorgs(self, response, period): 568 | for event in response: 569 | orgc = event["Event"]["Orgc"]["uuid"] 570 | if self.key_organisations.get(orgc, False): 571 | attributesqt = 0 572 | attributesqt_ids = 0 573 | attributesqt = len(event["Event"]["Attribute"]) 574 | for attr in event["Event"]["Attribute"]: 575 | if attr["to_ids"] == 1: 576 | attributesqt_ids += 1 577 | 578 | for misp_object in event["Event"]["Object"]: 579 | attributesqt += len(misp_object["Attribute"]) 580 | for attr in misp_object["Attribute"]: 581 | if attr["to_ids"] == 1: 582 | attributesqt_ids += 1 583 | 584 | self.data["statistics-keyorgs"][orgc][period]["events"] += 1 585 | self.data["statistics-keyorgs"][orgc][period]["attributes"] += attributesqt 586 | self.data["statistics-keyorgs"][orgc][period]["attributes_ids"] += attributesqt_ids 587 | 588 | def _build_misp_filter(self, current_page, published, date_filter): 589 | filter_params = { 590 | "limit": self.config["misp_page_size"], 591 | "page": current_page, 592 | "published": published, 593 | "tags": self.config["reporting_filter"] 594 | } 595 | if self.config["reporting_filter_timestamp"] == "timestamp": 596 | filter_params["timestamp"] = date_filter 597 | filter_params["published"] = None 598 | elif self.config["reporting_filter_timestamp"] == "published": 599 | filter_params["publish_timestamp"] = date_filter 600 | else: # default to published 601 | filter_params["publish_timestamp"] = date_filter 602 | return filter_params 603 | 604 | def _get_data_for_reporting_period(self, published=True): 605 | response = [] 606 | if not self.data_for_reporting_period: 607 | current_page = 1 608 | while True: 609 | if not published: 610 | filter_params = self._build_misp_filter(current_page, published, self.config["reporting_period"]) 611 | else: 612 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], self.config["reporting_period"]) 613 | tmp_reponse = self.misp.search("events", **filter_params) 614 | if len(tmp_reponse) > 0: 615 | response = response + tmp_reponse 616 | else: 617 | break 618 | current_page += 1 619 | self.data_for_reporting_period = response 620 | return self.data_for_reporting_period 621 | 622 | def _get_data_for_today(self, published=True): 623 | response = [] 624 | if not self.data_for_today: 625 | current_page = 1 626 | while True: 627 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], "1d") 628 | tmp_reponse = self.misp.search("events", **filter_params) 629 | if len(tmp_reponse) > 0: 630 | response = response + tmp_reponse 631 | else: 632 | break 633 | current_page += 1 634 | self.data_for_today = response 635 | return self.data_for_today 636 | 637 | def _convert_attribute_category(self, category): 638 | found_key = None 639 | for key, values in self.attribute_summary.items(): 640 | if category in values: 641 | found_key = key 642 | break 643 | if found_key: 644 | return found_key 645 | return self.attribute_other 646 | 647 | def _request_get(self, endpoint): 648 | response = requests.get("{}/{}".format(self.config["misp_url"], endpoint), headers=self.misp_headers, verify=self.config["misp_verifycert"]) 649 | if response.ok: 650 | return response 651 | elif 400 <= response.status_code < 500: 652 | self.logger.error(f"[{response.status_code}] Client Error: {response.reason}") 653 | elif 500 <= response.status_code < 600: 654 | self.logger.error(f"[{response.status_code}] Server Error: {response.reason}") 655 | else: 656 | self.logger.error(f"[{response.status_code}] Other: {response.reason}") 657 | return False 658 | 659 | def get_misp_statistics(self): 660 | statistics = {"event_count": self.data["statistics"]["event_count"], 661 | "attribute_count": self.data["statistics"]["attribute_count"], 662 | "user_count": self.data["statistics"]["user_count"], 663 | "org_count": self.data["statistics"]["org_count"], 664 | "local_org_count": self.data["statistics"]["local_org_count"]} 665 | 666 | today_statistics = {"today_event_count": self.data["today-attributes"], 667 | "today_attribute_count": self.data["today-attributes"], 668 | "today_attribute_ids_count": self.data["today-attributes_ids"]} 669 | 670 | filtered_statistics_attributes = { 671 | key: value[1] 672 | #for key, value in self.data["statistics-attributes"].items() 673 | for key, value in self.data.get("statistics-attributes", {}).items() 674 | } 675 | 676 | self.statistics = statistics 677 | self.today_statistics = today_statistics 678 | self.statistics_attributes = filtered_statistics_attributes 679 | 680 | def _process_get_contributors_orgs(self, response, period): 681 | for event in response: 682 | orgc = event["Event"]["Orgc"]["uuid"] 683 | orgc_id = event["Event"]["Orgc"]["id"] 684 | if orgc_id in self.data["contributor_orgs"]: 685 | self.data["contributor_orgs"][orgc_id][period] += 1 686 | self.data["contributor_orgs"][orgc_id]["uuid"] = orgc 687 | 688 | def get_contributors(self): 689 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 690 | self.data["contributor_orgs"] = {} 691 | 692 | contributor_orgs = self._request_get("/users/statistics/orgs/scope:all") 693 | if contributor_orgs and len(contributor_orgs.json()) > 0: 694 | contributor_orgs_json = contributor_orgs.json() 695 | for contributor_org in contributor_orgs_json: 696 | if "eventCount" in contributor_orgs_json[contributor_org]: 697 | entry = { 698 | "name": contributor_orgs_json[contributor_org]["name"], 699 | "id": contributor_orgs_json[contributor_org]["id"], 700 | "uuid": False, 701 | "all_time": contributor_orgs_json[contributor_org]["eventCount"], 702 | "reporting-period": 0, 703 | "today": 0 704 | } 705 | self.data["contributor_orgs"][contributor_orgs_json[contributor_org]["id"]] = entry 706 | 707 | response = self._get_data_for_reporting_period() 708 | self._process_get_contributors_orgs(response, "reporting-period") 709 | response = self._get_data_for_today() 710 | self._process_get_contributors_orgs(response, "today") 711 | 712 | sorted_data = dict(sorted( 713 | self.data["contributor_orgs"].items(), key=lambda item: int(item[1]["all_time"]), reverse=True)) 714 | self.data["contributor_orgs"] = sorted_data 715 | -------------------------------------------------------------------------------- /clsReportingStatistics.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | from matplotlib.ticker import MultipleLocator 14 | import numpy as np 15 | from collections import defaultdict 16 | 17 | 18 | class ReportingStatistics: 19 | def __init__(self, config, logger, statistics, today_statistics, statistics_attributes, output_dir="report_output"): 20 | self.config = config 21 | self.logger = logger 22 | 23 | self.statistics = statistics 24 | self.today_statistics = today_statistics 25 | self.statistics_attributes = statistics_attributes 26 | 27 | self.report_date = datetime.now().strftime("%Y-%m-%d") 28 | self.report_misp_server = self.config["misp_url"] 29 | self.reporting_period = self.config["reporting_period"] 30 | self.output_dir = self.config["output_dir"] 31 | 32 | def render_statistics(self): 33 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 34 | 35 | statistics_json = {"misp_server": self.report_misp_server, 36 | "report_date": self.report_date, 37 | "statistics": f"{self.statistics}", 38 | "today_statistics": f"{self.today_statistics}", 39 | "today_statistics_attributes": f"{self.statistics_attributes}", 40 | } 41 | 42 | try: 43 | json_file = "{}/statistics.json".format(self.config["output_dir"]) 44 | with open(json_file, "w") as out_file: 45 | json.dump(statistics_json, out_file, indent=4) 46 | except Exception as e: 47 | self.logger.error("Error writing statistics json file: {}".format(e)) 48 | self.logger.info("Render statistics") 49 | -------------------------------------------------------------------------------- /config.py.default: -------------------------------------------------------------------------------- 1 | config = { 2 | "logfile": "/var/www/MISP/app/tmp/logs/misp-reporting.log", 3 | "logname": "misp-reporting", 4 | "log_incomplete": False, 5 | 6 | "reporting_period": "30d", 7 | "reporting_filter": None, 8 | "reporting_eventdetails_onlykeyorgs": True, 9 | "reporting_trending_count": 6, 10 | "reporting_filter_attribute_type_ids": True, 11 | "reporting_filter_timestamp": "published", 12 | "reporting_filter_published": True, 13 | 14 | "print_event_details": False, 15 | 16 | "misp_url": "", 17 | "misp_verifycert": False, 18 | "misp_key": "", 19 | "misp_page_size": 200, 20 | 21 | "misp_infrastructure_monitor": [ 22 | #{ 23 | #"servername": {"misp_url": "https://misp", "misp_verifycert": False, "misp_key": "1111111111111111111111111111111111111111"}, 24 | #} 25 | ], 26 | 27 | "cve_url": "https://cvepremium.circl.lu/api/cve/", 28 | "vulnerability_lookup_url": "https://vulnerability.circl.lu/vuln/", 29 | 30 | "reporting_cve_highlight": 7, 31 | 32 | "filter_sector": "misp-galaxy:sector", 33 | "filter_sector_count": 10, 34 | "filter_geo": "misp-galaxy:target-information", 35 | "filter_geo_count": 10, 36 | "filter_ttp_actors": ["misp-galaxy:threat-actor", "misp-galaxy:mitre-intrusion-set"], 37 | "filter_ttp_actors_count": 15, 38 | "filter_ttp_pattern": ["misp-galaxy:mitre-attack-pattern"], 39 | "filter_ttp_pattern_count": 15, 40 | 41 | "attribute_summary": {"network": ["ip-src", "ip-dst", "ip-src|port", "ip-dst|port"], 42 | "hashes": ["sha1", "sha256", "md5", "filename|sha256", "filename|sha1", "filename|sha256", "filename|md5", "filename|sha256"], 43 | "domain": ["hostname", "domain", "domain|ip"], 44 | "url": ["url"], 45 | "email": ["email", "email-src", "email-dst"], 46 | "vulnerability": ["vulnerability", "cpe", "weakness"]}, 47 | "attribute_other": "Other", 48 | 49 | "key_organisations": {"5cf66e53-b5f8-43e7-be9a-49880a3b4631": {"logo": "5cf66e53-b5f8-43e7-be9a-49880a3b4631.png"}, 50 | "55f6ea5e-2c60-40e5-964f-47a8950d210f": {"logo": "55f6ea5e-2c60-40e5-964f-47a8950d210f.png"}, 51 | "56a64d7a-63dc-4471-bce9-4accc25ed029": {"logo": "56a64d7a-63dc-4471-bce9-4accc25ed029.png"}, 52 | "56e02aba-9dd0-49ea-bd95-026e55fec118": {"logo": "56e02aba-9dd0-49ea-bd95-026e55fec118.png"}, 53 | "5677fc72-f46c-49f3-b5f3-7245ff32448e": {"logo": "5677fc72-f46c-49f3-b5f3-7245ff32448e.png"}}, 54 | 55 | "output_assets": "assets", 56 | "output_dir": "/var/www/MISP/app/webroot/misp-reporting", 57 | "install_dir": "/var/www/MISP/misp-custom/misp-reporting", 58 | "template_css": "templates/style.css", 59 | "template_html": "templates/template.html", 60 | "template_curation_html": "templates/template_curation.html", 61 | "template_infrastructure_html": "templates/template_infrastructure.html", 62 | "template_contributors_html": "templates/template_contributors.html", 63 | 64 | "noimage_path": "no-image-32.png", 65 | "logo": "assets/cudeso.png", 66 | 67 | "distribution": ["Your organisation", "This community", "Connected communities", "All communities", "Sharing group"], 68 | "analysis": ["Initial", "Ongoing", "Completed"], 69 | "threatlevel_key_mapping": {'1': 'High', '2': 'Medium', '3': 'Low', '4': 'Undefined'}, 70 | "tlp_ignore_graph": ['tlp:unclear', 'tlp:ex:chr', 'tlp:amber+strict'], 71 | "workflow_complete": "workflow:state=\"complete\"", 72 | "workflow_incomplete": "workflow:state=\"incomplete\"", 73 | 74 | } 75 | -------------------------------------------------------------------------------- /docs/misp-reporting-curation-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-curation-1.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-curation-2.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-curation-3.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-curation-4.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-1.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-2.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-3.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-4.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-5.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-6.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-7.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-8.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/d89756a4f0ca779e8e3501aa6c803c839c95bfb1/docs/misp-reporting-summary-9.png -------------------------------------------------------------------------------- /reporting.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | import sys 4 | import json 5 | import requests 6 | from datetime import datetime, timedelta 7 | import logging 8 | from pymisp import * 9 | import inspect 10 | import os 11 | import matplotlib.pyplot as plt 12 | from jinja2 import Template 13 | import plotly.express as px 14 | import matplotlib.cm as cm 15 | import numpy as np 16 | from config import config 17 | from clsReporting import * 18 | from clsReportingData import * 19 | from clsReportingStatistics import * 20 | 21 | 22 | def main(config): 23 | logger.info("Start {}".format(config["logname"])) 24 | 25 | data = ReportingData(config, logger) 26 | data.get_statistics() 27 | data.get_today_events_attributes() 28 | data.get_trending_events_attributes() 29 | data.get_statistics_attributes() 30 | data.get_statistics_keyorgs() 31 | data.get_threatlevel() 32 | data.get_tlplevel() 33 | #data.get_eventdetails() 34 | data.get_target_sector() 35 | data.get_target_geo() 36 | data.get_ttp() 37 | data.get_vulnerabilities() 38 | data.get_curation() 39 | 40 | data.get_misp_statistics() 41 | data.get_contributors() 42 | 43 | data.get_infrastructure() 44 | 45 | reporting = Reporting(config, logger, data.data) 46 | reporting.write_index() 47 | reporting.render_report() 48 | reporting.render_curation_report() 49 | reporting.render_infrastructure() 50 | reporting.render_contributors() 51 | 52 | 53 | statistics = ReportingStatistics(config, logger, data.statistics, data.today_statistics, data.statistics_attributes) 54 | statistics.render_statistics() 55 | 56 | logger.info("End ".format(config["logname"])) 57 | 58 | 59 | if __name__ == '__main__': 60 | # export PYTHONIOENCODING='utf8' 61 | logger = logging.getLogger(config["logname"]) 62 | logger.setLevel(logging.DEBUG) 63 | ch = logging.FileHandler(config["logfile"], mode='a') 64 | ch.setLevel(logging.DEBUG) 65 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') 66 | ch.setFormatter(formatter) 67 | logger.addHandler(ch) 68 | 69 | main(config) 70 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | certifi 2 | charset-normalizer 3 | contourpy 4 | cycler 5 | Deprecated 6 | fonttools 7 | idna 8 | importlib-resources 9 | jinja2 10 | kaleido 11 | kiwisolver 12 | MarkupSafe 13 | matplotlib>=3.8.0 14 | numpy>=1.26.0 15 | packaging 16 | pandas>=2.1.0 17 | pillow 18 | plotly 19 | prettyprint 20 | publicsuffixlist 21 | pymisp 22 | pyparsing 23 | python-dateutil 24 | pytz 25 | pyzmq 26 | requests 27 | rpds-py 28 | six 29 | tenacity 30 | tzdata 31 | urllib3 32 | wrapt 33 | zipp 34 | zmq 35 | -------------------------------------------------------------------------------- /templates/style-blue.css: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: 'Roboto', Arial, sans-serif; 3 | font-size: 14px; 4 | margin: 0; 5 | padding: 0; 6 | background-color: #f2f2f2; 7 | color: #333; 8 | } 9 | header { 10 | background-color: #003366;/* #FF6600;*/ 11 | color: white; 12 | padding: 10px 20px; 13 | } 14 | header h1 { 15 | margin: 0; 16 | font-size: 1.5rem; 17 | } 18 | header .date { 19 | font-size: 0.8rem; 20 | } 21 | .logo-container { 22 | text-align: center; 23 | margin-top: 10px; 24 | } 25 | .logo-container img { 26 | max-height: 50px; 27 | } 28 | .container { 29 | max-width: 900px; 30 | margin: 20px auto; 31 | background: white; 32 | padding: 20px; 33 | border-radius: 5px; 34 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 35 | } 36 | .button-bar { 37 | display: flex; 38 | justify-content: flex-start; 39 | gap: 10px; 40 | margin-bottom: 20px; 41 | } 42 | .button-bar a { 43 | text-decoration: none; 44 | padding: 10px 15px; 45 | border-radius: 5px; 46 | font-weight: bold; 47 | transition: background-color 0.3s, box-shadow 0.3s; 48 | color: white; 49 | } 50 | .button-bar a.summary { 51 | background-color: #003366; /* #FF6600;*/ 52 | } 53 | .button-bar a.summary:hover { 54 | background-color: #003366; /* #e65c00;*/ 55 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 56 | } 57 | .button-bar a.section { 58 | background-color: #004080; /* #FF8800;*/ 59 | } 60 | .button-bar a.section:hover { 61 | background-color: #003366; /* #e67a00;*/ 62 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 63 | } 64 | .button-bar a.login { 65 | margin-left: auto; 66 | display: flex; 67 | background-color: #00509e; 68 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 69 | } 70 | .button-bar a.login:hover { 71 | background-color: #004080; 72 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 73 | } 74 | .charts { 75 | display: flex; 76 | justify-content: space-between; 77 | align-items: flex-start; 78 | gap: 10px; 79 | text-align: center; 80 | } 81 | .chart { 82 | flex: 1; 83 | } 84 | .chart.full-width { 85 | width: 100%; 86 | } 87 | .tables { 88 | display: flex; 89 | justify-content: space-between; 90 | gap: 10px; 91 | } 92 | .table { 93 | flex: 1; 94 | border-collapse: collapse; 95 | width: 100%; 96 | margin: 20px 0; 97 | } 98 | th, td { 99 | border: 1px solid #ddd; 100 | padding: 6px 8px; 101 | text-align: center; 102 | } 103 | th { 104 | background-color: #003366; /* #FF6600;*/ 105 | color: white; 106 | } 107 | td.align-right { 108 | text-align: right; 109 | } 110 | td.align-left { 111 | text-align: left; 112 | } 113 | td.align-topleft { 114 | text-align: left; 115 | vertical-align: top; 116 | } 117 | .table td a { 118 | color: #003366; /* #FF6600;*/ 119 | text-decoration: none; 120 | } 121 | 122 | .table td a:hover { 123 | color: #002d5c; /* #e65c00;*/ 124 | text-decoration: underline; 125 | } 126 | .separator { 127 | margin: 30px 0; 128 | border-top: 2px solid #FF6600; 129 | } 130 | footer { 131 | text-align: center; 132 | font-size: 12px; 133 | color: gray; 134 | margin-top: 20px; 135 | } 136 | 137 | h2 span.sub_h2 { 138 | font-size: 0.8rem; 139 | color: #898787; 140 | font-style: italic; 141 | font-weight: normal; 142 | } 143 | 144 | .red { 145 | color: #d9534f; /* Bootstrap-style red */ 146 | } 147 | .orange { 148 | color: #f0ad4e; /* Bootstrap-style orange */ 149 | } 150 | .green { 151 | color: #5cb85c; /* Bootstrap-style green */ 152 | } 153 | 154 | .status-default { 155 | color: #333; /* Default text colour */ 156 | } 157 | -------------------------------------------------------------------------------- /templates/style-red.css: -------------------------------------------------------------------------------- 1 | :root { 2 | --header-bg: #802c59; 3 | --header-text: #ffffff; 4 | --footer-bg: #511c39; 5 | --footer-text: #f0f0f0; 6 | --button-bg: #802c59; 7 | --button-hover: #c64012; 8 | --button-text: #ffffff; 9 | --body-bg: #f2f2f2; 10 | --container-bg: #ffffff; 11 | --text-color: #333333; 12 | --border-color: #dddddd; 13 | } 14 | 15 | body { 16 | font-family: 'Roboto', Arial, sans-serif; 17 | font-size: 14px; 18 | margin: 0; 19 | padding: 0; 20 | background-color: var(--body-bg); 21 | color: var(--text-color); 22 | } 23 | header { 24 | background-color: var(--header-bg); 25 | color: var(--header-text); 26 | padding: 10px 20px; 27 | } 28 | header h1 { 29 | margin: 0; 30 | font-size: 1.5rem; 31 | } 32 | header .date { 33 | font-size: 0.8rem; 34 | opacity: 0.8; 35 | } 36 | 37 | .logo-container { 38 | text-align: center; 39 | margin-top: 10px; 40 | } 41 | .logo-container img { 42 | max-height: 50px; 43 | } 44 | 45 | .container { 46 | max-width: 900px; 47 | margin: 20px auto; 48 | background: var(--container-bg); 49 | padding: 20px; 50 | border-radius: 5px; 51 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 52 | } 53 | 54 | .button-bar { 55 | display: flex; 56 | justify-content: flex-start; 57 | gap: 10px; 58 | margin-bottom: 20px; 59 | } 60 | .button-bar a { 61 | text-decoration: none; 62 | padding: 10px 15px; 63 | border-radius: 5px; 64 | font-weight: bold; 65 | transition: background-color 0.3s, box-shadow 0.3s; 66 | color: var(--button-text); 67 | background-color: var(--button-bg); 68 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 69 | } 70 | .button-bar a:hover { 71 | background-color: var(--button-hover); 72 | } 73 | 74 | .button-bar a.summary, 75 | .button-bar a.section, 76 | .button-bar a.login { 77 | } 78 | .button-bar a.login { 79 | margin-left: auto; 80 | display: flex; 81 | } 82 | 83 | .charts, 84 | .tables { 85 | display: flex; 86 | justify-content: space-between; 87 | gap: 10px; 88 | text-align: center; 89 | } 90 | 91 | .table { 92 | flex: 1; 93 | border-collapse: collapse; 94 | width: 100%; 95 | margin: 20px 0; 96 | background: var(--container-bg); 97 | border: 1px solid var(--border-color); 98 | } 99 | th, td { 100 | border: 1px solid var(--border-color); 101 | padding: 6px 8px; 102 | text-align: center; 103 | } 104 | th { 105 | background-color: #802c59; 106 | color: var(--button-text); 107 | } 108 | td.align-right { text-align: right; } 109 | td.align-left { text-align: left; } 110 | td.align-topleft { text-align: left; vertical-align: top; } 111 | .table td a { 112 | color: var(--button-bg); 113 | text-decoration: none; 114 | } 115 | .table td a:hover { 116 | color: var(--button-hover); 117 | text-decoration: underline; 118 | } 119 | .separator { 120 | margin: 30px 0; 121 | border-top: 2px solid var(--button-bg); 122 | } 123 | 124 | footer { 125 | background-color: var(--footer-bg); 126 | color: var(--footer-text); 127 | text-align: center; 128 | font-size: 12px; 129 | padding: 10px 0; 130 | } 131 | 132 | h2 span.sub_h2 { 133 | font-size: 0.8rem; 134 | color: #898787; 135 | font-style: italic; 136 | font-weight: normal; 137 | } 138 | 139 | .red { 140 | color: #d9534f; /* Bootstrap-style red */ 141 | } 142 | .orange { 143 | color: #f0ad4e; /* Bootstrap-style orange */ 144 | } 145 | .green { 146 | color: #5cb85c; /* Bootstrap-style green */ 147 | } 148 | 149 | .status-default { 150 | color: #333; /* Default text colour */ 151 | } 152 | -------------------------------------------------------------------------------- /templates/style.css: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: 'Roboto', Arial, sans-serif; 3 | font-size: 14px; 4 | margin: 0; 5 | padding: 0; 6 | background-color: #f2f2f2; 7 | color: #333; 8 | } 9 | header { 10 | background-color: #FF6600; 11 | color: white; 12 | padding: 10px 20px; 13 | } 14 | header h1 { 15 | margin: 0; 16 | font-size: 1.5rem; 17 | } 18 | header .date { 19 | font-size: 0.8rem; 20 | } 21 | .logo-container { 22 | text-align: center; 23 | margin-top: 10px; 24 | } 25 | .logo-container img { 26 | max-height: 50px; 27 | } 28 | .container { 29 | max-width: 900px; 30 | margin: 20px auto; 31 | background: white; 32 | padding: 20px; 33 | border-radius: 5px; 34 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 35 | } 36 | .button-bar { 37 | display: flex; 38 | justify-content: flex-start; 39 | gap: 10px; 40 | margin-bottom: 20px; 41 | } 42 | .button-bar a { 43 | text-decoration: none; 44 | padding: 10px 15px; 45 | border-radius: 5px; 46 | font-weight: bold; 47 | transition: background-color 0.3s, box-shadow 0.3s; 48 | color: white; 49 | } 50 | .button-bar a.summary { 51 | background-color: #FF6600; 52 | } 53 | .button-bar a.summary:hover { 54 | background-color: #e65c00; 55 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 56 | } 57 | .button-bar a.section { 58 | background-color: #FF9900; 59 | } 60 | .button-bar a.section:hover { 61 | background-color: #e68a00; 62 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 63 | } 64 | .button-bar a.login { 65 | margin-left: auto; 66 | display: flex; 67 | background-color: #FF9900; 68 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 69 | } 70 | .button-bar a.login:hover { 71 | background-color: #e68a00; 72 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 73 | } 74 | .charts { 75 | display: flex; 76 | justify-content: space-between; 77 | align-items: flex-start; 78 | gap: 10px; 79 | text-align: center; 80 | } 81 | .chart { 82 | flex: 1; 83 | } 84 | .chart.full-width { 85 | width: 100%; 86 | } 87 | .tables { 88 | display: flex; 89 | justify-content: space-between; 90 | gap: 10px; 91 | } 92 | .table { 93 | flex: 1; 94 | border-collapse: collapse; 95 | width: 100%; 96 | margin: 20px 0; 97 | } 98 | th, td { 99 | border: 1px solid #ddd; 100 | padding: 6px 8px; 101 | text-align: center; 102 | } 103 | th { 104 | background-color: #FF6600; 105 | color: white; 106 | } 107 | td.align-right { 108 | text-align: right; 109 | } 110 | td.align-left { 111 | text-align: left; 112 | } 113 | td.align-topleft { 114 | text-align: left; 115 | vertical-align: top; 116 | } 117 | .table td a { 118 | color: #FF6600; 119 | text-decoration: none; 120 | } 121 | 122 | .table td a:hover { 123 | color: #e65c00; 124 | text-decoration: underline; 125 | } 126 | .separator { 127 | margin: 30px 0; 128 | border-top: 2px solid #FF6600; 129 | } 130 | footer { 131 | text-align: center; 132 | font-size: 12px; 133 | color: gray; 134 | margin-top: 20px; 135 | } 136 | 137 | h2 span.sub_h2 { 138 | font-size: 0.8rem; 139 | color: #898787; 140 | font-style: italic; 141 | font-weight: normal; 142 | } 143 | 144 | .red { 145 | color: #d9534f; /* Bootstrap-style red */ 146 | } 147 | .orange { 148 | color: #f0ad4e; /* Bootstrap-style orange */ 149 | } 150 | .green { 151 | color: #5cb85c; /* Bootstrap-style green */ 152 | } 153 | 154 | .status-default { 155 | color: #333; /* Default text colour */ 156 | } 157 | -------------------------------------------------------------------------------- /templates/template.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Top contributors 24 | Infrastructure Monitoring 25 | 32 |
33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 |
Total eventsTotal attributesCorrelationsUsersOrganisationsLocal organisations
{{ summary.events }}{{ summary.attributes }}{{ summary.correlations }}{{ summary.users }}{{ summary.organisations }}{{ summary.local_organisations }}
56 | 57 |

Reporting period: Last {{ reporting_period }} {{ summary.period }} 58 | {% if report_misp_filters %} 59 |
Filters: {{ report_misp_filters }} 60 | {% endif %} 61 |
Published at {{ report_timestamp_hm }}

62 |
63 | 64 |
65 |

Summary of {{reporting_filter_timestamp}} events

66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 |
Events (24h)Attributes (24h)Attr to_ids (24h)Events ({{ reporting_period }})Attributes ({{ reporting_period }})Attr to_ids ({{ reporting_period }})
{{ summary.today_events }}{{ summary.today_attributes }}{{ summary.today_attributes_ids }}{{ summary.period_events }}{{ summary.period_attributes }}{{ summary.period_attributes_ids }}
88 | 89 |

Trending events and attributes for {{reporting_filter_timestamp}} events

90 |
91 |
92 | Trending {{reporting_filter_timestamp}} events 93 |
94 |
95 | Trending attributes in {{reporting_filter_timestamp}} events 96 |
97 |
98 |
99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | {% for month, value in trending_events.items() %} 108 | 109 | 110 | 111 | 112 | {% endfor %} 113 | 114 |
PeriodEvents
{{ month }}{{ value }}
115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | {% for month, value in trending_attributes.items() %} 125 | 126 | 127 | 128 | 129 | 130 | {% endfor %} 131 | 132 |
PeriodAttributesAttr to_ids
{{ month }}{{ value[0] }}{{ value[1] }}
133 |
134 | 135 |

Attributes type distribution for {{reporting_filter_timestamp}} events {{ attributes_with_ids_or_not}}

136 |
137 |
138 | Attributes type distribution 139 |
140 |
141 | Attributes type distribution 142 |
143 |
144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | {% for attr_type, count in attributes_type.items() %} 154 | 155 | 156 | 157 | 158 | 159 | {% endfor %} 160 | 161 |
Attribute type24h{{ reporting_period }}
{{ attr_type }}{{ count[1] }}{{ count[0] }}
162 |

Activity of key organisations for {{reporting_filter_timestamp}} events

163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | {% for org_name, org_data in keyorgs.items() %} 178 | 179 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | {% endfor %} 193 | 194 |
 OrganisationEvents (24h)Attributes (24h)Attr to_ids (24h)Events ({{ reporting_period }})Attributes ({{ reporting_period }})Attr to_ids ({{ reporting_period }})
180 | {% if org_data.logo %} 181 | {{ org_name }} 182 | {% endif %} 183 | {{ org_name }}{{ org_data['today_events'] }}{{ org_data['today_attributes'] }}{{ org_data['today_attributes_ids'] }}{{ org_data['period_events'] }}{{ org_data['period_attributes'] }}{{ org_data['period_attributes_ids'] }}
195 |
196 | 197 |
198 |

Threat level and TLP of {{reporting_filter_timestamp}} events ({{ reporting_period }})

199 |
200 |
201 | Threat level 202 |
203 |
204 | TLP 205 |
206 |
207 |
208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | {% for level, count in threatlevel.items() %} 217 | 218 | 219 | 220 | 221 | {% endfor %} 222 | 223 |
LevelCount
{{ level }}{{ count }}
224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | {% for tlp, count in tlp.items() %} 233 | 234 | 235 | 236 | 237 | {% endfor %} 238 | 239 | 240 |
TLPCount
{{ tlp }}{{ count }}
(tlp:white is translated to tlp:clear)
241 |
242 | 243 | {% if print_event_details %} 244 |

Event details

245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | {% for event in detailed_events %} 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | {% endfor %} 275 | 276 |
IDDateOrganisationDistributionAnalysisThreat LevelPublishedTLPInfoIndicators
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.distribution }}{{ event.analysis }}{{ event.threat_level }}{{ event.published }}{{ event.tlp }}{{ event.info }}{{ event.indicators }}
277 | {% endif %} 278 |
279 | 280 |
281 |

Geo target information of {{reporting_filter_timestamp}} events ({{ reporting_period }})

282 |
283 | Geo targeting map 284 |
285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | {% for country, count in target_geo.items() %} 294 | 295 | 296 | 297 | 298 | {% endfor %} 299 | 300 |
CountryCount
{{ country }}{{ count }}
301 |

Sector targeting of {{reporting_filter_timestamp}} events ({{ reporting_period }})

302 |
303 | Sector targeting 304 |
305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | {% for sector, count in target_sector.items() %} 314 | 315 | 316 | 317 | 318 | {% endfor %} 319 | 320 |
SectorCount
{{ sector }}{{ count }}
321 |
322 | 323 |
324 |

TTPs of {{reporting_filter_timestamp}} events ({{ reporting_period }})

325 |
326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | {% for actor, count in ttp_actors.items() %} 335 | 336 | 337 | 338 | 339 | {% endfor %} 340 | 341 |
ActorsCount
{{ actor }}{{ count }}
342 |
343 |
344 | 345 | 346 | 347 | 348 | 349 | 350 | 351 | 352 | {% for pattern, count in ttp_pattern.items() %} 353 | 354 | 355 | 356 | 357 | {% endfor %} 358 | 359 |
PatternsCount
{{ pattern }}{{ count }}
360 |
361 |
362 | 363 |
364 |

Vulnerabilities and weaknesses found in {{reporting_filter_timestamp}} events ({{ reporting_period }})

365 |
366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | {% for cve, detail in vulnerabilities.items() %} 378 | 379 | 380 | 381 | 382 | 387 | 388 | 389 | {% endfor %} 390 | 391 |
VulnerabilityEventsSummaryCVSSv3Count
{{ cve }}MISP{{ detail.summary }}{{ detail.cvss3 }} 383 | {% if detail.cvss3|float > cve_highlight|float %} 384 | ⚠ 385 | {% endif %} 386 | {{ detail.count }}
392 |
393 |
394 | 395 | 399 | 400 | 401 | -------------------------------------------------------------------------------- /templates/template_contributors.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Top contributors 24 | Infrastructure Monitoring 25 | 32 |
33 | 34 |

Reporting period: Last {{ reporting_period }} {{ summary.period }} 35 | {% if report_misp_filters %} 36 |
Filters: {{ report_misp_filters }} 37 | {% endif %} 38 |
Published at {{ report_timestamp_hm }}

39 |
40 | 41 |
42 |

Contributors

43 |

Organisations

44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | {% for org_name, org_data in contributors_org.items() %} 56 | 57 | 62 | 63 | 64 | 65 | 66 | 67 | {% endfor %} 68 | 69 |
 OrganisationEvents (24h)Events (30d)Events
(all time, published and not-published)
58 | {% if org_data.logo %} 59 | {{ org_name }} 60 | {% endif %} 61 | {{ org_name }}{{ org_data['today_events'] }}{{ org_data['period_events'] }}{{ org_data['alltime_events'] }}
70 |
71 | 72 |
73 |

Users

74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | {% for org_name, org_data in contributors_user.items() %} 86 | 87 | 92 | 93 | 94 | 95 | 96 | 97 | {% endfor %} 98 | 99 |
UserOrganisationEvents (24h)Events (30d)Events (all time)
88 | {% if org_data.logo %} 89 | {{ org_name }} 90 | {% endif %} 91 | {{ org_name }}{{ org_data['today_events'] }}{{ org_data['period_events'] }}{{ org_data['all_time'] }}
100 |
101 | 102 | 106 | 107 | 108 | -------------------------------------------------------------------------------- /templates/template_curation.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Top contributors 24 | Infrastructure Monitoring 25 | 32 |
33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 46 | 48 | 50 | 51 | 52 |
Not curated (24h)Not curated with threat level high (24h)Not curated from completely reliable sources (24h)
{{ curation_incomplete_today_count }} 45 |
Events		{{ curation_incomplete_high_count }} 47 |
Events		{{ curation_incomplete_adm_high_count}} 49 |
Events
53 | 54 |

Reporting period: Last {{ reporting_period }} {{ summary.period }} 55 | {% if report_misp_filters %} 56 |
Filters: {{ report_misp_filters }} 57 | {% endif %} 58 |
Published at {{ report_timestamp_hm }}

59 |
60 | 61 |
62 |

Curation summary

63 |

Curated and not curated

64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 |
PeriodCuratedNot curated
Last 24h{{ curation_complete_today_count }}{{ curation_incomplete_today_count }}
Last {{ reporting_period }}{{ curation_complete_count }}{{ curation_incomplete_count }}
85 | 86 |

Event dates (event dates indicate when a threat occurred and are set by the event creator)

87 |
88 |
89 | Bubble graph of curation results 90 |
91 |
92 | 93 |

Curation for last {{ reporting_period }}

94 |
95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | {% for org, value in curation_complete_org.items() %} 104 | 105 | 106 | 107 | 108 | {% endfor %} 109 | 110 |
OrganisationCurated events
{{ org }}{{ value }}
111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | {% for org, value in curation_incomplete_org.items() %} 120 | 121 | 122 | 123 | 124 | {% endfor %} 125 | 126 |
OrganisationNot curated events
{{ org }}{{ value }}
127 |
128 |
129 | 130 |
131 |

Events waiting for curation - Threat level high

132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | {% for event in curation_incomplete_high %} 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | {% endfor %} 152 | 153 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
154 | 155 |

Events waiting for curation - Admiralty scale : Source Reliability: Completely reliable (A)

156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | {% for event in curation_incomplete_adm_high %} 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | {% endfor %} 176 | 177 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
178 | 179 |

Events waiting for curation

180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | {% for event in curation_incomplete %} 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | {% endfor %} 200 | 201 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
202 | 203 |

Curated events

204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | {% for event in curation_complete %} 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | {% endfor %} 224 | 225 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
226 | 227 |
228 | 229 | 233 | 234 | 235 | -------------------------------------------------------------------------------- /templates/template_infrastructure.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Top contributors 24 | Infrastructure Monitoring 25 | 32 |
33 | 34 |

Published at {{ report_timestamp_hm }}.

35 |
36 | 37 |
38 |

Infrastructure overview

39 |

Server status

40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | {% for server_name, server_info in report_misp_results.items() %} 59 | 60 | 61 | 62 | 69 | {% if server_info.status == "OK" %} 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | {% else %} 79 | 80 | {% endif %} 81 | 82 | {% endfor %} 83 | 84 |
ServerLocationStatusVersionZMQRedisEnrichmentImportExportCache WorkerDefault Worker
{{ server_name }}{{ server_info.url }} 63 | {% if server_info.status == "OK" %} 64 | ✅ 65 | {% else %} 66 | ❌ 67 | {% endif %} 68 | {{ server_info.server.version }}{% if server_info.server.zmqStatus == 0 %}✅{% else %}❌{% endif %}{% if server_info.server.redis == "OK" %}✅{% else %}❌{% endif %}{% if server_info.server.moduleStatus_enrichment == 0 %}✅{% else %}❌{% endif %}{% if server_info.server.moduleStatus_import == 0 %}✅{% else %}❌{% endif %}{% if server_info.server.moduleStatus_export == 0 %}✅{% else %}❌{% endif %}{% if server_info.server.workers.cache == "OK" %}✅{% else %}❌{% endif %}{% if server_info.server.workers.default == "OK" %}✅{% else %}❌{% endif %}–
85 | 86 | {% for server_name, server_info in report_misp_results.items() %} 87 |

Remote servers for {{ server_name }}

88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | {% for remote in server_info.remote_servers %} 100 | 101 | 102 | 103 | 104 | 109 | 110 | 111 | 112 | {% else %} 113 | 114 | 115 | 116 | {% endfor %} 117 | 118 |
IDNameURLStatusInfo
{{ remote.id }}{{ remote.name }}{{ remote.url }} 105 | {% set color_class = remote.status_color if remote.status_color in ['red', 'orange', 'green'] else 'status-default' %} 106 | 107 | {{ remote.status }} 108 | {{ remote.status_extra }}
No remote servers available
119 | {% endfor %} 120 | 121 |
122 | 123 | 124 | 125 | 126 | 130 | 131 | 132 | --------------------------------------------------------------------------------