├── .github └── workflows │ └── jekyll-gh-pages.yml ├── .gitignore ├── README.md ├── assets ├── 55f6ea5e-2c60-40e5-964f-47a8950d210f.png ├── 5677fc72-f46c-49f3-b5f3-7245ff32448e.png ├── 56a64d7a-63dc-4471-bce9-4accc25ed029.png ├── 56e02aba-9dd0-49ea-bd95-026e55fec118.png ├── 57573875-59fc-4667-b85c-7361d5d09a03.png ├── 580eebab-6048-42c1-945a-50200a09010e.png └── 5cf66e53-b5f8-43e7-be9a-49880a3b4631.png ├── clsReporting.py ├── clsReportingData.py ├── clsReportingStatistics.py ├── config.py.default ├── docs ├── misp-reporting-curation-1.png ├── misp-reporting-curation-2.png ├── misp-reporting-curation-3.png ├── misp-reporting-curation-4.png ├── misp-reporting-summary-1.png ├── misp-reporting-summary-2.png ├── misp-reporting-summary-3.png ├── misp-reporting-summary-4.png ├── misp-reporting-summary-5.png ├── misp-reporting-summary-6.png ├── misp-reporting-summary-7.png ├── misp-reporting-summary-8.png └── misp-reporting-summary-9.png ├── reporting.py ├── requirements.txt └── templates ├── style-blue.css ├── style-red.css ├── style.css ├── template.html ├── template_curation.html └── template_infrastructure.html /.github/workflows/jekyll-gh-pages.yml: -------------------------------------------------------------------------------- 1 | # Sample workflow for building and deploying a Jekyll site to GitHub Pages 2 | name: Deploy Jekyll with GitHub Pages dependencies preinstalled 3 | 4 | on: 5 | # Runs on pushes targeting the default branch 6 | push: 7 | branches: ["master"] 8 | 9 | # Allows you to run this workflow manually from the Actions tab 10 | workflow_dispatch: 11 | 12 | # Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages 13 | permissions: 14 | contents: read 15 | pages: write 16 | id-token: write 17 | 18 | # Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. 19 | # However, do NOT cancel in-progress runs as we want to allow these production deployments to complete. 20 | concurrency: 21 | group: "pages" 22 | cancel-in-progress: false 23 | 24 | jobs: 25 | # Build job 26 | build: 27 | runs-on: ubuntu-latest 28 | steps: 29 | - name: Checkout 30 | uses: actions/checkout@v4 31 | - name: Setup Pages 32 | uses: actions/configure-pages@v5 33 | - name: Build with Jekyll 34 | uses: actions/jekyll-build-pages@v1 35 | with: 36 | source: ./ 37 | destination: ./_site 38 | - name: Upload artifact 39 | uses: actions/upload-pages-artifact@v3 40 | 41 | # Deployment job 42 | deploy: 43 | environment: 44 | name: github-pages 45 | url: ${{ steps.deployment.outputs.page_url }} 46 | runs-on: ubuntu-latest 47 | needs: build 48 | steps: 49 | - name: Deploy to GitHub Pages 50 | id: deployment 51 | uses: actions/deploy-pages@v4 52 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | config.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MISP reporting 2 | 3 | **MISP reporting** lets you explore the threat information stored in your MISP instance. It provides an overview of  4 | - the trend in the volume of events and attributes 5 | - the distribution of attribute types 6 | - recent activity by key organisations 7 | - threat level and TLP classification of events 8 | - targeted geographies and sectors 9 | - TTPs used by threat actors 10 | - vulnerabilities and weaknesses referenced in threat events 11 | 12 | **MISP reporting** queries your instance, summarises the data, and outputs **HTML reports** together with a **JSON statistics** file. The generated **graphs** can also be embedded in other reporting tools. 13 | 14 | # Setup 15 | 16 | MISP reporting is written in Python and requires a MISP API key. 17 | 18 | Because the web server must read and write the files, ownership must be set to the web‑server user. On Ubuntu this user is `www-data`; on Red Hat it is `apache`. Run the following commands as the appropriate user (replace `www-data` with `apache` on Red Hat). 19 | 20 | **Step 1: Create the reporting directory and clone the repository** 21 | 22 | Custom MISP scripts are best placed in a dedicated directory such as `/var/www/MISP/misp-custom/`. Create it if necessary and clone the project. 23 | 24 | ```bash 25 | sudo -u www-data mkdir /var/www/MISP/misp-custom/ 26 | sudo -u www-data git clone https://github.com/cudeso/misp-reporting.git 27 | cd misp-reporting 28 | ``` 29 | 30 | **Step 2: Set up a Python virtual environment and install dependencies** 31 | 32 | To keep system packages untouched, create a dedicated virtual environment named `venv`. If `virtualenv` is unavailable you may use `python3 -m venv venv`. 33 | 34 | ```bash 35 | sudo -u www-data virtualenv venv 36 | sudo -u www-data venv/bin/pip install -r requirements.txt 37 | ``` 38 | 39 | **Step 3: Configure the application** 40 | 41 | Copy the default configuration and edit it to suit your environment. 42 | 43 | ```bash 44 | sudo -u www-data cp config.py.default config.py 45 | ``` 46 | 47 | At a minimum you must set 48 | 49 | - **misp_url** – the URL of your MISP instance 50 | - **misp_key** – an API key (ideally for an account with the *admin* role) 51 | - **output_dir** – where reports will be written 52 | - **install_dir** – where MISP reporting is installed 53 | 54 | **Step 4: Create the output directories** 55 | 56 | Reports are written to `output_dir`, usually `/var/www/MISP/app/webroot/misp-reporting`. The directory must also contain a copy of the *assets* folder, which holds the logos of key MISP organisations. 57 | 58 | ```bash 59 | sudo -u www-data mkdir /var/www/MISP/app/webroot/misp-reporting/ 60 | sudo -u www-data cp -r /var/www/MISP/misp-custom/misp-reporting/assets /var/www/MISP/app/webroot/misp-reporting/ 61 | chown -R www-data:www-data /var/www/MISP/app/webroot/misp-reporting/ 62 | ``` 63 | 64 | **Step 5: Add your organisation’s logo** 65 | 66 | Copy your logo (and those of other key organisations) into the `assets` directory: 67 | 68 | ```bash 69 | sudo -u www-data cp logo.png /var/www/MISP/app/webroot/misp-reporting/assets/ 70 | ``` 71 | 72 | **Step 6: Run the reporting script** 73 | 74 | Generate the reports: 75 | 76 | ```bash 77 | sudo -u www-data /var/www/MISP/misp-custom/misp-reporting/venv/bin/python /var/www/MISP/misp-custom/misp-reporting/reporting.py 78 | ``` 79 | 80 | The reports are now available at `https:///misp-reporting/`. 81 | 82 | # Cronjob 83 | 84 | To run the report daily, add a cronjob for the web‑server user: 85 | 86 | ```bash 87 | crontab -u www-data -e 88 | ``` 89 | 90 | Insert the following line (adjust the path if required): 91 | 92 | ```bash 93 | 00 6 * * * cd /var/www/MISP/misp-custom/misp-reporting/ ; /var/www/MISP/misp-custom/misp-reporting/venv/bin/python /var/www/MISP/misp-custom/misp-reporting/reporting.py 94 | ``` 95 | 96 | # Reports 97 | 98 | Three report types are planned. The **summary** and **curation overview** are complete; **infrastructure monitoring** is in progress. 99 | 100 | ## Summary report 101 | 102 | The summary report presents high‑level statistics for your MISP server. The header also links to the other reports and provides a button to log in to MISP. 103 | 104 | ![docs/misp-reporting-summary-1.png](docs/misp-reporting-summary-1.png) 105 | 106 | It begins with a summary of published events. Based on `reporting_period`, `reporting_trending_count`, and `reporting_filter`, you see the trend in published events and their attributes. 107 | 108 | ![docs/misp-reporting-summary-2.png](docs/misp-reporting-summary-2.png) 109 | 110 | The next section examines attribute details, showing which attribute types appeared during the last 24 hours and other periods. The breakdown is controlled by `attribute_summary`. 111 | 112 | ![docs/misp-reporting-summary-3.png](docs/misp-reporting-summary-3.png) 113 | 114 | A further section highlights key organisations (defined in `key_organisations`), listing event and attribute counts for the past 24 hours and the current reporting period. Direct links open the corresponding events in MISP. 115 | 116 | ![docs/misp-reporting-summary-4.png](docs/misp-reporting-summary-4.png) 117 | 118 | Threat level and Traffic Light Protocol (TLP) designations follow, enabling you to spot spikes in high‑severity or specific TLP‑tagged events at a glance. 119 | 120 | ![docs/misp-reporting-summary-5.png](docs/misp-reporting-summary-5.png) 121 | 122 | Two charts then show which countries or regions were targeted and which sectors were attacked, using galaxy tags set in `filter_geo` and `filter_sector`. 123 | 124 | ![docs/misp-reporting-summary-6.png](docs/misp-reporting-summary-6.png) 125 | 126 | ![docs/misp-reporting-summary-7.png](docs/misp-reporting-summary-7.png) 127 | 128 | Similarly, threat actors (`filter_ttp_actors`) and attack techniques (`filter_ttp_pattern`) are listed. 129 | 130 | ![docs/misp-reporting-summary-8.png](docs/misp-reporting-summary-8.png) 131 | 132 | Finally, the report summarises all CVEs referenced in events. Additional details, such as descriptions and CVSS scores, are retrieved from cvepremium.circl.lu (`cve_url`). CVEs above the threshold set in `reporting_cve_highlight` are emphasised. 133 | 134 | ![docs/misp-reporting-summary-9.png](docs/misp-reporting-summary-9.png) 135 | 136 | ## Curation report 137 | 138 | The curation report shows how many threat events have been curated—either automatically or manually. A typical **curation process** checks events against the MISP warninglists and the CIRCL Hashlookup service; matches lead to indicators being marked non‑actionable (the *to_ids* flag is cleared). Curation may also enrich events with contextual data. In this setup, curated events are placed in the local workflow state *complete*; uncurated events remain *incomplete*. 139 | 140 | The opening section lists how many events are awaiting curation, how many high‑threat events are waiting, and how many events from completely reliable sources are pending. Links take you straight to each set in MISP. 141 | 142 | ![docs/misp-reporting-curation-1.png](docs/misp-reporting-curation-1.png) 143 | 144 | Next, the report plots the dates of curated and uncurated events so you can see whether outstanding items are recent or historic. 145 | 146 | ![docs/misp-reporting-curation-2.png](docs/misp-reporting-curation-2.png) 147 | 148 | You can also view the volume of curated versus uncurated events per organisation. 149 | 150 | ![docs/misp-reporting-curation-3.png](docs/misp-reporting-curation-3.png) 151 | 152 | Finally, a table lists the individual events, highlighting those with high threat levels and those from fully reliable sources. This helps you prioritise urgent reviews. For automation ideas see the MISP playbook [Curate threat events](https://misp.github.io/misp-playbooks/misp-playbooks/pb_curate_misp_events-with_output.html). 153 | 154 | ![docs/misp-reporting-curation-4.png](docs/misp-reporting-curation-4.png) 155 | 156 | ## Statistics file 157 | 158 | Besides the HTML reports, misp‑reporting creates a JSON file, `statistics.json`, suitable for other tools: 159 | 160 | ``` 161 | { 162 | "misp_server": "https://misp", 163 | "report_date": "2025-04-16", 164 | "statistics": "{'event_count': 138584, 'attribute_count': 6877874, 'user_count': 2, 'org_count': 1124, 'local_org_count': 2}", 165 | "today_statistics": "{'today_event_count': 36598, 'today_attribute_count': 36598, 'today_attribute_ids_count': 29370}", 166 | "today_statistics_attributes": "{'hashes': 4017, 'url': 295, 'domain': 22422, 'Other': 607, 'email': 22, 'network': 2007}" 167 | } 168 | ``` 169 | 170 | ## Graphics 171 | 172 | All images are stored in the report directory and may be reused elsewhere: 173 | 174 | - `attributes_bar_chart.png` 175 | - `attributes_type_bar_chart.png` 176 | - `curated_events_bubble_chart.png` 177 | - `geo_targeting_map.png` 178 | - `threatlevel_bar_chart.png` 179 | - `threat_levels_bar.png` 180 | - `tlp_pie_chart.png` 181 | - `attributes_trending.png` 182 | - `attributes_type_daily_bar_chart.png` 183 | - `events_trending.png` 184 | - `threatlevel_pie_chart.png` 185 | - `threat_levels_pie.png` 186 | - `trending_graph.png` 187 | 188 | ## Prevent unauthenticated access 189 | 190 | By default, everything under `/var/www/MISP/app/webroot/misp-reporting` is publicly accessible. To restrict access, modify MISP core as explained in [github.com/MISP/MISP/issues/10272](https://github.com/MISP/MISP/issues/10272) by editing `/var/www/MISP/app/AttributesController.php`. 191 | 192 | # Configuration options 193 | 194 | The configuration is handled in `config.py`. 195 | 196 | Set the log file (`logfile`), logger name (`logname`), and whether to log uncurated events (`log_incomplete`): 197 | 198 | ``` 199 | config = { 200 | "logfile": "/var/www/MISP/app/tmp/logs/misp-reporting.log", 201 | "logname": "misp-reporting", 202 | "log_incomplete": False, 203 | ``` 204 | 205 | Define the length of the reporting period (`reporting_period`), how many periods to display (`reporting_trending_count`), and any event filter (`reporting_filter`). `reporting_eventdetails_onlykeyorgs` determines whether to print event details later (see `print_event_details`). `reporting_filter_timestamp` lets you trend on either *timestamp* or *published* dates. 206 | 207 | ``` 208 | "reporting_period": "30d", 209 | "reporting_filter": None, 210 | "reporting_eventdetails_onlykeyorgs": True, 211 | "reporting_trending_count": 6, 212 | "reporting_filter_attribute_type_ids": True, 213 | "reporting_filter_timestamp": "published", 214 | "reporting_filter_published": True, 215 | 216 | "print_event_details": False, 217 | ``` 218 | 219 | MISP connection settings follow. Provide the instance URL, whether to verify the certificate, and an API key. `misp_page_size` controls pagination. `cve_url` and `vulnerability_lookup_url` supply vulnerability information. `reporting_cve_highlight` sets the CVSS threshold for highlighting CVEs. 220 | 221 | ``` 222 | "misp_url": "", 223 | "misp_verifycert": False, 224 | "misp_key": "", 225 | "misp_page_size": 200, 226 | "cve_url": "https://cvepremium.circl.lu/api/cve/", 227 | "vulnerability_lookup_url": "https://vulnerability.circl.lu/vuln/", 228 | 229 | "reporting_cve_highlight": 7, 230 | ``` 231 | 232 | Filter options: 233 | 234 | ``` 235 | "filter_sector": "misp-galaxy:sector", 236 | "filter_sector_count": 10, 237 | "filter_geo": "misp-galaxy:target-information", 238 | "filter_geo_count": 10, 239 | "filter_ttp_actors": ["misp-galaxy:threat-actor", "misp-galaxy:mitre-intrusion-set"], 240 | "filter_ttp_actors_count": 15, 241 | "filter_ttp_pattern": ["misp-galaxy:mitre-attack-pattern"], 242 | "filter_ttp_pattern_count": 15, 243 | ``` 244 | 245 | Attribute groupings: 246 | 247 | ``` 248 | "attribute_summary": {"network": ["ip-src", "ip-dst", "ip-src|port", "ip-dst|port"], 249 | "hashes": ["sha1", "sha256", "md5", "filename|sha256", "filename|sha1", "filename|sha256", "filename|md5", "filename|sha256"], 250 | "domain": ["hostname", "domain", "domain|ip"], 251 | "url": ["url"], 252 | "email": ["email", "email-src", "email-dst"], 253 | "vulnerability": ["vulnerability", "cpe", "weakness"]}, 254 | "attribute_other": "Other", 255 | ``` 256 | 257 | Key organisations and their logos: 258 | 259 | ``` 260 | "key_organisations": {"5cf66e53-b5f8-43e7-be9a-49880a3b4631": {"logo": "5cf66e53-b5f8-43e7-be9a-49880a3b4631.png"}, 261 | "55f6ea5e-2c60-40e5-964f-47a8950d210f": {"logo": "55f6ea5e-2c60-40e5-964f-47a8950d210f.png"}, 262 | "56a64d7a-63dc-4471-bce9-4accc25ed029": {"logo": "56a64d7a-63dc-4471-bce9-4accc25ed029.png"}, 263 | "56e02aba-9dd0-49ea-bd95-026e55fec118": {"logo": "56e02aba-9dd0-49ea-bd95-026e55fec118.png"}, 264 | "5677fc72-f46c-49f3-b5f3-7245ff32448e": {"logo": "5677fc72-f46c-49f3-b5f3-7245ff32448e.png"}}, 265 | ``` 266 | 267 | Installation paths, templates, and appearance: 268 | 269 | ``` 270 | "output_assets": "assets", 271 | "output_dir": "/var/www/MISP/app/webroot/misp-reporting", 272 | "install_dir": "/var/www/MISP/misp-custom/misp-reporting", 273 | "template_css": "templates/style.css", 274 | "template_html": "templates/template.html", 275 | "template_curation_html": "templates/template_curation.html", 276 | "template_infrastructure_html": "templates/template_infrastructure.html", 277 | "noimage_path": "no-image-32.png", 278 | "logo": "assets/cudeso.png", 279 | 280 | "distribution": ["Your organisation", "This community", "Connected communities", "All communities", "Sharing group"], 281 | "analysis": ["Initial", "Ongoing", "Completed"], 282 | "threatlevel_key_mapping": {'1': 'High', '2': 'Medium', '3': 'Low', '4': 'Undefined'}, 283 | "tlp_ignore_graph": ['tlp:unclear', 'tlp:ex:chr', 'tlp:amber+strict'], 284 | "workflow_complete": "workflow:state=\"complete\"", 285 | "workflow_incomplete": "workflow:state=\"incomplete\"", 286 | } 287 | ``` 288 | -------------------------------------------------------------------------------- /assets/55f6ea5e-2c60-40e5-964f-47a8950d210f.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/55f6ea5e-2c60-40e5-964f-47a8950d210f.png -------------------------------------------------------------------------------- /assets/5677fc72-f46c-49f3-b5f3-7245ff32448e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/5677fc72-f46c-49f3-b5f3-7245ff32448e.png -------------------------------------------------------------------------------- /assets/56a64d7a-63dc-4471-bce9-4accc25ed029.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/56a64d7a-63dc-4471-bce9-4accc25ed029.png -------------------------------------------------------------------------------- /assets/56e02aba-9dd0-49ea-bd95-026e55fec118.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/56e02aba-9dd0-49ea-bd95-026e55fec118.png -------------------------------------------------------------------------------- /assets/57573875-59fc-4667-b85c-7361d5d09a03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/57573875-59fc-4667-b85c-7361d5d09a03.png -------------------------------------------------------------------------------- /assets/580eebab-6048-42c1-945a-50200a09010e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/580eebab-6048-42c1-945a-50200a09010e.png -------------------------------------------------------------------------------- /assets/5cf66e53-b5f8-43e7-be9a-49880a3b4631.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/assets/5cf66e53-b5f8-43e7-be9a-49880a3b4631.png -------------------------------------------------------------------------------- /clsReporting.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | from matplotlib.ticker import MultipleLocator 14 | import numpy as np 15 | from collections import defaultdict 16 | 17 | 18 | class Reporting: 19 | def __init__(self, config, logger, data, output_dir="report_output"): 20 | self.config = config 21 | self.logger = logger 22 | self.data = data 23 | self.data_for_report = {} 24 | 25 | self.report_date = datetime.now().strftime("%Y-%m-%d") 26 | self.report_misp_server = self.config["misp_url"] 27 | self.reporting_period = self.config["reporting_period"] 28 | self.output_dir = self.config["output_dir"] 29 | self.template_css = "{}/{}".format(self.config["install_dir"], self.config["template_css"]) 30 | self.template_html = "{}/{}".format(self.config["install_dir"], self.config["template_html"]) 31 | self.template_curation_html = "{}/{}".format(self.config["install_dir"], self.config["template_curation_html"]) 32 | self.template_infrastructure_html = "{}/{}".format(self.config["install_dir"], self.config["template_infrastructure_html"]) 33 | self.assets_dir = os.path.join(self.output_dir, self.config["output_assets"]) 34 | os.makedirs(self.assets_dir, exist_ok=True) 35 | 36 | self.events_trending_path = os.path.join(self.output_dir, "events_trending.png") 37 | self.attributes_trending_path = os.path.join(self.output_dir, "attributes_trending.png") 38 | self.attributes_type_bar_chart_path = os.path.join(self.output_dir, "attributes_type_bar_chart.png") 39 | self.attributes_type_daily_bar_chart_path = os.path.join(self.output_dir, "attributes_type_daily_bar_chart.png") 40 | self.threatlevel_bar_chart_path = os.path.join(self.output_dir, "threatlevel_bar_chart.png") 41 | self.tlp_pie_chart_path = os.path.join(self.output_dir, "tlp_pie_chart.png") 42 | self.geo_targeting_map_path = os.path.join(self.output_dir, "geo_targeting_map.png") 43 | self.sector_targeting_bar_chart_path = os.path.join(self.output_dir, "sector_targeting_bar_chart.png") 44 | self.curated_events_bubble_path = os.path.join(self.output_dir, "curated_events_bubble_chart.png") 45 | self.noimage_path = self.config["noimage_path"] 46 | 47 | self.threatlevel_key_mapping = self.config["threatlevel_key_mapping"] 48 | self.tlp_ignore_graph = self.config["tlp_ignore_graph"] 49 | self.attribute_summary = self.config["attribute_summary"] 50 | self.attribute_other = self.config["attribute_other"] 51 | self.key_organisations = self.config["key_organisations"] 52 | 53 | if self.config["misp_verifycert"] is False: 54 | import urllib3 55 | urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 56 | self.misp = PyMISP(self.config["misp_url"], self.config["misp_key"], self.config["misp_verifycert"]) 57 | 58 | def write_index(self): 59 | html_content = """ 60 | 61 | 62 | 63 | Redirecting... 64 | 65 | 66 | 67 | 68 | 69 |

If you are not automatically redirected, please click here.

70 | 71 | 72 | """ 73 | 74 | with open("{}/index.html".format(self.config["output_dir"]), 'w') as f: 75 | f.write(html_content) 76 | 77 | def render_infrastructure(self): 78 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 79 | 80 | template_css_file = self.template_css 81 | with open(template_css_file, "r") as f: 82 | css_content = f.read() 83 | 84 | template_file = self.template_infrastructure_html 85 | with open(template_file, "r") as f: 86 | html_template = f.read() 87 | 88 | # Render the HTML 89 | template = Template(html_template) 90 | html_content = template.render( 91 | css=css_content, 92 | title="MISP Infrastructure summary", 93 | logo=self.config["logo"], 94 | report_date=self.report_date, 95 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 96 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 97 | report_misp_server=self.report_misp_server, 98 | 99 | ) 100 | 101 | # Save the HTML file 102 | output_html_path = os.path.join(self.output_dir, "misp_infrastructure.html") 103 | with open(output_html_path, "w") as f: 104 | f.write(html_content) 105 | return True 106 | 107 | def render_curation_report(self): 108 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 109 | 110 | # ############### General statistics 111 | key = "statistics" 112 | if key in self.data: 113 | dataset = self.data[key] 114 | updated_dataset = {} 115 | days = int(self.config["reporting_period"].strip("d")) 116 | current_date = datetime.now() 117 | past_date = current_date - timedelta(days=days) 118 | reporting_period = self.config["reporting_period"] 119 | updated_dataset["period"] = f"(until {past_date.strftime('%Y-%m-%d')})" 120 | if self.config["reporting_filter"] is not None: 121 | updated_dataset["period"] = "{}
MISP filters: {}".format(updated_dataset["period"], self.config["reporting_filter"]) 122 | #updated_dataset["events"] = dataset["event_count"] 123 | #updated_dataset["attributes"] = dataset["attribute_count"] 124 | self.data_for_report[key] = updated_dataset 125 | self.logger.debug(" Created {}".format(key)) 126 | else: 127 | self.data_for_report[key] = {} 128 | self.logger.error(" Not found: {}".format(key)) 129 | 130 | # ############### Curation events 131 | key = "curation_complete" 132 | curation_complete_count = 0 133 | curation_complete_today_count = 0 134 | curation_incomplete_count = 0 135 | curation_incomplete_today_count = 0 136 | curation_complete_events = [] 137 | curation_incomplete_events = [] 138 | if key in self.data: 139 | dataset = self.data[key] 140 | curation_complete_count = len(dataset) 141 | curation_complete_events = dataset 142 | self.logger.debug(" Created {}".format(key)) 143 | else: 144 | self.data_for_report[key] = {} 145 | self.logger.error(" Not found: {}".format(key)) 146 | 147 | if "curation_complete_today" in self.data: 148 | dataset = self.data["curation_complete_today"] 149 | curation_complete_today_count = len(dataset) 150 | self.logger.debug(" Created {}".format("curation_complete_today")) 151 | else: 152 | self.data_for_report["curation_complete_today"] = {} 153 | self.logger.error(" Not found: {}".format("curation_complete_today")) 154 | 155 | key = "curation_incomplete" 156 | if key in self.data: 157 | dataset = self.data[key] 158 | curation_incomplete_count = len(dataset) 159 | curation_incomplete_events = dataset 160 | self.logger.debug(" Created {}".format(key)) 161 | else: 162 | self.data_for_report[key] = {} 163 | self.logger.error(" Not found: {}".format(key)) 164 | 165 | if "curation_incomplete_today" in self.data: 166 | dataset = self.data["curation_incomplete_today"] 167 | curation_incomplete_today_count = len(dataset) 168 | self.logger.debug(" Created {}".format("curation_incomplete_today")) 169 | else: 170 | self.data_for_report["curation_incomplete_today"] = {} 171 | self.logger.error(" Not found: {}".format("curation_incomplete_today")) 172 | 173 | if "curation_incomplete_high" in self.data: 174 | dataset = self.data["curation_incomplete_high"] 175 | curation_incomplete_high_count = len(dataset) 176 | curation_incomplete_high_events = dataset 177 | self.logger.debug(" Created {}".format("curation_incomplete_high")) 178 | else: 179 | self.data_for_report["curation_incomplete_high"] = {} 180 | self.logger.error(" Not found: {}".format("curation_incomplete_high")) 181 | 182 | if "curation_incomplete_adm_high" in self.data: 183 | dataset = self.data["curation_incomplete_adm_high"] 184 | curation_incomplete_adm_high_count = len(dataset) 185 | curation_incomplete_adm_high_events = dataset 186 | self.logger.debug(" Created {}".format("curation_incomplete_adm_high")) 187 | else: 188 | self.data_for_report["curation_incomplete_adm_high"] = {} 189 | self.logger.error(" Not found: {}".format("curation_incomplete_adm_high")) 190 | 191 | key1 = "curation_incomplete_date" 192 | key2 = "curation_complete_date" 193 | if key1 in self.data and key2 in self.data: 194 | self.data_for_report[key1] = self._aggregate_by_month(self.data[key1]) 195 | self.data_for_report[key2] = self._aggregate_by_month(self.data[key2]) 196 | 197 | all_months = set(self.data_for_report[key1].keys()) | set(self.data_for_report[key2].keys()) 198 | if not all_months: 199 | self.logger.error("Not all months for {} or {}".format(key1, key2)) 200 | else: 201 | sorted_months = sorted(all_months, key=lambda x: datetime.strptime(x, "%Y-%m")) 202 | earliest, latest = sorted_months[0], sorted_months[-1] 203 | 204 | full_months = self._month_range(earliest, latest) 205 | values1 = [self.data_for_report[key1].get(m, 0) for m in full_months] 206 | values2 = [self.data_for_report[key2].get(m, 0) for m in full_months] 207 | 208 | self.create_bubble_chart(values1, values2, full_months, self.curated_events_bubble_path, "Event dates", "Not curated", "Curated", True) 209 | self.logger.debug(" Created {} and {}".format(key1, key2)) 210 | else: 211 | self.data_for_report[key1] = {} 212 | self.data_for_report[key2] = {} 213 | self.logger.error(" Not found: {} or {}".format(key1, key2)) 214 | 215 | key1 = "curation_orgs_complete" 216 | key2 = "curation_orgs_incomplete" 217 | if key1 in self.data and key2 in self.data: 218 | dataset = self.data[key1] 219 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 220 | self.data_for_report[key1] = sorted_data 221 | 222 | dataset = self.data[key2] 223 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 224 | self.data_for_report[key2] = sorted_data 225 | else: 226 | self.data_for_report[key1] = {} 227 | self.data_for_report[key2] = {} 228 | self.logger.error(" Not found: {} or {}".format(key1, key2)) 229 | 230 | template_css_file = self.template_css 231 | with open(template_css_file, "r") as f: 232 | css_content = f.read() 233 | 234 | template_file = self.template_curation_html 235 | with open(template_file, "r") as f: 236 | html_template = f.read() 237 | 238 | # Render the HTML 239 | template = Template(html_template) 240 | html_content = template.render( 241 | css=css_content, 242 | title="MISP Curation summary", 243 | logo=self.config["logo"], 244 | report_date=self.report_date, 245 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 246 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 247 | reporting_period=self.config["reporting_period"], 248 | 249 | report_misp_server=self.report_misp_server, 250 | summary=self.data_for_report.get("statistics", {}), 251 | curation_incomplete_today_count=curation_incomplete_today_count, 252 | curation_complete_today_count=curation_complete_today_count, 253 | curation_complete_count=curation_complete_count, 254 | curation_incomplete_count=curation_incomplete_count, 255 | 256 | curation_complete=curation_complete_events, 257 | curation_incomplete=curation_incomplete_events, 258 | 259 | curation_incomplete_high=curation_incomplete_high_events, 260 | curation_incomplete_high_count=curation_incomplete_high_count, 261 | curation_incomplete_adm_high=curation_incomplete_adm_high_events, 262 | curation_incomplete_adm_high_count=curation_incomplete_adm_high_count, 263 | 264 | curation_complete_org=self.data_for_report["curation_orgs_complete"], 265 | curation_incomplete_org=self.data_for_report["curation_orgs_incomplete"], 266 | 267 | curated_events_bubble=os.path.basename(self.curated_events_bubble_path) 268 | ) 269 | 270 | # Save the HTML file 271 | output_html_path = os.path.join(self.output_dir, "misp_curation.html") 272 | with open(output_html_path, "w") as f: 273 | f.write(html_content) 274 | return True 275 | 276 | def _aggregate_by_month(self, data_dict): 277 | monthly_data = defaultdict(int) 278 | for date_str, count in data_dict.items(): 279 | year_month = date_str[:7] # YYYY-MM 280 | monthly_data[year_month] += count 281 | return dict(monthly_data) 282 | 283 | def _month_range(self, start_ym, end_ym): 284 | start = datetime.strptime(start_ym, "%Y-%m") 285 | end = datetime.strptime(end_ym, "%Y-%m") 286 | current = start 287 | result = [] 288 | while current <= end: 289 | result.append(current.strftime("%Y-%m")) 290 | year = current.year 291 | month = current.month 292 | if month == 12: 293 | year += 1 294 | month = 1 295 | else: 296 | month += 1 297 | current = datetime(year, month, 1) 298 | return result 299 | 300 | def create_bubble_chart(self, values1, values2, full_months, output_path, title, data1_label, data2_label, full_width=False): 301 | figsize = (8, 4) if full_width else (6, 4) 302 | plt.figure(figsize=figsize) 303 | x_positions = list(range(len(full_months))) 304 | sizes1 = [v * 50 for v in values1] 305 | sizes2 = [v * 50 for v in values2] 306 | plt.scatter(x_positions, [0]*len(full_months), s=sizes1, alpha=0.6, c="#D35400", edgecolors="black", label=data1_label) 307 | plt.scatter(x_positions, [1]*len(full_months), s=sizes2, alpha=0.6, c="#F39C12", edgecolors="black", label=data2_label) 308 | 309 | plt.title(title, fontsize=10) 310 | 311 | tick_positions = x_positions[::6] 312 | tick_labels = [full_months[i] for i in tick_positions] 313 | plt.xticks(tick_positions, tick_labels, rotation=45, ha="right", fontsize=8) 314 | 315 | plt.ylim(-0.5, 1.5) 316 | plt.yticks([0, 1], [data1_label, data2_label]) 317 | plt.xlabel(" ", fontsize=8) 318 | plt.tight_layout() 319 | plt.savefig(output_path, dpi=100) 320 | plt.close() 321 | 322 | def render_report(self): 323 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 324 | 325 | self.logger.debug("Removing older images") 326 | for img in [self.events_trending_path, 327 | self.attributes_trending_path, 328 | self.attributes_type_bar_chart_path, 329 | self.attributes_type_daily_bar_chart_path, 330 | self.threatlevel_bar_chart_path, 331 | self.tlp_pie_chart_path, 332 | self.geo_targeting_map_path, 333 | self.sector_targeting_bar_chart_path 334 | ]: 335 | if os.path.exists(img): 336 | os.remove(img) 337 | 338 | # ############### Trending events 339 | key = "trending-events" 340 | if key in self.data: 341 | dataset = self.data[key] 342 | days = int(self.config["reporting_period"].strip("d")) 343 | sorted_keys = sorted(dataset.keys(), reverse=True) 344 | updated_dataset = {} 345 | highest_key = sorted_keys[0] 346 | updated_dataset[f"{highest_key + days}d-{highest_key}d"] = dataset[highest_key] 347 | for start, end in zip(sorted_keys[1:], sorted_keys): 348 | updated_dataset[f"{end}d-{start}d"] = dataset[start] 349 | self.data_for_report[key] = updated_dataset 350 | self.create_trending_graph(self.data_for_report[key], self.events_trending_path, "Trending events") 351 | self.logger.debug(" Created {}".format(self.events_trending_path)) 352 | else: 353 | self.events_trending_path = self.noimage_path 354 | self.data_for_report[key] = {} 355 | self.logger.error(" Not found: {}".format(key)) 356 | 357 | # ############### Trending attributes 358 | key = "trending-attributes" 359 | if key in self.data: 360 | dataset = self.data[key] 361 | dataset2 = self.data["trending-attributes_ids"] 362 | 363 | days = int(self.config["reporting_period"].strip("d")) 364 | sorted_keys = sorted(dataset.keys(), reverse=True) 365 | updated_dataset = {} 366 | highest_key = sorted_keys[0] 367 | updated_dataset[f"{highest_key + days}d-{highest_key}d"] = [dataset[highest_key], dataset2[highest_key]] 368 | for start, end in zip(sorted_keys[1:], sorted_keys): 369 | updated_dataset[f"{end}d-{start}d"] = [dataset[start], dataset2[start]] 370 | 371 | self.data_for_report[key] = updated_dataset 372 | self.create_trending_graph_double(self.data_for_report[key], self.attributes_trending_path, "Trending attributes", "Attributes", "With to_ids") 373 | self.logger.debug(" Created {}".format(self.attributes_trending_path)) 374 | else: 375 | self.attributes_trending_path = self.noimage_path 376 | self.data_for_report[key] = {} 377 | self.logger.error(" Not found: {}".format(key)) 378 | 379 | # ############### Statistics attributes 380 | key = "statistics-attributes" 381 | if key in self.data: 382 | dataset = self.data[key] 383 | self.data_for_report[key] = dataset 384 | self.create_bar_chart(self.data_for_report[key], self.attributes_type_bar_chart_path, "Attributes type distribution ({})".format(self.config["reporting_period"]), full_width=False, value_index=0) 385 | self.create_bar_chart(self.data_for_report[key], self.attributes_type_daily_bar_chart_path, "Attributes type distribution (24h)", full_width=False, value_index=1) 386 | self.data_for_report[key] = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 387 | self.logger.debug(" Created {}".format(self.attributes_type_bar_chart_path)) 388 | self.logger.debug(" Created {}".format(self.attributes_type_daily_bar_chart_path)) 389 | else: 390 | self.attributes_type_bar_chart_path = self.noimage_path 391 | self.attributes_type_daily_bar_chart_path = self.noimage_path 392 | self.data_for_report[key] = {} 393 | self.logger.error(" Not found: {}".format(key)) 394 | 395 | # ############### Statistics threatlevel 396 | key = "statistics-threatlevel" 397 | if key in self.data: 398 | dataset = self.data[key] 399 | updated_dataset = {self.threatlevel_key_mapping[key]: value for key, value in dataset.items()} 400 | self.data_for_report[key] = updated_dataset 401 | self.create_bar_chart(self.data_for_report[key], self.threatlevel_bar_chart_path, "Threat level", value_index=-1) 402 | self.logger.debug(" Created {}".format(self.threatlevel_bar_chart_path)) 403 | else: 404 | self.threatlevel_bar_chart_path = self.noimage_path 405 | self.data_for_report[key] = {} 406 | self.logger.error(" Not found: {}".format(key)) 407 | 408 | # ############### Statistics TLP 409 | key = "statistics-tlp" 410 | if key in self.data: 411 | dataset = self.data[key] 412 | updated_dataset = {} 413 | for delkey in dataset: 414 | if delkey not in self.tlp_ignore_graph: 415 | updated_dataset[delkey] = dataset[delkey] 416 | self.data_for_report[key] = dataset 417 | self.create_pie_chart(updated_dataset, self.tlp_pie_chart_path, "TLP", colors=["red", "orange", "green", "#d3d3d3", "#e8e6e6", "gray"]) 418 | self.logger.debug(" Created {}".format(self.tlp_pie_chart_path)) 419 | else: 420 | self.tlp_pie_chart_path = self.noimage_path 421 | self.data_for_report[key] = {} 422 | self.logger.error(" Not found: {}".format(key)) 423 | 424 | # ############### Statistics key organisations 425 | key = "statistics-keyorgs" 426 | if key in self.data: 427 | dataset = self.data[key] 428 | updated_dataset = {} 429 | for uuid in dataset: 430 | if uuid in self.key_organisations: 431 | try: 432 | org = self.misp.get_organisation(uuid) 433 | if "Organisation" in org: 434 | org_name = org["Organisation"]["name"] 435 | logo = self.key_organisations[uuid]["logo"] 436 | period_events = dataset[uuid]["reporting-period"]["events"] 437 | period_attributes = dataset[uuid]["reporting-period"]["attributes"] 438 | period_attributes_ids = dataset[uuid]["reporting-period"]["attributes_ids"] 439 | today_events = dataset[uuid]["today"]["events"] 440 | today_attributes = dataset[uuid]["today"]["attributes"] 441 | today_attributes_ids = dataset[uuid]["today"]["attributes_ids"] 442 | updated_dataset[org_name] = {"logo": f"{logo}", "org_uuid": f"{uuid}", "period_events": f"{period_events}", 443 | "period_attributes": f"{period_attributes}", 444 | "period_attributes_ids": f"{period_attributes_ids}", 445 | "today_events": f"{today_events}", 446 | "today_attributes": f"{today_attributes}", 447 | "today_attributes_ids": f"{today_attributes_ids}"} 448 | else: 449 | self.logger.error("Unable to get organisation info for {}".format(uuid)) 450 | except Exception as e: 451 | self.logger.error("Unable to get organisation info for {} - {}".format(uuid, e)) 452 | self.data_for_report[key] = updated_dataset 453 | self.logger.debug(" Created {}".format(key)) 454 | else: 455 | self.data_for_report[key] = {} 456 | self.logger.error(" Not found: {}".format(key)) 457 | 458 | # ############### General statistics 459 | key = "statistics" 460 | if key in self.data: 461 | dataset = self.data[key] 462 | updated_dataset = {} 463 | days = int(self.config["reporting_period"].strip("d")) 464 | current_date = datetime.now() 465 | past_date = current_date - timedelta(days=days) 466 | reporting_period = self.config["reporting_period"] 467 | updated_dataset["period"] = f"(until {past_date.strftime('%Y-%m-%d')})" 468 | if self.config["reporting_filter"] is not None: 469 | updated_dataset["period"] = "{}
MISP filters: {}".format(updated_dataset["period"], self.config["reporting_filter"]) 470 | if "trending-events" in self.data: 471 | updated_dataset["period_events"] = self.data["trending-events"][0] 472 | updated_dataset["period_attributes"] = self.data["trending-attributes"][0] 473 | updated_dataset["period_attributes_ids"] = self.data["trending-attributes_ids"][0] 474 | else: 475 | updated_dataset["period_events"] = "No data" 476 | updated_dataset["period_attributes"] = "No data" 477 | updated_dataset["period_attributes_ids"] = "No data" 478 | if "today-events" in self.data: 479 | updated_dataset["today_events"] = self.data["today-events"] 480 | updated_dataset["today_attributes"] = self.data["today-attributes"] 481 | updated_dataset["today_attributes_ids"] = self.data["today-attributes_ids"] 482 | else: 483 | updated_dataset["today_events"] = "No data" 484 | updated_dataset["today_attributes"] = "No data" 485 | updated_dataset["today_attributes_ids"] = "No data" 486 | updated_dataset["events"] = dataset["event_count"] 487 | updated_dataset["attributes"] = dataset["attribute_count"] 488 | updated_dataset["correlations"] = dataset["correlation_count"] 489 | updated_dataset["organisations"] = dataset["org_count"] 490 | updated_dataset["local_organisations"] = dataset["local_org_count"] 491 | updated_dataset["users"] = dataset["user_count"] 492 | self.data_for_report[key] = updated_dataset 493 | self.logger.debug(" Created {}".format(key)) 494 | else: 495 | self.data_for_report[key] = {} 496 | self.logger.error(" Not found: {}".format(key)) 497 | 498 | # ############### Event details 499 | key = "detailed_events" 500 | if key in self.data: 501 | dataset = self.data[key] 502 | self.data_for_report[key] = dataset 503 | self.logger.debug(" Created {}".format(key)) 504 | else: 505 | self.data_for_report[key] = {} 506 | self.logger.error(" Not found: {}".format(key)) 507 | 508 | # ############### Targeting geo 509 | key = "targeting-geo" 510 | if key in self.data and len(self.data[key]) > 0: 511 | dataset = self.data[key] 512 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 513 | if self.config["filter_geo_count"] > 0: 514 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_geo_count"]]) 515 | else: 516 | self.data_for_report[key] = sorted_data 517 | self.create_geo_targeting_map(self.data_for_report[key], self.geo_targeting_map_path) 518 | self.logger.debug(" Created {}".format(self.events_trending_path)) 519 | else: 520 | self.geo_targeting_map_path = self.noimage_path 521 | self.data_for_report[key] = {} 522 | self.logger.error(" Not found: {}".format(key)) 523 | 524 | # ############### Targeting sector 525 | key = "targeting-sector" 526 | if key in self.data and len(self.data[key]) > 0: 527 | dataset = self.data[key] 528 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 529 | if self.config["filter_sector_count"] > 0: 530 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_sector_count"]]) 531 | else: 532 | self.data_for_report[key] = sorted_data 533 | self.create_horizontal_bar_chart(self.data_for_report[key], self.sector_targeting_bar_chart_path, "Sector targeting") 534 | self.logger.debug(" Created {}".format(self.events_trending_path)) 535 | else: 536 | self.sector_targeting_bar_chart_path = self.noimage_path 537 | self.data_for_report[key] = {} 538 | self.logger.error(" Not found: {}".format(key)) 539 | 540 | # ############### TTP pattern 541 | key = "ttp_pattern" 542 | if key in self.data and len(self.data[key]) > 0: 543 | dataset = self.data[key] 544 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 545 | if self.config["filter_ttp_pattern_count"] > 0: 546 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_ttp_pattern_count"]]) 547 | else: 548 | self.data_for_report[key] = sorted_data 549 | self.logger.debug(" Created {}".format(key)) 550 | else: 551 | self.data_for_report[key] = {} 552 | self.logger.error(" Not found: {}".format(key)) 553 | 554 | # ############### TTP actors 555 | key = "ttp_actors" 556 | if key in self.data and len(self.data[key]) > 0: 557 | dataset = self.data[key] 558 | self.data_for_report[key] = dataset 559 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1], reverse=True)) 560 | if self.config["filter_ttp_actors_count"] > 0: 561 | self.data_for_report[key] = dict(list(sorted_data.items())[:self.config["filter_ttp_actors_count"]]) 562 | else: 563 | self.data_for_report[key] = sorted_data 564 | self.logger.debug(" Created {}".format(key)) 565 | else: 566 | self.data_for_report[key] = {} 567 | self.logger.error(" Not found: {}".format(key)) 568 | 569 | # ############### Vulnerabilities 570 | key = "vulnerabilities" 571 | if key in self.data and len(self.data[key]) > 0: 572 | dataset = self.data[key] 573 | self.data_for_report[key] = {} 574 | sorted_data = dict(sorted(dataset.items(), key=lambda item: item[1]["count"], reverse=True)) 575 | self.data_for_report[key] = sorted_data 576 | self.logger.debug(" Created {}".format(key)) 577 | else: 578 | self.data_for_report[key] = {} 579 | self.logger.error(" Not found: {}".format(key)) 580 | 581 | # ############### Labels 582 | if self.config["reporting_filter_timestamp"] == "timestamp": 583 | reporting_filter_timestamp = "recently changed" 584 | else: 585 | reporting_filter_timestamp = "published" 586 | 587 | if self.config["reporting_filter_attribute_type_ids"] == True: 588 | attributes_with_ids_or_not = "(only attributes where to_ids is set to true)" 589 | else: 590 | attributes_with_ids_or_not = "(all attributes, regardless of the to_ids flag)" 591 | 592 | template_css_file = self.template_css 593 | with open(template_css_file, "r") as f: 594 | css_content = f.read() 595 | 596 | template_file = self.template_html 597 | with open(template_file, "r") as f: 598 | html_template = f.read() 599 | 600 | template = Template(html_template) 601 | html_content = template.render( 602 | css=css_content, 603 | title="MISP Summary", 604 | logo=self.config["logo"], 605 | report_date=self.report_date, 606 | report_timestamp=datetime.now().strftime('%Y%m%d %H%M%S'), 607 | report_timestamp_hm=datetime.now().strftime('%Y-%m-%d'), 608 | reporting_period=self.config["reporting_period"], 609 | report_misp_server=self.report_misp_server, 610 | summary=self.data_for_report.get("statistics", {}), 611 | trending_events=self.data_for_report.get("trending-events", {}), 612 | trending_attributes=self.data_for_report.get("trending-attributes", {}), 613 | detailed_events=self.data_for_report["detailed_events"], 614 | print_event_details=self.config["print_event_details"], 615 | attributes_type=self.data_for_report["statistics-attributes"], 616 | threatlevel=self.data_for_report["statistics-threatlevel"], 617 | tlp=self.data_for_report["statistics-tlp"], 618 | keyorgs=self.data_for_report["statistics-keyorgs"], 619 | target_geo=self.data_for_report["targeting-geo"], 620 | target_sector=self.data_for_report["targeting-sector"], 621 | ttp_pattern=self.data_for_report["ttp_pattern"], 622 | ttp_actors=self.data_for_report["ttp_actors"], 623 | vulnerabilities=self.data_for_report["vulnerabilities"], 624 | 625 | events_trending_path=os.path.basename(self.events_trending_path), 626 | attributes_trending_path=os.path.basename(self.attributes_trending_path), 627 | attributes_type_bar_chart_path=os.path.basename(self.attributes_type_bar_chart_path), 628 | attributes_type_daily_bar_chart_path=os.path.basename(self.attributes_type_daily_bar_chart_path), 629 | threatlevel_bar_chart_path=os.path.basename(self.threatlevel_bar_chart_path), 630 | tlp_pie_chart_path=os.path.basename(self.tlp_pie_chart_path), 631 | geo_targeting_map_path=os.path.basename(self.geo_targeting_map_path), 632 | sector_targeting_bar_chart_path=os.path.basename(self.sector_targeting_bar_chart_path), 633 | 634 | reporting_filter_timestamp=reporting_filter_timestamp, 635 | vulnerability_lookup_url=self.config["vulnerability_lookup_url"], 636 | attributes_with_ids_or_not=attributes_with_ids_or_not, 637 | cve_highlight=self.config["reporting_cve_highlight"], 638 | ) 639 | 640 | output_html_path = os.path.join(self.output_dir, "misp_summary.html") 641 | with open(output_html_path, "w") as f: 642 | f.write(html_content) 643 | return True 644 | 645 | def create_geo_targeting_map(self, data, output_path): 646 | countries = list(data.keys()) 647 | counts = list(data.values()) 648 | fig = px.choropleth( 649 | locations=countries, 650 | locationmode="country names", 651 | color=counts, 652 | labels={"color": "Count"}, 653 | ) 654 | fig.write_image(output_path) 655 | 656 | def create_horizontal_bar_chart(self, data, output_path, title): 657 | labels = list(data.keys()) 658 | values = list(data.values()) 659 | 660 | if all(v == 0 for v in values): 661 | values = [0.1] * len(values) # Avoid fully empty chart 662 | 663 | cmap = plt.colormaps['Oranges'] 664 | colors = [cmap(i / len(labels)) for i in range(len(labels))] 665 | 666 | plt.figure(figsize=(6, 4)) 667 | plt.barh(labels, values, color=colors) 668 | plt.title(title) 669 | plt.xlabel("Count") 670 | 671 | # If all values are just dummy 0.1 672 | if all(v == 0.1 for v in values): 673 | plt.text( 674 | 0.5, 0.5, 675 | "No data available", 676 | fontsize=12, ha="center", 677 | transform=plt.gca().transAxes 678 | ) 679 | # Just keep a simple 0,1 scale 680 | plt.xticks([0, 1]) 681 | else: 682 | max_val = max(values) 683 | # We'll consider integer ticks up to at least ceil of max_val 684 | max_int = int(np.ceil(max_val)) 685 | 686 | if max_int > 5: 687 | # Generate exactly 5 ticks (0 to max_val) 688 | ticks = np.linspace(0, max_val, 5) 689 | # Round them to integers if desired 690 | tick_labels = [int(round(t)) for t in ticks] 691 | plt.xticks(ticks, tick_labels) 692 | else: 693 | # Less than or equal to 5, so just show integer ticks 694 | ticks = range(0, max_int + 1) 695 | plt.xticks(ticks, ticks) 696 | 697 | plt.tight_layout() 698 | plt.savefig(output_path, dpi=100) 699 | plt.close() 700 | 701 | def create_bar_chart(self, data, output_path, title, full_width=False, value_index=0): 702 | labels = list(data.keys()) 703 | if value_index == -1: 704 | values = list(data.values()) 705 | else: 706 | values = [v[value_index] for v in data.values()] 707 | 708 | if all(v == 0 for v in values): 709 | values = [0.1] * len(values) # Avoid fully empty chart 710 | 711 | figsize = (8, 4) if full_width else (4, 3) 712 | 713 | cmap = plt.colormaps['Oranges'] 714 | colors = [cmap(i / len(labels)) for i in range(len(labels))] 715 | 716 | plt.figure(figsize=figsize) 717 | plt.bar(labels, values, color=colors) 718 | plt.title(title, fontsize=10) 719 | plt.ylabel("Count", fontsize=8) 720 | 721 | ax = plt.gca() 722 | ax.xaxis.set_major_locator(MultipleLocator(1)) 723 | plt.xticks(fontsize=8, rotation=45) 724 | plt.yticks(fontsize=8) 725 | 726 | # If dummy values were used, indicate it 727 | if all(v == 0.1 for v in values): 728 | plt.text(0.5, 0.5, "No data available", fontsize=12, ha="center", transform=plt.gca().transAxes) 729 | 730 | plt.tight_layout() 731 | plt.savefig(output_path, dpi=100) 732 | plt.close() 733 | 734 | def create_pie_chart(self, data, output_path, title, colors): 735 | labels = list(data.keys()) 736 | sizes = list(data.values()) 737 | 738 | if sum(sizes) == 0: 739 | self.logger.info("No data to display in pie chart.") 740 | plt.figure(figsize=(4, 3)) 741 | plt.pie( 742 | [1], # Single value to create a full circle 743 | labels=["No data"], 744 | colors=["lightgrey"], 745 | startangle=90, 746 | wedgeprops={'width': 0.4} 747 | ) 748 | plt.title(title, fontsize=10) 749 | plt.axis("equal") 750 | plt.tight_layout() 751 | plt.savefig(output_path, dpi=100) 752 | plt.close() 753 | else: 754 | plt.figure(figsize=(4, 3)) 755 | plt.pie( 756 | sizes, 757 | labels=labels, 758 | autopct="%1.1f%%", 759 | startangle=90, 760 | colors=colors, 761 | wedgeprops={'width': 0.4} 762 | ) 763 | plt.title(title, fontsize=10) 764 | plt.axis("equal") 765 | plt.tight_layout() 766 | plt.savefig(output_path, dpi=100) 767 | plt.close() 768 | 769 | def create_trending_graph(self, data, output_path, title): 770 | months = list(data.keys()) 771 | values = list(data.values()) 772 | 773 | plt.figure(figsize=(4, 3)) 774 | plt.plot(months, values, marker="o", color="#FF6600") 775 | plt.title(title, fontsize=10) 776 | plt.ylabel("Count", fontsize=8) 777 | plt.xticks(fontsize=8, rotation=45) 778 | plt.yticks(fontsize=8) 779 | plt.grid(True, linestyle='--', alpha=0.7) 780 | plt.tight_layout() 781 | plt.savefig(output_path, dpi=100) 782 | plt.close() 783 | 784 | def create_trending_graph_double(self, data, output_path, title, label1, label2): 785 | months = list(data.keys()) 786 | pairs = list(data.values()) 787 | 788 | # Separate out the first/second data points in each pair 789 | first_values = [p[0] for p in pairs] 790 | second_values = [p[1] for p in pairs] 791 | 792 | plt.figure(figsize=(4, 3)) 793 | plt.plot(months, first_values, marker="o", color="#ffcc00", label=label1) 794 | plt.plot(months, second_values, marker="o", color="#ff1e00", label=label2) 795 | 796 | plt.title(title, fontsize=10) 797 | plt.ylabel("Count", fontsize=8) 798 | plt.xticks(fontsize=8, rotation=45) 799 | plt.yticks(fontsize=8) 800 | plt.grid(True, linestyle='--', alpha=0.7) 801 | plt.legend(fontsize=8) 802 | plt.tight_layout() 803 | plt.savefig(output_path, dpi=100) 804 | plt.close() 805 | -------------------------------------------------------------------------------- /clsReportingData.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta, timezone 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | import numpy as np 14 | 15 | 16 | class ReportingData(): 17 | def __init__(self, config, logger): 18 | self.filters = config["reporting_filter"] 19 | self.config = config 20 | self.logger = logger 21 | self.misp_headers = {"Authorization": self.config["misp_key"], "Content-Type": "application/json", "Accept": "application/json"} 22 | if self.config["misp_verifycert"] is False: 23 | import urllib3 24 | urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 25 | self.misp = PyMISP(self.config["misp_url"], self.config["misp_key"], self.config["misp_verifycert"]) 26 | self.data = {} 27 | self.data_for_reporting_period = False 28 | self.data_for_today = False 29 | 30 | self.attribute_summary = self.config["attribute_summary"] 31 | self.attribute_other = self.config["attribute_other"] 32 | self.key_organisations = self.config["key_organisations"] 33 | self.threatlevel_key_mapping = self.config["threatlevel_key_mapping"] 34 | 35 | self.workflow_complete = self.config["workflow_complete"] 36 | self.workflow_incomplete = self.config["workflow_incomplete"] 37 | 38 | self.filter_sector = self.config["filter_sector"] 39 | self.filter_geo = self.config["filter_geo"] 40 | 41 | self.filter_ttp_actors = self.config["filter_ttp_actors"] 42 | self.filter_ttp_pattern = self.config["filter_ttp_pattern"] 43 | 44 | def print(self): 45 | print(self.data) 46 | 47 | def get_statistics(self): 48 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 49 | 50 | statistics = self._request_get("/users/statistics") 51 | self.data["statistics"] = {"user_count": 0, "org_count": 0, "local_org_count": 0, "event_count": 0, "attribute_count": 0, 52 | "correlation_count": 0} 53 | if statistics and statistics.json().get("stats", False): 54 | json_statistics = statistics.json()["stats"] 55 | self.data["statistics"]["user_count"] = json_statistics["user_count"] 56 | self.data["statistics"]["org_count"] = json_statistics["org_count"] 57 | self.data["statistics"]["local_org_count"] = json_statistics["local_org_count"] 58 | self.data["statistics"]["event_count"] = json_statistics["event_count"] 59 | self.data["statistics"]["attribute_count"] = json_statistics["attribute_count"] 60 | self.data["statistics"]["correlation_count"] = json_statistics["correlation_count"] 61 | 62 | def get_today_events_attributes(self): 63 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 64 | 65 | self.data["today-events"] = {} 66 | self.data["today-attributes"] = {} 67 | self.data["today-attributes_ids"] = {} 68 | 69 | response = self._get_data_for_today() 70 | self.data["today-events"] = len(response) 71 | 72 | attributesqt = 0 73 | attributesqt_ids = 0 74 | for event in response: 75 | attributesqt += len(event["Event"]["Attribute"]) 76 | for attr in event["Event"]["Attribute"]: 77 | if attr["to_ids"] == 1: 78 | attributesqt_ids += 1 79 | for misp_object in event["Event"]["Object"]: 80 | attributesqt += len(misp_object["Attribute"]) 81 | for attr in misp_object["Attribute"]: 82 | if attr["to_ids"] == 1: 83 | attributesqt_ids += 1 84 | self.data["today-attributes"] = attributesqt 85 | self.data["today-attributes_ids"] = attributesqt_ids 86 | 87 | def get_trending_events_attributes(self): 88 | # Get the trends data 89 | # - First the data for the reporting period 90 | # - Then the data for the last reporting_trending_count (fe. 6) periods 91 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 92 | 93 | self.data["trending-events"] = {} 94 | self.data["trending-attributes"] = {} 95 | self.data["trending-attributes_ids"] = {} 96 | 97 | days = int(''.join(filter(str.isdigit, self.config["reporting_period"]))) 98 | self.logger.debug(" Get {}".format(days)) 99 | response = self._get_data_for_reporting_period() 100 | self.data["trending-events"][0] = len(response) 101 | 102 | attributesqt = 0 103 | attributesqt_ids = 0 104 | for event in response: 105 | attributesqt += len(event["Event"]["Attribute"]) 106 | for attr in event["Event"]["Attribute"]: 107 | if attr["to_ids"] == 1: 108 | attributesqt_ids += 1 109 | for misp_object in event["Event"]["Object"]: 110 | attributesqt += len(misp_object["Attribute"]) 111 | for attr in misp_object["Attribute"]: 112 | if attr["to_ids"] == 1: 113 | attributesqt_ids += 1 114 | self.data["trending-attributes"][0] = attributesqt 115 | self.data["trending-attributes_ids"][0] = attributesqt_ids 116 | 117 | count = 1 118 | while self.config["reporting_trending_count"] > count: 119 | start_period = days * count 120 | end_period = days * (count + 1) 121 | #timestamp_filter = ["{}d".format(start_period), "{}d".format(end_period)] 122 | self.logger.debug(" Get {} - {}".format(start_period, end_period)) 123 | 124 | current_page = 1 125 | tmp_len = 0 126 | response = [] 127 | while True: 128 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], ["{}d".format(start_period), "{}d".format(end_period)]) 129 | tmp_response = self.misp.search("events", **filter_params) 130 | if len(tmp_response) > 0: 131 | tmp_len = tmp_len + len(tmp_response) 132 | response += tmp_response 133 | else: 134 | break 135 | current_page += 1 136 | self.data["trending-events"][start_period] = tmp_len 137 | 138 | attributesqt = 0 139 | attributesqt_ids = 0 140 | for event in response: 141 | attributesqt += len(event["Event"]["Attribute"]) 142 | for attr in event["Event"]["Attribute"]: 143 | if attr["to_ids"] == 1: 144 | attributesqt_ids += 1 145 | for misp_object in event["Event"]["Object"]: 146 | attributesqt += len(misp_object["Attribute"]) 147 | for attr in misp_object["Attribute"]: 148 | if attr["to_ids"] == 1: 149 | attributesqt_ids += 1 150 | self.data["trending-attributes"][start_period] = attributesqt 151 | self.data["trending-attributes_ids"][start_period] = attributesqt_ids 152 | 153 | count += 1 154 | 155 | def get_statistics_attributes(self): 156 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 157 | self.data["statistics-attributes"] = {} 158 | 159 | response_reporting_period = self._get_data_for_reporting_period() 160 | self._process_attribute_counts(response_reporting_period, index=0) 161 | response_today = self._get_data_for_today() 162 | self._process_attribute_counts(response_today, index=1) 163 | 164 | def _process_attribute_counts(self, events, index): 165 | for event in events: 166 | for attribute in event["Event"]["Attribute"]: 167 | if self.config["reporting_filter_attribute_type_ids"] and attribute["to_ids"] == 0: 168 | continue 169 | 170 | attribute_type = self._convert_attribute_category(attribute["type"]) 171 | if attribute_type and attribute_type not in self.data["statistics-attributes"]: 172 | self.data["statistics-attributes"][attribute_type] = [0, 0] 173 | self.data["statistics-attributes"][attribute_type][index] += 1 174 | 175 | for misp_object in event["Event"]["Object"]: 176 | for attr in misp_object["Attribute"]: 177 | if self.config["reporting_filter_attribute_type_ids"] and attr["to_ids"] == 0: 178 | continue 179 | 180 | attribute_type = self._convert_attribute_category(attr["type"]) 181 | if attribute_type and attribute_type not in self.data["statistics-attributes"]: 182 | self.data["statistics-attributes"][attribute_type] = [0, 0] 183 | self.data["statistics-attributes"][attribute_type][index] += 1 184 | 185 | def get_statistics_keyorgs(self): 186 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 187 | self.data["statistics-keyorgs"] = {} 188 | 189 | org_uuid_list = list(self.key_organisations) 190 | if len(org_uuid_list) > 0: 191 | for orgc in org_uuid_list: 192 | self.data["statistics-keyorgs"][orgc] = {"reporting-period": {"events": 0, "attributes": 0, "attributes_ids": 0}, 193 | "today": {"events": 0, "attributes": 0, "attributes_ids": 0}} 194 | 195 | response = self._get_data_for_reporting_period() 196 | self._process_get_statistics_keyorgs(response, "reporting-period") 197 | response = self._get_data_for_today() 198 | self._process_get_statistics_keyorgs(response, "today") 199 | 200 | def get_threatlevel(self): 201 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 202 | self.data["statistics-threatlevel"] = {"1": 0, "2": 0, "3": 0, "4": 0} 203 | response = self._get_data_for_reporting_period() 204 | for event in response: 205 | threat_level_id = event["Event"]["threat_level_id"] 206 | self.data["statistics-threatlevel"][threat_level_id] += 1 207 | 208 | def get_tlplevel(self): 209 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 210 | self.data["statistics-tlp"] = {"tlp:red": 0, "tlp:amber": 0, "tlp:amber+strict": 0, "tlp:green": 0, "tlp:clear": 0, "tlp:ex:chr": 0, "tlp:unclear": 0} 211 | response = self._get_data_for_reporting_period() 212 | 213 | for event in response: 214 | tags = event["Event"].get("Tag", []) 215 | if len(tags) > 0: 216 | for tag in tags: 217 | if tag["name"].startswith("tlp:"): 218 | if tag["name"] == "tlp:white": 219 | tag_tlp = "tlp:clear" 220 | else: 221 | tag_tlp = tag["name"] 222 | self.data["statistics-tlp"][tag_tlp] += 1 223 | 224 | def get_eventdetails(self): 225 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 226 | self.data["detailed_events"] = [] 227 | response = self._get_data_for_reporting_period() 228 | self.distribution = self.config["distribution"] 229 | self.analysis_state = self.config["analysis"] 230 | 231 | for event in response: 232 | process_event = False 233 | if self.config["reporting_eventdetails_onlykeyorgs"]: 234 | if event["Event"]["Orgc"]["uuid"] in self.config["key_organisations"]: 235 | process_event = True 236 | else: 237 | process_event = True 238 | if process_event: 239 | tags = event["Event"].get("Tag", []) 240 | tag_tlp = "" 241 | if len(tags) > 0: 242 | for tag in tags: 243 | if tag["name"].startswith("tlp:"): 244 | if tag["name"] == "tlp:white": 245 | tag_tlp = "tlp:clear" 246 | else: 247 | tag_tlp = tag["name"] 248 | entry = {"date": event["Event"]["date"], 249 | "id": event["Event"]["id"], 250 | "org": event["Event"]["Orgc"]["name"], 251 | "distribution": self.distribution[int(event["Event"]["distribution"])], 252 | "analysis": self.analysis_state[int(event["Event"]["analysis"])], 253 | "published": event["Event"]["published"], 254 | "threat_level": self.threatlevel_key_mapping[event["Event"]["threat_level_id"]], 255 | "tlp": tag_tlp, 256 | "info": event["Event"]["info"][:30], 257 | "indicators": event["Event"]["attribute_count"]} 258 | self.data["detailed_events"].append(entry) 259 | 260 | def get_ttp(self): 261 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 262 | self.data["ttp_pattern"] = {} 263 | self.data["ttp_actors"] = {} 264 | response = self._get_data_for_reporting_period() 265 | for event in response: 266 | tags = event["Event"].get("Tag", []) 267 | if len(tags) > 0: 268 | for tag in tags: 269 | for ttp in self.filter_ttp_pattern: 270 | if ttp in tag["name"]: 271 | item = tag["name"].split("{}=".format(ttp))[1].replace("\"", "") 272 | if item in self.data["ttp_pattern"]: 273 | self.data["ttp_pattern"][item] += 1 274 | else: 275 | self.data["ttp_pattern"][item] = 1 276 | for ttp in self.filter_ttp_actors: 277 | if ttp in tag["name"]: 278 | item = tag["name"].split("{}=".format(ttp))[1].replace("\"", "") 279 | if item in self.data["ttp_actors"]: 280 | self.data["ttp_actors"][item] += 1 281 | else: 282 | self.data["ttp_actors"][item] = 1 283 | 284 | def get_target_sector(self): 285 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 286 | self.data["targeting-sector"] = {} 287 | response = self._get_data_for_reporting_period() 288 | for event in response: 289 | tags = event["Event"].get("Tag", []) 290 | if len(tags) > 0: 291 | for tag in tags: 292 | if self.filter_sector in tag["name"]: 293 | sector = tag["name"].split("{}=".format(self.filter_sector))[1].replace("\"", "") 294 | if sector in self.data["targeting-sector"]: 295 | self.data["targeting-sector"][sector] += 1 296 | else: 297 | self.data["targeting-sector"][sector] = 1 298 | 299 | def get_target_geo(self): 300 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 301 | self.data["targeting-geo"] = {} 302 | response = self._get_data_for_reporting_period() 303 | for event in response: 304 | tags = event["Event"].get("Tag", []) 305 | if len(tags) > 0: 306 | for tag in tags: 307 | if self.filter_geo in tag["name"]: 308 | geo = tag["name"].split("{}=".format(self.filter_geo))[1].replace("\"", "") 309 | if geo in self.data["targeting-geo"]: 310 | self.data["targeting-geo"][geo] += 1 311 | else: 312 | self.data["targeting-geo"][geo] = 1 313 | 314 | def get_vulnerabilities(self): 315 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 316 | self.data["vulnerabilities"] = {} 317 | tmp_data = {} 318 | response = self._get_data_for_reporting_period() 319 | for event in response: 320 | for attribute in event["Event"]["Attribute"]: 321 | if attribute["type"] == "vulnerability": 322 | if attribute["value"] in tmp_data: 323 | tmp_data[attribute["value"]] += 1 324 | else: 325 | tmp_data[attribute["value"]] = 1 326 | for misp_object in event["Event"]["Object"]: 327 | for attribute in misp_object["Attribute"]: 328 | if attribute["type"] == "vulnerability": 329 | if attribute["value"] in tmp_data: 330 | tmp_data[attribute["value"]] += 1 331 | else: 332 | tmp_data[attribute["value"]] = 1 333 | 334 | for cve in tmp_data: 335 | cve_url = self.config["cve_url"] 336 | try: 337 | response = requests.get(f"{cve_url}/{cve}") 338 | cve_data = response.json() 339 | cvss_base_score = cve_data["containers"]["cna"]["metrics"][0]["cvssV3_1"]["baseScore"] 340 | cve_summary = cve_data["containers"]["cna"]["descriptions"][0]["value"] 341 | except Exception as e: 342 | self.logger.debug("Unable to get CVE details for {} - {}".format(cve, e)) 343 | cvss_base_score = "?" 344 | cve_summary = "" 345 | entry = {"count": tmp_data[cve], "summary": cve_summary, "cvss3": cvss_base_score} 346 | self.data["vulnerabilities"][cve] = entry 347 | 348 | def get_curation(self): 349 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 350 | self.data["curation_complete"] = [] 351 | self.data["curation_complete_today"] = [] 352 | self.data["curation_incomplete"] = [] 353 | self.data["curation_incomplete_today"] = [] 354 | self.data["curation_complete_date"] = {} 355 | self.data["curation_incomplete_date"] = {} 356 | self.data["curation_orgs_complete"] = {} 357 | self.data["curation_orgs_incomplete"] = {} 358 | self.data["curation_incomplete_high"] = [] 359 | self.data["curation_incomplete_adm_high"] = [] 360 | 361 | self.distribution = self.config["distribution"] 362 | self.analysis_state = self.config["analysis"] 363 | 364 | # Reset to get curation data (published, and not published) 365 | self.data_for_reporting_period = None 366 | response = self._get_data_for_reporting_period(published=None) 367 | 368 | for event in response: 369 | complete_event = False 370 | entry = {"date": event["Event"]["date"], 371 | "id": event["Event"]["id"], 372 | "org": event["Event"]["Orgc"]["name"], 373 | "info": event["Event"]["info"][:30], 374 | "indicators": event["Event"]["attribute_count"]} 375 | if event["Event"]["published"]: 376 | tags = event["Event"].get("Tag", []) 377 | if len(tags) > 0: 378 | for tag in tags: 379 | if tag["name"] == self.workflow_complete: 380 | publish_timestamp_str = event["Event"]["publish_timestamp"] 381 | publish_timestamp = int(publish_timestamp_str) 382 | publish_time = datetime.fromtimestamp(publish_timestamp, tz=timezone.utc) 383 | now = datetime.now(timezone.utc) 384 | if (now - publish_time) < timedelta(days=1): 385 | self.data["curation_complete_today"].append(entry) 386 | self.data["curation_complete"].append(entry) 387 | 388 | if event["Event"]["date"] in self.data["curation_complete_date"]: 389 | self.data["curation_complete_date"][event["Event"]["date"]] += 1 390 | else: 391 | self.data["curation_complete_date"][event["Event"]["date"]] = 1 392 | if event["Event"]["Orgc"]["name"] in self.data["curation_orgs_complete"]: 393 | self.data["curation_orgs_complete"][event["Event"]["Orgc"]["name"]] += 1 394 | else: 395 | self.data["curation_orgs_complete"][event["Event"]["Orgc"]["name"]] = 1 396 | complete_event = True 397 | break 398 | 399 | if not complete_event: 400 | if self.config["log_incomplete"]: 401 | self.logger.debug("Consider event {} {} as incomplete".format(event["Event"]["id"], event["Event"]["info"])) 402 | publish_timestamp_str = event["Event"]["publish_timestamp"] 403 | publish_timestamp = int(publish_timestamp_str) 404 | publish_time = datetime.fromtimestamp(publish_timestamp, tz=timezone.utc) 405 | now = datetime.now(timezone.utc) 406 | if (now - publish_time) < timedelta(days=1): 407 | self.data["curation_incomplete_today"].append(entry) 408 | 409 | # Additional reporting for curation of today 410 | if event["Event"]["threat_level_id"] == "1": 411 | self.data["curation_incomplete_high"].append(entry) 412 | tags = event["Event"].get("Tag", []) 413 | for tag in tags: 414 | if tag["name"] == "admiralty-scale:source-reliability=\"a\"": 415 | self.data["curation_incomplete_adm_high"].append(entry) 416 | break 417 | 418 | self.data["curation_incomplete"].append(entry) 419 | 420 | if event["Event"]["date"] in self.data["curation_incomplete_date"]: 421 | self.data["curation_incomplete_date"][event["Event"]["date"]] += 1 422 | else: 423 | self.data["curation_incomplete_date"][event["Event"]["date"]] = 1 424 | if event["Event"]["Orgc"]["name"] in self.data["curation_orgs_incomplete"]: 425 | self.data["curation_orgs_incomplete"][event["Event"]["Orgc"]["name"]] += 1 426 | else: 427 | self.data["curation_orgs_incomplete"][event["Event"]["Orgc"]["name"]] = 1 428 | 429 | def get_infrastructure(self): 430 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 431 | 432 | def _process_get_statistics_keyorgs(self, response, period): 433 | for event in response: 434 | orgc = event["Event"]["Orgc"]["uuid"] 435 | if self.key_organisations.get(orgc, False): 436 | attributesqt = 0 437 | attributesqt_ids = 0 438 | attributesqt = len(event["Event"]["Attribute"]) 439 | for attr in event["Event"]["Attribute"]: 440 | if attr["to_ids"] == 1: 441 | attributesqt_ids += 1 442 | 443 | for misp_object in event["Event"]["Object"]: 444 | attributesqt += len(misp_object["Attribute"]) 445 | for attr in misp_object["Attribute"]: 446 | if attr["to_ids"] == 1: 447 | attributesqt_ids += 1 448 | 449 | self.data["statistics-keyorgs"][orgc][period]["events"] += 1 450 | self.data["statistics-keyorgs"][orgc][period]["attributes"] += attributesqt 451 | self.data["statistics-keyorgs"][orgc][period]["attributes_ids"] += attributesqt_ids 452 | 453 | def _build_misp_filter(self, current_page, published, date_filter): 454 | filter_params = { 455 | "limit": self.config["misp_page_size"], 456 | "page": current_page, 457 | "published": published, 458 | "tags": self.config["reporting_filter"] 459 | } 460 | if self.config["reporting_filter_timestamp"] == "timestamp": 461 | filter_params["timestamp"] = date_filter 462 | filter_params["published"] = None 463 | elif self.config["reporting_filter_timestamp"] == "published": 464 | filter_params["publish_timestamp"] = date_filter 465 | else: # default to published 466 | filter_params["publish_timestamp"] = date_filter 467 | return filter_params 468 | 469 | def _get_data_for_reporting_period(self, published=True): 470 | response = [] 471 | if not self.data_for_reporting_period: 472 | current_page = 1 473 | while True: 474 | if not published: 475 | filter_params = self._build_misp_filter(current_page, published, self.config["reporting_period"]) 476 | else: 477 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], self.config["reporting_period"]) 478 | tmp_reponse = self.misp.search("events", **filter_params) 479 | if len(tmp_reponse) > 0: 480 | response = response + tmp_reponse 481 | else: 482 | break 483 | current_page += 1 484 | self.data_for_reporting_period = response 485 | return self.data_for_reporting_period 486 | 487 | def _get_data_for_today(self, published=True): 488 | response = [] 489 | if not self.data_for_today: 490 | current_page = 1 491 | while True: 492 | filter_params = self._build_misp_filter(current_page, self.config["reporting_filter_published"], "1d") 493 | tmp_reponse = self.misp.search("events", **filter_params) 494 | if len(tmp_reponse) > 0: 495 | response = response + tmp_reponse 496 | else: 497 | break 498 | current_page += 1 499 | self.data_for_today = response 500 | return self.data_for_today 501 | 502 | def _convert_attribute_category(self, category): 503 | found_key = None 504 | for key, values in self.attribute_summary.items(): 505 | if category in values: 506 | found_key = key 507 | break 508 | if found_key: 509 | return found_key 510 | return self.attribute_other 511 | 512 | def _request_get(self, endpoint): 513 | response = requests.get("{}/{}".format(self.config["misp_url"], endpoint), headers=self.misp_headers, verify=self.config["misp_verifycert"]) 514 | if response.ok: 515 | return response 516 | elif 400 <= response.status_code < 500: 517 | self.logger.error(f"[{response.status_code}] Client Error: {response.reason}") 518 | elif 500 <= response.status_code < 600: 519 | self.logger.error(f"[{response.status_code}] Server Error: {response.reason}") 520 | else: 521 | self.logger.error(f"[{response.status_code}] Other: {response.reason}") 522 | return False 523 | 524 | def get_misp_statistics(self): 525 | statistics = {"event_count": self.data["statistics"]["event_count"], 526 | "attribute_count": self.data["statistics"]["attribute_count"], 527 | "user_count": self.data["statistics"]["user_count"], 528 | "org_count": self.data["statistics"]["org_count"], 529 | "local_org_count": self.data["statistics"]["local_org_count"]} 530 | 531 | today_statistics = {"today_event_count": self.data["today-attributes"], 532 | "today_attribute_count": self.data["today-attributes"], 533 | "today_attribute_ids_count": self.data["today-attributes_ids"]} 534 | 535 | filtered_statistics_attributes = { 536 | key: value[1] 537 | for key, value in self.data["statistics-attributes"].items() 538 | } 539 | 540 | self.statistics = statistics 541 | self.today_statistics = today_statistics 542 | self.statistics_attributes = filtered_statistics_attributes 543 | -------------------------------------------------------------------------------- /clsReportingStatistics.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import requests 4 | from datetime import datetime, timedelta 5 | import logging 6 | from pymisp import * 7 | import inspect 8 | import os 9 | import matplotlib.pyplot as plt 10 | from jinja2 import Template 11 | import plotly.express as px 12 | import matplotlib.cm as cm 13 | from matplotlib.ticker import MultipleLocator 14 | import numpy as np 15 | from collections import defaultdict 16 | 17 | 18 | class ReportingStatistics: 19 | def __init__(self, config, logger, statistics, today_statistics, statistics_attributes, output_dir="report_output"): 20 | self.config = config 21 | self.logger = logger 22 | 23 | self.statistics = statistics 24 | self.today_statistics = today_statistics 25 | self.statistics_attributes = statistics_attributes 26 | 27 | self.report_date = datetime.now().strftime("%Y-%m-%d") 28 | self.report_misp_server = self.config["misp_url"] 29 | self.reporting_period = self.config["reporting_period"] 30 | self.output_dir = self.config["output_dir"] 31 | 32 | def render_statistics(self): 33 | self.logger.debug("Started {}".format(inspect.currentframe().f_code.co_name)) 34 | 35 | statistics_json = {"misp_server": self.report_misp_server, 36 | "report_date": self.report_date, 37 | "statistics": f"{self.statistics}", 38 | "today_statistics": f"{self.today_statistics}", 39 | "today_statistics_attributes": f"{self.statistics_attributes}", 40 | } 41 | 42 | try: 43 | json_file = "{}/statistics.json".format(self.config["output_dir"]) 44 | with open(json_file, "w") as out_file: 45 | json.dump(statistics_json, out_file, indent=4) 46 | except Exception as e: 47 | self.logger.error("Error writing statistics json file: {}".format(e)) 48 | self.logger.info("Render statistics") 49 | -------------------------------------------------------------------------------- /config.py.default: -------------------------------------------------------------------------------- 1 | config = { 2 | "logfile": "/var/www/MISP/app/tmp/logs/misp-reporting.log", 3 | "logname": "misp-reporting", 4 | "log_incomplete": False, 5 | 6 | "reporting_period": "30d", 7 | "reporting_filter": None, 8 | "reporting_eventdetails_onlykeyorgs": True, 9 | "reporting_trending_count": 6, 10 | "reporting_filter_attribute_type_ids": True, 11 | "reporting_filter_timestamp": "published", 12 | "reporting_filter_published": True, 13 | 14 | "print_event_details": False, 15 | 16 | "misp_url": "", 17 | "misp_verifycert": False, 18 | "misp_key": "", 19 | "misp_page_size": 200, 20 | "cve_url": "https://cvepremium.circl.lu/api/cve/", 21 | "vulnerability_lookup_url": "https://vulnerability.circl.lu/vuln/", 22 | 23 | "reporting_cve_highlight": 7, 24 | 25 | "filter_sector": "misp-galaxy:sector", 26 | "filter_sector_count": 10, 27 | "filter_geo": "misp-galaxy:target-information", 28 | "filter_geo_count": 10, 29 | "filter_ttp_actors": ["misp-galaxy:threat-actor", "misp-galaxy:mitre-intrusion-set"], 30 | "filter_ttp_actors_count": 15, 31 | "filter_ttp_pattern": ["misp-galaxy:mitre-attack-pattern"], 32 | "filter_ttp_pattern_count": 15, 33 | 34 | "attribute_summary": {"network": ["ip-src", "ip-dst", "ip-src|port", "ip-dst|port"], 35 | "hashes": ["sha1", "sha256", "md5", "filename|sha256", "filename|sha1", "filename|sha256", "filename|md5", "filename|sha256"], 36 | "domain": ["hostname", "domain", "domain|ip"], 37 | "url": ["url"], 38 | "email": ["email", "email-src", "email-dst"], 39 | "vulnerability": ["vulnerability", "cpe", "weakness"]}, 40 | "attribute_other": "Other", 41 | 42 | "key_organisations": {"5cf66e53-b5f8-43e7-be9a-49880a3b4631": {"logo": "5cf66e53-b5f8-43e7-be9a-49880a3b4631.png"}, 43 | "55f6ea5e-2c60-40e5-964f-47a8950d210f": {"logo": "55f6ea5e-2c60-40e5-964f-47a8950d210f.png"}, 44 | "56a64d7a-63dc-4471-bce9-4accc25ed029": {"logo": "56a64d7a-63dc-4471-bce9-4accc25ed029.png"}, 45 | "56e02aba-9dd0-49ea-bd95-026e55fec118": {"logo": "56e02aba-9dd0-49ea-bd95-026e55fec118.png"}, 46 | "5677fc72-f46c-49f3-b5f3-7245ff32448e": {"logo": "5677fc72-f46c-49f3-b5f3-7245ff32448e.png"}}, 47 | 48 | "output_assets": "assets", 49 | "output_dir": "/var/www/MISP/app/webroot/misp-reporting", 50 | "install_dir": "/var/www/MISP/misp-custom/misp-reporting", 51 | "template_css": "templates/style.css", 52 | "template_html": "templates/template.html", 53 | "template_curation_html": "templates/template_curation.html", 54 | "template_infrastructure_html": "templates/template_infrastructure.html", 55 | "noimage_path": "no-image-32.png", 56 | "logo": "assets/cudeso.png", 57 | 58 | "distribution": ["Your organisation", "This community", "Connected communities", "All communities", "Sharing group"], 59 | "analysis": ["Initial", "Ongoing", "Completed"], 60 | "threatlevel_key_mapping": {'1': 'High', '2': 'Medium', '3': 'Low', '4': 'Undefined'}, 61 | "tlp_ignore_graph": ['tlp:unclear', 'tlp:ex:chr', 'tlp:amber+strict'], 62 | "workflow_complete": "workflow:state=\"complete\"", 63 | "workflow_incomplete": "workflow:state=\"incomplete\"", 64 | 65 | } 66 | -------------------------------------------------------------------------------- /docs/misp-reporting-curation-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-curation-1.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-curation-2.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-curation-3.png -------------------------------------------------------------------------------- /docs/misp-reporting-curation-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-curation-4.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-1.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-2.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-3.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-4.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-5.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-6.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-7.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-8.png -------------------------------------------------------------------------------- /docs/misp-reporting-summary-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cudeso/misp-reporting/6122a5b86e9ed8c6663886b5e3882285dd21b9af/docs/misp-reporting-summary-9.png -------------------------------------------------------------------------------- /reporting.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | import sys 4 | import json 5 | import requests 6 | from datetime import datetime, timedelta 7 | import logging 8 | from pymisp import * 9 | import inspect 10 | import os 11 | import matplotlib.pyplot as plt 12 | from jinja2 import Template 13 | import plotly.express as px 14 | import matplotlib.cm as cm 15 | import numpy as np 16 | from config import config 17 | from clsReporting import * 18 | from clsReportingData import * 19 | from clsReportingStatistics import * 20 | 21 | 22 | def main(config): 23 | logger.info("Start {}".format(config["logname"])) 24 | 25 | data = ReportingData(config, logger) 26 | data.get_statistics() 27 | data.get_today_events_attributes() 28 | data.get_trending_events_attributes() 29 | data.get_statistics_attributes() 30 | data.get_statistics_keyorgs() 31 | data.get_threatlevel() 32 | data.get_tlplevel() 33 | #data.get_eventdetails() 34 | data.get_target_sector() 35 | data.get_target_geo() 36 | data.get_ttp() 37 | data.get_vulnerabilities() 38 | data.get_curation() 39 | data.get_infrastructure() 40 | data.get_misp_statistics() 41 | 42 | reporting = Reporting(config, logger, data.data) 43 | reporting.write_index() 44 | reporting.render_report() 45 | reporting.render_curation_report() 46 | reporting.render_infrastructure() 47 | 48 | statistics = ReportingStatistics(config, logger, data.statistics, data.today_statistics, data.statistics_attributes) 49 | statistics.render_statistics() 50 | 51 | logger.info("End ".format(config["logname"])) 52 | 53 | 54 | if __name__ == '__main__': 55 | # export PYTHONIOENCODING='utf8' 56 | logger = logging.getLogger(config["logname"]) 57 | logger.setLevel(logging.DEBUG) 58 | ch = logging.FileHandler(config["logfile"], mode='a') 59 | ch.setLevel(logging.DEBUG) 60 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') 61 | ch.setFormatter(formatter) 62 | logger.addHandler(ch) 63 | 64 | main(config) 65 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | certifi==2024.7.4 2 | charset-normalizer==3.3.2 3 | contourpy==1.1.1 4 | cycler==0.12.1 5 | Deprecated==1.2.14 6 | fonttools==4.55.3 7 | idna==3.7 8 | importlib-resources==6.4.5 9 | jinja2==3.1.4 10 | kaleido==0.2.1 11 | kiwisolver==1.4.7 12 | MarkupSafe==2.1.5 13 | matplotlib==3.7.5 14 | numpy==1.24.4 15 | packaging==24.2 16 | pandas==2.0.3 17 | pillow==10.4.0 18 | plotly==5.24.1 19 | prettyprint==0.1.5 20 | publicsuffixlist==1.0.1.20240702 21 | pymisp==2.4.194 22 | pyparsing==3.1.4 23 | python-dateutil==2.9.0.post0 24 | pytz==2024.2 25 | pyzmq==26.0.3 26 | requests==2.32.3 27 | rpds-py==0.19.0 28 | six==1.16.0 29 | tenacity==9.0.0 30 | tzdata==2024.2 31 | urllib3==2.2.2 32 | wrapt==1.16.0 33 | zipp==3.20.2 34 | zmq==0.0.0 -------------------------------------------------------------------------------- /templates/style-blue.css: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: 'Roboto', Arial, sans-serif; 3 | font-size: 14px; 4 | margin: 0; 5 | padding: 0; 6 | background-color: #f2f2f2; 7 | color: #333; 8 | } 9 | header { 10 | background-color: #003366;/* #FF6600;*/ 11 | color: white; 12 | padding: 10px 20px; 13 | } 14 | header h1 { 15 | margin: 0; 16 | font-size: 1.5rem; 17 | } 18 | header .date { 19 | font-size: 0.8rem; 20 | } 21 | .logo-container { 22 | text-align: center; 23 | margin-top: 10px; 24 | } 25 | .logo-container img { 26 | max-height: 50px; 27 | } 28 | .container { 29 | max-width: 900px; 30 | margin: 20px auto; 31 | background: white; 32 | padding: 20px; 33 | border-radius: 5px; 34 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 35 | } 36 | .button-bar { 37 | display: flex; 38 | justify-content: flex-start; 39 | gap: 10px; 40 | margin-bottom: 20px; 41 | } 42 | .button-bar a { 43 | text-decoration: none; 44 | padding: 10px 15px; 45 | border-radius: 5px; 46 | font-weight: bold; 47 | transition: background-color 0.3s, box-shadow 0.3s; 48 | color: white; 49 | } 50 | .button-bar a.summary { 51 | background-color: #003366; /* #FF6600;*/ 52 | } 53 | .button-bar a.summary:hover { 54 | background-color: #003366; /* #e65c00;*/ 55 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 56 | } 57 | .button-bar a.curation { 58 | background-color: #004080; /* #FF8800;*/ 59 | } 60 | .button-bar a.curation:hover { 61 | background-color: #003366; /* #e67a00;*/ 62 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 63 | } 64 | .button-bar a.infrastructure { 65 | background-color: #00509e; /*#FF9900;*/ 66 | } 67 | .button-bar a.infrastructure:hover { 68 | background-color: #004080; /* #e68a00;*/ 69 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 70 | } 71 | .button-bar a.login { 72 | margin-left: auto; 73 | display: flex; 74 | background-color: #00509e; 75 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 76 | } 77 | .button-bar a.login:hover { 78 | background-color: #004080; 79 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 80 | } 81 | .charts { 82 | display: flex; 83 | justify-content: space-between; 84 | align-items: flex-start; 85 | gap: 10px; 86 | text-align: center; 87 | } 88 | .chart { 89 | flex: 1; 90 | } 91 | .chart.full-width { 92 | width: 100%; 93 | } 94 | .tables { 95 | display: flex; 96 | justify-content: space-between; 97 | gap: 10px; 98 | } 99 | .table { 100 | flex: 1; 101 | border-collapse: collapse; 102 | width: 100%; 103 | margin: 20px 0; 104 | } 105 | th, td { 106 | border: 1px solid #ddd; 107 | padding: 6px 8px; 108 | text-align: center; 109 | } 110 | th { 111 | background-color: #003366; /* #FF6600;*/ 112 | color: white; 113 | } 114 | td.align-right { 115 | text-align: right; 116 | } 117 | td.align-left { 118 | text-align: left; 119 | } 120 | td.align-topleft { 121 | text-align: left; 122 | vertical-align: top; 123 | } 124 | .table td a { 125 | color: #003366; /* #FF6600;*/ 126 | text-decoration: none; 127 | } 128 | 129 | .table td a:hover { 130 | color: #002d5c; /* #e65c00;*/ 131 | text-decoration: underline; 132 | } 133 | .separator { 134 | margin: 30px 0; 135 | border-top: 2px solid #FF6600; 136 | } 137 | footer { 138 | text-align: center; 139 | font-size: 12px; 140 | color: gray; 141 | margin-top: 20px; 142 | } 143 | 144 | h2 span.sub_h2 { 145 | font-size: 0.8rem; 146 | color: #898787; 147 | font-style: italic; 148 | font-weight: normal; 149 | } 150 | -------------------------------------------------------------------------------- /templates/style-red.css: -------------------------------------------------------------------------------- 1 | :root { 2 | --header-bg: #802c59; 3 | --header-text: #ffffff; 4 | --footer-bg: #511c39; 5 | --footer-text: #f0f0f0; 6 | --button-bg: #802c59; 7 | --button-hover: #c64012; 8 | --button-text: #ffffff; 9 | --body-bg: #f2f2f2; 10 | --container-bg: #ffffff; 11 | --text-color: #333333; 12 | --border-color: #dddddd; 13 | } 14 | 15 | body { 16 | font-family: 'Roboto', Arial, sans-serif; 17 | font-size: 14px; 18 | margin: 0; 19 | padding: 0; 20 | background-color: var(--body-bg); 21 | color: var(--text-color); 22 | } 23 | header { 24 | background-color: var(--header-bg); 25 | color: var(--header-text); 26 | padding: 10px 20px; 27 | } 28 | header h1 { 29 | margin: 0; 30 | font-size: 1.5rem; 31 | } 32 | header .date { 33 | font-size: 0.8rem; 34 | opacity: 0.8; 35 | } 36 | 37 | .logo-container { 38 | text-align: center; 39 | margin-top: 10px; 40 | } 41 | .logo-container img { 42 | max-height: 50px; 43 | } 44 | 45 | .container { 46 | max-width: 900px; 47 | margin: 20px auto; 48 | background: var(--container-bg); 49 | padding: 20px; 50 | border-radius: 5px; 51 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 52 | } 53 | 54 | .button-bar { 55 | display: flex; 56 | justify-content: flex-start; 57 | gap: 10px; 58 | margin-bottom: 20px; 59 | } 60 | .button-bar a { 61 | text-decoration: none; 62 | padding: 10px 15px; 63 | border-radius: 5px; 64 | font-weight: bold; 65 | transition: background-color 0.3s, box-shadow 0.3s; 66 | color: var(--button-text); 67 | background-color: var(--button-bg); 68 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 69 | } 70 | .button-bar a:hover { 71 | background-color: var(--button-hover); 72 | } 73 | 74 | .button-bar a.summary, 75 | .button-bar a.curation, 76 | .button-bar a.infrastructure, 77 | .button-bar a.login { 78 | } 79 | .button-bar a.login { 80 | margin-left: auto; 81 | display: flex; 82 | } 83 | 84 | .charts, 85 | .tables { 86 | display: flex; 87 | justify-content: space-between; 88 | gap: 10px; 89 | text-align: center; 90 | } 91 | 92 | .table { 93 | flex: 1; 94 | border-collapse: collapse; 95 | width: 100%; 96 | margin: 20px 0; 97 | background: var(--container-bg); 98 | border: 1px solid var(--border-color); 99 | } 100 | th, td { 101 | border: 1px solid var(--border-color); 102 | padding: 6px 8px; 103 | text-align: center; 104 | } 105 | th { 106 | background-color: #802c59; 107 | color: var(--button-text); 108 | } 109 | td.align-right { text-align: right; } 110 | td.align-left { text-align: left; } 111 | td.align-topleft { text-align: left; vertical-align: top; } 112 | .table td a { 113 | color: var(--button-bg); 114 | text-decoration: none; 115 | } 116 | .table td a:hover { 117 | color: var(--button-hover); 118 | text-decoration: underline; 119 | } 120 | .separator { 121 | margin: 30px 0; 122 | border-top: 2px solid var(--button-bg); 123 | } 124 | 125 | footer { 126 | background-color: var(--footer-bg); 127 | color: var(--footer-text); 128 | text-align: center; 129 | font-size: 12px; 130 | padding: 10px 0; 131 | } 132 | 133 | h2 span.sub_h2 { 134 | font-size: 0.8rem; 135 | color: #898787; 136 | font-style: italic; 137 | font-weight: normal; 138 | } 139 | -------------------------------------------------------------------------------- /templates/style.css: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: 'Roboto', Arial, sans-serif; 3 | font-size: 14px; 4 | margin: 0; 5 | padding: 0; 6 | background-color: #f2f2f2; 7 | color: #333; 8 | } 9 | header { 10 | background-color: #FF6600; 11 | color: white; 12 | padding: 10px 20px; 13 | } 14 | header h1 { 15 | margin: 0; 16 | font-size: 1.5rem; 17 | } 18 | header .date { 19 | font-size: 0.8rem; 20 | } 21 | .logo-container { 22 | text-align: center; 23 | margin-top: 10px; 24 | } 25 | .logo-container img { 26 | max-height: 50px; 27 | } 28 | .container { 29 | max-width: 900px; 30 | margin: 20px auto; 31 | background: white; 32 | padding: 20px; 33 | border-radius: 5px; 34 | box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1); 35 | } 36 | .button-bar { 37 | display: flex; 38 | justify-content: flex-start; 39 | gap: 10px; 40 | margin-bottom: 20px; 41 | } 42 | .button-bar a { 43 | text-decoration: none; 44 | padding: 10px 15px; 45 | border-radius: 5px; 46 | font-weight: bold; 47 | transition: background-color 0.3s, box-shadow 0.3s; 48 | color: white; 49 | } 50 | .button-bar a.summary { 51 | background-color: #FF6600; 52 | } 53 | .button-bar a.summary:hover { 54 | background-color: #e65c00; 55 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 56 | } 57 | .button-bar a.curation { 58 | background-color: #FF8800; 59 | } 60 | .button-bar a.curation:hover { 61 | background-color: #e67a00; 62 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 63 | } 64 | .button-bar a.infrastructure { 65 | background-color: #FF9900; 66 | } 67 | .button-bar a.infrastructure:hover { 68 | background-color: #e68a00; 69 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 70 | } 71 | .button-bar a.login { 72 | margin-left: auto; 73 | display: flex; 74 | background-color: #FF9900; 75 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 76 | } 77 | .button-bar a.login:hover { 78 | background-color: #e68a00; 79 | box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2); 80 | } 81 | .charts { 82 | display: flex; 83 | justify-content: space-between; 84 | align-items: flex-start; 85 | gap: 10px; 86 | text-align: center; 87 | } 88 | .chart { 89 | flex: 1; 90 | } 91 | .chart.full-width { 92 | width: 100%; 93 | } 94 | .tables { 95 | display: flex; 96 | justify-content: space-between; 97 | gap: 10px; 98 | } 99 | .table { 100 | flex: 1; 101 | border-collapse: collapse; 102 | width: 100%; 103 | margin: 20px 0; 104 | } 105 | th, td { 106 | border: 1px solid #ddd; 107 | padding: 6px 8px; 108 | text-align: center; 109 | } 110 | th { 111 | background-color: #FF6600; 112 | color: white; 113 | } 114 | td.align-right { 115 | text-align: right; 116 | } 117 | td.align-left { 118 | text-align: left; 119 | } 120 | td.align-topleft { 121 | text-align: left; 122 | vertical-align: top; 123 | } 124 | .table td a { 125 | color: #FF6600; 126 | text-decoration: none; 127 | } 128 | 129 | .table td a:hover { 130 | color: #e65c00; 131 | text-decoration: underline; 132 | } 133 | .separator { 134 | margin: 30px 0; 135 | border-top: 2px solid #FF6600; 136 | } 137 | footer { 138 | text-align: center; 139 | font-size: 12px; 140 | color: gray; 141 | margin-top: 20px; 142 | } 143 | 144 | h2 span.sub_h2 { 145 | font-size: 0.8rem; 146 | color: #898787; 147 | font-style: italic; 148 | font-weight: normal; 149 | } -------------------------------------------------------------------------------- /templates/template.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Infrastructure Monitoring 24 | 31 |
32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 |
Total eventsTotal attributesCorrelationsUsersOrganisationsLocal organisations
{{ summary.events }}{{ summary.attributes }}{{ summary.correlations }}{{ summary.users }}{{ summary.organisations }}{{ summary.local_organisations }}
55 | 56 |

Reporting period: Last {{ reporting_period }} {{ summary.period }}. Published at {{ report_timestamp_hm }}

57 |
58 | 59 |
60 |

Summary of {{reporting_filter_timestamp}} events

61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 |
Events (24h)Attributes (24h)Attr to_ids (24h)Events ({{ reporting_period }})Attributes ({{ reporting_period }})Attr to_ids ({{ reporting_period }})
{{ summary.today_events }}{{ summary.today_attributes }}{{ summary.today_attributes_ids }}{{ summary.period_events }}{{ summary.period_attributes }}{{ summary.period_attributes_ids }}
83 | 84 |

Trending events and attributes for {{reporting_filter_timestamp}} events

85 |
86 |
87 | Trending {{reporting_filter_timestamp}} events 88 |
89 |
90 | Trending attributes in {{reporting_filter_timestamp}} events 91 |
92 |
93 |
94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | {% for month, value in trending_events.items() %} 103 | 104 | 105 | 106 | 107 | {% endfor %} 108 | 109 |
PeriodEvents
{{ month }}{{ value }}
110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | {% for month, value in trending_attributes.items() %} 120 | 121 | 122 | 123 | 124 | 125 | {% endfor %} 126 | 127 |
PeriodAttributesAttr to_ids
{{ month }}{{ value[0] }}{{ value[1] }}
128 |
129 | 130 |

Attributes type distribution for {{reporting_filter_timestamp}} events {{ attributes_with_ids_or_not}}

131 |
132 |
133 | Attributes type distribution 134 |
135 |
136 | Attributes type distribution 137 |
138 |
139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | {% for attr_type, count in attributes_type.items() %} 149 | 150 | 151 | 152 | 153 | 154 | {% endfor %} 155 | 156 |
Attribute type24h{{ reporting_period }}
{{ attr_type }}{{ count[1] }}{{ count[0] }}
157 |

Activity of key organisations for {{reporting_filter_timestamp}} events

158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | {% for org_name, org_data in keyorgs.items() %} 173 | 174 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | {% endfor %} 188 | 189 |
 OrganisationEvents (24h)Attributes (24h)Attr to_ids (24h)Events ({{ reporting_period }})Attributes ({{ reporting_period }})Attr to_ids ({{ reporting_period }})
175 | {% if org_data.logo %} 176 | {{ org_name }} 177 | {% endif %} 178 | {{ org_name }}{{ org_data['today_events'] }}{{ org_data['today_attributes'] }}{{ org_data['today_attributes_ids'] }}{{ org_data['period_events'] }}{{ org_data['period_attributes'] }}{{ org_data['period_attributes_ids'] }}
190 |
191 | 192 |
193 |

Threat level and TLP of {{reporting_filter_timestamp}} events ({{ reporting_period }})

194 |
195 |
196 | Threat level 197 |
198 |
199 | TLP 200 |
201 |
202 |
203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | {% for level, count in threatlevel.items() %} 212 | 213 | 214 | 215 | 216 | {% endfor %} 217 | 218 |
LevelCount
{{ level }}{{ count }}
219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | {% for tlp, count in tlp.items() %} 228 | 229 | 230 | 231 | 232 | {% endfor %} 233 | 234 | 235 |
TLPCount
{{ tlp }}{{ count }}
(tlp:white is translated to tlp:clear)
236 |
237 | 238 | {% if print_event_details %} 239 |

Event details

240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | {% for event in detailed_events %} 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | {% endfor %} 270 | 271 |
IDDateOrganisationDistributionAnalysisThreat LevelPublishedTLPInfoIndicators
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.distribution }}{{ event.analysis }}{{ event.threat_level }}{{ event.published }}{{ event.tlp }}{{ event.info }}{{ event.indicators }}
272 | {% endif %} 273 |
274 | 275 |
276 |

Geo target information of {{reporting_filter_timestamp}} events ({{ reporting_period }})

277 |
278 | Geo targeting map 279 |
280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | {% for country, count in target_geo.items() %} 289 | 290 | 291 | 292 | 293 | {% endfor %} 294 | 295 |
CountryCount
{{ country }}{{ count }}
296 |

Sector targeting of {{reporting_filter_timestamp}} events ({{ reporting_period }})

297 |
298 | Sector targeting 299 |
300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | {% for sector, count in target_sector.items() %} 309 | 310 | 311 | 312 | 313 | {% endfor %} 314 | 315 |
SectorCount
{{ sector }}{{ count }}
316 |
317 | 318 |
319 |

TTPs of {{reporting_filter_timestamp}} events ({{ reporting_period }})

320 |
321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | {% for actor, count in ttp_actors.items() %} 330 | 331 | 332 | 333 | 334 | {% endfor %} 335 | 336 |
ActorsCount
{{ actor }}{{ count }}
337 |
338 |
339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | {% for pattern, count in ttp_pattern.items() %} 348 | 349 | 350 | 351 | 352 | {% endfor %} 353 | 354 |
PatternsCount
{{ pattern }}{{ count }}
355 |
356 |
357 | 358 |
359 |

Vulnerabilities and weaknesses found in {{reporting_filter_timestamp}} events ({{ reporting_period }})

360 |
361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | {% for cve, detail in vulnerabilities.items() %} 373 | 374 | 375 | 376 | 377 | 382 | 383 | 384 | {% endfor %} 385 | 386 |
VulnerabilityEventsSummaryCVSSv3Count
{{ cve }}MISP{{ detail.summary }}{{ detail.cvss3 }} 378 | {% if detail.cvss3|float > cve_highlight|float %} 379 | ⚠ 380 | {% endif %} 381 | {{ detail.count }}
387 |
388 |
389 | 390 | 394 | 395 | 396 | -------------------------------------------------------------------------------- /templates/template_curation.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Infrastructure Monitoring 24 | 31 |
32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 45 | 47 | 49 | 50 | 51 |
Not curated (24h)Not curated with threat level high (24h)Not curated from completely reliable sources (24h)
{{ curation_incomplete_today_count }} 44 |
Events		{{ curation_incomplete_high_count }} 46 |
Events		{{ curation_incomplete_adm_high_count}} 48 |
Events
52 | 53 |

Reporting period: Last {{ reporting_period }} {{ summary.period }}. Published at {{ report_timestamp_hm }}

54 |
55 | 56 |
57 |

Curation summary

58 |

Curated and not curated

59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 |
PeriodCuratedNot curated
Last 24h{{ curation_complete_today_count }}{{ curation_incomplete_today_count }}
Last {{ reporting_period }}{{ curation_complete_count }}{{ curation_incomplete_count }}
80 | 81 |

Event dates (event dates indicate when a threat occurred and are set by the event creator)

82 |
83 |
84 | Bubble graph of curation results 85 |
86 |
87 | 88 |

Curation for last {{ reporting_period }}

89 |
90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | {% for org, value in curation_complete_org.items() %} 99 | 100 | 101 | 102 | 103 | {% endfor %} 104 | 105 |
OrganisationCurated events
{{ org }}{{ value }}
106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | {% for org, value in curation_incomplete_org.items() %} 115 | 116 | 117 | 118 | 119 | {% endfor %} 120 | 121 |
OrganisationNot curated events
{{ org }}{{ value }}
122 |
123 |
124 | 125 |
126 |

Events waiting for curation - Threat level high

127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | {% for event in curation_incomplete_high %} 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | {% endfor %} 147 | 148 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
149 | 150 |

Events waiting for curation - Admiralty scale : Source Reliability: Completely reliable (A)

151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | {% for event in curation_incomplete_adm_high %} 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | {% endfor %} 171 | 172 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
173 | 174 |

Events waiting for curation

175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | {% for event in curation_incomplete %} 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | {% endfor %} 195 | 196 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
197 | 198 |

Curated events

199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | {% for event in curation_complete %} 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | {% endfor %} 219 | 220 |
IDDateOrganisationEvent titleAttributes
{{ event.id }}{{ event.date }}{{ event.org }}{{ event.info }}{{ event.indicators }}
221 | 222 |
223 | 224 | 228 | 229 | 230 | -------------------------------------------------------------------------------- /templates/template_infrastructure.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ title }} 7 | 8 | 9 | 10 | 11 | 12 |
13 |

{{ title }}

14 |
{{ report_date }}
15 |
16 |
17 | Logo 18 |
19 |
20 |
21 | Summary 22 | Curation Overview 23 | Infrastructure Monitoring 24 | 31 |
32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 |
Remote serversFeedsWorkers
---
49 | 50 |

Published at {{ report_timestamp_hm }}.

51 |

Development in progress

52 |
53 | 54 |
55 |

Remote servers

56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 |
ServerUserStatusEvents
----
74 | 75 | 79 | 80 | 81 | --------------------------------------------------------------------------------