Access denied
8 |Your IP address has been permanently blocked due to excessive amount of queries or another abuse of the service. If you think this is an error or you need a special access to the service for legitimate reasons, contact admins at "nerd@cesnet.cz"
9 | 10 | 11 | -------------------------------------------------------------------------------- /scripts/get_ttl_token_stats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Get statistics on number of IP records by TTL tokens 3 | # grep is used because mongosh puts an empty line at the end that we don't want there 4 | mongosh nerd --quiet --eval 'db.ip.aggregate([{$project: {ttl: {$objectToArray: "$_ttl"}}}, {$unwind: "$ttl"}, {$group: {"_id": "$ttl.k", cnt: {$sum: 1}}}, {$sort: {cnt: -1}}]).forEach( function(x) { print(x["_id"] + "\t" + x["cnt"]); })' | grep -v "^$" 5 | -------------------------------------------------------------------------------- /install/install_configure_bind.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Install and configure BIND (supports DNS queries made by various modules) 3 | 4 | BASEDIR=$(dirname $0) 5 | . $BASEDIR/common.sh 6 | 7 | echob "=============== Install & Configure BIND ===============" 8 | 9 | echob "** Installing BIND **" 10 | yum install -y -q bind bind-utils 11 | 12 | echob "** Starting BIND **" 13 | systemctl enable named.service 14 | systemctl restart named.service 15 | -------------------------------------------------------------------------------- /NERDweb/wsgi-debug.py: -------------------------------------------------------------------------------- 1 | from nerd_main import app 2 | 3 | # Enable logging of exceptions into Apache log file 4 | app.debug = True 5 | 6 | # Enable interactive debuggin in web browser. 7 | # ! Don't use this on production server, it allows to run arbitarary code ! 8 | from werkzeug.debug import DebuggedApplication 9 | application = DebuggedApplication(app, True) 10 | 11 | # Enable testing mode of NERDweb. 12 | # This allows anyone to log in as admininstrator ("devel autologin")! 13 | from nerd_main import config 14 | config.testing = True 15 | -------------------------------------------------------------------------------- /scripts/set_api_token.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Set API token to given user (in 'users' table in PSQL DB 'nerd'). 3 | # If no token is specified, a random string of 10 alphanumeric characters is used. 4 | 5 | if [ -z "$1" ]; then 6 | echo -e "Usage:\n $0 user_name [token]" >&2 7 | exit 1 8 | fi 9 | user=$1 10 | if [ -z "$2" ]; then 11 | token="$(Feed - {{name}} 5 |6 | Description: 7 |
8 | {{description}} 9 |
10 |
11 | 12 |
-
13 |
- Type of feed:
- {{feed_type}} 14 |
- Links 15 | {% if firehol_link %} 16 |
- FireHOL link: {{ firehol_link }} 17 | {% endif %} 18 | {% if provider_link %} 19 |
- Provider link: {{ provider_link }} 20 | {% endif %} 21 | {% if url %} 22 |
- Feed link: {{ url }} 23 | {% endif %} 24 |
Ports:{% for val in data.ports|sort %}{{ val }}{%- endfor -%}
3 | {% if data.os -%} 4 |OS: {{ data.os }}
5 | {% endif -%} 6 | {% if data.tags -%} 7 |Tags: {% for val in data.tags|sort %}{{ val }}{%- endfor -%}
8 | {% endif -%} 9 | 10 | {%- elif "No information available" in data.error -%} 11 |-- no information available --
12 | {%- elif data.error == "timeout" -%}
13 | Error: RPC timeout
14 | {%- else -%}
15 | Error while requesting shodan api: {{ data.error }}
16 | {%- endif %}
--------------------------------------------------------------------------------
/install/munin/nerd_tasks_by_src:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | sources="blacklists
4 | misp_receiver
5 | otx_receiver
6 | updater
7 | warden_receiver
8 | updater_manager
9 | web
10 | misp_updater
11 | "
12 |
13 | if [[ "$1" == "config" ]]; then
14 | cat <<'END'
15 | graph_title Tasks processed by source
16 | graph_category nerd
17 | graph_vlabel Number of tasks per 5 min
18 | END
19 | for each_source in ${sources}; do
20 | echo "${each_source}.label ${each_source}"
21 | echo "${each_source}.draw AREASTACK"
22 | echo "${each_source}.min 0"
23 | done
24 | exit 0
25 | fi
26 |
27 | ecl_reader /etc/nerd/event_logging.yml -g tasks_by_src -i 5m | sed -E 's/:([0-9]+)$/.value \1/'
--------------------------------------------------------------------------------
/NERDweb/robots.txt:
--------------------------------------------------------------------------------
1 | User-Agent: *
2 | Disallow: /nerd/login/
3 | Disallow: /nerd/_ips_count
4 | Disallow: /nerd/api/
5 | Disallow: /nerd/ajax/
6 | Disallow: /nerd/pdns/
7 | Disallow: /nerd/misp_event/
8 | Disallow: /Shibboleth.sso
9 |
10 | # Disallow some bots that generate a lot of traffic and we think it is not useful
11 |
12 | # OpenAI bot crawling training data - it doesn't make sense to use NERDs dynamic data for training (bots for search and user actions are still allowed)
13 | User-Agent: GPTBot
14 | Disallow: /nerd/
15 | Disallow: /Shibboleth.sso
16 |
17 | # Moz.com crawler for their backlink checker - lot of traffic and useless to search backlinks in NERD data
18 | User-Agent: DotBot
19 | Disallow: /nerd/
20 | Disallow: /Shibboleth.sso
21 |
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # NERD - Network Entity Reputation Database
2 |
3 | NERD is a software and a service which acquires, stores and aggregates various data about known malicious network entities (mostly IP addresses) and provides them in a comprehensible way to users.
4 |
5 | The main NERD instance runs at [nerd.cesnet.cz](https://nerd.cesnet.cz/).
6 |
7 | See the [project wiki](https://github.com/CESNET/NERD/wiki) for more information.
8 |
9 | ---
10 |
11 | _This software was developed within the scope of the Security Research
12 | Programme of the Czech Republic 2015 - 2020 (BV III / 1 VS) granted by
13 | the Ministry of the Interior of the Czech Republic under the project No.
14 | VI20162019029 The Sharing and analysis of security events in the Czech
15 | Republic._
16 |
--------------------------------------------------------------------------------
/install/munin/nerd_shodan:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | if [[ "$1" == "config" ]]; then
4 | cat <Account information
5 | 6 | 7 |Name: {{ user.get('name', '[unknown]') }}
8 |User ID: {{ user.id }}
9 |Login type: {{ user.login_type }}
10 |Groups: {{ user.groups|join(', ') }}
11 | 12 |13 |
API token: {{ token.value }}
14 | 18 | 19 | {% if passwd_form %} 20 |21 |
Change password:
22 | 29 | {% endif %} 30 | {% endblock %} 31 | -------------------------------------------------------------------------------- /NERDweb/templates/ipblock.html: -------------------------------------------------------------------------------- 1 | {% extends "layout.html" %} 2 | {% block body %} 3 | 4 |IP block
5 | 6 | {% if not ipblock or not ac('ipsearch') %} 7 | {# Print nothing if no IP-block was passed or insufficient permissions #} 8 | {% else %} 9 | 10 |11 | 12 |
13 | {{ attr }} {{ val }}
38 | {% else %}
39 | {{ attr }} {{ val }}
40 | {% endif %}
41 | {% endfor %}
42 |
43 |
44 | {% endif %}{# if not found #}
45 | {% endif %}{# if nothing specified #}
46 |
47 | {% endblock %}
48 |
--------------------------------------------------------------------------------
/install/configure_supervisor.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | # Optional parameter "open" opens http port on all interfaces (default: localhost only)
3 | # Use only in development environment (Vagrant), never in production!
4 |
5 | BASEDIR=$(dirname $0)
6 | . $BASEDIR/common.sh
7 |
8 | echob "=============== Configure Supervisor ==============="
9 |
10 | open_port=0
11 | if [ "$1" = "open" ]; then
12 | open_port=1
13 | fi
14 |
15 | echob "** Copying supervisor config files **"
16 |
17 | # Copy main configuration file
18 | cp $BASEDIR/supervisord.conf /etc/nerd/supervisord.conf
19 | if [ $open_port = 1 ]; then
20 | # replace "localhost:9001" (which should be in the config file) by "*:9001"
21 | echor "WARNING: SUPERVISORD HTTP PORT IS OPEN ON ALL INTERFACES!"
22 | sed -i "s/^port=[^:]*/port=*/" /etc/nerd/supervisord.conf
23 | fi
24 |
25 | # Copy files specifying individual NERD components running under Supervisor
26 | mkdir -p /etc/nerd/supervisord.conf.d/
27 | cp $BASEDIR/supervisord.conf.d/* /etc/nerd/supervisord.conf.d/
28 |
29 | chown -R nerd:nerd /etc/nerd/supervisord.conf
30 | chown -R nerd:nerd /etc/nerd/supervisord.conf.d/
31 |
32 | echob "** Create nerdctl script **"
33 | echob "'nerdctl' is an alias for 'supervisorctl' with NERD configuration file"
34 |
35 | echo '#!/bin/sh
36 | supervisorctl -c /etc/nerd/supervisord.conf $@' >/usr/bin/nerdctl
37 | chmod +x /usr/bin/nerdctl
38 |
39 | echob "** Set up supervisord systemd unit **"
40 |
41 | cp $BASEDIR/nerd-supervisor.service /etc/systemd/system/nerd-supervisor.service
42 | systemctl daemon-reload
43 | #systemctl enable nerd-supervisor
44 | #systemctl restart nerd-supervisor
45 |
46 | echoy "** TO RUN NERD, START ITS SUPERVISOR: systemctl start nerd-supervisor"
47 |
--------------------------------------------------------------------------------
/NERDweb/templates/org.html:
--------------------------------------------------------------------------------
1 | {% extends "layout.html" %}
2 | {% block body %}
3 |
4 | 14 | {% if rec %}{{ "%.3f"|format(rec.rep)|replace("0.",".") if rec.rep is defined else "--" }}{% endif %} 15 | {{ ipblock }}{{ rec.name if rec.name else '' }} 16 |
17 | 18 | {% if not rec %} 19 |Record not found in database
20 | {% else %} 21 | 22 |-
23 | {% for attr,val in rec|dictsort %}
24 | {% if attr.startswith("_") and not ac('internal_attrs') %}
25 | {# pass (hide attrs starting with '_' from normal users) #}
26 | {% elif attr == "org" %}
27 |
- Organization 28 |
- {{ val }} 29 | {% elif attr == "ips" %} 30 |
- IPs ({{val|length}}) 31 |
- {{ ip }}
34 | {% endfor %} 35 |
Organization
5 | 6 | {% if not org or not ac('ipsearch') %} 7 | {# Print nothing if no org was passed or insufficient permissions #} 8 | {% else %} 9 | 10 |11 | 12 |
13 | IP Blocks ({{val|length}})
33 | 5%} class="scrollable"{% endif %}>
34 | {% for ipb in val|sort %}
35 | {{ ipb }}
36 | {% endfor %}
37 | {{ attr }} {{ val|replace("\n","
"|safe) }}
40 | {% elif val|is_date %}
41 | {{ attr }} {{ val }}
42 | {% else %}
43 | {{ attr }} {{ val }}
44 | {% endif %}
45 | {% endfor %}
46 |
47 |
48 | {% endif %}{# if not found #}
49 | {% endif %}{# if nothing specified #}
50 |
51 | {% endblock %}
52 |
--------------------------------------------------------------------------------
/NERDweb/templates/bgppref.html:
--------------------------------------------------------------------------------
1 | {% extends "layout.html" %}
2 | {% block body %}
3 |
4 | {{ rec.name if rec.name else org }}{{ '('+org+')' if rec.name else '' }}
14 | 15 | {% if not rec %} 16 |Record not found in database
17 | {% else %} 18 | 19 | 20 |-
21 | {% for attr,val in rec|dictsort %}
22 | {% if attr.startswith("_") and not ac('internal_attrs') %}
23 | {# pass (hide attrs starting with '_' from normal users) #}
24 | {% elif attr == "asns" %}
25 |
- Autonomous Systems ({{val|length}}) 26 |
- {{ 'AS' + asn|string }}
29 | {% endfor %} 30 |
"|safe) }}
BGP prefix
5 | 6 | {% if not bgppref or not ac('ipsearch') %} 7 | {# Print nothing if no prefix was passed or insufficient permissions #} 8 | {% else %} 9 | 10 |11 | 12 |
13 | IPs ({{val|length}})
35 | 5%} class="scrollable"{% endif %}>
36 | {% for ip in val|sort %}
37 | {{ ip }}
38 | {% endfor %}
39 | {{ attr }} {{ val }}
42 | {% else %}
43 | {{ attr }} {{ val }}
44 | {% endif %}
45 | {% endfor %}
46 |
47 |
48 | {% endif %}{# if not found #}
49 | {% endif %}{# if nothing specified #}
50 |
51 | {% endblock %}
52 |
--------------------------------------------------------------------------------
/scripts/add_user.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | if [[ "$#" -ne 5 ]]; then
4 | echo "Create a local user in NERD user database."
5 | echo "Usage:"
6 | echo " $0 username list,of,groups name email org"
7 | echo
8 | echo "Username should be prefixed with 'local:' or 'shibboleth:'"
9 | echo "Groups are plain strings separated by commas, e.g.: registered,trusted"
10 | exit 1
11 | fi
12 |
13 | die () { echo "Exiting due to error"; exit 2; }
14 |
15 | username=$1
16 | groups=$2
17 | name=$3
18 | email=$4
19 | org=$5
20 |
21 | groups_for_sql="{\"$(sed 's/, */","/g' <<<"$groups")\"}"
22 |
23 | echo "Adding a new user with the following parameters:"
24 | echo " Username: $username"
25 | echo " Groups: $groups_for_sql"
26 | echo " Full name: $name"
27 | echo " Email: $email"
28 | echo " Organization: $org"
29 | echo
30 | if [[ $username =~ ^local: ]]; then
31 | read -p "Enter password for the new user (or leave empty to generate a random one): " pass
32 | echo
33 | if [[ "$pass" == "" ]]; then
34 | pass="$(/dev/null; then
6 | echo "(Re)configure RabbitMQ exchanges and queues for NERD workers." >&2
7 | echo "Number of workers must be a non-negative integer." >&2
8 | echo "Zero means to remove all NERD exchanges and queues." >&2
9 | echo >&2
10 | echo "Usage: $0 number_of_workers" >&2
11 | exit 1
12 | fi
13 |
14 | N=$1
15 |
16 | echo "** Removing all NERD exchanges and queues **"
17 |
18 | exchange_list=$(rabbitmqadmin list exchanges name -f tsv | grep "^nerd-.*-task-exchange")
19 | for q in $exchange_list
20 | do
21 | rabbitmqadmin delete exchange name=$q
22 | done
23 |
24 | queue_list=$(rabbitmqadmin list queues name -f tsv | grep "^nerd-worker-")
25 | for q in $queue_list
26 | do
27 | rabbitmqadmin delete queue name=$q
28 | done
29 |
30 | if [[ "$N" -eq 0 ]]; then
31 | exit 0
32 | fi
33 |
34 | echo "** Setting up exchanges and queues for $N workers **"
35 |
36 | # Declare exchanges (normal and priority)
37 | rabbitmqadmin declare exchange name=nerd-main-task-exchange type=direct durable=true
38 | rabbitmqadmin declare exchange name=nerd-priority-task-exchange type=direct durable=true
39 |
40 | # Declare queues for N workers
41 | for i in $(seq 0 $(($N-1)))
42 | do
43 | rabbitmqadmin declare queue name=nerd-worker-$i durable=true 'arguments={"x-max-length": 100, "x-overflow": "reject-publish"}'
44 | rabbitmqadmin declare queue name=nerd-worker-$i-pri durable=true
45 | done
46 |
47 | # Bind queues to exchanges
48 | for i in $(seq 0 $(($N-1)))
49 | do
50 | rabbitmqadmin declare binding source=nerd-main-task-exchange destination=nerd-worker-$i routing_key=$i
51 | rabbitmqadmin declare binding source=nerd-priority-task-exchange destination=nerd-worker-$i-pri routing_key=$i
52 | done
53 |
--------------------------------------------------------------------------------
/install/install_configure_munin.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | # Install and configure Munin to monitor the host and some metrics of NERD processing.
3 | #
4 | # Note:
5 | # - munin is the master, which queries status of (potentially multiple) hosts
6 | # - munin-node runs on the host and provides provides information to the maser
7 | # In this case, the same host runs both munin (master) and munin-node
8 |
9 | BASEDIR=$(dirname $0)
10 | . $BASEDIR/common.sh
11 |
12 | echob "=============== Install & configure Munin ==============="
13 |
14 | yum -y -q install munin munin-node
15 |
16 | # ** Configure munin-node **
17 | # Allow connections from localhost only
18 | sed -i -e 's/^host \*$/# host \*/' -e 's/^#\s*host 127.0.0.1/host 127.0.0.1/' /etc/munin/munin-node.conf
19 |
20 | # Install and enable NERD plugins
21 | cp $BASEDIR/munin/* /usr/share/munin/plugins/
22 | chmod +x /usr/share/munin/plugins/nerd_*
23 | ln -s /usr/share/munin/plugins/nerd_* /etc/munin/plugins/
24 | # except nerd_mongo_rs, since Mongo Replica-set is not configured by default - remove the symlink to disable plugin
25 | rm /etc/munin/plugins/nerd_mongo_rs
26 |
27 | # Enable Apache and named (BIND) plugins
28 | ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/
29 | ln -s /usr/share/munin/plugins/named /etc/munin/plugins/
30 |
31 | # Enable & run munin-node
32 | systemctl enable munin-node
33 | systemctl restart munin-node
34 |
35 | # Enable & run munin (a script has to be run periodically, older versions were run by cron, now it's done using systemd timer)
36 | systemctl enable munin.timer
37 | systemctl start munin.timer
38 |
39 | # ** Enable web access **
40 | # Copy prepared config file for Apache
41 | cp $BASEDIR/httpd/munin.conf /etc/httpd/conf.d/munin.conf
42 |
43 | touch /etc/munin/munin-htpasswd
44 |
45 | systemctl reload httpd
46 |
47 | echoy
48 | echoy "INFO: Munin is available at http://14 | {% if rec %}{{ "%.6f"|format(rec.rep)|replace("0.",".") if rec.rep is defined else "--" }}{% endif %} 15 | {{ bgppref }} 16 |
17 | 18 | {% if not rec %} 19 |Record not found in database
20 | {% else %} 21 | 22 |-
23 | {% for attr,val in rec|dictsort %}
24 | {% if attr.startswith("_") and not ac('internal_attrs') %}
25 | {# pass (hide attrs starting with '_' from normal users) #}
26 | {% elif attr == "asn" %}
27 |
- Origin ASN(s) ({{val|length}}) 28 |
- {{ 'AS' + asn|string }}
31 | {% endfor %} 32 |
Autonomous system number (ASN)
5 | 6 | {# Note: The onsubmit script overrides funcionality of Submit button to put IP address to custom URL #} 7 | 12 | 13 | {% if not asn or not ac('assearch') %} 14 | {# Print nothing if no ASN was passed #} 15 | {% else %} 16 | 17 |18 | 19 |
20 | Organization
45 | {{ val }}
46 | {% elif val|is_date %}
47 | {{ attr }} {{ val }}
48 | {% else %}
49 | {{ attr }} {{ val }}
50 | {% endif %}
51 | {% endfor %}
52 |
53 |
54 |
55 | {% endif %}{# if not found #}
56 | {% endif %}{# if nothing specified #}
57 |
58 | {% endblock %}
59 |
--------------------------------------------------------------------------------
/client-scripts/nerd.nse:
--------------------------------------------------------------------------------
1 | local http = require "http"
2 | local ipOps = require "ipOps"
3 | local io = require "io"
4 | local stdnse = require "stdnse"
5 | local json = require "json"
6 |
7 |
8 | description = [[
9 | Looks up info about the target in the NERD system.
10 | ]]
11 |
12 | ---
13 | -- @args nerd Takes the following optional argument:
14 | -- * AS{{ asn }} {{ rec.name }}
21 | 22 | {% if not rec %} 23 |Record not found in database
24 | {% else %} 25 | 26 |-
27 | {% for attr,val in rec|dictsort %}
28 | {% if attr.startswith("_") and not ac('internal_attrs') %}
29 | {# pass (hide attrs starting with '_' from normal users) #}
30 | {% elif attr == "ctry" %}
31 |
- Country
- 32 | 33 | {{ ctrydata.names.get(val, '?') }} 34 | {% elif attr == "rir" %} 35 |
- RIR
- {{ val|upper }} 36 | {% elif attr == "bgppref" %} 37 |
- BGP Prefixes ({{val|length}}) 38 |
- {{ pref }}
41 | {% endfor %} 42 |
nerd.apifile=file Path to file with NERD API key, default is ./nerdapifile.
15 | -- @usage
16 | -- # Basic usage:
17 | -- nmap target --script nerd
18 | -- nmap target --script nerd --script-args nerd.apifile=/home/user/apifile
19 | --
20 | -- @output
21 | -- Host script results:
22 | -- |_nerd: IP not found in NERD
23 | --
24 | -- Host script results:
25 | -- |_nerd: {"asn": [], "bgppref": "", "bl": [], "geo": {"ctry": "CZ"}, "hostname": "ns.cesnet.cz", "ip": "195.113.144.194", "ipblock": "", "rep": 0.2, "tags": []}
26 |
27 | author = "Tomas Cejka
6 |  }})
7 |
36 |
37 | {% else %}
38 |
7 |
8 |
35 | No account
9 |You have authenticated as "{{ user.id }}" ({{ user.name }}), but there is no account registered for this name (yet).
10 | {% if request_sent %} 11 |
12 |
15 | {% else %}
16 | Account creation request was sent to NERD admins.
13 |You will be notified to {{ form.email.data }} when the account is created.
14 |You can request creation of a new account below.
17 |Note: An account is currently only needed to:
18 |
24 |
32 | {% endif %}
33 | 18 |
-
19 |
- Access the API
20 | - Enable privileged access to see some sensitive information (i.e. more details about detected events). This is available only to trusted partners from CSIRT community.
21 |
Note that an account request have to be manually approved.
34 |ERROR: This should be page for authenticated but not registered users, but no user info was found in session.
39 | {% endif %} 40 | 41 | {% endblock %} 42 | -------------------------------------------------------------------------------- /install/cron/nerd: -------------------------------------------------------------------------------- 1 | #MAILTO=someone@example.org 2 | 3 | # Update metainformation about numbers of IPs with particular event Category and Node 4 | */30 * * * * nerd mongosh --quiet nerd /nerd/scripts/update_db_meta_info.js 5 | # Compute reputation scores of BGP prefixes once an hour 6 | 55 * * * * nerd mongosh --quiet nerd /nerd/scripts/set_prefix_repscore.js 7 | 8 | # Generate list of IPs and reputation scores every hour 9 | # (generate to a temp file and rename to avoid clients reading an incomplete file) 10 | 00 * * * * nerd /nerd/scripts/generate_ip_rep_list.sh > /data/web_data/ip_rep.csv.tmp && mv /data/web_data/ip_rep.csv{.tmp,} 11 | 00 * * * * nerd /nerd/scripts/generate_blocklist.sh 0.5 | sort -n > /data/web_data/bad_ips.txt.tmp && mv /data/web_data/bad_ips.txt{.tmp,} 12 | 00 * * * * nerd /nerd/scripts/generate_blocklist.sh 0.2 | sort -n > /data/web_data/bad_ips_med_conf.txt.tmp && mv /data/web_data/bad_ips_med_conf.txt{.tmp,} 13 | 14 | # Remove old IDEA messages from PostgreSQL every day at 03:00 15 | # (enable if local PSQL is used to store alerts from Warden) 16 | #0 03 * * * nerd /nerd/scripts/nerd_clean_eventdb.sh > /dev/null 17 | 18 | # Download GeoIP database every Tuesday evening 19 | # Fill API_KEY and uncomment 20 | # TODO: It's probably needed to somehow notify NERDd that it needs to reload the database 21 | #10 23 * * 2 nerd /nerd/scripts/download_maxmind_geolite.sh #FILL_THE_API_KEY_HERE_AND_UNCOMMENT 22 | 23 | # rsync Uceprotect blacklist 3 times a day 24 | 40 01,09,17 * * * nerd rsync -azq rsync-mirrors.uceprotect.net::RBLDNSD-ALL/dnsbl-1.uceprotect.net /data/blacklists/uceprotect-level1 25 | # rsync PSBL blacklist 3 times a day 26 | 41 01,09,17 * * * nerd rsync -zq psbl-mirror.surriel.com::psbl/psbl.txt /data/blacklists/psbl.txt 27 | 28 | # Export Crowdsec community blacklist to CSV 29 | # (enable if you're using Crowdsec) 30 | #42 01,09,17 * * * nerd sqlite3 -csv /opt/crowdsec/data/crowdsec.db "SELECT value, scenario FROM decisions WHERE origin='CAPI';" > /data/blacklists/crowdsec.csv 31 | 32 | # Check Apache log for 5xx errors every hour. If grep produces output, it's sent to the email contact. 33 | # Run at the end of every hour and simply filter all log lines with the current hour (not perfect, but simple) 34 | #59 * * * * root egrep "$(date +\%d/\%b/\%Y:\%H).* 5[0-9][0-9] [0-9]+" /var/log/httpd/ssl_access_log -------------------------------------------------------------------------------- /scripts/download_maxmind_geolite.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Download MaxMind's GeoLite2 database to /data/geoip/GeoLite2-City.mmdb 4 | # 5 | # MaxMind no longer allows to download the data without registration. 6 | # Therefore, this script must be called manually with licence key passed. 7 | # 8 | # Usage: 9 | # ./donwload_maxmind_geolite.sh LICENCE_KEY 10 | # 11 | # 12 | 13 | KEY=$1 14 | if [ -z "$KEY" ]; then 15 | echo "Licence key is needed (just register for free on maxmind.com)." >&2 16 | echo "Usage:" >&2 17 | echo " ./download_maxmind_geolite.sh LICENCE_KEY" >&2 18 | echo "Run as user 'nerd' or root." >&2 19 | exit 1 20 | fi 21 | 22 | user=$(whoami) 23 | if [[ "$user" != "nerd" && "$user" != "root" ]]; then 24 | echo "Run as user 'nerd' or root." >&2 25 | exit 2 26 | fi 27 | 28 | # exit when any command fails 29 | set -e 30 | 31 | echo "Downloading MaxMind GeoLite2 database to /data/geoip/GeoLite2-City.mmdb" 32 | # Needed by geolocation module 33 | mkdir -p /data/geoip 34 | cd /data/geoip 35 | # Download archive using the licence key 36 | wget -q "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=${KEY}&suffix=tar.gz" -O GeoLite2-City.tar.gz 37 | # The archive contains a directory named by creation date, containing the DB 38 | # and some txt file - extract just the DB file (*/GeoLite2-City.mmdb) to 39 | # current dir 40 | tar -xzf GeoLite2-City.tar.gz '*/GeoLite2-City.mmdb' --strip-components=1 41 | 42 | echo "Downloading MaxMind GeoLite2 databases (CSV) to /data/geoip/GeoLite2-*.csv" 43 | # Needed by FMP computation scripts 44 | # three files needed: GeoLite2-ASN-Blocks-IPv4.csv, GeoLite2-Country-Blocks-IPv4.csv, GeoLite2-Country-Locations-en.csv 45 | wget -q "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN-CSV&license_key=${KEY}&suffix=zip" -O GeoLite2-ASN-CSV.zip 46 | unzip -joq GeoLite2-ASN-CSV.zip 'GeoLite2-ASN-CSV_*/GeoLite2-ASN-Blocks-IPv4.csv' 47 | wget -q "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=${KEY}&suffix=zip" -O GeoLite2-Country-CSV.zip 48 | unzip -joq GeoLite2-Country-CSV.zip 'GeoLite2-Country-CSV_*/GeoLite2-Country-Blocks-IPv4.csv' 'GeoLite2-Country-CSV_*/GeoLite2-Country-Locations-en.csv' 49 | 50 | if [[ "$user" == "root" ]]; then 51 | echo "Setting ownership to 'nerd' account" 52 | chown -R nerd:nerd /data/geoip 53 | fi 54 | 55 | echo "Done" 56 | -------------------------------------------------------------------------------- /NERDd/modules/eml_asn_rank.py: -------------------------------------------------------------------------------- 1 | """ 2 | NERD module querying ASN ranking API of "The Email Laundry (EML)" company. 3 | """ 4 | 5 | from core.basemodule import NERDModule 6 | import g 7 | 8 | import logging 9 | import requests 10 | 11 | class EML_ASN_rank(NERDModule): 12 | """ 13 | EML ASN rank module. 14 | 15 | Queries newly added ASNs in EML's API for their ASN rank. 16 | 17 | Event flow specification: 18 | asn: !NEW -> get_rank -> eml_rank 19 | """ 20 | 21 | def __init__(self): 22 | self.log = logging.getLogger('EML_ASN_rank') 23 | #self.log.setLevel("DEBUG") 24 | self.url = g.config.get('eml_api.url', None) 25 | self.apikey = g.config.get('eml_api.key', None) 26 | if not self.url or not self.apikey: 27 | self.log.warning("API URL or key not set, EML ASN rank module disabled.") 28 | return 29 | self.log.debug("EML ASN Rank module initialized") 30 | 31 | g.um.register_handler( 32 | self.get_rank, 33 | 'asn', 34 | ('!NEW','every1d','!refresh_eml_rank'), 35 | ('eml_rank',) 36 | ) 37 | 38 | def get_rank(self, ekey, rec, updates): 39 | """ 40 | Query the EML API to get ASN rank. 41 | 42 | Arguments: 43 | ekey -- two-tuple of entity type and key, e.g. ('ip', '192.0.2.42') 44 | rec -- record currently assigned to the key 45 | updates -- list of all attributes whose update triggered this call and 46 | their new values (or events and their parameters) as a list of 47 | 2-tuples: [(attr, val), (!event, param), ...] 48 | 49 | 50 | Returns: 51 | List of update requests. 52 | """ 53 | etype, key = ekey 54 | if etype != 'asn': 55 | return None 56 | 57 | try: 58 | r = requests.get('{}asn/{}?key={}'.format(self.url, key, self.apikey)) 59 | r.raise_for_status() 60 | data = r.json() 61 | rank = float(data['asnrankinfo']['asnrank']) 62 | #self.log.debug("Setting EML ASN Rank of {} to {}".format(key, rank)) 63 | except Exception as e: 64 | self.log.error("Can't get rank for AS{}: {}".format(key, repr(e))) 65 | return None 66 | 67 | return [('set', 'eml_rank', rank)] 68 | -------------------------------------------------------------------------------- /NERDweb/templates/data.html: -------------------------------------------------------------------------------- 1 | {% extends "layout.html" %} 2 | {% block body %} 3 | 4 |Downloadable data
5 | 6 |-
7 |
- List of all IPs in NERD database with their reputation scores:
8 | {% if file_sizes['ip_rep.csv'] is number %}ip_rep.csv ({{file_sizes['ip_rep.csv']|filesizeformat}}){% else %}ERROR File not found{% endif %}
9 |
- Don't use this file as a blocklist as is! It contains every single IP reported as malicious by some of our data sources and it probably contains some false positives (FP). Look at the associated reputation score, which summarizes the number of alerts, their age, and number of different sources that reported the IP. Higher score means lower probability of FP. So, if this data is to be used as a blocklist, we recommend to select only the IPs with reputation score greater than some threshold. Or use one of the files below, instead.
11 | - List of malicious IPs (high confidence):
12 | {% if file_sizes['bad_ips.txt'] is number %}bad_ips.txt ({{file_sizes['bad_ips.txt']|filesizeformat}}){% else %}ERROR File not found{% endif %}
13 |
- IP addresses with reputation score greater than 0.5 (means many recent reports from multiple sources). Very low chance of false positives. IPs tagged as research scanners are not included.
15 | - List of malicious IPs (medium confidence):
16 | {% if file_sizes['bad_ips_med_conf.txt'] is number %}bad_ips_med_conf.txt ({{file_sizes['bad_ips_med_conf.txt']|filesizeformat}}){% else %}ERROR File not found{% endif %}
17 |
- IP addresses with reputation score greater than 0.2. This also includes less active IPs or those reported by just one source (but still multiple recent alerts are needed). Mostly reliable, but may contain a few false positives. IPs tagged as research scanners are not included.
19 |
We recommend downloading it few minutes after a whole hour (i.e. between xx:01 and xx:05). 22 | 23 | {% endblock %} 24 | -------------------------------------------------------------------------------- /NERDd/core/db.py: -------------------------------------------------------------------------------- 1 | """ 2 | NERD entity database wrapper. 3 | 4 | Provides EntityDatabase class -- an abstract layer above the database system 5 | implementing entity database. 6 | """ 7 | 8 | # import mongo 9 | 10 | class UnknownEntityType(ValueError): 11 | pass 12 | 13 | class EntityDatabase: 14 | """ 15 | Abstract layer above the entity database. It provides an interface for 16 | database operations independet on underlying database system. 17 | 18 | This is a trivial in-memory database based on Python dict - useful only for 19 | debugging/testing. 20 | """ 21 | # List of known/supported entity types - currently only IP addresses (both IPv4 and IPv6 are treated the same) 22 | _supportedTypes = ['ip'] 23 | 24 | def __init__(self, config): 25 | """ 26 | Connect to database and initialize all internal structures as neccessary. 27 | """ 28 | self._db = {'ip': {}} 29 | 30 | # def __del__(self): 31 | # """ 32 | # Destructor. Close connection to database. 33 | # """ 34 | # pass 35 | 36 | def getEntityTypes(self): 37 | """ 38 | Return list of known entity types. 39 | 40 | Currently only 'ip' type is supported. 41 | """ 42 | return self._supportedTypes 43 | 44 | 45 | def get(self, etype, key): 46 | """ 47 | Return record of given entity. 48 | 49 | Arguments: 50 | etype entity type (str), e.g. 'ip' 51 | key entity identifier (str), e.g. '192.0.2.42' 52 | 53 | Return the record as JSON document or None if it is not present in the database. 54 | 55 | Raise UnknownEntityType if there is not database collection for given etype. 56 | """ 57 | if etype not in self._supportedTypes: 58 | raise UnknownEntityType("There is no collection for entity type "+str(etype)) 59 | 60 | return self._db[etype].get(key, None) 61 | 62 | def put(self, etype, key, record): 63 | """ 64 | Store a record into the database or replace old with a new one. 65 | 66 | Arguments: 67 | etype entity type (str), e.g. 'ip' 68 | key entity identifier (str), e.g. '192.0.2.42' 69 | record JSON document with properties of the entity to be stored in DB 70 | """ 71 | self._db[etype][key] = record 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /NERDd/core/scheduler.py: -------------------------------------------------------------------------------- 1 | """ 2 | NERD scheduler - allows modules to register functions (callables) to be run at 3 | specified times or intervals (like cron does). 4 | 5 | Based on APScheduler package 6 | """ 7 | 8 | import logging 9 | from apscheduler.schedulers.background import BackgroundScheduler 10 | from apscheduler.triggers.cron import CronTrigger 11 | 12 | class Scheduler(): 13 | """ 14 | NERD scheduler - allows modules to register functions (callables) to be run 15 | at specified times or intervals (like cron does). 16 | """ 17 | def __init__(self): 18 | self.log = logging.getLogger("Scheduler") 19 | #self.log.setLevel("DEBUG") 20 | logging.getLogger("apscheduler.scheduler").setLevel("WARNING") 21 | logging.getLogger("apscheduler.executors.default").setLevel("WARNING") 22 | self.sched = BackgroundScheduler(timezone="UTC") 23 | self.last_job_id = 0 24 | 25 | def start(self): 26 | self.log.debug("Scheduler start") 27 | self.sched.start() 28 | 29 | def stop(self): 30 | self.log.debug("Scheduler stop") 31 | self.sched.shutdown() 32 | 33 | def register(self, func, year=None, month=None, day=None, week=None, 34 | day_of_week=None, hour=None, minute=None, second=None, timezone="UTC", 35 | args=None, kwargs=None): 36 | """ 37 | Register a function to be run at specified times. 38 | 39 | func - function or method to be called 40 | year,month,day,week,day_of_week,hour,minute,second - 41 | cron-like specification of when the function should be called, 42 | see docs of apscheduler.triggers.cron for details 43 | https://apscheduler.readthedocs.io/en/latest/modules/triggers/cron.html 44 | timezone - Timezone for time specification (default is UTC). 45 | args, kwargs - arguments passed to func 46 | 47 | Return job ID (integer). 48 | """ 49 | self.last_job_id += 1 50 | trigger = CronTrigger(year, month, day, week, day_of_week, hour, minute, 51 | second, timezone=timezone) 52 | self.sched.add_job(func, trigger, args, kwargs, coalesce=True, max_instances=1, id=str(self.last_job_id)) 53 | self.log.debug("Registered function {0} to be called at {1}".format(func.__qualname__, trigger)) 54 | return self.last_job_id 55 | 56 | def pause_job(self, id): 57 | """Pause job with given ID""" 58 | self.sched.pause_job(str(id)) 59 | 60 | def resume_job(self, id): 61 | """Resume previously paused job with given ID""" 62 | self.sched.resume_job(str(id)) 63 | 64 | -------------------------------------------------------------------------------- /NERDd/modules/test_module.py: -------------------------------------------------------------------------------- 1 | """NERD module for testing UpdateManager""" 2 | 3 | from core.basemodule import NERDModule 4 | import time 5 | 6 | class TestModule (NERDModule): 7 | """ 8 | Testing NERD module. 9 | 10 | Event flow specification: 11 | !NEW -> setType -> type, type_desc 12 | !NEW -> setA -> A 13 | A -> setB -> B 14 | A, B -> setC -> C 15 | !sleep -> sleep -> None 16 | """ 17 | 18 | def __init__(self): 19 | g.um.register_handler( 20 | self.setB, # function (or bound method) to call 21 | ('A',), # tuple/list/set of attributes to watch (their update triggers call of the registered method) 22 | ('B',) # tuple/list/set of attributes the method may change 23 | ) 24 | g.um.register_handler( 25 | self.setA, 'ip', ('!NEW',), ('A',) 26 | ) 27 | g.um.register_handler( 28 | self.setType, 'ip', ('!NEW',), ('type','type_desc') 29 | ) 30 | g.um.register_handler( 31 | self.setC, 'ip', ('A','B'), ('C') 32 | ) 33 | g.um.register_handler( 34 | self.sleep, 'ip', ('!sleep',), None 35 | ) 36 | 37 | def setB(self, ekey, rec, updates): 38 | """ 39 | Set a 'B' attribute to the value of an 'A' attribute concatenated with 40 | 'extended by B'. 41 | 42 | Arguments: 43 | etype, key -- entity type and key (e.g. 'ip', '192.0.2.42') 44 | rec -- record currently assigned to the key 45 | updates -- specification of updates that triggerd this call 46 | 3-tuple (op, attr, val) or ('event', name, param) 47 | 48 | Returns: 49 | List of 3-tuples describing requested attribute updates or events. 50 | """ 51 | return [('set', 'B', rec['A'] + ' extended by B')] 52 | 53 | def setA(self, ekey, rec, updates): 54 | return [('set', 'A', 'Value of A')] 55 | 56 | def setType(self, ekey, rec, updates): 57 | return [ 58 | ('set', 'type', ekey[0]), 59 | ('set', 'type_desc', 'This is record of type {}'.format(ekey[0])), 60 | ] 61 | 62 | def setC(self, ekey, rec, updates): 63 | update_reqs = [] 64 | for op,key,val in updates: 65 | update_reqs.append(('add_to_set', 'C', '{} changed'.format(key))) 66 | return update_reqs 67 | 68 | def sleep(self, ekey, rec, updates): 69 | for event in updates: 70 | assert(event[1] == '!sleep') 71 | print() 72 | print("Sleeping ...") 73 | print("zZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZzZ") 74 | print() 75 | time.sleep(event[2]) 76 | return None 77 | -------------------------------------------------------------------------------- /install/configure_apache.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Configure Apache to serve NERDweb 3 | # Pass base location without trailing slash (e.g. "/nerd" or "/") as parameter 4 | # (default is "/") 5 | # Add "-d" parameter to install in debug mode 6 | 7 | BASEDIR=$(dirname $0) 8 | . $BASEDIR/common.sh 9 | 10 | echob "=============== Configure Apache ===============" 11 | 12 | # (in case base_loc is server root, httpd needs "/", while nerd needs "") 13 | base_loc_httpd="/" # location without trailing slash (except when "/") 14 | base_loc_httpd_slash="/" # location with trailing slash 15 | base_loc_nerd="" 16 | debug=0 17 | 18 | while getopts "d" opt; do 19 | case "$opt" in 20 | d) 21 | debug=1 22 | ;; 23 | *) 24 | echo "Unknown parameter" 25 | exit 1 26 | ;; 27 | esac 28 | done 29 | shift $((OPTIND-1)) 30 | 31 | # If base_loc is set and not root directory (which is default and treated specially) 32 | if [ -n "$1" -a "$1" != "/" ] ; then 33 | base_loc_httpd="$1" 34 | base_loc_httpd_slash="$1/" 35 | base_loc_nerd="$1" 36 | fi 37 | 38 | echob "Base location: $base_loc_httpd" 39 | if [ "$debug" = 1 ] ; then 40 | echor "WARNING: SETTING UP DEBUG MODE!" 41 | fi 42 | 43 | echob "** Installing Apache and WSGI **" 44 | yum install -q -y httpd httpd-devel mod_wsgi 45 | pip3 -q install mod_wsgi 46 | 47 | # Replace the stock mod_wsgi.so with the one from Python36 48 | rm -f /usr/lib64/httpd/modules/mod_wsgi.so 49 | path="$(pip3 show mod_wsgi 2>/dev/null | sed -n '/Location: / s/Location: //p')" 50 | if [ -z "$path" ] ; then 51 | echor "ERROR: Can't find the path to mod_wsgi python package, can't create symlink to it. Apache won't start." >&2 52 | else 53 | ln -s "$path"/mod_wsgi/server/mod_wsgi-py36.*.so /usr/lib64/httpd/modules/mod_wsgi.so 54 | fi 55 | 56 | echob "** Setting up configuration files **" 57 | if [ "$debug" = 1 ] ; then 58 | cp $BASEDIR/httpd/nerd-debug.conf /etc/httpd/conf.d/nerd.conf 59 | else 60 | cp $BASEDIR/httpd/nerd.conf /etc/httpd/conf.d/nerd.conf 61 | fi 62 | # Set up base loc in both apache conf and NERD conf 63 | sed -i -E "s|^Define\s+NERDBaseLoc\s+.*$|Define NERDBaseLoc $base_loc_httpd|" /etc/httpd/conf.d/nerd.conf 64 | sed -i -E "s|^Define\s+NERDBaseLocS\s+.*$|Define NERDBaseLocS $base_loc_httpd_slash|" /etc/httpd/conf.d/nerd.conf 65 | sed -i -E "s|^base_url:.*$|base_url: \"$base_loc_nerd\"|" /etc/nerd/nerdweb.yml 66 | 67 | # Set up random "secret" number for Flask 68 | secret=$(head -c 24 /dev/urandom | base64) 69 | sed -i -E "s|^secret_key: \"!!! CHANGE THIS !!!\"|secret_key: \"$secret\"|" /etc/nerd/nerdweb.yml 70 | 71 | echob "** Setting up firewall (allow port 80, 443) **" 72 | iptables -I INPUT 1 -p TCP --dport 80 -j ACCEPT 73 | iptables -I INPUT 1 -p TCP --dport 443 -j ACCEPT 74 | iptables-save > /etc/sysconfig/iptables 75 | 76 | echob "** Starting Apache **" 77 | systemctl enable httpd 78 | systemctl restart httpd 79 | -------------------------------------------------------------------------------- /NERDweb/shodan_rpc_client.py: -------------------------------------------------------------------------------- 1 | import pika 2 | import uuid 3 | import time 4 | 5 | from common.config import read_config 6 | 7 | TIMEOUT = 10 # how many seconds to wait for RPC reply 8 | 9 | DEFAULT_RMQ_SETTINGS = { 10 | 'host': "localhost", 11 | 'port': 5672, 12 | 'virtual_host': "/", 13 | 'username': "guest", 14 | 'password': "guest" 15 | } 16 | 17 | # config - load nerd.yml 18 | config = read_config("/etc/nerd/nerd.yml") 19 | rmq_settings = DEFAULT_RMQ_SETTINGS 20 | rmq_settings.update(config.get('rabbitmq', {})) 21 | 22 | 23 | class ShodanRpcClient(object): 24 | def __init__(self): 25 | rmq_creds = pika.PlainCredentials(rmq_settings['username'], rmq_settings['password']) 26 | rmq_params = pika.ConnectionParameters(rmq_settings['host'], rmq_settings['port'], rmq_settings['virtual_host'], 27 | rmq_creds) 28 | self.connection = pika.BlockingConnection(rmq_params) 29 | self.channel = self.connection.channel() 30 | 31 | result = self.channel.queue_declare(queue='', arguments={'x-message-ttl' : 30000}, exclusive=True) 32 | self.callback_queue = result.method.queue 33 | self.response = None 34 | self.corr_id = None 35 | self.channel.basic_consume(queue=self.callback_queue, on_message_callback=self.on_response, auto_ack=True) 36 | 37 | def __del__(self): 38 | if not self.channel.connection.is_closed: 39 | try: 40 | self.channel.close() 41 | except (pika.exceptions.ConnectionClosed, pika.exceptions.StreamLostError): # for case it's been closed by the server 42 | pass 43 | 44 | def on_response(self, ch, method, props, body): 45 | if self.corr_id == props.correlation_id: 46 | self.response = body 47 | 48 | def call(self, ip): 49 | #print("(shodan_rpc_client) Request for {} received".format(str(ip))) 50 | self.response = None 51 | self.corr_id = str(uuid.uuid4()) 52 | self.channel.basic_publish(exchange='', 53 | routing_key='shodan_rpc_queue', 54 | properties=pika.BasicProperties( 55 | reply_to=self.callback_queue, 56 | correlation_id=self.corr_id, 57 | ), 58 | body=str(ip)) 59 | #print("(shodan_rpc_client) Request sent to RabbitMQ, waiting for response...") 60 | 61 | start = time.time() 62 | while self.response is None: 63 | self.connection.process_data_events(time_limit=2) 64 | if time.time() - start > TIMEOUT: 65 | return '{"error": "timeout"}' 66 | 67 | #print("(shodan_rpc_client) Response received, returning to web client") 68 | return str(self.response.decode('utf8')) 69 | -------------------------------------------------------------------------------- /NERDweb/static/status-box.js: -------------------------------------------------------------------------------- 1 | /* JS code for status-box (normally only available to admins) */ 2 | 3 | /**** Show/hide & enable/disable *****/ 4 | 5 | var refresh_status_enable = false; // Enable/disable refresh status 6 | var refreshing_status = false; // Whether refresh is in progress (to avoid multiple parallel calls it they are very slow and refresh interval is small) 7 | 8 | function toggle_status_box() { 9 | var div = $('#status-block > div'); 10 | var a = $('#status_box_toggle'); 11 | if (a.text() == "▲") { 12 | div.hide(); 13 | a.text("▼"); 14 | refresh_status_enable = false; 15 | $("#status_refresh_toggle").text("disabled"); 16 | } 17 | else { 18 | div.show(); 19 | a.text("▲"); 20 | } 21 | } 22 | 23 | function toggle_status_refresh() { 24 | if (refresh_status_enable) { 25 | refresh_status_enable=false; 26 | $("#status_refresh_toggle").text("disabled"); 27 | } 28 | else { 29 | refresh_status_enable=true; 30 | $("#status_refresh_toggle").text("enabled"); 31 | } 32 | } 33 | 34 | /***** AJAX request to get NERD status information *****/ 35 | function refresh_status(force=false) { 36 | if (!force && (!refresh_status_enable || refreshing_status)) { 37 | return; 38 | } 39 | refreshing_status = true; 40 | $("#status-block .refresh-spinner").css('visibility', 'visible'); 41 | $.getJSON( 42 | URL_GET_STATUS, 43 | function(data) { 44 | $("#status-cnt-ip").text(data.cnt_ip); 45 | $("#status-cnt-bgppref").text(data.cnt_bgppref); 46 | $("#status-cnt-asn").text(data.cnt_asn); 47 | $("#status-cnt-ipblock").text(data.cnt_ipblock); 48 | $("#status-cnt-org").text(data.cnt_org); 49 | $("#status-updates").text(data.updates_processed); 50 | $("#status-disk-usage").text(data.disk_usage); 51 | $("#status-idea-queue").text(data.idea_queue); 52 | // Set width of bar and its color 53 | var bar_width = (data.idea_queue * 100 / 10000); 54 | $("#status-idea-queue-bar div").css('width', bar_width + "%"); 55 | if (bar_width < 50) { 56 | $("#status-idea-queue-bar div").css('background-color', '#0c0'); 57 | } 58 | else if (bar_width < 75) { 59 | $("#status-idea-queue-bar div").css('background-color', '#ea0'); 60 | } 61 | else { 62 | $("#status-idea-queue-bar div").css('background-color', '#c00'); 63 | } 64 | } 65 | ) 66 | .always(function() { 67 | refreshing_status = false; 68 | $("#status-block .refresh-spinner").css('visibility', 'hidden'); 69 | }); 70 | } 71 | 72 | // Set automatic refresh every 2s 73 | var status_refresh_timer = window.setInterval(refresh_status, 2000); 74 | // Run refresh once on load 75 | $(function() { refresh_status(true) }); 76 | -------------------------------------------------------------------------------- /NERDweb/static/main.js: -------------------------------------------------------------------------------- 1 | /* NERD web - main JS code, common for whole web */ 2 | 3 | function create_event_table(data) { /* data are "dataset" field of a DOM node with "data-" attributes set */ 4 | if (data.table == "") { 5 | return "No events"; 6 | } 7 | var cats = data.cats.split(","); 8 | var dates = data.dates.split(","); 9 | var table = []; 10 | var table_rows = data.table.split(";"); 11 | for (i = 0; i < table_rows.length; i++) { 12 | table.push(table_rows[i].split(",")); 13 | } 14 | var nodes = data.nodes.split(","); 15 | 16 | var content = "
| "; 17 | content += cats.join(" | "); 18 | content += " | |
|---|---|---|
| "+dates[i]+" | "; 21 | content += table[i].join(" | "); 22 | content += " |
Visualization of IPv4 address space
15 | 16 |The visualization uses Hilbert curve to display IPv4 address space. Each pixel on the plane represents a certain network, its value (and color) is the sum of reputation scores of IP addresses in that network.
17 | 18 | {% if ipvis_url and ipvis_token %} 19 | 20 | 21 | 95 | {% else %} 96 |IP visualization backend not configured.
97 | {% endif %} 98 | 99 | {% endblock %} 100 | -------------------------------------------------------------------------------- /NERDweb/static/jquery.multiselect.css: -------------------------------------------------------------------------------- 1 | .ms-options-wrap, 2 | .ms-options-wrap * { 3 | box-sizing: border-box; 4 | } 5 | 6 | .ms-options-wrap > button:focus, 7 | .ms-options-wrap > button { 8 | position: relative; 9 | width: 100%; 10 | text-align: left; 11 | border: 1px solid #aaa; 12 | background-color: #fff; 13 | padding: 5px 20px 5px 5px; 14 | margin-top: 1px; 15 | font-size: 13px; 16 | color: #aaa; 17 | outline-offset: -2px; 18 | white-space: nowrap; 19 | } 20 | 21 | .ms-options-wrap > button > span { 22 | display: inline-block; 23 | } 24 | 25 | .ms-options-wrap > button[disabled] { 26 | background-color: #e5e9ed; 27 | color: #808080; 28 | opacity: 0.6; 29 | } 30 | 31 | .ms-options-wrap > button:after { 32 | content: ' '; 33 | height: 0; 34 | position: absolute; 35 | top: 50%; 36 | right: 5px; 37 | width: 0; 38 | border: 6px solid rgba(0, 0, 0, 0); 39 | border-top-color: #999; 40 | margin-top: -3px; 41 | } 42 | 43 | .ms-options-wrap.ms-has-selections > button { 44 | color: #333; 45 | } 46 | 47 | .ms-options-wrap > .ms-options { 48 | position: absolute; 49 | left: 0; 50 | width: 100%; 51 | margin-top: 1px; 52 | margin-bottom: 20px; 53 | background: white; 54 | z-index: 2000; 55 | border: 1px solid #aaa; 56 | overflow: auto; 57 | visibility: hidden; 58 | } 59 | 60 | .ms-options-wrap.ms-active > .ms-options { 61 | visibility: visible 62 | } 63 | 64 | .ms-options-wrap > .ms-options > .ms-search input { 65 | width: 100%; 66 | padding: 4px 5px; 67 | border: none; 68 | border-bottom: 1px groove; 69 | outline: none; 70 | } 71 | 72 | .ms-options-wrap > .ms-options .ms-selectall { 73 | display: inline-block; 74 | font-size: .9em; 75 | text-transform: lowercase; 76 | text-decoration: none; 77 | } 78 | .ms-options-wrap > .ms-options .ms-selectall:hover { 79 | text-decoration: underline; 80 | } 81 | 82 | .ms-options-wrap > .ms-options > .ms-selectall.global { 83 | margin: 4px 5px; 84 | } 85 | 86 | .ms-options-wrap > .ms-options > ul, 87 | .ms-options-wrap > .ms-options > ul > li.optgroup ul { 88 | list-style-type: none; 89 | padding: 0; 90 | margin: 0; 91 | } 92 | 93 | .ms-options-wrap > .ms-options > ul li.ms-hidden { 94 | display: none; 95 | } 96 | 97 | .ms-options-wrap > .ms-options > ul > li.optgroup { 98 | padding: 5px; 99 | } 100 | .ms-options-wrap > .ms-options > ul > li.optgroup + li.optgroup { 101 | border-top: 1px solid #aaa; 102 | } 103 | 104 | .ms-options-wrap > .ms-options > ul > li.optgroup .label { 105 | display: block; 106 | padding: 5px 0 0 0; 107 | font-weight: bold; 108 | } 109 | 110 | .ms-options-wrap > .ms-options > ul label { 111 | position: relative; 112 | display: inline-block; 113 | width: 100%; 114 | padding: 4px 4px 4px 20px; 115 | margin: 1px 0; 116 | border: 1px dotted transparent; 117 | } 118 | .ms-options-wrap > .ms-options.checkbox-autofit > ul label, 119 | .ms-options-wrap > .ms-options.hide-checkbox > ul label { 120 | padding: 4px; 121 | } 122 | 123 | .ms-options-wrap > .ms-options > ul label.focused, 124 | .ms-options-wrap > .ms-options > ul label:hover { 125 | background-color: #efefef; 126 | border-color: #999; 127 | } 128 | 129 | .ms-options-wrap > .ms-options > ul li.selected label { 130 | background-color: #efefef; 131 | border-color: transparent; 132 | } 133 | 134 | .ms-options-wrap > .ms-options > ul input[type="checkbox"] { 135 | margin: 0 5px 0 0; 136 | position: absolute; 137 | left: 4px; 138 | top: 7px; 139 | } 140 | 141 | .ms-options-wrap > .ms-options.hide-checkbox > ul input[type="checkbox"] { 142 | position: absolute !important; 143 | height: 1px; 144 | width: 1px; 145 | overflow: hidden; 146 | clip: rect(1px 1px 1px 1px); 147 | clip: rect(1px, 1px, 1px, 1px); 148 | } 149 | -------------------------------------------------------------------------------- /scripts/get_iana_assignment_files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | #-*- encoding: utf-8 -*- 3 | 4 | import subprocess 5 | import ipaddress 6 | import csv 7 | 8 | SPECIAL_PURPOSE_ADDRESS ='\ 9 | 0 16777215 Reserved:arin\n\ 10 | 167772160 184549375 Reserved:arin\n\ 11 | 1681915904 1686110207 Reserved:arin\n\ 12 | 2130706432 2147483647 Reserved:arin\n\ 13 | 2851995648 2852061183 Reserved:arin\n\ 14 | 2886729728 2887778303 Reserved:arin\n\ 15 | 3221225472 3221225727 Reserved:arin\n\ 16 | 3221225984 3221226239 Reserved:arin\n\ 17 | 3223307264 3223307519 Reserved:arin\n\ 18 | 3224682752 3224683007 Reserved:arin\n\ 19 | 3227017984 3227018239 Reserved:arin\n\ 20 | 3232235520 3232301055 Reserved:arin\n\ 21 | 3232706560 3232706815 Reserved:arin\n\ 22 | 3323068416 3323199487 Reserved:arin\n\ 23 | 3325256704 3325256959 Reserved:arin\n\ 24 | 3405803776 3405804031 Reserved:apnic\n\ 25 | 4026531840 4294967295 Reserved\n\ 26 | 4294967295 4294967295 Reserved\n' 27 | 28 | SPECIAL_PURPOSE_ASN ='\ 29 | 0,Reserved:ripe\n\ 30 | 1,arin\n\ 31 | 112,Reserved:arin\n\ 32 | 113,arin\n\ 33 | 23456,Reserved:arin\n\ 34 | 23457,arin\n\ 35 | 64496,Reserved:ripe\n\ 36 | 131072,apnic\n\ 37 | 4200000000,Reserved:ripe\n' 38 | 39 | DOWNLOAD_IP_COMMAND = '\ 40 | loc=("lacnic" "ripe" "arin" "afrinic" "apnic");\ 41 | rirs=("lacnic" "ripencc" "arin" "afrinic" "apnic");\ 42 | for i in ${!rirs[*]};\ 43 | do \ 44 | url="https://ftp."${loc[$i]}".net/pub/stats/"${rirs[$i]}"/delegated-"${rirs[$i]}"-extended-latest";\ 45 | echo "$url";\ 46 | wget -q "$url";\ 47 | cat "delegated-"${rirs[$i]}"-extended-latest" | grep "ipv4" | awk \'BEGIN { FS = "|"} ; {print $4","$5","$1}\' | tail -n +2 >> csv_tmp;\ 48 | rm -f "delegated-"${rirs[$i]}"-extended-latest";\ 49 | done' 50 | 51 | DOWNLOAD_ASN_COMMAND = '\ 52 | wget -q https://www.iana.org/assignments/as-numbers/as-numbers-1.csv;\ 53 | wget -q https://www.iana.org/assignments/as-numbers/as-numbers-2.csv;\ 54 | cat as-numbers-1.csv | tr " " "," | egrep "ARIN|APNIC|RIPE|AFRINIC|LACNIC" | awk \'BEGIN { FS = ","} ; {print $1","tolower($4)}\' >> asn_tmp;\ 55 | cat as-numbers-2.csv | tr " " "," | egrep "ARIN|APNIC|RIPE|AFRINIC|LACNIC|Unallocated" | awk \'BEGIN { FS = ","} ; {if ($2 == "Unallocated")print $1","$2; else print $1","tolower($4);}\' >> asn_tmp;\ 56 | rm -f as-numbers-1.csv as-numbers-2.csv' 57 | 58 | SORT_UNIQ_COMMAND_IPV4 = 'cat trans_tmp | sort -n -k 1,1 | tr " " "," > nerd-whois-ipv4.csv' 59 | SORT_UNIQ_COMMAND_ASN = 'cat asn_tmp2 | tr "," " " | sort -n -k1,1 | uniq -f 1 | tr " " "," > nerd-whois-asn.csv' 60 | 61 | CLEANUP_COMMAND = 'rm -f csv_tmp trans_tmp asn_tmp asn_tmp2' 62 | 63 | print("Downloading list of IP block allocations from FTP servers...") 64 | 65 | subprocess.call(DOWNLOAD_IP_COMMAND, shell=True, executable='/bin/bash') 66 | 67 | r = open('csv_tmp', 'r') 68 | w = open('trans_tmp', 'w') 69 | datareader = csv.reader(r, delimiter=',') 70 | 71 | print("Converting IP representation to long uint...") 72 | 73 | for row in datareader: 74 | rir = 'ripe' if row[2] == "ripencc" else row[2] 75 | first_ip = int(ipaddress.ip_address(row[0])) 76 | last_ip = first_ip + int(row[1]) - 1 77 | w.write(str(first_ip) + ' ' + str(last_ip) + ' ' + rir + '\n') 78 | 79 | w.write(SPECIAL_PURPOSE_ADDRESS) 80 | r.close() 81 | w.close() 82 | 83 | print("Removing duplicities...") 84 | 85 | subprocess.call(SORT_UNIQ_COMMAND_IPV4, shell=True, executable='/bin/bash') 86 | 87 | print("Downloading ASN allocation tables from IANA...") 88 | 89 | subprocess.call(DOWNLOAD_ASN_COMMAND, shell=True, executable='/bin/bash') 90 | 91 | r = open('asn_tmp', 'r') 92 | w = open('asn_tmp2', 'w') 93 | datareader = csv.reader(r, delimiter=',') 94 | 95 | for row in datareader: 96 | asn = row[0].split('-') 97 | w.write(asn[0] + ',' + row[1] + '\n') 98 | 99 | w.write(SPECIAL_PURPOSE_ASN) 100 | r.close() 101 | w.close() 102 | 103 | print("Cleaning up temporary files...") 104 | 105 | subprocess.call(SORT_UNIQ_COMMAND_ASN, shell=True, executable='/bin/bash') 106 | subprocess.call(CLEANUP_COMMAND, shell=True, executable='/bin/bash') 107 | 108 | print('Done!') 109 | -------------------------------------------------------------------------------- /etc/event_logging.yml: -------------------------------------------------------------------------------- 1 | # Configuration of a EventCountLogger system, which allows to count arbitrary events across multiple processes 2 | # (using shared counters in Redis) and in various time intervals. 3 | redis: 4 | host: localhost 5 | port: 6379 6 | db: 2 # Index of Redis DB used for the counters (it shouldn't be used for anything else) 7 | 8 | # Each "group" specifies a set of "events" which are handled together. 9 | groups: 10 | rec_ops: 11 | # Number of tasks processed by entity type and record operation. 12 | # Each task is counted as exactly one event: 13 | # - updated = normal task, one or more attributes of an existing record were updated 14 | # - created = task resulted in creation of a new record 15 | # - removed = task resulted in deletion of a record 16 | # - noop = task with no operations (or only weak operations and the record doesn't exist), no change in DB 17 | events: 18 | - ip_updated 19 | - ip_created 20 | - ip_removed 21 | - ip_noop 22 | - asn_updated 23 | - asn_created 24 | - asn_removed 25 | - asn_noop 26 | - bgppref_updated 27 | - bgppref_created 28 | - bgppref_removed 29 | - bgppref_noop 30 | - ipblock_updated 31 | - ipblock_created 32 | - ipblock_removed 33 | - ipblock_noop 34 | - org_updated 35 | - org_created 36 | - org_removed 37 | - org_noop 38 | auto_declare_events: true 39 | intervals: [ "5s", "5m" ] 40 | sync-interval: 1 41 | 42 | # Number of processed tasks by their source (TODO) 43 | tasks_by_src: 44 | events: 45 | - blacklists 46 | - misp_receiver 47 | - otx_receiver 48 | - updater 49 | - warden_receiver 50 | - updater_manager 51 | - web 52 | - misp_updater 53 | - dshield 54 | auto_declare_events: true 55 | intervals: ["5s", "5m"] 56 | sync-interval: 1 57 | 58 | # Logging of various errors (currently only errors in modules) 59 | errors: 60 | events: ["error_in_module"] 61 | auto_declare_events: true 62 | intervals: ["5m"] 63 | 64 | # Web - access to individual endpoints 65 | web_endpoints: 66 | events: 67 | - '/' 68 | - '/noaccount' 69 | - '/account' 70 | - '/set_effective_groups' 71 | - '/ips' 72 | - '/ips_count' 73 | - '/ips_download' 74 | - '/ip' 75 | - '/ajax/fetch_ip_data' 76 | - '/ajax/is_ip_prepared' 77 | - '/ajax/ip_events' 78 | - '/misp_event' 79 | - '/asn' 80 | - '/ipblock' 81 | - '/org' 82 | - '/bgppref' 83 | - '/status' 84 | - '/iplist' 85 | - '/map' 86 | - '/data' 87 | - '/data/ip_rep_csv' 88 | - '/data/bad_ips_txt' 89 | - '/data/bad_ips_med_conf_txt' 90 | - '/api/user_info' 91 | - '/api/ip' 92 | - '/api/ip/rep' 93 | - '/api/ip/fmp' 94 | # - '/api/ip/test' 95 | - '/api/ip/full' 96 | - '/api/search/ip' 97 | - '/api/prefix' 98 | - '/api/bad_prefixes' 99 | - '/api/ip/bulk' 100 | - '/pdns/ip' 101 | - '/api/shodan-info' 102 | # 5 min interval for Munin 103 | intervals: ["5m"] 104 | # Cache counts locally, push to Redis every 5 seconds 105 | sync-interval: 5 106 | 107 | # Web - error responses 108 | web_errors: 109 | events: 110 | - 400_bad_request 111 | - 403_no_auth_header 112 | - 403_invalid_token 113 | - 403_unauthorized 114 | - 404_api_bad_path 115 | - 404_api_ip_not_found 116 | - 429_rate_limit_api 117 | - 429_rate_limit_web 118 | - 503_db_error 119 | - 5xx_other 120 | # 5 min interval for Munin 121 | intervals: [ "5m" ] 122 | # Cache counts locally, push to Redis every 5 seconds 123 | sync-interval: 5 124 | 125 | 126 | # Shodan InternetDB module 127 | shodan: 128 | events: 129 | - add_or_update_data 130 | - no_data 131 | - remove_old_data 132 | - skipped 133 | - rate_limit 134 | - unexpected_reply 135 | # 5 min interval for Munin 136 | intervals: [ "5m" ] 137 | sync-interval: 5 138 | -------------------------------------------------------------------------------- /scripts/shodan_requester.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import sys 5 | import requests 6 | import argparse 7 | import json 8 | from datetime import datetime 9 | import logging 10 | 11 | from cachetools import TTLCache 12 | import pika 13 | 14 | # Add to path the "one directory above the current file location" to find modules from "common" 15 | sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))) 16 | 17 | from common.config import read_config 18 | 19 | LOGFORMAT = "%(asctime)-15s,%(name)s [%(levelname)s] %(message)s" 20 | LOGDATEFORMAT = "%Y-%m-%dT%H:%M:%S" 21 | logging.basicConfig(level=logging.INFO, format=LOGFORMAT, datefmt=LOGDATEFORMAT) 22 | 23 | logger = logging.getLogger('Shodan_requester') 24 | 25 | # parse arguments 26 | parser = argparse.ArgumentParser( 27 | prog="shodan_requester.py", 28 | description="NERD standalone, which will get info about IP from Shodan service when requested." 29 | ) 30 | parser.add_argument('-c', '--config', metavar='FILENAME', default='/etc/nerd/nerd.yml', 31 | help='Path to configuration file (default: /etc/nerd/nerd.yml)') 32 | parser.add_argument("-v", dest="verbose", action="store_true", help="Verbose mode") 33 | args = parser.parse_args() 34 | 35 | if args.verbose: 36 | logger.setLevel("DEBUG") 37 | 38 | # config - load nerd.yml 39 | logger.info("Loading config file {}".format(args.config)) 40 | config = read_config(args.config) 41 | 42 | api_key = config.get('shodan_api_key') 43 | rmq_settings = config.get('rabbitmq') 44 | if not api_key: 45 | logger.error("Cannot load Shodan API key, make sure it is properly configured in {}.".format(args.config)) 46 | sys.exit(1) 47 | 48 | try: 49 | rmq_creds = pika.PlainCredentials(rmq_settings['username'], rmq_settings['password']) 50 | rmq_params = pika.ConnectionParameters(rmq_settings['host'], rmq_settings['port'], rmq_settings['virtual_host'], 51 | rmq_creds) 52 | except KeyError: 53 | logger.error("RabbitMQ settings are not configured properly, make sure it is properly configured in {}.".format( 54 | args.config)) 55 | sys.exit(1) 56 | 57 | connection = pika.BlockingConnection(rmq_params) 58 | channel = connection.channel() 59 | channel.queue_declare(queue='shodan_rpc_queue', arguments={'x-message-ttl' : 30000}) # set ttl of messages to 30 sec 60 | 61 | # dictionary in format: { 'ipaddr': {ttl: time, data: data}} 62 | cache = TTLCache(maxsize=128, ttl=3600) 63 | 64 | 65 | def get_shodan_data(ip): 66 | if ip in cache: 67 | logger.debug("cache hit for {}".format(ip)) 68 | data = cache[ip] 69 | else: 70 | url = 'https://api.shodan.io/shodan/host/{ip}?key={api_key}'.format(ip=ip, api_key=api_key) 71 | resp = requests.get(url) 72 | if resp.status_code == 200: 73 | data = resp.content 74 | else: 75 | if resp.status_code != 404: 76 | logger.error("Error response for url: {}\n{}".format(url, resp.content)) 77 | try: 78 | response_dict = json.loads(resp.text) 79 | except Exception: 80 | return json.dumps({"error": "Shodan returned invalid response"}) 81 | data = json.dumps({"error": response_dict["error"] if "error" in response_dict else "Unknown error"}) 82 | 83 | if resp.status_code == 200 or resp.status_code == 404: 84 | # store returned data (or info that there's no data) to cache 85 | cache[ip] = data 86 | 87 | return data 88 | 89 | 90 | def on_request(ch, method, props, body): 91 | ip = str(body, 'utf-8') 92 | logger.debug("Received request for '{}'".format(ip)) 93 | response = get_shodan_data(ip) 94 | ch.basic_publish(exchange='', 95 | routing_key=props.reply_to, 96 | properties=pika.BasicProperties(correlation_id=props.correlation_id), 97 | body=response) 98 | ch.basic_ack(delivery_tag=method.delivery_tag) 99 | logger.debug("Sent response for '{}'".format(ip)) 100 | 101 | 102 | channel.basic_qos(prefetch_count=1) 103 | channel.basic_consume(queue='shodan_rpc_queue', on_message_callback=on_request) 104 | 105 | logger.info("*** Shodan request handler started, awaiting RPC requests ***") 106 | channel.start_consuming() 107 | -------------------------------------------------------------------------------- /common/eventdb_mentat.py: -------------------------------------------------------------------------------- 1 | """ 2 | Proxy to event database in external Mentat instance. 3 | 4 | Provides MentatEventDatabase class -- a proxy for reading data from Mentat API. 5 | Mentat is supposed to receive the same data as NERD, but via its own channel. 6 | This module only reads the data, storage is not implemented. 7 | """ 8 | from __future__ import print_function 9 | 10 | import json 11 | import logging 12 | import datetime 13 | import requests 14 | 15 | from common.utils import parse_rfc_time 16 | 17 | 18 | class BadEntityType(ValueError): 19 | pass 20 | 21 | class NotConfigured(RuntimeError): 22 | pass 23 | 24 | class GatewayError(RuntimeError): 25 | pass 26 | 27 | class MentatEventDBProxy: 28 | """ 29 | Event database reading IDEA messages from external Mentat via its API. 30 | """ 31 | 32 | def __init__(self, config): 33 | """ 34 | Initialize all internal structures as necessary. 35 | """ 36 | self.log = logging.getLogger('MentatEventDBProxy') 37 | #self.log.setLevel('DEBUG') 38 | 39 | # Load URL and API key from config 40 | self.base_url = config.get('eventdb_mentat.url', None) # should point to Mentat base API (e.g. https://example.com/mentat/) 41 | self.api_key = config.get('eventdb_mentat.api_key', None) 42 | 43 | # Check presence and validity of config (only print error if invalid) 44 | if not self.base_url: 45 | self.log.error("Mentat API used but URL not configured ('eventdb_mentat.url' config entry is missing)") 46 | elif not (self.base_url.startswith("https://") or self.base_url.startswith("http://")): 47 | self.log.error("Invalid URL of Mentat API") 48 | self.base_url = None 49 | elif not self.base_url.endswith("/"): 50 | self.base_url += "/" # ensure base_url ends with a slash 51 | if not self.api_key: 52 | self.log.error("Mentat API used but api_key not configured ('eventdb_mentat.api_key' config entry is missing)") 53 | 54 | def get(self, etype, key, limit=None, dt_from=None): 55 | """ 56 | Return all events where given IP is among Sources. 57 | 58 | Arguments: 59 | etype entity type (str), must be 'ip' 60 | key entity identifier (str), e.g. '192.0.2.42' 61 | limit max number of returned events 62 | dt_from minimal value of DetectTime (datetime) 63 | 64 | Return a list of IDEA messages (strings). 65 | 66 | Raise BadEntityType if etype is not 'ip'. 67 | """ 68 | if etype != 'ip': 69 | raise BadEntityType("etype must be 'ip'") 70 | 71 | if not self.base_url or not self.api_key: 72 | raise NotConfigured("Mentat DB connection not properly configured") 73 | 74 | # Prepare request to Mentat API 75 | url = self.base_url + "api/events/search?submit=Search" 76 | url += "&source_addrs="+key 77 | url += "&limit=" + (str(limit) if limit is not None else "100") 78 | if dt_from: 79 | url += "&dt_from=" + dt_from.strftime("%Y-%m-%d %H:%M:%S") 80 | data = {"api_key": self.api_key} 81 | # Send request 82 | try: 83 | resp = requests.post(url, data) 84 | except Exception as e: 85 | raise GatewayError("Can't get data from Mentat database: " + str(e)) 86 | # Parse response 87 | if resp.status_code == 400 and "search query quota" in resp.text: 88 | raise GatewayError("Search query quota exceeded, try again later.") 89 | try: 90 | result = resp.json()['items'] 91 | except (ValueError, KeyError): 92 | #self.log.error("Invalid data received from Mentat database: req. URL: '{}', req. body: '{}', resp. code: {}, resp. body: '{}'".format(resp.request.url, resp.request.body, resp.status_code, resp.text)) 93 | raise GatewayError("Invalid data received from Mentat database") 94 | 95 | return result 96 | 97 | 98 | def put(self, ideas): 99 | """ 100 | Does nothing. Implemented only for compatibility with other EventDB layers. 101 | 102 | Arguments: 103 | ideas list of IDEA message parsed into Python-native structures 104 | """ 105 | return 106 | -------------------------------------------------------------------------------- /NERDd/modules/event_type_counter.py: -------------------------------------------------------------------------------- 1 | """ 2 | NERD module determines main types of attack going from IP according to given parameters. 3 | """ 4 | 5 | from core.basemodule import NERDModule 6 | 7 | import g 8 | 9 | from datetime import datetime, timedelta 10 | import logging 11 | import os 12 | import re 13 | 14 | date_regex = re.compile("[0-9]{4}-[0-9]{2}-[0-9]{2}") 15 | 16 | class EventTypeCounter(NERDModule): 17 | """ 18 | EventTypeCounter module updates main event types for IP which triggered this module. 19 | Behavior of this module can be parameterized by time period, minimal total 20 | number of events and minimal required percentage per type (threshold) for 21 | event type determination. 22 | 23 | Event flow specification: 24 | [ip] 'events_meta.total' -> count_type() -> 'events_meta.types' 25 | """ 26 | 27 | def __init__(self): 28 | self.log = logging.getLogger("EventTypeCounter") 29 | self.event_days = g.config.get("event_type_counter.days", None) 30 | self.event_threshold = g.config.get("event_type_counter.threshold", 5) 31 | self.event_min = g.config.get("event_type_counter.min_num_of_events", 0) 32 | 33 | g.um.register_handler( 34 | self.count_type, 35 | 'ip', 36 | ('events_meta.total',), 37 | ('events_meta.types',) 38 | ) 39 | 40 | 41 | def count_type(self, ekey, rec, updates): 42 | """ 43 | Counts number of events for each event type for given time period (or counts all 44 | event if time period is not set). 45 | If total number of events is equal or greater than minimal required number of events and 46 | percentage of specific event type is equal or greater than threshold, this specific 47 | event type extends list of main event type for IP which triggered this module. 48 | 49 | Arguments: 50 | ekey -- two-tuple of entity type and key, e.g. ('ip', '192.0.2.42') 51 | rec -- record currently assigned to the key 52 | updates -- list of all attributes whose update triggered this call and 53 | their new values (or events and their parameters) as a list of 54 | 2-tuples: [(attr, val), (!event, param), ...] 55 | 56 | Return: 57 | List of update requests. 58 | """ 59 | 60 | etype, key = ekey 61 | if etype != 'ip': 62 | return None 63 | 64 | if 'events' not in rec: 65 | return None # No Warden event, nothing to do 66 | 67 | ret = [] 68 | total_events = 0 69 | types = {} # Map: event_type -> count 70 | 71 | # Count number of events per type in last "event_days" days 72 | if self.event_days is None: 73 | minday = None 74 | else: 75 | minday = datetime.utcnow().date() - timedelta(days=self.event_days) 76 | # Let's compare days as strings - it works thanks to ISO format 77 | # and it's faster than to convert all string keys in DB to datetime 78 | minday = minday.strftime("%Y-%m-%d") 79 | for evtrec in rec['events']: 80 | if minday and evtrec['date'] < minday: 81 | continue 82 | cat = evtrec['cat'] 83 | n = evtrec['n'] 84 | total_events += n 85 | if cat not in types: 86 | types[cat] = n 87 | else: 88 | types[cat] += n 89 | 90 | if total_events < self.event_min: 91 | # if self.event_days is not None: 92 | # self.log.debug("In last {} days only {} events happened for IP {} (minimal number of events for classification is {}).".format(self.event_days, total_events, key, self.event_min)) 93 | # else: 94 | # self.log.debug("Only {} events happened for IP {} (minimal number of events for classification is {}).".format(total_events, key, self.event_min)) 95 | return [('set', 'events_meta.types', [])] 96 | 97 | for event_type in types: 98 | if (types[event_type]/total_events*100) >= self.event_threshold: 99 | # self.log.debug("Event type {} exceed {}% threshold for IP {} ({} events from {}).".format(event_type, self.event_threshold, key, types[event_type], total_events)) 100 | ret.append(event_type) 101 | # else: 102 | # self.log.debug("Event type {} doesn't exceed {}% threshold for IP {} ({} events from {}).".format(event_type, self.event_threshold, key, types[event_type], total_events)) 103 | 104 | return [('set', 'events_meta.types', ret)] 105 | -------------------------------------------------------------------------------- /scripts/fix_ref_cnt.js: -------------------------------------------------------------------------------- 1 | // This script recomputes all references between deifferent entities in NERD 2 | // database and set "_ref_cnt" fields accordingly. 3 | // If some record has no reference (an inconsistency which shouldn't normally 4 | // happen) it's removed. 5 | // 6 | // IMPORTANT: Stop NERDd before running the script! Database must not be 7 | // changed while the script is running. 8 | // 9 | // Note: JavaScript (and MongoShell) treats all numbers as floats. To store int 10 | // it must be written as NumberInt(0). 11 | 12 | 13 | // ** ip -> bgppref ** 14 | // Set _ref_cnt in "bgppref" records to number of IPs pointing to it from "ip" records 15 | // Reset _ref_cnt to 0 in all records 16 | db.bgppref.update({}, {$set: {_ref_cnt: NumberInt(0)}}, {multi: true}); 17 | // Count references and set _ref_cnt 18 | db.ip.aggregate([ 19 | {$match: {bgppref: {$exists: true}}}, 20 | {$project: {_id: 1, bgppref: 1}}, 21 | {$group: {_id: "$bgppref", cnt: {$sum: NumberInt(1)}}} 22 | ]).forEach( function(x) { 23 | db.bgppref.update({_id: x._id}, {$set: {_ref_cnt: NumberInt(x.cnt)}}) 24 | }); 25 | // Delete records with _ref_cnt = 0 (shouldn't normally happen) 26 | res = db.bgppref.remove({_ref_cnt: 0}); 27 | if (res["nRemoved"] > 0) { 28 | print("NOTICE: Removed " + res["nRemoved"] + " 'bgppref' records with no reference."); 29 | } 30 | 31 | 32 | // ** ip -> ipblock ** 33 | // Set _ref_cnt in "ipblock" records to number of IPs pointing to it from "ip" records 34 | // Reset _ref_cnt to 0 in all records 35 | db.ipblock.update({}, {$set: {_ref_cnt: NumberInt(0)}}, {multi: true}); 36 | // Count references and set _ref_cnt 37 | db.ip.aggregate([ 38 | {$match: {ipblock: {$exists: true}}}, 39 | {$project: {_id: 1, ipblock: 1}}, 40 | {$group: {_id: "$ipblock", cnt: {$sum: NumberInt(1)}}} 41 | ]).forEach( function(x) { 42 | db.ipblock.update({_id: x._id}, {$set: {_ref_cnt: NumberInt(x.cnt)}}) 43 | }); 44 | // Delete records with _ref_cnt = 0 (shouldn't normally happen) 45 | res = db.ipblock.remove({_ref_cnt: 0}); 46 | if (res["nRemoved"] > 0) { 47 | print("NOTICE: Removed " + res["nRemoved"] + " 'ipblock' records with no reference."); 48 | } 49 | 50 | 51 | // ** bgppref <-> asn ** 52 | // Reset array of pointers in "asn" to [] 53 | db.asn.update({}, {$set: {bgppref: []}}, {multi: true}); 54 | // For each "asn", set its list of pointers to all "bgpprefs" which point to the "asn" 55 | db.bgppref.aggregate([ 56 | {$match: {asn: {$exists: true}}}, 57 | {$project: {_id: 1, asn: 1}}, 58 | {$unwind: "$asn"}, 59 | {$group: {_id: "$asn", bgppref: {$push: "$_id"}}} 60 | ]).forEach( function(x) { 61 | db.asn.update({_id: x._id}, {$set: {bgppref: x.bgppref}}) 62 | }); 63 | // Delete records with empty list of pointers (shouldn't normally happen) 64 | res = db.asn.remove({bgppref: {$size: 0}}); 65 | if (res["nRemoved"] > 0) { 66 | print("NOTICE: Removed " + res["nRemoved"] + " 'asn' records with no reference."); 67 | } 68 | 69 | // Reset array of pointers in "bgppref" to [] 70 | db.bgppref.update({}, {$set: {asn: []}}, {multi: true}); 71 | // For each "bgppref", set its list of pointers to all "asns" which point to the "bgppref" 72 | db.asn.aggregate([ 73 | {$match: {bgppref: {$exists: true}}}, 74 | {$project: {_id: 1, bgppref: 1}}, 75 | {$unwind: "$bgppref"}, 76 | {$group: {_id: "$bgppref", asn: {$push: "$_id"}}} 77 | ]).forEach( function(x) { 78 | db.bgppref.update({_id: x._id}, {$set: {asn: x.asn.map(n => NumberInt(n))}}) 79 | }); 80 | // Delete records with empty list of pointers (shouldn't normally happen) 81 | res = db.bgppref.remove({asn: {$size: 0}}); 82 | if (res["nRemoved"] > 0) { 83 | print("NOTICE: Removed " + res["nRemoved"] + " 'bgppref' records with no reference."); 84 | } 85 | 86 | 87 | // ** asn/ipblock -> org ** 88 | // Set _ref_cnt in "org" records to number of IPs pointing to it from "asn" and "ipblock" records 89 | // Reset _ref_cnt to 0 in all records 90 | db.org.update({}, {$set: {_ref_cnt: NumberInt(0)}}, {multi: true}); 91 | db.asn.aggregate([ 92 | {$match: {org: {$exists: true}}}, 93 | {$project: {_id: 1, org: 1}}, 94 | {$group: {_id: "$org", cnt: {$sum: NumberInt(1)}}} 95 | ]).forEach( function(x) { 96 | db.org.update({_id: x._id}, {$inc: {_ref_cnt: NumberInt(x.cnt)}}) 97 | }); 98 | db.ipblock.aggregate([ 99 | {$match: {org: {$exists: true}}}, 100 | {$project: {_id: 1, org: 1}}, 101 | {$group: {_id: "$org", cnt: {$sum: NumberInt(1)}}} 102 | ]).forEach( function(x) { 103 | db.org.update({_id: x._id}, {$inc: {_ref_cnt: NumberInt(x.cnt)}}) 104 | }); 105 | // Delete records with _ref_cnt = 0 (shouldn't normally happen) 106 | res = db.org.remove({_ref_cnt: 0}); 107 | if (res["nRemoved"] > 0) { 108 | print("NOTICE: Removed " + res["nRemoved"] + " 'org' records with no reference."); 109 | } 110 | -------------------------------------------------------------------------------- /NERDd/modules/hostname.py: -------------------------------------------------------------------------------- 1 | """ 2 | Module classifies type of service associated to given IP according to hostname. 3 | """ 4 | 5 | from core.basemodule import NERDModule 6 | 7 | import requests 8 | import re 9 | 10 | import g 11 | 12 | import datetime 13 | import logging 14 | import os 15 | 16 | class HostnameClass(NERDModule): 17 | """ 18 | HostnameClass module. 19 | Classifies IP according to hostname and given list of known domains and regular expressions 20 | 21 | Event flow specification: 22 | [ip] 'hostname' -> hostname_classify() -> 'hostname_class' 23 | """ 24 | 25 | def __init__(self): 26 | self.log = logging.getLogger("hostname_class") 27 | #self.log.setLevel("DEBUG") 28 | self.regex_hostname = g.config.get("hostname_tagging.regex_tagging", []) 29 | self.regex_ip_in_hostname = g.config.get("hostname_tagging.regex_tagging_ip_in_hostname", []) 30 | self.regex_hostname = [(re.compile(regex, flags=re.ASCII), tag) for regex,tag in self.regex_hostname] 31 | self.known_domains = self.convert_domain_list_to_dict(g.config.get("hostname_tagging.known_domains", [])) 32 | 33 | g.um.register_handler( 34 | self.hostname_classify, 35 | 'ip', 36 | ('hostname','!refresh_hostname_class'), # !refresh_hostname_class is called only manually, e.g. after change of tag configuration 37 | ('hostname_class',) 38 | ) 39 | 40 | def convert_domain_list_to_dict(self, domain_list): 41 | """ 42 | Converts list of domains and types of service to dictionary because of better time complexity of search operation. 43 | 44 | Arguments: 45 | domain_list -- list of lists of known domains and their type of service 46 | 47 | Return: 48 | Dicitonary with known domain as a key and type of service as a value 49 | """ 50 | 51 | ret = {} 52 | for domain in domain_list: 53 | ret[domain[0]] = domain[1] 54 | return ret 55 | 56 | def hostname_classify(self, ekey, rec, updates): 57 | """ 58 | Searches each hostname portion in known domain dictionary and sets type of service if match found. 59 | Tries to match hostname with regular expression and sets type of service if matches. 60 | 61 | Arguments: 62 | ekey -- two-tuple of entity type and key, e.g. ('ip', '192.0.2.42') 63 | rec -- record currently assigned to the key 64 | updates -- list of all attributes whose update triggered this call and 65 | their new values (or events and their parameters) as a list of 66 | 2-tuples: [(attr, val), (!event, param), ...] 67 | 68 | Returns: 69 | List of update requests. 70 | """ 71 | 72 | etype, key = ekey 73 | if etype != 'ip': 74 | return None 75 | 76 | hostname = rec["hostname"] 77 | 78 | if hostname is None: 79 | self.log.debug("Hostname attribute is not filled for IP ({}).".format(key)) 80 | return None 81 | 82 | tags = [] 83 | 84 | dot_count = hostname.count(".") 85 | 86 | for i in range(0,dot_count): 87 | portion = hostname.split(".",i)[-1] 88 | if portion in self.known_domains: 89 | tag = self.known_domains[portion] 90 | self.log.debug("Hostname ({}) ends with domain {} and has been classified as {}.".format(hostname, portion, tag)) 91 | if tag not in tags: 92 | tags.append(tag) 93 | break 94 | 95 | # second simple implementation of ip in host check 96 | ip_in_host_prob = 0 97 | for octet in set(key.split(".")): 98 | if octet in hostname: 99 | ip_in_host_prob += 25 100 | if ip_in_host_prob >= 50: 101 | tags.append("ip_in_hostname") 102 | 103 | for regex in self.regex_hostname: 104 | search = regex[0].search(hostname) 105 | if search: 106 | tag = regex[1] 107 | if tag == "ip_in_hostname": 108 | # check if captured ip address is really ip address of this hostname 109 | ip_adress_matched = all([group in key for group in search.groups() if group]) 110 | if not ip_adress_matched: 111 | continue 112 | 113 | self.log.debug("Hostname ({}) matches regex {} and has been classified as {}.".format(hostname, regex[0].pattern, tag)) 114 | if tag not in tags: 115 | tags.append(tag) 116 | 117 | if tags: 118 | return [('set', 'hostname_class', tags)] 119 | else: 120 | # If hostname_class existed previously but no rule matches now, remove the key 121 | return [('remove', 'hostname_class')] 122 | -------------------------------------------------------------------------------- /NERDd/modules/event_counter.py: -------------------------------------------------------------------------------- 1 | """ 2 | NERD module summarizing number of events in last day, week and month (30 days). 3 | 4 | Should be triggered at least once a day for every address. 5 | """ 6 | 7 | from core.basemodule import NERDModule 8 | import g 9 | 10 | import datetime 11 | 12 | EWMA_ALPHA = 0.25 # a parameter (there's no strong reason for the value selected, I just feel that 0.25 gives reasonable weights for the 7 day long period) 13 | EWMA_WEIGHTS = [(EWMA_ALPHA * (1 - EWMA_ALPHA)**i) for i in range(7)] 14 | 15 | # TODO - (re)compute sets of nodes for 1, 7 and 30 days as well (or do it in frontend?) 16 | 17 | class EventCounter(NERDModule): 18 | """ 19 | Module counting number of recent events. 20 | 21 | Periodically updates number of events in the last 1, 7 and 30 days. 22 | 23 | It's always the number of events in the current day (since 00:00 UTC) plus 24 | number of events in 1, 7 or 30 previous days. 25 | Therefore, "total1" contains number of events in previous 24 to 48 hours, 26 | depending on the time this function was triggered. 27 | 28 | The !refresh_event_count should be triggered when no event was added to 29 | the IP for more than 24 hours. 30 | 31 | Event flow specification: 32 | events.total -> count_events -> events_meta.{total1,total7,total30} 33 | !every1d -> count_events -> events_meta.{total1,total7,total30} 34 | 35 | # TODO: for now this hooks on events.total which is updated by event_receiver on every new event 36 | since it can't be hooked on events.