├── README.md ├── elastic-demo ├── README.md ├── export-filebeat-misp.ndjson └── export-misp.ndjson ├── misp-restore-botvrij.sh └── provision-demo ├── .gitignore ├── clear-elastic.sh ├── import-event.py ├── import-loop-events.sh └── provision.py /README.md: -------------------------------------------------------------------------------- 1 | # misp-training-environment 2 | 3 | Goal: have a training environment where users can be Site and Org admin, alter MISP settings and play around with the MISP features. Restore the "playground" on a daily basis, without user interaction. 4 | 5 | ## MISP-cloud 6 | 7 | The training environment is based on [MISP-cloud](https://github.com/MISP/misp-cloud) and the OSINT feed from [botvrij.eu](https://botvrij.eu/data/feed-osint/) 8 | 9 | ## Approach 10 | 11 | 1. Install MISP-cloud (AWS) 12 | 1. Change basic credentials and API keys. Run system update. 13 | 1. Change MISP URL and basic settings. 14 | 1. Change auditing features of MISP to include client_ips in the logs 15 | 1. Update rsyslog to seperate misp logs from 'normal' syslog 16 | 1. Configure the OSINT feed 17 | 1. Add demo users and demo organization 18 | 1. Truncate the logs (via mysql) 19 | 1. Create a backup with misp-backup 20 | 1. Create a cron job to restore the backup daily 21 | 1. Ship logs to external log collector 22 | 1. Enable Cloudflare 23 | 24 | The restore resets all changes done by the users in the demo environment, including MISP system configuration changes. It **does not** restore the users or remove new created users/organisations. This allows demo users to keep access with their newly created users (and API keys), after the restore has happened. Cleanup of demo users (and organisations) is a manual action. 25 | 26 | The script misp-restore-botvrij.sh is slightly altered compared to the original misp-restore. If does not require user-input (it can run from cron) and contains the Mysql root username. 27 | 28 | ## Cron-job 29 | 30 | 30 6 * * * root /var/www/MISP/tools/misp-backup/misp-restore-botvrij.sh /opt/mispbackups/MISP-Backup-clean.tar.gz 31 | 32 | # MISP and a webshell via web interface 33 | 34 | Install **ShellInABox** and edit the config file. 35 | 36 | ``` 37 | sudo apt install openssl shellinabox 38 | sudo vi /etc/default/shellinabox 39 | ``` 40 | 41 | Add these settings to have ShellInABox listen on the localhost and connect to a local system. Replace `192.168.171.10` with the IP addresss of the host you'd like to connect to. 42 | 43 | ``` 44 | SHELLINABOX_ARGS="--no-beep --localhost-only" 45 | OPTS="-s /:SSH:192.168.171.10" 46 | ``` 47 | 48 | Stop Apache and install **nginx** to be used as a reverse proxy and add a config file. Nginx will use the SSL password created for MISP. 49 | 50 | ``` 51 | sudo systemctl stop apache2 52 | sudo apt-get install nginx 53 | sudo /etc/nginx/sites-enabled/misp-template.conf 54 | ``` 55 | 56 | ``` 57 | server { 58 | listen 80; 59 | return 301 https://$host$request_uri; 60 | } 61 | 62 | server{ 63 | 64 | listen 443 ssl; 65 | ssl_certificate /etc/ssl/private/misp.local.crt; 66 | ssl_certificate_key /etc/ssl/private/misp.local.key; 67 | 68 | location / 69 | { 70 | proxy_pass https://127.0.0.1:8443/; 71 | proxy_redirect default; 72 | proxy_set_header Host $host; 73 | proxy_set_header X-Real-IP $remote_addr; 74 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 75 | client_max_body_size 10m; 76 | client_body_buffer_size 128k; 77 | proxy_connect_timeout 90; 78 | proxy_send_timeout 90; 79 | proxy_read_timeout 90; 80 | proxy_ssl_verify off; 81 | proxy_buffer_size 4k; 82 | proxy_buffers 4 32k; 83 | proxy_busy_buffers_size 64k; 84 | proxy_temp_file_write_size 64k; 85 | } 86 | location /shell/ 87 | { 88 | proxy_pass https://127.0.0.1:4200/; 89 | proxy_redirect default; 90 | proxy_set_header Host $host; 91 | proxy_set_header X-Real-IP $remote_addr; 92 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 93 | client_max_body_size 10m; 94 | client_body_buffer_size 128k; 95 | proxy_connect_timeout 90; 96 | proxy_send_timeout 90; 97 | proxy_read_timeout 90; 98 | proxy_ssl_verify off; 99 | proxy_buffer_size 4k; 100 | proxy_buffers 4 32k; 101 | proxy_busy_buffers_size 64k; 102 | proxy_temp_file_write_size 64k; 103 | } 104 | } 105 | ``` 106 | 107 | Then make sure you configure Apache to listen on 8443 and restart the web server. 108 | 109 | ``` 110 | vi /etc/apache2/ports.conf 111 | ``` 112 | 113 | ``` 114 | Listen 8080 115 | 116 | 117 | Listen 8443 118 | 119 | 120 | 121 | Listen 8443 122 | 123 | ``` 124 | 125 | ``` 126 | vi /etc/apache2/sites-enabled/misp-ssl.conf 127 | ``` 128 | 129 | ``` 130 | 131 | ``` 132 | 133 | -------------------------------------------------------------------------------- /elastic-demo/README.md: -------------------------------------------------------------------------------- 1 | Kibana Saved objects for Elastic demo 2 | -------------------------------------------------------------------------------- /misp-restore-botvrij.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # 3 | # 4 | # Inspired from daverstephens@gmail.com and @alexanderjaeger work on misp-backup.sh 5 | # 6 | # Apache, MISP and mariaDB/MySQL should be installed before running this script 7 | # 8 | # Only works with a database on localhost 9 | # 10 | # Add the MYSQL ROOT password!!!! 11 | # 12 | # 13 | 14 | # This makes use of the standard variables used by the installer 15 | eval "$(curl -fsSL https://raw.githubusercontent.com/MISP/MISP/2.4/docs/generic/globalVariables.md | grep -v \`\`\`)" 16 | MISPvars > /dev/null 2>&1 17 | 18 | # Leave empty for NO debug messages, if run with set -x or bash -x it will enable DEBUG by default 19 | DEBUG= 20 | 21 | MySQLRPassword="CHANGEME" 22 | 23 | case "$-" in 24 | *x*) NO_PROGRESS=1; DEBUG=1 ;; 25 | *) NO_PROGRESS=0 ;; 26 | esac 27 | 28 | ## Functions 29 | 30 | # Dynamic horizontal spacer 31 | space () { 32 | if [[ "$NO_PROGRESS" == "1" ]]; then 33 | return 34 | fi 35 | # Check terminal width 36 | num=`tput cols` 37 | for i in `seq 1 $num`; do 38 | echo -n "-" 39 | done 40 | echo "" 41 | } 42 | 43 | # Make sure the target has enough free space 44 | checkDiskFree () { 45 | if [[ ! -e $1 ]]; then 46 | echo "$1 does not exist, creating" 47 | mkdir -p $1 48 | fi 49 | threshhold=90 50 | free=$(df -l --sync --output=pcent $1 |tail -1|cut -f 1 -d% | tr -d \ ) 51 | if [[ "$free" > "$threshhold" ]]; then 52 | space 53 | echo "Your destination folder is $threshhold% full." 54 | space 55 | exit 1 56 | fi 57 | } 58 | 59 | # Check if variable is empty 60 | checkVar () { 61 | [[ -z $1 ]] && echo "$1 is empty, please investigate." && exit 1 62 | } 63 | 64 | # Small function to import SQL export 65 | mysqlImport () { 66 | myDB=$(mysqlshow |grep -o $MISPDB) 67 | if [[ "$myDB" == "$MISPDB" ]]; then 68 | TmpUsersSql="$(mktemp --tmpdir=$OutputDirName)" 69 | echo -n "$myDB detected" 70 | REMOVE="y" 71 | # Export Users table 72 | mysqldump --opt --host $MISPDBHost -u $MySQLRUser -p$MySQLRPassword $myDB users > $TmpUsersSql 73 | [[ "$REMOVE" == "y" ]] && mysql -h $MISPDBHost -u $MySQLRUser -p$MySQLRPassword -e "DROP DATABASE IF EXISTS $myDB" 74 | 75 | mysql -h $MISPDBHost -u $MySQLRUser -p$MySQLRPassword -Bse "CREATE DATABASE IF NOT EXISTS $MISPDB ;\ 76 | GRANT USAGE ON *.* TO '$MySQLUUser'@'localhost' IDENTIFIED BY '$MySQLUPass';\ 77 | GRANT ALL PRIVILEGES ON $MISPDB.* TO '$MySQLUUser'@'localhost' ; \ 78 | FLUSH PRIVILEGES;" 79 | 80 | mysql -s -h $MISPDBHost -u $MySQLUUser -p$MySQLUPass $MISPDB < $BackupDir/MISPbackupfile.sql 81 | 82 | mysql -s -h $MISPDBHost -u $MySQLRUser -p$MySQLRPassword -D $MISPDB < $TmpUsersSql 83 | rm $TmpUsersSql 84 | fi 85 | } 86 | echo '-- Starting MISP restore process' 87 | 88 | FILE=./misp-backup.conf 89 | 90 | # Extract base directory where this script is and cd into it 91 | cd "${0%/*}" 92 | 93 | # Set to the current webroot owner 94 | WWW_USER=$(ls -l $0 |awk {'print $3'}|tail -1) 95 | 96 | MySQLRUser="root" 97 | 98 | # In most cases the owner of the cake script is also the user as which it should be executed. 99 | if [[ "$USER" != "$WWW_USER" ]]; then 100 | echo "You run this script as $USER and the owner of the backup script is $WWW_USER, this should be your web server user. FYI." 101 | fi 102 | 103 | # Check if run as root 104 | if [[ "$EUID" != "0" ]]; then 105 | echo "Please run the backup script as root" 106 | exit 1 107 | fi 108 | 109 | if [ ! -z $1 ] && [ -f $1 ]; then 110 | BackupFile=$1 111 | else 112 | echo 'Specify backup file by running ./misp-restore.sh PATH_TO_ARCHIVE.tar.gz' 113 | exit 1 114 | fi 115 | 116 | # Source configuration file 117 | if [ -f $FILE ]; then 118 | echo "--- File $(pwd)$FILE exists." 119 | . $FILE 120 | else 121 | echo "--- Config File $FILE does not exist. Please enter values manually" 122 | echo -n '--- Where would you like to decompress backup files (Eg. /tmp)? ' 123 | read OutputDirName 124 | fi 125 | 126 | checkDiskFree OutputDirName 127 | 128 | # Decompress archive 129 | BackupDir=$OutputDirName/$(basename -s ".tar.gz" $BackupFile) 130 | mkdir $BackupDir 131 | echo '--- Decompressing files' 132 | tar zxpf $BackupFile -C $BackupDir 133 | 134 | # Fill in any missing values with defaults 135 | # MISP path detector 136 | if [[ -z $UNATTENDED ]]; then 137 | if [[ -z $PATH_TO_MISP ]]; then 138 | if [[ "$(locate > /dev/null 2> /dev/null ; echo $?)" != "127" ]]; then 139 | if [[ "$(locate MISP/app/webroot/index.php |wc -l)" > 1 ]]; then 140 | echo "We located more then 1 MISP/app/webroot, reverting to manual" 141 | PATH_TO_MISP=${PATH_TO_MISP:-$(locate MISP/app/webroot/index.php|sed 's/\/app\/webroot\/index\.php//')} 142 | echo -n 'Please enter the base path of your MISP install (e.g /var/www/MISP): ' 143 | read PATH_TO_MISP 144 | fi 145 | fi 146 | fi 147 | 148 | if [[ -d $PATH_TO_MISP ]] && [[ -f $PATH_TO_MISP/VERSION.json ]]; then 149 | echo "$PATH_TO_MISP exists and seems to include an existing install." 150 | #echo "We can move it out of the way, or leave as is." 151 | #echo -n "Move $PATH_TO_MISP to $OutputDirName/MISP-$(date +%Y%m%d)?" 152 | MOVE="n" 153 | #read MOVE 154 | [[ "$MOVE" == "y" ]] && mv $PATH_TO_MISP $OutputDirName/MISP-$(date +%Y%m%d) && mkdir $PATH_TO_MISP 155 | fi 156 | fi 157 | 158 | # database.php 159 | MySQLUUser=$(grep -o -P "(?<='login' => ').*(?=')" $BackupDir/Config/database.php) ; checkVar MySQLUUser 160 | MySQLUPass=$(grep -o -P "(?<='password' => ').*(?=')" $BackupDir/Config/database.php) ; checkVar MySQLUPass 161 | MISPDB=$(grep -o -P "(?<='database' => ').*(?=')" $BackupDir/Config/database.php) ; checkVar MISPDB 162 | DB_Port=$(grep -o -P "(?<='port' => ).*(?=,)" $BackupDir/Config/database.php) ; checkVar DB_Port 163 | MISPDBHost=$(grep -o -P "(?<='host' => ').*(?=')" $BackupDir/Config/database.php) ; checkVar MISPDBHost 164 | 165 | # config.php 166 | Salt=$(grep -o -P "(?<='salt' => ').*(?=')" $BackupDir/Config/config.php) ; checkVar Salt 167 | BaseURL=$(grep -o -P "(?<='baseurl' => ').*(?=')" $BackupDir/Config/config.php) # BaseURL can be empty 168 | OrgName=$(grep -o -P "(?<='org' => ').*(?=')" $BackupDir/Config/config.php) ; checkVar OrgName 169 | LogEmail=$(grep -o -P "(?<='email' => ').*(?=')" $BackupDir/Config/config.php|head -1) ; checkVar LogEmail 170 | AdminEmail=$(grep -o -P "(?<='contact' => ').*(?=')" $BackupDir/Config/config.php) ; checkVar AdminEmail 171 | GnuPGEmail=$(sed -n -e '/GnuPG/,$p' $BackupDir/Config/config.php|grep -o -P "(?<='email' => ').*(?=')") ; checkVar GnuPGEmail 172 | GnuPGHomeDir=$(grep -o -P "(?<='homedir' => ').*(?=')" $BackupDir/Config/config.php) ; checkVar GnuPGHomeDir 173 | GnuPGPass=$(grep -o -P "(?<='password' => ').*(?=')" $BackupDir/Config/config.php) ; checkVar GnuPGPass 174 | 175 | if [[ -f /tmp/MISPbackupfile.sql ]]; then 176 | mysqlImport 177 | rm /tmp/MISPbackupfile.sql 178 | exit 179 | fi 180 | 181 | # Restore backup files 182 | echo "--- Copy of GnuPG files" 183 | mkdir -p $GnuPGHomeDir 184 | cp $BackupDir/*.gpg $GnuPGHomeDir/ 185 | cp $BackupDir/random_seed $GnuPGHomeDir/ 186 | 187 | 188 | echo "--- Copy of org and images and files" 189 | cp -pr $BackupDir/orgs $PATH_TO_MISP/app/webroot/img/ 190 | cp -pr $BackupDir/custom $PATH_TO_MISP/app/webroot/img/ 191 | cp -pr $BackupDir/files $PATH_TO_MISP/app/ 192 | 193 | 194 | # Restore MISP Config files 195 | echo "--- Copy of app/Config files" 196 | cp -p $BackupDir/Config/bootstrap.php $PATH_TO_MISP/app/Config 197 | cp -p $BackupDir/Config/config.php $PATH_TO_MISP/app/Config 198 | cp -p $BackupDir/Config/core.php $PATH_TO_MISP/app/Config 199 | cp -p $BackupDir/Config/database.php $PATH_TO_MISP/app/Config 200 | 201 | # Permissions 202 | echo "--- Setting persmissions" 203 | chown -R $WWW_USER:$WWW_USER /var/www/MISP 204 | chmod -R 750 /var/www/MISP 205 | chmod -R g+ws /var/www/MISP/app/tmp 206 | chmod -R g+ws /var/www/MISP/app/files 207 | chmod -R g+ws /var/www/MISP/app/files/scripts/tmp 208 | 209 | 210 | 211 | # Restore DB 212 | echo "--- Starting MySQL database Restore" 213 | #echo -n '---- Please enter your MySQL root account username: ' 214 | MySQLRUser="root" 215 | #read MySQLRUser 216 | 217 | echo "---- Creating database and user if not exists with data found in MISP configuration files" 218 | mysqlImport 219 | 220 | ##if [[ "$returnCode" != "0" ]]; then 221 | ## echo "Something went wrong, MySQL returned: $returnCode" 222 | ## echo "The MISPbackupfile.sql will be copied to: /tmp for you to handle this situation manually." 223 | ## echo "You can also run this script again and we will retry to import the SQL." 224 | ## cp $BackupDir/MISPbackupfile.sql /tmp > /dev/null 2> /dev/null 225 | ## rm -rf $BackupDir 226 | ## exit $returnCode 227 | ##fi 228 | 229 | 230 | rm -rf $BackupDir 231 | 232 | if [[ ! -z $BaseURL ]]; then 233 | echo "-- MISP Restore Complete! URL: $BaseURL" 234 | else 235 | echo "-- MISP Restore Complete!" 236 | fi 237 | -------------------------------------------------------------------------------- /provision-demo/.gitignore: -------------------------------------------------------------------------------- 1 | misp-events/* 2 | -------------------------------------------------------------------------------- /provision-demo/clear-elastic.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | curl -XPOST "http://127.0.0.1:9200/filebeat-*/_delete_by_query?conflicts=proceed" -H 'Content-Type: application/json' -d'{ "query": { "match_all": {} }}' 4 | -------------------------------------------------------------------------------- /provision-demo/import-event.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import sys 4 | sys.path.insert(0, "/var/www/MISP/PyMISP/examples/") 5 | from pymisp import PyMISP, MISPAttribute, MISPEvent, MISPEventBlocklist, ExpandedPyMISP 6 | from keys import misp_url, misp_key, misp_verifycert, misp_client_cert 7 | import json 8 | import argparse 9 | import re 10 | import time 11 | 12 | if __name__ == '__main__': 13 | parser = argparse.ArgumentParser(description='Import events from json file.') 14 | parser.add_argument("-i", "--input", required=True, help="Input file") 15 | args = parser.parse_args() 16 | 17 | misp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert, debug=False, proxies=False) 18 | subst = "\"timestamp\": \"{ts}\"".format(ts=int(time.time())) 19 | regex = r"\"timestamp\": \"[0-9]{10}\"" 20 | subst_publish = "\"publish_timestamp\": \"{ts}\"".format(ts=int(time.time())) 21 | regex_publish = r"\"publish_timestamp\": \"[0-9]{10}\"" 22 | if args.input: 23 | with open(args.input, 'r') as f: 24 | for e in f: 25 | event = json.loads(e)['response'][0] 26 | eventid = event["Event"]["id"] 27 | eventuuid = event["Event"]["uuid"] 28 | eventinfo = event["Event"]["info"] 29 | 30 | # Delete event and remove it from the blocklist 31 | misp.delete_event(eventuuid) 32 | misp.delete_event_blocklist(eventuuid) 33 | 34 | # Fetch the new event, replace timestamps and push the event 35 | res = json.dumps(event) 36 | res = re.sub(regex, subst, res, 0, re.MULTILINE) 37 | res = re.sub(regex_publish, subst_publish, res, 0, re.MULTILINE) 38 | res = misp.add_event(res) 39 | print("----------------------------------------") 40 | if 'errors' in res: 41 | print(res) 42 | else: 43 | print("Processed {eventid} - {eventinfo}".format(eventid=eventid,eventinfo=eventinfo)) 44 | print("----------------------------------------") 45 | -------------------------------------------------------------------------------- /provision-demo/import-loop-events.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | PAUSE_BETWEEN_EVENTS=1m 4 | BASE_PATH="/home/ubuntu/provision-demo" 5 | 6 | echo "Remember to run this script through screen!" 7 | echo "-------------------------------------------" 8 | echo " " 9 | 10 | for FILE in $BASE_PATH/misp-events/*.json ; do date && echo Process $FILE && $BASE_PATH/import-event.py -i $FILE && sleep $PAUSE_BETWEEN_EVENTS; done 11 | 12 | -------------------------------------------------------------------------------- /provision-demo/provision.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import sys 4 | sys.path.insert(0, "/var/www/MISP/PyMISP/examples/") 5 | from pymisp import ExpandedPyMISP, MISPUser, MISPOrganisation 6 | from keys import misp_url, misp_key, misp_verifycert, misp_client_cert 7 | 8 | max_users = 10 9 | 10 | default_password = "{user}" 11 | default_siteadmin = "admin-{user}@DEMO" 12 | default_user = "user-{user}@DEMO" 13 | org_name = "training - DEMO" 14 | 15 | api = ExpandedPyMISP(misp_url, misp_key, misp_verifycert) 16 | 17 | def provision(): 18 | # Add demo organisation 19 | org = MISPOrganisation() 20 | org.name = org_name 21 | org.nationality = "Belgium" 22 | org.sector = "IT" 23 | org.type = "Training organisation" 24 | org.local = 1 25 | 26 | #add_org = api.add_organisation(org, pythonify=True) 27 | #orgid = add_org.id 28 | orgid = 11 29 | 30 | # Add demo users 31 | current_user = 1 32 | while current_user <= max_users: 33 | user = MISPUser() 34 | user.email = default_siteadmin.format(user=current_user) 35 | user.password = default_password.format(user=current_user) 36 | user.enable_password = 1 37 | user.change_pw = 0 38 | user.org_id = orgid 39 | user.role_id = 1 # Site admin 40 | add_user = api.add_user(user, pythonify=True) 41 | print("Add user %s" % user.email) 42 | 43 | user = MISPUser() 44 | user.email = default_user.format(user=current_user) 45 | user.password = default_password.format(user=current_user) 46 | user.enable_password = 1 47 | user.change_pw = 0 48 | user.org_id = orgid 49 | user.role_id = 3 # User 50 | add_user = api.add_user(user, pythonify=True) 51 | print("Add user %s" % user.email) 52 | 53 | current_user += 1 54 | 55 | def unprovision(): 56 | users = api.users() 57 | current_user = 1 58 | while current_user <= max_users: 59 | for el in users: 60 | obj = el['User'] 61 | if obj['email'] == default_siteadmin.format(user=current_user): 62 | user = MISPUser() 63 | user.id = obj['id'] 64 | api.delete_user(user) 65 | print("delete %s" % obj['email']) 66 | elif obj['email'] == default_user.format(user=current_user): 67 | user = MISPUser() 68 | user.id = obj['id'] 69 | api.delete_user(user) 70 | print("delete %s" % obj['email']) 71 | current_user +=1 72 | 73 | 74 | #provision() 75 | #unprovision() 76 | --------------------------------------------------------------------------------