├── .gitignore ├── README.md ├── bro ├── README.md ├── day_1 │ ├── AdvancedLogging0.md │ ├── AdvancedLoggingLogstash.md │ ├── AdvancedLoggingRsyslog.md │ ├── BuildFromSource.md │ ├── Config.md │ ├── LoadScript.md │ ├── Scripting.md │ ├── abox │ │ └── Vagrantfile │ ├── blacklists.bro │ ├── bro.json │ └── bro2.json ├── day_2 │ ├── ClusterAddNode.md │ ├── ClusterBasics.md │ ├── ClusterConf.md │ ├── Cron.md │ ├── Manager.md │ ├── boxes │ │ ├── Vagrantfile │ │ ├── ripid.txt │ │ └── scripts │ ├── broctl-setcap.md │ ├── cheat4addingworker.md │ └── fixdeceasedmaster.md ├── day_3 │ ├── BasicConf.md │ ├── LoadPcaps.md │ ├── README.md │ ├── csv2elasticsearch.md │ ├── nosweat.png │ ├── scripts2cluster.md │ └── threadHack.sh ├── day_intro │ ├── AptGetInstall.md │ ├── BTIGEK-easybutton-singlehost.sh │ ├── BasicConf.md │ ├── BasicLogging.md │ ├── BasicScripting.md │ ├── README.md │ ├── brodello │ │ ├── Vagrantfile │ │ └── grafana.tar.gz │ ├── dummy │ │ ├── Vagrantfile │ │ └── scripts │ │ │ └── dummy-install-bro.sh │ ├── empty │ │ ├── Vagrantfile │ │ └── scripts │ │ │ └── dummy-install-bro.sh │ ├── timing.md │ └── timing.sh ├── day_last │ └── README.md └── notes.txt ├── common ├── AnyInstall.txt ├── Elastic.md ├── ElasticInstall.txt ├── ElasticsearchAttachemnt.md ├── Kibana4.md ├── MertixInstall.txt ├── Telegraf.md ├── git-most.md ├── metrics ├── pfring.md ├── rsyslogBasics.md └── vagrant │ ├── CreateUbu14Box.bash │ └── bb │ ├── Vagrantfile │ ├── keys │ ├── vm.rb │ └── vsphere.rb │ └── scripts │ ├── install-bro.sh │ ├── install-elastic.sh │ ├── install-evebox.sh │ ├── install-grafana.sh │ ├── install-influxdb.sh │ ├── install-kibana.sh │ ├── install-logstash.sh │ ├── install-metrix-server.sh │ ├── install-moloch.sh │ ├── install-salt-master.sh │ ├── install-salt-minion.sh │ ├── install-scirius.sh │ ├── install-suri.sh │ ├── install-syslog-server.sh │ └── install-telegraf.sh ├── moloch ├── README.md ├── day_1 │ ├── BuildFromSource.md │ ├── Settings.md │ ├── Vagrantfile │ ├── moloch-capture.conf │ ├── moloch-viewer.conf │ ├── my-custom-install.sh │ └── wtfit.md ├── day_2 │ ├── MoloClusterNode.md │ ├── MultipleHostConfig.md │ ├── Readme.md │ ├── Vagrantfile │ ├── boxes │ │ ├── Vagrantfile │ │ └── scripts │ ├── moloch-capture.conf │ └── moloch-viewer.conf ├── day_3 │ ├── Cluster2Cluster.md │ └── lua.md ├── day_intro │ ├── MVGK-easybutton-singlehost.sh │ ├── README.md │ ├── dummy │ │ ├── Vagrantfile │ │ └── dummy-install-something.sh │ ├── easybutton │ │ ├── 2016-02-28-traffic-analysis-exercise-2.png │ │ ├── 2016-02-28-traffic-analysis-exercise.png │ │ ├── MVGK-autoportrait.png │ │ ├── Vagrantfile │ │ └── vsroom-moloch-sessions-demo.mov │ └── empty │ │ └── Vagrantfile └── day_last │ └── README.md ├── saynomore.png └── suricata ├── README.md ├── day_1 ├── BuildFromSource.md ├── CEE.md ├── Config.md ├── ConfigureHelp.md ├── ConfigureProfilingLuaJit.md ├── DNSProbe.md ├── FileExtraction.md ├── FlowProbe.md ├── MakeInstallFull.md ├── README.md ├── Reputation.md ├── RuleManagement.md ├── RuleProfiling.md ├── RuleThresholding.md ├── SingleBoxtuning.md ├── SingleHostConfig.md ├── TLSMonitor.md ├── abox │ └── Vagrantfile ├── boxes │ ├── Vagrantfile │ └── scripts │ │ └── install-suricata.sh ├── build.txt └── rsyslog.md ├── day_2 ├── Grafana.md ├── README.md ├── SetUpElastic.md ├── SetUpMaster.md ├── SetUpMetrics.md ├── SetUpMinions.md ├── admin.md ├── boxes │ ├── Vagrantfile │ └── scripts │ │ ├── evebox-server │ │ ├── install-bro-master-and-workers.sh │ │ ├── install-elastic.sh │ │ ├── install-evebox.sh │ │ ├── install-grafana.sh │ │ ├── install-influxdb.sh │ │ ├── install-kibana4.sh │ │ ├── install-logstash-for-bro-csv.sh │ │ ├── install-logstash.sh │ │ ├── install-moloch-capture.sh │ │ ├── install-moloch-viewer.sh │ │ ├── install-moloch.sh │ │ ├── install-salt-master.sh │ │ ├── install-salt-minion.sh │ │ ├── install-scirius.sh │ │ ├── install-suricata.sh │ │ └── install-telegraf.sh ├── homenet.md ├── notes.md ├── pushrules.md ├── suriN.dot └── suriN.png ├── day_3 ├── DumpCounters.md ├── Eve2Pcap.md ├── EvilBit.md ├── LoadPcaps.md ├── LuaOutput.md ├── LuaRules.md ├── MultiTenancy.md ├── NICAccess.md └── pcap-feeder ├── day_intro ├── Amsterdam.md ├── AppProtos.md ├── AptGetInstall.md ├── AptGetInstallPPA.md ├── BasicConf.md ├── BasicLogging.md ├── EveView.md ├── README.md ├── SSEE-easybutton-singlehost.sh ├── SimpleRule.md ├── ams_install.sh ├── amstelredamme │ ├── README.md │ └── Vagrantfile ├── dummy │ ├── Vagrantfile │ └── dummy-install-suricata.sh ├── empty │ └── Vagrantfile ├── etc.md ├── help.md ├── keywords.md ├── log.md └── runmodes.md └── day_last └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | common/vagrant/bb/.vagrant 3 | common/vagrant/bb/lastrun.txt 4 | suricata/day_1/abox/.vagrant 5 | suricata/day_1/boxes/.vagrant 6 | suricata/day_1/boxes/lastrun.txt 7 | suricata/day_2/boxes/.vagrant 8 | suricata/day_2/boxes/lastrun.txt 9 | suricata/day_intro/amstelredamme/.vagrant 10 | suricata/day_intro/dummy/.vagrant 11 | suricata/day_intro/empty/.vagrant 12 | bro/day_intro/empty/.vagrant 13 | bro/day_intro/empty/.vagrant 14 | bro/day_intro/brodello/.vagrant 15 | bro/day_2/boxes/.vagrant 16 | 17 | bro/day_2/boxes/lastrun.txt 18 | 19 | suricata/day_1/lastrun.txt 20 | 21 | moloch/day_intro/easybutton/.vagrant 22 | moloch/day_intro/dummy/.vagrant 23 | 24 | moloch/day_intro/boxes/.vagrant 25 | moloch/day_1/.vagrant 26 | moloch/day_2/.vagrant 27 | moloch/day_2/lastrun.txt 28 | moloch/day_2/boxes/.vagrant 29 | moloch/day_2/boxes/lastrun.txt 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > * moving to https://github.com/ccdcoe/CDMCS 2 | 3 | # Cyber Defence Monitoring Course Suite (CDMCS) 4 | 5 | ![sAYCyber!](/saynomore.png) 6 | 7 | This material has been designed to be taught in a classroom environment. 8 | 9 | The material is missing some of the contextual concepts and ideas that will be covered in class. 10 | 11 | This is *3 x ( .5 + 3 + .5 )* days of material for any intermediate-level dev-ops who has some experience with other security|monitoring tools and wants to learn Suricata, Bro and Moloch. We believe these classes are perfect for anyone who wants a jump start in learning Suricata, Bro and Moloch or who wants a more thorough understanding of the three and their internals. 12 | 13 | ## Minimal Qualified Student 14 | 15 | **TODO ....** 16 | 17 | ## Before You Come To Class 18 | 19 | Please visit: 20 | 21 | * http://suricata-ids.org/ 22 | * https://www.bro.org/ 23 | * https://github.com/aol/moloch 24 | 25 | Historical material: 26 | * [vagrant multi-machine: Moloch, Bro,Suricata,ElasticSearch,Kibana](https://github.com/hillar/vagrant_moloch_bro_suricata) 27 | * [Getting started with 28 | Network Forensics](http://slides.com/hillar/network-forensics#/) 29 | 30 | ## Go to the material 31 | 32 | .| [Suricata](./suricata/README.md) | [Bro](./bro/README.md) | [Moloch](./moloch/README.md) 33 | --- | --- | --- | --- 34 | day 0 :: intro| [*](/suricata/README.md#day-0) | [*](/bro/README.md#day-0) | [*](/moloch/README.md#day-0) 35 | day 1 :: single box| [*](/suricata/README.md#day-1--single-box) | [*](/bro/README.md#day-1--single-box) | [*](/moloch/README.md#day-1--single-box) 36 | day 2 :: cluster| [*](/suricata/README.md#day-2--cluster) | [*](/bro/README.md#day-2--cluster) | [*](/moloch/README.md#day-2--cluster) 37 | day 3 :: advanced| [*](/suricata/README.md#day-3--advanced-usage)| [*](/bro/README.md#day-3--advanced-usage)| [*](/moloch/README.md#day-3--advanced-usage) 38 | day +1 :: wrapup| [*](/suricata/README.md#day-1)| [*](/bro/README.md#day-1) | [*](./moloch/day_last/README.md) 39 | 40 | ## Current Classes 41 | 42 | * Suricata :: [Rule-based Threat Detection Course](https://ccdcoe.org/cyber-defence-monitoring-course-suite-module-1.html) 43 | * Bro :: [Semantic Network Security Monitoring Course](https://ccdcoe.org/cyber-defence-monitoring-course-suite-module-2.html) 44 | * Moloch :: [Large-Scale Packet Capture Analysis Course](https://ccdcoe.org/cyber-defence-monitoring-course-suite-module-3.html) 45 | 46 | ## Teachers 47 | 48 | * [hillar](https://github.com/hillar) 49 | * [regit](https://github.com/regit) 50 | * [markuskont](https://github.com/markuskont) 51 | * [mpihelgas](https://github.com/mpihelgas) 52 | 53 | 54 | ## Thanks 55 | [pevma](https://github.com/pevma), [inliniac](https://github.com/inliniac), [jasonish](https://github.com/jasonish) 56 | -------------------------------------------------------------------------------- /bro/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Day 0 3 | 4 | * 13:00 - 13:45 [Intro](/bro/day_intro/README.md) [apt-get install](/bro/day_intro/AptGetInstall.md) 5 | * 14:00 - 14:45 [vagrant](/bro/day_intro/BTIGEK-easybutton-singlehost.sh) 6 | * 15:00 - 15:45 [basic config](/bro/day_intro/BasicConf.md) 7 | * 16:00 - 16:45 [basic logging](/bro/day_intro/BasicLogging.md) 8 | 9 | ## Day 1 :: Single Box 10 | 11 | * 09:00 - 09:45 [build from source](/bro/day_1/BuildFromSource.md) 12 | * 10:00 - 10:45 [config](/bro/day_1/Config.md) 13 | * 11:00 - 11:45 [Structured logging](/bro/day_1/AdvancedLogging0.md),[logstash](/bro/day_1/AdvancedLoggingLogstash.md) 14 | 15 | 16 | * 13:00 - 13:45 [script loading](/bro/day_1/LoadScript.md) 17 | * 14:00 - 14:45 [intro to](/bro/day_intro/BasicScripting.md) [script writing](/bro/day_intro/Scripting.md) 18 | * 15:00 - 15:45 [viewing output]() 19 | * 16:00 - 16:45 20 | 21 | 22 | ## Day 2 :: Cluster 23 | 24 | * 09:00 - 09:45 [intro](/bro/day_2/ClusterBasics.md),[Metrix](/suricata/day_2/SetUpMetrics.md),[Elastic](/suricata/day_2/SetUpElastic.md), [Minions](/suricata/day_2/SetUpMinions.md), [Master](/suricata/day_2/SetUpMaster.md), [and script](/bro/day_2/boxes/Vagrantfile) 25 | * 10:00 - 10:45 [manager](/bro/day_2/Manager.md) 26 | * 11:00 - 11:45 [Basic Cluster Configuration](/bro/day_2/ClusterConf.md) 27 | 28 | 29 | 30 | * 13:00 - 15:45 [adding node](/bro/day_2/ClusterAddNode.md) 31 | * 16:00 - 16:45 [housekeeping](/bro/day_2/Cron.md) 32 | 33 | 34 | ## Day 3 :: Usage 35 | 36 | * 09:00 - 09:45 [digging into numbers](/common/Telegraf.md) 37 | * 10:00 - 10:45 [digging into text](/common/Kibana4.md) 38 | * 11:00 - 11:45 [loading pcaps](/bro/day_3/LoadPcaps.md) 39 | 40 | 41 | * 13:00 - 13:45 [loading csv](/bro/day_3/csv2elasticsearch.md) 42 | * 14:00 - 14:45 [loading scripts to cluster](/bro/day_3/scripts2cluster.md) 43 | * 15:00 - 15:45 44 | * 16:00 - 16:45 45 | 46 | ## Day +1 47 | 48 | * 09:00 - 09:45 []() 49 | * 10:00 - 10:45[]() 50 | * 11:00 - 12:00[feedback, contact exchange, thanks, etc]() 51 | -------------------------------------------------------------------------------- /bro/day_1/AdvancedLogging0.md: -------------------------------------------------------------------------------- 1 | # Structured logging 2 | 3 | * https://www.bro.org/sphinx/scripts/policy/tuning/json-logs.bro.html 4 | 5 | ## Unstructured conn log 6 | 7 | ``` 8 | 1449516638.400427 CreaaN3V0Fcx0Nj4xc 192.168.11.11 59518 192.168.12.12 443 tcp http 0.040555 88 170 SF -- 0 ShADadfF 7 460 7 542 (empty) 9 | ``` 10 | 11 | ## Structured conn log 12 | 13 | ``` 14 | {"ts":1449516638.400427,"uid":"CeHdP448pqXC61wZC6","id.orig_h":"192.168.11.11","id.orig_p":59518,"id.resp_h":"192.168.12.12","id.resp_p":443,"proto":"tcp","service":"http","duration":0.040555,"orig_bytes":88,"resp_bytes":170,"conn_state":"SF","missed_bytes":0,"history":"ShADadfF","orig_pkts":7,"orig_ip_bytes":460,"resp_pkts":7,"resp_ip_bytes":542,"tunnel_parents":[]} 15 | ``` -------------------------------------------------------------------------------- /bro/day_1/AdvancedLoggingLogstash.md: -------------------------------------------------------------------------------- 1 | # Advanced Logging 2 | 3 | This page assumes an existing logstash installation from elastic repository 4 | 5 | * https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html 6 | * https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html 7 | * https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html 8 | * https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_20_mapping_changes.html 9 | * https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html 10 | * https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html 11 | * https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html 12 | 13 | # Locate logstash configuration directory 14 | 15 | ``` 16 | ls -la /etc/logstash/conf.d/ 17 | ``` 18 | 19 | # View logstash daemon logs 20 | 21 | ``` 22 | tail -f /var/log/logstash/logstash.{err,log} 23 | ``` 24 | 25 | # Start and stop daemon 26 | 27 | ``` 28 | service logstash start 29 | service logstash stop 30 | ``` 31 | 32 | Be patient! 33 | 34 | # Verify logstash configuration 35 | 36 | ``` 37 | sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest 38 | ``` 39 | 40 | # Verify logstash permissions 41 | 42 | ``` 43 | grep bro /etc/passwd && adduser logstash bro 44 | ``` 45 | 46 | ``` 47 | ls -la //spool/bro 48 | ``` 49 | 50 | # Using the file input plugin 51 | 52 | ``` 53 | vim /etc/logstash/conf.d/10-bro.conf 54 | ``` 55 | 56 | ``` 57 | input { 58 | file { 59 | type => "bro-conn_log" 60 | start_position => "end" 61 | sincedb_path => "/var/tmp/.bro_conn_sincedb" 62 | path => "/opt/bro/logs/current/conn.log" 63 | } 64 | } 65 | ``` 66 | 67 | Or... 68 | 69 | ``` 70 | input { 71 | file { 72 | type => "bro_log" 73 | start_position => "end" 74 | path => "/opt/bro/logs/current/*.log" 75 | } 76 | } 77 | ``` 78 | 79 | # Elasticsearch output 80 | 81 | ``` 82 | output { 83 | # stdout { codec => rubydebug } 84 | elasticsearch { 85 | hosts => "127.0.0.1" 86 | index => "bro-%{+YYYY.MM.dd.HH}" 87 | } 88 | } 89 | ``` 90 | 91 | # Opening the JSON 92 | 93 | ``` 94 | filter { 95 | json { 96 | source => "message" 97 | } 98 | } 99 | ``` 100 | 101 | If statements are your friends... 102 | 103 | filter { 104 | if type == "bro_log" { 105 | json { 106 | source => "message" 107 | } 108 | } 109 | } 110 | 111 | # Normalizing field names 112 | 113 | ## Easy but expensive 114 | 115 | ``` 116 | de_dot {} 117 | ``` 118 | 119 | ## Might be better 120 | 121 | ``` 122 | mutate { 123 | gsub => [ 124 | "message", "id\.orig_h", "id_orig_h", 125 | "message", "id\.resp_h", "id_resp_h", 126 | "message", "id\.orig_p", "id_orig_p", 127 | "message", "id\.resp_p", "id_resp_p" 128 | ] 129 | } 130 | ``` 131 | 132 | # Timestamping for Kibana 133 | 134 | ## Quick and dirty 135 | 136 | ``` 137 | date { 138 | match => [ "ts", "UNIX" ] 139 | } 140 | ``` 141 | 142 | ## Challenge - use Bro and Elasticsearch 143 | 144 | * https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-JSON::TimestampFormat 145 | * https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html 146 | * https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html 147 | * [Bro ES template - with syntax error](/bro/day_1/bro.json) 148 | * [Bro ES template - working](/bro/day_1/bro2.json) 149 | -------------------------------------------------------------------------------- /bro/day_1/AdvancedLoggingRsyslog.md: -------------------------------------------------------------------------------- 1 | [Getting started with Rsyslog](/common/rsyslogBasics.md) 2 | 3 | * http://www.rsyslog.com/doc/v8-stable/configuration/templates.html 4 | * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html 5 | 6 | # Invoking file input module 7 | 8 | ``` 9 | vim /etc/rsyslog.d/70-bro.conf 10 | ``` 11 | 12 | ## Old syntax 13 | 14 | ``` 15 | $ModLoad imfile # 16 | $InputFileName /usr/local/bro/logs/current/conn.log 17 | $InputFileTag bro_conn: 18 | $InputFileStateFile stat-bro_conn 19 | $InputFileSeverity info 20 | $InputFileFacility local7 21 | ``` 22 | 23 | No wildcards for rsyslog! 24 | 25 | ``` 26 | service rsyslog restart 27 | ``` 28 | 29 | Activate the poller 30 | 31 | ``` 32 | $InputRunFileMonitor 33 | ``` 34 | 35 | Setting poller interval 36 | 37 | ``` 38 | $InputFilePollingInterval 1 39 | ``` 40 | 41 | ## New syntax 42 | 43 | ``` 44 | module(load="imfile" PollingInterval="1") #needs to be done just once 45 | 46 | # File 1 47 | input(type="imfile" 48 | File="/usr/local/bro/logs/current/conn.log" 49 | Tag="bro_conn" 50 | Severity="info" 51 | Facility="local7") 52 | ``` -------------------------------------------------------------------------------- /bro/day_1/BuildFromSource.md: -------------------------------------------------------------------------------- 1 | # Building Bro from source 2 | 3 | see https://www.bro.org/sphinx/install/install.html#installing-from-source 4 | 5 | ## Dependencies 6 | 7 | ``` 8 | apt-get install \ 9 | cmake \ 10 | make \ 11 | gcc \ 12 | g++ \ 13 | flex \ 14 | bison \ 15 | libpcap-dev \ 16 | libgeoip-dev \ 17 | libssl-dev \ 18 | python-dev \ 19 | zlib1g-dev \ 20 | libmagic-dev \ 21 | swig2.0 \ 22 | ``` 23 | 24 | ## libcaf 25 | 26 | > Robin Sommer | 22 Jul 21:39 2015 27 | 28 | > Re: Build Errors - Libcaf 29 | 30 | > On Wed, Jul 22, 2015 at 12:06 -0700, anthony kasza wrote: 31 | >> I'm building Bro from source and am receiving cmake errors "Could NOT find 32 | >> Libcaf". Is this a requirement to build Bro now? 33 | 34 | >Yes, since last night. :) 35 | 36 | ``` 37 | wget https://github.com/actor-framework/actor-framework/archive/0.14.4.tar.gz 38 | tar -xzf 0.14.4.tar.gz 39 | cd actor-framework-0.14.4/ 40 | ./configure --no-examples 41 | make 42 | make install 43 | ``` 44 | 45 | ## get the source 46 | ``` 47 | git clone --recursive git://git.bro.org/bro 48 | ``` 49 | 50 | ## configure, make install 51 | 52 | ``` 53 | ./configure --help 54 | ``` 55 | 56 | ``` 57 | ./configure 58 | ``` 59 | 60 | ``` 61 | make 62 | ... 63 | make[1]: Leaving directory `/root/bro/build' 64 | 65 | real 15m11.303s 66 | user 13m46.980s 67 | sys 1m19.839s 68 | ``` 69 | 70 | ``` 71 | sudo make install 72 | ``` 73 | 74 | ``` 75 | /usr/local/bro/bin/bro -v 76 | version 2.4-313 77 | ``` 78 | 79 | # Cleaning up 80 | 81 | Following commands must be entered while inside the source code directory. 82 | 83 | ``` 84 | make clean 85 | ``` 86 | 87 | ``` 88 | make distclean 89 | ``` 90 | 91 | Bro has no make uninstall command :( 92 | 93 | # Creating a deb package 94 | 95 | ``` 96 | apt-get install checkinstall 97 | checkinstall --help 98 | ``` -------------------------------------------------------------------------------- /bro/day_1/Config.md: -------------------------------------------------------------------------------- 1 | # Single host configuration 2 | 3 | See: 4 | * https://www.bro.org/sphinx/quickstart/index.html#a-minimal-starting-configuration 5 | 6 | # Home networks 7 | 8 | ``` 9 | cat /opt/bro/etc/networks.cfg 10 | ``` 11 | 12 | Do not forget IPv6! 13 | 14 | 15 | # Listening interface 16 | 17 | ``` 18 | grep 'interface' -B4 /opt/bro/etc/node.cfg 19 | ``` 20 | 21 | # Working modes 22 | * Standalone 23 | 24 | * Clustered - also multithreading on a single host 25 | https://www.bro.org/sphinx-git/cluster/index.html#on-host-flow-balancing 26 | 27 | 28 | 29 | # Notifications, log rotation, directories 30 | 31 | ``` 32 | vim /opt/bro/etc/broctl.cfg 33 | ``` 34 | 35 | # Broccoli 36 | BRO Client COmmunications LIbrary 37 | 38 | https://www.bro.org/sphinx/components/broccoli/broccoli-manual.html 39 | 40 | 41 | # BroControl 42 | Remember to install broctl configuration. 43 | See notes from [basic config](/bro/day_intro/BasicConf.md) 44 | 45 | ## Tasks 46 | * Start a Bro instance and check if it is running. 47 | * Find out which nodes are running? 48 | * Which interfaces are monitored? 49 | * What is the current packet count of the node? 50 | * Find all running Bro processes. 51 | * What type is your Bro instance? 52 | * Try restarting the Bro instance. 53 | * Stop Bro and check if it has stopped. 54 | * Exit BroControl 55 | 56 | # Housekeeping tasks 57 | Insert the following entry into the crontab of the user running BroControl: 58 | ``` 59 | 0-59/5 * * * * $PREFIX/bin/broctl cron 60 | ``` 61 | 62 | * Regular housekeeping tasks 63 | * Verifying that the process is running (restarting in case of abnormal termination) 64 | -------------------------------------------------------------------------------- /bro/day_1/LoadScript.md: -------------------------------------------------------------------------------- 1 | # Load a scripts 2 | 3 | see 4 | 5 | * https://www.bro.org/sphinx/quickstart/#telling-bro-which-scripts-to-load 6 | 7 | ``` 8 | ./bro -r some.pcap some_nice_script.bro 9 | ``` 10 | 11 | ``` 12 | ./bro -r lotofcerts.pcap protocols/ssl/validate-certs 13 | ``` 14 | -------------------------------------------------------------------------------- /bro/day_1/Scripting.md: -------------------------------------------------------------------------------- 1 | #Write a script 2 | 3 | > do not mix with plugins 4 | > see https://www.bro.org/sphinx/devel/plugins.html?highlight=exercise#builtin-language-elements 5 | 6 | go to 7 | 8 | * http://try.bro.org/#/trybro?example=hello 9 | * http://www.malware-traffic-analysis.net/training-exercises.html 10 | 11 | see 12 | 13 | * https://gist.github.com/sethhall 14 | 15 | 16 | see also 17 | 18 | * https://github.com/search?utf8=%E2%9C%93&q=language%3ABro&type=Repositories&ref=advsearch&l=Bro&l= 19 | * ... 20 | * https://github.com/michalpurzynski/bro-gramming 21 | * https://github.com/CrowdStrike/cs-bro 22 | * https://github.com/sethhall/credit-card-exposure 23 | * https://github.com/BrashEndeavours/bro-scripts 24 | * ... 25 | -------------------------------------------------------------------------------- /bro/day_1/abox/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.120" 2 | vm_box = 'ubu14s' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :source do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'source' 10 | box.vm.provision "shell", inline: "uptime" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "4"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /bro/day_1/blacklists.bro: -------------------------------------------------------------------------------- 1 | # Generates a notice when a connection is established to an IP which is in a blacklist file. 2 | # TODO: Different file/table for known malicious domains 3 | # 4 | # Thanks to SgtMalicious for initial functionality 5 | # https://gist.github.com/SgtMalicious/7a4b3524ba4de2e96801 6 | # 7 | 8 | @load base/frameworks/notice 9 | 10 | module MalwareSite; 11 | 12 | 13 | type Idx: record { 14 | ip: addr; 15 | }; 16 | 17 | 18 | global addresses: set[addr]; 19 | 20 | event bro_init() { 21 | # get an example list from https://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt 22 | # remember to insert the table header into the file: #fieldsip 23 | Input::add_table([$source="/usr/local/bro/share/bro/site/compromised-ips.txt",$name="addresses",$idx=Idx,$destination=addresses]); 24 | } 25 | 26 | 27 | export { 28 | redef enum Notice::Type += { 29 | Connection_Detected, 30 | #Domain_Hit, 31 | }; 32 | } 33 | 34 | function check(c: connection) { 35 | local id = c$id; 36 | 37 | local orig = id$orig_h; 38 | local resp = id$resp_h; 39 | 40 | if ( resp in addresses ) 41 | NOTICE([$note=Connection_Detected, 42 | $msg=fmt("Connection to known malware site %s from %s detected.", resp, orig)]); 43 | } 44 | 45 | event connection_established(c: connection) { 46 | MalwareSite::check(c); 47 | } 48 | 49 | #event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) { 50 | # if ( query in domains ) { 51 | # NOTICE([$note=Domain_Hit, 52 | # $conn=c, 53 | # $msg=fmt("A domain from the MALWARE domains report seen: %s", query), 54 | # $identifier=cat(query)]); 55 | # } 56 | #} 57 | 58 | -------------------------------------------------------------------------------- /bro/day_1/bro.json: -------------------------------------------------------------------------------- 1 | { 2 | "order" : 0, 3 | "template" : "bro-*", 4 | "settings" : { 5 | "index.refresh_interval" : "60s", 6 | "index.number_of_replicas" : "0", 7 | "index.number_of_shards" : "2" 8 | }, 9 | "mappings" : { 10 | "_default_" : { "dynamic_templates" : [ { 11 | "string_fields" : { 12 | "mapping" : { 13 | "index" : "analyzed", 14 | "omit_norms" : true, 15 | "type" : "string", 16 | "fields" : { 17 | "raw" : { 18 | "index" : "not_analyzed", 19 | "ignore_above" : 256, 20 | "type" : "string" 21 | } 22 | } 23 | }, 24 | "match" : "*", 25 | "match_mapping_type" : "string" 26 | } 27 | } ], 28 | "properties" : { 29 | "@version" : { 30 | "index" : "not_analyzed", 31 | "type" : "string" 32 | }, 33 | "ts" : { 34 | "index" : "analyzed", 35 | "type" : "date", 36 | }, 37 | "conn_state" : { 38 | "index" : "not_analyzed", 39 | "type" : "string" 40 | }, 41 | "history" : { 42 | "index" : "not_analyzed", 43 | "type" : "string" 44 | }, 45 | "id_orig_h" : { 46 | "index" : "not_analyzed", 47 | "type" : "string" 48 | }, 49 | "id_resp_h" : { 50 | "index" : "not_analyzed", 51 | "type" : "string" 52 | }, 53 | "id_orig_p" : { 54 | "index" : "analyzed", 55 | "type" : "integer" 56 | }, 57 | "id_resp_p" : { 58 | "index" : "analyzed", 59 | "type" : "integer" 60 | }, 61 | "local_orig" : { 62 | "index" : "not_analyzed", 63 | "type" : "string" 64 | }, 65 | "local_resp" : { 66 | "index" : "not_analyzed", 67 | "type" : "string" 68 | }, 69 | "missed_bytes" : { 70 | "index" : "analyzed", 71 | "type" : "integer" 72 | }, 73 | "orig_ip_bytes" : { 74 | "index" : "analyzed", 75 | "type" : "integer" 76 | }, 77 | "orig_pkts" : { 78 | "index" : "analyzed", 79 | "type" : "integer" 80 | }, 81 | "proto" : { 82 | "index" : "not_analyzed", 83 | "type" : "string" 84 | }, 85 | "resp_ip_bytes" : { 86 | "index" : "analyzed", 87 | "type" : "integer" 88 | }, 89 | "resp_pkts" : { 90 | "index" : "analyzed", 91 | "type" : "integer" 92 | }, 93 | "uid" : { 94 | "index" : "not_analyzed", 95 | "type" : "string" 96 | } 97 | }, 98 | "_all" : { 99 | "enabled" : false 100 | } 101 | } 102 | }, 103 | "aliases" : { } 104 | } 105 | -------------------------------------------------------------------------------- /bro/day_1/bro2.json: -------------------------------------------------------------------------------- 1 | { 2 | "order" : 0, 3 | "template" : "bro-*", 4 | "settings" : { 5 | "index.refresh_interval" : "10s", 6 | "index.number_of_replicas" : "0", 7 | "index.number_of_shards" : "2" 8 | }, 9 | "mappings" : { 10 | "_default_" : { 11 | "dynamic_templates" : [ { 12 | "string_fields" : { 13 | "mapping" : { 14 | "index" : "analyzed", 15 | "omit_norms" : true, 16 | "type" : "string", 17 | "fields" : { 18 | "raw" : { 19 | "index" : "not_analyzed", 20 | "ignore_above" : 256, 21 | "type" : "string" 22 | } 23 | } 24 | }, 25 | "match" : "*", 26 | "match_mapping_type" : "string" 27 | } 28 | } ], 29 | "properties" : { 30 | "ts" : { 31 | "index" : "analyzed", 32 | "type" : "date", 33 | "format": "epoch_millis" 34 | }, 35 | "@version" : { 36 | "index" : "not_analyzed", 37 | "type" : "string" 38 | }, 39 | "syslog_pri" : { 40 | "index" : "analyzed", 41 | "type" : "integer" 42 | }, 43 | "syslog_timestamp" : { 44 | "index" : "not_analyzed", 45 | "type" : "string" 46 | }, 47 | "syslog_hostname" : { 48 | "index" : "not_analyzed", 49 | "type" : "string" 50 | }, 51 | "syslog_program" : { 52 | "index" : "not_analyzed", 53 | "type" : "string" 54 | }, 55 | "syslog_message" : { 56 | "index" : "not_analyzed", 57 | "type" : "string" 58 | }, 59 | "http_vhost" : { 60 | "index" : "not_analyzed", 61 | "type" : "string" 62 | }, 63 | "http_port" : { 64 | "index" : "analyzed", 65 | "type" : "integer" 66 | }, 67 | "http_clientip" : { 68 | "index" : "not_analyzed", 69 | "type" : "string" 70 | }, 71 | "http_ident" :{ 72 | "index" : "not_analyzed", 73 | "type" : "string" 74 | }, 75 | "http_auth" : { 76 | "index" : "not_analyzed", 77 | "type" : "string" 78 | }, 79 | "http_timestamp" : { 80 | "index" : "not_analyzed", 81 | "type" : "string" 82 | }, 83 | "http_method" : { 84 | "index" : "not_analyzed", 85 | "type" : "string" 86 | }, 87 | "http_request" : { 88 | "index" : "not_analyzed", 89 | "type" : "string" 90 | }, 91 | "http_version" : { 92 | "index" : "not_analyzed", 93 | "type" : "string" 94 | }, 95 | "http_response" : { 96 | "index" : "analyzed", 97 | "type" : "integer" 98 | }, 99 | "http_bytes" : { 100 | "index" : "analyzed", 101 | "type" : "integer" 102 | }, 103 | "http_referer" : { 104 | "index" : "not_analyzed", 105 | "type" : "string" 106 | }, 107 | "http_useragent" : { 108 | "index" : "not_analyzed", 109 | "type" : "string" 110 | }, 111 | "name" : { 112 | "index" : "not_analyzed", 113 | "type" : "string" 114 | }, 115 | "os" : { 116 | "index" : "not_analyzed", 117 | "type" : "string" 118 | }, 119 | "os_name" : { 120 | "index" : "not_analyzed", 121 | "type" : "string" 122 | }, 123 | "device" : { 124 | "index" : "not_analyzed", 125 | "type" : "string" 126 | }, 127 | "major" : { 128 | "index" : "analyzed", 129 | "type" : "integer" 130 | }, 131 | "minor" : { 132 | "index" : "analyzed", 133 | "type" : "integer" 134 | } 135 | }, 136 | "_all" : { 137 | "enabled" : true 138 | } 139 | } 140 | }, 141 | "aliases" : { } 142 | } 143 | -------------------------------------------------------------------------------- /bro/day_2/ClusterAddNode.md: -------------------------------------------------------------------------------- 1 | # Adding Bro node 2 | 3 | * https://help.ubuntu.com/community/SSH/OpenSSH/Keys 4 | 5 | # All nodes 6 | 7 | ## User creation 8 | 9 | ``` 10 | useradd bro -N 11 | ``` 12 | 13 | ``` 14 | useradd --help 15 | ``` 16 | 17 | ## Setting home directory 18 | 19 | ``` 20 | usermod -d /opt/bro bro 21 | ``` 22 | 23 | ## Create SSH key directory in user home 24 | 25 | ``` 26 | mkdir /opt/bro/.ssh 27 | chown -R bro:bro /opt/bro 28 | chmod 750 /opt/bro/.ssh 29 | ``` 30 | 31 | ## Configure SSH public key 32 | 33 | ``` 34 | echo "" > /opt/bro/.ssh/authorized_keys 35 | ``` 36 | 37 | # Master 38 | 39 | ## Execute command as another user 40 | 41 | ``` 42 | su - bro -s /bin/bash -c 'whoami' 43 | ``` 44 | 45 | ## Generate SSH keys 46 | 47 | ``` 48 | ssh-keygen -t rsa -f /opt/bro/.ssh/id_rsa 49 | ``` 50 | 51 | ## View rsa public key 52 | 53 | ``` 54 | cat /opt/bro/.ssh/id_rsa.pub 55 | ``` 56 | 57 | ## Store SSH fingerprints 58 | 59 | ### Lazy admin 60 | 61 | ``` 62 | vim /opt/bro/.ssh/config 63 | ``` 64 | ``` 65 | Host 192.168.0.* 66 | StrictHostKeyChecking no 67 | ``` 68 | 69 | ### Proper way 70 | 71 | ``` 72 | ssh-keyscan 192.168.56.111 192.168.56.112 192.168.56.113 > /opt/bro/.ssh/known_hosts 73 | ``` 74 | 75 | ### Deploy bro configuration 76 | 77 | ``` 78 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl deploy' 79 | ``` -------------------------------------------------------------------------------- /bro/day_2/ClusterBasics.md: -------------------------------------------------------------------------------- 1 | # BRO cluster 2 | 3 | see https://www.bro.org/sphinx/cluster/index.html 4 | 5 | https://www.bro.org/sphinx-git/scripts/base/frameworks/cluster/main.bro.html#namespace-Cluster 6 | 7 | ![main parts of cluster](https://www.bro.org/sphinx/_images/deployment.png) 8 | 9 | ## Worker 10 | 11 | * sniffs traffic and does analysis 12 | 13 | ## Proxy 14 | 15 | * manages synchronized state 16 | 17 | ## Manager 18 | 19 | * waits for workers 20 | * stores logs 21 | * dedups notices 22 | -------------------------------------------------------------------------------- /bro/day_2/ClusterConf.md: -------------------------------------------------------------------------------- 1 | # BRO cluster Configuration 2 | 3 | see: 4 | 5 | * https://www.bro.org/sphinx/cluster/index.html 6 | * https://www.bro.org/sphinx-git/configuration/index.html 7 | 8 | ![main parts of cluster](https://www.bro.org/sphinx/_images/deployment.png) 9 | 10 | ## manager host configuration 11 | 12 | ### $PREFIX/etc/node.cfg 13 | ``` 14 | [manager] 15 | type=manager 16 | host=10.0.0.10 17 | 18 | [proxy-1] 19 | type=proxy 20 | host=10.0.0.10 21 | 22 | [worker-1] 23 | type=worker 24 | host=10.0.0.11 25 | interface=eth0 26 | 27 | [worker-2] 28 | type=worker 29 | host=10.0.0.12 30 | interface=eth0 31 | ``` 32 | 33 | ### BroControl 34 | ``` 35 | $PREFIX/etc/broctl.cfg 36 | ``` 37 | 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /bro/day_2/Cron.md: -------------------------------------------------------------------------------- 1 | # BRO Cron 2 | 3 | > The BroControl cron command performs housekeeping tasks, such as checking whether Bro is running or not (and starting or stopping to match the expected state, as needed), checking if there is sufficient free disk space, etc. This command is intended to be run from a cron job, rather than interactively by a user. 4 | 5 | 6 | 7 | # Elasticsearch Curator 8 | 9 | 10 | > Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your indices. 11 | 12 | ## but what if *broctl cron* & *curator* is not doing what you want ? 13 | 14 | * write your own bash||perl||ruby||.. script 15 | 16 | see https://github.com/aol/moloch/blob/ab8c088f43a5d957cee4af08f9959b0202815758/viewer/viewer.js#L830 17 | -------------------------------------------------------------------------------- /bro/day_2/Manager.md: -------------------------------------------------------------------------------- 1 | # Manager 2 | 3 | see 4 | 5 | * [manager provision script](/suricata/day_2/boxes/scripts/install-bro-master-and-workers.sh) 6 | -------------------------------------------------------------------------------- /bro/day_2/boxes/ripid.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/bro/day_2/boxes/ripid.txt -------------------------------------------------------------------------------- /bro/day_2/boxes/scripts: -------------------------------------------------------------------------------- 1 | ../../../suricata/day_2/boxes/scripts -------------------------------------------------------------------------------- /bro/day_2/broctl-setcap.md: -------------------------------------------------------------------------------- 1 | # Fixing the setcap every time broctl deploys configuration to workers 2 | 3 | ## A: Install a broctl plugin that does it for you 4 | see: https://github.com/PingTrip/broctl-setcap 5 | 6 | 7 | ## B: Workaround 8 | 9 | Run all these commands after deploy or make into a "deploy" script for yourself 10 | 11 | ``` 12 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl deploy' 13 | 14 | # Even though workers crashed, to clean up, you should run 'stop' for workers 15 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl stop' 16 | salt "*$BROS*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/bro' 17 | salt "*$BROS*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/capstats' 18 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl start' 19 | ``` 20 | 21 | -------------------------------------------------------------------------------- /bro/day_2/cheat4addingworker.md: -------------------------------------------------------------------------------- 1 | # cheating... 2 | 3 | ``` 4 | STUDENTNO=18 5 | BROS="student-$STUDENTNO-bro-worker-b" 6 | salt "*$BROS*" test.ping 7 | salt "*$BROS*" cmd.run 'addgroup --system bro --quiet' 8 | salt "*$BROS*" cmd.run 'adduser --system --home /opt/bro --no-create-home --ingroup bro --disabled-password --shell /bin/bash bro' 9 | salt "*$BROS*" cmd.run 'mkdir -p /opt/bro/.ssh' 10 | salt-cp "*$BROS*" /opt/bro/.ssh/id_rsa.pub /opt/bro/.ssh/bro-manager.pub 11 | salt "*$BROS*" cmd.run 'cat /opt/bro/.ssh/bro-manager.pub >> /opt/bro/.ssh/authorized_keys' 12 | cat /etc/hosts | grep -v 127| grep $(hostname) > /opt/bro/hosts 13 | salt-cp "*$BROS*" /opt/bro/hosts /opt/bro/hosts 14 | salt "*$BROS*" cmd.run 'cat /opt/bro/hosts >> /etc/hosts' 15 | salt "*$BROS*" cmd.run 'chown -R bro:bro /opt/bro;' 16 | 17 | echo "[worker-10.242.11.183]" >> /opt/bro/etc/node.cfg 18 | echo "type=worker" >> /opt/bro/etc/node.cfg 19 | echo "host=10.242.11.183" >> /opt/bro/etc/node.cfg 20 | echo "interface=eth1" >> /opt/bro/etc/node.cfg 21 | 22 | echo "Host 10.242.11.183" >> /opt/bro/.ssh/config 23 | echo " Hostname 10.242.11.183" >> /opt/bro/.ssh/config 24 | echo " IdentityFile ~/.ssh/id_rsa" >> /opt/bro/.ssh/config 25 | echo " StrictHostKeyChecking no" >> /opt/bro/.ssh/config 26 | 27 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl status' 28 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl deploy' 29 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl stop' 30 | salt "*$BROS*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/bro' 31 | ``` 32 | -------------------------------------------------------------------------------- /bro/day_2/fixdeceasedmaster.md: -------------------------------------------------------------------------------- 1 | # fix deceased master ;( 2 | 3 | ``` 4 | ssh someuser@192.168.1.2 'sudo service salt-minion stop; sudo rm -rf /etc/salt/pki /var/cache/salt /var/run/salt; sudo service salt-minion start; sudo service salt-minion restart;' 5 | ``` 6 | 7 | * see also pssh ;) 8 | -------------------------------------------------------------------------------- /bro/day_3/BasicConf.md: -------------------------------------------------------------------------------- 1 | # Configuration 2 | 3 | > These are the basic configuration changes to make for a minimal BroControl installation that will manage a single Bro instance on the localhost: 4 | 5 | >In $PREFIX/etc/node.cfg, set the right interface to monitor. 6 | >In $PREFIX/etc/networks.cfg, comment out the default settings and add the networks that Bro will consider local to the monitored environment. 7 | >In $PREFIX/etc/broctl.cfg, change the MailTo email address to a desired recipient and the LogRotationInterval to a desired log archival frequency. 8 | 9 | * https://www.bro.org/sphinx/quickstart/index.html#a-minimal-starting-configuration 10 | -------------------------------------------------------------------------------- /bro/day_3/LoadPcaps.md: -------------------------------------------------------------------------------- 1 | # Load Pcaps (...or bash for loop 101) 2 | 3 | * https://www.bro.org/sphinx-git/frameworks/logging.html 4 | * https://www.bro.org/bro-exchange-2013/exercises/faf.html 5 | * [In case you have a lot of large pcaps](/bro/day_3/threadHack.sh) 6 | 7 | ![nosweat!](/bro/day_3/nosweat.png) 8 | 9 | ## Checksum errors 10 | 11 | ``` 12 | 1449511310.447338 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted. 13 | ``` 14 | 15 | ## Fix this issue 16 | 17 | ``` 18 | WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. 19 | ``` -------------------------------------------------------------------------------- /bro/day_3/README.md: -------------------------------------------------------------------------------- 1 | https://github.com/CrowdStrike/cs-bro 2 | https://github.com/michalpurzynski/bro-gramming 3 | -------------------------------------------------------------------------------- /bro/day_3/csv2elasticsearch.md: -------------------------------------------------------------------------------- 1 | # CSV 2 ELASTIC 2 | 3 | > on your manager box as user root 4 | 5 | 1. pip install csv2es 6 | 1. csv2es --help 7 | 1. mkdir -p /home/yellow/tmp 8 | 1. cd /home/yellow/tmp/ 9 | 1. zcat /opt/bro/logs/2016-03-16/known_services.2* | head 10 | 1. zcat /opt/bro/logs/2016-03-16/known_services.22\:05\:17-23\:00\:00.log.gz > test.csv 11 | 1. head test.csv 12 | 1. vi test.csv 13 | 1. csv2es --host http://10.242.11.XX0:9200/ --tab --delete-index --index-name known --doc-type known_services --import-file test.csv 14 | 1. open http://10.242.11.XX0:9200/_plugin/head 15 | 16 | 17 | see 18 | * https://github.com/search?utf8=%E2%9C%93&q=csv+json+elasticsearch 19 | * http://www.codedependant.net/2012/04/12/handling-large-files-with-nodejs-and-elastic-searc/ 20 | 21 | > Now an often over looked fact when dealing with IO is that, typically, computers can read files much faster than they can write a file. 22 | 23 | ---- 24 | 25 | 26 | curl -XPOST "http://10.242.11.180:9200/known/_search?pretty" -d' 27 | { 28 | "size": 0, 29 | "aggregations": { 30 | "service": { 31 | "terms": { 32 | "field": "service" 33 | }, 34 | "aggs": { 35 | "servers": { 36 | "terms": { 37 | "field": "host" 38 | } 39 | } 40 | } 41 | } 42 | } 43 | }' 44 | -------------------------------------------------------------------------------- /bro/day_3/nosweat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/bro/day_3/nosweat.png -------------------------------------------------------------------------------- /bro/day_3/scripts2cluster.md: -------------------------------------------------------------------------------- 1 | # Exercise: load your custom script to cluster 2 | 3 | > Local policy scripts are located in share/bro/site. In the stand-alone setup, a single file called local.bro gets loaded automatically. In the cluster setup, the same local.bro gets loaded, followed by one of three other files: *local-manager.bro, local-worker.bro*, and *local-proxy.bro* are loaded by the manager, workers, and proxy, respectively. 4 | 5 | Try editing local.bro and update cluster conf: 6 | 7 | 1. su - bro -s /bin/bash -c 'bin/broctl start' 8 | 1. su - bro -s /bin/bash -c 'bin/broctl status' 9 | 1. su - bro -s /bin/bash -c 'bin/broctl check' 10 | 1. vi /opt/bro/share/bro/site/local.bro 11 | 1. su - bro -s /bin/bash -c 'bin/broctl check' 12 | 1. su - bro -s /bin/bash -c 'bin/broctl update' 13 | 1. su - bro -s /bin/bash -c 'bin/broctl status' 14 | 15 | ---- 16 | 17 | Get and add a new script from https://gist.github.com/search?utf8=%E2%9C%93&q=language%3Abro 18 | 19 | 1. mkdir /opt/bro/share/bro/site/customscripts 20 | 1. cd /opt/bro/share/bro/site/customscripts 21 | 1. wget .... 22 | 1. vi /opt/bro/share/bro/site/local.bro 23 | 1. su - bro -s /bin/bash -c 'bin/broctl check' 24 | 1. su - bro -s /bin/bash -c 'bin/broctl update' 25 | 1. su - bro -s /bin/bash -c 'bin/broctl status' 26 | 1. su - bro -s /bin/bash -c 'bin/broctl scripts' 27 | 1. su - bro -s /bin/bash -c 'bin/broctl scripts' | grep customscripts 28 | 29 | No scripts were sent to other instances? Explanation: 30 | 31 | > update - Update configuration of nodes on the fly 32 | 33 | > install - Update broctl installation/configuration 34 | 35 | Scripts are part of 'installation', so you need to run 'install' instead of 'update'. 36 | 37 | ``` 38 | ... 39 | su - bro -s /bin/bash -c 'bin/broctl install' 40 | ... 41 | ``` 42 | 43 | ---- 44 | 45 | > BROS="worker" 46 | > salt "\*$BROS\*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/bro' 47 | -------------------------------------------------------------------------------- /bro/day_3/threadHack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | thread_hack() { 4 | logdir=$1 5 | filepath=$2 6 | count_cmd=$(pgrep -c bro) 7 | threads_max=`grep '^core id' /proc/cpuinfo | wc -l` 8 | echo $logdir 9 | if [ $count_cmd -lt $threads_max ] ; then 10 | echo "Processing $filepath" 11 | if [ ! -d $logdir ]; then echo "$logdir does not exist. Creating"; mkdir -p $logdir; fi 12 | cd $logdir && $BROBIN -r $filepath local & 13 | else 14 | echo "Sleeping" 15 | sleep 1 16 | thread_hack $logdir $filepath 17 | fi 18 | } 19 | BASEDIR=/vagrant/pcap/ 20 | DESTDIR=/vagrant/log/ 21 | 22 | BROBIN='/usr/local/bro/bin/bro' 23 | 24 | export LANG=C 25 | export LC_ALL=C 26 | 27 | mkdir -p $DESTDIR 28 | 29 | for pcap_path in `find $BASEDIR -type f -name '*.pcap'`; do 30 | pcap_log_dir="$DESTDIR/`echo $pcap_path | perl -ne 'if (m/.+\/([\d\w-]+)\.pcap/){ print "$1\n"; } '`" 31 | thread_hack $pcap_log_dir $pcap_path 32 | done -------------------------------------------------------------------------------- /bro/day_intro/AptGetInstall.md: -------------------------------------------------------------------------------- 1 | # Installing from repository 2 | 3 | * https://www.bro.org/download/packages.html 4 | 5 | # Modify apt sources list file 6 | 7 | Remember to edit the following line to reflect your Ubuntu version. I.e. package compiled for 14.10 will not work on 14.04. 8 | 9 | ``` 10 | echo 'deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/bro.list 11 | ``` 12 | 13 | You can verify with the following command. 14 | 15 | ``` 16 | cat /etc/lsb-release 17 | ``` 18 | 19 | # Add release key 20 | 21 | ``` 22 | wget http://download.opensuse.org/repositories/network:bro/xUbuntu_14.04/Release.key 23 | sudo apt-key add - < Release.key 24 | ``` 25 | 26 | # Update package lists 27 | 28 | ``` 29 | apt-get update 30 | ``` 31 | 32 | # Install bro 33 | 34 | ``` 35 | apt-get -y install bro 36 | ``` 37 | 38 | # Verify APT package version 39 | 40 | ``` 41 | root@bro-empty:~# apt-cache policy bro 42 | bro: 43 | Installed: 2.4.1-0 44 | Candidate: 2.4.1-0 45 | Version table: 46 | *** 2.4.1-0 0 47 | 500 http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ Packages 48 | 100 /var/lib/dpkg/status 49 | ``` 50 | 51 | # Deploy scripts and verify daemon 52 | 53 | ``` 54 | /opt/bro/bin/broctl deploy 55 | ``` 56 | 57 | ``` 58 | /opt/bro/bin/broctl status 59 | ``` 60 | 61 | # Update only 62 | 63 | ``` 64 | /opt/bro/bin/broctl install 65 | ``` 66 | 67 | # RTFM 68 | 69 | ``` 70 | root@bro-empty:/opt/bro/etc# /opt/bro/bin/broctl help 71 | 72 | BroControl Version 1.4 73 | 74 | capstats [] [] - Report interface statistics with capstats 75 | check [] - Check configuration before installing it 76 | cleanup [--all] [] - Delete working dirs (flush state) on nodes 77 | config - Print broctl configuration 78 | cron [--no-watch] - Perform jobs intended to run from cron 79 | cron enable|disable|? - Enable/disable "cron" jobs 80 | deploy - Check, install, and restart 81 | df [] - Print nodes' current disk usage 82 | diag [] - Output diagnostics for nodes 83 | exec - Execute shell command on all hosts 84 | exit - Exit shell 85 | install - Update broctl installation/configuration 86 | netstats [] - Print nodes' current packet counters 87 | nodes - Print node configuration 88 | peerstatus [] - Print status of nodes' remote connections 89 | print [] - Print values of script variable at nodes 90 | process [] [-- ] - Run Bro (with options and scripts) on trace 91 | quit - Exit shell 92 | restart [--clean] [] - Stop and then restart processing 93 | scripts [-c] [] - List the Bro scripts the nodes will load 94 | start [] - Start processing 95 | status [] - Summarize node status 96 | stop [] - Stop processing 97 | top [] - Show Bro processes ala top 98 | update [] - Update configuration of nodes on the fly 99 | 100 | Commands provided by plugins: 101 | 102 | ps.bro [] - Show Bro processes on nodes' systems 103 | ``` -------------------------------------------------------------------------------- /bro/day_intro/BasicConf.md: -------------------------------------------------------------------------------- 1 | # Configuration 2 | 3 | > These are the basic configuration changes to make for a minimal BroControl installation that will manage a single Bro instance on the localhost: 4 | 5 | ``` 6 | - base directory for you installation 7 | ``` 8 | 9 | >In $PREFIX/etc/node.cfg, set the right interface to monitor. 10 | >In $PREFIX/etc/networks.cfg, comment out the default settings and add the networks that Bro will consider local to the monitored environment. 11 | >In $PREFIX/etc/broctl.cfg, change the MailTo email address to a desired recipient and the LogRotationInterval to a desired log archival frequency. 12 | 13 | * https://www.bro.org/sphinx/quickstart/index.html#a-minimal-starting-configuration 14 | 15 | ``` 16 | root@bro-empty:/opt/bro/etc# ls -la /opt/bro/etc/ 17 | total 24 18 | drwxrwsr-x 2 root bro 4096 Mar 8 14:01 . 19 | drwxr-xr-x 8 root root 4096 Mar 8 14:01 .. 20 | -rw-rw-r-- 1 root bro 2272 Sep 6 2015 broccoli.conf 21 | -rw-rw-r-- 1 root bro 2611 Sep 8 17:33 broctl.cfg 22 | -rw-rw-r-- 1 root bro 225 Sep 6 2015 networks.cfg 23 | -rw-rw-r-- 1 root bro 644 Sep 6 2015 node.cfg 24 | ``` 25 | 26 | # Home networks 27 | 28 | ``` 29 | cat /etc/networks.cfg 30 | ``` 31 | 32 | # Listening interface 33 | 34 | ``` 35 | grep 'interface' -B4 /etc/node.cfg 36 | ``` 37 | 38 | # Notifications, log rotation, directories 39 | 40 | ``` 41 | vim /etc/broctl.cfg 42 | ``` 43 | 44 | --- 45 | # BroControl 46 | 47 | Start up BroControl from the bin directory 48 | 49 | ``` 50 | /bin/broctl 51 | ``` 52 | 53 | The first time that you run BroControl, you must install the BroControl configuration 54 | ``` 55 | [BroControl] > install 56 | ``` 57 | 58 | To see help: 59 | ``` 60 | [BroControl] > help 61 | ``` 62 | 63 | ## Tasks 64 | * Print and examine the current broctl configuration 65 | * Start a Bro instance and check if it is running. 66 | * See if there are any logs already being generated? 67 | 68 | -------------------------------------------------------------------------------- /bro/day_intro/BasicLogging.md: -------------------------------------------------------------------------------- 1 | # Logging 2 | 3 | > Once Bro has been deployed in an environment and monitoring live traffic, it will, in its default configuration, begin to produce human-readable ASCII logs. Each log file, produced by Bro’s Logging Framework, is populated with organized, mostly connection-oriented data. As the standard log files are simple ASCII data, working with the data contained in them can be done from a command line terminal once you have been familiarized with the types of data that can be found in each file. 4 | 5 | 6 | * https://www.bro.org/sphinx/frameworks/logging.html#framework-logging 7 | * https://www.bro.org/sphinx/logs/index.html 8 | * https://www.bro.org/bro-workshop-2011/solutions/logs/ 9 | 10 | 11 | # Log types 12 | 13 | * [conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html) 14 | * [dns](https://www.bro.org/sphinx/scripts/base/protocols/dns/main.bro.html) 15 | * [http](https://www.bro.org/sphinx/scripts/base/protocols/http/main.bro.html) 16 | * [tls](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html) 17 | * [smtp](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html) 18 | * [notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html) 19 | * [weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html) 20 | * [etc...](https://www.bro.org/sphinx/script-reference/scripts.html) 21 | 22 | # Daemon logs 23 | 24 | ``` 25 | grep log /opt/bro/etc/broctl.cfg 26 | ``` 27 | 28 | ## Log archive 29 | 30 | ``` 31 | root@bro-empty:/opt/bro/etc# ls -la 32 | total 12 33 | drwxrws--- 3 root bro 4096 Mar 8 14:20 . 34 | drwxr-xr-x 8 root root 4096 Mar 8 14:01 .. 35 | drwxr-sr-x 2 root bro 4096 Mar 8 14:20 2016-03-08 36 | lrwxrwxrwx 1 root bro 18 Mar 8 14:17 current -> /opt/bro/spool/bro 37 | ``` 38 | 39 | ## Actively written logs 40 | 41 | ``` 42 | ls -la /opt/bro/spool/bro 43 | ``` 44 | 45 | # Runtime 46 | 47 | ``` 48 | cd /tmp 49 | /opt/bro/bin/bro -r local 50 | tail *.log 51 | ``` 52 | 53 | # ASCII tools 54 | 55 | 56 | ``` 57 | /opt/bro/bin/bro-cut --help 58 | /opt/bro/bin/bro-cut service resp_bytes id.resp_h < /opt/bro/logs/current/conn.log 59 | ``` 60 | 61 | Timestamps 62 | ``` 63 | /opt/bro/bin/bro-cut -d ts id.orig_h id.orig_p id.resp_h id.resp_p < /opt/bro/logs/current/conn.log 64 | ``` 65 | 66 | # Unique Event Identifier 67 | 68 | ``` 69 | /opt/bro/bin/bro-cut uid < /opt/bro/logs/current/conn.log 70 | ``` 71 | 72 | # Tasks 73 | 74 | * Extract hour, minute and second values in human readable format from connection logs. 75 | * Find corresponding log entries for each unique identifier in connection log. -------------------------------------------------------------------------------- /bro/day_intro/BasicScripting.md: -------------------------------------------------------------------------------- 1 | # scripting 2 | 3 | > Bro includes an event-driven scripting language that provides the primary means to extend and customize Bro’s functionality. 4 | 5 | see 6 | 7 | * https://www.bro.org/sphinx/scripting/index.html 8 | * https://www.bro.org/development/howtos/script-conventions.html 9 | * https://www.bro.org/sphinx/script-reference/index.html 10 | 11 | 12 | 13 | # Getting started with packaged scripts 14 | 15 | ``` 16 | find /opt/bro/ -type f -name 'local*' 17 | ``` 18 | -------------------------------------------------------------------------------- /bro/day_intro/README.md: -------------------------------------------------------------------------------- 1 | # BRO 2 | 3 | ![logo](https://www.bro.org/images/bro-eyes.png) 4 | 5 | ## https://www.bro.org 6 | 7 | > Bro is a powerful network analysis framework that is much different from the typical IDS you may know. 8 | 9 | > Key to understanding Bro lies in realizing that even though the system comes with such powerful functionality out of the box, fundamentally it represents a platform for traffic analyses that’s fully customizable and extensible: Bro provides users with a domain-specific, Turing-complete scripting language for expressing arbitrary analysis tasks. Conceptually, you can think of Bro as a “domain-specific Python” (or Perl): just like Python, the system comes with a large set of pre-built functionality (the “standard library”), yet you are not limited to what the system ships with but can put Bro to use in novel ways by writing your own code. Indeed, all of Bro’s default analyses, including all the logging, is the result of such scripts; there’s no specific analysis hard-coded into the core of system. 10 | 11 | ## https://github.com/bro/bro 12 | 13 | > git clone --recursive git://git.bro.org/bro 14 | 15 | #### please see 16 | 17 | * https://www.bro.org/ 18 | * https://www.bro.org/documentation/index.html 19 | * https://www.bro.org/sphinx/scripting/ -------------------------------------------------------------------------------- /bro/day_intro/brodello/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.112" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Brodello do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Brodello' 10 | box.vm.provision :shell, :path => "../BTIGEK-easybutton-singlehost.sh", args: "eth1" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /bro/day_intro/brodello/grafana.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/bro/day_intro/brodello/grafana.tar.gz -------------------------------------------------------------------------------- /bro/day_intro/dummy/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.111" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Dummy do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Dummy' 10 | box.vm.provision :shell, :path => "./scripts/dummy-install-bro.sh", args: "eth1" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /bro/day_intro/dummy/scripts/dummy-install-bro.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # this script 3 | # 1) adds bro to atp sources 4 | # 2) uodates debs list 5 | # 3) installs bro from package 6 | 7 | echo 'deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/bro.list 8 | apt-get update 9 | apt-get -y --force-yes install bro 10 | -------------------------------------------------------------------------------- /bro/day_intro/empty/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.110" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :empty do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Empty' 10 | box.vm.provision "shell", inline: "ifconfig; ping -c 1 www.ee; netstat -ntple" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /bro/day_intro/empty/scripts/dummy-install-bro.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # this script 3 | # 1) adds bro to atp sources 4 | # 2) uodates debs list 5 | # 3) installs bro from package 6 | 7 | echo 'deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/bro.list 8 | apt-get update 9 | apt-get -y --force-yes install bro 10 | -------------------------------------------------------------------------------- /bro/day_intro/timing.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo "# Timing" > ./timing.md 3 | cd empty/ 4 | echo "## Emtpy" >> ../timing.md 5 | echo "#### status" >> ../timing.md 6 | echo "\`\`\`bash" >> ../timing.md 7 | (time vagrant status) 1>> ../timing.md 2>> ../timing.md 8 | echo "\`\`\`" >> ../timing.md 9 | echo "#### up" >> ../timing.md 10 | echo "\`\`\`bash" >> ../timing.md 11 | (time vagrant up) 1>> ../timing.md 2>> ../timing.md 12 | echo "\`\`\`" >> ../timing.md 13 | echo "#### destroy" >> ../timing.md 14 | echo "\`\`\`bash" >> ../timing.md 15 | (time vagrant destroy -f) 1>> ../timing.md 2>> ../timing.md 16 | echo "\`\`\`" >> ../timing.md 17 | cd dummy/ 18 | echo "## Dummy" >> ../timing.md 19 | echo "#### status" >> ../timing.md 20 | echo "\`\`\`bash" >> ../timing.md 21 | (time vagrant status) 1>> ../timing.md 2>> ../timing.md 22 | echo "\`\`\`" >> ../timing.md 23 | echo "#### up" >> ../timing.md 24 | echo "\`\`\`bash" >> ../timing.md 25 | (time vagrant up) 1>> ../timing.md 2>> ../timing.md 26 | echo "\`\`\`" >> ../timing.md 27 | echo "#### destroy" >> ../timing.md 28 | echo "\`\`\`bash" >> ../timing.md 29 | (time vagrant destroy -f) 1>> ../timing.md 2>> ../timing.md 30 | echo "\`\`\`" >> ../timing.md 31 | -------------------------------------------------------------------------------- /bro/day_last/README.md: -------------------------------------------------------------------------------- 1 | 2 | * how did we get here 3 | * how does it all hang together 4 | * how much still remains to be understood 5 | -------------------------------------------------------------------------------- /bro/notes.txt: -------------------------------------------------------------------------------- 1 | 2 | 3 | https://github.com/EmergingThreats/bro 4 | -------------------------------------------------------------------------------- /common/AnyInstall.txt: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | SALT_MASTER=$1 4 | METRICS_SERVER=$2 5 | 6 | INSTALL_DIR=/provision 7 | 8 | echo "LC_ALL=en_US.UTF-8" >> /etc/environment 9 | apt-get -y install salt-minion 10 | echo "master: ${SALT_MASTER}" >> /etc/salt/minion 11 | service salt-minion restart 12 | 13 | 14 | TLGF=0.10.2 15 | 16 | mkdir -p ${INSTALL_DIR}/telegraf 17 | cd ${INSTALL_DIR}/telegraf 18 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 19 | wget -q wget http://get.influxdb.org/telegraf/telegraf_${TLGF}-1_amd64.deb 20 | fi 21 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 22 | echo "$(date) ${NAME} $0[$$]: {telegaf: {status:ERROR, msg: missing telegraf_${TLGF}_amd64.deb}" 23 | exit -1 24 | else 25 | echo -e "Y"|dpkg -i telegraf_${TLGF}-1_amd64.deb > /dev/null 26 | # urls = ["http://localhost:8086"] # required 27 | sed -i -e 's,http://localhost,http://'${METRICS_SERVER}',g' /etc/telegraf/telegraf.conf 28 | # interval = "10s" 29 | sed -i -e 's,interval = "10s",interval = "1s",g' /etc/telegraf/telegraf.conf 30 | # flush_interval = "10s" 31 | sed -i -e 's,flush_interval = "1s",flush_interval = "60s",g' /etc/telegraf/telegraf.conf 32 | echo "[[inputs.net]]" >> /etc/telegraf/telegraf.conf 33 | echo "[[inputs.netstat]]" >> /etc/telegraf/telegraf.conf 34 | 35 | service telegraf start 36 | fi 37 | -------------------------------------------------------------------------------- /common/Elastic.md: -------------------------------------------------------------------------------- 1 | # elasticsearch 2 | 3 | see 4 | * https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html 5 | * https://www.elastic.co/guide/en/elasticsearch/reference/2.2/modules-node.html 6 | * https://www.elastic.co/guide/en/elasticsearch/reference/2.2/modules-http.html#_disable_http 7 | 8 | 9 | 1. master :: node.master=true, node.data=false, http.enabled=true 10 | 1. data :: node.master=false, node.data=true, http.enabled=false 11 | 1. client :: node.master=false, node.data=false, http.enabled=true 12 | 13 | ``` 14 | 15 | wget -q https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-2.2.0.deb 16 | 17 | dpkg -i elasticsearch-${ES}.deb 18 | 19 | service elasticsearch stop 20 | 21 | /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head 22 | 23 | vi /ets/elasticsearch/elasticsearch.yml 24 | 25 | service elasticsearch start 26 | 27 | 28 | ``` 29 | -------------------------------------------------------------------------------- /common/ElasticInstall.txt: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | IP=$1 4 | NAME=$2 5 | CLUSTER=$3 6 | COUNTER=$4 7 | UNICASTHOSTS=$5 8 | TYPE=$6 #master,client or data(default) 9 | 10 | echo "adding new node to cluster: ${CLUSTER} node: ${NAME} bind: ${IP} unicast host: ${UNICASTHOSTS}" 11 | 12 | INSTALL_DIR=/provision 13 | 14 | ES=2.2.0 15 | 16 | mkdir -p ${INSTALL_DIR}/elasticsearch 17 | cd ${INSTALL_DIR}/elasticsearch 18 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 19 | wget -q https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES}.deb 20 | fi 21 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 22 | echo "$(date) ${NAME} $0[$$]: {elastic: {status:ERROR, msg: missing elasticsearch-${ES}.deb}" 23 | exit -1 24 | else 25 | echo -e "Y" | dpkg -i elasticsearch-${ES}.deb 26 | service elasticsearch stop 27 | /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head 28 | 29 | echo "cluster.name: ${CLUSTER}" > /etc/elasticsearch/elasticsearch.yml 30 | echo "node.name: ${NAME} " >> /etc/elasticsearch/elasticsearch.yml 31 | echo "node.max_local_storage_nodes: 1 " >> /etc/elasticsearch/elasticsearch.yml 32 | echo "index.number_of_replicas: 0 " >> /etc/elasticsearch/elasticsearch.yml 33 | echo "index.fielddata.cache: node " >> /etc/elasticsearch/elasticsearch.yml 34 | echo "indices.fielddata.cache.size: 40% " >> /etc/elasticsearch/elasticsearch.yml 35 | echo "http.compression: true " >> /etc/elasticsearch/elasticsearch.yml 36 | echo "bootstrap.mlockall: true " >> /etc/elasticsearch/elasticsearch.yml 37 | echo "network.host: {$IP}" >> /etc/elasticsearch/elasticsearch.yml 38 | echo "discovery.zen.minimum_master_nodes: 2" >> /etc/elasticsearch/elasticsearch.yml 39 | echo "discovery.zen.ping.multicast.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 40 | echo 'discovery.zen.ping.unicast.hosts: ["'${UNICASTHOSTS}'"]' >> /etc/elasticsearch/elasticsearch.yml 41 | if [ "$TYPE" == "master" ]; 42 | then 43 | echo "node.master: true" >> /etc/elasticsearch/elasticsearch.yml 44 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 45 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 46 | else 47 | if [ "$TYPE" == "client" ]; 48 | then 49 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 50 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 51 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 52 | else 53 | # defaults to data node 54 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 55 | echo "node.data: true" >> /etc/elasticsearch/elasticsearch.yml 56 | echo "http.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 57 | fi 58 | fi 59 | service elasticsearch start 60 | 61 | echo "[elasticsearch]" >> /etc/opt/telegraf/telegraf.conf 62 | echo 'servers = ["http://'${IP}':9200"]' >> /etc/opt/telegraf/telegraf.conf 63 | echo " local = true" >> /etc/opt/telegraf/telegraf.conf 64 | echo " cluster_health = false" >> /etc/opt/telegraf/telegraf.conf 65 | service telegraf restart 66 | fi 67 | -------------------------------------------------------------------------------- /common/ElasticsearchAttachemnt.md: -------------------------------------------------------------------------------- 1 | # Mapper Attachments Type for Elasticsearch 2 | 3 | see 4 | 5 | * https://github.com/elastic/elasticsearch-mapper-attachments/tree/v3.1.1/#version-311-for-elasticsearch-21 6 | 7 | 8 | ``` 9 | 10 | apt-get install -y openjdk-7-jre-headless 11 | 12 | wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.1.1/elasticsearch-2.1.1.deb 13 | 14 | sudo dpkg -i elasticsearch-2.1.1.deb 15 | 16 | /usr/share/elasticsearch/bin/plugin install elasticsearch/elasticsearch-mapper-attachments/3.1.1 17 | 18 | ``` 19 | -------------------------------------------------------------------------------- /common/Kibana4.md: -------------------------------------------------------------------------------- 1 | # Kibana4 2 | 3 | * https://www.elastic.co/products/kibana 4 | * https://www.elastic.co/guide/en/kibana/current/index.html 5 | * https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations 6 | * http://www.lucenetutorial.com/lucene-query-syntax.html 7 | * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html 8 | 9 | ## Config file 10 | 11 | ``` 12 | grep "elasticsearch.url" /opt/kibana/config/kibana.yml 13 | ``` 14 | 15 | ## Lucene search syntax 16 | 17 | ``` 18 | event_type:"alert" AND alert.severity:2 19 | ``` 20 | 21 | --- 22 | [next : looking at the numbers](/suricata/day_2/Grafana.md) 23 | -------------------------------------------------------------------------------- /common/MertixInstall.txt: -------------------------------------------------------------------------------- 1 | #METRICS_SERVER=$1 2 | 3 | METRICS_SERVER=$(hostname -f) 4 | 5 | 6 | 7 | INSTALL_DIR=/provision 8 | 9 | INFLX="0.10.0" 10 | 11 | mkdir -p ${INSTALL_DIR}/influxdb 12 | cd ${INSTALL_DIR}/influxdb 13 | if [ ! -f "influxdb_${INFLX}-1_amd64.deb" ]; then 14 | wget -q https://s3.amazonaws.com/influxdb/influxdb_${INFLX}-1_amd64.deb 15 | fi 16 | if [ ! -f "influxdb_${INFLX}-1_amd64.deb" ]; then 17 | echo "$(date) ${NAME} $0[$$]: {influxdb: {status:ERROR, msg: missing influxdb_${INFLX}-1_amd64.deb}" 18 | exit -1 19 | else 20 | dpkg -i influxdb_${INFLX}-1_amd64.deb 21 | #prepare for telegraf 22 | curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE telegraf" 23 | curl -G http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY one_day_only ON telegraf DURATION 1d REPLICATION 1 DEFAULT" 24 | #sed -i -e 's,localhost,'${METRICS_SERVER}',g' /etc/influxdb/influxdb.conf 25 | #service influxdb restart 26 | fi 27 | 28 | TLGF=0.10.2 29 | 30 | mkdir -p ${INSTALL_DIR}/telegraf 31 | cd ${INSTALL_DIR}/telegraf 32 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 33 | wget -q wget http://get.influxdb.org/telegraf/telegraf_${TLGF}-1_amd64.deb 34 | fi 35 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 36 | echo "$(date) ${NAME} $0[$$]: {telegaf: {status:ERROR, msg: missing telegraf_${TLGF}_amd64.deb}" 37 | exit -1 38 | else 39 | echo -e "Y"|dpkg -i telegraf_${TLGF}-1_amd64.deb > /dev/null 40 | # urls = ["http://localhost:8086"] # required 41 | sed -i -e 's,http://localhost,http://'${METRICS_SERVER}',g' /etc/telegraf/telegraf.conf 42 | # interval = "10s" 43 | sed -i -e 's,interval = "10s",interval = "1s",g' /etc/telegraf/telegraf.conf 44 | # flush_interval = "10s" 45 | sed -i -e 's,flush_interval = "1s",flush_interval = "60s",g' /etc/telegraf/telegraf.conf 46 | echo "[[inputs.net]]" >> /etc/telegraf/telegraf.conf 47 | echo "[[inputs.netstat]]" >> /etc/telegraf/telegraf.conf 48 | 49 | service telegraf start 50 | fi 51 | 52 | 53 | 54 | KPCTR=0.10.0 55 | 56 | mkdir -p ${INSTALL_DIR}/kapacitor 57 | cd ${INSTALL_DIR}/kapacitor 58 | if [ ! -f "kapacitor_${KPCTR}-1_amd64.deb" ]; then 59 | wget -q https://s3.amazonaws.com/influxdb/kapacitor_${KPCTR}-1_amd64.deb 60 | fi 61 | if [ ! -f "kapacitor_${KPCTR}-1_amd64.deb" ]; then 62 | echo "$(date) ${NAME} $0[$$]: {kapacitor: {status:WARNING, msg: missing kapacitor_${KPCTR}-1_amd64.deb}" 63 | #exit -1 64 | else 65 | dpkg -i kapacitor_${KPCTR}-1_amd64.deb 66 | # urls = ["http://localhost:8086"] 67 | sed -i -e 's,hostname = "localhost",hostname = "'${METRICS_SERVER}'",g' /etc/kapacitor/kapacitor.conf 68 | urls = ["http://localhost:8086"] 69 | #sed -i -e 's,urls = urls = ["http://localhost:8086"],urls = ["http://'${METRICS_SERVER}':8086"],g' /etc/kapacitor/kapacitor.conf 70 | service kapacitor start 71 | kapacitor define -name cpu_alert -type stream -tick ./cpu_alert.tick -dbrp telegraf.default 72 | fi 73 | 74 | CHRNGRF=0.10.0 75 | 76 | mkdir -p ${INSTALL_DIR}/chronograf 77 | cd ${INSTALL_DIR}/chronograf 78 | if [ ! -f "chronograf_${CHRNGRF}_amd64.deb" ]; then 79 | wget -q https://s3.amazonaws.com/get.influxdb.org/chronograf/chronograf_${CHRNGRF}_amd64.deb 80 | fi 81 | if [ ! -f "chronograf_${CHRNGRF}_amd64.deb" ]; then 82 | echo "$(date) ${NAME} $0[$$]: {chronograf: {status:WARNING, msg: missing chronograf_${CHRNGRF}_amd64.deb}" 83 | else 84 | dpkg -i chronograf_${CHRNGRF}_amd64.deb 85 | sed -i -e 's,Bind = "127.0.0.1:10000",Bind = "'${METRICS_SERVER}':10000",g' /opt/chronograf/config.toml 86 | fi 87 | 88 | GRFN="2.6.0" 89 | mkdir -p ${INSTALL_DIR}/grafana 90 | cd ${INSTALL_DIR}/grafana 91 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 92 | wget -q https://grafanarel.s3.amazonaws.com/builds/grafana_${GRFN}_amd64.deb 93 | fi 94 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 95 | echo "$(date) ${NAME} $0[$$]: {grafana: {status:WARNING, msg: missing grafana_${GRFN}_amd64.deb}" 96 | #exit -1 97 | else 98 | apt-get install -y adduser libfontconfig 99 | dpkg -i grafana_${GRFN}_amd64.deb 100 | update-rc.d grafana-server defaults 95 10 101 | sed -i -e 's,domain = localhost,domain = '${METRICS_SERVER}',g' /etc/grafana/grafana.ini 102 | service grafana-server start 103 | fi 104 | -------------------------------------------------------------------------------- /common/Telegraf.md: -------------------------------------------------------------------------------- 1 | # Telegraf 2 | -------------------------------------------------------------------------------- /common/git-most.md: -------------------------------------------------------------------------------- 1 | # gitlab + mattermost 2 | 3 | ```bash 4 | #!/bin/bash 5 | # 6 | # this script installs gitlab and enables mattermost 7 | # 8 | 9 | if [ "$(id -u)" != "0" ]; then 10 | echo "ERROR - This script must be run as root" 1>&2 11 | exit 1 12 | fi 13 | 14 | curl -s https://packages.gitlab.com/install/repositories/gitlab/nightly-builds/script.deb.sh | sudo bash 15 | DEBIAN_FRONTEND=noninteractive apt-get install -y gitlab-ce 16 | 17 | cat < /etc/gitlab/gitlab.rb 18 | 19 | #auto generated by $0 on $(date) 20 | external_url 'http://gitlab.test' 21 | logrotate['enable'] = true 22 | mattermost_external_url 'http://mattermost.test' 23 | mattermost['enable'] = true 24 | mattermost['team_site_name'] = "test chat" 25 | mattermost['team_enable_team_listing'] = true 26 | 27 | EOT 28 | 29 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 30 | 31 | cat <> /etc/hosts 32 | #auto added by $0 on $(date) 33 | $IP gitlab.test 34 | $IP mattermost.test 35 | #end 36 | EOT 37 | 38 | gitlab-ctl reconfigure 39 | 40 | #apt-get install ruby-dev 41 | #gitlab create_user email=john@olindata.com name='john Doe' username=john 42 | #sudo -u mattermost /opt/gitlab/embedded/bin/mattermost -config /var/opt/gitlab/mattermost/config.json -version 43 | #sudo -u mattermost /opt/gitlab/embedded/bin/mattermost -config /var/opt/gitlab/mattermost/config.json -create_user -team_name="ls16" -email="user3@example.com" -password="mypassword" 44 | ``` 45 | -------------------------------------------------------------------------------- /common/metrics: -------------------------------------------------------------------------------- 1 | # collect some metrics 2 | 3 | ``` 4 | METRICS_SERVER=$(hostname -f) 5 | ``` 6 | 7 | ## influxdb 8 | 9 | see 10 | * https://github.com/influxdata/influxdb#features 11 | * https://github.com/influxdata/influxdb-r 12 | 13 | ``` 14 | 15 | 16 | wget -q https://s3.amazonaws.com/influxdb/influxdb_nightly_amd64.deb 17 | sudo dpkg -i influxdb_nightly_amd64.deb 18 | sed -i -e 's,localhost,'${METRICS_SERVER}',g' /etc/influxdb/influxdb.conf 19 | service influxdb restart 20 | 21 | ``` 22 | 23 | ## telegraf 24 | 25 | see https://github.com/influxdata/telegraf#telegraf- 26 | 27 | ``` 28 | TLGF=0.2.4 29 | 30 | wget -q https://s3.amazonaws.com/get.influxdb.org/telegraf/telegraf_${TLGF}_amd64.deb 31 | sudo dpkg -i telegraf_${TLGF}_amd64.deb 32 | sed -i -e 's,http://localhost,http://'${METRICS_SERVER}',g' /etc/opt/telegraf/telegraf.conf 33 | sed -i -e 's,interval = "10s",interval = "1s",g' /etc/opt/telegraf/telegraf.conf 34 | sed -i -e 's,flush_interval = "1s",flush_interval = "60s",g' /etc/opt/telegraf/telegraf.conf 35 | service telegraf restart 36 | 37 | ``` 38 | 39 | ## kapacitor 40 | 41 | see https://github.com/influxdata/kapacitor#basic-example 42 | 43 | 44 | ``` 45 | KPCTR=0.2.4 46 | 47 | wget -q https://s3.amazonaws.com/influxdb/kapacitor_${KPCTR}-1_amd64.deb 48 | sudo dpkg -i kapacitor_${KPCTR}-1_amd64.deb 49 | sed -i -e 's,hostname = "localhost",hostname = "'${METRICS_SERVER}'",g' /etc/kapacitor/kapacitor.conf 50 | urls = ["http://localhost:8086"] 51 | sed -i -e 's,urls = urls = ["http://localhost:8086"],urls = ["http://'${METRICS_SERVER}':8086"],g' /etc/kapacitor/kapacitor.conf 52 | service kapacitor restart 53 | 54 | kapacitor define -name cpu_alert -type stream -tick ./cpu_alert.tick -dbrp telegraf.default 55 | 56 | ``` 57 | 58 | ## chronograf 59 | 60 | see https://docs.influxdata.com/chronograf/v0.4/ 61 | 62 | ``` 63 | wget https://s3.amazonaws.com/get.influxdb.org/chronograf/chronograf_0.4.0_amd64.deb 64 | sudo dpkg -i chronograf_0.4.0_amd64.deb 65 | ``` 66 | 67 | ## grafana 68 | 69 | see http://grafana.org 70 | 71 | 72 | ``` 73 | wget -q https://grafanarel.s3.amazonaws.com/builds/grafana_latest_amd64.deb 74 | sudo dpkg -i grafana_latest_amd64.deb 75 | sudo update-rc.d grafana-server defaults 95 10 76 | sed -i -e 's,domain = localhost,domain = '${METRICS_SERVER}',g' /etc/grafana/grafana.ini 77 | service grafana-server start 78 | ``` 79 | -------------------------------------------------------------------------------- /common/pfring.md: -------------------------------------------------------------------------------- 1 | # PF RING 2 | 3 | 4 | ``` 5 | wget http://sourceforge.net/projects/ntop/files/PF_RING/PF_RING-6.0.3.tar.gz 6 | tar xvzf PF_RING-6.0.3.tar.gz 7 | cd PF_RING-6.0.3/ 8 | make 9 | cd kernel 10 | sudo make install 11 | cd ../userland/lib 12 | sudo make install 13 | 14 | 15 | 16 | ``` 17 | 18 | 19 | 20 | 21 | ``` 22 | wget http://apt.ntop.org/14.04/all/apt-ntop.deb 23 | sudo dpkg -i apt-ntop.deb 24 | sudo apt-get clean all 25 | sudo apt-get update 26 | sudo apt-get install pfring 27 | ``` 28 | -------------------------------------------------------------------------------- /common/rsyslogBasics.md: -------------------------------------------------------------------------------- 1 | # Rsyslog 2 | 3 | * http://www.rsyslog.com/ubuntu-repository/ 4 | * http://www.rsyslog.com/tag/mmjsonparse/ 5 | * http://www.rsyslog.com/doc/mmjsonparse.html 6 | * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html 7 | 8 | ``` 9 | apt-cache policy rsyslog 10 | rsyslog: 11 | Installed: 7.4.4-1ubuntu2.6 12 | Candidate: 8.16.0-0adiscon1trusty1 13 | Version table: 14 | 8.16.0-0adiscon1trusty1 0 15 | 500 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu/ trusty/main amd64 Packages 16 | *** 7.4.4-1ubuntu2.6 0 17 | 500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages 18 | 100 /var/lib/dpkg/status 19 | ``` 20 | 21 | # Installing missing modules 22 | 23 | ``` 24 | sudo apt-get install rsyslog-mmjsonparse rsyslog-elasticsearch -y 25 | ``` 26 | 27 | ``` 28 | sudo service rsyslog restart 29 | ``` 30 | 31 | # Verify daemon 32 | 33 | ``` 34 | grep rsyslogd /var/log/syslog 35 | ``` 36 | 37 | ``` 38 | Feb 25 11:47:56 box rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="2499" x-info="http://www.rsyslog.com"] exiting on signal 15. 39 | Feb 25 11:47:56 box rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="31019" x-info="http://www.rsyslog.com"] start 40 | Feb 25 11:47:56 box rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.16.0 try http://www.rsyslog.com/e/2307 ] 41 | Feb 25 11:47:56 box rsyslogd: rsyslogd's groupid changed to 104 42 | Feb 25 11:47:56 box rsyslogd: rsyslogd's userid changed to 101 -------------------------------------------------------------------------------- /common/vagrant/CreateUbu14Box.bash: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # create and add updated & upgraded ubuntu 14 box to vagrant box list 4 | # time 5-10ish minutes 5 | 6 | # exit if ubu14 is listed as a box 7 | if [ "$(vagrant box list|grep ubu14|awk '{print $1}')" = "ubu14" ] 8 | then 9 | echo "ubu14 is already listed, nothing to do ;)" 10 | echo "simply run" 11 | echo "vagrant init ubu14" 12 | exit 13 | fi 14 | 15 | # exit if Vagrantfile exist 16 | if [ -f "./Vagrantfile" ]; 17 | then 18 | echo "./Vagrantfile exist, exiting"; 19 | exit 20 | fi 21 | 22 | # create ./Vagrantfile 23 | cat > ./Vagrantfile < /etc/vagrant_provisioned_at 27 | apt-get -y --purge remove nfs-kernel-server nfs-common portmap rpcbind puppet puppet-common chef chef-zero 28 | apt-get update >> /etc/vagrant_provisioned_at 29 | apt-get upgrade >> /etc/vagrant_provisioned_at 30 | apt-get -y install openjdk-7-jdk >> /etc/vagrant_provisioned_at 31 | apt-get -y install git >> /etc/vagrant_provisioned_at 32 | apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev build-essential autoconf automake libtool libpcap-dev libnet1-dev libyaml-0-2 libyaml-dev pkg-config zlib1g zlib1g-dev libcap-ng-dev libcap-ng0 make libmagic-dev >> /etc/vagrant_provisioned_at 33 | apt-get clean 34 | date >> /etc/vagrant_provisioned_at 35 | SCRIPT 36 | Vagrant.configure(2) do |config| 37 | config.vm.synced_folder ".", "/vagrant", disabled: true 38 | config.vm.box = "ubuntu/trusty64" 39 | config.vm.provision "shell", inline: \$script 40 | end 41 | DELIM 42 | 43 | # run & provision & package & add & clean up 44 | vagrant up 45 | vagrant halt 46 | vagrant package --output ubu14.box 47 | vagrant box add ubu14 ubu14.box 48 | vagrant destroy -f 49 | rm ./Vagrantfile 50 | rm ./ubu14.box 51 | rmdir -p .vagrant/machines/default/virtualbox/ 52 | 53 | if [ "$(vagrant box list|grep ubu14|awk '{print $1}')" = "ubu14" ] 54 | then 55 | echo "ubu14 is now listed as available box, done" 56 | echo "simply run" 57 | echo "vagrant init ubu14" 58 | 59 | fi 60 | -------------------------------------------------------------------------------- /common/vagrant/bb/keys/vm.rb: -------------------------------------------------------------------------------- 1 | module VMKeys 2 | VM_USER = 'yellow' 3 | VM_PASSWORD = 'yellow' 4 | end 5 | -------------------------------------------------------------------------------- /common/vagrant/bb/keys/vsphere.rb: -------------------------------------------------------------------------------- 1 | module VSphereKeys 2 | VSPHERE_USER = 'Hillar.Aarelaid' 3 | VSPHERE_PASSWORD = 'secret.passowrd' 4 | VSPHERE_HOST = '10.0.21.122' 5 | VSPHERE_CLUSTER = 'BigCluster' 6 | VSPHERE_TEMPLATE_PATH = '/Development/Hillar/' 7 | VSPHERE_VM_PATH = '/Testing/Hillar/' 8 | end 9 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-bro.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | echo 'deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/bro.list 5 | sudo apt-get update 6 | sudo apt-get install bro 7 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-elastic.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) installs elasticsearch as 5 | # a) master no data 6 | # b) client no data 7 | # c) dana no http 8 | # 2) set unicast hosts 9 | # 3) sets cluster 10 | # 4) add elasticsearch to telegraf config 11 | # 12 | 13 | 14 | if [ "$(id -u)" != "0" ]; then 15 | echo "ERROR - This script must be run as root" 1>&2 16 | exit 1 17 | fi 18 | 19 | IP=$1 20 | NAME=$2 21 | CLUSTER=$3 22 | COUNTER=$4 23 | UNICASTHOSTS=$5 24 | TYPE=$6 #master,client or data(default) 25 | 26 | echo "adding new node to cluster: ${CLUSTER} node: ${NAME} bind: ${IP} unicast host: ${UNICASTHOSTS}" 27 | 28 | INSTALL_DIR=/provision 29 | 30 | ES=2.2.0 31 | 32 | mkdir -p ${INSTALL_DIR}/elasticsearch 33 | cd ${INSTALL_DIR}/elasticsearch 34 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 35 | wget -4 -q https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES}.deb 36 | fi 37 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 38 | echo "$(date) ${NAME} $0[$$]: {elastic: {status:ERROR, msg: missing elasticsearch-${ES}.deb}" 39 | exit -1 40 | else 41 | echo -e "Y" | dpkg -i elasticsearch-${ES}.deb 2>&1 > /dev/null 42 | service elasticsearch stop 43 | /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head 2>&1 > /dev/null 44 | echo "# generated ${date} by $0" > /etc/elasticsearch/elasticsearch.yml 45 | echo "cluster.name: ${CLUSTER}" >> /etc/elasticsearch/elasticsearch.yml 46 | echo "node.name: ${NAME} " >> /etc/elasticsearch/elasticsearch.yml 47 | echo "node.max_local_storage_nodes: 1 " >> /etc/elasticsearch/elasticsearch.yml 48 | echo "index.number_of_replicas: 0 " >> /etc/elasticsearch/elasticsearch.yml 49 | echo "index.fielddata.cache: node " >> /etc/elasticsearch/elasticsearch.yml 50 | echo "indices.fielddata.cache.size: 40% " >> /etc/elasticsearch/elasticsearch.yml 51 | echo "http.compression: true " >> /etc/elasticsearch/elasticsearch.yml 52 | echo "bootstrap.mlockall: true " >> /etc/elasticsearch/elasticsearch.yml 53 | echo "network.host: ${IP}" >> /etc/elasticsearch/elasticsearch.yml 54 | echo "# split brain here ;( " >> /etc/elasticsearch/elasticsearch.yml 55 | echo "discovery.zen.minimum_master_nodes: 1" >> /etc/elasticsearch/elasticsearch.yml 56 | echo "discovery.zen.ping.multicast.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 57 | echo 'discovery.zen.ping.unicast.hosts: ['${UNICASTHOSTS}']' >> /etc/elasticsearch/elasticsearch.yml 58 | if [ "$TYPE" == "master" ]; 59 | then 60 | echo "node.master: true" >> /etc/elasticsearch/elasticsearch.yml 61 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 62 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 63 | else 64 | if [ "$TYPE" == "client" ]; 65 | then 66 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 67 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 68 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 69 | else 70 | # defaults to data node 71 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 72 | echo "node.data: true" >> /etc/elasticsearch/elasticsearch.yml 73 | echo "http.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 74 | fi 75 | fi 76 | service elasticsearch start 77 | 78 | echo "[elasticsearch]" >> /etc/telegraf/telegraf.conf 79 | echo 'servers = ["http://'${IP}':9200"]' >> /etc/telegraf/telegraf.conf 80 | echo " local = true" >> /etc/telegraf/telegraf.conf 81 | echo " cluster_health = false" >> /etc/telegraf/telegraf.conf 82 | service telegraf restart 83 | fi 84 | 85 | netstat -ntlpe 86 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-evebox.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs evebox 5 | # 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | ELASTIC=$1 14 | 15 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 16 | HOSTNAME=$(hostname -f) 17 | 18 | echo "installing evebox on ${IP} ${HOSTNAME} sets elasticsearch to ${ELASTIC} ..." 19 | 20 | apt-get -y install unzip 21 | cd /opt/ 22 | wget -4 -q https://bintray.com/artifact/download/jasonish/evebox/evebox-linux-amd64.zip 23 | unzip evebox-linux-amd64.zip 24 | /opt/evebox-linux-amd64/evebox --version 25 | echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 26 | echo "http.cors.allow-origin: \"/.*/\"" >> /etc/elasticsearch/elasticsearch.yml 27 | service elasticsearch restart 28 | nohup /opt/evebox-linux-amd64/evebox > /var/log/evebox.log 2>&1 & 29 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-grafana.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs grafana 5 | # 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | echo "installing grafana on ${IP} ${HOSTNAME} ..." 18 | 19 | INSTALL_DIR=/provision 20 | 21 | 22 | GRFN="2.6.0" 23 | mkdir -p ${INSTALL_DIR}/grafana 24 | cd ${INSTALL_DIR}/grafana 25 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 26 | wget -4 -q https://grafanarel.s3.amazonaws.com/builds/grafana_${GRFN}_amd64.deb 27 | fi 28 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 29 | echo "$(date) ${HOSTNAME} $0[$$]: {grafana: {status:WARNING, msg: missing grafana_${GRFN}_amd64.deb}" 30 | #exit -1 31 | else 32 | apt-get install -y adduser libfontconfig 33 | echo -e "Y"|dpkg -i grafana_${GRFN}_amd64.deb 34 | update-rc.d grafana-server defaults 95 10 35 | sed -i -e 's,domain = localhost,domain = '${IP}',g' /etc/grafana/grafana.ini 36 | service grafana-server start 37 | fi 38 | netstat -lntpe | grep grafana 39 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-influxdb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs influxdb 5 | # 2) sets influxdb to $1 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | BIND=$1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | echo "installing influxdb on ${IP} ${HOSTNAME} setting bind to ${BIND}..." 18 | 19 | INSTALL_DIR=/provision 20 | 21 | INFLX="0.10.0" 22 | 23 | mkdir -p ${INSTALL_DIR}/influxdb 24 | cd ${INSTALL_DIR}/influxdb 25 | if [ ! -f "influxdb_${INFLX}-1_amd64.deb" ]; then 26 | wget -q https://s3.amazonaws.com/influxdb/influxdb_${INFLX}-1_amd64.deb 27 | fi 28 | if [ ! -f "influxdb_${INFLX}-1_amd64.deb" ]; then 29 | echo "$(date) ${NAME} $0[$$]: {influxdb: {status:ERROR, msg: missing influxdb_${INFLX}-1_amd64.deb}" 30 | exit -1 31 | else 32 | echo -e "Y"|dpkg -i influxdb_${INFLX}-1_amd64.deb 2>&1 > /dev/null 33 | #prepare for telegraf 34 | curl -s -G http://localhost:8086/query --data-urlencode "q=DROP DATABASE telegraf" 35 | sleep 1 36 | curl -s -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE telegraf" 37 | curl -s -G http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY one_day_only ON telegraf DURATION 1d REPLICATION 1 DEFAULT" 38 | #sed -i -e 's,localhost,'${METRICS_SERVER}',g' /etc/influxdb/influxdb.conf 39 | #service influxdb restart 40 | service telegraf restart 41 | fi 42 | netstat -lntpe | grep influx 43 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-kibana.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # installs kibana 5 | # 6 | 7 | if [ "$(id -u)" != "0" ]; then 8 | echo "ERROR - This script must be run as root" 1>&2 9 | exit 1 10 | fi 11 | 12 | INSTALL_DIR=/provision 13 | KBN=4.3.1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | 18 | mkdir -p ${INSTALL_DIR}/kibana 19 | cd ${INSTALL_DIR}/kibana 20 | if [ ! -f "kibana-${KBN}-linux-x64.tar.gz" ]; then 21 | wget -4 -q https://download.elastic.co/kibana/kibana/kibana-${KBN}-linux-x64.tar.gz 22 | fi 23 | mkdir /opt 24 | cd /opt 25 | tar -xzf ${INSTALL_DIR}/kibana/kibana-${KBN}-linux-x64.tar.gz 26 | ln -sf /opt/kibana-4.3.1-linux-x64 /opt/kibana 27 | /opt/kibana/bin/kibana plugin -i kibana/timelion 28 | #chown -R kibana.kibana /opt/kibana/optimize/ 29 | # server.host: "0.0.0.0" 30 | sed -i -e 's,# server.host: "0.0.0.0",server.host: "'${IP}'",g' /opt/kibana/config/kibana.yml 31 | # elasticsearch.url: "http://10.242.11.29:9200" 32 | sed -i -e 's,# elasticsearch.url: "http://localhost:9200",elasticsearch.url: "http://'${IP}':9200",g' /opt/kibana/config/kibana.yml 33 | 34 | nohup /opt/kibana/bin/kibana > /var/log/kibana.log 2>&1 & 35 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-logstash.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs logstash 5 | # 2) sets elastic to $1 in conf.d/suricata.conf 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | ELASTIC=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 14 | 15 | 16 | # Markus, can you replace logststash with rsyslog, but still keep the "good" tempalte, so scirius can still get data 17 | # logstash 18 | echo 'deb http://packages.elasticsearch.org/logstash/2.2/debian stable main' > /etc/apt/sources.list.d/logstash.list 19 | apt-get update > /dev/null 2>&1 20 | apt-get -y --force-yes install logstash > /dev/null 2>&1 21 | 22 | #stealing amsterdam losgstash conf 23 | wget -4 -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/logstash/logstash.conf -O /etc/logstash/conf.d/suricata.conf 24 | # hosts => elasticsearch 25 | sed -i -e 's,hosts => elasticsearch,hosts => "'${ELASTIC}'"\n index => "logstash-%{+YYYY.MM.dd.HH}",g' /etc/logstash/conf.d/suricata.conf 26 | #fix this hack 27 | chmod 777 /var/log/suricata/eve.json 28 | service logstash start 29 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-metrix-server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo "$0 got params: $1 $2 $3 $4" 3 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-moloch.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # will build moloch 5 | # 6 | 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | echo "going to build moloch, take your time, or coffe or ..." 14 | 15 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 16 | INSTALL_DIR=/provision 17 | 18 | mkdir -p ${INSTALL_DIR}/moloch 19 | cd ${INSTALL_DIR}/moloch 20 | if [ ! -f "master.tar.gz" ]; then 21 | wget -4 -q https://github.com/aol/moloch/archive/master.tar.gz 22 | tar -xzf master.tar.gz 23 | fi 24 | cd moloch-master 25 | 26 | #echo -e "\n\n\n\n\n\n\n\n"|./easybutton-singlehost.sh 27 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-salt-master.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) adds ppa:saltstack/salt 5 | # 2) installs salt-master 6 | # 3) accepts all minions contacting in 10 secs 7 | # 8 | 9 | 10 | if [ "$(id -u)" != "0" ]; then 11 | echo "ERROR - This script must be run as root" 1>&2 12 | exit 1 13 | fi 14 | 15 | echo "installing salt-master on ${hostname} ..." 16 | 17 | echo "LC_ALL=en_US.UTF-8" >> /etc/environment 18 | echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4 19 | 20 | add-apt-repository -y ppa:saltstack/salt 2>&1 > /dev/null 21 | apt-get update 2>&1 > /dev/null 22 | apt-get -y install salt-master 2>&1 > /dev/null 23 | echo "waiting for minions.." 24 | sleep 11 25 | salt-key -L 26 | salt-key -A -y 27 | sleep 1 28 | salt '*' test.ping 29 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-salt-minion.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) adds ppa:saltstack/salt 5 | # 2) installs salt-minion 6 | # 3) sets master to $1 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | MASTER=$1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | echo "installing salt-minion on ${IP} ${HOSTNAME} setting master to ${MASTER}..." 18 | 19 | echo "LC_ALL=en_US.UTF-8" >> /etc/environment 20 | #echo "${MASTER} saltmaster" >> /etc/hosts 21 | #hostname `cat /etc/hostname` 22 | 23 | echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4 24 | 25 | add-apt-repository -y ppa:saltstack/salt > /dev/null 2>&1 26 | apt-get update > /dev/null 2>&1 27 | apt-get -y install salt-minion > /dev/null 2>&1 28 | echo "master: ${MASTER}" >> /etc/salt/minion 29 | service salt-minion restart 30 | sleep 1 31 | tail /var/log/salt/minion 32 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-suri.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 0) sets gro,gso,lro,tso off 5 | # 1) addp oisf ppa 6 | # 2) installs suricata 7 | # 3) sets suricata conf as amsterdam 8 | # 4) 9 | 10 | 11 | if [ "$(id -u)" != "0" ]; then 12 | echo "ERROR - This script must be run as root" 1>&2 13 | exit 1 14 | fi 15 | 16 | ETH=$1 17 | SCIRIUS=$2 18 | 19 | ethtool -K $ETH tx off sg off gro off gso off lro off tso off 20 | 21 | #suricata 22 | add-apt-repository -y ppa:oisf/suricata-stable 2>&1 > /dev/null 23 | apt-get update 2>&1 > /dev/null 24 | apt-get -y install suricata 2>&1 > /dev/null 25 | service suricata stop 26 | #stealing amsterdam suricata conf 27 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/suricata.yaml -O /etc/suricata/suricata.yaml 28 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/threshold.config -O /etc/suricata/threshold.config 29 | 30 | # - interface: eth0 31 | sed -i -e 's,- interface: eth0,- interface: eth1,g' /etc/suricata/suricata.yaml 32 | #fake scirius rules 33 | #todo: get it from master 34 | touch /etc/suricata/rules/scirius.rules 35 | service suricata start 36 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-syslog-server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo "$0 got params: $1 $2 $3 $4" 3 | -------------------------------------------------------------------------------- /common/vagrant/bb/scripts/install-telegraf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs telegraf 5 | # 2) sets influxdb to $1 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | service telegraf stop 14 | 15 | MASTER=$1 16 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 17 | HOSTNAME=$(hostname -f) 18 | 19 | echo "installing telegraf on ${IP} ${HOSTNAME} setting influxdb to ${MASTER}..." 20 | 21 | 22 | TLGF=0.10.4.1 23 | INSTALL_DIR=/provision 24 | 25 | mkdir -p ${INSTALL_DIR}/telegraf 26 | cd ${INSTALL_DIR}/telegraf 27 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 28 | wget -4 -q wget http://get.influxdb.org/telegraf/telegraf_${TLGF}-1_amd64.deb 29 | fi 30 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 31 | echo "$(date) ${NAME} $0[$$]: {telegaf: {status:ERROR, msg: missing telegraf_${TLGF}_amd64.deb}" 32 | exit -1 33 | else 34 | echo -e "Y"|dpkg -i telegraf_${TLGF}-1_amd64.deb > /dev/null 35 | # urls = ["http://localhost:8086"] # required 36 | sed -i -e 's,http://localhost,http://'${MASTER}',g' /etc/telegraf/telegraf.conf 37 | # interval = "10s" 38 | sed -i -e 's,interval = "10s",interval = "1s",g' /etc/telegraf/telegraf.conf 39 | # flush_interval = "10s" 40 | sed -i -e 's,flush_interval = "1s",flush_interval = "60s",g' /etc/telegraf/telegraf.conf 41 | echo "[[inputs.net]]" >> /etc/telegraf/telegraf.conf 42 | echo "[[inputs.netstat]]" >> /etc/telegraf/telegraf.conf 43 | 44 | service telegraf restart 45 | fi 46 | -------------------------------------------------------------------------------- /moloch/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Day 0 3 | 4 | * 13:00 - 13:45 [Intro]() 5 | * 14:00 - 14:45 [easy button single host install]() 6 | * 15:00 - 15:45 [basic config]() 7 | * 16:00 - 16:45 [viewer basics]() 8 | 9 | ## Day 1 :: Single Box 10 | 11 | * 09:00 - 09:45 12 | * 10:00 - 10:45 13 | * 11:00 - 11:45 14 | 15 | 16 | * 13:00 - 13:45 17 | * 14:00 - 14:45 18 | * 15:00 - 15:45 19 | * 16:00 - 16:45 20 | 21 | 22 | * [build from source](/moloch/day_1/BuildFromSource.md) 23 | * [settings](/moloch/day_1/Settings.md) 24 | * [right-click]() 25 | * [Tagger]() 26 | * [Wise]() 27 | 28 | 29 | ## Day 2 :: Cluster 30 | * 09:00 - 09:45 31 | * 10:00 - 10:45 32 | * 11:00 - 11:45 33 | 34 | 35 | * 13:00 - 13:45 36 | * 14:00 - 14:45 37 | * 15:00 - 15:45 38 | * 16:00 - 16:45 39 | 40 | 41 | 42 | * [Multiple Hosts configuration](/moloch/day_2/MultipleHostConfig.md) 43 | 44 | 45 | 46 | 47 | ## Day 3 :: Advanced usage 48 | 49 | * 09:00 - 09:45 50 | * 10:00 - 10:45 51 | * 11:00 - 11:45 52 | 53 | 54 | * 13:00 - 13:45 55 | * 14:00 - 14:45 56 | * 15:00 - 15:45 57 | * 16:00 - 16:45 58 | 59 | 60 | * [Connecting clusters](/moloch/day_3/Cluster2Cluster.md) 61 | * [Plugins]() 62 | * [Multi-Viewer]() 63 | * [API]() 64 | 65 | 66 | ## Day +1 67 | 68 | * 09:00 - 09:45 69 | * 10:00 - 10:45 70 | * 11:00 - 11:45 71 | -------------------------------------------------------------------------------- /moloch/day_1/BuildFromSource.md: -------------------------------------------------------------------------------- 1 | # Build Moloch from source 2 | 3 | see https://github.com/aol/moloch/blob/master/README.rst#building-and-installing 4 | -------------------------------------------------------------------------------- /moloch/day_1/Settings.md: -------------------------------------------------------------------------------- 1 | # Settings 2 | 3 | see https://github.com/aol/moloch/wiki/Settings 4 | -------------------------------------------------------------------------------- /moloch/day_1/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.113" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :single do |box| 6 | box.vm.box = vm_box 7 | #box.vm.synced_folder ".", "/vagrant", disabled: false 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'single' 10 | box.vm.provision :shell, :path => "./my-custom-install.sh", args: "first_arg second_arg" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /moloch/day_1/moloch-capture.conf: -------------------------------------------------------------------------------- 1 | description "Upstart configuration for Capture" 2 | start on started elasticsearch 3 | stop on stopping elasticsearch 4 | respawn limit 10 5 5 | env CAPTURE_HOME=/data/moloch/ 6 | 7 | pre-start script 8 | chdir $CAPTURE_HOME 9 | exec /bin/rm -f $CAPTURE_HOME/logs/capture.log.old 10 | exec /bin/mv $CAPTURE_HOME/logs/capture.log $CAPTURE_HOME/logs/capture.log.old 11 | end script 12 | 13 | script 14 | chdir $CAPTURE_HOME 15 | exec $CAPTURE_HOME/bin/moloch-capture -c $CAPTURE_HOME/etc/config.ini > $CAPTURE_HOME/logs/capture.log 2>&1 16 | end script 17 | -------------------------------------------------------------------------------- /moloch/day_1/moloch-viewer.conf: -------------------------------------------------------------------------------- 1 | description "Upstart configuration for Viewer" 2 | start on started elasticsearch 3 | stop on stopping elasticsearch 4 | respawn limit 10 5 5 | env VIEWER_HOME=/data/moloch/ 6 | export NODE_ENV=production 7 | 8 | pre-start script 9 | chdir $VIEWER_HOME/viewer 10 | exec /bin/rm -f $VIEWER_HOME/logs/viewer.log.old 11 | exec /bin/mv $VIEWER_HOME/logs/viewer.log $VIEWER_HOME/logs/viewer.log.old 12 | end script 13 | 14 | script 15 | chdir $VIEWER_HOME/viewer 16 | exec $VIEWER_HOME/bin/node-viewer viewer.js -c $VIEWER_HOME/etc/config.ini > $VIEWER_HOME/logs/viewer.log 2>&1 17 | end script 18 | -------------------------------------------------------------------------------- /moloch/day_1/my-custom-install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # This script will install .. 4 | 5 | if [ "$(id -u)" != "0" ]; then 6 | echo "ERROR - This script must be run as root" 1>&2 7 | exit 1 8 | fi 9 | 10 | echo "got arg1: $1 arg2: $2" 11 | 12 | 13 | mkdir -p /vagrant/provision 14 | cd /vagrant/provision 15 | 16 | if [ ! -f "master.tar.gz" ]; then 17 | echo "downloading moloch source..." 18 | time wget -4 -q https://github.com/aol/moloch/archive/master.tar.gz 19 | fi 20 | 21 | ES=2.3.2 22 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 23 | echo "downloaging elastic $ES" 24 | wget -4 -q https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES}.deb 25 | fi 26 | 27 | echo "using: " 28 | for i in $(ls); 29 | do 30 | ls -tlah $i 31 | done 32 | 33 | #mkdir -p /build 34 | #cd /build/ 35 | #tar -xzf /vagrant/provision/master.tar.gz 36 | #cd moloch-master/ 37 | #ls -tlah 38 | 39 | # https://github.com/aol/moloch/blob/16e21a275eef042f635686213247e8cde4729754/capture/reader-libpcap.c#L58 40 | # ethtool -K INTERFACE tx off sg off gro off gso off lro off tso off 41 | -------------------------------------------------------------------------------- /moloch/day_1/wtfit.md: -------------------------------------------------------------------------------- 1 | # you got a pcap 2 | 3 | 4 | ```bash 5 | for pcap in `find /srv/ -type f -name '*.pcap*'`; do echo "`tcpdump -r $pcap -c1 -tttt 2> /dev/null | perl -ne 'if (m/^(.+)\.\d+ IP/){print "$1"}'` : $pcap" ; done >> /tmp/list_combined.txt 6 | ``` 7 | 8 | ```bash 9 | #!/bin/bash 10 | echo "first;last;pcap" 11 | for pcap in $(find /srv/ -type f -name '*.pcap*'|head); 12 | do 13 | first=$(tcpdump -r $pcap -c1 -tt 2> /dev/null| cut -f1 -d".") 14 | last=$(tcpdump -r $pcap -tt 2> /dev/null|tail -1| cut -f1 -d".") 15 | echo "$first;$last;$pcap" 16 | done 17 | ``` 18 | 19 | > tcpdump -r $pcap -tt 2> /dev/null|tail -1| cut -f1 -d"." <- is slow 20 | 21 | ```bash 22 | #!/bin/bash 23 | 24 | filename=$1 25 | 26 | # for pcap in `find /srv/ -type f -name '*.pcap*'`; do echo "`tcpdump -r $pcap -c1 -tttt 2> /dev/null | perl -ne 'if (m/^(.+)\.\d+ IP/){print "$1"}'` : $pcap" ; done >> filename 27 | 28 | 29 | echo "# start $(date)" > /tmp/kala 30 | line=$(wc -l $filename | cut -f1 -d" ") 31 | while [[ "$line" != "0" ]] ; do 32 | tmp=$(head -1 $filename) 33 | time=$(echo $tmp | cut -f1,2 -d" "|sed 's/ //g'|sed 's/-//g'|sed 's/://g') 34 | file=$(echo $tmp | cut -f4 -d" ") 35 | log=$(echo $file| sed 's/\//_/g') 36 | sed -i '1d' $filename 37 | if [[ "$file" != "" ]]; 38 | then 39 | i=$(pgrep moloch-capture | wc -l) 40 | echo "already running $i" 41 | while [[ "$i" > "2" ]] 42 | do 43 | sleep 5 44 | i=$(pgrep moloch-capture | wc -l) 45 | echo "$i" 46 | done 47 | echo "$time $file" >> /tmp/kala 48 | echo $file 49 | /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -r $file > /tmp/log.$log 2>&1 & 50 | sleep 1 51 | fi 52 | line=$(wc -l $filename | cut -f1 -d" ") 53 | echo $line 54 | done 55 | echo "# end $(date)" >> /tmp/kala 56 | ``` 57 | -------------------------------------------------------------------------------- /moloch/day_2/MoloClusterNode.md: -------------------------------------------------------------------------------- 1 | # ElasticSearch cluster nodes 2 | -------------------------------------------------------------------------------- /moloch/day_2/MultipleHostConfig.md: -------------------------------------------------------------------------------- 1 | # Multiple Host Configuration 2 | 3 | see 4 | * https://github.com/aol/moloch/wiki/Multiple-Host-HOWTO 5 | * https://github.com/aol/moloch/wiki/Architecture#multiple-hosts-monitoring-multiple-network-segments 6 | -------------------------------------------------------------------------------- /moloch/day_2/Readme.md: -------------------------------------------------------------------------------- 1 | # multi hosts 2 | 3 | ``` 4 | tar cvfz moloch_and_nodejs.tgz --exclude /data/moloch/elasticsearch* --exclude '/data/moloch/data' 5 | ``` 6 | -------------------------------------------------------------------------------- /moloch/day_2/boxes/scripts: -------------------------------------------------------------------------------- 1 | ../../../suricata/day_2/boxes/scripts -------------------------------------------------------------------------------- /moloch/day_2/moloch-capture.conf: -------------------------------------------------------------------------------- 1 | description "Upstart configuration for Capture" 2 | start on started elasticsearch 3 | stop on stopping elasticsearch 4 | respawn limit 10 5 5 | env CAPTURE_HOME=/opt/moloch/ 6 | 7 | pre-start script 8 | chdir $CAPTURE_HOME 9 | exec /bin/rm -f $CAPTURE_HOME/logs/capture.log.old 10 | exec /bin/mv $CAPTURE_HOME/logs/capture.log $CAPTURE_HOME/logs/capture.log.old 11 | end script 12 | 13 | script 14 | chdir $CAPTURE_HOME 15 | exec $CAPTURE_HOME/bin/moloch-capture -c $CAPTURE_HOME/etc/config.ini > $CAPTURE_HOME/logs/capture.log 2>&1 16 | end script 17 | -------------------------------------------------------------------------------- /moloch/day_2/moloch-viewer.conf: -------------------------------------------------------------------------------- 1 | description "Upstart configuration for Viewer" 2 | start on started elasticsearch 3 | stop on stopping elasticsearch 4 | respawn limit 10 5 5 | env VIEWER_HOME=/opt/moloch/ 6 | export NODE_ENV=production 7 | 8 | pre-start script 9 | chdir $VIEWER_HOME/viewer 10 | exec /bin/rm -f $VIEWER_HOME/logs/viewer.log.old 11 | exec /bin/mv $VIEWER_HOME/logs/viewer.log $VIEWER_HOME/logs/viewer.log.old 12 | end script 13 | 14 | script 15 | chdir $VIEWER_HOME/viewer 16 | exec nodejs viewer.js -c $VIEWER_HOME/etc/config.ini > $VIEWER_HOME/logs/viewer.log 2>&1 17 | end script 18 | -------------------------------------------------------------------------------- /moloch/day_3/Cluster2Cluster.md: -------------------------------------------------------------------------------- 1 | # Cluster 2 Cluster 2 | 3 | see https://github.com/aol/moloch/wiki/Settings#moloch-clusters 4 | 5 | ``` 6 | [moloch-clusters] 7 | cluster1=url:https://moloch.example.com:8005;passwordSecret:password;name:Cluster 8 | cluster2=url:https://cluster2.example.com:8005;passwordSecret:foo;name:Test Cluster 9 | ``` 10 | -------------------------------------------------------------------------------- /moloch/day_3/lua.md: -------------------------------------------------------------------------------- 1 | # lua 2 | 3 | see 4 | * https://github.com/aol/moloch/commit/7f03eeffced17ee56f902659df77ac40d6675d6b 5 | * https://github.com/aol/moloch/blob/master/capture/plugins/lua/README.md 6 | 7 | 8 | 9 | ```lua 10 | -- To use add 11 | -- plugins=lua.so 12 | -- lauFiles=/path/moloch.lua 13 | -- This script is run per packetThread, so any globals will be shared 14 | -- for all packets processed by this thread. 15 | 16 | -- session - userdata session object 17 | -- data - the binary data 18 | -- direction - socket direction 19 | -- return - -1 means stop parsing 20 | function foo_parser(session, data, direction) 21 | print("\nparser:\n", "\nsession:", session, "\nwhich:", which, "\n") 22 | moloch_field_add_string(session, "protocols", "newprotocol"..cnt) 23 | return -1; 24 | end 25 | 26 | -- session - userdata session object 27 | -- data - the binary data 28 | -- direction - socket direction 29 | function foo_classify(session, data, direction) 30 | print ("classify", session, which, data) 31 | moloch_session_add_tag(session, "foofoobarbar"..cnt) 32 | moloch_field_add_string(session, "user", "luauser"..cnt) 33 | moloch_field_add_int(session, portSocksField, 12345 + cnt) 34 | moloch_parsers_register(session, foo_parser) 35 | cnt = cnt+1 36 | end 37 | 38 | -- name, offset to check for match in data, binary match data, "function name" 39 | moloch_parsers_classifier_register_tcp("all", 0, "", "foo_classify") 40 | moloch_parsers_classifier_register_udp("all", 0, "", "foo_classify") 41 | 42 | -- convert expression to fieldId 43 | portSocksField = moloch_field_by_exp("port.socks"); 44 | 45 | -- Just to make things added unique and make sure things are working. :) 46 | cnt = 0; -- remember global per thread 47 | ``` 48 | -------------------------------------------------------------------------------- /moloch/day_intro/README.md: -------------------------------------------------------------------------------- 1 | # Moloch 2 | 3 | ![owl](http://molo.ch/moloch_2x2.png) 4 | 5 | ## http://molo.ch/ 6 | 7 | > Moloch is an open source, large scale packet capturing (PCAP), indexing and database system. A simple web interface is provided for PCAP browsing, searching, and exporting. APIs are exposed that allow PCAP data and JSON-formatted session data to be downloaded directly. Simple security is implemented by using HTTPS and HTTP digest password support or by using apache in front. Moloch is not meant to replace IDS engines but instead work along side them to store and index all the network traffic in standard PCAP format, providing fast access. Moloch is built to be deployed across many systems and can scale to handle multiple gigabits/sec of traffic. 8 | 9 | ## https://github.com/aol/moloch 10 | 11 | > The Moloch system is comprised of 3 components 12 | 13 | > * **capture** - A single-threaded C application that runs per network interface. It is possible to run multiple capture processes per machine if there are multiple interfaces to monitor. 14 | > * **viewer** - A node.js application that runs per capture machine and handles the web interface and transfer of PCAP files. 15 | > * **elasticsearch** - The search database technology powering Moloch. 16 | 17 | 18 | > Impatient? 19 | 20 | > git clone https://github.com/aol/moloch.git 21 | 22 | > cd moloch 23 | 24 | > ./easybutton-singlehost.sh 25 | -------------------------------------------------------------------------------- /moloch/day_intro/dummy/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.111" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Dummy do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Dummy' 10 | box.vm.provision :shell, :path => "./dummy-install-something.sh", args: "eth1" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /moloch/day_intro/dummy/dummy-install-something.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "installing nothing" 4 | -------------------------------------------------------------------------------- /moloch/day_intro/easybutton/2016-02-28-traffic-analysis-exercise-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/moloch/day_intro/easybutton/2016-02-28-traffic-analysis-exercise-2.png -------------------------------------------------------------------------------- /moloch/day_intro/easybutton/2016-02-28-traffic-analysis-exercise.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/moloch/day_intro/easybutton/2016-02-28-traffic-analysis-exercise.png -------------------------------------------------------------------------------- /moloch/day_intro/easybutton/MVGK-autoportrait.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/moloch/day_intro/easybutton/MVGK-autoportrait.png -------------------------------------------------------------------------------- /moloch/day_intro/easybutton/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.112" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Moloch do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Moloch' 10 | box.vm.provision :shell, :path => "../MVGK-easybutton-singlehost.sh", args: "eth1" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /moloch/day_intro/easybutton/vsroom-moloch-sessions-demo.mov: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/moloch/day_intro/easybutton/vsroom-moloch-sessions-demo.mov -------------------------------------------------------------------------------- /moloch/day_intro/empty/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.110" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Empty do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Empty' 10 | box.vm.provision "shell", inline: "ifconfig; ping -c 1 www.ee; netstat -ntple" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /moloch/day_last/README.md: -------------------------------------------------------------------------------- 1 | 2 | * how did we get here 3 | * how does it all hang together 4 | * how much still remains to be understood 5 | -------------------------------------------------------------------------------- /saynomore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/saynomore.png -------------------------------------------------------------------------------- /suricata/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Day 0 3 | 4 | * 13:00 - 13:45 [Intro](/suricata/day_intro/README.md), vagrant, docker, [Amsterdam](/suricata/day_intro/Amsterdam.md) 5 | * 14:00 - 14:45 [apt-get install](/suricata/day_intro/AptGetInstall.md), [PPA](/suricata/day_intro/AptGetInstallPPA.md) 6 | * 15:00 - 15:45 [basic config](/suricata/day_intro/BasicConf.md),[basic logging](/suricata/day_intro/BasicLogging.md), [view events](/suricata/day_intro/EveView.md) 7 | * 16:00 - 16:45 [writing first rule](/suricata/day_intro/SimpleRule.md) 8 | 9 | 10 | ## Day 1 :: Single Box 11 | 12 | * 09:00 - 09:45 [intro](/suricata/day_1/README.md),[build from source](/suricata/day_1/BuildFromSource.md) 13 | * 10:00 - 10:45 [Config](/suricata/day_1/SingleHostConfig.md) 14 | * 11:00 - 11:45 Rules [management](/suricata/day_1/RuleManagement.md) 15 | 16 | 17 | 18 | * 13:00 - 13:45 [thresholding](/suricata/day_1/RuleThresholding.md) 19 | * 14:00 - 14:45 [reputation](/suricata/day_1/Reputation.md) 20 | * 15:00 - 16:45 [CEE](/suricata/day_1/CEE.md), [rsyslog](/suricata/day_1/rsyslog.md) 21 | 22 | 23 | ## Day 2 :: Cluster 24 | 25 | * 09:00 - 11:45 [Tuning Considerations](/suricata/day_1/SingleBoxtuning.md) 26 | 27 | 28 | * 13:00 - 13:45 [intro](/suricata/day_2/README.md),[Metrix](/suricata/day_2/SetUpMetrics.md),[Elastic](/suricata/day_2/SetUpElastic.md), [Minions](/suricata/day_2/SetUpMinions.md), [Master](/suricata/day_2/SetUpMaster.md) 29 | * 14:00 - 16:45 [work with boxes](/suricata/day_2/notes.md) 30 | 31 | 32 | 33 | ## Day 3 :: Advanced usage 34 | 35 | * 09:00 - 11:45 [diving into nigthly alerts](/common/Kibana4.md) 36 | 37 | 38 | 39 | 40 | * 13:00 - 13:45 [Multi Tenancy](/suricata/day_3/MultiTenancy.md) 41 | * 14:00 - 14:45 [Loading many pcaps](/suricata/day_3/LoadPcaps.md) 42 | * 15:00 - 15:45 43 | * 16:00 - 16:45 44 | 45 | 46 | * [Lua scripted rules](/suricata/day_3/LuaRules.md) 47 | * [Lua Output](/suricata/day_3/LuaOutput.md) 48 | * Extraction [file](/suricata/day_1/FileExtraction.md) 49 | * [TLS monitor](/suricata/day_1/TLSMonitor.md) , [DNS probe](/suricata/day_1/DNSProbe.md) , [flow probe](/suricata/day_1/FlowProbe.md) 50 | * [evil bit](/suricata/day_3/EvilBit.md) 51 | * [rule profiling](/suricata/day_1/RuleProfiling.md), 52 | * [pf_packet, af_ring, ..](/suricata/day_3/NICAccess.md) 53 | 54 | 55 | ## Day +1 56 | 57 | * 09:00 - 09:45 [Internal counters](/suricata/day_3/DumpCounters.md) 58 | * 10:00 - 10:45 [Converting logs to pcap](/suricata/day_3/Eve2Pcap.md) 59 | * 11:00 - 11:45 [pgp key signing, contact excange, thanks, etc ...]() 60 | * 12:00 The End 61 | -------------------------------------------------------------------------------- /suricata/day_1/BuildFromSource.md: -------------------------------------------------------------------------------- 1 | # Building Suricata from source 2 | 3 | see 4 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Installation_from_GIT 5 | * http://pevma.blogspot.se/2013/12/suricata-cocktails-handy-one-liners.html 6 | * https://github.com/obsrvbl/suricata-service 7 | * https://github.com/jasonish/suricata-rpms 8 | 9 | ### Dependencies 10 | 11 | ``` 12 | sudo apt-get -y install \ 13 | libpcre3 \ 14 | libpcre3-dbg \ 15 | libpcre3-dev \ 16 | build-essential \ 17 | autoconf \ 18 | automake \ 19 | libtool \ 20 | libpcap-dev \ 21 | libnet1-dev \ 22 | libyaml-0-2 \ 23 | libyaml-dev \ 24 | pkg-config \ 25 | zlib1g \ 26 | zlib1g-dev \ 27 | libcap-ng-dev \ 28 | libcap-ng0 \ 29 | make \ 30 | libmagic-dev 31 | ``` 32 | 33 | Or the lazy way to get most of them: 34 | 35 | ``` 36 | sudo apt-get build-dep suricata 37 | ``` 38 | 39 | ### Searching for packages 40 | 41 | ``` 42 | sudo apt-cache search pcre 43 | ``` 44 | 45 | ``` 46 | sudo apt-cache policy libpcre3-dev 47 | ``` 48 | 49 | ### get the source 50 | ``` 51 | git clone git://phalanx.openinfosecfoundation.org/oisf.git 52 | ``` 53 | 54 | ``` 55 | cd oisf 56 | git clone https://github.com/OISF/libhtp.git -b 0.5.x 57 | ``` 58 | ### configure, make install 59 | 60 | ``` 61 | ./autogen.sh 62 | ``` 63 | 64 | ``` 65 | ./configure 66 | ``` 67 | 68 | * [configure --help](/suricata/day_1/ConfigureHelp.md) 69 | * [configure --enable-profiling --enable-luajit](/suricata/day_1/ConfigureProfilingLuaJit.md) 70 | 71 | ``` 72 | make 73 | ``` 74 | 75 | ``` 76 | sudo make install 77 | ``` 78 | 79 | * [make install-full](/suricata/day_1/MakeInstallFull.md) 80 | 81 | 82 | ``` 83 | sudo ldconfig 84 | ``` 85 | 86 | ``` 87 | root@secx:/home/student/oisf# suricata -V 88 | This is Suricata version 3.0dev (rev 44a444b) 89 | ``` 90 | 91 | ### Cleaning up 92 | 93 | ``` 94 | sudo make uninstall 95 | ``` 96 | 97 | ``` 98 | make clean 99 | ``` 100 | 101 | ``` 102 | make distclean 103 | ``` 104 | 105 | --- 106 | 107 | [next : Single host configuration](/suricata/day_1/SingleHostConfig.md) 108 | 109 | 110 | -------------------------------------------------------------------------------- /suricata/day_1/CEE.md: -------------------------------------------------------------------------------- 1 | # Common Event Expression 2 | 3 | * https://cee.mitre.org/ 4 | * http://www.rsyslog.com/tag/cee-enhanced/ 5 | 6 | ## Log format 7 | 8 | ``` 9 | Feb 25 11:23:42 suricata suricata[26526]: @cee: {"timestamp":"2015-12-07T19:30:54.863188+0000","flow_id":139635731853600,"pcap_cnt":142,"event_type":"alert","src_ip":"192.168.11.11","src_port":59523,"dest_ip":"192.168.12.12","dest_port":443,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2013926,"rev":8,"signature":"ET POLICY HTTP traffic on port 443 (POST)","category":"Potentially Bad Traffic","severity":2}} 10 | ``` 11 | 12 | ## Suricata configuration 13 | 14 | ``` 15 | grep cee -B2 -A3 /etc/suricata/suricata.yaml 16 | ``` 17 | 18 | [next: rsyslog](/suricata/day_1/rsyslog.md) -------------------------------------------------------------------------------- /suricata/day_1/Config.md: -------------------------------------------------------------------------------- 1 | # Configuration 2 | 3 | see 4 | * http://jasonish-suricata.readthedocs.org/en/latest/configuration/index.html 5 | * http://pevma.blogspot.com.ee/2015/10/suricata-with-afpacket-memory-of-it-all.html 6 | -------------------------------------------------------------------------------- /suricata/day_1/ConfigureProfilingLuaJit.md: -------------------------------------------------------------------------------- 1 | # ./configure --enable-profiling --enable-luajit 2 | 3 | * apt-get install libluajit-5.1-dev 4 | * apt-get install libjansson-dev 5 | 6 | ``` 7 | root@secx:/home/student/oisf# ./configure --enable-profiling --enable-luajit 8 | checking whether make supports nested variables... yes 9 | 10 | 11 | ... 12 | 13 | checking that generated files are newer than configure... done 14 | configure: creating ./config.status 15 | config.status: creating htp/htp_version.h 16 | config.status: creating Makefile 17 | config.status: creating htp.pc 18 | config.status: creating htp/Makefile 19 | config.status: creating test/Makefile 20 | config.status: creating docs/Makefile 21 | config.status: creating htp_config_auto_gen.h 22 | config.status: htp_config_auto_gen.h is unchanged 23 | config.status: executing depfiles commands 24 | config.status: executing libtool commands 25 | 26 | Suricata Configuration: 27 | AF_PACKET support: yes 28 | PF_RING support: no 29 | NFQueue support: no 30 | NFLOG support: no 31 | IPFW support: no 32 | Netmap support: no 33 | DAG enabled: no 34 | Napatech enabled: no 35 | 36 | Unix socket enabled: yes 37 | Detection enabled: yes 38 | 39 | libnss support: no 40 | libnspr support: no 41 | libjansson support: yes 42 | hiredis support: no 43 | Prelude support: no 44 | PCRE jit: yes 45 | LUA support: yes, through luajit 46 | libluajit: yes 47 | libgeoip: no 48 | Non-bundled htp: no 49 | Old barnyard2 support: no 50 | CUDA enabled: no 51 | 52 | Suricatasc install: yes 53 | 54 | Unit tests enabled: no 55 | Debug output enabled: no 56 | Debug validation enabled: no 57 | Profiling enabled: yes 58 | Profiling locks enabled: no 59 | Coccinelle / spatch: no 60 | 61 | Generic build parameters: 62 | Installation prefix: /usr/local 63 | Configuration directory: /usr/local/etc/suricata/ 64 | Log directory: /usr/local/var/log/suricata/ 65 | 66 | --prefix /usr/local 67 | --sysconfdir /usr/local/etc 68 | --localstatedir /usr/local/var 69 | 70 | Host: x86_64-unknown-linux-gnu 71 | Compiler: gcc (exec name) / gcc (real) 72 | GCC Protect enabled: no 73 | GCC march native enabled: yes 74 | GCC Profile enabled: no 75 | Position Independent Executable enabled: no 76 | CFLAGS -g -O2 -march=native 77 | PCAP_CFLAGS -I/usr/include 78 | SECCFLAGS 79 | 80 | To build and install run 'make' and 'make install'. 81 | 82 | You can run 'make install-conf' if you want to install initial configuration 83 | files to /usr/local/etc/suricata/. Running 'make install-full' will install configuration 84 | and rules and provide you a ready-to-run suricata. 85 | 86 | To install Suricata into /usr/bin/suricata, have the config in 87 | /etc/suricata and use /var/log/suricata as log dir, use: 88 | ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ 89 | ``` 90 | -------------------------------------------------------------------------------- /suricata/day_1/DNSProbe.md: -------------------------------------------------------------------------------- 1 | # DNS Probe 2 | 3 | Suricata does stateful DNS tracking for UDP and TCP 4 | 5 | ```json 6 | { 7 | "_index": "logstash-2016.02.03", 8 | "_type": "SELKS", 9 | "_id": "AVKo0dtdE4DnavQE_p1P", 10 | "_score": null, 11 | "_source": { 12 | "timestamp": "2016-02-03T22:27:51.201423+0200", 13 | "flow_id": 140371952721904, 14 | "in_iface": "eth2", 15 | "event_type": "dns", 16 | "src_ip": "10.22.3.20", 17 | "src_port": 50398, 18 | "dest_ip": "10.42.2.2", 19 | "dest_port": 53, 20 | "proto": "UDP", 21 | "dns": { 22 | "type": "query", 23 | "id": 24746, 24 | "rrname": "traffic2.ex", 25 | "rrtype": "A", 26 | "tx_id": 0 27 | }, 28 | "@version": "1", 29 | "@timestamp": "2016-02-03T20:27:51.201Z", 30 | "host": "rtws-yellow-1-medium-2", 31 | "path": "/var/log/suricata/eve.json", 32 | "type": "SELKS" 33 | } 34 | } 35 | ``` 36 | 37 | #### elastic template 38 | 39 | ```json 40 | "dns": { 41 | "properties": { 42 | "id": { 43 | "type": "long" 44 | }, 45 | "rrname": { 46 | "norms": { 47 | "enabled": false 48 | }, 49 | "fielddata": { 50 | "format": "disabled" 51 | }, 52 | "type": "string", 53 | "fields": { 54 | "raw": { 55 | "index": "not_analyzed", 56 | "ignore_above": 256, 57 | "type": "string" 58 | } 59 | } 60 | }, 61 | "rrtype": { 62 | "norms": { 63 | "enabled": false 64 | }, 65 | "fielddata": { 66 | "format": "disabled" 67 | }, 68 | "type": "string", 69 | "fields": { 70 | "raw": { 71 | "index": "not_analyzed", 72 | "ignore_above": 256, 73 | "type": "string" 74 | } 75 | } 76 | }, 77 | "tx_id": { 78 | "type": "long" 79 | }, 80 | "rcode": { 81 | "norms": { 82 | "enabled": false 83 | }, 84 | "fielddata": { 85 | "format": "disabled" 86 | }, 87 | "type": "string", 88 | "fields": { 89 | "raw": { 90 | "index": "not_analyzed", 91 | "ignore_above": 256, 92 | "type": "string" 93 | } 94 | } 95 | }, 96 | "type": { 97 | "norms": { 98 | "enabled": false 99 | }, 100 | "fielddata": { 101 | "format": "disabled" 102 | }, 103 | "type": "string", 104 | "fields": { 105 | "raw": { 106 | "index": "not_analyzed", 107 | "ignore_above": 256, 108 | "type": "string" 109 | } 110 | } 111 | }, 112 | "ttl": { 113 | "type": "long" 114 | }, 115 | "rdata": { 116 | "norms": { 117 | "enabled": false 118 | }, 119 | "fielddata": { 120 | "format": "disabled" 121 | }, 122 | "type": "string", 123 | "fields": { 124 | "raw": { 125 | "index": "not_analyzed", 126 | "ignore_above": 256, 127 | "type": "string" 128 | } 129 | } 130 | } 131 | } 132 | } 133 | ``` 134 | -------------------------------------------------------------------------------- /suricata/day_1/FileExtraction.md: -------------------------------------------------------------------------------- 1 | # File Extraction 2 | 3 | > File extraction depends on a lot of things to be fully done.. 4 | 5 | 6 | ##see 7 | 8 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/MD5 9 | 10 | #### fresh blog posts 11 | * https://stoq-framework.blogspot.com.ee/2016/02/using-stoq-with-suricatas-file.html 12 | 13 | ### slides 14 | * Generic slides with filestore keywords after slide 12: https://home.regit.org/wp-content/uploads/2015/02/suricata-netfilter-prc.pdf 15 | 16 | ### config 17 | ``` yaml 18 | - file-store: 19 | enabled: yes # set to yes to enable 20 | log-dir: files # directory to store the files 21 | force-magic: yes # force logging magic on all stored files 22 | force-md5: yes # force logging of md5 checksums 23 | #waldo: file.waldo # waldo file to store the file_id across runs 24 | 25 | # output module to log files tracked in a easily parsable json format 26 | - file-log: 27 | enabled: yes 28 | filename: files-json.log 29 | append: yes 30 | #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram' 31 | 32 | force-magic: yes # force logging magic on all logged files 33 | force-md5: yes # force logging of md5 checksums 34 | 35 | ``` 36 | 37 | #### stream reassembly depth. Set this to 0 (unlimited) 38 | ``` yaml 39 | stream: 40 | memcap: 32mb 41 | checksum-validation: yes # reject wrong csums 42 | inline: auto # auto will use inline mode in IPS mode, yes or no set it statically 43 | reassembly: 44 | memcap: 128mb 45 | depth: 0 #1mb 46 | 47 | ``` 48 | 49 | #### http request / response body sizes. Set those to 0 (unlimited) 50 | ``` yaml 51 | 52 | libhtp: 53 | 54 | default-config: 55 | personality: IDS 56 | 57 | # Can be specified in kb, mb, gb. Just a number indicates 58 | # it's in bytes. 59 | request-body-limit: 0 #100kb 60 | response-body-limit: 0 #100kb 61 | ``` 62 | 63 | 64 | 65 | 66 | #### rules that contain the "filestore" keyword. 67 | 68 | ``` 69 | # grep filestore /usr/local/etc/suricata/rules/* | grep -v "#" 70 | /usr/local/etc/suricata/rules/files.rules:alert http any any -> any any (msg:"FILE store all"; filestore; noalert; sid:15; rev:1;) 71 | 72 | ``` 73 | 74 | 75 | ### files 76 | 77 | ``` 78 | # ls /usr/local/var/log/suricata//files 79 | file.1 file.1.meta 80 | ``` 81 | 82 | 83 | ---- 84 | 85 | ## no files ;( 86 | 87 | 88 | 89 | ### start from cmd 90 | 91 | ``` 92 | [51155] 20/1/2016 -- 14:54:53 - (detect.c:416) (ProcessSigFiles) -- Loading rule file: /usr/local/etc/suricata/rules/files.rules 93 | 94 | [51066] 20/1/2016 -- 14:29:20 - (log-filestore.c:476) (LogFilestoreLogInitCtx) -- storing files in /usr/local/var/log/suricata//files 95 | [51066] 20/1/2016 -- 14:29:20 - (util-logopenfile.c:298) (SCConfLogOpenGeneric) -- file-log output device (regular) initialized: files-js 96 | 97 | [51067] 20/1/2016 -- 14:29:20 - (util-ioctl.c:202) (GetIfaceOffloading) -- NIC offloading on eth0: GRO: unset, LRO: unset 98 | 99 | [51067] 20/1/2016 -- 14:29:41 - (util-checksum.c:86) (ChecksumAutoModeCheck) -- No packets with invalid checksum, assuming checksum offloading is NOT used 100 | 101 | ``` 102 | 103 | 104 | ## no md5 ;( 105 | 106 | ### start from cmd 107 | ``` 108 | [51155] 20/1/2016 -- 14:55:17 - (log-filestore.c:473) (LogFilestoreLogInitCtx) -- md5 calculation requires linking against libnss 109 | [51155] 20/1/2016 -- 14:55:17 - (log-file.c:431) (LogFileLogInitCtx) -- md5 calculation requires linking against libnss 110 | ``` 111 | 112 | ``` 113 | root@secx:/usr/local/var/log/suricata/files# grep MD5 file.*.meta 114 | 115 | root@secx:/usr/local/var/log/suricata/files# /usr/local/bin/suricata -c /usr/local/etc/suricata/suricata.yaml --dump-config | grep md5 116 | outputs.1.eve-log.types.4.files.force-md5 = no 117 | outputs.13.file-store.force-md5 = yes 118 | outputs.14.file-log.force-md5 = yes 119 | app-layer.protocols.smtp.mime.body-md5 = no 120 | 121 | ``` 122 | 123 | 124 | ## push files to elasticsearch for content index 125 | 126 | see 127 | 128 | * http://tika.apache.org/1.11/formats.html#Full_list_of_Supported_Formats 129 | 130 | ``` 131 | curl -XDELETE http://localhost:9200/suricata 132 | ``` 133 | 134 | ``` bash 135 | curl -XPOST http://localhost:9200/suricata -d '{ 136 | "mappings": { 137 | "file": { 138 | "properties": { 139 | "content": { "type": "attachment" } 140 | }}}}' 141 | ``` 142 | 143 | ``` bash 144 | curl -XPOST http://localhost:9200/suricata/file/86601 -d " 145 | { 146 | \"content\": \"$(openssl base64 -in file.86601)\" 147 | } 148 | " 149 | ``` 150 | 151 | ``` bash 152 | curl -XPOST http://localhost:9200/suricata/file/_search -d ' 153 | { 154 | "query": { 155 | "query_string": { 156 | "query": "kustutatud" 157 | }}} 158 | ' 159 | ``` 160 | -------------------------------------------------------------------------------- /suricata/day_1/FlowProbe.md: -------------------------------------------------------------------------------- 1 | # Flow Probe 2 | 3 | Suricata keeps ‘flow’ records: 4 | 5 | * bidirectional 6 | * uses 5 or 7 tuple depending on VLAN support 7 | * used for storing various ‘states’ 8 | * TCP tracking and reassembly 9 | * HTTP parsing 10 | * Flow records are updated per packet 11 | * Flow records time out 12 | 13 | Flow output records: 14 | 15 | * IP protocol, source, destination, source port, destination port 16 | * packet count, bytes count 17 | * start time stamp (first packet), end time stamp (last packet) 18 | * L7 protocol as detected based on traffic content 19 | * TCP 20 | * flags seen 21 | * state at flow end 22 | 23 | ```yaml 24 | # bi-directional flows 25 | - flow 26 | ``` 27 | 28 | 29 | ```json 30 | { 31 | "_index": "logstash-2016.02.03", 32 | "_type": "SELKS", 33 | "_id": "AVKox_drE4DnavQE_df3", 34 | "_score": null, 35 | "_source": { 36 | "timestamp": "2016-02-03T22:17:03.002103+0200", 37 | "flow_id": 140023364304784, 38 | "event_type": "flow", 39 | "src_ip": "10.242.3.7", 40 | "src_port": 7653, 41 | "dest_ip": "39.255.255.25", 42 | "dest_port": 1900, 43 | "proto": "UDP", 44 | "flow": { 45 | "pkts_toserver": 1, 46 | "pkts_toclient": 0, 47 | "bytes_toserver": 143, 48 | "bytes_toclient": 0, 49 | "start": "2016-02-03T22:16:32.263423+0200", 50 | "end": "2016-02-03T22:16:32.263423+0200", 51 | "age": 0, 52 | "state": "new", 53 | "reason": "timeout" 54 | }, 55 | "@version": "1", 56 | "@timestamp": "2016-02-03T20:17:03.002Z", 57 | "host": "rtws-yellow-1-medium-1", 58 | "path": "/var/log/suricata/eve.json", 59 | "type": "SELKS" 60 | } 61 | ``` 62 | 63 | ```yaml 64 | # uni-directional flows 65 | - netflow 66 | ``` 67 | 68 | ```json 69 | { 70 | "_index": "logstash-2016.02.03", 71 | "_type": "SELKS", 72 | "_id": "AVKozJ01E4DnavQE_jPq", 73 | "_score": null, 74 | "_source": { 75 | "timestamp": "2016-02-03T22:22:07.002256+0200", 76 | "flow_id": 140023364362240, 77 | "event_type": "netflow", 78 | "src_ip": "10.22.0.10", 79 | "src_port": 35034, 80 | "dest_ip": "18.62.222.25", 81 | "dest_port": 24497, 82 | "proto": "TCP", 83 | "netflow": { 84 | "pkts": 34, 85 | "bytes": 33300, 86 | "start": "2016-02-03T22:17:45.364052+0200", 87 | "end": "2016-02-03T22:21:06.760810+0200", 88 | "age": 201 89 | }, 90 | "tcp": { 91 | "tcp_flags": "00" 92 | }, 93 | "@version": "1", 94 | "@timestamp": "2016-02-03T20:22:07.002Z", 95 | "host": "rtws-yellow-1-medium-1", 96 | "path": "/var/log/suricata/eve.json", 97 | "type": "SELKS", 98 | "geoip": { 99 | "ip": "178.62.222.205", 100 | "country_code2": "RU", 101 | "country_code3": "RUS", 102 | "country_name": "Russian Federation", 103 | "continent_code": "EU", 104 | "latitude": 60, 105 | "longitude": 100, 106 | "location": [ 107 | 100, 108 | 60 109 | ], 110 | "coordinates": [ 111 | 100, 112 | 60 113 | ] 114 | } 115 | } 116 | ``` 117 | -------------------------------------------------------------------------------- /suricata/day_1/MakeInstallFull.md: -------------------------------------------------------------------------------- 1 | # make install-full 2 | 3 | ``` 4 | root@secx:/home/student/oisf# make install-full 5 | Making install in libhtp 6 | 7 | ... 8 | 9 | install -d "/usr/local/etc/suricata/rules" 10 | /usr/bin/wget -qO - http://rules.emergingthreats.net/open/suricata-2.0/emerging.rules.tar.gz | tar -x -z -C "/usr/local/etc/suricata/" -f - 11 | 12 | You can now start suricata by running as root something like '/usr/local/bin/suricata -c /usr/local/etc/suricata//suricata.yaml -i eth0'. 13 | 14 | If a library like libhtp.so is not found, you can run suricata with: 15 | 'LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/suricata -c /usr/local/etc/suricata//suricata.yaml -i eth0'. 16 | 17 | While rules are installed now, it's highly recommended to use a rule manager for maintaining rules. 18 | The two most common are Oinkmaster and Pulledpork. For a guide see: 19 | https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Rule_Management_with_Oinkmaster 20 | 21 | ``` 22 | -------------------------------------------------------------------------------- /suricata/day_1/README.md: -------------------------------------------------------------------------------- 1 | # Single host setup 2 | 3 | The objective is simple: be able to run Suricata on single box and treat the whole traffic (1|10 Gb) with a decent number (10000ish) of rules. 4 | 5 | With the constraint not to use any non official system code (plain system and kernel if we omit a driver). 6 | 7 | see 8 | 9 | * https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/ 10 | * http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source.html 11 | 12 | ---- 13 | 14 | [next : build from source](/suricata/day_1/BuildFromSource.md) 15 | -------------------------------------------------------------------------------- /suricata/day_1/Reputation.md: -------------------------------------------------------------------------------- 1 | # Reputation 2 | 3 | see 4 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/IP_Reputation 5 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/IPReputationConfig 6 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/IPReputationFormat 7 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/IPReputationRules 8 | 9 | 10 | 11 | #### Categories file 12 | 13 | The categories file provides a mapping between a category number, short name and long description. 14 | 15 | 16 | ``` 17 | ,, 18 | ``` 19 | 20 | #### Reputation file 21 | The reputation file lists a reputation score for hosts in the categories. 22 | 23 | ``` 24 | ,, 25 | ``` 26 | 27 | #### Rule file 28 | ``` 29 | iprep:,,, 30 | ``` 31 | 32 | #### configuration 33 | 34 | ```yaml 35 | #reputation-categories-file: /usr/local/etc/suricata/iprep/categories.txt 36 | #default-reputation-path: /usr/local/etc/suricata/iprep 37 | #reputation-files: 38 | # - reputation.list 39 | 40 | default-rule-path: /usr/local/etc/suricata/rules 41 | rule-files: 42 | - ../iprep/iprep.rules 43 | ``` 44 | 45 | #### Reload 46 | 47 | Only the reputation files will be reloaded, the categories file won’t be. 48 | If categories change, Suricata should be restarted. 49 | -------------------------------------------------------------------------------- /suricata/day_1/RuleManagement.md: -------------------------------------------------------------------------------- 1 | # Rule management 2 | 3 | see 4 | 5 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricatayaml#Rule-files 6 | 7 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Rule_Management_with_Oinkmaster 8 | 9 | * https://github.com/StamusNetworks/Scirius 10 | 11 | * https://github.com/codeweaver33/mob-boss 12 | 13 | ## Config 14 | 15 | ``` yaml 16 | default-rule-path: /etc/suricata/rules/ 17 | rule-files: 18 | - backdoor.rules 19 | - bad-traffic.rules 20 | - chat.rules 21 | - ddos.rules 22 | - .... 23 | ``` 24 | 25 | ## Reloading rules 26 | 27 | Rules need to be reloaded to be active. 28 | 29 | You can send a signal to Suricata 30 | 31 | ```bash 32 | killall -USR2 suricata 33 | ``` 34 | 35 | or use the unix socket command interface 36 | 37 | ```bash 38 | suricatasc -c "reload-rules" 39 | ``` 40 | -------------------------------------------------------------------------------- /suricata/day_1/RuleProfiling.md: -------------------------------------------------------------------------------- 1 | # Rule Profiling 2 | 3 | see: 4 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Rule_Profiling 5 | * https://github.com/inliniac/suricata/blob/master/src/util-profiling-rules.c 6 | 7 | 8 | Profiling can be disabled in config, but it will still have a performance impact if compiled in. 9 | 10 | 11 | ``` 12 | rules: 13 | enabled: yes 14 | filename: rule_perf.log 15 | append: yes 16 | # Sort options: ticks, avgticks, checks, matches, maxticks 17 | sort: avgticks 18 | json: true 19 | 20 | ``` 21 | -------------------------------------------------------------------------------- /suricata/day_1/RuleThresholding.md: -------------------------------------------------------------------------------- 1 | # Rule Thresholding 2 | 3 | see https://github.com/inliniac/suricata/blob/master/threshold.config 4 | 5 | Thresholding feature is used to reduce the number of logged alerts for noisy rules. 6 | 7 | 8 | ``` 9 | threshold gen_id , sig_id , type , track , count , seconds 10 | ``` 11 | -------------------------------------------------------------------------------- /suricata/day_1/SingleBoxtuning.md: -------------------------------------------------------------------------------- 1 | @regit 2 | 3 | # Tuning 4 | 5 | ## eth1 6 | 7 | #### Verify that the offloading features are off 8 | ```bash 9 | ethtool -k eth1 10 | ``` 11 | 12 | #### Verify the send and receive buffers are the same as the hardware maximum values 13 | ```bash 14 | ethtool -g eth1 15 | ``` 16 | 17 | #### Verify the flow hash indirection table 18 | ```bash 19 | ethtool -x eth1 20 | bash 21 | ``` 22 | 23 | #### Verify that the IRQ affinity is set correctly, the output bellow shows only the first 4 CPU's 24 | ```bash 25 | cat /proc/interrupts | grep 'CPU\|eth1' 26 | ``` 27 | ---- 28 | 29 | ### turn off offloading 30 | ```bash 31 | ethtool -K eth1 rx off 32 | ethtool -K eth1 tx off 33 | ethtool -K eth1 sg off 34 | ethtool -K eth1 tso off 35 | ethtool -K eth1 gso off 36 | ethtool -K eth1 gro off 37 | ethtool -K eth1 lro off 38 | ethtool -K eth1 rxvlan off 39 | ethtool -K eth1 txvlan off 40 | ethtool -K eth1 rxhash off 41 | ``` 42 | 43 | ### set buffers 44 | 45 | 1. Get the hardware RX/TX maximum and current 46 | ```bash 47 | PRESET=$(ethtool -g $1 | tr '\n' ' ' | sed 's/.*RX:\s\+\([0-9]\+\).*TX:\s\+\([0-9]\+\).*RX:\s\+\([0-9]\+\).*TX:\s\+\([0-9]\+\).*/\1 \2 \3 \4/g') 48 | ``` 49 | 1. Set receive and transmit buffers to the hardware maximum 50 | ```bash 51 | ethtool -G $1 rx $(echo $PRESET | cut -f 1 -d " ") tx $(echo $PRESET | cut -f 2 -d " ") 52 | ``` 53 | 54 | ### balance flowhash 55 | 56 | * Balance evenly per CPU 57 | 58 | ```bash 59 | ethtool -X $1 equal $(cat /proc/cpuinfo | grep processor | wc -l) 60 | ``` 61 | 62 | ### set affinity 63 | 64 | ```bash 65 | MAX=$(cat /proc/cpuinfo | grep processor | wc -l) 66 | 67 | # Since the receive/transmit interrupts name index starts at 0, subtract 1 from the maximum 68 | let "MAX=$MAX-1" 69 | 70 | # The mask that will define the affinity 71 | MASK=1 72 | 73 | for INDEX in $(seq 0 1 $MAX); do 74 | IRQ=$(cat /proc/interrupts | grep $1-rxtx-$INDEX"$" | sed 's/\s\([0-9]\+\)\(.*\)/\1/g') 75 | 76 | # Apply the mask to the current IRQ 77 | printf "%X" $MASK > /proc/irq/$IRQ/smp_affinity 78 | 79 | # Duplicate the next mask value 80 | let "MASK=$MASK+$MASK" 81 | done 82 | ``` 83 | 84 | ### make it persistent 85 | 86 | Those configurations need to be persistent when the system is power cycled. To do that one can leverage the */sbin/ifup-local* script ;) 87 | 88 | ## Suricata configuration 89 | 90 | ### Capture configuration 91 | 92 | ```YAML 93 | af-packet: 94 | - interface: eth0 95 | threads: 16 96 | use-mmap: yes 97 | cluster-id: 42 98 | cluster-type: cluster_flow 99 | ring-size: 30000 100 | ``` 101 | 102 | ### Memory settings 103 | 104 | * memcaps 105 | * preallocation 106 | 107 | ### Detection engine settings 108 | 109 | ```YAML 110 | detect: 111 | profile: medium 112 | custom-values: 113 | toclient-groups: 3 114 | toserver-groups: 25 115 | sgh-mpm-context: auto 116 | inspection-recursion-limit: 3000 117 | 118 | mpm-algo: ac #ac-bs ac-gfbs 119 | 120 | ``` 121 | 122 | ### CPU affinity settings 123 | 124 | For worker mode: 125 | 126 | ```YAML 127 | threading: 128 | set-cpu-affinity: yes 129 | cpu-affinity: 130 | - management-cpu-set: 131 | cpu: [ "all" ] # include only these cpus in affinity settings 132 | - detect-cpu-set: 133 | cpu: [ "all" ] 134 | mode: "exclusive" # run detect threads in these cpus 135 | prio: 136 | default: "high" 137 | ``` 138 | 139 | ```YAML 140 | threading: 141 | set-cpu-affinity: yes 142 | cpu-affinity: 143 | - management-cpu-set: 144 | cpu: [ "all" ] # include only these cpus in affinity settings 145 | - receive-cpu-set: 146 | cpu: [ "all" ] # include only these cpus in affinity settings 147 | mode: "exclusive" # run detect threads in these cpus 148 | prio: 149 | default: "high" 150 | - detect-cpu-set: 151 | cpu: [ "all" ] 152 | mode: "exclusive" # run detect threads in these cpus 153 | threads: 48 154 | - verdict-cpu-set: 155 | cpu: [ 0 ] 156 | prio: 157 | default: "high" 158 | detect-thread-ratio: 1.5 159 | ``` 160 | -------------------------------------------------------------------------------- /suricata/day_1/TLSMonitor.md: -------------------------------------------------------------------------------- 1 | # TLS Monitor 2 | 3 | see 4 | 5 | * https://www.stamus-networks.com/2015/07/24/finding-self-signed-tls-certificates-suricata-and-luajit-scripting/ 6 | * https://resources.sei.cmu.edu/asset_files/Presentation/2016_017_001_449890.pdf 7 | 8 | 9 | #### eve 10 | ```json 11 | { 12 | "_index": "logstash-2016.02.03", 13 | "_type": "SELKS", 14 | "_id": "AVKnLZNJBzrbu4sSBuRs", 15 | "_score": 1, 16 | "_source": { 17 | "timestamp": "2016-02-03T14:48:46.484856+0200", 18 | "event_type": "engine", 19 | "engine": { 20 | "message": "enabling 'eve-log' module 'tls'" 21 | }, 22 | "@version": "1", 23 | "@timestamp": "2016-02-03T12:48:46.484Z", 24 | "host": "rtws-yellow-1-medium-1", 25 | "path": "/var/log/suricata/suricata.json", 26 | "type": "SELKS" 27 | } 28 | } 29 | ``` 30 | 31 | #### TLS Logging Example 32 | 33 | ```json 34 | 35 | { 36 | "_index": "logstash-2016.02.03", 37 | "_type": "SELKS", 38 | "_id": "AVKo7VVns9d5M18pys8A", 39 | "_score": null, 40 | "_source": { 41 | "timestamp": "2016-02-03T22:57:50.899152+0200", 42 | "flow_id": 139854564246144, 43 | "in_iface": "eth1", 44 | "event_type": "tls", 45 | "src_ip": "10.242.11.11", 46 | "src_port": 52226, 47 | "dest_ip": "103.234.36.144", 48 | "dest_port": 443, 49 | "proto": "TCP", 50 | "tls": { 51 | "subject": "CN=www.kzjcxnoy2jwvznhlr.net", 52 | "issuerdn": "CN=www.aczcjywnbjiuoy.com", 53 | "fingerprint": "a8:e5:5d:6c:d6:0a:2b:09:d9:9b:b2:6f:51:0c:7a:4c:11:55:93:b4", 54 | "version": "TLS 1.2" 55 | }, 56 | "@version": "1", 57 | "@timestamp": "2016-02-03T20:57:50.899Z", 58 | "host": "rtws-yellow-1-medium-3", 59 | "path": "/var/log/suricata/eve.json", 60 | "type": "SELKS" 61 | } 62 | } 63 | ``` 64 | -------------------------------------------------------------------------------- /suricata/day_1/abox/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.12" 2 | vm_box = 'ubu14' 3 | 4 | 5 | Vagrant.configure('2') do |config| 6 | config.vm.define :abox do |box| 7 | box.vm.box = vm_box 8 | box.vm.synced_folder ".", "/vagrant", disabled: true 9 | box.vm.network :private_network, ip: "#{IP}" 10 | box.vm.hostname = 'abox' 11 | box.vm.provision "shell", inline: "ifconfig eth1; netstat -ntple" 12 | box.vm.provider :virtualbox do |vb| 13 | vb.customize ["modifyvm", :id, "--memory", "4096"] 14 | vb.customize ["modifyvm", :id, "--cpus", "4"] 15 | end 16 | end 17 | end 18 | -------------------------------------------------------------------------------- /suricata/day_1/boxes/scripts/install-suricata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 0) sets gro,gso,lro,tso off 5 | # 1) addp oisf ppa 6 | # 2) installs suricata 7 | # 3) sets suricata conf as amsterdam 8 | # 4) 9 | 10 | 11 | if [ "$(id -u)" != "0" ]; then 12 | echo "ERROR - This script must be run as root" 1>&2 13 | exit 1 14 | fi 15 | 16 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 17 | HOSTNAME=$(hostname -f) 18 | 19 | ETH=$1 20 | echo "installing suricata on $IP $HOSTNAME setting * off on $ETH" 21 | ethtool -K $ETH tx off sg off gro off gso off lro off tso off 22 | 23 | #suricata 24 | add-apt-repository -y ppa:oisf/suricata-stable > /dev/null 2>&1 25 | apt-get update > /dev/null 2>&1 26 | apt-get -y install suricata > /dev/null 2>&1 27 | service suricata stop 28 | #stealing amsterdam suricata conf 29 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/suricata.yaml -O /etc/suricata/suricata.yaml 30 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/threshold.config -O /etc/suricata/threshold.config 31 | 32 | # - interface: eth0 33 | sed -i -e 's,- interface: eth0,- interface: '${ETH}',g' /etc/suricata/suricata.yaml 34 | #fake scirius rules 35 | #todo: get it from master 36 | touch /etc/suricata/rules/scirius.rules 37 | -------------------------------------------------------------------------------- /suricata/day_1/rsyslog.md: -------------------------------------------------------------------------------- 1 | [Getting started with Rsyslog](/common/rsyslogBasics.md) 2 | 3 | # Simple filtering 4 | 5 | ``` 6 | vim /etc/rsyslog.d/60-suricata-tag-to-file.conf 7 | ``` 8 | 9 | ``` 10 | if $syslogtag contains 'suricata' then /var/log/suricata-tag.log 11 | ``` 12 | 13 | # Filtering using JSON parser 14 | 15 | ``` 16 | vim /etc/rsyslog.d/61-suricata-cee-to-file.conf 17 | ``` 18 | ``` 19 | module(load="mmjsonparse") 20 | 21 | action(type="mmjsonparse") 22 | 23 | if $parsesuccess == "OK" then action( 24 | type="omfile" 25 | dirCreateMode="0700" 26 | FileCreateMode="0644" 27 | File="/var/log/suricata-cee.log" 28 | ) 29 | ``` 30 | 31 | # Enable high precision timestamps 32 | 33 | ``` 34 | sudo vim /etc/rsyslog.conf 35 | ``` 36 | ``` 37 | #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat 38 | ``` 39 | ``` 40 | sudo service rsyslog restart 41 | ``` 42 | 43 | # Parsing syslog timestamp 44 | 45 | ``` 46 | template(name="suricata-index" type="list") { 47 | constant(value="suricata-") 48 | property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4") 49 | constant(value=".") 50 | property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7") 51 | constant(value=".") 52 | property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10") 53 | } 54 | ``` 55 | 56 | # Invoking a template for dynamic naming 57 | 58 | ``` 59 | local5.info action( 60 | type="omfile" 61 | dirCreateMode="0700" 62 | FileCreateMode="0644" 63 | DynaFile="suricata-index" 64 | ) 65 | ``` 66 | 67 | # Defining custom log message format 68 | 69 | ``` 70 | template(name="JSON" type="list") { 71 | property(name="$!all-json") 72 | } 73 | ``` 74 | 75 | # Invoking elasticsearch output module 76 | 77 | ``` 78 | action( 79 | type="omelasticsearch" 80 | template="JSON" 81 | server="127.0.0.1" 82 | serverport="9200" 83 | searchIndex="suricata-index" 84 | ) 85 | ``` 86 | 87 | # Final configuration 88 | 89 | ``` 90 | module(load="omelasticsearch") 91 | module(load="mmjsonparse") 92 | 93 | template(name="suricata-index" type="list") { 94 | constant(value="suricata-") 95 | property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4") 96 | constant(value=".") 97 | property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7") 98 | constant(value=".") 99 | property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10") 100 | } 101 | 102 | template(name="JSON" type="list") { 103 | property(name="$!all-json") 104 | } 105 | 106 | if $syslogtag contains 'suricata' and $msg startswith ' @cee:' then { 107 | 108 | action(type="mmjsonparse") 109 | 110 | if $parsesuccess == "OK" then action( 111 | type="omelasticsearch" 112 | template="JSON" 113 | server="127.0.0.1" 114 | serverport="9200" 115 | searchIndex="suricata-index" 116 | dynSearchIndex="on" 117 | ) 118 | 119 | } 120 | ``` -------------------------------------------------------------------------------- /suricata/day_2/Grafana.md: -------------------------------------------------------------------------------- 1 | # grafana 2 | 3 | > *graph* and *dashboard* builder for visualizing **time series metrics**. 4 | 5 | read: 6 | 7 | * http://docs.grafana.org/guides/gettingstarted/ 8 | * http://docs.grafana.org/installation/troubleshooting/ 9 | * http://docs.grafana.org/datasources/influxdb/ 10 | * http://docs.grafana.org/datasources/elasticsearch/ 11 | * https://docs.influxdata.com/influxdb/v0.10/query_language/functions/ 12 | * http://www.lucenetutorial.com/lucene-query-syntax.html 13 | -------------------------------------------------------------------------------- /suricata/day_2/README.md: -------------------------------------------------------------------------------- 1 | # suricata on more than one box 2 | 3 | Running commands on remote hosts, also known as remote execution, is the core feature we need. 4 | 5 | We need also: 6 | 7 | * to know, how hw is holding 8 | * good fast responsive place to put all our stuff 9 | 10 | ![graphvizrocks](./suriN.png) 11 | 12 | see [source](./suriN.dot) 13 | 14 | ---- 15 | 16 | [next : metrix](/suricata/day_2/SetUpMetrics.md) 17 | -------------------------------------------------------------------------------- /suricata/day_2/SetUpElastic.md: -------------------------------------------------------------------------------- 1 | # set up elasticsearch 2 | 3 | 4 | we use: 5 | 6 | * no_http for data nodes 7 | * no_data for master & client nodes 8 | 9 | see [install script](./boxes/scripts/install-elastic.sh) 10 | 11 | see also 12 | 13 | * https://www.elastic.co/guide/en/elasticsearch/reference/2.2/modules-node.html#modules-node 14 | * https://github.com/aol/moloch/wiki/Multiple-Host-HOWTO 15 | * https://www.elastic.co/products/elasticsearch 16 | 17 | ---- 18 | 19 | [next : minions](/suricata/day_2/SetUpMinions.md) 20 | -------------------------------------------------------------------------------- /suricata/day_2/SetUpMaster.md: -------------------------------------------------------------------------------- 1 | # set up master 2 | 3 | see [install script](./boxes/scripts/install-salt-master.sh) 4 | 5 | see also 6 | 7 | * https://docs.saltstack.com/en/latest/ref/configuration/master.html 8 | 9 | ---- 10 | 11 | [next : admin box](/suricata/day_2/admin.md) 12 | -------------------------------------------------------------------------------- /suricata/day_2/SetUpMetrics.md: -------------------------------------------------------------------------------- 1 | # set up metrics 2 | 3 | * ethx ? 4 | * cpu ? 5 | * ram ? 6 | * disk ? 7 | * ... 8 | 9 | good old munin, syweb, nagios, .... 10 | 11 | however, we use: 12 | 13 | * [telegraf](./boxes/scripts/install-telegraf.sh) to collect metrix 14 | * [influxdb](./boxes/scripts/install-influxdb.sh) to hold metrix 15 | * [grafana](./boxes/scripts/install-grafana.sh) to show metrix 16 | 17 | see 18 | * http://oss.oetiker.ch/rrdtool/ 19 | * https://github.com/influxdata/telegraf 20 | * https://github.com/influxdata/influxdb 21 | * https://influxdata.com/wp-content/themes/influx/images/TICK-Stack.png 22 | * https://github.com/grafana/grafana 23 | * http://play.grafana.org 24 | 25 | 26 | ---- 27 | 28 | [next : elastic](/suricata/day_2/SetUpElastic.md) 29 | -------------------------------------------------------------------------------- /suricata/day_2/SetUpMinions.md: -------------------------------------------------------------------------------- 1 | # set up Minions 2 | 3 | again a long list like cfengine, ansible, pupet,.... 4 | 5 | 6 | see [install script](./boxes/scripts/install-salt-minion.sh) 7 | 8 | * https://docs.saltstack.com/en/latest/ref/configuration/minion.html 9 | 10 | 11 | 12 | 13 | ---- 14 | 15 | [next : master](/suricata/day_2/SetUpMaster.md) 16 | -------------------------------------------------------------------------------- /suricata/day_2/admin.md: -------------------------------------------------------------------------------- 1 | # admin box 2 | 3 | see https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/Vagrantfile#L154 4 | 5 | tldr Amstelredamme + salt-master + influxdb + grafana 6 | 7 | [next : go over sceleton](/suricata/day_2/notes.md) 8 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-bro-master-and-workers.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # This script will install bro master 4 | # and set up ssh keys on workers and proxy's 5 | # 6 | # ! depends on salt 7 | # 8 | # params : 9 | # $1 manager ip 10 | # $2 salt key pattern to match bro workers and proxyes 11 | # $3 proxyes ip list, sepatarated with comma 12 | # $4 workers ip list, sepatarated with comma 13 | 14 | 15 | if [ "$(id -u)" != "0" ]; then 16 | echo "ERROR - This script must be run as root" 1>&2 17 | exit 1 18 | fi 19 | 20 | echo "installing bro with:" 21 | echo "manager: $1" 22 | echo "pattern: $2" 23 | echo "proxyes: $3" 24 | echo "workers: $4" 25 | echo "wait .." 26 | 27 | MANAGER=$1 28 | BROS=$2 29 | PROXYS=$3 30 | WORKERS=$4 31 | 32 | # add bro repo and install 33 | echo "deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_14.04/ /" >> /etc/apt/sources.list.d/bro.list 34 | wget -4 -q http://download.opensuse.org/repositories/network:bro/xUbuntu_14.04/Release.key 35 | apt-key add - < Release.key > /dev/null 2>&1 36 | apt-get -qq update > /dev/null 2>&1 37 | apt-get -y install bro > /dev/null 2>&1 38 | #prepare master key and ssh conf 39 | adduser --system --home /opt/bro --no-create-home --ingroup bro --disabled-password --shell /bin/false bro 40 | mkdir /opt/bro/.ssh && chown -R bro:bro /opt/bro && chmod 750 /opt/bro/.ssh 41 | su - bro -s /bin/bash -c 'echo -en "\n\n"|ssh-keygen -t rsa -f /opt/bro/.ssh/id_rsa' 42 | echo "# auto generated by $0 @ $(date) " > /opt/bro/.ssh/config 43 | echo "$PROXYS,$WORKERS" | sed 's/,/\n/g'|while read IP; 44 | do 45 | echo "Host $IP" >> /opt/bro/.ssh/config 46 | echo " Hostname $IP" >> /opt/bro/.ssh/config 47 | echo " IdentityFile ~/.ssh/id_rsa" >> /opt/bro/.ssh/config 48 | echo " StrictHostKeyChecking no" >> /opt/bro/.ssh/config 49 | done 50 | 51 | #distribute to nodes via salt 52 | salt-key -A -y > /dev/null 2>&1 53 | salt-key -L 54 | sleep 1 55 | salt "*$BROS*" test.ping > /dev/null 2>&1 56 | salt "*$BROS*" cmd.run 'addgroup --system bro --quiet' > /dev/null 2>&1 57 | salt "*$BROS*" cmd.run 'adduser --system --home /opt/bro --no-create-home --ingroup bro --disabled-password --shell /bin/bash bro' > /dev/null 2>&1 58 | salt "*$BROS*" cmd.run 'mkdir -p /opt/bro/.ssh' > /dev/null 2>&1 59 | salt-cp "*$BROS*" /opt/bro/.ssh/id_rsa.pub /opt/bro/.ssh/bro-manager.pub > /dev/null 2>&1 60 | salt "*$BROS*" cmd.run 'cat /opt/bro/.ssh/bro-manager.pub >> /opt/bro/.ssh/authorized_keys' > /dev/null 2>&1 61 | #salt "*$BROS*" cmd.run 'setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro' 62 | #salt "*$BROS*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/bro' 63 | cat /etc/hosts | grep -v 127| grep $(hostname) > /opt/bro/hosts 64 | salt-cp "*$BROS*" /opt/bro/hosts /opt/bro/hosts > /dev/null 2>&1 65 | salt "*$BROS*" cmd.run 'cat /opt/bro/hosts >> /etc/hosts' > /dev/null 2>&1 66 | salt "*$BROS*" cmd.run 'chown -R bro:bro /opt/bro;' > /dev/null 2>&1 67 | 68 | # create node.cfg 69 | cat > /opt/bro/etc/node.cfg<> /opt/bro/etc/node.cfg 77 | echo "type=proxy" >> /opt/bro/etc/node.cfg 78 | echo "host=$IP" >> /opt/bro/etc/node.cfg 79 | done 80 | echo "$WORKERS" | sed 's/,/\n/g'|while read IP; 81 | do 82 | echo "[worker-$IP]" >> /opt/bro/etc/node.cfg 83 | echo "type=worker" >> /opt/bro/etc/node.cfg 84 | echo "host=$IP" >> /opt/bro/etc/node.cfg 85 | echo "interface=eth1" >> /opt/bro/etc/node.cfg 86 | done 87 | chown -R bro:bro /opt/bro 88 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl deploy' > /dev/null 2>&1 89 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl stop' > /dev/null 2>&1 90 | salt "*$BROS*" cmd.run 'setcap "CAP_NET_RAW+eip" /opt/bro/bin/bro' 91 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl start' > /dev/null 2>&1 92 | sleep 3 93 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl status' 94 | echo "to start using bro use 'su - bro -s /bin/bash'" 95 | # tell telegraf there is bro 96 | cat > /etc/telegraf/telegraf.d/bro.conf < /dev/null 2>&1 101 | salt-cp "*$BROS*" /etc/telegraf/telegraf.d/bro.conf /etc/telegraf/telegraf.d/bro.conf > /dev/null 2>&1 102 | salt "*$BROS*" service.restart telegraf > /dev/null 2>&1 103 | 104 | su - bro -s /bin/bash -c '/opt/bro/bin/broctl stop' > /dev/null 2>&1 105 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-elastic.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) installs elasticsearch as 5 | # a) master no data 6 | # b) client no data 7 | # c) dana no http 8 | # 2) set unicast hosts 9 | # 3) sets cluster 10 | # 4) add elasticsearch to telegraf config 11 | # 12 | 13 | 14 | if [ "$(id -u)" != "0" ]; then 15 | echo "ERROR - This script must be run as root" 1>&2 16 | exit 1 17 | fi 18 | 19 | IP=$1 20 | NAME=$2 21 | CLUSTER=$3 22 | COUNTER=$4 23 | UNICASTHOSTS=$5 24 | TYPE=$6 #master,client or data(default) 25 | DATAPATH="/srv" 26 | 27 | echo "adding new node to cluster: ${CLUSTER} node: ${NAME} bind: ${IP} unicast host: ${UNICASTHOSTS} type: ${TYPE}" 28 | 29 | INSTALL_DIR=/var/cache/wget 30 | 31 | ES=2.3.2 32 | 33 | mkdir -p ${INSTALL_DIR} 34 | cd ${INSTALL_DIR} 35 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 36 | wget -N -P /var/cache/wget -4 -q https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES}.deb 37 | fi 38 | if [ ! -f "elasticsearch-${ES}.deb" ]; then 39 | echo "$(date) ${NAME} $0[$$]: {elastic: {status:ERROR, msg: missing elasticsearch-${ES}.deb}" 40 | exit -1 41 | else 42 | echo -e "Y" | dpkg -i elasticsearch-${ES}.deb > /dev/null 2>&1 43 | service elasticsearch stop > /dev/null 2>&1 44 | /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head 2>&1 > /dev/null 45 | echo "# generated ${date} by $0" > /etc/elasticsearch/elasticsearch.yml 46 | mkdir -p $DATAPATH 47 | chown elasticsearch $DATAPATH 48 | echo "path.data: ${DATAPATH}" >> /etc/elasticsearch/elasticsearch.yml 49 | echo "cluster.name: ${CLUSTER}" >> /etc/elasticsearch/elasticsearch.yml 50 | echo "node.name: ${NAME} " >> /etc/elasticsearch/elasticsearch.yml 51 | echo "node.max_local_storage_nodes: 1 " >> /etc/elasticsearch/elasticsearch.yml 52 | echo "index.number_of_replicas: 0 " >> /etc/elasticsearch/elasticsearch.yml 53 | echo "index.fielddata.cache: node " >> /etc/elasticsearch/elasticsearch.yml 54 | echo "indices.fielddata.cache.size: 40% " >> /etc/elasticsearch/elasticsearch.yml 55 | echo "http.compression: true " >> /etc/elasticsearch/elasticsearch.yml 56 | echo "bootstrap.mlockall: true " >> /etc/elasticsearch/elasticsearch.yml 57 | echo "network.host: ${IP}" >> /etc/elasticsearch/elasticsearch.yml 58 | echo "# split brain here ;( " >> /etc/elasticsearch/elasticsearch.yml 59 | echo "discovery.zen.minimum_master_nodes: 1" >> /etc/elasticsearch/elasticsearch.yml 60 | echo "discovery.zen.ping.multicast.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 61 | echo 'discovery.zen.ping.unicast.hosts: ['${UNICASTHOSTS}']' >> /etc/elasticsearch/elasticsearch.yml 62 | if [ "$TYPE" == "master" ]; 63 | then 64 | echo "node.master: true" >> /etc/elasticsearch/elasticsearch.yml 65 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 66 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 67 | else 68 | if [ "$TYPE" == "client" ]; 69 | then 70 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 71 | echo "node.data: false" >> /etc/elasticsearch/elasticsearch.yml 72 | echo "http.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 73 | else 74 | # defaults to data node 75 | echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml 76 | echo "node.data: true" >> /etc/elasticsearch/elasticsearch.yml 77 | echo "http.enabled: false" >> /etc/elasticsearch/elasticsearch.yml 78 | fi 79 | fi 80 | service elasticsearch start > /dev/null 2>&1 81 | #sleep 2 82 | #service elasticsearch status 83 | #service telegraf stop > /dev/null 2>&1 84 | if [ "$TYPE" == "master" ]; then 85 | cat > /etc/telegraf/telegraf.d/elasticsearch.conf < /etc/telegraf/telegraf.d/elasticsearch.conf < /dev/null 2>&1 99 | #sleep 1 100 | #service telegraf status 101 | 102 | 103 | fi 104 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-evebox.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs evebox 5 | # 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | ELASTIC=$1 14 | 15 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 16 | HOSTNAME=$(hostname -f) 17 | 18 | echo "installing evebox on ${IP} ${HOSTNAME} sets elasticsearch to ${ELASTIC} ..." 19 | 20 | apt-get -y install unzip > /dev/null 2>&1 21 | cd /opt/ 22 | wget -4 -q https://bintray.com/artifact/download/jasonish/evebox/evebox-linux-amd64.zip 23 | unzip -o evebox-linux-amd64.zip 24 | /opt/evebox-linux-amd64/evebox --version 25 | echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml 26 | echo "http.cors.allow-origin: \"/.*/\"" >> /etc/elasticsearch/elasticsearch.yml 27 | service elasticsearch restart > /dev/null 2>&1 28 | 29 | ln -s /opt/evebox-linux-amd64 /opt/evebox 30 | ln -s /opt/evebox-linux-amd64/evebox /opt/evebox-linux-amd64/evebox-server 31 | addgroup --system evebox --quiet 32 | adduser --system --home /opt/evebox --no-create-home --ingroup evebox --disabled-password --shell /bin/false evebox 33 | > /dev/null 2>&1 34 | touch /etc/default/evebox-server 35 | mkdir -p /etc/evebox 36 | touch /etc/evebox/evebox.conf 37 | wget -4 -q https://raw.githubusercontent.com/hillar/CDMCS/master/suricata/day_2/boxes/scripts/evebox-server -O /etc/init.d/evebox-server 38 | chmod +x /etc/init.d/evebox-server 39 | update-rc.d evebox-server defaults 95 10 > /dev/null 2>&1 40 | service evebox-server start > /dev/null 2>&1 41 | 42 | 43 | cat > /etc/telegraf/telegraf.d/evebox.conf < /dev/null 2>&1 48 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-grafana.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs grafana 5 | # 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 14 | HOSTNAME=$(hostname -f) 15 | 16 | echo "installing grafana on ${IP} ${HOSTNAME}" 17 | 18 | INSTALL_DIR=/var/cache/wget 19 | 20 | #https://grafanarel.s3.amazonaws.com/builds/grafana_3.0.0-beta71462173753_amd64.deb 21 | GRFN="3.0.0-beta71462173753" 22 | mkdir -p ${INSTALL_DIR} 23 | cd ${INSTALL_DIR} 24 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 25 | wget -N -P /var/cache/wget -4 -q https://grafanarel.s3.amazonaws.com/builds/grafana_${GRFN}_amd64.deb 26 | fi 27 | if [ ! -f "grafana_${GRFN}_amd64.deb" ]; then 28 | echo "$(date) ${HOSTNAME} $0[$$]: {grafana: {status:WARNING, msg: missing grafana_${GRFN}_amd64.deb}" 29 | #exit -1 30 | else 31 | #rm /var/lib/grafana/grafana.db 32 | apt-get install -y adduser libfontconfig > /dev/null 2>&1 33 | echo -e "Y"|dpkg -i grafana_${GRFN}_amd64.deb > /dev/null 2>&1 34 | update-rc.d grafana-server defaults 95 10 > /dev/null 2>&1 35 | sed -i -e 's,domain = localhost,domain = '${IP}',g' /etc/grafana/grafana.ini 36 | service grafana-server start > /dev/null 2>&1 37 | sleep 1 38 | apt-get -y install sqlite > /dev/null 2>&1 39 | sqlite3 /var/lib/grafana/grafana.db "INSERT INTO \"data_source\" VALUES(1,1,0,\"influxdb\",\"telegraf\",\"proxy\",\"http://$IP:8086\",\"admin\",\"admin\",\"telegraf\",0,\"\",\"\",1,\"{}\",\"2016-03-07 18:21:44\",\"2016-03-07 18:50:56\",0);" 40 | 41 | #service grafana-server status 42 | #service telegraf stop > /dev/null 2>&1 43 | cat > /etc/telegraf/telegraf.d/grafana.conf < /dev/null 2>&1 48 | #sleep 1 49 | #service telegraf status 50 | fi 51 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-influxdb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs influxdb 5 | # 2) sets influxdb to $1 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | BIND=$1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | echo "installing influxdb on ${IP} ${HOSTNAME} setting bind to ${BIND}..." 18 | 19 | INSTALL_DIR=/var/cache/wget 20 | 21 | 22 | 23 | #https://dl.influxdata.com/influxdb/releases/influxdb_0.13.0~rc2_amd64.deb 24 | INFLX="0.13.0~rc2" 25 | 26 | mkdir -p ${INSTALL_DIR} 27 | cd ${INSTALL_DIR} 28 | if [ ! -f "influxdb_${INFLX}_amd64.deb" ]; then 29 | wget -N -P /var/cache/wget -4 -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLX}_amd64.deb 30 | fi 31 | if [ ! -f "influxdb_${INFLX}_amd64.deb" ]; then 32 | echo "$(date) ${NAME} $0[$$]: {influxdb: {status:ERROR, msg: missing influxdb_${INFLX}_amd64.deb}" 33 | exit -1 34 | else 35 | echo -e "Y"|dpkg -i influxdb_${INFLX}_amd64.deb > /dev/null 2>&1 36 | service influxdb start > /dev/null 2>&1 37 | sleep 1 38 | service influxdb status 39 | #prepare for telegraf 40 | 41 | service telegraf stop > /dev/null 2>&1 42 | curl -s -XPOST http://localhost:8086/query --data-urlencode "q=DROP DATABASE telegraf" 43 | curl -s -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE telegraf" 44 | curl -s -XPOST http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY one_day_only ON telegraf DURATION 1d REPLICATION 1 DEFAULT" 45 | #sed -i -e 's,localhost,'${METRICS_SERVER}',g' /etc/influxdb/influxdb.conf 46 | cat > /etc/telegraf/telegraf.d/influxdb.conf < /dev/null 2>&1 53 | sleep 1 54 | service telegraf status 55 | fi 56 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-kibana4.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # installs kibana 5 | # 6 | 7 | if [ "$(id -u)" != "0" ]; then 8 | echo "ERROR - This script must be run as root" 1>&2 9 | exit 1 10 | fi 11 | 12 | KBN=4.4 13 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 14 | HOSTNAME=$(hostname -f) 15 | 16 | ELASTIC=$1 17 | echo "installing kibana$KBN on $IP $HOSTNAME setting elastic to $ELASTIC" 18 | 19 | 20 | 21 | wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add - > /dev/null 2>&1 22 | echo 'deb http://packages.elastic.co/kibana/'${KBN}'/debian stable main' > /etc/apt/sources.list.d/kibana.list 23 | apt-get update > /dev/null 2>&1 24 | apt-get -y install kibana > /dev/null 2>&1 25 | /opt/kibana/bin/kibana plugin -i kibana/timelion > /dev/null 2>&1 26 | #chown -R kibana.kibana /opt/kibana/optimize/ 27 | # server.host: "0.0.0.0" 28 | sed -i -e 's,# server.host: "0.0.0.0",server.host: "'${IP}'",g' /opt/kibana/config/kibana.yml 29 | # elasticsearch.url: "http://10.242.11.29:9200" 30 | sed -i -e 's,# elasticsearch.url: "http://localhost:9200",elasticsearch.url: "http://'${ELASTIC}':9200",g' /opt/kibana/config/kibana.yml 31 | chown -R kibana:kibana /opt/kibana/optimize/ 32 | service kibana start > /dev/null 2>&1 33 | cat > /etc/telegraf/telegraf.d/kibana.conf < /dev/null 2>&1 38 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-logstash-for-bro-csv.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs logstash 5 | # 2) sets elastic to $1 in conf.d/bro-*.conf 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 13 | HOSTNAME=$(hostname -f) 14 | ELASTIC=$1 15 | echo "installing logstash on $IP $HOSTNAME setting elasticsearch on $ELASTIC" 16 | 17 | #ELASTIC=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 18 | echo 'deb http://packages.elasticsearch.org/logstash/2.2/debian stable main' > /etc/apt/sources.list.d/logstash.list 19 | apt-get update > /dev/null 2>&1 20 | apt-get -y --force-yes install logstash > /dev/null 2>&1 21 | /opt/logstash/bin/plugin install logstash-filter-translate > /dev/null 2>&1 22 | /opt/logstash/bin/plugin install logstash-filter-de_dot > /dev/null 2>&1 23 | #stealing from Tim Molter 24 | cd /etc/logstash/conf.d/ 25 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-conn_log.conf 26 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-dns_log.conf 27 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-files_log.conf 28 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-http_log.conf 29 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-notice_log.conf 30 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-ssh_log.conf 31 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-ssl_log.conf 32 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-weird_log.conf 33 | wget -q https://raw.githubusercontent.com/timmolter/logstash-dfir/master/conf_files/bro/bro-x509_log.conf 34 | # elasticsearch { host => localhost } 35 | sed -i -e 's,host => localhost,hosts => "'${ELASTIC}'"\n index => "logstash-%{+YYYY.MM.dd.HH}",g' /etc/logstash/conf.d/*.conf 36 | sed -i -e '/date/i \ \ \ \ \de_dot{ }' /etc/logstash/conf.d/*.conf 37 | #sed -i -e 's/id.orig_h/id_orig_h/g' -e 's/id.resp_h/id_resp_h/g' -e 's/id.orig_p/id_orig_p/g' -e 's/id.resp_p/id_resp_p/g' /etc/logstash/conf.d/*.conf 38 | #path => "/nsm/bro/logs/current/conn.log" 39 | sed -i -e 's,/nsm/bro/logs/current/,/opt/bro/logs/current/,g' /etc/logstash/conf.d/*.conf 40 | 41 | 42 | sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest 43 | 44 | adduser logstash bro 45 | service logstash start > /dev/null 2>&1 46 | cat > /etc/telegraf/telegraf.d/logstash.conf < /dev/null 2>&1 51 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-logstash.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs logstash 5 | # 2) sets elastic to $1 in conf.d/suricata.conf 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 13 | HOSTNAME=$(hostname -f) 14 | ELASTIC=$1 15 | echo "installing logstash on $IP $HOSTNAME setting elasticsearch on $ELASTIC" 16 | 17 | #ELASTIC=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 18 | echo 'deb http://packages.elasticsearch.org/logstash/2.2/debian stable main' > /etc/apt/sources.list.d/logstash.list 19 | apt-get update > /dev/null 2>&1 20 | apt-get -y --force-yes install logstash > /dev/null 2>&1 21 | #stealing amsterdam losgstash conf 22 | wget -4 -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/logstash/conf.d/logstash.conf -O /etc/logstash/conf.d/suricata.conf 23 | # hosts => elasticsearch 24 | sed -i -e 's,hosts => elasticsearch,hosts => "'${ELASTIC}'"\n index => "logstash-%{+YYYY.MM.dd.HH}",g' /etc/logstash/conf.d/suricata.conf 25 | #fix this hack 26 | chmod 777 /var/log/suricata/eve.json 27 | service logstash start > /dev/null 2>&1 28 | cat > /etc/telegraf/telegraf.d/logstash.conf < /dev/null 2>&1 33 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-moloch-capture.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/suricata/day_2/boxes/scripts/install-moloch-capture.sh -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-moloch-viewer.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/suricata/day_2/boxes/scripts/install-moloch-viewer.sh -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-salt-master.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) adds ppa:saltstack/salt 5 | # 2) installs salt-master 6 | # 3) accepts all minions contacting in 10 secs 7 | # 8 | 9 | if [ "$(id -u)" != "0" ]; then 10 | echo "ERROR - This script must be run as root" 1>&2 11 | exit 1 12 | fi 13 | 14 | echo "installing salt-master on $(hostname -f) ..." 15 | 16 | echo "LC_ALL=en_US.UTF-8" >> /etc/environment 17 | echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4 18 | add-apt-repository -y ppa:saltstack/salt > /dev/null 2>&1 19 | apt-get update > /dev/null 2>&1 20 | apt-get -y install salt-master > /dev/null 2>&1 21 | #echo "waiting for minions.." 22 | #sleep 11 23 | #salt-key -L 24 | #salt-key -A -y 25 | #sleep 3 26 | #salt '*' test.ping 27 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-salt-minion.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 1) adds ppa:saltstack/salt 5 | # 2) installs salt-minion 6 | # 3) sets master to $1 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | MASTER=$1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | echo "installing salt-minion on ${IP} ${HOSTNAME} setting master to ${MASTER}..." 17 | echo "LC_ALL=en_US.UTF-8" >> /etc/environment 18 | echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4 19 | add-apt-repository -y ppa:saltstack/salt > /dev/null 2>&1 20 | apt-get update > /dev/null 2>&1 21 | apt-get -y install salt-minion > /dev/null 2>&1 22 | service salt-minion stop > /dev/null 2>&1 23 | echo "master: ${MASTER}" > /etc/salt/minion 24 | echo "id: ${HOSTNAME}" >> /etc/salt/minion 25 | service salt-minion start > /dev/null 2>&1 26 | sleep 1 27 | service salt-minion status 28 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-scirius.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ELASTICIP=$1 4 | EVEBOXIP=$2 5 | KIBANAIP=$3 6 | 7 | #cd /opt 8 | #wget https://github.com/inliniac/suricata/archive/master.tar.gz 9 | #tar -xzf master.tar.gz 10 | #cd suricata-master/scripts/suricatasc/ 11 | #python setup.py intall 12 | #suricata 13 | add-apt-repository -y ppa:oisf/suricata-stable > /dev/null 2>&1 14 | apt-get update > /dev/null 2>&1 15 | apt-get -y install suricata --force-yes > /dev/null 2>&1 16 | service suricata stop 17 | 18 | # Scirius 19 | # see https://github.com/StamusNetworks/scirius#installation-and-setup 20 | DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y wget python-pip python-dev git gcc > /dev/null 2>&1 21 | cd /tmp 22 | wget -4 -q https://github.com/StamusNetworks/scirius/archive/master.tar.gz > /dev/null 2>&1 23 | mkdir -p /opt/selks/sciriusdata 24 | cd /opt/selks 25 | tar zxf /tmp/master.tar.gz 26 | ln -sf /opt/selks/scirius-master /opt/selks/scirius 27 | cd /opt/selks/scirius 28 | pip install -r requirements.txt > /dev/null 2>&1 29 | ln -s /etc/scirius/local_settings.py /opt/selks/scirius/scirius/ 30 | pip install -U six > /dev/null 2>&1 31 | pip install urllib3 --upgrade > /dev/null 2>&1 32 | mkdir /etc/scirius 33 | cat > /etc/scirius/local_settings.py < /dev/null 2>&1 50 | python manage.py makemigrations > /dev/null 2>&1 51 | python manage.py migrate > /dev/null 2>&1 52 | python manage.py createcachetable my_cache_table > /dev/null 2>&1 53 | python manage.py addsource "ETOpen Ruleset" https://rules.emergingthreats.net/open/suricata-2.0.7/emerging.rules.tar.gz http sigs > /dev/null 2>&1 54 | python manage.py addsource "SSLBL abuse.ch" https://sslbl.abuse.ch/blacklist/sslblacklist.rules http sig > /dev/null 2>&1 55 | python manage.py defaultruleset "Default SELKS ruleset" > /dev/null 2>&1 56 | python manage.py disablecategory "Default SELKS ruleset" stream-events > /dev/null 2>&1 57 | python manage.py addsuricata $(hostname) "Suricata on SELKS" /etc/suricata/rules "Default SELKS ruleset" > /dev/null 2>&1 58 | python manage.py updatesuricata > /dev/null 2>&1 59 | suricata -T -c /etc/suricata/suricata.yaml 60 | # set u:p to admin:password 61 | echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'admin@localhost', 'admin')" | python manage.py shell 62 | python manage.py runserver 0.0.0.0:8000 > /var/log/scirius.log 2>&1 & 63 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-suricata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 0) sets gro,gso,lro,tso off 5 | # 1) addp oisf ppa 6 | # 2) installs suricata 7 | # 3) sets suricata conf as amsterdam 8 | # 4) 9 | 10 | 11 | if [ "$(id -u)" != "0" ]; then 12 | echo "ERROR - This script must be run as root" 1>&2 13 | exit 1 14 | fi 15 | 16 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 17 | HOSTNAME=$(hostname -f) 18 | 19 | ETH=$1 20 | echo "installing suricata on $IP $HOSTNAME setting * off on $ETH" 21 | ethtool -K $ETH tx off sg off gro off gso off lro off tso off 22 | 23 | #suricata 24 | add-apt-repository -y ppa:oisf/suricata-stable > /dev/null 2>&1 25 | apt-get update > /dev/null 2>&1 26 | apt-get -y install suricata > /dev/null 2>&1 27 | service suricata stop 28 | #stealing amsterdam suricata conf 29 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/suricata.yaml -O /etc/suricata/suricata.yaml 30 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/threshold.config -O /etc/suricata/threshold.config 31 | 32 | # - interface: eth0 33 | sed -i -e 's,- interface: eth0,- interface: '${ETH}',g' /etc/suricata/suricata.yaml 34 | #fake scirius rules 35 | #todo: get it from master 36 | touch /etc/suricata/rules/scirius.rules 37 | service suricata start > /dev/null 2>&1 38 | #sleep 2 39 | #service suricata status 40 | cat > /etc/telegraf/telegraf.d/suricata.conf < /dev/null 2>&1 45 | #sleep 1 46 | #service telegraf status 47 | -------------------------------------------------------------------------------- /suricata/day_2/boxes/scripts/install-telegraf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script: 4 | # 1) installs telegraf 5 | # 2) sets influxdb to $1 6 | # 7 | 8 | if [ "$(id -u)" != "0" ]; then 9 | echo "ERROR - This script must be run as root" 1>&2 10 | exit 1 11 | fi 12 | 13 | MASTER=$1 14 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 15 | HOSTNAME=$(hostname -f) 16 | 17 | echo "installing telegraf on ${IP} ${HOSTNAME} setting influxdb to ${MASTER}" 18 | 19 | #http://get.influxdb.org/telegraf/telegraf_0.12.1-1_amd64.deb 20 | TLGF=0.12.1 21 | 22 | INSTALL_DIR=/var/cache/wget 23 | 24 | mkdir -p ${INSTALL_DIR} 25 | cd ${INSTALL_DIR} 26 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 27 | wget -N -P /var/cache/wget -4 -q http://get.influxdb.org/telegraf/telegraf_${TLGF}-1_amd64.deb 28 | fi 29 | if [ ! -f "telegraf_${TLGF}-1_amd64.deb" ]; then 30 | echo "$(date) ${NAME} $0[$$]: {telegraf: {status:ERROR, msg: missing telegraf_${TLGF}_amd64.deb}" 31 | exit -1 32 | else 33 | rm -rf /var/log/telegraf 34 | rm -rf /etc/telegraf 35 | apt-get -y remove --purge telegraf > /dev/null 2>&1 36 | #echo -e "Y"| 37 | dpkg -i telegraf_${TLGF}-1_amd64.deb > /dev/null 2>&1 38 | 39 | cat > /etc/telegraf/telegraf.conf < /etc/telegraf/telegraf.d/common.conf < /etc/telegraf/telegraf.d/telegraf.conf < /dev/null 2>&1 69 | #sleep 1 70 | #service telegraf status 71 | fi 72 | -------------------------------------------------------------------------------- /suricata/day_2/homenet.md: -------------------------------------------------------------------------------- 1 | # set homenet 2 | 3 | hints: 4 | 5 | * use salt-cp 6 | * use *include* aka split configuration 7 | 8 | [next : push new ruleset to all suricata boxes](/suricata/day_2/pushrules.md) 9 | -------------------------------------------------------------------------------- /suricata/day_2/notes.md: -------------------------------------------------------------------------------- 1 | ## check prepared sceleton 2 | 3 | * XX <- studentnumber [1..18] 4 | 5 | 6 | * 10.242.11.XX0 student-XX-admin https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/Vagrantfile#L158 7 | * 10.242.11.XX1 student-XX-suricata-1 https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/Vagrantfile#L120 8 | * 10.242.11.XX9 student-XX-elasticsearch-1 https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/Vagrantfile#L88 9 | 10 | 11 | 12 | 13 | ### ssh into admin box 14 | 15 | 1. ssh user@10.242.11.XX0 16 | 1. w 17 | 1. last 18 | 1. cat /etc/salt/minion 19 | ``` 20 | master: 10.242.11.9 21 | id: student-XX-admin 22 | ``` 23 | 1. ps aux | less 24 | 1. sudo -i 25 | 1. salt-key -L 26 | 1. salt-key -A -y 27 | 1. salt '\*' test.ping 28 | 1. salt '\*' service.status telegraf 29 | 1. salt '\*suricata\*' service.status suricata 30 | 1. salt '\*suricata\*' service.status logstash 31 | 1. salt '\*elastic\*' service.status elasticsearch 32 | 1. service elasticsearch status 33 | 1. service influxdb status 34 | 1. service telegraf status 35 | 1. netstat -ntple 36 | 37 | 38 | ### :9200 elasticsearch 39 | 40 | 1. http://10.242.11.XX0:9200/ 41 | 1. http://10.242.11.XX0:9200/_cat 42 | 1. http://10.242.11.XX0:9200/_cat/nodes?v 43 | 44 | ``` 45 | $ curl -XGET http://10.242.11.XX0:9200/_cat/nodes?v 46 | ``` 47 | 48 | 49 | > open in browser http://10.242.11.XX0:9200/_plugin/head 50 | 51 | 52 | 1. click *Indices* 53 | 1. click *Browser* 54 | 1. click *Overview* 55 | 56 | 57 | ### :5636 EveBox 58 | 59 | > open in browser http://10.242.11.XX0:5636/ https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/scripts/install-evebox.sh#L30 60 | 61 | 62 | 1. click *settings* 63 | 1. set *Elastic Search URL* to *http://10.242.11.XX0:9200* 64 | 1. click *save* 65 | 1. click *Events* 66 | 1. click *any row* 67 | 1. click *back* 68 | 69 | ### :8083, :8086 influxdb 70 | 71 | 1. http://10.242.11.XX0:8086/query?q=SHOW+DATABASES 72 | 1. http://10.242.11.XX0:8086/query?q=SHOW+MEASUREMENTS&db=telegraf 73 | 1. http://10.242.11.XX0:8086/query?q=SHOW+TAG+VALUES+FROM+%22cpu%22+WITH+KEY+%3D+%22host%22&db=telegraf 74 | 75 | ``` 76 | $ curl -XGET 'http://somehostname:8086/query?db=mydb' --data-urlencode 'q=SHOW MEASUREMENTS' 77 | ``` 78 | 79 | > open in browser http://10.242.11.XX0:8083/ 80 | 81 | 82 | 1. set current database to *telegraf* 83 | 1. click *Query Templates* 84 | 1. select *SHOW DATABASES* 85 | 1. loop around ... 86 | 87 | 88 | 89 | ### :5601 Kibana4 90 | 91 | > open in browser http://10.242.11.XX0:5601 92 | 93 | 94 | 1. set *Index name or pattern* 95 | 1. set *Time-field name* 96 | 1. click *Discover* tab on top left 97 | 98 | ### :3000 Grafana 99 | 100 | > open in browser http://10.242.11.XX0:3000/ 101 | 102 | #### add first datasource 103 | 104 | 1. click *Data Sources* (http://10.242.11.XX0:3000/datasources) 105 | 1. click *Add new* (http://10.242.11.XX0:3000/datasources/new) 106 | 1. set *Name* to *telemetry* 107 | 1. tick *default* 108 | 1. set *Type* to *InfluxDB 0.9.x* 109 | 1. set *Url* to *http://10.242.11.XX0:8086* 110 | 1. set *Database* to *telegraf* 111 | 1. set *User* and *Password* to *admin* 112 | 1. click *Test Connection* 113 | 1. click *Save* 114 | 115 | #### create first dashboard and graph 116 | 117 | 1. click *Dashboards* 118 | 1. click *Home* 119 | 1. click *New* 120 | 1. click *small green box on top left* 121 | 1. choose *add panel* - *graph* 122 | 1. set *From* to *cpu* 123 | 1. set *Where* to *cpu* = *cpu-total* 124 | 1. set *Select* to *usage_idle* 125 | 1. set *group by* to *host* 126 | 1. click *floppy image* on top 127 | 1. click *Panel Title* 128 | 1. select *edit* 129 | 1. play around ... 130 | 1. click *floppy image* on top 131 | 132 | ### :8000 Scirius 133 | 134 | 1. open in browser http://10.242.11.XX0:8000/ https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/scripts/install-scirius.sh#L62 135 | 1. user *admin* password *admin* https://github.com/hillar/CDMCS/blob/master/suricata/day_2/boxes/scripts/install-scirius.sh#L61 136 | 1. click on *elasticsearch* 137 | 1. look around ... 138 | 139 | --- 140 | [next : set homenet](/suricata/day_2/homenet.md) 141 | -------------------------------------------------------------------------------- /suricata/day_2/pushrules.md: -------------------------------------------------------------------------------- 1 | # push new rulesets 2 | 3 | * ? push or pull ;) 4 | -------------------------------------------------------------------------------- /suricata/day_2/suriN.dot: -------------------------------------------------------------------------------- 1 | digraph G { 2 | 3 | {rank=min gigamon} 4 | {rank=max admin} 5 | 6 | gigamon [label="traffic mirror"] 7 | suri1 [label="suricata 1"] 8 | suri2 [label="suricata .."] 9 | suri3 [label="suricata N"] 10 | ela1 [label="elasticsearch 1"] 11 | ela2 [label="elasticsearch .."] 12 | ela3 [label="elasticsearch N"] 13 | admin [label=" salt + scirius + evebox + kibana + grafana + influxdb"] 14 | 15 | # data 16 | gigamon -> {suri1 suri2 suri3} -> {ela1 ela2 ela3} -> admin [color="red"] 17 | # master 18 | admin -> {suri1 suri2 suri3 ela1 ela2 ela3} [style="dotted", color="green" ] 19 | # metrix 20 | {suri1 suri2 suri3 ela1 ela2 ela3} -> admin [style="dotted", color="blue" ] 21 | 22 | } 23 | -------------------------------------------------------------------------------- /suricata/day_2/suriN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hillar/CDMCS/a524559199258beb0cd7f6ec57e909ed3dfb2813/suricata/day_2/suriN.png -------------------------------------------------------------------------------- /suricata/day_3/DumpCounters.md: -------------------------------------------------------------------------------- 1 | # internal counters 2 | 3 | see https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Interacting_via_Unix_Socket 4 | 5 | ! https://github.com/inliniac/suricata/blob/fc7f090cd3268a3a624177cef32a9576b74975e3/src/counters.c#L758 6 | 7 | https://gist.github.com/hillar/309e93d5b555095d07b9 8 | 9 | https://github.com/influxdb/telegraf 10 | 11 | https://github.com/influxdb/influxdb 12 | 13 | https://github.com/grafana/grafana 14 | -------------------------------------------------------------------------------- /suricata/day_3/Eve2Pcap.md: -------------------------------------------------------------------------------- 1 | # Convert eve log to pcap 2 | 3 | see 4 | * http://idstools.readthedocs.org/en/latest/tools/eve2pcap.html 5 | * https://github.com/jasonish/py-idstools/blob/master/idstools/scripts/eve2pcap.py 6 | * https://github.com/Maxtors/evepcapparser 7 | 8 | ``` 9 | pip install idstools 10 | ``` 11 | -------------------------------------------------------------------------------- /suricata/day_3/EvilBit.md: -------------------------------------------------------------------------------- 1 | # Evil bit 2 | 3 | see 4 | 5 | * https://www.ietf.org/rfc/rfc3514.txt 6 | * https://github.com/regit/luaevilbit 7 | 8 | 9 | 10 | 11 | #### 0x0 12 | 13 | If the bit is set to 0, the packet has no evil intent. Hosts, 14 | network elements, etc., SHOULD assume that the packet is 15 | harmless, and SHOULD NOT take any defensive measures. (We note 16 | that this part of the spec is already implemented by many common 17 | desktop operating systems.) 18 | 19 | #### 0x1 20 | 21 | If the bit is set to 1, the packet has evil intent. Secure 22 | systems SHOULD try to defend themselves against such packets. 23 | Insecure systems MAY chose to crash, be penetrated, etc. 24 | -------------------------------------------------------------------------------- /suricata/day_3/LoadPcaps.md: -------------------------------------------------------------------------------- 1 | # load pcaps 2 | 3 | see 4 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Interacting_via_Unix_Socket 5 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Multi_Tenancy 6 | * https://home.regit.org/2012/09/a-new-unix-command-mode-in-suricata/ 7 | * https://www.networktotal.com 8 | 9 | ## Configuration 10 | 11 | ``` 12 | grep 'unix-command' -B5 -A2 /etc/suricata/suricata.yaml 13 | ``` 14 | 15 | ``` 16 | suricata --help | grep unix 17 | ``` 18 | 19 | ## For loop is easy? 20 | 21 | ``` 22 | for pcap in `find /pcapdir -type f -name '*.pcap'` ; do 23 | echo "I am doing stuff with $pcap" 24 | done 25 | ``` 26 | 27 | ## Using existing tool to interact with socket 28 | 29 | ``` 30 | suricatasc --help 31 | ``` -------------------------------------------------------------------------------- /suricata/day_3/LuaOutput.md: -------------------------------------------------------------------------------- 1 | # LUA output 2 | 3 | see https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Lua_Output 4 | 5 | see (old) https://gist.github.com/hillar/aeae0b6d12de4ccd8ced#file-suricata_flow2ela-lua 6 | 7 | ## build with LuaJit 8 | 9 | 10 | 11 | ## enable in config 12 | 13 | ``` yaml 14 | outputs: 15 | - lua: 16 | enabled: yes 17 | scripts-dir: /etc/suricata/lua-output/ 18 | scripts: 19 | - tcp-data.lua 20 | - flow.lua 21 | 22 | ``` 23 | ## needs["type", "protocol"] 24 | 25 | ## outputs to .. 26 | 27 | * file 28 | * http 29 | * ... 30 | -------------------------------------------------------------------------------- /suricata/day_3/LuaRules.md: -------------------------------------------------------------------------------- 1 | # LUA scripts in the rules 2 | 3 | see 4 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Lua_scripting 5 | 6 | * http://blog.inliniac.net/2013/04/18/suricata-lua-scripting-flowvar-access/ 7 | 8 | * http://blog.inliniac.net/2013/04/22/suricata-lua-scripting-flowint-access/ 9 | 10 | * https://github.com/EmergingThreats/et-luajit-scripts 11 | 12 | ## build with LuaJit 13 | 14 | ## Config 15 | 16 | ## Init 17 | 18 | ## Match 19 | -------------------------------------------------------------------------------- /suricata/day_3/MultiTenancy.md: -------------------------------------------------------------------------------- 1 | # Multi Tenancy 2 | 3 | Multi tenancy support allows for different rule sets with different rule vars. 4 | 5 | 6 | 7 | see http://jasonish-suricata.readthedocs.org/en/latest/configuration/multi-tenant.html 8 | 9 | and also 10 | * https://github.com/inliniac/suricata/blob/37a64bdd45936875598925f7ccde420e4efdc450/src/unix-manager.c#L880 11 | * https://github.com/inliniac/suricata/blob/37a64bdd45936875598925f7ccde420e4efdc450/src/runmode-unix-socket.c#L440 12 | 13 | 14 | 15 | 16 | ## configuration 17 | 18 | ``` yaml 19 | multi-detect: 20 | enabled: yes 21 | #selector: direct # direct or vlan 22 | selector: vlan 23 | loaders: 3 24 | 25 | tenants: 26 | - id: 1 27 | yaml: tenant-1.yaml 28 | - id: 2 29 | yaml: tenant-2.yaml 30 | - id: 3 31 | yaml: tenant-3.yaml 32 | 33 | mappings: 34 | - vlan-id: 1000 35 | tenant-id: 1 36 | - vlan-id: 2000 37 | tenant-id: 2 38 | - vlan-id: 1112 39 | tenant-id: 3 40 | 41 | 42 | ``` 43 | 44 | ## Unix Socket 45 | 46 | ``` 47 | register-tenant-handler 1 vlan 1000 48 | 49 | unregister-tenant-handler 4 vlan 1111 50 | 51 | ``` 52 | -------------------------------------------------------------------------------- /suricata/day_3/NICAccess.md: -------------------------------------------------------------------------------- 1 | # NIC Access 2 | 3 | see 4 | 5 | * http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source_4.html 6 | -------------------------------------------------------------------------------- /suricata/day_3/pcap-feeder: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | from __future__ import print_function 4 | import os 5 | from suricatasc import * 6 | 7 | 8 | SOCKET_PATH = "/home/eric/builds/suricata/var/run/suricata/suricata-command.socket" 9 | 10 | sc = SuricataSC(SOCKET_PATH, verbose=False) 11 | 12 | sc.connect() 13 | 14 | DIRECTORY = '/home/eric/git/oisf/benches/' 15 | for pcap in os.listdir(DIRECTORY): 16 | if os.path.isfile(os.path.join(DIRECTORY,pcap)) and pcap.endswith(".pcap"): 17 | print(pcap) 18 | outputdir = os.path.join(DIRECTORY, pcap + '.log') 19 | try: 20 | os.mkdir(outputdir) 21 | except OSError: 22 | os.unlink(os.path.join(outputdir, "eve.json")) 23 | pass 24 | (command, arguments) = sc.parse_command("pcap-file %s %s" % (os.path.join(DIRECTORY, pcap), outputdir)); 25 | res = sc.send_command(command, arguments) 26 | print(json.dumps(res)) 27 | 28 | sc.close() 29 | 30 | -------------------------------------------------------------------------------- /suricata/day_intro/Amsterdam.md: -------------------------------------------------------------------------------- 1 | # Amsterdam 2 | 3 | Docker based Suricata, ELK, Scirius, Evebox 4 | 5 | see https://github.com/StamusNetworks/Amsterdam 6 | 7 | see also https://github.com/hillar/CDMCS/blob/master/suricata/day_intro/SSEE-easybutton-singlehost.sh 8 | 9 | ---- 10 | 11 | [next: install](/suricata/day_intro/AptGetInstall.md) 12 | -------------------------------------------------------------------------------- /suricata/day_intro/AppProtos.md: -------------------------------------------------------------------------------- 1 | # supported app layer protocols 2 | 3 | ``` 4 | root@secx:~# suricata --list-app-layer-protos 5 | =========Supported App Layer Protocols========= 6 | http 7 | ftp 8 | smtp 9 | tls 10 | ssh 11 | imap 12 | msn 13 | smb 14 | dcerpc 15 | dns 16 | ``` 17 | 18 | ---- 19 | 20 | [next: keywords](/suricata/day_intro/keywords.md) 21 | 22 | [jump to: configuration](/suricata/day_intro/BasicConf.md) 23 | 24 | -------------------------------------------------------------------------------- /suricata/day_intro/AptGetInstall.md: -------------------------------------------------------------------------------- 1 | # apt-get install suricata 2 | 3 | 4 | ### system information 5 | 6 | ``` 7 | root@secx:~# uname -a 8 | Linux secx 3.19.0-30-generic #34~14.04.1-Ubuntu SMP Fri Oct 2 22:09:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux 9 | ``` 10 | 11 | ### install 12 | 13 | ``` 14 | root@secx:~# apt-get install suricata 15 | Reading package lists... Done 16 | Building dependency tree 17 | Reading state information... Done 18 | The following extra packages will be installed: 19 | libauthen-sasl-perl libencode-locale-perl libfile-listing-perl 20 | 21 | ... 22 | 23 | Setting up suricata (1.4.7-1ubuntu1.1) ... 24 | * suricata disabled, please adjust the configuration to your needs 25 | * and then set RUN to 'yes' in /etc/default/suricata to enable it. 26 | Setting up libwww-perl (6.05-2) ... 27 | Setting up oinkmaster (2.0-4) ... 28 | Processing triggers for libc-bin (2.19-0ubuntu6.6) ... 29 | Processing triggers for ureadahead (0.100.0-16) ... 30 | ``` 31 | 32 | ### version info 33 | 34 | ``` 35 | root@secx:~# suricata -V 36 | This is Suricata version 1.4.7 RELEASE 37 | ``` 38 | 39 | ### build info 40 | 41 | ``` 42 | root@secx:~# suricata --build-info 43 | This is Suricata version 1.4.7 RELEASE 44 | Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW PCRE_JIT HAVE_NSS HAVE_LUAJIT HAVE_LIBJANSSON 45 | 64-bits, Little-endian architecture 46 | GCC version 4.8.2, C version 199901 47 | __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 48 | __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 49 | __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 50 | __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 51 | compiled with -fstack-protector 52 | compiled with _FORTIFY_SOURCE=2 53 | compiled with libhtp 0.2.14, linked against 0.2.14 54 | Suricata Configuration: 55 | AF_PACKET support: yes 56 | PF_RING support: no 57 | NFQueue support: yes 58 | IPFW support: no 59 | DAG enabled: no 60 | Napatech enabled: no 61 | Unix socket enabled: yes 62 | 63 | libnss support: yes 64 | libnspr support: yes 65 | libjansson support: yes 66 | Prelude support: yes 67 | PCRE jit: yes 68 | libluajit: yes 69 | libgeoip: no 70 | Non-bundled htp: yes 71 | Old barnyard2 support: no 72 | CUDA enabled: no 73 | 74 | Suricatasc install: yes 75 | 76 | Unit tests enabled: no 77 | Debug output enabled: no 78 | Debug validation enabled: no 79 | Profiling enabled: no 80 | Profiling locks enabled: no 81 | 82 | Generic build parameters: 83 | Installation prefix (--prefix): /usr 84 | Configuration directory (--sysconfdir): /etc/suricata/ 85 | Log directory (--localstatedir) : /var/log/suricata/ 86 | 87 | Host: x86_64-pc-linux-gnu 88 | GCC binary: gcc 89 | GCC Protect enabled: yes 90 | GCC march native enabled: no 91 | GCC Profile enabled: no 92 | 93 | ``` 94 | 95 | ---- 96 | 97 | [next: add PPA](/suricata/day_intro/AptGetInstallPPA.md) 98 | -------------------------------------------------------------------------------- /suricata/day_intro/BasicConf.md: -------------------------------------------------------------------------------- 1 | # configuration 2 | 3 | see 4 | * http://jasonish-suricata.readthedocs.org/en/latest/configuration/suricata-yaml.html 5 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricatayaml 6 | 7 | ## Yaml 8 | 9 | Suricata uses the Yaml format for configuration. 10 | 11 | see http://www.yaml.org 12 | 13 | ### specify configuration file location 14 | 15 | ``` 16 | suricata -c /some/directory/suricata.yaml 17 | ``` 18 | 19 | ### test configuration file 20 | 21 | ``` 22 | suricata -c /etc/suricata/suricata.yaml -T 23 | #or 24 | suricata -c /etc/suricata/suricata.yaml -T -v 25 | ``` 26 | > notice the missing threshold.config 27 | > see http://permalink.gmane.org/gmane.comp.security.ids.oisf.user/1738 28 | 29 | ``` 30 | root@secx:~# touch /etc/suricata/threshold.config 31 | root@secx:~# /usr/bin/suricata -c /etc/suricata/suricata.yaml -T 32 | 6/1/2016 -- 14:25:47 - - Running suricata under test mode 33 | 6/1/2016 -- 14:25:47 - - This is Suricata version 2.0.11 RELEASE 34 | ``` 35 | 36 | ## HOME_NET 37 | 38 | As a bare minimum, you should define your home network: 39 | ``` 40 | root@secx:~# grep HOME_NET /etc/suricata/suricata.yaml 41 | HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 42 | EXTERNAL_NET: "!$HOME_NET" 43 | HTTP_SERVERS: "$HOME_NET" 44 | SMTP_SERVERS: "$HOME_NET" 45 | SQL_SERVERS: "$HOME_NET" 46 | DNS_SERVERS: "$HOME_NET" 47 | TELNET_SERVERS: "$HOME_NET" 48 | DNP3_SERVER: "$HOME_NET" 49 | DNP3_CLIENT: "$HOME_NET" 50 | MODBUS_CLIENT: "$HOME_NET" 51 | MODBUS_SERVER: "$HOME_NET" 52 | ENIP_CLIENT: "$HOME_NET" 53 | ENIP_SERVER: "$HOME_NET" 54 | ``` 55 | 56 | Review default ports according to your environments: 57 | ``` 58 | root@secx:~# grep '_PORTS' /etc/suricata/suricata.yaml 59 | HTTP_PORTS: "80" 60 | SHELLCODE_PORTS: "!80" 61 | ORACLE_PORTS: 1521 62 | SSH_PORTS: 22 63 | DNP3_PORTS: 20000 64 | MODBUS_PORTS: 502 65 | ``` 66 | 67 | ## Rules 68 | 69 | ``` 70 | root@secx:~# grep -A51 rule-path /etc/suricata/suricata.yaml 71 | default-rule-path: /etc/suricata/rules 72 | rule-files: 73 | - botcc.rules 74 | - ciarmy.rules 75 | - compromised.rules 76 | - drop.rules 77 | - dshield.rules 78 | - emerging-activex.rules 79 | - emerging-attack_response.rules 80 | - emerging-chat.rules 81 | - emerging-current_events.rules 82 | - emerging-dns.rules 83 | - emerging-dos.rules 84 | - emerging-exploit.rules 85 | - emerging-ftp.rules 86 | - emerging-games.rules 87 | - emerging-icmp_info.rules 88 | # - emerging-icmp.rules 89 | - emerging-imap.rules 90 | - emerging-inappropriate.rules 91 | - emerging-malware.rules 92 | - emerging-misc.rules 93 | - emerging-mobile_malware.rules 94 | - emerging-netbios.rules 95 | - emerging-p2p.rules 96 | - emerging-policy.rules 97 | - emerging-pop3.rules 98 | - emerging-rpc.rules 99 | - emerging-scada.rules 100 | - emerging-scan.rules 101 | - emerging-shellcode.rules 102 | - emerging-smtp.rules 103 | - emerging-snmp.rules 104 | - emerging-sql.rules 105 | - emerging-telnet.rules 106 | - emerging-tftp.rules 107 | - emerging-trojan.rules 108 | - emerging-user_agents.rules 109 | - emerging-voip.rules 110 | - emerging-web_client.rules 111 | - emerging-web_server.rules 112 | - emerging-web_specific_apps.rules 113 | - emerging-worm.rules 114 | - tor.rules 115 | - decoder-events.rules # available in suricata sources under rules dir 116 | - stream-events.rules # available in suricata sources under rules dir 117 | - http-events.rules # available in suricata sources under rules dir 118 | - smtp-events.rules # available in suricata sources under rules dir 119 | - dns-events.rules # available in suricata sources under rules dir 120 | - tls-events.rules # available in suricata sources under rules dir 121 | # - modbus-events.rules # available in suricata sources under rules dir 122 | - app-layer-events.rules # available in suricata sources under rules dir 123 | ``` 124 | 125 | 126 | 127 | ### do you need rules !? 128 | 129 | ``` 130 | root@secx:~# suricata -c /etc/suricata/suricata.yaml --disable-detection 131 | 6/1/2016 -- 14:47:28 - - detection engine disabled 132 | 6/1/2016 -- 14:47:28 - - This is Suricata version 2.0.11 RELEASE 133 | ``` 134 | 135 | How could this mode prove useful? 136 | 137 | 138 | ## logs 139 | 140 | ``` 141 | root@secx:~# grep log-dir /etc/suricata/suricata.yaml 142 | default-log-dir: /var/log/suricata/ 143 | ... 144 | ``` 145 | 146 | More of logging in the next chapter 147 | 148 | ---- 149 | 150 | [next : basic logging](/suricata/day_intro/BasicLogging.md) 151 | -------------------------------------------------------------------------------- /suricata/day_intro/BasicLogging.md: -------------------------------------------------------------------------------- 1 | # logging 2 | 3 | * http://jasonish-suricata.readthedocs.org/en/latest/output/index.html 4 | * [Files in log directory](/suricata/day_intro/log.md) 5 | 6 | ## file 7 | 8 | Line-based logs are easily human readable but contain limited information. Furthermore, machine processing can be expensive in terms of CPU and IO. 9 | 10 | ``` 11 | grep '\- fast' -A4 -B1 /etc/suricata/suricata.yaml 12 | tail /var/log/suricata/fast.log 13 | ``` 14 | 15 | ``` 16 | 12/07/2015-19:30:03.307554 [**] [1:2200074:1] SURICATA TCPv4 invalid checksum [**] [Classification: (null)] [Priority: 3] {TCP} 192.168.11.11:42336 -> 192.168.12.12:80 17 | ``` 18 | 19 | ### unified2 aka Barnyard2 20 | 21 | Binary format for disk efficiency. Thus, IDS spends minimal resources for storing data. Further processing is offloaded to other tools. 22 | 23 | * https://github.com/firnsy/barnyard2 24 | * https://github.com/jasonish/py-idstools 25 | * https://github.com/threatstack/unified2 26 | * http://threatstack.github.io/pigsty/ 27 | 28 | ### EVE (Extensible Event Format) 29 | 30 | Starting in 2.0, Suricata can output alerts, connections and internal statistics through JSON. 31 | 32 | * http://www.json.org/ 33 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/EveJSONFormat 34 | 35 | ``` 36 | { 37 | "timestamp": "2015-12-07T19:29:55.093707+0000", 38 | "flow_id": 1.4000333881421e+14, 39 | "event_type": "http", 40 | "src_ip": "192.168.11.11", 41 | "src_port": 42334, 42 | "dest_ip": "192.168.12.12", 43 | "dest_port": 80, 44 | "proto": "TCP", 45 | "tx_id": 0, 46 | "http": { 47 | "hostname": "192.168.12.12", 48 | "url": "\/index.html?crap=1449516595", 49 | "http_content_type": "text\/html", 50 | "http_method": "GET", 51 | "protocol": "HTTP\/1.1", 52 | "status": 200, 53 | "length": 1 54 | } 55 | } 56 | ``` 57 | 58 | ``` 59 | grep 'eve-log' -A10 /etc/suricata/suricata.yaml 60 | ``` 61 | 62 | Following event types can be enabled in Suricata 3. 63 | 64 | * Alert 65 | * HTTP 66 | * DNS 67 | * TLS 68 | * Files 69 | * drop 70 | * SMTP 71 | * SSH 72 | * Stats 73 | * Flow 74 | * Netflow 75 | 76 | ``` 77 | cat /var/log/suricata/eve.json | jq -c 'select(.event_type=="alert")' 78 | ``` 79 | 80 | ## syslog 81 | 82 | Suricata can alert via syslog. 83 | 84 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Syslog_Alerting_Compatibility 85 | * https://www.ietf.org/rfc/rfc3164.txt 86 | * https://tools.ietf.org/html/rfc5424 87 | * http://www.rsyslog.com/doc/master/index.html 88 | * https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html-single/index.html 89 | * http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html 90 | 91 | ## stuff 92 | 93 | ``` 94 | root@secx:/usr/local/var/log/suricata# cat eve.json | perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c 95 | 1060 alert 96 | 29841 dns 97 | 12351 fileinfo 98 | 12438 http 99 | 8219 ssh 100 | 44329 stats 101 | ``` 102 | 103 | Or with jq: 104 | 105 | ``` 106 | cat eve.json | jq -c '.event_type' | sort | uniq -c 107 | ``` 108 | 109 | ---- 110 | 111 | [next: view events](/suricata/day_intro/EveView.md) 112 | -------------------------------------------------------------------------------- /suricata/day_intro/EveView.md: -------------------------------------------------------------------------------- 1 | # View events 2 | 3 | see 4 | 5 | * https://github.com/jasonish/evebox 6 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_logstash_kibana_and_suricata_json_output 7 | * https://github.com/pevma/Suricata-Logstash-Templates 8 | 9 | * https://github.com/Snorby/snorby/ 10 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/suricata_snorby_and_barnyard2_set_up_guide 11 | -------------------------------------------------------------------------------- /suricata/day_intro/README.md: -------------------------------------------------------------------------------- 1 | # Suricata 2 | 3 | ![logo](https://idsips.files.wordpress.com/2012/09/suricata.png) 4 | 5 | ## http://suricata-ids.org/ 6 | 7 | > Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. 8 | 9 | > Suricata implements a complete signature language to match on known threats, policy violations and malicious behaviour. Suricata will also detect many anomalies in the traffic it inspects. 10 | 11 | > Suricata will automatically detect protocols such as HTTP on any port and apply the proper detection and logging logic. Suricata can log HTTP requests, log and store TLS certificates, extract files from flows and store them to disk. 12 | 13 | #### please see 14 | 15 | * http://suricata-ids.org/ 16 | * http://planet.suricata-ids.org/ 17 | * http://www.openinfosecfoundation.org/ 18 | * https://github.com/inliniac/suricata 19 | * http://jasonish-suricata.readthedocs.org/en/latest/index.html 20 | 21 | 22 | #### please remember 23 | 24 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams 25 | 26 | ---- 27 | 28 | [next: all the nice/graphical thing for start](/suricata/day_intro/Amsterdam.md) 29 | -------------------------------------------------------------------------------- /suricata/day_intro/SimpleRule.md: -------------------------------------------------------------------------------- 1 | # Writing Rule 2 | 3 | > Do not write rules, buy from professionals ! 4 | 5 | see https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricata_Rules 6 | 7 | ``` 8 | alert tcp any any -> any 443 (msg:"SURICATA Port 443 but not SSL/TLS"; app-layer-protocol:!tls; threshold: type limit, track by_src, seconds 180, count 1; classtype:bad-unknown; sid:990001;) 9 | ``` 10 | 11 | A rule consists of the following: 12 | * action 13 | * header 14 | * rule-options 15 | 16 | ## Action 17 | 18 | 19 | * alert - This is the action we want to perform on the rule 20 | 21 | * pass - This can be compared to “ACCEPT” in iptables, in that if the packet matches this rule it’ll be accepted through. 22 | * drop - The packet doesn’t get processed any further down the chain and the sender isn’t notified. This is akin to the “DROP” target in iptables, where it will silently remove the packet from the network stack. 23 | * reject - This acts the same as drop but will also notify the sender that the packet has been removed from the stack. 24 | 25 | ## header 26 | 27 | * First keyword: protocol with protocol recognition 28 | * Second part: IP params includin variable 29 | 30 | ## Rule options 31 | 32 | * content matching 33 | * meta data 34 | * threshold configuration 35 | 36 | ## Write some simple signature 37 | 38 | ### Amsterdam 39 | 40 | Config file are in ee/config directory. To update suricata config file, you have 41 | to edit ee/config/suricata/suricata.yaml and add a reference to custom.rules: 42 | 43 | ```YAML 44 | default-rule-path: /etc/suricata/rules 45 | rule-files: 46 | - scirius.rules 47 | - custom.rules 48 | ``` 49 | 50 | Restart amsterdam. 51 | 52 | You can also edit the file from the container by changing their right to read write. 53 | 54 | To do so, edit the ee/docker-compose.yml and change suricata.yaml and threshold.config to rw: 55 | 56 | ```YAML 57 | suricata: 58 | build: /home/eric/builds/amsterdam/ee/docker/suricata 59 | volumes: 60 | - /home/eric/builds/amsterdam/ee/config/suricata/suricata.yaml:/etc/suricata/suricata.yaml:rw 61 | - /home/eric/builds/amsterdam/ee/config/suricata/threshold.config:/etc/suricata/threshold.config:rw 62 | ``` 63 | 64 | Then create a shell in suricata container: 65 | 66 | ```bash 67 | docker exec -ti ee_suricata_1 bash 68 | ``` 69 | 70 | Once in the shell, you can install vim 71 | 72 | ```bash 73 | apt-get install vim 74 | vim /etc/suricata/rules/custom.rules 75 | ``` 76 | 77 | ### Exercises 78 | 79 | * Write rules on query to a specific website http_hostname 80 | * Check some text in the content of the response 81 | * Write a rules checking TLS fingerprint of a HTTPS website 82 | * Alert on all PNG files seen on HTTP and SMTP 83 | -------------------------------------------------------------------------------- /suricata/day_intro/ams_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | apt-get install docker.io 4 | pip install docker-compose amsterdam 5 | export COMPOSE_API_VERSION=1.18 6 | amsterdam -d ee -i eth0 setup 7 | amsterdam -d ee start 8 | -------------------------------------------------------------------------------- /suricata/day_intro/amstelredamme/README.md: -------------------------------------------------------------------------------- 1 | #amstelredamme 2 | 3 | 4 | >Q: why ? 5 | 6 | >A: beacuse to build a complete single-machine SuricataSciriusEvebox system. This is good for a demo and can also be used as a starting point for a real production deployment. 7 | 8 | see 9 | 10 | * https://github.com/hillar/CDMCS/blob/master/suricata/day_intro/Amsterdam.md 11 | * https://github.com/hillar/CDMCS/blob/master/suricata/day_intro/SSEE-easybutton-singlehost.sh 12 | -------------------------------------------------------------------------------- /suricata/day_intro/amstelredamme/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.11" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Amstelredamme do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'amstelredamme' 10 | box.vm.provision :shell, :path => "../SSEE-easybutton-singlehost.sh" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "4"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /suricata/day_intro/dummy/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.111" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Dummy do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Dummy' 10 | box.vm.provision :shell, :path => "./dummy-install-suricata.sh", args: "eth1" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /suricata/day_intro/dummy/dummy-install-suricata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # this script 4 | # 0) sets gro,gso,lro,tso off 5 | # 1) addp oisf ppa 6 | # 2) installs suricata 7 | # 3) sets suricata conf as amsterdam 8 | # 4) 9 | 10 | 11 | if [ "$(id -u)" != "0" ]; then 12 | echo "ERROR - This script must be run as root" 1>&2 13 | exit 1 14 | fi 15 | 16 | IP=$(ifconfig eth0 2>/dev/null|grep 'inet addr'|cut -f2 -d':'|cut -f1 -d' ') 17 | HOSTNAME=$(hostname -f) 18 | 19 | ETH=$1 20 | echo "installing suricata on $IP $HOSTNAME setting * off on $ETH" 21 | ethtool -K $ETH tx off sg off gro off gso off lro off tso off 22 | 23 | #suricata 24 | add-apt-repository -y ppa:oisf/suricata-stable 25 | apt-get update 26 | apt-get -y install suricata 27 | service suricata stop 28 | #stealing amsterdam suricata conf 29 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/suricata.yaml -O /etc/suricata/suricata.yaml 30 | wget -q https://raw.githubusercontent.com/StamusNetworks/Amsterdam/master/src/config/suricata/threshold.config -O /etc/suricata/threshold.config 31 | 32 | # - interface: eth0 33 | sed -i -e 's,- interface: eth0,- interface: '${ETH}',g' /etc/suricata/suricata.yaml 34 | #fake scirius rules 35 | #todo: get it from master 36 | touch /etc/suricata/rules/scirius.rules 37 | service suricata start 38 | -------------------------------------------------------------------------------- /suricata/day_intro/empty/Vagrantfile: -------------------------------------------------------------------------------- 1 | IP = "192.168.11.110" 2 | vm_box = 'ubu14' 3 | 4 | Vagrant.configure('2') do |config| 5 | config.vm.define :Empty do |box| 6 | box.vm.box = vm_box 7 | box.vm.synced_folder ".", "/vagrant", disabled: true 8 | box.vm.network :private_network, ip: "#{IP}" 9 | box.vm.hostname = 'Empty' 10 | box.vm.provision "shell", inline: "ifconfig; ping -c 1 www.ee; netstat -ntple" 11 | box.vm.provider :virtualbox do |vb| 12 | vb.customize ["modifyvm", :id, "--memory", "4096"] 13 | vb.customize ["modifyvm", :id, "--cpus", "2"] 14 | end 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /suricata/day_intro/etc.md: -------------------------------------------------------------------------------- 1 | ## what is under /etc 2 | 3 | ``` 4 | root@secx:~# tree /etc/suricata/ 5 | /etc/suricata/ 6 | |-- classification.config 7 | |-- reference.config 8 | |-- rules 9 | | |-- botcc.portgrouped.rules 10 | | |-- botcc.rules 11 | | |-- BSD-License.txt 12 | | |-- ciarmy.rules 13 | | |-- classification.config 14 | | |-- compromised-ips.txt 15 | | |-- compromised.rules 16 | | |-- decoder-events.rules 17 | | |-- dns-events.rules 18 | | |-- drop.rules 19 | | |-- dshield.rules 20 | | |-- emerging-activex.rules 21 | | |-- emerging-attack_response.rules 22 | | |-- emerging-chat.rules 23 | | |-- emerging.conf 24 | | |-- emerging-current_events.rules 25 | | |-- emerging-deleted.rules 26 | | |-- emerging-dns.rules 27 | | |-- emerging-dos.rules 28 | | |-- emerging-exploit.rules 29 | | |-- emerging-ftp.rules 30 | | |-- emerging-games.rules 31 | | |-- emerging-icmp_info.rules 32 | | |-- emerging-icmp.rules 33 | | |-- emerging-imap.rules 34 | | |-- emerging-inappropriate.rules 35 | | |-- emerging-info.rules 36 | | |-- emerging-malware.rules 37 | | |-- emerging-misc.rules 38 | | |-- emerging-mobile_malware.rules 39 | | |-- emerging-netbios.rules 40 | | |-- emerging-p2p.rules 41 | | |-- emerging-policy.rules 42 | | |-- emerging-pop3.rules 43 | | |-- emerging-rpc.rules 44 | | |-- emerging-scada.rules 45 | | |-- emerging-scan.rules 46 | | |-- emerging-shellcode.rules 47 | | |-- emerging-smtp.rules 48 | | |-- emerging-snmp.rules 49 | | |-- emerging-sql.rules 50 | | |-- emerging-telnet.rules 51 | | |-- emerging-tftp.rules 52 | | |-- emerging-trojan.rules 53 | | |-- emerging-user_agents.rules 54 | | |-- emerging-voip.rules 55 | | |-- emerging-web_client.rules 56 | | |-- emerging-web_server.rules 57 | | |-- emerging-web_specific_apps.rules 58 | | |-- emerging-worm.rules 59 | | |-- files.rules 60 | | |-- gen-msg.map 61 | | |-- gpl-2.0.txt 62 | | |-- http-events.rules 63 | | |-- LICENSE 64 | | |-- rbn-malvertisers.rules 65 | | |-- rbn.rules 66 | | |-- reference.config 67 | | |-- sid-msg.map 68 | | |-- smtp-events.rules 69 | | |-- stream-events.rules 70 | | |-- suricata-1.2-prior-open.yaml 71 | | |-- suricata-open.txt 72 | | |-- tls-events.rules 73 | | |-- tor.rules 74 | | `-- unicode.map 75 | `-- suricata.yaml 76 | 77 | 1 directory, 69 files 78 | ``` 79 | 80 | ---- 81 | 82 | [next: /var/log/suricata/*](/suricata/day_intro/log.md) 83 | 84 | [jump to: configuration](/suricata/day_intro/BasicConf.md) 85 | -------------------------------------------------------------------------------- /suricata/day_intro/help.md: -------------------------------------------------------------------------------- 1 | ## help 2 | 3 | ``` 4 | root@secx:~# suricata -h 5 | Suricata 2.0.11 6 | USAGE: suricata [OPTIONS] [BPF FILTER] 7 | 8 | -c : path to configuration file 9 | -T : test configuration file (use with -c) 10 | -i : run in pcap live mode 11 | -F : bpf filter file 12 | -r : run in pcap file/offline mode 13 | -q : run in inline nfqueue mode 14 | -s : path to signature file loaded in addition to suricata.yaml settings (optional) 15 | -S : path to signature file loaded exclusively (optional) 16 | -l : default log directory 17 | -D : run as daemon 18 | -k [all|none] : force checksum check (all) or disabled it (none) 19 | -V : display Suricata version 20 | -v[v] : increase default Suricata verbosity 21 | --list-app-layer-protos : list supported app layer protocols 22 | --list-keywords[=all|csv|] : list keywords implemented by the engine 23 | --list-runmodes : list supported runmodes 24 | --runmode : specific runmode modification the engine should run. The argument 25 | supplied should be the id for the runmode obtained by running 26 | --list-runmodes 27 | --engine-analysis : print reports on analysis of different sections in the engine and exit. 28 | Please have a look at the conf parameter engine-analysis on what reports 29 | can be printed 30 | --pidfile : write pid to this file 31 | --init-errors-fatal : enable fatal failure on signature init error 32 | --disable-detection : disable detection engine 33 | --dump-config : show the running configuration 34 | --build-info : display build information 35 | --pcap[=] : run in pcap mode, no value select interfaces from suricata.yaml 36 | --pcap-buffer-size : size of the pcap buffer value from 0 - 2147483647 37 | --af-packet[=] : run in af-packet mode, no value select interfaces from suricata.yaml 38 | --user : run suricata as this user after init 39 | --group : run suricata as this group after init 40 | --erf-in : process an ERF file 41 | --unix-socket[=] : use unix socket to control suricata work 42 | --set name=value : set a configuration value 43 | 44 | 45 | To run the engine with default configuration on interface eth0 with signature file "signatures.rules", run the command as: 46 | 47 | suricata -c suricata.yaml -s signatures.rules -i eth0 48 | 49 | ``` 50 | 51 | #### see 52 | 53 | * http://jasonish-suricata.readthedocs.org/en/latest/command-line-options.html 54 | * https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Command_Line_Options 55 | 56 | ---- 57 | 58 | [next: /etc/suricata/*](/suricata/day_intro/etc.md) 59 | 60 | [jump to: configuration](/suricata/day_intro/BasicConf.md) 61 | 62 | -------------------------------------------------------------------------------- /suricata/day_intro/log.md: -------------------------------------------------------------------------------- 1 | ### what was logged 2 | 3 | ``` 4 | root@secx:~# tree /var/log/suricata/ 5 | /var/log/suricata/ 6 | |-- certs 7 | |-- core 8 | |-- eve.json 9 | |-- fast.log 10 | |-- files 11 | |-- http.log 12 | |-- stats.log 13 | |-- suricata-start.log 14 | `-- unified2.alert.1452070536 15 | 16 | 3 directories, 6 files 17 | ``` 18 | ---- 19 | 20 | [next: app layer protocols](/suricata/day_intro/AppProtos.md) 21 | 22 | [jump to: configuration](/suricata/day_intro/BasicConf.md) 23 | 24 | -------------------------------------------------------------------------------- /suricata/day_last/README.md: -------------------------------------------------------------------------------- 1 | 2 | * how did we get here 3 | * how does it all hang together 4 | * how much still remains to be understood 5 | --------------------------------------------------------------------------------