├── Information2Knowledge.md ├── DataFlow.md ├── Data2Info2Knowledge.md ├── ScreenRealEstate.md ├── Data2Information.md ├── .gitignore └── README.md /Information2Knowledge.md: -------------------------------------------------------------------------------- 1 | #Information 2 Knowledge 2 | 3 | .... 4 | -------------------------------------------------------------------------------- /DataFlow.md: -------------------------------------------------------------------------------- 1 | # Data flow 2 | 3 | ``` 4 | host -[metrix]-> influxdb 5 | host -[(sys)log]-> rsyslog 6 | switch -[rspan]-> tcpdump 7 | router -[netflow]-> nfcapd 8 | 9 | ``` 10 | -------------------------------------------------------------------------------- /Data2Info2Knowledge.md: -------------------------------------------------------------------------------- 1 | # how it works ... 2 | 3 | * [DATA Flow](DataFlow.md) 4 | * [DATA -> INFORMATION](Data2Information.md) 5 | * [INFORMATION -> KNOWLEDGE ](Information2Knowledge.md) 6 | * [Display KNOWLEDGE, INFORMATION & DATA](ScreenRealEstate.md) 7 | -------------------------------------------------------------------------------- /ScreenRealEstate.md: -------------------------------------------------------------------------------- 1 | # Screen real estate 2 | 3 | * 4 x HD 4 | * 1 x 'average' projector 5 | 6 | ``` 7 | 8 | alerta -[alert]-> projector(custom_viz) 9 | 10 | kapacitor -[warnings]-> HD(alerta) 11 | influx -[metrix]->HD(grafana) 12 | suricata -[ids alerts]-> HD(evebox?) 13 | .. -[]-> HD(..?) 14 | 15 | ``` 16 | -------------------------------------------------------------------------------- /Data2Information.md: -------------------------------------------------------------------------------- 1 | # Data 2 Information 2 | 3 | 4 | ## parsion log files 5 | 6 | * [telegraf.logparser](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logparser) 7 | 8 | 9 | ``` 10 | 11 | logfileX -> telegraf.logparser -> influxdb 12 | 13 | 14 | ``` 15 | 16 | ## parsing pcap's 17 | 18 | * moloch 19 | * suricata 20 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled Lua sources 2 | luac.out 3 | 4 | # luarocks build files 5 | *.src.rock 6 | *.zip 7 | *.tar.gz 8 | 9 | # Object files 10 | *.o 11 | *.os 12 | *.ko 13 | *.obj 14 | *.elf 15 | 16 | # Precompiled Headers 17 | *.gch 18 | *.pch 19 | 20 | # Libraries 21 | *.lib 22 | *.a 23 | *.la 24 | *.lo 25 | *.def 26 | *.exp 27 | 28 | # Shared objects (inc. Windows DLLs) 29 | *.dll 30 | *.so 31 | *.so.* 32 | *.dylib 33 | 34 | # Executables 35 | *.exe 36 | *.out 37 | *.app 38 | *.i*86 39 | *.x86_64 40 | *.hex 41 | 42 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Frankencoding 2 | 3 | https://twitter.com/hughcards/status/423952995240648704 4 | 5 | You're busted! - and just with Bubble Gum And Baling Wire... ok and some Duct Tape is also used. 6 | 7 | ### About 8 | 9 | This repository houses a list of open-source tools, libraries, projects, etc that can be used to build awesome security stacks. 10 | 11 | ### Contents 12 | 13 | ## Packet capture and intrusion detection 14 | 15 | * [Suricata IDS](https://github.com/OISF/suricata) - Suricata is a network IDS, IPS and NSM engine. 16 | * [Documentation](https://suricata-update.readthedocs.io/en/latest/) 17 | * [CCDCOE course materials](https://github.com/ccdcoe/CDMCS/tree/master/Suricata) 18 | * [Suricata update](https://suricata-update.readthedocs.io/en/latest/) 19 | * [Detect newly created TLS certificates](https://gist.github.com/markuskont/87e545c8b82fa00caf091ed6a2fedc8f) 20 | * [Moloch](https://github.com/aol/moloch) - Moloch is an open source, large scale, full packet capturing, indexing, and database system. 21 | * [Wiki](https://github.com/aol/moloch/wiki) *deprecated* 22 | * [Official documentation](https://molo.ch/learn) 23 | * [API endpoints](https://molo.ch/api) 24 | * [CCDCOE course materials](https://github.com/ccdcoe/CDMCS/tree/master/Moloch), [example WISE plugin](https://github.com/markuskont/moloch/blob/custom/wisePlugins/wiseService/source.ls19.js), [writing a wise plugin](https://github.com/ccdcoe/CDMCS/tree/master/Moloch/wise#writing-a-wise-plugin) 25 | * [Zeek](https://github.com/zeek/zeek) - Zeek is a powerful network analysis framework that is much different from the typical IDS you may know. 26 | * [Documentation](https://www.zeek.org/documentation/index.html) 27 | * [Scripts for detecting cobal strike](https://github.com/ccdcoe/zeeky) 28 | 29 | ### Libraries 30 | * [py-idstools](https://github.com/jasonish/py-idstools) - idstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool) 31 | * [Go NIDS](https://github.com/google/gonids) - gonids is a library to parse IDS rules, with a focus primarily on Suricata rule compatibility. 32 | * [Gopacket](https://github.com/google/gopacket) - Provides packet processing capabilities for Go 33 | 34 | ## Logging 35 | 36 | ### Sources 37 | 38 | * [Snoopy](https://github.com/a2o/snoopy) - Log every executed command to syslog (a.k.a. Snoopy Logger). 39 | * [Improve logging format](https://github.com/ccdcoe/frankenstack/blob/master/states/blue/snoopy/config/config.ini), [normalize snoopy messages using liblognorm](https://github.com/ccdcoe/frankenstack/blob/master/states/yellow/logserver/config-rsyslog/snoopy.rulebase) 40 | * [Sysmon](https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon) - Make Windows logging great again 41 | * [SwiftOnSecurity configs](https://github.com/SwiftOnSecurity/sysmon-config) - Sysmon configuration file template with default high-quality event tracing 42 | * [Sysmon modular](https://github.com/olafhartong/sysmon-modular) - A repository of sysmon configuration modules 43 | 44 | ### Collecting, shipping, stream processing 45 | 46 | * [Rsyslog](https://github.com/rsyslog/rsyslog) - RSYSLOG is the rocket-fast system for log processing. 47 | * [Documentation](https://rsyslog.readthedocs.io/en/latest/index.html) 48 | * [liblognorm](https://github.com/rsyslog/liblognorm) - A fast samples-based log normalization library. 49 | * [Syslog-ng](https://github.com/syslog-ng/syslog-ng) - free and open-source implementation of the syslog protocol for Unix and Unix-like systems. 50 | * [Fever](https://github.com/DCSO/fever) - fast, extensible, versatile event router for Suricata's EVE-JSON format 51 | 52 | ### Correlation 53 | 54 | * [Simple Event Correlator](https://github.com/simple-evcorr/sec) - SEC is an event correlation tool for advanced event processing. 55 | 56 | ## Databases 57 | 58 | ### Search engines 59 | 60 | * [Visibility Across Space and Time (VAST)](https://github.com/tenzir/vast) - is a scalable foundation for a security operations center (SOC): a rich data model for security data, high-throughput ingestion of telemetry, low-latency search, and flexible export in various formats. 61 | 62 | ### Graph 63 | * [Neo4j](https://github.com/neo4j/neo4j) - Neo4j is the world’s leading Graph Database. 64 | 65 | ### SQL 66 | 67 | * [SQLite](https://github.com/sqlite/sqlite) - SQLite is a relational database management system contained in a C library. In contrast to many other database management systems, SQLite is not a client–server database engine. Rather, it is embedded into the end program. 68 | * [MariaDB](https://github.com/MariaDB/server) - MariaDB server is a community developed fork of MySQL server. 69 | * [CockroachDB](https://github.com/cockroachdb/cockroach) - the open source, cloud-native SQL database. 70 | 71 | ### NoSQL 72 | 73 | * [Cassandra](https://github.com/apache/cassandra) - Apache Cassandra is a highly-scalable partitioned row store. 74 | * [Elasticsearch](https://github.com/elastic/elasticsearch) - Open Source, Distributed, RESTful Search Engine. 75 | * [Install elastic with docker](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) 76 | * [InfluxDB](https://github.com/influxdata/influxdb) - Scalable datastore for metrics, events, and real-time analytics. 77 | * [Prometheus](https://github.com/prometheus/prometheus) - The Prometheus monitoring system and time series database. 78 | 79 | ## Web interfaces and API-s 80 | 81 | ### Alerts 82 | 83 | * [Scirius](https://github.com/StamusNetworks/scirius) - Scirius is a web application for Suricata ruleset management. 84 | * [scirius-docker](https://github.com/StamusNetworks/scirius-docker) 85 | * [Evebox](https://github.com/jasonish/evebox) - Web Based Event Viewer (GUI) for Suricata EVE Events in Elastic Search 86 | * [Alerta](https://github.com/alerta/alerta) - Alerta monitoring system 87 | * [Alerta web UI](https://github.com/alerta/alerta-webui) - Alerta Web UI 7.0 88 | 89 | ### Generic Viz 90 | 91 | * [Kibana](https://github.com/elastic/kibana) - Your window into the Elastic Stack 92 | * [Grafana](https://github.com/grafana/grafana) - The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More 93 | 94 | ## Programming languages and dev tools 95 | 96 | * [Rust](https://rustup.rs/) - Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — and enable you to eliminate many classes of bugs at compile-time. 97 | * [Learn](https://doc.rust-lang.org/book/index.html) 98 | * [Dark arts](https://doc.rust-lang.org/nomicon/) 99 | * [Golang](https://golang.org/) - Go is an open source programming language that makes it easy to build simple, reliable, and efficient software. 100 | * [Learn](https://learn.go.dev/) 101 | * [Julia](https://julialang.org/) - Julia is a high-level, high-performance dynamic language for technical computing. 102 | * [R](https://www.r-project.org/) - R is a free software environment for statistical computing and graphics. 103 | * [nvim-r](https://github.com/jalvesaq/Nvim-R) - Vim plugin to work with R 104 | 105 | ### Editors 106 | 107 | * [neovim](https://github.com/neovim/neovim) - Vim-fork focused on extensibility and usability 108 | * [Conquer of Completion](https://github.com/neoclide/coc.nvim) - Intellisense engine for vim8 & neovim, full language server protocol support as VSCode 109 | * [coc-rls](https://github.com/neoclide/coc-rls) - Rust language server support for coc.nvim 110 | * [coc-python](https://github.com/neoclide/coc-python) - Python extension for coc.nvim 111 | * [plug](https://github.com/junegunn/vim-plug) - Minimalist Vim Plugin Manager 112 | * [vim-go](https://github.com/fatih/vim-go) - Go development plugin for Vim 113 | * [vscode](https://github.com/microsoft/vscode) - Visual Studio Code is a streamlined code editor with support for development operations like debugging, task running, and version control. 114 | * [juno](https://github.com/JunoLab) - Juno is a powerful, free environment for the Julia language. 115 | * [Jupyter lab](https://github.com/jupyterlab/jupyterlab) - An extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture. 116 | * [IJulia](https://github.com/JuliaLang/IJulia.jl) - IJulia is a Julia-language backend combined with the Jupyter interactive environment (also used by IPython). 117 | * [gophernotes](https://github.com/gopherdata/gophernotes) - The Go kernel for Jupyter notebooks and nteract. 118 | 119 | ### Libraries 120 | 121 | * [Pandas](https://pandas.pydata.org/pandas-docs/stable/) - powerful Python data analysis toolkit 122 | * [matplotlob](https://github.com/matplotlib/matplotlib) - plotting with Python 123 | 124 | ## Data science 125 | 126 | ### Tools 127 | 128 | * [LogCluster](https://github.com/ristov/ristov.github.io/tree/master/logcluster) - experimental Perl-based tool for log file clustering and mining line patterns from log files 129 | 130 | ### Libraries 131 | 132 | * [scikit-learn](https://github.com/scikit-learn/scikit-learn) - machine learning in Python 133 | * [TensorFlow](https://github.com/tensorflow/tensorflow) - An Open Source Machine Learning Framework for Everyone 134 | * [TensorFlow-Examples](https://github.com/aymericdamien/TensorFlow-Examples) - TensorFlow Tutorial and Examples for Beginners (support TF v1 & v2) 135 | * [Ngraph](https://github.com/anvaka/ngraph) - a set of graph related algorithms. 136 | * [ngraph.pixel](https://github.com/anvaka/ngraph.pixel) - fast graph renderer based on low level ShaderMaterial from three.js 137 | * [Cayley](https://github.com/cayleygraph/cayley) - an open-source graph to be a part of the developer's toolbox 138 | 139 | ## Message queue and data pipelining 140 | 141 | * [Heka](https://github.com/mozilla-services/heka/) - Heka is a tool for collecting and collating data from a number of different sources, performing "in-flight" processing of collected data, and delivering the results to any number of destinations for further analysis. 142 | * [Hindsight](https://github.com/mozilla-services/hindsight) - Hindsight is lighter weight and faster data pipeline with delivery guarantees to replace Heka. 143 | * [nanomsg](https://github.com/nanomsg/nanomsg) - The nanomsg library is a simple high-performance implementation of several "scalability protocols". 144 | * [mangos](https://github.com/go-mangos/mangos) - Package mangos is an implementation in pure Go of the SP ("Scalable Protocols") protocols. 145 | * [Kapacitor](https://github.com/influxdata/kapacitor) - Framework for processing, monitoring, and alerting on time series data. 146 | * [Kafka](https://github.com/apache/kafka) - A distributed streaming platform. 147 | * [Apache Pulsar](https://github.com/apache/pulsar) - Distributed pub-sub messaging system. 148 | 149 | ## Hunting 150 | 151 | ### Intelligence platforms 152 | 153 | * [Malware Information Sharing Platform and Threat Sharing (MISP)](https://github.com/MISP/MISP) - Open Source Threat Intelligence and Sharing Platform 154 | * [Semi-Automated Cyber Threat Intelligence - ACT](https://github.com/mnemonic-no/act) - The main objective of the ACT project is to develop a platform for cyber threat intelligence to uncover cyber attacks, cyber espionage and sabotage. 155 | * [MITRE ATT&CK](https://mitre-attack.github.io/attack-navigator/enterprise/) - Globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. 156 | 157 | ### Playbooks 158 | 159 | * [Threat hunters playbook](https://github.com/hunters-forge/ThreatHunter-Playbook) - A Threat hunter's playbook to aid the development of techniques and hypothesis for hunting campaigns. 160 | 161 | ## Metrics and alerting 162 | 163 | * [Telegraf](https://github.com/influxdata/telegraf) - Telegraf is an plugin-driven agent for collecting & reporting metrics. 164 | 165 | ## Reading materials 166 | 167 | ---- 168 | 169 | [https://random-blather.com/2014/04/28/information-isnt-power/](Data2Info2Knowledge.md) 170 | 171 | ## see also 172 | 173 | * [Cyber Defence Monitoring Course Suite](https://github.com/ccdcoe/CDMCS) 174 | * [Technical Courses](https://ccdcoe.org/event/technical-courses.html) 175 | * [Cyber Defence Exercises](https://ccdcoe.org/event/cyber-defence-exercises.html) 176 | --------------------------------------------------------------------------------