├── .gitignore
├── INSTALL.md
├── README.md
├── battalion
├── domain-scan
    ├── domain-scan.sh
    └── scripts
    │   ├── apply-whitelist.py
    │   ├── dnstwist.sh
    │   ├── eyewitness.sh
    │   ├── filter-a-records.sh
    │   ├── filter-ip-addresses.sh
    │   ├── filter-subdomains.sh
    │   ├── find-drupal-domains.sh
    │   ├── find-http-domains.sh
    │   ├── find-wordpress-domains.sh
    │   ├── map-cname-records.py
    │   ├── nmap-aggressive.sh
    │   ├── nmap-basic.sh
    │   ├── parse-wpscan.py
    │   ├── run-domain-scan.sh
    │   ├── shodan.sh
    │   ├── whatweb.sh
    │   ├── whois.sh
    │   └── wpscan.sh
├── install-dependencies
├── install
    ├── download-tools.sh
    ├── install-kali-2016-2.sh
    └── install-ubuntu.sh
└── user-scan
    ├── name-bad-word-list
    ├── scripts
        ├── build-emails-from-names.sh
        ├── clean-harvested-names.sh
        ├── harvest-linkedin.sh
        ├── hibp-filter.sh
        ├── hibp-scan.sh
        ├── hunter-filter.sh
        ├── possible-emails.sh
        └── strip-name.sh
    ├── standard-email-list
    └── user-scan.sh


/.gitignore:
--------------------------------------------------------------------------------
1 | tools/
2 | api-keys/
3 | sample-input/
4 | 


--------------------------------------------------------------------------------
/INSTALL.md:
--------------------------------------------------------------------------------
  1 | Battalion Dependency Installation Guide
  2 | =======================================
  3 | 
  4 | This guide is intended to detail the required tools and steps to install all of the
  5 | tools and dependencies required by Battalion.
  6 | 
  7 | # Target Operating Systems
  8 | 
  9 | This guide is currently targeted towards Ubuntu-based Linux distributions. The general steps are the same though specific instructions may vary. Please see the scripts located within the `install` directory for per-distribution steps.
 10 | 
 11 | # About the Installation Scripts
 12 | 
 13 | The provided installation scripts consolidate the procedure outlined here. These scripts assume that the current user has `sudo` access and starts within the desired directory. Please keep in mind that _dependency installation requires sudo access_. We recommend reading the script for your distribution along with any tool-provided scripts so you understand what you're executing.
 14 | 
 15 | ## What do the scripts _not_ provide?
 16 | 
 17 | These scripts assume you will setup and manage your own Ruby and Python versions, and as such do not automatically install these packages.
 18 | 
 19 | ## Automatic Installation Script
 20 | 
 21 | ```bash
 22 | $ git clone https://github.com/theabraxas/Battalion.git
 23 | $ apt install gem
 24 | $ cd Battalion
 25 | $ ./install-dependencies <target>
 26 | ```
 27 | 
 28 | ## Supported Distributions
 29 | 
 30 | - [Ubuntu](install/install-ubuntu.sh): `./install-dependencies ubuntu`
 31 | - [Kali 2016.2](install/install-kali-2016-2.sh): `./install-dependencies kali-2016-2`
 32 | 
 33 | # Scanning Tools
 34 | 
 35 | Battalion leverages the following tools:
 36 | 
 37 | - [dnsrecon](https://github.com/darkoperator/dnsrecon)
 38 | - [EyeWitness](https://github.com/ChrisTruncer/EyeWitness)
 39 | - [Nmap](https://nmap.org/)
 40 | - [curl](https://curl.haxx.se/)
 41 | - [TheHarvester](https://github.com/laramies/theHarvester)
 42 | - [WhatWeb](https://github.com/urbanadventurer/WhatWeb)
 43 | - [whois](https://github.com/weppos/whois)
 44 | - [wpscan](https://github.com/wpscanteam/wpscan)
 45 | - [dnstwist](https://github.com/elceef/dnstwist)
 46 | 
 47 | # Manual Installation Steps
 48 | 
 49 | ## Clone Battalion and Clone Tools
 50 | 
 51 | The [download-tools.sh](download-tools.sh) script provided by Battalion will clone all of the required tools from GitHub and check out the appropriate tagged version.
 52 | 
 53 | ```bash
 54 | $ git clone https://github.com/theabraxas/Battalion
 55 | $ cd Battalion
 56 | $ ./download-tools.sh
 57 | ```
 58 | 
 59 | ## Install Tools from Package
 60 | 
 61 | These tools are available via the system package manager and do not require manual installation.
 62 | 
 63 | ```bash
 64 | $ sudo apt-get install jq nmap curl
 65 | ```
 66 | 
 67 | ## Install RVM and Install Ruby Version
 68 | 
 69 | ### Kali Linux
 70 | 
 71 | Kali comes with Python 2.7 and Ruby 2.3.1 by default in the 2016.2 version and does not require any additional setup. Other versions of Kali may need additional Ruby/Python setup.
 72 | 
 73 | ### Other Installs
 74 | 
 75 | We recommend using RVM to manage your Ruby versions and dependencies. Please see [RVM Installation Guide](https://rvm.io/rvm/install) for instructions on installing RVM. The tools we are utilizing prefer Ruby 2.3.1:
 76 | 
 77 | ```bash
 78 | $ rvm install 2.3.1
 79 | ```
 80 | 
 81 | Either set version 2.3.1 as the default, or run the following before utilizing Battalion:
 82 | 
 83 | ```bash
 84 | $ rvm use 2.3.1
 85 | ```
 86 | 
 87 | ## Install Python 2.7 
 88 | 
 89 | ### Kali Linux
 90 | 
 91 | Kali comes with Python 2.7 and Ruby 2.3.1 by default in the 2016.2 version and does not require any additional setup. Other versions of Kali may need additional Ruby/Python setup.
 92 | 
 93 | ### Other Installs
 94 | 
 95 | ```bash
 96 | $ sudo apt-get install python2.7 python-pip
 97 | $ sudo pip install --upgrade pip
 98 | ```
 99 | 
100 | ## Setup Tools from GitHub
101 | 
102 | ### dnsrecon
103 | 
104 | ```bash
105 | $ cd tools/dnsrecon
106 | $ pip install -r requirements.txt
107 | ```
108 | 
109 | ### dnstwist
110 | 
111 | Note that some system-level dependencies are required. We currently ignore the `ssdeep` dependency since Battalion does not use it.
112 | 
113 | ```bash
114 | $ cd tools/dnstwist
115 | $ sudo apt-get install libgeoip-dev libffi-dev
116 | $ pip install GeoIP==1.3.2 dnspython==1.14.0 requests=2.11.1 whois==0.7
117 | ```
118 | 
119 | ### EyeWitness
120 | 
121 | EyeWitness provides a setup script that installs a number of dependencies. The Battalion installation scripts utilize this setup script, which requires `sudo`. Please read through this script if you have any concerns.
122 | 
123 | ```bash
124 | $ cd tools/EyeWitness
125 | $ sudo ./setup/setup.sh
126 | ```
127 | 
128 | ### theHarvester
129 | 
130 | ```bash
131 | $ pip install requests
132 | ```
133 | 
134 | ### WhatWeb
135 | 
136 | ```bash
137 | $ gem install json
138 | ```
139 | 
140 | ### wpscan
141 | 
142 | ```bash
143 | $ cd tools/wpscan
144 | $ sudo apt-get install libcurl4-openssl-dev libxml2 libxml2-dev libxslt1-dev build-essential libgmp-dev zlib1g-dev
145 | $ gem install bundler
146 | $ bundle install --without test
147 | ```
148 | 
149 | ### whois
150 | 
151 | ```bash
152 | $ gem install whois
153 | ```
154 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | Battalion
 2 | =========
 3 | Battalion is a tool designed to automate a huge portion of a standard pentest. By supplying only a domain name and website site Battalion goes through the various passive and active reconaissance tasks, enumerates publicly accessible sites and services, identifies potential misconfigurations or vulnerable technologies, discoves and identifies breached accounts, build reports, and much more.
 4 | 
 5 | Ultimately Battalion will automate beyond reconaissance and go so far as to trigger phishing campaigns, automatically exploit some discovered vulnerabilities, and come with post-exploitation options.
 6 | 
 7 | Try out Battalion and send us any feedback! https://github.com/eidolonpg and I are excited to build out this tool and make it as comprehensive and efficient as possible!
 8 | 
 9 | # Installation
10 | 
11 | Battalion depends on a number of tools - please see the primary documentation in the [Battalion Installation Guide](INSTALL.md) for more information. This distribution also includes scripts for some system types in the `install` directory. The installation documentation provides more information on these scripts.
12 | 
13 | # Using Battalion
14 | 
15 | ## Example: Scanning a Domain and Users
16 | 
17 | ```bash
18 | $ ./battalion.sh --name "Test Scan" --out /home/user/scans/company \
19 |     --company "My Company" --domain "company.com" --nmap
20 | ```
21 | 
22 | This scan for `My Company` will produce results in the directory `/home/user/scans/company`. The domain scan would be based on the specified domain `company.com`, whereas the user scan is based upon the company name `My Company`. This scan also enables a light Nmap scan on the detected domains.
23 | 
24 | ## Required Parameters for All Scans
25 | 
26 | - `--name <scan name>`: The scan name
27 | - `--out  <directory>`: The output directory (absolute path)
28 | 
29 | ## Required Parameters for Domain Scans
30 | 
31 | - `--domain <domain name>`: The domain name to scan
32 | 
33 | ## Required Parameters for User Scans
34 | 
35 | - `--company <company name>`: The company name per LinkedIn, used for user scraping
36 | 
37 | ## Optional Parameters
38 | 
39 | - `--email-domain <domain name>`: Allows a different email domain to be configured. Use this if the primary domain is `x.com` but users receive mail at `y.com` addresses.
40 | - `--subdomain-list <file>`: Specify a file that provides potential subdomains, this is used by the dnsrecon tool. That tool provides some high-quality default lists.
41 | - `--nmap`: Enable light touch nmap scanning of subdomains
42 | - `--nmap-aggressive`: (Long running!) This is a VERY intense scan on each subdomain, approximately 10 minutes per subdomain.
43 | - `--shodan <api key>`: Specify a Shodan API key and enables a Shodan scan
44 | - `--hunter <api key>`: Sets a Hunter.io API Key and enables Hunter in the user scan. This will vastly speed up the user scan!
45 | - `--timeout-http <seconds>`: Specify a timeout in seconds for HTTP detection
46 | - `--timeout-eyewitness <seconds>`: Specify a timeout in seconds for EyeWitness individual scans
47 | 
48 | ## Disabling Major Scan Types
49 | 
50 | - `--disable-user`: Disable the user scan
51 | - `--disable-domain`: Disable the domain scan
52 | 
53 | # Scan Output
54 | 
55 | Battalion produces a number of directories which help categorize raw output. All of these directories will be created at the location specified by the `--out` parameter by the Battalion script.
56 | 
57 | ## Expected Scan Time (User)
58 | 
59 | The current default scan time for the user scan is rather large -- over 20 minutes. We recomment acquiring a [Hunter](https://hunter.io) API key to expidite this process.
60 | 
61 | ## Expected Scan Time (Domain)
62 | 
63 | Scans depend very much on the 'size' of the target, where the size is deteremined by the number of users and the number of detected domain records. Small targets usually take at least a few minutes to complete.
64 | 
65 | # Disclaimer
66 | 
67 | This utility has been created purely for the purposes of research and for improving defense, and is not intended to be used to attack systems except where explicitly authorized. Project maintainers are not responsible or liable for misuse of the software. Use responsibly.
68 | 


--------------------------------------------------------------------------------
/battalion:
--------------------------------------------------------------------------------
  1 | #!/bin/bash
  2 | # ______________________________________________
  3 | # |  ____        _   _        _ _               |
  4 | # | | __ )  __ _| |_| |_ __ _| (_) ___  _ __    |
  5 | # | |  _ \ / _` | __| __/ _` | | |/ _ \| '_ \   |
  6 | # | | |_) | (_| | |_| || (_| | | | (_) | | | |  |
  7 | # | |____/ \__,_|\__|\__\__,_|_|_|\___/|_| |_|  |
  8 | # |_____________________________________________|                                         
  9 | #
 10 | 
 11 | 
 12 | usage() {
 13 |     echo "Usage: battalion --name <name> --out <dir> --domain <domain> --company <company> [optional parameters] [--help]"
 14 |     echo ""
 15 |     echo "Required Parameters:"
 16 |     echo "    --name           <scan name>     Scan name that appears in output."
 17 |     echo "    --out            <absolute path> Output directory for scan, should be an absolute path that exists."
 18 |     echo ""
 19 |     echo "Required Parameters for Domain Scan:"
 20 |     echo "    --domain         <domain>        Domain being targeted by scan (example: google.com)."
 21 |     echo ""
 22 |     echo "Required Parameters for User Scan:"
 23 |     echo "    --company        <company name>  Company name as it appears on LinkedIn." 
 24 |     echo ""
 25 |     echo "Optional Parameters:"
 26 |     echo "    --subdomain-list     <file>    File containing subdomains to verify."
 27 |     echo "    --whitelist          <file>    File containing IP whitelist."
 28 |     echo "    --email-domain       <domain>  Used to set an email domain name (default is domain name)."
 29 |     echo "    --email-format       <format>  Used to set an email format. This prevents discovery"
 30 |     echo "                                   tasks from running and greatly increases scan speed."
 31 |     echo "                                   Formats will look {first}.{l} for 'firstname.lastinitial' formats."
 32 |     echo "    --nmap                         If this flag is set, Nmap scanning is added to the domain scan."
 33 |     echo "    --nmap-aggressive              If this flag is set, aggressive Nmap scanning is enabled for the domain."
 34 |     echo "    --shodan             <api key> Sets a Shodan API Key and adds Shodan to the domain scan."
 35 |     echo "    --hunter             <api key> Sets a Hunter API Key and enables Hunter in the User scan."
 36 |     echo "    --timeout-http       <seconds> Configure the timeout in seconds for HTTP detection."
 37 |     echo "    --timeout-eyewitness <seconds> Configure the timeout in seconds for EyeWitness."
 38 |     echo ""
 39 |     echo "Scan Types:"
 40 |     echo "    --disable-domain         Disable the domain scan."
 41 |     echo "    --disable-user           Disable the user scan."
 42 |     echo ""
 43 |     echo "Additional Parameters:"
 44 |     echo "    --help Display this usage text."
 45 |     echo ""
 46 |     echo "Subdomain Lists:"
 47 |     echo -n "The subdomain list file is used to specify potential subdomains to test to see if they exist."
 48 |     echo -n "You can generate a file yourself, or you can use a premade file. The dnsrecon tool that Battalion "
 49 |     echo -n -e "uses to perform these scans provides lists that we recommend utilizing.\n"
 50 |     echo ""
 51 |     echo "IP Whitelist:"
 52 |     echo -n "The IP Whitelist file is used to specify IP addresses (one per line) allowed"
 53 |     echo -n "for scanning. All domain-level scans after the initial domain scan are affected."
 54 |     echo ""
 55 | }
 56 | 
 57 | usage_short() {
 58 |     echo "Usage: battalion --name <name> --out <dir> --domain <domain> --company <company> --email-domain <domain> [optional parameters] [--help]"
 59 |     echo "Please use --help for more information."
 60 | }
 61 | 
 62 | export SCRIPT_DIRECTORY="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 63 | export BATTALION_DNSRECON_HOME=${BATTALION_DNSRECON_HOME:-$SCRIPT_DIRECTORY/tools/dnsrecon}
 64 | export BATTALION_EYEWITNESS_HOME=${BATTALION_EYEWITNESS_HOME:-$SCRIPT_DIRECTORY/tools/EyeWitness}
 65 | export BATTALION_HARVESTER_HOME=${BATTALION_HARVESTER_HOME:-$SCRIPT_DIRECTORY/tools/theHarvester}
 66 | export BATTALION_WHATWEB_HOME=${BATTALION_WHATWEB_HOME:-$SCRIPT_DIRECTORY/tools/WhatWeb}
 67 | export BATTALION_WPSCAN_HOME=${BATTALION_WPSCAN_HOME:-$SCRIPT_DIRECTORY/tools/wpscan}
 68 | export BATTALION_DNSTWIST_HOME=${BATTALION_DNSTWIST_HOME:-$SCRIPT_DIRECTORY/tools/dnstwist}
 69 | 
 70 | if [ $# -eq 0 ]; then
 71 |     usage_short
 72 |     exit 0
 73 | fi
 74 | 
 75 | DOMAIN_SCAN_ENABLED=true
 76 | USER_SCAN_ENABLED=true
 77 | 
 78 | while [[ $# -gt 0 ]]
 79 | do
 80 |     KEY="$1"
 81 | 
 82 |     case $KEY in
 83 |         --disable-domain)
 84 |             export DOMAIN_SCAN_ENABLED=false
 85 |             ;;
 86 |         --disable-user)
 87 |             export USER_SCAN_ENABLED=false
 88 |             ;;
 89 |         --name)
 90 |             SCAN_NAME="$2"
 91 |             shift
 92 |             ;;
 93 |         --out)
 94 |             SCAN_DIRECTORY="$2"
 95 |             shift
 96 |             ;;
 97 |         --domain)
 98 |             DOMAIN_TARGET="$2"
 99 |             shift
100 |             ;;
101 |         --subdomain-list)
102 |             DOMAIN_SUBDOMAIN_LIST="$2"
103 |             shift
104 |             ;;
105 |         --whitelist)
106 |             IP_WHITELIST_FILE="$2"
107 |             shift
108 |             ;;
109 |         --timeout-http)
110 |             DOMAIN_HTTP_SCAN_TIMEOUT="$2"
111 |             shift
112 |             ;;
113 |         --timeout-eyewitness)
114 |             EYEWITNESS_TIMEOUT="$2"
115 |             shift
116 |             ;;
117 |         --nmap)
118 |             NMAP_ENABLED=true
119 |             ;;
120 |         --nmap-aggressive)
121 |             NMAP_AGGRESSIVE_ENABLED=true
122 |             ;;
123 |         --shodan)
124 |             export SHODAN_ENABLED=true
125 |             SHODAN_API_KEY="$2"
126 |             shift
127 |             ;;
128 |         --hunter)
129 |             export HUNTER_ENABLED=true
130 |             HUNTER_API_KEY="$2"
131 |             shift
132 |             ;;
133 |         --company)
134 |             COMPANY_NAME="$2"
135 |             shift
136 |             ;;
137 |         --email-domain)
138 |             EMAIL_DOMAIN="$2"
139 |             shift
140 |             ;;
141 |         --email-format)
142 |             SPECIFIED_EMAIL_FORMAT="$2"
143 |             shift
144 |             ;;
145 |         --help)
146 |             usage
147 |             exit 0
148 |             ;;
149 |         *)
150 |             echo "Unrecognized parameter '$1'"
151 |             echo ""
152 |             exit 1
153 |             ;;
154 |     esac
155 |     shift
156 | done
157 | 
158 | export SCAN_NAME
159 | export DOMAIN_TARGET
160 | export SHODAN_API_KEY
161 | export HUNTER_API_KEY
162 | export COMPANY_NAME
163 | export DOMAIN_SUBDOMAIN_LIST=${DOMAIN_SUBDOMAIN_LIST:-$BATTALION_DNSRECON_HOME/subdomains-top1mil-20000.txt}
164 | export IP_WHITELIST_FILE
165 | export DOMAIN_HTTP_SCAN_TIMEOUT=${DOMAIN_HTTP_SCAN_TIMEOUT:-3}
166 | export EYEWITNESS_TIMEOUT=${EYEWITNESS_TIMEOUT:-15}
167 | export NMAP_ENABLED=${NMAP_ENABLED:-false}
168 | export NMAP_AGGRESSIVE_ENABLED=${NMAP_AGGRESSIVE_ENABLED:-false}
169 | export SHODAN_ENABLED=${SHODAN_ENABLED:-false}
170 | export HUNTER_ENABLED=${HUNTER_ENABLED:-false}
171 | export EMAIL_DOMAIN=${EMAIL_DOMAIN:-$DOMAIN_TARGET}
172 | 
173 | CONFIGURATION_ERROR=false
174 | 
175 | if [ -z "${SCAN_NAME}" ]; then
176 |     echo "[Error] The scan name must be configured."
177 |     CONFIGURATION_ERROR=true
178 | fi
179 | 
180 | if [ -z "${SCAN_DIRECTORY}" ]; then
181 |     echo "[Error] The scan directory must be configured."
182 |     CONFIGURATION_ERROR=true
183 | fi
184 | 
185 | if $DOMAIN_SCAN_ENABLED && [ -z "${DOMAIN_TARGET}" ]; then
186 |     echo "[Error] The domain target must be configured for domain scans."
187 |     CONFIGURATION_ERROR=true
188 | fi
189 | 
190 | if $DOMAIN_SCAN_ENABLED && [ -z "${DOMAIN_SUBDOMAIN_LIST}" ]; then
191 |     echo "[Error] A valid subdomain list file must be configured for domain scans."
192 |     CONFIGURATION_ERROR=true
193 | fi
194 | 
195 | if $USER_SCAN_ENABLED && [ -z "${COMPANY_NAME}" ]; then
196 |     echo "[Error] If the user scan is enabled a company name must be configured."
197 |     CONFIGURATION_ERROR=true
198 | fi
199 | 
200 | if $CONFIGURATION_ERROR; then
201 |     exit 1
202 | fi
203 | 
204 | # 
205 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
206 | # Prepare scan directory structure.
207 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
208 | # 
209 | 
210 | export SCAN_DIRECTORY
211 | export DOMAIN_DIRECTORY=$SCAN_DIRECTORY/domain
212 | export WHATWEB_DIRECTORY=$SCAN_DIRECTORY/whatweb
213 | export HTTP_DIRECTORY=$SCAN_DIRECTORY/http
214 | export NMAP_DIRECTORY=$SCAN_DIRECTORY/nmap
215 | export WORDPRESS_DIRECTORY=$SCAN_DIRECTORY/wordpress
216 | export EYEWITNESS_DIRECTORY=$SCAN_DIRECTORY/eyewitness-report
217 | export WHOIS_DIRECTORY=$SCAN_DIRECTORY/whois
218 | export SHODAN_DIRECTORY=$SCAN_DIRECTORY/shodan
219 | export USER_DIRECTORY=$SCAN_DIRECTORY/user
220 | export REPORT_DIRECTORY=$SCAN_DIRECTORY/report
221 | 
222 | build_dir() {
223 |     mkdir -p "${1}" >/dev/null || true
224 | }
225 | 
226 | build_dir "${SCAN_DIRECTORY}"
227 | build_dir "${DOMAIN_DIRECTORY}"
228 | build_dir "${WHATWEB_DIRECTORY}"
229 | build_dir "${HTTP_DIRECTORY}"
230 | build_dir "${NMAP_DIRECTORY}"
231 | build_dir "${WORDPRESS_DIRECTORY}"
232 | build_dir "${EYEWITNESS_DIRECTORY}"
233 | build_dir "${WHOIS_DIRECTORY}"
234 | build_dir "${SHODAN_DIRECTORY}"
235 | build_dir "${USER_DIRECTORY}"
236 | build_dir "${REPORT_DIRECTORY}"
237 | 
238 | # 
239 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
240 | # Scan Startup
241 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
242 | # 
243 | 
244 | # First acquire the Host IP address for reporting purposes.
245 | export HOST_IP_ADDRESS="$(curl -s ipinfo.io | grep ip | cut -d\" -f4)"
246 | 
247 | echo ">> Battalion <<"
248 | echo ""
249 | echo "Running scan '$SCAN_NAME' from IP address $HOST_IP_ADDRESS"
250 | echo ''
251 | 
252 | export DOMAIN_SCAN_SCRIPTS=$SCRIPT_DIRECTORY/domain-scan/scripts
253 | export USER_SCAN_SCRIPTS=$SCRIPT_DIRECTORY/user-scan/scripts
254 | 
255 | if $DOMAIN_SCAN_ENABLED ; then
256 |     $SCRIPT_DIRECTORY/domain-scan/domain-scan.sh &
257 |     DOMAIN_SCAN_PID=$!
258 |     echo -e "\t+ Domain scan is enabled and is running with PID ${DOMAIN_SCAN_PID}."
259 | else
260 |     echo -e "\t- Domain scan is disabled."
261 | fi
262 | 
263 | # Note that due to how the user scan polls HaveIBeenPwned, it currently runs extremely
264 | # slowly and will probably outlive the rest of the scanning.
265 | if $USER_SCAN_ENABLED ; then
266 |     export LINKEDIN_RESULTS=${SCAN_DIRECTORY}/user/linkedin-users.txt
267 |     export POSSIBLE_EMAILS=${SCAN_DIRECTORY}/user/possible-emails.txt
268 |     export COMPROMISED_STYLE=${SCAN_DIRECTORY}/user/compromised-style.txt
269 |     export PROBABLE_EMAILS=${SCAN_DIRECTORY}/user/probable-emails.txt
270 |     export COMPROMISED_EMAILS=${SCAN_DIRECTORY}/user/compromised-emails.txt
271 |     export SPECIFIED_EMAIL_FORMAT
272 | 
273 |     $SCRIPT_DIRECTORY/user-scan/user-scan.sh &
274 |     USER_SCAN_PID=$!
275 |     echo -e "\t+ User scan is enabled and is running with PID ${USER_SCAN_PID}."
276 | else
277 |     echo -e "\t- User scan is disabled."
278 | fi
279 | 
280 | echo ""
281 | 
282 | # 
283 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
284 | # Wait for the scans to complete 
285 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
286 | # 
287 | 
288 | if $DOMAIN_SCAN_ENABLED ; then
289 |     wait $DOMAIN_SCAN_PID
290 |     echo "Domain scan complete."
291 | fi
292 | 
293 | if $USER_SCAN_ENABLED ; then
294 |     wait $USER_SCAN_PID
295 |     echo "User scan complete."
296 | fi
297 | 
298 | # 
299 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
300 | # Part X - Complete!
301 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
302 | # 
303 | 
304 | echo ""
305 | echo "Completed scan '${SCAN_NAME}' in directory ${SCAN_DIRECTORY}"
306 | 


--------------------------------------------------------------------------------
/domain-scan/domain-scan.sh:
--------------------------------------------------------------------------------
  1 | #!/bin/bash
  2 | 
  3 | export DOMAIN_SCAN_SCRIPTS=$SCRIPT_DIRECTORY/domain-scan/scripts
  4 | 
  5 | # 
  6 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  7 | # Part 1 - dnsrecon domain scan
  8 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  9 | # 
 10 | echo "> Executing domain scan on $DOMAIN_TARGET using subdomain list $DOMAIN_SUBDOMAIN_LIST. This may take awhile, there is no stdout during this phase"
 11 | 
 12 | BASE_DOMAIN_REPORT=$DOMAIN_DIRECTORY/base-domain-report.txt
 13 | FULL_DOMAIN_REPORT=$DOMAIN_DIRECTORY/backup-domain-report.txt
 14 | WHITELISTED_RECORDS=$DOMAIN_DIRECTORY/whitelisted-records.txt
 15 | REJECTED_RECORDS=$DOMAIN_DIRECTORY/rejected-records.txt
 16 | A_RECORD_LIST=$DOMAIN_DIRECTORY/a-records.txt
 17 | IP_ADDRESS_LIST=$DOMAIN_DIRECTORY/a-record-ip-addresses.txt
 18 | SUBDOMAIN_LIST=$DOMAIN_DIRECTORY/subdomains.txt
 19 | CNAME_LIST=$DOMAIN_DIRECTORY/cnames.txt
 20 | HTTP_SUBDOMAIN_LIST=$DOMAIN_DIRECTORY/http-subdomains.txt
 21 | DNSTWIST_LIST=$DOMAIN_DIRECTORY/dnstwist.txt
 22 | 
 23 | $DOMAIN_SCAN_SCRIPTS/run-domain-scan.sh \
 24 |     "$DOMAIN_TARGET" 15 $DOMAIN_SUBDOMAIN_LIST \
 25 |     > $BASE_DOMAIN_REPORT
 26 | 
 27 | # Apply the IP whitelist if necessary
 28 | if [ ! -z $IP_WHITELIST_FILE ] && [ -f $IP_WHITELIST_FILE ]; then
 29 |     echo ""
 30 |     echo "> Applying IP Whitelist from file $IP_WHITELIST_FILE"
 31 | 
 32 |     touch $WHITELISTED_RECORDS
 33 |     touch $REJECTED_RECORDS
 34 | 
 35 |     python $DOMAIN_SCAN_SCRIPTS/apply-whitelist.py \
 36 |         $BASE_DOMAIN_REPORT $IP_WHITELIST_FILE $WHITELISTED_RECORDS $REJECTED_RECORDS
 37 | 
 38 |     cp -f $BASE_DOMAIN_REPORT $FULL_DOMAIN_REPORT
 39 |     cp -f $WHITELISTED_RECORDS $BASE_DOMAIN_REPORT
 40 | fi
 41 | 
 42 | # Produce a list of only the A records
 43 | $DOMAIN_SCAN_SCRIPTS/filter-a-records.sh $BASE_DOMAIN_REPORT \
 44 |     > $A_RECORD_LIST
 45 | 
 46 | # Produce a list of all IP Addresses of A records that match our primary domain name.
 47 | $DOMAIN_SCAN_SCRIPTS/filter-ip-addresses.sh $A_RECORD_LIST "$DOMAIN_TARGET" \
 48 |     > $IP_ADDRESS_LIST
 49 | 
 50 | # Identify all valid subdomains that match our primary domain name.
 51 | $DOMAIN_SCAN_SCRIPTS/filter-subdomains.sh $BASE_DOMAIN_REPORT "$DOMAIN_TARGET" \
 52 |     > $SUBDOMAIN_LIST
 53 | 
 54 | # Map CNAME records back to IP addresses
 55 | $DOMAIN_SCAN_SCRIPTS/map-cname-records.py "$DOMAIN_TARGET" $BASE_DOMAIN_REPORT \
 56 |     > $CNAME_LIST
 57 | 
 58 | # Identify all subdomains that support HTTP(s) connections.
 59 | echo ""
 60 | echo "> Identifying domains that support HTTP(s) connections for additional scanning"
 61 | 
 62 | cat $SUBDOMAIN_LIST \
 63 |     | jq -M -r -c '.subdomains | .[]' \
 64 |     | $DOMAIN_SCAN_SCRIPTS/find-http-domains.sh "$DOMAIN_HTTP_SCAN_TIMEOUT" \
 65 |     > $HTTP_SUBDOMAIN_LIST
 66 | 
 67 | echo -e "\t- Identified $(cat $HTTP_SUBDOMAIN_LIST | wc -l) subdomains that support HTTP(s)"
 68 | 
 69 | echo ""
 70 | echo "> Executing dnstwist scan on $DOMAIN_TARGET to find similar looking registered domains"
 71 | 
 72 | $DOMAIN_SCAN_SCRIPTS/dnstwist.sh "$DOMAIN_TARGET" \
 73 |      > "$DNSTWIST_LIST"
 74 | 
 75 | 
 76 | # 
 77 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 78 | # Part 2a - EyeWitness
 79 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 80 | # 
 81 | echo ""
 82 | echo "> Using EyeWitness to visually inspect all HTTP(s) subdomains."
 83 | 
 84 | $DOMAIN_SCAN_SCRIPTS/eyewitness.sh $BATTALION_EYEWITNESS_HOME \
 85 |     $SCAN_DIRECTORY/eyewitness-report \
 86 |     $HTTP_SUBDOMAIN_LIST \
 87 |     $EYEWITNESS_TIMEOUT &
 88 | 
 89 | EYEWITNESS_PID=$!
 90 | 
 91 | # 
 92 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 93 | # Part 2b - WhatWeb
 94 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 95 | # 
 96 | echo "> Using WhatWeb to analyze all HTTP(s) subdomains."
 97 | 
 98 | $DOMAIN_SCAN_SCRIPTS/whatweb.sh $WHATWEB_DIRECTORY $HTTP_SUBDOMAIN_LIST &
 99 | WHATWEB_PID=$!
100 | 
101 | #
102 | # Part 2 - Wait for processes to complete.
103 | #
104 | echo ""
105 | echo "> Waiting for EyeWitness and WhatWeb to complete..."
106 | echo -e "\t+ EyeWitness abort:(kill $EYEWITNESS_PID)"
107 | echo -e "\t+ WhatWeb    abort:(kill $WHATWEB_PID)"
108 | echo ""
109 | 
110 | wait $EYEWITNESS_PID
111 | wait $WHATWEB_PID
112 | 
113 | # 
114 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
115 | # Part 3 - WhatWeb Filtering
116 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
117 | # 
118 | 
119 | WORDPRESS_LIST=$HTTP_DIRECTORY/wordpress-domains
120 | $DOMAIN_SCAN_SCRIPTS/find-wordpress-domains.sh $WHATWEB_DIRECTORY \
121 |     | uniq -i \
122 |     > $WORDPRESS_LIST
123 | 
124 | echo "> Extracted $(cat $WORDPRESS_LIST | wc -l) domains using WordPress from WhatWeb results"
125 | 
126 | # 
127 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
128 | # Part 4 - Nmap (If Enabled)
129 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
130 | # 
131 | if $NMAP_ENABLED ; then
132 |     echo ""
133 |     echo "> Executing Nmap scan on all subdomains."
134 | 
135 |     cat $SUBDOMAIN_LIST | jq -M -r -c '.subdomains | .[]' | uniq -i | $DOMAIN_SCAN_SCRIPTS/nmap-basic.sh $NMAP_DIRECTORY
136 | 
137 |     echo -e "\t- Produced $(ls -1 ${NMAP_DIRECTORY} | wc -l) Nmap reports."
138 | 
139 | elif $NMAP_AGGRESSIVE_ENABLED ; then
140 |     echo ""
141 |     echo "> Executing Aggressive Nmap scan on all subdomains."
142 | 
143 |     cat $SUBDOMAIN_LIST | jq -M -r -c '.subdomains | .[]' | uniq -i | $DOMAIN_SCAN_SCRIPTS/nmap-aggressive.sh $NMAP_DIRECTORY
144 | 
145 |     echo -e "\t- Produced $(ls -1 ${NMAP_DIRECTORY} | wc -l) Nmap reports."
146 | 
147 | else
148 |     echo ""
149 |     echo "! Nmap disabled for this scan"
150 | fi
151 | 
152 | # 
153 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
154 | # Part 5 - WPScan for WordPress Domains
155 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
156 | # 
157 | echo ""
158 | echo "> Scanning $(cat ${WORDPRESS_LIST} | wc -l) domains for WordPress information and vulnerabilities."
159 | $DOMAIN_SCAN_SCRIPTS/wpscan.sh $WORDPRESS_LIST
160 | 
161 | echo -e "\t- Executed $(ls -1 $WORDPRESS_DIRECTORY | wc -l) WordPress scans."
162 | 
163 | 
164 | # 
165 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
166 | # Part 6 - WHOIS 
167 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
168 | # 
169 | echo ""
170 | echo "> Parsing WHOIS report for $DOMAIN_TARGET"
171 | 
172 | $DOMAIN_SCAN_SCRIPTS/whois.sh "$DOMAIN_TARGET" $WHOIS_DIRECTORY
173 | 
174 | 
175 | # 
176 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
177 | # Part 7 - Shodan (If Enabled)
178 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
179 | # 
180 | 
181 | if $SHODAN_ENABLED ; then
182 |     echo ""
183 |     echo "> Executing Shodan scan on $(cat $IP_ADDRESS_LIST | wc -l) IP addresses."
184 | 
185 |     cat $IP_ADDRESS_LIST | jq -M -r -c '.primaryIPAddresses | .[]' | $DOMAIN_SCAN_SCRIPTS/shodan.sh $SHODAN_DIRECTORY "${SHODAN_API_KEY}"
186 | 
187 |     echo -e "\t- Produced $(ls -1 $SHODAN_DIRECTORY | wc -l) Shodan reports."
188 | else
189 |     echo ""
190 |     echo "! Shodan disabled for this scan"
191 | fi
192 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/apply-whitelist.py:
--------------------------------------------------------------------------------
 1 | import json
 2 | import sys
 3 | 
 4 | if len(sys.argv) < 5:
 5 |     print('[Error] Expected arguments:')
 6 |     print('Scan results, IP whitelist, selected output file, rejected output file.')
 7 |     sys.exit(0)
 8 | 
 9 | scanFile = sys.argv[1]
10 | whitelistFile = sys.argv[2]
11 | 
12 | with open(scanFile) as data:
13 |     scan = json.load(data)
14 | 
15 | with open(whitelistFile) as data:
16 |     whitelist = [l.strip() for l in data.readlines()]
17 | 
18 | # Search through every record. Ensure that only targets that
19 | # match the IP whitelist make it through.
20 | selected = []
21 | rejected = []
22 | for record in scan['domainRecords']:
23 |     if record['target'] in whitelist:
24 |         selected.append(record)
25 |     else:
26 |         rejected.append(record)
27 | 
28 | selectedOut = { 'domainRecords': selected }
29 | rejectedOut = { 'domainRecords': rejected }
30 | 
31 | with open(sys.argv[3], 'w') as out:
32 |     json.dump(selectedOut, out)
33 | 
34 | with open(sys.argv[4], 'w') as out:
35 |     json.dump(rejectedOut, out)
36 | 
37 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/dnstwist.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | #
3 | # Generate a DNSTwist report using DNSTwist
4 | # Arguments specify to only find registered domains .
5 | 
6 | TARGET_DOMAIN="${1}"
7 | 
8 | ${BATTALION_DNSTWIST_HOME}/dnstwist.py $TARGET_DOMAIN -j -r -b -m -t 15 < /dev/stdin
9 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/eyewitness.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Invoke EyeWitness upon a list of subdomains. Note that we need to change directories to ensure that
 4 | # EyeWitness will work properly. This does not operate on standard input/output, and EyeWitness will 
 5 | # create and populate a new directory.
 6 | 
 7 | EYEWITNESS_HOME=$1
 8 | REPORT_TARGET=$2
 9 | SUBDOMAIN_LIST=$3
10 | TIMEOUT_SECONDS="${4}"
11 | 
12 | rm -fr ${REPORT_TARGET} || true
13 | 
14 | OLD_PWD=$(pwd)
15 | cd $EYEWITNESS_HOME
16 | 
17 | ./EyeWitness.py \
18 |     -f "${SUBDOMAIN_LIST}" \
19 |     -d "${REPORT_TARGET}" \
20 |     --timeout ${TIMEOUT_SECONDS} \
21 |     --headless --rdp --vnc --no-prompt --prepend-https \
22 |     > /dev/null
23 | 
24 | cd ${OLD_PWD}
25 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/filter-a-records.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | # We want to filter out all non-A-record elements.
4 | cat $1 | jq -M -c '.domainRecords | [.[] | if (.recordType == "A") then . else empty end] | unique_by(.domain | ascii_downcase) | {aRecords: .}'
5 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/filter-ip-addresses.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | A_RECORD_LIST=$1
4 | DOMAIN_TARGET=$2
5 | 
6 | cat $A_RECORD_LIST | jq -M -c '.aRecords | [.[] | if (.domain | contains("'"$DOMAIN_TARGET"'")) then .target else empty end] | unique | {primaryIPAddresses: .}'
7 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/filter-subdomains.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Search for all subdomains of the target domain.
 4 | # We want to match on all domains that contain the target domain.
 5 | 
 6 | BASE_DOMAIN_REPORT=$1
 7 | DOMAIN_TARGET="$2"
 8 | 
 9 | cat $BASE_DOMAIN_REPORT | jq -M -c '.domainRecords | [.[] | if (.domain | contains("'"$DOMAIN_TARGET"'")) then .domain else empty end | ascii_downcase] | unique | {subdomains: .}'
10 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/find-drupal-domains.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Read WhatWeb scan results and extract Drupal information.
 4 | # We know that each file in the WhatWeb output directory represents
 5 | # a single subdomain, so we scan each file for Drupal support.
 6 | 
 7 | WHATWEB_RAW_DATA=${1}
 8 | 
 9 | for WHATWEB_RAW_FILE in ${WHATWEB_RAW_DATA}/*.txt; do
10 |     FILE_BASENAME=`basename ${WHATWEB_RAW_FILE}`
11 |     SUBDOMAIN=${FILE_BASENAME%.*}
12 |     INDICATOR=$(grep '\[\s*Drupal\s*\]' ${WHATWEB_RAW_FILE})
13 | 
14 |     if [ ! -z "${INDICATOR}" ]; then
15 |         echo ${SUBDOMAIN}
16 |     fi
17 | done
18 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/find-http-domains.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Reads STDIN - each line should represent a subdomain to check.
 4 | # Uses curl to verify whether the subdomain supports HTTP. Only retain
 5 | # subdomains that expose standard HTTP ports.
 6 | 
 7 | CONNECT_TIMEOUT=${1}
 8 | 
 9 | check_https() {
10 |     SUBDOMAIN=${1}
11 |     curl -s \
12 |         --connect-timeout ${CONNECT_TIMEOUT} --max-time ${CONNECT_TIMEOUT} \
13 |         "${SUBDOMAIN}:443" >/dev/null && echo "${SUBDOMAIN}"
14 | }
15 | 
16 | check_ports() {
17 |     SUBDOMAIN=${1}
18 |     curl -s \
19 |         --connect-timeout ${CONNECT_TIMEOUT} --max-time ${CONNECT_TIMEOUT} \
20 |         "${SUBDOMAIN}:80" >/dev/null && echo "${SUBDOMAIN}" || check_https ${SUBDOMAIN}
21 | }
22 | 
23 | while read SUBDOMAIN
24 | do
25 |     # For each subdomain, include it if it supports port 80 or port 443
26 |     (>&2 echo -e "\t+ Checking ports 80/443 for domain [${SUBDOMAIN}]")
27 |     check_ports ${SUBDOMAIN}
28 | done < /dev/stdin
29 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/find-wordpress-domains.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Read WhatWeb scan results and extract WordPress information.
 4 | # We know that each file in the WhatWeb output directory represents
 5 | # a single subdomain, so we scan each file for WordPress support.
 6 | 
 7 | WHATWEB_RAW_DATA=${1}
 8 | 
 9 | for WHATWEB_RAW_FILE in ${WHATWEB_RAW_DATA}/*.txt; do
10 |     FILE_BASENAME=`basename ${WHATWEB_RAW_FILE}`
11 |     SUBDOMAIN=${FILE_BASENAME%.*}
12 |     INDICATOR=$(grep '\[\s*WordPress\s*\]' ${WHATWEB_RAW_FILE})
13 | 
14 |     if [ ! -z "${INDICATOR}" ]; then
15 |         echo ${SUBDOMAIN}
16 |     fi
17 | done
18 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/map-cname-records.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | import sys
 3 | 
 4 | domain = sys.argv[1]
 5 | filename = sys.argv[2]
 6 | 
 7 | def parse_file(filename):
 8 |     "Load the file as a record."
 9 |     with open(filename) as f:
10 |         return [parse_line(line.strip('\n')) for line in f.readlines()]
11 | 
12 | def parse_line(line):
13 |     "Parse a line by breaking it apart by spaces."
14 |     parts = line.split()
15 |     if len(parts) != 3:
16 |         return []
17 |     else:
18 |         return parts
19 | 
20 | def make_a_record_map(records, domain):
21 |     "Given a list of records and a domain, produce a mapping of A records to IP Address."
22 |     result = {}
23 | 
24 |     for record in records:
25 |         if len(record) == 3 and record[0] == "A" and domain in record[1]:
26 |             result[record[1]] = record[2]
27 |     
28 |     return result
29 | 
30 | def make_cname_record_map(records, domain):
31 |     "Given a list of records and a domain, produce a mapping of subdomains to domains."
32 |     result = {}
33 | 
34 |     for record in records:
35 |         if len(record) == 3 and record[0] == "CNAME" and record[2].endswith(domain):
36 |             result[record[1]] = record[2]
37 | 
38 |     return result
39 | 
40 | records = parse_file(filename)
41 | a_records = make_a_record_map(records, domain)
42 | cname_records = make_cname_record_map(records, domain)
43 | 
44 | for subdomain, domain in cname_records.items():
45 |     if domain in a_records:
46 |         print("{} {} {}".format(subdomain, domain, a_records[domain]))
47 | 
48 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/nmap-aggressive.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | OUTPUT_DIRECTORY="${1}"
 4 | 
 5 | while read SUBDOMAIN
 6 | do
 7 |     (>&2 echo -e "\t+ Executing aggressive NMAP scan for domain [$SUBDOMAIN]")
 8 |     nmap -T4 -A -Pn -sC -v -p- $SUBDOMAIN -oX "$OUTPUT_DIRECTORY/$SUBDOMAIN.xml" > "$OUTPUT_DIRECTORY/$SUBDOMAIN.txt"
 9 | done < /dev/stdin
10 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/nmap-basic.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | OUTPUT_DIRECTORY="${1}"
 4 | 
 5 | while read SUBDOMAIN
 6 | do
 7 |     (>&2 echo -e "\t+ Executing light Nmap scan for domain [$SUBDOMAIN]")
 8 |     nmap -T4 --open -F -v -Pn $SUBDOMAIN -oX "$OUTPUT_DIRECTORY/$SUBDOMAIN.xml" > "$OUTPUT_DIRECTORY/$SUBDOMAIN.txt"
 9 | done < /dev/stdin
10 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/parse-wpscan.py:
--------------------------------------------------------------------------------
  1 | import sys
  2 | 
  3 | inputfile = sys.argv[1]
  4 | outputfile = sys.argv[2]
  5 | 
  6 | elements = {}
  7 | jsonfile = ""
  8 | with open(inputfile, "r") as f:
  9 | 	lines = f.read().split("\n\n") #Splits WPscan data based on empty lines, this is essentially the logical groups
 10 | 	counter = 0
 11 | 	for line in lines: 
 12 | 		if "[+] URL" in line:
 13 | 			TargetSite = line
 14 | 		groups = line.split("\n") #Split data based on newlines, essentially elements per group
 15 | 		elements["var"+str(counter)] = groups
 16 | 		counter +=1
 17 | 
 18 | theme_and_plugin_safe = {} #themes and plugins with no associated vulns
 19 | theme_and_plugin_vulns = {} #themes and plugins with known associated vulns
 20 | generic_data = {} #Generic data about the scan and target
 21 | generic_data["Generic Data"] = []
 22 | 
 23 | for group in elements:
 24 | 	title = ""
 25 | 	theme_and_plugin_safe_holder = []
 26 | 	theme_and_plugin_vuln_holder = []
 27 | 	generic_data_holder = []
 28 | 
 29 | 	#grouptype checks
 30 | 	theme_and_plugin_safe_check = False
 31 | 	theme_and_plugin_vuln_check = False
 32 | 	generic_data_check = False
 33 | 	for i in elements[group]:
 34 | 		if "[!] Title" in i: #This finds themes and plugins with known vulnerabilities
 35 | 			title = i
 36 | 			theme_and_plugin_vuln_check = True
 37 | 		elif "[+] URL" in i or "[i] Wordpress Version" in i or "[+] Finished: " in i or "theme in use" in i or "[+] robots.txt" in i or "plugins found:" in i:
 38 | 			generic_data_holder.append(i)
 39 | 			generic_data_check = True
 40 | 		elif "[+] Name:" in i:
 41 | 			title = i
 42 | 			theme_and_plugin_safe_holder.append(i)
 43 | 			theme_and_plugin_safe_check = True
 44 | 		elif theme_and_plugin_vuln_check == True:
 45 | 			theme_and_plugin_vuln_holder.append(i)
 46 | 		elif theme_and_plugin_safe_check == True:
 47 | 			theme_and_plugin_safe_holder.append(i)
 48 | 		elif generic_data_check == True:
 49 | 			generic_data_holder.append(i)
 50 | 		else: 
 51 | 			break
 52 | 
 53 | 	if len(theme_and_plugin_vuln_holder) > 1:
 54 | 		theme_and_plugin_vulns[title] = theme_and_plugin_vuln_holder
 55 | 	if generic_data_check == True:
 56 | 		generic_data["Generic Data"].append(generic_data_holder)
 57 | 	if len(theme_and_plugin_safe_holder) > 1:
 58 | 		theme_and_plugin_safe[title] = theme_and_plugin_safe_holder
 59 | 
 60 | #Generic Data Placeholders
 61 | ScanDuration = ""
 62 | RequestsDone = ""
 63 | MemoryUsed = ""
 64 | Robots = ""
 65 | RobotLocation = ""
 66 | WordpressThemeInUse = ""
 67 | readme_present = "False"
 68 | interesting_headers = "Interesting Header"
 69 | interesting_header_json = '"Interesting Headers" : {'
 70 | Registration_enabled = "false"
 71 | xml_rpc = ""
 72 | Upload_dir = ""
 73 | Includes_dir = ""
 74 | TargetSite = TargetSite.split("[+]")[1].rstrip()
 75 | TargetSite = TargetSite.split(": ")[1] 
 76 | SafeTheme_SafePlugin = ""
 77 | jsonGenerator = '"Safe themes and plugins" : {'
 78 | 
 79 | 
 80 | 
 81 | for i in theme_and_plugin_safe: #themes and plugins with no detected vulns _ plus JSON generation
 82 | 	SafeTheme_SafePlugin = i.split("Name: ")[1]
 83 | 	jsonGenerator = jsonGenerator + ('"{ThemePlugin}" : {{ ').format(ThemePlugin=SafeTheme_SafePlugin)
 84 | 	for i in theme_and_plugin_safe[i]:
 85 | 		if "Latest version" in i:
 86 | 			jsonGenerator += '"Latest Version" : "%s", ' % i.split(": ")[1].rstrip()
 87 | 		elif "Location:" in i:
 88 | 			jsonGenerator += '"Location" : "%s", ' % str(i.split(": ")[1:])[2:-2]
 89 | 		elif "out of date" in i:
 90 | 			jsonGenerator += '"Latest version" : "%s", ' % i.split("version is ")[2]
 91 | 	jsonGenerator = jsonGenerator[:-2] + '}, '
 92 | jsonGenerator = jsonGenerator[:-3] + '}'
 93 | 
 94 | #############################################
 95 | 
 96 | jsonGenerator2 = '"Vulnerable themes and plugins" : {'
 97 | 
 98 | if len(theme_and_plugin_vulns) > 1:
 99 | 	for i in theme_and_plugin_vulns: #Theme and plugin vulnerability Data
100 | 		referenceCounter = 1
101 | 		AdditionalCounter = 1
102 | 		VulnTheme_VulnPlugin = i.split("Title: ")[1]
103 | 		jsonGenerator2 = jsonGenerator2 + ('"{ThemePlugin}" : {{ ').format(ThemePlugin=VulnTheme_VulnPlugin)
104 | 		jsonGenerator2 += '"Vulnerability Type" : "%s", ' % VulnTheme_VulnPlugin.split(" - ")[1]
105 | 		for i in theme_and_plugin_vulns[i]:
106 | 			if "Reference" in i:
107 | 				url = i.split("Reference: ")[1]
108 | 				jsonGenerator2 += '"Reference%s" : "%s", ' % (str(referenceCounter), str(i.split(": ")[1:])[2:-2])
109 | 				referenceCounter += 1
110 | 			elif "Fixed in:" in i:
111 | 				fixed_version = i.split("in: ")[1]
112 | 				jsonGenerator2 += '"Fixed in" : "%s", ' % i.split(": ")[1]
113 | 			else:
114 | 				jsonGenerator2 += '"Additional Data%s" : "%s", ' % (str(AdditionalCounter), i.split(": ")[1])
115 | 				AdditionalCounter += 1
116 | 		jsonGenerator2 = jsonGenerator2[:-2] + '}, '
117 | else:
118 | 	jsonGenerator2 += 'none : "none"   '
119 | jsonGenerator2 = jsonGenerator2[:-2] + '}'
120 | 
121 | #convert mixed dictionary to single dictionary
122 | generic_data_final ={}
123 | templist = []
124 | for i in generic_data:
125 | 	for i in generic_data[i]:
126 | 		for j in i:
127 | 			templist.append(j)
128 | generic_data_final["Generic Scan Data"] = templist
129 | 
130 | for j in generic_data_final: #Generic Data Output
131 | 	header_counter = 1
132 | 	for i in generic_data_final[j]:
133 | 		if "Elapsed time" in i:
134 | 			ScanDuration = i.split("e: ")[1]
135 | 		elif "Requests Done" in i:
136 | 			RequestsDone = i.split(": ")[1]
137 | 		elif "Memory used" in i:
138 | 			MemoryUsed = i.split("used: ")[1]
139 | 		elif "robots.txt" in i:
140 | 			Robots = "True"
141 | 			RobotLocation = i.split(": ")[1]
142 | 			RobotLocation = RobotLocation[:-1]
143 | 		elif "file exists exposing" in i:
144 | 			readme_present = i.split("'")[1]
145 | 		elif "theme in use" in i:
146 | 			WordpressThemeInUse = i.split(": ")[1]
147 | 		elif "Interesting header" in i:
148 | 			headerdata = i.split(": ")[1:]
149 | 			header = ""
150 | 			for i in headerdata:
151 | 				header += str(i) + " "
152 | 			interesting_header_json += '"Interesting Header%s" : "%s", ' % (header_counter, header.replace('"',"'"))
153 | 			header_counter+=1
154 | 		elif "Registration is enabled" in i:
155 | 			Registration_enabled = i.split(": ")[1]
156 | 		elif "XML-RPC" in i:
157 | 			xml_rpc = i.split(": ")[1]
158 | 		elif "Upload directory" in i:
159 | 			Upload_dir = i.split(": ")[1]
160 | 		elif "Includes directory" in i:
161 | 			Includes_dir = i.split(": ")[1]
162 | 
163 | 
164 | interesting_header_json= interesting_header_json[:-2] + "}"
165 | 
166 | if interesting_header_json == '"Interesting Headers" :}':
167 | 	interesting_header_json = '"Interesting Headers" : {"none" : "none"}'
168 | 
169 | 
170 | jsonfile = ('{{"WPScan_Site" : "{URL}", "Generic_Scan_Data" : \
171 | 	{{ "ScanDuration" : "{ScanDuration}", "Requests Done" : "{RequestsDone}", "Memory Used" : "{MemoryUsed}", "robots.txt present?" : "{Robots}", \
172 | 	"Robots.txt Location" : "{RobotLocation}", "Readme File exposing version?" : "{Readme}", "Registration Enabled?" : "{Reg}", "XML-RPC?" : "{XML}", \
173 | 	"Exposed Uploads Directory?" : "{Upload}", "Exposed Includes Directory" : "{Include}", {headerJson}, "Theme In Use" : "{WordpressThemeInUse}"}}, {SafeThemePluginJSON}}}, {VulnThemeJson}}}'\
174 | 	).format(URL=TargetSite, ScanDuration=ScanDuration,RequestsDone=RequestsDone,MemoryUsed=MemoryUsed,Robots=Robots,RobotLocation=RobotLocation, \
175 | 	WordpressThemeInUse=WordpressThemeInUse, SafeThemePluginJSON=jsonGenerator, VulnThemeJson=jsonGenerator2, Readme=readme_present, \
176 | 	headerJson=interesting_header_json, Reg=Registration_enabled, XML= xml_rpc, Upload=Upload_dir, Include=Includes_dir)
177 | 
178 | with open(outputfile, 'w') as f:
179 |     f.write(jsonfile)
180 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/run-domain-scan.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Generate a Domain report using DNSRecon.
 4 | # We strip some of the unimportant banner text and summary information.
 5 | 
 6 | TARGET_DOMAIN="${1}"
 7 | DOMAIN_SCAN_THREAD_LIMIT="${2}"
 8 | SUBDOMAIN_LIST_FILE="${3}"
 9 | 
10 | function execute_domain_scan() {
11 |     ${BATTALION_DNSRECON_HOME}/dnsrecon.py \
12 |         -d "${TARGET_DOMAIN}" \
13 |         -t brt \
14 |         -D ${SUBDOMAIN_LIST_FILE} \
15 |         --threads ${DOMAIN_SCAN_THREAD_LIMIT} \
16 |         | tail -n +2 \
17 |         | head -n -1 \
18 |         | cut -c 7- \
19 |         | grep -v ":" \
20 |         | sort -k1,1
21 | }
22 | 
23 | function convert_record() {
24 |     RECORD_TYPE=$(echo "$1" | cut -d' ' -f1)
25 |     DOMAIN=$(echo "$1" | cut -d' ' -f2)
26 |     TARGET=$(echo "$1" | cut -d' ' -f3)
27 | 
28 |     echo -n '{"recordType":"'"$RECORD_TYPE"'","domain":"'"$DOMAIN"'","target":"'"$TARGET"'"}'
29 | }
30 | 
31 | function convert_raw_to_json() {
32 |     IS_FIRST_LINE=true
33 |     while read -r LINE; do
34 |         # Each line will either be blank or have the format:
35 |         # RECORD_TYPE DOMAIN TARGET
36 |         # where each element is separated by a single space.
37 |         if [ ! -z "$LINE" ]; then
38 |             if $IS_FIRST_LINE; then
39 |                 IS_FIRST_LINE=false
40 |             else
41 |                 echo -n ","
42 |             fi
43 |             convert_record "$LINE"
44 |         fi
45 |     done <<< "$1"
46 | }
47 | 
48 | RAW_OUTPUT=$(execute_domain_scan)
49 | 
50 | echo -n '{"domainRecords":['
51 | convert_raw_to_json "$RAW_OUTPUT"
52 | echo -n ']}'
53 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/shodan.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | SCAN_DIRECTORY="${1}"
 4 | SHODAN_API_KEY="${2}"
 5 | 
 6 | while read IP_ADDRESS
 7 | do
 8 |     (>&2 echo -e "\t+ Executing Shodan scan for IP address [${IP_ADDRESS}]")
 9 |     curl -s -X GET \
10 |         "https://api.shodan.io/shodan/host/${IP_ADDRESS}?key=${SHODAN_API_KEY}" \
11 |         > "${SCAN_DIRECTORY}/${IP_ADDRESS}.txt"
12 | done < /dev/stdin
13 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/whatweb.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | SCAN_DIRECTORY="${1}"
 4 | SUBDOMAIN_FILE="${2}"
 5 | 
 6 | while read SUBDOMAIN; do
 7 |     # For each subdomain, execute WhatWeb and dump the raw results to a file.
 8 |     OUTPUT="${SCAN_DIRECTORY}/${SUBDOMAIN}.txt"
 9 |     JSONOUTPUT="${SCAN_DIRECTORY}/${SUBDOMAIN}.json"
10 |     ${BATTALION_WHATWEB_HOME}/whatweb --color=never --no-errors -a1 -v "${SUBDOMAIN}" --log-json="${JSONOUTPUT}" > "${OUTPUT}"
11 | done <${SUBDOMAIN_FILE}
12 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/whois.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash 
 2 | 
 3 | DOMAIN_TARGET=${1}
 4 | WHOIS_SCAN_DIRECTORY=${2}
 5 | FILE_BASENAME="${WHOIS_SCAN_DIRECTORY}/${DOMAIN_TARGET}"
 6 | 
 7 | ruby -r 'whois' -e 'puts Whois::Client.new(:timeout => 30).lookup("'"${DOMAIN_TARGET}"'")' > "${FILE_BASENAME}.raw.txt" 
 8 | 
 9 | RNAME=`grep "Registrant Name" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
10 | REMAIL=`grep "Registrant Email" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
11 | RADDR1=`grep "Registrant Street" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
12 | RADDR2=`grep "Registrant City" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
13 | RADDR3=`grep "Registrant State" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
14 | RADDR4=`grep "Registrant Postal" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
15 | TNAME=`grep "Tech Name" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
16 | TEMAIL=`grep "Tech Email" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
17 | TADDR1=`grep "Tech Street" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
18 | TADDR2=`grep "Tech City" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
19 | TADDR3=`grep "Tech State" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
20 | TADDR4=`grep "Tech Postal" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
21 | REGI=`grep -A0 "Registrar:" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //' | tail -n1`
22 | REXPD=`grep "Registrar Registration Expiration Date" "${FILE_BASENAME}.raw.txt" | sed 's/.*: //'`
23 | 
24 | echo '{"whois" : {"registrantName":"'"$RNAME"'","registrantEmail":"'"$RMEAIL"'","registrantAddress":"'"$RADDR1 $RADDR2 $RADDR3 $RADDR4"'","techName":"'"$TNAME"'","techEmail":"'"$TEMAIL"'","techAddress":"'"$TADDR1 $TADDR2 $TADDR3 $TADDR4"'","registrar":"'"$REGI"'","domainExirationDate":"'"$REXPD"'"}}' > "${FILE_BASENAME}.json"
25 |  
26 | 
27 | cat "${FILE_BASENAME}.raw.txt" \
28 |     | grep -E 'Registrar:|Referral URL:|Creation Date:|Expiration Date:|DNSSEC:|Registratnt Name:|Registrant Organization:|Registrant Street:|Registrant City:|Registrant State/Province:|Registrant Phone:|Registrant Email:|Admin Name:|Admin Street:|Admin City:|Admin State/Province:|Admin Phone:|Admin Email:|Tech Name:|Tech Street:|Tech City:|Tech State/Province:|Tech Phone:|Tech Email:' \
29 |     | grep --invert-match 'Registrar Registration Expiration Date:' \
30 |     | sed -e 's/^[[:space:]]*//' \
31 |     > "${FILE_BASENAME}.txt"
32 | 


--------------------------------------------------------------------------------
/domain-scan/scripts/wpscan.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | WORDPRESS_DOMAINS=${1}
 4 | 
 5 | OLD_PWD=$(pwd)
 6 | while read WP_DOMAIN; do
 7 |     RAW_OUTPUT=$WORDPRESS_DIRECTORY/$WP_DOMAIN.txt
 8 |     JSON_OUTPUT=$WORDPRESS_DIRECTORY/$WP_DOMAIN.json
 9 | 
10 |     cd $BATTALION_WPSCAN_HOME
11 | 
12 |     if [ ! -z "$WP_DOMAIN" ]; then
13 |         (>&2 echo -e "\t+ Running wpscan for domain [$WP_DOMAIN]")
14 |         ruby wpscan.rb \
15 |             --batch --no-color --follow-redirection \
16 |             --url "$WP_DOMAIN" \
17 |             > $RAW_OUTPUT
18 |     fi
19 | 
20 |     cd $OLD_PWD
21 | 
22 |     python $DOMAIN_SCAN_SCRIPTS/parse-wpscan.py $RAW_OUTPUT $JSON_OUTPUT
23 | 
24 | done <${WORDPRESS_DOMAINS}
25 | 
26 | 


--------------------------------------------------------------------------------
/install-dependencies:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | export SCRIPT_DIRECTORY="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 4 | 
 5 | usage() {
 6 |     echo "Usage: ./install-dependencies <target>"
 7 |     echo ""
 8 |     echo "Available Targets:"
 9 |     echo "ubuntu, kali-2016-2"
10 | }
11 | 
12 | # Goals:
13 | # - Move download-tools.sh to the install/ directory
14 | # - Create singular installation script (choose target)
15 | # - Make directories "smart"
16 | 
17 | INSTALL_TARGET="$1"
18 | 
19 | if [ -z "${INSTALL_TARGET}" ]; then
20 |     echo "[Error] Please specify a target platform."
21 |     echo ""
22 |     echo "Usage: ./install.sh <target>"
23 |     echo ""
24 |     echo "Available Targets:"
25 |     echo "ubuntu, kali-2016-2"
26 | 
27 |     exit 1
28 | fi
29 | 
30 | if [ "$INSTALL_TARGET" = "--help" ]; then
31 |     usage
32 |     exit 0
33 | fi
34 | 
35 | $SCRIPT_DIRECTORY/install/download-tools.sh "$SCRIPT_DIRECTORY"
36 | $SCRIPT_DIRECTORY/install/install-${INSTALL_TARGET}.sh "$SCRIPT_DIRECTORY"
37 | 


--------------------------------------------------------------------------------
/install/download-tools.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | BASE_DIRECTORY=$1
 4 | 
 5 | mkdir -p $BASE_DIRECTORY/tools || true >/dev/null 2>&1
 6 | 
 7 | git clone https://github.com/darkoperator/dnsrecon $BASE_DIRECTORY/tools/dnsrecon
 8 | cd $BASE_DIRECTORY/tools/dnsrecon
 9 | git checkout tags/v0.8.9 -b battalion
10 | cd $BASE_DIRECTORY
11 | 
12 | git clone https://github.com/ChrisTruncer/EyeWitness $BASE_DIRECTORY/tools/EyeWitness
13 | cd $BASE_DIRECTORY/tools/EyeWitness
14 | git checkout tags/2.2.2 -b battalion
15 | cd $BASE_DIRECTORY
16 | 
17 | git clone https://github.com/laramies/theHarvester $BASE_DIRECTORY/tools/theHarvester
18 | cd $BASE_DIRECTORY/tools/theHarvester
19 | git checkout tags/2.7 -b battalion
20 | cd $BASE_DIRECTORY
21 | 
22 | git clone https://github.com/urbanadventurer/WhatWeb $BASE_DIRECTORY/tools/WhatWeb
23 | 
24 | git clone https://github.com/wpscanteam/wpscan $BASE_DIRECTORY/tools/wpscan
25 | cd $BASE_DIRECTORY/tools/wpscan
26 | git checkout tags/2.9.2 -b battalion
27 | cd $BASE_DIRECTORY
28 | 
29 | git clone https://github.com/elceef/dnstwist $BASE_DIRECTORY/tools/dnstwist
30 | cd $BASE_DIRECTORY/tools/dnstwist
31 | git checkout tags/v1.02 -b battalion
32 | cd $BASE_DIRECTORY
33 | 


--------------------------------------------------------------------------------
/install/install-kali-2016-2.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/sh
 2 | 
 3 | sudo apt-get install jq nmap libgeoip-dev libffi-dev libcurl4-openssl-dev libxml2 libxml2-dev libxslt1-dev build-essential libgmp-dev zlib1g-dev curl
 4 | 
 5 | pip install requests
 6 | 
 7 | cd tools/dnsrecon
 8 | pip install -r requirements.txt
 9 | 
10 | cd ../dnstwist
11 | pip install GeoIP==1.3.2 dnspython==1.14.0 requests==2.11.1 whois==0.7
12 | 
13 | cd ../EyeWitness
14 | sudo -H ./setup/setup.sh
15 | 
16 | gem install json
17 | gem install whois
18 | 
19 | cd ../wpscan
20 | gem install bundler
21 | bundle install --without test
22 | 


--------------------------------------------------------------------------------
/install/install-ubuntu.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/sh
 2 | 
 3 | sudo apt-get install jq nmap libgeoip-dev libffi-dev libcurl4-openssl-dev libxml2 libxml2-dev libxslt1-dev build-essential libgmp-dev zlib1g-dev curl
 4 | 
 5 | pip install requests
 6 | 
 7 | cd tools/dnsrecon
 8 | pip install -r requirements.txt
 9 | 
10 | cd ../dnstwist
11 | pip install GeoIP==1.3.2 dnspython==1.14.0 requests==2.11.1 whois==0.7
12 | 
13 | cd ../EyeWitness
14 | sudo -H ./setup/setup.sh
15 | 
16 | gem install json
17 | gem install whois
18 | 
19 | cd ../wpscan
20 | gem install bundler
21 | bundle install --without test
22 | 


--------------------------------------------------------------------------------
/user-scan/name-bad-word-list:
--------------------------------------------------------------------------------
 1 | Mr.
 2 | Mrs.
 3 | Sr.
 4 | Jr.
 5 | CPA
 6 | PMP
 7 | Dr.
 8 | ,
 9 | .
10 | CEO
11 | CTO
12 | CIO
13 | CFO
14 | 


--------------------------------------------------------------------------------
/user-scan/scripts/build-emails-from-names.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Given a list of names - one per line - generate a list of email addresses
 4 | # from those names.
 5 | 
 6 | while read LINE
 7 | do
 8 |     echo "$LINE@$EMAIL_DOMAIN"
 9 | done < ${1:-/dev/stdin}
10 | 


--------------------------------------------------------------------------------
/user-scan/scripts/clean-harvested-names.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | # Apply a filter to all of the names to strip bad words and characters.
4 | 
5 | while read NAME
6 | do
7 |     $USER_SCAN_SCRIPTS/strip-name.sh "$NAME" < $SCRIPT_DIRECTORY/user-scan/name-bad-word-list
8 | done < "${1:-/dev/stdin}"
9 | 


--------------------------------------------------------------------------------
/user-scan/scripts/harvest-linkedin.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | COMPANY_NAME="$1"
 4 | RESULT_LIMIT="$2"
 5 | BASE_SKIP=19
 6 | REAL_SKIP=$((${BASE_SKIP} + (${RESULT_LIMIT} / 100)))
 7 | 
 8 | if [ "${REAL_SKIP}" -lt "20" ]; then
 9 |     REAL_SKIP=20
10 | fi
11 | 
12 | # -d <target> -- for LinkedIn this is the company name
13 | # -l <limit>  -- number of results
14 | 
15 | python ${BATTALION_HARVESTER_HOME}/theHarvester.py -d "${COMPANY_NAME}" -b linkedin -l ${RESULT_LIMIT} \
16 |     | tail -n +${REAL_SKIP} \
17 |     | sed 's/^ *//;s/ *$//' \
18 |     | uniq -u 
19 | 


--------------------------------------------------------------------------------
/user-scan/scripts/hibp-filter.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Given a set of possible emails, attempt to narrow down the list using
 4 | # HaveIBeenPwned -- if any emails match, consider them "valid" and select
 5 | # that format in general.
 6 | #
 7 | # The input should look like:
 8 | #
 9 | # email_style|email
10 | #
11 | # Please see the possible-emails.sh script for more information on this
12 | # format. Matched emails will output their type.
13 | 
14 | HIBP_REQUEST="https://haveibeenpwned.com/api/v2/breachedaccount/"
15 | HIBP_PARAMS="?truncateResponse=True"
16 | 
17 | CURRENT_COUNT=0
18 | 
19 | print_progress() {
20 |     let CURRENT_COUNT=CURRENT_COUNT+1
21 |     echo -ne "\rScanning emails to find format.  Scanned $CURRENT_COUNT emails." >&2
22 | }
23 | 
24 | handle_potential_email() {
25 |     IFS='|' read -r -a DESCRIPTOR <<< "${1}"
26 |     STYLE="${DESCRIPTOR[0]}"
27 |     EMAIL="${DESCRIPTOR[1]}"
28 | 
29 |     RESULT="$(curl -s -w "\n\n<%{http_code}>" ${HIBP_REQUEST}${EMAIL}${HIBP_PARAMS})"
30 |     STATUS_CODE="$(echo $RESULT | tail -n 1)"
31 |     BODY="$(echo "$RESULT" | head -n -2)"
32 | 
33 |     if [ "$STATUS_CODE" = "<200>" ] && [ ! -z "${BODY// }" ]; then
34 |         echo "$STYLE"
35 |     fi
36 | }
37 | 
38 | while read LINE
39 | do
40 |     EMAIL_RESULT="$(handle_potential_email "${LINE}")"
41 |     print_progress
42 | 
43 |     if [ -z "$EMAIL_RESULT" ]; then
44 |         sleep 1.6
45 |     else
46 |         # Break on the first result. This tells us the "Valid format" to select
47 |         echo $EMAIL_RESULT
48 |         exit 0
49 |     fi
50 | done < /dev/stdin
51 | 


--------------------------------------------------------------------------------
/user-scan/scripts/hibp-scan.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | HIBP_REQUEST="https://haveibeenpwned.com/api/v2/breachedaccount/"
 4 | HIBP_PARAMS="?truncateResponse=True"
 5 | 
 6 | CURRENT_COUNT=0
 7 | 
 8 | print_progress() {
 9 |     let CURRENT_COUNT=CURRENT_COUNT+1
10 |     echo -ne "\rScanned $CURRENT_COUNT emails." >&2
11 | }
12 | 
13 | handle_potential_email() {
14 |     EMAIL="$1"
15 | 
16 |     RESULT=`curl -s -w "\n\n<%{http_code}>" ${HIBP_REQUEST}${EMAIL}${HIBP_PARAMS}`
17 |     STATUS_CODE="$(echo "$RESULT" | tail -n 1)"
18 |     BODY="$(echo "$RESULT" | head -n -2)"
19 | 
20 |     if [ "$STATUS_CODE" = "<200>" ] && [ ! -z "${BODY// }" ]; then
21 |         echo "{\"email\":\"$EMAIL\",\"breaches\":$BODY}"
22 |     else
23 |         if [ "$STATUS_CODE" = "<429>" ]; then
24 |             >&2 echo "Exceeded acceptable rate for HaveIBeenPwned, waiting 5 seconds to recover."
25 | 
26 |             # Immediately sleep for 5 seconds if we manage to exceed our rate.
27 |             # We want to be polite to HIBP, so give it some time to cool down.
28 |             sleep 5
29 |         fi
30 |     fi
31 | }
32 | 
33 | while read LINE
34 | do
35 |     handle_potential_email "${LINE}"
36 |     print_progress
37 | 
38 |     # Ensure that we keep this cooldown up-to-date with what HIBP requires.
39 |     sleep 1.6
40 | done < /dev/stdin
41 | 


--------------------------------------------------------------------------------
/user-scan/scripts/hunter-filter.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | # Use hunter to determine the most common email pattern, returning that pattern
 4 | # to use for HIBP.
 5 | 
 6 | # Note that each unique scan made against Hunter.io will consume a credit from your
 7 | # account. Please only enable Hunter if you have credits available and want to use
 8 | # them to augment your Battalion results.
 9 | 
10 | HUNTER_API_REQUEST="https://api.hunter.io/v2/domain-search?domain=${DOMAIN_TARGET}&api_key=${HUNTER_API_KEY}"
11 | HUNTER_RESPONSE="$(curl -s -w "\n\n<%{http_code}>" $HUNTER_API_REQUEST)"
12 | STATUS_CODE="$(echo "$HUNTER_RESPONSE" | tail -n 1)"
13 | BODY="$(echo "$HUNTER_RESPONSE" | head -n -2)"
14 | 
15 | # Expect response: 200 OK for successful results (reject everything else)
16 | if [ "$STATUS_CODE" = "<200>" ]; then
17 |     # Parse the response (JSON) using jq
18 |     PATTERN="$(echo "$BODY" | jq -M -c -r '.data.pattern')"
19 |     echo "$PATTERN"
20 | else
21 |     >&2 echo "! Failed to query Hunter.io and received a status code '$STATUS_CODE'"
22 |     >&2 echo "! Please ensure that you are utilizing a valid API key."
23 | fi
24 | 


--------------------------------------------------------------------------------
/user-scan/scripts/possible-emails.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Given a domain and a list of names, generate a set of potential emails
 4 | # for every name. The output format is:
 5 | #
 6 | # email_style|email
 7 | #
 8 | # About Email Styles
 9 | # ==============================================
10 | # Email styles are different combinations of name parts. We utilize four variables:
11 | # 
12 | # - {first} = First name
13 | # - {last}  = Last name
14 | # - {f}     = First letter of first name
15 | # - {l}     = First letter of last name
16 | #
17 | # Those variables might be mixed with common characters as well. For instances we
18 | # might have {f}{last}@domain.com or {first}-{last}@domain.com
19 | #
20 | 
21 | EMAIL_DOMAIN=$1
22 | 
23 | handle_name() {
24 |     FIRST_NAME=$1
25 |     LAST_NAME=$2
26 |     FIRST_LETTER=${1:0:1}
27 |     LAST_LETTER=${2:0:1}
28 | 
29 |     # First name followed by last name
30 |     echo "{first}.{last}|${FIRST_NAME}.${LAST_NAME}@${EMAIL_DOMAIN}"
31 |     echo "{first}_{last}|${FIRST_NAME}_${LAST_NAME}@${EMAIL_DOMAIN}"
32 |     echo "{first}-{last}|${FIRST_NAME}-${LAST_NAME}@${EMAIL_DOMAIN}"
33 | 
34 |     # Last name followed by first name
35 |     echo "{last}.{first}|${LAST_NAME}.${FIRST_NAME}@${EMAIL_DOMAIN}"
36 |     echo "{last}_{first}|${LAST_NAME}_${FIRST_NAME}@${EMAIL_DOMAIN}"
37 |     echo "{last}-{first}|${LAST_NAME}-${FIRST_NAME}@${EMAIL_DOMAIN}"
38 | 
39 |     # Letter followed by name
40 |     echo "{f}{last}|${FIRST_LETTER}${LAST_NAME}@${EMAIL_DOMAIN}"
41 |     echo "{l}{first}|${LAST_LETTER}${FIRST_NAME}@${EMAIL_DOMAIN}"
42 |     
43 |     # Name followed by letter
44 |     echo "{last}{f}|${LAST_NAME}${FIRST_LETTER}@${EMAIL_DOMAIN}"
45 |     echo "{first}{l}|${FIRST_NAME}${LAST_LETTER}@${EMAIL_DOMAIN}"
46 | }
47 | 
48 | while read LINE
49 | do
50 |     # Process an individual line. For the name, generate all possible emails.
51 |     IFS=' ' read -r -a FULL_NAME <<< "${LINE}"
52 |     handle_name "${FULL_NAME[0]}" "${FULL_NAME[1]}"
53 | done < /dev/stdin
54 | 


--------------------------------------------------------------------------------
/user-scan/scripts/strip-name.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | # Strip matched words from names. This is used to remove patterns like:
 4 | # 'Jr.', 'Sr.', 'CPA', etc. from names.
 5 | #
 6 | # The second input (or stdin) is a file containing the words to strip, one per line.
 7 | # The first input is the line to strip from.
 8 | 
 9 | # echo ${TMP//Mr.} | sed 's/^[ \t]*//' | sed 's/[ \t]*$//' | sed 's/  */ /'
10 | 
11 | LINE="$1"
12 | 
13 | while read STRIPPED_WORD
14 | do
15 |     LINE="${LINE//$STRIPPED_WORD}"
16 | done < "${2:-/dev/stdin}"
17 | 
18 | echo $LINE \
19 |     | sed 's/^[ \t]*//' \
20 |     | sed 's/[ \t]*$//' \
21 |     | sed 's/[ \t][ \t]*/ /'
22 | 
23 | 


--------------------------------------------------------------------------------
/user-scan/standard-email-list:
--------------------------------------------------------------------------------
1 | support
2 | info
3 | webmaster
4 | marketing
5 | sales
6 | orders
7 | 


--------------------------------------------------------------------------------
/user-scan/user-scan.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | echo "> Attempting to harvest users for company '$COMPANY_NAME' from LinkedIn."
 4 | 
 5 | $USER_SCAN_SCRIPTS/harvest-linkedin.sh "$COMPANY_NAME" 100 \
 6 |     | $USER_SCAN_SCRIPTS/clean-harvested-names.sh \
 7 |     | uniq -u \
 8 |     > $LINKEDIN_RESULTS
 9 | 
10 | echo -e "\t+ Found $(cat $LINKEDIN_RESULTS | wc -l) users."
11 | echo ""
12 | 
13 | # Generate a list of possible emails from our LinkedIn results.
14 | # See the script for more information, we just combine names in different ways.
15 | echo "> Populating user emails using the domain $EMAIL_DOMAIN"
16 | touch $POSSIBLE_EMAILS
17 | $USER_SCAN_SCRIPTS/possible-emails.sh "$EMAIL_DOMAIN" \
18 |     < $LINKEDIN_RESULTS \
19 |     > $POSSIBLE_EMAILS
20 | 
21 | touch $COMPROMISED_STYLE
22 | if [ ! -z "$SPECIFIED_EMAIL_FORMAT" ]; then
23 |     # The user has specified an email format. Hunter and scanning are both disabled.
24 |     echo -e "> User has specified email style '$SPECIFIED_EMAIL_FORMAT'"
25 |     echo "$SPECIFIED_EMAIL_FORMAT" > $COMPROMISED_STYLE
26 | elif $HUNTER_ENABLED; then
27 |     # The user has chosen to use Hunter. Rather than scan using HIBP we'll
28 |     # just look up the best pattern.
29 |     echo -e "> Attempting to identify compromised email style based on Hunter.io."
30 |     echo -e "> This operation uses the Hunter.io API Key that you supplied and will use its credits."
31 | 
32 |     $USER_SCAN_SCRIPTS/hunter-filter.sh \
33 |         > $COMPROMISED_STYLE
34 | else
35 |     # The user does not want to use Hunter. We'll scan possible emails one-by-one
36 |     # until we encounter a HIBP match and choose that style.
37 |     echo -e "> Attempting to identify compromised email style based on HaveIBeenPwned."
38 |     echo -e "> Scanning a set of $(cat $POSSIBLE_EMAILS | wc -l) potential emails."
39 | 
40 |     $USER_SCAN_SCRIPTS/hibp-filter.sh \
41 |         < $POSSIBLE_EMAILS \
42 |         > $COMPROMISED_STYLE
43 | fi
44 | 
45 | # We have identified a style (or haven't found anything useful) -- if we DO have
46 | # a style of email that seems probable, we want to run an HIBP scan against all
47 | # email addresses of that form.
48 | touch $PROBABLE_EMAILS
49 | touch $COMPROMISED_EMAILS
50 | 
51 | # Append the Battalion standard email list to the probable emails so that they always get
52 | # scanned. It's a small list of non-user accounts that have a good chance of existing.
53 | $USER_SCAN_SCRIPTS/build-emails-from-names.sh $SCRIPT_DIRECTORY/user-scan/standard-email-list > $PROBABLE_EMAILS
54 | 
55 | if [ -s $COMPROMISED_STYLE ]; then
56 |     echo -e "\t+ Identified the style '$(cat $COMPROMISED_STYLE)'\n"
57 | 
58 |     # Filter the possible emails by the pattern that we determined to be the most probable.
59 |     # Next we'll run every email matching the pattern through HIBP in an attempt to identify
60 |     # any compromised emails.
61 |     cat $POSSIBLE_EMAILS \
62 |         | grep -F "$(cat $COMPROMISED_STYLE)" \
63 |         | cut -d\| -f 2 \
64 |         >> $PROBABLE_EMAILS
65 | fi
66 | 
67 | echo -e "> Identified $(cat $PROBABLE_EMAILS | wc -l) probable emails."
68 | echo -e "> Scanning probable emails using HaveIBeenPwned.\n"
69 | 
70 | cat $PROBABLE_EMAILS \
71 |     | $USER_SCAN_SCRIPTS/hibp-scan.sh \
72 |     > $COMPROMISED_EMAILS
73 | 
74 | echo ""
75 | 


--------------------------------------------------------------------------------