├── log
    └── README.md
├── res
    ├── sniff-paste-pic.jpg
    ├── sql-statements.txt
    ├── nmapfilter.conf
    └── regexesToAdd.json
├── requirements.txt
├── .gitignore
├── settings.ini
├── README.md
└── sniff-paste.py


/log/README.md:
--------------------------------------------------------------------------------
1 | Scraper logs located here
2 | 


--------------------------------------------------------------------------------
/res/sniff-paste-pic.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/needmorecowbell/sniff-paste/HEAD/res/sniff-paste-pic.jpg


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | requests==2.10.0
2 | colorlog==2.7.0
3 | lxml==3.4.4
4 | SQLAlchemy==1.2.9
5 | nmap==0.0.1
6 | cssselect==1.0.3
7 | ipaddress==1.0.22
8 | 


--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
 1 | *.log
 2 | *.db
 3 | log/*
 4 | !log/README.md
 5 | out/*.txt
 6 | *.html
 7 | *.xml
 8 | 
 9 | configdebug.py
10 | settings.debug.ini
11 | 
12 | __pycache__/
13 | 
14 | 


--------------------------------------------------------------------------------
/settings.ini:
--------------------------------------------------------------------------------
 1 | [GENERAL]
 2 | PasteLimit = 50
 3 | PBLink = http://pastebin.com/
 4 | DownloadWorkers = 2
 5 | NewPasteCheckInterval = 5
 6 | ConnectionRetryInterval = 30
 7 | IPBlockedWaitTime = 3600
 8 | 
 9 | [LOGGING]
10 | RotationLog = log/pastebin-scraper.log
11 | MaxRotationSize = 2097152
12 | RotationBackupCount = 3
13 | 
14 | [STDOUT]
15 | Enable = yes
16 | ContentDisplayLimit = 100
17 | ShowName = yes
18 | ShowLang = yes
19 | ShowLink = yes
20 | ShowData = yes
21 | DataEncoding = utf-8
22 | 
23 | [MYSQL]
24 | Enable = yes
25 | TableName = sniff_paste
26 | Host = 127.0.0.1
27 | Port = 3306
28 | Username = root
29 | Password = password
30 | 
31 | 


--------------------------------------------------------------------------------
/res/sql-statements.txt:
--------------------------------------------------------------------------------
 1 | sql-statements= {
 2 |     "Top 10 IP Occurences in Pastes":"SELECT ip, COUNT(*) AS magnitude from ips Group By ip Order by magnitude DESC Limit 10",
 3 |     "Top 10 Email Occurences in Pastes":"SELECT email, COUNT(*) AS magnitude from emails Group By phone Order by magnitude DESC Limit 10",
 4 |     "Top 10 Phone Number Occurences in Pastes":"SELECT phone, COUNT(*) AS magnitude from phones Group By phone Order by magnitude DESC Limit 10",
 5 |     "Top 10 URL Occurences in Pastes":"SELECT url, COUNT(*) AS magnitude from links Group By url Order by magnitude DESC Limit 10",
 6 |     "Top 10 Pastes with Highest IP Counts":"SELECT link, COUNT(link) AS frequency FROM ips GROUP BY link ORDER by frequency DESC Limit 10",
 7 |     "Top 10 Pastes with Highest Email Counts":"SELECT link, COUNT(link) AS frequency FROM emails GROUP BY link ORDER by frequency DESC Limit 10",
 8 |     "Top 10 Pastes with Highest URL Counts":"SELECT link, COUNT(link) AS frequency FROM links GROUP BY link ORDER by frequency DESC Limit 10"
 9 | }
10 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | ## Sniff-Paste: OSINT Pastebin Harvester
 2 | 
 3 | <p align="center">
 4 |     <img src="res/sniff-paste-pic.jpg" width="400"></img>
 5 | </p>
 6 | 
 7 | Multithreaded pastebin scraper, scrapes to mysql database, then reads pastes for noteworthy information.
 8 | 
 9 | Use sniff-paste.py  to go through the entire process of collection, logging, and harvest automatically. The scraper can be set to a paste limit of 0 to scrape indefinitely. If scraped indefinitely, press ctrl + c to stop scraping, any useful information will be in the database, along with a link back to the original paste it was found in.
10 | 
11 | 
12 | ## Installation
13 | 
14 | `sudo apt install libxslt-dev python3-lxml python3-nmap xsltproc mysql-server`
15 | 
16 | `pip3 install -r requirements.txt`
17 | 
18 |  - Create database named `sniff_paste` in mysql server
19 |  - Fill in settings.ini
20 | 
21 | `python3 sniff-paste.py`
22 | 
23 | This will scrape pastebin for the latest number of pastes, then run analysis for ip addresses, emails, and phone numbers. It filters out duplicates and runs scans on some of the harvested data.
24 | 
25 | ## Database Structure 
26 | - `sniff_paste` -- root db
27 | 	- `pastes` -- stores paste with full text, date, link, title, and language
28 | 	- `emails` -- stores emails with extension to paste
29 | 	- `links` -- stores urls with extension to paste
30 | 	- `ip` -- stores ip with connectivity and extension to paste
31 | 	- `phones` -- stores phone numbers with extension to paste
32 | 	- `secrets` -- stores secret type with extension to paste
33 | 	- `ports` -- stores port scan info (port, status, service, version, ip)
34 | 	- `cryptos` -- stores cryptocurrency findings with extension to paste 
35 | 
36 | **Crypto findings are not certain to be valid, consider them low probability findings**
37 | 
38 | 
39 | ## Notes
40 | 
41 | - Please contribute! If there's an error let me know -- even better if you can fix it :)
42 | 	- Regex Contributions would be very helpful, and should be pretty easy to add!
43 | - This tool is in the process of a bigger update, where the scraper can send all new pastes to my new project needmorecowbell/Funnel. I'm trying to consolidate all of my osint tools into one streamlined solution.
44 | 


--------------------------------------------------------------------------------
/res/nmapfilter.conf:
--------------------------------------------------------------------------------
  1 | 0.0.0.0/8
  2 | 10.0.0.0/8
  3 | 100.64.0.0/10
  4 | 127.0.0.0/8
  5 | 169.254.0.0/16
  6 | 172.16.0.0/12
  7 | 192.0.0.0/24
  8 | 192.0.0.0/29
  9 | 192.0.0.170/32
 10 | 192.0.0.171/32
 11 | 192.0.2.0/24
 12 | 192.88.99.0/24
 13 | 192.168.0.0/16
 14 | 198.18.0.0/15
 15 | 198.51.100.0/24
 16 | 203.0.113.0/24
 17 | 240.0.0.0/4
 18 | 255.255.255.255/32
 19 | 153.11.0.0/16
 20 | 4.53.201.0/24
 21 | 5.152.179.0/24
 22 | 8.12.162.0/24
 23 | 8.12.163.0/24
 24 | 8.12.164.0/24
 25 | 8.14.84.0/22
 26 | 8.14.145.0/24
 27 | 8.14.146.0/24
 28 | 8.14.147.0/24
 29 | 8.17.250.0/24
 30 | 8.17.251.0/24
 31 | 8.17.252.0/24
 32 | 23.27.0.0/16
 33 | 23.231.128.0/17
 34 | 37.72.172.0/23
 35 | 38.72.200.0/22
 36 | 50.93.192.0/24
 37 | 50.93.193.0/24
 38 | 50.93.194.0/24
 39 | 50.93.195.0/24
 40 | 50.93.196.0/24
 41 | 50.93.197.0/24
 42 | 50.115.128.0/20
 43 | 50.117.0.0/17
 44 | 50.118.128.0/17
 45 | 63.141.222.0/24
 46 | 64.62.253.0/24
 47 | 64.92.96.0/19
 48 | 64.145.79.0/24
 49 | 64.145.82.0/23
 50 | 64.158.146.0/23
 51 | 65.49.24.0/24
 52 | 65.49.93.0/24
 53 | 65.162.192.0/22
 54 | 66.79.160.0/19
 55 | 66.160.191.0/24
 56 | 68.68.96.0/20
 57 | 69.46.64.0/19
 58 | 69.176.80.0/20
 59 | 72.13.80.0/20
 60 | 72.52.76.0/24
 61 | 74.82.43.0/24
 62 | 74.82.160.0/19
 63 | 74.114.88.0/22
 64 | 74.115.0.0/24
 65 | 74.115.2.0/24
 66 | 74.115.4.0/24
 67 | 74.122.100.0/22
 68 | 75.127.0.0/24
 69 | 103.251.91.0/24
 70 | 108.171.32.0/24
 71 | 108.171.42.0/24
 72 | 108.171.52.0/24
 73 | 108.171.62.0/24
 74 | 118.193.78.0/23
 75 | 130.93.16.0/23
 76 | 136.0.0.0/16
 77 | 142.111.0.0/16
 78 | 142.252.0.0/16
 79 | 146.82.55.93
 80 | 149.54.136.0/21
 81 | 149.54.152.0/21
 82 | 166.88.0.0/16
 83 | 172.252.0.0/16
 84 | 173.245.64.0/19
 85 | 173.245.194.0/23
 86 | 173.245.220.0/22
 87 | 173.252.192.0/18
 88 | 178.18.16.0/22
 89 | 178.18.26.0/24
 90 | 178.18.27.0/24
 91 | 178.18.28.0/24
 92 | 178.18.29.0/24
 93 | 183.182.22.0/24
 94 | 192.92.114.0/24
 95 | 192.155.160.0/19
 96 | 192.177.0.0/16
 97 | 192.186.0.0/18
 98 | 192.249.64.0/20
 99 | 192.250.240.0/20
100 | 194.110.214.0/24
101 | 198.12.120.0/24
102 | 198.12.121.0/24
103 | 198.12.122.0/24
104 | 198.144.240.0/20
105 | 199.33.120.0/24
106 | 199.33.124.0/22
107 | 199.48.147.0/24
108 | 199.68.196.0/22
109 | 199.127.240.0/21
110 | 199.187.168.0/22
111 | 199.188.238.0/23
112 | 199.255.208.0/24
113 | 203.12.6.0/24
114 | 204.13.64.0/21
115 | 204.16.192.0/21
116 | 204.19.238.0/24
117 | 204.74.208.0/20
118 | 205.159.189.0/24
119 | 205.164.0.0/18
120 | 205.209.128.0/18
121 | 206.108.52.0/23
122 | 206.165.4.0/24
123 | 208.77.40.0/21
124 | 208.80.4.0/22
125 | 208.123.223.0/24
126 | 209.51.185.0/24
127 | 209.54.48.0/20
128 | 209.107.192.0/23
129 | 209.107.210.0/24
130 | 209.107.212.0/24
131 | 211.156.110.0/23
132 | 216.83.33.0/24
133 | 216.83.34.0/24
134 | 216.83.35.0/24
135 | 216.83.36.0/24
136 | 216.83.37.0/24
137 | 216.83.38.0/24
138 | 216.83.39.0/24
139 | 216.83.40.0/24
140 | 216.83.42.0/24
141 | 216.83.43.0/24
142 | 216.83.44.0/24
143 | 216.83.45.0/24
144 | 216.83.46.0/24
145 | 216.83.47.0/24
146 | 216.83.48.0/24
147 | 216.83.49.0/24
148 | 216.83.51.0/24
149 | 216.83.52.0/24
150 | 216.83.53.0/24
151 | 216.83.54.0/24
152 | 216.83.55.0/24
153 | 216.83.56.0/24
154 | 216.83.57.0/24
155 | 216.83.58.0/24
156 | 216.83.59.0/24
157 | 216.83.60.0/24
158 | 216.83.61.0/24
159 | 216.83.62.0/24
160 | 216.83.63.0/24
161 | 216.151.183.0/24
162 | 216.151.190.0/23
163 | 216.172.128.0/19
164 | 216.185.36.0/24
165 | 216.218.233.0/24
166 | 216.224.112.0/20
167 | 194.77.40.242
168 | 194.77.40.246
169 | 165.160.0.0/16
170 | 129.123.0.0/16
171 | 144.39.0.0/16
172 | 204.113.91.0/24
173 | 


--------------------------------------------------------------------------------
/res/regexesToAdd.json:
--------------------------------------------------------------------------------
 1 | "ip" : {
 2 |     "ipv6": "i/^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$/",
 3 | 
 4 |     "ipv4": "/^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/"
 5 | },
 6 | 
 7 | "links": {
 8 |     "youtube-channel":"/https?:\/\/(www\.)?youtube.com\/channel\/UC([-_a-z0-9]{22})/i",
 9 |     "youtube-video":"/https?:\/\/(?:youtu\.be\/|(?:[a-z]{2,3}\.)?youtube\.com\/watch(?:\?|#\!)v=)([\w-]{11}).*/gi",
10 |     "onion-link":"(?:^[a-z2-7]{16}\.onion$)|(?:^[a-z2-7]{56}\.onion$)",
11 |     "https":"https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+",
12 |     "http":"http?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+"
13 | },
14 | 
15 | "general":{
16 |     "street":"\d{1,4} [\w\s]{1,20}(?:street|st|avenue|ave|road|rd|highway|hwy|square|sq|trail|trl|drive|dr|court|ct|park|parkway|pkwy|circle|cir|boulevard|blvd)\W?(?=\s|$)",
17 |     "credit-cards":"((?:(?:\\d{4}[- ]?){3}\\d{4}|\\d{15,16}))(?![\\d])",
18 |     "emails":"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+"
19 | 
20 | },
21 | 
22 | "secrets" : {                                                                                                                                               
23 |    "Slack Token": "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})",                                                                                   
24 |    "RSA private key": "-----BEGIN RSA PRIVATE KEY-----",                                                                                                         
25 |    "SSH (OPENSSH) private key": "-----BEGIN OPENSSH PRIVATE KEY-----",                                                                                           
26 |    "SSH (DSA) private key": "-----BEGIN DSA PRIVATE KEY-----",                                                                                                  
27 |    "SSH (EC) private key": "-----BEGIN EC PRIVATE KEY-----",                                                                                                 
28 |    "PGP private key block": "-----BEGIN PGP PRIVATE KEY BLOCK-----",                                                                                         
29 |    "Facebook Oauth": "[f|F][a|A][c|C][e|E][b|B][o|O][o|O][k|K].*['|\"][0-9a-f]{32}['|\"]",                                                                 
30 |    "Twitter Oauth": "[t|T][w|W][i|I][t|T][t|T][e|E][r|R].*['|\"][0-9a-zA-Z]{35,44}['|\"]",                                                                
31 |    "GitHub": "[g|G][i|I][t|T][h|H][u|U][b|B].*[['|\"]0-9a-zA-Z]{35,40}['|\"]",                                                                              
32 |    "Google Oauth": "(\"client_secret\":\"[a-zA-Z0-9-_]{24}\")",                                                                                             
33 |    "AWS API Key": "AKIA[0-9A-Z]{16}",                                                                                                                         
34 |    "Heroku API Key": "[h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}",                                           
35 |    "Generic Secret": "[s|S][e|E][c|C][r|R][e|E][t|T].*['|\"][0-9a-zA-Z]{32,45}['|\"]"                                                                         
36 | },
37 | 
38 | "cryptos":{                                                                                                                                                   
39 |    "bitcoin-address" : "[13][a-km-zA-HJ-NP-Z1-9]{25,34}" ,                                                                                                       
40 |    "bitcoin-uri" : "bitcoin:([13][a-km-zA-HJ-NP-Z1-9]{25,34})" ,                                                                                                 
41 |    "bitcoin-xpub-key" : "(xpub[a-km-zA-HJ-NP-Z1-9]{100,108})(\\?c=\\d*&h=bip\\d{2,3})?" ,                                                                       
42 |    "monero-address": "(?:^4[0-9AB][1-9A-HJ-NP-Za-km-z]{93}$)",                                                                                                   
43 |    "ethereum-address": "(?:^0x[a-fA-F0-9]{40}$)",                                                                                                                
44 |    "litecoin-address":"(?:^[LM3][a-km-zA-HJ-NP-Z1-9]{26,33}$)",                                                                                                  
45 |    "bitcoin-cash-address":"(?:^[13][a-km-zA-HJ-NP-Z1-9]{33}$)",                                                                                                  
46 |    "dash-address":"(?:^X[1-9A-HJ-NP-Za-km-z]{33}$)",                                                                                                             
47 |    "ripple-address":"(?:^r[0-9a-zA-Z]{33}$)",                                                                                                                    
48 |    "neo-address":"(?:^A[0-9a-zA-Z]{33}$)",                                                                                                                       
49 |    "dogecoin-address":"(?:^D{1}[5-9A-HJ-NP-U]{1}[1-9A-HJ-NP-Za-km-z]{32}$)"                                                                                      
50 | }
51 | 


--------------------------------------------------------------------------------
/sniff-paste.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python3
  2 | import json
  3 | from ipaddress import IPv4Address as IPv4
  4 | from ipaddress import IPv4Network as IPv4Net
  5 | import logging
  6 | import socket
  7 | import logging.handlers
  8 | import os
  9 | import sys
 10 | import threading
 11 | import time
 12 | from datetime import datetime
 13 | from os import path
 14 | import re
 15 | import requests
 16 | from lxml import html
 17 | 
 18 | import nmap
 19 | 
 20 | import configparser
 21 | import queue
 22 | from colorlog import ColoredFormatter
 23 | 
 24 | from sqlalchemy import Column, Integer, String, DateTime, Boolean
 25 | from sqlalchemy.dialects.mysql import LONGTEXT
 26 | 
 27 | 
 28 | debug = False
 29 | 
 30 | IPStack = []
 31 | 
 32 | 
 33 | secretRegexes = {
 34 |     "Slack Token": "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})",
 35 |     "RSA private key": "-----BEGIN RSA PRIVATE KEY-----",
 36 |     "SSH (OPENSSH) private key": "-----BEGIN OPENSSH PRIVATE KEY-----",
 37 |     "SSH (DSA) private key": "-----BEGIN DSA PRIVATE KEY-----",
 38 |     "SSH (EC) private key": "-----BEGIN EC PRIVATE KEY-----",
 39 |     "PGP private key block": "-----BEGIN PGP PRIVATE KEY BLOCK-----",
 40 |     "Facebook Oauth": "[f|F][a|A][c|C][e|E][b|B][o|O][o|O][k|K].*['|\"][0-9a-f]{32}['|\"]",
 41 |     "Twitter Oauth": "[t|T][w|W][i|I][t|T][t|T][e|E][r|R].*['|\"][0-9a-zA-Z]{35,44}['|\"]",
 42 |     "GitHub": "[g|G][i|I][t|T][h|H][u|U][b|B].*[['|\"]0-9a-zA-Z]{35,40}['|\"]",
 43 |     "Google Oauth": "(\"client_secret\":\"[a-zA-Z0-9-_]{24}\")",
 44 |     "AWS API Key": "AKIA[0-9A-Z]{16}",
 45 |     "Heroku API Key": "[h|H][e|E][r|R][o|O][k|K][u|U].*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}",
 46 |     "Generic Secret": "[s|S][e|E][c|C][r|R][e|E][t|T].*['|\"][0-9a-zA-Z]{32,45}['|\"]"
 47 | }
 48 | 
 49 | cryptoRegexes={
 50 |     "bitcoin-address" : "[13][a-km-zA-HJ-NP-Z1-9]{25,34}" ,
 51 |     "bitcoin-uri" : "bitcoin:([13][a-km-zA-HJ-NP-Z1-9]{25,34})" ,
 52 |     "bitcoin-xpub-key" : "(xpub[a-km-zA-HJ-NP-Z1-9]{100,108})(\\?c=\\d*&h=bip\\d{2,3})?" ,
 53 |     "monero-address": "(?:^4[0-9AB][1-9A-HJ-NP-Za-km-z]{93}$)",
 54 |     "ethereum-address": "(?:^0x[a-fA-F0-9]{40}$)",
 55 |     "litecoin-address":"(?:^[LM3][a-km-zA-HJ-NP-Z1-9]{26,33}$)",
 56 |     "bitcoin-cash-address":"(?:^[13][a-km-zA-HJ-NP-Z1-9]{33}$)",
 57 |     "dash-address":"(?:^X[1-9A-HJ-NP-Za-km-z]{33}$)",
 58 |     "ripple-address":"(?:^r[0-9a-zA-Z]{33}$)",
 59 |     "neo-address":"(?:^A[0-9a-zA-Z]{33}$)",
 60 |     "dogecoin-address":"(?:^D{1}[5-9A-HJ-NP-U]{1}[1-9A-HJ-NP-Za-km-z]{32}$)"
 61 | }
 62 | 
 63 | class PasteDBConnector(object):
 64 |     supported = ('MYSQL')
 65 | 
 66 |     def __init__(self, db, **kwargs):
 67 |         try:
 68 |             self.logger = logging.getLogger('pastebin-scraper')
 69 |             from sqlalchemy.ext.declarative import declarative_base
 70 |         except ImportError:
 71 |             self.logger.error('SQLAlchemy import failed. Make sure the SQLAlchemy Python library '
 72 |                               'is installed! To check your existing installation run: '
 73 |                               'python3 -c "import sqlalchemy;print(sqlalchemy.__version__)"')
 74 |         self.db = db
 75 |         self.Base = declarative_base()
 76 |         self.engine = self._get_db_engine(**kwargs)
 77 |         self.session = self._get_db_session(self.engine)
 78 | 
 79 |         #Create tables for credentials
 80 |         self.paste_model = self._get_paste_model(self.Base, **kwargs)
 81 |         self.email_model = self._get_email_model(self.Base)
 82 |         self.link_model = self._get_link_model(self.Base)
 83 |         self.ip_model = self._get_ip_model(self.Base)
 84 |         self.phone_model = self._get_phone_model(self.Base)
 85 |         self.secret_model = self._get_secret_model(self.Base)
 86 |         self.crypto_model = self._get_crypto_model(self.Base)
 87 |         self.port_model = self._get_port_model(self.Base)
 88 |         self.Base.metadata.create_all(self.engine)
 89 | 
 90 |         #Nmap Worker
 91 |             
 92 |         nmapper = threading.Thread(target=self._scan_network)
 93 |         nmapper.setDaemon(True)
 94 |         nmapper.start()
 95 | 
 96 |     def isFiltered(self,ip):
 97 |         with open("res/nmapfilter.conf") as ipFilters:
 98 |             for networkString in ipFilters:
 99 |                 ip=IPv4(ip)
100 |                 network=IPv4Net(networkString.rstrip("\r\n"))
101 |                 if(ip in network):
102 | 
103 |                     self.logger.debug("["+str(ip)+"] is a filtered address, do not scan.")
104 |                     return True
105 |         return False
106 | 
107 | 
108 |     def _get_db_engine(self, **kwargs):
109 |         from sqlalchemy import create_engine
110 | 
111 |         if self.db == 'MYSQL':
112 |             # use the mysql-python connector
113 |             location = 'mysql+pymysql://'
114 |             location += '{username}:{password}@{host}:{port}'.format(
115 |                 host=kwargs.pop('host'),
116 |                 port=kwargs.pop('port'),
117 |                 username=kwargs.pop('username'),
118 |                 password=kwargs.pop('password'),
119 |             )
120 |             location += '/{table_name}?charset={charset}'.format(
121 |                 table_name=kwargs.pop('table_name'),
122 |                 charset='utf8'
123 |             )
124 |             
125 |             self.logger.info('Using MySQL')
126 |             return create_engine(location)
127 |         
128 |     def _get_db_session(self, engine):
129 |         from sqlalchemy.orm import sessionmaker
130 |         return sessionmaker(bind=engine)()
131 | 
132 |     def _get_paste_model(self, base, **kwargs):
133 |         class Paste(base):
134 |             __tablename__ = "pastes"
135 | 
136 |             id = Column(Integer, primary_key=True)
137 |             name = Column('name', String(60))
138 |             lang = Column('language', String(30))
139 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
140 |             date = Column('date', DateTime())
141 |             
142 |             data = Column('data', LONGTEXT(charset='utf8'))
143 |       
144 |             def __repr__(self):
145 |                 return "<Paste(id=%s, name='%s', lang='%s', link='%s', date='%s', data='%s')" %\
146 |                        (self.id,
147 |                         self.name,
148 |                         self.lang,
149 |                         self.link,
150 |                         str(self.date),
151 |                         self.data[:10])
152 |         return Paste
153 | 
154 |     def _get_email_model(self, base):
155 |         class Email(base):
156 |             __tablename__ = "emails"
157 |             id = Column(Integer, primary_key=True)
158 |             email = Column('email', String(90))
159 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
160 |       
161 |             def __repr__(self):
162 |                 return "<Email(id=%s, email='%s', link='%s')" %\
163 |                        (self.id,
164 |                         self.email,
165 |                         self.link)
166 |         return Email
167 | 
168 |     def _get_link_model(self, base):
169 |         class Link(base):
170 |             __tablename__ = "links"
171 | 
172 |             id = Column(Integer, primary_key=True)
173 |             url = Column('url', String(60))
174 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
175 |       
176 |             def __repr__(self):
177 |                 return "<Link(id=%s, url='%s', link='%s')" %\
178 |                        (self.id,
179 |                         self.url,
180 |                         self.link)
181 |         return Link
182 | 
183 |     def _get_phone_model(self, base):
184 |         class Phone(base):
185 |             __tablename__ = "phones"
186 | 
187 |             id = Column(Integer, primary_key=True)
188 |             phone = Column('phone', String(60))
189 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
190 |       
191 |             def __repr__(self):
192 |                 return "<Phone(id=%s, phone='%s', link='%s')" %\
193 |                        (self.id,
194 |                         self.phone,
195 |                         self.link)
196 |         return Phone
197 | 
198 |     def _get_secret_model(self, base):
199 |         class Secret(base):
200 |             __tablename__ = "secrets"
201 | 
202 |             id = Column(Integer, primary_key=True)
203 |             secret = Column('secret', String(60))
204 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
205 |       
206 |             def __repr__(self):
207 |                 return "<Secret(id=%s, secret='%s', link='%s')" %\
208 |                        (self.id,
209 |                         self.secret,
210 |                         self.link)
211 |         return Secret
212 | 
213 |     def _get_crypto_model(self, base):
214 |         class Crypto(base):
215 |             __tablename__ = "cryptos"
216 | 
217 |             id = Column(Integer, primary_key=True)
218 |             genre = Column('genre', String(60))
219 |             content = Column('content', String(60))
220 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
221 |       
222 |             def __repr__(self):
223 |                 return "<Crypto(id=%s, genre='%s', content='%s', link='%s' )" %\
224 |                        (self.id,
225 |                         self.genre,
226 |                         self.content,
227 |                         self.link
228 |                        )
229 |         return Crypto
230 | 
231 | 
232 | 
233 |     def _get_port_model(self, base):
234 |         class Port(base):
235 |             __tablename__ = "ports"
236 | 
237 |             id = Column(Integer, primary_key=True)
238 |             ip = Column('ip', String(60))
239 |             port = Column('port', String(10))      
240 |             service = Column('service', String(60))
241 |             status = Column('status', String(40))
242 |             version= Column('version', String(60))
243 |             def __repr__(self):
244 |                 return "<Port(id=%s, ip='%s', port='%s', service='%s', status='%s', version='%s')" %\
245 |                     (self.id,
246 |                      self.ip,
247 |                      self.port,
248 |                      self.service,
249 |                      self.status,
250 |                      self.version
251 |                     )
252 |         return Port
253 | 
254 |     def _get_ip_model(self, base):
255 |         class IP(base):
256 |             __tablename__ = "ips"
257 | 
258 |             id = Column(Integer, primary_key=True)
259 |             ip = Column('ip', String(60))
260 |             online = Column('online', Boolean())      
261 |             link = Column('link', String(28))  # Assuming format http://pastebin.com/XXXXXXXX
262 |             def __repr__(self):
263 |                 return "<IP(id=%s, ip='%s', online='%s', link='%s')" %\
264 |                        (self.id,
265 |                         self.ip,
266 |                         self.online,
267 |                         self.link)
268 |         return IP
269 | 
270 |     def harvest(self,pasteLink, data):
271 |         
272 |         print("Harvesting Data From: "+pasteLink)
273 |         ips = re.findall( r'[0-9]+(?:\.[0-9]+){3}', str(data ) )
274 |         emails = re.findall( r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+", str(data ) )
275 |         phoneNumbers= re.findall( r"\(?\b[2-9][0-9]{2}\)?[-. ]?[2-9][0-9]{2}[-. ]?[0-9]{4}\b", str(data))
276 | 
277 |         urls = re.findall( 'https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+', str(data))
278 |         urls += re.findall( 'http?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+', str(data))
279 |         
280 |         secrets = None
281 | 
282 |         
283 |         # Remove all duplicates
284 |         ips = list(set(ips))
285 |         emails= list(set(emails))
286 |         phoneNumbers = list(set(phoneNumbers))
287 |         urls= list(set(urls))
288 | 
289 | 
290 |         for finding in urls:
291 |             url_model = self.link_model(
292 |                 url=finding,
293 |                 link=pasteLink
294 |             )
295 |             try:
296 |                 self.session.add(url_model)
297 |                 self.session.commit()
298 |             except:
299 |                 print("Error: Url model not committed, session rollback")
300 |                 self.session.rollback()
301 |             #ADD TO SQL +=url_model
302 |  
303 |         for finding in ips:
304 |             #print("IP: "+finding)
305 |             try:
306 |                 socket.inet_aton(finding)
307 |                 IPStack.append((finding, pasteLink))
308 |             except socket.error:
309 |                 print("Invalid ip: "+finding)
310 |                 #ret = os.system("ping -o -c 2 -W 500 "+finding)
311 |                 #pingRes = ret != 0 
312 | 
313 |         for finding in emails:
314 |             email_model = self.email_model(
315 |                 email=finding,
316 |                 link=pasteLink
317 |             )
318 |             try:
319 |                 self.session.add(email_model)
320 |                 self.session.commit()
321 |             except:
322 |                 print("Error: Email model not committed, session rollback")
323 |                 self.session.rollback()
324 | 
325 |         for finding in phoneNumbers:
326 |             try:
327 |                 if( ' ' in finding or '-' in finding or '(' in finding or ')' in finding):
328 |                     phone_model = self.phone_model(
329 |                         phone=finding,
330 |                         link=pasteLink
331 |                     )
332 |                     try:
333 |                         self.session.add(phone_model)
334 |                         self.session.commit()
335 |                     except Exception as e:
336 |                         print("Error: Phone model not committed, session rollback: "+str(e))
337 |                         self.session.rollback()
338 | 
339 |             except:
340 |                print("Error: Phone number submission")  
341 | 
342 |         totalSecrets= 0 
343 |         for key, value in secretRegexes.items():
344 |             secrets = re.findall(value, str(data))
345 |             
346 |             for secret in secrets:
347 |                 totalSecrets+=1
348 |                 print("\t"+key+": "+secret)
349 |                 secret_model = self.secret_model(
350 |                     secret=key,
351 |                     link=pasteLink
352 |                 )
353 |                 try:
354 |                     self.session.add(secret_model)
355 |                     self.session.commit()
356 |                 except:
357 |                     print("Error: Secret model not committed, session rollback")
358 |                     self.session.rollback()
359 | 
360 | 
361 |         totalCryptoFindings= 0 
362 |         for key, value in cryptoRegexes.items():
363 |             findings = re.findall(value, str(data))
364 |             
365 |             for finding in findings:
366 |                 totalCryptoFindings+=1
367 |                 crypto_model = self.crypto_model(
368 |                     genre=key,
369 |                     content=finding,
370 |                     link=pasteLink
371 |                 )
372 |                 try:
373 |                     self.session.add(crypto_model)
374 |                     self.session.commit()
375 |                 except:
376 |                     print("Error: Crypto model not committed, session rollback")
377 |                     self.session.rollback()
378 | 
379 | 
380 | 
381 |                    
382 |         print("IPS: "+str(len(ips)))
383 |         print("Emails: "+str(len(emails)))
384 |         print("URLS: "+str(len(urls)))
385 |         print("Phones: "+str(len(phoneNumbers)))
386 |         print("Secrets: "+str(totalSecrets))
387 |         print("Cryptos: "+str(totalCryptoFindings))
388 |         print("\n\n")
389 | 
390 |     def add(self, paste, data):
391 |         paste_model = self.paste_model(
392 |             name=paste[0],
393 |             lang=paste[1],
394 |             link=paste[2],
395 |             date=datetime.now(),
396 |             data=data.content.replace(b'\\', b'\\\\').decode('unicode-escape')
397 |         )
398 | 
399 |         #Start Harvest Here
400 |         self.logger.debug('Harvesting From model ' + str(paste_model))
401 |         self.harvest(paste[2], data.content.replace(b'\\', b'\\\\').decode('unicode-escape'))
402 | 
403 |         try:
404 |             self.session.add(paste_model)
405 |             self.session.commit()
406 |         except:
407 |             self.logger.error(
408 |                 'An error occurred while adding a paste to %s: %s' %
409 |                 (self.db, sys.exc_info()[0])
410 |             )
411 | 
412 |     def _scan_network(self):
413 |         
414 |         nm = nmap.PortScanner()
415 | 
416 |         while True:
417 | 
418 |             if(IPStack): #Check for connectivity, parse nmap if valid
419 | 
420 |                 finding, pasteLink = IPStack.pop()
421 |                 
422 |                 print("NMAP WORKER CALLED ["+finding+"]")
423 |                 print("Left on stack: " +str(len(IPStack)))
424 |                 if(self.isFiltered(finding)==False):
425 | 
426 |                     self.logger.debug('Nmap scan on IP: ' + finding)
427 |                     try:
428 |                         nm.scan(finding, arguments='-sV')
429 |                     except Exception as e:
430 |                         print("Nmap scan error: "+str(e))
431 | 
432 |                     print("Nmap Scan ["+finding+"] Complete")
433 |                     try:
434 |                         state= nm[finding]['status']['state']
435 |                     except:
436 |                         state= "down"
437 | 
438 |                     if("up" in state):
439 |                         portScan= nm[finding]['tcp']
440 | 
441 |                 
442 |                         self.logger.debug(finding+": up\tports: "+str(portScan))
443 |                         for key, value in portScan.items():
444 |                             print(finding+":"+str(key))
445 | 
446 |                             product= value['product']
447 |                             ver= value['version']
448 |                             state= value['state']
449 |                             name= value['name']
450 | 
451 |                             try:
452 |                  
453 |                                 print("Name: "+name+"\n\tProduct: "+product+"\n\tVersion: "+ver+"\n\tState: "+state)
454 | 
455 |                                 port_model = self.port_model(
456 |                                     ip = finding,
457 |                                     port = key,
458 |                                     service = product,
459 |                                     status = state,
460 |                                     version = ver
461 |                                 )
462 | 
463 |                                 try:
464 |                                     self.session.add(port_model)
465 |                                     self.session.commit()
466 |                                 except:
467 |                                     self.session.rollback()
468 |                                     print("Error: Port Commit Issue, session rollback")
469 |                             except Exception as e:
470 |                                 print("Exception error while pushing port model: "+str(e))
471 | 
472 |                             try:
473 |                                 ip_model = self.ip_model(
474 |                                 ip=finding,
475 |                                 online= True,
476 |                                 link=pasteLink
477 |                                 )
478 |                                 try:
479 |                                     self.session.add(ip_model)
480 |                                     self.session.commit()
481 |                                 except:
482 |                                     self.session.rollback()
483 |                                     print("Error pushing to mysql: "+ str(e))
484 |                             except:
485 |                                 print("Error: IP Model issue")
486 | 
487 |                     else: #Else put a normal entry in for pastebin
488 |                         self.logger.debug("IP["+finding+"] down")
489 |                         try:
490 |                             ip_model = self.ip_model(
491 |                                 ip=finding,
492 |                                 online= False,
493 |                                 link=pasteLink
494 |                             )
495 |                             try:
496 |                                 self.session.add(ip_model)
497 |                                 self.session.commit()
498 |                             except:
499 |                                 self.session.rollback()
500 |                                 print("Error: IP Commit Issue, session rollback")
501 | 
502 |                         except Exception as e:
503 |                             print("Error pushing to mysql: "+ str(e))                    
504 | 
505 |         else: #Else put a nmap filtered kentry for pastebin
506 |             self.logger.debug("IP["+finding+"] is a filtered address")
507 |             try:
508 |                 ip_model = self.ip_model(
509 |                     ip=finding,
510 |                     online= False,
511 |                     link=pasteLink
512 |                 )
513 |                 try:
514 |                     self.session.add(ip_model)
515 |                     self.session.commit()
516 |                 except:
517 |                     self.session.rollback()
518 |                     print("Error: IP Commit Issue, session rollback")
519 | 
520 |             except Exception as e:
521 |                 print("Error pushing to mysql: "+ str(e))                    
522 | 
523 | 
524 | 
525 | class PastebinScraper(object):
526 |     def __init__(self):
527 |         # Read and split config
528 |         self.config = configparser.ConfigParser()
529 |         if(debug):
530 |             self.config.read('settings.debug.ini')
531 |         else:
532 |             self.config.read('settings.ini')
533 | 
534 |         self.conf_general = self.config['GENERAL']
535 |         self.conf_logging = self.config['LOGGING']
536 |         self.conf_stdout = self.config['STDOUT']
537 |         self.conf_mysql = self.config['MYSQL']
538 | 
539 |         # Internals
540 |         self.unlimited_pastes = self.conf_general.getint('PasteLimit') == 0
541 |         self.pastes = queue.Queue(maxsize=8)
542 |         self.pastes_seen = set()
543 | 
544 |         # Init the logger
545 |         self.logger = logging.getLogger('pastebin-scraper')
546 |         self.logger.setLevel(logging.DEBUG)
547 | 
548 |         # Set up log rotation
549 |         rotation = logging.handlers.RotatingFileHandler(
550 |             filename=self.conf_logging['RotationLog'],
551 |             maxBytes=self.conf_logging.getint('MaxRotationSize'),
552 |             backupCount=self.conf_logging.getint('RotationBackupCount')
553 |         )
554 |         rotation.setLevel(logging.DEBUG)
555 |         formatter = logging.Formatter('%(asctime)s|%(levelname)-8s| %(message)s')
556 |         rotation.setFormatter(formatter)
557 |         self.logger.addHandler(rotation)
558 | 
559 |         console = logging.StreamHandler()
560 |         console.setLevel(logging.INFO)
561 |         formatter = ColoredFormatter(
562 |             '%(log_color)s%(asctime)s|[%(levelname)-4s] %(message)s%(reset)s', '%H:%M:%S'
563 |         )
564 |         console.setFormatter(formatter)
565 |         self.logger.addHandler(console)
566 | 
567 |         if not (self.conf_stdout.getboolean('Enable') or self.conf_mysql.getboolean('Enable')):
568 |             self.logger.error('No output method specified! Please set at least one output method '
569 |                               'in the settings.ini to \'yes\'.')
570 |             raise RuntimeError('No output method specified!')
571 | 
572 | 
573 |         # DB connectors if needed
574 |         self.mysql_conn = None
575 |         if self.conf_mysql.getboolean('Enable'):
576 |             self.logger.debug('Initializing MySQL connector')
577 |             self.mysql_conn = PasteDBConnector(
578 |                 db='MYSQL',
579 |                 host=self.conf_mysql['Host'],
580 |                 port=self.conf_mysql['Port'],
581 |                 username=self.conf_mysql['Username'],
582 |                 password=self.conf_mysql['Password'],
583 |                 table_name=self.conf_mysql['TableName']
584 |             )
585 |      
586 |     def _get_paste_data(self):
587 |         paste_limit = self.conf_general.getint('PasteLimit')
588 |         pb_link = self.conf_general['PBLINK']
589 |         paste_counter = 0
590 |         self.logger.info('No scrape limit set - scraping indefinitely' if self.unlimited_pastes
591 |                         else 'Paste limit: ' + str(paste_limit))
592 | 
593 |         while self.unlimited_pastes or (paste_counter < paste_limit):
594 |             page = self._handle_data_download(pb_link)
595 | 
596 |             self.logger.debug('Got {} - {} from {}'.format(
597 |                 page.status_code,
598 |                 page.reason,
599 |                 pb_link
600 |             ))
601 |             tree = html.fromstring(page.content)
602 |             pastes = tree.cssselect('ul.right_menu li')
603 |             for paste in pastes:
604 |                 if not self.unlimited_pastes \
605 |                    and (paste_counter >= paste_limit):
606 |                     # Break for limits % 8 != 0
607 |                     break
608 |                 name_link = paste.cssselect('a')[0]
609 |                 name = name_link.text_content().strip()
610 |                 href = name_link.get('href')[1:]  # Get rid of leading /
611 |                 data = paste.cssselect('span')[0].text_content().split('|')
612 |                 language = None
613 |                 if len(data) == 2:
614 |                     # Got language
615 |                     language = data[0].strip()
616 |                 paste_data = (name, language, href)
617 |                 self.logger.debug('Paste scraped: ' + str(paste_data))
618 |                 if paste_data[2] not in self.pastes_seen:
619 |                     # New paste detected
620 |                     self.logger.debug('Scheduling new paste:' + str(paste_data))
621 |                     self.pastes_seen.add(paste_data[2])
622 |                     self.pastes.put(paste_data)
623 |                     delay = self.conf_general.getint('NewPasteCheckInterval')
624 |                     time.sleep(delay)
625 |                     paste_counter += 1
626 |                     self.logger.debug('Paste counter now at ' + str(paste_counter))
627 |                     if paste_counter % 100 == 0:
628 |                         self.logger.info('Scheduled %d pastes' % paste_counter)
629 | 
630 | 
631 | 
632 |     def _download_paste(self):
633 |         while True:
634 |             paste = self.pastes.get()  # (name, lang, href)
635 |             self.logger.debug('Fetching raw paste ' + paste[2])
636 |             link = self.conf_general['PBLink'] + 'raw/' + paste[2]
637 |             data = self._handle_data_download(link)
638 | 
639 |             self.logger.debug('Fetched {} with {} - {}'.format(
640 |                 link,
641 |                 data.status_code,
642 |                 data.reason
643 |             ))
644 |             if self.conf_stdout.getboolean('Enable'):
645 |                 self._write_to_stdout(paste, data)
646 |             if self.conf_mysql.getboolean('Enable'):
647 |                 self._write_to_mysql(paste, data)
648 | 
649 |     def _handle_data_download(self, link):
650 |         while True:
651 |             try:
652 |                 data = requests.get(link)
653 |             except:
654 |                 retry = self.conf_general.getint('ConnectionRetryInterval')
655 |                 self.logger.debug(
656 |                     'Error connecting to %s: Retry in %ss, TRACE: %s' %
657 |                     (link, retry, sys.exc_info())
658 |                 )
659 |                 self.logger.info('Connection problems - trying again in %ss' % retry)
660 |                 time.sleep(retry)
661 |             else:
662 |                 if data.status_code == 403 and b'Pastebin.com has blocked your IP' in data.content:
663 |                     self.logger.info('Our IP has been blocked. Trying again in an hour.')
664 |                     time.sleep(self.conf_general.getint('IPBlockedWaitTime'))
665 |                 return data
666 | 
667 |     def _assemble_output(self, conf, paste, data):
668 |         output = ''
669 |         if conf.getboolean('ShowName'):
670 |             output += 'Name: %s\n' % paste[0]
671 |         if conf.getboolean('ShowLang'):
672 |             output += 'Lang: %s\n' % paste[1]
673 |         if conf.getboolean('ShowLink'):
674 |             output += 'Link: %s\n' % (self.conf_general['PBLink'] + paste[2])
675 |         if conf.getboolean('ShowData'):
676 |             encoding = conf['DataEncoding']
677 |             limit = conf.getint('ContentDisplayLimit')
678 |             if limit > 0:
679 |                 output += '\n%s\n\n' % data.content.decode(encoding)[:limit]
680 |             else:
681 |                 output += '\n%s\n\n' % data.content.decode(encoding)
682 |         return output
683 | 
684 |     def _write_to_stdout(self, paste, data):
685 |         output = self._assemble_output(self.conf_stdout, paste, data)
686 |         sys.stdout.write(output)
687 | 
688 |     def _write_to_mysql(self, paste, data):
689 |         self.mysql_conn.add(paste, data)
690 | 
691 | 
692 |     def run(self):
693 |         for i in range(self.conf_general.getint('DownloadWorkers')):
694 |             t = threading.Thread(target=self._download_paste)
695 |             t.setDaemon(True)
696 |             t.start()
697 | 
698 |         s = threading.Thread(target=self._get_paste_data)
699 |         s.start()
700 |         s.join()
701 | 
702 | 
703 | if __name__ == '__main__':
704 |     ps = PastebinScraper()
705 |     ps.run()
706 | 


--------------------------------------------------------------------------------