├── .gitignore ├── Dockerfile ├── LICENSE ├── Procfile ├── README.md ├── certstream ├── __init__.py ├── certlib.py ├── util.py ├── watcher.py └── webserver.py ├── html ├── .babelrc ├── .gitignore ├── README.md ├── index.html ├── package-lock.json ├── package.json ├── src │ ├── assets │ │ ├── certstream-bg.png │ │ ├── certstream-overview.png │ │ ├── demo1.gif │ │ ├── demo2.gif │ │ ├── demo3.gif │ │ ├── doghead.png │ │ ├── favicon.png │ │ ├── logo.png │ │ └── rolling-transition.png │ ├── index.html │ ├── main.js │ ├── main.scss │ └── vues │ │ ├── App.vue │ │ ├── FeedWatcher.vue │ │ ├── Frontpage.vue │ │ └── TopPanel.vue └── webpack.config.js ├── requirements.txt ├── run_server.py └── runtime.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *,cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # dotenv 80 | .env 81 | 82 | # virtualenv 83 | .venv/ 84 | venv/ 85 | ENV/ 86 | 87 | # Spyder project settings 88 | .spyderproject 89 | 90 | # Rope project settings 91 | .ropeproject 92 | 93 | .idea/ 94 | 95 | *.sqlite3 96 | phishfinder/secrets.py 97 | 98 | celerybeat-schedule 99 | .DS_Store 100 | 101 | *.retry -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3 2 | 3 | WORKDIR /usr/src/app 4 | 5 | COPY requirements.txt ./ 6 | RUN pip install --no-cache-dir -r requirements.txt 7 | 8 | COPY . . 9 | 10 | EXPOSE 8080 11 | 12 | CMD [ "python", "./run_server.py" ] 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Cali Dog Security 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | web: python run_server.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 |

CertStream-Server

4 |

Aggregate and broadcast SSL certs as they're issued live.

5 |

6 | 7 | **Certstream-server** is a server written in python3 to aggregate, parse, and stream certificate data from multiple [certificate transparency logs](https://www.certificate-transparency.org/what-is-ct). It leverages concurrency to be relatively efficient and handle many clients at the same time. 8 | 9 | ## Getting Started 10 | 11 | Getting up and running is pretty easy (especially if you use Heroku, as we include a Procfile!), simply clone this repo and run: 12 | 13 | ``` 14 | # Install Dependencies 15 | pip install -r requirements.txt 16 | 17 | # Run CertStream 18 | python run_server.py 19 | ``` 20 | 21 | Or if python3 isn't your default interpreter: 22 | 23 | ``` 24 | # Install Dependencies 25 | pip3 install -r requirements.txt 26 | 27 | # Run CertStream 28 | python3 run_server.py 29 | ``` 30 | 31 | **Note** Running the server requires at least python 3.6, as we use the new `async`/`await` syntax internally. 32 | 33 | This will open up an http/websocket server on port 8080 (override this by setting a `PORT` environment variable). Connecting to it with a websocket client will subscribe you to a live aggregated stream of certificates and heartbeat messages. 34 | 35 | Connecting over a normal HTTP connection will show the certstream frontpage (currently being re-written). 36 | 37 | ## HTTP Routes 38 | 39 | `/latest.json` - Get the most recent 25 certificates CertStream has seen 40 | 41 | `/example.json` - Get the most recent certificate CertStream has seen 42 | 43 | `/stats` - Get statistics on the connected clients (override by setting the `STATS_URL` environment variable - we do this for [certstream.calidog.io](https://certstream.calidog.io)). 44 | 45 | ## Websocket Channels 46 | 47 | Channels are a feature that is currently in development (and aren't quite in yet), but the idea is that it will allow you to get a subset of the data instead of an entire data packet for each update message you receive. Sometimes you only care about the leaf certificate (so we'll omit the chain), and sometimes you don't want a copy of the certificates themselves (in which case we'll omit those). 48 | 49 | ## Data Structures 50 | 51 | There are currently only 2 data structures that certstream produces: 52 | 53 | **Heartbeats** 54 | 55 | ``` 56 | { 57 | "message_type": "heartbeat", 58 | "timestamp": 1508532970.93171 59 | } 60 | ``` 61 | 62 | **Certificate Updates** 63 | 64 | ``` 65 | { 66 | "message_type": "certificate_update", 67 | "data": { 68 | "update_type": "X509LogEntry", 69 | "leaf_cert": { 70 | "subject": { 71 | "aggregated": "/CN=e-zigarette-liquid-shop.de", 72 | "C": null, 73 | "ST": null, 74 | "L": null, 75 | "O": null, 76 | "OU": null, 77 | "CN": "e-zigarette-liquid-shop.de" 78 | }, 79 | "extensions": { 80 | "keyUsage": "Digital Signature, Key Encipherment", 81 | "extendedKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication", 82 | "basicConstraints": "CA:FALSE", 83 | "subjectKeyIdentifier": "AC:4C:7B:3C:E9:C8:7F:CB:E2:7D:5D:64:F2:25:0C:89:C2:AE:F0:5E", 84 | "authorityKeyIdentifier": "keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\n", 85 | "authorityInfoAccess": "OCSP - URI:http://ocsp.int-x3.letsencrypt.org\nCA Issuers - URI:http://cert.int-x3.letsencrypt.org/\n", 86 | "subjectAltName": "DNS:e-zigarette-liquid-shop.de, DNS:www.e-zigarette-liquid-shop.de", 87 | "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n CPS: http://cps.letsencrypt.org\n User Notice:\n Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\n" 88 | }, 89 | "not_before": 1508123861.0, 90 | "not_after": 1515899861.0, 91 | "as_der": "::BASE64_DER_CERT::", 92 | "all_domains": [ 93 | "e-zigarette-liquid-shop.de", 94 | "www.e-zigarette-liquid-shop.de" 95 | ] 96 | }, 97 | "chain": [ 98 | { 99 | "subject": { 100 | "aggregated": "/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3", 101 | "C": "US", 102 | "ST": null, 103 | "L": null, 104 | "O": "Let's Encrypt", 105 | "OU": null, 106 | "CN": "Let's Encrypt Authority X3" 107 | }, 108 | "extensions": { 109 | "basicConstraints": "CA:TRUE, pathlen:0", 110 | "keyUsage": "Digital Signature, Certificate Sign, CRL Sign", 111 | "authorityInfoAccess": "OCSP - URI:http://isrg.trustid.ocsp.identrust.com\nCA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c\n", 112 | "authorityKeyIdentifier": "keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\n", 113 | "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n CPS: http://cps.root-x1.letsencrypt.org\n", 114 | "crlDistributionPoints": "\nFull Name:\n URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl\n", 115 | "subjectKeyIdentifier": "A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1" 116 | }, 117 | "not_before": 1458232846.0, 118 | "not_after": 1615999246.0, 119 | "as_der": "::BASE64_DER_CERT::" 120 | }, 121 | { 122 | "subject": { 123 | "aggregated": "/O=Digital Signature Trust Co./CN=DST Root CA X3", 124 | "C": null, 125 | "ST": null, 126 | "L": null, 127 | "O": "Digital Signature Trust Co.", 128 | "OU": null, 129 | "CN": "DST Root CA X3" 130 | }, 131 | "extensions": { 132 | "basicConstraints": "CA:TRUE", 133 | "keyUsage": "Certificate Sign, CRL Sign", 134 | "subjectKeyIdentifier": "C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10" 135 | }, 136 | "not_before": 970348339.0, 137 | "not_after": 1633010475.0, 138 | "as_der": "::BASE64_DER_CERT::" 139 | } 140 | ], 141 | "cert_index": 19587936, 142 | "seen": 1508483726.8601687, 143 | "source": { 144 | "url": "mammoth.ct.comodo.com", 145 | "name": "Comodo 'Mammoth' CT log" 146 | } 147 | } 148 | } 149 | ``` 150 | -------------------------------------------------------------------------------- /certstream/__init__.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | import asyncio 4 | 5 | import uvloop 6 | asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) 7 | 8 | from certstream.certlib import MerkleTreeHeader 9 | from certstream.watcher import TransparencyWatcher 10 | from certstream.webserver import WebServer 11 | 12 | logging.basicConfig(format='[%(levelname)s:%(name)s] %(asctime)s - %(message)s', level=logging.INFO) 13 | 14 | def run(): 15 | logging.info("Starting CertStream...") 16 | 17 | loop = asyncio.get_event_loop() 18 | 19 | watcher = TransparencyWatcher(loop) 20 | webserver = WebServer(loop, watcher) 21 | 22 | asyncio.ensure_future(asyncio.gather(*watcher.get_tasks())) 23 | 24 | webserver.run_server() 25 | 26 | if __name__ == "__main__": 27 | run() -------------------------------------------------------------------------------- /certstream/certlib.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import datetime 3 | import logging 4 | import time 5 | 6 | from collections import OrderedDict 7 | 8 | from OpenSSL import crypto 9 | from construct import Struct, Byte, Int16ub, Int64ub, Enum, Bytes, \ 10 | Int24ub, this, GreedyBytes, GreedyRange, Terminated, Embedded 11 | 12 | 13 | MerkleTreeHeader = Struct( 14 | "Version" / Byte, 15 | "MerkleLeafType" / Byte, 16 | "Timestamp" / Int64ub, 17 | "LogEntryType" / Enum(Int16ub, X509LogEntryType=0, PrecertLogEntryType=1), 18 | "Entry" / GreedyBytes 19 | ) 20 | 21 | Certificate = Struct( 22 | "Length" / Int24ub, 23 | "CertData" / Bytes(this.Length) 24 | ) 25 | 26 | CertificateChain = Struct( 27 | "ChainLength" / Int24ub, 28 | "Chain" / GreedyRange(Certificate), 29 | ) 30 | 31 | PreCertEntry = Struct( 32 | "LeafCert" / Certificate, 33 | Embedded(CertificateChain), 34 | Terminated 35 | ) 36 | 37 | def dump_extensions(certificate): 38 | extensions = {} 39 | for x in range(certificate.get_extension_count()): 40 | extension_name = "" 41 | try: 42 | extension_name = certificate.get_extension(x).get_short_name() 43 | 44 | if extension_name == b'UNDEF': 45 | continue 46 | 47 | extensions[extension_name.decode('latin-1')] = certificate.get_extension(x).__str__() 48 | except: 49 | try: 50 | extensions[extension_name.decode('latin-1')] = "NULL" 51 | except Exception as e: 52 | logging.debug("Extension parsing error -> {}".format(e)) 53 | return extensions 54 | 55 | def serialize_certificate(certificate): 56 | subject = certificate.get_subject() 57 | not_before_datetime = datetime.datetime.strptime(certificate.get_notBefore().decode('ascii'), "%Y%m%d%H%M%SZ") 58 | not_after_datetime = datetime.datetime.strptime(certificate.get_notAfter().decode('ascii'), "%Y%m%d%H%M%SZ") 59 | return { 60 | "subject": { 61 | "aggregated": repr(certificate.get_subject())[18:-2], 62 | "C": subject.C, 63 | "ST": subject.ST, 64 | "L": subject.L, 65 | "O": subject.O, 66 | "OU": subject.OU, 67 | "CN": subject.CN 68 | }, 69 | "extensions": dump_extensions(certificate), 70 | "not_before": not_before_datetime.timestamp(), 71 | "not_after": not_after_datetime.timestamp(), 72 | "serial_number": '{0:x}'.format(int(certificate.get_serial_number())), 73 | "fingerprint": str(certificate.digest("sha1"),'utf-8'), 74 | "as_der": base64.b64encode( 75 | crypto.dump_certificate( 76 | crypto.FILETYPE_ASN1, certificate 77 | ) 78 | ).decode('utf-8') 79 | } 80 | 81 | def add_all_domains(cert_data): 82 | all_domains = [] 83 | 84 | # Apparently we have certificates with null CNs....what? 85 | if cert_data['leaf_cert']['subject']['CN']: 86 | all_domains.append(cert_data['leaf_cert']['subject']['CN']) 87 | 88 | subject_alternative_name = cert_data['leaf_cert']['extensions'].get('subjectAltName') 89 | 90 | if subject_alternative_name: 91 | for entry in subject_alternative_name.split(', '): 92 | if entry.startswith('DNS:'): 93 | all_domains.append(entry.replace('DNS:', '')) 94 | 95 | cert_data['leaf_cert']['all_domains'] = list(OrderedDict.fromkeys(all_domains)) 96 | 97 | return cert_data 98 | 99 | def parse_ctl_entry(entry, operator_information): 100 | mtl = MerkleTreeHeader.parse(base64.b64decode(entry['leaf_input'])) 101 | 102 | cert_data = {} 103 | 104 | if mtl.LogEntryType == "X509LogEntryType": 105 | cert_data['update_type'] = "X509LogEntry" 106 | chain = [crypto.load_certificate(crypto.FILETYPE_ASN1, Certificate.parse(mtl.Entry).CertData)] 107 | extra_data = CertificateChain.parse(base64.b64decode(entry['extra_data'])) 108 | for cert in extra_data.Chain: 109 | chain.append(crypto.load_certificate(crypto.FILETYPE_ASN1, cert.CertData)) 110 | else: 111 | cert_data['update_type'] = "PreCertEntry" 112 | extra_data = PreCertEntry.parse(base64.b64decode(entry['extra_data'])) 113 | chain = [crypto.load_certificate(crypto.FILETYPE_ASN1, extra_data.LeafCert.CertData)] 114 | 115 | for cert in extra_data.Chain: 116 | chain.append( 117 | crypto.load_certificate(crypto.FILETYPE_ASN1, cert.CertData) 118 | ) 119 | 120 | cert_data.update({ 121 | "leaf_cert": serialize_certificate(chain[0]), 122 | "chain": [serialize_certificate(x) for x in chain[1:]], 123 | "cert_index": entry['index'], 124 | "seen": time.time() 125 | }) 126 | 127 | add_all_domains(cert_data) 128 | 129 | cert_data['source'] = { 130 | "url": operator_information['url'], 131 | "name": operator_information['description'] 132 | } 133 | 134 | return cert_data 135 | -------------------------------------------------------------------------------- /certstream/util.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | 3 | def pretty_date(time=False): 4 | """ 5 | Get a datetime object or a int() Epoch timestamp and return a 6 | pretty string like 'an hour ago', 'Yesterday', '3 months ago', 7 | 'just now', etc 8 | """ 9 | now = datetime.now() 10 | if type(time) is int: 11 | diff = now - datetime.fromtimestamp(time) 12 | elif isinstance(time, datetime): 13 | diff = now - time 14 | elif not time: 15 | diff = now - now 16 | second_diff = diff.seconds 17 | day_diff = diff.days 18 | 19 | if day_diff < 0: 20 | return '' 21 | 22 | if day_diff == 0: 23 | if second_diff < 10: 24 | return "just now" 25 | if second_diff < 60: 26 | return str(second_diff) + " seconds ago" 27 | if second_diff < 120: 28 | return "a minute ago" 29 | if second_diff < 3600: 30 | return str(second_diff / 60) + " minutes ago" 31 | if second_diff < 7200: 32 | return "an hour ago" 33 | if second_diff < 86400: 34 | return str(second_diff / 3600) + " hours ago" 35 | if day_diff == 1: 36 | return "Yesterday" 37 | if day_diff < 7: 38 | return str(day_diff) + " days ago" 39 | if day_diff < 31: 40 | return str(day_diff / 7) + " weeks ago" 41 | if day_diff < 365: 42 | return str(day_diff / 30) + " months ago" 43 | return str(day_diff / 365) + " years ago" 44 | 45 | def get_ip(request): 46 | peer_info = request.transport.get_extra_info('peername') 47 | 48 | ip = "UNKNOWN" 49 | 50 | if peer_info is not None: 51 | ip, port = peer_info 52 | 53 | if 'X-Forwarded-For' in request.headers: 54 | ip = request.headers.get('X-Forwarded-For') 55 | 56 | return ip -------------------------------------------------------------------------------- /certstream/watcher.py: -------------------------------------------------------------------------------- 1 | import aiohttp 2 | import asyncio 3 | import logging 4 | import math 5 | import requests 6 | import sys 7 | import os 8 | 9 | from certstream.certlib import parse_ctl_entry 10 | 11 | 12 | class TransparencyWatcher(object): 13 | # These are a list of servers that we shouldn't even try to connect to. In testing they either had bad 14 | # DNS records, resolved to un-routable IP addresses, or didn't have valid SSL certificates. 15 | BAD_CT_SERVERS = [ 16 | "alpha.ctlogs.org", 17 | "clicky.ct.letsencrypt.org", 18 | "ct.akamai.com", 19 | "ct.filippo.io/behindthesofa", 20 | "ct.gdca.com.cn", 21 | "ct.izenpe.com", 22 | "ct.izenpe.eus", 23 | "ct.sheca.com", 24 | "ct.startssl.com", 25 | "ct.wosign.com", 26 | "ctlog.api.venafi.com", 27 | "ctlog.gdca.com.cn", 28 | "ctlog.sheca.com", 29 | "ctlog.wosign.com", 30 | "ctlog2.wosign.com", 31 | "flimsy.ct.nordu.net:8080", 32 | "log.certly.io", 33 | "nessie2021.ct.digicert.com/log", 34 | "plausible.ct.nordu.net", 35 | "www.certificatetransparency.cn/ct", 36 | ] 37 | 38 | MAX_BLOCK_SIZE = 64 39 | 40 | def __init__(self, _loop): 41 | self.loop = _loop 42 | self.stopped = False 43 | self.logger = logging.getLogger('certstream.watcher') 44 | 45 | self.stream = asyncio.Queue(maxsize=3000) 46 | 47 | self.logger.info("Initializing the CTL watcher") 48 | 49 | def _initialize_ts_logs(self): 50 | try: 51 | self.transparency_logs = requests.get('https://www.gstatic.com/ct/log_list/all_logs_list.json').json() 52 | except Exception as e: 53 | self.logger.fatal("Invalid response from certificate directory! Exiting :(") 54 | sys.exit(1) 55 | 56 | self.logger.info("Retrieved transparency log with {} entries to watch.".format(len(self.transparency_logs['logs']))) 57 | for entry in self.transparency_logs['logs']: 58 | if entry['url'].endswith('/'): 59 | entry['url'] = entry['url'][:-1] 60 | self.logger.info(" + {}".format(entry['description'])) 61 | 62 | async def _print_memory_usage(self): 63 | import objgraph 64 | import gc 65 | 66 | while True: 67 | print("Stream backlog : {}".format(self.stream.qsize())) 68 | gc.collect() 69 | objgraph.show_growth() 70 | await asyncio.sleep(60) 71 | 72 | def get_tasks(self): 73 | self._initialize_ts_logs() 74 | 75 | coroutines = [] 76 | 77 | if os.getenv("DEBUG_MEMORY", False): 78 | coroutines.append(self._print_memory_usage()) 79 | 80 | for log in self.transparency_logs['logs']: 81 | if log['url'] not in self.BAD_CT_SERVERS: 82 | coroutines.append(self.watch_for_updates_task(log)) 83 | return coroutines 84 | 85 | def stop(self): 86 | self.logger.info('Got stop order, exiting...') 87 | self.stopped = True 88 | for task in asyncio.Task.all_tasks(): 89 | task.cancel() 90 | 91 | async def watch_for_updates_task(self, operator_information): 92 | try: 93 | latest_size = 0 94 | name = operator_information['description'] 95 | while not self.stopped: 96 | try: 97 | async with aiohttp.ClientSession(loop=self.loop) as session: 98 | async with session.get("https://{}/ct/v1/get-sth".format(operator_information['url'])) as response: 99 | info = await response.json() 100 | except aiohttp.ClientError as e: 101 | self.logger.info('[{}] Exception -> {}'.format(name, e)) 102 | await asyncio.sleep(600) 103 | continue 104 | 105 | tree_size = info.get('tree_size') 106 | 107 | # TODO: Add in persistence and id tracking per log 108 | if latest_size == 0: 109 | latest_size = tree_size 110 | 111 | if latest_size < tree_size: 112 | self.logger.info('[{}] [{} -> {}] New certs found, updating!'.format(name, latest_size, tree_size)) 113 | 114 | try: 115 | async for result_chunk in self.get_new_results(operator_information, latest_size, tree_size): 116 | for entry in result_chunk: 117 | cert_data = parse_ctl_entry(entry, operator_information) 118 | await self.stream.put(cert_data) 119 | 120 | except aiohttp.ClientError as e: 121 | self.logger.info('[{}] Exception -> {}'.format(name, e)) 122 | await asyncio.sleep(600) 123 | continue 124 | 125 | except Exception as e: 126 | print("Encountered an exception while getting new results! -> {}".format(e)) 127 | return 128 | 129 | latest_size = tree_size 130 | else: 131 | self.logger.debug('[{}][{}|{}] No update needed, continuing...'.format(name, latest_size, tree_size)) 132 | 133 | await asyncio.sleep(30) 134 | except Exception as e: 135 | print("Encountered an exception while getting new results! -> {}".format(e)) 136 | return 137 | 138 | async def get_new_results(self, operator_information, latest_size, tree_size): 139 | # The top of the tree isn't actually a cert yet, so the total_size is what we're aiming for 140 | total_size = tree_size - latest_size 141 | start = latest_size 142 | 143 | end = start + self.MAX_BLOCK_SIZE 144 | 145 | chunks = math.ceil(total_size / self.MAX_BLOCK_SIZE) 146 | 147 | self.logger.info("Retrieving {} certificates ({} -> {}) for {}".format(tree_size-latest_size, latest_size, tree_size, operator_information['description'])) 148 | async with aiohttp.ClientSession(loop=self.loop) as session: 149 | for _ in range(chunks): 150 | # Cap the end to the last record in the DB 151 | if end >= tree_size: 152 | end = tree_size - 1 153 | 154 | assert end >= start, "End {} is less than start {}!".format(end, start) 155 | assert end < tree_size, "End {} is less than tree_size {}".format(end, tree_size) 156 | 157 | url = "https://{}/ct/v1/get-entries?start={}&end={}".format(operator_information['url'], start, end) 158 | 159 | async with session.get(url) as response: 160 | certificates = await response.json() 161 | if 'error_message' in certificates: 162 | print("error!") 163 | 164 | for index, cert in zip(range(start, end+1), certificates['entries']): 165 | cert['index'] = index 166 | 167 | yield certificates['entries'] 168 | 169 | start += self.MAX_BLOCK_SIZE 170 | 171 | end = start + self.MAX_BLOCK_SIZE + 1 172 | 173 | class DummyTransparencyWatcher(object): 174 | stream = asyncio.Queue() 175 | def get_tasks(self): 176 | return [] 177 | 178 | if __name__ == "__main__": 179 | loop = asyncio.get_event_loop() 180 | watcher = TransparencyWatcher(loop) 181 | loop.run_until_complete(asyncio.gather(*watcher.get_tasks())) 182 | -------------------------------------------------------------------------------- /certstream/webserver.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import collections 3 | import json 4 | import logging 5 | import os 6 | import time 7 | import ssl 8 | 9 | from aiohttp import web 10 | from aiohttp.web_urldispatcher import Response 11 | from aiohttp.web_ws import WebSocketResponse 12 | 13 | from certstream.util import pretty_date, get_ip 14 | 15 | WebsocketClientInfo = collections.namedtuple( 16 | 'WebsocketClientInfo', 17 | ['external_ip', 'queue', 'connection_time'] 18 | ) 19 | 20 | STATIC_INDEX = ''' 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 |
30 | 31 | 32 | '''.format(time.time()) 33 | 34 | class WebServer(object): 35 | def __init__(self, _loop, transparency_watcher): 36 | self.active_sockets = [] 37 | self.recently_seen = collections.deque(maxlen=25) 38 | self.stats_url = os.getenv("STATS_URL", 'stats') 39 | self.logger = logging.getLogger('certstream.webserver') 40 | 41 | self.loop = _loop 42 | self.watcher = transparency_watcher 43 | 44 | self.app = web.Application(loop=self.loop) 45 | 46 | self._add_routes() 47 | 48 | def run_server(self): 49 | self.mux_stream = asyncio.ensure_future(self.mux_ctl_stream()) 50 | self.heartbeat_coro = asyncio.ensure_future(self.ws_heartbeats()) 51 | 52 | if os.environ.get("NOSSL", False): 53 | ssl_ctx = None 54 | else: 55 | ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) 56 | ssl_ctx.load_cert_chain(certfile=os.getenv("SERVER_CERT", "server.crt"), keyfile=os.getenv("SERVER_KEY", "server.key")) 57 | 58 | web.run_app( 59 | self.app, 60 | port=int(os.environ.get('PORT', 8080)), 61 | ssl_context=ssl_ctx 62 | ) 63 | 64 | def _add_routes(self): 65 | self.app.router.add_get("/latest.json", self.latest_json_handler) 66 | self.app.router.add_get("/example.json", self.example_json_handler) 67 | self.app.router.add_get("/{}".format(self.stats_url), self.stats_handler) 68 | self.app.router.add_get('/', self.root_handler) 69 | self.app.router.add_get('/develop', self.dev_handler) 70 | 71 | async def mux_ctl_stream(self): 72 | while True: 73 | cert_data = await self.watcher.stream.get() 74 | 75 | data_packet = { 76 | "message_type": "certificate_update", 77 | "data": cert_data 78 | } 79 | 80 | self.recently_seen.append(data_packet) 81 | 82 | for client in self.active_sockets: 83 | try: 84 | client.queue.put_nowait(data_packet) 85 | except asyncio.QueueFull: 86 | pass 87 | 88 | 89 | async def dev_handler(self, request): 90 | # If we have a websocket request 91 | if request.headers.get("Upgrade"): 92 | ws = web.WebSocketResponse() 93 | 94 | await ws.prepare(request) 95 | 96 | try: 97 | for message in self.recently_seen: 98 | message_json = json.dumps(message) 99 | await ws.send_str(message_json) 100 | except asyncio.CancelledError: 101 | print('websocket cancelled') 102 | 103 | await ws.close() 104 | 105 | return ws 106 | 107 | return web.Response( 108 | body=json.dumps( 109 | { 110 | "error": "Please use this url with a websocket client!" 111 | }, 112 | indent=4 113 | ), 114 | content_type="application/json", 115 | ) 116 | 117 | async def root_handler(self, request): 118 | resp = WebSocketResponse() 119 | available = resp.can_prepare(request) 120 | if not available: 121 | return Response(body=STATIC_INDEX, content_type="text/html") 122 | 123 | await resp.prepare(request) 124 | 125 | client_queue = asyncio.Queue(maxsize=500) 126 | 127 | client = WebsocketClientInfo( 128 | external_ip=get_ip(request), 129 | queue=client_queue, 130 | connection_time=int(time.time()), 131 | ) 132 | 133 | try: 134 | self.logger.info('Client {} joined.'.format(client.external_ip)) 135 | self.active_sockets.append(client) 136 | while True: 137 | message = await client_queue.get() 138 | message_json = json.dumps(message) 139 | await resp.send_str(message_json) 140 | 141 | finally: 142 | self.active_sockets.remove(client) 143 | self.logger.info('Client {} disconnected.'.format(client.external_ip)) 144 | 145 | async def latest_json_handler(self, _): 146 | return web.Response( 147 | body=json.dumps( 148 | { 149 | "messages": list(self.recently_seen) 150 | }, 151 | indent=4 152 | ), 153 | headers={"Access-Control-Allow-Origin": "*"}, 154 | content_type="application/json", 155 | ) 156 | 157 | async def example_json_handler(self, _): 158 | if self.recently_seen: 159 | return web.Response( 160 | body=json.dumps(list(self.recently_seen)[0], indent=4), 161 | headers={"Access-Control-Allow-Origin": "*"}, 162 | content_type="application/json", 163 | ) 164 | else: 165 | return web.Response( 166 | body="{}", 167 | headers={"Access-Control-Allow-Origin": "*"}, 168 | content_type="application/json" 169 | ) 170 | 171 | async def stats_handler(self, _): 172 | clients = {} 173 | for client in self.active_sockets: 174 | client_identifier = "{}-{}".format(client.external_ip, client.connection_time) 175 | clients[client_identifier] = { 176 | "ip_address": client.external_ip, 177 | "conection_time": client.connection_time, 178 | "connection_length": pretty_date(client.connection_time), 179 | "queue_size": client.queue.qsize(), 180 | } 181 | 182 | return web.Response( 183 | body=json.dumps({ 184 | "connected_client_count": len(self.active_sockets), 185 | "clients": clients 186 | }, indent=4 187 | ), 188 | content_type="application/json", 189 | ) 190 | 191 | async def ws_heartbeats(self): 192 | self.logger.info("Starting WS heartbeat coro...") 193 | while True: 194 | await asyncio.sleep(30) 195 | self.logger.debug("Sending ping...") 196 | timestamp = time.time() 197 | for client in self.active_sockets: 198 | await client.queue.put({ 199 | "message_type": "heartbeat", 200 | "timestamp": timestamp 201 | }) 202 | 203 | if __name__ == "__main__": 204 | from certstream.watcher import TransparencyWatcher 205 | loop = asyncio.get_event_loop() 206 | watcher = TransparencyWatcher(loop) 207 | webserver = WebServer(loop, watcher) 208 | asyncio.ensure_future(asyncio.gather(*watcher.get_tasks())) 209 | webserver.run_server() 210 | -------------------------------------------------------------------------------- /html/.babelrc: -------------------------------------------------------------------------------- 1 | { 2 | "presets": [ 3 | ["env", { "modules": false }], 4 | ] 5 | } 6 | -------------------------------------------------------------------------------- /html/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | node_modules/ 3 | dist/ 4 | npm-debug.log 5 | yarn-error.log 6 | 7 | # Editor directories and files 8 | .idea 9 | *.suo 10 | *.ntvs* 11 | *.njsproj 12 | *.sln 13 | -------------------------------------------------------------------------------- /html/README.md: -------------------------------------------------------------------------------- 1 | # certstream-html 2 | 3 | > The CertStream frontend 4 | 5 | ## Build Setup 6 | 7 | ``` bash 8 | # install dependencies 9 | npm install 10 | 11 | # serve with hot reload at localhost:8080 12 | npm run dev 13 | 14 | # build for production with minification 15 | npm run build 16 | ``` 17 | 18 | For detailed explanation on how things work, consult the [docs for vue-loader](http://vuejs.github.io/vue-loader). 19 | -------------------------------------------------------------------------------- /html/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
10 | 11 | 12 | -------------------------------------------------------------------------------- /html/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "certstream-html", 3 | "description": "The CertStream frontend", 4 | "version": "1.0.0", 5 | "author": "fitblip ", 6 | "private": true, 7 | "scripts": { 8 | "dev": "cross-env NODE_ENV=development webpack-dev-server --open --hot --host 0.0.0.0 --content-base src --debug", 9 | "build": "cross-env NODE_ENV=production webpack --progress --hide-modules", 10 | "deploy": "gsutil rsync -a all-users -d dist/ gs://certstream-prod/" 11 | }, 12 | "dependencies": { 13 | "animejs": "^2.2.0", 14 | "axios": "^0.16.2", 15 | "babel-preset-es2015": "^6.24.1", 16 | "babel-preset-stage-2": "^6.24.1", 17 | "debounce": "^1.0.2", 18 | "devicon": "^2.0.0", 19 | "robust-websocket": "^0.3.0", 20 | "typed.js": "^2.0.6", 21 | "v-tooltip": "^2.0.0-rc.2", 22 | "vue": "^2.5.2", 23 | "vue-json-tree": "^0.3.3", 24 | "vue-json-tree-view": "^2.1.1", 25 | "vue-scrollto": "^2.7.8" 26 | }, 27 | "devDependencies": { 28 | "animate.css": "^3.5.2", 29 | "babel-core": "^6.26.0", 30 | "babel-loader": "^6.0.0", 31 | "babel-preset-env": "^1.6.1", 32 | "bulma": "^0.6.0", 33 | "clean-webpack-plugin": "^0.1.17", 34 | "cross-env": "^3.0.0", 35 | "css-loader": "^0.25.0", 36 | "file-loader": "^0.9.0", 37 | "font-awesome": "^4.7.0", 38 | "html-webpack-harddisk-plugin": "^0.1.0", 39 | "html-webpack-plugin": "^2.30.1", 40 | "image-webpack-loader": "^3.4.2", 41 | "node-sass": "^4.5.0", 42 | "sass-loader": "^5.0.1", 43 | "style-loader": "^0.19.0", 44 | "url-loader": "^0.6.2", 45 | "vue-loader": "^12.1.0", 46 | "vue-template-compiler": "^2.5.2", 47 | "webpack": "^2.6.1", 48 | "webpack-dev-server": "^2.9.3" 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /html/src/assets/certstream-bg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/certstream-bg.png -------------------------------------------------------------------------------- /html/src/assets/certstream-overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/certstream-overview.png -------------------------------------------------------------------------------- /html/src/assets/demo1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/demo1.gif -------------------------------------------------------------------------------- /html/src/assets/demo2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/demo2.gif -------------------------------------------------------------------------------- /html/src/assets/demo3.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/demo3.gif -------------------------------------------------------------------------------- /html/src/assets/doghead.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/doghead.png -------------------------------------------------------------------------------- /html/src/assets/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/favicon.png -------------------------------------------------------------------------------- /html/src/assets/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/logo.png -------------------------------------------------------------------------------- /html/src/assets/rolling-transition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CaliDog/certstream-server-python/790718da384d3710e7842bd32b8367d2e142cc14/html/src/assets/rolling-transition.png -------------------------------------------------------------------------------- /html/src/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
11 | 12 | 13 | -------------------------------------------------------------------------------- /html/src/main.js: -------------------------------------------------------------------------------- 1 | import Vue from 'vue' 2 | import App from './vues/App.vue' 3 | 4 | require('./main.scss'); 5 | 6 | window.component = new Vue({ 7 | el: '#app', 8 | render: h => h(App) 9 | }); 10 | -------------------------------------------------------------------------------- /html/src/main.scss: -------------------------------------------------------------------------------- 1 | $fa-font-path: '~font-awesome/fonts'; 2 | 3 | @import '~font-awesome/scss/font-awesome.scss'; 4 | @import "~animate.css/animate.css"; 5 | @import "~bulma/css/bulma.css"; 6 | @import "~devicon/devicon.css"; 7 | @import "~devicon/devicon-colors.css"; -------------------------------------------------------------------------------- /html/src/vues/App.vue: -------------------------------------------------------------------------------- 1 | 6 | 7 | 21 | 22 | 31 | -------------------------------------------------------------------------------- /html/src/vues/FeedWatcher.vue: -------------------------------------------------------------------------------- 1 | 71 | 72 | 212 | 213 | 445 | -------------------------------------------------------------------------------- /html/src/vues/Frontpage.vue: -------------------------------------------------------------------------------- 1 | 188 | 189 | 190 | 404 | 405 | 967 | -------------------------------------------------------------------------------- /html/src/vues/TopPanel.vue: -------------------------------------------------------------------------------- 1 | 6 | 7 | 21 | 22 | 31 | -------------------------------------------------------------------------------- /html/webpack.config.js: -------------------------------------------------------------------------------- 1 | const path = require('path'); 2 | 3 | const webpack = require('webpack'); 4 | 5 | const HtmlWebpackPlugin = require('html-webpack-plugin'); 6 | 7 | const CleanWebpackPlugin = require('clean-webpack-plugin'); 8 | 9 | module.exports = { 10 | entry: './src/main.js', 11 | output: { 12 | path: path.resolve(__dirname, './dist/'), 13 | filename: 'build.js', 14 | }, 15 | module: { 16 | rules: [ 17 | { 18 | test: /\.vue$/, 19 | loader: 'vue-loader', 20 | options: { 21 | loaders: { 22 | // Since sass-loader (weirdly) has SCSS as its default parse mode, we map 23 | // the "scss" and "sass" values for the lang attribute to the right configs here. 24 | // other preprocessors should work out of the box, no loader config like this necessary. 25 | 'scss': 'vue-style-loader!css-loader!sass-loader', 26 | 'sass': 'vue-style-loader!css-loader!sass-loader?indentedSyntax' 27 | } 28 | // other vue-loader options go here 29 | } 30 | }, 31 | { 32 | test: /\.(scss|sass)$/, 33 | loaders: ['style-loader', 'css-loader', 'sass-loader'], 34 | exclude: /node_modules/ 35 | }, 36 | { test: /\.woff(2)?(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: "url-loader?limit=10000&minetype=application/font-woff" }, 37 | { test: /\.(ttf|eot|svg)(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: "file-loader" }, 38 | { 39 | test: /\.js$/, 40 | loader: 'babel-loader', 41 | exclude: /node_modules/ 42 | }, 43 | { 44 | test: /\.(png|jpg|gif|svg)$/, 45 | loader: 'file-loader', 46 | options: { 47 | name: '[name].[ext]?[hash]' 48 | } 49 | }, 50 | { 51 | test: /\.(png|jpg|gif|svg)$/, 52 | loader: 'image-webpack-loader', 53 | options: { 54 | optipng: { 55 | enabled: true, 56 | optimizationLevel: 7, 57 | }, 58 | bypassOnDebug: true 59 | } 60 | }, 61 | ], 62 | }, 63 | resolve: { 64 | alias: { 65 | 'vue$': 'vue/dist/vue.esm.js' 66 | } 67 | }, 68 | devServer: { 69 | historyApiFallback: true, 70 | noInfo: false, 71 | host: '0.0.0.0', 72 | disableHostCheck: true 73 | }, 74 | performance: { 75 | hints: false 76 | }, 77 | devtool: '#eval-source-map', 78 | plugins: [ 79 | new CleanWebpackPlugin(['dist']), 80 | new HtmlWebpackPlugin({ 81 | template: 'index.html' 82 | }), 83 | ] 84 | }; 85 | 86 | if (process.env.NODE_ENV === 'production') { 87 | module.exports.devtool = '#source-map' 88 | module.exports.output.publicPath = "https://storage.googleapis.com/certstream-prod/"; 89 | // http://vue-loader.vuejs.org/en/workflow/production.html 90 | module.exports.plugins = (module.exports.plugins || []).concat([ 91 | new webpack.DefinePlugin({ 92 | 'process.env': { 93 | NODE_ENV: '"production"' 94 | } 95 | }), 96 | new webpack.optimize.UglifyJsPlugin({ 97 | sourceMap: false, 98 | compress: { 99 | warnings: false 100 | } 101 | }), 102 | new webpack.LoaderOptionsPlugin({ 103 | minimize: true 104 | }) 105 | ]) 106 | } 107 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | construct 2 | uvloop 3 | aiohttp 4 | aioprocessing 5 | PyOpenSSL 6 | websockets 7 | requests -------------------------------------------------------------------------------- /run_server.py: -------------------------------------------------------------------------------- 1 | import certstream 2 | 3 | certstream.run() -------------------------------------------------------------------------------- /runtime.txt: -------------------------------------------------------------------------------- 1 | python-3.6.2 --------------------------------------------------------------------------------