├── .gitignore ├── LICENSE.md ├── README.md ├── docker-compose.yml ├── postgres ├── postgres_clear.sql └── postgres_init.sql ├── smtpd ├── Dockerfile └── src │ ├── app.py │ ├── mailer.py │ ├── plugin_manager.py │ ├── plugins │ ├── directory_example │ │ ├── __init__.py │ │ └── test_module.py │ ├── example.py │ └── plugin.py │ ├── requirements.txt │ ├── smtpd.cfg.default │ ├── storage.py │ ├── test │ ├── attachment.txt │ └── send_email.py │ ├── test_storage.py │ └── util │ └── mailslurper_import.py └── web ├── Dockerfile └── src ├── app.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | smtpd/src/smtpd.cfg 2 | *__pycache__* 3 | *.pyc 4 | .idea/ 5 | .vscode 6 | smtpd/src/.pytest_cache/ 7 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) [2018] [Brenton Morris] 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Sarlacc 2 | 3 | This is an SMTP server that I use in my malware lab to collect spam from infected hosts. 4 | 5 | It will collect all mail items sent to it in a postgres database, storing all attachments in mongodb. 6 | 7 | This is work in progress code and there will probably be bugs but it does everything I need. 8 | 9 | Warning: There will most likely be breaking changes as I flesh out the plugin API. Once it has stabilized I will give this a version number and try not to break anything else. 10 | 11 | 12 | ## Getting Started 13 | 14 | ### docker-compose 15 | 16 | To get started with docker-compose, simply run `docker-compose up`. 17 | 18 | The server will then be listening for SMTP connections on port `2500`. 19 | 20 | #### Data 21 | To ensure proper data persistence, data for both postgres and mongodb is stored in docker volumes. 22 | 23 | 24 | ### Production 25 | 26 | If installing in a production environment which requires a proper setup, an install of mongodb and postgresql will be required. 27 | To configure sarlacc, copy the default config file to `smtpd/src/smtpd.cfg` and override the settings you wish to change: 28 | ``` 29 | cp smtpd/src/smtpd.cfg.default smtpd/src/smtpd.cfg 30 | $EDITOR smtpd/src/smtpd.cfg 31 | ``` 32 | Then edit the file with your required configuration. 33 | 34 | You can use the `postgres/postgres_init.sql` script to initialize the database for use with sarlacc. 35 | ``` 36 | psql -h localhost -U postgres < postgres/postgres_init.sql 37 | ``` 38 | 39 | If you want to use different credentials (you should) then modify the `postgres/postgres_init.sql` and the config file for the smtp server appropriately. 40 | 41 | cd into the `smtpd/src` directory: 42 | ``` 43 | cd smtpd/src 44 | ``` 45 | 46 | Install the dependencies: 47 | ``` 48 | pip install -r requirements.txt 49 | ``` 50 | 51 | Start the server: 52 | ``` 53 | ./app.py 54 | ``` 55 | 56 | The server will then be listening for SMTP connections on port `2500`. 57 | 58 | 59 | ### Requirements 60 | 61 | python3.5 62 | 63 | 64 | 65 | ## Web Client 66 | 67 | The web client has not been built yet, to view the data you will need to manually interact with the databases. 68 | 69 | 70 | 71 | ## Plugins 72 | 73 | You can extend sarlacc via plugins. Simply drop a python file (or a directory with an `__init__.py` file) into `smtpd/src/plugins`. There are example's of both types of plugins at `smtpd/src/plugins/example.py` and `smtpd/src/plugins/directory_example`. 74 | 75 | To get a full idea of what events are available for the plugins to be notified by, check out the `smtpd/src/plugins/plugin.py` file. 76 | 77 | Plugins are also exposed to the internal storage API, from which you can pull email items, recipients, attachments, tag attachments etc etc. Take a look at the `smtpd/src/storage.py` file for more info on how to use this. 78 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | services: 3 | mongodb: 4 | image: "mvertes/alpine-mongo" 5 | restart: always 6 | ports: 7 | - "127.0.0.1:27017:27017" 8 | volumes: 9 | - mongodb:/data/db 10 | 11 | postgres: 12 | image: "postgres:9.2-alpine" 13 | # restart: always 14 | ports: 15 | - "127.0.0.1:5432:5432" 16 | volumes: 17 | - postgres:/var/lib/postgresql/data 18 | environment: 19 | PGDATA: /var/lib/postgresql/data/pgdata 20 | POSTGRES_DB: sarlacc 21 | POSTGRES_USER: user 22 | 23 | smtpd: 24 | build: ./smtpd 25 | ports: 26 | - "2500:2500" 27 | 28 | web: 29 | build: ./web 30 | ports: 31 | - "5000:5000" 32 | 33 | volumes: 34 | mongodb: 35 | postgres: 36 | 37 | -------------------------------------------------------------------------------- /postgres/postgres_clear.sql: -------------------------------------------------------------------------------- 1 | delete from attachment; 2 | delete from mailrecipient; 3 | delete from mailitem; 4 | delete from body; 5 | delete from recipient; 6 | -------------------------------------------------------------------------------- /postgres/postgres_init.sql: -------------------------------------------------------------------------------- 1 | CREATE DATABASE sarlacc; 2 | 3 | \connect sarlacc 4 | 5 | DROP TABLE attachment CASCADE; 6 | DROP TABLE recipient CASCADE; 7 | DROP TABLE mailrecipient CASCADE; 8 | DROP TABLE mailitem CASCADE; 9 | DROP TABLE body CASCADE; 10 | 11 | CREATE TABLE body ( 12 | id SERIAL PRIMARY KEY, 13 | sha256 text, 14 | content text 15 | ); 16 | 17 | CREATE TABLE mailitem ( 18 | id SERIAL PRIMARY KEY, 19 | datesent timestamp, 20 | subject text, 21 | fromaddress text, 22 | bodyid integer REFERENCES body (id) 23 | ); 24 | 25 | CREATE TABLE recipient ( 26 | id SERIAL PRIMARY KEY, 27 | emailaddress text 28 | ); 29 | 30 | CREATE TABLE mailrecipient ( 31 | id SERIAL PRIMARY KEY, 32 | recipientid integer REFERENCES recipient (id), 33 | mailid integer REFERENCES mailitem (id) 34 | ); 35 | 36 | CREATE TABLE attachment ( 37 | id SERIAL PRIMARY KEY, 38 | mailid integer REFERENCES mailitem (id), 39 | sha256 text, 40 | filename text 41 | ); 42 | 43 | CREATE ROLE "user"; 44 | ALTER ROLE "user" WITH LOGIN; 45 | GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO "user"; 46 | GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO "user"; 47 | GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "user"; 48 | 49 | -------------------------------------------------------------------------------- /smtpd/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.6 2 | ADD ./src /smtpd 3 | WORKDIR /smtpd 4 | RUN pip install -r requirements.txt 5 | CMD python -u app.py 6 | 7 | -------------------------------------------------------------------------------- /smtpd/src/app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import asyncio 5 | import threading 6 | import sys 7 | import logging 8 | from configparser import ConfigParser 9 | 10 | from mailer import MailHandler, CustomIdentController 11 | from plugin_manager import PluginManager 12 | 13 | 14 | logger = logging.getLogger() 15 | 16 | 17 | def main(): 18 | # Read config 19 | config = ConfigParser() 20 | config.readfp(open(os.path.dirname(os.path.abspath(__file__)) + "/smtpd.cfg.default")) 21 | config.read(["smtpd.cfg",]) 22 | 23 | # Configure the logger 24 | logging.basicConfig(level=getattr(logging, config["logging"]["log_level"].upper()), 25 | format='%(levelname)s: %(asctime)s %(message)s', 26 | datefmt='%m/%d/%Y %I:%M:%S %p') 27 | 28 | loop = asyncio.get_event_loop() 29 | 30 | # Init plugin manager 31 | plugin_manager = PluginManager(loop) 32 | 33 | logger.info("Starting smtpd on {}:{}".format(config["smtpd"]["host"], config["smtpd"]["port"])) 34 | cont = CustomIdentController( 35 | MailHandler(loop, config, plugin_manager), 36 | loop=loop, 37 | ident_hostname=config["smtpd"]["hostname"], 38 | ident=config["smtpd"]["ident"], 39 | hostname=config["smtpd"]["host"], 40 | port=config["smtpd"]["port"]) 41 | cont.start() 42 | 43 | # Ugly but whatever, wait until the controller thread finishes (wtf why do they start a thread) 44 | threads = threading.enumerate() 45 | for thread in threads: 46 | if not threading.current_thread() == thread: 47 | thread.join() 48 | 49 | plugin_manager.stop_plugins() 50 | 51 | 52 | if __name__ == "__main__": 53 | main() 54 | 55 | -------------------------------------------------------------------------------- /smtpd/src/mailer.py: -------------------------------------------------------------------------------- 1 | import email 2 | import re 3 | import asyncio 4 | import logging 5 | import functools 6 | from datetime import datetime 7 | from aiosmtpd.controller import Controller 8 | from aiosmtpd.smtp import SMTP as Server 9 | from base64 import b64decode 10 | import storage 11 | 12 | 13 | logger = logging.getLogger() 14 | 15 | 16 | async def create_mailer(handler, loop, ident_hostname, ident, **kwargs): 17 | """Creates and initializes a ``Mailer`` object 18 | 19 | Args: 20 | loop -- asyncio loop. 21 | ident_hostname -- the hostname to use in the ident message. 22 | ident -- the version string. 23 | 24 | Returns: 25 | The Mailer object. 26 | """ 27 | 28 | return Mailer(handler, loop, ident_hostname, ident, **kwargs) 29 | 30 | 31 | class CustomIdentController(Controller): 32 | def __init__(self, handler, loop, ident_hostname, ident, **kwargs): 33 | """Init method for ``CustomIdentController``. 34 | 35 | Args: 36 | handler -- the smtpd MailHandler object. 37 | loop -- the asyncio loop. 38 | ident_hostname -- the hostname to use in the ident message. 39 | ident -- the version string. 40 | """ 41 | 42 | self.loop = loop 43 | self.ident_hostname = ident_hostname 44 | self.ident = ident 45 | super(CustomIdentController, self).__init__(handler, loop=loop, **kwargs) 46 | 47 | def factory(self): 48 | """``CustomIdentController`` factory. 49 | 50 | Overrides ``super.factory()``. 51 | Creates an aiosmtpd server object. 52 | 53 | Returns: 54 | Returns the server object. 55 | """ 56 | 57 | server = Server(self.handler) 58 | server.hostname = self.ident_hostname 59 | server.__ident__ = self.ident 60 | return server 61 | 62 | 63 | class MailHandler: 64 | def __init__(self, loop, config, plugin_manager): 65 | """The init method for the ``MailHandler`` class. 66 | 67 | Args: 68 | loop -- the ``asyncio`` loop. 69 | config -- the sarlacc ``config`` object. 70 | plugin_manager -- the sarlacc ``plugin_manager`` object. 71 | """ 72 | 73 | self.loop = loop 74 | self.config = config 75 | self.plugin_manager = plugin_manager 76 | loop.create_task(self.init_store()) 77 | 78 | 79 | async def init_store(self): 80 | """Intialize the storage backend. 81 | 82 | This will create the storage backend and load and run any plugins. 83 | """ 84 | 85 | # Init storage handlers 86 | self.store = await storage.create_storage(self.config, self.plugin_manager, self.loop) 87 | 88 | self.plugin_manager.load_plugins(self.store, "plugins") 89 | self.plugin_manager.run_plugins() 90 | 91 | 92 | async def handle_DATA(self, server, session, envelope): 93 | """DATA header handler. 94 | 95 | Overrides ``super.handle_DATA`` 96 | This will be called when a DATA header is received by the mail server. 97 | 98 | Args: 99 | server -- the ``aiosmtpd`` server. 100 | session -- the ``aiosmtpd`` session. 101 | envelope -- the data envelope. 102 | 103 | Returns: 104 | The response string to send to the client. 105 | """ 106 | 107 | subject = "" 108 | to_address_list = envelope.rcpt_tos 109 | from_address = envelope.mail_from 110 | body = None 111 | attachments = [] 112 | date_sent = datetime.now() 113 | 114 | # Parse message 115 | try: 116 | message = email.message_from_string(envelope.content.decode("utf8", errors="replace")) 117 | subject = message["subject"] 118 | if message.is_multipart(): 119 | for part in message.get_payload(): 120 | if "Content-Disposition" in part and "attachment;" in part["Content-Disposition"]: 121 | filename = None 122 | matches = re.findall(r'filename=".*"', part["Content-Disposition"]) 123 | if len(matches) > 0: 124 | a = matches[0].index('"') 125 | b = matches[0].index('"', a + 1) 126 | filename = matches[0][a + 1:b] 127 | content = part.get_payload(decode=True) 128 | 129 | attachments.append({ 130 | "content": content, 131 | "filename": filename}) 132 | elif "Content-Type" in part and "text/plain" in part["Content-Type"]: 133 | body = part.get_payload() 134 | elif "Content-Type" in part and "text/html" in part["Content-Type"]: 135 | body = part.get_payload() 136 | else: 137 | # This is gross 138 | if "Content-Disposition" in message and "attachment;" in message["Content-Disposition"]: 139 | filename = None 140 | matches = re.findall(r'filename=".*"', message["Content-Disposition"]) 141 | if len(matches) > 0: 142 | a = matches[0].index('"') 143 | b = matches[0].index('"', a + 1) 144 | filename = matches[0][a + 1:b] 145 | content = message.get_payload(decode=True) 146 | 147 | attachments.append({ 148 | "content": content, 149 | "filename": filename}) 150 | elif "Content-Type" in message and "text/plain" in message["Content-Type"]: 151 | body = message.get_payload() 152 | elif "Content-Type" in message and "text/html" in message["Content-Type"]: 153 | body = message.get_payload() 154 | 155 | asyncio.ensure_future(self.store.store_email( 156 | subject = subject, 157 | to_address_list = to_address_list, 158 | from_address = from_address, 159 | body = body, 160 | attachments = attachments, 161 | date_sent = date_sent)) 162 | 163 | except: 164 | logger.error("Failed to parse mail") 165 | e = sys.exc_info()[0] 166 | logger.error(e) 167 | 168 | return "250 Message accepted for delivery" 169 | -------------------------------------------------------------------------------- /smtpd/src/plugin_manager.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import traceback 4 | from importlib import import_module 5 | from enum import Enum 6 | 7 | 8 | logger = logging.getLogger() 9 | 10 | 11 | class PluginManager(): 12 | def __init__(self, loop): 13 | """Init method for PluginManager. 14 | 15 | Args: 16 | loop -- asyncio loop. 17 | """ 18 | 19 | self.loop = loop 20 | self.plugins = [] 21 | 22 | 23 | def load_plugins(self, store, directory): 24 | """Load plugins from plugin directory. 25 | 26 | Args: 27 | store -- sarlacc store object (provides interface to backend storage). 28 | directory -- path to the directory to load plugins from. 29 | """ 30 | 31 | for name in os.listdir(os.path.join(os.path.dirname(os.path.abspath(__file__)), directory)): 32 | full_path = os.path.join("plugins", name) 33 | if name.startswith("__"): 34 | logger.info("continuing") 35 | continue 36 | elif name.endswith(".py") and not name == "plugin.py": 37 | module_name = name[:-3] 38 | self.__import_module(module_name, store) 39 | elif os.path.isdir(full_path) and os.path.exists(os.path.join(full_path, "__init__.py")): 40 | # This module is in it's own directory 41 | self.__import_module(name, store) 42 | 43 | 44 | def __import_module(self, module_name, store): 45 | """Import a module 46 | 47 | Args: 48 | module_name -- the name of the module to load 49 | store -- sarlacc store object (provides interface to backend storage) 50 | """ 51 | 52 | try: 53 | logger.info("Loading: %s", module_name) 54 | module = import_module("plugins." + module_name) 55 | self.plugins.append(module.Plugin(logger, store)) 56 | logger.info("Loaded plugins/{}".format(module_name)) 57 | except Exception as e: 58 | logger.error("Failed to load plugin/{}".format(module_name)) 59 | logger.error(traceback.format_exc()) 60 | 61 | 62 | def run_plugins(self): 63 | """Run the plugins. 64 | 65 | Calls all the currently loaded plugin's `run` methods. 66 | """ 67 | 68 | for plugin in self.plugins: 69 | plugin.run() 70 | 71 | 72 | def stop_plugins(self): 73 | """Stop the plugins. 74 | 75 | Calls all the currently loaded plugin's `stop` methods. 76 | """ 77 | 78 | for plugin in self.plugins: 79 | plugin.stop() 80 | 81 | 82 | async def emit_new_email_address(self, *args, **kwargs): 83 | """Emit "new email address" signal. 84 | 85 | Inform all plugins that an email address that hasn't been seen before was detected. 86 | """ 87 | 88 | for plugin in self.plugins: 89 | self.loop.create_task(plugin.new_email_address(*args, **kwargs)) 90 | 91 | 92 | async def emit_new_attachment(self, *args, **kwargs): 93 | """Emit "new attachment" signal. 94 | 95 | Inform all plugins that a new attachment that hasn't been seen before was detected. 96 | """ 97 | 98 | for plugin in self.plugins: 99 | self.loop.create_task(plugin.new_attachment(*args, **kwargs)) 100 | 101 | 102 | async def emit_new_mail_item(self, *args, **kwargs): 103 | """Emit "new mail item" signal. 104 | 105 | Inform all plugins that a new email has been received. 106 | """ 107 | 108 | for plugin in self.plugins: 109 | self.loop.create_task(plugin.new_mail_item(*args, **kwargs)) 110 | 111 | -------------------------------------------------------------------------------- /smtpd/src/plugins/directory_example/__init__.py: -------------------------------------------------------------------------------- 1 | from plugins.plugin import SarlaccPlugin 2 | from . import test_module 3 | 4 | class Plugin(SarlaccPlugin): 5 | def run(self): 6 | self.logger.info("This is an example plugin in it's own directory") 7 | 8 | 9 | async def new_attachment(self, _id, sha256, content, filename, tags): 10 | self.logger.info("Plugin alerting to new attachment with sha256: %s", sha256) 11 | 12 | # Example usage of the storage API 13 | attachment = await self.store.get_attachment_by_sha256(sha256) 14 | # attachment = { 15 | # tags[]: a list of tag strings attached to this attachment, 16 | # sha256: the sha256 hash of this attachment, 17 | # content: the raw file, 18 | # filename: the filename, 19 | # _id: the id of the attachment's postgresql record 20 | # } 21 | 22 | 23 | async def new_email_address(self, _id, email_address): 24 | self.logger.info("Plugin alerting to new email address: %s", email_address) 25 | 26 | 27 | async def new_mail_item(self, _id, subject, recipients, from_address, body, date_sent, attachments): 28 | self.logger.info("Plugin alerting to new mail item with subject: %s", subject) 29 | -------------------------------------------------------------------------------- /smtpd/src/plugins/directory_example/test_module.py: -------------------------------------------------------------------------------- 1 | print("Test module loaded") 2 | -------------------------------------------------------------------------------- /smtpd/src/plugins/example.py: -------------------------------------------------------------------------------- 1 | from plugins.plugin import SarlaccPlugin 2 | 3 | class Plugin(SarlaccPlugin): 4 | def run(self): 5 | self.logger.info("This is an example plugin") 6 | 7 | 8 | async def new_attachment(self, _id, sha256, content, filename, tags): 9 | self.logger.info("Plugin alerting to new attachment with sha256: %s", sha256) 10 | 11 | # Example usage of the storage API 12 | attachment = await self.store.get_attachment_by_sha256(sha256) 13 | # attachment = { 14 | # tags[]: a list of tag strings attached to this attachment, 15 | # sha256: the sha256 hash of this attachment, 16 | # content: the raw file, 17 | # filename: the filename, 18 | # _id: the id of the attachment's postgresql record 19 | # } 20 | 21 | 22 | async def new_email_address(self, _id, email_address): 23 | self.logger.info("Plugin alerting to new email address: %s", email_address) 24 | 25 | 26 | async def new_mail_item(self, _id, subject, recipients, from_address, body, date_sent, attachments): 27 | self.logger.info("Plugin alerting to new mail item with subject: %s", subject) 28 | -------------------------------------------------------------------------------- /smtpd/src/plugins/plugin.py: -------------------------------------------------------------------------------- 1 | class SarlaccPlugin: 2 | def __init__(self, logger, store): 3 | """Init method for SarlaccPlugin. 4 | 5 | Args: 6 | logger -- sarlacc logger object. 7 | store -- sarlacc store object. 8 | """ 9 | 10 | self.logger = logger 11 | self.store = store 12 | 13 | 14 | def run(self): 15 | """Runs the plugin. 16 | 17 | This method should be overridden if a plugin needs to do any initial work that isn't purely 18 | initialization. This could include starting any long running jobs / threads. 19 | """ 20 | 21 | pass 22 | 23 | 24 | def stop(self): 25 | """Stops the plugin. 26 | 27 | This method should be overridden if a plugin needs to do any extra cleanup before stopping. 28 | This could include stopping any previously started jobs / threads. 29 | """ 30 | 31 | pass 32 | 33 | 34 | async def new_attachment(self, _id, sha256, content, filename, tags): 35 | """New attachment signal. 36 | 37 | This method is called when a new, previously unseen attachment is detected. 38 | Override this method to be informed about this event. 39 | 40 | Args: 41 | _id -- the attachment postgresql record id. 42 | sha256 -- the sha256 hash of the attachment. 43 | content -- the raw file. 44 | filename -- the filename of the attachment. 45 | tags -- any tags attached to the attachment. 46 | """ 47 | 48 | pass 49 | 50 | 51 | async def new_email_address(self, _id, email_address): 52 | """New email address signal. 53 | 54 | This method is called when a new, previously unseen recipient email address is detected. 55 | Override this method to be informed about this event. 56 | 57 | Args: 58 | _id -- the email address postgresql record id. 59 | email_address -- the email address. 60 | """ 61 | 62 | pass 63 | 64 | 65 | async def new_mail_item(self, _id, subject, recipients, from_address, body, date_sent, attachments): 66 | """New email signal. 67 | 68 | This method is called when an email is received. 69 | Override this method to be informed about this event. 70 | 71 | Args: 72 | _id -- the mail item postgresql record id. 73 | subject -- the email subject. 74 | recipients -- a list of recipient email addresses. 75 | from_address -- the email address in the email's "from" header. 76 | body -- the body of the email. 77 | date_sent -- the date and time the email was sent. 78 | attachments -- a list of attachment objects in the following format: 79 | { 80 | content: the content of the attachment (raw file), 81 | filename: the name of the attachment filename 82 | } 83 | """ 84 | 85 | pass 86 | -------------------------------------------------------------------------------- /smtpd/src/requirements.txt: -------------------------------------------------------------------------------- 1 | asyncio 2 | pytest-asyncio 3 | aiosmtpd 4 | pymongo 5 | motor 6 | psycopg2 7 | ConfigParser 8 | aiopg 9 | -------------------------------------------------------------------------------- /smtpd/src/smtpd.cfg.default: -------------------------------------------------------------------------------- 1 | [smtpd] 2 | host = 0.0.0.0 3 | port = 2500 4 | hostname = mx.google.com 5 | ident = ESMTP m66si1320047qki.162 - gsmtp 6 | 7 | [mongodb] 8 | host = mongodb 9 | port = 27017 10 | 11 | [postgres] 12 | host = postgres 13 | database = sarlacc 14 | user = user 15 | password = user 16 | 17 | [logging] 18 | log_level = INFO 19 | -------------------------------------------------------------------------------- /smtpd/src/storage.py: -------------------------------------------------------------------------------- 1 | from motor.motor_asyncio import AsyncIOMotorClient 2 | import psycopg2 3 | import aiopg 4 | import hashlib 5 | import time 6 | import logging 7 | 8 | 9 | logger = logging.getLogger() 10 | 11 | 12 | async def create_storage(config, plugin_manager, loop): 13 | """Creates and initializes a storage object. 14 | 15 | Args: 16 | plugin_manager -- sarlacc plugin_manager object. 17 | loop -- asyncio loop. 18 | 19 | Returns: 20 | The storage object. 21 | """ 22 | 23 | storage = StorageControl(config, plugin_manager, loop) 24 | await storage._init() 25 | return storage 26 | 27 | 28 | class StorageControl: 29 | def __init__(self, config, plugin_manager, loop): 30 | """Init method for StorageControl class. 31 | 32 | Args: 33 | config -- sarlacc config object 34 | plugin_manager -- sarlacc plugin_manager object 35 | loop -- asyncio loop 36 | """ 37 | 38 | self.config = config 39 | self.plugin_manager = plugin_manager 40 | self.loop = loop 41 | 42 | self.mongo = AsyncIOMotorClient("mongodb://{}:{}".format( 43 | config['mongodb']['host'], 44 | config['mongodb']['port'])) 45 | 46 | 47 | async def _init(self): 48 | """Async init method to be called once inside event loop. 49 | 50 | Used to initialize postgres so we can await on the connect method. 51 | """ 52 | 53 | self.postgres = await self.try_connect_postgres( 54 | host=self.config['postgres']['host'], 55 | database=self.config['postgres']['database'], 56 | user=self.config['postgres']['user'], 57 | password=self.config['postgres']['password']) 58 | 59 | try: 60 | async with self.postgres.acquire() as conn: 61 | async with conn.cursor() as curs: 62 | # create tables if they don't already exist 63 | await curs.execute(''' 64 | CREATE TABLE body ( 65 | id SERIAL PRIMARY KEY, 66 | sha256 text, 67 | content text 68 | ); 69 | 70 | CREATE TABLE mailitem ( 71 | id SERIAL PRIMARY KEY, 72 | datesent timestamp, 73 | subject text, 74 | fromaddress text, 75 | bodyid integer REFERENCES body (id) 76 | ); 77 | 78 | CREATE TABLE recipient ( 79 | id SERIAL PRIMARY KEY, 80 | emailaddress text 81 | ); 82 | 83 | CREATE TABLE mailrecipient ( 84 | id SERIAL PRIMARY KEY, 85 | recipientid integer REFERENCES recipient (id), 86 | mailid integer REFERENCES mailitem (id) 87 | ); 88 | 89 | CREATE TABLE attachment ( 90 | id SERIAL PRIMARY KEY, 91 | mailid integer REFERENCES mailitem (id), 92 | sha256 text, 93 | filename text 94 | ); 95 | ''') 96 | logger.debug("Created fresh database") 97 | except: 98 | pass 99 | 100 | 101 | async def __get_sha256(self, data): 102 | """Calculate sha256 hash of data. 103 | 104 | Args: 105 | data -- the data to hash. 106 | 107 | Returns: 108 | The sha256 hash. 109 | """ 110 | 111 | m = hashlib.sha256() 112 | m.update(data) 113 | return m.hexdigest() 114 | 115 | 116 | async def try_connect_postgres(self, host, user, password, database): 117 | """Loop forever and attempt to connect to postgres. 118 | 119 | Args: 120 | host -- the hostname to connect to. 121 | user -- the username to authenticate as. 122 | password -- the password to authenticate with. 123 | database -- the name of the database to use. 124 | 125 | Returns: 126 | The connected postgres client. 127 | """ 128 | 129 | while True: 130 | logger.info("Trying to connect to postgres... {}@{}/{}".format(user, host, database)) 131 | logger.debug("loop: {}".format(self.loop)) 132 | try: 133 | postgres = await aiopg.create_pool( 134 | loop=self.loop, 135 | host=host, 136 | user=user, 137 | database=database, 138 | password=password) 139 | logger.info("Successfully connected to postgres") 140 | return postgres 141 | except: 142 | logger.warn("Failed to connect to postgres") 143 | time.sleep(5) 144 | 145 | 146 | async def get_attachment_by_selector(self, selector): 147 | """Gets an attachment using a mongodb query. 148 | 149 | Args: 150 | selector -- a query object for mongodb as documented here: https://docs.mongodb.com/manual/reference/method/db.collection.findOne/ 151 | 152 | Returns: 153 | The attachment object in the following format: 154 | { 155 | tags[]: a list of tag strings attached to this attachment, 156 | sha256: the sha256 hash of this attachment, 157 | content: the raw file, 158 | filename: the filename, 159 | _id: the id of the attachment's postgresql record 160 | } 161 | """ 162 | 163 | sarlacc = self.mongo["sarlacc"] 164 | return await sarlacc["samples"].find_one(selector) 165 | 166 | 167 | async def get_attachment_by_id(self, _id): 168 | """Gets an attachment by it's id. 169 | 170 | Args: 171 | _id -- the id of the attachment. 172 | 173 | Returns: 174 | The attachment object in the following format: 175 | { 176 | tags[]: a list of tag strings attached to this attachment, 177 | sha256: the sha256 hash of this attachment, 178 | content: the raw file, 179 | filename: the filename, 180 | _id: the id of the attachment's postgresql record 181 | } 182 | """ 183 | 184 | async with self.postgres.acquire() as conn: 185 | async with conn.cursor() as curs: 186 | await curs.execute(''' 187 | SELECT * FROM attachment 188 | WHERE id=%s; 189 | ''', 190 | (_id,)) 191 | attachment_record = await curs.fetchone() 192 | 193 | attachment = await self.get_attachment_by_selector({"sha256": attachment_record[2]}) 194 | 195 | return { 196 | "_id": attachment_record[0], 197 | "content": attachment["content"], 198 | "filename": attachment["filename"], 199 | "tags": attachment["tags"], 200 | "sha256": attachment["sha256"]} 201 | 202 | 203 | async def get_attachment_by_sha256(self, sha256): 204 | """Gets an attachment by it's sha256 hash. 205 | 206 | Args: 207 | sha256 -- the hash to search for. 208 | 209 | Returns: 210 | The attachment object in the following format: 211 | { 212 | tags[]: a list of tag strings attached to this attachment, 213 | sha256: the sha256 hash of this attachment, 214 | content: the raw file, 215 | filename: the filename, 216 | _id: the id of the attachment's postgresql record 217 | } 218 | """ 219 | 220 | async with self.postgres.acquire() as conn: 221 | async with conn.cursor() as curs: 222 | await curs.execute(''' 223 | SELECT * FROM attachment 224 | WHERE sha256=%s; 225 | ''', 226 | (sha256,)) 227 | attachment_record = await curs.fetchone() 228 | 229 | attachment = await self.get_attachment_by_selector({"sha256": attachment_record[2]}) 230 | 231 | return { 232 | "_id": attachment_record[0], 233 | "content": attachment["content"], 234 | "filename": attachment["filename"], 235 | "tags": attachment["tags"], 236 | "sha256": attachment["sha256"]} 237 | 238 | 239 | async def add_attachment_tag(self, sha256, tag): 240 | """Adds a tag to an attachment. 241 | 242 | Args: 243 | sha256 -- the hash of the attachment to tag. 244 | tag -- the string to tag it with. 245 | """ 246 | 247 | sarlacc = self.mongo["sarlacc"] 248 | await sarlacc["samples"].update_one( 249 | {"sha256": sha256}, 250 | {"$addToSet": 251 | {"tags": tag}}) 252 | 253 | 254 | async def get_email_attachments(self, email_id, content=True): 255 | """Gets an email's attachments. 256 | 257 | Args: 258 | email_id -- the id of the mailitem to get attachments for. 259 | content (boolean) -- set this to False to omit the attachment's actual file content. 260 | Defaults to True. 261 | 262 | Returns: 263 | A list of email attachment objects in the following format: 264 | [{ 265 | tags[]: a list of tag strings attached to this attachment, 266 | sha256: the sha256 hash of this attachment, 267 | content: the raw file, 268 | filename: the filename, 269 | _id: the id of the attachment's postgresql record 270 | }] 271 | """ 272 | 273 | async with self.postgres.acquire() as conn: 274 | async with conn.cursor() as curs: 275 | await curs.execute(''' 276 | SELECT * FROM attachment 277 | WHERE mailid=%s 278 | ''', 279 | (email_id,)) 280 | attachment_records = await curs.fetchall() 281 | 282 | attachments = [] 283 | for record in attachment_records: 284 | # Fetch the content 285 | sarlacc = self.mongo["sarlacc"] 286 | logger.info("Fetching attachment with sha256: %s", record[2]) 287 | 288 | if content: 289 | attachment_info = await sarlacc["samples"].find_one({"sha256": record[2]}) 290 | attachments.append({ 291 | "_id": record[0], 292 | "sha256": record[2], 293 | "filename": record[3], 294 | "content": attachment_info["content"], 295 | "tags": attachment_info["tags"]}) 296 | else: 297 | attachment_info = await sarlacc["samples"].find_one({"sha256": record[2]}, 298 | {"tags": True}) 299 | attachments.append({ 300 | "_id": record[0], 301 | "sha256": record[2], 302 | "filename": record[3], 303 | "tags": attachment_info["tags"]}) 304 | 305 | return attachments 306 | 307 | 308 | async def get_email_recipients(self, email_id): 309 | """Gets an email's recipients. 310 | 311 | Args: 312 | email_id -- the id of the mailitem to get recipients for. 313 | 314 | Returns: 315 | A list of email recipients 316 | ["user@example.com", ...] 317 | """ 318 | async with self.postgres.acquire() as conn: 319 | async with conn.cursor() as curs: 320 | await curs.execute(''' 321 | SELECT * FROM mailrecipient 322 | LEFT JOIN recipient on recipient.id = mailrecipient.recipientid 323 | WHERE mailrecipient.mailid=%s 324 | ''', 325 | (email_id,)) 326 | recipient_records = await curs.fetchall() 327 | 328 | recipients = [] 329 | for record in recipient_records: 330 | recipients.append(record[4]) 331 | 332 | return recipients 333 | 334 | 335 | async def get_email_by_selector(self, selector, attachment_content=True): 336 | """Get email by sql query. 337 | 338 | Gets a mail item using a selector object. 339 | 340 | Args: 341 | selector -- a dict containing values to query for in the following format: 342 | { 343 | "column_name_0": value, 344 | "column_name_1": value 345 | ... 346 | "column_name_n": value 347 | } 348 | Where "column_name" is the name of the column and value is the value you search 349 | for in the `where` clause of the sql query. 350 | attachment_content (boolean): set to false to not return actual file content for attachments. 351 | This is useful if the file is very large. Defaults to True. 352 | 353 | Returns: 354 | An email object in the following format: 355 | { 356 | _id: the id of the email record in postgres, 357 | date_send: the date and time the email was sent, 358 | subject: the email subject, 359 | from_address: the email address in the from header, 360 | recipients: the list of recipient email addresses 361 | body_id: the id of the body record in postgres, 362 | body_sha256: the sha256 hash of the body, 363 | body_content: the content of the body, 364 | attachments: a list of email attachment objects in the following format: 365 | Note: to get attachment content see the get_email_attachments method. 366 | [{ 367 | tags[]: a list of tag strings attached to this attachment, 368 | sha256: the sha256 hash of this attachment, 369 | filename: the filename, 370 | _id: the id of the attachment's postgresql record 371 | }] 372 | } 373 | 374 | Example: 375 | Lets say I wish to get an email that has the subject "test" and the sending email 376 | address "from@example.com", simply use the following: 377 | {"subject": "test", "from_address": "from@example.com"} 378 | """ 379 | 380 | # A list of selector keys matched with the full column name they represent and their value 381 | # (defaults to None) 382 | # 383 | # This also doubles as a whitelist for column names allowed in the query to prevent sqli. 384 | whitelist_columns = { 385 | "_id": "mailitem.id", 386 | "date_sent": "mailitem.datesent", 387 | "subject": "mailitem.subject", 388 | "from_address": "mailitem.fromaddress", 389 | "body_id": "mailitem.bodyid", 390 | "body_sha256": "body.sha256", 391 | "body_content": "body.content"} 392 | 393 | and_operator = False 394 | query_string = """SELECT * FROM mailitem 395 | LEFT JOIN body ON body.id = mailitem.bodyid """ 396 | 397 | values = () 398 | 399 | # Loop over values and add them to the query if they're whitelisted 400 | for key, value in selector.items(): 401 | if key in whitelist_columns: 402 | if not and_operator: 403 | and_operator = True 404 | query_string += "WHERE " 405 | else: 406 | query_string += "AND " 407 | query_string += whitelist_columns[key] + "=%s " 408 | values = values + (value,) 409 | else: 410 | logger.warning("Detected selector key not specified in the whitelist. Key: %s", key) 411 | 412 | query_string += ";" 413 | 414 | async with self.postgres.acquire() as conn: 415 | async with conn.cursor() as curs: 416 | 417 | await curs.execute(query_string, 418 | values) 419 | email = await curs.fetchone() 420 | 421 | return { 422 | "_id": email[0], 423 | "date_sent": email[1], 424 | "subject": email[2], 425 | "from_address": email[3], 426 | "recipients": await self.get_email_recipients(email[0]), 427 | "body_id": email[4], 428 | "body_sha256": email[6], 429 | "body_content": email[7], 430 | "attachments": await self.get_email_attachments(email[0]) 431 | } 432 | 433 | 434 | 435 | async def get_email_by_id(self, email_id): 436 | """Get email by id. 437 | 438 | Gets a mail item by it's id. 439 | 440 | Args: 441 | email_id -- the id of the mail item. 442 | 443 | Returns: 444 | An email object in the following format: 445 | { 446 | _id: the id of the email record in postgres, 447 | date_send: the date and time the email was sent, 448 | subject: the email subject, 449 | from_address: the email address in the from header, 450 | body_id: the id of the body record in postgres, 451 | body_sha256: the sha256 hash of the body, 452 | body_content: the content of the body, 453 | attachments: a list of email attachment objects in the following format: 454 | [{ 455 | tags[]: a list of tag strings attached to this attachment, 456 | sha256: the sha256 hash of this attachment, 457 | content: the raw file, 458 | filename: the filename, 459 | _id: the id of the attachment's postgresql record 460 | }] 461 | } 462 | """ 463 | 464 | async with self.postgres.acquire() as conn: 465 | async with conn.cursor() as curs: 466 | await curs.execute(''' 467 | SELECT * FROM mailitem 468 | LEFT JOIN body ON body.id = mailitem.bodyid 469 | WHERE mailitem.id=%s; 470 | ''', 471 | (email_id,)) 472 | email = await curs.fetchone() 473 | return { 474 | "_id": email[0], 475 | "date_sent": email[1], 476 | "subject": email[2], 477 | "from_address": email[3], 478 | "recipients": await self.get_email_recipients(email[0]), 479 | "body_id": email[4], 480 | "body_sha256": email[6], 481 | "body_content": email[7], 482 | "attachments": await self.get_email_attachments(email_id) 483 | } 484 | 485 | 486 | 487 | async def store_email(self, subject, to_address_list, from_address, body, date_sent, attachments): 488 | """A new email item. 489 | 490 | Args: 491 | subject -- the subject of the email. 492 | to_address_list -- a list of recipient email addresses. 493 | from_address -- the email address in the from header. 494 | body -- the email body. 495 | date_send -- the date and time the email was sent. 496 | attachments -- a list of attachment objects in the following format: 497 | { 498 | content: the content of the attachment (raw file), 499 | filename: the name of the attachment filename 500 | } 501 | """ 502 | 503 | logger.debug("-" * 80) 504 | logger.debug("Subject: %s", subject) 505 | logger.debug("to_address_list: %s", to_address_list) 506 | logger.debug("from_address: %s", from_address) 507 | logger.debug("body: %s", body) 508 | logger.debug("attachment count: %s", len(attachments)) 509 | logger.debug("date_sent: %s", date_sent) 510 | logger.debug("-" * 80) 511 | 512 | async with self.postgres.acquire() as conn: 513 | body_sha256 = await self.__get_sha256(body.encode("utf-8")) 514 | async with conn.cursor() as curs: 515 | logger.debug("curs: {}".format(curs)) 516 | # insert if not existing already, otherwise return existing record 517 | await curs.execute(''' 518 | WITH s AS ( 519 | SELECT id, sha256, content 520 | FROM body 521 | WHERE sha256 = %s 522 | ), i as ( 523 | INSERT INTO body (sha256, content) 524 | SELECT %s, %s 525 | WHERE NOT EXISTS (SELECT 1 FROM s) 526 | RETURNING id, sha256, content 527 | ) 528 | SELECT id, sha256, content 529 | FROM i 530 | UNION ALL 531 | SELECT id, sha256, content 532 | FROM s; 533 | ''', 534 | (body_sha256, body_sha256, body,)) 535 | bodyRecord = await curs.fetchone() 536 | bodyId = bodyRecord[0] 537 | logger.debug("Body ID: {}".format(bodyId)) 538 | 539 | # add a mailitem 540 | await curs.execute("INSERT INTO mailitem (datesent, subject, fromaddress, bodyid) values (%s, %s, %s, %s) returning *;", 541 | (date_sent, subject, from_address, bodyId,)) 542 | mailitem = await curs.fetchone() 543 | 544 | # add recipients 545 | recipientList = [] 546 | for recipient in to_address_list: 547 | # insert if not existing already, otherwise return existing record 548 | await curs.execute(''' 549 | WITH s AS ( 550 | SELECT id, emailaddress 551 | FROM recipient 552 | WHERE emailaddress = %s 553 | ), i as ( 554 | INSERT INTO recipient (emailaddress) 555 | SELECT %s 556 | WHERE NOT EXISTS (SELECT 1 FROM s) 557 | RETURNING id, emailaddress 558 | ) 559 | SELECT id, emailaddress 560 | FROM i 561 | UNION ALL 562 | SELECT id, emailaddress 563 | FROM s; 564 | ''', 565 | (recipient, recipient,)) 566 | recipientRecord = await curs.fetchone() 567 | recipientList.append(recipientRecord) 568 | 569 | # link this recipient to the mailitem 570 | await curs.execute("INSERT INTO mailrecipient (recipientid, mailid) values (%s, %s);", (recipientRecord[0], mailitem[0])) 571 | 572 | # check if this is a new email address in the recipient list and if so, inform registered plugins 573 | if recipient is not recipientRecord[1]: 574 | # new email address 575 | await self.plugin_manager.emit_new_email_address( 576 | _id=recipientRecord[0], 577 | email_address=recipientRecord[1]) 578 | 579 | 580 | if attachments != None: 581 | for attachment in attachments: 582 | attachment_sha256 = await self.__get_sha256(attachment["content"]) 583 | attachment["sha256"] = attachment_sha256 584 | attachment["tags"] = [] 585 | await curs.execute("INSERT INTO attachment (sha256, mailid, filename) values (%s, %s, %s) returning *;", 586 | (attachment_sha256, mailitem[0], attachment["filename"],)) 587 | 588 | attachment_record = await curs.fetchone() 589 | attachment["id"] = attachment_record[0] 590 | 591 | # check if attachment has been seen, if not store it in mongodb 592 | sarlacc = self.mongo['sarlacc'] 593 | 594 | logger.info("Checking if attachment already in db") 595 | write_result = await sarlacc["samples"].update( 596 | {"sha256": attachment_sha256}, 597 | { 598 | "sha256": attachment_sha256, 599 | "content": attachment["content"], 600 | "filename": attachment["filename"], 601 | "tags": [] 602 | }, 603 | True) 604 | 605 | if not write_result["updatedExisting"]: 606 | # inform plugins of new attachment 607 | await self.plugin_manager.emit_new_attachment( 608 | _id=attachment_record[0], 609 | sha256=attachment_sha256, 610 | content=attachment["content"], 611 | filename=attachment["filename"], 612 | tags=[]) 613 | 614 | # inform plugins 615 | await self.plugin_manager.emit_new_mail_item(mailitem[0], subject, to_address_list, from_address, body, date_sent, attachments) 616 | 617 | -------------------------------------------------------------------------------- /smtpd/src/test/attachment.txt: -------------------------------------------------------------------------------- 1 | testing attachment 2 | 3 | this 4 | is 5 | a 6 | test 7 | -------------------------------------------------------------------------------- /smtpd/src/test/send_email.py: -------------------------------------------------------------------------------- 1 | # Settings 2 | 3 | SMTP_SERVER = 'localhost' 4 | SMTP_PORT = 2500 5 | SMTP_USERNAME = 'myusername' 6 | SMTP_PASSWORD = '$uper$ecret' 7 | SMTP_FROM = 'sender@example.com' 8 | SMTP_TO = ['recipient@example.com', 'test@example.com'] 9 | SMTP_SUBJECT = 'testing' 10 | 11 | TEXT_FILENAME = 'attachment.txt' 12 | MESSAGE = """This is the message 13 | to be sent to the client. 14 | """ 15 | 16 | import os 17 | import sys 18 | import smtplib, email 19 | from email import encoders 20 | 21 | TEXT_FILEPATH = os.path.dirname(os.path.abspath(__file__)) + "/" + TEXT_FILENAME 22 | 23 | # Now construct the message 24 | msg = email.MIMEMultipart.MIMEMultipart() 25 | body = email.MIMEText.MIMEText(MESSAGE) 26 | attachment = email.MIMEBase.MIMEBase('text', 'plain') 27 | attachment.set_payload(open(TEXT_FILEPATH).read()) 28 | attachment.add_header('Content-Disposition', 'attachment', filename=os.path.basename(TEXT_FILENAME)) 29 | encoders.encode_base64(attachment) 30 | msg.attach(attachment) 31 | msg.attach(body) 32 | msg.add_header('From', SMTP_FROM) 33 | msg.add_header('To', ", ".join(SMTP_TO)) 34 | msg.add_header('Subject', SMTP_SUBJECT) 35 | 36 | # Now send the message 37 | mailer = smtplib.SMTP(SMTP_SERVER, SMTP_PORT) 38 | # EDIT: mailer is already connected 39 | # mailer.connect() 40 | # mailer.login(SMTP_USERNAME, SMTP_PASSWORD) 41 | mailer.sendmail(SMTP_FROM, SMTP_TO, msg.as_string()) 42 | mailer.close() 43 | print("done") 44 | -------------------------------------------------------------------------------- /smtpd/src/test_storage.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import storage 3 | import os 4 | import pytest 5 | from plugin_manager import PluginManager 6 | from configparser import ConfigParser 7 | import datetime 8 | 9 | 10 | @pytest.yield_fixture(scope="module") 11 | def event_loop(request): 12 | loop = asyncio.get_event_loop_policy().new_event_loop() 13 | yield loop 14 | loop.close() 15 | 16 | 17 | @pytest.fixture(scope="module") 18 | def config(): 19 | # Read config 20 | config = ConfigParser() 21 | config.readfp(open(os.path.dirname(os.path.abspath(__file__)) + "/smtpd.cfg.default")) 22 | config.read(["smtpd.cfg",]) 23 | return config 24 | 25 | 26 | @pytest.mark.asyncio 27 | @pytest.fixture(scope="module") 28 | async def plugin_manager(event_loop): 29 | return PluginManager(event_loop) 30 | 31 | 32 | @pytest.mark.asyncio 33 | @pytest.fixture(scope="module") 34 | async def store(event_loop, config, plugin_manager): 35 | return await storage.create_storage(config, plugin_manager, event_loop) 36 | 37 | 38 | @pytest.mark.asyncio 39 | async def test_create_Storage(event_loop, config, plugin_manager): 40 | store = await storage.create_storage(config, plugin_manager, event_loop) 41 | 42 | 43 | @pytest.mark.asyncio 44 | async def test_store_email(store): 45 | """This also doubles as a test for get_email_by_selector 46 | """ 47 | now = datetime.datetime.now() 48 | await store.store_email( 49 | "test subject", 50 | ["test_recipient_1@example.com"], 51 | "from_address@example.com", 52 | "test body", 53 | now, 54 | [{"content": b"Test content", 55 | "filename": "testfile.txt"}]) 56 | 57 | email = await store.get_email_by_selector({"date_sent": now}) 58 | 59 | assert email["subject"] == "test subject" 60 | assert "test_recipient_1@example.com" == email["recipients"][0] 61 | assert len(email["recipients"]) == 1 62 | assert email["date_sent"] == now 63 | assert email["attachments"][0]["content"] == b"Test content" 64 | assert email["attachments"][0]["filename"] == "testfile.txt" 65 | 66 | -------------------------------------------------------------------------------- /smtpd/src/util/mailslurper_import.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os, sys 4 | sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) 5 | 6 | import storage 7 | import mysql.connector 8 | import psycopg2 9 | import re 10 | from base64 import b64encode, b64decode 11 | from configparser import ConfigParser 12 | import asyncio 13 | 14 | 15 | def cleanup_address(addr): 16 | return addr[1:len(addr)-1] 17 | 18 | 19 | async def main(): 20 | config = ConfigParser() 21 | config.read("./smtpd.cfg") 22 | 23 | 24 | store = storage.StorageControl(config) 25 | 26 | cnx = mysql.connector.connect( 27 | user="root", password="root", 28 | host="localhost", 29 | database="sarlacc") 30 | 31 | mysql_cursor = cnx.cursor() 32 | 33 | mysql_cursor.execute("SELECT dateSent, fromAddress, toAddressList, subject, body FROM mailitem;") 34 | 35 | for (dateSent, fromAddress, toAddressList, subject, body) in mysql_cursor: 36 | # tidy up fromAddress 37 | fromAddress = cleanupAddress(re.findall(r"<(.*?)>", fromAddress)[0]) 38 | 39 | # tidy up toaAdressList 40 | toAddressList = re.findall(r"<(.*?)>", toAddressList) 41 | 42 | body = str(b64decode(body)) 43 | 44 | store.store_email(subject, toAddressList, fromAddress, body, dateSent, []) 45 | 46 | mysql_cursor.close() 47 | cnx.close() 48 | 49 | 50 | loop = asyncio.get_event_loop() 51 | loop.run_until_complete(main()) 52 | loop.close() 53 | -------------------------------------------------------------------------------- /web/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.6 2 | ADD ./src /sarlacc 3 | WORKDIR /sarlacc 4 | RUN pip install -r requirements.txt 5 | CMD ["python", "app.py"] 6 | -------------------------------------------------------------------------------- /web/src/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from redis import Redis 3 | 4 | app = Flask(__name__) 5 | redis = Redis(host='redis', port=6379) 6 | 7 | @app.route('/') 8 | def hello(): 9 | count = redis.incr('hits') 10 | return 'Testing. Hello World! I have been seen {} times.\n'.format(count) 11 | 12 | if __name__ == "__main__": 13 | app.run(host="0.0.0.0", debug=True) 14 | -------------------------------------------------------------------------------- /web/src/requirements.txt: -------------------------------------------------------------------------------- 1 | flask 2 | redis 3 | --------------------------------------------------------------------------------