├── Dockerfile ├── LICENSE ├── README.md ├── app ├── config-sample.json ├── main.py └── plugins │ ├── __init__.py │ ├── hybrid_analysis.py │ ├── malshare.py │ ├── virusbay.py │ └── virustotal.py ├── docker-compose.yml └── requirements.txt /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.4-alpine 2 | ADD . /code 3 | WORKDIR /code 4 | RUN pip install -r requirements.txt 5 | WORKDIR /code/app 6 | CMD ["python", "main.py"] -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Paul 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | metasearch 2 | =========== 3 | 4 | Purpose: stop searching for sample hashes on 10 different sites. 5 | This is a simple Python3 Flask application running on port 5000 interacting with various platforms (TBC) and caching the results in a Redis database for faster responses. 6 | 7 | ### Installation 8 | 9 | Git clone the repository: 10 | 11 | ```bash 12 | $ git clone https://github.com/PaulSec/metasearch-public.git 13 | $ cd metasearch-public 14 | ``` 15 | 16 | 17 | Add your API tokens (and Redis parameters) for the specific plugins in the app/config-sample.json file: 18 | 19 | ```json 20 | { 21 | "hybrid_analysis": { 22 | "api": "XXXXXXXXXXXXXXXXXX", 23 | "secret": "XXXXXXXXXXXXXXXXXX" 24 | }, 25 | "malshare": { 26 | "api": "XXXXXXXXXXXXXXXXXX" 27 | }, 28 | "redis_host": "redis", 29 | "redis_port": 6379 30 | } 31 | ``` 32 | 33 | 34 | Finally, rename it from ```config-sample.json``` to ```config.json``` 35 | 36 | ### Quickstart (with docker-compose) 37 | 38 | Then, use docker-compose in the metasearch directory: 39 | 40 | ```bash 41 | $ docker-compose up 42 | Recreating metasearch_web_1 ... 43 | Recreating metasearch_web_1 44 | Starting metasearch_redis_1 ... 45 | Recreating metasearch_web_1 ... done 46 | Attaching to metasearch_redis_1, metasearch_web_1 47 | redis_1 | 1:C 23 Feb 20:12:16.838 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 48 | redis_1 | 1:C 23 Feb 20:12:16.840 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1, just started 49 | redis_1 | 1:C 23 Feb 20:12:16.840 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 50 | redis_1 | 1:M 23 Feb 20:12:16.845 * Running mode=standalone, port=6379. 51 | redis_1 | 1:M 23 Feb 20:12:16.845 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 52 | redis_1 | 1:M 23 Feb 20:12:16.845 # Server initialized 53 | redis_1 | 1:M 23 Feb 20:12:16.845 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 54 | redis_1 | 1:M 23 Feb 20:12:16.848 * DB loaded from disk: 0.003 seconds 55 | redis_1 | 1:M 23 Feb 20:12:16.848 * Ready to accept connections 56 | web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) 57 | web_1 | * Restarting with stat 58 | web_1 | * Debugger is active! 59 | web_1 | * Debugger PIN: 216-090-375 60 | web_1 | 172.20.0.1 - - [23/Feb/2018 20:12:45] "GET /plugins HTTP/1.1" 200 - 61 | ``` 62 | 63 | The service is accessible at ```http://0.0.0.0:5000```. You can check by typing: 64 | 65 | ```bash 66 | $ docker ps -a 67 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 68 | 3ed6edac232d metasearch_web "python main.py" About an hour ago Up About an hour 0.0.0.0:5000->5000/tcp metasearch_web_1 69 | 6bddda639254 redis:alpine "docker-entrypoint..." 2 hours ago Up About an hour 6379/tcp metasearch_redis_1 70 | ``` 71 | 72 | ### Interacting with the API 73 | 74 | Those are the different API endpoint accessible: 75 | 76 | HTTP Method | URI | HTTP Method 77 | --- | --- | --- 78 | |GET | /plugins | Lists all the plugins loaded within the application | 79 | |GET | /hybrid_analysis/hash | Will check the hash provided on [Hybrid-analysis](https://www.hybrid-analysis.com/) | 80 | |GET | /virustotal/hash | Will check the hash provided on [VirusTotal](https://www.virustotal.com/) | 81 | |GET | /malshare/hash | Will check the hash provided on [MalShare](http://malshare.com/) | 82 | |GET | /virusbay/hash | Will check the hash provided on [VirusBay](https://beta.virusbay.io) | 83 | |GET | /search/hash | Will check on all the platforms listed above | 84 | 85 | ### Examples: 86 | 87 | ##### Retrieving all the plugins 88 | 89 | ```bash 90 | $ curl http://0.0.0.0:5000/plugins -s | jq . 91 | [ 92 | "virustotal", 93 | "malshare", 94 | "virusbay", 95 | "hybrid_analysis" 96 | ] 97 | ``` 98 | 99 | #### Looking up ```d84769d63aa6b8718ab4bd86e27e26a4``` on MalShare. 100 | 101 | ```bash 102 | $ curl http://0.0.0.0:5000/malshare/d84769d63aa6b8718ab4bd86e27e26a4 -s | jq . 103 | { 104 | "found": true, 105 | "data": { 106 | "SHA1": "78cac2c75b0fe9e7d3819341a451dabcad4d7678", 107 | "MD5": "d84769d63aa6b8718ab4bd86e27e26a4", 108 | "F_TYPE": "PE32", 109 | "SHA256": "c2c855b71cc8b1c1c731f4cadab8a24db4cd8b66f8583cb9640c35d296baf6b0", 110 | "SOURCES": [ 111 | "http://109.234.36.233/bot/Miner/bin/Release/LoaderBot.exe" 112 | ], 113 | "SSDEEP": "384:fKxvDuPNItH19GTXjdh8duujYcV6AUwJFZb:f44atV9AhsfYcV6Dw9b" 114 | }, 115 | "name": "malshare" 116 | } 117 | ``` 118 | 119 | 120 | ##### Looking up ```2dd395cbd297e8b40a4b64b3bb21e655``` on all the platforms. 121 | 122 | ```bash 123 | $ curl http://0.0.0.0:5000/search/2dd395cbd297e8b40a4b64b3bb21e655 -s | jq . | more 124 | [ 125 | { 126 | "links": { 127 | "self": "https://www.virustotal.com/ui/search?query=2dd395cbd297e8b40a4b64b3bb21e655&relationships[url]=network_location%2Clast_serving_ip_address&relationships[comment]=author%2Citem" 128 | }, 129 | "data": [ 130 | { 131 | "attributes": { 132 | "names": [ 133 | "482931ee6c24d9ead3e4024b62106286992cfa3d", 134 | "bash" 135 | ], 136 | "elf_info": { 137 | "imports": [ 138 | [ 139 | "__deregister_frame_info", 140 | "NOTYPE" 141 | ], 142 | [ 143 | "__pthread_initialize_minimal", 144 | "NOTYPE" 145 | ], 146 | 147 | [..redacted..] 148 | 149 | "type": "file" 150 | } 151 | ], 152 | "found": true, 153 | "name": "virustotal" 154 | }, 155 | { 156 | "found": false, 157 | "data": [], 158 | "name": "malshare" 159 | }, 160 | { 161 | "search": [ 162 | { 163 | "tags": [ 164 | { 165 | "__v": 0, 166 | "isHash": false, 167 | "_id": "5a3b6199697fdd3b4ded78f6", 168 | "lowerCaseName": "elf", 169 | "name": "elf" 170 | }, 171 | { 172 | "__v": 0, 173 | "isHash": false, 174 | "_id": "5a3b6199697fdd3b4ded78f7", 175 | "lowerCaseName": "linux", 176 | "name": "linux" 177 | [..redacted..] 178 | ``` 179 | 180 | License 181 | ======== 182 | 183 | This project has been released under MIT License. 184 | Contributions are more than welcome. Ping me on Twitter [@PaulWebSec](https://twitter.com/PaulWebSec) if you want some help for that. 185 | -------------------------------------------------------------------------------- /app/config-sample.json: -------------------------------------------------------------------------------- 1 | { 2 | "hybrid_analysis": { 3 | "api": "XXXXXXXXXXXXXXXXXX", 4 | "secret": "XXXXXXXXXXXXXXXXXX" 5 | }, 6 | "malshare": { 7 | "api": "XXXXXXXXXXXXXXXXXX" 8 | }, 9 | "redis_host": "redis", 10 | "redis_port": 6379 11 | } -------------------------------------------------------------------------------- /app/main.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/python 2 | # -*- coding:utf-8 -*- 3 | 4 | from flask import Flask, json, jsonify 5 | import requests, json, sys, redis, os 6 | 7 | app = Flask(__name__) 8 | 9 | @app.route('/') 10 | def index(): 11 | return "Hello !" 12 | 13 | # Endpoint to list all the plugins 14 | @app.route('/plugins') 15 | def retrieve_plugins(): 16 | plugins_names = list(plugins) 17 | print(plugins_names) 18 | response = app.response_class( 19 | response=json.dumps(plugins_names), 20 | status=200, 21 | mimetype='application/json' 22 | ) 23 | return response 24 | 25 | # Search with all the plugins 26 | @app.route('/search/') 27 | def search(query): 28 | # Iterate over all the plugins 29 | result = [] 30 | for plugin in plugins: 31 | # checking in the Redis if the entry is already there 32 | test_redis = cache.get('{}_{}'.format(plugin, query)) 33 | if test_redis: 34 | test_redis = test_redis.decode('utf-8') 35 | tmp_result = json.loads(test_redis) 36 | else: 37 | tmp_result = plugins[plugin]['check'](query) 38 | cache.setex('{}_{}'.format(plugin, query), json.dumps(tmp_result), 3600) 39 | result.append(tmp_result) 40 | 41 | response = app.response_class( 42 | response=json.dumps(result), 43 | status=200, 44 | mimetype='application/json' 45 | ) 46 | return response 47 | 48 | @app.route('//') 49 | def search_provider(provider, query): 50 | # Checking if the provider is listed 51 | if provider not in list(plugins): 52 | response = app.response_class( 53 | response=None, 54 | status=404 55 | ) 56 | return response 57 | 58 | # checking in the Redis if the entry is already there 59 | test_redis = cache.get('{}_{}'.format(provider, query)) 60 | if test_redis: 61 | test_redis = test_redis.decode('utf-8') 62 | result = json.loads(test_redis) 63 | response = app.response_class( 64 | response=json.dumps(result), 65 | status=200 if result['found'] else 404, 66 | mimetype='application/json' 67 | ) 68 | return response 69 | 70 | # If not, it fetches it 71 | result = plugins[provider]['check'](query) 72 | cache.setex('{}_{}'.format(provider, query), json.dumps(result), 3600) 73 | response = app.response_class( 74 | response=json.dumps(result), 75 | status=200 if result['found'] else 404, 76 | mimetype='application/json' 77 | ) 78 | return response 79 | 80 | if __name__ == '__main__': 81 | 82 | path = "plugins/" 83 | plugins = {} 84 | 85 | # Retrieve the configuration 86 | with open('config.json') as f: 87 | config = json.loads(f.read()) 88 | 89 | # Load plugins 90 | sys.path.insert(0, path) 91 | for f in os.listdir(path): 92 | fname, ext = os.path.splitext(f) 93 | if ext == '.py': 94 | try: 95 | mod = __import__(fname) 96 | plugin_config = None 97 | if fname in config: 98 | plugin_config = config[fname] 99 | plugin = mod.Plugin(plugin_config) 100 | tmp_res = plugin.register() 101 | plugin_name = list(tmp_res)[0] 102 | plugin_functions = tmp_res[plugin_name] 103 | plugins[plugin_name] = plugin_functions 104 | # print(plugins) 105 | except Exception as err: 106 | print('[!] Problem loading plugin with file {}'.format(fname)) 107 | print(err) 108 | 109 | cache = redis.Redis(host=config['redis_host'], port=config['redis_port']) 110 | print(cache) 111 | 112 | app.run(host='0.0.0.0', debug=True) 113 | 114 | -------------------------------------------------------------------------------- /app/plugins/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PaulSec/metasearch-public/75218d20dc366b1ad37992a3c146289098a953e1/app/plugins/__init__.py -------------------------------------------------------------------------------- /app/plugins/hybrid_analysis.py: -------------------------------------------------------------------------------- 1 | import requests 2 | 3 | plugin_name = 'hybrid_analysis' 4 | config = None 5 | 6 | def check(query): 7 | api = config['api'] 8 | secret = config['secret'] 9 | url = 'https://www.hybrid-analysis.com/api/search?query={}&secret={}&apikey={}'.format(query, secret, api) 10 | req = requests.get(url).json() 11 | res = req 12 | res['found'] = True if res['response']['result'] != [] else False 13 | res['name'] = 'hybrid-analysis' 14 | return res 15 | 16 | class Plugin: 17 | def __init__(self, conf): 18 | global config 19 | config = conf 20 | 21 | def register(self): 22 | return {plugin_name: {'check': check}} -------------------------------------------------------------------------------- /app/plugins/malshare.py: -------------------------------------------------------------------------------- 1 | import requests 2 | 3 | plugin_name = 'malshare' 4 | config = None 5 | 6 | def check(query): 7 | API_KEY = config['api'] 8 | url ='https://malshare.com/api.php?api_key={}&action=details&hash={}'.format(API_KEY, query) 9 | print(url) 10 | req = requests.get(url) 11 | res = {} 12 | res['found'] = True if b'Sample not found by hash' not in req.content else False 13 | res['data'] = req.json() if res['found'] else [] 14 | res['name'] = 'malshare' 15 | return res 16 | 17 | class Plugin: 18 | def __init__(self, conf): 19 | global config 20 | config = conf 21 | 22 | def register(self): 23 | return {plugin_name: {'check': check}} -------------------------------------------------------------------------------- /app/plugins/virusbay.py: -------------------------------------------------------------------------------- 1 | import requests 2 | 3 | plugin_name = 'virusbay' 4 | 5 | def check(query): 6 | url = 'https://beta.virusbay.io/sample/search?q={}'.format(query) 7 | req = requests.get(url).json() 8 | res = req 9 | res['found'] = True if req['search'] != [] else False 10 | res['name'] = 'virusbay' 11 | return res 12 | 13 | class Plugin: 14 | def __init__(self, conf): 15 | pass 16 | 17 | def register(self): 18 | return {plugin_name: {'check': check}} 19 | -------------------------------------------------------------------------------- /app/plugins/virustotal.py: -------------------------------------------------------------------------------- 1 | import requests 2 | 3 | plugin_name = 'virustotal' 4 | 5 | def check(query): 6 | url = 'https://www.virustotal.com/ui/search?query={}&relationships[url]=network_location%2Clast_serving_ip_address&relationships[comment]=author%2Citem'.format(query) 7 | req = requests.get(url).json() 8 | res = req 9 | res['found'] = True if len(req['data']) > 0 and 'attributes' in req['data'][0] else False 10 | res['name'] = 'virustotal' 11 | return res 12 | 13 | class Plugin: 14 | def __init__(self, conf): 15 | pass 16 | 17 | def register(self): 18 | return {plugin_name: {'check': check}} -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | web: 4 | build: . 5 | ports: 6 | - "5000:5000" 7 | redis: 8 | image: "redis:alpine" -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | flask 2 | requests 3 | redis --------------------------------------------------------------------------------