├── requirements.txt ├── .gitignore ├── slackconfig.py ├── LICENSE ├── README.md └── keye.py /requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | keye.db 3 | 4 | -------------------------------------------------------------------------------- /slackconfig.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | #Slack webhook for notifications 4 | posting_webhook = "https://hooks.slack.com/services/" 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Çlirim Emini 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## What is Keye? 2 | Keye is a reconnaissance tool that was written in Python with SQLite3 integrated. After adding a single URL, or a list of URLs, it will make a request to these URLs and try to detect changes based on their response's body length. This tool is supposed to be scheduled to run periodically at fixed times, dates, or intervals (Ideally each day). Recognized changes of URLs will be sent to Slack workspace with a notification push. 3 | 4 | #### Thanks to [Yassine Aboukir](https://twitter.com/Yassineaboukir) for his help and support on this project. 5 | ## Requirements 6 | - Virtual Private Server (VPS) running on Unix. 7 | - Python 2.x or 3.x. 8 | - Free Slack workspace. 9 | 10 | ## Installation & Configuration 11 | First of all, you will have to clone the tool from Github to your server: 12 | > $ git clone https://github.com/clirimemini/Keye.git && cd Keye 13 | 14 | Install the dependencies: 15 | 16 | >$ pip install -r requirements.txt (or pip3 install -r requirements.txt) 17 | 18 | Next step is to create a Slack workspace, to where new recognized changes of URLs will be sent to. Just browse to [https://slack.com/](https://slack.com/) and create a free account. 19 | 20 | Then, create a channel on your workspace. 21 | 22 | Next, we need to generate a Webhook URL for our channel so we can leverage Slack API. 23 | 24 | Browse to [https://api.slack.com/apps](https://api.slack.com/apps) and create a new application. 25 | 26 | Browse to `Incoming Webhooks` and create a Webhook, and link it to the channel that we created previously. 27 | 28 | You will obtain a link in the following format: 29 | 30 | `https://hooks.slack.com/services/XXXXXXX/BF0XXXXME/XXXXXXXXXXXXX` 31 | 32 | Copy that link and edit `slackconfig.py` 33 | 34 | Now, we need to add a new Cron job to schedule execution of Keye at given time. To do it, type: 35 | > $ crontab -e 36 | 37 | Add the following line at the end of the Cron file: 38 | 39 | `0 */12 * * * cd /root/Keye/ && /usr/bin/python /root/Keye/keye.py` 40 | 41 | Now, we're done with installing and configuring Keye. 42 | 43 | ## Usage 44 | > $ python keye.py -h 45 | 46 | Short Form | Long Form | Description 47 | ------------- | ------------- |------------- 48 | -s | --singleurl | Single URL to add. E.g: http://google.com 49 | -ul | --urlslist | File with new URLs to add. E.g: urls.txt 50 | -rm | --remove | URL to remove from database. E.g: http://google.com 51 | -d | --display | Display all monitored URLs. 52 | 53 | ## Feedback and issues? 54 | If you have any feedback, anything that you want to see implemented or if you're running into issues using Keye, please feel free to file an issue on [https://github.com/clirimemini/Keye/issues](https://github.com/clirimemini/Keye/issues) 55 | -------------------------------------------------------------------------------- /keye.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import argparse 4 | import requests 5 | import sqlite3 6 | import os 7 | from slackconfig import * 8 | import json 9 | requests.packages.urllib3.disable_warnings() 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('-s', '--singleurl', help='Single URL to add. E.g: http://google.com', dest='singleurl') 13 | parser.add_argument('-ul', '--urlslist', help='File with new urls to add. E.g: urls.txt', dest='urlslist') 14 | parser.add_argument('-rm', '--remove', help='URL to remove from database. E.g: http://google.com', dest='urltoremove') 15 | parser.add_argument('-d', '--display', help='Display all monitored URLs', required = False, nargs='?', const="True", dest='displayurls') 16 | args = parser.parse_args() 17 | 18 | def db_install(): 19 | if (os.path.isfile('./keye.db')) == False: 20 | db = sqlite3.connect('keye.db') 21 | cursor = db.cursor() 22 | cursor.execute(''' 23 | CREATE TABLE urls(id INTEGER PRIMARY KEY, url TEXT, 24 | reslength INTEGER)''') 25 | db.commit() 26 | db.close() 27 | else: 28 | pass 29 | 30 | def addsingleurl(): 31 | url = args.singleurl 32 | request(url) 33 | 34 | def addurlsfromlist(): 35 | urlslist = open(args.urlslist, "r") 36 | for url in urlslist: 37 | url = url.rstrip() 38 | request(url) 39 | 40 | def request(url): 41 | try: 42 | if not "http" in url: 43 | url = "http://" + url 44 | req = requests.get(url, allow_redirects=True, verify=False, timeout=5) 45 | reslength = len(req.text) 46 | try: 47 | if not check_if_present(url): 48 | committodb(url, reslength) 49 | print("We have successfully added the URL to be monitored.") 50 | else: 51 | print("This URL already exists on the database.") 52 | except Exception as e: 53 | print(e) 54 | 55 | except: 56 | try: 57 | url = url.replace("http://", "https://") 58 | req = requests.get(url, allow_redirects=True, timeout=5) 59 | reslength = len(req.text) 60 | if not check_if_present(url): 61 | committodb(url, reslength) 62 | print("We have successfully added the URL to be monitored.") 63 | else: 64 | print("This URL already exists on the database.") 65 | print("We have successfully added the URL to be monitored.") 66 | except Exception as e: 67 | print("We could not connect to {} due to following error: {}".format(url, e)) 68 | 69 | def committodb(url, reslength): 70 | try: 71 | cursor.execute('''INSERT INTO urls(url, reslength) 72 | VALUES(?,?)''', (url, reslength)) 73 | db.commit() 74 | except Exception as e: 75 | print(e) 76 | 77 | def getfromdb(): 78 | try: 79 | cursor.execute('''SELECT id, url, reslength FROM urls''') 80 | all_rows = cursor.fetchall() 81 | for row in all_rows: 82 | id = row[0] 83 | url = row[1] 84 | reslength = str(row[2]) 85 | connect(id, url, reslength) 86 | except Exception as e: 87 | print(e) 88 | 89 | def connect(id, url, reslength): 90 | try: 91 | req = requests.get(url, allow_redirects=True, verify=False, timeout=5) 92 | newreslength = len(req.text) 93 | if int(newreslength) == int(reslength): 94 | pass 95 | else: 96 | notify(url) 97 | cursor.execute('''UPDATE urls SET reslength = ? WHERE id = ? ''', (newreslength, id)) 98 | db.commit() 99 | except Exception as e: 100 | print("We could not connect to {} due to following error: {}".format(url, e)) 101 | 102 | def removefromdb(): 103 | urltoremove = args.urltoremove 104 | try: 105 | cursor.execute('''DELETE FROM urls WHERE url = ? ''', (urltoremove,)) 106 | db.commit() 107 | print("\nWe have successfully removed the URL from the database.") 108 | except Exception as e: 109 | print(e) 110 | 111 | def displayurls(): 112 | try: 113 | cursor.execute('''SELECT url from urls''') 114 | all_rows = cursor.fetchall() 115 | for row in all_rows: 116 | print(row[0]) 117 | except Exception as e: 118 | print("We couldn't retrieve URLs due to following error {}".format(e)) 119 | 120 | 121 | def check_if_present(url): 122 | try: 123 | cursor.execute('''SELECT * from urls where url = ? ''',(url,)) 124 | if cursor.fetchall(): 125 | return True 126 | else: 127 | return False 128 | except Exception as e: 129 | return False 130 | 131 | 132 | def notify(url): 133 | webhook_url = posting_webhook 134 | slack_data = {'text': 'Changes detected on: ' + url} 135 | sendnotification = requests.post(webhook_url, data=json.dumps(slack_data), headers={'Content-Type': 'application/json'}) 136 | 137 | db_install() 138 | db = sqlite3.connect('keye.db') 139 | cursor = db.cursor() 140 | 141 | if args.singleurl: 142 | addsingleurl() 143 | elif args.urlslist: 144 | addurlsfromlist() 145 | elif args.urltoremove: 146 | removefromdb() 147 | elif args.displayurls: 148 | displayurls() 149 | else: 150 | getfromdb() 151 | 152 | db.close() 153 | --------------------------------------------------------------------------------