├── .gitmodules ├── LICENSE ├── README.md ├── extension_guide.md ├── ideas.md ├── images ├── architecture.png ├── card_creation.png ├── logo.png ├── modules_engine.png └── telegram_alerts.png ├── local_install.md └── tour_hosted_version.md /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "OSINT-compass-engine"] 2 | path = OSINT-compass-engine 3 | url = https://github.com/elpato-dev/OSINT-compass-engine.git 4 | branch = main 5 | [submodule "OSINT-compass-portal"] 6 | path = OSINT-compass-portal 7 | url = https://github.com/elpato-dev/OSINT-compass-portal.git 8 | branch = main 9 | [submodule "OSINT-compass-alert-cron"] 10 | path = OSINT-compass-alert-cron 11 | url = https://github.com/elpato-dev/OSINT-compass-alert-cron.git 12 | branch = main 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | Copyright 2023 elpato-dev 179 | 180 | Licensed under the Apache License, Version 2.0 (the "License"); 181 | you may not use this file except in compliance with the License. 182 | You may obtain a copy of the License at 183 | 184 | http://www.apache.org/licenses/LICENSE-2.0 185 | 186 | Unless required by applicable law or agreed to in writing, software 187 | distributed under the License is distributed on an "AS IS" BASIS, 188 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 189 | See the License for the specific language governing permissions and 190 | limitations under the License. 191 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OSINT-compass 2 | 3 | ![logo](https://github.com/elpato-dev/OSINT-compass/blob/main/images/logo.png) 4 | 5 | ## Team Members 6 | This project was developed by [@her0marodeur](https://github.com/her0marodeur), [@elutac](https://github.com/elutac) and [@FlexusDev](https://github.com/FlexusDev). We welcome contributions from the open-source community. If you have any questions or suggestions, please feel free to open an issue or submit a pull request. Thank you for your interest in OSINT-compass! 7 | 8 | ## Tool Description 9 | OSINT-compass is a tool that allows users to collect, monitor, and analyze open-source information from various sources. It provides an integrated search engine to gather relevant information and a sentiment monitoring interface. 10 | 11 | Its main purpose is to allow organizations to provide an easy to use, easy to extend, graphical tool to researchers. If you want you can also selfhost it (not straight forward yet). 12 | 13 | The alerting capabilities also make it easy for researchers to keep an eye on a multitude of topics and be alerted on relevant changes. 14 | 15 | We would absolutely love, if you would integrate your own tools or APIs into OSINT-compass. <3 16 | 17 | ## Usage 18 | The README of the repository [OSINT-compass-engine](https://github.com/elpato-dev/OSINT-compass-engine/blob/main/README.md) has a detailed documentation of the API and its endpoints. 19 | 20 | We hope the web interface on https://osint-compass-portal.onrender.com is self explanatory but we will provide some guidelines on how to use it. 21 | 22 | A full tour of the hosted version can be found [here](https://github.com/elpato-dev/OSINT-compass/blob/main/tour_hosted_version.md). 23 | 24 | ### Start Page 25 | 26 | The start page provides a search bar with three buttons below: 27 | - term: just enter a term like `brazil` or `america president` 28 | - results: 29 | - recent news from news API, with an overall sentiment score between -1 (negative) and 1 (positve) 30 | - real range of sentiment score is probably between -0.3 and 0.03 31 | - recent tweets from Twitter API, with an overall sentiment score between -1 (negative) and 1 (positve) 32 | - real range of sentiment score is probably between -0.3 and 0.03 33 | - tweets are not filtered by relevance and may include offensive content 34 | - links to wikipedia articles related to the term 35 | - a frequency count of words used in the news and tweets (extend the card by clicking on it) 36 | - email: enter an email address in the format `email@domain.com` 37 | - get results form different sources 38 | - currently pingutils and spycloud since they need no API key and have no rate limiting 39 | - can be easily extended with other sources like emailrep and HaveIBeenPwned 40 | - domain: enter a domain in the format `domain.com` 41 | - get results form different sources 42 | - currently checks for robots.txt, subdomains and if it is listed on the waybackmachine 43 | - can be easily extended with other sources 44 | - snscrape: enter a term that will be searched on reddit 45 | - get reddit results 46 | - get sentiment score 47 | - not filtered so might include offensive content 48 | 49 | ### Alerting 50 | 51 | To use the altering functionality of your web project, please follow these steps: 52 | 53 | 1. Set up a Telegram bot and obtain your Telegram Bot Token. 54 | 55 | 2. Enter your Telegram Bot Token into the local .env file of the OSINT-compass-alert-cron by adding the following line: `TELEGRAM_BOT_TOKEN=` 56 | 57 | 3. To set an alert for a specific search term, enter the following information: 58 | - Search term: The term for which you want to monitor the Sentiment Score. 59 | - Communication channel: At the moment, only Telegram is supported 60 | - Communication channel information: Enter the Chat ID where you want to receive the alerts 61 | - Trigger values: Enter the values that should trigger the alert 62 | 63 | You can only alert for lesser than or greater than or both but currently the API needs input for both. So please input 1 for greater than, if you only want an alert for lesser than. Also input -1 for lesser than, if you want to only alert for greater than. In the other field input the desired value. Keep in mind that realistic ranges are between -0.3 and 0.3. 64 | 65 | 4. Once you have entered all the necessary information, you need to run the cronjob as described in `local_install.md`. The cronjob will periodically check for the trigger values and send alerts to your designated Telegram chat when the values are met. 66 | 67 | Note: Please ensure that the Telegram bot has the necessary permissions to send messages to the designated chat. 68 | 69 | If everything worked fine, you should be able to recieve alerts like this: 70 | 71 | ![telegram_alerts](https://github.com/elpato-dev/OSINT-compass/blob/main/images/telegram_alerts.png) 72 | 73 | If you want to trigger the alert checking manually just send a get request to this URL: `/alert?apikey=`. 74 | 75 | ## Extending the tool 76 | 77 | The tool was designed with extensibility in mind. Check out the guide on how to extend it [here](https://github.com/elpato-dev/OSINT-compass/blob/main/extension_guide.md). 78 | 79 | ## Installation 80 | Installation currently is not very easy, since the tool consists of multiple services deployed on render.com and docker containers. A hosted instance can be found here: https://osint-compass-portal.onrender.com 81 | 82 | Basically what you would do is take the submodules and deploy them to render.com. OSINT-compass-engine as a web service and and OSINT-compass-portal as a static website. 83 | 84 | For alerting you need to deploy a Postgresql datbase on render. Additionally you need something that can run cron jobs (on render they are paid only) and run the OSINT-compass-alert-cron in what ever intervall you want it to send out alerts. 85 | 86 | For installing the tools locally we provide a guide [here](https://github.com/elpato-dev/OSINT-compass/blob/main/local_install.md) . 87 | 88 | To simplify this process in the future we plan on releasing a Docker image, that combines all the services. 89 | 90 | ## Additional Information 91 | Future ideas: 92 | - implement a single Docker image to make self hosting easier 93 | - add more tools and make the current ones more extensive 94 | - timelines of sentiment score 95 | - add subjectivity score from TextBlob 96 | - JSON export of results 97 | - guide on how to host it on render 98 | - other alert channels 99 | - adding closed APIs like haveibeenpwned and checks if they are used 100 | - adding a telegram bot that allows to query the API 101 | - implement dark mode 102 | 103 | Limitations: 104 | - Self hosting the tool is possible but was not documented during the bellingcat hackathon, since it is currently quite some effort. 105 | - I would suspect the tool to have at least some security vulnerabilities. We tried to do our best, but please do not provide very sensitive information to it. 106 | - The sentiment score currently seems to have a low range, so do not really expect to see one above 0.3 or below -0.3 . 107 | - The tweets are currently not filtered by relevance so you might see some offensive material... sorry... 108 | 109 | Archictecture principles: 110 | - The tool aims to be very modular, to allow anyone to integrate a GUI for their tool or API into OSINT-compass. 111 | - If you are only after data collection automation you could also use the API as a standalone service. 112 | 113 | ## Submodules 114 | This repository includes three submodules: the `OSINT-compass-engine`, `OSINT-compass-portal` and `OSINT-compass-API-cron` modules. 115 | 116 | The `OSINT-compass-engine` submodule contains the backend of the OSINT-compass tool, which handles the API, data retrieval and storage, processing, and analysis. 117 | 118 | The `OSINT-compass-portal` submodule provides a user-friendly interface to interact with the tool. 119 | 120 | The `OSINT-compass-API-cron` submodule includes a Python script thought to be run as a cron job to send alerts. Currently it is implemented as an API to allow tesing of the alerting functionality. 121 | 122 | All three submodules point to a working version (commit) of the corresponding respository. The main branches of the repositories may differ from that working version (commit). 123 | 124 | ## Acknowledgments 125 | 126 | We would like to thank Bellingcat for hosting the exciting hackathon during which the tool was created. 127 | They provided a lot of valuable feedback and we hope our tool can help with the amazing work they are doing. 128 | 129 | This tool would not be possible wothout the other tools and APIs it is using. 130 | We are thankful for their contribution to the OSINT community. 131 | 132 | The tool uses snscrape: https://github.com/JustAnotherArchivist/snscrape 133 | 134 | The tool queries the following websites and APIs: 135 | - Waybackmachine: https://web.archive.org/ 136 | - Spycloud: https://spycloud.com/ 137 | - ECA pingutil: https://eva.pingutil.com/ 138 | - NewsAPI: https://newsapi.org 139 | - Twitter: https://twitter.com 140 | -------------------------------------------------------------------------------- /extension_guide.md: -------------------------------------------------------------------------------- 1 | # OSINT-compass extension guide 2 | 3 | This page details how to integrate your own tool or API into OSINT-compass. If you have any questions or need help, please contact us. You can open a pull request with new features or ceate an issue for ideas. 4 | 5 | ## General design 6 | 7 | The tool consists of the following parts: 8 | - an engine (exposes the functionality via API and retrieves the data from tools, APIs and other soures) 9 | - written in Python/Flask 10 | - a web interface (allows the user to interact with all the tools in a GUI) 11 | - written in Angular 12 | - a database for storing the alerts 13 | - currently Postgresql 14 | - an alert cron job 15 | - has to be configured to run periodically to check the database for alerts and send alerts out, when the criteria is met 16 | - written in Python 17 | 18 | The following image illustrates the architecture: 19 | 20 | ![oc_modules](https://github.com/elpato-dev/OSINT-compass/blob/main/images/architecture.png) 21 | 22 | ## Extending the API 23 | 24 | The following diagram shows the integration of the modules: 25 | 26 | ![oc_modules](https://github.com/elpato-dev/OSINT-compass/blob/main/images/modules_engine.png) 27 | 28 | The arrows show which module feeds data to which other module. 29 | 30 | For you to implement a tool or API into OSINT-compass you should first add an API endpoint in the [main.py](https://github.com/elpato-dev/OSINT-compass-engine/blob/main/main.py). 31 | 32 | Let's look at the email endpoint for example: 33 | 34 | ```text 35 | @app.route('/email', methods=['GET']) #define the endpoint and method 36 | @require_api_key # protects it with the apikey so request has to include apikey= 37 | def email_endpoint(): 38 | email = request.args.get('email') # get the args you need 39 | if not email: 40 | return jsonify({'error': 'email argument is required.'}), 400 # do some error handling (check if required args are there) 41 | result = get_email_data(email) # call the method for data collection and get result (is available bc of "from emailgetter import get_email_data" at the top) 42 | return jsonify(result) # returns the result you retrieved as JSON as response to the request 43 | ``` 44 | 45 | You can add your API endpoint in the same style and step one is done. 46 | 47 | ## Adding a module 48 | 49 | Continuing with our email example let's look at the email modules implementation ([emailgetter.py](https://github.com/elpato-dev/OSINT-compass-engine/blob/main/emailgetter.py)) (The current implementation looks a bit different from this one, but as this one is easier, we explain it using this one) 50 | 51 | ```text 52 | import requests # import needed modules 53 | 54 | def get_email_data(email): # define method that will be imported and called in main.py 55 | 56 | # do the necessary data collection 57 | 58 | # query spycloud 59 | spycloud_response = requests.get("https://portal.spycloud.com/endpoint/enriched-stats/" + email) 60 | spycloud_data = spycloud_response.json() 61 | 62 | # query pingutil 63 | 64 | pingutil_response = requests.get('https://api.eva.pingutil.com/email?email=' + email, verify=False) 65 | pingutil_data = pingutil_response.json() 66 | 67 | # formatting of the data in a format that the UI can handle 68 | 69 | email_data = { 70 | "sources":[ 71 | 72 | { 73 | "title": "spycloud", 74 | "content": spycloud_data 75 | }, 76 | { 77 | "title": "pingutil", 78 | "content": pingutil_data 79 | } 80 | 81 | ] 82 | } 83 | return email_data # returning the data (that is what the UI gets) 84 | 85 | ``` 86 | 87 | 88 | While other JSON structures can be implemented in the UI the following is the easiest way for now. 89 | It will create a simple card style layout in the frontend.: 90 | 91 | ```text 92 | { 93 | "sources":[ 94 | 95 | { 96 | "title": , 97 | "content": 98 | }, ... (how ever many cards of the format above you want) 99 | 100 | ] 101 | } 102 | ``` 103 | 104 | Note that this card has not much formatting besides title and content, so please do not put too much stuff into the content. 105 | 106 | And that is it, the API is now exposing your tool: 107 | 108 | ```text 109 | https:///?=&apikey= 110 | ``` 111 | 112 | Note: For the email and domain functionality (and any other future functionality that uses that simple card style) adding new sources is very easy. Since the UI already dynamically adds all the JSON objects from the sources list, you can just query your source in the python script that gets the data. Then you only need to add it tho the sources list as explained above and the UI will display it in that defined card format. 113 | 114 | ## Creation of simple cards (recursive display) 115 | 116 | When the frontend application is receiving a JSON input like shown above it has the ability to loop through all the JSON objects in the sources list. 117 | It will create a card for each JSON object. The title will be used as the title for the card. The frontend will try to unravel to JSON in `content` and find a way to display it nicely. 118 | 119 | The following example shows how it will be done: 120 | 121 | ![card_creation](https://github.com/elpato-dev/OSINT-compass/blob/main/images/card_creation.png) 122 | 123 | ## Extending the UI 124 | 125 | 126 | The UI can be extened in many ways. One that we want to highlight, is the very easy option to add new, simple search modules. 127 | 128 | ### Adding an endpoint 129 | 130 | Go to the [search-page.compnent.ts](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/app/search-page/search-page.component.ts) 131 | 132 | #### Step 1 : 133 | Add a string for your tool to the categories list. This will be the name of the button that will be added below the searchbar: 134 | 135 | ```plaintext 136 | categories: string[] = [ 137 | "Term", 138 | "E-Mail", 139 | "Domain", 140 | "snscrape" 141 | ]; 142 | ``` 143 | #### Step 2 : 144 | Add your tool to the switch statement. Here you define the endpoint-name and how your results are displayed. 145 | 146 | ```plaintext 147 | switch (this.selectedCategory) { 148 | case "Term" : endpoint = "term"; display = "term"; break; 149 | case "E-Mail" : endpoint = "email"; display = "recursive"; break; 150 | case "Domain" : endpoint = "domain"; display = "recursive"; break; 151 | case "snscrape" : endpoint = "snscrape"; display = "snscrape"; break; 152 | } 153 | ``` 154 | #### Step 3 (optional) : 155 | 156 | To change paramter-style of your endpoint you have to add a case to the swtich statement in the Compass-Service [compass-api.service.ts ](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/services/compassapi/compass-api.service.ts) 157 | 158 | ```plaintext 159 | switch (endpoint) { 160 | case "snscrape" : 161 | url = this.baseURL + '/' + endpoint + '?term=' + term + '&entries=10&reddit=true&apikey=' + this.apikey; 162 | break; 163 | default : 164 | url = this.baseURL + '/' + endpoint + '?' + endpoint + '=' + term + '&apikey=' + this.apikey; 165 | break; 166 | } 167 | ``` 168 | 169 | This will result in the frontend making an API request like the following `/?=` 170 | 171 | Notice that the endpoint name and name of the parameter have to be the same and it currently only supports one parameter. The frontend expects a response in the JSON format shown [here](#JSON). 172 | 173 | ### Adding a display-page 174 | 175 | There are already three display-pages implemented. 176 | - term 177 | - recursive (recommended) 178 | - snscrape 179 | 180 | If you want to define your own display-page do the following. 181 | 182 | #### Step 1: 183 | Add a component to handel the displaying of the data. 184 | #### Step 2: 185 | Add the component to the [SearchComponent](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/app/search-page/search-page.component.ts)'s [html](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/app/search-page/search-page.component.html). 186 | #### Step 3: 187 | ResultData can be passed via import. Look into [SnscrapeDisplayComponent](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/app/result-page/snscrape-display/) for a implemented example. 188 | -------------------------------------------------------------------------------- /ideas.md: -------------------------------------------------------------------------------- 1 | # General Idea 2 | Wie wäre es mit einem Tool namens "OSINT Compass"? Das Tool würde es Benutzern ermöglichen, verschiedene Open-Source-Informationen aus verschiedenen Quellen zu sammeln, zu organisieren und zu analysieren. Es könnte Funktionen wie eine integrierte Suchmaschine haben, um relevante Informationen aus verschiedenen Quellen zu sammeln, eine Schnittstelle zur Überwachung von Social-Media-Plattformen und eine Möglichkeit, Daten in verschiedenen Formaten zu speichern und zu organisieren. 3 | Darüber hinaus könnte OSINT Compass auch Machine-Learning-Technologien verwenden, um Benutzern dabei zu helfen, wichtige Trends und Muster in den gesammelten Informationen zu erkennen. Das Tool könnte auch Funktionen haben, die es Benutzern ermöglichen, Berichte und Präsentationen auf der Grundlage der gesammelten Informationen zu erstellen, um Einblicke in wichtige Themen oder Trends zu gewinnen. 4 | # Feature Ideas 5 | Features: 6 | Integrierte Suchmaschine: Eine integrierte Suchmaschine, die es Benutzern ermöglicht, relevante Informationen aus verschiedenen Quellen zu sammeln. 7 | 8 | Social Media Monitoring: Eine Schnittstelle zur Überwachung von Social-Media-Plattformen wie Twitter, Facebook, LinkedIn und Instagram. 9 | 10 | Metadaten-Extraktion: Möglichkeit zur Extraktion von Metadaten aus Bildern und Videos. 11 | 12 | Geolokalisierung: Möglichkeit zur Lokalisierung von Fotos oder Videos über Geo-Tags und GPS-Informationen. 13 | 14 | Verhaltensanalyse: Eine Funktion zur Analyse von Benutzerverhalten in sozialen Medien oder anderen Online-Plattformen. 15 | 16 | Sentimentanalyse: Möglichkeit zur Analyse des allgemeinen Stimmungsbildes über ein bestimmtes Thema oder eine bestimmte Person. 17 | 18 | Entitätsanalyse: Eine Funktion zur Identifizierung von Personen, Organisationen und Orten in Texten und Bildern. 19 | 20 | Machine Learning: Machine-Learning-Technologien, die helfen können, wichtige Trends und Muster in den gesammelten Informationen zu erkennen. 21 | 22 | Export- und Speicherfunktionen: Möglichkeit, Daten in verschiedenen Formaten zu speichern und zu organisieren. 23 | 24 | Berichtsfunktionen: Möglichkeit zur Erstellung von Berichten und Präsentationen auf der Grundlage der gesammelten Informationen. 25 | 26 | Zusammenarbeit: Eine Funktion zur Zusammenarbeit von mehreren Benutzern an einem Projekt. 27 | 28 | Datensicherheit: Ein sicheres System zum Schutz der gesammelten Informationen. 29 | 30 | Anonymität: Eine Möglichkeit, die Identität des Benutzers zu verschleiern, um die Sicherheit der Analyse zu erhöhen. 31 | 32 | E-Mail-Monitoring: Eine Funktion zur Überwachung von E-Mails. 33 | 34 | Dark Web Monitoring: Eine Schnittstelle zur Überwachung des Dark Web. 35 | 36 | API-Integration: Möglichkeit, APIs von anderen Tools und Diensten zu integrieren, um weitere Datenquellen zu nutzen. 37 | 38 | Live-Überwachung: Möglichkeit zur Live-Überwachung bestimmter Quellen. 39 | 40 | Benachrichtigungen: Eine Funktion zur Benachrichtigung von Benutzern über wichtige Ereignisse oder Änderungen in den gesammelten Informationen. 41 | 42 | Workflow-Management: Eine Funktion zur Optimierung der Arbeitsabläufe bei der Sammlung, Organisation und Analyse von Informationen. 43 | 44 | Erkennung von Fälschungen: Eine Funktion zur Erkennung von gefälschten oder manipulierten Inhalten. 45 | 46 | Datenvisualisierung: Möglichkeit, gesammelte Informationen visuell darzustellen, z.B. in Form von Diagrammen oder Heatmaps. 47 | 48 | Automatisierung: Möglichkeit, die Sammlung und Analyse von Informationen zu automatisieren, z.B. durch die Erstellung von Skripten. 49 | 50 | Datenbereinigung: Möglichkeit zur automatischen Bereinigung von Daten und Entfernung von Duplikaten. 51 | 52 | Keyword-Tracking: Eine Funktion zur Verfolgung bestimmter Keywords oder Phrasen in den gesammelten Informationen. 53 | 54 | Anpassbare Dashboards: Möglichkeit, das Dashboard und die Benutzeroberfläche an die Bedürfnisse des Benutzers anzupassen. 55 | 56 | Workflow-Integration: Möglichkeit, den Workflow in anderen Tools und Diensten zu integrieren, z.B. in Projektmanagement-Tools oder E-Mail-Programme. 57 | 58 | Datenbank-Integration: Möglichkeit, die gesammelten Informationen in einer Datenbank zu speichern und abzurufen. 59 | 60 | Textanalyse: Eine Funktion zur Analyse von Texten und der Identifizierung von wichtigen Schlüsselwörtern und Themen. 61 | 62 | Sprachübersetzung: Möglichkeit, Informationen in verschiedenen Sprachen automatisch zu übersetzen. 63 | 64 | Personalisierte Warnungen: Eine Funktion zur Erstellung von personalisierten Warnungen, z.B. wenn sich eine bestimmte Person auf einer Social-Media-Plattform äußert. 65 | 66 | Content Curation: Eine Funktion zur Kuratierung von Inhalten, d.h. zur Identifizierung der relevantesten Informationen und deren Präsentation auf eine übersichtliche Art und Weise. 67 | 68 | Erweiterbarkeit: Möglichkeit, das Tool durch Plugins und Erweiterungen zu erweitern, z.B. durch die Integration von neuen Datenquellen oder Analyse-Tools. 69 | 70 | Cloud-basiert: Möglichkeit, das Tool in der Cloud zu hosten und von überall aus darauf zuzugreifen. 71 | 72 | Multitasking: Möglichkeit, mehrere Aufgaben gleichzeitig auszuführen, z.B. durch die Verwendung von Tabs oder Arbeitsbereichen. 73 | 74 | Echtzeit-Kollaboration: Möglichkeit, in Echtzeit mit anderen Benutzern zu kommunizieren und zusammenzuarbeiten, z.B. durch den Einsatz von Chat- oder Videokonferenz-Tools. 75 | 76 | Historische Datenanalyse: Eine Funktion zur Analyse historischer Daten und zur Identifizierung von Trends und Mustern über einen längeren Zeitraum. 77 | 78 | Privatsphäre-Einstellungen: Möglichkeit, die Privatsphäre-Einstellungen für die Sammlung und Analyse von Informationen anzupassen, z.B. um unerwünschte Inhalte auszublenden oder bestimmte Quellen auszuschließen. 79 | 80 | Integrierte Analyse-Tools: Möglichkeit, integrierte Analyse-Tools für bestimmte Aufgaben zu verwenden, z.B. für die Gesichtserkennung oder die Analyse von Social-Media-Netzwerken. 81 | 82 | Risikobewertung: Möglichkeit, eine Risikobewertung für die gesammelten Informationen durchzuführen, z.B. durch die Verwendung von Bewertungskriterien wie Zuverlässigkeit oder Relevanz. 83 | 84 | Compliance: Möglichkeit, das Tool gemäß den geltenden Compliance-Standards zu konfigurieren, z.B. durch die Einhaltung von Datenschutz- und Datensicherheitsrichtlinien. 85 | 86 | Integration mit Alerting-Systemen: Möglichkeit, das Tool mit Alerting-Systemen zu integrieren, um Benachrichtigungen über wichtige Ereignisse oder Entwicklungen zu erhalten. 87 | 88 | Benutzer-Authentifizierung: Möglichkeit, die Benutzer-Authentifizierung durchzuführen, um sicherzustellen, dass nur autorisierte Benutzer Zugriff auf das Tool und die gesammelten Informationen haben. 89 | 90 | Benutzer-Management: Möglichkeit, Benutzer zu verwalten und deren Zugriffsrechte und -berechtigungen zu konfigurieren, um sicherzustellen, dass nur autorisierte Benutzer auf die gesammelten Informationen zugreifen können. 91 | 92 | Content-Moderation: Möglichkeit, Inhalte zu moderieren, z.B. um unangemessene Inhalte auszublenden oder zu entfernen. 93 | 94 | Geolokalisierung: Möglichkeit, Informationen anhand ihres geografischen Standorts zu identifizieren und zu analysieren, z.B. durch die Verwendung von GPS-Koordinaten oder IP-Adressen. 95 | 96 | Integration mit Daten-Quellen: Möglichkeit, Datenquellen wie Social-Media-Plattformen oder Nachrichtenportale direkt in das Tool zu integrieren, um die Sammlung von Informationen zu automatisieren. 97 | 98 | Interaktive Datenvisualisierung: Möglichkeit, interaktive Datenvisualisierungen zu erstellen, um den Benutzern eine bessere Übersicht über die gesammelten Informationen zu geben. 99 | 100 | Zeitleisten-Erstellung: Möglichkeit, eine Zeitleiste der Ereignisse zu erstellen, um die Entwicklung von Ereignissen im Laufe der Zeit zu verfolgen. 101 | 102 | Machine Learning-Integration: Möglichkeit, Machine Learning-Technologien in das Tool zu integrieren, um automatische Analysen und Vorhersagen zu generieren. 103 | 104 | Berichterstellung: Möglichkeit, Berichte über die gesammelten Informationen und Analysen zu erstellen, um den Benutzern eine Zusammenfassung der Ergebnisse zu geben. 105 | -------------------------------------------------------------------------------- /images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elpato-dev/OSINT-compass/e8b9501f733761842ac18a0c880d22842572f4a3/images/architecture.png -------------------------------------------------------------------------------- /images/card_creation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elpato-dev/OSINT-compass/e8b9501f733761842ac18a0c880d22842572f4a3/images/card_creation.png -------------------------------------------------------------------------------- /images/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elpato-dev/OSINT-compass/e8b9501f733761842ac18a0c880d22842572f4a3/images/logo.png -------------------------------------------------------------------------------- /images/modules_engine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elpato-dev/OSINT-compass/e8b9501f733761842ac18a0c880d22842572f4a3/images/modules_engine.png -------------------------------------------------------------------------------- /images/telegram_alerts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elpato-dev/OSINT-compass/e8b9501f733761842ac18a0c880d22842572f4a3/images/telegram_alerts.png -------------------------------------------------------------------------------- /local_install.md: -------------------------------------------------------------------------------- 1 | # Complete local installation guide for OSINT-compass 2 | 3 | OSINT-compass needs [Python](https://www.python.org/downloads/) with [pip](https://pypi.org/project/pip/) and [NodeJS with npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) to be installed. 4 | 5 | ## Setting up a postgres database and telegram bot 6 | 7 | This step is only necessary, if you want to use the alering functionality. 8 | 9 | We provide links to guides that explain how to do it: 10 | 11 | - Telegram Bot: https://sendpulse.com/knowledge-base/chatbot/telegram/create-telegram-chatbot 12 | - Postgesql database: https://1kevinson.com/how-to-create-a-postgres-database-in-docker/ 13 | 14 | You also need to create the following table: 15 | 16 | ```plaintext 17 | CREATE TABLE alerts ( 18 | id SERIAL PRIMARY KEY, 19 | term VARCHAR NOT NULL, 20 | scorelt NUMERIC(3,1), 21 | scoregt NUMERIC(3,1), 22 | contact_method VARCHAR, 23 | contact_details VARCHAR 24 | ); 25 | ``` 26 | 27 | ## OSINT-compass Engine 28 | 29 | 1. Clone the repo `git clone https://github.com/elpato-dev/OSINT-compass-engine.git` from [here](https://github.com/elpato-dev/OSINT-compass-engine) 30 | 31 | 2. Install the requirements `pip install -r requirements.txt` 32 | 33 | 3. Uncomment the last two lines from main.py so it looks like this: 34 | ```plaintext 35 | # Remove before deploying to render 36 | if __name__ == '__main__': 37 | app.run() 38 | ``` 39 | 4. Copy the contents of [example_env.txt](https://github.com/elpato-dev/OSINT-compass-engine/blob/main/example_env.txt) into a `.env` file and fill in the needed information. 40 | 5. Run the app `python3 main.py` . 41 | 6. It now should be locally available on `http://127.0.0.1:5000` 42 | 43 | ## OSINT-compass Alerter 44 | 45 | 1. Clone the repo `git clone https://github.com/elpato-dev/OSINT-compass-alert-cron.git` from [here](https://github.com/elpato-dev/OSINT-compass-alert-cron) 46 | 47 | 2. Install the requirements `pip install -r requirements.txt` 48 | 3. Exchange the url of your API endpoint in line 28 in the [alerter.py](https://github.com/elpato-dev/OSINT-compass-alert-cron/blob/main/alerter.py) 49 | 50 | 4. uncomment the last two lines from api.py so it looks like this: 51 | ```plaintext 52 | #remove before deploying on render 53 | if __name__ == '__main__': 54 | app.run(port=1337) 55 | ``` 56 | 5. Copy the contents of [example_env.txt](https://github.com/elpato-dev/OSINT-compass-engine/blob/main/example_env.txt) into a `.env` file and fill in the needed information. The API key that is used for accessing the engine API is also used for protecting the alerter API 57 | 6. Run the app `python3 main.py` . 58 | 7. It now should be locally available on `http://127.0.0.1:1337` 59 | 8. You can trigger alerts via `http://127.0.0.1:1337/alert?apikey=` 60 | 9. To run alerts setup something that periodically calls the alert API endpoint. (e.g. a cron job) 61 | 62 | 63 | ## OSINT-compass Portal 64 | 65 | 1. Clone the code from [here](https://github.com/elpato-dev/OSINT-compass-portal) `git clone https://github.com/elpato-dev/OSINT-compass-portal.git` 66 | 2. Run `npm install` in the root directory of the project 67 | 3. edit the `baseURL` and `apikey` to match your API endpoint in the [environment.ts](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/environments/environment.ts) and [environment.development.ts](https://github.com/elpato-dev/OSINT-compass-portal/blob/main/src/environments/environment.development.ts) 68 | 4. Run `npm start` in the root directory of the project. 69 | 5. The project will be available on 127.0.0.1:4200. 70 | 6. If you wnat to deploy it for production, check [this](https://angular.io/guide/deployment). 71 | -------------------------------------------------------------------------------- /tour_hosted_version.md: -------------------------------------------------------------------------------- 1 | # OSINT-compass-portal hosted version tour 2 | 3 | The hosted version can be found on: https://osint-compass-portal.onrender.com/ . 4 | 5 | ## The start page 6 | 7 | The start page provides access to all the search functionalities of OSINT-compass. 8 | The following search features are implemented in the hosted version: 9 | 10 | ### Term search 11 | 12 | The term search functionality lets you see recent news and tweets for the term you enter as well as related links from wikipedia. 13 | Furthermore it provides basic sentiment analysis: 14 | 15 | ![image](https://user-images.githubusercontent.com/101996103/233848009-e065d2e1-57e8-49b7-b378-ffa3ccadb782.png) 16 | 17 | ### E-Mail search 18 | 19 | The email search functionality lets you see basic data for an email address. 20 | 21 | - Pingutil: displays basic info about the email 22 | - Spycloud: provides basic breach occurence data of the provided email address and its domain 23 | 24 | ![image](https://user-images.githubusercontent.com/101996103/233847949-ab9ad650-5abe-48e5-acbd-a0189d933d2f.png) 25 | 26 | 27 | ### Domain search 28 | 29 | The domain search functionality provides basic information on a domain: 30 | - If it has a robots.txt it will be displayed 31 | - If the URL was indexed by waybackmachine you can click on the latest index of it 32 | 33 | ![image](https://user-images.githubusercontent.com/101996103/233848064-5a17cad7-97ec-4b2b-a749-1b235c6e2e0f.png) 34 | 35 | ### snscrape 36 | This functionality shows that other tools can be integrated in our tool. Currently only the reddit functionality of snscrape is available, since many other functionalities are currently broken. 37 | 38 | The snscrape functionality provides reddit posts for the term you specify: 39 | - information about the post 40 | - post content 41 | - sentiment analysis 42 | 43 | ![image](https://user-images.githubusercontent.com/101996103/233848102-9c828f92-5058-4c2e-aa38-d1334ba603c2.png) 44 | 45 | 46 | ## The alerting functionality 47 | 48 | When you press the "ALERT ME" button in the top right corner the alert dialog will open. 49 | Currently alerting only supports telegram. 50 | 51 | The alerts will be send out every hour at xx:18 but you can trigger them manually here https://osint-compass-alerter.onrender.com/alert?apikey=mysuperkey . 52 | 53 | You can only alert for lesser than or greater than or both but currently the API needs input for both. So please input 1 for greater than, if you only want an alert for lesser than. Also input -1 for lesser than, if you want to only alert for greater than. In the other field input the desired value. Keep in mind that realistic ranges are between -0.3 and 0.3. 54 | 55 | **IMPORTANT: To receive alerts on telegram please start a chat with @osintcompassbot on Telegram.** 56 | 57 | ![image](https://user-images.githubusercontent.com/101996103/233840173-5b79e9f8-8f9f-425e-a04d-f0b972a0de21.png) 58 | 59 | 60 | You will get an alert like the following: 61 | 62 | ![image](https://user-images.githubusercontent.com/101996103/233841791-ab9ce64b-2f9f-439a-a814-2fc01d44a105.png) 63 | 64 | --------------------------------------------------------------------------------