├── __init__.py ├── subprober ├── __init__.py ├── modules │ ├── __init__.py │ ├── cli │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── cli.pypy310.pyc │ │ │ ├── cli.cpython-312.pyc │ │ │ ├── __init__.pypy310.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── cli.py │ ├── dns │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── dns.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── dns.py │ ├── tls │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── tls.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── tls.py │ ├── banner │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── banner.pypy310.pyc │ │ │ ├── __init__.pypy310.pyc │ │ │ ├── banner.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── banner.py │ ├── config │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── config.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── config.py │ ├── core │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── core.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── core.py │ ├── extender │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── extender.cpython-312.pyc │ │ └── extender.py │ ├── filters │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── filters.cpython-312.pyc │ │ └── filters.py │ ├── hash │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── hash.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── hash.py │ ├── help │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── help.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── help.py │ ├── jarmhash │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── jarmhash.cpython-312.pyc │ │ └── jarmhash.py │ ├── logger │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── logger.pypy310.pyc │ │ │ ├── __init__.pypy310.pyc │ │ │ ├── logger.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── logger.py │ ├── matchers │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── matchers.cpython-312.pyc │ │ └── matchers.py │ ├── save │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── save.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── save.py │ ├── update │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── update.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── update.py │ ├── utils │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── utils.cpython-312.pyc │ │ │ └── __init__.cpython-312.pyc │ │ └── utils.py │ ├── validate │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── headless.cpython-312.pyc │ │ └── headless.py │ ├── version │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── version.cpython-312.pyc │ │ └── version.py │ ├── screenshot │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── screenshot.cpython-312.pyc │ │ └── screenshot.py │ ├── websocket │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-312.pyc │ │ │ └── websocket.cpython-312.pyc │ │ └── websocket.py │ ├── __pycache__ │ │ ├── __init__.pypy310.pyc │ │ ├── handler.pypy310.pyc │ │ ├── handler.cpython-312.pyc │ │ └── __init__.cpython-312.pyc │ └── handler.py ├── subprober.py └── updatelog.md ├── requirements.txt ├── .github └── workflows │ └── python-publish.yml ├── LICENSE ├── setup.py └── README.md /__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/cli/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/dns/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/tls/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/banner/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/config/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/core/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/extender/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/filters/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/hash/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/help/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/jarmhash/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/logger/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/matchers/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/save/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/update/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/validate/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/version/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/screenshot/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/websocket/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /subprober/modules/__pycache__/__init__.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/__pycache__/__init__.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/__pycache__/handler.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/__pycache__/handler.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/cli/__pycache__/cli.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/cli/__pycache__/cli.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/__pycache__/handler.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/__pycache__/handler.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/cli/__pycache__/cli.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/cli/__pycache__/cli.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/dns/__pycache__/dns.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/dns/__pycache__/dns.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/tls/__pycache__/tls.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/tls/__pycache__/tls.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/banner/__pycache__/banner.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/banner/__pycache__/banner.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/cli/__pycache__/__init__.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/cli/__pycache__/__init__.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/core/__pycache__/core.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/core/__pycache__/core.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/hash/__pycache__/hash.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/hash/__pycache__/hash.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/help/__pycache__/help.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/help/__pycache__/help.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/logger/__pycache__/logger.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/logger/__pycache__/logger.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/save/__pycache__/save.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/save/__pycache__/save.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/banner/__pycache__/__init__.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/banner/__pycache__/__init__.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/cli/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/cli/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/dns/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/dns/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/logger/__pycache__/__init__.pypy310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/logger/__pycache__/__init__.pypy310.pyc -------------------------------------------------------------------------------- /subprober/modules/tls/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/tls/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/utils/__pycache__/utils.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/utils/__pycache__/utils.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/banner/__pycache__/banner.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/banner/__pycache__/banner.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/config/__pycache__/config.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/config/__pycache__/config.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/core/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/core/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/hash/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/hash/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/help/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/help/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/logger/__pycache__/logger.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/logger/__pycache__/logger.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/save/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/save/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/update/__pycache__/update.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/update/__pycache__/update.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/utils/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/utils/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/banner/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/banner/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/config/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/config/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/extender/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/extender/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/extender/__pycache__/extender.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/extender/__pycache__/extender.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/filters/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/filters/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/filters/__pycache__/filters.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/filters/__pycache__/filters.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/jarmhash/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/jarmhash/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/jarmhash/__pycache__/jarmhash.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/jarmhash/__pycache__/jarmhash.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/logger/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/logger/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/matchers/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/matchers/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/matchers/__pycache__/matchers.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/matchers/__pycache__/matchers.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/update/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/update/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/validate/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/validate/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/validate/__pycache__/headless.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/validate/__pycache__/headless.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/version/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/version/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/version/__pycache__/version.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/version/__pycache__/version.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/subprober.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.handler import Main 2 | 3 | def main(): 4 | 5 | Main() 6 | 7 | if __name__ == '__main__': 8 | 9 | main() 10 | -------------------------------------------------------------------------------- /subprober/modules/screenshot/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/screenshot/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/websocket/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/websocket/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/websocket/__pycache__/websocket.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/websocket/__pycache__/websocket.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/screenshot/__pycache__/screenshot.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RevoltSecurities/SubProber/HEAD/subprober/modules/screenshot/__pycache__/screenshot.cpython-312.pyc -------------------------------------------------------------------------------- /subprober/modules/config/config.py: -------------------------------------------------------------------------------- 1 | import appdirs 2 | import asyncio 3 | import os 4 | def configdir(): 5 | try: 6 | dir = appdirs.user_config_dir() 7 | config = f"{dir}/subprober" 8 | if not os.path.exists(config): 9 | os.makedirs(config) 10 | return config 11 | return config 12 | except (KeyboardInterrupt, asyncio.CancelledError): 13 | exit(1) 14 | except Exception as e: 15 | pass -------------------------------------------------------------------------------- /subprober/modules/extender/extender.py: -------------------------------------------------------------------------------- 1 | import platform 2 | import resource 3 | 4 | def extender(): 5 | try: 6 | soft , hard = resource.getrlimit(resource.RLIMIT_NOFILE) 7 | new = 100000 8 | osname = platform.system() 9 | if osname == "Linux" or osname == "Darwin": 10 | resource.setrlimit(resource.RLIMIT_NOFILE, (new, hard)) 11 | except KeyboardInterrupt as e: 12 | exit(1) 13 | except Exception as e: 14 | pass -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | aiodns>=3.2.0 2 | aiofiles>=24.1.0 3 | aiojarm>=0.2.2 4 | alive_progress>=3.2.0 5 | appdirs>=1.4.4 6 | art>=6.4 7 | asynciolimiter>=1.1.1 8 | beautifulsoup4>=4.12.3 9 | colorama>=0.4.6 10 | cryptography>=44.0.0 11 | fake_useragent>=1.5.1 12 | httpx>=0.28.1 13 | mmh3>=5.0.1 14 | playwright>=1.49.1 15 | Requests>=2.32.3 16 | rich>=13.9.4 17 | setuptools>=75.2.0 18 | simhash>=2.1.2 19 | urllib3>=1.26.18 20 | uvloop>=0.21.0 21 | websockets>=14.1 22 | bs4>=0.0.2 23 | lxml>=5.3.0 24 | -------------------------------------------------------------------------------- /subprober/modules/version/version.py: -------------------------------------------------------------------------------- 1 | import requests 2 | 3 | def Gitversion(): 4 | url = f"https://api.github.com/repos/RevoltSecurities/Subprober/releases/latest" 5 | try: 6 | response = requests.get(url, verify=True, timeout=10) 7 | if response.status_code == 200: 8 | data = response.json() 9 | latest = data.get('tag_name') 10 | return latest 11 | except KeyboardInterrupt as e: 12 | exit(1) 13 | except Exception as e: 14 | return None -------------------------------------------------------------------------------- /subprober/modules/jarmhash/jarmhash.py: -------------------------------------------------------------------------------- 1 | import aiojarm 2 | from subprober.modules.logger.logger import logger 3 | from subprober.modules.utils.utils import GetDomain 4 | import asyncio 5 | 6 | async def jarmhash(url, args,port=443) -> str: 7 | try: 8 | domain = GetDomain(url) 9 | result = await aiojarm.scan(domain, port) 10 | return result[3] 11 | except (KeyboardInterrupt, asyncio.CancelledError): 12 | exit(1) 13 | except Exception as e: 14 | logger(f"Excepiton occured in jarm fingerprint generate module due to: {e}, {type(e)}", "warn", args.no_color) -------------------------------------------------------------------------------- /subprober/modules/banner/banner.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | from art import * 3 | import random 4 | from subprober.modules.logger.logger import random,bold,reset,white,colors 5 | random_color = random.choice(colors) 6 | 7 | 8 | def banner(): 9 | tool_name = "subprober" 10 | fonts = ["big", "ogre", "shadow", "script", "colossal" , "smslant", "graffiti", "slant"] 11 | selected_font = random.choice(fonts) 12 | banner = text2art(f"{tool_name}", font=selected_font) 13 | banner = f"""{bold}{random_color}{banner}{reset} 14 | {bold}{white}- RevoltSecurities{reset}\n""" 15 | return banner -------------------------------------------------------------------------------- /subprober/updatelog.md: -------------------------------------------------------------------------------- 1 | # SubProber V3.0.1 Updates 2 | 3 | ### Whats Changed 4 | 5 | - **Fixed the issue in response html content parsing** 6 | - **Handled the parser exception, if `lxml` parser is missed then automatically errors handled** 7 | - **Improved the requirements of packages to install missing pacakges** 8 | 9 | ### Information: 10 | 11 | **If any of our users faced empty titles in output then indicates that `lxml` and `bs4` packages are not installed properly, We request that users to install the missing packages with the following command** 12 | 13 | ``` 14 | pip install -U beautifulsoup4 lxml bs4 --break-system-packages 15 | ``` 16 | 17 | -------------------------------------------------------------------------------- /subprober/modules/save/save.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.logger.logger import logger 2 | import aiofiles 3 | import asyncio 4 | import json 5 | 6 | async def save(filename: str, content: str, Json: bool, nocolor=False) -> None: 7 | try: 8 | async with aiofiles.open(filename, "a") as streamw: 9 | if Json: 10 | await streamw.write(json.dumps(content, indent=4)+ "\n") 11 | else: 12 | await streamw.write(content + '\n') 13 | except (KeyboardInterrupt,asyncio.CancelledError): 14 | exit(1) 15 | except Exception as e: 16 | logger(f"Excpetion occured in the save module due to: {e}, {type(e)}, {content}", "warn", nocolor) -------------------------------------------------------------------------------- /subprober/modules/update/update.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from rich.console import Console 3 | from rich.markdown import Markdown 4 | from subprober.modules.logger.logger import logger 5 | 6 | console = Console() 7 | 8 | def updatelog(): 9 | try: 10 | url = f"https://raw.githubusercontent.com/RevoltSecurities/SubProber/main/subprober/updatelog.md" 11 | response = requests.get(url, timeout=20, stream=True) 12 | if response.status_code == 200: 13 | loader = response.text 14 | console.print(Markdown(loader)) 15 | else: 16 | logger("Unable to get the new updates of the subprober, please try to vist here: https://github.com/RevoltSecurities/SubProber/blob/main/subprober/updatelog.md", "info") 17 | quit() 18 | except Exception as e: 19 | pass -------------------------------------------------------------------------------- /subprober/modules/websocket/websocket.py: -------------------------------------------------------------------------------- 1 | import websockets 2 | import websockets.connection 3 | from subprober.modules.logger.logger import logger 4 | import asyncio 5 | 6 | async def AsyncWebsocket(url, args) -> str: 7 | try: 8 | if url.startswith("https://"): 9 | wsurl = url.replace("https://", "wss://", 1) 10 | elif url.startswith("http://"): 11 | wsurl = url.replace("http://", "ws://", 1) 12 | 13 | async with websockets.connect(wsurl) as socket: 14 | return "allowed" 15 | except (KeyboardInterrupt, asyncio.CancelledError): 16 | exit(1) 17 | except websockets.exceptions.InvalidStatusCode: 18 | return "disallowed" 19 | except Exception as e: 20 | if args.verbose: 21 | logger(f"Exception occured in the async websocket module due to: {e}, {type(e)}", "warn", args.no_color) 22 | return "disallowed" -------------------------------------------------------------------------------- /.github/workflows/python-publish.yml: -------------------------------------------------------------------------------- 1 | name: Release to PyPI 2 | 3 | on: 4 | 5 | release: 6 | 7 | types: 8 | - created 9 | 10 | jobs: 11 | 12 | deploy: 13 | 14 | runs-on: ubuntu-latest 15 | 16 | steps: 17 | 18 | - name: Check out code 19 | 20 | uses: actions/checkout@v2 21 | 22 | 23 | - name: Set up Python 24 | 25 | uses: actions/setup-python@v2 26 | 27 | with: 28 | 29 | python-version: 3.x # Choose your Python version 30 | 31 | - name: Install dependencies 32 | run: | 33 | python3 -m pip install --upgrade pip 34 | pip install setuptools wheel twine 35 | 36 | - name: Build and publish 37 | run: | 38 | python3 setup.py sdist bdist_wheel 39 | python3 -m twine upload dist/*.tar.gz 40 | env: 41 | 42 | TWINE_USERNAME: ${{ secrets.PYPI_USER }} 43 | 44 | TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} 45 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2025 RevoltSecurities 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /subprober/modules/dns/dns.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.logger.logger import logger 2 | from subprober.modules.utils.utils import GetDomain 3 | import aiodns 4 | import asyncio 5 | import socket 6 | 7 | async def dns(resolver: aiodns.DNSResolver, url, record_type, args) -> list[str]: 8 | try: 9 | domain = GetDomain(url) 10 | results = await resolver.query(domain, record_type) 11 | await asyncio.sleep(0.0000001) 12 | if record_type == 'CNAME': 13 | return [results.cname] if hasattr(results, 'cname') else [] 14 | elif record_type == 'A': 15 | return [result.host for result in results] if results else [] 16 | elif record_type == 'AAAA': 17 | return [result.host for result in results] if results else [] 18 | else: 19 | return [] 20 | except (KeyboardInterrupt, asyncio.CancelledError): 21 | exit(1) 22 | except (socket.gaierror, aiodns.error.DNSError, TimeoutError): 23 | return [] 24 | except Exception as e: 25 | if args.verbose: 26 | logger(f"Exception occurred in the dns resolver module due to: {e}, {type(e)}", "warn", args.no_color) 27 | return [] -------------------------------------------------------------------------------- /subprober/modules/validate/headless.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.logger.logger import logger 2 | from playwright.async_api import async_playwright 3 | import asyncio 4 | import sys 5 | 6 | async def check_browsers(verbose=False, colored=False) -> bool: 7 | try: 8 | async with async_playwright() as playwright: 9 | await playwright.chromium.launch() 10 | return True 11 | except Exception as e: 12 | if verbose: 13 | logger(f"Browsers are not installed for Headless screenshots", "warn", colored) 14 | return False 15 | 16 | async def install_browsers(verbose, colored=False): 17 | try: 18 | process = await asyncio.create_subprocess_exec( 19 | sys.executable, '-m', 'playwright', 'install','chromium' 20 | ) 21 | 22 | stdout, stderr = await process.communicate() 23 | if process.returncode == 0: 24 | logger("Browsers installed successfully", "verbose", colored) 25 | exit(0) 26 | else: 27 | logger(f"Browsers installation failed, please try again manually with command: playwright install chromium", "warn", colored) 28 | exit(1) 29 | except Exception as e: 30 | if verbose: 31 | logger(f"Exception occured in the browser installing module due to: {e}, {type(e)}") 32 | exit(1) -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | with open("README.md", "r") as streamr: 4 | long_description = streamr.read() 5 | 6 | 7 | setup( 8 | name='subprober', 9 | version='3.0.1', 10 | author='D. Sanjai Kumar', 11 | author_email='bughunterz0047@gmail.com', 12 | long_description=long_description, 13 | long_description_content_type="text/markdown", 14 | url="https://github.com/RevoltSecurities/Subprober", 15 | description='Subprober - An essential HTTP multi-purpose Probing Tool for Penetration Testers and Security Researchers with Asynchronous httpx client support', 16 | packages=find_packages(), 17 | install_requires=[ 18 | 'aiodns>=3.2.0', 19 | 'aiofiles>=24.1.0', 20 | 'aiojarm>=0.2.2', 21 | 'alive_progress>=3.2.0', 22 | 'appdirs>=1.4.4', 23 | 'art>=6.4', 24 | 'asynciolimiter>=1.1.1', 25 | 'beautifulsoup4>=4.12.3', 26 | 'bs4>=0.0.2', 27 | 'lxml>=5.3.0', 28 | 'colorama>=0.4.6', 29 | 'cryptography>=44.0.0', 30 | 'fake_useragent>=1.5.1', 31 | 'httpx>=0.28.1', 32 | 'mmh3>=5.0.1', 33 | 'playwright>=1.49.1', 34 | 'Requests>=2.32.3', 35 | 'rich>=13.9.4', 36 | 'setuptools>=75.2.0', 37 | 'simhash>=2.1.2', 38 | 'urllib3>=1.26.18', 39 | 'uvloop>=0.21.0', 40 | 'websockets>=14.1' 41 | ], 42 | entry_points={ 43 | 'console_scripts': [ 44 | 'subprober = subprober.subprober:main' 45 | ] 46 | }, 47 | ) 48 | -------------------------------------------------------------------------------- /subprober/modules/matchers/matchers.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | async def match_by_code(response, code_list=None) -> bool: 4 | if not code_list: 5 | return True 6 | for code in code_list: 7 | if int(response.status_code) == (code): 8 | return True 9 | return False 10 | 11 | async def match_code_range(response, code=None) -> bool: 12 | if code is None: 13 | return True 14 | min_code, max_code = map(int, code.split("-")) 15 | if min_code is None or max_code is None: 16 | return True 17 | if min_code <= response.status_code <= max_code: 18 | return True 19 | return False 20 | 21 | async def match_url_path_contains(response, paths=None) -> bool: 22 | if not paths: 23 | return True 24 | for path in paths: 25 | if str(path) in str(response.url.path): 26 | return True 27 | return False 28 | 29 | async def match_word_body(response, words=None) -> bool: 30 | if not words: 31 | return True 32 | for word in words: 33 | if str(word) in response.text: 34 | return True 35 | return False 36 | 37 | async def match_by_ints(requested_code, code_list=None) -> bool: 38 | if not code_list: 39 | return True 40 | for code in code_list: 41 | if int(requested_code) == int(code): 42 | return True 43 | return False 44 | 45 | async def match_by_regex(response, regexes=None) -> bool: 46 | if not regexes: 47 | return True 48 | for regex in regexes: 49 | if re.search(regex, response.text): 50 | return True 51 | return False 52 | 53 | async def match_response_time(response, max_time=None) -> bool: 54 | if max_time is None: 55 | return True 56 | if float(response.elapsed.total_seconds()) <= float(max_time): 57 | return True 58 | return False -------------------------------------------------------------------------------- /subprober/modules/filters/filters.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | async def filter_by_code(response, code_list=None) -> bool: 4 | if not code_list: 5 | return True 6 | for code in code_list: 7 | if int(response.status_code) == int(code): 8 | return False 9 | return True 10 | 11 | async def filter_code_range(response, code=None) -> bool: 12 | if code is None: 13 | return True 14 | min_code,max_code = map(int, code.split("-")) 15 | if min_code is None or max_code is None: 16 | return True 17 | if min_code <= response.status_code <= max_code: 18 | return False 19 | return True 20 | 21 | async def filter_url_path_contains(response, paths=None) -> bool: 22 | if not paths: 23 | return True 24 | for path in paths: 25 | if str(path) in str(response.url.path): 26 | return False 27 | return True 28 | 29 | async def filter_word_body(response, words=None) -> bool: 30 | if not words: 31 | return True 32 | for word in words: 33 | if str(word) in response.text: 34 | return False 35 | return True 36 | 37 | async def filter_by_ints(requested_code, code_list=None) -> bool: 38 | if not code_list: 39 | return True 40 | for code in code_list: 41 | if int(requested_code) == int(code): 42 | return False 43 | return True 44 | 45 | async def filter_by_regex(response, regexes=None) -> bool: 46 | if not regexes: 47 | return True 48 | for regex in regexes: 49 | if re.search(regex, response.text): 50 | return False 51 | return True 52 | 53 | async def filter_response_time(response, max_time=None) -> bool: 54 | if max_time is None: 55 | return True 56 | if float(response.elapsed.total_seconds()) > float(max_time): 57 | return False 58 | return True 59 | -------------------------------------------------------------------------------- /subprober/modules/logger/logger.py: -------------------------------------------------------------------------------- 1 | from colorama import Fore,Style,init 2 | import sys 3 | import random 4 | init() 5 | 6 | red = Fore.RED 7 | green = Fore.GREEN 8 | magenta = Fore.MAGENTA 9 | cyan = Fore.CYAN 10 | mixed = Fore.RED + Fore.BLUE 11 | blue = Fore.BLUE 12 | yellow = Fore.YELLOW 13 | white = Fore.WHITE 14 | reset = Style.RESET_ALL 15 | bold = Style.BRIGHT 16 | colors = [ green, cyan, blue] 17 | random_color = random.choice(colors) 18 | 19 | 20 | def logger(message: str, level="info", nocolored=False): 21 | if level == "info": 22 | if nocolored: 23 | leveler = "INFO" 24 | else: 25 | leveler = f"{bold}{blue}INFO{reset}" 26 | 27 | elif level == "warn": 28 | if nocolored: 29 | leveler = "WRN" 30 | else: 31 | leveler = f"{bold}{red}WRN{reset}" 32 | 33 | elif level == "error": 34 | if nocolored: 35 | leveler = "ERR" 36 | else: 37 | leveler = f"{bold}{red}ERR{reset}" 38 | 39 | elif level == "verbose": 40 | if nocolored: 41 | leveler = "VRB" 42 | else: 43 | leveler = f"{bold}{green}VRB{reset}" 44 | 45 | elif level == "debug": 46 | if nocolored: 47 | leveler = "DBG" 48 | else: 49 | leveler = f"{bold}{cyan}DBG{reset}" 50 | 51 | else: 52 | if nocolored: 53 | leveler = f"{level.upper()}" 54 | else: 55 | leveler = f"{bold}{green}{level.upper()}{reset}" 56 | 57 | if nocolored: 58 | print(f"[{leveler}]: {message}", file=sys.stderr) 59 | else: 60 | print(f"[{bold}{blue}{leveler}{reset}]: {bold}{white}{message}{reset}", file=sys.stderr) 61 | 62 | def bannerlog(banner: str): 63 | print(banner, file=sys.stderr) 64 | 65 | def stdinlog(message): 66 | print(message) -------------------------------------------------------------------------------- /subprober/modules/hash/hash.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | import mmh3 3 | from simhash import Simhash 4 | from subprober.modules.logger.logger import logger 5 | import asyncio 6 | 7 | def tokenize_for_simhash(text: str) -> list[str]: 8 | return text.lower().split() 9 | 10 | async def Hashgen(response: str, algorithms: list[str], args) -> dict[str]: 11 | try: 12 | response = response.encode("utf-8", errors="ignore").decode("utf-8", errors="ignore") 13 | hashed = {} 14 | 15 | for alg in algorithms: 16 | try: 17 | algo = alg.strip().lower() 18 | if algo == "md5": 19 | hashed[algo] = hashlib.md5(response.encode("utf-8")).hexdigest() 20 | elif algo == "sha1": 21 | hashed[algo] = hashlib.sha1(response.encode("utf-8")).hexdigest() 22 | elif algo == "sha256": 23 | hashed[algo] = hashlib.sha256(response.encode("utf-8")).hexdigest() 24 | elif algo == "sha512": 25 | hashed[algo] = hashlib.sha512(response.encode("utf-8")).hexdigest() 26 | elif algo == "mmh3": 27 | hashed[algo] = str(mmh3.hash(response.encode("utf-8"))) 28 | elif algo == "simhash": 29 | tokens = tokenize_for_simhash(response) 30 | hashed[algo] = str(Simhash(tokens).value) 31 | else: 32 | if args.verbose: 33 | logger(f"Undefined hash algorithm: {algo}", "error", args.no_color) 34 | except Exception as inner_e: 35 | logger( 36 | f"Error with algorithm '{algo}': {inner_e}, {type(inner_e)}", 37 | "error", 38 | args.no_color, 39 | ) 40 | except (KeyboardInterrupt, asyncio.CancelledError): 41 | exit(1) 42 | except Exception as e: 43 | if args.verbose: 44 | logger(f"Exception occurred in Hashgen: {e}, {type(e)} {algo}", "warn", args.no_color) 45 | finally: 46 | return hashed 47 | -------------------------------------------------------------------------------- /subprober/modules/screenshot/screenshot.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.logger.logger import logger 2 | import asyncio 3 | from playwright.async_api import async_playwright 4 | from subprober.modules.utils.utils import Useragents 5 | import os 6 | import base64 7 | 8 | class Headless: 9 | def __init__( 10 | self, 11 | args, 12 | ) -> None: 13 | 14 | self.args = args 15 | self.sandbox = True if os.geteuid() == 0 else False 16 | self.savepath = None 17 | self.chrome_options = [] 18 | 19 | def setup(self) -> None: 20 | try: 21 | self.savepath = self.args.screenshot_path if self.args.screenshot_path else os.path.join(os.getcwd(),"screenshots") 22 | os.makedirs(self.savepath,exist_ok=True) 23 | 24 | if self.args.headless_options: 25 | self.chrome_options = self.args.headless_options.split(",") 26 | except Exception as e: 27 | if self.args.verbose: 28 | logger(f"Exception occured in the headless setup module due to: {e}, {type(e)}", "error", self.args.no_color) 29 | 30 | 31 | async def run(self, url: str, results: dict[str,str]) -> None: 32 | try: 33 | 34 | output = url.replace("://", "_").replace("/", "_") 35 | output_path = os.path.join(self.savepath,f"{output}.png") if not self.args.save_pdf else os.path.join(self.savepath,f"{output}.pdf") 36 | 37 | async with async_playwright() as playwright: 38 | 39 | headers = {"Upgrade-Insecure-Requests": "1"} 40 | 41 | launchers= { 42 | "headless": True, 43 | "timeout": self.args.screenshot_timeout*1000, 44 | "args": ["--ignore-certificate-errors", 45 | "--disable-gpu", 46 | "--disable-crash-reporter", 47 | "--disable-notifications", 48 | "--hide-scrollbars", 49 | "--mute-audio"]+ self.chrome_options 50 | } 51 | 52 | if os.geteuid() == 0: 53 | launchers["args"].extend(["--no-sandbox", "--disable-setuid-sandbox"]) 54 | 55 | if self.args.screenshot_headers: 56 | for header in self.screenshot_headers: 57 | key,value = header.split(":",1) 58 | headers[key.strip()] = value.strip() 59 | 60 | headers["User-Agent"] = Useragents() if self.args.random_agent else "git+Subprober/V2.XD" 61 | path = self.args.system_chrome_path if self.args.system_chrome_path else None 62 | 63 | if self.args.proxy: 64 | launchers["proxy"] = {"server": self.args.proxy} 65 | 66 | browser = await playwright.chromium.launch(**launchers, chromium_sandbox=self.sandbox, executable_path=path) 67 | page = await browser.new_page(extra_http_headers=headers,ignore_https_errors=True) 68 | await page.goto(url) 69 | await asyncio.sleep(self.args.screenshot_idle) 70 | 71 | if self.args.save_pdf : 72 | bytess = await page.pdf(path=output_path) 73 | else: 74 | bytess = await page.screenshot(path=output_path, full_page=True) 75 | 76 | if self.args.full_output or self.args.include_bytes and self.args.json: 77 | results["HeadlessBody"] = base64.b64encode(bytess).decode() 78 | 79 | if self.args.full_output or self.args.json: 80 | results["ScreenshotPath"] = output_path 81 | 82 | except Exception as e: 83 | if self.args.verbose: 84 | logger(f"Exception occured in the headless run module due to: {e}, {type(e)}", "error", self.args.no_color) 85 | finally: 86 | await browser.close() -------------------------------------------------------------------------------- /subprober/modules/utils/utils.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import os 3 | from urllib.parse import urlparse,urlunparse, urljoin 4 | from subprober.modules.logger.logger import logger 5 | import aiofiles 6 | from fake_useragent import UserAgent 7 | from itertools import islice 8 | from requests.cookies import RequestsCookieJar 9 | from urllib.parse import urlparse 10 | 11 | def extract_cookies(cookie_jar): 12 | formatted_cookies = [] 13 | if cookie_jar: 14 | for cookie in cookie_jar: 15 | cookie_data = { 16 | "name": cookie.name, 17 | "value": cookie.value, 18 | "domain": cookie.domain, 19 | "path": cookie.path, 20 | "expires": cookie.expires, 21 | "secure": cookie.secure, 22 | "http_only": cookie.has_nonstandard_attr('HttpOnly') 23 | } 24 | formatted_cookies.append(cookie_data) 25 | return formatted_cookies 26 | 27 | async def Reader(file: str, args) -> list[str]: 28 | try: 29 | content = [] 30 | async with aiofiles.open(file, "r") as streamr: 31 | data = await streamr.read() 32 | data = data.splitlines() 33 | for d in data: 34 | content.append(d) 35 | return content 36 | except (KeyboardInterrupt, asyncio.CancelledError): 37 | exit(1) 38 | except PermissionError: 39 | logger(f"{file} have insufficient permission to read", "warn", args.no_color) 40 | exit(1) 41 | except FileNotFoundError: 42 | logger(f"{file}: no such file or directory exist", "error", args.no_color) 43 | exit(1) 44 | except Exception as e: 45 | logger(f"Exception occured in return reader due to: {e}, {type(e)}", "warn", args.no_color) 46 | exit(1) 47 | 48 | async def permissions(filename, args) -> bool: 49 | try: 50 | async with aiofiles.open(filename, mode='a') as file: 51 | pass 52 | return True 53 | except (KeyboardInterrupt, asyncio.CancelledError): 54 | exit(1) 55 | except PermissionError: 56 | logger(f"{filename} have insufficient permission to write", "warn", args.no_color) 57 | exit(1) 58 | except Exception as e: 59 | logger(f"Exception occured in util permission checker due to: {e}, {type(e)}", "warn", args.no_color) 60 | 61 | 62 | def GetDomain(url) -> str: 63 | try: 64 | parsed_url = urlparse(url) 65 | domain = parsed_url.hostname 66 | return domain 67 | except (KeyboardInterrupt, asyncio.CancelledError): 68 | exit(1) 69 | except Exception as e: 70 | pass 71 | 72 | def extractor(data): 73 | extracted = [] 74 | final = data.split(",") 75 | extracted.extend([hash for hash in final]) 76 | 77 | def Useragents() -> str: 78 | return UserAgent().random 79 | 80 | def chunker(data, size=100000): 81 | it = iter(data) 82 | while chunk := list(islice(it, size)): 83 | yield chunk 84 | 85 | def string_to_str_list(words) -> list[str] | None: 86 | if words is None: 87 | return None 88 | values = [str(word) for word in words.split(",")] 89 | return values 90 | 91 | def string_to_int_list(words) -> list[int] | None: 92 | if words is None: 93 | return None 94 | values = [int(num) for num in words.split(",")] 95 | return values 96 | 97 | 98 | def validate_urls(urls: list[str], http_include=False) -> list[str]: 99 | validated_urls = [] 100 | for url in urls: 101 | parsed = urlparse(url) 102 | if parsed.scheme in ("http", "https"): 103 | validated_urls.append(url) 104 | else: 105 | validated_urls.append(f"https://{url}") 106 | if not http_include: 107 | validated_urls.append(f"http://{url}") 108 | return list(set(validated_urls)) 109 | 110 | def add_ports_to_urls(urls: list[str], ports=None) -> list[str]: 111 | urls_with_ports = [] 112 | for url in urls: 113 | parsed = urlparse(url) 114 | scheme = parsed.scheme 115 | 116 | if ports: 117 | for port in ports: 118 | new_netloc = f"{parsed.hostname}:{port}" 119 | urls_with_ports.append(urlunparse(parsed._replace(netloc=new_netloc))) 120 | else: 121 | new_netloc = f"{parsed.hostname}" 122 | urls_with_ports.append(urlunparse(parsed._replace(netloc=new_netloc))) 123 | return urls_with_ports 124 | 125 | 126 | def add_paths_to_urls(urls: list[str], paths=None)-> list[str]: 127 | if not paths: 128 | return urls 129 | urls_with_paths = [] 130 | for url in urls: 131 | for path in paths: 132 | urls_with_paths.append(urljoin(url, path)) 133 | return urls_with_paths -------------------------------------------------------------------------------- /subprober/modules/tls/tls.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from cryptography import x509 3 | from cryptography.hazmat.backends import default_backend 4 | from cryptography.hazmat.primitives import hashes 5 | from subprober.modules.logger.logger import logger 6 | 7 | 8 | async def tlsinfo(args, network_stream=None) -> dict: 9 | tlsinfo ={} 10 | if network_stream: 11 | try: 12 | ssl_bin = network_stream.get_extra_info("ssl_object").getpeercert(True) 13 | except Exception as e: 14 | return tlsinfo 15 | cert = x509.load_der_x509_certificate(ssl_bin, default_backend()) 16 | tlsinfo["Serial Number"] = cert.serial_number 17 | tlsinfo["Version"] = cert.version.name 18 | tlsinfo["Signature Algorithm"] = cert.signature_algorithm_oid._name, 19 | tlsinfo["Issuer"] = {attr.oid._name: attr.value for attr in cert.issuer} 20 | tlsinfo["Subject"] = {attr.oid._name: attr.value for attr in cert.subject} 21 | tlsinfo["Validity"] = {"Not Before (UTC)": cert.not_valid_before_utc.strftime("%Y-%m-%dT%H:%M:%S"), "Not After (UTC)": cert.not_valid_after_utc.strftime("%Y-%m-%dT%H:%M:%S"),} 22 | tlsinfo["Public Key Algorithm"] = cert.public_key().__class__.__name__ 23 | tlsinfo["Pub Key Size"] = cert.public_key().key_size 24 | tlsinfo["SHA-1 Fingerprint"] = cert.fingerprint(hashes.SHA1()).hex() 25 | tlsinfo["SHA-256 Fingerprint"] = cert.fingerprint(hashes.SHA256()).hex() 26 | tlsinfo["MD5 Fingerprint"] = cert.fingerprint(hashes.MD5()).hex() 27 | tlsinfo["Subject Alternative Names (SANs)"] = [] 28 | tlsinfo["Key Usage"] = {} 29 | tlsinfo["Extended Key Usage"] = [] 30 | tlsinfo["Certificate Policies"] = [] 31 | tlsinfo["Basic Constraints"] = {} 32 | tlsinfo["Authority Information Access"] = [] 33 | tlsinfo["CRL Distribution Points"] = [] 34 | tlsinfo["OCSP URLs"]=[] 35 | 36 | try: 37 | san_extension = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName) 38 | tlsinfo["Subject Alternative Names (SANs)"] = san_extension.value.get_values_for_type(x509.DNSName) 39 | except x509.ExtensionNotFound: 40 | pass 41 | 42 | try: 43 | key_usage = cert.extensions.get_extension_for_class(x509.KeyUsage) 44 | tlsinfo["Key Usage"] = { 45 | "Digital Signature": key_usage.value.digital_signature, 46 | "Content Commitment": key_usage.value.content_commitment, 47 | "Key Encipherment": key_usage.value.key_encipherment, 48 | "Data Encipherment": key_usage.value.data_encipherment, 49 | "Key Agreement": key_usage.value.key_agreement, 50 | "Key Cert Sign": key_usage.value.key_cert_sign, 51 | "CRL Sign": key_usage.value.crl_sign, 52 | } 53 | except x509.ExtensionNotFound: 54 | pass 55 | 56 | try: 57 | ext_key_usage = cert.extensions.get_extension_for_class(x509.ExtendedKeyUsage) 58 | tlsinfo["Extended Key Usage"] = [eku.dotted_string for eku in ext_key_usage.value] 59 | except x509.ExtensionNotFound: 60 | pass 61 | 62 | try: 63 | cert_policies = cert.extensions.get_extension_for_oid(x509.ExtensionOID.CERTIFICATE_POLICIES).value 64 | tlsinfo["Certificate Policies"] = [{"Policy": policy.policy_identifier.dotted_string} for policy in cert_policies] 65 | except x509.ExtensionNotFound: 66 | pass 67 | 68 | try: 69 | authority_info = cert.extensions.get_extension_for_oid(x509.ExtensionOID.AUTHORITY_INFORMATION_ACCESS).value 70 | tlsinfo["Authority Information Access"] = [f"{desc.access_method.dotted_string}: {desc.access_location.value}" for desc in authority_info] 71 | except x509.ExtensionNotFound: 72 | pass 73 | 74 | try: 75 | crl_distribution_points = cert.extensions.get_extension_for_oid(x509.ExtensionOID.CRL_DISTRIBUTION_POINTS).value 76 | tlsinfo["CRL Distribution Points"] = [dp.full_name[0].value for dp in crl_distribution_points if dp.full_name] 77 | except x509.ExtensionNotFound: 78 | pass 79 | 80 | try: 81 | ocsp = cert.extensions.get_extension_for_oid(x509.ExtensionOID.AUTHORITY_INFORMATION_ACCESS).value 82 | tlsinfo["OCSP URLs"] = [ desc.access_location.value for desc in ocsp if desc.access_method == x509.AuthorityInformationAccessOID.OCSP ] 83 | except x509.ExtensionNotFound: 84 | pass 85 | 86 | try: 87 | basic_constraints = cert.extensions.get_extension_for_class(x509.BasicConstraints) 88 | tlsinfo["Basic Constraints"] = {"CA": basic_constraints.value.ca, "Path Length": basic_constraints.value.path_length} 89 | except x509.ExtensionNotFound: 90 | pass 91 | 92 | return tlsinfo 93 | else: 94 | return tlsinfo 95 | 96 | -------------------------------------------------------------------------------- /subprober/modules/cli/cli.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | from subprober.modules.logger.logger import logger 4 | 5 | 6 | 7 | def cli(): 8 | 9 | try: 10 | parser = argparse.ArgumentParser(add_help=False,usage=argparse.SUPPRESS,exit_on_error=False ) 11 | parser.add_argument("-f", "--filename", type=str) 12 | parser.add_argument("-h", "--help", action="store_true") 13 | parser.add_argument("-u", "--url", type=str ) 14 | parser.add_argument("-o", "--output", type=str) 15 | parser.add_argument("-c", "--concurrency", type=int, default=100) 16 | parser.add_argument("-tl", "--title", action="store_true") 17 | parser.add_argument("-to", "--timeout", type=int, default=10) 18 | parser.add_argument("-sv", "--server", action="store_true") 19 | parser.add_argument("-l", "--location", action="store_true") 20 | parser.add_argument("-wc", "--word-count", action="store_true") 21 | parser.add_argument("-apt", "--application-type", action="store_true") 22 | parser.add_argument("-fc", "--filter-code", type=str) 23 | parser.add_argument("-mc", "--match-code", type=str) 24 | parser.add_argument("-s", "--silent", action="store_true") 25 | parser.add_argument("-v", "--verbose", action="store_true") 26 | parser.add_argument("-p", "--path", type=str) 27 | parser.add_argument("-px", "--proxy", type=str) 28 | parser.add_argument("-ar", "--allow-redirect", action="store_true") 29 | parser.add_argument("-nc", "--no-color", action="store_true") 30 | parser.add_argument("-up", "--update", action="store_true") 31 | parser.add_argument("-das", "--disable-auto-save", action="store_true") 32 | parser.add_argument("-dhp", "--disable-http-probe", action="store_true") 33 | parser.add_argument("-X", "--method", type=str, choices=["get", "post", "head", "put", "delete", "patch", "trace", "connect", "options"], default="get") 34 | parser.add_argument("-H", "--header", action="append") 35 | parser.add_argument("-sc", "--status-code", action="store_true") 36 | parser.add_argument("-ra", "--random-agent", action="store_true") 37 | parser.add_argument("-ss", "--screenshot", action="store_true") 38 | parser.add_argument("-st", "--screenshot-timeout", type=int, default=15) 39 | parser.add_argument("-sp", "--screenshot-path", type=str) 40 | parser.add_argument("-ip", "--ipaddress", action="store_true") 41 | parser.add_argument("-cn", "--cname", action="store_true") 42 | parser.add_argument("-maxr", "--max-redirection", type=int, default=10) 43 | parser.add_argument("-sup", "--show-updates", action="store_true") 44 | parser.add_argument("-http2", "--http2", action="store_true") 45 | parser.add_argument("-htv", "--http-version", action="store_true") 46 | parser.add_argument("-hrs", "--http-reason", action="store_true") 47 | parser.add_argument("-jarm", "--jarm-fingerprint", action="store_true") 48 | parser.add_argument("-rpt", "--response-time", action="store_true") 49 | parser.add_argument("-sd", "--secret-debug", action="store_true") # this isn't for you! 50 | parser.add_argument("-rtl", "--rate-limit", type=int, default=1000) 51 | parser.add_argument("-d", "--delay", type=float, default=0.5) 52 | parser.add_argument("-lc", "--line-count", action="store_true") 53 | parser.add_argument("-pt", "--port", type=str) 54 | parser.add_argument("-rts", "--retries", type=int, default=0) 55 | parser.add_argument("-wss", "--websocket", action="store_true") 56 | parser.add_argument("-dmt", "--display-method", action="store_true") 57 | parser.add_argument("-J", "--json", action="store_true") 58 | parser.add_argument("-bp", "--body-preview", action="store_true") 59 | parser.add_argument("-hash", "--hash", type=str) 60 | parser.add_argument("-cl", "--content-length", action="store_true") 61 | parser.add_argument("-tls", "--tls", action="store_true") 62 | parser.add_argument("-scp", "--system-chrome-path", action="store_true") 63 | parser.add_argument("-aaa", "--aaa-records", action="store_true") 64 | parser.add_argument("-fcr", "--filter-code-range", type=str) 65 | parser.add_argument("-mcr", "--match-code-range", type=str) 66 | parser.add_argument("-mrt", "--match-response-time", type=float) 67 | parser.add_argument("-frt","--filter-response-time", type=float) 68 | parser.add_argument("-ms","--match-string", type=str) 69 | parser.add_argument("-fs","--filter-string", type=str) 70 | parser.add_argument("-mr","--match-regex", type=str) 71 | parser.add_argument("-fr","--filter-regex", type=str) 72 | parser.add_argument("-mpt", "--match-path", type=str) 73 | parser.add_argument("-fpt","--filter-path", type=str) 74 | parser.add_argument("-ml", "--match-length", type=str) 75 | parser.add_argument("-fl", "--filter-length", type=str) 76 | parser.add_argument("-mlc", "--match-line-count", type=str) 77 | parser.add_argument("-flc","--filter-line-count", type=str) 78 | parser.add_argument("-mwc","--match-word-count",type=str) 79 | parser.add_argument("-fwc","--filter-word-count",type=str) 80 | parser.add_argument("-rdu", "--redirect-urls", action="store_true") 81 | parser.add_argument("-rdh", "--redirect-history", action="store_true") 82 | parser.add_argument("-rsc", "--redirect-status-codes", action="store_true") 83 | parser.add_argument("-rqh", "--request-headers", action="store_true") 84 | parser.add_argument("-rsh", "--response-headers", action="store_true") 85 | parser.add_argument("-fo", "--full-output", action="store_true") 86 | parser.add_argument("-sni", "--sni-hostname", type=str) 87 | parser.add_argument("-HH", "--screenshot-headers", action="append") 88 | parser.add_argument("-pdf", "--save-pdf", action="store_true") 89 | parser.add_argument("-sid", "--screenshot-idle", default=1, type=int) 90 | parser.add_argument("-hos", "--headless-options", type=str) 91 | parser.add_argument("-icb", "--include-bytes", action="store_true") 92 | parser.add_argument("-sct", "--screenshot-threads", type=int, default=40) 93 | return parser.parse_args() 94 | except argparse.ArgumentError as e: 95 | print(f"Please use the command for more information: subprober -h {e}", "warn") 96 | exit(1) 97 | except argparse.ArgumentTypeError as e: 98 | print(f"Please use the command for more information: subprober -h {e}", "warn") 99 | exit(1) 100 | except KeyboardInterrupt as e: 101 | exit(1) 102 | except Exception as e: 103 | logger(f"Exception occured in the cli module due to: {e}, {type(e)}") 104 | exit(1) -------------------------------------------------------------------------------- /subprober/modules/handler.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | from colorama import Fore, Style, init 3 | import sys 4 | import os 5 | import random 6 | 7 | init() 8 | 9 | red = Fore.RED 10 | green = Fore.GREEN 11 | magenta = Fore.MAGENTA 12 | cyan = Fore.CYAN 13 | mixed = Fore.RED + Fore.BLUE 14 | blue = Fore.BLUE 15 | yellow = Fore.YELLOW 16 | white = Fore.WHITE 17 | lm = Fore.LIGHTMAGENTA_EX 18 | reset = Style.RESET_ALL 19 | bold = Style.BRIGHT 20 | colors = [ green, cyan, blue] 21 | random_color = random.choice(colors) 22 | 23 | from subprober.modules.cli.cli import cli 24 | from subprober.modules.banner.banner import banner 25 | from subprober.modules.config.config import configdir 26 | from subprober.modules.extender.extender import extender 27 | from subprober.modules.help.help import help 28 | from subprober.modules.version.version import Gitversion 29 | from subprober.modules.core.core import Subprober 30 | from subprober.modules.logger.logger import logger, bannerlog 31 | from subprober.modules.update.update import updatelog 32 | from subprober.modules.utils.utils import * 33 | from subprober.modules.validate.headless import check_browsers, install_browsers 34 | 35 | extender() 36 | args = cli() 37 | configpath = configdir() 38 | banner = banner() 39 | git = "v3.0.1" 40 | 41 | def version(): 42 | try: 43 | latest = Gitversion() 44 | if latest and latest == git: 45 | print(f"[{blue}{bold}version{reset}]:{bold}{white}subprober current version {git} ({green}latest{reset}{bold}{white}){reset}", file=sys.stderr) 46 | elif latest and latest != git: 47 | print(f"[{blue}{bold}Version{reset}]: {bold}{white}subprober current version {git} ({red}outdated{reset}{bold}{white}){reset}", file=sys.stderr) 48 | else: 49 | logger(f"Unable to get the latest version of subprober", "warn", args.no_color) 50 | except (KeyboardInterrupt, asyncio.CancelledError): 51 | exit(1) 52 | except Exception as e: 53 | if args.verbose: 54 | logger(f"Exception occured in the version check module due to: {e}, {type(e)}", "warn", args.no_color) 55 | 56 | async def Updates(): 57 | try: 58 | if args.show_updates: 59 | updatelog() 60 | exit(0) 61 | 62 | if args.update: 63 | latest = Gitversion() 64 | if latest and latest == git: 65 | logger("Subprober is already in latest version", "info", args.no_color) 66 | else: 67 | logger(f"Downloading latest version of subprober", "info", args.no_color) 68 | 69 | process = await asyncio.create_subprocess_exec( 70 | "pip", "install", "-U", "git+https://github.com/RevoltSecurities/Subprober", "--break-system-packages", 71 | stdout=asyncio.subprocess.PIPE, 72 | stderr=asyncio.subprocess.PIPE 73 | ) 74 | stdout, stderr = await process.communicate() 75 | if process.returncode == 0: 76 | logger("successfully updated subprober to latest version", "info", args.no_color) 77 | updatelog() 78 | exit(0) 79 | else: 80 | logger("failed for updating latest version of subprober, try to update manually", "warn", args.no_color) 81 | exit(1) 82 | except Exception as e: 83 | if args.secret_debug: 84 | print(f"Exception in handler update: {e}, {type(e)}") 85 | 86 | async def start(urls) -> None: 87 | try: 88 | urls = validate_urls(urls,args.disable_http_probe) 89 | urls = add_ports_to_urls(urls, ports= string_to_str_list(args.port) if args.port else None) 90 | if args.path: 91 | path = await Reader(args.path,args) if os.path.isfile(args.path) else string_to_str_list(args.path) 92 | else: 93 | path = None 94 | 95 | urls = add_paths_to_urls(urls,path) 96 | 97 | subprober = Subprober( 98 | urls, 99 | args, 100 | hashes=string_to_str_list(args.hash), 101 | mc=string_to_int_list(args.match_code), 102 | fc=string_to_int_list(args.filter_code), 103 | mcr=args.match_code_range, 104 | fcr=args.filter_code_range, 105 | ms=string_to_str_list(args.match_string), 106 | fs=string_to_str_list(args.filter_string), 107 | mrg=string_to_str_list(args.match_regex), 108 | frg=string_to_str_list(args.filter_regex), 109 | mpt=string_to_str_list(args.match_path), 110 | fpt=string_to_str_list(args.filter_path), 111 | ml=string_to_int_list(args.match_length), 112 | fl=string_to_int_list(args.filter_length), 113 | mlc=string_to_int_list(args.match_line_count), 114 | flc=string_to_int_list(args.filter_line_count), 115 | mwc=string_to_int_list(args.match_word_count), 116 | fwc=string_to_int_list(args.filter_word_count), 117 | mrt=args.match_response_time, 118 | frt=args.filter_response_time 119 | ) 120 | await subprober.start() 121 | exit(0) 122 | except (KeyboardInterrupt, asyncio.CancelledError): 123 | exit(1) 124 | except Exception as e: 125 | logger(f"Exception occured in the handler start module due to: {e}, {type(e)}","warn", args.no_color) 126 | 127 | async def handler(): 128 | try: 129 | if args.help: 130 | bannerlog(banner) 131 | help() 132 | exit(0) 133 | 134 | if not args.silent: 135 | bannerlog(banner) 136 | version() 137 | 138 | if args.update or args.show_updates: 139 | await Updates() 140 | exit(0) 141 | 142 | if args.screenshot: 143 | browser = await check_browsers(args.verbose, args.no_color) 144 | if browser: 145 | if args.verbose: 146 | logger(f"Browsers are already installed!", "verbose", args.no_color) 147 | if not browser: 148 | logger(f"Installing Browsers for Headless modes", "info", args.no_color) 149 | await install_browsers(args.verbose, args.no_color) 150 | 151 | if args.url: 152 | if args.output: 153 | await permissions(args.output, args) 154 | urls = string_to_str_list(args.url) 155 | await start(urls) 156 | exit(0) 157 | 158 | if args.filename: 159 | if args.output: 160 | await permissions(args.output, args) 161 | urls = await Reader(args.filename,args) 162 | if urls is None: 163 | if args.verbose: 164 | logger(f"Error occured in the urls file reader due to no data found","error",args.no_color) 165 | await start(urls) 166 | exit(0) 167 | 168 | if sys.stdin.isatty(): 169 | logger(f"no inputs provided for subprober", "warn", args.no_color) 170 | exit(1) 171 | 172 | else: 173 | if args.output: 174 | await permissions(args.output, args) 175 | urls = [domain.strip() for domain in sys.stdin if domain.strip()] 176 | await start(urls) 177 | exit(0) 178 | except (KeyboardInterrupt,asyncio.CancelledError): 179 | exit(1) 180 | except Exception as e: 181 | logger(f"Exception occured in the handler module due to: {e}, {type(e)}","warn",args.no_color) 182 | 183 | def Main(): 184 | try: 185 | asyncio.run(handler()) 186 | except (KeyboardInterrupt, asyncio.CancelledError): 187 | exit(1) -------------------------------------------------------------------------------- /subprober/modules/help/help.py: -------------------------------------------------------------------------------- 1 | from subprober.modules.logger.logger import blue, bold, white, reset 2 | 3 | def help(): 4 | print(f""" 5 | {bold}{white}Subprober - An essential HTTP multi-purpose Probing Tool for Penetration Testers and Security Researchers with Asynchronous httpx client support 6 | 7 | {bold}[{bold}{blue}Description{reset}{bold}{white}]{reset} : 8 | 9 | {bold}{white}Subprober is a high-performance tool designed for probing and extracting vital information efficiently with Asynchronous concurrency performance{reset} 10 | 11 | {bold}[{bold}{blue}Options{reset}{bold}{white}]{reset}:{reset}{bold}{white} 12 | 13 | {bold}[{bold}{blue}INPUT{reset}{bold}{white}]{reset}:{reset}{bold}{white} 14 | 15 | -f, --filename specify the filename containing a list of Urls to probe 16 | -u, --url specify a Url to probe and supports comma-separated values (-u google.com,https://hackerone.com) 17 | stdin/stdout subprober supports both stdin/stdout and enables -nc to pipe the output of subprober 18 | 19 | {bold}[{bold}{blue}PROBES{reset}{bold}{white}]{reset}:{reset}{bold}{white} 20 | 21 | -sc, --status-code display the status code of the host 22 | -tl, --title display the title of host 23 | -sv, --server display the server name of the host 24 | -wc, --word-count display the HTTP response word count 25 | -lc, --line-count display the HTTP response line count 26 | -cl, --content-length display the HTTP response content length 27 | -l , --location display the redirected location of the host 28 | -apt, --application-type display the content type of the host 29 | -ip, --ipaddress display the IPs of the host 30 | -cn, --cname display the CNAMEs of the host 31 | -aaa, --aaa-records display the AAAA records of the host 32 | -htv, --http-version display the server supported HTTP version of the host 33 | -hrs, --http-reason display the reason for HTTP connection of the host 34 | -jarm, --jarm-fingerprint display the JARM fingerprint hash of the host 35 | -rpt, --response-time display the response time for the successful request 36 | -wss, --websocket display the server supports websockets 37 | -hash, --hash display response body in hash format (supported hashes: md5, mmh3, simhash, sha1, sha256, sha512) 38 | -dmt, --display-method display the method of the HTTP request 39 | -bp, --body-preview display the HTTP response body in first n number of characters (default: 100) 40 | 41 | {bold}[{bold}{blue}CONFIG{reset}{bold}{white}]{reset}:{reset}{bold}{white} 42 | 43 | -dhp, --disable-http-probe disables subprober from probing HTTP protocols and only for HTTPS when no protocol is specified 44 | -X, --method request methods to probe and get response (supported: get, post, head, put, delete, patch, trace, connect, options) (default: get) 45 | -H, --header add custom headers for probing and -H can be used multiple times to pass multiple header values (ex: -H application/json -H X-Forwarded-Host: 127.0.0.1) 46 | -ra, --random-agent enable Random User-Agent to use for probing and applies same to screenshots. (default: subprober/Alpha) 47 | -px, --proxy specify a proxy to send the requests through it (ex: http://127.0.0.1:8080) 48 | -ar, --allow-redirect enable following redirections 49 | -maxr, --max-redirection set max value to follow redirections (default: 10) 50 | -http2, --http2 enable to request with HTTP/2 support (default: Http/1.1) 51 | -sni, --sni-hostname set custom TLS SNI host name for requests. 52 | 53 | {bold}[{bold}{blue}MISCELLANEOUS{reset}{bold}{white}]{reset}:{reset}{bold}{white} 54 | 55 | -p, --path specify a path or text file of paths for probing and getting results (example: -p admin.php or -p paths.txt) 56 | -pt, --port set custom port for making HTTP request and default ports are 80,443 based on the url scheme 57 | -tls, --tls grabs the TLS data for the requested host 58 | 59 | {bold}[{bold}{blue}HEADLESS{reset}{bold}{white}]{reset}:{reset}{bold}{white} 60 | 61 | -ss, --screenshot enable to take screenshots of the page using headless browsers with asynchronous performance 62 | -st, --screenshot-timeout set a timeout value for taking screenshots (default: 15) 63 | -scp, --system-chrome-path specify the executable path of the chromedriver to use system chrome to take screenshots 64 | -pdf, --save-pdf enable to save the screenshot image in the pdf format (default: png) 65 | -HH , --screenshot-headers add custom headers for authenticated screenshots 66 | -icb, --include-bytes enable to include the screenshot bytes in output when json output enabled 67 | -hos, --headless-options set additional chrome headless browser options and supports comma-separated values (-ho "--start-maximized") 68 | -sid, --screenshot-idle set custom idle time in seconds before taking screenshots (default: 1) 69 | -sp, --screenshot-path specify a directory path to store screenshot results (default: currentdir/screenshots) 70 | 71 | {bold}[{bold}{blue}MATCHERS{reset}{bold}{white}]{reset}:{reset}{bold}{white} 72 | 73 | -mc, --match-code match http response by specified status codes and supports comma-separated values (-mc 200,302) 74 | -mcr, --match-code-range match http response by specified status code range and supports single value (-mcr 200-299) 75 | -ms, --match-string match http response containing the specified string and supports comma-separated values (-ms admin,login) 76 | -mr, --match-regex match http response matching the specified regex and supports comma-separated values (-mr .*admin.*,.*login.*) 77 | -mpt, --match-path match http response by URL path and supports comma-separated values (-mpt /admin/wp-ajax.php,/wp-json) 78 | -ml, --match-length match http response by specified response length and supports comma-separated values (-ml 1024,2048) 79 | -mlc, --match-line-count match http response by specified response line count and supports comma-separated values (-mlc 10,50) 80 | -mwc, --match-word-count match http response by specified word count and supports comma-separated values (-mwc 100,500) 81 | -mrt, --match-response-time match http response exceeding the specified minimum response time in seconds (-mrt 2.30) 82 | 83 | {bold}[{bold}{blue}FILTERS{reset}{bold}{white}]{reset}:{reset}{bold}{white} 84 | 85 | -fc, --filter-code filter http response by specified status codes and supports comma-separated values (-fc 404,500) 86 | -fcr, --filter-code-range filter http response by specified status code range and supports single value (-fcr 400-499) 87 | -fs, --filter-string filter http response containing the specified string and supports comma-separated values (-fs error,not found) 88 | -fr, --filter-regex filter http response matching the specified regex and supports comma-separated values (-fr .*admin.*,.*login.*) 89 | -fpt, --filter-path filter http response by URL path and supports comma-separated values (-fpt /error,404.html) 90 | -fl, --filter-length filter http response by specified response length and supports comma-separated values (-fl 1024,2048) 91 | -flc, --filter-line-count filter http response by specified response line count and supports comma-separated values (-flc 10,50) 92 | -fwc, --filter-word-count filter http response by specified response word count and supports comma-separated values (-fwc 100,500) 93 | -frt, --filter-response-time filter http response exceeding the specified maximum response time in seconds (-frt 2.30) 94 | 95 | {bold}[{bold}{blue}OUTPUT{reset}{bold}{white}]{reset}:{reset}{bold}{white} 96 | 97 | -o, --output define the output filename to store the results of the probing operation. 98 | -das, --disable-auto-save disable the auto-save of results when no output file is specified. 99 | -J, --json store and display output in JSON format (includes only data from enabled options). 100 | -rdu, --redirect-urls display the redirect URLs in the output (requires -J and -ar to enabled to enabled). 101 | -rdh, --redirect-history display the full redirect history (requires -J and -ar to enabled). 102 | -rsc, --redirect-status-codes display the status codes for redirections (requires -J and -ar to enabled). 103 | -rqh, --request-headers include request headers in the output (requires -J and -ar to enabled). 104 | -rsh, --response-headers include response headers in the output (requires -J and -ar to enabled). 105 | -fo, --full-output include all available data in the output (requires -J to enabled and doesn't overrides websocket,jarm,hashes options). 106 | 107 | {bold}[{bold}{blue}RATE-LIMIT{reset}{bold}{white}]{reset}:{reset}{bold}{white} 108 | 109 | -c, --concurrency set the concurrency level for sending http requests (default: 100) 110 | -rtl, --rate-limit set a rate limit for sending a maximum number of requests per second (default: 1000) 111 | -sct, --screenshot-threads set a threads level for taking screenshots (default: 40) 112 | 113 | {bold}[{bold}{blue}Optimization{reset}{bold}{white}]{reset}:{reset}{bold}{white} 114 | 115 | -to, --timeout set a custom timeout value for sending requests. 116 | -d, --delay set a delay in seconds before sending each request (default: 0.5) 117 | -rts, --retries set a number of retries if a request fails to connect (default: 0) 118 | 119 | {bold}[{bold}{blue}UPDATES{reset}{bold}{white}]{reset}:{reset}{bold}{white} 120 | 121 | -up, --update update subprober to the latest version (pip required to be installed) 122 | -sup, --show-updates display the current or latest version of subprober updates 123 | 124 | {bold}[{bold}{blue}DEBUG{reset}{bold}{white}]{reset}:{reset}{bold}{white} 125 | 126 | -h, --help display this help message and exit! 127 | -s, --silent enable silent mode to suppress the display of Subprober banner and version information. 128 | -v, --verbose enable verbose mode to display error results on the console. 129 | -nc, --no-color enable to display the output without any CLI colors{reset}""") 130 | exit(0) 131 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Subprober 2 | An essential HTTP multi-purpose Probing Tool for Penetration Testers and Security Researchers with Asynchronous httpx client support 3 | 4 | ![GitHub last commit](https://img.shields.io/github/last-commit/RevoltSecurities/Subprober) ![GitHub release (latest by date)](https://img.shields.io/github/v/release/RevoltSecurities/Subprober) [![GitHub license](https://img.shields.io/github/license/RevoltSecurities/Subprober)](https://github.com/RevoltSecurities/Subprober/blob/main/LICENSE) 5 | 6 | ### Overview 7 | 8 | Subprober is a powerful and efficient tool designed for penetration testers and security professionals. This release introduces several enhancements, bug fixes, and new features to elevate your probing experience. Subprober facilitates fast and reliable information extraction, making it an invaluable asset for penetration testing workflows. 9 | 10 |

11 | 12 |
13 |

14 | 15 | - Fast and configurable probings 16 | - Supported Inputs: hosts, URLS, IPs 17 | - Supports multiple methods http requests 18 | - Supports proxies and customizable Header for probing 19 | - Progress your probing tasks 20 | 21 | 22 | ### Subprober Probing Configuration: 23 | 24 | | Probes | Default Check | Probes | Default Check | 25 | |----------------------|---------------|-------------------------|---------------| 26 | | Url | True | UrlScheme | False | 27 | | Title | True | Ports | False | 28 | | Status code | True | Paths | False | 29 | | Response Length | True | HTTP2 | False | 30 | | Server | True | Response Body Hash | False | 31 | | Content Type | True | HTTP Version | True | 32 | | Follow redirection | False | HTTP Method | True | 33 | | Path | False | Body Preview | True | 34 | | Redirect location | False | Redirect History | True | 35 | | Max redirection | False | Response Reason | True | 36 | | IP address of Host | False | Word Count | True | 37 | | Cname of Host | False | AAAA Record of Host | False | 38 | | Jarm | False | Response Time | True | 39 | | Web Socket | True | Line Count | True | 40 | | TLS Data | False | Redirect Location | True | 41 | 42 | 43 | 44 | ### Installation 45 | 46 | **To install Subprober you need python latest version to be installed and then you can follow the below steps to install subprober** 47 | 48 | **PIP Installation:** 49 | 50 | ```bash 51 | pip install git+https://github.com/RevoltSecurities/Subprober.git 52 | subprober -h 53 | ``` 54 | 55 | **PIPX Installation:** 56 | ```bash 57 | pipx install git+https://github.com/RevoltSecurities/Subprober.git 58 | subprober -h 59 | ``` 60 | 61 | **GIT Installation:** 62 | ```bash 63 | git clone https://github.com/RevoltSecurities/SubProber.git 64 | cd Subprober 65 | pip install . 66 | subprober -h 67 | ``` 68 | 69 | 70 | ### Usage 71 | 72 | ```yaml 73 | subprober -h 74 | 75 | _____ __ ____ __ 76 | / ___/__ __/ /_ / __ \_________ / /_ ___ _____ 77 | \__ \/ / / / __ \/ /_/ / ___/ __ \/ __ \/ _ \/ ___/ 78 | ___/ / /_/ / /_/ / ____/ / / /_/ / /_/ / __/ / 79 | /____/\__,_/_.___/_/ /_/ \____/_.___/\___/_/ 80 | 81 | 82 | 83 | - RevoltSecurities 84 | 85 | 86 | Subprober - An essential HTTP multi-purpose Probing Tool for Penetration Testers and Security Researchers with Asynchronous httpx client support 87 | 88 | [Description] : 89 | 90 | Subprober is a high-performance tool designed for probing and extracting vital information efficiently with Asynchronous concurrency performance 91 | 92 | [Options]: 93 | 94 | [INPUT]: 95 | 96 | -f, --filename specify the filename containing a list of Urls to probe 97 | -u, --url specify a Url to probe and supports comma-separated values (-u google.com,https://hackerone.com) 98 | stdin/stdout subprober supports both stdin/stdout and enables -nc to pipe the output of subprober 99 | 100 | [PROBES]: 101 | 102 | -sc, --status-code display the status code of the host 103 | -tl, --title display the title of host 104 | -sv, --server display the server name of the host 105 | -wc, --word-count display the HTTP response word count 106 | -lc, --line-count display the HTTP response line count 107 | -cl, --content-length display the HTTP response content length 108 | -l , --location display the redirected location of the host 109 | -apt, --application-type display the content type of the host 110 | -ip, --ipaddress display the IPs of the host 111 | -cn, --cname display the CNAMEs of the host 112 | -aaa, --aaa-records display the AAAA records of the host 113 | -htv, --http-version display the server supported HTTP version of the host 114 | -hrs, --http-reason display the reason for HTTP connection of the host 115 | -jarm, --jarm-fingerprint display the JARM fingerprint hash of the host 116 | -rpt, --response-time display the response time for the successful request 117 | -wss, --websocket display the server supports websockets 118 | -hash, --hash display response body in hash format (supported hashes: md5, mmh3, simhash, sha1, sha256, sha512) 119 | -dmt, --display-method display the method of the HTTP request 120 | -bp, --body-preview display the HTTP response body in first n number of characters (default: 100) 121 | 122 | [CONFIG]: 123 | 124 | -dhp, --disable-http-probe disables subprober from probing HTTP protocols and only for HTTPS when no protocol is specified 125 | -X, --method request methods to probe and get response (supported: get, post, head, put, delete, patch, trace, connect, options) (default: get) 126 | -H, --header add custom headers for probing and -H can be used multiple times to pass multiple header values (ex: -H application/json -H X-Forwarded-Host: 127.0.0.1) 127 | -ra, --random-agent enable Random User-Agent to use for probing and applies same to screenshots. (default: subprober/Alpha) 128 | -px, --proxy specify a proxy to send the requests through it (ex: http://127.0.0.1:8080) 129 | -ar, --allow-redirect enable following redirections 130 | -maxr, --max-redirection set max value to follow redirections (default: 10) 131 | -http2, --http2 enable to request with HTTP/2 support (default: Http/1.1) 132 | -sni, --sni-hostname set custom TLS SNI host name for requests. 133 | 134 | [MISCELLANEOUS]: 135 | 136 | -p, --path specify a path or text file of paths for probing and getting results (example: -p admin.php or -p paths.txt) 137 | -pt, --port set custom port for making HTTP request and default ports are 80,443 based on the url scheme 138 | -tls, --tls grabs the TLS data for the requested host 139 | 140 | [HEADLESS]: 141 | 142 | -ss, --screenshot enable to take screenshots of the page using headless browsers with asynchronous performance 143 | -st, --screenshot-timeout set a timeout value for taking screenshots (default: 15) 144 | -scp, --system-chrome-path specify the executable path of the chromedriver to use system chrome to take screenshots 145 | -pdf, --save-pdf enable to save the screenshot image in the pdf format (default: png) 146 | -HH , --screenshot-headers add custom headers for authenticated screenshots 147 | -icb, --include-bytes enable to include the screenshot bytes in output when json output enabled 148 | -hos, --headless-options set additional chrome headless browser options and supports comma-separated values (-ho "--start-maximized") 149 | -sid, --screenshot-idle set custom idle time in seconds before taking screenshots (default: 1) 150 | -sp, --screenshot-path specify a directory path to store screenshot results (default: currentdir/screenshots) 151 | 152 | [MATCHERS]: 153 | 154 | -mc, --match-code match http response by specified status codes and supports comma-separated values (-mc 200,302) 155 | -mcr, --match-code-range match http response by specified status code range and supports single value (-mcr 200-299) 156 | -ms, --match-string match http response containing the specified string and supports comma-separated values (-ms admin,login) 157 | -mr, --match-regex match http response matching the specified regex and supports comma-separated values (-mr .*admin.*,.*login.*) 158 | -mpt, --match-path match http response by URL path and supports comma-separated values (-mpt /admin/wp-ajax.php,/wp-json) 159 | -ml, --match-length match http response by specified response length and supports comma-separated values (-ml 1024,2048) 160 | -mlc, --match-line-count match http response by specified response line count and supports comma-separated values (-mlc 10,50) 161 | -mwc, --match-word-count match http response by specified word count and supports comma-separated values (-mwc 100,500) 162 | -mrt, --match-response-time match http response exceeding the specified minimum response time in seconds (-mrt 2.30) 163 | 164 | [FILTERS]: 165 | 166 | -fc, --filter-code filter http response by specified status codes and supports comma-separated values (-fc 404,500) 167 | -fcr, --filter-code-range filter http response by specified status code range and supports single value (-fcr 400-499) 168 | -fs, --filter-string filter http response containing the specified string and supports comma-separated values (-fs error,not found) 169 | -fr, --filter-regex filter http response matching the specified regex and supports comma-separated values (-fr .*admin.*,.*login.*) 170 | -fpt, --filter-path filter http response by URL path and supports comma-separated values (-fpt /error,404.html) 171 | -fl, --filter-length filter http response by specified response length and supports comma-separated values (-fl 1024,2048) 172 | -flc, --filter-line-count filter http response by specified response line count and supports comma-separated values (-flc 10,50) 173 | -fwc, --filter-word-count filter http response by specified response word count and supports comma-separated values (-fwc 100,500) 174 | -frt, --filter-response-time filter http response exceeding the specified maximum response time in seconds (-frt 2.30) 175 | 176 | [OUTPUT]: 177 | 178 | -o, --output define the output filename to store the results of the probing operation. 179 | -das, --disable-auto-save disable the auto-save of results when no output file is specified. 180 | -J, --json store and display output in JSON format (includes only data from enabled options). 181 | -rdu, --redirect-urls display the redirect URLs in the output (requires -J and -ar to enabled to enabled). 182 | -rdh, --redirect-history display the full redirect history (requires -J and -ar to enabled). 183 | -rsc, --redirect-status-codes display the status codes for redirections (requires -J and -ar to enabled). 184 | -rqh, --request-headers include request headers in the output (requires -J and -ar to enabled). 185 | -rsh, --response-headers include response headers in the output (requires -J and -ar to enabled). 186 | -fo, --full-output include all available data in the output (requires -J to enabled and doesn't overrides websocket,jarm,hashes options). 187 | 188 | [RATE-LIMIT]: 189 | 190 | -c, --concurrency set the concurrency level for sending http requests (default: 100) 191 | -rtl, --rate-limit set a rate limit for sending a maximum number of requests per second (default: 1000) 192 | -sct, --screenshot-threads set a threads level for taking screenshots (default: 40) 193 | 194 | [Optimization]: 195 | 196 | -to, --timeout set a custom timeout value for sending requests. 197 | -d, --delay set a delay in seconds before sending each request (default: 0.5) 198 | -rts, --retries set a number of retries if a request fails to connect (default: 0) 199 | 200 | [UPDATES]: 201 | 202 | -up, --update update subprober to the latest version (pip required to be installed) 203 | -sup, --show-updates display the current or latest version of subprober updates 204 | 205 | [DEBUG]: 206 | 207 | -h, --help display this help message and exit! 208 | -s, --silent enable silent mode to suppress the display of Subprober banner and version information. 209 | -v, --verbose enable verbose mode to display error results on the console. 210 | -nc, --no-color enable to display the output without any CLI colors 211 | ``` 212 | 213 | 214 | ### About: 215 | 216 | The **SubProber** is a cutting-edge tool developed by **RevoltSecurities** to empower Security Researchers and Penetration Testers. Designed with efficiency and precision in mind, SubProber streamlines reconnaissance and enhances vulnerability detection. Released under the MIT License, it reflects our commitment to fostering innovation and collaboration within the open-source community. 217 | 218 | At **RevoltSecurities**, we aim to support researchers by providing advanced automation tools that simplify complex tasks, enabling professionals to focus on securing modern infrastructures. 219 | 220 | -------------------------------------------------------------------------------- /subprober/modules/core/core.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import httpx 3 | import warnings 4 | import argparse 5 | import aiodns 6 | from asynciolimiter import Limiter 7 | from datetime import datetime 8 | import json 9 | from httpx import AsyncHTTPTransport 10 | from alive_progress import alive_bar 11 | from bs4 import XMLParsedAsHTMLWarning, MarkupResemblesLocatorWarning, BeautifulSoup, FeatureNotFound 12 | from subprober.modules.logger.logger import logger, stdinlog, bold,white,green,blue,cyan,magenta,yellow,red,reset,random_color 13 | from subprober.modules.websocket.websocket import AsyncWebsocket 14 | from subprober.modules.dns.dns import dns 15 | from subprober.modules.jarmhash.jarmhash import jarmhash 16 | from subprober.modules.hash.hash import Hashgen 17 | from subprober.modules.utils.utils import Useragents, chunker,extract_cookies 18 | from subprober.modules.tls.tls import tlsinfo 19 | from subprober.modules.filters.filters import * 20 | from subprober.modules.matchers.matchers import * 21 | from subprober.modules.screenshot.screenshot import Headless 22 | from subprober.modules.save.save import save 23 | 24 | class Subprober: 25 | def __init__( 26 | self, 27 | urls: list[str], 28 | args: argparse.ArgumentParser.parse_args, 29 | nameservers=["8.8.8.8", "1.1.1.1"], 30 | hashes = [], 31 | mc = None, 32 | fc = None, 33 | mcr = None, 34 | fcr = None, 35 | mpt = None, 36 | fpt = None, 37 | ms = None, 38 | fs = None, 39 | mrg = None, 40 | frg = None, 41 | mrt = None, 42 | frt = None, 43 | mlc = None, 44 | flc = None, 45 | mwc = None, 46 | fwc = None, 47 | ml = None, 48 | fl = None, 49 | ) -> None: 50 | 51 | self.urls = urls 52 | self.args = args 53 | self.semaphore = asyncio.Semaphore(self.args.concurrency) 54 | self.scsem = asyncio.Semaphore(self.args.screenshot_threads) 55 | self.nameservers = nameservers 56 | self.loop = asyncio.get_event_loop() 57 | self.resolver = aiodns.DNSResolver(nameservers=self.nameservers, rotate=True, loop=self.loop) 58 | self.rate_limit = Limiter(rate=self.args.rate_limit/1) 59 | self.hashes = hashes 60 | self.mc = mc 61 | self.fc = fc 62 | self.mcr = mcr 63 | self.fcr = fcr 64 | self.mpt = mpt 65 | self.fpt = fpt 66 | self.ms = ms 67 | self.fs = fs 68 | self.mrg = mrg 69 | self.frg = frg 70 | self.mrt = mrt 71 | self.frt = frt 72 | self.mlc = mlc 73 | self.flc = flc 74 | self.mwc = mwc 75 | self.fwc = fwc 76 | self.ml = ml 77 | self.fl = fl 78 | self.screenshots = None 79 | 80 | async def request(self, method: str, url: str, client: httpx.AsyncClient) -> httpx.Response: 81 | try: 82 | await self.rate_limit.wait() 83 | headers = {} 84 | if self.args.header: 85 | for header in self.args.header: 86 | name, value = header.split(':', 1) 87 | headers[name.strip()] = value.strip() 88 | headers["User-Agent"] = Useragents() if self.args.random_agent else "git+Subprober/V2.XD" 89 | if self.args.sni_hostname: 90 | extensions = {"sni_hostname": f"{self.args.sni_hostname}"} 91 | else: 92 | extensions = None 93 | response = await client.request(method.upper(), url,headers=headers,follow_redirects=self.args.allow_redirect, extensions=extensions) 94 | return response 95 | except (httpx.ConnectError, httpx.ConnectTimeout): 96 | pass 97 | except (KeyboardInterrupt, asyncio.CancelledError): 98 | exit(1) 99 | except Exception as e: 100 | if self.args.verbose: 101 | logger(f"Exception occured in the subprober request module due to: {e}, {type(e)}, {url}", "warn", self.args.no_color) 102 | return None 103 | 104 | async def responsed(self, response: httpx.Response, url: str) -> dict: 105 | try: 106 | network_streams = response.extensions.get("network_stream") 107 | server_addr = network_streams.get_extra_info("server_addr") if network_streams else None 108 | results = {} 109 | results["Title"] = "" 110 | results["Timestamp"] = datetime.now().strftime("%Y-%m-%dT%H:%M:%S") 111 | results["Url"] = str(url) 112 | results["FinalUrl"] = str(response.url) 113 | results["IsRedirect"] = response.has_redirect_location 114 | 115 | if self.args.full_output or self.args.redirect_history and self.args.json and self.args.allow_redirect: 116 | results["RedirectHistory"] = [{"url": str(r.url), "status_code": r.status_code} for r in response.history] 117 | 118 | if self.args.full_output or self.args.redirect_urls and self.args.json and self.args.allow_redirect: 119 | results["RedirectedUrls"] = [str(redirect.url) for redirect in response.history] if response.history else [] 120 | 121 | if self.args.full_output or self.args.redirect_status_codes and self.args.json and self.args.allow_redirect: 122 | results["RedirectedStatusCode"] = [int(redirect.status_code) for redirect in response.history] if response.history else [] 123 | 124 | results["HttpVersion"] = response.http_version 125 | results["ResponseReason"] = response.reason_phrase 126 | results["StatusCode"] = response.status_code 127 | results["ResponseTime"] = response.elapsed.total_seconds() 128 | 129 | results["Method"] = response.request.method 130 | results["Host"] = str(response.request.url.host) if response.request.url.host else "" 131 | results["Port"] = str(response.request.url.port) if response.request.url.port else "" 132 | results["ServerAddress"] = str([server_addr[0]]) if server_addr is not None else "" 133 | results["ServerPort"] = str([server_addr[1]]) if server_addr is not None else "" 134 | 135 | if self.args.full_output or self.args.request_headers and self.args.json: 136 | results["RequestHeaders"] = dict(response.request.headers) if response.request.headers else {} 137 | 138 | if self.args.full_output or self.args.response_headers and self.args.json: 139 | results["ResponseHeaders"] = dict(response.headers) if response.headers else {} 140 | 141 | results["Cookies"] = extract_cookies(response.cookies.jar) if response.cookies.jar else [] 142 | results["Length"] = len(response.text) 143 | results["LineCount"] = len(response.text.splitlines()) 144 | results["WordCount"] = len(response.text.split()) 145 | results["BodyPreview"] = response.text[:100] if response.text else "" 146 | 147 | try: 148 | with warnings.catch_warnings(): 149 | warnings.filterwarnings("ignore", category=UserWarning) 150 | warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning) 151 | warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning) 152 | soup = BeautifulSoup(response.text, "lxml") 153 | title_tag = soup.title 154 | title = title_tag.string if title_tag else "" 155 | results["Title"] = title 156 | except FeatureNotFound: 157 | if self.args.verbose: 158 | logger(f"Looks like your beautifulsoup4, lxml, bs4 not in latest version, please update it", "warn", self.args.no_color) 159 | results["Title"] = "" 160 | 161 | if self.args.hash: 162 | hashes= await Hashgen(response.text, self.hashes, self.args) 163 | results["Hash"] = hashes if hashes else {} 164 | 165 | if self.args.jarm_fingerprint: 166 | jarmhashes = await jarmhash(url, self.args) 167 | results["JarmHash"] = jarmhashes 168 | 169 | if self.args.full_output or self.args.cname: 170 | cname = await dns(self.resolver, url, "CNAME", self.args) 171 | results["Cname"] = cname 172 | 173 | if self.args.full_output or self.args.ipaddress: 174 | ips = await dns(self.resolver, url, "A", self.args) 175 | results["A"] = ips 176 | 177 | if self.args.full_output or self.args.aaa_records: 178 | aaaa = await dns(self.resolver, url, "AAAA", self.args) 179 | results["AAAA"] = aaaa 180 | 181 | results["Server"] = response.headers.get("server", "") 182 | 183 | content_type = response.headers.get("Content-Type", "") 184 | if content_type: 185 | content_type = content_type.split(";")[0].strip() 186 | results["ContentType"] = content_type 187 | else: 188 | results["ContentType"] = "" 189 | 190 | 191 | if self.args.websocket: 192 | websocket = await AsyncWebsocket(url, self.args) 193 | results["Websockets"] = websocket 194 | 195 | 196 | if self.args.full_output or self.args.tls and self.args.json: 197 | tls = await tlsinfo(self.args, network_streams) 198 | results["TLS"] = tls 199 | 200 | if await match_by_code(response, self.mc) and \ 201 | await match_code_range(response, self.mcr) and\ 202 | await match_url_path_contains(response, self.mpt)and\ 203 | await match_word_body(response, self.ms) and \ 204 | await match_by_regex(response,self.mrg)and \ 205 | await match_response_time(response, self.mrt)and \ 206 | await filter_by_code(response, self.fc) and \ 207 | await filter_code_range(response, self.fcr) and \ 208 | await filter_url_path_contains(response, self.fpt) and \ 209 | await filter_word_body(response, self.fs) and \ 210 | await filter_by_regex(response, self.frg) and \ 211 | await filter_response_time(response, self.frt) and \ 212 | await match_by_ints(response.status_code, self.ml) and \ 213 | await filter_by_ints(response.status_code, self.fl) and \ 214 | await match_by_ints(response.status_code, self.mlc) and \ 215 | await filter_by_ints(response.status_code, self.flc) and \ 216 | await match_by_ints(response.status_code, self.mwc) and \ 217 | await filter_by_ints(response.status_code, self.fwc): 218 | 219 | if self.args.screenshot: 220 | await self.scsem.acquire() 221 | await self.screenshots.run(results["Url"],results=results) 222 | else: 223 | results = None 224 | 225 | except (KeyboardInterrupt, asyncio.CancelledError): 226 | logger("CTRL+C Pressed!", "debug", self.args.no_color) 227 | exit(1) 228 | except Exception as e: 229 | if self.args.verbose: 230 | logger(f"Exception occured in the response handler module due to: {e}, {type(e)}, {url}", "warn", self.args.no_color) 231 | finally: 232 | if self.args.screenshot: 233 | self.scsem.release() 234 | return results 235 | 236 | 237 | async def manage(self, url, method, bar, session: httpx.AsyncClient) -> None: 238 | try: 239 | self.semaphore.release() 240 | await asyncio.sleep(self.args.delay) 241 | response = await self.request(method, url,session) 242 | 243 | if response is None: 244 | return 245 | results = await self.responsed(response, url) 246 | 247 | if results is None: 248 | return 249 | 250 | if self.args.json: 251 | stdinlog(json.dumps(results, ensure_ascii=False)) 252 | 253 | if self.args.output: 254 | await save(self.args.output, results, self.args.json, self.args.no_color) 255 | 256 | else: 257 | 258 | if not self.args.no_color: 259 | Url = f"{bold}{white}{results["Url"]}" 260 | 261 | sc = results["StatusCode"] 262 | statuscode = sc if self.args.status_code else "" 263 | 264 | if self.args.status_code: 265 | if statuscode >=200 and statuscode <=299: 266 | StatusCode = f"{bold}{white}[{reset}{bold}{bold}{green}{statuscode}{reset}{bold}{white}]{reset}" 267 | elif statuscode >=300 and statuscode <=399: 268 | StatusCode = f"{bold}{white}[{reset}{bold}{bold}{yellow}{statuscode}{reset}{bold}{white}]{reset}" 269 | else: 270 | StatusCode = f"{bold}{white}[{reset}{bold}{red}{statuscode}{reset}{bold}{white}]{reset}" 271 | else: 272 | StatusCode = "" 273 | 274 | Jarm = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{magenta}{results["JarmHash"]}{reset}{bold}{white}]{reset}" if self.args.jarm_fingerprint else "" 275 | Title = f"{bold}{white}[{reset}{bold}{cyan}{results["Title"]}{reset}{bold}{white}]{reset}" if self.args.title else "" 276 | Server = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{magenta}{results["Server"]}{reset}{bold}{white}]{reset}" if self.args.server else "" 277 | Wc = f"{bold}{white}[{reset}{bold}{green}{results["WordCount"]}{reset}{bold}{white}]{reset}" if self.args.word_count else "" 278 | Lc = f"{bold}{white}[{reset}{bold}{red}{results["LineCount"]}{reset}{bold}{white}]{reset}" if self.args.line_count else "" 279 | Lt = f"{bold}{white}[{reset}{bold}{green}{results["Length"]}{reset}{bold}{white}]{reset}" if self.args.content_length else "" 280 | Lo = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{magenta}{results["FinalUrl"]}{reset}{bold}{white}]{reset}" if self.args.location and self.args.allow_redirect else "" 281 | Apt = f"{bold}{white}[{reset}{bold}{yellow}{results["ContentType"]}{reset}{bold}{white}]{reset}" if self.args.application_type else "" 282 | A = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{yellow}{",".join(map(str, results["A"]))}{reset}{bold}{white}]{reset}" if self.args.ipaddress else "" 283 | Cn = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{green}{",".join(map(str, results["Cname"]))}{reset}{bold}{white}]{reset}" if self.args.cname else "" 284 | AAA = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{cyan}{",".join(map(str, results["AAAA"]))}{reset}{bold}{white}]{reset}" if self.args.aaa_records else "" 285 | Htv = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{blue}{results["HttpVersion"]}{reset}{bold}{white}]{reset}" if self.args.http_version else "" 286 | Htr = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{magenta}{results["ResponseReason"]}{reset}{bold}{white}]{reset}" if self.args.http_reason else "" 287 | Rpt = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{random_color}{results["ResponseTime"]}{reset}{bold}{white}]{reset}" if self.args.response_time else "" 288 | Wss = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{random_color}websocket: {results["Websockets"]}{reset}{bold}{white}]{reset}" if self.args.websocket else "" 289 | Hsh = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{random_color}{",".join(map(str, results["Hash"].values()))}{reset}{bold}{white}]{reset}" if self.args.hash else "" 290 | Dmt = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{random_color}{results["Method"]}{reset}{bold}{white}]{reset}" if self.args.display_method else "" 291 | Bpv = f"{bold}{white}[{reset}{bold}{white}{reset}{bold}{random_color}{results["BodyPreview"]}{reset}{bold}{white}]{reset}" if self.args.body_preview else "" 292 | else: 293 | Url = f"{results['Url']}" 294 | StatusCode = f"[{results["StatusCode"]}]" if self.args.status_code else "" 295 | Jarm = f"[{results['JarmHash']}]" if self.args.jarm_fingerprint else "" 296 | Title = f"[{results['Title']}]" if self.args.title else "" 297 | Server = f"[{results['Server']}]" if self.args.server else "" 298 | 299 | Wc = f"[{results['WordCount']}]" if self.args.word_count else "" 300 | Lc = f"[{results['LineCount']}]" if self.args.line_count else "" 301 | Lt = f"[{results['Length']}]" if self.args.content_length else "" 302 | Lo = f"[{results['FinalUrl']}]" if self.args.location and self.args.allow_redirect else "" 303 | Apt = f"[{results['ContentType']}]" if self.args.application_type else "" 304 | A = f"[{','.join(map(str, results['A']))}]" if self.args.ipaddress else "" 305 | Cn = f"[{','.join(map(str, results['Cname']))}]" if self.args.cname else "" 306 | AAA = f"[{','.join(map(str, results['AAAA']))}]" if self.args.aaa_records else "" 307 | Htv = f"[{results['HttpVersion']}]" if self.args.http_version else "" 308 | Htr = f"[{results['ResponseReason']}]" if self.args.http_reason else "" 309 | Rpt = f"[{results['ResponseTime']}]" if self.args.response_time else "" 310 | Wss = f"[websocket: {results['Websockets']}]" if self.args.websocket else "" 311 | Hsh = f"[{','.join(map(str, results['Hash'].values()))}]" if self.args.hash else "" 312 | Dmt = f"[{results['Method']}]" if self.args.display_method else "" 313 | Bpv = f"[{results['BodyPreview']}]" if self.args.body_preview else "" 314 | 315 | output = f"{Url} {StatusCode}{Jarm}{Title}{Server}{Apt}{Wc}{Lc}{Lt}{Lo}{A}{Cn}{AAA}{Htv}{Htr}{Rpt}{Wss}{Hsh}{Dmt}{Bpv}" 316 | stdinlog(output) 317 | if self.args.output: 318 | await save(self.args.output, output, self.args.json, self.args.no_color) 319 | 320 | except (KeyboardInterrupt, asyncio.CancelledError): 321 | logger("CTRL+C Pressed!", "debug", self.args.no_color) 322 | exit(1) 323 | except Exception as e: 324 | logger(f"Exception occured in subprober core manager module due to: {e}, {type(e)}", "warn", self.args.no_color) 325 | finally: 326 | bar() 327 | 328 | async def start(self) -> None: 329 | try: 330 | 331 | if self.args.screenshot: 332 | self.screenshots = Headless(self.args) 333 | self.screenshots.setup() 334 | 335 | if self.args.retries: 336 | transport = AsyncHTTPTransport(retries=self.args.retries) 337 | else: 338 | transport = None 339 | 340 | timeout = httpx.Timeout(connect=self.args.timeout, pool=self.args.concurrency*2, write=None, read=80.0) 341 | limits = httpx.Limits(max_connections=self.args.concurrency*2, max_keepalive_connections=self.args.concurrency*2) 342 | 343 | async with httpx.AsyncClient(verify=False, limits=limits, http2=self.args.http2, max_redirects=self.args.max_redirection, proxy=self.args.proxy, transport=transport, timeout=timeout) as session: 344 | with alive_bar(title="SubProber", total=len(self.urls), enrich_print=False) as bar: 345 | for chunk in chunker(self.urls): 346 | tasks = [] 347 | for url in chunk: 348 | await self.semaphore.acquire() 349 | task = asyncio.create_task(self.manage(url, self.args.method,bar,session)) 350 | tasks.append(task) 351 | await asyncio.gather(*tasks, return_exceptions=True) 352 | except (KeyboardInterrupt, asyncio.CancelledError): 353 | logger("CTRL+C Pressed!", "debug", self.args.no_color) 354 | exit(1) 355 | except Exception as e: 356 | if self.args.verbose: 357 | logger(f"Exception occured in the subprober start module due to: {e}, {type(e)}", "warn", self.args.no_color) --------------------------------------------------------------------------------