├── README.md ├── build.sh ├── config ├── Config.yml ├── pc_ua.json ├── phantomjs_liunx_x64 └── phantomjs_windows.exe ├── imgs ├── five.png ├── four.png ├── help.png ├── logo.jpg ├── one.png ├── threes.png └── two.png ├── lib ├── Controller.py ├── WebpackFind.py ├── __init__.py ├── common │ ├── Printlog.py │ ├── __init__.py │ ├── cmdline.py │ ├── downloadjs.py │ ├── fileoperation.py │ ├── regular.py │ ├── toupdate.py │ ├── urlrequest.py │ └── utils.py └── vuln │ ├── __init__.py │ └── unauthorized.py ├── main.py ├── requirements.txt └── version.txt /README.md: -------------------------------------------------------------------------------- 1 | # Webpackfind-信息收集工具 2 | 3 | 郑重声明：文中所涉及的技术、思路和工具仅供以安全为目的的学习交流使用，任何人不得将其用于非法用途以及盈利等目的，否则后果自行承担。 4 | 5 | ## 0x01 介绍 6 | 7 | 作者：[小洲](https://github.com/xz-zone) 8 | 9 | 团队：[横戈安全团队](imgs/logo.jpg)，未来一段时间将陆续开源工具，欢迎关注微信公众号： 10 | 11 | ![logo](imgs/logo.jpg) 12 | 13 | 定位：协助红队人员快速的信息收集，测绘目标资产，寻找薄弱点。 14 | 15 | 语言：python3开发 16 | 17 | 功能：一条龙服务，只需要输入根域名即可一键化收集敏感信息。具体案例见下文。 18 | 19 | 调用： 20 | * 脚本借用了JSFinderjs内容提取脚本，感谢Threezh1作者。 21 | * 脚本借用了HaE内容提取脚本，感谢gh0stkey作者。 22 | 23 | 支持环境：Ubuntu、Centos、Windows。 24 | 25 | 26 | ## 0x02 安装 27 | 28 | 为了避免踩坑,建议安装在如下环境中 29 | 30 | * 当前用户对该目录有写权限，不然扫描结果无法生成。root权限即可 31 | * Python环境必须是3.7以上，因为使用了异步。建议VPS环境是ubuntu20，默认是python3.8。安装模块的时候切记不要走豆瓣的源 32 | 33 | ``` 34 | Ubuntu 运行环境 35 | chmod 777 build.sh 36 | ./build.sh 37 | 38 | Windows模块安装 39 | python3 -m pip install -r requirements.txt 40 | ``` 41 | ![threes](imgs/threes.png) 42 | 43 | `python3 webpackfind.py` 44 | 45 | ![help](imgs/help.png) 46 | 47 | ## 0x03 效果展示 48 | 49 | 自动爬取网站 50 | 51 | ![one](imgs/one.png) 52 | 53 | 自动爬取本地文件 54 | 55 | ![two](imgs/two.png) 56 | 57 | 读取txt循环读取url 58 | 59 | ![five](imgs/five.png) 60 | 61 | ## 0x04 使用方法 62 | 63 | | 语法 | 功能 | 64 | | :------------------------------------------------------- | :-------------------------------------------- | 65 | | python3 webpackfind.py -u http://domain.com | 自动化遍历URL中里面js | 66 | | python3 webpackfind.py -j ./js/test.com/ | 遍历本地目录中的js文件并且格式化js文件、提取信息泄露 | 67 | | python3 webpackfind.py -f ./url.txt | 读取本地url文件批量遍历url中js | 68 | | python3 webpackfind.py -c "name=test;" -u http://domain.com | 设置自定义Cookie，场景:需要登录才能爬取js信息 | 69 | | python3 webpackfind.py -d "Authorization: bearer 123456\nContent-Type: application/json\n" -u http://domain.com | 设置自定义headers | 70 | | python3 webpackfind.py -update 1 | 检查本地`webpackfind.py`是否与github.com是否一致 | 71 | 72 | ## 0x05 版本更新 73 | 74 | 2021-08-17 初始版本提交 75 | 76 | 2021-08-18 版本整体修改为面向对象编程，自动识别是否存在其他子域名 77 | ![four](imgs/four.png) 78 | 79 | 2021-08-19 修复build.sh 不能执行问题 80 | 81 | 2021-08-23 修复了manifest js无法读取遍历BUG，感谢 chhyx2 师傅提交的bug。 82 | 83 | 2021-08-24 优化细节,解决windows生成目录问题。 84 | 85 | 2021-08-26 优化细节,解决获取js路径问题,添加读取文本url地址进行扫描js。 86 | 87 | 2021-09-02 优化细节,解决PhantomJS给拦截问题。 88 | 89 | 2021-09-03 优化细节,解决提取域名正则问题。 90 | 91 | 2021-09-07 优化细节,解决目录名称更好查找，添加自动判断是否更新。 92 | 93 | 2021-09-22 优化细节,解决首页存在webpackjs代码问题。 94 | 95 | 2022-07-21 优化细节,解决本地js代码文件未格式化问题,优化加更新参数才检查更新。 96 | 97 | 2022-08-22 重构核心代码,兼容更多webpack版本问题。 98 | 99 | 2022-08-23 优化细节,优化批量扫描,解决写入路径问题,增加Cookies认证遍历读取。 100 | 101 | 2022-08-24 优化phantomjs文件过大问题,优化提示文字,增加协议识别模块。 102 | 103 | 2022-08-25 优化细节，增加批量扫描单线程改多线程。 104 | 105 | 2022-08-27 优化线程池问题,增加“HaE”规则库。 106 | 107 | 2022-08-30 增加进度条，自动化识别路径Url。 108 | 109 | 2022-08-31 优化线程池问题,压缩运行时间。 110 | 111 | 2022-09-01 优化整体压缩美化代码。 112 | 113 | 2022-09-06 优化匹配正则，输出文件。 114 | 115 | 2022-09-09 增加自定义herder头部。 116 | 117 | 2022-09-19 优化匹配正则。 118 | 119 | 2022-09-23 优化细节。 120 | 121 | 2022-09-28 优化细节。 122 | 123 | 2022-11-28 优化细节，匹配更多api接口。 124 | 125 | ## 0x06 反馈 126 | 127 | Webpackfind 是一个免费且开源的项目，我们欢迎任何人为其开发和进步贡献力量。 128 | 129 | * 在使用过程中出现任何问题，可以通过 issues 来反馈。 130 | * Bug 的修复可以直接提交 Pull Request 到 dev 分支。 131 | * 如果是增加新的功能特性，请先创建一个 issue 并做简单描述以及大致的实现方法，提议被采纳后，就可以创建一个实现新特性的 Pull Request。 132 | * 欢迎对说明文档做出改善，帮助更多的人使用 Webpackfind。 133 | * 贡献代码请提交 PR 至 dev 分支，master 分支仅用于发布稳定可用版本。 134 | 135 | *提醒：和项目相关的问题最好在 issues 中反馈，这样方便其他有类似问题的人可以快速查找解决方法，并且也避免了我们重复回答一些问题。* 136 | 137 | ## Stargazers over time 138 | 139 | [![Stargazers over time](https://starchart.cc/xz-zone/Webpackfind.svg)](https://starchart.cc/xz-zone/Webpackfind) 140 | 141 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | apt install libfontconfig1 3 | apt install python3 -y 4 | apt install python3-pip --fix-missing -y 5 | apt install python3-setuptools -y 6 | python3 -m pip install --upgrade pip 7 | python3 -m pip install -r requirements.txt -------------------------------------------------------------------------------- /config/Config.yml: -------------------------------------------------------------------------------- 1 | rules: 2 | - rule: 3 | - color: green 4 | engine: dfa 5 | loaded: true 6 | name: Shiro 7 | regex: (=deleteMe|rememberMe=) 8 | scope: any 9 | sensitive: true 10 | - color: green 11 | engine: dfa 12 | loaded: true 13 | name: JSON Web Token 14 | regex: (eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9._-]{10,}|eyJ[A-Za-z0-9_\/+-]{10,}\.[A-Za-z0-9._\/+-]{10,}) 15 | scope: any 16 | sensitive: true 17 | - color: green 18 | engine: dfa 19 | loaded: true 20 | name: Swagger UI 21 | regex: ((swagger-ui.html)|(\"swagger\":)|(Swagger UI)|(swaggerUi)) 22 | scope: response body 23 | sensitive: true 24 | - color: green 25 | engine: dfa 26 | loaded: true 27 | name: Ueditor 28 | regex: (ueditor\.(config|all)\.js) 29 | scope: response body 30 | sensitive: true 31 | type: Fingerprint 32 | - rule: 33 | - color: yellow 34 | engine: nfa 35 | loaded: true 36 | name: Email 37 | regex: (([a-z0-9][_|\.])*[a-z0-9]+@([a-z0-9][-|_|\.])*[a-z0-9]+\.((?!js|css|jpg|jpeg|png|ico)[a-z]{2,})) 38 | scope: response 39 | sensitive: false 40 | - color: orange 41 | engine: nfa 42 | loaded: true 43 | name: Chinese IDCard 44 | regex: '[^0-9]((\d{8}(0\d|10|11|12)([0-2]\d|30|31)\d{3}$)|(\d{6}(18|19|20)\d{2}(0[1-9]|10|11|12)([0-2]\d|30|31)\d{3}(\d|X|x)))[^0-9]' 45 | scope: response body 46 | sensitive: true 47 | - color: orange 48 | engine: nfa 49 | loaded: true 50 | name: Chinese Mobile Number 51 | regex: '[^\w]((?:(?:\+|00)86)?1(?:(?:3[\d])|(?:4[5-79])|(?:5[0-35-9])|(?:6[5-7])|(?:7[0-8])|(?:8[\d])|(?:9[189]))\d{8})[^\w]' 52 | scope: response body 53 | sensitive: true 54 | - color: cyan 55 | engine: nfa 56 | loaded: true 57 | name: Internal IP Address 58 | regex: '[^0-9]((127\.0\.0\.1)|(10\.\d{1,3}\.\d{1,3}\.\d{1,3})|(172\.((1[6-9])|(2\d)|(3[01]))\.\d{1,3}\.\d{1,3})|(192\.168\.\d{1,3}\.\d{1,3}))' 59 | scope: response 60 | sensitive: true 61 | - color: green 62 | engine: nfa 63 | loaded: true 64 | name: MAC Address 65 | regex: (^([a-fA-F0-9]{2}(:[a-fA-F0-9]{2}){5})|[^a-zA-Z0-9]([a-fA-F0-9]{2}(:[a-fA-F0-9]{2}){5})) 66 | scope: response 67 | sensitive: true 68 | - color: orange 69 | engine: nfa 70 | loaded: false 71 | name: Chinese Bank Card ID 72 | regex: '[^0-9]([1-9]\d{12,18})[^0-9]' 73 | scope: response 74 | sensitive: true 75 | type: Basic Information 76 | - rule: 77 | - color: cyan 78 | engine: dfa 79 | loaded: true 80 | name: RCE Paramters 81 | regex: ((cmd=)|(exec=)|(command=)|(execute=)|(ping=)|(query=)|(jump=)|(code=)|(reg=)|(do=)|(func=)|(arg=)|(option=)|(load=)|(process=)|(step=)|(read=)|(function=)|(feature=)|(exe=)|(module=)|(payload=)|(run=)|(daemon=)|(upload=)|(dir=)|(download=)|(log=)|(ip=)|(cli=)) 82 | scope: request 83 | sensitive: true 84 | - color: yellow 85 | engine: dfa 86 | loaded: true 87 | name: Java Deserialization 88 | regex: (javax\.faces\.ViewState) 89 | scope: response 90 | sensitive: true 91 | - color: cyan 92 | engine: dfa 93 | loaded: true 94 | name: Debug Logic Parameters 95 | regex: ((access=)|(adm=)|(admin=)|(alter=)|(cfg=)|(clone=)|(config=)|(create=)|(dbg=)|(debug=)|(delete=)|(disable=)|(edit=)|(enable=)|(exec=)|(execute=)|(grant=)|(load=)|(make=)|(modify=)|(rename=)|(reset=)|(root=)|(shell=)|(test=)|(toggl=)) 96 | scope: request 97 | sensitive: true 98 | - color: cyan 99 | engine: nfa 100 | loaded: true 101 | name: URL As A Value 102 | regex: (=(https?://.*|https?%3(a|A)%2(f|F)%2(f|F).*)) 103 | scope: request 104 | sensitive: true 105 | - color: yellow 106 | engine: dfa 107 | loaded: true 108 | name: Upload Form 109 | regex: (type=\"file\") 110 | scope: response body 111 | sensitive: true 112 | - color: cyan 113 | engine: nfa 114 | loaded: true 115 | name: DoS Paramters 116 | regex: ((size=)|(page=)|(num=)) 117 | scope: request 118 | sensitive: false 119 | - color: yellow 120 | engine: nfa 121 | loaded: true 122 | name: JSONP Response 123 | regex: (^(\w.*?)\() 124 | scope: response body 125 | sensitive: false 126 | type: Maybe Vulnerability 127 | - rule: 128 | - color: yellow 129 | engine: dfa 130 | loaded: true 131 | name: OSS 132 | regex: ([A|a]ccess[K|k]ey[I|i][d|D]|[A|a]ccess[K|k]ey[S|s]ecret) 133 | scope: any 134 | sensitive: true 135 | - color: green 136 | engine: nfa 137 | loaded: true 138 | name: Amazon AWS URL 139 | regex: (((([a-zA-Z0-9._-]+\.s3|s3)(\.|\-)+[a-zA-Z0-9._-]+|[a-zA-Z0-9._-]+\.s3|s3)\.amazonaws\.com)|(s3:\/\/[a-zA-Z0-9-\.\_]+)|(s3.console.aws.amazon.com\/s3\/buckets\/[a-zA-Z0-9-\.\_]+)|(amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})|(ec2-[0-9-]+.cd-[a-z0-9-]+.compute.amazonaws.com)|(us[_-]?east[_-]?1[_-]?elb[_-]?amazonaws[_-]?com)) 140 | scope: response body 141 | sensitive: true 142 | - color: green 143 | engine: nfa 144 | loaded: true 145 | name: Amazon AWS Region 146 | regex: ((us(-gov)?|ap|ca|cn|eu|sa)-(central|(north|south)?(east|west)?)-\d) 147 | scope: response body 148 | sensitive: true 149 | - color: blue 150 | engine: nfa 151 | loaded: true 152 | name: SSH Private Key 153 | regex: ([-]+BEGIN [^\s]+ PRIVATE KEY[-]) 154 | scope: response body 155 | sensitive: true 156 | - color: green 157 | engine: nfa 158 | loaded: true 159 | name: Windows File/Dir Path 160 | regex: '[^\w](([a-zA-Z]:\\(?:\w+\\?)*)|([a-zA-Z]:\\(?:\w+\\)*\w+\.\w+))' 161 | scope: response 162 | sensitive: true 163 | - color: yellow 164 | engine: nfa 165 | loaded: true 166 | name: Password Field 167 | regex: ((|'|")([p](ass|wd|asswd|assword))(|'|")(:|=)( |)('|")(.*?)('|")(|,)) 168 | scope: response body 169 | sensitive: false 170 | - color: green 171 | engine: nfa 172 | loaded: true 173 | name: Username Field 174 | regex: ((|'|")([u](ser|name|ame|sername))(|'|")(:|=)( |)('|")(.*?)('|")(|,)) 175 | scope: response body 176 | sensitive: false 177 | - color: green 178 | engine: dfa 179 | loaded: true 180 | name: WeCom Key 181 | regex: ([c|C]or[p|P]id|[c|C]orp[s|S]ecret) 182 | scope: response body 183 | sensitive: true 184 | - color: yellow 185 | engine: nfa 186 | loaded: true 187 | name: JDBC Connection 188 | regex: (jdbc:[a-z:]+://[a-z0-9\.\-_:;=/@?,&]+) 189 | scope: any 190 | sensitive: false 191 | - color: yellow 192 | engine: nfa 193 | loaded: true 194 | name: Authorization Header 195 | regex: ((basic [a-z0-9=:_\+\/-]{5,100})|(bearer [a-z0-9_.=:_\+\/-]{5,100})) 196 | scope: response body 197 | sensitive: false 198 | - color: green 199 | engine: nfa 200 | loaded: true 201 | name: Github Access Token 202 | regex: ([a-z0-9_-]*:[a-z0-9_\-]+@github\.com*) 203 | scope: response body 204 | sensitive: false 205 | - color: cyan 206 | engine: nfa 207 | loaded: true 208 | name: Microsoft Teams Webhook 209 | regex: (https://outlook\.office\.com/webhook/[a-z0-9@-]+/IncomingWebhook/[a-z0-9-]+/[a-z0-9-]+) 210 | scope: response body 211 | sensitive: false 212 | - color: cyan 213 | engine: nfa 214 | loaded: true 215 | name: Zoho Webhook 216 | regex: (https://creator\.zoho\.com/api/[a-z0-9/_.-]+\?authtoken=[a-z0-9]+) 217 | scope: response body 218 | sensitive: false 219 | - color: cyan 220 | engine: nfa 221 | loaded: true 222 | name: Sonarqube Token 223 | regex: (sonar.{0,50}(?:"|\'|`)?[0-9a-f]{40}(?:"|\'|`)?) 224 | scope: response body 225 | sensitive: false 226 | type: Sensitive Information 227 | - rule: 228 | - color: gray 229 | engine: nfa 230 | loaded: true 231 | name: Linkfinder 232 | regex: (?:"|')(((?:[a-zA-Z]{1,10}://|//)[^"'/]{1,}\.[a-zA-Z]{2,}[^"']{0,})|((?:/|\.\./|\./)[^"'><,;|*()(%%$^/\\\[\]][^"'><,;|()]{1,})|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{1,}\.(?:[a-zA-Z]{1,4}|action)(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{3,}(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-]{1,}\.(?:php|asp|aspx|jsp|json|action|html|js|txt|xml)(?:[\?|#][^"|']{0,}|)))(?:"|') 233 | scope: response body 234 | sensitive: true 235 | - color: pink 236 | engine: dfa 237 | loaded: true 238 | name: Source Map 239 | regex: (\.js\.map) 240 | scope: response body 241 | sensitive: true 242 | - color: magenta 243 | engine: nfa 244 | loaded: true 245 | name: HTML Notes 246 | regex: () 247 | scope: response body 248 | sensitive: true 249 | - color: green 250 | engine: dfa 251 | loaded: true 252 | name: Create Script 253 | regex: (createElement\(\"script\"\)) 254 | scope: response body 255 | sensitive: true 256 | type: Other 257 | -------------------------------------------------------------------------------- /config/pc_ua.json: -------------------------------------------------------------------------------- 1 | {"chrome": ["Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36", "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36", "Mozilla/5.0 (Windows NT 4.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2117.157 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1866.237 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/4E423F", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36 Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.517 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1664.3 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1664.3 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.16 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36", "Mozilla/5.0 (X11; CrOS i686 4319.74.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1467.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1464.0 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1500.55 Safari/537.36", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.90 Safari/537.36", "Mozilla/5.0 (X11; NetBSD) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36", "Mozilla/5.0 (X11; CrOS i686 3912.101.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.60 Safari/537.17", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko) Chrome/24.0.1295.0 Safari/537.15", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.14 (KHTML, like Gecko) Chrome/24.0.1292.0 Safari/537.14"], "opera": ["Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16", "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14", "Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0) Opera 12.14", "Opera/12.80 (Windows NT 5.1; U; en) Presto/2.10.289 Version/12.02", "Opera/9.80 (Windows NT 6.1; U; es-ES) Presto/2.9.181 Version/12.00", "Opera/9.80 (Windows NT 5.1; U; zh-sg) Presto/2.9.181 Version/12.00", "Opera/12.0(Windows NT 5.2;U;en)Presto/22.9.168 Version/12.00", "Opera/12.0(Windows NT 5.1;U;en)Presto/22.9.168 Version/12.00", "Mozilla/5.0 (Windows NT 5.1) Gecko/20100101 Firefox/14.0 Opera/12.0", "Opera/9.80 (Windows NT 6.1; WOW64; U; pt) Presto/2.10.229 Version/11.62", "Opera/9.80 (Windows NT 6.0; U; pl) Presto/2.10.229 Version/11.62", "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52", "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 Version/11.52", "Opera/9.80 (Windows NT 5.1; U; en) Presto/2.9.168 Version/11.51", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; de) Opera 11.51", "Opera/9.80 (X11; Linux x86_64; U; fr) Presto/2.9.168 Version/11.50", "Opera/9.80 (X11; Linux i686; U; hu) Presto/2.9.168 Version/11.50", "Opera/9.80 (X11; Linux i686; U; ru) Presto/2.8.131 Version/11.11", "Opera/9.80 (X11; Linux i686; U; es-ES) Presto/2.8.131 Version/11.11", "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/5.0 Opera 11.11", "Opera/9.80 (X11; Linux x86_64; U; bg) Presto/2.8.131 Version/11.10", "Opera/9.80 (Windows NT 6.0; U; en) Presto/2.8.99 Version/11.10", "Opera/9.80 (Windows NT 5.1; U; zh-tw) Presto/2.8.131 Version/11.10", "Opera/9.80 (Windows NT 6.1; Opera Tablet/15165; U; en) Presto/2.8.149 Version/11.1", "Opera/9.80 (X11; Linux x86_64; U; Ubuntu/10.10 (maverick); pl) Presto/2.7.62 Version/11.01", "Opera/9.80 (X11; Linux i686; U; ja) Presto/2.7.62 Version/11.01", "Opera/9.80 (X11; Linux i686; U; fr) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.1; U; sv) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 6.0; U; pl) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 5.2; U; ru) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 5.1; U;) Presto/2.7.62 Version/11.01", "Opera/9.80 (Windows NT 5.1; U; cs) Presto/2.7.62 Version/11.01", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101213 Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.7.62 Version/11.01", "Mozilla/5.0 (Windows NT 6.1; U; nl; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.01", "Mozilla/5.0 (Windows NT 6.1; U; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.01", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; de) Opera 11.01", "Opera/9.80 (X11; Linux x86_64; U; pl) Presto/2.7.62 Version/11.00", "Opera/9.80 (X11; Linux i686; U; it) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.6.37 Version/11.00", "Opera/9.80 (Windows NT 6.1; U; pl) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.1; U; ko) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.1; U; fi) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.1; U; en-GB) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.1 x64; U; en) Presto/2.7.62 Version/11.00", "Opera/9.80 (Windows NT 6.0; U; en) Presto/2.7.39 Version/11.00"], "firefox": ["Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1", "Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0", "Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0", "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/29.0", "Mozilla/5.0 (X11; OpenBSD amd64; rv:28.0) Gecko/20100101 Firefox/28.0", "Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0", "Mozilla/5.0 (Windows NT 6.1; rv:27.3) Gecko/20130101 Firefox/27.3", "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:27.0) Gecko/20121011 Firefox/27.0", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0", "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0", "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20130405 Firefox/23.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20130406 Firefox/23.0", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:23.0) Gecko/20131011 Firefox/23.0", "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20130405 Firefox/22.0", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:22.0) Gecko/20130328 Firefox/22.0", "Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20130405 Firefox/22.0", "Mozilla/5.0 (Microsoft Windows NT 6.2.9200.0); rv:22.0) Gecko/20130405 Firefox/22.0", "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/21.0.1", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/21.0.1", "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:21.0.0) Gecko/20121011 Firefox/21.0.0", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20130331 Firefox/21.0", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:21.0) Gecko/20130514 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.2; rv:21.0) Gecko/20130326 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130401 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130331 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130330 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130401 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130328 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20130401 Firefox/21.0", "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20130331 Firefox/21.0", "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Windows NT 5.0; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0", "Mozilla/5.0 (Windows NT 6.2; Win64; x64;) Gecko/20100101 Firefox/20.0", "Mozilla/5.0 (Windows x86; rv:19.0) Gecko/20100101 Firefox/19.0", "Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20100101 Firefox/19.0", "Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/18.0.1", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0.6"], "internetexplorer": ["Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko", "Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko", "Mozilla/5.0 (compatible; MSIE 10.6; Windows NT 6.1; Trident/5.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727) 3gpp-gba UNTRUSTED/1.0", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/4.0; InfoPath.2; SV1; .NET CLR 2.0.50727; WOW64)", "Mozilla/5.0 (compatible; MSIE 10.0; Macintosh; Intel Mac OS X 10_7_3; Trident/6.0)", "Mozilla/4.0 (Compatible; MSIE 8.0; Windows NT 5.2; Trident/6.0)", "Mozilla/4.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)", "Mozilla/1.22 (compatible; MSIE 10.0; Windows 3.1)", "Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))", "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; Media Center PC 6.0; InfoPath.3; MS-RTC LM 8; Zune 4.7)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; Media Center PC 6.0; InfoPath.3; MS-RTC LM 8; Zune 4.7", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 4.0; InfoPath.3; MS-RTC LM 8; .NET4.0C; .NET4.0E)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; chromeframe/12.0.742.112)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 4.0; Tablet PC 2.0; InfoPath.3; .NET4.0C; .NET4.0E)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; yie8)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET CLR 1.1.4322; .NET4.0C; Tablet PC 2.0)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; FunWebProducts)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; chromeframe/13.0.782.215)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; chromeframe/11.0.696.57)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) chromeframe/10.0.648.205", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; GTB7.4; InfoPath.1; SV1; .NET CLR 2.8.52393; WOW64; en-US)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; chromeframe/11.0.696.57)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/4.0; GTB7.4; InfoPath.3; SV1; .NET CLR 3.1.76908; WOW64; en-US)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.4; InfoPath.2; SV1; .NET CLR 3.3.69573; WOW64; en-US)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; InfoPath.1; SV1; .NET CLR 3.8.36217; WOW64; en-US)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; .NET CLR 2.7.58687; SLCC2; Media Center PC 5.0; Zune 3.4; Tablet PC 3.6; InfoPath.3)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Media Center PC 4.0; SLCC1; .NET CLR 3.0.04320)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; SLCC1; .NET CLR 1.1.4322)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.0; Trident/4.0; InfoPath.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 3.0.04506.30)", "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.0; Trident/4.0; FBSMTWB; .NET CLR 2.0.34861; .NET CLR 3.0.3746.3218; .NET CLR 3.5.33652; msn OptimizedIE8;ENUS)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.3; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MS-RTC LM 8)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.2)", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 3.0)"], "safari": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A", "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10", "Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko ) Version/5.1 Mobile/9B176 Safari/7534.48.3", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; de-at) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; da-dk) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1", "Mozilla/5.0 (Windows; U; Windows NT 6.1; tr-TR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.1; ko-KR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.1; fr-FR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.1; cs-CZ) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.0; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; zh-cn) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; zh-cn) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; sv-se) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; ko-kr) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; fr-fr) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; es-es) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-gb) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; de-de) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27", "Mozilla/5.0 (Windows; U; Windows NT 6.1; sv-SE) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.1; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.0; hu-HU) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.0; de-DE) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 5.1; it-IT) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-us) AppleWebKit/534.16+ (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; fr-ch) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; de-de) AppleWebKit/534.15+ (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; ar) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Android 2.2; Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4", "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-HK) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 6.0; tr-TR) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 6.0; nb-NO) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 6.0; fr-FR) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; zh-cn) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5"]} -------------------------------------------------------------------------------- /config/phantomjs_liunx_x64: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/config/phantomjs_liunx_x64 -------------------------------------------------------------------------------- /config/phantomjs_windows.exe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/config/phantomjs_windows.exe -------------------------------------------------------------------------------- /imgs/five.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/five.png -------------------------------------------------------------------------------- /imgs/four.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/four.png -------------------------------------------------------------------------------- /imgs/help.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/help.png -------------------------------------------------------------------------------- /imgs/logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/logo.jpg -------------------------------------------------------------------------------- /imgs/one.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/one.png -------------------------------------------------------------------------------- /imgs/threes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/threes.png -------------------------------------------------------------------------------- /imgs/two.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/imgs/two.png -------------------------------------------------------------------------------- /lib/Controller.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | from lib.common.utils import Utils 5 | from lib.WebpackFind import webpackfind_class 6 | from lib.common.toupdate import ToUpdate 7 | from lib.common.cmdline import CommandLines 8 | from lib.common.fileoperation import FileOperation 9 | from queue import Queue 10 | from tqdm import * 11 | 12 | class Controller(): 13 | 14 | def __init__(self, options): 15 | self.options = options 16 | 17 | def start(self): 18 | # 判断cookie是否为空 19 | if self.options.cookies == None: 20 | self.options.cookies = "" 21 | 22 | # 声明全局调用headers 23 | headers = {"User-Agent": FileOperation().uarand(), "Cookie": self.options.cookies} 24 | if self.options.headers != None: 25 | headers = Utils().handle_headers(headers, self.options.headers) 26 | 27 | # 声明全局uuid 28 | uuid = Utils().random_uuid() + "/" 29 | 30 | if self.options.update != 0: 31 | # 检查更新 32 | ToUpdate().get_version() 33 | 34 | elif self.options.jsfile != None: 35 | 36 | webpackfind_class([], uuid, None, headers).jsfile_main(self.options.jsfile) 37 | 38 | elif self.options.urlfile != None: 39 | 40 | pbar = tqdm(total=1, position=0, desc="完成进度") 41 | 42 | webpackfind_class([], uuid, pbar, headers).main(self.options.urlfile) 43 | 44 | pbar.close() 45 | 46 | elif self.options.file != None: 47 | 48 | try: 49 | # 声明多线程队列 50 | urllist = Queue(-1) 51 | file = open(str(self.options.file), 'r', encoding='utf-8') 52 | filelist = file.readlines() 53 | for i in range(len(filelist)): 54 | urllist.put([filelist[i].strip(), len(filelist)]) 55 | 56 | pbar = tqdm(total=len(filelist), position=0, desc="完成进度") 57 | threads = [] 58 | for num in range(1, len(filelist) + 1): 59 | t = webpackfind_class(urllist, uuid, pbar, headers) 60 | threads.append(t) 61 | t.start() 62 | 63 | for t in threads: 64 | t.join() 65 | 66 | pbar.close() 67 | 68 | except Exception as e: 69 | tqdm.write(CommandLines().help()) 70 | else: 71 | tqdm.write(CommandLines().help()) 72 | 73 | # 清空phantomjs日志 74 | FileOperation().rmphantomjslog() 75 | -------------------------------------------------------------------------------- /lib/WebpackFind.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import os, threading 5 | from lib.common.utils import Utils 6 | from lib.common.fileoperation import FileOperation 7 | from lib.common.downloadjs import DownloadJs 8 | from lib.common.regular import Regular 9 | from lib.vuln.unauthorized import unauthorized 10 | from lib.common.urlrequest import UrlRequest 11 | from lib.common.Printlog import Printlog 12 | from urllib.parse import urlparse 13 | from bs4 import BeautifulSoup 14 | 15 | class webpackfind_class(threading.Thread): 16 | 17 | def __init__(self, urllist=[], uuid="", pbar=None, headers={}): 18 | threading.Thread.__init__(self) 19 | self.urllist = urllist 20 | self.uuid = uuid 21 | self.domain = "" 22 | # 初始化加载规则库 23 | self.rulesJson = FileOperation().getyml() 24 | self.log = None 25 | self.domainlog = "" 26 | self.pbar = pbar 27 | self.headers = headers 28 | 29 | # js读取本地文件 30 | def jsfile_main(self, jsfile=""): 31 | try: 32 | # 清空日志文件 33 | FileOperation().rmlogfile(jsfile) 34 | 35 | self.log = Printlog(jsfile).get_logger() 36 | 37 | # 遍历读取文件目录中的文件 38 | eachinfo = FileOperation().eachFile(jsfile, 1) 39 | 40 | # 遍历读取文件路径 41 | eachfile = FileOperation().eachFile(jsfile, 2) 42 | 43 | # 匹配规则库 44 | Regular().run_Matching_rules(os.path.basename(os.path.realpath(jsfile)), jsfile, eachfile, self.rulesJson) 45 | 46 | # 在所有的urls中提取出目标站的子域名 47 | Regular().find_subdomain(os.path.basename(os.path.realpath(jsfile)), eachinfo, jsfile, os.path.basename(os.path.realpath(jsfile))) 48 | 49 | # 探测未授权接口 50 | unauthorized(os.path.basename(os.path.realpath(jsfile)), self.headers, os.path.basename(os.path.realpath(jsfile)), self.rulesJson, jsfile).run(eachfile) 51 | 52 | self.log.info("{} 【{}】 扫描成功 路径：{}".format(Utils().tellTime(), os.path.basename(os.path.realpath(jsfile)), jsfile)) 53 | 54 | except Exception as e: 55 | self.log.error("{} 【错误异常】：{}".format(Utils().tellTime(), str(e))) 56 | 57 | # 获取js文件url 58 | def get_js(self, domain="", path=""): 59 | 60 | domain = UrlRequest(self.headers).Auto_select_protocol(domain) 61 | 62 | self.log.info("{} 【{}】 协议头检测：{}".format(Utils().tellTime(), self.domainlog, str(domain))) 63 | 64 | self.domain = domain 65 | 66 | url = [] 67 | 68 | new_domain = "" 69 | 70 | if urlparse(domain).netloc: 71 | 72 | # url自动化遍历读取文件 73 | content, js_array = UrlRequest(self.headers).phantomjs_requests(domain) 74 | 75 | for i in range(len(js_array)): 76 | url.append(js_array[i]) 77 | 78 | if content: 79 | 80 | content = BeautifulSoup(content, "html.parser") 81 | 82 | script = content.find_all('script') 83 | 84 | for a in range(len(script)): 85 | if script: 86 | try: 87 | for a in range(len(script)): 88 | if urlparse(script[a].get("src")).netloc != "": 89 | if urlparse(self.domain).netloc != urlparse(script[a].get("src")).netloc: 90 | domain = script[a].get("src") 91 | else: 92 | domain = self.domain 93 | else: 94 | domain = self.domain 95 | 96 | if script[a].get("src") != None and script[a].get("src")[0:2] == "//": 97 | scr_path = os.path.basename(script[a].get("src").replace("./", "/")) 98 | if os.path.basename(script[a].get("src").replace("./", "/")).find("?") != -1: 99 | scr_path = os.path.basename(script[a].get("src").replace("./", "/"))[0:os.path.basename(script[a].get("src").replace("./", "/")).find("?")] 100 | if len(js_array) > 0: 101 | if Utils().is_array_value(js_array, Utils().get_filename(scr_path)) == True: 102 | continue 103 | elif domain == None: 104 | # 读取js文件路径 105 | js_content = FileOperation().resolve_path(str(script[a])) 106 | for i in range(len(js_content)): 107 | url.append(self.domain + "/" + js_content[i]) 108 | url = list(set(url)) 109 | else: 110 | try: 111 | scheme = "http:" 112 | if urlparse(domain).scheme != "": 113 | scheme = str(urlparse(domain).scheme) + ":" 114 | if urlparse(domain).path: 115 | new_domain = scheme + "//" + urlparse(domain).netloc + "/" + str(urlparse(domain).path).split("/")[1] 116 | else: 117 | new_domain = scheme + "//" + urlparse(domain).netloc 118 | if script[a].get("data-main") != None: 119 | domain_url = new_domain + os.path.normpath(script[a].get("src").replace("./", "/")) 120 | if urlparse(domain).path: 121 | if script[a].get("data-main").find(str(urlparse(new_domain).path).split("/")[1]) != -1: 122 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + os.path.normpath(script[a].get("data-main")).replace("\\", "/").replace("./", "/") 123 | if os.path.normpath(script[a].get("data-main")).replace("\\", "/").replace("./", "/")[:1] != "/": 124 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + "/" + os.path.normpath(script[a].get("data-main")).replace("\\", "/").replace("./", "/") 125 | url.append(domain_url) 126 | elif script[a].get("src") != None: 127 | if "http" in script[a].get("src"): 128 | url.append(script[a].get("src").replace("./", "/")) 129 | elif "runtime." in script[a].get("src") or "app." in script[a].get("src") or "finance." in script[a].get("src") or "vendor." in script[a].get("src") or "manifest." in script[a].get("src"): 130 | if urlparse(os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/")).netloc == "": 131 | domain_url = new_domain + os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/") 132 | else: 133 | domain_url = new_domain + str(scheme + os.path.normpath(script[a].get("src")).replace("\\","/").replace("./", "/")).replace(new_domain, "") 134 | if urlparse(new_domain).path: 135 | if os.path.normpath(script[a].get("src")).replace("\\","/").replace("./", "/")[:1] != "/": 136 | if urlparse(os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/")).netloc == "": 137 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + urlparse(new_domain).path + "/" + os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/") 138 | else: 139 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + urlparse(new_domain).path + "/" + str(scheme + os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/")).replace(new_domain, "") 140 | else: 141 | if urlparse(os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/")).netloc == "": 142 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + urlparse(new_domain).path + os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/") 143 | else: 144 | domain_url = urlparse(new_domain).scheme + "://" + urlparse(new_domain).netloc + urlparse(new_domain).path + str(scheme + os.path.normpath(script[a].get("src")).replace("\\", "/").replace("./", "/")).replace(new_domain, "") 145 | content = UrlRequest(self.headers).Extract_html(domain_url) 146 | if content == None: 147 | try: 148 | if domain == None: 149 | domainfname = os.path.join(path, str(urlparse(self.domain).netloc).replace(":", "_") + "_error_js_url_list.txt") 150 | else: 151 | domainfname = os.path.join(path, str(urlparse(domain).netloc).replace(":", "_") + "_error_js_url_list.txt") 152 | FileOperation().save_result(domainfname, domain) 153 | except Exception as e: 154 | self.log.error("{} 【写入文件失败】：{}".format(Utils().tellTime(), str(e))) 155 | continue 156 | else: 157 | # 读取js文件路径 158 | js_content = FileOperation().resolve_path(str(content)) 159 | for i in range(len(js_content)): 160 | url.append(new_domain + "/" + js_content[i]) 161 | url = list(set(url)) 162 | else: 163 | script_text = script[a].get("src").replace("./", "/") 164 | if script_text[0] != "/": 165 | script_text = "/" + script_text 166 | if len(js_array) > 0: 167 | if Utils().is_array_value(js_array, os.path.basename(script[a].get("src").replace("./", "/"))) == True: 168 | continue 169 | 170 | if urlparse(script_text).scheme == "": 171 | if urlparse(script_text).netloc != "": 172 | url.append(scheme + script_text) 173 | else: 174 | url.append(new_domain + script_text) 175 | else: 176 | url.append(new_domain + script_text) 177 | except Exception as e: 178 | self.log.error("{} 【解析js出错】：{}".format(Utils().tellTime(), str(e))) 179 | continue 180 | except Exception as e: 181 | self.log.error("{} 【解析js出错】：{}".format(Utils().tellTime(), str(e))) 182 | pass 183 | 184 | # 解决重复url问题 185 | new_url = [] 186 | for u in range(len(url)): 187 | if Utils().White_list_domain(url[u]): 188 | url[u] = url[u].replace("///", "/").replace("\\", "/").replace("./", "/").replace("///", "/") 189 | if url[u].find("?") != -1: 190 | new_url.append(url[u][:url[u].find("?")]) 191 | elif url[u].find(";") != -1: 192 | new_url.append(url[u][:url[u].find(";")]) 193 | else: 194 | if urlparse(url[u]).netloc == "": 195 | continue 196 | else: 197 | new_url.append(url[u]) 198 | url = list(set(new_url)) 199 | try: 200 | if domain == None: 201 | domainfname = os.path.join(path, str(urlparse(self.domain).netloc).replace(":", "_") + "_js_url_list.txt") 202 | else: 203 | domainfname = os.path.join(path, str(urlparse(domain).netloc).replace(":", "_") + "_js_url_list.txt") 204 | for u in range(len(url)): 205 | FileOperation().save_result(domainfname, str(url[u])) 206 | return url 207 | except Exception as e: 208 | self.log.error("{} 【写入文件失败】：{}".format(Utils().tellTime(), str(e))) 209 | return url 210 | elif content == None: 211 | try: 212 | fname = os.path.join(path, str(urlparse(new_domain).netloc).replace(":", "_") + "_error_js_url_list.txt") 213 | FileOperation().save_result(fname, domain) 214 | return url 215 | except Exception as e: 216 | self.log.error("{} 【写入文件失败】：{}".format(Utils().tellTime(), str(e))) 217 | return url 218 | else: 219 | self.log.error("{} 【扫描失败】：{}".format(Utils().tellTime(), str(domain))) 220 | return url 221 | 222 | # 入口函数 223 | def main(self, domain=""): 224 | # 做下vue url存在 /#/ 问题 225 | if "/#/" in domain: 226 | domain = domain.split("/#/")[0] 227 | 228 | # 声明保存路径 229 | if urlparse(domain).netloc == "": 230 | path = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))), 'tmp', self.uuid, str(domain).replace(":", "_"), '') 231 | self.domainlog = str(domain) 232 | else: 233 | path = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))), 'tmp', self.uuid, str(urlparse(domain).netloc).replace(":", "_"), '') 234 | self.domainlog = str(urlparse(domain).netloc) 235 | 236 | FileOperation().mkdir(path) 237 | 238 | self.log = Printlog(path).get_logger() 239 | 240 | self.log.info("{} 【{}】 初始化创建路径：{}".format(Utils().tellTime(), self.domainlog, path)) 241 | 242 | # 读取js文件路径 243 | info = self.get_js(domain, path) 244 | 245 | if info: 246 | 247 | self.log.info("{} 【{}】 获取到js路径，数量：{}".format(Utils().tellTime(), self.domainlog, len(info))) 248 | 249 | # 扫描结果存入 result.txt 250 | FileOperation().save_result(os.path.join(path, 'result.txt'), "【URL】：{}".format(str(domain)), "w") 251 | 252 | # 批量下载js 253 | DownloadJs(self.domainlog, info, path, self.headers).run() 254 | 255 | # 遍历读取文件目录中的文件 256 | eachinfo = FileOperation().eachFile(path, 1) 257 | 258 | # 遍历读取文件路径 259 | eachfile = FileOperation().eachFile(path, 2) 260 | 261 | # 匹配规则库 262 | Regular().run_Matching_rules(self.domainlog, path, eachfile, self.rulesJson) 263 | 264 | # 在所有的urls中提取出目标站的子域名 265 | Regular().find_subdomain(self.domainlog, eachinfo, path, domain) 266 | 267 | # 探测未授权接口 268 | unauthorized(self.domainlog, self.headers, domain, self.rulesJson, path).run(eachfile) 269 | 270 | self.log.info("{} 【{}】 扫描成功 路径：{}".format(Utils().tellTime(), self.domainlog, str(path))) 271 | 272 | else: 273 | 274 | self.log.info("{} 【扫描失败】：{}".format(Utils().tellTime(), str(domain))) 275 | 276 | self.pbar.update() 277 | 278 | # 主函数 279 | def run(self): 280 | while not self.urllist.empty(): 281 | url, i = self.urllist.get() 282 | self.main(url) 283 | self.urllist.task_done() 284 | 285 | 286 | 287 | -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/lib/__init__.py -------------------------------------------------------------------------------- /lib/common/Printlog.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | import logging, time, os 4 | 5 | class Printlog(): 6 | def __init__(self, path="", logger=None): 7 | self.logger = logging.getLogger(logger) 8 | self.logger.setLevel(logging.NOTSET) 9 | self.log_time = time.strftime("%Y-%m-%d %H:%M:%S") 10 | self.path = path 11 | 12 | def info(self, message): 13 | self.fontColor('\033[0;34m%s\033[0m') 14 | self.logger.info(message) 15 | 16 | def set_logger(self): 17 | if not self.logger.handlers: 18 | self.fh = logging.FileHandler(os.path.join(self.path, "run_log.txt"), "w", encoding="utf-8") 19 | self.fh.setLevel(logging.DEBUG) 20 | self.chd = logging.StreamHandler() 21 | self.chd.setLevel(logging.INFO) 22 | self.formatter = logging.Formatter("[%(levelname)s] %(asctime)s %(filename)s -> %(funcName)s line:%(lineno)d: %(message)s\n") 23 | self.formatter_info = logging.Formatter() 24 | self.chd.setFormatter(self.formatter_info) 25 | self.fh.setFormatter(self.formatter) 26 | self.logger.addHandler(self.fh) 27 | self.logger.addHandler(self.chd) 28 | 29 | def get_logger(self): 30 | Printlog.set_logger(self) 31 | return self.logger 32 | 33 | def remove_log_handler(self): 34 | self.logger.removeHandler(self.fh) 35 | self.logger.removeHandler(self.chd) 36 | self.fh.close() 37 | self.chd.close() 38 | -------------------------------------------------------------------------------- /lib/common/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/lib/common/__init__.py -------------------------------------------------------------------------------- /lib/common/cmdline.py: -------------------------------------------------------------------------------- 1 | # !/usr/bin/env python3 2 | # -*- encoding: utf-8 -*- 3 | 4 | import argparse 5 | 6 | class CommandLines(): 7 | 8 | # 参数输入 9 | def cmd(self): 10 | parser = argparse.ArgumentParser(description=self.help(), add_help=False) 11 | parser.add_argument("-u", "--urlfile", help="自动化遍历URL中里面js") 12 | parser.add_argument("-j", "--jsfile", help="遍历本地目录中的js文件并且格式化js文件、提取信息泄露") 13 | parser.add_argument("-f", "--file", help="读取本地url文件批量遍历url中js") 14 | parser.add_argument("-c", "--cookies", help="设置自定义Cookie，场景:需要登录才能爬取js信息") 15 | parser.add_argument("-d", "--headers", help="设置自定义headers") 16 | parser.add_argument("-update", "--update", type=int, default=0, help="检查版本更新") 17 | return parser.parse_args() 18 | 19 | # 帮助文档输出 20 | def help(self): 21 | return ''' 22 | 例子： 23 | 自动化遍历URL中里面js 24 | python3 webpackfind.py -u http://domain.com 25 | 26 | 遍历本地目录中的js文件并且格式化js文件、提取信息泄露 27 | python3 webpackfind.py -j ./js/test.com/ 28 | 29 | 读取本地url文件批量遍历url中js 30 | python3 webpackfind.py -f ./url.txt 31 | 32 | 设置自定义Cookie，场景:需要登录才能爬取js信息 33 | python3 webpackfind.py -c "name=test;" -u http://domain.com 34 | 35 | 设置自定义headers 36 | python3 webpackfind.py -d "Authorization: bearer 123456\nContent-Type: application/json\n" -u http://domain.com 37 | 38 | 检查本地`webpackfind.py`是否与github.com是否一致 39 | python3 webpackfind.py -update 1 40 | ''' -------------------------------------------------------------------------------- /lib/common/downloadjs.py: -------------------------------------------------------------------------------- 1 | # !/usr/bin/env python3 2 | # -*- encoding: utf-8 -*- 3 | 4 | import jsbeautifier 5 | from lib.common.urlrequest import UrlRequest 6 | from lib.common.Printlog import Printlog 7 | from lib.common.utils import Utils 8 | from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED 9 | 10 | class DownloadJs(): 11 | 12 | def __init__(self, domainlog="", urllist=[], path="", headers={}): 13 | self.urllist = urllist 14 | self.path = path 15 | self.headers = headers 16 | self.log = Printlog(path).get_logger() 17 | self.downloadpbar = None 18 | self.success = 0 19 | self.error = 0 20 | self.domainlog = domainlog 21 | 22 | # 主函数 23 | def run(self): 24 | 25 | # 创建线程池 26 | pool = ThreadPoolExecutor(20) 27 | 28 | allTask = [pool.submit(self.Download, url) for url in self.urllist] 29 | 30 | # 开启异步线程池 31 | wait(allTask, return_when=ALL_COMPLETED) 32 | 33 | # log日志输出 34 | self.log.info("{} 【{}】 【下载完成】总数：{} 成功：{} 失败：{}".format(Utils().tellTime(), self.domainlog, str(len(self.urllist)), str(self.success), str(self.error))) 35 | 36 | def Download(self, url): 37 | 38 | url = url.replace("//", "/").replace(":/", "://") 39 | 40 | # 请求接口 41 | content = UrlRequest(self.headers).Extract_html(url) 42 | 43 | if content: 44 | self.success = self.success + 1 45 | fname = self.path + "/" + url.split('/')[-1] 46 | try: 47 | fp = open(fname, "at", encoding='utf-8') 48 | fp.write(jsbeautifier.beautify(content)) 49 | fp.close() 50 | except Exception as e: 51 | self.log.error("{} 【{}】【写入文件失败】：{}".format(Utils().tellTime(), self.domainlog, str(e))) 52 | return 53 | else: 54 | self.error = self.error + 1 55 | return -------------------------------------------------------------------------------- /lib/common/fileoperation.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import os, json, random, jsbeautifier, yaml, re, js2py 5 | from lib.common.regular import Regular 6 | 7 | class FileOperation(): 8 | 9 | # 写入文件 10 | def save_result(self, filename="", content="", jurisdiction="at"): 11 | fp = open(filename, jurisdiction, encoding='utf-8') 12 | fp.write(content + "\n") 13 | fp.close() 14 | 15 | # 创建文件夹 16 | def mkdir(self, path): 17 | path = path.strip() 18 | path = path.rstrip("\\") 19 | isExists = os.path.exists(path) 20 | if not isExists: 21 | os.makedirs(path) 22 | return True 23 | else: 24 | self.del_file(path) 25 | return True 26 | 27 | # 清空文件夹内容 28 | def del_file(self, path): 29 | ls = os.listdir(path) 30 | for i in ls: 31 | c_path = os.path.join(path, i) 32 | if os.path.isdir(c_path): 33 | self.del_file(c_path) 34 | else: 35 | os.remove(c_path) 36 | 37 | # 遍历指定目录，显示目录下的所有文件名 38 | def eachFile(self, filepath="", type=1): 39 | url = [] 40 | pathDir = os.listdir(filepath) 41 | for allDir in pathDir: 42 | if os.path.splitext(allDir)[-1] != ".txt": 43 | child = os.path.join('%s%s' % (filepath, allDir)) 44 | if type == 1: 45 | info = self.readFile(child) 46 | if info: 47 | for u in range(len(info)): 48 | url.append(info[u]) 49 | else: 50 | url.append(child) 51 | return url 52 | 53 | # 遍历指定目录，并格式化js源码 54 | def eachFormatJs(self, filepath): 55 | pathDir = os.listdir(filepath) 56 | for allDir in pathDir: 57 | child = os.path.join('%s%s' % (filepath, allDir)) 58 | # 读取文件 59 | fopen = open(child, 'r', encoding='utf-8') 60 | txt = jsbeautifier.beautify(fopen.read()) 61 | fopen.close() 62 | # 写入文件 63 | file_object = open(child, 'w', encoding='utf-8') 64 | file_object.write(txt) 65 | file_object.close() 66 | 67 | # 读取文件内容并打印 68 | def readFile(self, filename): 69 | url = [] 70 | fopen = open(filename, 'r', encoding='utf-8') 71 | data = Regular().Extract_URL(fopen.read()) 72 | self.save_result(os.path.join(os.path.dirname(filename), "result.txt"), "【+】" + filename) 73 | for x in data: 74 | self.save_result(os.path.join(os.path.dirname(filename),"result.txt"), " " + str(x)) 75 | url.append(x) 76 | fopen.close() 77 | return url 78 | 79 | # 随机获取ua库 80 | def uarand(self): 81 | ie_type = ["chrome", "opera", "firefox", "internetexplorer", "safari"] 82 | with open(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'config', 'pc_ua.json'), 'r', encoding='utf8') as fp: 83 | json_data = json.load(fp) 84 | return json_data[ie_type[random.randint(0, len(ie_type) - 1)]][random.randint(0, len(json_data[ie_type[random.randint(0, len(ie_type) - 1)]]) - 1)] 85 | 86 | # 读取yml规则库 87 | def getyml(self): 88 | rulesJson = [] 89 | try: 90 | file = open(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'config', 'Config.yml'), 'r', encoding="utf-8") 91 | file_data = file.read() 92 | file.close() 93 | data = yaml.load(file_data, Loader=yaml.FullLoader) 94 | rules = data.get("rules") 95 | # Fingerprint 指纹 96 | # Basic Information 基本信息 97 | # Maybe Vulnerability 可能是脆弱性 98 | # Sensitive Information 敏感信息 99 | # Other 另外 100 | for i in range(len(rules)): 101 | for r in range(len(rules[i]["rule"])): 102 | rulesJson.append({"type_name": rules[i]["type"], "name": rules[i]["rule"][r]["name"], "regex": rules[i]["rule"][r]["regex"]}) 103 | return rulesJson 104 | except: 105 | return rulesJson 106 | 107 | # 读取js文件路径 108 | def resolve_path(self, content=""): 109 | r = re.compile(r"\w\.p\+\"(.*?)\.js", re.DOTALL) 110 | result = r.findall(str(content)) 111 | urllist = [] 112 | for jsCode in result: 113 | if len(jsCode) < 30000: 114 | jsCode = "\"" + jsCode + ".js\"" 115 | variable = re.findall(r'\[.*?\]', jsCode) 116 | if "[" and "]" in variable[0]: 117 | variable = variable[0].replace("[", "").replace("]", "") 118 | jsCodeFunc = "function js_compile(%s){js_url=" % (variable) + jsCode + "\nreturn js_url}" 119 | pattern_jscode = re.compile(r"\(\{\}\[(.*?)\]\|\|.\)", re.DOTALL) 120 | flag_code = pattern_jscode.findall(jsCodeFunc) 121 | if flag_code: 122 | jsCodeFunc = jsCodeFunc.replace("({}[%s]||%s)" % (flag_code[0], flag_code[0]), flag_code[0]) 123 | pattern1 = re.compile(r"\{(.*?)\:") 124 | pattern2 = re.compile(r"\,(.*?)\:") 125 | nameList1 = pattern1.findall(jsCode) 126 | nameList2 = pattern2.findall(jsCode) 127 | nameList = nameList1 + nameList2 128 | nameList = list(set(nameList)) 129 | Func = js2py.eval_js(jsCodeFunc) 130 | for name in nameList: 131 | if "\"" in name: 132 | name = name.replace("\"", "") 133 | content = Func(name) 134 | if "undefined" not in content: 135 | urllist.append(content) 136 | return list(set(urllist)) 137 | 138 | # 清空phantomjs日志 139 | def rmphantomjslog(self): 140 | # 清楚日志文件 141 | try: 142 | os.remove(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'tmp', 'ghostdriver.log')) 143 | except Exception as e: 144 | pass 145 | 146 | # 清空日志文件 147 | def rmlogfile(self, filepath): 148 | pathDir = os.listdir(filepath) 149 | for allDir in pathDir: 150 | if allDir.endswith(".txt"): 151 | try: 152 | os.remove(os.path.join(filepath, allDir)) 153 | except Exception as e: 154 | pass -------------------------------------------------------------------------------- /lib/common/regular.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import re, os 5 | from lib.common.utils import Utils 6 | from lib.common.Printlog import Printlog 7 | from urllib.parse import urlparse 8 | from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED 9 | 10 | class Regular(): 11 | 12 | def __init__(self): 13 | self.log = "" 14 | 15 | # 从js内容提取URL。返回链接列表：js_url[] 16 | def Extract_URL(self, Js_content): 17 | pattern_raw = r""" 18 | (?:"|') # Start newline delimiter 19 | ( 20 | ((?:[a-zA-Z]{1,10}://|//) # Match a scheme [a-Z]*1-10 or // 21 | [^"'/]{1,}\. # Match a domainname (any character + dot) 22 | [a-zA-Z]{2,}[^"']{0,}) # The domainextension and/or path 23 | | 24 | ((?:/|\.\./|\./) # Start with /,../,./ 25 | [^"'><,;| *()(%%$^/\\\[\]] # Next character can't be... 26 | [^"'><,;|()]{1,}) # Rest of the characters can't be 27 | | 28 | ([a-zA-Z0-9_\-/]{1,}/ # Relative endpoint with / 29 | [a-zA-Z0-9_\-/]{1,} # Resource name 30 | \.(?:[a-zA-Z]{1,4}|action) # Rest + extension (length 1-4 or action) 31 | (?:[\?|#][^"|']{0,}|)) # ? or # mark with parameters 32 | | 33 | ([a-zA-Z0-9_\-/]{1,}/ # REST API (no extension) with / 34 | [a-zA-Z0-9_\-/]{3,} # Proper REST endpoints usually have 3+ chars 35 | (?:[\?|#][^"|']{0,}|)) # ? or # mark with parameters 36 | | 37 | ([a-zA-Z0-9_\-]{1,} # filename 38 | \.(?:php|asp|aspx|jsp|json| 39 | action|html|js|txt|xml) # . + extension 40 | (?:[\?|#][^"|']{0,}|)) # ? or # mark with parameters 41 | ) 42 | (?:"|') # End newline delimiter 43 | """ 44 | pattern = re.compile(pattern_raw, re.VERBOSE) 45 | result = re.finditer(pattern, str(Js_content)) 46 | if result == None: 47 | return None 48 | js_url = [] 49 | for match in result: 50 | if match.group() not in js_url: 51 | js_url.append(match.group().strip('"').strip("'")) 52 | 53 | rule = r"['\"]+[0-9a-zA-Z-:\./?=]{4,}['\"]+" 54 | r = re.compile(rule) 55 | result = r.findall(str(Js_content)) 56 | for i in result: 57 | if "." in i: 58 | if "/" in i: 59 | js_url.append(i.strip('"').strip("'")) 60 | js_url = Utils().unique(js_url) 61 | return js_url 62 | 63 | # 在所有的urls中提取出目标站的子域名 64 | def find_subdomain(self, domainlog="", urls=[], mainurl="", domain=""): 65 | 66 | self.log = Printlog(mainurl).get_logger() 67 | 68 | self.log.info("{} 【{}】 子域名提取规则模块加载完成".format(Utils().tellTime(), domainlog)) 69 | 70 | if urlparse(domain).netloc: 71 | fname = os.path.join(mainurl, str(urlparse(domain).netloc).replace(":", "_") + "_url_list.txt") 72 | else: 73 | fname = os.path.join(mainurl, str(urlparse(domain).path).replace(":", "_") + "_url_list.txt") 74 | pattern = re.compile( 75 | r'^(([a-zA-Z]{1})|([a-zA-Z]{1}[a-zA-Z]{1})|' 76 | r'([a-zA-Z]{1}[0-9]{1})|([0-9]{1}[a-zA-Z]{1})|' 77 | r'([a-zA-Z0-9][-_.a-zA-Z0-9]{0,61}[a-zA-Z0-9]))\.' 78 | r'([a-zA-Z]{2,13}|[a-zA-Z0-9-]{2,30}.[a-zA-Z]{2,3})$' 79 | ) 80 | miandomain = urlparse(mainurl).netloc 81 | positions = Utils().find_last(domain, ".") 82 | if len(positions) > 1: 83 | miandomain = domain[positions[-2] + 1:].replace("/", "") 84 | subdomains = [] 85 | for url in urls: 86 | suburl = urlparse(url) 87 | subdomain = suburl.netloc 88 | if subdomain.strip() == "": 89 | continue 90 | if miandomain in subdomain: 91 | if subdomain not in subdomains: 92 | if Utils().White_list_domain(subdomain): 93 | if pattern.match(subdomain): 94 | if Utils().filter_content(subdomain): 95 | subdomains.append(subdomain) 96 | self.save_result(fname, subdomain) 97 | else: 98 | check_ip = re.compile('(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5]):(6[0-5]{2}[0-3][0-5]|[1-5]\d{4}|[1-9]\d{1,3}|[0-9])|(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])') 99 | if check_ip.match(subdomain): 100 | if Utils().filter_content(subdomain): 101 | subdomains.append(subdomain) 102 | self.save_result(fname, subdomain) 103 | subdomains = list(set(subdomains)) 104 | new_subdomains = [] 105 | for i in range(len(subdomains)): 106 | new_subdomains.append([i, subdomains[i]]) 107 | 108 | # 子域名表格方式输出 109 | Utils().gettable(["序号", "子域名"], "扫描完成：{}".format(domain), new_subdomains, os.path.join(mainurl, "table_result.txt")) 110 | 111 | self.log.info("{} 【{}】 子域名提取结果保存路径：{}".format(Utils().tellTime(), domainlog, os.path.join(mainurl, "table_result.txt"))) 112 | 113 | self.log.info("{} 【{}】 子域名提取规则模块扫描完成".format(Utils().tellTime(), domainlog)) 114 | 115 | return subdomains 116 | 117 | def run_Matching_rules(self, domainlog="", path="", filepath=[], rulesJson=[]): 118 | 119 | self.log = Printlog(path).get_logger() 120 | 121 | self.log.info("{} 【{}】 Hae规则匹配模块加载完成".format(Utils().tellTime(), domainlog)) 122 | 123 | # 创建线程池 124 | pool = ThreadPoolExecutor(20) 125 | 126 | allTask = [pool.submit(self.Matching_rules, url, rulesJson) for url in filepath] 127 | 128 | # 开启异步线程池 129 | wait(allTask, return_when=ALL_COMPLETED) 130 | 131 | self.log.info("{} 【{}】 Hae规则匹配结果保存路径：{}".format(Utils().tellTime(), domainlog, os.path.join(path, "table_result.txt"))) 132 | 133 | self.log.info("{} 【{}】 Hae规则匹配模块扫描完成".format(Utils().tellTime(), domainlog)) 134 | 135 | # 匹配规则库 136 | def Matching_rules(self, filename="", rulesJson=[]): 137 | try: 138 | fopen = open(filename, 'r', encoding='utf-8') 139 | data = str(fopen.read()) 140 | fopen.close() 141 | array = [] 142 | count = 0 143 | for i in range(len(rulesJson)): 144 | try: 145 | r = re.compile(rulesJson[i]['regex']) 146 | result = r.findall(str(data)) 147 | for ib in range(len(result)): 148 | if Utils().filter_content(result[ib][0]): 149 | array.append([rulesJson[i]['type_name'], rulesJson[i]['name'], result[ib][0]]) 150 | count = count + 1 151 | except Exception as e: 152 | pass 153 | if count != 0: 154 | self.save_result(os.path.join(os.path.dirname(filename), "/result_rules.txt"), "【+】" + filename) 155 | # 子域名表格方式输出 156 | 157 | Utils().gettable(["模块", "名称", "内容"], "路径：{}".format(filename), array, os.path.join(os.path.dirname(filename), "table_result.txt")) 158 | 159 | except Exception as e: 160 | return 161 | 162 | # 写入文件 163 | def save_result(self, filename="", content="", jurisdiction="at"): 164 | fp = open(filename, jurisdiction, encoding='utf-8') 165 | fp.write(content + "\n") 166 | fp.close() -------------------------------------------------------------------------------- /lib/common/toupdate.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import os, requests 5 | from tqdm import * 6 | from lib.common.fileoperation import FileOperation 7 | 8 | class ToUpdate(): 9 | 10 | # 检查版本更新 11 | def get_version(self): 12 | path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'version.txt') 13 | if os.path.exists(path): 14 | try: 15 | with open(path, "rt", encoding="UTF-8") as f: 16 | now_version = f.read().strip() 17 | print("目前版本: \n{}\n".format(now_version)) 18 | version_url = "https://raw.githubusercontent.com/xz-zone/Webpackfind/master/version.txt" 19 | res = requests.get(url=version_url, headers={"User-Agent": FileOperation().uarand()}, timeout=10, verify=False) 20 | if res.status_code == 200: 21 | new_version = res.text.strip() 22 | if now_version == new_version: 23 | tqdm.write("目前版本最新") 24 | else: 25 | add_version = str(str(new_version.replace("\r", "").replace("\n", "")).replace(str(now_version.replace("\n", "")), "")).replace("。", "。\n") 26 | if add_version: 27 | print("目前版本最新") 28 | print("更新内容如下:\n{}".format(add_version)) 29 | print("目前版本非最新，建议及时更新...\n地址: https://github.com/xz-zone/Webpackfind/\n") 30 | else: 31 | print("获取版本信息失败...") 32 | except Exception as e: 33 | print("获取版本信息失败...") 34 | else: 35 | print("目前版本非最新，建议及时更新...\n地址: https://github.com/xz-zone/Webpackfind/") -------------------------------------------------------------------------------- /lib/common/urlrequest.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import requests, chardet, platform, os 5 | from urllib.parse import urlparse 6 | from lib.common.fileoperation import FileOperation 7 | from tqdm import * 8 | from selenium import webdriver 9 | from selenium.webdriver import DesiredCapabilities 10 | from requests.packages import urllib3 11 | from requests.packages.urllib3.exceptions import InsecureRequestWarning 12 | requests.packages.urllib3.disable_warnings(InsecureRequestWarning) 13 | 14 | class UrlRequest(): 15 | 16 | def __init__(self, headers={}): 17 | urllib3.disable_warnings() 18 | self.headers = headers 19 | 20 | # 提取页面源码，返回值为页面源码 21 | def Extract_html(self, url): 22 | try: 23 | re = requests.get(url, headers=self.headers, timeout=10, verify=False, allow_redirects=False) 24 | if re.status_code == 200: 25 | cont = re.content 26 | charset = chardet.detect(cont)['encoding'] 27 | raw = cont.decode(charset) 28 | return raw 29 | else: 30 | return None 31 | except Exception as e: 32 | return None 33 | 34 | # 自动选择协议头 35 | def Auto_select_protocol(self, url): 36 | try: 37 | scheme = urlparse(url).scheme 38 | # 自动判断url地址是否添加http/https协议 39 | if "http" in scheme or "https" in scheme: 40 | return url 41 | else: 42 | if self.Extract_html("http://" + url) == None: 43 | if self.Extract_html("https://" + url) == None: 44 | tqdm.write("【扫描失败】：{}\n".format(str(url))) 45 | return False 46 | else: 47 | url = "https://" + url 48 | else: 49 | url = "http://" + url 50 | return url 51 | except Exception as e: 52 | return False 53 | 54 | # 调用phantomjs模仿浏览器请求 55 | def phantomjs_requests(self, url): 56 | ua = "" 57 | if "user-agent" in self.headers: 58 | ua = self.headers["user-agent"] 59 | if "User-Agent" in self.headers: 60 | ua = self.headers["User-Agent"] 61 | content = "" 62 | js_array = [] 63 | sys = platform.system() 64 | if sys == "Windows": 65 | jspath = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'config', 'phantomjs_windows.exe') 66 | elif sys == "Linux": 67 | jspath = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'config', 'phantomjs_liunx_x64') 68 | else: 69 | return False 70 | try: 71 | desired_capabilities = DesiredCapabilities.PHANTOMJS.copy() 72 | ua = FileOperation().uarand() 73 | for key, value in self.headers.items(): 74 | desired_capabilities['phantomjs.page.customHeaders.{}'.format(key)] = value 75 | desired_capabilities["phantomjs.page.settings.userAgent"] = ua 76 | desired_capabilities["phantomjs.page.settings.resourceTimeout"] = True 77 | desired_capabilities["phantomjs.page.settings.disk-cache"] = True 78 | desired_capabilities["phantomjs.page.settings.loadImages"] = False 79 | driver = webdriver.PhantomJS(executable_path=jspath, desired_capabilities=desired_capabilities, service_args=['--ignore-ssl-errors=true'], service_log_path=os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), 'tmp', 'ghostdriver.log')) 80 | driver.set_page_load_timeout(10) 81 | driver.set_script_timeout(5) 82 | driver.get(url) 83 | content = driver.page_source 84 | if urlparse(driver.current_url).netloc != urlparse(url).netloc: 85 | content = self.Extract_html(url) 86 | else: 87 | # 解决layui模块加载问题 88 | js = """ 89 | var jspath = [] 90 | for(var i=0;i<=document.scripts.length;i++){ 91 | try { 92 | if(document.scripts[i].src != ""){ 93 | jspath.push(document.scripts[i].src) 94 | } 95 | } catch(err){ 96 | break; 97 | } 98 | } 99 | return jspath 100 | """ 101 | js_array = driver.execute_script(js) 102 | return content, js_array 103 | except Exception as e: 104 | content = self.Extract_html(url) 105 | return content, js_array -------------------------------------------------------------------------------- /lib/common/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import time 5 | import re 6 | from prettytable import PrettyTable 7 | from uuid import uuid4 8 | from urllib.parse import urlparse 9 | 10 | class Utils(): 11 | 12 | def __init__(self): 13 | self.White = ["w3.org", "example.com", "purl.org", "microsoft.com", "openxmlformats.org", "purl.oclc.org", "docs.oasis-open.org", "openoffice.org", "raphaeljs.com", "bing.com", "wallstreetcn.com", "mozilla.org", "mozilla.org"] 14 | 15 | # 使用set对列表去重，并保持列表原来顺序 16 | def unique(self, arr): 17 | arr1 = list(set(arr)) 18 | arr1.sort(key=arr.index) 19 | return arr1 20 | 21 | # 白名单判断是否过滤那个域名 22 | def White_list_domain(self, domain=""): 23 | for t in range(len(self.White)): 24 | if self.White[t] in domain: 25 | return False 26 | return True 27 | 28 | # 返回一个列表，内容为目标字符在字符串中的位置 [3, 7, 10] 29 | def find_last(self, string, str): 30 | positions = [] 31 | last_position = -1 32 | while True: 33 | position = string.find(str, last_position + 1) 34 | if position == -1: break 35 | last_position = position 36 | positions.append(position) 37 | return positions 38 | 39 | # 随机生成uuid 40 | def random_uuid(self): 41 | # 声明全局uuid 42 | return str(uuid4()).split('-')[-1] 43 | 44 | # 表格输出 45 | def gettable(self, field_names=[], title="", array=[], path=""): 46 | if len(array) != 0: 47 | tb = PrettyTable(align="l", header=True, padding_width=5, field_names=field_names, title=title) 48 | count = 0 49 | for i in range(len(array)): 50 | count = count + 1 51 | tb.add_row(array[i]) 52 | 53 | if path != "": 54 | self.save_result(path, tb.get_string()) 55 | 56 | # 写入文件 57 | def save_result(self, filename="", content="", jurisdiction="at"): 58 | fp = open(filename, jurisdiction, encoding='utf-8') 59 | fp.write(content + "\n") 60 | fp.close() 61 | 62 | # 时间输出 63 | def tellTime(self): 64 | localtime = "[" + str(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time()))) + "] " 65 | return localtime 66 | 67 | # 判断数组中是否存在某个值 68 | def is_array_value(self, array=[], value=""): 69 | if value == "": 70 | return False 71 | for t in range(len(array)): 72 | if array[t] in value: 73 | return False 74 | return True 75 | 76 | # 获取文件名 77 | def get_filename(self, url_str): 78 | url = urlparse(url_str) 79 | i = len(url.path) - 1 80 | while i > 0: 81 | if url.path[i] == '/': 82 | break 83 | i = i - 1 84 | filename = url.path[i+1:len(url.path)] 85 | if not filename.strip(): 86 | return False 87 | return filename 88 | 89 | # headers头部信息处理 90 | def handle_headers(self, primaryheaders={}, new_headers=""): 91 | new_headers = new_headers.split("\\n") 92 | while "" in new_headers: 93 | new_headers.remove("") 94 | for i in new_headers: 95 | if ": " not in i: 96 | break 97 | primaryheaders[i.split(": ")[0]] = i.split(": ")[1] 98 | return primaryheaders 99 | 100 | # 过滤文件内容 101 | def filter_content(self, content=""): 102 | if len(content.strip()) > 2: 103 | if re.match(r"^./(.*)", content.strip(), re.M|re.I) == None: 104 | if re.match(r"^/(.*).js$", content.strip(), re.M|re.I) == None: 105 | filter = ["DD/MM/YYYY", "MM/DD/YYYY", "MM/D/YYYY", "YYYY/MM/DD", "YYYY/MM/DD", "YYYY/M/D", "DD/M/YYYY", "application/x-www-form-urlencoded", "multipart/form-data", "application/xml", "application/json", "/./", "image/webp", "application/octet-stream", "text/html", "image/jpeg", "image/gif", "image/png", "text/plain", "text/xml", "static/js", "text/css", "static/css", "static/js/", "static/css/", "http://www.w3.org/2000/svg", "http://www.w3.org/1999/xlink", "http://www.w3.org/1998/Math/MathML", "http://www.w3.org/TR/REC-html40", "http://www.w3.org/2000/xmlns/", "http://www.w3.org/XML/1998/namespace"] 106 | if content.strip() not in filter: 107 | return True 108 | return False -------------------------------------------------------------------------------- /lib/vuln/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xz-zone/Webpackfind/ad573bdeeea44cacd464622c34d71db36403ae67/lib/vuln/__init__.py -------------------------------------------------------------------------------- /lib/vuln/unauthorized.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import re, requests, chardet, os 5 | from lib.common.utils import Utils 6 | from lib.common.regular import Regular 7 | from requests.packages import urllib3 8 | from urllib.parse import urlparse 9 | from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED 10 | from requests.packages.urllib3.exceptions import InsecureRequestWarning 11 | requests.packages.urllib3.disable_warnings(InsecureRequestWarning) 12 | from lib.common.Printlog import Printlog 13 | 14 | 15 | # 未授权检测 16 | class unauthorized(): 17 | 18 | def __init__(self, domainlog="", headers={}, domain="", rulesJson=[], path=""): 19 | urllib3.disable_warnings() 20 | self.headers = headers 21 | self.rulesJson = rulesJson 22 | self.domain = domain 23 | self.path = path 24 | self.api_result = [] 25 | self.rules_result = [] 26 | self.domainlog = domainlog 27 | 28 | def run(self, filepath=[]): 29 | 30 | self.log = Printlog(self.path).get_logger() 31 | 32 | self.log.info("{} 【{}】 未授权检测模块加载完成".format(Utils().tellTime(), self.domainlog)) 33 | 34 | if len(filepath) > 0: 35 | 36 | # 创建线程池 37 | pool = ThreadPoolExecutor(len(filepath)) 38 | 39 | allTask = [pool.submit(self.api_unauthorized, path) for path in filepath] 40 | 41 | # 开启异步线程池 42 | wait(allTask, return_when=ALL_COMPLETED) 43 | 44 | # 表格方式输出 45 | Utils().gettable(["URL", "标题", "响应状态"], "扫描网站：{}".format(self.domain), self.api_result, os.path.join(self.path, 'table_result.txt')) 46 | 47 | Utils().gettable(["模块", "名称", "URL", "内容"], "扫描网站：{}".format(self.domain), self.rules_result, os.path.join(self.path, 'table_result.txt')) 48 | 49 | self.log.info("{} 【{}】 未授权检测模块结果保存路径：{}".format(Utils().tellTime(), self.domainlog, os.path.join(self.path, 'table_result.txt'))) 50 | 51 | # log日志输出 52 | self.log.info("{} 【{}】 未授权检测模块扫描完成".format(Utils().tellTime(), self.domainlog)) 53 | 54 | # API未授权探测并检测接口标题 55 | def api_unauthorized(self, filename=""): 56 | try: 57 | fopen = open(filename, 'r', encoding='utf-8') 58 | data = str(fopen.read()) 59 | fopen.close() 60 | data = Regular().Extract_URL(data) 61 | if len(data) != 0: 62 | for i in range(len(data)): 63 | data[i] = data[i].replace('./', '/').replace('\\', '') 64 | if data[i][0:2] != "//" and data[i][0] == "/": 65 | try: 66 | if Utils().filter_content(data[i].replace('', '')): 67 | url = self.domain + data[i].replace('', '') 68 | status_code = 200 69 | info = requests.get(url, headers=self.headers, timeout=10, verify=False, allow_redirects=False) 70 | try: 71 | cont = info.content 72 | charset = chardet.detect(info)['encoding'] 73 | html_doc = cont.decode(charset) 74 | title = re.findall('(.+)', html_doc) 75 | title = str(title[0]) 76 | status_code = info.status_code 77 | except Exception as e: 78 | title = "未识别" 79 | if info.status_code == 404: 80 | url = urlparse(self.domain).scheme + "://" + urlparse(self.domain).netloc + "/" + data[i].replace('', '') 81 | info1 = requests.get(url, headers=self.headers, timeout=10, verify=False, allow_redirects=False) 82 | try: 83 | cont = info1.content 84 | charset = chardet.detect(info1)['encoding'] 85 | html_doc = cont.decode(charset) 86 | title = re.findall('(.+)', html_doc) 87 | title = str(title[0]) 88 | status_code = info1.status_code 89 | except Exception as e: 90 | title = "未识别" 91 | self.api_result.append([str(url), title, str(status_code)]) 92 | for i in range(len(self.rulesJson)): 93 | try: 94 | r = re.compile(self.rulesJson[i]['regex']) 95 | result = r.findall(str(info.text)) 96 | for ib in range(len(result)): 97 | if Utils().filter_content(data[i].replace('', '')) and Utils().filter_content(result[ib][0]): 98 | self.rules_result.append([self.rulesJson[i]['type_name'], self.rulesJson[i]['name'], str(url), result[ib][0]]) 99 | except Exception as e: 100 | pass 101 | except Exception as e: 102 | pass 103 | except Exception as e: 104 | pass 105 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | from lib.common.cmdline import CommandLines 5 | from lib.Controller import Controller 6 | 7 | class webpackfind(): 8 | def __init__(self, options): 9 | self.options = options 10 | 11 | def start(self): 12 | t = Controller(self.options) 13 | t.start() 14 | 15 | if __name__ == '__main__': 16 | cmd = CommandLines().cmd() 17 | webpackfind(cmd).start() -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | selenium==2.48.0 2 | jsbeautifier==1.14.0 3 | bs4==0.0.1 4 | idna<2.9 5 | requests==2.21.0 6 | prettytable==3.3.0 7 | PyYaml==6.0 8 | tqdm==4.64.0 9 | Js2Py==0.71 -------------------------------------------------------------------------------- /version.txt: -------------------------------------------------------------------------------- 1 | 2021-08-17 初始版本提交。 2 | 2021-08-18 版本整体修改为面向对象编程，自动识别是否存在其他子域名。 3 | 2021-08-19 修复build.sh 不能执行问题。 4 | 2021-08-23 修复了manifest js无法读取遍历BUG，感谢 chhyx2 师傅提交的bug。 5 | 2021-08-24 优化细节,解决windows生成目录问题。 6 | 2021-08-26 优化细节,解决获取js路径问题,添加读取文本url地址进行扫描js。 7 | 2021-09-02 优化细节,解决PhantomJS给拦截问题。 8 | 2021-09-03 优化细节,解决提取域名正则问题。 9 | 2021-09-07 优化细节,解决目录名称更好查找，添加自动判断是否更新。 10 | 2021-09-22 优化细节,解决首页存在webpackjs代码问题。 11 | 2022-07-21 优化细节,解决本地js代码文件未格式化问题,优化加更新参数才检查更新。 12 | 2022-08-22 重构核心代码,兼容更多webpack版本问题。 13 | 2022-08-23 优化细节,优化批量扫描,解决写入路径问题,增加Cookies认证遍历读取。 14 | 2022-08-24 优化phantomjs文件过大问题,优化提示文字,增加协议识别模块。 15 | 2022-08-25 优化细节,增加批量扫描单线程改多线程。 16 | 2022-08-27 优化线程池问题,增加“HaE”规则库。 17 | 2022-08-30 增加进度条,自动化识别路径Url。 18 | 2022-08-31 优化线程池问题,压缩运行时间。 19 | 2022-09-01 优化整体压缩美化代码。 20 | 2022-09-06 优化匹配正则,输出文件。 21 | 2022-09-09 增加自定义herder头部。 22 | 2022-09-19 优化匹配正则。 23 | 2022-09-23 优化细节。 24 | 2022-09-28 优化细节。 25 | 2022-11-28 优化细节，匹配更多api接口。 --------------------------------------------------------------------------------