├── .github └── FUNDING.yml ├── list.txt ├── psfuzz_example.md ├── config.json ├── LICENSE ├── lists ├── 403header_bypass.txt ├── 403bypass.txt └── useragents.txt ├── README.md └── main.go /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | patreon: proviesec 4 | -------------------------------------------------------------------------------- /list.txt: -------------------------------------------------------------------------------- 1 | htaccess 2 | admin 3 | wp-admin 4 | .git 5 | error 6 | config.xml 7 | config.txt 8 | backup.bz 9 | backup.zip 10 | readme.html 11 | log 12 | public 13 | static 14 | vendor 15 | config 16 | robots.txt 17 | configuration 18 | git 19 | docker 20 | monitoring 21 | api 22 | ws_ftp.ini 23 | -------------------------------------------------------------------------------- /psfuzz_example.md: -------------------------------------------------------------------------------- 1 | ``` go run main.go -url https://www.google.com/ -d dir-full.txt -c 2 -o googleoutputfile -s true -fscn 404,301,302 ``` 2 | 3 | ``` go run main.go -url https://www.exmaple.com/ -c 1 -o example -s true -fscn 403,404 -b true ``` 4 | 5 | ``` go run main.go -url https://www.exmaple.com/ ``` 6 | -------------------------------------------------------------------------------- /config.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://example.com", 3 | "dirlist": "default", 4 | "generate_payload": true, 5 | "showStatus": false, 6 | "filterTestLength": true, 7 | "filterPossible404": false, 8 | "onlydomains": false, 9 | "redirect": true, 10 | "recursionLevel": 0, 11 | "bypass": true, 12 | "checkBackslash": false, 13 | "bypassTooManyRequests": true, 14 | "concurrency": 5, 15 | "output": "output.txt", 16 | "helpmode": false, 17 | "filterWrongStatus200": true, 18 | "filterWrongSubdomain": false 19 | } 20 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Proviesec 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /lists/403header_bypass.txt: -------------------------------------------------------------------------------- 1 | Base-Url: 127.0.0.1 2 | Client-IP: 127.0.0.1 3 | Http-Url: 127.0.0.1 4 | Proxy-Host: 127.0.0.1 5 | Proxy-Url: 127.0.0.1 6 | Real-Ip: 127.0.0.1 7 | Redirect: 127.0.0.1 8 | Referer: 127.0.0.1 9 | Referrer: 127.0.0.1 10 | Refferer: 127.0.0.1 11 | Request-Uri: 127.0.0.1 12 | Uri: 127.0.0.1 13 | Url: 127.0.0.1 14 | X-Client-IP: 127.0.0.1 15 | X-Custom-IP-Authorization: 127.0.0.1 16 | X-Forward-For: 127.0.0.1 17 | X-Forwarded-By: 127.0.0.1 18 | X-Forwarded-For-Original: 127.0.0.1 19 | X-Forwarded-For: 127.0.0.1 20 | X-Forwarded-Host: 127.0.0.1 21 | X-Forwarded-Port: 443 22 | X-Forwarded-Port: 4443 23 | X-Forwarded-Port: 80 24 | X-Forwarded-Port: 8080 25 | X-Forwarded-Port: 8443 26 | X-Forwarded-Scheme: http 27 | X-Forwarded-Scheme: https 28 | X-Forwarded-Server: 127.0.0.1 29 | X-Forwarded: 127.0.0.1 30 | X-Forwarder-For: 127.0.0.1 31 | X-Host: 127.0.0.1 32 | X-Http-Destinationurl: 127.0.0.1 33 | X-Http-Host-Override: 127.0.0.1 34 | X-Original-Remote-Addr: 127.0.0.1 35 | X-Original-Url: 127.0.0.1 36 | X-Originating-IP: 127.0.0.1 37 | X-Proxy-Url: 127.0.0.1 38 | X-Real-Ip: 127.0.0.1 39 | X-Remote-Addr: 127.0.0.1 40 | X-Remote-IP: 127.0.0.1 41 | X-Rewrite-Url: 127.0.0.1 42 | X-True-IP: 127.0.0.1 43 | -------------------------------------------------------------------------------- /lists/403bypass.txt: -------------------------------------------------------------------------------- 1 | # 2 | #? 3 | %09 4 | %09%3b 5 | %09.. 6 | %09; 7 | %20 8 | %23 9 | %23%3f 10 | %252f%252f 11 | %252f/ 12 | %2e%2e 13 | %2e%2e/ 14 | %2f 15 | %2f%20%23 16 | %2f%23 17 | %2f%2f 18 | %2f%3b%2f 19 | %2f%3b%2f%2f 20 | %2f%3f 21 | %2f%3f/ 22 | %2f/ 23 | %2f;? 24 | %2f?; 25 | %3b 26 | %3b%09 27 | %3b%2f%2e%2e 28 | %3b%2f%2e%2e%2f%2e%2e%2f%2f 29 | %3b%2f%2e. 30 | %3b%2f.. 31 | %3b/%2e%2e/..%2f%2f 32 | %3b/%2e. 33 | %3b/%2f%2f../ 34 | %3b/.. 35 | %3b//%2f../ 36 | %3f%23 37 | %3f%3f 38 | %3f.php 39 | .. 40 | ..%00/ 41 | ..%00/; 42 | ..%00;/ 43 | ..%09 44 | ..%0d/ 45 | ..%0d/; 46 | ..%0d;/ 47 | ..%5c/ 48 | ..%ff/ 49 | ..%ff/; 50 | ..%ff;/ 51 | ../ 52 | ..;%00/ 53 | ..;%0d/ 54 | ..;%ff/ 55 | ..;\ 56 | ..;\; 57 | ..\ 58 | ..\; 59 | .html 60 | .json 61 | / 62 | /# 63 | /%20 64 | /%20# 65 | /%20%23 66 | /%23 67 | /%252e%252e%252f/ 68 | /%252e%252e%253b/ 69 | /%252e%252f/ 70 | /%252e%253b/ 71 | /%252e/ 72 | /%252f 73 | /%2e%2e 74 | /%2e%2e%2f/ 75 | /%2e%2e%3b/ 76 | /%2e%2e/ 77 | /%2e%2f/ 78 | /%2e%3b/ 79 | /%2e%3b// 80 | /%2e/ 81 | /%2e// 82 | /%2f 83 | /%3b/ 84 | /.. 85 | /..%2f 86 | /..%2f..%2f 87 | /..%2f..%2f..%2f 88 | /../ 89 | /../../ 90 | /../../../ 91 | /../../..// 92 | /../..// 93 | /../..//../ 94 | /../..;/ 95 | /.././../ 96 | /../.;/../ 97 | /..// 98 | /..//../ 99 | /..//../../ 100 | /..//..;/ 101 | /../;/ 102 | /../;/../ 103 | /..;%2f 104 | /..;%2f..;%2f 105 | /..;%2f..;%2f..;%2f 106 | /..;/ 107 | /..;/../ 108 | /..;/..;/ 109 | /..;// 110 | /..;//../ 111 | /..;//..;/ 112 | /..;/;/ 113 | /..;/;/..;/ 114 | /./ 115 | /.// 116 | /.;/ 117 | /.;// 118 | // 119 | //.. 120 | //../../ 121 | //..; 122 | //./ 123 | //.;/ 124 | ///.. 125 | ///../ 126 | ///..// 127 | ///..; 128 | ///..;/ 129 | ///..;// 130 | //;/ 131 | /;/ 132 | /;// 133 | /;? 134 | /;x 135 | /;x/ 136 | /? 137 | /?; 138 | /x/../ 139 | /x/..// 140 | /x/../;/ 141 | /x/..;/ 142 | /x/..;// 143 | /x/..;/;/ 144 | /x//../ 145 | /x//..;/ 146 | /x/;/../ 147 | /x/;/..;/ 148 | ; 149 | ;%09 150 | ;%09.. 151 | ;%09..; 152 | ;%09; 153 | ;%2F.. 154 | ;%2f%2e%2e 155 | ;%2f%2e%2e%2f%2e%2e%2f%2f 156 | ;%2f%2f/../ 157 | ;%2f.. 158 | ;%2f..%2f%2e%2e%2f%2f 159 | ;%2f..%2f..%2f%2f 160 | ;%2f..%2f/ 161 | ;%2f..%2f/..%2f 162 | ;%2f..%2f/../ 163 | ;%2f../%2f..%2f 164 | ;%2f../%2f../ 165 | ;%2f..//..%2f 166 | ;%2f..//../ 167 | ;%2f../// 168 | ;%2f..///; 169 | ;%2f..//;/ 170 | ;%2f..//;/; 171 | ;%2f../;// 172 | ;%2f../;/;/ 173 | ;%2f../;/;/; 174 | ;%2f..;/// 175 | ;%2f..;//;/ 176 | ;%2f..;/;// 177 | ;%2f/%2f../ 178 | ;%2f//..%2f 179 | ;%2f//../ 180 | ;%2f//..;/ 181 | ;%2f/;/../ 182 | ;%2f/;/..;/ 183 | ;%2f;//../ 184 | ;%2f;/;/..;/ 185 | ;/%2e%2e 186 | ;/%2e%2e%2f%2f 187 | ;/%2e%2e%2f/ 188 | ;/%2e%2e/ 189 | ;/%2e. 190 | ;/%2f%2f../ 191 | ;/%2f/..%2f 192 | ;/%2f/../ 193 | ;/.%2e 194 | ;/.%2e/%2e%2e/%2f 195 | ;/.. 196 | ;/..%2f 197 | ;/..%2f%2f../ 198 | ;/..%2f..%2f 199 | ;/..%2f/ 200 | ;/..%2f// 201 | ;/../ 202 | ;/../%2f/ 203 | ;/../../ 204 | ;/../..// 205 | ;/.././../ 206 | ;/../.;/../ 207 | ;/..// 208 | ;/..//%2e%2e/ 209 | ;/..//%2f 210 | ;/..//../ 211 | ;/../// 212 | ;/../;/ 213 | ;/../;/../ 214 | ;/..; 215 | ;/.;. 216 | ;//%2f../ 217 | ;//.. 218 | ;//../../ 219 | ;///.. 220 | ;///../ 221 | ;///..// 222 | ;? 223 | ;x 224 | ;x/ 225 | ;x; 226 | ? 227 | ?# 228 | ?.php 229 | ?; 230 | ?? 231 | /// 232 | /%2f/ 233 | //%2f 234 | %2f/%2f 235 | %2f%2f%2f 236 | %2f// 237 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![PSFuzz](https://user-images.githubusercontent.com/6010786/176360134-adc6d195-60b0-4628-af06-b6b42afaffae.png) 2 | ![](https://us-central1-progress-markdown.cloudfunctions.net/progress/85) 3 | # PSFuzz - ProvieSec Fuzz Scanner - Web path/file discovery 4 | [![License](https://img.shields.io/badge/license-MIT-_red.svg)](https://opensource.org/licenses/MIT) 5 | [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/dwisiswant0/go-dork/issues) 6 | [![Twitter](https://img.shields.io/twitter/follow/proviesec?label=Follow)](https://twitter.com/proviesec) 7 | 8 | PSFuzz - ProvieSec Fuzz Scanner is a web path and file discovery tool designed to scan web directories, files, and other endpoints efficiently. It allows security professionals to explore potential vulnerabilities through fuzzing by sending a wide range of requests to various URLs, using both predefined and dynamically generated payloads. 9 | 10 | The tool supports custom configurations, such as recursive directory scanning, status code filtering, payload generation, request throttling, and more. PSFuzz is built with high concurrency to handle multiple requests in parallel while also allowing rate limiting. 11 | 12 | It includes several bypass techniques for common restrictions, and it is flexible enough to detect possible false positives, such as 404-like responses. The tool outputs findings to a file or directly to the console for further analysis. 13 | 14 | 15 | Proviesec logo 16 | 17 | Buy Me A Coffee 18 | 19 | Table of Contents 20 | ------------ 21 | * [Introduction](#introduction) 22 | * [Installation](#installation--usage) 23 | * [Wordlists](#wordlists-important) 24 | * [Options](#options) 25 | * [How to use](#how-to-use) 26 | * [Todos](#todos) 27 | * [Tips](#tips) 28 | * [License](#license) 29 | 30 | Introduction 31 | ------------ 32 | 33 | :star: Star us on GitHub — it motivates a lot! :star: 34 | 35 | Web path discovery 36 | 37 | Discover with ProSecFuzz hidden files and directories on a web server. 38 | 39 | ## Disclaimer: DONT BE A JERK! 40 | Needless to mention, please use this tool very very carefully. The authors won't be responsible for any consequences. 41 | 42 | # Installation & Usage 43 | 44 | ```go get https://github.com/Proviesec/PSFuzz``` 45 | ```go install github.com/Proviesec/PSFuzz@latest``` 46 | 47 | Wordlists 48 | --------------- 49 | **Summary:** 50 | - the Wordlist is a text file, each line is a path. 51 | - Here you get suitable lists: https://github.com/Proviesec/directory-payload-list 52 | 53 | Options 54 | --------------- 55 | **Required** 56 | * `-u`/`-url` Example: `-u https://www.google.com` 57 | 58 | **Optional** 59 | * `-o`/`-output` Example: `-o google_output` Default: output.txt 60 | * `-d`/`-dirlist` Example: `-d list.txt` Default is https://raw.githubusercontent.com/Proviesec/directory-payload-list/main/directory-full-list.txt 61 | * `-s`/`-status` Example: `-s true` Default:false only Status Code 200 62 | * `-c`/`-concurrency` Example: `-c 5 ` 63 | * `-fscn`/`-filterStatusCodeNot` Example: `-fscn 404` Don't show response status code 404 64 | * `-fsc`/`-filterStatusCode` Example: `-fsc 200,301` Show only response status code 200 and 301 65 | * `-fl`/`-filterLength` Example: `-fl 122,1234,1235,1236` or `-fl 122,1234-1236` Show only the response with this length (or length range) 66 | * `-fln`/`-filterLengthNot` Example: `-fln 122,1234,1235,1236` or `-fln 122,1234-1236` Show not this response with this length (or length range) 67 | * `-fm`/`-filterMatchWord` Example: `-fm admin` 68 | * `-rah`/`-requestAddHeader` Example: `-rah Host:127.0.0.1` 69 | * `-tr`/`-throttleRate` Example: `-tr 10` -> max requests per second 70 | * `-b`/`-bypass` Example: `-b true` -> bypass status code: 401,402,403 71 | * `-g`/`-generate_payload` Example: `-g 100` -> generate a,aa,ab,abc,aaa,abb,bbc 72 | * `-od` /`-onlydomains` Example: `-od true` Show only domains in the outputfile (no status code) 73 | * checkBackslash Example -checkBackslash true 74 | * `-t` /`-filterTestLength` Example: `-t true` make a test request and check if any other request has the same length, if yes, then skip the result for this request 75 | * `fws` / `filterWrongStatus200` Example: `-fws true` - Don´t show: in title: "Access Gateway", "Not Found", "Error"/"ERROR", "403", "Bad Request" ,"Forbidden", "500", "Internal Server Error" and body length <= 1 76 | 77 | 78 | # Attack Configuration / Patterns 79 | 80 | # Response Analysis 81 | 82 | # Example 83 | ``` 84 | go run main.go -url https://www.google.com/ -d dir-full.txt -c 2 -o testest -s true -fscn 404,301,302 85 | 86 | go run main.go -url https://www.google.com/ -d dir-full.txt -c 2 -o googletest -s true -fl 122,1565-1569 -fln 1566-1568 87 | ``` 88 | 89 | ![image](https://user-images.githubusercontent.com/6010786/180856727-0d8791af-6076-417c-94a8-05bc786b5a4d.png) 90 | 91 | ![image](https://user-images.githubusercontent.com/6010786/180856025-6922fc14-9baf-4ba7-b5c0-6d2073c5b0c2.png) 92 | 93 | # Todos 94 | 95 | 96 | ## General 97 | - [x] Multi requests 98 | - [x] Optional param output 99 | - [x] check https or http 100 | - [x] Logo and Version output 101 | - [ ] Marketing 102 | - [ ] Tryhackme room - Link: 103 | - [ ] Youtube Video 104 | - [x] help mode (-h) 105 | - [x] check backslah 106 | - [ ] optional config file 107 | - [x] load config 108 | - [ ] save config 109 | - [x] json file 110 | - [ ] config for "dont show" in title/body 111 | - [ ] Proxy 112 | - [x] throttle 113 | - [x] detect "too many requests" 114 | - [ ] Output 115 | - [x] TXT 116 | - [x] CSV 117 | - [ ] Json 118 | - [ ] HTML 119 | - [ ] Send Mail? 120 | - [x] Progress bar 121 | - [ ] list of sites 122 | - [ ] Parameter 123 | - [ ] random payload generator 124 | - [ ] choice of dirlist from proviesec github repo 125 | - [ ] subdomain list from proviesec github repo 126 | - [ ] Port List 127 | - [x] Length 128 | - [x] Length range show and not show 129 | - [x] Response Status List show 130 | - [x] Response Status Range show 131 | - [x] Response Status List not show 132 | - [x] Response Status Range not show 133 | - [x] Filter content type 134 | - [x] Words match list title/page 135 | - [x] Set Optional Header 136 | - [ ] scan subdirs with depth (list: admin/public/static) 137 | - [ ] exclude subdirs (list: js/img) 138 | - [ ] add default-extensions (yml,php,aspx,jsp,html,js) 139 | - [ ] lowercase 140 | - [ ] uppercase 141 | - [ ] Min response-size 142 | - [ ] Max response-size 143 | - [ ] Set request Timeout 144 | - [ ] Add Cookies 145 | - [ ] quite Mode 146 | - [ ] random user-agent 147 | - [x] show only the urls 148 | - [x] add user agent 149 | - [ ] username /password basic Auth 150 | 151 | ## Attack 152 | - [x] make GET requests 153 | - [ ] make put request 154 | - [ ] make POST requests 155 | - [ ] try PUT/DELETE/PATCH 156 | - [x] Wordlist txt parameter 157 | - [x] Wildcard parameter 158 | - [ ] List of URLs 159 | - [ ] depth by dir 160 | - [ ] Word list 161 | - [ ] Automatic Word list for any file html,txt, php.. 162 | - [ ] payload generator, include, start or end with specific word and max length 163 | - [ ] file ending as parameter list 164 | - [ ] get list from any url 165 | - [x] get list from proviesec github account default 166 | - [ ] multiple word lists 167 | - [ ] Crlf scan 168 | - [ ] open redirect scan 169 | - [ ] fuzzing parameter (from a-z) 170 | - [ ] fuzzing http verbs 171 | - [ ] Wordlist formats, upper lower 172 | 173 | ## Response Analysis 174 | - [x] show response status 175 | - [x] count words 176 | - [ ] show response time 177 | - [ ] show lines 178 | - [ ] dump the response in files 179 | - [ ] analyse the response with AI 180 | - [ ] Fingerprint Software (Wordpress/php/java/Apache/nginx etc.) 181 | - [ ] CORS analyse 182 | - [ ] bypass 183 | - [x] 403 Bypass, config 184 | - [ ] Status bypass 185 | - [ ] Words match list title/page/header 186 | - [ ] output the match line 187 | - [x] Show positiv false: status 200, but title 404 188 | - [ ] Show possible block response, after x requests "403 or too many request" 189 | - [x] Show possible false 200, same length of a random site 190 | - [x] Show confident value, if the folder/file not in the response 191 | - [ ] Intilligence 192 | - [x] Automatically detect false 200 (really 404) 193 | - [ ] too many rediretcs and then restart again, with the exclusion of 194 | - [ ] Show the most unique target 195 | - [x] Show titel of Page 196 | - [x] Show Response Body Length 197 | - [x] filter possibile 404 198 | - [ ] show content type 199 | - [ ] Fingerprint check 200 | - [ ] fuzz Parameter check (normal Response vs. with paramter) 201 | - [ ] show reflected cookie 202 | - [ ] show reflected params 203 | - [ ] show reflected base64 params 204 | - [ ] search for interesting strings 205 | - [ ] compare two scans 206 | - [ ] save scan 207 | - [ ] load scan 208 | - [x] Redirect handler - 301... -> Can be activated via parameter 209 | - [ ] Show Redirect URL 210 | - [ ] Skip Status filter if redirect true (via parameter) 211 | 212 | 213 | # Example 214 | `go run main.go -url https://www.google.com -d list.txt -s true -c 2` 215 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bufio" 5 | "context" 6 | "encoding/json" 7 | "flag" 8 | "fmt" 9 | "io" 10 | "io/ioutil" 11 | "log" 12 | "net/http" 13 | "os" 14 | "regexp" 15 | "strconv" 16 | "strings" 17 | "sync" 18 | "time" 19 | ) 20 | 21 | const MAX_CONCURRENT_JOBS = 2 22 | 23 | type result struct { 24 | sumValue int 25 | multiplyValue int 26 | } 27 | 28 | var mutex = &sync.Mutex{} 29 | 30 | var statuscount = map[string]int{} 31 | 32 | var default_payload_url = "https://raw.githubusercontent.com/Proviesec/directory-payload-list/main/directory-full-list.txt" 33 | var fav_payload_url = "https://raw.githubusercontent.com/Proviesec/directory-files-payload-lists/main/directory-proviesec-favorite-list.txt" 34 | 35 | var url string 36 | var dirlist string 37 | var generate_payload string 38 | var generate_payload_length int 39 | var showStatus string 40 | var redirect string 41 | var recursionLevel int 42 | var bypass string 43 | var bypassTooManyRequests string 44 | var concurrency int 45 | var output string 46 | var onlydomains string 47 | var requestAddHeader string 48 | var requestAddAgent string 49 | var checkBackslash string 50 | var helpmode string 51 | var filterWrongStatus200 string 52 | var filterWrongSubdomain string 53 | var filterStatusCode string 54 | var filterTestLength string 55 | var filterPossible404 string 56 | var filterContentType string 57 | var filterContentTypeList []string 58 | var filterMatchWord string 59 | var filterStatusCodeList []string 60 | var filterStatusNot string 61 | var filterStatusNotList []string 62 | var filterLength string 63 | var filterLengthList []string 64 | var filterMaxLength int 65 | var filterMinLength int 66 | var filterLengthNot string 67 | var filterLengthNotList []string 68 | var lengthcount = map[int]int{} 69 | var testlength int 70 | var configfile string 71 | 72 | type Config struct { 73 | URL string `json:"url"` 74 | Dirlist string `json:"dirlist"` 75 | GeneratePayload bool `json:"generate_payload"` 76 | ShowStatus bool `json:"showStatus"` 77 | FilterTestLength bool `json:"filterTestLength"` 78 | FilterPossible404 bool `json:"filterPossible404"` 79 | OnlyDomains bool `json:"onlydomains"` 80 | Redirect bool `json:"redirect"` 81 | RecursionLevel int `json:"recursionLevel"` 82 | Bypass bool `json:"bypass"` 83 | CheckBackslash bool `json:"checkBackslash"` 84 | BypassTooManyRequests bool `json:"bypassTooManyRequests"` 85 | Concurrency int `json:"concurrency"` 86 | Output string `json:"output"` 87 | Helpmode bool `json:"helpmode"` 88 | FilterWrongStatus200 bool `json:"filterWrongStatus200"` 89 | FilterWrongSubdomain bool `json:"filterWrongSubdomain"` 90 | } 91 | 92 | var throttleRate int 93 | var throttleTicker *time.Ticker 94 | 95 | var ( 96 | requestCount int 97 | startTime time.Time 98 | lastReportedRate int 99 | ) 100 | 101 | func init() { 102 | setupFlags() 103 | loadConfigOverrides() 104 | 105 | filterStatusCodeList = strings.Split(filterStatusCode, ",") 106 | filterStatusNotList = strings.Split(filterStatusNot, ",") 107 | filterContentTypeList = strings.Split(filterContentType, ",") 108 | generate_payload_length, _ = strconv.Atoi(generate_payload) 109 | 110 | if generate_payload_length < 0 || generate_payload_length > 20000 { 111 | generate_payload_length = 20000 112 | } 113 | 114 | filterLengthList = strings.Split(filterLength, ",") 115 | filterLengthNotList = strings.Split(filterLengthNot, ",") 116 | 117 | // Initialize throttle ticker if throttleRate is set 118 | if throttleRate > 0 { 119 | throttleTicker = time.NewTicker(time.Second / time.Duration(throttleRate)) 120 | } 121 | 122 | // If the identified URL has neither http or https infront of it. Create both and scan them. 123 | if !strings.Contains(url, "http://") && !strings.Contains(url, "https://") { 124 | url = "https://" + url 125 | } 126 | // if the URL does not end with a /, add it. 127 | if !strings.HasSuffix(url, "/") && checkBackslash != "true" { 128 | url = url + "/" 129 | } 130 | startTime = time.Now() 131 | requestCount = 0 132 | lastReportedRate = 0 133 | } 134 | 135 | func setupFlags() { 136 | // get config file from the command line 137 | flag.StringVar(&configfile, "configfile", "", "Config file") 138 | flag.StringVar(&configfile, "cf", "", "Config file") 139 | 140 | // get url parameter from name "url" in the command line 141 | flag.StringVar(&url, "url", "", "URL") 142 | flag.StringVar(&url, "u", "", "URL") 143 | 144 | // get directoryList parameter from name "directoryList" in the command line 145 | flag.StringVar(&dirlist, "dirlist", "", "Directory List") 146 | flag.StringVar(&dirlist, "d", "default", "Directory List") 147 | 148 | //get generate_payload parameter from name "generate_payload" in the command line 149 | flag.StringVar(&generate_payload, "generate_payload", "", "Generate Payload") 150 | flag.StringVar(&generate_payload, "g", "", "Generate Payload") 151 | 152 | // get status parameter from the command line 153 | flag.StringVar(&showStatus, "showStatus", "false", "show status") 154 | flag.StringVar(&showStatus, "s", "false", "show status") 155 | 156 | // get testlength parameter from the command line 157 | flag.StringVar(&filterTestLength, "filterTestLength", "false", "filterTestLength") 158 | flag.StringVar(&filterTestLength, "t", "false", "filterTestLength") 159 | 160 | // get filterpossible404 parameter from the command line 161 | flag.StringVar(&filterPossible404, "filterPossible404", "false", "filterPossible404") 162 | flag.StringVar(&filterPossible404, "p404", "false", "filterPossible404") 163 | 164 | // get onlydomains parameter from the command line 165 | flag.StringVar(&onlydomains, "onlydomains", "false", "only domains") 166 | flag.StringVar(&onlydomains, "od", "false", "only domains") 167 | 168 | // get concurrency parameter from the command line 169 | flag.StringVar(&redirect, "redirect", "true", "redirect") 170 | flag.StringVar(&redirect, "r", "true", "redirect") 171 | 172 | // get recursionLevel parameter from the command line 173 | flag.IntVar(&recursionLevel, "recursionLevel", 0, "recursionLevel") 174 | flag.IntVar(&recursionLevel, "rl", 0, "recursionLevel") 175 | 176 | // get bypass parameter from the command line 177 | flag.StringVar(&bypass, "bypass", "false", "bypass") 178 | flag.StringVar(&bypass, "b", "false", "bypass") 179 | 180 | // get checkBackslash parameter from the command line 181 | flag.StringVar(&checkBackslash, "checkBackslash", "false", "checkBackslash") 182 | flag.StringVar(&checkBackslash, "cb", "false", "checkBackslash") 183 | 184 | // get bypassTooManyRequests parameter from the command line 185 | flag.StringVar(&bypassTooManyRequests, "bypassTooManyRequests", "false", "bypassTooManyRequests") 186 | flag.StringVar(&bypassTooManyRequests, "btr", "false", "bypassTooManyRequests") 187 | 188 | // get concurrency parameter from the command line 189 | flag.IntVar(&concurrency, "concurrency", 1, "concurrency") 190 | flag.IntVar(&concurrency, "c", 1, "concurrency") 191 | 192 | // get output parameter from the command line 193 | flag.StringVar(&output, "output", "", "output") 194 | flag.StringVar(&output, "o", "", "output") 195 | 196 | // get helpmode parameter from the command line 197 | flag.StringVar(&helpmode, "helpmode", "false", "helpmode") 198 | flag.StringVar(&helpmode, "h", "false", "helpmode") 199 | 200 | // get filterWrongStatus200 parameter from the command line 201 | flag.StringVar(&filterWrongStatus200, "filterWrongStatus200", "false", "filterWrongStatus200") 202 | flag.StringVar(&filterWrongStatus200, "fws", "false", "filterWrongStatus200") 203 | 204 | //get filterWrongSubdomain parameter from the command line 205 | flag.StringVar(&filterWrongSubdomain, "filterWrongSubdomain", "false", "filterWrongSubdomain") 206 | flag.StringVar(&filterWrongSubdomain, "fwd", "false", "filterWrongSubdomain") 207 | 208 | // get filterContentType parameter from the command line 209 | flag.StringVar(&filterContentType, "filterContentType", "", "filterContentType") 210 | flag.StringVar(&filterContentType, "f", "", "filterContentType") 211 | 212 | // get filterMatchWord parameter from the command line 213 | flag.StringVar(&filterMatchWord, "filterMatchWord", "", "filterMatchWord") 214 | flag.StringVar(&filterMatchWord, "fm", "", "filterMatchWord") 215 | 216 | // get list of show status codes from the command line 217 | flag.StringVar(&filterStatusCode, "filterStatusCode", "", "Show only this status code") 218 | flag.StringVar(&filterStatusCode, "fsc", "", "Show only this status code") 219 | 220 | // get list of status code to be filtered from the command line 221 | flag.StringVar(&filterStatusNot, "filterStatusCodeNot", "", "Show not this status code") 222 | flag.StringVar(&filterStatusNot, "fscn", "", "Show not this status code") 223 | 224 | // get length from the command line 225 | flag.StringVar(&filterLength, "filterLength", "-1", "Show only response length") 226 | flag.StringVar(&filterLength, "fl", "-1", "Show only response length") 227 | 228 | // get not show length from the command line 229 | flag.StringVar(&filterLengthNot, "filterLengthNot", "-1", "Don´t show this response length") 230 | flag.StringVar(&filterLengthNot, "fln", "-1", "Don´t show this response length") 231 | 232 | // get filterMaxLength from the command line 233 | flag.IntVar(&filterMaxLength, "filterMaxLength", -1, "Show only response length less than") 234 | flag.IntVar(&filterMaxLength, "fml", -1, "Show only response length less than") 235 | 236 | // get filterMinLength from the command line 237 | flag.IntVar(&filterMinLength, "filterMinLength", -1, "Show only response length greater than") 238 | flag.IntVar(&filterMinLength, "fmg", -1, "Show only response length greater than") 239 | 240 | // get add header from the command line 241 | flag.StringVar(&requestAddHeader, "requestAddHeader", "", "Add header to request") 242 | flag.StringVar(&requestAddHeader, "rah", "", "Add header to request") 243 | 244 | // get add agent from the command line 245 | flag.StringVar(&requestAddAgent, "requestAddAgent", "", "Add agent to request") 246 | flag.StringVar(&requestAddAgent, "raa", "", "Add agent to request") 247 | 248 | // get throttleRate from the command line 249 | flag.IntVar(&throttleRate, "throttleRate", 0, "Throttle rate (requests per second)") 250 | flag.IntVar(&throttleRate, "tr", 0, "Throttle rate (requests per second)") 251 | 252 | flag.Parse() 253 | } 254 | 255 | func loadConfigOverrides() { 256 | config, err := loadConfig("config.json") 257 | if err != nil { 258 | fmt.Println(err) 259 | } 260 | 261 | if url == "" { 262 | url = config.URL 263 | } 264 | 265 | if dirlist == "default" { 266 | if config.Dirlist == "" { 267 | dirlist = config.Dirlist 268 | } 269 | } 270 | } 271 | 272 | func updateRequestRate() { 273 | requestCount++ 274 | elapsed := time.Since(startTime).Seconds() 275 | if elapsed > 0 { 276 | currentRate := int(float64(requestCount) / elapsed) 277 | 278 | // Report the rate every 5 seconds or if it has significantly changed 279 | if time.Now().Second()%1 == 0 && (lastReportedRate == 0 || abs(currentRate-lastReportedRate) > 10) { 280 | fmt.Printf("\tAktuelle Anfragenrate: %d Anfragen pro Sekunde\n", currentRate) 281 | lastReportedRate = currentRate 282 | } 283 | } 284 | } 285 | 286 | func abs(x int) int { 287 | if x < 0 { 288 | return -x 289 | } 290 | return x 291 | } 292 | 293 | func contains(s []string, str string) bool { 294 | for _, v := range s { 295 | if v == str { 296 | return true 297 | } 298 | } 299 | 300 | return false 301 | } 302 | 303 | func GetResponseDetails(response *http.Response) (string, int, string, int, int) { 304 | // Get the response body as a string 305 | dataInBytes, _ := ioutil.ReadAll(response.Body) 306 | pageContent := string(dataInBytes) 307 | 308 | pageTitle := "" 309 | pageBody := "" 310 | // Find a substr 311 | titleStartIndex := strings.Index(pageContent, "") 312 | if titleStartIndex == -1 { 313 | pageTitle = "No title element found" 314 | } else { 315 | // <title> = length = 7 316 | titleStartIndex += 7 317 | 318 | // Find the index of the closing tag 319 | titleEndIndex := strings.Index(pageContent, "") 320 | if titleEndIndex == -1 { 321 | pageTitle = "No closing tag for title found." 322 | } else { 323 | pageTitle = "Page title:" + string([]byte(pageContent[titleStartIndex:titleEndIndex])) 324 | } 325 | } 326 | 327 | matchWord := "" 328 | if filterMatchWord != "" { 329 | // if filterMatchWord in pageContent then output the 10 postions before and after the matchWord 330 | if strings.Contains(pageContent, filterMatchWord) { 331 | matchWordIndex := strings.Index(pageContent, filterMatchWord) 332 | if matchWordIndex > 10 { 333 | matchWord = pageContent[matchWordIndex-10 : matchWordIndex+10] 334 | } else { 335 | matchWord = pageContent[0 : matchWordIndex+10] 336 | } 337 | } 338 | } 339 | bodyStartIndex := strings.Index(pageContent, "") 340 | if bodyStartIndex != -1 { 341 | // = length = 6 342 | bodyStartIndex += 6 343 | 344 | // Find the index of the closing tag 345 | bodyEndIndex := strings.Index(pageContent, "") 346 | if bodyEndIndex != -1 { 347 | pageBody = string([]byte(pageContent[bodyStartIndex:bodyEndIndex])) 348 | } 349 | } 350 | // Match non-space character sequences. 351 | re := regexp.MustCompile(`[\S]+`) 352 | 353 | // Find all matches and return count. 354 | resultsCount := re.FindAllString(pageContent, -1) 355 | 356 | return pageTitle, len(pageContent), matchWord, len(resultsCount), len(pageBody) 357 | } 358 | 359 | func lineCounter(r io.Reader) (int, error) { 360 | count := 0 361 | scanner := bufio.NewScanner(r) 362 | 363 | for scanner.Scan() { 364 | count++ 365 | } 366 | 367 | return count, scanner.Err() 368 | } 369 | 370 | func createPayload(length int) []byte { 371 | // Create a byte slice of the specified length 372 | payload := make([]byte, length) 373 | // Fill the slice with the sequence of characters "a", "b", "c", etc. 374 | for i := 0; i < length; i++ { 375 | payload[i] = byte('a' + (i % 26)) 376 | } 377 | // Return the payload 378 | 379 | return payload 380 | 381 | } 382 | 383 | func NextAlias(last string) string { 384 | if last == "" { 385 | return "a" 386 | } else if last[len(last)-1] == 'z' { 387 | return last[:len(last)-1] + "aa" 388 | } else { 389 | return last[:len(last)-1] + string(last[len(last)-1]+1) 390 | } 391 | } 392 | 393 | func urlFuzzScanner(directoryList []string) { 394 | // open the text file directoryList and read the lines in it 395 | // if directoryList is default or fav then use one of the default lists 396 | if directoryList[0] == "default" || directoryList[0] == "fav" { 397 | 398 | directoryListUrl := default_payload_url 399 | if directoryList[0] == "fav" { 400 | directoryListUrl = fav_payload_url 401 | } 402 | 403 | dir_resp, err := http.Get(directoryListUrl) 404 | 405 | if err != nil { 406 | log.Fatal(err) 407 | } 408 | defer dir_resp.Body.Close() 409 | if dir_resp.StatusCode == 200 { 410 | // get the response body as a string 411 | dataInBytes, _ := ioutil.ReadAll(dir_resp.Body) 412 | pageContent := string(dataInBytes) 413 | payload_create, err := os.Create("default_payload.txt") // Truncates if file already exists, be careful! 414 | if err != nil { 415 | log.Fatalf("failed creating file: %s", err) 416 | } 417 | defer payload_create.Close() 418 | _, err = payload_create.WriteString(pageContent) 419 | } 420 | directoryList[0] = "default_payload.txt" 421 | } 422 | if generate_payload == "true" { 423 | payload_create, err := os.Create("random_payload.txt") // Truncates if file already exists, be careful! 424 | if err != nil { 425 | log.Fatalf("failed creating file: %s", err) 426 | } 427 | last := "" 428 | for i := 0; i < generate_payload_length; i++ { 429 | next := NextAlias(last) 430 | _, err = payload_create.WriteString(next + "") 431 | last = next 432 | } 433 | directoryList[0] = "random_payload.txt" 434 | } 435 | 436 | file, err := os.Open(directoryList[0]) 437 | if err != nil { 438 | fmt.Fprint(os.Stdout, "\r"+err.Error()+strings.Repeat(" ", 100)+"\n") 439 | } 440 | defer file.Close() 441 | // read the lines in the text file 442 | 443 | file_lines, err := os.OpenFile(directoryList[0], os.O_RDONLY, 0444) 444 | if err != nil { 445 | fmt.Fprint(os.Stdout, "\r"+err.Error()+strings.Repeat(" ", 100)+"\n") 446 | } 447 | defer file_lines.Close() 448 | count_lines, _ := lineCounter(file_lines) 449 | if concurrency <= 0 { 450 | concurrency = MAX_CONCURRENT_JOBS 451 | } 452 | 453 | concurrent := make(chan int, concurrency) 454 | 455 | scanner := bufio.NewScanner(file) 456 | count := 0 457 | 458 | if output == "" { 459 | output = "output.txt" 460 | } else { 461 | output = output + ".txt" 462 | } 463 | 464 | file_create, err := os.Create(output) // Truncates if file already exists, be careful! 465 | if err != nil { 466 | log.Fatalf("failed creating file: %s", err) 467 | } 468 | defer file_create.Close() 469 | 470 | if filterTestLength == "true" { 471 | 472 | random_string := "213804asdad32432sdf" 473 | url_test := "" 474 | if strings.Contains(url, "#PSFUZZ#") { 475 | url_test = strings.Replace(url, "#PSFUZZ#", random_string, 1) 476 | } else { 477 | url_test = url + random_string 478 | } 479 | resp_test, err := sendRequest(url_test, "") 480 | if err != nil { 481 | // Handle the error here and continue the execution of the program 482 | fmt.Println("Error occurred while sending request:", err) 483 | return 484 | } 485 | _, testlength, _, _, _ = GetResponseDetails(resp_test) 486 | } 487 | 488 | for scanner.Scan() { 489 | percent := (count * 100 / count_lines) 490 | fill := strings.Repeat("x", percent) + strings.Repeat("-", 100-percent) 491 | _, _ = fmt.Fprint(os.Stdout, "\r[") 492 | _, _ = fmt.Fprintf(os.Stdout, "%s]", fill) 493 | p := int(count * 100 / (count_lines + 1)) 494 | _, _ = fmt.Fprintf(os.Stdout, "\t%d %%", p) 495 | 496 | word := scanner.Text() 497 | // check if the line is empty 498 | if word == "" { 499 | continue 500 | } 501 | 502 | concurrent <- 1 503 | count++ 504 | 505 | go func(count int, url string, showStatus string) { 506 | // find the wildcard in the url 507 | if strings.Contains(url, "#PSFUZZ#") { 508 | url = strings.Replace(url, "#PSFUZZ#", word, 1) 509 | } else { 510 | url = url + word 511 | } 512 | // write the result to the file 513 | testUrl(url, showStatus, file_create, false, requestAddHeader, bypass) 514 | <-concurrent 515 | }(count, url, showStatus) 516 | } 517 | return 518 | } 519 | 520 | func sendRequest(url string, requestHeader string) (*http.Response, error) { 521 | // Throttle request if necessary 522 | if throttleRate > 0 { 523 | <-throttleTicker.C 524 | } 525 | if requestHeader != "" { 526 | requestHeader = requestAddHeader 527 | } 528 | // create a new http client 529 | client := &http.Client{ 530 | CheckRedirect: func(req *http.Request, via []*http.Request) error { 531 | return http.ErrUseLastResponse 532 | }, 533 | } 534 | // create a new request 535 | req, err := http.NewRequest("GET", url, nil) 536 | if err != nil { 537 | fmt.Fprint(os.Stdout, "\r"+err.Error()+strings.Repeat(" ", 100)+"\n") 538 | return nil, err 539 | } 540 | // set the user agent 541 | if requestAddAgent != "" { 542 | req.Header.Set("User-Agent", requestAddAgent) 543 | } else { 544 | req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (zz99)") 545 | } 546 | 547 | // if requestHeader is not empty then add the headers to the request 548 | if requestHeader != "" { 549 | headers := strings.Split(requestHeader, ",") 550 | for _, header := range headers { 551 | header = strings.TrimSpace(header) 552 | headerSplit := strings.Split(header, ":") 553 | if len(headerSplit) == 2 { 554 | req.Header.Set(headerSplit[0], headerSplit[1]) 555 | } 556 | } 557 | } 558 | 559 | // define the request with a timeout of 2 seconds 560 | ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) 561 | defer cancel() 562 | // make the request 563 | resp, err := client.Do(req.WithContext(ctx)) 564 | if err != nil { 565 | fmt.Fprint(os.Stdout, "\r"+err.Error()+strings.Repeat(" ", 100)+"\n") 566 | return nil, err 567 | } 568 | return resp, nil 569 | } 570 | 571 | func testUrl(url string, showStatus string, file_create *os.File, redirected bool, requestHeader string, bypassResponse string) { 572 | resp, err := sendRequest(url, requestHeader) 573 | 574 | if err != nil { 575 | // Handle the error here and continue the execution of the program 576 | fmt.Println("Error occurred while sending request:", err) 577 | return 578 | } 579 | 580 | mutex.Lock() 581 | statuscount[resp.Status] = statuscount[resp.Status] + 1 582 | mutex.Unlock() 583 | 584 | responseAnalyse(resp, url, showStatus, file_create, redirected, bypassResponse) 585 | } 586 | 587 | func responseAnalyse(resp *http.Response, url string, showStatus string, file_create *os.File, redirected bool, bypassResponse string) { 588 | // create output string variable 589 | var outputString string 590 | if checkStatus(strconv.Itoa(resp.StatusCode)) && checkContentType(resp.Header.Get("Content-Type")) { 591 | title, length, matchWord, countWords, _ := GetResponseDetails(resp) 592 | if ((filterMatchWord != "" && matchWord != "") || filterMatchWord == "") && ((contains(filterLengthList, strconv.Itoa(length)) || contains(filterLengthList, "-1")) && (!contains(filterLengthNotList, strconv.Itoa(length)) || contains(filterLengthNotList, "-1")) || checkLength(strconv.Itoa(length))) { 593 | 594 | // save the length of the response 595 | mutex.Lock() 596 | lengthcount[length]++ 597 | mutex.Unlock() 598 | 599 | if filterWrongStatus200 == "true" { 600 | if strings.Contains(title, "Technical subdomain") || strings.Contains(title, "Page Not Available") || strings.Contains(title, "Access Gateway") || strings.Contains(title, "Not Found") || strings.Contains(title, "ERROR") || strings.Contains(title, "Error") || strings.Contains(title, "Forbidden") || strings.Contains(title, "Bad Request") || strings.Contains(title, "Internal Server Error") || strings.Contains(title, "Bad Gateway") || length <= 1 { 601 | return 602 | } 603 | } 604 | if filterWrongSubdomain == "true" { 605 | if length == 21 || length == 180 || strings.Contains(title, "Origin DNS error") || resp.StatusCode == http.StatusPermanentRedirect { 606 | return 607 | } 608 | } 609 | if filterTestLength == "true" { 610 | if length == testlength { 611 | return 612 | } 613 | } 614 | if strings.Contains(title, "404") { 615 | if filterPossible404 == "true" { 616 | return 617 | } 618 | title += " -- possibile a 404" 619 | } 620 | if filterMatchWord != "" && matchWord != "" { 621 | title += " -- MATCH: " + matchWord + " --" 622 | } 623 | if redirected { 624 | outputString = "redirected to " 625 | } 626 | outputString += url + " - " + resp.Status + " " + strings.Repeat(" ", 100) + "\n" + title + " Length:" + strconv.Itoa(length) + " Words:" + strconv.Itoa(countWords) + "\n" 627 | 628 | // convert resp.ContentLength to string 629 | fmt.Fprint(os.Stdout, "\r"+outputString) 630 | if redirected { 631 | fmt.Fprint(os.Stdout, "redirected to ") 632 | } 633 | if redirect == "true" && (resp.StatusCode == http.StatusFound || resp.StatusCode == http.StatusMovedPermanently) { //status code 302 634 | redirUrl, _ := resp.Location() 635 | testUrl(redirUrl.String(), showStatus, file_create, true, requestAddHeader, bypassResponse) 636 | } 637 | if onlydomains == "true" { 638 | // if url string start with http:// or https:// then remove it 639 | url_string := url 640 | if strings.HasPrefix(url_string, "http://") { 641 | url_string = strings.Replace(url_string, "http://", "", 1) 642 | } 643 | if strings.HasPrefix(url_string, "https://") { 644 | url_string = strings.Replace(url_string, "https://", "", 1) 645 | } 646 | outputString = url_string + "\n" 647 | } 648 | } 649 | if (resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusUnauthorized) && bypassResponse == "true" { 650 | bypassStatusCode40x(url, showStatus, file_create) 651 | } 652 | // if the status code is 429 then wait 5 seconds and retry 653 | if bypassTooManyRequests == "true" && resp.StatusCode == http.StatusTooManyRequests { 654 | time.Sleep(5 * time.Second) 655 | testUrl(url, showStatus, file_create, redirected, requestAddHeader, bypassResponse) 656 | } 657 | } 658 | _, _ = file_create.WriteString(outputString) 659 | } 660 | 661 | func bypassStatusCode40x(url string, showStatus string, file_create *os.File) { 662 | 663 | arrayPath := [13]string{"/*", "//.", "/%2e/", "/%2f/", "/./", "/", "/*/", "/..;/", "/..%3B/", "////", "/%20", "%00", "#test"} 664 | for _, element := range arrayPath { 665 | time.Sleep(time.Second) 666 | testUrl(url+element, showStatus, file_create, false, "", "false") 667 | } 668 | arrayExtensions := [10]string{".yml", ".php", ".html", ".zip", ".txt", ".yaml", ".wadl", ".htm", ".asp", ".aspx"} 669 | for _, element := range arrayExtensions { 670 | time.Sleep(time.Second) 671 | testUrl(url+element, showStatus, file_create, false, "", "false") 672 | } 673 | arrayHeader := [6]string{"X-Custom-IP-Authorization127.0.0.1", "Host:Localhost", "X-Forwarded-For:127.0.0.1:80", "X-Forwarded-For:http://127.0.0.1", "X-Custom-IP-Authorization:127.0.0.1", "Content-Length:0"} 674 | for _, element := range arrayHeader { 675 | time.Sleep(time.Second) 676 | testUrl(url, showStatus, file_create, false, element, "false") 677 | } 678 | } 679 | func checkContentType(contentType string) bool { 680 | if contains(filterContentTypeList, contentType) || contains(filterContentTypeList, "") { 681 | return true 682 | } 683 | return false 684 | } 685 | 686 | func checkStatus(s string) bool { 687 | if (contains(filterStatusCodeList, s) || showStatus == "true") && !contains(filterStatusNotList, s) { 688 | return true 689 | } 690 | for _, v := range filterStatusNotList { 691 | // check if in v is the string "-" Example 200-250 and compare the two numbers 692 | if strings.Contains(v, "-") { 693 | // split the string in two parts 694 | splitted := strings.Split(v, "-") 695 | // check if the length is in the range 696 | if splitted[0] <= s && s <= splitted[1] { 697 | return false 698 | } 699 | } 700 | } 701 | for _, v := range filterStatusCodeList { 702 | // check if in v is the string "-" Example 400-405 and compare the two numbers 703 | if strings.Contains(v, "-") { 704 | // split the string in two parts 705 | splitted := strings.Split(v, "-") 706 | // check if the length is in the range 707 | if splitted[0] <= s && s <= splitted[1] { 708 | return true 709 | } 710 | } 711 | } 712 | return false 713 | } 714 | 715 | func loadConfig(file string) (*Config, error) { 716 | var config Config 717 | configFile, err := os.Open(file) 718 | defer configFile.Close() 719 | if err != nil { 720 | return nil, err 721 | } 722 | jsonParser := json.NewDecoder(configFile) 723 | err = jsonParser.Decode(&config) 724 | return &config, err 725 | } 726 | 727 | func checkLength(s string) bool { 728 | // if filterMaxLength is bigger than 0 then check if the length is smaller than the filterMaxLength 729 | if filterMaxLength > 0 { 730 | // convert the string to int 731 | i, _ := strconv.Atoi(s) 732 | if i > filterMaxLength { 733 | return false 734 | } 735 | } 736 | // if filterMinLength is bigger than 0 then check if the length is bigger than the filterMinLength 737 | if filterMinLength > 0 { 738 | i, _ := strconv.Atoi(s) 739 | if filterMinLength > i { 740 | return false 741 | } 742 | } 743 | for _, v := range filterLengthNotList { 744 | // check if in v is the string "-" Example 10-200 and compare the two numbers 745 | if strings.Contains(v, "-") { 746 | // split the string in two parts 747 | splitted := strings.Split(v, "-") 748 | // check if the length is in the range 749 | if splitted[0] <= s && s <= splitted[1] { 750 | return false 751 | } 752 | } 753 | } 754 | for _, v := range filterLengthList { 755 | // check if in v is the string "-" Example 10-200 and compare the two numbers 756 | if strings.Contains(v, "-") { 757 | // split the string in two parts 758 | splitted := strings.Split(v, "-") 759 | // check if the length is in the range 760 | if splitted[0] <= s && s <= splitted[1] { 761 | return true 762 | } 763 | } 764 | } 765 | return false 766 | } 767 | 768 | func main() { 769 | fmt.Fprint(os.Stdout, "PSFuzz - Payload Scanner\n") 770 | fmt.Fprint(os.Stdout, "Version: 0.8.0\n") 771 | fmt.Fprint(os.Stdout, "Author: Proviesec\n") 772 | // ouput ascii art 773 | fmt.Fprint(os.Stdout, ` 774 | %%%%%%%%%%% %%%%%%%%%%%% %%%%%%%%%% %%%% %%%% %%%%%%%%%%%% %%%%%%%%%%%% 775 | %%%%%%%%%%%% %%%%%%%%%%% %%%% %%%% %%%% %%%% %%%% 776 | %%%% %%%% %%%% %%%% %%%% %% %% 777 | %%%% % %%% %%%%%%%%%% %%%% %%%% %%% %%% 778 | %%%%%%%%% %%%%% %%%% %%%% %%%% %%%% %%%% 779 | %%%% %%%%%%%%%%%%% %%%% %%%%%%%#%%%% %%%%%%%%%%%% %%%%%%%%%%%% 780 | %%% %%%%%%%%%% %% %%%%%% %%%%%%%% %%%%%%%% 781 | 782 | `) 783 | 784 | directoryList := strings.Split(dirlist, ",") 785 | 786 | // check the directory list, if the found in the url 787 | urlFuzzScanner(directoryList) 788 | fmt.Fprint(os.Stdout, "\n") 789 | fmt.Println(statuscount) // map[string]int 790 | fmt.Println(lengthcount) 791 | } 792 | -------------------------------------------------------------------------------- /lists/useragents.txt: -------------------------------------------------------------------------------- 1 | !Susie (http://www.sync2it.com/susie) 2 | <a href='http://www.unchaos.com/' 3 | <b 4 | <http://www.sygol.com/ 5 | ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) 6 | (DreamPassport/3.0; isao/MyDiGiRabi) 7 | (Privoxy/1.0) 8 | */Nutch-0.9-dev 9 | +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) 10 | -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de 11 | 123spider-Bot (Version: 1.02, powered by www.123spider.de 12 | 192.comAgent 13 | 1st ZipCommander (Net) - http://www.zipcommander.com/ 14 | 2Bone_LinkChecker/1.0 libwww-perl/5.64 15 | 4anything.com LinkChecker v2.0 16 | 8484 Boston Project v 1.0 17 | :robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html ) 18 | A-Online Search 19 | A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27 20 | A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 21 | aardvark-crawler 22 | AbachoBOT 23 | AbachoBOT (Mozilla compatible) 24 | ABCdatos BotLink/5.xx.xxx#BBL 25 | Aberja Checkomat 26 | abot/0.1 (abot; http://www.abot.com; abot@abot.com) 27 | About/0.1libwww-perl/5.47 28 | Accelatech RSSCrawler/0.4 29 | accoona 30 | Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) 31 | Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) 32 | Ace Explorer 33 | Ack (http://www.ackerm.com/) 34 | AcoiRobot 35 | Acoon Robot v1.50.001 36 | Acoon Robot v1.52 (http://www.acoon.de) 37 | Acoon-Robot 4.0.x.[xx] (http://www.acoon.de) 38 | Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com) 39 | Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) 40 | ActiveBookmark 1.x 41 | Activeworlds 42 | ActiveWorlds/3.xx (xxx) 43 | Ad Muncher v4.xx.x 44 | Ad Muncher v4x Build xxxxx 45 | Adaxas Spider (http://www.adaxas.net/) 46 | Advanced Browser (http://www.avantbrowser.com) 47 | AESOP_com_SpiderMan 48 | agadine/1.x.x (+http://www.agada.de) 49 | Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) 50 | AgentName/0.1 libwww-perl/5.48 51 | AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China) 52 | AideRSS/1.0 (aiderss.com) 53 | aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) 54 | aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com) 55 | Akregator/1.2.9; librss/remnants 56 | Aladin/3.324 57 | Alcatel-BG3/1.0 UP.Browser/5.0.3.1.2 58 | Aleksika Spider/1.0 (+http://www.aleksika.com/) 59 | AlertInfo 2.0 (Powered by Newsbrain) 60 | AlkalineBOT/1.3 61 | AlkalineBOT/1.4 (1.4.0326.0 RTM) 62 | Allesklar/0.1 libwww-perl/5.46 63 | Alligator 1.31 (www.nearsoftware.com) 64 | Allrati/1.1 (+) 65 | AltaVista Intranet V2.0 AVS EVAL search@freeit.com 66 | AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net 67 | AltaVista Intranet V2.0 evreka.com crawler@evreka.com 68 | AltaVista V2.0B crawler@evreka.com 69 | amaya/x.xx libwww/x.x.x 70 | AmfibiBOT 71 | Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com) 72 | Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) 73 | amibot 74 | Amiga-AWeb/3.4.167SE 75 | AmigaVoyager/3.4.4 (MorphOS/PPC native) 76 | AmiTCP Miami (AmigaOS 2.04) 77 | Amoi 8512/R21.0 NF-Browser/3.3 78 | amzn_assoc 79 | AnnoMille spider 0.1 alpha - http://www.annomille.it 80 | annotate_google; http://ponderer.org/download/annotate_google.user.js 81 | Anonymized by ProxyOS: http://www.megaproxy.com 82 | Anonymizer/1.1 83 | AnswerBus (http://www.answerbus.com/) 84 | AnswerChase PROve x.0 85 | AnswerChase x.0 86 | ANTFresco/x.xx 87 | antibot-V1.1.5/i586-linux-2.2 88 | AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) 89 | Apexoo Spider 1.x 90 | Aplix HTTP/1.0.1 91 | Aplix_SANYO_browser/1.x (Japanese) 92 | Aplix_SEGASATURN_browser/1.x (Japanese) 93 | Aport 94 | appie 1.1 (www.walhello.com) 95 | Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102 96 | Apple-PubSub/65.1.1 97 | ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) 98 | ArachBot 99 | Arachnoidea (arachnoidea@euroseek.com) 100 | aranhabot 101 | ArchitextSpider 102 | archive.org_bot 103 | Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com) 104 | Arikus_Spider 105 | Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt) 106 | ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) 107 | Asahina-Antenna/1.x 108 | Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) 109 | ask.24x.info 110 | AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com) 111 | asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) 112 | ASPSeek/1.2.5 113 | ASPseek/1.2.9d 114 | ASPSeek/1.2.x 115 | ASPSeek/1.2.xa 116 | ASPseek/1.2.xx 117 | ASPSeek/1.2.xxpre 118 | ASSORT/0.10 119 | asterias/2.0 120 | AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) 121 | Atomic_Email_Hunter/4.0 122 | Atomz/1.0 123 | atSpider/1.0 124 | Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) 125 | AU-MIC/2.0 MMP/2.0 126 | AUDIOVOX-SMT5600 127 | augurfind 128 | augurnfind V-1.x 129 | autoemailspider 130 | autohttp 131 | autowebdir 1.1 (www.autowebdir.com) 132 | AV Fetch 1.0 133 | Avant Browser (http://www.avantbrowser.com) 134 | AVSearch-1.0(peter.turney@nrc.ca) 135 | AVSearch-2.0-fusionIdx-14-CompetitorWebSites 136 | AVSearch-3.0(AltaVista/AVC) 137 | AWeb 138 | axadine/ (Axadine Crawler; http://www.axada.de/; ) 139 | AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine. 140 | Azureus 2.x.x.x 141 | BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) 142 | BaboomBot/1.x.x (+http://www.baboom.us) 143 | BackStreet Browser 3.x 144 | BaiduImagespider+(+http://www.baidu.jp/search/s308.html) 145 | BaiDuSpider 146 | Baiduspider+(+http://help.baidu.jp/system/05.html) 147 | Baiduspider+(+http://www.baidu.com/search/spider.htm) 148 | Baiduspider+(+http://www.baidu.com/search/spider_jp.html) 149 | Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) 150 | BanBots/1.2 (spider@banbots.com) 151 | Barca/2.0.xxxx 152 | BarcaPro/1.4.xxxx 153 | BarraHomeCrawler (albertof@barrahome.org) 154 | bCentral Billing Post-Process 155 | bdcindexer_2.6.2 (research@bdc) 156 | BDFetch 157 | BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686) 158 | BeamMachine/0.5 (dead link remover of www.beammachine.net) 159 | beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) 160 | BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) 161 | BeebwareDirectory/v0.01 162 | Big Brother (http://pauillac.inria.fr/~fpottier/) 163 | Big Fish v1.0 164 | BigBrother/1.6e 165 | BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) 166 | BIGLOTRON (Beta 2;GNU/Linux) 167 | Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) 168 | Bilbo/2.3b-UNIX 169 | BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) 170 | BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) 171 | billbot wjj@cs.cmu.edu 172 | Bitacle bot/1.1 173 | Bitacle Robot (V:1.0;) (http://www.bitacle.com) 174 | Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx) 175 | BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0) 176 | BlackWidow 177 | Blaiz-Bee/1.0 (+http://www.blaiz.net) 178 | Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) 179 | Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net) 180 | BlitzBOT@tricus.net 181 | BlitzBOT@tricus.net (Mozilla compatible) 182 | BlockNote.Net 183 | BlogBot/1.x 184 | BlogBridge 2.13 (http://www.blogbridge.com/) 185 | Bloglines Title Fetch/1.0 (http://www.bloglines.com) 186 | Bloglines-Images/0.1 (http://www.bloglines.com) 187 | Bloglines/3.1 (http://www.bloglines.com) 188 | BlogMap (http://www.feedmap.net) 189 | Blogpulse (info@blogpulse.com) 190 | BlogPulseLive (support@blogpulse.com) 191 | BlogSearch/1.x +http://www.icerocket.com/ 192 | blogsearchbot-pumpkin-3 193 | BlogsNowBot, V 2.01 (+http://www.blogsnow.com/) 194 | BlogVibeBot-v1.1 (spider@blogvibe.nl) 195 | blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) 196 | BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) 197 | BlogzIce/1.0 +http://www.icerocket.com/ 198 | BloobyBot 199 | Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) 200 | bluefish 0.6 HTML editor 201 | BMCLIENT 202 | BMLAUNCHER 203 | Bobby/4.0.x RPT-HTTPClient/0.3-3E 204 | boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) 205 | boitho.com-robot/1.x 206 | boitho.com-robot/1.x (http://www.boitho.com/bot.html) 207 | Bookdog/x.x 208 | Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/) 209 | Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] 210 | Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta) 211 | BookmarkBase(2/;http://bookmarkbase.com) 212 | Bot mailto:craftbot@yahoo.com 213 | BPImageWalker/2.0 (www.bdbrandprotect.com) 214 | BravoBrian bstop.bravobrian.it 215 | BravoBrian SpiderEngine MarcoPolo 216 | BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) 217 | BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) 218 | BSDSeek/1.0 219 | BStop.BravoBrian.it Agent Detector 220 | BTbot/0.x (+http://www.btbot.com/btbot.html) 221 | BTWebClient/180B(9704) 222 | BuildCMS crawler (http://www.buildcms.com/crawler) 223 | Bulkfeeds/r1752 (http://bulkfeeds.net/) 224 | BullsEye 225 | bumblebee@relevare.com 226 | BunnySlippers 227 | BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) 228 | Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) 229 | BW-C-2.0 230 | bwh3_user_agent 231 | Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) 232 | Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) 233 | CamelHttpStream/1.0 234 | Cancer Information and Support International; 235 | carleson/1.0 236 | Carnegie_Mellon_University_Research_WebBOT-- 237 | Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html 238 | Catall Spider 239 | CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) 240 | CCBot/1.0 (+http://www.commoncrawl.org/bot.html) 241 | ccubee/x.x 242 | CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0 243 | CE-Preload 244 | CentiverseBot 245 | CentiverseBot - investigator 246 | CentiverseBot/3.0 (http://www.centiverse-project.net) 247 | Ceramic Tile Installation Guide (http://www.floorstransformed.com) 248 | CERN-LineMode/2.15 249 | cfetch/1.0 250 | CFNetwork/x.x 251 | cg-eye interactive 252 | Charon/1.x (Amiga) 253 | Chat Catcher/1.0 254 | Checkbot/1.xx LWP/5.xx 255 | CheckLinks/1.x.x 256 | CheckUrl 257 | CheckWeb 258 | Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp) 259 | China Local Browse 2.6 260 | Chitika ContentHit 1.0 261 | ChristCRAWLER 2.0 262 | CHttpClient by Open Text Corporation 263 | CipinetBot (http://www.cipinet.com/bot.html) 264 | Cityreview Robot (+http://www.cityreview.org/crawler/) 265 | CJ Spider/ 266 | CJB.NET Proxy 267 | ClariaBot/1.0 268 | Claymont.com 269 | CloakDetect/0.9 (+http://fulltext.seznam.cz/) 270 | Clushbot/2.x (+http://www.clush.com/bot.html) 271 | Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) 272 | Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) 273 | Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) 274 | Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) 275 | COAST WebMaster Pro/4.x.x.xx (Windows NT) 276 | CoBITSProbe 277 | Cocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious) 278 | Cogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html) 279 | ColdFusion 280 | ColdFusion (BookmarkTracker.com) 281 | collage.cgi/1.xx 282 | combine/0.0 283 | Combine/2.0 http://combine.it.lth.se/ 284 | Combine/3 http://combine.it.lth.se/ 285 | Combine/x.0 286 | cometrics-bot, http://www.cometrics.de 287 | Commerce Browser Center 288 | complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com) 289 | Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu 290 | Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) 291 | Contact 292 | ContactBot/0.2 293 | ContentSmartz 294 | contype 295 | Convera Internet Spider V6.x 296 | ConveraCrawler/0.2 297 | ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) 298 | ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) 299 | CoolBot 300 | Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3 301 | CoralWebPrx/0.1.1x (See http://coralcdn.org/) 302 | cosmos/0.8_(robot@xyleme.com) 303 | cosmos/0.9_(robot@xyleme.com) 304 | CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com) 305 | CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml) 306 | Covac TexAs Arachbot 307 | CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1) 308 | Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) 309 | Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) 310 | CrawlConvera0.1 (CrawlConvera@yahoo.com) 311 | Crawler 312 | Crawler (cometsearch@cometsystems.com) 313 | Crawler admin@crawler.de 314 | Crawler V 0.2.x admin@crawler.de 315 | crawler@alexa.com 316 | CrawlerBoy Pinpoint.com 317 | Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) 318 | CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) 319 | Cricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0 320 | CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) 321 | csci_b659/0.13 322 | CSE HTML Validator Professional (http://www.htmlvalidator.com/) 323 | Cuam Ver0.050bx 324 | Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ 325 | curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4 326 | curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled) 327 | curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6) 328 | curl/7.9.x (win32) libcurl 7.9.x 329 | CurryGuide SiteScan 1.1 330 | Custo x.x (www.netwu.com) 331 | Custom Spider www.bisnisseek.com /1.0 332 | Cyberdog/2.0 (Macintosh; 68k) 333 | CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) 334 | CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com) 335 | CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) 336 | CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com) 337 | DA 3.5 (www.lidan.com) 338 | DA 4.0 339 | DA 4.0 (www.downloadaccelerator.com) 340 | DA 5.0 341 | DA 7.0 342 | DAP x.x 343 | Dart Communications PowerTCP 344 | DataCha0s/2.0 345 | DataFountains/DMOZ Downloader 346 | DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml) 347 | DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml) 348 | DataparkSearch/4.47 (+http://dataparksearch.org/bot) 349 | DataparkSearch/4.xx (http://www.dataparksearch.org/) 350 | DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com) 351 | DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com) 352 | DatenBot( http://www.sicher-durchs-netz.de/bot.html) 353 | DaviesBot/1.7 (www.wholeweb.net) 354 | daypopbot/0.x 355 | dbDig(http://www.prairielandconsulting.com) 356 | DBrowse 1.4b 357 | DBrowse 1.4d 358 | DC-Sakura/x.xx 359 | dCSbot/1.1 360 | DDD 361 | dds explorer v1.0 beta 362 | de.searchengine.comBot 1.2 (http://de.searchengine.com/spider) 363 | DeadLinkCheck/0.4.0 libwww-perl/5.xx 364 | Deep Link Calculator v1.0 365 | deepak-USC/ISI 366 | DeepIndex 367 | DeepIndex ( http://www.zetbot.com ) 368 | DeepIndex (www.en.deepindex.com) 369 | DeepIndexer.ca 370 | del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) 371 | DeleGate/9.0.5-fix1 372 | Demo Bot DOT 16b 373 | Demo Bot Z 16b 374 | Denmex websearch (http://search.denmex.com) 375 | Der große BilderSauger 2.00u 376 | dev-spider2.searchpsider.com/1.3b 377 | DevComponents.com HtmlDocument Object 378 | DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) 379 | Diamond/x.0 380 | DiamondBot 381 | Digger/1.0 JDK/1.3.0rc3 382 | DigOut4U 383 | DIIbot/1.2 384 | Dillo/0.8.5-i18n-misc 385 | Dillo/0.x.x 386 | disastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us) 387 | DISCo Pump x.x 388 | disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) 389 | disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) 390 | DittoSpyder 391 | dlman 392 | dloader(NaverRobot)/1.0 393 | DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com 394 | DoCoMo/1.0/Nxxxi/c10 395 | DoCoMo/1.0/Nxxxi/c10/TB 396 | DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) 397 | DoCoMo/2.0 P900iV(c100;TB;W24H11) 398 | DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via translate.google.com) 399 | DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) 400 | DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) 401 | DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804 402 | dodgebot/experimental 403 | DonutP; Windows98SE 404 | Doubanbot/1.0 (bot@douban.com http://www.douban.com) 405 | Download Demon/3.x.x.x 406 | Download Druid 2.x 407 | Download Express 1.0 408 | Download Master 409 | Download Ninja 3.0 410 | Download Wonder 411 | Download-Tipp Linkcheck (http://download-tipp.de/) 412 | Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/) 413 | DownloadDirect.1.0 414 | Dr.Web (R) online scanner: http://online.drweb.com/ 415 | Dragonfly File Reader 416 | Drecombot/1.0 (http://career.drecom.jp/bot.html) 417 | Drupal (+http://drupal.org/) 418 | DSurf15a 01 419 | DSurf15a 71 420 | DSurf15a 81 421 | DSurf15a VA 422 | DTAAgent 423 | dtSearchSpider 424 | Dual Proxy 425 | DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) 426 | Dumbot(version 0.1 beta - dumbfind.com) 427 | Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) 428 | Dumbot(version 0.1 beta) 429 | e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) 430 | e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) 431 | eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com) 432 | EARTHCOM.info/1.x [www.earthcom.info] 433 | EARTHCOM.info/1.xbeta [www.earthcom.info] 434 | EasyDL/3.xx 435 | EasyDL/3.xx http://keywen.com/Encyclopedia/Bot 436 | EBrowse 1.4b 437 | eCatch/3.0 438 | EchO!/2.0 439 | Educate Search VxB 440 | egothor/3.0a (+http://www.xdefine.org/robot.html) 441 | EgotoBot/4.8 (+http://www.egoto.com/about.htm) 442 | ejupiter.com 443 | EldoS TimelyWeb/3.x 444 | elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) 445 | ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html) 446 | ELinks (0.x.x; Linux 2.4.20 i586; 132x60) 447 | ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43) 448 | EmailSiphon 449 | EmailSpider 450 | EmailWolf 1.00 451 | EmeraldShield.com WebBot 452 | EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx) 453 | EMPAS_ROBOT 454 | EnaBot/1.x (http://www.enaball.com/crawler.html) 455 | endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo) 456 | Enfish Tracker 457 | Enterprise_Search/1.0 458 | Enterprise_Search/1.0.xxx 459 | Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) 460 | envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) 461 | envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) 462 | EroCrawler 463 | ES.NET_Crawler/2.0 (http://search.innerprise.net/) 464 | eseek-larbin_2.6.2 (crawler@exactseek.com) 465 | ESISmartSpider 466 | eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) 467 | ESurf15a 15 468 | EuripBot/0.x (+http://www.eurip.com) GetFile 469 | EuripBot/0.x (+http://www.eurip.com) GetRobots 470 | EuripBot/0.x (+http://www.eurip.com) PreCheck 471 | Eurobot/1.0 (http://www.ayell.eu) 472 | EvaalSE - bot@evaal.com 473 | eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) 474 | Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp) 475 | Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp) 476 | Exabot-Images/1.0 477 | Exabot-Test/1.0 478 | Exabot/2.0 479 | Exabot/3.0 480 | ExactSearch 481 | ExactSeek Crawler/0.1 482 | exactseek-crawler-2.63 (crawler@exactseek.com) 483 | exactseek-pagereaper-2.63 (crawler@exactseek.com) 484 | exactseek.com 485 | Exalead NG/MimeLive Client (convert/http/0.120) 486 | Excalibur Internet Spider V6.5.4 487 | Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) 488 | exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) 489 | exooba/exooba crawler (exooba; exooba) 490 | ExperimentalHenrytheMiragoRobot 491 | Expired Domain Sleuth 492 | Express WebPictures (www.express-soft.com) 493 | ExtractorPro 494 | Extreme Picture Finder 495 | EyeCatcher (Download-tipp.de)/1.0 496 | Factbot 1.09 (see http://www.factbites.com/webmasters.php) 497 | factbot : http://www.factbites.com/robots 498 | FaEdit/2.0.x 499 | FairAd Client 500 | FANGCrawl/0.01 501 | FARK.com link verifier 502 | Fast Crawler Gold Edition 503 | FAST Enterprise Crawler 6 (Experimental) 504 | FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/ 505 | FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com) 506 | FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) 507 | FAST Enterprise Crawler 6 used by FAST (FAST) 508 | FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) 509 | FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) 510 | FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) 511 | FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de) 512 | FAST Enterprise Crawler/6 (www.fastsearch.com) 513 | FAST Enterprise Crawler/6.4 (helpdesk at fast.no) 514 | FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0) 515 | FAST MetaWeb Crawler (helpdesk at fastsearch dot com) 516 | Fast PartnerSite Crawler 517 | FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 518 | FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 519 | FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no 520 | FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no 521 | FAST-WebCrawler/3.2 test 522 | FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) 523 | FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) 524 | FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) 525 | FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler) 526 | FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 527 | FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) 528 | FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 529 | FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) 530 | FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 531 | FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 532 | FAST-WebCrawler/3.x Multimedia 533 | FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no) 534 | fastbot crawler beta 2.0 (+http://www.fastbot.de) 535 | FastBug http://www.ay-up.com 536 | FastCrawler 3.0.1 (crawler@1klik.dk) 537 | FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com) 538 | Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/) 539 | FavIconizer 540 | favo.eu crawler/0.6 (http://www.favo.eu) 541 | FavOrg 542 | Favorites Checking (http://campulka.net) 543 | Favorites Sweeper v.2.03 544 | Faxobot/1.0 545 | FDM 1.x 546 | FDM 2.x 547 | Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) 548 | Feed24.com 549 | Feed::Find/0.0x 550 | Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1) 551 | FeedChecker/0.01 552 | FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP) 553 | Feedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html) 554 | Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) 555 | FeedForAll rss2html.php v2 556 | FeedHub FeedDiscovery/1.0 (http://www.feedhub.com) 557 | FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com) 558 | Feedjit Favicon Crawler 1.0 559 | Feedreader 3.xx (Powered by Newsbrain) 560 | Feedshow/x.0 (http://www.feedshow.com; 1 subscriber) 561 | FeedshowOnline (http://www.feedshow.com) 562 | Feedster Crawler/3.0; Feedster, Inc. 563 | FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector 564 | Felix - Mixcat Crawler (+http://mixcat.com) 565 | fetch libfetch/2.0 566 | FFC Trap Door Spider 567 | Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) 568 | Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com) 569 | Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) 570 | fileboost.net/1.0 (+http://www.fileboost.net) 571 | FileHound x.x 572 | Filtrbox/1.0 573 | FindAnISP.com_ISP_Finder_v99a 574 | Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) 575 | findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/) 576 | FineBot 577 | Finjan-prefetch 578 | Firefly/1.0 579 | Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) 580 | Firefox (kastaneta03@hotmail.com) 581 | Firefox_1.0.6 (kasparek@naparek.cz) 582 | FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov 583 | firstsbot 584 | Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com) 585 | FlashGet 586 | FLATARTS_FAVICO 587 | Flexum spider 588 | Flexum/2.0 589 | FlickBot 2.0 RPT-HTTPClient/0.3-3 590 | flunky 591 | fly/6.01 libwww/4.0D 592 | flyindex.net 1.0/http://www.flyindex.net 593 | FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) 594 | FocusedSampler/1.0 595 | Folkd.com Spider/0.1 beta 1 (www.folkd.com) 596 | FollowSite Bot ( http://www.followsite.com/bot.html ) 597 | FollowSite.com ( http://www.followsite.com/b.html ) 598 | Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots 599 | Francis/1.0 (francis@neomo.de http://www.neomo.de/) 600 | Franklin Locator 1.8 601 | free-downloads.net download-link validator /0.1 602 | FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com) 603 | Frelicbot/1.0 +http://www.frelic.com/ 604 | FreshDownload/x.xx 605 | FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com 606 | FSurf15a 01 607 | FTB-Bot http://www.findthebest.co.uk/ 608 | Full Web Bot 0416B 609 | Full Web Bot 0516B 610 | Full Web Bot 2816B 611 | FuseBulb.Com 612 | FyberSpider (+http://www.fybersearch.com/fyberspider.php) 613 | Gagglebot 614 | GAIS Robot/1.0B2 615 | Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php) 616 | Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) 617 | GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) 618 | Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com) 619 | gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) 620 | Gamespy_Arcade 621 | GammaSpider/1.0 622 | gazz/x.x (gazz@nttrd.com) 623 | geckobot 624 | Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 625 | generic_crawler/01.0217/ 626 | GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727) 627 | genieBot (http://64.5.245.11/faq/faq.html) 628 | geniebot wgao@genieknows.com 629 | GeoBot/1.0 630 | GeonaBot 1.x; http://www.geona.com/ 631 | geourl/2.0b2 632 | GeoURLBot 1.0 (http://geourl.org) 633 | GetBot 634 | GetRight/3.x.x 635 | GetRight/4.5xx 636 | GetRight/4.x 637 | GetRight/4.x[a-e] 638 | GetRight/6.1 (Pro) 639 | GetRightPro/6.0beta2 640 | GetWeb/0.1 libwww-perl/5.16 641 | GhostRouteHunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net) 642 | gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) 643 | Gigabot/2.0 (gigablast.com) 644 | Gigabot/2.0/gigablast.com/spider.html 645 | Gigabot/2.0; http://www.gigablast.com/spider.html 646 | Gigabot/2.0att 647 | Gigabot/3.0 (http://www.gigablast.com/spider.html) 648 | Gigabot/x.0 649 | GigabotSiteSearch/2.0 (sitesearch.gigablast.com) 650 | GNODSPIDER (www.gnod.net) 651 | Go!Zilla 3.x (www.gozilla.com) 652 | Go!Zilla/4.x.x.xx 653 | Go-Ahead-Got-It/1.1 654 | Goblin/0.9 (http://www.goguides.org/) 655 | Goblin/0.9.x (http://www.goguides.org/goblin-info.html) 656 | GoForIt.com 657 | GOFORITBOT ( http://www.goforit.com/about/ ) 658 | GoGuides.Org Link Check 659 | GoldenFeed Spider 1.0 (http://www.goldenfeed.com) 660 | Goldfire Server 661 | gonzo1[P] +http://www.suchen.de/popups/faq.jsp 662 | gonzo2[P] +http://www.suchen.de/faq.html 663 | Goofer/0.2 664 | Google Talk 665 | googlebot (larbin2.6.0@unspecified.mail) 666 | Googlebot-Image/1.0 667 | Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) 668 | Googlebot/2.1 ( http://www.google.com/bot.html) 669 | Googlebot/2.1 ( http://www.googlebot.com/bot.html) 670 | Googlebot/Test ( http://www.googlebot.com/bot.html) 671 | Gordon's Spider/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com) 672 | GrapeFX/0.3 libwww/5.4.0 673 | great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) 674 | GreatNews/1.0 675 | GreenBrowser 676 | gridwell (http://search.gridwell.com) 677 | GrigorBot 0.8 (http://www.grigor.biz/bot.html) 678 | Gromit/1.0 679 | grub crawler(http://www.grub.org) 680 | grub-client 681 | gsa-crawler (Enterprise; GID-01422; jplastiras@google.com) 682 | gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) 683 | gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com) 684 | gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) 685 | gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com) 686 | GSiteCrawler/v1.xx rev. xxx (http://gsitecrawler.com/) 687 | Guestbook Auto Submitter 688 | Gulliver/1.3 689 | Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) 690 | Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) 691 | GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) 692 | GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) 693 | Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/6.2.2.4; UPG1; UP/4.0; Embedded) 694 | HappyFunBot/1.1 695 | Harvest-NG/1.0.2 696 | Haste/0.12 (HOME: http://haste.kytoon.com/) 697 | Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) 698 | Hatena Mobile Gateway/1.0 699 | Hatena Pagetitle Agent/1.0 700 | Hatena RSS/0.3 (http://r.hatena.ne.jp) 701 | HatenaScreenshot/1.0 (checker) 702 | hbtronix.spider.2 -- http://hbtronix.de/spider.php 703 | HeinrichderMiragoRobot 704 | HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) 705 | Helix/1.x ( http://www.sitesearch.ca/helix/) 706 | HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) 707 | HenrytheMiragoRobot 708 | HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) 709 | hgrepurl/1.0 710 | Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3 711 | HiDownload 712 | Hippias/0.9 Beta 713 | HitList 714 | Hitwise Spider v1.0 http://www.hitwise.com 715 | HLoader 716 | holmes/3.11 (http://morfeo.centrum.cz/bot) 717 | holmes/3.9 (onet.pl) 718 | holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl) 719 | holmes/x.x 720 | HolmesBot (http://holmes.ge) 721 | HomePageSearch(hpsearch.uni-trier.de) 722 | Homerbot: www.homerweb.com 723 | Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com) 724 | HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at 725 | HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at 726 | HotJava/1.0.1/JRE1.1.x 727 | Hotzonu/x.0 728 | HPL/Nutch-0.9 - 729 | htdig/3.1.6 (http://computerorgs.com) 730 | htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) 731 | htdig/3.1.x (root@localhost) 732 | Html Link Validator (www.lithopssoft.com) 733 | HTML2JPG Blackbox, http://www.html2jpg.com 734 | HTML2JPG Enterprise 735 | HTMLParser/1.x 736 | HTTP Retriever 737 | http://Anonymouse.org/ (Unix) 738 | http://Ask.24x.Info/ (http://narres.it/) 739 | http://hilfe.acont.de/bot.html ACONTBOT 740 | http://OzySoftware.com/Index.html 741 | http://www.almaden.ibm.com/cs/crawler 742 | http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com] 743 | http://www.almaden.ibm.com/cs/crawler [wf216] 744 | http://www.istarthere.com_spider@istarthere.com 745 | http://www.monogol.de 746 | http://www.trendtech.dk/spider.asp) 747 | HTTP::Lite/2.x.x 748 | HTTPEyes 749 | HTTPResume v. 1.x 750 | httpunit/1.5 751 | httpunit/1.x 752 | Hybrid/1.2 [en] (OS Independent) 753 | HyperEstraier/1.x.xx 754 | i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) 755 | IAArchiver-1.0 756 | iaskspider 757 | iaskspider2 (iask@staff.sina.com.cn) 758 | ia_archiver 759 | ia_archiver-web.archive.org 760 | ia_archiver/1.6 761 | IBrowse/2.2 (AmigaOS 3.5) 762 | IBrowse/2.2 (Windows 3.1) 763 | iCab/2.5.2 (Macintosh; I; PPC) 764 | ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) 765 | ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) 766 | iCCrawler (http://www.iccenter.net) 767 | ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) 768 | ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86) 769 | ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html) 770 | ichiro/x.0 (ichiro@nttr.co.jp) 771 | IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) 772 | IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) 773 | ICOO Loader v.x.x.x 774 | ICRA_label_spider/x.0 775 | icsbot-0.1 776 | IDA 777 | ideare - SignSite/1.x 778 | iearthworm/1.0, iearthworm@yahoo.com.cn 779 | IEFav172Free 780 | iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers) 781 | igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html) 782 | iGetter/1.x (Macintosh;G;PPC) 783 | iGetter/2 (Macintosh; U; PPC Mac OS X; en) 784 | IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) 785 | ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com) 786 | ilial/Nutch-0.9-dev 787 | IlseBot/1.x 788 | IlTrovatore-Setaccio ( http://www.iltrovatore.it) 789 | Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) 790 | IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html) 791 | Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) 792 | iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....) 793 | IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) 794 | ImageVisu/v4.x.x 795 | ImageWalker/2.0 (www.bdbrandprotect.com) 796 | Incutio HttpClient v0.x 797 | IncyWincy data gatherer(webmaster@loopimprovements.com 798 | IncyWincy page crawler(webmaster@loopimprovements.com 799 | IncyWincy(http://www.look.com) 800 | IncyWincy(http://www.loopimprovements.com/robot.html) 801 | IncyWincy/2.1(loopimprovements.com/robot.html) 802 | IndexTheWeb.com Crawler7 803 | Industry Program 1.0.x 804 | Inet library 805 | InetURL/1.0 806 | info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca 807 | Infoaxe./Nutch-0.9 808 | infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl) 809 | InfoFly/1.0 (http://www.versions-project.org/) 810 | InfoLink/1.x 811 | INFOMINE/8.0 Adders 812 | INFOMINE/8.0 RemoteServices 813 | INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents) 814 | InfoNaviRobot(F107) 815 | InfoSeek Sidewinder/0.9 816 | InfoSeek Sidewinder/1.0A 817 | InfoSeek Sidewinder/1.1A 818 | Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX) 819 | Infoseek SideWinder/2.0B (Linux 2.4 i686) 820 | INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp) 821 | Inktomi Search 822 | InnerpriseBot/1.0 (http://www.innerprise.com/) 823 | Insitor.com search and find world wide! 824 | Insitornaut 825 | InstallShield DigitalWizard 826 | integrity/1.6 827 | Intelix/0.x (cs; http://www.microton.cz/intelix/; microton@@microton.cz) 828 | Interarchy/x.x.x (InterarchyCrawler) 829 | Internet Ninja x.0 830 | InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apache 831 | InternetLinkAgent/3.1 832 | InternetSeer.com 833 | intraVnews/1.x 834 | IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) 835 | IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.com 836 | http://www.ip2location.com 837 | IP2MapBot/1.1 <a href=http://www.ip2map.com 838 | IPiumBot laurion(dot)com 839 | IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) 840 | Iria/1.xxa 841 | IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler) 842 | IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) 843 | IrssiUrlLog/0.2 844 | Irvine/1.x.x 845 | ISC Systems iRc Search 2.1 846 | iSiloX/4.xx Windows/32 847 | isurf (tszhu@canada.com) 848 | iTunes/x.x.x 849 | IUPUI Research Bot v 1.9a 850 | iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml) 851 | iVia/4.0 CanonizeUrl (http://infomine.ucr.edu/iVia/useragents.shtml 852 | IWAgent/ 1.0 - www.brandprotect.com 853 | J-PHONE/3.0/J-SH07 854 | Jabot/6.x (http://odin.ingrid.org/) 855 | Jabot/7.x.x (http://odin.ingrid.org/) 856 | Jack 857 | Jakarta Commons-HttpClient/2.0xxx 858 | Jakarta Commons-HttpClient/3.0-rcx 859 | Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com) 860 | Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com) 861 | Java 1.1 862 | Java/1.4.1_01 863 | Java1.0.21.0 864 | Java1.1.xx.x 865 | Java1.3.0rc1 866 | Java1.3.x 867 | Java1.4.0 868 | Jayde Crawler. http://www.jayde.com 869 | JBH Agent 2.0 870 | jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0) 871 | JCheckLinks/0.1 RPT-HTTPClient/0.3-1 872 | JDK/1.1 873 | Jeode/1.x.x 874 | Jetbot/1.0 875 | JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/) 876 | JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/) 877 | JetCar 878 | Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0 879 | JoBo/1.x (http://www.matuschek.net/jobo.html) 880 | JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html) 881 | JobSpider_BA/1.1 882 | JOC Web Spider 883 | JordoMedia/1.0 RSS File Reader (http://www.jordomedia.com) 884 | Journster [alpha] (http://journster.com/) 885 | Journster.com RSS/Atom aggregator 0.5 (http://www.journster.com/bot.phtml) 886 | JRTS Check Favorites Utility 887 | JRTwine Software Check Favorites Utility 888 | Jyxobot/x 889 | K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011 890 | k2spider 891 | KAIST AITrc Crawler 892 | KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com) 893 | kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) 894 | kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com) 895 | Kapere (http://www.kapere.com) 896 | Kazehakase/0.x.x.[x] 897 | KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) 898 | Kenjin Spider 899 | Kevin http://dznet.com/kevin/ 900 | Kevin http://websitealert.net/kevin/ 901 | KE_1.0/2.0 libwww/5.2.8 902 | KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de) 903 | kinja-imagebot (http://www.kinja.com/) 904 | kinjabot (http://www.kinja.com) 905 | KIT-Fireball/2.0 906 | KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5) 907 | Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0) 908 | KnowItAll(knowitall@cs.washington.edu) 909 | Knowledge.com/0.x 910 | Kontiki Client x.xx 911 | Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com) 912 | KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com) 913 | kuloko-bot/0.x 914 | kulokobot www.kuloko.com kuloko@backweave.com 915 | kulturarw3/0.1 916 | KummHttp/1.1 (compatible; KummClient; Linux rulez) 917 | KWC-KX9/1109 UP.Browser/6.2.3.9.g.1.107 (GUI) MMP/2.0 UP.Link/6.3.0.0.0 918 | Labrador/0.2; http://ir.dcs.gla.ac.uk/labrador; craigm@dcs.gla.ac.uk 919 | Lachesis 920 | lanshanbot/1.0 921 | lanshanbot/1.0 (+http://search.msn.com/msnbot.htm) 922 | LapozzBot/1.4 ( http://robot.lapozz.com) 923 | LapozzBot/1.5 (+http://robot.lapozz.hu) 924 | larbin (samualt9@bigfoot.com) 925 | LARBIN-EXPERIMENTAL (efp@gmx.net) 926 | larbin_2.1.1 larbin2.1.1@somewhere.com 927 | larbin_2.2.0 (crawl@compete.com) 928 | larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) 929 | larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) 930 | larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) 931 | larbin_2.6.0 (larbin2.6.0@unspecified.mail) 932 | larbin_2.6.1 (larbin2.6.1@unspecified.mail) 933 | larbin_2.6.2 (hamasaki@grad.nii.ac.jp) 934 | larbin_2.6.2 (larbin2.6.2@unspecified.mail) 935 | larbin_2.6.2 (listonATccDOTgatechDOTedu) 936 | larbin_2.6.2 (pimenas@systems.tuc.gr) 937 | larbin_2.6.2 (tom@lemurconsulting.com) 938 | larbin_2.6.2 (vitalbox1@hotmail.com) 939 | larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) 940 | larbin_2.6.3 (wgao@genieknows.com) 941 | larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi 942 | larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) 943 | larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/) 944 | lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com) 945 | lc/$ROADS::Version libwww-perl/5.00 946 | lcabotAccept: */* 947 | LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; robot@yoriwa.com) 948 | LECodeChecker/3.0 libgetdoc/1.0 949 | LeechGet 200x (www.leechget.de) 950 | LEIA/2.90 951 | LEIA/3.01pr (LEIAcrawler; [SNIP]) 952 | LetsCrawl.com/1.0 +http://letscrawl.com/ 953 | LexiBot/1.00 954 | LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 955 | LG/U8138/v1.0 956 | Libby_1.1/libwww-perl/5.47 957 | libcurl-agent/1.0 958 | LibertyW (+http://www.lw01.com) 959 | libWeb/clsHTTP -- hiongun@kt.co.kr 960 | libwww-perl/5.41 961 | libwww-perl/5.45 962 | libwww-perl/5.48 963 | libwww-perl/5.50 964 | libwww-perl/5.52 FP/2.1 965 | libwww-perl/5.52 FP/4.0 966 | libwww-perl/5.53 967 | libwww-perl/5.63 968 | libwww-perl/5.64 969 | libwww-perl/5.65 970 | libwww-perl/5.800 971 | libwww/5.3.2 972 | Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/) 973 | Liferea/1.x.x (Linux; es_ES.UTF-8; http://liferea.sf.net/) 974 | LightningDownload/1.0beta2 975 | LightningDownload/1.x.x 976 | LightningDownload/1.x.x [Accelerated x] 977 | LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) 978 | Lincoln State Web Browser 979 | Link Valet Online 1.x 980 | LinkAlarm/2.x 981 | Linkbot 982 | linkbot 983 | Linkbot x.0 984 | LinkCheck (linkcheck@inter7.com http://www.inter7.com/linkcheck) 985 | LinkLint-checkonly/2.x.x 986 | LinkLint-spider/2.x.x 987 | linknzbot 988 | LinkPimpin v1.0 989 | LinkProver 2.1 990 | Links (0.9x; Linux 2.4.7-10 i686) 991 | Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24) 992 | Links (2.xpre7; Linux 2.4.18 i586; x) 993 | Links - http://gossamer-threads.com/scripts/links/ 994 | Links 2.0 (http://gossamer-threads.com/scripts/links/) 995 | Links SQL (http://gossamer-threads.com/scripts/links-sql/) 996 | Links4US-Crawler, (+http://links4us.com/) 997 | LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo) 998 | LinkScan/9.0g Unix 999 | LinkScan/x.x Unix 1000 | LinksManager.com (http://linksmanager.com/linkchecker.html) 1001 | LinkSonar/1.35 1002 | LinkSweeper/1.x 1003 | LinkWalker 1004 | link_check3.plx libwww-perl/5.65 1005 | ListBidBot (freelance job spider http://listbid.com)<a href=http://listbid.com 1006 | LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/) 1007 | Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) 1008 | LMQueueBot/0.2 1009 | lmspider (lmspider@scansoft.com) 1010 | LNSpiderguy 1011 | LocalBot/1.0 ( http://www.localbot.co.uk/) 1012 | LocalcomBot/1.2.x ( http://www.local.com/bot.htm) 1013 | Lockstep Spider/1.0 1014 | Look.com 1015 | Lotus-Notes/4.5 ( Windows-NT ) 1016 | LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT) 1017 | Lovel as 1.0 ( +http://www.everatom.com) 1018 | LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu) 1019 | LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) 1020 | luchs.at URL checker 1021 | Lunascape 1022 | lwp-trivial/1.32 1023 | lwp-trivial/1.34 1024 | lwp-trivial/1.35 1025 | LWP::Simple/5.22 1026 | LWP::Simple/5.36 1027 | LWP::Simple/5.48 1028 | LWP::Simple/5.50 1029 | LWP::Simple/5.51 1030 | LWP::Simple/5.53 1031 | LWP::Simple/5.63 1032 | LWP::Simple/5.803 1033 | Lycos_Spider_(modspider) 1034 | Lycos_Spider_(T-Rex) 1035 | Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04) 1036 | Lynx/2.6 libwww-FM/2.14 1037 | Lynx/2.8 (;http://seebot.org) 1038 | Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6 1039 | Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net) 1040 | Mac Finder 1.0.xx 1041 | Mackster( http://www.ukwizz.com ) 1042 | Mag-Net 1043 | MagicWML/1.0 (forcewml) 1044 | MagpieRSS/0.7x (+http://magpierss.sf.net) 1045 | Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com) 1046 | Mail.Ru/1.0 1047 | mailto:webcraft@bea.com 1048 | mammoth/1.0 ( http://www.sli-systems.com/) 1049 | MantraAgent 1050 | MapoftheInternet.com ( http://MapoftheInternet.com) 1051 | Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) 1052 | Marketwave Hit List 1053 | Martini 1054 | MARTINI 1055 | Marvin v0.3 1056 | MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) 1057 | Mass Downloader 2.x 1058 | MasterSeek 1059 | Mata Hari/2.00 1060 | Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address) 1061 | maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) 1062 | McBot/5.001 (windows; U; NT4.0; en-us) 1063 | MDbot/1.0 (+http://www.megadownload.net/bot.html) 1064 | Media Player Classic 1065 | MediaCrawler-1.0 (Experimental) 1066 | Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) 1067 | MediaSearch/0.1 1068 | MegaSheep v1.0 (www.searchuk.com internet sheep) 1069 | Megite2.0 (http://www.megite.com) 1070 | Mercator-1.x 1071 | Mercator-2.0 1072 | Mercator-Scrub-1.1 1073 | Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com) 1074 | MetaGer-LinkChecker 1075 | MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) 1076 | MetaGer_PreChecker0.1 1077 | MetaProducts Download Express/1.x 1078 | Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) 1079 | metatagsdir/0.7 (+http://metatagsdir.com/directory/) 1080 | MFC Foundation Class Library 4.0 1081 | MFC_Tear_Sample 1082 | MFHttpScan 1083 | MicroBaz 1084 | Microsoft Data Access Internet Publishing Provider Cache Manager 1085 | Microsoft Data Access Internet Publishing Provider DAV 1086 | Microsoft Data Access Internet Publishing Provider Protocol Discovery 1087 | Microsoft Log Parser 2.2 1088 | Microsoft Small Business Indexer 1089 | Microsoft URL Control - 6.00.8xxx 1090 | MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) 1091 | Microsoft_Internet_Explorer_5.00.438 (fjones@isd.net) 1092 | MIIxpc/4.2 1093 | Mindjet MindManager 1094 | minibot 1095 | miniRank/1.6 (Website ranking; www.minirank.com; robot) 1096 | MiracleAlphaTest 1097 | Missauga Locate 1.0.0 1098 | Missigua Locator 1.9 1099 | Missouri College Browse 1100 | Mister Pix II 2.02a 1101 | Mister PiX version.dll 1102 | Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr) 1103 | Miva (AlgoFeedback@miva.com) 1104 | Mizzu Labs 2.2 1105 | MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+) 1106 | MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) 1107 | MJBot (SEO assessment) 1108 | MLBot (www.metadatalabs.com) 1109 | MnogoSearch/3.2.xx 1110 | Mo College 1.9 1111 | moget/x.x (moget@goo.ne.jp) 1112 | mogimogi/1.0 1113 | moiNAG 0.02 1114 | MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) 1115 | monkeyagent 1116 | MoonBrowser (version 0.41 Beta4) 1117 | Moreoverbot/x.00 (+http://www.moreover.com) 1118 | Morris - Mixcat Crawler ( http://mixcat.com) 1119 | Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml) 1120 | Motorola-V3m Obigo 1121 | Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) 1122 | MovableType/x.x 1123 | mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) 1124 | Mozi! 1125 | Mozilla 1126 | Mozilla (libwhisker/2.4) 1127 | Mozilla (Mozilla@somewhere.com) 1128 | Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) 1129 | Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE) 1130 | Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10) 1131 | Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT) 1132 | Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2 1133 | Mozilla/2.0 1134 | Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC) 1135 | Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16) 1136 | Mozilla/2.0 (compatible; Ask Jeeves) 1137 | Mozilla/2.0 (compatible; Ask Jeeves/Teoma) 1138 | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) 1139 | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html) 1140 | Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) 1141 | Mozilla/2.0 (compatible; MS FrontPage x.0) 1142 | Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC) 1143 | Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT) 1144 | Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95) 1145 | Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95) 1146 | Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1) 1147 | Mozilla/2.0 (compatible; MSIE 3.0B; Win32) 1148 | Mozilla/2.0 (compatible; NEWT ActiveX; Win32) 1149 | Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) 1150 | Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98) 1151 | Mozilla/2.01 (Win16; I) 1152 | Mozilla/2.02Gold (Win95; I) 1153 | Mozilla/3.0 (compatible) 1154 | Mozilla/3.0 (compatible; AvantGo 3.2) 1155 | Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com) 1156 | Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 1.0.1.34) 1157 | Mozilla/3.0 (compatible; Indy Library) 1158 | Mozilla/3.0 (compatible; Linkman) 1159 | Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) 1160 | Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) 1161 | Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) 1162 | Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64) 1163 | Mozilla/3.0 (compatible; NetPositive/2.2) 1164 | Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1 1165 | Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2 1166 | Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95) 1167 | Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x) 1168 | Mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail) 1169 | Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) 1170 | Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator http://www.relsoftware.com/ link validation software 1171 | Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows) 1172 | Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) 1173 | Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) 1174 | Mozilla/3.0 (Compatible;Viking/1.8) 1175 | Mozilla/3.0 (DreamPassport/3.0) 1176 | Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) 1177 | Mozilla/3.0 (Liberate DTV 1.1) 1178 | Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US) 1179 | Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1180 | Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1181 | Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1182 | Mozilla/3.0 (Vagabondo/1.1 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) 1183 | Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.nl/) 1184 | Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) 1185 | Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) 1186 | Mozilla/3.0 (Win16; I) 1187 | Mozilla/3.0 (Win95; I) 1188 | Mozilla/3.0 (WinNT; I) 1189 | Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE) 1190 | Mozilla/3.0 (X11; I; OSF1 V4.0 alpha) 1191 | Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK) 1192 | Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0) 1193 | Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0) 1194 | Mozilla/3.01 (Compatible; Links2Go Similarity Engine) 1195 | Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2) 1196 | Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC) 1197 | Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486) 1198 | Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m) 1199 | Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32) 1200 | Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02) 1201 | Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309) 1202 | Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon) 1203 | Mozilla/3.x (I-Opener 1.1; Netpliance) 1204 | Mozilla/4.0 1205 | Mozilla/4.0 (agadine3.0) www.agada.de 1206 | Mozilla/4.0 (Compatible); URLBase 6 1207 | Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) 1208 | Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 1209 | Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 1210 | Mozilla/4.0 (compatible; <a href=http://www.reget.com 1211 | Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx) 1212 | Mozilla/4.0 (compatible; Arachmo) 1213 | Mozilla/4.0 (compatible; BorderManager 3.0) 1214 | Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org) 1215 | Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) 1216 | Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0) 1217 | Mozilla/4.0 (compatible; Check&Get 3.0; Windows NT) 1218 | Mozilla/4.0 (compatible; ChristCrawler.com ChristCrawler@ChristCENTRAL.com) 1219 | Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) 1220 | Mozilla/4.0 (compatible; DAUMOA-video; +http://ws.daum.net/aboutkr.html) 1221 | Mozilla/4.0 (compatible; DepSpid/5.0x; +http://about.depspid.net) 1222 | Mozilla/4.0 (compatible; DnloadMage 1.0) 1223 | Mozilla/4.0 (compatible; FastCrawler3 support-fastcrawler3@fast.no) 1224 | Mozilla/4.0 (compatible; FDSE robot) 1225 | Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php) 1226 | Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub.org) 1227 | Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http://grub.org) 1228 | Mozilla/4.0 (compatible; grub-client-2.x) 1229 | Mozilla/4.0 (compatible; ibisBrowser) 1230 | Mozilla/4.0 (compatible; ICS 1.2.xxx) 1231 | Mozilla/4.0 (compatible; IE-Favorites-Check-0.5) 1232 | Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at) 1233 | Mozilla/4.0 (compatible; KeepNI web site monitor) 1234 | Mozilla/4.0 (compatible; Link Utility; http://net-promoter.com) 1235 | Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT) 1236 | Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98) 1237 | Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC) 1238 | Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95) 1239 | Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) 1240 | Mozilla/4.0 (compatible; MSIE 4.01; Windows 95) 1241 | Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq) 1242 | Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com) 1243 | Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320) 1244 | Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE) 1245 | Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) 1246 | Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft 1247 | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR 1248 | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive 1249 | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest Designs 1250 | Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via <B 1251 | Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) 1252 | Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1) 1253 | Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; sureseeker.com) 1254 | Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021) 1255 | Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) 1256 | Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17 1257 | Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 1258 | Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/) 1259 | Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 1260 | Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] 1261 | Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx http://www.kyosoft.com) 1262 | Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent 1263 | Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) 1264 | Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com) 1265 | Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com/; http://www.galaxy.com/info/crawler.html) 1266 | Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) 1267 | Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.com)) 1268 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0) 1269 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x) 1270 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo) 1271 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL) 1272 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0) 1273 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1) 1274 | Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler) 1275 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) 1276 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) 1277 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache Request 1278 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request 1279 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705) 1280 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF) 1281 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5) 1282 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a) 1283 | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E 1284 | Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.10 1285 | Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0) 1286 | Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2) 1287 | Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] 1288 | Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html) 1289 | Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0 1290 | Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x) 1291 | Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x) 1292 | Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.com.au) 1293 | Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; support@illumit.com; http://www.illumit.com/Products/weblight/) 1294 | Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; http://www.Abolimba.de) 1295 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3) 1296 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;) 1297 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html) 1298 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test; http://tuezilla.de/test-odp-links-agent.html) 1299 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322) 1300 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (dns_admin@c-a-s-h.com) 1301 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;) 1302 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5)) 1303 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer) 1304 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 http://www.cs.washington.edu/research/networking/websys/) 1305 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0) 1306 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M) 1307 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM) 1308 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0) 1309 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0) 1310 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon) 1311 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler) 1312 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04) 1313 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net) 1314 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1) 1315 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; StumbleUpon.com 1.760; .NET CLR 1.1.4322) 1316 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler) 1317 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html ) 1318 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) 1319 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser 5.0.0.0) 1320 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (0.4.1.98); ezPeer+ v1.0 (0.5.0.00); .NET CLR 1.1.4322; MSIECrawler) 1321 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) 1322 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218)) 1323 | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) 1324 | Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0 1325 | Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft) 1326 | Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727) 1327 | Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0) 1328 | Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648) 1329 | Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0) 1330 | Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3) 1331 | Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html) 1332 | Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea) 1333 | Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) 1334 | Mozilla/4.0 (compatible; Netcraft Web Server Survey) 1335 | Mozilla/4.0 (compatible; NetPromoter Spider;http://www.net-promoter.com/) 1336 | Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en] 1337 | Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) 1338 | Mozilla/4.0 (compatible; RSS Popper) 1339 | Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0) 1340 | Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com) 1341 | Mozilla/4.0 (compatible; SPENG) 1342 | Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98) 1343 | Mozilla/4.0 (compatible; Synapse) 1344 | Mozilla/4.0 (compatible; WebCapture 3.0; Windows) 1345 | Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5) 1346 | Mozilla/4.0 (compatible; WSN Links) 1347 | Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) 1348 | Mozilla/4.0 (compatible; www.galaxy.com) 1349 | Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT) 1350 | Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) 1351 | Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) 1352 | Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818) 1353 | Mozilla/4.0 (fantomBrowser) 1354 | Mozilla/4.0 (fantomCrew Browser) 1355 | Mozilla/4.0 (hhjhj@yahoo.com) 1356 | Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) 1357 | Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0 1358 | Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) 1359 | Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html) 1360 | Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com) 1361 | Mozilla/4.0 (Sleek Spider/1.2) 1362 | Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net) 1363 | Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) 1364 | Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) 1365 | Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) 1366 | Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) 1367 | Mozilla/4.0 efp@gmx.net 1368 | Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0) 1369 | Mozilla/4.0 [en] (Ask Jeeves Corporate Spider) 1370 | Mozilla/4.0(compatible; Zealbot 1.0) 1371 | Mozilla/4.01 (compatible; NORAD National Defence Network) 1372 | Mozilla/4.01 [en](Win95;I) 1373 | Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u) 1374 | Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp) 1375 | Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712) 1376 | Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22) 1377 | Mozilla/4.05 (Macintosh; I; 68K Nav) 1378 | Mozilla/4.05 (Macintosh; I; PPC Nav) 1379 | Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m) 1380 | Mozilla/4.08 [en] (Win98; U ;Nav) 1381 | Mozilla/4.08 [en] (WinNT; U) 1382 | Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8 1383 | Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2 1384 | Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) 1385 | Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC) 1386 | Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC) 1387 | Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC) 1388 | Mozilla/4.5 RPT-HTTPClient/0.3-2 1389 | Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I) 1390 | Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586) 1391 | Mozilla/4.6 [en] (http://www.cnet.com/) 1392 | Mozilla/4.61 [de] (OS/2; I) 1393 | Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows) 1394 | Mozilla/4.7 1395 | Mozilla/4.7 (compatible; http://eidetica.com/spider) 1396 | Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com) 1397 | Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2 1398 | Mozilla/4.7 (compatible; Whizbang) 1399 | Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler) 1400 | Mozilla/4.7 [en](BecomeBot@exava.com) 1401 | Mozilla/4.7 [en](Exabot@exava.com) 1402 | Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I) 1403 | Mozilla/4.72 [en] (BACS http://www.ba.be) 1404 | Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686) 1405 | Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha) 1406 | Mozilla/4.76 (Windows 98; U) Opera 5.12 [en] 1407 | Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386) 1408 | Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u) 1409 | Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32) 1410 | Mozilla/5.0 1411 | Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 1412 | Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 1413 | Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/) 1414 | Mozilla/5.0 (compatible) GM RSS Panel X 1415 | Mozilla/5.0 (compatible; +http://www.evri.com/evrinid) 1416 | Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 1417 | Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com) 1418 | Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) 1419 | Mozilla/5.0 (compatible; AnsearchBot/1.x; +http://www.ansearch.com.au/) 1420 | Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html) 1421 | Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org) 1422 | Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane Katrina 1423 | Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) 1424 | Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; +http://www.banbots.com) 1425 | Mozilla/5.0 (compatible; BecomeBot/1.23; http://www.become.com/webmasters.html) 1426 | Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; http://www.become.com/webmasters.html) 1427 | Mozilla/5.0 (compatible; BecomeBot/2.0beta; http://www.become.com/webmasters.html) 1428 | Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; http://www.become.com/site_owners.html) 1429 | Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html) 1430 | Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers) 1431 | Mozilla/5.0 (compatible; Bot; +http://pressemitteilung.ws/spamfilter 1432 | Mozilla/5.0 (compatible; BuzzRankingBot/1.0; +http://www.buzzrankingbot.com/) 1433 | Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) 1434 | Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) 1435 | Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) 1436 | Mozilla/5.0 (compatible; Custo 3 (Netwu.com); Windows NT 5.1) 1437 | Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) 1438 | Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com) 1439 | Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com) 1440 | Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) 1441 | Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info) 1442 | Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu) 1443 | Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) 1444 | Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) 1445 | Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml) 1446 | Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html) 1447 | mozilla/5.0 (compatible; genevabot http://www.healthdash.com) 1448 | Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12 1449 | Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) 1450 | mozilla/5.0 (compatible; heritrix/1.0.4 http://innovationblog.com) 1451 | Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) 1452 | Mozilla/5.0 (compatible; heritrix/1.12.1 +http://newstin.com/) 1453 | Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) 1454 | Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com] 1455 | mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org) 1456 | Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) 1457 | Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) 1458 | Mozilla/5.0 (compatible; heritrix/1.5.0 http://www.l3s.de/~kohlschuetter/projects/crawling/) 1459 | Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 http://pandora.nla.gov.au/crawl.html) 1460 | Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) 1461 | Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) 1462 | Mozilla/5.0 (compatible; Heritrix/1.8.0 http://www.hanzoarchives.com) 1463 | Mozilla/5.0 (compatible; heritrix/1.x.x +http://www.accelobot.com) 1464 | Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com) 1465 | Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) 1466 | Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ ) 1467 | Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ ) 1468 | Mozilla/5.0 (compatible; HyperixScoop/1.3; +http://www.hyperix.com) 1469 | Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0) 1470 | Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) 1471 | Mozilla/5.0 (compatible; InterseekWeb/3.x) 1472 | Mozilla/5.0 (compatible; Jim +http://www.hanzoarchives.com) 1473 | Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encoding 1474 | Mozilla/5.0 (compatible; Konqueror/2.1.1; X11) 1475 | Mozilla/5.0 (compatible; Konqueror/2.2.2) 1476 | Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686) 1477 | Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) 1478 | Mozilla/5.0 (compatible; LemSpider 0.1) 1479 | Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html) 1480 | Mozilla/5.0 (compatible; LinkStash Bookmark Manager; http://www.xrayz.co.uk/) 1481 | Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) 1482 | Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker) 1483 | Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) 1484 | Mozilla/5.0 (compatible; OnetSzukaj/5.0; http://szukaj.onet.pl) 1485 | Mozilla/5.0 (compatible; PagestackerBot; http://www.pagestacker.com) 1486 | Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001 1487 | Mozilla/5.0 (compatible; PEAR HTTP_Request class; http://feed.moo.jp/) 1488 | Mozilla/5.0 (compatible; Phonifier; +http://www.phonifier.com) 1489 | Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) 1490 | Mozilla/5.0 (compatible; pogodak.ba/3.x) 1491 | Mozilla/5.0 (compatible; Pogodak.hr/3.1) 1492 | Mozilla/5.0 (compatible; Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html) 1493 | Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) 1494 | Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) 1495 | Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ ) 1496 | Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) 1497 | Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) 1498 | Mozilla/5.0 (compatible; ShunixBot/1.x.x +http://www.shunix.com/robot.htm) 1499 | Mozilla/5.0 (compatible; ShunixBot/1.x; http://www.shunix.com/bot.htm) 1500 | Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) 1501 | Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 1502 | Mozilla/5.0 (compatible; SpurlBot/0.2) 1503 | Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) 1504 | Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;) 1505 | Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) 1506 | Mozilla/5.0 (compatible; Theophrastus/x.x; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php) 1507 | Mozilla/5.0 (compatible; TridentSpider/3.1) 1508 | Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 1509 | Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) 1510 | Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - http://www.monsterli.ch/phpwebbrain/) 1511 | Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) 1512 | Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0;http://www.wowlemmings.com/kathune.html) 1513 | Mozilla/5.0 (compatible; XTbot/1.0v; +http://www.externaltest.com) 1514 | Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 1515 | Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) 1516 | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 1517 | Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) 1518 | Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) 1519 | Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) 1520 | Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/) 1521 | Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com] 1522 | Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org) 1523 | Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 +http://pandora.nla.gov.au/crawl.html) 1524 | Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm) 1525 | Mozilla/5.0 (compatible;MAINSEEK_BOT) 1526 | Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/) 1527 | Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b5) Gecko/2008032619 Firefox/3.0b5 1528 | Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20021219 Chimera/0.6 1529 | Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20030306 Camino/0.7 1530 | Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/xx (KHTML like Gecko) OmniWeb/v5xx.xx 1531 | Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/xxx.x (KHTML like Gecko) Safari/12x.x 1532 | Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 1533 | Mozilla/5.0 (research@mediatrec.com) 1534 | Mozilla/5.0 (Sage) 1535 | Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1536 | Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1537 | Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en] 1538 | Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) 1539 | Mozilla/5.0 (Version: xxxx Type:xx) 1540 | Mozilla/5.0 (wgao@genieknows.com) 1541 | Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 1542 | Mozilla/5.0 (Windows; U; Win98; en-US; rv:x.xx) Gecko/20030423 Firebird Browser/0.6 1543 | Mozilla/5.0 (Windows; U; Win9x; en; Stable) Gecko/20020911 Beonex/0.8.1-stable 1544 | Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com 1545 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.2.153.1 Safari/525.19 1546 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060731 Firefox/1.5.0.5 Flock/0.7.4.1 1547 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.4/Megaupload x.0 1548 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1 Orca/1.1 beta 3 1549 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x 1550 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xx) Gecko/20030504 Mozilla Firebird/0.6 1551 | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xxx) Gecko/20041027 Mnenhy/0.6.0.104 1552 | Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) 1553 | Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5 1554 | Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3 1555 | Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@health 1556 | Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com 1557 | Mozilla/5.0 (X11; Linux i686; U;rv: 1.7.13) Gecko/20070322 Kazehakase/0.4.4.1 1558 | Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6/6.01 1559 | Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.0.2) Gecko/20060309 SeaMonkey/1.0 1560 | Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2) 1561 | Mozilla/5.0 (X11; U; Linux i686; en-US; Nautilus/1.0Final) Gecko/20020408 1562 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801 1563 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130 1564 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3 1565 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Epiphany/1.2.1 1566 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot 1567 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061129 BonEcho/2.0 1568 | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20061205 Iceweasel/2.0.0.1 (Debian-2.0.0.1+dfsg-2) 1569 | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9a8) Gecko/2007100619 GranParadiso/3.0a8 1570 | Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224 1571 | Mozilla/5.0 gURLChecker/0.x.x (Linux) 1572 | Mozilla/5.0 URL-Spider 1573 | Mozilla/5.0 usww.com-Spider-for-w8.net 1574 | Mozilla/5.0 wgao@genieknows.com 1575 | Mozilla/5.0 whoiam [http://www.axxus.de/] 1576 | Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) 1577 | MQbot metaquerier.cs.uiuc.edu/crawler 1578 | MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) 1579 | MSFrontPage/4.0 1580 | MSIE 4.0 (Win95) 1581 | MSIE-5.13 (larbin@unspecified.mail) 1582 | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) 1583 | msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) 1584 | MSNBOT/0.xx (http://search.msn.com/msnbot.htm) 1585 | msnbot/x.xx ( http://search.msn.com/msnbot.htm) 1586 | MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) 1587 | MSNPTC/1.0 1588 | MSProxy/2.0 1589 | MSRBOT 1590 | MSRBOT (http://research.microsoft.com/research/sv/msrbot) 1591 | Mulder, VCR-1.0 1592 | multiBlocker browser 1593 | multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) 1594 | MultiText/0.1 1595 | MusicWalker2.0 ( http://www.somusical.com) 1596 | MVAClient 1597 | My WinHTTP Connection 1598 | myDaemon 1599 | MyGetRight/1.0.0 1600 | MyGetRight/1.0b 1601 | Mylinea.com Crawler 2.0 1602 | mylinkcheck/1.02 1603 | Naamah 1.0.1/Blogbot (http://blogbot.de/) 1604 | Naamah 1.0a/Blogbot (http://blogbot.de/) 1605 | NABOT/5.0 1606 | nabot_1.0 1607 | NameOfAgent (CMS Spider) 1608 | naoFavicon4IE/1.xx 1609 | NASA Search 1.0 1610 | NationalDirectory-WebSpider/1.3 1611 | NationalDirectoryAddURL/1.0 1612 | NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) 1613 | NaverBot_dloader/1.5 1614 | NavissoBot 1615 | NavissoBot/1.7 (+http://navisso.com/) 1616 | NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) 1617 | Nebullabot/2.2 (http://bot.nebulla.info) 1618 | NEC Research Agent -- compuman at research.nj.nec.com 1619 | NEC-Hayek/1.0 1620 | Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com) 1621 | NetAnts/1.2x 1622 | NETCOMplete/x.xx 1623 | NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) 1624 | NetLookout/2.24 1625 | Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) 1626 | NetMechanic Vx.0 1627 | NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/) 1628 | NetNoseCrawler/v1.0 1629 | Netprospector JavaCrawler 1630 | NetPumper/x.xx 1631 | NetResearchServer(http://www.look.com) 1632 | NetResearchServer/x.x(loopimprovements.com/robot.html) 1633 | NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com) 1634 | NetSprint -- 2.0 1635 | NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) 1636 | NetZippy 1637 | NeuralBot/0.2 1638 | newsearchengine (ThisUser@unspecified.mail) 1639 | NewsGator FetchLinks extension/0.2.0 (http://graemef.com) 1640 | NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers) 1641 | NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot) 1642 | NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x 1643 | NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) 1644 | NG/1.0 1645 | NG/4.0.1229 1646 | nicebot 1647 | NICO/1.0 1648 | Nikita the Spider (http://NikitaTheSpider.com/) 1649 | NITLE Blog Spider/0.01 1650 | Nitro Downloader 1.x (www.klsofttools.com) 1651 | Noago Spider 1652 | Nocilla/1.0 1653 | Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com 1654 | Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi) 1655 | Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) 1656 | Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0) 1657 | NokodoBot/1.x (+http://nokodo.com/bot.htm) 1658 | Norbert the Spider(Burf.com) 1659 | noxtrumbot/1.0 (crawler@noxtrum.com) 1660 | noyona_0_1 1661 | NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) 1662 | NPBot (http://www.nameprotect.com/botinfo.html) 1663 | NPBot-1/2.0 1664 | Nsauditor/1.x 1665 | NSPlayer/10.0.0.xxxx WMFSDK/10.0 1666 | nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com) 1667 | nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com) 1668 | nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) 1669 | Nucleus SiteList LinkChecker/1.1 1670 | nuSearch Spider <a href='http://www.nusearch.com' 1671 | NuSearch Spider (compatible; MSIE 6.0) 1672 | NuSearch Spider www.nusearch.com 1673 | Nutch 1674 | Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com) 1675 | Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu ) 1676 | NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) 1677 | NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) 1678 | NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) 1679 | NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) 1680 | NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) 1681 | nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) 1682 | NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com) 1683 | obidos-bot (just looking for books.) 1684 | ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) 1685 | ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) 1686 | oBot ((compatible;Win32)) 1687 | Ocelli/1.x (http://www.globalspec.com/Ocelli) 1688 | Octopus 1689 | Octora Beta - www.octora.com 1690 | Octora Beta Bot - www.octora.com 1691 | Offline Explorer 1.* 1692 | OliverPerry 1693 | OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car & shopping search (64.62.175.xxx) 1694 | OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler 1695 | OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler 1696 | OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer 1697 | onCHECK Linkchecker von www.scientec.de fuer www.onsinn.de 1698 | onCHECK-Robot, www.onsearch.de 1699 | Onet.pl SA- http://szukaj.onet.pl 1700 | online link validator (http://www.dead-links.com/) 1701 | Online24-Bot (Version: 1.0x, powered by www.online24.de) 1702 | OntoSpider/1.0 libwww-perl/5.65 1703 | OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) 1704 | OpenAcoon v4.0.x (www.openacoon.de) 1705 | Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) 1706 | Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) 1707 | Openfind Robot/1.1A2 1708 | OpenISearch/1.x (www.openisearch.com) 1709 | OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) 1710 | OpenTextSiteCrawler/2.9.2 1711 | OpenWebSpider/0.x.x (http://www.openwebspider.org) 1712 | OpenWebSpider/x 1713 | Opera/5.0 (Linux 2.0.38 i386; U) [en] 1714 | Opera/5.11 (Windows ME; U) [ru] 1715 | Opera/5.12 (Windows 98; U) [en] 1716 | Opera/6.01 (larbin@unspecified.mail) 1717 | Opera/6.x (Linux 2.4.8-26mdk i686; U) [en] 1718 | Opera/6.x (Windows NT 4.0; U) [de] 1719 | Opera/7.x (Windows NT 5.1; U) [en] 1720 | Opera/8.xx (Windows NT 5.1; U; en) 1721 | Opera/9.0 (Windows NT 5.1; U; en) 1722 | Opera/9.00 (Windows NT 5.1; U; de) 1723 | Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1 1724 | OpidooBOT (larbin2.6.3@unspecified.mail) 1725 | OPWV-SDK UP.Browser/7.0.2.3.119 (GUI) MMP/2.0 Push/PO 1726 | Oracle Application Server Web Cache 10g 1727 | Oracle iMTCrawler 1728 | Oracle Ultra Search 1729 | OrangeSpider 1730 | Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) 1731 | Orca Browser (http://www.orcabrowser.com) 1732 | OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54) 1733 | OutfoxBot/0.x (For internet experiments; http://; outfox.agent@gmail.com) 1734 | OutfoxMelonBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com) 1735 | Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 1736 | OWR_Crawler 0.1 1737 | ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) 1738 | PADLibrary Spider 1739 | PageBitesHyperBot/600 (http://www.pagebites.com/) 1740 | Pagebull http://www.pagebull.com/ 1741 | Pagestacker Bot 1742 | page_verifier (http://www.securecomputing.com/goto/pv) 1743 | PagmIEDownload 1744 | parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 1745 | ParaSite/1.0b (http://www.ianett.com/parasite/) 1746 | Patwebbot (http://www.herz-power.de/technik.html) 1747 | pavuk/0.9pl29b i686-pc-linux-gnu 1748 | PBrowse 1.4b 1749 | pd02_1.0.0 pd02_1.0.0@dzimi@post.sk 1750 | PEAR HTTP_Request class ( http://pear.php.net/ ) 1751 | PEERbot www.peerbot.com 1752 | PeopleChat/Search_Engine 1753 | PEval 1.4b 1754 | PHP/3.x.xx 1755 | PHP/4.0.4pl1 1756 | PHP/4.0.6 1757 | PHP/4.1.1 1758 | PHP/4.1.2 1759 | PicoSearch/1.0 1760 | Piffany_Web_Scraper_v0.x 1761 | Piffany_Web_Spider_v0.x 1762 | PigeonBot1.0 BETA 1763 | PingALink Monitoring Services 1.0 1764 | PingALink Monitoring Services 1.0 (http://www.pingalink.com) 1765 | Pingdom GIGRIB (http://www.pingdom.com) 1766 | pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com) 1767 | pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) 1768 | Pita 1769 | Pizilla++ ver 2.45 1770 | PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) 1771 | Plagger/0.x.xx (http://plagger.org/) 1772 | PlagiarBot/1.0 1773 | PlantyNet_WebRobot_V1.9 dhkang@plantynet.com 1774 | plinki/0.1 (you got plinked! (thats a good thing..); http://www.plinki.com; crawl@plinki.com) 1775 | PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) 1776 | Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com) 1777 | Pockey-GetHTML/4.12.0 (Win32; GUI; ix86) 1778 | Pockey-GetHTML/x.xx 1779 | Pockey/x.x.x 1780 | Pockey7.x.x(WIN32GUI) 1781 | POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000) 1782 | Poirot 1783 | polybot 1.0 (http://cis.poly.edu/polybot/) 1784 | Pompos/1.x http://dir.com/pompos.html 1785 | Pompos/1.x pompos@iliad.fr 1786 | Popdexter/1.0 1787 | Port Huron Labs 1788 | PortalBSpider/2.0 (spider@portalb.com) 1789 | portalmmm/2.0 S500i(c20;TB) 1790 | PostFavorites 1791 | potbot 1.0 1792 | PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) 1793 | PrivacyFinder Cache Bot v1.0 1794 | PrivacyFinder/1.1 1795 | Privoxy/3.0 (Anonymous) 1796 | Production Bot 0116B 1797 | Production Bot 2016B 1798 | Production Bot DOT 3016B 1799 | Program Shareware 1.0.2 1800 | Progressive Download 1801 | Progressive Download HTTP check 1802 | Project XP5 [2.03.07-111203] 1803 | PROve AnswerBot 4.0 1804 | ProWebGuide Link Checker (http://www.prowebguide.com) 1805 | psbot/0.1 (+http://www.picsearch.com/bot.html) 1806 | PSurf15a 11 1807 | PSurf15a 51 1808 | PSurf15a VA 1809 | psycheclone 1810 | PubCrawl (pubcrawl.stanford.edu) 1811 | puf/0.91beta6a (Linux 2.2.18; i686) 1812 | puf/0.93.2a (Linux 2.4.18; i686) 1813 | pulseBot (pulse Web Miner) 1814 | PureSight 1815 | PuxaRapido v1.0 1816 | PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) 1817 | PycURL 1818 | PycURL/7.xx.x 1819 | Python-urllib/1.1x 1820 | Python-urllib/2.0a1 1821 | Qango.com Web Directory (http://www.qango.com/) 1822 | QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavis 1823 | QPCreep Test Rig ( We are not indexing- just testing ) 1824 | QuepasaCreep ( crawler@quepasacorp.com ) 1825 | QuepasaCreep v0.9.1x 1826 | QueryN Metasearch 1827 | Quicksilver (Blacktree,MacOSX) 1828 | QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9) 1829 | QweeryBot/3.01 ( http://qweerybot.qweery.nl) 1830 | Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) 1831 | R6_CommentReader_(www.radian6.com/crawler) 1832 | R6_FeedFetcher_(www.radian6.com/crawler) 1833 | rabaz (rabaz at gigabaz dot com) 1834 | RaBot/1.0 Agent-admin/phortse@hanmail.net 1835 | Rainbot1.1 1836 | ramBot xtreme x.x 1837 | RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com) 1838 | RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) 1839 | Rank Exec (rankexec.com) Reciprocal Link Manager 1.x/bot 1840 | Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn) 1841 | Rational SiteCheck (Windows NT) 1842 | RAYSPIDER/Nutch-0.9 1843 | ReadABlog Spider (compatible; 1.1; feed update; www.readablog.com) 1844 | RealDownload/4.0.0.4x 1845 | REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) 1846 | Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) 1847 | Reaper/2.0x (+http://www.sitesearch.ca/reaper) 1848 | REBOL Core 2.x.x.x.x 1849 | REBOL View 1.x.x.x.x 1850 | RebusnetBot (+http://www.rebusnet.biz) 1851 | RebusnetPADBot/1.5x (+http://www.rebusnet.biz) 1852 | reciprocal links checker (http://www.recip-links.com/) 1853 | RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) 1854 | RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) 1855 | RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net) 1856 | RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) 1857 | RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) 1858 | REL Link Checker Lite x.x 1859 | RepoMonkey Bait & Tackle/v1.01 1860 | Rewebber/1.2 libwww-perl/5.41 1861 | rico/0.1 1862 | RixBot (http://babelserver.org/rix) 1863 | RMA/1.0 (compatible; RealMedia) 1864 | RoboCrawl (http://www.canadiancontent.net) 1865 | RoboCrawl (www.canadiancontent.net) 1866 | RoboPal (http://www.findpal.com/) 1867 | Robot/www.pj-search.com 1868 | Robot: NutchCrawler- Owner: wdavies@acm.org 1869 | Robot@SuperSnooper.Com 1870 | Robozilla/1.0 1871 | Rome Client (http://tinyurl.com/64t5n) Ver: 0.9 1872 | Rotondo/3.1 libwww/5.3.1 1873 | RPT-HTTPClient/0.3-x 1874 | RRC (crawler_admin@bigfoot.com) 1875 | RssBandit/1.5.0.10 (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; http://www.rssbandit.org) (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; ) 1876 | RSSMicro.com RSS/Atom Feed Robot 1877 | RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw) 1878 | RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw) 1879 | RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0 1880 | RSurf15a 41 1881 | RSurf15a 51 1882 | RSurf15a 81 1883 | Rubbot/1.0 (+http://rubhub.com/) 1884 | RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html) 1885 | RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) 1886 | Rumours-Agent 1887 | RX Bar 1888 | S&L Spider (http://search.hirners.com/) 1889 | S.T.A.L.K.E.R. (http://www.seo-tools.net/en/bot.aspx) 1890 | SafariBookmarkChecker (+http://www.coriolis.ch/) 1891 | sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) 1892 | SandCrawler - Compatibility Testing 1893 | SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) 1894 | SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) 1895 | savvybot/0.2 1896 | SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) 1897 | SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) 1898 | SBL-BOT (http://sbl.net) 1899 | ScanWeb 1900 | ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com) 1901 | schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp) 1902 | Science Traveller International 1X/1.0 1903 | ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) 1904 | Scooter-3.0.EU 1905 | Scooter-3.0.FS 1906 | Scooter-3.0.HD 1907 | Scooter-3.0.VNS 1908 | Scooter-3.0QI 1909 | Scooter-3.2 1910 | Scooter-3.2.BT 1911 | Scooter-3.2.DIL 1912 | Scooter-3.2.EX 1913 | Scooter-3.2.JT 1914 | Scooter-3.2.NIV 1915 | Scooter-3.2.SF0 1916 | Scooter-3.2.snippet 1917 | Scooter-3.3dev 1918 | Scooter-ARS-1.1 1919 | Scooter-ARS-1.1-ih 1920 | scooter-venus-3.0.vns 1921 | Scooter-W3-1.0 1922 | Scooter-W3.1.2 1923 | Scooter/1.0 1924 | Scooter/1.0 scooter@pa.dec.com 1925 | Scooter/1.1 (custom) 1926 | Scooter/2.0 G.R.A.B. V1.1.0 1927 | Scooter/2.0 G.R.A.B. X2.0 1928 | Scooter/3.3 1929 | Scooter/3.3.QA.pczukor 1930 | Scooter/3.3.vscooter 1931 | Scooter/3.3_SF 1932 | Scooter2_Mercator_x-x.0 1933 | Scooter_bh0-3.0.3 1934 | Scooter_trk3-3.0.3 1935 | Scope (Mars+) 1936 | ScoutAbout 1937 | ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ 1938 | scoutmaster 1939 | Scrubby/2.x (http://www.scrubtheweb.com/) 1940 | Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html) 1941 | Search+ 1942 | Search-Engine-Studio 1943 | search.ch V1.4 1944 | search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) 1945 | Search/1.0 (http://www.innerprise.net/es-spider.asp) 1946 | searchbot admin@google.com 1947 | SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com) 1948 | SearchdayBot 1949 | SearchExpress Spider0.99 1950 | SearchGuild/DMOZ/Experiment (searchguild@gmail.com) 1951 | SearchGuild_DMOZ_Experiment (chris@searchguild.com) 1952 | Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) 1953 | Searchmee! Spider v0.98a 1954 | SearchSight/2.0 (http://SearchSight.com/) 1955 | SearchSpider.com/1.1 1956 | Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com) 1957 | SearchTone2.0 - IDEARE 1958 | Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 1959 | Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) 1960 | Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 1961 | Seeker.lookseek.com 1962 | semaforo.net 1963 | Semager/1.1 (http://www.semager.de/blog/semager-bots/) 1964 | Semager/1.x (http://www.semager.de) 1965 | semanticdiscovery/0.x 1966 | Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) 1967 | Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) 1968 | SeznamBot/1.0 1969 | SeznamBot/1.0 (+http://fulltext.seznam.cz/) 1970 | SeznamBot/2.0-test (+http://fulltext.sblog.cz/) 1971 | ShablastBot 1.0 1972 | Shareaza v1.x.x.xx 1973 | SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com) 1974 | Shim Crawler 1975 | Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) 1976 | ShopWiki/1.0 ( +http://www.shopwiki.com/) 1977 | ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) 1978 | Shoula.com Crawler 2.0 1979 | SietsCrawler/1.1 (+http://www.siets.biz) 1980 | Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) 1981 | Siigle Orumcex v.001 Turkey (http://www.siigle.com) 1982 | silk/1.0 1983 | silk/1.0 (+http://www.slider.com/silk.htm)/3.7 1984 | SimpleFavPanel/1.2 1985 | Simpy 1.x; http://www.simpy.com/ 1986 | Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com) 1987 | Sirketcebot/v.01 (http://www.sirketce.com/bot.html) 1988 | SiteBar/3.x.x (Bookmark Server; http://sitebar.org/) 1989 | SiteBar/x.x 1990 | SiteBar/x.x.x (Bookmark Server; http://sitebar.org/) 1991 | sitecheck.internetseer.com 1992 | sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com) 1993 | SiteRecon+(xx) 1994 | SiteSnagger 1995 | SiteSpider +(http://www.SiteSpider.com/) 1996 | SiteSucker/1.x.x 1997 | SiteTaggerBot (http://www.sitetagger.com/bot.htm) 1998 | SiteTruth.com site rating system 1999 | SiteWinder 2000 | SiteXpert 2001 | Skampy/0.9.x (http://www.skaffe.com/skampy-info.html) 2002 | Skimpy/0.x (http://www.skaffe.com/skampy-info.html) 2003 | Skywalker/0.1 (Skywalker; anonymous; anonymous) 2004 | Slarp/0.1 2005 | Sleipnir 2006 | Sleipnir Version 1.xx 2007 | Sleipnir Version2.x 2008 | Sleipnir/2.xx 2009 | Slider_Search_v1-de 2010 | SlimBrowser 2011 | Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 2012 | Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 2013 | Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 2014 | Slurpy Verifier/1.0 2015 | SlySearch (slysearch@slysearch.com) 2016 | SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html 2017 | SlySearch/1.x http://www.slysearch.com 2018 | SmartDownload/1.2.67 (Win32; Jan 12 1999) 2019 | SmartDownload/1.2.77 (Win32; Feb 1 2000) 2020 | SmartDownload/1.2.77 (Win32; Jun 19 2001) 2021 | smartwit.com 2022 | SmiffyDCMetaSpider/1.0 2023 | sna-0.0.1 (mikemuzio@msn.com) 2024 | sna-0.0.1 mikeelliott@hotmail.com 2025 | snap.com beta crawler v0 2026 | Snapbot/1.0 2027 | Snapbot/1.0 (Snap Shots, +http://www.snap.com) 2028 | Snappy/1.1 ( http://www.urltrends.com/ ) 2029 | Snarfer/0.x.x (http://www.snarfware.com/) 2030 | SnoopRob/x.x 2031 | Snoopy v1.xx 2032 | Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2) 2033 | Snoopy_v0.xx 2034 | SnykeBot/0.6 (http://www.snyke.com) 2035 | SocSciBot () 2036 | SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 2037 | SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) 2038 | Softizerbot (http://www.softizer.com) 2039 | sogou develop spider 2040 | Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) 2041 | sogou spider 2042 | Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) 2043 | sohu agent 2044 | sohu-search 2045 | Sosospider+(+http://help.soso.com/webspider.htm) 2046 | Space Bison/0.02 [fu] (Win67; X; SK) 2047 | SpeedDownload/1.x 2048 | speedfind ramBot xtreme 8.1 2049 | Speedy Spider (Beta/x.x; speedy@entireweb.com) 2050 | Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) 2051 | Speedy_Spider (http://www.entireweb.com) 2052 | Sphere Scout&v4.0 - scout at sphere dot com 2053 | Sphider 2054 | Spida/0.1 2055 | Spider-Sleek/2.0 (+http://search-info.com/linktous.html) 2056 | spider.batsch.com 2057 | Spider.TerraNautic.net - v:1.04 2058 | spider.yellopet.com - www.yellopet.com 2059 | Spider/maxbot.com admin@maxbot.com 2060 | SpiderKU/0.x 2061 | SpiderMan 2062 | SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) 2063 | Spinne/2.0 2064 | Spinne/2.0 med 2065 | Spinne/2.0 med_AH 2066 | Spock Crawler (http://www.spock.com/crawler) 2067 | sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de) 2068 | sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com) 2069 | SQ Webscanner 2070 | Squid-Prefetch 2071 | squidclam 2072 | SquidClamAV_Redirector 1.x.x 2073 | Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) 2074 | Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) 2075 | Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu) 2076 | SSurf15a 11 2077 | StackRambler/x.x 2078 | Stamina/1.4 2079 | Star Downloader 2080 | StarDownloader/1.xx 2081 | stat statcrawler@gmail.com 2082 | Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) 2083 | Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) 2084 | Strategic Board Bot (+http://www.strategicboard.com) 2085 | Submission Spider at surfsafely.com 2086 | suchbaer.de 2087 | suchbaer.de (CrawlerAgent v0.103) 2088 | suchbot 2089 | Suchknecht.at-Robot 2090 | suchpadbot/1.0 (+http://www.suchpad.de) 2091 | Sunrise XP/2.x 2092 | Sunrise/0.42g (Windows XP) 2093 | SuperBot/x.x (Win32) 2094 | SuperBot/x.x.x.xx (Windows XP) 2095 | Superdownloads Spiderman 2096 | SURF 2097 | SurferF3 1/0 2098 | SurfMaster 2099 | SurveyBot/2.2 <a href='http://www.whois.sc' 2100 | SurveyBot/2.3 (Whois Source) 2101 | suzuran 2102 | SWB/V1.4 (HP) 2103 | swbot/0.9c libwww/5.3.1 2104 | Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm) 2105 | SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.html 2106 | SygolBot http://www.sygol.net 2107 | Sylera/1.2.x 2108 | SyncBot 2109 | SyncIT/x.x 2110 | Syndirella/0.91pre 2111 | SynoBot 2112 | Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler 2113 | Szukacz/1.x 2114 | Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) 2115 | T-Online Browser 2116 | tags2dir.com/0.8 (+http://tags2dir.com/directory/) 2117 | Tagword (http://tagword.com/dmoz_survey.php) 2118 | Tagyu Agent/1.0 2119 | Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot) 2120 | TAMU_CS_IRL_CRAWLER/1.0 2121 | targetblaster.com/0.9k 2122 | TargetYourNews.com bot 2123 | TCDBOT/Nutch-0.8 (PhD student research;http://www.tcd.ie; mcgettrs at t c d dot IE) 2124 | TE 2125 | TeamSoft WinInet Component 2126 | TECOMAC-Crawler/0.x 2127 | Tecomi Bot (http://www.tecomi.com/bot.htm) 2128 | Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) 2129 | Teleport Pro/1.2x(.1xxx) 2130 | Teoma MP 2131 | teomaagent crawler-admin@teoma.com 2132 | teomaagent1 [crawler-admin@teoma.com] 2133 | teoma_agent1 2134 | Teradex Mapper; mapper@teradex.com; http://www.teradex.com 2135 | TeragramCrawler 2136 | terraminds-bot/1.0 (support@terraminds.de) 2137 | TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) 2138 | Test spider 2139 | TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com) 2140 | The Expert HTML Source Viewer (http://www.expert-html.com) 2141 | TheRarestParser/0.2a (http://therarestwords.com/) 2142 | TheSuBot/0.1 (www.thesubot.de) 2143 | thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) 2144 | thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de) 2145 | timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html 2146 | http://www.timelyweb.com/ 2147 | TinEye/1.1 (http://tineye.com/crawler.html) 2148 | tivraSpider/1.0 (crawler@tivra.com) 2149 | TJG/Spider 2150 | TJvMultiHttpGrabber Component 2151 | Tkensaku/x.x(http://www.tkensaku.com/q.html) 2152 | toCrawl/UrlDispatcher 2153 | Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com) 2154 | TOPOS robot/1.1 (http://www.topos.com.ua/) 2155 | Toutatis x-xx.x (hoppa.com) 2156 | Toutatis x.x (hoppa.com) 2157 | Toutatis x.x-x 2158 | traazibot/testengine (+http://www.traazi.de) 2159 | Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) 2160 | Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com) 2161 | Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) 2162 | Trampelpfad-Spider 2163 | Trampelpfad-Spider-v0.1 2164 | tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck) 2165 | TSurf15a 11 2166 | TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17 2167 | TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8 2168 | Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com) 2169 | TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) 2170 | Turnpike Emporium LinkChecker/0.1 2171 | TutorGig/1.5 (+http://www.tutorgig.com/crawler) 2172 | Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) 2173 | Twiceler www.cuill.com/robots.html 2174 | Twiceler-0.9 http://www.cuill.com/twiceler/robot.html 2175 | Twisted PageGetter 2176 | Twitturly / v0.x 2177 | Twotrees Reactive Filter V2.0 2178 | Tycoon Agent/Nutch-1.0-dev 2179 | TygoBot 2180 | TygoProwler 2181 | UCmore 2182 | UCMore Crawler App 2183 | UCWEB5.1 2184 | UDM 2185 | UdmSearch/3.1.x 2186 | UIowaCrawler/1.0 2187 | UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) 2188 | Ultraseek 2189 | Under the Rainbow 2.2 2190 | unknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.com 2191 | UofTDB_experiment (leehyun@cs.toronto.edu) 2192 | UP.Browser/3.01-IG01 UP.Link/3.2.3.4 2193 | updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) 2194 | updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) 2195 | UPG1 UP/4.0 (compatible; Blazer 1.0) 2196 | Uptimebot 2197 | UptimeBot(www.uptimebot.com) 2198 | URI::Fetch/0.06 2199 | URL Spider Pro/x.xx (innerprise.net) 2200 | URLBase/6.x 2201 | URLBlaze 2202 | urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html 2203 | URLGetFile 2204 | URL_Spider_Pro/x.x 2205 | URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) 2206 | User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com 2207 | User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.com 2208 | User-Agent: LjSEEK Picture-Bot /1.0 contact@ljseek.com 2209 | User-Agent: FileHeap! file downloader (http://www.fileheap.com) 2210 | User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) 2211 | User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) 2212 | user-agent=Mozilla/3.01Gold 2213 | USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) 2214 | UtilMind HTTPGet 2215 | Utopia WebWasher 3.0 2216 | uTorrent/1500 2217 | VadixBot 2218 | Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile 2219 | Vagabondo/1.x MT (webagent@wise-guys.nl) 2220 | Vagabondo/2.0 MT 2221 | Vagabondo/2.0 MT (webagent at wise-guys dot nl) 2222 | Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) 2223 | Vagabondo/3.0 (webagent at wise-guys dot nl) 2224 | Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) 2225 | VayalaCreep-v0.0.1 (haploid@haploid.com) 2226 | Vayala|Creep-v0.0.1 (codepoet@wildties.com) 2227 | vb wininet 2228 | versus 0.2 (+http://versus.integis.ch) 2229 | versus crawler eda.baykan@epfl.ch 2230 | Verticrawlbot 2231 | VeryGoodSearch.com.DaddyLongLegs 2232 | verzamelgids.nl - Networking4all Bot/x.x 2233 | Verzamelgids/2.2 (http://www.verzamelgids.nl) 2234 | Vespa Crawler 2235 | virus_detector (virus_harvester@securecomputing.com) 2236 | VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) 2237 | Visicom Toolbar 2238 | Vision Research Lab image spider at vision.ece.ucsb.edu 2239 | VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN team 2240 | VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) 2241 | Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) 2242 | voyager-hc/1.0 2243 | voyager/1.0 2244 | voyager/2.0 (http://www.kosmix.com/html/crawler.html) 2245 | VSE/1.0 (testcrawler@hotmail.com) 2246 | VSE/1.0 (testcrawler@vivisimo.com) 2247 | vspider 2248 | vspider/3.x 2249 | VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu 2250 | W3C-checklink/3.x.x.x libwww-perl/5.xx 2251 | W3C-checklink/4.x [4.xx] libwww-perl/5.xxx 2252 | W3C-WebCon/5.x.x libwww/5.x.x 2253 | W3CLineMode/5.4.0 libwww/5.x.x 2254 | W3CRobot/5.4.0 libwww/5.4.0 2255 | W3C_Validator/1.xxx libwww-perl/5.xx 2256 | w3m/0.x.xx 2257 | W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de 2258 | wadaino.jp-crawler 0.2 (http://wadaino.jp/) 2259 | WannaBe (Macintosh; PPC) 2260 | WapOnWindows 1.0 2261 | Watchfire WebXM 1.0 2262 | WAVcheck 1.0.x (http://www.webbanalys.se/apps/WAVcheck/) 2263 | Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) 2264 | Waypath development crawler - info at waypath dot com 2265 | Waypath Scout v2.x - info at waypath dot com 2266 | WDG_Validator/1.1 2267 | Web Image Collector 2268 | Web Link Validator 1.5 2269 | Web Snooper 2270 | web-bekannt (Version: 1.02, powered by www.internetservice-franken.de) 2271 | web-bekannt (Version: 1.02, powered by www.web-bekannt.de) 2272 | Web-Bot V1.03 2273 | Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3 2274 | web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org) 2275 | WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) 2276 | WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) 2277 | WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) 2278 | WebAuto/3.4xxx (WinNT; I) 2279 | webbandit/4.xx.0 2280 | WebBug/5.x 2281 | Webclipping.com 2282 | webcollage/1.xx 2283 | WebCompass 2.0 2284 | WebCopier vx.x 2285 | WebCopier vx.xa 2286 | WebCorp/1.0 2287 | webcrawl.net 2288 | WebDownloader for X x.xx 2289 | Webdup/0.9 2290 | WebFetch 2291 | webfetch/5.x.x 2292 | WebFilter Robot 1.0 2293 | WebFilter Robot 1.x 2294 | WebFindBot(http://www.web-find.com) 2295 | Webglimpse 2.xx.x (http://webglimpse.net) 2296 | webGobbler/1.x.x 2297 | webhack 2298 | WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages ) 2299 | WebLight/4.x.x (support@illumit.com; http://www.illumit.com/Products/weblight/) 2300 | Weblink's checker/ 2301 | Weblog Attitude Diffusion 1.0 2302 | webmeasurement-bot, http://rvs.informatik.uni-leipzig.de 2303 | WebMiner/x.x [en] (Win98; I) 2304 | WeBoX/0.xx 2305 | WebPix 1.0 (www.netwu.com) 2306 | WebQL 2307 | WebRACE/1.1 (University of Cyprus- Distributed Crawler) 2308 | WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) 2309 | WebReaper vx.x - www.webreaper.net 2310 | WebReaper [info@webreaper.net] 2311 | WebReaper [webreaper@webreaper.net] 2312 | WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU) 2313 | WebSearchBench WebCrawler v0.1(Experimental) 2314 | WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/ 2315 | Website Explorer/0.9.x.x 2316 | Website eXtractor 2317 | WebsiteWorth v1.0 2318 | Webspinne/1.0 webmaster@webspinne.de 2319 | Websquash.com (Add url robot) 2320 | WebStat/1.0 (Unix; beta; 20040314) 2321 | Webster v0.3 ( http://webster.healeys.net/ ) 2322 | webster-internet.de pad browser 2323 | WebStripper/2.xx 2324 | WebTrafficExpress/x.0 2325 | WebTrends/3.0 (WinNT) 2326 | WebVac (webmaster@pita.stanford.edu) 2327 | WebVal/1.0 2328 | Webverzeichnis.de - Telefon: 01908 / 26005 2329 | WebVulnCrawl.unknown/1.0 libwww-perl/5.803 2330 | WebWatcherMonitor/2.01 2331 | WebZIP/x.x (http://www.spidersoft.com) 2332 | Wells Search II 2333 | WEP Search 00 2334 | West Wind Internet Protocols 4.xx 2335 | WFARC 2336 | Wget/1.x(.x)GNU wget http://www.gnu.org/software/wget/wget.html - file downloader 2337 | Wget/1.x+cvs-stable (Red Hat modified) 2338 | Wget/1.x.x+cvs 2339 | Whatsup/x.x 2340 | whatUseek_winona/3.0 2341 | WhizBang! Lab 2342 | Wildsoft Surfer 2343 | Willow Internet Crawler by Twotrees V2.1 2344 | WinampMPEG/2.00 (larbin@unspecified.mail) 2345 | WincerSong Agent v1.0 2346 | Windows-Media-Player/10.00.00.xxxx 2347 | WinGet 1.1 2348 | WinHTTP Example/1.0 2349 | WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) 2350 | WinPodder (http://winpodder.com) 2351 | WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0) 2352 | Wir sind die Borg (Version: 1.03, Sie wurden Assimiliert +http://www.yammba.com/suchmaschine/bot.html) 2353 | WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) 2354 | WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) 2355 | WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) 2356 | WiseWire-Spider2 2357 | wish-project (http://wish.slis.tsukuba.ac.jp/) 2358 | WordChampBot 2359 | WordPress/x.x.x.x PHP/4.x.xx 2360 | worio heritrix bot (+http://worio.com/) 2361 | woriobot ( http://www.worio.com/) 2362 | WorldLight 2363 | WorQmada/1.0 2364 | Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com) 2365 | Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 2366 | WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.de 2367 | WSB, http://websearchbench.cs.uni-dortmund.de 2368 | wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) 2369 | Wusage/x.0@boutell.com 2370 | Wwlib/Linux 2371 | WWSBOT 1.x [--- http://www.analyzer.nu ---] 2372 | WWW-Mechanize/1.1x 2373 | www.arianna.it 2374 | www.business-socket.com registry verify/1.x 2375 | www.doweb.co.uk crawler 2376 | www4mail/2.x libwww-FM/2.14 (Unix; I) 2377 | WWWC/1.0x 2378 | WWWeasel Robot v1.00 (http://wwweasel.de) 2379 | WWWOFFLE/2.x 2380 | wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de) 2381 | wxDownload Fast 2382 | X-Crawler 2383 | Xaldon WebSpider 2384 | Xenu Link Sleuth 1.xx 2385 | Xenu's Link Sleuth 1.x[a-z] 2386 | Xerka WebBot v1.0.0 [UPVOpenDir] 2387 | xine/1.0 2388 | xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) 2389 | XMLSlurp/0.1 libwww-perl/5.805 2390 | XRL/2.00b1 (Linux; i686; en-us) (+http://metamark.net/about) 2391 | Xylix 2392 | xyro_(xcrawler@cosmos.inria.fr) 2393 | Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 2394 | Y!J-SRD/1.0 2395 | Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 2396 | Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386) 2397 | Y!TunnelPro 2398 | yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en) 2399 | yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net 2400 | Yahoo Pipes 1.0 2401 | Yahoo! Mindset 2402 | Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) 2403 | Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) 2404 | Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) 2405 | Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) 2406 | Yahoo-Test/4.0 2407 | Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler 2408 | YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide) 2409 | YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) 2410 | YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) 2411 | YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html) 2412 | YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) 2413 | YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) 2414 | YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) 2415 | Yandex/1.01.001 (compatible; Win16; I) 2416 | Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/) 2417 | yarienavoir.net/0.2 2418 | Yeti 2419 | Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) 2420 | Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) 2421 | yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de) 2422 | YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) 2423 | yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail) 2424 | yoogliFetchAgent/0.1 2425 | yoono/1.0 web-crawler/1.0 2426 | YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine 2427 | YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine 2428 | YTunnelPro 2429 | Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) 2430 | Zao-Crawler 2431 | Zao-Crawler 0.2b 2432 | Zao/0.1 (http://www.kototoi.org/zao/) 2433 | ZBot/1.00 (icaulfield@zeus.com) 2434 | Zearchit 2435 | ZeBot_lseek.net (bot@ze.bz) 2436 | ZeBot_www.ze.bz (ze.bz@hotmail.com) 2437 | zedzo.digest/0.1 (http://www.zedzo.com/) 2438 | Zend_Http_Client 2439 | zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com] 2440 | zerxbot/Version 0.6 libwww-perl/5.79 2441 | Zeus ThemeSite Viewer Webster Pro V2.9 Win32 2442 | Zeus xxxxx Webster Pro V2.9 Win32 2443 | Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com) 2444 | Ziggy -- The Clown From Hell!! 2445 | ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) 2446 | ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) 2447 | Zippy v2.0 - Zippyfinder.com 2448 | Zoo Tycoon 2 Client -- http://www.zootycoon.com 2449 | ZoomSpider - wrensoft.com 2450 | Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler 2451 | zspider/0.9-dev http://feedback.redkolibri.com/ 2452 | ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) 2453 | Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (FM Scene 4.6.1) 2454 | Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5) 2455 | --------------------------------------------------------------------------------