├── .gitignore
├── README.md
├── demo-file.js
└── extract.rb


/.gitignore:
--------------------------------------------------------------------------------
1 | .ruby-version
2 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | During reconnaissance (recon) it is often helpful to get a quick overview of all the relative endpoints in a file. These days web applications have frontend pipelines that make it harder for humans to understand minified code. This tool contains a nifty regular expression to find and extract the relative URLs in such files. This can help surface new targets for security researchers to look at. It can also be used to periodically compare the results of the same file, to see which new endpoints have been deployed. History has shown that this is a goldmine for bug bounty hunters.
 2 | 
 3 | # Usage
 4 | The tool only accepts input on `STDIN`. Any command output can be piped into the file to extract relative URLs. To include an excerpt of the line it was matched in, use the  `--show-line` flag.
 5 | 
 6 |  - `cat demo-file.js | ./extract.rb`
 7 |  - `cat demo-file.js | ./extract.rb --show-line`
 8 |  - `curl -s https://hackerone.com/hacktivity | ./extract.rb`
 9 | 
10 | # Demo
11 | A demo file is included in the repository. Run the commands as shown above to extract relative URLs from a minified JavaScript file. An excerpt of the output is shown below.
12 | 
13 | ```
14 | ...
15 | /vulnerability-coordination-maturity-model/user_data
16 | /mail_subscriptions/update_monthly_digest_subscription
17 | /subscription.json?subscribe=true
18 | /subscription.json?subscribe=false
19 | /terms/finder
20 | /resources/download-web-hacking-101
21 | /settings/reputation/log
22 | /plugins/servlet/webhooks
23 | /secure/project/ViewProjects.jspa
24 | ...
25 | ```
26 | 


--------------------------------------------------------------------------------
/extract.rb:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env ruby
 2 | 
 3 | # frozen_string_literal: true
 4 | 
 5 | contents = ''
 6 | 
 7 | while (line = STDIN.gets)
 8 |   contents += line
 9 | end
10 | 
11 | REGEX = %r{(^.*?("|')(/[\w\d\W\?/&=\#\.\!:_-]*?)(\2).*$)}.freeze
12 | EXCERPT_FORMAT = "------------------------------------------------\r\n%s\r\n"
13 | 
14 | def sanitize_non_ascii(string)
15 |   encoding_options = {
16 |     invalid: :replace,
17 |     undef: :replace,
18 |     replace: '_'
19 |   }
20 | 
21 |   string.encode(Encoding.find('ASCII'), **encoding_options)
22 | end
23 | 
24 | matched_endpoints = []
25 | 
26 | sanitize_non_ascii(contents).gsub(/;/, "\n").scan(REGEX).map do |string|
27 |   next if matched_endpoints.include?(string[2])
28 | 
29 |   matched_endpoints << string[2]
30 | 
31 |   puts string[2]
32 | 
33 |   puts format(EXCERPT_FORMAT, string[0]) if ARGV[0] == '--show-line'
34 | end
35 | 


--------------------------------------------------------------------------------