├── .gitignore
├── .travis.yml
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── lib
    └── prerender_rails.rb
├── prerender_rails.gemspec
└── test
    ├── lib
        └── prerender_rails.rb
    └── test_helper.rb


/.gitignore:
--------------------------------------------------------------------------------
 1 | *.gem
 2 | *.rbc
 3 | .bundle
 4 | .config
 5 | .yardoc
 6 | Gemfile.lock
 7 | InstalledFiles
 8 | _yardoc
 9 | coverage
10 | doc/
11 | lib/bundler/man
12 | pkg
13 | rdoc
14 | spec/reports
15 | test/tmp
16 | test/version_tmp
17 | tmp
18 | .DS_Store


--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
1 | language: ruby
2 | rvm:
3 |   - 2.5
4 | before_install:
5 |   - gem install bundler


--------------------------------------------------------------------------------
/Gemfile:
--------------------------------------------------------------------------------
1 | source 'https://rubygems.org'
2 | 
3 | # Specify your gem's dependencies in prerender_rails.gemspec
4 | gemspec
5 | 


--------------------------------------------------------------------------------
/LICENSE.txt:
--------------------------------------------------------------------------------
 1 | Copyright (c) 2013 Todd Hooper
 2 | 
 3 | MIT License
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining
 6 | a copy of this software and associated documentation files (the
 7 | "Software"), to deal in the Software without restriction, including
 8 | without limitation the rights to use, copy, modify, merge, publish,
 9 | distribute, sublicense, and/or sell copies of the Software, and to
10 | permit persons to whom the Software is furnished to do so, subject to
11 | the following conditions:
12 | 
13 | The above copyright notice and this permission notice shall be
14 | included in all copies or substantial portions of the Software.
15 | 
16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
23 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | Prerender Rails [![Build Status](https://travis-ci.org/prerender/prerender_rails.png)](https://travis-ci.org/prerender/prerender_rails) [![Gem Version](https://badge.fury.io/rb/prerender_rails.png)](http://badge.fury.io/rb/prerender_rails)
  2 | ===========================
  3 | 
  4 | Google, Facebook, Twitter, and Bing are constantly trying to view your website... but Google is the only crawler that executes a meaningful amount of JavaScript and Google even admits that they can execute JavaScript weeks after actually crawling. Prerender allows you to serve the full HTML of your website back to Google and other crawlers so that they don't have to execute any JavaScript. [Google recommends using Prerender.io](https://developers.google.com/search/docs/guides/dynamic-rendering) to prevent indexation issues on sites with large amounts of JavaScript.
  5 | 
  6 | Prerender is perfect for Angular SEO, React SEO, Vue SEO, and any other JavaScript framework.
  7 | 
  8 | This middleware intercepts requests to your Node.js website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the JavaScript for that page. That HTML is then returned to the crawler.
  9 | 
 10 | `Note` Make sure you have more than one webserver thread/process running because the prerender service will make a request to your server to render the HTML.
 11 | 
 12 | Add this line to your application's Gemfile:
 13 | 
 14 |     gem 'prerender_rails'
 15 | 
 16 | And in `config/environment/production.rb`, add this line:
 17 | 
 18 | ```ruby
 19 | 	config.middleware.use Rack::Prerender
 20 | ```
 21 | 
 22 | or if you have an account on [prerender.io](https://prerender.io/) and want to use your token:
 23 | 
 24 | ```ruby
 25 | 	config.middleware.use Rack::Prerender, prerender_token: 'YOUR_TOKEN'
 26 | ```
 27 | 
 28 | `Note` If you're testing locally, you'll need to run the [prerender server](https://github.com/prerender/prerender) locally so that it has access to your server.
 29 | 
 30 | ## Testing
 31 | 
 32 | When testing make sure you're not using a single threaded application server like default WEBrick one, use Puma or Unicorn to prevent a deadlock when the Prerender Service needs to render your HTML on the fly.
 33 | 
 34 | The best way to test the prerendered page is to [set the User Agent of your browser to Googlebot's user agent](https://developers.google.com/web/tools/chrome-devtools/device-mode/override-user-agent) and visit your URL directly. If you View Source on that URL, you should see the static HTML version of the page with the `<script>` tags removed from the page. If you still see `<script>` tags then that means the middleware isn't set up properly yet.
 35 | 
 36 | `Note` If you're testing locally, you'll need to run the [prerender server](https://github.com/prerender/prerender) locally so that it has access to your server.
 37 | 
 38 | ## How it works
 39 | 1. The middleware checks to make sure we should show a prerendered page
 40 | 	1. The middleware checks if the request is from a crawler by checking the user agent string against a default list of crawler user agents
 41 | 	2. The middleware checks to make sure we aren't requesting a resource (js, css, etc...)
 42 | 	3. (optional) The middleware checks to make sure the url is in the whitelist
 43 | 	4. (optional) The middleware checks to make sure the url isn't in the blacklist
 44 | 2. The middleware makes a `GET` request to the [prerender service](https://github.com/prerender/prerender)(phantomjs server) for the page's prerendered HTML
 45 | 3. Return that HTML to the crawler
 46 | 
 47 | # Customization
 48 | 
 49 | ### Whitelist
 50 | 
 51 | Whitelist a single url path or multiple url paths. Compares using regex, so be specific when possible. If a whitelist is supplied, only url's containing a whitelist path will be prerendered.
 52 | ```ruby
 53 | config.middleware.use Rack::Prerender, whitelist: '^/search'
 54 | ```
 55 | ```ruby
 56 | config.middleware.use Rack::Prerender, whitelist: ['/search', '/users/.*/profile']
 57 | ```
 58 | 
 59 | ### Blacklist
 60 | 
 61 | Blacklist a single url path or multiple url paths. Compares using regex, so be specific when possible. If a blacklist is supplied, all url's will be prerendered except ones containing a blacklist path.
 62 | ```ruby
 63 | config.middleware.use Rack::Prerender, blacklist: '^/search'
 64 | ```
 65 | ```ruby
 66 | config.middleware.use Rack::Prerender, blacklist: ['/search', '/users/.*/profile']
 67 | ```
 68 | 
 69 | ### before_render
 70 | 
 71 | This method is intended to be used for caching, but could be used to save analytics or anything else you need to do for each crawler request. If you return a string from before_render, the middleware will serve that to the crawler instead of making a request to the prerender service.
 72 | ```ruby
 73 | config.middleware.use Rack::Prerender,
 74 | 	before_render: (Proc.new do |env|
 75 | 		# do whatever you need to do.
 76 | 	end)
 77 | ```
 78 | 
 79 | ### after_render
 80 | 
 81 | This method is intended to be used for caching, but could be used to save analytics or anything else you need to do for each crawler request. This method is a noop and is called after the prerender service returns HTML.
 82 | ```ruby
 83 | config.middleware.use Rack::Prerender,
 84 | 	after_render: (Proc.new do |env, response|
 85 | 		# do whatever you need to do.
 86 | 	end)
 87 | ```
 88 | 
 89 | ### build_rack_response_from_prerender
 90 | 
 91 | This method is intended to be used to modify the response before it is sent to the crawler. Use this method to add/remove response headers, or do anything else before the request is sent.
 92 | ```ruby
 93 | config.middleware.use Rack::Prerender,
 94 | 	build_rack_response_from_prerender: (Proc.new do |response, prerender_response|
 95 | 		# response is already populated with the prerender status code, html, and headers
 96 | 		# prerender_response is the response that came back from the prerender service
 97 | 	end)
 98 | ```
 99 | 
100 | ### protocol
101 | 
102 | Option to hard-set the protocol for Prerender accessing your site instead of the middleware figuring out the protocol based on the request.
103 | ```ruby
104 | config.middleware.use Rack::Prerender, protocol: 'https'
105 | ```
106 | 
107 | ## Caching
108 | 
109 | This rails middleware is ready to be used with [redis](http://redis.io/) or [memcached](http://memcached.org/) to return prerendered pages in milliseconds.
110 | 
111 | When setting up the middleware in `config/environment/production.rb`, you can add a `before_render` method and `after_render` method for caching.
112 | 
113 | Here's an example testing a local redis cache:
114 | 
115 | _Put this in `config/environment/development.rb`, and add `gem 'redis'` to your Gemfile._
116 | 
117 | ```ruby
118 | require 'redis'
119 | @redis = Redis.new
120 | config.middleware.use Rack::Prerender,
121 |   before_render: (Proc.new do |env|
122 |     @redis.get(Rack::Request.new(env).url)
123 |   end),
124 |   after_render: (Proc.new do |env, response|
125 |     @redis.set(Rack::Request.new(env).url, response.body)
126 |   end)
127 | ```
128 | 
129 | ## Using your own prerender service
130 | 
131 | We host a Prerender server at [prerender.io](https://prerender.io/) so that you can work on more important things, but if you've deployed the prerender service on your own... set the `PRERENDER_SERVICE_URL` environment variable so that this middleware points there instead. Otherwise, it will default to the service already deployed by [prerender.io](https://prerender.io/).
132 | 
133 | 	$ export PRERENDER_SERVICE_URL=<new url>
134 | 
135 | Or on heroku:
136 | 
137 | 	$ heroku config:add PRERENDER_SERVICE_URL=<new url>
138 | 
139 | As an alternative, you can pass `prerender_service_url` in the options object during initialization of the middleware
140 | 
141 | ``` ruby
142 | config.middleware.use Rack::Prerender, prerender_service_url: '<new url>'
143 | ```
144 | 
145 | ## License
146 | 
147 | The MIT License (MIT)
148 | 
149 | Copyright (c) 2013 Todd Hooper &lt;todd@prerender.io&gt;
150 | 
151 | Permission is hereby granted, free of charge, to any person obtaining a copy
152 | of this software and associated documentation files (the "Software"), to deal
153 | in the Software without restriction, including without limitation the rights
154 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
155 | copies of the Software, and to permit persons to whom the Software is
156 | furnished to do so, subject to the following conditions:
157 | 
158 | The above copyright notice and this permission notice shall be included in
159 | all copies or substantial portions of the Software.
160 | 
161 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
162 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
163 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
164 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
165 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
166 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
167 | THE SOFTWARE.
168 | 


--------------------------------------------------------------------------------
/Rakefile:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env rake
 2 | require "bundler/gem_tasks"
 3 |  
 4 | require 'rake/testtask'
 5 |  
 6 | Rake::TestTask.new do |t|
 7 |   t.libs << 'lib/prerender_rails'
 8 |   t.test_files = FileList['test/lib/prerender_rails.rb']
 9 |   t.verbose = true
10 | end
11 |  
12 | task :default => :test
13 | 


--------------------------------------------------------------------------------
/lib/prerender_rails.rb:
--------------------------------------------------------------------------------
  1 | module Rack
  2 |   class Prerender
  3 |     require 'net/http'
  4 |     require 'active_support'
  5 | 
  6 |     def initialize(app, options={})
  7 |       @connection_header_values = [
  8 |         'close',
  9 |         'keep-alive'
 10 |       ].freeze
 11 |       @hop_by_hop_headers = [
 12 |         'Connection',
 13 |         'Keep-Alive',
 14 |         'Public',
 15 |         'Proxy-Authenticate',
 16 |         'Transfer-Encoding',
 17 |         'Upgrade'
 18 |       ].freeze
 19 | 
 20 |       @crawler_user_agents = [
 21 |         'googlebot',
 22 |         'Google-InspectionTool',
 23 |         'yahoo',
 24 |         'bingbot',
 25 |         'baiduspider',
 26 |         'facebookexternalhit',
 27 |         'twitterbot',
 28 |         'rogerbot',
 29 |         'linkedinbot',
 30 |         'embedly',
 31 |         'bufferbot',
 32 |         'quora link preview',
 33 |         'showyoubot',
 34 |         'outbrain',
 35 |         'pinterest/0.',
 36 |         'developers.google.com/+/web/snippet',
 37 |         'www.google.com/webmasters/tools/richsnippets',
 38 |         'slackbot',
 39 |         'vkShare',
 40 |         'W3C_Validator',
 41 |         'redditbot',
 42 |         'Applebot',
 43 |         'WhatsApp',
 44 |         'flipboard',
 45 |         'tumblr',
 46 |         'bitlybot',
 47 |         'SkypeUriPreview',
 48 |         'nuzzel',
 49 |         'Discordbot',
 50 |         'Google Page Speed',
 51 |         'Qwantify',
 52 |         'Chrome-Lighthouse',
 53 |         'TelegramBot'
 54 |       ]
 55 | 
 56 |       @extensions_to_ignore = [
 57 |         '.js',
 58 |         '.css',
 59 |         '.xml',
 60 |         '.less',
 61 |         '.png',
 62 |         '.jpg',
 63 |         '.jpeg',
 64 |         '.gif',
 65 |         '.pdf',
 66 |         '.doc',
 67 |         '.txt',
 68 |         '.ico',
 69 |         '.rss',
 70 |         '.zip',
 71 |         '.mp3',
 72 |         '.rar',
 73 |         '.exe',
 74 |         '.wmv',
 75 |         '.doc',
 76 |         '.avi',
 77 |         '.ppt',
 78 |         '.mpg',
 79 |         '.mpeg',
 80 |         '.tif',
 81 |         '.wav',
 82 |         '.mov',
 83 |         '.psd',
 84 |         '.ai',
 85 |         '.xls',
 86 |         '.mp4',
 87 |         '.m4a',
 88 |         '.swf',
 89 |         '.dat',
 90 |         '.dmg',
 91 |         '.iso',
 92 |         '.flv',
 93 |         '.m4v',
 94 |         '.torrent'
 95 |       ]
 96 | 
 97 |       @options = options
 98 |       @options[:whitelist] = [@options[:whitelist]] if @options[:whitelist].is_a? String
 99 |       @options[:blacklist] = [@options[:blacklist]] if @options[:blacklist].is_a? String
100 |       @extensions_to_ignore = @options[:extensions_to_ignore] if @options[:extensions_to_ignore]
101 |       @crawler_user_agents = @options[:crawler_user_agents] if @options[:crawler_user_agents]
102 |       @app = app
103 |     end
104 | 
105 | 
106 |     def call(env)
107 |       if should_show_prerendered_page(env)
108 | 
109 |         cached_response = before_render(env)
110 | 
111 |         if cached_response
112 |           return cached_response.finish
113 |         end
114 | 
115 |         prerendered_response = get_prerendered_page_response(env)
116 | 
117 |         if prerendered_response
118 |           response = build_rack_response_from_prerender(prerendered_response)
119 |           after_render(env, prerendered_response)
120 |           return response.finish
121 |         end
122 |       end
123 | 
124 |       @app.call(env)
125 |     end
126 | 
127 | 
128 |     def should_show_prerendered_page(env)
129 |       user_agent = env['HTTP_USER_AGENT']
130 |       buffer_agent = env['HTTP_X_BUFFERBOT']
131 |       prerender_agent = env['HTTP_X_PRERENDER']
132 |       is_requesting_prerendered_page = false
133 | 
134 |       return false if !user_agent
135 |       return false if env['REQUEST_METHOD'] != 'GET'
136 | 
137 |       request = Rack::Request.new(env)
138 | 
139 |       is_requesting_prerendered_page = true if Rack::Utils.parse_query(request.query_string).has_key?('_escaped_fragment_')
140 | 
141 |       #if it is a bot...show prerendered page
142 |       is_requesting_prerendered_page = true if @crawler_user_agents.any? { |crawler_user_agent| user_agent.downcase.include?(crawler_user_agent.downcase) }
143 | 
144 |       #if it is BufferBot...show prerendered page
145 |       is_requesting_prerendered_page = true if buffer_agent
146 | 
147 |       #if it is Prerender...don't show prerendered page
148 |       is_requesting_prerendered_page = false if prerender_agent
149 | 
150 |       #if it is a bot and is requesting a resource...dont prerender
151 |       return false if @extensions_to_ignore.any? { |extension| request.fullpath.include? extension }
152 | 
153 |       #if it is a bot and not requesting a resource and is not whitelisted...dont prerender
154 |       return false if @options[:whitelist].is_a?(Array) && @options[:whitelist].all? { |whitelisted| !Regexp.new(whitelisted).match(request.fullpath) }
155 | 
156 |       #if it is a bot and not requesting a resource and is blacklisted(url or referer)...dont prerender
157 |       if @options[:blacklist].is_a?(Array) && @options[:blacklist].any? { |blacklisted|
158 |           blacklistedUrl = false
159 |           blacklistedReferer = false
160 |           regex = Regexp.new(blacklisted)
161 | 
162 |           blacklistedUrl = !!regex.match(request.fullpath)
163 |           blacklistedReferer = !!regex.match(request.referer) if request.referer
164 | 
165 |           blacklistedUrl || blacklistedReferer
166 |         }
167 |         return false
168 |       end
169 | 
170 |       return is_requesting_prerendered_page
171 |     end
172 | 
173 | 
174 |     def get_prerendered_page_response(env)
175 |       begin
176 |         url = URI.parse(build_api_url(env))
177 |         headers = {
178 |           'User-Agent' => env['HTTP_USER_AGENT'],
179 |           'Accept-Encoding' => 'gzip',
180 |           'X-Prerender-Int-Type' => 'Ruby'
181 |         }
182 |         headers['X-Prerender-Token'] = ENV['PRERENDER_TOKEN'] if ENV['PRERENDER_TOKEN']
183 |         headers['X-Prerender-Token'] = @options[:prerender_token] if @options[:prerender_token]
184 |         req = Net::HTTP::Get.new(url.request_uri, headers)
185 |         req.basic_auth(ENV['PRERENDER_USERNAME'], ENV['PRERENDER_PASSWORD']) if @options[:basic_auth]
186 |         http = Net::HTTP.new(url.host, url.port)
187 |         http.use_ssl = true if url.scheme == 'https'
188 |         response = http.request(req)
189 |         if response['Content-Encoding'] == 'gzip'
190 |           response.body = ActiveSupport::Gzip.decompress(response.body)
191 |           response['Content-Length'] = response.body.bytesize
192 |           response.delete('Content-Encoding')
193 |         end
194 | 
195 |         hop_by_hop_headers = @hop_by_hop_headers
196 |         connection = response['Connection']
197 |         if connection
198 |           connection_hop_by_hop_headers = connection.split(',').
199 |                                             map(&:strip).
200 |                                             map(&:downcase).
201 |                                             difference(@connection_header_values)
202 |           hop_by_hop_headers = connection_hop_by_hop_headers.
203 |                                  concat(hop_by_hop_headers)
204 |         end
205 |         hop_by_hop_headers.each { |h| response.delete(h) }
206 | 
207 |         response
208 |       rescue
209 |         nil
210 |       end
211 |     end
212 | 
213 | 
214 |     def build_api_url(env)
215 |       new_env = env
216 |       if env["CF-VISITOR"]
217 |         match = /"scheme":"(http|https)"/.match(env['CF-VISITOR'])
218 |         new_env["HTTPS"] = true and new_env["rack.url_scheme"] = "https" and new_env["SERVER_PORT"] = 443 if (match && match[1] == "https")
219 |         new_env["HTTPS"] = false and new_env["rack.url_scheme"] = "http" and new_env["SERVER_PORT"] = 80 if (match && match[1] == "http")
220 |       end
221 | 
222 |       if env["X-FORWARDED-PROTO"]
223 |         new_env["HTTPS"] = true and new_env["rack.url_scheme"] = "https" and new_env["SERVER_PORT"] = 443 if env["X-FORWARDED-PROTO"].split(',')[0] == "https"
224 |         new_env["HTTPS"] = false and new_env["rack.url_scheme"] = "http" and new_env["SERVER_PORT"] = 80 if env["X-FORWARDED-PROTO"].split(',')[0] == "http"
225 |       end
226 | 
227 |       if @options[:protocol]
228 |         new_env["HTTPS"] = true and new_env["rack.url_scheme"] = "https" and new_env["SERVER_PORT"] = 443 if @options[:protocol] == "https"
229 |         new_env["HTTPS"] = false and new_env["rack.url_scheme"] = "http" and new_env["SERVER_PORT"] = 80 if @options[:protocol] == "http"
230 |       end
231 | 
232 |       url = Rack::Request.new(new_env).url
233 |       prerender_url = get_prerender_service_url()
234 |       forward_slash = prerender_url[-1, 1] == '/' ? '' : '/'
235 |       "#{prerender_url}#{forward_slash}#{url}"
236 |     end
237 | 
238 | 
239 |     def get_prerender_service_url
240 |       @options[:prerender_service_url] || ENV['PRERENDER_SERVICE_URL'] || 'http://service.prerender.io/'
241 |     end
242 | 
243 | 
244 |     def build_rack_response_from_prerender(prerendered_response)
245 |       response = Rack::Response.new(prerendered_response.body, prerendered_response.code, prerendered_response)
246 | 
247 |       @options[:build_rack_response_from_prerender].call(response, prerendered_response) if @options[:build_rack_response_from_prerender]
248 | 
249 |       response
250 |     end
251 | 
252 |     def before_render(env)
253 |       return nil unless @options[:before_render]
254 | 
255 |       cached_render = @options[:before_render].call(env)
256 | 
257 |       if cached_render && cached_render.is_a?(String)
258 |         Rack::Response.new(cached_render, 200, { 'Content-Type' => 'text/html; charset=utf-8' })
259 |       elsif cached_render && cached_render.is_a?(Rack::Response)
260 |         cached_render
261 |       else
262 |         nil
263 |       end
264 |     end
265 | 
266 | 
267 |     def after_render(env, response)
268 |       return true unless @options[:after_render]
269 |       @options[:after_render].call(env, response)
270 |     end
271 |   end
272 | end
273 | 


--------------------------------------------------------------------------------
/prerender_rails.gemspec:
--------------------------------------------------------------------------------
 1 | # coding: utf-8
 2 | 
 3 | Gem::Specification.new do |spec|
 4 |   spec.name          = "prerender_rails"
 5 |   spec.version       = "1.11.0"
 6 |   spec.authors       = ["Todd Hooper"]
 7 |   spec.email         = ["todd@prerender.io"]
 8 |   spec.description   = %q{Rails middleware to prerender your javascript heavy pages on the fly by a phantomjs service}
 9 |   spec.summary       = %q{Prerender your backbone/angular/javascript rendered application on the fly when search engines crawl}
10 |   spec.homepage      = "https://github.com/prerender/prerender_rails"
11 |   spec.license       = "MIT"
12 | 
13 |   spec.files         = `git ls-files`.split($/)
14 |   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
15 |   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
16 |   spec.require_paths = ["lib"]
17 | 
18 |   spec.add_dependency 'rack', '~> 2.2.2'
19 |   spec.add_dependency 'activesupport', '>= 0'
20 | 
21 |   spec.add_development_dependency "bundler", "~> 2.1.2"
22 |   spec.add_development_dependency "rake"
23 |   spec.add_development_dependency "webmock"
24 | end
25 | 


--------------------------------------------------------------------------------
/test/lib/prerender_rails.rb:
--------------------------------------------------------------------------------
  1 | require_relative '../test_helper'
  2 | 
  3 | describe Rack::Prerender do
  4 | 
  5 |   bot = 'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
  6 |   user = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36'
  7 | 
  8 |   before :each do
  9 |     @app = lambda do |params|
 10 |       [200, {}, ""]
 11 |     end
 12 | 
 13 |     @prerender = Rack::Prerender.new(@app)
 14 |   end
 15 | 
 16 | 
 17 |   it "should return a prerendered response for a crawler with the returned status code and headers" do
 18 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
 19 |     stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => bot }).to_return(:body => "<html></html>", :status => 301, :headers => { 'Location' => 'http://google.com'})
 20 |     response = Rack::Prerender.new(@app).call(request)
 21 | 
 22 |     assert_equal response[2], ["<html></html>"]
 23 |     assert_equal response[0], 301
 24 |     assert_equal( { 'location' => 'http://google.com'}, response[1] )
 25 |   end
 26 | 
 27 | 
 28 |   it "should return a prerendered reponse if user is a bot by checking for _escaped_fragment_" do
 29 |     request = Rack::MockRequest.env_for "/path?_escaped_fragment_=", "HTTP_USER_AGENT" => user
 30 |     stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => user }).to_return(:body => "<html></html>")
 31 |     response = Rack::Prerender.new(@app).call(request)
 32 | 
 33 |     assert_equal ["<html></html>"], response[2]
 34 |   end
 35 | 
 36 | 
 37 |   it "should continue to app routes if the url is a bad url with _escaped_fragment_" do
 38 |     request = Rack::MockRequest.env_for "/path?query=string?_escaped_fragment_=", "HTTP_USER_AGENT" => user
 39 |     response = Rack::Prerender.new(@app).call(request)
 40 | 
 41 |     assert_equal "", response[2]
 42 |   end
 43 | 
 44 | 
 45 |   it "should continue to app routes if the request is not a GET" do
 46 |     request = Rack::MockRequest.env_for "/path?_escaped_fragment_=", { "HTTP_USER_AGENT" => user, "REQUEST_METHOD" => "POST" }
 47 |     response = Rack::Prerender.new(@app).call(request)
 48 | 
 49 |     assert_equal "", response[2]
 50 |   end
 51 | 
 52 | 
 53 |   it "should continue to app routes if user is not a bot by checking agent string" do
 54 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => user
 55 |     response = Rack::Prerender.new(@app).call(request)
 56 | 
 57 |     assert_equal "", response[2]
 58 |   end
 59 | 
 60 | 
 61 |   it "should continue to app routes if contains X-Prerender header" do
 62 |     request = Rack::MockRequest.env_for "/path?_escaped_fragment_=", "HTTP_USER_AGENT" => user, "HTTP_X_PRERENDER" => "1"
 63 |     response = Rack::Prerender.new(@app).call(request)
 64 | 
 65 |     assert_equal "", response[2]
 66 |   end
 67 | 
 68 | 
 69 |   it "should continue to app routes if user is a bot, but the bot is requesting a resource file" do
 70 |     request = Rack::MockRequest.env_for "/main.js?anyQueryParam=true", "HTTP_USER_AGENT" => bot
 71 |     response = Rack::Prerender.new(@app).call(request)
 72 | 
 73 |     assert_equal "", response[2]
 74 |   end
 75 | 
 76 | 
 77 |   it "should continue to app routes if the url is not part of the regex specific whitelist" do
 78 |     request = Rack::MockRequest.env_for "/saved/search/blah?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
 79 |     response = Rack::Prerender.new(@app, whitelist: ['^/search', '/help']).call(request)
 80 | 
 81 |     assert_equal "", response[2]
 82 |   end
 83 | 
 84 | 
 85 |   it "should set use_ssl to true for https prerender_service_url" do
 86 |     @prerender = Rack::Prerender.new(@app, prerender_service_url: 'https://service.prerender.io/')
 87 | 
 88 |     request = Rack::MockRequest.env_for "/search/things/123/page?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
 89 |     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
 90 |     response = @prerender.call(request)
 91 | 
 92 |     assert_equal ["<html></html>"], response[2]
 93 |   end
 94 | 
 95 | 
 96 |   it "should return a prerendered response if the url is part of the regex specific whitelist" do
 97 |     request = Rack::MockRequest.env_for "/search/things/123/page?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
 98 |     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
 99 |     response = Rack::Prerender.new(@app, whitelist: ['^/search.*page', '/help']).call(request)
100 | 
101 |     assert_equal ["<html></html>"], response[2]
102 |   end
103 | 
104 | 
105 |   it "should continue to app routes if the url is part of the regex specific blacklist" do
106 |     request = Rack::MockRequest.env_for "/search/things/123/page", "HTTP_USER_AGENT" => bot
107 |     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
108 | 
109 |     assert_equal "", response[2]
110 |   end
111 | 
112 |   it "should continue to app routes if the hashbang url is part of the regex specific blacklist" do
113 |     request = Rack::MockRequest.env_for "?_escaped_fragment_=/search/things/123/page", "HTTP_USER_AGENT" => bot
114 |     response = Rack::Prerender.new(@app, blacklist: ['/search', '/help']).call(request)
115 | 
116 |     assert_equal "", response[2]
117 |   end
118 | 
119 |   it "should return a prerendered response if the url is not part of the regex specific blacklist" do
120 |     request = Rack::MockRequest.env_for "/profile/search/blah", "HTTP_USER_AGENT" => bot
121 |     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
122 |     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
123 | 
124 |     assert_equal ["<html></html>"], response[2]
125 |   end
126 | 
127 | 
128 |   it "should continue to app routes if the referer is part of the regex specific blacklist" do
129 |     request = Rack::MockRequest.env_for "/api/results", "HTTP_USER_AGENT" => bot, "HTTP_REFERER" => '/search'
130 |     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
131 | 
132 |     assert_equal "", response[2]
133 |   end
134 | 
135 | 
136 |   it "should return a prerendered response if the referer is not part of the regex specific blacklist" do
137 |     request = Rack::MockRequest.env_for "/api/results", "HTTP_USER_AGENT" => bot, "HTTP_REFERER" => '/profile/search'
138 |     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
139 |     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
140 | 
141 |     assert_equal ["<html></html>"], response[2]
142 |   end
143 | 
144 | 
145 |   it "should return a prerendered response if a string is returned from before_render" do
146 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
147 |     response = Rack::Prerender.new(@app, before_render: Proc.new do |env| '<html>cached</html>' end).call(request)
148 | 
149 |     assert_equal ["<html>cached</html>"], response[2]
150 |   end
151 | 
152 | 
153 |   it "should return a prerendered response if a response is returned from before_render" do
154 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
155 |     response = Rack::Prerender.new(@app, before_render: Proc.new do |env| Rack::Response.new('<html>cached2</html>', 200, { 'test' => 'test2Header'}) end).call(request)
156 | 
157 |     assert_equal ["<html>cached2</html>"], response[2]
158 |     assert_equal response[0], 200
159 |     assert_equal( { 'test' => 'test2Header'}, response[1] )
160 |   end
161 | 
162 |   it "should return a prerendered response stripped of hop-by-hop headers" do
163 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
164 |     stub_request(:get, @prerender.build_api_url(request)).
165 |       with(:headers => { 'User-Agent' => bot }).
166 |       to_return(:body => "<html></html>", :status => 401, :headers => {
167 |                   'Content-Type' => 'text/html',
168 |                   'Transfer-Encoding' => 'Chunked',
169 |                   'Connection' => 'Keep-Alive',
170 |                   'Keep-Alive' => 'timeout=5, max=100',
171 |                   'Public' => 'GET HEAD',
172 |                   'Proxy-Authenticate' => 'Basic',
173 |                   'X-Drop-Test' => 'ShouldAlwaysHappen',
174 |                   'Upgrade' => 'dummy'
175 |                 })
176 |     response = Rack::Prerender.new(@app).call(request)
177 | 
178 |     assert_equal response[2], ["<html></html>"]
179 |     assert_equal response[0], 401
180 |     assert_equal( { 'content-type' => 'text/html', 'x-drop-test' => 'ShouldAlwaysHappen'}, response[1] )
181 |   end
182 | 
183 | it "should return a prerendered response stripped of custom-defined hop-by-hop headers" do
184 |     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
185 |     stub_request(:get, @prerender.build_api_url(request)).
186 |       with(:headers => { 'User-Agent' => bot }).
187 |       to_return(:body => "<html></html>", :status => 200, :headers => {
188 |                   'Content-Type' => 'text/html',
189 |                   'Connection' => 'Close, X-Drop-Test',
190 |                   'X-Drop-Test' => 'ShouldNeverHappen'
191 |                 })
192 |     response = Rack::Prerender.new(@app).call(request)
193 | 
194 |     assert_equal response[2], ["<html></html>"]
195 |     assert_equal response[0], 200
196 |     assert_equal( { 'content-type' => 'text/html' }, response[1] )
197 |   end
198 | 
199 |   describe '#buildApiUrl' do
200 |     it "should build the correct api url with the default url" do
201 |       request = Rack::MockRequest.env_for "https://google.com/search?q=javascript"
202 |       ENV['PRERENDER_SERVICE_URL'] = nil
203 |       assert_equal 'http://service.prerender.io/https://google.com/search?q=javascript', @prerender.build_api_url(request)
204 |     end
205 | 
206 | 
207 |     it "should build the correct api url with an environment variable url" do
208 |       ENV['PRERENDER_SERVICE_URL'] = 'http://prerenderurl.com'
209 |       request = Rack::MockRequest.env_for "https://google.com/search?q=javascript"
210 |       assert_equal 'http://prerenderurl.com/https://google.com/search?q=javascript', @prerender.build_api_url(request)
211 |       ENV['PRERENDER_SERVICE_URL'] = nil
212 |     end
213 | 
214 | 
215 |     it "should build the correct api url with an initialization variable url" do
216 |       @prerender = Rack::Prerender.new(@app, prerender_service_url: 'http://prerenderurl.com')
217 |       request = Rack::MockRequest.env_for "https://google.com/search?q=javascript"
218 |       assert_equal 'http://prerenderurl.com/https://google.com/search?q=javascript', @prerender.build_api_url(request)
219 |     end
220 | 
221 | 
222 |     it "should build the correct https api url with an initialization variable url" do
223 |       @prerender = Rack::Prerender.new(@app, prerender_service_url: 'https://prerenderurl.com')
224 |       request = Rack::MockRequest.env_for "https://google.com/search?q=javascript"
225 |       assert_equal 'https://prerenderurl.com/https://google.com/search?q=javascript', @prerender.build_api_url(request)
226 |     end
227 | 
228 | 
229 |     # Check CF-Visitor header in order to Work behind CloudFlare with Flexible SSL (https://support.cloudflare.com/hc/en-us/articles/200170536)
230 |     it "should build the correct api url for the Cloudflare Flexible SSL support" do
231 |       request = Rack::MockRequest.env_for "http://google.com/search?q=javascript", { 'CF-VISITOR' => '"scheme":"https"'}
232 |       ENV['PRERENDER_SERVICE_URL'] = nil
233 |       assert_equal 'http://service.prerender.io/https://google.com/search?q=javascript', @prerender.build_api_url(request)
234 |     end
235 | 
236 | 
237 |     # Check X-Forwarded-Proto because Heroku SSL Support terminates at the load balancer
238 |     it "should build the correct api url for the Heroku SSL Addon support with single value" do
239 |       request = Rack::MockRequest.env_for "http://google.com/search?q=javascript", { 'X-FORWARDED-PROTO' => 'https'}
240 |       ENV['PRERENDER_SERVICE_URL'] = nil
241 |       assert_equal 'http://service.prerender.io/https://google.com/search?q=javascript', @prerender.build_api_url(request)
242 |     end
243 | 
244 | 
245 |     # Check X-Forwarded-Proto because Heroku SSL Support terminates at the load balancer
246 |     it "should build the correct api url for the Heroku SSL Addon support with double value" do
247 |       request = Rack::MockRequest.env_for "http://google.com/search?q=javascript", { 'X-FORWARDED-PROTO' => 'https,http'}
248 |       ENV['PRERENDER_SERVICE_URL'] = nil
249 |       assert_equal 'http://service.prerender.io/https://google.com/search?q=javascript', @prerender.build_api_url(request)
250 |     end
251 | 
252 |     it "should build the correct api url for the protocol option" do
253 |       @prerender = Rack::Prerender.new(@app, protocol: 'https')
254 |       request = Rack::MockRequest.env_for "http://google.com/search?q=javascript"
255 |       ENV['PRERENDER_SERVICE_URL'] = nil
256 |       assert_equal 'http://service.prerender.io/https://google.com/search?q=javascript', @prerender.build_api_url(request)
257 |     end
258 |   end
259 | 
260 | end
261 | 


--------------------------------------------------------------------------------
/test/test_helper.rb:
--------------------------------------------------------------------------------
1 | require 'minitest/autorun'
2 | require 'webmock/minitest'
3 | require 'rack'
4 | require File.expand_path('../../lib/prerender_rails.rb', __FILE__)


--------------------------------------------------------------------------------