├── .gitignore ├── .npmignore ├── .travis.yml ├── CHANGELOG.md ├── README.md ├── index.js ├── package-lock.json ├── package.json └── test ├── express3-spec.js ├── express4-spec.js ├── index-spec.js └── support ├── express3 ├── app.js ├── package.json ├── public │ ├── javascripts │ │ └── todo.js │ ├── partials │ │ └── todo.html │ └── stylesheets │ │ └── todo.css ├── routes │ ├── index.js │ └── user.js └── views │ ├── index.html │ └── index.jade └── express4 ├── app.js ├── bin └── www ├── package.json ├── public ├── javascripts │ └── todo.js ├── partials │ └── todo.html └── stylesheets │ └── todo.css ├── routes ├── index.js └── users.js └── views ├── error.pug ├── index.html ├── index.pug └── layout.pug /.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | .DS_Store -------------------------------------------------------------------------------- /.npmignore: -------------------------------------------------------------------------------- 1 | test 2 | .travis.yml 3 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: node_js 2 | node_js: 3 | - "10" 4 | script: 5 | - cd test/support/express3 && npm install 6 | - cd ../express4 && npm install 7 | - cd ../../.. 8 | - npm test 9 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 3.8.0 (2024-02-20) 2 | Added: 3 | - Add X-Prerender-Int-Type to recognized user integration type 4 | 5 | ## 3.7.0 (2023-05-22) 6 | Added: 7 | - Add Google-InspectionTool to the recognized user agents 8 | 9 | ## 3.6.0 (2023-03-15) 10 | Added: 11 | 12 | - .webp, .woff2 extensions to the list of extensions that are not prerendered 13 | - Add AhrefsBot to the recognized user agents 14 | - Add AhrefsSiteAudit to the recognized user agents 15 | - Add Iframely to the recognized user agents 16 | - Add screaming frog to the recognized user agents 17 | 18 | ## 3.5.0 () 19 | Added: 20 | 21 | - cancelRender flag in afterRender callback 22 | 23 | ## 3.4.2 (2022-03-21) 24 | Fixed: 25 | 26 | - Extensions must be at the end of the pathname. 27 | Solves cases when the URL contained pieces of extensions like www.example.com/this.is.data.of.something.html - In this case the because of .dat the URL was filtered 28 | 29 | ## 3.4.1 (2022-01-17) 30 | Added: 31 | 32 | - Add seznam.cz bot to the recognized user agents 33 | 34 | ## 3.4.0 (2021-11-09) 35 | Changed: 36 | 37 | - Replace requests lib with the native nodejs http and https modules 38 | 39 | ## 3.3.0 (2021-11-04) 40 | Changed: 41 | 42 | - Add telegram bot to the recognized user agents 43 | 44 | ## 3.2.5 (2019-07-30) 45 | Changed: 46 | 47 | - Updated lodash and mocha to resolve security vulnerability warning 48 | 49 | ## 3.2.4 (2019-04-08) 50 | Changed: 51 | 52 | - Updated devDependencies to latest versions for running tests 53 | 54 | ## 3.2.3 (2019-04-08) 55 | Changed: 56 | 57 | - Modified request dependency from ^2.83.0 to ^2.88.0 and added package-lock.json file 58 | 59 | ## 3.2.2 (2019-03-27) 60 | Added: 61 | 62 | - Added Chrome-Lighthouse user agent so Google page speed insight tool will show prerendered pages 63 | 64 | ## 3.2.1 (2018-08-29) 65 | Added: 66 | 67 | - Looks like I accidentally added `yahoo` instead of `yandex` in the last commit. Fixing that by adding yandex now 68 | 69 | ## 3.2.0 (2018-07-18) 70 | Added: 71 | 72 | - Added Googlebot, Bingbot, and Yandex to user agent check for new Dynamic Rendering and phase-out of escaped fragment URLs 73 | - Added check for x-prerender header for mobile adaptive feature for websites that serve different HTML to mobile vs desktop crawlers 74 | 75 | ## 3.1.1 (2018-03-01) 76 | Changed: 77 | 78 | - Fixing test that checked for the old endpoint 79 | 80 | ## 3.1.0 (2018-03-01) 81 | Changed: 82 | 83 | - Point to https endpoint for prerender service instead of http 84 | 85 | ## 3.0.0 (2018-03-01) 86 | Removed: 87 | 88 | - Node 0.10 support 89 | - Node 0.12 support 90 | 91 | Changed: 92 | 93 | - Bumped request dependency version 94 | 95 | Added: 96 | 97 | - Node 8 to travis.yml 98 | 99 | ## 2.8.0 (2018-02-21) 100 | Added: 101 | 102 | - Added Bitrix24 and Xing user agents. 103 | 104 | ## 2.7.4 (2017-09-22) 105 | Changed: 106 | 107 | - Fixing wrong version of request. :) 108 | 109 | ## 2.7.3 (2017-09-22) 110 | Changed: 111 | 112 | - Pinned node request module to 2.81.1 since they bumped a dependency version that required a newer version of node. prerender-node still supports older versions of node so pinned an earlier version of request until we officially dont support older versions of node. 113 | 114 | ## 2.7.2 (2017-07-05) 115 | New features: 116 | 117 | - Added pinterestbot to user agents being checked. 118 | 119 | ## 2.7.1 (2017-04-13) 120 | New features: 121 | 122 | - Added Qwantify to user agents being checked. 123 | 124 | Changed 125 | 126 | - Added `host` parameter documentation to README 127 | 128 | ## 2.7.0 (2016-12-20) 129 | New features: 130 | 131 | - Adds an option to be able to pass options to the request sent to the prerender. 132 | 133 | 134 | ## 2.6.0 (2016-12-13) 135 | New features: 136 | 137 | - Check for Google Page Speed user agent in order to send prerendered pages when URLs are tested through their tool 138 | 139 | Changed 140 | 141 | - extensionsToIgnore check is now case insensitive 142 | 143 | ## 2.5.0 (2016-10-26) 144 | 145 | Changed 146 | 147 | - When creating the URL to send to Prerender, check to see if `x-forwarded-host` is set and use that before using the `host` header. 148 | 149 | ## 2.4.0 (2016-8-30) 150 | 151 | New features: 152 | 153 | - Check for Skype user agent in order to send prerendered pages when URLs are shared through Skype chat 154 | - Check for nuzzel user agent 155 | - Check for discordbot user agent 156 | 157 | Changed 158 | 159 | - Don't send the `host` header if `forwardHeaders` is enabled to prevent issues with servers that take the Host header and apply it to the URL 160 | 161 | ## 2.3.0 (2016-6-1) 162 | 163 | New features: 164 | 165 | - Check for Tumblr user agent in order to send prerendered pages when URLs are shared through Tumblr 166 | - Check for bitlybot user agent in order to send prerendered pages when URLs are shared through Bitly 167 | 168 | ## 2.2.2 (2016-5-8) 169 | 170 | Bugfixes: 171 | 172 | - fix compatibility with Node.js v6.0.0 173 | 174 | ## 2.2.1 (2016-3-21) 175 | 176 | Bugfixes: 177 | 178 | - Check for more specific pinterest crawler user agent since their iOS app changes the browser user agent to include "pinterest" 179 | - Added svg to list of extensions to check 180 | 181 | ## 2.2.0 (2015-12-29) 182 | 183 | New features: 184 | 185 | - Check for Whatsapp user agent in order to send prerendered pages when URLs are shared through Whatsapp 186 | 187 | ## 2.1.0 (2015-09-17) 188 | 189 | New features: 190 | 191 | - Check for Applebot user agent in order to send prerendered pages to apple for Siri on iOS9 192 | 193 | ## 2.0.2 (2015-07-24) 194 | 195 | Bugfixes: 196 | 197 | - Make sure we pass through the error, request, and Prerender response to afterRender 198 | 199 | Fixed an issue where the original request and Prerender response wasn't getting passed to afterRender after recent change to pass err through. Now works correctly: `afterRender(err, req, prerenderResponse)` 200 | 201 | ## 2.0.1 (2015-07-20) 202 | 203 | Bugfixes: 204 | 205 | - Send HTTPS URLs through correctly. 206 | 207 | Fixed an issue where `http` was always added to the URLs sent to Prerender even if the URL was https. 208 | 209 | ## 2.0.0 (2015-07-15) 210 | 211 | Bugfixes: 212 | 213 | - Stop swallowing errors if the prerender service fails. 214 | 215 | Before: If the prerender service errors out (e.g., not available and the request times out), the error is swallowed and `next` called with no args. 216 | 217 | After: error is passed through to `next`, and `prerender.afterRenderFn` is handed `err` as the first arg. Makes it much easier to know if you're misconfiguring something or forgot to start the prerender server locally when testing. 218 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Prerender Node [](https://travis-ci.org/prerender/prerender-node) [](http://badge.fury.io/js/prerender-node) 2 | =========================== 3 | 4 | Google, Facebook, Twitter, and Bing are constantly trying to view your website... but Google is the only crawler that executes a meaningful amount of JavaScript and Google even admits that they can execute JavaScript weeks after actually crawling. Prerender allows you to serve the full HTML of your website back to Google and other crawlers so that they don't have to execute any JavaScript. [Google recommends using Prerender.io](https://developers.google.com/search/docs/guides/dynamic-rendering) to prevent indexation issues on sites with large amounts of JavaScript. 5 | 6 | Prerender is perfect for Angular SEO, React SEO, Vue SEO, and any other JavaScript framework. 7 | 8 | This middleware intercepts requests to your Node.js website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the JavaScript for that page. That HTML is then returned to the crawler. 9 | 10 | via npm: 11 | 12 | $ npm install prerender-node --save 13 | 14 | And when you set up your express app, add: 15 | 16 | ```js 17 | app.use(require('prerender-node')); 18 | ``` 19 | 20 | or if you have an account on [prerender.io](https://prerender.io/) and want to use your token: 21 | 22 | ```js 23 | app.use(require('prerender-node').set('prerenderToken', 'YOUR_TOKEN')); 24 | ``` 25 | 26 | `Note` If you're testing locally, you'll need to run the [prerender server](https://github.com/prerender/prerender) locally so that it has access to your server. 27 | 28 | This middleware is tested with Express3 and Express4, but has no explicit dependency on either. 29 | 30 | ## Testing 31 | 32 | The best way to test the prerendered page is to [set the User Agent of your browser to Googlebot's user agent](https://developers.google.com/web/tools/chrome-devtools/device-mode/override-user-agent) and visit your URL directly. If you View Source on that URL, you should see the static HTML version of the page with the ` 7 | 8 | 9 | 10 | 11 |
12 | 13 | 14 |