├── .github ├── FUNDING.yml ├── ISSUE_TEMPLATE.yml └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── History.md ├── LICENSE ├── Readme.md ├── _examples ├── go │ ├── Readme.md │ ├── main.go │ └── static.json └── node │ ├── Readme.md │ ├── package-lock.json │ ├── package.json │ ├── public │ └── style.css │ ├── server.js │ ├── static.json │ └── views │ ├── index.pug │ └── post.pug ├── cmd └── staticgen │ └── main.go ├── config.go ├── events.go ├── go.mod ├── go.sum ├── internal ├── crawler │ ├── crawler.go │ └── crawler_test.go └── deduplicator │ ├── deduplicator.go │ └── deduplicator_test.go ├── reporter.go └── staticgen.go /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | github: tj -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.yml: -------------------------------------------------------------------------------- 1 | ## Prerequisites 2 | 3 | * [ ] I searched to see if the issue already exists. 4 | 5 | ## Description 6 | 7 | Describe the bug or feature. 8 | 9 | ## Steps to Reproduce 10 | 11 | Describe the steps required to reproduce the issue if applicable. 12 | 13 | ## Slack 14 | 15 | Join us on Slack https://chat.apex.sh/ 16 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please open an issue and discuss changes before spending time on them, unless the change is trivial or an issue already exists. 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .envrc 2 | dist 3 | build/ 4 | node_modules/ 5 | test 6 | /static.json 7 | -------------------------------------------------------------------------------- /History.md: -------------------------------------------------------------------------------- 1 | 2 | v1.1.0 / 2020-09-23 3 | =================== 4 | 5 | * add Allow404 support 6 | 7 | v1.0.1 / 2020-01-29 8 | =================== 9 | 10 | * add `version` command to output the Staticgen version number 11 | 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2019 TJ Holowaychuk tj@tjholowaychuk.com 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | 'Software'), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be 14 | included in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 21 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 22 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # Staticgen 2 | 3 | A static website generator that lets you use HTTP servers and frameworks you already know. Just tell Staticgen how to start your server, then watch it crawl your site and generate a static version with all of the pages and assets required. 4 | 5 | ## About 6 | 7 | If you're unfamiliar, you can actually use the decades-old [wget command](https://apex.sh/blog/post/pre-render-wget/) to output a static website from a dynamic one, this project is purpose-built for the same idea, letting your team to use whatever HTTP servers and frameworks you're already familiar with, in any language. 8 | 9 | I haven't done any scientific benchmarks or comparisons yet, but here are some results on my 2014 8-core MBP: 10 | 11 | - Compiles 3,296 pages of the [Signal v. Noise](https://m.signalvnoise.com/) blog in 1 second 12 | - Compiles my [Apex Software](https://apex.sh/) site in 150ms 13 | 14 | ## Installation 15 | 16 | Via [gobinaries.com](https://gobinaries.com): 17 | 18 | ```sh 19 | $ curl -sf https://gobinaries.com/tj/staticgen/cmd/staticgen | sh 20 | ``` 21 | 22 | ## Configuration 23 | 24 | Configuration is stored within a `./static.json` file in your project's root directory. The following options are available: 25 | 26 | - __command__ — The server command executed before crawling. 27 | - __url__ — The target website to crawl. Defaults to `"http://127.0.0.1:3000"`. 28 | - __dir__ — The static website output directory. Defaults to `"build"`. 29 | - __pages__ — A list of paths added to crawl, typically including unlinked pages such as landing pages. Defaults to `[]`. 30 | - __concurrency__ — The number of concurrent pages to crawl. Defaults to `30`. 31 | 32 | ## Guide 33 | 34 | First create the `./static.json` configuration file, for example here's the config for Go server, the only required property is `command`: 35 | 36 | ```json 37 | { 38 | "command": "go run main.go", 39 | "concurrency": 50, 40 | "dir": "dist" 41 | } 42 | ``` 43 | 44 | Below is an example of a Node.js server, note that `NODE_ENV` is assigned to production so that optimizations such as Express template caches are used to improve serving performance. 45 | 46 | ```json 47 | { 48 | "command": "NODE_ENV=production node server.js" 49 | } 50 | ``` 51 | 52 | Run the `staticgen` command to start the pre-rendering process: 53 | 54 | ``` 55 | $ staticgen 56 | ``` 57 | 58 | Staticgen executes the `command` you provided, waits for the server to become available on the `url` configured. The pages and assets are copied to the `dir` configured and then your server is shut down. 59 | 60 | By default the timeout for the generation process is 15 minutes, depending on your situation you may want to increase or decrease this with the `-t, --timeout` flag, here are some examples: 61 | 62 | ``` 63 | $ staticgen -t 30s 64 | $ staticgen -t 15m 65 | $ staticgen -t 1h 66 | ``` 67 | 68 | When launching the `command`, Staticgen sets the `STATICGEN` environment variable to `1`, allowing you to alter behaviour if necessary. 69 | 70 | To view the pre-rendered site run the following command to start a static file server and open the browser: 71 | 72 | ``` 73 | $ staticgen serve 74 | ``` 75 | 76 | See the [examples](./_examples) directory for full examples. 77 | 78 | ## Notes 79 | 80 | Staticgen does not pre-render using a headless browser, this makes it faster, however it means that you cannot rely on client-side JavaScript manipulating the page. 81 | 82 | 83 | --- 84 | 85 | [![GoDoc](https://godoc.org/github.com/tj/staticgen?status.svg)](https://godoc.org/github.com/tj/staticgen) 86 | ![](https://img.shields.io/badge/license-MIT-blue.svg) 87 | ![](https://img.shields.io/badge/status-stable-green.svg) 88 | 89 | ## Sponsors 90 | 91 | This project is sponsored by [CTO.ai](https://cto.ai/), making it easy for development teams to create and share workflow automations without leaving the command line. 92 | 93 | [![](https://apex-software.imgix.net/github/sponsors/cto.png)](https://cto.ai/) 94 | 95 | And my [GitHub sponsors](https://github.com/sponsors/tj): 96 | 97 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/0) 98 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/1) 99 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/2) 100 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/3) 101 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/4) 102 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/5) 103 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/6) 104 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/7) 105 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/8) 106 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/9) 107 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/10) 108 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/11) 109 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/12) 110 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/13) 111 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/14) 112 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/15) 113 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/16) 114 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/17) 115 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/18) 116 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/19) 117 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/20) 118 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/21) 119 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/22) 120 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/23) 121 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/24) 122 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/25) 123 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/26) 124 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/27) 125 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/28) 126 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/29) 127 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/30) 128 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/31) 129 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/32) 130 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/33) 131 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/34) 132 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/35) 133 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/36) 134 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/37) 135 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/38) 136 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/39) 137 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/40) 138 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/41) 139 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/42) 140 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/43) 141 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/44) 142 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/45) 143 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/46) 144 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/47) 145 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/48) 146 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/49) 147 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/50) 148 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/51) 149 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/52) 150 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/53) 151 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/54) 152 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/55) 153 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/56) 154 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/57) 155 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/58) 156 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/59) 157 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/60) 158 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/61) 159 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/62) 160 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/63) 161 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/64) 162 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/65) 163 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/66) 164 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/67) 165 | [](https://sponsors-api-u2fftug6kq-uc.a.run.app/sponsor/profile/68) 166 | 167 | -------------------------------------------------------------------------------- /_examples/go/Readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Go 3 | 4 | This example is used to illustrate the performance capabilities, without being held back by the speed of frameworks. 5 | 6 | To compile the site, run `staticgen` in the project's directory, for example: 7 | 8 | ``` 9 | $ cd _examples/go 10 | $ staticgen 11 | $ ls build 12 | ``` -------------------------------------------------------------------------------- /_examples/go/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "log" 6 | "net/http" 7 | "strconv" 8 | ) 9 | 10 | // Post model. 11 | type Post struct { 12 | ID int 13 | Title string 14 | Teaser string 15 | Body string 16 | } 17 | 18 | var posts []Post 19 | 20 | func init() { 21 | for i := 0; i < 10000; i++ { 22 | posts = append(posts, Post{ 23 | ID: i, 24 | Title: fmt.Sprintf("Blog post #%d", i), 25 | Teaser: "Some content here.", 26 | Body: "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse venenatis sem sit amet felis imperdiet porta. Cras eget cursus ante. Suspendisse lobortis aliquam felis feugiat vehicula. Phasellus id mattis lacus. Morbi pellentesque nibh ut turpis finibus porttitor. Vivamus vestibulum dui vel mi sagittis dictum. Ut faucibus commodo magna vel auctor. Donec ut egestas nunc, at ullamcorper mi. Maecenas in fringilla mauris. Mauris non enim eu diam elementum ornare quis at est. Morbi et semper ex, vitae cursus lectus. Nulla auctor, nibh vel tempus ultrices, erat metus auctor ligula, vel tincidunt quam quam ac ligula. Nullam in elit dui. In tellus quam, suscipit eu ultrices sit amet, volutpat eu tellus. Integer pretium et dolor eu faucibus. Proin aliquam blandit rhoncus.", 27 | }) 28 | } 29 | } 30 | 31 | func main() { 32 | http.HandleFunc("/", list) 33 | http.Handle("/posts/", http.StripPrefix("/posts/", http.HandlerFunc(post))) 34 | fmt.Printf("Server starting on :3000\n") 35 | log.Fatal(http.ListenAndServe(":3000", nil)) 36 | } 37 | 38 | func list(w http.ResponseWriter, r *http.Request) { 39 | w.Header().Set("Content-Type", "text/html") 40 | for _, p := range posts { 41 | fmt.Fprintf(w, ` 42 |
43 |

%s

44 |

%s

45 | View article 46 |
47 | `, p.Title, p.Teaser, p.ID) 48 | } 49 | } 50 | 51 | func post(w http.ResponseWriter, r *http.Request) { 52 | id, _ := strconv.ParseInt(r.URL.Path, 10, 64) 53 | p := posts[id] 54 | w.Header().Set("Content-Type", "text/html") 55 | fmt.Fprintf(w, ` 56 |

%s

57 |

%s

58 | `, p.Title, p.Body) 59 | } 60 | -------------------------------------------------------------------------------- /_examples/go/static.json: -------------------------------------------------------------------------------- 1 | { 2 | "command": "go run main.go", 3 | "concurrency": 50 4 | } -------------------------------------------------------------------------------- /_examples/node/Readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Node 3 | 4 | Setup: 5 | 6 | ``` 7 | $ npm install 8 | ``` 9 | 10 | To compile the site, run `staticgen` in the project's directory, for example: 11 | 12 | ``` 13 | $ cd _examples/node 14 | $ staticgen 15 | $ ls build 16 | ``` 17 | 18 | Note that `NODE_ENV=production` is used in the static.json configuration, this improves the performance with template compilation caching. -------------------------------------------------------------------------------- /_examples/node/package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "example", 3 | "requires": true, 4 | "lockfileVersion": 1, 5 | "dependencies": { 6 | "@types/babel-types": { 7 | "version": "7.0.7", 8 | "resolved": "https://registry.npmjs.org/@types/babel-types/-/babel-types-7.0.7.tgz", 9 | "integrity": "sha512-dBtBbrc+qTHy1WdfHYjBwRln4+LWqASWakLHsWHR2NWHIFkv4W3O070IGoGLEBrJBvct3r0L1BUPuvURi7kYUQ==" 10 | }, 11 | "@types/babylon": { 12 | "version": "6.16.5", 13 | "resolved": "https://registry.npmjs.org/@types/babylon/-/babylon-6.16.5.tgz", 14 | "integrity": "sha512-xH2e58elpj1X4ynnKp9qSnWlsRTIs6n3tgLGNfwAGHwePw0mulHQllV34n0T25uYSu1k0hRKkWXF890B1yS47w==", 15 | "requires": { 16 | "@types/babel-types": "*" 17 | } 18 | }, 19 | "accepts": { 20 | "version": "1.3.7", 21 | "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz", 22 | "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==", 23 | "requires": { 24 | "mime-types": "~2.1.24", 25 | "negotiator": "0.6.2" 26 | } 27 | }, 28 | "acorn": { 29 | "version": "3.3.0", 30 | "resolved": "https://registry.npmjs.org/acorn/-/acorn-3.3.0.tgz", 31 | "integrity": "sha1-ReN/s56No/JbruP/U2niu18iAXo=" 32 | }, 33 | "acorn-globals": { 34 | "version": "3.1.0", 35 | "resolved": "https://registry.npmjs.org/acorn-globals/-/acorn-globals-3.1.0.tgz", 36 | "integrity": "sha1-/YJw9x+7SZawBPqIDuXUZXOnMb8=", 37 | "requires": { 38 | "acorn": "^4.0.4" 39 | }, 40 | "dependencies": { 41 | "acorn": { 42 | "version": "4.0.13", 43 | "resolved": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz", 44 | "integrity": "sha1-EFSVrlNh1pe9GVyCUZLhrX8lN4c=" 45 | } 46 | } 47 | }, 48 | "align-text": { 49 | "version": "0.1.4", 50 | "resolved": "https://registry.npmjs.org/align-text/-/align-text-0.1.4.tgz", 51 | "integrity": "sha1-DNkKVhCT810KmSVsIrcGlDP60Rc=", 52 | "requires": { 53 | "kind-of": "^3.0.2", 54 | "longest": "^1.0.1", 55 | "repeat-string": "^1.5.2" 56 | } 57 | }, 58 | "array-flatten": { 59 | "version": "1.1.1", 60 | "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", 61 | "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" 62 | }, 63 | "asap": { 64 | "version": "2.0.6", 65 | "resolved": "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz", 66 | "integrity": "sha1-5QNHYR1+aQlDIIu9r+vLwvuGbUY=" 67 | }, 68 | "babel-runtime": { 69 | "version": "6.26.0", 70 | "resolved": "https://registry.npmjs.org/babel-runtime/-/babel-runtime-6.26.0.tgz", 71 | "integrity": "sha1-llxwWGaOgrVde/4E/yM3vItWR/4=", 72 | "requires": { 73 | "core-js": "^2.4.0", 74 | "regenerator-runtime": "^0.11.0" 75 | } 76 | }, 77 | "babel-types": { 78 | "version": "6.26.0", 79 | "resolved": "https://registry.npmjs.org/babel-types/-/babel-types-6.26.0.tgz", 80 | "integrity": "sha1-o7Bz+Uq0nrb6Vc1lInozQ4BjJJc=", 81 | "requires": { 82 | "babel-runtime": "^6.26.0", 83 | "esutils": "^2.0.2", 84 | "lodash": "^4.17.4", 85 | "to-fast-properties": "^1.0.3" 86 | } 87 | }, 88 | "babylon": { 89 | "version": "6.18.0", 90 | "resolved": "https://registry.npmjs.org/babylon/-/babylon-6.18.0.tgz", 91 | "integrity": "sha512-q/UEjfGJ2Cm3oKV71DJz9d25TPnq5rhBVL2Q4fA5wcC3jcrdn7+SssEybFIxwAvvP+YCsCYNKughoF33GxgycQ==" 92 | }, 93 | "body-parser": { 94 | "version": "1.19.0", 95 | "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz", 96 | "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==", 97 | "requires": { 98 | "bytes": "3.1.0", 99 | "content-type": "~1.0.4", 100 | "debug": "2.6.9", 101 | "depd": "~1.1.2", 102 | "http-errors": "1.7.2", 103 | "iconv-lite": "0.4.24", 104 | "on-finished": "~2.3.0", 105 | "qs": "6.7.0", 106 | "raw-body": "2.4.0", 107 | "type-is": "~1.6.17" 108 | } 109 | }, 110 | "bytes": { 111 | "version": "3.1.0", 112 | "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz", 113 | "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==" 114 | }, 115 | "camelcase": { 116 | "version": "1.2.1", 117 | "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-1.2.1.tgz", 118 | "integrity": "sha1-m7UwTS4LVmmLLHWLCKPqqdqlijk=" 119 | }, 120 | "center-align": { 121 | "version": "0.1.3", 122 | "resolved": "https://registry.npmjs.org/center-align/-/center-align-0.1.3.tgz", 123 | "integrity": "sha1-qg0yYptu6XIgBBHL1EYckHvCt60=", 124 | "requires": { 125 | "align-text": "^0.1.3", 126 | "lazy-cache": "^1.0.3" 127 | } 128 | }, 129 | "character-parser": { 130 | "version": "2.2.0", 131 | "resolved": "https://registry.npmjs.org/character-parser/-/character-parser-2.2.0.tgz", 132 | "integrity": "sha1-x84o821LzZdE5f/CxfzeHHMmH8A=", 133 | "requires": { 134 | "is-regex": "^1.0.3" 135 | } 136 | }, 137 | "clean-css": { 138 | "version": "4.2.1", 139 | "resolved": "https://registry.npmjs.org/clean-css/-/clean-css-4.2.1.tgz", 140 | "integrity": "sha512-4ZxI6dy4lrY6FHzfiy1aEOXgu4LIsW2MhwG0VBKdcoGoH/XLFgaHSdLTGr4O8Be6A8r3MOphEiI8Gc1n0ecf3g==", 141 | "requires": { 142 | "source-map": "~0.6.0" 143 | } 144 | }, 145 | "cliui": { 146 | "version": "2.1.0", 147 | "resolved": "https://registry.npmjs.org/cliui/-/cliui-2.1.0.tgz", 148 | "integrity": "sha1-S0dXYP+AJkx2LDoXGQMukcf+oNE=", 149 | "requires": { 150 | "center-align": "^0.1.1", 151 | "right-align": "^0.1.1", 152 | "wordwrap": "0.0.2" 153 | } 154 | }, 155 | "constantinople": { 156 | "version": "3.1.2", 157 | "resolved": "https://registry.npmjs.org/constantinople/-/constantinople-3.1.2.tgz", 158 | "integrity": "sha512-yePcBqEFhLOqSBtwYOGGS1exHo/s1xjekXiinh4itpNQGCu4KA1euPh1fg07N2wMITZXQkBz75Ntdt1ctGZouw==", 159 | "requires": { 160 | "@types/babel-types": "^7.0.0", 161 | "@types/babylon": "^6.16.2", 162 | "babel-types": "^6.26.0", 163 | "babylon": "^6.18.0" 164 | } 165 | }, 166 | "content-disposition": { 167 | "version": "0.5.3", 168 | "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz", 169 | "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==", 170 | "requires": { 171 | "safe-buffer": "5.1.2" 172 | } 173 | }, 174 | "content-type": { 175 | "version": "1.0.4", 176 | "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz", 177 | "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==" 178 | }, 179 | "cookie": { 180 | "version": "0.4.0", 181 | "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz", 182 | "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==" 183 | }, 184 | "cookie-signature": { 185 | "version": "1.0.6", 186 | "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz", 187 | "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw=" 188 | }, 189 | "core-js": { 190 | "version": "2.6.9", 191 | "resolved": "https://registry.npmjs.org/core-js/-/core-js-2.6.9.tgz", 192 | "integrity": "sha512-HOpZf6eXmnl7la+cUdMnLvUxKNqLUzJvgIziQ0DiF3JwSImNphIqdGqzj6hIKyX04MmV0poclQ7+wjWvxQyR2A==" 193 | }, 194 | "debug": { 195 | "version": "2.6.9", 196 | "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", 197 | "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==", 198 | "requires": { 199 | "ms": "2.0.0" 200 | } 201 | }, 202 | "decamelize": { 203 | "version": "1.2.0", 204 | "resolved": "https://registry.npmjs.org/decamelize/-/decamelize-1.2.0.tgz", 205 | "integrity": "sha1-9lNNFRSCabIDUue+4m9QH5oZEpA=" 206 | }, 207 | "depd": { 208 | "version": "1.1.2", 209 | "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz", 210 | "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=" 211 | }, 212 | "destroy": { 213 | "version": "1.0.4", 214 | "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz", 215 | "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA=" 216 | }, 217 | "doctypes": { 218 | "version": "1.1.0", 219 | "resolved": "https://registry.npmjs.org/doctypes/-/doctypes-1.1.0.tgz", 220 | "integrity": "sha1-6oCxBqh1OHdOijpKWv4pPeSJ4Kk=" 221 | }, 222 | "ee-first": { 223 | "version": "1.1.1", 224 | "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", 225 | "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0=" 226 | }, 227 | "encodeurl": { 228 | "version": "1.0.2", 229 | "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz", 230 | "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=" 231 | }, 232 | "escape-html": { 233 | "version": "1.0.3", 234 | "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", 235 | "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg=" 236 | }, 237 | "esutils": { 238 | "version": "2.0.3", 239 | "resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz", 240 | "integrity": "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==" 241 | }, 242 | "etag": { 243 | "version": "1.8.1", 244 | "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", 245 | "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=" 246 | }, 247 | "express": { 248 | "version": "4.17.1", 249 | "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz", 250 | "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==", 251 | "requires": { 252 | "accepts": "~1.3.7", 253 | "array-flatten": "1.1.1", 254 | "body-parser": "1.19.0", 255 | "content-disposition": "0.5.3", 256 | "content-type": "~1.0.4", 257 | "cookie": "0.4.0", 258 | "cookie-signature": "1.0.6", 259 | "debug": "2.6.9", 260 | "depd": "~1.1.2", 261 | "encodeurl": "~1.0.2", 262 | "escape-html": "~1.0.3", 263 | "etag": "~1.8.1", 264 | "finalhandler": "~1.1.2", 265 | "fresh": "0.5.2", 266 | "merge-descriptors": "1.0.1", 267 | "methods": "~1.1.2", 268 | "on-finished": "~2.3.0", 269 | "parseurl": "~1.3.3", 270 | "path-to-regexp": "0.1.7", 271 | "proxy-addr": "~2.0.5", 272 | "qs": "6.7.0", 273 | "range-parser": "~1.2.1", 274 | "safe-buffer": "5.1.2", 275 | "send": "0.17.1", 276 | "serve-static": "1.14.1", 277 | "setprototypeof": "1.1.1", 278 | "statuses": "~1.5.0", 279 | "type-is": "~1.6.18", 280 | "utils-merge": "1.0.1", 281 | "vary": "~1.1.2" 282 | } 283 | }, 284 | "finalhandler": { 285 | "version": "1.1.2", 286 | "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz", 287 | "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==", 288 | "requires": { 289 | "debug": "2.6.9", 290 | "encodeurl": "~1.0.2", 291 | "escape-html": "~1.0.3", 292 | "on-finished": "~2.3.0", 293 | "parseurl": "~1.3.3", 294 | "statuses": "~1.5.0", 295 | "unpipe": "~1.0.0" 296 | } 297 | }, 298 | "forwarded": { 299 | "version": "0.1.2", 300 | "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz", 301 | "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=" 302 | }, 303 | "fresh": { 304 | "version": "0.5.2", 305 | "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz", 306 | "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=" 307 | }, 308 | "function-bind": { 309 | "version": "1.1.1", 310 | "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.1.tgz", 311 | "integrity": "sha512-yIovAzMX49sF8Yl58fSCWJ5svSLuaibPxXQJFLmBObTuCr0Mf1KiPopGM9NiFjiYBCbfaa2Fh6breQ6ANVTI0A==" 312 | }, 313 | "has": { 314 | "version": "1.0.3", 315 | "resolved": "https://registry.npmjs.org/has/-/has-1.0.3.tgz", 316 | "integrity": "sha512-f2dvO0VU6Oej7RkWJGrehjbzMAjFp5/VKPp5tTpWIV4JHHZK1/BxbFRtf/siA2SWTe09caDmVtYYzWEIbBS4zw==", 317 | "requires": { 318 | "function-bind": "^1.1.1" 319 | } 320 | }, 321 | "http-errors": { 322 | "version": "1.7.2", 323 | "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz", 324 | "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==", 325 | "requires": { 326 | "depd": "~1.1.2", 327 | "inherits": "2.0.3", 328 | "setprototypeof": "1.1.1", 329 | "statuses": ">= 1.5.0 < 2", 330 | "toidentifier": "1.0.0" 331 | } 332 | }, 333 | "iconv-lite": { 334 | "version": "0.4.24", 335 | "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz", 336 | "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==", 337 | "requires": { 338 | "safer-buffer": ">= 2.1.2 < 3" 339 | } 340 | }, 341 | "inherits": { 342 | "version": "2.0.3", 343 | "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", 344 | "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" 345 | }, 346 | "ipaddr.js": { 347 | "version": "1.9.0", 348 | "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.0.tgz", 349 | "integrity": "sha512-M4Sjn6N/+O6/IXSJseKqHoFc+5FdGJ22sXqnjTpdZweHK64MzEPAyQZyEU3R/KRv2GLoa7nNtg/C2Ev6m7z+eA==" 350 | }, 351 | "is-buffer": { 352 | "version": "1.1.6", 353 | "resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-1.1.6.tgz", 354 | "integrity": "sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w==" 355 | }, 356 | "is-expression": { 357 | "version": "3.0.0", 358 | "resolved": "https://registry.npmjs.org/is-expression/-/is-expression-3.0.0.tgz", 359 | "integrity": "sha1-Oayqa+f9HzRx3ELHQW5hwkMXrJ8=", 360 | "requires": { 361 | "acorn": "~4.0.2", 362 | "object-assign": "^4.0.1" 363 | }, 364 | "dependencies": { 365 | "acorn": { 366 | "version": "4.0.13", 367 | "resolved": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz", 368 | "integrity": "sha1-EFSVrlNh1pe9GVyCUZLhrX8lN4c=" 369 | } 370 | } 371 | }, 372 | "is-promise": { 373 | "version": "2.1.0", 374 | "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-2.1.0.tgz", 375 | "integrity": "sha1-eaKp7OfwlugPNtKy87wWwf9L8/o=" 376 | }, 377 | "is-regex": { 378 | "version": "1.0.4", 379 | "resolved": "https://registry.npmjs.org/is-regex/-/is-regex-1.0.4.tgz", 380 | "integrity": "sha1-VRdIm1RwkbCTDglWVM7SXul+lJE=", 381 | "requires": { 382 | "has": "^1.0.1" 383 | } 384 | }, 385 | "js-stringify": { 386 | "version": "1.0.2", 387 | "resolved": "https://registry.npmjs.org/js-stringify/-/js-stringify-1.0.2.tgz", 388 | "integrity": "sha1-Fzb939lyTyijaCrcYjCufk6Weds=" 389 | }, 390 | "jstransformer": { 391 | "version": "1.0.0", 392 | "resolved": "https://registry.npmjs.org/jstransformer/-/jstransformer-1.0.0.tgz", 393 | "integrity": "sha1-7Yvwkh4vPx7U1cGkT2hwntJHIsM=", 394 | "requires": { 395 | "is-promise": "^2.0.0", 396 | "promise": "^7.0.1" 397 | } 398 | }, 399 | "kind-of": { 400 | "version": "3.2.2", 401 | "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-3.2.2.tgz", 402 | "integrity": "sha1-MeohpzS6ubuw8yRm2JOupR5KPGQ=", 403 | "requires": { 404 | "is-buffer": "^1.1.5" 405 | } 406 | }, 407 | "lazy-cache": { 408 | "version": "1.0.4", 409 | "resolved": "https://registry.npmjs.org/lazy-cache/-/lazy-cache-1.0.4.tgz", 410 | "integrity": "sha1-odePw6UEdMuAhF07O24dpJpEbo4=" 411 | }, 412 | "lodash": { 413 | "version": "4.17.15", 414 | "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz", 415 | "integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A==" 416 | }, 417 | "longest": { 418 | "version": "1.0.1", 419 | "resolved": "https://registry.npmjs.org/longest/-/longest-1.0.1.tgz", 420 | "integrity": "sha1-MKCy2jj3N3DoKUoNIuZiXtd9AJc=" 421 | }, 422 | "media-typer": { 423 | "version": "0.3.0", 424 | "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", 425 | "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=" 426 | }, 427 | "merge-descriptors": { 428 | "version": "1.0.1", 429 | "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz", 430 | "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E=" 431 | }, 432 | "methods": { 433 | "version": "1.1.2", 434 | "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", 435 | "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4=" 436 | }, 437 | "mime": { 438 | "version": "1.6.0", 439 | "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz", 440 | "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==" 441 | }, 442 | "mime-db": { 443 | "version": "1.40.0", 444 | "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.40.0.tgz", 445 | "integrity": "sha512-jYdeOMPy9vnxEqFRRo6ZvTZ8d9oPb+k18PKoYNYUe2stVEBPPwsln/qWzdbmaIvnhZ9v2P+CuecK+fpUfsV2mA==" 446 | }, 447 | "mime-types": { 448 | "version": "2.1.24", 449 | "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.24.tgz", 450 | "integrity": "sha512-WaFHS3MCl5fapm3oLxU4eYDw77IQM2ACcxQ9RIxfaC3ooc6PFuBMGZZsYpvoXS5D5QTWPieo1jjLdAm3TBP3cQ==", 451 | "requires": { 452 | "mime-db": "1.40.0" 453 | } 454 | }, 455 | "ms": { 456 | "version": "2.0.0", 457 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", 458 | "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g=" 459 | }, 460 | "negotiator": { 461 | "version": "0.6.2", 462 | "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz", 463 | "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==" 464 | }, 465 | "object-assign": { 466 | "version": "4.1.1", 467 | "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", 468 | "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM=" 469 | }, 470 | "on-finished": { 471 | "version": "2.3.0", 472 | "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz", 473 | "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=", 474 | "requires": { 475 | "ee-first": "1.1.1" 476 | } 477 | }, 478 | "parseurl": { 479 | "version": "1.3.3", 480 | "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", 481 | "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==" 482 | }, 483 | "path-parse": { 484 | "version": "1.0.6", 485 | "resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.6.tgz", 486 | "integrity": "sha512-GSmOT2EbHrINBf9SR7CDELwlJ8AENk3Qn7OikK4nFYAu3Ote2+JYNVvkpAEQm3/TLNEJFD/xZJjzyxg3KBWOzw==" 487 | }, 488 | "path-to-regexp": { 489 | "version": "0.1.7", 490 | "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz", 491 | "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w=" 492 | }, 493 | "promise": { 494 | "version": "7.3.1", 495 | "resolved": "https://registry.npmjs.org/promise/-/promise-7.3.1.tgz", 496 | "integrity": "sha512-nolQXZ/4L+bP/UGlkfaIujX9BKxGwmQ9OT4mOt5yvy8iK1h3wqTEJCijzGANTCCl9nWjY41juyAn2K3Q1hLLTg==", 497 | "requires": { 498 | "asap": "~2.0.3" 499 | } 500 | }, 501 | "proxy-addr": { 502 | "version": "2.0.5", 503 | "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.5.tgz", 504 | "integrity": "sha512-t/7RxHXPH6cJtP0pRG6smSr9QJidhB+3kXu0KgXnbGYMgzEnUxRQ4/LDdfOwZEMyIh3/xHb8PX3t+lfL9z+YVQ==", 505 | "requires": { 506 | "forwarded": "~0.1.2", 507 | "ipaddr.js": "1.9.0" 508 | } 509 | }, 510 | "pug": { 511 | "version": "2.0.4", 512 | "resolved": "https://registry.npmjs.org/pug/-/pug-2.0.4.tgz", 513 | "integrity": "sha512-XhoaDlvi6NIzL49nu094R2NA6P37ijtgMDuWE+ofekDChvfKnzFal60bhSdiy8y2PBO6fmz3oMEIcfpBVRUdvw==", 514 | "requires": { 515 | "pug-code-gen": "^2.0.2", 516 | "pug-filters": "^3.1.1", 517 | "pug-lexer": "^4.1.0", 518 | "pug-linker": "^3.0.6", 519 | "pug-load": "^2.0.12", 520 | "pug-parser": "^5.0.1", 521 | "pug-runtime": "^2.0.5", 522 | "pug-strip-comments": "^1.0.4" 523 | } 524 | }, 525 | "pug-attrs": { 526 | "version": "2.0.4", 527 | "resolved": "https://registry.npmjs.org/pug-attrs/-/pug-attrs-2.0.4.tgz", 528 | "integrity": "sha512-TaZ4Z2TWUPDJcV3wjU3RtUXMrd3kM4Wzjbe3EWnSsZPsJ3LDI0F3yCnf2/W7PPFF+edUFQ0HgDL1IoxSz5K8EQ==", 529 | "requires": { 530 | "constantinople": "^3.0.1", 531 | "js-stringify": "^1.0.1", 532 | "pug-runtime": "^2.0.5" 533 | } 534 | }, 535 | "pug-code-gen": { 536 | "version": "2.0.2", 537 | "resolved": "https://registry.npmjs.org/pug-code-gen/-/pug-code-gen-2.0.2.tgz", 538 | "integrity": "sha512-kROFWv/AHx/9CRgoGJeRSm+4mLWchbgpRzTEn8XCiwwOy6Vh0gAClS8Vh5TEJ9DBjaP8wCjS3J6HKsEsYdvaCw==", 539 | "requires": { 540 | "constantinople": "^3.1.2", 541 | "doctypes": "^1.1.0", 542 | "js-stringify": "^1.0.1", 543 | "pug-attrs": "^2.0.4", 544 | "pug-error": "^1.3.3", 545 | "pug-runtime": "^2.0.5", 546 | "void-elements": "^2.0.1", 547 | "with": "^5.0.0" 548 | } 549 | }, 550 | "pug-error": { 551 | "version": "1.3.3", 552 | "resolved": "https://registry.npmjs.org/pug-error/-/pug-error-1.3.3.tgz", 553 | "integrity": "sha512-qE3YhESP2mRAWMFJgKdtT5D7ckThRScXRwkfo+Erqga7dyJdY3ZquspprMCj/9sJ2ijm5hXFWQE/A3l4poMWiQ==" 554 | }, 555 | "pug-filters": { 556 | "version": "3.1.1", 557 | "resolved": "https://registry.npmjs.org/pug-filters/-/pug-filters-3.1.1.tgz", 558 | "integrity": "sha512-lFfjNyGEyVWC4BwX0WyvkoWLapI5xHSM3xZJFUhx4JM4XyyRdO8Aucc6pCygnqV2uSgJFaJWW3Ft1wCWSoQkQg==", 559 | "requires": { 560 | "clean-css": "^4.1.11", 561 | "constantinople": "^3.0.1", 562 | "jstransformer": "1.0.0", 563 | "pug-error": "^1.3.3", 564 | "pug-walk": "^1.1.8", 565 | "resolve": "^1.1.6", 566 | "uglify-js": "^2.6.1" 567 | } 568 | }, 569 | "pug-lexer": { 570 | "version": "4.1.0", 571 | "resolved": "https://registry.npmjs.org/pug-lexer/-/pug-lexer-4.1.0.tgz", 572 | "integrity": "sha512-i55yzEBtjm0mlplW4LoANq7k3S8gDdfC6+LThGEvsK4FuobcKfDAwt6V4jKPH9RtiE3a2Akfg5UpafZ1OksaPA==", 573 | "requires": { 574 | "character-parser": "^2.1.1", 575 | "is-expression": "^3.0.0", 576 | "pug-error": "^1.3.3" 577 | } 578 | }, 579 | "pug-linker": { 580 | "version": "3.0.6", 581 | "resolved": "https://registry.npmjs.org/pug-linker/-/pug-linker-3.0.6.tgz", 582 | "integrity": "sha512-bagfuHttfQOpANGy1Y6NJ+0mNb7dD2MswFG2ZKj22s8g0wVsojpRlqveEQHmgXXcfROB2RT6oqbPYr9EN2ZWzg==", 583 | "requires": { 584 | "pug-error": "^1.3.3", 585 | "pug-walk": "^1.1.8" 586 | } 587 | }, 588 | "pug-load": { 589 | "version": "2.0.12", 590 | "resolved": "https://registry.npmjs.org/pug-load/-/pug-load-2.0.12.tgz", 591 | "integrity": "sha512-UqpgGpyyXRYgJs/X60sE6SIf8UBsmcHYKNaOccyVLEuT6OPBIMo6xMPhoJnqtB3Q3BbO4Z3Bjz5qDsUWh4rXsg==", 592 | "requires": { 593 | "object-assign": "^4.1.0", 594 | "pug-walk": "^1.1.8" 595 | } 596 | }, 597 | "pug-parser": { 598 | "version": "5.0.1", 599 | "resolved": "https://registry.npmjs.org/pug-parser/-/pug-parser-5.0.1.tgz", 600 | "integrity": "sha512-nGHqK+w07p5/PsPIyzkTQfzlYfuqoiGjaoqHv1LjOv2ZLXmGX1O+4Vcvps+P4LhxZ3drYSljjq4b+Naid126wA==", 601 | "requires": { 602 | "pug-error": "^1.3.3", 603 | "token-stream": "0.0.1" 604 | } 605 | }, 606 | "pug-runtime": { 607 | "version": "2.0.5", 608 | "resolved": "https://registry.npmjs.org/pug-runtime/-/pug-runtime-2.0.5.tgz", 609 | "integrity": "sha512-P+rXKn9un4fQY77wtpcuFyvFaBww7/91f3jHa154qU26qFAnOe6SW1CbIDcxiG5lLK9HazYrMCCuDvNgDQNptw==" 610 | }, 611 | "pug-strip-comments": { 612 | "version": "1.0.4", 613 | "resolved": "https://registry.npmjs.org/pug-strip-comments/-/pug-strip-comments-1.0.4.tgz", 614 | "integrity": "sha512-i5j/9CS4yFhSxHp5iKPHwigaig/VV9g+FgReLJWWHEHbvKsbqL0oP/K5ubuLco6Wu3Kan5p7u7qk8A4oLLh6vw==", 615 | "requires": { 616 | "pug-error": "^1.3.3" 617 | } 618 | }, 619 | "pug-walk": { 620 | "version": "1.1.8", 621 | "resolved": "https://registry.npmjs.org/pug-walk/-/pug-walk-1.1.8.tgz", 622 | "integrity": "sha512-GMu3M5nUL3fju4/egXwZO0XLi6fW/K3T3VTgFQ14GxNi8btlxgT5qZL//JwZFm/2Fa64J/PNS8AZeys3wiMkVA==" 623 | }, 624 | "qs": { 625 | "version": "6.7.0", 626 | "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz", 627 | "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==" 628 | }, 629 | "range-parser": { 630 | "version": "1.2.1", 631 | "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", 632 | "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==" 633 | }, 634 | "raw-body": { 635 | "version": "2.4.0", 636 | "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz", 637 | "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==", 638 | "requires": { 639 | "bytes": "3.1.0", 640 | "http-errors": "1.7.2", 641 | "iconv-lite": "0.4.24", 642 | "unpipe": "1.0.0" 643 | } 644 | }, 645 | "regenerator-runtime": { 646 | "version": "0.11.1", 647 | "resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.11.1.tgz", 648 | "integrity": "sha512-MguG95oij0fC3QV3URf4V2SDYGJhJnJGqvIIgdECeODCT98wSWDAJ94SSuVpYQUoTcGUIL6L4yNB7j1DFFHSBg==" 649 | }, 650 | "repeat-string": { 651 | "version": "1.6.1", 652 | "resolved": "https://registry.npmjs.org/repeat-string/-/repeat-string-1.6.1.tgz", 653 | "integrity": "sha1-jcrkcOHIirwtYA//Sndihtp15jc=" 654 | }, 655 | "resolve": { 656 | "version": "1.12.0", 657 | "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.12.0.tgz", 658 | "integrity": "sha512-B/dOmuoAik5bKcD6s6nXDCjzUKnaDvdkRyAk6rsmsKLipWj4797iothd7jmmUhWTfinVMU+wc56rYKsit2Qy4w==", 659 | "requires": { 660 | "path-parse": "^1.0.6" 661 | } 662 | }, 663 | "right-align": { 664 | "version": "0.1.3", 665 | "resolved": "https://registry.npmjs.org/right-align/-/right-align-0.1.3.tgz", 666 | "integrity": "sha1-YTObci/mo1FWiSENJOFMlhSGE+8=", 667 | "requires": { 668 | "align-text": "^0.1.1" 669 | } 670 | }, 671 | "safe-buffer": { 672 | "version": "5.1.2", 673 | "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", 674 | "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==" 675 | }, 676 | "safer-buffer": { 677 | "version": "2.1.2", 678 | "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", 679 | "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==" 680 | }, 681 | "send": { 682 | "version": "0.17.1", 683 | "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz", 684 | "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==", 685 | "requires": { 686 | "debug": "2.6.9", 687 | "depd": "~1.1.2", 688 | "destroy": "~1.0.4", 689 | "encodeurl": "~1.0.2", 690 | "escape-html": "~1.0.3", 691 | "etag": "~1.8.1", 692 | "fresh": "0.5.2", 693 | "http-errors": "~1.7.2", 694 | "mime": "1.6.0", 695 | "ms": "2.1.1", 696 | "on-finished": "~2.3.0", 697 | "range-parser": "~1.2.1", 698 | "statuses": "~1.5.0" 699 | }, 700 | "dependencies": { 701 | "ms": { 702 | "version": "2.1.1", 703 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz", 704 | "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg==" 705 | } 706 | } 707 | }, 708 | "serve-static": { 709 | "version": "1.14.1", 710 | "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz", 711 | "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==", 712 | "requires": { 713 | "encodeurl": "~1.0.2", 714 | "escape-html": "~1.0.3", 715 | "parseurl": "~1.3.3", 716 | "send": "0.17.1" 717 | } 718 | }, 719 | "setprototypeof": { 720 | "version": "1.1.1", 721 | "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz", 722 | "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw==" 723 | }, 724 | "source-map": { 725 | "version": "0.6.1", 726 | "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", 727 | "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==" 728 | }, 729 | "statuses": { 730 | "version": "1.5.0", 731 | "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz", 732 | "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=" 733 | }, 734 | "to-fast-properties": { 735 | "version": "1.0.3", 736 | "resolved": "https://registry.npmjs.org/to-fast-properties/-/to-fast-properties-1.0.3.tgz", 737 | "integrity": "sha1-uDVx+k2MJbguIxsG46MFXeTKGkc=" 738 | }, 739 | "toidentifier": { 740 | "version": "1.0.0", 741 | "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz", 742 | "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==" 743 | }, 744 | "token-stream": { 745 | "version": "0.0.1", 746 | "resolved": "https://registry.npmjs.org/token-stream/-/token-stream-0.0.1.tgz", 747 | "integrity": "sha1-zu78cXp2xDFvEm0LnbqlXX598Bo=" 748 | }, 749 | "type-is": { 750 | "version": "1.6.18", 751 | "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz", 752 | "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==", 753 | "requires": { 754 | "media-typer": "0.3.0", 755 | "mime-types": "~2.1.24" 756 | } 757 | }, 758 | "uglify-js": { 759 | "version": "2.8.29", 760 | "resolved": "https://registry.npmjs.org/uglify-js/-/uglify-js-2.8.29.tgz", 761 | "integrity": "sha1-KcVzMUgFe7Th913zW3qcty5qWd0=", 762 | "requires": { 763 | "source-map": "~0.5.1", 764 | "uglify-to-browserify": "~1.0.0", 765 | "yargs": "~3.10.0" 766 | }, 767 | "dependencies": { 768 | "source-map": { 769 | "version": "0.5.7", 770 | "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.5.7.tgz", 771 | "integrity": "sha1-igOdLRAh0i0eoUyA2OpGi6LvP8w=" 772 | } 773 | } 774 | }, 775 | "uglify-to-browserify": { 776 | "version": "1.0.2", 777 | "resolved": "https://registry.npmjs.org/uglify-to-browserify/-/uglify-to-browserify-1.0.2.tgz", 778 | "integrity": "sha1-bgkk1r2mta/jSeOabWMoUKD4grc=", 779 | "optional": true 780 | }, 781 | "unpipe": { 782 | "version": "1.0.0", 783 | "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", 784 | "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=" 785 | }, 786 | "utils-merge": { 787 | "version": "1.0.1", 788 | "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz", 789 | "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM=" 790 | }, 791 | "vary": { 792 | "version": "1.1.2", 793 | "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", 794 | "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=" 795 | }, 796 | "void-elements": { 797 | "version": "2.0.1", 798 | "resolved": "https://registry.npmjs.org/void-elements/-/void-elements-2.0.1.tgz", 799 | "integrity": "sha1-wGavtYK7HLQSjWDqkjkulNXp2+w=" 800 | }, 801 | "window-size": { 802 | "version": "0.1.0", 803 | "resolved": "https://registry.npmjs.org/window-size/-/window-size-0.1.0.tgz", 804 | "integrity": "sha1-VDjNLqk7IC76Ohn+iIeu58lPnJ0=" 805 | }, 806 | "with": { 807 | "version": "5.1.1", 808 | "resolved": "https://registry.npmjs.org/with/-/with-5.1.1.tgz", 809 | "integrity": "sha1-+k2qktrzLE6pTtRTyB8EaGtXXf4=", 810 | "requires": { 811 | "acorn": "^3.1.0", 812 | "acorn-globals": "^3.0.0" 813 | } 814 | }, 815 | "wordwrap": { 816 | "version": "0.0.2", 817 | "resolved": "https://registry.npmjs.org/wordwrap/-/wordwrap-0.0.2.tgz", 818 | "integrity": "sha1-t5Zpu0LstAn4PVg8rVLKF+qhZD8=" 819 | }, 820 | "yargs": { 821 | "version": "3.10.0", 822 | "resolved": "https://registry.npmjs.org/yargs/-/yargs-3.10.0.tgz", 823 | "integrity": "sha1-9+572FfdfB0tOMDnTvvWgdFDH9E=", 824 | "requires": { 825 | "camelcase": "^1.0.2", 826 | "cliui": "^2.1.0", 827 | "decamelize": "^1.0.0", 828 | "window-size": "0.1.0" 829 | } 830 | } 831 | } 832 | } 833 | -------------------------------------------------------------------------------- /_examples/node/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "example", 3 | "private": true, 4 | "dependencies": { 5 | "express": "^4.17.1", 6 | "pug": "^2.0.4" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /_examples/node/public/style.css: -------------------------------------------------------------------------------- 1 | body { 2 | font: 16px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; 3 | padding: 2rem; 4 | margin: 0; 5 | } -------------------------------------------------------------------------------- /_examples/node/server.js: -------------------------------------------------------------------------------- 1 | 2 | const express = require('express') 3 | const app = express() 4 | 5 | app.set('view engine', 'pug') 6 | app.use(express.static('public')) 7 | 8 | // generate fake posts, normally these would come 9 | // from disk, or a database. 10 | const posts = [] 11 | 12 | for (let i = 0; i < 1000; i++) { 13 | posts.push({ 14 | id: i, 15 | title: `Blog Post #${i}`, 16 | teaser: 'Some content here.', 17 | body: `Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse venenatis sem sit amet felis imperdiet porta. Cras eget cursus ante. Suspendisse lobortis aliquam felis feugiat vehicula. Phasellus id mattis lacus. Morbi pellentesque nibh ut turpis finibus porttitor. Vivamus vestibulum dui vel mi sagittis dictum. Ut faucibus commodo magna vel auctor. Donec ut egestas nunc, at ullamcorper mi. Maecenas in fringilla mauris. Mauris non enim eu diam elementum ornare quis at est. Morbi et semper ex, vitae cursus lectus. Nulla auctor, nibh vel tempus ultrices, erat metus auctor ligula, vel tincidunt quam quam ac ligula. Nullam in elit dui. In tellus quam, suscipit eu ultrices sit amet, volutpat eu tellus. Integer pretium et dolor eu faucibus. Proin aliquam blandit rhoncus.`, 18 | }) 19 | } 20 | 21 | app.get('/posts/:id', (req, res) => { 22 | const post = posts[req.params.id] 23 | res.render('post', { post }) 24 | }) 25 | 26 | app.get('/', (req, res) => { 27 | res.render('index', { posts }) 28 | }) 29 | 30 | console.log('Server starting on :3000') 31 | app.listen(3000) -------------------------------------------------------------------------------- /_examples/node/static.json: -------------------------------------------------------------------------------- 1 | { 2 | "command": "NODE_ENV=production node server.js" 3 | } -------------------------------------------------------------------------------- /_examples/node/views/index.pug: -------------------------------------------------------------------------------- 1 | html 2 | head 3 | title Blog Posts 4 | link(rel='stylesheet' href='/style.css') 5 | body 6 | h1 Blog Posts 7 | each post in posts 8 | article 9 | h2= post.title 10 | p= post.teaser 11 | a(href='/posts/' + post.id) View article 12 | -------------------------------------------------------------------------------- /_examples/node/views/post.pug: -------------------------------------------------------------------------------- 1 | html 2 | head 3 | title= post.title 4 | link(rel='stylesheet' href='/style.css') 5 | body 6 | h1= post.title 7 | p= post.body -------------------------------------------------------------------------------- /cmd/staticgen/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "net" 7 | "net/http" 8 | "os" 9 | "os/signal" 10 | "time" 11 | 12 | "github.com/apex/httplog" 13 | "github.com/apex/log" 14 | "github.com/apex/log/handlers/text" 15 | "github.com/pkg/browser" 16 | "github.com/tj/kingpin" 17 | 18 | "github.com/tj/staticgen" 19 | ) 20 | 21 | // version of staticgen. 22 | var version string 23 | 24 | // http transport. 25 | var transport = &http.Transport{ 26 | DialContext: (&net.Dialer{ 27 | Timeout: 30 * time.Second, 28 | }).DialContext, 29 | MaxIdleConns: 100, 30 | MaxIdleConnsPerHost: 100, 31 | IdleConnTimeout: time.Minute, 32 | } 33 | 34 | // http client. 35 | var client = &http.Client{ 36 | Timeout: time.Second * 10, 37 | Transport: transport, 38 | } 39 | 40 | // main. 41 | func main() { 42 | app := kingpin.New("staticgen", "Static website generator") 43 | dir := app.Flag("chdir", "Change working directory").Short('C').Default(".").String() 44 | 45 | log.SetHandler(text.Default) 46 | 47 | app.PreAction(func(_ *kingpin.ParseContext) error { 48 | return os.Chdir(*dir) 49 | }) 50 | 51 | generateCmd(app) 52 | serveCmd(app) 53 | versionCmd(app) 54 | 55 | _, err := app.Parse(os.Args[1:]) 56 | if err != nil { 57 | log.Fatalf("error: %s\n", err) 58 | } 59 | } 60 | 61 | // generateCmd command. 62 | func generateCmd(app *kingpin.Application) { 63 | cmd := app.Command("generate", "Generate static website").Default() 64 | timeout := cmd.Flag("timeout", "Timeout of website generation").Short('t').Default("15m").String() 65 | cmd.Action(func(_ *kingpin.ParseContext) error { 66 | // generator 67 | g := staticgen.Generator{ 68 | HTTPClient: client, 69 | } 70 | 71 | // parse timeout 72 | d, err := time.ParseDuration(*timeout) 73 | if err != nil { 74 | return fmt.Errorf("parsing duration: %w", err) 75 | } 76 | 77 | // timeout 78 | ctx, cancel := context.WithTimeout(context.Background(), d) 79 | defer cancel() 80 | 81 | // trap interrupt and quit 82 | ch := make(chan os.Signal, 1) 83 | signal.Notify(ch, os.Interrupt) 84 | go func() { 85 | log.Infof("Received signal %s — quitting\n", <-ch) 86 | cancel() 87 | }() 88 | 89 | // reporting 90 | events := make(chan staticgen.Event, 1000) 91 | 92 | var r staticgen.Reporter 93 | g.Report(events) 94 | done := r.Report(events) 95 | 96 | // start 97 | err = g.Run(ctx) 98 | if err != nil { 99 | return fmt.Errorf("crawling: %w", err) 100 | } 101 | 102 | close(events) 103 | <-done 104 | return nil 105 | }) 106 | } 107 | 108 | // serveCmd command. 109 | func serveCmd(app *kingpin.Application) { 110 | cmd := app.Command("serve", "Serve the generated website") 111 | addr := cmd.Flag("address", "Bind address").Default("localhost:3000").String() 112 | cmd.Action(func(_ *kingpin.ParseContext) error { 113 | var c staticgen.Config 114 | 115 | err := c.Load("static.json") 116 | if err != nil { 117 | return fmt.Errorf("loading configuration: %w", err) 118 | } 119 | 120 | _ = browser.OpenURL("http://" + *addr) 121 | 122 | server := http.FileServer(http.Dir(c.Dir)) 123 | fmt.Printf("Starting static file server on %s\n", *addr) 124 | return http.ListenAndServe(*addr, httplog.New(server)) 125 | }) 126 | } 127 | 128 | // versionCmd command. 129 | func versionCmd(app *kingpin.Application) { 130 | cmd := app.Command("version", "Output the version.").Hidden() 131 | cmd.Action(func(_ *kingpin.ParseContext) error { 132 | fmt.Printf("Staticgen %s\n", version) 133 | return nil 134 | }) 135 | } 136 | -------------------------------------------------------------------------------- /config.go: -------------------------------------------------------------------------------- 1 | package staticgen 2 | 3 | import ( 4 | "github.com/tj/go-config" 5 | ) 6 | 7 | // Config is the static website generator configuration. 8 | type Config struct { 9 | // URL is the target website to crawl. Defaults to "http://127.0.0.1:3000". 10 | URL string `json:"url"` 11 | 12 | // Dir is the static website output directory. Defaults to "build". 13 | Dir string `json:"dir"` 14 | 15 | // Command is the optional server command executed before crawling. 16 | Command string `json:"command"` 17 | 18 | // Pages is a list of paths added to crawl, typically 19 | // including unlinked pages such as error pages, 20 | // landing pages and so on. 21 | Pages []string `json:"pages"` 22 | 23 | // Concurrency is the number of concurrent pages to crawl. Defaults to 30. 24 | Concurrency int `json:"concurrency"` 25 | 26 | // Allow404 can be enabled to opt-in to pages resulting in a 404, 27 | // which otherwise lead to an error. 28 | Allow404 bool `json:"allow_404"` 29 | } 30 | 31 | // Load configuration from the given path. 32 | func (c *Config) Load(path string) error { 33 | if c.URL == "" { 34 | c.URL = "http://127.0.0.1:3000" 35 | } 36 | 37 | if c.Dir == "" { 38 | c.Dir = "build" 39 | } 40 | 41 | if c.Concurrency == 0 { 42 | c.Concurrency = 30 43 | } 44 | 45 | err := config.Load(path, c) 46 | if err != nil { 47 | return err 48 | } 49 | 50 | return nil 51 | } 52 | -------------------------------------------------------------------------------- /events.go: -------------------------------------------------------------------------------- 1 | package staticgen 2 | 3 | import ( 4 | "time" 5 | ) 6 | 7 | // Event is an event. 8 | type Event interface { 9 | event() 10 | } 11 | 12 | // EventStartingServer . 13 | type EventStartingServer struct { 14 | Command string 15 | URL string 16 | } 17 | 18 | // EventStartedServer . 19 | type EventStartedServer struct { 20 | Command string 21 | URL string 22 | } 23 | 24 | // EventStoppingServer . 25 | type EventStoppingServer struct{} 26 | 27 | // EventStartCrawl . 28 | type EventStartCrawl struct{} 29 | 30 | // EventStopCrawl . 31 | type EventStopCrawl struct{} 32 | 33 | // EventVisitedResource . 34 | type EventVisitedResource struct { 35 | Target 36 | Duration time.Duration 37 | StatusCode int 38 | Filename string 39 | Error error 40 | } 41 | 42 | // event implementation. 43 | func (e EventStartingServer) event() {} 44 | func (e EventStartedServer) event() {} 45 | func (e EventStoppingServer) event() {} 46 | func (e EventStartCrawl) event() {} 47 | func (e EventStopCrawl) event() {} 48 | func (e EventVisitedResource) event() {} 49 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/tj/staticgen 2 | 3 | go 1.12 4 | 5 | require ( 6 | github.com/PuerkitoBio/goquery v1.5.0 7 | github.com/alecthomas/assert v0.0.0-20170929043011-405dbfeb8e38 // indirect 8 | github.com/alecthomas/colour v0.0.0-20160524082231-60882d9e2721 // indirect 9 | github.com/alecthomas/repr v0.0.0-20181024024818-d37bc2a10ba1 // indirect 10 | github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect 11 | github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d // indirect 12 | github.com/apex/httplog v0.0.0-20170124183939-d677fdf2ae1f 13 | github.com/apex/log v1.1.1 14 | github.com/dustin/go-humanize v1.0.0 15 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4 16 | github.com/tj/assert v0.0.0-20190920132354-ee03d75cd160 17 | github.com/tj/go-config v1.3.0 18 | github.com/tj/kingpin v2.5.0+incompatible 19 | golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297 // indirect 20 | ) 21 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/PuerkitoBio/goquery v1.5.0 h1:uGvmFXOA73IKluu/F84Xd1tt/z07GYm8X49XKHP7EJk= 2 | github.com/PuerkitoBio/goquery v1.5.0/go.mod h1:qD2PgZ9lccMbQlc7eEOjaeRlFQON7xY8kdmcsrnKqMg= 3 | github.com/alecthomas/assert v0.0.0-20170929043011-405dbfeb8e38 h1:smF2tmSOzy2Mm+0dGI2AIUHY+w0BUc+4tn40djz7+6U= 4 | github.com/alecthomas/assert v0.0.0-20170929043011-405dbfeb8e38/go.mod h1:r7bzyVFMNntcxPZXK3/+KdruV1H5KSlyVY0gc+NgInI= 5 | github.com/alecthomas/colour v0.0.0-20160524082231-60882d9e2721 h1:JHZL0hZKJ1VENNfmXvHbgYlbUOvpzYzvy2aZU5gXVeo= 6 | github.com/alecthomas/colour v0.0.0-20160524082231-60882d9e2721/go.mod h1:QO9JBoKquHd+jz9nshCh40fOfO+JzsoXy8qTHF68zU0= 7 | github.com/alecthomas/repr v0.0.0-20181024024818-d37bc2a10ba1 h1:GDQdwm/gAcJcLAKQQZGOJ4knlw+7rfEQQcmwTbt4p5E= 8 | github.com/alecthomas/repr v0.0.0-20181024024818-d37bc2a10ba1/go.mod h1:xTS7Pm1pD1mvyM075QCDSRqH6qRLXylzS24ZTpRiSzQ= 9 | github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM= 10 | github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= 11 | github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d h1:UQZhZ2O0vMHr2cI+DC1Mbh0TJxzA3RcLoMsFw+aXw7E= 12 | github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho= 13 | github.com/andybalholm/cascadia v1.0.0 h1:hOCXnnZ5A+3eVDX8pvgl4kofXv2ELss0bKcqRySc45o= 14 | github.com/andybalholm/cascadia v1.0.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y= 15 | github.com/apex/httplog v0.0.0-20170124183939-d677fdf2ae1f h1:FmzTEg6R4xZlBPy+ZpXOf302NkstVo01qFtPX3tjXG0= 16 | github.com/apex/httplog v0.0.0-20170124183939-d677fdf2ae1f/go.mod h1:6XwPOAECoJrkW1w7n3CkBZQdH2nVZ3zSusB37O8PZlE= 17 | github.com/apex/log v1.1.1 h1:BwhRZ0qbjYtTob0I+2M+smavV0kOC8XgcnGZcyL9liA= 18 | github.com/apex/log v1.1.1/go.mod h1:Ls949n1HFtXfbDcjiTTFQqkVUrte0puoIBfO3SVgwOA= 19 | github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy8kCu4PNA+aP7WUV72eXWJeP9/r3/K9aLE= 20 | github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys= 21 | github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= 22 | github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59 h1:WWB576BN5zNSZc/M9d/10pqEx5VHNhaQ/yOVAkmj5Yo= 23 | github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I= 24 | github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= 25 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 26 | github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= 27 | github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= 28 | github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= 29 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 30 | github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= 31 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 32 | github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 33 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 34 | github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= 35 | github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0= 36 | github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= 37 | github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= 38 | github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= 39 | github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= 40 | github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE= 41 | github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= 42 | github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= 43 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 44 | github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= 45 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4 h1:49lOXmGaUpV9Fz3gd7TFZY106KVlPVa5jcYD1gaQf98= 46 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA= 47 | github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= 48 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 49 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 50 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 51 | github.com/rogpeppe/fastuuid v1.1.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ= 52 | github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ= 53 | github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= 54 | github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM= 55 | github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM= 56 | github.com/smartystreets/gunit v1.0.0/go.mod h1:qwPWnhz6pn0NnRBP++URONOVyNkPyr4SauJk4cUOwJs= 57 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 58 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 59 | github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= 60 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 61 | github.com/tj/assert v0.0.0-20171129193455-018094318fb0 h1:Rw8kxzWo1mr6FSaYXjQELRe88y2KdfynXdnK72rdjtA= 62 | github.com/tj/assert v0.0.0-20171129193455-018094318fb0/go.mod h1:mZ9/Rh9oLWpLLDRpvE+3b7gP/C2YyLFYxNmcLnPTMe0= 63 | github.com/tj/assert v0.0.0-20190920132354-ee03d75cd160 h1:NSWpaDaurcAJY7PkL8Xt0PhZE7qpvbZl5ljd8r6U0bI= 64 | github.com/tj/assert v0.0.0-20190920132354-ee03d75cd160/go.mod h1:mZ9/Rh9oLWpLLDRpvE+3b7gP/C2YyLFYxNmcLnPTMe0= 65 | github.com/tj/go-config v1.3.0 h1:tzt1FKVcWklCBTWU7oXzzx5fWMe4+DrO98CW8Fov6fU= 66 | github.com/tj/go-config v1.3.0/go.mod h1:Kvv5shvb2QHrLwN77dhoTPJ0mSzfunmewr75pMwDp/c= 67 | github.com/tj/go-elastic v0.0.0-20171221160941-36157cbbebc2/go.mod h1:WjeM0Oo1eNAjXGDx2yma7uG2XoyRZTq1uv3M/o7imD0= 68 | github.com/tj/go-kinesis v0.0.0-20171128231115-08b17f58cb1b/go.mod h1:/yhzCV0xPfx6jb1bBgRFjl5lytqVqZXEaeqWP8lTEao= 69 | github.com/tj/go-spin v1.1.0/go.mod h1:Mg1mzmePZm4dva8Qz60H2lHwmJ2loum4VIrLgVnKwh4= 70 | github.com/tj/kingpin v2.5.0+incompatible h1:nZWdCABGeebLFX5Ha/rYqxgEQpSXYWh5N9Dx2sGR0Bs= 71 | github.com/tj/kingpin v2.5.0+incompatible/go.mod h1:/babRmtQneL+pp+Yi24s2gukswokaKCR4gfjGbnjHBk= 72 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 73 | golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 74 | golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 75 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 76 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a h1:gOpx8G595UYyvj8UK4+OFyY4rx037g3fmfhe5SasG3U= 77 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 78 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 79 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 80 | golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297 h1:k7pJ2yAPLPgbskkFdhRCsA77k2fySZ1zf2zCjvQCiIM= 81 | golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 82 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 83 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 84 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 85 | golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 86 | golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= 87 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 88 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 89 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 90 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 91 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 92 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 93 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 94 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= 95 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 96 | -------------------------------------------------------------------------------- /internal/crawler/crawler.go: -------------------------------------------------------------------------------- 1 | // Package crawler provides a website crawler. 2 | package crawler 3 | 4 | import ( 5 | "bytes" 6 | "context" 7 | "fmt" 8 | "io" 9 | "io/ioutil" 10 | "net/http" 11 | "net/url" 12 | "path/filepath" 13 | "regexp" 14 | "strings" 15 | "sync" 16 | "time" 17 | 18 | dom "github.com/PuerkitoBio/goquery" 19 | "github.com/tj/staticgen/internal/deduplicator" 20 | ) 21 | 22 | // atImportRe is the regexp used for parsing @import directives. 23 | var atImportRe = regexp.MustCompile(`@import *"([^"]+)"`) 24 | 25 | // A Target is a target URL to crawl, with optional Parent page URL. 26 | type Target struct { 27 | Parent *url.URL 28 | URL *url.URL 29 | } 30 | 31 | // A Resource is representation of the response to a Target request, 32 | // for a particular page or asset. 33 | type Resource struct { 34 | Target 35 | StatusCode int 36 | Duration time.Duration 37 | Body io.ReadCloser 38 | Error error 39 | } 40 | 41 | // A Crawler is in charge of visiting or "crawling" 42 | // all pages and assets of a particular URL. 43 | type Crawler struct { 44 | URL *url.URL 45 | Concurrency int 46 | Allow404 bool 47 | HTTPClient *http.Client 48 | 49 | pending sync.WaitGroup 50 | resources chan Resource 51 | targets chan Target 52 | duplicates deduplicator.Deduplicator 53 | done chan struct{} 54 | } 55 | 56 | // Run starts the crawling process and waits for completion. 57 | func (c *Crawler) Run(ctx context.Context) error { 58 | err := c.Start(ctx) 59 | if err != nil { 60 | return fmt.Errorf("starting: %w", err) 61 | } 62 | 63 | err = c.Wait() 64 | if err != nil { 65 | return fmt.Errorf("waiting: %w", err) 66 | } 67 | 68 | return nil 69 | } 70 | 71 | // Start crawling workers asynchronously. Use Wait() to block until completion. 72 | func (c *Crawler) Start(ctx context.Context) error { 73 | // defaults 74 | if c.Concurrency == 0 { 75 | c.Concurrency = 1 76 | } 77 | 78 | if c.HTTPClient == nil { 79 | c.HTTPClient = http.DefaultClient 80 | } 81 | 82 | // setup 83 | c.resources = make(chan Resource) 84 | c.targets = make(chan Target) 85 | c.done = make(chan struct{}) 86 | 87 | // initial page 88 | c.Queue(c.URL) 89 | 90 | // start workers 91 | ctx, cancel := context.WithCancel(ctx) 92 | for i := 0; i < c.Concurrency; i++ { 93 | go c.crawl(ctx) 94 | } 95 | 96 | // wait for completion 97 | go func() { 98 | c.pending.Wait() 99 | cancel() 100 | close(c.done) 101 | }() 102 | 103 | return nil 104 | } 105 | 106 | // Queue a given URL. This method is non-blocking. 107 | func (c *Crawler) Queue(u *url.URL) { 108 | c.pending.Add(1) 109 | go func() { 110 | c.targets <- Target{URL: u} 111 | }() 112 | } 113 | 114 | // Wait for all pending targets to be crawled. 115 | func (c *Crawler) Wait() error { 116 | <-c.done 117 | return nil 118 | } 119 | 120 | // Resources returns a channel of resources visited by the crawler. 121 | func (c *Crawler) Resources() <-chan Resource { 122 | return c.resources 123 | } 124 | 125 | // crawl all targets, discover additional links, 126 | // and publish resources visited to the Resources() channel. 127 | func (c *Crawler) crawl(ctx context.Context) { 128 | for { 129 | select { 130 | case <-ctx.Done(): 131 | return 132 | case t := <-c.targets: 133 | urls, r, err := c.visit(ctx, t) 134 | 135 | // send resource error 136 | if err != nil { 137 | r.Error = err 138 | select { 139 | case c.resources <- r: 140 | c.pending.Done() 141 | case <-ctx.Done(): 142 | return 143 | } 144 | return 145 | } 146 | 147 | // queue urls 148 | urls = c.duplicates.Filter(urls) 149 | c.pending.Add(len(urls)) 150 | go c.queue(urls, t) 151 | 152 | // send resource 153 | select { 154 | case c.resources <- r: 155 | c.pending.Done() 156 | case <-ctx.Done(): 157 | return 158 | } 159 | } 160 | } 161 | } 162 | 163 | // visit a target and return any additional targets to crawl. 164 | func (c *Crawler) visit(ctx context.Context, t Target) ([]*url.URL, Resource, error) { 165 | start := time.Now() 166 | r := Resource{Target: t} 167 | 168 | // request 169 | req, err := http.NewRequest("GET", t.URL.String(), nil) 170 | if err != nil { 171 | return nil, r, err 172 | } 173 | 174 | req = req.WithContext(ctx) 175 | 176 | // response 177 | res, err := c.HTTPClient.Do(req) 178 | if err != nil { 179 | return nil, r, err 180 | } 181 | 182 | r.StatusCode = res.StatusCode 183 | r.Duration = time.Since(start) 184 | r.Body = res.Body 185 | 186 | // ignore 404s 187 | if res.StatusCode == 404 && c.Allow404 { 188 | return nil, r, nil 189 | } 190 | 191 | // http error 192 | if res.StatusCode >= 300 { 193 | return nil, r, fmt.Errorf("%s response", res.Status) 194 | } 195 | 196 | // file handling 197 | switch filepath.Ext(t.URL.Path) { 198 | case ".css": 199 | defer res.Body.Close() 200 | var buf bytes.Buffer 201 | body := io.TeeReader(res.Body, &buf) 202 | urls, err := visitCSS(body, c.URL, t.URL) 203 | r.Body = ioutil.NopCloser(&buf) 204 | return urls, r, err 205 | case ".html", ".htm", "": 206 | defer res.Body.Close() 207 | var buf bytes.Buffer 208 | body := io.TeeReader(res.Body, &buf) 209 | urls, err := visitHTML(body, c.URL, t.URL) 210 | r.Body = ioutil.NopCloser(&buf) 211 | return urls, r, err 212 | default: 213 | return nil, r, nil 214 | } 215 | } 216 | 217 | // queue the given urls with parent target. 218 | func (c *Crawler) queue(urls []*url.URL, t Target) { 219 | for _, u := range urls { 220 | c.targets <- Target{ 221 | URL: u, 222 | Parent: t.URL, 223 | } 224 | } 225 | } 226 | 227 | // visitCSS returns targets found in a CSS file. 228 | func visitCSS(r io.Reader, root, u *url.URL) ([]*url.URL, error) { 229 | b, err := ioutil.ReadAll(r) 230 | if err != nil { 231 | return nil, err 232 | } 233 | 234 | return parseImports(b, root, u) 235 | } 236 | 237 | // parseImports returns CSS imports. 238 | func parseImports(b []byte, root, u *url.URL) (urls []*url.URL, err error) { 239 | matches := atImportRe.FindAllSubmatch(b, -1) 240 | for _, m := range matches { 241 | s := string(m[1]) 242 | 243 | target, err := url.Parse(s) 244 | if err != nil { 245 | return nil, fmt.Errorf("parsing css import %q: %w", s, err) 246 | } 247 | 248 | resolved := u.ResolveReference(target) 249 | 250 | if follow(root, resolved) { 251 | urls = append(urls, resolved) 252 | } 253 | } 254 | 255 | return 256 | } 257 | 258 | // visitHTML returns targets found in an HTML file. 259 | func visitHTML(r io.Reader, root, u *url.URL) ([]*url.URL, error) { 260 | doc, err := dom.NewDocumentFromReader(r) 261 | if err != nil { 262 | return nil, err 263 | } 264 | 265 | return parseLinks(doc, root, u) 266 | } 267 | 268 | // parseLinks returns resolved target urls in the document. 269 | func parseLinks(doc *dom.Document, root, u *url.URL) (urls []*url.URL, err error) { 270 | doc.Find("a, link").Each(func(i int, s *dom.Selection) { 271 | href := s.AttrOr("href", s.AttrOr("src", "")) 272 | if href == "" { 273 | return 274 | } 275 | 276 | target, err := url.Parse(href) 277 | if err != nil { 278 | return 279 | } 280 | 281 | target.Fragment = "" 282 | target.RawQuery = "" 283 | 284 | resolved := u.ResolveReference(target) 285 | 286 | if follow(root, resolved) { 287 | urls = append(urls, resolved) 288 | } 289 | }) 290 | 291 | return urls, nil 292 | } 293 | 294 | // follow returns true if URL u should be followed. 295 | func follow(root, u *url.URL) bool { 296 | // invalid scheme 297 | if u.Scheme != "https" && u.Scheme != "http" { 298 | return false 299 | } 300 | 301 | // cross domain 302 | if u.Host != root.Host { 303 | return false 304 | } 305 | 306 | // path prefix 307 | if !strings.HasPrefix(u.Path, root.Path) { 308 | return false 309 | } 310 | 311 | return true 312 | } 313 | -------------------------------------------------------------------------------- /internal/crawler/crawler_test.go: -------------------------------------------------------------------------------- 1 | package crawler_test 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "io" 7 | "io/ioutil" 8 | "net/http" 9 | "net/url" 10 | "testing" 11 | "time" 12 | 13 | "github.com/tj/assert" 14 | 15 | "github.com/tj/staticgen/internal/crawler" 16 | ) 17 | 18 | // Test . 19 | func TestCrawler(t *testing.T) { 20 | u, _ := url.Parse("https://apex.sh") 21 | 22 | c := crawler.Crawler{ 23 | URL: u, 24 | Concurrency: 10, 25 | } 26 | 27 | ctx, cancel := context.WithTimeout(context.Background(), time.Second*10) 28 | defer cancel() 29 | 30 | go func() { 31 | for { 32 | select { 33 | case r := <-c.Resources(): 34 | if err := r.Error; err == nil { 35 | fmt.Printf("GET %s -> %s (%s)\n", r.URL, http.StatusText(r.StatusCode), r.Duration.Round(time.Millisecond)) 36 | io.Copy(ioutil.Discard, r.Body) 37 | r.Body.Close() 38 | } else { 39 | fmt.Printf("GET %s -> error %s\n", r.URL, r.Error) 40 | } 41 | case <-ctx.Done(): 42 | fmt.Printf("reporter done\n") 43 | return 44 | } 45 | } 46 | }() 47 | 48 | err := c.Run(ctx) 49 | assert.NoError(t, err, "wait") 50 | 51 | fmt.Printf("done\n") 52 | } 53 | -------------------------------------------------------------------------------- /internal/deduplicator/deduplicator.go: -------------------------------------------------------------------------------- 1 | // Package deduplicator provides URL deduplication. 2 | package deduplicator 3 | 4 | import ( 5 | "net/url" 6 | "strings" 7 | "sync" 8 | ) 9 | 10 | // A Deduplicator manages the deduplication of URLs, allowing you 11 | // to filter out those which you've already visited. The zero value 12 | // is valid. 13 | type Deduplicator struct { 14 | mu sync.Mutex 15 | visited map[string]struct{} 16 | } 17 | 18 | // Filter implementation. 19 | func (d *Deduplicator) Filter(urls []*url.URL) (filtered []*url.URL) { 20 | d.mu.Lock() 21 | defer d.mu.Unlock() 22 | 23 | if d.visited == nil { 24 | d.visited = make(map[string]struct{}) 25 | } 26 | 27 | for _, u := range urls { 28 | u = normalize(u) 29 | 30 | _, ok := d.visited[u.String()] 31 | if ok { 32 | continue 33 | } 34 | 35 | d.visited[u.String()] = struct{}{} 36 | filtered = append(filtered, u) 37 | } 38 | 39 | return 40 | } 41 | 42 | // normalize returns a URL with its path normalized, 43 | // stripping the tailing "/" if present, treating 44 | // "/blog/" and "/blog" as the same. 45 | func normalize(u *url.URL) *url.URL { 46 | if strings.HasSuffix(u.Path, "/") { 47 | c := *u 48 | c.Path = strings.TrimRight(c.Path, "/") 49 | return &c 50 | } 51 | return u 52 | } 53 | -------------------------------------------------------------------------------- /internal/deduplicator/deduplicator_test.go: -------------------------------------------------------------------------------- 1 | package deduplicator_test 2 | 3 | import ( 4 | "net/url" 5 | "testing" 6 | 7 | "github.com/tj/assert" 8 | 9 | "github.com/tj/staticgen/internal/deduplicator" 10 | ) 11 | 12 | // Test filtering. 13 | func TestDeduplicator_Filter(t *testing.T) { 14 | var d deduplicator.Deduplicator 15 | 16 | apex, _ := url.Parse("https://apex.sh") 17 | netflix, _ := url.Parse("https://netflix.com") 18 | youtube, _ := url.Parse("https://youtube.com") 19 | facebook, _ := url.Parse("https://facebook.com") 20 | 21 | urls := []*url.URL{apex, netflix, youtube} 22 | 23 | urls = d.Filter(urls) 24 | assert.Len(t, urls, 3) 25 | 26 | urls = d.Filter(urls) 27 | assert.Len(t, urls, 0) 28 | 29 | urls = append(urls, facebook) 30 | urls = d.Filter(urls) 31 | assert.Len(t, urls, 1) 32 | } 33 | -------------------------------------------------------------------------------- /reporter.go: -------------------------------------------------------------------------------- 1 | package staticgen 2 | 3 | import ( 4 | "net/http" 5 | "time" 6 | 7 | "github.com/apex/log" 8 | "github.com/dustin/go-humanize" 9 | ) 10 | 11 | // A Reporter outputs a human-friendly report of events. 12 | type Reporter struct { 13 | count int64 14 | start time.Time 15 | } 16 | 17 | // Report on the given event channel. Returns a channel which is 18 | // closed when the event channel is closed, and all reporting has 19 | // been completed. 20 | func (r *Reporter) Report(ch <-chan Event) <-chan struct{} { 21 | done := make(chan struct{}) 22 | 23 | go func() { 24 | defer close(done) 25 | for e := range ch { 26 | switch e := e.(type) { 27 | case EventStartCrawl: 28 | r.start = time.Now() 29 | case EventStartingServer: 30 | log.Infof("Starting server with command %q", e.Command) 31 | log.Infof("Waiting for server to listen on %s", e.URL) 32 | case EventStartedServer: 33 | log.Infof("Server is listening for requests") 34 | case EventStoppingServer: 35 | log.Infof("Stopping server, sending SIGTERM") 36 | case EventVisitedResource: 37 | r.count++ 38 | if e.Error == nil { 39 | log.Infof("GET %s —— %s —— %s (%s)", e.URL, e.Filename, http.StatusText(e.StatusCode), e.Duration.Round(time.Millisecond)) 40 | } else { 41 | log.Errorf("GET %s —— %s (error: %s)", e.URL, http.StatusText(e.StatusCode), e.Error) 42 | } 43 | case EventStopCrawl: 44 | log.Infof("Completed %s resources in %s", humanize.Comma(r.count), time.Since(r.start).Round(time.Millisecond)) 45 | } 46 | } 47 | }() 48 | 49 | return done 50 | } 51 | -------------------------------------------------------------------------------- /staticgen.go: -------------------------------------------------------------------------------- 1 | // Package staticgen provides static website generation from a live server. 2 | package staticgen 3 | 4 | import ( 5 | "bytes" 6 | "context" 7 | "fmt" 8 | "io" 9 | "net/http" 10 | "net/url" 11 | "os" 12 | "os/exec" 13 | "path" 14 | "path/filepath" 15 | "strings" 16 | "sync" 17 | "syscall" 18 | "time" 19 | 20 | "github.com/apex/log" 21 | 22 | "github.com/tj/staticgen/internal/crawler" 23 | ) 24 | 25 | // Target is a target URL. 26 | type Target struct { 27 | Parent *url.URL 28 | URL *url.URL 29 | } 30 | 31 | // Generator is a static website generator. 32 | type Generator struct { 33 | // Config used for crawling and producing the static website. 34 | Config 35 | 36 | // HTTPClient ... 37 | HTTPClient *http.Client 38 | 39 | // crawler 40 | crawler crawler.Crawler 41 | wg sync.WaitGroup 42 | 43 | // server command 44 | cmd *exec.Cmd 45 | out bytes.Buffer 46 | 47 | // events 48 | events chan<- Event 49 | } 50 | 51 | // Run starts the configured server command, starts to perform crawling, 52 | // and waits for completion before shutting down the configured server. 53 | func (g *Generator) Run(ctx context.Context) error { 54 | err := g.Start(ctx) 55 | if err != nil { 56 | return fmt.Errorf("starting: %w", err) 57 | } 58 | 59 | err = g.Wait() 60 | if err != nil { 61 | return fmt.Errorf("waiting: %w", err) 62 | } 63 | 64 | defer g.emit(EventStopCrawl{}) 65 | if err := g.stopCommand(ctx); err != nil { 66 | return fmt.Errorf("stopping: %w", err) 67 | } 68 | 69 | return nil 70 | } 71 | 72 | // Start loads configuration from ./static.json, starts the 73 | // configured server, and begins the crawling process. 74 | func (g *Generator) Start(ctx context.Context) error { 75 | // load configuration 76 | err := g.Config.Load("static.json") 77 | if err != nil { 78 | return fmt.Errorf("loading configuration: %w", err) 79 | } 80 | 81 | // remove output dir 82 | err = os.RemoveAll(g.Dir) 83 | if err != nil { 84 | return fmt.Errorf("removing output directory: %w", err) 85 | } 86 | 87 | // create output dir 88 | err = os.MkdirAll(g.Dir, 0755) 89 | if err != nil { 90 | return fmt.Errorf("creating output directory: %w", err) 91 | } 92 | 93 | // start command 94 | err = g.startCommand(ctx) 95 | if err != nil { 96 | return fmt.Errorf("starting command: %w", err) 97 | } 98 | 99 | // parse url 100 | u, err := url.Parse(g.URL) 101 | if err != nil { 102 | return fmt.Errorf("parsing url: %w", err) 103 | } 104 | 105 | // setup crawler 106 | g.crawler = crawler.Crawler{ 107 | URL: u, 108 | Allow404: g.Allow404, 109 | Concurrency: g.Concurrency, 110 | HTTPClient: g.HTTPClient, 111 | } 112 | 113 | // start workers 114 | ctx, cancel := context.WithCancel(ctx) 115 | for i := 0; i < g.Concurrency; i++ { 116 | g.wg.Add(1) 117 | go func() { 118 | g.saveLoop(ctx) 119 | g.wg.Done() 120 | }() 121 | } 122 | 123 | // start crawling 124 | g.emit(EventStartCrawl{}) 125 | err = g.crawler.Start(ctx) 126 | if err != nil { 127 | cancel() 128 | return fmt.Errorf("starting crawler: %w", err) 129 | } 130 | 131 | // queue pages 132 | g.queuePages(u) 133 | 134 | // wait for crawling to complete, 135 | // then exit the save loops. 136 | go func() { 137 | g.crawler.Wait() 138 | cancel() 139 | }() 140 | 141 | return nil 142 | } 143 | 144 | // Wait for crawling to complete. 145 | func (g *Generator) Wait() error { 146 | g.wg.Wait() 147 | return nil 148 | } 149 | 150 | // Report registers a channel for reporting on events. 151 | func (g *Generator) Report(ch chan<- Event) { 152 | g.events = ch 153 | } 154 | 155 | // queuePages queues the configured Pages relative to the given url. 156 | func (g *Generator) queuePages(u *url.URL) { 157 | for _, p := range g.Pages { 158 | t := *u 159 | t.Path = p 160 | g.crawler.Queue(&t) 161 | } 162 | } 163 | 164 | // saveLoop saves the crawler resources to disk. 165 | func (g *Generator) saveLoop(ctx context.Context) { 166 | for { 167 | select { 168 | case <-ctx.Done(): 169 | return 170 | case r := <-g.crawler.Resources(): 171 | err := g.save(r) 172 | if err != nil { 173 | log.WithError(err).WithField("url", r.URL.String()).Error("error saving") 174 | } 175 | } 176 | } 177 | } 178 | 179 | // save a resource to disk. 180 | func (g *Generator) save(r crawler.Resource) error { 181 | // determine local path for the file 182 | ext := filepath.Ext(r.URL.Path) 183 | dir, file := path.Split(r.URL.Path) 184 | dst := filepath.Join(g.Dir, dir, file) 185 | 186 | // save html into directories with index.html 187 | if file != "index.html" && (ext == ".html" || ext == "") { 188 | file = strings.Replace(file, ".html", "", 1) 189 | dst = filepath.Join(g.Dir, dir, file, "index.html") 190 | } 191 | 192 | g.emit(EventVisitedResource{ 193 | Target: Target(r.Target), 194 | Duration: r.Duration, 195 | StatusCode: r.StatusCode, 196 | Error: r.Error, 197 | Filename: dst, 198 | }) 199 | 200 | // request error, don't copy to disk 201 | if r.Error != nil { 202 | return nil 203 | } 204 | 205 | defer r.Body.Close() 206 | return writeFile(r.Body, dst) 207 | } 208 | 209 | // startCommand starts the configured server command. 210 | func (g *Generator) startCommand(ctx context.Context) error { 211 | if g.Command == "" { 212 | return nil 213 | } 214 | 215 | g.emit(EventStartingServer{ 216 | Command: g.Command, 217 | URL: g.URL, 218 | }) 219 | 220 | // start 221 | g.cmd = exec.Command("sh", "-c", g.Command) 222 | g.cmd.Env = append(os.Environ(), "STATICGEN=1") 223 | g.cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} 224 | g.cmd.Stdout = &g.out 225 | g.cmd.Stderr = &g.out 226 | err := g.cmd.Start() 227 | if err != nil { 228 | return err 229 | } 230 | 231 | // timeout 232 | ctx, cancel := context.WithTimeout(ctx, time.Second*15) 233 | defer cancel() 234 | 235 | // wait 236 | err = waitForListen(ctx, g.URL) 237 | if err != nil { 238 | return fmt.Errorf("waiting for app to start: %w", err) 239 | } 240 | 241 | g.emit(EventStartedServer{ 242 | Command: g.Command, 243 | URL: g.URL, 244 | }) 245 | 246 | return nil 247 | } 248 | 249 | // stopCommand stops the configured command. 250 | func (g *Generator) stopCommand(ctx context.Context) error { 251 | if g.Command == "" { 252 | return nil 253 | } 254 | 255 | g.emit(EventStoppingServer{}) 256 | 257 | pgid, err := syscall.Getpgid(g.cmd.Process.Pid) 258 | if err != nil { 259 | return fmt.Errorf("getting process group id: %w", err) 260 | } 261 | 262 | err = syscall.Kill(-pgid, syscall.SIGTERM) 263 | if err != nil { 264 | return fmt.Errorf("kill: %w", err) 265 | } 266 | 267 | _ = g.cmd.Wait() 268 | return nil 269 | } 270 | 271 | // emit an event. 272 | func (g *Generator) emit(e Event) { 273 | if g.events != nil { 274 | g.events <- e 275 | } 276 | } 277 | 278 | // writeFile writes to filename and ensures the directory exists. 279 | func writeFile(r io.Reader, filename string) error { 280 | dir := filepath.Dir(filename) 281 | 282 | err := os.MkdirAll(dir, 0755) 283 | if err != nil { 284 | return err 285 | } 286 | 287 | f, err := os.Create(filename) 288 | if err != nil { 289 | return err 290 | } 291 | 292 | _, err = io.Copy(f, r) 293 | if err != nil { 294 | return err 295 | } 296 | 297 | return f.Close() 298 | } 299 | 300 | // waitForListen blocks until `addr` is listening. 301 | func waitForListen(ctx context.Context, url string) error { 302 | for { 303 | select { 304 | case <-ctx.Done(): 305 | return ctx.Err() 306 | case <-time.After(time.Second): 307 | if isListening(url) { 308 | return nil 309 | } 310 | } 311 | } 312 | } 313 | 314 | // isListening returns true if there's an HTTP server listening on url. 315 | func isListening(url string) bool { 316 | _, err := http.Head(url) 317 | return err == nil 318 | } 319 | --------------------------------------------------------------------------------