├── requirements.txt ├── shot.png ├── shots.yml ├── .github └── workflows │ └── shots.yml └── README.md /requirements.txt: -------------------------------------------------------------------------------- 1 | shot-scraper 2 | -------------------------------------------------------------------------------- /shot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pietroppeter/cattura-il-post-it/main/shot.png -------------------------------------------------------------------------------- /shots.yml: -------------------------------------------------------------------------------- 1 | - url: https://www.ilpost.it/ 2 | output: shot.png 3 | width: 1000 4 | height: 2000 5 | -------------------------------------------------------------------------------- /.github/workflows/shots.yml: -------------------------------------------------------------------------------- 1 | name: Take screenshots 2 | 3 | on: 4 | push: 5 | workflow_dispatch: 6 | 7 | permissions: 8 | contents: write 9 | 10 | jobs: 11 | shot-scraper: 12 | runs-on: ubuntu-latest 13 | if: ${{ github.repository != 'simonw/shot-scraper-template' }} 14 | steps: 15 | - uses: actions/checkout@v3 16 | - name: Set up Python 3.10 17 | uses: actions/setup-python@v3 18 | with: 19 | python-version: "3.10" 20 | - uses: actions/cache@v3 21 | name: Configure pip caching 22 | with: 23 | path: ~/.cache/pip 24 | key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }} 25 | restore-keys: | 26 | ${{ runner.os }}-pip- 27 | - name: Cache Playwright browsers 28 | uses: actions/cache@v3 29 | with: 30 | path: ~/.cache/ms-playwright/ 31 | key: ${{ runner.os }}-browsers 32 | - name: Install dependencies 33 | run: | 34 | pip install -r requirements.txt 35 | - name: Install Playwright dependencies 36 | run: | 37 | shot-scraper install 38 | - uses: actions/github-script@v6 39 | name: Create shots.yml if missing on first run 40 | with: 41 | script: | 42 | const fs = require('fs'); 43 | if (!fs.existsSync('shots.yml')) { 44 | const desc = context.payload.repository.description; 45 | let line = ''; 46 | if (desc && (desc.startsWith('http://') || desc.startsWith('https://'))) { 47 | line = `- url: ${desc}` + '\n output: shot.png\n height: 800'; 48 | } else { 49 | line = '# - url: https://www.example.com/\n# output: shot.png\n# height: 800'; 50 | } 51 | fs.writeFileSync('shots.yml', line + '\n'); 52 | } 53 | - name: Take shots 54 | run: | 55 | shot-scraper multi shots.yml 56 | - name: Commit and push 57 | run: |- 58 | git config user.name "Automated" 59 | git config user.email "actions@users.noreply.github.com" 60 | git add -A 61 | timestamp=$(date -u) 62 | git commit -m "${timestamp}" || exit 0 63 | git pull --rebase 64 | git push 65 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # shot-scraper-template 2 | 3 | Quickly create a new GitHub repository that takes automated screenshots of a web page using [shot-scraper](https://github.com/simonw/shot-scraper). 4 | 5 | Read more about how this works in [Instantly create a GitHub repository to take screenshots of a web page](https://simonwillison.net/2022/Mar/14/shot-scraper-template/). 6 | 7 | [simonw/simonwillison-net-shot](https://github.com/simonw/simonwillison-net-shot) is an example repository created using this template. 8 | 9 | ## How to get started 10 | 11 | Visit https://github.com/simonw/shot-scraper-template/generate 12 | 13 | Screenshot of the interface for creating a new repository with this template, showing the URL pasted into the description field 14 | 15 | Pick a name for your new repository, and paste **the URL** of the page you would like to take screenshots of into the **description field** (including the `http://` or `https://`). 16 | 17 | Then click **Create repository from template**. 18 | 19 | Your new repository will be created, and a script will run which will do the following: 20 | 21 | - Add a `shots.yml` file to your repository containing the URL of the page you requested 22 | - Take a screenshot of that URL and add that to your repository as a file called `shot.png` 23 | 24 | You can then edit that `shots.yml` file to customize your screenshot, or add more URLs - see below. 25 | 26 | If the script does not run when the repository is first created you may need to **Enable Actions** first: 27 | 28 | - Click the "Actions" tab 29 | - Clice "Enable Actions" 30 | - Run the "Take screenshots" workflow as described below 31 | 32 | ## Re-taking the screenshot 33 | 34 | To re-take the screenshot: 35 | 36 | - Click "Actions" 37 | - Select the "Take screenshots" workflow 38 | - Click the "Run workflow" menu item 39 | - Click the green "Run workflow" button 40 | 41 | image 42 | 43 | The repository will keep a history of every previous version of each screenshot, which is useful for keeping track of visual changes to a page. 44 | 45 | ## Configuring the screenshots 46 | 47 | The initial `shots.yml` file in your repository should look like this: 48 | 49 | ```yaml 50 | - url: https://simonwillison.net/ 51 | output: shot.png 52 | height: 800 53 | ``` 54 | 55 | To change the name of the file that the screenshot is saved to, change `output: shot.png` to a different file name. 56 | 57 | To take a full height image of the page, remove the `height: 800` line. 58 | 59 | To add additional screenshots, add them to the YAML file like this: 60 | 61 | ```yaml 62 | - url: https://simonwillison.net/ 63 | output: shot.png 64 | height: 800 65 | - url: https://www.example.com/ 66 | output: example.png 67 | height: 800 68 | ``` 69 | Other useful options include: 70 | 71 | - `wait: 3000` to add a 3 second delay before taking the shot (in case some things need more time to load) 72 | - `javascript: ...` to execute custom JavaScript before taking the shot - to activate menus or hide elements or similar 73 | - `quality: 80` to save a smaller, lower quality JPEG image 74 | 75 | This example takes a shot of the LA Times homepage after hiding ads and the terms of service prompt: 76 | 77 | ```yaml 78 | - url: https://www.latimes.com/ 79 | output: latimes.jpg 80 | width: 1600 81 | height: 1600 82 | quality: 80 83 | wait: 2000 84 | javascript: | 85 | document.querySelectorAll( 86 | '[data-ad-rendered],#ensNotifyBanner' 87 | ).forEach(el => el.style.display = 'none') 88 | ``` 89 | Further options are described in the [shot-scraper README file](https://github.com/simonw/shot-scraper#taking-multiple-screenshots). 90 | 91 | ## Installing fonts for more languages 92 | 93 | The default Ubuntu used by GitHub Actions does not include fonts for many languages, including Chinese and Japanese. 94 | 95 | You can modify the `shots.yml` file to install extra fonts by adding this section, between the "Cache Playwright browsers" and "Install dependencies" steps: 96 | 97 | ```yaml 98 | - name: Cache Playwright browsers 99 | uses: actions/cache@v2 100 | with: 101 | path: ~/.cache/ms-playwright/ 102 | key: ${{ runner.os }}-browsers 103 | - name: Install extra fonts 104 | run: | 105 | sudo apt-get install fonts-arphic-ukai fonts-arphic-uming fonts-ipafont-mincho fonts-ipafont-gothic fonts-unfonts-core 106 | - name: Install dependencies 107 | run: | 108 | pip install -r requirements.txt 109 | ``` 110 | --------------------------------------------------------------------------------