├── .gitignore ├── README.md ├── bin ├── patreon-downloader.js ├── process-mp3.sh ├── process-wav.sh ├── transfer-files-to-media-centre.sh └── update-kodi-audio-library.sh ├── files └── .gitkeep ├── js ├── cookies.js ├── files.js └── lazyLoadImages.js ├── package-lock.json ├── package.json ├── persistence └── .gitkeep └── run.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | *.iml 3 | 4 | node_modules 5 | persistence 6 | 7 | cookies.json 8 | files 9 | notes.txt 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Patreon downloader 2 | 3 | ### Summary 4 | 5 | A collection of scripts to enable automatic downloading, tagging and organising of audio files posted by Patreon creators. The script uses [Puppeteer](https://developers.google.com/web/tools/puppeteer/) to control an instance of the Chromium browser. A Patreon account is required, and if you want to access subscription-only content, you will need to pay for it. This script purely replaces a manual process with an automated one. It can be run on a cron job or by some other scheduling mechanism. 6 | 7 | The script supports downloading of mp3 and wav files. mp3 files will also have a comment (COMM) id3v2 tag added, the content of which will be the text content of the blog post, followed by any tags set by the creator. The post's associated image will also be embeded as album art in the file. As wav does not officially support such metadata, no such tagging/embedding will occur. Instead a [cue sheet](https://en.wikipedia.org/wiki/Cue_sheet_(computing)) will be written and transferred along with the audio and artwork files. 8 | 9 | The script keeps track of any files downloaded in a text file and will not re-download them. Once files have been downloaded they will be moved to a final destination directory (for me, a hard disk attached to my media centre), either on the same machine using `mv` or elsewhere using `scp`. 10 | 11 | Once the file is downloaded, an update of the audio library in a Kodi installation is triggered (given the correct configuration). 12 | 13 | ### Dependencies 14 | * [nodeJS](https://nodejs.org) 15 | * [cURL](https://curl.haxx.se/) 16 | * [id3 A.K.A. id3mtag](https://squell.github.io/id3/) (do not be confused by [id3](http://manpages.ubuntu.com/manpages/cosmic/en/man1/id3.1.html) on Ubuntu systems, this is not the same tool) 17 | * [eyeD3](https://eyed3.readthedocs.io/en/latest/) (requires Python 2.7+ and libmagic) 18 | 19 | To install id3mtag on macOS, refer to my blog post at [https://endofhome.github.io/2019/01/11/compiling_id3mtag_for_macos.html](https://endofhome.github.io/2019/01/11/compiling_id3mtag_for_macos.html) 20 | 21 | eyeD3 can be installed via `pip`: ```pip install eyeD3``` 22 | It requires a working installation of `libmagic`. On macOS this can easily be installed via Homebrew: ```brew install libmagic```, or on Ubuntu, ```apt-get install libmagic```. 23 | 24 | ### Configuration 25 | 26 | As well as the required environment variables (listed below), the script requires a logged-in session cookie. I dumped my cookies to a JSON file using a browser extension. The only cookie required (at the time of writing) is `session_id` for the domain `.patreon.com`. The script requires the session cookie to be stored as a JSON object inside a JSON array in a file named `cookies.json` at root level. 27 | 28 | Environment variables: 29 | 30 | | Variable | Description| 31 | |----------|------------| 32 | | PATREON_ARTIST | The Patreon artist/creator that you are supporting, as per the URL of their Patreon page | 33 | | PATREON_ARTIST_NAME | The name of the artist/creator | 34 | | DESTINATION_DIRECTORY | The directory you want to save your files in. For me, a directory in a disk attached to my media centre. This directory doesn't have to be on the same machine as you are running the script on, if you have an ssh key stored the script will be able to use `scp` to transfer the files | 35 | | DESTINATION_MACHINE_NAME | Name of the machine where the destination directory exists | 36 | | DESTINATION_MACHINE_USERNAME | Username for the account you will use to `scp` into the destination machine | 37 | | DESTINATION_MACHINE_HOST | Host for the destination machine (used for `scp`) | 38 | | KODI_USERNAME | Username for Kodi installation | 39 | | KODI_PASSWORD | Password for Kodi installation | 40 | | KODI_PORT | Port for Kodi installation | 41 | 42 | ### How do I run it? 43 | 44 | `npm i` to install the nodeJS dependencies. 45 | 46 | Install other dependencies mentioned above, and ensure the `cookies.json` file and necessary environment variables are present. 47 | 48 | `./run.sh` will start the process. 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | -------------------------------------------------------------------------------- /bin/patreon-downloader.js: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | const puppeteer = require('puppeteer'); 4 | const loadCookies = require("../js/cookies"); 5 | const getLazyLoadedImages = require("../js/lazyLoadImages"); 6 | const downloadTagAndOrganiseFiles = require("../js/files"); 7 | 8 | async function main() { 9 | const requiredConfig = [ 10 | "PATREON_ARTIST", 11 | "PATREON_ARTIST_NAME", 12 | "DESTINATION_DIRECTORY", 13 | "DESTINATION_MACHINE_NAME", 14 | "DESTINATION_MACHINE_USERNAME", 15 | "DESTINATION_MACHINE_HOST", 16 | "KODI_USERNAME", 17 | "KODI_PASSWORD", 18 | "KODI_PORT" 19 | ]; 20 | const config = validateConfig(requiredConfig); 21 | 22 | const browser = await puppeteer.launch({headless: true}); 23 | const page = await browser.newPage(); 24 | const waitTime = 0.5; 25 | 26 | await loadCookies(page); 27 | console.log("Starting Chromium and looking for new songs to download..."); 28 | await page.goto(`https://www.patreon.com/${config.PATREON_ARTIST}/posts`, {timeout: 100000, waitUntil: 'networkidle0'}); 29 | await getLazyLoadedImages(page, waitTime); 30 | 31 | const songs = await page.evaluate(() => { 32 | // all executed in the context of the browser 33 | const numberOfElementsToTest = 30; 34 | 35 | return [...document.querySelectorAll('[data-tag="post-card"]')] 36 | .slice(0, numberOfElementsToTest) 37 | .filter(postCard => hasFile(postCard)) 38 | .map(postCard => { 39 | const fileDownloadLink = postCard.querySelector('a[data-tag="post-file-download"]'); 40 | console.log("processing " + fileDownloadLink.textContent); 41 | const secondAudioPlayerElement = postCard.querySelectorAll('[aria-label="Audio Player"]')[1]; 42 | const style = secondAudioPlayerElement.firstChild.firstChild.getAttribute('style'); 43 | const songNotes = postContent(postCard); 44 | const date = dateFor(postCard); 45 | const year = date.split('-')[0]; 46 | const artworkUrl = style.slice(23) 47 | .split('') 48 | .reverse() 49 | .slice(3) 50 | .reverse() 51 | .join(''); 52 | 53 | return { 54 | title: postCard.querySelector('[data-tag="post-title"]').textContent, 55 | file: fileDownloadLink.textContent, 56 | url: fileDownloadLink.href, 57 | notes: validatedNotes(songNotes), 58 | tags: tagsOrEmptyString(postCard.querySelector('[data-tag="post-tags"]')), 59 | artwork: artworkUrl, 60 | publishedDate: date, 61 | year: year 62 | } 63 | }); 64 | 65 | function hasFile(postCard) { 66 | return postCard.querySelector('a[data-tag="post-file-download"]') !== null 67 | } 68 | 69 | function postContent(postCard) { 70 | const postContentCollapsed = postCard.querySelector('[data-tag="post-content-collapse"]'); 71 | const postContent = postContentCollapsed ? collapsedContent(postContentCollapsed) : uncollapsedContent(postCard); 72 | return postContent.map(element => element.textContent); 73 | 74 | function collapsedContent(postContent) { 75 | const containsParagraphs = postContent.querySelectorAll('p'); 76 | 77 | return containsParagraphs.length !== 0 ? [...containsParagraphs] : arrayOfPostContentCollapsed(postContent) 78 | } 79 | 80 | function uncollapsedContent(postCard) { 81 | const result = []; 82 | result.push(postCard.querySelector('[data-tag="post-content"]')); 83 | return result; 84 | } 85 | 86 | function arrayOfPostContentCollapsed(postContent) { 87 | const notesHtmlWithBrTags = postContent.firstChild.firstChild.firstChild.firstChild.innerHTML; 88 | const splitByBrTag = notesHtmlWithBrTags.split('

'); 89 | const result = []; 90 | [1, 2].forEach(() => { // duplicating the notes so they match the output of finding by

tag 91 | splitByBrTag.forEach(note => 92 | result.push({textContent: note}) 93 | ); 94 | }); 95 | return result; 96 | } 97 | } 98 | 99 | function dateFor(postCard) { 100 | const months = { 101 | Jan: '01', 102 | Feb: '02', 103 | Mar: '03', 104 | Apr: '04', 105 | May: '05', 106 | Jun: '06', 107 | Jul: '07', 108 | Aug: '08', 109 | Sep: '09', 110 | Oct: '10', 111 | Nov: '11', 112 | Dec: '12' 113 | }; 114 | 115 | const textContent = postCard.querySelector('[data-tag="post-published-at"]').textContent; 116 | try { 117 | const timeStripped = textContent.replace(/ at .*/g, ''); 118 | let split = timeStripped.split(" "); 119 | split[0] = months[split[0]]; 120 | 121 | if (commaIsPresentIn(timeStripped)) { 122 | const year = split.splice(2, 1); 123 | split.unshift(year[0]); 124 | split[2] = padWithZero(split[2].replace(',', '')); 125 | return split.join('-') 126 | } else { 127 | split.unshift(new Date().getFullYear().toString()); 128 | split[2] = padWithZero(split[2]); 129 | return split.join('-') 130 | } 131 | 132 | } catch (e) { 133 | throw Error(`couldn't parse date: ${textContent}`) 134 | } 135 | 136 | function commaIsPresentIn(string) { 137 | return string.includes(',') 138 | } 139 | 140 | function padWithZero(string) { 141 | if (string.length === 1) { 142 | return '0' + string; 143 | } else { 144 | return string; 145 | } 146 | } 147 | } 148 | 149 | function validatedNotes(songNotes) { 150 | if (songNotes.length % 2 === 0) { 151 | const firstHalf = []; 152 | const secondHalf = []; 153 | songNotes.forEach((songNote, index) => { 154 | if (index < (songNotes.length / 2)) { 155 | firstHalf.push(songNote); 156 | } else { 157 | secondHalf.push(songNote); 158 | } 159 | }); 160 | 161 | if (firstHalf.length !== secondHalf.length) { 162 | throw Error(`Halves are of unequal length. First half: ${firstHalf.length}, second half: ${secondHalf.length}`) 163 | } 164 | 165 | firstHalf.forEach((note, index) => { 166 | if (note !== secondHalf[index]) { 167 | throw Error(`Notes are not duplicates. First note: ${note}, second note: ${secondHalf[index]}`) 168 | } 169 | }); 170 | 171 | return firstHalf; 172 | } else { 173 | return songNotes; 174 | } 175 | } 176 | 177 | function tagsOrEmptyString(postTags) { 178 | if (postTags !== null) { 179 | return [...postTags.firstChild.querySelector('div').children] 180 | .map(tagLink => tagLink.textContent) 181 | .join(","); 182 | } else { 183 | return "" 184 | } 185 | } 186 | 187 | }); 188 | 189 | // print songs JSON 190 | // console.log(JSON.stringify(songs)); 191 | 192 | browser.close(); 193 | try { 194 | downloadTagAndOrganiseFiles(songs, config.PATREON_ARTIST_NAME); 195 | } catch (e) { 196 | console.log(e); 197 | } 198 | 199 | 200 | function validateConfig(requiredConfig) { 201 | const missingConfig = []; 202 | const validatedConfig = {}; 203 | 204 | requiredConfig.forEach(configItem => { 205 | const result = process.env[configItem]; 206 | 207 | if (result === undefined) { 208 | missingConfig.push(configItem) 209 | } else { 210 | validatedConfig[configItem] = result; 211 | } 212 | }); 213 | 214 | if (missingConfig.length > 0) { 215 | throw Error(`\n\nMissing environment variables: \n${missingConfig.map(item => { 216 | return item + "\n" 217 | }).join("") 218 | }`); 219 | } else { 220 | return validatedConfig; 221 | } 222 | } 223 | } 224 | 225 | main(); 226 | -------------------------------------------------------------------------------- /bin/process-mp3.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o nounset 4 | set -o pipefail 5 | 6 | URL=$1 7 | FILE_PATH=$2 8 | COMMENT=$3 9 | YEAR=$4 10 | TITLE=$5 11 | ARTWORK_URL=$6 12 | ARTIST_NAME=$7 13 | 14 | __dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 15 | 16 | echo "downloading ${TITLE}" 17 | curl -sL ${URL} --output "${FILE_PATH}" 18 | 19 | echo "tagging ${TITLE}" 20 | id3 -2 -c "$COMMENT" -y ${YEAR} "${FILE_PATH}" >/dev/null 2>&1 21 | 22 | echo "checking tags" 23 | METADATA=$(echo "$(id3 "${FILE_PATH}")") 24 | 25 | echo "checking for ALBUM tag" 26 | ALBUM_TAG=$(echo "$METADATA" | grep Album | cut -d ':' -f2 | tr -d '[:space:]') 27 | if [[ -z "${ALBUM_TAG}" ]]; then 28 | echo "ALBUM tag is missing - setting to 'Patreon Posts'" 29 | id3 -2 -l "Patreon Posts" "${FILE_PATH}" >/dev/null 2>&1 30 | else 31 | echo "ALBUM tag is present" 32 | fi 33 | 34 | echo "checking for ARTIST tag" 35 | ARTIST_TAG=$(echo "$METADATA" | grep Artist | cut -d ':' -f2 | tr -d '[:space:]') 36 | if [[ -z "${ARTIST_TAG}" ]]; then 37 | echo "ARTIST tag is missing - setting to '${ARTIST_NAME}'" 38 | id3 -2 -a "${ARTIST_NAME}" "${FILE_PATH}" >/dev/null 2>&1 39 | else 40 | echo "ARTIST tag is present" 41 | fi 42 | 43 | echo "checking for TITLE tag" 44 | TITLE_TAG=$(echo "$METADATA" | grep Title | cut -d ':' -f2 | tr -d '[:space:]') 45 | if [[ -z "${TITLE_TAG}" ]]; then 46 | echo "TITLE tag is missing - setting to '${TITLE}'" 47 | id3 -2 -t "${TITLE}" "${FILE_PATH}" >/dev/null 2>&1 48 | else 49 | echo "TITLE tag is present" 50 | fi 51 | 52 | echo "downloading artwork for ${TITLE}" 53 | TITLE_NO_WHITESPACE=$(echo ${TITLE} | sed -e 's/ /_/g') 54 | ARTWORK_FILE_PATH="$(dirname "${BASH_SOURCE[0]}")/../files/cover-${TITLE_NO_WHITESPACE}.jpg" 55 | curl -sL ${ARTWORK_URL} --output "${ARTWORK_FILE_PATH}" 56 | 57 | echo "embedding artwork" 58 | /usr/local/bin/eyeD3 --add-image "${ARTWORK_FILE_PATH}:FRONT_COVER" "${FILE_PATH}" >/dev/null 2>&1 59 | 60 | rm "${ARTWORK_FILE_PATH}" 61 | 62 | source "${__dir}/transfer-files-to-media-centre.sh" "${TITLE}" "${FILE_PATH}" 63 | -------------------------------------------------------------------------------- /bin/process-wav.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o nounset 4 | set -o pipefail 5 | 6 | URL=$1 7 | FILE_PATH=$2 8 | COMMENT=$3 9 | YEAR=$4 10 | TITLE=$5 11 | ARTWORK_URL=$6 12 | ARTIST_NAME=$7 13 | 14 | __dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 15 | 16 | echo "downloading ${TITLE}" 17 | curl -sL ${URL} --output "${FILE_PATH}" 18 | 19 | echo "writing cue sheet for ${TITLE}" 20 | CUE_FILE_PATH="$(dirname "$FILE_PATH")/$(basename "$FILE_PATH" .wav).cue" 21 | touch "$CUE_FILE_PATH" 22 | cat < "$CUE_FILE_PATH" 23 | PERFORMER "${ARTIST_NAME}" 24 | TITLE "Patreon Posts" 25 | REM DATE "${YEAR}" 26 | FILE "$(basename "$FILE_PATH")" WAVE 27 | TRACK 01 AUDIO 28 | TITLE "${TITLE}" 29 | PERFORMER "${ARTIST_NAME}" 30 | REM COMMENT "${COMMENT}" 31 | INDEX 01 00:00:00 32 | EOF 33 | 34 | echo "downloading artwork for ${TITLE}" 35 | echo "will not embed artwork, saving for later processing." 36 | ARTWORK_FILE_PATH="$(dirname "$FILE_PATH")/$(basename "$FILE_PATH" .wav).jpg" 37 | curl -sL ${ARTWORK_URL} --output "${ARTWORK_FILE_PATH}" 38 | 39 | source "${__dir}/transfer-files-to-media-centre.sh" "${TITLE}" "${FILE_PATH}" "$CUE_FILE_PATH" "$ARTWORK_FILE_PATH" 40 | -------------------------------------------------------------------------------- /bin/transfer-files-to-media-centre.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o errexit 4 | set -o pipefail 5 | 6 | TITLE=$1 7 | FILE_PATH=$2 8 | CUE_FILE_PATH=${3:-} 9 | ARTWORK_FILE_PATH=${4:-} 10 | 11 | __dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 12 | 13 | echo "transferring ${TITLE} to media centre" 14 | 15 | if [[ $OSTYPE = "darwin"* ]]; then 16 | DESTINATION=$(echo ${DESTINATION_DIRECTORY} | sed -e 's/ /\\ /g') 17 | else 18 | DESTINATION=${DESTINATION_DIRECTORY} 19 | fi 20 | 21 | if [[ $(uname -a | awk '{print $2}') = ${DESTINATION_MACHINE_NAME} ]]; then 22 | echo "destination directory located on this machine, using mv" 23 | mv "${FILE_PATH}" "${DESTINATION}" 24 | if [[ -n "$CUE_FILE_PATH" ]]; then 25 | mv "${CUE_FILE_PATH}" "${DESTINATION}" 26 | fi 27 | if [[ -n "$ARTWORK_FILE_PATH" ]]; then 28 | mv "${ARTWORK_FILE_PATH}" "${DESTINATION}" 29 | fi 30 | else 31 | echo "destination directory located on a remote machine, using scp" 32 | scp "${FILE_PATH}" ${DESTINATION_MACHINE_USERNAME}@${DESTINATION_MACHINE_HOST}:"${DESTINATION}" 33 | rm "${FILE_PATH}" 34 | if [[ -n "$CUE_FILE_PATH" ]]; then 35 | scp "${CUE_FILE_PATH}" ${DESTINATION_MACHINE_USERNAME}@${DESTINATION_MACHINE_HOST}:"${DESTINATION}" 36 | rm "${CUE_FILE_PATH}" 37 | fi 38 | if [[ -n "$ARTWORK_FILE_PATH" ]]; then 39 | scp "${ARTWORK_FILE_PATH}" ${DESTINATION_MACHINE_USERNAME}@${DESTINATION_MACHINE_HOST}:"${DESTINATION}" 40 | rm "${ARTWORK_FILE_PATH}" 41 | fi 42 | fi 43 | 44 | "${__dir}/update-kodi-audio-library.sh" 45 | 46 | echo "$(date +"%Y-%m-%d %H:%M:%S"),$TITLE" >> "$(dirname "${BASH_SOURCE[0]}")/../persistence/downloaded.txt" 47 | echo "processing ${TITLE} completed" 48 | -------------------------------------------------------------------------------- /bin/update-kodi-audio-library.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o nounset 4 | set -o errexit 5 | set -o pipefail 6 | 7 | echo "updating Kodi audio library" 8 | curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "id": "patreon-downloader"}' -H 'content-type: application/json;' http://${KODI_USERNAME}:${KODI_PASSWORD}@${DESTINATION_MACHINE_HOST}:${KODI_PORT}/jsonrpc >/dev/null 2>&1 9 | -------------------------------------------------------------------------------- /files/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/endofhome/patreon-downloader/ac4caef6b91b57debdbd09b211f725ae2e408cda/files/.gitkeep -------------------------------------------------------------------------------- /js/cookies.js: -------------------------------------------------------------------------------- 1 | module.exports = async function loadCookies(page) { 2 | const cookiesArr = require(`../cookies`); 3 | if (cookiesArr.length !== 0) { 4 | await cookiesArr.forEach(cookie => { 5 | page.setCookie(cookie) 6 | }); 7 | console.log('Session has been loaded in the browser'); 8 | } 9 | }; -------------------------------------------------------------------------------- /js/files.js: -------------------------------------------------------------------------------- 1 | const fs = require('fs'); 2 | const path = require('path'); 3 | const exec = require('child_process').exec; 4 | 5 | module.exports = function downloadTagAndOrganiseFiles(songs, artistName) { 6 | let downloaded; 7 | try { 8 | downloaded = fs.readFileSync("persistence/downloaded.txt", "utf8") 9 | } catch (e) { 10 | downloaded = ""; 11 | } 12 | 13 | const downloadedTitles = downloaded 14 | .split("\n") 15 | .filter(Boolean) 16 | .map(line => { 17 | return line.split(",")[1]; 18 | }); 19 | 20 | songs.map(song => { 21 | if (downloadedTitles.includes(song.title)) { 22 | console.log(`Already downloaded ${song.title}`); 23 | } else { 24 | if (song.file.endsWith('.mp3')) { 25 | processMp3(song); 26 | } else if (song.file.endsWith('.wav')) { 27 | processWav(song); 28 | } else { 29 | console.log("This filename has no file extension. Assuming it's an mp3"); 30 | processMp3(song) 31 | } 32 | } 33 | }); 34 | 35 | function escapeDoubleQuotes(s) { 36 | return s.replace(/"/g, '\\"'); 37 | } 38 | 39 | function reformatNotesForShellScript(song) { 40 | return escapeDoubleQuotes(song.notes.join('\n')); 41 | } 42 | 43 | function processMp3(song) { 44 | const filename = song.file.replace(".mp3", "") + ".mp3"; 45 | const mp3Command = `"${path.resolve(__dirname, '../bin/process-mp3.sh')}" "${song.url}" "${path.resolve(__dirname, '../files/' + filename)}" "${reformatNotesForShellScript(song)}\n\n${escapeDoubleQuotes(song.tags)}" ${song.year} "${escapeDoubleQuotes(song.title)}" "${song.artwork}" "${artistName}"`; 46 | download(mp3Command); 47 | } 48 | 49 | function processWav(song) { 50 | const wavCommand = `"${path.resolve(__dirname, '../bin/process-wav.sh')}" "${song.url}" "${path.resolve(__dirname, '../files/' + song.file)}" "${reformatNotesForShellScript(song)}\n\n${escapeDoubleQuotes(song.tags)}" ${song.year} "${escapeDoubleQuotes(song.title)}" "${song.artwork}" "${artistName}"`; 51 | download(wavCommand); 52 | } 53 | 54 | function download(command) { 55 | exec(command, (error, stdout, stderr) => { 56 | console.log(`${stdout}`); 57 | console.log(`${stderr}`); 58 | if (error !== null) { 59 | console.log(`exec error: ${error}`); 60 | } 61 | }); 62 | } 63 | }; 64 | -------------------------------------------------------------------------------- /js/lazyLoadImages.js: -------------------------------------------------------------------------------- 1 | module.exports = async function getLazyLoadedImages(page, waitTime) { 2 | // Scroll one viewport at a time, pausing to let content load 3 | // based on https://gist.github.com/schnerd/b550b7c05d4a57d8374082aaae714881 by schnerd 4 | function wait(ms) { 5 | return new Promise(resolve => setTimeout(() => resolve(), ms)); 6 | } 7 | 8 | const bodyHandle = await page.$('body'); 9 | const {height} = await bodyHandle.boundingBox(); 10 | await bodyHandle.dispose(); 11 | 12 | const viewportHeight = page.viewport().height; 13 | let viewportIncr = 0; 14 | // (height * 8) is a bit sloppy - I need to spend some time to find a better solution. 15 | while (viewportIncr + viewportHeight < height * 8) { 16 | await page.evaluate(_viewportHeight => { 17 | // scroll by half of the viewport at a time 18 | window.scrollBy(0, _viewportHeight / 2); 19 | 20 | const buttons = [...document.querySelectorAll('button')]; 21 | const loadMoreButton = buttons.find(button => { 22 | const loadMoreDiv = [...button.querySelectorAll('div')].find(div => { 23 | return div.textContent === 'Load more' 24 | }); 25 | if (loadMoreDiv !== undefined) { 26 | return button; 27 | } 28 | }); 29 | if (loadMoreButton !== undefined) { 30 | loadMoreButton.click(); 31 | } 32 | }, viewportHeight); 33 | await wait(waitTime * 1000); 34 | viewportIncr = viewportIncr + viewportHeight; 35 | } 36 | }; 37 | -------------------------------------------------------------------------------- /package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "patreon-downloader", 3 | "version": "0.0.1", 4 | "lockfileVersion": 1, 5 | "requires": true, 6 | "dependencies": { 7 | "agent-base": { 8 | "version": "4.2.1", 9 | "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-4.2.1.tgz", 10 | "integrity": "sha512-JVwXMr9nHYTUXsBFKUqhJwvlcYU/blreOEUkhNR2eXZIvwd+c+o5V4MgDPKWnMS/56awN3TRzIP+KoPn+roQtg==", 11 | "requires": { 12 | "es6-promisify": "^5.0.0" 13 | } 14 | }, 15 | "async-limiter": { 16 | "version": "1.0.0", 17 | "resolved": "https://registry.npmjs.org/async-limiter/-/async-limiter-1.0.0.tgz", 18 | "integrity": "sha512-jp/uFnooOiO+L211eZOoSyzpOITMXx1rBITauYykG3BRYPu8h0UcxsPNB04RR5vo4Tyz3+ay17tR6JVf9qzYWg==" 19 | }, 20 | "balanced-match": { 21 | "version": "1.0.0", 22 | "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.0.tgz", 23 | "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c=" 24 | }, 25 | "brace-expansion": { 26 | "version": "1.1.11", 27 | "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", 28 | "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==", 29 | "requires": { 30 | "balanced-match": "^1.0.0", 31 | "concat-map": "0.0.1" 32 | } 33 | }, 34 | "buffer-from": { 35 | "version": "1.1.1", 36 | "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.1.tgz", 37 | "integrity": "sha512-MQcXEUbCKtEo7bhqEs6560Hyd4XaovZlO/k9V3hjVUF/zwW7KBVdSK4gIt/bzwS9MbR5qob+F5jusZsb0YQK2A==" 38 | }, 39 | "concat-map": { 40 | "version": "0.0.1", 41 | "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", 42 | "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s=" 43 | }, 44 | "concat-stream": { 45 | "version": "1.6.2", 46 | "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.6.2.tgz", 47 | "integrity": "sha512-27HBghJxjiZtIk3Ycvn/4kbJk/1uZuJFfuPEns6LaEvpvG1f0hTea8lilrouyo9mVc2GWdcEZ8OLoGmSADlrCw==", 48 | "requires": { 49 | "buffer-from": "^1.0.0", 50 | "inherits": "^2.0.3", 51 | "readable-stream": "^2.2.2", 52 | "typedarray": "^0.0.6" 53 | } 54 | }, 55 | "core-util-is": { 56 | "version": "1.0.2", 57 | "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz", 58 | "integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac=" 59 | }, 60 | "debug": { 61 | "version": "4.1.0", 62 | "resolved": "https://registry.npmjs.org/debug/-/debug-4.1.0.tgz", 63 | "integrity": "sha512-heNPJUJIqC+xB6ayLAMHaIrmN9HKa7aQO8MGqKpvCA+uJYVcvR6l5kgdrhRuwPFHU7P5/A1w0BjByPHwpfTDKg==", 64 | "requires": { 65 | "ms": "^2.1.1" 66 | } 67 | }, 68 | "es6-promise": { 69 | "version": "4.2.5", 70 | "resolved": "https://registry.npmjs.org/es6-promise/-/es6-promise-4.2.5.tgz", 71 | "integrity": "sha512-n6wvpdE43VFtJq+lUDYDBFUwV8TZbuGXLV4D6wKafg13ldznKsyEvatubnmUe31zcvelSzOHF+XbaT+Bl9ObDg==" 72 | }, 73 | "es6-promisify": { 74 | "version": "5.0.0", 75 | "resolved": "http://registry.npmjs.org/es6-promisify/-/es6-promisify-5.0.0.tgz", 76 | "integrity": "sha1-UQnWLz5W6pZ8S2NQWu8IKRyKUgM=", 77 | "requires": { 78 | "es6-promise": "^4.0.3" 79 | } 80 | }, 81 | "extract-zip": { 82 | "version": "1.6.7", 83 | "resolved": "https://registry.npmjs.org/extract-zip/-/extract-zip-1.6.7.tgz", 84 | "integrity": "sha1-qEC0uK9kAyZMjbV/Txp0Mz74H+k=", 85 | "requires": { 86 | "concat-stream": "1.6.2", 87 | "debug": "2.6.9", 88 | "mkdirp": "0.5.1", 89 | "yauzl": "2.4.1" 90 | }, 91 | "dependencies": { 92 | "debug": { 93 | "version": "2.6.9", 94 | "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", 95 | "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==", 96 | "requires": { 97 | "ms": "2.0.0" 98 | } 99 | }, 100 | "ms": { 101 | "version": "2.0.0", 102 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", 103 | "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g=" 104 | } 105 | } 106 | }, 107 | "fd-slicer": { 108 | "version": "1.0.1", 109 | "resolved": "https://registry.npmjs.org/fd-slicer/-/fd-slicer-1.0.1.tgz", 110 | "integrity": "sha1-i1vL2ewyfFBBv5qwI/1nUPEXfmU=", 111 | "requires": { 112 | "pend": "~1.2.0" 113 | } 114 | }, 115 | "fs.realpath": { 116 | "version": "1.0.0", 117 | "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", 118 | "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8=" 119 | }, 120 | "glob": { 121 | "version": "7.1.3", 122 | "resolved": "https://registry.npmjs.org/glob/-/glob-7.1.3.tgz", 123 | "integrity": "sha512-vcfuiIxogLV4DlGBHIUOwI0IbrJ8HWPc4MU7HzviGeNho/UJDfi6B5p3sHeWIQ0KGIU0Jpxi5ZHxemQfLkkAwQ==", 124 | "requires": { 125 | "fs.realpath": "^1.0.0", 126 | "inflight": "^1.0.4", 127 | "inherits": "2", 128 | "minimatch": "^3.0.4", 129 | "once": "^1.3.0", 130 | "path-is-absolute": "^1.0.0" 131 | } 132 | }, 133 | "https-proxy-agent": { 134 | "version": "2.2.1", 135 | "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-2.2.1.tgz", 136 | "integrity": "sha512-HPCTS1LW51bcyMYbxUIOO4HEOlQ1/1qRaFWcyxvwaqUS9TY88aoEuHUY33kuAh1YhVVaDQhLZsnPd+XNARWZlQ==", 137 | "requires": { 138 | "agent-base": "^4.1.0", 139 | "debug": "^3.1.0" 140 | }, 141 | "dependencies": { 142 | "debug": { 143 | "version": "3.2.6", 144 | "resolved": "https://registry.npmjs.org/debug/-/debug-3.2.6.tgz", 145 | "integrity": "sha512-mel+jf7nrtEl5Pn1Qx46zARXKDpBbvzezse7p7LqINmdoIk8PYP5SySaxEmYv6TZ0JyEKA1hsCId6DIhgITtWQ==", 146 | "requires": { 147 | "ms": "^2.1.1" 148 | } 149 | } 150 | } 151 | }, 152 | "inflight": { 153 | "version": "1.0.6", 154 | "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", 155 | "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=", 156 | "requires": { 157 | "once": "^1.3.0", 158 | "wrappy": "1" 159 | } 160 | }, 161 | "inherits": { 162 | "version": "2.0.3", 163 | "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", 164 | "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" 165 | }, 166 | "isarray": { 167 | "version": "1.0.0", 168 | "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz", 169 | "integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE=" 170 | }, 171 | "mime": { 172 | "version": "2.4.0", 173 | "resolved": "https://registry.npmjs.org/mime/-/mime-2.4.0.tgz", 174 | "integrity": "sha512-ikBcWwyqXQSHKtciCcctu9YfPbFYZ4+gbHEmE0Q8jzcTYQg5dHCr3g2wwAZjPoJfQVXZq6KXAjpXOTf5/cjT7w==" 175 | }, 176 | "minimatch": { 177 | "version": "3.0.4", 178 | "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz", 179 | "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==", 180 | "requires": { 181 | "brace-expansion": "^1.1.7" 182 | } 183 | }, 184 | "minimist": { 185 | "version": "0.0.8", 186 | "resolved": "http://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz", 187 | "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=" 188 | }, 189 | "mkdirp": { 190 | "version": "0.5.1", 191 | "resolved": "http://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz", 192 | "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=", 193 | "requires": { 194 | "minimist": "0.0.8" 195 | } 196 | }, 197 | "ms": { 198 | "version": "2.1.1", 199 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz", 200 | "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg==" 201 | }, 202 | "once": { 203 | "version": "1.4.0", 204 | "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", 205 | "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=", 206 | "requires": { 207 | "wrappy": "1" 208 | } 209 | }, 210 | "path-is-absolute": { 211 | "version": "1.0.1", 212 | "resolved": "http://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", 213 | "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18=" 214 | }, 215 | "pend": { 216 | "version": "1.2.0", 217 | "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz", 218 | "integrity": "sha1-elfrVQpng/kRUzH89GY9XI4AelA=" 219 | }, 220 | "process-nextick-args": { 221 | "version": "2.0.0", 222 | "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.0.tgz", 223 | "integrity": "sha512-MtEC1TqN0EU5nephaJ4rAtThHtC86dNN9qCuEhtshvpVBkAW5ZO7BASN9REnF9eoXGcRub+pFuKEpOHE+HbEMw==" 224 | }, 225 | "progress": { 226 | "version": "2.0.3", 227 | "resolved": "https://registry.npmjs.org/progress/-/progress-2.0.3.tgz", 228 | "integrity": "sha512-7PiHtLll5LdnKIMw100I+8xJXR5gW2QwWYkT6iJva0bXitZKa/XMrSbdmg3r2Xnaidz9Qumd0VPaMrZlF9V9sA==" 229 | }, 230 | "proxy-from-env": { 231 | "version": "1.0.0", 232 | "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.0.0.tgz", 233 | "integrity": "sha1-M8UDmPcOp+uW0h97gXYwpVeRx+4=" 234 | }, 235 | "puppeteer": { 236 | "version": "1.11.0", 237 | "resolved": "https://registry.npmjs.org/puppeteer/-/puppeteer-1.11.0.tgz", 238 | "integrity": "sha512-iG4iMOHixc2EpzqRV+pv7o3GgmU2dNYEMkvKwSaQO/vMZURakwSOn/EYJ6OIRFYOque1qorzIBvrytPIQB3YzQ==", 239 | "requires": { 240 | "debug": "^4.1.0", 241 | "extract-zip": "^1.6.6", 242 | "https-proxy-agent": "^2.2.1", 243 | "mime": "^2.0.3", 244 | "progress": "^2.0.1", 245 | "proxy-from-env": "^1.0.0", 246 | "rimraf": "^2.6.1", 247 | "ws": "^6.1.0" 248 | } 249 | }, 250 | "readable-stream": { 251 | "version": "2.3.6", 252 | "resolved": "http://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz", 253 | "integrity": "sha512-tQtKA9WIAhBF3+VLAseyMqZeBjW0AHJoxOtYqSUZNJxauErmLbVm2FW1y+J/YA9dUrAC39ITejlZWhVIwawkKw==", 254 | "requires": { 255 | "core-util-is": "~1.0.0", 256 | "inherits": "~2.0.3", 257 | "isarray": "~1.0.0", 258 | "process-nextick-args": "~2.0.0", 259 | "safe-buffer": "~5.1.1", 260 | "string_decoder": "~1.1.1", 261 | "util-deprecate": "~1.0.1" 262 | } 263 | }, 264 | "rimraf": { 265 | "version": "2.6.2", 266 | "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-2.6.2.tgz", 267 | "integrity": "sha512-lreewLK/BlghmxtfH36YYVg1i8IAce4TI7oao75I1g245+6BctqTVQiBP3YUJ9C6DQOXJmkYR9X9fCLtCOJc5w==", 268 | "requires": { 269 | "glob": "^7.0.5" 270 | } 271 | }, 272 | "safe-buffer": { 273 | "version": "5.1.2", 274 | "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", 275 | "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==" 276 | }, 277 | "string_decoder": { 278 | "version": "1.1.1", 279 | "resolved": "http://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", 280 | "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", 281 | "requires": { 282 | "safe-buffer": "~5.1.0" 283 | } 284 | }, 285 | "typedarray": { 286 | "version": "0.0.6", 287 | "resolved": "https://registry.npmjs.org/typedarray/-/typedarray-0.0.6.tgz", 288 | "integrity": "sha1-hnrHTjhkGHsdPUfZlqeOxciDB3c=" 289 | }, 290 | "util-deprecate": { 291 | "version": "1.0.2", 292 | "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", 293 | "integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8=" 294 | }, 295 | "wrappy": { 296 | "version": "1.0.2", 297 | "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", 298 | "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=" 299 | }, 300 | "ws": { 301 | "version": "6.1.2", 302 | "resolved": "https://registry.npmjs.org/ws/-/ws-6.1.2.tgz", 303 | "integrity": "sha512-rfUqzvz0WxmSXtJpPMX2EeASXabOrSMk1ruMOV3JBTBjo4ac2lDjGGsbQSyxj8Odhw5fBib8ZKEjDNvgouNKYw==", 304 | "requires": { 305 | "async-limiter": "~1.0.0" 306 | } 307 | }, 308 | "yauzl": { 309 | "version": "2.4.1", 310 | "resolved": "https://registry.npmjs.org/yauzl/-/yauzl-2.4.1.tgz", 311 | "integrity": "sha1-lSj0QtqxsihOWLQ3m7GU4i4MQAU=", 312 | "requires": { 313 | "fd-slicer": "~1.0.1" 314 | } 315 | } 316 | } 317 | } 318 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "patreon-downloader", 3 | "version": "0.0.1", 4 | "description": "Automatically download new files from Patreon", 5 | "main": "patreon-downloader.js", 6 | "scripts": { 7 | "test": "echo \"Error: no test specified\" && exit 1" 8 | }, 9 | "author": "", 10 | "license": "ISC", 11 | "dependencies": { 12 | "puppeteer": "^1.11.0" 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /persistence/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/endofhome/patreon-downloader/ac4caef6b91b57debdbd09b211f725ae2e408cda/persistence/.gitkeep -------------------------------------------------------------------------------- /run.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | DIR=${pwd} 4 | cd "$(dirname "$0")" && ./bin/patreon-downloader.js 5 | cd ${DIR} 6 | --------------------------------------------------------------------------------