├── LICENSE ├── README.md └── ytdlrc /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Brian Hardisty 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ytdlrc 2 | 3 | [![Codacy grade](https://img.shields.io/codacy/grade/fb1e5b8b80374c15b05877cf2bebfd31.svg)](https://www.codacy.com/app/bardisty/ytdlrc?utm_source=github.com&utm_medium=referral&utm_content=bardisty/ytdlrc&utm_campaign=Badge_Grade) 4 | ![GitHub file size in bytes](https://img.shields.io/github/size/bardisty/ytdlrc/ytdlrc.svg) 5 | [![GitHub](https://img.shields.io/github/license/bardisty/ytdlrc.svg?color=blue)](https://github.com/bardisty/ytdlrc/blob/master/LICENSE) 6 | 7 | `ytdlrc` is a simple shell script wrapper for `youtube-dl` and `rclone`. It 8 | downloads videos and metadata via `youtube-dl` and moves each file on 9 | completion to an `rclone` remote, e.g. Google Drive. 10 | 11 |
12 | 13 | Table of Contents 14 | 15 | 31 |
32 | 33 | ## About 34 | 35 | * Designed to be executed via cron job or manually. 36 | 37 | * Ideal for use on VPS's with small disk space as it moves files on 38 | completion rather than copying them. In testing I downloaded / uploaded 39 | ~500GB of videos on a VPS with a 15GB SSD. 40 | 41 | * This can easily be changed to keep the files if you have the disk 42 | space; see the `rclone_command` variable to do so. 43 | 44 | * Includes metadata (xattrs, \*.info.json, \*.description, \*.jpg). 45 | 46 | * Reads a file named `snatch.list` to know which URL's / channels / 47 | playlists you want to download and monitor for new videos. 48 | 49 | * Lines beginning with `#` are ignored. 50 | 51 | * Completed video ID's are saved in a file named `archive.list`; this 52 | prevents `youtube-dl` from re-downloading videos that have already been 53 | processed and moved to the rclone remote. 54 | 55 | * By default, it creates / uses the following structure: 56 | 57 | * `~/ytdlrc/stage` - download directory 58 | * `~/ytdlrc/snatch.list` - list of usernames or URL's to monitor / download 59 | * `~/ytdlrc/archive.list` - list of completed video ID's 60 | 61 | These paths can be changed by modifying the `ytdl_*` variables. 62 | 63 | * In the download directory, subfolders are created for each line in the 64 | `snatch.list`. The name of the subfolders depends on whether the 65 | processed line is a playlist or a channel. This results in the following 66 | structure: 67 | 68 | * `~/ytdlrc/stage/Some_Channel_Name/{downloaded files}` 69 | * `~/ytdlrc/stage/Another_Channel_Name/{downloaded files}` 70 | * `~/ytdlrc/stage/Some_Playlist_Name/{downloaded files}` 71 | * `~/ytdlrc/stage/Another_Playlist_Name/{downloaded files}` 72 | 73 | Upon download completion a file is moved to the rclone remote, creating 74 | a structure such as: 75 | 76 | * `remote:archive/youtube/Some_Channel_Name/{downloaded files}` 77 | * `remote:archive/youtube/Another_Channel_Name/{downloaded files}` 78 | * `remote:archive/youtube/Some_Playlist_Name/{downloaded files}` 79 | * `remote:archive/youtube/Another_Playlist_Name/{downloaded files}` 80 | 81 | The path before the channel / playlist names (`remote:archive/youtube`) 82 | is set via the `rclone_destination` variable. 83 | 84 | * Downloaded files are saved with the following output template: 85 | 86 | `"%(uploader)s.%(upload_date)s.%(title)s.%(resolution)s.%(id)s.%(ext)s"` 87 | 88 | This results in filenames such as: 89 | 90 | * `Channel_Name.20170307.Video_Title.1920x1080.J---aiyznGQ.mp4` 91 | * `Channel_Name.20170307.Video_Title.1920x1080.J---aiyznGQ.info.json` 92 | * `Channel_Name.20170307.Video_Title.1920x1080.J---aiyznGQ.description` 93 | * `Channel_Name.20170307.Video_Title.1920x1080.J---aiyznGQ.jpg` 94 | 95 | See the `ytdl_output_template` variable if you wish to use a different 96 | output template. 97 | 98 | ## Example Output 99 | 100 | These examples are what you see with `debug=true`. 101 | 102 | ### Example: Initial execution 103 | 104 | ```text 105 | [YTDLRC] Lock file doesn't exist. Attempting to create /tmp/ytdlrc.lock... 106 | [YTDLRC] Creating '/tmp/ytdlrc.lock' succeeded. Continuing... 107 | [YTDLRC] Creating download directory: /home/brian/ytdlrc/stage 108 | [YTDLRC] Creating snatch list: /home/brian/ytdlrc/snatch.list 109 | [YTDLRC] Creating archive list: /home/brian/ytdlrc/archive.list 110 | [YTDLRC] Process complete. Removing lock file. 111 | ``` 112 | 113 | ### Example: Downloading all videos from ytuser:kurzgesagt 114 | 115 | `ytuser:kurzgesagt` is the only line in our `snatch.list`. 116 | 117 | #### First run 118 | 119 | ```text 120 | [YTDLRC] Lock file doesn't exist. Attempting to create /tmp/ytdlrc.lock... 121 | [YTDLRC] Creating '/tmp/ytdlrc.lock' succeeded. Continuing... 122 | [YTDLRC] Processing ytuser:kurzgesagt... 123 | [YTDLRC] Grabbing 'playlist_title' from 'ytuser:kurzgesagt'... 124 | [YTDLRC] 'playlist_title' is 'Uploads_from_Kurzgesagt_In_a_Nutshell' 125 | [YTDLRC] Trimming off 'Uploads_from_' from 'Uploads_from_Kurzgesagt_In_a_Nutshell'... 126 | [YTDLRC] New 'playlist_title' is 'Kurzgesagt_In_a_Nutshell' 127 | [debug] System config: [] 128 | [debug] User config: [] 129 | [debug] Custom config: [] 130 | [debug] Command-line args: ['-4', '--continue', '--download-archive', '/home/brian/ytdlrc/archive.list', '--exec', "/usr/bin/rclone move '{}' 'acd:testing/Kurzgesagt_In_a_Nutshell' --config '/home/brian/.rclone.conf' -v --stats 1s", '--ignore-errors', '--no-overwrites', '--restrict-filenames', '--write-description', '--write-info-json', '--write-thumbnail', '--xattrs', '-f', 'bestvideo+bestaudio', '-o', '/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/%(uploader)s.%(upload_date)s.%(title)s.%(resolution)s.%(id)s.%(ext)s', '--verbose', 'ytuser:kurzgesagt'] 131 | [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 132 | [debug] youtube-dl version 2017.03.07 133 | [debug] Python version 3.6.0 - Linux-4.9.11-1-ARCH-x86_64-with-arch 134 | [debug] exe versions: ffmpeg 3.2.4, ffprobe 3.2.4, rtmpdump 2.4 135 | [debug] Proxy map: {} 136 | [youtube:user] kurzgesagt: Downloading channel page 137 | [youtube:playlist] UUsXVk37bltHxD1rDPwtNM8Q: Downloading webpage 138 | [download] Downloading playlist: Uploads from Kurzgesagt – In a Nutshell 139 | [youtube:playlist] playlist Uploads from Kurzgesagt – In a Nutshell: Downloading 58 videos 140 | [download] Downloading video 1 of 58 141 | [youtube] DHyUYg8X31c: Downloading webpage 142 | [youtube] DHyUYg8X31c: Downloading video info webpage 143 | [youtube] DHyUYg8X31c: Extracting video information 144 | [youtube] DHyUYg8X31c: Downloading MPD manifest 145 | [info] Writing video description to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.description 146 | [info] Writing video description metadata as JSON to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.info.json 147 | [youtube] DHyUYg8X31c: Downloading thumbnail ... 148 | [youtube] DHyUYg8X31c: Writing thumbnail to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.jpg 149 | WARNING: Requested formats are incompatible for merge and will be merged into mkv. 150 | [debug] Invoking downloader on 'https://r8---sn-n4v7kne7.googlevideo.com/videoplayback?id=0c7c94620f17df57&itag=299&source=youtube&requiressl=yes&mn=sn-n4v7kne7&mm=31&pl=20&initcwndbps=8067500&mv=m&ms=au&ratebypass=yes&mime=video/mp4&gir=yes&clen=97795996&lmt=1487912067914680&dur=394.266&signature=D4B482B33A060CAD522317EBCF5BA3288C4C95.276EACC1ABBDFFAE143AB016DB9079A47D8329EC&upn=Wj36IpbSSvI&mt=1489800442&key=dg_yt0&ip=104.131.132.15&ipbits=0&expire=1489822135&sparams=ip,ipbits,expire,id,itag,source,requiressl,mn,mm,pl,initcwndbps,mv,ms,ratebypass,mime,gir,clen,lmt,dur' 151 | [download] Destination: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f299.mp4 152 | [download] 100% of 93.27MiB in 00:01 153 | [debug] Invoking downloader on 'https://r8---sn-n4v7kne7.googlevideo.com/videoplayback?pl=20&ei=Vo3MWIusHurD-APq5JnoDg&clen=6926640&itag=251&gir=yes&upn=TwHyVFumtCo&signature=4ADE59C2CE26DABFB4D6418F61D65F10DF13E25B.1B06F7DE826DD862F39066A6D16B90B12D04EE5D&mime=audio%2Fwebm&initcwndbps=8067500&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&ipbits=0&requiressl=yes&keepalive=yes&mn=sn-n4v7kne7&mm=31&mt=1489800442&dur=394.281&id=o-AM978v3HI34Q_CjNzW-0QaneJHqs_vHmTJJzH32vY0-f&lmt=1487796412297530&key=yt6&ip=104.131.132.15&mv=m&source=youtube&ms=au&expire=1489822134&ratebypass=yes' 154 | [download] Destination: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f251.webm 155 | [download] 100% of 6.61MiB in 00:00 156 | [ffmpeg] Merging formats into "/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.mkv" 157 | [debug] ffmpeg command line: ffmpeg -y -i file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f299.mp4 -i file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f251.webm -c copy -map 0:v:0 -map 1:a:0 file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.temp.mkv 158 | Deleting original file /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f299.mp4 (pass -k to keep) 159 | Deleting original file /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.f251.webm (pass -k to keep) 160 | [metadata] Writing metadata to file's xattrs 161 | [exec] Executing command: /usr/bin/rclone move '/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.mkv' 'acd:testing/Kurzgesagt_In_a_Nutshell' --config '/home/brian/.rclone.conf' -v --stats 1s 162 | 2017/03/17 18:29:00 Using RCLONE_CONFIG_PASS password. 163 | 2017/03/17 18:29:00 rclone: Version "v1.35-54-gff8f11dβ" starting with parameters ["/usr/bin/rclone" "move" "/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.mkv" "acd:testing/Kurzgesagt_In_a_Nutshell" "--config" "/home/brian/.rclone.conf" "-v" "--stats" "1s"] 164 | 2017/03/17 18:29:00 acdp: Saved new token in config file 165 | 2017/03/17 18:29:03 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Modify window not supported 166 | 2017/03/17 18:29:03 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Waiting for checks to finish 167 | 2017/03/17 18:29:03 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Waiting for transfers to finish 168 | 2017/03/17 18:29:04 169 | Transferred: 0 Bytes (0 Bytes/s) 170 | Errors: 0 171 | Checks: 0 172 | Transferred: 0 173 | Elapsed time: 4.1s 174 | Transferring: 175 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 0% done, 0 Bytes/s, ETA: - 176 | 177 | 2017/03/17 18:29:05 178 | Transferred: 13.312 MBytes (2.590 MBytes/s) 179 | Errors: 0 180 | Checks: 0 181 | Transferred: 0 182 | Elapsed time: 5.1s 183 | Transferring: 184 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 13% done, 5.278 MBytes/s, ETA: 16s 185 | 186 | 2017/03/17 18:29:06 187 | Transferred: 41.312 MBytes (6.707 MBytes/s) 188 | Errors: 0 189 | Checks: 0 190 | Transferred: 0 191 | Elapsed time: 6.1s 192 | Transferring: 193 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 41% done, 6.744 MBytes/s, ETA: 8s 194 | 195 | 2017/03/17 18:29:07 196 | Transferred: 68.062 MBytes (9.532 MBytes/s) 197 | Errors: 0 198 | Checks: 0 199 | Transferred: 0 200 | Elapsed time: 7.1s 201 | Transferring: 202 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 68% done, 8.003 MBytes/s, ETA: 3s 203 | 204 | 2017/03/17 18:29:08 205 | Transferred: 95.500 MBytes (11.727 MBytes/s) 206 | Errors: 0 207 | Checks: 0 208 | Transferred: 0 209 | Elapsed time: 8.1s 210 | Transferring: 211 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 95% done, 9.293 MBytes/s, ETA: 0s 212 | 213 | 2017/03/17 18:29:09 214 | Transferred: 99.754 MBytes (10.913 MBytes/s) 215 | Errors: 0 216 | Checks: 0 217 | Transferred: 0 218 | Elapsed time: 9.1s 219 | Transferring: 220 | * ...Become_Conscious.1920x1080.DHyUYg8X31c.mkv: 100% done, 9.473 MBytes/s, ETA: 0s 221 | 222 | 2017/03/17 18:29:21 Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.mkv: Copied (new) 223 | 2017/03/17 18:29:21 Kurzgesagt_In_a_Nutshell.20170223.Do_Robots_Deserve_Rights_What_if_Machines_Become_Conscious.1920x1080.DHyUYg8X31c.mkv: Deleted 224 | 2017/03/17 18:29:21 225 | Transferred: 99.754 MBytes (4.598 MBytes/s) 226 | Errors: 0 227 | Checks: 1 228 | Transferred: 1 229 | Elapsed time: 21.6s 230 | 2017/03/17 18:29:21 Go routines at exit 13 231 | [download] Downloading video 2 of 58 232 | [youtube] RVMZxH1TIIQ: Downloading webpage 233 | [youtube] RVMZxH1TIIQ: Downloading video info webpage 234 | [youtube] RVMZxH1TIIQ: Extracting video information 235 | [youtube] RVMZxH1TIIQ: Downloading MPD manifest 236 | [info] Writing video description to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.description 237 | [info] Writing video description metadata as JSON to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.info.json 238 | [youtube] RVMZxH1TIIQ: Downloading thumbnail ... 239 | [youtube] RVMZxH1TIIQ: Writing thumbnail to: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.jpg 240 | WARNING: Requested formats are incompatible for merge and will be merged into mkv. 241 | [debug] Invoking downloader on 'https://r12---sn-n4v7kn76.googlevideo.com/videoplayback?id=455319c47d532084&itag=299&source=youtube&requiressl=yes&mn=sn-n4v7kn76&mm=31&mv=m&initcwndbps=8067500&pl=20&ms=au&ratebypass=yes&mime=video/mp4&gir=yes&clen=96704077&lmt=1486111643921647&dur=416.166&key=dg_yt0&upn=05NaEV1LBgE&signature=35638BDF9D1E278A874D767C7215EF5A9EE8BA23.2CB508CE60975702F17ABD906AC0C079966AD66A&mt=1489800442&ip=104.131.132.15&ipbits=0&expire=1489822162&sparams=ip,ipbits,expire,id,itag,source,requiressl,mn,mm,mv,initcwndbps,pl,ms,ratebypass,mime,gir,clen,lmt,dur' 242 | [download] Destination: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f299.mp4 243 | [download] 100% of 92.22MiB in 00:02 244 | [debug] Invoking downloader on 'https://r12---sn-n4v7kn76.googlevideo.com/videoplayback?mv=m&source=youtube&ms=au&lmt=1485958101260093&ip=104.131.132.15&key=yt6&mt=1489800442&mn=sn-n4v7kn76&mm=31&id=o-APMAlzJZmlhyM8gJr2Qpia97TJJC-xyNgHdb-x3ftfMO&dur=416.161&gir=yes&clen=7619194&itag=251&ei=cY3MWKenLszL-gP_u4O4Bw&pl=20&initcwndbps=8067500&upn=NJAuw32Dclw&signature=B4CF4BDE47558BAD446C092DD53E8C9E62324A8B.73D00E820A74DD2122C00B502364FCECF99969B4&mime=audio%2Fwebm&requiressl=yes&keepalive=yes&expire=1489822161&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&ipbits=0&ratebypass=yes' 245 | [download] Destination: /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f251.webm 246 | [download] 100% of 7.27MiB in 00:00 247 | [ffmpeg] Merging formats into "/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv" 248 | [debug] ffmpeg command line: ffmpeg -y -i file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f299.mp4 -i file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f251.webm -c copy -map 0:v:0 -map 1:a:0 file:/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.temp.mkv 249 | Deleting original file /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f299.mp4 (pass -k to keep) 250 | Deleting original file /home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.f251.webm (pass -k to keep) 251 | [metadata] Writing metadata to file's xattrs 252 | [exec] Executing command: /usr/bin/rclone move '/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv' 'acd:testing/Kurzgesagt_In_a_Nutshell' --config '/home/brian/.rclone.conf' -v --stats 1s 253 | 2017/03/17 18:29:27 Using RCLONE_CONFIG_PASS password. 254 | 2017/03/17 18:29:27 rclone: Version "v1.35-54-gff8f11dβ" starting with parameters ["/usr/bin/rclone" "move" "/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/Kurzgesagt_In_a_Nutshell.20170201.Why_Earth_Is_A_Prison_and_How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv" "acd:testing/Kurzgesagt_In_a_Nutshell" "--config" "/home/brian/.rclone.conf" "-v" "--stats" "1s"] 255 | 2017/03/17 18:29:29 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Modify window not supported 256 | 2017/03/17 18:29:29 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Waiting for checks to finish 257 | 2017/03/17 18:29:29 Encrypted amazon drive root 'crypt/i04dv4pst54cb73aviioosi6z0/itz12de1esv28z1fob78x6p5q3cp4tw1nboxe4g5u8nswp96qsnt': Waiting for transfers to finish 258 | 2017/03/17 18:29:30 259 | Transferred: 1.688 MBytes (617.097 kBytes/s) 260 | Errors: 0 261 | Checks: 0 262 | Transferred: 0 263 | Elapsed time: 2.8s 264 | Transferring: 265 | * ...How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv: 1% done, 0 Bytes/s, ETA: - 266 | 267 | 2017/03/17 18:29:31 268 | Transferred: 21.062 MBytes (5.513 MBytes/s) 269 | Errors: 0 270 | Checks: 0 271 | Transferred: 0 272 | Elapsed time: 3.8s 273 | Transferring: 274 | * ...How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv: 21% done, 3.614 MBytes/s, ETA: 21s 275 | 276 | 2017/03/17 18:29:32 277 | Transferred: 47.438 MBytes (9.882 MBytes/s) 278 | Errors: 0 279 | Checks: 0 280 | Transferred: 0 281 | Elapsed time: 4.8s 282 | Transferring: 283 | * ...How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv: 47% done, 4.707 MBytes/s, ETA: 11s 284 | 285 | 2017/03/17 18:29:33 286 | Transferred: 79 MBytes (13.613 MBytes/s) 287 | Errors: 0 288 | Checks: 0 289 | Transferred: 0 290 | Elapsed time: 5.8s 291 | Transferring: 292 | * ...How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv: 79% done, 6.243 MBytes/s, ETA: 3s 293 | 294 | 2017/03/17 18:29:34 295 | Transferred: 99.365 MBytes (14.612 MBytes/s) 296 | Errors: 0 297 | Checks: 0 298 | Transferred: 0 299 | Elapsed time: 6.8s 300 | Transferring: 301 | * ...How_To_Escape_It.1920x1080.RVMZxH1TIIQ.mkv: 100% done, 7.948 MBytes/s, ETA: 0s 302 | ``` 303 | 304 | This will continue until all videos / metadata are downloaded and moved to 305 | the rclone remote. 306 | 307 | #### Second run 308 | 309 | ```text 310 | [YTDLRC] Lock file doesn't exist. Creating lock file and continuing... 311 | [YTDLRC] Processing ytuser:kurzgesagt... 312 | [YTDLRC] Grabbing 'playlist_title' from 'ytuser:kurzgesagt'... 313 | [YTDLRC] 'playlist_title' is 'Uploads_from_Kurzgesagt_In_a_Nutshell' 314 | [YTDLRC] Trimming off 'Uploads_from_' from 'Uploads_from_Kurzgesagt_In_a_Nutshell'... 315 | [YTDLRC] New 'playlist_title' is 'Kurzgesagt_In_a_Nutshell' 316 | [debug] System config: [] 317 | [debug] User config: [] 318 | [debug] Custom config: [] 319 | [debug] Command-line args: ['-4', '--continue', '--download-archive', '/home/brian/ytdlrc/archive.list', '--exec', "/usr/bin/rclone move '{}' 'acd:archive/youtube/Kurzgesagt_In_a_Nutshell' --config '/home/brian/.rclone.conf' -v --stats 1s", '--ignore-errors', '--no-overwrites', '--restrict-filenames', '--write-description', '--write-info-json', '--write-thumbnail', '--xattrs', '-f', 'bestvideo+bestaudio', '-o', '/home/brian/ytdlrc/stage/Kurzgesagt_In_a_Nutshell/%(uploader)s.%(upload_date)s.%(title)s.%(resolution)s.%(id)s.%(ext)s', '--verbose', 'ytuser:kurzgesagt'] 320 | [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 321 | [debug] youtube-dl version 2017.03.07 322 | [debug] Python version 3.6.0 - Linux-4.9.11-1-ARCH-x86_64-with-arch 323 | [debug] exe versions: ffmpeg 3.2.4, ffprobe 3.2.4, rtmpdump 2.4 324 | [debug] Proxy map: {} 325 | [youtube:user] kurzgesagt: Downloading channel page 326 | [youtube:playlist] UUsXVk37bltHxD1rDPwtNM8Q: Downloading webpage 327 | [download] Downloading playlist: Uploads from Kurzgesagt – In a Nutshell 328 | [youtube:playlist] playlist Uploads from Kurzgesagt – In a Nutshell: Downloading 58 videos 329 | [download] Downloading video 1 of 58 330 | [download] Do Robots Deserve Rights? What if Machines Become Conscious? has already been recorded in archive 331 | [download] Downloading video 2 of 58 332 | [download] Why Earth Is A Prison and How To Escape It has already been recorded in archive 333 | [download] Downloading video 3 of 58 334 | [download] Overpopulation – The Human Explosion Explained has already been recorded in archive 335 | [download] Downloading video 4 of 58 336 | [download] A New History for Humanity – The Human Era has already been recorded in archive 337 | [download] Downloading video 5 of 58 338 | [download] The Most Gruesome Parasites – Neglected Tropical Diseases – NTDs has already been recorded in archive 339 | [download] Downloading video 6 of 58 340 | [download] Fusion Power Explained – Future or Failure has already been recorded in archive 341 | [download] Downloading video 7 of 58 342 | [download] The Most Efficient Way to Destroy the Universe – False Vacuum has already been recorded in archive 343 | [download] Downloading video 8 of 58 344 | [download] How To Eradicate One Of Our Deadliest Enemies – Gene Drive & Malaria has already been recorded in archive 345 | [download] Downloading video 9 of 58 346 | [download] Genetic Engineering Will Change Everything Forever – CRISPR has already been recorded in archive 347 | [download] Downloading video 10 of 58 348 | [download] Death From Space — Gamma-Ray Bursts Explained has already been recorded in archive 349 | [download] Downloading video 11 of 58 350 | [download] What Happened Before History? Human Origins has already been recorded in archive 351 | [download] Downloading video 12 of 58 352 | [download] What Are You? has already been recorded in archive 353 | [download] Downloading video 13 of 58 354 | [download] How Far Can We Go? Limits of Humanity. has already been recorded in archive 355 | [download] Downloading video 14 of 58 356 | [download] Safe and Sorry – Terrorism & Mass Surveillance has already been recorded in archive 357 | [download] Downloading video 15 of 58 358 | [download] Space Elevator – Science Fiction or the Future of Mankind? has already been recorded in archive 359 | [download] Downloading video 16 of 58 360 | [download] The Antibiotic Apocalypse Explained has already been recorded in archive 361 | [download] Downloading video 17 of 58 362 | [download] Why The War on Drugs Is a Huge Failure has already been recorded in archive 363 | [download] Downloading video 18 of 58 364 | [download] The Last Star in the Universe – Red Dwarfs Explained has already been recorded in archive 365 | [download] Downloading video 19 of 58 366 | [download] What Is Something? has already been recorded in archive 367 | [download] Downloading video 20 of 58 368 | [download] Black Holes Explained – From Birth to Death has already been recorded in archive 369 | [download] Downloading video 21 of 58 370 | [download] Quantum Computers Explained – Limits of Human Technology has already been recorded in archive 371 | [download] Downloading video 22 of 58 372 | [download] How Facebook is Stealing Billions of Views has already been recorded in archive 373 | [download] Downloading video 23 of 58 374 | [download] Addiction has already been recorded in archive 375 | [download] Downloading video 24 of 58 376 | [download] What Is Light? has already been recorded in archive 377 | [download] Downloading video 25 of 58 378 | [download] The European Refugee Crisis and Syria Explained has already been recorded in archive 379 | [download] Downloading video 26 of 58 380 | [download] What is Dark Matter and Dark Energy? has already been recorded in archive 381 | [download] Downloading video 27 of 58 382 | [download] What if there was a black hole in your pocket? has already been recorded in archive 383 | [download] Downloading video 28 of 58 384 | [download] The Death Of Bees Explained – Parasites, Poison and Humans has already been recorded in archive 385 | [download] Downloading video 29 of 58 386 | [download] The Fermi Paradox II — Solutions and Ideas – Where Are All The Aliens? has already been recorded in archive 387 | [download] Downloading video 30 of 58 388 | [download] The Fermi Paradox — Where Are All The Aliens? (1/2) has already been recorded in archive 389 | [download] Downloading video 31 of 58 390 | [download] 3 Reasons Why Nuclear Energy Is Terrible! 2/3 has already been recorded in archive 391 | [download] Downloading video 32 of 58 392 | [download] 3 Reasons Why Nuclear Energy Is Awesome! 3/3 has already been recorded in archive 393 | [download] Downloading video 33 of 58 394 | [download] Nuclear Energy Explained: How does it work? 1/3 has already been recorded in archive 395 | [download] Downloading video 34 of 58 396 | [download] Banking Explained – Money and Credit has already been recorded in archive 397 | [download] Downloading video 35 of 58 398 | [download] Measles Explained — Vaccinate or Not? has already been recorded in archive 399 | [download] Downloading video 36 of 58 400 | [download] How Small Is An Atom? Spoiler: Very Small. has already been recorded in archive 401 | [download] Downloading video 37 of 58 402 | [download] The Ultimate Conspiracy Debunker has already been recorded in archive 403 | [download] Downloading video 38 of 58 404 | [download] What Is Life? Is Death Real? has already been recorded in archive 405 | [download] Downloading video 39 of 58 406 | [download] The Ebola Virus Explained — How Your Body Fights For Survival has already been recorded in archive 407 | [download] Downloading video 40 of 58 408 | [download] Is War Over? — A Paradox Explained has already been recorded in archive 409 | [download] Downloading video 41 of 58 410 | [download] Atoms As Big As Mountains — Neutron Stars Explained has already been recorded in archive 411 | [download] Downloading video 42 of 58 412 | [download] Everything You Need to Know About Planet Earth has already been recorded in archive 413 | [download] Downloading video 43 of 58 414 | [download] The Immune System Explained I – Bacteria Infection has already been recorded in archive 415 | [download] Downloading video 44 of 58 416 | [download] Iraq Explained -- ISIS, Syria and War has already been recorded in archive 417 | [download] Downloading video 45 of 58 418 | [download] Are You Alone? (In The Universe) has already been recorded in archive 419 | [download] Downloading video 46 of 58 420 | [download] How to catch a Dwarf Planet -- Triton MM#3 has already been recorded in archive 421 | [download] Downloading video 47 of 58 422 | [download] The Moons of Mars Explained -- Phobos & Deimos MM#2 has already been recorded in archive 423 | [download] Downloading video 48 of 58 424 | [download] How Big is the Moon? MM#1 has already been recorded in archive 425 | [download] Downloading video 49 of 58 426 | [download] Help us make more Videos for Kurzgesagt has already been recorded in archive 427 | [download] Downloading video 50 of 58 428 | [download] Who Invented the Internet? And Why? has already been recorded in archive 429 | [download] Downloading video 51 of 58 430 | [download] The Beginning of Everything -- The Big Bang has already been recorded in archive 431 | [download] Downloading video 52 of 58 432 | [download] Three Ways to Destroy the Universe has already been recorded in archive 433 | [download] Downloading video 53 of 58 434 | [download] The History and Future of Everything -- Time has already been recorded in archive 435 | [download] Downloading video 54 of 58 436 | [download] How The Stock Exchange Works (For Dummies) has already been recorded in archive 437 | [download] Downloading video 55 of 58 438 | [download] The Gulf Stream Explained has already been recorded in archive 439 | [download] Downloading video 56 of 58 440 | [download] Fracking explained: opportunity or danger has already been recorded in archive 441 | [download] Downloading video 57 of 58 442 | [download] The Solar System -- our home in space has already been recorded in archive 443 | [download] Downloading video 58 of 58 444 | [download] How Evolution works has already been recorded in archive 445 | [download] Finished downloading playlist: Uploads from Kurzgesagt – In a Nutshell 446 | [YTDLRC] Process complete. Removing lock file. 447 | ``` 448 | 449 | All available videos were already downloaded and moved to the rclone remote, 450 | so there was nothing to be done on the second run other than check for new 451 | uploads. 452 | 453 | ## Installation 454 | 455 | 1. [Download](https://github.com/bardisty/ytdlrc/archive/master.zip) or 456 | clone the repository: 457 | 458 | `git clone https://github.com/bardisty/ytdlrc` 459 | 460 | 2. `cd` into the directory: 461 | 462 | `cd ytdlrc` 463 | 464 | 3. Open `ytdlrc` in your text editor and see [Usage](#usage). 465 | 466 | ## Usage 467 | 468 | 1. Ensure the path to your rclone configuration file is correct or modify 469 | it if need be: 470 | 471 | ```shell 472 | rclone_config="${HOME}/.config/rclone/rclone.conf" 473 | ``` 474 | 475 | 2. Modify the `rclone_destination` variable with your rclone remote 476 | destination path, e.g.: 477 | 478 | ```diff 479 | - rclone_destination="remote:archive/youtube" 480 | + rclone_destination="gdrive:archive/youtube" 481 | ``` 482 | 483 | 3. Run the script once to generate the working directory, download 484 | directory, snatch list, and archive list: 485 | 486 | ```text 487 | $ ./ytdlrc 488 | [YTDLRC] Creating download directory: /home/brian/ytdlrc/stage 489 | [YTDLRC] Creating snatch list: /home/brian/ytdlrc/snatch.list 490 | [YTDLRC] Creating archive list: /home/brian/ytdlrc/archive.list 491 | [YTDLCR] Process complete. Removing lock file. 492 | ``` 493 | 494 | 4. Put the URL's / channels / playlists you want to download in the 495 | `snatch.list` file, one per line, e.g.: 496 | 497 | * `ytuser:username` 498 | * `https://www.youtube.com/user/username` 499 | * `https://www.youtube.com/playlist?list=PLK9Sc5q_4K6aNajVLKtkaAB1JGmKyccf2` 500 | 501 | 5. *(Optional / Recommended)* Run the script once or twice with debugging 502 | enabled (`debug=true`) to ensure everything is okie dokie. Disable 503 | debugging when done if you don't want `ytdlrc` to spam your system log 504 | when executed via cron. 505 | 506 | 6. Set up a cron job to execute the script however often you want, e.g.: 507 | 508 | ```crontab 509 | # Every 6 hours 510 | 30 */6 * * * /home/your-user/bin/ytdlrc 511 | ``` 512 | 513 | * If you set up the cron job by moving the `ytdlrc` file to one of the 514 | `/etc/cron.*/` directories, you may want to modify the 515 | `rclone_config` variable with the path to your rclone config, 516 | otherwise it will look inside `/root` for your config file: 517 | 518 | ```diff 519 | - rclone_config="${HOME}/.config/rclone/rclone.conf" 520 | + rclone_config="/home/your-user/.config/rclone/rclone.conf" 521 | ``` 522 | 523 | You may also want to modify the `ytdl_root_dir` variable so runtime 524 | files aren't created inside `/root`: 525 | 526 | ```diff 527 | - ytdl_root_dir="${HOME}/ytdlrc" 528 | + ytdl_root_dir="/home/your-user/ytdlrc" 529 | ``` 530 | 531 | 7. *(Optional)* Commit your configuration changes so you can easily remain 532 | updated with the upstream version of the script: 533 | 534 | ```shell 535 | git add ytdlrc 536 | git commit -m "Modify options to my taste" 537 | ``` 538 | 539 | To update the script to the latest version simply run `git pull` and 540 | merge any changes. 541 | 542 | ## Requirements 543 | 544 | * [coreutils](https://www.gnu.org/software/coreutils/coreutils.html) 545 | * [ffmpeg](https://ffmpeg.org/) 546 | * [rclone >= v1.43](http://rclone.org/) 547 | * [youtube-dl](https://rg3.github.io/youtube-dl/) 548 | 549 | ## License 550 | 551 | [MIT](LICENSE) 552 | -------------------------------------------------------------------------------- /ytdlrc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | # 3 | # dP dP dP 4 | # 88 88 88 5 | # dP dP d8888P .d888b88 88 88d888b. .d8888b. 6 | # 88 88 88 88' `88 88 88' `88 88' `"" 7 | # 88. .88 88 88. .88 88 88 88. ... 8 | # .--------------- `8888P88 dP `88888P8 dP dP `88888P' --------------. 9 | # |::::::::::::::::......88 ::..::::.......:..:..::::::::......::::::::::::::::| 10 | # |:::::::::::::::::d8888P ::::::::::::::::::::::::::::::::::::::::::::::::::::| 11 | # |:::::::::::::::::......:::::::::::::::::::::::::::::::::::::::::::::::::::::| 12 | # |----------------------------------------------------------------------------| 13 | # | file | ytdlrc | 14 | # | desc | Downloads files via `youtube-dl` and moves them via `rclone` | 15 | # | author | bardisty | 16 | # | source | https://github.com/bardisty/ytdlrc | 17 | # | modified | Mon Apr 22 2019 16:59:07 PDT -0700 | 18 | # `----------------------------------------------------------------------------' 19 | 20 | 21 | # Enable debugging for verbose output 22 | debug=false 23 | 24 | # Working directory for the script 25 | ytdl_root_dir="${HOME}/ytdlrc" 26 | 27 | # Where to download files before moving them to the rclone remote 28 | ytdl_download_dir="${ytdl_root_dir%/}/stage" 29 | 30 | # List of URL's / channels / playlists to download, e.g.: 31 | # * `ytuser:username` 32 | # * `https://www.youtube.com/user/username` 33 | # * `https://www.youtube.com/playlist?list=PLK9Sc5q_4K6aNajVLKtkaAB1JGmKyccf2` 34 | ytdl_snatch_list="${ytdl_root_dir%/}/snatch.list" 35 | 36 | # List of downloaded/processed files; prevents youtube-dl from 37 | # redownloading files that have already been moved to the rclone remote 38 | ytdl_archive_list="${ytdl_root_dir%/}/archive.list" 39 | 40 | # What video format(s) to grab 41 | ytdl_format="bestvideo+bestaudio/best" 42 | 43 | # Output template for downloaded filenames 44 | ytdl_output_template="%(uploader)s.%(upload_date)s.%(title)s.%(resolution)s.%(id)s.%(ext)s" 45 | 46 | # Write metadata to file's xattrs? 47 | # * Requires the `attr` package and a filesystem that supports extended 48 | # attributes 49 | ytdl_write_metadata_to_xattrs=false 50 | 51 | # Write subtitle file (if available) 52 | ytdl_write_subtitles=true 53 | 54 | # Write automatically generated subtitle file (YouTube only) 55 | ytdl_write_automatic_subtitles=true 56 | 57 | # Download all the available subtitles of the video 58 | # * If `false`, only subtitles matching the specified language(s) 59 | # in the `ytdl_subtitle_lang` setting will be downloaded 60 | # * If `true`, downloading an entire channel may take considerably more 61 | # time to process 62 | ytdl_write_all_subtitles=false 63 | 64 | # What subtitle format to use 65 | ytdl_subtitle_format="srt/best" 66 | 67 | # What subtitle languages to download (separated by commas) 68 | # * Use `youtube-dl --list-subs` to view available language tags 69 | # * Ignored if `ytdl_write_all_subtitles=true` 70 | ytdl_subtitle_lang="en" 71 | 72 | # What value to grab from the first video in a playlist 73 | # * This affects the directory hierarchy on the rclone remote, e.g.: 74 | # `remote:archive/youtube/{playlist_title}/{downloaded files}` 75 | # * The default value - `playlist_title` - will play well with YouTube 76 | # usernames/channels (ytuser:username or URL to channel), as well as 77 | # playlists containing videos from different uploaders. However it 78 | # will not play well with individual YouTube video URL's, as these 79 | # usually return "NA" as the playlist title. A value of `uploader` 80 | # will play well with individual videos and entire channels but not 81 | # with playlists containing videos from different uploaders. 82 | ytdl_video_value="playlist_title" 83 | 84 | # If `youtube-dl` is unable to pull `ytdl_video_value` (default: 85 | # playlist_title) from the first video in a playlist, use this value instead 86 | # * See the `ytdl_skip_on_fail` option if you would rather skip the 87 | # playlist altogether so videos are never downloaded / uploaded into a 88 | # directory that doesn't match the name of the playlist / channel 89 | ytdl_default_video_value="unknown-playlist" 90 | 91 | # Skip playlist if `youtube-dl` is unable to pull `ytdl_video_value` 92 | # (default: playlist_title) from the first video in a playlist 93 | # * Set to `false` if you don't want to skip a playlist if `youtube-dl` 94 | # fails to pull the `ytdl_video_value` (default: playlist_title). 95 | # Instead, videos will be downloaded / uploaded into a directory named 96 | # after `ytdl_default_video_value` (default: "unknown-playlist") 97 | ytdl_skip_on_fail=true 98 | 99 | # By default, the directories videos are saved in have titlecase names, e.g. 100 | # `Some_Channel_Name`. Set to `true` if you prefer lowercase names. 101 | ytdl_lowercase_directories=false 102 | 103 | # Path to rclone config 104 | rclone_config="${HOME}/.config/rclone/rclone.conf" 105 | 106 | # Should we `copy` or `move` files? 107 | rclone_command="move" 108 | 109 | # Where to store downloaded files on the rclone remote 110 | rclone_destination="remote:archive/youtube" 111 | 112 | # Rclone flags 113 | # * Add any additional rclone flags here, e.g.: "--syslog --bwlimit 5M" 114 | # * "--config" is already set, see `rclone_config` variable above 115 | # * If debug=true, "-vv --stats 1s --progress" is appended to rclone 116 | # command 117 | # * If debug=false, "-q" is appended to rclone command 118 | rclone_flags="--transfers 8 --checkers 16 --acd-upload-wait-per-gb 5m" 119 | 120 | 121 | # ---------- You probably don't need to edit anything below this ---------- 122 | 123 | 124 | # POSIX-friendly variable printing 125 | say() { 126 | # Prefix for all printouts 127 | printout_prefix="${text_bold}${text_yellow}[YTDLRC]${text_reset}" 128 | printf %s\\n "${printout_prefix} ${1}" 129 | } 130 | 131 | 132 | # Error printouts 133 | say_error() { 134 | # Prefix for error printouts 135 | printout_prefix_error="${text_bold}${text_red}[Error]${text_reset}" 136 | say "${printout_prefix_error} ${1}" >&2 137 | } 138 | 139 | 140 | say_debug() { 141 | message_type="$2" 142 | if [ "$debug" = true ]; then 143 | 144 | if [ "$message_type" = "success" ]; then 145 | printout_prefix_debug="${text_bold}${text_gray}[Debug]${text_green} [OK]${text_reset}" 146 | else 147 | printout_prefix_debug="${text_bold}${text_gray}[Debug]${text_reset}" 148 | fi 149 | 150 | say "${printout_prefix_debug} ${1}" 151 | 152 | fi 153 | } 154 | 155 | 156 | # Determine if a directory is empty 157 | # Source: http://www.etalabs.net/sh_tricks.html 158 | is_empty() { 159 | cd "$1" || return 0 160 | set -- .[!.]* ; test -f "$1" && return 1 161 | set -- ..?* ; test -f "$1" && return 1 162 | set -- * ; test -f "$1" && return 1 163 | return 0 164 | } 165 | 166 | 167 | command_exists() { 168 | cmd="$1" 169 | 170 | if eval type type > /dev/null 2>&1; then 171 | eval type "$cmd" > /dev/null 2>&1 172 | else 173 | command -v "$cmd" > /dev/null 2>&1 174 | fi 175 | 176 | return $? 177 | } 178 | 179 | 180 | # Get a value from the video (default is playlist_title) and store it in a 181 | # variable 182 | # * Required to store videos on rclone remote in separate directories 183 | # using the pulled value 184 | get_video_value() { 185 | # Printout example: "Grabbing 'playlist_title' from 'ytuser:username'..." 186 | say_debug "Grabbing '${1}' from '${3}'..." 187 | 188 | video_value=$( 189 | youtube-dl \ 190 | --force-ipv4 \ 191 | --get-filename \ 192 | --output "%(${1})s" \ 193 | --playlist-items "$2" \ 194 | --restrict-filenames \ 195 | "$3" 196 | ) 197 | 198 | # Assign a default value if `youtube-dl` was unable to pull 199 | # `playlist_title` from the video 200 | video_value="${video_value:-$ytdl_default_video_value}" 201 | } 202 | 203 | 204 | download_all_the_things() { 205 | # shellcheck disable=SC2086 206 | youtube-dl \ 207 | --force-ipv4 \ 208 | --continue \ 209 | --download-archive "$ytdl_archive_list" \ 210 | --exec "rclone $rclone_command \ 211 | '{}' '${rclone_destination%/}/${video_value}' \ 212 | --config '$rclone_config' \ 213 | $rclone_flags \ 214 | $rclone_debug_flags" \ 215 | --format "$ytdl_format" \ 216 | --ignore-config \ 217 | --ignore-errors \ 218 | --no-overwrites \ 219 | --output "${ytdl_download_dir%/}/${video_value}/${ytdl_output_template}" \ 220 | --restrict-filenames \ 221 | --write-description \ 222 | --write-info-json \ 223 | --write-thumbnail \ 224 | "$ytdl_debug_flags" \ 225 | $ytdl_subtitle_flags \ 226 | $ytdl_xattrs_flag \ 227 | "$1" 228 | } 229 | 230 | 231 | check_rclone_version() { 232 | minimum_required_version=1.43 233 | rclone_version=$(rclone --version|awk '/rclone/ { print $2 }') 234 | version_prefix="v" 235 | version_dev="-DEV" 236 | version_beta="-beta" 237 | 238 | if_contains_string() { 239 | if test "${rclone_version#*$1}" != "$rclone_version"; then 240 | rclone_version=$(printf %s\\n "$rclone_version"|$2) 241 | fi 242 | } 243 | 244 | if ! command_exists awk; then 245 | say_error "Command not found: awk" 246 | delete_lock_file 247 | exit 127 248 | fi 249 | 250 | say_debug "Checking if rclone meets minimum required version (${minimum_required_version})..." 251 | 252 | # If beta, trim off everything after version number 253 | if_contains_string "$version_beta" "cut -d - -f 1" 254 | 255 | # If dev, trim off everything after version number 256 | if_contains_string "$version_dev" "cut -d - -f 1" 257 | 258 | # Trim off `v` from version number 259 | if_contains_string "$version_prefix" "sed s/^${version_prefix}//" 260 | 261 | meets_minimum_required_version=$(awk -v current="$rclone_version" \ 262 | -v required="$minimum_required_version" \ 263 | 'BEGIN{print current /dev/null 2>&1; then 295 | if command_exists tput; then 296 | text_reset=$(tput sgr0) 297 | text_bold=$(tput bold) 298 | text_red=$(tput setaf 1) 299 | text_yellow=$(tput setaf 3) 300 | text_green=$(tput setaf 2) 301 | text_gray=$(tput setaf 8) 302 | fi 303 | fi 304 | 305 | 306 | # Where to store temp runtime files (FIFO and lock file) 307 | temp_file_dir="/tmp" 308 | 309 | # Where to store FIFO for reading the snatch list 310 | fifo="${temp_file_dir%/}/ytdlrc.fifo" 311 | 312 | # Where to store lock file to prevent the script from running more than once 313 | # at a time 314 | lock_file="${temp_file_dir%/}/ytdlrc.lock" 315 | 316 | 317 | # Set `youtube-dl` and `rclone` command flags depending on debugging status 318 | if [ "$debug" = true ]; then 319 | ytdl_debug_flags="--verbose" 320 | rclone_debug_flags="-vv --stats 1s --progress" 321 | else 322 | ytdl_debug_flags="--quiet" 323 | rclone_debug_flags="-q" 324 | fi 325 | 326 | if [ "$ytdl_write_subtitles" = true ] || [ "$ytdl_write_automatic_subtitles" = true ]; then 327 | ytdl_subtitle_flags="--sub-format ${ytdl_subtitle_format}" 328 | 329 | # Set `youtube-dl` flags for writing subtitle files 330 | if [ "$ytdl_write_subtitles" = true ]; then 331 | ytdl_subtitle_flags="${ytdl_subtitle_flags} --write-sub" 332 | fi 333 | 334 | # Set `youtube-dl` flags for writing automatic subtitles 335 | if [ "$ytdl_write_automatic_subtitles" = true ]; then 336 | ytdl_subtitle_flags="${ytdl_subtitle_flags} --write-auto-sub" 337 | fi 338 | 339 | # Set `youtube-dl` flags for writing all subtitles or only specified languages 340 | if [ "$ytdl_write_all_subtitles" = true ]; then 341 | # Append `--all-subs` flag 342 | ytdl_subtitle_flags="${ytdl_subtitle_flags} --all-subs" 343 | else 344 | # Append `--sub-lang` flag 345 | ytdl_subtitle_flags="${ytdl_subtitle_flags} --sub-lang ${ytdl_subtitle_lang}" 346 | fi 347 | 348 | fi 349 | 350 | 351 | # Catch CTRL-C; clean up temp files and exit script to prevent further 352 | # commands from executing 353 | trap 'delete_lock_file && rm "$fifo" && exit 0' 2 354 | 355 | 356 | # Check for existence of lock file 357 | if [ -f "$lock_file" ]; then 358 | # Lock file exists; do nothing 359 | say_debug "Lock file exists: ${lock_file}" 360 | say_debug "Exiting..." 361 | exit 0 362 | else 363 | # Lock file doesn't exist; create it 364 | say_debug "Lock file doesn't exist. Attempting to create '${lock_file}'..." 365 | # Check if temp file directory exists 366 | if [ -d "${temp_file_dir%/}" ]; then 367 | # Temp file directory exists; attempt to create lock file 368 | if touch "$lock_file"; then 369 | say_debug "Creating '${lock_file}' succeeded. Continuing..." success 370 | else 371 | say_error "Could not create lock file '${lock_file}'. Exiting..." 372 | exit 1 373 | fi 374 | else 375 | # Temp file directory does not exist; attempt to create it 376 | say_debug "Temp file directory '${temp_file_dir%/}' not found. Attempting to create it..." 377 | if mkdir -p "${temp_file_dir%/}"; then 378 | # Attempt to create lock file 379 | say_debug "Creating '${temp_file_dir%/}' succeeded. Creating lock file..." success 380 | if touch "$lock_file"; then 381 | say_debug "Creating '${lock_file}' succeeded. Continuing..." success 382 | else 383 | say_error "Could not create lock file '${lock_file}'. Exiting..." 384 | exit 1 385 | fi 386 | else 387 | say_error "Could not create temp directory '${temp_file_dir%/}'. Exiting..." 388 | exit 1 389 | fi 390 | fi 391 | fi 392 | 393 | 394 | # Generate required runtime directory and files if they don't already exist 395 | if [ ! -d "${ytdl_download_dir%/}" ]; then 396 | say "Creating download directory: ${ytdl_download_dir%/}" 397 | if ! mkdir -p "${ytdl_download_dir%/}"; then 398 | say_error "Could not create download directory '${ytdl_download_dir%/}'. Exiting..." 399 | delete_lock_file 400 | exit 1 401 | fi 402 | fi 403 | 404 | if [ ! -f "$ytdl_snatch_list" ]; then 405 | say "Creating snatch list: ${ytdl_snatch_list}" 406 | if ! touch "$ytdl_snatch_list"; then 407 | say_error "Could not create snatch list '${ytdl_snatch_list}'. Exiting..." 408 | delete_lock_file 409 | exit 1 410 | fi 411 | fi 412 | 413 | if [ ! -f "$ytdl_archive_list" ]; then 414 | say "Creating archive list: ${ytdl_archive_list}" 415 | if ! touch "$ytdl_archive_list"; then 416 | say_error "Could not create archive list '${ytdl_archive_list}'. Exiting..." 417 | delete_lock_file 418 | exit 1 419 | fi 420 | fi 421 | 422 | 423 | if [ ! -s "$ytdl_snatch_list" ]; then 424 | say_error "${ytdl_snatch_list} is empty. Exiting..." 425 | delete_lock_file 426 | exit 1 427 | fi 428 | 429 | 430 | say_debug "Checking required commands..." 431 | required_commands="youtube-dl ffmpeg rclone" 432 | for cmd in $required_commands; do 433 | if ! command_exists "$cmd"; then 434 | say_error "Command not found: ${cmd}" 435 | delete_lock_file 436 | exit 127 437 | else 438 | say_debug "Command found: ${cmd}" success 439 | fi 440 | done 441 | 442 | 443 | # Ensure rclone meets minimum required version 444 | check_rclone_version 445 | 446 | 447 | # Check if rclone config exists 448 | say_debug "Checking if rclone configuration file exists..." 449 | if [ ! -f "$rclone_config" ]; then 450 | say_error "Rclone configuration not found: ${rclone_config}" 451 | delete_lock_file 452 | exit 1 453 | else 454 | say_debug "Using rclone configuration: ${rclone_config}" success 455 | fi 456 | 457 | # Check if rclone remote has any issues 458 | say_debug "Checking rclone remote for any issues..." 459 | if ! rclone about "$rclone_destination" > /dev/null 2>&1; then 460 | say_error "Could not read rclone remote '${rclone_destination}'. See the \`rclone_destination\` setting." 461 | say_error "If the remote looks correct, check for issues by running: \`rclone about ${rclone_destination}\`" 462 | delete_lock_file 463 | exit 1 464 | else 465 | say_debug "Remote exists. No issues found." success 466 | fi 467 | 468 | 469 | # Ensure xattrs are supported, if enabled 470 | if [ "$ytdl_write_metadata_to_xattrs" = true ]; then 471 | 472 | if ! command_exists attr; then 473 | say_error "Command not found: attr" 474 | say_error "Please install the \`attr\` package or set \`ytdl_write_metadata_to_xattrs\` to \`false\`." 475 | delete_lock_file 476 | exit 127 477 | fi 478 | 479 | xattr_test_file="${ytdl_download_dir%/}/ytdlrc_xattr_test" 480 | if touch "$xattr_test_file"; then 481 | if ! setfattr -n "user.testAttr" -v "attribute value" "$xattr_test_file" > /dev/null 2>&1; then 482 | say_error "Extended attributes not supported." 483 | say_error "Please set \`ytdl_write_metadata_to_xattrs\` to \`false\`." 484 | rm "$xattr_test_file" 485 | delete_lock_file 486 | exit 1 487 | else 488 | ytdl_xattrs_flag="--xattrs" 489 | rm "$xattr_test_file" 490 | fi 491 | else 492 | say_error "Could not create xattrs test file. Does ${ytdl_download_dir%/} exist?" 493 | say_error "You can bypass this by setting \`ytdl_write_metadata_to_xattrs\` to \`false\`." 494 | delete_lock_file 495 | exit 1 496 | fi 497 | 498 | fi 499 | 500 | 501 | # Read through `ytdl_snatch_list` file only if it contains URL's / 502 | # usernames, assign each line to the `url` variable 503 | # * We do this instead of using youtube-dl's `--batch-file` option in 504 | # order to save playlist/channel names to a variable for an improved 505 | # directory hierarchy on the rclone remote destination 506 | if [ -s "$ytdl_snatch_list" ]; then 507 | mkfifo "$fifo" 508 | grep -v '^ *#' < "$ytdl_snatch_list" > "$fifo" & 509 | while IFS= read -r url; do 510 | if [ -n "$url" ]; then 511 | say "Processing ${url}..." 512 | get_video_value "$ytdl_video_value" "1" "$url" 513 | 514 | if [ "$video_value" = "$ytdl_default_video_value" ]; then 515 | # Failed to grab video value, try the 2nd video before proceeding... 516 | say_debug "Failed to grab '${ytdl_video_value}' from '${url}'. Trying 2nd video instead..." 517 | get_video_value "$ytdl_video_value" "2" "$url" 518 | if [ "$video_value" = "$ytdl_default_video_value" ]; then 519 | # Failed again, if true skip, if false continue with the default 520 | # value (`ytdl_default_video_value`) 521 | if [ "$ytdl_skip_on_fail" = true ]; then 522 | say_debug "Failed to grab '${ytdl_video_value}' from '${url}' after 2 attempts. Skipping..." 523 | continue 524 | else 525 | # Printout example: "Unable to grab 'playlist_title' from '{url}'. Using 526 | # default value 'unknown-playlist' instead." 527 | say_debug "Unable to grab '${ytdl_video_value}' from '${url}'. Using default value '${ytdl_default_video_value}' instead." 528 | fi 529 | fi 530 | fi 531 | 532 | # Process value returned by `get_video_value()` 533 | if [ "$video_value" != "$ytdl_default_video_value" ]; then 534 | # Printout example: "'playlist_title' is 'Uploads_from_Some_Channel_Name'" 535 | say_debug "'${ytdl_video_value}' is '${video_value}'" success 536 | 537 | # Trim off 'Uploads_from_' when returning titles of playlists that are just 538 | # all the videos a user has uploaded 539 | if [ "$ytdl_video_value" = "playlist_title" ]; then 540 | string_to_trim="Uploads_from_" 541 | if test "${video_value#*$string_to_trim}" != "$video_value"; then 542 | # Printout example: "Trimming off 'Uploads_from_' from 'Uploads_from_Channel_Name'" 543 | say_debug "Trimming off '${string_to_trim}' from '${video_value}'..." 544 | video_value="$(printf %s\\n "$video_value"|sed "s/^${string_to_trim}//")" 545 | # Printout example: "New 'playlist_title' is 'Channel_Name'" 546 | say_debug "New '${ytdl_video_value}' is '${video_value}'" success 547 | fi 548 | fi 549 | 550 | if [ "$ytdl_lowercase_directories" = true ]; then 551 | # Printout example: "Converting 'Channel_Name' to lowercase..." 552 | say_debug "Converting '${video_value}' to lowercase..." 553 | video_value=$(printf %s\\n "$video_value"|tr '[:upper:]' '[:lower:]') 554 | # Printout example: "New 'playlist_title' is 'channel_name' 555 | say_debug "New '${ytdl_video_value}' is '${video_value}'" success 556 | fi 557 | fi 558 | 559 | download_all_the_things "$url" 560 | 561 | # youtube-dl's `--exec` option only processes the video file and not any 562 | # metadata files (.json, .jpg, .description, etc.). `rclone` is executed 563 | # again to ensure metadata gets moved to the rclone remote as well. 564 | # * The download directory check is to ensure we don't run `rclone` 565 | # unnecessarily in the event no new videos/metadata are downloaded 566 | download_directory="${ytdl_download_dir%/}/${video_value}" 567 | if [ -d "$download_directory" ]; then 568 | say_debug "Uploading metadata to rclone remote..." 569 | # shellcheck disable=SC2086 570 | rclone "$rclone_command" "$download_directory" \ 571 | "${rclone_destination%/}/${video_value}" --config "$rclone_config" \ 572 | $rclone_flags $rclone_debug_flags 573 | fi 574 | 575 | # Delete leftover download directory if: 576 | # - it exists 577 | # - it doesn't contain any files 578 | # - `rclone_command` is set to move 579 | if [ -d "$download_directory" ] && is_empty "$download_directory" && [ "$rclone_command" = "move" ]; then 580 | say_debug "Removing leftover download directory: ${download_directory}" 581 | rmdir "$download_directory" 582 | fi 583 | 584 | fi 585 | done < "$fifo" 586 | rm "$fifo" 587 | fi 588 | 589 | say "Process completed at $(date --iso-8601=seconds)." 590 | delete_lock_file 591 | 592 | --------------------------------------------------------------------------------