├── .gitmodules ├── Readme.md ├── example ├── add_channel.sh.example ├── callback.bash.example ├── channels.bash.example ├── isBanned.bash.example ├── start.bash.example └── test.bash.example ├── ipv6 └── ipv6.bash └── scripts └── vtbackup.init.sh /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "live-dl"] 2 | path = live-dl 3 | url = https://github.com/lekoOwO/live-dl 4 | branch = master 5 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # vtbackup 2 | 3 | This project contains all scripts you need to automatically grab live streams from several channels. 4 | 5 | Highlights: 6 | 7 | 1. Auto download live streams when it starts 8 | 9 | 2. Callback function. Allowing to upload to GDrive, OneDrive...(rclone supported) after videos are downloaded. 10 | 11 | 3. Address Pool to prevent HTTP 429 error from Youtube. 12 | 13 | 4. Using a light-weight Alpine Linux! 14 | 15 | 5. Telegram notification 16 | 17 | 6. Monitored channels are easy to be added 18 | 19 | ## Requirements 20 | 21 | - Alpine Linux 22 | - a IPv6 prefix (optional, strongly recommended) 23 | - a large enough Cloud Drive (optional, recommended) 24 | - Telegram bot token (optional) 25 | 26 | ## Installation 27 | 28 | Run `git clone --recursive https://github.com/lekoOwO/vtbackup .` on `/root` 29 | 30 | ## Init 31 | 32 | Install all dependencies by running `sh /root/scripts/live-dl.init.sh`. 33 | 34 | ## Config 35 | 36 | ### 1. Config live-dl 37 | 38 | Config live-dl with its example config file. 39 | 40 | ### 2. IPv6 Pool 41 | 42 | It's recommended to config a IPv6 Pool to prevent Youtube HTTP 429 error. 43 | 44 | Make sure that you are in a SLAAC (IPv6 stateless) environment. 45 | 46 | #### 2.a Config script 47 | 48 | fill `PREFIX`, `DEV`, `CIDR` in `/root/ipv6/ipv6.bash` and make it executable. 49 | 50 | It's using `${PREFIX}${SUFFIX}::1` to `${PREFIX}${SUFFIX}::ff` by default, change it if you want (just modify the for-loop part, easy.) 51 | 52 | #### 2.b Config live-dl 53 | 54 | ``` 55 | address_pool: true 56 | address_pool_file: /root/ipv6/address.txt 57 | ``` 58 | 59 | ### 3. Callback 60 | 61 | #### 3.a Config live-dl 62 | 63 | ``` 64 | run_callback: true 65 | callback: 66 | executable: /root/scripts/callback.bash 67 | ``` 68 | 69 | #### 3.b1 Using lekoOwO's script 70 | 71 | Download rclone and config your storage. 72 | 73 | A example callback file is provided, modify it to meet your need :D 74 | 75 | Downloaded video will be uploaded to your cloud storage and deleted locally. 76 | 77 | #### 3.b2 Use your own solution 78 | 79 | Write your own callback script, yay! 80 | 81 | `EXECUTABLE "${OUTPUT_PATH}.mp4" "$BASE_DIR/" $VIDEO_ID $FULLTITLE $UPLOADER $UPLOAD_DATE` will be called after the video is downloaded. 82 | 83 | Save your callback script to `/root/scripts/callback.bash` and make it executable. 84 | 85 | ### 4. YTArchive 86 | 87 | [YTArchive](https://github.com/Kethsar/ytarchive) is a convinient script to archive youtube videos. 88 | 89 | `config.yml` 90 | ``` 91 | use_ytarchive: true 92 | ytarchive: 93 | executable: YTARCHIVE_PY_LOCATION 94 | cookie: YOUR_COOKIE_TXT 95 | ``` 96 | 97 | ### 5. Telegram Notification 98 | 99 | This part helps you to know whether you are 429ed by receiving messages on a telegram channel every x minutes. 100 | 101 | #### 5.a isBanned.bash 102 | 103 | Fill your `CHANNEL_ID`, `CHAT_ID`, `BOT_TOKEN` in. 104 | 105 | #### 5.b Make it run every 15 minutes 106 | 107 | ``` 108 | cd /etc/periodic/15min/ 109 | ln /root/scripts/isBanned.bash 110 | mv ./isBanned.bash ./isBanned 111 | chmod +x ./isBanned 112 | ``` 113 | 114 | ##### 4.b.1 Enable crontab 115 | `rc-service crond start && rc-update add crond default` 116 | 117 | ### 6. Startup script 118 | 119 | Look at `start.bash.example`. 120 | 121 | Make it executable. 122 | 123 | #### 6.a Make in run at boot 124 | 125 | ``` 126 | rc-update add local default 127 | cd /etc/local.d/ 128 | ln /root/start.bash 129 | mv ./start.bash ./live-dl.start 130 | chmod +x ./live-dl.start 131 | ``` 132 | 133 | ### 7. Add monitored channel 134 | 135 | `add_channel.sh` is a convinient script to add a monitored channel! 136 | 137 | Don't forget to edit it (`DEFAULT_TELEGRAM_CHANNEL_ID`)! -------------------------------------------------------------------------------- /example/add_channel.sh.example: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | TELEGRAM_CHANNEL_ID="CHANNEL_ID" 3 | 4 | read -p 'Channel Name: ' name 5 | read -p 'Channel ID: ' channel_id 6 | 7 | sed -i "/CHANNEL_IDS=(/a \\ \\ [\"$name\"]=\"$channel_id\"" /root/scripts/channels.bash 8 | 9 | cat >> /root/live-dl/config.yml </dev/null | grep -oP 'Adding segment \K\d+(?= to queue)' 2>/dev/null | head -n 1) 38 | # last_segment=$(cat "${FILEPATH_WITHOUT_EXT}.log" | grep -oP 'Adding segment \K\d+(?= to queue)' | tail -n 1) 39 | # segments="${first_segment} ~ ${last_segment}" 40 | 41 | # ytarchive 42 | segments=$(cat "${FILEPATH_WITHOUT_EXT}.log" | grep -oP 'Video fragments: \K(\d+)' | tail -n 1) 43 | 44 | # ----- Evalulate variables END ----- # 45 | 46 | function upload { 47 | local drive="$1" 48 | local excludeExt="$2" 49 | 50 | for i in "${exts[@]}"; do 51 | if [[ "$excludeExt" = "$i" ]]; then 52 | continue 53 | fi 54 | local_file="${FILEPATH_WITHOUT_EXT}.${i}" 55 | 56 | _output="${local_file##${base_dir}}" 57 | _output_folder=$(dirname "${_output}") 58 | _output_filename=$(basename "${_output}") 59 | 60 | rclone copyto --no-traverse "${local_file}" "${drive}:/${_output_folder}/${FILENAME_WITHOUT_EXT}/${_output_filename}" 61 | done 62 | 63 | rclone copyto --no-traverse "${video_file}" "${drive}:/${output_folder}/${FILENAME_WITHOUT_EXT}/${output_filename}" 64 | } 65 | 66 | function clearFile { 67 | for i in "${exts[@]}"; do 68 | rm -rf "${FILEPATH_WITHOUT_EXT}.${i}" 69 | done 70 | 71 | rm -rf "${video_file}" 72 | } 73 | 74 | function getDuration { 75 | local seconds=`ffmpeg -i "${video_file}" 2>&1 | awk '/Duration/ {split($2,a,":");print a[1]*3600+a[2]*60+a[3]}'` 76 | local result=`date -d@$seconds -u +%H:%M:%S` 77 | 78 | echo "$result" 79 | } 80 | 81 | # ----- Post variable evaluation ----- # 82 | 83 | duration=`getDuration` 84 | 85 | # ----- Post variable evaluation END ----- # 86 | 87 | # ----- Jobs ----- # 88 | 89 | # Upload to cloud. 90 | upload "MyCloudDriveA" 91 | 92 | # Upload to cloud, ignoring .log files. 93 | upload "MyCloudDriveB" "log" 94 | 95 | # Get file link 96 | link=`rclone link "MyCloudDriveB:/${output_folder}/${FILENAME_WITHOUT_EXT}"` 97 | 98 | # Log to TG 99 | tg_type="sendPhoto" 100 | tg_body='{ 101 | "chat_id": "'"$CHAT_ID"'", 102 | "photo": "'"$photoUrl"'", 103 | "caption": "'"${output_folder}"'\n\n\`'"${FILENAME_WITHOUT_EXT}"'\`\n\nSegment: '"${segments}"'\nDuration: '"${duration}"'\n\n*UPLOADED*", 104 | "parse_mode": "Markdown", 105 | "reply_markup": { 106 | "inline_keyboard": [ 107 | [ 108 | {"text": "Google Drive", "url": "'"$link"'"}, 109 | {"text": "YouTube Video", "url": "https://www.youtube.com/watch?v='"$video_id"'"} 110 | ] 111 | ] 112 | } 113 | }' 114 | curl \ 115 | -4 \ 116 | --connect-timeout 5 \ 117 | --max-time 10 \ 118 | --retry 5 \ 119 | --retry-delay 0 \ 120 | --retry-max-time 40 \ 121 | -s \ 122 | -X POST \ 123 | -H 'Content-Type: application/json' \ 124 | -d "${tg_body}" \ 125 | "https://api.telegram.org/bot${BOT_TOKEN}/${tg_type}" \ 126 | > /dev/null & 127 | 128 | # Upload if channel name contains "牧野白", ignoring .log files. 129 | if [[ "$channel_name" =~ "牧野白" ]]; then 130 | upload "MyCloudDriveC" "log" & 131 | fi 132 | 133 | # Wait for upload to finish 134 | wait 135 | 136 | # Clear up. 137 | clearFile 138 | 139 | # ----- Jobs END ----- # -------------------------------------------------------------------------------- /example/channels.bash.example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | declare -A CHANNEL_IDS 4 | CHANNEL_IDS=( 5 | ["Makino Shiro Ch. 牧野白"]="UCbZcxNKrC0a6IZYBowvzAUg" 6 | ["Gawr Gura Ch. hololive-EN"]="UCoSrY_IQQVpmIRZ9Xf-y93g" 7 | ) 8 | 9 | mapfile -d '' CHANNEL_IDS_SORTED < <(printf '%s\0' "${!CHANNEL_IDS[@]}" | sort -z) -------------------------------------------------------------------------------- /example/isBanned.bash.example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | CHANNEL_ID="TEST_YOUTUBE_CHANNEL_ID" 3 | ADDRESS_POOL="/root/ipv6/address.txt" 4 | CHAT_ID="SUCCESS_CHANNEL_ID" 5 | FAILED_CHAT_ID="FAILED_CHAT_ID" 6 | BOT_TOKEN="BOT_TOKEN" 7 | 8 | # Look at test.bash.example 9 | TEST_SCRIPT_PATH="/root/scripts/test.bash" 10 | 11 | chatId="${CHAT_ID}" 12 | 13 | ip=$(shuf -n 1 "${ADDRESS_POOL}") 14 | 15 | USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36" 16 | 17 | # Check the internet connection. 18 | echo "Checking the internet..." 19 | echo "Using IP: ${ip}" 20 | 21 | ping -6 -I "${ip}" -c1 "google.com" 1>/dev/null 2>/dev/null 22 | SUCCESS=$? 23 | if [ $SUCCESS -eq "0" ]; then 24 | echo "Internet is OK." 25 | else 26 | echo "Internet Failed." 27 | echo "Try using SLAAC IP..." 28 | ping -6 -I "$SLAAC_IP" -c1 "google.com" 1>/dev/null 2>/dev/null 29 | SUCCESS=$? 30 | if [ $SUCCESS -eq "0" ]; then 31 | echo "SLAAC Success." 32 | 33 | echo "Retrying original IP." 34 | ping -6 -I "$ip" -c1 "google.com" 1>/dev/null 2>/dev/null 35 | SUCCESS=$? 36 | if [ $SUCCESS -eq "0" ]; then 37 | echo "Internet is OK." 38 | else 39 | status="INTERNET FAILED - SLAAC SUCCESS" 40 | chatId="${FAILED_CHAT_ID}" 41 | fi 42 | else 43 | status="INTERNET FAILED - SLAAC FAILED" 44 | chatId="${FAILED_CHAT_ID}" 45 | fi 46 | fi 47 | 48 | echo "Checking if Youtube blocked us..." 49 | result=`curl -sIL -X GET --compressed -H "User-Agent: $USER_AGENT" --interface $ip \ 50 | "https://www.youtube.com/channel/$CHANNEL_ID/live" | tac | tac | head -n 1 | cut -d ' ' -f2` 51 | 52 | isBanned="false" 53 | # Send to Telegram 54 | if [ ${#status} -eq "0" ]; then 55 | if [ "$result" != "200" ]; then 56 | status="**BANNED** - ${result}" 57 | disable_notification="false" 58 | isBanned="true" 59 | chatId="${FAILED_CHAT_ID}" 60 | else 61 | status="OK" 62 | disable_notification="true" 63 | fi 64 | fi 65 | echo "status: $status" 66 | 67 | echo "Checking live-dl status..." 68 | liveDlResult="$(/bin/bash $TEST_SCRIPT_PATH)" 69 | echo "$liveDlResult" 70 | 71 | tg_body_text="IP Status: $status\nIP: \`$ip\`\n\n\`${liveDlResult}\`" 72 | 73 | tg_type="sendMessage" 74 | tg_body='{ 75 | "chat_id": "'"${chatId}"'", 76 | "text": "'"${tg_body_text}"'", 77 | "disable_notification": "'"$disable_notification"'", 78 | "parse_mode": "Markdown" 79 | }' 80 | 81 | 82 | curl -4 -s -X POST -H 'Content-Type: application/json' -d "$tg_body" "https://api.telegram.org/bot$BOT_TOKEN/$tg_type" > /dev/null & -------------------------------------------------------------------------------- /example/start.bash.example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | /root/ipv6/ipv6.bash 4 | 5 | . /root/scripts/channels.bash 6 | 7 | mkdir -p /tmp/live-dl 8 | 9 | for key in "${!CHANNEL_IDS[@]}"; do 10 | PM2_HOME='/root/.pm2' pm2 start /bin/bash --name "$key" -- /root/live-dl/live-dl "https://www.youtube.com/channel/${CHANNEL_IDS[${key}]}" 11 | done 12 | 13 | /root/scripts/isBanned.bash & -------------------------------------------------------------------------------- /example/test.bash.example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Look at channels.bash.example 4 | . /root/scripts/channels.bash 5 | 6 | LIVE_DL_PATH="/root/live-dl/live-dl" 7 | 8 | for name in "${CHANNEL_IDS_SORTED[@]}"; do 9 | id="${CHANNEL_IDS[${name}]}" 10 | 11 | count=$(ps aux | grep "$id" | grep "$LIVE_DL_PATH" | wc -l) 12 | if [ $count -eq "0" ]; then 13 | status="❌" 14 | else 15 | status="" 16 | if [[ $(ps aux | grep "$name" | grep "streamlink" | wc -l) -ne "0" ]] || \ 17 | [[ $(ps aux | grep "$name" | grep "ytarchive" | wc -l) -ne "0" ]]; then 18 | status="🔴" 19 | elif [ $(ps aux | grep "$name" | grep "ffmpeg" | wc -l) -ne "0" ]; then 20 | status="🔄" 21 | elif [ $(ps aux | grep "$name" | grep "rclone" | wc -l) -ne "0" ]; then 22 | status="📤" 23 | else 24 | status="✅" 25 | fi 26 | fi 27 | echo "${status}${name}" 28 | done -------------------------------------------------------------------------------- /ipv6/ipv6.bash: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DEV="eth0" 4 | CIDR="64" 5 | 6 | PREFIX="2001:0123:4567:89a" 7 | SUFFIX="0" 8 | 9 | isReboot="${1:-false}" 10 | 11 | OUTPUT_FILE="/root/ipv6/address.txt" 12 | if [ -f $OUTPUT_FILE ]; then 13 | echo "Deleting old IPv6 addresses..." 14 | OLD_SUFFIX="" 15 | while IFS= read -r line; do 16 | if [ ${#OLD_SUFFIX} -eq 0 ]; then 17 | sp=`echo $line | grep -oP '.+:\K.+?(?=::.+)'` 18 | OLD_SUFFIX=${sp: -1} 19 | fi 20 | ip a del "$line/$CIDR" dev $DEV 21 | done < "$OUTPUT_FILE" 22 | SUFFIX="$(((0x$OLD_SUFFIX + 1)%16))" 23 | SUFFIX=$(printf "%x\n" $SUFFIX) 24 | fi 25 | rm "$OUTPUT_FILE" 26 | touch "$OUTPUT_FILE" 27 | 28 | X_PREFIX="${PREFIX}${SUFFIX}::" 29 | for (( num=1; num <= 0xff; num+=1 )); do 30 | suffix=$(printf "%x\n" $num) 31 | ip="${X_PREFIX}${suffix}" 32 | 33 | ip addr add "$ip/$CIDR" dev $DEV 34 | echo "$ip" >> "$OUTPUT_FILE" 35 | done -------------------------------------------------------------------------------- /scripts/vtbackup.init.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | apk add --no-cache aria2 bash ffmpeg python3 py3-pip perl build-base curl jq git nano exiv2-dev coreutils grep nodejs npm 3 | apk add exiv2 --no-cache --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community 4 | apk add --no-cache atomicparsley --repository=http://dl-cdn.alpinelinux.org/alpine/edge/testing 5 | pip3 install yq 6 | npm i -g pm2 --------------------------------------------------------------------------------