├── README.md ├── rclone.conf ├── rclone_mount ├── rclone_unmount └── rclone_upload /README.md: -------------------------------------------------------------------------------- 1 | # Rclone Mount & Upload Scripts for Plex Users 2 | 3 | Note: This project is no longer supported 4 | 5 | Collection of scripts to create rclone google mounts to allow fast launch times with Plex (or Emby). 6 | 7 | The main thread for more support: https://forums.unraid.net/topic/75436-guide-how-to-use-rclone-to-mount-cloud-drives-and-play-files/. 8 | 9 | Credits: 10 | 11 | Thanks to SenPaiBox and the Unraid community for help in refining the scripts. 12 | 13 | Unraid Users Requirements: 14 | 44 | Non-unRaid Users 45 |

46 | Other users need to install rclone and use their preferred way to schedule cron jobs. The scripts install mergerfs, which I think should work for other systems. 47 |

  • Optional: Create Service Accounts (follow steps 1-4).
    • 48 | 50 |

    51 | How It Works 52 |

      53 |
    1. Rclone is used to access files on your google drive and to mount them in a folder on your server e.g. mount a gdrive remote called gdrive_vfs: at /mnt/user/mount_rclone/gdrive_vfs
      2. 54 |
    2. Mergerfs is used to merge files from your rclone mount (/mnt/user/mount_rclone/gdrive_vfs) with local files that exist on your server and haven't been uploaded yet (e.g. /mnt/user/local/gdrive_vfs) in a new mount /mnt/user/mount_unionfs/gdrive_vfs
      3. 55 | 60 |
    3. An upload script is used to upload files in the background from the local folder to the remote. This activity is masked by mergerfs i.e. to plex, radarr etc files haven't 'moved'
      4. 61 |
    62 | Getting Started 63 |
      64 |
    1. Rclone remote setup
      2. 65 | 72 |

      73 |

    2. Once complete your rclone_config file should look something like this:
      3. 74 |

      75 | [gdrive] 76 |
      type = drive 77 |
      client_id = UNIQUE CLIENT_ID 78 |
      client_secret = MATCHING_UNIQUE_SECRET 79 |
      scope = drive 80 |
      root_folder_id = xxxx 81 |
      token = {"xxxxx"} 82 |
      server_side_across_configs = true 83 |

      84 | [gdrive_media_vfs] 85 |
      type = crypt 86 |
      remote = gdrive:crypt 87 |
      filename_encryption = standard 88 |
      directory_name_encryption = true 89 |
      password = PASSWORD1 90 |
      password2 = PASSWORD2 91 |

      92 |
    3. Or, like this if using service accounts:
      4. 93 |

      94 | 95 | 96 | [gdrive] 97 |
      type = drive 98 |
      scope = drive 99 |
      service_account_file = /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive.json 100 |
      team_drive = TEAM DRIVE ID 101 |
      server_side_across_configs = true 102 | 103 | [gdrive_media_vfs] 104 |
      type = crypt 105 |
      remote = gdrive:crypt 106 |
      filename_encryption = standard 107 |
      directory_name_encryption = true 108 |
      password = PASSWORD1 109 |
      password2 = PASSWORD2 110 |

      111 | 112 | 113 | If you need help doing this, please consult the forum thread above. 114 |

      115 | It is advisable to create your own client_id to avoid API bans. More Details 116 |

      117 |

    4. Mount script
      5. 118 | 130 |

      131 |
    5. Upload script
      6. 132 | 145 |

      146 |
    6. Cleanup script
      7. 147 | 150 |
    151 |

    152 | Using Mergerfs 153 |

    154 | Once the scripts are added you should have a new folder created at /mnt/user/mount_mergerfs/name_of_remote.  155 |

    165 | Troubleshooting 166 |

    167 | If you need to unmount manually the command to use is: 168 | 169 | fusermount -uz /path/to/remote 170 | -------------------------------------------------------------------------------- /rclone.conf: -------------------------------------------------------------------------------- 1 | # Sample rclone conf 2 | 3 | [gdrive] 4 | type = drive 5 | client_id = xxxxxxxx 6 | client_secret = xxxxxxxxxxxxx 7 | scope = drive 8 | token = {"access_token":"xxxxxxxxxxxxxxxxxx"} 9 | server_side_across_configs = true 10 | 11 | [gdrive_media_vfs] 12 | type = crypt 13 | remote = gdrive:crypt 14 | filename_encryption = standard 15 | directory_name_encryption = true 16 | password = xxxxxxxxxxxxxxx 17 | password2 = xxxxxxxxxxxxxxxxxxxxxxxxxxx 18 | -------------------------------------------------------------------------------- /rclone_mount: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ###################### 4 | #### Mount Script #### 5 | ###################### 6 | ## Version 0.96.9.3 ## 7 | ###################### 8 | 9 | ####### EDIT ONLY THESE SETTINGS ####### 10 | 11 | # INSTRUCTIONS 12 | # 1. Change the name of the rclone remote and shares to match your setup 13 | # 2. NOTE: enter RcloneRemoteName WITHOUT ':' 14 | # 3. Optional: include custom command and bind mount settings 15 | # 4. Optional: include extra folders in mergerfs mount 16 | 17 | # REQUIRED SETTINGS 18 | RcloneRemoteName="gdrive_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data 19 | RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone 20 | RcloneMountDirCacheTime="720h" # rclone dir cache time 21 | LocalFilesShare="/mnt/user/local" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable 22 | RcloneCacheShare="/mnt/user0/mount_rclone" # location of rclone cache files without trailing slash e.g. /mnt/user0/mount_rclone 23 | RcloneCacheMaxSize="400G" # Maximum size of rclone cache 24 | RcloneCacheMaxAge="336h" # Maximum age of cache files 25 | MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable 26 | DockerStart="nzbget plex sonarr radarr ombi" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page 27 | MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount 28 | 29 | # Note: Again - remember to NOT use ':' in your remote name above 30 | 31 | # OPTIONAL SETTINGS 32 | 33 | # Add extra paths to mergerfs mount in addition to LocalFilesShare 34 | LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable 35 | LocalFilesShare3="ignore" 36 | LocalFilesShare4="ignore" 37 | 38 | # Add extra commands or filters 39 | Command1="--rc" 40 | Command2="" 41 | Command3="" 42 | Command4="" 43 | Command5="" 44 | Command6="" 45 | Command7="" 46 | Command8="" 47 | 48 | CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter 49 | RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address 50 | NetworkAdapter="eth0" # choose your network adapter. eth0 recommended 51 | VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them 52 | 53 | ####### END SETTINGS ####### 54 | 55 | ############################################################################### 56 | ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### 57 | ############################################################################### 58 | 59 | ####### Preparing mount location variables ####### 60 | RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount 61 | LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount 62 | MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location 63 | 64 | ####### create directories for rclone mount and mergerfs mounts ####### 65 | mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files 66 | mkdir -p $RcloneCacheShare/cache/$RcloneRemoteName # for cache files 67 | if [[ $LocalFilesShare == 'ignore' ]]; then 68 | echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." 69 | LocalFilesLocation="/tmp/$RcloneRemoteName" 70 | eval mkdir -p $LocalFilesLocation 71 | else 72 | echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." 73 | eval mkdir -p $LocalFilesLocation/"$MountFolders" 74 | fi 75 | mkdir -p $RcloneMountLocation 76 | 77 | if [[ $MergerfsMountShare == 'ignore' ]]; then 78 | echo "$(date "+%d.%m.%Y %T") INFO: Not creating MergerFS folders as requested." 79 | else 80 | echo "$(date "+%d.%m.%Y %T") INFO: Creating MergerFS folders." 81 | mkdir -p $MergerFSMountLocation 82 | fi 83 | 84 | 85 | ####### Check if script is already running ####### 86 | echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" 87 | echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." 88 | if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then 89 | echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." 90 | exit 91 | else 92 | echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." 93 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 94 | fi 95 | 96 | ####### Checking have connectivity ####### 97 | 98 | echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" 99 | ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform 100 | if [ $? -eq 0 ]; then # ping returns exit status 0 if successful 101 | echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" 102 | else 103 | echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" 104 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 105 | exit 106 | fi 107 | 108 | ####### Create Rclone Mount ####### 109 | 110 | # Check If Rclone Mount Already Created 111 | if [[ -f "$RcloneMountLocation/mountcheck" ]]; then 112 | echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." 113 | else 114 | echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." 115 | # Creating mountcheck file in case it doesn't already exist 116 | echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." 117 | touch mountcheck 118 | rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse 119 | # Check bind option 120 | if [[ $CreateBindMount == 'Y' ]]; then 121 | echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" 122 | ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform 123 | if [ $? -eq 0 ]; then # ping returns exit status 0 if successful 124 | echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" 125 | else 126 | echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" 127 | ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber 128 | fi 129 | echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" 130 | else 131 | RCloneMountIP="" 132 | echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" 133 | fi 134 | # create rclone mount 135 | rclone mount \ 136 | $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ 137 | --allow-other \ 138 | --umask 000 \ 139 | --dir-cache-time $RcloneMountDirCacheTime \ 140 | --attr-timeout $RcloneMountDirCacheTime \ 141 | --log-level INFO \ 142 | --poll-interval 10s \ 143 | --cache-dir=$RcloneCacheShare/cache/$RcloneRemoteName \ 144 | --drive-pacer-min-sleep 10ms \ 145 | --drive-pacer-burst 1000 \ 146 | --vfs-cache-mode full \ 147 | --vfs-cache-max-size $RcloneCacheMaxSize \ 148 | --vfs-cache-max-age $RcloneCacheMaxAge \ 149 | --vfs-read-ahead 1G \ 150 | --bind=$RCloneMountIP \ 151 | $RcloneRemoteName: $RcloneMountLocation & 152 | 153 | # Check if Mount Successful 154 | echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" 155 | # slight pause to give mount time to finalise 156 | sleep 5 157 | echo "$(date "+%d.%m.%Y %T") INFO: continuing..." 158 | if [[ -f "$RcloneMountLocation/mountcheck" ]]; then 159 | echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." 160 | else 161 | echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" 162 | docker stop $DockerStart 163 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 164 | exit 165 | fi 166 | fi 167 | 168 | ####### Start MergerFS Mount ####### 169 | 170 | if [[ $MergerfsMountShare == 'ignore' ]]; then 171 | echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." 172 | else 173 | if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then 174 | echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." 175 | else 176 | # check if mergerfs already installed 177 | if [[ -f "/bin/mergerfs" ]]; then 178 | echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" 179 | else 180 | # Build mergerfs binary 181 | echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." 182 | mkdir -p /mnt/user/appdata/other/rclone/mergerfs 183 | docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build 184 | mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin 185 | # check if mergerfs install successful 186 | echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" 187 | sleep 5 188 | if [[ -f "/bin/mergerfs" ]]; then 189 | echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." 190 | else 191 | echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." 192 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 193 | exit 194 | fi 195 | fi 196 | # Create mergerfs mount 197 | echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." 198 | # Extra Mergerfs folders 199 | if [[ $LocalFilesShare2 != 'ignore' ]]; then 200 | echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." 201 | LocalFilesShare2=":$LocalFilesShare2" 202 | else 203 | LocalFilesShare2="" 204 | fi 205 | if [[ $LocalFilesShare3 != 'ignore' ]]; then 206 | echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." 207 | LocalFilesShare3=":$LocalFilesShare3" 208 | else 209 | LocalFilesShare3="" 210 | fi 211 | if [[ $LocalFilesShare4 != 'ignore' ]]; then 212 | echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." 213 | LocalFilesShare4=":$LocalFilesShare4" 214 | else 215 | LocalFilesShare4="" 216 | fi 217 | # make sure mergerfs mount point is empty 218 | mv $MergerFSMountLocation $LocalFilesLocation 219 | mkdir -p $MergerFSMountLocation 220 | # mergerfs mount command 221 | mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true 222 | # check if mergerfs mount successful 223 | echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." 224 | if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then 225 | echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." 226 | else 227 | echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." 228 | docker stop $DockerStart 229 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 230 | exit 231 | fi 232 | fi 233 | fi 234 | 235 | ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### 236 | 237 | # only start dockers once 238 | if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then 239 | echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." 240 | else 241 | # Check CA Appdata plugin not backing up or restoring 242 | if [ -f "/tmp/ca.backup2/tempFiles/backupInProgress" ] || [ -f "/tmp/ca.backup2/tempFiles/restoreInProgress" ] ; then 243 | echo "$(date "+%d.%m.%Y %T") INFO: Appdata Backup plugin running - not starting dockers." 244 | else 245 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started 246 | echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." 247 | docker start $DockerStart 248 | fi 249 | fi 250 | 251 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running 252 | echo "$(date "+%d.%m.%Y %T") INFO: Script complete" 253 | 254 | exit 255 | -------------------------------------------------------------------------------- /rclone_unmount: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ####################### 4 | ### Cleanup Script #### 5 | ####################### 6 | #### Version 0.9.2 #### 7 | ####################### 8 | 9 | echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***" 10 | 11 | ####### Cleanup Tracking Files ####### 12 | 13 | echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***" 14 | 15 | find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete 16 | find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete 17 | find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete 18 | echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***" 19 | 20 | exit 21 | -------------------------------------------------------------------------------- /rclone_upload: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ###################### 4 | ### Upload Script #### 5 | ###################### 6 | ### Version 0.95.5 ### 7 | ###################### 8 | 9 | ####### EDIT ONLY THESE SETTINGS ####### 10 | 11 | # INSTRUCTIONS 12 | # 1. Edit the settings below to match your setup 13 | # 2. NOTE: enter RcloneRemoteName WITHOUT ':' 14 | # 3. Optional: Add additional commands or filters 15 | # 4. Optional: Use bind mount settings for potential traffic shaping/monitoring 16 | # 5. Optional: Use service accounts in your upload remote 17 | # 6. Optional: Use backup directory for rclone sync jobs 18 | 19 | # REQUIRED SETTINGS 20 | RcloneCommand="move" # choose your rclone command e.g. move, copy, sync 21 | RcloneRemoteName="gdrive_vfs" # Name of rclone remote mount WITHOUT ':'. 22 | RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. 23 | LocalFilesShare="/mnt/user/local" # location of the local files without trailing slash you want to rclone to use 24 | RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone 25 | MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y 26 | ModSort="ascending" # "ascending" oldest files first, "descending" newest files first 27 | 28 | # Note: Again - remember to NOT use ':' in your remote name above 29 | 30 | # Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight. 31 | BWLimit1Time="01:00" 32 | BWLimit1="off" 33 | BWLimit2Time="08:00" 34 | BWLimit2="15M" 35 | BWLimit3Time="16:00" 36 | BWLimit3="12M" 37 | 38 | # OPTIONAL SETTINGS 39 | 40 | # Add name to upload job 41 | JobName="_daily_upload" # Adds custom string to end of checker file. Useful if you're running multiple jobs against the same remote. 42 | 43 | # Add extra commands or filters 44 | Command1="--exclude downloads/**" 45 | Command2="" 46 | Command3="" 47 | Command4="" 48 | Command5="" 49 | Command6="" 50 | Command7="" 51 | Command8="" 52 | 53 | # Bind the mount to an IP address 54 | CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter. 55 | RCloneMountIP="192.168.1.253" # Choose IP to bind upload to. 56 | NetworkAdapter="eth0" # choose your network adapter. eth0 recommended. 57 | VirtualIPNumber="1" # creates eth0:x e.g. eth0:1. 58 | 59 | # Use Service Accounts. Instructions: https://github.com/xyou365/AutoRclone 60 | UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts. 61 | ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files. 62 | ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload". 63 | CountServiceAccounts="15" # Integer number of service accounts to use. 64 | 65 | # Is this a backup job 66 | BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName 67 | BackupRemoteLocation="backup" # choose location on mount for deleted sync files 68 | BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files 69 | BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y 70 | 71 | ####### END SETTINGS ####### 72 | 73 | ############################################################################### 74 | ##### DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING ##### 75 | ############################################################################### 76 | 77 | ####### Preparing mount location variables ####### 78 | if [[ $BackupJob == 'Y' ]]; then 79 | LocalFilesLocation="$LocalFilesShare" 80 | echo "$(date "+%d.%m.%Y %T") INFO: *** Backup selected. Files will be copied or synced from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" 81 | else 82 | LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" 83 | echo "$(date "+%d.%m.%Y %T") INFO: *** Rclone move selected. Files will be moved from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" 84 | fi 85 | 86 | RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location of rclone mount 87 | 88 | ####### create directory for script files ####### 89 | mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName #for script files 90 | 91 | ####### Check if script already running ########## 92 | echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_upload script for ${RcloneUploadRemoteName} ***" 93 | if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName" ]]; then 94 | echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." 95 | exit 96 | else 97 | echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." 98 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName 99 | fi 100 | 101 | ####### check if rclone installed ########## 102 | echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully." 103 | if [[ -f "$RcloneMountLocation/mountcheck" ]]; then 104 | echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." 105 | else 106 | echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." 107 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName 108 | exit 109 | fi 110 | 111 | ####### Rotating serviceaccount.json file if using Service Accounts ####### 112 | if [[ $UseServiceAccountUpload == 'Y' ]]; then 113 | cd /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/ 114 | CounterNumber=$(find -name 'counter*' | cut -c 11,12) 115 | CounterCheck="1" 116 | if [[ "$CounterNumber" -ge "$CounterCheck" ]];then 117 | echo "$(date "+%d.%m.%Y %T") INFO: Counter file found for ${RcloneUploadRemoteName}." 118 | else 119 | echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneUploadRemoteName}. Creating counter_1." 120 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 121 | CounterNumber="1" 122 | fi 123 | ServiceAccount="--drive-service-account-file=$ServiceAccountDirectory/$ServiceAccountFile$CounterNumber.json" 124 | echo "$(date "+%d.%m.%Y %T") INFO: Adjusted service_account_file for upload remote ${RcloneUploadRemoteName} to ${ServiceAccountFile}${CounterNumber}.json based on counter ${CounterNumber}." 125 | else 126 | echo "$(date "+%d.%m.%Y %T") INFO: Uploading using upload remote ${RcloneUploadRemoteName}" 127 | ServiceAccount="" 128 | fi 129 | 130 | ####### Upload files ########## 131 | 132 | # Check bind option 133 | if [[ $CreateBindMount == 'Y' ]]; then 134 | echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" 135 | ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform 136 | if [ $? -eq 0 ]; then # ping returns exit status 0 if successful 137 | echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" 138 | else 139 | echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for upload to remote ${RcloneUploadRemoteName}" 140 | ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber 141 | fi 142 | else 143 | RCloneMountIP="" 144 | fi 145 | 146 | # Remove --delete-empty-src-dirs if rclone sync or copy 147 | if [[ $RcloneCommand == 'move' ]]; then 148 | echo "$(date "+%d.%m.%Y %T") INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload." 149 | DeleteEmpty="--delete-empty-src-dirs " 150 | else 151 | echo "$(date "+%d.%m.%Y %T") INFO: *** Not using rclone move - will remove --delete-empty-src-dirs to upload." 152 | DeleteEmpty="" 153 | fi 154 | 155 | # Check --backup-directory 156 | if [[ $BackupJob == 'Y' ]]; then 157 | echo "$(date "+%d.%m.%Y %T") INFO: *** Will backup to ${BackupRemoteLocation} and use ${BackupRemoteDeletedLocation} as --backup-directory with ${BackupRetention} retention for ${RcloneUploadRemoteName}." 158 | LocalFilesLocation="$LocalFilesShare" 159 | BackupDir="--backup-dir $RcloneUploadRemoteName:$BackupRemoteDeletedLocation" 160 | else 161 | BackupRemoteLocation="" 162 | BackupRemoteDeletedLocation="" 163 | BackupRetention="" 164 | BackupDir="" 165 | fi 166 | 167 | # process files 168 | rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount $BackupDir \ 169 | --user-agent="$RcloneUploadRemoteName" \ 170 | -vv \ 171 | --buffer-size 512M \ 172 | --drive-chunk-size 512M \ 173 | --tpslimit 8 \ 174 | --checkers 8 \ 175 | --transfers 4 \ 176 | --order-by modtime,$ModSort \ 177 | --min-age $MinimumAge \ 178 | $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ 179 | --exclude *fuse_hidden* \ 180 | --exclude *_HIDDEN \ 181 | --exclude .recycle** \ 182 | --exclude .Recycle.Bin/** \ 183 | --exclude *.backup~* \ 184 | --exclude *.partial~* \ 185 | --drive-stop-on-upload-limit \ 186 | --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \ 187 | --bind=$RCloneMountIP $DeleteEmpty 188 | 189 | # Delete old files from mount 190 | if [[ $BackupJob == 'Y' ]]; then 191 | echo "$(date "+%d.%m.%Y %T") INFO: *** Removing files older than ${BackupRetention} from $BackupRemoteLocation for ${RcloneUploadRemoteName}." 192 | rclone delete --min-age $BackupRetention $RcloneUploadRemoteName:$BackupRemoteDeletedLocation 193 | fi 194 | 195 | ####### Remove Control Files ########## 196 | 197 | # update counter and remove other control files 198 | if [[ $UseServiceAccountUpload == 'Y' ]]; then 199 | if [[ "$CounterNumber" == "$CountServiceAccounts" ]];then 200 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* 201 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 202 | echo "$(date "+%d.%m.%Y %T") INFO: Final counter used - resetting loop and created counter_1." 203 | else 204 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* 205 | CounterNumber=$((CounterNumber+1)) 206 | touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_$CounterNumber 207 | echo "$(date "+%d.%m.%Y %T") INFO: Created counter_${CounterNumber} for next upload run." 208 | fi 209 | else 210 | echo "$(date "+%d.%m.%Y %T") INFO: Not utilising service accounts." 211 | fi 212 | 213 | # remove dummy file 214 | rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName 215 | echo "$(date "+%d.%m.%Y %T") INFO: Script complete" 216 | 217 | exit 218 | --------------------------------------------------------------------------------