├── .gitignore ├── README.md ├── config ├── config_init ├── fcomp ├── fcopy ├── filter ├── fmove ├── foldercopy ├── folderlist ├── fsync ├── listcopy ├── listsets ├── listshcut ├── sets └── tv_example ├── shcut ├── src_dest_folders └── src_dest_tv_example └── src_folders └── src_tv_example /.gitignore: -------------------------------------------------------------------------------- 1 | archive/* 2 | backup/* 3 | sets/* 4 | src_folders/* 5 | src_dest_folders/* 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### Foldercopy - quick copy utility for Google Team/Shared Drives with string matches, using rclone (or lclone or gclone) 2 | 3 | - `foldercopy` copies folders matching any `filter` 4 | 5 | - foldercopy also copies shortcuts for folders using `./shcut set_name filter` 6 | 7 | NOTE: `foldercopy` requires a local list of folders. Create a set in `/sets/` and run `./folderlist set_name` to generate / update your list(s). The file is stored in `/src_folders` and can be updated as often as needed. 8 | 9 | 10 | Syntax: 11 | - `./folderlist tv` generates a file listing all folders in the source remotes in the set file `tv`. Subsequent searches are instant and copying can begin without re-scanning the remote. 12 | - `./foldercopy tv Bananas` will copy any folders that include the string `Bananas` (ignore caps) between folders defined in the set file called `tv` 13 | - `./foldercopy tv Bananas 2007` will copy any folders that include the strings `Bananas` and `2007` 14 | - `./foldercopy -d my_remote:comedy tv Bananas` will copy any folders that include the string `Bananas` to the `my_remote:comedy` folder 15 | - flags must come before the set name. `./foldercopy tv Bananas -d my_remote:comedy` will NOT work 16 | 17 | - After running a command foldercopy will give you a list and a choice 18 | - `y` to copy/sync/move all folders listed. 19 | - `n` to abort 20 | - `s` to select one of the folders shown in the list 21 | - `c` to change or refine the filter 22 | 23 | - Folder `sets` map the source and destination pairs in files in the `/sets` folder. These pairs can be as simple or complex as you like. For example a file `/sets/tv` could contain 24 | ``` 25 | my_td:video bak_td:video 26 | my_td:documentary bak_td:documentary 27 | my_td:nature bak_td:nature 28 | ``` 29 | - `./folderlist tv` will generate a list of all folders in my_td:video , documentary and nature as well as a mapping of all folders to bak_td:. These lists are saved in `/src_folders` and `src_dest_folders` directories. 30 | - There is a `config` file with a number of basic settings for default folders, actions, etc. 31 | - Many settings use flags, and some have short command alternatives. Most short commands accept flags: 32 | ``` 33 | FLAG OR COMMAND 34 | -c|--copy) fcopy => copy (This is the default action) 35 | -m|--move) fmove => move 36 | -s|--sync) fsync => sync 37 | -z|--shcut) shcut => create shortcuts of the source folders 38 | fcomp => compare two folder lists 39 | listsets => shows all of your set files. Select to print contentss 40 | -r|--rclone => use rclone (This is the default app for copying) 41 | -l|--lclone) => use lclone 42 | -g|--gclone) => use gclone 43 | -a|--auto) => copy without user confirmation 44 | -t|--test) => print command with executing 45 | -d|--dest) => set destination manually (e.g. -d remote:folder) 46 | -f|--flag) => add a flag to rclone. Each flag requires -f: `-f --flag1 -f flag2` 47 | -u|--update) => update folderlist, then run filter and other commands 48 | ``` 49 | - `foldercopy` works with rclone, gclone and lclone (l3uddz's rclone_gclone fork). This can be set in the `config` file or with flags 50 | - `./listcopy 51 | - `./fcomp set1 set2` creates a file in src_folders listing folders in set1 that are not in set2. `fcomp` will then ask if you want to copy (or sync or move) those folders. 52 | 53 | 54 | HINTS: 55 | - searches can use partial and multiple filters 56 | - e.g You have a folderlist called `videos` which includes a folders called /holiday/2014/italy and /holiday/2015/japan 57 | - `./fcopy videos hol 14` will find `/holiday/2014/italy` 58 | - `./fcopy videos 201` will find `/holiday/2014/italy` and `/holiday/2015/japan` 59 | - `./fcopy videos 2014 holiday` will find nothing, as the holiday precedes 2014 in the string 60 | -------------------------------------------------------------------------------- /config: -------------------------------------------------------------------------------- 1 | clone="lclone" 2 | action="copy" 3 | set_folder="sets" 4 | src_folders="src_folders" 5 | src_dest_folders="src_dest_folders" 6 | flags="--drive-server-side-across-configs --transfers=8 --tpslimit=5 --tpslimit-burst=32 -vP --fast-list" 7 | 8 | 9 | 10 | ### folder and parameter init 11 | IFS1="|" 12 | mkdir -p $src_folders $src_dest_folders $set_folder backup/$(date +%u)/ 13 | cp -r *_folders backup/$(date +%u)/ 14 | src_all="$src_folders/src_$1" 15 | src_dest_all="$src_dest_folders/src_dest_$1" 16 | -------------------------------------------------------------------------------- /config_init: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | source config 3 | for arg in "$@";do 4 | case $arg in 5 | -r|--rclone) clone="rclone"; shift ;; 6 | -l|--lclone) clone="lclone"; shift ;; 7 | -g|--gclone) clone="gclone"; shift ;; 8 | -c|--copy) action="copy" ; shift ;; 9 | -m|--move) action="move" ; shift ;; 10 | -s|--sync) action="sync" ; shift ;; 11 | -a|--auto) auto=true ; shift ;; 12 | -t|--test) test="echo " ; shift ;; 13 | -d|--dest) dest_="$2" ; shift 2 ;; 14 | -f|--flag) flags+=" $2 " ; shift 2 ;; 15 | -z|--shcut) action="backend shortcut";shift ;; 16 | -u|--update) update=true ; shift ;; 17 | esac 18 | done 19 | set_name="$1" 20 | args="$@" 21 | -------------------------------------------------------------------------------- /fcomp: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # syntax `./fcomp setname1 setname2` creates a differential file in src_folders 3 | source config_init $@ 4 | set $args 5 | outfile="$src_folders"/folders_in_"$1"_not_in_"$2" 6 | comm -23 "$src_folders"/src_"$1" "$src_folders"/src_"$2" | tee $outfile 7 | 8 | read -r -p "$action these folders? y/n " resp 9 | if [ $resp != "y" ];then 10 | echo "Exiting fcomp";exit 11 | else 12 | source listcopy "$outfile" "$1" 13 | fi -------------------------------------------------------------------------------- /fcopy: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source foldercopy -c "$@" -------------------------------------------------------------------------------- /filter: -------------------------------------------------------------------------------- 1 | - */Specials/** 2 | - */Season ** 3 | -------------------------------------------------------------------------------- /fmove: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source foldercopy -m "$@" -------------------------------------------------------------------------------- /foldercopy: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source config_init $@ 4 | set $args 5 | if [ $test ];then echo "===RUNNING IN TEST MODE===";fi 6 | infile="$src_folders/src_$1" 7 | infile2="$src_dest_folders/src_dest_$1" 8 | shift 9 | raw_filter="$@" 10 | if [ ! -z $update ];then source folderlist $set_name;fi 11 | 12 | check_auto () 13 | { if [ -z $auto ];then 14 | read -r -p "Run $action for all folders (y); select one (s); change filter (c); exit (e)? y/s/c/e: " resp 15 | if [ $resp == "s" ];then OLDIFS=$IFS;IFS=$'\n' 16 | select choice in $(echo -e "$src_list");do 17 | raw_filter="$choice" 18 | IFS=$OLDIFS 19 | filter_list 20 | break 21 | done 22 | elif [ $resp == "c" ];then 23 | read -r -p "Change filter to: " raw_filter 24 | filter_list 25 | elif [ $resp == "y" ];then echo 26 | else exit;fi 27 | fi 28 | } 29 | 30 | filter_list () 31 | { filter=`echo "$raw_filter" | sed -r 's/ /.*/g'` 32 | src_list=`cat $infile | grep -i -E "$filter"` 33 | if [ -z "$src_list" ];then 34 | echo -e "The filter \"$raw_filter\" does not match any folders in $infile. Try again" 35 | else 36 | echo -e "The filter matches:\n$src_list\n\n`echo "$src_list" | wc -l` folder(s) match filter\n" 37 | check_auto 38 | fi 39 | } 40 | 41 | run_foldercopy () 42 | { cat $infile2 | grep -i -E "$filter" |\ 43 | while IFS=$IFS1 read src dest; do 44 | if [ ! -z "$src" ];then 45 | if [ ! -z "$dest_" ];then dest="$dest_/${src/*\//}";fi 46 | if [ "$action" == "backend shortcut" ];then 47 | src1=${src#*:};src="${src%:*}:" 48 | dest1=${dest#*:};dest="-o target=${dest%:*}:";fi 49 | if [ "$action" == "backend shortcut" ];then 50 | set -x;$test "$clone" $action "$src" "$src1" $dest "$dest1" $flags;set +x 51 | else 52 | set -x;$test "$clone" $action "$src" "$dest" $flags;set +x 53 | fi 54 | fi 55 | done 56 | } 57 | 58 | filter_list 59 | run_foldercopy -------------------------------------------------------------------------------- /folderlist: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | source config $@ 3 | >$src_all;>$src_dest_all 4 | 5 | cat "$set_folder/$1" | while read src dest; do 6 | if [ ! -z "$src" ];then 7 | src_norm=`echo $src | sed -E 's/[^[:alnum:]]/_/g'` 8 | outfile="$src_folders/src_$src_norm";>$outfile 9 | echo -e "Creating list of source ($src) folders" 10 | rclone tree --noindent --noreport --full-path -d --filter-from filter "$src" |\ 11 | sort|uniq|tee $outfile -a $src_all >/dev/null 12 | if [ ! -z "$dest" ];then 13 | echo -e "Creating list of source ($src) and dest ($dest) folders" 14 | cat $outfile | while read folder; do 15 | echo -e "$src$folder$IFS1$dest$folder" >> $src_dest_all 16 | done 17 | fi 18 | rm $outfile 19 | fi 20 | done 21 | 22 | for i in `ls $src_folders`;do sort -o "$src_folders/$i" "$src_folders/$i";done 23 | for j in `ls $src_dest_folders`;do sort -o "$src_dest_folders/$j" "$src_dest_folders/$j";done 24 | echo -e "`cat $src_all|wc -l` folders in $src_all set" 25 | -------------------------------------------------------------------------------- /fsync: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source foldercopy -s "$@" -------------------------------------------------------------------------------- /listcopy: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # syntax: `./listcopy filter_file set_file` copies folder that match filters 3 | # in filter_file from source to destination in the set_file 4 | filter_file="$1" 5 | shift 6 | listcopy_args="$@" 7 | while IFS= read -r filter;do 8 | source foldercopy -a "$listcopy_args" $filter 9 | done <$filter_file -------------------------------------------------------------------------------- /listsets: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | source config_init 3 | echo -e "\nChoose set to see source-destination folder pairs\n" 4 | select choice in `ls sets`;do 5 | cat sets/$choice 6 | exit 7 | done -------------------------------------------------------------------------------- /listshcut: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # syntax: `./listshcut filter_file set_file` copies a shortcut of folders that match filters 3 | # in filter_file from source to destination in the set_file 4 | filter_file="$1" 5 | shift 6 | listcopy_args="$@" 7 | while IFS= read -r filter;do 8 | source shcut -a "$listcopy_args" $filter 9 | done <$filter_file -------------------------------------------------------------------------------- /sets/tv_example: -------------------------------------------------------------------------------- 1 | my_td:video bak_td:video 2 | -------------------------------------------------------------------------------- /shcut: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source foldercopy --shcut "$@" -------------------------------------------------------------------------------- /src_dest_folders/src_dest_tv_example: -------------------------------------------------------------------------------- 1 | my_td:video/First Birthday|bak_td:video/First Birthday 2 | my_td:video/Second Birthday|bak_td:video/Second Birthday 3 | my_td:video/Third Birthday|bak_td:video/Third Birthday 4 | -------------------------------------------------------------------------------- /src_folders/src_tv_example: -------------------------------------------------------------------------------- 1 | /First Birthday 2 | /Second Birthday 3 | /Third Birthday 4 | --------------------------------------------------------------------------------