├── script.template ├── plist.template ├── install.sh └── README.md /script.template: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | {{UNISON_PATH}} -batch -ignore="Name {{{IGNORE_FILES}}}" "{{LOCAL_PATH}}" "{{CLOUD_PATH}}" 4 | -------------------------------------------------------------------------------- /plist.template: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Label 6 | {{LABEL}} 7 | ProgramArguments 8 | 9 | /bin/bash 10 | {{SCRIPT_PATH}} 11 | 12 | WatchPaths 13 | 14 | {{LOCAL_PATH}} 15 | {{CLOUD_PATH}} 16 | 17 | RunAtLoad 18 | 19 | StandardOutPath 20 | {{LOG_FILE}} 21 | StandardErrorPath 22 | {{ERR_FILE}} 23 | LowPriorityIO 24 | 25 | Nice 26 | 1 27 | 28 | 29 | -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # v1.0.0 4 | # 5 | # Usage: 6 | # - Call script to register sync script with launchd. 7 | # - Call with `--no-logs` to disable logging. 8 | # - Call with `--uninstall` or `--remove` to unregister from launchd and clean up files. 9 | 10 | # Adjust the paths to match your system (do not end the path with /). 11 | # Path to local (working) projects folder 12 | local_path="${HOME}/LocalDocs/Projects" 13 | 14 | # Path to cloud projects folder (node_modules, etc. are omitted). 15 | # 16 | # Note: if you're using iCloud on a system before Sierra, the Documents folder 17 | # can be found at "${HOME}/Library/Mobile Documents/com~apple~CloudDocs" 18 | cloud_path="${HOME}/Documents/Projects" 19 | 20 | # Comma-separated list of files to ignore. 21 | # Example: "node_modules,*.log" -> ignore all paths containing `node_modules` and any files ending with `*.log`. 22 | # For more details see: http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#ignore 23 | ignore_files="node_modules,bower_components,*.log,.DS_Store" 24 | 25 | # If you want, change log destination here (irellevant with --no-logs flag). 26 | log_file="/var/log/${label}.out.log" 27 | err_file="/var/log/${label}.err.log" 28 | 29 | ########################################################################## 30 | # No need to modify the code below, unless you know what you're doing :D # 31 | ########################################################################## 32 | 33 | # Path to script and launchd config. 34 | label="com.markogresak.projects.CloudSyncIgnore" 35 | script_path="/usr/local/bin/${label}.sh" 36 | plist_path="${HOME}/Library/LaunchAgents/${label}.plist" 37 | 38 | # If config already exists, unload it before updating it. 39 | if [ -f $plist_path ]; then 40 | launchctl unload $plist_path 41 | fi 42 | 43 | if [[ "$1" == "--uninstall" || "$1" == "--remove" ]]; then 44 | rm -f $script_path $plist_path 45 | if [ -f $log_file ] || [ -f $err_file ]; then 46 | echo "The script will attempt to remove log files. This requires sudo access, so the shell will ask you for password." 47 | sudo rm -f $log_file $err_file 48 | fi 49 | echo "Sync script successfully removed. Thanks for giving it a chance. If you have any suggestions for improvement, please let me know by submitting an issue." 50 | exit 51 | fi 52 | 53 | # Check for unison command and fail if not found. 54 | if ! command -v unison >/dev/null 2>&1; then 55 | echo "Command 'unison' not found. Install it (brew install unison) and try this script again." 56 | exit 1 57 | fi 58 | 59 | # If `--no-logs` flag is used, use /dev/null as stdout and stderr. 60 | if [[ "$1" == "--no-logs" ]]; then 61 | log_file="/dev/null" 62 | err_file="/dev/null" 63 | else 64 | echo "The script will attempt to create log files. This requires sudo access, so the shell will ask you for password." 65 | 66 | # Create/clear log files (requires sudo to allow modifying files in /var/log) and fix log file permissions. 67 | sudo sh -c 'echo "" > $0' "$log_file" 68 | sudo sh -c 'echo "" > $0' "$err_file" 69 | sudo chown `whoami` "$log_file" "$err_file" 70 | echo -e "Log files were successfully created.\n" 71 | fi 72 | 73 | # Create actual files based of .template files. 74 | sed "s|{{LOCAL_PATH}}|${local_path}|; 75 | s|{{CLOUD_PATH}}|${cloud_path}|; 76 | s|{{SCRIPT_PATH}}|${script_path}|; 77 | s|{{LABEL}}|${label}|; 78 | s|{{LOG_FILE}}|${log_file}|; 79 | s|{{ERR_FILE}}|${err_file}|" plist.template > $plist_path 80 | sed "s|{{UNISON_PATH}}|$(which unison)|; 81 | s|{{IGNORE_FILES}}|${ignore_files}|; 82 | s|{{LOCAL_PATH}}|${local_path}|; 83 | s|{{CLOUD_PATH}}|${cloud_path}|;" script.template > $script_path 84 | 85 | # Load launchd config. 86 | launchctl load $plist_path 87 | 88 | echo "Sync script added. It will be triggered any time any of files inside local or iCloud project folder changes." 89 | echo "I hope this script will help make your life a little easier :)" 90 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Cloud sync ignore files 2 | 3 | > Make Cloud sync ignore certain project files 4 | 5 | ### Why? 6 | 7 | I am using iCloud to share data between my macs, including my project files. I noticed that `node_modules` and similar project dependencies caused iCloud to take awfully long to sync thousands upon thousands of files, despite a fast internet connection. 8 | 9 | I was looking for a solution to prevent `node_modules` from uploading, but iCloud doesn't seem to offer ignoring files. So I started to put together this solution/hack. It's not iCloud-specific; you can use it with any service which doesn't provide selective sync or ignoring files. 10 | 11 | ### How it works? 12 | 13 | The script expects a local project directory, which outside the cloud synced folders. The project directory is in sync with a duplicate project folder inside the cloud drive, but this clone doesn't contain project dependencies. This setup makes the size of the clone smaller and more importantly, faster to sync, thanks to a lot fewer files. 14 | 15 | 16 | Syncing works both ways, so when new files are downloaded from the cloud, they are copied over to the local project directory. 17 | 18 | The syncing works via [unison](https://www.cis.upenn.edu/~bcpierce/unison/index.html) CLI tool. Initially, I tried with `rsync`, but it doesn't support a simple bidirectional sync setup. 19 | 20 | The sync script will ignore: 21 | - `node_modules` folder 22 | - `bower_components` folder 23 | - `*.log` files (rails apps can produce large logs and those are stored directly inside the project directory) 24 | - `.DS_Store` files 25 | 26 | You can change the ignored files by modifying the `ignore_files` variable in `install.sh`. With the current version, the setup is based on personal project types (mostly rails and node), but it's open to suggestions. 27 | 28 | ### Installing 29 | 30 | *Note*: This currently works only on macOS because of the use of `launchctl`, but the rest is platform-agnostic (limited by [`unison` support](https://github.com/bcpierce00/unison/wiki/Downloading-Unison)). It should not be a lot of work to swap in a platform-specific service manager (e.g. `systemctl`). But I am not an active Linux or Windows user, so I did not test the install script on other platforms. PRs welcome :smile: 31 | 32 | Due to stricter macOS security policy, you must ensure `bash` has the Full Disk Access permission 33 |
34 | _Thanks to @chrisblossom for pointing it out (see [#5](https://github.com/markogresak/cloud-ignore-files/issues/5#issuecomment-771240855))._ 35 | 36 | 1. Install [unison](https://www.cis.upenn.edu/~bcpierce/unison/download.html) CLI tool. The easiest way is `brew install unison`. 37 | 2. Clone or download this repository and `cd` into the folder. 38 | 3. Check `install.sh` script and edit paths to match your system setup. Check variables `local_path`, `cloud_path` and `ignore_files`, which can be found at the top of the script. 39 | 4. Run `./install.sh`. 40 | 41 | By default, the script is configured to add log files in `/var/log`. To do this, the script requires `sudo` access, so the script will ask for the password during installation. You can disable logging by running the install script as `./install.sh --no-logs`, which also skips the password prompt. 42 | 43 | ### Updating config 44 | 45 | If you make changes to `install.sh` or templates and want to update your config, just run `./install.sh` again and it will re-generate and reload config. The updated config should start working immediately. But `unison` might take a moment to sync, depending on the size of the synced folder. 46 | 47 | ### Uninstalling 48 | 49 | Run `./install.sh --uninstall` to unload and remove the config, remove the syncing script and log files. 50 | 51 | 52 | ### Performance 53 | 54 | As far as software goes, it all comes down to `unison` performance, which seems to be quite fast. On my MacBook Pro (13" retina, late 2013: 2.4 GHz CPU, 8gb RAM, 256gb SSD) with projects folder size of 2.5GB with ~125k files or 62k without counting `node_modules`, it took about 75s to init, i.e. clone the whole project folder. Real cases will probably see only a few files changed here and there, which should sync instantaneously. 55 | 56 | 57 | ### Related solutions 58 | 59 | - [iCloud-NoSync](https://github.com/tsdexter/iCloud-NoSync): An Automator utility to have iCloud Sync ignore an entire folder without losing access to the folder path 60 | 61 | ### Credits 62 | 63 | Thank you @tatums for the [rsyc-icloud-hack](https://github.com/tatums/rsyc-icloud-hack) project, it helped me a lot to shape this project! 64 | --------------------------------------------------------------------------------