├── C2C_Transfer.ipynb
├── Images
├── rc1.png
├── rc10.png
├── rc11.png
├── rc12.png
├── rc13.png
├── rc14.png
├── rc15.png
├── rc16.png
├── rc17.png
├── rc2.png
├── rc21.png
├── rc22.png
├── rc3-2.png
├── rc3.png
├── rc31.png
├── rc4-edit.png
├── rc5-edit.png
├── rc6.png
├── rc7.png
├── rc8.png
├── rc9.png
└── rcsdrive.png
├── LICENSE
├── README.md
└── Tutorial.md
/C2C_Transfer.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "C2C Transfer.ipynb",
7 | "provenance": [],
8 | "collapsed_sections": [],
9 | "include_colab_link": true
10 | },
11 | "kernelspec": {
12 | "name": "python3",
13 | "display_name": "Python 3"
14 | }
15 | },
16 | "cells": [
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {
20 | "id": "view-in-github",
21 | "colab_type": "text"
22 | },
23 | "source": [
24 | ""
25 | ]
26 | },
27 | {
28 | "cell_type": "code",
29 | "metadata": {
30 | "id": "V_b5UJR6i-2Q",
31 | "colab_type": "code",
32 | "colab": {}
33 | },
34 | "source": [
35 | "#*************************************** README **********************************************\n",
36 | "# !ipython notebook\n",
37 | "#\n",
38 | "# C2C Transfer\n",
39 | "# [C]loud 2 [C]loud Transfer script for GoogleColab \n",
40 | "# script by Lawliet_D2C \"Gokup\"--gokupxxxxx@xxx.com--https://github.com/gokupistol\n",
41 | "# Version: 1.0\n",
42 | "# Created: 09-24-2019 13:30\n",
43 | "# Last Modified: 10-25-2019 17:30\n",
44 | "#\n",
45 | "#*************************************** NOTES ***********************************************\n",
46 | "#\n",
47 | "# To run this you need to have rclone.conf in Google Drive at /content/drive/My Drive/Config/\n",
48 | "# To install and configure Rclone please visit my github repository.\n",
49 | "#\n",
50 | "#**************************************** ***** **********************************************"
51 | ],
52 | "execution_count": 0,
53 | "outputs": []
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {
58 | "id": "Q3N_96KhkFzq",
59 | "colab_type": "text"
60 | },
61 | "source": [
62 | "**Mount Google Drive**\n",
63 | "1. Run the code and open the link shown in output window.\n",
64 | "2. Sign in to your google drive account.\n",
65 | "3. Copy and Paste authorization code in white box and press Enter.\n"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "metadata": {
71 | "id": "SwSWEDyM_3pf",
72 | "colab_type": "code",
73 | "colab": {}
74 | },
75 | "source": [
76 | "from google.colab import drive\n",
77 | "drive.mount('/content/drive')"
78 | ],
79 | "execution_count": 0,
80 | "outputs": []
81 | },
82 | {
83 | "cell_type": "markdown",
84 | "metadata": {
85 | "id": "aohyHFO7nBfi",
86 | "colab_type": "text"
87 | },
88 | "source": [
89 | "**Install Rclone on Runtime Cloud** \n",
90 | "\n",
91 | "* Simply Run following code\n"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "metadata": {
97 | "id": "s6mySd-lAiur",
98 | "colab_type": "code",
99 | "colab": {}
100 | },
101 | "source": [
102 | "!curl https://rclone.org/install.sh | sudo bash"
103 | ],
104 | "execution_count": 0,
105 | "outputs": []
106 | },
107 | {
108 | "cell_type": "markdown",
109 | "metadata": {
110 | "id": "8xC0seIVnj1T",
111 | "colab_type": "text"
112 | },
113 | "source": [
114 | "**Copy and replace rclone config file**\n",
115 | "* Run following code to get the path and run next to replace rclone.conf of Colab with our configuration file.\n",
116 | "\n"
117 | ]
118 | },
119 | {
120 | "cell_type": "code",
121 | "metadata": {
122 | "id": "WNhBjFxk93ov",
123 | "colab_type": "code",
124 | "colab": {}
125 | },
126 | "source": [
127 | "!rclone config file"
128 | ],
129 | "execution_count": 0,
130 | "outputs": []
131 | },
132 | {
133 | "cell_type": "code",
134 | "metadata": {
135 | "id": "OdxdhW8Z1xTs",
136 | "colab_type": "code",
137 | "colab": {}
138 | },
139 | "source": [
140 | "!cp /content/drive/My\\ Drive/Config/rclone.conf /root/.config/rclone/"
141 | ],
142 | "execution_count": 0,
143 | "outputs": []
144 | },
145 | {
146 | "cell_type": "markdown",
147 | "metadata": {
148 | "id": "u0R4Sb7toT-I",
149 | "colab_type": "text"
150 | },
151 | "source": [
152 | "**Check wheather it is working properly:**\n",
153 | "\n",
154 | "\n",
155 | "* This will give list of cloud storage configured in rclone.conf\n"
156 | ]
157 | },
158 | {
159 | "cell_type": "code",
160 | "metadata": {
161 | "id": "nQo659P8A6PC",
162 | "colab_type": "code",
163 | "colab": {}
164 | },
165 | "source": [
166 | "!rclone listremotes"
167 | ],
168 | "execution_count": 0,
169 | "outputs": []
170 | },
171 | {
172 | "cell_type": "code",
173 | "metadata": {
174 | "id": "kBDCPTpZmCXF",
175 | "colab_type": "code",
176 | "colab": {}
177 | },
178 | "source": [
179 | "#List directories in remote:\n",
180 | "!rclone lsd gdrive:"
181 | ],
182 | "execution_count": 0,
183 | "outputs": []
184 | },
185 | {
186 | "cell_type": "markdown",
187 | "metadata": {
188 | "id": "f__n_Kwso0Cb",
189 | "colab_type": "text"
190 | },
191 | "source": [
192 | "**Run next command to Copy data from one clud to another**\n",
193 | "\n",
194 | "\n",
195 | "* Replace \"gdrive\" with your source remote name and Shared with path of folder you want to transfer. \n",
196 | "* Replace \"onedrive\" with your destination remote name and Backup with path of folder to where you want to transfer.\n",
197 | "\n",
198 | "\n",
199 | "---\n",
200 | "\n",
201 | "\n",
202 | "\n",
203 | "Flags details:\n",
204 | "\n",
205 | "* --transfers=N : Number of file transfers to run in parallel. \n",
206 | "* --checkers=N : Number of checkers to run in parallel, they do the equality checking of files during a sync.\n",
207 | "* --stats=5s --stats-one-line --stats-log-level NOTICE : To display stats of progress.\n",
208 | "* -v or --verbose : Will tell you about each file that is transferred.\n",
209 | "\n",
210 | "Note: Try to change number of transfers and checkers if transfer rate is slow.\n",
211 | "\n",
212 | "For more info visit: https://rclone.org/docs/\n",
213 | "\n",
214 | "Examples:\n",
215 | "\n",
216 | "* `!rclone copy remote1:path/to/folder/or/file remote2:path/to/folder -v`\n",
217 | "\n",
218 | "* `!rclone copy remote1:path/to/folder/or/file remote2:path/to/folder --transfers=32 --checkers=32 --stats=5s --stats-one-line --stats-log-level NOTICE`\n",
219 | "\n"
220 | ]
221 | },
222 | {
223 | "cell_type": "code",
224 | "metadata": {
225 | "id": "yUaBZ1m3A_dU",
226 | "colab_type": "code",
227 | "colab": {}
228 | },
229 | "source": [
230 | "# rclone copy\n",
231 | "!rclone copy gdrive:'path_to_folder' onedrive:'path_to_folder' --transfers=32 --checkers=32 --stats=5s --stats-one-line --stats-log-level NOTICE"
232 | ],
233 | "execution_count": 0,
234 | "outputs": []
235 | },
236 | {
237 | "cell_type": "markdown",
238 | "metadata": {
239 | "id": "JMJ4T9hsrvTp",
240 | "colab_type": "text"
241 | },
242 | "source": [
243 | "**Sync data from one clud to another**\n",
244 | "\n",
245 | "\n",
246 | "\n",
247 | "`!rclone sync remote1:path remote2:path [flags]`\n",
248 | "\n",
249 | "The following command sync entire onedrive to dropbox's backup folder.\n",
250 | "\n",
251 | "\n",
252 | "\n",
253 | "---\n",
254 | "\n",
255 | "\n",
256 | "\n",
257 | "**Note: Transfer Speed ∝ Number of files and Size of files.**\n",
258 | "\n",
259 | "If you have many small files, speed may be slow -> try increasing trackers and checkers.\n",
260 | "If you have less number of files but huge in size then -> try reducing them.\n",
261 | "\n",
262 | "---\n",
263 | "\n",
264 | "You can then use team folders like this remote:/TeamFolder and remote:/TeamFolder/path/to/file.\n"
265 | ]
266 | },
267 | {
268 | "cell_type": "code",
269 | "metadata": {
270 | "id": "3uEWSrgqs1zC",
271 | "colab_type": "code",
272 | "colab": {}
273 | },
274 | "source": [
275 | "# rclone sync\n",
276 | "!rclone sync gdrive: dropbox: --transfers=32 --checkers=32 --stats=10s --stats-one-line --stats-log-level NOTICE"
277 | ],
278 | "execution_count": 0,
279 | "outputs": []
280 | },
281 | {
282 | "cell_type": "markdown",
283 | "metadata": {
284 | "id": "O5A5x7d1VSTb",
285 | "colab_type": "text"
286 | },
287 | "source": [
288 | "**Example to download direct links to cloud storage**"
289 | ]
290 | },
291 | {
292 | "cell_type": "code",
293 | "metadata": {
294 | "id": "l57OoLysVRy5",
295 | "colab_type": "code",
296 | "colab": {}
297 | },
298 | "source": [
299 | "# rclone copyurl\n",
300 | "!rclone copyurl 'https://cdn-xxx.com/xxx/url/hits.txt' gdrive:'Download/hits.txt' -v"
301 | ],
302 | "execution_count": 0,
303 | "outputs": []
304 | },
305 | {
306 | "cell_type": "markdown",
307 | "metadata": {
308 | "id": "PPCE_B0AWFve",
309 | "colab_type": "text"
310 | },
311 | "source": [
312 | "Example to move data from google drive to shared drive"
313 | ]
314 | },
315 | {
316 | "cell_type": "code",
317 | "metadata": {
318 | "id": "mXg425ZoWFDf",
319 | "colab_type": "code",
320 | "colab": {}
321 | },
322 | "source": [
323 | "#rclone move\n",
324 | "!rclone move ccsfdrive:asp dropbox:asp --transfers=32 --checkers=32 --stats=60s --stats-one-line --stats-log-level NOTICE"
325 | ],
326 | "execution_count": 0,
327 | "outputs": []
328 | },
329 | {
330 | "cell_type": "markdown",
331 | "metadata": {
332 | "id": "cNSLOBEhsDTa",
333 | "colab_type": "text"
334 | },
335 | "source": [
336 | "**Some useful rclone commands:**\n",
337 | "\n",
338 | "* rclone config - Enter an interactive configuration session.\n",
339 | "* rclone copy - Copy files from source to dest, skipping already copied.\n",
340 | "* rclone sync - Make source and dest identical, modifying destination only.\n",
341 | "* rclone move - Move files from source to dest.\n",
342 | "* rclone delete - Remove the contents of path.\n",
343 | "* rclone purge - Remove the path and all of its contents.\n",
344 | "* rclone mkdir - Make the path if it doesn’t already exist.\n",
345 | "* rclone rmdir - Remove the path.\n",
346 | "* rclone rmdirs - Remove any empty directories under the path.\n",
347 | "* rclone check - Check if the files in the source and destination match.\n",
348 | "* rclone ls - List all the objects in the path with size and path.\n",
349 | "* rclone lsd - List all directories/containers/buckets in the path.\n",
350 | "* rclone lsl - List all the objects in the path with size, modification time and path.\n",
351 | "* rclone about - Get quota information from the remote.\n",
352 | "\n",
353 | "For more command info visit: https://rclone.org/docs/"
354 | ]
355 | }
356 | ]
357 | }
--------------------------------------------------------------------------------
/Images/rc1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc1.png
--------------------------------------------------------------------------------
/Images/rc10.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc10.png
--------------------------------------------------------------------------------
/Images/rc11.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc11.png
--------------------------------------------------------------------------------
/Images/rc12.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc12.png
--------------------------------------------------------------------------------
/Images/rc13.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc13.png
--------------------------------------------------------------------------------
/Images/rc14.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc14.png
--------------------------------------------------------------------------------
/Images/rc15.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc15.png
--------------------------------------------------------------------------------
/Images/rc16.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc16.png
--------------------------------------------------------------------------------
/Images/rc17.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc17.png
--------------------------------------------------------------------------------
/Images/rc2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc2.png
--------------------------------------------------------------------------------
/Images/rc21.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc21.png
--------------------------------------------------------------------------------
/Images/rc22.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc22.png
--------------------------------------------------------------------------------
/Images/rc3-2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc3-2.png
--------------------------------------------------------------------------------
/Images/rc3.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc3.png
--------------------------------------------------------------------------------
/Images/rc31.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc31.png
--------------------------------------------------------------------------------
/Images/rc4-edit.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc4-edit.png
--------------------------------------------------------------------------------
/Images/rc5-edit.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc5-edit.png
--------------------------------------------------------------------------------
/Images/rc6.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc6.png
--------------------------------------------------------------------------------
/Images/rc7.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc7.png
--------------------------------------------------------------------------------
/Images/rc8.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc8.png
--------------------------------------------------------------------------------
/Images/rc9.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rc9.png
--------------------------------------------------------------------------------
/Images/rcsdrive.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nguyenanhtuan1008/c2c-transfer/b4e568a4332fb546f3351ae66258d399266bbeda/Images/rcsdrive.png
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2019 Lawliet L
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Cloud to Cloud Transfer and Backup
2 | A Simple Google Colab Notebook to Transfer and Backup Data from One Cloud Storage to Another Using Rclone
3 |
4 | **Best way to sync clouds, create multiple backup and transfer.**
5 |
6 | [](https://colab.research.google.com/github/gokupistol/c2c-transfer/blob/master/C2C_Transfer.ipynb)
7 |
8 | ### Tutorial
9 |
10 | Follow the [How to install and configure rclone on local machine](/Tutorial.md)
11 |
12 | How to Use Colab Notebook:
13 |
14 | 0. Copy rclone.conf in google drive as mention in above tutorial.
15 | 1. Click the badge which says 'Open in Colab'.
16 | 2. Goto **File > Save a copy in Drive...** (a new tab opens with the copy of this notebook).
17 | 3. Run the whole notebook in sequence.
18 | 4. Follow directions there.
19 |
--------------------------------------------------------------------------------
/Tutorial.md:
--------------------------------------------------------------------------------
1 |