├── 按本地文件列表删除文件--PHP ├── README.md └── delete_file.php ├── 统计目录大小--Python2.7 ├── README.MD └── count_directory_size.py ├── 列指定目录下的文件--Python2.7 ├── README.MD └── list_file.py ├── .gitignore ├── 将指定目录下的文件迁移到新的服务 ├── readme.md └── transfer_file.py ├── 删除指定目录下的文件--Python2.7 ├── README.MD └── delete_file.py ├── README.md ├── 指定目录缓存刷新--Python2.7 └── purges.py └── 下载指定目录文件--Python2.7 └── download_file_with_iter.py /按本地文件列表删除文件--PHP/README.md: -------------------------------------------------------------------------------- 1 | ## 使用说明 2 | 3 | 4 | 

 5 | $upyun = new upyun("bucket", "user", "password", false);
 6 | $upyun->delfile("del.txt"); 
 7 | 
 8 | 
 9 |
10 | 11 | bucket : 服务名 12 | user : 操作员账号 13 | pass : 操作员密码 14 | false : 是否进行异步删除，如果要进行异步删除则更改为 true 15 | del.txt : 需要删除的文件URI列表，一行一条URI 16 | 17 | * 运行脚本生成的 delsucc.txt 代表删除成功的文件，delfail.txt 代表删除失败的文件。 18 | -------------------------------------------------------------------------------- /统计目录大小--Python2.7/README.MD: -------------------------------------------------------------------------------- 1 | ####依赖: 2 | 3 | sudo pip install requests 4 | 5 | 6 | 7 | ####统计目录内文件大小的方法: 8 | 9 | ``` 10 | bucket = '' 11 | username = '' 12 | password = '' 13 | 14 | path = '' 15 | ``` 16 | 17 | 在脚本中填入上述参数。分别是服务名，操作员名，操作员密码，和待统计的路径。 18 | 19 | 然后 python count_directory_size.py 进行删除. 20 | 21 | 后台运行: 22 | 23 | 1.Centos/RedHad 24 | ``` 25 | yum -y install screen 26 | screen -S count_directory_size.py 27 | python count_directory_size.py 28 | ``` 29 | 30 | 2.Debian/Ubuntu 31 | 32 | ``` 33 | sudo apt-get install screen 34 | screen -S count_directory_size.py 35 | python count_directory_size.py 36 | ``` 37 | 38 | 通过screen在后台运行脚本的好处就是即使本地与远程ssh主机断开后，程序依然能够正常运行 39 | ``` 40 | ~$ screen -ls #查看当前后台运行的脚本 41 | 42 | $ screen -r test #进入已经创建的后台任务中 43 | ``` 44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /列指定目录下的文件--Python2.7/README.MD: -------------------------------------------------------------------------------- 1 | 在脚本内部填写: 2 | 3 | ``` 4 | # ----------------------- 5 | bucket = '' # 服务名 6 | username = '' # 操作员名 7 | password = '' # 操作员密码 8 | 9 | path = '' # 需要列文件列表的起始目录 10 | # ----------------------- 11 | ``` 12 | 13 | 填写无误后, 直接运行 14 | 15 | ``` 16 | python list_file_with_iter.py 17 | 18 | ``` 19 | 20 | 如果文件量巨大，也可以使用 screen 工具， 或者 nohup 将进程挂载。 21 | 22 | 比如： 23 | 24 | ``` 25 | nohup python list_file_with_iter.py & 26 | ``` 27 | 28 | 或者: 29 | 30 | 31 | 后台运行: 32 | 33 | 1.Centos/RedHad 34 | ``` 35 | yum -y install screen 36 | screen -S list_file_with_iter 37 | python list_file_with_iter.py 38 | ``` 39 | 40 | 2.Debian/Ubuntu 41 | 42 | ``` 43 | sudo apt-get install screen 44 | screen -S list_file_with_iter 45 | python list_file_with_iter.py 46 | ``` 47 | 48 | 查看 screen 49 | ``` 50 | $ screen -ls #查看当前后台运行的脚本 51 | 52 | $ screen -r xxxx #进入已经创建的后台任务中 53 | ``` -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *,cover 46 | .hypothesis/ 47 | .idea/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | 56 | # Sphinx documentation 57 | docs/_build/ 58 | 59 | # PyBuilder 60 | target/ 61 | 62 | #Ipython Notebook 63 | .ipynb_checkpoints 64 | -------------------------------------------------------------------------------- /将指定目录下的文件迁移到新的服务/readme.md: -------------------------------------------------------------------------------- 1 | #### 依赖 2 | 3 | 脚本依赖又拍云 SDK 4 | 5 | ``` 6 | pip install upyun 7 | ``` 8 | 9 | #### 参数说明 10 | 11 | 使用脚本之前, 需要在脚本第 11 行 和 24 行 填写一下相应的参数。 12 | 13 | 比如: 我要从 "bucket_one" 这个空间拉取 '/' 目录下面所有的文件, 访问域名为 techs.upyun.com。 然后保存到 bucket_two 的 "/bucket_one" 目录中去, 那我就需要填写如下参数 14 | 15 | ```python 16 | # ----------待拉取的服务名操作员信息------------- 17 | origin_bucket = 'buckeet_one' # (必填) 待拉取的服务名 18 | origin_username = 'username' # (必填) 待拉取的服务名下授权的操作员名 19 | origin_password = 'password' # (必填) 待拉取服务名下授权操作员的密码 20 | host = 'http://techs.upyun.com' # (必填) 待拉取的服务名的访问域名, 请使用 http// 或者 https:// 开头, 比如 'http://techs.upyun.com' 21 | origin_path = '/' # (必填) 待拉取的资源路径 (默认会拉取根目录下面的所有目录的文件) 22 | # -------------------------------------------- 23 | 24 | # ----------目标迁移服务名, 操作员信息------------- 25 | target_bucket = 'bucket_two' # (必填) 文件迁移的目标服务名 26 | target_username = 'username' # (必填) 文件迁移的目标服务名的授权操作员名 27 | target_password = 'password' # (必填) 文件迁移的目标服务名的授权操作员的密码 28 | save_as_prefix = '/bucket_one' # (选填) 目标服务名的保存路径的前置路径 (如果不填写, 默认迁移后的路径和原路径相同) 29 | # -------------------------------------------- 30 | ``` 31 | 32 | 33 | -------------------------------------------------------------------------------- /删除指定目录下的文件--Python2.7/README.MD: -------------------------------------------------------------------------------- 1 | ####依赖: 2 | 3 | sudo pip install requests 4 | 5 | 6 | ###视频教程 7 | 8 | https://techs.b0.upaiyun.com/videos/cdnpage/delete_file.html 9 | 10 | 11 | ####删除方法: 12 | 13 | ``` 14 | bucket = '' 15 | username = '' 16 | password = '' 17 | 18 | path = '' 19 | ``` 20 | 21 | 在脚本中填入上述参数。分别是服务名，操作员名，操作员密码，和待删除的路径。 22 | 23 | 然后 python delete_file.py 进行删除. 24 | 25 | 后台运行: 26 | 27 | 1.Centos/RedHad 28 | ``` 29 | yum -y install screen 30 | screen -S delete_upyun_file 31 | python delete_file.py 32 | ``` 33 | 34 | 2.Debian/Ubuntu 35 | 36 | ``` 37 | sudo apt-get install screen 38 | screen -S delete_upyun_file 39 | python delete_file.py 40 | ``` 41 | 42 | 通过screen在后台运行脚本的好处就是即使本地与远程ssh主机断开后，程序依然能够正常运行 43 | ``` 44 | ~$ screen -ls #查看当前后台运行的脚本 45 | There are screens on: 46 | 3003.test (03/24/2016 10:58:26 AM) (Attached) 47 | 2860.delete (03/24/2016 10:57:52 AM) (Detached) 48 | 6828.py (03/16/2016 08:50:22 PM) (Detached) 49 | 3 Sockets in /var/run/screen/S-fangwenjun. 50 | $ screen -r test #进入已经创建的后台任务中 51 | ``` 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## 脚本使用说明 2 | 3 | 填写 username bucket password 然后运行，如果提示需要安装python的包可以使用 pip install 去安装，例如 4 | 5 | 6 | 7 | $pip install requests 8 | 9 | 10 | ## 视频教程 11 | 12 | 1.删除文件： 13 | 14 | https://techs.b0.upaiyun.com/videos/cdnpage/delete_file.html 15 | 16 | 17 | 后台运行: 18 | 19 | 1.Centos/RedHad 20 | ``` 21 | yum -y install screen 22 | screen -S delete_upyun_file 23 | python delete_file.py 24 | ``` 25 | 26 | 2.Debian/Ubuntu 27 | 28 | ``` 29 | sudo apt-get install screen 30 | screen -S delete_upyun_file 31 | python delete_file.py 32 | ``` 33 | 34 | 通过screen在后台运行脚本的好处就是即使本地与远程ssh主机断开后，程序依然能够正常运行 35 | ``` 36 | ~$ screen -ls #查看当前后台运行的脚本 37 | There are screens on: 38 | 3003.test (03/24/2016 10:58:26 AM) (Attached) 39 | 2860.delete (03/24/2016 10:57:52 AM) (Detached) 40 | 6828.py (03/16/2016 08:50:22 PM) (Detached) 41 | 3 Sockets in /var/run/screen/S-fangwenjun. 42 | $ screen -r test #进入已经创建的后台任务中 43 | ``` 44 | 45 | 46 | ### 更新历史 47 | 48 | 2016-3-24： 49 | 50 | 1.增加用户交互功能，无需进入脚本输入参数 51 | 2.输入密码不可见 52 | 3.修复2处可能抛异常的地方 53 | 54 | 2017-07-12: 55 | 56 | 1. 因接口变动, headers['X-List-Limit'] 字段里面的值修改为str类型 57 | 58 | -------------------------------------------------------------------------------- /按本地文件列表删除文件--PHP/delete_file.php: -------------------------------------------------------------------------------- 1 | bucketname = $bucketname; 12 | $this->username = $username; 13 | $this->password = $password; 14 | $this->isasync = $isasync; 15 | 16 | } 17 | 18 | public function del($dstpath){ 19 | 20 | $uri = "/{$this->bucketname}$dstpath"; 21 | $date = gmdate('D, d M Y H:i:s \G\M\T'); 22 | $signature = base64_encode(hash_hmac("sha1", "DELETE&$uri&$date", md5("{$this->password}"), true)); 23 | $header = array("Authorization:UPYUN {$this->username}:$signature", "Date:$date", "x-upyun-async:{$this->isasync}"); 24 | 25 | $ch = curl_init(self::END_POINT.$uri); 26 | curl_setopt($ch, CURLOPT_HTTPHEADER, $header); 27 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 28 | curl_setopt($ch, CURLOPT_POST, 1); 29 | curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE"); 30 | $rsp_body = curl_exec($ch); 31 | $rsp_code = curl_getinfo($ch, CURLINFO_HTTP_CODE); 32 | curl_close($ch); 33 | return array("code"=>$rsp_code,"msg"=>$rsp_body); 34 | 35 | } 36 | 37 | public function delfile($path){ 38 | $fh = fopen($path,"rb") or die("Unable to open file."); 39 | while(!feof($fh)){ 40 | $fileline = fgets($fh); 41 | $fileline = str_replace("\n", "", $fileline); 42 | $resdel = self::del($fileline); 43 | if($resdel["code"] == 200){ 44 | file_put_contents("delsucc.txt",$fileline."\n", FILE_APPEND); 45 | echo "DELETE ".$fileline." SUCCESS\n"; 46 | }else{ 47 | file_put_contents("delfail.txt",$fileline."\ncode: ".$resdel["code"]." message: ".$resdel["msg"]."\n\n", FILE_APPEND); 48 | echo "DELETE ".$fileline." FAILED\n"; 49 | } 50 | } 51 | } 52 | 53 | } 54 | 55 | $upyun = new upyun("bucket", "user", "password", false); 56 | 57 | $upyun->delfile("del.txt"); 58 | 59 | 60 | 61 | ?> -------------------------------------------------------------------------------- /统计目录大小--Python2.7/count_directory_size.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | from __future__ import division 4 | from base64 import b64encode 5 | import requests 6 | import urllib 7 | import Queue 8 | 9 | # ----------------------- 10 | bucket = '' 11 | username = '' 12 | password = '' 13 | 14 | path = '/' 15 | # ----------------------- 16 | 17 | queue = Queue.LifoQueue() 18 | 19 | 20 | def do_http_request(method, key, upyun_iter): 21 | uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 22 | if isinstance(uri, unicode): 23 | uri = uri.encode('utf-8') 24 | uri = urllib.quote(uri) 25 | headers = { 26 | 'Authorization': 'Basic ' + b64encode(username + ':' + password), 27 | 'User-Agent': 'up-python-script', 28 | 'X-List-Limit': '300' 29 | } 30 | if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg': 31 | headers['x-list-iter'] = upyun_iter 32 | 33 | url = "http://v0.api.upyun.com" + uri 34 | requests.adapters.DEFAULT_RETRIES = 5 35 | session = requests.session() 36 | try: 37 | response = session.request(method, url, headers=headers, timeout=30) 38 | status = response.status_code 39 | if status == 200: 40 | content = response.content 41 | try: 42 | iter_header = response.headers['x-upyun-list-iter'] 43 | except Exception as e: 44 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 45 | data = { 46 | 'content': content, 47 | 'iter_header': iter_header 48 | } 49 | return data 50 | else: 51 | return None 52 | except Exception as e: 53 | return None 54 | 55 | 56 | def getlist(key, upyun_iter): 57 | result = do_http_request('GET', key, upyun_iter) 58 | if not result: 59 | return None 60 | content = result['content'] 61 | items = content.split('\n') 62 | content = [dict(zip(['name', 'type', 'size', 'time'], 63 | x.split('\t'))) for x in items] + result['iter_header'].split() 64 | return content 65 | 66 | 67 | def count_dir_size(path): 68 | upyun_iter = None 69 | size = 0 70 | while True: 71 | while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg': 72 | res = getlist(path, upyun_iter) 73 | if res: 74 | upyun_iter = res[-1] 75 | for i in res[:-1]: 76 | try: 77 | if not i['name']: 78 | continue 79 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 80 | if i['type'] == 'F': 81 | queue.put(new_path) 82 | elif i['type'] == 'N': 83 | print 'size ++ ----> {0} B'.format(size) 84 | size += int(i['size']) 85 | except Exception as e: 86 | print e 87 | else: 88 | if not queue.empty(): 89 | path = queue.get() 90 | upyun_iter = None 91 | queue.task_done() 92 | else: 93 | if not queue.empty(): 94 | path = queue.get() 95 | upyun_iter = None 96 | queue.task_done() 97 | else: 98 | break 99 | return size / 1024 / 1024 / 1024 100 | 101 | 102 | if __name__ == '__main__': 103 | size = count_dir_size(path) 104 | print "Job's Done!" 105 | print 'your path: "{0}" , total size: "{1}" GB'.format(path, size) 106 | -------------------------------------------------------------------------------- /列指定目录下的文件--Python2.7/list_file.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from base64 import b64encode 5 | import requests 6 | import urllib 7 | import Queue 8 | 9 | # ----------------------- 10 | bucket = '' 11 | username = '' 12 | password = '' 13 | 14 | path = '' 15 | # ----------------------- 16 | 17 | queue = Queue.LifoQueue() 18 | 19 | 20 | def record_request(url, status): 21 | if status: 22 | with open('file_list.txt', 'a') as file: 23 | file.write(url + '\n') 24 | else: 25 | with open('list_failed_path.txt', 'a') as failed_file: 26 | failed_file.write(url + '\n') 27 | 28 | 29 | def do_http_request(method, key, upyun_iter): 30 | uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 31 | if isinstance(uri, unicode): 32 | uri = uri.encode('utf-8') 33 | uri = urllib.quote(uri) 34 | headers = { 35 | 'Authorization': 'Basic ' + b64encode(username + ':' + password), 36 | 'User-Agent': 'up-python-script', 37 | 'X-List-Limit': '300' 38 | } 39 | if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg': 40 | headers['x-list-iter'] = upyun_iter 41 | 42 | url = "http://v0.api.upyun.com" + uri 43 | requests.adapters.DEFAULT_RETRIES = 5 44 | session = requests.session() 45 | try: 46 | response = session.request(method, url, headers=headers, timeout=30) 47 | status = response.status_code 48 | if status == 200: 49 | content = response.content 50 | try: 51 | iter_header = response.headers['x-upyun-list-iter'] 52 | except Exception as e: 53 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 54 | data = { 55 | 'content': content, 56 | 'iter_header': iter_header 57 | } 58 | return data 59 | else: 60 | record_request(uri, False) 61 | return None 62 | except Exception as e: 63 | record_request(uri, False) 64 | return None 65 | 66 | 67 | def getlist(key, upyun_iter): 68 | result = do_http_request('GET', key, upyun_iter) 69 | if not result: 70 | return None 71 | content = result['content'] 72 | items = content.split('\n') 73 | content = [dict(zip(['name', 'type', 'size', 'time'], 74 | x.split('\t'))) for x in items] + result['iter_header'].split() 75 | return content 76 | 77 | 78 | def print_file_with_iter(path): 79 | upyun_iter = None 80 | while True: 81 | while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg': 82 | res = getlist(path, upyun_iter) 83 | if res: 84 | upyun_iter = res[-1] 85 | for i in res[:-1]: 86 | try: 87 | if not i['name']: 88 | continue 89 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 90 | if i['type'] == 'F': 91 | queue.put(new_path) 92 | elif i['type'] == 'N': 93 | print new_path 94 | record_request(new_path, True) 95 | except Exception as e: 96 | print e 97 | else: 98 | if not queue.empty(): 99 | path = queue.get() 100 | upyun_iter = None 101 | queue.task_done() 102 | else: 103 | if not queue.empty(): 104 | path = queue.get() 105 | upyun_iter = None 106 | queue.task_done() 107 | else: 108 | break 109 | 110 | 111 | if __name__ == '__main__': 112 | print_file_with_iter(path) 113 | print "Job's Done!" 114 | -------------------------------------------------------------------------------- /删除指定目录下的文件--Python2.7/delete_file.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from base64 import b64encode 5 | import requests 6 | import urllib 7 | import Queue 8 | 9 | # ----------------------- 10 | bucket = '' 11 | username = '' 12 | password = '' 13 | 14 | path = '' 15 | # ----------------------- 16 | queue = Queue.LifoQueue() 17 | queue_list = Queue.LifoQueue() 18 | 19 | 20 | def record_request(url, status): 21 | if status: 22 | with open('deleted_file_list.txt', 'a') as file: 23 | file.write(url + '\n') 24 | else: 25 | with open('list_failed_path.txt', 'a') as failed_file: 26 | failed_file.write(url + '\n') 27 | 28 | 29 | def do_http_request(method, key, upyun_iter): 30 | uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 31 | if isinstance(uri, unicode): 32 | uri = uri.encode('utf-8') 33 | uri = urllib.quote(uri) 34 | headers = { 35 | 'Authorization': 'Basic ' + b64encode(username + ':' + password), 36 | 'User-Agent': 'up-python-delete-script', 37 | 'X-List-Limit': '300' 38 | } 39 | if method is not 'DELETE': 40 | if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg': 41 | headers['x-list-iter'] = upyun_iter 42 | 43 | url = "http://v0.api.upyun.com" + uri 44 | requests.adapters.DEFAULT_RETRIES = 5 45 | session = requests.session() 46 | try: 47 | response = session.request(method, url, headers=headers, timeout=30) 48 | status = response.status_code 49 | if status == 200 and method != 'DELETE': 50 | content = response.content 51 | try: 52 | iter_header = response.headers['x-upyun-list-iter'] 53 | except Exception as e: 54 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 55 | data = { 56 | 'content': content, 57 | 'iter_header': iter_header 58 | } 59 | return data 60 | elif status == 200 and method == 'DELETE': 61 | return True 62 | else: 63 | print 'status: ' + str(status) + '--->' + url 64 | record_request(uri, False) 65 | except Exception as e: 66 | record_request(uri, False) 67 | 68 | 69 | def getlist(key, upyun_iter): 70 | result = do_http_request('GET', key, upyun_iter) 71 | if not result: 72 | return None 73 | content = result['content'] 74 | items = content.split('\n') 75 | content = [dict(zip(['name', 'type', 'size', 'time'], 76 | x.split('\t'))) for x in items] + result['iter_header'].split() 77 | return content 78 | 79 | 80 | def list_file_with_iter(path): 81 | upyun_iter = None 82 | while True: 83 | while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg': 84 | res = getlist(path, upyun_iter) 85 | if res: 86 | upyun_iter = res[-1] 87 | for i in res[:-1]: 88 | try: 89 | if not i['name']: 90 | if delete_file(path): 91 | print 'folder deleted' + path 92 | continue 93 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 94 | if i['type'] == 'F': 95 | queue.put(new_path) 96 | queue_list.put(new_path) 97 | elif i['type'] == 'N': 98 | result = delete_file(new_path) 99 | print 'file deleted--->' + new_path 100 | record_request(new_path, True) 101 | except Exception as e: 102 | print e 103 | else: 104 | if not queue.empty(): 105 | path = queue.get() 106 | upyun_iter = None 107 | queue.task_done() 108 | else: 109 | if not queue.empty(): 110 | path = queue.get() 111 | upyun_iter = None 112 | queue.task_done() 113 | else: 114 | while not queue_list.empty(): 115 | delete_file(queue_list.get()) 116 | queue_list.task_done() 117 | break 118 | 119 | 120 | def delete_file(key): 121 | return do_http_request('DELETE', key, None) 122 | 123 | 124 | if __name__ == '__main__': 125 | list_file_with_iter(path) 126 | print "Job's Done!" 127 | -------------------------------------------------------------------------------- /指定目录缓存刷新--Python2.7/purges.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 说明 4 | # 使用此脚本需要安装 requests 库，使用pip install requests==2.10.0 安装 5 | # 具体的使用教程，请参考 https://awen.me/archives/801.html 说明 6 | 7 | from base64 import b64encode 8 | import requests 9 | import urllib 10 | import Queue 11 | import hashlib 12 | import datetime 13 | import getpass 14 | 15 | # ----------------------- 16 | bucket = raw_input("Please enter your serverName:") 17 | username = raw_input("Please enter your userName:") 18 | password = getpass.getpass("Plaser enter your Password:") 19 | # ----------------------- 20 | 21 | queue = Queue.LifoQueue() 22 | count = 0 23 | 24 | 25 | def httpdate_rfc1123(dt): 26 | """Return a string representation of a date according to RFC 1123 27 | (HTTP/1.1). 28 | 29 | The supplied date must be in UTC. 30 | """ 31 | weekday = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"][dt.weekday()] 32 | month = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", 33 | "Oct", "Nov", "Dec"][dt.month - 1] 34 | return "%s, %02d %s %04d %02d:%02d:%02d GMT" % \ 35 | (weekday, dt.day, month, dt.year, dt.hour, dt.minute, dt.second) 36 | 37 | 38 | def run(purge): 39 | date = httpdate_rfc1123(datetime.datetime.utcnow()) 40 | token = hashlib.md5(password).hexdigest() 41 | sign = hashlib.md5(purge + "&" + bucket + "&" + 42 | date + "&" + token).hexdigest() 43 | 44 | Header = { 45 | "Authorization": 'UpYun ' + bucket + ':' + username + ':' + sign, 46 | "Date": date, 47 | "Content-Type": "application/x-www-form-urlencoded", 48 | } 49 | post = urllib.urlencode({'purge': purge}) 50 | r = requests.post("http://purge.upyun.com/purge/", post, headers=Header) 51 | return r.status_code 52 | 53 | 54 | def do_http_request(method, key, upyun_iter): 55 | uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 56 | if isinstance(uri, unicode): 57 | uri = uri.encode('utf-8') 58 | uri = urllib.quote(uri) 59 | headers = {} 60 | headers['Authorization'] = "Basic " + b64encode(username + ':' + password) 61 | headers['User-Agent'] = "uptechs" 62 | headers['X-List-Limit'] = '300' 63 | if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg': 64 | headers['x-list-iter'] = upyun_iter 65 | 66 | url = "http://v0.api.upyun.com" + uri 67 | requests.adapters.DEFAULT_RETRIES = 5 68 | session = requests.session() 69 | try: 70 | response = session.request(method, url, headers=headers, timeout=30) 71 | status = response.status_code 72 | if status == 200: 73 | content = response.content 74 | try: 75 | iter_header = response.headers['x-upyun-list-iter'] 76 | except Exception as e: 77 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 78 | return content + "`" + str(iter_header) 79 | else: 80 | return None 81 | except Exception as e: 82 | pass 83 | 84 | 85 | def getlist(key, upyun_iter): 86 | content = do_http_request('GET', key, upyun_iter) 87 | if not content: 88 | return None 89 | content = content.split("`") 90 | items = content[0].split('\n') 91 | content = [dict(zip(['name', 'type', 'size', 'time'], 92 | x.split('\t'))) for x in items] + content[1].split() 93 | return content 94 | 95 | 96 | def print_file_with_iter(path): 97 | upyun_iter = None 98 | while True: 99 | while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg': 100 | res = getlist(path, upyun_iter) 101 | if res: 102 | upyun_iter = res[-1] 103 | for i in res[:-1]: 104 | try: 105 | if not i['name']: 106 | continue 107 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 108 | if i['type'] == 'F': 109 | queue.put(new_path) 110 | elif i['type'] == 'N': 111 | url = 'http://' + bucket + '.b0.upaiyun.com' + new_path 112 | if run(url) == 200: 113 | print url+' ---> purge success' 114 | 115 | except Exception as e: 116 | print e 117 | else: 118 | if not queue.empty(): 119 | path = queue.get() 120 | upyun_iter = None 121 | queue.task_done() 122 | else: 123 | break 124 | 125 | 126 | if __name__ == '__main__': 127 | path = raw_input("input path:") 128 | print_file_with_iter(path) 129 | print '刷新全部完成' 130 | -------------------------------------------------------------------------------- /将指定目录下的文件迁移到新的服务/transfer_file.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # author: support@upyun.com 4 | 5 | from base64 import b64encode 6 | import requests 7 | import upyun 8 | import urllib 9 | import Queue 10 | 11 | # ----------待拉取的服务名操作员信息------------- 12 | origin_bucket = '' # (必填) 待拉取的服务名 13 | origin_username = '' # (必填) 待拉取的服务名下授权的操作员名 14 | origin_password = '' # (必填) 待拉取服务名下授权操作员的密码 15 | host = '' # (必填) 待拉取的服务名的访问域名, 请使用 http// 或者 https:// 开头, 比如 'http://techs.upyun.com' 16 | origin_path = '/' # (必填) 待拉取的资源路径 (默认会拉取根目录下面的所有目录的文件) 17 | # -------------------------------------------- 18 | 19 | # ----------目标迁移服务名, 操作员信息------------- 20 | target_bucket = '' # (必填) 文件迁移的目标服务名 21 | target_username = '' # (必填) 文件迁移的目标服务名的授权操作员名 22 | target_password = '' # (必填) 文件迁移的目标服务名的授权操作员的密码 23 | save_as_prefix = '' # (选填) 目标服务名的保存路径的前置路径 (如果不填写, 默认迁移后的路径和原路径相同) 24 | # -------------------------------------------- 25 | 26 | notify_url = 'http://your_notify_url' # 将回调地址改成自己的服务器地址, 用来接收又拍云 POST 过来的异步拉取结果 27 | 28 | # -------------------------------------------- 29 | 30 | 31 | queue = Queue.LifoQueue() 32 | 33 | 34 | def push_tasks(url, up): 35 | fetch_data = [ 36 | { 37 | 'url': host + url, # 需要拉取文件的 URL 38 | 'random': False, # 是否追加随机数, 默认 false 39 | 'overwrite': True, # 是否覆盖，默认 True 40 | 'save_as': url 41 | } 42 | ] 43 | 44 | result = up.put_tasks(fetch_data, notify_url, 'spiderman') 45 | return result 46 | 47 | 48 | def do_http_request(method, key, upyun_iter): 49 | uri = '/' + origin_bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 50 | if isinstance(uri, unicode): 51 | uri = uri.encode('utf-8') 52 | uri = urllib.quote(uri) 53 | headers = { 54 | 'Authorization': 'Basic ' + b64encode(origin_username + ':' + origin_password), 55 | 'User-Agent': 'up-python-script', 56 | 'X-List-Limit': '300' 57 | } 58 | if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg': 59 | headers['x-list-iter'] = upyun_iter 60 | 61 | url = "http://v0.api.upyun.com" + uri 62 | requests.adapters.DEFAULT_RETRIES = 5 63 | session = requests.session() 64 | try: 65 | response = session.request(method, url, headers=headers, timeout=30) 66 | status = response.status_code 67 | if status == 200: 68 | content = response.content 69 | try: 70 | iter_header = response.headers['x-upyun-list-iter'] 71 | except Exception as e: 72 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 73 | data = { 74 | 'content': content, 75 | 'iter_header': iter_header 76 | } 77 | return data 78 | else: 79 | return None 80 | except Exception as e: 81 | return None 82 | 83 | 84 | def sort_data(key, upyun_iter): 85 | result = do_http_request('GET', key, upyun_iter) 86 | if not result: 87 | return None 88 | content = result['content'] 89 | items = content.split('\n') 90 | content = [dict(zip(['name', 'type', 'size', 'time'], 91 | x.split('\t'))) for x in items] + result['iter_header'].split() 92 | return content 93 | 94 | 95 | def get_list(path): 96 | upyun_iter = None 97 | up = upyun.UpYun(target_bucket, target_username, target_password) 98 | while True: 99 | while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg': 100 | res = sort_data(path, upyun_iter) 101 | if res: 102 | upyun_iter = res[-1] 103 | for i in res[:-1]: 104 | try: 105 | if not i['name']: 106 | continue 107 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 108 | if i['type'] == 'F': 109 | queue.put(new_path) 110 | elif i['type'] == 'N': 111 | print new_path 112 | if save_as_prefix: 113 | new_path = save_as_prefix + new_path 114 | push_tasks(new_path, up) 115 | except Exception as e: 116 | print e 117 | else: 118 | if not queue.empty(): 119 | path = queue.get() 120 | upyun_iter = None 121 | queue.task_done() 122 | else: 123 | if not queue.empty(): 124 | path = queue.get() 125 | upyun_iter = None 126 | queue.task_done() 127 | else: 128 | break 129 | 130 | 131 | if __name__ == '__main__': 132 | get_list(origin_path) 133 | -------------------------------------------------------------------------------- /下载指定目录文件--Python2.7/download_file_with_iter.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | 5 | from base64 import b64encode 6 | import requests 7 | import os 8 | import urllib 9 | import time 10 | import getpass 11 | 12 | # ----------------------- 13 | bucket = raw_input("Please enter your serverName:") 14 | username = raw_input("Please enter your userName:") 15 | password = getpass.getpass("Plaser enter your Password:") 16 | # ----------------------- 17 | 18 | count = 0 19 | requests_count = 0 20 | 21 | 22 | def do_http_request(method, key, params, of): 23 | chunk_size = 8192 24 | global requests_count 25 | uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key) 26 | if isinstance(uri, unicode): 27 | uri = uri.encode('utf-8') 28 | uri = urllib.quote(uri) 29 | 30 | headers = {} 31 | headers['Authorization'] = 'Basic ' + b64encode(username + ':' + password) 32 | headers['User-Agent'] = "UPYUN_DOWNLOAD_SCRIPT" 33 | if params is not None: 34 | if params['x-list-iter'] is not None or not 'g2gCZAAEbmV4dGQAA2VvZg': 35 | headers['X-List-Iter'] = params['x-list-iter'] 36 | headers['x-list-limit'] = '300' 37 | 38 | URL = "http://v0.api.upyun.com" + uri 39 | requests.adapters.DEFAULT_RETRIES = 5 40 | session = requests.session() 41 | response = session.request(method, URL, headers=headers, timeout=30) 42 | status = response.status_code 43 | if status == 200: 44 | if method == 'GET' and of: 45 | readsofar = 0 46 | try: 47 | total_size = int(response.headers['content-length']) 48 | except (KeyError, TypeError): 49 | total_size = 0 50 | 51 | hdr = None 52 | for chunk in response.iter_content(chunk_size): 53 | if chunk and hdr: 54 | readsofar += len(chunk) 55 | if readsofar != total_size: 56 | hdr.update(readsofar) 57 | else: 58 | hdr.finish() 59 | if not chunk: 60 | break 61 | of.write(chunk) 62 | return True 63 | elif method == 'GET' and of is None: 64 | 65 | content = response.content 66 | try: 67 | iter_header = response.headers['x-upyun-list-iter'] 68 | except Exception as e: 69 | iter_header = 'g2gCZAAEbmV4dGQAA2VvZg' 70 | return content + "`" + str(status) + "`" + str(iter_header) 71 | elif method == 'HEAD': 72 | return response.headers['content-length'] 73 | else: 74 | print 'status: ' + str(status) 75 | if requests_count == 4: 76 | requests_count = 0 77 | with open('download_error.txt', 'a') as f: 78 | f.write(uri + '\n') 79 | else: 80 | requests_count += 1 81 | time.sleep(1) 82 | do_http_request('GET', key, params, of=None) 83 | 84 | 85 | def getlist(key, params): 86 | content = do_http_request('GET', key, params, of=None) 87 | if content: 88 | content = content.split("`") 89 | items = content[0].split('\n') 90 | content = [dict(zip(['name', 'type', 'size', 'time'], 91 | x.split('\t'))) for x in items] + content[1].split() + content[2].split() 92 | return content 93 | else: 94 | return None 95 | 96 | 97 | def download_file(path, params): 98 | global count 99 | res = getlist(path, params) 100 | for i in res[:-2]: 101 | 102 | if not i['name']: 103 | continue 104 | new_path = path + i['name'] if path == '/' else path + '/' + i['name'] 105 | try: 106 | if i['type'] == 'F': 107 | download_file_with_iter(new_path) 108 | else: 109 | try: 110 | if not os.path.exists(bucket + path): 111 | os.makedirs(bucket + path) 112 | except OSError as e: 113 | print 'something wrong when mkdir: ' + str(e) 114 | save_path = os.getcwd() + '/' + bucket + new_path 115 | content_length = do_http_request('HEAD', new_path, params=None, of=None) 116 | if not os.path.isfile(save_path) or os.path.getsize(save_path) == 0 or int(os.path.getsize( 117 | save_path)) != int(content_length): 118 | with open(save_path, 'wb') as f: 119 | download_result = do_http_request('GET', new_path, params=None, of=f) 120 | if download_result: 121 | count += 1 122 | print str(count) + '---->' + save_path 123 | else: 124 | count += 1 125 | print str(count) + '----> file already downloaded' 126 | except Exception as e: 127 | print str(e) 128 | with open('download_error.txt', 'a') as f: 129 | f.write(new_path + '\n') 130 | if res[-1] != 'g2gCZAAEbmV4dGQAA2VvZg': 131 | params = { 132 | 'x-list-iter': res[-1] 133 | } 134 | download_file(path, params) 135 | 136 | 137 | def download_file_with_iter(path): 138 | params = { 139 | 'x-list-iter': None 140 | } 141 | return download_file(path, params) 142 | 143 | 144 | if __name__ == '__main__': 145 | if len(str.strip(bucket)) == 0 or len(str.strip(username)) == 0 or len(str.strip(password)) == 0: 146 | print "401 buket or username and password is null" 147 | else: 148 | path = raw_input('input a path( e.g: "/" ) : ') 149 | while len(path) == 0: 150 | path = raw_input('input a path( e.g: "/" ) : ') 151 | download_file_with_iter(path) 152 | print str(count) + ':Done!' 153 | --------------------------------------------------------------------------------