├── README.md ├── geeAssetMigratorExample.py ├── .gitignore └── geeAssetMigratorLib.py /README.md: -------------------------------------------------------------------------------- 1 | # geeMigrator 2 | > Method to facilitate migration of GEE assets from one account to another 3 | * Recursively migrates all assets under a user-specified asset to another user-specified asset 4 | * Handles migrating between two different accounts and: 5 | * Authentication to different accounts 6 | * Setting access-control list (ACL) permissions (who can read and write to a given asset folder/imageCollection) 7 | 8 | ## Primary POCs 9 | * Primary setup contact (contact for any issues with setting up the migration example and running it): 10 | * Josh Heyer- joshua.heyer@usda.gov 11 | 12 | * Primary authors (contact if any major bugs are found or feature requests): 13 | * Ian Housman- ian.housman@usda.gov 14 | * Leah Campbell- leah.campbell@usda.gov 15 | 16 | ## Dependencies 17 | * Python 3 18 | * earthengine-api 19 | 20 | ## Using 21 | * Ensure you have Python 3 installed 22 | * 23 | 24 | * Ensure the Google Earth Engine api is installed and up-to-date 25 | * `pip install earthengine-api --upgrade` 26 | * `conda update -c conda-forge earthengine-api` 27 | 28 | * Clone or download this repository 29 | * Recommended: `git clone https://github.com/gee-community/geeMigrator` 30 | * If downloading, download .zip and unzip the file. 31 | 32 | * Running script 33 | * Open `geeAssetMigratorExample.py` script and update the following variables: 34 | * Set the specified `sourceRoot` and `destinationRoot` to the asset level you would like to copy assets from and to (these must already exist) 35 | * E.g. if you would like to migrate all assets under a personal user account: `sourceRoot = 'users/fromUsername' and destinationRoot = 'users/toUsername'` 36 | * E.g. if you would like to migrate some assets under a personal user account: `sourceRoot = 'users/fromUsername/someFolder' and destinationRoot = 'users/toUsername/someFolder'` 37 | * E.g. if you would like to migrate some assets under a legacy project to a GCP project: `sourceRoot = 'projects/someProject/someFolder' and destinationRoot = 'projects/someGCPProject/someFolder'` 38 | * Updating the ACL permissions (who can read or write to a given asset folder or imageCollection). Source assets and destination assets can have their permissions updated. This is essential if the destination assets fall under a different account than the source assets. If this is the case, you must add the user, group, service account, or project as a writer to the source assets by listing it under the `sourceWriters` list. 39 | 40 | * E.g. `sourceWriters = ['user:destinationEmail@domain.com']` 41 | * The basic workflow within the script is: 42 | * `setupCredentialsForMigration` - Ensure tokens are available that have access to both the `sourceRoot` and `destinationRoot` 43 | * Tokens will be created if they do not exist by prompting the user to log into the appropriate account 44 | * The ee.Authenticate method will be used. This opens a login screen in your browser. You must log into the account that has access to the appropriate root folder, copy the token into the command prompt and press ENTER. 45 | * This process will run until a token can be found that has access to both the `sourceRoot` and `destinationRoot`. If this requires different tokens, it will automatically be handled. 46 | * `batchChangePermissions` - Add the destination account listed under `sourceWriters` to all assets under the `sourceRoot` 47 | * `copyAssetTree` - Copy all assets under the `sourceRoot` to the `destinationRoot` and update destination assets ACLs if specified under `changeDestinationPermissions` 48 | 49 | * DANGER!!!! Optional method to use is `deleteAssetTree`. This method will delete all assets under the specified asset level. This cannot be undone, but is useful for cleaning up any asset tree. 50 | * E.g. if `migrate.deleteAssetTree('users/fromUsername/someFolder')` is specified, all folders, imageCollections, images, and tables under `users/fromUsername/someFolder` will be deleted. 51 | * Run `geeAssetMigratorExample.py` script 52 | -------------------------------------------------------------------------------- /geeAssetMigratorExample.py: -------------------------------------------------------------------------------- 1 | import geeAssetMigratorLib as migrate 2 | ################################################################################################### 3 | # This script recursively migrates Google Earth Engine assets from one repository to a new one, sets permissions, and deletes old assets if needed. 4 | ################################################################################################### 5 | """ 6 | Copyright 2020 Ian Housman and Leah Campbell 7 | 8 | Licensed under the Apache License, Version 2.0 (the "License"); 9 | you may not use this file except in compliance with the License. 10 | You may obtain a copy of the License at 11 | 12 | http://www.apache.org/licenses/LICENSE-2.0 13 | 14 | Unless required by applicable law or agreed to in writing, software 15 | distributed under the License is distributed on an "AS IS" BASIS, 16 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | See the License for the specific language governing permissions and 18 | limitations under the License. 19 | """ 20 | ################################################################################################### 21 | ################################################################################################### 22 | #Set up tokens for migrating 23 | #The method handles migrating assets from one account to another and sets up two different credentials to take care of 24 | #any permission conflicts 25 | #It will also work if you are migrating within a single account 26 | #Specify the directory to store the tokens (will be created if it doesn't exist) 27 | #Set to None if you want to use the default credentials dir in c:\Users\userName\.config\earthengine 28 | token_dir = None#'c:/tokens3' 29 | 30 | # Repository you are moving from: 31 | sourceRoot = 'users/iwhousman' 32 | 33 | 34 | # Repository you are moving to: 35 | #Will be created if it does not exist (will default to a FOLDER if it does not exist) 36 | # destinationRoot = 'users/usfs-ihousman' 37 | destinationRoot = 'projects/USFS/migrationTest' 38 | 39 | # If the credential you're using does not have editor permissions in the source repository, add them under sourceWriters 40 | # Must include 'user:' prefix if it is a individual's Email or 'group:' if it is a Google Group 41 | sourceReaders = [] 42 | sourceWriters = ['user:ian.housman@usda.gov'] 43 | source_all_users_can_read = False 44 | 45 | #Optionally, update the permissions of the copied destination assets 46 | changeDestinationPermissions = False 47 | destinationReaders = [] 48 | destinationWriters = ['user:iwhousman@gmail.com'] 49 | destination_all_users_can_read = False 50 | ################################################################################################### 51 | #Step 1: set up credentials for the source and destination 52 | source_token, destination_token = migrate.setupCredentialsForMigration(token_dir,sourceRoot,destinationRoot,overwrite = False) 53 | 54 | #Step 2: Initialize GEE using the source token and update permissions on the source folder 55 | migrate.initializeFromToken(source_token) 56 | migrate.batchChangePermissions(None, sourceRoot,sourceReaders,sourceWriters, source_all_users_can_read) 57 | 58 | #Step 3: Initialize GEE using the destination token and copy folders 59 | migrate.initializeFromToken(destination_token) 60 | migrate.copyAssetTree(sourceRoot,destinationRoot,changeDestinationPermissions,destinationReaders,destinationWriters,destination_all_users_can_read) 61 | ################################################################################################### 62 | ################################################################################################### 63 | ################################################################################################### 64 | # !!!!!!!!!!!! DANGER !!!!!!!!!!!! 65 | # !!!!!!!!!!!! DANGER !!!!!!!!!!!! 66 | # !!!!!!!!!!!! DANGER !!!!!!!!!!!! 67 | # !!!!!!!!!!!! DANGER !!!!!!!!!!!! 68 | #Delete folder tree if you would like to clean up any asset trees 69 | # migrate.initializeFromToken(source_token) 70 | # migrate.deleteAssetTree(sourceRoot) 71 | 72 | 73 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | 4 | # User-specific files 5 | *.suo 6 | *.user 7 | *.userosscache 8 | *.sln.docstates 9 | 10 | # User-specific files (MonoDevelop/Xamarin Studio) 11 | *.userprefs 12 | 13 | # Build results 14 | [Dd]ebug/ 15 | [Dd]ebugPublic/ 16 | [Rr]elease/ 17 | [Rr]eleases/ 18 | x64/ 19 | x86/ 20 | bld/ 21 | [Bb]in/ 22 | [Oo]bj/ 23 | [Ll]og/ 24 | 25 | # Visual Studio 2015 cache/options directory 26 | .vs/ 27 | # Uncomment if you have tasks that create the project's static files in wwwroot 28 | #wwwroot/ 29 | 30 | # MSTest test Results 31 | [Tt]est[Rr]esult*/ 32 | [Bb]uild[Ll]og.* 33 | 34 | # NUNIT 35 | *.VisualState.xml 36 | TestResult.xml 37 | 38 | # Build Results of an ATL Project 39 | [Dd]ebugPS/ 40 | [Rr]eleasePS/ 41 | dlldata.c 42 | 43 | # DNX 44 | project.lock.json 45 | project.fragment.lock.json 46 | artifacts/ 47 | 48 | *_i.c 49 | *_p.c 50 | *_i.h 51 | *.ilk 52 | *.meta 53 | *.obj 54 | *.pch 55 | *.pdb 56 | *.pgc 57 | *.pgd 58 | *.rsp 59 | *.sbr 60 | *.tlb 61 | *.tli 62 | *.tlh 63 | *.tmp 64 | *.tmp_proj 65 | *.log 66 | *.vspscc 67 | *.vssscc 68 | .builds 69 | *.pidb 70 | *.svclog 71 | *.scc 72 | 73 | # Chutzpah Test files 74 | _Chutzpah* 75 | 76 | # Visual C++ cache files 77 | ipch/ 78 | *.aps 79 | *.ncb 80 | *.opendb 81 | *.opensdf 82 | *.sdf 83 | *.cachefile 84 | *.VC.db 85 | *.VC.VC.opendb 86 | 87 | # Visual Studio profiler 88 | *.psess 89 | *.vsp 90 | *.vspx 91 | *.sap 92 | 93 | # TFS 2012 Local Workspace 94 | $tf/ 95 | 96 | # Guidance Automation Toolkit 97 | *.gpState 98 | 99 | # ReSharper is a .NET coding add-in 100 | _ReSharper*/ 101 | *.[Rr]e[Ss]harper 102 | *.DotSettings.user 103 | 104 | # JustCode is a .NET coding add-in 105 | .JustCode 106 | 107 | # TeamCity is a build add-in 108 | _TeamCity* 109 | 110 | # DotCover is a Code Coverage Tool 111 | *.dotCover 112 | 113 | # NCrunch 114 | _NCrunch_* 115 | .*crunch*.local.xml 116 | nCrunchTemp_* 117 | 118 | # MightyMoose 119 | *.mm.* 120 | AutoTest.Net/ 121 | 122 | # Web workbench (sass) 123 | .sass-cache/ 124 | 125 | # Installshield output folder 126 | [Ee]xpress/ 127 | 128 | # DocProject is a documentation generator add-in 129 | DocProject/buildhelp/ 130 | DocProject/Help/*.HxT 131 | DocProject/Help/*.HxC 132 | DocProject/Help/*.hhc 133 | DocProject/Help/*.hhk 134 | DocProject/Help/*.hhp 135 | DocProject/Help/Html2 136 | DocProject/Help/html 137 | 138 | # Click-Once directory 139 | publish/ 140 | 141 | # Publish Web Output 142 | *.[Pp]ublish.xml 143 | *.azurePubxml 144 | # TODO: Comment the next line if you want to checkin your web deploy settings 145 | # but database connection strings (with potential passwords) will be unencrypted 146 | #*.pubxml 147 | *.publishproj 148 | 149 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 150 | # checkin your Azure Web App publish settings, but sensitive information contained 151 | # in these scripts will be unencrypted 152 | PublishScripts/ 153 | 154 | # NuGet Packages 155 | *.nupkg 156 | # The packages folder can be ignored because of Package Restore 157 | **/packages/* 158 | # except build/, which is used as an MSBuild target. 159 | !**/packages/build/ 160 | # Uncomment if necessary however generally it will be regenerated when needed 161 | #!**/packages/repositories.config 162 | # NuGet v3's project.json files produces more ignoreable files 163 | *.nuget.props 164 | *.nuget.targets 165 | 166 | # Microsoft Azure Build Output 167 | csx/ 168 | *.build.csdef 169 | 170 | # Microsoft Azure Emulator 171 | ecf/ 172 | rcf/ 173 | 174 | # Windows Store app package directories and files 175 | AppPackages/ 176 | BundleArtifacts/ 177 | Package.StoreAssociation.xml 178 | _pkginfo.txt 179 | 180 | # Visual Studio cache files 181 | # files ending in .cache can be ignored 182 | *.[Cc]ache 183 | # but keep track of directories ending in .cache 184 | !*.[Cc]ache/ 185 | 186 | # Others 187 | ClientBin/ 188 | ~$* 189 | *~ 190 | *.dbmdl 191 | *.dbproj.schemaview 192 | *.jfm 193 | *.pfx 194 | *.publishsettings 195 | node_modules/ 196 | orleans.codegen.cs 197 | 198 | # Since there are multiple workflows, uncomment next line to ignore bower_components 199 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 200 | #bower_components/ 201 | 202 | # RIA/Silverlight projects 203 | Generated_Code/ 204 | 205 | # Backup & report files from converting an old project file 206 | # to a newer Visual Studio version. Backup files are not needed, 207 | # because we have git ;-) 208 | _UpgradeReport_Files/ 209 | Backup*/ 210 | UpgradeLog*.XML 211 | UpgradeLog*.htm 212 | 213 | # SQL Server files 214 | *.mdf 215 | *.ldf 216 | 217 | # Business Intelligence projects 218 | *.rdl.data 219 | *.bim.layout 220 | *.bim_*.settings 221 | 222 | # Microsoft Fakes 223 | FakesAssemblies/ 224 | 225 | # GhostDoc plugin setting file 226 | *.GhostDoc.xml 227 | 228 | # Node.js Tools for Visual Studio 229 | .ntvs_analysis.dat 230 | 231 | # Visual Studio 6 build log 232 | *.plg 233 | 234 | # Visual Studio 6 workspace options file 235 | *.opt 236 | 237 | # Visual Studio LightSwitch build output 238 | **/*.HTMLClient/GeneratedArtifacts 239 | **/*.DesktopClient/GeneratedArtifacts 240 | **/*.DesktopClient/ModelManifest.xml 241 | **/*.Server/GeneratedArtifacts 242 | **/*.Server/ModelManifest.xml 243 | _Pvt_Extensions 244 | 245 | # Paket dependency manager 246 | .paket/paket.exe 247 | paket-files/ 248 | 249 | # FAKE - F# Make 250 | .fake/ 251 | 252 | # JetBrains Rider 253 | .idea/ 254 | *.sln.iml 255 | 256 | # CodeRush 257 | .cr/ 258 | 259 | # Python Tools for Visual Studio (PTVS) 260 | __pycache__/ 261 | *.pyc 262 | 263 | -------------------------------------------------------------------------------- /geeAssetMigratorLib.py: -------------------------------------------------------------------------------- 1 | import ee,os,json,re,shutil,ctypes,glob,secrets 2 | from google.oauth2.credentials import Credentials 3 | ################################################################################################### 4 | """ 5 | Copyright 2020 Ian Housman and Leah Campbell 6 | 7 | Licensed under the Apache License, Version 2.0 (the "License"); 8 | you may not use this file except in compliance with the License. 9 | You may obtain a copy of the License at 10 | 11 | http://www.apache.org/licenses/LICENSE-2.0 12 | 13 | Unless required by applicable law or agreed to in writing, software 14 | distributed under the License is distributed on an "AS IS" BASIS, 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | See the License for the specific language governing permissions and 17 | limitations under the License. 18 | """ 19 | ################################################################################################### 20 | #Library of functions for recursively migrating Google Earth Engine assets from one repository to a new one, sets permissions, and deletes old assets if needed. 21 | ################################################################################################### 22 | def check_end(in_path, add = '/'): 23 | if in_path[-len(add):] != add: 24 | out = in_path + add 25 | else: 26 | out = in_path 27 | return out 28 | ################################################################################################### 29 | #Function to authenticate to a user account and create a custom token at a specified path 30 | #If there is an existing token, it will be retained in the default .config location 31 | #Optionally, this token can be used to initialize GEE 32 | def custom_authenticate(token_path_name,initialize = True,overwrite = False): 33 | token_path_name = os.path.normpath(token_path_name) 34 | #Ensure token directory exists 35 | if os.path.exists(os.path.dirname(token_path_name)) == False:os.makedirs(os.path.dirname(token_path_name)) 36 | 37 | #If the token doesn't already exist or overwrite == True, create it 38 | if os.path.exists(token_path_name) == False or overwrite: 39 | 40 | #Rename any existing credentials 41 | existing_credentials_moved = False 42 | if os.path.exists(ee.oauth.get_credentials_path()): 43 | shutil.move(ee.oauth.get_credentials_path(),ee.oauth.get_credentials_path()+'_temp') 44 | existing_credentials_moved = True 45 | 46 | #Get new credentials and move them to the specified location 47 | ee.Authenticate() 48 | default_token = ee.oauth.get_credentials_path() 49 | shutil.move(default_token,token_path_name) 50 | 51 | #Rename the existing credentials back to the default 52 | if existing_credentials_moved: 53 | shutil.move(ee.oauth.get_credentials_path()+'_temp',ee.oauth.get_credentials_path()) 54 | 55 | #Initialize using the credentials if chosen 56 | if initialize: 57 | initializeFromToken(token_path_name) 58 | ################################################################################################### 59 | #Function to add a list of root assets for a given GEE token file 60 | #Stores the paths in the same json as the token 61 | def addRoots(token_path_name): 62 | print('Adding root paths to:', token_path_name) 63 | 64 | initializeFromToken(token_path_name) 65 | available_roots = [i['id'] for i in ee.data.getAssetRoots()] 66 | 67 | token = json.load(open(token_path_name)) 68 | token['roots'] = available_roots 69 | # print(available_roots,token) 70 | o = open(token_path_name,'w') 71 | o.write(json.dumps(token)) 72 | o.close() 73 | ################################################################################################### 74 | #Function to initialize from specified token 75 | #Does not un-initialize any existing initializations, but will point to this set of credentials 76 | def initializeFromToken(token_path_name): 77 | print('Initializing GEE using:',token_path_name) 78 | refresh_token = json.load(open(token_path_name))['refresh_token'] 79 | c = Credentials( 80 | None, 81 | refresh_token=refresh_token, 82 | token_uri=ee.oauth.TOKEN_URI, 83 | client_id=ee.oauth.CLIENT_ID, 84 | client_secret=ee.oauth.CLIENT_SECRET, 85 | scopes=ee.oauth.SCOPES) 86 | ee.Initialize(c) 87 | ################################################################################################### 88 | #Function to set up a new token and check if it works for specified root 89 | def setupToken(token_dir,root,overwrite = False,name = ''): 90 | if not token_dir:token_dir = os.path.dirname(ee.oauth.get_credentials_path()) 91 | token_dir = check_end(token_dir) 92 | root = fixAssetPath(root) 93 | token = token_dir+'credentials_'+secrets.token_urlsafe(32) # '_'.join(root.split('/')) 94 | 95 | if os.path.exists(token) == False or overwrite: 96 | a = ctypes.windll.user32.MessageBoxW(0, 'Please authenticate to the account that has access to the "{}" asset folder.\n\nMake sure you select the account that has owner level permissions for the asset folder you are copying from.\n\nSometimes the authentication URL will open in a browser instance that does not immediately list the account. You can copy and paste the URL into another browser instance if needed.\n\nThe token will be stored as "{}"'.format(root,token), "!!!! IMPORTANT !!!! Set up {}token".format(name+ ' '), 1) 97 | if a == 1: 98 | custom_authenticate(token,initialize = False,overwrite = overwrite) 99 | addRoots(token) 100 | return token 101 | ################################################################################################### 102 | #Function to set up credentials that have access to a specified asset root folder 103 | #This function will run until it can find a token that has the root listed under the available assetRoots 104 | def smartGetCredentials(root, token_dir = os.path.dirname(ee.oauth.get_credentials_path()),overwrite = False,name = ''): 105 | if not token_dir:token_dir = os.path.dirname(ee.oauth.get_credentials_path()) 106 | #House keeping 107 | if os.path.exists(token_dir) == False:os.makedirs(token_dir) 108 | token_dir = check_end(token_dir) 109 | root = fixAssetPath(root) 110 | 111 | #Try to find a token with asset roots 112 | #Asset roots will be added to token json if they are not already there 113 | can_write = False 114 | iteration = 1 115 | while not can_write: 116 | #List available tokens (assumes only token files are in token_dir) 117 | os.chdir(token_dir) 118 | tokens = glob.glob('*') 119 | 120 | #Find tokens that contain the root as an available asset root 121 | for token in tokens: 122 | can_write = canWrite(token,root) 123 | 124 | if can_write: 125 | out_token = token 126 | break 127 | #If no available tokens have access to the root, set up a new one 128 | if not can_write: 129 | if iteration > 1: 130 | ctypes.windll.user32.MessageBoxW(0, 'The account you just authenticated to does not have access to {}. Please retry and ensure you select an account that does have access to {}.'.format(root,root), "!!!! IMPORTANT !!!! Set up {}token failed. RETRY!".format(name+ ' '), 1) 131 | 132 | out_token = setupToken(token_dir,root,overwrite = overwrite,name = name) 133 | iteration += 1 134 | 135 | # ctypes.windll.user32.MessageBoxW(0, 'Successfully found token that has access to {}'.format(root), "!!!! SUCCESS !!!!", 1) 136 | print('Successfully found token that has access to {}'.format(root)) 137 | return out_token 138 | ################################################################################################## 139 | #Get root dir of an asset 140 | def getAssetRoot(asset): 141 | return re.search("^projects/[^/]+/assets/", asset).group(0) 142 | ################################################################################################### 143 | #See if a given token path can access a specified asset root 144 | def canWrite(token,root): 145 | 146 | #House keeping 147 | root = check_end(root) 148 | root = fixAssetPath(root) 149 | 150 | #See if there are stored roots in json 151 | can_write = False 152 | token_dict = json.load(open(token)) 153 | print('roots' not in token_dict.keys()) 154 | 155 | #First try to add the owner roots and see if that is listed 156 | if 'roots' not in token_dict.keys(): 157 | addRoots(token) 158 | token_dict = json.load(open(token)) 159 | root_matches = [i for i in token_dict['roots'] if root.find(i) == 0] 160 | if root_matches != []:can_write = True 161 | 162 | #If root is still not listed for token, try seeing if the asset root quota can be acquired 163 | if not can_write: 164 | try: 165 | initializeFromToken(token) 166 | quota = ee.data.getAssetRootQuota(getAssetRoot(root)) 167 | can_write = True 168 | 169 | #Cache it if it has access 170 | token_dict['roots'].append(root) 171 | o = open(token,'w') 172 | o.write(json.dumps(token_dict)) 173 | o.close() 174 | except Exception as e: 175 | print('Token {} cannot access {}'.format(token,root)) 176 | 177 | return can_write 178 | ################################################################################################### 179 | #Function to get credentials set up for two accounts for asset migration 180 | def setupCredentialsForMigration(token_dir,sourceRoot,destinationRoot,overwrite = False): 181 | #Get tokens 182 | source_token = smartGetCredentials(sourceRoot, token_dir,overwrite = overwrite,name = 'Source') 183 | destination_token = smartGetCredentials(destinationRoot, token_dir,overwrite = overwrite,name = 'Destination') 184 | return source_token, destination_token 185 | # ################################################################################################### 186 | def fixAssetPath(path,legacy_prefix = 'projects/earthengine-legacy/assets/',regex = "^projects/[^/]+/assets/.*$"): 187 | if re.match(regex,path) == None: 188 | path = legacy_prefix + path 189 | return path 190 | ################################################################################################### 191 | #Function to get all folders, imageCollections, images, and tables under a given folder or imageCollection level 192 | def getTree(fromRoot,toRoot,treeList = None): 193 | pathPrefix = 'projects/earthengine-legacy/assets/' 194 | 195 | #Handle inconsistencies with the earthengine-legacy prefix 196 | fromRoot = fixAssetPath(fromRoot) 197 | toRoot = fixAssetPath(toRoot) 198 | 199 | #Initialize treeList with original root directories if None 200 | if treeList == None: 201 | try: 202 | rootType = ee.data.getAsset(fromRoot)['type'] 203 | except: 204 | rootType = 'FOLDER' 205 | 206 | treeList = [[rootType,fromRoot,toRoot]] 207 | 208 | #Clean up the given paths 209 | if(fromRoot[-1] != '/'):fromRoot += '/' 210 | if(toRoot[-1] != '/'):toRoot += '/' 211 | 212 | 213 | 214 | # print(fromRoot,toRoot,treeList) 215 | #List assets 216 | assets = ee.data.listAssets({'parent':fromRoot})['assets'] 217 | 218 | 219 | #Reursively walk down the tree 220 | nextLevels = [] 221 | for asset in assets: 222 | fromID = asset['name'] 223 | fromType = asset['type'] 224 | toID = fromID.replace(fromRoot,toRoot) 225 | 226 | if fromType in ['FOLDER','IMAGE_COLLECTION']: 227 | nextLevels.append([fromID,toID]) 228 | treeList.append([fromType,fromID,toID]) 229 | 230 | 231 | for i1,i2 in nextLevels: 232 | getTree(i1,i2,treeList) 233 | return treeList 234 | ################################################################################################### 235 | #Function for setting permissions for all files under a specified root level 236 | #Either a list of assets a root to start from can be provided 237 | def batchChangePermissions(assetList = None,root = None,readers = [],writers = [], all_users_can_read = False): 238 | if assetList == None: 239 | assetList = [i[1] for i in getTree(root,root)] 240 | 241 | for assetID in assetList: 242 | print('Changing permissions for: {}'.format(assetID)) 243 | try: 244 | ee.data.setAssetAcl(assetID, json.dumps({u'writers': writers, u'all_users_can_read': all_users_can_read, u'readers': readers})) 245 | except Exception as e: 246 | print(e) 247 | ################################################################################################### 248 | #Function to copy all folders, imageCollections, images, and tables under a given folder or imageCollection level 249 | #Permissions can also be set here 250 | def copyAssetTree(fromRoot,toRoot,changePermissions = False,readers = [],writers = [],all_users_can_read = False): 251 | print('Getting asset tree list') 252 | treeList = getTree(fromRoot,toRoot) 253 | 254 | #Iterate across all assets and copy and create when appropriate 255 | for fromType,fromID,toID in treeList: 256 | if fromType in ['FOLDER','IMAGE_COLLECTION']: 257 | try: 258 | print('Creating {}: {}'.format(fromType,toID)) 259 | ee.data.createAsset({'type':fromType, 'name': toID}) 260 | except Exception as e: 261 | print(e) 262 | else: 263 | try: 264 | print('Copying {}: {}'.format(fromType,toID)) 265 | ee.data.copyAsset(fromID,toID,False) 266 | 267 | except Exception as e: 268 | print(e) 269 | print() 270 | print() 271 | 272 | if changePermissions: 273 | batchChangePermissions(assetList = [i[2] for i in treeList],root = None,readers = readers,writers = writers, all_users_can_read = all_users_can_read) 274 | ################################################################################################### 275 | #Function to delete all folders, imageCollections, images, and tables under a given folder or imageCollection level 276 | def deleteAssetTree(root): 277 | answer = input('Are you sure you want to delete all assets under {}? (y = yes, n = no): '.format(root)) 278 | print(answer) 279 | if answer.lower() == 'y': 280 | answer = input('You answered yes. Just double checking. Are you really sure you want to delete all assets under {}? (y = yes, n = no): '.format(root)) 281 | if answer.lower() == 'y': 282 | treeList = getTree(root,root)[1:] 283 | treeList.reverse() 284 | for fromType, ID1,ID2 in treeList: 285 | print('Deleting {}'.format(ID1)) 286 | try: 287 | ee.data.deleteAsset(ID1) 288 | except Exception as e: 289 | print(e) 290 | --------------------------------------------------------------------------------