├── LICENSE ├── README.md ├── immutable ├── README.md ├── list-immutable.txt ├── makeimmutable.sh └── makeimmutablelistpolicy.txt ├── list ├── README.md ├── list.sh ├── listpol_all.txt ├── listpol_immut.txt ├── listpol_mig.txt ├── listpol_pmig.txt └── listpol_res.txt ├── noobaaGlacier ├── Readme.md ├── migrate.pol ├── recall.pol ├── recallGlacier.sh ├── setExpire.sh └── setexpire.pol ├── premigrate ├── ltfsee_premig.sh ├── premig_fset_all.txt └── readme.md ├── quota-migration ├── README.md ├── callback-quota.sh ├── quota-listpol.txt └── quota-migpol.txt ├── recall ├── LTFS-EE │ ├── ltfsee_recall.sh │ ├── readme.md │ ├── recall_migOnly_policy.txt │ └── recall_migPmig_policy.txt ├── README.md ├── mmpolicyExec-hsm.sample.patch └── recpol.txt ├── receiver ├── policy_receiver_age.txt ├── policy_receiver_all.txt ├── policy_receiver_mig.txt ├── readme.md └── receiver.sh ├── runpolicy ├── readme.md └── runpolicy.sh └── sample-policies ├── list-byState.txt ├── list-byStateWithTapeID.txt ├── list-byTapeID.txt ├── migrate-all.txt ├── migrate-fset.txt ├── premigrate-all.txt ├── premigrate-fset.txt ├── readme.md └── recall-all.txt /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2020 Nils Haustein 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | Copyright 2024 Nils Haustein, released under the [MIT license](LICENSE) 3 | 4 | This project includes scripts and policies for IBM Spectrum Scale ILM in combination with IBM Spectrum Archive Enterprise Edition and IBM Spectrum Protect for Space Management. For more inforemation refer to the [IBM Spectrum Scale ILM policy whitepaper](https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102642). 5 | 6 | ## Folder [immutable](immutable/) - Script to set files to immutable 7 | 8 | This folder contains a script and policies to identify files that are not immutable in an immutable fileset and sets these identified files to immutable. The `makeimmutable.sh` script is an external pool script that receives the input from an EXTERNAL LIST policy and sets all files identified by the LIST policy to immutable using the `mmchattr -i yes` command. The retention period is defined in the policy itself and is applied to each file by setting the retention time to current date and time plus retention period 9 | 10 | ------------------------ 11 | 12 | ## Folder [list](list/) - List policy script 13 | 14 | This folder contains a wrapper script for LIST policies. The main purpose is to list the numbers and optionally the file names of file in accordance to their migration state. This wrapper script can be dynamically extended to use other list policies. 15 | 16 | ------------------------ 17 | 18 | ## Folder [noobaaGlacier](noobaaGlacier/) - policies and scripts for NooBaa Glacier 19 | 20 | This folder contains policies and scripts allowing to automatically manage migration and recalls of S3 GLACIER objects ingested through NooBaa endpoints in IBM Storage Scale file systems. 21 | 22 | The policy in `migrate.pol` facilitates the migration of objects from the Storage Scale file system to tape. 23 | 24 | The policy `recall.pol` facilitates the recall of objects from tape. It is implemented as EXTERNAL LIST policy that invokes the external script `recallGlacier.sh`) to accommodate the recall of the selected files. 25 | 26 | The policy `setexpire.pol` facilitates setting the user.noobaa attributes. It is implemented as EXTERNAL LIST policy that invokes the external script `setExpire.sh` to accommodate setting the attributes of the selected files. 27 | 28 | ------------------------ 29 | 30 | ## Folder [premigrate](/premigrate) - Premigrate policies and scripts 31 | 32 | This folder contains script and policies to perform premigrates using migrate policies. It provides an interface script that wraps perform pre-migration instead of migration. It contains an interface script for Spectrum Archive. 33 | 34 | ------------------------ 35 | 36 | ## Folder [quota-migration](/quota-migration) - migration when quota limits have reached 37 | 38 | This folder contains a callback script and policies facilitating migration of files for filesets where the soft quota limit has been reached. The callback script (`callback-quota.sh`) is invoked when the event `softQuotaExceeded` is triggered for a fileset. This script invokes an EXTERNAL LIST policy to identify files in the fileset that qualify for migration based on the quota limits and a migration policy that migrates the files identified by the list policy to IBM Spectrum Archive EE. 39 | 40 | ------------------------ 41 | 42 | ## Folder [Recall](recall/) - wrapper script for tape optimized recalls 43 | 44 | This folder contains script and policies for tape optimized recalls with Spectrum Protect for Space Management and Spectrum Archive Enterprise Edition . The scripts are customized interface scripts that perform recall instead of migration. 45 | 46 | ------------------------ 47 | 48 | ## Folder [receiver](receiver/) - external script invoked by EXTERNAL LIST policy 49 | 50 | This folder includes an interface script that receives file lists from an EXTERNAL LIST policies and processes the files contained in the file list. The processing in this script extracts the file names contained in the file list and writes these file names to an extra file. Of course you can add other operations for the files. 51 | 52 | ------------------------ 53 | 54 | ## Folder [runpolicy](runpolicy/) - wrapper for mmapplypolicy 55 | 56 | This folder contains a wrapper script for the `mmapplypolicy` command. It is invoked in a simplified way and allows either pass further options for the `mmapplypolicy` commmand or encode these options in an internal variable. 57 | 58 | ------------------------ 59 | 60 | ## Folder [sample-policies](sample-policies/) - sample ILM policies 61 | 62 | This folder includes sample policies for migration, recall and list. 63 | 64 | 65 | -------------------------------------------------------------------------------- /immutable/README.md: -------------------------------------------------------------------------------- 1 | # Introduction: 2 | 3 | This script can be used in combination with the policy engine and a proper LIST policy to automatically set files to immutable with a defined retention period. The retention time for each files is set to current date and time plus retention period defined in the policy. It is an interface script for LIST policy invoked by mmapplypolicy. 4 | 5 | 6 | ## Prerequisite: 7 | 8 | The makeimmutable.sh script needs to be installed on one or more Spectrum Scale nodes in a the same directory. The directory where this program is installed is referenced in the EXTERNAL list rule. 9 | 10 | ## Example Policy: 11 | 12 | This example policy will identify all files ending with `.mp3` and `.pdf` in the immutable fileset `worm`that are not set to immutable and invoke the external script /root/silo/makeimmutable.sh. This external script sets the specified retention period to the selected files and make these files immutable. The external script is installed in directory: `root/silo/makeimmutable.sh` on all nodes that participate in this operation. 13 | 14 | The following rules show how to set all files ending with `.mp3` to 1 day retention (`OPTS '1'`): 15 | 16 | /* define exclude list */ 17 | define( exclude_list, 18 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 19 | PATH_NAME LIKE '%/.ltfsee/%' OR 20 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 21 | PATH_NAME LIKE '%/.snapshots/%' OR 22 | NAME LIKE '.mmbackupShadow%' OR 23 | NAME LIKE 'mmbackup%') 24 | 25 | /* define immutable attribute */ 26 | define( immutable, MISC_ATTRIBUTES LIKE '%X%') 27 | 28 | /* define external list with makeimmutable.sh script to set mp3 files in fileset 'worm' to immutable with 1 day retention */ 29 | RULE EXTERNAL LIST 'setmp3' EXEC '/root/silo/makeimmutable.sh' OPTS '1' 30 | RULE 'mp3' LIST 'setmp3' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.mp3') 31 | 32 | 33 | In the same policy there can be additional EXTERNAL LIST rules which set other types of files to immutable with different retention period. For example the rules below set 2 days retention times for files ending with `.pdf` (`OPTS '2'`). Add this set of rules to the policy above when needed: 34 | 35 | /* define external list with makeimmutable.sh script to set pdf files in fileset 'worm' to immutable with 1 day retention */ 36 | RULE EXTERNAL LIST 'setpdf' EXEC '/root/silo/makeimmutable.sh' OPTS '2' 37 | RULE 'pdf' LIST 'setpdf' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.pdf') 38 | 39 | 40 | An example of this policy can be found in [makeimmutablelistpolicy.txt](makeimmutablelistpolicy.txt). 41 | 42 | 43 | ## Invokation: 44 | 45 | To execute the policy use the mmapplypolicy command. This will invoke the external script with file lists including the filenames of the files that have been identied. Here is an example, some parameters have to be changed: 46 | 47 | mmapplypolicy fsname -P policyfile -N nodename -B 1000 -m 1 -s localworkdir -g globalworkdir --single-instance [-I test] 48 | 49 | 50 | The parameters of the command are the following: 51 | 52 | fsname is the name of the file system or a fully qualified path of a fileset or directory 53 | 54 | -P policyfile is the file including the rules for this policy. An example can be found in [makeimmutablelistpolicy.txt](makeimmutablelistpolicy.txt) 55 | 56 | -N specifies a one or more Spectrum Scale node name where the makeimmutable.sh script is installed. 57 | 58 | -m specifies the number of concurrent external script instances to be launched per node. In the example above it is set to one which means that there is one instance running 59 | 60 | -B specifies the number of file names in one file list. In this case it would be 1000 files per file list. 61 | 62 | -s specifies a local directory used to store temporary files created by the policy engine. There must be sufficient space in this directory. The default directory is /tmp. 63 | 64 | -g specifies a directory that is accessible be all cluster nodes. It can be in the file system that is processed by the policy engine or it can be in a different file system. The default is specified by the Spectrum Scale configuration parameter `sharedTmpDir`. 65 | 66 | --single-instance specifies that only one instance of the policy engine can run. If another instance is already running then this command will abort 67 | 68 | -I test means that the policy is tested for syntax. No -I means that the policy performs file selection and executes script. 69 | 70 | 71 | 72 | ## Processing: 73 | 74 | With the proper policy this script is invoked by mmapplypolicy with the following parameters: 75 | $1 operation (list, test) 76 | $2 file system name or name of filelist 77 | $3 optional parameter defined in LIST policy under OPTS, defines retention period in days relative to current date 78 | 79 | The script implements the TEST and LIST operation. 80 | Upon TEST it tests if the file system name given with $2 exists and exits with 0. 81 | Upon LIST it parses the filelist passed with the parameter $2, extracts the file names and sets the retention period to current-date plus retention-time. The retention-time is passed with the parameter $3 and encoded in the policy. 82 | 83 | ## Output: 84 | 85 | Sets files identified to immutable with retention period define in policy relative to current date and time (default is defined as $DEFRETTIME). 86 | Write runtime information and debugging messages to log file specified with the parameter `$LOGFILE` in the makeimmutable.sh script. 87 | 88 | ## Supplemental List Policy 89 | 90 | It may be worth to periodically check if all files have been set to immutable. The supplemental list policy [list-immutable.txt](list-immutable.txt) can be used for this. This policy will list any files in fileset `worm` which are not immutable. To list only files of a certain type a WHERE clause can be added to the LIST rule, like this: 91 | 92 | ... WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.mp3') 93 | 94 | 95 | To run this policy use the `mmapplypolicy` command: 96 | mmapplypolicy fsname -P policy -f ./my -I defer 97 | 98 | 99 | The resulting file list is stored in file `my.list.immut` 100 | -------------------------------------------------------------------------------- /immutable/list-immutable.txt: -------------------------------------------------------------------------------- 1 | /* define exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* define immutable attribute */ 12 | define( immutable, MISC_ATTRIBUTES LIKE '%X%') 13 | 14 | /* external list rule */ 15 | RULE EXTERNAL LIST 'immut' EXEC '' 16 | RULE 'listimmut' LIST 'immut' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) 17 | 18 | /* to run this policy: mmapplypolicy fsname -P policy -f ./gpfs -I defer */ 19 | /* result is written to ./gpfs.list.immut */ -------------------------------------------------------------------------------- /immutable/makeimmutable.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2020 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | 26 | # Program: makeimmutable.sh 27 | # 28 | # Description: 29 | #------------- 30 | # Interface script for LIST policy invoked by mmapplypolicy 31 | # Sets selected files to immutable with retention time given in policy. 32 | # 33 | # Prerequisite: 34 | #------------- 35 | # EXTERNAL list policy that identifies files that are not immutable. 36 | # 37 | # Input: 38 | #------- 39 | # invoked by mmapplypolicy with the following parameters: 40 | # $1 operation (list, test) 41 | # $2 file system name or name of filelist 42 | # $3 optional parameter defined in LIST policy under OPTS, defines retention period in days relative to current date 43 | # 44 | # Output: 45 | #-------- 46 | # Sets files identified to immutable with retention period define in policy (default is defined as $DEFRETTIME 47 | # Write runtime information and debugging messages to log file $LOGFILE 48 | # 49 | # Example Policy: 50 | #----------------- 51 | #/* define exclude list */ 52 | #define( exclude_list, 53 | # (PATH_NAME LIKE '%/.SpaceMan/%' OR 54 | # PATH_NAME LIKE '%/.ltfsee/%' OR 55 | # PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 56 | # PATH_NAME LIKE '%/.snapshots/%' OR 57 | # NAME LIKE '.mmbackupShadow%' OR 58 | # NAME LIKE 'mmbackup%') 59 | #/* define immutable attribute */ 60 | #define( immutable, MISC_ATTRIBUTES LIKE '%X%') 61 | #/* define external list with makeimmutable.sh script to set mp3 files in fileset 'worm' to immutable with 1 day retention */ 62 | #RULE EXTERNAL LIST 'setmp3' EXEC '/root/silo/makeimmutable.sh' OPTS '1' 63 | #RULE 'mp3' LIST 'setmp3' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.mp3') 64 | # 65 | # Invokation: 66 | #------------- 67 | # mmapplypolicy fsname -P policyfile [-N nodenames -m threads -B bucketsize] 68 | # Policy file must include a policy as shown above. 69 | # 70 | # Change History 71 | #---------------- 72 | # 10/09/12 first implementation based on startbackup 73 | # 12/20/15 implementation for immutability, some streamlining of existing code 74 | # 06/22/20 minor adjustments 75 | 76 | #---------------------------------- 77 | #These parameters can be adjusted 78 | #----------------------------------- 79 | # define the path for the log file 80 | MYPATH="./makeimmutable/" 81 | # logfile used for system_log function, logs are appended 82 | LOGFILE=$MYPATH"makeimmutable.log" 83 | # sets the log level for the system log, everything below that number is logged 84 | LOGLEVEL=1 85 | # default retention time 86 | DEFRETTIME=0 87 | 88 | #---------------------------------- 89 | # Constants 90 | #---------------------------------- 91 | # GPFS path name to be used with all GPFS commands 92 | gpfsPath="/usr/lpp/mmfs/bin" 93 | 94 | 95 | ## Append to the system log 96 | ## Usage: system_log 97 | system_log () { 98 | SEV=$1 99 | 100 | case $SEV in 101 | [0-9]) ;; 102 | *) SEV=1;; 103 | esac 104 | 105 | LINE=$2 106 | if [ $LOGLEVEL -ge $SEV ] ; then 107 | if [[ -z "$LINE" ]]; then 108 | echo -e "INTERNAL WARNING: Improper value given to system_log function ($@)" >> $LOGFILE 109 | else 110 | echo -e "$LINE" >> $LOGFILE 111 | fi 112 | fi 113 | } 114 | 115 | ## Print a message to the stdout 116 | ## Usage: user_log 117 | user_log () { 118 | echo -e "Makeimmutable $@" 119 | } 120 | 121 | 122 | ## Get current date and time 123 | ## Usage: get_cur_date_time 124 | get_cur_date_time(){ 125 | echo "$(date +"%Y-%b-%d %H:%M:%S")" 126 | } 127 | 128 | #++++++++++++++++++++++++++ MAIN ++++++++++++++++++++++++++++++++++++++ 129 | user_log "$(get_cur_date_time) makeimmutable invoked by policy engine" 130 | 131 | # check the path for logging 132 | if [[ ! -d $MYPATH ]] then 133 | mkdir -p $MYPATH 134 | rc=$? 135 | if (( rc > 0 )) then 136 | user_log "ERROR: failed to create directory $MYPATH, check permissions" 137 | exit 1 138 | fi 139 | fi 140 | 141 | system_log 1 "=========================================================================" 142 | system_log 1 "$(get_cur_date_time) makeimmutable invoked with arguments: $*" 143 | 144 | ## Parse Arguments & execute 145 | #$1 is the policy operation (list, migrate, etc) 146 | op=$1 147 | #$2 is the policy file name 148 | polFile=$2 149 | #$3 is the option given in the EXTERNAL LIST rule with OPTS '..' should be retention time here 150 | retTime=$3 151 | 152 | ## this is required, as the script may be called multiple times during 153 | ## the same backup (if there are too many files to process). 154 | 155 | case $op in 156 | TEST ) 157 | user_log "INFO: TEST option received for directory $polFile." 158 | system_log 1 "INFO: TEST option received for $polFile" 159 | if [[ ! -z "$polFile" ]] then 160 | if [[ -d "$polFile" ]] then 161 | user_log "INFO: TEST directory $polFile exists." 162 | system_log 1 "INFO: Directory $polFile exists." 163 | else 164 | user_log "WARNING: TEST directory $polFile does not exists." 165 | system_log 1 "WARNING: Directory $polFile does not exist." 166 | fi 167 | fi 168 | ;; 169 | LIST ) 170 | user_log "INFO: LIST option received, starting makeimmutable task" 171 | system_log 1 "INFO: LIST option received with file name $polFile and options $retTime" 172 | 173 | #set retention time to default if not set 174 | if [[ -z $retTime ]] then 175 | retTime=$DEFRETTIME 176 | fi 177 | 178 | itemNum=0 179 | 180 | numEntries=$(wc -l $polFile | awk '{print $1}') 181 | system_log 1 "INFO: setting retention time $retTime day(s) for $numEntries files" 182 | #consider a function for this 183 | cat $polFile | while read line 184 | do 185 | # use set to get file name, does tolerate blanks 186 | set $line 187 | shift 4 188 | fName="$*" 189 | 190 | rc=0 191 | # perhaps check if file exists 192 | if [[ ! -a $fName ]] then 193 | system_log 1 "WARNING: file $fName does not exist." 194 | user_log "DEBUG: file $fName does not exist." 195 | else 196 | $gpfsPath/mmchattr -E $(date +%Y-%m-%d@%H:%M:%S -d "$DATE + $retTime day") "$fName" 197 | rc=$? 198 | if (( rc == 0 )) then 199 | $gpfsPath/mmchattr -i yes "$fName" 200 | (( rc=rc+$? )) 201 | fi 202 | if (( rc == 0 )) then 203 | system_log 1 "INFO: Retention time $retTime for file $fName set successful." 204 | user_log "INFO: Retention time $retTime for file $fName set successful." 205 | else 206 | system_log 1 "WARNING: Setting retention time $retTime for file $fName failed (rc=$rc)" 207 | user_log "WARNING: Setting retention time $retTime for file $fName failed (rc=$rc)" 208 | fi 209 | fi 210 | (( itemNum=itemNum+1 )) 211 | done 212 | system_log 1 "INFO: retention time set for $itemNum out of $numEntries files" 213 | user_log "INFO: retention time set for $itemNum out of $numEntries files" 214 | ;; 215 | REDO ) 216 | user_log "INFO: REDO option received, doing nothing" 217 | system_log 1 "INFO: REDO option received with file name $polFile and options $retTime" 218 | ;; 219 | * ) 220 | user_log "WARNING: Unknown option $op received, doing nothing" 221 | system_log 1 "WARNUNG: UNKNOWN option ($op) received with file name $polFile and options $retTime" 222 | ;; 223 | esac 224 | 225 | user_log "$(get_cur_date_time) makeimmutable ended" 226 | system_log 1 "$(get_cur_date_time) makeimmutable ended" 227 | 228 | exit 0 -------------------------------------------------------------------------------- /immutable/makeimmutablelistpolicy.txt: -------------------------------------------------------------------------------- 1 | /* define exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* define immutable attribute */ 12 | define( immutable, MISC_ATTRIBUTES LIKE '%X%') 13 | 14 | /* define external list with makeimmutable.sh script to set mp3 files in fileset 'worm' to immutable with 1 day retention */ 15 | RULE EXTERNAL LIST 'setmp3' EXEC '/root/silo/makeimmutable.sh' OPTS '1' 16 | RULE 'mp3' LIST 'setmp3' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.mp3') 17 | 18 | /* define external list with makeimmutable.sh script to set pdf files in fileset 'worm' to immutable with 1 day retention */ 19 | RULE EXTERNAL LIST 'setpdf' EXEC '/root/silo/makeimmutable.sh' OPTS '2' 20 | RULE 'pdf' LIST 'setpdf' FOR FILESET ('worm') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.pdf') 21 | 22 | -------------------------------------------------------------------------------- /list/README.md: -------------------------------------------------------------------------------- 1 | # List policy script 2 | 3 | ## Description: 4 | 5 | This is a wrapper script to run predefined and custom LIST policies. LIST policies can be used to find files matching certain criteria, such as file that are migrated or files that are premigrated. There are some predefined LIST policies included (see section [Running predefined LIST policies](#Running-predefined-LIST-policies)). Further custom LIST policies can be created and run by this wrapper script (see section [Running custom LIST policies](#Running-custom-LIST-policies)). 6 | 7 | 8 | 9 | ## Running predefined LIST policies 10 | 11 | The wrapper script `list.sh` lists the number of files according to the state given as command line parameter. Optionally the file names can be listed (option `-v`) 12 | 13 | For installation, copy the wrapper script `list.sh` along with the predefined policy file examples named `listpol_*.txt` to the same directory. 14 | 15 | The syntax is: 16 | 17 | # list.sh state fspath [-v -s] 18 | 19 | Options: 20 | state is the name of the policy to be executed according to the above naming conventions 21 | fspath is the file system or directory path subject for the list policy 22 | -v list the file names instead of the number of files 23 | -s specify the local work directory for the policy engine (default is /tmp) 24 | 25 | Predefined states are: 26 | mig list all migrated files 27 | pmig list all premigrated files 28 | res list all resident files 29 | all list all premigrated 30 | 31 | The script checks if the list policy file exists, executes the approporate LIST policy using `mmapplypolicy` and prints the number of files matching the state. The actual path and file names matching the LIST rule are written to `/tmp/gpfs.list.`. The directory of the output files can be changed in the script by the parameter `$ofPrefix`. 32 | 33 | The output of the policy run is written to STDOUT. 34 | 35 | For example, to list the number of in file system /gpfs/archive that are migrated use the following command: 36 | 37 | # list.sh mig /gpfs/archive 38 | [I] 2020-06-23@16:55:05.206 Directory entries scanned: 838. 39 | [I] 2020-06-23@16:55:06.106 Parallel-piped sort and policy evaluation. 838 files scanned. 40 | [I] 2020-06-23@16:55:06.399 Piped sorting and candidate file choosing. 141 records scanned. 41 | [I] 2020-06-23@16:55:06.437 Policy execution. 0 files dispatched. 42 | ============================================================================== 43 | Files that are in state mig: 44 | Number of files with state mig: 141 (filename: /tmp/gpfs.list.mig) 45 | 46 | 47 | The file list with the file names is stored in `/tmp/gpfs.list.mig` (until the program is executed again with the same state operation). 48 | 49 | You can also create your own list policies and run it by the wrapper script `list.sh`. 50 | 51 | 52 | ## Running custom LIST policies 53 | 54 | The wrapper script `list.sh` can also be used to run your own custom policies. To create a custom policy two things must be considered: 55 | 56 | 1. Create a custom list policy where the EXTERNAL LIST name defines the state of the files to be listed. For example, the following EXTERNAL LIST name is `immut`. The associated LIST rules lists immutable files: 57 | 58 | /* Define exclude list to exclude SpaceMan and snapshots */ 59 | define( exclude_list, 60 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 61 | PATH_NAME LIKE '%/.ltfsee/%' OR 62 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 63 | PATH_NAME LIKE '%/.snapshots/%' OR 64 | NAME LIKE '.mmbackupShadow%' OR 65 | NAME LIKE 'mmbackup%') 66 | ) 67 | 68 | /* find immutable files */ 69 | define(is_immutable,(MISC_ATTRIBUTES LIKE '%X%')) 70 | 71 | RULE EXTERNAL LIST 'immut' EXEC '' 72 | RULE 'immutable_files' LIST 'immut' WHERE (is_immutable) AND ( NOT (exclude_list) ) 73 | 74 | 75 | 2. Store this policy in a file named: `listpol_immut.txt`. Thus, the name of the EXTERNAL LIST must match the file name pattern of the policy file. 76 | 77 | 78 | To run this policy with the `list.sh` script, copy the policy file to the directory where `list.sh` is located or alternatively to the location specified by parameter `$pfPrefix` within the `list.sh`script. Now run the script: 79 | 80 | # list.sh immut /gpfs/archive 81 | 82 | 83 | This display list the number of immutable files in the file system `gpfs/archive`. The file list with the file names is stored in `/tmp/gpfs.list.immut` (until the program is executed again with the same state operation). 84 | -------------------------------------------------------------------------------- /list/list.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2019 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | 26 | # 27 | # flexible framework to run list policies 28 | # - create a list policy where the external rule name is identical to the op-code given to this program 29 | # - the op-code given to this program is the first argument 30 | # - the policy file must include the op-code in the filename: listpol_.txt 31 | # - this policy file is stored in the path configurable below ($pfPrefix) 32 | # - run the program with the state: list.sh 33 | # - optional argument -v allows to show verbose output (file names) 34 | # - optional argument -s allows to set the working directory for the policy engine output files 35 | # 36 | 37 | # ADJUST define pathname and prefix for policy files 38 | # policy file must be named: $pfPrefix_$op.txt 39 | pfPrefix="./listpol" 40 | 41 | # define local work directory for mmapplypolicy and output files, default is /tmp 42 | workDir="/tmp" 43 | 44 | # ADJUST define file system name 45 | fsName="" 46 | 47 | # define output format: 0 - number of files only; 1 - filenames 48 | numbersOnly=1 49 | 50 | # Constant: GPFS path 51 | gpfsPath="/usr/lpp/mmfs/bin" 52 | 53 | 54 | #function syntax 55 | function syntax 56 | { 57 | echo 58 | echo "Syntax: list state filesystem [-v -s workDir]" 59 | echo " This program lists the files according to the HSM state, Valid states are:" 60 | echo " mig: list all migrated files" 61 | echo " pmig: list all premigrated files" 62 | echo " res: list all resident files" 63 | echo " all: provides statistic about all states" 64 | echo 65 | echo " Filesystem: is the name of the file system or directory" 66 | echo " -v: shows the file names selected by the policy (default is number of files)" 67 | echo " -s workDir: specify the working directory for the policy engine output files (default is $workDir)" 68 | echo 69 | } 70 | 71 | #check first argument to be state 72 | if [[ -z "$1" ]]; 73 | then 74 | echo "Error: state not specified." 75 | syntax 76 | exit 1 77 | else 78 | op=$1 79 | fi 80 | 81 | # second argument must be file system path 82 | if [[ -z "$2" ]]; 83 | then 84 | echo "Error: file system not specified" 85 | syntax 86 | exit 1 87 | else 88 | fsName=$2 89 | if [[ ! -d $fsName ]]; 90 | then 91 | echo "Error: file system $fsName does not exist." 92 | syntax 93 | exit 1 94 | fi 95 | fi 96 | 97 | # now parse the rest of the arguments 98 | shift 2 99 | # echo "DEBUG: args=$*" 100 | 101 | while [[ ! -z "$1" ]]; 102 | do 103 | case "$1" in 104 | "-v") numbersOnly=0;; 105 | "-s") shift 1 106 | workDir=$1 107 | if [[ ! -d "$workDir" ]]; 108 | then 109 | echo "Error: working directory specified by -s $workDir does not exist." 110 | syntax 111 | exit 1 112 | fi ;; 113 | *) echo "Error: Unknown Argument received ($1). " 114 | syntax 115 | exit 1;; 116 | esac 117 | shift 1 118 | done 119 | 120 | #define prefix for policy output file 121 | #policy output file is named: $ofPrefix.list.$op 122 | ofPrefix="$workDir""/gpfs" 123 | 124 | #define logfile name 125 | logfile="$workDir""/gpfslist_mmapply.log" 126 | 127 | 128 | #check if policy file exists 129 | polfile="$pfPrefix""_""$op"".txt" 130 | # echo "DEBUG: policy file name is: $polfile" 131 | if [[ ! -a $polfile ]]; 132 | then 133 | echo "ERROR: Policyfile $polfile does not exist in $pfPrefix. Provide the file or use correct state." 134 | syntax 135 | exit 1 136 | fi 137 | 138 | #delete previous files to not get confused 139 | for s in mig pmig res; 140 | do 141 | outfile="$ofPrefix"".""list"".""$s" 142 | rm -f $outfile 2>&1 143 | done 144 | 145 | # if state=all then run run mmapplypolicy and subsequently determine the number of respective files 146 | 147 | # run mmapplypolicy 148 | $gpfsPath/mmapplypolicy $fsName -P $polfile -s $workDir -f $ofPrefix -I defer > $logfile 149 | rc=$? 150 | echo "==============================================================================" 151 | if (( rc == 0 )); 152 | then 153 | echo "Files that are in state $op:" 154 | if [[ "$op" = "all" ]]; 155 | then 156 | for s in mig pmig res; 157 | do 158 | outfile="$ofPrefix"".""list"".""$s" 159 | if [[ ! -a "$outfile" ]]; 160 | then 161 | num=0 162 | else 163 | num=$(wc -l $outfile | awk '{print $1}') 164 | fi 165 | echo " Number of files with state $s: $num (filename: $outfile)" 166 | done 167 | else 168 | #create name of policy output file 169 | outfile="$ofPrefix"".""list"".""$op" 170 | # echo "DEBUG: out file name is: $outfile" 171 | if (( numbersOnly )); 172 | then 173 | if [[ ! -a "$outfile" ]]; 174 | then 175 | num=0 176 | else 177 | num=$(wc -l $outfile | awk '{print $1}') 178 | fi 179 | echo " Number of files with state $op: $num (filename: $outfile)" 180 | else 181 | if [[ ! -a "$outfile" ]]; 182 | then 183 | echo "WARNING: The policy did not identify any files according to the policy.!" 184 | else 185 | cat $outfile 186 | echo "---------------------------------------------------------------------------" 187 | echo "INFO: See file $outfile" 188 | echo "---------------------------------------------------------------------------" 189 | fi 190 | fi 191 | fi 192 | else 193 | echo "ERROR: mmapplypolicy returned error (rc=$rc), check log ($logfile)" 194 | fi 195 | 196 | exit 0 197 | -------------------------------------------------------------------------------- /list/listpol_all.txt: -------------------------------------------------------------------------------- 1 | /* Define exclude list to exclude SpaceMan and snapshots */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* Define is migrated */ 12 | define( is_migrated,(MISC_ATTRIBUTES LIKE '%V%') ) 13 | 14 | /* list rule to list all migrated files */ 15 | RULE EXTERNAL LIST 'mig' EXEC '' 16 | RULE 'list_mig' LIST 'mig' SHOW ( varchar(file_size) ) WHERE ( is_migrated ) AND ( NOT (exclude_list) ) 17 | 18 | /* Define is premigrated */ 19 | define( is_premigrated,(MISC_ATTRIBUTES LIKE '%M%' AND MISC_ATTRIBUTES NOT LIKE '%V%') ) 20 | 21 | /* list rule to list all premigrated files */ 22 | RULE EXTERNAL LIST 'pmig' EXEC '' 23 | RULE 'list_pmig' LIST 'pmig' SHOW ( varchar(file_size) ) WHERE ( is_premigrated ) AND ( NOT (exclude_list) ) 24 | 25 | /* Define is resident */ 26 | define( is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%') ) 27 | 28 | /* list rule to list all resident files */ 29 | RULE EXTERNAL LIST 'res' EXEC '' 30 | RULE 'list_res' LIST 'res' SHOW ( varchar(file_size) ) WHERE ( is_resident ) AND ( NOT (exclude_list) ) 31 | -------------------------------------------------------------------------------- /list/listpol_immut.txt: -------------------------------------------------------------------------------- 1 | /* Define exclude list to exclude SpaceMan and snapshots */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* find immutable files */ 12 | define(is_immutable,(MISC_ATTRIBUTES LIKE '%X%')) 13 | 14 | RULE EXTERNAL LIST 'immut' EXEC '' 15 | RULE 'immutable_files' LIST 'immut' WHERE (is_immutable) AND ( NOT (exclude_list) ) 16 | 17 | -------------------------------------------------------------------------------- /list/listpol_mig.txt: -------------------------------------------------------------------------------- 1 | /* Define exclude list to exclude SpaceMan and snapshots */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* Define is migrated */ 12 | define( is_migrated,(MISC_ATTRIBUTES LIKE '%V%') ) 13 | 14 | /* list rule to list all migrated files */ 15 | RULE EXTERNAL LIST 'mig' EXEC '' 16 | RULE 'list_mig' LIST 'mig' WHERE ( is_migrated ) AND ( NOT (exclude_list) ) 17 | -------------------------------------------------------------------------------- /list/listpol_pmig.txt: -------------------------------------------------------------------------------- 1 | /* Define exclude list to exclude SpaceMan and snapshots */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* Define is premigrated */ 12 | define( is_premigrated,(MISC_ATTRIBUTES LIKE '%M%' AND MISC_ATTRIBUTES NOT LIKE '%V%') ) 13 | 14 | /* list rule to list all premigrated files */ 15 | RULE EXTERNAL LIST 'pmig' EXEC '' 16 | RULE 'list_pmig' LIST 'pmig' WHERE ( is_premigrated ) AND ( NOT (exclude_list) ) 17 | -------------------------------------------------------------------------------- /list/listpol_res.txt: -------------------------------------------------------------------------------- 1 | /* Define exclude list to exclude SpaceMan and snapshots */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* Define is resident */ 12 | define( is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%') ) 13 | 14 | /* list rule to list all resident files */ 15 | RULE EXTERNAL LIST 'res' EXEC '' 16 | RULE 'list_res' LIST 'res' WHERE ( is_resident ) AND ( NOT (exclude_list) ) 17 | -------------------------------------------------------------------------------- /noobaaGlacier/Readme.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | This project includes policies and scripts allowing to automatically manage migration and recalls of S3 GLACIER objects ingested through NooBaa endpoints in IBM Storage Scale file systems. 4 | 5 | 6 | ## System environment 7 | 8 | The systems applicable for the policies and scripts includes an IBM Storage Scale file system that is space managed by IBM Storage Archive Enterprise Edition. The open-source software NooBaa is installed on one or more IBM Storage Scale cluster nodes and provides the S3 object storage endpoints to the S3 users and applications. Objects and buckets ingested via the NooBaa S3 endpoints are stored as files and directories in the Storage Scale file system. 9 | 10 | NooBaa supports the AWS S3 and the AWS S3 Glacier API. 11 | 12 | 13 | ## AWS S3 Glacier 14 | 15 | AWS S3 Glacier provides additional API functions on top of the S3 API. The S3 Glacier API facilitates retrieval or restoration of objects from high latency media like tapes. Object stored in a Glacier storage class cannot be retrieved using normal S3 GET requests. Instead, the S3 user must first issue the Restore-Object request provided by the S3 Glacier API to request the recall from tape. After the object was recalled from tape, the user can use the normal S3 GET operations to retrieve the object. 16 | 17 | With the Restore-Object request, the S3 user can signal the retrieval of the object. The Restore-Object request allows to specify an expiration time period in days. After the recall, the object shall be retained on disk for the specified expiration time period. 18 | 19 | This means, a S3 Glacier object is not immediatelly accessible. Instead the user issues a Restore-Object request and waits for an agree time period. During this time period more Restore-Object requests can be accumulated and objects can be recalled from tape in an optimized manner. The time period the user must wait before he can access the object is defined as service level agreement (SLA). 20 | 21 | Here is an example of the Restore-Object command using the AWS CLI where a expiration time of 1 days is specified: 22 | 23 | ``` 24 | # aws s3api restore-object --bucket glacier1 --key coldile0 --restore-request '{"Days": 1}' 25 | ``` 26 | 27 | 28 | After issuing the Restore-Object command, the user can check if the object is available for access by using the Head-Object command. 29 | 30 | ``` 31 | # aws s3api head-object --bucket glacier1 --key coldfile0 32 | { 33 | "AcceptRanges": "bytes", 34 | "Restore": "ongoing-request=\"true\"", 35 | "LastModified": "Mon, 11 Mar 2024 14:30:28 GMT", 36 | "ContentLength": 6449152, 37 | "ETag": "\"mtime-czqzravhx5a8-ino-mxj\"", 38 | "ContentType": "application/octet-stream", 39 | "Metadata": { 40 | "storage_class": "GLACIER" 41 | }, 42 | "StorageClass": "GLACIER" 43 | } 44 | 45 | ``` 46 | 47 | The `Restore` parameter is set to `ongoing-request=true`, which means that the object is being retrieved from the high latency media. The object can still not be retrieved using the normal GET object request. Some magic needs to happen in the system, that is described in the next section. 48 | 49 | 50 | 51 | ## NooBaa and S3 Glacier 52 | 53 | NooBaa leverages extended attributes of the files in the Storage Scale file system. When the S3 user stores an object in storage class GLACIER, then NooBaa sets an attribute `user.storage_class=GLACIER`. With this attribute an object is marked as Glacier object. 54 | 55 | When the user issues an Restore-Object request for an object, then NooBaa sets an attribute `user.noobaa.restore.request="1"` The value of `1` denotes the expiration time period of 1 day. This means after the recall the object is kept on disk for 1 day. The expireation time period can be any integer value. 56 | 57 | To make the object available for retrieval using a normal GET request, the attribute `user.noobaa.restore.expiry` must be set to a valid time stamp in the future and the attribute `user.noobaa.restore.request` must be deleted (or set to `false`). The time stamp encoded in attribute `user.noobaa.restore.expiry` represents the expiration date. If the expiration date has expired, then the object is no longer retrievable. The time stamp in attribute `user.noobaa.restore.expiry` is set in ISO format: `yyyy-mm-ddTHH:MM:SSZ`. Setting and deleting these attribute is not automated in NooBaa and must be triggered within the system. In this project we provide a policy along with an external script to accomplish this. 58 | 59 | For example, when the attribute `user.noobaa.restore.expiry` is set to a future date and the attribute `user.noobaa.restore.request` is deleted, then the user can GET the object. The Head-Object request shows that there is no ongoing restore-request and that the restore can be performed until the expiry date (parameter `Restore` is set to `ongoing-request=false, expiry-date=some-date-in-the future` 60 | 61 | ``` 62 | # s3u1api head-object --bucket glacier1 --key coldfile0 63 | { 64 | "AcceptRanges": "bytes", 65 | "Restore": "ongoing-request=\"false\", expiry-date=\"Wed, 13 Mar 2024 00:00:00 GMT\"", 66 | "LastModified": "Mon, 11 Mar 2024 14:30:28 GMT", 67 | "ContentLength": 6449152, 68 | "ETag": "\"mtime-czqzravhx5a8-ino-mxj\"", 69 | "ContentType": "application/octet-stream", 70 | "Metadata": { 71 | "storage_class": "GLACIER" 72 | }, 73 | "StorageClass": "GLACIER" 74 | } 75 | 76 | ``` 77 | 78 | NooBaa represents the storage class and restore-request in extended attributes of the files associated with S3 Glacier objects. This enables automation of migration and recalls and facilitates tape optimized operations because multiple requests for recalls can be accumulated and executed as tape optimized recalls. 79 | 80 | The policies and script included in this project take care for migration, recalls and setting the NooBaa specific extended attributes. These policies and scripts can be simply automated and execute the S3 Glacier workflows unattended. 81 | 82 | 83 | 84 | # Policies and scripts 85 | 86 | In this section the policies and scripts are explained that facilitate migration, recalls and attribute setting for Glacier objects. 87 | 88 | 89 | ## Workflow 90 | 91 | The typical workflow for Glacier objects is the following 92 | 93 | 1. The S3 user issues a PUT request for an object with storage-class set to GLACIER into a bucket 94 | 2. An automated process migrates objects with storage-class set to GLACIER to tape. This is the [Migration](#Migration) process. 95 | 3. The S3 user issues a Restore-Object request for an object. 96 | 4. An automated process recalls objects that have a restore-request from tape. This is the [Recall](#Recall) process. 97 | 5. An automated process sets the attributes of objects that were recalled. This is the [Set attributes](#Set-attributes) process. 98 | 99 | 100 | 101 | ## Migration 102 | 103 | The migration of objects from the Storage Scale file system to tape is accommodated by the [migration policy](migrate.pol). This policy uses Storage Archive EE to perform the migration to tape. 104 | 105 | This migration policy migrates files matching the following conditions: 106 | storage_class attribute set to GLACIER AND 107 | NOT migrated AND 108 | ( ( user.noobaa.restore.request is set AND user.noobaa.restore.expiry is NOT set) OR user.noobaa.restore.expiry is expired ) 109 | 110 | This policy migrates all files that were newly ingested with storage-class set to Glacier and also files where the date and time encoded in attribute `user.noobaa.restore.expiry` is expired. 111 | 112 | 113 | ### Adjusting the migration policy 114 | 115 | The [migration policy](migrate.pol) must be adjusted. The EXTERNAL POOL rule must reflect the tape pool(s) used as destination for the migration. Adjust the parameter `-p poolname` whereby `poolname` is the name of the tape pools. If Storage Archive is configured with multiple libraries, then the pool name must also include the library name (e.g., `-p poolname@libname`). When multiple copies must be created on different tape pools, then this parameter must include all pool names (e.g., `-p poolname1@libname1 poolname2@libname2`) 116 | 117 | ``` 118 | RULE 'extPool' EXTERNAL POOL 'ltfs' EXEC '/opt/ibm/ltfsee/bin/eeadm' OPTS '-p poolname' 119 | ``` 120 | 121 | 122 | The second adjustment must be done in the MIGRATE rule. If objects must be migrated from specific filesets, then these fileset name must be added to the FOR FILESET() clause. In the example below, the migration scope is limited to `fileset1`and `fileset2`. If the scope not limited to filesets and applies to the entire file system, then the FOR FILESET() clause can be removed. 123 | 124 | ``` 125 | RULE 'migGlacier' MIGRATE FROM POOL 'system' TO POOL 'ltfs' FOR FILESET('fileset1' 'fileset1') WHERE 126 | ``` 127 | 128 | 129 | ### Executing the migration policy 130 | 131 | Once the migration policy was adjusted, it can be executed by the policy engine using the following command: 132 | 133 | ``` 134 | # mmapplypolicy [path-or-device] -P migrate.pol -m [threads] -N [archiveNodes] -B [bucket-size]--single-instance 135 | ``` 136 | 137 | The following parameters for the `mmapplypolicy` command must be considered: 138 | 139 | - `path-or-device` Path name of file system name. For example: `/ibm/fs1` 140 | - `-m threads` Number of parallel threads per node. This should be equivalent to the number of drives per node minus 1. 141 | - `-N nodes` Names of the Storage Archive nodes. 142 | - `-B bucket-size` Number of files to be migrated by one migrate operation. Depends on file size. A good value may be between 1000 and 20000. 143 | - `--single-instance` Ensures that not more than one instance of this policy is executed at a time. 144 | 145 | 146 | The execution of the migration policy may be scheduled to run 1 - 2 times a day, depending on the requirements. 147 | 148 | 149 | ## Recall 150 | 151 | The recall of objects from tape is accommodated by the [recall policy](recall.pol) which is executed by the Storage Scale policy engine. The recall policy is an EXTERNAL LIST policy that invokes the external script [recallGlacier.sh](recallGlacier.sh) to accommodate the recall of the selected files. 152 | 153 | The [recall policy](recall.pol) selects all files matching the following conditions: 154 | storage_class attribute set to GLACIER AND 155 | migrated AND 156 | user.noobaa.restore.request is set 157 | 158 | File names that are selected by the recall policy are packed in file list and passed on to the [recall script](recallGlacier.sh). This script performs the tape optimized recall of all files in the list by executing the Storage Archive `eeadm recall filelist` command. 159 | 160 | 161 | ### Adjusting the recall policy and script 162 | 163 | The [recall policy](recall.pol) must be adjusted. The EXTERNAL LIST rule must point to the exact path and file name where the [recall script](recallGlacier.sh) is located. In the example below the external script path and name is `/usr/local/bin/recallGlacier.sh`. 164 | 165 | ``` 166 | RULE 'extlist' EXTERNAL LIST 'recall' EXEC '/usr/local/bin/recallGlacier.sh' 167 | ``` 168 | 169 | The second adjustment must be done in the LIST rule. If objects must be recalled from specific filesets, then these fileset name must be added to the FOR FILESET() clause. In the example below, the recall scope is limited to `fileset1`and `fileset2`. If the scope is not limited to filesets and applies to the entire file system, then the FOR FILESET() clause can be removed. 170 | 171 | ``` 172 | RULE 'listRec' LIST 'recall' FOR FILESET('fileset1' 'fileset1') WHERE 173 | ``` 174 | 175 | 176 | In the [recall script](recallGlacier.sh), the variable `MYPATH` must be set to a valid path. The script will write the log file into this path. The path must be valid for each Storage Archive node. In the following example the log file path is set to `/var/log/glacier/recall`: 177 | 178 | ``` 179 | # define paths for log files and output files 180 | MYPATH="/var/log/glacier/recall" 181 | ``` 182 | 183 | 184 | ### Executing the recall policy and script 185 | 186 | Once the recall policy and script was adjusted, it can be executed by the policy engine using the following command: 187 | 188 | ``` 189 | # mmapplypolicy [path-or-device] -P recall.pol -m [threads] -N [archiveNodes] -B [bucket-size] --single-instance 190 | ``` 191 | 192 | The following parameters for the `mmapplypolicy` command must be considered: 193 | 194 | - `path-or-device` Path name of file system name. For example: `/ibm/fs1` 195 | - `-m threads` Number of parallel threads per node. This should be equivalent to the number of drives per node minus 1. 196 | - `-N nodes` Names of the Storage Archive nodes. 197 | - `-B bucket-size` Number of files to be migrated by one migrate operation. Depends on file size. A good value may be between 1000 and 20000. 198 | - `--single-instance` Ensures that not more than one instance of this policy is executed at a time. 199 | 200 | 201 | The schedule for the execution of the recall policy depends on the service level specifying the time period a S3 user must wait for the object to be retrievable after the user issued the Restore-Object request. Depending of the number of files in the file system and filesets, realistic time periods for retrieval may be 4 - 8 hours. If the time period for the retrieval is 4 hours, than the recall policy should be executed every 3 hours to give it some lead time for recalls. 202 | 203 | 204 | Note, that the [recall script](recallGlacier.sh) appends the output into a log file (recall.log) that is stored in path specified by `MYPATH`. Consider implementing log rotation to prevent the file system to be filled up. 205 | 206 | 207 | 208 | ## Set Attributes 209 | 210 | Setting the user.noobaa attributes is accommodated by the [set-attributes policy](setexpire.pol) that is executed by the Storage Scale policy engine. The set-attribute policy is an EXTERNAL LIST policy that invokes the external script [setExpire.sh](setExpire.sh) to accommodate setting the attributes of the selected files. 211 | 212 | The [set-attributes policy](setExpire.pol) selects all files matching the following conditions: 213 | storage_class attribute set to GLACIER AND 214 | NOT migrated AND 215 | user.noobaa.restore.request is set 216 | 217 | File names that are selected by the set-attributes policy are packed in file list and passed on to the [set-attributes script](setExpire.sh). For each file in the file list, the set-attributes script performs the following steps: 218 | - Calculates expiry-date based on current time plus the time period in days encoded in the attribute user.noobaa.restore.request 219 | - Set the attribute user.noobaa.restore.expiry to the value of the calculated expiry-date 220 | - Remove the attribute user.noobaa.restore.request 221 | 222 | 223 | To assure that recalled files (objects) are retrievable by the S3 user, **the set-attributes policy must be executed right after the recall**. This is because the recall is initiated for files that are migrated and have the attribute user.noobaa.restore.request set. After the recall, the files become retrievable as object via the S3 API when the attribute user.noobaa.restore.request is not present (or set to `false`) and then the date and time stamp encoded in attribute user.noobaa.restore.expire is in the future. This is taken care by the set-attributes policy and the set-attributes script. 224 | 225 | 226 | ### Adjusting the set-attribute policy and script 227 | 228 | The [set-attribute policy](recall.pol) must be adjusted. The EXTERNAL LIST rule must point to the exact path and file name of the[set-attributes script](setExpire.sh). In the example below the external script path and name is `/usr/local/bin/setExpire.sh`. 229 | 230 | ``` 231 | RULE 'extlist' EXTERNAL LIST 'setExpiry' EXEC '/usr/local/bin/setExpire.sh' 232 | ``` 233 | 234 | The second adjustment must be done in the LIST rule. If objects must be processed from specific filesets, then these fileset names must be added to the FOR FILESET() clause. In the example below the processing scope is limited to `fileset1`and `fileset2`. If the processing scope is not limited to fileset and applies to the entire file system, then the FOR FILESET() clause can be removed. 235 | 236 | ``` 237 | RULE 'listFiles' LIST 'setExpiry' FOR FILESET('fileset1' 'fileset1') WHERE 238 | ``` 239 | 240 | 241 | In the [set-attributes script](setExpire.sh), the variable `MYPATH` must be set to a valid path. The script will write the log file into this path. The path must be valid for each Storage Archive node. In the following example the log file path is set to `/var/log/glacier/setexpire`: 242 | 243 | ``` 244 | # define paths for log files and output files 245 | MYPATH="/var/log/glacier/setexpire" 246 | ``` 247 | 248 | 249 | ### Executing the set-attribute policy and script 250 | 251 | Once the set-attributes policy and script was adjusted, it can be executed by the policy engine using the following command: 252 | 253 | ``` 254 | # mmapplypolicy [path-or-device] -P setexpire.pol -m [threads] -N [Nodes] -B [bucket-size] --single-instance 255 | ``` 256 | 257 | The following parameters for the `mmapplypolicy` command must be considered: 258 | 259 | - `path-or-device` Path name of file system name. For example: `/ibm/fs1` 260 | - `-m threads` Number of parallel threads per node. This should be equivalent to the number of drives per node minus 1. 261 | - `-N nodes` Names of nodes that execute the policy and the script. This is **not** limited the Storage Archive nodes. 262 | - `-B bucket-size` Number of files to be migrated by one migrate operation. Depends on file size. A good value may be between 1000 and 20000. 263 | - `--single-instance` Ensures that not more than one instance of this policy is executed at a time. 264 | 265 | 266 | The schedule for the execution of the set-attribute policy depends on the schedule of the recall policy. The set-attribute policy must be executed right after the recall to assure that files get the right attributes set and can be retrieved by the S3 user. 267 | 268 | Note, that the [set-attribute script](setExpire.sh) appends the output into a log file (setexpire.log) that is stored in path specified by `MYPATH`. Consider implementing log rotation to prevent the file system to be filled up. 269 | 270 | -------------------------------------------------------------------------------- /noobaaGlacier/migrate.pol: -------------------------------------------------------------------------------- 1 | /* exclude rule */ 2 | RULE 'exclude' EXCLUDE WHERE 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%.mmbackupCfg/%' OR 7 | PATH_NAME LIKE '%/.snapshots/%' OR 8 | NAME LIKE '.mmbackupShadow%' OR 9 | NAME LIKE 'mmbackup%') 10 | 11 | /* define macro */ 12 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) 13 | 14 | /* Migrate policy for testing */ 15 | RULE 'extPool' EXTERNAL POOL 'ltfs' EXEC '/opt/ibm/ltfsee/bin/eeadm' OPTS '-p poolname' 16 | /* Limit bucket-size to 20 GB, value given in KB */ 17 | SIZE 20971520 18 | RULE 'migGlacier' MIGRATE FROM POOL 'system' TO POOL 'ltfs' FOR FILESET('buckets') WHERE 19 | FILE_SIZE > 0 AND 20 | xattr('user.storage_class') = 'GLACIER' AND 21 | NOT (is_migrated) AND 22 | (( xattr('user.noobaa.restore.request') IS NULL AND xattr('user.noobaa.restore.expiry') IS NULL ) OR 23 | CURRENT_TIMESTAMP >= TIMESTAMP(CONCAT(CONCAT(SUBSTR(xattr('user.noobaa.restore.expiry'), 0, 10), ' '), SUBSTR(xattr('user.noobaa.restore.expiry'), 12, 8))) ) 24 | 25 | /*Invokation: 26 | # mmapplypolicy [path-or-device] -P migrate.pol -m [drives-1] -N [archiveNodes] -B [bucket-size]--single-instance 27 | -m number of parallel threads per node, number of drives per node minus 1 28 | -N nodes exuting the policy, archive nodes 29 | -B bucket-size per thread, between 1000 and 20000 30 | */ 31 | -------------------------------------------------------------------------------- /noobaaGlacier/recall.pol: -------------------------------------------------------------------------------- 1 | /* exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%.mmbackupCfg/%' OR 7 | PATH_NAME LIKE '%/.snapshots/%' OR 8 | NAME LIKE '.mmbackupShadow%' OR 9 | NAME LIKE 'mmbackup%') 10 | ) 11 | 12 | /* define is_migrated */ 13 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) 14 | 15 | /* list files to be recalled and invoke the recall script */ 16 | RULE 'extlist' EXTERNAL LIST 'recall' EXEC '/path-to/recallGlacier.sh' 17 | RULE 'listRec' LIST 'recall' FOR FILESET('buckets') WHERE 18 | NOT (exclude_list) AND 19 | xattr('user.storage_class') = 'GLACIER' AND 20 | is_migrated AND 21 | xattr('user.noobaa.restore.request') IS NOT NULL 22 | 23 | 24 | /*Invokation: 25 | # mmapplypolicy [path-or-device] -P recall.pol -m [drives-1] -N [archiveNodes] -B [bucket-size] --single-instance 26 | -m number of parallel threads per node, number of drives per node minus 1 27 | -N nodes exuting the policy, archive nodes 28 | -B bucket-size per thread, between 1000 and 20000 29 | */ -------------------------------------------------------------------------------- /noobaaGlacier/recallGlacier.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2024 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: recall.sh 27 | # 28 | # Description: 29 | # Interface script for LIST policy invoked by mmapplypolicy 30 | # Performs recall of files provided in filelist 31 | # 32 | # Prerequisite: 33 | # EXTERNAL list policy that identifies files that must be processed 34 | # 35 | # Input: 36 | # invoked by mmapplypolicy with the following parameters: 37 | # $1 operation (list, test) 38 | # $2 file system name or name of filelist 39 | # $3 optional parameter defined in LIST policy under OPTS 40 | # 41 | # Processing: 42 | # recalls files in file list 43 | # 44 | # Output: 45 | # Write runtime information and debugging messages to log file $LOGFILE 46 | # 47 | # Example Policy: 48 | # /* list files to be recalled */ 49 | # RULE 'extlist' EXTERNAL LIST 'recall' EXEC '' 50 | # RULE 'listRec' LIST 'recall' FOR FILESET('buckets') WHERE 51 | # NOT (exclude_list) AND 52 | # xattr('user.storage_class') = 'GLACIER' AND 53 | # is_migrated AND 54 | # xattr('user.noobaa.restore.request') IS NOT NULL 55 | # 56 | # 57 | # Change History 58 | # 7.5.2024 first implementation based on receiver script 59 | 60 | #global variables for this script 61 | #---------------------------------- 62 | # define paths for log files and output files 63 | MYPATH="./recall" 64 | # logfile used for system_log function 65 | LOGFILE=$MYPATH/"recall.log" 66 | # sets the log level for the system log, everything below that number is logged 67 | LOGLEVEL=1 68 | # set the default option for the file list processing in case $3 is not given 69 | DEFOPTS="" 70 | 71 | #Program specific global vars: 72 | #------------------------------ 73 | # global vars 74 | restoreAttrName="user.noobaa.restore.request" 75 | expireAttrName="user.noobaa.restore.expiry" 76 | gpfsPath="/usr/lpp/mmfs/bin" 77 | eePath="/opt/ibm/ltfsee/bin" 78 | 79 | 80 | #=================================================================================== 81 | # Functions 82 | #=================================================================================== 83 | 84 | ## Append to the system log 85 | ## Usage: system_log 86 | system_log () { 87 | SEV=$1 88 | 89 | case $SEV in 90 | [0-9]) ;; 91 | *) SEV=1;; 92 | esac 93 | 94 | LINE=$2 95 | if [ $LOGLEVEL -ge $SEV ] ; then 96 | if [[ -z "$LINE" ]]; then 97 | echo -e "RECALL INTERNAL WARNING: Improper value given to system_log function ($@)" >> $LOGFILE 98 | else 99 | echo -e "RECALL: $LINE" >> $LOGFILE 100 | fi 101 | fi 102 | } 103 | 104 | ## Print a message to the stdout 105 | ## Usage: user_log 106 | user_log () { 107 | echo -e "RECALL $@" 108 | } 109 | 110 | 111 | ## Get current date and time 112 | ## Usage: get_cur_date_time 113 | get_cur_date_time(){ 114 | echo "$(date +"%Y-%b-%d %H:%M:%S")" 115 | } 116 | 117 | #=================================================================================== 118 | # Main 119 | #=================================================================================== 120 | user_log "$(get_cur_date_time) recall.sh invoked by policy engine" 121 | 122 | # check the path for logging 123 | if [[ ! -d $MYPATH ]] then 124 | mkdir -p $MYPATH 125 | rc=$? 126 | if (( rc > 0 )) then 127 | system_log 1 "ERROR: failed to create directory $MYPATH, check permissions" 128 | user_log "ERROR: failed to create directory $MYPATH, check permissions" 129 | exit 1 130 | fi 131 | fi 132 | 133 | system_log 1 "=========================================================================" 134 | system_log 1 "$(get_cur_date_time) recall invoked with arguments: $*" 135 | 136 | ## Parse Arguments & execute 137 | #$1 is the policy operation (list, migrate, etc) 138 | op=$1 139 | #$2 is the policy file name 140 | polFile=$2 141 | #$3 is the option given in the EXTERNAL LIST rule with OPTS '..' should be retention time here 142 | option=$3 143 | 144 | ## this is required, as the script may be called multiple times during 145 | ## the same backup (if there are too many files to process). 146 | 147 | case $op in 148 | TEST ) 149 | user_log "INFO: TEST option received for directory $polFile." 150 | system_log 1 "INFO: TEST option received for $polFile" 151 | if [[ ! -z "$polFile" ]] then 152 | if [[ -d "$polFile" ]] then 153 | user_log "INFO: TEST directory $polFile exists." 154 | system_log 1 "INFO: Directory $polFile exists." 155 | else 156 | user_log "WARNING: TEST directory $polFile does not exists." 157 | system_log 1 "WARNING: Directory $polFile does not exist." 158 | fi 159 | fi 160 | ;; 161 | LIST ) 162 | user_log "INFO: LIST option received, starting recall task" 163 | system_log 1 "INFO: LIST option received with file name $polFile and options $option" 164 | 165 | #set option to default if not set 166 | if [[ -z $option ]] then 167 | option=$DEFOPTS 168 | fi 169 | 170 | # process the files 171 | recallRc=0 172 | numEntries=$(wc -l $polFile | awk '{print $1}') 173 | system_log 1 "INFO: Recalling $numEntries files" 174 | user_log "INFO: Recalling $numEntries files" 175 | $eePath/eeadm recall $polFile 2>&1 >> $LOGFILE 176 | recallRc=$? 177 | 178 | ##### 179 | # evaluate return code and log message 180 | if (( recallRc > 0 )); then 181 | system_log 1 "WARNING: Recall ended with return code $recallRc" 182 | user_log "WARNING: Recall ended with return code $recallRc" 183 | fi 184 | ;; 185 | REDO ) 186 | user_log "INFO: REDO option received, doing nothing" 187 | system_log 1 "INFO: REDO option received with file name $polFile and options $option" 188 | ;; 189 | * ) 190 | user_log "WARNING: Unknown option $op received, doing nothing" 191 | system_log 1 "WARNUNG: UNKNOWN option ($op) received with file name $polFile and options $option" 192 | ;; 193 | esac 194 | 195 | user_log "$(get_cur_date_time) recall ended" 196 | system_log 1 "$(get_cur_date_time) recall ended" 197 | 198 | # exit 0 if things are OK 199 | exit 0 -------------------------------------------------------------------------------- /noobaaGlacier/setExpire.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2024 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: setExpire.sh 27 | # 28 | # Description: 29 | # Interface script for LIST policy invoked by mmapplypolicy 30 | # Sets the user.noobaa.restore.expiry attribute if user.noobaa.restore.request is set and files are not migrated 31 | # 32 | # Prerequisite: 33 | # EXTERNAL list policy that identifies files that must be processed 34 | # 35 | # Input: 36 | # invoked by mmapplypolicy with the following parameters: 37 | # $1 operation (list, test) 38 | # $2 file system name or name of filelist 39 | # $3 optional parameter defined in LIST policy under OPTS 40 | # 41 | # Processing: 42 | # calculate the expiration time based on user.noobaa.restore.request attribute 43 | # sets the expiration time in user.noobaa.restore.expiry attribute 44 | # deletes user.noobaa.restore.request attribute 45 | # 46 | # Output: 47 | # Write runtime information and debugging messages to log file $LOGFILE 48 | # 49 | # Example Policy: 50 | # RULE 'extlist' EXTERNAL LIST 'restNotMig' EXEC '/root/silo/tapecloud/policies/setExpire.sh' 51 | # RULE 'listRest' LIST 'restNotMig' FOR FILESET('buckets') WHERE 52 | # NOT (exclude_list) AND 53 | # xattr('user.storage_class') = 'GLACIER' AND 54 | # NOT (is_migrated) AND 55 | # xattr('user.noobaa.restore.request') IS NOT NULL 56 | # 57 | # Change History 58 | # 6.5.2024 first implementation based on receiver script 59 | 60 | 61 | #global variables for this script 62 | #---------------------------------- 63 | # define paths for log files and output files 64 | MYPATH="./setExpire" 65 | # logfile used for system_log function 66 | LOGFILE=$MYPATH/"setExpire.log" 67 | # sets the log level for the system log, everything below that number is logged 68 | LOGLEVEL=1 69 | # set the default option for the file list processing in case $3 is not given 70 | DEFOPTS="" 71 | 72 | #Program specific global vars: 73 | #------------------------------ 74 | # global vars 75 | restoreAttrName="user.noobaa.restore.request" 76 | expireAttrName="user.noobaa.restore.expiry" 77 | gpfsPath="/usr/lpp/mmfs/bin" 78 | 79 | 80 | #=================================================================================== 81 | # Functions 82 | #=================================================================================== 83 | 84 | ## Append to the system log 85 | ## Usage: system_log 86 | system_log () { 87 | SEV=$1 88 | 89 | case $SEV in 90 | [0-9]) ;; 91 | *) SEV=1;; 92 | esac 93 | 94 | LINE=$2 95 | if [ $LOGLEVEL -ge $SEV ] ; then 96 | if [[ -z "$LINE" ]]; then 97 | echo -e "SETEXPIRE INTERNAL WARNING: Improper value given to system_log function ($@)" >> $LOGFILE 98 | else 99 | echo -e "SETEXPIRE: $LINE" >> $LOGFILE 100 | fi 101 | fi 102 | } 103 | 104 | ## Print a message to the stdout 105 | ## Usage: user_log 106 | user_log () { 107 | echo -e "SETEXPIRE $@" 108 | } 109 | 110 | 111 | ## Get current date and time 112 | ## Usage: get_cur_date_time 113 | get_cur_date_time(){ 114 | echo "$(date +"%Y-%b-%d %H:%M:%S")" 115 | } 116 | 117 | #=================================================================================== 118 | # Main 119 | #=================================================================================== 120 | user_log "$(get_cur_date_time) setExpire.sh invoked by policy engine" 121 | 122 | # check the path for logging 123 | if [[ ! -d $MYPATH ]] then 124 | mkdir -p $MYPATH 125 | rc=$? 126 | if (( rc > 0 )) then 127 | system_log 1 "ERROR: failed to create directory $MYPATH, check permissions" 128 | user_log "ERROR: failed to create directory $MYPATH, check permissions" 129 | exit 1 130 | fi 131 | fi 132 | 133 | system_log 1 "=========================================================================" 134 | system_log 1 "$(get_cur_date_time) setExpire invoked with arguments: $*" 135 | 136 | ## Parse Arguments & execute 137 | #$1 is the policy operation (list, migrate, etc) 138 | op=$1 139 | #$2 is the policy file name 140 | polFile=$2 141 | #$3 is the option given in the EXTERNAL LIST rule with OPTS '..' should be retention time here 142 | option=$3 143 | 144 | ## this is required, as the script may be called multiple times during 145 | ## the same backup (if there are too many files to process). 146 | 147 | case $op in 148 | TEST ) 149 | user_log "INFO: TEST option received for directory $polFile." 150 | system_log 1 "INFO: TEST option received for $polFile" 151 | if [[ ! -z "$polFile" ]] then 152 | if [[ -d "$polFile" ]] then 153 | user_log "INFO: TEST directory $polFile exists." 154 | system_log 1 "INFO: Directory $polFile exists." 155 | else 156 | user_log "WARNING: TEST directory $polFile does not exists." 157 | system_log 1 "WARNING: Directory $polFile does not exist." 158 | fi 159 | fi 160 | ;; 161 | LIST ) 162 | user_log "INFO: LIST option received, starting setExpire task" 163 | system_log 1 "INFO: LIST option received with file name $polFile and options $option" 164 | 165 | #set option to default if not set 166 | if [[ -z $option ]] then 167 | option=$DEFOPTS 168 | fi 169 | 170 | # process the files 171 | itemNum=0 172 | numEntries=$(wc -l $polFile | awk '{print $1}') 173 | system_log 1 "INFO: Start processing $numEntries files" 174 | user_log "INFO: Start processing $numEntries files" 175 | cat $polFile | while read line 176 | do 177 | # use set to get file name, does tolerate blanks 178 | set $line 179 | shift 4 180 | fName="$*" 181 | 182 | # get restore.request days 183 | reqDays=$($gpfsPath/mmlsattr -n $restoreAttrName $fName | grep "$restoreAttrName" | cut -d'"' -f 2 2>>$LOGFILE) 184 | 185 | # calculate expiration time 186 | expDate=$(date +"%Y-%m-%dT%H:%M:%S.000Z" -d "$DATE + $reqDays day") 187 | system_log 1 "DEBUG: Processing file $fName with expiring in $reqDays at $expDate" 188 | 189 | # set expiration time 190 | $gpfsPath/mmchattr --set-attr $expireAttrName="$expDate" $fName 2>&1 >> $LOGFILE 191 | 192 | # remove restore attr 193 | $gpfsPath/mmchattr --delete-attr $restoreAttrName $fName 2>&1 >> $LOGFILE 194 | 195 | ((itemNum=itemNum+1)) 196 | done 197 | system_log 1 "INFO: Processed $itemNum out of $numEntries files" 198 | user_log "INFO: Processed $itemNum out of $numEntries files" 199 | ;; 200 | REDO ) 201 | user_log "INFO: REDO option received, doing nothing" 202 | system_log 1 "INFO: REDO option received with file name $polFile and options $option" 203 | ;; 204 | * ) 205 | user_log "WARNING: Unknown option $op received, doing nothing" 206 | system_log 1 "WARNUNG: UNKNOWN option ($op) received with file name $polFile and options $option" 207 | ;; 208 | esac 209 | 210 | user_log "$(get_cur_date_time) setExpire ended" 211 | system_log 1 "$(get_cur_date_time) setExpire ended" 212 | 213 | # exit 0 if things are OK 214 | exit 0 -------------------------------------------------------------------------------- /noobaaGlacier/setexpire.pol: -------------------------------------------------------------------------------- 1 | /* exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%.mmbackupCfg/%' OR 7 | PATH_NAME LIKE '%/.snapshots/%' OR 8 | NAME LIKE '.mmbackupShadow%' OR 9 | NAME LIKE 'mmbackup%') 10 | ) 11 | 12 | /* define is_migrated */ 13 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) 14 | 15 | /* list files with restore request that are not migrated and invoke setExpire script to set user.noobaa.restore.expiry and remove user.noobaa.restore.request */ 16 | RULE 'extlist' EXTERNAL LIST 'setExpiry' EXEC '/path-to/setExpire.sh' 17 | RULE 'listFiles' LIST 'setExpiry' FOR FILESET('buckets') WHERE 18 | NOT (exclude_list) AND 19 | xattr('user.storage_class') = 'GLACIER' AND 20 | NOT (is_migrated) AND 21 | xattr('user.noobaa.restore.request') IS NOT NULL 22 | 23 | /*Invokation: 24 | # mmapplypolicy [path-or-device] -P setexpire.pol -m [num] -N [nodes] -B [bucket-size] --single-instance 25 | -m number of parallel threads per node 26 | -N nodes exuting the policy 27 | -B bucket-size per thread, between 1000 and 20000 28 | */ 29 | 30 | -------------------------------------------------------------------------------- /premigrate/ltfsee_premig.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2020 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: ltfsee_premigrate 27 | # 28 | # Description: 29 | # Interface script for MIGRATE policy invoked by mmapplypolicy 30 | # Invokes ltfsee premigrate -s filelist -p pool@library 31 | # 32 | # Prerequisite: 33 | # EXTERNAL pool policy that identifies files to be premigrated. 34 | # 35 | # Input: 36 | # invoked by mmapplypolicy with the following parameters: 37 | # $1 operation (list, test) 38 | # $2 file system name or name of filelist 39 | # $3 target for migration as pool@library name 40 | # 41 | # Output: 42 | # Writes runtime information to STDOUT - ends up in mmapplypolicy output 43 | # 44 | # 45 | # Invokation: 46 | # mmapplypolicy fsname -P policyfile 47 | # 48 | # Change History 49 | # 10/09/12 first implementation based GAD startbackup 50 | # 12/20/15 implementation for immutability, some streamlining of existing code 51 | # 12/21/15 create the general receiver 52 | # 08/23/17 ltfsee premigrate 53 | # 08/31/17 streamline 54 | # 06/23/20 adopt to eeadm command, improve messaging 55 | 56 | #global variables for this script 57 | # set the default option in case $3 is not give, allows to specify the pool in the syntax "-p pool@lib", just in case the pool is not set in the external pool definition of the migrate policy 58 | DEFOPTS="" 59 | # define ltfsee directory 60 | LTFSEEDIR=/opt/ibm/ltfsee/bin 61 | 62 | #++++++++++++++++++++++++++ MAIN ++++++++++++++++++++++++++++++++++++++ 63 | echo "================================================================================================" 64 | echo "$(date +"%Y-%b-%d %H:%M:%S") LTFSEE_PREMIG invoked with arguments: $*" 65 | 66 | ## Parse Arguments & execute 67 | #$1 is the policy operation (list, migrate, etc) 68 | op=$1 69 | #$2 is the policy file name 70 | polFile=$2 71 | #$3 is the option given in the EXTERNAL LIST rule with OPTS '..' should be the pool 72 | shift 2 73 | option=$* 74 | 75 | ## evaluate the operation passed by mmapplypolicy and act upon it 76 | case $op in 77 | # there will always be a TEST call with $2 being the file system path 78 | TEST ) 79 | echo "LTFSEE_PREMIG INFO: TEST option received for $polFile" 80 | if [[ ! -z "$polFile" ]] then 81 | if [[ ! -d "$polFile" ]] then 82 | echo "LTFSEE_PREMIG WARNING: TEST directory $polFile does not exists." 83 | fi 84 | fi 85 | ;; 86 | MIGRATE ) 87 | # this is the actual migrate call where we call ltfsee premigrate 88 | echo "LTFSEE_PREMIG INFO: MIGRATE option received with file name $polFile and options $option" 89 | #set option to default if not set 90 | if [[ -z $option ]] then 91 | if [[ -z $DEFOPTS ]] then 92 | echo "LTFSEE_PREMIG ERROR: Pool name not specified in the external pool rule." 93 | exit 1 94 | else 95 | option=$DEFOPTS 96 | fi 97 | fi 98 | 99 | echo "LTFSEE_PREMIG INFO: Start processing files $polFile with ltfsee premigrate to pool $option" 100 | $LTFSEEDIR/eeadm premigrate $polFile $option 101 | rc=$? 102 | if (( rc != 0 )) then 103 | echo "LTFSEE_PREMIG WARNING: ltfsee premigrated ended with return code $rc" 104 | fi 105 | ;; 106 | REDO ) 107 | echo "LTFSEE_PREMIG INFO: REDO option received with file name $polFile and options $option" 108 | ;; 109 | * ) 110 | echo "LTFSEE_PREMIG WARNING: UNKNOWN operation ($op) received with file name $polFile and options $option" 111 | ;; 112 | esac 113 | 114 | echo "$(date +"%Y-%b-%d %H:%M:%S") LTFSEE_PREMIG ended" 115 | echo 116 | exit 0 117 | -------------------------------------------------------------------------------- /premigrate/premig_fset_all.txt: -------------------------------------------------------------------------------- 1 | /* Use external script to premigrate all resident files to LTFS */ 2 | 3 | /* define exclude rule*/ 4 | RULE 'exclude' EXCLUDE WHERE 5 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 6 | PATH_NAME LIKE '%/.ltfsee/%' OR 7 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 8 | PATH_NAME LIKE '%/.snapshots/%' OR 9 | NAME LIKE '.mmbackupShadow%' OR 10 | NAME LIKE 'mmbackup%') 11 | 12 | 13 | /* define macros */ 14 | define( is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%') ) 15 | define( is_empty,(KB_ALLOCATED=0) ) 16 | 17 | /* Define external pool ltfsPremig with customized interface script that does premigrate instead of migrated*/ 18 | RULE EXTERNAL POOL 'ltfsPremig' 19 | EXEC '/root/silo/premig/ltfsee_premig.sh' /* full path to ltfsee command must be specified */ 20 | OPTS '-p test@eelib1' /* this is our pool in EE which is given by the runpolicy script*/ 21 | 22 | /* (Pre)Migrate everything from system to ltfsPremig for fileset test that is resident and no 0 bytes*/ 23 | RULE 'PMigToExt' MIGRATE FROM POOL 'system' TO POOL 'ltfsPremig' FOR FILESET ('test') 24 | WHERE ( (is_resident) AND NOT (is_empty) ) 25 | 26 | -------------------------------------------------------------------------------- /premigrate/readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Script: ltfsee_premig.sh 3 | 4 | ## Description 5 | 6 | This script can be used as an interface script in combination with a migration policy to pre-migrate files from IBM Spectrum Scale to IBM Spectrum Archive EE. This script receives the file list from the policy engine according to the MIGRATE rule and feeds this into the `ltfsee_premig.sh` script. 7 | 8 | 9 | ## Prerequisite 10 | 11 | Copy the `ltfsee_premig.sh` script to a directory of the executing EE node. To run the policy on multiple EE nodes the script must be placed in the same directory on each node. Consider placing it in a shared file system. 12 | 13 | Adjust the premigration policy according to your needs and store it in the same directory as the `ltfsee_premig.sh` script. The file [premig_fset_all.txt](premig_fset_all.txt) is an example for a premigration policy. 14 | 15 | Note, the directory where the `ltfsee_premig.sh` script is stored must be entered into to the EXEC clause of the EXTERNAL POOL rule 16 | 17 | 18 | ## Invokation 19 | 20 | The premigration script `ltfsee_premig.sh` is invoked with the policy engine: 21 | 22 | 23 | # mmapplypolicy fsname -P policyfile -N nodenames --single-instance 24 | 25 | Options: 26 | fsname is the file system name, file system path or the file system path with a subdirectory. 27 | -P policyfile is the policy file including the EXTERNAL POOL rule specifying the `ltfsee_premig.sh` interface script and the MIGRATE rule. 28 | -N nodenames node name or node class name that executes this policy. Must be Spectrum Archive EE nodes 29 | --single-instance run only one instance of this policy. 30 | 31 | 32 | An example for the `policyfile` in file [premig_fset_all.txt](premig_fset_all.txt). Further options can be specified with the `mmapplypolicy` command. 33 | 34 | 35 | ## Processing 36 | 37 | The policy engine (`mmapplypolicy`) identifies files according to the migrate rule and passes these files to the interface script `ltfsee_premig.sh`. The interface script `ltfsee_premig.sh` premigrates the selected files using the command: `eeadm premigrate filelist -p pool@library`. 38 | 39 | 40 | ## Output 41 | 42 | Output of the script is logged to STDOUT and ends up within the output of the mmapplypolicy command 43 | 44 | 45 | -------------------------------------------------------------------------------- /quota-migration/README.md: -------------------------------------------------------------------------------- 1 | ## Description 2 | 3 | Spectrum Scale Information Lifecycle Management policies can be configured to automatically migrate files from one file system pool to another if a pre-defined occupation limit of a pool is met. For example in an environment with two file system pools (e.g. system and silver) files can be placed in pool `system` and if pool `system` reaches an occupation of 80% then files can be migrated to pool `silver`. This migration policy however takes the entire file system occupation into account and not the occupation of a particular fileset. 4 | 5 | To take the occupation of a fileset into account, the standard ILM policies are not effective. However it is possible to migrate files pertaining to a particular fileset if the fileset soft quota limit is met. Migrating files pertaining to a particular fileset from one file system pool to another only makes sense if the files for a particular fileset are placed in a particular pool. For example, all files for a fileset name fset1 can be placed in pool `system`. 6 | 7 | This project provides a description of the methodology along with some sample code and policies to accomplish this. 8 | 9 | 10 | ### Methodology 11 | 12 | To migrate files pertaining to a particular fileset of a Spectrum Scale file system when the fileset has reached the pre-defined soft quota limit the following components are required: 13 | 14 | 1. An EXTERNAL LIST policy [quota-listpol.txt](quota-listpol.txt) that identifies files according to quota limits using the clause `THRESHOLD (resourceclass)` in the EXTERNAL LIST rule and the clause THRESHOLD(%high, %low) in the LIST rule. The parameter `resourceclass` in the EXTERNAL LIST rule can be `FILESET_QUOTAS` for hard quota limits or `FILESET_QUOTA_SOFT` for soft quota limits. Files are identified if the quota occupation in the fileset is above the THRESHOLD defined in the LIST rule. The LIST policy identifies files located in pool `system` that brings the occupation in the fileset down to a low threshold. The LIST policy stores the list of identified files in a pre-defined location. This list of files is used as input for the MIGRATION policy. 15 | 16 | 2. A MIGRATE policy [quota-migpol.txt](quota-migpol.txt) takes the list of files generated by the EXTERNAL LIST policy, evaluates these against the migration rule and migrates selected files from pool `system` to pool `ltfs` that is managed by IBM Spectrum Archive Enterprise Edition. 17 | 18 | 3. A callback script [callback-quota.sh](callback-quota.sh) that is invoked when the Spectrum Scale event `softQuotaExceeded` is raised for a fileset. This callback script runs the EXTERNAL LIST policy of step 1 to identify files according to the quota limits and invokes the migration policy of step 2 with the files being identified. Thus, the callback script is invoked when the event `softQuotaExceeded` is triggered and coordinates the LIST and MIGRATE policy runs. 19 | 20 | 4. Last but not least, the Spectrum Scale callback is defined that receives the event `softQuotaExceeded` and runs the callback script `callback-quota.sh`passing on certain parameters including the file system and fileset name that triggered this even Quota callback. 21 | 22 | 23 | * Note: * 24 | The event `softQuotaExceeded` is triggered on the file system manager. Any node with a manager role in the cluster can become file system manager. Therefore all components explained above have to be installed on all nodes with manager role. 25 | 26 | 27 | The next sections explain how to install and configure these components. 28 | 29 | 30 | ## Preparation 31 | 32 | Setup the placement policy for the filesets placing all files stored in this fileset directory to a particular pool. The following placement policy example places all files stored in fileset `test` in the pool `system`. All other files are placed in pool `data`: 33 | 34 | /* fileset placement rule */ 35 | RULE 'placefset' SET POOL 'system' FOR FILESET ('test') 36 | /* default placement rule */ 37 | RULE 'default' SET POOL 'data' 38 | 39 | 40 | Configure quota for the filesets that should be migrated when the quota limit is reached. 41 | 42 | 43 | Copy the `callback-quota.sh` script and the two policy files (`quota-listpol.txt` and `quota-migpol.txt`) into the same directory on all nodes in your cluster that have manager role. Since all nodes in a cluster have access to the common Spectrum Scale file systems you can copy these files into a directory of the Spectrum Scale file system. 44 | 45 | 46 | Adjust configurable parameters in the `callback-quota.sh` script: 47 | 48 | 49 | | Parameter | Description | 50 | | ----------|-------------| 51 | | workDir | define the working directory where output files of the policy runs are stored. It may also be used to define the path names of the policy files. This directory must exist. | 52 | | logDir | define the directory where the log files are stored. This directory must exist. | 53 | | logF | define the file name of the log file. The log file is stored under directory specified by `logDir` and is continuously appended. Configure log rotation for this file when required. | 54 | | outfile | define the path and file name prefix of the file list created by the EXTERNAL LIST policy. | 55 | | listPol | define the path and file name of the policy file including the EXTERNAL LIST policy [quota-listpol.txt](quota-listpol.txt) | 56 | | migPol | define the path and file name of the policy file including the MIGRATE policy [quota-migpol.txt](quota-migpol.txt) | 57 | | migPolOpts | define options for the `mmapplypolicy` command running the migration. When migrating to Spectrum Archive these should include the EE node names (`-N`), the number of threads (`-m`), the bucket size (`-B`), etc. | 58 | | eePool | define the pool name for the tape pool when migrating to Spectrum Archive EE. The tape pool name given here in the syntax `pool@library` substitutes the parameter `EEPOOL`in the migrate policy [quota-migpol.txt](quota-migpol.txt). When not migrating to Spectrum Archive EE then leave this parameter blank and adjust the migrate policy. | 59 | 60 | 61 | 62 | Adjust the policy file [quota-listpol.txt](quota-listpol.txt) according to your needs. 63 | 64 | 65 | Adjust the policy file [quota-migpol.txt](quota-migpol.txt) according to your needs 66 | 67 | 68 | Create the Spectrum Scale callback: 69 | 70 | # mmaddcallback SOFTQUOTA-MIGRATION --command /path/callback-quota.sh --event softQuotaExceeded --parms "%eventName %fsName %filesetName" 71 | 72 | The callback is invoked when the event `softQuotaExceeded` is triggered for the file system on the file system manager. The callback launches the script `/path/callback-quota.sh` (the path must be spelled out and it must be identical for all manager nodes). The callback passes the following parameters to the callback script: 73 | `eventName` is the event `softQuotaExceeded` 74 | `fsName` file system name where the event was triggerd for 75 | `fsetname` fileset name where the event was triggered for. If the event was triggered for the entire file system then the fileset name is `root`. 76 | 77 | 78 | Test if the callback script is effective. For example by temporarily reducing the soft quota limit and storing some files in the fileset. Monitor the GPFS logs on the active file system manager for the file system to see that the event `softQuotaExceeded` is triggered and that the callback script `callback-quota.sh` is launched and files are migrated. 79 | 80 | 81 | ## Processing 82 | 83 | The `callback-quota.sh` script performs the following steps: 84 | - check if preconfigured parameters are correct 85 | - check if parameters passed by the callback are correct 86 | - run the EXTERNAL LIST policy and create list of selected files based on quota consumption 87 | - run the MIGRATE policy using the file list produced by the EXTERNAL LIST policy 88 | 89 | 90 | ## Output 91 | 92 | The `callback-quota.sh` writes STDIN and STDERR to the log file defined by parameter `$logF` 93 | 94 | Return codes: 95 | 96 | 0: Good 97 | 98 | 1: Error 99 | 100 | 101 | ## Notes 102 | 103 | - The callback is configured cluster wide for all nodes. 104 | - The callback script along with the policy files must be installed on all manager nodes because the event is triggered on the file system manager only. 105 | - The event "softQuotaExceeded" is only triggered once per fileset when the condition is met. It expects the space consumption to decrease under the quota limits. If this is the case and after a while the quota limit is reached again then this event is triggered again. Otherwise, if the space consumption does not decrease then the event might not be triggered again. Therefore it is important to make sure that the callback script works 100 % and that it alerts the admin if not. 106 | - In order to retrigger the event the softquota limits can be increased or files can be move out and back in again. Some delay between moving files out of the files and in should be planned (5 - 10 min). 107 | - Consider placing the temporary file generated by mmapplypolicy in a directory with sufficient space. Use the parameter -s with the mmapplypolicy command for this. 108 | 109 | -------------------------------------------------------------------------------- /quota-migration/callback-quota.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2020 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: callback-quota.sh 27 | # 28 | # Description: 29 | # ------------- 30 | # This script along with some policies can accomplish fileset level migration when the quota limit of a fileset is met. 31 | # Script that can be invoked by a Spectrum Scale callback upon the event softQuotaExceeded. 32 | # Executes a list policies that identifies files that are above the quota limit. 33 | # Executes a MIGRATE policy that migrates the files identified by the list policy. 34 | # 35 | # Prerequisite: 36 | # ------------- 37 | # Filesets should be aligned to file system pools using placement policies 38 | # Fileset quota limits must be configured 39 | # This script along with the required policy files must be available on all manager nodes 40 | # callback catching the event softQuotaExceeded must be configured: 41 | # mmaddcallback SOFTQUOTA-MIGRATION --command /path/callback-quota.sh --event softQuotaExceeded --parms "%eventName %fsName %filesetName" 42 | # 43 | # Input: 44 | # ------------- 45 | # invoked by the callback with the following parameters: 46 | # eventName: is the event `softQuotaExceeded` 47 | # fsName: file system name where the event was triggerd for 48 | # fsetname: fileset name where the event was triggered for. If the event was triggered for the entire file system then the fileset name is `root`. 49 | # 50 | # Output: 51 | # ------- 52 | # Identifies and migrates file for the fileset that triggered the event softQuotaExceeded 53 | # Write runtime information and debugging messages to log file $logF 54 | # 55 | # Change History 56 | # 08/29/19 first implementation 57 | # 07/21/20 streamlining, removed nodeclass and ssh to node and added migrate policy options 58 | # 59 | # 60 | # ---------------------------------------------------------------------------- 61 | # define configurable parameters 62 | # ---------------------------------------------------------------------------- 63 | # the workdir must exist on all manager nodes who can become file system manager 64 | workDir="/root/silo/quotamigration" 65 | # name of the file generated by the external list policy 66 | outFile=$workDir"/qFiles" 67 | # name of the LIST policy. The policy files must be accessible on all manager nodes 68 | listPol=$workDir"/quota-listpol.txt" 69 | # name of the MIGRATE policy. The policy files must be accessible on all manager nodes 70 | migPol=$workDir"/quota-migpol.txt" 71 | # the log directory must exist 72 | logDir="/root/silo/quotamigration/log" 73 | # The log file is appended to, you have to take care of logfile rotation 74 | logF=$logDir"/quota-callback.log" 75 | # define default options for the migration policy run 76 | migPolOpts="-N ltfsee-1 -m 1 -B 1000" 77 | # define the pool name for the EE Pool when migrating to Spectrum Archive EE 78 | eePool="test@eelib1" 79 | 80 | # ---------------------------------------------------------------------------- 81 | # define constants 82 | # ---------------------------------------------------------------------------- 83 | # define GPFS binary path 84 | gpfsPath="/usr/lpp/mmfs/bin" 85 | 86 | # ---------------------------------------------------------------------------- 87 | # assign parameters given to the script 88 | # ---------------------------------------------------------------------------- 89 | evName=$1 90 | fsName=$2 91 | fsetName=$3 92 | 93 | 94 | # check prerequisites 95 | if [[ ! -d "$logDir" ]]; 96 | then 97 | echo "ERROR: logging directory $logDir does not exist. exiting." 98 | exit 1 99 | fi 100 | 101 | # present a banner 102 | echo -e "\n\n==============================================================================" >> $logF 103 | echo "$(date) Program $0 started on $(hostname)" | tee -a $logF 104 | echo -e "------------------------------------------------------------------------------" >> $logF 105 | 106 | # check more prerequisites 107 | if [[ ! -d "$workDir" ]]; 108 | then 109 | echo "ERROR: working directory $workDir does not exist. exiting." | tee -a $logF 110 | exit 1 111 | fi 112 | if [[ ! -a "$listPol" ]]; 113 | then 114 | echo "ERROR: List policy file $listPol does not exist. exiting." | tee -a $logF 115 | exit 1 116 | fi 117 | if [[ ! -a "$migPol" ]]; 118 | then 119 | echo "ERROR: Migrate policy file $MigPol does not exist. exiting." | tee -a $logF 120 | exit 1 121 | fi 122 | 123 | #check parameters given to the script 124 | if [[ -z $evName ]]; 125 | then 126 | echo "ERROR: event name not specified." | tee -a $logF 127 | echo "Syntax: $0 event-name filesystem-name fileset-name" | tee -a $logF 128 | exit 1 129 | fi 130 | if [[ -z $fsName ]]; 131 | then 132 | echo "ERROR: filesystem name not specified." | tee -a $logF 133 | echo "Syntax: $0 event-name filesystem-name fileset-name" | tee -a $logF 134 | exit 1 135 | fi 136 | if [[ -z $fsetName ]]; 137 | then 138 | echo "ERROR: fileset name not specified." | tee -a $logF 139 | echo "Syntax: $0 event-name filesystem-name fileset-name" | tee -a $logF 140 | exit 1 141 | fi 142 | 143 | echo "INFO: received event $evName for file system $fsName and fileset $fsetName" | tee -a $logF 144 | 145 | # set the name of the output file to qfiles-fsname-fsetname 146 | outFile=$outFile"-"$fsName"-"$fsetName 147 | 148 | # delete last output file and invoke list policy 149 | echo "INFO: running list policy for filesystem $fsName and file set $fsetName" >> $logF 150 | rm -f $outFile.list.softquota >> $logF 2>&1 151 | rm -f $outFile.miglist >> $logF 2>&1 152 | $gpfsPath/mmapplypolicy $fsName -P $listPol -f $outFile -M FSETNAME=$fsetName --single-instance -I defer >> $logF 2>&1 153 | rc=$? 154 | if (( rc != 0 )); then 155 | echo "ERROR: LIST policy failed with rc=$rc" | tee -a $logF 156 | exit 1 157 | fi 158 | 159 | # run the migrate policy, the output file name is $outFile.list.softquota 160 | rc=0 161 | if [[ ! -a "$outFile.list.softquota" ]]; then 162 | echo >> $logF 163 | echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> $logF 164 | echo "WARNING: no outputfile ($outFile) has been created, skipping migration." | tee -a $logF 165 | echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> $logF 166 | else 167 | echo >> $logF 168 | # prepare input file list ($outFile.miglist) for migrate policy by extracting the file names 169 | awk -F '[ ]' '{ for(i=7; i<=NF; i++) printf "%s",$i (i==NF?ORS:OFS) }' $outFile.list.softquota > $outFile.miglist 170 | 171 | # if eepool is set then use this pool for migration, if not it might have been set in the migrate policy 172 | if [[ ! -z $eePool ]]; then 173 | echo "INFO: running migration for fileset $fsetName using policy: $migPol, input file: $outFile.miglist and options: $migPolOpts -M EEPOOL=-p $eePool" >> $logF 174 | $gpfsPath/mmapplypolicy $fsName -P $migPol --single-instance -M FSETNAME=$fsetName -i $outFile.miglist $migPolOpts -M EEPOOL="-p $eePool" >> $logF 2>&1 175 | rc=$? 176 | else 177 | echo "INFO: running migration for fileset $fsetName using policy: $migPol, input file: $outFile.miglist and options: $migPolOpts" >> $logF 178 | $gpfsPath/mmapplypolicy $fsName -P $migPol --single-instance -M FSETNAME=$fsetName -i $outFile.miglist $migPolOpts >> $logF 2>&1 179 | rc=$? 180 | fi 181 | fi 182 | 183 | exitrc=0 184 | if (( rc != 0 )); then 185 | echo "ERROR: MIGRATE policy failed with rc=$rc" | tee -a $logF 186 | exitrc=1 187 | fi 188 | 189 | 190 | echo >> $logF 191 | echo "INFO: $(date) Program finished on node $(hostname)" | tee -a $logF 192 | echo >> $logF 193 | 194 | exit $exitrc 195 | 196 | -------------------------------------------------------------------------------- /quota-migration/quota-listpol.txt: -------------------------------------------------------------------------------- 1 | /* runs a list policy that identifies files if soft quota limit is met. Files are selected for a fileset and based on access time with the oldest files first */ 2 | 3 | /* exclude list */ 4 | define( exclude_list, 5 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 6 | PATH_NAME LIKE '%/.ltfsee/%' OR 7 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 8 | PATH_NAME LIKE '%/.snapshots/%' OR 9 | NAME LIKE '.mmbackupShadow%' OR 10 | NAME LIKE 'mmbackup%') 11 | ) 12 | 13 | /* define macro for files that allocate more than 0 KB */ 14 | define(is_empty, (KB_ALLOCATED=0)) 15 | 16 | /* define macro for access age */ 17 | define(access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) 18 | 19 | /* check softquota against threshold and select files */ 20 | RULE EXTERNAL LIST 'softquota' THRESHOLD 'FILESET_QUOTA_SOFT' 21 | 22 | /* identify files exceeeding the THRESHOLD in the given fileset stored on system pool that are not empty and that are not in exclude list */ 23 | RULE 'quotalist' LIST 'softquota' FROM POOL 'system' THRESHOLD(90,80) WEIGHT(access_age) FOR FILESET ('FSETNAME') WHERE NOT (is_empty) and NOT (exclude_list) 24 | -------------------------------------------------------------------------------- /quota-migration/quota-migpol.txt: -------------------------------------------------------------------------------- 1 | /* runs a migrate policy that migrates files for a particular fileset that are not yet migrated */ 2 | 3 | /* define exclude rule*/ 4 | RULE 'exclude' EXCLUDE WHERE 5 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 6 | PATH_NAME LIKE '%/.ltfsee/%' OR 7 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 8 | PATH_NAME LIKE '%/.snapshots/%' OR 9 | NAME LIKE '.mmbackupShadow%' OR 10 | NAME LIKE 'mmbackup%') 11 | 12 | /* define macro for files that allocate more than 0 KB */ 13 | define(is_empty,(KB_ALLOCATED=0)) 14 | 15 | /* define external pool, the name of the tape pool is provided via EEPOOL */ 16 | RULE 'extpool' EXTERNAL POOL 'ltfs' EXEC '/opt/ibm/ltfsee/bin/eeadm' 17 | OPTS 'EEPOOL' 18 | SIZE 10485760 19 | 20 | /* define migration to ltfs pool for the fileset specified by FSETNAME*/ 21 | RULE 'quotaMig' MIGRATE FROM POOL 'system' TO POOL 'ltfs' FOR FILESET ('FSETNAME') 22 | WHERE NOT (is_empty) 23 | -------------------------------------------------------------------------------- /recall/LTFS-EE/ltfsee_recall.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2019 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: ltfsee_recall 27 | # 28 | # Description: 29 | # Interface script for MIGRATE policy invoked by mmapplypolicy 30 | # Invokes eeadm recall -l libname filelist 31 | # 32 | # Prerequisite: 33 | # EXTERNAL pool policy that identifies files to be recalled. Samples of recall policies are provided in the GitHub repository 34 | # 35 | # Input: 36 | # invoked by mmapplypolicy with the following parameters: 37 | # $1 operation (migrate, test) 38 | # $2 file system name or name of filelist 39 | # $3 library name 40 | # 41 | # Output: 42 | # Writes runtime information to STDOUT - ends up in mmapplypolicy output 43 | # 44 | # Invokation: 45 | # mmapplypolicy fsname -P policyfile 46 | # 47 | # Change History 48 | # 10/09/12 first implementation based GAD startbackup 49 | # 12/20/15 implementation for immutability, some streamlining of existing code 50 | # 12/21/15 create the general receiver 51 | # 09/01/17 ltfsee recall 52 | # 06/23/20 adapt eeadm recall command, improved messaging 53 | 54 | #global variables for this script 55 | # set the default option for the eeadm recall command in case $3 is not give (like --resident or -l libname) 56 | DEFOPTS="" 57 | # define ltfsee directory 58 | LTFSEEDIR=/opt/ibm/ltfsee/bin 59 | 60 | #++++++++++++++++++++++++++ MAIN ++++++++++++++++++++++++++++++++++++++ 61 | echo "================================================================================================" 62 | echo "$(date +"%Y-%b-%d %H:%M:%S") EE_RECALL invoked with arguments: $*" 63 | 64 | ## Parse Arguments & execute 65 | #$1 is the policy operation (list, migrate, etc) 66 | op=$1 67 | #$2 is the policy file name 68 | polFile=$2 69 | #$3 options passed to the eeadm recall command (like --resident and -l libname) 70 | shift 2 71 | option=$* 72 | 73 | ## evaluate the operation passed by mmapplypolicy and act upon it 74 | case $op in 75 | # there will always be a TEST call with $2 being the file system path 76 | TEST ) 77 | echo "EE_RECALL INFO: TEST option received for $polFile" 78 | if [[ ! -z "$polFile" ]] then 79 | if [[ ! -d "$polFile" ]] then 80 | echo "EE_RECALL WARNING: TEST directory $polFile does not exists." 81 | fi 82 | fi 83 | ;; 84 | RECALL | MIGRATE | LIST ) 85 | # this is the actual migrate call where we call eeadm recall 86 | echo "EE_RECALL INFO: $op option received with file name $polFile and options $option" 87 | #set option to default if not set with $3 88 | if [[ -z $option ]] then 89 | if [[ ! -z $DEFOPTS ]] then 90 | option=$DEFOPTS 91 | fi 92 | fi 93 | 94 | echo "EE_RECALL INFO: Start processing files $polFile with eeadm recall from library $option" 95 | $LTFSEEDIR/eeadm recall $polFile $option 96 | rc=$? 97 | if (( rc != 0 )) then 98 | echo "EE_RECALL WARNING: eeadm recall ended with return code $rc" 99 | fi 100 | ;; 101 | REDO ) 102 | echo "EE_RECALL INFO: REDO option received with file name $polFile and options $option" 103 | ;; 104 | * ) 105 | echo "EE_RECALL WARNING: UNKNOWN operation ($op) received with file name $polFile and options $option" 106 | ;; 107 | esac 108 | 109 | echo "$(date +"%Y-%b-%d %H:%M:%S") EE_RECALL ended" 110 | echo 111 | exit 0 112 | -------------------------------------------------------------------------------- /recall/LTFS-EE/readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Script: ltfsee_recall.sh 3 | 4 | 5 | ## Description 6 | 7 | This script can be used as an interface script in combination with a migration policy to recall files from Spectrum Archive EE to Spectrum Scale. This script receives the file list from the policy engine according to the MIGRATE rule and feeds this into the ltfsee recall command. 8 | 9 | Please notice that IBM Spectrum Archive Enterprise Edition version 1.3.0.7 and above has built in capabilities to recall files from LTFS Tape to IBM Spectrum Scale disk. For more details see the IBM Spectrum Archive EE Knowledge Center - section [Manual-recall-with-the mmapplypolicy-command](https://www.ibm.com/support/knowledgecenter/en/ST9MBR_1.3.0/ltfs_ee_recall_mmapplypolicy.html). If you are using this version of IBM Spectrum Archive EE it is recommended to use the built in approach. 10 | 11 | The script `ltfsee_recall.sh` along with the policies has been last tested with IBM Spectrum Archive EE version 1.3.0.7. 12 | 13 | ## Prerequisite 14 | 15 | Copy the `ltfsee_recall.sh` script to a directory of the executing EE node. To run the policy on multiple EE nodes the script must be placed in the same directory on each node. Consider placing it in a shared file system. 16 | 17 | Adjust the recall policy according to your needs and store it in the same directory as the `ltfsee_recall.sh` script. The directory name where the `ltfsee_recall.sh` script is stored on all executing nodes must be updated in the EXTERNAL POOL rule. 18 | 19 | There are two policies provided: 20 | 21 | [recall_migOnly_policy.txt](recall_migOnly_policy.txt): 22 | This policy recalls all migrated files that match clause in the macro `recall_dir`. Only migrated files are recalled. To recall migrated files directly to resident state the option `--resident` has to be added to the EXTERNAL POOL rule in the clause: `OPTS '--resident'`. Note, you cannot recall pre-migrated files, unless you recall to resident state. 23 | 24 | [recall_migPmig_policy.txt](recall_migPmig_policy.txt): 25 | This policy recalls all pre-migrated and migrated files to resident state that match clause in the macro `recall_dir`. 26 | 27 | If the `ltfsee_recall.sh` script and the policies are stored in directory `/usr/local/bin` then the EXTERNAL POOL rule EXEC clause must be adjusted to: 28 | 29 | RULE 'extpool' EXTERNAL POOL 'ltfs' EXEC '/usr/local/bin/ltfsee_recall.sh' 30 | 31 | 32 | ## Invokation 33 | 34 | The recall script `ltfsee_recall.sh` is invoked with the policy engine: 35 | 36 | 37 | # mmapplypolicy fsname -P policyfile -N nodenames --single-instance 38 | 39 | Options: 40 | fsname is the file system name, file system path or the file system path with a subdirectory. 41 | -P policyfile is the policy file including the EXTERNAL POOL rule specifying the `ltfsee_premig.sh` interface script and the MIGRATE rule. 42 | -N nodenames node name or node class name that executes this policy. Must be Spectrum Archive EE nodes 43 | --single-instance run only one instance of this policy. 44 | 45 | 46 | Two example for the `policyfile` are provided in [recall_migOnly_policy.txt](recall_migOnly_policy.txt) and [recall_migPmig_policy.txt](recall_migPmig_policy.txt). Further options can be specified with the `mmapplypolicy` command. 47 | 48 | 49 | ## Processing 50 | 51 | The policy engine (`mmapplypolicy`) identifies files according to the migrate rule and passes these files to the interface script `ltfsee_recall.sh`. The interface script `ltfsee_recall.sh` recalls the selected files using the command: `eeadm recall filelist [--resident]`. The option `--resident` can be specified via the OPTS clause in the EXTERNAL POOL definition of the policies. 52 | 53 | 54 | ## Output 55 | 56 | Output of the script is logged to STDOUT and ends up within the output of the mmapplypolicy command 57 | 58 | -------------------------------------------------------------------------------- /recall/LTFS-EE/recall_migOnly_policy.txt: -------------------------------------------------------------------------------- 1 | /* define the path and file name pattern to be recalled ('%' means all files) */ 2 | define(recall_dir, (PATH_NAME LIKE '%')) 3 | 4 | /* define external pool */ 5 | RULE 'extpool' EXTERNAL POOL 'ltfs' EXEC '/root/silo/recall/ltfsee_recall.sh' 6 | 7 | /* define migration rule to recall migrated files only*/ 8 | RULE 'recall' MIGRATE FROM POOL 'ltfs' TO POOL 'system' FOR FILESET ('test') WHERE 9 | (XATTR('dmapi.IBMPMig') IS NULL) AND NOT (XATTR('dmapi.IBMTPS') IS NULL) and (recall_dir) 10 | -------------------------------------------------------------------------------- /recall/LTFS-EE/recall_migPmig_policy.txt: -------------------------------------------------------------------------------- 1 | /* define the path and file name pattern to be recalled ('%' means all files) */ 2 | define(recall_dir, (PATH_NAME LIKE '%')) 3 | 4 | /* define external pool */ 5 | RULE 'extpool' EXTERNAL POOL 'ltfs' EXEC '/root/silo/recall/ltfsee_recall.sh' OPTS '--resident' 6 | 7 | /* define migration rule recall of premigrated and migrated files */ 8 | RULE 'recall' MIGRATE FROM POOL 'ltfs' TO POOL 'system' FOR FILESET ('test') WHERE 9 | NOT (XATTR('dmapi.IBMTPS') IS NULL) and (recall_dir) 10 | -------------------------------------------------------------------------------- /recall/README.md: -------------------------------------------------------------------------------- 1 | # Recall policy with TSM HSM 2 | 3 | ## Description 4 | The policy engine can be used to recall large numbers of files from an external pool, e.g. from tape. The policy file provided with this project can act as a template for performing such bulk recalls. 5 | 6 | ## Usage 7 | 8 | 1. Modify the sample policy `recpol.txt` to select the files you intend to recall. Line 14 in that policy contains the selection criteria: 9 | 10 | ``` 11 | ... 12 | define(recall_dir, (PATH_NAME LIKE '/sample_fs/sample_dir/%')) 13 | ... 14 | ``` 15 | 16 | Adjust the path as required. 17 | 18 | 2. Run the policy to recall the actual data: 19 | 20 | ``` 21 | # mmapplypolicy -P recpol.txt -N node1,node2 -m 4 –B 1000 -s [-I test] 22 | ``` 23 | 24 | - `-N node1,node2` 25 | 26 | Specifies the HSM nodes or nodeclass to perform the policy scan, as well as the subsequent recall operation. 27 | 28 | - `-m 4 (ThreadLevel)` 29 | 30 | With two HSM nodes recalling the actual data, this will result in ~8 tape mounts on the TSM server. However, this number is not guaranteed... So you will need to monitor TSM server activity and potentially adjust this setting as required. 31 | 32 | - `‐B 1000 (MaxFiles)` 33 | 34 | A batch size of 1000-2000 files is probably the minimum, and will only make sense for fewer large files. In directories with many small files it is suggested to raise this parameter. 35 | 36 | - `-s ` 37 | 38 | Provide a directory with sufficient capacity as scratch space for temporary data (e.g. file lists) used during the policy scan and recall operation. 39 | 40 | - `-I test` 41 | 42 | Optionally performs a dry-run (test) of the recall operation. This will evaluate the policy with the given path definition, and report on the amount of data which would be transferred. Remove this argument to perform the actual data recall. 43 | 44 | 3. It is recommended to create multiple copies of the `recpol.txt` file in order to recall independent batches of files as required. 45 | 46 | ## Further optimization 47 | 48 | The HSM recall script which is shipped with (certain versions of) Spectrum Scale does not facilitate a Spectrum Protect tape-optimized recall by default. However, the script can be modified in order to optimize performance during bulk recalls using the policy engine. 49 | 50 | Furthermore, bulk-recalls are frequently used to ultimately recall data from external pools prior to a system migration, or in preparation of disabling HSM altogether. Performance can be improved in both such scenarios by recalling files into resident state. By default, files would be recalled into premigrated state - but the HSM recall script can be modified to recall data into resident state straight away. 51 | 52 | Both these modifications can be implemented by applying the patch which is shipped with this project on all HSM nodes: 53 | 54 | ``` 55 | # patch -b /usr/lpp/mmfs/samples/ilm/mmpolicyExec-hsm.sample mmpolicyExec-hsm.sample.patch 56 | ``` 57 | 58 | Furthermore, the mountpoint of the filesystem needs to be adapted in fore mentioned policy. Modify the sample policy `recpol.txt` to contain the filesystem mountpoint to which you intend to recall to. Line 20 in that policy contains the external pool definition: 59 | 60 | ``` 61 | ... 62 | RULE 'hsmexternalpool' EXTERNAL POOL 'hsm' EXEC '/usr/lpp/mmfs/samples/ilm/mmpolicyExec-hsm.sample' OPTS '-v -fs=/sample_fs' 63 | ... 64 | ``` 65 | 66 | Note that this modification needs to be performed on each HSM node in order to enable Spectrum Protect tape-optimized recall operation. 67 | -------------------------------------------------------------------------------- /recall/mmpolicyExec-hsm.sample.patch: -------------------------------------------------------------------------------- 1 | --- mmpolicyExec-hsm.sample.orig 2017-03-03 13:16:40.336792141 +0100 2 | +++ mmpolicyExec-hsm.sample 2017-03-03 13:19:37.048514666 +0100 3 | @@ -96,6 +96,7 @@ 4 | # Initialize global vars 5 | $Verbose = 0; 6 | $VerboseOption = ""; 7 | +$FileSystem = ""; 8 | 9 | 10 | # Set binding to HSM commands 11 | @@ -107,7 +108,7 @@ 12 | # Command Options Files 13 | $MigrateFormat = "%s %s -filelist=%s"; 14 | $PremigrateFormat = "%s %s -premigrate -filelist=%s"; 15 | -$RecallFormat = "%s %s -filelist=%s"; 16 | +$RecallFormat = "%s %s -resident -filelist=%s %s"; 17 | 18 | # Set command options 19 | $VerboseCommandOption = "-detail"; 20 | @@ -135,6 +136,10 @@ 21 | $Verbose = 1; 22 | $VerboseOption = $VerboseCommandOption; 23 | } 24 | + elsif($opt =~ m/-fs=(.*)/) { 25 | + $opt =~ s/-fs=//g; 26 | + $FileSystem = $opt; 27 | + } 28 | } 29 | 30 | print "$0 $command $filelist @ARGV\n" if ($Verbose > 0); 31 | @@ -249,7 +254,7 @@ 32 | } 33 | elsif ($command eq "RECALL") { 34 | $syscmd = sprintf($RecallFormat, 35 | - $RecallCommand, $VerboseOption, $hsmfilelist); 36 | + $RecallCommand, $VerboseOption, $hsmfilelist, $FileSystem); 37 | } 38 | print "$syscmd\n" if ($Verbose > 0); 39 | -------------------------------------------------------------------------------- /recall/recpol.txt: -------------------------------------------------------------------------------- 1 | /* define some file attributes */ 2 | define(not_zero_byte, (FILE_SIZE > 0)) 3 | define(is_premigrated, (MISC_ATTRIBUTES LIKE '%M' AND MISC_ATTRIBUTES NOT LIKE '%V')) 4 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V')) 5 | define(is_resident, (MISC_ATTRIBUTES NOT LIKE '%M%')) 6 | 7 | /* define access age */ 8 | define(access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) 9 | 10 | /* define path to be recalled */ 11 | /********************************/ 12 | /*** path needs to be adapted ***/ 13 | /********************************/ 14 | define(recall_dir, (PATH_NAME LIKE '/sample_fs/sample_dir/%')) 15 | 16 | /* define external pool script */ 17 | /**************************************/ 18 | /*** filesystem needs to be adapted ***/ 19 | /**************************************/ 20 | RULE 'hsmexternalpool' EXTERNAL POOL 'hsm' EXEC '/usr/lpp/mmfs/samples/ilm/mmpolicyExec-hsm.sample' OPTS '-v -fs=/sample_fs' 21 | 22 | /* define recall rule */ 23 | RULE 'recall' MIGRATE FROM POOL 'hsm' TO POOL 'system' WHERE (NOT (is_resident) AND 24 | NOT (is_premigrated) AND 25 | (not_zero_byte) AND 26 | (recall_dir)) 27 | -------------------------------------------------------------------------------- /receiver/policy_receiver_age.txt: -------------------------------------------------------------------------------- 1 | /* define exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* define the interface script */ 12 | RULE EXTERNAL LIST 'recieveAge' EXEC '/usr/local/bin/receiver.sh' 13 | 14 | 15 | RULE 'list_age' LIST 'recieveAge' WHERE (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME) > 2) AND NOT (exclude_list) 16 | -------------------------------------------------------------------------------- /receiver/policy_receiver_all.txt: -------------------------------------------------------------------------------- 1 | /* define exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | 12 | /* define the interface script */ 13 | RULE EXTERNAL LIST 'recieveAll' EXEC '/usr/local/bin/receiver.sh' 14 | 15 | RULE 'list_all' LIST 'recieveAll' 16 | 17 | -------------------------------------------------------------------------------- /receiver/policy_receiver_mig.txt: -------------------------------------------------------------------------------- 1 | /* define exclude list */ 2 | define( exclude_list, 3 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 4 | PATH_NAME LIKE '%/.ltfsee/%' OR 5 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 6 | PATH_NAME LIKE '%/.snapshots/%' OR 7 | NAME LIKE '.mmbackupShadow%' OR 8 | NAME LIKE 'mmbackup%') 9 | ) 10 | 11 | /* define migrated state */ 12 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) 13 | 14 | /* define the interface script */ 15 | RULE EXTERNAL LIST 'recieveMig' EXEC '/usr/local/bin/receiver.sh' 16 | 17 | /* define the rule to select migrated files in fileset test */ 18 | RULE 'listMig' LIST 'recieveMig' WHERE (is_migrated) AND NOT (exclude_list) 19 | -------------------------------------------------------------------------------- /receiver/readme.md: -------------------------------------------------------------------------------- 1 | 2 | ## Introduction 3 | 4 | EXTERNAL LIST policies can be used select files based on criteria and process these selected files by a custom interface script. The script `receiver.sh` is an interface script that can be used in combination with EXTERNAL LIST policies to process the selected files. 5 | 6 | An EXTERNAL LIST policy consists of a EXTERNAL LIST rule and a LIST RULE. The EXTERNAL LIST rule defines the interface script and allow to pass parameter to this script. The LIST rule defines criteria for the selection of files. The policy example below, defines the program `/user/local/bin/receiver.sh` as interface script and selects files in fileset `test` that have migrated. 7 | 8 | /* define exclude list */ 9 | define( exclude_list, 10 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 11 | PATH_NAME LIKE '%/.ltfsee/%' OR 12 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 13 | PATH_NAME LIKE '%/.snapshots/%' OR 14 | NAME LIKE '.mmbackupShadow%' OR 15 | NAME LIKE 'mmbackup%') 16 | ) 17 | 18 | /* define migrated state */ 19 | define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) 20 | 21 | /* define the interface script */ 22 | RULE EXTERNAL LIST 'recieve-mig' EXEC 'usr/local/bin/receiver.sh' 23 | 24 | /* define the rule to select migrated files in fileset test */ 25 | RULE 'listMig' LIST 'recieve-mig' FOR FILESET ('test') WHERE (is_migrated) AND NOT (exclude_list) 26 | 27 | 28 | This policy can be invoked with the command: 29 | 30 | # mmapplypolicy fsname -P policyfile -B 1000 -m 2 -N node1,node2 --single-instance 31 | 32 | The parameter `fsname` denotes the name or the path of the file system or subdirectory. 33 | 34 | The parameter `-P policyfile` denotes the file name that includes the EXTERNAL LIST policy. 35 | 36 | The IBM Spectrum Scale policy engine (`mmapplypolicy`) first runs the LIST rule and selects all migrated files in fileset `test`. The full path and filenames of these files are put in file lists, whereby each file list contains 1000 file name (parameter `-B 1000`). Each file list is passed to one instance of the interface script defined by the EXTERNAL LIST rule (`/usr/local/bin/receiver.sh`). The policy engine starts 2 instances of the interface script per node (parameter `-m 2`). The nodes that execute the `receiver.sh` program are node1 and node2 (`-N node1,node2`). 37 | 38 | When invoked by the policy engine the `receiver.sh` script obtains two arguments: 39 | 40 | $1: is the operation of the policy, in this case it is LIST. 41 | $2: is the name of the file list containing the path and file names 42 | 43 | 44 | The file list passed to the `receiver.sh` script in argument `$2` has the following format: 45 | 46 | 48900 1741777473 0 -- /filesystem/file1 47 | 48 | The three first numbers are IBM Spectrum Scale internal numbers (inodenumber, inodegeneration, snapid). The path and file name of the selected files is the 5th field in the file list. The `receiver.sh` extracts the path and file name and writes it to an output file `$OUTPUTFILE`. At this point you can implement your own logic to process the files. 49 | 50 | It is also possible to pass further arguments to the `receiver.sh` script by using the OPTS clause in the EXTERNAL LIST rule. The following example passes the string `migrated` to the `receiver.sh` script: 51 | 52 | RULE EXTERNAL LIST 'receiver' EXEC 'usr/local/bin/receiver.sh' OPTS 'migrated' 53 | 54 | The OPTS clause allows to pass arguments for the processing of the file. 55 | 56 | The script `receiver.sh` provided in this repository does not process any file yet, it just write the path and file names to a file list. 57 | 58 | 59 | 60 | ## Preparation 61 | 62 | Adjust the parameters in the script `receiver.sh` when required: 63 | 64 | | Parameter | Description | 65 | | ----------|-------------| 66 | | MYPATH | define paths for log files and output files. Default is `./receiver`. | 67 | | LOGFILE | log file name that is stored in `$MYPATH`. Default is `` receiver.log` | 68 | | OUTPUTFILE | name of the file that contains the file names identified by the policy engine. This is not the file that is produced by the policy engine but the file that just includes the path and file names selected by the policy engine. Default is `receiver.out` | 69 | | DEFOPTS | Default options used for the `mmapplypolicy` command. | 70 | | LOGLEVEL | Defines the depth of logging. Default is `1`. Currently there is only level `1` implemented | 71 | 72 | 73 | Copy the `receiver.sh` script to a common directory of all IBM Spectrum Scale nodes that execute the script. 74 | 75 | 76 | Create the EXTERNAL LIST policy according to the needs and specify the path of the `receiver.sh` script in the EXTERNAL LIST rule. There are three examples policy files: 77 | 78 | [policy_receiver_mig.txt](policy_receiver_mig.txt): selects all file that are migrated and invokes `receiver.sh` with the selected path and file names. 79 | 80 | [policy_receiver_age.txt](policy_receiver_age.txt): selects all file that have not been accessed for more than 2 days and invokes `receiver.sh` with the selected path and file names. 81 | 82 | [policy_receiver_all.txt](policy_receiver_all.txt): selects all file and invokes `receiver.sh` with the selected path and file names. 83 | 84 | 85 | Run the policy engine on one node: 86 | 87 | # mmapplypolicy fsname -P policyfile -B 1000 -m 2 -N node1,node2 --single-instance 88 | 89 | The policy engine selects the files in accordance to the LIST rulé and executes the interface script by passing the selected file names to the script. 90 | 91 | Running the policy engine can be automated. The GitHub project [Spectrum Scale Automation](https://github.com/nhaustein/SpectrumScaleAutomation) provides the tooling for this. 92 | 93 | 94 | ## Output: 95 | 96 | The `receiver.sh` script creates a subdirectory specified by the script parameter `$MYPATH` and logs messages in a log file denoted by parameter `$LOGFILE`. The resulting list including path and file of the selected files is written to the file denoted by `OUTPUTFILE`. 97 | 98 | 99 | ## Combination with runpolicy.sh 100 | 101 | Instead of running the `mmapplypolicy` command manually you can combine this with the [runpoliy script](runpolicy/readme.md). 102 | 103 | ./runpolicy run fsname policyfile 104 | 105 | Parameter `fsname` is the name of the file system where the policy should run. 106 | Parameter `policyfile` is the name of the policy file including the EXTERNAL LIST policy that invokes that interface script `receiver.sh`. 107 | 108 | 109 | -------------------------------------------------------------------------------- /receiver/receiver.sh: -------------------------------------------------------------------------------- 1 | #! /bin/ksh 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2019 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program: receiver 27 | # 28 | # Description: 29 | # Interface script for LIST policy invoked by mmapplypolicy 30 | # Prints files identified by policy to output file 31 | # 32 | # Prerequisite: 33 | # EXTERNAL list policy that identifies files that are not immutable. 34 | # 35 | # Input: 36 | # invoked by mmapplypolicy with the following parameters: 37 | # $1 operation (list, test) 38 | # $2 file system name or name of filelist 39 | # $3 optional parameter defined in LIST policy under OPTS 40 | # 41 | # Output: 42 | # Prints file names identified by policy to $OUTPUTFILE 43 | # Write runtime information and debugging messages to log file $LOGFILE 44 | # 45 | # Example Policy: 46 | # /* define macros */ 47 | # define( exclude_list, (PATH_NAME LIKE '%/.SpaceMan/%' OR PATH_NAME LIKE '%/.snapshots/%' OR NAME LIKE '%mmbackup%' )) 48 | # define( immutable, MISC_ATTRIBUTES LIKE '%X%') 49 | # RULE EXTERNAL LIST 'setmp3' EXEC '/root/silo/receiver.sh' OPTS 'TEST' 50 | # RULE 'mp3' LIST 'setmp3' FOR FILESET ('native') WHERE NOT (exclude_list) and NOT (immutable) and (NAME LIKE '%.mp3') 51 | # 52 | # Invokation: 53 | # mmapplypolicy fsname -P policyfile 54 | # 55 | # Change History 56 | # 10/09/12 first implementation based GAD startbackup 57 | # 12/20/15 implementation for immutability, some streamlining of existing code 58 | # 12/21/15 create the general receiver 59 | # 06/26/20 Add name of the program to the output messages. 60 | 61 | #global variables for this script 62 | #---------------------------------- 63 | # define paths for log files and output files 64 | MYPATH="./receiver" 65 | # logfile used for system_log function 66 | LOGFILE=$MYPATH/"receiver.log" 67 | # outfile is used to print file names to 68 | OUTPUTFILE=$MYPATH/"receiver.out" 69 | # sets the log level for the system log, everything below that number is logged 70 | LOGLEVEL=1 71 | # set the default option for the file list processing in case $3 is not given 72 | DEFOPTS="" 73 | 74 | 75 | ## Append to the system log 76 | ## Usage: system_log 77 | system_log () { 78 | SEV=$1 79 | 80 | case $SEV in 81 | [0-9]) ;; 82 | *) SEV=1;; 83 | esac 84 | 85 | LINE=$2 86 | if [ $LOGLEVEL -ge $SEV ] ; then 87 | if [[ -z "$LINE" ]]; then 88 | echo -e "RECEIVER INTERNAL WARNING: Improper value given to system_log function ($@)" >> $LOGFILE 89 | else 90 | echo -e "RECEIVER: $LINE" >> $LOGFILE 91 | fi 92 | fi 93 | } 94 | 95 | ## Print a message to the stdout 96 | ## Usage: user_log 97 | user_log () { 98 | echo -e "RECEIVER $@" 99 | } 100 | 101 | 102 | ## Get current date and time 103 | ## Usage: get_cur_date_time 104 | get_cur_date_time(){ 105 | echo "$(date +"%Y-%b-%d %H:%M:%S")" 106 | } 107 | 108 | #++++++++++++++++++++++++++ MAIN ++++++++++++++++++++++++++++++++++++++ 109 | user_log "$(get_cur_date_time) receiver.sh invoked by policy engine" 110 | 111 | # check the path for logging 112 | if [[ ! -d $MYPATH ]] then 113 | mkdir -p $MYPATH 114 | rc=$? 115 | if (( rc > 0 )) then 116 | system_log "ERROR: failed to create directory $MYPATH, check permissions" 117 | user_log "ERROR: failed to create directory $MYPATH, check permissions" 118 | exit 1 119 | fi 120 | fi 121 | 122 | system_log 1 "=========================================================================" 123 | system_log 1 "$(get_cur_date_time) receiver invoked with arguments: $*" 124 | 125 | ## Parse Arguments & execute 126 | #$1 is the policy operation (list, migrate, etc) 127 | op=$1 128 | #$2 is the policy file name 129 | polFile=$2 130 | #$3 is the option given in the EXTERNAL LIST rule with OPTS '..' should be retention time here 131 | option=$3 132 | 133 | ## this is required, as the script may be called multiple times during 134 | ## the same backup (if there are too many files to process). 135 | 136 | case $op in 137 | TEST ) 138 | user_log "INFO: TEST option received for directory $polFile." 139 | system_log 1 "INFO: TEST option received for $polFile" 140 | if [[ ! -z "$polFile" ]] then 141 | if [[ -d "$polFile" ]] then 142 | user_log "INFO: TEST directory $polFile exists." 143 | system_log 1 "INFO: Directory $polFile exists." 144 | else 145 | user_log "WARNING: TEST directory $polFile does not exists." 146 | system_log 1 "WARNING: Directory $polFile does not exist." 147 | fi 148 | fi 149 | 150 | # delete the $OUTPUTFILE if it exists in order to get a clean start 151 | if [[ -a $OUTPUTFILE ]] then 152 | rm -f $OUTPUTFILE 153 | fi 154 | ;; 155 | LIST ) 156 | user_log "INFO: LIST option received, starting receiver task" 157 | system_log 1 "INFO: LIST option received with file name $polFile and options $option" 158 | 159 | #set option to default if not set 160 | if [[ -z $option ]] then 161 | option=$DEFOPTS 162 | fi 163 | 164 | # process the files 165 | itemNum=0 166 | numEntries=$(wc -l $polFile | awk '{print $1}') 167 | system_log 1 "INFO: Start processing $numEntries files, outputfile=$OUTPUTFILE" 168 | user_log "INFO: Start processing $numEntries files, outputfile=$OUTPUTFILE" 169 | cat $polFile | while read line 170 | do 171 | # use set to get file name, does tolerate blanks 172 | set $line 173 | shift 4 174 | fName="$*" 175 | 176 | # perhaps check if file exists 177 | echo "$fName" >> $OUTPUTFILE 178 | ((itemNum=itemNum+1)) 179 | done 180 | system_log 1 "INFO: Processed $itemNum out of $numEntries files" 181 | user_log "INFO: Processed $itemNum out of $numEntries files" 182 | ;; 183 | REDO ) 184 | user_log "INFO: REDO option received, doing nothing" 185 | system_log 1 "INFO: REDO option received with file name $polFile and options $option" 186 | ;; 187 | * ) 188 | user_log "WARNING: Unknown option $op received, doing nothing" 189 | system_log 1 "WARNUNG: UNKNOWN option ($op) received with file name $polFile and options $option" 190 | ;; 191 | esac 192 | 193 | user_log "$(get_cur_date_time) receiver ended" 194 | system_log 1 "$(get_cur_date_time) receiver ended" 195 | 196 | # exit 0 if things are OK 197 | exit 0 -------------------------------------------------------------------------------- /runpolicy/readme.md: -------------------------------------------------------------------------------- 1 | ## Introduction 2 | 3 | The script `runpolicy.sh` is a wrapper for `mmapplypolicy`command that runs a policy for file system directory provided as input. When started with no additional arguments, `runpolicy.sh` will not execute the policy and run in test mode. This is usefull to check the policy syntax. With additional command line parameters you can run the policy and provide addition options to the `mmapplypolicy` command. 4 | 5 | 6 | ## Preparation 7 | 8 | Copy the program `runpolicy.sh` to a directory of a Spectrum Scale cluster node where you want to run the script. 9 | 10 | 11 | Adjust the variables in the `runpolicy.sh` script: 12 | 13 | | Parameter | Description | 14 | | ----------|-------------| 15 | | DEFAULT_OPTS | defines the default parameters for the policy engine. These are used if not other parameters are specified in the argument `opts` of the `runpolicy.sh` script | 16 | 17 | Encoding default parameter in the script variable `DEFAULT_OPTS` is useful, when these parameters are static. Alternatively, parameters for the policy engine can be bassed with the `runpoliy.sh` commmand. 18 | 19 | 20 | Create and copy the file including the policy to a directory of the Spectrum Scale where you want to run the script. Find more guidance for the policies in section [Sample Policies](#Sample-policies) 21 | 22 | 23 | 24 | ## Running the script 25 | 26 | The scipt can be invoked like this: 27 | 28 | runpolicy.sh filesystem-directory policyfile [opts] 29 | filesystem: file system path and directory subject for the policy run (mandatory) 30 | policyfile: name of the file including the policy (mandatory) 31 | opts : options for mmapplypolicy, such as -i -m -B -n --single-instance -M param=value (optional) 32 | 33 | **Notes:** 34 | The sequence of parameters matters. The parameter `filesystem-path-directory` has to come first, followed by name of the `policyfile`. 35 | 36 | The default run mode is `test`. If you want to run the policy in a different mode specify `-I yes` for MIGRATE policies or `-I defer -f dirPrefix` for list policies in parameter `opts`. 37 | 38 | Parameter `opts` comes last and can include specific paramaters controlling `mmapplypolicy`. If parameter `opts` is not specified then the parameters encoded in the script variable `$DEFAULT_OPTS` are used. If parameter `opts` is specified the variable `$DEFAULT_OPTS` is ignored. 39 | 40 | Find some examples to run the script along with some sample policies below. 41 | 42 | 43 | ### Sample policies 44 | 45 | In folder [sample-policies](../sample-policies/) you can find some useful sample policies: 46 | 47 | - Policy [migrate-all.txt](../sample-policies/migrate-all.txt) migrates all files that are not migrated, including pre-migrated files. To run the policy: 48 | 49 | 50 | runpolicy.sh path migrate-all.txt -M EEPOOL=pool1@lib1 [opts] 51 | 52 | 53 | The parameter `-M EEPOOL=pool1@lib1` denotes the destination pool for the migration and must be an existing pool in the Spectrum Archive EE system. Additional options for the policy engine should be configured. These options are passed to the `mmapplypolicy` command should include: `-B bucketsize -m threads -N nodenames --single-instance`. These options can either be encoded in the command line parameter `opts`, or these options can be encoded in the script variable `DEFAULT_OPTS`. 54 | 55 | 56 | - Policy [migrate-fset.txt](../sample-policies/migrate-fset.txt) migrates all files for a particular fileset that are not migrated, including pre-migrated files. To run the policy: 57 | 58 | 59 | runpolicy.sh path migrate-fset.txt -M EEPOOL=pool1@lib1 -M FSET=filesetname [opts] 60 | 61 | 62 | The parameter `-M EEPOOL=pool1@lib1` denotes the destination pool for the migration and must be an existing pool in the Spectrum Archive EE system. The parameter `-M FSET=filesetname` denotes the fileset name subject for migration. Additional options for the policy engine should be configured. These options are passed to the `mmapplypolicy` command should include: `-B bucketsize -m threads -N nodenames --single-instance`. These options can either be encoded in the command line parameter `opts`, or these options can be encoded in the script variable `DEFAULT_OPTS`. 63 | 64 | - Policy [premigrate-all.txt](../sample-policies/premigrate-all.txt) premigrates all resident files. To run the policy: 65 | 66 | 67 | runpolicy.sh path premigrate-all.txt -M EEPOOL=pool1@lib1 [opts] 68 | 69 | 70 | The parameter `-M EEPOOL=pool1@lib1` denotes the destination pool for the premigration and must be an existing pool in the Spectrum Archive EE system. Additional options for the policy engine should be configured. These options are passed to the `mmapplypolicy` command should include: `-B bucketsize -m threads -N nodenames --single-instance`. These options can either be encoded in the command line parameter `opts`, or these options can be encoded in the script variable `DEFAULT_OPTS`. 71 | 72 | 73 | - Policy [premigrate-fset.txt](../sample-policies/premigrate-fset.txt) migrates all files for a particular fileset that are not migrated, including pre-migrated files. To run the policy: 74 | 75 | 76 | runpolicy.sh path premigrate-fset.txt -M EEPOOL=pool1@lib1 -M FSET=filesetname [opts] 77 | 78 | 79 | The parameter `-M EEPOOL=pool1@lib1` denotes the destination pool for the premigration and must be an existing pool in the Spectrum Archive EE system. The parameter `-M FSET=filesetname` denotes the fileset name subject for premigration. Additional options for the policy engine should be configured. These options are passed to the `mmapplypolicy` command should include: `-B bucketsize -m threads -N nodenames --single-instance`. These options can either be encoded in the command line parameter `opts`, or these options can be encoded in the script variable `DEFAULT_OPTS`. 80 | 81 | 82 | - Policy [recall-all.txt](../sample-policies/recall-all.txt) recalls all files located in a specified directory (`RECAllDIR`). After recall files are in status `premigrated`. To run the policy: 83 | 84 | 85 | runpolicy.sh path recall-all.txt -M RECALLDIR=recallpath [opts] 86 | 87 | 88 | The parameter `-M RECALLDIR=recallpath` denotes the path in the Spectrum Scale file system subject for recall. All migrated files in this path are being recalled. Additional options for the policy engine should be configured. These options are passed to the `mmapplypolicy` command should include: `-B bucketsize -m threads -N nodenames --single-instance`. These options can either be encoded in the command line parameter `opts`, or these options can be encoded in the script variable `DEFAULT_OPTS`. 89 | 90 | - Policy [list-byState.txt](../sample-policies/list-byState.txt) creates lists of all files in accordance to their migration state. For each migration state (resident, migrated, premigrated) a separate file list is created. To run the policy: 91 | 92 | 93 | runpolicy.sh path list-byState.txt -I defer -f dirPrefix 94 | 95 | 96 | The resulting file lists are stored in a directory denoted by parameter `-f dirPrefix`. File names of resident files are stored in file `dirPrefix.list.r`, file names of premigrated files are stored in file `dirPrefix.list.p` and file names of migrated files are stored in file `dirPrefix.list.m`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file lists would be named: `/tmp/files.list.r`, `/tmp/files.list.p` and `/tmp/files.list.`. 97 | Instead of specifying the parameter `-I defer -f dirPrefix` in the command line, these can also be encoded in the script variable `DEFAULT_OPTS`. 98 | 99 | 100 | - Policy [list-byTapeID.txt](../sample-policies/list-byTapeID.txt) creates a list of all files that are located on a particular tape ID. The tape ID is specified in the command line. To run the policy: 101 | 102 | 103 | runpolicy.sh path list-byTapeID.txt -I defer -f dirPrefix -M TAPEID=volser 104 | 105 | The tape ID for which the files should be listed in provided with parameter `-M TAPEID=volser`. 106 | The resulting file list is stored in a directory denoted by parameter `-f dirPrefix` and named `dirPrefix.list.filesontape`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file list would be named: `/tmp/files.list.filesontape`. This file list includes all path and file names of files stored on the subject tape ID. This is includes files that have the primary or a secondary copy on the subject tape. 107 | 108 | 109 | - Policy [list-byStateWithTapeID.txt](../sample-policies/list-byStateWithTapeID.txt) creates a lists of all files that that are in premigrated and migrated state includin the tape ID where these files are (pre)migrated to. To run the policy: 110 | 111 | 112 | runpolicy.sh path list-byStateWithTapeID.txt -I defer -f dirPrefix 113 | 114 | The resulting file lists are stored in a directory denoted by parameter `-f dirPrefix`. File names and tape ID of premigrated files are stored in file `dirPrefix.list.premigTapeID` and file names and tape ID of migrated files are stored in file `dirPrefix.list.migTapeID`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file lists would be named: `/tmp/files.list.premigTapeID` and `/tmp/files.list.migTapeID`. 115 | Instead of specifying the parameter `-I defer -f dirPrefix` in the command line, these can also be encoded in the script variable `DEFAULT_OPTS`. 116 | 117 | 118 | ## Output 119 | 120 | The `runpolicy.sh` script runs the `mmapplypolicy` command and logs all output to the console. The script output is marked with by token `RUNPOLICY`. 121 | -------------------------------------------------------------------------------- /runpolicy/runpolicy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ################################################################################ 3 | # The MIT License (MIT) # 4 | # # 5 | # Copyright (c) 2019 Nils Haustein # 6 | # # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy # 8 | # of this software and associated documentation files (the "Software"), to deal# 9 | # in the Software without restriction, including without limitation the rights # 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # 11 | # copies of the Software, and to permit persons to whom the Software is # 12 | # furnished to do so, subject to the following conditions: # 13 | # # 14 | # The above copyright notice and this permission notice shall be included in # 15 | # all copies or substantial portions of the Software. # 16 | # # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,# 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE# 23 | # SOFTWARE. # 24 | ################################################################################ 25 | # 26 | # Program name: runpolicy.sh 27 | # 28 | # Author: N. Haustein 29 | # 30 | # Description: 31 | # Wrapper for mmapplypolicy 32 | # 33 | # Prerequisite: 34 | # policy file must exist and must be correct 35 | # when required add optional parameters for mmapplypolicy 36 | # 37 | # Syntax: 38 | # runpolicy mode filesystem policyfile [opts] 39 | # filesystem: GPFS file system and directory subject for the policy run 40 | # policyfile: name of the file including the policy 41 | # opts : options for mmapplypolicy, such as -I yes -m 3 -n 1 --single-instance -M param=value" (optional). Default options can be set with script parameter DEFAULT_OPTS 42 | # Note, the sequence of the parameters matters. If not mmappylpolicy options are defined the policy is run in test mode. Specify other run mode explicitely with -I 43 | # 44 | # Output: 45 | # writes STDOUT and STDERR to console 46 | # 47 | # Change history 48 | #----------------- 49 | # 07/06/20 streamlining, allow to specify mmapplypolicy parameters 50 | # 51 | 52 | # adjustable parameters 53 | # --------------------- 54 | # default options for mmapplypolicy in case there are options specified with the runpolicy.sh command 55 | DEFAULT_OPTS="-m 3 -n 1 --single-instance" 56 | 57 | 58 | # Parameter assignment 59 | # -------------------- 60 | #mode=$1 61 | fs=$1 62 | pol=$2 63 | # we assign opts later 64 | opts="" 65 | 66 | hostname=$(hostname) 67 | LOGLEVEL=1 68 | MMAPPLYPOLICY_OPTS="" 69 | 70 | 71 | syntax () 72 | { 73 | echo "SYNTAX: runpolicy filesystem-directory policyfile [opts]" 74 | echo " filesystem: GPFS file system path and directory subject for the policy run (mandatory)" 75 | echo " policyfile: name of the file including the policy (mandatory)" 76 | echo " opts : options for mmapplypolicy, such as II -m -n --single-instance -M param=value (optional)" 77 | echo "" 78 | echo " Note: The sequence of parameters matters. File system path has to come first, followed by policyfile." 79 | echo " The default run mode is test. If you want to run the policy specify -I yes for MIGRATE policies " 80 | echo " or -I defer -f /tmp/prefix for list policies as parameter opts. " 81 | echo 82 | exit 1 83 | } 84 | 85 | #************** MAIN ************************** 86 | echo "=====================================================" 87 | echo "RUNPOLICY INFO: $(date +"%d/%m/%y %H:%M:%S") Starting policy on $hostname" 88 | 89 | # check if file system was specified and check if path exists 90 | if [[ -z "$fs" ]]; then 91 | echo "RUNPOLICY ERROR: filesystem not specified." 92 | syntax 93 | elif [[ ! -d "$fs" ]]; then 94 | echo "RUNPOLICY ERROR: filesystem $fs does not exist." 95 | syntax 96 | fi 97 | 98 | # check if policyfile was specified and exists 99 | if [[ -z "$pol" ]]; then 100 | echo "RUNPOLICY ERROR: policy file $pol not specified." 101 | syntax 102 | elif [[ ! -f $pol ]]; then 103 | echo "RUNPOLICY ERROR: policy file $pol does not exist." 104 | syntax 105 | fi 106 | 107 | # we have file system name and policy file, now lets assign the rest of the options 108 | # all the rest of the parameters are considered for mmapplypolicy (no parsing) 109 | shift 2 110 | opts="$*" 111 | 112 | # assign mmapplypolicy options 113 | if [[ ! -z "$opts" ]]; then 114 | MMAPPLYPOLICY_OPTS="$opts" 115 | else 116 | if [[ ! -z "$DEFAULT_OPTS" ]]; then 117 | MMAPPLYPOLICY_OPTS="$DEFAULT_OPTS" 118 | else 119 | echo "RUNPOLICY INFO: not further mmapplypolicy options specified." 120 | fi 121 | fi 122 | echo "RUNPOLICY DEBUG: MMAPPLYPOLICY_OPTS=$MMAPPLYPOLICY_OPTS" 123 | 124 | # if -I is specified in the options with -I then use this, otherwise set -I test 125 | if [[ -z $(echo $MMAPPLYPOLICY_OPTS | grep "\-I ") ]]; then 126 | echo "RUNPOLICY DEBUG: setting run mode to: test." 127 | MMAPPLYPOLICY_OPTS=" -I test "$MMAPPLYPOLICY_OPTS 128 | else 129 | echo "RUNPOLICY DEBUG: run mode was set as parameter ($MMAPPLYPOLICY_OPTS)." 130 | fi 131 | 132 | echo "RUNPOLICY INFO: running policy on file system $fs with policyfile $pol and options: $MMAPPLYPOLICY_OPTS." 133 | 134 | #mmapplypolicy $fs -P $pol "$MMAPPLYPOLICY_OPTS" -M "LASTRUNDATE=$LASTRUNDATE" -M "LASTRUNTIME=$LASTRUNTIME" -L $LOGLEVEL 135 | # echo "RUNPOLICY DEBUG: mmapplypolicy $fs -P $pol "$MMAPPLYPOLICY_OPTS" " 136 | mmapplypolicy $fs -P $pol $MMAPPLYPOLICY_OPTS 137 | rc=$? 138 | if (( rc > 0 )); then 139 | echo "RUNPOLICY ERROR: mmapplypolicy returned an error (rc=$rc)" 140 | else 141 | echo "RUNPOLICY INFO: mmapplypolicy succeeded (rc=$rc)" 142 | fi 143 | 144 | echo "RUNPOLICY INFO: $(date +"%d/%m/%y %H:%M:%S") Finishing policy on $hostname" 145 | exit $rc 146 | -------------------------------------------------------------------------------- /sample-policies/list-byState.txt: -------------------------------------------------------------------------------- 1 | /* policy lists file by state */ 2 | 3 | /* define exclude list */ 4 | define( exclude_list, 5 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 6 | PATH_NAME LIKE '%/.ltfsee/%' OR 7 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 8 | PATH_NAME LIKE '%/.snapshots/%' OR 9 | NAME LIKE '.mmbackupShadow%' OR 10 | NAME LIKE 'mmbackup%') 11 | ) 12 | 13 | /* define macros for states */ 14 | define(is_premigrated,(MISC_ATTRIBUTES LIKE '%M%' AND MISC_ATTRIBUTES NOT LIKE '%V%')) 15 | define(is_migrated,(MISC_ATTRIBUTES LIKE '%V%')) 16 | define(is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%')) 17 | 18 | /* define external lists */ 19 | RULE EXTERNAL LIST 'r' EXEC '' 20 | RULE EXTERNAL LIST 'p' EXEC '' 21 | RULE EXTERNAL LIST 'm' EXEC '' 22 | 23 | /* rules to identify files by state */ 24 | RULE 'r_files' LIST 'r' WHERE (is_resident) AND (NOT exclude_list) 25 | RULE 'p_files' LIST 'p' WHERE (is_premigrated) AND (NOT exclude_list) 26 | RULE 'm_files' LIST 'm' WHERE (is_migrated) AND (NOT exclude_list) 27 | 28 | /* Invokation: 29 | mmapplypolicy path -P thispolicyfile -I defer -f ./prefix 30 | or 31 | runpolicy path thispolicyfile -I defer -f ./prefix 32 | 33 | --> resulting file lists including path and file names are stored under name: prefix.list.m (migrated files, prefix.list.r (resident files), prefix.llist.p (premigrated files) 34 | */ 35 | -------------------------------------------------------------------------------- /sample-policies/list-byStateWithTapeID.txt: -------------------------------------------------------------------------------- 1 | /* this policy lists all files by state and the tape ID */ 2 | /* to run this policy: mmapplypolicy filesystem -P policyfile -f ./filename-I defer */ 3 | /* resulting files are stored under name filename.migrated, filename.resident, filename.premigrate */ 4 | 5 | /* define macros for states */ 6 | define(is_premigrated,(MISC_ATTRIBUTES LIKE '%M%' AND MISC_ATTRIBUTES NOT LIKE '%V%')) 7 | define(is_migrated,(MISC_ATTRIBUTES LIKE '%V%')) 8 | 9 | /* define exclude list */ 10 | define( exclude_list, 11 | (PATH_NAME LIKE '%/.SpaceMan/%' OR 12 | PATH_NAME LIKE '%/.ltfsee/%' OR 13 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR 14 | PATH_NAME LIKE '%/.snapshots/%' OR 15 | NAME LIKE '.mmbackupShadow%' OR 16 | NAME LIKE 'mmbackup%') 17 | ) 18 | 19 | /* define external lists */ 20 | RULE EXTERNAL LIST 'migTapeID' EXEC '' 21 | RULE EXTERNAL LIST 'premigTapeID' EXEC '' 22 | 23 | /* show migrated files and tape */ 24 | RULE 'MIGRATED' LIST 'migTapeID' 25 | FROM POOL 'system' 26 | SHOW('migrated ' || xattr('dmapi.IBMTPS')) 27 | WHERE ( is_migrated AND NOT exclude_list ) 28 | 29 | /* show premigrated files and tape */ 30 | RULE 'PREMIGRATED' LIST 'premigTapeID' 31 | FROM POOL 'system' 32 | SHOW('premigrated ' || xattr('dmapi.IBMTPS')) 33 | WHERE (is_premigrated AND NOT exclude_list) 34 | 35 | /* Invokation: 36 | mmapplypolicy path -P thispolicyfile -I defer -f ./prefix 37 | or 38 | runpolicy path thispolicyfile -I defer -f ./prefix 39 | 40 | --> resulting file list including path and file names are stored under name prefix.list.migTapeID (migrated files with Tape ID, prefix.list.premigTapeID (premigrated files with tape ID) 41 | */ 42 | -------------------------------------------------------------------------------- /sample-policies/list-byTapeID.txt: -------------------------------------------------------------------------------- 1 | /* policy lists files on a particular tape by state */ 2 | /* the substitiution variable TAPEID has to passed from mmapplypolicy -M TAPEID=123456L8 */ 3 | 4 | RULE EXTERNAL LIST 'fileontape' EXEC '' 5 | RULE 'findfiles' LIST 'filesontape' show (XATTR('dmapi.IBMTPS')) where XATTR('dmapi.IBMTPS') like '%TAPEID%' 6 | 7 | /* Invokation: 8 | mmapplypolicy path -P thispolicyfile -M TAPEID=123456L8 -f prefix -I defer 9 | or 10 | runpolicy path thispolicyfile -M TAPEID=123456L8 -f prefix -I defer 11 | 12 | --> file list including the path and file names of files stored on TAPEID is stored under name prefix.list.filesontape 13 | */ 14 | -------------------------------------------------------------------------------- /sample-policies/migrate-all.txt: -------------------------------------------------------------------------------- 1 | /* migrate all files */ 2 | /* the substitiution variable EEPOOL has to passed from mmapplypolicy -M EEPOOL=pool1@lib1 */ 3 | 4 | /* Define exclude list to exclude SpaceMan and snapshots */ 5 | RULE 'exclude' EXCLUDE WHERE 6 | (PATH_NAME LIKE '%/.SpaceMan/%' OR /* HSM directory used for TSM and LTFS */ 7 | PATH_NAME LIKE '%/.ltfsee/%' OR /* LTFS metadata directory*/ 8 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR /* default global shared directory used by mmapplypolicy */ 9 | PATH_NAME LIKE '%/.snapshots/%' OR /* snapshot */ 10 | NAME LIKE '.mmbackupShadow%' OR /* mmbackup shadow database */ 11 | NAME LIKE 'mmbackup%') /* other mmbackup files */ 12 | 13 | /* Define LTFS as external pool */ 14 | RULE EXTERNAL POOL 'ltfs' 15 | EXEC '/opt/ibm/ltfsee/bin/eeadm' /* full path to ltfsee command must be specified */ 16 | OPTS 'EEPOOL' /* this is our pool in LTFS EE which is given by the runpolicy script*/ 17 | SIZE 10485760 18 | 19 | /* here comes the migration rule whereby the FILESYSTEM is given by the runpolicy script*/ 20 | RULE 'ee-all-mig' MIGRATE FROM POOL 'system' TO POOL 'ltfs' 21 | WHERE (KB_ALLOCATED > 0) 22 | 23 | /* Invokation: 24 | mmapplypolicy path -P thispolicyfile --single-instance -N eenodes -m 1 -M EEPOOL=pool1@lib1 25 | or 26 | runpolicy path thispolicyfile -M EEPOOL=pool1@lib1 -I yes 27 | */ 28 | -------------------------------------------------------------------------------- /sample-policies/migrate-fset.txt: -------------------------------------------------------------------------------- 1 | /* migrate all files for a particular fileset */ 2 | /* the substitiution variable EEPOOL has to passed from mmapplypolicy -M EEPOOL=pool1@lib1 */ 3 | /* the substitiution variable FSET has to passed from mmapplypolicy -M FSET=fsetname */ 4 | 5 | /* Define exclude list to exclude SpaceMan and snapshots */ 6 | RULE 'exclude' EXCLUDE WHERE 7 | (PATH_NAME LIKE '%/.SpaceMan/%' OR /* HSM directory used for TSM and LTFS */ 8 | PATH_NAME LIKE '%/.ltfsee/%' OR /* LTFS metadata directory*/ 9 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR /* default global shared directory used by mmapplypolicy */ 10 | PATH_NAME LIKE '%/.snapshots/%' OR /* snapshot */ 11 | NAME LIKE '.mmbackupShadow%' OR /* mmbackup shadow database */ 12 | NAME LIKE 'mmbackup%') /* other mmbackup files */ 13 | 14 | /* Define LTFS as external pool */ 15 | RULE EXTERNAL POOL 'ltfs' 16 | EXEC '/opt/ibm/ltfsee/bin/eeadm' /* full path to ltfsee command must be specified */ 17 | OPTS 'EEPOOL' /* this is our pool in LTFS EE which is given by the runpolicy script*/ 18 | SIZE 10485760 19 | 20 | /* here comes the migration rule whereby the FILESYSTEM is given by the runpolicy script*/ 21 | RULE 'ee-fset-mig' MIGRATE FROM POOL 'system' TO POOL 'ltfs' FOR FILESET ('FSET') 22 | WHERE (KB_ALLOCATED > 0) 23 | 24 | /* Invokation: 25 | mmapplypolicy path -P thispolicyfile --single-instance -N eenodes -m 1 -M EEPOOL=pool1@lib1 -M FSET=fsetname 26 | or 27 | runpolicy path thispolicyfile -M EEPOOL=pool1@lib1 -M FSET=fsetname -I yes 28 | */ 29 | -------------------------------------------------------------------------------- /sample-policies/premigrate-all.txt: -------------------------------------------------------------------------------- 1 | /* premigrate all resident files */ 2 | /* the substitiution variable EEPOOL has to passed from mmapplypolicy -M EEPOOL=pool1@lib1 */ 3 | 4 | /* define macro for resident files */ 5 | define( is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%') ) 6 | 7 | /* Define exclude list to exclude SpaceMan and snapshots */ 8 | RULE 'exclude' EXCLUDE WHERE 9 | (PATH_NAME LIKE '%/.SpaceMan/%' OR /* HSM directory used for TSM and LTFS */ 10 | PATH_NAME LIKE '%/.ltfsee/%' OR /* LTFS metadata directory*/ 11 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR /* default global shared directory used by mmapplypolicy */ 12 | PATH_NAME LIKE '%/.snapshots/%' OR /* snapshot */ 13 | NAME LIKE '.mmbackupShadow%' OR /* mmbackup shadow database */ 14 | NAME LIKE 'mmbackup%') /* other mmbackup files */ 15 | 16 | /* Define LTFS as external pool */ 17 | RULE EXTERNAL POOL 'ltfs' 18 | EXEC '/opt/ibm/ltfsee/bin/eeadm' /* full path to ltfsee command must be specified */ 19 | OPTS 'EEPOOL' /* this is our pool in LTFS EE which is given by the runpolicy script*/ 20 | SIZE 10485760 21 | 22 | /* here comes the premigration rule whereby the FILESYSTEM is given by the runpolicy script*/ 23 | /* see the THRESHOLD(high%, low%, premig%), kicks in at high% and premig% */ 24 | RULE 'ee-all-premig' MIGRATE FROM POOL 'system' THRESHOLD (0,100,0) TO POOL 'ltfs' 25 | WHERE (is_resident) AND (FILE_SIZE > 0) 26 | 27 | /* Invokation: 28 | mmapplypolicy path -P thispolicyfile --single-instance -N eenodes -m 1 -M EEPOOL=pool1@lib1 -M FSET=fsetname 29 | or 30 | runpolicy path thispolicyfile -M EEPOOL=pool1@lib1 -M FSET=fsetname -I yes 31 | */ 32 | -------------------------------------------------------------------------------- /sample-policies/premigrate-fset.txt: -------------------------------------------------------------------------------- 1 | /* premigrate all resident files from fileset */ 2 | /* the substitiution variable EEPOOL has to passed from mmapplypolicy -M EEPOOL=pool1@lib1 */ 3 | /* the substitiution variable FSET has to passed from mmapplypolicy -M FSET=fsetname */ 4 | 5 | 6 | /* define macro for resident files */ 7 | define( is_resident,(MISC_ATTRIBUTES NOT LIKE '%M%') ) 8 | 9 | /* Define exclude list to exclude SpaceMan and snapshots */ 10 | RULE 'exclude' EXCLUDE WHERE 11 | (PATH_NAME LIKE '%/.SpaceMan/%' OR /* HSM directory used for TSM and LTFS */ 12 | PATH_NAME LIKE '%/.ltfsee/%' OR /* LTFS metadata directory*/ 13 | PATH_NAME LIKE '%/.mmSharedTmpDir/%' OR /* default global shared directory used by mmapplypolicy */ 14 | PATH_NAME LIKE '%/.snapshots/%' OR /* snapshot */ 15 | NAME LIKE '.mmbackupShadow%' OR /* mmbackup shadow database */ 16 | NAME LIKE 'mmbackup%') /* other mmbackup files */ 17 | 18 | /* Define LTFS as external pool */ 19 | RULE EXTERNAL POOL 'ltfs' 20 | EXEC '/opt/ibm/ltfsee/bin/eeadm' /* full path to ltfsee command must be specified */ 21 | OPTS 'EEPOOL' /* this is our pool in LTFS EE which is given by the runpolicy script*/ 22 | SIZE 10485760 23 | 24 | /* here comes the premigration rule whereby the FILESYSTEM is given by the runpolicy script*/ 25 | /* see the THRESHOLD(high%, low%, premig%), kicks in at high% and premig% */ 26 | RULE 'ee-fset-premig' MIGRATE FROM POOL 'system' THRESHOLD (0,100,0) TO POOL 'ltfs' FOR FILESET ('FSET') 27 | WHERE (is_resident) AND (FILE_SIZE > 0) 28 | 29 | /* Invokation: 30 | mmapplypolicy path -P thispolicyfile --single-instance -N eenodes -m 1 -M EEPOOL=pool1@lib1 -M FSET=fsetname 31 | or 32 | runpolicy path thispolicyfile -M EEPOOL=pool1@lib1 -M FSET=fsetname -I yes 33 | */ 34 | -------------------------------------------------------------------------------- /sample-policies/readme.md: -------------------------------------------------------------------------------- 1 | ## Introduction 2 | 3 | This folder contains some sample policy files. This readme briefly describes the policies contained in the policy files and how to invoke it using the IBM Spectrum Scale policy engine. 4 | 5 | These examples have been tested in a test environment. There is no guarantee that these policies work in any environment. Furthermore it requires knowledge about the policy rule language and policy engine parameters. Refer to the [IBM Spectrum Scale ILM policy whitepaper](https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102642) for more information about the RULES and policy engine parameters. 6 | 7 | 8 | ## Policy file examples 9 | 10 | In this section the sample policy files located in this folder are explained, along with the command to run the policy engine using these policy files. The description below contains some basic parameters for the `mmapplypolicy` command, further parameters may be required. 11 | 12 | - Policy [migrate-all.txt](../sample-policies/migrate-all.txt) migrates all files that are not migrated, including pre-migrated files. The following command can be used to run this policy: 13 | 14 | 15 | mmapplypolicy path -P migrate-all.txt -M EEPOOL=pool@lib -N -m -B --single-instance 16 | 17 | 18 | The parameter `-M EEPOOL=pool@lib` denotes the destination pool for the migration and `pool@lib` must specify an existing pool in the Spectrum Archive EE system. It substitutes the variable `EEPOOL` within the policy. Additional parameters `-N -m -B --single-instance` specify the Spectrum Archive EE node names (`-N`), the number of `eeadm migrate` threads per node (`-m`), the number of files per file list (`-B`) and to run as single instance (`--single-instance`). Futher parameters may be required. 19 | 20 | 21 | - Policy [migrate-fset.txt](../sample-policies/migrate-fset.txt) migrates all files for a particular fileset that are not migrated, including pre-migrated files. To run the policy: 22 | 23 | 24 | mmapplypolicy path -P migrate-fset.txt -M EEPOOL=pool@lib -M FSET=filesetname -N -m -B --single-instance 25 | 26 | 27 | The parameter `-M EEPOOL=pool@lib` denotes the destination pool for the migration and `pool@lib` must specify an existing pool in the Spectrum Archive EE system. The parameter `-M FSET=filesetname` denotes the fileset name subject for migration and the file set named `filesetname` must exist within the specified `path`. 28 | Additional parameters `-N -m -B --single-instance` specify the Spectrum Archive EE node names (`-N`), the number of `eeadm migrate` threads per node (`-m`), the number of files per file list (`-B`) and to run as single instance (`--single-instance`). Futher parameters may be required. 29 | 30 | 31 | - Policy [premigrate-all.txt](../sample-policies/premigrate-all.txt) premigrates all resident files. To run the policy: 32 | 33 | 34 | mmapplypolicy path -P premigrate-all.txt -M EEPOOL=pool@lib -N -m -B --single-instance 35 | 36 | 37 | The parameter `-M EEPOOL=pool@lib` denotes the destination pool for the premigration and `pool@lib` must specify an existing pool in the Spectrum Archive EE system. It substitutes the variable `EEPOOL` within the policy. Additional parameters `-N -m -B --single-instance` specify the Spectrum Archive EE node names (`-N`), the number of `eeadm migrate` threads per node (`-m`), the number of files per file list (`-B`) and to run as single instance (`--single-instance`). Futher parameters may be required. 38 | 39 | 40 | - Policy [premigrate-fset.txt](../sample-policies/premigrate-fset.txt) migrates all files for a particular fileset that are not migrated, including pre-migrated files. To run the policy: 41 | 42 | 43 | mmapplypolicy path -P premigrate-fset.txt -M EEPOOL=pool@lib -M FSET=filesetname -N -m -B --single-instance 44 | 45 | 46 | The parameter `-M EEPOOL=pool@lib` denotes the destination pool for the premigration and `pool@lib` must specify an existing pool in the Spectrum Archive EE system. The parameter `-M FSET=filesetname` denotes the fileset name subject for premigration and the file set named `filesetname` must exist within the specified `path`. 47 | Additional parameters `-N -m -B --single-instance` specify the Spectrum Archive EE node names (`-N`), the number of `eeadm migrate` threads per node (`-m`), the number of files per file list (`-B`) and to run as single instance (`--single-instance`). Futher parameters may be required. 48 | 49 | 50 | - Policy [recall-all.txt](../sample-policies/recall-all.txt) recalls all files located in a specified directory (`RECAllDIR`). After recall files are in status `premigrated`. To run the policy: 51 | 52 | 53 | mmapplypolicy path -P recall-all.txt -M RECALLDIR=recallpath -N -m -B --single-instance 54 | 55 | 56 | The parameter `-M RECALLDIR=recallpath` denotes the path in the Spectrum Scale file system subject for recall. All migrated files in this path are being recalled. Additional parameters `-N -m -B --single-instance` specify the Spectrum Archive EE node names (`-N`), the number of `eeadm recall` threads per node (`-m`), the number of files per file list (`-B`) and to run as single instance (`--single-instance`). Futher parameters may be required. 57 | 58 | 59 | - Policy [list-byState.txt](../sample-policies/list-byState.txt) creates lists of all files in accordance to their migration state. For each migration state (resident, migrated, premigrated) a separate file list is created. To run the policy: 60 | 61 | 62 | mmapplypolicy path -P list-byState.txt -I defer -f dirPrefix 63 | 64 | 65 | The resulting file lists are stored in a directory denoted by parameter `-f dirPrefix`. File names of resident files are stored in file `dirPrefix.list.r`, file names of premigrated files are stored in file `dirPrefix.list.p` and file names of migrated files are stored in file `dirPrefix.list.m`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file lists would be named: `/tmp/files.list.r`, `/tmp/files.list.p` and `/tmp/files.list.m`. 66 | 67 | 68 | - Policy [list-byTapeID.txt](../sample-policies/list-byTapeID.txt) creates a list of all files that are located on a particular tape ID. The tape ID is specified in the command line. To run the policy: 69 | 70 | 71 | mmapplypolicy path -P list-byTapeID.txt -I defer -f dirPrefix -M TAPEID=volser 72 | 73 | The tape ID for which the files should be listed in provided with parameter `-M TAPEID=volser`. 74 | The resulting file list is stored in a directory denoted by parameter `-f dirPrefix` and named `dirPrefix.list.fileontape`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file list would be named: `/tmp/files.list.filesontape`. This file list includes all path and file names of files stored on the subject tape ID. This is includes files that have the primary or a secondary copy on the subject tape. 75 | 76 | 77 | - Policy [list-byStateWithTapeID.txt](../sample-policies/list-byStateWithTapeID.txt) creates a lists of all files that that are in premigrated and migrated state including the tape ID where these files are (pre)migrated to. To run the policy: 78 | 79 | 80 | mmapplypolicy path -P list-byStateWithTapeID.txt -I defer -f dirPrefix 81 | 82 | The resulting file lists are stored in a directory denoted by parameter `-f dirPrefix`. File names and tape ID of premigrated files are stored in file `dirPrefix.list.premigTapeID` and file names and tape ID of migrated files are stored in file `dirPrefix.list.migTapeID`. For example if the parameter `dirPrefix` is set to `/tmp/files`, then the file lists would be named: `/tmp/files.list.premigTapeID` and `/tmp/files.list.migTapeID`. 83 | 84 | -------------------------------------------------------------------------------- /sample-policies/recall-all.txt: -------------------------------------------------------------------------------- 1 | /* recall all files from the given file system and directory */ 2 | /* the substitiution variable RECALLDIR has to passed from mmapplypolicy -M RECALLDIR=/path */ 3 | 4 | 5 | /* define macros */ 6 | define(recall_dir, (PATH_NAME LIKE '%RECALLDIR%')) 7 | 8 | /* Define LTFS as external pool */ 9 | RULE EXTERNAL POOL 'ltfs' 10 | EXEC '/opt/ibm/ltfsee/bin/eeadm' /* full path to ltfsee command must be specified */ 11 | 12 | /* here comes the migration rule whereby the FILESYSTEM is given by the runpolicy script*/ 13 | RULE 'recall' MIGRATE FROM POOL 'ltfs' TO POOL 'system' 14 | WHERE (XATTR('dmapi.IBMPMig') IS NULL) AND 15 | NOT (XATTR('dmapi.IBMTPS') IS NULL) AND (recall_dir) 16 | 17 | /* Invokation: 18 | mmapplypolicy path -P thispolicyfile --single-instance -N eenodes -m 1 -M RECALLDIR=/path 19 | or 20 | runpolicy path thispolicyfile -M EEPOOL=pool1@lib1 -I yes -M RECALLDIR=/path 21 | */ 22 | --------------------------------------------------------------------------------