├── scheme.png
├── .github
    └── workflows
    │   └── shellcheck.yml
├── LICENSE
├── CHANGELOG.md
├── README.md
└── pgbackrest_auto


/scheme.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vitabaks/pgbackrest_auto/HEAD/scheme.png


--------------------------------------------------------------------------------
/.github/workflows/shellcheck.yml:
--------------------------------------------------------------------------------
 1 | name: 'ShellCheck'
 2 | 
 3 | on:
 4 |   push:
 5 |     branches:
 6 |       - master
 7 |   pull_request:
 8 |     branches:
 9 |       - master
10 | 
11 | jobs:
12 |   shellcheck:
13 |     name: Shellcheck
14 |     runs-on: ubuntu-latest
15 |     steps:
16 |     - uses: actions/checkout@v2
17 |     - name: Run ShellCheck
18 |       uses: ludeeus/action-shellcheck@master
19 |       with:
20 |         severity: error
21 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2019 Vitaliy Kukharik
 4 | Copyright (c) 2020 parcIT GmbH
 5 | 
 6 | Permission is hereby granted, free of charge, to any person obtaining a copy
 7 | of this software and associated documentation files (the "Software"), to deal
 8 | in the Software without restriction, including without limitation the rights
 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10 | copies of the Software, and to permit persons to whom the Software is
11 | furnished to do so, subject to the following conditions:
12 | 
13 | The above copyright notice and this permission notice shall be included in all
14 | copies or substantial portions of the Software.
15 | 
16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22 | SOFTWARE.
23 | 


--------------------------------------------------------------------------------
/CHANGELOG.md:
--------------------------------------------------------------------------------
 1 | ## Version 1.3.1 (17 June 2022)
 2 | 
 3 | - stop postgres before running pg_checksums
 4 | - function dummy_dump: specify the full path to pg_dump
 5 | - function pg_checksums: the logic of processing the test result has been changed
 6 | 
 7 | ## Version 1.3 (17 June 2022)
 8 | 
 9 |  - check data checksums with pg_checksums. "--checksums" (and "--checkdb") option (requires `postgresql-<version>-pg-checksums` package for PostgreSQL version 11 and below)
10 |  - "--dummy-dump" option. Verify that data can be read out. Check with pg_dump >> /dev/null
11 |  - "--amcheck" option. Validate Indexes (verify the logical consistency of the structure of indexes and heap relations)
12 |  - "--norestore" option to check already existing clusters
13 |  - automatic determine Postgres version from pgbackrest info
14 |  - automatic create a new postgres cluster (initdb) to restore to the path specified in the "--to" option (if it does not exist)
15 |  - determine postgresql parameters from pg_controldata and configure postgresql.conf accordingly after restore
16 |  - compare DB and filesystem size before restore
17 |  - remove colors in log messages
18 |  - remove dependencies - gawk, ansi2html.sh
19 |  - a little code refactoring
20 | 
21 | ## Version 1.2 (20 Apr 2020)
22 | 
23 | - Bug Fixes: The pg_logical_validation() function was creating the amcheck extension, but did not actually perform indexes checking with bt_index_parent_check for PostgreSQL version 11 or later.
24 | 
25 | ## Version 1.1 (27 Jan 2020)
26 | 
27 | - Compatibility with PostgreSQL versions 11 and 12.
28 | 
29 | ## Version 1.0 (27 Jan 2020)
30 | 
31 | - Initial release
32 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # pgbackrest_auto
  2 | 
  3 | This is a simple script to automate the process of restore PostgreSQL databases from backup using [pgbackrest](https://github.com/pgbackrest/pgbackrest). With the possibility **validate for physical and logical database corruption**, and (optional) sending the final report to DBA an e-mail address.
  4 | 
  5 | #### Data Validation
  6 | 
  7 | >PostgreSQL has supported page-level checksums since 9.3. **If page checksums are enabled** pgbackrest will [validate](https://pgbackrest.org/configuration.html#section-backup/option-checksum-page) the checksums for every file that is copied during a backup. All page checksums are validated during a full backup and checksums in files that have changed are validated during differential and incremental backups.
  8 | This feature allows page-level corruption to be detected early.
  9 | 
 10 | `pgbackrest_auto` runs **additional advanced checks on your data** after the recovery stage. In case of PostgreSQL data restore and recovery, the stage of physical data validation is performed (all data blocks are sequentially read with pg_dump);
 11 | Databases that successfully passed the stage of physical validation additionally pass the stage of logical validation using the [amcheck](https://github.com/petergeoghegan/amcheck) extension. Tthe logical integrity of the structure of the B-Tree indexes and tables related to the target index relation is checked (used `bt_index_parent_check` with `heapallindexed` argument is `true`).
 12 | 
 13 | ![pgbackrest_auto_scheme](./scheme.png)
 14 | 
 15 | #### Example
 16 | ```
 17 | pgbackrest_auto --from=epgdb --to=/bkpdata/rst/epgdb --checkdb
 18 | ```
 19 | ###### result:
 20 | ```
 21 | 2022-06-15 14:13:13 INFO: [STEP 1]: Starting
 22 | 2022-06-15 14:13:13 INFO: Starting. Restore Type: Full PostgreSQL Restore FROM Stanza: epgdb --> TO Directory: /bkpdata/rst/epgdb
 23 | 2022-06-15 14:13:14 INFO: Starting. Restore Settings: immediate
 24 | 2022-06-15 14:13:14 INFO: Starting. Run settings: Backup host: localhost
 25 | 2022-06-15 14:13:14 INFO: Starting. Run settings: Log: /var/log/pgbackrest/pgbackrest_auto_epgdb.log
 26 | 2022-06-15 14:13:14 INFO: Starting. Run settings: Lock run: /tmp/pgbackrest_auto_epgdb.lock
 27 | 2022-06-15 14:13:14 INFO: Starting. PostgreSQL instance: epgdb
 28 | 2022-06-15 14:13:14 INFO: Starting. PostgreSQL version: 11
 29 | 2022-06-15 14:13:14 INFO: Starting. PostgreSQL port: 5432
 30 | 2022-06-15 14:13:14 INFO: Starting. PostgreSQL Database Validation: yes
 31 | 2022-06-15 14:13:14 WARN: Restoring to /bkpdata/rst/epgdb Waiting 30 seconds. The directory will be overwritten. If mistake, press ^C
 32 | 2022-06-15 14:13:44 INFO: [STEP 2]: Stopping PostgreSQL
 33 | 2022-06-15 14:13:44 INFO: attempt: 1/3600
 34 | 2022-06-15 14:13:44 INFO: PostgreSQL check status
 35 | 2022-06-15 14:13:44 INFO: PostgreSQL instance epgdb not running
 36 | 2022-06-15 14:13:44 INFO: [STEP 3]: Restoring from backup
 37 | 2022-06-15 14:13:44 INFO: Restore from backup started. Type: Full PostgreSQL Restore
 38 | 2022-06-15 14:13:44 INFO: See detailed log in the file /var/log/pgbackrest/epgdb-restore.log
 39 | pgbackrest --config=/tmp/pgbackrest_auto.conf --repo1-host=localhost --repo1-host-user=postgres --stanza=epgdb --pg1-path=/bkpdata/rst/epgdb  --type=immediate --repo1-path=/bkpdata/pgbackrest --delta restore --process-max=4 --log-level-console=error --log-level-file=detail --recovery-option=recovery_target_action=promote --tablespace-map-all=/bkpdata/rst/epgdb/remapped_tablespaces
 40 | 2022-06-15 14:13:52 INFO: Restore from backup done
 41 | 2022-06-15 14:13:52 INFO: [STEP 4]: PostgreSQL Starting for recovery
 42 | 2022-06-15 14:13:52 INFO: PostgreSQL start
 43 | 2022-06-15 14:14:02 INFO: attempt: 1/3600
 44 | 2022-06-15 14:14:03 INFO: PostgreSQL instance epgdb started and accepting connections
 45 | 2022-06-15 14:14:03 INFO: [STEP 5]: PostgreSQL Recovery Checking
 46 | 2022-06-15 14:14:03 INFO: Checking if restoring from archive is done
 47 | 2022-06-15 14:14:03 INFO: Replayed:
 48 | 2022-06-15 14:14:03 INFO: Restoring from archive is done
 49 | 2022-06-15 14:14:03 INFO: Restore done
 50 | 2022-06-15 14:14:03 INFO: [STEP 6]: Validate for physical database corruption
 51 | 2022-06-15 14:14:04 INFO: Start data validation for database postgres
 52 | 2022-06-15 14:14:04 INFO: ... starting pg_dump -p 5432 -d postgres >> /dev/null
 53 | 2022-06-15 14:14:06 INFO: Data validation in the database postgres - Successful
 54 | 2022-06-15 14:14:06 INFO: Start data validation for database epg
 55 | 2022-06-15 14:14:06 INFO: ... starting pg_dump -p 5432 -d epg >> /dev/null
 56 | 2022-06-15 14:14:12 INFO: Data validation in the database epg - Successful
 57 | 2022-06-15 14:14:12 INFO: Start data validation for database ota
 58 | 2022-06-15 14:14:12 INFO: ... starting pg_dump -p 5432 -d ota >> /dev/null
 59 | 2022-06-15 14:14:13 INFO: Data validation in the database ota - Successful
 60 | 2022-06-15 14:14:13 INFO: [STEP 7]: Validate for logical database corruption
 61 | 2022-06-15 14:14:13 INFO: pg_checksums: starting data checksums validation
 62 | 2022-06-15 14:14:21 INFO: pg_checksums: data checksums validation - Successful
 63 | 2022-06-15 14:14:22 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database postgres
 64 | 2022-06-15 14:14:22 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database epg
 65 | 2022-06-15 14:14:59 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database ota
 66 | 2022-06-15 14:15:23 INFO: Finish
 67 | ```
 68 | 
 69 | 
 70 | #### pgbackrest_auto --help
 71 | ```
 72 | Automatic Restore and Validate for physical and logical database corruption
 73 | 
 74 | Support three types of restore:
 75 |         1) Restore last backup  (recovery to earliest consistent point) [default]
 76 |         2) Restore latest       (recovery to the end of the archive stream)
 77 |         3) Restore to the point (recovery to restore point)
 78 | 
 79 | Important: Run on the nodes on which you want to restore the backup
 80 | 
 81 | Usage: /usr/bin/pgbackrest_auto --from=STANZANAME --to=DATA_DIRECTORY [ --datname=DATABASE [...] ] [ --recovery-type=( default | immediate | time ) ] [ --recovery-target=TIMELINE  [ --backup-set=SET ] [ --pgver= ] [ --checkdb ] [ --clear ] [ --report ] ]
 82 | 
 83 | --from=STANZANAME
 84 |         Stanza from which you need to restore from a backup
 85 | 
 86 | --to=DATA_DIRECTORY
 87 |         PostgreSQL Data directory Path to restore from a backup
 88 |         a PostgreSQL database cluster (PGDATA) will be automatically created if it does not exist
 89 |         Example: /bkpdata/rst/app-db
 90 | 
 91 | --datname=DATABASE [...]
 92 |         Database name to be restored (After this you MUST drop other databases)
 93 |         Note that built-in databases (template0, template1, and postgres) are always restored.
 94 |         To be restore more than one database specify them in brackets separated by spaces.
 95 |         Example: --datname="db1 db2"
 96 | 
 97 | --recovery-type=TYPE
 98 |         immediate - recover only until the database becomes consistent           (Type 1. Restore last backup)  [default]
 99 |         default   - recover to the end of the archive stream                     (Type 2. Restore latest)
100 |         time      - recover to the time specified in --recovery-target           (Type 3. Restore to the point)
101 | 
102 | --recovery-target=TIMELINE
103 |         time - recovery point time. The time stamp up to which recovery will proceed.
104 |         if --recovery-type=time
105 |         Example: "2022-06-14 09:00:00"
106 | 
107 | --backup-set=SET
108 |         If you need to restore not the most recent backup. Example few days ago.
109 |         Get info of backup. Login to pgbackrest server. User postgres
110 |         pgbackrest --stanza=[STANZA NAME] info
111 |         And get it. Example:
112 |                     incr backup: 20220611-000004F_20220614-000003D
113 |         This is the name of SET: 20220611-000004F_20220614-000003D
114 | 
115 | --pgver=VERSION
116 |         PostgreSQL cluster (instance) version [ optional ]
117 |         by default, the PostgreSQL version will be determined from the pgbackrest info
118 | 
119 | --dummy-dump
120 |         Verify that data can be read out. Check with pg_dump
121 | 
122 | --checksums
123 |         Check data checksums
124 | 
125 | --amcheck
126 |         Validate Indexes (verify the logical consistency of the structure of indexes and heap relations)
127 | 
128 | --checkdb
129 |         Validate for Physical and Logical Database Corruption (includes: dummy-dump, checksums, amcheck)
130 | 
131 | --clear
132 |         Clear PostgreSQL Data directory after Restore (the path was specified in the "--to" parameter ) [ optional ]
133 | 
134 | --report
135 |         Send report to mail address
136 | 
137 | --norestore
138 |         Do not restore a stanza but use an already existing cluster
139 | 
140 | --config=/path/to/pgbackrest.conf
141 |         The path to the custom pgbackrest configuration file [ optional ]
142 | 
143 | --custom-options=
144 |         Costom options for pgBackRest [ optional ]
145 |         This includes all the options that may also be configured in pgbackrest.conf
146 |         Example: --option1=value --option2=value --option3=value
147 |         See all available options: https://pgbackrest.org/configuration.html
148 | 
149 | --process-max=
150 |         Max processes to use for restore and validate (default 1).
151 | 
152 | EXAMPLES:
153 | ( example stanza "app-db" , backup host "localhost" (default value) )
154 | 
155 | | Restore last backup:
156 | 
157 |     /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db
158 | 
159 | | Restore backup made a few days ago:
160 | 
161 |     /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D
162 | 
163 | | Restore backup made a few days ago and pick time:
164 | 
165 |     /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target="2022-06-14 09:00:00"
166 | 
167 | | Restore backup made a few days ago and pick time. And we have restore only one database with the name "app_db":
168 | 
169 |     /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target="2022-06-14 09:00:00" --datname=app_db
170 | 
171 | | Restore and Validate of databases:
172 | 
173 |     /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb
174 | 
175 | ```
176 | 
177 | ---
178 | 
179 |  :bulb: You can use this script to daily automatically check your backups, immediately after the completion of the backup process.
180 | 
181 | ###### Example of Cron jobs:
182 | 
183 | ```
184 | #=== pgbackrest - Backup PostgreSQL ====================
185 | 
186 | 01 00 * * 6 if pgbackrest --stanza=app-db --type=full backup; then pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb --clear --report; fi
187 | 01 00 * * 0-5 if pgbackrest --stanza=app-db --type=diff backup; then pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb --clear --report; fi
188 | 
189 | 30 00 * * 6 if pgbackrest --stanza=apdb-cluster --type=full backup; then pgbackrest_auto --from=apdb-cluster --to=/bkpdata/rst/apdb-cluster --checkdb --clear --report; fi
190 | 30 00 * * 0-5 if pgbackrest --stanza=apdb-cluster --type=diff backup; then pgbackrest_auto --from=apdb-cluster --to=/bkpdata/rst/apdb-cluster --checkdb --clear --report; fi
191 | 
192 | 00 01 * * 6 if pgbackrest --stanza=dbs-eu --type=full backup; then pgbackrest_auto --from=dbs-eu--to=/bkpdata/rst/dbs-eu --checkdb --clear --report; fi
193 | 00 01 * * 0-5 if pgbackrest --stanza=dbs-eu --type=diff backup; then pgbackrest_auto --from=dbs-eu--to=/bkpdata/rst/dbs-eu --checkdb --clear --report; fi
194 | 
195 | #=======================================================
196 | ```
197 | 
198 | ## Compatibility
199 | RedHat and Debian based distros
200 | 
201 | ###### PostgreSQL versions:
202 | all supported PostgreSQL versions
203 | 
204 | ## Requirements
205 | `pgbackrest` and `jq` packages.
206 | 
207 | for `--checksums` (and `--checkdb`):
208 |   - `postgresql-<version>-pg-checksums` package (if PostgreSQL version <= 11)
209 | 
210 | for `--amcheck` (and `--checkdb`):
211 |   - `postgresql-<version>-amcheck` package (if PostgreSQL version <= 10)
212 |     (_the amcheck extension will be automatically installed to the restored databases_)
213 | 
214 | for `--report`:
215 |   - `sendemail` package
216 |   - specify smtp parameters `smtp_server`, `mail_from`, `mail_to` in the pgbackrest_auto script file.
217 | 
218 | Run as user: `postgres`
219 | 
220 | If your PostgreSQL is installed somewhere other than the default installation path, please specify the `PG_BIN_DIR` variable in the script file.
221 | 
222 | ## Installation
223 | 1. Download and copy the `pgbackrest_auto` script to `/usr/bin/` directory
224 | 2. Grant execute rights on the scripts
225 | 
226 | Example:
227 | ```
228 | wget https://raw.githubusercontent.com/vitabaks/pgbackrest_auto/master/pgbackrest_auto
229 | sudo mv pgbackrest_auto /usr/bin/
230 | sudo chown postgres:postgres /usr/bin/pgbackrest_auto
231 | sudo chmod 750 /usr/bin/pgbackrest_auto
232 | ```
233 | 
234 | 
235 | ## Logging
236 | Log file: `/var/log/pgbackrest/pgbackrest_auto_<STANZANAME>.log`
237 | 
238 | In addition, the script execution is written in syslog. Get the pgbackrest_auto log:
239 | ```
240 | sudo grep pgbackrest_auto /var/log/syslog
241 | ```
242 | 
243 | ## License
244 | Licensed under the MIT License. See the [LICENSE](./LICENSE) file for details.
245 | 
246 | ## Author
247 | Vitaliy Kukharik (PostgreSQL DBA) vitabaks@gmail.com
248 | 
249 | ## Feedback, bug-reports, requests, ...
250 | Are [welcome](https://github.com/vitabaks/pgbackrest_auto/issues)!
251 | #### Help Wanted! If you noticed a bug or a missing feature or just have an idea of how this project could be enhanced, please feel free to file an issue.
252 | 


--------------------------------------------------------------------------------
/pgbackrest_auto:
--------------------------------------------------------------------------------
  1 | #!/bin/bash
  2 | # Author: Vitaliy Kukharik (vitabaks@gmail.com)
  3 | # Title: /usr/bin/pgbackrest_auto - Automatic Restore and Validate for physical and logical database corruption
  4 | 
  5 | # Dependencies: "pgbackrest", "jq" packages;
  6 | # for "--checksums" (and "--checkdb"): postgresql-<version>-pg-checksums package (if PostgreSQL version <= 11)
  7 | # for "--amcheck" (and "--checkdb"): postgresql-<version>-amcheck package (if PostgreSQL version <= 10)
  8 | # for "--report": sendemail
  9 | # Run as user: postgres
 10 | 
 11 | ver="1.6.0"
 12 | 
 13 | # variables for function "sendmail()"
 14 | smtp_server="10.128.64.5:25"
 15 | mail_from="backuper@my-domain.com"
 16 | mail_to="dba@my-domain.com"
 17 | attach_report=false  # or 'true'
 18 | 
 19 | # Send report to mail address
 20 | function sendmail(){
 21 |     SMTP="${smtp_server}"
 22 |     EMAIL="${mail_to}"
 23 |     SUBJECT="postgres restore report for '${FROM}': $(date +%Y-%m-%d) (auto-generated)"
 24 |     MAIL_FROM="${mail_from}"
 25 | 
 26 |     # send mail
 27 |     if [ "$attach_report" = true ]; then
 28 |         sendemail -v -o message-content-type=text -o message-file="${log}" -f "${MAIL_FROM}" -t "${EMAIL}" -u "${SUBJECT}" -s "${SMTP}" -a "${log}"
 29 |     else
 30 |         sendemail -v -o message-content-type=text -o message-file="${log}" -f "${MAIL_FROM}" -t "${EMAIL}" -u "${SUBJECT}" -s "${SMTP}"
 31 |     fi
 32 | }
 33 | 
 34 | function info(){
 35 |     msg="$1"
 36 |     echo -e "$(date "+%F %T") INFO: $msg"
 37 |     logger -p user.notice -t "$(basename "$0")" "$msg"
 38 | }
 39 | function warnmsg(){
 40 |     msg="$1"
 41 |     echo -e "$(date "+%F %T") \e[33mWARN:\e[0m $msg"
 42 |     logger -p user.notice -t "$(basename "$0")" "$msg"
 43 |     return 1
 44 | }
 45 | function error(){
 46 |     msg="$1"
 47 |     echo -e "$(date "+%F %T") \e[91mERROR:\e[0m $msg"
 48 |     sed -i 's/Result_status=1/Result_status=0/g' "${status_file}"
 49 |     logger -p user.error -t "$(basename "$0")" "$msg"
 50 |     # sendmail
 51 |     if [[ "${REPORT}" = "yes" ]]; then sendmail; fi
 52 |     # remove lock file
 53 |     if [ -f "${lock}" ]; then rm "${lock}"; fi
 54 |     exit 1
 55 | }
 56 | 
 57 | 
 58 | while getopts ":-:" optchar; do
 59 |     [[ "${optchar}" == "-" ]] || continue
 60 |     case "${OPTARG}" in
 61 |         config=* )
 62 |             CONFIG=${OPTARG#*=}
 63 |             ;;
 64 |         from=* )
 65 |             FROM=${OPTARG#*=}
 66 |             ;;
 67 |         to=* )
 68 |             TO=${OPTARG#*=}
 69 |             ;;
 70 |         datname=* )
 71 |             DATNAME=${OPTARG#*=}
 72 |             ;;
 73 |         backup-set=* )
 74 |             BACKUPSET=${OPTARG#*=}
 75 |             ;;
 76 |         recovery-type=* )
 77 |             RECOVERYTYPE=${OPTARG#*=}
 78 |             ;;
 79 |         recovery-target=* )
 80 |             RECOVERYTARGET=${OPTARG#*=}
 81 |             ;;
 82 |         pgver=* )
 83 |             PGVER=${OPTARG#*=}
 84 |             ;;
 85 |         dummy-dump )
 86 |             DUMMYDUMP=yes
 87 |             ;;
 88 |         checksums )
 89 |             CHECKSUMS=yes
 90 |             ;;
 91 |         amcheck )
 92 |             AMCHECK=yes
 93 |             ;;
 94 |         checkdb )
 95 |             CHECKDB=yes
 96 |             ;;
 97 |         clear )
 98 |             CLEAR=yes
 99 |             ;;
100 |         report )
101 |             REPORT=yes
102 |             ;;
103 |         norestore )
104 |             NORESTORE=yes
105 |             ;;
106 |         custom-options=* )
107 |             CUSTOMOTIONS=${OPTARG#*=}
108 |             ;;
109 |         process-max=* )
110 |             PROCESS_MAX=${OPTARG#*=}
111 |             ;;
112 |     esac
113 | done
114 | 
115 | 
116 | function help(){
117 | echo -e "
118 | Automatic Restore and Validate for physical and logical database corruption
119 | 
120 | Support three types of restore:
121 |         1) Restore last backup  (recovery to earliest consistent point) [default]
122 |         2) Restore latest       (recovery to the end of the archive stream)
123 |         3) Restore to the point (recovery to restore point)
124 | 
125 | Important: Run on the nodes on which you want to restore the backup
126 | 
127 | Usage: $0 --from=STANZANAME --to=DATA_DIRECTORY [ --datname=DATABASE [...] ] [ --recovery-type=( default | immediate | time ) ] [ --recovery-target=TIMELINE  [ --backup-set=SET ] [ --pgver= ] [ --checkdb ] [ --clear ] [ --report ] ]
128 | 
129 | --from=STANZANAME
130 |         Stanza from which you need to restore from a backup
131 | 
132 | --to=DATA_DIRECTORY
133 |         PostgreSQL Data directory Path to restore from a backup
134 |         a PostgreSQL database cluster (PGDATA) will be automatically created if it does not exist
135 |         Example: /bkpdata/rst/app-db
136 | 
137 | --datname=DATABASE [...]
138 |         Database name to be restored (After this you MUST drop other databases)
139 |         Note that built-in databases (template0, template1, and postgres) are always restored.
140 |         To be restore more than one database specify them in brackets separated by spaces.
141 |         Example: --datname=\"db1 db2\"
142 | 
143 | --recovery-type=TYPE
144 |         immediate - recover only until the database becomes consistent           (Type 1. Restore last backup)  [default]
145 |         default   - recover to the end of the archive stream                     (Type 2. Restore latest)
146 |         time      - recover to the time specified in --recovery-target           (Type 3. Restore to the point)
147 | 
148 | --recovery-target=TIMELINE
149 |         time - recovery point time. The time stamp up to which recovery will proceed.
150 |         if --recovery-type=time
151 |         Example: \"2022-06-14 09:00:00\"
152 | 
153 | --backup-set=SET
154 |         If you need to restore not the most recent backup. Example few days ago.
155 |         Get info of backup. Login to pgbackrest server. User postgres
156 |         pgbackrest --stanza=[STANZA NAME] info
157 |         And get it. Example:
158 |                     incr backup: 20220611-000004F_20220614-000003D
159 |         This is the name of SET: 20220611-000004F_20220614-000003D
160 | 
161 | --pgver=VERSION
162 |         PostgreSQL cluster (instance) version [ optional ]
163 |         by default, the PostgreSQL version will be determined from the pgbackrest info
164 | 
165 | --dummy-dump
166 |         Verify that data can be read out. Check with pg_dump
167 | 
168 | --checksums
169 |         Check data checksums
170 | 
171 | --amcheck
172 |         Validate Indexes (verify the logical consistency of the structure of indexes and heap relations)
173 | 
174 | --checkdb
175 |         Validate for Physical and Logical Database Corruption (includes: dummy-dump, checksums, amcheck)
176 | 
177 | --clear
178 |         Clear PostgreSQL Data directory after Restore (the path was specified in the \"--to\" parameter ) [ optional ]
179 | 
180 | --report
181 |         Send report to mail address
182 | 
183 | --norestore
184 |         Do not restore a stanza but use an already existing cluster
185 | 
186 | --config=/path/to/pgbackrest.conf
187 |         The path to the custom pgbackrest configuration file [ optional ]
188 | 
189 | --custom-options=
190 |         Costom options for pgBackRest [ optional ]
191 | 	This includes all the options that may also be configured in pgbackrest.conf
192 |         Example: "--option1=value --option2=value --option3=value"
193 |         See all available options: https://pgbackrest.org/configuration.html
194 | 
195 | --process-max=
196 |         Max processes to use for restore and validate (default 1).
197 | 
198 | EXAMPLES:
199 | ( example stanza \"app-db\" , backup host \"localhost\" (default value) )
200 | 
201 | | Restore last backup:
202 | 
203 |     $0 --from=app-db --to=/bkpdata/rst/app-db
204 | 
205 | | Restore backup made a few days ago:
206 | 
207 |     $0 --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D
208 | 
209 | | Restore backup made a few days ago and pick time:
210 | 
211 |     $0 --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target=\"2022-06-14 09:00:00\"
212 | 
213 | | Restore backup made a few days ago and pick time. And we have restore only one database with the name \"app_db\":
214 | 
215 |     $0 --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target=\"2022-06-14 09:00:00\" --datname=app_db
216 | 
217 | | Restore and Validate of databases:
218 | 
219 |     $0 --from=app-db --to=/bkpdata/rst/app-db --checkdb
220 | "
221 | exit
222 | }
223 | [ "$1" = "-h" ] || [ "$1" = "--help" ] || [ "$1" = "help" ] && help
224 | if [ "$1" = "-v" ] || [ "$1" = "--version" ] || [ "$1" = "version" ]; then echo "$0 version ${ver}" && exit; fi
225 | 
226 | USR=$(whoami)
227 | if [ "$USR" != 'postgres' ]; then
228 |     error "$0 must be run as postgres"
229 | fi
230 | 
231 | # check if sendemail exists
232 | if [[ "${REPORT}" = "yes" ]]; then
233 |     if ! command -v sendemail &> /dev/null
234 |     then
235 |         warnmsg "sendemail could not be found. Please install the sendemail package"
236 |         exit
237 |     fi
238 | fi
239 | 
240 | # check if jq exists
241 | if ! command -v jq &> /dev/null
242 | then
243 |     warnmsg "jq could not be found. Please install the jq package"
244 |     exit
245 | fi
246 | 
247 | # Log file
248 | log="/var/log/pgbackrest/pgbackrest_auto_${FROM}.log"
249 | 
250 | # Status file - contains status of restore for monitoring
251 | status_file="/var/log/pgbackrest/pgbackrest_auto_${FROM}.status"
252 | ## 1 - success
253 | ## 0 - error
254 | 
255 | # Lock file
256 | lock="/tmp/pgbackrest_auto_${FROM}.lock"
257 | exec 9>"${lock}"
258 | flock -n 9 || exit
259 | 
260 | [[ -z "${FROM}" ]] && error "--from is missing"
261 | [[ -z "${TO}" ]] && error "--to is missing"
262 | [[ -z $RECOVERYTYPE && -n $RECOVERYTARGET ]] && error "--recovery-type is missing"
263 | if [[ $RECOVERYTYPE != default ]]; then
264 |     [[ -n $RECOVERYTYPE && -z $RECOVERYTARGET ]] && error "--recovery-target is missing"
265 | fi
266 | # default recovery-type = immediate
267 | if [[ -z $RECOVERYTYPE ]]; then RECOVERYTYPE="immediate"; fi
268 | 
269 | [[ $RECOVERYTYPE = immediate || $RECOVERYTYPE = default || $RECOVERYTYPE = time ]] || error "--recovery-type=( immediate | default | time )"
270 | [[ $RECOVERYTYPE = default && -n $RECOVERYTARGET ]] && error "Not use --recovery-type=default with --recovery-target"
271 | if [[ -n $DATNAME && -n $CHECKDB ]]; then error "Not use --checkdb with --datname. It work with only Full PostgreSQL Restore"; fi
272 | 
273 | # PostgreSQL variables
274 | 
275 | # version
276 | if [[ -z $PGVER ]]; then
277 |     # get the postgres version from stanza
278 |     PGVER=$(pgbackrest info --stanza="${FROM}" --output=json | jq -r '.[].db[-1]."version"')
279 |     if [[ "$PGVER" = "null" ]]; then
280 |         error "could not determine the postgres version using \"pgbackrest info --stanza=${FROM}\""
281 |     fi
282 | fi
283 | 
284 | # data diretory
285 | PGDATA="${TO}"
286 | 
287 | # bin diretory
288 | PG_BIN_DIR=""
289 | 
290 | # try to define the bin directory (if PG_BIN_DIR is not set)
291 | if [[ -z "${PG_BIN_DIR}" ]]; then
292 |     if [[ -d /usr/lib/postgresql/"${PGVER}"/bin ]]; then
293 |         PG_BIN_DIR=/usr/lib/postgresql/"${PGVER}"/bin
294 |         PG_CONF_DIR="${PGDATA}"
295 |     elif [[ -d /usr/pgsql-"${PGVER}"/bin ]]; then
296 |         PG_BIN_DIR=/usr/pgsql-"${PGVER}"/bin
297 |         PG_CONF_DIR="${PGDATA}"
298 |     else
299 |         error "The bin directory for PostgreSQL ${PGVER} was not found. \
300 |     Please check if the appropriate version of PostgreSQL packages is installed."
301 |     fi
302 | fi
303 | 
304 | # check if pg_ctl exists
305 | if ! command -v "${PG_BIN_DIR}"/pg_ctl &> /dev/null; then
306 |     warnmsg "${PG_BIN_DIR}/pg_ctl command not be found. Make sure that the PG_BIN_DIR variable is set correctly."
307 |     exit
308 | fi
309 | 
310 | # check if a directory exists
311 | if [[ ! -d "${PGDATA}" ]]; then
312 |     if ! mkdir -p "${PGDATA}"; then
313 |         warnmsg "cannot create directory ${PGDATA}"
314 |         exit
315 |     fi
316 | fi
317 | 
318 | # Compare DB and filesystem size before restore
319 | # get the database size from the given backup-set or last backup
320 | if [[ -n $BACKUPSET ]]; then
321 |     DBSIZE=$(pgbackrest info --stanza="${FROM}" --output=json | jq -r '.[].backup[] |  select(.label == "${BACKUPSET}") | .info.size')
322 | fi
323 | if [[ -z $BACKUPSET ]]; then
324 |     DBSIZE=$(pgbackrest info --stanza="${FROM}" --output=json | jq -r '.[].backup[-1].info.size')
325 | fi
326 | # check the disk space
327 | DIRSIZE=$(df "$TO" | tail -1 | awk '{print $4}')
328 | if [[ $(( "$DIRSIZE * 1000" )) -le $DBSIZE && "$NORESTORE" != "yes" ]];
329 | then
330 |     error "Not enough disk space for restore to $TO"
331 | fi
332 | 
333 | # checkdb_mode
334 | if [[ -z $CHECKDB ]]; then
335 |     [[ "${DUMMYDUMP}" = "yes" ]] && CHECKDB_MODE+="dummy-dump "
336 |     [[ "${CHECKSUMS}" = "yes" ]] && CHECKDB_MODE+="checksums "
337 |     [[ "${AMCHECK}" = "yes" ]] && CHECKDB_MODE+="amcheck "
338 |     if [[ -z $DUMMYDUMP && -z $CHECKSUMS && -z $AMCHECK ]]; then CHECKDB_MODE="No"; fi
339 | else CHECKDB_MODE="yes"
340 | fi
341 | 
342 | # restore_type_msg
343 | if [[ -z $DATNAME && $RECOVERYTYPE = time ]]; then
344 |     restore_type_msg="Full PostgreSQL Restore with Point-in-Time"
345 | elif [[ -z $DATNAME ]]; then
346 |     restore_type_msg="Full PostgreSQL Restore"
347 | elif [[ -n $DATNAME && $RECOVERYTYPE = time ]]; then
348 |     restore_type_msg="Partial PostgreSQL Restore with Point-in-Time"
349 | elif [[ -n $DATNAME ]]; then
350 |     restore_type_msg="Partial PostgreSQL Restore"
351 | fi
352 | 
353 | # process-max default
354 | if [[ -z $PROCESS_MAX  ]]; then
355 |     PROCESS_MAX=1
356 | fi
357 | 
358 | function sigterm_handler(){
359 |     info "Recieved QUIT|TERM|INT signal"
360 |     error "Clean up and exit"
361 | }
362 | 
363 | trap sigterm_handler QUIT TERM INT
364 | 
365 | function check_errcode(){
366 |     # ARG: "error message"
367 |     [[ $? -ne 0 ]] && error "${1}"
368 | }
369 | 
370 | function check_mistake_run(){
371 |     if [[ -n "${PGDATA}" ]]; then
372 |         warnmsg "Restoring to ${TO} Waiting 30 seconds. The directory will be overwritten. If mistake, press ^C"
373 |         sleep 30s
374 |     fi
375 | }
376 | 
377 | function pg_port_pick(){
378 |     port_from=5432
379 |     port_to=6432
380 |     # allocate a free port from the port range
381 |     for (( port=$port_from; port<=$port_to; port++)); do
382 |         if ! "${PG_BIN_DIR}"/pg_isready -qp "$port"; then
383 |         PGPORT=$port
384 |         break
385 |         fi
386 |     done
387 |     # check if the port is allocated
388 |     if [[ -z $PGPORT ]]; then
389 |       error "pg_port_pick: Can't allocate Postgres port"
390 |     fi
391 | }
392 | 
393 | function cycle_simple(){
394 |     # ARG: command
395 |     # Assign variable 'status' = "ok" or "er"
396 |     status=
397 |     cmd=$1
398 |     attempt=1
399 |     limit=3600
400 |     while [[ $attempt -le $limit ]]; do
401 |         info "attempt: ${attempt}/${limit}"
402 |         $cmd
403 |         if [[ "$status" = "ok" ]]; then
404 |             # Ready to work
405 |             break
406 |         elif [[ "$status" = "er" ]]; then
407 |             error "exit"
408 |         fi
409 |         ((attempt++))
410 | 	sleep 1s
411 |     done
412 | 
413 |     [[ $attempt -ge $limit && $status != ok ]] && error "attempt limit exceeded"
414 | }
415 | 
416 | function pg_stop_check(){
417 |     sleep=10
418 |     # Use with function cycle_simple
419 |     info "PostgreSQL check status"
420 |     "${PG_BIN_DIR}"/pg_ctl status -D "${PGDATA}" &> /dev/null
421 |     code=$?
422 |     if [[ $code -eq 3 ]]; then
423 |         info "PostgreSQL instance not running"
424 |         status=ok
425 |     elif [[ $code -eq 0 ]]; then
426 |         info "Wait PostgreSQL instance stop: wait ${sleep}s"
427 |         sleep ${sleep}s
428 |     elif [[ $code -eq 4 ]]; then
429 |         status=ok
430 |     else
431 |         warnmsg "PostgreSQL check failed"
432 |         status=er
433 |     fi
434 | }
435 | 
436 | function pg_stop(){
437 |     "${PG_BIN_DIR}"/pg_ctl status -D "${PGDATA}" &> /dev/null
438 |     code=$?
439 |     if [[ $code -eq 0 ]]; then
440 |         info "PostgreSQL stop"
441 |         if "${PG_BIN_DIR}"/pg_ctl stop -D "${PGDATA}" -m fast -w -t 1800 &> /dev/null
442 |         then
443 |             info "PostgreSQL instance stopped"
444 |         else
445 |             warnmsg "PostgreSQL instance stop failed"
446 |         fi
447 |     fi
448 | }
449 | 
450 | function pgisready(){
451 |     if "${PG_BIN_DIR}"/pg_isready -qp "${PGPORT}"
452 |     then
453 |         info "PostgreSQL instance ${PGPORT} started and accepting connections"
454 |         status=ok
455 |         return 0
456 |     else
457 |         warnmsg "PostgreSQL instance ${PGPORT} no response"
458 |         return 1
459 |     fi
460 | }
461 | 
462 | function pg_start(){
463 |     pg_port_pick
464 |     "${PG_BIN_DIR}"/pg_ctl status -D "${PGDATA}" &> /dev/null
465 |     code=$?
466 |     if [[ $code -eq 0 ]]; then
467 |         info "PostgreSQL is already running"
468 |     else
469 |         info "Starting PostgreSQL on port ${PGPORT}"
470 |         if ! "${PG_BIN_DIR}"/pg_ctl -o "-p ${PGPORT}" start -D "${PGDATA}" -w -t 36000 \
471 |             > /tmp/pgbackrest_auto_pg_start_${FROM}.log 2>&1
472 |         then
473 |             error "PostgreSQL instance ${PGPORT} start failed"
474 |         else
475 |             pgisready 1> /dev/null
476 |         fi
477 |     fi
478 | }
479 | 
480 | function pgbackrest_exec(){
481 |     # config
482 |     if [[ -n $CONFIG ]]; then
483 |         pgbackrest_conf="${CONFIG}"
484 |     else
485 |         pgbackrest_conf=/tmp/pgbackrest.conf
486 |         if [ ! -f $pgbackrest_conf ]; then touch $pgbackrest_conf; fi
487 |     fi
488 |     # recovery_opt
489 |     if [[ $PGVER = 9.4 || $PGVER = 94 ]]; then
490 |         recovery_opt="pause_at_recovery_target=false"
491 |     else
492 |         recovery_opt="recovery_target_action=promote"
493 |     fi
494 |     # pgbackrest_opt
495 |     [[ -n "${BACKUPSET}" ]] && pgbackrest_opt="--set=${BACKUPSET}"
496 |     [[ -n "${DATNAME}" ]] && for db in ${DATNAME}; do pgbackrest_opt+=" --db-include=${db}"; done
497 |     if [[ "${RECOVERYTYPE}" = "default" || "${RECOVERYTYPE}" = "time" ]]; then
498 |         [[ -n "${RECOVERYTYPE}" ]] && pgbackrest_opt+=" --type=${RECOVERYTYPE}"
499 |         [[ -n "${RECOVERYTARGET}" ]] && pgbackrest_opt+=" --target=\"${RECOVERYTARGET}\""
500 |     else
501 |         pgbackrest_opt+=" --type=immediate"
502 |     fi
503 |     # tablespace-map-all
504 |     mkdir -p "${TO}"_remapped_tablespaces
505 |     # config file/custom options or set default options
506 |     if [[ $pgbackrest_conf = "/tmp/pgbackrest.conf" && -f /etc/pgbackrest.conf && -z "${CUSTOMOTIONS}" ]]; then
507 |         grep -q "repo1-path" /etc/pgbackrest.conf && pgbackrest_opt+=" --$(bash -c "grep \"repo1-path=\" /etc/pgbackrest.conf")"
508 |         pgbackrest_opt+=" --repo1-host=localhost --repo1-host-user=postgres"
509 |     fi
510 |     if [[ -n "${CUSTOMOTIONS}" ]]; then pgbackrest_opt+=" ${CUSTOMOTIONS}"; fi
511 |     # detail_rst_log
512 |     detail_rst_log="/var/log/pgbackrest/$FROM-restore.log"
513 |     if [ -f "${detail_rst_log}" ]; then info "See detailed log in the file ${detail_rst_log}"; fi
514 |     info "Restore from backup started. Type: $restore_type_msg"
515 |     # execute pgbackrest
516 |     echo "pgbackrest --config=${pgbackrest_conf} --stanza=${FROM} --pg1-path=${TO} ${pgbackrest_opt} --delta restore --process-max=${PROCESS_MAX} --log-level-console=error --log-level-file=detail --recovery-option=${recovery_opt} --tablespace-map-all=${TO}_remapped_tablespaces"
517 |     if bash -c "pgbackrest --config=${pgbackrest_conf} --stanza=${FROM} --pg1-path=${TO} ${pgbackrest_opt} --delta restore --process-max=${PROCESS_MAX} --log-level-console=error --log-level-file=detail --recovery-option=${recovery_opt} --tablespace-map-all=${TO}_remapped_tablespaces"
518 |     then
519 |         info "Restore from backup done"
520 |         sed -i 's/Restore_from_backup=0/Restore_from_backup=1/g' "${status_file}"
521 |     else
522 |         error "Restore from backup failed"
523 |     fi
524 | }
525 | 
526 | function pg_info_replay(){
527 |     if [[ "${RECOVERYTYPE}" = "time" ]]; then
528 |         info "RECOVERYTYPE time"
529 |         result=$(psql -p "${PGPORT}" -h 127.0.0.1 -tAXc "SELECT pg_last_xact_replay_timestamp(), '${RECOVERYTARGET}' - pg_last_xact_replay_timestamp()")
530 |     else
531 |         result=$(psql -p "${PGPORT}" -h 127.0.0.1 -tAXc "SELECT pg_last_xact_replay_timestamp()")
532 |     fi
533 |     while IFS='|' read -r replay_timestamp left_timestamp; do
534 |         if [[ -n "${left_timestamp}" ]]; then
535 |             info "Replayed: ${replay_timestamp} Left: ${left_timestamp}"
536 |         else
537 |             info "Replayed: ${replay_timestamp}"
538 |         fi
539 |     done <<< "${result}"
540 | }
541 | 
542 | function pg_check_recovery(){
543 |     state=$(psql -p "${PGPORT}" -h 127.0.0.1 -tAXc 'SELECT pg_is_in_recovery()') 2>/dev/null
544 |     pg_info_replay
545 |     # Is the restore complete? YES
546 |     if [ "$state" = "f" ]; then
547 |         recovery=ok
548 |     # Is the restore complete? No
549 |     elif [ "$state" = "t" ]; then
550 |         sleep 10
551 |     else
552 |     # Is everything all right? check connection with PostgreSQL
553 |         pgisready 1> /dev/null || check_errcode "exit"
554 |         recovery=er
555 |     fi
556 | }
557 | 
558 | # verify that data can be read out. Check with pg_dump >> /dev/null
559 | function dummy_dump(){
560 |     if ! pgisready 1> /dev/null; then pg_start cycle_simple pgisready; fi
561 |     sed -i 's/Data_validation=0/Data_validation=1/g' "${status_file}"
562 |     databases=$(bash -c "psql -p ${PGPORT} -h 127.0.0.1 -tAXc \"select datname from pg_database where not datistemplate\"")
563 |         for db in $databases; do
564 |             info "Start data validation for database $db"
565 |             if pgisready 1> /dev/null; then
566 |                 if [[ "$PROCESS_MAX" == '1' ]]; then
567 |                     # single-threaded dump is slower, but requires less disk I/O due to redirection to /dev/null
568 |                     info "  starting pg_dump -p ${PGPORT} -h 127.0.0.1 -d $db >> /dev/null"
569 |                     if ! "${PG_BIN_DIR}"/pg_dump -p "${PGPORT}" -h 127.0.0.1 -d "$db" >> /dev/null
570 |                     then
571 |                         sed -i 's/Data_validation=1/Data_validation=0/g' "${status_file}"
572 |                         error "Data validation in the database $db - Failed"
573 |                     else
574 |                         info "Data validation in the database $db - Successful"
575 |                     fi
576 |                 else
577 |                     # Parallel dump speeds up the process for large databases,
578 |                     # but requires writing data and more disk space for the dump.
579 |                     mkdir -p "${PGDATA}"/dump/
580 |                     rm -rf "${PGDATA}"/dump/"$db"
581 |                     info "  starting pg_dump -p ${PGPORT} -h 127.0.0.1 -d $db -F d -j ${PROCESS_MAX} -f ${PGDATA}/dump/$db"
582 |                     if ! "${PG_BIN_DIR}"/pg_dump -p "${PGPORT}" -h 127.0.0.1 -d "$db" -F d -j "${PROCESS_MAX}" -f "${PGDATA}"/dump/"$db"
583 |                     then
584 |                         sed -i 's/Data_validation=1/Data_validation=0/g' "${status_file}"
585 |                         error "Data validation in the database $db - Failed"
586 |                     else
587 |                         info "Data validation in the database $db - Successful"
588 |                     fi
589 |                 fi
590 |             fi
591 |         done
592 | }
593 | 
594 | # checksums - check data checksums
595 | function pg_checksums(){
596 |     local pg_checksums_command
597 |     local pg_checksums_result
598 | 
599 |     # Determine the checksums command based on PostgreSQL version
600 |     if [[ "$PGVER" -le "11" ]]; then
601 |         if command -v "${PG_BIN_DIR}"/pg_checksums &> /dev/null; then
602 |             pg_checksums_command="pg_checksums"
603 |         elif command -v "${PG_BIN_DIR}"/pg_verify_checksums &> /dev/null; then
604 |             pg_checksums_command="pg_verify_checksums"
605 |         else
606 |             warnmsg "Checksum command not found. Please install the postgresql-$PGVER-pg-checksums package."
607 |             exit 1
608 |         fi
609 |     else
610 |         pg_checksums_command="pg_checksums"
611 |     fi
612 | 
613 |     if pgisready 1> /dev/null; then pg_stop cycle_simple pg_stop_check; fi
614 |         info "pg_checksums: starting data checksums validation"
615 |         sed -i 's/PG_checksums_validation=0/PG_checksums_validation=1/g' "${status_file}"
616 |         if [ "$pg_checksums_command" == "pg_verify_checksums" ]; then
617 |             pg_checksums_result=$("${PG_BIN_DIR}"/pg_verify_checksums -D "${PGDATA}" | grep "Bad checksums")
618 |         else
619 |             pg_checksums_result=$("${PG_BIN_DIR}"/pg_checksums -c -D "${PGDATA}" | grep "Bad checksums")
620 |         fi
621 |         if [[ $pg_checksums_result != "Bad checksums:  0" ]]
622 |         then
623 |             warnmsg "pg_checksums: data checksums validation result: $pg_checksums_result"
624 |             sed -i 's/PG_checksums_validation=1/PG_checksums_validation=0/g' "${status_file}"
625 |             error "pg_checksums: data checksums validation - Failed"
626 |         else
627 |             info "pg_checksums: data checksums validation - Successful"
628 |     fi
629 | }
630 | 
631 | # amcheck CREATE EXTENSION if not exists
632 | function amcheck_exists(){
633 |     if [ "$PGVER" -le "10" ]; then
634 |         extension='amcheck_next'
635 |     else
636 |         extension='amcheck'
637 |     fi
638 |     if ! psql -v "ON_ERROR_STOP" -p "${PGPORT}" -h 127.0.0.1 -U postgres -d "$db" -tAXc "CREATE EXTENSION if not exists $extension" &> /dev/null
639 |     then
640 |         error "CREATE EXTENSION $extension failed"
641 |     fi
642 | }
643 | 
644 | # amcheck - verify the logical consistency of the structure of PostgreSQL B-Tree indexes
645 | function amcheck(){
646 |     if ! pgisready 1> /dev/null; then pg_start cycle_simple pgisready; fi
647 |     sed -i 's/Amcheck_validation=0/Amcheck_validation=1/g' "${status_file}"
648 |     databases=$(bash -c "psql -p ${PGPORT} -h 127.0.0.1 -tAXc \"select datname from pg_database where not datistemplate\"")
649 |     for db in $databases; do
650 |         if pgisready 1> /dev/null; then
651 |             if amcheck_exists; then
652 |                 info "amcheck: verify the logical consistency of the structure of indexes and heap relations in the database $db"
653 |                 if [[ "$PGVER" -lt '14' ]]; then
654 |                     # If the PostgreSQL version is less than 14, use the bt_index_parent_check function for each index in single-threaded mode.
655 |                     indexes=$(psql -p "${PGPORT}" -h 127.0.0.1 -d "$db" -tXAc "SELECT quote_ident(n.nspname)||'.'||quote_ident(c.relname) FROM pg_index i JOIN pg_opclass op ON i.indclass[0] = op.oid JOIN pg_am am ON op.opcmethod = am.oid JOIN pg_class c ON i.indexrelid = c.oid JOIN pg_namespace n ON c.relnamespace = n.oid WHERE am.amname = 'btree' AND n.nspname NOT IN ('pg_catalog', 'pg_toast') AND c.relpersistence != 't' AND c.relkind = 'i' AND i.indisready AND i.indisvalid")
656 |                     for index in $indexes; do
657 |                         if ! psql -v ON_ERROR_STOP=on -p "${PGPORT}" -h 127.0.0.1 -d "$db" -tAXc "select bt_index_parent_check('${index}', heapallindexed => true)" 1> /dev/null
658 |                         then
659 |                             sed -i 's/Amcheck_validation=1/Amcheck_validation=0/g' "${status_file}"
660 |                             error "amcheck: logical validation for index ${index} ( database $db ) - Failed"
661 |                         fi
662 |                     done
663 |                 else
664 |                     # Use pg_amcheck (added in PostgreSQL 14)
665 |                     if ! "${PG_BIN_DIR}"/pg_amcheck -p "${PGPORT}" -h 127.0.0.1 -d "$db" --parent-check --heapallindexed --progress --jobs="${PROCESS_MAX}"
666 |                     then
667 |                         sed -i 's/Amcheck_validation=1/Amcheck_validation=0/g' "${status_file}"
668 |                         error "amcheck: logical validation for database $db - Failed"
669 |                     fi
670 |                 fi
671 |             fi
672 |         fi
673 |     done
674 | }
675 | 
676 | function postgresql_conf(){
677 |     # create pg_hba.conf file if not exist
678 |     if [ ! -f "${PG_CONF_DIR}"/pg_hba.conf ]; then touch "${PG_CONF_DIR}"/pg_hba.conf; fi
679 |     # add an access rule for localhost only
680 |     echo "host all all 127.0.0.1/32 trust" > "${PG_CONF_DIR}"/pg_hba.conf
681 |     # create postgresql.conf file if not exist
682 |     if [ ! -f "${PG_CONF_DIR}"/postgresql.conf ]; then touch "${PG_CONF_DIR}"/postgresql.conf; fi
683 |     # listen_addresses
684 |     echo "listen_addresses = '127.0.0.1'" > "${PG_CONF_DIR}"/postgresql.conf
685 |     # determine postgresql parameters from pg_controldata and configure postgresql.conf
686 |     # max_connections
687 |     max_connections=$("${PG_BIN_DIR}"/pg_controldata "${PGDATA}" | grep max_connections | awk '{print $3}')
688 |     echo "max_connections = ${max_connections}" >> "${PG_CONF_DIR}"/postgresql.conf
689 |     # max_worker_processes
690 |     max_worker_processes=$("${PG_BIN_DIR}"/pg_controldata "${PGDATA}" | grep max_worker_processes | awk '{print $3}')
691 |     echo "max_worker_processes = ${max_worker_processes}" >> "${PG_CONF_DIR}"/postgresql.conf
692 |     # max_prepared_transactions
693 |     max_prepared_transactions=$("${PG_BIN_DIR}"/pg_controldata "${PGDATA}" | grep max_prepared_xacts | awk '{print $3}')
694 |     echo "max_prepared_transactions = ${max_prepared_transactions}" >> "${PG_CONF_DIR}"/postgresql.conf
695 |     # max_locks_per_transaction
696 |     max_locks_per_transaction=$("${PG_BIN_DIR}"/pg_controldata "${PGDATA}" | grep max_locks_per_xact | awk '{print $3}')
697 |     echo "max_locks_per_transaction = ${max_locks_per_transaction}" >> "${PG_CONF_DIR}"/postgresql.conf
698 | }
699 | 
700 | 
701 | ### MAIN ###
702 | STEP=1
703 | rm -f "${log}"
704 | touch "${log}"
705 | exec &> >(tee -a "${log}")
706 | info "[STEP $((STEP++))]: Starting"
707 | # Reset values in status file before new restore
708 | printf "Restore_from_backup=0\nRestoring_from_archive=0\nData_validation=0\nPG_checksums_validation=0\nAmcheck_validation=0\nResult_status=1" > "${status_file}"
709 | if [[ "$NORESTORE" = "yes" ]]; then
710 |     info "Starting. Skipping restore."
711 |     info "Starting. Run settings: Log: ${log}"
712 |     info "Starting. Run settings: Lock run: ${lock}"
713 |     info "Starting. PostgreSQL version: ${PGVER}"
714 |     info "Starting. PostgreSQL data directory: ${PGDATA}"
715 |     info "Starting. PostgreSQL Database Validation: ${CHECKDB_MODE}"
716 |     if [[ "${CLEAR}" = "yes" ]]; then info "Starting. Clear Data Directory after restore: ${CLEAR}";fi
717 |     info "[STEP $((STEP++))]: PostgreSQL Starting"
718 |     pg_start
719 |     cycle_simple pgisready
720 |     sed -i 's/Restore_from_backup=0/Restore_from_backup=1/g' "${status_file}"
721 |     sed -i 's/Restoring_from_archive=0/Restore_from_archive=1/g' "${status_file}"
722 | else
723 |     info "Starting. Restore Type: ${restore_type_msg} FROM Stanza: ${FROM} --> TO Directory: ${TO}"
724 |     info "Starting. Restore Settings: ${RECOVERYTYPE} ${RECOVERYTARGET} ${BACKUPSET} ${DATNAME}"
725 |     info "Starting. Run settings: Log: ${log}"
726 |     info "Starting. Run settings: Lock run: ${lock}"
727 |     info "Starting. PostgreSQL version: ${PGVER}"
728 |     info "Starting. PostgreSQL data directory: ${PGDATA}"
729 |     info "Starting. PostgreSQL Database Validation: ${CHECKDB_MODE}"
730 |     if [[ "${CLEAR}" = "yes" ]]; then info "Starting. Clear Data Directory after restore: ${CLEAR}";fi
731 |     check_mistake_run
732 |     info "[STEP $((STEP++))]: Stopping PostgreSQL"
733 |     pg_stop
734 |     cycle_simple pg_stop_check
735 |     info "[STEP $((STEP++))]: Restoring from backup"
736 |     # Restore from backup
737 |     pgbackrest_exec
738 |     # get postgresql parameters from pg_controldata
739 |     postgresql_conf
740 |     info "[STEP $((STEP++))]: PostgreSQL Starting for recovery"
741 |     pg_start
742 |     cycle_simple pgisready
743 |     info "[STEP $((STEP++))]: PostgreSQL Recovery Checking"
744 |     # Expect recovery result
745 |     while true; do
746 |         info "Checking if restoring from archive is done"
747 |         pg_check_recovery
748 |         if [[ "${recovery}" = "ok" ]]; then
749 |             info "Restoring from archive is done"
750 |             sed -i 's/Restoring_from_archive=0/Restore_from_archive=1/g' "${status_file}"
751 |             break
752 |         elif [[ "${recovery}" = "er" ]]; then
753 |             warnmsg "Restoring from archive failed"
754 |             pg_stop; check_errcode "exit"
755 |         fi
756 |     done
757 | fi
758 | if [[ "${CHECKDB_MODE}" != "No" ]]; then
759 |     # checksums - check data checksums
760 |     if [[ "${CHECKDB}" = "yes" || "${CHECKSUMS}" = "yes" ]]; then
761 |     info "[STEP $((STEP++))]: Verify data checksums"
762 |     pg_checksums
763 |     else
764 |     sed -i 's/PG_checksums_validation=0/PG_checksums_validation=1/g' "${status_file}"
765 |     fi
766 |     # verify that data can be read out. Check with pg_dump >> /dev/null
767 |     if [[ "${CHECKDB}" = "yes" || "${DUMMYDUMP}" = "yes" ]]; then
768 |     info "[STEP $((STEP++))]: Verify that data can be read out"
769 |     dummy_dump
770 |     else
771 |     sed -i 's/Data_validation=0/Data_validation=1/g' "${status_file}"
772 |     fi
773 |     # amcheck - verify the logical consistency of the structure of PostgreSQL B-Tree indexes
774 |     if [[ "${CHECKDB}" = "yes" || "${AMCHECK}" = "yes" ]]; then
775 |     info "[STEP $((STEP++))]: Verify indexes"
776 |     amcheck
777 |     else
778 |     sed -i 's/Amcheck_validation=0/Amcheck_validation=1/g' "${status_file}"
779 |     fi
780 | else
781 |     sed -i 's/PG_checksums_validation=0/PG_checksums_validation=1/g' "${status_file}"
782 |     sed -i 's/Data_validation=0/Data_validation=1/g' "${status_file}"
783 |     sed -i 's/Amcheck_validation=0/Amcheck_validation=1/g' "${status_file}"
784 | fi
785 | # [ optional ] clear data directory
786 | if [[ "${CLEAR}" = "yes" ]]; then
787 |     info "[STEP $((STEP++))]: Stopping PostgreSQL and Clear Data Directory"
788 |     pg_stop
789 |     cycle_simple pg_stop_check
790 |     if [[ $code -eq 3 ]]; then
791 |         rm -rf "${TO}"
792 |         rm -rf "${TO}"_remapped_tablespaces
793 |     fi
794 | fi
795 | if [[ "${REPORT}" = "yes" ]]; then
796 |     info "[STEP $((STEP++))]: Send report to mail address"
797 |     sendmail
798 | fi
799 | 
800 | #set result status of restore for zabbix
801 | if grep -q 0 "${status_file}"; then
802 |     sed -i 's/Result_status=1/Result_status=0/g' "${status_file}"
803 | fi
804 | 
805 | # remove lock file
806 | if [ -f "${lock}" ]; then
807 |     rm "${lock}"
808 | fi
809 | info "Finish"
810 | 
811 | exit
812 | 


--------------------------------------------------------------------------------