├── README.md
├── elasticsearch-backup-jobs.sh
├── elasticsearch-snapshot-rotation.sh
├── elasticsearch-take-snapshot.sh
└── shell-variables


/README.md:
--------------------------------------------------------------------------------
  1 | Elasticsearch Backup Shell Script
  2 | ==============================================
  3 | 
  4 | About
  5 | -------------
  6 | 
  7 | This scripts
  8 | 
  9 | Original script are provided by @karelbemelmans
 10 | 
 11 | - [Elasticsearch backup script with snapshot rotation · Karel Bemelmans](https://www.karelbemelmans.com/2015/03/elasticsearch-backup-script-with-snapshot-rotation/)
 12 | 
 13 | Our customized script replace URL to shell variable and split file with function.
 14 | 
 15 | Configuration
 16 | ------------------------
 17 | 
 18 | ### Shell variable
 19 | 
 20 | edit `shell-variables`
 21 | 
 22 | - `URL`: Elasticsearch base url
 23 | - `REPO`: Snapshot repository name
 24 | - `LIMIT`: Number of snapshots to keep
 25 | 
 26 | Requirements
 27 | ------------------------
 28 | 
 29 | 1. Elasticsearch 1.4, 2, 5 maybe
 30 | 1. curl
 31 | 1. jq JSON Parser command
 32 | 
 33 | Usage
 34 | -----------------
 35 | 
 36 | clone scripts your preferred directory. In this case we assume cloned `/etc/bacula/pre-backup.d`.
 37 | 
 38 | ```
 39 | git clone https://github.com/UnicastInc/elasticsearch-shell-backup.git /etc/bacula/pre-backup.d
 40 | ```
 41 | 
 42 | call simply.
 43 | 
 44 | ```
 45 | elasticsearch-backup-jobs.sh
 46 | ```
 47 | 
 48 | This script simply execute `elasticsearch-take-snapshot.sh`, `elasticsearch-snapshot-rotation.sh` internally.
 49 | 
 50 | Schedule regular intervals
 51 | ---------------------------------------
 52 | 
 53 | ### cron
 54 | 
 55 | Most easiest way to backup.
 56 | 
 57 | #### crontab
 58 | 
 59 | ```
 60 | 45 23 * * *    /etc/bacula/pre-backup.d/elasticsearch-backup-jobs.sh
 61 | ```
 62 | 
 63 | or drop into `/etc/cron.{daily,weekly,monthly}` ...
 64 | 
 65 | ### Bacula
 66 | 
 67 | Specify in `bacula-dir.conf`.
 68 | 
 69 | ```
 70 | Job {
 71 |   ClientRunBeforeJob = "/etc/bacula/pre-backup.d/elasticsearch-backup-jobs.sh"
 72 | }
 73 | ```
 74 | 
 75 | Bacula File Daemon side, create `/etc/bacula/pre-backup.d/elasticsearch-backup-jobs.sh`
 76 | 
 77 | ```bash
 78 | #!/bin/bash
 79 | 
 80 | # Elasticsearch snapshot lifecycle
 81 | /etc/bacula/pre-backup.d/elasticsearch-backup-jobs.sh
 82 | ```
 83 | 
 84 | ### Another
 85 | 
 86 | - Jenkins, GitLab CI or another CIs.
 87 | 
 88 | For Test
 89 | ---------------
 90 | 
 91 | ### Take a snapshot
 92 | 
 93 | ```
 94 | elasticsearch-take-snapshot.sh
 95 | ```
 96 | 
 97 | ### Rotate snapshots
 98 | 
 99 | ```
100 | elasticsearch-snapshot-rotation.sh
101 | ```
102 | 
103 | Hardening Security
104 | ------------------------------------------
105 | 
106 | ```bash
107 | chmod 700 /etc/bacula/pre-backup.d
108 | chmod 600 shell-variables
109 | ```
110 | 
111 | Tool commands
112 | ------------------------
113 | 
114 | ### Get snapshot list
115 | 
116 | ```
117 | curl -X GET http://localhost:9200/_snapshot/es_backup/_all 2>/dev/null | jq -r '.snapshots [] .snapshot'
118 | ```
119 | 
120 | 


--------------------------------------------------------------------------------
/elasticsearch-backup-jobs.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | base_dir="$(dirname "$0")"
4 | source $base_dir/shell-variables
5 | 
6 | $base_dir/elasticsearch-take-snapshot.sh
7 | $base_dir/elasticsearch-snapshot-rotation.sh
8 | 
9 | 


--------------------------------------------------------------------------------
/elasticsearch-snapshot-rotation.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | #
 3 | # Clean up script for old elasticsearch snapshots.
 4 | # 23/2/2014 karel@narfum.eu
 5 | #
 6 | # You need the jq binary:
 7 | # - yum install jq
 8 | # - apt-get install jq
 9 | # - or download from http://stedolan.github.io/jq/
10 | 
11 | base_dir="$(dirname "$0")"
12 | source $base_dir/shell-variables
13 | 
14 | # Get a list of snapshots that we want to delete
15 | echo "curl -s -XGET \"$URL/_snapshot/$REPO/_all\" | jq -r \".snapshots[:-${LIMIT}][].snapshot\""
16 | SNAPSHOTS=`curl -s -XGET "$URL/_snapshot/$REPO/_all" | jq -r ".snapshots[:-${LIMIT}][].snapshot"`
17 | 
18 | echo Snapshot List:
19 | echo $SNAPSHOTS
20 | 
21 | # Loop over the results and delete each snapshot
22 | for SNAPSHOT in $SNAPSHOTS
23 | do
24 |   echo "Deleting snapshot: $SNAPSHOT"
25 |   curl -s -XDELETE "$URL/_snapshot/$REPO/$SNAPSHOT?pretty" | jq '.'
26 |   if [ "$?" != "0" ]; then
27 |     echo Could not delete $SNAPSHOT.
28 |   fi
29 | done
30 | echo "Done!"
31 | 
32 | 


--------------------------------------------------------------------------------
/elasticsearch-take-snapshot.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | base_dir="$(dirname "$0")"
4 | source $base_dir/shell-variables
5 | 
6 | curl -XPUT "$URL/_snapshot/$REPO/$SNAPSHOT?wait_for_completion=true" | jq '.'
7 | 
8 | 


--------------------------------------------------------------------------------
/shell-variables:
--------------------------------------------------------------------------------
 1 | #!/bin/bash
 2 | 
 3 | # Elasticsearch endpoint URL
 4 | URL=http://localhost:9200
 5 | # Elasticsearch snapshot repository name
 6 | REPO=es_backup
 7 | # Number of snapshots to keep
 8 | LIMIT=30
 9 | # For test
10 | #LIMIT=3
11 | # Snapshot naming convention
12 | SNAPSHOT=`date +%Y%m%d-%H%M%S`
13 | 
14 | 


--------------------------------------------------------------------------------