├── README.md ├── Vagrant.md ├── tutorial ├── exercises.odt ├── exercises.pdf ├── exercises.txt ├── interactives.txt ├── pgReplicationTutorial.odp └── pgReplicationTutorial.pdf └── vagrant ├── Dockerfile ├── Vagrantfile ├── bootstrap.sh └── setup ├── dbprep.sh ├── package_install.sh ├── pgbench ├── #transfer1.bench# ├── balancecheck1.bench ├── balancecheck2.bench ├── balancecheck3.bench ├── balancecheck4.bench ├── branchreport.bench ├── deposit.bench ├── hang.bench ├── runbench.sh ├── runbench_replica1.sh ├── runbench_replica1.sh~ ├── transfer1.bench └── withdrawal.bench └── postgres ├── .pgpass ├── archive_logs.sh ├── earth.dump ├── fouine.html ├── libdata.dump ├── libdata.users.sql ├── master ├── pg_hba.conf └── postgresql.conf ├── pgbouncer.default ├── pgbouncer.ini ├── replica1 ├── pg_hba.conf ├── postgresql.conf ├── recovery.conf ├── recovery.conf.replica1.archiving ├── recovery.conf.replica1.dual └── recovery.conf.replica1.slot ├── replica2 ├── pg_hba.conf ├── postgresql.conf └── recovery.conf └── userlist.txt /README.md: -------------------------------------------------------------------------------- 1 | pgReplicationTutorial 2 | ===================== 3 | 4 | This repository contains files for the PostgreSQL Binary 5 | Replication tutorial. 6 | 7 | These files are required to perform the hands-on exercises. 8 | Importantly, there is significant setup required in order to 9 | do the hands-on exercises which needs to happen _before_ the 10 | tutorial starts, so please read the below. 11 | 12 | **All steps through Vagrant Up / Docker Pull should be completed before 13 | getting to the tutorial. The tutorial will not include any help on getting these 14 | systems running.** 15 | 16 | Requirements 17 | ============= 18 | 19 | Laptop Harware: 20 | 21 | * running Linux, OSX, or Windows XP or later 22 | * 64-bit (32-bit may be possible, see below) 23 | * at least 1GB of RAM, 2GB or more preferred 24 | * 300MB free disk space 25 | 26 | Software and Wetware: 27 | 28 | * terminal program capable of ssh 29 | * familiarity with the bash/linux command line 30 | * familiarity with one or more command-line text editors 31 | * vagrant and virtualbox (see below) OR Docker 1.5.0 or later 32 | 33 | Vagrant vs. Docker 34 | ================== 35 | 36 | Docker is the recommended platform for running the execises, because it 37 | is much easier to set up, and requires far less system resources to run 38 | when you're taking the tutorial. However, using Docker does require: 39 | 40 | * A Linux laptop, with Linux kernel 3.8 or later 41 | * Package lxc-docker installed running version 1.5 or later 42 | 43 | OR: 44 | 45 | * A windows or Mac laptop running the Docker Toolbox or Kitematic 46 | 47 | [Docker Toolbox](https://www.docker.com/toolbox) is available from Docker. 48 | We are not able to provide detailed instructions or assistance on configuring 49 | Docker Toolbox on Windows or Mac. The Toolbox page, however, has an excellent 50 | tutorial. 51 | 52 | If you are unable to get Docker working, or are simply more comfortable 53 | with Vagrant, then please see the Vagrant instructions in Vagrant.md after 54 | you have installed the tutorial exercises. 55 | 56 | Installing Tutorial Exercises 57 | ============================= 58 | 59 | Install the tutorial exercises on your machine one of two 60 | ways below. Installing them is **required** if you are using 61 | Vagrant, and is optional if you are using Docker, but helpful. 62 | 63 | **Preferred Method**: Git Checkout from the Github repo. The 64 | repository is here: https://github.com/jberkus/pgReplicationTutorial, 65 | and you can clone it by: 66 | 67 | git clone https://github.com/jberkus/pgReplicationTutorial.git 68 | 69 | **Alternate Method**: if you're not comfortable with git, download 70 | the tarball from: 71 | 72 | https://dl.dropboxusercontent.com/u/5132935/pgReplicationTutorial.tgz 73 | 74 | This will require the programs "tar" and "gzip" to expand, as follows: 75 | 76 | tar -p -xvf pgReplicationTutorial.tgz 77 | 78 | There is also a "zip" formatted archive which should be more friendly for 79 | Windows users. However, this zip file **will not work for Vagrant execution**, 80 | so it is only suitable for Docker users. You can obtain this zip file from github: 81 | 82 | https://github.com/jberkus/pgReplicationTutorial/archive/master.zip 83 | 84 | The ReplicationTutorial directory should be placed somewhere 85 | you have disk space available. 86 | 87 | Docker Installation 88 | =================== 89 | 90 | The Docker image for this tutorial is 64-bit and was built with Docker 1.7.1. 91 | If you are on a 32-bit machine, or if you are limited to using Docker 1.4 or older, 92 | I recommend that you scroll back up and follow the Vagrant install instructions 93 | instead. 94 | 95 | If you're working on Windows or OSX, you'll need to run commands using the 96 | Docker Terminal which is a part of the Docker Toolbox. 97 | 98 | First pull the image: 99 | 100 | docker pull jberkus/pgreplicationtutorial 101 | 102 | While a pull is sufficient, I suggest that you test the image by doing: 103 | 104 | docker run -it --rm jberkus/pgreplicationtutorial 105 | 106 | This should log you into a command prompt as root. If you type "exit" the container 107 | will halt and erase itself. 108 | 109 | Other Files In This Package 110 | =========================== 111 | 112 | The pgReplicationTutorial package also contains the following files 113 | in the Tutorial directory: 114 | 115 | exercises.txt 116 | ------------- 117 | 118 | This is a set of commands which you will be running during the hands-on 119 | exercises for the tutorial. These are provided in a text file so that 120 | you can copy & paste, instead of trying to read every letter and dash 121 | from the instructor's screen. 122 | 123 | pgReplicationTutorial.odt/pdf 124 | ----------------------------- 125 | 126 | These are copies of the slides for the tutorial. 127 | 128 | ChangeLog 129 | ========= 130 | 131 | **0.6** Updated for Postgres Open 2015. PostGIS removed to shrink image. Docker instructions provided 132 | as primary recommended option. 133 | 134 | **0.5.1** multiple fixes to Docker image. 135 | 136 | **0.5** Added PostgreSQL 9.4 replication slots to the tutorial as exercise. Added some additional tuning and configuration portions. 137 | 138 | **0.4** Version produced for LinuxConf.AU 2015. Removes PostGIS exercises. Upgrades to PostgreSQL 9.4. Adds Docker support. 139 | 140 | **0.3** Version produced for FOSS4G 2014. Includes information about replicating PostGIS. Interactive "play" demonstrations have have been removed due to their time-consuming nature. 141 | 142 | **0.2** Initial version produced for DjangoCon 2013. 143 | 144 | License 145 | ======= 146 | 147 | The pgReplicationTutorial is Copyright 2013-2015 Josh Berkus 148 | and PostgreSQL Experts Inc. 149 | 150 | All slides, text, instructions and similar content in this tutorial are 151 | licensed [Creative Commons Attribution-ShareAlike 3.0] 152 | (http://creativecommons.org/licenses/by-sa/3.0/us/) 153 | 154 | Code exercises, virtual machines, containers and sample databases 155 | are licensed under the 156 | [Gnu Public License Version 2] 157 | (http://www.gnu.org/licenses/gpl-2.0.html) 158 | 159 | All other rights reserved. 160 | -------------------------------------------------------------------------------- /Vagrant.md: -------------------------------------------------------------------------------- 1 | Vagrant Instructions 2 | ==================== 3 | 4 | These are installation instructions for participants who are, for whatever reason, 5 | unable to use Docker. First, install the Tutorial Exercises from the main instructions. 6 | 7 | Alternative: Installing the Base Software: Vagrant 8 | ================================================== 9 | 10 | First, you will need to install Vagrant and VirtualBox if you do not 11 | already have them. If you do already have them, please make sure that 12 | you have at least these minimum versions: 13 | 14 | * Vagrant 1.4.1 or later 15 | * VirtualBox 4.2 or later 16 | 17 | Otherwise, you're going to need to install them. Fortunately, both have 18 | downloads for most OSes and extensive documentation. And when you get 19 | used to them, you'll find Vagrant an invaluable tool for testing. 20 | 21 | Vagrant Installation: 22 | 23 | * https://docs.vagrantup.com/v2/installation/ 24 | * https://docs.vagrantup.com/v2/getting-started/ 25 | 26 | VirtualBox Installation: 27 | 28 | * https://www.virtualbox.org/wiki/Downloads 29 | * https://www.virtualbox.org/wiki/End-user_documentation 30 | 31 | Installing the Virtual Machine Image: Vagrant 32 | ============================================= 33 | 34 | Once you've installed Vagrant and VirtualBox, you'll need to get an 35 | operating system image, or "box". Please download one of the following 36 | large files: 37 | 38 | If you have a 64-bit laptop: http://files.vagrantup.com/precise64.box 39 | 40 | If you have a 32-bit laptop: http://files.vagrantup.com/precise32.box 41 | 42 | Then run the following command from the folder where you downloaded it: 43 | 44 | * 64-bit: vagrant box add precise precise64.box 45 | * 32-bit: vagrant box add precise precise32.box 46 | 47 | Vagrant Up 48 | ========== 49 | 50 | The first time you do vagrant up, it will require an internet connection 51 | with significant bandwidth and around 1/2 hour. As such, **you should do 52 | it at home, before you get to the conference or the tutorial.** 53 | 54 | Open your terminal program. Navigate to pgReplicationTutorial/vagrant 55 | directory. Type the following: 56 | 57 | vagrant up 58 | 59 | This will launche the precise box (VM), install a bunch of software on 60 | it, and start it up. You will see some brief errors during the startup, many of them in relation to not having a tty. 61 | Only errors which involve more than 4 lines of errors in a row are significant; others can be ignored. 62 | Verify that you can log into it with: 63 | 64 | vagrant ssh 65 | 66 | Now log out with "exit". Shut down the VM, but leave it set up in preparation 67 | for the tutorial: 68 | 69 | vagrant suspend -------------------------------------------------------------------------------- /tutorial/exercises.odt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/tutorial/exercises.odt -------------------------------------------------------------------------------- /tutorial/exercises.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/tutorial/exercises.pdf -------------------------------------------------------------------------------- /tutorial/exercises.txt: -------------------------------------------------------------------------------- 1 | Exercises for PostgreSQL Replication Tutorial 2 | ============================================= 3 | 4 | $EDITOR below is the editor of your choice. Options include: 5 | 6 | * vi 7 | * nano (recommended if you're unfamiliar with Linux editors) 8 | * jmacs (emacs clone) 9 | * joe 10 | 11 | Lines in (parentheses) are comments and shouldn't be pasted/typed in. 12 | 13 | The ^ carat symbol is short for the [Ctrl] key. Thus ^c is "hold down [Ctrl], press [c]". 14 | 15 | If using Vagrant: Start Vagrant 16 | ------------------------------- 17 | 18 | vagrant up 19 | vagrant ssh 20 | sudo su - 21 | 22 | If using Docker: Start Docker and Postgres 23 | ------------------------------------------ 24 | 25 | docker run -it jberkus/pgreplicationtutorial 26 | su - postgres -c "/usr/bin/pg_ctl -w -D /var/lib/postgresql/9.4/master start" 27 | 28 | tmux 29 | ---- 30 | 31 | tmux 32 | ^b c 33 | su - postgres 34 | ^b n 35 | ^b n 36 | (you should be in "postgres" shell) 37 | 38 | optional: tab-completion exercise 39 | 40 | basic 2-node replication 41 | ------------------------ 42 | 43 | cd 9.4 44 | ls 45 | pg_basebackup --help | less 46 | pg_basebackup -x -P -D replica1 47 | cp /setup/postgres/replica1/* replica1/ 48 | pg_ctl -D replica1 start 49 | ps -aux | grep postgres 50 | psql -p 5432 libdata 51 | \dt 52 | \q 53 | psql -p 5433 libdata 54 | \dt 55 | \q 56 | createdb -p 5432 -U bench bench 57 | pgbench -i -s 10 -U bench bench 58 | psql -p 5432 bench 59 | \dt 60 | \q 61 | psql -p 5433 bench 62 | \dt 63 | delete from pgbench_history; 64 | \q 65 | 66 | Replication configuration 67 | ------------------------- 68 | 69 | $EDITOR master/postgresql.conf 70 | $EDITOR replica1/postgresql.conf 71 | $EDITOR replica1/recovery.conf 72 | 73 | (here we are just looking at the contents of these files, not modifying them) 74 | 75 | Administering replication 76 | ------------------------- 77 | 78 | ^b n 79 | (root shell) 80 | cd /setup/pgbench 81 | ./runbench.sh 82 | ^b n 83 | psql -p 5432 bench 84 | select count(*) from pgbench_history; 85 | (repeat above several times) 86 | \x 87 | select * from pg_stat_activity; 88 | select * from pg_stat_replication; 89 | select pg_xlog_location_diff(write_location,replay_location) from pg_stat_replication; 90 | (repeat above 2-4 times) 91 | \q 92 | psql -p 5433 bench 93 | select count(*) from pgbench_history; 94 | select pg_is_in_recovery(); 95 | select pg_last_xlog_receive_location(); 96 | select pg_last_xlog_receive_location(), pg_last_xlog_replay_location(); 97 | select pg_xlog_location_diff(pg_last_xlog_receive_location(), pg_last_xlog_replay_location()); 98 | select pg_last_xact_replay_timestamp(); 99 | select now() - pg_last_xact_replay_timestamp(); 100 | select pg_xlog_replay_pause(); 101 | select pg_xlog_location_diff(pg_last_xlog_receive_location(), pg_last_xlog_replay_location()); 102 | (repeat above twice) 103 | select now() - pg_last_xact_replay_timestamp(); 104 | select pg_xlog_replay_resume(); 105 | \q 106 | 107 | Replication and Security 108 | ------------------------ 109 | 110 | psql -p 5432 111 | create role replicator password 'replicate' login replication; 112 | \du 113 | \q 114 | $EDITOR master/pg_hba.conf 115 | (comment out postgres - replication lines) 116 | $EDITOR replica1/recovery.conf 117 | (change primary_conninfo to use replicator user) 118 | cp /setup/postgres/.pgpass ~/ 119 | chmod 700 ~/.pgpass 120 | $EDITOR ~/.pgpass 121 | (check password info for replicator user) 122 | pg_ctl -D master reload 123 | pg_ctl -D replica1 restart 124 | ps -aux | grep replicator 125 | psql -p 5432 126 | select * from pg_stat_replication; 127 | \q 128 | 129 | Cloning and Archiving 130 | --------------------- 131 | 132 | ^b n 133 | ^c 134 | #(pgbench killed) 135 | ^b n 136 | pg_ctl -D replica1 stop 137 | rm -rf replica1/* 138 | $EDITOR master/postgresql.conf 139 | (edit to enable archiving by uncommenting the "archive" lines) 140 | pg_ctl -D master restart 141 | ps aux | grep archiver 142 | ^b n 143 | ./runbench.sh 144 | ^b n 145 | psql -p 5432 146 | select pg_start_backup('replica1'); 147 | select pg_is_in_backup(); 148 | \q 149 | ls ~/wal_archive/ 150 | rsync -av --exclude="pg_xlog/*" --exclude="postmaster.pid" master/* replica1/ 151 | psql -p 5432 152 | select pg_stop_backup(); 153 | \q 154 | cp /setup/postgres/replica1/* replica1/ 155 | cp replica1/recovery.conf.replica1.archiving replica1/recovery.conf 156 | $EDITOR replica1/recovery.conf 157 | pg_ctl -D replica1 start 158 | ps aux | grep startup 159 | psql -p 5432 160 | create table test(test text); 161 | select * from pg_stat_replication; 162 | \q 163 | tail -f /var/log/postgresql/postgresql-replica1 164 | psql -p 5433 165 | \dt 166 | (repeat above until "test" table appears) 167 | \c bench 168 | select count(*) from pgbench_history; 169 | (repeat above several times until result changes) 170 | \q 171 | 172 | Dual Replication 173 | ---------------- 174 | 175 | cp replica1/recovery.conf.replica1.dual replica1/recovery.conf 176 | $EDITOR replica1/recovery.conf 177 | ^b n 178 | pg_ctl -D replica1 restart 179 | tail -f /var/log/postgresql/postgresql-replica1 180 | ^c 181 | psql -p 5432 182 | select * from pg_stat_replication; 183 | \q 184 | 185 | 186 | Failover and Failback 187 | --------------------- 188 | 189 | ^b n 190 | (check if runbench is still running, if not run the below) 191 | ./runbench.sh 192 | ^b n 193 | pg_ctl -D master -m immediate stop 194 | ^b n 195 | ^b n 196 | pg_ctl -D replica1 promote 197 | ls replica1/ 198 | psql -p 5433 199 | select pg_is_in_recovery(); 200 | create table test2(test text); 201 | \q 202 | ^b n 203 | ./runbench_replica1.sh 204 | ^b n 205 | rm -rf master/* 206 | pg_basebackup -X stream -P -p 5433 -U replicator -D master 207 | cp /setup/postgres/master/* master/ 208 | cp replica1/recovery.done master/recovery.conf 209 | $EDITOR master/recovery.conf 210 | (edit recovery.conf to connect to replica1) 211 | pg_ctl -D master start 212 | tail -f /var/log/postgresql/postgresql-master 213 | ^c 214 | psql -p 5433 215 | select * from pg_stat_replication; 216 | \q 217 | pg_ctl -D replica1 -m fast stop 218 | touch master/PROMOTE 219 | (wait) 220 | psql -p 5432 221 | select pg_is_in_recovery(); 222 | \q 223 | mv replica1/recovery.done replica1/recovery.conf 224 | pg_ctl -D replica1 start 225 | psql -p 5432 226 | select * from pg_stat_replication; 227 | \q 228 | 229 | Query Lag 230 | --------- 231 | 232 | $EDITOR replica1/postgresql.conf 233 | (look at streaming_delay settings) 234 | psql -p 5433 libdata 235 | begin; 236 | set transaction_isolation = 'repeatable read'; 237 | select * from copies; 238 | ^b n 239 | psql -U postgres -p 5432 libdata 240 | vacuum full copies; 241 | select * from pg_stat_replication; 242 | select pg_xlog_location_diff(write_location, replay_location) from pg_stat_replication; 243 | ^b n 244 | select * from copies; 245 | ( wait 10-20 sec. ) 246 | select * from copies; 247 | ( repeat above until error ) 248 | \q 249 | ^b n 250 | \q 251 | ^b n (postgres shell) 252 | 253 | Synchronous Replication 254 | ----------------------- 255 | 256 | $EDITOR master/postgresql.conf 257 | (add "replica1" to list of synchronous_standby_names 258 | $EDITOR replica1/postgresql.conf 259 | (turn on hot_standby_feedback) 260 | pg_ctl -D master reload 261 | pg_ctl -D replica1 reload 262 | ^b n 263 | runbench.sh 264 | ^b n 265 | psql -p 5432 266 | select * from pg_stat_replication; 267 | \timing 268 | begin; 269 | insert into test values ('test1'); 270 | commit; 271 | begin; 272 | set synchronous_commit='local'; 273 | insert into test values ('test2'); 274 | commit; 275 | \q 276 | pg_ctl -D replica1 stop 277 | psql -p 5432 278 | begin; 279 | insert into test values ('test3'); 280 | commit; 281 | ^c 282 | select * from test; 283 | begin; 284 | set synchronous_commit='local'; 285 | insert into test values ('test4'); 286 | commit; 287 | \q 288 | $EDITOR master/postgresql.conf 289 | (remove synchronous replica) 290 | pg_ctl -D master reload 291 | pg_ctl -D replica1 start 292 | 293 | Cascading Replication 294 | --------------------- 295 | 296 | pg_basebackup -p 5433 -x -P -U replicator -D replica2 297 | cp /setup/postgres/replica2/* replica2/ 298 | pg_ctl -D replica2 start 299 | psql -p 5432 300 | select * from pg_stat_replication; 301 | \q 302 | psql -p 5433 303 | select * from pg_stat_replication; 304 | \q 305 | psql -p 5434 306 | select pg_is_in_recovery(); 307 | \q 308 | pg_ctl -D replica1 promote 309 | psql -p 5432 310 | select * from pg_stat_replication; 311 | \q 312 | psql -p 5433 313 | select * from pg_stat_replication; 314 | \q 315 | pg_ctl -D master stop 316 | 317 | pgbouncer LB/Failover 318 | --------------------- 319 | ^b n 320 | service pgbouncer start 321 | $EDITOR /etc/pgbouncer/pgbouncer.ini 322 | ^b n 323 | psql -p 6432 bench 324 | show port; 325 | create table branches2 as select * from pgbench_branches; 326 | \q 327 | psql -p 6432 bench_ro 328 | show port; 329 | drop table branches2; 330 | \q 331 | pg_ctl -D replica1 -m fast stop 332 | pg_ctl -D replica2 promote 333 | psql -p 6432 bench 334 | ^b n 335 | $EDITOR /etc/pgbouncer/pgbouncer.ini 336 | (edit the rw connections so that the point to replica2) 337 | service pgbouncer restart 338 | ^b n 339 | psql -p 6432 bench 340 | show port; 341 | drop table branches2; 342 | \q 343 | 344 | replication slots 345 | ----------------- 346 | 347 | pg_ctl -D replica2 -m fast stop 348 | pg_ctl -D master start 349 | rm -rf replica1/* 350 | pg_basebackup -x -P -D replica1 351 | cp /setup/postgres/replica1/* replica1/ 352 | mv replica1/recovery.conf.replica1.slot replica1/recovery.conf 353 | psql -p 5432 354 | SELECT * FROM pg_create_physical_replication_slot('replica1'); 355 | \q 356 | less replica1/recovery.conf 357 | pg_ctl -D replica1 start 358 | ^b n 359 | cd /setup/postgres/pgbench 360 | ./runbench.sh 361 | ^b n 362 | psql -p 5432 363 | select * from pg_replication_slots; 364 | \q 365 | pg_ctl -D replica1 stop 366 | psql -p 5432 367 | select * from pg_replication_slots; 368 | # repeat above 3 times 369 | select pg_xlog_location_diff(pg_current_xlog_location(), restart_lsn) from pg_replication_slots; 370 | \q 371 | pg_ctl -D replica1 start 372 | psql -p 5432 373 | select * from pg_replication_slots; 374 | # repeat above 3 times 375 | select pg_xlog_location_diff(pg_current_xlog_location(), restart_lsn) from pg_replication_slots; 376 | 377 | \q 378 | 379 | 380 | Exit 381 | ---- 382 | 383 | ^b d 384 | exit 385 | exit 386 | vagrant halt (or destroy) -------------------------------------------------------------------------------- /tutorial/interactives.txt: -------------------------------------------------------------------------------- 1 | Interactives 2 | ============ 3 | 4 | Streaming Async Replication 5 | --------------------------- 6 | 7 | 3 vols 8 | 9 | 1: you are the master 10 | 2: you are pg_basebackup 11 | 3: clone 12 | 13 | These cards represent blocks of data. 14 | 15 | Stand the master in front 16 | Ok, pgbasebackup, you're going to clone the master 17 | Now now clone the master 18 | Ok, you are the replica 19 | Replica, signal the master: 20 | "Hey, I'm ready for replication" 21 | That's the WalReceiver 22 | Master, send a replication connection to the replica. 23 | Ok, now start passing data blocks 24 | Replica, drop the connection. 25 | Replica, pick the connection back up. 26 | 27 | Archiving Replication 28 | --------------------- 29 | 30 | 3 vols: 31 | 32 | 1 master 33 | 2 clone 34 | 3 writer client 35 | 4 admin 36 | 37 | Stand the master in front 38 | 39 | admin, your goal is to clone the master and establish a replication connection 40 | as fast as you can. 41 | writer, your job is to pass data writes (cards) to the master as fast as you can once cloning starts. 42 | master, I want you to drop data pages on the floor if the replica isn't ready 43 | clone, don't do anything to help. 44 | 45 | so, second time: 46 | 47 | same as the first: only master, you put data pages on the table once cloning starts. 48 | 49 | failover and failback 50 | --------------------- 51 | 52 | 1 master 53 | 2 replica 54 | 3 writer 55 | 56 | writer, pass data writes ever couple seconds to the master 57 | master, pass them to the replica 58 | now, master, you're down (sit down). don't hand over that card. 59 | replica, you're the new master. you're promoted! congrats! 60 | since you're coming up, you'll start a new *timeline* 61 | please start using this suite. 62 | writer, start passing cards to the new master. 63 | old master, you're back up. 64 | but now you're a replica 65 | let's start replication the other way. 66 | ooops! what about that card you're holding? 67 | data corruption! danger will robinson, danger! 68 | this is why you get a "cannot switch timeline" error. 69 | now master, we're gonna reclone you from the new master 70 | and establish replication 71 | this card is now permanently lost 72 | then, we're going to switch directions. 73 | 74 | split-brain 75 | ----------- 76 | 77 | same group 78 | 79 | here's all the network connections 80 | writer, you need to decide who to write to 81 | now, drop the connection between master & replica 82 | replica, you are now a master 83 | writer, who do you write to? 84 | 85 | remastering 86 | ----------- 87 | 88 | add third person 89 | you are replica2 90 | master pass red card to replica1 and black to replica2 91 | now, master is down 92 | how to choose which replica is new master? 93 | 94 | replication lag 95 | --------------- 96 | 97 | 1 writer 98 | 2 master 99 | 3 replica 100 | 4 reader 101 | 102 | explain vacuum, blocking and replicas 103 | 2 reasons for lag: replica busy, 104 | and file operations 105 | writer, pass cards to master 106 | these two jokers are "vacuum" cards 107 | insert them whereever you like 108 | reader, request cards from replica 109 | hold on to cards for a while 110 | if joker comes while reader is holding a card 111 | don't take any more replication until 112 | reader gives back card 113 | go 114 | 115 | synchronous replication 116 | ----------------------- 117 | 118 | 1 writer 119 | 2 master 120 | 3 replica 121 | 122 | writer, pass cards 123 | pairs, here's the card and here's the ack 124 | master, pass card to replica 125 | replica, pass ack back to master 126 | master pass ack back to writer 127 | go 128 | 129 | cascading replication 130 | --------------------- 131 | 132 | 1 master 133 | 2 replica 134 | 3 2nd replica 135 | 4. writer 136 | 137 | writer, pass cards to master 138 | replica, hold onto cards 139 | now, clone replica 140 | replica, pass cards to 2nd replica 141 | master, you're down 142 | replica, you're promoted 143 | writer, pass cards to new master 144 | 145 | load-balancing 146 | -------------- 147 | 148 | 1 application 149 | 2 master 150 | 3 replica 151 | 4 proxy 152 | 153 | application, red cards are reads, black are writes 154 | pass them all to master 155 | now start passing them faster 156 | replica, you're idle. act idle 157 | wouldn't it be nice if we could make the replica do some work? 158 | one way: 159 | application, pass red cards to replica, black to master 160 | another way: 161 | proxy, you're in the middle 162 | proxy, you'll sort red from black cards 163 | now, application, I want you to pass a black card 164 | and then ask for it back really quickly 165 | 166 | -------------------------------------------------------------------------------- /tutorial/pgReplicationTutorial.odp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/tutorial/pgReplicationTutorial.odp -------------------------------------------------------------------------------- /tutorial/pgReplicationTutorial.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/tutorial/pgReplicationTutorial.pdf -------------------------------------------------------------------------------- /vagrant/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:precise 2 | 3 | RUN apt-key adv --keyserver pgp.mit.edu --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8 4 | 5 | ENV PG_MAJOR 9.4 6 | 7 | ENV PG_VERSION 9.4+169.pgdg14.04+1 8 | 9 | ENV DEBIAN_FRONTEND=noninteractive 10 | 11 | RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* \ 12 | && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 13 | 14 | ENV LANG en_US.utf8 15 | 16 | COPY setup /setup 17 | 18 | RUN /setup/package_install.sh 19 | 20 | RUN /setup/dbprep.sh 21 | 22 | CMD ["bash"] -------------------------------------------------------------------------------- /vagrant/Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | Vagrant.configure("2") do |config| 5 | # All Vagrant configuration is done here. The most common configuration 6 | # options are documented and commented below. For a complete reference, 7 | # please see the online documentation at vagrantup.com. 8 | 9 | # Every Vagrant virtual environment requires a box to build off of. 10 | config.vm.box = "precise" 11 | 12 | # Create a forwarded port mapping which allows access to a specific port 13 | # within the machine from a port on the host machine. In the example below, 14 | # accessing "localhost:15432" will access the postgres port on the guest machine. 15 | # config.vm.network :forwarded_port, guest: 5432, host: 15432 16 | 17 | # Create a private network, which allows host-only access to the machine 18 | # using a specific IP. 19 | config.vm.network :private_network, ip: "192.168.199.101" 20 | 21 | # Create a public network, which generally matched to bridged network. 22 | # Bridged networks make the machine appear as another physical device on 23 | # your network. 24 | # config.vm.network :public_network 25 | # config.vm.network :private_network, ip: "44.44.44.100" 26 | 27 | # Share an additional folder to the guest VM. The first argument is 28 | # the path on the host to the actual folder. The second argument is 29 | # the path on the guest to mount the folder. And the optional third 30 | # argument is a set of non-required options. 31 | config.vm.synced_folder "setup", "/setup" 32 | 33 | # Provider-specific configuration so you can fine-tune various 34 | # backing providers for Vagrant. These expose provider-specific options. 35 | # Example for VirtualBox: 36 | # 37 | config.vm.provider :virtualbox do |vb| 38 | # # Don't boot with headless mode 39 | # vb.gui = true 40 | # 41 | # Use VBoxManage to customize the VM. For example to change memory: 42 | # memory is set to 384MB 43 | vb.customize ["modifyvm", :id, "--memory", "384"] 44 | vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] 45 | vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"] 46 | end 47 | 48 | # provision 49 | config.vm.provision :shell, :path => "bootstrap.sh" 50 | 51 | 52 | end 53 | -------------------------------------------------------------------------------- /vagrant/bootstrap.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | echo 'starting provisioning' 4 | 5 | # setup apt-get to pull from apt.postgresql.org 6 | 7 | echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main 9.4" > /etc/apt/sources.list.d/pgdg.list 8 | wget -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | apt-key add - 9 | 10 | # update apt 11 | apt-get update 12 | apt-get -y -q install pgdg-keyring 13 | 14 | # install postgresql and a bunch of accessories 15 | apt-get -y -q install postgresql-client-9.4 16 | apt-get -y -q install postgresql-9.4 17 | apt-get -y -q install postgresql-contrib-9.4 18 | apt-get -y -q install postgresql-plpython-9.4 19 | apt-get -y -q install pgbouncer 20 | 21 | # install PostGIS 22 | #apt-get -y -q install postgresql-9.4-postgis-2.1 23 | 24 | # install alternate editor and tmux 25 | apt-get -y -q install joe 26 | apt-get -y -q install tmux 27 | 28 | /setup/dbprep.sh 29 | 30 | echo '' 31 | echo 'vagrant loaded and ready for tutorial' 32 | echo 'version 0.5.1 of pgReplicationTutorial environment' 33 | echo 'now with PostgreSQL 9.4' 34 | 35 | exit 0 36 | 37 | 38 | 39 | 40 | -------------------------------------------------------------------------------- /vagrant/setup/dbprep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # stop postgres 4 | service postgresql stop 5 | 6 | # copy configuration files 7 | mv /var/lib/postgresql/9.4/main /var/lib/postgresql/9.4/master 8 | cp /setup/postgres/master/* /var/lib/postgresql/9.4/master/ 9 | mkdir /var/lib/postgresql/9.4/master/conf.d 10 | cp /setup/postgres/pgbouncer.ini /etc/pgbouncer/ 11 | cp /setup/postgres/userlist.txt /etc/pgbouncer/ 12 | cp /setup/postgres/pgbouncer.default /etc/default/pgbouncer 13 | chown -R postgres /etc/pgbouncer 14 | 15 | # create the archiving and replica directories 16 | mkdir /var/lib/postgresql/wal_archive 17 | mkdir /var/lib/postgresql/9.4/replica1 18 | mkdir /var/lib/postgresql/9.4/replica2 19 | chown -R postgres /var/lib/postgresql/* 20 | chmod 700 /var/lib/postgresql/9.4/replica1 21 | chmod 700 /var/lib/postgresql/9.4/replica2 22 | 23 | #link pg_ctl and friends 24 | ln -s /usr/lib/postgresql/9.4/bin/pg_ctl /usr/bin/pg_ctl 25 | ln -s /usr/lib/postgresql/9.4/bin/initdb /usr/bin/initdb 26 | #ln -s /usr/lib/postgresql/9.4/bin/pg_archivecleanup /usr/bin/pg_archivecleanup 27 | 28 | # restart postgresql 29 | su - postgres -c "/usr/bin/pg_ctl -w -D /var/lib/postgresql/9.4/master start" 30 | 31 | # load the libdata database 32 | psql -U postgres -f /setup/postgres/libdata.users.sql postgres 33 | pg_restore -e -U postgres -d libdata /setup/postgres/libdata.dump 34 | # load a postGIS database 35 | #pg_restore -e -U postgres -O -x -d earth /setup/postgres/earth.dump 36 | 37 | exit 0 38 | -------------------------------------------------------------------------------- /vagrant/setup/package_install.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | echo 'starting provisioning' 4 | 5 | #cp -rp /import /setup 6 | #chmod -R +rx /setup 7 | 8 | # install wget 9 | 10 | apt-get update 11 | apt-get -y -q install wget 12 | apt-get -y -q install apt-utils 13 | 14 | # setup apt-get to pull from apt.postgresql.org 15 | 16 | echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main 9.4" > /etc/apt/sources.list.d/pgdg.list 17 | wget -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | apt-key add - 18 | 19 | # update apt 20 | apt-get update 21 | apt-get -y -q install pgdg-keyring 22 | 23 | # install some basics 24 | apt-get -y -q install libreadline-dev 25 | apt-get -y -q install less 26 | 27 | # install postgresql and a bunch of accessories 28 | apt-get -y -q install postgresql-client-9.4 29 | apt-get -y -q install postgresql-9.4 30 | apt-get -y -q install postgresql-contrib-9.4 31 | apt-get -y -q install postgresql-plpython-9.4 32 | apt-get -y -q install pgbouncer 33 | 34 | # install PostGIS 35 | #apt-get -y -q install postgresql-9.4-postgis-2.1 36 | 37 | # install alternate editor and tmux and other utilities 38 | apt-get -y -q install joe 39 | apt-get -y -q install nano 40 | apt-get -y -q install vim 41 | apt-get -y -q install tmux 42 | apt-get -y -q install rsync 43 | 44 | echo '' 45 | echo 'container loaded and ready for tutorial' 46 | echo 'version 0.7 of pgReplicationTutorial environment' 47 | echo 'now with 9.4 and docker support' 48 | 49 | exit 0 50 | 51 | 52 | 53 | 54 | -------------------------------------------------------------------------------- /vagrant/setup/pgbench/#transfer1.bench#: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \set ntellers 10 * :scale 4 | \setrandom aida 1 100 5 | \setrandom aidb 1 100 6 | \setrandom bida 1 :nbranches 7 | \setrandom bidb 1 :nbranches 8 | \setrandom tid 1 :ntellers 9 | \setrandom delta 1 1000 10 | \setrandom randsleep 100 250 11 | BEGIN; 12 | UPDATE pgbench_accounts SET abalance = abalance - :delta WHERE aid = :aida; 13 | SELECT abalance FROM pgbench_accounts WHERE aid = :aida; 14 | \sleep :randsleep ms 15 | UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aidb; 16 | SELECT abalance FROM pgbench_accounts WHERE aid = :aidb; 17 | \sleep :randsleep ms 18 | UPDATE pgbench_branches SET bbalance = bbalance - :delta WHERE bid = :bida; 19 | UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bidb; 20 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bida, :aida, :delta, CURRENT_TIMESTAMP); 21 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bidb, :aidb, :delta, CURRENT_TIMESTAMP); 22 | END; -------------------------------------------------------------------------------- /vagrant/setup/pgbench/balancecheck1.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set naccounts 100000 * :scale 3 | \setrandom aid 1 :naccounts 4 | \setrandom delta 1 1000 5 | \setrandom randsleep 100 250 6 | SELECT abalance > :delta FROM pgbench_accounts WHERE aid = :aid; 7 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/balancecheck2.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set naccounts 100000 * :scale 3 | \setrandom aid 1 :naccounts 4 | \setrandom delta 1 1000 5 | \setrandom randsleep 10 50 6 | SELECT abalance > :delta FROM pgbench_accounts WHERE aid = :aid; 7 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/balancecheck3.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set naccounts 100000 * :scale 3 | \setrandom aid 1 :naccounts 4 | \setrandom delta 1 1000 5 | \setrandom randsleep 50 100 6 | SELECT abalance > :delta FROM pgbench_accounts WHERE aid = :aid; 7 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/balancecheck4.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set naccounts 100000 * :scale 3 | \setrandom aid 1 :naccounts 4 | \setrandom delta 1 1000 5 | \setrandom randsleep 50 100 6 | SELECT abalance > :delta FROM pgbench_accounts WHERE aid = :aid; 7 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/branchreport.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \setrandom bid 1 :nbranches 4 | \setrandom randsleep 100 250 5 | SELECT bid, count(aid), sum(abalance) FROM pgbench_accounts WHERE bid = :bid GROUP BY bid; 6 | SELECT bid, tid, count(*), sum(delta) FROM pgbench_history WHERE bid = :bid GROUP BY bid, tid ORDER BY bid, tid; 7 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/deposit.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \set ntellers 10 * :scale 4 | \set naccounts 100000 * :scale 5 | \setrandom aid 1 :naccounts 6 | \setrandom bid 1 :nbranches 7 | \setrandom tid 1 :ntellers 8 | \setrandom delta 1 1000 9 | \setrandom randsleep 50 200 10 | BEGIN; 11 | UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 12 | SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 13 | UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 14 | UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 15 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 16 | END; 17 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/pgbench/hang.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \set ntellers 10 * :scale 4 | \setrandom aida 1 100 5 | \setrandom bida 1 :nbranches 6 | \setrandom tid 1 :ntellers 7 | \setrandom delta 1 100 8 | BEGIN; 9 | UPDATE pgbench_accounts SET abalance = abalance - :delta WHERE aid = :aida; 10 | SELECT abalance FROM pgbench_accounts WHERE aid = :aida; 11 | \sleep 600 s 12 | END; -------------------------------------------------------------------------------- /vagrant/setup/pgbench/runbench.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | pgbench -U bench -c 3 bench -T 3600 -f balancecheck1.bench -f balancecheck2.bench -f balancecheck3.bench -f transfer1.bench -f deposit.bench -f withdrawal.bench -f branchreport.bench 4 | -------------------------------------------------------------------------------- /vagrant/setup/pgbench/runbench_replica1.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | pgbench -U bench -p 5433 -c 3 bench -T 3600 -f balancecheck1.bench -f balancecheck2.bench -f balancecheck3.bench -f transfer1.bench -f deposit.bench -f withdrawal.bench -f branchreport.bench 4 | -------------------------------------------------------------------------------- /vagrant/setup/pgbench/runbench_replica1.sh~: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | pgbench -U bench -c 3 bench -T 3600 -f balancecheck1.bench -f balancecheck2.bench -f balancecheck3.bench -f transfer1.bench -f deposit.bench -f withdrawal.bench -f branchreport.bench 4 | -------------------------------------------------------------------------------- /vagrant/setup/pgbench/transfer1.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \set ntellers 10 * :scale 4 | \setrandom aida 1 100 5 | \setrandom aidb 1 100 6 | \setrandom bida 1 :nbranches 7 | \setrandom bidb 1 :nbranches 8 | \setrandom tid 1 :ntellers 9 | \setrandom delta 1 1000 10 | \setrandom randsleep 500 1500 11 | BEGIN; 12 | UPDATE pgbench_accounts SET abalance = abalance - :delta WHERE aid = :aida; 13 | SELECT abalance FROM pgbench_accounts WHERE aid = :aida; 14 | \sleep :randsleep ms 15 | UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aidb; 16 | SELECT abalance FROM pgbench_accounts WHERE aid = :aidb; 17 | \sleep :randsleep ms 18 | UPDATE pgbench_branches SET bbalance = bbalance - :delta WHERE bid = :bida; 19 | UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bidb; 20 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bida, :aida, :delta, CURRENT_TIMESTAMP); 21 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bidb, :aidb, :delta, CURRENT_TIMESTAMP); 22 | END; -------------------------------------------------------------------------------- /vagrant/setup/pgbench/withdrawal.bench: -------------------------------------------------------------------------------- 1 | \set scale 10 2 | \set nbranches :scale 3 | \set ntellers 10 * :scale 4 | \set naccounts 100000 * :scale 5 | \setrandom aid 1 :naccounts 6 | \setrandom bid 1 :nbranches 7 | \setrandom tid 1 :ntellers 8 | \setrandom delta 1 1000 9 | \setrandom randsleep 50 200 10 | BEGIN; 11 | UPDATE pgbench_accounts SET abalance = abalance - :delta WHERE aid = :aid; 12 | SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 13 | UPDATE pgbench_tellers SET tbalance = tbalance - :delta WHERE tid = :tid; 14 | UPDATE pgbench_branches SET bbalance = bbalance - :delta WHERE bid = :bid; 15 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 16 | END; 17 | \sleep :randsleep ms -------------------------------------------------------------------------------- /vagrant/setup/postgres/.pgpass: -------------------------------------------------------------------------------- 1 | localhost:*:*:postgres:superuser 2 | localhost:*:*:bench:benchmark 3 | localhost:*:*:replicator:replicate 4 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/archive_logs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ARCHIVEDIR=/var/lib/postgresql/wal_archive 4 | WALFROM=$1 5 | WALDEST=$2 6 | NOARCHIVEFILE=/etc/postgresql/9.4/master/NOARCHIVING 7 | 8 | # See whether we want all archiving off 9 | test -f ${NOARCHIVEFILE} && exit 0 10 | 11 | rsync --quiet --archive ${WALFROM} ${ARCHIVEDIR}/${WALDEST} 12 | if [ $? -ne 0 ]; then 13 | exit 1 14 | fi 15 | 16 | exit 0 -------------------------------------------------------------------------------- /vagrant/setup/postgres/earth.dump: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/vagrant/setup/postgres/earth.dump -------------------------------------------------------------------------------- /vagrant/setup/postgres/fouine.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/vagrant/setup/postgres/fouine.html -------------------------------------------------------------------------------- /vagrant/setup/postgres/libdata.dump: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberkus/pgReplicationTutorial/423d9545c65ecc99ab5448f46c5b5a74f601adb4/vagrant/setup/postgres/libdata.dump -------------------------------------------------------------------------------- /vagrant/setup/postgres/libdata.users.sql: -------------------------------------------------------------------------------- 1 | -- users and database for the libdata database 2 | 3 | CREATE ROLE libdata PASSWORD 'books' LOGIN; 4 | 5 | CREATE DATABASE libdata; 6 | 7 | ALTER DATABASE libdata OWNER TO libdata; 8 | 9 | ALTER ROLE postgres PASSWORD 'superuser'; 10 | 11 | CREATE ROLE bench PASSWORD 'benchmark' LOGIN CREATEDB; 12 | 13 | CREATE DATABASE earth; 14 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/master/pg_hba.conf: -------------------------------------------------------------------------------- 1 | # PostgreSQL Client Authentication Configuration File 2 | # =================================================== 3 | # 4 | # Refer to the "Client Authentication" section in the PostgreSQL 5 | # documentation for a complete description of this file. A short 6 | # synopsis follows. 7 | # 8 | # This file controls: which hosts are allowed to connect, how clients 9 | # are authenticated, which PostgreSQL user names they can use, which 10 | # databases they can access. Records take one of these forms: 11 | # 12 | # local DATABASE USER METHOD [OPTIONS] 13 | # host DATABASE USER ADDRESS METHOD [OPTIONS] 14 | # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] 15 | # hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] 16 | 17 | # Put your actual configuration here 18 | # ---------------------------------- 19 | 20 | # TYPE DATABASE USER ADDRESS METHOD 21 | 22 | # initial local access setup, which is trust for all 23 | # configured this way so that we can deal with security later 24 | 25 | local all all trust 26 | host all all 127.0.0.1/32 trust 27 | host all all ::1/128 trust 28 | 29 | # Allow replication connections from localhost, by a user with the 30 | # replication privilege. 31 | # Replication connections are not included in any of the above. 32 | # Initially will be set to "trust" for easy demo. 33 | 34 | local replication postgres trust 35 | host replication postgres 127.0.0.1/32 trust 36 | host replication postgres ::1/128 trust 37 | 38 | # more serious configuration, also using a seperate "replication" user 39 | 40 | local replication replicator md5 41 | host replication replicator 127.0.0.1/32 md5 42 | host replication replicator ::1/128 md5 43 | 44 | # example public connection 45 | # hostssl replication replicator 192.168.0.0/24 md5 46 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/master/postgresql.conf: -------------------------------------------------------------------------------- 1 | # ------------------------------------------- 2 | # Simple PostgreSQL Configuration File v. 9.4 3 | # For Replication Tutorial 4 | # ------------------------------------------- 5 | 6 | # The settings below are appropriate for doing 7 | # a replication demo on a laptop. They are 8 | # not suitable for production use. 9 | 10 | # CONNECTIONS 11 | 12 | listen_addresses = '*' 13 | max_connections = 20 14 | port = 5432 15 | 16 | # RESOURCES 17 | 18 | shared_buffers = 32MB 19 | work_mem = 1MB 20 | maintenance_work_mem = 2MB 21 | dynamic_shared_memory_type = posix 22 | max_worker_processes = 8 23 | 24 | # WAL SETTINGS (non-replication) 25 | 26 | checkpoint_segments = 4 27 | checkpoint_completion_target = 0.0 28 | 29 | # ============================================ 30 | # Replication Settings 31 | # ============================================ 32 | 33 | # The settings below control replication 34 | # These are the ones you will manipulate in this 35 | # tutorial. 36 | 37 | # MASTER 38 | # these settings apply only to the master server 39 | # they will be ignored on non-cascading replicas 40 | 41 | # use replication level of logging 42 | # required 43 | # change from 9.3: now using level "logical" 44 | wal_level = logical 45 | 46 | # number of replication connections to support 47 | # pg_basebackup requires 2-3 48 | # good rule is: 49 | # ( planned_replicas X 2 ) + 3 50 | # note that cascading replicas need this too 51 | max_wal_senders = 5 52 | 53 | # number of transaction log segments to keep 54 | # in case replicas need them to catch up 55 | # generally you want 1 hour to 12 hours of 56 | # logs, depending on how big that is 57 | wal_keep_segments = 6 58 | 59 | # number of replication "slots" allowed for replicas 60 | # to track how far replicas are caught up 61 | # as an alternative to wal_keep_segments 62 | max_replication_slots = 3 63 | 64 | # keepalive for the walsender, so that we know if 65 | # the replica has vanished, even if we have no 66 | # data to send. 67 | wal_sender_timeout = 15s 68 | 69 | # delay VACUUM runs so that we don't cause query lag 70 | vacuum_defer_cleanup_age = 0 71 | 72 | # SYNCHRONOUS REPLICATION 73 | 74 | # list of appnames of connections which are 75 | # synchornous standbys. The list is in priority 76 | # order. 77 | # synchronous_standby_names = '' 78 | 79 | # ARCHIVING 80 | # controls file-based archiving of transaction logs 81 | 82 | # are we using WAL archiving? Note that turning 83 | # this setting on requires a restart 84 | # archive_mode = on 85 | 86 | # if we are archiving, what script do we call? 87 | # use a script here, NOT just "cp" 88 | # archive_command = '/setup/postgres/archive_logs.sh %p %f' 89 | 90 | # if we are archiving, we also want to force a 91 | # file to be written every so often even if there's 92 | # not enough data. 93 | # archive_timeout = 30s 94 | 95 | # REPLICA 96 | # replica settings are ignored on the master and only 97 | # used on the replica 98 | 99 | # are we accepting read-only queries while being 100 | # replicated to? 101 | hot_standby = on 102 | 103 | # maximum delay on before applying updates from the 104 | # master and cancelling queries, for each replication 105 | # method 106 | max_standby_streaming_delay = 30s 107 | max_standby_archive_delay = 60s 108 | 109 | # whether to send messages about snapshot visibility 110 | # to the master, to delay vacuum and limit query cancel 111 | # turn it on for synch replication. 112 | hot_standby_feedback = off 113 | 114 | # keepalive for the walreceiver, so that we know when 115 | # the master has gone offline 116 | wal_receiver_timeout = 60s 117 | 118 | # =============================================== 119 | 120 | # CONFIG FILE INCLUDES 121 | 122 | include_dir = 'conf.d' 123 | 124 | # LOGGING SETTINGS 125 | 126 | # These settings control logging 127 | # so that you can check how replication is going. 128 | 129 | log_destination = 'stderr' 130 | logging_collector = on 131 | log_directory = '/var/log/postgresql/' 132 | log_filename = 'postgresql-master' 133 | log_line_prefix = '[%u @ %d @ %m]:' 134 | log_truncate_on_rotation = on 135 | log_connections = on 136 | 137 | # File Locations 138 | # -------------- 139 | # configured for demo 140 | # these have been changed from the standard PostgreSQL file 141 | # locations on Ubuntu 142 | 143 | data_directory = '/var/lib/postgresql/9.4/master' 144 | external_pid_file = '/var/run/postgresql/9.4-master.pid' 145 | unix_socket_directories = '/var/run/postgresql' 146 | 147 | 148 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/pgbouncer.default: -------------------------------------------------------------------------------- 1 | 2 | # Set to 1 after you've configured pgbouncer in 3 | # /etc/pgbouncer 4 | 5 | START=1 6 | 7 | # It is possible to change the options the daemon is started with. 8 | # The default setting is shown below, uncomment and apply your changes 9 | # if you need to modify it. 10 | 11 | #OPTS="-d /etc/pgbouncer/pgbouncer.ini" 12 | 13 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/pgbouncer.ini: -------------------------------------------------------------------------------- 1 | ;; database name = connect string 2 | ;; 3 | ;; connect string params: 4 | ;; dbname= host= port= user= password= 5 | ;; client_encoding= datestyle= timezone= 6 | ;; pool_size= connect_query= 7 | [databases] 8 | bench = host=localhost port=5433 9 | libdata = host=localhost port=5433 10 | postgres = host=localhost port=5433 11 | bench_ro = host=localhost port=5434 dbname=bench 12 | libdata_ro = host=localhost port=5434 dbname=libdata 13 | postgres_ro = host=localhost port=5434 dbname=postgres 14 | 15 | ;; Configuration section 16 | [pgbouncer] 17 | 18 | ;;; 19 | ;;; Administrative settings 20 | ;;; 21 | 22 | logfile = /var/log/postgresql/pgbouncer.log 23 | pidfile = /var/run/postgresql/pgbouncer.pid 24 | 25 | ;;; 26 | ;;; Where to wait for clients 27 | ;;; 28 | 29 | ; ip address or * which means all ip-s 30 | listen_addr = * 31 | listen_port = 6432 32 | 33 | ; unix socket is also used for -R. 34 | ; On debian it should be /var/run/postgresql 35 | ;unix_socket_dir = /tmp 36 | ;unix_socket_mode = 0777 37 | ;unix_socket_group = 38 | unix_socket_dir = /var/run/postgresql 39 | 40 | ;;; 41 | ;;; Authentication settings 42 | ;;; 43 | 44 | ; any, trust, plain, crypt, md5 45 | auth_type = md5 46 | ;auth_file = /8.0/main/global/pg_auth 47 | auth_file = /etc/pgbouncer/userlist.txt 48 | 49 | ;;; 50 | ;;; Users allowed into database 'pgbouncer' 51 | ;;; 52 | 53 | ; comma-separated list of users, who are allowed to change settings 54 | admin_users = postgres 55 | 56 | ; comma-separated list of users who are just allowed to use SHOW command 57 | ;stats_users = stats, root 58 | 59 | ;;; 60 | ;;; Pooler personality questions 61 | ;;; 62 | 63 | ; When server connection is released back to pool: 64 | ; session - after client disconnects 65 | ; transaction - after transaction finishes 66 | ; statement - after statement finishes 67 | pool_mode = transaction 68 | 69 | ; Query for cleaning connection immediately after releasing from client. 70 | ; No need to put ROLLBACK here, pgbouncer does not reuse connections 71 | ; where transaction is left open. 72 | ; Empty if statement pooling is in use. 73 | ; 74 | ;server_reset_query = DISCARD ALL 75 | 76 | ;;; 77 | ;;; Connection limits 78 | ;;; 79 | 80 | ; total number of clients that can connect 81 | max_client_conn = 50 82 | 83 | ; default pool size. 20 is good number when transaction pooling 84 | ; is in use, in session pooling it needs to be the number of 85 | ; max clients you want to handle at any moment 86 | default_pool_size = 6 87 | 88 | ; how many additional connection to allow in case of trouble 89 | reserve_pool_size = 10 90 | 91 | ; if a clients needs to wait more than this many seconds, use reserve pool 92 | reserve_pool_timeout = 3 93 | 94 | ; log if client connects or server connection is made 95 | ;log_connections = 1 96 | 97 | ; log if and why connection was closed 98 | ;log_disconnections = 1 99 | 100 | ; log error messages pooler sends to clients 101 | ;log_pooler_errors = 1 102 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/pg_hba.conf: -------------------------------------------------------------------------------- 1 | # PostgreSQL Client Authentication Configuration File 2 | # =================================================== 3 | # 4 | # Refer to the "Client Authentication" section in the PostgreSQL 5 | # documentation for a complete description of this file. A short 6 | # synopsis follows. 7 | # 8 | # This file controls: which hosts are allowed to connect, how clients 9 | # are authenticated, which PostgreSQL user names they can use, which 10 | # databases they can access. Records take one of these forms: 11 | # 12 | # local DATABASE USER METHOD [OPTIONS] 13 | # host DATABASE USER ADDRESS METHOD [OPTIONS] 14 | # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] 15 | # hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] 16 | 17 | # Put your actual configuration here 18 | # ---------------------------------- 19 | 20 | # TYPE DATABASE USER ADDRESS METHOD 21 | 22 | # initial local access setup, which is trust for all 23 | # configured this way so that we can deal with security later 24 | 25 | local all all trust 26 | host all all 127.0.0.1/32 trust 27 | host all all ::1/128 trust 28 | 29 | # Allow replication connections from localhost, by a user with the 30 | # replication privilege. 31 | # Replication connections are not included in any of the above. 32 | # Initially will be set to "trust" for easy demo. 33 | 34 | local replication postgres trust 35 | host replication postgres 127.0.0.1/32 trust 36 | host replication postgres ::1/128 trust 37 | 38 | # more serious configuration, also using a seperate "replication" user 39 | 40 | local replication replicator md5 41 | host replication replicator 127.0.0.1/32 md5 42 | host replication replicator ::1/128 md5 43 | 44 | # example public connection 45 | # hostssl replication replicator 192.168.0.0/24 md5 46 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/postgresql.conf: -------------------------------------------------------------------------------- 1 | # ------------------------------------------- 2 | # Simple PostgreSQL Configuration File v. 9.3 3 | # For Replication Tutorial 4 | # ------------------------------------------- 5 | 6 | # The settings below are appropriate for doing 7 | # a replication demo on a laptop. They are 8 | # not suitable for production use. 9 | 10 | # CONNECTIONS 11 | 12 | listen_addresses = '*' 13 | max_connections = 20 14 | port = 5433 15 | 16 | # RESOURCES 17 | 18 | shared_buffers = 32MB 19 | work_mem = 1MB 20 | maintenance_work_mem = 2MB 21 | dynamic_shared_memory_type = posix 22 | max_worker_processes = 8 23 | 24 | # WAL SETTINGS (non-replication) 25 | 26 | checkpoint_segments = 4 27 | checkpoint_completion_target = 0.0 28 | 29 | # ============================================ 30 | # Replication Settings 31 | # ============================================ 32 | 33 | # The settings below control replication 34 | # These are the ones you will manipulate in this 35 | # tutorial. 36 | 37 | # MASTER 38 | # these settings apply only to the master server 39 | # they will be ignored on non-cascading replicas 40 | 41 | # use replication level of logging 42 | # required 43 | wal_level = logical 44 | 45 | # number of replication connections to support 46 | # pg_basebackup requires 2-3 47 | # good rule is: 48 | # ( planned_replicas X 2 ) + 3 49 | # note that cascading replicas need this too 50 | max_wal_senders = 5 51 | 52 | # number of transaction log segments to keep 53 | # in case replicas need them to catch up 54 | # generally you want 1 hour to 12 hours of 55 | # logs, depending on how big that is 56 | wal_keep_segments = 6 57 | 58 | # number of replication "slots" allowed for replicas 59 | # to track how far replicas are caught up 60 | # as an alternative to wal_keep_segments 61 | max_replication_slots = 3 62 | 63 | # keepalive for the walsender, so that we know if 64 | # the replica has vanished, even if we have no 65 | # data to send. 66 | wal_sender_timeout = 15s 67 | 68 | # delay VACUUM runs so that we don't cause query lag 69 | vacuum_defer_cleanup_age = 0 70 | 71 | # SYNCHRONOUS REPLICATION 72 | 73 | # list of appnames of connections which are 74 | # synchornous standbys. The list is in priority 75 | # order. 76 | # synchronous_standby_names = '' 77 | 78 | # ARCHIVING 79 | # controls file-based archiving of transaction logs 80 | 81 | # are we using WAL archiving? Note that turning 82 | # this setting on requires a restart 83 | # archive_mode = on 84 | 85 | # if we are archiving, what script do we call? 86 | # use a script here, NOT just "cp" 87 | # archive_command = '/setup/postgres/archive_logs.sh %p %f' 88 | 89 | # if we are archiving, we also want to force a 90 | # file to be written every so often even if there's 91 | # not enough data. 92 | # archive_timeout = 30s 93 | 94 | # REPLICA 95 | # replica settings are ignored on the master and only 96 | # used on the replica 97 | 98 | # are we accepting read-only queries while being 99 | # replicated to? 100 | hot_standby = on 101 | 102 | # maximum delay on before applying updates from the 103 | # master and cancelling queries, for each replication 104 | # method 105 | max_standby_streaming_delay = 30s 106 | max_standby_archive_delay = 60s 107 | 108 | # whether to send messages about snapshot visibility 109 | # to the master, to delay vacuum and limit query cancel 110 | # turn it on for synch replication. 111 | hot_standby_feedback = off 112 | 113 | # keepalive for the walreceiver, so that we know when 114 | # the master has gone offline 115 | wal_receiver_timeout = 60s 116 | 117 | # =============================================== 118 | 119 | # CONFIG FILE INCLUDES 120 | 121 | include_dir = 'conf.d' 122 | 123 | # LOGGING SETTINGS 124 | 125 | # These settings control logging 126 | # so that you can check how replication is going. 127 | 128 | log_destination = 'stderr' 129 | logging_collector = on 130 | log_directory = '/var/log/postgresql/' 131 | log_filename = 'postgresql-replica1' 132 | log_line_prefix = '[%u @ %d @ %m]:' 133 | log_truncate_on_rotation = on 134 | log_connections = on 135 | 136 | # File Locations 137 | # -------------- 138 | # configured for demo 139 | # these have been changed from the standard PostgreSQL file 140 | # locations on Ubuntu 141 | 142 | data_directory = '/var/lib/postgresql/9.4/replica1' 143 | external_pid_file = '/var/run/postgresql/9.4-replica1.pid' 144 | unix_socket_directories = '/var/run/postgresql' 145 | 146 | 147 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/recovery.conf: -------------------------------------------------------------------------------- 1 | # ------------------------------- 2 | # PostgreSQL recovery config file 3 | # ------------------------------- 4 | # 5 | # Edit this file to provide the parameters that PostgreSQL needs to 6 | # perform an archive recovery of a database, or to act as a replication 7 | # standby. 8 | # 9 | # If "recovery.conf" is present in the PostgreSQL data directory, it is 10 | # read on postmaster startup. After successful recovery, it is renamed 11 | # to "recovery.done" to ensure that we do not accidentally re-enter 12 | # archive recovery or standby mode. 13 | # 14 | #--------------------------------------------------------------------------- 15 | # STREAMING PARAMETERS 16 | #--------------------------------------------------------------------------- 17 | 18 | # below parameters should be set for streaming replication servers 19 | 20 | # standby_mode 21 | # When standby_mode is enabled, the PostgreSQL server will work as a 22 | # standby. It will continuously wait for the additional XLOG records, using 23 | # restore_command and/or primary_conninfo. 24 | 25 | standby_mode = on 26 | 27 | # primary_conninfo 28 | # If set, the PostgreSQL server will try to connect to the primary using this 29 | # connection string and receive XLOG records continuously. 30 | # Required for streaming replication 31 | 32 | primary_conninfo = 'host=localhost port=5432 user=postgres application_name=replica1' 33 | 34 | # recovery_target_timeline 35 | # Set to 'latest' to follow the master if it switches timeline 36 | # requires version 9.3 or WAL file archiving to be effective 37 | 38 | recovery_target_timeline = 'latest' 39 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/recovery.conf.replica1.archiving: -------------------------------------------------------------------------------- 1 | # ------------------------------- 2 | # PostgreSQL recovery config file 3 | # ------------------------------- 4 | # 5 | # Edit this file to provide the parameters that PostgreSQL needs to 6 | # perform an archive recovery of a database, or to act as a replication 7 | # standby. 8 | # 9 | # If "recovery.conf" is present in the PostgreSQL data directory, it is 10 | # read on postmaster startup. After successful recovery, it is renamed 11 | # to "recovery.done" to ensure that we do not accidentally re-enter 12 | # archive recovery or standby mode. 13 | # 14 | #--------------------------------------------------- 15 | # ARCHIVING Replication Parameters 16 | #--------------------------------------------------- 17 | 18 | # set the below if you are also doing replication via WAL file 19 | # archiving, with or without streaming 20 | 21 | # standby_mode 22 | # 23 | # When standby_mode is enabled, the PostgreSQL server will work as a 24 | # standby. It will continuously wait for the additional XLOG records, using 25 | # restore_command and/or primary_conninfo. 26 | 27 | standby_mode = on 28 | 29 | # restore command 30 | # specifies the shell command that is executed to copy log files 31 | # back from archival storage. The command string may contain %f, 32 | # which is replaced by the name of the desired log file, and %p, 33 | # which is replaced by the absolute path to copy the log file to. 34 | # 35 | # It is important that the command return nonzero exit status on failure. 36 | # The command *will* be asked for log files that are not present in the 37 | # archive; it must return nonzero when so asked. 38 | # 39 | # NOTE that the basename of %p will be different from %f; do not 40 | # expect them to be interchangeable. 41 | 42 | restore_command = 'cp /var/lib/postgresql/wal_archive/%f %p' 43 | 44 | # archive_cleanup_command 45 | # 46 | # specifies an optional shell command to execute at every restartpoint. 47 | # This can be useful for cleaning up the archive of a standby server. 48 | 49 | archive_cleanup_command = 'pg_archivecleanup /var/lib/postgresql/wal_archive %r' 50 | 51 | # recovery_target_timeline 52 | # Set to 'latest' to follow the master if it switches timeline 53 | # requires version 9.3 or WAL file archiving to be effective 54 | 55 | recovery_target_timeline = 'latest' 56 | 57 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/recovery.conf.replica1.dual: -------------------------------------------------------------------------------- 1 | # ------------------------------- 2 | # PostgreSQL recovery config file 3 | # ------------------------------- 4 | # 5 | # Edit this file to provide the parameters that PostgreSQL needs to 6 | # perform an archive recovery of a database, or to act as a replication 7 | # standby. 8 | # 9 | # If "recovery.conf" is present in the PostgreSQL data directory, it is 10 | # read on postmaster startup. After successful recovery, it is renamed 11 | # to "recovery.done" to ensure that we do not accidentally re-enter 12 | # archive recovery or standby mode. 13 | # 14 | #--------------------------------------------------------------------------- 15 | # STREAMING PARAMETERS 16 | #--------------------------------------------------------------------------- 17 | 18 | # below parameters should be set for streaming replication servers 19 | 20 | # standby_mode 21 | # When standby_mode is enabled, the PostgreSQL server will work as a 22 | # standby. It will continuously wait for the additional XLOG records, using 23 | # restore_command and/or primary_conninfo. 24 | 25 | standby_mode = on 26 | 27 | # primary_conninfo 28 | # If set, the PostgreSQL server will try to connect to the primary using this 29 | # connection string and receive XLOG records continuously. 30 | # Required for streaming replication 31 | 32 | primary_conninfo = 'host=localhost port=5432 user=replicator application_name=replica1' 33 | 34 | # recovery_target_timeline 35 | # Set to 'latest' to follow the master if it switches timeline 36 | # requires version 9.3 or WAL file archiving to be effective 37 | 38 | recovery_target_timeline = 'latest' 39 | 40 | # create trigger file for failover 41 | # if this file exists, failover is initiated 42 | 43 | trigger_file = 'PROMOTE' 44 | 45 | #--------------------------------------------------- 46 | # ARCHIVING Replication Parameters 47 | #--------------------------------------------------- 48 | 49 | # set the below if you are also doing replication via WAL file 50 | # archiving, with or without streaming 51 | 52 | # restore command 53 | # specifies the shell command that is executed to copy log files 54 | # back from archival storage. The command string may contain %f, 55 | # which is replaced by the name of the desired log file, and %p, 56 | # which is replaced by the absolute path to copy the log file to. 57 | # 58 | # It is important that the command return nonzero exit status on failure. 59 | # The command *will* be asked for log files that are not present in the 60 | # archive; it must return nonzero when so asked. 61 | # 62 | # NOTE that the basename of %p will be different from %f; do not 63 | # expect them to be interchangeable. 64 | 65 | restore_command = 'cp /var/lib/postgresql/wal_archive/%f %p' 66 | 67 | # archive_cleanup_command 68 | # 69 | # specifies an optional shell command to execute at every restartpoint. 70 | # This can be useful for cleaning up the archive of a standby server. 71 | 72 | archive_cleanup_command = 'pg_archivecleanup /var/lib/postgresql/wal_archive %r' -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica1/recovery.conf.replica1.slot: -------------------------------------------------------------------------------- 1 | # ------------------------------- 2 | # PostgreSQL recovery config file 3 | # ------------------------------- 4 | # 5 | # Edit this file to provide the parameters that PostgreSQL needs to 6 | # perform an archive recovery of a database, or to act as a replication 7 | # standby. 8 | # 9 | # If "recovery.conf" is present in the PostgreSQL data directory, it is 10 | # read on postmaster startup. After successful recovery, it is renamed 11 | # to "recovery.done" to ensure that we do not accidentally re-enter 12 | # archive recovery or standby mode. 13 | # 14 | #--------------------------------------------------------------------------- 15 | # STREAMING PARAMETERS 16 | #--------------------------------------------------------------------------- 17 | 18 | # below parameters should be set for streaming replication servers 19 | 20 | # standby_mode 21 | # When standby_mode is enabled, the PostgreSQL server will work as a 22 | # standby. It will continuously wait for the additional XLOG records, using 23 | # restore_command and/or primary_conninfo. 24 | 25 | standby_mode = on 26 | 27 | # primary_conninfo 28 | # If set, the PostgreSQL server will try to connect to the primary using this 29 | # connection string and receive XLOG records continuously. 30 | # Required for streaming replication 31 | 32 | primary_conninfo = 'host=localhost port=5432 user=replicator application_name=replica1' 33 | 34 | # replication slot 35 | # set to the slot on the master for this replica. this slot must already exist 36 | 37 | primary_slot_name = 'replica1' 38 | 39 | # recovery_target_timeline 40 | # Set to 'latest' to follow the master if it switches timeline 41 | # requires version 9.3 or WAL file archiving to be effective 42 | 43 | recovery_target_timeline = 'latest' 44 | 45 | # create trigger file for failover 46 | # if this file exists, failover is initiated 47 | 48 | trigger_file = 'PROMOTE' 49 | 50 | # for delayed standbys. add a delay before applying new transactions 51 | # in seconds 52 | 53 | # recovery_min_apply_delay = 3600 54 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica2/pg_hba.conf: -------------------------------------------------------------------------------- 1 | # PostgreSQL Client Authentication Configuration File 2 | # =================================================== 3 | # 4 | # Refer to the "Client Authentication" section in the PostgreSQL 5 | # documentation for a complete description of this file. A short 6 | # synopsis follows. 7 | # 8 | # This file controls: which hosts are allowed to connect, how clients 9 | # are authenticated, which PostgreSQL user names they can use, which 10 | # databases they can access. Records take one of these forms: 11 | # 12 | # local DATABASE USER METHOD [OPTIONS] 13 | # host DATABASE USER ADDRESS METHOD [OPTIONS] 14 | # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] 15 | # hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] 16 | 17 | # Put your actual configuration here 18 | # ---------------------------------- 19 | 20 | # TYPE DATABASE USER ADDRESS METHOD 21 | 22 | # initial local access setup, which is trust for all 23 | # configured this way so that we can deal with security later 24 | 25 | local all all trust 26 | host all all 127.0.0.1/32 trust 27 | host all all ::1/128 trust 28 | 29 | # Allow replication connections from localhost, by a user with the 30 | # replication privilege. 31 | # Replication connections are not included in any of the above. 32 | # Initially will be set to "trust" for easy demo. 33 | 34 | local replication postgres trust 35 | host replication postgres 127.0.0.1/32 trust 36 | host replication postgres ::1/128 trust 37 | 38 | # more serious configuration, also using a seperate "replication" user 39 | 40 | local replication replicator md5 41 | host replication replicator 127.0.0.1/32 md5 42 | host replication replicator ::1/128 md5 43 | 44 | # example public connection 45 | # hostssl replication replicator 192.168.0.0/24 md5 46 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica2/postgresql.conf: -------------------------------------------------------------------------------- 1 | # ------------------------------------------- 2 | # Simple PostgreSQL Configuration File v. 9.3 3 | # For Replication Tutorial 4 | # ------------------------------------------- 5 | 6 | # The settings below are appropriate for doing 7 | # a replication demo on a laptop. They are 8 | # not suitable for production use. 9 | 10 | # CONNECTIONS 11 | 12 | listen_addresses = '*' 13 | max_connections = 20 14 | port = 5434 15 | 16 | # RESOURCES 17 | 18 | shared_buffers = 32MB 19 | work_mem = 1MB 20 | maintenance_work_mem = 2MB 21 | dynamic_shared_memory_type = posix 22 | max_worker_processes = 8 23 | 24 | # WAL SETTINGS (non-replication) 25 | 26 | checkpoint_segments = 4 27 | checkpoint_completion_target = 0.0 28 | 29 | # ============================================ 30 | # Replication Settings 31 | # ============================================ 32 | 33 | # The settings below control replication 34 | # These are the ones you will manipulate in this 35 | # tutorial. 36 | 37 | # MASTER 38 | # these settings apply only to the master server 39 | # they will be ignored on non-cascading replicas 40 | 41 | # use replication level of logging 42 | # required 43 | wal_level = logical 44 | 45 | # number of replication connections to support 46 | # pg_basebackup requires 2-3 47 | # good rule is: 48 | # ( planned_replicas X 2 ) + 3 49 | # note that cascading replicas need this too 50 | max_wal_senders = 5 51 | 52 | # number of transaction log segments to keep 53 | # in case replicas need them to catch up 54 | # generally you want 1 hour to 12 hours of 55 | # logs, depending on how big that is 56 | wal_keep_segments = 6 57 | 58 | # number of replication "slots" allowed for replicas 59 | # to track how far replicas are caught up 60 | # as an alternative to wal_keep_segments 61 | max_replication_slots = 3 62 | 63 | # keepalive for the walsender, so that we know if 64 | # the replica has vanished, even if we have no 65 | # data to send. 66 | wal_sender_timeout = 15s 67 | 68 | # delay VACUUM runs so that we don't cause query lag 69 | vacuum_defer_cleanup_age = 0 70 | 71 | # SYNCHRONOUS REPLICATION 72 | 73 | # list of appnames of connections which are 74 | # synchornous standbys. The list is in priority 75 | # order. 76 | # synchronous_standby_names = '' 77 | 78 | # ARCHIVING 79 | # controls file-based archiving of transaction logs 80 | 81 | # are we using WAL archiving? Note that turning 82 | # this setting on requires a restart 83 | # archive_mode = on 84 | 85 | # if we are archiving, what script do we call? 86 | # use a script here, NOT just "cp" 87 | # archive_command = '/setup/postgres/archive_logs.sh %p %f' 88 | 89 | # if we are archiving, we also want to force a 90 | # file to be written every so often even if there's 91 | # not enough data. 92 | # archive_timeout = 30s 93 | 94 | # REPLICA 95 | # replica settings are ignored on the master and only 96 | # used on the replica 97 | 98 | # are we accepting read-only queries while being 99 | # replicated to? 100 | hot_standby = on 101 | 102 | # maximum delay on before applying updates from the 103 | # master and cancelling queries, for each replication 104 | # method 105 | max_standby_streaming_delay = 30s 106 | max_standby_archive_delay = 60s 107 | 108 | # whether to send messages about snapshot visibility 109 | # to the master, to delay vacuum and limit query cancel 110 | # turn it on for synch replication. 111 | hot_standby_feedback = off 112 | 113 | # keepalive for the walreceiver, so that we know when 114 | # the master has gone offline 115 | wal_receiver_timeout = 60s 116 | 117 | # =============================================== 118 | 119 | # CONFIG FILE INCLUDES 120 | 121 | include_dir = 'conf.d' 122 | 123 | # LOGGING SETTINGS 124 | 125 | # These settings control logging 126 | # so that you can check how replication is going. 127 | 128 | log_destination = 'stderr' 129 | logging_collector = on 130 | log_directory = '/var/log/postgresql/' 131 | log_filename = 'postgresql-replica2' 132 | log_line_prefix = '[%u @ %d @ %m]:' 133 | log_truncate_on_rotation = on 134 | log_connections = on 135 | 136 | # File Locations 137 | # -------------- 138 | # configured for demo 139 | # these have been changed from the standard PostgreSQL file 140 | # locations on Ubuntu 141 | 142 | data_directory = '/var/lib/postgresql/9.4/replica2' 143 | external_pid_file = '/var/run/postgresql/9.4-replica2.pid' 144 | unix_socket_directories = '/var/run/postgresql' 145 | 146 | 147 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/replica2/recovery.conf: -------------------------------------------------------------------------------- 1 | # ------------------------------- 2 | # PostgreSQL recovery config file 3 | # ------------------------------- 4 | # 5 | # Edit this file to provide the parameters that PostgreSQL needs to 6 | # perform an archive recovery of a database, or to act as a replication 7 | # standby. 8 | # 9 | # If "recovery.conf" is present in the PostgreSQL data directory, it is 10 | # read on postmaster startup. After successful recovery, it is renamed 11 | # to "recovery.done" to ensure that we do not accidentally re-enter 12 | # archive recovery or standby mode. 13 | # 14 | #--------------------------------------------------------------------------- 15 | # STREAMING PARAMETERS 16 | #--------------------------------------------------------------------------- 17 | 18 | # below parameters should be set for streaming replication servers 19 | 20 | # standby_mode 21 | # When standby_mode is enabled, the PostgreSQL server will work as a 22 | # standby. It will continuously wait for the additional XLOG records, using 23 | # restore_command and/or primary_conninfo. 24 | 25 | standby_mode = on 26 | 27 | # primary_conninfo 28 | # If set, the PostgreSQL server will try to connect to the primary using this 29 | # connection string and receive XLOG records continuously. 30 | # Required for streaming replication 31 | 32 | primary_conninfo = 'host=localhost port=5433 user=replicator application_name=replica2' 33 | 34 | # recovery_target_timeline 35 | # Set to 'latest' to follow the master if it switches timeline 36 | # requires version 9.3 or WAL file archiving to be effective 37 | 38 | recovery_target_timeline = 'latest' 39 | -------------------------------------------------------------------------------- /vagrant/setup/postgres/userlist.txt: -------------------------------------------------------------------------------- 1 | "postgres" "superuser" 2 | "bench" "benchmark" 3 | "libdata" "books" 4 | --------------------------------------------------------------------------------