├── .env.example ├── .gitignore ├── README.md ├── database ├── PDOConnector.php ├── database-connection.php └── public │ └── index.php ├── docker-compose.yml └── docker ├── .gitignore ├── mysql ├── Dockerfile ├── dumps │ └── .gitignore ├── primary │ ├── logs │ │ └── .gitignore │ └── my.conf ├── replica-1 │ ├── logs │ │ └── .gitignore │ └── my.conf └── replica-2 │ ├── logs │ └── .gitignore │ └── my.conf ├── nginx ├── Dockerfile └── default.conf └── php ├── Dockerfile └── php.ini /.env.example: -------------------------------------------------------------------------------- 1 | PRIMARY_MYSQL_DATABASE=main_db 2 | PRIMARY_MYSQL_ROOT_PASSWORD=secret 3 | PRIMARY_MYSQL_USER=user 4 | PRIMARY_MYSQL_PASSWORD=secret 5 | PRIMARY_MYSQL_PORT=3307 6 | 7 | REPLICA_1_MYSQL_DATABASE=replica_db 8 | REPLICA_1_MYSQL_ROOT_PASSWORD=secret 9 | REPLICA_1_MYSQL_USER=user 10 | REPLICA_1_MYSQL_PASSWORD=secret 11 | REPLICA_1_MYSQL_PORT=3308 12 | 13 | REPLICA_2_MYSQL_DATABASE=replica_db 14 | REPLICA_2_MYSQL_ROOT_PASSWORD=secret 15 | REPLICA_2_MYSQL_USER=user 16 | REPLICA_2_MYSQL_PASSWORD=secret 17 | REPLICA_2_MYSQL_PORT=3309 18 | 19 | PHP_PORT=9000 20 | NGINX_PORT=80 21 | 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | docker/nginx/logs/* 3 | .idea -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Database Replication Setup 2 | 3 | After doing some study on highly available systems, I saw how database replication can be used 4 | to seperate read and write queries to speed them up respectively. 5 | 6 | The main DB is optimized to support faster writes (no indexes setup and ideally no read requests) 7 | and used for writes only, while the replica DB is optimized (properly indexed) for faster reads 8 | and no writes. 9 | 10 | This Repo contains a containerized implementation of database replication with one main db for writes 11 | and two replica DBs for reads, you can have as many replicas as possible, you just need to add more 12 | replica mysql containers and follow the steps to listen for the replica events, more on that later. 13 | 14 | 15 | ## Setup Containers 16 | 17 | To run this setup, you will need to have `Docker` installed and running on your system. 18 | 19 | Open your terminal in the project folder and run `cp .env.example .env` to copy the env file. 20 | 21 | Run `docker-compose up -d` and wait for the services to finish build and be available. 22 | 23 | 24 | ## Setup Replication 25 | 26 | *Assumption: You have a terminal access open at this project folder* 27 | 28 | 29 | 30 | The needed config for the WRITE and READ DBs are already setup and can be found inside the 31 | `docker/mysql` folder. 32 | 33 | ### WRITE DATABASE SETUP 34 | We will start with setting up the WRITE DB. 35 | 36 | Open the terminal for your WRITE DB and run: 37 | 38 | ```mysql 39 | docker-compose exec primary-sql bash 40 | ``` 41 | 42 | This will open up a bash command line interface for the primary-sql container, then you 43 | login to mysql, using: 44 | 45 | ```shell 46 | mysql -u root -p 47 | ``` 48 | 49 | It will prompt for your password and you provide it, if you're sticking to the env defaults, that would be the word `secret` 50 | 51 | After gaining access to the MySQL interface, copy the codes below and run it in there. 52 | 53 | ```mysql 54 | create user ‘replica’@’%’ identified by ‘password’; 55 | grant replication slave on *.* to ‘replica’@’%’; 56 | flush privileges; 57 | ``` 58 | 59 | This sets up a user that will be in charge of replication, and grants replication abilities to it. 60 | 61 | Next, you need to grab the binary log details for the WRITE database which will be used later to provision the READ database 62 | 63 | Still in the WRITE database mysql terminal, run: 64 | 65 | ```mysql 66 | use main_db; 67 | show master status; 68 | ``` 69 | This will bring out a table containing the binary log file and the position, copy this details to somewhere safe. 70 | 71 | Now exit the MySQL terminal and then exit the WRITE database terminal too entirely. We are done with the setup 72 | 73 | ### READ DATABASE SETUP 74 | 75 | Open the terminal for your WRITE DB and run: 76 | 77 | ```mysql 78 | docker-compose exec replica-sql-1 bash 79 | ``` 80 | 81 | This will open up a bash command line interface for the primary-sql container, then you 82 | login to mysql, using: 83 | 84 | ```shell 85 | mysql -u root -p 86 | ``` 87 | 88 | It will prompt for your password and you provide it, if you're sticking to the env defaults, that would be the word `secret` 89 | 90 | After gaining access to the MySQL interface, copy the codes below and run it in there. 91 | 92 | *Before running the command below, change `MASTER_LOG_FILE = 'mysql-bin.000001', MASTER_LOG_POS = 107` values to the values you got from 93 | the master setup.* 94 | 95 | ```mysql 96 | stop slave; 97 | CHANGE MASTER TO MASTER_HOST = 'primary-sql', MASTER_USER = 'replica', MASTER_PASSWORD = 'password', MASTER_LOG_FILE = 'mysql-bin.000001', MASTER_LOG_POS = 107; 98 | start slave; 99 | ``` 100 | Then run `show slave status\G;` and look out for the following parameters: 101 | 102 | ```mysql 103 | Slave_IO_State: Waiting for master to send event 104 | 105 | Master_Host: primary-sql 106 | Slave_IO_Running: Yes 107 | Slave_SQL_Running: Yes 108 | ``` 109 | 110 | If the last two parameters are not running then there is a setup error and you might need to either look for where you missed a step 111 | or check in with ChatGPT with the specific error code. 112 | 113 | That is all the setup, now the replica DB will be picking up commands made on the primary DB 114 | 115 | You can repeat this steps on the `replica-sql-2` database and as many databases you want. 116 | 117 | ## Testing 118 | 119 | Run a command on your master mysql terminal, and then check that it is ran too on your replica db. 120 | An easy example would be creating a table on the READ database and checking that same table exists on the replica table. 121 | 122 | ## The Docker File 123 | 124 | The Docker file contains 3 services which are instances of the mysql image: 125 | - primary-sql 126 | - replica-sql-1 127 | - replica-sql-2 128 | 129 | The `primary-sql` instance acts as the WRITE DB while the `replica-sql-1` and `replica-sql-2` 130 | acts as the READ DB. 131 | 132 | Each respective mysql container has a folder in the `docker/mysql` folder and contains the config 133 | files and logs folder. You can update it to suit your custom needs but the default details there are 134 | just right to get us going. 135 | 136 | 137 | Implementation Resource Aid: [MySQL DB Replication.](https://thilinamad.medium.com/mysql-db-replication-63786ac8241e) and ChatGPT 138 | 139 | 140 | ### Todo 141 | 142 | [] Add a webserver to visualize the replication -------------------------------------------------------------------------------- /database/PDOConnector.php: -------------------------------------------------------------------------------- 1 | host = $host; 13 | 14 | return $this; 15 | } 16 | 17 | public function setUserName(string $username) 18 | { 19 | $this->username = $username; 20 | 21 | return $this; 22 | } 23 | 24 | public function setPassword(string $password) 25 | { 26 | $this->password = $password; 27 | 28 | return $this; 29 | } 30 | 31 | public function setDBName(string $database) 32 | { 33 | $this->database = $database; 34 | 35 | return $this; 36 | } 37 | 38 | public function connect(array $options = []) 39 | { 40 | try { 41 | return new PDO( 42 | "mysql:host=$this->host;dbname=$this->database", 43 | $this->username, 44 | $this->password, 45 | $options 46 | ); 47 | } catch(PDOException $e) { 48 | echo "Error!:". $e->getMessage() . "
"; 49 | die(); 50 | } 51 | } 52 | } -------------------------------------------------------------------------------- /database/database-connection.php: -------------------------------------------------------------------------------- 1 | setHostName('primary-sql') 6 | ->setUserName('root') 7 | ->setPassword('secret') 8 | ->setDBName('main_db') 9 | ->connect(); 10 | 11 | $firstReplicaDBConnection = (new PDOConnector) 12 | ->setHostName('replica-sql-1') 13 | ->setUserName('user') 14 | ->setPassword('secret') 15 | ->setDBName('replica_db') 16 | ->connect(); 17 | 18 | $secondReplicaDBConnection = (new PDOConnector) 19 | ->setHostName('replica-sql-2') 20 | ->setUserName('user') 21 | ->setPassword('secret') 22 | ->setDBName('replica_db') 23 | ->connect(); 24 | -------------------------------------------------------------------------------- /database/public/index.php: -------------------------------------------------------------------------------- 1 | 4 | 5 | 6 | 7 | 8 | 9 | 11 | 12 | Database Ish 13 | 14 | 15 | 16 | query('show master status'); 18 | 19 | //var_dump($statement->fetch()); 20 | ?> 21 | 22 |
23 | 24 |
25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | primary-sql: 5 | env_file: 6 | - .env 7 | container_name: main-sql 8 | image: mysql 9 | build: 10 | context: ./docker/mysql 11 | environment: 12 | MYSQL_DATABASE: ${PRIMARY_MYSQL_DATABASE} 13 | MYSQL_ROOT_PASSWORD: ${PRIMARY_MYSQL_ROOT_PASSWORD} 14 | MYSQL_USER: ${PRIMARY_MYSQL_USER} 15 | MYSQL_PASSWORD: ${PRIMARY_MYSQL_PASSWORD} 16 | ports: 17 | - ${PRIMARY_MYSQL_PORT}:3306 18 | volumes: 19 | - ./docker/mysql/primary/my.conf:/etc/mysql/conf.d/my.cnf 20 | - ./docker/mysql/primary/logs:/var/log/mysql:rw 21 | - ./docker/mysql/dumps:/var/log/dumps:rw 22 | - main:/var/lib/mysql 23 | depends_on: 24 | - replica-sql-1 25 | - replica-sql-2 26 | 27 | replica-sql-1: 28 | env_file: 29 | - .env 30 | container_name: replica-sql-1 31 | image: mysql 32 | build: 33 | context: ./docker/mysql 34 | environment: 35 | MYSQL_DATABASE: ${REPLICA_1_MYSQL_DATABASE} 36 | MYSQL_ROOT_PASSWORD: ${REPLICA_1_MYSQL_ROOT_PASSWORD} 37 | MYSQL_USER: ${REPLICA_1_MYSQL_USER} 38 | MYSQL_PASSWORD: ${REPLICA_1_MYSQL_PASSWORD} 39 | ports: 40 | - ${REPLICA_1_MYSQL_PORT}:3306 41 | volumes: 42 | - ./docker/mysql/replica-1/my.conf:/etc/mysql/conf.d/my.cnf 43 | - ./docker/mysql/replica-1/logs:/var/log/mysql:rw 44 | - ./docker/mysql/dumps:/var/log/dumps:rw 45 | - replication-1:/var/lib/mysql 46 | 47 | replica-sql-2: 48 | env_file: 49 | - .env 50 | container_name: replica-sql-2 51 | image: mysql 52 | build: 53 | context: ./docker/mysql 54 | environment: 55 | MYSQL_DATABASE: ${REPLICA_2_MYSQL_DATABASE} 56 | MYSQL_ROOT_PASSWORD: ${REPLICA_2_MYSQL_ROOT_PASSWORD} 57 | MYSQL_USER: ${REPLICA_2_MYSQL_USER} 58 | MYSQL_PASSWORD: ${REPLICA_2_MYSQL_PASSWORD} 59 | ports: 60 | - ${REPLICA_2_MYSQL_PORT}:3306 61 | volumes: 62 | - ./docker/mysql/replica-2/my.conf:/etc/mysql/conf.d/my.cnf 63 | - ./docker/mysql/replica-2/logs:/var/log/mysql:rw 64 | - ./docker/mysql/dumps:/var/log/dumps:rw 65 | - replication-2:/var/lib/mysql 66 | 67 | web-server: 68 | env_file: 69 | - .env 70 | container_name: web-server 71 | image: 'web-server' 72 | build: 73 | context: ./docker/nginx 74 | ports: 75 | - ${NGINX_PORT}:80 76 | restart: unless-stopped 77 | depends_on: 78 | - php-fpm 79 | volumes: 80 | - ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf 81 | - ./docker/nginx/logs/:/var/log/nginx 82 | - ./database:/var/www/html/database:rw 83 | 84 | php-fpm: 85 | env_file: 86 | - .env 87 | container_name: php-srv 88 | image: php-srv 89 | build: 90 | context: ./docker/php 91 | ports: 92 | - ${PHP_PORT}:9000 93 | restart: unless-stopped 94 | depends_on: 95 | - primary-sql 96 | - replica-sql-1 97 | - replica-sql-2 98 | volumes: 99 | - ./docker/php/php.ini:/usr/local/etc/php/php.ini 100 | - ./database:/var/www/html/database:rw 101 | 102 | volumes: 103 | main: 104 | replication-1: 105 | replication-2: 106 | -------------------------------------------------------------------------------- /docker/.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elishaukpong/database-replication/81e1dcad102c6e85e7815fdf1a76df76c1276d76/docker/.gitignore -------------------------------------------------------------------------------- /docker/mysql/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mysql:8.0.19 2 | -------------------------------------------------------------------------------- /docker/mysql/dumps/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /docker/mysql/primary/logs/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /docker/mysql/primary/my.conf: -------------------------------------------------------------------------------- 1 | [mysqld] 2 | bind-address = 0.0.0.0 3 | 4 | server-id = 1 5 | log_bin = /var/log/mysql/mysql-bin.log 6 | log_bin_index =/var/log/mysql/mysql-bin.log.index 7 | relay_log = /var/log/mysql/mysql-relay-bin 8 | relay_log_index = /var/log/mysql/mysql-relay-bin.index 9 | 10 | binlog-do-db = main_db 11 | 12 | default_authentication_plugin=mysql_native_password 13 | 14 | collation-server=utf8_general_ci 15 | character-set-server=utf8 16 | max_allowed_packet=512MB -------------------------------------------------------------------------------- /docker/mysql/replica-1/logs/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore 3 | -------------------------------------------------------------------------------- /docker/mysql/replica-1/my.conf: -------------------------------------------------------------------------------- 1 | [mysqld] 2 | bind-address = 0.0.0.0 3 | 4 | server-id = 2 5 | log_bin = /var/log/mysql/mysql-bin.log 6 | log_bin_index =/var/log/mysql/mysql-bin.log.index 7 | relay_log = /var/log/mysql/mysql-relay-bin 8 | relay_log_index = /var/log/mysql/mysql-relay-bin.index 9 | 10 | binlog-do-db = replica_db 11 | 12 | replicate-do-db = main_db 13 | 14 | default_authentication_plugin=mysql_native_password 15 | 16 | collation-server=utf8_general_ci 17 | character-set-server=utf8 18 | max_allowed_packet=512MB -------------------------------------------------------------------------------- /docker/mysql/replica-2/logs/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore 3 | -------------------------------------------------------------------------------- /docker/mysql/replica-2/my.conf: -------------------------------------------------------------------------------- 1 | [mysqld] 2 | bind-address = 0.0.0.0 3 | 4 | server-id = 3 5 | log_bin = /var/log/mysql/mysql-bin.log 6 | log_bin_index =/var/log/mysql/mysql-bin.log.index 7 | relay_log = /var/log/mysql/mysql-relay-bin 8 | relay_log_index = /var/log/mysql/mysql-relay-bin.index 9 | 10 | binlog-do-db = replica_db 11 | 12 | replicate-do-db = main_db 13 | 14 | default_authentication_plugin=mysql_native_password 15 | 16 | collation-server=utf8_general_ci 17 | character-set-server=utf8 18 | max_allowed_packet=512MB -------------------------------------------------------------------------------- /docker/nginx/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM nginx:latest 2 | -------------------------------------------------------------------------------- /docker/nginx/default.conf: -------------------------------------------------------------------------------- 1 | server { 2 | listen 80 default_server; 3 | listen [::]:80 default_server; 4 | 5 | server_name database-replication.test; 6 | 7 | index index.php index.html; 8 | 9 | error_log /var/log/nginx/error.log; 10 | access_log /var/log/nginx/access.log; 11 | 12 | root /var/www/html/database/public; 13 | 14 | client_max_body_size 512M; 15 | 16 | proxy_read_timeout 600; 17 | proxy_connect_timeout 600; 18 | proxy_send_timeout 600; 19 | proxy_busy_buffers_size 512k; 20 | proxy_buffers 4 512k; 21 | proxy_buffer_size 256k; 22 | 23 | 24 | location / { 25 | try_files $uri $document_root/index.php$is_args$args; 26 | } 27 | 28 | location ~ \.php$ { 29 | fastcgi_pass php-fpm:9000; 30 | try_files $uri /index.php =404; 31 | fastcgi_split_path_info ^(.+\.php)(/.+)$; 32 | fastcgi_index index.php; 33 | fastcgi_read_timeout 600; 34 | fastcgi_buffers 16 32k; 35 | fastcgi_buffer_size 64k; 36 | fastcgi_busy_buffers_size 64k; 37 | fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; 38 | 39 | include fastcgi_params; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /docker/php/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM php:8.2-fpm 2 | 3 | # Set working directory 4 | WORKDIR /var/www 5 | 6 | USER root 7 | 8 | RUN apt update && apt -f install -y \ 9 | build-essential \ 10 | libpng-dev \ 11 | libjpeg62-turbo-dev \ 12 | libfreetype6-dev \ 13 | locales \ 14 | zip \ 15 | jpegoptim optipng pngquant gifsicle \ 16 | vim \ 17 | unzip \ 18 | git \ 19 | openssh-server \ 20 | curl \ 21 | libzip-dev \ 22 | libfontconfig1 \ 23 | libxrender1 \ 24 | libpng-dev \ 25 | make \ 26 | nano \ 27 | iputils-ping \ 28 | ssh \ 29 | sudo 30 | 31 | # Install php extensions 32 | RUN apt -f install -y libgmp-dev re2c libmhash-dev libmcrypt-dev file libtidy-dev \ 33 | && ln -s /usr/include/x86_64-linux-gnu/gmp.h /usr/local/include/ \ 34 | && docker-php-ext-configure gmp \ 35 | && docker-php-ext-install gmp \ 36 | && docker-php-ext-install pdo_mysql zip exif pcntl bcmath \ 37 | && docker-php-ext-configure gd --with-freetype --with-jpeg \ 38 | && docker-php-ext-install gd \ 39 | && docker-php-ext-install tidy \ 40 | && docker-php-ext-configure tidy \ 41 | && docker-php-ext-enable tidy 42 | 43 | # Install composer 44 | RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer 45 | 46 | # Verify installation 47 | RUN composer --version 48 | 49 | # sudo www-data 50 | RUN echo "www-data ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 51 | 52 | USER www-data 53 | -------------------------------------------------------------------------------- /docker/php/php.ini: -------------------------------------------------------------------------------- 1 | [PHP] 2 | upload_max_filesize=512M 3 | post_max_size=512M 4 | memory_limit=2048M 5 | max_execution_time=1000 6 | pcre.backtrack_limit=1000000 7 | variables_order = EGPCS 8 | --------------------------------------------------------------------------------