├── .gitignore ├── .travis.yml ├── AUTHORS ├── LICENSE ├── Makefile ├── README.md ├── chlog-gen.sh ├── config.go ├── daemon.go ├── database.go ├── debian ├── changelog ├── compat ├── control ├── copyright ├── rules └── source │ └── format ├── docker-compose.yml ├── docker ├── pgcheck-plproxy │ ├── Dockerfile │ └── entrypoint.sh └── pgcheck-postgres │ ├── Dockerfile │ ├── entrypoint.sh │ ├── fake.sql │ ├── pg.conf │ ├── pg_hba.conf │ ├── pgbouncer.ini │ ├── trusted_keys │ └── pgdg.gpg │ └── userlist.txt ├── host.go ├── http.go ├── http_test.go ├── pgcheck.go ├── pgcheck_test.go ├── priority.go ├── priority_test.go ├── samples ├── etc │ └── pgcheck.yml └── sql │ ├── 00_pgproxy.sql │ ├── 30_get_cluster_partitions.sql │ ├── 30_get_cluster_version.sql │ ├── 30_inc_cluster_version.sql │ ├── 50_is_master.sql │ ├── 50_select_part.sql │ └── 99_data.sql └── tests ├── environment.py ├── features └── main.feature ├── pgcheck.featureset ├── requirements.txt └── steps ├── __init__.py ├── database.py ├── helpers.py ├── main.py └── moby.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyo 2 | *.pyc 3 | *.sw[o-s] 4 | pgcheck 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | sudo: required 3 | cache: false 4 | go: 5 | - "1.11" 6 | services: 7 | - docker 8 | before_install: 9 | - sudo pip install -r tests/requirements.txt 10 | script: 11 | - go get github.com/lib/pq github.com/spf13/viper github.com/stretchr/testify/assert 12 | - go test 13 | - make docker 14 | - make check-world 15 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | (C) YANDEX LLC, 2014-2019 2 | 3 | People that contributed to it: 4 | 5 | Vladimir Borodin 6 | Evgeny Dyukov 7 | Sergey Lavrinenko 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | (C) YANDEX LLC, 2014-2019 2 | 3 | The Source Code called "pgcheck" available at 4 | https://github.com/yandex/pgcheck is subject to the terms of 5 | the GNU Lesser General Public License, v. 3.0 (hereinafter - LGPL). 6 | The text of the LGPL is the following: 7 | 8 | GNU LESSER GENERAL PUBLIC LICENSE 9 | Version 3, 29 June 2007 10 | 11 | Copyright (C) 2007 Free Software Foundation, Inc. 12 | Everyone is permitted to copy and distribute verbatim copies 13 | of this license document, but changing it is not allowed. 14 | 15 | 16 | This version of the GNU Lesser General Public License incorporates 17 | the terms and conditions of version 3 of the GNU General Public 18 | License, supplemented by the additional permissions listed below. 19 | 20 | 0. Additional Definitions. 21 | 22 | As used herein, "this License" refers to version 3 of the GNU Lesser 23 | General Public License, and the "GNU GPL" refers to version 3 of the GNU 24 | General Public License. 25 | 26 | "The Library" refers to a covered work governed by this License, 27 | other than an Application or a Combined Work as defined below. 28 | 29 | An "Application" is any work that makes use of an interface provided 30 | by the Library, but which is not otherwise based on the Library. 31 | Defining a subclass of a class defined by the Library is deemed a mode 32 | of using an interface provided by the Library. 33 | 34 | A "Combined Work" is a work produced by combining or linking an 35 | Application with the Library. The particular version of the Library 36 | with which the Combined Work was made is also called the "Linked 37 | Version". 38 | 39 | The "Minimal Corresponding Source" for a Combined Work means the 40 | Corresponding Source for the Combined Work, excluding any source code 41 | for portions of the Combined Work that, considered in isolation, are 42 | based on the Application, and not on the Linked Version. 43 | 44 | The "Corresponding Application Code" for a Combined Work means the 45 | object code and/or source code for the Application, including any data 46 | and utility programs needed for reproducing the Combined Work from the 47 | Application, but excluding the System Libraries of the Combined Work. 48 | 49 | 1. Exception to Section 3 of the GNU GPL. 50 | 51 | You may convey a covered work under sections 3 and 4 of this License 52 | without being bound by section 3 of the GNU GPL. 53 | 54 | 2. Conveying Modified Versions. 55 | 56 | If you modify a copy of the Library, and, in your modifications, a 57 | facility refers to a function or data to be supplied by an Application 58 | that uses the facility (other than as an argument passed when the 59 | facility is invoked), then you may convey a copy of the modified 60 | version: 61 | 62 | a) under this License, provided that you make a good faith effort to 63 | ensure that, in the event an Application does not supply the 64 | function or data, the facility still operates, and performs 65 | whatever part of its purpose remains meaningful, or 66 | 67 | b) under the GNU GPL, with none of the additional permissions of 68 | this License applicable to that copy. 69 | 70 | 3. Object Code Incorporating Material from Library Header Files. 71 | 72 | The object code form of an Application may incorporate material from 73 | a header file that is part of the Library. You may convey such object 74 | code under terms of your choice, provided that, if the incorporated 75 | material is not limited to numerical parameters, data structure 76 | layouts and accessors, or small macros, inline functions and templates 77 | (ten or fewer lines in length), you do both of the following: 78 | 79 | a) Give prominent notice with each copy of the object code that the 80 | Library is used in it and that the Library and its use are 81 | covered by this License. 82 | 83 | b) Accompany the object code with a copy of the GNU GPL and this license 84 | document. 85 | 86 | 4. Combined Works. 87 | 88 | You may convey a Combined Work under terms of your choice that, 89 | taken together, effectively do not restrict modification of the 90 | portions of the Library contained in the Combined Work and reverse 91 | engineering for debugging such modifications, if you also do each of 92 | the following: 93 | 94 | a) Give prominent notice with each copy of the Combined Work that 95 | the Library is used in it and that the Library and its use are 96 | covered by this License. 97 | 98 | b) Accompany the Combined Work with a copy of the GNU GPL and this license 99 | document. 100 | 101 | c) For a Combined Work that displays copyright notices during 102 | execution, include the copyright notice for the Library among 103 | these notices, as well as a reference directing the user to the 104 | copies of the GNU GPL and this license document. 105 | 106 | d) Do one of the following: 107 | 108 | 0) Convey the Minimal Corresponding Source under the terms of this 109 | License, and the Corresponding Application Code in a form 110 | suitable for, and under terms that permit, the user to 111 | recombine or relink the Application with a modified version of 112 | the Linked Version to produce a modified Combined Work, in the 113 | manner specified by section 6 of the GNU GPL for conveying 114 | Corresponding Source. 115 | 116 | 1) Use a suitable shared library mechanism for linking with the 117 | Library. A suitable mechanism is one that (a) uses at run time 118 | a copy of the Library already present on the user's computer 119 | system, and (b) will operate properly with a modified version 120 | of the Library that is interface-compatible with the Linked 121 | Version. 122 | 123 | e) Provide Installation Information, but only if you would otherwise 124 | be required to provide such information under section 6 of the 125 | GNU GPL, and only to the extent that such information is 126 | necessary to install and execute a modified version of the 127 | Combined Work produced by recombining or relinking the 128 | Application with a modified version of the Linked Version. (If 129 | you use option 4d0, the Installation Information must accompany 130 | the Minimal Corresponding Source and Corresponding Application 131 | Code. If you use option 4d1, you must provide the Installation 132 | Information in the manner specified by section 6 of the GNU GPL 133 | for conveying Corresponding Source.) 134 | 135 | 5. Combined Libraries. 136 | 137 | You may place library facilities that are a work based on the 138 | Library side by side in a single library together with other library 139 | facilities that are not Applications and are not covered by this 140 | License, and convey such a combined library under terms of your 141 | choice, if you do both of the following: 142 | 143 | a) Accompany the combined library with a copy of the same work based 144 | on the Library, uncombined with any other library facilities, 145 | conveyed under the terms of this License. 146 | 147 | b) Give prominent notice with the combined library that part of it 148 | is a work based on the Library, and explaining where to find the 149 | accompanying uncombined form of the same work. 150 | 151 | 6. Revised Versions of the GNU Lesser General Public License. 152 | 153 | The Free Software Foundation may publish revised and/or new versions 154 | of the GNU Lesser General Public License from time to time. Such new 155 | versions will be similar in spirit to the present version, but may 156 | differ in detail to address new problems or concerns. 157 | 158 | Each version is given a distinguishing version number. If the 159 | Library as you received it specifies that a certain numbered version 160 | of the GNU Lesser General Public License "or any later version" 161 | applies to it, you have the option of following the terms and 162 | conditions either of that published version or of any later version 163 | published by the Free Software Foundation. If the Library as you 164 | received it does not specify a version number of the GNU Lesser 165 | General Public License, you may choose any version of the GNU Lesser 166 | General Public License ever published by the Free Software Foundation. 167 | 168 | If the Library as you received it specifies that a proxy can decide 169 | whether future versions of the GNU Lesser General Public License shall 170 | apply, that proxy's public statement of acceptance of any version is 171 | permanent authorization for you to choose that version for the 172 | Library. 173 | 174 | A copy of the LGPL is also available at https://www.gnu.org/licenses/lgpl.html. 175 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: clean all 2 | 3 | all: unit-tests docker check 4 | 5 | build: 6 | GOOS=linux GOARCH=amd64 go build -o pgcheck 7 | 8 | unit-tests: 9 | go test 10 | 11 | unit: unit-tests 12 | 13 | docker-build: 14 | docker build --rm --no-cache -t pgcheck-postgres docker/pgcheck-postgres/ 15 | docker build --rm --no-cache -t pgcheck-plproxy docker/pgcheck-plproxy/ 16 | 17 | docker-env: 18 | docker-compose down 19 | docker-compose up -d 20 | 21 | check: 22 | behave --show-timings --stop --tags=-long @tests/pgcheck.featureset 23 | 24 | check-world: 25 | behave --show-timings --stop @tests/pgcheck.featureset 26 | 27 | docker: build docker-build docker-env 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Pgcheck 2 | [![Go Report Card](https://goreportcard.com/badge/github.com/yandex/pgcheck)](https://goreportcard.com/report/github.com/yandex/pgcheck) 3 | [![Build Status](https://travis-ci.org/yandex/pgcheck.svg?branch=master)](https://travis-ci.org/yandex/pgcheck) 4 | 5 | Tool for monitoring backend databases from PL/Proxy hosts and changing `plproxy.get_cluster_partitions` function output. 6 | 7 | ## How does it work? 8 | 9 | Pgcheck checks health status of each host of PostgreSQL clusters and assigns them priorities. Right now the assigned priorities are: 10 | * `0` - the master, 11 | * `10` - asynchronous replica in the same (as plproxy-host) datacenter, 12 | * `20` - asynchronous replica in any other datacenter, 13 | * `100` - dead hosts. 14 | 15 | In our environment plproxy-host, when taking decision where to route the query, by default takes host with the lowest priority. So in general all queries go to the master. If it fails, the queries are routed to one of the replicas. This gives us read-only degradation in case of master fail. 16 | 17 | If you set `replics_weights = yes` in config-file, replics priorities diffs would be calculated depending on its load. The resulting priority is increase by this diff. The load of the replica will be calculated depending on PostgreSQL client connections (all connections from pg_stat_activity, not only in `active` state). 18 | 19 | If you set `account_replication_lag = yes`, replics priorities would be also increased by one for each second of replication replay delay. Replication delay in seconds is measured with [repl_mon](https://github.com/dev1ant/repl_mon). 20 | 21 | In our environment information about shards, hosts and their priorities is kept in special tables (you can see sqls for creating them in `samples/sql` directory). Pgcheck created a goroutine for each cluster defined in config-file. 22 | The loop inside the goroutine is executed every `iteration_timeout` and it refreshes the value for field `priority` in the `priorities` table and assigns next values: 23 | * `0` if the host is master, 24 | * `100` if the host is dead, in any case, 25 | * `calculated_prio+prio_diff`, if the host is replica and alive. 26 | Because of network flaps frequent changes of priorities may occur, so there are parameters `quorum` and `hysterisis` for every cluster. Details for them are below. 27 | 28 | The `prio_diff` field in table `hosts` is taken in account when assigning current priorities for replics. It may be needed for changing priority of any host by hand. 29 | 30 | ## Installation 31 | 32 | Compile the binary with standard `go build` and install it where you want. 33 | The binary will be statically linked, so without any dependencies but it will not create database(s) with needed tables and functions and it will not install needed config-files. Samples for sql-files (creating needed schema and functions) and config files can be found in the `samples` directory. In our environment they come from different package and are managed by our SCM. 34 | 35 | You could also see an example of deploying pgcheck in tests infrastructure. 36 | 37 | ## Config-file 38 | 39 | Sample config file can be found in `samples/etc/pgcheck.yml`. All the parameters have comments with explanations except for two of them - `quorum` and `hysteris`. 40 | 41 | For each host in memory there will be stored information about `quorum+hysterisis` last priorities. If `quorum` of them are the same as the last one, it will be assigned. For example, if `quorum = 3` and `hysterisis = 2`, there will be floating window of 5 last priorities and the logic will be next: 42 | * `[0, 100, 0, 100, 100]` - priority `100` will be assigned, 43 | * `[100, 100, 0, 100, 0]` - priority will not be changed, 44 | * `[100, 100, 0, 0, 0]` - priority will be changed to `0`, 45 | * `[100, 100, 0, 0, 100]` - priority will be changed to `100`. 46 | -------------------------------------------------------------------------------- /chlog-gen.sh: -------------------------------------------------------------------------------- 1 | cat > debian/changelog< $(date +%a\,\ %d\ %b\ %Y\ %H:%M:%S\ %z) 7 | EOH 8 | -------------------------------------------------------------------------------- /config.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/spf13/viper" 7 | ) 8 | 9 | // 10 | // We declare all types here and all fields in them exportable 11 | // so that viper module could unmarshal to them 12 | // 13 | 14 | // Config is (surprise) for storing configuration 15 | type Config struct { 16 | LogFile string `mapstructure:"log_file"` 17 | DC string `mapstructure:"my_dc"` 18 | Timeout int `mapstructure:"iteration_timeout"` 19 | Databases map[string]DBConfig 20 | } 21 | 22 | // DBConfig stores config of a single DB 23 | type DBConfig struct { 24 | LocalConnString string `mapstructure:"local_conn_string"` 25 | AppendConnString string `mapstructure:"append_conn_string"` 26 | Quorum uint 27 | Hysterisis uint 28 | } 29 | 30 | func parseConfig() Config { 31 | viper.SetConfigName("pgcheck") 32 | viper.SetConfigType("yaml") 33 | viper.AddConfigPath("/etc/") 34 | err := viper.ReadInConfig() 35 | if err != nil { 36 | log.Fatal(err) 37 | } 38 | 39 | var config Config 40 | err = viper.Unmarshal(&config) 41 | if err != nil { 42 | log.Fatal(err) 43 | } 44 | return config 45 | } 46 | -------------------------------------------------------------------------------- /daemon.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | "os" 6 | "os/signal" 7 | "syscall" 8 | ) 9 | 10 | func initLogging(config *Config) { 11 | if config.LogFile == "" { 12 | return 13 | } 14 | 15 | f, err := os.OpenFile(config.LogFile, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644) 16 | if err != nil { 17 | log.Fatal("Error opening file: ", err) 18 | } 19 | 20 | log.SetOutput(f) 21 | } 22 | 23 | func handleSignals() { 24 | sigChannel := make(chan os.Signal) 25 | signal.Notify(sigChannel, 26 | syscall.SIGHUP, 27 | syscall.SIGINT, 28 | syscall.SIGTERM, 29 | syscall.SIGQUIT, 30 | ) 31 | for { 32 | sig := <-sigChannel 33 | log.Println("Got signal: ", sig) 34 | // TODO: handle signals properly 35 | if sig != syscall.SIGHUP { 36 | break 37 | } 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /database.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "log" 6 | "time" 7 | ) 8 | 9 | type database struct { 10 | name string 11 | config DBConfig 12 | pool *sql.DB 13 | } 14 | 15 | func createPool(connStr *string, wait bool) (*sql.DB, error) { 16 | for { 17 | db, err := sql.Open("postgres", *connStr) 18 | if err == nil { 19 | db.SetConnMaxLifetime(time.Hour) 20 | db.SetMaxIdleConns(5) 21 | return db, err 22 | } 23 | 24 | if !wait { 25 | return nil, err 26 | } 27 | 28 | log.Printf("Connection to '%s' failed: %s", *connStr, err) 29 | time.Sleep(time.Second) 30 | defer db.Close() 31 | } 32 | } 33 | 34 | func getPool(host *host) (*sql.DB, error) { 35 | var db *sql.DB 36 | var err error 37 | if host.connectionPool != nil { 38 | // Reuse already created connection pool if possible 39 | return host.connectionPool, nil 40 | } 41 | 42 | db, err = createPool(&host.connStr, false) 43 | if err != nil { 44 | return nil, err 45 | } 46 | host.connectionPool = db 47 | return db, nil 48 | } 49 | -------------------------------------------------------------------------------- /debian/changelog: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yandex/pgcheck/977c96962f9e69bab655f18a41f6a05d5f6b292f/debian/changelog -------------------------------------------------------------------------------- /debian/compat: -------------------------------------------------------------------------------- 1 | 9 2 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Source: pgcheck 2 | Section: unknown 3 | Priority: optional 4 | Maintainer: Vladimir Borodin 5 | Build-Depends: debhelper (>= 8.0.0) 6 | Standards-Version: 3.9.4 7 | Homepage: https://github.com/yandex/pgcheck 8 | #Vcs-Git: git://github.com:yandex/pgcheck.git 9 | 10 | Package: pgcheck 11 | Architecture: any 12 | Depends: ${misc:Depends}, ${python:Depends}, python-psycopg2, python-daemon, python-lockfile, python-requests 13 | Description: Tool for changing PL/Proxy hosts behaviour. 14 | -------------------------------------------------------------------------------- /debian/copyright: -------------------------------------------------------------------------------- 1 | Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ 2 | Upstream-Name: pgcheck 3 | Source: 4 | 5 | Files: * 6 | Copyright: 2014-2019 Vladimir Borodin 7 | License: LGPL-3 8 | 9 | License: LGPL-3 10 | GNU LESSER GENERAL PUBLIC LICENSE 11 | Version 3, 29 June 2007 12 | . 13 | Copyright (C) 2007 Free Software Foundation, Inc. 14 | Everyone is permitted to copy and distribute verbatim copies 15 | of this license document, but changing it is not allowed. 16 | . 17 | This version of the GNU Lesser General Public License incorporates 18 | the terms and conditions of version 3 of the GNU General Public 19 | License, supplemented by the additional permissions listed below. 20 | . 21 | 0. Additional Definitions. 22 | . 23 | As used herein, "this License" refers to version 3 of the GNU Lesser 24 | General Public License, and the "GNU GPL" refers to version 3 of the GNU 25 | General Public License. 26 | . 27 | "The Library" refers to a covered work governed by this License, 28 | other than an Application or a Combined Work as defined below. 29 | . 30 | An "Application" is any work that makes use of an interface provided 31 | by the Library, but which is not otherwise based on the Library. 32 | Defining a subclass of a class defined by the Library is deemed a mode 33 | of using an interface provided by the Library. 34 | . 35 | A "Combined Work" is a work produced by combining or linking an 36 | Application with the Library. The particular version of the Library 37 | with which the Combined Work was made is also called the "Linked 38 | Version". 39 | . 40 | The "Minimal Corresponding Source" for a Combined Work means the 41 | Corresponding Source for the Combined Work, excluding any source code 42 | for portions of the Combined Work that, considered in isolation, are 43 | based on the Application, and not on the Linked Version. 44 | . 45 | The "Corresponding Application Code" for a Combined Work means the 46 | object code and/or source code for the Application, including any data 47 | and utility programs needed for reproducing the Combined Work from the 48 | Application, but excluding the System Libraries of the Combined Work. 49 | . 50 | 1. Exception to Section 3 of the GNU GPL. 51 | . 52 | You may convey a covered work under sections 3 and 4 of this License 53 | without being bound by section 3 of the GNU GPL. 54 | . 55 | 2. Conveying Modified Versions. 56 | . 57 | If you modify a copy of the Library, and, in your modifications, a 58 | facility refers to a function or data to be supplied by an Application 59 | that uses the facility (other than as an argument passed when the 60 | facility is invoked), then you may convey a copy of the modified 61 | version: 62 | . 63 | a) under this License, provided that you make a good faith effort to 64 | ensure that, in the event an Application does not supply the 65 | function or data, the facility still operates, and performs 66 | whatever part of its purpose remains meaningful, or 67 | . 68 | b) under the GNU GPL, with none of the additional permissions of 69 | this License applicable to that copy. 70 | . 71 | 3. Object Code Incorporating Material from Library Header Files. 72 | . 73 | The object code form of an Application may incorporate material from 74 | a header file that is part of the Library. You may convey such object 75 | code under terms of your choice, provided that, if the incorporated 76 | material is not limited to numerical parameters, data structure 77 | layouts and accessors, or small macros, inline functions and templates 78 | (ten or fewer lines in length), you do both of the following: 79 | . 80 | a) Give prominent notice with each copy of the object code that the 81 | Library is used in it and that the Library and its use are 82 | covered by this License. 83 | . 84 | b) Accompany the object code with a copy of the GNU GPL and this license 85 | document. 86 | . 87 | 4. Combined Works. 88 | . 89 | You may convey a Combined Work under terms of your choice that, 90 | taken together, effectively do not restrict modification of the 91 | portions of the Library contained in the Combined Work and reverse 92 | engineering for debugging such modifications, if you also do each of 93 | the following: 94 | . 95 | a) Give prominent notice with each copy of the Combined Work that 96 | the Library is used in it and that the Library and its use are 97 | covered by this License. 98 | . 99 | b) Accompany the Combined Work with a copy of the GNU GPL and this license 100 | document. 101 | . 102 | c) For a Combined Work that displays copyright notices during 103 | execution, include the copyright notice for the Library among 104 | these notices, as well as a reference directing the user to the 105 | copies of the GNU GPL and this license document. 106 | . 107 | d) Do one of the following: 108 | . 109 | 0) Convey the Minimal Corresponding Source under the terms of this 110 | License, and the Corresponding Application Code in a form 111 | suitable for, and under terms that permit, the user to 112 | recombine or relink the Application with a modified version of 113 | the Linked Version to produce a modified Combined Work, in the 114 | manner specified by section 6 of the GNU GPL for conveying 115 | Corresponding Source. 116 | . 117 | 1) Use a suitable shared library mechanism for linking with the 118 | Library. A suitable mechanism is one that (a) uses at run time 119 | a copy of the Library already present on the user's computer 120 | system, and (b) will operate properly with a modified version 121 | of the Library that is interface-compatible with the Linked 122 | Version. 123 | . 124 | e) Provide Installation Information, but only if you would otherwise 125 | be required to provide such information under section 6 of the 126 | GNU GPL, and only to the extent that such information is 127 | necessary to install and execute a modified version of the 128 | Combined Work produced by recombining or relinking the 129 | Application with a modified version of the Linked Version. (If 130 | you use option 4d0, the Installation Information must accompany 131 | the Minimal Corresponding Source and Corresponding Application 132 | Code. If you use option 4d1, you must provide the Installation 133 | Information in the manner specified by section 6 of the GNU GPL 134 | for conveying Corresponding Source.) 135 | . 136 | 5. Combined Libraries. 137 | . 138 | You may place library facilities that are a work based on the 139 | Library side by side in a single library together with other library 140 | facilities that are not Applications and are not covered by this 141 | License, and convey such a combined library under terms of your 142 | choice, if you do both of the following: 143 | . 144 | a) Accompany the combined library with a copy of the same work based 145 | on the Library, uncombined with any other library facilities, 146 | conveyed under the terms of this License. 147 | . 148 | b) Give prominent notice with the combined library that part of it 149 | is a work based on the Library, and explaining where to find the 150 | accompanying uncombined form of the same work. 151 | . 152 | 6. Revised Versions of the GNU Lesser General Public License. 153 | . 154 | The Free Software Foundation may publish revised and/or new versions 155 | of the GNU Lesser General Public License from time to time. Such new 156 | versions will be similar in spirit to the present version, but may 157 | differ in detail to address new problems or concerns. 158 | . 159 | Each version is given a distinguishing version number. If the 160 | Library as you received it specifies that a certain numbered version 161 | of the GNU Lesser General Public License "or any later version" 162 | applies to it, you have the option of following the terms and 163 | conditions either of that published version or of any later version 164 | published by the Free Software Foundation. If the Library as you 165 | received it does not specify a version number of the GNU Lesser 166 | General Public License, you may choose any version of the GNU Lesser 167 | General Public License ever published by the Free Software Foundation. 168 | . 169 | If the Library as you received it specifies that a proxy can decide 170 | whether future versions of the GNU Lesser General Public License shall 171 | apply, that proxy's public statement of acceptance of any version is 172 | permanent authorization for you to choose that version for the 173 | Library. 174 | . 175 | A copy of the LGPL is also available at https://www.gnu.org/licenses/lgpl.html. 176 | . 177 | On Debian systems, the complete text of the GNU Lesser General 178 | Public License version 3 can be found in "/usr/share/common-licenses/LGPL-3". 179 | -------------------------------------------------------------------------------- /debian/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | # -*- makefile -*- 3 | 4 | # FIXME: debian/links is an ugly way to simulate python-support 5 | 6 | %: 7 | dh $@ 8 | 9 | override_dh_usrlocal: 10 | -------------------------------------------------------------------------------- /debian/source/format: -------------------------------------------------------------------------------- 1 | 3.0 (native) 2 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2.2' 2 | 3 | services: 4 | shard01-dc1: 5 | image: pgcheck-postgres 6 | ports: 7 | - "5432" 8 | - "6432" 9 | hostname: shard01-dc1 10 | domainname: pgcheck.net 11 | init: true 12 | extra_hosts: 13 | - "shard01-dc2.pgcheck.net:192.168.233.11" 14 | - "shard01-dc3.pgcheck.net:192.168.233.12" 15 | networks: 16 | net: 17 | ipv4_address: 192.168.233.10 18 | shard01-dc2: 19 | image: pgcheck-postgres 20 | ports: 21 | - "5432" 22 | - "6432" 23 | command: [ "replica", "shard01-dc1.pgcheck.net" ] 24 | hostname: shard01-dc2 25 | domainname: pgcheck.net 26 | init: true 27 | extra_hosts: 28 | - "shard01-dc1.pgcheck.net:192.168.233.10" 29 | - "shard01-dc3.pgcheck.net:192.168.233.12" 30 | networks: 31 | net: 32 | ipv4_address: 192.168.233.11 33 | shard01-dc3: 34 | image: pgcheck-postgres 35 | ports: 36 | - "5432" 37 | - "6432" 38 | command: [ "replica", "shard01-dc1.pgcheck.net" ] 39 | hostname: shard01-dc3 40 | domainname: pgcheck.net 41 | init: true 42 | extra_hosts: 43 | - "shard01-dc1.pgcheck.net:192.168.233.10" 44 | - "shard01-dc2.pgcheck.net:192.168.233.11" 45 | networks: 46 | net: 47 | ipv4_address: 192.168.233.12 48 | 49 | shard02-dc1: 50 | image: pgcheck-postgres 51 | ports: 52 | - "5432" 53 | - "6432" 54 | command: [ "replica", "shard02-dc2.pgcheck.net" ] 55 | hostname: shard02-dc1 56 | domainname: pgcheck.net 57 | init: true 58 | extra_hosts: 59 | - "shard02-dc2.pgcheck.net:192.168.233.14" 60 | - "shard02-dc3.pgcheck.net:192.168.233.15" 61 | networks: 62 | net: 63 | ipv4_address: 192.168.233.13 64 | shard02-dc2: 65 | image: pgcheck-postgres 66 | ports: 67 | - "5432" 68 | - "6432" 69 | hostname: shard02-dc2 70 | domainname: pgcheck.net 71 | init: true 72 | extra_hosts: 73 | - "shard02-dc1.pgcheck.net:192.168.233.13" 74 | - "shard02-dc3.pgcheck.net:192.168.233.15" 75 | networks: 76 | net: 77 | ipv4_address: 192.168.233.14 78 | shard02-dc3: 79 | image: pgcheck-postgres 80 | ports: 81 | - "5432" 82 | - "6432" 83 | command: [ "replica", "shard02-dc2.pgcheck.net" ] 84 | hostname: shard02-dc3 85 | domainname: pgcheck.net 86 | init: true 87 | extra_hosts: 88 | - "shard02-dc1.pgcheck.net:192.168.233.13" 89 | - "shard02-dc2.pgcheck.net:192.168.233.14" 90 | networks: 91 | net: 92 | ipv4_address: 192.168.233.15 93 | 94 | plproxy: 95 | image: pgcheck-plproxy 96 | volumes: 97 | - ./:/tmp/ 98 | ports: 99 | - "6432" 100 | - "8080" 101 | hostname: plproxy 102 | domainname: pgcheck.net 103 | init: true 104 | extra_hosts: 105 | - "shard01-dc1.pgcheck.net:192.168.233.10" 106 | - "shard01-dc2.pgcheck.net:192.168.233.11" 107 | - "shard01-dc3.pgcheck.net:192.168.233.12" 108 | - "shard02-dc1.pgcheck.net:192.168.233.13" 109 | - "shard02-dc2.pgcheck.net:192.168.233.14" 110 | - "shard02-dc3.pgcheck.net:192.168.233.15" 111 | networks: 112 | net: 113 | ipv4_address: 192.168.233.16 114 | 115 | networks: 116 | net: 117 | driver: bridge 118 | ipam: 119 | driver: default 120 | config: 121 | - subnet: 192.168.233.0/24 122 | gateway: 192.168.233.1 123 | -------------------------------------------------------------------------------- /docker/pgcheck-plproxy/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM pgcheck-postgres 2 | MAINTAINER Vladimir Borodin 3 | 4 | ARG pg_version=10 5 | 6 | ENV DEBIAN_FRONTEND noninteractive 7 | ENV PG_VERSION ${pg_version} 8 | 9 | USER root 10 | 11 | COPY entrypoint.sh /entrypoint.sh 12 | 13 | RUN apt-get update -qq \ 14 | && apt-get install -y \ 15 | postgresql-${pg_version}-plproxy 16 | 17 | EXPOSE 6432 18 | 19 | USER postgres 20 | 21 | CMD ["master"] 22 | ENTRYPOINT ["/entrypoint.sh"] 23 | -------------------------------------------------------------------------------- /docker/pgcheck-plproxy/entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | if [ "$1" = 'bash' ] 3 | then 4 | /bin/bash 5 | elif [ "$1" = 'master' ] 6 | then 7 | sudo cp /tmp/samples/etc/pgcheck.yml /etc/pgcheck.yml 8 | sudo cp /tmp/samples/etc/pgcheck-hosts.json /etc/pgcheck-hosts.json 9 | sudo mkdir -p /var/log/pgcheck /var/run/pgcheck 10 | sudo chown postgres /var/log/pgcheck /var/run/pgcheck 11 | 12 | pgbouncer -d /etc/pgbouncer/pgbouncer.ini 13 | sudo pg_ctlcluster 10 main start 14 | while ! psql -c "select 1" > /dev/null 15 | do 16 | echo "PostgreSQL has not started yet. Sleeping for a second." 17 | sleep 1 18 | done 19 | 20 | psql -c "CREATE DATABASE db1" 21 | psql db1 -f /tmp/samples/sql/00_pgproxy.sql 22 | psql db1 -f /tmp/samples/sql/30_get_cluster_partitions.sql 23 | psql db1 -f /tmp/samples/sql/30_get_cluster_version.sql 24 | psql db1 -f /tmp/samples/sql/30_inc_cluster_version.sql 25 | psql db1 -f /tmp/samples/sql/50_is_master.sql 26 | psql db1 -f /tmp/samples/sql/50_select_part.sql 27 | psql db1 -f /tmp/samples/sql/99_data.sql 28 | 29 | /tmp/pgcheck 30 | else 31 | eval "$@" 32 | fi 33 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:xenial 2 | MAINTAINER Vladimir Borodin 3 | 4 | ARG pg_version=10 5 | 6 | ENV DEBIAN_FRONTEND noninteractive 7 | ENV PG_VERSION ${pg_version} 8 | 9 | COPY trusted_keys /tmp/trusted_keys 10 | 11 | RUN apt-key add /tmp/trusted_keys/pgdg.gpg \ 12 | && rm -rf /tmp/trusted_keys \ 13 | && echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ 14 | && apt-get update -qq \ 15 | && apt-get install -y \ 16 | postgresql-${pg_version} \ 17 | postgresql-contrib-${pg_version} \ 18 | pgbouncer \ 19 | sudo \ 20 | && echo "postgres ALL=(ALL) NOPASSWD: ALL" >>/etc/sudoers 21 | 22 | COPY entrypoint.sh /entrypoint.sh 23 | COPY pg.conf /etc/postgresql/${pg_version}/main/ 24 | COPY pg_hba.conf /etc/postgresql/${pg_version}/main/ 25 | COPY pgbouncer.ini /etc/pgbouncer/pgbouncer.ini 26 | COPY userlist.txt /etc/pgbouncer/userlist.txt 27 | COPY fake.sql /tmp/fake.sql 28 | RUN echo "include 'pg.conf'" >> /etc/postgresql/${pg_version}/main/postgresql.conf 29 | 30 | EXPOSE 5432 6432 31 | 32 | USER postgres 33 | 34 | CMD ["master"] 35 | ENTRYPOINT ["/entrypoint.sh"] 36 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | if [ "$1" = 'bash' ] 3 | then 4 | /bin/bash 5 | elif [ "$1" = 'master' ] 6 | then 7 | pgbouncer -d /etc/pgbouncer/pgbouncer.ini 8 | sudo pg_ctlcluster 10 main start 9 | while ! psql -c "select 1" > /dev/null 10 | do 11 | echo "PostgreSQL has not started yet. Sleeping for a second." 12 | sleep 1 13 | done 14 | psql -c "CREATE DATABASE db1" 15 | psql db1 -f /tmp/fake.sql 16 | # Loop is needed for generating some write activity for fake repl_mon 17 | while true; do 18 | psql db1 -c "INSERT INTO tmp_table VALUES (current_timestamp)" >/dev/null 19 | sleep 3 20 | done 21 | elif [ "$1" = 'replica' ] 22 | then 23 | while ! psql -h $2 -c "select 1" > /dev/null 24 | do 25 | echo "Master has not started yet. Sleeping for a second." 26 | sleep 1 27 | done; 28 | pgbouncer -d /etc/pgbouncer/pgbouncer.ini 29 | rm -rf /var/lib/postgresql/${PG_VERSION}/main/ 30 | /usr/lib/postgresql/${PG_VERSION}/bin/pg_basebackup \ 31 | -D /var/lib/postgresql/${PG_VERSION}/main/ \ 32 | --write-recovery-conf \ 33 | --wal-method=fetch \ 34 | -h $2 35 | /usr/lib/postgresql/${PG_VERSION}/bin/postgres \ 36 | -D /var/lib/postgresql/${PG_VERSION}/main/ \ 37 | -c config_file=/etc/postgresql/${PG_VERSION}/main/postgresql.conf 38 | else 39 | eval "$@" 40 | fi 41 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/fake.sql: -------------------------------------------------------------------------------- 1 | CREATE FUNCTION public.pgcheck_poll( 2 | OUT is_master boolean, 3 | OUT lag integer, 4 | OUT sessions_ratio float 5 | ) AS $$ 6 | DECLARE 7 | closed boolean; 8 | sessions integer; 9 | BEGIN 10 | closed := current_setting('pgcheck.closed', true); 11 | IF closed IS true THEN 12 | RAISE EXCEPTION 'Database is closed from load (pgcheck.closed = %)', closed; 13 | END IF; 14 | 15 | SELECT NOT pg_is_in_recovery() 16 | INTO is_master; 17 | 18 | IF is_master THEN 19 | SELECT 0 INTO lag; 20 | ELSE 21 | SELECT ROUND(extract(epoch FROM clock_timestamp() - ts))::integer 22 | FROM public.repl_mon 23 | INTO lag; 24 | END IF; 25 | 26 | SELECT count(*) 27 | FROM pg_stat_activity 28 | INTO sessions; 29 | 30 | SELECT sessions / setting::float 31 | FROM pg_settings 32 | INTO sessions_ratio 33 | WHERE name = 'max_connections'; 34 | END 35 | $$ LANGUAGE plpgsql SECURITY DEFINER; 36 | 37 | 38 | CREATE VIEW public.repl_mon AS 39 | SELECT pg_last_xact_replay_timestamp() AS ts, 40 | pg_last_wal_replay_lsn() AS location, 41 | 2 AS replics, 42 | 'hostname' AS master; 43 | 44 | 45 | CREATE TABLE tmp_table (ts timestamptz); 46 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/pg.conf: -------------------------------------------------------------------------------- 1 | listen_addresses = '*' 2 | shared_buffers = 16MB 3 | temp_buffers = 1MB 4 | work_mem = 1MB 5 | stats_temp_directory = 'pg_stat_tmp' 6 | fsync = off 7 | wal_keep_segments = 2 8 | track_commit_timestamp = on 9 | log_hostname = on 10 | pgcheck.closed = off 11 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/pg_hba.conf: -------------------------------------------------------------------------------- 1 | local all all trust 2 | host all all 0.0.0.0/0 trust 3 | host all all ::/0 trust 4 | host replication all 0.0.0.0/0 trust 5 | host replication all ::/0 trust 6 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/pgbouncer.ini: -------------------------------------------------------------------------------- 1 | [databases] 2 | * = host=127.0.0.1 3 | [pgbouncer] 4 | logfile = /var/log/postgresql/pgbouncer.log 5 | pidfile = /var/run/postgresql/pgbouncer.pid 6 | listen_addr = * 7 | listen_port = 6432 8 | auth_type = trust 9 | auth_file = /etc/pgbouncer/userlist.txt 10 | admin_users = postgres 11 | stats_users = postgres 12 | pool_mode = transaction 13 | server_reset_query = 14 | server_reset_query_always = 0 15 | ignore_startup_parameters = extra_float_digits 16 | server_check_delay = 30 17 | application_name_add_host = 1 18 | max_client_conn = 1000 19 | default_pool_size = 100 20 | min_pool_size = 0 21 | log_connections = 1 22 | log_disconnections = 1 23 | log_pooler_errors = 1 24 | server_idle_timeout = 20 25 | server_connect_timeout = 3 26 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/trusted_keys/pgdg.gpg: -------------------------------------------------------------------------------- 1 | -----BEGIN PGP PUBLIC KEY BLOCK----- 2 | Version: GnuPG v1 3 | 4 | mQINBE6XR8IBEACVdDKT2HEH1IyHzXkb4nIWAY7echjRxo7MTcj4vbXAyBKOfjja 5 | UrBEJWHN6fjKJXOYWXHLIYg0hOGeW9qcSiaa1/rYIbOzjfGfhE4x0Y+NJHS1db0V 6 | G6GUj3qXaeyqIJGS2z7m0Thy4Lgr/LpZlZ78Nf1fliSzBlMo1sV7PpP/7zUO+aA4 7 | bKa8Rio3weMXQOZgclzgeSdqtwKnyKTQdXY5MkH1QXyFIk1nTfWwyqpJjHlgtwMi 8 | c2cxjqG5nnV9rIYlTTjYG6RBglq0SmzF/raBnF4Lwjxq4qRqvRllBXdFu5+2pMfC 9 | IZ10HPRdqDCTN60DUix+BTzBUT30NzaLhZbOMT5RvQtvTVgWpeIn20i2NrPWNCUh 10 | hj490dKDLpK/v+A5/i8zPvN4c6MkDHi1FZfaoz3863dylUBR3Ip26oM0hHXf4/2U 11 | A/oA4pCl2W0hc4aNtozjKHkVjRx5Q8/hVYu+39csFWxo6YSB/KgIEw+0W8DiTII3 12 | RQj/OlD68ZDmGLyQPiJvaEtY9fDrcSpI0Esm0i4sjkNbuuh0Cvwwwqo5EF1zfkVj 13 | Tqz2REYQGMJGc5LUbIpk5sMHo1HWV038TWxlDRwtOdzw08zQA6BeWe9FOokRPeR2 14 | AqhyaJJwOZJodKZ76S+LDwFkTLzEKnYPCzkoRwLrEdNt1M7wQBThnC5z6wARAQAB 15 | tBxQb3N0Z3JlU1FMIERlYmlhbiBSZXBvc2l0b3J5iQI9BBMBCAAnAhsDBQsJCAcD 16 | BRUKCQgLBRYCAwEAAh4BAheABQJS6RUZBQkOhCctAAoJEH/MfUaszEz4zmQP/2ad 17 | HtuaXL5Xu3C3NGLha/aQb9iSJC8z5vN55HMCpsWlmslCBuEr+qR+oZvPkvwh0Io/ 18 | 8hQl/qN54DMNifRwVL2n2eG52yNERie9BrAMK2kNFZZCH4OxlMN0876BmDuNq2U6 19 | 7vUtCv+pxT+g9R1LvlPgLCTjS3m+qMqUICJ310BMT2cpYlJx3YqXouFkdWBVurI0 20 | pGU/+QtydcJALz5eZbzlbYSPWbOm2ZSS2cLrCsVNFDOAbYLtUn955yXB5s4rIscE 21 | vTzBxPgID1iBknnPzdu2tCpk07yJleiupxI1yXstCtvhGCbiAbGFDaKzhgcAxSIX 22 | 0ZPahpaYLdCkcoLlfgD+ar4K8veSK2LazrhO99O0onRG0p7zuXszXphO4E/WdbTO 23 | yDD35qCqYeAX6TaB+2l4kIdVqPgoXT/doWVLUK2NjZtd3JpMWI0OGYDFn2DAvgwP 24 | xqKEoGTOYuoWKssnwLlA/ZMETegak27gFAKfoQlmHjeA/PLC2KRYd6Wg2DSifhn+ 25 | 2MouoE4XFfeekVBQx98rOQ5NLwy/TYlsHXm1n0RW86ETN3chj/PPWjsi80t5oepx 26 | 82azRoVu95LJUkHpPLYyqwfueoVzp2+B2hJU2Rg7w+cJq64TfeJG8hrc93MnSKIb 27 | zTvXfdPtvYdHhhA2LYu4+5mh5ASlAMJXD7zIOZt2iEYEEBEIAAYFAk6XSO4ACgkQ 28 | xa93SlhRC1qmjwCg9U7U+XN7Gc/dhY/eymJqmzUGT/gAn0guvoX75Y+BsZlI6dWn 29 | qaFU6N8HiQIcBBABCAAGBQJOl0kLAAoJEExaa6sS0qeuBfEP/3AnLrcKx+dFKERX 30 | o4NBCGWr+i1CnowupKS3rm2xLbmiB969szG5TxnOIvnjECqPz6skK3HkV3jTZaju 31 | v3sR6M2ItpnrncWuiLnYcCSDp9TEMpCWzTEgtrBlKdVuTNTeRGILeIcvqoZX5w+u 32 | i0eBvvbeRbHEyUsvOEnYjrqoAjqUJj5FUZtR1+V9fnZp8zDgpOSxx0LomnFdKnhj 33 | uyXAQlRCA6/roVNR9ruRjxTR5ubteZ9ubTsVYr2/eMYOjQ46LhAgR+3Alblu/WHB 34 | MR/9F9//RuOa43R5Sjx9TiFCYol+Ozk8XRt3QGweEH51YkSYY3oRbHBb2Fkql6N6 35 | YFqlLBL7/aiWnNmRDEs/cdpo9HpFsbjOv4RlsSXQfvvfOayHpT5nO1UQFzoyMVpJ 36 | 615zwmQDJT5Qy7uvr2eQYRV9AXt8t/H+xjQsRZCc5YVmeAo91qIzI/tA2gtXik49 37 | 6yeziZbfUvcZzuzjjxFExss4DSAwMgorvBeIbiz2k2qXukbqcTjB2XqAlZasd6Ll 38 | nLXpQdqDV3McYkP/MvttWh3w+J/woiBcA7yEI5e3YJk97uS6+ssbqLEd0CcdT+qz 39 | +Waw0z/ZIU99Lfh2Qm77OT6vr//Zulw5ovjZVO2boRIcve7S97gQ4KC+G/+QaRS+ 40 | VPZ67j5UMxqtT/Y4+NHcQGgwF/1iiQI9BBMBCAAnAhsDBQsJCAcDBRUKCQgLBRYC 41 | AwEAAh4BAheABQJQeSssBQkDwxbfAAoJEH/MfUaszEz4bgkP/0AI0UgDgkNNqplA 42 | IpE/pkwem2jgGpJGKurh2xDu6j2ZL+BPzPhzyCeMHZwTXkkI373TXGQQP8dIa+RD 43 | HAZ3iijw4+ISdKWpziEUJjUk04UMPTlN+dYJt2EHLQDD0VLtX0yQC/wLmVEH/REp 44 | oclbVjZR/+ehwX2IxOIlXmkZJDSycl975FnSUjMAvyzty8P9DN0fIrQ7Ju+BfMOM 45 | TnUkOdp0kRUYez7pxbURJfkM0NxAP1geACI91aISBpFg3zxQs1d3MmUIhJ4wHvYB 46 | uaR7Fx1FkLAxWddre/OCYJBsjucE9uqc04rgKVjN5P/VfqNxyUoB+YZ+8Lk4t03p 47 | RBcD9XzcyOYlFLWXbcWxTn1jJ2QMqRIWi5lzZIOMw5B+OK9LLPX0dAwIFGr9WtuV 48 | J2zp+D4CBEMtn4Byh8EaQsttHeqAkpZoMlrEeNBDz2L7RquPQNmiuom15nb7xU/k 49 | 7PGfqtkpBaaGBV9tJkdp7BdH27dZXx+uT+uHbpMXkRrXliHjWpAw+NGwADh/Pjmq 50 | ExlQSdgAiXy1TTOdzxKH7WrwMFGDK0fddKr8GH3f+Oq4eOoNRa6/UhTCmBPbryCS 51 | IA7EAd0Aae9YaLlOB+eTORg/F1EWLPm34kKSRtae3gfHuY2cdUmoDVnOF8C9hc0P 52 | bL65G4NWPt+fW7lIj+0+kF19s2PviQI9BBMBCAAnAhsDBQsJCAcDBRUKCQgLBRYC 53 | AwEAAh4BAheABQJRKm2VBQkINsBBAAoJEH/MfUaszEz4RTEP/1sQHyjHaUiAPaCA 54 | v8jw/3SaWP/g8qLjpY6ROjLnDMvwKwRAoxUwcIv4/TWDOMpwJN+CJIbjXsXNYvf9 55 | OX+UTOvq4iwi4ADrAAw2xw+Jomc6EsYla+hkN2FzGzhpXfZFfUsuphjY3FKL+4hX 56 | H+R8ucNwIz3yrkfc17MMn8yFNWFzm4omU9/JeeaafwUoLxlULL2zY7H3+QmxCl0u 57 | 6t8VvlszdEFhemLHzVYRY0Ro/ISrR78CnANNsMIy3i11U5uvdeWVCoWV1BXNLzOD 58 | 4+BIDbMB/Do8PQCWiliSGZi8lvmj/sKbumMFQonMQWOfQswTtqTyQ3yhUM1LaxK5 59 | PYq13rggi3rA8oq8SYb/KNCQL5pzACji4TRVK0kNpvtxJxe84X8+9IB1vhBvF/Ji 60 | /xDd/3VDNPY+k1a47cON0S8Qc8DA3mq4hRfcgvuWy7ZxoMY7AfSJOhleb9+PzRBB 61 | n9agYgMxZg1RUWZazQ5KuoJqbxpwOYVFja/stItNS4xsmi0lh2I4MNlBEDqnFLUx 62 | SvTDc22c3uJlWhzBM/f2jH19uUeqm4jaggob3iJvJmK+Q7Ns3WcfhuWwCnc1+58d 63 | iFAMRUCRBPeFS0qd56QGk1r97B6+3UfLUslCfaaA8IMOFvQSHJwDO87xWGyxeRTY 64 | IIP9up4xwgje9LB7fMxsSkCDTHOk 65 | =s3DI 66 | -----END PGP PUBLIC KEY BLOCK----- 67 | -------------------------------------------------------------------------------- /docker/pgcheck-postgres/userlist.txt: -------------------------------------------------------------------------------- 1 | "postgres" "" 2 | -------------------------------------------------------------------------------- /host.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "fmt" 6 | "log" 7 | "time" 8 | ) 9 | 10 | type hostInfo struct { 11 | Name string `json:"name"` 12 | connStr string 13 | DC string `json:"dc"` 14 | prioDiff int 15 | partID int 16 | } 17 | 18 | type hostState struct { 19 | IsAlive bool `json:"alive"` 20 | IsPrimary bool `json:"primary"` 21 | ReplicationLag uint `json:"replication_lag"` 22 | SessionsRatio float64 `json:"sessions_ratio"` 23 | } 24 | 25 | var defaultHostState = hostState{ 26 | IsAlive: false, 27 | IsPrimary: false, 28 | ReplicationLag: 0, 29 | SessionsRatio: 0, 30 | } 31 | 32 | type hostAux struct { 33 | connectionPool *sql.DB 34 | statesChan chan hostState 35 | LastStates []hostState `json:"last_states"` 36 | } 37 | 38 | type host struct { 39 | hostInfo 40 | hostState 41 | hostPrio 42 | hostAux 43 | } 44 | 45 | func buildHostInfo(db *database, hostname string) *host { 46 | var host host 47 | var ( 48 | dc sql.NullString 49 | prioDiff sql.NullInt64 50 | ) 51 | // We assume here that one host may be strictly in one shard 52 | row := db.pool.QueryRow( 53 | `SELECT h.host_name, c.conn_string, h.dc, h.prio_diff, p.part_id, p.priority 54 | FROM plproxy.priorities p JOIN 55 | plproxy.hosts h USING (host_id) JOIN 56 | plproxy.connections c USING (conn_id) 57 | WHERE host_name = $1`, hostname) 58 | err := row.Scan(&host.Name, &host.connStr, &dc, &prioDiff, &host.partID, &host.CurrentPrio) 59 | if err != nil { 60 | log.Printf("Host %s is wrong: %v", hostname, err) 61 | } 62 | 63 | if dc.Valid { 64 | host.DC = dc.String 65 | } 66 | if prioDiff.Valid { 67 | host.prioDiff = int(prioDiff.Int64) 68 | } 69 | host.connStr = fmt.Sprintf("%s %s", host.connStr, db.config.AppendConnString) 70 | 71 | maxStatesCount := int(db.config.Quorum + db.config.Hysterisis) 72 | host.statesChan = make(chan hostState, maxStatesCount*3) 73 | host.LastStates = make([]hostState, 0, maxStatesCount) 74 | 75 | host.connectionPool, err = createPool(&host.connStr, false) 76 | if err != nil { 77 | log.Printf("Could not create connection pool for %s", hostname) 78 | } 79 | 80 | return &host 81 | } 82 | 83 | func getHostState(host *host, config *Config, dbname string) *hostState { 84 | dbConfig := config.Databases[dbname] 85 | maxStatesCount := int(dbConfig.Quorum + dbConfig.Hysterisis) 86 | go sendStateToStatesChan(host, &config.DC) 87 | 88 | if len(host.statesChan) > maxStatesCount { 89 | for i := maxStatesCount; i < len(host.statesChan); i++ { 90 | <-host.statesChan 91 | } 92 | } 93 | 94 | var state hostState 95 | timeout := time.Second*time.Duration(config.Timeout) + 100*time.Millisecond 96 | select { 97 | case x := <-host.statesChan: 98 | state = x 99 | case <-time.After(timeout): 100 | log.Printf("Getting status of %s timed out", host.Name) 101 | state = defaultHostState 102 | } 103 | 104 | return &state 105 | } 106 | 107 | func sendStateToStatesChan(host *host, myDC *string) { 108 | c := host.statesChan 109 | 110 | db, err := getPool(host) 111 | if err != nil { 112 | log.Printf("Connection to %s failed: %v", host.Name, err) 113 | c <- defaultHostState 114 | return 115 | } 116 | 117 | state := fillState(db, host) 118 | c <- state 119 | } 120 | 121 | func fillState(db *sql.DB, host *host) hostState { 122 | state := defaultHostState 123 | 124 | var isMaster bool 125 | var replicationLag uint 126 | var sessionsRatio float64 127 | row := db.QueryRow(`SELECT is_master, lag, sessions_ratio 128 | FROM public.pgcheck_poll()`) 129 | err := row.Scan(&isMaster, &replicationLag, &sessionsRatio) 130 | if err != nil { 131 | log.Printf("Checking %s failed: %v", host.Name, err) 132 | return state 133 | } 134 | 135 | state.IsAlive = true 136 | state.IsPrimary = isMaster 137 | state.ReplicationLag = replicationLag 138 | state.SessionsRatio = sessionsRatio 139 | return state 140 | } 141 | 142 | func updateLastStates(host *host, state *hostState, maxStatesCount int) *[]hostState { 143 | var startFrom int 144 | if len(host.LastStates) != maxStatesCount { 145 | startFrom = 0 146 | } else { 147 | startFrom = 1 148 | } 149 | result := append(host.LastStates[startFrom:], *state) 150 | //log.Printf("Last states of %s are: %v", host.name, result) 151 | return &result 152 | } 153 | 154 | func updateState(hostsInfo *map[string]*host, hostname string, state *hostState, quorum uint, myDC string) { 155 | hosts := *hostsInfo 156 | host := hosts[hostname] 157 | neededPrio := stateToPrio(host, state, &myDC) 158 | 159 | if prioIsNear(host.CurrentPrio, neededPrio) { 160 | hosts[hostname].hostState = *state 161 | return 162 | } 163 | 164 | var cnt uint 165 | for i := range hosts[hostname].LastStates { 166 | prio := stateToPrio(host, &host.LastStates[i], &myDC) 167 | if prioIsNear(prio, neededPrio) { 168 | cnt++ 169 | } 170 | } 171 | if cnt >= quorum { 172 | hosts[hostname].NeededPrio = neededPrio 173 | hosts[hostname].hostState = *state 174 | } else { 175 | hosts[hostname].NeededPrio = hosts[hostname].CurrentPrio 176 | } 177 | } 178 | -------------------------------------------------------------------------------- /http.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "log" 7 | "net/http" 8 | ) 9 | 10 | type statsState struct { 11 | dbname string 12 | hostsMap map[string]*host 13 | shardsMap map[int][]string 14 | } 15 | 16 | // StatsState is a type for marshaling to JSON 17 | type StatsState struct { 18 | HostsMap map[string]host `json:"hosts"` 19 | ShardsMap map[int][]string `json:"shards"` 20 | } 21 | 22 | // Statistics contains current statistics 23 | type Statistics struct { 24 | TotalShards uint `json:"total_shards"` 25 | NormalShards uint `json:"normal_shards"` 26 | ReadOnlyShards uint `json:"read_only_shards"` 27 | NoReplicShards uint `json:"shards_without_replicas"` 28 | SplitBrainShards uint `json:"split_brain_shards"` 29 | FullyDeadShards uint `json:"fully_dead_shards"` 30 | AliveHosts uint `json:"alive_hosts"` 31 | DeadHosts uint `json:"dead_hosts"` 32 | } 33 | 34 | var currentState = make(map[string]StatsState) 35 | 36 | func currentStateHandler(w http.ResponseWriter, r *http.Request) { 37 | state, err := json.Marshal(currentState) 38 | if err != nil { 39 | log.Println(err) 40 | return 41 | } 42 | w.Header().Set("Content-type", "application/json") 43 | fmt.Fprintf(w, string(state)) 44 | } 45 | 46 | func statisticsHandler(w http.ResponseWriter, r *http.Request) { 47 | var statistic Statistics 48 | for _, state := range currentState { 49 | for shardID := range state.ShardsMap { 50 | countStatsForShard(&statistic, &state, shardID) 51 | } 52 | } 53 | 54 | stats, err := json.Marshal(statistic) 55 | if err != nil { 56 | log.Println(err) 57 | return 58 | } 59 | w.Header().Set("Content-type", "application/json") 60 | fmt.Fprintf(w, string(stats)) 61 | } 62 | 63 | func startStatsServer(config Config, ch chan statsState) { 64 | for db := range config.Databases { 65 | currentState[db] = StatsState{} 66 | } 67 | 68 | go startUpdatingStats(ch) 69 | 70 | http.HandleFunc("/state", currentStateHandler) 71 | http.HandleFunc("/stats", statisticsHandler) 72 | log.Print(http.ListenAndServe(":8080", nil)) 73 | } 74 | 75 | func startUpdatingStats(ch chan statsState) { 76 | for { 77 | x := <-ch 78 | currentState[x.dbname] = state2State(x) 79 | } 80 | } 81 | 82 | func countStatsForShard(statistic *Statistics, state *StatsState, shardID int) { 83 | shardHosts := state.ShardsMap[shardID] 84 | aliveMasters := 0 85 | aliveReplics := 0 86 | notDelayedReplics := 0 87 | 88 | for _, fqdn := range shardHosts { 89 | hostState := state.HostsMap[fqdn] 90 | 91 | if hostState.IsAlive { 92 | statistic.AliveHosts++ 93 | if hostState.IsPrimary { 94 | aliveMasters++ 95 | } else { 96 | aliveReplics++ 97 | if hostState.ReplicationLag < 90 { 98 | notDelayedReplics++ 99 | } 100 | } 101 | } else { 102 | statistic.DeadHosts++ 103 | } 104 | } 105 | 106 | statistic.TotalShards++ 107 | if aliveMasters == 0 { 108 | if aliveReplics == 0 { 109 | statistic.FullyDeadShards++ 110 | } else { 111 | statistic.ReadOnlyShards++ 112 | } 113 | } else if aliveMasters > 1 { 114 | statistic.SplitBrainShards++ 115 | // Below aliveMasters == 1 116 | } else if aliveReplics == 0 || notDelayedReplics == 0 { 117 | statistic.NoReplicShards++ 118 | } else if aliveReplics > 0 { 119 | statistic.NormalShards++ 120 | } 121 | } 122 | 123 | func state2State(in statsState) StatsState { 124 | tmpState := StatsState{} 125 | tmpState.ShardsMap = in.shardsMap 126 | 127 | tmpState.HostsMap = make(map[string]host) 128 | for k := range in.hostsMap { 129 | tmpState.HostsMap[k] = *in.hostsMap[k] 130 | } 131 | 132 | return tmpState 133 | } 134 | -------------------------------------------------------------------------------- /http_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/assert" 8 | ) 9 | 10 | type CountStatsForShardTestCase struct { 11 | name string 12 | inputState []byte 13 | neededStat Statistics 14 | } 15 | 16 | const shardID = 0 17 | 18 | var shardsMap = map[int][]string{ 19 | shardID: { 20 | "shard01-dc1.pgcheck.net", 21 | "shard01-dc2.pgcheck.net", 22 | "shard01-dc3.pgcheck.net", 23 | }, 24 | } 25 | 26 | var CountStatsForShardTestCases = []CountStatsForShardTestCase{ 27 | { 28 | "NormalShard", 29 | []byte(`{ 30 | "shard01-dc1.pgcheck.net": { 31 | "name": "shard01-dc1.pgcheck.net", 32 | "dc": "DC1", 33 | "alive": true, 34 | "primary": true, 35 | "replication_lag": 0, 36 | "sessions_ratio": 0.08, 37 | "needed_prio": 0, 38 | "current_prio": 0, 39 | "last_states": [] 40 | }, 41 | "shard01-dc2.pgcheck.net": { 42 | "name": "shard01-dc2.pgcheck.net", 43 | "dc": "DC2", 44 | "alive": true, 45 | "primary": false, 46 | "replication_lag": 0, 47 | "sessions_ratio": 0.05, 48 | "needed_prio": 20, 49 | "current_prio": 20, 50 | "last_states": [] 51 | }, 52 | "shard01-dc3.pgcheck.net": { 53 | "name": "shard01-dc3.pgcheck.net", 54 | "dc": "DC3", 55 | "alive": true, 56 | "primary": false, 57 | "replication_lag": 0, 58 | "sessions_ratio": 0.05, 59 | "needed_prio": 25, 60 | "current_prio": 25, 61 | "last_states": [] 62 | } 63 | }`), 64 | Statistics{1, 1, 0, 0, 0, 0, 3, 0}, 65 | }, 66 | { 67 | "DeadReplica", 68 | []byte(`{ 69 | "shard01-dc1.pgcheck.net": { 70 | "name": "shard01-dc1.pgcheck.net", 71 | "dc": "DC1", 72 | "alive": true, 73 | "primary": true, 74 | "replication_lag": 0, 75 | "sessions_ratio": 0.08, 76 | "needed_prio": 0, 77 | "current_prio": 0, 78 | "last_states": [] 79 | }, 80 | "shard01-dc2.pgcheck.net": { 81 | "name": "shard01-dc2.pgcheck.net", 82 | "dc": "DC2", 83 | "alive": false, 84 | "primary": false, 85 | "replication_lag": 0, 86 | "sessions_ratio": 0, 87 | "needed_prio": 100, 88 | "current_prio": 100, 89 | "last_states": [] 90 | }, 91 | "shard01-dc3.pgcheck.net": { 92 | "name": "shard01-dc3.pgcheck.net", 93 | "dc": "DC3", 94 | "alive": true, 95 | "primary": false, 96 | "replication_lag": 0, 97 | "sessions_ratio": 0.05, 98 | "needed_prio": 25, 99 | "current_prio": 25, 100 | "last_states": [] 101 | } 102 | }`), 103 | Statistics{1, 1, 0, 0, 0, 0, 2, 1}, 104 | }, 105 | { 106 | "ReadOnlyShard", 107 | []byte(`{ 108 | "shard01-dc1.pgcheck.net": { 109 | "name": "shard01-dc1.pgcheck.net", 110 | "dc": "DC1", 111 | "alive": false, 112 | "primary": false, 113 | "replication_lag": 0, 114 | "sessions_ratio": 0, 115 | "needed_prio": 100, 116 | "current_prio": 100, 117 | "last_states": [] 118 | }, 119 | "shard01-dc2.pgcheck.net": { 120 | "name": "shard01-dc2.pgcheck.net", 121 | "dc": "DC2", 122 | "alive": true, 123 | "primary": false, 124 | "replication_lag": 10, 125 | "sessions_ratio": 0.05, 126 | "needed_prio": 30, 127 | "current_prio": 30, 128 | "last_states": [] 129 | }, 130 | "shard01-dc3.pgcheck.net": { 131 | "name": "shard01-dc3.pgcheck.net", 132 | "dc": "DC3", 133 | "alive": true, 134 | "primary": false, 135 | "replication_lag": 15, 136 | "sessions_ratio": 0.05, 137 | "needed_prio": 40, 138 | "current_prio": 40, 139 | "last_states": [] 140 | } 141 | }`), 142 | Statistics{1, 0, 1, 0, 0, 0, 2, 1}, 143 | }, 144 | { 145 | "NoReplicShard", 146 | []byte(`{ 147 | "shard01-dc1.pgcheck.net": { 148 | "name": "shard01-dc1.pgcheck.net", 149 | "dc": "DC1", 150 | "alive": true, 151 | "primary": true, 152 | "replication_lag": 0, 153 | "sessions_ratio": 0.08, 154 | "needed_prio": 0, 155 | "current_prio": 0, 156 | "last_states": [] 157 | }, 158 | "shard01-dc2.pgcheck.net": { 159 | "name": "shard01-dc2.pgcheck.net", 160 | "dc": "DC2", 161 | "alive": false, 162 | "primary": false, 163 | "replication_lag": 0, 164 | "sessions_ratio": 0, 165 | "needed_prio": 100, 166 | "current_prio": 100, 167 | "last_states": [] 168 | }, 169 | "shard01-dc3.pgcheck.net": { 170 | "name": "shard01-dc3.pgcheck.net", 171 | "dc": "DC3", 172 | "alive": true, 173 | "primary": false, 174 | "replication_lag": 100, 175 | "sessions_ratio": 0.05, 176 | "needed_prio": 120, 177 | "current_prio": 120, 178 | "last_states": [] 179 | } 180 | }`), 181 | Statistics{1, 0, 0, 1, 0, 0, 2, 1}, 182 | }, 183 | { 184 | "SplitBrainShard", 185 | []byte(`{ 186 | "shard01-dc1.pgcheck.net": { 187 | "name": "shard01-dc1.pgcheck.net", 188 | "dc": "DC1", 189 | "alive": true, 190 | "primary": true, 191 | "replication_lag": 0, 192 | "sessions_ratio": 0.08, 193 | "needed_prio": 100, 194 | "current_prio": 100, 195 | "last_states": [] 196 | }, 197 | "shard01-dc2.pgcheck.net": { 198 | "name": "shard01-dc2.pgcheck.net", 199 | "dc": "DC2", 200 | "alive": true, 201 | "primary": true, 202 | "replication_lag": 0, 203 | "sessions_ratio": 0.16, 204 | "needed_prio": 100, 205 | "current_prio": 100, 206 | "last_states": [] 207 | }, 208 | "shard01-dc3.pgcheck.net": { 209 | "name": "shard01-dc3.pgcheck.net", 210 | "dc": "DC3", 211 | "alive": true, 212 | "primary": false, 213 | "replication_lag": 10, 214 | "sessions_ratio": 0.05, 215 | "needed_prio": 20, 216 | "current_prio": 20, 217 | "last_states": [] 218 | } 219 | }`), 220 | Statistics{1, 0, 0, 0, 1, 0, 3, 0}, 221 | }, 222 | { 223 | "FullyDeadShard", 224 | []byte(`{ 225 | "shard01-dc1.pgcheck.net": { 226 | "name": "shard01-dc1.pgcheck.net", 227 | "dc": "DC1", 228 | "alive": false, 229 | "primary": false, 230 | "replication_lag": 0, 231 | "sessions_ratio": 0, 232 | "needed_prio": 100, 233 | "current_prio": 100, 234 | "last_states": [] 235 | }, 236 | "shard01-dc2.pgcheck.net": { 237 | "name": "shard01-dc2.pgcheck.net", 238 | "dc": "DC2", 239 | "alive": false, 240 | "primary": false, 241 | "replication_lag": 0, 242 | "sessions_ratio": 0, 243 | "needed_prio": 100, 244 | "current_prio": 100, 245 | "last_states": [] 246 | }, 247 | "shard01-dc3.pgcheck.net": { 248 | "name": "shard01-dc3.pgcheck.net", 249 | "dc": "DC3", 250 | "alive": false, 251 | "primary": false, 252 | "replication_lag": 0, 253 | "sessions_ratio": 0, 254 | "needed_prio": 0, 255 | "current_prio": 0, 256 | "last_states": [] 257 | } 258 | }`), 259 | Statistics{1, 0, 0, 0, 0, 1, 0, 3}, 260 | }, 261 | } 262 | 263 | func TestCountStatsForShard(t *testing.T) { 264 | for _, test := range CountStatsForShardTestCases { 265 | testOneCountStatsForShardCase(t, test) 266 | } 267 | } 268 | 269 | func testOneCountStatsForShardCase(t *testing.T, test CountStatsForShardTestCase) { 270 | t.Run(test.name, func(t *testing.T) { 271 | hostsMap := make(map[string]host) 272 | err := json.Unmarshal(test.inputState, &hostsMap) 273 | if err != nil { 274 | t.Error(err) 275 | } 276 | 277 | state := StatsState{hostsMap, shardsMap} 278 | var statistic Statistics 279 | countStatsForShard(&statistic, &state, shardID) 280 | 281 | assert.Equal(t, test.neededStat, statistic, "Wrong statistics") 282 | }) 283 | } 284 | -------------------------------------------------------------------------------- /pgcheck.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "log" 6 | "time" 7 | 8 | _ "github.com/lib/pq" 9 | ) 10 | 11 | const maxChannelsSize = 100 12 | 13 | func main() { 14 | var config Config 15 | config = parseConfig() 16 | initLogging(&config) 17 | 18 | statsChan := make(chan statsState, maxChannelsSize) 19 | go startStatsServer(config, statsChan) 20 | 21 | for db := range config.Databases { 22 | go processDB(db, &config, statsChan) 23 | } 24 | 25 | handleSignals() 26 | } 27 | 28 | func processDB(dbname string, config *Config, statsChan chan statsState) { 29 | var db database 30 | db.name = dbname 31 | db.config = config.Databases[dbname] 32 | db.pool, _ = createPool(&db.config.LocalConnString, true) 33 | defer db.pool.Close() 34 | 35 | hosts := buildInitialHostsInfo(&db) 36 | shards := buildShardsInfo(hosts) 37 | 38 | for { 39 | updateHostsState(hosts, config, dbname) 40 | correctPrioForHostsInShard(shards, hosts) 41 | updatePriorities(db.pool, hosts) 42 | sendStats(statsChan, dbname, hosts, shards) 43 | time.Sleep(time.Second) 44 | } 45 | } 46 | 47 | func buildInitialHostsInfo(db *database) *map[string]*host { 48 | rows, err := db.pool.Query("SELECT distinct(host_name) FROM plproxy.hosts") 49 | if err != nil { 50 | log.Printf("%s: %s", db.name, err) 51 | } 52 | defer rows.Close() 53 | 54 | hosts := make(map[string]*host) 55 | 56 | for rows.Next() { 57 | var hostname string 58 | if err := rows.Scan(&hostname); err != nil { 59 | log.Printf("%s: %s", db.name, err) 60 | } 61 | 62 | hosts[hostname] = buildHostInfo(db, hostname) 63 | } 64 | 65 | if err := rows.Err(); err != nil { 66 | log.Printf("%s: %s", db.name, err) 67 | } 68 | 69 | return &hosts 70 | } 71 | 72 | func buildShardsInfo(hostsInfo *map[string]*host) *map[int][]string { 73 | hosts := *hostsInfo 74 | shards := make(map[int][]string) 75 | for hostname := range hosts { 76 | shard := hosts[hostname].partID 77 | shards[shard] = append(shards[shard], hostname) 78 | } 79 | return &shards 80 | } 81 | 82 | func updateHostsState(hostsInfo *map[string]*host, wholeConfig *Config, dbname string) { 83 | config := wholeConfig.Databases[dbname] 84 | maxStatesCount := int(config.Quorum + config.Hysterisis) 85 | 86 | hosts := *hostsInfo 87 | for hostname := range hosts { 88 | host := hosts[hostname] 89 | state := getHostState(host, wholeConfig, dbname) 90 | hosts[hostname].LastStates = *updateLastStates(host, state, maxStatesCount) 91 | updateState(&hosts, hostname, state, config.Quorum, wholeConfig.DC) 92 | } 93 | } 94 | 95 | func correctPrioForHostsInShard(shardsInfo *map[int][]string, hostsInfo *map[string]*host) { 96 | shards := *shardsInfo 97 | hosts := *hostsInfo 98 | 99 | for partID, hostsList := range shards { 100 | var masters []string 101 | for _, h := range hostsList { 102 | if hosts[h].IsAlive && hosts[h].IsPrimary { 103 | masters = append(masters, h) 104 | } 105 | } 106 | //log.Printf("%d: %d masters", partID, len(masters)) 107 | if len(masters) > 1 { 108 | log.Printf("%d masters in shard %d. Changing priority for "+ 109 | "all of them to %d", len(masters), partID, deadHostPrio) 110 | for _, h := range masters { 111 | hosts[h].NeededPrio = deadHostPrio 112 | } 113 | } 114 | 115 | if len(masters) != 1 { 116 | for _, h := range hostsList { 117 | if !hosts[h].IsPrimary && hosts[h].IsAlive { 118 | // Do not account replication lag since master is dead 119 | // and nothing is written 120 | hosts[h].NeededPrio = 121 | hosts[h].NeededPrio - priority(hosts[h].ReplicationLag) 122 | } 123 | } 124 | } 125 | } 126 | } 127 | 128 | func updatePriorities(db *sql.DB, hostsInfo *map[string]*host) { 129 | hosts := *hostsInfo 130 | for hostname := range hosts { 131 | s := hosts[hostname] 132 | //log.Printf("%s: current %d, needed %d", s.name, s.currentPrio, s.neededPrio) 133 | if s.CurrentPrio != s.NeededPrio { 134 | hosts[hostname].CurrentPrio = s.NeededPrio 135 | updateHostPriority(db, hostname, s.NeededPrio) 136 | log.Printf("Priority of host %s has been changed to %d ", 137 | hostname, s.NeededPrio) 138 | } 139 | } 140 | } 141 | 142 | func sendStats(ch chan statsState, dbname string, hosts *map[string]*host, shards *map[int][]string) { 143 | if len(ch) == maxChannelsSize { 144 | log.Printf("Could not send stats for DB %s since channel is full", dbname) 145 | return 146 | } 147 | ch <- statsState{dbname, *hosts, *shards} 148 | } 149 | -------------------------------------------------------------------------------- /pgcheck_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | ) 8 | 9 | type correctPrioForHostsInShardTestCase struct { 10 | name string 11 | inHostsInfo map[string]*host 12 | neededPriorities map[string]priority 13 | } 14 | 15 | var correctPrioForHostsInShardTestCases = []correctPrioForHostsInShardTestCase{ 16 | { 17 | "Normal", 18 | map[string]*host{ 19 | "shard01-dc1.pgcheck.net": &host{ 20 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 21 | hostState{true, true, 0, 0.08}, 22 | hostPrio{0, 0}, 23 | hostAux{nil, nil, nil}, 24 | }, 25 | "shard01-dc2.pgcheck.net": &host{ 26 | hostInfo{"shard01-dc2.pgcheck.net", "", "DC2", 0, 0}, 27 | hostState{true, false, 1, 0.05}, 28 | hostPrio{11, 11}, 29 | hostAux{nil, nil, nil}, 30 | }, 31 | "shard01-dc3.pgcheck.net": &host{ 32 | hostInfo{"shard01-dc3.pgcheck.net", "", "DC3", 0, 0}, 33 | hostState{true, false, 0, 0.05}, 34 | hostPrio{20, 20}, 35 | hostAux{nil, nil, nil}, 36 | }, 37 | }, 38 | map[string]priority{ 39 | "shard01-dc1.pgcheck.net": 0, 40 | "shard01-dc2.pgcheck.net": 11, 41 | "shard01-dc3.pgcheck.net": 20, 42 | }, 43 | }, 44 | { 45 | "Split brain without replication lag", 46 | map[string]*host{ 47 | "shard01-dc1.pgcheck.net": &host{ 48 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 49 | hostState{true, true, 0, 0.08}, 50 | hostPrio{0, 0}, 51 | hostAux{nil, nil, nil}, 52 | }, 53 | "shard01-dc2.pgcheck.net": &host{ 54 | hostInfo{"shard01-dc2.pgcheck.net", "", "DC2", 0, 0}, 55 | hostState{true, true, 0, 0.08}, 56 | hostPrio{0, 0}, 57 | hostAux{nil, nil, nil}, 58 | }, 59 | "shard01-dc3.pgcheck.net": &host{ 60 | hostInfo{"shard01-dc3.pgcheck.net", "", "DC3", 0, 0}, 61 | hostState{true, false, 0, 0.05}, 62 | hostPrio{20, 20}, 63 | hostAux{nil, nil, nil}, 64 | }, 65 | }, 66 | map[string]priority{ 67 | "shard01-dc1.pgcheck.net": 100, 68 | "shard01-dc2.pgcheck.net": 100, 69 | "shard01-dc3.pgcheck.net": 20, 70 | }, 71 | }, 72 | { 73 | "Split brain with replication lag", 74 | map[string]*host{ 75 | "shard01-dc1.pgcheck.net": &host{ 76 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 77 | hostState{true, true, 0, 0.08}, 78 | hostPrio{0, 0}, 79 | hostAux{nil, nil, nil}, 80 | }, 81 | "shard01-dc2.pgcheck.net": &host{ 82 | hostInfo{"shard01-dc2.pgcheck.net", "", "DC2", 0, 0}, 83 | hostState{true, true, 0, 0.08}, 84 | hostPrio{0, 0}, 85 | hostAux{nil, nil, nil}, 86 | }, 87 | "shard01-dc3.pgcheck.net": &host{ 88 | hostInfo{"shard01-dc3.pgcheck.net", "", "DC3", 0, 0}, 89 | hostState{true, false, 100500, 0.05}, 90 | hostPrio{100520, 100520}, 91 | hostAux{nil, nil, nil}, 92 | }, 93 | }, 94 | map[string]priority{ 95 | "shard01-dc1.pgcheck.net": 100, 96 | "shard01-dc2.pgcheck.net": 100, 97 | "shard01-dc3.pgcheck.net": 20, 98 | }, 99 | }, 100 | { 101 | "No master with replication lag", 102 | map[string]*host{ 103 | "shard01-dc1.pgcheck.net": &host{ 104 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 105 | hostState{false, false, 0, 0}, 106 | hostPrio{100, 100}, 107 | hostAux{nil, nil, nil}, 108 | }, 109 | "shard01-dc2.pgcheck.net": &host{ 110 | hostInfo{"shard01-dc2.pgcheck.net", "", "DC2", 0, 0}, 111 | hostState{true, false, 100500, 0.08}, 112 | hostPrio{100510, 100510}, 113 | hostAux{nil, nil, nil}, 114 | }, 115 | "shard01-dc3.pgcheck.net": &host{ 116 | hostInfo{"shard01-dc3.pgcheck.net", "", "DC3", 0, 0}, 117 | hostState{true, false, 100500, 0.05}, 118 | hostPrio{100520, 100520}, 119 | hostAux{nil, nil, nil}, 120 | }, 121 | }, 122 | map[string]priority{ 123 | "shard01-dc1.pgcheck.net": 100, 124 | "shard01-dc2.pgcheck.net": 10, 125 | "shard01-dc3.pgcheck.net": 20, 126 | }, 127 | }, 128 | } 129 | 130 | func TestCorrectPrioForHostsInShard(t *testing.T) { 131 | for _, test := range correctPrioForHostsInShardTestCases { 132 | t.Run(test.name, func(t *testing.T) { 133 | // We use shardsMap from http_test.go, ugly but works 134 | correctPrioForHostsInShard(&shardsMap, &test.inHostsInfo) 135 | resultPrioritites := make(map[string]priority) 136 | for h := range test.neededPriorities { 137 | resultPrioritites[h] = test.inHostsInfo[h].NeededPrio 138 | } 139 | assert.Equal(t, test.neededPriorities, resultPrioritites, "Needed priorities mismatch") 140 | }) 141 | } 142 | } 143 | -------------------------------------------------------------------------------- /priority.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "log" 6 | ) 7 | 8 | type priority uint 9 | 10 | const ( 11 | alivePrimaryPrio priority = 0 12 | aliveStandbyPrio priority = 10 13 | deadHostPrio priority = 100 14 | ) 15 | 16 | type hostPrio struct { 17 | CurrentPrio priority 18 | NeededPrio priority 19 | } 20 | 21 | func prioIsNear(currentPrio, newPrio priority) bool { 22 | const magic = 5 23 | lower := int(currentPrio) - magic 24 | upper := int(currentPrio) + magic 25 | if int(newPrio) >= lower && int(newPrio) <= upper { 26 | return true 27 | } 28 | return false 29 | } 30 | 31 | func stateToPrio(host *host, state *hostState, myDC *string) priority { 32 | if !state.IsAlive { 33 | return deadHostPrio 34 | } else if state.IsPrimary { 35 | return alivePrimaryPrio 36 | } 37 | 38 | prio := aliveStandbyPrio 39 | if host.DC != *myDC { 40 | prio += 10 41 | } 42 | prio += priority(state.ReplicationLag) 43 | prio += priority(state.SessionsRatio * 100.0 / 2) 44 | 45 | // prioDiff may be negative so we cast everything to int first 46 | // and then back to priority type 47 | p := int(prio) + host.prioDiff 48 | if p < 0 { 49 | p = int(deadHostPrio) - p 50 | } 51 | prio = priority(p) 52 | 53 | return prio 54 | } 55 | 56 | func updateHostPriority(db *sql.DB, hostname string, prio priority) { 57 | _, err := db.Exec( 58 | `UPDATE plproxy.priorities 59 | SET priority = $1 60 | WHERE host_id = ( 61 | SELECT host_id FROM plproxy.hosts WHERE host_name = $2 62 | )`, prio, hostname) 63 | if err != nil { 64 | log.Printf("Setting priority for %s failed: %v", hostname, err) 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /priority_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | ) 8 | 9 | type stateToPrioTestCase struct { 10 | name string 11 | inHost host 12 | inState hostState 13 | inDC string 14 | neededPrio priority 15 | } 16 | 17 | var stateToPrioTestCases = []stateToPrioTestCase{ 18 | { 19 | "Alive master", 20 | host{ 21 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 22 | hostState{}, 23 | hostPrio{}, 24 | hostAux{}, 25 | }, 26 | hostState{true, true, 0, 0.08}, 27 | "DC2", 28 | 0, 29 | }, 30 | { 31 | "Dead host", 32 | host{ 33 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 34 | hostState{}, 35 | hostPrio{}, 36 | hostAux{}, 37 | }, 38 | hostState{false, false, 0, 0}, 39 | "DC2", 40 | 100, 41 | }, 42 | { 43 | "Alive replica in other DC", 44 | host{ 45 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 46 | hostState{}, 47 | hostPrio{}, 48 | hostAux{}, 49 | }, 50 | hostState{true, false, 0, 0.08}, 51 | "DC2", 52 | 24, 53 | }, 54 | { 55 | "Alive replica in same DC", 56 | host{ 57 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 58 | hostState{}, 59 | hostPrio{}, 60 | hostAux{}, 61 | }, 62 | hostState{true, false, 0, 0.08}, 63 | "DC1", 64 | 14, 65 | }, 66 | { 67 | "Alive replica with replication lag", 68 | host{ 69 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 70 | hostState{}, 71 | hostPrio{}, 72 | hostAux{}, 73 | }, 74 | hostState{true, false, 10, 0.08}, 75 | "DC2", 76 | 34, 77 | }, 78 | { 79 | "Alive overloaded replica", 80 | host{ 81 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 82 | hostState{}, 83 | hostPrio{}, 84 | hostAux{}, 85 | }, 86 | hostState{true, false, 0, 0.9}, 87 | "DC2", 88 | 65, 89 | }, 90 | { 91 | "Alive delayed and overloaded replica", 92 | host{ 93 | hostInfo{"shard01-dc1.pgcheck.net", "", "DC1", 0, 0}, 94 | hostState{}, 95 | hostPrio{}, 96 | hostAux{}, 97 | }, 98 | hostState{true, false, 50, 0.9}, 99 | "DC2", 100 | 115, 101 | }, 102 | } 103 | 104 | func TestStateToPrio(t *testing.T) { 105 | for _, test := range stateToPrioTestCases { 106 | t.Run(test.name, func(t *testing.T) { 107 | prio := stateToPrio(&test.inHost, &test.inState, &test.inDC) 108 | assert.Equal(t, int(test.neededPrio), int(prio), "Wrong priority") 109 | }) 110 | } 111 | assert.Equal(t, 1, 1, "Shit") 112 | } 113 | -------------------------------------------------------------------------------- /samples/etc/pgcheck.yml: -------------------------------------------------------------------------------- 1 | # user name to run daemon as 2 | daemon_user: postgres 3 | log_file: 4 | # debug, info. warning, error, critical 5 | log_level: info 6 | pid_file: /var/run/pgcheck/pgcheck.pid 7 | # path, where to run script from ($PWD) 8 | working_dir: /tmp 9 | # time between iterations (in seconds) 10 | iteration_timeout: 1 11 | # assign or not priorities for replicas depending on load 12 | replics_weights: yes 13 | # account or not replication lag (replay_location) of replica 14 | # right now pgcheck increases base_prio by one on every megabyte of lag 15 | account_replication_lag: yes 16 | # DC of PL/Proxy host 17 | my_dc: DC1 18 | # list of DBs which to poll 19 | databases: 20 | db1: 21 | # connection string to local database with tables about shards, hosts and priorities 22 | local_conn_string: dbname=db1 user=postgres sslmode=disable 23 | # additional parameters for connections strings for shards databases 24 | append_conn_string: user=postgres sslmode=disable connect_timeout=1 25 | # number of checks, after which the priority can be changed 26 | quorum: 2 27 | # number of priorities, about which we additionally store information 28 | hysterisis: 1 -------------------------------------------------------------------------------- /samples/sql/00_pgproxy.sql: -------------------------------------------------------------------------------- 1 | -- handler function 2 | CREATE OR REPLACE FUNCTION plproxy_call_handler () 3 | RETURNS language_handler AS 'plproxy' LANGUAGE C; 4 | 5 | -- language 6 | CREATE OR REPLACE LANGUAGE plproxy HANDLER plproxy_call_handler; 7 | 8 | -- validator function 9 | CREATE OR REPLACE FUNCTION plproxy_fdw_validator (text[], oid) 10 | RETURNS boolean AS 'plproxy' LANGUAGE C; 11 | 12 | -- foreign data wrapper 13 | DROP FOREIGN DATA WRAPPER IF EXISTS plproxy CASCADE; 14 | CREATE FOREIGN DATA WRAPPER plproxy VALIDATOR plproxy_fdw_validator; 15 | 16 | drop schema if exists plproxy cascade; 17 | create schema plproxy; 18 | 19 | create table plproxy.parts 20 | ( 21 | part_id integer not null 22 | ); 23 | 24 | create table plproxy.hosts 25 | ( 26 | host_id integer not null, 27 | host_name varchar(100) not null, 28 | dc varchar(10), 29 | prio_diff smallint null 30 | ); 31 | 32 | create table plproxy.connections 33 | ( 34 | conn_id integer not null, 35 | conn_string varchar(255) not null 36 | ); 37 | 38 | create table plproxy.versions 39 | ( 40 | version bigint default 0 not null 41 | ); 42 | 43 | insert into plproxy.versions values (1); 44 | 45 | create table plproxy.priorities 46 | ( 47 | part_id integer not null, 48 | host_id integer not null, 49 | conn_id integer not null, 50 | priority smallint default 100 not null 51 | ); 52 | 53 | create table plproxy.config 54 | ( 55 | key varchar(50) not null, 56 | value varchar(255) 57 | ); 58 | 59 | create table plproxy.key_ranges 60 | ( 61 | range_id integer not null, 62 | part_id integer not null, 63 | start_key bigint not null, 64 | end_key bigint not null 65 | ); 66 | 67 | 68 | -------------------------------------------------------------------------------- /samples/sql/30_get_cluster_partitions.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION plproxy.get_cluster_partitions(i_cluster_name text) 2 | RETURNS SETOF text 3 | LANGUAGE plpgsql 4 | AS $function$ 5 | declare 6 | r record; 7 | min_priority integer; 8 | tmp integer; 9 | begin 10 | if (i_cluster_name = 'rw') then 11 | min_priority:=0; 12 | elsif (i_cluster_name = 'ro') then 13 | min_priority:=1; 14 | else 15 | raise exception 'Unknown cluster'; 16 | end if; 17 | for r in 18 | select distinct(part_id) from plproxy.parts order by part_id 19 | loop 20 | select min(priority) into tmp from plproxy.priorities p where p.part_id=r.part_id and priority>=min_priority; 21 | if (tmp is null or tmp >= 100) then 22 | min_priority:=0; 23 | end if; 24 | return next (select conn_string from plproxy.connections c, plproxy.priorities p where p.part_id=r.part_id and p.conn_id=c.conn_id and priority>=min_priority order by priority limit 1); 25 | end loop; 26 | return; 27 | end; 28 | $function$ 29 | -------------------------------------------------------------------------------- /samples/sql/30_get_cluster_version.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION plproxy.get_cluster_version(i_cluster_name text) 2 | RETURNS integer 3 | LANGUAGE plpgsql 4 | AS $function$ 5 | DECLARE 6 | ver integer; 7 | BEGIN 8 | IF i_cluster_name in ('rw', 'ro') THEN 9 | select version into ver from plproxy.versions; 10 | RETURN ver; 11 | END IF; 12 | RAISE EXCEPTION 'Unknown cluster'; 13 | END; 14 | $function$ 15 | -------------------------------------------------------------------------------- /samples/sql/30_inc_cluster_version.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION plproxy.inc_cluster_version() 2 | RETURNS trigger 3 | LANGUAGE plpgsql 4 | AS $function$ 5 | BEGIN 6 | update plproxy.versions set version = version + 1; 7 | return null; 8 | END; 9 | $function$; 10 | 11 | DROP TRIGGER IF EXISTS update_cluster_version on plproxy.priorities; 12 | CREATE TRIGGER update_cluster_version AFTER INSERT or DELETE or UPDATE on plproxy.priorities for each statement execute procedure plproxy.inc_cluster_version(); 13 | -------------------------------------------------------------------------------- /samples/sql/50_is_master.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION plproxy.is_master(i_key bigint) 2 | RETURNS smallint AS $$ 3 | CLUSTER 'rw'; 4 | RUN ON plproxy.select_part(i_key); 5 | TARGET public.is_master; 6 | $$ LANGUAGE plproxy; 7 | -------------------------------------------------------------------------------- /samples/sql/50_select_part.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION plproxy.select_part(i_key bigint) 2 | RETURNS integer 3 | LANGUAGE plpgsql 4 | AS $function$ 5 | declare 6 | part bigint; 7 | begin 8 | select part_id into part from plproxy.key_ranges where i_key between start_key and end_key; 9 | if (part is null) then 10 | raise exception 'No range for this key'; 11 | end if; 12 | return part; 13 | end; 14 | $function$ 15 | -------------------------------------------------------------------------------- /samples/sql/99_data.sql: -------------------------------------------------------------------------------- 1 | INSERT INTO plproxy.parts VALUES (0), (1); 2 | INSERT INTO plproxy.key_ranges VALUES (0, 0, 0, 511), (1, 1, 512, 1023); 3 | INSERT INTO plproxy.hosts (host_id, host_name, dc, prio_diff) VALUES 4 | (1, 'shard01-dc1.pgcheck.net', 'DC1', NULL), 5 | (2, 'shard01-dc2.pgcheck.net', 'DC2', NULL), 6 | (3, 'shard01-dc3.pgcheck.net', 'DC3', 5), 7 | (4, 'shard02-dc1.pgcheck.net', 'DC1', NULL), 8 | (5, 'shard02-dc2.pgcheck.net', 'DC2', NULL), 9 | (6, 'shard02-dc3.pgcheck.net', 'DC3', NULL); 10 | INSERT INTO plproxy.connections VALUES 11 | (1, 'host=shard01-dc1.pgcheck.net port=6432 dbname=db1'), 12 | (2, 'host=shard01-dc2.pgcheck.net port=6432 dbname=db1'), 13 | (3, 'host=shard01-dc3.pgcheck.net port=6432 dbname=db1'), 14 | (4, 'host=shard02-dc1.pgcheck.net port=6432 dbname=db1'), 15 | (5, 'host=shard02-dc2.pgcheck.net port=6432 dbname=db1'), 16 | (6, 'host=shard02-dc3.pgcheck.net port=6432 dbname=db1'); 17 | INSERT INTO plproxy.priorities VALUES 18 | (0, 1, 1, 100), 19 | (0, 2, 2, 10), 20 | (0, 3, 3, 20), 21 | (1, 4, 4, 10), 22 | (1, 5, 5, 0), 23 | (1, 6, 6, 20); 24 | -------------------------------------------------------------------------------- /tests/environment.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import docker 6 | 7 | from steps import moby 8 | 9 | 10 | def before_all(context): 11 | """ 12 | Setup environment 13 | """ 14 | # Connect to docker daemon 15 | context.docker = docker.from_env() 16 | 17 | context.containers = {} 18 | for container in context.docker.containers.list(): 19 | name = ''.join(container.name.split('_')[1:-1]) 20 | context.containers[name] = container 21 | 22 | context.plproxy_port = moby.get_container_tcp_port( 23 | moby.get_container_by_name('plproxy'), 6432) 24 | 25 | 26 | def after_step(context, step): 27 | debug = True if os.environ.get('DEBUG', 0) != 0 else False 28 | if debug and step.status == 'failed': 29 | # -- ENTER DEBUGGER: Zoom in on failure location. 30 | # NOTE: Use IPython debugger, same for pdb (basic python debugger). 31 | try: 32 | import ipdb 33 | except ImportError: 34 | import pdb as ipdb 35 | ipdb.post_mortem(step.exc_traceback) 36 | -------------------------------------------------------------------------------- /tests/features/main.feature: -------------------------------------------------------------------------------- 1 | Feature: Infrastructure works correctly 2 | 3 | Scenario: Pgcheck initial values are correct 4 | Given a deployed cluster 5 | Then connection strings for "rw" cluster are: 6 | | get_cluster_partitions | 7 | | host=shard01-dc1.pgcheck.net port=6432 dbname=db1 | 8 | | host=shard02-dc2.pgcheck.net port=6432 dbname=db1 | 9 | And connection strings for "ro" cluster are: 10 | | get_cluster_partitions | 11 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 12 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | 13 | 14 | 15 | Scenario: Dead host is handled correctly 16 | Given a deployed cluster 17 | When we pause "shard01-dc1" container 18 | Then within 5 seconds connection strings for "rw" cluster changes to: 19 | | get_cluster_partitions | 20 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 21 | | host=shard02-dc2.pgcheck.net port=6432 dbname=db1 | 22 | 23 | When we unpause "shard01-dc1" container 24 | Then within 5 seconds connection strings for "rw" cluster changes to: 25 | | get_cluster_partitions | 26 | | host=shard01-dc1.pgcheck.net port=6432 dbname=db1 | 27 | | host=shard02-dc2.pgcheck.net port=6432 dbname=db1 | 28 | 29 | When we pause "shard01-dc2" container 30 | Then within 5 seconds connection strings for "ro" cluster changes to: 31 | | get_cluster_partitions | 32 | | host=shard01-dc3.pgcheck.net port=6432 dbname=db1 | 33 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | 34 | 35 | When we unpause "shard01-dc2" container 36 | Then within 5 seconds connection strings for "ro" cluster changes to: 37 | | get_cluster_partitions | 38 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 39 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | 40 | 41 | 42 | Scenario: Delayed replica is handled correctly 43 | Given a deployed cluster 44 | When we pause replay on "shard02-dc1" 45 | Then within 20 seconds connection strings for "ro" cluster changes to: 46 | | get_cluster_partitions | 47 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 48 | | host=shard02-dc3.pgcheck.net port=6432 dbname=db1 | 49 | 50 | When we resume replay on "shard02-dc1" 51 | Then within 20 seconds connection strings for "ro" cluster changes to: 52 | | get_cluster_partitions | 53 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 54 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | 55 | 56 | 57 | @long 58 | Scenario: All delayed replics are handled correctly 59 | Given a deployed cluster 60 | When we pause replay on "shard02-dc1" 61 | And we pause replay on "shard02-dc3" 62 | Then within 100 seconds connection strings for "ro" cluster changes to: 63 | | get_cluster_partitions | 64 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 65 | | host=shard02-dc2.pgcheck.net port=6432 dbname=db1 | 66 | 67 | When we resume replay on "shard02-dc1" 68 | And we resume replay on "shard02-dc3" 69 | Then within 10 seconds connection strings for "ro" cluster changes to: 70 | | get_cluster_partitions | 71 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 72 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | 73 | 74 | 75 | Scenario: Overloaded host is handled correctly 76 | Given a deployed cluster 77 | When we open a lot of connections to "shard02-dc1" 78 | Then within 10 seconds connection strings for "ro" cluster changes to: 79 | | get_cluster_partitions | 80 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 81 | | host=shard02-dc3.pgcheck.net port=6432 dbname=db1 | 82 | 83 | When we close a lot of connections to "shard02-dc1" 84 | Then within 10 seconds connection strings for "ro" cluster changes to: 85 | | get_cluster_partitions | 86 | | host=shard01-dc2.pgcheck.net port=6432 dbname=db1 | 87 | | host=shard02-dc1.pgcheck.net port=6432 dbname=db1 | -------------------------------------------------------------------------------- /tests/pgcheck.featureset: -------------------------------------------------------------------------------- 1 | features/main.feature 2 | -------------------------------------------------------------------------------- /tests/requirements.txt: -------------------------------------------------------------------------------- 1 | psycopg2-binary 2 | docker 3 | git+https://github.com/behave/behave 4 | -------------------------------------------------------------------------------- /tests/steps/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yandex/pgcheck/977c96962f9e69bab655f18a41f6a05d5f6b292f/tests/steps/__init__.py -------------------------------------------------------------------------------- /tests/steps/database.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import contextlib 4 | import logging 5 | from collections import namedtuple 6 | 7 | import psycopg2 8 | 9 | LOG = logging.getLogger(__name__) 10 | 11 | 12 | class QueryResult(namedtuple('QueryResult', ['records', 'errcode', 'errmsg'])): 13 | """ 14 | Simple class to return results from PostgreSQL 15 | """ 16 | pass 17 | 18 | 19 | class Connection(object): 20 | """ 21 | A class for connections to PostgreSQL 22 | """ 23 | def __init__(self, connstring): 24 | self.connstring = connstring 25 | self._conn = psycopg2.connect(self.connstring) 26 | self._conn.autocommit = True 27 | self.errcode = None 28 | self.errmsg = None 29 | 30 | def __create_cursor(self): 31 | cursor = self._conn.cursor() 32 | return cursor 33 | 34 | def __exec_query(self, query, **kwargs): 35 | cur = self.__create_cursor() 36 | self.errcode = None 37 | self.errmsg = None 38 | try: 39 | cur.execute(query, kwargs) 40 | except psycopg2.Error as e: 41 | self.errcode = e.pgcode 42 | self.errmsg = e.pgerror 43 | return cur 44 | 45 | def __get_names(self, cur): 46 | return [r[0].lower() for r in cur.description] 47 | 48 | def __plain_format(self, cur): 49 | names = self.__get_names(cur) 50 | for row in cur.fetchall(): 51 | yield dict(zip(names, tuple(row))) 52 | 53 | def get(self, query, **kwargs): 54 | """ 55 | Method to execute query and return result 56 | """ 57 | with contextlib.closing(self.__exec_query(query, **kwargs)) as cur: 58 | records = list(self.__plain_format(cur)) if self.errcode is None else [] 59 | return QueryResult( 60 | records, 61 | self.errcode, 62 | self.errmsg 63 | ) 64 | 65 | def get_func(self, name, **kwargs): 66 | """ 67 | Method to execute function and return result 68 | """ 69 | arg_names = ', '.join('{0} => %({0})s'.format(k) for k in kwargs) 70 | q = 'SELECT * FROM {name}({arg_names})'.format( 71 | name=name, 72 | arg_names=arg_names, 73 | ) 74 | res = self.get(q, **kwargs) 75 | LOG.info(q, kwargs) 76 | LOG.info(res) 77 | return res 78 | -------------------------------------------------------------------------------- /tests/steps/helpers.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import logging 5 | import time 6 | import threading 7 | 8 | LOG = logging.getLogger('helpers') 9 | 10 | 11 | def retry_on_assert(function): 12 | """ 13 | Decorator for retrying. It catches AssertionError 14 | while timeout not exceeded. 15 | """ 16 | 17 | def wrapper(*args, **kwargs): 18 | timeout = kwargs['timeout'] 19 | max_time = time.time() + timeout 20 | while True: 21 | try: 22 | return function(*args, **kwargs) 23 | except AssertionError as error: 24 | LOG.info('%s call asserted: %s', function.__name__, error) 25 | # raise exception if timeout exceeded 26 | if time.time() > max_time: 27 | raise 28 | time.sleep(1) 29 | 30 | return wrapper 31 | 32 | 33 | def assert_results_are_equal(expected_table, result): 34 | """ 35 | Function that asserts if results in two tables are not the same 36 | """ 37 | expected_rows_num = len(expected_table.rows) 38 | actual_rows_num = len(result.records) 39 | assert expected_rows_num == actual_rows_num, \ 40 | "Expected {} rows, got {}".format(expected_rows_num, 41 | actual_rows_num) 42 | 43 | if expected_rows_num > 0: 44 | expected_headings = expected_table.headings 45 | actual_headings = list(result.records[0].keys()) 46 | LOG.info(expected_headings) 47 | assert expected_headings == actual_headings, \ 48 | "Tables structure mismatch, expected: {}".format(expected_headings) 49 | 50 | for i in range(0, expected_rows_num): 51 | expected = {} 52 | for j in range(0, len(expected_headings)): 53 | expected[expected_headings[j]] = expected_table.rows[i][j] 54 | 55 | actual = result.records[i] 56 | assert expected == actual, "Incorrect result in line " \ 57 | "{}, expected {}, got {}".format(i, expected, actual) 58 | 59 | 60 | def run_threads(count, func, *args, **kwargs): 61 | """ 62 | Function to start `count` threads and run `func` in each on them 63 | """ 64 | for _ in range(0, count): 65 | thread = threading.Thread(target=func, args=args, kwargs=kwargs) 66 | thread.start() 67 | -------------------------------------------------------------------------------- /tests/steps/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import logging 5 | import time 6 | 7 | import database 8 | import helpers 9 | import moby 10 | 11 | LOG = logging.getLogger(__name__) 12 | 13 | TIMEOUT = 10 14 | ITERATIONS = 6 15 | PLPROXY_CONN_STRING = "host=localhost port={port} dbname=db1 " + \ 16 | "user=postgres connect_timeout=1" 17 | 18 | 19 | @given(u'a deployed cluster') 20 | def given_deployed_cluster(context): 21 | """ 22 | Checks that cluster has been started up 23 | 24 | Since everything is started with compose here we should check 25 | that everything has been deployed to PL/Proxy host 26 | """ 27 | for _ in range(0, ITERATIONS): 28 | try: 29 | conn = database.Connection( 30 | PLPROXY_CONN_STRING.format(port=context.plproxy_port)) 31 | res = conn.get('SELECT * FROM plproxy.priorities') 32 | LOG.info(res) 33 | if res.errcode is None and res.records: 34 | return 35 | time.sleep(TIMEOUT) 36 | except Exception as err: # pylint: disable=W0703 37 | LOG.info(err) 38 | time.sleep(TIMEOUT) 39 | 40 | 41 | @then(u'connection strings for "{cluster}" cluster are') 42 | def then_connection_strings_are(context, cluster): 43 | """ 44 | Function to check connection strings 45 | """ 46 | conn = database.Connection( 47 | PLPROXY_CONN_STRING.format(port=context.plproxy_port)) 48 | res = conn.get_func( 49 | 'plproxy.get_cluster_partitions', i_cluster_name=cluster) 50 | LOG.info(context.table.rows) 51 | helpers.assert_results_are_equal(context.table, res) 52 | 53 | 54 | @then(u'within {timeout:d} seconds connection strings for "{cluster}" cluster changes to') 55 | @helpers.retry_on_assert 56 | def then_connection_strings_become(context, timeout, cluster): # pylint: disable=W0613 57 | """ 58 | Simple helper with retries on asserts 59 | """ 60 | then_connection_strings_are(context, cluster) 61 | 62 | 63 | @when(u'we {action} "{container_name}" container') 64 | def when_action_container(context, action, container_name): 65 | """ 66 | Function that performs actions with docker containers 67 | """ 68 | container = context.containers[container_name] 69 | moby.container_action(container, action) 70 | 71 | 72 | @when(u'we {action} replay on "{container_name}"') 73 | def when_action_replay(context, action, container_name): # pylint: disable=W0613 74 | """ 75 | Function that performs actions with WAL replay on replica 76 | """ 77 | conn_string = moby.container_conn_string(container_name) 78 | statement = 'SELECT pg_wal_replay_{action}()'.format(action=action) 79 | 80 | conn = database.Connection(conn_string) 81 | res = conn.get(statement) 82 | if res.errcode: 83 | LOG.info(res) 84 | raise RuntimeError('Could not execute statement') 85 | 86 | 87 | @when(u'we {action} a lot of connections to "{container_name}"') 88 | def when_action_connections(context, action, container_name): # pylint: disable=W0613 89 | """ 90 | Function to open/close many connections to container PostgreSQL 91 | """ 92 | 93 | def execute_pg_sleep(conn_string): 94 | """ 95 | Simple helper to call inside thread 96 | """ 97 | conn = database.Connection(conn_string) 98 | res = conn.get('SELECT now(), pg_sleep(60)') 99 | LOG.info(res) 100 | 101 | conn_string = moby.container_conn_string(container_name) 102 | 103 | if action == 'open': 104 | helpers.run_threads(50, execute_pg_sleep, conn_string=conn_string) 105 | elif action == 'close': 106 | conn = database.Connection(conn_string) 107 | res = conn.get("""SELECT pg_terminate_backend(pid) FROM pg_stat_activity 108 | WHERE query ~ 'pg_sleep' AND pid != pg_backend_pid()""") 109 | LOG.info(res) -------------------------------------------------------------------------------- /tests/steps/moby.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import docker 5 | 6 | DOCKER = docker.from_env() 7 | 8 | def get_container_by_name(name): 9 | """ 10 | Returns container object 11 | """ 12 | for container in DOCKER.containers.list(): 13 | if container.attrs['Config']['Hostname'] == name: 14 | return container 15 | 16 | 17 | def get_container_tcp_port(container, port): 18 | """ 19 | Returns exposed to host container port 20 | """ 21 | binding = container.attrs['NetworkSettings']['Ports'].get( 22 | '{port}/tcp'.format(port=port)) 23 | if binding: 24 | return binding[0]['HostPort'] 25 | 26 | 27 | def container_action(container, action): 28 | """ 29 | Performs desired action with container 30 | """ 31 | if action == 'pause': 32 | container.pause() 33 | elif action == 'unpause': 34 | container.unpause() 35 | else: 36 | raise RuntimeError('Unsupported action') 37 | 38 | 39 | def container_conn_string(container): 40 | """ 41 | Returns connection string to container 42 | """ 43 | port = get_container_tcp_port(get_container_by_name(container), 6432) 44 | conn_string = "host=localhost port={port} dbname=db1 user=postgres " + \ 45 | "connect_timeout=1" 46 | return conn_string.format(port=port) --------------------------------------------------------------------------------