├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | CC0 1.0 Universal 2 | 3 | Statement of Purpose 4 | 5 | The laws of most jurisdictions throughout the world automatically confer 6 | exclusive Copyright and Related Rights (defined below) upon the creator and 7 | subsequent owner(s) (each and all, an "owner") of an original work of 8 | authorship and/or a database (each, a "Work"). 9 | 10 | Certain owners wish to permanently relinquish those rights to a Work for the 11 | purpose of contributing to a commons of creative, cultural and scientific 12 | works ("Commons") that the public can reliably and without fear of later 13 | claims of infringement build upon, modify, incorporate in other works, reuse 14 | and redistribute as freely as possible in any form whatsoever and for any 15 | purposes, including without limitation commercial purposes. These owners may 16 | contribute to the Commons to promote the ideal of a free culture and the 17 | further production of creative, cultural and scientific works, or to gain 18 | reputation or greater distribution for their Work in part through the use and 19 | efforts of others. 20 | 21 | For these and/or other purposes and motivations, and without any expectation 22 | of additional consideration or compensation, the person associating CC0 with a 23 | Work (the "Affirmer"), to the extent that he or she is an owner of Copyright 24 | and Related Rights in the Work, voluntarily elects to apply CC0 to the Work 25 | and publicly distribute the Work under its terms, with knowledge of his or her 26 | Copyright and Related Rights in the Work and the meaning and intended legal 27 | effect of CC0 on those rights. 28 | 29 | 1. Copyright and Related Rights. A Work made available under CC0 may be 30 | protected by copyright and related or neighboring rights ("Copyright and 31 | Related Rights"). Copyright and Related Rights include, but are not limited 32 | to, the following: 33 | 34 | i. the right to reproduce, adapt, distribute, perform, display, communicate, 35 | and translate a Work; 36 | 37 | ii. moral rights retained by the original author(s) and/or performer(s); 38 | 39 | iii. publicity and privacy rights pertaining to a person's image or likeness 40 | depicted in a Work; 41 | 42 | iv. rights protecting against unfair competition in regards to a Work, 43 | subject to the limitations in paragraph 4(a), below; 44 | 45 | v. rights protecting the extraction, dissemination, use and reuse of data in 46 | a Work; 47 | 48 | vi. database rights (such as those arising under Directive 96/9/EC of the 49 | European Parliament and of the Council of 11 March 1996 on the legal 50 | protection of databases, and under any national implementation thereof, 51 | including any amended or successor version of such directive); and 52 | 53 | vii. other similar, equivalent or corresponding rights throughout the world 54 | based on applicable law or treaty, and any national implementations thereof. 55 | 56 | 2. Waiver. To the greatest extent permitted by, but not in contravention of, 57 | applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and 58 | unconditionally waives, abandons, and surrenders all of Affirmer's Copyright 59 | and Related Rights and associated claims and causes of action, whether now 60 | known or unknown (including existing as well as future claims and causes of 61 | action), in the Work (i) in all territories worldwide, (ii) for the maximum 62 | duration provided by applicable law or treaty (including future time 63 | extensions), (iii) in any current or future medium and for any number of 64 | copies, and (iv) for any purpose whatsoever, including without limitation 65 | commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes 66 | the Waiver for the benefit of each member of the public at large and to the 67 | detriment of Affirmer's heirs and successors, fully intending that such Waiver 68 | shall not be subject to revocation, rescission, cancellation, termination, or 69 | any other legal or equitable action to disrupt the quiet enjoyment of the Work 70 | by the public as contemplated by Affirmer's express Statement of Purpose. 71 | 72 | 3. Public License Fallback. Should any part of the Waiver for any reason be 73 | judged legally invalid or ineffective under applicable law, then the Waiver 74 | shall be preserved to the maximum extent permitted taking into account 75 | Affirmer's express Statement of Purpose. In addition, to the extent the Waiver 76 | is so judged Affirmer hereby grants to each affected person a royalty-free, 77 | non transferable, non sublicensable, non exclusive, irrevocable and 78 | unconditional license to exercise Affirmer's Copyright and Related Rights in 79 | the Work (i) in all territories worldwide, (ii) for the maximum duration 80 | provided by applicable law or treaty (including future time extensions), (iii) 81 | in any current or future medium and for any number of copies, and (iv) for any 82 | purpose whatsoever, including without limitation commercial, advertising or 83 | promotional purposes (the "License"). The License shall be deemed effective as 84 | of the date CC0 was applied by Affirmer to the Work. Should any part of the 85 | License for any reason be judged legally invalid or ineffective under 86 | applicable law, such partial invalidity or ineffectiveness shall not 87 | invalidate the remainder of the License, and in such case Affirmer hereby 88 | affirms that he or she will not (i) exercise any of his or her remaining 89 | Copyright and Related Rights in the Work or (ii) assert any associated claims 90 | and causes of action with respect to the Work, in either case contrary to 91 | Affirmer's express Statement of Purpose. 92 | 93 | 4. Limitations and Disclaimers. 94 | 95 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 96 | surrendered, licensed or otherwise affected by this document. 97 | 98 | b. Affirmer offers the Work as-is and makes no representations or warranties 99 | of any kind concerning the Work, express, implied, statutory or otherwise, 100 | including without limitation warranties of title, merchantability, fitness 101 | for a particular purpose, non infringement, or the absence of latent or 102 | other defects, accuracy, or the present or absence of errors, whether or not 103 | discoverable, all to the greatest extent permissible under applicable law. 104 | 105 | c. Affirmer disclaims responsibility for clearing rights of other persons 106 | that may apply to the Work or any use thereof, including without limitation 107 | any person's Copyright and Related Rights in the Work. Further, Affirmer 108 | disclaims responsibility for obtaining any necessary consents, permissions 109 | or other rights required for any use of the Work. 110 | 111 | d. Affirmer understands and acknowledges that Creative Commons is not a 112 | party to this document and has no duty or obligation with respect to this 113 | CC0 or use of the Work. 114 | 115 | For more information, please see 116 | 117 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PostgreSQL Commandments 2 | 3 | [](https://www.postgresql.org/) 4 | 5 | Short axioms about how to use Postgres. 6 | 7 |
8 | Disclaimer 9 | 10 | - These are not recommendations from the official PostgreSQL community. 11 | 12 | - They are my personal observations based on my own experience as PostgreSQL DBA for the past 10 years. 13 | 14 | - You can follow or ignore them but the main course of these commandments - prioritise data safety over performance. 15 |
16 | 17 | # 18 | 19 |
20 | Avoid long transactions. 21 | 22 | Long, idle and especially write transactions acquire and hold locks on tuples, preventing their cleanup by vacuum. 23 | 24 | Look at the performance of the pgbench benchmark (8 clients read and write to the database during 1 hour): 25 | 26 | ``` 27 | pgbench -c8 -P 60 -T 3600 -U postgres pgbench 28 | starting vacuum...end. 29 | progress: 60.0 s, 9506.3 tps, lat 0.841 ms stddev 0.390 30 | progress: 120.0 s, 5262.1 tps, lat 1.520 ms stddev 0.517 31 | progress: 180.0 s, 3801.8 tps, lat 2.104 ms stddev 0.757 32 | progress: 240.0 s, 2960.0 tps, lat 2.703 ms stddev 0.830 33 | progress: 300.0 s, 2575.8 tps, lat 3.106 ms stddev 0.891 34 | ``` 35 | ...in the end 36 | ``` 37 | progress: 3300.0 s, 759.5 tps, lat 10.533 ms stddev 2.554 38 | progress: 3360.0 s, 751.8 tps, lat 10.642 ms stddev 2.604 39 | progress: 3420.0 s, 743.6 tps, lat 10.759 ms stddev 2.655 40 | progress: 3480.0 s, 739.1 tps, lat 10.824 ms stddev 2.662 41 | progress: 3540.0 s, 742.5 tps, lat 10.774 ms stddev 2.579 42 | progress: 3600.0 s, 868.2 tps, lat 9.215 ms stddev 2.569 43 | ``` 44 | As you can see, performance is dropped dramatically over a short period of time. 45 | 46 | Now, look at the vacuum logs. 47 | ``` 48 | tuples: 0 removed, 692428 remain, 691693 are dead but not yet removable, oldest xmin: 62109160 49 | tuples: 0 removed, 984009 remain, 983855 are dead but not yet removable, oldest xmin: 62109160 50 | tuples: 0 removed, 1176821 remain, 1176821 are dead but not yet removable, oldest xmin: 62109160 51 | tuples: 0 removed, 1494122 remain, 1494122 are dead but not yet removable, oldest xmin: 62109160 52 | tuples: 0 removed, 2022284 remain, 2022284 are dead but not yet removable, oldest xmin: 62109160 53 | tuples: 0 removed, 2756298 remain, 2756153 are dead but not yet removable, oldest xmin: 62109160 54 | tuples: 0 removed, 3500913 remain, 3500693 are dead but not yet removable, oldest xmin: 62109160 55 | tuples: 0 removed, 4631448 remain, 4631354 are dead but not yet removable, oldest xmin: 62109160 56 | tuples: 0 removed, 5377941 remain, 5374941 are dead but not yet removable, oldest xmin: 62109160 57 | ``` 58 | Pay attention on the number of dead but not yet removable rows. Their number increases continuously during the benchmark. Also, you can see that oldest the xmin is constant. 59 | 60 | --- 61 |
62 | 63 |
64 | Avoid idle transactions. 65 | 66 | "Idle transactions" is the special case when an application starts transactions with `BEGIN` command and doesn't close them correctly (with `COMMIT`,`ROLLBACK` or `END` command). This might occur due to many different reasons on the application side: absent or wrong error handling inside application code when working with transactions, working with remote data sources like other databases or API from open transactions, etc. 67 | Negative effects are the same as in case of long write transactions - performance degradation. See the example above. 68 | 69 | --- 70 |
71 | 72 | Don't disable autovacuum. 73 | 74 | Don't use 'fsync=off' in production. 75 | 76 | Don't remove anything from $DATADIR. 77 | 78 |
79 | Don't use 'kill -9' against Postgres processes. 80 | 81 | PostgreSQL’s official documentation states: 82 | 83 | > It is best not to use SIGKILL to shut down the server. Doing so will prevent the server from releasing shared memory and semaphores, which might then have to be done manually before a new server can be started. 84 | 85 | Moreover, using SIGKILL against even a single Postgres backend forces to immediately terminate all other backends, re-initialize internal structures and run recovery from last check point, at which database cluster is not available for clients and applications until recovery ends. 86 | 87 | In the example below, you can see how Postgres handles SIGKILL: 88 | 1) Process with PID 9774 is terminated by SIGKILL. 89 | 2) Postgres terminates the rest of its processes (20 processes in total). 90 | 3) Reinitializes and runs automatic recovery process (which may take a while in various scenarios). 91 | 4) Finishes recovery and starts accepting connections. (edited) 92 | 93 | ``` 94 | 1549 @ from [] LOG: server process (PID 9774) was terminated by signal 9: Killed 95 | 1549 @ from [] DETAIL: Failed process was running: SELECT abalance FROM pgbench_accounts WHERE aid = 729760; 96 | 1549 @ from [] LOG: terminating any other active server processes 97 | 9773 postgres@pgbench from [local] [idle] WARNING: terminating connection because of crash of another server process 98 | 9773 postgres@pgbench from [local] [idle] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 99 | 9773 postgres@pgbench from [local] [idle] HINT: In a moment you should be able to reconnect to the database and repeat your command. 100 | 1816 @ from [] WARNING: terminating connection because of crash of another server process 101 | 1816 @ from [] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 102 | 1816 @ from [] HINT: In a moment you should be able to reconnect to the database and repeat your command. 103 | 9768 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 104 | 9768 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 105 | 9768 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 106 | 9782 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 107 | 9782 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 108 | 9782 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 109 | 9764 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 110 | 9764 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 111 | 9764 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 112 | 9770 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 113 | 9770 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 114 | 9770 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 115 | 9769 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 116 | 9769 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 117 | 9769 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 118 | 9772 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 119 | 9772 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 120 | 9772 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 121 | 9779 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 122 | 9779 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 123 | 9779 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 124 | 9780 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 125 | 9780 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 126 | 9780 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 127 | 9775 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 128 | 9775 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 129 | 9775 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 130 | 9776 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 131 | 9776 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 132 | 9776 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 133 | 9771 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 134 | 9771 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 135 | 9771 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 136 | 9766 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 137 | 9766 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 138 | 9766 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 139 | 9765 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 140 | 9765 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 141 | 9765 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 142 | 9781 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 143 | 9781 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 144 | 9781 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 145 | 9777 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 146 | 9777 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 147 | 9777 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 148 | 9767 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 149 | 9767 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 150 | 9767 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 151 | 9778 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 152 | 9778 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 153 | 9778 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 154 | 9783 postgres@pgbench from [local] [SELECT] WARNING: terminating connection because of crash of another server process 155 | 9783 postgres@pgbench from [local] [SELECT] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 156 | 9783 postgres@pgbench from [local] [SELECT] HINT: In a moment you should be able to reconnect to the database and repeat your command. 157 | 1549 @ from [] LOG: all server processes terminated; reinitializing 158 | 9817 @ from [] LOG: database system was interrupted; last known up at 2018-12-03 15:01:23 +05 159 | 9817 @ from [] LOG: database system was not properly shut down; automatic recovery in progress 160 | 9817 @ from [] LOG: redo starts at 8/F3C72E70 161 | 9817 @ from [] LOG: invalid record length at 8/F3C7A390: wanted 24, got 0 162 | 9817 @ from [] LOG: redo done at 8/F3C7A358 163 | 9817 @ from [] LOG: last completed transaction was at log time 2018-12-03 21:39:14.667678+05 164 | 9817 @ from [] LOG: checkpoint starting: end-of-recovery immediate 165 | 9817 @ from [] LOG: checkpoint complete: wrote 7 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.000 s, sync=0.016 s, total=0.042 s; sync files=7, longest=0.008 s, average=0.002 s; distance=29 kB, estimate=29 kB 166 | 1549 @ from [] LOG: database system is ready to accept connections 167 | ``` 168 | --- 169 |
170 | 171 | 172 | Don't delete rows by billions at a time. 173 | 174 | Avoid creating unnecessary indexes. 175 | 176 | The more indexes table has, the slower you can write into it. 177 | 178 | Don't use 'listen_addresses = *' on hosts with public access. 179 | 180 | Use connection pooler when there are too many connections. 181 | 182 | Postgres doesn't 'hang' at shutdown... It has *checkpoint*. 183 | 184 | Don't use minor releases that are less than 3, in production. 185 | 186 | VACUUM FULL completely blocks access to the table. 187 | 188 | Use CONCURRENTLY as much as possible. 189 | 190 | The bigger the table, the slower the count(*). 191 | 192 | Hot standby != backup. 193 | 194 | Don't trust non-validated backup. 195 | 196 |
197 | Use partitioning for archived data. 198 | Let’s assume we have some logs or events stored in the database 199 | 200 | ``` 201 | =# select pg_size_pretty(pg_relation_size('history_log')); 202 | pg_size_pretty 203 | ---------------- 204 | 155 GB 205 | 206 | =# select count(*) from history_log; 207 | count 208 | ----------- 209 | 2102342910 210 | ``` 211 | 212 | At some point, we’ll decide to clean old events 213 | 214 | ``` 215 | =# delete from history_log where updated_at < '2018-11-01'; 216 | DELETE 1885782465 217 | Time: 725220.719 ms (12:05.221) 218 | ``` 219 | 220 | The query would take twelve minutes to complete. However, during this action there is a less obvious process that takes place - query would generate certain amount of WAL that will then need to be transferred into all standbys. 221 | Ok, let’s check how many rows there are in the table now. 222 | 223 | ``` 224 | =# select count(*) from history_log; 225 | count 226 | ----------- 227 | 216560445 228 | 229 | =# select 100 * 1885782465::bigint / 2102342910; 230 | ?column? 231 | ---------- 232 | 89 233 | ``` 234 | 235 | It seems we deleted something around of 89% of the table. Let’s check its size. 236 | 237 | ``` 238 | select pg_size_pretty(pg_relation_size('history_log')); 239 | pg_size_pretty 240 | ---------------- 241 | 155 GB 242 | ``` 243 | 244 | Huh, the size hasn’t been changed?! 245 | 246 | The thing is, Postgres never performs real deletion. It just marks rows as removed. Later on, space occupied by these “removed” rows will be cleared by vacuum and available space can again be used for new rows, however, this space still belongs to table (in some rare circumstances, table can be truncated and free space will returned to the file system). 247 | 248 | Using partitioning with historical data allow you 1) to drop old data quickly, 2) without overhead related to WAL generation 3) and free up space immediately. 249 | 250 | --- 251 |
252 | 253 | Upstream and master candidate should be the same. 254 | 255 | Don't re-invent queues, use PgQ. 256 | 257 |
258 | Keep logs outside of $DATADIR. 259 | 260 | Keeping logs in $DATADIR is configured by default in most of installations and has some drawbacks: 261 | - logs are included into the backups made by pg_basebackup-based backup tools (barman, wal-g, etc.) 262 | - in case of writing too many logs: 263 | - logs might take too much space and postgres might crash due to 'No left space on device' error. 264 | - writing logs might add extra load on IO subsystem and performance of user queries might suffer. 265 | 266 | Better configuration is placing logs outside of $DATADIR, for example into `/var/log/postgresql` 267 | 268 | --- 269 |
270 | # 271 | 272 | :elephant: Contributions welcome. Add commandments or ideas through [pull requests](https://github.com/lesovsky/postgres-commandments/pulls) or create an [issue](https://github.com/lesovsky/postgres-commandments/issues) to start a discussion. 273 | --------------------------------------------------------------------------------