├── COPYING.GPLv3 ├── COPYING.LGPL ├── README.md ├── bin ├── cls_cleanup ├── cls_maintenance ├── cls_maintenance_node ├── cls_master ├── cls_online_switch ├── cls_rebuild_slave ├── cls_repair_by_pg_rewind ├── cls_reset_master ├── cls_standby_node ├── cls_start ├── cls_status ├── cls_stop ├── cls_unmaintenance ├── cls_unmaintenance_node └── cls_unstandby_node ├── doc ├── lvsdr.md └── muti_with_lvs.md ├── gencfg.sh ├── install.sh ├── lib └── common.sh ├── ra ├── expgsql ├── lvsdr └── lvsdr-realsvr ├── setup.sh ├── template ├── config_dual.ini.sample ├── config_muti.ini.sample ├── config_muti_with_lvs.ini.sample ├── dual.pcs.template ├── muti.pcs.template └── muti_with_lvs.pcs.template ├── test └── failover_test.py └── tools ├── check_active_slave ├── distlock ├── distlock_ddl.sql ├── distlock_init.sql └── get_active_slaves /COPYING.GPLv3: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /COPYING.LGPL: -------------------------------------------------------------------------------- 1 | GNU LESSER GENERAL PUBLIC LICENSE 2 | Version 2.1, February 1999 3 | 4 | Copyright (C) 1991, 1999 Free Software Foundation, Inc. 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | [This is the first released version of the Lesser GPL. It also counts 10 | as the successor of the GNU Library Public License, version 2, hence 11 | the version number 2.1.] 12 | 13 | Preamble 14 | 15 | The licenses for most software are designed to take away your 16 | freedom to share and change it. By contrast, the GNU General Public 17 | Licenses are intended to guarantee your freedom to share and change 18 | free software--to make sure the software is free for all its users. 19 | 20 | This license, the Lesser General Public License, applies to some 21 | specially designated software packages--typically libraries--of the 22 | Free Software Foundation and other authors who decide to use it. You 23 | can use it too, but we suggest you first think carefully about whether 24 | this license or the ordinary General Public License is the better 25 | strategy to use in any particular case, based on the explanations below. 26 | 27 | When we speak of free software, we are referring to freedom of use, 28 | not price. Our General Public Licenses are designed to make sure that 29 | you have the freedom to distribute copies of free software (and charge 30 | for this service if you wish); that you receive source code or can get 31 | it if you want it; that you can change the software and use pieces of 32 | it in new free programs; and that you are informed that you can do 33 | these things. 34 | 35 | To protect your rights, we need to make restrictions that forbid 36 | distributors to deny you these rights or to ask you to surrender these 37 | rights. These restrictions translate to certain responsibilities for 38 | you if you distribute copies of the library or if you modify it. 39 | 40 | For example, if you distribute copies of the library, whether gratis 41 | or for a fee, you must give the recipients all the rights that we gave 42 | you. You must make sure that they, too, receive or can get the source 43 | code. If you link other code with the library, you must provide 44 | complete object files to the recipients, so that they can relink them 45 | with the library after making changes to the library and recompiling 46 | it. And you must show them these terms so they know their rights. 47 | 48 | We protect your rights with a two-step method: (1) we copyright the 49 | library, and (2) we offer you this license, which gives you legal 50 | permission to copy, distribute and/or modify the library. 51 | 52 | To protect each distributor, we want to make it very clear that 53 | there is no warranty for the free library. Also, if the library is 54 | modified by someone else and passed on, the recipients should know 55 | that what they have is not the original version, so that the original 56 | author's reputation will not be affected by problems that might be 57 | introduced by others. 58 | 59 | Finally, software patents pose a constant threat to the existence of 60 | any free program. We wish to make sure that a company cannot 61 | effectively restrict the users of a free program by obtaining a 62 | restrictive license from a patent holder. Therefore, we insist that 63 | any patent license obtained for a version of the library must be 64 | consistent with the full freedom of use specified in this license. 65 | 66 | Most GNU software, including some libraries, is covered by the 67 | ordinary GNU General Public License. This license, the GNU Lesser 68 | General Public License, applies to certain designated libraries, and 69 | is quite different from the ordinary General Public License. We use 70 | this license for certain libraries in order to permit linking those 71 | libraries into non-free programs. 72 | 73 | When a program is linked with a library, whether statically or using 74 | a shared library, the combination of the two is legally speaking a 75 | combined work, a derivative of the original library. The ordinary 76 | General Public License therefore permits such linking only if the 77 | entire combination fits its criteria of freedom. The Lesser General 78 | Public License permits more lax criteria for linking other code with 79 | the library. 80 | 81 | We call this license the "Lesser" General Public License because it 82 | does Less to protect the user's freedom than the ordinary General 83 | Public License. It also provides other free software developers Less 84 | of an advantage over competing non-free programs. These disadvantages 85 | are the reason we use the ordinary General Public License for many 86 | libraries. However, the Lesser license provides advantages in certain 87 | special circumstances. 88 | 89 | For example, on rare occasions, there may be a special need to 90 | encourage the widest possible use of a certain library, so that it becomes 91 | a de-facto standard. To achieve this, non-free programs must be 92 | allowed to use the library. A more frequent case is that a free 93 | library does the same job as widely used non-free libraries. In this 94 | case, there is little to gain by limiting the free library to free 95 | software only, so we use the Lesser General Public License. 96 | 97 | In other cases, permission to use a particular library in non-free 98 | programs enables a greater number of people to use a large body of 99 | free software. For example, permission to use the GNU C Library in 100 | non-free programs enables many more people to use the whole GNU 101 | operating system, as well as its variant, the GNU/Linux operating 102 | system. 103 | 104 | Although the Lesser General Public License is Less protective of the 105 | users' freedom, it does ensure that the user of a program that is 106 | linked with the Library has the freedom and the wherewithal to run 107 | that program using a modified version of the Library. 108 | 109 | The precise terms and conditions for copying, distribution and 110 | modification follow. Pay close attention to the difference between a 111 | "work based on the library" and a "work that uses the library". The 112 | former contains code derived from the library, whereas the latter must 113 | be combined with the library in order to run. 114 | 115 | GNU LESSER GENERAL PUBLIC LICENSE 116 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 117 | 118 | 0. This License Agreement applies to any software library or other 119 | program which contains a notice placed by the copyright holder or 120 | other authorized party saying it may be distributed under the terms of 121 | this Lesser General Public License (also called "this License"). 122 | Each licensee is addressed as "you". 123 | 124 | A "library" means a collection of software functions and/or data 125 | prepared so as to be conveniently linked with application programs 126 | (which use some of those functions and data) to form executables. 127 | 128 | The "Library", below, refers to any such software library or work 129 | which has been distributed under these terms. A "work based on the 130 | Library" means either the Library or any derivative work under 131 | copyright law: that is to say, a work containing the Library or a 132 | portion of it, either verbatim or with modifications and/or translated 133 | straightforwardly into another language. (Hereinafter, translation is 134 | included without limitation in the term "modification".) 135 | 136 | "Source code" for a work means the preferred form of the work for 137 | making modifications to it. For a library, complete source code means 138 | all the source code for all modules it contains, plus any associated 139 | interface definition files, plus the scripts used to control compilation 140 | and installation of the library. 141 | 142 | Activities other than copying, distribution and modification are not 143 | covered by this License; they are outside its scope. The act of 144 | running a program using the Library is not restricted, and output from 145 | such a program is covered only if its contents constitute a work based 146 | on the Library (independent of the use of the Library in a tool for 147 | writing it). Whether that is true depends on what the Library does 148 | and what the program that uses the Library does. 149 | 150 | 1. You may copy and distribute verbatim copies of the Library's 151 | complete source code as you receive it, in any medium, provided that 152 | you conspicuously and appropriately publish on each copy an 153 | appropriate copyright notice and disclaimer of warranty; keep intact 154 | all the notices that refer to this License and to the absence of any 155 | warranty; and distribute a copy of this License along with the 156 | Library. 157 | 158 | You may charge a fee for the physical act of transferring a copy, 159 | and you may at your option offer warranty protection in exchange for a 160 | fee. 161 | 162 | 2. You may modify your copy or copies of the Library or any portion 163 | of it, thus forming a work based on the Library, and copy and 164 | distribute such modifications or work under the terms of Section 1 165 | above, provided that you also meet all of these conditions: 166 | 167 | a) The modified work must itself be a software library. 168 | 169 | b) You must cause the files modified to carry prominent notices 170 | stating that you changed the files and the date of any change. 171 | 172 | c) You must cause the whole of the work to be licensed at no 173 | charge to all third parties under the terms of this License. 174 | 175 | d) If a facility in the modified Library refers to a function or a 176 | table of data to be supplied by an application program that uses 177 | the facility, other than as an argument passed when the facility 178 | is invoked, then you must make a good faith effort to ensure that, 179 | in the event an application does not supply such function or 180 | table, the facility still operates, and performs whatever part of 181 | its purpose remains meaningful. 182 | 183 | (For example, a function in a library to compute square roots has 184 | a purpose that is entirely well-defined independent of the 185 | application. Therefore, Subsection 2d requires that any 186 | application-supplied function or table used by this function must 187 | be optional: if the application does not supply it, the square 188 | root function must still compute square roots.) 189 | 190 | These requirements apply to the modified work as a whole. If 191 | identifiable sections of that work are not derived from the Library, 192 | and can be reasonably considered independent and separate works in 193 | themselves, then this License, and its terms, do not apply to those 194 | sections when you distribute them as separate works. But when you 195 | distribute the same sections as part of a whole which is a work based 196 | on the Library, the distribution of the whole must be on the terms of 197 | this License, whose permissions for other licensees extend to the 198 | entire whole, and thus to each and every part regardless of who wrote 199 | it. 200 | 201 | Thus, it is not the intent of this section to claim rights or contest 202 | your rights to work written entirely by you; rather, the intent is to 203 | exercise the right to control the distribution of derivative or 204 | collective works based on the Library. 205 | 206 | In addition, mere aggregation of another work not based on the Library 207 | with the Library (or with a work based on the Library) on a volume of 208 | a storage or distribution medium does not bring the other work under 209 | the scope of this License. 210 | 211 | 3. You may opt to apply the terms of the ordinary GNU General Public 212 | License instead of this License to a given copy of the Library. To do 213 | this, you must alter all the notices that refer to this License, so 214 | that they refer to the ordinary GNU General Public License, version 2, 215 | instead of to this License. (If a newer version than version 2 of the 216 | ordinary GNU General Public License has appeared, then you can specify 217 | that version instead if you wish.) Do not make any other change in 218 | these notices. 219 | 220 | Once this change is made in a given copy, it is irreversible for 221 | that copy, so the ordinary GNU General Public License applies to all 222 | subsequent copies and derivative works made from that copy. 223 | 224 | This option is useful when you wish to copy part of the code of 225 | the Library into a program that is not a library. 226 | 227 | 4. You may copy and distribute the Library (or a portion or 228 | derivative of it, under Section 2) in object code or executable form 229 | under the terms of Sections 1 and 2 above provided that you accompany 230 | it with the complete corresponding machine-readable source code, which 231 | must be distributed under the terms of Sections 1 and 2 above on a 232 | medium customarily used for software interchange. 233 | 234 | If distribution of object code is made by offering access to copy 235 | from a designated place, then offering equivalent access to copy the 236 | source code from the same place satisfies the requirement to 237 | distribute the source code, even though third parties are not 238 | compelled to copy the source along with the object code. 239 | 240 | 5. A program that contains no derivative of any portion of the 241 | Library, but is designed to work with the Library by being compiled or 242 | linked with it, is called a "work that uses the Library". Such a 243 | work, in isolation, is not a derivative work of the Library, and 244 | therefore falls outside the scope of this License. 245 | 246 | However, linking a "work that uses the Library" with the Library 247 | creates an executable that is a derivative of the Library (because it 248 | contains portions of the Library), rather than a "work that uses the 249 | library". The executable is therefore covered by this License. 250 | Section 6 states terms for distribution of such executables. 251 | 252 | When a "work that uses the Library" uses material from a header file 253 | that is part of the Library, the object code for the work may be a 254 | derivative work of the Library even though the source code is not. 255 | Whether this is true is especially significant if the work can be 256 | linked without the Library, or if the work is itself a library. The 257 | threshold for this to be true is not precisely defined by law. 258 | 259 | If such an object file uses only numerical parameters, data 260 | structure layouts and accessors, and small macros and small inline 261 | functions (ten lines or less in length), then the use of the object 262 | file is unrestricted, regardless of whether it is legally a derivative 263 | work. (Executables containing this object code plus portions of the 264 | Library will still fall under Section 6.) 265 | 266 | Otherwise, if the work is a derivative of the Library, you may 267 | distribute the object code for the work under the terms of Section 6. 268 | Any executables containing that work also fall under Section 6, 269 | whether or not they are linked directly with the Library itself. 270 | 271 | 6. As an exception to the Sections above, you may also combine or 272 | link a "work that uses the Library" with the Library to produce a 273 | work containing portions of the Library, and distribute that work 274 | under terms of your choice, provided that the terms permit 275 | modification of the work for the customer's own use and reverse 276 | engineering for debugging such modifications. 277 | 278 | You must give prominent notice with each copy of the work that the 279 | Library is used in it and that the Library and its use are covered by 280 | this License. You must supply a copy of this License. If the work 281 | during execution displays copyright notices, you must include the 282 | copyright notice for the Library among them, as well as a reference 283 | directing the user to the copy of this License. Also, you must do one 284 | of these things: 285 | 286 | a) Accompany the work with the complete corresponding 287 | machine-readable source code for the Library including whatever 288 | changes were used in the work (which must be distributed under 289 | Sections 1 and 2 above); and, if the work is an executable linked 290 | with the Library, with the complete machine-readable "work that 291 | uses the Library", as object code and/or source code, so that the 292 | user can modify the Library and then relink to produce a modified 293 | executable containing the modified Library. (It is understood 294 | that the user who changes the contents of definitions files in the 295 | Library will not necessarily be able to recompile the application 296 | to use the modified definitions.) 297 | 298 | b) Use a suitable shared library mechanism for linking with the 299 | Library. A suitable mechanism is one that (1) uses at run time a 300 | copy of the library already present on the user's computer system, 301 | rather than copying library functions into the executable, and (2) 302 | will operate properly with a modified version of the library, if 303 | the user installs one, as long as the modified version is 304 | interface-compatible with the version that the work was made with. 305 | 306 | c) Accompany the work with a written offer, valid for at 307 | least three years, to give the same user the materials 308 | specified in Subsection 6a, above, for a charge no more 309 | than the cost of performing this distribution. 310 | 311 | d) If distribution of the work is made by offering access to copy 312 | from a designated place, offer equivalent access to copy the above 313 | specified materials from the same place. 314 | 315 | e) Verify that the user has already received a copy of these 316 | materials or that you have already sent this user a copy. 317 | 318 | For an executable, the required form of the "work that uses the 319 | Library" must include any data and utility programs needed for 320 | reproducing the executable from it. However, as a special exception, 321 | the materials to be distributed need not include anything that is 322 | normally distributed (in either source or binary form) with the major 323 | components (compiler, kernel, and so on) of the operating system on 324 | which the executable runs, unless that component itself accompanies 325 | the executable. 326 | 327 | It may happen that this requirement contradicts the license 328 | restrictions of other proprietary libraries that do not normally 329 | accompany the operating system. Such a contradiction means you cannot 330 | use both them and the Library together in an executable that you 331 | distribute. 332 | 333 | 7. You may place library facilities that are a work based on the 334 | Library side-by-side in a single library together with other library 335 | facilities not covered by this License, and distribute such a combined 336 | library, provided that the separate distribution of the work based on 337 | the Library and of the other library facilities is otherwise 338 | permitted, and provided that you do these two things: 339 | 340 | a) Accompany the combined library with a copy of the same work 341 | based on the Library, uncombined with any other library 342 | facilities. This must be distributed under the terms of the 343 | Sections above. 344 | 345 | b) Give prominent notice with the combined library of the fact 346 | that part of it is a work based on the Library, and explaining 347 | where to find the accompanying uncombined form of the same work. 348 | 349 | 8. You may not copy, modify, sublicense, link with, or distribute 350 | the Library except as expressly provided under this License. Any 351 | attempt otherwise to copy, modify, sublicense, link with, or 352 | distribute the Library is void, and will automatically terminate your 353 | rights under this License. However, parties who have received copies, 354 | or rights, from you under this License will not have their licenses 355 | terminated so long as such parties remain in full compliance. 356 | 357 | 9. You are not required to accept this License, since you have not 358 | signed it. However, nothing else grants you permission to modify or 359 | distribute the Library or its derivative works. These actions are 360 | prohibited by law if you do not accept this License. Therefore, by 361 | modifying or distributing the Library (or any work based on the 362 | Library), you indicate your acceptance of this License to do so, and 363 | all its terms and conditions for copying, distributing or modifying 364 | the Library or works based on it. 365 | 366 | 10. Each time you redistribute the Library (or any work based on the 367 | Library), the recipient automatically receives a license from the 368 | original licensor to copy, distribute, link with or modify the Library 369 | subject to these terms and conditions. You may not impose any further 370 | restrictions on the recipients' exercise of the rights granted herein. 371 | You are not responsible for enforcing compliance by third parties with 372 | this License. 373 | 374 | 11. If, as a consequence of a court judgment or allegation of patent 375 | infringement or for any other reason (not limited to patent issues), 376 | conditions are imposed on you (whether by court order, agreement or 377 | otherwise) that contradict the conditions of this License, they do not 378 | excuse you from the conditions of this License. If you cannot 379 | distribute so as to satisfy simultaneously your obligations under this 380 | License and any other pertinent obligations, then as a consequence you 381 | may not distribute the Library at all. For example, if a patent 382 | license would not permit royalty-free redistribution of the Library by 383 | all those who receive copies directly or indirectly through you, then 384 | the only way you could satisfy both it and this License would be to 385 | refrain entirely from distribution of the Library. 386 | 387 | If any portion of this section is held invalid or unenforceable under any 388 | particular circumstance, the balance of the section is intended to apply, 389 | and the section as a whole is intended to apply in other circumstances. 390 | 391 | It is not the purpose of this section to induce you to infringe any 392 | patents or other property right claims or to contest validity of any 393 | such claims; this section has the sole purpose of protecting the 394 | integrity of the free software distribution system which is 395 | implemented by public license practices. Many people have made 396 | generous contributions to the wide range of software distributed 397 | through that system in reliance on consistent application of that 398 | system; it is up to the author/donor to decide if he or she is willing 399 | to distribute software through any other system and a licensee cannot 400 | impose that choice. 401 | 402 | This section is intended to make thoroughly clear what is believed to 403 | be a consequence of the rest of this License. 404 | 405 | 12. If the distribution and/or use of the Library is restricted in 406 | certain countries either by patents or by copyrighted interfaces, the 407 | original copyright holder who places the Library under this License may add 408 | an explicit geographical distribution limitation excluding those countries, 409 | so that distribution is permitted only in or among countries not thus 410 | excluded. In such case, this License incorporates the limitation as if 411 | written in the body of this License. 412 | 413 | 13. The Free Software Foundation may publish revised and/or new 414 | versions of the Lesser General Public License from time to time. 415 | Such new versions will be similar in spirit to the present version, 416 | but may differ in detail to address new problems or concerns. 417 | 418 | Each version is given a distinguishing version number. If the Library 419 | specifies a version number of this License which applies to it and 420 | "any later version", you have the option of following the terms and 421 | conditions either of that version or of any later version published by 422 | the Free Software Foundation. If the Library does not specify a 423 | license version number, you may choose any version ever published by 424 | the Free Software Foundation. 425 | 426 | 14. If you wish to incorporate parts of the Library into other free 427 | programs whose distribution conditions are incompatible with these, 428 | write to the author to ask for permission. For software which is 429 | copyrighted by the Free Software Foundation, write to the Free 430 | Software Foundation; we sometimes make exceptions for this. Our 431 | decision will be guided by the two goals of preserving the free status 432 | of all derivatives of our free software and of promoting the sharing 433 | and reuse of software generally. 434 | 435 | NO WARRANTY 436 | 437 | 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO 438 | WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. 439 | EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR 440 | OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY 441 | KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE 442 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 443 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE 444 | LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME 445 | THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 446 | 447 | 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN 448 | WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY 449 | AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU 450 | FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR 451 | CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE 452 | LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING 453 | RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A 454 | FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF 455 | SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH 456 | DAMAGES. 457 | 458 | END OF TERMS AND CONDITIONS 459 | 460 | How to Apply These Terms to Your New Libraries 461 | 462 | If you develop a new library, and you want it to be of the greatest 463 | possible use to the public, we recommend making it free software that 464 | everyone can redistribute and change. You can do so by permitting 465 | redistribution under these terms (or, alternatively, under the terms of the 466 | ordinary General Public License). 467 | 468 | To apply these terms, attach the following notices to the library. It is 469 | safest to attach them to the start of each source file to most effectively 470 | convey the exclusion of warranty; and each file should have at least the 471 | "copyright" line and a pointer to where the full notice is found. 472 | 473 | 474 | Copyright (C) 475 | 476 | This library is free software; you can redistribute it and/or 477 | modify it under the terms of the GNU Lesser General Public 478 | License as published by the Free Software Foundation; either 479 | version 2.1 of the License, or (at your option) any later version. 480 | 481 | This library is distributed in the hope that it will be useful, 482 | but WITHOUT ANY WARRANTY; without even the implied warranty of 483 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 484 | Lesser General Public License for more details. 485 | 486 | You should have received a copy of the GNU Lesser General Public 487 | License along with this library; if not, write to the Free Software 488 | Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 489 | 490 | Also add information on how to contact you by electronic and paper mail. 491 | 492 | You should also get your employer (if you work as a programmer) or your 493 | school, if any, to sign a "copyright disclaimer" for the library, if 494 | necessary. Here is a sample; alter the names: 495 | 496 | Yoyodyne, Inc., hereby disclaims all copyright interest in the 497 | library `Frob' (a library for tweaking knobs) written by James Random Hacker. 498 | 499 | , 1 April 1990 500 | Ty Coon, President of Vice 501 | 502 | That's all there is to it! 503 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Pacemaker High Availability for PostgreSQL 3 | 4 | ## 简介 5 | 原生的基于Pacemaker+Corosync搭建PostgreSQL流复制HA集群配置和使用都比较复杂，因此封装了一些常用的集群配置和操作命令，目的在于简化集群的部署和使用。同时对Resource Agent 4.2.0的pgsql RA进行了增强，引入分布式锁服务，防止双节点集群出现脑裂，并确保同步复制下failover后数据不丢失。 但是，你还是必须了解Pacemaker的相关概念和基本操作，用于解决封装脚本处理不了的问题。 6 | 7 | **本工具已停止更新，建议优先使用其他PG HA工具，比如Patroni，可参考 [基于Patroni的PostgreSQL高可用环境部署](https://github.com/ChenHuajun/blog_xqhx/blob/main/2020/2020-09-07-%E5%9F%BA%E4%BA%8EPatroni%E7%9A%84PostgreSQL%E9%AB%98%E5%8F%AF%E7%94%A8%E7%8E%AF%E5%A2%83%E9%83%A8%E7%BD%B2.md)** 8 | 9 | ## 功能特性 10 | 1. 秒级故障转移 11 | 2. 支持双节点集群和多节点集群 12 | 3. 支持同步复制和异步复制 13 | 4. 同步复制下failover零数据丢失 14 | 5. 提供读写VIP和只读VIP，集群的拓扑结构对应用透明 15 | 16 | ## 基本架构和原理 17 | 1. Pacemaker + Corosync作为集群基础软件，Corosync负责集群通信和成员关系管理，Pacemaker负责资源管理。 18 | 2. 集群用到资源包括PostgreSQL和VIP等，PostgreSQL对应的Resource Agent(RA)为expgsql，expgsql负责实施PostgreSQL的起停，监视，failover等操作。 19 | 3. 集群初始启动时expgsql通过比较所有节点的xlog位置，找出xlog最新的节点作为Master，其它节点作为Slave通过读写VIP连接到Master上进行WAL复制。 20 | 4. 集群启动后expgsql不断监视PostgreSQL的健康状况，当expgsql发现PostgreSQL资源故障时报告给Pacemaker，由Pacemaker实施相应动作。 21 | - 如果是PostgreSQL进程故障，原地重启PostgreSQL，并且该节点上的fail-count加1。 22 | - fail-count累加到3时不再分配PostgreSQL资源到这个节点。如果该节点为Master，会提升一个Slave为Master，即发起failover。 23 | 5. Corosync发现节点故障(主机或网络故障)时，Pacemaker也根据情况实施相应动作。 24 | - 对多节点集群，未包含过半节点成员的分区将主动释放本分区内的所有资源，包括PostgreSQL和VIP。 25 | - 合法的分区中如果没有Master，Pacemaker会提升一个Slave为Master，即发起failover。 26 | 6. Master上的expgsql会不断监视Slave的复制健康状况，同步复制下会选定一个Slave作为同步Slave。 27 | 7. 当同步Slave出现故障时，Master上的expgsql会临时将同步复制切换到异步复制，防止Master上的写操作被hang住。如果故障Slave恢复或存在另一个健康的Slave，再切换到同步复制。 28 | 8. 为防止集群分区后，Slave升级为新Master而旧Master切换到异步复制导致脑裂和数据双写，引入分布式锁服务进行仲裁。Slave升级为新Master和旧Master切换到异步复制前必须先取得锁，避免这两件事同时发生。失去锁的Master会主动停止PostgreSQL进程，防止出现双主。 29 | 9. 如果分布锁服务发生故障而所有PostgreSQL节点都是健康的，expgsql会忽视锁服务，即不影响集群服务。但在分布锁服务故障期间，Master发生节点故障(注意区分节点故障和资源故障)，集群将无法正常failover。 30 | 10. 同步复制下只有同步Slave才有资格成为候选Master，加上有分布式锁的防护，可以确保failover后数据不丢失。 31 | 11. 集群初始启动和每次failover时通过`pg_ctl promote`提升Slave为Master并使时间线加1，同时记录Master节点名，时间线和切换时的xlog位置到集群CIB。 32 | 12. 集群重启时根据集群CIB中记录的信息确定Master节点，并保持时间线不变。 33 | 13. expgsql启动PostgreSQL前会检查该节点的时间线和xlog，如果和集群CIB中记录的信息有冲突，将报错。需要人工通过`cls_repair_by_pg_rewind`等手段修复。 34 | 14. 读写VIP和Master节点绑定，只读VIP和其中一个Slave绑定，应用只需访问VIP，无需关心具体访问哪个节点。 35 | 36 | 37 | ## 集群操作命令一览 38 | 1. `cls_start` 39 | 启动集群 40 | 2. `cls_stop` 41 | 停止集群 42 | 3. `cls_online_switch` 43 | 在线主从切换,对多节点集群当前不支持指定新Master。在多节点的同步复制下，只有pgsql-data-status值为“STREAMING|SYNC”的节点，即同步复制节点可以作为候选master。如果希望指定其它节点作为新的master，可以在master上执行下面的操作，然后等待pgsql-data-status更新。 44 | 45 | su - postgres 46 | echo "synchronous_standby_names = 'node3'" > /var/lib/pgsql/tmp/rep_mode.conf 47 | pg_ctl -D /home/postgresql/data reload 48 | exit 49 | 50 | 4. `cls_master` 51 | 输出当前Master节点名 52 | 5. `cls_status` 53 | 显示集群状态 54 | 6. `cls_cleanup` 55 | 清除资源状态和fail-count。在某个节点上资源失败次数(fail-count)超过3次Pacemaker将不再分配该资源到此节点，人工修复故障后需要调用cleanup让Pacemkaer重新尝试启动资源。 56 | 7. `cls_reset_master` [master] 57 | 设置`pgsql_REPL_INFO`使指定的节点成为Master；如未指定Master，则清除`pgsql_REPL_INFO`让Pacemaker重新在所有节点中选出xlog位置最新的节点作为Master。仅用于集群中没有任何节点满足Master条件情况下的紧急修复。 58 | 8. `cls_repair_by_pg_rewind` 59 | 通过`pg_rewind`修复当前节点，主要用于旧Master的修复，回退超出时间线分叉点的那部分更新，并和新Master建立复制关系。`pg_rewind`仅在PostgreSQL 9.5以上版本提供 60 | 9. `cls_rebuild_slave` 61 | 通过`pg_basebackup`在当前节点重建Slave。执行该命令前需要停止当前节点上的PostgreSQL进程并清空旧的数据目录。 62 | 10. `cls_maintenance` 63 | 切换集群到维护模式使所有资源脱离Pacemaker的控制。当需要重启Pacemaker和Corosync又不能停止PostgreSQL服务时，可以先调用这个命令，Pacemaker和Corosync重启完成后再调用用cls_unmaintenance恢复为普通模式。 64 | 11. `cls_unmaintenance` 65 | 从维护模式恢复到普通模式。 66 | 12. `cls_maintenance_node` 67 | 使节点进入维护模式。维护模式和unmanage resource相比的区别是会取消monitor，比unmanage更彻底。 68 | 13. `cls_unmaintenance_node` 69 | 解除节点的维护模式。 70 | 14. `cls_standby_node` 71 | 释放某节点上所有资源。可用于特定节点的维护，比如升级。 72 | 15. `cls_unstandby_node` 73 | 恢复`cls_standby_node`产生的节点standby状态。 74 | 75 | 以上命令必须以root用户执行 76 | 77 | ## 依赖软件 78 | - pacemaker 79 | - corosync 80 | - pcs 81 | - ipvsadm 82 | 83 | ## 支持的PostgreSQL版本 84 | - PostgreSQL 9.6 85 | - PostgreSQL 10 86 | - PostgreSQL 11 87 | 88 | ## 安装 89 | 90 | 安装过程以在以下环境下部署双节点HA集群为例说明。 91 | 多节点的安装示例请参考:[基于Pacemaker的PostgreSQL一主多从读负载均衡集群搭建](https://github.com/ChenHuajun/pha4pgsql/blob/master/doc/muti_with_lvs.md) 92 | 93 | - OS:CentOS 7.1 94 | - 节点1主机名:node1 95 | - 节点2主机名:node2 96 | - writer_vip:192.168.41.136 97 | - reader_vip:192.168.41.137 98 | - 用作分布式锁服务的PostgreSQL的连接字符串:"host=node3 port=5439 dbname=postgres user=postgres" 99 | 100 | ### Linux集群环境安装与配置 101 | #### 环境准备 102 | 1. 所有节点设置时钟同步 103 | 104 | cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 105 | ntpdate time.windows.com && hwclock -w 106 | 107 | 2. 所有节点设置独立的主机名(node1，node2) 108 | 109 | hostnamectl set-hostname node1 110 | 111 | 3. 设置对所有节点的域名解析(修改/etc/hosts) 112 | 4. 所有节点间设置SSH互信 113 | 114 | #### 禁用防火墙 115 | 在所有节点执行： 116 | 117 | setenforce 0 118 | sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config 119 | systemctl disable firewalld.service 120 | systemctl stop firewalld.service 121 | iptables -F 122 | 123 | 如果开启防火墙需要开放postgres，pcsd和corosync的端口。 124 | 125 | * postgres:5432/tcp 126 | * pcsd:2224/tcp 127 | * corosync:5405/udp 128 | 129 | #### 安装Pacemaker和Corosync及相关软件包 130 | 在所有节点执行： 131 | 132 | yum install -y pacemaker corosync pcs ipvsadm 133 | 134 | 注：如果OS自带的Pacemaker比较旧，建议下载新版的。之前在Pacemaker 1.1.7上遇到了不少Bug，因此不建议使用这个版本或更老的版本。配置LVS(使用`muti_with_lvs.pcs.template`模板)支持需要安装ipvsadm 135 | 136 | #### 启用pcsd服务 137 | 在所有节点执行： 138 | 139 | systemctl start pcsd.service 140 | systemctl enable pcsd.service 141 | 142 | #### 设置hacluster用户密码 143 | 在所有节点执行： 144 | 145 | echo hacluster | passwd hacluster --stdin 146 | 147 | #### 集群认证 148 | 在任何一个节点上执行: 149 | 150 | pcs cluster auth -u hacluster -p hacluster node1 node2 151 | 152 | #### 同步配置 153 | 在任何一个节点上执行: 154 | 155 | pcs cluster setup --last_man_standing=1 --name pgcluster node1 node2 156 | 157 | #### 启动集群 158 | 在任何一个节点上执行: 159 | 160 | pcs cluster start --all 161 | 162 | ### 安装和配置PostgreSQL 163 | 164 | #### 安装PostgreSQL 165 | 在所有节点执行： 166 | 167 | yum install postgresql-server 168 | 169 | OS自带的PostgreSQL往往比较旧，可参考http://www.postgresql.org/download/linux/ ，安装最新版PostgreSQL. 170 | 171 | #### 创建主数据库 172 | 在node1节点执行： 173 | 174 | 1. 创建数据目录 175 | 176 | mkdir -p /data/postgresql/data 177 | chown -R postgres:postgres /data/postgresql/ 178 | chmod 0700 /data/postgresql/data 179 | 180 | 2. 初始化db 181 | 182 | su - postgres 183 | initdb -D /data/postgresql/data/ 184 | 185 | 3. 修改postgresql.conf 186 | 187 | listen_addresses = '*' 188 | wal_level = hot_standby 189 | wal_log_hints = on 190 | synchronous_commit = on 191 | max_wal_senders=5 192 | wal_keep_segments = 32 193 | hot_standby = on 194 | wal_sender_timeout = 5000 195 | wal_receiver_status_interval = 2 196 | max_standby_streaming_delay = -1 197 | max_standby_archive_delay = -1 198 | restart_after_crash = off 199 | hot_standby_feedback = on 200 | 201 | 注：PostgreSQL9.5以上版本设置"`wal_log_hints = on`"可以使用`pg_rewind`修复旧Master。 202 | 203 | 204 | 4. 修改`pg_hba.conf` 205 | 206 | local all all trust 207 | host all all 192.168.41.0/24 md5 208 | host replication all 192.168.41.0/24 md5 209 | 210 | 5. 启动postgres 211 | 212 | pg_ctl -D /data/postgresql/data/ start 213 | 214 | 6. 创建复制用户 215 | 216 | createuser --login --replication replication -P 217 | 218 | 9.5以上版本如需要支持pg_rewind，需加上“-s”选项。 219 | 220 | createuser --login --replication replication -P -s 221 | 222 | #### 创建备数据库 223 | 在node2节点执行： 224 | 225 | 1. 创建数据目录 226 | 227 | mkdir -p /data/postgresql/data 228 | chown -R postgres:postgres /data/postgresql/ 229 | chmod 0700 /data/postgresql/data 230 | 231 | 2. 创建基础备份 232 | 233 | su - postgres 234 | pg_basebackup -h node1 -U replication -D /data/postgresql/data/ -X stream -P 235 | 236 | 237 | #### 停止PostgreSQL服务 238 | 在node1上执行: 239 | 240 | pg_ctl -D /data/postgresql/data/ stop 241 | 242 | ### 配置分布式锁服务 243 | 分布式锁服务的作用是防止双节点集群出现脑裂。当网络发生故障形成分区时，备可能会被提升为主，同时旧主会将同步复制切换到异步复制，这可能导致数据丢失。通过分布式锁服务可以确保新主的提升和旧主的切换到异步复制同时只能有一个成功。 244 | 245 | 分布式锁服务通过HA集群外部的另外一个PostgreSQL服务实现。需要事先创建锁表。 246 | 247 | create table if not exists distlock(lockname text primary key,owner text not null,ts timestamptz not null,expired_time interval not null); 248 | 249 | 可选地，可以创建锁的历史表，每次锁的owner变更(主从角色切换)都会记录到历史表(distlock_history)中。 250 | 251 | create table if not exists distlock_history(id serial primary key,lockname text not null,owner text not null,ts timestamptz not null,expired_time interval not null); 252 | 253 | CREATE OR REPLACE FUNCTION distlock_log_update() RETURNS trigger AS $$ 254 | BEGIN 255 | 256 | IF TG_OP = 'INSERT' or NEW.owner <> OLD.owner THEN 257 | INSERT INTO distlock_history(lockname, owner, ts, expired_time) values(NEW.lockname, NEW.owner, NEW.ts, NEW.expired_time); 258 | END IF; 259 | RETURN NEW; 260 | END; 261 | $$ LANGUAGE plpgsql; 262 | 263 | 264 | DROP TRIGGER IF EXISTS distlock_log_update ON distlock; 265 | 266 | CREATE TRIGGER distlock_log_update AFTER INSERT OR UPDATE ON distlock 267 | FOR EACH ROW EXECUTE PROCEDURE distlock_log_update(); 268 | 269 | 270 | ### 安装和配置pha4pgsql 271 | 在任意一个节点上执行: 272 | 273 | 1. 下载pha4pgsql 274 | 275 | git clone git://github.com/Chenhuajun/pha4pgsql.git 276 | 277 | 2. 编辑config.ini 278 | 279 | pcs_template=dual.pcs.template 280 | OCF_ROOT=/usr/lib/ocf 281 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 282 | pha4pgsql_dir=/opt/pha4pgsql 283 | writer_vip=192.168.41.136 284 | reader_vip=192.168.41.137 285 | node1=node1 286 | node2=node2 287 | vip_nic=eno33554984 288 | vip_cidr_netmask=24 289 | pgsql_pgctl=/usr/bin/pg_ctl 290 | pgsql_psql=/usr/bin/psql 291 | pgsql_pgdata=/data/postgresql/data 292 | pgsql_pgdata=5432 293 | pgsql_restore_command="" 294 | pgsql_rep_mode=sync 295 | pgsql_repuser=replication 296 | pgsql_reppassord=replication 297 | pgsql_enable_distlock=true 298 | pgsql_distlock_psql_cmd='/usr/bin/psql \\"host=node3 port=5439 dbname=postgres user=postgres connect_timeout=5\\"' 299 | pgsql_distlock_lockname=pgsql_cls1 300 | 301 | 需要根据实际环境修改上面的参数。当多个集群使用锁服务时，确保每个集群的`pgsql_distlock_lockname`值必须是唯一的。 302 | 303 | template目录下有预定义的参数模板。 304 | - `config_dual.ini.sample` 305 | 双节点参数模板，通过同步复制和分布式锁仲裁防止脑裂 306 | - `config_muti.ini.sample` 307 | 多节点参数模板，通过no-quorum-policy="stop"防止脑裂 308 | - `config_muti_with_lvs.ini.sample` 309 | 多节点参数模板，并引入LVS做读负载均衡 310 | 311 | 3. 安装pha4pgsql 312 | 313 | sh install.sh 314 | 315 | 这一步会拷贝需要的脚本到本地和远程机器上，并生成集群的资源配置文件。 316 | 317 | [root@node1 pha4pgsql]# cat config.pcs 318 | 319 | pcs cluster cib pgsql_cfg 320 | 321 | pcs -f pgsql_cfg property set no-quorum-policy="ignore" 322 | pcs -f pgsql_cfg property set stonith-enabled="false" 323 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 324 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 325 | 326 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 327 | ip="192.168.41.136" \ 328 | nic="eno33554984" \ 329 | cidr_netmask="24" \ 330 | op start timeout="60s" interval="0s" on-fail="restart" \ 331 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 332 | op stop timeout="60s" interval="0s" on-fail="block" 333 | 334 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 335 | ip="192.168.41.137" \ 336 | nic="eno33554984" \ 337 | cidr_netmask="24" \ 338 | op start timeout="60s" interval="0s" on-fail="restart" \ 339 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 340 | op stop timeout="60s" interval="0s" on-fail="block" 341 | 342 | pcs -f pgsql_cfg resource create pgsql expgsql \ 343 | pgctl="/usr/bin/pg_ctl" \ 344 | psql="/usr/bin/psql" \ 345 | pgdata="5432" \ 346 | pgport="" \ 347 | rep_mode="sync" \ 348 | node_list="node1 node2" \ 349 | restore_command="" \ 350 | primary_conninfo_opt="user=replication password=replication keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 351 | master_ip="192.168.41.136" \ 352 | restart_on_promote="false" \ 353 | enable_distlock="true" \ 354 | distlock_lock_cmd="/opt/pha4pgsql/tools/distlock '/usr/bin/psql \"host=node3 port=5439 dbname=postgres user=postgres connect_timeout=5\"' lock distlock:pgsql_cls1 @owner 9 12" \ 355 | distlock_unlock_cmd="/opt/pha4pgsql/tools/distlock '/usr/bin/psql \"host=node3 port=5439 dbname=postgres user=postgres connect_timeout=5\"' unlock distlock:pgsql_cls1 @owner" \ 356 | distlock_lockservice_deadcheck_nodelist="node1 node2" \ 357 | op start timeout="60s" interval="0s" on-fail="restart" \ 358 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 359 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 360 | op promote timeout="60s" interval="0s" on-fail="restart" \ 361 | op demote timeout="60s" interval="0s" on-fail="stop" \ 362 | op stop timeout="60s" interval="0s" on-fail="block" \ 363 | op notify timeout="60s" interval="0s" 364 | 365 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 366 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 367 | migration-threshold="3" target-role="Master" 368 | 369 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 370 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 371 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 372 | 373 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 374 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 375 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 376 | 377 | pcs cluster cib-push pgsql_cfg 378 | 379 | 380 | 可以根据需要对config.pcs做相应修改，在执行下面的配置脚本 381 | 382 | ./setup.sh 383 | 384 | 注意，安装和配置过程只需在一个节点上执行即可。 385 | 386 | 4. 设置环境变量 387 | 388 | export PATH=/opt/pha4pgsql/bin:$PATH 389 | 390 | 4. 启动集群 391 | 392 | cls_start 393 | 394 | 5. 确认集群状态 395 | 396 | cls_status 397 | 398 | `cls_status`的输出示例如下： 399 | 400 | [root@node1 pha4pgsql]# cls_status 401 | Last updated: Fri Apr 22 02:01:01 2016 402 | Last change: Fri Apr 22 02:01:00 2016 by root via crm_resource on node1 403 | Stack: corosync 404 | Current DC: node2 (2) - partition with quorum 405 | Version: 1.1.12-a14efad 406 | 2 Nodes configured 407 | 4 Resources configured 408 | 409 | 410 | Online: [ node1 node2 ] 411 | 412 | Full list of resources: 413 | 414 | vip-master (ocf::heartbeat:IPaddr2): Started node1 415 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 416 | Master/Slave Set: msPostgresql [pgsql] 417 | Masters: [ node1 ] 418 | Slaves: [ node2 ] 419 | 420 | Node Attributes: 421 | * Node node1: 422 | + #cluster-name : pgcluster 423 | + #site-name : pgcluster 424 | + master-pgsql : 1000 425 | + pgsql-data-status : LATEST 426 | + pgsql-master-baseline : 00000000070000D0 427 | + pgsql-status : PRI 428 | * Node node2: 429 | + #cluster-name : pgcluster 430 | + #site-name : pgcluster 431 | + master-pgsql : -INFINITY 432 | + pgsql-data-status : STREAMING|ASYNC 433 | + pgsql-status : HS:async 434 | 435 | Migration summary: 436 | * Node node2: 437 | * Node node1: 438 | 439 | pgsql_REPL_INFO:node1|1|00000000070000D0 440 | 441 | 检查集群的健康状态。完全健康的集群需要满足以下条件： 442 | 443 | 1. msPostgresql在每个节点上都已启动 444 | 2. 在其中一个节点上msPostgresql处于Master状态，其它的为Salve状态 445 | 3. Salve节点的data-status值是以下中的一个 446 | - STREAMING|SYNC 447 | 同步复制Slave 448 | - STREAMING|POTENTIAL 449 | 候选同步复制Slave 450 | - STREAMING|ASYNC 451 | 异步复制Slave 452 | 453 | 454 | pgsql-data-status的取值详细可参考下面的说明 455 | 456 | The transitional state of data is displayed. This state remains after stopping pacemaker. When starting pacemaker next time, this state is used to judge whether my data is old or not. 457 | DISCONNECT 458 | Master changes other node state into DISCONNECT if Master can't detect connection of replication because of LAN failure or breakdown of Slave and so on. 459 | {state}|{sync_state} 460 | Master changes other node state into {state}|{sync_state} if Master detects connection of replication. 461 | {state} and {sync_state} means state of replication which is retrieved using "select state and sync_state from pg_stat_replication" on Master. 462 | For example, INIT, CATCHUP, and STREAMING are displayed in {state} and ASYNC, SYNC are displayed in {sync_state} 463 | LATEST 464 | It's displayed when it's Master. 465 | These states are the transitional state of final data, and it may be not consistent with the state of actual data. For instance, During PRI, the state is "LATEST". But the node is stopped or down, this state "LATEST" is maintained if Master doesn't exist in other nodes. It never changes to "DISCONNECT" for oneself. When other node newly is promoted, this new Master changes the state of old Master to "DISCONNECT". When any node can not become Master, this "LATEST" will be keeped. 466 | 467 | `pgsql_REPL_INFO`的3段内容分别指当前master，上次提升前的时间线和xlog位置。 468 | 469 | pgsql_REPL_INFO:node1|1|00000000070000D0 470 | 471 | ## 故障测试 472 | 473 | ### Master上的postgres进程故障 474 | 475 | 1. 强制杀死Master上的postgres进程 476 | 477 | [root@node1 pha4pgsql]# killall postgres 478 | 479 | 2. 检查集群状态 480 | 由于设置了migration-threshold="3"，发生一次普通的错误，Pacemaker会在原地重新启动postgres进程，不发生主从切换。 481 | （如果Master的物理机或网络发生故障，直接进行failover。） 482 | 483 | [root@node1 pha4pgsql]# cls_status 484 | Last updated: Fri Apr 22 02:03:17 2016 485 | Last change: Fri Apr 22 02:03:10 2016 by root via crm_resource on node1 486 | Stack: corosync 487 | Current DC: node2 (2) - partition with quorum 488 | Version: 1.1.12-a14efad 489 | 2 Nodes configured 490 | 4 Resources configured 491 | 492 | 493 | Online: [ node1 node2 ] 494 | 495 | Full list of resources: 496 | 497 | vip-master (ocf::heartbeat:IPaddr2): Started node1 498 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 499 | Master/Slave Set: msPostgresql [pgsql] 500 | Masters: [ node1 ] 501 | Slaves: [ node2 ] 502 | 503 | Node Attributes: 504 | * Node node1: 505 | + #cluster-name : pgcluster 506 | + #site-name : pgcluster 507 | + master-pgsql : 1000 508 | + pgsql-data-status : LATEST 509 | + pgsql-master-baseline : 0000000007000250 510 | + pgsql-status : PRI 511 | * Node node2: 512 | + #cluster-name : pgcluster 513 | + #site-name : pgcluster 514 | + master-pgsql : 100 515 | + pgsql-data-status : STREAMING|SYNC 516 | + pgsql-status : HS:sync 517 | 518 | Migration summary: 519 | * Node node2: 520 | * Node node1: 521 | pgsql: migration-threshold=3 fail-count=1 last-failure='Mon Apr 18 09:14:28 2016' 522 | 523 | Failed actions: 524 | pgsql_monitor_3000 on node1 'unknown error' (1): call=205, status=complete, exit-reason='none', last-rc-change='Fri Apr 22 02:02:56 2016', queued=0ms, exec=0ms 525 | 526 | 527 | pgsql_REPL_INFO:node1|1|00000000070000D0 528 | 529 | 530 | 3. 再强制杀死Master上的postgres进程2次后检查集群状态。 531 | 这时已经发生了failover，产生了新的Master，并提升了时间线。 532 | 533 | [root@node1 pha4pgsql]# cls_status 534 | Last updated: Fri Apr 22 02:07:33 2016 535 | Last change: Fri Apr 22 02:07:31 2016 by root via crm_resource on node1 536 | Stack: corosync 537 | Current DC: node2 (2) - partition with quorum 538 | Version: 1.1.12-a14efad 539 | 2 Nodes configured 540 | 4 Resources configured 541 | 542 | 543 | Online: [ node1 node2 ] 544 | 545 | Full list of resources: 546 | 547 | vip-master (ocf::heartbeat:IPaddr2): Started node2 548 | vip-slave (ocf::heartbeat:IPaddr2): Stopped 549 | Master/Slave Set: msPostgresql [pgsql] 550 | Masters: [ node2 ] 551 | Stopped: [ node1 ] 552 | 553 | Node Attributes: 554 | * Node node1: 555 | + #cluster-name : pgcluster 556 | + #site-name : pgcluster 557 | + master-pgsql : -INFINITY 558 | + pgsql-data-status : DISCONNECT 559 | + pgsql-status : STOP 560 | * Node node2: 561 | + #cluster-name : pgcluster 562 | + #site-name : pgcluster 563 | + master-pgsql : 1000 564 | + pgsql-data-status : LATEST 565 | + pgsql-master-baseline : 0000000007000410 566 | + pgsql-status : PRI 567 | 568 | Migration summary: 569 | * Node node2: 570 | * Node node1: 571 | pgsql: migration-threshold=3 fail-count=3 last-failure='Mon Apr 18 09:18:58 2016' 572 | 573 | Failed actions: 574 | pgsql_monitor_3000 on node1 'not running' (7): call=237, status=complete, exit-reason='none', last-rc-change='Fri Apr 22 02:07:26 2016', queued=0ms, exec=0ms 575 | 576 | 577 | pgsql_REPL_INFO:node2|2|0000000007000410 578 | 579 | 580 | 4. 修复旧Master 581 | 582 | 可通过`pg_basebackup`修复旧Master 583 | 584 | # su - postgres 585 | $ rm -rf /data/postgresql/data 586 | $ pg_basebackup -h 192.168.41.136 -U postgres -D /data/postgresql/data -X stream -P 587 | $ exit 588 | # pcs resource cleanup msPostgresql 589 | 590 | 如果恢复失败，请检查PostgreSQL和Pacemaker日志文件。 591 | 592 | 通过`pg_baseback`修复旧Master。`cls_rebuild_slave`是对`pg_basebackup`的包装，主要多了执行结果状态的检查。 593 | 594 | [root@node1 pha4pgsql]# rm -rf /data/postgresql/data 595 | [root@node1 pha4pgsql]# cls_rebuild_slave 596 | 22636/22636 kB (100%), 1/1 tablespace 597 | All resources/stonith devices successfully cleaned up 598 | wait for recovery complete 599 | ..... 600 | slave recovery of node1 successed 601 | [root@node1 pha4pgsql]# cls_status 602 | Last updated: Fri Apr 22 02:40:48 2016 603 | Last change: Fri Apr 22 02:40:36 2016 by root via crm_resource on node1 604 | Stack: corosync 605 | Current DC: node2 (2) - partition with quorum 606 | Version: 1.1.12-a14efad 607 | 2 Nodes configured 608 | 4 Resources configured 609 | 610 | 611 | Online: [ node1 node2 ] 612 | 613 | Full list of resources: 614 | 615 | vip-master (ocf::heartbeat:IPaddr2): Started node2 616 | vip-slave (ocf::heartbeat:IPaddr2): Started node1 617 | Master/Slave Set: msPostgresql [pgsql] 618 | Masters: [ node2 ] 619 | Slaves: [ node1 ] 620 | 621 | Node Attributes: 622 | * Node node1: 623 | + #cluster-name : pgcluster 624 | + #site-name : pgcluster 625 | + master-pgsql : 100 626 | + pgsql-data-status : STREAMING|SYNC 627 | + pgsql-status : HS:sync 628 | * Node node2: 629 | + #cluster-name : pgcluster 630 | + #site-name : pgcluster 631 | + master-pgsql : 1000 632 | + pgsql-data-status : LATEST 633 | + pgsql-master-baseline : 0000000007000410 634 | + pgsql-status : PRI 635 | 636 | Migration summary: 637 | * Node node2: 638 | * Node node1: 639 | 640 | pgsql_REPL_INFO:node2|2|0000000007000410 641 | 642 | 643 | 9.5以上版本还可以通过`pg_rewind`修复旧Master 644 | 645 | [root@node1 pha4pgsql]# cls_repair_by_pg_rewind 646 | connected to server 647 | servers diverged at WAL position 0/7000410 on timeline 2 648 | rewinding from last common checkpoint at 0/7000368 on timeline 2 649 | reading source file list 650 | reading target file list 651 | reading WAL in target 652 | need to copy 67 MB (total source directory size is 85 MB) 653 | 69591/69591 kB (100%) copied 654 | creating backup label and updating control file 655 | syncing target data directory 656 | Done! 657 | All resources/stonith devices successfully cleaned up 658 | wait for recovery complete 659 | .... 660 | slave recovery of node1 successed 661 | 662 | 663 | 664 | ### Master网络故障 665 | 666 | 1. 故障前的集群状态 667 | 668 | 故障前的Master是node1 669 | 670 | [root@node1 pha4pgsql]# cls_status 671 | Last updated: Fri Apr 22 11:28:26 2016 672 | Last change: Fri Apr 22 11:25:56 2016 by root via crm_resource on node1 673 | Stack: corosync 674 | Current DC: node2 (2) - partition with quorum 675 | Version: 1.1.12-a14efad 676 | 2 Nodes configured 677 | 4 Resources configured 678 | 679 | 680 | Online: [ node1 node2 ] 681 | 682 | Full list of resources: 683 | 684 | vip-master (ocf::heartbeat:IPaddr2): Started node1 685 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 686 | Master/Slave Set: msPostgresql [pgsql] 687 | Masters: [ node1 ] 688 | Slaves: [ node2 ] 689 | 690 | Node Attributes: 691 | * Node node1: 692 | + #cluster-name : pgcluster 693 | + #site-name : pgcluster 694 | + master-pgsql : 1000 695 | + pgsql-data-status : LATEST 696 | + pgsql-master-baseline : 0000000009044898 697 | + pgsql-status : PRI 698 | * Node node2: 699 | + #cluster-name : pgcluster 700 | + #site-name : pgcluster 701 | + master-pgsql : 100 702 | + pgsql-data-status : STREAMING|SYNC 703 | + pgsql-status : HS:sync 704 | 705 | Migration summary: 706 | * Node node2: 707 | * Node node1: 708 | 709 | pgsql_REPL_INFO:node1|12|0000000009044898 710 | 711 | 2. 阻断Master和其它节点的通信 712 | 713 | [root@node1 pha4pgsql]# iptables -A INPUT -j DROP -s node2 714 | [root@node1 pha4pgsql]# iptables -A OUTPUT -j DROP -d node2 715 | [root@node1 pha4pgsql]# iptables -A INPUT -j DROP -s node3 716 | [root@node1 pha4pgsql]# iptables -A OUTPUT -j DROP -d node3 717 | 718 | 3. 等10几秒后检查集群状态 719 | 720 | 在node1(旧Master)上查看，由于失去分布式锁，node1已经停止了部署在自身上面的所有资源。 721 | 722 | [root@node1 pha4pgsql]# cls_status 723 | Last updated: Fri Apr 22 11:34:46 2016 724 | Last change: Fri Apr 22 11:25:56 2016 by root via crm_resource on node1 725 | Stack: corosync 726 | Current DC: node1 (1) - partition with quorum 727 | Version: 1.1.12-a14efad 728 | 2 Nodes configured 729 | 4 Resources configured 730 | 731 | 732 | Online: [ node1 ] 733 | OFFLINE: [ node2 ] 734 | 735 | Full list of resources: 736 | 737 | vip-master (ocf::heartbeat:IPaddr2): Stopped 738 | vip-slave (ocf::heartbeat:IPaddr2): Stopped 739 | Master/Slave Set: msPostgresql [pgsql] 740 | Stopped: [ node1 node2 ] 741 | 742 | Node Attributes: 743 | * Node node1: 744 | + #cluster-name : pgcluster 745 | + #site-name : pgcluster 746 | + master-pgsql : -INFINITY 747 | + pgsql-data-status : LATEST 748 | + pgsql-status : STOP 749 | 750 | Migration summary: 751 | * Node node1: 752 | pgsql: migration-threshold=3 fail-count=2 last-failure='Fri Apr 22 11:34:23 2016' 753 | 754 | Failed actions: 755 | pgsql_promote_0 on node1 'unknown error' (1): call=990, status=complete, exit-reason='none', last-rc-change='Fri Apr 22 11:34:15 2016', queued=0ms, exec=7756ms 756 | 757 | 758 | pgsql_REPL_INFO:node1|12|0000000009044898 759 | 760 | 761 | 在node2上查看，发现node2已经被提升为新Master，PostgreSQL的时间线也从12增长到了13。 762 | 763 | [root@node2 ~]# cls_status 764 | Last updated: Sun May 8 01:02:04 2016 765 | Last change: Sun May 8 00:57:47 2016 by root via crm_resource on node1 766 | Stack: corosync 767 | Current DC: node2 (2) - partition with quorum 768 | Version: 1.1.12-a14efad 769 | 2 Nodes configured 770 | 4 Resources configured 771 | 772 | 773 | Online: [ node2 ] 774 | OFFLINE: [ node1 ] 775 | 776 | Full list of resources: 777 | 778 | vip-master (ocf::heartbeat:IPaddr2): Started node2 779 | vip-slave (ocf::heartbeat:IPaddr2): Stopped 780 | Master/Slave Set: msPostgresql [pgsql] 781 | Masters: [ node2 ] 782 | Stopped: [ node1 ] 783 | 784 | Node Attributes: 785 | * Node node2: 786 | + #cluster-name : pgcluster 787 | + #site-name : pgcluster 788 | + master-pgsql : 1000 789 | + pgsql-data-status : LATEST 790 | + pgsql-master-baseline : 0000000009045828 791 | + pgsql-status : PRI 792 | 793 | Migration summary: 794 | * Node node2: 795 | 796 | pgsql_REPL_INFO:node2|13|0000000009045828 797 | 798 | 请注意，这时发生了网络分区，node1和node2各自保存的集群状态是不同的。 799 | 800 | 4. 恢复node1上的网络 801 | 802 | [root@node1 pha4pgsql]# iptables -F 803 | 804 | 5. 再次在node1上检查集群状态 805 | 再次在node1上检查集群状态，发现node1和node2两个分区合并后，集群采纳了node2的配置而不是node1，这正是我们想要的（由于node2上的集群配置的版本更高，所以采纳node2而不是node1的配置)。同时，Pacemaker试图重新启动node1上的PostgreSQL进程时，发现它的最近一次checkpoint位置大于等于上次时间线提升的位置，不能作为Slave连到新Master上所以报错并阻止它上线。 806 | 807 | [root@node1 pha4pgsql]# cls_status 808 | Last updated: Fri Apr 22 11:49:44 2016 809 | Last change: Sun May 8 00:57:47 2016 by root via crm_resource on node1 810 | Stack: corosync 811 | Current DC: node2 (2) - partition with quorum 812 | Version: 1.1.12-a14efad 813 | 2 Nodes configured 814 | 4 Resources configured 815 | 816 | 817 | Online: [ node1 node2 ] 818 | 819 | Full list of resources: 820 | 821 | vip-master (ocf::heartbeat:IPaddr2): Started node2 822 | vip-slave (ocf::heartbeat:IPaddr2): Started node1 823 | Master/Slave Set: msPostgresql [pgsql] 824 | Masters: [ node2 ] 825 | Stopped: [ node1 ] 826 | 827 | Node Attributes: 828 | * Node node1: 829 | + #cluster-name : pgcluster 830 | + #site-name : pgcluster 831 | + master-pgsql : -INFINITY 832 | + pgsql-data-status : DISCONNECT 833 | + pgsql-status : STOP 834 | * Node node2: 835 | + #cluster-name : pgcluster 836 | + #site-name : pgcluster 837 | + master-pgsql : 1000 838 | + pgsql-data-status : LATEST 839 | + pgsql-master-baseline : 0000000009045828 840 | + pgsql-status : PRI 841 | 842 | Migration summary: 843 | * Node node2: 844 | * Node node1: 845 | pgsql: migration-threshold=3 fail-count=1000000 last-failure='Sun May 8 01:12:57 2016' 846 | 847 | Failed actions: 848 | pgsql_start_0 on node1 'unknown error' (1): call=1022, status=complete, exit-reason='The master's timeline forked off current database system timeline 13 before latest checkpoint location 0000000009045828, REPL_IN', last-rc-change='Fri Apr 22 11:49:35 2016', queued=0ms, exec=2123ms 849 | 850 | 851 | pgsql_REPL_INFO:node2|13|0000000009045828 852 | 853 | 854 | 6. 修复node1(旧Master) 855 | 856 | 修复node1(旧Master)的方法和前面一样，可使用`cls_repair_slave`、`cls_repair_by_pg_rewind`，或者直接使用`pg_basebackup`、`pg_rewind`。 857 | 858 | [root@node1 pha4pgsql]# cls_repair_slave 859 | connected to server 860 | servers diverged at WAL position 0/9045828 on timeline 13 861 | rewinding from last common checkpoint at 0/9045780 on timeline 13 862 | reading source file list 863 | reading target file list 864 | reading WAL in target 865 | need to copy 211 MB (total source directory size is 229 MB) 866 | 216927/216927 kB (100%) copied 867 | creating backup label and updating control file 868 | syncing target data directory 869 | Done! 870 | All resources/stonith devices successfully cleaned up 871 | wait for recovery complete 872 | .......... 873 | slave recovery of node1 successed 874 | 875 | ### Slave上的PostgreSQL进程故障 876 | 1. 强制杀死Slave上的postgres进程 877 | 878 | [root@node2 pha4pgsql]# killall postgres 879 | 880 | 2. 检查集群状态 881 | 由于设置了migration-threshold="3"，发生一次普通的错误，Pacemaker会在原地重新启动postgres进程。 882 | 883 | [root@node2 ~]# cls_status 884 | Last updated: Sun May 8 01:34:36 2016 885 | Last change: Sun May 8 01:33:01 2016 by root via crm_resource on node1 886 | Stack: corosync 887 | Current DC: node2 (2) - partition with quorum 888 | Version: 1.1.12-a14efad 889 | 2 Nodes configured 890 | 4 Resources configured 891 | 892 | 893 | Online: [ node1 node2 ] 894 | 895 | Full list of resources: 896 | 897 | vip-master (ocf::heartbeat:IPaddr2): Started node1 898 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 899 | Master/Slave Set: msPostgresql [pgsql] 900 | Masters: [ node1 ] 901 | Slaves: [ node2 ] 902 | 903 | Node Attributes: 904 | * Node node1: 905 | + #cluster-name : pgcluster 906 | + #site-name : pgcluster 907 | + master-pgsql : 1000 908 | + pgsql-data-status : LATEST 909 | + pgsql-master-baseline : 00000000090650F8 910 | + pgsql-status : PRI 911 | * Node node2: 912 | + #cluster-name : pgcluster 913 | + #site-name : pgcluster 914 | + master-pgsql : 100 915 | + pgsql-data-status : STREAMING|SYNC 916 | + pgsql-status : HS:sync 917 | 918 | Migration summary: 919 | * Node node2: 920 | pgsql: migration-threshold=3 fail-count=1 last-failure='Sun May 8 01:32:44 2016' 921 | * Node node1: 922 | 923 | Failed actions: 924 | pgsql_monitor_4000 on node2 'not running' (7): call=227, status=complete, exit-reason='none', last-rc-change='Sun May 8 01:32:44 2016', queued=0ms, exec=0ms 925 | 926 | 927 | pgsql_REPL_INFO:node1|14|00000000090650F8 928 | 929 | 3. 再强制杀死Master上的postgres进程2次后检查集群状态。 930 | 931 | fail-count增加到3后，Pacemaker不再启动PostgreSQL，保持其为停止状态。 932 | 933 | [root@node2 ~]# cls_status 934 | Last updated: Sun May 8 01:36:16 2016 935 | Last change: Sun May 8 01:36:07 2016 by root via crm_resource on node1 936 | Stack: corosync 937 | Current DC: node2 (2) - partition with quorum 938 | Version: 1.1.12-a14efad 939 | 2 Nodes configured 940 | 4 Resources configured 941 | 942 | 943 | Online: [ node1 node2 ] 944 | 945 | Full list of resources: 946 | 947 | vip-master (ocf::heartbeat:IPaddr2): Started node1 948 | vip-slave (ocf::heartbeat:IPaddr2): Stopped 949 | Master/Slave Set: msPostgresql [pgsql] 950 | Masters: [ node1 ] 951 | Stopped: [ node2 ] 952 | 953 | Node Attributes: 954 | * Node node1: 955 | + #cluster-name : pgcluster 956 | + #site-name : pgcluster 957 | + master-pgsql : 1000 958 | + pgsql-data-status : LATEST 959 | + pgsql-master-baseline : 00000000090650F8 960 | + pgsql-status : PRI 961 | * Node node2: 962 | + #cluster-name : pgcluster 963 | + #site-name : pgcluster 964 | + master-pgsql : -INFINITY 965 | + pgsql-data-status : DISCONNECT 966 | + pgsql-status : STOP 967 | 968 | Migration summary: 969 | * Node node2: 970 | pgsql: migration-threshold=3 fail-count=3 last-failure='Sun May 8 01:36:08 2016' 971 | * Node node1: 972 | 973 | Failed actions: 974 | pgsql_monitor_4000 on node2 'not running' (7): call=240, status=complete, exit-reason='none', last-rc-change='Sun May 8 01:36:08 2016', queued=0ms, exec=0ms 975 | 976 | 977 | pgsql_REPL_INFO:node1|14|00000000090650F8 978 | 979 | 同时，Master(node1)上的复制模式被自动切换到异步复制，防止写操作hang住。 980 | 981 | [root@node1 pha4pgsql]# tail /var/lib/pgsql/tmp/rep_mode.conf 982 | synchronous_standby_names = '' 983 | 984 | 4. 修复Salve 985 | 在node2上执行`cls_cleanup`，清除fail-count后，Pacemaker会再次启动PostgreSQL进程。 986 | 987 | [root@node2 ~]# cls_cleanup 988 | All resources/stonith devices successfully cleaned up 989 | [root@node2 ~]# cls_status 990 | Last updated: Sun May 8 01:43:13 2016 991 | Last change: Sun May 8 01:43:08 2016 by root via crm_resource on node1 992 | Stack: corosync 993 | Current DC: node2 (2) - partition with quorum 994 | Version: 1.1.12-a14efad 995 | 2 Nodes configured 996 | 4 Resources configured 997 | 998 | 999 | Online: [ node1 node2 ] 1000 | 1001 | Full list of resources: 1002 | 1003 | vip-master (ocf::heartbeat:IPaddr2): Started node1 1004 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 1005 | Master/Slave Set: msPostgresql [pgsql] 1006 | Masters: [ node1 ] 1007 | Slaves: [ node2 ] 1008 | 1009 | Node Attributes: 1010 | * Node node1: 1011 | + #cluster-name : pgcluster 1012 | + #site-name : pgcluster 1013 | + master-pgsql : 1000 1014 | + pgsql-data-status : LATEST 1015 | + pgsql-master-baseline : 00000000090650F8 1016 | + pgsql-status : PRI 1017 | * Node node2: 1018 | + #cluster-name : pgcluster 1019 | + #site-name : pgcluster 1020 | + master-pgsql : 100 1021 | + pgsql-data-status : STREAMING|SYNC 1022 | + pgsql-status : HS:sync 1023 | 1024 | Migration summary: 1025 | * Node node2: 1026 | * Node node1: 1027 | 1028 | pgsql_REPL_INFO:node1|14|00000000090650F8 1029 | 1030 | 同时，Master(node1)上的复制模式又自动切换回到同步复制。 1031 | 1032 | [root@node1 pha4pgsql]# tail /var/lib/pgsql/tmp/rep_mode.conf 1033 | synchronous_standby_names = 'node2' 1034 | 1035 | ## 多节点集群的设置 1036 | 1037 | 参考pha4pgsql\template\config_muti.ini.sample的例子，编辑config.ini 1038 | 1039 | pcs_template=muti.pcs.template 1040 | OCF_ROOT=/usr/lib/ocf 1041 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 1042 | pha4pgsql_dir=/opt/pha4pgsql 1043 | writer_vip=192.168.41.136 1044 | reader_vip=192.168.41.137 1045 | node1=node1 1046 | node2=node2 1047 | node3=node3 1048 | vip_nic=eno33554984 1049 | vip_cidr_netmask=24 1050 | pgsql_pgctl=/usr/pgsql-9.5/bin/pg_ctl 1051 | pgsql_psql=/usr/pgsql-9.5/bin/psql 1052 | pgsql_pgdata=/data/postgresql/data 1053 | pgsql_pgport=5433 1054 | pgsql_restore_command="" 1055 | pgsql_rep_mode=sync 1056 | pgsql_repuser=replication 1057 | pgsql_reppassord=replication 1058 | pgsql_enable_distlock=false 1059 | pgsql_distlock_psql_cmd='/usr/bin/psql \\"host=node3 port=5439 dbname=postgres user=postgres connect_timeout=5\\"' 1060 | pgsql_distlock_lockname=pgsql_cls1 1061 | 1062 | 然后执行安装和配置 1063 | 1064 | sh install.sh 1065 | 1066 | 生成的资源配置如下，可以根据情况修改： 1067 | 1068 | [root@node1 pha4pgsql]# cat config.pcs 1069 | 1070 | pcs cluster cib pgsql_cfg 1071 | 1072 | pcs -f pgsql_cfg property set no-quorum-policy="stop" 1073 | pcs -f pgsql_cfg property set stonith-enabled="false" 1074 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 1075 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 1076 | 1077 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 1078 | ip="192.168.41.136" \ 1079 | nic="eno33554984" \ 1080 | cidr_netmask="24" \ 1081 | op start timeout="60s" interval="0s" on-fail="restart" \ 1082 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 1083 | op stop timeout="60s" interval="0s" on-fail="block" 1084 | 1085 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 1086 | ip="192.168.41.137" \ 1087 | nic="eno33554984" \ 1088 | cidr_netmask="24" \ 1089 | op start timeout="60s" interval="0s" on-fail="restart" \ 1090 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 1091 | op stop timeout="60s" interval="0s" on-fail="block" 1092 | 1093 | pcs -f pgsql_cfg resource create pgsql expgsql \ 1094 | pgctl="/usr/pgsql-9.5/bin/pg_ctl" \ 1095 | psql="/usr/pgsql-9.5/bin/psql" \ 1096 | pgdata="/data/postgresql/data" \ 1097 | pgport="5433" \ 1098 | rep_mode="sync" \ 1099 | node_list="node1 node2 node3 " \ 1100 | restore_command="" \ 1101 | primary_conninfo_opt="user=replication password=replication keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 1102 | master_ip="192.168.41.136" \ 1103 | restart_on_promote="false" \ 1104 | enable_distlock="false" \ 1105 | distlock_lock_cmd="/opt/pha4pgsql/tools/distlock '' lock distlock:pgsql_cls1 @owner 9 12" \ 1106 | distlock_unlock_cmd="/opt/pha4pgsql/tools/distlock '' unlock distlock:pgsql_cls1 @owner" \ 1107 | distlock_lockservice_deadcheck_nodelist="node1 node2 node3 " \ 1108 | op start timeout="60s" interval="0s" on-fail="restart" \ 1109 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 1110 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 1111 | op promote timeout="60s" interval="0s" on-fail="restart" \ 1112 | op demote timeout="60s" interval="0s" on-fail="stop" \ 1113 | op stop timeout="60s" interval="0s" on-fail="block" \ 1114 | op notify timeout="60s" interval="0s" 1115 | 1116 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 1117 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 1118 | migration-threshold="3" target-role="Master" 1119 | 1120 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 1121 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 1122 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 1123 | 1124 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 1125 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 1126 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 1127 | 1128 | pcs cluster cib-push pgsql_cfg 1129 | 1130 | 确定资源定义后进行配置 1131 | 1132 | ./setup.sh 1133 | 1134 | ## 错误排查 1135 | 出现故障时，可通过以下方法排除故障 1136 | 1137 | 1. 确认集群服务是否OK 1138 | 1139 | pcs status 1140 | 1141 | 2. 查看错误日志 1142 | 1143 | PostgreSQL的错误日志 1144 | /var/log/messages 1145 | /var/log/cluster/corosync.log 1146 | 1147 | Pacemaker输出的日志非常多，可以进行过滤。 1148 | 1149 | 只看Pacemaker的资源调度（在Current DC节点上执行)： 1150 | 1151 | grep Initiating /var/log/messages 1152 | 1153 | 只查看expgsql RA的输出： 1154 | 1155 | grep expgsql /var/log/messages 1156 | 1157 | ##其它故障的处理 1158 | ### 无Master时的修复 1159 | 1160 | 如果切换失败或其它原因导致集群中没有Master，可以参考下面的步骤修复 1161 | 1162 | #### 方法1：使用cleanup修复 1163 | 1164 | cls_cleanup 1165 | 1166 | 大部分情况，cleanup就可以找到Master。而且应该首先使用cleanup。如果不成功，再采用下面的方法 1167 | 1168 | #### 方法2：人工修复复制关系 1169 | 1. 将资源脱离集群管理 1170 | 1171 | cls_unmanage 1172 | 1173 | 2. 人工修复PostgreSQL，建立复制关系 1174 | 至于master的选取，可以选择`pgsql_REPL_INFO`中的master节点，或根据xlog位置确定。 1175 | 3. 在所有节点上停止PostgreSQL 1176 | 4. 清除状态并恢复集群管理 1177 | 1178 | cls_manage 1179 | cls_reset_master 1180 | 1181 | #### 方法3：快速恢复Master节点再恢复Slave 1182 | 可以明确指定将哪个节点作为Master，省略则通过xlog位置比较确定master 1183 | 1184 | cls_reset_master [master] 1185 | 1186 | 1187 | ### 疑难的Pacemaker问题的处理 1188 | 有时候可能会遇到一些顽固的问题，Pacemaker不按期望的动作，或某个资源处于错误状态却无法清除。 1189 | 这时最简单的办法就是清除CIB重新设置。可执行下面的命令完成。 1190 | 1191 | ./setup.sh [master] 1192 | 1193 | 如果不指定master，并且PostgreSQL进程是活动的，通过当前PostgreSQL进程的主备关系决定谁是master。 1194 | 如果当前没有处于主的PostgreSQL进程，通过比较xlog位置确定谁作为master。 1195 | 1196 | 在PostgreSQL服务启动期间，正常情况下，执行setup.sh不会使服务停止。 1197 | 1198 | setup.sh还可以完全取代前面的`cls_reset_master`。 1199 | 1200 | ### fail-count的清除 1201 | 如果某个节点上有资源的fail-count不为0，最好将其清除，即使当前资源是健康的。 1202 | 1203 | cls_cleanup 1204 | 1205 | ## 注意事项 1206 | 1. ./setup.sh会清除CIB，对Pacemaker资源定义的修改应该写到config.pcs里，防止下次执行setup.sh丢失。 1207 | 2. 有些包装后的脚本容易超时，比如`cls_rebuild_slave`。此时可能执行还没有完成的，需要通过`cls_status`或日志进行确认。 1208 | 1209 | 1210 | ## 附录1：对pgsql RA的修改 1211 | 本项目使用的expgsql RA是在Resource Agent 3.9.7中的pgsql RA的基础上做的修改。修改内容如下： 1212 | 1213 | 1. 引入分布式锁服务防止双节点集群出现脑裂，并防止在failover过程中丢失数据。 1214 | promote和monitor的同步复制切换为异步复制前都需要先获取锁，因此确保这两件事不能同时发生，也就防止了在同步复制模式下failover出现数据丢失。相应的引入以下参数： 1215 | 1216 | - `enable_distlock` 1217 | 是否启动分布式锁仲裁，对双节集群建议启用。 1218 | - `distlock_lock_cmd` 1219 | 分布式锁服务的加锁命令 1220 | - `distlock_unlock_cmd` 1221 | 分布式锁服务的解锁命令 1222 | - `distlock_lockservice_deadcheck_nodelist` 1223 | 无法访问分布式锁服务时，需要做二次检查的节点列表，通过ssh连接到这些节点后再获取锁。如果节点列表中所有节点都无法访问分布式锁服务，认为分布式锁服务失效，按已获得锁处理。如果节点列表中任何一个节点本身无法访问，按未获得锁处理。 1224 | 1225 | 并且内置了一个基于PostgreSQL的分布式锁实现，即tools\distlock。 1226 | 1227 | 在网络不稳定的极端情况下，主从分区后可能一会只有Master可以连上分布式锁服务，一会只有Slave可以连上分布式锁服务，导致在一个很偶然的小时间窗口内出现双主而且是异步复制，当然这种场景发生的概率极低。 1228 | 1229 | 1230 | 2. 根据Master是否发生变更动态采取restart或`pg_ctl promote`的方式提升Slave为Master。 1231 | 当Master发生变更时采用`pg_ctl promote`的方式提升Slave为Master；未发生变更时采用restart的方式提升。 1232 | 相应地废弃原pgsql RA的`restart_on_promote`参数。 1233 | 1234 | 3. 记录PostgreSQL上次时间线切换前的时间线和xlog位置信息 1235 | 这些信息记录在集群配置变量`pgsql_REPL_INFO`中。`pgsql_REPL_INFO`的值由以下3个部分组成,通过‘|’连接在一起。 1236 | 1237 | - Master节点名 1238 | - `pg_ctl promote`前的时间线 1239 | - `pg_ctl promote`前的时间线的结束位置 1240 | 1241 | RA启动时，会检查当前节点和`pgsql_REPL_INFO`中记录的状态是否有冲突，如有报错不允许资源启动。 1242 | 因为有这个检查废弃原pgsql RA的PGSQL.lock锁文件。 1243 | 1244 | 4. 资源启动时通过`pgsql_REPL_INFO`中记录的Master节点名，继续沿用原Master。 1245 | 通过这种方式加速集群的启动，并避免不必要的主从切换。集群仅在初始启动`pgsql_REPL_INFO`的值为空时，才通过xlog比较确定哪个节点作为Master。 1246 | 1247 | 1248 | 关于pgsql RA的原始功能请参考：[PgSQL Replicated Cluster](http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster) 1249 | 1250 | ## 附录2：参考 1251 | - [PostgreSQL流复制高可用的原理与实践](http://www.postgres.cn/news/viewone/1/124) 1252 | - [PgSQL Replicated Cluster](http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster) 1253 | - [Pacemaker+Corosync搭建PostgreSQL集群](http://my.oschina.net/aven92/blog/518928) 1254 | -------------------------------------------------------------------------------- /bin/cls_cleanup: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ChenHuajun/pha4pgsql/1688fb609d6687c448b96ce3df7a572805244547/bin/cls_cleanup -------------------------------------------------------------------------------- /bin/cls_maintenance: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_maintenance 5 | #InPut:none 6 | #OutPut:none 7 | # 8 | #0 正常 9 | #1 一般错误 10 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 11 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 12 | ########################################################### 13 | 14 | cd $(dirname "$0") 15 | . ../lib/common.sh 16 | 17 | # switch to maintenance mode 18 | pcs property set maintenance-mode=true -------------------------------------------------------------------------------- /bin/cls_maintenance_node: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_maintenance_node 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | if [ $# -lt 1 ]; then 19 | echo "Usage: cls_maintenance_node " >&2 20 | exit 1 21 | fi 22 | 23 | nodename="$1" 24 | 25 | pcs property set --node "$nodename" maintenance=on 26 | if [ $? -ne 0 ]; then 27 | echo "failed to execute \"pcs property set --node "$nodename" maintenance=on\"" >&2 28 | exit 1 29 | fi 30 | -------------------------------------------------------------------------------- /bin/cls_master: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_master 5 | #InPut:无 6 | #OutPut:当前Master所在节点名 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | pgsql_status 19 | if [ $? -ne 0 ]; then 20 | exit 1 21 | fi 22 | 23 | echo "$MASTER_NODE" 24 | 25 | -------------------------------------------------------------------------------- /bin/cls_online_switch: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_online_switch 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #10 不符合主从切换条件（比如当前没有Master） 12 | #11 主从切换失败 13 | #100 输入参数错误 14 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 15 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 16 | ########################################################### 17 | 18 | cd $(dirname "$0") 19 | . ../lib/common.sh 20 | 21 | #指定新的master可能导致切换后数据丢失，暂不支持 22 | #newmaster=$1 23 | 24 | pgsql_status 25 | 26 | #Check if all server is alive 27 | if [ -z "$MASTER_NODE" ]; then 28 | echo "There is no master" >&2 29 | exit 10 30 | fi 31 | 32 | for node in $node1 $node2 $node3 $othernodes 33 | do 34 | is_alive=0 35 | for alive_node in $MASTER_NODE $SLAVE_NODES 36 | do 37 | if [ $node = $alive_node ];then 38 | is_alive=1 39 | break 40 | fi 41 | done 42 | 43 | if [ $is_alive -ne 1 ]; then 44 | dead_nodes="$dead_nodes $node" 45 | fi 46 | done 47 | 48 | if [ -n "$dead_nodes" ];then 49 | echo "All nodes must alive, But there are dead nodes: $dead_nodes" >&2 50 | exit 10 51 | fi 52 | 53 | #Check the new master hostname 54 | if [ -n "$newmaster" ]; then 55 | if [ "$newmaster" = "$MASTER_NODE" ]; then 56 | echo "$newmaster is the current master" >&2 57 | exit 100 58 | fi 59 | 60 | #Check if the new master is a HA node 61 | is_HA_node=0 62 | for node in $node1 $node2 $node3 63 | do 64 | if [ $node = "$newmaster" ]; then 65 | is_HA_node=1 66 | break 67 | fi 68 | done 69 | 70 | if [ $is_HA_node -ne 1 ]; then 71 | echo "the new master must be a HA node" >&2 72 | exit 100 73 | fi 74 | 75 | #set the new master node's location score for Master role to "INFINITY" 76 | pcs resource clear msPostgresql 77 | 78 | pcs resource move msPostgresql $newmaster --master --wait 79 | if [ $? -ne 0 ]; then 80 | pcs resource clear msPostgresql >/dev/null 2>&1 81 | echo 'failed to execute "pcs resource move msPostgresql ..."' >&2 82 | exit 1 83 | fi 84 | else 85 | #set the old master node's location score for Master role to "-INFINITY" 86 | pcs resource clear msPostgresql 87 | 88 | pcs resource move msPostgresql --master --wait 89 | if [ $? -ne 0 ]; then 90 | pcs resource clear msPostgresql >/dev/null 2>&1 91 | echo 'failed to execute "pcs resource move msPostgresql ..."' >&2 92 | exit 1 93 | fi 94 | fi 95 | 96 | pcs resource clear msPostgresql 97 | 98 | # expgsql demote stop the postgres process, so it will raise a error from monitor, 99 | # so, it's needed to cleanup failcounts 100 | pcs resource cleanup msPostgresql >/dev/null 2>&1 101 | 102 | #Check if switch completed 103 | echo "wait for switch complete" 104 | OLD_MASTER_NODE="$MASTER_NODE" 105 | for i in {1..60} 106 | do 107 | pgsql_status 108 | rc=$? 109 | if [ $rc -ne 0 ]; then 110 | exit $rc 111 | fi 112 | 113 | if [ -n "$MASTER_NODE" -a "$MASTER_NODE" != "$OLD_MASTER_NODE" ]; then 114 | break 115 | fi 116 | echo -n "." 117 | sleep 1 118 | done 119 | 120 | echo 121 | 122 | if [ -z "$MASTER_NODE" -o "$MASTER_NODE" = "$OLD_MASTER_NODE" ]; then 123 | echo "online switch timeout" 124 | exit 11 125 | fi 126 | 127 | #wait for vip and other resource switch complete 128 | ./cls_start 129 | 130 | echo "online switch successed,the new master is $MASTER_NODE" 131 | 132 | 133 | -------------------------------------------------------------------------------- /bin/cls_rebuild_slave: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_rebuild_slave 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #10 不符合重建slave的前提条件（比如当前没有Master） 12 | #11 重建slave失败 13 | #100 输入参数错误 14 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 15 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 16 | ########################################################### 17 | 18 | cd $(dirname "$0") 19 | . ../lib/common.sh 20 | 21 | nodename=`hostname` 22 | 23 | pgsql_status 24 | 25 | #Check if master is alive 26 | if [ -z "$MASTER_NODE" ]; then 27 | echo "There is no master, can not recovery" >&2 28 | exit 10 29 | fi 30 | 31 | if [ "$nodename" = "$MASTER_NODE" ]; then 32 | echo "$nodename is the current master, recovery does not needed" >&2 33 | exit 100 34 | fi 35 | 36 | data_status=`crm_attribute -l forever -N $nodename -n "pgsql-data-status" -Q -q 2>/dev/null` 37 | 38 | if [ "$data_status" = "STREAMING|ASYNC" -o "$data_status" = "STREAMING|POTENTIAL" -o "$data_status" = "STREAMING|SYNC" -o "$data_status" = "LATEST" ]; then 39 | echo "The current node is healthy, recovery does not needed" >&2 40 | exit 100 41 | fi 42 | 43 | if [ -d $pgsql_pgdata ]; then 44 | 45 | echo "The PostgreSQL data directory $pgsql_pgdata exists, you must remove it first!" >&2 46 | exit 100 47 | fi 48 | 49 | # mkdir -p $pgsql_pgdata 50 | # chown postgres:postgres $pgsql_pgdata 51 | # chmod 700 $pgsql_pgdata 52 | # if [ $? -ne 0 ]; then 53 | # echo "filed to create PostgreSQL data directory $pgsql_pgdata" >&2 54 | # exit 1 55 | # fi 56 | 57 | ./cls_standby_node $nodename 58 | if [ $? -ne 0 ]; then 59 | echo "failed to call cls_standby_node $nodename" >&2 60 | exit 1 61 | fi 62 | 63 | PG_BASEBACKUP=$(dirname "$pgsql_pgctl")/pg_basebackup 64 | 65 | su - postgres -c "PGPASSWORD=$pgsql_reppassord $PG_BASEBACKUP -h $writer_vip -p $pgsql_pgport -U $pgsql_repuser -D $pgsql_pgdata -X stream -P" 66 | if [ $? -ne 0 ]; then 67 | echo "failed to call \"$PG_BASEBACKUP -h $writer_vip -p $pgsql_pgport -U $pgsql_repuser -D $pgsql_pgdata -X stream -P\"" >&2 68 | ./cls_unstandby_node $nodename >/dev/null 2>/dev/null 69 | exit 1 70 | fi 71 | 72 | echo "pg_basebackup complete!" 73 | 74 | ./cls_unstandby_node $nodename 75 | 76 | ./cls_cleanup 77 | 78 | #Check if recovery completed 79 | echo "wait for recovery complete" 80 | OLD_MASTER_NODE="$MASTER_NODE" 81 | for i in {1..60} 82 | do 83 | data_status=`crm_attribute -l forever -N $nodename -n "pgsql-data-status" -Q` 84 | rc=$? 85 | if [ $rc -ne 0 ]; then 86 | exit 1 87 | fi 88 | 89 | if [ "$data_status" = "STREAMING|ASYNC" -o "$data_status" = "STREAMING|POTENTIAL" -o "$data_status" = "STREAMING|SYNC" ]; then 90 | break 91 | fi 92 | echo -n "." 93 | sleep 1 94 | done 95 | 96 | echo 97 | 98 | if [ "$data_status" != "STREAMING|ASYNC" -a "$data_status" != "STREAMING|POTENTIAL" -a "$data_status" != "STREAMING|SYNC" ]; then 99 | echo "slave recovery of $nodename timeout" 100 | exit 11 101 | fi 102 | 103 | echo "slave recovery of $nodename successed" 104 | 105 | 106 | -------------------------------------------------------------------------------- /bin/cls_repair_by_pg_rewind: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_repair_slave 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #10 不符合修复Slave的条件（比如当前没有Master） 12 | #11 修复Slave失败 13 | #100 输入参数错误 14 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 15 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 16 | ########################################################### 17 | 18 | cd $(dirname "$0") 19 | . ../lib/common.sh 20 | 21 | nodename=`hostname` 22 | 23 | pgsql_status 24 | 25 | #Check if master is alive 26 | if [ -z "$MASTER_NODE" ]; then 27 | echo "There is no master, can not recovery" >&2 28 | exit 10 29 | fi 30 | 31 | if [ "$nodename" = "$MASTER_NODE" ]; then 32 | echo "$nodename is the current master, recovery does not needed" >&2 33 | exit 100 34 | fi 35 | 36 | data_status=`crm_attribute -l forever -N $nodename -n "pgsql-data-status" -Q` 37 | if [ $? -ne 0 ]; then 38 | echo "failed to get pgsql-data-status of $nodename" >&2 39 | exit 1 40 | fi 41 | 42 | if [ "$data_status" = "STREAMING|ASYNC" -o "$data_status" = "STREAMING|POTENTIAL" -o "$data_status" = "STREAMING|SYNC" -o "$data_status" = "LATEST" ]; then 43 | echo "The current node is healthy, recovery does not needed" >&2 44 | exit 100 45 | fi 46 | 47 | ./cls_standby_node $nodename 48 | if [ $? -ne 0 ]; then 49 | echo "failed to call cls_standby_node $nodename" >&2 50 | exit 1 51 | fi 52 | 53 | PG_REWIND=$(dirname "$pgsql_pgctl")/pg_rewind 54 | 55 | su - postgres -c "$PG_REWIND -D $pgsql_pgdata --source-server='host=$writer_vip port=$pgsql_pgport dbname=template1 user=$pgsql_repuser password=$pgsql_reppassord' -P" 56 | if [ $? -ne 0 ]; then 57 | echo "failed to call \"$PG_REWIND -D $pgsql_pgdata --source-server='host=$writer_vip' port=$pgsql_pgport -P\"" >&2 58 | ./cls_unstandby_node $nodename >/dev/null 2>/dev/null 59 | exit 1 60 | fi 61 | 62 | echo "pg_rewind complete!" 63 | 64 | ./cls_unstandby_node $nodename 65 | 66 | ./cls_cleanup 67 | 68 | #Check if recovery completed 69 | echo "wait for recovery complete" 70 | OLD_MASTER_NODE="$MASTER_NODE" 71 | for i in {1..60} 72 | do 73 | data_status=`crm_attribute -l forever -N $nodename -n "pgsql-data-status" -Q` 74 | rc=$? 75 | if [ $rc -ne 0 ]; then 76 | exit 1 77 | fi 78 | 79 | if [ "$data_status" = "STREAMING|ASYNC" -o "$data_status" = "STREAMING|POTENTIAL" -o "$data_status" = "STREAMING|SYNC" ]; then 80 | break 81 | fi 82 | echo -n "." 83 | sleep 1 84 | done 85 | 86 | echo 87 | 88 | if [ "$data_status" != "STREAMING|ASYNC" -a "$data_status" != "STREAMING|POTENTIAL" -a "$data_status" != "STREAMING|SYNC" ]; then 89 | echo "slave recovery of $nodename timeout" 90 | exit 11 91 | fi 92 | 93 | echo "slave recovery of $nodename successed" 94 | 95 | 96 | -------------------------------------------------------------------------------- /bin/cls_reset_master: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_reset_master 5 | #InPut:新Master节点名，缺省由Pacemaker根据xlog位置比较确定Master 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #10 不符合恢复master的条件（比如当前没有Master） 12 | #11 恢复master失败 13 | #100 输入参数错误 14 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 15 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 16 | ########################################################### 17 | 18 | cd $(dirname "$0") 19 | . ../lib/common.sh 20 | 21 | nodename=`hostname` 22 | newmaster=$1 23 | 24 | pgsql_status 25 | if [ $? -ne 0 ]; then 26 | echo "failed to execute \"pgsql_status\"">&2 27 | return 1 28 | fi 29 | 30 | #Check if master is alive 31 | if [ -n "$MASTER_NODE" ]; then 32 | echo "Current master exists, reset master is not allowed" >&2 33 | exit 10 34 | fi 35 | 36 | ./cls_stop 37 | 38 | # clear pgsql-data-status attribute from all node 39 | for node in `crm_node -l|awk '{print $2}'` 40 | do 41 | crm_attribute -l forever -N ${node} -n "pgsql-data-status" -D 42 | done 43 | 44 | if [ -n "$newmaster" ]; then 45 | crm_attribute --type crm_config --name pgsql_REPL_INFO -s pgsql_replication -v "$newmaster" 46 | else 47 | crm_attribute --type crm_config --name pgsql_REPL_INFO -s pgsql_replication -D 48 | fi 49 | 50 | ./cls_cleanup 51 | 52 | ./cls_start 53 | 54 | #Check if master startup completed 55 | echo "wait for master startup complete" 56 | for i in {1..60} 57 | do 58 | pgsql_status 59 | rc=$? 60 | if [ $rc -ne 0 ]; then 61 | exit 1 62 | fi 63 | 64 | if [ -n "$MASTER_NODE" ]; then 65 | break 66 | fi 67 | echo -n "." 68 | sleep 1 69 | done 70 | 71 | echo 72 | 73 | if [ -z "$MASTER_NODE" ]; then 74 | echo "failed to startup the new master" 75 | exit 11 76 | fi 77 | 78 | echo "master reset and startup successed, the master is $MASTER_NODE" 79 | 80 | 81 | -------------------------------------------------------------------------------- /bin/cls_standby_node: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_standby 5 | #InPut:要下线的节点名 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | if [ $# -lt 1 ]; then 19 | echo "Usage: cls_standby " >&2 20 | exit 1 21 | fi 22 | 23 | nodename="$1" 24 | 25 | pcs cluster standby "$nodename" 26 | if [ $? -ne 0 ]; then 27 | echo "failed to execute \"pcs cluster unstandby $nodename\"" >&2 28 | exit 1 29 | fi 30 | 31 | #Check if resource stoped in the standby node 32 | check_with_timeout "check_node_standbyed $nodename" 60 33 | if [ $? -ne 0 ]; then 34 | echo "stop resource in standby node timeout" >&2 35 | exit 1 36 | fi -------------------------------------------------------------------------------- /bin/cls_start: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_start 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #2 启动Slave失败 12 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 13 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 14 | ########################################################### 15 | 16 | cd $(dirname "$0") 17 | . ../lib/common.sh 18 | 19 | pcs resource cleanup msPostgresql 20 | 21 | pcs resource enable msPostgresql --wait 22 | rc=$? 23 | if [ $rc -ne 0 ]; then 24 | echo "failed to execute \"pcs resource enable msPostgresql --wait\" rc=$rc" >&2 25 | exit 1 26 | fi 27 | 28 | #Check if resource started normally 29 | check_with_timeout "check_resource_started msPostgresql vip-master" 5 30 | if [ $? -ne 0 ]; then 31 | echo "start msPostgresql and vip-master timeout" >&2 32 | exit 1 33 | fi 34 | 35 | check_with_timeout "check_resource_started vip-slave" 5 36 | if [ $? -ne 0 ]; then 37 | echo "start vip-slave timeout" >&2 38 | exit 2 39 | fi 40 | 41 | check_with_timeout "check_replication_ok" 30 42 | if [ $? -ne 0 ]; then 43 | echo "start replication timeout" >&2 44 | exit 2 45 | fi 46 | 47 | echo "start resource successful" 48 | -------------------------------------------------------------------------------- /bin/cls_status: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_status 5 | #InPut:无 6 | #OutPut:当前集群的状态(crm_mon -Afr1的输出) 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | crm_mon -Afr1 19 | 20 | repl_info=`crm_attribute --type crm_config --name pgsql_REPL_INFO -s pgsql_replication -G -q 2>/dev/null` 21 | echo 22 | echo "pgsql_REPL_INFO:$repl_info" 23 | 24 | 25 | 26 | -------------------------------------------------------------------------------- /bin/cls_stop: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_stop 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | pcs resource disable msPostgresql --wait 19 | rc=$? 20 | if [ $rc -ne 0 ]; then 21 | echo "failed to execute \"pcs resource disable msPostgresql --wait\" rc=$rc" >&2 22 | exit 1 23 | fi 24 | 25 | #Check if resources stoped normally 26 | check_with_timeout "check_resource_stoped msPostgresql vip-master vip-slave" 5 27 | if [ $? -ne 0 ]; then 28 | echo "stop resources timeout" >&2 29 | exit 1 30 | fi 31 | 32 | echo "stop resource successful" 33 | 34 | -------------------------------------------------------------------------------- /bin/cls_unmaintenance: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_unmaintenance 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | 19 | # switch maintenance mode to normal 20 | pcs property unset maintenance-mode 21 | -------------------------------------------------------------------------------- /bin/cls_unmaintenance_node: -------------------------------------------------------------------------------- 1 | ##!/bin/bash 2 | 3 | ########################################################### 4 | #cls_unmaintenance_node 5 | #InPut:无 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 一般错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | if [ $# -lt 1 ]; then 19 | echo "Usage: cls_maintenance_node " >&2 20 | exit 1 21 | fi 22 | 23 | nodename="$1" 24 | 25 | pcs property set --node "$nodename" maintenance=off 26 | if [ $? -ne 0 ]; then 27 | echo "failed to execute \"pcs property set --node "$nodename" maintenance=off\"" >&2 28 | exit 1 29 | fi 30 | 31 | -------------------------------------------------------------------------------- /bin/cls_unstandby_node: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########################################################### 4 | #cls_unstandby 5 | #InPut:准备上线的节点名 6 | #OutPut:无 7 | # 8 | #return 9 | #0 正常 10 | #1 错误 11 | #101 无法获取crm信息，比如pacemaker,corosync服务未启动(保留) 12 | #102 分区未达到法定票数或和任意一个另外的HA节点状态不一致(保留) 13 | ########################################################### 14 | 15 | cd $(dirname "$0") 16 | . ../lib/common.sh 17 | 18 | if [ $# -lt 1 ]; then 19 | echo "Usage: cls_unstandby " >&2 20 | exit 1 21 | fi 22 | 23 | nodename="$1" 24 | 25 | pcs cluster unstandby "$nodename" 26 | if [ $? -ne 0 ]; then 27 | echo "failed to execute \"pcs cluster unstandby $nodename\"" >&2 28 | exit 1 29 | fi 30 | 31 | 32 | -------------------------------------------------------------------------------- /doc/lvsdr.md: -------------------------------------------------------------------------------- 1 | # 基于Pacemkaer Resource Agent的LVS负载均衡 2 | 3 | ## 前言 4 | 对于有主从状态的集群，实现读负载均衡需要将读请求分发到各Slave上，并且主从发生切换后，要自动调整分发策略。然而，当前主流的LVS监控方案，keepalived或Pacemaker + ldirectord并不能很好的支持这一点，它们需要在发生failover后修改相应的配置文件，这并不是非常方便。为了把LVS更好集成到Pacemaker集群里，将LVS相关的控制操作包装成资源代理，并提供多种灵活的方式动态配置real server。当前只支持工作在LVS的DR模式。 5 | 6 | ## 功能 7 | 1. Director Server和Real Server上内核参数的自动设置 8 | 2. 负载均衡策略等LVS参数的配置 9 | 3. Real Server权重的配置 10 | 4. 多种Real Server列表配置方式 11 | * a)静态列表 12 | * b)根据资源依赖动态设置 13 | * c)根据节点属性动态设置 14 | * d)以上a,b,c 3种的组合 15 | * e)根据外部脚本动态获取 16 | 5. 多种Real Server的健康检查方式 17 | * 由Real Server对应的RA(如pgsql)检查 18 | * 由外部脚本动态检查 19 | 6. 在线动态调整Real Server 20 | Real Server列表发生变更时，已建立的到非故障Real Server的连接不受影响。 21 | 22 | 23 | ## 配置参数 24 | * vip 25 | 26 | 虚拟服务的VIP. 27 | 28 | * port 29 | 30 | 虚拟服务的端口号 31 | 32 | * `virtual_service_options` 33 | 34 | 传递给ipvsadm的虚拟服务的选项，比如"-s rr". 35 | 36 | * `default_weight` 37 | 38 | Real Server的缺省权重，默认为1. 39 | 40 | * `weight_of_realservers` 41 | 42 | Real Server的host和权重组合的列表，设置形式为"node1,weight1 node2,weight2 ..."。 43 | 如果省略权重，则使用`default_weight`。使用了`realserver_dependent_resource`， 44 | `realserver_dependent_attribute_name`或`realserver_dependent_attribute_value`参数时， 45 | host必须是节点的主机名。 46 | 47 | * `realserver_dependent_resource` 48 | 49 | Real Server依赖的资源，只有被分配了该资源的节点会被加入到LVS的real server列表中。 50 | 如果`realserver_get_real_servers_script`不为空，该参数将失效。 51 | 52 | * `realserver_dependent_attribute_name` 53 | 54 | Real Server依赖的节点属性名。 55 | 如果`realserver_get_real_servers_script`或`realserver_check_active_slave_script`不为空，该参数将失效。 56 | 57 | * `realserver_dependent_attribute_value` 58 | 59 | Real Server依赖的节点属性值的正则表达式，比如对于pgsql RA的从节点，可以设置为"HS:sync|HS:potential|HS:async" 60 | 如果`realserver_get_real_servers_script`或`realserver_check_active_slave_script`不为空，该参数将失效。 61 | 62 | * `realserver_get_real_servers_script` 63 | 64 | 动态获取Real Server列表的脚本。该脚本输出空格分隔得Real Server列表。 65 | 66 | 67 | * `realserver_check_active_real_server_script` 68 | 动态检查Real Server健康的脚本。该脚本接收节点名作为参数。 69 | 如果`realserver_get_real_servers_script`或`realserver_check_active_slave_script`不为空，该参数将失效。 70 | 71 | 72 | ## 安装 73 | 74 | ## 前提需求 75 | 76 | * Pacemaker 77 | * Corosync 78 | * ipvsadm 79 | 80 | ### 获取RA脚本 81 | 82 | wget https://raw.githubusercontent.com/ChenHuajun/pha4pgsql/master/ra/lvsdr 83 | wget https://raw.githubusercontent.com/ChenHuajun/pha4pgsql/master/ra/lvsdr-realsvr 84 | 85 | ### 安装RA脚本 86 | 87 | cp lvsdr lvsdr-realsvr /usr/lib/ocf/resource.d/heartbeat/ 88 | chmod +x /usr/lib/ocf/resource.d/heartbeat/lvsdr 89 | chmod +x /usr/lib/ocf/resource.d/heartbeat/lvsdr-realsvr 90 | 91 | 92 | ## 使用示例 93 | 94 | 以下是PostgreSQL主从集群中读负载均衡的配置示例，读请求通过LVS的RR负载均衡策略平均分散到2个Slave节点上。 95 | 96 | ## 示例1:通过资源依赖和节点属性动态设置Real Server列表 97 | 98 | 配置`lvsdr-realsvr`和每个expgsql的Slave资源部署在一起 99 | 100 | pcs -f pgsql_cfg resource create lvsdr-realsvr lvsdr-realsvr \ 101 | vip="192.168.0.237" \ 102 | nic_lo="lo:0" \ 103 | op start timeout="60s" interval="0s" on-fail="restart" \ 104 | op monitor timeout="30s" interval="60s" on-fail="restart" \ 105 | op stop timeout="60s" interval="0s" on-fail="block" 106 | 107 | pcs -f pgsql_cfg resource clone lvsdr-realsvr clone-node-max=1 notify=false 108 | 109 | pcs -f pgsql_cfg constraint colocation add lvsdr with vip-slave INFINITY 110 | pcs -f pgsql_cfg constraint colocation add lvsdr-realsvr-clone with Slave msPostgresql INFINITY 111 | 112 | 配值lvsdr的real server列表依赖于`lvsdr-realsvr-clone`资源和节点的`pgsql-status`特定属性值 113 | 114 | pcs -f pgsql_cfg resource create lvsdr lvsdr \ 115 | vip="192.168.0.237" \ 116 | port="5432" \ 117 | realserver_dependent_resource="lvsdr-realsvr-clone" \ 118 | realserver_dependent_attribute_name="pgsql-status" \ 119 | realserver_dependent_attribute_value="HS:sync|HS:potential|HS:async" \ 120 | virtual_service_options="-s rr" \ 121 | op start timeout="60s" interval="0s" on-fail="restart" \ 122 | op monitor timeout="30s" interval="5s" on-fail="restart" \ 123 | op stop timeout="60s" interval="0s" on-fail="block" 124 | 125 | 126 | 完整的配置如下: 127 | 128 | pcs cluster cib pgsql_cfg 129 | 130 | pcs -f pgsql_cfg property set no-quorum-policy="stop" 131 | pcs -f pgsql_cfg property set stonith-enabled="false" 132 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 133 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 134 | 135 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 136 | ip="192.168.0.236" \ 137 | nic="eno16777736" \ 138 | cidr_netmask="24" \ 139 | op start timeout="60s" interval="0s" on-fail="restart" \ 140 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 141 | op stop timeout="60s" interval="0s" on-fail="block" 142 | 143 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 144 | ip="192.168.0.237" \ 145 | nic="eno16777736" \ 146 | cidr_netmask="24" \ 147 | op start timeout="60s" interval="0s" on-fail="restart" \ 148 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 149 | op stop timeout="60s" interval="0s" on-fail="block" 150 | 151 | pcs -f pgsql_cfg resource create pgsql expgsql \ 152 | pgctl="/usr/pgsql-9.5/bin/pg_ctl" \ 153 | psql="/usr/pgsql-9.5/bin/psql" \ 154 | pgdata="/data/postgresql/data" \ 155 | pgport="5432" \ 156 | rep_mode="sync" \ 157 | node_list="node1 node2 node3 " \ 158 | restore_command="" \ 159 | primary_conninfo_opt="user=replication password=replication keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 160 | master_ip="192.168.0.236" \ 161 | restart_on_promote="false" \ 162 | op start timeout="60s" interval="0s" on-fail="restart" \ 163 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 164 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 165 | op promote timeout="60s" interval="0s" on-fail="restart" \ 166 | op demote timeout="60s" interval="0s" on-fail="stop" \ 167 | op stop timeout="60s" interval="0s" on-fail="block" \ 168 | op notify timeout="60s" interval="0s" 169 | 170 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 171 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 172 | migration-threshold="3" target-role="Master" 173 | 174 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 175 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 176 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 177 | 178 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 179 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 180 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 181 | 182 | pcs -f pgsql_cfg resource create lvsdr lvsdr \ 183 | vip="192.168.0.237" \ 184 | port="5432" \ 185 | realserver_dependent_resource="lvsdr-realsvr-clone" \ 186 | realserver_dependent_attribute_name="pgsql-status" \ 187 | realserver_dependent_attribute_value="HS:sync|HS:potential|HS:async" \ 188 | virtual_service_options="-s rr" \ 189 | op start timeout="60s" interval="0s" on-fail="restart" \ 190 | op monitor timeout="30s" interval="5s" on-fail="restart" \ 191 | op stop timeout="60s" interval="0s" on-fail="block" 192 | 193 | pcs -f pgsql_cfg resource create lvsdr-realsvr lvsdr-realsvr \ 194 | vip="192.168.0.237" \ 195 | nic_lo="lo:0" \ 196 | op start timeout="60s" interval="0s" on-fail="restart" \ 197 | op monitor timeout="30s" interval="60s" on-fail="restart" \ 198 | op stop timeout="60s" interval="0s" on-fail="block" 199 | 200 | pcs -f pgsql_cfg resource clone lvsdr-realsvr clone-node-max=1 notify=false 201 | 202 | pcs -f pgsql_cfg constraint colocation add lvsdr with vip-slave INFINITY 203 | pcs -f pgsql_cfg constraint colocation add lvsdr-realsvr-clone with Slave msPostgresql INFINITY 204 | 205 | pcs -f pgsql_cfg constraint order stop vip-slave then start vip-slave symmetrical=false score=0 206 | pcs -f pgsql_cfg constraint order stop vip-master then start vip-master symmetrical=false score=0 207 | pcs -f pgsql_cfg constraint order start lvsdr-realsvr-clone then start lvsdr symmetrical=false score=0 208 | pcs -f pgsql_cfg constraint order start lvsdr then start vip-slave symmetrical=false score=0 209 | 210 | pcs cluster cib-push pgsql_cfg 211 | 212 | ### 示例2:从Master节点查询Slave节点 213 | 214 | 配值lvsdr的real server列表通过`get_active_slaves`脚本在Master上动态查询Slave节点。 215 | 216 | pcs -f pgsql_cfg resource create lvsdr lvsdr \ 217 | vip="192.168.0.237" \ 218 | port="5432" \ 219 | default_weight="0" \ 220 | weight_of_realservers="node1,1 node2,1 node3,1 192.168.0.234,1" \ 221 | realserver_get_real_servers_script="/opt/pha4pgsql/tools/get_active_slaves /usr/pgsql/bin/psql \"host=192.168.0.236 port=5432 dbname=postgres user=replication password=replication connect_timeout=5\"" \ 222 | virtual_service_options="-s rr" \ 223 | op start timeout="60s" interval="0s" on-fail="restart" \ 224 | op monitor timeout="30s" interval="5s" on-fail="restart" \ 225 | op stop timeout="60s" interval="0s" on-fail="block" 226 | 227 | 采用这种方式可以将Pacemaker集群以外的Slave作为real server加入到LVS。对这样的节点需要进行下面的设置 228 | 229 | 1. 设置作为LVS real server的系统参数 230 | 231 | echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore 232 | echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce 233 | echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 234 | echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 235 | 236 | 2. 在lo网卡上添加读VIP 237 | 238 | ip a add 192.168.0.237/32 dev lo:0 239 | 240 | 3. 设置Slave节点连接信息中`application_name`为该节点的主机名或ip地址。 241 | 242 | [root@node4 pha4pgsql]# cat /data/postgresql/data/recovery.conf 243 | standby_mode = 'on' 244 | primary_conninfo = 'host=192.168.0.236 port=5432 application_name=192.168.0.234 user=replication password=replication keepalives_idle=60 keepalives_interval=5 keepalives_count=5' 245 | restore_command = '' 246 | recovery_target_timeline = 'latest' 247 | 248 | 249 | ### 示例3:直接连接Slave检查节点健康状况 250 | 251 | 通过`default_weight`和`weight_of_realservers`指定real server一览，并通过调用`check_active_slave`脚本，依次连接到real server中的每个节点上检查其是否可以连接并且是Slave。 252 | 253 | pcs -f pgsql_cfg resource create lvsdr lvsdr \ 254 | vip="192.168.0.237" \ 255 | port="5432" \ 256 | default_weight="1" \ 257 | weight_of_realservers="node1 node2 node3 192.168.0.234" \ 258 | realserver_check_active_real_server_script="/opt/pha4pgsql/tools/check_active_slave /usr/pgsql/bin/psql \"port=5432 dbname=postgres user=replication password=replication connect_timeout=5\" -h" \ 259 | virtual_service_options="-s rr" \ 260 | op start timeout="60s" interval="0s" on-fail="restart" \ 261 | op monitor timeout="30s" interval="5s" on-fail="restart" \ 262 | op stop timeout="60s" interval="0s" on-fail="block" 263 | 264 | pcs resource update lvsdr default_weight="1" 265 | pcs resource update lvsdr weight_of_realservers="node1 node2 node3 192.168.0.234" 266 | pcs resource update lvsdr realserver_dependent_resource="" 267 | pcs resource update lvsdr realserver_get_real_servers_script="" 268 | pcs resource update lvsdr realserver_check_active_real_server_script="/opt/pha4pgsql/tools/check_active_slave /usr/pgsql/bin/psql \"port=5432 dbname=postgres user=replication password=replication connect_timeout=5\" -h" 269 | 270 | -------------------------------------------------------------------------------- /doc/muti_with_lvs.md: -------------------------------------------------------------------------------- 1 | 2 | # 基于Pacemaker的PostgreSQL一主多从读负载均衡集群搭建 3 | 4 | ## 简介 5 | PostgreSQL的HA方案有很多种，本文演示基于Pacemaker的PostgreSQL一主多从读负载均衡集群搭建。 6 | 搭建过程并不是使用原始的Pacemaker pgsql RA脚本，而使用以下我修改和包装的脚本集pha4pgsql。 7 | 8 | * [https://github.com/ChenHuajun/pha4pgsql](https://github.com/ChenHuajun/pha4pgsql) 9 | 10 | ### 目标集群特性 11 | 1. 秒级自动failover 12 | 2. failover零数据丢失(防脑裂) 13 | 3. 支持在线主从切换 14 | 4. 支持读写分离 15 | 5. 支持读负载均衡 16 | 6. 支持动态增加和删除只读节点 17 | 18 | 19 | ### 环境 20 | 21 | - OS:CentOS 7.3 22 | - 节点1:node1(192.168.0.231) 23 | - 节点2:node2(192.168.0.232) 24 | - 节点2:node3(192.168.0.233) 25 | - writer_vip:192.168.0.236 (无任何指向的同网段ip) 26 | - reader_vip:192.168.0.237 (无任何指向的同网段ip) 27 | 28 | ### 依赖软件 29 | - pacemaker 30 | - corosync 31 | - pcs 32 | - ipvsadm 33 | 34 | ## 安装与配置 35 | ### 环境准备 36 | 1. 所有节点设置时钟同步 37 | 38 | cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 39 | ntpdate time.windows.com && hwclock -w 40 | 41 | 2. 所有节点设置独立的主机名(node1，node2，node3) 42 | 43 | hostnamectl set-hostname node1 44 | 45 | 3. 设置对所有节点的域名解析 46 | 47 | $ vi /etc/hosts 48 | ... 49 | 192.168.0.231 node1 50 | 192.168.0.232 node2 51 | 192.168.0.233 node3 52 | 53 | 4. 在所有节点上禁用SELINUX 54 | 55 | $ setenforce 0 56 | $ vi /etc/selinux/config 57 | ... 58 | SELINUX=disabled 59 | 60 | 5. 在所有节点上禁用防火墙 61 | 62 | systemctl disable firewalld.service 63 | systemctl stop firewalld.service 64 | 65 | 如果开启防火墙需要开放postgres，pcsd和corosync的端口。参考[CentOS 7防火墙设置示例](http://blog.chinaunix.net/uid-20726500-id-5748864.html) 66 | 67 | * postgres:5432/tcp 68 | * pcsd:2224/tcp 69 | * corosync:5405/udp 70 | 71 | ### 安装和配置Pacemaker+Corosync集群软件 72 | 73 | #### 安装Pacemaker和Corosync及相关软件包 74 | 在所有节点执行： 75 | 76 | yum install -y pacemaker corosync pcs ipvsadm 77 | 78 | #### 启用pcsd服务 79 | 在所有节点执行： 80 | 81 | systemctl start pcsd.service 82 | systemctl enable pcsd.service 83 | 84 | #### 设置hacluster用户密码 85 | 在所有节点执行： 86 | 87 | echo hacluster | passwd hacluster --stdin 88 | 89 | #### 集群认证 90 | 在任何一个节点上执行: 91 | 92 | pcs cluster auth -u hacluster -p hacluster node1 node2 node3 93 | 94 | #### 同步配置 95 | 在任何一个节点上执行: 96 | 97 | pcs cluster setup --last_man_standing=1 --name pgcluster node1 node2 node3 98 | 99 | #### 启动集群 100 | 在任何一个节点上执行: 101 | 102 | pcs cluster start --all 103 | 104 | ### 安装和配置PostgreSQL 105 | 106 | #### 安装PostgreSQL 107 | 安装9.2以上的PostgreSQL，本文通过PostgreSQL官方yum源安装CentOS 7.3对应的PostgreSQL 9.6 108 | 109 | * [https://yum.postgresql.org/](https://yum.postgresql.org/) 110 | 111 | 在所有节点执行： 112 | 113 | yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 114 | 115 | yum install -y postgresql96 postgresql96-contrib postgresql96-libs postgresql96-server postgresql96-devel 116 | 117 | ln -sf /usr/pgsql-9.6 /usr/pgsql 118 | echo 'export PATH=/usr/pgsql/bin:$PATH' >>~postgres/.bash_profile 119 | 120 | 121 | #### 创建Master数据库 122 | 在node1节点执行： 123 | 124 | 1. 创建数据目录 125 | 126 | mkdir -p /pgsql/data 127 | chown -R postgres:postgres /pgsql/ 128 | chmod 0700 /pgsql/data 129 | 130 | 2. 初始化db 131 | 132 | su - postgres 133 | initdb -D /pgsql/data/ 134 | 135 | 3. 修改postgresql.conf 136 | 137 | listen_addresses = '*' 138 | wal_level = hot_standby 139 | wal_log_hints = on 140 | synchronous_commit = on 141 | max_wal_senders=5 142 | wal_keep_segments = 32 143 | hot_standby = on 144 | wal_sender_timeout = 5000 145 | wal_receiver_status_interval = 2 146 | max_standby_streaming_delay = -1 147 | max_standby_archive_delay = -1 148 | restart_after_crash = off 149 | hot_standby_feedback = on 150 | 151 | 注：1.设置"`wal_log_hints = on`"可以使用`pg_rewind`修复旧Master。 152 | 153 | 154 | 4. 修改`pg_hba.conf` 155 | 156 | local all all trust 157 | host all all 192.168.0.0/24 md5 158 | host replication all 192.168.0.0/24 md5 159 | 160 | 5. 启动postgres 161 | 162 | pg_ctl -D /pgsql/data/ start 163 | 164 | 6. 创建复制用户 165 | 166 | createuser --login --replication replication -P -s 167 | 168 | 注:加上“-s”选项可支持`pg_rewind`。 169 | 170 | #### 创建Slave数据库 171 | 在node2和node3节点执行： 172 | 173 | 1. 创建数据目录 174 | 175 | mkdir -p /pgsql/data 176 | chown -R postgres:postgres /pgsql/ 177 | chmod 0700 /pgsql/data 178 | 179 | 2. 创建基础备份 180 | 181 | su - postgres 182 | pg_basebackup -h node1 -U replication -D /pgsql/data/ -X stream -P 183 | 184 | 185 | #### 停止PostgreSQL服务 186 | 在node1上执行: 187 | pg_ctl -D /pgsql/data/ stop 188 | 189 | 190 | ### 安装和配置pha4pgsql 191 | 在任意一个节点上执行: 192 | 193 | 1. 下载pha4pgsql 194 | 195 | cd /opt 196 | git clone git://github.com/Chenhuajun/pha4pgsql.git 197 | 198 | 2. 拷贝config.ini 199 | 200 | cd /opt/pha4pgsql 201 | cp template/config_muti_with_lvs.ini.sample config.ini 202 | 203 | 注:如果不需要配置基于LVS的负载均衡，可使用模板`config_muti.ini.sample` 204 | 205 | 3. 修改config.ini 206 | 207 | pcs_template=muti_with_lvs.pcs.template 208 | OCF_ROOT=/usr/lib/ocf 209 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 210 | pha4pgsql_dir=/opt/pha4pgsql 211 | writer_vip=192.168.0.236 212 | reader_vip=192.168.0.237 213 | node1=node1 214 | node2=node2 215 | node3=node3 216 | othernodes="" 217 | vip_nic=ens37 218 | vip_cidr_netmask=24 219 | pgsql_pgctl=/usr/pgsql/bin/pg_ctl 220 | pgsql_psql=/usr/pgsql/bin/psql 221 | pgsql_pgdata=/pgsql/data 222 | pgsql_pgport=5432 223 | pgsql_restore_command="" 224 | pgsql_rep_mode=sync 225 | pgsql_repuser=replication 226 | pgsql_reppassord=replication 227 | 228 | 注: 229 | 1、vip_nic表示当前机器的网卡名称，通过ifconfig可以查到name 230 | 2、如果你安装pg 的路径和文档不一致的话，以上路径要核对清楚 231 | 232 | 4. 安装pha4pgsql (用于同步pg数据的工具) 233 | 234 | sh install.sh 235 | ./setup.sh 236 | 237 | 执行install.sh使用了scp拷贝文件，中途会多次要求输入其它节点的root账号。 238 | install.sh执行会生成Pacemaker的配置脚本/opt/pha4pgsql/config.pcs，可以根据情况对其中的参数进行调优后再执行setup.sh。 239 | 240 | 241 | 5. 设置环境变量 242 | 243 | export PATH=/opt/pha4pgsql/bin:$PATH 244 | echo 'export PATH=/opt/pha4pgsql/bin:$PATH' >>/root/.bash_profile 245 | 246 | 6. 启动集群 247 | 248 | cls_start 249 | 250 | 7. 确认集群状态 251 | 252 | cls_status 253 | 254 | cls_status的输出如下： 255 | 256 | [root@node1 pha4pgsql]# cls_status 257 | Stack: corosync 258 | Current DC: node1 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum 259 | Last updated: Wed Jan 11 00:53:58 2017 Last change: Wed Jan 11 00:45:54 2017 by root via crm_attribute on node1 260 | 261 | 3 nodes and 9 resources configured 262 | 263 | Online: [ node1 node2 node3 ] 264 | 265 | Full list of resources: 266 | 267 | vip-master (ocf::heartbeat:IPaddr2): Started node1 268 | vip-slave (ocf::heartbeat:IPaddr2): Started node2 269 | Master/Slave Set: msPostgresql [pgsql] 270 | Masters: [ node1 ] 271 | Slaves: [ node2 node3 ] 272 | lvsdr (ocf::heartbeat:lvsdr): Started node2 273 | Clone Set: lvsdr-realsvr-clone [lvsdr-realsvr] 274 | Started: [ node2 node3 ] 275 | Stopped: [ node1 ] 276 | 277 | Node Attributes: 278 | * Node node1: 279 | + master-pgsql : 1000 280 | + pgsql-data-status : LATEST 281 | + pgsql-master-baseline : 00000000050001B0 282 | + pgsql-status : PRI 283 | * Node node2: 284 | + master-pgsql : 100 285 | + pgsql-data-status : STREAMING|SYNC 286 | + pgsql-status : HS:sync 287 | * Node node3: 288 | + master-pgsql : -INFINITY 289 | + pgsql-data-status : STREAMING|ASYNC 290 | + pgsql-status : HS:async 291 | 292 | Migration Summary: 293 | * Node node2: 294 | * Node node3: 295 | * Node node1: 296 | 297 | pgsql_REPL_INFO:node1|1|00000000050001B0 298 | 299 | 检查集群的健康状态。完全健康的集群需要满足以下条件： 300 | 301 | 1. msPostgresql在每个节点上都已启动 302 | 2. 在其中一个节点上msPostgresql处于Master状态，其它的为Salve状态 303 | 3. Salve节点的data-status值是以下中的一个 304 | - STREAMING|SYNC 305 | 同步复制Slave 306 | - STREAMING|POTENTIAL 307 | 候选同步复制Slave 308 | - STREAMING|ASYNC 309 | 异步复制Slave 310 | 311 | `pgsql_REPL_INFO`的3段内容分别指当前master，上次提升前的时间线和xlog位置。 312 | 313 | pgsql_REPL_INFO:node1|1|00000000050001B0 314 | 315 | 316 | LVS配置在node2上 317 | 318 | [root@node2 ~]# ipvsadm -L 319 | IP Virtual Server version 1.2.1 (size=4096) 320 | Prot LocalAddress:Port Scheduler Flags 321 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 322 | TCP node2:postgres rr 323 | -> node2:postgres Route 1 0 0 324 | -> node3:postgres Route 1 0 0 325 | 326 | ## 故障测试 327 | 328 | ### Master故障 329 | 330 | 1. 停止Master上的网卡模拟故障 331 | 332 | [root@node1 pha4pgsql]# ifconfig ens37 down 333 | 334 | 335 | 2. 检查集群状态 336 | 337 | Pacemaker已经将Master和写VIP切换到node2上 338 | 339 | [root@node2 ~]# cls_status 340 | resource msPostgresql is NOT running 341 | Stack: corosync 342 | Current DC: node2 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum 343 | Last updated: Wed Jan 11 01:25:08 2017 Last change: Wed Jan 11 01:21:26 2017 by root via crm_attribute on node2 344 | 345 | 3 nodes and 9 resources configured 346 | 347 | Online: [ node2 node3 ] 348 | OFFLINE: [ node1 ] 349 | 350 | Full list of resources: 351 | 352 | vip-master (ocf::heartbeat:IPaddr2): Started node2 353 | vip-slave (ocf::heartbeat:IPaddr2): Started node3 354 | Master/Slave Set: msPostgresql [pgsql] 355 | Masters: [ node2 ] 356 | Slaves: [ node3 ] 357 | Stopped: [ node1 ] 358 | lvsdr (ocf::heartbeat:lvsdr): Started node3 359 | Clone Set: lvsdr-realsvr-clone [lvsdr-realsvr] 360 | Started: [ node3 ] 361 | Stopped: [ node1 node2 ] 362 | 363 | Node Attributes: 364 | * Node node2: 365 | + master-pgsql : 1000 366 | + pgsql-data-status : LATEST 367 | + pgsql-master-baseline : 00000000050008E0 368 | + pgsql-status : PRI 369 | * Node node3: 370 | + master-pgsql : 100 371 | + pgsql-data-status : STREAMING|SYNC 372 | + pgsql-status : HS:sync 373 | 374 | Migration Summary: 375 | * Node node2: 376 | * Node node3: 377 | 378 | pgsql_REPL_INFO:node2|2|00000000050008E0 379 | 380 | 381 | LVS和读VIP被移到了node3上 382 | 383 | [root@node3 ~]# ipvsadm -L 384 | IP Virtual Server version 1.2.1 (size=4096) 385 | Prot LocalAddress:Port Scheduler Flags 386 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 387 | TCP node3:postgres rr 388 | -> node3:postgres Route 1 0 0 389 | 390 | 391 | 3. 修复旧Master的网卡 392 | 393 | 在旧Master node1上，postgres进程还在（注1）。但是由于配置的是同步复制，数据无法写入不会导致脑裂。 394 | 395 | [root@node1 pha4pgsql]# ps -ef|grep postgres 396 | root 20295 2269 0 01:35 pts/0 00:00:00 grep --color=auto postgres 397 | postgres 20556 1 0 00:45 ? 00:00:01 /usr/pgsql-9.6/bin/postgres -D /pgsql/data -c config_file=/pgsql/data/postgresql.conf 398 | postgres 20566 20556 0 00:45 ? 00:00:00 postgres: logger process 399 | postgres 20574 20556 0 00:45 ? 00:00:00 postgres: checkpointer process 400 | postgres 20575 20556 0 00:45 ? 00:00:00 postgres: writer process 401 | postgres 20576 20556 0 00:45 ? 00:00:00 postgres: stats collector process 402 | postgres 22390 20556 0 00:45 ? 00:00:00 postgres: wal writer process 403 | postgres 22391 20556 0 00:45 ? 00:00:00 postgres: autovacuum launcher process 404 | 405 | 启动网卡后，postgres进程被停止 406 | 407 | [root@node1 pha4pgsql]# ifconfig ens37 up 408 | [root@node1 pha4pgsql]# ps -ef|grep postgres 409 | root 21360 2269 0 01:36 pts/0 00:00:00 grep --color=auto postgres 410 | [root@node1 pha4pgsql]# cls_status 411 | resource msPostgresql is NOT running 412 | Stack: corosync 413 | Current DC: node2 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum 414 | Last updated: Wed Jan 11 01:36:20 2017 Last change: Wed Jan 11 01:36:00 2017 by hacluster via crmd on node2 415 | 416 | 3 nodes and 9 resources configured 417 | 418 | Online: [ node1 node2 node3 ] 419 | 420 | Full list of resources: 421 | 422 | vip-master (ocf::heartbeat:IPaddr2): Started node2 423 | vip-slave (ocf::heartbeat:IPaddr2): Started node3 424 | Master/Slave Set: msPostgresql [pgsql] 425 | Masters: [ node2 ] 426 | Slaves: [ node3 ] 427 | Stopped: [ node1 ] 428 | lvsdr (ocf::heartbeat:lvsdr): Started node3 429 | Clone Set: lvsdr-realsvr-clone [lvsdr-realsvr] 430 | Started: [ node3 ] 431 | Stopped: [ node1 node2 ] 432 | 433 | Node Attributes: 434 | * Node node1: 435 | + master-pgsql : -INFINITY 436 | + pgsql-data-status : DISCONNECT 437 | + pgsql-status : STOP 438 | * Node node2: 439 | + master-pgsql : 1000 440 | + pgsql-data-status : LATEST 441 | + pgsql-master-baseline : 00000000050008E0 442 | + pgsql-status : PRI 443 | * Node node3: 444 | + master-pgsql : 100 445 | + pgsql-data-status : STREAMING|SYNC 446 | + pgsql-status : HS:sync 447 | 448 | Migration Summary: 449 | * Node node2: 450 | * Node node3: 451 | * Node node1: 452 | pgsql: migration-threshold=3 fail-count=1000000 last-failure='Wed Jan 11 01:36:08 2017' 453 | 454 | Failed Actions: 455 | * pgsql_start_0 on node1 'unknown error' (1): call=278, status=complete, exitreason='The master's timeline forked off current database system timeline 2 before latest checkpoint location 0000000005000B80, REPL_INF', 456 | last-rc-change='Wed Jan 11 01:36:07 2017', queued=0ms, exec=745ms 457 | 458 | 459 | pgsql_REPL_INFO:node2|2|00000000050008E0 460 | 461 | 462 | 注1：这是通过`ifconfig ens37 down`停止网卡模拟故障的特殊现象（或者说是corosync的bug），Pacemkaer的日志中不停的输出以下警告。在实际的物理机宕机或网卡故障时，故障节点会由于失去quorum，postgres进程会被Pacemaker主动停止。 463 | 464 | [43260] node3 corosyncwarning [MAIN ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly. 465 | 466 | 4. 修复旧Master(node1)并作为Slave加入集群 467 | 468 | 通过pg_rewind修复旧Master 469 | 470 | [root@node1 pha4pgsql]# cls_repair_by_pg_rewind 471 | resource msPostgresql is NOT running 472 | resource msPostgresql is NOT running 473 | resource msPostgresql is NOT running 474 | connected to server 475 | servers diverged at WAL position 0/50008E0 on timeline 2 476 | rewinding from last common checkpoint at 0/5000838 on timeline 2 477 | reading source file list 478 | reading target file list 479 | reading WAL in target 480 | need to copy 99 MB (total source directory size is 117 MB) 481 | 102359/102359 kB (100%) copied 482 | creating backup label and updating control file 483 | syncing target data directory 484 | Done! 485 | pg_rewind complete! 486 | resource msPostgresql is NOT running 487 | resource msPostgresql is NOT running 488 | Waiting for 1 replies from the CRMd. OK 489 | wait for recovery complete 490 | ..... 491 | slave recovery of node1 successed 492 | 493 | 检查集群状态 494 | 495 | [root@node1 pha4pgsql]# cls_status 496 | Stack: corosync 497 | Current DC: node2 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum 498 | Last updated: Wed Jan 11 01:39:30 2017 Last change: Wed Jan 11 01:37:35 2017 by root via crm_attribute on node2 499 | 500 | 3 nodes and 9 resources configured 501 | 502 | Online: [ node1 node2 node3 ] 503 | 504 | Full list of resources: 505 | 506 | vip-master (ocf::heartbeat:IPaddr2): Started node2 507 | vip-slave (ocf::heartbeat:IPaddr2): Started node3 508 | Master/Slave Set: msPostgresql [pgsql] 509 | Masters: [ node2 ] 510 | Slaves: [ node1 node3 ] 511 | lvsdr (ocf::heartbeat:lvsdr): Started node3 512 | Clone Set: lvsdr-realsvr-clone [lvsdr-realsvr] 513 | Started: [ node1 node3 ] 514 | Stopped: [ node2 ] 515 | 516 | Node Attributes: 517 | * Node node1: 518 | + master-pgsql : -INFINITY 519 | + pgsql-data-status : STREAMING|ASYNC 520 | + pgsql-status : HS:async 521 | * Node node2: 522 | + master-pgsql : 1000 523 | + pgsql-data-status : LATEST 524 | + pgsql-master-baseline : 00000000050008E0 525 | + pgsql-status : PRI 526 | * Node node3: 527 | + master-pgsql : 100 528 | + pgsql-data-status : STREAMING|SYNC 529 | + pgsql-status : HS:sync 530 | + pgsql-xlog-loc : 000000000501F118 531 | 532 | Migration Summary: 533 | * Node node2: 534 | * Node node3: 535 | * Node node1: 536 | 537 | pgsql_REPL_INFO:node2|2|00000000050008E0 538 | 539 | 540 | ### Slave故障 541 | 542 | LVS配置在node3上，2个real server 543 | 544 | [root@node3 ~]# ipvsadm -L 545 | IP Virtual Server version 1.2.1 (size=4096) 546 | Prot LocalAddress:Port Scheduler Flags 547 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 548 | TCP node3:postgres rr 549 | -> node1:postgres Route 1 0 0 550 | -> node3:postgres Route 1 0 0 551 | 552 | 在其中一个Slave(node1)上停止网卡 553 | 554 | [root@node1 pha4pgsql]# ifconfig ens37 down 555 | 556 | Pacemaker已自动修改LVS的real server配置 557 | 558 | [root@node3 ~]# ipvsadm -L 559 | IP Virtual Server version 1.2.1 (size=4096) 560 | Prot LocalAddress:Port Scheduler Flags 561 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 562 | TCP node3:postgres rr 563 | -> node3:postgres Route 1 0 0 564 | 565 | 566 | ## 添加Slave扩容读负载均衡 567 | 568 | 目前配置的是1主2从集群，2个Slave通过读VIP+LVS做读负载均衡，如果读负载很高可以添加额外的Slave扩展读性能。 569 | 把更多的Slave直接添加到Pacemaker集群中可以达到这个目的，但过多的节点数会增加Pacemaker+Corosync集群的复杂性和通信负担（Corosync的通信是一个环路，节点数越多，时延越大）。所以不把额外的Slave加入Pacemaker集群，仅仅加到LVS的real server中，并让lvsdr监视Slave的健康状况，动态更新LVS的real server列表。方法如下： 570 | 571 | 572 | ### 创建额外的Slave数据库 573 | 准备第4台机器node4(192.168.0.234),并在该机器上执行以下命令创建新的Slave 574 | 575 | 1. 禁用SELINUX 576 | 577 | $ setenforce 0 578 | $ vi /etc/selinux/config 579 | ... 580 | SELINUX=disabled 581 | 582 | 2. 禁用防火墙 583 | 584 | systemctl disable firewalld.service 585 | systemctl stop firewalld.service 586 | 587 | 3. 安装PostgreSQL 588 | 589 | yum install -y https://yum.postgresql.org/9.6/redhat/rhel-7.3-x86_64/pgdg-centos96-9.6-3.noarch.rpm 590 | 591 | yum install -y postgresql96 postgresql96-contrib postgresql96-libs postgresql96-server postgresql96-devel 592 | 593 | ln -sf /usr/pgsql-9.6 /usr/pgsql 594 | echo 'export PATH=/usr/pgsql/bin:$PATH' >>~postgres/.bash_profile 595 | 596 | 4. 创建数据目录 597 | 598 | mkdir -p /pgsql/data 599 | chown -R postgres:postgres /pgsql/ 600 | chmod 0700 /pgsql/data 601 | 602 | 5. 创建Salve备份 603 | 604 | 从当前的Master节点(即写VIP 192.168.0.236)拉取备份创建Slave 605 | 606 | su - postgres 607 | pg_basebackup -h 192.168.0.236 -U replication -D /pgsql/data/ -X stream -P 608 | 609 | 610 | 6. 编辑postgresql.conf 611 | 612 | 将postgresql.conf中的下面一行删掉 613 | 614 | $vi /pgsql/data/postgresql.conf 615 | ... 616 | #include '/var/lib/pgsql/tmp/rep_mode.conf' # added by pgsql RA 617 | 618 | 7. 编辑recovery.conf 619 | 620 | $vi /pgsql/data/recovery.conf 621 | standby_mode = 'on' 622 | primary_conninfo = 'host=192.168.0.236 port=5432 application_name=192.168.0.234 user=replication password=replication keepalives_idle=60 keepalives_interval=5 keepalives_count=5' 623 | restore_command = '' 624 | recovery_target_timeline = 'latest' 625 | 626 | 627 | 上面的`application_name`设置为本节点的IP地址192.168.0.234 628 | 629 | 8. 启动Slave 630 | 631 | pg_ctl -D /pgsql/data/ start 632 | 633 | 在Master上检查postgres wal sender进程，新建的Slave(192.168.0.234)已经和Master建立了流复制。 634 | 635 | [root@node1 pha4pgsql]# ps -ef|grep '[w]al sender' 636 | postgres 32387 111175 0 12:15 ? 00:00:00 postgres: wal sender process replication 192.168.0.234(47894) streaming 0/7000220 637 | postgres 116675 111175 0 12:01 ? 00:00:00 postgres: wal sender process replication 192.168.0.233(33652) streaming 0/7000220 638 | postgres 117079 111175 0 12:01 ? 00:00:00 postgres: wal sender process replication 192.168.0.232(40088) streaming 0/7000220 639 | 640 | ### 配置LVS real server 641 | 642 | 1. 设置系统参数 643 | 644 | echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore 645 | echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce 646 | echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 647 | echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 648 | 649 | 2. 在lo网卡上添加读VIP 650 | 651 | ip a add 192.168.0.237/32 dev lo:0 652 | 653 | ### 将新建的Slave加入到LVS中 654 | 655 | 现在LVS的配置中还没有把新的Slave作为real server加入 656 | 657 | [root@node3 ~]# ipvsadm 658 | IP Virtual Server version 1.2.1 (size=4096) 659 | Prot LocalAddress:Port Scheduler Flags 660 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 661 | TCP node3:postgres rr 662 | -> node2:postgres Route 1 0 0 663 | -> node3:postgres Route 1 0 0 664 | 665 | 在Pacemaker集群的任意一个节点(node1,node2或node3)上，修改lvsdr RA的配置，加入新的real server。 666 | 667 | [root@node2 ~]# pcs resource update lvsdr realserver_get_real_servers_script="/opt/pha4pgsql/tools/get_active_slaves /usr/pgsql/bin/psql \"host=192.168.0.236 port=5432 dbname=postgres user=replication password=replication connect_timeout=5\"" 668 | 669 | 设置`realserver_get_real_servers_script`参数后，lvsdr会通过脚本获取LVS的real server列表，这里的`get_active_slaves`会通过写VIP连接到Master节点获取所有以连接到Master的Slave的`application_name`作为real server。设置后新的Slave 192.168.0.234已经被加入到real server 列表中了。 670 | 671 | [root@node2 ~]# ipvsadm 672 | IP Virtual Server version 1.2.1 (size=4096) 673 | Prot LocalAddress:Port Scheduler Flags 674 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 675 | TCP node2:postgres rr 676 | -> node2:postgres Route 1 0 0 677 | -> node3:postgres Route 1 0 0 678 | -> 192.168.0.234:postgres Route 1 0 0 679 | 680 | 681 | ### 测试读负载均衡 682 | 在当前的Master节点(node1)上通过读VIP访问postgres,可以看到psql会轮询连接到3个不同的Slave上。 683 | 684 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 685 | 2017-01-14 12:01:48.068455+08 686 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 687 | 2017-01-14 12:01:12.222412+08 688 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 689 | 2017-01-14 12:15:19.614782+08 690 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 691 | 2017-01-14 12:01:48.068455+08 692 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 693 | 2017-01-14 12:01:12.222412+08 694 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" -tAc "select pg_postmaster_start_time()" 695 | 2017-01-14 12:15:19.614782+08 696 | 697 | 下面测试Salve节点发生故障的场景。 698 | 先连接到其中一台Slave 699 | 700 | [root@node1 pha4pgsql]# psql "host=192.168.0.237 port=5432 dbname=postgres user=replication password=replication" 701 | psql (9.6.1) 702 | Type "help" for help. 703 | 704 | 当前连接在node4上 705 | 706 | [root@node4 ~]# ps -ef|grep postgres 707 | postgres 11911 1 0 12:15 pts/0 00:00:00 /usr/pgsql-9.6/bin/postgres -D /pgsql/data 708 | postgres 11912 11911 0 12:15 ? 00:00:00 postgres: logger process 709 | postgres 11913 11911 0 12:15 ? 00:00:00 postgres: startup process recovering 000000090000000000000007 710 | postgres 11917 11911 0 12:15 ? 00:00:00 postgres: checkpointer process 711 | postgres 11918 11911 0 12:15 ? 00:00:00 postgres: writer process 712 | postgres 11920 11911 0 12:15 ? 00:00:00 postgres: stats collector process 713 | postgres 11921 11911 0 12:15 ? 00:00:04 postgres: wal receiver process streaming 0/7000CA0 714 | postgres 12004 11911 0 13:19 ? 00:00:00 postgres: replication postgres 192.168.0.231(42116) idle 715 | root 12006 2230 0 13:19 pts/0 00:00:00 grep --color=auto postgres 716 | 717 | 强制杀死node4上的postgres进程 718 | 719 | [root@node4 ~]# killall postgres 720 | 721 | lvsdr探测到node4挂了后会自动将其从real server列表中摘除 722 | 723 | [root@node2 ~]# ipvsadm 724 | IP Virtual Server version 1.2.1 (size=4096) 725 | Prot LocalAddress:Port Scheduler Flags 726 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 727 | TCP node2:postgres rr 728 | -> node2:postgres Route 1 0 0 729 | -> node3:postgres Route 1 0 0 730 | 731 | 732 | psql执行下一条SQL时就会自动连接到其它Slave上。 733 | 734 | postgres=# select pg_postmaster_start_time(); 735 | FATAL: terminating connection due to administrator command 736 | server closed the connection unexpectedly 737 | This probably means the server terminated abnormally 738 | before or while processing the request. 739 | The connection to the server was lost. Attempting reset: Succeeded. 740 | postgres=# select pg_postmaster_start_time(); 741 | pg_postmaster_start_time 742 | ------------------------------- 743 | 2017-01-14 12:01:48.068455+08 744 | (1 row) 745 | 746 | 747 | ### 指定静态的real server列表 748 | 749 | 有时候不希望将所有连接到Master的Slave都加入到LVS的real server中，比如某个`Slave`可能实际上是`pg_receivexlog`。 750 | 这时可以在lvsdr上指定静态的real server列表作为白名单。 751 | 752 | #### 方法1: 753 | 754 | 通过`default_weight`和`weight_of_realservers`指定各个real server的权重，将不想参与到负载均衡的Slave的权重设置为0。 755 | 并且还是通过在Master上查询Slave一览的方式监视Slave健康状态。 756 | 757 | 下面在Pacemaker集群的任意一个节点(node1,node2或node3)上，修改lvsdr RA的配置，设置有效的real server列表为node,node2和node3。 758 | 759 | pcs resource update lvsdr default_weight="0" 760 | pcs resource update lvsdr weight_of_realservers="node1,1 node2,1 node3,1" 761 | pcs resource update lvsdr realserver_get_real_servers_script="/opt/pha4pgsql/tools/get_active_slaves /usr/pgsql/bin/psql \"host=192.168.0.236 port=5432 dbname=postgres user=replication password=replication connect_timeout=5\"" 762 | 763 | 在lvsdr所在节点上检查LVS的状态，此时node4(192.168.0.234)的权重为0，LVS不会往node4上转发请求。 764 | 765 | [root@node2 ~]# ipvsadm 766 | IP Virtual Server version 1.2.1 (size=4096) 767 | Prot LocalAddress:Port Scheduler Flags 768 | -> RemoteAddress:Port Forward Weight ActiveConn InActConn 769 | 770 | TCP node2:postgres rr 771 | -> node2:postgres Route 1 0 0 772 | -> node3:postgres Route 1 0 0 773 | -> 192.168.0.234:postgres Route 0 0 0 774 | 775 | #### 方法2: 776 | 777 | 通过`default_weight`和`weight_of_realservers`指定real server一览，并通过调用`check_active_slave`脚本，依次连接到real server中的每个节点上检查其是否可以连接并且是Slave。 778 | 779 | pcs resource update lvsdr default_weight="1" 780 | pcs resource update lvsdr weight_of_realservers="node1 node2 node3 192.168.0.234" 781 | pcs resource update lvsdr realserver_dependent_resource="" 782 | pcs resource update lvsdr realserver_get_real_servers_script="" 783 | pcs resource update lvsdr realserver_check_active_real_server_script="/opt/pha4pgsql/tools/check_active_slave /usr/pgsql/bin/psql \"port=5432 dbname=postgres user=replication password=replication connect_timeout=5\" -h" 784 | 785 | 推荐采用方法1，因为每次健康检查只需要1次连接。 786 | 787 | 788 | ## 参考 789 | - [Pacemaker High Availability for PostgreSQL](https://github.com/ChenHuajun/pha4pgsql) 790 | - [PostgreSQL流复制高可用的原理与实践](http://www.postgres.cn/news/viewone/1/124) 791 | - [PgSQL Replicated Cluster](http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster) 792 | - [Pacemaker+Corosync搭建PostgreSQL集群](http://my.oschina.net/aven92/blog/518928) 793 | -------------------------------------------------------------------------------- /gencfg.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # create config.pcs via configure file template 3 | 4 | cd "$(dirname "$0")" 5 | . ./config.ini 6 | 7 | template_file=template/${pcs_template} 8 | 9 | # generate config.pcs 10 | sed -e 's#${pha4pgsql_dir}#'${pha4pgsql_dir}#g \ 11 | -e 's#${writer_vip}#'${writer_vip}#g \ 12 | -e 's#${reader_vip}#'${reader_vip}#g \ 13 | -e 's#${node1}#'${node1}#g \ 14 | -e 's#${node2}#'${node2}#g \ 15 | -e 's#${node3}#'${node3}#g \ 16 | -e 's#${othernodes}#'${othernodes}#g \ 17 | -e 's#${vip_nic}#'${vip_nic}#g \ 18 | -e 's#${vip_cidr_netmask}#'${vip_cidr_netmask}#g \ 19 | -e 's#${pgsql_pgctl}#'${pgsql_pgctl}#g \ 20 | -e 's#${pgsql_psql}#'${pgsql_psql}#g \ 21 | -e 's#${pgsql_pgdata}#'${pgsql_pgdata}#g \ 22 | -e 's#${pgsql_pghost}#'${pgsql_pghost}#g \ 23 | -e 's#${pgsql_pgport}#'${pgsql_pgport}#g \ 24 | -e 's#${pgsql_socketdir}#'${pgsql_socketdir}#g \ 25 | -e 's#${pgsql_rep_mode}#'${pgsql_rep_mode}#g \ 26 | -e 's#${pgsql_repuser}#'"${pgsql_repuser}#g" \ 27 | -e 's#${pgsql_reppassord}#'"${pgsql_reppassord}#g" \ 28 | -e 's#${pgsql_restore_command}#'"${pgsql_restore_command}#g" \ 29 | -e 's#${pgsql_enable_distlock}#'${pgsql_enable_distlock}#g \ 30 | -e 's#${pgsql_distlock_psql_cmd}#'"${pgsql_distlock_psql_cmd}#g" \ 31 | -e 's#${pgsql_distlock_lockname}#'${pgsql_distlock_lockname}#g \ 32 | $template_file > config.pcs 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd "$(dirname "$0")" 4 | . ./config.ini 5 | 6 | if [ -z "${pha4pgsql_dir}" -o -z "${OCF_ROOT}" ]; then 7 | echo "Please set pha4pgsql_dir and OCF_ROOT in the config.ini first!" 8 | exit 1 9 | fi 10 | 11 | echo "generate config.pcs..." 12 | sh ./gencfg.sh 13 | if [ $? -ne 0 ]; then 14 | echo 'failed to execute "gencfg.sh"' >&2 15 | exit 1 16 | fi 17 | 18 | chmod 600 config.ini config.pcs 19 | chmod +x tools/* ra/* *.sh bin/cls_* 20 | 21 | # copy scripts 22 | for node in ${node1} ${node2} ${node3} ${othernodes} 23 | do 24 | if [ "`hostname`" == "$node" ]; then 25 | mkdir -p ${pha4pgsql_dir} 26 | if [ "`pwd`" != "${pha4pgsql_dir}" ]; then 27 | cp -rf * ${pha4pgsql_dir}/ 28 | fi 29 | cp -f ra/* ${OCF_ROOT}/resource.d/heartbeat/ 30 | else 31 | ssh "$node" mkdir -p ${pha4pgsql_dir} 32 | scp -rp * ${node}:${pha4pgsql_dir}/ 33 | scp -p ra/* ${node}:${OCF_ROOT}/resource.d/heartbeat/ 34 | fi 35 | done 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /lib/common.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | . ../config.ini 4 | 5 | #check cluster status 6 | ########################################################### 7 | #mysql_status() 8 | #InPut:无 9 | #OutPut: 10 | #设置以下SHELL内部变量 11 | # CLUSTER_STATUS: "crm_mon -Afr1"的输出 12 | # MASTER_NODE: pgsql Master所在节点 13 | # SLAVE_NODES: pgsql Slave所在节点列表 14 | # 15 | #return 16 | #0 Normal 17 | #1 无法获取cib信息，比如pacemaker,corosync服务未启动 18 | #2 分区未达到法定票数或和任意一个另外的HA节点状态不一致 19 | ########################################################### 20 | pgsql_status() { 21 | 22 | # crm_verify -L 23 | # if [ $? -ne 0 ]; then 24 | # echo "failed to execute \"crm_verify -L\"">&2 25 | # return 1 26 | # fi 27 | 28 | CLUSTER_STATUS=`crm_mon -Afr1` 29 | if [ $? -ne 0 ]; then 30 | echo "failed to execute \"crm_mon -Afr1\"">&2 31 | return 1 32 | fi 33 | 34 | if [ -n "$node3" ]; then 35 | echo "$CLUSTER_STATUS"|grep "partition with quorum" >/dev/null 36 | if [ $? -ne 0 ]; then 37 | echo "partition WITHOUT quorum" >&2 38 | return 2 39 | fi 40 | 41 | if [ "$use_ssh" = "yes" ]; then 42 | # 测试发现几个节点的crm状态可能会不一致，保险起见需要进行检查。 43 | # 但是网络出现故障（丢包而不是拒绝）时，ssh命令可能会阻塞，故需要设置连接超时。 44 | localnode=`hostname` 45 | local_status=`echo "$CLUSTER_STATUS"|grep -E "Current DC:|Online:"` 46 | for hanode in $node1 $node2 $node3 47 | do 48 | if [ "$hanode" != "$localnode" ]; then 49 | remote_status=`ssh -o ConnectTimeout=2 $hanode crm_mon -Afr1|grep -E "Current DC:|Online:"` 50 | if [ "$remote_status" = "$local_status" ]; then 51 | break; 52 | fi 53 | fi 54 | done 55 | 56 | if [ "$remote_status" != "$local_status" ]; then 57 | echo "inconsistentat status with other HA nodes" >&2 58 | return 2 59 | fi 60 | fi 61 | 62 | fi 63 | 64 | pgsql_locates=`crm_resource --resource msPostgresql --locate` 65 | if [ $? -ne 0 ]; then 66 | echo "failed to execute \"crm_resource --resource msPostgresql --locate\"">&2 67 | return 1 68 | fi 69 | 70 | MASTER_NODE=`echo "$pgsql_locates" | grep "running on:" | grep Master | awk '{print $6}'` 71 | SLAVE_NODES=`echo "$pgsql_locates" | grep "running on:" | grep -v Master | awk '{print $6}'` 72 | 73 | # translate \n to space 74 | SLAVE_NODES=`echo $SLAVE_NODES` 75 | 76 | return 0 77 | } 78 | 79 | check_with_timeout() 80 | { 81 | func="$1" 82 | timeout=$2 83 | 84 | start=`date +%s` 85 | expire=`expr $timeout + $start` 86 | 87 | while true 88 | do 89 | output=`$func 2>&1` 90 | if [ $? -eq 0 ]; then 91 | if [ -n "$output" ];then 92 | echo "Successful to call \"$func\" :$output" 93 | fi 94 | return 0 95 | fi 96 | 97 | if [ `date +%s` -gt $expire ]; then 98 | echo "Tried $timeout seconds and failed to call \"$func\" :$output" 99 | return 1 100 | fi 101 | sleep 1 102 | done 103 | } 104 | 105 | check_resource_started() 106 | { 107 | for resource in $* 108 | do 109 | crm_resource --resource $resource --locate 2>/dev/null | grep "is running on:" >/dev/null 110 | if [ $? -ne 0 ]; then 111 | return 1 112 | fi 113 | done 114 | 115 | return 0 116 | } 117 | 118 | check_resource_stoped() 119 | { 120 | for resource in $* 121 | do 122 | crm_resource --resource $resource --locate 2>/dev/null | grep "is running on:" >/dev/null 123 | if [ $? -eq 0 ]; then 124 | return 1 125 | fi 126 | done 127 | 128 | return 0 129 | } 130 | 131 | check_node_standbyed() 132 | { 133 | nodename=$1 134 | 135 | for resource in $RESOURCE_LIST 136 | do 137 | crm_resource --resource $resource --locate 2>/dev/null | grep "is running on: $nodename" >/dev/null 138 | if [ $? -eq 0 ]; then 139 | return 1 140 | fi 141 | done 142 | 143 | return 0 144 | } 145 | 146 | check_replication_ok() 147 | { 148 | local data_status 149 | 150 | for node in $node1 $node2 $node3 $othernodes 151 | do 152 | data_status=`crm_attribute -l forever -N "$node" -n "pgsql-data-status" -G -q 2>/dev/null` 153 | if [ "$data_status" != "LATEST" -a "$data_status" != "STREAMING|ASYNC" -a "$data_status" != "STREAMING|POTENTIAL" -a "$data_status" != "STREAMING|SYNC" ]; then 154 | return 1 155 | fi 156 | done 157 | 158 | return 0 159 | } 160 | 161 | pgsql_status 162 | rc=$? 163 | if [ $rc -ne 0 ]; then 164 | exit `expr 100 + $rc` 165 | fi -------------------------------------------------------------------------------- /ra/lvsdr: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # 4 | # Resource Agent script for LVS(DR) 5 | # 6 | # Authors: Chen Huajun 7 | # 8 | ####################################################################### 9 | # Initialization: 10 | 11 | : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} 12 | . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs 13 | 14 | ####################################################################### 15 | 16 | #OCF_RESKEY_realserver_dependent_resource_default="lvsdr-realsvr-clone" 17 | OCF_RESKEY_realserver_dependent_attribute_name_default="readable" 18 | OCF_RESKEY_realserver_dependent_attribute_value_default="1" 19 | OCF_RESKEY_virtual_service_options_default="-s rr" 20 | OCF_RESKEY_default_weight_default="1" 21 | 22 | #: ${OCF_RESKEY_realserver_dependent_resource=${OCF_RESKEY_realserver_dependent_resource_default}} 23 | : ${OCF_RESKEY_realserver_dependent_attribute_name=${OCF_RESKEY_realserver_dependent_attribute_name_default}} 24 | : ${OCF_RESKEY_realserver_dependent_attribute_value=${OCF_RESKEY_realserver_dependent_attribute_value_default}} 25 | : ${OCF_RESKEY_virtual_service_options=${OCF_RESKEY_virtual_service_options_default}} 26 | : ${OCF_RESKEY_default_weight=${OCF_RESKEY_default_weight_default}} 27 | 28 | ####################################################################### 29 | # Convenience variables 30 | 31 | CRM_RESOURCE="${HA_SBIN_DIR}/crm_resource" 32 | CRM_NODE="${HA_SBIN_DIR}/crm_node" 33 | CRM_ATTRIBUTE="${HA_SBIN_DIR}/crm_attribute" 34 | 35 | ####################################################################### 36 | # update heatbeat file for purpose of monitoring 37 | 38 | if [ -n "$OCF_RESKEY_heatbeat_file" ];then 39 | echo "$(date +%s)" >$OCF_RESKEY_heatbeat_file 40 | fi 41 | 42 | ####################################################################### 43 | 44 | meta_data() { 45 | cat < 47 | 48 | 49 | 1.0 50 | 51 | 52 | Resource Agent script for LVS(DR). 53 | #Currently only one lvsdr RA is allowed within a cluster. 54 | 55 | Resource Agent script for LVS(DR) 56 | 57 | 58 | 59 | 60 | Vitrual IP for virtual service. 61 | 62 | Vitrual IP for virtual service 63 | 64 | 65 | 66 | 67 | 68 | Port for virtual service. 69 | 70 | Port for virtual service 71 | 72 | 73 | 74 | 75 | 76 | Additional virtual service options for ipvsadm -A,such as "-s rr". 77 | 78 | Additional virtual service options 79 | 80 | 81 | 82 | 83 | 84 | Default weight of real server. 85 | 86 | Default weight of real server 87 | 88 | 89 | 90 | 91 | 92 | Weight define of real servers in form of "node1,weight1 node2,weight2 ...". 93 | Real servers without weight definition will use the value of default_weight . 94 | The "node1" must be the hostname of this node, when use realserver_dependent_resource, 95 | realserver_dependent_attribute_name or realserver_dependent_attribute_value to 96 | check health of real servres. 97 | 98 | weight define of real servers 99 | 100 | 101 | 102 | 103 | 104 | File to record timestamp of last activity of this resource agent for purpose of monitoring. 105 | If heatbeat_file be setted, write seconds since 1970-01-01 00:00:00 UTC to heatbeat_file 106 | every time this resource agent be invoked. 107 | 108 | file to record timestamp of last activity of this resource agent for purpose of monitoring 109 | 110 | 111 | 112 | 113 | 114 | The dependent resource for a node to become a real server. 115 | If this is setted to "", all node will become a candidate real server. 116 | This parameter will be disabled if realserver_get_real_servers_script is not empty. 117 | 118 | The dependent resource for a node to become a real server 119 | 120 | 121 | 122 | 123 | 124 | An node attribute that indicate whether a node can become a real server. 125 | If this is setted to "", all node will become a candidate real server. 126 | This parameter will be disabled if realserver_get_real_servers_script or realserver_check_active_slave_script is not empty. 127 | 128 | An node attribute that indicate whether a node can become a real server 129 | 130 | 131 | 132 | 133 | 134 | The expect attribute value for a node which can become a real server. 135 | The value can be a regular expression, such as "HS:sync|HS:potential|HS:async". 136 | This parameter will be disabled if realserver_get_real_servers_script or realserver_check_active_slave_script is not empty. 137 | 138 | The expect attribute value for a node which can become a real server 139 | 140 | 141 | 142 | 143 | 144 | The script which return the space separated active slaves. 145 | 146 | The script which return the space separated active slaves 147 | 148 | 149 | 150 | 151 | 152 | The script which check if the gave node can act as a real server. 153 | The gave node is pass as command line parameter. 154 | This parameter will be disabled if realserver_get_real_servers_script is not empty. 155 | 156 | The script which return the space separated active slaves 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | END 171 | } 172 | 173 | ####################################################################### 174 | 175 | lvsdr_usage() { 176 | cat </dev/null)" 243 | echo "$readable"| grep -Ex "${OCF_RESKEY_realserver_dependent_attribute_value}" >/dev/null 244 | if [ $? -eq 0 ]; then 245 | realsvr_list="$realsvr_list $realsvr" 246 | let i+=1 247 | fi 248 | done 249 | else 250 | realsvr_list="$candidate_realservers" 251 | fi 252 | 253 | echo "$realsvr_list" 254 | } 255 | 256 | # Update lvs real server list according to real servers's status dynamically 257 | update_lvs() { 258 | local realsvr_list 259 | 260 | #1. get lived real server 261 | realsvr_list=("$(get_lived_realservers)") 262 | 263 | #update lvs setup according ${realsvr_list[@]} 264 | #2. delete invalid real server from LVS virtual service 265 | cur_realservers="$(ipvsadm -L -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} | grep -v "RemoteAddress" |grep "\->"|cut -d" " -f 4|cut -d":" -f 1)" 266 | for realserver in $cur_realservers ; do 267 | echo " ${realsvr_list[@]} " | grep -F " $realserver " >/dev/null 268 | if [ $? -ne 0 ]; then 269 | ipvsadm -d -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} -r $realserver 270 | if [ $? -ne 0 ]; then 271 | ocf_exit_reason "fail to delete real server via ipvsadm" 272 | return $OCF_ERR_GENERIC 273 | fi 274 | fi 275 | done 276 | 277 | #3. add or update the LVS real server within ${realsvr_list[@]} 278 | for realserver in ${realsvr_list[@]}; do 279 | weight="$(get_weight $realserver)" 280 | 281 | echo "$cur_realservers"|grep -x "$realserver" >/dev/null 282 | if [ $? -eq 0 ]; then 283 | current_weight="$(ipvsadm -L -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} |grep " ${realserver}:" |awk '{print $4}')" 284 | if [ $current_weight != $weight ]; then 285 | ipvsadm -e -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} -r $realserver -w $weight 286 | if [ $? -ne 0 ]; then 287 | ocf_exit_reason "fail to update real server via ipvsadm" 288 | return $OCF_ERR_GENERIC 289 | fi 290 | fi 291 | else 292 | ipvsadm -a -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} -r $realserver -w $weight 293 | if [ $? -ne 0 ]; then 294 | ocf_exit_reason "fail to add real server via ipvsadm" 295 | return $OCF_ERR_GENERIC 296 | fi 297 | fi 298 | done 299 | 300 | } 301 | 302 | lvsdr_start() { 303 | #lvsdr_monitor 304 | #if [ $? = $OCF_SUCCESS ]; then 305 | # return $OCF_SUCCESS 306 | #fi 307 | 308 | # Since this is the Director we must be able to forward packets 309 | echo 1 > /proc/sys/net/ipv4/ip_forward 310 | echo 1 > /proc/sys/net/ipv4/vs/expire_nodest_conn 311 | 312 | # Clear all iptables rules. 313 | #iptables -F 314 | # Reset iptables counters. 315 | #iptables -Z 316 | # Clear all ipvsadm rules/services. 317 | #ipvsadm -C 318 | ipvsadm -D -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} 2>/dev/null 319 | 320 | # Add an IP virtual service for VIP ${OCF_RESKEY_vip}:${OCF_RESKEY_port} 321 | ipvsadm -A -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} ${OCF_RESKEY_virtual_service_options} 322 | if [ $? -ne 0 ]; then 323 | ocf_exit_reason "fail to add virtual server via ipvsadm" 324 | return $OCF_ERR_GENERIC 325 | fi 326 | } 327 | 328 | lvsdr_stop() { 329 | # Stop forwarding packets 330 | #echo 0 > /proc/sys/net/ipv4/ip_forward 331 | #echo 0 > /proc/sys/net/ipv4/vs/expire_nodest_conn 332 | 333 | # Reset ipvsadm 334 | ipvsadm -D -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} 335 | #ipvsadm -C 336 | #if [ $? -ne 0 ]; then 337 | # ocf_exit_reason "fail to clear LVS table via ipvsadm" 338 | # return $OCF_ERR_GENERIC 339 | #fi 340 | 341 | return $OCF_SUCCESS 342 | } 343 | 344 | lvsdr_monitor() { 345 | 346 | # check virtual server for ${OCF_RESKEY_vip}:${OCF_RESKEY_port} 347 | ipvsadm -L -t ${OCF_RESKEY_vip}:${OCF_RESKEY_port} >/dev/null 2>&1 348 | if [ $? -ne 0 ]; then 349 | if ocf_is_probe; then 350 | ocf_log info "These is no virtual service definition for this virtual servie" 351 | else 352 | ocf_log err "These is no virtual service definition for this virtual servie" 353 | fi 354 | return $OCF_NOT_RUNNING 355 | fi 356 | 357 | #update Real Servers 358 | update_lvs 359 | if [ $? -ne 0 ]; then 360 | return $OCF_ERR_GENERIC 361 | fi 362 | return $OCF_SUCCESS 363 | } 364 | 365 | lvsdr_validate() { 366 | return $OCF_SUCCESS 367 | } 368 | 369 | ########################################################################## 370 | # If DEBUG_LOG is set, make this resource agent easy to debug: set up the 371 | # debug log and direct all output to it. Otherwise, redirect to /dev/null. 372 | # The log directory must be a directory owned by root, with permissions 0700, 373 | # and the log must be writable and not a symlink. 374 | ########################################################################## 375 | DEBUG_LOG="/tmp/lvsdr.ocf.ra.debug/log" 376 | if [ "${DEBUG_LOG}" -a -w "${DEBUG_LOG}" -a ! -L "${DEBUG_LOG}" ]; then 377 | DEBUG_LOG_DIR="${DEBUG_LOG%/*}" 378 | if [ -d "${DEBUG_LOG_DIR}" ]; then 379 | exec 9>>"$DEBUG_LOG" 380 | exec 2>&9 381 | date >&9 382 | echo "$*" >&9 383 | env | grep OCF_ | sort >&9 384 | set -x 385 | else 386 | exec 9>/dev/null 387 | fi 388 | fi 389 | 390 | # Set debug log output 391 | case "$1" in 392 | start|stop|promote|demote) 393 | write_log="Yes" 394 | ;; 395 | monitor) 396 | if ocf_is_probe; then 397 | write_log="Yes" 398 | fi 399 | ;; 400 | *) 401 | write_log="No" 402 | ;; 403 | esac 404 | 405 | if [ "$write_log" = "Yes" ]; then 406 | ocf_log info "##### ${OCF_RESOURCE_TYPE}(${OCF_RESOURCE_INSTANCE}):${__OCF_ACTION} begin" 407 | fi 408 | 409 | case $__OCF_ACTION in 410 | meta-data) meta_data 411 | exit $OCF_SUCCESS 412 | ;; 413 | start) lvsdr_start;; 414 | stop) lvsdr_stop;; 415 | monitor) lvsdr_monitor;; 416 | validate-all) lvsdr_validate;; 417 | usage|help) lvsdr_usage 418 | exit $OCF_SUCCESS 419 | ;; 420 | *) lvsdr_usage 421 | exit $OCF_ERR_UNIMPLEMENTED 422 | ;; 423 | esac 424 | rc=$? 425 | 426 | if [ "$write_log" = "Yes" ]; then 427 | ocf_log info "##### ${OCF_RESOURCE_TYPE}(${OCF_RESOURCE_INSTANCE}):${__OCF_ACTION} end (rc=$rc)" 428 | fi 429 | 430 | exit $rc 431 | 432 | -------------------------------------------------------------------------------- /ra/lvsdr-realsvr: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # 4 | # Resource Agent script for LVS(DR) Real Server 5 | # 6 | # Authors: Chen Huajun 7 | # 8 | ####################################################################### 9 | # Initialization: 10 | 11 | : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} 12 | . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs 13 | 14 | ####################################################################### 15 | OCF_RESKEY_nic_lo_default="lo:0" 16 | 17 | : ${OCF_RESKEY_nic_lo=${OCF_RESKEY_nic_lo_default}} 18 | 19 | ####################################################################### 20 | meta_data() { 21 | cat < 23 | 24 | 25 | 1.0 26 | 27 | 28 | Resource Agent script for LVS(DR) Real Server. 29 | #Currently only one lvsdr-realsvr resource(primitive or clone) is allowed within a cluster. 30 | 31 | Resource Agent script for LVS(DR) Real Server 32 | 33 | 34 | 35 | 36 | Vitrual IP for virtual service. 37 | 38 | Vitrual IP for virtual service 39 | 40 | 41 | 42 | 43 | 44 | Loopback network device label for realserver,such as "lo:0". 45 | 46 | Loopback network device label for realserver 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | END 61 | } 62 | 63 | ####################################################################### 64 | 65 | lvsdr_realsvr_usage() { 66 | cat < /proc/sys/net/ipv4/conf/lo/arp_ignore 76 | echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce 77 | echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 78 | echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 79 | 80 | ip addr|grep "${OCF_RESKEY_nic_lo}"|grep ${OCF_RESKEY_vip} 81 | if [ $? -eq 0 ]; then 82 | return $OCF_SUCCESS 83 | fi 84 | 85 | ifconfig lo down 86 | ifconfig lo up 87 | if [ $? -ne 0 ]; then 88 | ocf_exit_reason "fail to up loopback device" 89 | return $OCF_ERR_GENERIC 90 | fi 91 | 92 | ifconfig ${OCF_RESKEY_nic_lo} ${OCF_RESKEY_vip} \ 93 | broadcast ${OCF_RESKEY_vip} netmask 255.255.255.255 up 94 | 95 | if [ $? -eq 0 ]; then 96 | return $OCF_SUCCESS 97 | else 98 | ocf_exit_reason "fail to up real server loopback device" 99 | return $OCF_ERR_GENERIC 100 | fi 101 | } 102 | 103 | lvsdr_realsvr_stop() { 104 | # Stop LVS-DR real server loopback device(s). 105 | ifconfig ${OCF_RESKEY_nic_lo} down 106 | #echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore 107 | #echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce 108 | #echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore 109 | #echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce 110 | 111 | return $OCF_SUCCESS 112 | } 113 | 114 | lvsdr_realsvr_monitor() { 115 | 116 | ip addr|grep "${OCF_RESKEY_nic_lo}"|grep ${OCF_RESKEY_vip} 117 | if [ $? -ne 0 ]; then 118 | if ocf_is_probe; then 119 | ocf_log info "These is no ip addr definition for this real server" 120 | else 121 | ocf_log err "These is no ip addr definition for this real server" 122 | fi 123 | return $OCF_NOT_RUNNING 124 | fi 125 | 126 | return $OCF_SUCCESS 127 | } 128 | 129 | lvsdr_realsvr_validate() { 130 | return $OCF_SUCCESS 131 | } 132 | 133 | # Set debug log output 134 | case "$1" in 135 | start|stop|promote|demote) 136 | write_log="Yes" 137 | ;; 138 | monitor) 139 | if ocf_is_probe; then 140 | write_log="Yes" 141 | fi 142 | ;; 143 | *) 144 | write_log="No" 145 | ;; 146 | esac 147 | 148 | if [ "$write_log" = "Yes" ]; then 149 | ocf_log info "##### ${OCF_RESOURCE_TYPE}(${OCF_RESOURCE_INSTANCE}):${__OCF_ACTION} begin" 150 | fi 151 | 152 | case $__OCF_ACTION in 153 | meta-data) meta_data 154 | exit $OCF_SUCCESS 155 | ;; 156 | start) lvsdr_realsvr_start;; 157 | stop) lvsdr_realsvr_stop;; 158 | monitor) lvsdr_realsvr_monitor;; 159 | validate-all) lvsdr_realsvr_validate;; 160 | usage|help) lvsdr_realsvr_usage 161 | exit $OCF_SUCCESS 162 | ;; 163 | *) lvsdr_realsvr_usage 164 | exit $OCF_ERR_UNIMPLEMENTED 165 | ;; 166 | esac 167 | rc=$? 168 | 169 | if [ "$write_log" = "Yes" ]; then 170 | ocf_log info "##### ${OCF_RESOURCE_TYPE}(${OCF_RESOURCE_INSTANCE}):${__OCF_ACTION} end (rc=$rc)" 171 | fi 172 | 173 | exit $rc 174 | 175 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd "$(dirname "$0")" 4 | . ./config.ini 5 | 6 | master=$1 7 | 8 | if [ ! -f config.pcs ]; then 9 | echo "config.pcs does not exists, please run install.sh/gencfg.sh to create it" 10 | fi 11 | 12 | # erase cib 13 | echo "erase cib..." 14 | # switch all node to maintenance mode 15 | for node in `crm_node -l|awk '{print $2}'` 16 | do 17 | pcs property set --node ${node} maintenance=on 18 | done 19 | 20 | cibadmin --erase -f 21 | if [ $? -ne 0 ]; then 22 | echo 'failed to execute "cibadmin --erase -f"' >&2 23 | exit 1 24 | fi 25 | 26 | pcs resource cleanup 27 | 28 | if [ -n "$master" ]; then 29 | echo "set the pgsql_REPL_INFO to $master" 30 | crm_attribute --type crm_config --name pgsql_REPL_INFO -s pgsql_replication -v "$master" 31 | fi 32 | 33 | # setup cib 34 | echo "setup cib..." 35 | sh config.pcs 36 | if [ $? -ne 0 ]; then 37 | echo 'failed to execute "sh config.pcs"' >&2 38 | exit 1 39 | fi 40 | 41 | # initialize data of distributed lock 42 | if [ "$pgsql_enable_distlock" = "true" ]; then 43 | echo "initialize data of distributed lock..." 44 | echo $pgsql_distlock_psql_cmd -vlockname=$pgsql_distlock_lockname -f tools/distlock_init.sql | tr -d '\\' | sh 45 | if [ $? -ne 0 ]; then 46 | echo 'failed to initialize data of distributed lock' >&2 47 | exit 1 48 | fi 49 | fi 50 | -------------------------------------------------------------------------------- /template/config_dual.ini.sample: -------------------------------------------------------------------------------- 1 | pcs_template=dual.pcs.template 2 | OCF_ROOT=/usr/lib/ocf 3 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 4 | pha4pgsql_dir=/opt/pha4pgsql 5 | writer_vip=192.168.0.236 6 | reader_vip=192.168.0.237 7 | node1=node1 8 | node2=node2 9 | vip_nic=eno16777736 10 | vip_cidr_netmask=24 11 | pgsql_pgctl=/usr/bin/pg_ctl 12 | pgsql_psql=/usr/bin/psql 13 | pgsql_pgdata=/data/postgresql/data 14 | pgsql_pgport=5432 15 | pgsql_restore_command="" 16 | pgsql_rep_mode=sync 17 | pgsql_repuser=replication 18 | pgsql_reppassord=replication 19 | pgsql_enable_distlock=true 20 | pgsql_distlock_psql_cmd='/usr/bin/psql \\"host=node3 port=5439 dbname=postgres user=postgres connect_timeout=5\\"' 21 | pgsql_distlock_lockname=pgsql_cls1 22 | -------------------------------------------------------------------------------- /template/config_muti.ini.sample: -------------------------------------------------------------------------------- 1 | pcs_template=muti.pcs.template 2 | OCF_ROOT=/usr/lib/ocf 3 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 4 | pha4pgsql_dir=/opt/pha4pgsql 5 | writer_vip=192.168.0.236 6 | reader_vip=192.168.0.237 7 | node1=node1 8 | node2=node2 9 | node3=node3 10 | othernodes="" 11 | vip_nic=eno16777736 12 | vip_cidr_netmask=24 13 | pgsql_pgctl=/usr/bin/pg_ctl 14 | pgsql_psql=/usr/bin/psql 15 | pgsql_pgdata=/data/postgresql/data 16 | pgsql_pgport=5432 17 | pgsql_restore_command="" 18 | pgsql_rep_mode=sync 19 | pgsql_repuser=replication 20 | pgsql_reppassord=replication -------------------------------------------------------------------------------- /template/config_muti_with_lvs.ini.sample: -------------------------------------------------------------------------------- 1 | pcs_template=muti_with_lvs.pcs.template 2 | OCF_ROOT=/usr/lib/ocf 3 | RESOURCE_LIST="msPostgresql vip-master vip-slave" 4 | pha4pgsql_dir=/opt/pha4pgsql 5 | writer_vip=192.168.0.236 6 | reader_vip=192.168.0.237 7 | node1=node1 8 | node2=node2 9 | node3=node3 10 | othernodes="" 11 | vip_nic=eno16777736 12 | vip_cidr_netmask=24 13 | pgsql_pgctl=/usr/bin/pg_ctl 14 | pgsql_psql=/usr/bin/psql 15 | pgsql_pgdata=/data/postgresql/data 16 | pgsql_pgport=5432 17 | pgsql_restore_command="" 18 | pgsql_rep_mode=sync 19 | pgsql_repuser=replication 20 | pgsql_reppassord=replication -------------------------------------------------------------------------------- /template/dual.pcs.template: -------------------------------------------------------------------------------- 1 | 2 | pcs cluster cib pgsql_cfg 3 | 4 | pcs -f pgsql_cfg property set no-quorum-policy="ignore" 5 | pcs -f pgsql_cfg property set stonith-enabled="false" 6 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 7 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 8 | 9 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 10 | ip="${writer_vip}" \ 11 | nic="${vip_nic}" \ 12 | cidr_netmask="${vip_cidr_netmask}" \ 13 | op start timeout="60s" interval="0s" on-fail="restart" \ 14 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 15 | op stop timeout="60s" interval="0s" on-fail="block" 16 | 17 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 18 | ip="${reader_vip}" \ 19 | nic="${vip_nic}" \ 20 | cidr_netmask="${vip_cidr_netmask}" \ 21 | op start timeout="60s" interval="0s" on-fail="restart" \ 22 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 23 | op stop timeout="60s" interval="0s" on-fail="block" 24 | 25 | pcs -f pgsql_cfg resource create pgsql expgsql \ 26 | pgctl="${pgsql_pgctl}" \ 27 | psql="${pgsql_psql}" \ 28 | pgdata="${pgsql_pgdata}" \ 29 | pgport="${pgsql_pgport}" \ 30 | rep_mode="${pgsql_rep_mode}" \ 31 | node_list="${node1} ${node2}" \ 32 | restore_command="${pgsql_restore_command}" \ 33 | primary_conninfo_opt="user=${pgsql_repuser} password=${pgsql_reppassord} keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 34 | master_ip="${writer_vip}" \ 35 | restart_on_promote="false" \ 36 | enable_distlock="${pgsql_enable_distlock}" \ 37 | distlock_lock_cmd="${pha4pgsql_dir}/tools/distlock '${pgsql_distlock_psql_cmd}' lock distlock:${pgsql_distlock_lockname} @owner 9 12" \ 38 | distlock_unlock_cmd="${pha4pgsql_dir}/tools/distlock '${pgsql_distlock_psql_cmd}' unlock distlock:${pgsql_distlock_lockname} @owner" \ 39 | distlock_lockservice_deadcheck_nodelist="${node1} ${node2}" \ 40 | op start timeout="60s" interval="0s" on-fail="restart" \ 41 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 42 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 43 | op promote timeout="60s" interval="0s" on-fail="restart" \ 44 | op demote timeout="60s" interval="0s" on-fail="stop" \ 45 | op stop timeout="60s" interval="0s" on-fail="block" \ 46 | op notify timeout="60s" interval="0s" 47 | 48 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 49 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 50 | migration-threshold="3" target-role="Master" 51 | 52 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 53 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 54 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 55 | 56 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 57 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 58 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 59 | 60 | pcs cluster cib-push pgsql_cfg 61 | -------------------------------------------------------------------------------- /template/muti.pcs.template: -------------------------------------------------------------------------------- 1 | 2 | pcs cluster cib pgsql_cfg 3 | 4 | pcs -f pgsql_cfg property set no-quorum-policy="stop" 5 | pcs -f pgsql_cfg property set stonith-enabled="false" 6 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 7 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 8 | 9 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 10 | ip="${writer_vip}" \ 11 | nic="${vip_nic}" \ 12 | cidr_netmask="${vip_cidr_netmask}" \ 13 | op start timeout="60s" interval="0s" on-fail="restart" \ 14 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 15 | op stop timeout="60s" interval="0s" on-fail="block" 16 | 17 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 18 | ip="${reader_vip}" \ 19 | nic="${vip_nic}" \ 20 | cidr_netmask="${vip_cidr_netmask}" \ 21 | op start timeout="60s" interval="0s" on-fail="restart" \ 22 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 23 | op stop timeout="60s" interval="0s" on-fail="block" 24 | 25 | pcs -f pgsql_cfg resource create pgsql expgsql \ 26 | pgctl="${pgsql_pgctl}" \ 27 | psql="${pgsql_psql}" \ 28 | pgdata="${pgsql_pgdata}" \ 29 | pgport="${pgsql_pgport}" \ 30 | rep_mode="${pgsql_rep_mode}" \ 31 | node_list="${node1} ${node2} ${node3} ${othernodes}" \ 32 | restore_command="${pgsql_restore_command}" \ 33 | primary_conninfo_opt="user=${pgsql_repuser} password=${pgsql_reppassord} keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 34 | master_ip="${writer_vip}" \ 35 | restart_on_promote="false" \ 36 | enable_distlock="${pgsql_enable_distlock}" \ 37 | distlock_lock_cmd="${pha4pgsql_dir}/tools/distlock '${pgsql_distlock_psql_cmd}' lock distlock:${pgsql_distlock_lockname} @owner 9 12" \ 38 | distlock_unlock_cmd="${pha4pgsql_dir}/tools/distlock '${pgsql_distlock_psql_cmd}' unlock distlock:${pgsql_distlock_lockname} @owner" \ 39 | distlock_lockservice_deadcheck_nodelist="${node1} ${node2} ${node3} ${othernodes}" \ 40 | op start timeout="60s" interval="0s" on-fail="restart" \ 41 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 42 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 43 | op promote timeout="60s" interval="0s" on-fail="restart" \ 44 | op demote timeout="60s" interval="0s" on-fail="stop" \ 45 | op stop timeout="60s" interval="0s" on-fail="block" \ 46 | op notify timeout="60s" interval="0s" 47 | 48 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 49 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 50 | migration-threshold="3" target-role="Master" 51 | 52 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 53 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 54 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 55 | 56 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 57 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 58 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 59 | 60 | pcs -f pgsql_cfg constraint location vip-slave rule id="loc-vip-slave-rule" score=1000 master-pgsql eq "HS:sync" 61 | 62 | pcs cluster cib-push pgsql_cfg 63 | -------------------------------------------------------------------------------- /template/muti_with_lvs.pcs.template: -------------------------------------------------------------------------------- 1 | 2 | pcs cluster cib pgsql_cfg 3 | 4 | pcs -f pgsql_cfg property set no-quorum-policy="stop" 5 | pcs -f pgsql_cfg property set stonith-enabled="false" 6 | pcs -f pgsql_cfg resource defaults resource-stickiness="1" 7 | pcs -f pgsql_cfg resource defaults migration-threshold="10" 8 | 9 | pcs -f pgsql_cfg resource create vip-master IPaddr2 \ 10 | ip="${writer_vip}" \ 11 | nic="${vip_nic}" \ 12 | cidr_netmask="${vip_cidr_netmask}" \ 13 | op start timeout="60s" interval="0s" on-fail="restart" \ 14 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 15 | op stop timeout="60s" interval="0s" on-fail="block" 16 | 17 | pcs -f pgsql_cfg resource create vip-slave IPaddr2 \ 18 | ip="${reader_vip}" \ 19 | nic="${vip_nic}" \ 20 | cidr_netmask="${vip_cidr_netmask}" \ 21 | op start timeout="60s" interval="0s" on-fail="restart" \ 22 | op monitor timeout="60s" interval="10s" on-fail="restart" \ 23 | op stop timeout="60s" interval="0s" on-fail="block" 24 | 25 | pcs -f pgsql_cfg resource create pgsql expgsql \ 26 | pgctl="${pgsql_pgctl}" \ 27 | psql="${pgsql_psql}" \ 28 | pgdata="${pgsql_pgdata}" \ 29 | pghost="${pgsql_pghost}" \ 30 | pgport="${pgsql_pgport}" \ 31 | socketdir="${pgsql_socketdir}" \ 32 | rep_mode="${pgsql_rep_mode}" \ 33 | node_list="${node1} ${node2} ${node3} ${othernodes}" \ 34 | restore_command="${pgsql_restore_command}" \ 35 | primary_conninfo_opt="user=${pgsql_repuser} password=${pgsql_reppassord} keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ 36 | master_ip="${writer_vip}" \ 37 | restart_on_promote="false" \ 38 | op start timeout="60s" interval="0s" on-fail="restart" \ 39 | op monitor timeout="60s" interval="4s" on-fail="restart" \ 40 | op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ 41 | op promote timeout="60s" interval="0s" on-fail="restart" \ 42 | op demote timeout="60s" interval="0s" on-fail="stop" \ 43 | op stop timeout="60s" interval="0s" on-fail="block" \ 44 | op notify timeout="60s" interval="0s" 45 | 46 | pcs -f pgsql_cfg resource master msPostgresql pgsql \ 47 | master-max=1 master-node-max=1 clone-node-max=1 notify=true \ 48 | migration-threshold="3" target-role="Master" 49 | 50 | pcs -f pgsql_cfg constraint colocation add vip-master with Master msPostgresql INFINITY 51 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-master symmetrical=false score=INFINITY 52 | pcs -f pgsql_cfg constraint order demote msPostgresql then stop vip-master symmetrical=false score=0 53 | 54 | pcs -f pgsql_cfg constraint colocation add vip-slave with Slave msPostgresql INFINITY 55 | pcs -f pgsql_cfg constraint order promote msPostgresql then start vip-slave symmetrical=false score=INFINITY 56 | pcs -f pgsql_cfg constraint order stop msPostgresql then stop vip-slave symmetrical=false score=0 57 | 58 | pcs -f pgsql_cfg resource create lvsdr lvsdr \ 59 | vip="${reader_vip}" \ 60 | port="${pgsql_pgport}" \ 61 | realserver_dependent_resource="lvsdr-realsvr-clone" \ 62 | realserver_dependent_attribute_name="pgsql-status" \ 63 | realserver_dependent_attribute_value="HS:sync|HS:potential|HS:async" \ 64 | virtual_service_options="-s rr" \ 65 | op start timeout="60s" interval="0s" on-fail="restart" \ 66 | op monitor timeout="30s" interval="5s" on-fail="restart" \ 67 | op stop timeout="60s" interval="0s" on-fail="block" 68 | 69 | pcs -f pgsql_cfg resource create lvsdr-realsvr lvsdr-realsvr \ 70 | vip="${reader_vip}" \ 71 | nic_lo="lo:0" \ 72 | op start timeout="60s" interval="0s" on-fail="restart" \ 73 | op monitor timeout="30s" interval="60s" on-fail="restart" \ 74 | op stop timeout="60s" interval="0s" on-fail="block" 75 | 76 | pcs -f pgsql_cfg resource clone lvsdr-realsvr clone-node-max=1 notify=false 77 | 78 | pcs -f pgsql_cfg constraint colocation add lvsdr with vip-slave INFINITY 79 | pcs -f pgsql_cfg constraint colocation add lvsdr-realsvr-clone with Slave msPostgresql INFINITY 80 | 81 | pcs -f pgsql_cfg constraint order stop vip-slave then start vip-slave symmetrical=false score=0 82 | pcs -f pgsql_cfg constraint order stop vip-master then start vip-master symmetrical=false score=0 83 | pcs -f pgsql_cfg constraint order start lvsdr-realsvr-clone then start lvsdr symmetrical=false score=0 84 | pcs -f pgsql_cfg constraint order start lvsdr then start vip-slave symmetrical=false score=0 85 | 86 | 87 | pcs cluster cib-push pgsql_cfg 88 | -------------------------------------------------------------------------------- /test/failover_test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | import psycopg2 3 | import time 4 | 5 | url="host=192.168.41.136 dbname=postgres user=postgres" 6 | 7 | conn = psycopg2.connect(url) 8 | conn.autocommit=True 9 | cur = conn.cursor() 10 | 11 | cur.execute("CREATE TABLE IF NOT EXISTS pgsql_ha_test(id integer PRIMARY KEY, num integer);") 12 | cur.execute("truncate pgsql_ha_test;") 13 | cur.execute("insert into pgsql_ha_test values(1,0);") 14 | cur.execute("show transaction_read_only") 15 | 16 | print "Update process had started,please kill the master..." 17 | i=0 18 | num=-1 19 | try: 20 | while True: 21 | i+=1 22 | cur.execute("UPDATE pgsql_ha_test set num = %s where id=1",(i,)) 23 | num=i 24 | if i % 1000 == 0: 25 | print time.time()," current num=",num 26 | 27 | except psycopg2.Error as e: 28 | print time.time()," The master has down, last num=",num 29 | print e.pgerror 30 | conn.close() 31 | 32 | time1= time.time() 33 | connect_success = False 34 | while connect_success == False: 35 | try: 36 | conn = psycopg2.connect(url) 37 | conn.autocommit=True 38 | connect_success=True 39 | except psycopg2.Error as e: 40 | time.sleep(1) 41 | 42 | time2= time.time() 43 | print time.time()," connect success after %f second"%(time2-time1) 44 | 45 | cur = conn.cursor() 46 | cur.execute("select num from pgsql_ha_test where id=1;") 47 | newnum = cur.fetchone()[0] 48 | print time.time()," the new num=",newnum 49 | 50 | if not (newnum==num or newnum==num+1): 51 | print "NG: Data lost!" 52 | conn.close() 53 | exit(1) 54 | 55 | # check if the new master support write 56 | #cur.execute("UPDATE pgsql_ha_test set num = num where id=1") 57 | cur.execute("show transaction_read_only") 58 | read_only = cur.fetchone()[0] 59 | if "off" != read_only: 60 | print "NG: failover to a readony node!" 61 | conn.close() 62 | exit(1) 63 | 64 | cur.close() 65 | conn.close() 66 | print "OK" 67 | 68 | -------------------------------------------------------------------------------- /tools/check_active_slave: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | usage() { 4 | cat < 6 | return 't' if is slave, otherwise return 'f' 7 | 8 | Examples: 9 | 10 | check_active_slave 'psql "port=5433 dbname=postgres user=replication password=replication"' 192.168.0.213 11 | check_active_slave psql \"port=5432 dbname=postgres user=replication password=replication\" -h 192.168.0.213 12 | 13 | EOF 14 | } 15 | 16 | 17 | if [ $# -lt 2 ]; then 18 | usage 19 | exit 1 20 | fi 21 | 22 | psql_cmd="$@" 23 | PSQL="$psql_cmd" 24 | 25 | output=`echo $PSQL -tA -c \""select pg_is_in_recovery()"\" | sh` 26 | 27 | if [ $? -ne 0 ];then 28 | if [ -n "$output" ];then 29 | echo "$output" >&2 30 | fi 31 | exit 1 32 | fi 33 | 34 | echo "$output" -------------------------------------------------------------------------------- /tools/distlock: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | usage() { 4 | cat < trylock|lock|unlock [expired_time [timeout] ] 6 | Display 1 if successed, 0 if not. 7 | 8 | Examples: 9 | 10 | trylock: Obtain a lock if available, and set the expired time to 10 seconds. 11 | distlock 'psql -hlocalhost -Upostgres' trylock distlock:1 node1 10 12 | 13 | lock: Obtain a lock, and set the expired time to 10 seconds. The timeout is 15 seconds. 14 | distlock 'psql -hlocalhost -Upostgres' lock distlock:1 node1 10 15 15 | 16 | unlock: Release a lock. 17 | distlock 'psql -hlocalhost -Upostgres' unlock distlock:1 node1 18 | 19 | EOF 20 | } 21 | 22 | check_with_timeout() 23 | { 24 | func=$1 25 | timeout=$2 26 | 27 | start=`date +%s` 28 | expire=`expr $timeout + $start` 29 | 30 | while true 31 | do 32 | output=`$func 2>&1` 33 | if [ $? -eq 0 ]; then 34 | if [ -n "$output" ];then 35 | echo "Successful to call \"$func\" :$output" 36 | fi 37 | return 0 38 | fi 39 | 40 | if [ `date +%s` -gt $expire ]; then 41 | echo "Tried $timeout seconds and failed to call \"$func\" :$output" 42 | return 1 43 | fi 44 | sleep 1 45 | done 46 | } 47 | 48 | trylock() { 49 | if [ -z "$expired_time" ]; then 50 | usage 51 | exit 1 52 | fi 53 | 54 | output=`echo $PSQL -tA -c \""update $table set owner='$owner',ts=now(),expired_time=interval '$expired_time second' where lockname='$lockname' and (owner = '$owner' or owner = '' or now() > ts + expired_time)"\" | sh` 55 | if [ $? -ne 0 ];then 56 | if [ -n "$output" ];then 57 | echo "$output" >&2 58 | fi 59 | return 1 60 | fi 61 | 62 | echo "$output" | awk '{print $2}' 63 | } 64 | 65 | unlock() { 66 | output=`echo $PSQL -tA -c \""update $table set owner='',ts=now(),expired_time=interval '-1 second' where lockname='$lockname' and owner='$owner'"\" | sh` 67 | if [ $? -ne 0 ];then 68 | if [ -n "$output" ];then 69 | echo "$output" >&2 70 | fi 71 | return 1 72 | fi 73 | 74 | echo "$output" | awk '{print $2}' 75 | } 76 | 77 | lock() { 78 | if [ -z "$expired_time" ]; then 79 | usage 80 | exit 1 81 | fi 82 | 83 | if [ -z "$timeout" ]; then 84 | timeout=2147483647 85 | fi 86 | 87 | start=`date +%s` 88 | expire=`expr $timeout + $start` 89 | 90 | while true 91 | do 92 | output=`trylock` 93 | if [ $? -ne 0 ]; then 94 | if [ -n "$output" ];then 95 | echo "$output" >&2 96 | fi 97 | return 1 98 | fi 99 | 100 | if [ "$output" = "1" ]; then 101 | echo "$output" 102 | return 0 103 | fi 104 | 105 | if [ `date +%s` -gt $expire ]; then 106 | echo "$output" 107 | return 0 108 | fi 109 | sleep 1 110 | done 111 | 112 | } 113 | 114 | if [ $# -lt 4 ]; then 115 | usage 116 | exit 1 117 | fi 118 | 119 | psql_cmd=$1 120 | operate=$2 121 | lockpath=$3 122 | table=${lockpath%%:*} 123 | lockname=${lockpath#*:} 124 | owner=$4 125 | expired_time=$5 126 | timeout="$6" 127 | 128 | PSQL="$psql_cmd" 129 | 130 | if [ -z "$table" -o -z "$lockname" -o -z "$owner" ]; then 131 | usage 132 | exit 1 133 | fi 134 | 135 | case "$operate" in 136 | trylock) trylock 137 | exit $?;; 138 | lock) lock 139 | exit $?;; 140 | unlock) unlock 141 | exit $?;; 142 | *) usage 143 | exit 1;; 144 | esac 145 | -------------------------------------------------------------------------------- /tools/distlock_ddl.sql: -------------------------------------------------------------------------------- 1 | create table if not exists distlock(lockname text primary key,owner text not null,ts timestamptz not null,expired_time interval not null); 2 | 3 | create table if not exists distlock_history(id serial primary key,lockname text not null,owner text not null,ts timestamptz not null,expired_time interval not null); 4 | 5 | CREATE OR REPLACE FUNCTION distlock_log_update() RETURNS trigger AS $$ 6 | BEGIN 7 | 8 | IF TG_OP = 'INSERT' or NEW.owner <> OLD.owner THEN 9 | INSERT INTO distlock_history(lockname, owner, ts, expired_time) values(NEW.lockname, NEW.owner, NEW.ts, NEW.expired_time); 10 | END IF; 11 | RETURN NEW; 12 | END; 13 | $$ LANGUAGE plpgsql; 14 | 15 | 16 | DROP TRIGGER IF EXISTS distlock_log_update ON distlock; 17 | 18 | CREATE TRIGGER distlock_log_update AFTER INSERT OR UPDATE ON distlock 19 | FOR EACH ROW EXECUTE PROCEDURE distlock_log_update(); 20 | -------------------------------------------------------------------------------- /tools/distlock_init.sql: -------------------------------------------------------------------------------- 1 | 2 | delete from distlock where lockname = :'lockname'; 3 | 4 | insert into distlock(lockname, owner, ts, expired_time) values(:'lockname', '', now(), interval '-1 second'); -------------------------------------------------------------------------------- /tools/get_active_slaves: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | usage() { 4 | cat < 6 | Display the list of active slaves, or empty if none or failed. 7 | psql_cmd must connect to the master of pgsql cluster 8 | 9 | Examples: 10 | 11 | get_active_slaves psql \"host=192.168.0.236 port=5432 dbname=postgres user=replication password=replication\" 12 | 13 | EOF 14 | } 15 | 16 | 17 | if [ $# -lt 1 ]; then 18 | usage 19 | exit 1 20 | fi 21 | 22 | psql_cmd="$@" 23 | PSQL="$psql_cmd" 24 | 25 | output=`echo $PSQL -tA -c \""select application_name from pg_stat_replication where sync_state in ('sync','potential','async') and not application_name is null"\" | sh` 26 | if [ $? -ne 0 ];then 27 | if [ -n "$output" ];then 28 | echo "$output" >&2 29 | fi 30 | exit 1 31 | fi 32 | 33 | echo "$output" | tr '\n' ' ' --------------------------------------------------------------------------------