├── LICENSE ├── README.md ├── __init__.py ├── api_handler.py ├── doc ├── SQLAdvisor.md ├── images │ ├── 13.png │ ├── 14.png │ ├── 1523172269065.png │ ├── 1523174081610.png │ ├── 1523174420906.png │ ├── 1523174766848.png │ ├── 1523175788884.png │ ├── 1523185047355.png │ ├── 1532581840639.png │ ├── 1532583500578.png │ ├── 1532583669092.png │ ├── 1532583812725.png │ ├── 1532584154244.png │ ├── 1532584401394.png │ ├── 1532585031943.png │ ├── 1532585074866.png │ ├── 1532585123544.png │ ├── 1532586216885.png │ ├── 1532587306400.png │ ├── 1532587410451.png │ ├── 1532587504312.png │ ├── 1532591634885.png │ ├── 1532592565485.png │ ├── 1532593048696.png │ ├── 1532593125690.png │ ├── 1532594104486.png │ ├── 1532594149567.png │ ├── 1532594209026.png │ ├── 1532594288107.png │ ├── architecture.png │ └── sonar_task.png ├── inception.md └── sonarqube.md ├── exec_inception.py ├── exec_sonar.py ├── exec_sqladvisor.py ├── requirements.txt └── settings.py /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## SQL 审核 审计 执行 备份 索引优化 ，代码质量管理等。 2 | 3 | [更新日志](https://github.com/ss1917/do_mg/releases) 4 | 5 | [在线访问](http://demo.opendevops.cn/) 6 | 7 | ### 代码质量管理 8 | #### 为什么要使用： 9 | ``` 10 | 糟糕的复杂度分布 11 | 文件、类、方法等，如果复杂度过高将难以改变，这会使得开发人员难以理解它们， 12 | 且如果没有自动化的单元测试，对于程序中的任何组件的改变都将可能导致需要全面的回归测试 13 | 重复 14 | 显然程序中包含大量复制粘贴的代码是质量低下的，sonar可以展示源码中重复严重的地方 15 | 缺乏单元测试 16 | sonar可以很方便地统计并展示单元测试覆盖率 17 | 没有代码标准 18 | sonar可以通过PMD,CheckStyle,Findbugs等等代码规则检测工具规范代码编写 19 | 没有足够的或者过多的注释 20 | 没有注释将使代码可读性变差，特别是当不可避免地出现人员变动时，程序的可读性将大幅下降 21 | 而过多的注释又会使得开发人员将精力过多地花费在阅读注释上，亦违背初衷 22 | 潜在的bug 23 | sonar可以通过PMD,CheckStyle,Findbugs等等代码规则检测工具检测出潜在的bug 24 | 糟糕的设计 25 | 通过sonar可以找出循环，展示包与包、类与类之间的相互依赖关系 26 | 可以检测自定义的架构规则，可以检测单个任务规则的应用情况 27 | 检测耦合 28 | ``` 29 | #### 简介 30 | **SonarQube 是一个开源的代码分析平台, 用来持续分析和评测项目源代码的质量。 通过SonarQube我们可以检测出项目中重复代码， 潜在bug， 代码风格问题，缺乏单元测试等问题， 并通过一个web ui展示出来** 31 | 32 | #### 环境部署 33 | 34 | [SonarQube 介绍 部署 配置](https://github.com/opendevops-cn/codo-check/tree/master/doc/sonarqube.md) 35 | 36 | #### 脚本部署 37 | 38 | - 脚本下载 39 | ``` 40 | mkdir -p /opt/ops_scripts/ && cd /opt/ops_scripts/ && git clone https://github.com/opendevops-cn/codo-check.git 41 | ``` 42 | - 配置 settings 43 | - 安装依赖 pip3 install -r requirements.txt 44 | 45 | **约束** 46 | - 当前主机可以执行 /usr/local/sonar-scanner/bin/sonar-scanner 命令。如何部署请看部署文档 47 | - 当前注意可以 拉取 相关git仓库的代码。可以使用git的 deploy_keys 48 | 49 | #### 使用说明 50 | **待完善** 51 | 52 | ### SQL 审核 审计 执行 备份 53 | #### Inception简介： 54 | **Inception是集审核、执行、回滚于一体的一个自动化运维系统，它是根据MySQL代码修改过来的，用它可以很明确的，详细的，准确的审核MySQL的SQL语句，它的工作模式和MySQL完全相同，可以直接使用MySQL客户端来连接，但不需要验证权限，它相对应用程序（上层审核流程系统等）而言，是一个服务器，在连接时需要指定服务器地址及Inception服务器的端口即可，而它相对要审核或执行的语句所对应的线上MySQL服务器来说，是一个客户端，它在内部需要实时的连接数据库服务器来获取所需要的信息，或者直接在在线上执行相应的语句及获取binlog等，Inception就是一个中间性质的服务** 55 | **本项目使用inception 结合CMDB，把SQL审核接入运维自动化平台，方便直接在运维平台使用，拥有审计，审核，执行，备份功能，并且可以结合代码发布使用** 56 | 57 | [Inception 介绍 部署 配置](https://github.com/opendevops-cn/codo-check/tree/master/doc/inception.md) 58 | #### 部署 配置 59 | - 部署参考部署文档 60 | - 因为作者已经删除此项目，所以需要此项功能请自行去github找相关代码 61 | - inception 配置参考 上述文档， 默认IP为127.0.0.1 端口6669， 密码为空 62 | #### 脚本部署 63 | ``` 64 | # 本脚本根据应用名称从CMDB查找当前应用的数据库信息，根据执行模板里面配置的强制主机上执行（强制主机须能连上要审核执行的数据库），数据库语句放在git仓库，通过http地址获取 65 | # 本次审核执行会在当前应用的所有主库执行，如果使用应用本身的分库分表，可能会出现错误,请自行根据报错信息处理问题 66 | ``` 67 | 68 | ### SQL索引优化 69 | #### 为什么要使用： 70 | ``` 71 | 例行 SQL 优化，不仅可以提升程序性能，还能够降低线上故障的概率。 72 | 目前常用的 SQL 优化方式包括但不限于：业务层优化、SQL逻辑优化、索引优化等。其中索引优化通常通过调整索引或新增索引从而达到 SQL 优化的目的。索引优化往往可以在短时间内产生非常巨大的效果。如果能够将索引优化转化成工具化、标准化的流程，减少人工介入的工作量，无疑会大大提高的工作效率。 73 | ``` 74 | 75 | #### SQLAdvisor简介 76 | **SQLAdvisor 是由美团点评公司北京DBA团队开发维护的 SQL 优化工具：输入SQL，输出索引优化建议。 它基于 MySQL 原生词法解析，再结合 SQL 中的 where 条件以及字段选择度、聚合条件、多表 Join 关系等最终输出最优的索引优化建议。目前 SQLAdvisor 在公司内部大量使用，较为成熟、稳定。** 77 | 78 | #### 环境部署 79 | [SQLAdvisor 部署 配置](https://github.com/opendevops-cn/codo-check/tree/master/doc/SQLAdvisor.md) 80 | 81 | #### 脚本部署 82 | 83 | - 脚本下载 如果已经下载，则可以省略当前步骤 84 | ``` 85 | mkdir -p /opt/ops_scripts/ && cd /opt/ops_scripts/ && git clone https://github.com/opendevops-cn/codo-check.git 86 | ``` 87 | - 配置 settings 88 | - 安装依赖 pip3 install -r requirements.txt 89 | 90 | **约束** 91 | - 当前主机可以执行 /usr/local/sonar-scanner/bin/sonar-scanner 命令。如何部署请看部署文档 92 | - 当前注意可以 拉取 相关git仓库的代码。可以使用git的 deploy_keys 93 | 94 | ## License 95 | 96 | Everything is [GPL v3.0](https://www.gnu.org/licenses/gpl-3.0.html). -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Contact : 191715030@qq.com 5 | Author : shenshuo 6 | Date : 2018/12/26 7 | Desc : 8 | """ -------------------------------------------------------------------------------- /api_handler.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Contact : 191715030@qq.com 5 | Author : shenshuo 6 | Date : 2018/12/25 7 | Desc : 8 | """ 9 | 10 | import pyotp 11 | import requests 12 | import json 13 | from settings import api_settings 14 | 15 | 16 | 17 | class API(object): 18 | def __init__(self): 19 | self.url = api_settings['api_gw'] 20 | self.user = api_settings['api_user'] 21 | self.pwd = api_settings['api_pwd'] 22 | self.key = api_settings.get('api_key') 23 | self.login_api = api_settings.get('login_api') 24 | self.publish_info_api = api_settings.get('publish_info_api') 25 | self.get_key_api = api_settings.get('get_key_api') 26 | self.mail_api = api_settings.get('mail_api') 27 | 28 | @property 29 | def get_mfa(self): 30 | t = pyotp.TOTP(self.key) 31 | return t.now() 32 | 33 | def login(self): 34 | try: 35 | headers = {"Content-Type": "application/json"} 36 | params = {"username": self.user, "password": self.pwd, "dynamic": self.get_mfa} 37 | result = requests.post(self.login_api, data=json.dumps(params), headers=headers) 38 | 39 | ret = json.loads(result.text) 40 | if ret['code'] == 0: 41 | return ret['auth_key'] 42 | else: 43 | print(ret) 44 | print(ret['msg']) 45 | exit(1) 46 | except Exception as e: 47 | print('[Error:] 用户:{} 接口登陆失败，错误信息：{}'.format(self.user, e)) 48 | 49 | def get_publish_name_info(self, publish_name): 50 | token = self.login() 51 | try: 52 | params = {'key': 'publish_name', 'value': publish_name} 53 | res = requests.get(self.publish_info_api, params=params, cookies=dict(auth_key=token)) 54 | ret = json.loads(res.content) 55 | if ret['code'] == 0: return ret['data'] 56 | except Exception as e: 57 | print('[Error:] Publish发布接口连接失败，错误信息：{}'.format(e)) 58 | exit(-2) 59 | 60 | def get_publish_all_info(self): 61 | '''获取发布配置所有信息''' 62 | try: 63 | token = self.login() 64 | except Exception as e: 65 | print(e) 66 | token = self.login() 67 | 68 | res = requests.get(self.publish_info_api, cookies=dict(auth_key=token)) 69 | ret = json.loads(res.content) 70 | if ret['code'] == 0: return ret['data'] 71 | 72 | def get_api_info(self, api_url): 73 | '''获取接口所有信息''' 74 | try: 75 | token = self.login() 76 | except Exception as e: 77 | print(e) 78 | token = self.login() 79 | 80 | res = requests.get(api_url, cookies=dict(auth_key=token)) 81 | ret = json.loads(res.content) 82 | return ret['data'] 83 | 84 | def send_mail_for_api(self, mail_list, mail_subject, mail_content): 85 | """发送邮件""" 86 | try: 87 | token = self.login() 88 | except Exception as e: 89 | print(e) 90 | token = self.login() 91 | 92 | req1 = requests.get(self.get_key_api, cookies=dict(auth_key=token)) 93 | if req1.status_code != 200: 94 | raise SystemExit(req1.status_code) 95 | 96 | ret1 = json.loads(req1.text) 97 | if ret1['code'] == 0: 98 | csrf_key = ret1['csrf_key'] 99 | else: 100 | raise SystemExit(ret1['code']) 101 | 102 | body = json.dumps({"to_list": mail_list, "subject": mail_subject, "content": mail_content, "subtype": "plain"}) 103 | res = requests.post(self.mail_api, data=body, cookies=dict(auth_key=token, csrf_key=csrf_key)) 104 | if res.status_code != 200: 105 | raise SystemExit(res.status_code) 106 | else: 107 | return json.loads(res.text)['msg'] 108 | -------------------------------------------------------------------------------- /doc/SQLAdvisor.md: -------------------------------------------------------------------------------- 1 | # SQLAdvisor说明 2 | 3 | ## 简介 4 | 5 | SQLAdvisor是一个分析SQL给出索引优化建议的工具。它基于MySQL原生态词法解析，结合分析SQL中的where条件、聚合条件、多表Join关系 给出索引优化建议。 6 | 7 | 主要功能：输出SQL索引优化建议 8 | 9 | 详细介绍官方文档 : https://github.com/Meituan-Dianping/SQLAdvisor/blob/master/doc/THEORY_PRACTICES.md 10 | 11 | ## 安装依赖 12 | centos版安装 13 | 14 | ``` 15 | #安装percona56 yum源，后续集成sqladvisor需要用到Percona客户端，优先安装避免与mysql客户端冲突。 16 | yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm 17 | 18 | yum -y install make cmake ncurses-devel gcc gcc-c++ openssl-devel libaio-devel libffi-devel glib2 glib2-devel m4 Percona-Server-shared-56 Percona-Server-client-56 19 | ``` 20 | ## 安装使用 21 | 22 | ``` 23 | #编译依赖项sqlparser 24 | git clone https://github.com/Meituan-Dianping/SQLAdvisor.git 25 | cd SQLAdvisor 26 | 1.cmake -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=debug -DCMAKE_INSTALL_PREFIX=/usr/local/sqlparser ./ 27 | 2.make && make install 28 | 29 | #安装SQLAdvisor源码 30 | cd SQLAdvisor/sqladvisor/ 31 | cmake -DCMAKE_BUILD_TYPE=debug ./ 32 | make 33 | cp sqladvisor /usr/bin/ 34 | 35 | #使用帮助 36 | sqladvisor --help 37 | 38 | #举个栗子 39 | sqladvisor -u root -p 123456 -P 3306 -h 10.0.0.24 -q "select permission_id from base_server_user_group_permission_relation;" -d work_permission_0 -v 1 40 | ``` 41 | 42 | ![1.7](images/1523185047355.png) 43 | 44 | * 编译完成后可删除整个sqladvisor目录，仅保留可执行文件sqladvisor。 45 | -------------------------------------------------------------------------------- /doc/images/13.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/13.png -------------------------------------------------------------------------------- /doc/images/14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/14.png -------------------------------------------------------------------------------- /doc/images/1523172269065.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523172269065.png -------------------------------------------------------------------------------- /doc/images/1523174081610.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523174081610.png -------------------------------------------------------------------------------- /doc/images/1523174420906.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523174420906.png -------------------------------------------------------------------------------- /doc/images/1523174766848.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523174766848.png -------------------------------------------------------------------------------- /doc/images/1523175788884.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523175788884.png -------------------------------------------------------------------------------- /doc/images/1523185047355.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1523185047355.png -------------------------------------------------------------------------------- /doc/images/1532581840639.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532581840639.png -------------------------------------------------------------------------------- /doc/images/1532583500578.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532583500578.png -------------------------------------------------------------------------------- /doc/images/1532583669092.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532583669092.png -------------------------------------------------------------------------------- /doc/images/1532583812725.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532583812725.png -------------------------------------------------------------------------------- /doc/images/1532584154244.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532584154244.png -------------------------------------------------------------------------------- /doc/images/1532584401394.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532584401394.png -------------------------------------------------------------------------------- /doc/images/1532585031943.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532585031943.png -------------------------------------------------------------------------------- /doc/images/1532585074866.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532585074866.png -------------------------------------------------------------------------------- /doc/images/1532585123544.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532585123544.png -------------------------------------------------------------------------------- /doc/images/1532586216885.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532586216885.png -------------------------------------------------------------------------------- /doc/images/1532587306400.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532587306400.png -------------------------------------------------------------------------------- /doc/images/1532587410451.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532587410451.png -------------------------------------------------------------------------------- /doc/images/1532587504312.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532587504312.png -------------------------------------------------------------------------------- /doc/images/1532591634885.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532591634885.png -------------------------------------------------------------------------------- /doc/images/1532592565485.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532592565485.png -------------------------------------------------------------------------------- /doc/images/1532593048696.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532593048696.png -------------------------------------------------------------------------------- /doc/images/1532593125690.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532593125690.png -------------------------------------------------------------------------------- /doc/images/1532594104486.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532594104486.png -------------------------------------------------------------------------------- /doc/images/1532594149567.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532594149567.png -------------------------------------------------------------------------------- /doc/images/1532594209026.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532594209026.png -------------------------------------------------------------------------------- /doc/images/1532594288107.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/1532594288107.png -------------------------------------------------------------------------------- /doc/images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/architecture.png -------------------------------------------------------------------------------- /doc/images/sonar_task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/opendevops-cn/codo-check/55d4bc7154c7073da525868e2ad3bf4772dc6d97/doc/images/sonar_task.png -------------------------------------------------------------------------------- /doc/inception.md: -------------------------------------------------------------------------------- 1 | # Inception总述及体系结构 2 | 3 | ## 总述 4 | 5 | Inception是集审核、执行、回滚于一体的一个自动化运维系统，它是根据MySQL代码修改过来的，用它可以很明确的，详细的，准确的审核MySQL的SQL语句，它的工作模式和MySQL完全相同，可以直接使用MySQL客户端来连接，但不需要验证权限，它相对应用程序（上层审核流程系统等）而言，是一个服务器，在连接时需要指定服务器地址及Inception服务器的端口即可，而它相对要审核或执行的语句所对应的线上MySQL服务器来说，是一个客户端，它在内部需要实时的连接数据库服务器来获取所需要的信息，或者直接在在线上执行相应的语句及获取binlog等，Inception就是一个中间性质的服务。图1.1所示为Inception的架构。 6 | 7 | ![1.1](images/architecture.png)Inception提供的功能很丰富，首先，它可以对提交的所有语句的语法分析，如果语法有问题，都会将相应的错误信息返回给审核者。 还提供语义分析，当一个表，库，列等信息不正确或者不符合规范的时候报错，或者使用了一个不存在的对象时报错等等。 还提供了很多针对SQL规范性约束的功能，这些DBA都是可以通过系统参数来配置的。 更高级的功能是，可以辅助DBA分析一条查询语句的性能，如果没有使用索引或者某些原因导致查询很慢，都可以检查。 8 | 9 | 还提供SQL语句的执行功能，可执行的语句类型包括常用的DML及DDL语句及truncate table等操作。 Inception 在执行 DML 时还提供生成回滚语句的功能，对应的操作记录及回滚语句会被存储在备份机器上面，备份机器通过配置Inception参数来指定。 10 | 11 | 12 | 13 | 14 | 15 | ## 安装 16 | 17 | centos版安装inception 18 | 19 | ``` 20 | #安装percona56 yum源，后续集成sqladvisor需要用到Percona客户端，优先安装避免与mysql客户端冲突。 21 | yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm 22 | 23 | yum -y install make cmake ncurses-devel gcc gcc-c++ openssl-devel libaio-devel libffi-devel glib2 glib2-devel m4 Percona-Server-shared-56 Percona-Server-client-56 24 | 25 | wget -c http://ftp.gnu.org/gnu/bison/bison-2.5.1.tar.gz 26 | tar zxvf bison-2.5.1.tar.gz && cd bison-2.5.1 && ./configure && make && make install 27 | 28 | cd /usr/local && wget -c https://github.com/mysql-inception/inception/archive/master.zip 29 | unzip master.zip 30 | cd inception-master/ 31 | sh inception_build.sh builddir linux 32 | mv builddir/ /usr/local/inception-master_2.1.50 33 | 34 | ``` 35 | 36 | 编译完成后，需要一个配置文件( inc.cnf ),此处只做示例用，详细配置参考官方说明：http://mysql-inception.github.io/inception-document/variables/ 37 | 38 | ``` 39 | [inception] 40 | general_log=1 41 | general_log_file=inception.log 42 | port=6669 43 | socket=/自己目录，请自行修改/inc.socket 44 | character-set-client-handshake=0 45 | character-set-server=utf8 46 | inception_remote_system_password=root 47 | inception_remote_system_user=wzf1 48 | inception_remote_backup_port=3306 49 | inception_remote_backup_host=127.0.0.1 50 | inception_support_charset=utf8mb4 51 | inception_enable_nullable=0 52 | inception_check_primary_key=1 53 | inception_check_column_comment=1 54 | inception_check_table_comment=1 55 | inception_osc_min_table_size=1 56 | inception_osc_bin_dir=/data/temp 57 | inception_osc_chunk_time=0.1 58 | inception_enable_blob_type=1 59 | inception_check_column_default_value=1 60 | ``` 61 | 62 | 63 | 64 | ## 启动，测试使用 65 | 66 | ``` 67 | #启动inception服务 68 | nohup /usr/local/inception-master/builddir/mysql/bin/Inception --defaults-file=/etc/inc.cnf & 69 | 70 | #通过mysql(percona)客户端连接 71 | mysql -uroot -h127.0.0.1 -P6669 72 | 73 | #测试命令，输出所有变量，如图1.2所示表示启动成功。 74 | inception get variables; 75 | ``` 76 | 77 | ![1.2](images/1523172269065.png) 78 | 79 | 80 | ## inc.cnf配置文件参数简要说明 81 | 82 | ``` 83 | [inception] 84 | general_log=1 85 | general_log_file=/var/log/inception.log 86 | port=6669 87 | socket=/tmp/inc.socket 88 | character-set-client-handshake=0 89 | character-set-server=utf8 90 | 91 | #备份相关 92 | #需要开启binlog 93 | inception_remote_system_password=123456 94 | inception_remote_system_user=user 95 | inception_remote_backup_port=3306 96 | inception_remote_backup_host=10.0.0.24 97 | 98 | #在DML语句中没有WHERE条件时，是不是要报错 99 | inception_check_dml_where=1 100 | #在DML语句中使用了LIMIT时，是不是要报错 101 | inception_check_dml_limit=1 102 | #在DML语句中使用了Order By时，是不是要报错 103 | inception_check_dml_orderby 104 | #Select*时是不是要报错 105 | inception_enable_select_star=1 106 | #order by rand时是不是报错 107 | inception_enable_orderby_rand=1 108 | #创建或者新增列时如果列为NULL，是不是报错 109 | inception_enable_nullable=1 110 | #是不是支持外键 111 | inception_enable_foreign_key=1 112 | #一个索引中，列的最大个数，超过这个数目则报错(1-64) 113 | inception_max_key_parts=5 114 | #在一个修改语句中，预计影响的最大行数，超过这个数就报错(1-max) 115 | inception_max_update_rows=10000 116 | #一个表中，最大的索引数目，超过这个数则报错(1-1024) 117 | inception_max_keys=16 118 | #建表指定的存储引擎不为Innodb，不报错 119 | inception_enable_not_innodb=0 120 | #表示在建表或者建库时支持的字符集，如果需要多个，则用逗号分隔，影响的范围是建表、设置会话字符集、修改表字符集属性等 121 | inception_support_charset=utf8mb4 122 | #建表时，表没有注释时报错 123 | inception_check_table_comment=1 124 | #建表时，列没有注释时报错 125 | inception_check_column_comment=1 126 | #建表时，如果没有主键，则报错 127 | inception_check_primary_key=1 128 | #是不是支持分区表 129 | inception_enable_partition_table=0 130 | #是不是支持enum,set,bit数据类型 131 | inception_enable_enum_set_bit=0 132 | #是不是要检查索引名字前缀为"idx_"，检查唯一索引前缀是不是"uniq_" 133 | inception_check_index_prefix=1 134 | #自增列是不是要为无符号型 135 | inception_enable_autoincrement_unsigned=1 136 | #当char类型的长度大于这个值时，就提示将其转换为VARCHAR(1-max) 137 | inception_max_char_length=16 138 | #当建表时自增列的值指定的不为1，则报错 139 | inception_check_autoincrement_init_value=1 140 | #当建表时自增列的类型不为int或者bigint时报错 141 | inception_check_autoincrement_datatype=1 142 | #建表时，如果没有为timestamp类型指定默认值，则报错 143 | inception_check_timestamp_default=0 144 | #允许列自己设置字符集 145 | inception_enable_column_charset=0 146 | #建表时，如果指定的自增列的名字不为ID，则报错，说明是有意义的，给提示 147 | inception_check_autoincrement_name=1 148 | #在多个改同一个表的语句出现时，报错，提示合成一个 149 | inception_merge_alter_table=1 150 | #检查在建表、修改列、新增列时，新的列属性是不是要有默认值 151 | inception_check_column_default_value=1 152 | #检查是不是支持BLOB字段，包括建表、修改列、新增列操作 153 | inception_enable_blob_type=1 154 | #检查在SQL语句中，是不是有标识符被写成MySQL的关键字，默认值为报警。 155 | inception_enable_identifer_keyword=1 156 | #这个参数的作用是为了匹配Python客户端每次自动设置auto_commit=0的，如果取消则会报错，针对Inception本身没有实际意义 157 | #auto_commit=0 158 | #这个参数实际上就是MySQL数据库原来的参数，因为Incpetion没有权限验证过程，那么为了实现更安全的访问，可以给Inception服务器的这个参数设置某台机器（Inception上层的应用程序）不地址，这样 159 | #其它非法程序是不可访问的，那么再加上Inception执行的选项中的用户名密码，对MySQL就更加安全 160 | bind_address=127.0.0.1 161 | #inception_user 162 | #inception_password 163 | #inception_enable_sql_statistic 164 | #inception_read_only 165 | #打开与关闭Inception对SQL语句中各种名字的检查，如果设置为ON，则如果发现名字中存在除数字字母下划线之外的字符时，报Identifier "invalidname" is invalid, valid options: [a-z,A-Z,0-9,_]. 166 | inception_check_identifier=1 167 | 168 | #inception_osc_min_table_size=1 169 | #inception_osc_bin_dir=/data/temp 170 | #inception_osc_chunk_time=0.1 171 | ``` 172 | 173 | 174 | 175 | # Inception使用方法 176 | 177 | 1. Inception规定，在语句的最开始位置，要加上inception_magic_start;语句，在执行语句块的最后加上inception_magic_commit;语句，这2个语句在 Inception 中都是合法的、具有标记性质的可被正确解析的 SQL 语句。被包围起来的所有需要审核或者执行的语句都必须要在每条之后加上分号，其实就是批量执行SQL语句。（包括use database语句之后也要加分号，这点与 MySQL 客户端不同），不然存在语法错误 178 | 2. 目前执行只支持通过C/C++接口、Python接口来对Inception访问，这一段必须是一次性的通过执行接口提交给Inception，那么在处理完成之后，Inception会返回一个结果集，来告诉我们这些语句中存在什么错误，或者是完全正常等等。 179 | 180 | 下面是一个python脚本访问inception的简单例子，指定连接测试的mysql服务器为10.0.0.24： 181 | 182 | ``` 183 | cat >inc-mysql.py <inc-mysql.py < Dockerfile < 49 | # sub 2048R/06855C1D 2015-05-25 50 | && gpg --keyserver ha.pool.sks-keyservers.net --recv-keys F1182E81C792928921DBCAB4CFCA4A29D26468DE \ 51 | 52 | && mkdir /opt \ 53 | && cd /opt \ 54 | && wget -O sonarqube.zip --no-verbose https://sonarsource.bintray.com/Distribution/sonarqube/sonarqube-$SONAR_VERSION.zip \ 55 | && wget -O sonarqube.zip.asc --no-verbose https://sonarsource.bintray.com/Distribution/sonarqube/sonarqube-$SONAR_VERSION.zip.asc \ 56 | && gpg --batch --verify sonarqube.zip.asc sonarqube.zip \ 57 | && unzip sonarqube.zip \ 58 | && mv sonarqube-$SONAR_VERSION sonarqube \ 59 | && chown -R sonarqube:sonarqube sonarqube \ 60 | && rm sonarqube.zip* \ 61 | && rm -rf $SONARQUBE_HOME/bin/* 62 | 63 | VOLUME "$SONARQUBE_HOME/data" 64 | 65 | WORKDIR $SONARQUBE_HOME 66 | COPY run.sh $SONARQUBE_HOME/bin/ 67 | ENTRYPOINT ["./bin/run.sh"] 68 | EOF 69 | 70 | cat > run.sh </data/sonarqube/docker-compose.yml <sonar-project.properties< 此任务放在最后一个并且检查是否有重大BUG 都不会影响发布，如果需要根据条件阻断发布，在任务编排中放在 前列，并且抛出错误，则任务进程会报错等待处理。 280 | 281 | ## gitlab-plugin 配合 gitlab-ci 完成每次 commit 代码检测 (这样对gitlab资源消耗比较严重，且配置复杂，此处仅供参考) 282 | 283 | ### Sonar GitLab Plugin 安装 284 | 285 | sonar应用市场搜索gitlab，安装. 286 | 287 | ![1532591634885](images/1532591634885.png) 288 | 289 | ### gitlab-runner安装 290 | 291 | 测试环境gitlab ci为10.8.4版本，需下载匹配版本的gitlab-runner 292 | 293 | ![1532592565485](images/1532592565485.png) 294 | 295 | **安装** 296 | 297 | ``` 298 | sudo wget -O /usr/bin/gitlab-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-linux-amd64 299 | chmod +x /usr/bin/gitlab-runner 300 | sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner 301 | 302 | #向gitlab注册runner 303 | gitlab-runner register 304 | ``` 305 | 306 | ```shell 307 | Running in system-mode. 308 | 309 | Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): 310 | http://gitlab.example.com/ 311 | Please enter the gitlab-ci token for this runner: 312 | deap7yTbt-J35euZzek9 313 | Please enter the gitlab-ci description for this runner: 314 | [localhost.localdomain]: my-test-runner 315 | Please enter the gitlab-ci tags for this runner (comma separated): 316 | my-test 317 | Whether to run untagged builds [true/false]: 318 | [false]: true 319 | Whether to lock Runner to current project [true/false]: 320 | [false]: true 321 | Registering runner... succeeded runner=deap7yTb 322 | Please enter the executor: ssh, docker-ssh+machine, shell, virtualbox, docker+machine, kubernetes, docker, docker-ssh, parallels: 323 | shell 324 | Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 325 | ``` 326 | 327 | 注册时需要的token来自gitlab-Admin Area-Runners。注册成功后启动`gitlab-runner start` 328 | 329 | 可以看到gitlab runners界面中出现了runner 330 | 331 | ![1532593125690](images/1532593125690.png) 332 | 333 | ### runner配合sonar-scanner实现代码commit后自动扫描 334 | 335 | 编写个简单脚本，完成2件事情，拉取指定项目，进入项目目录执行sonar-scanner扫描代码 336 | 337 | ```shell 338 | #!/bin/bash 339 | if [ $# -ne 2 ] 340 | then 341 | echo "arguments error!" 342 | exit 1 343 | else 344 | deploy_path="/data/$1/$2" 345 | if [ ! -d "$deploy_path" ] 346 | then 347 | project_path="git@gitlab.example.com:"$1/$2".git" 348 | git clone $project_path $deploy_path 349 | else 350 | cd $deploy_path 351 | git pull 352 | fi 353 | fi 354 | 355 | cd $deploy_path;sonar-scanner 356 | if [ $? -eq 0 ]; then 357 | echo "sonarqube code-analyze over." 358 | fi 359 | ``` 360 | 361 | 编写`.gitlab-ci.yml`配置文件，放入指定项目仓库根目录 362 | 363 | ``` 364 | cat >gitlab-ci.yml <= 5: 14 | self.client_flag |= CLIENT.MULTI_RESULTS 15 | """ 16 | 17 | import fire 18 | import base64 19 | import subprocess 20 | import warnings 21 | import socket 22 | import re 23 | from api_handler import API 24 | 25 | warnings.filterwarnings('ignore') 26 | import pymysql 27 | 28 | 29 | 30 | def _is_ip(value): 31 | """检测是否是IP""" 32 | ipv4_re = re.compile(r'^(25[0-5]|2[0-4]\d|[0-1]?\d?\d)(\.(25[0-5]|2[0-4]\d|[0-1]?\d?\d)){3}$') 33 | if ipv4_re.match(value): return True 34 | 35 | 36 | def conver_url2ip(db_host): 37 | '''RDS URL 解析成IP,不然host过长,导致inception无法备份''' 38 | if not _is_ip(db_host): 39 | old_host = db_host 40 | db_host = socket.gethostbyname(db_host) 41 | print('[HOST URL] %s ===> [IP]%s' % (old_host, db_host)) 42 | return db_host 43 | 44 | 45 | def get_conf(db_info): 46 | """获取SQL配置生产配置文件""" 47 | db_host, db_port, db_user, db_pwd = str(base64.b64decode(db_info), 'utf-8').split(',,,') 48 | db_host = conver_url2ip(db_host) 49 | if db_pwd == 'null': 50 | db_pwd = '' 51 | connstr_target = dict(host=db_host, port=db_port, user=db_user, password=db_pwd, charset='utf8mb4') 52 | connstr_inception = {'host': '127.0.0.1', 'port': 6669, 'user': '', 'password': '', 'db': '', 53 | 'charset': 'utf8mb4'} 54 | 55 | return connstr_target, connstr_inception 56 | 57 | 58 | def get_sql(db_file): 59 | gitlab = db_file.split('blob') 60 | if len(gitlab) != 2: 61 | return -1, '请输入正确的文件名' 62 | 63 | db_file_list = db_file.split('/') 64 | git_domain, git_group, git_app, git_branch = db_file_list[2], db_file_list[3], db_file_list[4], db_file_list[6] 65 | git_url = 'git@{}:{}/{}.git'.format(git_domain, git_group, git_app) 66 | print('db file path is',git_url) 67 | sql_file = '/tmp/codes/{}/{}'.format(git_app, db_file.split('blob')[1].replace('/' + git_branch + '/', '')) 68 | cmd = 'mkdir -p /tmp/codes/ && cd /tmp/codes/ && rm -rf {} && git clone {} && cd {} && git checkout {}'.format( 69 | git_app, git_url, git_app, git_branch) 70 | 71 | sub = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 72 | stdout, stderr = sub.communicate() 73 | ret = sub.returncode 74 | if ret == 0: 75 | with open(sql_file, "r") as f1: 76 | return 0, f1.read() 77 | else: 78 | return -1, stderr 79 | 80 | 81 | def exec_inception_v2(db_info, db_file): 82 | # ### 获取要执行的SQL 83 | exec_code, exec_sql = get_sql(db_file) 84 | if exec_code != 0: 85 | print(db_file) 86 | exit(-1) 87 | 88 | # 执行还是校验 89 | operation = '--execute=1' 90 | # operation = '--enable-execute;--enable-ignore-warnings;--enable-force' 91 | 92 | # 发布目标服务器 93 | connstr_target, connstr_inception = get_conf(db_info) 94 | 95 | try: 96 | # 将待执行的sql语句组合成inception识别的格式 97 | sql_with_format = '''/*--user={};--password={};--host={};{};--port={};*/ inception_magic_start;\n {} \ninception_magic_commit;'''.format( 98 | connstr_target['user'], 99 | connstr_target['password'], 100 | connstr_target['host'], 101 | operation, 102 | connstr_target['port'], exec_sql) 103 | 104 | # 连接至inception 服务器 105 | conn_inception = pymysql.connect(host=connstr_inception.get('host', '127.0.0.1'), 106 | port=connstr_inception.get('port', 6669), 107 | user=connstr_inception.get('user', ''), 108 | password=connstr_inception.get('password', ''), 109 | charset=connstr_inception.get('charset', 'utf8mb4')) 110 | 111 | cur = conn_inception.cursor() 112 | 113 | cur.execute(sql_with_format) 114 | result = cur.fetchall() 115 | num_fields = len(cur.description) 116 | field_names = [i[0] for i in cur.description] 117 | # print(field_names) 118 | # 打印出来Inception对MySQL语句的审计结果 119 | result_code = [] 120 | for row in result: 121 | print(row[0], "|", row[1], "|", row[2], "|", row[3], "|", row[4], "|", row[5], "|", row[6], "|", row[7], 122 | "|", row[8], "|", row[9], "|", row[10]) 123 | result_code.append(row[2]) 124 | 125 | cur.close() 126 | conn_inception.close() 127 | 128 | if 2 in result_code or 1 in result_code: 129 | print('host:{} error, please check the log !!!'.format(connstr_target['host'])) 130 | exit(-1) 131 | else: 132 | print('host:{} success !!!'.format(connstr_target['host'])) 133 | 134 | except Exception as err: 135 | print(err) 136 | exit(-2) 137 | finally: 138 | print('********************') 139 | 140 | 141 | def main(publish_name, db_file): 142 | obj = API() 143 | result = obj.get_publish_name_info(publish_name) 144 | publish_hosts_api, mail_to = result[0]['publish_hosts_api'], result[0]['mail_to'] 145 | if publish_hosts_api: 146 | res = obj.get_api_info(publish_hosts_api) 147 | for db in res['db_list']: 148 | if db['db_type'] == 'MySQL' and db['db_role'] == 'master': 149 | db_info = db['db_info'] 150 | exec_inception_v2(db_info, db_file) 151 | 152 | if __name__ == '__main__': 153 | fire.Fire(main) 154 | 155 | 156 | ###python3.6 /opt/ops_scripts/codo-check/exec_inception.py shenshuo_server-test http://gitlab.shinezoneserver.com/ops/123/blob/master/ss/shenyingzhi.txt -------------------------------------------------------------------------------- /exec_sonar.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Contact : 191715030@qq.com 5 | Author : shenshuo 6 | Date : 2018/8/9 7 | Desc : 执行代码扫描 8 | """ 9 | 10 | import fire 11 | import os 12 | import subprocess 13 | from settings import sonar_info 14 | from api_handler import API 15 | 16 | 17 | def get_config_info(publish_name): 18 | obj = API() 19 | result = obj.get_publish_name_info(publish_name) 20 | return result[0]['repository'], result[0]['mail_to'] 21 | 22 | 23 | def exec_sonar_v2(publish_name, git_repository, publish_tag, mail_to): 24 | app_name = git_repository.split('/')[-1].replace(".git", "") 25 | print(app_name) 26 | ### 删除已经存在的，然后创建 27 | app_path = os.path.join('/var/www/version/', app_name) 28 | os.system('mkdir -p {}'.format(app_path)) 29 | cmd1 = 'cd {} && rm -rf *'.format(app_path) 30 | os.system(cmd1) 31 | ## 代码获取完毕 32 | cmd = "cd {} && git clone -b {} {}".format(app_path, publish_tag, git_repository) 33 | print(cmd) 34 | 35 | sub1 = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 36 | stdout, stderr = sub1.communicate() 37 | ret = sub1.returncode 38 | if ret != 0: 39 | print(stdout, stderr) 40 | exit(-2) 41 | 42 | ## 写入扫描文件 43 | print("start checking code") 44 | sonar_file = os.path.join(app_path, app_name, "sonar-project.properties") 45 | with open(sonar_file, 'w') as f: 46 | f.write( 47 | "sonar.projectKey={key}\n" 48 | "sonar.projectName={name}\n" 49 | "sonar.projectVersion=1.0\n" 50 | "sonar.sources=.\n" 51 | "sonar.login={sonar_login}\n".format(key=app_name, name=publish_name, 52 | sonar_login=sonar_info.get('sonar_login'))) 53 | 54 | sub2 = subprocess.Popen("cd {}/{} && /usr/local/sonar-scanner/bin/sonar-scanner".format(app_path, app_name), 55 | shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 56 | stdout, stderr = sub2.communicate() 57 | 58 | for i in stdout.decode('utf-8').split('\n'): 59 | print(i) 60 | 61 | print("mail_to: ", mail_to) 62 | msg = "代码检查完毕！\n项目：{}\n应用：{}\n标签：{}\n用户名: shinezone \n密码：shinezone \n详情：{}{}".format( 63 | publish_name, app_name, publish_tag, sonar_info.get('sonar_domain'), app_name) 64 | obj = API() 65 | 66 | result = obj.send_mail_for_api(mail_to, '代码检查', msg) 67 | print(result) 68 | print("successful") 69 | 70 | 71 | def main(publish_name, publish_tag): 72 | if publish_name == 'PUBLISH_NAME' or publish_tag == 'PUBLISH_TAG': 73 | print("[Error:] PUBLISH_NAME PUBLISH_TAG 不能为空") 74 | exit(-1) 75 | 76 | git_repository, mail_to = get_config_info(publish_name) 77 | if git_repository: 78 | exec_sonar_v2(publish_name, git_repository, publish_tag, mail_to) 79 | else: 80 | print('[Error:] git repository is none !!!') 81 | 82 | 83 | if __name__ == '__main__': 84 | fire.Fire(main) 85 | -------------------------------------------------------------------------------- /exec_sqladvisor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*-coding:utf-8-*- 3 | """ 4 | author : shenshuo 5 | date : 2018年4月24日 6 | desc : 执行SQL优化 7 | 1. 更新生成配置 data: 2018年6月1日 8 | """ 9 | 10 | import fire 11 | import base64 12 | import subprocess 13 | from shortuuid import uuid 14 | from api_handler import API 15 | 16 | 17 | def get_conf(dbname, db_info): 18 | """获取SQL配置生产配置文件""" 19 | conf_file = '/tmp/' + uuid() + '.ini' 20 | if len(str(base64.b64decode(db_info)).split(',,,')) < 4: 21 | print('db_info error') 22 | exit(-100) 23 | 24 | db_host, db_port, db_user, db_pwd = str(base64.b64decode(db_info), 'utf-8').split(',,,') 25 | if db_pwd == 'null': 26 | db_pwd = '' 27 | with open(conf_file, 'w') as f: 28 | f.write("[sqladvisor]\n" 29 | "username={db_user}\n" 30 | "password={db_pwd}\n" 31 | "host={db_host}\n" 32 | "port={db_port}\n" 33 | "dbname={dbname}\n" 34 | .format(db_user=db_user, db_pwd=db_pwd, db_host=db_host, db_port=db_port, dbname=dbname)) 35 | return conf_file, db_host 36 | 37 | 38 | def exec_sqladvisor_v2(conf_file, sqls): 39 | all_result = '' 40 | 41 | for s in sqls.split(';'): 42 | if s: 43 | sub2 = subprocess.Popen('sqladvisor -f {} -q "{};" -v 1'.format(conf_file, s), shell=True, 44 | stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 45 | stdout, stderr = sub2.communicate() 46 | result = stdout.decode('utf-8').replace('\n\n', '\n') 47 | all_result = '{}\n{}'.format(all_result, result) 48 | 49 | return all_result 50 | 51 | 52 | def main(publish_name, db_name, sqls): 53 | if publish_name == 'PUBLISH_NAME' or db_name == 'DB_NAME': 54 | print("[Error:] PUBLISH_NAME DB_NAME 不能为空") 55 | exit(-1) 56 | 57 | obj = API() 58 | result = obj.get_publish_name_info(publish_name) 59 | publish_hosts_api, mail_to = result[0]['publish_hosts_api'], result[0]['mail_to'] 60 | 61 | print(publish_hosts_api, mail_to) 62 | all_msg = '' 63 | if publish_hosts_api: 64 | res = obj.get_api_info(publish_hosts_api) 65 | for db in res['db_list']: 66 | if db['db_type'] == 'MySQL' and db['db_role'] == 'master': 67 | db_info = db['db_info'] 68 | conf_file, db_host = get_conf(db_name, db_info) 69 | print('db host is {}'.format(db_host)) 70 | print("conf_file is " + conf_file) 71 | all_msg = all_msg + 'db host is {}\n'.format(db_host) 72 | all_msg = all_msg + "conf_file is {}\n".format(conf_file) 73 | result_msg = exec_sqladvisor_v2(conf_file, sqls) 74 | print(result_msg) 75 | all_msg = all_msg + result_msg 76 | 77 | ### 发送邮件 78 | msg = "SQL优化完毕！\n库名：{}\n语句：{}\n{}".format(db_name, sqls, all_msg) 79 | obj = API() 80 | 81 | print(obj.send_mail_for_api(mail_to, 'SQL索引优化', msg)) 82 | print("successful") 83 | else: 84 | print('[Error:] git repository is none !!!') 85 | 86 | 87 | if __name__ == '__main__': 88 | fire.Fire(main) 89 | 90 | ### 例如 91 | ### python3.6 /opt/ops_scripts/codo-check/exec_sqladvisor.py shenshuo_server-test shen '11224;dfsdfsd' -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | fire 2 | requests 3 | pyotp -------------------------------------------------------------------------------- /settings.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Contact : 191715030@qq.com 5 | Author : shenshuo 6 | Date : 2018/8/9 7 | Desc : 8 | """ 9 | 10 | api_gw = 'http://gw.shinezone.net.cn/api' 11 | mail_api = '{}/mg/v2/notifications/mail/'.format(api_gw) 12 | login_api = '{}/accounts/login/'.format(api_gw) 13 | publish_info_api = '{}/task/v2/task_other/publish_cd/'.format(api_gw) 14 | get_key_api = '{}/task/v2/task/accept/'.format(api_gw) 15 | 16 | ### cmdb 读取get， 邮件发送post 用户信息get 17 | api_user = 'publish' 18 | api_pwd = 'shenshuo' 19 | api_key = 'JJFTQNLXHBYWSVSCINNEWNDFJRKWGY2UNJTTSVTLMN3UGUKXPFDE4NDH' 20 | 21 | api_settings = dict( 22 | api_user=api_user, 23 | api_pwd=api_pwd, 24 | api_key=api_key, 25 | api_gw=api_gw, 26 | mail_api=mail_api, 27 | login_api=login_api, 28 | publish_info_api=publish_info_api, 29 | get_key_api=get_key_api, 30 | ) 31 | 32 | 33 | sonar_domain = 'http://ops-sonar.shinezone.net.cn/dashboard/index/' 34 | sonar_login = '0e8510011b3fb281d1737e13b7fdceadd6b601b4' 35 | sonar_info =dict( 36 | sonar_login =sonar_login, 37 | sonar_domain = sonar_domain 38 | ) 39 | --------------------------------------------------------------------------------