├── LICENSE.md ├── README.md ├── emaill2git.py ├── lines_data_prep ├── git_prep.py └── pw_prep.py ├── subject_data_gen ├── git │ └── commit_subject_generator.py └── patchwork │ └── pwSubjectFull.py ├── subject_matching.py ├── validation_tests └── read_test_output.py └── web └── matches_to_files.py /LICENSE.md: -------------------------------------------------------------------------------- 1 | ### GNU GENERAL PUBLIC LICENSE 2 | 3 | Version 3, 29 June 2007 4 | 5 | Copyright (C) 2007 Free Software Foundation, Inc. 6 | 7 | 8 | Everyone is permitted to copy and distribute verbatim copies of this 9 | license document, but changing it is not allowed. 10 | 11 | ### Preamble 12 | 13 | The GNU General Public License is a free, copyleft license for 14 | software and other kinds of works. 15 | 16 | The licenses for most software and other practical works are designed 17 | to take away your freedom to share and change the works. By contrast, 18 | the GNU General Public License is intended to guarantee your freedom 19 | to share and change all versions of a program--to make sure it remains 20 | free software for all its users. We, the Free Software Foundation, use 21 | the GNU General Public License for most of our software; it applies 22 | also to any other work released this way by its authors. You can apply 23 | it to your programs, too. 24 | 25 | When we speak of free software, we are referring to freedom, not 26 | price. Our General Public Licenses are designed to make sure that you 27 | have the freedom to distribute copies of free software (and charge for 28 | them if you wish), that you receive source code or can get it if you 29 | want it, that you can change the software or use pieces of it in new 30 | free programs, and that you know you can do these things. 31 | 32 | To protect your rights, we need to prevent others from denying you 33 | these rights or asking you to surrender the rights. Therefore, you 34 | have certain responsibilities if you distribute copies of the 35 | software, or if you modify it: responsibilities to respect the freedom 36 | of others. 37 | 38 | For example, if you distribute copies of such a program, whether 39 | gratis or for a fee, you must pass on to the recipients the same 40 | freedoms that you received. You must make sure that they, too, receive 41 | or can get the source code. And you must show them these terms so they 42 | know their rights. 43 | 44 | Developers that use the GNU GPL protect your rights with two steps: 45 | (1) assert copyright on the software, and (2) offer you this License 46 | giving you legal permission to copy, distribute and/or modify it. 47 | 48 | For the developers' and authors' protection, the GPL clearly explains 49 | that there is no warranty for this free software. For both users' and 50 | authors' sake, the GPL requires that modified versions be marked as 51 | changed, so that their problems will not be attributed erroneously to 52 | authors of previous versions. 53 | 54 | Some devices are designed to deny users access to install or run 55 | modified versions of the software inside them, although the 56 | manufacturer can do so. This is fundamentally incompatible with the 57 | aim of protecting users' freedom to change the software. The 58 | systematic pattern of such abuse occurs in the area of products for 59 | individuals to use, which is precisely where it is most unacceptable. 60 | Therefore, we have designed this version of the GPL to prohibit the 61 | practice for those products. If such problems arise substantially in 62 | other domains, we stand ready to extend this provision to those 63 | domains in future versions of the GPL, as needed to protect the 64 | freedom of users. 65 | 66 | Finally, every program is threatened constantly by software patents. 67 | States should not allow patents to restrict development and use of 68 | software on general-purpose computers, but in those that do, we wish 69 | to avoid the special danger that patents applied to a free program 70 | could make it effectively proprietary. To prevent this, the GPL 71 | assures that patents cannot be used to render the program non-free. 72 | 73 | The precise terms and conditions for copying, distribution and 74 | modification follow. 75 | 76 | ### TERMS AND CONDITIONS 77 | 78 | #### 0. Definitions. 79 | 80 | "This License" refers to version 3 of the GNU General Public License. 81 | 82 | "Copyright" also means copyright-like laws that apply to other kinds 83 | of works, such as semiconductor masks. 84 | 85 | "The Program" refers to any copyrightable work licensed under this 86 | License. Each licensee is addressed as "you". "Licensees" and 87 | "recipients" may be individuals or organizations. 88 | 89 | To "modify" a work means to copy from or adapt all or part of the work 90 | in a fashion requiring copyright permission, other than the making of 91 | an exact copy. The resulting work is called a "modified version" of 92 | the earlier work or a work "based on" the earlier work. 93 | 94 | A "covered work" means either the unmodified Program or a work based 95 | on the Program. 96 | 97 | To "propagate" a work means to do anything with it that, without 98 | permission, would make you directly or secondarily liable for 99 | infringement under applicable copyright law, except executing it on a 100 | computer or modifying a private copy. Propagation includes copying, 101 | distribution (with or without modification), making available to the 102 | public, and in some countries other activities as well. 103 | 104 | To "convey" a work means any kind of propagation that enables other 105 | parties to make or receive copies. Mere interaction with a user 106 | through a computer network, with no transfer of a copy, is not 107 | conveying. 108 | 109 | An interactive user interface displays "Appropriate Legal Notices" to 110 | the extent that it includes a convenient and prominently visible 111 | feature that (1) displays an appropriate copyright notice, and (2) 112 | tells the user that there is no warranty for the work (except to the 113 | extent that warranties are provided), that licensees may convey the 114 | work under this License, and how to view a copy of this License. If 115 | the interface presents a list of user commands or options, such as a 116 | menu, a prominent item in the list meets this criterion. 117 | 118 | #### 1. Source Code. 119 | 120 | The "source code" for a work means the preferred form of the work for 121 | making modifications to it. "Object code" means any non-source form of 122 | a work. 123 | 124 | A "Standard Interface" means an interface that either is an official 125 | standard defined by a recognized standards body, or, in the case of 126 | interfaces specified for a particular programming language, one that 127 | is widely used among developers working in that language. 128 | 129 | The "System Libraries" of an executable work include anything, other 130 | than the work as a whole, that (a) is included in the normal form of 131 | packaging a Major Component, but which is not part of that Major 132 | Component, and (b) serves only to enable use of the work with that 133 | Major Component, or to implement a Standard Interface for which an 134 | implementation is available to the public in source code form. A 135 | "Major Component", in this context, means a major essential component 136 | (kernel, window system, and so on) of the specific operating system 137 | (if any) on which the executable work runs, or a compiler used to 138 | produce the work, or an object code interpreter used to run it. 139 | 140 | The "Corresponding Source" for a work in object code form means all 141 | the source code needed to generate, install, and (for an executable 142 | work) run the object code and to modify the work, including scripts to 143 | control those activities. However, it does not include the work's 144 | System Libraries, or general-purpose tools or generally available free 145 | programs which are used unmodified in performing those activities but 146 | which are not part of the work. For example, Corresponding Source 147 | includes interface definition files associated with source files for 148 | the work, and the source code for shared libraries and dynamically 149 | linked subprograms that the work is specifically designed to require, 150 | such as by intimate data communication or control flow between those 151 | subprograms and other parts of the work. 152 | 153 | The Corresponding Source need not include anything that users can 154 | regenerate automatically from other parts of the Corresponding Source. 155 | 156 | The Corresponding Source for a work in source code form is that same 157 | work. 158 | 159 | #### 2. Basic Permissions. 160 | 161 | All rights granted under this License are granted for the term of 162 | copyright on the Program, and are irrevocable provided the stated 163 | conditions are met. This License explicitly affirms your unlimited 164 | permission to run the unmodified Program. The output from running a 165 | covered work is covered by this License only if the output, given its 166 | content, constitutes a covered work. This License acknowledges your 167 | rights of fair use or other equivalent, as provided by copyright law. 168 | 169 | You may make, run and propagate covered works that you do not convey, 170 | without conditions so long as your license otherwise remains in force. 171 | You may convey covered works to others for the sole purpose of having 172 | them make modifications exclusively for you, or provide you with 173 | facilities for running those works, provided that you comply with the 174 | terms of this License in conveying all material for which you do not 175 | control copyright. Those thus making or running the covered works for 176 | you must do so exclusively on your behalf, under your direction and 177 | control, on terms that prohibit them from making any copies of your 178 | copyrighted material outside their relationship with you. 179 | 180 | Conveying under any other circumstances is permitted solely under the 181 | conditions stated below. Sublicensing is not allowed; section 10 makes 182 | it unnecessary. 183 | 184 | #### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 185 | 186 | No covered work shall be deemed part of an effective technological 187 | measure under any applicable law fulfilling obligations under article 188 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 189 | similar laws prohibiting or restricting circumvention of such 190 | measures. 191 | 192 | When you convey a covered work, you waive any legal power to forbid 193 | circumvention of technological measures to the extent such 194 | circumvention is effected by exercising rights under this License with 195 | respect to the covered work, and you disclaim any intention to limit 196 | operation or modification of the work as a means of enforcing, against 197 | the work's users, your or third parties' legal rights to forbid 198 | circumvention of technological measures. 199 | 200 | #### 4. Conveying Verbatim Copies. 201 | 202 | You may convey verbatim copies of the Program's source code as you 203 | receive it, in any medium, provided that you conspicuously and 204 | appropriately publish on each copy an appropriate copyright notice; 205 | keep intact all notices stating that this License and any 206 | non-permissive terms added in accord with section 7 apply to the code; 207 | keep intact all notices of the absence of any warranty; and give all 208 | recipients a copy of this License along with the Program. 209 | 210 | You may charge any price or no price for each copy that you convey, 211 | and you may offer support or warranty protection for a fee. 212 | 213 | #### 5. Conveying Modified Source Versions. 214 | 215 | You may convey a work based on the Program, or the modifications to 216 | produce it from the Program, in the form of source code under the 217 | terms of section 4, provided that you also meet all of these 218 | conditions: 219 | 220 | - a) The work must carry prominent notices stating that you modified 221 | it, and giving a relevant date. 222 | - b) The work must carry prominent notices stating that it is 223 | released under this License and any conditions added under 224 | section 7. This requirement modifies the requirement in section 4 225 | to "keep intact all notices". 226 | - c) You must license the entire work, as a whole, under this 227 | License to anyone who comes into possession of a copy. This 228 | License will therefore apply, along with any applicable section 7 229 | additional terms, to the whole of the work, and all its parts, 230 | regardless of how they are packaged. This License gives no 231 | permission to license the work in any other way, but it does not 232 | invalidate such permission if you have separately received it. 233 | - d) If the work has interactive user interfaces, each must display 234 | Appropriate Legal Notices; however, if the Program has interactive 235 | interfaces that do not display Appropriate Legal Notices, your 236 | work need not make them do so. 237 | 238 | A compilation of a covered work with other separate and independent 239 | works, which are not by their nature extensions of the covered work, 240 | and which are not combined with it such as to form a larger program, 241 | in or on a volume of a storage or distribution medium, is called an 242 | "aggregate" if the compilation and its resulting copyright are not 243 | used to limit the access or legal rights of the compilation's users 244 | beyond what the individual works permit. Inclusion of a covered work 245 | in an aggregate does not cause this License to apply to the other 246 | parts of the aggregate. 247 | 248 | #### 6. Conveying Non-Source Forms. 249 | 250 | You may convey a covered work in object code form under the terms of 251 | sections 4 and 5, provided that you also convey the machine-readable 252 | Corresponding Source under the terms of this License, in one of these 253 | ways: 254 | 255 | - a) Convey the object code in, or embodied in, a physical product 256 | (including a physical distribution medium), accompanied by the 257 | Corresponding Source fixed on a durable physical medium 258 | customarily used for software interchange. 259 | - b) Convey the object code in, or embodied in, a physical product 260 | (including a physical distribution medium), accompanied by a 261 | written offer, valid for at least three years and valid for as 262 | long as you offer spare parts or customer support for that product 263 | model, to give anyone who possesses the object code either (1) a 264 | copy of the Corresponding Source for all the software in the 265 | product that is covered by this License, on a durable physical 266 | medium customarily used for software interchange, for a price no 267 | more than your reasonable cost of physically performing this 268 | conveying of source, or (2) access to copy the Corresponding 269 | Source from a network server at no charge. 270 | - c) Convey individual copies of the object code with a copy of the 271 | written offer to provide the Corresponding Source. This 272 | alternative is allowed only occasionally and noncommercially, and 273 | only if you received the object code with such an offer, in accord 274 | with subsection 6b. 275 | - d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | - e) Convey the object code using peer-to-peer transmission, 288 | provided you inform other peers where the object code and 289 | Corresponding Source of the work are being offered to the general 290 | public at no charge under subsection 6d. 291 | 292 | A separable portion of the object code, whose source code is excluded 293 | from the Corresponding Source as a System Library, need not be 294 | included in conveying the object code work. 295 | 296 | A "User Product" is either (1) a "consumer product", which means any 297 | tangible personal property which is normally used for personal, 298 | family, or household purposes, or (2) anything designed or sold for 299 | incorporation into a dwelling. In determining whether a product is a 300 | consumer product, doubtful cases shall be resolved in favor of 301 | coverage. For a particular product received by a particular user, 302 | "normally used" refers to a typical or common use of that class of 303 | product, regardless of the status of the particular user or of the way 304 | in which the particular user actually uses, or expects or is expected 305 | to use, the product. A product is a consumer product regardless of 306 | whether the product has substantial commercial, industrial or 307 | non-consumer uses, unless such uses represent the only significant 308 | mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to 312 | install and execute modified versions of a covered work in that User 313 | Product from a modified version of its Corresponding Source. The 314 | information must suffice to ensure that the continued functioning of 315 | the modified object code is in no case prevented or interfered with 316 | solely because modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or 331 | updates for a work that has been modified or installed by the 332 | recipient, or for the User Product in which it has been modified or 333 | installed. Access to a network may be denied when the modification 334 | itself materially and adversely affects the operation of the network 335 | or violates the rules and protocols for communication across the 336 | network. 337 | 338 | Corresponding Source conveyed, and Installation Information provided, 339 | in accord with this section must be in a format that is publicly 340 | documented (and with an implementation available to the public in 341 | source code form), and must require no special password or key for 342 | unpacking, reading or copying. 343 | 344 | #### 7. Additional Terms. 345 | 346 | "Additional permissions" are terms that supplement the terms of this 347 | License by making exceptions from one or more of its conditions. 348 | Additional permissions that are applicable to the entire Program shall 349 | be treated as though they were included in this License, to the extent 350 | that they are valid under applicable law. If additional permissions 351 | apply only to part of the Program, that part may be used separately 352 | under those permissions, but the entire Program remains governed by 353 | this License without regard to the additional permissions. 354 | 355 | When you convey a copy of a covered work, you may at your option 356 | remove any additional permissions from that copy, or from any part of 357 | it. (Additional permissions may be written to require their own 358 | removal in certain cases when you modify the work.) You may place 359 | additional permissions on material, added by you to a covered work, 360 | for which you have or can give appropriate copyright permission. 361 | 362 | Notwithstanding any other provision of this License, for material you 363 | add to a covered work, you may (if authorized by the copyright holders 364 | of that material) supplement the terms of this License with terms: 365 | 366 | - a) Disclaiming warranty or limiting liability differently from the 367 | terms of sections 15 and 16 of this License; or 368 | - b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | - c) Prohibiting misrepresentation of the origin of that material, 372 | or requiring that modified versions of such material be marked in 373 | reasonable ways as different from the original version; or 374 | - d) Limiting the use for publicity purposes of names of licensors 375 | or authors of the material; or 376 | - e) Declining to grant rights under trademark law for use of some 377 | trade names, trademarks, or service marks; or 378 | - f) Requiring indemnification of licensors and authors of that 379 | material by anyone who conveys the material (or modified versions 380 | of it) with contractual assumptions of liability to the recipient, 381 | for any liability that these contractual assumptions directly 382 | impose on those licensors and authors. 383 | 384 | All other non-permissive additional terms are considered "further 385 | restrictions" within the meaning of section 10. If the Program as you 386 | received it, or any part of it, contains a notice stating that it is 387 | governed by this License along with a term that is a further 388 | restriction, you may remove that term. If a license document contains 389 | a further restriction but permits relicensing or conveying under this 390 | License, you may add to a covered work material governed by the terms 391 | of that license document, provided that the further restriction does 392 | not survive such relicensing or conveying. 393 | 394 | If you add terms to a covered work in accord with this section, you 395 | must place, in the relevant source files, a statement of the 396 | additional terms that apply to those files, or a notice indicating 397 | where to find the applicable terms. 398 | 399 | Additional terms, permissive or non-permissive, may be stated in the 400 | form of a separately written license, or stated as exceptions; the 401 | above requirements apply either way. 402 | 403 | #### 8. Termination. 404 | 405 | You may not propagate or modify a covered work except as expressly 406 | provided under this License. Any attempt otherwise to propagate or 407 | modify it is void, and will automatically terminate your rights under 408 | this License (including any patent licenses granted under the third 409 | paragraph of section 11). 410 | 411 | However, if you cease all violation of this License, then your license 412 | from a particular copyright holder is reinstated (a) provisionally, 413 | unless and until the copyright holder explicitly and finally 414 | terminates your license, and (b) permanently, if the copyright holder 415 | fails to notify you of the violation by some reasonable means prior to 416 | 60 days after the cessation. 417 | 418 | Moreover, your license from a particular copyright holder is 419 | reinstated permanently if the copyright holder notifies you of the 420 | violation by some reasonable means, this is the first time you have 421 | received notice of violation of this License (for any work) from that 422 | copyright holder, and you cure the violation prior to 30 days after 423 | your receipt of the notice. 424 | 425 | Termination of your rights under this section does not terminate the 426 | licenses of parties who have received copies or rights from you under 427 | this License. If your rights have been terminated and not permanently 428 | reinstated, you do not qualify to receive new licenses for the same 429 | material under section 10. 430 | 431 | #### 9. Acceptance Not Required for Having Copies. 432 | 433 | You are not required to accept this License in order to receive or run 434 | a copy of the Program. Ancillary propagation of a covered work 435 | occurring solely as a consequence of using peer-to-peer transmission 436 | to receive a copy likewise does not require acceptance. However, 437 | nothing other than this License grants you permission to propagate or 438 | modify any covered work. These actions infringe copyright if you do 439 | not accept this License. Therefore, by modifying or propagating a 440 | covered work, you indicate your acceptance of this License to do so. 441 | 442 | #### 10. Automatic Licensing of Downstream Recipients. 443 | 444 | Each time you convey a covered work, the recipient automatically 445 | receives a license from the original licensors, to run, modify and 446 | propagate that work, subject to this License. You are not responsible 447 | for enforcing compliance by third parties with this License. 448 | 449 | An "entity transaction" is a transaction transferring control of an 450 | organization, or substantially all assets of one, or subdividing an 451 | organization, or merging organizations. If propagation of a covered 452 | work results from an entity transaction, each party to that 453 | transaction who receives a copy of the work also receives whatever 454 | licenses to the work the party's predecessor in interest had or could 455 | give under the previous paragraph, plus a right to possession of the 456 | Corresponding Source of the work from the predecessor in interest, if 457 | the predecessor has it or can get it with reasonable efforts. 458 | 459 | You may not impose any further restrictions on the exercise of the 460 | rights granted or affirmed under this License. For example, you may 461 | not impose a license fee, royalty, or other charge for exercise of 462 | rights granted under this License, and you may not initiate litigation 463 | (including a cross-claim or counterclaim in a lawsuit) alleging that 464 | any patent claim is infringed by making, using, selling, offering for 465 | sale, or importing the Program or any portion of it. 466 | 467 | #### 11. Patents. 468 | 469 | A "contributor" is a copyright holder who authorizes use under this 470 | License of the Program or a work on which the Program is based. The 471 | work thus licensed is called the contributor's "contributor version". 472 | 473 | A contributor's "essential patent claims" are all patent claims owned 474 | or controlled by the contributor, whether already acquired or 475 | hereafter acquired, that would be infringed by some manner, permitted 476 | by this License, of making, using, or selling its contributor version, 477 | but do not include claims that would be infringed only as a 478 | consequence of further modification of the contributor version. For 479 | purposes of this definition, "control" includes the right to grant 480 | patent sublicenses in a manner consistent with the requirements of 481 | this License. 482 | 483 | Each contributor grants you a non-exclusive, worldwide, royalty-free 484 | patent license under the contributor's essential patent claims, to 485 | make, use, sell, offer for sale, import and otherwise run, modify and 486 | propagate the contents of its contributor version. 487 | 488 | In the following three paragraphs, a "patent license" is any express 489 | agreement or commitment, however denominated, not to enforce a patent 490 | (such as an express permission to practice a patent or covenant not to 491 | sue for patent infringement). To "grant" such a patent license to a 492 | party means to make such an agreement or commitment not to enforce a 493 | patent against the party. 494 | 495 | If you convey a covered work, knowingly relying on a patent license, 496 | and the Corresponding Source of the work is not available for anyone 497 | to copy, free of charge and under the terms of this License, through a 498 | publicly available network server or other readily accessible means, 499 | then you must either (1) cause the Corresponding Source to be so 500 | available, or (2) arrange to deprive yourself of the benefit of the 501 | patent license for this particular work, or (3) arrange, in a manner 502 | consistent with the requirements of this License, to extend the patent 503 | license to downstream recipients. "Knowingly relying" means you have 504 | actual knowledge that, but for the patent license, your conveying the 505 | covered work in a country, or your recipient's use of the covered work 506 | in a country, would infringe one or more identifiable patents in that 507 | country that you have reason to believe are valid. 508 | 509 | If, pursuant to or in connection with a single transaction or 510 | arrangement, you convey, or propagate by procuring conveyance of, a 511 | covered work, and grant a patent license to some of the parties 512 | receiving the covered work authorizing them to use, propagate, modify 513 | or convey a specific copy of the covered work, then the patent license 514 | you grant is automatically extended to all recipients of the covered 515 | work and works based on it. 516 | 517 | A patent license is "discriminatory" if it does not include within the 518 | scope of its coverage, prohibits the exercise of, or is conditioned on 519 | the non-exercise of one or more of the rights that are specifically 520 | granted under this License. You may not convey a covered work if you 521 | are a party to an arrangement with a third party that is in the 522 | business of distributing software, under which you make payment to the 523 | third party based on the extent of your activity of conveying the 524 | work, and under which the third party grants, to any of the parties 525 | who would receive the covered work from you, a discriminatory patent 526 | license (a) in connection with copies of the covered work conveyed by 527 | you (or copies made from those copies), or (b) primarily for and in 528 | connection with specific products or compilations that contain the 529 | covered work, unless you entered into that arrangement, or that patent 530 | license was granted, prior to 28 March 2007. 531 | 532 | Nothing in this License shall be construed as excluding or limiting 533 | any implied license or other defenses to infringement that may 534 | otherwise be available to you under applicable patent law. 535 | 536 | #### 12. No Surrender of Others' Freedom. 537 | 538 | If conditions are imposed on you (whether by court order, agreement or 539 | otherwise) that contradict the conditions of this License, they do not 540 | excuse you from the conditions of this License. If you cannot convey a 541 | covered work so as to satisfy simultaneously your obligations under 542 | this License and any other pertinent obligations, then as a 543 | consequence you may not convey it at all. For example, if you agree to 544 | terms that obligate you to collect a royalty for further conveying 545 | from those to whom you convey the Program, the only way you could 546 | satisfy both those terms and this License would be to refrain entirely 547 | from conveying the Program. 548 | 549 | #### 13. Use with the GNU Affero General Public License. 550 | 551 | Notwithstanding any other provision of this License, you have 552 | permission to link or combine any covered work with a work licensed 553 | under version 3 of the GNU Affero General Public License into a single 554 | combined work, and to convey the resulting work. The terms of this 555 | License will continue to apply to the part which is the covered work, 556 | but the special requirements of the GNU Affero General Public License, 557 | section 13, concerning interaction through a network will apply to the 558 | combination as such. 559 | 560 | #### 14. Revised Versions of this License. 561 | 562 | The Free Software Foundation may publish revised and/or new versions 563 | of the GNU General Public License from time to time. Such new versions 564 | will be similar in spirit to the present version, but may differ in 565 | detail to address new problems or concerns. 566 | 567 | Each version is given a distinguishing version number. If the Program 568 | specifies that a certain numbered version of the GNU General Public 569 | License "or any later version" applies to it, you have the option of 570 | following the terms and conditions either of that numbered version or 571 | of any later version published by the Free Software Foundation. If the 572 | Program does not specify a version number of the GNU General Public 573 | License, you may choose any version ever published by the Free 574 | Software Foundation. 575 | 576 | If the Program specifies that a proxy can decide which future versions 577 | of the GNU General Public License can be used, that proxy's public 578 | statement of acceptance of a version permanently authorizes you to 579 | choose that version for the Program. 580 | 581 | Later license versions may give you additional or different 582 | permissions. However, no additional obligations are imposed on any 583 | author or copyright holder as a result of your choosing to follow a 584 | later version. 585 | 586 | #### 15. Disclaimer of Warranty. 587 | 588 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 589 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 590 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT 591 | WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT 592 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 593 | A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND 594 | PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE 595 | DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR 596 | CORRECTION. 597 | 598 | #### 16. Limitation of Liability. 599 | 600 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 601 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR 602 | CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 603 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES 604 | ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT 605 | NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR 606 | LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM 607 | TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER 608 | PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 609 | 610 | #### 17. Interpretation of Sections 15 and 16. 611 | 612 | If the disclaimer of warranty and limitation of liability provided 613 | above cannot be given local legal effect according to their terms, 614 | reviewing courts shall apply local law that most closely approximates 615 | an absolute waiver of all civil liability in connection with the 616 | Program, unless a warranty or assumption of liability accompanies a 617 | copy of the Program in return for a fee. 618 | 619 | END OF TERMS AND CONDITIONS 620 | 621 | ### How to Apply These Terms to Your New Programs 622 | 623 | If you develop a new program, and you want it to be of the greatest 624 | possible use to the public, the best way to achieve this is to make it 625 | free software which everyone can redistribute and change under these 626 | terms. 627 | 628 | To do so, attach the following notices to the program. It is safest to 629 | attach them to the start of each source file to most effectively state 630 | the exclusion of warranty; and each file should have at least the 631 | "copyright" line and a pointer to where the full notice is found. 632 | 633 | 634 | Copyright (C) 635 | 636 | This program is free software: you can redistribute it and/or modify 637 | it under the terms of the GNU General Public License as published by 638 | the Free Software Foundation, either version 3 of the License, or 639 | (at your option) any later version. 640 | 641 | This program is distributed in the hope that it will be useful, 642 | but WITHOUT ANY WARRANTY; without even the implied warranty of 643 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 644 | GNU General Public License for more details. 645 | 646 | You should have received a copy of the GNU General Public License 647 | along with this program. If not, see . 648 | 649 | Also add information on how to contact you by electronic and paper 650 | mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands \`show w' and \`show c' should show the 661 | appropriate parts of the General Public License. Of course, your 662 | program's commands might be different; for a GUI interface, you would 663 | use an "about box". 664 | 665 | You should also get your employer (if you work as a programmer) or 666 | school, if any, to sign a "copyright disclaimer" for the program, if 667 | necessary. For more information on this, and how to apply and follow 668 | the GNU GPL, see . 669 | 670 | The GNU General Public License does not permit incorporating your 671 | program into proprietary programs. If your program is a subroutine 672 | library, you may consider it more useful to permit linking proprietary 673 | applications with the library. If this is what you want to do, use the 674 | GNU Lesser General Public License instead of this License. But first, 675 | please read . -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # email2git 2 | 3 | ## Background Information 4 | 5 | The Linux project's email-based reviewing process is highly effective in filtering open source contributions on their way from mailing list discussions towards Linus Torvalds' Git repository. However, once integrated, it can be difficult to link Git commits back to their review comments in mailing list discussions, especially when considering commits that underwent multiple versions (and hence review rounds), that belong to a multi-patch series, or that were cherry-picked. 6 | 7 | As an answer to these and other issues, we created email2git, a patch retrieving system built for the Linux kernel. For a given commit, the tool is capable of finding the email patch as well as the email conversation that took place during the review process. 8 | 9 | This repository contains the scripts used to retrieve the original email patch and discussion that introduced linux commits. The data comes from two different sources: the Linux git repository and [patchwork](https://github.com/getpatchwork/patchwork). Patchwork tracks the patches sent to a mailing list and organises the review in a user firendly maner. 10 | 11 | The commit-patch matches are accessible [here](http://mcis.polymtl.ca/~courouble/email2git/). 12 | 13 | 14 | ## Using Email2git for your Project 15 | 16 | If your project uses an email-based contribution workflow, you can use email2git to provide your community with better context around your project's commits. 17 | 18 | Following are the steps I followed to run Email2git on the Linux Kernel. Keep in mind that the Kernel has 700k + commits, which represents a large amount of data to parse. The following steps were implemented to speed up the matching process. 19 | 20 | The process is composed of two major steps: 21 | 22 | ### Email subject matching: 23 | 24 | This steps leverages the patch subject / commit summary concept. Depending on the linux subsystem, the email subject might often be used as the "commit summary" as the patch makes it to the main linux tree. 25 | 26 | The scripts used to prepare the data are in [subject_data_gen](https://github.com/alexcourouble/email2git/tree/master/subject_data_gen). 27 | 28 | [commit_subject_generator.py](https://github.com/alexcourouble/email2git/blob/master/subject_data_gen/git/commit_subject_generator.py) reads a git log output and [pwSubjectFull.py](https://github.com/alexcourouble/email2git/blob/master/subject_data_gen/patchwork/pwSubjectFull.py) reads a data dump from the patchwork database. 29 | 30 | These two scripts will generate the datasets used by [subject_matching.py](https://github.com/alexcourouble/email2git/blob/master/subject_matching.py) to execute the subject matching. 31 | 32 | ### Targeted lines-based matching: 33 | 34 | This step read the author and the files of patches and commit to make targeted patch/git diff line comparisons. 35 | 36 | In this step, we also have to generate the necessary data with the scripts in [line_data_prep](https://github.com/alexcourouble/email2git/tree/master/lines_data_prep). Once the data is generated, we run [email2git.py](https://github.com/alexcourouble/email2git/blob/master/emaill2git.py) to generate the list of matches. 37 | 38 | 39 | -------------------------------------------------------------------------------- /emaill2git.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | """ 18 | import sqlite3 19 | import cPickle as pickle 20 | import difflib 21 | import time 22 | import sys 23 | 24 | # LINES 25 | # PATCHES_PICKLED = '/Users/alexandrecourouble/Desktop/email2git_data/PATCHES_PICKLED_test.txt' 26 | # COMMITS_PICKLED = '/Users/alexandrecourouble/Desktop/email2git_data/COMMITS_PICKLED_test.txt' 27 | # COMMIT_MAP = '/Users/alexandrecourouble/Desktop/email2git_data/COMMIT_MAP_PICKLED.txt' 28 | 29 | # DB_PATH = "/Users/alexandrecourouble/Desktop/email2git_data/lookupDB.db" 30 | 31 | PATCHES_PICKLED = '/home-students/courouble/email2git_data/PATCHES_PICKLED_test.txt' 32 | COMMITS_PICKLED = '/home-students/courouble/email2git_data/COMMITS_PICKLED_test.txt' 33 | COMMIT_MAP = '/home-students/courouble/email2git_data/COMMIT_MAP_PICKLED.txt' 34 | 35 | DB_PATH = "/home-students/courouble/email2git_data/lookupDB.db" 36 | 37 | MATCH_RATIO = .4 38 | 39 | PATCH_FILE_MAP = {} 40 | PATCH_FILE_MAP_SHORT = {} 41 | COMMIT_FILE_MAP = {} 42 | 43 | PATCH_AUTHOR_MAP = {} 44 | COMMIT_AUTHOR_MAP = {} 45 | 46 | patches = {} 47 | commits = {} 48 | 49 | subjectMatches = {} 50 | lineMatches = {} 51 | 52 | subjectMatchesCount = 0 53 | lineMatchesCount = 0 54 | 55 | 56 | MATCHED_CID = set([]) 57 | MATCHED_PWID = set([]) 58 | 59 | 60 | notMatched = [] 61 | 62 | 63 | 64 | 65 | def getLineMatches(): 66 | readPW() 67 | readGit() 68 | print "========================= BEGENING =========================" 69 | print "========================= " + len(MATCHED_CID) + " =========================" 70 | doAuthMapMatching() 71 | print "========================= doAuthMapMatching() =========================" 72 | print "========================= " + len(MATCHED_CID) + " =========================" 73 | createFilePathMap() 74 | print "========================= createFilePathMap() =========================" 75 | print "========================= " + len(MATCHED_CID) + " =========================" 76 | doFileMapMatching() 77 | print "========================= doFileMapMatching() =========================" 78 | print "========================= " + len(MATCHED_CID) + " =========================" 79 | createFileNameMap() 80 | print "========================= createFileNameMap() =========================" 81 | print "========================= " + len(MATCHED_CID) + " =========================" 82 | doFileNameMapMatching() 83 | print "========================= doFileNameMapMatching() =========================" 84 | print "========================= " + len(MATCHED_CID) + " =========================" 85 | # doBruteMatching() 86 | 87 | 88 | def readPW(): 89 | global PATCH_AUTHOR_MAP 90 | global patches 91 | with open(PATCHES_PICKLED) as f: 92 | print "Reading patches data" 93 | patches = pickle.load(f) 94 | 95 | for i in patches: 96 | auth = patches[i]["author"] 97 | if auth not in PATCH_AUTHOR_MAP: 98 | PATCH_AUTHOR_MAP[auth] = [] 99 | PATCH_AUTHOR_MAP[auth].append(i) 100 | print 'Found:',len(patches),"patches." 101 | 102 | def readGit(): 103 | global commits 104 | global notMatched 105 | global COMMIT_FILE_MAP 106 | global COMMIT_AUTHOR_MAP 107 | # opening commit file map 108 | with open(COMMIT_MAP) as f: 109 | print "Reading commit map" 110 | COMMIT_FILE_MAP = pickle.load(f) 111 | print len(COMMIT_FILE_MAP), "commits in COMMIT_FILE_MAP" 112 | 113 | # opening commit pickle 114 | with open(COMMITS_PICKLED) as f: 115 | print "Reading commit data" 116 | commits = pickle.load(f) 117 | print "Found:", len(commits), "commits" 118 | 119 | # creating cid -> author email map. 120 | for i in commits: 121 | COMMIT_AUTHOR_MAP[i] = commits[i]["personid"] 122 | 123 | 124 | 125 | 126 | def doAuthMapMatching(): 127 | global MATCHED_PWID 128 | global MATCHED_CID 129 | print "========================================" 130 | print "Starting author map based line matches" 131 | # iterate through the author map 132 | countNoAuth = 0 133 | for i in COMMIT_AUTHOR_MAP: 134 | if COMMIT_AUTHOR_MAP[i] in PATCH_AUTHOR_MAP: 135 | # print i, PATCH_AUTHOR_MAP[COMMIT_AUTHOR_MAP[i]] 136 | for j in PATCH_AUTHOR_MAP[COMMIT_AUTHOR_MAP[i]]: 137 | compareDiffs(i,j,MATCH_RATIO) 138 | else: 139 | # print "not found: ",COMMIT_AUTHOR_MAP[i] 140 | countNoAuth += 1 141 | print "number of commits in author map:", len(COMMIT_AUTHOR_MAP) 142 | print "not found: ", countNoAuth 143 | print "len(MATCHED_CID)", len(MATCHED_CID) 144 | print "len(MATCHED_PWID)", len(MATCHED_PWID) 145 | 146 | 147 | 148 | 149 | def createFilePathMap(): 150 | for i in patches: 151 | # CREATING FILE BASED DICT {filePath : pwid} 152 | for j in patches[i]["files"]: 153 | filename = j 154 | if filename not in PATCH_FILE_MAP: 155 | PATCH_FILE_MAP[filename] = [] 156 | PATCH_FILE_MAP[filename].append(i) 157 | 158 | 159 | 160 | def doFileMapMatching(): 161 | global MATCHED_PWID 162 | global MATCHED_CID 163 | print "========================================" 164 | print "Starting file map based line matches" 165 | countNoFile = 0 166 | # iterate through cid-file map 167 | for i in COMMIT_FILE_MAP: 168 | # if i in notMatched: # taking only commits that have NOT been matched by subject 169 | # for all the files j touched by commit i 170 | for j in COMMIT_FILE_MAP[i]: 171 | # find all PWID that touched that file: 172 | # Have to check if file is in FILE-PWID map 173 | name_only = j.split("/")[-1] 174 | if j in PATCH_FILE_MAP: 175 | for k in PATCH_FILE_MAP[j]: 176 | if i not in MATCHED_CID and k not in MATCHED_PWID: 177 | compareDiffs(i,k,MATCH_RATIO) 178 | elif name_only in PATCH_FILE_MAP_SHORT: 179 | for k in PATCH_FILE_MAP_SHORT[name_only]: 180 | if i not in MATCHED_CID and k not in MATCHED_PWID: 181 | compareDiffs(i,k,MATCH_RATIO) 182 | else: 183 | # print j ,"NOT FOUND" 184 | countNoFile += 1 185 | 186 | print "not found: ", countNoFile 187 | print "len(MATCHED_CID)", len(MATCHED_CID) 188 | print "len(MATCHED_PWID)", len(MATCHED_PWID) 189 | 190 | 191 | 192 | def createFileNameMap(): 193 | for i in patches: 194 | # CREATING FILE BASED DICT {filePath : pwid} 195 | for j in patches[i]["files"]: 196 | name_only = j.split("/")[-1] 197 | if name_only not in PATCH_FILE_MAP_SHORT: 198 | PATCH_FILE_MAP_SHORT[name_only] = [] 199 | PATCH_FILE_MAP_SHORT[name_only].append(i) 200 | 201 | 202 | def doFileNameMapMatching(): 203 | global MATCHED_PWID 204 | global MATCHED_CID 205 | print "========================================" 206 | print "Starting filename map based line matches" 207 | countNoFile = 0 208 | # iterate through cid-file map 209 | for i in COMMIT_FILE_MAP: 210 | # print i 211 | # if i in notMatched: # taking only commits that have NOT been matched by subject 212 | # for all the files j touched by commit i 213 | for j in COMMIT_FILE_MAP[i]: 214 | # find all PWID that touched that file: 215 | # Have to check if file is in FILE-PWID map 216 | name_only = j.split("/")[-1] 217 | if name_only in PATCH_FILE_MAP_SHORT and name_only not in ["Kconfig","Makefile","Kbuild","MAINTAINERS"]: 218 | # print name_only 219 | # print PATCH_FILE_MAP_SHORT[name_only] 220 | for k in PATCH_FILE_MAP_SHORT[name_only]: 221 | if i not in MATCHED_CID and k not in MATCHED_PWID: 222 | compareDiffs(i,k,MATCH_RATIO) 223 | else: 224 | # print name_only ,"NOT FOUND" 225 | countNoFile += 1 226 | 227 | print "not found: ", countNoFile 228 | print "len(MATCHED_CID)", len(MATCHED_CID) 229 | print "len(MATCHED_PWID)", len(MATCHED_PWID) 230 | 231 | 232 | 233 | def doBruteMatching(): 234 | print "Starting brute force matching on remaining of matches" 235 | global MATCHED_PWID 236 | global MATCHED_CID 237 | connPW = sqlite3.connect(DB_PATH) # connecting 238 | connPW.text_factory = str 239 | cp = connPW.cursor() # creating cursor 240 | for i in commits: 241 | if i not in MATCHED_CID: 242 | commitTime = commits[i]["commitTime"] 243 | authorTime = commits[i]["authorTime"] 244 | count = 0 245 | for j in cp.execute("select pwid from lines where ? < date and date < ?",(authorTime,commitTime)): 246 | count += 1 247 | if j not in MATCHED_PWID and j[0] in patches: 248 | # print j[0], authorTime, commitTime 249 | # print patches[j[0]]["time"] 250 | compareDiffs(i,j[0],.5) 251 | # print count 252 | 253 | 254 | def compareDiffs(cid,pwid,threshold): 255 | global MATCHED_PWID 256 | global MATCHED_CID 257 | global commits 258 | global patches 259 | doCheck = True 260 | 261 | # git diff:commits[cid]["lines"] 262 | if cid in commits: 263 | gitDiff = commits[cid]["lines"] 264 | else: 265 | doCheck = False 266 | # patch diff: patches[pwid]["lines"] 267 | if pwid in patches: 268 | pwDiff = patches[pwid]["lines"] 269 | else: 270 | # print pwid 271 | doCheck = False 272 | 273 | if doCheck: 274 | # print 'commits[cid]["time"],patches[pwid]["time"]', commits[cid]["time"], patches[pwid]["time"] 275 | # print commits[cid]["authorTime"], patches[pwid]["time"], commits[cid]["commitTime"] 276 | if patches[pwid]["time"] != "NULL": 277 | if int(commits[cid]["commitTime"]) < int(patches[pwid]["time"]) and int(patches[pwid]["time"]) > int(commits[cid]["authorTime"]) - 86400: 278 | doCheck = False 279 | 280 | if doCheck: 281 | sm=difflib.SequenceMatcher(None,gitDiff,pwDiff) 282 | ratio = sm.ratio() 283 | if ratio > threshold: 284 | MATCHED_CID.add(cid) 285 | MATCHED_PWID.add(pwid) 286 | # if len(MATCHED_CID) % 10 == 0: 287 | # print len(MATCHED_CID) 288 | # print cid, pwid, ratio 289 | if cid not in lineMatches: 290 | lineMatches[cid] = set([]) 291 | lineMatches[cid].add(pwid) 292 | 293 | # remove and pwid from the dict 294 | del patches[pwid] 295 | 296 | 297 | 298 | if __name__ == '__main__': 299 | start = time.time() 300 | 301 | 302 | if sys.argv > 1: 303 | if "test" in sys.argv: 304 | OUTPUT ='/home-students/courouble/email2git_data/test_results/LINES_OUTPUT.txt' 305 | else: 306 | OUTPUT ='/home-students/courouble/email2git_data/LINES_OUTPUT.txt' 307 | 308 | getLineMatches() 309 | 310 | print "MATCHED_PWID:", len(MATCHED_PWID) 311 | print "MATCHED_CID:", len(MATCHED_CID) 312 | 313 | # print "Subject commit matched: ", len(subjectMatches) 314 | # print "lines commit matched: ", len(lineMatches) 315 | # print "subject commits total: ", subjectMatchesCount 316 | print "len(patches)", len(patches) 317 | print "len(commits)", len(commits) 318 | 319 | print "Created in", time.time() - start 320 | 321 | # with open("subjectMatches.txt") as f: 322 | # f.write(pickle.dumps(subjectMatches)) 323 | 324 | with open(OUTPUT,"w") as f: 325 | f.write(pickle.dumps(lineMatches)) 326 | -------------------------------------------------------------------------------- /lines_data_prep/git_prep.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | 20 | Reading the a customised git log output 21 | 22 | To get files touched by commits: 23 | git log --no-merges --name-only --pretty=oneline --after="2009-01-01" > ~/Desktop/email2git_data/raw_data/git_file_map.txt 24 | git log --no-merges --name-only --pretty=oneline --after="2009-01-01" > /home-students/courouble/email2git_data/raw_data/git_file_map.txt 25 | 26 | To get the commit diffs: 27 | git log -p --no-merges --pretty=format:"%H;%an;%ae;%ct;%at" --after="2009-01-01" > ~/Desktop/email2git_data/raw_data/commits_short.txt 28 | git log -p --no-merges --pretty=format:"%H;%an;%ae;%ct;%at" --after="2009-01-01" > /home-students/courouble/email2git_data/raw_data/commits_short.txt 29 | 30 | """ 31 | import re 32 | import cPickle as pickle 33 | import sys 34 | import sqlite3 35 | 36 | class Commit: 37 | def __innit__(self,cid,name,email,lines,commitTime, authorTime): 38 | self.cid = cid 39 | self.name = name 40 | self.email = email 41 | self.lines = lines 42 | self.commitTime = commitTime 43 | self.authorTime = authorTime 44 | 45 | # MATCHED_CID_INPUT = '/Users/alexandrecourouble/Desktop/email2git_data/subject_ouput/matched_cid_pickled.txt' 46 | MATCHED_CID_INPUT = '/home-students/courouble/email2git_data/subject_ouput/matched_cid_pickled.txt' 47 | 48 | # INPUT_CID_FILE_MAP = "/Users/alexandrecourouble/Desktop/email2git_data/raw_data/git_file_map.txt" 49 | INPUT_CID_FILE_MAP = "/home-students/courouble/email2git_data/raw_data/git_file_map.txt" 50 | # INPUT_COMMIT_FILE = "/Users/alexandrecourouble/Desktop/email2git_data/raw_data/commits_short.txt" 51 | INPUT_COMMIT_FILE = "/home-students/courouble/email2git_data/raw_data/commits_short.txt" 52 | # PERSONS_DB = '/Users/alexandrecourouble/Desktop/email2git_data/linux_persons.db' 53 | PERSONS_DB = '/home-students/courouble/email2git_data/linux_persons.db' 54 | 55 | # OUTPUT_MAP = "/Users/alexandrecourouble/Desktop/email2git_data/COMMIT_MAP_PICKLED.txt" 56 | OUTPUT_MAP = "/home-students/courouble/email2git_data/COMMIT_MAP_PICKLED.txt" 57 | # OUTPUT_COMMITS = "/Users/alexandrecourouble/Desktop/email2git_data/COMMITS_PICKLED_test.txt" 58 | OUTPUT_COMMITS = "/home-students/courouble/email2git_data/COMMITS_PICKLED_test.txt" 59 | 60 | COMMIT_FILE_MAP = {} 61 | 62 | COMMITS = {} 63 | 64 | persons = {} 65 | 66 | def readCIDMap(): 67 | with open(INPUT_CID_FILE_MAP) as f: 68 | currentCommit = "" 69 | for i in f: 70 | line = i.strip("\n") 71 | if re.match(r'\b[0-9a-f]{40}\b', line): 72 | currentCommit = line[:40] 73 | if currentCommit not in MATCHED_CID: 74 | COMMIT_FILE_MAP[currentCommit] = [] 75 | else: 76 | if currentCommit in COMMIT_FILE_MAP: 77 | COMMIT_FILE_MAP[currentCommit].append(line) 78 | 79 | 80 | def readCommits(): 81 | unique_people_id_email = {} 82 | unique_people_id_name = {} 83 | # reading persons DB 84 | conn = sqlite3.connect(PERSONS_DB) 85 | c = conn.cursor() 86 | for i in c.execute("SELECT personid, name, lcemail FROM persons"): 87 | email = i[2].replace("<","").replace(">","") 88 | unique_people_id_email[email] = i[0] 89 | unique_people_id_name[i[1]] = i[0] 90 | conn.close() 91 | 92 | with open(INPUT_COMMIT_FILE) as f: 93 | cid = "" 94 | authorName = "" 95 | authorEmail = "" 96 | commitTime = "" 97 | authorTime = "" 98 | personid = "" 99 | lines = [] 100 | 101 | for i in f: 102 | line = i.strip("\n") 103 | if re.match(r'\b[0-9a-f]{40}\b', line): 104 | # submit previously found data 105 | if cid != "" and cid not in MATCHED_CID: 106 | # COMMITS[cid] = Commit(cid,authorName,authorEmail,lines) 107 | COMMITS[cid] = {"name":authorName,"email":authorEmail,"lines":lines,"commitTime":commitTime,"authorTime":authorTime,"personid":personid} 108 | # create next commit and reset LINES 109 | split = line.split(";") 110 | cid = split[0] 111 | authorName = split[1] 112 | authorEmail = split[2] 113 | commitTime = split[3] 114 | authorTime = split[4] 115 | if authorEmail in unique_people_id_email: 116 | personid = unique_people_id_email[authorEmail] 117 | elif authorName in unique_people_id_name: 118 | personid = unique_people_id_name[authorName] 119 | else: 120 | personid = authorEmail 121 | lines = [] 122 | elif line.startswith("+++") or line.startswith("---"): 123 | pass 124 | elif line.startswith("+") or line.startswith("-"): 125 | lines.append(line) 126 | 127 | 128 | if __name__ == '__main__': 129 | 130 | if sys.argv > 1: 131 | if "test" not in sys.argv: 132 | 133 | # opening matched cid set from subject matching 134 | with open(MATCHED_CID_INPUT) as f: 135 | MATCHED_CID = pickle.load(f) 136 | print "Read", len(MATCHED_CID), "subject matched cid. Type:" , type(MATCHED_CID) 137 | 138 | elif "test" in sys.argv: 139 | MATCHED_CID = set([]) 140 | 141 | readCIDMap() 142 | readCommits() 143 | 144 | with open(OUTPUT_MAP,"w") as f: 145 | f.write(pickle.dumps(COMMIT_FILE_MAP)) 146 | 147 | with open(OUTPUT_COMMITS,"w") as f: 148 | f.write(pickle.dumps(COMMITS)) 149 | 150 | 151 | print COMMIT_FILE_MAP 152 | print len(COMMIT_FILE_MAP) 153 | -------------------------------------------------------------------------------- /lines_data_prep/pw_prep.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | 20 | 21 | how to get the data from db: 22 | mysql -u gitmine -p -h 172.30.200.1 KORG_PATCHWORK -e "select id, submitter_id, content from patchwork_patch where date > '2017-01-01';" > patches_short.txt 23 | 24 | mysql -u gitmine -p -h 172.30.200.1 KORG_PATCHWORK -e "select id, submitter_id, content from patchwork_patch where date > '2009-01-01';" > patches_short.txt 25 | 26 | This script will go through the output from the wuery and create a lighter file containing the info required for the line-based matching. 27 | 28 | 29 | """ 30 | # try: 31 | import cPickle as pickle 32 | import sqlite3 33 | import sys 34 | # except: 35 | # import pickle 36 | 37 | # reading pwid that were already matched (no need to include them) 38 | 39 | 40 | # MATCHED_PWID_INPUT = '/Users/alexandrecourouble/Desktop/email2git_data/subject_ouput_tmp/matched_pwid_pickled.txt' 41 | 42 | # INPUT_FILE_PATH = '/Users/alexandrecourouble/Desktop/email2git_data/raw_data/patches_short_time.txt' 43 | # NAME_MAP_PATH = '/Users/alexandrecourouble/Desktop/email2git_data/raw_data/name_map_short.txt' 44 | # PERSONS_DB = '/Users/alexandrecourouble/Desktop/email2git_data/linux_persons.db' 45 | 46 | # OUTPUT_PATH = '/Users/alexandrecourouble/Desktop/email2git_data/PATCHES_PICKLED_test.txt' 47 | # DB_PATH = "/Users/alexandrecourouble/Desktop/email2git_data/lookupDB.db" 48 | 49 | 50 | MATCHED_PWID_INPUT = '/home-students/courouble/email2git_data/subject_ouput/matched_pwid_pickled.txt' 51 | 52 | INPUT_FILE_PATH = '/home-students/courouble/email2git_data/raw_data/patches_short_time.txt' 53 | NAME_MAP_PATH = '/home-students/courouble/email2git_data/raw_data/name_map_short.txt' 54 | PERSONS_DB = '/home-students/courouble/email2git_data/linux_persons.db' 55 | 56 | OUTPUT_PATH = '/home-students/courouble/email2git_data/PATCHES_PICKLED_test.txt' 57 | DB_PATH = "/home-students/courouble/email2git_data/lookupDB.db" 58 | 59 | 60 | PATCHES = {} 61 | PEOPLE = {} 62 | 63 | 64 | class Patch: 65 | def __init__(self, pwid, author, files, time, lines): 66 | self.pwid = pwid 67 | self.author = author 68 | self.files = files 69 | self.time = time 70 | self.lines = lines 71 | 72 | 73 | class People: 74 | def __init__(self, pid, name, email, personid): 75 | self.pid = pid 76 | self.name = name 77 | self.email = email 78 | self.personid = personid # personid found in DMG's people DB 79 | 80 | 81 | def getPeople(): 82 | unique_people_id_email = {} 83 | unique_people_id_name = {} 84 | # reading persons DB 85 | conn = sqlite3.connect(PERSONS_DB) 86 | c = conn.cursor() 87 | for i in c.execute("SELECT personid, name, lcemail FROM persons"): 88 | email = i[2].replace("<","").replace(">","") 89 | unique_people_id_email[email] = i[0] 90 | unique_people_id_name[i[1]] = i[0] 91 | conn.close() 92 | 93 | 94 | with open(NAME_MAP_PATH) as f: 95 | for i in f: 96 | personid = "" 97 | split = i.strip("\n").split("\t") 98 | if split[1] in unique_people_id_email: 99 | personid = unique_people_id_email[split[1]] 100 | elif split[2] in unique_people_id_name: 101 | personid = unique_people_id_name[split[2]] 102 | else: 103 | personid = split[1] 104 | 105 | # id, name, email, personid 106 | PEOPLE[split[0]] = People(split[0],split[2],split[1],personid) 107 | 108 | 109 | def readDataFile(): 110 | global PATCHES 111 | global PEOPLE 112 | # limit = 0 113 | countNoAuthor = 0 114 | with open(INPUT_FILE_PATH) as f: 115 | for i in f: 116 | # limit += 1 117 | split = i.split("\t") 118 | # print split 119 | if len(split) > 1: 120 | # data points: 121 | pwid = split[0] 122 | if split[1] in PEOPLE: 123 | author = PEOPLE[split[1]] 124 | else: 125 | author = People("","","","") 126 | countNoAuthor += 1 127 | time = split[2] 128 | files = [] 129 | lines = [] 130 | 131 | if pwid not in MATCHED_PWID: 132 | a = 'h' 133 | for j in split[3].split("\\n"): 134 | if j.startswith("+++"): # get file name 135 | files.append(j.replace("+++ b/","")) 136 | elif j.startswith("---"): 137 | pass 138 | elif j.startswith("+") or j.startswith("-"): 139 | lines.append(j) 140 | 141 | PATCHES[pwid] = Patch(pwid, author, files, time, lines) 142 | 143 | # if limit == 20: break 144 | print "Could not find: ", countNoAuthor 145 | 146 | 147 | if __name__ == '__main__': 148 | 149 | if sys.argv > 1: 150 | if "test" not in sys.argv: 151 | # opening matched pwid set from subject matching 152 | with open(MATCHED_PWID_INPUT) as f: 153 | MATCHED_PWID = pickle.load(f) 154 | print "Read", len(MATCHED_PWID), "subject matched PWID. Type:" , type(MATCHED_PWID) 155 | elif "test" in sys.argv: 156 | MATCHED_PWID = set([]) 157 | 158 | 159 | getPeople() 160 | readDataFile() 161 | 162 | out = {} 163 | table = [] 164 | for i in PATCHES: 165 | 166 | out[i] = {} 167 | out[i]["lines"] = PATCHES[i].lines 168 | out[i]["files"] = PATCHES[i].files 169 | out[i]["time"] = PATCHES[i].time 170 | out[i]["author"] = PATCHES[i].author.personid 171 | 172 | table.append((int(i),PATCHES[i].time)) 173 | 174 | 175 | # print out 176 | with open(OUTPUT_PATH,"w") as f: 177 | f.write(pickle.dumps(out)) 178 | 179 | # creating db for faster pwid lookup 180 | 181 | conn = sqlite3.connect(DB_PATH) # connecting 182 | conn.text_factory = str 183 | c = conn.cursor() #creating cursor 184 | c.execute('''create table lines(pwid int, date date)''') # creating the table 185 | c.executemany('INSERT INTO lines VALUES (?,?)', table) # importing all the data 186 | conn.commit() 187 | conn.close() # we're done! 188 | 189 | 190 | 191 | 192 | 193 | -------------------------------------------------------------------------------- /subject_data_gen/git/commit_subject_generator.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | 20 | 21 | 22 | This script will create a database containing the first line of the commit message, the commit ID, and the commit time. 23 | 24 | How to run: 25 | enter the follwoing command in the linux git repo. The path should be the path to this script. The date is the lower limit of the git log data 26 | local execution command: 27 | git log --no-merges --pretty=format:"%H %ct %s" --after={2009-01-01} | python /Users/alexandrecourouble/Desktop/email2git/subject_data_gen/git/commit_subject_generator.py 28 | 29 | server command: 30 | git log --no-merges --pretty=format:"%H %ct %s" --after={2009-01-01} | python /home-students/courouble/email2git/subject_data_gen/git/commit_subject_generator.py 31 | 32 | by Alex Courouble 33 | """ 34 | import os 35 | import sys, sqlite3 36 | DB_PATH = '/home-students/courouble/email2git_data/commit_subject.db' 37 | 38 | # getting data from stdin and storing it in a list of tuples before populating sqlite DB 39 | data = sys.stdin.read() 40 | lines = data.split('\n') 41 | table = [] 42 | for i in lines: 43 | split = i.split(' ') 44 | if len(split) < 4: 45 | table.append(tuple(split)) 46 | 47 | 48 | 49 | if os.path.isfile(DB_PATH): 50 | os.remove(DB_PATH) 51 | 52 | 53 | # creating a connection to the DB and inputing the data 54 | conn = sqlite3.connect(DB_PATH) # connecting 55 | conn.text_factory = str 56 | c = conn.cursor() #creating cursor 57 | c.execute('''create table subject(cid char(40), commit_time int, subject text)''') # creating the table 58 | c.executemany('INSERT INTO subject VALUES (?,?,?)', table) # importing all the data 59 | c.execute('CREATE INDEX subjectidx on subject(subject,commit_time)') 60 | c.execute('CREATE INDEX subjectidx2 on subject(cid)') 61 | conn.commit() 62 | conn.close() # we're done! 63 | 64 | -------------------------------------------------------------------------------- /subject_data_gen/patchwork/pwSubjectFull.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | 20 | 21 | Script used to create a DB from the patchwork email subject data dump. 22 | 23 | in PWDB: "select id, name from patchwork_patch" 24 | 25 | By Alex Courouble 26 | """ 27 | import sqlite3, os 28 | INPUT_FILE_PATH = '/home-students/courouble/email2git_data/raw_data/subject_pw.txt' 29 | DB_PATH = '/home-students/courouble/email2git_data/pwSubject_short.db' # output 30 | 31 | 32 | table = [] 33 | 34 | with open(INPUT_FILE_PATH,"r") as s: 35 | for i in s: 36 | patchData = i.split("\t") 37 | patchId = patchData[0] 38 | patchSubject = patchData[1] 39 | # print patchId," ",patchSubject 40 | table.append((patchId,patchSubject)) 41 | 42 | if os.path.isfile(DB_PATH): 43 | os.remove(DB_PATH) 44 | 45 | conn = sqlite3.connect(DB_PATH) # connecting 46 | conn.text_factory = str 47 | c = conn.cursor() #creating cursor 48 | c.execute('''create table subject(pwid int, subject text)''') # creating the table 49 | c.executemany('INSERT INTO subject VALUES (?,?)', table) # importing all the data 50 | conn.commit() 51 | conn.close() # we're done! -------------------------------------------------------------------------------- /subject_matching.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | """ 18 | 19 | try: 20 | import cPickle as pickle 21 | except: 22 | import pickle 23 | print "Could not import cPickle. Using regular pickle implementation. Expect performance loss." 24 | 25 | import sqlite3 26 | import sys 27 | 28 | 29 | PW_DB = '/home-students/courouble/email2git_data/pwSubject_short.db' 30 | GIT_DB = '/home-students/courouble/email2git_data/commit_subject.db' 31 | OUTPUT = '/home-students/courouble/email2git_data/subject_ouput/SUBJECT_OUTPUT.txt' 32 | 33 | MATCHED_CID_OUTPUT = '/home-students/courouble/email2git_data/subject_ouput/matched_cid_pickled.txt' 34 | MATCHED_PWID_OUTPUT = '/home-students/courouble/email2git_data/subject_ouput/matched_pwid_pickled.txt' 35 | 36 | 37 | subjectMatches = {} 38 | 39 | MATCHED_PWID = set([]) 40 | MATCHED_CID = set([]) 41 | 42 | def getSubjectMatches(): 43 | print "Finding subject matches" 44 | 45 | connGIT = sqlite3.connect(GIT_DB) # connecting 46 | connGIT.text_factory = str 47 | cg = connGIT.cursor() #creating cursor 48 | gitCommits = {} 49 | for row in cg.execute('select * from subject'): 50 | gitCommits[row[0]] = (row[1],row[2]) 51 | connGIT.close() 52 | 53 | connPW = sqlite3.connect(PW_DB) # connecting 54 | connPW.text_factory = str 55 | cp = connPW.cursor() # creating cursor 56 | 57 | for i in gitCommits: 58 | string = '%'+gitCommits[i][1]+'%' 59 | count = 0 60 | for j in cp.execute("""select * from subject where subject like ?;""", (string,)): 61 | # print str(j[0])+','+i 62 | if i not in subjectMatches: 63 | subjectMatches[i] = set([]) 64 | subjectMatches[i].add(str(j[0])) 65 | MATCHED_PWID.add(str(j[0])) 66 | MATCHED_CID.add(i) 67 | 68 | # printing remaining cids 69 | connPW.close() 70 | 71 | 72 | 73 | 74 | 75 | if sys.argv > 1: 76 | if "test" in sys.argv: 77 | OUTPUT ='/Users/alexandrecourouble/Desktop/email2git_data/test_results/SUBJECT_OUTPUT.txt' 78 | 79 | 80 | getSubjectMatches() 81 | 82 | print "Matched ", len(MATCHED_CID), "commits" 83 | print "Matched ", len(MATCHED_PWID), "patches" 84 | 85 | # Outputing data: 86 | # matched cids: 87 | with open(MATCHED_CID_OUTPUT, "w") as f: 88 | f.write(pickle.dumps(MATCHED_CID)) 89 | 90 | # matched pwid: 91 | with open(MATCHED_PWID_OUTPUT, "w") as f: 92 | f.write(pickle.dumps(MATCHED_PWID)) 93 | 94 | 95 | # MATCHES 96 | with open(OUTPUT, "w") as f: 97 | f.write(pickle.dumps(subjectMatches)) 98 | 99 | -------------------------------------------------------------------------------- /validation_tests/read_test_output.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | reading and comparing matches outputed by subject and line 20 | """ 21 | import pickle 22 | try: 23 | import cPickle as pickle 24 | except: 25 | import pickle 26 | print "Could not import cPickle. Using regular pickle implementation. Expect performance loss." 27 | 28 | 29 | subject_matches = "/Users/alexandrecourouble/Desktop/email2git_data/test_results/SUBJECT_OUTPUT.txt" 30 | lines_matches = '/Users/alexandrecourouble/Desktop/email2git_data/test_results/LINES_OUTPUT.txt' 31 | 32 | subject = {} 33 | subject_set = set([]) 34 | 35 | lines = {} 36 | lines_set = set([]) 37 | 38 | # reading subject matches 39 | with open(subject_matches) as f: 40 | subject = pickle.load(f) 41 | 42 | # populating set with unique match string (cid + pwid) 43 | nmb_s_matches = 0 44 | for i in subject: 45 | for j in subject[i]: 46 | subject_set.add(i+j) 47 | nmb_s_matches += 1 48 | 49 | with open(lines_matches) as f: 50 | lines = pickle.load(f) 51 | 52 | count = 0 53 | nmb_l_matches = 0 54 | for i in lines: 55 | for j in lines[i]: 56 | nmb_l_matches += 1 57 | if i+j in subject_set: 58 | count += 1 59 | 60 | print count 61 | 62 | print "Number of SUBJECT matches :", nmb_s_matches 63 | print "Number of LINES matches :", nmb_l_matches -------------------------------------------------------------------------------- /web/matches_to_files.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2017 Alex Courouble 3 | 4 | This program is free software: you can redistribute it and/or modify 5 | it under the terms of the GNU General Public License as published by 6 | the Free Software Foundation, either version 3 of the License, or 7 | (at your option) any later version. 8 | 9 | This program is distributed in the hope that it will be useful, 10 | but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | GNU General Public License for more details. 13 | 14 | You should have received a copy of the GNU General Public License 15 | 16 | along with this program. If not, see . 17 | 18 | 19 | 20 | Reads PICKLED matches from subject and line matches files 21 | 22 | Input: 23 | - LINES_OUTPUT.txt 24 | - SUBJECT_OUTPUT.txt 25 | - commit_dates.txt 26 | - patch_mailinglist_data.txt 27 | - mailing_lists_ids.txt 28 | """ 29 | 30 | LINE_MATCHES = "LINES_OUTPUT.txt" 31 | SUBJECT_MATCHES = "SUBJECT_OUTPUT.txt" 32 | 33 | CID_DATE = "commit_dates.txt" 34 | PATCH_MAILINGLIST = "mailing_list_data/patch_mailinglist_data.txt" 35 | ML_NAMES = "mailing_list_data/mailing_lists_ids.txt" 36 | 37 | OUTPUT = "/Users/alexandrecourouble/Documents/public_html/cregit_web/matches/" 38 | 39 | import cPickle as pickle 40 | 41 | class Match(): 42 | def __init__(self, cid, pwid, mailing_list, matchType, date): 43 | self.cid = cid 44 | self.pwid = pwid 45 | self.mailing_list = mailing_list 46 | self.matchType = matchType 47 | self.date = date 48 | 49 | 50 | def readMailingListData(): 51 | ml_map = {} 52 | patches_data = {} 53 | 54 | print "Reading Mailing List data" 55 | # Reading mailing lists ids 56 | with open(ML_NAMES) as f: 57 | for i in f: 58 | split = i.strip("\n").split(",") 59 | ml_map[split[0]] = split[1] 60 | 61 | 62 | # reading patches and their mailing list 63 | with open(PATCH_MAILINGLIST) as f: 64 | for i in f: 65 | split = i.strip("\n").split("\t") 66 | patches_data[split[0]] = (ml_map[split[1]],split[2]) 67 | 68 | return patches_data 69 | 70 | 71 | def readMatches(): 72 | print "Rading Matches" 73 | 74 | # new line matches 75 | with open(LINE_MATCHES) as f: 76 | parseMatches(pickle.load(f),"l") 77 | 78 | # new subject matches 79 | with open(SUBJECT_MATCHES) as f: 80 | parseMatches(pickle.load(f),"s") 81 | 82 | 83 | 84 | def parseMatches(matchesDict,matchType): 85 | print "Read", len(matchesDict),"line matches" 86 | 87 | for i in matchesDict: 88 | for j in matchesDict[i]: 89 | pwid = j 90 | cid = i 91 | mailing_list = patches_data[pwid][0] 92 | date = patches_data[pwid][1] 93 | # matches[i] = Match(cid,pwid,mailing_list,matchType,date) 94 | if i not in matches: 95 | matches[i] = set([]) 96 | matches[i].add(Match(cid,pwid,mailing_list,matchType,date)) 97 | 98 | 99 | 100 | if __name__ == '__main__': 101 | matches = {} 102 | 103 | patches_data = readMailingListData() 104 | 105 | readMatches() 106 | 107 | for i in matches: 108 | # creating file to write matches 109 | file_name = OUTPUT + i +".txt" 110 | with open(file_name,"w") as f: 111 | for j in matches[i]: 112 | lineToWrite = j.pwid+","+j.mailing_list+","+j.date+","+j.matchType+"\n" 113 | f.write(lineToWrite) 114 | 115 | 116 | 117 | 118 | --------------------------------------------------------------------------------