├── .flake8 ├── .pre-commit-config.yaml ├── LICENSE.txt ├── MANIFEST.in ├── README.md ├── assets ├── screenshot-TreeContext.svg └── screenshot-encoding.svg ├── grep_ast ├── __init__.py ├── dump.py ├── grep_ast.py ├── main.py ├── parsers.py └── tsl.py ├── requirements.txt ├── scripts ├── build-and-release.sh └── screenshot.sh ├── setup.py └── tests └── test_parsers.py /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = E203,W503 3 | max-line-length = 100 4 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pycqa/isort 3 | rev: 5.12.0 4 | hooks: 5 | - id: isort 6 | args: ["--profile", "black"] 7 | - repo: https://github.com/psf/black 8 | rev: 23.3.0 9 | hooks: 10 | - id: black 11 | args: ["--line-length", "100", "--preview"] 12 | - repo: https://github.com/pycqa/flake8 13 | rev: 6.0.0 14 | hooks: 15 | - id: flake8 16 | args: ["--show-source"] 17 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include requirements.txt 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # grep-ast 2 | 3 | Grep source code files and see matching lines with 4 | useful context that show how they fit into the code. 5 | See the loops, functions, methods, classes, etc 6 | that contain all the matching lines. 7 | Get a sense of what's inside a matched class or function definition. 8 | You see relevant code from every layer of the 9 | abstract syntax tree, above and below the matches. 10 | 11 | By default, grep-AST recurses the current directory to search all source code files. 12 | It respects `.gitignore`, so it will usually "do the right thing" in most repos 13 | if you just do `grep-ast ` without specifying any filenames. 14 | 15 | You can also invoke `grep-ast` as `gast` for convenience. 16 | 17 | Grep-AST is built with [tree-sitter](https://tree-sitter.github.io/tree-sitter/) and 18 | [tree-sitter-languages](https://github.com/grantjenks/py-tree-sitter-languages). 19 | So it supports a lot of popular [code languages](https://github.com/paul-gauthier/grep-ast/blob/main/grep_ast/parsers.py). 20 | 21 | ## Install 22 | 23 | ```bash 24 | python -m pip install git+https://github.com/paul-gauthier/grep-ast.git 25 | ``` 26 | 27 | ## Usage 28 | 29 | Basic usage: 30 | 31 | ```bash 32 | grep-ast [pattern] [filenames...] 33 | ``` 34 | 35 | Full options list: 36 | 37 | ``` 38 | usage: grep_ast.py [-h] [-i] [--color] [--no-color] [--encoding ENCODING] [--languages] [--verbose] 39 | [pat] [filenames ...] 40 | 41 | positional arguments: 42 | pat the pattern to search for 43 | filenames the files to display 44 | 45 | options: 46 | -h, --help show this help message and exit 47 | -i, --ignore-case ignore case distinctions 48 | --color force color printing 49 | --no-color disable color printing 50 | --encoding ENCODING file encoding 51 | --languages print the parsers table 52 | --verbose enable verbose output 53 | ``` 54 | 55 | ## Examples 56 | 57 | Here we search for **"encoding"** in the source to this tool. 58 | These results mainly highlight how `grep-ast` 59 | shows you how the matches fit into the code base. 60 | 61 | aider screencast 62 | 63 | Here we search for **"TreeContext"** in the source to this tool. 64 | These results mainly highlight how `grep-ast` 65 | helps you understand the *contents* of a matching 66 | named code block (class, function, method, etc). 67 | 68 | aider screencast 69 | -------------------------------------------------------------------------------- /assets/screenshot-TreeContext.svg: -------------------------------------------------------------------------------- 1 | 2 | 12 | 13 | $ 14 | g 15 | r 16 | e 17 | p 18 | - 19 | a 20 | s 21 | t 22 | T 23 | r 24 | e 25 | e 26 | C 27 | o 28 | n 29 | t 30 | e 31 | x 32 | t 33 | g 34 | r 35 | e 36 | p 37 | _ 38 | a 39 | s 40 | t 41 | / 42 | _ 43 | _ 44 | i 45 | n 46 | i 47 | t 48 | _ 49 | _ 50 | . 51 | p 52 | y 53 | : 54 | │ 55 | # 56 | n 57 | o 58 | q 59 | a 60 | : 61 | F 62 | 4 63 | 0 64 | 1 65 | │ 66 | █ 67 | f 68 | r 69 | o 70 | m 71 | . 72 | g 73 | r 74 | e 75 | p 76 | _ 77 | a 78 | s 79 | t 80 | i 81 | m 82 | p 83 | o 84 | r 85 | t 86 | T 87 | r 88 | e 89 | e 90 | C 91 | o 92 | n 93 | t 94 | e 95 | x 96 | t 97 | │ 98 | f 99 | r 100 | o 101 | m 102 | . 103 | p 104 | a 105 | r 106 | s 107 | e 108 | r 109 | s 110 | i 111 | m 112 | p 113 | o 114 | r 115 | t 116 | f 117 | i 118 | l 119 | e 120 | n 121 | a 122 | m 123 | e 124 | _ 125 | t 126 | o 127 | _ 128 | l 129 | a 130 | n 131 | g 132 | g 133 | r 134 | e 135 | p 136 | _ 137 | a 138 | s 139 | t 140 | / 141 | g 142 | r 143 | e 144 | p 145 | _ 146 | a 147 | s 148 | t 149 | . 150 | p 151 | y 152 | : 153 | │ 154 | # 155 | ! 156 | / 157 | u 158 | s 159 | r 160 | / 161 | b 162 | i 163 | n 164 | / 165 | e 166 | n 167 | v 168 | p 169 | y 170 | t 171 | h 172 | o 173 | n 174 | │ 175 | │ 176 | i 177 | m 178 | p 179 | o 180 | r 181 | t 182 | r 183 | e 184 | │ 185 | ⋮ 186 | . 187 | . 188 | . 189 | │ 190 | █ 191 | c 192 | l 193 | a 194 | s 195 | s 196 | T 197 | r 198 | e 199 | e 200 | C 201 | o 202 | n 203 | t 204 | e 205 | x 206 | t 207 | : 208 | │ 209 | d 210 | e 211 | f 212 | _ 213 | _ 214 | i 215 | n 216 | i 217 | t 218 | _ 219 | _ 220 | ( 221 | │ 222 | s 223 | e 224 | l 225 | f 226 | , 227 | │ 228 | f 229 | i 230 | l 231 | e 232 | n 233 | a 234 | m 235 | e 236 | , 237 | │ 238 | c 239 | o 240 | d 241 | e 242 | , 243 | │ 244 | c 245 | o 246 | l 247 | o 248 | r 249 | = 250 | F 251 | a 252 | l 253 | s 254 | e 255 | , 256 | │ 257 | v 258 | e 259 | r 260 | b 261 | o 262 | s 263 | e 264 | = 265 | F 266 | a 267 | l 268 | s 269 | e 270 | , 271 | │ 272 | l 273 | i 274 | n 275 | e 276 | _ 277 | n 278 | u 279 | m 280 | b 281 | e 282 | r 283 | = 284 | F 285 | a 286 | l 287 | s 288 | e 289 | , 290 | │ 291 | p 292 | a 293 | r 294 | e 295 | n 296 | t 297 | _ 298 | c 299 | o 300 | n 301 | t 302 | e 303 | x 304 | t 305 | = 306 | T 307 | r 308 | u 309 | e 310 | , 311 | │ 312 | c 313 | h 314 | i 315 | l 316 | d 317 | _ 318 | c 319 | o 320 | n 321 | t 322 | e 323 | x 324 | t 325 | = 326 | T 327 | r 328 | u 329 | e 330 | , 331 | │ 332 | l 333 | a 334 | s 335 | t 336 | _ 337 | l 338 | i 339 | n 340 | e 341 | = 342 | T 343 | r 344 | u 345 | e 346 | , 347 | ⋮ 348 | . 349 | . 350 | . 351 | │ 352 | s 353 | e 354 | l 355 | f 356 | . 357 | f 358 | i 359 | l 360 | e 361 | n 362 | a 363 | m 364 | e 365 | = 366 | f 367 | i 368 | l 369 | e 370 | n 371 | a 372 | m 373 | e 374 | ⋮ 375 | . 376 | . 377 | . 378 | │ 379 | f 380 | o 381 | r 382 | i 383 | i 384 | n 385 | r 386 | a 387 | n 388 | g 389 | e 390 | ( 391 | s 392 | e 393 | l 394 | f 395 | . 396 | n 397 | u 398 | m 399 | _ 400 | l 401 | i 402 | n 403 | e 404 | s 405 | ) 406 | : 407 | │ 408 | h 409 | e 410 | a 411 | d 412 | e 413 | r 414 | = 415 | s 416 | o 417 | r 418 | t 419 | e 420 | d 421 | ( 422 | s 423 | e 424 | l 425 | f 426 | . 427 | h 428 | e 429 | a 430 | d 431 | e 432 | r 433 | [ 434 | i 435 | ] 436 | ) 437 | ⋮ 438 | . 439 | . 440 | . 441 | │ 442 | s 443 | e 444 | l 445 | f 446 | . 447 | h 448 | e 449 | a 450 | d 451 | e 452 | r 453 | [ 454 | i 455 | ] 456 | = 457 | h 458 | e 459 | a 460 | d 461 | _ 462 | s 463 | t 464 | a 465 | r 466 | t 467 | , 468 | h 469 | e 470 | a 471 | d 472 | _ 473 | e 474 | n 475 | d 476 | │ 477 | ⋮ 478 | . 479 | . 480 | . 481 | │ 482 | r 483 | e 484 | t 485 | u 486 | r 487 | n 488 | │ 489 | ⋮ 490 | . 491 | . 492 | . 493 | │ 494 | d 495 | e 496 | f 497 | a 498 | d 499 | d 500 | _ 501 | c 502 | o 503 | n 504 | t 505 | e 506 | x 507 | t 508 | ( 509 | s 510 | e 511 | l 512 | f 513 | ) 514 | : 515 | │ 516 | i 517 | f 518 | n 519 | o 520 | t 521 | s 522 | e 523 | l 524 | f 525 | . 526 | l 527 | i 528 | n 529 | e 530 | s 531 | _ 532 | o 533 | f 534 | _ 535 | i 536 | n 537 | t 538 | e 539 | r 540 | e 541 | s 542 | t 543 | : 544 | ⋮ 545 | . 546 | . 547 | . 548 | │ 549 | s 550 | e 551 | l 552 | f 553 | . 554 | c 555 | l 556 | o 557 | s 558 | e 559 | _ 560 | s 561 | m 562 | a 563 | l 564 | l 565 | _ 566 | g 567 | a 568 | p 569 | s 570 | ( 571 | ) 572 | │ 573 | │ 574 | d 575 | e 576 | f 577 | a 578 | d 579 | d 580 | _ 581 | c 582 | h 583 | i 584 | l 585 | d 586 | _ 587 | c 588 | o 589 | n 590 | t 591 | e 592 | x 593 | t 594 | ( 595 | s 596 | e 597 | l 598 | f 599 | , 600 | i 601 | ) 602 | : 603 | │ 604 | i 605 | f 606 | n 607 | o 608 | t 609 | s 610 | e 611 | l 612 | f 613 | . 614 | n 615 | o 616 | d 617 | e 618 | s 619 | [ 620 | i 621 | ] 622 | : 623 | ⋮ 624 | . 625 | . 626 | . 627 | │ 628 | f 629 | o 630 | r 631 | c 632 | h 633 | i 634 | l 635 | d 636 | i 637 | n 638 | c 639 | h 640 | i 641 | l 642 | d 643 | r 644 | e 645 | n 646 | : 647 | │ 648 | i 649 | f 650 | l 651 | e 652 | n 653 | ( 654 | s 655 | e 656 | l 657 | f 658 | . 659 | s 660 | h 661 | o 662 | w 663 | _ 664 | l 665 | i 666 | n 667 | e 668 | s 669 | ) 670 | > 671 | c 672 | u 673 | r 674 | r 675 | e 676 | n 677 | t 678 | l 679 | y 680 | _ 681 | s 682 | h 683 | o 684 | w 685 | i 686 | n 687 | g 688 | + 689 | m 690 | a 691 | x 692 | _ 693 | t 694 | o 695 | _ 696 | s 697 | h 698 | o 699 | w 700 | : 701 | ⋮ 702 | . 703 | . 704 | . 705 | │ 706 | s 707 | e 708 | l 709 | f 710 | . 711 | a 712 | d 713 | d 714 | _ 715 | p 716 | a 717 | r 718 | e 719 | n 720 | t 721 | _ 722 | s 723 | c 724 | o 725 | p 726 | e 727 | s 728 | ( 729 | c 730 | h 731 | i 732 | l 733 | d 734 | _ 735 | s 736 | t 737 | a 738 | r 739 | t 740 | _ 741 | l 742 | i 743 | n 744 | e 745 | ) 746 | │ 747 | ⋮ 748 | . 749 | . 750 | . 751 | │ 752 | d 753 | e 754 | f 755 | c 756 | l 757 | o 758 | s 759 | e 760 | _ 761 | s 762 | m 763 | a 764 | l 765 | l 766 | _ 767 | g 768 | a 769 | p 770 | s 771 | ( 772 | s 773 | e 774 | l 775 | f 776 | ) 777 | : 778 | ⋮ 779 | . 780 | . 781 | . 782 | │ 783 | c 784 | l 785 | o 786 | s 787 | e 788 | d 789 | _ 790 | s 791 | h 792 | o 793 | w 794 | = 795 | s 796 | e 797 | t 798 | ( 799 | s 800 | e 801 | l 802 | f 803 | . 804 | s 805 | h 806 | o 807 | w 808 | _ 809 | l 810 | i 811 | n 812 | e 813 | s 814 | ) 815 | ⋮ 816 | . 817 | . 818 | . 819 | │ 820 | s 821 | e 822 | l 823 | f 824 | . 825 | s 826 | h 827 | o 828 | w 829 | _ 830 | l 831 | i 832 | n 833 | e 834 | s 835 | = 836 | c 837 | l 838 | o 839 | s 840 | e 841 | d 842 | _ 843 | s 844 | h 845 | o 846 | w 847 | │ 848 | │ 849 | d 850 | e 851 | f 852 | f 853 | o 854 | r 855 | m 856 | a 857 | t 858 | ( 859 | s 860 | e 861 | l 862 | f 863 | ) 864 | : 865 | │ 866 | i 867 | f 868 | n 869 | o 870 | t 871 | s 872 | e 873 | l 874 | f 875 | . 876 | s 877 | h 878 | o 879 | w 880 | _ 881 | l 882 | i 883 | n 884 | e 885 | s 886 | : 887 | ⋮ 888 | . 889 | . 890 | . 891 | │ 892 | f 893 | o 894 | r 895 | i 896 | , 897 | l 898 | i 899 | n 900 | e 901 | i 902 | n 903 | e 904 | n 905 | u 906 | m 907 | e 908 | r 909 | a 910 | t 911 | e 912 | ( 913 | s 914 | e 915 | l 916 | f 917 | . 918 | l 919 | i 920 | n 921 | e 922 | s 923 | ) 924 | : 925 | │ 926 | i 927 | f 928 | i 929 | n 930 | o 931 | t 932 | i 933 | n 934 | s 935 | e 936 | l 937 | f 938 | . 939 | s 940 | h 941 | o 942 | w 943 | _ 944 | l 945 | i 946 | n 947 | e 948 | s 949 | : 950 | │ 951 | i 952 | f 953 | d 954 | o 955 | t 956 | s 957 | : 958 | │ 959 | i 960 | f 961 | s 962 | e 963 | l 964 | f 965 | . 966 | l 967 | i 968 | n 969 | e 970 | _ 971 | n 972 | u 973 | m 974 | b 975 | e 976 | r 977 | : 978 | │ 979 | o 980 | u 981 | t 982 | p 983 | u 984 | t 985 | + 986 | = 987 | " 988 | . 989 | . 990 | . 991 | ⋮ 992 | . 993 | . 994 | . 995 | \ 996 | n 997 | " 998 | │ 999 | e 1000 | l 1001 | s 1002 | e 1003 | : 1004 | │ 1005 | o 1006 | u 1007 | t 1008 | p 1009 | u 1010 | t 1011 | + 1012 | = 1013 | " 1014 | ⋮ 1015 | . 1016 | . 1017 | . 1018 | \ 1019 | n 1020 | " 1021 | │ 1022 | d 1023 | o 1024 | t 1025 | s 1026 | = 1027 | F 1028 | a 1029 | l 1030 | s 1031 | e 1032 | ⋮ 1033 | . 1034 | . 1035 | . 1036 | │ 1037 | d 1038 | o 1039 | t 1040 | s 1041 | = 1042 | T 1043 | r 1044 | u 1045 | e 1046 | │ 1047 | │ 1048 | r 1049 | e 1050 | t 1051 | u 1052 | r 1053 | n 1054 | o 1055 | u 1056 | t 1057 | p 1058 | u 1059 | t 1060 | │ 1061 | ⋮ 1062 | . 1063 | . 1064 | . 1065 | │ 1066 | d 1067 | e 1068 | f 1069 | w 1070 | a 1071 | l 1072 | k 1073 | _ 1074 | t 1075 | r 1076 | e 1077 | e 1078 | ( 1079 | s 1080 | e 1081 | l 1082 | f 1083 | , 1084 | n 1085 | o 1086 | d 1087 | e 1088 | , 1089 | d 1090 | e 1091 | p 1092 | t 1093 | h 1094 | = 1095 | 0 1096 | ) 1097 | : 1098 | │ 1099 | s 1100 | t 1101 | a 1102 | r 1103 | t 1104 | = 1105 | n 1106 | o 1107 | d 1108 | e 1109 | . 1110 | s 1111 | t 1112 | a 1113 | r 1114 | t 1115 | _ 1116 | p 1117 | o 1118 | i 1119 | n 1120 | t 1121 | ⋮ 1122 | . 1123 | . 1124 | . 1125 | │ 1126 | r 1127 | e 1128 | t 1129 | u 1130 | r 1131 | n 1132 | s 1133 | t 1134 | a 1135 | r 1136 | t 1137 | _ 1138 | l 1139 | i 1140 | n 1141 | e 1142 | , 1143 | e 1144 | n 1145 | d 1146 | _ 1147 | l 1148 | i 1149 | n 1150 | e 1151 | g 1152 | r 1153 | e 1154 | p 1155 | _ 1156 | a 1157 | s 1158 | t 1159 | / 1160 | m 1161 | a 1162 | i 1163 | n 1164 | . 1165 | p 1166 | y 1167 | : 1168 | │ 1169 | # 1170 | ! 1171 | / 1172 | u 1173 | s 1174 | r 1175 | / 1176 | b 1177 | i 1178 | n 1179 | / 1180 | e 1181 | n 1182 | v 1183 | p 1184 | y 1185 | t 1186 | h 1187 | o 1188 | n 1189 | │ 1190 | │ 1191 | i 1192 | m 1193 | p 1194 | o 1195 | r 1196 | t 1197 | a 1198 | r 1199 | g 1200 | p 1201 | a 1202 | r 1203 | s 1204 | e 1205 | ⋮ 1206 | . 1207 | . 1208 | . 1209 | │ 1210 | f 1211 | r 1212 | o 1213 | m 1214 | . 1215 | d 1216 | u 1217 | m 1218 | p 1219 | i 1220 | m 1221 | p 1222 | o 1223 | r 1224 | t 1225 | d 1226 | u 1227 | m 1228 | p 1229 | # 1230 | n 1231 | o 1232 | q 1233 | a 1234 | : 1235 | F 1236 | 4 1237 | 0 1238 | 1 1239 | █ 1240 | f 1241 | r 1242 | o 1243 | m 1244 | . 1245 | g 1246 | r 1247 | e 1248 | p 1249 | _ 1250 | a 1251 | s 1252 | t 1253 | i 1254 | m 1255 | p 1256 | o 1257 | r 1258 | t 1259 | T 1260 | r 1261 | e 1262 | e 1263 | C 1264 | o 1265 | n 1266 | t 1267 | e 1268 | x 1269 | t 1270 | │ 1271 | f 1272 | r 1273 | o 1274 | m 1275 | . 1276 | p 1277 | a 1278 | r 1279 | s 1280 | e 1281 | r 1282 | s 1283 | i 1284 | m 1285 | p 1286 | o 1287 | r 1288 | t 1289 | P 1290 | A 1291 | R 1292 | S 1293 | E 1294 | R 1295 | S 1296 | │ 1297 | ⋮ 1298 | . 1299 | . 1300 | . 1301 | │ 1302 | d 1303 | e 1304 | f 1305 | p 1306 | r 1307 | o 1308 | c 1309 | e 1310 | s 1311 | s 1312 | _ 1313 | f 1314 | i 1315 | l 1316 | e 1317 | n 1318 | a 1319 | m 1320 | e 1321 | ( 1322 | f 1323 | i 1324 | l 1325 | e 1326 | n 1327 | a 1328 | m 1329 | e 1330 | , 1331 | a 1332 | r 1333 | g 1334 | s 1335 | ) 1336 | : 1337 | │ 1338 | t 1339 | r 1340 | y 1341 | : 1342 | │ 1343 | w 1344 | i 1345 | t 1346 | h 1347 | o 1348 | p 1349 | e 1350 | n 1351 | ( 1352 | f 1353 | i 1354 | l 1355 | e 1356 | n 1357 | a 1358 | m 1359 | e 1360 | , 1361 | " 1362 | r 1363 | " 1364 | , 1365 | e 1366 | n 1367 | c 1368 | o 1369 | d 1370 | i 1371 | n 1372 | g 1373 | = 1374 | a 1375 | r 1376 | g 1377 | s 1378 | . 1379 | e 1380 | n 1381 | c 1382 | o 1383 | d 1384 | i 1385 | n 1386 | g 1387 | ) 1388 | a 1389 | s 1390 | f 1391 | i 1392 | l 1393 | e 1394 | : 1395 | │ 1396 | c 1397 | o 1398 | d 1399 | e 1400 | = 1401 | f 1402 | i 1403 | l 1404 | e 1405 | . 1406 | r 1407 | e 1408 | a 1409 | d 1410 | ( 1411 | ) 1412 | │ 1413 | e 1414 | x 1415 | c 1416 | e 1417 | p 1418 | t 1419 | U 1420 | n 1421 | i 1422 | c 1423 | o 1424 | d 1425 | e 1426 | D 1427 | e 1428 | c 1429 | o 1430 | d 1431 | e 1432 | E 1433 | r 1434 | r 1435 | o 1436 | r 1437 | : 1438 | ⋮ 1439 | . 1440 | . 1441 | . 1442 | │ 1443 | t 1444 | r 1445 | y 1446 | : 1447 | █ 1448 | t 1449 | c 1450 | = 1451 | T 1452 | r 1453 | e 1454 | e 1455 | C 1456 | o 1457 | n 1458 | t 1459 | e 1460 | x 1461 | t 1462 | ( 1463 | │ 1464 | f 1465 | i 1466 | l 1467 | e 1468 | n 1469 | a 1470 | m 1471 | e 1472 | , 1473 | c 1474 | o 1475 | d 1476 | e 1477 | , 1478 | c 1479 | o 1480 | l 1481 | o 1482 | r 1483 | = 1484 | a 1485 | r 1486 | g 1487 | s 1488 | . 1489 | c 1490 | o 1491 | l 1492 | o 1493 | r 1494 | , 1495 | v 1496 | e 1497 | r 1498 | b 1499 | o 1500 | s 1501 | e 1502 | = 1503 | a 1504 | r 1505 | g 1506 | s 1507 | . 1508 | v 1509 | e 1510 | r 1511 | b 1512 | o 1513 | s 1514 | e 1515 | , 1516 | l 1517 | i 1518 | n 1519 | e 1520 | _ 1521 | n 1522 | u 1523 | m 1524 | b 1525 | e 1526 | r 1527 | = 1528 | a 1529 | r 1530 | g 1531 | s 1532 | . 1533 | l 1534 | i 1535 | n 1536 | e 1537 | _ 1538 | n 1539 | u 1540 | m 1541 | b 1542 | e 1543 | r 1544 | │ 1545 | ) 1546 | │ 1547 | e 1548 | x 1549 | c 1550 | e 1551 | p 1552 | t 1553 | V 1554 | a 1555 | l 1556 | u 1557 | e 1558 | E 1559 | r 1560 | r 1561 | o 1562 | r 1563 | : 1564 | │ 1565 | r 1566 | e 1567 | t 1568 | u 1569 | r 1570 | n 1571 | │ 1572 | ⋮ 1573 | . 1574 | . 1575 | . 1576 | │ 1577 | p 1578 | r 1579 | i 1580 | n 1581 | t 1582 | ( 1583 | ) 1584 | │ 1585 | ⋮ 1586 | . 1587 | . 1588 | . 1589 | │ 1590 | i 1591 | f 1592 | _ 1593 | _ 1594 | n 1595 | a 1596 | m 1597 | e 1598 | _ 1599 | _ 1600 | = 1601 | = 1602 | " 1603 | _ 1604 | _ 1605 | m 1606 | a 1607 | i 1608 | n 1609 | _ 1610 | _ 1611 | " 1612 | : 1613 | │ 1614 | r 1615 | e 1616 | s 1617 | = 1618 | m 1619 | a 1620 | i 1621 | n 1622 | ( 1623 | ) 1624 | │ 1625 | s 1626 | y 1627 | s 1628 | . 1629 | e 1630 | x 1631 | i 1632 | t 1633 | ( 1634 | r 1635 | e 1636 | s 1637 | ) 1638 | 1639 | -------------------------------------------------------------------------------- /assets/screenshot-encoding.svg: -------------------------------------------------------------------------------- 1 | 2 | 12 | 13 | $ 14 | g 15 | r 16 | e 17 | p 18 | - 19 | a 20 | s 21 | t 22 | e 23 | n 24 | c 25 | o 26 | d 27 | i 28 | n 29 | g 30 | g 31 | r 32 | e 33 | p 34 | _ 35 | a 36 | s 37 | t 38 | / 39 | m 40 | a 41 | i 42 | n 43 | . 44 | p 45 | y 46 | g 47 | r 48 | e 49 | p 50 | _ 51 | a 52 | s 53 | t 54 | / 55 | m 56 | a 57 | i 58 | n 59 | . 60 | p 61 | y 62 | : 63 | │ 64 | # 65 | ! 66 | / 67 | u 68 | s 69 | r 70 | / 71 | b 72 | i 73 | n 74 | / 75 | e 76 | n 77 | v 78 | p 79 | y 80 | t 81 | h 82 | o 83 | n 84 | │ 85 | │ 86 | i 87 | m 88 | p 89 | o 90 | r 91 | t 92 | a 93 | r 94 | g 95 | p 96 | a 97 | r 98 | s 99 | e 100 | ⋮ 101 | . 102 | . 103 | . 104 | │ 105 | d 106 | e 107 | f 108 | m 109 | a 110 | i 111 | n 112 | ( 113 | ) 114 | : 115 | │ 116 | # 117 | P 118 | a 119 | r 120 | s 121 | e 122 | c 123 | o 124 | m 125 | m 126 | a 127 | n 128 | d 129 | l 130 | i 131 | n 132 | e 133 | a 134 | r 135 | g 136 | u 137 | m 138 | e 139 | n 140 | t 141 | s 142 | │ 143 | p 144 | a 145 | r 146 | s 147 | e 148 | r 149 | = 150 | a 151 | r 152 | g 153 | p 154 | a 155 | r 156 | s 157 | e 158 | . 159 | A 160 | r 161 | g 162 | u 163 | m 164 | e 165 | n 166 | t 167 | P 168 | a 169 | r 170 | s 171 | e 172 | r 173 | ( 174 | ) 175 | │ 176 | p 177 | a 178 | r 179 | s 180 | e 181 | r 182 | . 183 | a 184 | d 185 | d 186 | _ 187 | a 188 | r 189 | g 190 | u 191 | m 192 | e 193 | n 194 | t 195 | ( 196 | " 197 | p 198 | a 199 | t 200 | t 201 | e 202 | r 203 | n 204 | " 205 | , 206 | n 207 | a 208 | r 209 | g 210 | s 211 | = 212 | " 213 | ? 214 | " 215 | , 216 | h 217 | e 218 | l 219 | p 220 | = 221 | " 222 | t 223 | h 224 | e 225 | p 226 | a 227 | t 228 | t 229 | e 230 | r 231 | n 232 | t 233 | o 234 | s 235 | e 236 | a 237 | r 238 | c 239 | h 240 | f 241 | o 242 | r 243 | " 244 | ) 245 | │ 246 | p 247 | a 248 | r 249 | s 250 | e 251 | r 252 | . 253 | a 254 | d 255 | d 256 | _ 257 | a 258 | r 259 | g 260 | u 261 | m 262 | e 263 | n 264 | t 265 | ( 266 | " 267 | f 268 | i 269 | l 270 | e 271 | n 272 | a 273 | m 274 | e 275 | s 276 | " 277 | , 278 | n 279 | a 280 | r 281 | g 282 | s 283 | = 284 | " 285 | * 286 | " 287 | , 288 | h 289 | e 290 | l 291 | p 292 | = 293 | " 294 | t 295 | h 296 | e 297 | f 298 | i 299 | l 300 | e 301 | s 302 | t 303 | o 304 | d 305 | i 306 | s 307 | p 308 | l 309 | a 310 | y 311 | " 312 | , 313 | d 314 | e 315 | f 316 | a 317 | u 318 | l 319 | t 320 | = 321 | " 322 | . 323 | " 324 | ) 325 | █ 326 | p 327 | a 328 | r 329 | s 330 | e 331 | r 332 | . 333 | a 334 | d 335 | d 336 | _ 337 | a 338 | r 339 | g 340 | u 341 | m 342 | e 343 | n 344 | t 345 | ( 346 | " 347 | - 348 | - 349 | e 350 | n 351 | c 352 | o 353 | d 354 | i 355 | n 356 | g 357 | " 358 | , 359 | d 360 | e 361 | f 362 | a 363 | u 364 | l 365 | t 366 | = 367 | " 368 | u 369 | t 370 | f 371 | 8 372 | " 373 | , 374 | h 375 | e 376 | l 377 | p 378 | = 379 | " 380 | f 381 | i 382 | l 383 | e 384 | e 385 | n 386 | c 387 | o 388 | d 389 | i 390 | n 391 | g 392 | " 393 | ) 394 | │ 395 | p 396 | a 397 | r 398 | s 399 | e 400 | r 401 | . 402 | a 403 | d 404 | d 405 | _ 406 | a 407 | r 408 | g 409 | u 410 | m 411 | e 412 | n 413 | t 414 | ( 415 | " 416 | - 417 | - 418 | l 419 | a 420 | n 421 | g 422 | u 423 | a 424 | g 425 | e 426 | s 427 | " 428 | , 429 | a 430 | c 431 | t 432 | i 433 | o 434 | n 435 | = 436 | " 437 | s 438 | t 439 | o 440 | r 441 | e 442 | _ 443 | t 444 | r 445 | u 446 | e 447 | " 448 | , 449 | h 450 | e 451 | l 452 | p 453 | = 454 | " 455 | s 456 | h 457 | o 458 | w 459 | s 460 | u 461 | p 462 | p 463 | o 464 | r 465 | t 466 | e 467 | d 468 | l 469 | a 470 | n 471 | g 472 | u 473 | a 474 | g 475 | e 476 | s 477 | " 478 | ) 479 | ⋮ 480 | . 481 | . 482 | . 483 | │ 484 | f 485 | o 486 | r 487 | f 488 | n 489 | a 490 | m 491 | e 492 | i 493 | n 494 | e 495 | n 496 | u 497 | m 498 | e 499 | r 500 | a 501 | t 502 | e 503 | _ 504 | f 505 | i 506 | l 507 | e 508 | s 509 | ( 510 | a 511 | r 512 | g 513 | s 514 | . 515 | f 516 | i 517 | l 518 | e 519 | n 520 | a 521 | m 522 | e 523 | s 524 | , 525 | s 526 | p 527 | e 528 | c 529 | ) 530 | : 531 | │ 532 | p 533 | r 534 | o 535 | c 536 | e 537 | s 538 | s 539 | _ 540 | f 541 | i 542 | l 543 | e 544 | n 545 | a 546 | m 547 | e 548 | ( 549 | f 550 | n 551 | a 552 | m 553 | e 554 | , 555 | a 556 | r 557 | g 558 | s 559 | ) 560 | │ 561 | ⋮ 562 | . 563 | . 564 | . 565 | │ 566 | d 567 | e 568 | f 569 | p 570 | r 571 | o 572 | c 573 | e 574 | s 575 | s 576 | _ 577 | f 578 | i 579 | l 580 | e 581 | n 582 | a 583 | m 584 | e 585 | ( 586 | f 587 | i 588 | l 589 | e 590 | n 591 | a 592 | m 593 | e 594 | , 595 | a 596 | r 597 | g 598 | s 599 | ) 600 | : 601 | │ 602 | t 603 | r 604 | y 605 | : 606 | █ 607 | w 608 | i 609 | t 610 | h 611 | o 612 | p 613 | e 614 | n 615 | ( 616 | f 617 | i 618 | l 619 | e 620 | n 621 | a 622 | m 623 | e 624 | , 625 | " 626 | r 627 | " 628 | , 629 | e 630 | n 631 | c 632 | o 633 | d 634 | i 635 | n 636 | g 637 | = 638 | a 639 | r 640 | g 641 | s 642 | . 643 | e 644 | n 645 | c 646 | o 647 | d 648 | i 649 | n 650 | g 651 | ) 652 | a 653 | s 654 | f 655 | i 656 | l 657 | e 658 | : 659 | │ 660 | c 661 | o 662 | d 663 | e 664 | = 665 | f 666 | i 667 | l 668 | e 669 | . 670 | r 671 | e 672 | a 673 | d 674 | ( 675 | ) 676 | │ 677 | e 678 | x 679 | c 680 | e 681 | p 682 | t 683 | U 684 | n 685 | i 686 | c 687 | o 688 | d 689 | e 690 | D 691 | e 692 | c 693 | o 694 | d 695 | e 696 | E 697 | r 698 | r 699 | o 700 | r 701 | : 702 | ⋮ 703 | . 704 | . 705 | . 706 | │ 707 | p 708 | r 709 | i 710 | n 711 | t 712 | ( 713 | ) 714 | │ 715 | ⋮ 716 | . 717 | . 718 | . 719 | │ 720 | i 721 | f 722 | _ 723 | _ 724 | n 725 | a 726 | m 727 | e 728 | _ 729 | _ 730 | = 731 | = 732 | " 733 | _ 734 | _ 735 | m 736 | a 737 | i 738 | n 739 | _ 740 | _ 741 | " 742 | : 743 | │ 744 | r 745 | e 746 | s 747 | = 748 | m 749 | a 750 | i 751 | n 752 | ( 753 | ) 754 | │ 755 | s 756 | y 757 | s 758 | . 759 | e 760 | x 761 | i 762 | t 763 | ( 764 | r 765 | e 766 | s 767 | ) 768 | 769 | -------------------------------------------------------------------------------- /grep_ast/__init__.py: -------------------------------------------------------------------------------- 1 | # noqa: F401 2 | 3 | from .grep_ast import TreeContext 4 | from .parsers import filename_to_lang 5 | -------------------------------------------------------------------------------- /grep_ast/dump.py: -------------------------------------------------------------------------------- 1 | import json 2 | import traceback 3 | 4 | 5 | def cvt(s): 6 | if isinstance(s, str): 7 | return s 8 | try: 9 | return json.dumps(s, indent=4) 10 | except TypeError: 11 | return str(s) 12 | 13 | 14 | def dump(*vals): 15 | # http://docs.python.org/library/traceback.html 16 | stack = traceback.extract_stack() 17 | vars = stack[-2][3] 18 | 19 | # strip away the call to dump() 20 | vars = "(".join(vars.split("(")[1:]) 21 | vars = ")".join(vars.split(")")[:-1]) 22 | 23 | vals = [cvt(v) for v in vals] 24 | has_newline = sum(1 for v in vals if "\n" in v) 25 | if has_newline: 26 | print("%s:" % vars) 27 | print(", ".join(vals)) 28 | else: 29 | print("%s:" % vars, ", ".join(vals)) 30 | -------------------------------------------------------------------------------- /grep_ast/grep_ast.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import re 4 | 5 | from .dump import dump # noqa: F401 6 | from .parsers import filename_to_lang 7 | from .tsl import get_parser 8 | 9 | 10 | class TreeContext: 11 | def __init__( 12 | self, 13 | filename, 14 | code, 15 | color=False, 16 | verbose=False, 17 | line_number=False, 18 | parent_context=True, 19 | child_context=True, 20 | last_line=True, 21 | margin=3, 22 | mark_lois=True, 23 | header_max=10, 24 | show_top_of_file_parent_scope=True, 25 | loi_pad=1, 26 | ): 27 | self.filename = filename 28 | self.color = color 29 | self.verbose = verbose 30 | self.line_number = line_number 31 | self.last_line = last_line 32 | self.margin = margin 33 | self.mark_lois = mark_lois 34 | self.header_max = header_max 35 | self.loi_pad = loi_pad 36 | self.show_top_of_file_parent_scope = show_top_of_file_parent_scope 37 | 38 | self.parent_context = parent_context 39 | self.child_context = child_context 40 | 41 | lang = filename_to_lang(filename) 42 | if not lang: 43 | raise ValueError(f"Unknown language for {filename}") 44 | 45 | # Get parser based on file extension 46 | parser = get_parser(lang) 47 | tree = parser.parse(bytes(code, "utf8")) 48 | 49 | self.lines = code.splitlines() 50 | self.num_lines = len(self.lines) + 1 51 | 52 | # color lines, with highlighted matches 53 | self.output_lines = dict() 54 | 55 | # Which scopes is each line part of? 56 | # A scope is the line number on which the scope started 57 | self.scopes = [set() for _ in range(self.num_lines)] 58 | 59 | # Which lines serve as a short "header" for the scope starting on that line 60 | self.header = [list() for _ in range(self.num_lines)] 61 | 62 | self.nodes = [list() for _ in range(self.num_lines)] 63 | 64 | root_node = tree.root_node 65 | self.walk_tree(root_node) 66 | 67 | if self.verbose: 68 | scope_width = max(len(str(set(self.scopes[i]))) for i in range(self.num_lines - 1)) 69 | for i in range(self.num_lines): 70 | header = sorted(self.header[i]) 71 | if self.verbose and i < self.num_lines - 1: 72 | scopes = str(sorted(set(self.scopes[i]))) 73 | print(f"{scopes.ljust(scope_width)}", i, self.lines[i]) 74 | 75 | if len(header) > 1: 76 | size, head_start, head_end = header[0] 77 | if size > self.header_max: 78 | head_end = head_start + self.header_max 79 | else: 80 | head_start = i 81 | head_end = i + 1 82 | 83 | self.header[i] = head_start, head_end 84 | 85 | self.show_lines = set() 86 | self.lines_of_interest = set() 87 | 88 | return 89 | 90 | def grep(self, pat, ignore_case): 91 | found = set() 92 | for i, line in enumerate(self.lines): 93 | if re.search(pat, line, re.IGNORECASE if ignore_case else 0): 94 | if self.color: 95 | highlighted_line = re.sub( 96 | pat, 97 | lambda match: f"\033[1;31m{match.group()}\033[0m", # noqa 98 | line, 99 | flags=re.IGNORECASE if ignore_case else 0, 100 | ) 101 | self.output_lines[i] = highlighted_line 102 | found.add(i) 103 | return found 104 | 105 | def add_lines_of_interest(self, line_nums): 106 | self.lines_of_interest.update(line_nums) 107 | 108 | def add_context(self): 109 | if not self.lines_of_interest: 110 | return 111 | 112 | self.done_parent_scopes = set() 113 | 114 | self.show_lines = set(self.lines_of_interest) 115 | 116 | if self.loi_pad: 117 | for line in list(self.show_lines): 118 | for new_line in range(line - self.loi_pad, line + self.loi_pad + 1): 119 | # if not self.scopes[line].intersection(self.scopes[new_line]): 120 | # continue 121 | if new_line >= self.num_lines: 122 | continue 123 | if new_line < 0: 124 | continue 125 | self.show_lines.add(new_line) 126 | 127 | if self.last_line: 128 | # add the bottom line (plus parent context) 129 | bottom_line = self.num_lines - 2 130 | self.show_lines.add(bottom_line) 131 | self.add_parent_scopes(bottom_line) 132 | 133 | if self.parent_context: 134 | for i in set(self.lines_of_interest): 135 | self.add_parent_scopes(i) 136 | 137 | if self.child_context: 138 | for i in set(self.lines_of_interest): 139 | self.add_child_context(i) 140 | 141 | # add the top margin lines of the file 142 | if self.margin: 143 | self.show_lines.update(range(self.margin)) 144 | 145 | self.close_small_gaps() 146 | 147 | def add_child_context(self, i): 148 | if not self.nodes[i]: 149 | return 150 | 151 | last_line = self.get_last_line_of_scope(i) 152 | size = last_line - i 153 | if size < 5: 154 | self.show_lines.update(range(i, last_line + 1)) 155 | return 156 | 157 | children = [] 158 | for node in self.nodes[i]: 159 | children += self.find_all_children(node) 160 | 161 | children = sorted( 162 | children, 163 | key=lambda node: node.end_point[0] - node.start_point[0], 164 | reverse=True, 165 | ) 166 | 167 | currently_showing = len(self.show_lines) 168 | max_to_show = 25 169 | min_to_show = 5 170 | percent_to_show = 0.10 171 | max_to_show = max(min(size * percent_to_show, max_to_show), min_to_show) 172 | 173 | for child in children: 174 | if len(self.show_lines) > currently_showing + max_to_show: 175 | break 176 | child_start_line = child.start_point[0] 177 | self.add_parent_scopes(child_start_line) 178 | 179 | def find_all_children(self, node): 180 | children = [node] 181 | for child in node.children: 182 | children += self.find_all_children(child) 183 | return children 184 | 185 | def get_last_line_of_scope(self, i): 186 | last_line = max(node.end_point[0] for node in self.nodes[i]) 187 | return last_line 188 | 189 | def close_small_gaps(self): 190 | # a "closing" operation on the integers in set. 191 | # if i and i+2 are in there but i+1 is not, I want to add i+1 192 | # Create a new set for the "closed" lines 193 | closed_show = set(self.show_lines) 194 | sorted_show = sorted(self.show_lines) 195 | for i in range(len(sorted_show) - 1): 196 | if sorted_show[i + 1] - sorted_show[i] == 2: 197 | closed_show.add(sorted_show[i] + 1) 198 | 199 | # pick up adjacent blank lines 200 | for i, line in enumerate(self.lines): 201 | if i not in closed_show: 202 | continue 203 | if self.lines[i].strip() and i < self.num_lines - 2 and not self.lines[i + 1].strip(): 204 | closed_show.add(i + 1) 205 | 206 | self.show_lines = closed_show 207 | 208 | def format(self): 209 | if not self.show_lines: 210 | return "" 211 | 212 | output = "" 213 | if self.color: 214 | # reset 215 | output += "\033[0m\n" 216 | 217 | dots = not (0 in self.show_lines) 218 | for i, line in enumerate(self.lines): 219 | if i not in self.show_lines: 220 | if dots: 221 | if self.line_number: 222 | output += "...⋮...\n" 223 | else: 224 | output += "⋮\n" 225 | dots = False 226 | continue 227 | 228 | if i in self.lines_of_interest and self.mark_lois: 229 | spacer = "█" 230 | if self.color: 231 | spacer = f"\033[31m{spacer}\033[0m" 232 | else: 233 | spacer = "│" 234 | 235 | line_output = f"{spacer}{self.output_lines.get(i, line)}" 236 | if self.line_number: 237 | line_output = f"{i + 1: 3}" + line_output 238 | output += line_output + "\n" 239 | 240 | dots = True 241 | 242 | return output 243 | 244 | def add_parent_scopes(self, i): 245 | if i in self.done_parent_scopes: 246 | return 247 | self.done_parent_scopes.add(i) 248 | 249 | if i >= len(self.scopes): 250 | return 251 | 252 | for line_num in self.scopes[i]: 253 | head_start, head_end = self.header[line_num] 254 | if head_start > 0 or self.show_top_of_file_parent_scope: 255 | self.show_lines.update(range(head_start, head_end)) 256 | 257 | if self.last_line: 258 | last_line = self.get_last_line_of_scope(line_num) 259 | self.add_parent_scopes(last_line) 260 | 261 | def walk_tree(self, node, depth=0): 262 | start = node.start_point 263 | end = node.end_point 264 | 265 | start_line = start[0] 266 | end_line = end[0] 267 | size = end_line - start_line 268 | 269 | self.nodes[start_line].append(node) 270 | 271 | # dump(start_line, end_line, node.text) 272 | if self.verbose and node.is_named: 273 | """ 274 | for k in dir(node): 275 | print(k, getattr(node, k)) 276 | """ 277 | print( 278 | " " * depth, 279 | node.type, 280 | f"{start_line}-{end_line}={size + 1}", 281 | node.text.splitlines()[0], 282 | self.lines[start_line], 283 | ) 284 | 285 | if size: 286 | self.header[start_line].append((size, start_line, end_line)) 287 | 288 | for i in range(start_line, end_line + 1): 289 | self.scopes[i].add(start_line) 290 | 291 | for child in node.children: 292 | self.walk_tree(child, depth + 1) 293 | 294 | return start_line, end_line 295 | -------------------------------------------------------------------------------- /grep_ast/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import os 5 | import sys 6 | from pathlib import Path 7 | 8 | import pathspec 9 | 10 | from .dump import dump # noqa: F401 11 | from .grep_ast import TreeContext 12 | from .parsers import PARSERS 13 | 14 | 15 | def main(): 16 | # Parse command line arguments 17 | parser = argparse.ArgumentParser() 18 | parser.add_argument("pattern", nargs="?", help="the pattern to search for") 19 | parser.add_argument("filenames", nargs="*", help="the files to display", default=".") 20 | parser.add_argument("--encoding", default="utf8", help="file encoding") 21 | parser.add_argument("--languages", action="store_true", help="show supported languages") 22 | parser.add_argument("-i", "--ignore-case", action="store_true", help="ignore case distinctions") 23 | parser.add_argument("--color", action="store_true", help="force color printing", default=None) 24 | parser.add_argument( 25 | "--no-color", action="store_false", help="disable color printing", dest="color" 26 | ) 27 | parser.add_argument("--no-gitignore", action="store_true", help="ignore .gitignore file") 28 | parser.add_argument("--verbose", action="store_true", help="enable verbose output") 29 | parser.add_argument("-n", "--line-number", action="store_true", help="display line numbers") 30 | args = parser.parse_args() 31 | 32 | # If stdout is not a terminal, set color to False 33 | if args.color is None: 34 | args.color = os.isatty(1) 35 | 36 | # If --languages is provided, print the parsers table and exit 37 | if args.languages: 38 | for ext, lang in sorted(PARSERS.items()): 39 | print(f"{ext}: {lang}") 40 | return 41 | elif not args.pattern: 42 | print("Please provide a pattern to search for") 43 | return 1 44 | 45 | gitignore = None 46 | if not args.no_gitignore: 47 | for parent in Path("./xxx").resolve().parents: 48 | potential_gitignore = parent / ".gitignore" 49 | if potential_gitignore.exists(): 50 | gitignore = potential_gitignore 51 | break 52 | 53 | if gitignore: 54 | with gitignore.open() as f: 55 | spec = pathspec.PathSpec.from_lines("gitwildmatch", f) 56 | else: 57 | spec = pathspec.PathSpec.from_lines("gitwildmatch", []) 58 | 59 | for fname in enumerate_files(args.filenames, spec): 60 | process_filename(fname, args) 61 | 62 | 63 | def enumerate_files(fnames, spec, use_spec=False): 64 | for fname in fnames: 65 | fname = Path(fname) 66 | 67 | # oddly, Path('.').name == "" so we will recurse it 68 | if fname.name.startswith(".") or use_spec and spec.match_file(fname): 69 | continue 70 | 71 | if fname.is_file(): 72 | yield str(fname) 73 | continue 74 | 75 | if fname.is_dir(): 76 | for sub_fnames in enumerate_files(fname.iterdir(), spec, True): 77 | yield sub_fnames 78 | 79 | 80 | def process_filename(filename, args): 81 | try: 82 | with open(filename, "r", encoding=args.encoding) as file: 83 | code = file.read() 84 | except UnicodeDecodeError: 85 | return 86 | 87 | try: 88 | tc = TreeContext( 89 | filename, code, color=args.color, verbose=args.verbose, line_number=args.line_number 90 | ) 91 | except ValueError: 92 | return 93 | 94 | loi = tc.grep(args.pattern, args.ignore_case) 95 | if not loi: 96 | return 97 | 98 | tc.add_lines_of_interest(loi) 99 | tc.add_context() 100 | 101 | print() 102 | print(f"{filename}:") 103 | 104 | print(tc.format(), end="") 105 | 106 | print() 107 | 108 | 109 | if __name__ == "__main__": 110 | res = main() 111 | sys.exit(res) 112 | -------------------------------------------------------------------------------- /grep_ast/parsers.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from .tsl import USING_TSL_PACK 4 | 5 | # Updated mapping of file extensions to parsers 6 | PARSERS = { 7 | ".py": "python", 8 | ".js": "javascript", 9 | ".mjs": "javascript", # mjs file extension stands for "module JavaScript." 10 | ".go": "go", 11 | ".bash": "bash", 12 | ".c": "c", 13 | ".cc": "cpp", 14 | ".cs": "c_sharp", 15 | ".cl": "commonlisp", 16 | ".cpp": "cpp", 17 | ".css": "css", 18 | ".dockerfile": "dockerfile", 19 | ".dot": "dot", 20 | ".el": "elisp", 21 | ".ex": "elixir", 22 | ".elm": "elm", 23 | ".et": "embedded_template", 24 | ".erl": "erlang", 25 | ".gomod": "gomod", 26 | ".hack": "hack", 27 | ".hs": "haskell", 28 | ".hcl": "hcl", 29 | ".html": "html", 30 | ".java": "java", 31 | ".jsdoc": "jsdoc", 32 | ".json": "json", 33 | ".jl": "julia", 34 | ".kt": "kotlin", 35 | ".lua": "lua", 36 | ".mk": "make", 37 | ".md": "markdown", # https://github.com/ikatyang/tree-sitter-markdown/issues/59 38 | ".m": "objc", 39 | ".ml": "ocaml", 40 | ".mli": "ocaml_interface", 41 | ".pl": "perl", 42 | ".php": "php", 43 | ".ql": "ql", 44 | ".r": "r", 45 | ".R": "r", 46 | ".regex": "regex", 47 | ".rst": "rst", 48 | ".rb": "ruby", 49 | ".rs": "rust", 50 | ".scala": "scala", 51 | ".sql": "sql", 52 | ".sqlite": "sqlite", 53 | ".tf": "hcl", 54 | ".toml": "toml", 55 | ".tsq": "tsq", 56 | ".tsx": "typescript", 57 | ".ts": "typescript", 58 | ".yaml": "yaml", 59 | } 60 | 61 | if USING_TSL_PACK: 62 | # Replace the PARSERS dictionary with a comprehensive mapping based on the language pack 63 | PARSERS = { 64 | # A 65 | ".as": "actionscript", 66 | ".adb": "ada", 67 | ".ads": "ada", 68 | ".agda": "agda", 69 | ".ino": "arduino", 70 | ".asm": "asm", 71 | ".s": "asm", 72 | ".astro": "astro", 73 | # B 74 | ".sh": "bash", 75 | ".bash": "bash", 76 | ".zsh": "bash", 77 | ".bean": "beancount", 78 | ".bib": "bibtex", 79 | ".bicep": "bicep", 80 | ".bb": "bitbake", 81 | ".bbappend": "bitbake", 82 | ".bbclass": "bitbake", 83 | # C 84 | ".c": "c", 85 | ".h": "c", 86 | ".cairo": "cairo", 87 | ".capnp": "capnp", 88 | ".chatito": "chatito", 89 | ".clar": "clarity", 90 | ".clj": "clojure", 91 | ".cljs": "clojure", 92 | ".cljc": "clojure", 93 | ".edn": "clojure", 94 | ".cmake": "cmake", 95 | "CMakeLists.txt": "cmake", 96 | ".lisp": "commonlisp", 97 | ".cl": "commonlisp", 98 | ".cpon": "cpon", 99 | ".cpp": "cpp", 100 | ".cc": "cpp", 101 | ".cxx": "cpp", 102 | ".hpp": "cpp", 103 | ".hxx": "cpp", 104 | ".h++": "cpp", 105 | ".cs": "csharp", 106 | ".css": "css", 107 | ".csv": "csv", 108 | ".cu": "cuda", 109 | ".cuh": "cuda", 110 | ".d": "d", 111 | # D 112 | ".dart": "dart", 113 | "Dockerfile": "dockerfile", 114 | ".dtd": "dtd", 115 | # E 116 | ".el": "elisp", 117 | ".ex": "elixir", 118 | ".exs": "elixir", 119 | ".elm": "elm", 120 | ".erl": "erlang", 121 | ".hrl": "erlang", 122 | # F 123 | ".fnl": "fennel", 124 | ".fir": "firrtl", 125 | ".fish": "fish", 126 | ".f": "fortran", 127 | ".f90": "fortran", 128 | ".f95": "fortran", 129 | ".f03": "fortran", 130 | ".f08": "fortran", 131 | ".fc": "func", 132 | # G 133 | ".gd": "gdscript", 134 | ".gitattributes": "gitattributes", 135 | ".gitcommit": "gitcommit", 136 | ".gitignore": "gitignore", 137 | ".gleam": "gleam", 138 | ".glsl": "glsl", 139 | ".vert": "glsl", 140 | ".frag": "glsl", 141 | ".gn": "gn", 142 | ".gni": "gn", 143 | ".go": "go", 144 | "go.mod": "gomod", 145 | "go.sum": "gosum", 146 | ".groovy": "groovy", 147 | ".launch": "gstlaunch", 148 | # H 149 | ".hack": "hack", 150 | ".ha": "hare", 151 | ".hs": "haskell", 152 | ".hx": "haxe", 153 | ".hcl": "hcl", 154 | ".tf": "hcl", 155 | ".tfvars": "hcl", 156 | ".heex": "heex", 157 | ".hlsl": "hlsl", 158 | ".html": "html", 159 | ".htm": "html", 160 | ".hypr": "hyprlang", 161 | # I 162 | ".ispc": "ispc", 163 | # J 164 | ".janet": "janet", 165 | ".java": "java", 166 | ".js": "javascript", 167 | ".jsx": "javascript", 168 | ".mjs": "javascript", 169 | ".jsdoc": "jsdoc", 170 | ".json": "json", 171 | ".jsonnet": "jsonnet", 172 | ".libsonnet": "jsonnet", 173 | ".jl": "julia", 174 | # K 175 | "Kconfig": "kconfig", 176 | ".kdl": "kdl", 177 | ".kt": "kotlin", 178 | ".kts": "kotlin", 179 | # L 180 | ".tex": "latex", 181 | ".sty": "latex", 182 | ".cls": "latex", 183 | ".ld": "linkerscript", 184 | ".ll": "llvm", 185 | ".td": "tablegen", 186 | ".lua": "lua", 187 | ".luadoc": "luadoc", 188 | ".luap": "luap", 189 | ".luau": "luau", 190 | # M 191 | ".magik": "magik", 192 | "Makefile": "make", 193 | ".mk": "make", 194 | ".md": "markdown", 195 | ".markdown": "markdown", 196 | ".m": "matlab", # Note: .m is used by both MATLAB and Objective-C, prioritizing MATLAB here 197 | ".mat": "matlab", 198 | ".mermaid": "mermaid", 199 | "meson.build": "meson", 200 | # N 201 | ".ninja": "ninja", 202 | ".nix": "nix", 203 | ".nqc": "nqc", 204 | # O 205 | # .m extension is handled under MATLAB section (dual use extension) 206 | ".mm": "objc", 207 | ".ml": "ocaml", 208 | ".mli": "ocaml_interface", 209 | ".odin": "odin", 210 | ".org": "org", 211 | # P 212 | ".pas": "pascal", 213 | ".pp": "pascal", 214 | ".pem": "pem", 215 | ".pl": "perl", 216 | ".pm": "perl", 217 | ".pgn": "pgn", 218 | ".php": "php", 219 | ".po": "po", 220 | ".pot": "po", 221 | ".pony": "pony", 222 | ".ps1": "powershell", 223 | ".psm1": "powershell", 224 | ".printf": "printf", 225 | ".prisma": "prisma", 226 | ".properties": "properties", 227 | ".proto": "proto", 228 | ".psv": "psv", 229 | ".purs": "purescript", 230 | "MANIFEST.in": "pymanifest", 231 | ".py": "python", 232 | # Q 233 | "qmldir": "qmldir", 234 | ".qml": "qmljs", 235 | # Q 236 | # R 237 | ".r": "r", 238 | ".R": "r", 239 | ".rkt": "racket", 240 | ".re2c": "re2c", 241 | ".inputrc": "readline", 242 | "requirements.txt": "requirements", 243 | ".ron": "ron", 244 | ".rst": "rst", 245 | ".rb": "ruby", 246 | ".rs": "rust", 247 | # S 248 | ".scala": "scala", 249 | ".sc": "scala", 250 | ".scm": "scheme", # .scm is primarily used for Scheme files 251 | ".ss": "scheme", 252 | ".scss": "scss", 253 | ".smali": "smali", 254 | ".smithy": "smithy", 255 | ".sol": "solidity", 256 | ".rq": "sparql", 257 | ".sql": "sql", 258 | ".nut": "squirrel", 259 | ".bzl": "starlark", 260 | "BUILD": "starlark", 261 | "WORKSPACE": "starlark", 262 | ".svelte": "svelte", 263 | ".swift": "swift", 264 | # T 265 | ".tcl": "tcl", 266 | ".thrift": "thrift", 267 | ".toml": "toml", 268 | ".tsv": "tsv", 269 | ".tsx": "typescript", 270 | ".twig": "twig", 271 | ".ts": "typescript", 272 | ".typ": "typst", 273 | # U 274 | ".rules": "udev", 275 | ".ungram": "ungrammar", 276 | ".tal": "uxntal", 277 | # V 278 | # Note: .v extension is used by both V language and Verilog 279 | # Prioritizing Verilog as it's more commonly used 280 | ".sv": "verilog", 281 | ".v": "verilog", 282 | # For V language, users may need to specify parser manually 283 | ".vhd": "vhdl", 284 | ".vhdl": "vhdl", 285 | ".vim": "vim", 286 | ".vimrc": "vim", 287 | ".vue": "vue", 288 | # W 289 | ".wgsl": "wgsl", 290 | # X 291 | ".XCompose": "xcompose", 292 | ".xml": "xml", 293 | ".svg": "xml", 294 | ".xsl": "xml", 295 | # Y 296 | ".yuck": "yuck", 297 | # Z 298 | ".zig": "zig", 299 | } 300 | 301 | 302 | def filename_to_lang(filename): 303 | # First check if the full filename (like "Dockerfile" or "go.mod") is in PARSERS 304 | basename = os.path.basename(filename) 305 | if basename in PARSERS: 306 | return PARSERS[basename] 307 | 308 | # If not found by full filename, check by extension 309 | file_extension = os.path.splitext(filename)[1] 310 | return PARSERS.get(file_extension) 311 | -------------------------------------------------------------------------------- /grep_ast/tsl.py: -------------------------------------------------------------------------------- 1 | try: 2 | from tree_sitter_language_pack import get_language, get_parser 3 | 4 | USING_TSL_PACK = True 5 | except ImportError: 6 | from tree_sitter_languages import get_language, get_parser 7 | 8 | USING_TSL_PACK = False 9 | 10 | __all__ = [get_parser, get_language, USING_TSL_PACK] 11 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | tree-sitter-language-pack 2 | pathspec 3 | -------------------------------------------------------------------------------- /scripts/build-and-release.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ## 4 | ## Bump the version in setup.py first 5 | ## 6 | 7 | # exit when any command fails 8 | set -e 9 | 10 | old dist build 11 | 12 | python3 -m build 13 | 14 | python3 -m twine upload dist/* 15 | -------------------------------------------------------------------------------- /scripts/screenshot.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # exit when any command fails 4 | set -e 5 | 6 | PAT=$1 7 | SVG=assets/screenshot-$PAT.svg 8 | 9 | CMD="grep-ast $PAT grep_ast/grep_ast.py" 10 | 11 | echo > tmp.txt 12 | echo "\$ $CMD" >> tmp.txt 13 | 14 | $CMD --color >> tmp.txt 15 | 16 | cat tmp.txt | ~/go/bin/ansisvg > $SVG 17 | 18 | open $SVG 19 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | from setuptools import find_packages, setup 4 | 5 | with open("requirements.txt") as f: 6 | requirements = f.read().splitlines() 7 | 8 | with open("README.md", "r", encoding="utf-8") as f: 9 | long_description = f.read() 10 | long_description = re.sub(r"\n!\[.*\]\(.*\)", "", long_description) 11 | 12 | setup( 13 | name="grep-ast", 14 | version="0.8.1", 15 | description="A tool to grep through the AST of a source file", 16 | url="https://github.com/paul-gauthier/grep-ast", 17 | long_description=long_description, 18 | long_description_content_type="text/markdown", 19 | packages=find_packages(), 20 | install_requires=requirements, 21 | entry_points={ 22 | "console_scripts": [ 23 | "grep-ast=grep_ast.main:main", 24 | "gast=grep_ast.main:main", 25 | ], 26 | }, 27 | classifiers=["License :: OSI Approved :: Apache Software License"], 28 | ) 29 | -------------------------------------------------------------------------------- /tests/test_parsers.py: -------------------------------------------------------------------------------- 1 | from tree_sitter_languages import get_parser 2 | 3 | from grep_ast.dump import dump # noqa: F401 4 | from grep_ast.parsers import PARSERS 5 | 6 | 7 | def test_get_parser_for_all_parsers(): 8 | for lang in PARSERS.values(): 9 | assert get_parser(lang) is not None 10 | --------------------------------------------------------------------------------