├── LICENSE ├── README.md ├── pythonparser └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- 1 | GNU LESSER GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | 9 | This version of the GNU Lesser General Public License incorporates 10 | the terms and conditions of version 3 of the GNU General Public 11 | License, supplemented by the additional permissions listed below. 12 | 13 | 0. Additional Definitions. 14 | 15 | As used herein, "this License" refers to version 3 of the GNU Lesser 16 | General Public License, and the "GNU GPL" refers to version 3 of the GNU 17 | General Public License. 18 | 19 | "The Library" refers to a covered work governed by this License, 20 | other than an Application or a Combined Work as defined below. 21 | 22 | An "Application" is any work that makes use of an interface provided 23 | by the Library, but which is not otherwise based on the Library. 24 | Defining a subclass of a class defined by the Library is deemed a mode 25 | of using an interface provided by the Library. 26 | 27 | A "Combined Work" is a work produced by combining or linking an 28 | Application with the Library. The particular version of the Library 29 | with which the Combined Work was made is also called the "Linked 30 | Version". 31 | 32 | The "Minimal Corresponding Source" for a Combined Work means the 33 | Corresponding Source for the Combined Work, excluding any source code 34 | for portions of the Combined Work that, considered in isolation, are 35 | based on the Application, and not on the Linked Version. 36 | 37 | The "Corresponding Application Code" for a Combined Work means the 38 | object code and/or source code for the Application, including any data 39 | and utility programs needed for reproducing the Combined Work from the 40 | Application, but excluding the System Libraries of the Combined Work. 41 | 42 | 1. Exception to Section 3 of the GNU GPL. 43 | 44 | You may convey a covered work under sections 3 and 4 of this License 45 | without being bound by section 3 of the GNU GPL. 46 | 47 | 2. Conveying Modified Versions. 48 | 49 | If you modify a copy of the Library, and, in your modifications, a 50 | facility refers to a function or data to be supplied by an Application 51 | that uses the facility (other than as an argument passed when the 52 | facility is invoked), then you may convey a copy of the modified 53 | version: 54 | 55 | a) under this License, provided that you make a good faith effort to 56 | ensure that, in the event an Application does not supply the 57 | function or data, the facility still operates, and performs 58 | whatever part of its purpose remains meaningful, or 59 | 60 | b) under the GNU GPL, with none of the additional permissions of 61 | this License applicable to that copy. 62 | 63 | 3. Object Code Incorporating Material from Library Header Files. 64 | 65 | The object code form of an Application may incorporate material from 66 | a header file that is part of the Library. You may convey such object 67 | code under terms of your choice, provided that, if the incorporated 68 | material is not limited to numerical parameters, data structure 69 | layouts and accessors, or small macros, inline functions and templates 70 | (ten or fewer lines in length), you do both of the following: 71 | 72 | a) Give prominent notice with each copy of the object code that the 73 | Library is used in it and that the Library and its use are 74 | covered by this License. 75 | 76 | b) Accompany the object code with a copy of the GNU GPL and this license 77 | document. 78 | 79 | 4. Combined Works. 80 | 81 | You may convey a Combined Work under terms of your choice that, 82 | taken together, effectively do not restrict modification of the 83 | portions of the Library contained in the Combined Work and reverse 84 | engineering for debugging such modifications, if you also do each of 85 | the following: 86 | 87 | a) Give prominent notice with each copy of the Combined Work that 88 | the Library is used in it and that the Library and its use are 89 | covered by this License. 90 | 91 | b) Accompany the Combined Work with a copy of the GNU GPL and this license 92 | document. 93 | 94 | c) For a Combined Work that displays copyright notices during 95 | execution, include the copyright notice for the Library among 96 | these notices, as well as a reference directing the user to the 97 | copies of the GNU GPL and this license document. 98 | 99 | d) Do one of the following: 100 | 101 | 0) Convey the Minimal Corresponding Source under the terms of this 102 | License, and the Corresponding Application Code in a form 103 | suitable for, and under terms that permit, the user to 104 | recombine or relink the Application with a modified version of 105 | the Linked Version to produce a modified Combined Work, in the 106 | manner specified by section 6 of the GNU GPL for conveying 107 | Corresponding Source. 108 | 109 | 1) Use a suitable shared library mechanism for linking with the 110 | Library. A suitable mechanism is one that (a) uses at run time 111 | a copy of the Library already present on the user's computer 112 | system, and (b) will operate properly with a modified version 113 | of the Library that is interface-compatible with the Linked 114 | Version. 115 | 116 | e) Provide Installation Information, but only if you would otherwise 117 | be required to provide such information under section 6 of the 118 | GNU GPL, and only to the extent that such information is 119 | necessary to install and execute a modified version of the 120 | Combined Work produced by recombining or relinking the 121 | Application with a modified version of the Linked Version. (If 122 | you use option 4d0, the Installation Information must accompany 123 | the Minimal Corresponding Source and Corresponding Application 124 | Code. If you use option 4d1, you must provide the Installation 125 | Information in the manner specified by section 6 of the GNU GPL 126 | for conveying Corresponding Source.) 127 | 128 | 5. Combined Libraries. 129 | 130 | You may place library facilities that are a work based on the 131 | Library side by side in a single library together with other library 132 | facilities that are not Applications and are not covered by this 133 | License, and convey such a combined library under terms of your 134 | choice, if you do both of the following: 135 | 136 | a) Accompany the combined library with a copy of the same work based 137 | on the Library, uncombined with any other library facilities, 138 | conveyed under the terms of this License. 139 | 140 | b) Give prominent notice with the combined library that part of it 141 | is a work based on the Library, and explaining where to find the 142 | accompanying uncombined form of the same work. 143 | 144 | 6. Revised Versions of the GNU Lesser General Public License. 145 | 146 | The Free Software Foundation may publish revised and/or new versions 147 | of the GNU Lesser General Public License from time to time. Such new 148 | versions will be similar in spirit to the present version, but may 149 | differ in detail to address new problems or concerns. 150 | 151 | Each version is given a distinguishing version number. If the 152 | Library as you received it specifies that a certain numbered version 153 | of the GNU Lesser General Public License "or any later version" 154 | applies to it, you have the option of following the terms and 155 | conditions either of that published version or of any later version 156 | published by the Free Software Foundation. If the Library as you 157 | received it does not specify a version number of the GNU Lesser 158 | General Public License, you may choose any version of the GNU Lesser 159 | General Public License ever published by the Free Software Foundation. 160 | 161 | If the Library as you received it specifies that a proxy can decide 162 | whether future versions of the GNU Lesser General Public License shall 163 | apply, that proxy's public statement of acceptance of any version is 164 | permanent authorization for you to choose that version for the 165 | Library. 166 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pythonparser 2 | 3 | Pythonparser is a python tool that produces a GumTree compatible Python AST. It uses the awesome `parso`module under the hood. 4 | 5 | ## Installation 6 | 7 | Pythonparser requires to use Python 3. To install it, just clone the repository and install the dependencies using 8 | 9 | ~~~ 10 | pip3 install -r requirements.txt 11 | ~~~~ 12 | 13 | Pythonpaser can be used as a standalone tool like this: 14 | 15 | ~~~ 16 | pythonparser /path/to/script.py 17 | ~~~~ 18 | 19 | -------------------------------------------------------------------------------- /pythonparser: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # This file is part of pythonparser. 4 | 5 | # pythonparser is free software: you can redistribute it and/or modify 6 | # it under the terms of the GNU Lesser General Public License as published by 7 | # the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # pythonparser is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 13 | # GNU Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public License 16 | # along with pythonparser. If not, see . 17 | # 18 | # Copyright 2020-2021 Jean-Rémy Falleri 19 | 20 | from xml.dom import minidom 21 | import parso 22 | import sys 23 | 24 | doc = minidom.Document() 25 | positions = [0] 26 | 27 | def main(file): 28 | parso_ast = parso.parse(read_file(file)) 29 | gumtree_ast = to_gumtree_node(parso_ast) 30 | doc.appendChild(gumtree_ast) 31 | process_node(parso_ast, gumtree_ast) 32 | xml = doc.toprettyxml() 33 | print(xml) 34 | 35 | def process_node(parso_node, gumtree_node): 36 | if parso_node.type == 'error_node': 37 | sys.exit(parso_node) 38 | 39 | for parso_child in parso_node.children: 40 | gumtree_child = to_gumtree_node(parso_child) 41 | if gumtree_child != None: 42 | gumtree_node.appendChild(gumtree_child) 43 | if hasattr(parso_child, 'children'): 44 | process_node(parso_child, gumtree_child) 45 | 46 | def to_gumtree_node(parso_node): 47 | if parso_node.type in ['keyword', 'newline', 'endmarker']: 48 | return 49 | if parso_node.type == 'operator' and parso_node.value in ['.', '(', ')', '[', ']', ':', ';']: 50 | return 51 | gumtree_node = doc.createElement('tree') 52 | gumtree_node.setAttribute("type", parso_node.type) 53 | start_pos = positions[parso_node.start_pos[0] - 1] + parso_node.start_pos[1] 54 | end_pos = positions[parso_node.end_pos[0] - 1] + parso_node.end_pos[1] 55 | length = end_pos - start_pos 56 | gumtree_node.setAttribute("pos", str(start_pos)) 57 | gumtree_node.setAttribute("length", str(length)) 58 | if (not hasattr(parso_node, 'children')) or len(parso_node.children) == 0: 59 | gumtree_node.setAttribute("label", parso_node.value) 60 | return gumtree_node 61 | 62 | def read_file(file): 63 | with open(file, 'r') as file: 64 | data = file.read() 65 | index = 0 66 | for chr in data: 67 | index += 1 68 | if chr == '\n': 69 | positions.append(index) 70 | return data 71 | 72 | if __name__ == '__main__': 73 | main(sys.argv[1]) -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | parso --------------------------------------------------------------------------------