├── .gitignore ├── LICENSE ├── Python.asdl ├── README.rst ├── TODO ├── asdl.py ├── asdl_c.py ├── asdl_test.py ├── docs └── ASDL Zephyr - Wang.pdf └── gen.sh /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | __pycache__ 21 | 22 | # Installer logs 23 | pip-log.txt 24 | 25 | # Unit test / coverage reports 26 | .coverage 27 | .tox 28 | nosetests.xml 29 | 30 | # Translations 31 | *.mo 32 | 33 | # Mr Developer 34 | .mr.developer.cfg 35 | .project 36 | .pydevproject 37 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | A. HISTORY OF THE SOFTWARE 2 | ========================== 3 | 4 | Python was created in the early 1990s by Guido van Rossum at Stichting 5 | Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands 6 | as a successor of a language called ABC. Guido remains Python's 7 | principal author, although it includes many contributions from others. 8 | 9 | In 1995, Guido continued his work on Python at the Corporation for 10 | National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) 11 | in Reston, Virginia where he released several versions of the 12 | software. 13 | 14 | In May 2000, Guido and the Python core development team moved to 15 | BeOpen.com to form the BeOpen PythonLabs team. In October of the same 16 | year, the PythonLabs team moved to Digital Creations (now Zope 17 | Corporation, see http://www.zope.com). In 2001, the Python Software 18 | Foundation (PSF, see http://www.python.org/psf/) was formed, a 19 | non-profit organization created specifically to own Python-related 20 | Intellectual Property. Zope Corporation is a sponsoring member of 21 | the PSF. 22 | 23 | All Python releases are Open Source (see http://www.opensource.org for 24 | the Open Source Definition). Historically, most, but not all, Python 25 | releases have also been GPL-compatible; the table below summarizes 26 | the various releases. 27 | 28 | Release Derived Year Owner GPL- 29 | from compatible? (1) 30 | 31 | 0.9.0 thru 1.2 1991-1995 CWI yes 32 | 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes 33 | 1.6 1.5.2 2000 CNRI no 34 | 2.0 1.6 2000 BeOpen.com no 35 | 1.6.1 1.6 2001 CNRI yes (2) 36 | 2.1 2.0+1.6.1 2001 PSF no 37 | 2.0.1 2.0+1.6.1 2001 PSF yes 38 | 2.1.1 2.1+2.0.1 2001 PSF yes 39 | 2.1.2 2.1.1 2002 PSF yes 40 | 2.1.3 2.1.2 2002 PSF yes 41 | 2.2 and above 2.1.1 2001-now PSF yes 42 | 43 | Footnotes: 44 | 45 | (1) GPL-compatible doesn't mean that we're distributing Python under 46 | the GPL. All Python licenses, unlike the GPL, let you distribute 47 | a modified version without making your changes open source. The 48 | GPL-compatible licenses make it possible to combine Python with 49 | other software that is released under the GPL; the others don't. 50 | 51 | (2) According to Richard Stallman, 1.6.1 is not GPL-compatible, 52 | because its license has a choice of law clause. According to 53 | CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 54 | is "not incompatible" with the GPL. 55 | 56 | Thanks to the many outside volunteers who have worked under Guido's 57 | direction to make these releases possible. 58 | 59 | 60 | B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON 61 | =============================================================== 62 | 63 | PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 64 | -------------------------------------------- 65 | 66 | 1. This LICENSE AGREEMENT is between the Python Software Foundation 67 | ("PSF"), and the Individual or Organization ("Licensee") accessing and 68 | otherwise using this software ("Python") in source or binary form and 69 | its associated documentation. 70 | 71 | 2. Subject to the terms and conditions of this License Agreement, PSF hereby 72 | grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, 73 | analyze, test, perform and/or display publicly, prepare derivative works, 74 | distribute, and otherwise use Python alone or in any derivative version, 75 | provided, however, that PSF's License Agreement and PSF's notice of copyright, 76 | i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 77 | 2011, 2012, 2013 Python Software Foundation; All Rights Reserved" are retained 78 | in Python alone or in any derivative version prepared by Licensee. 79 | 80 | 3. In the event Licensee prepares a derivative work that is based on 81 | or incorporates Python or any part thereof, and wants to make 82 | the derivative work available to others as provided herein, then 83 | Licensee hereby agrees to include in any such work a brief summary of 84 | the changes made to Python. 85 | 86 | 4. PSF is making Python available to Licensee on an "AS IS" 87 | basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 88 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND 89 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 90 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT 91 | INFRINGE ANY THIRD PARTY RIGHTS. 92 | 93 | 5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 94 | FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS 95 | A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, 96 | OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 97 | 98 | 6. This License Agreement will automatically terminate upon a material 99 | breach of its terms and conditions. 100 | 101 | 7. Nothing in this License Agreement shall be deemed to create any 102 | relationship of agency, partnership, or joint venture between PSF and 103 | Licensee. This License Agreement does not grant permission to use PSF 104 | trademarks or trade name in a trademark sense to endorse or promote 105 | products or services of Licensee, or any third party. 106 | 107 | 8. By copying, installing or otherwise using Python, Licensee 108 | agrees to be bound by the terms and conditions of this License 109 | Agreement. 110 | 111 | 112 | BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 113 | ------------------------------------------- 114 | 115 | BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 116 | 117 | 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an 118 | office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the 119 | Individual or Organization ("Licensee") accessing and otherwise using 120 | this software in source or binary form and its associated 121 | documentation ("the Software"). 122 | 123 | 2. Subject to the terms and conditions of this BeOpen Python License 124 | Agreement, BeOpen hereby grants Licensee a non-exclusive, 125 | royalty-free, world-wide license to reproduce, analyze, test, perform 126 | and/or display publicly, prepare derivative works, distribute, and 127 | otherwise use the Software alone or in any derivative version, 128 | provided, however, that the BeOpen Python License is retained in the 129 | Software, alone or in any derivative version prepared by Licensee. 130 | 131 | 3. BeOpen is making the Software available to Licensee on an "AS IS" 132 | basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 133 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND 134 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 135 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT 136 | INFRINGE ANY THIRD PARTY RIGHTS. 137 | 138 | 4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE 139 | SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS 140 | AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY 141 | DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 142 | 143 | 5. This License Agreement will automatically terminate upon a material 144 | breach of its terms and conditions. 145 | 146 | 6. This License Agreement shall be governed by and interpreted in all 147 | respects by the law of the State of California, excluding conflict of 148 | law provisions. Nothing in this License Agreement shall be deemed to 149 | create any relationship of agency, partnership, or joint venture 150 | between BeOpen and Licensee. This License Agreement does not grant 151 | permission to use BeOpen trademarks or trade names in a trademark 152 | sense to endorse or promote products or services of Licensee, or any 153 | third party. As an exception, the "BeOpen Python" logos available at 154 | http://www.pythonlabs.com/logos.html may be used according to the 155 | permissions granted on that web page. 156 | 157 | 7. By copying, installing or otherwise using the software, Licensee 158 | agrees to be bound by the terms and conditions of this License 159 | Agreement. 160 | 161 | 162 | CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 163 | --------------------------------------- 164 | 165 | 1. This LICENSE AGREEMENT is between the Corporation for National 166 | Research Initiatives, having an office at 1895 Preston White Drive, 167 | Reston, VA 20191 ("CNRI"), and the Individual or Organization 168 | ("Licensee") accessing and otherwise using Python 1.6.1 software in 169 | source or binary form and its associated documentation. 170 | 171 | 2. Subject to the terms and conditions of this License Agreement, CNRI 172 | hereby grants Licensee a nonexclusive, royalty-free, world-wide 173 | license to reproduce, analyze, test, perform and/or display publicly, 174 | prepare derivative works, distribute, and otherwise use Python 1.6.1 175 | alone or in any derivative version, provided, however, that CNRI's 176 | License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 177 | 1995-2001 Corporation for National Research Initiatives; All Rights 178 | Reserved" are retained in Python 1.6.1 alone or in any derivative 179 | version prepared by Licensee. Alternately, in lieu of CNRI's License 180 | Agreement, Licensee may substitute the following text (omitting the 181 | quotes): "Python 1.6.1 is made available subject to the terms and 182 | conditions in CNRI's License Agreement. This Agreement together with 183 | Python 1.6.1 may be located on the Internet using the following 184 | unique, persistent identifier (known as a handle): 1895.22/1013. This 185 | Agreement may also be obtained from a proxy server on the Internet 186 | using the following URL: http://hdl.handle.net/1895.22/1013". 187 | 188 | 3. In the event Licensee prepares a derivative work that is based on 189 | or incorporates Python 1.6.1 or any part thereof, and wants to make 190 | the derivative work available to others as provided herein, then 191 | Licensee hereby agrees to include in any such work a brief summary of 192 | the changes made to Python 1.6.1. 193 | 194 | 4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" 195 | basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 196 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND 197 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 198 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT 199 | INFRINGE ANY THIRD PARTY RIGHTS. 200 | 201 | 5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 202 | 1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS 203 | A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, 204 | OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 205 | 206 | 6. This License Agreement will automatically terminate upon a material 207 | breach of its terms and conditions. 208 | 209 | 7. This License Agreement shall be governed by the federal 210 | intellectual property law of the United States, including without 211 | limitation the federal copyright law, and, to the extent such 212 | U.S. federal law does not apply, by the law of the Commonwealth of 213 | Virginia, excluding Virginia's conflict of law provisions. 214 | Notwithstanding the foregoing, with regard to derivative works based 215 | on Python 1.6.1 that incorporate non-separable material that was 216 | previously distributed under the GNU General Public License (GPL), the 217 | law of the Commonwealth of Virginia shall govern this License 218 | Agreement only as to issues arising under or with respect to 219 | Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this 220 | License Agreement shall be deemed to create any relationship of 221 | agency, partnership, or joint venture between CNRI and Licensee. This 222 | License Agreement does not grant permission to use CNRI trademarks or 223 | trade name in a trademark sense to endorse or promote products or 224 | services of Licensee, or any third party. 225 | 226 | 8. By clicking on the "ACCEPT" button where indicated, or by copying, 227 | installing or otherwise using Python 1.6.1, Licensee agrees to be 228 | bound by the terms and conditions of this License Agreement. 229 | 230 | ACCEPT 231 | 232 | 233 | CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 234 | -------------------------------------------------- 235 | 236 | Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, 237 | The Netherlands. All rights reserved. 238 | 239 | Permission to use, copy, modify, and distribute this software and its 240 | documentation for any purpose and without fee is hereby granted, 241 | provided that the above copyright notice appear in all copies and that 242 | both that copyright notice and this permission notice appear in 243 | supporting documentation, and that the name of Stichting Mathematisch 244 | Centrum or CWI not be used in advertising or publicity pertaining to 245 | distribution of the software without specific, written prior 246 | permission. 247 | 248 | STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO 249 | THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND 250 | FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE 251 | FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 252 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 253 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT 254 | OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 255 | -------------------------------------------------------------------------------- /Python.asdl: -------------------------------------------------------------------------------- 1 | -- ASDL's six builtin types are identifier, int, string, bytes, object, singleton 2 | 3 | module Python 4 | { 5 | mod = Module(stmt* body) 6 | | Interactive(stmt* body) 7 | | Expression(expr body) 8 | 9 | -- not really an actual node but useful in Jython's typesystem. 10 | | Suite(stmt* body) 11 | 12 | stmt = FunctionDef(identifier name, arguments args, 13 | stmt* body, expr* decorator_list, expr? returns) 14 | | ClassDef(identifier name, 15 | expr* bases, 16 | keyword* keywords, 17 | expr? starargs, 18 | expr? kwargs, 19 | stmt* body, 20 | expr* decorator_list) 21 | | Return(expr? value) 22 | 23 | | Delete(expr* targets) 24 | | Assign(expr* targets, expr value) 25 | | AugAssign(expr target, operator op, expr value) 26 | 27 | -- use 'orelse' because else is a keyword in target languages 28 | | For(expr target, expr iter, stmt* body, stmt* orelse) 29 | | While(expr test, stmt* body, stmt* orelse) 30 | | If(expr test, stmt* body, stmt* orelse) 31 | | With(withitem* items, stmt* body) 32 | 33 | | Raise(expr? exc, expr? cause) 34 | | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) 35 | | Assert(expr test, expr? msg) 36 | 37 | | Import(alias* names) 38 | | ImportFrom(identifier? module, alias* names, int? level) 39 | 40 | | Global(identifier* names) 41 | | Nonlocal(identifier* names) 42 | | Expr(expr value) 43 | | Pass | Break | Continue 44 | 45 | -- XXX Jython will be different 46 | -- col_offset is the byte offset in the utf8 string the parser uses 47 | attributes (int lineno, int col_offset) 48 | 49 | -- BoolOp() can use left & right? 50 | expr = BoolOp(boolop op, expr* values) 51 | | BinOp(expr left, operator op, expr right) 52 | | UnaryOp(unaryop op, expr operand) 53 | | Lambda(arguments args, expr body) 54 | | IfExp(expr test, expr body, expr orelse) 55 | | Dict(expr* keys, expr* values) 56 | | Set(expr* elts) 57 | | ListComp(expr elt, comprehension* generators) 58 | | SetComp(expr elt, comprehension* generators) 59 | | DictComp(expr key, expr value, comprehension* generators) 60 | | GeneratorExp(expr elt, comprehension* generators) 61 | -- the grammar constrains where yield expressions can occur 62 | | Yield(expr? value) 63 | | YieldFrom(expr value) 64 | -- need sequences for compare to distinguish between 65 | -- x < 4 < 3 and (x < 4) < 3 66 | | Compare(expr left, cmpop* ops, expr* comparators) 67 | | Call(expr func, expr* args, keyword* keywords, 68 | expr? starargs, expr? kwargs) 69 | | Num(object n) -- a number as a PyObject. 70 | | Str(string s) -- need to specify raw, unicode, etc? 71 | | Bytes(bytes s) 72 | | NameConstant(singleton value) 73 | | Ellipsis 74 | 75 | -- the following expression can appear in assignment context 76 | | Attribute(expr value, identifier attr, expr_context ctx) 77 | | Subscript(expr value, slice slice, expr_context ctx) 78 | | Starred(expr value, expr_context ctx) 79 | | Name(identifier id, expr_context ctx) 80 | | List(expr* elts, expr_context ctx) 81 | | Tuple(expr* elts, expr_context ctx) 82 | 83 | -- col_offset is the byte offset in the utf8 string the parser uses 84 | attributes (int lineno, int col_offset) 85 | 86 | expr_context = Load | Store | Del | AugLoad | AugStore | Param 87 | 88 | slice = Slice(expr? lower, expr? upper, expr? step) 89 | | ExtSlice(slice* dims) 90 | | Index(expr value) 91 | 92 | boolop = And | Or 93 | 94 | operator = Add | Sub | Mult | Div | Mod | Pow | LShift 95 | | RShift | BitOr | BitXor | BitAnd | FloorDiv 96 | 97 | unaryop = Invert | Not | UAdd | USub 98 | 99 | cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn 100 | 101 | comprehension = (expr target, expr iter, expr* ifs) 102 | 103 | excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) 104 | attributes (int lineno, int col_offset) 105 | 106 | arguments = (arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults, 107 | arg? kwarg, expr* defaults) 108 | 109 | arg = (identifier arg, expr? annotation) 110 | attributes (int lineno, int col_offset) 111 | 112 | -- keyword arguments supplied to call 113 | keyword = (identifier arg, expr value) 114 | 115 | -- import name with optional 'as' alias. 116 | alias = (identifier name, identifier? asname) 117 | 118 | withitem = (expr context_expr, expr? optional_vars) 119 | } 120 | 121 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Status - please read this first! 2 | ================================ 3 | 4 | This code has been submitted to upstream CPython in 5 | `early 2014 `_. I keep the separate 6 | repository alive mostly for historic interest and for folks who want to play 7 | with real ASDL-based code generators in a simpler environment than the 8 | CPython repository. 9 | 10 | asdl_parser 11 | =========== 12 | 13 | Standalone ASDL parser for upstream CPython 3.x. 14 | 15 | The parser is in a single file - asdl.py; it contains a hand-written lexer and a 16 | recursive-descent parser. 17 | 18 | Note: Python.asdl (the ASDL definition file for Python) and asdl_c.py (emitter 19 | for Python-ast.[hc]) are copied over from the CPython repository (default 20 | branch); I applied some very small cleanups to asdl_c.py, mainly 21 | because asdl.py produces cleaner ASTs than the old Spark-based parser. When run, 22 | it produces exactly the same Python-ast.[hc] as in upstream CPython. 23 | 24 | Python version 25 | ============== 26 | 27 | The officially required version is Python 3.3, but should run with any 3.x 28 | 29 | License 30 | ======= 31 | 32 | Same as CPython: Python Software Foundation License (LICENSE file included 33 | here). 34 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | * Update asdl_c.py... can be done separately 2 | 3 | Modernization of asdl_c.py: 4 | 5 | * s/getopt/argparse/ 6 | * INC_DIR/SRC_DIR globals to communicate stuff into main() - meh 7 | * Get rid of XXXs - at this point we can assume they're resolved 8 | * Remove limitation of not being able to produce both C and H simultaneously 9 | * Node-speficic hacks in reflow_lines?? 10 | 11 | Makefiles: 12 | 13 | * Why run asdl_c twice / parse the ASDL twice? - because makefile rules want 14 | to have a single target. 15 | 16 | -------------------------------------------------------------------------------- /asdl.py: -------------------------------------------------------------------------------- 1 | #------------------------------------------------------------------------------- 2 | # Parser for ASDL [1] definition files. Reads in an ASDL description and parses 3 | # it into an AST that describes it. 4 | # 5 | # The EBNF we're parsing here: Figure 1 of the paper [1]. Extended to support 6 | # modules and attributes after a product. Words starting with Capital letters 7 | # are terminals. Literal tokens are in "double quotes". Others are 8 | # non-terminals. Id is either TokenId or ConstructorId. 9 | # 10 | # module ::= "module" Id "{" [definitions] "}" 11 | # definitions ::= { TypeId "=" type } 12 | # type ::= product | sum 13 | # product ::= fields ["attributes" fields] 14 | # fields ::= "(" { field, "," } field ")" 15 | # field ::= TypeId ["?" | "*"] [Id] 16 | # sum ::= constructor { "|" constructor } ["attributes" fields] 17 | # constructor ::= ConstructorId [fields] 18 | # 19 | # [1] "The Zephyr Abstract Syntax Description Language" by Wang, et. al. See 20 | # http://asdl.sourceforge.net/ 21 | #------------------------------------------------------------------------------- 22 | from collections import namedtuple 23 | import re 24 | 25 | __all__ = [ 26 | 'builtin_types', 'parse', 'AST', 'Module', 'Type', 'Constructor', 27 | 'Field', 'Sum', 'Product', 'VisitorBase', 'Check', 'check'] 28 | 29 | # The following classes define nodes into which the ASDL description is parsed. 30 | # Note: this is a "meta-AST". ASDL files (such as Python.asdl) describe the AST 31 | # structure used by a programming language. But ASDL files themselves need to be 32 | # parsed. This module parses ASDL files and uses a simple AST to represent them. 33 | # See the EBNF at the top of the file to understand the logical connection 34 | # between the various node types. 35 | 36 | builtin_types = set( 37 | ['identifier', 'string', 'bytes', 'int', 'object', 'singleton']) 38 | 39 | class AST: 40 | def __repr__(self): 41 | raise NotImplementedError 42 | 43 | class Module(AST): 44 | def __init__(self, name, dfns): 45 | self.name = name 46 | self.dfns = dfns 47 | self.types = {type.name: type.value for type in dfns} 48 | 49 | def __repr__(self): 50 | return 'Module({0.name}, {0.dfns})'.format(self) 51 | 52 | class Type(AST): 53 | def __init__(self, name, value): 54 | self.name = name 55 | self.value = value 56 | 57 | def __repr__(self): 58 | return 'Type({0.name}, {0.value})'.format(self) 59 | 60 | class Constructor(AST): 61 | def __init__(self, name, fields=None): 62 | self.name = name 63 | self.fields = fields or [] 64 | 65 | def __repr__(self): 66 | return 'Constructor({0.name}, {0.fields})'.format(self) 67 | 68 | class Field(AST): 69 | def __init__(self, type, name=None, seq=False, opt=False): 70 | self.type = type 71 | self.name = name 72 | self.seq = seq 73 | self.opt = opt 74 | 75 | def __repr__(self): 76 | if self.seq: 77 | extra = ", seq=True" 78 | elif self.opt: 79 | extra = ", opt=True" 80 | else: 81 | extra = "" 82 | if self.name is None: 83 | return 'Field({0.type}{1})'.format(self, extra) 84 | else: 85 | return 'Field({0.type}, {0.name}{1})'.format(self, extra) 86 | 87 | class Sum(AST): 88 | def __init__(self, types, attributes=None): 89 | self.types = types 90 | self.attributes = attributes or [] 91 | 92 | def __repr__(self): 93 | if self.attributes: 94 | return 'Sum({0.types}, {0.attributes})'.format(self) 95 | else: 96 | return 'Sum({0.types})'.format(self) 97 | 98 | class Product(AST): 99 | def __init__(self, fields, attributes=None): 100 | self.fields = fields 101 | self.attributes = attributes or [] 102 | 103 | def __repr__(self): 104 | if self.attributes: 105 | return 'Product({0.fields}, {0.attributes})'.format(self) 106 | else: 107 | return 'Product({0.fields})'.format(self) 108 | 109 | # A generic visitor for the meta-AST that describes ASDL. This can be used by 110 | # emitters. Note that this visitor does not provide a generic visit method, so a 111 | # subclass needs to define visit methods from visitModule to as deep as the 112 | # interesting node. 113 | # We also define a Check visitor that makes sure the parsed ASDL is well-formed. 114 | 115 | class VisitorBase: 116 | """Generic tree visitor for ASTs.""" 117 | def __init__(self): 118 | self.cache = {} 119 | 120 | def visit(self, obj, *args): 121 | klass = obj.__class__ 122 | meth = self.cache.get(klass) 123 | if meth is None: 124 | methname = "visit" + klass.__name__ 125 | meth = getattr(self, methname, None) 126 | self.cache[klass] = meth 127 | if meth: 128 | try: 129 | meth(obj, *args) 130 | except Exception as e: 131 | print("Error visiting %r: %s" % (obj, e)) 132 | raise 133 | 134 | class Check(VisitorBase): 135 | """A visitor that checks a parsed ASDL tree for correctness. 136 | 137 | Errors are printed and accumulated. 138 | """ 139 | def __init__(self): 140 | super().__init__() 141 | self.cons = {} 142 | self.errors = 0 143 | self.types = {} 144 | 145 | def visitModule(self, mod): 146 | for dfn in mod.dfns: 147 | self.visit(dfn) 148 | 149 | def visitType(self, type): 150 | self.visit(type.value, str(type.name)) 151 | 152 | def visitSum(self, sum, name): 153 | for t in sum.types: 154 | self.visit(t, name) 155 | 156 | def visitConstructor(self, cons, name): 157 | key = str(cons.name) 158 | conflict = self.cons.get(key) 159 | if conflict is None: 160 | self.cons[key] = name 161 | else: 162 | print('Redefinition of constructor {}'.format(key)) 163 | print('Defined in {} and {}'.format(conflict, name)) 164 | self.errors += 1 165 | for f in cons.fields: 166 | self.visit(f, key) 167 | 168 | def visitField(self, field, name): 169 | key = str(field.type) 170 | l = self.types.setdefault(key, []) 171 | l.append(name) 172 | 173 | def visitProduct(self, prod, name): 174 | for f in prod.fields: 175 | self.visit(f, name) 176 | 177 | def check(mod): 178 | """Check the parsed ASDL tree for correctness. 179 | 180 | Return True if success. For failure, the errors are printed out and False 181 | is returned. 182 | """ 183 | v = Check() 184 | v.visit(mod) 185 | 186 | for t in v.types: 187 | if t not in mod.types and not t in builtin_types: 188 | v.errors += 1 189 | uses = ", ".join(v.types[t]) 190 | print('Undefined type {}, used in {}'.format(t, uses)) 191 | return not v.errors 192 | 193 | # The ASDL parser itself comes next. The only interesting external interface 194 | # here is the top-level parse function. 195 | 196 | def parse(filename): 197 | """Parse ASDL from the given file and return a Module node describing it.""" 198 | with open(filename) as f: 199 | parser = ASDLParser() 200 | return parser.parse(f.read()) 201 | 202 | # Types for describing tokens in an ASDL specification. 203 | class TokenKind: 204 | """TokenKind is provides a scope for enumerated token kinds.""" 205 | (ConstructorId, TypeId, Equals, Comma, Question, Pipe, Asterisk, 206 | LParen, RParen, LBrace, RBrace) = range(11) 207 | 208 | operator_table = { 209 | '=': Equals, ',': Comma, '?': Question, '|': Pipe, '(': LParen, 210 | ')': RParen, '*': Asterisk, '{': LBrace, '}': RBrace} 211 | 212 | Token = namedtuple('Token', 'kind value lineno') 213 | 214 | class ASDLSyntaxError(Exception): 215 | def __init__(self, msg, lineno=None): 216 | self.msg = msg 217 | self.lineno = lineno or '' 218 | 219 | def __str__(self): 220 | return 'Syntax error on line {0.lineno}: {0.msg}'.format(self) 221 | 222 | def tokenize_asdl(buf): 223 | """Tokenize the given buffer. Yield Token objects.""" 224 | for lineno, line in enumerate(buf.splitlines(), 1): 225 | for m in re.finditer(r'\s*(\w+|--.*|.)', line.strip()): 226 | c = m.group(1) 227 | if c[0].isalpha(): 228 | # Some kind of identifier 229 | if c[0].isupper(): 230 | yield Token(TokenKind.ConstructorId, c, lineno) 231 | else: 232 | yield Token(TokenKind.TypeId, c, lineno) 233 | elif c[:2] == '--': 234 | # Comment 235 | break 236 | else: 237 | # Operators 238 | try: 239 | op_kind = TokenKind.operator_table[c] 240 | except KeyError: 241 | raise ASDLSyntaxError('Invalid operator %s' % c, lineno) 242 | yield Token(op_kind, c, lineno) 243 | 244 | class ASDLParser: 245 | """Parser for ASDL files. 246 | 247 | Create, then call the parse method on a buffer containing ASDL. 248 | This is a simple recursive descent parser that uses tokenize_asdl for the 249 | lexing. 250 | """ 251 | def __init__(self): 252 | self._tokenizer = None 253 | self.cur_token = None 254 | 255 | def parse(self, buf): 256 | """Parse the ASDL in the buffer and return an AST with a Module root. 257 | """ 258 | self._tokenizer = tokenize_asdl(buf) 259 | self._advance() 260 | return self._parse_module() 261 | 262 | def _parse_module(self): 263 | if self._at_keyword('module'): 264 | self._advance() 265 | else: 266 | raise ASDLSyntaxError( 267 | 'Expected "module" (found {})'.format(self.cur_token.value), 268 | self.cur_token.lineno) 269 | name = self._match(self._id_kinds) 270 | self._match(TokenKind.LBrace) 271 | defs = self._parse_definitions() 272 | self._match(TokenKind.RBrace) 273 | return Module(name, defs) 274 | 275 | def _parse_definitions(self): 276 | defs = [] 277 | while self.cur_token.kind == TokenKind.TypeId: 278 | typename = self._advance() 279 | self._match(TokenKind.Equals) 280 | type = self._parse_type() 281 | defs.append(Type(typename, type)) 282 | return defs 283 | 284 | def _parse_type(self): 285 | if self.cur_token.kind == TokenKind.LParen: 286 | # If we see a (, it's a product 287 | return self._parse_product() 288 | else: 289 | # Otherwise it's a sum. Look for ConstructorId 290 | sumlist = [Constructor(self._match(TokenKind.ConstructorId), 291 | self._parse_optional_fields())] 292 | while self.cur_token.kind == TokenKind.Pipe: 293 | # More constructors 294 | self._advance() 295 | sumlist.append(Constructor( 296 | self._match(TokenKind.ConstructorId), 297 | self._parse_optional_fields())) 298 | return Sum(sumlist, self._parse_optional_attributes()) 299 | 300 | def _parse_product(self): 301 | return Product(self._parse_fields(), self._parse_optional_attributes()) 302 | 303 | def _parse_fields(self): 304 | fields = [] 305 | self._match(TokenKind.LParen) 306 | while self.cur_token.kind == TokenKind.TypeId: 307 | typename = self._advance() 308 | is_seq, is_opt = self._parse_optional_field_quantifier() 309 | id = (self._advance() if self.cur_token.kind in self._id_kinds 310 | else None) 311 | fields.append(Field(typename, id, seq=is_seq, opt=is_opt)) 312 | if self.cur_token.kind == TokenKind.RParen: 313 | break 314 | elif self.cur_token.kind == TokenKind.Comma: 315 | self._advance() 316 | self._match(TokenKind.RParen) 317 | return fields 318 | 319 | def _parse_optional_fields(self): 320 | if self.cur_token.kind == TokenKind.LParen: 321 | return self._parse_fields() 322 | else: 323 | return None 324 | 325 | def _parse_optional_attributes(self): 326 | if self._at_keyword('attributes'): 327 | self._advance() 328 | return self._parse_fields() 329 | else: 330 | return None 331 | 332 | def _parse_optional_field_quantifier(self): 333 | is_seq, is_opt = False, False 334 | if self.cur_token.kind == TokenKind.Asterisk: 335 | is_seq = True 336 | self._advance() 337 | elif self.cur_token.kind == TokenKind.Question: 338 | is_opt = True 339 | self._advance() 340 | return is_seq, is_opt 341 | 342 | def _advance(self): 343 | """ Return the value of the current token and read the next one into 344 | self.cur_token. 345 | """ 346 | cur_val = None if self.cur_token is None else self.cur_token.value 347 | try: 348 | self.cur_token = next(self._tokenizer) 349 | except StopIteration: 350 | self.cur_token = None 351 | return cur_val 352 | 353 | _id_kinds = (TokenKind.ConstructorId, TokenKind.TypeId) 354 | 355 | def _match(self, kind): 356 | """The 'match' primitive of RD parsers. 357 | 358 | * Verifies that the current token is of the given kind (kind can 359 | be a tuple, in which the kind must match one of its members). 360 | * Returns the value of the current token 361 | * Reads in the next token 362 | """ 363 | if (isinstance(kind, tuple) and self.cur_token.kind in kind or 364 | self.cur_token.kind == kind 365 | ): 366 | value = self.cur_token.value 367 | self._advance() 368 | return value 369 | else: 370 | raise ASDLSyntaxError( 371 | 'Unmatched {} (found {})'.format(kind, self.cur_token.kind), 372 | self.cur_token.lineno) 373 | 374 | def _at_keyword(self, keyword): 375 | return (self.cur_token.kind == TokenKind.TypeId and 376 | self.cur_token.value == keyword) 377 | -------------------------------------------------------------------------------- /asdl_c.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | """Generate C code from an ASDL description.""" 3 | 4 | import os, sys 5 | 6 | import asdl 7 | 8 | TABSIZE = 4 9 | MAX_COL = 80 10 | 11 | def get_c_type(name): 12 | """Return a string for the C name of the type. 13 | 14 | This function special cases the default types provided by asdl. 15 | """ 16 | if name in asdl.builtin_types: 17 | return name 18 | else: 19 | return "%s_ty" % name 20 | 21 | def reflow_lines(s, depth): 22 | """Reflow the line s indented depth tabs. 23 | 24 | Return a sequence of lines where no line extends beyond MAX_COL 25 | when properly indented. The first line is properly indented based 26 | exclusively on depth * TABSIZE. All following lines -- these are 27 | the reflowed lines generated by this function -- start at the same 28 | column as the first character beyond the opening { in the first 29 | line. 30 | """ 31 | size = MAX_COL - depth * TABSIZE 32 | if len(s) < size: 33 | return [s] 34 | 35 | lines = [] 36 | cur = s 37 | padding = "" 38 | while len(cur) > size: 39 | i = cur.rfind(' ', 0, size) 40 | # XXX this should be fixed for real 41 | if i == -1 and 'GeneratorExp' in cur: 42 | i = size + 3 43 | assert i != -1, "Impossible line %d to reflow: %r" % (size, s) 44 | lines.append(padding + cur[:i]) 45 | if len(lines) == 1: 46 | # find new size based on brace 47 | j = cur.find('{', 0, i) 48 | if j >= 0: 49 | j += 2 # account for the brace and the space after it 50 | size -= j 51 | padding = " " * j 52 | else: 53 | j = cur.find('(', 0, i) 54 | if j >= 0: 55 | j += 1 # account for the paren (no space after it) 56 | size -= j 57 | padding = " " * j 58 | cur = cur[i+1:] 59 | else: 60 | lines.append(padding + cur) 61 | return lines 62 | 63 | def is_simple(sum): 64 | """Return True if a sum is a simple. 65 | 66 | A sum is simple if its types have no fields, e.g. 67 | unaryop = Invert | Not | UAdd | USub 68 | """ 69 | for t in sum.types: 70 | if t.fields: 71 | return False 72 | return True 73 | 74 | 75 | class EmitVisitor(asdl.VisitorBase): 76 | """Visit that emits lines""" 77 | 78 | def __init__(self, file): 79 | self.file = file 80 | self.identifiers = set() 81 | super(EmitVisitor, self).__init__() 82 | 83 | def emit_identifier(self, name): 84 | name = str(name) 85 | if name in self.identifiers: 86 | return 87 | self.emit("_Py_IDENTIFIER(%s);" % name, 0) 88 | self.identifiers.add(name) 89 | 90 | def emit(self, s, depth, reflow=True): 91 | # XXX reflow long lines? 92 | if reflow: 93 | lines = reflow_lines(s, depth) 94 | else: 95 | lines = [s] 96 | for line in lines: 97 | line = (" " * TABSIZE * depth) + line + "\n" 98 | self.file.write(line) 99 | 100 | 101 | class TypeDefVisitor(EmitVisitor): 102 | def visitModule(self, mod): 103 | for dfn in mod.dfns: 104 | self.visit(dfn) 105 | 106 | def visitType(self, type, depth=0): 107 | self.visit(type.value, type.name, depth) 108 | 109 | def visitSum(self, sum, name, depth): 110 | if is_simple(sum): 111 | self.simple_sum(sum, name, depth) 112 | else: 113 | self.sum_with_constructors(sum, name, depth) 114 | 115 | def simple_sum(self, sum, name, depth): 116 | enum = [] 117 | for i in range(len(sum.types)): 118 | type = sum.types[i] 119 | enum.append("%s=%d" % (type.name, i + 1)) 120 | enums = ", ".join(enum) 121 | ctype = get_c_type(name) 122 | s = "typedef enum _%s { %s } %s;" % (name, enums, ctype) 123 | self.emit(s, depth) 124 | self.emit("", depth) 125 | 126 | def sum_with_constructors(self, sum, name, depth): 127 | ctype = get_c_type(name) 128 | s = "typedef struct _%(name)s *%(ctype)s;" % locals() 129 | self.emit(s, depth) 130 | self.emit("", depth) 131 | 132 | def visitProduct(self, product, name, depth): 133 | ctype = get_c_type(name) 134 | s = "typedef struct _%(name)s *%(ctype)s;" % locals() 135 | self.emit(s, depth) 136 | self.emit("", depth) 137 | 138 | 139 | class StructVisitor(EmitVisitor): 140 | """Visitor to generate typedefs for AST.""" 141 | 142 | def visitModule(self, mod): 143 | for dfn in mod.dfns: 144 | self.visit(dfn) 145 | 146 | def visitType(self, type, depth=0): 147 | self.visit(type.value, type.name, depth) 148 | 149 | def visitSum(self, sum, name, depth): 150 | if not is_simple(sum): 151 | self.sum_with_constructors(sum, name, depth) 152 | 153 | def sum_with_constructors(self, sum, name, depth): 154 | def emit(s, depth=depth): 155 | self.emit(s % sys._getframe(1).f_locals, depth) 156 | enum = [] 157 | for i in range(len(sum.types)): 158 | type = sum.types[i] 159 | enum.append("%s_kind=%d" % (type.name, i + 1)) 160 | 161 | emit("enum _%(name)s_kind {" + ", ".join(enum) + "};") 162 | 163 | emit("struct _%(name)s {") 164 | emit("enum _%(name)s_kind kind;", depth + 1) 165 | emit("union {", depth + 1) 166 | for t in sum.types: 167 | self.visit(t, depth + 2) 168 | emit("} v;", depth + 1) 169 | for field in sum.attributes: 170 | # rudimentary attribute handling 171 | type = str(field.type) 172 | assert type in asdl.builtin_types, type 173 | emit("%s %s;" % (type, field.name), depth + 1); 174 | emit("};") 175 | emit("") 176 | 177 | def visitConstructor(self, cons, depth): 178 | if cons.fields: 179 | self.emit("struct {", depth) 180 | for f in cons.fields: 181 | self.visit(f, depth + 1) 182 | self.emit("} %s;" % cons.name, depth) 183 | self.emit("", depth) 184 | 185 | def visitField(self, field, depth): 186 | # XXX need to lookup field.type, because it might be something 187 | # like a builtin... 188 | ctype = get_c_type(field.type) 189 | name = field.name 190 | if field.seq: 191 | if field.type == 'cmpop': 192 | self.emit("asdl_int_seq *%(name)s;" % locals(), depth) 193 | else: 194 | self.emit("asdl_seq *%(name)s;" % locals(), depth) 195 | else: 196 | self.emit("%(ctype)s %(name)s;" % locals(), depth) 197 | 198 | def visitProduct(self, product, name, depth): 199 | self.emit("struct _%(name)s {" % locals(), depth) 200 | for f in product.fields: 201 | self.visit(f, depth + 1) 202 | for field in product.attributes: 203 | # rudimentary attribute handling 204 | type = str(field.type) 205 | assert type in asdl.builtin_types, type 206 | self.emit("%s %s;" % (type, field.name), depth + 1); 207 | self.emit("};", depth) 208 | self.emit("", depth) 209 | 210 | 211 | class PrototypeVisitor(EmitVisitor): 212 | """Generate function prototypes for the .h file""" 213 | 214 | def visitModule(self, mod): 215 | for dfn in mod.dfns: 216 | self.visit(dfn) 217 | 218 | def visitType(self, type): 219 | self.visit(type.value, type.name) 220 | 221 | def visitSum(self, sum, name): 222 | if is_simple(sum): 223 | pass # XXX 224 | else: 225 | for t in sum.types: 226 | self.visit(t, name, sum.attributes) 227 | 228 | def get_args(self, fields): 229 | """Return list of C argument into, one for each field. 230 | 231 | Argument info is 3-tuple of a C type, variable name, and flag 232 | that is true if type can be NULL. 233 | """ 234 | args = [] 235 | unnamed = {} 236 | for f in fields: 237 | if f.name is None: 238 | name = f.type 239 | c = unnamed[name] = unnamed.get(name, 0) + 1 240 | if c > 1: 241 | name = "name%d" % (c - 1) 242 | else: 243 | name = f.name 244 | # XXX should extend get_c_type() to handle this 245 | if f.seq: 246 | if f.type == 'cmpop': 247 | ctype = "asdl_int_seq *" 248 | else: 249 | ctype = "asdl_seq *" 250 | else: 251 | ctype = get_c_type(f.type) 252 | args.append((ctype, name, f.opt or f.seq)) 253 | return args 254 | 255 | def visitConstructor(self, cons, type, attrs): 256 | args = self.get_args(cons.fields) 257 | attrs = self.get_args(attrs) 258 | ctype = get_c_type(type) 259 | self.emit_function(cons.name, ctype, args, attrs) 260 | 261 | def emit_function(self, name, ctype, args, attrs, union=True): 262 | args = args + attrs 263 | if args: 264 | argstr = ", ".join(["%s %s" % (atype, aname) 265 | for atype, aname, opt in args]) 266 | argstr += ", PyArena *arena" 267 | else: 268 | argstr = "PyArena *arena" 269 | margs = "a0" 270 | for i in range(1, len(args)+1): 271 | margs += ", a%d" % i 272 | self.emit("#define %s(%s) _Py_%s(%s)" % (name, margs, name, margs), 0, 273 | reflow=False) 274 | self.emit("%s _Py_%s(%s);" % (ctype, name, argstr), False) 275 | 276 | def visitProduct(self, prod, name): 277 | self.emit_function(name, get_c_type(name), 278 | self.get_args(prod.fields), [], union=False) 279 | 280 | 281 | class FunctionVisitor(PrototypeVisitor): 282 | """Visitor to generate constructor functions for AST.""" 283 | 284 | def emit_function(self, name, ctype, args, attrs, union=True): 285 | def emit(s, depth=0, reflow=True): 286 | self.emit(s, depth, reflow) 287 | argstr = ", ".join(["%s %s" % (atype, aname) 288 | for atype, aname, opt in args + attrs]) 289 | if argstr: 290 | argstr += ", PyArena *arena" 291 | else: 292 | argstr = "PyArena *arena" 293 | self.emit("%s" % ctype, 0) 294 | emit("%s(%s)" % (name, argstr)) 295 | emit("{") 296 | emit("%s p;" % ctype, 1) 297 | for argtype, argname, opt in args: 298 | if not opt and argtype != "int": 299 | emit("if (!%s) {" % argname, 1) 300 | emit("PyErr_SetString(PyExc_ValueError,", 2) 301 | msg = "field %s is required for %s" % (argname, name) 302 | emit(' "%s");' % msg, 303 | 2, reflow=False) 304 | emit('return NULL;', 2) 305 | emit('}', 1) 306 | 307 | emit("p = (%s)PyArena_Malloc(arena, sizeof(*p));" % ctype, 1); 308 | emit("if (!p)", 1) 309 | emit("return NULL;", 2) 310 | if union: 311 | self.emit_body_union(name, args, attrs) 312 | else: 313 | self.emit_body_struct(name, args, attrs) 314 | emit("return p;", 1) 315 | emit("}") 316 | emit("") 317 | 318 | def emit_body_union(self, name, args, attrs): 319 | def emit(s, depth=0, reflow=True): 320 | self.emit(s, depth, reflow) 321 | emit("p->kind = %s_kind;" % name, 1) 322 | for argtype, argname, opt in args: 323 | emit("p->v.%s.%s = %s;" % (name, argname, argname), 1) 324 | for argtype, argname, opt in attrs: 325 | emit("p->%s = %s;" % (argname, argname), 1) 326 | 327 | def emit_body_struct(self, name, args, attrs): 328 | def emit(s, depth=0, reflow=True): 329 | self.emit(s, depth, reflow) 330 | for argtype, argname, opt in args: 331 | emit("p->%s = %s;" % (argname, argname), 1) 332 | assert not attrs 333 | 334 | 335 | class PickleVisitor(EmitVisitor): 336 | 337 | def visitModule(self, mod): 338 | for dfn in mod.dfns: 339 | self.visit(dfn) 340 | 341 | def visitType(self, type): 342 | self.visit(type.value, type.name) 343 | 344 | def visitSum(self, sum, name): 345 | pass 346 | 347 | def visitProduct(self, sum, name): 348 | pass 349 | 350 | def visitConstructor(self, cons, name): 351 | pass 352 | 353 | def visitField(self, sum): 354 | pass 355 | 356 | 357 | class Obj2ModPrototypeVisitor(PickleVisitor): 358 | def visitProduct(self, prod, name): 359 | code = "static int obj2ast_%s(PyObject* obj, %s* out, PyArena* arena);" 360 | self.emit(code % (name, get_c_type(name)), 0) 361 | 362 | visitSum = visitProduct 363 | 364 | 365 | class Obj2ModVisitor(PickleVisitor): 366 | def funcHeader(self, name): 367 | ctype = get_c_type(name) 368 | self.emit("int", 0) 369 | self.emit("obj2ast_%s(PyObject* obj, %s* out, PyArena* arena)" % (name, ctype), 0) 370 | self.emit("{", 0) 371 | self.emit("int isinstance;", 1) 372 | self.emit("", 0) 373 | 374 | def sumTrailer(self, name, add_label=False): 375 | self.emit("", 0) 376 | # there's really nothing more we can do if this fails ... 377 | error = "expected some sort of %s, but got %%R" % name 378 | format = "PyErr_Format(PyExc_TypeError, \"%s\", obj);" 379 | self.emit(format % error, 1, reflow=False) 380 | if add_label: 381 | self.emit("failed:", 1) 382 | self.emit("Py_XDECREF(tmp);", 1) 383 | self.emit("return 1;", 1) 384 | self.emit("}", 0) 385 | self.emit("", 0) 386 | 387 | def simpleSum(self, sum, name): 388 | self.funcHeader(name) 389 | for t in sum.types: 390 | line = ("isinstance = PyObject_IsInstance(obj, " 391 | "(PyObject *)%s_type);") 392 | self.emit(line % (t.name,), 1) 393 | self.emit("if (isinstance == -1) {", 1) 394 | self.emit("return 1;", 2) 395 | self.emit("}", 1) 396 | self.emit("if (isinstance) {", 1) 397 | self.emit("*out = %s;" % t.name, 2) 398 | self.emit("return 0;", 2) 399 | self.emit("}", 1) 400 | self.sumTrailer(name) 401 | 402 | def buildArgs(self, fields): 403 | return ", ".join(fields + ["arena"]) 404 | 405 | def complexSum(self, sum, name): 406 | self.funcHeader(name) 407 | self.emit("PyObject *tmp = NULL;", 1) 408 | for a in sum.attributes: 409 | self.visitAttributeDeclaration(a, name, sum=sum) 410 | self.emit("", 0) 411 | # XXX: should we only do this for 'expr'? 412 | self.emit("if (obj == Py_None) {", 1) 413 | self.emit("*out = NULL;", 2) 414 | self.emit("return 0;", 2) 415 | self.emit("}", 1) 416 | for a in sum.attributes: 417 | self.visitField(a, name, sum=sum, depth=1) 418 | for t in sum.types: 419 | line = "isinstance = PyObject_IsInstance(obj, (PyObject*)%s_type);" 420 | self.emit(line % (t.name,), 1) 421 | self.emit("if (isinstance == -1) {", 1) 422 | self.emit("return 1;", 2) 423 | self.emit("}", 1) 424 | self.emit("if (isinstance) {", 1) 425 | for f in t.fields: 426 | self.visitFieldDeclaration(f, t.name, sum=sum, depth=2) 427 | self.emit("", 0) 428 | for f in t.fields: 429 | self.visitField(f, t.name, sum=sum, depth=2) 430 | args = [f.name for f in t.fields] + [a.name for a in sum.attributes] 431 | self.emit("*out = %s(%s);" % (t.name, self.buildArgs(args)), 2) 432 | self.emit("if (*out == NULL) goto failed;", 2) 433 | self.emit("return 0;", 2) 434 | self.emit("}", 1) 435 | self.sumTrailer(name, True) 436 | 437 | def visitAttributeDeclaration(self, a, name, sum=sum): 438 | ctype = get_c_type(a.type) 439 | self.emit("%s %s;" % (ctype, a.name), 1) 440 | 441 | def visitSum(self, sum, name): 442 | if is_simple(sum): 443 | self.simpleSum(sum, name) 444 | else: 445 | self.complexSum(sum, name) 446 | 447 | def visitProduct(self, prod, name): 448 | ctype = get_c_type(name) 449 | self.emit("int", 0) 450 | self.emit("obj2ast_%s(PyObject* obj, %s* out, PyArena* arena)" % (name, ctype), 0) 451 | self.emit("{", 0) 452 | self.emit("PyObject* tmp = NULL;", 1) 453 | for f in prod.fields: 454 | self.visitFieldDeclaration(f, name, prod=prod, depth=1) 455 | self.emit("", 0) 456 | for f in prod.fields: 457 | self.visitField(f, name, prod=prod, depth=1) 458 | args = [f.name for f in prod.fields] 459 | self.emit("*out = %s(%s);" % (name, self.buildArgs(args)), 1) 460 | self.emit("return 0;", 1) 461 | self.emit("failed:", 0) 462 | self.emit("Py_XDECREF(tmp);", 1) 463 | self.emit("return 1;", 1) 464 | self.emit("}", 0) 465 | self.emit("", 0) 466 | 467 | def visitFieldDeclaration(self, field, name, sum=None, prod=None, depth=0): 468 | ctype = get_c_type(field.type) 469 | if field.seq: 470 | if self.isSimpleType(field): 471 | self.emit("asdl_int_seq* %s;" % field.name, depth) 472 | else: 473 | self.emit("asdl_seq* %s;" % field.name, depth) 474 | else: 475 | ctype = get_c_type(field.type) 476 | self.emit("%s %s;" % (ctype, field.name), depth) 477 | 478 | def isSimpleSum(self, field): 479 | # XXX can the members of this list be determined automatically? 480 | return field.type in ('expr_context', 'boolop', 'operator', 481 | 'unaryop', 'cmpop') 482 | 483 | def isNumeric(self, field): 484 | return get_c_type(field.type) in ("int", "bool") 485 | 486 | def isSimpleType(self, field): 487 | return self.isSimpleSum(field) or self.isNumeric(field) 488 | 489 | def visitField(self, field, name, sum=None, prod=None, depth=0): 490 | ctype = get_c_type(field.type) 491 | if field.opt: 492 | check = "exists_not_none(obj, &PyId_%s)" % (field.name,) 493 | else: 494 | check = "_PyObject_HasAttrId(obj, &PyId_%s)" % (field.name,) 495 | self.emit("if (%s) {" % (check,), depth, reflow=False) 496 | self.emit("int res;", depth+1) 497 | if field.seq: 498 | self.emit("Py_ssize_t len;", depth+1) 499 | self.emit("Py_ssize_t i;", depth+1) 500 | self.emit("tmp = _PyObject_GetAttrId(obj, &PyId_%s);" % field.name, depth+1) 501 | self.emit("if (tmp == NULL) goto failed;", depth+1) 502 | if field.seq: 503 | self.emit("if (!PyList_Check(tmp)) {", depth+1) 504 | self.emit("PyErr_Format(PyExc_TypeError, \"%s field \\\"%s\\\" must " 505 | "be a list, not a %%.200s\", tmp->ob_type->tp_name);" % 506 | (name, field.name), 507 | depth+2, reflow=False) 508 | self.emit("goto failed;", depth+2) 509 | self.emit("}", depth+1) 510 | self.emit("len = PyList_GET_SIZE(tmp);", depth+1) 511 | if self.isSimpleType(field): 512 | self.emit("%s = _Py_asdl_int_seq_new(len, arena);" % field.name, depth+1) 513 | else: 514 | self.emit("%s = _Py_asdl_seq_new(len, arena);" % field.name, depth+1) 515 | self.emit("if (%s == NULL) goto failed;" % field.name, depth+1) 516 | self.emit("for (i = 0; i < len; i++) {", depth+1) 517 | self.emit("%s value;" % ctype, depth+2) 518 | self.emit("res = obj2ast_%s(PyList_GET_ITEM(tmp, i), &value, arena);" % 519 | field.type, depth+2, reflow=False) 520 | self.emit("if (res != 0) goto failed;", depth+2) 521 | self.emit("asdl_seq_SET(%s, i, value);" % field.name, depth+2) 522 | self.emit("}", depth+1) 523 | else: 524 | self.emit("res = obj2ast_%s(tmp, &%s, arena);" % 525 | (field.type, field.name), depth+1) 526 | self.emit("if (res != 0) goto failed;", depth+1) 527 | 528 | self.emit("Py_CLEAR(tmp);", depth+1) 529 | self.emit("} else {", depth) 530 | if not field.opt: 531 | message = "required field \\\"%s\\\" missing from %s" % (field.name, name) 532 | format = "PyErr_SetString(PyExc_TypeError, \"%s\");" 533 | self.emit(format % message, depth+1, reflow=False) 534 | self.emit("return 1;", depth+1) 535 | else: 536 | if self.isNumeric(field): 537 | self.emit("%s = 0;" % field.name, depth+1) 538 | elif not self.isSimpleType(field): 539 | self.emit("%s = NULL;" % field.name, depth+1) 540 | else: 541 | raise TypeError("could not determine the default value for %s" % field.name) 542 | self.emit("}", depth) 543 | 544 | 545 | class MarshalPrototypeVisitor(PickleVisitor): 546 | 547 | def prototype(self, sum, name): 548 | ctype = get_c_type(name) 549 | self.emit("static int marshal_write_%s(PyObject **, int *, %s);" 550 | % (name, ctype), 0) 551 | 552 | visitProduct = visitSum = prototype 553 | 554 | 555 | class PyTypesDeclareVisitor(PickleVisitor): 556 | 557 | def visitProduct(self, prod, name): 558 | self.emit("static PyTypeObject *%s_type;" % name, 0) 559 | self.emit("static PyObject* ast2obj_%s(void*);" % name, 0) 560 | if prod.attributes: 561 | for a in prod.attributes: 562 | self.emit_identifier(a.name) 563 | self.emit("static char *%s_attributes[] = {" % name, 0) 564 | for a in prod.attributes: 565 | self.emit('"%s",' % a.name, 1) 566 | self.emit("};", 0) 567 | if prod.fields: 568 | for f in prod.fields: 569 | self.emit_identifier(f.name) 570 | self.emit("static char *%s_fields[]={" % name,0) 571 | for f in prod.fields: 572 | self.emit('"%s",' % f.name, 1) 573 | self.emit("};", 0) 574 | 575 | def visitSum(self, sum, name): 576 | self.emit("static PyTypeObject *%s_type;" % name, 0) 577 | if sum.attributes: 578 | for a in sum.attributes: 579 | self.emit_identifier(a.name) 580 | self.emit("static char *%s_attributes[] = {" % name, 0) 581 | for a in sum.attributes: 582 | self.emit('"%s",' % a.name, 1) 583 | self.emit("};", 0) 584 | ptype = "void*" 585 | if is_simple(sum): 586 | ptype = get_c_type(name) 587 | tnames = [] 588 | for t in sum.types: 589 | tnames.append(str(t.name)+"_singleton") 590 | tnames = ", *".join(tnames) 591 | self.emit("static PyObject *%s;" % tnames, 0) 592 | self.emit("static PyObject* ast2obj_%s(%s);" % (name, ptype), 0) 593 | for t in sum.types: 594 | self.visitConstructor(t, name) 595 | 596 | def visitConstructor(self, cons, name): 597 | self.emit("static PyTypeObject *%s_type;" % cons.name, 0) 598 | if cons.fields: 599 | for t in cons.fields: 600 | self.emit_identifier(t.name) 601 | self.emit("static char *%s_fields[]={" % cons.name, 0) 602 | for t in cons.fields: 603 | self.emit('"%s",' % t.name, 1) 604 | self.emit("};",0) 605 | 606 | class PyTypesVisitor(PickleVisitor): 607 | 608 | def visitModule(self, mod): 609 | self.emit(""" 610 | typedef struct { 611 | PyObject_HEAD 612 | PyObject *dict; 613 | } AST_object; 614 | 615 | static void 616 | ast_dealloc(AST_object *self) 617 | { 618 | Py_CLEAR(self->dict); 619 | Py_TYPE(self)->tp_free(self); 620 | } 621 | 622 | static int 623 | ast_traverse(AST_object *self, visitproc visit, void *arg) 624 | { 625 | Py_VISIT(self->dict); 626 | return 0; 627 | } 628 | 629 | static void 630 | ast_clear(AST_object *self) 631 | { 632 | Py_CLEAR(self->dict); 633 | } 634 | 635 | static int 636 | ast_type_init(PyObject *self, PyObject *args, PyObject *kw) 637 | { 638 | _Py_IDENTIFIER(_fields); 639 | Py_ssize_t i, numfields = 0; 640 | int res = -1; 641 | PyObject *key, *value, *fields; 642 | fields = _PyObject_GetAttrId((PyObject*)Py_TYPE(self), &PyId__fields); 643 | if (!fields) 644 | PyErr_Clear(); 645 | if (fields) { 646 | numfields = PySequence_Size(fields); 647 | if (numfields == -1) 648 | goto cleanup; 649 | } 650 | res = 0; /* if no error occurs, this stays 0 to the end */ 651 | if (PyTuple_GET_SIZE(args) > 0) { 652 | if (numfields != PyTuple_GET_SIZE(args)) { 653 | PyErr_Format(PyExc_TypeError, "%.400s constructor takes %s" 654 | "%zd positional argument%s", 655 | Py_TYPE(self)->tp_name, 656 | numfields == 0 ? "" : "either 0 or ", 657 | numfields, numfields == 1 ? "" : "s"); 658 | res = -1; 659 | goto cleanup; 660 | } 661 | for (i = 0; i < PyTuple_GET_SIZE(args); i++) { 662 | /* cannot be reached when fields is NULL */ 663 | PyObject *name = PySequence_GetItem(fields, i); 664 | if (!name) { 665 | res = -1; 666 | goto cleanup; 667 | } 668 | res = PyObject_SetAttr(self, name, PyTuple_GET_ITEM(args, i)); 669 | Py_DECREF(name); 670 | if (res < 0) 671 | goto cleanup; 672 | } 673 | } 674 | if (kw) { 675 | i = 0; /* needed by PyDict_Next */ 676 | while (PyDict_Next(kw, &i, &key, &value)) { 677 | res = PyObject_SetAttr(self, key, value); 678 | if (res < 0) 679 | goto cleanup; 680 | } 681 | } 682 | cleanup: 683 | Py_XDECREF(fields); 684 | return res; 685 | } 686 | 687 | /* Pickling support */ 688 | static PyObject * 689 | ast_type_reduce(PyObject *self, PyObject *unused) 690 | { 691 | PyObject *res; 692 | _Py_IDENTIFIER(__dict__); 693 | PyObject *dict = _PyObject_GetAttrId(self, &PyId___dict__); 694 | if (dict == NULL) { 695 | if (PyErr_ExceptionMatches(PyExc_AttributeError)) 696 | PyErr_Clear(); 697 | else 698 | return NULL; 699 | } 700 | if (dict) { 701 | res = Py_BuildValue("O()O", Py_TYPE(self), dict); 702 | Py_DECREF(dict); 703 | return res; 704 | } 705 | return Py_BuildValue("O()", Py_TYPE(self)); 706 | } 707 | 708 | static PyMethodDef ast_type_methods[] = { 709 | {"__reduce__", ast_type_reduce, METH_NOARGS, NULL}, 710 | {NULL} 711 | }; 712 | 713 | static PyGetSetDef ast_type_getsets[] = { 714 | {"__dict__", PyObject_GenericGetDict, PyObject_GenericSetDict}, 715 | {NULL} 716 | }; 717 | 718 | static PyTypeObject AST_type = { 719 | PyVarObject_HEAD_INIT(&PyType_Type, 0) 720 | "_ast.AST", 721 | sizeof(AST_object), 722 | 0, 723 | (destructor)ast_dealloc, /* tp_dealloc */ 724 | 0, /* tp_print */ 725 | 0, /* tp_getattr */ 726 | 0, /* tp_setattr */ 727 | 0, /* tp_reserved */ 728 | 0, /* tp_repr */ 729 | 0, /* tp_as_number */ 730 | 0, /* tp_as_sequence */ 731 | 0, /* tp_as_mapping */ 732 | 0, /* tp_hash */ 733 | 0, /* tp_call */ 734 | 0, /* tp_str */ 735 | PyObject_GenericGetAttr, /* tp_getattro */ 736 | PyObject_GenericSetAttr, /* tp_setattro */ 737 | 0, /* tp_as_buffer */ 738 | Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 739 | 0, /* tp_doc */ 740 | (traverseproc)ast_traverse, /* tp_traverse */ 741 | (inquiry)ast_clear, /* tp_clear */ 742 | 0, /* tp_richcompare */ 743 | 0, /* tp_weaklistoffset */ 744 | 0, /* tp_iter */ 745 | 0, /* tp_iternext */ 746 | ast_type_methods, /* tp_methods */ 747 | 0, /* tp_members */ 748 | ast_type_getsets, /* tp_getset */ 749 | 0, /* tp_base */ 750 | 0, /* tp_dict */ 751 | 0, /* tp_descr_get */ 752 | 0, /* tp_descr_set */ 753 | offsetof(AST_object, dict),/* tp_dictoffset */ 754 | (initproc)ast_type_init, /* tp_init */ 755 | PyType_GenericAlloc, /* tp_alloc */ 756 | PyType_GenericNew, /* tp_new */ 757 | PyObject_GC_Del, /* tp_free */ 758 | }; 759 | 760 | 761 | static PyTypeObject* make_type(char *type, PyTypeObject* base, char**fields, int num_fields) 762 | { 763 | PyObject *fnames, *result; 764 | int i; 765 | fnames = PyTuple_New(num_fields); 766 | if (!fnames) return NULL; 767 | for (i = 0; i < num_fields; i++) { 768 | PyObject *field = PyUnicode_FromString(fields[i]); 769 | if (!field) { 770 | Py_DECREF(fnames); 771 | return NULL; 772 | } 773 | PyTuple_SET_ITEM(fnames, i, field); 774 | } 775 | result = PyObject_CallFunction((PyObject*)&PyType_Type, "s(O){sOss}", 776 | type, base, "_fields", fnames, "__module__", "_ast"); 777 | Py_DECREF(fnames); 778 | return (PyTypeObject*)result; 779 | } 780 | 781 | static int add_attributes(PyTypeObject* type, char**attrs, int num_fields) 782 | { 783 | int i, result; 784 | _Py_IDENTIFIER(_attributes); 785 | PyObject *s, *l = PyTuple_New(num_fields); 786 | if (!l) 787 | return 0; 788 | for (i = 0; i < num_fields; i++) { 789 | s = PyUnicode_FromString(attrs[i]); 790 | if (!s) { 791 | Py_DECREF(l); 792 | return 0; 793 | } 794 | PyTuple_SET_ITEM(l, i, s); 795 | } 796 | result = _PyObject_SetAttrId((PyObject*)type, &PyId__attributes, l) >= 0; 797 | Py_DECREF(l); 798 | return result; 799 | } 800 | 801 | /* Conversion AST -> Python */ 802 | 803 | static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) 804 | { 805 | Py_ssize_t i, n = asdl_seq_LEN(seq); 806 | PyObject *result = PyList_New(n); 807 | PyObject *value; 808 | if (!result) 809 | return NULL; 810 | for (i = 0; i < n; i++) { 811 | value = func(asdl_seq_GET(seq, i)); 812 | if (!value) { 813 | Py_DECREF(result); 814 | return NULL; 815 | } 816 | PyList_SET_ITEM(result, i, value); 817 | } 818 | return result; 819 | } 820 | 821 | static PyObject* ast2obj_object(void *o) 822 | { 823 | if (!o) 824 | o = Py_None; 825 | Py_INCREF((PyObject*)o); 826 | return (PyObject*)o; 827 | } 828 | #define ast2obj_singleton ast2obj_object 829 | #define ast2obj_identifier ast2obj_object 830 | #define ast2obj_string ast2obj_object 831 | #define ast2obj_bytes ast2obj_object 832 | 833 | static PyObject* ast2obj_int(long b) 834 | { 835 | return PyLong_FromLong(b); 836 | } 837 | 838 | /* Conversion Python -> AST */ 839 | 840 | static int obj2ast_singleton(PyObject *obj, PyObject** out, PyArena* arena) 841 | { 842 | if (obj != Py_None && obj != Py_True && obj != Py_False) { 843 | PyErr_SetString(PyExc_ValueError, 844 | "AST singleton must be True, False, or None"); 845 | return 1; 846 | } 847 | *out = obj; 848 | return 0; 849 | } 850 | 851 | static int obj2ast_object(PyObject* obj, PyObject** out, PyArena* arena) 852 | { 853 | if (obj == Py_None) 854 | obj = NULL; 855 | if (obj) { 856 | if (PyArena_AddPyObject(arena, obj) < 0) { 857 | *out = NULL; 858 | return -1; 859 | } 860 | Py_INCREF(obj); 861 | } 862 | *out = obj; 863 | return 0; 864 | } 865 | 866 | static int obj2ast_identifier(PyObject* obj, PyObject** out, PyArena* arena) 867 | { 868 | if (!PyUnicode_CheckExact(obj) && obj != Py_None) { 869 | PyErr_SetString(PyExc_TypeError, "AST identifier must be of type str"); 870 | return 1; 871 | } 872 | return obj2ast_object(obj, out, arena); 873 | } 874 | 875 | static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena) 876 | { 877 | if (!PyUnicode_CheckExact(obj) && !PyBytes_CheckExact(obj)) { 878 | PyErr_SetString(PyExc_TypeError, "AST string must be of type str"); 879 | return 1; 880 | } 881 | return obj2ast_object(obj, out, arena); 882 | } 883 | 884 | static int obj2ast_bytes(PyObject* obj, PyObject** out, PyArena* arena) 885 | { 886 | if (!PyBytes_CheckExact(obj)) { 887 | PyErr_SetString(PyExc_TypeError, "AST bytes must be of type bytes"); 888 | return 1; 889 | } 890 | return obj2ast_object(obj, out, arena); 891 | } 892 | 893 | static int obj2ast_int(PyObject* obj, int* out, PyArena* arena) 894 | { 895 | int i; 896 | if (!PyLong_Check(obj)) { 897 | PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj); 898 | return 1; 899 | } 900 | 901 | i = (int)PyLong_AsLong(obj); 902 | if (i == -1 && PyErr_Occurred()) 903 | return 1; 904 | *out = i; 905 | return 0; 906 | } 907 | 908 | static int add_ast_fields(void) 909 | { 910 | PyObject *empty_tuple, *d; 911 | if (PyType_Ready(&AST_type) < 0) 912 | return -1; 913 | d = AST_type.tp_dict; 914 | empty_tuple = PyTuple_New(0); 915 | if (!empty_tuple || 916 | PyDict_SetItemString(d, "_fields", empty_tuple) < 0 || 917 | PyDict_SetItemString(d, "_attributes", empty_tuple) < 0) { 918 | Py_XDECREF(empty_tuple); 919 | return -1; 920 | } 921 | Py_DECREF(empty_tuple); 922 | return 0; 923 | } 924 | 925 | static int exists_not_none(PyObject *obj, _Py_Identifier *id) 926 | { 927 | int isnone; 928 | PyObject *attr = _PyObject_GetAttrId(obj, id); 929 | if (!attr) { 930 | PyErr_Clear(); 931 | return 0; 932 | } 933 | isnone = attr == Py_None; 934 | Py_DECREF(attr); 935 | return !isnone; 936 | } 937 | 938 | """, 0, reflow=False) 939 | 940 | self.emit("static int init_types(void)",0) 941 | self.emit("{", 0) 942 | self.emit("static int initialized;", 1) 943 | self.emit("if (initialized) return 1;", 1) 944 | self.emit("if (add_ast_fields() < 0) return 0;", 1) 945 | for dfn in mod.dfns: 946 | self.visit(dfn) 947 | self.emit("initialized = 1;", 1) 948 | self.emit("return 1;", 1); 949 | self.emit("}", 0) 950 | 951 | def visitProduct(self, prod, name): 952 | if prod.fields: 953 | fields = name+"_fields" 954 | else: 955 | fields = "NULL" 956 | self.emit('%s_type = make_type("%s", &AST_type, %s, %d);' % 957 | (name, name, fields, len(prod.fields)), 1) 958 | self.emit("if (!%s_type) return 0;" % name, 1) 959 | if prod.attributes: 960 | self.emit("if (!add_attributes(%s_type, %s_attributes, %d)) return 0;" % 961 | (name, name, len(prod.attributes)), 1) 962 | else: 963 | self.emit("if (!add_attributes(%s_type, NULL, 0)) return 0;" % name, 1) 964 | 965 | def visitSum(self, sum, name): 966 | self.emit('%s_type = make_type("%s", &AST_type, NULL, 0);' % 967 | (name, name), 1) 968 | self.emit("if (!%s_type) return 0;" % name, 1) 969 | if sum.attributes: 970 | self.emit("if (!add_attributes(%s_type, %s_attributes, %d)) return 0;" % 971 | (name, name, len(sum.attributes)), 1) 972 | else: 973 | self.emit("if (!add_attributes(%s_type, NULL, 0)) return 0;" % name, 1) 974 | simple = is_simple(sum) 975 | for t in sum.types: 976 | self.visitConstructor(t, name, simple) 977 | 978 | def visitConstructor(self, cons, name, simple): 979 | if cons.fields: 980 | fields = cons.name+"_fields" 981 | else: 982 | fields = "NULL" 983 | self.emit('%s_type = make_type("%s", %s_type, %s, %d);' % 984 | (cons.name, cons.name, name, fields, len(cons.fields)), 1) 985 | self.emit("if (!%s_type) return 0;" % cons.name, 1) 986 | if simple: 987 | self.emit("%s_singleton = PyType_GenericNew(%s_type, NULL, NULL);" % 988 | (cons.name, cons.name), 1) 989 | self.emit("if (!%s_singleton) return 0;" % cons.name, 1) 990 | 991 | 992 | class ASTModuleVisitor(PickleVisitor): 993 | 994 | def visitModule(self, mod): 995 | self.emit("static struct PyModuleDef _astmodule = {", 0) 996 | self.emit(' PyModuleDef_HEAD_INIT, "_ast"', 0) 997 | self.emit("};", 0) 998 | self.emit("PyMODINIT_FUNC", 0) 999 | self.emit("PyInit__ast(void)", 0) 1000 | self.emit("{", 0) 1001 | self.emit("PyObject *m, *d;", 1) 1002 | self.emit("if (!init_types()) return NULL;", 1) 1003 | self.emit('m = PyModule_Create(&_astmodule);', 1) 1004 | self.emit("if (!m) return NULL;", 1) 1005 | self.emit("d = PyModule_GetDict(m);", 1) 1006 | self.emit('if (PyDict_SetItemString(d, "AST", (PyObject*)&AST_type) < 0) return NULL;', 1) 1007 | self.emit('if (PyModule_AddIntMacro(m, PyCF_ONLY_AST) < 0)', 1) 1008 | self.emit("return NULL;", 2) 1009 | for dfn in mod.dfns: 1010 | self.visit(dfn) 1011 | self.emit("return m;", 1) 1012 | self.emit("}", 0) 1013 | 1014 | def visitProduct(self, prod, name): 1015 | self.addObj(name) 1016 | 1017 | def visitSum(self, sum, name): 1018 | self.addObj(name) 1019 | for t in sum.types: 1020 | self.visitConstructor(t, name) 1021 | 1022 | def visitConstructor(self, cons, name): 1023 | self.addObj(cons.name) 1024 | 1025 | def addObj(self, name): 1026 | self.emit('if (PyDict_SetItemString(d, "%s", (PyObject*)%s_type) < 0) return NULL;' % (name, name), 1) 1027 | 1028 | 1029 | _SPECIALIZED_SEQUENCES = ('stmt', 'expr') 1030 | 1031 | def find_sequence(fields, doing_specialization): 1032 | """Return True if any field uses a sequence.""" 1033 | for f in fields: 1034 | if f.seq: 1035 | if not doing_specialization: 1036 | return True 1037 | if str(f.type) not in _SPECIALIZED_SEQUENCES: 1038 | return True 1039 | return False 1040 | 1041 | def has_sequence(types, doing_specialization): 1042 | for t in types: 1043 | if find_sequence(t.fields, doing_specialization): 1044 | return True 1045 | return False 1046 | 1047 | 1048 | class StaticVisitor(PickleVisitor): 1049 | CODE = '''Very simple, always emit this static code. Override CODE''' 1050 | 1051 | def visit(self, object): 1052 | self.emit(self.CODE, 0, reflow=False) 1053 | 1054 | 1055 | class ObjVisitor(PickleVisitor): 1056 | 1057 | def func_begin(self, name): 1058 | ctype = get_c_type(name) 1059 | self.emit("PyObject*", 0) 1060 | self.emit("ast2obj_%s(void* _o)" % (name), 0) 1061 | self.emit("{", 0) 1062 | self.emit("%s o = (%s)_o;" % (ctype, ctype), 1) 1063 | self.emit("PyObject *result = NULL, *value = NULL;", 1) 1064 | self.emit('if (!o) {', 1) 1065 | self.emit("Py_INCREF(Py_None);", 2) 1066 | self.emit('return Py_None;', 2) 1067 | self.emit("}", 1) 1068 | self.emit('', 0) 1069 | 1070 | def func_end(self): 1071 | self.emit("return result;", 1) 1072 | self.emit("failed:", 0) 1073 | self.emit("Py_XDECREF(value);", 1) 1074 | self.emit("Py_XDECREF(result);", 1) 1075 | self.emit("return NULL;", 1) 1076 | self.emit("}", 0) 1077 | self.emit("", 0) 1078 | 1079 | def visitSum(self, sum, name): 1080 | if is_simple(sum): 1081 | self.simpleSum(sum, name) 1082 | return 1083 | self.func_begin(name) 1084 | self.emit("switch (o->kind) {", 1) 1085 | for i in range(len(sum.types)): 1086 | t = sum.types[i] 1087 | self.visitConstructor(t, i + 1, name) 1088 | self.emit("}", 1) 1089 | for a in sum.attributes: 1090 | self.emit("value = ast2obj_%s(o->%s);" % (a.type, a.name), 1) 1091 | self.emit("if (!value) goto failed;", 1) 1092 | self.emit('if (_PyObject_SetAttrId(result, &PyId_%s, value) < 0)' % a.name, 1) 1093 | self.emit('goto failed;', 2) 1094 | self.emit('Py_DECREF(value);', 1) 1095 | self.func_end() 1096 | 1097 | def simpleSum(self, sum, name): 1098 | self.emit("PyObject* ast2obj_%s(%s_ty o)" % (name, name), 0) 1099 | self.emit("{", 0) 1100 | self.emit("switch(o) {", 1) 1101 | for t in sum.types: 1102 | self.emit("case %s:" % t.name, 2) 1103 | self.emit("Py_INCREF(%s_singleton);" % t.name, 3) 1104 | self.emit("return %s_singleton;" % t.name, 3) 1105 | self.emit("default:", 2) 1106 | self.emit('/* should never happen, but just in case ... */', 3) 1107 | code = "PyErr_Format(PyExc_SystemError, \"unknown %s found\");" % name 1108 | self.emit(code, 3, reflow=False) 1109 | self.emit("return NULL;", 3) 1110 | self.emit("}", 1) 1111 | self.emit("}", 0) 1112 | 1113 | def visitProduct(self, prod, name): 1114 | self.func_begin(name) 1115 | self.emit("result = PyType_GenericNew(%s_type, NULL, NULL);" % name, 1); 1116 | self.emit("if (!result) return NULL;", 1) 1117 | for field in prod.fields: 1118 | self.visitField(field, name, 1, True) 1119 | for a in prod.attributes: 1120 | self.emit("value = ast2obj_%s(o->%s);" % (a.type, a.name), 1) 1121 | self.emit("if (!value) goto failed;", 1) 1122 | self.emit('if (_PyObject_SetAttrId(result, &PyId_%s, value) < 0)' % a.name, 1) 1123 | self.emit('goto failed;', 2) 1124 | self.emit('Py_DECREF(value);', 1) 1125 | self.func_end() 1126 | 1127 | def visitConstructor(self, cons, enum, name): 1128 | self.emit("case %s_kind:" % cons.name, 1) 1129 | self.emit("result = PyType_GenericNew(%s_type, NULL, NULL);" % cons.name, 2); 1130 | self.emit("if (!result) goto failed;", 2) 1131 | for f in cons.fields: 1132 | self.visitField(f, cons.name, 2, False) 1133 | self.emit("break;", 2) 1134 | 1135 | def visitField(self, field, name, depth, product): 1136 | def emit(s, d): 1137 | self.emit(s, depth + d) 1138 | if product: 1139 | value = "o->%s" % field.name 1140 | else: 1141 | value = "o->v.%s.%s" % (name, field.name) 1142 | self.set(field, value, depth) 1143 | emit("if (!value) goto failed;", 0) 1144 | emit('if (_PyObject_SetAttrId(result, &PyId_%s, value) == -1)' % field.name, 0) 1145 | emit("goto failed;", 1) 1146 | emit("Py_DECREF(value);", 0) 1147 | 1148 | def emitSeq(self, field, value, depth, emit): 1149 | emit("seq = %s;" % value, 0) 1150 | emit("n = asdl_seq_LEN(seq);", 0) 1151 | emit("value = PyList_New(n);", 0) 1152 | emit("if (!value) goto failed;", 0) 1153 | emit("for (i = 0; i < n; i++) {", 0) 1154 | self.set("value", field, "asdl_seq_GET(seq, i)", depth + 1) 1155 | emit("if (!value1) goto failed;", 1) 1156 | emit("PyList_SET_ITEM(value, i, value1);", 1) 1157 | emit("value1 = NULL;", 1) 1158 | emit("}", 0) 1159 | 1160 | def set(self, field, value, depth): 1161 | if field.seq: 1162 | # XXX should really check for is_simple, but that requires a symbol table 1163 | if field.type == "cmpop": 1164 | # While the sequence elements are stored as void*, 1165 | # ast2obj_cmpop expects an enum 1166 | self.emit("{", depth) 1167 | self.emit("Py_ssize_t i, n = asdl_seq_LEN(%s);" % value, depth+1) 1168 | self.emit("value = PyList_New(n);", depth+1) 1169 | self.emit("if (!value) goto failed;", depth+1) 1170 | self.emit("for(i = 0; i < n; i++)", depth+1) 1171 | # This cannot fail, so no need for error handling 1172 | self.emit("PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)asdl_seq_GET(%s, i)));" % value, 1173 | depth+2, reflow=False) 1174 | self.emit("}", depth) 1175 | else: 1176 | self.emit("value = ast2obj_list(%s, ast2obj_%s);" % (value, field.type), depth) 1177 | else: 1178 | ctype = get_c_type(field.type) 1179 | self.emit("value = ast2obj_%s(%s);" % (field.type, value), depth, reflow=False) 1180 | 1181 | 1182 | class PartingShots(StaticVisitor): 1183 | 1184 | CODE = """ 1185 | PyObject* PyAST_mod2obj(mod_ty t) 1186 | { 1187 | if (!init_types()) 1188 | return NULL; 1189 | return ast2obj_mod(t); 1190 | } 1191 | 1192 | /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */ 1193 | mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode) 1194 | { 1195 | mod_ty res; 1196 | PyObject *req_type[3]; 1197 | char *req_name[] = {"Module", "Expression", "Interactive"}; 1198 | int isinstance; 1199 | 1200 | req_type[0] = (PyObject*)Module_type; 1201 | req_type[1] = (PyObject*)Expression_type; 1202 | req_type[2] = (PyObject*)Interactive_type; 1203 | 1204 | assert(0 <= mode && mode <= 2); 1205 | 1206 | if (!init_types()) 1207 | return NULL; 1208 | 1209 | isinstance = PyObject_IsInstance(ast, req_type[mode]); 1210 | if (isinstance == -1) 1211 | return NULL; 1212 | if (!isinstance) { 1213 | PyErr_Format(PyExc_TypeError, "expected %s node, got %.400s", 1214 | req_name[mode], Py_TYPE(ast)->tp_name); 1215 | return NULL; 1216 | } 1217 | if (obj2ast_mod(ast, &res, arena) != 0) 1218 | return NULL; 1219 | else 1220 | return res; 1221 | } 1222 | 1223 | int PyAST_Check(PyObject* obj) 1224 | { 1225 | if (!init_types()) 1226 | return -1; 1227 | return PyObject_IsInstance(obj, (PyObject*)&AST_type); 1228 | } 1229 | """ 1230 | 1231 | class ChainOfVisitors: 1232 | def __init__(self, *visitors): 1233 | self.visitors = visitors 1234 | 1235 | def visit(self, object): 1236 | for v in self.visitors: 1237 | v.visit(object) 1238 | v.emit("", 0) 1239 | 1240 | common_msg = "/* File automatically generated by %s. */\n\n" 1241 | 1242 | def main(srcfile, dump_module=False): 1243 | argv0 = sys.argv[0] 1244 | components = argv0.split(os.sep) 1245 | argv0 = os.sep.join(components[-2:]) 1246 | auto_gen_msg = common_msg % argv0 1247 | mod = asdl.parse(srcfile) 1248 | if dump_module: 1249 | print('Parsed Module:') 1250 | print(mod) 1251 | if not asdl.check(mod): 1252 | sys.exit(1) 1253 | if INC_DIR: 1254 | p = "%s/%s-ast.h" % (INC_DIR, mod.name) 1255 | f = open(p, "w") 1256 | f.write(auto_gen_msg) 1257 | f.write('#include "asdl.h"\n\n') 1258 | c = ChainOfVisitors(TypeDefVisitor(f), 1259 | StructVisitor(f), 1260 | PrototypeVisitor(f), 1261 | ) 1262 | c.visit(mod) 1263 | f.write("PyObject* PyAST_mod2obj(mod_ty t);\n") 1264 | f.write("mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode);\n") 1265 | f.write("int PyAST_Check(PyObject* obj);\n") 1266 | f.close() 1267 | 1268 | if SRC_DIR: 1269 | p = os.path.join(SRC_DIR, str(mod.name) + "-ast.c") 1270 | f = open(p, "w") 1271 | f.write(auto_gen_msg) 1272 | f.write('#include \n') 1273 | f.write('\n') 1274 | f.write('#include "Python.h"\n') 1275 | f.write('#include "%s-ast.h"\n' % mod.name) 1276 | f.write('\n') 1277 | f.write("static PyTypeObject AST_type;\n") 1278 | v = ChainOfVisitors( 1279 | PyTypesDeclareVisitor(f), 1280 | PyTypesVisitor(f), 1281 | Obj2ModPrototypeVisitor(f), 1282 | FunctionVisitor(f), 1283 | ObjVisitor(f), 1284 | Obj2ModVisitor(f), 1285 | ASTModuleVisitor(f), 1286 | PartingShots(f), 1287 | ) 1288 | v.visit(mod) 1289 | f.close() 1290 | 1291 | if __name__ == "__main__": 1292 | import sys 1293 | import getopt 1294 | 1295 | INC_DIR = '' 1296 | SRC_DIR = '' 1297 | dump_module = False 1298 | opts, args = getopt.getopt(sys.argv[1:], "dh:c:") 1299 | for o, v in opts: 1300 | if o == '-h': 1301 | INC_DIR = v 1302 | if o == '-c': 1303 | SRC_DIR = v 1304 | if o == '-d': 1305 | dump_module = True 1306 | if INC_DIR and SRC_DIR: 1307 | print('Must specify exactly one output file') 1308 | sys.exit(1) 1309 | elif len(args) != 1: 1310 | print('Must specify single input file') 1311 | sys.exit(1) 1312 | main(args[0], dump_module) 1313 | -------------------------------------------------------------------------------- /asdl_test.py: -------------------------------------------------------------------------------- 1 | # Simple testing / sanity-checking for asdl.py 2 | # Assumes some things about the current Python.asdl, which is used as input. 3 | 4 | import sys, unittest 5 | import asdl 6 | 7 | 8 | class TestAsdlParser(unittest.TestCase): 9 | @classmethod 10 | def setUpClass(cls): 11 | # Parse Python.asdl into a ast.Module and run the check on it. 12 | # There's no need to do this for each test method, hence setUpClass. 13 | cls.mod = asdl.parse('./Python.asdl') 14 | cls.assertTrue(asdl.check(cls.mod), 'Module validation failed') 15 | 16 | def setUp(self): 17 | # alias stuff from the class, for convenience 18 | self.mod = TestAsdlParser.mod 19 | self.types = self.mod.types 20 | 21 | def test_module(self): 22 | self.assertEqual(self.mod.name, 'Python') 23 | self.assertIn('stmt', self.types) 24 | self.assertIn('expr', self.types) 25 | self.assertIn('mod', self.types) 26 | 27 | def test_definitions(self): 28 | defs = self.mod.dfns 29 | self.assertIsInstance(defs[0], asdl.Type) 30 | self.assertIsInstance(defs[0].value, asdl.Sum) 31 | 32 | self.assertIsInstance(self.types['withitem'], asdl.Product) 33 | self.assertIsInstance(self.types['alias'], asdl.Product) 34 | 35 | def test_product(self): 36 | alias = self.types['alias'] 37 | self.assertEqual( 38 | str(alias), 39 | 'Product([Field(identifier, name), Field(identifier, asname, opt=True)])') 40 | 41 | def test_attributes(self): 42 | stmt = self.types['stmt'] 43 | self.assertEqual(len(stmt.attributes), 2) 44 | self.assertEqual(str(stmt.attributes[0]), 'Field(int, lineno)') 45 | self.assertEqual(str(stmt.attributes[1]), 'Field(int, col_offset)') 46 | 47 | def test_constructor_fields(self): 48 | ehandler = self.types['excepthandler'] 49 | self.assertEqual(len(ehandler.types), 1) 50 | self.assertEqual(len(ehandler.attributes), 2) 51 | 52 | cons = ehandler.types[0] 53 | self.assertIsInstance(cons, asdl.Constructor) 54 | self.assertEqual(len(cons.fields), 3) 55 | 56 | f0 = cons.fields[0] 57 | self.assertEqual(f0.type, 'expr') 58 | self.assertEqual(f0.name, 'type') 59 | self.assertTrue(f0.opt) 60 | 61 | f1 = cons.fields[1] 62 | self.assertEqual(f1.type, 'identifier') 63 | self.assertEqual(f1.name, 'name') 64 | self.assertTrue(f1.opt) 65 | 66 | f2 = cons.fields[2] 67 | self.assertEqual(f2.type, 'stmt') 68 | self.assertEqual(f2.name, 'body') 69 | self.assertFalse(f2.opt) 70 | self.assertTrue(f2.seq) 71 | 72 | def test_visitor(self): 73 | class CustomVisitor(asdl.VisitorBase): 74 | def __init__(self): 75 | super().__init__() 76 | self.names_with_seq = [] 77 | 78 | def visitModule(self, mod): 79 | for dfn in mod.dfns: 80 | self.visit(dfn) 81 | 82 | def visitType(self, type): 83 | self.visit(type.value) 84 | 85 | def visitSum(self, sum): 86 | for t in sum.types: 87 | self.visit(t) 88 | 89 | def visitConstructor(self, cons): 90 | for f in cons.fields: 91 | if f.seq: 92 | self.names_with_seq.append(cons.name) 93 | 94 | v = CustomVisitor() 95 | v.visit(self.types['mod']) 96 | self.assertEqual(v.names_with_seq, ['Module', 'Interactive', 'Suite']) 97 | 98 | 99 | if __name__ == '__main__': 100 | unittest.main() 101 | -------------------------------------------------------------------------------- /docs/ASDL Zephyr - Wang.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eliben/asdl_parser/44a2ca40b39712dc6f05ec3e57570c2d7199670a/docs/ASDL Zephyr - Wang.pdf -------------------------------------------------------------------------------- /gen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -eu 3 | set -x 4 | 5 | python3 asdl_c.py -h /tmp Python.asdl 6 | python3 asdl_c.py -c /tmp Python.asdl 7 | --------------------------------------------------------------------------------