├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── cparser-scm-1.rockspec ├── cparser.lua ├── lcdecl ├── lcpp └── tests ├── README ├── spec └── cparser_spec.lua ├── testbit.lua ├── testcpp.sh ├── testmacro.c ├── testrecmacro.c ├── testvarmacro.c ├── tst1.c ├── tst2.c ├── tst3.c ├── tst4.c ├── tst5.c ├── tst6.c └── tstenum.h /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled Lua sources 2 | luac.out 3 | 4 | # luarocks build files 5 | *.src.rock 6 | *.zip 7 | *.tar.gz 8 | 9 | # Object files 10 | *.o 11 | *.os 12 | *.ko 13 | *.obj 14 | *.elf 15 | 16 | # Precompiled Headers 17 | *.gch 18 | *.pch 19 | 20 | # Libraries 21 | *.lib 22 | *.a 23 | *.la 24 | *.lo 25 | *.def 26 | *.exp 27 | 28 | # Shared objects (inc. Windows DLLs) 29 | *.dll 30 | *.so 31 | *.so.* 32 | *.dylib 33 | 34 | # Executables 35 | *.exe 36 | *.out 37 | *.app 38 | *.i*86 39 | *.x86_64 40 | *.hex 41 | 42 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.fb.com/codeofconduct/) so that you can understand what actions will and will not be tolerated. 4 | 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to CPARSER 2 | 3 | We want to make contributing to this project as easy and transparent as 4 | possible. 5 | 6 | ## Pull Requests 7 | 8 | We actively welcome your pull requests. 9 | 1. Fork the repo and create your branch from `master`. 10 | 2. If you've added code that should be tested, add tests 11 | 3. If you've changed APIs, update the documentation. 12 | 4. Ensure the test suite passes. 13 | 5. Make sure your code lints. 14 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 15 | 16 | ## Contributor License Agreement ("CLA") 17 | 18 | In order to accept your pull request, we need you to submit a CLA. You only need 19 | to do this once to work on any of Facebook's open source projects. 20 | 21 | Complete your CLA here: 22 | 23 | ## Issues 24 | 25 | We use GitHub issues to track public bugs. Please ensure your description is 26 | clear and has sufficient instructions to be able to reproduce the issue. 27 | 28 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 29 | disclosure of security bugs. In those cases, please go through the process 30 | outlined on that page and do not file a public issue. 31 | 32 | ## License 33 | 34 | By contributing to CPARSER, you agree that your contributions will be licensed 35 | under its MIT license. 36 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Facebook, Inc. and its affiliates. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CParser 2 | 3 | This pure Lua module implements (1) a standard compliant C 4 | preprocessor with a couple useful extensions, and (2) a parser that 5 | provides a Lua friendly description of all global declarations and 6 | definitions in a C header or C program file. 7 | 8 | The driver program `lcpp` invokes the preprocessor and outputs 9 | preprocessed code. Although it can be used as a replacement for the 10 | normal preprocessor, it is more useful as an extra preprocessing step 11 | (see option `-Zpass` which is on by default.) The same capabilities 12 | are offered by functions `cparser.cpp` and `cparser.cppTokenIterator` 13 | provided by the module `cparser`. 14 | 15 | The driver program `lcdecl` analyzes a C header file and a C program 16 | file and outputs a short descriptions of the declarations and 17 | definitions. This program is mostly useful to understand the 18 | representations produced by the `cparser` function 19 | `cparser.declarationIterator`. 20 | 21 | This code is licensed under the MIT license found in 22 | the LICENSE file in the root directory of this source tree. 23 | 24 | --- 25 | 26 | ## Program `lcpp` 27 | 28 | ### Synopsis 29 | 30 | ```sh 31 | lcpp [options] inputfile.c [-o outputfile.c] 32 | ``` 33 | Preprocess file `inputfile.c` and write the preprocessed code into 34 | file `outputfile.c` or to the standard output. 35 | 36 | ### Options 37 | 38 | The following options are recognized: 39 | 40 | - `-Werror` 41 | Cause all warning to be treated as errors. 42 | Note that parsing cannot resume after an error. 43 | The parser simply throws a Lua error. 44 | 45 | - `-w` 46 | Do not print warning messages. 47 | 48 | - `-D`*sym*`[=`*val*`]` 49 | Define preprocessor symbol `sym` to value `val`. 50 | The default value of `val` is `1`. 51 | Note that it is possible to define function-like symbols 52 | with syntax `-Dsym(args)=val`. 53 | 54 | - `-U`*sym* 55 | Undefine preprocessor symbol `sym`. 56 | 57 | - `-I`*dir* 58 | Add directory `dir` to the search path for included files. Note 59 | that there is no default search path. When an include file is not 60 | found the include directive is simply ignored with a warning (but 61 | see also option `-Zpass`). Therefore all include directives are 62 | ignored unless one uses option `-I` to specify the search path. 63 | 64 | - `-I-` 65 | Marks the beginning of the system include path. When an included 66 | file is given with angle brackets, (as in `#include `), 67 | one only searches directories specified by the `-I` options that 68 | follow `-I-`. Therefore all these include directives are ignored 69 | unless one uses option `-I-` followed by one or more option `-I`. 70 | 71 | - `-dM` 72 | Instead of producing the preprocessed file, 73 | dumps all macros defined at the end of the parse. 74 | 75 | - `-Zcppdef` 76 | Run the native preprocessor using command `cpp -dM < dev/null` 77 | and copy its predefined symbols. This is useful when using 78 | `lcpp` as a full replacement for the standard preprocessor. 79 | 80 | - `-Zpass` 81 | This option is enabled by default (use `-Znopass` to disable) 82 | and indicates that the output of `lcpp` is going to be 83 | reprocessed by a C preprocessor and compiler. 84 | This option triggers the following behavior: 85 | 86 | * The preprocessor directives `#pragma` and `#ident` 87 | are copied verbatim into the output file. 88 | * When the included file cannot be found in the provided 89 | search path, the preprocessor directive `#include` is 90 | copied into the output file. 91 | * Preprocessor directives prefixed with a double `##` are copied 92 | verbatim into the output file with a single `#` prefix. This 93 | feature is useful for `#if` directives that depend on symbols 94 | defined by unresolved `#include` directives. 95 | 96 | - `-std=(c|gnu)(89|99|11)` 97 | This option selects a C dialect. 98 | In the context of the preprocessor, this impacts 99 | the symbols predefined by `lcpp` and potentially enables 100 | GCC extensions of the variadic macro definition syntax. 101 | * Symbol `__CPARSER__` is always defined with value <1>. 102 | * Symbols `__STDC__` and `__STDC_VERSION__` are either defined by 103 | option `-Zcppdef` or take values suitable for the target C 104 | dialect. 105 | * Symbols `__GNUC__` and `__GNUC_MINOR__` are either defined by 106 | option `-Zcppdef` or are defined to values `4` and `2` if the 107 | target dialect starts with string `gnu`. 108 | 109 | This can be further adjusted using the `-D` or `-U` options. 110 | The default dialect is `gnu99`. 111 | 112 | 113 | ### Preprocessor extensions 114 | 115 | The `lcpp` preprocessor implements several useful nonstandard features. 116 | The main feature are multiline macros. The other features are mostly 117 | here because they make multiline macros more useful. 118 | 119 | #### String comparison in conditional expressions 120 | 121 | The C standard specifies that the expressions following `#if` 122 | directives are constant expressions of integral type. However this 123 | processor also handles strings. The only valid operations on strings 124 | are the equality and ordering comparisons. This is quite useful to 125 | make special cases for certain values of the parameters of a multiline 126 | macro, as shown later. 127 | 128 | #### Multiline macros 129 | 130 | Preprocessor directives `#defmacro` and `#endmacro` can be used to 131 | define a function-like macro whose body spans several lines. The 132 | `#defmacro` directive contains the macro name and a mandatory argument 133 | list. The body of the macro is composed of all the following lines up 134 | to the matching `#endmacro`. This offers several benefits: 135 | 136 | * The line numbers of the macro-expansion is preserved. This ensures 137 | that the compiler produces error messages with meaningful line 138 | numbers. 139 | 140 | * The multi-line macro can contain preprocessor directives. 141 | Conditional directives are very useful in this context. Note 142 | however that preprocessor definitions (with `#define`, `#defmacro`, 143 | or `#undef`) nested inside multiline macros are only valid within 144 | the macro. 145 | 146 | * The standard stringification `#` and token concatenation `##` 147 | operators can be used freely in the body of multiline macros. Note 148 | that these operators only work with the parameters of the multiline 149 | macros and not with ordinary preprocessor definitions. This is 150 | consistent with the standard behavior of these operators in ordinary 151 | preprocessor macros. 152 | 153 | Example 154 | 155 | ```C 156 | #defmacro DEFINE_VDOT(TNAME, TYPE) 157 | TYPE TNAME##Vector_dot(TYPE *a, TYPE *b, int n) 158 | { 159 | /* try cblas */ 160 | #if #TYPE == "float" 161 | return cblas_sdot(n, a, 1, b, 1); 162 | #elif #TYPE == "double" 163 | return cblas_ddot(n, a, 1, b, 1); 164 | #else 165 | int i; 166 | TYPE s = 0; 167 | for(i=0;i>", cparser.declToString(decl)) end 454 | ``` 455 | 456 | 457 | ##### `cparser.typeToString(ty,nam)` 458 | 459 | This function produces a string suitable for 460 | declaring a variable `nam` with type `ty` in a C program. 461 | 462 | Argument `ty` is a type data structure. 463 | Argument `nam` should be a string representing a legal identifier. 464 | However it defaults to `%s` in order to compute a format string 465 | suitable for the standard Lua function `string.format`. 466 | 467 | Module `cparser` represents each type with a tree whose nodes are Lua 468 | tables tagged by their `tag` field. These tables are equipped with a 469 | convenient metatable method that prints them compactly by first 470 | displaying the tag then the remaining fields using the standard Lua 471 | construct. 472 | 473 | For instance, the type `const int` is printed as 474 | ```Lua 475 | Qualified{t=Type{n="int"},const=true} 476 | ``` 477 | and corresponds to 478 | ```Lua 479 | { 480 | tag="Qualified", 481 | const=true, 482 | t= { 483 | tag="Type", 484 | n = "int" 485 | } 486 | } 487 | ``` 488 | 489 | The following tags are used to represent types. 490 | 491 | * `Type{n=name}` is used to represent a named type `name`. There is 492 | only one instance of each named type. Names can be made of multiple 493 | keywords, such as `int` or `unsigned long int`, they can also be 494 | typedef identifiers, such as `size_t`, or composed names, such as 495 | `struct foo` or `enum bar`. This construct can also contain a field 496 | `_def` that points to the definition of the named type if such a 497 | definition is known. 498 | 499 | * `Qualified{t=basetype,...}` is used to represent a qualified variant 500 | of `basetype`. Fields named `const`, `volatile`, or `restrict` are 501 | set to true to represent the applicable type qualifiers. When the 502 | type appears in function parameters and the base type is a pointer, 503 | a field named `static` may contain the guaranteed array size. 504 | 505 | * `Pointer{t=basetype}` is used to represent a pointer to an object of 506 | type `basetype`. This construct may also contains a field 507 | `block=true` to indicate that the pointer refers to a code block (a 508 | C extension found in Apple compilers) or a field `ref=true` to 509 | indicate a reference type (a C extension inspired by C++.) 510 | 511 | * `Array{t=basetype,size=s}` is used to represent an array of object 512 | of type `basetype`. The optional field `size` contains the array 513 | size when an array size is specified. The size is usually an 514 | integer. However there are situations in which the parser is unable 515 | to evaluate the size, for instance because it relies on the C 516 | keyword `sizeof(x)`. In such cases, field `size` is a string 517 | containing a C expression for the size. 518 | 519 | * `Struct{}` and `Union{}` are used to represent the corresponding C 520 | types. The optional field `n` contains the structure tag. Each entry 521 | is represented by a `Pair{type,name}` construct located at 522 | successive integer indices. This means that the type of the third entry 523 | of structure type `ty` can be accessed as `ty[3][1]` and the 524 | corresponding name is `ty[3][2]`. In the case of `Struct{}` tables, 525 | the pairs optionally contain a field `bitfield` to indicate a 526 | bitfield size for the structure entry. Field `bitfield` usually 527 | contains a small integer but can also contain a string representing 528 | a C expression (just like field `size` in the `Array{}` construct.) 529 | 530 | * `Enum{}` is used to represent an enumerated type. The optional 531 | field `n` may contain the enumeration tag name. The enumeration 532 | constants are reprsented as `Pair{name,value}` located at 533 | successive integer indices. The second pair element is only 534 | given when the C code contains an explicit value. It can be an 535 | integer or an expression strint (just like field `size` in `Array{}`). 536 | 537 | * `Function{t=returntype}` is used to represent functions returning an 538 | object of type `returntype`. Field `withoutProto` is set to `true` 539 | when the function does not provide a prototype. Otherwise the 540 | arguments are described by `Pair{type,name}` located as integer 541 | indices. The prototype of variadic functions end with a 542 | `Pair{ellipsis=true}` to represent the `...` argument. 543 | 544 | The `Qualified{}`, `Function{}`, `Struct{}`, `Union{}`, and `Enum{}` 545 | tables may additionally have a field `attr` whose contents represents 546 | attribute information, such as C11 attributes `[[...]]`, MSVC-style 547 | attributes `__declspec(...)` or GNU attributes `__attribute__(...)`. 548 | This is representing by an array containing all the attribute tokens 549 | (on odd indices) and their locations (on even indices). 550 | 551 | 552 | ##### `cparser.stringToType(s)` 553 | 554 | Parses string `s` as an abstract type name or a variable declaration 555 | and returns the type object and possibly the variable name. This 556 | function returns `nil` when the string cannot be interpreted as a type 557 | or a declaration, or when the declaration specifies a storage class. 558 | 559 | Example 560 | 561 | ```Lua 562 | > return cparser.stringToType("int(*)(const char*)") 563 | Pointer{t=Function{Pair{Pointer{t=Qualified{const=true,t=Type{n="char"}}}},t=Type{n="int"}}} nil 564 | ``` 565 | 566 | 567 | 568 | ##### `cparser.declToString(decl)` 569 | 570 | This function produces a string that describes the data structures 571 | returned by the declaration iterator. There are in fact three kinds 572 | of data structures. All these structures have very similar fields. 573 | In particular, field `where` always contains the location of the 574 | definition or declaration. 575 | 576 | * `TypeDef{name=n,sclass=s,type=ty}` represents a type definition. 577 | This construct is produced in two different situations. When the C 578 | program contains a `typedef` keyword, field `sclass` contains the 579 | string `"typedef"`, field `name` contains the new type name, and 580 | field `type` contains the type description. When the C program 581 | defines a tagged `struct`, `union`, or `enum` type, field `sclass` 582 | contains the string `"[typetag]"`, field `name` contains the tagged 583 | type name (e.g, `"struct foo"`), and field `type` contains the type 584 | definition (e.g., `Struct{...}`). 585 | 586 | * `Declaration{name=n,sclass=s,type=ty,...}` represents the declaration 587 | of a variable or function that is defined elsewhere. Field `name` 588 | gives the variable or function name. Field `type` gives its type. 589 | Field `sclass` can be empty, `"extern"`, or `"static"`. 590 | 591 | * `Definition{name=n,sclass=s,type=ty...}` represents the definition 592 | of a constant, a variable, or a function. Field `name` again gives 593 | the name, field `type` gives its type, field `sclass` gives its 594 | storage class, and field `init` may contain an array of tokens and 595 | token locations representing a variable initializer or a function 596 | body. Constant definitions may also have a field `intval` contaning 597 | the value of an integer constant. This field works like the size of 598 | an array: it often contains a small integer but can also contains a 599 | string representing the C expression that the parser was unable to 600 | evaluate for one reason or another. Enumeration constants are 601 | reported with storage class `"[enum]"` and with a constant integer 602 | type containing an additional field `_enum` that points to the 603 | corresponding enumerated type. 604 | 605 | * `CppEvent{directive=dir,...}` describes certain preprocessor events 606 | that are potentially relevant to a C API. In particular, the 607 | definition of an object-like macro `s` with an integer value `v` is 608 | reported as `CppEvent{directive="define",name="s",intval=v}` and its 609 | deletion as `CppEvent{directive="undef",name="s"}`. Finally, 610 | `CppEvent{directive="include",name="fspec"}` indicates that an 611 | include directive was not resolved. 612 | -------------------------------------------------------------------------------- /cparser-scm-1.rockspec: -------------------------------------------------------------------------------- 1 | -- Copyright (c) Facebook, Inc. and its affiliates. 2 | -- This source code is licensed under the MIT license found in the 3 | -- LICENSE file in the root directory of this source tree. 4 | 5 | 6 | package = 'cparser' 7 | version = 'scm-1' 8 | source = { 9 | url = 'git://github.com/facebookresearch/cparser', 10 | tag = 'scm-1' 11 | } 12 | description = { 13 | summary = 'A compact C preprocessor and declaration parser written in pure Lua', 14 | homepage = 'https://github.com/facebookresearch/CParser', 15 | license = 'MIT', 16 | } 17 | dependencies = { 18 | 'lua >= 5.1' 19 | } 20 | build = { 21 | type = 'builtin', 22 | install = { 23 | bin = { 24 | "lcdecl", 25 | "lcpp" 26 | } 27 | }, 28 | modules = { 29 | ['cparser'] = 'cparser.lua', 30 | } 31 | } 32 | 33 | -- vim: syntax=lua 34 | -------------------------------------------------------------------------------- /cparser.lua: -------------------------------------------------------------------------------- 1 | -- Copyright (c) Facebook, Inc. and its affiliates. 2 | -- This source code is licensed under the MIT license found in the 3 | -- LICENSE file in the root directory of this source tree. 4 | -- 5 | -- Lua module to preprocess and parse C declarations. 6 | -- (Leon Bottou, 2015) 7 | 8 | 9 | 10 | -- standard libs 11 | local string = require 'string' 12 | local coroutine = require 'coroutine' 13 | local table = require 'table' 14 | local io = require 'io' 15 | 16 | -- Lua 5.1 to 5.3 compatibility 17 | local unpack = unpack or table.unpack 18 | 19 | -- Debugging 20 | local DEBUG = true 21 | if DEBUG then pcall(require,'strict') end 22 | 23 | -- luacheck: globals cparser 24 | -- luacheck: ignore 43 4/ti 4/li 25 | -- luacheck: ignore 212/.*_ 26 | -- luacheck: ignore 211/is[A-Z].* 211/Type 27 | -- luacheck: ignore 542 28 | 29 | 30 | --------------------------------------------------- 31 | --------------------------------------------------- 32 | --------------------------------------------------- 33 | -- ALL UGLY HACKS SHOULD BE HERE 34 | 35 | 36 | -- Sometimes we cannot find system include files but need to know at 37 | -- least things about them. For instance, certain system include files 38 | -- define alternate forms for keywords. 39 | 40 | local knownIncludeQuirks = {} 41 | 42 | knownIncludeQuirks[""] = { -- c99 43 | "#ifndef complex", "# define complex _Complex", "#endif" 44 | } 45 | 46 | knownIncludeQuirks[""] = { -- c99 47 | "#ifndef bool", "# define bool _Bool", "#endif" 48 | } 49 | 50 | knownIncludeQuirks[""] = { -- c11 51 | "#ifndef alignof", "# define alignof _Alignof", "#endif", 52 | "#ifndef alignas", "# define alignas _Alignas", "#endif" 53 | } 54 | 55 | knownIncludeQuirks[""] = { -- c11 56 | "#ifndef noreturn", "# define noreturn _Noreturn", "#endif" 57 | } 58 | 59 | knownIncludeQuirks[""] = { -- c11 60 | "#ifndef thread_local", "# define thread_local _Thread_local", "#endif" 61 | } 62 | 63 | knownIncludeQuirks[""] = { -- c++ 64 | "#define and &&", "#define and_eq &=", "#define bitand &", "#define bitor |", 65 | "#define compl ~", "#define not !", "#define not_eq !=", "#define or ||", 66 | "#define or_eq |=", "#define xor ^", "#define xor_eq ^=" 67 | } 68 | 69 | 70 | --------------------------------------------------- 71 | --------------------------------------------------- 72 | --------------------------------------------------- 73 | -- TAGGED TABLES 74 | 75 | 76 | -- Utilities to produce and print tagged tables. 77 | -- The tag name is simply the contents of table key . 78 | -- Function returns a node constructor 79 | -- 80 | -- Example: 81 | -- 82 | -- > Foo = newTag('Foo') 83 | -- > Bar = newTag('Bar') 84 | -- 85 | -- > print( Foo{const=true,next=Bar{name="Hello"}} ) 86 | -- Foo{next=Bar{name="Hello"},const=true} 87 | -- 88 | -- > print( Bar{name="hi!", Foo{1}, Foo{2}, Foo{3}} ) 89 | -- Bar{Foo{1},Foo{2},Foo{3},name="hi!"} 90 | 91 | local function newTag(tag) 92 | -- the printing function 93 | local function tostr(self) 94 | local function str(x) 95 | if type(x)=='string' then 96 | return string.format("%q",x):gsub("\\\n","\\n") 97 | elseif type(x)=='table' and not getmetatable(x) then 98 | return "{..}" 99 | else 100 | return tostring(x) 101 | end 102 | end 103 | local p = string.format("%s{", self.tag or "Node") 104 | local s = {} 105 | local seqlen = 0 106 | for i=1,#self do 107 | if self[i] then seqlen=i else break end end 108 | for i=1,seqlen do 109 | s[1+#s] = str(self[i]) end 110 | for k,v in pairs(self) do 111 | if type(k) == 'number' then 112 | if k<1 or k>seqlen then 113 | s[1+#s] = string.format("[%s]=%s",k,str(v)) end 114 | elseif type(k) ~= 'string' then 115 | s.extra = true 116 | elseif k:find("^_") and type(v)=='table' then 117 | s[1+#s] = string.format("%s={..}",k) -- hidden 118 | elseif k ~= 'tag' then 119 | s[1+#s] = string.format("%s=%s",k,str(v)) end 120 | end 121 | if s.extra then s[1+#s] = "..." end 122 | return p .. table.concat(s,',') .. '}' 123 | end 124 | -- the constructor 125 | return function(t) -- must be followed by a table constructor 126 | t = t or {} 127 | assert(type(t)=='table') 128 | setmetatable(t, { __tostring=tostr } ) 129 | t.tag = tag 130 | return t 131 | end 132 | end 133 | 134 | -- hack to print any table: print(Node(nn)) 135 | local Node = newTag(nil) -- luacheck: ignore 211 136 | 137 | --------------------------------------------------- 138 | --------------------------------------------------- 139 | --------------------------------------------------- 140 | -- UTILITIES 141 | 142 | 143 | -- Many functions below have an optional argument 'options' which is 144 | -- simply an array of compiler-like options that are specified in the 145 | -- toplevel call and passed to nearly all functions. Because it 146 | -- provides a good communication channel across the code components, 147 | -- many named fields are also used for multiple purposes. The 148 | -- following function is called at the beginning of the user facing 149 | -- functions to make a copy of the user provided option array and 150 | -- setup some of these fields. 151 | 152 | local function copyOptions(options) 153 | options = options or {} 154 | assert(type(options)=='table') 155 | local noptions = {} 156 | -- copy options 157 | for k,v in ipairs(options) do noptions[k]=v end 158 | -- copy user modifiable named fields 159 | noptions.sizeof = options.sizeof -- not used yet 160 | noptions.alignof = options.alignof -- not used yet 161 | -- create reversed hash 162 | noptions.hash = {} 163 | for i,v in ipairs(options) do 164 | noptions.hash[v] = i 165 | end 166 | -- compute dialect flags 167 | local dialect = 'gnu99' 168 | for _,v in ipairs(options) do 169 | if v:find("^%-std=%s*[^%s]") then 170 | dialect = v:match("^%-std=%s*(.-)%s*$") 171 | end 172 | end 173 | noptions.dialect = dialect 174 | noptions.dialectGnu = dialect:find("^gnu") 175 | noptions.dialect99 = dialect:find("9[9x]$") 176 | noptions.dialect11 = dialect:find("1[1x]$") 177 | noptions.dialectAnsi = not noptions.dialectGnu 178 | noptions.dialectAnsi = noptions.dialectAnsi and not noptions.dialect99 179 | noptions.dialectAnsi = noptions.dialectAnsi and not noptions.dialect11 180 | -- return 181 | return noptions 182 | end 183 | 184 | 185 | -- This function tests whether a particular option has been given. 186 | 187 | local function hasOption(options, opt) 188 | assert(options) 189 | assert(options.silent or options.hash) 190 | return options.hash and options.hash[opt] 191 | end 192 | 193 | 194 | -- Generic functions for error messages 195 | 196 | local function xmessage(err, options, lineno, message, ...) 197 | local msg = string.format("cparser: (%s) ",lineno) 198 | msg = msg .. string.format(message,...) 199 | if options.silent then 200 | if err == 'error' then error(msg, 0) end 201 | else 202 | if err == 'warning' and hasOption(options, "-Werror") then err = 'error' end 203 | if err == 'error' or not hasOption(options, "-w") then print(msg) end 204 | if err == 'error' then error("cparser: aborted",0) end 205 | end 206 | end 207 | 208 | local function xwarning(options, lineno, message, ...) 209 | xmessage('warning', options, lineno, message, ...) 210 | end 211 | 212 | local function xerror(options, lineno, message, ...) 213 | xmessage('error', options, lineno, message, ...) 214 | end 215 | 216 | local function xassert(cond, ...) 217 | if not cond then xerror(...) end 218 | end 219 | 220 | local function xdebug(lineno,message,...) 221 | local msg = string.format("\t\t[%s] ", lineno) 222 | msg = msg .. string.format(message,...) 223 | print(msg) 224 | end 225 | 226 | 227 | -- Nil-safe max 228 | 229 | local function max(a,b) 230 | a = a or b 231 | b = b or a 232 | return a > b and a or b 233 | end 234 | 235 | 236 | -- Deep table comparison 237 | -- (not very efficient, no loop detection) 238 | 239 | local function tableCompare(a,b) 240 | if a == b then 241 | return true 242 | elseif type(a) == 'table' and type(b) == 'table' then 243 | for k,v in pairs(a) do 244 | if not tableCompare(v,b[k]) then return false end 245 | end 246 | for k,v in pairs(b) do 247 | if not tableCompare(a[k],v) then return false end 248 | end 249 | return true 250 | else 251 | return false 252 | end 253 | end 254 | 255 | 256 | -- Concatenate two possibly null arrays 257 | 258 | local function tableAppend(a1, a2) 259 | if not a1 then 260 | return a2 261 | elseif not a2 then 262 | return a1 263 | else 264 | local a = {} 265 | for _,v in ipairs(a1) do a[1+#a] = v end 266 | for _,v in ipairs(a2) do a[1+#a] = v end 267 | return a 268 | end 269 | end 270 | 271 | -- Concatenate strings from table (skipping non-string content.) 272 | 273 | local function tableConcat(a) 274 | local b = {} 275 | for _,v in ipairs(a) do 276 | if type(v) == 'string' then b[1+#b]=v end end 277 | return table.concat(b) 278 | end 279 | 280 | 281 | -- Evaluate a lua expression, return nil on error. 282 | 283 | local function evalLuaExpression(s) 284 | assert(type(s)=='string') 285 | local f = load(string.gmatch(s,".*")) 286 | local function r(status,...) 287 | if status then return ... end end 288 | return r(pcall(f or error)) 289 | end 290 | 291 | 292 | -- Bitwise manipulations 293 | -- try lua53 operators otherwise revert to iterative version 294 | 295 | local bit = evalLuaExpression([[ 296 | local bit = {} 297 | function bit.bnot(a) return ~a end 298 | function bit.bor(a,b) return a | b end 299 | function bit.band(a,b) return a & b end 300 | function bit.bxor(a,b) return a ~ b end 301 | function bit.lshift(a,b) return a < 0 and b < 0 and ~((~a) << b) or a << b end 302 | return bit 303 | ]]) 304 | 305 | if not bit then 306 | local function bor(a,b) 307 | local r, c, d = 0, 1, -1 308 | while a > 0 or b > 0 or a < -1 or b < -1 do 309 | if a % 2 > 0 or b % 2 > 0 then r = r + c end 310 | a, b, c, d = math.floor(a / 2), math.floor(b / 2), c * 2, d * 2 end 311 | if a < 0 or b < 0 then r = r + d end 312 | return r end 313 | bit = {} 314 | function bit.bnot(a) return -1-a end 315 | function bit.bor(a,b) return bor(a,b) end 316 | function bit.band(a,b) return -1-bor(-1-a,-1-b) end 317 | function bit.bxor(a,b) return bor(-1-bor(a,-1-b),-1-bor(-1-a,b)) end 318 | function bit.lshift(a,b) return math.floor(a * 2 ^ b) end 319 | end 320 | 321 | 322 | -- Coroutine helpers. 323 | -- This code uses many coroutines that yield lines or tokens. 324 | -- All functions that can yield take an options table as first argument. 325 | 326 | 327 | -- Wrap a coroutine f into an iterator 328 | -- The options and all the extra arguments are passed 329 | -- to the coroutine when it starts. Together with the 330 | -- above calling convention, this lets us specify 331 | -- coroutine pipelines (see example in function "cpp".) 332 | 333 | local function wrap(options, f, ...) 334 | local function g(...) coroutine.yield(nil) f(...) end 335 | local c = coroutine.create(g) 336 | coroutine.resume(c, options, ...) 337 | local function r(s,...) 338 | if not s then local m = ... ; error(m, 0) end 339 | return ... 340 | end 341 | return function() 342 | if coroutine.status(c) ~= 'dead' then 343 | return r(coroutine.resume(c)) 344 | end 345 | end 346 | end 347 | 348 | 349 | -- Collect coroutine outputs into an array 350 | -- The options and the extra arguments are passed to the coroutine. 351 | 352 | local function callAndCollect(options, f, ...) -- Bell Labs nostalgia 353 | local collect = {} 354 | for s in wrap(options, f, ...) do 355 | collect[1+#collect] = s 356 | end 357 | return collect 358 | end 359 | 360 | 361 | -- Yields all outputs from iterator iter. 362 | -- Argument options is ignored. 363 | 364 | local function yieldFromIterator(options_, iter) 365 | local function yes(v,...) coroutine.yield(v,...) return v end 366 | while yes(iter()) do end 367 | end 368 | 369 | 370 | -- Yields all values from array . 371 | -- This function successively yields all values in the table. 372 | -- Every yield is augmented with all extra arguments passed to the function. 373 | -- Argument options is ignored. 374 | 375 | local function yieldFromArray(options_, arr, ...) 376 | for _,v in ipairs(arr) do 377 | coroutine.yield(v, ...) 378 | end 379 | end 380 | 381 | 382 | 383 | 384 | 385 | --------------------------------------------------- 386 | --------------------------------------------------- 387 | --------------------------------------------------- 388 | -- INITIAL PREPROCESSING 389 | 390 | 391 | 392 | -- A routine that pulls lines from a line iterator 393 | -- and yields them together with a location 394 | -- composed of the optional prefix, a colon, and a line number. 395 | -- Argument options is ignored. 396 | -- Lua provides good line iterators such as: 397 | -- io.lines(filename) filedesc:lines() str:gmatch("[^\n]+") 398 | 399 | local function yieldLines(options_,lineIterator,prefix) 400 | prefix = prefix or "" 401 | assert(type(prefix)=='string') 402 | local n = 0 403 | for s in lineIterator do 404 | n = n + 1 405 | coroutine.yield(s, string.format("%s:%d", prefix, n)) 406 | end 407 | end 408 | 409 | 410 | -- A routine that obtains lines from coroutine , 411 | -- joins lines terminated by a backslash, and yield the 412 | -- resulting lines. The coroutine is initialized with 413 | -- argument and all extra arguments. 414 | -- Reference: https://gcc.gnu.org/onlinedocs/cpp/Initial-processing.html (3) 415 | 416 | local function joinLines(options, lines, ...) 417 | local li = wrap(options, lines, ...) 418 | for s, n in li do 419 | while type(s) == 'string' and s:find("\\%s*$") do 420 | local t = li() or "" 421 | s = s:gsub("\\%s*$", "") .. t 422 | end 423 | coroutine.yield(s, n) 424 | end 425 | end 426 | 427 | 428 | -- A routine that obtain lines from coroutine , eliminate the 429 | -- comments and yields the resulting lines. The coroutine is 430 | -- initialized with argument and all extra arguments. 431 | -- Reference: https://gcc.gnu.org/onlinedocs/cpp/Initial-processing.html (4) 432 | 433 | local function eliminateComments(options, lines, ...) 434 | local lineIterator = wrap(options, lines, ...) 435 | local s,n = lineIterator() 436 | while type(s) == 'string' do 437 | local inString = false 438 | local q = s:find("[\'\"\\/]", 1) 439 | while q ~= nil do 440 | if hasOption(options,"-d:comments") then 441 | xdebug(n, "comment: [%s][%s] %s",s:sub(1,q-1),s:sub(q),inString) 442 | end 443 | local c = s:byte(q) 444 | if inString then 445 | if c == 92 then -- \ 446 | q = q + 1 447 | elseif c == inString then 448 | inString = false 449 | end 450 | else 451 | if c == 34 or c == 39 then -- " or ' 452 | inString = c 453 | elseif c == 47 and s:byte(q+1) == 47 then -- "//" 454 | s = s:sub(1,q-1) 455 | elseif c == 47 and s:byte(q+1) == 42 then -- "/*" 456 | local p = s:find("%*/",q+2) 457 | if p ~= nil then 458 | s = s:sub(1,q-1) .. " " .. s:sub(p+2) 459 | else 460 | s = s:sub(1,q-1) 461 | local ss,pp 462 | repeat 463 | ss = lineIterator() 464 | xassert(ss ~= nil, options, n, "Unterminated comment") 465 | pp = ss:find("%*/") 466 | until pp 467 | s = s .. " " .. ss:sub(pp+2) 468 | end 469 | end 470 | end 471 | q = s:find("[\'\"\\/]", q+1) 472 | end 473 | coroutine.yield(s, n) 474 | s, n = lineIterator() 475 | end 476 | end 477 | 478 | 479 | 480 | --------------------------------------------------- 481 | --------------------------------------------------- 482 | --------------------------------------------------- 483 | -- TOKENIZER 484 | 485 | 486 | 487 | local keywordTable = { 488 | ------ Standard keywords 489 | "auto", "break", "case", "char", "const", "continue", "default", "do", 490 | "double", "else", "enum", "extern", "float", "for", "goto", "if", "int", 491 | "long", "register", "return", "short", "signed", "sizeof", "static", "struct", 492 | "switch", "typedef", "union", "unsigned", "void", "volatile", "while", 493 | ------ Nonstandard or dialect specific keywords do not belong here 494 | ------ because the main function of this table is to say which 495 | ------ identifiers cannot be variable names. 496 | } 497 | 498 | local punctuatorTable = { 499 | "+", "-", "*", "/", "%", "&", "|", "^", ">>", "<<", "~", 500 | "=", "+=", "-=", "*=", "/=", "%=", "&=", "|=", "^=", ">>=", "<<=", 501 | "(", ")", "[", "]", "{", "}", "++", "--", 502 | "==", "!=", ">=", "<=", ">", "<", "&&", "||", "!", 503 | ".", "->", "*", "&", "?", ":", "::", "->*", ".*", ";", ",", 504 | "#", "##", "...", "@", "\\" -- preprocessor stuff 505 | } 506 | 507 | local keywordHash = {} 508 | for _,v in ipairs(keywordTable) do 509 | keywordHash[v] = true 510 | end 511 | 512 | local punctuatorHash = {} 513 | for _,v in ipairs(punctuatorTable) do 514 | local l = v:len() 515 | local b = v:byte() 516 | punctuatorHash[v] = true 517 | punctuatorHash[b] = max(l,punctuatorHash[b]) 518 | end 519 | 520 | 521 | -- The following functions test the types of the tokens returned by the tokenizer. 522 | -- They should not be applied to arbitrary strings. 523 | 524 | local function isSpace(tok) 525 | return type(tok) == 'string' and tok:find("^%s") ~= nil end 526 | local function isNewline(tok) -- Subtype of space 527 | return type(tok) == 'string' and tok:find("^\n") ~= nil end 528 | local function isNumber(tok) 529 | return type(tok) == 'string' and tok:find("^[.0-9]") ~= nil end 530 | local function isString(tok) 531 | if type(tok) ~= 'string' then return false end 532 | return tok:find("^[\'\"]") ~= nil end 533 | local function isHeaderName(tok) 534 | if type(tok) ~= 'string' then return false end 535 | return tok:find("^\"") or tok:find("^<") and tok:find(">$") end 536 | local function isPunctuator(tok) 537 | return type(tok) == 'string' and punctuatorHash[tok] ~= nil end 538 | local function isIdentifier(tok) 539 | return type(tok) == 'string' and tok:find("^[A-Za-z_$]") ~= nil end 540 | local function isKeyword(tok) -- Subtype of identifier 541 | return keywordHash[tok] ~= nil end 542 | local function isName(tok) -- Subtype of identifier 543 | return isIdentifier(tok) and not keywordHash[tok] end 544 | 545 | -- Magic tokens are used to mark macro expansion boundaries (see expandMacros.) 546 | local function isMagic(tok) 547 | return tok and type(tok) ~= 'string' end 548 | local function isBlank(tok) -- Treats magic token as space. 549 | return isMagic(tok) or isSpace(tok) end 550 | 551 | -- The tokenizeLine() function takes a line, splits it into tokens, 552 | -- and yields tokens and locations. The number tokens are the weird 553 | -- preprocessor numbers defined by ansi c. The string tokens include 554 | -- character constants and angle-bracket delimited strings occuring 555 | -- after an include directive. Every line begins with a newline 556 | -- token giving the proper indentation. All subsequent spaces 557 | -- are reduced to a single space character. 558 | 559 | local function tokenizeLine(options, s, n, notNewline) 560 | -- little optimization for multiline macros 561 | -- s may be an array of precomputed tokens 562 | if type(s) == 'table' then 563 | return yieldFromArray(options, s, n) 564 | end 565 | -- normal operation 566 | assert(type(s) == 'string') 567 | local p = s:find("[^%s]") 568 | -- produce a newline token 569 | if p and not notNewline then 570 | local r = '\n' .. s:sub(1,p-1) 571 | coroutine.yield(r, n) 572 | end 573 | -- produce one token 574 | local function token() 575 | local b, l, r 576 | if hasOption(options, "-d:tokenize") then 577 | xdebug(n, "[%s][%s]",s:sub(1,p-1),s:sub(p)) 578 | end 579 | -- space 580 | l = s:find("[^%s]", p) 581 | if l == nil then 582 | return nil 583 | elseif l > p then 584 | p = l 585 | return " ", n 586 | end 587 | -- identifier 588 | r = s:match("^[a-zA-Z_$][a-zA-Z0-9_$]*", p) 589 | if r ~= nil then 590 | p = p + r:len() 591 | return r, n 592 | end 593 | -- preprocessor numbers 594 | r = s:match("^%.?[0-9][0-9a-zA-Z._]*", p) 595 | if r ~= nil then 596 | l = r:len() 597 | while r:find("[eEpP]$") and s:find("^[-+]", p+l) do 598 | r = r .. s:match("^[-+][0-9a-zA-Z._]*", p+l) 599 | l = r:len() 600 | end 601 | p = p + l 602 | return r, n 603 | end 604 | -- angle-delimited strings in include directives 605 | b = s:byte(p) 606 | if b == 60 and s:find("^%s*#%s*include") then 607 | r = s:match("^<[^>]+>", p) 608 | if r ~= nil then 609 | p = p + r:len() 610 | return r, n 611 | end 612 | end 613 | -- punctuator 614 | l = punctuatorHash[b] 615 | if l ~= nil then 616 | while l > 0 do 617 | r = s:sub(p,p+l-1) 618 | if punctuatorHash[r] then 619 | p = p + l 620 | return r, n 621 | end 622 | l = l - 1 623 | end 624 | end 625 | -- string 626 | if b == 34 or b == 39 then -- quotes 627 | local q = p 628 | repeat 629 | q = s:find("[\'\"\\]", q + 1) 630 | l = s:byte(q) 631 | xassert(q ~= nil, options, n, "Unterminated string or character constant") 632 | if l == 92 then 633 | q = q + 1 634 | end 635 | until l == b 636 | r = s:sub(p,q) 637 | p = q + 1 638 | return r, n 639 | end 640 | -- other stuff (we prefer to signal an error here) 641 | xerror(options, n,"Unrecognized character (%s)", s:sub(p)) 642 | end 643 | -- loop 644 | if p then 645 | for tok,tokn in token do 646 | coroutine.yield(tok, tokn) 647 | end 648 | end 649 | end 650 | 651 | 652 | -- Obtain lines from coroutine , 653 | -- and yields their tokens. The coroutine is initialized with 654 | -- argument and all extra arguments. 655 | 656 | local function tokenize(options, lines, ...) 657 | for s,n in wrap(options, lines, ...) do 658 | tokenizeLine(options, s, n) 659 | end 660 | end 661 | 662 | 663 | --------------------------------------------------- 664 | --------------------------------------------------- 665 | --------------------------------------------------- 666 | -- PREPROCESSING 667 | 668 | 669 | -- Preprocessing is performed by two coroutines. The first one 670 | -- processes all the preprocessor directives and yields the remaining 671 | -- lines. The second one processes tokens from the remaining lines and 672 | -- perform macro expansions. Both take a table of macro definitions as 673 | -- argument. The first one writes into the table and the second one 674 | -- reads from it. 675 | -- 676 | -- Each macro definition is an array of tokens (for a single line 677 | -- macro) or a table whose entry <"lines"> contains an array of arrays 678 | -- of tokens (#defmacro). If the macro takes arguments, the entry 679 | -- <"args"> contains a list of argument names. If the macro is 680 | -- recursive (#defrecmacro), the entry is set. 681 | -- Alternatively, the macro definition may be a function called at 682 | -- macro-expansion time. This provides for complicated situations. 683 | 684 | 685 | -- forward declarations 686 | local function expandMacros() end 687 | local function processDirectives() end 688 | 689 | 690 | -- Starting with the second coroutine which takes a token producing 691 | -- coroutine and yields the preprocessed tokens. Argument macros is 692 | -- the macro definition table. 693 | 694 | -- The standard mandates that the result of a macro-expansion must be 695 | -- scanned for further macro invocations whose argunent list possibly 696 | -- consume tokens that follow the macro-expansion. This means that one 697 | -- cannot recursively call expandMacros but one must prepend the 698 | -- macro-expansion in front of the remaining tokens. The standard also 699 | -- mandates that the result of any macro-expansion must be marked to 700 | -- prevent recursive invocation of the macro that generated it, 701 | -- whether when expanding macro arguments or expanding the macro 702 | -- itself. We achieve this by bracketing every macro-expansion with 703 | -- magic tokens that track which macro definitions must be disabled. 704 | -- These magic tokens are removed later in the coroutines 705 | -- or . 706 | 707 | expandMacros = function(options, macros, tokens, ...) 708 | -- basic iterator 709 | local ti = wrap(options, tokens, ...) 710 | -- prepending tokens in front of the token stream 711 | local prepend = {} 712 | local function prependToken(s,n) 713 | table.insert(prepend,{s,n}) end 714 | local function prependTokens(pti) 715 | local pos = 1+#prepend 716 | for s,n in pti do table.insert(prepend,pos,{s,n}) end end 717 | local ti = function() 718 | if #prepend > 0 then return unpack(table.remove(prepend)) 719 | else return ti() end end 720 | -- iterator that handles magic tokens to update macro definition table 721 | local ti = function() 722 | local s,n = ti() 723 | while type(s) == 'table' do 724 | if s.tag == 'push' then 725 | local nmacros = {} 726 | setmetatable(nmacros, {__index=macros}) 727 | if s.symb then nmacros[s.symb] = false end 728 | macros = nmacros 729 | elseif s.tag == 'pop' then 730 | local mt = getmetatable(macros) 731 | if mt and mt.__index then macros = mt.__index end 732 | end 733 | coroutine.yield(s,n) 734 | s,n = ti() 735 | end 736 | return s,n 737 | end 738 | -- redefine ti() to ensure tok,n remain up-to-date 739 | local tok,n = ti() 740 | local ti = function() tok,n=ti() return tok,n end 741 | -- collect one macro arguments into an array 742 | -- stop when reaching a closing parenthesis or a comma 743 | local function collectArgument(ti, varargs) 744 | local count = 0 745 | local tokens = {} 746 | ti() 747 | while isSpace(tok) do 748 | tok = ti() 749 | end 750 | while tok do 751 | if tok == ')' and count == 0 then 752 | break 753 | elseif tok == ')' then 754 | count = count - 1 755 | elseif tok == '(' then 756 | count = count + 1 757 | elseif tok == ',' and count == 0 and not varargs then 758 | break 759 | end 760 | if isSpace(tok) then tok = " " end 761 | tokens[1+#tokens] = tok 762 | tok = ti() 763 | end 764 | if #tokens > 0 and isSpace(tokens[#tokens]) then 765 | tokens[#tokens] = nil 766 | end 767 | return tokens 768 | end 769 | -- collects all macro arguments 770 | local function collectArguments(ti,def,ntok,nn) 771 | local args = def.args 772 | local nargs = { [0]={} } 773 | if #args == 0 then ti() end 774 | for _,name in ipairs(args) do 775 | if tok == ')' and name == "__VA_ARGS__" then 776 | nargs[0][name] = { negComma=true } 777 | nargs[name] = { negComma=true } 778 | else 779 | xassert(tok=='(' or tok==',', options, nn, "not enough arguments for macro '%s'", ntok) 780 | local arg = collectArgument(ti, name == "__VA_ARGS__") 781 | nargs[0][name] = arg 782 | nargs[name] = callAndCollect(options, expandMacros, macros, yieldFromArray, arg, nn) 783 | end 784 | end 785 | if def.nva then -- named variadic argument (implies dialectGnu) 786 | nargs[def.nva] = nargs["__VA_ARGS__"] 787 | nargs[0][def.nva] = nargs[0]["__VA_ARGS__"] 788 | end 789 | xassert(tok, options, nn, "unterminated arguments for macro '%s'", ntok) 790 | xassert(tok==')', options, nn, "too many arguments for macro '%s'", ntok) 791 | return nargs 792 | end 793 | -- coroutine that substitute the macro arguments 794 | -- and stringification and concatenation are handled here 795 | local function substituteArguments(options, def, nargs, n, inDirective) 796 | local uargs = nargs[0] or nargs -- unexpanded argument values 797 | if inDirective then nargs = uargs end -- use unexpanded arguments in directives 798 | -- prepare loop 799 | local i,j,k = 1,1,1 800 | while def[i] do 801 | if isBlank(def[i]) then 802 | -- copy blanks 803 | coroutine.yield(def[i], n) 804 | else 805 | -- positions j and k on next non-space tokens 806 | local function updateJandK() 807 | if j <= i then j=i 808 | repeat j=j+1 until def[j] == nil or not isBlank(def[j]) end 809 | if k <= j then k=j 810 | repeat k=k+1 until def[k] == nil or not isBlank(def[k]) end 811 | end 812 | updateJandK() 813 | -- alternatives 814 | if def[i]=='#' and def[j] and nargs[def[j]] then 815 | -- stringification (with the weird quoting rules) 816 | local v = { '\"' } 817 | for _,t in ipairs(uargs[def[j]]) do 818 | if type(t)=='string' then 819 | if t:find("^%s+$") then t = ' ' end 820 | if t:find("^[\'\"]") then t = string.format("%q", t):sub(2,-2) end 821 | v[1+#v] = t end end 822 | v[1+#v] = '\"' 823 | coroutine.yield(tableConcat(v), n) 824 | i = j 825 | elseif def.nva and def[i]==',' and def[j]=='##' and def[k]==def.nva then 826 | -- named variadic macro argument with ## to signal negative comma (gcc crap) 827 | if nargs[def.nva].negComma then i=i+1 end 828 | while i < j do coroutine.yield(def[i], n) ; i=i+1 end 829 | elseif def[i]==',' and def[j]=='__VA_ARGS__' and def[k]==')' then 830 | -- __VA_ARGS__ with implied negative comma semantics 831 | if nargs[def[j]].negComma then i=i+1 end 832 | while i < j do coroutine.yield(def[i], n) ; i=i+1 end 833 | i = j-1 834 | elseif def[j]=='##' and def[k] and not inDirective then 835 | -- concatenation 836 | local u = {} 837 | local function addToU(s) 838 | if nargs[s] then for _,v in ipairs(uargs[s]) do u[1+#u] = v end 839 | else u[1+#u]=s end end 840 | addToU(def[i]) 841 | while def[j] == '##' and def[k] do 842 | addToU(def[k]) 843 | i = k 844 | updateJandK() 845 | end 846 | tokenizeLine(options, tableConcat(u), n, true) 847 | elseif nargs[def[i]] then 848 | -- substitution 849 | yieldFromArray(options, nargs[def[i]], n) 850 | else 851 | -- copy 852 | coroutine.yield(def[i], n) 853 | end 854 | end 855 | i = i + 1 856 | end 857 | end 858 | -- main loop 859 | local newline, directive = true, false 860 | while tok ~= nil do 861 | -- detects Zpassed directives 862 | if newline and tok == '#' then 863 | newline, directive = false, true 864 | elseif not isBlank(tok) then 865 | newline = false 866 | elseif isNewline(tok) then 867 | newline, directive = true, false 868 | end 869 | -- process code 870 | local def = macros[tok] 871 | if not def or directive then 872 | -- not a macro 873 | coroutine.yield(tok, n) 874 | elseif type(def) == 'function' then 875 | -- magic macro 876 | def(ti,tok,n) 877 | elseif def.args == nil then 878 | -- object-like macro 879 | prependToken({tag='pop'},n) 880 | prependTokens(wrap(options, substituteArguments, def, {}, n)) 881 | prependToken({tag='push', symb=tok},n) 882 | else 883 | -- function-like macro 884 | local ntok, nn = tok,n 885 | local spc = false 886 | ti() 887 | if isSpace(tok) then spc=true ti() end 888 | if (tok ~= '(') then 889 | coroutine.yield(ntok, nn) 890 | if spc then coroutine.yield(' ', n) end 891 | if tok then prependToken(tok,n) end 892 | else 893 | local nargs = collectArguments(ti,def,ntok,nn) 894 | if def.lines == nil then 895 | -- single-line function-like macro 896 | prependToken({tag='pop'},n) 897 | prependTokens(wrap(options, substituteArguments, def, nargs, nn)) 898 | prependToken({tag='push', symb=ntok},nn) 899 | else 900 | -- multi-line function-like macro 901 | local lines = def.lines 902 | -- a coroutine that yields the macro definition 903 | local function yieldMacroLines() 904 | local count = 0 905 | for i=1,#lines,2 do 906 | local ls,ln = lines[i], lines[i+1] 907 | -- are we possibly in a cpp directive 908 | local dir = false 909 | if ls[2] and ls[2]:find('^#') then 910 | dir = isIdentifier(ls[3]) and ls[3] or ls[4] 911 | end 912 | if dir and nargs[dir] then 913 | dir = false -- leading stringification 914 | elseif dir == 'defmacro' then 915 | count = count + 1 -- entering a multiline macto 916 | elseif dir == 'endmacro' then 917 | count = count - 1 -- leaving a multiline macro 918 | end 919 | dir = dir or count > 0 920 | -- substitute 921 | ls = callAndCollect(options,substituteArguments,ls,nargs,ln,dir) 922 | -- compute lines (optimize speed by passing body lines as tokens) 923 | local j=1 924 | while isBlank(ls[j]) do j=j+1 end 925 | if ls[j] and ls[j]:find("^#") then -- but not directives 926 | ls = ls[1]:sub(2) .. tableConcat(ls, nil, 2) 927 | end 928 | coroutine.yield(ls,ln) 929 | end 930 | end 931 | -- recursively reenters preprocessing subroutines in order to handle 932 | -- preprocessor directives located inside the macro expansion. As a result 933 | -- we cannot expand macro invocations that extend beyond the macro-expansion. 934 | local nmacros = {} 935 | setmetatable(nmacros,{__index=macros}) 936 | if not def.recursive then nmacros[ntok]=false end 937 | if not def.recursive then coroutine.yield({tag='push',symb=ntok}) end 938 | expandMacros(options, nmacros, tokenize, processDirectives, nmacros, yieldMacroLines) 939 | if not def.recursive then coroutine.yield({tag='pop'}) end 940 | end 941 | end 942 | end 943 | ti() 944 | end 945 | end 946 | 947 | 948 | -- Processing conditional directive requires evaluating conditions 949 | -- This function takes an iterator on preprocessed expression tokens 950 | -- and computes the value. This does not handle defined(X) expressions. 951 | -- Optional argument resolver is a function that takes an indentifer 952 | -- name and returns a value. Otherwise zero is assumed 953 | 954 | local function evaluateCppExpression(options, tokenIterator, n, resolver) 955 | -- redefine token iterator to skip spaces and update tok 956 | local tok 957 | local function ti() 958 | repeat tok = tokenIterator() 959 | until not isBlank(tok) return tok 960 | end 961 | -- operator tables 962 | local unaryOps = { 963 | ["!"] = function(v) return v == 0 and 1 or 0 end, 964 | ["~"] = function(v) return bit.bnot(v) end, 965 | ["+"] = function(v) return v end, 966 | ["-"] = function(v) return -v end, 967 | ["L"] = function(v) return v end 968 | } 969 | local binaryOps = { 970 | ["*"] = function(a,b) return a * b end, 971 | ["/"] = function(a,b) xassert(b~=0,options,n,"division by zero"); return math.floor(a / b) end, 972 | ["%"] = function(a,b) xassert(b~=0,options,n,"division by zero"); return a % b end, 973 | ["+"] = function(a,b) return a + b end, 974 | ["-"] = function(a,b) return a - b end, 975 | [">>"] = function(a,b) return bit.lshift(a, -b) end, 976 | ["<<"] = function(a,b) return bit.lshift(a, b) end, 977 | [">="] = function(a,b) return a >= b and 1 or 0 end, 978 | ["<="] = function(a,b) return a <= b and 1 or 0 end, 979 | [">"] = function(a,b) return a > b and 1 or 0 end, 980 | ["<"] = function(a,b) return a < b and 1 or 0 end, 981 | ["=="] = function(a,b) return a == b and 1 or 0 end, 982 | ["!="] = function(a,b) return a ~= b and 1 or 0 end, 983 | ["&"] = function(a,b) return bit.band(a,b) end, 984 | ["^"] = function(a,b) return bit.bxor(a,b) end, 985 | ["|"] = function(a,b) return bit.bor(a,b) end, 986 | ["&&"] = function(a,b) return (a ~= 0 and b ~= 0) and 1 or 0 end, 987 | ["||"] = function(a,b) return (a ~= 0 or b ~= 0) and 1 or 0 end, 988 | } 989 | local binaryPrec = { 990 | ["*"] = 1, ["/"] = 1, ["%"] = 1, 991 | ["+"] = 2, ["-"] = 2, 992 | [">>"] = 3, ["<<"] = 3, 993 | [">="] = 4, ["<="] = 4, ["<"] = 4, [">"] = 4, 994 | ["=="] = 5, ["!="] = 5, 995 | ["&"] = 6, ["^"] = 7, ["|"] = 8, 996 | ["&&"] = 9, ["||"] = 10 997 | } 998 | -- forward 999 | local function evaluate() end 1000 | -- unary operations 1001 | local function evalUnary() 1002 | if unaryOps[tok] then 1003 | local op = unaryOps[tok] 1004 | ti(); return op(evalUnary()) 1005 | elseif tok == '(' then 1006 | ti(); local v = evaluate() 1007 | xassert(tok == ')', options, n, "missing closing parenthesis") 1008 | ti(); return v 1009 | elseif tok == 'defined' then -- magic macro should have removed this 1010 | xerror(options, n, "syntax error after ") 1011 | elseif isIdentifier(tok) then 1012 | local v = type(resolver) == 'function' and resolver(tok,ti) 1013 | ti(); return v or 0 1014 | elseif isNumber(tok) then 1015 | local v = tok:gsub("[ULul]+$","") 1016 | if v:find("^0[0-7]+$") then 1017 | v = tonumber(v,8) -- octal 1018 | elseif v:find("^0[bB][01]+") then 1019 | v = tonumber(v:sub(3),2) -- binary 1020 | else 1021 | v = tonumber(v) -- lua does the rest 1022 | end 1023 | xassert(v and v == math.floor(v), options, n, "syntax error (invalid integer '%s')", tok) 1024 | ti(); return v 1025 | elseif isString(tok) then 1026 | local v = '""' 1027 | if tok:find("^'") then -- interpret character constant as number 1028 | v = evalLuaExpression(string.format("return string.byte(%s)", tok)) 1029 | xassert(type(v)=='number', options, n, "syntax error (invalid value '%s')", tok) 1030 | ti() 1031 | else 1032 | while isString(tok) do 1033 | xassert(tok:find('^"'), options, n, "syntax error (invalid value '%s')", tok) 1034 | v = v:gsub('"$','') .. tok:gsub('^"','') 1035 | ti() 1036 | end 1037 | end 1038 | return v 1039 | end 1040 | xerror(options, n, "syntax error (invalid value '%s')", tok) 1041 | end 1042 | -- binary operations 1043 | local function evalBinary(p) 1044 | if p == 0 then 1045 | return evalUnary() 1046 | else 1047 | local val = evalBinary(p-1) 1048 | while binaryPrec[tok] == p do 1049 | local op = binaryOps[tok] ti() 1050 | local oval = evalBinary(p-1) 1051 | xassert(p==4 or p==5 or type(val)=='number', options, n, 1052 | "expression uses arithmetic operators on strings") 1053 | xassert(type(val) == type(oval), options, n, 1054 | "expression compares numbers and strings") 1055 | val = op(val,oval) 1056 | end 1057 | return val 1058 | end 1059 | end 1060 | -- eval ternary conditonal 1061 | local function evalTernary() 1062 | local c = evalBinary(10) 1063 | if tok ~= '?' then return c end 1064 | ti() local v1 = evalBinary(10) 1065 | xassert(tok == ':', options, n, "expecting ':' after '?'") 1066 | ti() local v2 = evalBinary(10) 1067 | if c == 0 then return v2 else return v1 end 1068 | end 1069 | 1070 | -- actual definition of evaluate 1071 | evaluate = function() 1072 | return evalTernary() 1073 | end 1074 | -- main function 1075 | ti() 1076 | xassert(tok, options, n, "constant expression expected"); 1077 | local result = evaluate() 1078 | if hasOption(options, "-d:eval") then 1079 | xdebug(n, "eval %s", result) 1080 | end 1081 | -- warn about garbage when called from cpp (but not when called with resolver) 1082 | while isBlank(tok) do ti() end 1083 | xassert(resolver or not tok, options, n, "garbage after conditional expression"); 1084 | return result 1085 | end 1086 | 1087 | 1088 | -- Now dealing with the coroutine that processes all directives. 1089 | -- This coroutine obtains lines from coroutine , 1090 | -- processes all directives, and yields remaining lines 1091 | 1092 | processDirectives = function(options, macros, lines, ...) 1093 | local li = wrap(options, lines, ...) 1094 | local s, n = li() 1095 | -- redefine li to make sure vars s and n are up-to-date 1096 | local li = function() s,n=li() return s,n end 1097 | -- directives store their current token in these vars 1098 | local dirtok, tok, spc 1099 | -- forward declaration 1100 | local processLine 1101 | -- the captureTable mechanism communicates certain preprocessor 1102 | -- events to the declaration parser in order to report them 1103 | -- to the user (the parsing does not depend on this). 1104 | -- if macros[1] is a table, the preprocessor will append 1105 | -- records to this table for the parser to process. 1106 | local function hasCaptureTable() 1107 | local captable = rawget(macros,1) 1108 | return captable and type(captable)=='table' and captable 1109 | end 1110 | local function addToCaptureTable(record) 1111 | local captable = hasCaptureTable() 1112 | if captable and record then captable[1+#captable] = record end 1113 | end 1114 | 1115 | -- simple directives 1116 | local function doIgnore() 1117 | if hasOption(options, "-Zpass") then coroutine.yield(s, n) end 1118 | end 1119 | local function doError() 1120 | xerror(options, n, "unexpected preprocessor directive #%s", dirtok) 1121 | end 1122 | local function doMessage() 1123 | local msg = s:match("^%s*#+%s*[a-z]*%s+([^%s].*)") 1124 | xmessage(dirtok, options, n, msg or '#' .. dirtok) 1125 | end 1126 | -- undef 1127 | local function doUndef(ti) 1128 | ti() 1129 | local nam = tok 1130 | xassert(isIdentifier(nam), options, n, "symbol expected after #undef") 1131 | if ti() then xwarning(options, n, "garbage after #undef directive") end 1132 | if hasOption(options, "-d:defines") then xdebug(n, "undef %s", nam) end 1133 | if hasCaptureTable() and macros[nam] and macros[nam].captured then 1134 | addToCaptureTable{directive='undef',name=nam, where=n} 1135 | end 1136 | macros[nam] = false -- false overrides inherited definitions 1137 | end 1138 | -- define 1139 | local function getMacroArguments(ti) 1140 | local args = {} 1141 | local msg = "argument list in function-like macro" 1142 | local nva = nil -- named variadic argument 1143 | ti() 1144 | while tok and tok ~= ')' do 1145 | local nam = tok 1146 | ti() 1147 | if options.dialectGnu and isIdentifier(nam) and tok == '...' then nam,nva=tok,nam ; ti() end 1148 | xassert(nam ~= "__VA_ARGS__", options, n, "name __VA_ARGS__ is not allowed here") 1149 | xassert(tok == ')' or nam ~= '...', options, n, "ellipsis in argument list must appear last") 1150 | xassert(tok == ')' or tok == ',', options, n, "bad " .. msg) 1151 | if tok == ',' then ti() end 1152 | if nam == '...' then nam = "__VA_ARGS__" end 1153 | xassert(isIdentifier(nam), options, n, "bad " .. msg) 1154 | args[1+#args] = nam 1155 | end 1156 | xassert(tok == ')', options, n, "unterminated " .. msg) 1157 | ti() 1158 | return args, nva 1159 | end 1160 | local function doDefine(ti) 1161 | xassert(isIdentifier(ti()), options, n, "symbol expected after #define") 1162 | local nam, args, nva = tok, nil, nil 1163 | -- collect arguments 1164 | if ti() == '(' and not spc then 1165 | args, nva = getMacroArguments(ti) 1166 | end 1167 | -- collect definition 1168 | local def = { tok, args = args, nva = nva } 1169 | while ti(true) do 1170 | def[1+#def] = tok 1171 | end 1172 | -- define macro 1173 | if macros[nam] and not tableCompare(def,macros[nam]) then 1174 | xwarning(options, n,"redefinition of preprocessor symbol '%s'", nam) 1175 | end 1176 | macros[nam] = def 1177 | -- debug 1178 | if hasOption(options, "-d:defines") then 1179 | if args then args = "(" .. tableConcat(args,",") .. ")" end 1180 | xdebug(n, "define %s%s = %s", nam, args or "", tableConcat(def,' ')) 1181 | end 1182 | -- capture integer macro definitions 1183 | if hasCaptureTable() and args == nil then 1184 | local i = 0 1185 | local v = callAndCollect(options, expandMacros, macros, yieldFromArray, def, n) 1186 | local function ti() i=i+1 return v[i] end 1187 | local ss,r = pcall(evaluateCppExpression,{silent=true}, ti, n, error) 1188 | if ss and type(r)=='number' then 1189 | def.captured = true 1190 | addToCaptureTable{directive='define', name=nam, intval=r, where=n} 1191 | end 1192 | end 1193 | end 1194 | -- defmacro 1195 | local function checkDirective(stop) 1196 | xassert(s, options, n, "unterminated macro (missing #%s)", stop) 1197 | local r = type(s) == 'string' and s:match("^%s*#%s*([a-z]+)") 1198 | if r == "endmacro" or r == "endif" then 1199 | if s:find(r .. "%s*[^%s]") then 1200 | xwarning(options, n, "garbage after #%s directive", r) 1201 | end 1202 | end 1203 | return r 1204 | end 1205 | local function doMacroLines(lines, stop) 1206 | while true do 1207 | li() 1208 | local ss = callAndCollect(options,tokenizeLine,s,n) 1209 | if #ss > 0 then lines[1+#lines] = ss ; lines[1+#lines] = n end 1210 | local r = checkDirective(stop) 1211 | if r == "endmacro" or r == "endif" then 1212 | xassert(r == stop, options,n, "unbalanced directives (got #%s instead of #%s)",r,stop) 1213 | return r 1214 | elseif r=="defmacro" then 1215 | doMacroLines(lines,"endmacro") 1216 | elseif r == "if" or r == "ifdef" or r == "ifndef" then 1217 | doMacroLines(lines,"endif") 1218 | end 1219 | end 1220 | end 1221 | local function doDefmacro(ti) 1222 | xassert(isIdentifier(ti()), options, n, "symbol expected after #defmacro") 1223 | local nam,nn = tok,n 1224 | xassert(ti()=='(', options, n, "argument list expected in #defmacro") 1225 | local args, nva = getMacroArguments(ti) 1226 | xassert(not tok, options, n, "garbage after argument list in #defmacro") 1227 | -- collect definition 1228 | local lines = {} 1229 | local def = { args=args, nva=nva, lines=lines, recursive=(dirtok=="defrecursivemacro") } 1230 | doMacroLines(lines,"endmacro") 1231 | lines[#lines] = nil 1232 | lines[#lines] = nil 1233 | if hasOption(options,"-d:directives") then 1234 | xdebug(n, "directive: #endmacro") 1235 | end 1236 | if macros[nam] and not tableCompare(def,macros[nam]) then 1237 | xwarning(options, n, "redefinition of preprocessor symbol '%s'", nam) 1238 | end 1239 | if hasOption(options, "-d:defines") then 1240 | xdebug(nn, "defmacro %s(%s) =", nam, tableConcat(args,',')) 1241 | for i=1,#lines,2 do 1242 | xdebug(lines[i+1], "\t%s", tableConcat(lines[i]):gsub("^\n","")) end 1243 | end 1244 | macros[nam] = def 1245 | end 1246 | -- include 1247 | local function doInclude(ti) 1248 | -- get filename 1249 | local pti = wrap(options, expandMacros, macros, yieldFromIterator, ti) 1250 | local tok = pti() 1251 | while isBlank(tok) do tok=pti() end 1252 | if tok == '<' then -- computed include 1253 | repeat local tok2 = pti() 1254 | tok = tok .. tostring(tok2) 1255 | until tok2==nil or tok2=='>' or isNewline(tok2) 1256 | tok = tok:gsub('%s>$','>') -- gcc does this 1257 | end 1258 | xassert(isHeaderName(tok), options, n, "malformed header name after #include") 1259 | local ttok = pti() 1260 | while isBlank(ttok) do ttok=pti() end 1261 | if ttok then xwarning(options, n, "garbage after #include directive") end 1262 | -- interpret filename 1263 | local sys = tok:byte() == 60 1264 | local min = dirtok=="include_next" and options.includedir or 0 1265 | local fname = evalLuaExpression(string.format("return '%s'", tok:sub(2,-2))) 1266 | local pname, fd, fdi 1267 | for i,v in ipairs(options) do 1268 | if v == "-I-" then 1269 | sys=false 1270 | elseif i > min and v:find("^%-I") and not sys then 1271 | pname = v:match("^%-I%s*(.*)") .. '/' .. fname 1272 | fdi, fd = i, io.open(pname, "r") 1273 | if fd then break end 1274 | end 1275 | end 1276 | if fd then 1277 | -- include file 1278 | if hasOption(options, "-d:include") then xdebug(n, "including %q", pname) end 1279 | local savedfdi = options.includedir 1280 | options.includedir = fdi -- saved index to implement include_next 1281 | processDirectives(options, macros, eliminateComments, joinLines, 1282 | yieldLines, fd:lines(), pname) 1283 | options.includedir = savedfdi 1284 | else 1285 | -- include file not found 1286 | if hasOption(options, "-Zpass") then 1287 | coroutine.yield(string.format("#include %s",tok), n) 1288 | else 1289 | xwarning(options, n, "include directive (%s) was unresolved", tok) 1290 | end 1291 | -- quirks 1292 | if knownIncludeQuirks[tok] then 1293 | processDirectives(options, macros, eliminateComments, joinLines, 1294 | yieldFromArray, knownIncludeQuirks[tok], n) 1295 | end 1296 | -- capture 1297 | if hasCaptureTable() then 1298 | addToCaptureTable{directive='include',name=tok, where=n} 1299 | end 1300 | end 1301 | end 1302 | -- conditionals 1303 | local function doConditionalBranch(execute) 1304 | checkDirective("endif") 1305 | while true do 1306 | li() 1307 | local r = checkDirective("endif") 1308 | if r == "else" or r == "elif" or r == "endif" then 1309 | return r 1310 | elseif execute then 1311 | processLine() 1312 | elseif r == "if" or r == "ifdef" or r == "ifndef" then 1313 | while doConditionalBranch(false) ~= "endif" do end 1314 | end 1315 | end 1316 | end 1317 | local function doConditional(result) 1318 | local r = doConditionalBranch(result) 1319 | if r == 'elif' and not result then 1320 | return processLine(true) 1321 | end 1322 | while r ~= "endif" do 1323 | r = doConditionalBranch(not result) 1324 | end 1325 | if hasOption(options,"-d:directives") then 1326 | xdebug(n, "directive: %s",s:gsub('^%s*','')) 1327 | end 1328 | end 1329 | local function doIfdef(ti) 1330 | ti() 1331 | xassert(isIdentifier(tok), options, n, "symbol expected after #%s", dirtok) 1332 | local result = macros[tok] 1333 | if ti() then xwarning(options, n, "garbage after #undef directive") end 1334 | if dirtok == 'ifndef' then result = not result end 1335 | doConditional(result) 1336 | end 1337 | local function doIf(ti) 1338 | -- magic macro for 'defined' 1339 | local nmacros = {} 1340 | setmetatable(nmacros,{__index=macros}) 1341 | nmacros['defined'] = function(ti) 1342 | local tok,n = ti() 1343 | if tok == '(' then tok = ti() 1344 | if ti() ~= ')' then tok = nil end end 1345 | if isIdentifier(tok) then 1346 | coroutine.yield(macros[tok] and "1" or "0", n) 1347 | else 1348 | coroutine.yield("defined", n) -- error 1349 | end 1350 | end 1351 | -- evaluate and branch 1352 | local pti = wrap(options, expandMacros, nmacros, yieldFromIterator, ti) 1353 | local result = evaluateCppExpression(options, pti, n) 1354 | doConditional(result ~= 0) 1355 | end 1356 | -- table of directives 1357 | local directives = { 1358 | ["else"] = doError, ["elif"] = doError, ["endif"] = doError, 1359 | ["pragma"] = doIgnore, ["ident"] = doIgnore, ["line"] = doIgnore, 1360 | ["error"] = doMessage, ["warning"] = doMessage, 1361 | ["if"] = doIf, ["ifdef"] = doIfdef, ["ifndef"] = doIfdef, 1362 | ["define"] = doDefine, ["undef"] = doUndef, 1363 | ["defmacro"] = doDefmacro, ["defrecursivemacro"] = doDefmacro, 1364 | ["endmacro"] = doError, 1365 | ["include"] = doInclude, ["include_next"] = doInclude, 1366 | } 1367 | -- process current line 1368 | processLine = function(okElif) 1369 | if type(s) == 'table' then 1370 | -- optimization for multiline macros: 1371 | -- When s is an an array of precomputed tokens, code is assumed. 1372 | coroutine.yield(s, n) 1373 | elseif not s:find("^%s*#") then 1374 | -- code 1375 | coroutine.yield(s, n) 1376 | elseif s:find("^%s*##") and hasOption(options, "-Zpass") then 1377 | -- pass 1378 | local ns = s:gsub("^(%s*)##","%1#") 1379 | coroutine.yield(ns, n) 1380 | else 1381 | if hasOption(options, "-d:directives") then 1382 | xdebug(n, "directive: %s",s:gsub('^%s*','')) 1383 | end 1384 | -- tokenize directive 1385 | local ti = wrap(options, tokenizeLine, s, n) 1386 | -- a token iterator that skips spaces unless told otherwise 1387 | local ti = function(keepSpaces) 1388 | tok = ti() 1389 | spc = isSpace(tok) 1390 | while not keepSpaces and isBlank(tok) do 1391 | tok = ti() 1392 | spc = spc or isSpace(tok) 1393 | end 1394 | return tok, n 1395 | end 1396 | -- start parsing directives 1397 | ti() 1398 | assert(tok=='#' or tok=='##') 1399 | if tok == '##' then 1400 | xwarning(options, n, "directive starts with ## without -Zpass") end 1401 | dirtok = ti() 1402 | if isIdentifier(tok) then 1403 | local f = directives[dirtok] 1404 | if okElif and dirtok == "elif" then f = doIf end 1405 | xassert(f, options, n, "unrecognized preprocessor directive #%s", tok) 1406 | f(ti) 1407 | elseif tok ~= nil then 1408 | xerror(options, n, "unrecognized preprocessor directive '#%s'", s:gsub("^%s*","")) 1409 | end 1410 | end 1411 | end 1412 | -- main loop 1413 | while s ~= nil do 1414 | processLine() 1415 | li() 1416 | end 1417 | end 1418 | 1419 | 1420 | -- This function yields initialization lines 1421 | 1422 | local function initialDefines(options) 1423 | -- cpp-extracted definitions 1424 | if hasOption(options, "-Zcppdef") then 1425 | local fd = io.popen("cpp -dM < /dev/null","r") 1426 | yieldLines(options, fd:lines(), "") 1427 | fd:close() 1428 | end 1429 | -- builtin definitions 1430 | local sb = { "#define __CPARSER__ 1" } 1431 | local function addDef(s,v) 1432 | sb[1+#sb] = string.format("#ifndef %s",s) 1433 | sb[1+#sb] = string.format("# define %s %s",s,v) 1434 | sb[1+#sb] = string.format("#endif") 1435 | end 1436 | addDef("__STDC__", "1") 1437 | local stdc = "199409L" 1438 | if options.dialect11 then stdc = "201112L" end 1439 | if options.dialect99 then stdc = "199901L" end 1440 | addDef("__STDC_VERSION__", stdc) 1441 | if options.dialectGnu then 1442 | addDef("__GNUC__", 4) 1443 | addDef("__GNUC_MINOR__", 2) 1444 | end 1445 | yieldLines(options, wrap(options, yieldFromArray, sb), "") 1446 | -- command line definitions 1447 | local sc = {} 1448 | for _,v in ipairs(options) do 1449 | local d 1450 | if v:find("^%-D(.*)=") then 1451 | d = v:gsub("^%-D%s*(.*)%s*=%s*(.-)%s*$","#define %1 %2") 1452 | elseif v:find("^%-D") then 1453 | d = v:gsub("^%-D%s*(.-)%s*$","#define %1 1") 1454 | elseif v:find("^%-U") then 1455 | d = v:gsub("^%-U%s*(.-)%s*$","#undef %1") 1456 | end 1457 | if d then sc[1+#sc] = d end 1458 | end 1459 | yieldLines(options, wrap(options, yieldFromArray, sc), "") 1460 | end 1461 | 1462 | 1463 | -- This function creates the initial macro directory 1464 | 1465 | local function initialMacros(options) 1466 | local macros = {} 1467 | -- magic macros 1468 | macros["__FILE__"] = function(_,_,n) 1469 | local f 1470 | if type(n) == 'string' then f=n:match("^[^:]*") end 1471 | coroutine.yield(string.format("%q", f or ""), n) 1472 | end 1473 | macros["__LINE__"] = function(_,_,n) 1474 | local d = n 1475 | if type(d) == 'string' then d=tonumber(d:match("%d*$")) end 1476 | coroutine.yield(string.format("%d", d or 0), n) 1477 | end 1478 | macros["__DATE__"] = function(_,_,n) 1479 | coroutine.yield(string.format("%q", os.date("%b %e %Y")), n) 1480 | end 1481 | macros["__TIME__"] = function(_,_,n) 1482 | coroutine.yield(string.format("%q", os.date("%T")), n) 1483 | end 1484 | -- initial macros 1485 | local li = wrap(options,processDirectives,macros,initialDefines) 1486 | for _ in li do end 1487 | -- return 1488 | return macros 1489 | end 1490 | 1491 | 1492 | -- This function prepares a string containing the definition of the 1493 | -- macro named in macro definition table , or nil if no 1494 | -- such definition exists. 1495 | 1496 | local function macroToString(macros, name) 1497 | local v = macros[name] 1498 | if type(v) == 'table' then 1499 | local dir = 'define' 1500 | if v.recursive and v.lines then 1501 | dir = 'defrecursivemacro' 1502 | elseif v.lines then 1503 | dir = 'defmacro' 1504 | end 1505 | local arr = {"#", dir, ' ', name } 1506 | if v.args then 1507 | arr[1+#arr] = '(' 1508 | for i,s in ipairs(v.args) do 1509 | if i ~= 1 then arr[1+#arr] = ',' end 1510 | if s == '__VA_ARGS__' then s = (v.nva or '') .. '...' end 1511 | arr[1+#arr] = s 1512 | end 1513 | arr[1+#arr] = ') ' 1514 | else 1515 | arr[1+#arr] = ' ' 1516 | end 1517 | for _,s in ipairs(v) do 1518 | arr[1+#arr] = s 1519 | end 1520 | if v.lines then 1521 | for i = 1, #v.lines, 2 do 1522 | local vl = v.lines[i] 1523 | arr[1+#arr] = '\n' 1524 | if type(vl)=='table' then vl=tableConcat(vl) end 1525 | arr[1+#arr] = vl:gsub('^%s?%s?',' '):gsub('^\n','') 1526 | end 1527 | arr[1+#arr] = '\n' 1528 | arr[1+#arr] = "#endmacro" 1529 | end 1530 | return tableConcat(arr) 1531 | end 1532 | end 1533 | 1534 | -- This function dumps all macros to file descriptor outputfile 1535 | 1536 | local function dumpMacros(macros, outputfile) 1537 | outputfile = outputfile or io.output() 1538 | assert(type(macros) == 'table') 1539 | assert(io.type(outputfile) == 'file') 1540 | for k,_ in pairs(macros) do 1541 | local s = macroToString(macros,k) 1542 | if s then outputfile:write(string.format("%s\n", s)) end 1543 | end 1544 | end 1545 | 1546 | 1547 | 1548 | -- A coroutine that filters out spaces, directives, and magic tokens 1549 | 1550 | local function filterSpaces(options, tokens, ...) 1551 | local ti = wrap(options, tokens, ...) 1552 | local tok,n = ti() 1553 | while tok do 1554 | -- skip directives 1555 | while isNewline(tok) do 1556 | tok,n = ti() 1557 | while isBlank(tok) do tok,n = ti() end 1558 | if tok == '#' then 1559 | while not isNewline(tok) do 1560 | tok, n = ti() 1561 | end 1562 | end 1563 | end 1564 | -- output nonspaces 1565 | if not isBlank(tok) then 1566 | coroutine.yield(tok, n) 1567 | end 1568 | tok,n = ti() 1569 | end 1570 | end 1571 | 1572 | -- This function takes a line iterator and an optional location prefix. 1573 | -- It returns a token iterator for the preprocessed tokens 1574 | -- and a table of macro definitions. 1575 | 1576 | local function cppTokenIterator(options, lines, prefix) 1577 | options = copyOptions(options) 1578 | prefix = prefix or "" 1579 | assert(type(options)=='table') 1580 | assert(type(lines)=='function') 1581 | assert(type(prefix)=='string') 1582 | local macros = initialMacros(options) 1583 | local ti = wrap(options, 1584 | filterSpaces, 1585 | expandMacros, macros, 1586 | tokenize, 1587 | processDirectives, macros, 1588 | eliminateComments, 1589 | joinLines, 1590 | yieldLines, lines, prefix) 1591 | return ti, macros 1592 | end 1593 | 1594 | 1595 | -- A coroutine that reconstructs lines from the preprocessed tokens 1596 | 1597 | local function preprocessedLines(options, tokens, ...) 1598 | local ti = wrap(options, tokens, ...) 1599 | local tok,n = ti() 1600 | while tok do 1601 | local curn = n 1602 | local curl = {} 1603 | if isNewline(tok) then 1604 | curn = n 1605 | curl[1+#curl] = tok:sub(2) 1606 | tok, n = ti() 1607 | end 1608 | while tok and not isNewline(tok) do 1609 | if not isMagic(tok) then curl[1+#curl] = tok end 1610 | tok, n = ti() 1611 | end 1612 | coroutine.yield(table.concat(curl), curn) 1613 | end 1614 | end 1615 | 1616 | 1617 | -- This function preprocesses file . 1618 | -- The optional argument specifies where to write the 1619 | -- preprocessed file and may be a string or a file descriptor. 1620 | -- The optional argument contains an array of option strings. 1621 | -- Note that option "-Zpass" is added, unless the option "-Znopass" is present. 1622 | 1623 | local function cpp(filename, outputfile, options) 1624 | -- handle optional arguments 1625 | options = copyOptions(options) 1626 | outputfile = outputfile or "-" 1627 | assert(type(filename)=='string') 1628 | assert(type(options)=='table') 1629 | local closeoutputfile = false 1630 | if io.type(outputfile) ~= 'file' then 1631 | assert(type(outputfile) == 'string') 1632 | if outputfile == '-' then 1633 | outputfile = io.output() 1634 | else 1635 | closeoutputfile = true 1636 | outputfile = io.open(outputfile,"w") 1637 | end 1638 | end 1639 | assert(io.type(outputfile) == 'file') 1640 | -- makes option -Zpass on by default 1641 | if not hasOption(options,"-Znopass") then 1642 | options.hash["-Zpass"] = true 1643 | end 1644 | -- prepare iterator 1645 | local dM = hasOption(options, "-dM") 1646 | local macros = initialMacros(options) 1647 | local li = wrap(options, 1648 | preprocessedLines, 1649 | expandMacros, macros, 1650 | tokenize, 1651 | processDirectives, macros, 1652 | eliminateComments, 1653 | joinLines, 1654 | yieldLines, io.lines(filename), filename) 1655 | -- iterate, inserting line markers 1656 | local lm = hasOption(options,"-Zpass") and "line" or "" 1657 | local cf, cn 1658 | for s,n in li do 1659 | if not dM and s:find("[^%s]") then 1660 | local xf, xn 1661 | if type(n) == 'number' then 1662 | xn = n 1663 | elseif type(n) == 'string' then 1664 | xf, xn = n:match("^([^:]*).-(%d*)$") 1665 | xn = tonumber(xn) 1666 | end 1667 | if cf ~= xf or cn ~= xn then 1668 | cf, cn = xf, xn 1669 | outputfile:write(string.format("#%s %d %q\n", lm, cn, cf)) 1670 | end 1671 | outputfile:write(s) 1672 | outputfile:write("\n") 1673 | cn = cn + 1 1674 | end 1675 | end 1676 | if dM then 1677 | dumpMacros(macros, outputfile) 1678 | end 1679 | if closeoutputfile then outputfile:close() end 1680 | end 1681 | 1682 | 1683 | 1684 | --------------------------------------------------- 1685 | --------------------------------------------------- 1686 | --------------------------------------------------- 1687 | -- PARSING DECLARATIONS 1688 | 1689 | -- Simple tuples are constructed using "tuple=Pair{a,b}" and accessed 1690 | -- as tuple[1],tuple[2] etc. Although this is named Pair, one can use 1691 | -- more than two args. 1692 | 1693 | local Pair = newTag('Pair') 1694 | 1695 | -- Types are represented by a series of tagged data structures. 1696 | -- Subfield usually contains the base type or the function return 1697 | -- type. Subfield contains the name of the structure, union, or 1698 | -- enum. Numerical indices are used for struct components and 1699 | -- function arguments. The construct Type{n=...} is used for named 1700 | -- types, including basic types, typedefs, and tagged struct, unions 1701 | -- or enums. When the named type has a better definition the hidden 1702 | -- field <_def> contains it. They should be constructed with function 1703 | -- namedType() because it is expected that there is only one copy of 1704 | -- each named type. The construct Qualified{t=...} is used to 1705 | -- represent const/volatile/restrict variants of the base type. 1706 | -- 1707 | -- Examples 1708 | -- long int a Type{n="long int"} 1709 | -- int *a Pointer{t=Type{n="int"}} 1710 | -- const int *a Pointer{t=Qualified{const=true,t=Type{n="int"}}} 1711 | -- int* const a Qualified{const=true,t=Pointer{t=Type{n="int"}}} 1712 | -- void foo(int bar) Function{Pair{Type{n="int"},"bar"},t=Type{n="void"}}} 1713 | -- int foo(void) Function{t=Type{n="int"}} 1714 | -- int foo() Function{t=Type{n="int"},withoutProto=true} 1715 | 1716 | local Type = newTag('Type') 1717 | local Qualified = newTag('Qualified') 1718 | local Pointer = newTag('Pointer') 1719 | local Array = newTag('Array') 1720 | local Enum = newTag('Enum') 1721 | local Struct = newTag('Struct') 1722 | local Union = newTag('Union') 1723 | local Function = newTag('Function') 1724 | 1725 | 1726 | -- This function creates a qualified variant of a type. 1727 | 1728 | local function addQualifier(ty, q) 1729 | assert(q=='const' or q=='volatile' or q=='restrict') 1730 | if ty.Tag ~= 'Qualified' then ty = Qualified{t=ty} end 1731 | ty[q] = true 1732 | return ty 1733 | end 1734 | 1735 | local function typeIs(ty,tag) 1736 | assert(ty) 1737 | if ty.tag == 'Qualified' then ty = ty.t end 1738 | return ty.tag == tag 1739 | end 1740 | 1741 | -- This function compares two types. When optional argument is 1742 | -- not false and types t1 or t2 are incomplete, the function returns 1743 | -- true if the types are compatible: an unsized array matches a sized 1744 | -- array, a function type without prototype matches one with a 1745 | -- prototype. Furthermore, if oki is <1>, the function will patch type 1746 | -- to contain the complete information. 1747 | 1748 | local function compareTypes(t1, t2, oki) 1749 | if t1 == t2 then 1750 | return t1 1751 | elseif t1.tag == 'Type' and t1._def then 1752 | return compareTypes(t1._def, t2, oki) 1753 | elseif t2.tag == 'Type' and t2._def then 1754 | return compareTypes(t1, t2._def, oki) 1755 | elseif t1.tag == 'Qualified' or t2.tag == 'Qualified' then 1756 | if t1.tag ~= 'Qualified' then 1757 | return compareTypes(Qualified{t=t1},t2) 1758 | elseif t2.tag ~= 'Qualified' then 1759 | return compareTypes(t1,Qualified{t=t2}) 1760 | else 1761 | if t1.const ~= t2.const then return false end 1762 | if t1.volatile ~= t2.volatile then return false end 1763 | if t1.restrict ~= t2.restrict then return false end 1764 | if oki==1 then t1.attr=tableAppend(t1.attr,t2.attr) end 1765 | return compareTypes(t1.t, t2.t, oki) 1766 | end 1767 | elseif t1.tag ~= t2.tag then 1768 | return false 1769 | elseif t1.tag == 'Pointer' then 1770 | if t1.block ~= t2.block then return false end 1771 | if t1.ref ~= t2.ref then return false end 1772 | return compareTypes(t1.t, t2.t, oki) 1773 | elseif t1.tag == 'Array' then 1774 | if compareTypes(t1.t, t2.t, oki) then 1775 | if t1.size == t2.size then return true end 1776 | if t1.size == nil or t2.size == nil then 1777 | if oki == 1 and t1.size == nil then t1.size = t2.size end 1778 | return oki 1779 | end 1780 | end 1781 | elseif t1.tag == 'Function' then 1782 | if compareTypes(t1.t, t2.t, oki) then 1783 | if oki==1 then t1.attr=tableAppend(t1.attr,t2.attr) end 1784 | if t1.withoutProto or t2.withoutProto then 1785 | if t1.withoutProto and t2.withoutProto then return true end 1786 | if oki == 1 and t1.withoutProto then 1787 | for i=1,#t2 do t1[i] = t2[i] end 1788 | t1.withoutProto = nil 1789 | end 1790 | return oki 1791 | elseif #t1 == #t2 then 1792 | for i=1,#t1 do 1793 | if t1[i][1] == nil or t2[i][1] == nil then 1794 | return t1[i].ellipsis and t2[i].ellipsis 1795 | elseif not compareTypes(t1[i][1],t2[i][1],oki) then 1796 | return false 1797 | end 1798 | end 1799 | return true 1800 | end 1801 | end 1802 | elseif t1.tag == 'Enum' then 1803 | return false 1804 | elseif #t1 == #t2 then -- struct or union 1805 | for i=1,#t1 do 1806 | if t1[i][2] ~= t2[i][2] then return false end 1807 | if t1[i].bitfield ~= t2[i].bitfield then return false end 1808 | if not compareTypes(t1[i][1],t2[i][1],oki) then return false end 1809 | end 1810 | if oki==1 then t1.attr=tableAppend(t1.attr,t2.attr) end 1811 | if t1.n == t2.n then return true end 1812 | if t1.n == nil or t2.n == nil then 1813 | if oki == 1 and t1.n == nil then t1.n = t2.n end 1814 | return oki 1815 | end 1816 | end 1817 | return false 1818 | end 1819 | 1820 | 1821 | -- Util 1822 | 1823 | local function spaceNeededBetweenTokens(t1,t2) 1824 | if not t1 or not t2 then return false end 1825 | local it1 = isIdentifier(t1) or isNumber(t1) 1826 | local it2 = isIdentifier(t2) or isNumber(t2) 1827 | if it1 and it2 then return true end 1828 | if it1 and not it2 or not it1 and it2 then return false end 1829 | local z = callAndCollect({silent=true},tokenizeLine,t1..t2,"internal",true) 1830 | return z[1]~=t1 or z[2]~=t2 1831 | end 1832 | 1833 | 1834 | -- Constructs a string suitable for declaring a variable of type 1835 | -- in a C program. Argument defaults to "%s". 1836 | 1837 | local function typeToString(ty, nam) 1838 | nam = nam or "%s" 1839 | assert(type(nam) == 'string') 1840 | local function parenthesize(nam) 1841 | return '(' .. nam .. ')' 1842 | end 1843 | local function insertword(word,nam) 1844 | if nam:find("^[A-Za-z0-9$_%%]") then nam = ' ' .. nam end 1845 | return word .. nam 1846 | end 1847 | local function makelist(ty,sep) 1848 | local s = '' 1849 | for i=1,#ty do 1850 | if i>1 then s = s .. sep end 1851 | if ty[i].ellipsis then s = s .. '...' 1852 | else s = s .. typeToString(ty[i][1], ty[i][2] or "") end 1853 | if ty[i].bitfield then 1854 | s = s .. ':' .. tostring(ty[i].bitfield) 1855 | end 1856 | end 1857 | return s 1858 | end 1859 | local function initstr(arr) 1860 | local s = {} 1861 | for i=1,#arr,2 do 1862 | if spaceNeededBetweenTokens(arr[i-2],arr[i]) then s[1+#s] = ' ' end 1863 | s[1+#s] = arr[i] 1864 | end 1865 | return table.concat(s) 1866 | end 1867 | local function insertqual(ty,nam) 1868 | if ty and ty.attr then nam = insertword(initstr(ty.attr),nam) end 1869 | if ty and ty.restrict then nam = insertword("restrict",nam) end 1870 | if ty and ty.volatile then nam = insertword("volatile",nam) end 1871 | if ty and ty.const then nam= insertword("const",nam) end 1872 | if ty and ty.static then nam= insertword("static",nam) end 1873 | return nam 1874 | end 1875 | -- main loop 1876 | while true do 1877 | local qty = nil 1878 | while ty.tag == 'Qualified' do 1879 | qty = ty ty = ty.t 1880 | end 1881 | if qty and qty.static and ty.tag == 'Pointer' then 1882 | ty = Array{t=ty.t, size=qty.static} 1883 | end 1884 | if ty.tag == 'Type' then 1885 | return insertqual(qty, insertword(ty.n, nam)) 1886 | elseif ty.tag == 'Pointer' then 1887 | local star = (ty.block and '^') or (ty.ref and '&') or '*' 1888 | nam = star .. insertqual(qty, nam) 1889 | ty = ty.t 1890 | elseif ty.tag == 'Array' then 1891 | local sz = ty.size or '' 1892 | if nam:find("^[*^]") then nam = parenthesize(nam) end 1893 | nam = nam .. '[' .. insertqual(qty, tostring(sz)) .. ']' 1894 | ty = ty.t 1895 | elseif ty.tag == 'Function' then 1896 | if nam:find("^[*^]") then nam = parenthesize(nam) end 1897 | if #ty == 0 and ty.withoutProto then nam = nam .. '()' 1898 | elseif #ty == 0 then nam = nam .. '(void)' 1899 | else nam = nam .. '(' .. makelist(ty,',') .. ')' end 1900 | if ty.attr then nam = nam .. initstr(ty.attr) end 1901 | if qty then nam = nam .. insertqual(qty,'') end 1902 | ty = ty.t 1903 | elseif ty.tag == 'Enum' then 1904 | local s = insertqual(qty, 'enum') 1905 | if ty.attr then s = s .. ' ' .. initstr(ty.attr) end 1906 | if ty.n then s = s .. ' ' .. ty.n end 1907 | s = s .. '{' 1908 | for i=1,#ty do 1909 | if i > 1 then s = s .. ',' end 1910 | s = s .. ty[i][1] 1911 | if ty[i][2] then 1912 | s = s .. '=' .. tostring(ty[i][2]) 1913 | end 1914 | end 1915 | return s .. '}' .. nam 1916 | else 1917 | local s = insertqual(qty, string.lower(ty.tag)) 1918 | if ty.attr then s = s .. ' ' .. initstr(ty.attr) end 1919 | if ty.n then s = s .. ' ' .. ty.n end 1920 | return s .. '{' .. makelist(ty,';') .. ';}' .. nam; 1921 | end 1922 | end 1923 | end 1924 | 1925 | 1926 | -- Tables Definition{} and Declaration{} represent variable and 1927 | -- constant definitions and declarations found in the code. Field 1928 | -- is the location of the definition or declaration, field 1929 | -- is the name of the variable or constant being defined, field 1930 | -- contains the type, field optionally contain the 1931 | -- initialization or the function body. Field contains the 1932 | -- storage class such as , , . Special storage 1933 | -- class '[enum]' is used to define enumeration constants. Table 1934 | -- TypeDef{} represents type definitions and contains pretty much the 1935 | -- same fields. Note that storage class is used for an 1936 | -- actual and storage class <[typetag]> is used when the 1937 | -- type definition results from a tagged structure union or enum. 1938 | -- CppEvent{} is used to report captured cpp events. 1939 | 1940 | local TypeDef = newTag('TypeDef') 1941 | local Definition = newTag('Definition') 1942 | local Declaration = newTag('Declaration') 1943 | local CppEvent = newTag('CppEvent') 1944 | 1945 | local function declToString(action) 1946 | local tag = action and action.tag 1947 | if tag == 'TypeDef' or tag == 'Definition' or tag == 'Declaration' then 1948 | local n = (action.sclass == '[typetag]') and "" or action.name 1949 | local s = typeToString(action.type, n) 1950 | if action.type.inline then 1951 | s = 'inline' .. ' ' .. s 1952 | end 1953 | if action.sclass then 1954 | s = action.sclass .. ' ' .. s 1955 | end 1956 | if action.intval then 1957 | s = s .. ' = ' .. action.intval 1958 | elseif action.init and typeIs(action.type, 'Function') then 1959 | s = s .. "{..}" 1960 | elseif action.init then 1961 | s = s .. "=.." 1962 | end 1963 | return s 1964 | elseif tag == 'CppEvent' then 1965 | local s = nil 1966 | if action.directive == 'include' then 1967 | s = string.format("#include %s", action.name) 1968 | elseif action.directive == 'define' then 1969 | s = string.format("#define %s %s", action.name, action.intval) 1970 | elseif action.directive == 'undef' then 1971 | s = string.format("#undef %s", action.name) 1972 | end 1973 | return s 1974 | end 1975 | end 1976 | 1977 | 1978 | -- The symbol table is implemented by a table that contains Type{} 1979 | -- nodes for type definitions (possibly with a hidden <_def> field 1980 | -- pointing to the full definition), Definition{} or Declaration{] 1981 | -- nodes for all other names. 1982 | 1983 | local function isTypeName(symtable, name) 1984 | local ty = symtable[name] 1985 | if ty and ty.tag == 'Type' then return ty end 1986 | return false 1987 | end 1988 | 1989 | local function newScope(symtable) 1990 | local newSymtable = {} 1991 | setmetatable(newSymtable, {__index=symtable}) 1992 | return newSymtable 1993 | end 1994 | 1995 | 1996 | -- Returns an iterator that can look tokens ahead. 1997 | -- Calling it without arguments works like an ordinary iterator. 1998 | -- Calling it with argument 0 returns the current token. 1999 | -- Calling it with a positive argument look positions ahead. 2000 | -- Calling it with argument -1 pushes back the last token. 2001 | 2002 | local function lookaheadTokenIterator(ti) 2003 | local tok,n = ti() 2004 | local fifo = {} 2005 | return function(arg) 2006 | if not arg then 2007 | if fifo[1] then 2008 | tok,n = unpack(fifo[1]) 2009 | table.remove(fifo,1) 2010 | return tok,n 2011 | else 2012 | tok,n = ti() 2013 | return tok,n 2014 | end 2015 | elseif arg == 0 then 2016 | return tok, n 2017 | elseif arg == -1 then 2018 | table.insert(fifo,1,{tok,n}) 2019 | return tok, n 2020 | else 2021 | assert(type(arg)=='number' and arg > 0) 2022 | while arg > #fifo do fifo[1+#fifo] = {ti()} end 2023 | return unpack(fifo[arg]) 2024 | end 2025 | end 2026 | end 2027 | 2028 | 2029 | -- Evaluation of constant expression. 2030 | -- We avoid writing a complete expression parser by reusing the cpp 2031 | -- expression parser and either returning an integer (when we can 2032 | -- evaluate) or a string containing the expression (when we can't) or 2033 | -- nil (when we are sure this is not a number). Array contains 2034 | -- tokens (odd indices) followed by location (even indices). Argument 2035 | -- is the symbol table. 2036 | -- The alternative is to write a proper expression parse with 2037 | -- constant folding as well as providing means to evaluate the value 2038 | -- of the sizeof and alignof operators. This but might be needed if 2039 | -- one wants to compute struct layouts. 2040 | 2041 | local function tryEvaluateConstantExpression(options, n, arr, symtable) 2042 | -- array initializers never are constant integers 2043 | if arr[1] == '{' then return nil,false end 2044 | -- try direct evaluation 2045 | local ari = -1 2046 | local function ti(arg) 2047 | if not arg then ari = ari + 2 ; arg = 0 end 2048 | if arg < 0 and ari > -1 then ari = ari - 2 ; arg = 1 end 2049 | return arr[ari+2*arg],arr[ari+2*arg+1] 2050 | end 2051 | local function rsym(tok) 2052 | local s = symtable and symtable[tok] 2053 | xassert(s and type(s.intval)=='number', {silent=true}, n, 2054 | "symbol '%s' does not resolve to a constant integer", s) 2055 | return s.intval 2056 | end 2057 | local ss,r = pcall(evaluateCppExpression, {silent=true}, ti, n, rsym) 2058 | if ss and type(r)=='number' and not ti(0) then return r,true end 2059 | if ss and type(r)~='number' and not ti(0) then return nil,false end 2060 | -- just return an expression string 2061 | local s = {} 2062 | for i=1,#arr,2 do 2063 | if spaceNeededBetweenTokens(arr[i-2],arr[i]) then 2064 | s[1+#s] = ' ' 2065 | end 2066 | if isName(arr[i]) and symtable[arr[i]] and symtable[arr[i]].eval then 2067 | s[1+#s] = string.format("(%s)", symtable[arr[i]].eval) 2068 | else 2069 | s[1+#s] = arr[i] 2070 | end 2071 | end 2072 | s = table.concat(s) 2073 | xwarning(options, n, "cparser cannot evaluate '%s' as an integer constant" 2074 | .. " and is using the literal expression instead", s) 2075 | return s, false 2076 | end 2077 | 2078 | 2079 | -- Specifier table. 2080 | -- This function return a table that categorize the meaning 2081 | -- of all the type specifier keywords. 2082 | 2083 | local function getSpecifierTable(options) 2084 | options.specifierTable = options.specifierTable or { 2085 | typedef = 'sclass', 2086 | extern = 'sclass', 2087 | static = 'sclass', 2088 | auto = 'sclass', 2089 | register = 'sclass', 2090 | void = 'type', 2091 | char = 'type', 2092 | float = 'type', 2093 | int = 'type', 2094 | double = 'type', 2095 | short = 'size', 2096 | long = 'size', 2097 | signed = 'sign', 2098 | unsigned = 'sign', 2099 | const = 'const', 2100 | volatile = 'volatile', 2101 | struct = 'struct', 2102 | union = 'struct', 2103 | enum = 'enum', 2104 | __inline__ = 'inline', -- gnu 2105 | __asm__ = 'attr', -- gnu 2106 | __restrict__ = 'restrict', -- gnu 2107 | __attribute__ = 'attr', -- gnu 2108 | __extension__ = 'extension', -- gnu 2109 | __pragma = 'attr', -- msvc 2110 | __asm = 'attr', -- msvc 2111 | __declspec = 'attr', -- msvc 2112 | __restrict = 'restrict', -- msvc 2113 | __inline = 'inline', -- msvc 2114 | __forceinline = 'inline', -- msvc 2115 | __cdecl = 'attr', -- msvc 2116 | __fastcall = 'attr', -- msvc 2117 | __stdcall = 'attr', -- msvc 2118 | __based = 'attr', -- msvc 2119 | __int8 = 'type', -- msvc 2120 | __int16 = 'type', -- msvc 2121 | __int32 = 'type', -- msvc 2122 | __int64 = 'type', -- msvc 2123 | _Bool = not options.dialectAnsi and 'type', 2124 | restrict = not options.dialectAnsi and 'restrict', 2125 | _Complex = not options.dialectAnsi and 'complex', 2126 | _Imaginary = not options.dialectAnsi and 'complex', 2127 | _Atomic = not options.dialectAnsi and 'atomic', 2128 | inline = not options.dialectAnsi and 'inline', 2129 | _Pragma = not options.dialectAnsi and "attr", 2130 | __thread = options.dialectGnu and "attr", 2131 | asm = options.dialectGnu and "attr", 2132 | _Alignas = options.dialect11 and "attr", 2133 | _Noreturn = options.dialect11 and "attr", 2134 | _Thread_local = options.dialect11 and "attr", 2135 | } 2136 | return options.specifierTable 2137 | end 2138 | 2139 | 2140 | -- This coroutine is the declaration parser 2141 | -- Argument is the global symbol table. 2142 | -- Argument is a coroutine that yields program tokens. 2143 | 2144 | local function parseDeclarations(options, globals, tokens, ...) 2145 | -- see processMacroCaptures around the end of this function 2146 | if type(options.macros) == 'table' then options.macros[1] = {} end 2147 | 2148 | -- define a lookahead token iterator that also ensures that 2149 | -- variables tok,n always contain the current token 2150 | local ti = lookaheadTokenIterator(wrap(options, tokens, ...)) 2151 | local tok,n = ti(0) 2152 | local ti = function(arg) 2153 | if arg then return ti(arg) end 2154 | tok,n = ti() 2155 | -- print(string.format("*** [%s] (%s)",tok,n)) 2156 | return tok,n 2157 | end 2158 | 2159 | -- this function is used to retrieve or construct Type{} nodes for 2160 | -- named types. Since the Type constructor should not be used we 2161 | -- override it with a function that calls assert(false) 2162 | local function namedType(symtable, nam) 2163 | local ty = symtable[nam] 2164 | if ty and ty.tag == 'Type' then 2165 | return ty 2166 | elseif ty and ty.tag ~= 'Type' then 2167 | local msg = " previous declaration at %s" 2168 | if rawget(symtable,nam) then 2169 | xerror(options, n, "type name '%s' conflicts with" .. msg, nam, ty.where) 2170 | else 2171 | xwarning(options, n, "type name '%s' shadows" .. msg, nam, ty.where) 2172 | end 2173 | end 2174 | ty = Type{n=nam} 2175 | symtable[nam] = ty 2176 | return ty 2177 | end 2178 | local function Type() assert(false) end 2179 | 2180 | -- unique id generator 2181 | local unique_int = 0 2182 | local function unique() 2183 | unique_int = unique_int + 1 2184 | return string.format("%s_%05d", options.unique_prefix or "__anon", unique_int) 2185 | end 2186 | 2187 | -- check that current token is one of the provided token strings 2188 | local function check(s1,s2) 2189 | if tok == s1 then return end 2190 | if tok == s2 then return end 2191 | if not s2 then 2192 | xerror(options,n,"expecting '%s' but got '%s'", s1, tok) 2193 | else 2194 | xerror(options,n,"expecting '%s' or '%s' but got '%s'", s1, s2, tok) 2195 | end 2196 | end 2197 | 2198 | -- record tokens into array arr if non nil 2199 | local function record(arr) 2200 | if arr then arr[1+#arr]=tok arr[1+#arr]=n end 2201 | end 2202 | 2203 | -- skip parenthesized expression stating on current token. 2204 | -- return nil if current token is not a left delimiter. 2205 | -- new current token immediately follow right delimiter. 2206 | -- optionally record tokens into arr and return arr 2207 | local function skipPar(arr) 2208 | local dleft = { ["("]=")", ["{"]="}", ["["]="]" } 2209 | local dright = { [")"]=1, ["}"]=1, ["]"]=1 } 2210 | local stok = dleft[tok] 2211 | if stok then 2212 | local sn = n 2213 | local ltok = tok 2214 | record(arr) ti() 2215 | while not dright[tok] do 2216 | xassert(tok, options, sn, "no matching '%s' for this '%s'", stok, ltok) 2217 | if dleft[tok] then skipPar(arr) else record(arr) ti() end 2218 | end 2219 | xassert(tok==stok, options, n, "expecting '%s' but got '%s'", tok, stok) 2220 | record(arr) ti() 2221 | return arr 2222 | end 2223 | end 2224 | 2225 | -- skip balanced tokens until reaching token s1 or s2 or s2. 2226 | -- in addition s1 may be a table whose keys are the stop token. 2227 | -- the new current token immediately follows the stop token. 2228 | -- optionally records tokens into arr and returns arr. 2229 | local function skipTo(arr,s1,s2,s3,s4) 2230 | local sn = n 2231 | while tok and tok ~= s1 and tok ~= s2 and tok ~= s3 and tok ~= s4 do 2232 | if type(s1) == 'table' and s1[tok] then break end 2233 | if not skipPar(arr) then record(arr) ti() end end 2234 | xassert(tok,options,sn,"unterminated expression") 2235 | return arr 2236 | end 2237 | 2238 | 2239 | -- processDeclaration. 2240 | -- Argument is the file/line of the declaration. 2241 | -- Argument is the current symbol table. 2242 | -- Argument is 'global', 'param', 'local' 2243 | local function processDeclaration(where, symtable, context, name, ty, sclass, init) 2244 | local dcl 2245 | -- handle type definitions 2246 | if sclass == 'typedef' or sclass == '[typetag]' then 2247 | local nty = namedType(symtable, name) 2248 | nty._def = ty 2249 | symtable[name] = nty 2250 | dcl = TypeDef{name=name,type=ty,where=where,sclass=sclass} 2251 | if context == 'global' then coroutine.yield(dcl) end 2252 | return 2253 | end 2254 | -- handle variable and constants 2255 | if typeIs(ty, 'Function') then 2256 | if init then 2257 | dcl = Definition{name=name,type=ty,sclass=sclass,where=where,init=init} 2258 | else 2259 | dcl = Declaration{name=name,type=ty,sclass=sclass,where=where} 2260 | end 2261 | else 2262 | if sclass == 'extern' 2263 | or ty.const and not init and sclass ~= '[enum]' 2264 | or ty.tag=='Array' and not ty.size and not init 2265 | then 2266 | xassert(not init,options,n,"extern declaration cannot have initializers") 2267 | dcl = Declaration{name=name,type=ty,sclass=sclass,where=where} 2268 | else 2269 | local v = ty.tag == 'Qualified' and ty.const and init or nil 2270 | if type(v) == 'table' then 2271 | v = tryEvaluateConstantExpression(options,where,init,symtable) 2272 | elseif type(v) == 'number' then -- happens when called from parseEnum 2273 | init = {tostring(v), where} 2274 | end 2275 | dcl = Definition{name=name,type=ty,sclass=sclass,where=where,init=init,intval=v} 2276 | end 2277 | end 2278 | -- check for duplicate declaration 2279 | local ddcl = dcl 2280 | if dcl.tag ~= 'TypeDef' then 2281 | local odcl = symtable[name] 2282 | local samescope = rawget(symtable, name) 2283 | -- compare types 2284 | if odcl and samescope then 2285 | if dcl.tag == 'Definition' and odcl.tag == 'Definition' 2286 | or not compareTypes(dcl.type,odcl.type,true) then 2287 | xerror(options,where, 2288 | "%s of symbol '%s' conflicts with earlier %s at %s", 2289 | string.lower(dcl.tag), name, 2290 | string.lower(odcl.tag), odcl.where) 2291 | end 2292 | if odcl.tag == 'Definition' then 2293 | ddcl = odcl 2294 | compareTypes(ddcl.type, dcl.type, 1) 2295 | else 2296 | compareTypes(ddcl.type, odcl.type, 1) 2297 | end 2298 | end 2299 | -- compare storage class 2300 | if odcl and dcl.sclass ~= odcl.sclass then 2301 | if dcl.sclass == 'static' or samescope and odcl.sclass == 'static' then 2302 | xerror(options, n, "inconsistent linkage for '%s' (previous at %s)", 2303 | name, odcl.where) 2304 | end 2305 | end 2306 | -- install dcl in symtable and yield global declarations 2307 | symtable[name] = ddcl 2308 | if context == 'global' then coroutine.yield(dcl) end 2309 | end 2310 | end 2311 | 2312 | -- forward declations of parsing functions 2313 | local parseDeclaration 2314 | local parseDeclarationSpecifiers 2315 | local parseDeclarator, parsePrototype 2316 | local parseEnum, parseStruct 2317 | 2318 | -- C declarations have a left part that contains a type 2319 | -- and comma separated right parts that contain the variable 2320 | -- name in expressions that mimic how one would use the 2321 | -- variable to obtain the type specified by the left part. 2322 | -- The left part is called a DeclarationSpecifier 2323 | -- and the right parts are called Declarators. 2324 | 2325 | -- token classification table for speeding up type parsing 2326 | local specifierTable = getSpecifierTable(options) 2327 | 2328 | -- appends attributes to table 2329 | local function isAttribute() 2330 | return specifierTable[tok]=='attr' or 2331 | options.dialect11 and tok=='[' and ti(1)=='[' 2332 | end 2333 | local function collectAttributes(arr) 2334 | while isAttribute() do 2335 | arr = arr or {} 2336 | if tok~='[' then arr[1+#arr]=tok; arr[1+#arr]=n; ti() end 2337 | if tok=='(' or tok =='[' then skipPar(arr) end 2338 | end 2339 | return arr 2340 | end 2341 | 2342 | -- This function parses the left part and returns the type, and a table 2343 | -- containing all the additional information we could collect, namely the 2344 | -- presence of an inline keyword or the tokens associated with 2345 | -- compiler-specific attribute syntax. 2346 | parseDeclarationSpecifiers = function(symtable, context, abstract) 2347 | local ty 2348 | local nn = {} 2349 | while true do 2350 | local ltok = tok 2351 | local p = specifierTable[tok] 2352 | if isAttribute() then 2353 | p = 'attr'; nn.attr = collectAttributes(nn.attr) 2354 | elseif p == 'enum' then 2355 | p = 'type'; ty = parseEnum(symtable, context, abstract, nn) 2356 | elseif p == 'struct' then 2357 | p = 'type'; ty = parseStruct(symtable, context, abstract, nn) 2358 | elseif p then 2359 | ti() 2360 | elseif isName(tok) then 2361 | local tt = isTypeName(symtable, tok) 2362 | local yes = not nn.type and not nn.size and not nn.sign and not nn.complex 2363 | if not tt then 2364 | local tok1 = ti(1) 2365 | local no = not abstract and tok1:find("^[;,[]") 2366 | if yes and not no then -- assume this is a type name 2367 | p = 'type'; ty = namedType(globals, tok); ti() 2368 | end 2369 | elseif yes or tt.tag ~= 'Type' or tt._def then 2370 | p = 'type'; ty = tt; ti() -- beware redefinition of inferred types 2371 | end 2372 | end 2373 | if not p then 2374 | break 2375 | end 2376 | if p == 'size' and ltok == 'long' and nn[p] == 'long' then 2377 | nn[p] = 'long long' 2378 | elseif p == 'attr' then 2379 | -- nothing 2380 | elseif p=='type' and nn[p] then 2381 | xerror(options,n,"conflicting types '%s' and '%s'", nn[p], ltok) 2382 | elseif nn[p] then 2383 | xerror(options,n,"conflicting type specifiers '%s' and '%s'", nn[p], ltok) 2384 | else 2385 | nn[p] = ltok 2386 | end 2387 | end 2388 | -- resolve multi-keyword type names 2389 | if not nn.type then 2390 | if nn.size or nn.sign then 2391 | nn.type = 'int' 2392 | elseif nn.complex then 2393 | xwarning(options, n, "_Complex used without a type, assuming 'double'") 2394 | nn.type = 'double' 2395 | elseif nn.sclass then 2396 | xwarning(options, n, "missing type specifier defaults to 'int'") 2397 | nn.type = 'int' 2398 | else 2399 | xerror(options, n, "missing type specifier") 2400 | end 2401 | end 2402 | if nn.type == 'char' then 2403 | if nn.sign then 2404 | nn.type=nn.sign..' '..nn.type nn.sign=nil end 2405 | elseif nn.type == 'int' then 2406 | if nn.size then 2407 | nn.type=nn.size..' '..nn.type nn.size=nil end 2408 | if nn.sign then 2409 | nn.type=nn.sign..' '..nn.type nn.sign=nil end 2410 | elseif nn.type == 'double' then 2411 | if nn.size and nn.size:find('long') then 2412 | nn.type=nn.size..' '..nn.type nn.size=nil end 2413 | if nn.complex then 2414 | nn.type=nn.complex..' '..nn.type nn.complex=nil end 2415 | elseif nn.type == 'float' then 2416 | if nn.complex then 2417 | nn.type='_Complex '..nn.type nn.complex=nil end 2418 | elseif type(nn.type)=='string' and nn.type:find('^__int%d+$') then 2419 | if nn.sign then 2420 | nn.type=nn.sign..' '..nn.type nn.sign=nil end 2421 | end 2422 | if nn.atomic then 2423 | nn.type='_Atomic '..nn.type nn.atomic = nil -- could be narrower 2424 | end 2425 | local msg = "qualifier '%s' cannot be applied to type '%s'" 2426 | xassert(not nn.sign, options, n, msg, nn.sign, nn.type) 2427 | xassert(not nn.size, options, n, msg, nn.size, nn.type) 2428 | xassert(not nn.complex, options, n, msg, nn.complex, nn.type) 2429 | xassert(not nn.atomic, options, n, msg, nn.atomic, nn.type) 2430 | -- signal meaningless register storage classes 2431 | local sclass = nn.sclass 2432 | local smsg = "storage class '%s' is not appropriate in this context" 2433 | if context == 'global' then 2434 | xassert(sclass~='register' and sclass~='auto', 2435 | options, n, smsg, sclass) 2436 | elseif context == 'param' then 2437 | xassert(sclass~='static' and sclass~='extern' and sclass~='typedef', 2438 | options, n, smsg, sclass) 2439 | end 2440 | -- return 2441 | if not ty then ty = namedType(globals, nn.type) end 2442 | if nn.const then ty = addQualifier(ty, 'const') end 2443 | if nn.volatile then ty = addQualifier(ty, 'volatile') end 2444 | xassert(not nn.restrict, options, n, 2445 | "qualifier '%s' is not adequate here", nn.restrict) 2446 | return ty, nn 2447 | end 2448 | 2449 | -- This function parse the right parts and returns the identifier 2450 | -- name, its type, and a storage class. Its arguments are the 2451 | -- outputs of the corresponding plus 2452 | -- the same arguments as . 2453 | parseDeclarator = function(ty, extra, symtable, context, abstract) 2454 | -- because of the curious syntax of c types, it turns out that 2455 | -- it is easier to construct the chain of types in reverse 2456 | local attr = collectAttributes() 2457 | local where = n 2458 | local name 2459 | local function parseRev() 2460 | local ty = nil 2461 | if isName(tok) then 2462 | xassert(not name, options, n, "extraneous identifier '%s'", tok) 2463 | name = tok 2464 | ti() 2465 | elseif tok == '*' or tok == '^' or tok == '&' then --pointer 2466 | local block = tok == '^' or nil -- code blocks (apple) 2467 | local ref = tok == '&' or nil -- reference type 2468 | ti() 2469 | local nt, pt 2470 | while tok=='const' or tok=='volatile' 2471 | or specifierTable[tok]=='restrict' or isAttribute(tok) do 2472 | nt = nt or Qualified{} 2473 | if isAttribute(tok) then 2474 | nt.attr = collectAttributes(nt.attr) 2475 | else 2476 | nt[specifierTable[tok]] = true 2477 | ti() 2478 | end 2479 | end 2480 | pt = parseRev() 2481 | if nt then nt.t = pt; pt = nt; end 2482 | ty = Pointer{t=pt, block=block, ref=ref} 2483 | elseif tok == '(' then 2484 | ti() 2485 | local p = specifierTable[tok] or isTypeName(tok) or isAttribute(tok) or tok == ')' 2486 | if abstract and p then 2487 | ty = parsePrototype(ty,symtable,context,abstract) 2488 | else 2489 | attr = collectAttributes(attr) 2490 | ty = parseRev() 2491 | check(')') ti() 2492 | end 2493 | elseif tok ~= '[' then 2494 | return ty 2495 | end 2496 | attr = collectAttributes(attr) 2497 | while tok == '(' or tok == '[' and ti(1) ~= '[' do 2498 | if tok == '(' then ti() 2499 | ty = parsePrototype(ty,symtable,context,abstract) 2500 | check(")") ti() 2501 | ty.attr = collectAttributes(ty.attr) 2502 | elseif tok == '[' then -- array 2503 | ti() 2504 | xassert(ty==nil or ty.tag ~= 'Function', options,n, 2505 | "functions cannot return arrays (they can return pointers)") 2506 | local nt = nil 2507 | while specifierTable[tok] =='restrict' or tok == 'const' 2508 | or tok == 'volatile' or options.dialect99 and tok == 'static' do 2509 | xassert(ty==nil or ty.tag~='Array', options, n, 2510 | "only the outer array indices can contain qualifiers") 2511 | xassert(tok~='static' or context=='param', options, n, 2512 | "static array qualifiers are only permitted in prototypes") 2513 | nt = nt or Qualified{} 2514 | nt[specifierTable[tok]] = true 2515 | ti() 2516 | end 2517 | if tok == ']' then 2518 | xassert(ty==nil or ty.tag~='Array', options, n, 2519 | "only the outer array can be specified without a size") 2520 | ty = Array{t=ty} 2521 | ti() 2522 | else 2523 | local size = skipTo({},']',',',';') 2524 | local v = tryEvaluateConstantExpression(options, n, size, symtable) 2525 | xassert(v, options, n, 2526 | "syntax error in array size specification") 2527 | xassert(type(v)~='number' or v>=0, options, n, 2528 | "invalid array size '%s'", v) 2529 | check(']') ti() 2530 | ty = Array{t=ty, size=v} 2531 | end 2532 | if nt then 2533 | if nt.sclass then 2534 | xassert(ty.size, options, n, "static in this context needs an array size") 2535 | nt.static = ty.size 2536 | nt.sclass = nil 2537 | end 2538 | nt.t = ty.t 2539 | ty.t = nt 2540 | end 2541 | end 2542 | end 2543 | return ty 2544 | end 2545 | -- get reversed type and reverse it back 2546 | local rty = parseRev() 2547 | while rty do 2548 | local nty = rty.t 2549 | rty.t = ty 2550 | ty = rty 2551 | rty = nty 2552 | -- syntax checks 2553 | if ty.tag == 'Pointer' and ty.block then 2554 | xassert(ty.t and ty.t.tag == 'Function', options, where, 2555 | "invalid use of code block operator '^'") 2556 | end 2557 | end 2558 | attr = collectAttributes(attr) 2559 | -- distribute inlines and attributes 2560 | if extra.inline then 2561 | local tt = ty 2562 | while tt and tt.tag ~= 'Function' do tt=tt.t end 2563 | xassert(tt, options, where, "only functions can be declared inline") 2564 | tt.inline = true 2565 | end 2566 | attr = tableAppend(extra.attr, attr) 2567 | if attr then 2568 | local tt = ty 2569 | while tt.tag == 'Pointer' do tt=tt.t end 2570 | if tt.tag ~= 'Function' and tt.tag ~= 'Struct' and tt.tag ~= 'Union' then tt = nil end 2571 | if tt == nil and ty == 'Qualified' then tt = ty end 2572 | if tt == nil then ty=Qualified{t=ty} tt=ty end 2573 | if tt then tt.attr = attr end 2574 | end 2575 | -- return 2576 | xassert(abstract or name, options, n, "an identifier was expected") 2577 | return name, ty, extra.sclass 2578 | end 2579 | 2580 | -- We are now ready to parse a declaration in the specified context 2581 | parseDeclaration = function(symtable, context) 2582 | -- parse declaration specifiers 2583 | local where = n 2584 | local lty, lextra = parseDeclarationSpecifiers(symtable,context,false) 2585 | -- loop over declarators 2586 | if isName(tok) or tok=='*' or tok=='&' or tok == '^' or tok=='(' or tok=='[' then 2587 | -- parse declarator 2588 | local name,ty,sclass = parseDeclarator(lty, lextra, symtable, context, false) 2589 | -- first declarator may be a function definition 2590 | if context == 'global' and name and typeIs(ty,'Function') and tok == '{' then 2591 | local body = skipPar({}) 2592 | xassert(sclass ~= 'typedef', options, where, 2593 | "storage class %s is not adequate for a function definition", sclass) 2594 | processDeclaration(where, symtable, context, name, ty, sclass, body) 2595 | return 2596 | end 2597 | -- process declarators 2598 | while true do 2599 | if typeIs(ty,'Function') then 2600 | if not where then error() end 2601 | processDeclaration(where, symtable, context, name, ty, sclass) 2602 | else 2603 | local init 2604 | if tok == '=' then 2605 | xassert(sclass ~= 'typedef', options, n, "a typedef cannot have an initializer") 2606 | ti() 2607 | init = skipTo({}, specifierTable, ';', ',') 2608 | end 2609 | processDeclaration(where, symtable, context, name, ty, sclass, init) 2610 | end 2611 | if tok ~= ',' then break else ti() end 2612 | where = n 2613 | name,ty,sclass = parseDeclarator(lty, lextra, symtable, context, false) 2614 | end 2615 | else 2616 | xassert(lextra.newtype, options, where, "empty declaration") 2617 | end 2618 | -- the end 2619 | check(';') ti() 2620 | end 2621 | 2622 | parsePrototype = function(rty,symtable,context_,abstract_) 2623 | local nsymtable = newScope(symtable) 2624 | local ty = Function{t=rty} 2625 | local i=0 2626 | while tok ~= ')' do 2627 | if tok == '...' then 2628 | i = i + 1 2629 | ty[i] = Pair{ellipsis=true} 2630 | ti() check(')') 2631 | else 2632 | local lty, lextra = parseDeclarationSpecifiers(nsymtable, 'param', true) 2633 | local pname, pty = parseDeclarator(lty, lextra, nsymtable, 'param', true) 2634 | local sty = pty.tag == 'Qualified' and pty.t or pty 2635 | if sty.tag == 'Type' and sty.n == 'void' then 2636 | xassert(i==0 and not pname and tok==')' and pty == sty, options, n, 2637 | "void in function parameters must appear first and alone") 2638 | return ty 2639 | else 2640 | if pty.tag == 'Array' then 2641 | pty = Pointer{t=pty.t} 2642 | elseif pty.tag == 'Qualified' and pty.t.tag == 'Array' then 2643 | pty.t = Pointer{t=pty.t.t} end 2644 | i = i + 1 2645 | local def 2646 | if tok == '=' then ti() def=skipTo({}, specifierTable,';',',') end 2647 | ty[i] = Pair{pty,pname,defval=def} 2648 | if tok == ',' then ti() else check(',',')') end 2649 | end 2650 | end 2651 | end 2652 | if i == 0 then ty.withoutProto = true end 2653 | return ty 2654 | end 2655 | 2656 | parseStruct = function(symtable, context, abstract_, nn) 2657 | check('struct', 'union') 2658 | local kind = tok ; ti() 2659 | nn.attr = collectAttributes(nn.attr) 2660 | local ttag, tnam 2661 | if isName(tok) then ttag=tok; tnam=kind..' '..ttag; nn.newtype=true; ti() end 2662 | nn.attr = collectAttributes(nn.attr) 2663 | if ttag and tok ~= '{' then return namedType(symtable, tnam) end 2664 | -- parse real struct definition 2665 | local ty 2666 | if kind == 'struct' then ty = Struct{n=ttag} else ty = Union{n=ttag} end 2667 | local where = n 2668 | check('{') ti() 2669 | while tok and tok ~= '}' do 2670 | where = n 2671 | local lty, lextra = parseDeclarationSpecifiers(symtable, context) 2672 | xassert(lextra.sclass == nil, options, where, 2673 | "storage class '%s' is not allowed here", lextra.sclass) 2674 | if tok == ';' then -- anonymous member 2675 | xassert(lty.tag=='Struct' or lty.tag=='Union' , 2676 | options, where, "empty declaration") 2677 | ty[1+#ty] = Pair{lty} 2678 | else 2679 | while true do 2680 | if tok == ':' then ti() -- unnamed bitfield 2681 | local size = skipTo({},',',';') 2682 | local v = tryEvaluateConstantExpression(options, where, size, symtable) 2683 | xassert(v, options, where, 2684 | "syntax error in bitfield specification") 2685 | xassert(type(v)~='number' or v>=0, options, where, 2686 | "invalid anonymous bitfield size (%s)", v) 2687 | ty[1+#ty] = Pair{lty,bitfield=v} 2688 | else 2689 | local pname, pty = parseDeclarator(lty, lextra, symtable, context) 2690 | if pty.tag == 'Array' and not pty.size then 2691 | xwarning(options, where, "unsized arrays are not allowed here (ignoring)") 2692 | elseif pty.tag == 'Function' then 2693 | xerror(options, where, "member functions are not allowed in C") 2694 | end 2695 | if tok == ':' then ti() 2696 | xassert(lty == pty, options, where, "bitfields must be of integral types") 2697 | local size = skipTo({},',',';') 2698 | local v = tryEvaluateConstantExpression(options,where,size,symtable) 2699 | xassert(v, options, where, 2700 | "syntax error in bitfield specification") 2701 | xassert(type(v)~='number' or v>0, options, where, 2702 | "invalid bitfield size (%s)", v) 2703 | ty[1+#ty] = Pair{pty,pname,bitfield=v} 2704 | else 2705 | ty[1+#ty] = Pair{pty,pname} 2706 | end 2707 | end 2708 | check(',',';') 2709 | if tok == ',' then ti() else break end 2710 | end 2711 | end 2712 | check(';','}') 2713 | if tok == ';' then ti() end 2714 | end 2715 | check("}") ti() 2716 | ty.attr = collectAttributes(nn.attr) 2717 | nn.attr = nil 2718 | -- name anonymous structs or enums (avoiding anonymous unions) 2719 | if not ttag and tok ~= ';' and hasOption(options,"-Ztag") then 2720 | ttag = unique() 2721 | tnam = kind .. " " .. ttag 2722 | ty.n = ttag 2723 | end 2724 | -- change tagged type as newtype 2725 | if ttag then 2726 | nn.newtype = true 2727 | processDeclaration(where, symtable, context, tnam, ty, '[typetag]') 2728 | return namedType(symtable, tnam) 2729 | else 2730 | return ty 2731 | end 2732 | end 2733 | 2734 | parseEnum = function(symtable, context, abstract_, nn) 2735 | local kind = tok ; ti() 2736 | nn.attr = collectAttributes(nn.attr) 2737 | local ttag, tnam 2738 | if isName(tok) then ttag=tok; tnam=kind..' '..ttag; nn.newtype=true; ti() end 2739 | nn.attr = collectAttributes(nn.attr) 2740 | if ttag and tok ~= '{' then return namedType(symtable, tnam) end 2741 | -- parse real struct definition 2742 | local i = 1 2743 | local v,a = 0,0 2744 | local ty = Enum{n=ttag} 2745 | local ity = Qualified{t=namedType(globals, "int"),const=true,_enum=ty} 2746 | local where = n 2747 | check('{') ti() 2748 | repeat 2749 | local nam = tok 2750 | local init 2751 | xassert(isName(nam),options,n,"identifier expected, got '%s'", tok) 2752 | collectAttributes(nil) -- parsed but lost for now 2753 | if ti() == '=' then ti() 2754 | init = skipTo({},',','}') 2755 | v = tryEvaluateConstantExpression(options, n, init, symtable) 2756 | xassert(v,options,n,"invalid value for enum constant") 2757 | a = 0 2758 | end 2759 | local x 2760 | if type(v) == 'number' then 2761 | x = v + a 2762 | elseif a > 0 then 2763 | x = string.format("%d+(%s)",a,v) 2764 | else 2765 | x = v 2766 | end 2767 | ty[i] = Pair{nam, init and v} 2768 | a = a + 1 2769 | i = i + 1 2770 | processDeclaration(n, symtable, context, nam, ity, '[enum]', x) 2771 | if tok == ',' then ti() else check(',','}') end 2772 | until tok == nil or tok == '}' 2773 | check('}') ti() 2774 | ty.attr = collectAttributes(nn.attr) 2775 | nn.attr = nil 2776 | -- name anonymous structs or enums 2777 | if not ttag and hasOption(options,"-Ztag") then 2778 | ttag = unique() 2779 | tnam = kind .. " " .. ttag 2780 | ty.n = ttag 2781 | end 2782 | -- change tagged type as newtype 2783 | nn.newtype = true 2784 | if ttag then 2785 | processDeclaration(where, symtable, context, tnam, ty, '[typetag]') 2786 | return namedType(symtable, tnam) 2787 | else 2788 | return ty 2789 | end 2790 | end 2791 | 2792 | -- When macros[1] is a table, the preprocessor attempts to 2793 | -- preprocess and evaluate the definition of object-like macros. If 2794 | -- the evaluation is successful, it adds it to the table. 2795 | local function processMacroCaptures() 2796 | local macros = options.macros 2797 | local captable = macros and macros[1] 2798 | if type(captable) == 'table' then 2799 | for _,v in ipairs(captable) do coroutine.yield(CppEvent(v)) end 2800 | macros[1] = {} 2801 | end 2802 | end 2803 | 2804 | -- main 2805 | if options.stringToType then 2806 | -- this is used to implement stringToType 2807 | local lty, lextra = parseDeclarationSpecifiers(globals, 'stringToType', true) 2808 | local pname, pty, psclass = parseDeclarator(lty, lextra, globals, 'stringToType', true) 2809 | while tok==';' do ti() end 2810 | xassert(not psclass, options, n, "storage class '%s' is not adequate in this context", psclass) 2811 | xassert(not tok, options, n, "garbage after type declaration") 2812 | return pty, pname 2813 | else 2814 | -- main loop 2815 | while tok do 2816 | while tok == ';' do ti() end 2817 | processMacroCaptures() 2818 | parseDeclaration(globals,"global") 2819 | processMacroCaptures() 2820 | end 2821 | return globals 2822 | end 2823 | end 2824 | 2825 | 2826 | 2827 | 2828 | -- converts a string into a type and possibly a variable name 2829 | 2830 | local function stringToType(s) 2831 | local options = { silent=true, stringToType=true } 2832 | local src = "<" .. s .. ">" 2833 | local ss,t,n = pcall(parseDeclarations, options, {}, 2834 | filterSpaces, tokenizeLine, s, src, true) 2835 | if not ss then return nil end 2836 | while t and t._def do t = t._def end 2837 | return t, n 2838 | end 2839 | 2840 | 2841 | 2842 | -- processes the typedef options <-Ttypename> 2843 | -- and create the initial symbol table. 2844 | 2845 | local function initialSymbols(options) 2846 | local symbols = {} 2847 | for _,v in ipairs(options) do 2848 | if v:find("^%-T") then 2849 | local d = v:gsub("^%-T%s*(.-)%s*$") 2850 | xassert(d and d:find("[A-Za-z_$][A-Za-z0-9_$]*"), 2851 | options,"", 2852 | "option -T must be followed by a valid identifier") 2853 | symbols[d] = TypeDef{n=d} 2854 | end 2855 | end 2856 | return symbols 2857 | end 2858 | 2859 | 2860 | -- this function return an iterator function that 2861 | -- successively returns actions as tagged tables 2862 | -- with tags TypeDef, VarDef, FuncDef, or Declaration. 2863 | 2864 | local function declarationIterator(options, lines, prefix) 2865 | options = copyOptions(options) 2866 | prefix = prefix or "" 2867 | local symbols = initialSymbols(options) 2868 | local macros = initialMacros(options) 2869 | assert(type(options)=='table') 2870 | assert(type(lines)=='function') 2871 | assert(type(prefix)=='string') 2872 | assert(type(symbols)=='table') 2873 | assert(type(macros)=='table') 2874 | options.macros = macros 2875 | options.symbols = symbols 2876 | local di = wrap(options, 2877 | parseDeclarations, symbols, 2878 | filterSpaces, 2879 | expandMacros, macros, 2880 | tokenize, 2881 | processDirectives, macros, 2882 | eliminateComments, 2883 | joinLines, 2884 | yieldLines, lines, prefix) 2885 | return di, symbols, macros 2886 | end 2887 | 2888 | 2889 | local function parse(filename, outputfile, options) 2890 | -- handle optional arguments 2891 | options = options or {} 2892 | outputfile = outputfile or "-" 2893 | assert(type(filename)=='string') 2894 | assert(type(options)=='table') 2895 | local closeoutputfile = false 2896 | if io.type(outputfile) ~= 'file' then 2897 | assert(type(outputfile) == 'string') 2898 | if outputfile == '-' then 2899 | outputfile = io.output() 2900 | else 2901 | closeoutputfile = true 2902 | outputfile = io.open(outputfile,"w") 2903 | end 2904 | end 2905 | assert(io.type(outputfile) == 'file') 2906 | -- go 2907 | local li = declarationIterator(options, io.lines(filename), filename) 2908 | outputfile:write("+--------------------------\n") 2909 | for action in li do 2910 | local s = declToString(action) 2911 | outputfile:write(string.format("| %s\n", tostring(action))) 2912 | if s then outputfile:write(string.format("| %s\n", s)) end 2913 | outputfile:write("+--------------------------\n") 2914 | end 2915 | if closeoutputfile then 2916 | outputfile:close() 2917 | end 2918 | end 2919 | 2920 | 2921 | --------------------------------------------------- 2922 | --------------------------------------------------- 2923 | --------------------------------------------------- 2924 | -- EXPORTS 2925 | 2926 | cparser = {} 2927 | 2928 | cparser.cpp = cpp 2929 | cparser.cppTokenIterator = cppTokenIterator 2930 | cparser.macroToString = macroToString 2931 | 2932 | cparser.parse = parse 2933 | cparser.declarationIterator = declarationIterator 2934 | cparser.typeToString = typeToString 2935 | cparser.stringToType = stringToType 2936 | cparser.declToString = declToString 2937 | 2938 | return cparser 2939 | -------------------------------------------------------------------------------- /lcdecl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env lua 2 | -- Copyright (c) Facebook, Inc. and its affiliates. 3 | -- This source code is licensed under the MIT license found in the 4 | -- LICENSE file in the root directory of this source tree. 5 | -- 6 | -- Front-end for the Lua C declaration parser 7 | 8 | local function smartrequire(s) 9 | local savedpath=package.path 10 | local thisfile=debug.getinfo(2).source 11 | if type(thisfile)=='string' and thisfile:find("^@") then 12 | local thisdir=thisfile:gsub("^@",""):gsub("[^/]*$","") 13 | package.path=thisdir.."?.lua;"..package.path 14 | end 15 | local r = require(s) 16 | package.path = savedpath 17 | return r 18 | end 19 | 20 | cparser = smartrequire 'cparser' 21 | io = require 'io' 22 | 23 | local function die(s,...) 24 | if s then print(string.format(s,...)) end 25 | error(nil) 26 | end 27 | 28 | local function usage() 29 | die([[ 30 | Usage: ldecl [options] inputfile.c [-o outputfile.txt] 31 | Main options: 32 | -Idirname : Add dirname to the include search path 33 | -I- : Marks beginning of system include search path 34 | -Dsym[=value] : Define a macro symbol 35 | -Usym : Undefine a macro symbol 36 | -Tsym : Declare that symbol is a type 37 | -Zcppdef : Extracts initial macro definitions from cpp 38 | -std=xxx : Select C dialect (c89,c99,c11,gnu89,gnu99,gnu11) 39 | ]]) 40 | end 41 | 42 | local options={} 43 | local outputfile 44 | local inputfile 45 | 46 | local outputi 47 | for i,v in ipairs{...} do 48 | if outputi == i then 49 | outputfile = v 50 | elseif v == '-o' and not outputfile then 51 | outputi = i+1 52 | elseif v:find("^-") then 53 | options[1+#options] = v 54 | elseif not inputfile then 55 | inputfile = v 56 | else 57 | usage() 58 | end 59 | end 60 | 61 | local function exists(fname) 62 | local fd = io.open(fname,"r") 63 | if fd then fd:close() return true end 64 | return false 65 | end 66 | 67 | if not inputfile then 68 | usage() 69 | elseif not exists(inputfile) then 70 | die("cparser: cannot read file '%s'", inputfile) 71 | end 72 | 73 | local outputfd 74 | if outputfile then 75 | outputfd = io.open(outputfile,"w") 76 | if not outputfd then 77 | die("cparser: cannot open '%s' for writing", outputfile) 78 | end 79 | end 80 | 81 | local s,m = pcall(cparser.parse, inputfile, outputfd, options) 82 | if io.type(outputfd)=='file' then outputfd:close() end 83 | if not s then die( not m:find("aborted") and m ) end 84 | -------------------------------------------------------------------------------- /lcpp: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env lua 2 | -- Copyright (c) Facebook, Inc. and its affiliates. 3 | -- This source code is licensed under the MIT license found in the 4 | -- LICENSE file in the root directory of this source tree. 5 | -- 6 | -- Front-end for the Lua C preprocessor. 7 | 8 | 9 | local function smartrequire(s) 10 | local savedpath=package.path 11 | local thisfile=debug.getinfo(2).source 12 | if type(thisfile)=='string' and thisfile:find("^@") then 13 | local thisdir=thisfile:gsub("^@",""):gsub("[^/]*$","") 14 | package.path=thisdir.."?.lua;"..package.path 15 | end 16 | local r = require(s) 17 | package.path = savedpath 18 | return r 19 | end 20 | 21 | cparser = smartrequire 'cparser' 22 | io = require 'io' 23 | 24 | local function die(s,...) 25 | if s then print(string.format(s,...)) end 26 | error(nil) 27 | end 28 | 29 | local function usage() 30 | die([[ 31 | Usage: lcpp [options] inputfile.c [-o outputfile.c] 32 | Main options: 33 | -Idirname : Add dirname to the include search path 34 | -I- : Marks beginning of system include search path 35 | -Dsym[=value] : Define a macro symbol. 36 | -Usym : Undefine a macro symbol. 37 | -Zcppdef : Extracts initial macro definitions from cpp. 38 | -Znopass : Do not copy unrecognized preprocessor directives. 39 | -dM : Dump final macro definitions. 40 | ]]) 41 | end 42 | 43 | local options={} 44 | local outputfile 45 | local inputfile 46 | 47 | 48 | local outputi 49 | for i,v in ipairs{...} do 50 | if outputi == i then 51 | outputfile = v 52 | elseif v == '-o' and not outputfile then 53 | outputi = i+1 54 | elseif v:find("^-") then 55 | options[1+#options] = v 56 | elseif not inputfile then 57 | inputfile = v 58 | else 59 | usage() 60 | end 61 | end 62 | 63 | local function exists(fname) 64 | local fd = io.open(fname,"r") 65 | if fd then fd:close() return true end 66 | return false 67 | end 68 | 69 | if not inputfile then 70 | usage() 71 | elseif not exists(inputfile) then 72 | die("cparser: cannot read file '%s'", inputfile) 73 | end 74 | 75 | local outputfd 76 | if outputfile then 77 | outputfd = io.open(outputfile,"w") 78 | if not outputfd then 79 | die("cparser: cannot open '%s' for writing", outputfile) 80 | end 81 | end 82 | 83 | local s,m = pcall(cparser.cpp, inputfile, outputfd, options) 84 | if io.type(outputfd)=='file' then outputfd:close() end 85 | if not s then die( not m:find("aborted") and m ) end 86 | 87 | 88 | -------------------------------------------------------------------------------- /tests/README: -------------------------------------------------------------------------------- 1 | TODO: make a good testing framework instead of random scripts 2 | -------------------------------------------------------------------------------- /tests/spec/cparser_spec.lua: -------------------------------------------------------------------------------- 1 | -- Copyright (c) Facebook, Inc. and its affiliates. 2 | -- This source code is licensed under the MIT license found in the 3 | -- LICENSE file in the root directory of this source tree. 4 | 5 | -- Note: This test case was originally contributed by 6 | -- Javier in git commit c6cf0b6fa31 7 | -- and licensed under the facebook CLA. */ 8 | 9 | 10 | describe('cparser', function() 11 | local cparser = require('cparser') 12 | 13 | local function get_enum_action(actions) 14 | local enum = {} 15 | 16 | for action in actions do 17 | enum = action 18 | end 19 | 20 | return enum 21 | end 22 | 23 | it('should be able to parse an enum', function() 24 | local actions = cparser.declarationIterator({}, io.lines('tests/tstenum.h'), 'tests/tstenum.h') 25 | 26 | local actual = get_enum_action(actions) 27 | 28 | local expected = { 29 | tag = 'TypeDef', 30 | name = 'SomeEnumeration', 31 | where = 'tests/tstenum.h:1', 32 | sclass = 'typedef', 33 | type = { 34 | tag = 'Enum', 35 | { tag = 'Pair', 'ValueOne' }, 36 | { tag = 'Pair', 'ValueTwo', 1 }, 37 | { tag = 'Pair', 'ValueThree' }, 38 | { tag = 'Pair', 'ValueFour', 0 } 39 | } 40 | } 41 | 42 | assert.are.same(expected, actual) 43 | end) 44 | end) 45 | -------------------------------------------------------------------------------- /tests/testbit.lua: -------------------------------------------------------------------------------- 1 | -- Copyright (c) Facebook, Inc. and its affiliates. 2 | -- This source code is licensed under the MIT license found in the 3 | -- LICENSE file in the root directory of this source tree. 4 | 5 | if false then -- set to true and use lua53 to get the correct output 6 | local bit = {} 7 | function bit.bnot(a) return ~a end 8 | function bit.bor(a,b) return a | b end 9 | function bit.band(a,b) return a & b end 10 | function bit.bxor(a,b) return a ~ b end 11 | function bit.lshift(a,b) return a < 0 and b < 0 and ~((~a) << b) or a << b end 12 | return bit 13 | else 14 | local function bor(a,b) 15 | local r, c, d = 0, 1, -1 16 | while a > 0 or b > 0 or a < -1 or b < -1 do 17 | if a % 2 > 0 or b % 2 > 0 then r = r + c end 18 | a, b, c, d = math.floor(a / 2), math.floor(b / 2), c * 2, d * 2 end 19 | if a < 0 or b < 0 then r = r + d end 20 | return r end 21 | bit = {} 22 | function bit.bnot(a) return -1-a end 23 | function bit.bor(a,b) return bor(a,b) end 24 | function bit.band(a,b) return -1-bor(-1-a,-1-b) end 25 | function bit.bxor(a,b) return bor(-1-bor(a,-1-b),-1-bor(-1-a,b)) end 26 | function bit.lshift(a,b) return math.floor(a * 2 ^ b) end 27 | end 28 | 29 | -- cparser = require 'cparser' 30 | 31 | local args = {} 32 | 33 | local s = 314 34 | for i=1,10000 do 35 | s = (s * 6900069) % 8989889 36 | z = (s % 2) > 0 and 1 or -1 37 | args[1+#args] = z * math.floor(s / 2) 38 | end 39 | 40 | local function test(s,f) 41 | for i=1,#args do 42 | local a = args[i] 43 | local b = args[1+(i+3)%(#args)] 44 | print(s,a,b,f(a,b)) 45 | end 46 | end 47 | 48 | test("bitnot",bit.bnot) 49 | test("bitand",bit.band) 50 | test("bitor",bit.bor) 51 | test("bitxor",bit.bxor) 52 | test("bitshift",function(a,b) 53 | local s = b % 2 54 | local bb = (math.floor(b/2) % 33) * (-1 ^ s) 55 | return bit.lshift(a,bb) end) 56 | 57 | -------------------------------------------------------------------------------- /tests/testcpp.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) Facebook, Inc. and its affiliates. 3 | # This source code is licensed under the MIT license found in the 4 | # LICENSE file in the root directory of this source tree. 5 | # 6 | # This script compares the outputs of the clang preprocessor 7 | # and the cparser preprocessor. Other than white-space 8 | # differences, we expect parity on MacOSX 10.10. 9 | # Note that we have to prevent clang from including 10 | # the compiler specific include files because 11 | # they contain weird constructs such as #include_next 12 | # and friends. I do not want to implement that. 13 | 14 | 15 | tmp=tmp$$- 16 | 17 | trap "rm 2>/dev/null ${tmp}*" 0 18 | 19 | usage() { 20 | echo 2>&1 "Usage: cppcomp.sh -I... -D... filename.c" 21 | } 22 | 23 | args= 24 | file= 25 | defs='"-Znopass"' 26 | for n in "$@" 27 | do 28 | case "$n" in 29 | -Z*) 30 | defs="$defs,\"$n\"" 31 | ;; 32 | -*) 33 | args="$args $n" 34 | defs="$defs,\"$n\"" 35 | ;; 36 | *) 37 | test -n "$file" && usage && exit 10 38 | file=$n 39 | esac 40 | done 41 | test -z "$file" && usage && exit 10 42 | 43 | defs="$defs,"'"-I.","-I-","-I/usr/include","-Zcppdef"' 44 | defs="$defs,"'"-D__has_builtin(x)=1","-D__has_feature(x)=1"' 45 | defs="$defs,"'"-D__has_attribute(x)=1","-D__has_extension(x)=1"' 46 | 47 | clang -E -nostdinc $args -I. -I/usr/include $file > ${tmp}-clang.c 48 | lua -lcparser -e 'cparser.cpp("'$file'", "'${tmp}-lua.c'", {'$defs'})' 49 | grep -v '^[[:space:]]*#' ${tmp}-clang.c | grep -v '^[[:space:]]*$' > ${tmp}-clang2.c 50 | grep -v '^[[:space:]]*#' ${tmp}-lua.c | grep -v '^[[:space:]]*$' > ${tmp}-lua2.c 51 | wc ${tmp}-* 52 | 53 | diff -u -b -B ${tmp}-clang2.c ${tmp}-lua2.c 54 | -------------------------------------------------------------------------------- /tests/testmacro.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | #include 7 | #include 8 | 9 | #defmacro DEFINE_SVV(TNAME, TYPE, OPNAME, OPTEXT) 10 | 11 | #define OPER(a,b) OPTEXT 12 | 13 | TYPE TNAME##Vector_##OPNAME(TYPE *a, TYPE *b, int n) 14 | { 15 | /* try cblas */ 16 | #if #TYPE == "float" && #OPNAME == "dot" 17 | return cblas_sdot(n, a, 1, b, 1); 18 | #elif #TYPE == "double" && #OPNAME == "dot" 19 | return cblas_ddot(n, a, 1, b, 1); 20 | #else 21 | int i; 22 | TYPE s = 0; 23 | # pragma unroll(i) 24 | for(i=0;i 0 8 | n, notseq(n-1) 9 | # else 10 | 0 11 | # endif 12 | #endmacro 13 | 14 | 15 | 16 | #defrecursivemacro sequence(n) 17 | # if n > 0 18 | n, sequence(n-1) 19 | # else 20 | 0 21 | # endif 22 | #endmacro 23 | 24 | #defrecursivemacro infinite(x) 25 | infinite(x) 26 | #endmacro 27 | 28 | int a[] = { notseq(10) }; 29 | 30 | int a[] = { sequence(10) }; 31 | 32 | #ifdef TESTLOOP 33 | infinite(0); 34 | #endif 35 | -------------------------------------------------------------------------------- /tests/testvarmacro.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | #define F_MACRO1(arg1, arg2, ...) \ 7 | do_f(arg1, arg2, __VA_ARGS__) 8 | #define F_MACRO2(arg1, arg2, args...) \ 9 | do_f(arg1, arg2, args) 10 | #define F_MACRO3(arg1, arg2, args...) \ 11 | do_f(arg1, arg2, ##args) 12 | 13 | 14 | extern void do_f(int,...); 15 | 16 | int main(int argc, char **argv) 17 | { 18 | F_MACRO1(argc,argv[0],argv[1],argv[2],argv[3]); 19 | F_MACRO1(argc,argv[0]); 20 | F_MACRO2(argc,argv[0],argv[1],argv[2],argv[3]); 21 | F_MACRO2(argc,argv[0]); 22 | F_MACRO3(argc,argv[0],argv[1],argv[2],argv[3]); 23 | F_MACRO3(argc,argv[0]); 24 | } 25 | -------------------------------------------------------------------------------- /tests/tst1.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define hash_hash # ## # 10 | #define mkstr(a) # a 11 | #define in_between(a) mkstr(a) 12 | #define join(c, d) in_between(c hash_hash d) 13 | char p[] = join(x, y); // equivalent to 14 | // char p[] = "x ## y"; 15 | -------------------------------------------------------------------------------- /tests/tst2.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define x 3 10 | #define f(a) f(x * (a)) 11 | #undef x 12 | #define x 2 13 | #define g f 14 | #define z z[0] 15 | #define h g(~ 16 | #define m(a) a(w) 17 | #define w 0,1 18 | #define t(a) a 19 | #define p() int 20 | #define q(x) x 21 | #define r(x,y) x ## y 22 | #define str(x) # x 23 | 24 | f(y+1) + f(f(z)) % t(t(g)(0) + t)(1); 25 | g(x+(3,4)-w) | h 5) & m 26 | (f)^m(m); 27 | p() i[q()] = { q(1), r(2,3), r(4,), r(,5), r(,) }; 28 | char c[2][6] = { str(hello), str() }; 29 | 30 | -------------------------------------------------------------------------------- /tests/tst3.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define str(s) # s 10 | #define xstr(s) str(s) 11 | #define debug(s, t) printf("x" # s "= %d, x" # t "= %s", \ 12 | x ## s, x ## t) 13 | #define INCFILE(n) vers ## n 14 | #define glue(a, b) a ## b 15 | #define xglue(a, b) glue(a, b) 16 | #define HIGHLOW "hello" 17 | #define LOW LOW ", world" 18 | debug(1, 2); 19 | fputs(str(strncmp("abc\0d", "abc", '\4') // this goes away 20 | == 0) str(: @\n), s); 21 | #include xstr(INCFILE(2).h) 22 | glue(HIGH, LOW); 23 | xglue(HIGH, LOW) 24 | -------------------------------------------------------------------------------- /tests/tst4.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define t(x,y,z) x ## y ## z 10 | int j[] = { t(1,2,3), t(,4,5), t(6,,7), t(8,9,), 11 | t(10,,), t(,11,), t(,,12), t(,,) }; 12 | -------------------------------------------------------------------------------- /tests/tst5.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define OBJ_LIKE (1-1) 10 | #define OBJ_LIKE /* white space */ (1-1) /* other */ 11 | #define FUNC_LIKE(a) ( a ) 12 | #define FUNC_LIKE( a )( /* note the white space */ \ 13 | a /* other stuff on this line 14 | */ ) 15 | 16 | -------------------------------------------------------------------------------- /tests/tst6.c: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test file is an excerpt of the C99 standard 7 | http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf */ 8 | 9 | #define debug(...) fprintf(stderr, __VA_ARGS__) 10 | #define showlist(...) puts(#__VA_ARGS__) 11 | #define report(test, ...) ((test) ? puts(#test) : printf(__VA_ARGS__)) 12 | debug("Flag"); 13 | debug("X = %d\n", x); 14 | showlist(The first, second, and third items.); 15 | report(x>y, "x is %d but y is %d", x, y); 16 | -------------------------------------------------------------------------------- /tests/tstenum.h: -------------------------------------------------------------------------------- 1 | /* Copyright (c) Facebook, Inc. and its affiliates. 2 | * This source code is licensed under the MIT license found in the 3 | * LICENSE file in the root directory of this source tree. 4 | */ 5 | 6 | /* Note: This test case was originally contributed by 7 | Javier in git commit c6cf0b6fa31 8 | with the name tst7.c and licensed under the facebook CLA. */ 9 | 10 | typedef enum 11 | { 12 | ValueOne, 13 | ValueTwo = 1, 14 | ValueThree, 15 | ValueFour = ValueOne 16 | } SomeEnumeration; 17 | --------------------------------------------------------------------------------