| Pattern Element | Meaning |
|---|---|
| [a-fx] | Square brackets denote a character class. The pattern at left can match the letters a-f or x. Piraha has no pre-defined character classes and does not support the notions of unions or intersections insided character classes available in Java's regular expression package. |
| [^a-fx] | This is a negated character entity. It will match any character except a-f or x. |
| [^] | This pattern will match any character. |
| x{n,m} | The quantifier pattern will match the pattern x a minimum of n and a maximum of m times. |
| x{n,} | This version of the quantifier pattern will match n or more occurrences of x |
| x{,m} | This version of the quantifier pattern will match at most m occurrences of x |
| x+ | A shorthand for x{1,} |
| x* | A shorthand for x{0,} |
| x? | A shorthand for x{0,1} |
| \1 | Match the first backreference within this rule. Any of backreferences 1 to 9 may be matched by using \2, \3, etc. |
| {name} | Match the pattern named by name. Save the match in a backreference. |
| {-name} | Match the pattern named by name, but do not capture backreferences.| |
| (?=x) | Match the pattern x as a zero-width lookahead assertion |
| (?!x) | A negative zero-width lookahead assertion for pattern x |
| (x|y|z) | An ordered set of preferred ways to match. If pattern x does not succeed, try pattern y, and then z. There can be any number of alternatives in this set. |
| \b | A word boundary. A word is composed of letters, numbers, and the underscore. |
| $ | Match the end of the string. |
| ^ | Match the beginning of the string. |
| (?i:x) | Ignore case when matching x. |
| (?-i:x) | Do not ignore case when matching x.| |
| {OpenWhite} | A zero-width lookahead assertion that a new indentation level has opened, starting at the current position. |
| {CloseWhite} | A zero-width lookahead assertion that an indentation level has closed, starting at the current position. |
| {BodyWhite} | 45 | Match the current indentation level. Initially this will be zero. It will increase when {OpenWhite} successfully matches, and decrease when {CloseWhite} successfully matches. 46 | |
Piraha uses a subset of the syntax familiar from regular expressions in the java library: 31 | java.util.regex.Pattern doc
32 | 33 |This document assumes you are already familiar with writing regular expressions using the regular expression engine that comes with java 1.4+.
34 | 35 |The Piraha API is similar, but instead of simply compiling patterns with a static method, each pattern is named and is compiled into a grammar.
36 | 37 |
38 | import edu.lsu.cct.piraha.*;
39 |
40 | public class Test {
41 | public static void main(String[] args) {
42 | // Instantiate a grammar
43 | Grammar g = new Grammar();
44 |
45 | // compile a pattern and name it HELLO
46 | g.compile("HELLO","(?i:hello world)");
47 |
48 | // get a matcher for the pattern named HELLO
49 | Matcher m = g.matcher("HELLO","Hello World!");
50 |
51 | // Look for a match!
52 | if(m.matches()) {
53 | System.out.println(m.group());
54 | }
55 |
56 | }
57 | }
58 |
59 |
60 | Basic differences are as follows: 61 |
"(a*)a" regardless of what string of characters you supply for text:
63 |
64 | import edu.lsu.cct.piraha.*;
65 |
66 | public class Test2 {
67 | public static void main(String[] args) {
68 | Grammar g = new Grammar();
69 | g.compile("alist","(a*)a");
70 | String text = " ... any text ...";
71 | Matcher m = g.matcher("alist",text);
72 | if(m.matches()) {
73 | // can't happen
74 | System.out.println(m.group());
75 | }
76 | }
77 | }
78 |
79 | All groups are independent non-capturing groups. What does this mean? It means that Piraha will fail when it gets a pattern and text like the one below. The reason is that the first version of the pattern "aaa" will match the first three characters of "aaaa", and that will leave only one "a" unmatched. Neither the sub-pattern "aaa" nor "aa" can match a single "a". However, the string "aaaaa" will succeed and the whole pattern will match.
82 |
83 | import edu.lsu.cct.piraha.*;
84 |
85 | public class Test3 {
86 | public static void main(String[] args) {
87 | Grammar g = new Grammar();
88 | g.compile("alist","(aaa|aa)$");
89 | Matcher m = g.matcher("alist","aaaa");
90 | if(m.matches()) {
91 | System.out.println(m.group());
92 | }
93 | }
94 | }
95 |
96 | The Pattern element {name} references a pattern element by name, and a pattern can reference itself recursively. This means that it's easy to matched balanced parenthesis in Piraha. In this example, we match a balanced angle bracket.
99 |
100 | import edu.lsu.cct.piraha.*;
101 |
102 | public class Test4 {
103 | public static void main(String[] args) {
104 | Grammar g = new Grammar();
105 | g.compile("d","<{d}>|[a-z]+");
106 | Matcher m = g.matcher("d","<<bar>>extra");
107 | if(m.find()) {
108 | System.out.println(m.group()); // prints <<bar>>
109 | }
110 | }
111 | }
112 | The basic pattern elements were documented in the Reference Card, and the basic 29 | API was documented in the Quick Start. These documents show you how to compile 30 | and use Piraha pattern expressions one by one. However, if you are describing a complex grammar, it is often 31 | more convenient to use a Grammar File.
32 |A Grammar File consists of pattern definitions of the form "name=value," where the name is a c-identifier 33 | (i.e. a sequence of letters, numbers, or the underscore), and value is a pattern or pegular expression. 34 |
It is not necessary to write Java code to make use of a Grammar File. Piraha comes with a generic grammar 48 | compiler. You may invoke it as follows:
49 |50 | $ java -cp piraha.jar edu.lsu.cct.piraha.examples.Generic peg-file src-file1 src-file2 ... 51 |52 |
The result of running the command will be a series of files with the suffix "pegout." They will provide 53 | a parse tree for your grammar in outline form. If you would prefer to see xml, you can ask for that. 54 | In the example below, we provide eqn.peg as an input file:
55 |
56 | skipper = [ \t\n]*
57 | num = [0-9]+
58 | mulop = [*/]
59 | mul = {num}( {mulop} {num})*
60 | addop = [+-]
61 | add = {mul}( {addop} {mul})*
62 | math = ( {add} )$
63 |
64 | We also provide eqn.in as an example source for the input file. 65 |
66 | 10+9-2*3-1 67 |68 |
We now run the command with --xml:
69 |70 | $ java -cp piraha.jar edu.lsu.cct.piraha.examples.Generic --xml eqn.peg eqn.in 71 | reading file: eqn.in 72 | writing file: eqn.xml 73 | SUCCESS: files-checked=[1] grammar=[eqn.peg] 74 |75 |
And obtain the output:
76 |77 | <math start='0' end='11' line='1'> 78 | <add start='0' end='10' line='1'> 79 | <mul start='0' end='2' line='1'> 80 | <num start='0' end='2' line='1'>10</num> 81 | </mul> 82 | <addop start='2' end='3' line='1'>+</addop> 83 | <mul start='3' end='4' line='1'> 84 | <num start='3' end='4' line='1'>9</num> 85 | </mul> 86 | <addop start='4' end='5' line='1'>-</addop> 87 | <mul start='5' end='8' line='1'> 88 | <num start='5' end='6' line='1'>2</num> 89 | <mulop start='6' end='7' line='1'>*</mulop> 90 | <num start='7' end='8' line='1'>3</num> 91 | </mul> 92 | <addop start='8' end='9' line='1'>-</addop> 93 | <mul start='9' end='10' line='1'> 94 | <num start='9' end='10' line='1'>1</num> 95 | </mul> 96 | </add> 97 | <text>10+9-2*3-1 </text> 98 | </math> 99 |100 |
Note that pattern names become xml node names, and each node has 101 | a start position in the text, and end position, a line number, 102 | and either child nodes, or the text matched by the node. 103 | In addition, the full input text is captured in the text node 104 | at the end.
105 |Other flags to Generic include --perl, which generates Perl5 data 106 | structures as output, or --python, which generates Python data 107 | structures.
108 | 109 |If you want to compile a grammar file within Java, you can do it 110 | like this: 111 |
112 | import edu.lsu.cct.piraha.*;
113 | import java.io.*;
114 |
115 | public class Gram {
116 | public static void main(String[] args) throws Exception {
117 | Grammar g = new Grammar();
118 | g.compileFile(new File("eqn.peg"));
119 | String contents = Grammar.readContents(new File("eqn.in"));
120 | Matcher m = g.matcher(contents);
121 | if(m.match(0)) { // test for match at 0
122 | m.dumpMatchesXML(); // Write matches to the screen in XML
123 | }
124 | }
125 | }
126 |
127 |
128 |
129 |
--------------------------------------------------------------------------------
/src/edu/lsu/cct/piraha/examples/Test.java:
--------------------------------------------------------------------------------
1 | package edu.lsu.cct.piraha.examples;
2 |
3 | import edu.lsu.cct.piraha.DebugOutput;
4 | import edu.lsu.cct.piraha.DebugVisitor;
5 | import edu.lsu.cct.piraha.Grammar;
6 | import edu.lsu.cct.piraha.Matcher;
7 | import edu.lsu.cct.piraha.ParseException;
8 | import edu.lsu.cct.piraha.Pattern;
9 |
10 | /**
11 | * Just a handful of basic tests
12 | * @author sbrandt
13 | *
14 | */
15 | public class Test {
16 |
17 |
18 | public static void test(String pat, String text, int matchsize) {
19 | Grammar g = new Grammar();
20 | Pattern p = g.compile("d", pat);//"a(0|1|2|3|4|5|6a|7|8|9)*x");
21 | String pdc = p.decompile();
22 | Pattern pe = null;
23 | DebugVisitor dv = new DebugVisitor();
24 | try {
25 | pe = g.compile("e",pdc);
26 | } catch (ParseException e) {
27 | p.visit(dv);
28 | DebugOutput.out.println("pat="+pat);
29 | DebugOutput.out.println("pdc="+pdc);
30 | throw new RuntimeException("parser decompile error "+pat+" != "+pdc,e);
31 | }
32 | if(!pe.equals(p)) {
33 | p.visit(dv);
34 | pe.visit(dv);
35 | throw new RuntimeException("decompile error "+pat+" != "+pe.decompile());
36 | }
37 | Matcher m = g.matcher(text);//"a6ax");
38 | //m.getPattern().diag();
39 | m.getPattern().visit(new DebugVisitor(DebugOutput.out));
40 | boolean found = m.matches();
41 | DebugOutput.out.print("match "+text+" using "+pat+" => "+found+" ");
42 | if(found) {
43 | DebugOutput.out.print("["+m.getBegin()+","+m.getEnd()+"]");
44 | m.dumpMatchesXML();
45 | }
46 | DebugOutput.out.println();
47 | DebugOutput.out.println();
48 | if(found) assert(matchsize == m.getEnd() - m.getBegin());
49 | else assert(matchsize == -1);
50 | }
51 |
52 | static boolean assertOn = false;
53 |
54 | static public boolean turnAssertOn() {
55 | assertOn = true;
56 | return true;
57 | }
58 |
59 | public static void main(String[] args) throws Exception {
60 | assert(turnAssertOn());
61 | if(!assertOn)
62 | throw new RuntimeException("Assertions are not enabled");
63 |
64 | test("(\\[[^\\]]*)+","[",1);
65 | test("a(0|1|2|3|4|5|6a|7|8|9)*x","a6a72x",6);
66 | test("a(0|1|2|3|4|5|6a|7|8|9)*x","a6a72",-1);
67 | test("<{d}>|x","<When building any grammar in Piraha, you start with patterns. We write a pattern for a number as "-?[0-9]+". This will match one or more digits, optionally preceded by a minus sign. We'll name this pattern num.
35 | Grammar math = new Grammar();
36 | math.compile("num","-?[0-9]+");
37 |
38 |
39 | We'll also need patterns to match the basic mathematical operations: add and subtract.
40 | 41 |
42 | math.compile("addop", "\\+|-");
43 |
44 |
45 | So far we have only written simple patterns. While they are named and part of an object called a grammar, they aren't really a grammar yet. The next step is 46 | to start combining these patterns together. We 47 | could write
48 | 49 |
50 | math.compile("addexpr", "{num}({addop}{num})*");
51 |
52 |
53 | This pattern would match 3 as well as 3+4 as well as 3+4-2 because the group (i.e. the parens) matches a + or - followed by a number, and this group can match zero or more times. When matching 3+4-2 the 3 matches the first instance of {num} the +4 matches the group, and -2 matches the group again.
This will allow us to parse a simple sequence of additions and subtractions.
56 | 57 |Let's take a look at how we'd use that code. 58 |
59 | Matcher m = math.matcher("expr", "3+4-2");
60 | boolean b = m.matches();
61 | if (b) {
62 | m.dumpMatches();
63 | System.out.println("eval=" + evalExpr(m));
64 | } else {
65 | System.out.println(m.near()); // report errors
66 | }
67 |
68 |
69 | To apply our patterns to actual text, we create a matcher with both the pattern name and the text we want to match. We then ask the Matcher if the pattern matches, and if it does we print the parse tree in a human readable form. If the parse fails, we print a diagnostic showing where the pattern matcher had trouble.
70 | 71 |In this example, the printed parse tree looks like this: 72 |
73 | [0] addexpr: 74 | [0] num=(3) 75 | [1] addop=(+) 76 | [2] num=(4) 77 | [3] addop=(-) 78 | [4] num=(2) 79 |80 | 81 |
The topmost node is addexpr, it has child nodes that are either num or addop. An equals sign shows us the value each node has.
82 | 83 |Evaluating this tree is simple. There is only a single class you need to understand in order to make use of it: Group. The top 5 methods in the listing below are the main ones you need to make sense of any parse tree.
84 | 85 |
86 | public class Group {
87 | // position in text where the match began
88 | public int getBegin();
89 | // position in text where the match ended
90 | public int getEnd();
91 | // number of child groups
92 | public int groupCount();
93 | // the nth child group
94 | public Group group(int n);
95 | // the name of the pattern that matched this text
96 | public String getPatternName();
97 |
98 | // the full input string
99 | public String getText();
100 | // the substring matched by this pattern
101 | public String toString();
102 | // show the parse tree
103 | public void dumpMatches();
104 | // show the parse tree in XML form
105 | public void dumpMatchesXML();
106 |
107 | }
108 |
109 |
110 | The Matcher is a subclass of group. This is a convenience that allows you to access the matches more easily.
111 | 112 |For the case of our calculator, we can use this Group object to compute the sum of all the addends. 113 |
114 | private static double evalExpr(Group match) {
115 | double answer = Double.parseDouble(match.group(0).toString());
116 | for(int i=1;i<match.groupCount();i+=2) {
117 | String op = match.group(i).toString();
118 | double addend = Double.parseDouble(match.group(i+1).toString());
119 | if("+".equals(op))
120 | answer += addend;
121 | else
122 | answer -= addend;
123 | }
124 | return answer;
125 | }
126 |
127 |
128 | To capture a calculator we want the multiplies and divides to work as well, and we'll want to recognize the order of operations (multiply and divide before add and subtract).
131 | 132 |To enable this feature, we'll replace {num} above with {mulexp}.
135 | math.compile("addexpr", "{mulexp}({addop}{mulexp})*");
136 | math.compile("mulexp", "{num}({mulop}{num})*");
137 |
138 |
139 | Now when we examine our parse tree, we'll find that {addexp} nodes have {mulexp} nodes underneath them instead of simple numbers.
142 | package edu.lsu.cct.piraha.examples;
143 |
144 | import edu.lsu.cct.piraha.Grammar;
145 | import edu.lsu.cct.piraha.Group;
146 | import edu.lsu.cct.piraha.Matcher;
147 |
148 | public class Calc {
149 | public static Grammar makeMath() {
150 | Grammar math = new Grammar();
151 | // All the various was a double precision number (or integer) can be
152 | // written
153 | math.compile("num","-?[0-9]+(\.[0-9]+)?");
154 |
155 | // The basic operators
156 | math.compile("addop", "\\+|-");
157 | math.compile("mulop", "\\*|/");
158 | math.compile("powop", "\\*\\*");
159 |
160 | // unary negation
161 | math.compile("neg", "-");
162 |
163 | // All the different ways we can stick things together, includes
164 | // parenthesis.
165 | // Note: The start of "expr" should not be {expr}
166 | // because that leads to infinite recursion.
167 | math.compile("expr", "{mulexp}({addop}{mulexp})*");
168 | math.compile("mulexp", "{powexp}({mulop}{powexp})*");
169 | math.compile("powexp", "{num}({powop}{num})*|\\({expr}\\)");
170 |
171 | return math;
172 | }
173 |
174 | Grammar math = makeMath();
175 |
176 | public double eval(String s) {
177 | Matcher m = math.matcher("expr",s.trim());
178 | if(m.matches())
179 | return evalExpr(m);
180 | else
181 | return Double.NaN;
182 | }
183 |
184 | public static void main(String[] args) {
185 | Grammar math = makeMath();
186 |
187 | Matcher m = math.matcher("expr", "1+2*4+4**3**2"); // answer is 262153
188 |
189 | boolean b = m.matches();
190 | System.out.println("match? " + b);
191 | if (b) {
192 | m.dumpMatchesXML();
193 | System.out.println("node count=" + count(m));
194 | System.out.println("eval=" + evalExpr(m));
195 | }
196 | }
197 |
198 | private static int count(Group m) {
199 | int n = 1;
200 | for (int i = 0; i < m.groupCount(); i++) {
201 | n += count(m.group(i));
202 | }
203 | return n;
204 | }
205 |
206 | private static double evalExpr(Group match) {
207 | String pn = match.getPatternName();
208 | if ("num".equals(pn)) {
209 | return Double.parseDouble(match.substring());
210 | } else if ("expr".equals(pn)) {
211 | double d = evalExpr(match.group(0));
212 | for(int i=1;i+1<match.groupCount();i+=2) {
213 | String op = match.group(i).substring();
214 | if("+".equals(op))
215 | d += evalExpr(match.group(i+1));
216 | else
217 | d -= evalExpr(match.group(i+1));
218 | }
219 | return d;
220 | } else if ("mulexp".equals(pn)) {
221 | double d = evalExpr(match.group(0));
222 | for(int i=1;i+1<match.groupCount();i+=2) {
223 | String op = match.group(i).substring();
224 | if("*".equals(op))
225 | d *= evalExpr(match.group(i+1));
226 | else
227 | d /= evalExpr(match.group(i+1));
228 | }
229 | return d;
230 | } else if ("powexp".equals(pn)) {
231 | int n = match.groupCount();
232 | double d = evalExpr(match.group(n-1));
233 | for(int i=n-2;i>0;i-=2) {
234 | d = Math.pow(evalExpr(match.group(i-1)), d);
235 | }
236 | return d;
237 | }
238 | return evalExpr(match.group(0));
239 | }
240 |
241 | }
242 |
243 |
244 |
245 |
246 |
--------------------------------------------------------------------------------
/src/edu/lsu/cct/piraha/Group.java:
--------------------------------------------------------------------------------
1 | package edu.lsu.cct.piraha;
2 |
3 | import java.io.PrintWriter;
4 | import java.util.LinkedList;
5 | import java.util.Stack;
6 |
7 | public class Group implements Cloneable {
8 | String patternName, text;
9 | int begin, end;
10 | Group replacement;
11 |
12 | public void setReplacement(Group replacement) {
13 | if(replacement.getPatternName().equals(patternName))
14 | this.replacement = replacement;
15 | else
16 | throw new MatchException("replacement group does not have patternName='"+patternName+"'");
17 | }
18 |
19 | public int getBegin() {
20 | if(replacement != null)
21 | return replacement.getBegin();
22 | return begin;
23 | }
24 | public int getEnd() {
25 | if(replacement != null)
26 | replacement.getEnd();
27 | return end;
28 | }
29 | LinkedListOne of the nice things at parsing expression grammars is how expressive they are, and how easy it is to build fairly complex results in a small space.
32 | 33 |The first step in creating our language will be to create the "skipper". The skipper is a pattern which matches things like white space and comments. A typical C-like skipper
34 | 35 |36 | skipper = \b([ \t\r\n]|//[^\n]*|/\*(\*[^/]|[^*])*\*/)* 37 |38 | 39 |
Let's consider what we have here. The "\b" pattern means we want this pattern to include a word boundary. Word boundaries are things like transitions between letters and symbols.
40 | 41 |The next types of things in the skipper are white space, [ \t\r\n]; a comment that goes until the end of the line, //[^\n]*; or a multi-line comment /\*(\*[^/]|[^*])*\*/. This last consists of the literal start and end sequences, /\* and \*/. In between it matches a sequence which consists of characters that either match [^*] or \*[^/].
Now we consider various types of basic pattern elements. The double quote:
44 | 45 |46 | dquote = "(\\.|[^"\\])*" 47 |48 | 49 |
The integer, with optional leading minus sign:
50 | 51 |52 | int = -?[0-9]+ 53 |54 | 55 |
A name (which is really a C-Identifier):
56 | 57 |58 | name = [a-zA-Z_][a-zA-Z0-9_]* 59 |60 | 61 |
Next we'll define a mathematical or logical expression, basing it loosely on the C-language precedence of operators. This is similar to what we already did with the calculator, except we'll include comparison and logical operators.
62 | 63 |mulop = *|/|% 66 | addop = +|- 67 | ltgt = <=?|>=? 68 | eq = [!=]= 69 | andor = &&|\|\|70 | 71 |
e1 = {fcall}|{index}|{name}|{int}|{dquote}|( {expr} )
74 | e2 = {e1}( {mulop} {e1})*
75 | e3 = {e2}( {addop} {e2})*
76 | e4 = {e3}( {ltgt} {e3})*
77 | e5 = {e4}( {eq} {e4})*
78 | expr = {e5}( {andor} {e5})*
79 |
80 | index = {name} [ {expr} ]
81 |
82 | fcall = {name} \( ({expr}( , {expr})*|) \)
83 |
84 |
85 | The index defines syntax for indexing an array.
86 | 87 |Wherever a blank space occurs in any pattern in the peg, the skipper is implicit. Thus, I could have written the index definition as follows:
88 | 89 |
90 | index = {name}{skipper}\[{skipper}{expr}{skipper}\]
91 |
92 |
93 | But it would be much harder to read.
94 | 95 |Note we've also included a function call definition (fcall) which consists of a name followed by parenthesis containing 0 or more arguments delimited by commas.
At this point we begin the meat of the language. This pattern, quite similar to our function call pattern above, matches a function definition:
98 | 99 |
100 | func = {name} \( ({name}( , {name})*|) \) {block}
101 |
102 |
103 | Everything here, except the definition of block is defined in terms of things we already know. Nothing in the peg syntax requires us to build our pattern definitions in order of use. We're attempting to build a minimal, but reasonably complete language. For this reason we'll define our block to consist of one of 5 types of statements. Control flow, including an if (with optional else), and a while. We'll also want the ability to assign to local variables, to call functions (e.g. printf("Hello, world"); and the ability to return from functions.
All of this can be expressed in a straightforward way in just a few lines.
106 | 107 |
108 | block = { ({statement} )*}
109 |
110 | statement = {if}|{while}|{return}|{assign}|{call}|{block}
111 |
112 | assign = ({index}|{name}) = {expr} ;
113 | call = {name} ( ({expr}( , {expr})*|) ) ;
114 | if = if ( {expr} ) {block} (else {block})?
115 | while = while ( {expr} ) {block}
116 | return = return {expr} \;
117 |
118 |
119 | Finally, we complete our peg by defining the program pattern. It matches any number of variable assignments and function definitions.
122 | program = ^ ({assign} |{func} )*$
123 |
124 |
125 |
128 | skipper = \b([ \t\r\n]|//[^\n]|/*(*[^/]|[^])*/)
129 |
130 | dquote = "(\.|[^"\])*"
131 |
132 | int = -?[0-9]+
133 |
134 | name = [a-zA-Z_][a-zA-Z0-9_]*
135 |
136 | func = {name} ( ({name}( , {name})*|) ) {block}
137 |
138 | block = { ({statement} )*}
139 |
140 | statement = {if}|{while}|{return}|{assign}|{call}|{block}
141 |
142 | index = {name} [ {expr} ]
143 |
144 | assign = ({index}|{name}) = {expr} ;
145 | call = {name} ( ({expr}( , {expr})*|) ) ;
146 | if = if ( {expr} ) {block} (else {block})?
147 | while = while ( {expr} ) {block}
148 | return = return {expr} \;
149 |
150 | mulop = *|/|% 153 | addop = +|- 154 | ltgt = <=?|>=? 155 | eq = [!=]= 156 | andor = &&|\|\|157 | 158 |
e1 = {fcall}|{index}|{name}|{int}|{dquote}|( {expr} )
161 | e2 = {e1}( {mulop} {e1})*
162 | e3 = {e2}( {addop} {e2})*
163 | e4 = {e3}( {ltgt} {e3})*
164 | e5 = {e4}( {eq} {e4})*
165 | expr = {e5}( {andor} {e5})*
166 |
167 | fcall = {name} ( ({expr}( , {expr})*|) )
168 |
169 | program = ^ ({assign} |{func} )*$
170 |
171 |
172 | The first step is to read in the grammar we've defined above
177 | and compile it. We use the compileFile() member function on
178 | the Grammar class to accomplish this:
180 | Grammar g = new Grammar();
181 |
182 | static String[] cmdArgs;
183 | public Cexample(String[] args) throws IOException {
184 | cmdArgs = args;
185 | g.compileFile(new File("Cexample.peg"));
186 | }
187 |
188 |
189 | Next, we provide a method within our code to compile files
190 | in our new language and report syntax errors. The near()
191 | function does the latter task.
195 |
196 | public void compile(File fname) throws IOException {
197 | String s = Grammar.readContents(fname);
198 | Matcher m = g.matcher("program",s);
199 | if(m.matches()) {
200 | program = m.group();
201 | } else {
202 | System.out.println(m.near());
203 | }
204 | }
205 |
206 |
207 | Our main method will do both of these things for us. It will also store our command
210 | public static void main(String[] args) throws Exception {
211 | Cexample c = new Cexample(args);
212 | c.compile(new File(args[0]));
213 | c.exec();
214 | }
215 |
216 |
217 | The bulk of the work is implementing the interpreter logic. We'll create a VarMap class to map variable names to values. Each time we invoke a function we'll create a new one of these. The VarMap has a prev field to point to definitions of variables that live farther down in the call stack.
Once our exec(Group,VarMap) function completes, we'll know the definitions of all functions and can locate and invoke main.
222 | class VarMap {
223 | VarMap prev;
224 | Map vars = new HashMap();
225 | }
226 |
227 | public void exec() {
228 | exec(program);
229 | }
230 |
231 | public void exec(Group g) {
232 | VarMap root = new VarMap();
233 | exec(g,root);
234 | if(funcMap.containsKey("main")) {
235 | List<Object> args = new ArrayList<Object>();
236 | for(int i=0;i<cmdArgs.length;i++)
237 | args.add(cmdArgs[i]);
238 | List<Object> arg = new ArrayList<Object>();
239 | arg.add(args);
240 | fcall("main",arg,root);
241 | }
242 | }
243 |
244 |
245 | The next thing to look at is the exec(Group,VarMap) function itself.
Two types of patterns are important for the exec function to understand the program. It must understand assignment statements for root or global variables, and it must understand function definitions.
248 | 249 |The "func" definition, if found, will contain a "name" as its first (0th) child. This is the name of the function, and the key we'll use to look up the function in our function map. The call to g.group(0).substring() retrieves this name from the parse tree.
Assignments to indexes are a little more complex. If the first child of the assignment statement is an index, it's an element of an array we are trying to assign. If not, it's a name. This latter case is simpler. We retrieve the name with g.group(0).substring(), and the expression being assigned with g.group(1). This latter needs to be evaluated before being assigned to the variables table, and eval(g.group(1),vm) performs this task.
For indexes, we have to look up the name of the array being indexed, the List<Object> used to represent the index, evaluate the index, the value, and finally set the element.
256 | Map funcMap = new HashMap();
257 |
258 | void exec(Group g,VarMap vm) {
259 | if("assign".equals(g.getPatternName())) {
260 | if(g.group(0).getPatternName().equals("index")) {
261 | List<Object> li = (List<Object>)eval(g.group(0).group(0),vm);
262 | Integer index = (Integer)eval(g.group(0).group(1),vm);
263 | Object value = eval(g.group(1),vm);
264 | li.set(index.intValue(),value);
265 | } else {
266 | vm.vars.put(g.group(0).substring(),eval(g.group(1),vm));
267 | }
268 | } else if("func".equals(g.getPatternName())) {
269 | funcMap.put(g.group(0).substring(),g);
270 | } else if( ....
271 | ....
272 |
273 |
274 | The eval() function is going to convert our mathematical expressions and values into objects. Let's take a brief look at that.
If eval sees an "int" or a "dquote", it simply stores the value into an appropriate form and returns it. For an int, that involves calling the Integer constructor. For a string, the initial and trailing quotes in g.substring() have to be removed, and the basic escape sequences processed ('\n' maps to line feed, '\r' to carriage return, etc.).
If we find an expression, then we have to process logical operators. If we have a single child element, we just evaluate it. If we have more, there will be an odd number, as the pattern matches values joined by logical "and" or "or" operators. We process these left to right and store each new sub result in val. This same basic logic will be used to construct all the different kinds of operators.
279 | 280 |
281 | Object eval(Group g,VarMap vm) {
282 | if("int".equals(g.getPatternName())) {
283 | return new Integer(g.substring());
284 | } else if("dquote".equals(g.getPatternName())) {
285 | String s = g.substring();
286 | return remap(s);
287 | } else if("expr".equals(g.getPatternName())) {
288 | Object val = eval(g.group(0),vm);
289 | for(int i=1;i<g.groupCount();i+=2) {
290 | String op = g.group(i).substring();
291 | Object val2 = eval(g.group(i+1),vm);
292 | boolean b1 = mkBoolean(val);
293 | boolean b2 = mkBoolean(val2);
294 | if("&&".equals(op))
295 | val = (b1 & b1) ? 1 : 0;
296 | else
297 | val = (b1 | b2) ? 1 : 0;
298 | }
299 | return val;
300 | ...
301 |
302 |
303 | Function calls are handled by the fcall() function. It takes the function name, the value of all arguments (evaluated by eval() and finally the current variable map.
Variable names are looked up by starting with the current variable map and checking for the variable name. If found, the value is returned. If not, the prev pointer is checked for additional definitions.
308 | ...
309 | } else if("fcall".equals(g.getPatternName())) {
310 | List<Object> args = new ArrayList<Object>();
311 | for(int i=1;i<g.groupCount();i++) {
312 | args.add(eval(g.group(i),vm));
313 | }
314 | return fcall(g.group(0).substring(),args,vm);
315 | } else if("name".equals(g.getPatternName())) {
316 | VarMap x = vm;
317 | String nm = g.substring();
318 | while(x != null) {
319 | if(vm.vars.containsKey(nm))
320 | return vm.vars.get(nm);
321 | else if(vm.prev == null)
322 | throw new Error("Undefined variable: '"+nm+"'");
323 | vm = vm.prev;
324 | }
325 | return null;
326 | ....
327 |
328 |
329 | Evaluating function calls is our next step. The fcall method
330 | starts by providing logic for "builtin" functions such as printf(). If we don't have a builtin function, we create a new VarMap, populate it with variable names.
The variable names are found inside the function definition. The first child of the "func" definition is the function name, the last is the function body. All the children in between are argument names. We fetch each of these in turn, assign values to the variables in the new VarMap and finally invoke the function body with our exec(Group,VarMap) function.
335 | Object fcall(String fn,List<Object> args,VarMap vm) {
336 | try {
337 | if("printf".equals(fn)) {
338 | Object[] pars = new Object[args.size()-1];
339 | for(int i=0;i<pars.length;i++)
340 | pars[i] = args.get(i+1);
341 | System.out.printf(args.get(0).toString(),pars);
342 | return 1;
343 | } else if("len".equals(fn)) {
344 | ....
345 | }
346 | VarMap local = new VarMap();
347 | local.prev = vm;
348 | Group func = funcMap.get(fn);
349 | if(func == null)
350 | throw new Error("unknown function "+fn);
351 | for(int i=1;i+1<func.groupCount();i++) {
352 | local.vars.put(func.group(i).substring(),args.get(i-1));
353 | }
354 | exec(func.group(func.groupCount()-1),local);
355 | } catch(ReturnException re) {
356 | return re.o;
357 | }
358 | return null;
359 | }
360 |
361 |
362 | Hopefully, this explains enough of the code that you can understand the complete code listing below. Here is a sample program written in our C-Script language.
363 | 364 |
365 | fib(n) {
366 | if(n < 2) {
367 | return n;
368 | }
369 | return fib(n-1)+fib(n-2);
370 | }
371 |
372 | main(args) {
373 | n = int(args[1]);
374 | printf("fib(%d)=%d\n",n,fib(n));
375 | }
376 |
377 |
378 | It is invoked as follows:
379 | 380 |
381 | $ java -cp piraha.peg:. Cexample fib.cs 15
382 | fib(15)=610
383 |
Possibly a more interesting example is this sort program:
386 | 387 |
388 | // Quicksort variant
389 | sort2(arr,start,end) {
390 | if(start >= end) {
391 | return 1;
392 | }
393 | lo = start;
394 | hi = end;
395 |
396 | // find the pivot
397 | r = rand();
398 | if(r < 0) {
399 | r = 0-r;
400 | }
401 | pivot = r % (hi - lo + 1) + lo;
402 |
403 | pv = arr[pivot];
404 | while(lo < hi) {
405 | lov = arr[lo];
406 | hiv = arr[hi];
407 | if(lov > hiv) {
408 | arr[lo]=hiv;
409 | arr[hi]=lov;
410 | } else {
411 | if(lov <= pv) {
412 | lo = lo + 1;
413 | } else {
414 | hi = hi - 1;
415 | }
416 | }
417 | }
418 | if(lo == end) {
419 | lo = lo - 1;
420 | }
421 | sort2(arr,start,lo);
422 | sort2(arr,lo+1,end);
423 |
424 | }
425 |
426 | sort(arr) {
427 | n = len(arr);
428 | sort2(arr,0,n-1);
429 | }
430 |
431 | main() {
432 | arr = mkarray();
433 | // make a random array
434 | i = 0;
435 | while(i < 100) {
436 | append(arr,rand() % 1000);
437 | i = i + 1;
438 | }
439 | sort(arr);
440 | printf("%s\n",arr);
441 | }
442 |
443 |
444 |
447 | import edu.lsu.cct.piraha.*;
448 | import java.io.*;
449 | import java.util.*;
450 |
451 | public class Cexample {
452 | static class ReturnException extends RuntimeException {
453 | Object o;
454 | ReturnException(Object o) {
455 | this.o = o;
456 | }
457 | }
458 |
459 | Grammar g = new Grammar();
460 |
461 | static String[] cmdArgs;
462 | public Cexample(String[] args) throws IOException {
463 | cmdArgs = args;
464 | g.compileFile(new File("Cexample.peg"));
465 | }
466 |
467 | Group program;
468 |
469 | public void compile(File fname) throws IOException {
470 | String s = Grammar.readContents(fname);
471 | Matcher m = g.matcher("program",s);
472 | if(m.matches()) {
473 | program = m.group();
474 | } else {
475 | System.out.println(m.near());
476 | }
477 | }
478 |
479 | public static void main(String[] args) throws Exception {
480 | Cexample c = new Cexample(args);
481 | c.compile(new File(args[0]));
482 | c.exec();
483 | }
484 |
485 | class VarMap {
486 | VarMap prev;
487 | Map<String,Object> vars = new HashMap<String,Object>();
488 | }
489 |
490 | public void exec() {
491 | exec(program);
492 | }
493 |
494 | public void exec(Group g) {
495 | VarMap root = new VarMap();
496 | exec(g,root);
497 | if(funcMap.containsKey("main")) {
498 | List<Object> args = new ArrayList<Object>();
499 | for(int i=0;i<cmdArgs.length;i++)
500 | args.add(cmdArgs[i]);
501 | List<Object> arg = new ArrayList<Object>();
502 | arg.add(args);
503 | fcall("main",arg,root);
504 | }
505 | }
506 |
507 | Map<String,Group> funcMap = new HashMap<String,Group>();
508 |
509 | void exec(Group g,VarMap vm) {
510 | if("assign".equals(g.getPatternName())) {
511 | if(g.group(0).getPatternName().equals("index")) {
512 | List<Object> li = (List<Object>)eval(g.group(0).group(0),vm);
513 | Integer index = (Integer)eval(g.group(0).group(1),vm);
514 | Object value = eval(g.group(1),vm);
515 | li.set(index.intValue(),value);
516 | } else {
517 | vm.vars.put(g.group(0).substring(),eval(g.group(1),vm));
518 | }
519 | } else if("func".equals(g.getPatternName())) {
520 | funcMap.put(g.group(0).substring(),g);
521 | } else if("program".equals(g.getPatternName())) {
522 | for(int i=0;i<g.groupCount();i++)
523 | exec(g.group(i),vm);
524 | } else if("block".equals(g.getPatternName())) {
525 | for(int i=0;i<g.groupCount();i++)
526 | exec(g.group(i),vm);
527 | } else if("statement".equals(g.getPatternName())) {
528 | for(int i=0;i<g.groupCount();i++)
529 | exec(g.group(i),vm);
530 | } else if("return".equals(g.getPatternName())) {
531 | //vm.vars.put("$ret",eval(g.group(0),vm));
532 | throw new ReturnException(eval(g.group(0),vm));
533 | } else if("while".equals(g.getPatternName())) {
534 | while(true) {
535 | Object o = eval(g.group(0),vm);
536 | boolean b = mkBoolean(o);
537 | if(b)
538 | exec(g.group(1),vm);
539 | else
540 | break;
541 | }
542 | } else if("if".equals(g.getPatternName())) {
543 | Object o = eval(g.group(0),vm);
544 | boolean b = mkBoolean(o);
545 | if(b)
546 | exec(g.group(1),vm);
547 | else if(g.groupCount() > 2)
548 | exec(g.group(2),vm);
549 | } else if("call".equals(g.getPatternName())) {
550 | List<Object> args = new ArrayList<Object>();
551 | for(int i=1;i<g.groupCount();i++) {
552 | args.add(eval(g.group(i),vm));
553 | }
554 | fcall(g.group(0).substring(),args,vm);
555 | } else {
556 | throw new Error(g.getPatternName());
557 | }
558 | }
559 |
560 | boolean mkBoolean(Object o) {
561 | if(o instanceof String) {
562 | String s = (String)o;
563 | if(s.length()==0)
564 | return false;
565 | } else if(o instanceof Integer) {
566 | Integer i = (Integer)o;
567 | if(i.intValue()==0)
568 | return false;
569 | }
570 | return true;
571 | }
572 |
573 | int cmp(Object o1,Object o2) {
574 | if(o1 instanceof Integer && o2 instanceof Integer) {
575 | Integer i1 = (Integer)o1;
576 | Integer i2 = (Integer)o2;
577 | return i1.compareTo(i2);
578 | } else if(o1 instanceof String && o2 instanceof String) {
579 | String s1 = (String)o1;
580 | String s2 = (String)o2;
581 | return s1.compareTo(s2);
582 | } else {
583 | throw new Error("cannot compare "+o1+" and "+o2);
584 | }
585 | }
586 |
587 | Object add(Object o1,Object o2) {
588 | if(o1 instanceof Integer && o2 instanceof Integer) {
589 | Integer i1 = (Integer)o1;
590 | Integer i2 = (Integer)o2;
591 | return i1 + i2;
592 | } else if(o1 instanceof String && o2 instanceof String) {
593 | String s1 = (String)o1;
594 | String s2 = (String)o2;
595 | return s1 + s2;
596 | } else {
597 | throw new Error("cannot add "+o1+" and "+o2);
598 | }
599 | }
600 |
601 | Object mul(Object o1,Object o2) {
602 | if(o1 instanceof Integer && o2 instanceof Integer) {
603 | Integer i1 = (Integer)o1;
604 | Integer i2 = (Integer)o2;
605 | return i1 * i2;
606 | } else if(o1 instanceof String && o2 instanceof Integer) {
607 | String s1 = (String)o1;
608 | Integer i2 = (Integer)o2;
609 | StringBuffer sb = new StringBuffer();
610 | for(int i=0;i<i2;i++)
611 | sb.append(s1);
612 | return sb.toString();
613 | } else {
614 | throw new Error("cannot mul "+o1+" and "+o2);
615 | }
616 | }
617 |
618 | Object sub(Object o1,Object o2) {
619 | if(o1 instanceof Integer && o2 instanceof Integer) {
620 | Integer i1 = (Integer)o1;
621 | Integer i2 = (Integer)o2;
622 | return i1 - i2;
623 | } else {
624 | throw new Error("cannot subtract "+o1+" and "+o2);
625 | }
626 | }
627 |
628 | Object rem(Object o1,Object o2) {
629 | if(o1 instanceof Integer && o2 instanceof Integer) {
630 | Integer i1 = (Integer)o1;
631 | Integer i2 = (Integer)o2;
632 | return i1 % i2;
633 | } else {
634 | throw new Error("cannot remainder "+o1+" and "+o2);
635 | }
636 | }
637 |
638 | Object div(Object o1,Object o2) {
639 | if(o1 instanceof Integer && o2 instanceof Integer) {
640 | Integer i1 = (Integer)o1;
641 | Integer i2 = (Integer)o2;
642 | return i1 / i2;
643 | } else {
644 | throw new Error("cannot divide "+o1+" and "+o2);
645 | }
646 | }
647 |
648 | final static java.util.Random rand = new java.util.Random();
649 |
650 | Object fcall(String fn,List<Object> args,VarMap vm) {
651 | try {
652 | if("printf".equals(fn)) {
653 | Object[] pars = new Object[args.size()-1];
654 | for(int i=0;i<pars.length;i++)
655 | pars[i] = args.get(i+1);
656 | System.out.printf(args.get(0).toString(),pars);
657 | return 1;
658 | } else if("len".equals(fn)) {
659 | List<Object> li = (List<Object>)args.get(0);
660 | return li.size();
661 | } else if("append".equals(fn)) {
662 | List<Object> li = (List<Object>)args.get(0);
663 | for(int i=1;i<args.size();i++)
664 | li.add(args.get(i));
665 | return 1;
666 | } else if("mkarray".equals(fn)) {
667 | return new ArrayList<Object>();
668 | } else if("int".equals(fn)) {
669 | return new Integer((String)args.get(0));
670 | } else if("rand".equals(fn)) {
671 | return new Integer(rand.nextInt());
672 | } else if("str".equals(fn)) {
673 | return args.get(0).toString();
674 | }
675 | VarMap local = new VarMap();
676 | local.prev = vm;
677 | Group func = funcMap.get(fn);
678 | if(func == null)
679 | throw new Error("unknown function "+fn);
680 | for(int i=1;i+1<func.groupCount();i++) {
681 | local.vars.put(func.group(i).substring(),args.get(i-1));
682 | }
683 | exec(func.group(func.groupCount()-1),local);
684 | } catch(ReturnException re) {
685 | return re.o;
686 | }
687 | return null;
688 | }
689 |
690 | Object eval(Group g,VarMap vm) {
691 | if("int".equals(g.getPatternName())) {
692 | return new Integer(g.substring());
693 | } else if("dquote".equals(g.getPatternName())) {
694 | String s = g.substring();
695 | return remap(s);
696 | } else if("expr".equals(g.getPatternName())) {
697 | Object val = eval(g.group(0),vm);
698 | for(int i=1;i<g.groupCount();i+=2) {
699 | String op = g.group(i).substring();
700 | Object val2 = eval(g.group(i+1),vm);
701 | boolean b1 = mkBoolean(val);
702 | boolean b2 = mkBoolean(val2);
703 | if("&&".equals(op))
704 | val = (b1 & b1) ? 1 : 0;
705 | else
706 | val = (b1 | b2) ? 1 : 0;
707 | }
708 | return val;
709 | } else if("e5".equals(g.getPatternName())) {
710 | Object val = eval(g.group(0),vm);
711 | for(int i=1;i<g.groupCount();i+=2) {
712 | String op = g.group(i).substring();
713 | Object val2 = eval(g.group(i+1),vm);
714 | if("==".equals(op))
715 | val = cmp(val,val2)==0 ? 1 : 0;
716 | else
717 | val = cmp(val,val2)==0 ? 0 : 1;
718 | }
719 | return val;
720 | } else if("e4".equals(g.getPatternName())) {
721 | Object val = eval(g.group(0),vm);
722 | for(int i=1;i<g.groupCount();i+=2) {
723 | String op = g.group(i).substring();
724 | Object val2 = eval(g.group(i+1),vm);
725 | if("<".equals(op))
726 | val = cmp(val,val2)< 0 ? 1 : 0;
727 | else if("<=".equals(op))
728 | val = cmp(val,val2)<=0 ? 1 : 0;
729 | else if(">".equals(op))
730 | val = cmp(val,val2)> 0 ? 1 : 0;
731 | else if(">=".equals(op))
732 | val = cmp(val,val2)>=0 ? 1 : 0;
733 | }
734 | return val;
735 | } else if("e3".equals(g.getPatternName())) {
736 | Object val = eval(g.group(0),vm);
737 | for(int i=1;i<g.groupCount();i+=2) {
738 | String op = g.group(i).substring();
739 | Object val2 = eval(g.group(i+1),vm);
740 | if("+".equals(op)) {
741 | val = add(val,val2);
742 | } else {
743 | val = sub(val,val2);
744 | }
745 | }
746 | return val;
747 | } else if("e2".equals(g.getPatternName())) {
748 | Object val = eval(g.group(0),vm);
749 | for(int i=1;i<g.groupCount();i+=2) {
750 | String op = g.group(i).substring();
751 | Object val2 = eval(g.group(i+1),vm);
752 | if("*".equals(op)) {
753 | val = mul(val,val2);
754 | } else if("%".equals(op)) {
755 | val = rem(val,val2);
756 | } else {
757 | val = div(val,val2);
758 | }
759 | }
760 | return val;
761 | } else if("e1".equals(g.getPatternName())) {
762 | return eval(g.group(0),vm);
763 | } else if("fcall".equals(g.getPatternName())) {
764 | List<Object> args = new ArrayList<Object>();
765 | for(int i=1;i<g.groupCount();i++) {
766 | args.add(eval(g.group(i),vm));
767 | }
768 | return fcall(g.group(0).substring(),args,vm);
769 | } else if("name".equals(g.getPatternName())) {
770 | VarMap x = vm;
771 | String nm = g.substring();
772 | while(x != null) {
773 | if(vm.vars.containsKey(nm))
774 | return vm.vars.get(nm);
775 | else if(vm.prev == null)
776 | throw new Error("Undefined variable: '"+nm+"'");
777 | vm = vm.prev;
778 | }
779 | return null;
780 | } else if("index".equals(g.getPatternName())) {
781 | VarMap x = vm;
782 | String nm = g.group(0).substring();
783 | Integer index = (Integer)eval(g.group(1),vm);
784 | while(x != null) {
785 | if(vm.vars.containsKey(nm)) {
786 | List<Object> li = (List<Object>)vm.vars.get(nm);
787 | return li.get(index);
788 | } else if(vm.prev == null)
789 | throw new Error("Undefined variable: '"+nm+"'");
790 | vm = vm.prev;
791 | }
792 | return null;
793 | } else {
794 | throw new Error(g.getPatternName());
795 | }
796 | }
797 | String remap(String s) {
798 | StringBuffer sb = new StringBuffer();
799 | for(int i=1;i+1<s.length();i++) {
800 | char c = s.charAt(i);
801 | if(c == '\\') {
802 | char c2 = s.charAt(++i);
803 | if(c2 == 'n')
804 | sb.append('\n');
805 | else if(c2 == 'r')
806 | sb.append('\r');
807 | else if(c2 == 't')
808 | sb.append('\t');
809 | else
810 | sb.append(c2);
811 | } else {
812 | sb.append(c);
813 | }
814 | }
815 | return sb.toString();
816 | }
817 |
818 | }
819 |
820 |
821 |
822 |
823 |
--------------------------------------------------------------------------------