├── LICENSE ├── README.md ├── examples ├── arithmetic-test.jsonnet ├── arithmetic.libsonnet ├── ip-parser-test.jsonnet └── ip-parser.libsonnet ├── parser-combinators-test.jsonnet ├── parser-combinators.libsonnet ├── test-utils.libsonnet └── tests.sh /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | Copyright 2023 Stanislaw Barzowski 179 | 180 | Licensed under the Apache License, Version 2.0 (the "License"); 181 | you may not use this file except in compliance with the License. 182 | You may obtain a copy of the License at 183 | 184 | http://www.apache.org/licenses/LICENSE-2.0 185 | 186 | Unless required by applicable law or agreed to in writing, software 187 | distributed under the License is distributed on an "AS IS" BASIS, 188 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 189 | See the License for the specific language governing permissions and 190 | limitations under the License. 191 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Parser Combinators for Jsonnet 2 | 3 | It is sometimes useful to validate a piece of text or extract data from it. A common way to do this is with regular expressions. Jsonnet does not support them at the moment, though. 4 | This library implements a more powerful approach - parser combinators. 5 | 6 | Let's start with a small example - parsing IPv4 addresses, such as `192.168.1.1`. This task is doable with regexps, [but a bit tricky and the result in not very readable](https://stackoverflow.com/questions/5284147/validating-ipv4-addresses-with-regexp). 7 | With this library, you can do it like this: 8 | 9 | ``` 10 | local int255 = pc.captureWithCheck(pc.int, function(_text, val) if val >= 0 && val <= 255 then [val, null] else [null, "out of range"]); 11 | 12 | local ipV4 = pc.seq([int255, '.', int255, '.', int255, '.', int255]); 13 | 14 | pc.runParser(ipV4, "192.168.1.1") 15 | ``` 16 | 17 | A more elaborate example, which shows how to parse and evaluate arithmetic expressions (e.g. `(2 + 2) * 2 + 2`) is available in `examples/` directory. 18 | 19 | ## State of the project and future work 20 | 21 | It reached a stage when it can be useful, but it is still fairly minimal. 22 | Its primary limitation is that error reporting is still very primitive in the included parsers and parser combinators. 23 | The set of provided parsers is quite minimal, it makes sense to add more. 24 | Potentially this can even grow beyond the parser combinators to a more general text processing library. 25 | -------------------------------------------------------------------------------- /examples/arithmetic-test.jsonnet: -------------------------------------------------------------------------------- 1 | local a = import 'arithmetic.libsonnet'; 2 | local pc = import '../parser-combinators.libsonnet'; 3 | 4 | true 5 | && std.assertEqual([4, null], pc.runParser(a.expr, "4")) 6 | && std.assertEqual([4, null], pc.runParser(a.expr, "2+2")) 7 | && std.assertEqual([42, null], pc.runParser(a.expr, "40+2")) 8 | && std.assertEqual([4, null], pc.runParser(a.expr1, "2 * 2")) 9 | && std.assertEqual([4, null], pc.runParser(a.expr, "2 * 2")) 10 | && std.assertEqual([5, null], pc.runParser(a.expr, "1 + 2 * 2")) 11 | && std.assertEqual([6, null], pc.runParser(a.expr, "(1 + 2) * 2")) 12 | -------------------------------------------------------------------------------- /examples/arithmetic.libsonnet: -------------------------------------------------------------------------------- 1 | local pc = import '../parser-combinators.libsonnet'; 2 | 3 | 4 | local operator1 = pc.capture(pc.any(["*", "/"])); 5 | local operator2 = pc.capture(pc.any(["+", "-"])); 6 | 7 | local funcs = { 8 | "+":: function(x, y) x + y, 9 | "-":: function(x, y) x - y, 10 | "/":: function(x, y) x / y, 11 | "*":: function(x, y) x * y, 12 | }; 13 | 14 | local calc(exprVal) = 15 | // expects [value, null] or [value, [operator, value]] 16 | if exprVal[1] == null then 17 | exprVal[0] 18 | else 19 | funcs[exprVal[1][0]](exprVal[0], exprVal[1][1]) 20 | ; 21 | 22 | local applyCalc(p) = pc.apply(p, calc); 23 | 24 | local in_paren = pc.apply(["(", expr2, ")"], function(x) x[1]), 25 | expr0 = pc.in_whitespace(pc.any([in_paren, pc.int])), 26 | expr1 = pc.in_whitespace(applyCalc([expr0, pc.optional([operator1, expr1])])), 27 | expr2 = pc.in_whitespace(applyCalc([expr1, pc.optional([operator2, expr2])])) 28 | ; 29 | local expr = expr2; 30 | 31 | { 32 | expr:: expr, 33 | expr1:: expr1, 34 | } 35 | -------------------------------------------------------------------------------- /examples/ip-parser-test.jsonnet: -------------------------------------------------------------------------------- 1 | local ip = import 'ip-parser.libsonnet', 2 | pc = import '../parser-combinators.libsonnet'; 3 | 4 | local testutils = import '../test-utils.libsonnet'; 5 | 6 | local assertParses = testutils.assertParses, assertMismatch = testutils.assertMismatch, assertError = testutils.assertError; 7 | 8 | true 9 | && assertParses([192, null, 168, null, 1, null, 1], ip.ipV4, "192.168.1.1") 10 | && assertParses([0, null, 0, null, 0, null, 0], ip.ipV4, "0.0.0.0") 11 | && assertParses([255, null, 255, null, 255, null, 255], ip.ipV4, "255.255.255.255") 12 | && assertMismatch(ip.ipV4, "192.168..1") 13 | && assertError("out of range", ip.ipV4, "192.168.1000.1") 14 | && assertError("out of range", ip.ipV4, "192.168.-1.1") 15 | -------------------------------------------------------------------------------- /examples/ip-parser.libsonnet: -------------------------------------------------------------------------------- 1 | local pc = import '../parser-combinators.libsonnet'; 2 | 3 | local int255 = pc.captureWithCheck(pc.int, function(_text, val) if val >= 0 && val <= 255 then [val, null] else [null, "out of range"]); 4 | 5 | local ipV4 = pc.seq([int255, '.', int255, '.', int255, '.', int255]); 6 | 7 | { 8 | ipV4: ipV4, 9 | } 10 | -------------------------------------------------------------------------------- /parser-combinators-test.jsonnet: -------------------------------------------------------------------------------- 1 | local pc = import 'parser-combinators.libsonnet'; 2 | local testutils = import 'test-utils.libsonnet'; 3 | 4 | local assertParses = testutils.assertParses, assertMismatch = testutils.assertMismatch; 5 | 6 | true 7 | && assertParses(null, pc.const("foo"), "foo") 8 | && assertMismatch(pc.const("foo"), "fo") 9 | && assertParses(null, pc.const(["f", "o", "o"]), ["f", "o", "o"]) 10 | && assertParses([null, null], pc.seq([pc.const("aaa"), pc.const("bbb")]), "aaabbb") 11 | && assertParses([null, null], pc.seq(["aaa", "bbb"]), "aaabbb") 12 | && assertMismatch(pc.seq(["aaa", "bbb"]), "aaa") 13 | && assertParses(null, pc.any(["aaa", "bbb"]), "aaa") 14 | && assertParses([null, null], ["aaa", "bbb"], "aaabbb") 15 | && assertParses(null, pc.any(["aaa", "bbb"]), "bbb") 16 | && assertMismatch(pc.any(["aaa", "bbb"]), "bb") 17 | && assertParses([null, null, null, null, null, null], pc.greedy("a"), "aaaaaa") 18 | && assertParses([null, null], pc.seq([pc.ignore(pc.greedy("a")), "b"]), "aaaaaab") 19 | && assertParses(null, pc.list(pc.noop, "a", ",", "."), "a,a,a,a.") 20 | && assertMismatch(pc.list("", "a", ",", "."), "a,a,a,a,.") 21 | && assertParses(null, pc.list("[", "a", ",", "]"), "[a,a,a,a]") 22 | && assertParses("aaa", pc.capture(pc.const("aaa")), "aaa") 23 | && assertParses(["aaa", null], pc.captureWith(pc.const("aaa"), function(c, orig) [c, orig]), "aaa") 24 | && assertParses(42, pc.setValue(pc.noop, 42), "aaa") 25 | && assertParses(42, pc.apply(pc.setValue(pc.noop, 21), function(x) x * 2), "aaa") 26 | 27 | 28 | 29 | && ( 30 | // Lexer tests: 31 | local 32 | lexer = pc.lex([identifier, operator, string], [whitespace]), 33 | identifier = pc.greedy(pc.alpha, minMatches=1), 34 | whitespace = pc.greedy(" ", minMatches=1), 35 | operator = pc.any(["==", "=", "+"]), 36 | string = ['"', pc.greedy(pc.any(["=", "+", " ", pc.alpha])), '"']; 37 | true 38 | && assertParses(["a"], lexer, "a") 39 | && assertParses(["a", "=", "b"], lexer, "a=b") 40 | && assertParses(["a", "=", "b"], lexer, "a = b") 41 | && assertParses(["a", "=", '"xxx"'], lexer, 'a = "xxx"') 42 | && assertParses(["a", "=", '"x x x"'], lexer, 'a = "x x x"') 43 | ) 44 | 45 | 46 | && ( 47 | // Batteries 48 | true 49 | && assertParses(123, pc.int, "123") 50 | && assertMismatch(pc.int, "foo") 51 | 52 | ) 53 | -------------------------------------------------------------------------------- /parser-combinators.libsonnet: -------------------------------------------------------------------------------- 1 | // Parser :: State -> (Value, State) 2 | // State = [Pos, Text, Err or null] 3 | 4 | local withError(state, err) = 5 | local pos = state[0], text = state[1], oldErr = state[2]; 6 | assert oldErr == null; 7 | [null, [pos, text, err]] 8 | ; 9 | 10 | local parseConst(pattern) = function(state) 11 | local pos = state[0], text = state[1], err = state[2]; 12 | if err != null then 13 | [null, state] 14 | else 15 | local length = std.length(pattern); 16 | local retrieved = text[pos:pos+length]; 17 | if retrieved == pattern then 18 | [null, [pos + length, text, null]] 19 | else 20 | withError(state, "mismatch") 21 | ; 22 | 23 | local noop = function(state) [null, state]; 24 | 25 | local 26 | normalize(protoParser) = 27 | if std.isString(protoParser) then 28 | parseConst(protoParser) 29 | else if std.isArray(protoParser) then 30 | parseSeq(protoParser) // not sure if good idea to have that implicit 31 | else if std.isFunction(protoParser) then 32 | protoParser 33 | else 34 | error "Expected a string or a function, got " + std.type(protoParser) 35 | , 36 | parseSeq(parsers) = function(state) 37 | local length = std.length(parsers); 38 | local ps = std.map(normalize, parsers); 39 | local parsers = ps; 40 | local parseSeqH(pIndex, state, val) = 41 | local err = state[2]; 42 | if pIndex >= length then 43 | [if err != null then null else val, state] 44 | else 45 | if err != null then 46 | [null, state] 47 | else 48 | local parsed = parsers[pIndex](state); 49 | parseSeqH(pIndex + 1, parsed[1], val + [parsed[0]]) 50 | ; 51 | parseSeqH(0, state, []) 52 | ; 53 | 54 | local parseAny(parsers) = function(state) 55 | // TODO(sbarzowski) handle fatal parsing errors 56 | local length = std.length(parsers); 57 | local ps = std.map(normalize, parsers); 58 | local parsers = ps; 59 | local err = state[2]; 60 | if err != null then 61 | [null, state] 62 | else 63 | local parseAnyH(pIndex, state) = 64 | if pIndex >= length then 65 | withError(state, "mismatch") 66 | else 67 | local parsed = parsers[pIndex](state); 68 | local newState = parsed[1]; 69 | local err = newState[2]; 70 | if err != null then 71 | parseAnyH(pIndex + 1, state) 72 | else 73 | parsed 74 | ; 75 | parseAnyH(0, state) 76 | ; 77 | 78 | 79 | local capture(parser) = 80 | local p = normalize(parser); 81 | local parser = p; 82 | function(state) 83 | local startPos = state[0], text = state[1]; 84 | local parsed = parser(state); 85 | local endPos = parsed[1][0], err = parsed[1][2]; 86 | [text[startPos:endPos], parsed[1]] 87 | ; 88 | 89 | local captureWithCheck(parser, f) = 90 | local p = normalize(parser); 91 | local parser = p; 92 | function(state) 93 | local startPos = state[0], text = state[1]; 94 | local parsed = parser(state); 95 | local endPos = parsed[1][0], err = parsed[1][2]; 96 | if err == null then 97 | local res = f(text[startPos:endPos], parsed[0]); 98 | local err = res[1]; 99 | if err == null then 100 | [res[0], parsed[1]] 101 | else 102 | withError(parsed[1], res[1]) 103 | else 104 | [null, parsed[1]] 105 | ; 106 | 107 | local captureWith(parser, f) = 108 | local p = normalize(parser); 109 | local parser = p; 110 | function(state) 111 | local startPos = state[0], text = state[1]; 112 | local parsed = parser(state); 113 | local endPos = parsed[1][0], err = parsed[1][2]; 114 | if err == null then 115 | [f(text[startPos:endPos], parsed[0]), parsed[1]] 116 | else 117 | [null, parsed[1]] 118 | ; 119 | 120 | local captureTextWith(parser, f) = 121 | captureWith(parser, function(text, _parsed) f(text)); 122 | 123 | local apply(parser, f) = 124 | local p = normalize(parser); 125 | local parser = p; 126 | function(state) 127 | local parsed = parser(state); 128 | [f(parsed[0]), parsed[1]] 129 | ; 130 | 131 | local setValue(parser, val) = 132 | local p = normalize(parser); 133 | local parser = p; 134 | function(state) 135 | local parsed = parser(state); 136 | [val, parsed[1]] 137 | ; 138 | 139 | local ignore(parser) = setValue(parser, null); 140 | 141 | // TODO(sbarzowski) add support for minimum and maximum number of matches 142 | local parseGreedy(parser, minMatches=null, maxMatches=null) = function(state) 143 | local p = normalize(parser); 144 | local parser = p; 145 | local err = state[2]; 146 | if err != null then 147 | state 148 | else 149 | local parseGreedyH(state, count, vals) = 150 | local parsed = parser(state); 151 | local val = parsed[0]; 152 | local newState = parsed[1]; 153 | local err = newState[2]; 154 | if err != null then 155 | // TODO handle critical 156 | if minMatches != null && count < minMatches then 157 | [null, "not enough matches"] 158 | else 159 | [vals, state] 160 | else if maxMatches != null && count + 1 == maxMatches then 161 | [vals + [val], newState] 162 | else 163 | parseGreedyH(newState, count + 1, vals + [val]) tailstrict 164 | ; 165 | parseGreedyH(state, 0, []) 166 | ; 167 | 168 | local parseCharFiltered(filter) = function(state) 169 | local startPos = state[0], text = state[1], err = state[2]; 170 | local len = std.length(text); 171 | if err != null then 172 | state 173 | else if startPos < len && filter(text[startPos]) then 174 | [text[startPos], [startPos + 1, text, null]] 175 | else 176 | withError(state, "mismatch") 177 | ; 178 | 179 | // TODO(sbarzowski) better name 180 | local parseCharMap(obj) = function(state) 181 | local startPos = state[0], text = state[1], err = state[2]; 182 | local len = std.length(text); 183 | local c = text[startPos]; 184 | if err != null then 185 | state 186 | else if startPos < len && std.objectHas(obj, c) then 187 | [obj[c], [startPos + 1, text, null]] 188 | else 189 | withError(state, "mismatch") 190 | ; 191 | 192 | // Batteries 193 | 194 | local optional(parser) = parseAny([parser, noop]); 195 | local digit = parseCharFiltered(function(c) c >= '0' && c <= '9'); 196 | local nonZeroDigit = parseCharFiltered(function(c) c >= '1' && c <= '9'); 197 | local alphaLower = parseCharFiltered(function(c) c >= 'a' && c <= 'z'); 198 | local alphaUpper = parseCharFiltered(function(c) c >= 'A' && c <= 'Z'); 199 | local alpha = parseCharFiltered(function(c) c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'); 200 | 201 | local jsonStringChar = parseCharFiltered(function(c) c != '\\' && c != '"' && std.codepoint(c) > 31); 202 | local escapedJsonStringChar = parseSeq(["\\", parseAny(["\"", "\\", 'b', 'f', 'n', 'r', 't', 'u'])]); 203 | 204 | local int = captureTextWith( 205 | parseAny([ 206 | [optional('-'), nonZeroDigit, parseGreedy(digit)], 207 | "0" 208 | ]), 209 | std.parseInt 210 | ); 211 | local jsonString = parseSeq(["\"", parseGreedy(parseAny(jsonStringChar, escapedJsonStringChar)), "\""]); 212 | 213 | local filterNull(parser) = apply(parser, function(arr) std.filter(function (x) x != null, arr)); 214 | 215 | local whitespace = parseGreedy(" "); 216 | local in_whitespace(p) = apply([whitespace, p, whitespace], function(x) x[1]); 217 | 218 | // High-level stuff 219 | 220 | local parseList(openP, elemP, delimP, closeP) = function(state) 221 | local op = normalize(openP), ep = normalize(elemP), dp = normalize(delimP), cp = normalize(closeP); 222 | local openP = op, elemP = ep, delimP = dp, closeP = cp; 223 | local err = state[2]; 224 | if err != null then 225 | state 226 | else 227 | local parseListH(state) = 228 | local parsed = elemP(state); 229 | local state = parsed[1]; 230 | local err = state[2]; 231 | if err != null then 232 | [null, state] 233 | else 234 | // TODO - if parseAny could signify which option it went with, it could be simplified 235 | local parsedDelim = delimP(state); 236 | local stateDelim = parsedDelim[1]; 237 | local errDelim = stateDelim[2]; 238 | if errDelim != null then 239 | // TODO check if fatal critical 240 | local parsedClose = closeP(state); 241 | local stateClose = parsedClose[1]; 242 | [null, stateClose] 243 | else 244 | parseListH(stateDelim) 245 | ; 246 | local parsed = openP(state); 247 | local state = parsed[1]; 248 | local err = state[2]; 249 | if err != null then 250 | [null, state] 251 | else 252 | parseListH(state) 253 | ; 254 | 255 | // TODO(sbarzowski) keywords and easy distinguishing between types 256 | // or should it be a separate step? 257 | local lex(parsers, ignoredParsers, captureFunc=capture) = 258 | local ps = std.map(normalize, parsers); 259 | local parsers = ps; 260 | local cParsers = std.map(function(p) captureFunc(p), parsers); 261 | local iParsers = std.map(ignore, ignoredParsers); 262 | filterNull(parseGreedy(parseAny(cParsers+iParsers))) 263 | ; 264 | 265 | // Utilities 266 | 267 | // TODO(sbarzowski) maybe add helpers for cases when it must be valid or when we want to just check if it's valid or not 268 | local runParser(parser, text) = 269 | local p = normalize(parser); 270 | local parser = p; 271 | // TODO(sbarzowski) configurable starting position 272 | local parsed = parser([0, text, null]); 273 | local result = [parsed[0], parsed[1][2]]; 274 | result 275 | ; 276 | 277 | { 278 | runParser:: runParser, 279 | any:: parseAny, 280 | list:: parseList, 281 | seq:: parseSeq, 282 | const:: parseConst, 283 | greedy:: parseGreedy, 284 | capture:: capture, 285 | captureWith:: captureWith, 286 | captureWithCheck:: captureWithCheck, 287 | apply:: apply, 288 | setValue:: setValue, 289 | ignore:: ignore, 290 | noop:: noop, 291 | charMap:: parseCharMap, 292 | 293 | // Batteries 294 | optional:: optional, 295 | alpha:: alpha, 296 | alphaLower:: alphaLower, 297 | alphaUpper:: alphaUpper, 298 | digit:: digit, 299 | int:: int, 300 | whitespace:: whitespace, 301 | in_whitespace:: in_whitespace, 302 | jsonString:: jsonString, 303 | 304 | lex::lex, 305 | } 306 | -------------------------------------------------------------------------------- /test-utils.libsonnet: -------------------------------------------------------------------------------- 1 | local pc = import 'parser-combinators.libsonnet'; 2 | 3 | local assertParses(expected, parser, input) = 4 | std.assertEqual([expected, null], pc.runParser(parser, input)); 5 | 6 | local assertMismatch(parser, input) = 7 | std.assertEqual([null, "mismatch"], pc.runParser(parser, input)); 8 | 9 | local assertError(err, parser, input) = 10 | std.assertEqual([null, err], pc.runParser(parser, input)); 11 | 12 | { 13 | assertParses:: assertParses, 14 | assertMismatch:: assertMismatch, 15 | assertError:: assertError, 16 | } 17 | -------------------------------------------------------------------------------- /tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | shopt -s globstar 5 | 6 | for i in **/*-test.jsonnet; do 7 | echo "$i" 8 | jsonnet "$i" 9 | done 10 | --------------------------------------------------------------------------------