├── .dkrc ├── .editorconfig ├── .envrc ├── .gitignore ├── LICENSE ├── README.md ├── bin └── jqmd ├── jqmd.md ├── package.sh ├── script ├── README.md ├── bootstrap ├── cibuild ├── clean ├── console ├── server ├── setup ├── test └── update └── specs ├── Data.cram.md ├── JQ.cram.md ├── Smoke-Test.cram.md └── y2j.cram.md /.dkrc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | dk use: cram # run tests using the "cram" functional test tool 4 | dk use: entr-watch # watch files and re-run tests or other commands 5 | dk use: shell-console # make the "console" command enter a subshell 6 | dk use: bash32 # enable doing tests/console/etc. in bash3.2 w/docker 7 | dk use: shellcheck # support running shellcheck (via docker if not installed) 8 | 9 | # Define overrides, new commands, functions, etc. here: 10 | 11 | # SC1090 = dynamic 'source' command 12 | # SC2128 = array/string mixing 13 | # SC2178 = array/string mixing 14 | SHELLCHECK_OPTS='-e SC1090,SC2128,SC2178' 15 | 16 | # Define overrides, new commands, functions, etc. here: 17 | 18 | on "boot" require mdsh github bashup/mdsh master bin/mdsh 19 | on "boot" require realpaths github bashup/realpaths master realpaths 20 | 21 | on "build" mdsh --out bin/jqmd --compile jqmd.md 22 | on "test" eval 'dk shellcheck /dev/stdin < <(mdsh --compile jqmd.md)' 23 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | [{jqmd.md,bin/jqmd}] 2 | indent_size = 4 3 | tab_width = 4 4 | 5 | -------------------------------------------------------------------------------- /.envrc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # basher installation scheme for dependencies; you can change this if you want, 4 | # so long as all the variables are correct. The .devkit/dk script will clone 5 | # basher to $BASHER_ROOT and look for binaries in $BASHER_INSTALL_BIN. 6 | 7 | export BASHER_PREFIX="$PWD/.deps" 8 | export BASHER_INSTALL_BIN="$BASHER_PREFIX/bin" 9 | export BASHER_INSTALL_MAN="$BASHER_PREFIX/man" 10 | 11 | # Dependencies are checked out here: 12 | export BASHER_PACKAGES_PATH="$BASHER_PREFIX" 13 | export BASHER_ROOT="$BASHER_PACKAGES_PATH/basherpm/basher" 14 | 15 | # Activate virtualenv if present 16 | [[ -f $BASHER_INSTALL_BIN/activate && -f $BASHER_INSTALL_BIN/python ]] && 17 | [[ ! "${VIRTUAL_ENV-}" || $VIRTUAL_ENV != "$BASHER_PREFIX" ]] && 18 | VIRTUAL_ENV_DISABLE_PROMPT=true source $BASHER_INSTALL_BIN/activate 19 | 20 | # Activate .composer/vendor/bin if PHP project 21 | [[ -f composer.json ]] && export PATH="$PWD/vendor/bin:$PATH" 22 | 23 | # Activate node_modules/.bin if Node project 24 | [[ -f package.json ]] && export PATH="$PWD/node_modules/.bin:$PATH" 25 | 26 | # $BASHER_INSTALL_BIN must be on PATH to use commands installed as deps 27 | [[ :$PATH: == *:$BASHER_INSTALL_BIN:* ]] || export PATH="$BASHER_INSTALL_BIN:$PATH" 28 | 29 | # You can add other variables you want available via direnv. Configuration 30 | # variables for devkit itself, however, should go in .dkrc unless they need 31 | # to be available via direnv as well. 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .deps 2 | .devkit 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 PJ Eby 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 6 | files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 7 | modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software 8 | is furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 11 | 12 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 13 | WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 15 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Literate jq+shell Programming with `jqmd` 2 | 3 | `jqmd` is a tool for writing well-documented, complex manipulations of YAML or JSON data structures using bash scripting and `jq`. It allows you to mix both kinds of code -- plus snippets of YAML or JSON data! -- within one or more markdown documents, making it easier to write scripts that do complex things like generate `docker-compose` configurations or manipulate serialized Wordpress options. 4 | 5 | `jqmd` is implemented as an extension of [`mdsh`](https://github.com/bashup/mdsh), which means you can extend it to process additional kinds of code blocks by defining functions inside your `shell @mdsh` blocks. But you do not need to install mdsh, and you can use `jqmd --compile` to make distributable scripts that don't require jqmd *or* mdsh. 6 | 7 | **Contents** 8 | 9 | 10 | 11 | - [Installation](#installation) 12 | - [Usage](#usage) 13 | * [Data Merging](#data-merging) 14 | * [Reusable Blocks](#reusable-blocks) 15 | * [Named Constants](#named-constants) 16 | - [Programming Models](#programming-models) 17 | * [Filters](#filters) 18 | * [Scripts](#scripts) 19 | * [Extensions](#extensions) 20 | - [Available Functions](#available-functions) 21 | * [Adding jq Code and Data](#adding-jq-code-and-data) 22 | * [JSON Escaping and Data Structures](#json-escaping-and-data-structures) 23 | * [Adding jq Options and Arguments](#adding-jq-options-and-arguments) 24 | * [Controlling jq Execution](#controlling-jq-execution) 25 | * [Command-line Arguments](#command-line-arguments) 26 | - [Supporting Additional Languages](#supporting-additional-languages) 27 | 28 | 29 | 30 | ### Installation 31 | 32 | If you have [`basher`](https://github.com/basherpm/basher) on your system, you can install jqmd with `basher install bashup/jqmd`; otherwise, just download the [jqmd executable](bin/jqmd), `chmod +x` it, and put it in a directory on your `PATH`. 33 | 34 | ### Usage 35 | 36 | Running `jqmd some-document.md args...` will read and interpret unindented, triple-backquote fenced code blocks from `some-document.md`, according to the language listed on the block: 37 | 38 | * `shell` -- interpreted as bash code, executed immediately. Shell blocks can invoke various jqmd functions as described later in this document. 39 | 40 | * `jq` -- jq code, which is added to a jq filter pipeline for execution at the end of the file, or to be run explicitly with the `RUN_JQ` function. Blocks written in jq can also be tagged with `@func` to turn them into shell functions instead of executing them immediately; see the section below on [reusable blocks](#reusable-blocks) for more details. 41 | 42 | * `jq defs` -- jq function definitions, which are accumulated over the course of the program run, and included at the start of any executed filter pipelines 43 | 44 | * `jq imports` -- jq module includes or imports, which are accumulated over the course of the program run, and included at the start of any executed filter pipelines (before the current set of `jq defs`). 45 | 46 | * `yaml`, `json` -- YAML data or JSON expressions, which are added to the jq filter pipeline as `jqmd_data(data)`. (Which turns the given data into a jq filter to modify an existing data structure; see [Data Merging](#data-merging), below for more details). Data blocks can also be tagged with `@func` and `!const` to turn them into shell functions or JQ constants instead of executing them immediately; see the sections below on [resusable blocks](#resusable-blocks) and [named constants](#named-constants) for more details. 47 | 48 | (Note: YAML data can only be processed if there is a `yaml2json` executable on `PATH`, the system `python` interpreter has PyYAML installed, or [yaml2json.php](https://packagist.org/packages/dirtsimple/yaml2json) is installed; otherwise an error will occur. (For best performance, we recommend installing a tool like this [yaml2json written in Go](https://github.com/bronze1man/yaml2json), as its process startup time alone is considerably smaller than that of Python or PHP.) 49 | 50 | Both YAML and JSON blocks can contain **jq string interpolation expressions**, denoted by ``\( )``. For example, a JSON block containing ``{ "foo": "\(env.BAR)"}`` will cause jq to insert the contents of the environment variable `BAR` into the data structure at the appropriate point. (Note that this means that if you have a backslash before a `(` in your YAML blocks and you *don't* want it to be treated as interpolation, you will need to add an extra backslash in front of it.) 51 | 52 | (In addition, `json` blocks do not have to be valid JSON: they can actually contain arbitrary jq expressions. The only real difference between a `json` block and a `jq` block is that a JSON block is automatically wrapped in a call to `jqmd_data()`.) 53 | 54 | (As with `mdsh`, you can extend the above list by defining appropriate hook functions in `shell @mdsh` blocks; see the section below on "Supporting Additional Languages" for more info.) 55 | 56 | Once all blocks have been executed or added to the filter pipeline, jq is run on standard input with the built-up filter pipeline, if any. (If the filtering pipeline is empty, jq is not run.) Filter pipeline elements are automatically separated with `|`, so you should not include a `|` at the beginning or end of your `jq` blocks or `APPLY` / `FILTER` code. 57 | 58 | As with `mdsh`, you can optionally make a markdown file directly executable by giving it a shebang line such as `#!/usr/bin/env jqmd`, or use a [shelldown header](https://github.com/bashup/mdsh#making-sourceable-scripts-and-handling-0) to make it executable, sourceable, and pretty. :) A sample shelldown header for jqmd might look like: 59 | 60 | ~~~markdown 61 | #!/usr/bin/env bash 62 | : ' 63 | 64 | 65 | # My Awesome Script 66 | 67 | ...markdown and code start here... 68 | ~~~ 69 | 70 | Also as with `mdsh`, you can run `jqmd --compile` to output a bash version of your script, with no external dependencies (other than jq and maybe `yaml2json` or PyYAML). `jqmd --compile` and `jqmd --eval` both inject the necessary jqmd runtime functions into the script so that it will work on systems without jqmd installed. (Note that unless your script uses the `YAML` or `yaml2json` functions at *runtime*, your script's users will not need it installed.) 71 | 72 | (If you'd like more information on compiling, sourcing, and shelldown headers, feel free to have a look at the [mdsh docs](https://github.com/bashup/mdsh)!) 73 | 74 | #### Data Merging 75 | 76 | In a jqmd program, one is often incrementally defining some sort of data structure (such as, e.g. a docker-compose project specification, or a set of Wordpress options). While jq expressions can be used directly to manipulate such a data structure, a more intuitive way to express such data structures is as a series of JSON or YAML blocks that are combined in some way. For this reason, jqmd defines an intuitive data structure merging function to apply such data blocks to an existing data structure. This merging function is exposed to jqmd programs as `jqmd::data($data)`, and is used by default to merge JSON and YAML data. The merge algorithm is as follows: 77 | 78 | * If `.` is an array, add `$data` to it (concatenating if `$data` is also an array, otherwise appending) 79 | * If `.` and `$data` are both objects, recursively merge their values using this same algorithm 80 | * In all other cases, return `$data` 81 | 82 | For most programs, this algorithm is sufficient to do most incremental data structure creation. If you have different needs, however, you can define a `jqmd_data` function of your own: JSON and YAML data are wrapped with a call to `jqmd_data`, but the default `jqmd_data` just calls `jqmd::data`. 83 | 84 | If you want to override the data merging for *all* data as of the start of the filter chain, you define a `jqmd_data` function in a `DEFINE` call or a `jq defs` block. Or, you can override it for just a few filters or blocks by defining it in an `APPLY` or `FILTER` call or `jq` block. Afterwards, you can restore the original data merging algorithm like this: 85 | 86 | ```shell 87 | FILTER 'def jqmd_data($data): jqmd::data($data) ; .' 88 | ``` 89 | 90 | #### Reusable Blocks 91 | 92 | Normally, code or data blocks are executed immediately, at the point they appear in the document. But for more complex scripts or libraries, this is a bit limiting. So jqmd allows you to turn blocks into shell functions, so they can be called more than once (or not at all), possibly with parameters. For example, the following markdown: 93 | 94 | ~~~markdown 95 | ```jq @func setElement key="$1" @val="$2" 96 | .[$key] = $val 97 | ``` 98 | 99 | ```yaml @func mksite SITE WP_HOME 100 | services: 101 | \($SITE): 102 | environment: 103 | WP_HOME: \($WP_HOME) 104 | ``` 105 | ~~~ 106 | 107 | ...expands into the following two shell functions: 108 | 109 | ```shell 110 | function setElement() { 111 | APPLY $'.[$key] = $val\n' \ 112 | key="$1" @val="$2" 113 | } 114 | 115 | function mksite() { 116 | APPLY $'jqmd_data({"services":{"\\($SITE)":{"environment":{"WP_HOME":"\\($WP_HOME)"}}}})\n' \ 117 | SITE WP_HOME 118 | } 119 | ``` 120 | 121 | Everything after the `@func name` part of the block opener becomes arguments to `APPLY`, which maps shell variables or other values to jq variables with the specified names. An `@` before an argument name means, "this variable or value is already JSON-encoded", and the absence of an `=` means "create a jq variable with the same name and value as this shell or environment variable". (Note: values after `=` should be quoted as shown above if they contain variables or shell parameters like `$1`.) 122 | 123 | So, our example `setElement` function takes two positional arguments and sets a key (given as a string) to a value (given as JSON data). So e.g. `setElement foo 42` would be equivalent to the jq expression `.foo = 42`. 124 | 125 | The second example function, `mksite`, sets the `WP_HOME` for a docker-compose service named `$SITE` with the *current* contents of `$SITE` and `$WP_HOME`. (Unlike normal docker-compose string interpolation -- which can only use one value for an environment variable -- this function can be called several times with different `SITE` and `WP_HOME` values to build up configuration for mutliple containers.) 126 | 127 | These are just a few examples of what you can do with reusable `@func` blocks. `@func` can only be used with `json`, `yaml`, or `jq` blocks. `jq` and `json` blocks can refer directly to parameter variables, while `yaml` blocks can only use string interpolation (`\( $var )` ) to insert string keys or values. `jq` blocks are applied as-is, while `json` and `yaml` blocks are wrapped in a call to `jqmd_data()` (as described in [Data Merging](#data-merging), above). 128 | 129 | #### Named Constants 130 | 131 | Data blocks can also be tagged as "named constants": a code block starting with e.g. `` ```yaml !const foo `` will have its contents defined as a zero-argument jq function named `foo`. 132 | 133 | That is, the following two code blocks do the exact same thing: 134 | 135 | ~~~markdown 136 | ```jq defs 137 | def pi: 3.14159; 138 | ``` 139 | ```json !const pi 140 | 3.14159 141 | ``` 142 | ~~~ 143 | 144 | 145 | ### Programming Models 146 | 147 | `jqmd` supports developing three types of programs: filters, scripts, and extensions. The main differences are that: 148 | 149 | * Filters typically run jq once, implicitly, at the end of the document, sending the output to stdout, 150 | * Scripts explicitly run jq multiple times or not at all, and 151 | * Extensions are shell scripts written using `jqmd` functions to create different markdown processing and/or jq support tools. 152 | 153 | #### Filters 154 | Filters are programs that build up a single giant jq pipeline, and then act as a filter, typically taking JSON input from stdin and sending the result to stdout. If your markdown document defines at least one filter, and doesn't use `RUN_JQ` or `CLEAR_FILTERS` to reset the pipeline, it's a filter. `jqmd` will automatically run `jq` to do the filtering from stdin to stdout, after the *entire markdown document* has been processed. If you don't want jq to read from stdin, you can use `JQ_OPTS -n` within your script to start the filter pipeline without any file input. (Similarly, you can use `JQ_OPTS -- somefile` to force jq to read input from a specific file instead of stdin.) 155 | 156 | #### Scripts 157 | 158 | If your program isn't a filter, it's probably a script. Scripts can run jq with shared imports, functions, and arguments, using the `RUN_JQ` function. (They must not add anything to the filter pipeline after the last `RUN_JQ` or `CLEAR_FILTERS` call, though, or `jqmd` will think the program's a filter!) 159 | 160 | You'll generally use this approach if your script needs to run jq multiple times with different inputs and filters. Each time a script uses the `CLEAR_FILTERS` or `RUN_JQ` functions, the filter pipeline is reset to empty and can then be built up again to run different operations. 161 | 162 | (Note: unlike the filter pipeline, jq options, arguments, imports, and defintions are *cumulative*. They can only be added to as the program executes, and cannot be reset. Thus, they are shared across all invocations of `RUN_JQ`. So anything specific to a given run of jq should be specified as a filter, or passed as an explicit command-line argument to `RUN_JQ`.) 163 | 164 | #### Extensions 165 | 166 | `jqmd` itself can be extended by other shell scripts, to make more-specialized tools or custom interpreters. Sourcing `jqmd` from a bash script will define all its functions, but not actually run a program. In this way, you can use all of the available functions described below (plus any of `mdsh`'s underlying API) in a shell script, rather than a markdown file. (You can also use or redefine jqmd and mdsh's internal functions, but those not documented here or in the mdsh documentation are subject to change without notice!) 167 | 168 | If you are sourcing `jqmd` (whether it's to write an extension or reuse its functions), you should also read the [mdsh docs](https://github.com/bashup/mdsh), since jqmd is an extension of mdsh. 169 | 170 | ### Available Functions 171 | 172 | Within `shell` blocks, many functions are available for your use. When passing `jq` code to them, it's best to use single quotes to avoid unwanted interpretation of $ variables or other quoting issues, e.g.: 173 | 174 | ```shell 175 | DEFINE ' 176 | def recursive_add($other): . as $original | 177 | reduce paths(type=="array") as $path ( 178 | (. // {}) * $other; setpath( $path; ($original | getpath($path)) + ($other | getpath($path)) ) 179 | ); 180 | ' 181 | DEFINE 'def jqmd_data($arg): recursive_add($arg);' 182 | ``` 183 | 184 | #### Adding jq Code and Data 185 | 186 | * `APPLY` *expr [`@`]name[`=`value]...* -- add *expr* to the jq filter pipeline, with the named jq variables bound to the specified values or the value of the corresponding shell variable. If *expr* is the empty string or `.`, the variables can be used by the entire filter chain past this point; otherwise they are only visible within *expr*. 187 | 188 | Each *name* must be a valid jq variable name (minus the leading `$`). If the `=`*value* is omitted, the value of the shell variable *name* is used. By default, the value is received by jq as a string, but if *name* is prefixed with `@`, then the value is interpreted as JSON. So, if you need to pass in a number, boolean, or other value already in JSON format (even a complex data structure) you can use `@` to pass it in -- even if it's untrusted user-supplied data. e.g.: 189 | 190 | ```shell 191 | APPLY 'some_func($foo; $bar)' @foo=42 @bar="$untrusted_json" 192 | ``` 193 | 194 | This code will call `some_func(42; $bar)` with jq's `$bar` variable set to the arbitrary JSON value from `$untrusted_json`, or else abort with an error during the jq run if `$untrusted_json` contains invalid JSON. 195 | 196 | * `IMPORTS` *arg* -- add the given jq `import` or `include` statements to a block that will appear at the very beginning of the jq "program". (Each statement must be terminated with `;`, as is standard for jq.) Imports are accumulated in the order they are processed, but *all* imports active as of a given jq run will be placed at the beginning of the overall program, as required by jq syntax. 197 | 198 | (This function is the programmatic equivalent of including a `jq imports` code block at the current point of execution.) 199 | 200 | * `DEFINE` *arg* -- add the given jq `def` statements to a block that will appear after the `IMPORTS`, but *before* any filters. (Each statement must be terminated with `;`, as is standard for jq.) 201 | 202 | This function is the programmatic equivalent of including a `jq defs` code block at the current point of execution. 203 | 204 | Note: you do **not** have to define all your functions this way. Functions can also be defined at the beginning of `FILTER` blocks or `jq`-tagged code blocks. The main benefits of using `DEFINE` or `jq defs` blocks are that: 205 | 206 | - They can be done "out of order" within a document: you can use a function in a `jq` or `FILTER` block *before* its `DEFINE` block appears, as long as the `DEFINE` happens before jq is actually run. 207 | 208 | - In a script that runs jq more than once, `IMPORTS` and `DEFINE` blocks persist across jq runs, while `jq` and `FILTER` blocks reset after every `RUN_JQ`. 209 | 210 | - While a `jq` or `FILTER` block *has* to include a filter expression of some kind (even if it's just `.`), `DEFINE` blocks can **only** contain definitions and comments. 211 | 212 | (Well, technically, you *can* include filtering expressions in a `DEFINE` block, but it's not recommended, and you would then have to end the block with a `|` to get a syntactically-correct jq program.) 213 | 214 | * `FILTER` *expr [args...]* -- add the given jq expression to the jq filter pipeline. The expression is automatically prefixed with `|` if any filter expressions have already been added to the pipeline. (This function is the programmatic equivalent of including a `jq` code block at the current point of execution.) 215 | 216 | If any arguments are supplied after *expr*, they are inserted as JSON-quoted strings wherever `%s` appears in it. (So `FILTER "foo(%s; %s)" bar baz` will expand to `foo("bar", "baz")`. In this way, you can insert arbitrary strings into a jq expression, even if they contain characters that must be escaped in JSON. 217 | 218 | If you are using arguments, *expr* is interpreted as a bash `printf` format string, which means that you must escape any actual `%` signs as `%%`, and should be careful with backslashes in it. (If you don't pass any *args* after the *expr*, these issues don't apply, as the string is used as-is.) 219 | 220 | Every `jq`-tagged code block or `FILTER` argument **must** contain a jq expression. Since jq expressions can begin with function definitions, this means that you can begin a filter with function definitions. This can be useful for redefining `jqmd_data` or other functions at various points within your filter pipeline, or to define functions that will only be used for one `RUN_JQ` pipeline. 221 | 222 | Bear in mind, however, that because a filter block *must* contain a valid jq expression, you may need to terminate your filter with a `.` if it contains only functions. For example, this bit of `jq` code is a valid filter, because it ends with a `.`: 223 | 224 | ```jq 225 | # Add as many functions as you like 226 | def f1($other): something; 227 | def f2: another(thing); 228 | 229 | # but finish with a '.' to create a no-op filtering expression 230 | . 231 | ``` 232 | 233 | This "end function-only filters with a ." rule applies whether you're using `jq`-tagged code blocks or the `FILTER` function. 234 | 235 | * `JSON` *data [args...]* -- a shortcut for `FILTER "jqmd_data(`*data*`)"` *args...*. This function is the programmatic equivalent of including a `json` code block at the current point of execution, but it can also include interpolated args, as with `FILTER` (and the same rules for `%s` and escaping `%` apply if you supply any *args*). 236 | 237 | * `YAML` *data* -- a shortcut for `FILTER "jqmd_data(`*data-converted-to-json*`)"`. This function is the programmatic equivalent of including a `yaml` code block at the current point of execution, and only works if there is a `yaml2json` converter on `PATH`, the system default `python` has PyYAML installed, or [yaml2json.php](https://packagist.org/packages/dirtsimple/yaml2json) is on the system `PATH`.) 238 | 239 | * `yaml2json` -- a filter that takes YAML or JSON input, and produces JSON output. The actual implementation is system-dependent, using either a `yam2json` command line tool, Python, or PHP, depending on what's available. This can be used to convert data, validate it, or to remove jq expressions from untrusted input. 240 | 241 | 242 | Notice that JSON and YAML blocks are always filtered through a `jqmd_data()` function, which by default does [data merging](#data-merging), but you can always redefine the function to do something different, even as part of a `FILTER` or jq block. (Just remember that while filters can begin with function definitions, they must each *end* with an expression, even if it's only a `.`.) 243 | 244 | Also note that data passed to the `JSON` and `YAML` functions *can contain jq interpolation expressions*, which means that you **must not pass untrusted data to them**. If you need to process a user-supplied JSON string, the simplest way is to use `JSON "( %s | fromjson)" "$untrusted_json"`. Alternately, you can call `ARGJSON someJQvarname "$untrusted_json"` to create the jq variable `$someJQvarname`, and then use it with e.g. `JSON '$someJQvarname'` . (Note the single quotes!) 245 | 246 | (If your user-supplied data is in YAML form, you can use the same approaches, but must convert it to JSON first.) 247 | 248 | #### JSON Escaping and Data Structures 249 | 250 | These functions don't do anything to jq or the filter pipeline; they simply escape, quote, or otherwise format values into JSON, returning the result(s) via `REPLY`. You can then use them to build up `FILTER` strings, or pipe them to jq as input. 251 | 252 | * `JSON-QUOTE` *strings...* -- set `REPLY` to an array containing the JSON-quoted version of *strings*. Each element in the resulting array will begin and end with double quotes, and have proper backslash escapes for contained control characters, double quotes, and backslashes. 253 | * `JSON-LIST` *strings...* -- set `REPLY` to a string representing a JSON list of the given *strings*. 254 | * `JSON-KV` *"key=val"...* -- set `REPLY` to a string representing a JSON object mapping from each given key to a string value. Keys cannot contain `=`. If an argument doesn't contain an `=`, its value is equal to its key. 255 | * `JSON-MAP` *assoc-array* -- (bash 4+ only) set `REPLY` to a string representing a JSON object containing the contents of the named *assoc-array* 256 | * `escape-ctrl-characters` *strings...* -- set `REPLY` to an array containing *strings* with control characters escaped as `\n`, `\t`, `\r`, or `\uXXXX`. This function is used internally by the other `JSON-x` functions when their argument(s) contain control characters. 257 | 258 | #### Adding jq Options and Arguments 259 | 260 | * `JQ_OPTS` *opts...* -- add *opts* to the jq command line being built up. Whenever jq is run (either explicitly using `RUN_JQ` or `CALL_JQ`, or implicitly at the end of the document), the given options will be part of the command line. 261 | * `ARG` *name value* -- define a jq variable named `$`*name*, with the supplied string value. (Shortcut for `JQ_OPTS --arg name value`.) 262 | * `ARGJSON` *name json-value* -- define a jq variable named `$`*name*, with the supplied JSON value. (Shortcut for `JQ_OPTS --argjson name json`.) This is especially useful for passing the output of other programs or data files as arguments to your jq code, e.g. `ARGJSON something "$(wp option get something --format=json)"`. 263 | * `ARGSTR` *string* and `ARGVAL` *json-value* -- these functions work like `ARG` and `ARGJSON`, but instead of you passing in an argument name, a unique argument name is automatically generated, and returned in `$REPLY`. The returned string will expand to the passed in-value in any jq expressions. 264 | 265 | 266 | (Note: the added options will reset to empty again after `RUN_JQ`, `CALL_JQ`, or `CLEAR_FILTERS`.) 267 | 268 | #### Controlling jq Execution 269 | 270 | * `RUN_JQ` *args...* -- invoke `$JQ_CMD` (`jq` by default) with the current `JQ_OPTS` and given *args*. If a "program" is given in `JQ_OPTS` (i.e., a non-option argument other than `--`), it's added to the filter pipeline, after any `IMPORTS` and `DEFINE` blocks established so far. Any `-f` or `--fromfile` options are similarly added to the filter pipeline, and multiple such files are allowed. (Unlike plain jq, which doesn't work properly with multiple `-f` options.) 271 | 272 | After jq is run, the filter pipeline is emptied with `CLEAR_FILTERS`. 273 | 274 | * `CALL_JQ` *args...* -- exactly like `RUN_JQ`, except that the output of `jq` is captured into `$REPLY`. You should use this instead of shell substitution to capture jq's output. 275 | 276 | * `CLEAR_FILTERS` -- reset the current filter pipeline and `JQ_OPTS` to empty. This can be used at the end of a script to keep `jqmd` from running jq on stdin/stdout. 277 | 278 | * `HAVE_FILTERS` -- succeeds if there is anything in the filter pipeline at the time of excution, fails otherwise. (i.e., you can use `if HAVE_FILTERS; then ...` to take action in a script based on the current filter state. 279 | 280 | Note: piping into `RUN_JQ` or `CALL_JQ`, or invoking them in a subshell or shell substituion will *not* reset the current filter pipeline. To capture jq's output, use `CALL_JQ` instead of shell substitution. To pipe input into jq, pass it as a post-`--` argument to `RUN_JQ` or `CALL_JQ`, e.g.: 281 | 282 | ~~~sh 283 | $ echo '"something"' | RUN_JQ . # WRONG: CLEAR_FILTERS won't run 284 | $ RUN_JQ . -- <(echo '"something"') # RIGHT: use process substitution instead of piping 285 | 286 | $ foo bar "$(RUN_JQ)" # WRONG: CLEAR_FILTERS won't run 287 | $ CALL_JQ; foo bar "$REPLY" # RIGHT 288 | ~~~ 289 | 290 | #### Command-line Arguments 291 | 292 | You can pass additional arguments to `jqmd`, after the path to the markdown file. These additional arguments are available as `$1`, `$2`, etc. within any top-level `shell` code in the markdown file. 293 | 294 | ### Supporting Additional Languages 295 | 296 | By default, `jqmd` only interprets unindented, triple-backquoted markdown blocks tagged as `shell`, `jq`, `jq defs`, `jq imports`, `yaml`, `yml`, or `json`. Unindented triple-backquoted blocks with any other tags are interpreted as data and assigned to shell variables, as described in the [mdsh docs on data blocks](https://github.com/bashup/mdsh#data-blocks). 297 | 298 | As with `mdsh`, however, you can define interpreters for other block types by defining `mdsh-lang-X` or `mdsh-compile-X` functions in `shell @mdsh` blocks, via a wrapper script, or as exported functions in your bash environment. (You can also override these functions to change jqmd's default interpretation of jq, YAML, or JSON blocks.) 299 | 300 | For more information on how to do this, see the [mdsh docs on processing non-shell languages](https://github.com/bashup/mdsh#processing-non-shell-languages), or consult the mdsh docs in general for more info on what you can do with jqmd. -------------------------------------------------------------------------------- /bin/jqmd: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # --- 3 | # This file is automatically generated from jqmd.md - DO NOT EDIT 4 | # --- 5 | 6 | # MIT License 7 | # 8 | # Copyright (c) 2017 PJ Eby 9 | # 10 | # Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 11 | # files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 12 | # modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software 13 | # is furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 18 | # WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 19 | # COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 20 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | 22 | set -euo pipefail # Strict mode 23 | mdsh-parse() { 24 | local cmd=$1 lno=0 block_start lang mdsh_block ln indent fence close_fence indent_remove 25 | local open_fence=$'^( {0,3})(~~~+|```+) *([^`]*)$' 26 | while ((lno++)); IFS= read -r ln; do 27 | if [[ $ln =~ $open_fence ]]; then 28 | indent=${BASH_REMATCH[1]} fence=${BASH_REMATCH[2]} lang=${BASH_REMATCH[3]} mdsh_block= 29 | block_start=$lno close_fence="^( {0,3})$fence+ *\$" indent_remove="^${indent// / ?}" 30 | while ((lno++)); IFS= read -r ln && ! [[ $ln =~ $close_fence ]]; do 31 | ! [[ $ln =~ $indent_remove ]] || ln=${ln#${BASH_REMATCH[0]}}; mdsh_block+=$ln$'\n' 32 | done 33 | lang="${lang%"${lang##*[![:space:]]}"}"; "$cmd" fenced "$lang" "$mdsh_block" 34 | fi 35 | done 36 | } 37 | mdsh-source() { 38 | local MDSH_FOOTER='' MDSH_SOURCE 39 | if [[ ${1:--} != '-' ]]; then 40 | MDSH_SOURCE="$1" 41 | mdsh-parse __COMPILE__ <"$1" 42 | else mdsh-parse __COMPILE__ 43 | fi 44 | ${MDSH_FOOTER:+ printf %s "$MDSH_FOOTER"}; MDSH_FOOTER= 45 | } 46 | mdsh-compile() ( # <-- force subshell to prevent escape of compile-time state 47 | mdsh-source "$@" 48 | ) 49 | __COMPILE__() { 50 | [[ $1 == fenced && $fence == $'```' && ! $indent ]] || return 0 # only unindented ``` code 51 | local mdsh_tag=$2 mdsh_lang tag_words 52 | mdsh-splitwords "$2" tag_words # check for command blocks first 53 | case ${tag_words[1]-} in 54 | '') mdsh_lang=${tag_words[0]-} ;; # fast exit for common case 55 | '@'*) 56 | mdsh_lang=${tag_words[1]#@} ;; # language alias: fall through to function lookup 57 | '!'*) 58 | mdsh_lang=${tag_words[0]}; set -- "$3" "$2" "$block_start"; eval "${2#*!}"; return 59 | ;; 60 | '+'*) 61 | printf 'mdsh_lang=%q; %s %q\n' "${tag_words[0]}" "${2#"${tag_words[0]}"*+}" "$3" 62 | return 63 | ;; 64 | '|'*) 65 | printf 'mdsh_lang=%q; ' "${tag_words[0]}" 66 | echo "${2#"${tag_words[0]}"*|} <<'\`\`\`'"; printf $'%s```\n' "$3" 67 | return 68 | ;; 69 | *) mdsh_lang=${2//[^_[:alnum:]]/_} # convert entire line to safe variable name 70 | esac 71 | mdsh-emit-block 72 | } 73 | mdsh-block() { 74 | local mdsh_lang=${1-${mdsh_lang-}} mdsh_block=${2-${mdsh_block-}} 75 | local block_start=${3-${block_start-}} mdsh_tag=${4-${mdsh_lang-}} tag_words 76 | mdsh-splitwords "$mdsh_tag" tag_words; mdsh-emit-block 77 | } 78 | mdsh-emit-block() { 79 | if fn-exists "mdsh-lang-$mdsh_lang"; then 80 | mdsh-rewrite "mdsh-lang-$mdsh_lang" "{" "} <<'\`\`\`'"; printf $'%s```\n' "$mdsh_block" 81 | elif fn-exists "mdsh-compile-$mdsh_lang"; then 82 | "mdsh-compile-$mdsh_lang" "$mdsh_block" "$mdsh_tag" "$block_start" 83 | else 84 | mdsh-misc "$mdsh_tag" "$mdsh_block" 85 | fi 86 | if fn-exists "mdsh-after-$mdsh_lang"; then 87 | mdsh-rewrite "mdsh-after-$mdsh_lang" 88 | fi 89 | } 90 | # split words in $1 into the array named by $2 (REPLY by default), without wildcard expansion 91 | # shellcheck disable=SC2206 # set -f is in effect 92 | mdsh-splitwords() { 93 | local f=$-; set -f; if [[ ${2-} ]]; then eval "$2"'=($1)'; else REPLY=($1); fi 94 | [[ $1 == *f* ]] || set +f 95 | } 96 | # fn-exists: succeed if argument is a function 97 | fn-exists() { declare -F -- "$1"; } >/dev/null 98 | # Output body of func $1, optionally replacing the opening/closing { and } with $2 and $3 99 | mdsh-rewrite() { 100 | local b='}' r; r="$(declare -f -- "$1")"; r=${r#*{ }; r=${r%\}*}; echo "${2-{}$r${3-$b}" 101 | } 102 | mdsh-misc() { mdsh-data "$@"; } # Treat unknown languages as data 103 | mdsh-compile-() { :; } # Ignore language-less blocks 104 | 105 | mdsh-compile-mdsh() { eval "$1"; } # Execute `mdsh` blocks in-line 106 | mdsh-compile-mdsh_main() { ! @is-main || eval "$1"; } 107 | 108 | mdsh-compile-shell() { printf '%s' "$1"; } # Copy `shell` blocks to the output 109 | mdsh-compile-shell_main() { ! @is-main || printf '%s' "$1"; } 110 | mdsh-data() { 111 | printf 'mdsh_raw_%s+=(%q)\n' "${1//[^_[:alnum:]]/_}" "$2" 112 | } 113 | mdsh-compile-shell_mdsh() { 114 | indent='' fence=$'```' __COMPILE__ fenced mdsh "$1" 115 | } 116 | mdsh-compile-shell_mdsh_main() { 117 | indent='' fence=$'```' __COMPILE__ fenced "mdsh main" "$1" 118 | } 119 | # Main program: check for arguments and run markdown script 120 | mdsh-main() { 121 | (($#)) || mdsh-error "Usage: %s [--out FILE] [ --compile | --eval ] markdownfile [args...]" "${0##*/}" 122 | case "$1" in 123 | --) mdsh-interpret "${@:2}" ;; 124 | --*|-?) fn-exists "mdsh.$1" || mdsh-error "%s: unrecognized option: %s" "${0##*/}" "$1" 125 | "mdsh.$1" "${@:2}" 126 | ;; 127 | -??*) mdsh-main "${1::2}" "-${1:2}" "${@:2}" ;; # split '-abc' into '-a -bc' and recurse 128 | *) mdsh-interpret "$@" ;; 129 | esac 130 | } 131 | # Run markdown file as main program, with $0 == $BASH_SOURCE == "" and 132 | # MDSH_ZERO pointing to the original $0. 133 | 134 | function mdsh-interpret() { 135 | printf -v cmd $'eval "$(%q --compile %q)"' "$0" "$1" 136 | MDSH_ZERO="$1" exec bash -c "$cmd" "" "${@:2}" 137 | } 138 | mdsh.--compile() { 139 | (($#)) || mdsh-error "Usage: %s --compile FILENAME..." "${0##*/}" 140 | ! fn-exists mdsh:file-header || mdsh:file-header 141 | for REPLY; do mdsh-compile "$REPLY"; done 142 | ! fn-exists mdsh:file-footer || mdsh:file-footer 143 | } 144 | 145 | mdsh.-c() { mdsh.--compile "$@"; } 146 | mdsh.--eval() { 147 | { (($# == 1)) && [[ $1 != - ]]; } || 148 | mdsh-error "Usage: %s --eval FILENAME" "${0##*/}" 149 | mdsh.--compile "$1" 150 | echo $'__status=$? eval \'return $__status || exit $__status\' 2>/dev/null' 151 | } 152 | 153 | mdsh.-E() { mdsh.--eval "$@"; } 154 | mdsh.--out() { 155 | REPLY=("$(set -e; mdsh-main "${@:2}")") 156 | mdsh-ok && exec echo "$REPLY" >"$1" # handle self-compiling properly 157 | } 158 | 159 | mdsh.-o() { mdsh.--out "$@"; } 160 | # mdsh-error: printf args to stderr and exit w/EX_USAGE (code 64) 161 | # shellcheck disable=SC2059 # argument is a printf format string 162 | mdsh-error() { exit 64 "$1" "${2-}" "${@:3}"; } 163 | mdsh.--help() { 164 | printf 'Usage: %s [--out FILE] [ --compile | --eval ] markdownfile [args...]\n' "${0##*/}" 165 | echo $' 166 | Run and/or compile code blocks from markdownfile(s) to bash. 167 | Use a filename of `-` to run or compile from stdin. 168 | 169 | Options: 170 | -h, --help Show this help message and exit 171 | -c, --compile MDFILE... Compile MDFILE(s) to bash and output on stdout. 172 | -E, --eval MDFILE Compile one file w/a shelldown-support footer line\n' 173 | } 174 | 175 | mdsh.-h() { mdsh.--help "$@"; } 176 | MDSH_LOADED_MODULES= 177 | MDSH_MODULE= 178 | 179 | @require() { 180 | flatname "$1" 181 | if ! [[ $MDSH_LOADED_MODULES == *"<$REPLY>"* ]]; then 182 | MDSH_LOADED_MODULES+="<$REPLY>"; local MDSH_MODULE=$1 183 | "${@:2}" 184 | fi 185 | } 186 | @is-main() { ! [[ $MDSH_MODULE ]]; } 187 | @module() { 188 | @is-main || return 0 189 | set -- "${1:-${MDSH_SOURCE-}}" 190 | echo "#!/usr/bin/env bash" 191 | echo "# ---" 192 | echo "# This file is automatically generated from ${1##*/} - DO NOT EDIT" 193 | echo "# ---" 194 | echo 195 | } 196 | @main() { 197 | @is-main || return 0 198 | MDSH_FOOTER=$'if [[ $0 == "${BASH_SOURCE-}" ]]; then '"$1"$' "$@"; exit; fi\n' 199 | } 200 | @comment() ( # subshell for cd 201 | ! [[ "${MDSH_SOURCE-}" == */* ]] || cd "${MDSH_SOURCE%/*}" 202 | sed -e 's/^\(.\)/# \1/; s/^$/#/;' "$@" 203 | echo 204 | ) 205 | # shellcheck disable=2059 206 | exit() { 207 | set -- "${1-$?}" "${@:2}" 208 | case $# in 0|1) : ;; 2) printf '%s\n' "$2" ;; *) printf "$2\\n" "${@:3}" ;; esac >&2 209 | builtin exit "$1" 210 | } 211 | mdsh-ok(){ return $?;} 212 | mdsh-embed() { 213 | local f=$1 base=${1##*/}; local boundary="# --- EOF $base ---" contents ctr= 214 | [[ $f == */* && -f $f ]] || f=$(command -v "$f") || { 215 | echo "Can't find module $1" >&2; return 69 # EX_UNAVAILABLE 216 | } 217 | contents=$'\n'$(<"$f")$'\n' 218 | while [[ $contents == *$'\n'"$boundary"$'\n'* ]]; do 219 | ((ctr++)); boundary="# --- EOF $base.$ctr ---" 220 | done 221 | printf $'{ if [[ $OSTYPE != cygwin && $OSTYPE != msys && -e /dev/fd/0 ]]; then source /dev/fd/0; else source <(cat); fi; } <<\'%s\'%s%s\n' "$boundary" "$contents" "$boundary" 222 | } 223 | mdsh-make() { 224 | [[ -f "$1" && -f "$2" && ! "$1" -nt "$2" && ! "$1" -ot "$2" ]] || { 225 | ( "${@:3}" && mdsh-main --out "$2" --compile "$1" ); mdsh-ok && touch -r "$1" "$2" 226 | } 227 | } 228 | mdsh-cache() { 229 | [[ -d "$1" ]] || mkdir -p "$1" 230 | flatname "${3:-$2}"; REPLY="$1/$REPLY"; mdsh-make "$2" "$REPLY" "${@:4}" 231 | } 232 | flatname() { 233 | REPLY="${1//\%/%25}"; REPLY="${REPLY//\//%2F}"; REPLY="${REPLY/#./%2E}" 234 | REPLY="${REPLY///%3E}" 235 | REPLY="${REPLY//\\/%5C}" 236 | } 237 | MDSH_CACHE= 238 | mdsh-use-cache() { 239 | if ! (($#)); then 240 | set -- "${XDG_CACHE_HOME:-${HOME:+$HOME/.cache}}" 241 | set -- "${1:+$1/mdsh}" 242 | fi 243 | MDSH_CACHE="$1" 244 | } 245 | mdsh-use-cache 246 | mdsh-run() { 247 | if [[ ${MDSH_CACHE-} ]]; then 248 | mdsh-cache "$MDSH_CACHE" "$1" "${2-}" 249 | mdsh-ok && source "$REPLY" "${@:3}" 250 | else run-markdown "$1" "${@:3}" 251 | fi 252 | } 253 | # run-markdown file args... 254 | # Compile `file` and source the result, passing along any positional arguments 255 | run-markdown() { 256 | REPLY=("$(set -e; mdsh-source "${1--}")"); mdsh-ok || return 257 | if [[ $BASH_VERSINFO == 3 ]]; then # bash 3 can't source from proc 258 | # shellcheck disable=SC1091 # shellcheck shouldn't try to read stdin 259 | source /dev/fd/0 "${@:2}" <<<"$REPLY" 260 | else source <(echo "$REPLY") "${@:2}" 261 | fi 262 | } 263 | mdsh_raw_bash_runtime+=($'#!/usr/bin/env bash\n\n# --- BEGIN jqmd runtime ---\n') 264 | mdsh_raw_bash_runtime+=($'jqmd_imports=\njqmd_filters=\njqmd_defines=\n\nHAVE_FILTERS() { [[ ${jqmd_filters-} ]]; }\nCLEAR_FILTERS() { unset jqmd_filters; JQ_OPTS=(jq); }\n\nIMPORTS() { jqmd_imports+="${jqmd_imports:+$\'\\n\'}$1"; }\nDEFINE() { jqmd_defines+="${jqmd_defines:+$\'\\n\'}$1"; }\nFILTER() {\n\tcase $# in\n\t1) jqmd_filters+="${jqmd_filters:+$\'\\n\'| }$1"; return ;;\n\t0) return ;;\n\tesac\n\tlocal REPLY ARGS=(printf -v REPLY "$1"); shift\n\tJSON-QUOTE "$@"; ARGS+=("${REPLY[@]}"); "${ARGS[@]}"; FILTER "$REPLY"\n}\n\nAPPLY() {\n\tlocal name filter=\'\' REPLY expr=${1-} lf=$\'\\n\'; shift\n\twhile (($#)); do\n\t\tname=${1#@}; REPLY=${name#*=}\n\t\t[[ $name == *=* ]] || REPLY=${!name}\n\t\tif ((${#REPLY} > 32)); then\n\t\t\tif [[ $1 == @* ]]; then ARGVAL "$REPLY"; else ARGSTR "$REPLY"; fi\n\t\telse\n\t\t\tJSON-QUOTE "$REPLY"; [[ $1 != @* ]] || REPLY="($REPLY|fromjson)"\n\t\tfi\n\t\tfilter+=" | $REPLY as \\$${name%%=*}"; shift\n\tdone\n\tfilter=${filter:3}; [[ ! $expr || $expr == . ]] || filter="( ${filter:+$filter | }$expr )"\n\t${filter:+FILTER "$filter"}\n}\n') 265 | mdsh_raw_bash_runtime+=($'JSON-QUOTE() {\n\tlocal LC_ALL=C\n\t[[ $* != *\\\\* ]] || set -- "${@//\\\\/\\\\\\\\}"\n\t[[ $* != *\'"\'* ]] || set -- "${@//\\"/\\\\\\"}"\n\tset -- "${@/#/\\"}"; set -- "${@/%/\\"}"\n\tif [[ $* != *[$\'\\x01\'-$\'\\x1F\']* ]]; then REPLY=("$@"); else escape-ctrlchars "$@"; fi\n}\n\nJSON-LIST() {\n\t(($#)) || { REPLY=("[]"); return; }\n\tJSON-QUOTE "$@"; REPLY=${REPLY[*]/%/,}; REPLY=("[${REPLY%,}]")\n}\n\nJSON-MAP() {\n\tlocal -n v=$1; local LC_ALL=C n=${v[@]+${#v[@]}}; ((${n:-0})) || { REPLY=("{}"); return; }\n\tlocal i=1 p=\'\\"${@:i:1}\\": \\"${@: \'"$n"\'+ i++:1}\\",\'; p="${v[*]/#*/$p}"\n\tset -- "${!v[@]}" "${v[@]}"\n\t[[ $* != *\\\\* ]] || set -- "${@//\\\\/\\\\\\\\}"\n\t[[ $* != *\\"* ]] || set -- "${@//\\"/\\\\\\"}"\n\teval "REPLY=(\\"{${p%,}}\\")"\n\t[[ $REPLY != *[$\'\\x01\'-$\'\\x1F\']* ]] || escape-ctrlchars "$REPLY"\n}\n\nJSON-KV() {\n\t(($#)) || { REPLY=("{}"); return; }\n\tlocal LC_ALL=C i=0 p=\'\\"${REPLY[i]}\\": \\"${REPLY[\'"$#"\'+i++]}\\",\'; p="${*/#*/$p}"\n\t[[ $* != *\\\\* ]] || set -- "${@//\\\\/\\\\\\\\}"\n\t[[ $* != *\\"* ]] || set -- "${@//\\"/\\\\\\"}"\n\tREPLY=("${@%%=*}" "${@#*=}"); eval "REPLY=(\\"{${p%,}}\\")"\n\t[[ $REPLY != *[$\'\\x01\'-$\'\\x1F\']* ]] || escape-ctrlchars "$REPLY"\n}\n\nescape-ctrlchars() {\n\tlocal LC_ALL=C\n\tset -- "${@//$\'\\n\'/\\\\n}"; set -- "${@//$\'\\r\'/\\\\r}"; set -- "${@//$\'\\t\'/\\\\t}" # \\n\\r\\t\n\tif [[ $* == *[$\'\\x01\'-$\'\\x1F\']* ]]; then\n\t\tlocal r s=$*; s=${s//[^$\'\\x01\'-$\'\\x1F\']}\n\t\twhile [[ $s ]]; do\n\t\t\tprintf -v r \\\\\\\\u%04x "\'${s:0:1}"; set -- "${@//${s:0:1}/$r}"\n\t\t\ts=${s//${s:0:1}/}\n\t\tdone\n\tfi\n\tREPLY=("$@")\n}\n\n') 266 | mdsh_raw_bash_runtime+=($'JQ_CMD=(jq)\nJQ_OPTS=(jq)\nJQ_SIZE_LIMIT=16000\nJQ_OPTS() { JQ_OPTS+=("$@"); }\nARG() { JQ_OPTS --arg "$1" "$2"; }\nARGJSON() { JQ_OPTS --argjson "$1" "$2"; }\nARGQUOTE() { ARGSTR "$1"; } # deprecated\nARGSTR() { REPLY=JQMD_QA_${#JQ_OPTS[@]}; ARG "$REPLY" "$1"; REPLY=\'$\'$REPLY; }\nARGVAL() { REPLY=JQMD_JA_${#JQ_OPTS[@]}; ARGJSON "$REPLY" "$1"; REPLY=\'$\'$REPLY; }\n') 267 | mdsh_raw_bash_runtime+=($'JQ_CMD() {\n\tlocal f= opt nargs cmd=("${JQ_CMD[@]}"); set -- "${JQ_OPTS[@]:1}" "$@"\n\n\twhile (($#)); do\n\t\tcase "$1" in\n\t\t-f|--fromfile)\n\t\t\topt=$(<"$2") || return 69\n\t\t\tFILTER "$opt"; shift 2; continue\n\t\t\t;;\n\t\t-L|--indent) nargs=2 ;;\n\t\t--arg|--argjson|--slurpfile|--argfile) nargs=3 ;;\n\t\t--) break ;; # rest of args are data files\n\t\t-*) nargs=1 ;;\n\t\t*) FILTER "$1"; break ;;\t# jq program: data files follow\n\t\tesac\n\t\tcmd+=("${@:1:$nargs}")\t# add $nargs args to cmd\n\t\tshift $nargs\n\tdone\n\n\tHAVE_FILTERS || FILTER . # jq needs at least one filter expression\n\tfor REPLY in "${jqmd_imports-}" "${jqmd_defines-}" "${jqmd_filters-}"; do\n\t\t[[ $REPLY ]] && f+=${f:+$\'\\n\'}$REPLY\n\tdone\n\n\tREPLY=("${cmd[@]}" "$f" "${@:2}")\n\tif ((${#f} > JQ_SIZE_LIMIT)); then REPLY=(JQ_WITH_FILE "${#cmd[@]}" "${REPLY[@]}"); fi\n\tCLEAR_FILTERS # cleanup for any re-runs\n}\n\nJQ_WITH_FILE() { local n=$(($1++2)); "${@:2:$1}" -f <(echo "${!n}") "${@:n+1}"; }\nRUN_JQ() { JQ_CMD "$@" && "${REPLY[@]}"; }\nCALL_JQ() { JQ_CMD "$@" && REPLY=("$("${REPLY[@]}")"); }\n') 268 | mdsh_raw_bash_runtime+=($'YAML() { y2j "$1"; JSON "$REPLY"; }\nJSON() { FILTER "jqmd_data($1)" "${@:2}"; }\n') 269 | mdsh_raw_bash_runtime+=($'DEFINE \'\ndef jqmd::blend($other; combine): . as $this | . * $other | . as $combined | with_entries(\n if (.key | in($this)) and (.key | in($other)) then\n .this = $this[.key] | .other = $other[.key] | combine\n else . end\n);\n\ndef jqmd::combine: (.this|type) as $this | (.other|type) as $other | .value =\n if $this == "array" then\n if $other == "array" then .this + .other else .this + [.other] end\n elif $this == "object" then\n if $other == "object" then\n .other as $o | (.this | jqmd::blend($o; jqmd::combine))\n else .other end\n else .other end; # everything else just overrides\n\ndef jqmd::data($data): {this: ., other:$data} | jqmd::combine | .value ;\ndef jqmd_data($data): jqmd::data($data) ;\n\'\n') 270 | mdsh_raw_bash_runtime+=($'y2j() {\n\tlocal p j; j="$(echo "$1" | yaml2json)" || return $?; REPLY=\n\twhile [[ $j == *\'\\\\(\'* ]]; do\n\t\tp=${j%%\'\\\\(\'*}; j=${j#"$p"\'\\\\(\'}\n\t\tif [[ $p =~ (^|[^\\\\])(\'\\\\\\\\\')*$ ]]; then\n\t\t\tp="${p}"\'\\(\' # odd, unbalance the backslash\n\t\telse\n\t\t\tp="${p}(" # even, remove one actual backslash\n\t\tfi\n\t\tREPLY+=$p\n\tdone\n\tREPLY+=$j\n}\n') 271 | mdsh_raw_bash_runtime+=($'yaml2json:cmd() { command yaml2json /dev/stdin; }\n\nyaml2json:py() {\n\tpython -c \'import sys, yaml, json; json.dump(yaml.safe_load(sys.stdin), sys.stdout)\'\n}\n\nyaml2json:php() { command yaml2json.php; }\n\nyaml2json() {\n\tlocal kind # auto-select between available yaml2json implementations\n\tfor kind in cmd py php; do\n\t\tREPLY=($(yaml2json:$kind < <(echo "a: {}") 2>/dev/null || true))\n\t\tprintf -v REPLY %s ${REPLY+"${REPLY[@]}"}\n\t\tif [[ "$REPLY" == \'{"a":{}}\' ]]; then\n\t\t\teval "yaml2json() { yaml2json:$kind; }"; yaml2json; return\n\t\tfi\n\tdone\n\texit 69 "To process YAML, must have one of: yaml2json, PyYAML, or yaml2json.php" # EX_UNAVAILABLE\n}\n\n# --- END jqmd runtime ---\n') 272 | # Language Support 273 | mdsh-compile-jq() { printf 'FILTER %q\n' "$1"$'\n'; } 274 | mdsh-compile-jq_defs() { printf 'DEFINE %q\n' "$1"$'\n'; } 275 | mdsh-compile-jq_imports() { printf 'IMPORTS %q\n' "$1"$'\n'; } 276 | 277 | mdsh-compile-yml() { y2j "$1"; mdsh-compile-json "$REPLY"; } 278 | mdsh-compile-yaml() { y2j "$1"; mdsh-compile-json "$REPLY"; } 279 | mdsh-compile-json() { mdsh-compile-jq "jqmd_data($1)"; } 280 | 281 | mdsh-compile-func() { 282 | case ${tag_words-} in 283 | yml|yaml) y2j "$1"; REPLY="jqmd_data($REPLY)"$'\n' ;; 284 | json*) REPLY="jqmd_data($1)"$'\n' ;; 285 | jq|javascript|js) REPLY=$1 ;; 286 | *) mdsh-error "Invalid language for function: '%s'" "${tag_words-}"; return 287 | esac 288 | printf 'function %s() {\n\tAPPLY %q \\\n\t\t%s\n}\n' "${tag_words[2]}" "$REPLY" \ 289 | "${2#*${tag_words}*${tag_words[1]}*${tag_words[2]}}" 290 | } 291 | 292 | const() { 293 | case "${mdsh_lang-}" in 294 | yaml|yml) y2j "$mdsh_block"; printf 'DEFINE %q\n' "def $1: $REPLY ;"$'\n' ;; 295 | json) printf 'DEFINE %q\n' "def $1: $mdsh_block ;"$'\n' ;; 296 | *) mdsh-error "Invalid language for constant: '%s'" "${mdsh_lang-}" 297 | esac 298 | } 299 | # Load the runtime so it's usable by mdsh 300 | printf -v REPLY '%s\n' "${mdsh_raw_bash_runtime[@]}"; eval "$REPLY" 301 | 302 | # Add runtime to the top of compiled (main) scripts 303 | printf -v REPLY 'mdsh:file-header() { ! @is-main || echo -n %q; }' "$REPLY"; eval "$REPLY" 304 | 305 | # Ensure (main) scripts process any leftover filters at end 306 | mdsh:file-footer() { ! @is-main || echo $'if [[ $0 == "${BASH_SOURCE[0]-}" ]] && HAVE_FILTERS; then RUN_JQ; fi'; } 307 | mdsh.--no-runtime() ( unset -f mdsh:file-header mdsh:file-footer; mdsh-main "$@"; ) 308 | mdsh.--yaml() ( 309 | fn-exists "yaml2json:${1-}" || exit $? "No such yaml2json processor: ${1-}" 310 | eval 'yaml2json() { yaml2json:'"$1"'; }' 311 | mdsh-main "${@:2}" 312 | ) 313 | 314 | mdsh.-R() { mdsh.--no-runtime "$@"; } 315 | mdsh.-y() { mdsh.--yaml "$@"; } 316 | if [[ $0 == "${BASH_SOURCE-}" ]]; then mdsh-main "$@"; exit; fi 317 | -------------------------------------------------------------------------------- /jqmd.md: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | : ' 3 | 4 | 5 | # jqmd - literate jq programming 6 | 7 | jqmd is an mdsh extension written as a literate program using mdsh. Within this source file, `shell` code blocks are the main program, while `bash runtime` blocks are captured as data to be used as part of the runtime compiled into jqmd programs. 8 | 9 | ### Contents 10 | 11 | 12 | 13 | - [File Header](#file-header) 14 | - [Runtime](#runtime) 15 | * [jq imports, filters, and defines](#jq-imports-filters-and-defines) 16 | * [JSON Data Structures](#json-data-structures) 17 | * [jq options and arguments](#jq-options-and-arguments) 18 | * [Invoking jq](#invoking-jq) 19 | * [YAML and JSON data](#yaml-and-json-data) 20 | + [jqmd::data](#jqmddata) 21 | + [YAML Interpolation](#yaml-interpolation) 22 | + [YAML to JSON Conversion (yaml2json)](#yaml-to-json-conversion-yaml2json) 23 | - [Main Program](#main-program) 24 | 25 | 26 | 27 | ## File Header 28 | 29 | The main program begins with a `#!` line and edit warning, followed by its license text, and the source of mdsh: 30 | 31 | ```shell mdsh 32 | @module jqmd.md 33 | @main mdsh-main 34 | 35 | @require pjeby/license @comment LICENSE 36 | @require bashup/mdsh mdsh-source "$BASHER_PACKAGES_PATH/bashup/mdsh/mdsh.md" 37 | ``` 38 | 39 | ## Runtime 40 | 41 | The runtime also begins with a `#!` line, so that compiled scripts will begin with one too: 42 | 43 | ```bash runtime 44 | #!/usr/bin/env bash 45 | 46 | # --- BEGIN jqmd runtime --- 47 | ``` 48 | 49 | ### jq imports, filters, and defines 50 | 51 | ```bash runtime 52 | jqmd_imports= 53 | jqmd_filters= 54 | jqmd_defines= 55 | 56 | HAVE_FILTERS() { [[ ${jqmd_filters-} ]]; } 57 | CLEAR_FILTERS() { unset jqmd_filters; JQ_OPTS=(jq); } 58 | 59 | IMPORTS() { jqmd_imports+="${jqmd_imports:+$'\n'}$1"; } 60 | DEFINE() { jqmd_defines+="${jqmd_defines:+$'\n'}$1"; } 61 | FILTER() { 62 | case $# in 63 | 1) jqmd_filters+="${jqmd_filters:+$'\n'| }$1"; return ;; 64 | 0) return ;; 65 | esac 66 | local REPLY ARGS=(printf -v REPLY "$1"); shift 67 | JSON-QUOTE "$@"; ARGS+=("${REPLY[@]}"); "${ARGS[@]}"; FILTER "$REPLY" 68 | } 69 | 70 | APPLY() { 71 | local name filter='' REPLY expr=${1-} lf=$'\n'; shift 72 | while (($#)); do 73 | name=${1#@}; REPLY=${name#*=} 74 | [[ $name == *=* ]] || REPLY=${!name} 75 | if ((${#REPLY} > 32)); then 76 | if [[ $1 == @* ]]; then ARGVAL "$REPLY"; else ARGSTR "$REPLY"; fi 77 | else 78 | JSON-QUOTE "$REPLY"; [[ $1 != @* ]] || REPLY="($REPLY|fromjson)" 79 | fi 80 | filter+=" | $REPLY as \$${name%%=*}"; shift 81 | done 82 | filter=${filter:3}; [[ ! $expr || $expr == . ]] || filter="( ${filter:+$filter | }$expr )" 83 | ${filter:+FILTER "$filter"} 84 | } 85 | ``` 86 | 87 | ### JSON Data Structures 88 | 89 | ```bash runtime 90 | JSON-QUOTE() { 91 | local LC_ALL=C 92 | [[ $* != *\\* ]] || set -- "${@//\\/\\\\}" 93 | [[ $* != *'"'* ]] || set -- "${@//\"/\\\"}" 94 | set -- "${@/#/\"}"; set -- "${@/%/\"}" 95 | if [[ $* != *[$'\x01'-$'\x1F']* ]]; then REPLY=("$@"); else escape-ctrlchars "$@"; fi 96 | } 97 | 98 | JSON-LIST() { 99 | (($#)) || { REPLY=("[]"); return; } 100 | JSON-QUOTE "$@"; REPLY=${REPLY[*]/%/,}; REPLY=("[${REPLY%,}]") 101 | } 102 | 103 | JSON-MAP() { 104 | local -n v=$1; local LC_ALL=C n=${v[@]+${#v[@]}}; ((${n:-0})) || { REPLY=("{}"); return; } 105 | local i=1 p='\"${@:i:1}\": \"${@: '"$n"'+ i++:1}\",'; p="${v[*]/#*/$p}" 106 | set -- "${!v[@]}" "${v[@]}" 107 | [[ $* != *\\* ]] || set -- "${@//\\/\\\\}" 108 | [[ $* != *\"* ]] || set -- "${@//\"/\\\"}" 109 | eval "REPLY=(\"{${p%,}}\")" 110 | [[ $REPLY != *[$'\x01'-$'\x1F']* ]] || escape-ctrlchars "$REPLY" 111 | } 112 | 113 | JSON-KV() { 114 | (($#)) || { REPLY=("{}"); return; } 115 | local LC_ALL=C i=0 p='\"${REPLY[i]}\": \"${REPLY['"$#"'+i++]}\",'; p="${*/#*/$p}" 116 | [[ $* != *\\* ]] || set -- "${@//\\/\\\\}" 117 | [[ $* != *\"* ]] || set -- "${@//\"/\\\"}" 118 | REPLY=("${@%%=*}" "${@#*=}"); eval "REPLY=(\"{${p%,}}\")" 119 | [[ $REPLY != *[$'\x01'-$'\x1F']* ]] || escape-ctrlchars "$REPLY" 120 | } 121 | 122 | escape-ctrlchars() { 123 | local LC_ALL=C 124 | set -- "${@//$'\n'/\\n}"; set -- "${@//$'\r'/\\r}"; set -- "${@//$'\t'/\\t}" # \n\r\t 125 | if [[ $* == *[$'\x01'-$'\x1F']* ]]; then 126 | local r s=$*; s=${s//[^$'\x01'-$'\x1F']} 127 | while [[ $s ]]; do 128 | printf -v r \\\\u%04x "'${s:0:1}"; set -- "${@//${s:0:1}/$r}" 129 | s=${s//${s:0:1}/} 130 | done 131 | fi 132 | REPLY=("$@") 133 | } 134 | 135 | ``` 136 | 137 | ### jq options and arguments 138 | 139 | ```bash runtime 140 | JQ_CMD=(jq) 141 | JQ_OPTS=(jq) 142 | JQ_SIZE_LIMIT=16000 143 | JQ_OPTS() { JQ_OPTS+=("$@"); } 144 | ARG() { JQ_OPTS --arg "$1" "$2"; } 145 | ARGJSON() { JQ_OPTS --argjson "$1" "$2"; } 146 | ARGQUOTE() { ARGSTR "$1"; } # deprecated 147 | ARGSTR() { REPLY=JQMD_QA_${#JQ_OPTS[@]}; ARG "$REPLY" "$1"; REPLY='$'$REPLY; } 148 | ARGVAL() { REPLY=JQMD_JA_${#JQ_OPTS[@]}; ARGJSON "$REPLY" "$1"; REPLY='$'$REPLY; } 149 | ``` 150 | 151 | ### Invoking jq 152 | 153 | ```bash runtime 154 | JQ_CMD() { 155 | local f= opt nargs cmd=("${JQ_CMD[@]}"); set -- "${JQ_OPTS[@]:1}" "$@" 156 | 157 | while (($#)); do 158 | case "$1" in 159 | -f|--fromfile) 160 | opt=$(<"$2") || return 69 161 | FILTER "$opt"; shift 2; continue 162 | ;; 163 | -L|--indent) nargs=2 ;; 164 | --arg|--argjson|--slurpfile|--argfile) nargs=3 ;; 165 | --) break ;; # rest of args are data files 166 | -*) nargs=1 ;; 167 | *) FILTER "$1"; break ;; # jq program: data files follow 168 | esac 169 | cmd+=("${@:1:$nargs}") # add $nargs args to cmd 170 | shift $nargs 171 | done 172 | 173 | HAVE_FILTERS || FILTER . # jq needs at least one filter expression 174 | for REPLY in "${jqmd_imports-}" "${jqmd_defines-}" "${jqmd_filters-}"; do 175 | [[ $REPLY ]] && f+=${f:+$'\n'}$REPLY 176 | done 177 | 178 | REPLY=("${cmd[@]}" "$f" "${@:2}") 179 | if ((${#f} > JQ_SIZE_LIMIT)); then REPLY=(JQ_WITH_FILE "${#cmd[@]}" "${REPLY[@]}"); fi 180 | CLEAR_FILTERS # cleanup for any re-runs 181 | } 182 | 183 | JQ_WITH_FILE() { local n=$(($1++2)); "${@:2:$1}" -f <(echo "${!n}") "${@:n+1}"; } 184 | RUN_JQ() { JQ_CMD "$@" && "${REPLY[@]}"; } 185 | CALL_JQ() { JQ_CMD "$@" && REPLY=("$("${REPLY[@]}")"); } 186 | ``` 187 | 188 | ### YAML and JSON data 189 | 190 | YAML and JSON blocks are just jq filter expressions wrapped in a call to `jqmd::data()` (or an appropriate alternative, selected by a `@data` directive in a mdsh block). 191 | 192 | ```bash runtime 193 | YAML() { y2j "$1"; JSON "$REPLY"; } 194 | JSON() { FILTER "jqmd_data($1)" "${@:2}"; } 195 | ``` 196 | 197 | #### jqmd::data 198 | 199 | The `jqmd::data` function provides a default wrapper to convert YAML or JSON data to a jq filter. It recursively merges dictionaries and concatenates (or appends to) arrays. 200 | 201 | ```bash runtime 202 | DEFINE ' 203 | def jqmd::blend($other; combine): . as $this | . * $other | . as $combined | with_entries( 204 | if (.key | in($this)) and (.key | in($other)) then 205 | .this = $this[.key] | .other = $other[.key] | combine 206 | else . end 207 | ); 208 | 209 | def jqmd::combine: (.this|type) as $this | (.other|type) as $other | .value = 210 | if $this == "array" then 211 | if $other == "array" then .this + .other else .this + [.other] end 212 | elif $this == "object" then 213 | if $other == "object" then 214 | .other as $o | (.this | jqmd::blend($o; jqmd::combine)) 215 | else .other end 216 | else .other end; # everything else just overrides 217 | 218 | def jqmd::data($data): {this: ., other:$data} | jqmd::combine | .value ; 219 | def jqmd_data($data): jqmd::data($data) ; 220 | ' 221 | ``` 222 | #### YAML Interpolation 223 | 224 | YAML blocks are allowed to have jq string interpolation expressions. But since such expressions are invalid JSON, and the yaml2json filter produces only valid JSON, it's necessary to selectively *invalidate* the resulting JSON before it becomes a JQ filter. Specifically, either one or two backslashes must be removed before every open parenthesis, depending on whether the total number of preceding backslashes is divisible by four (i.e., the original number of backslashes was divisible by two.) 225 | 226 | ```bash runtime 227 | y2j() { 228 | local p j; j="$(echo "$1" | yaml2json)" || return $?; REPLY= 229 | while [[ $j == *'\\('* ]]; do 230 | p=${j%%'\\('*}; j=${j#"$p"'\\('} 231 | if [[ $p =~ (^|[^\\])('\\\\')*$ ]]; then 232 | p="${p}"'\(' # odd, unbalance the backslash 233 | else 234 | p="${p}(" # even, remove one actual backslash 235 | fi 236 | REPLY+=$p 237 | done 238 | REPLY+=$j 239 | } 240 | ``` 241 | #### YAML to JSON Conversion (yaml2json) 242 | 243 | The `yaml2json` function is a wrapper that automatically selects one of `yaml2json:cmd`, `yaml2json:py`, or `yaml2json:php` to perform YAML to JSON conversions. It does so by piping a small YAML input to each function and testing whether the result is a valid JSON conversion. 244 | 245 | ```bash runtime 246 | yaml2json:cmd() { command yaml2json /dev/stdin; } 247 | 248 | yaml2json:py() { 249 | python -c 'import sys, yaml, json; json.dump(yaml.safe_load(sys.stdin), sys.stdout)' 250 | } 251 | 252 | yaml2json:php() { command yaml2json.php; } 253 | 254 | yaml2json() { 255 | local kind # auto-select between available yaml2json implementations 256 | for kind in cmd py php; do 257 | REPLY=($(yaml2json:$kind < <(echo "a: {}") 2>/dev/null || true)) 258 | printf -v REPLY %s ${REPLY+"${REPLY[@]}"} 259 | if [[ "$REPLY" == '{"a":{}}' ]]; then 260 | eval "yaml2json() { yaml2json:$kind; }"; yaml2json; return 261 | fi 262 | done 263 | exit 69 "To process YAML, must have one of: yaml2json, PyYAML, or yaml2json.php" # EX_UNAVAILABLE 264 | } 265 | 266 | # --- END jqmd runtime --- 267 | ``` 268 | 269 | ## Main Program 270 | 271 | The non-runtime part of the program defines hooks for mdsh to be able to compile jq, yaml, and json blocks and constants: 272 | 273 | ```shell 274 | # Language Support 275 | mdsh-compile-jq() { printf 'FILTER %q\n' "$1"$'\n'; } 276 | mdsh-compile-jq_defs() { printf 'DEFINE %q\n' "$1"$'\n'; } 277 | mdsh-compile-jq_imports() { printf 'IMPORTS %q\n' "$1"$'\n'; } 278 | 279 | mdsh-compile-yml() { y2j "$1"; mdsh-compile-json "$REPLY"; } 280 | mdsh-compile-yaml() { y2j "$1"; mdsh-compile-json "$REPLY"; } 281 | mdsh-compile-json() { mdsh-compile-jq "jqmd_data($1)"; } 282 | 283 | mdsh-compile-func() { 284 | case ${tag_words-} in 285 | yml|yaml) y2j "$1"; REPLY="jqmd_data($REPLY)"$'\n' ;; 286 | json*) REPLY="jqmd_data($1)"$'\n' ;; 287 | jq|javascript|js) REPLY=$1 ;; 288 | *) mdsh-error "Invalid language for function: '%s'" "${tag_words-}"; return 289 | esac 290 | printf 'function %s() {\n\tAPPLY %q \\\n\t\t%s\n}\n' "${tag_words[2]}" "$REPLY" \ 291 | "${2#*${tag_words}*${tag_words[1]}*${tag_words[2]}}" 292 | } 293 | 294 | const() { 295 | case "${mdsh_lang-}" in 296 | yaml|yml) y2j "$mdsh_block"; printf 'DEFINE %q\n' "def $1: $REPLY ;"$'\n' ;; 297 | json) printf 'DEFINE %q\n' "def $1: $mdsh_block ;"$'\n' ;; 298 | *) mdsh-error "Invalid language for constant: '%s'" "${mdsh_lang-}" 299 | esac 300 | } 301 | ``` 302 | 303 | It also evals the runtime, and defines header/footer hooks to embed the runtime in compiled scripts, and to ensure jq is run at the script's end if there are any leftover filters: 304 | 305 | ```shell 306 | # Load the runtime so it's usable by mdsh 307 | printf -v REPLY '%s\n' "${mdsh_raw_bash_runtime[@]}"; eval "$REPLY" 308 | 309 | # Add runtime to the top of compiled (main) scripts 310 | printf -v REPLY 'mdsh:file-header() { ! @is-main || echo -n %q; }' "$REPLY"; eval "$REPLY" 311 | 312 | # Ensure (main) scripts process any leftover filters at end 313 | mdsh:file-footer() { ! @is-main || echo $'if [[ $0 == "${BASH_SOURCE[0]-}" ]] && HAVE_FILTERS; then RUN_JQ; fi'; } 314 | ``` 315 | 316 | It also defines a few command line options for controlling compilation: 317 | 318 | ```shell 319 | mdsh.--no-runtime() ( unset -f mdsh:file-header mdsh:file-footer; mdsh-main "$@"; ) 320 | mdsh.--yaml() ( 321 | fn-exists "yaml2json:${1-}" || exit $? "No such yaml2json processor: ${1-}" 322 | eval 'yaml2json() { yaml2json:'"$1"'; }' 323 | mdsh-main "${@:2}" 324 | ) 325 | 326 | mdsh.-R() { mdsh.--no-runtime "$@"; } 327 | mdsh.-y() { mdsh.--yaml "$@"; } 328 | ``` 329 | -------------------------------------------------------------------------------- /package.sh: -------------------------------------------------------------------------------- 1 | BINS=bin/jqmd 2 | -------------------------------------------------------------------------------- /script/README.md: -------------------------------------------------------------------------------- 1 | ## Scripts To Rule Them All 2 | 3 | The scripts in this directory are a [Scripts To Rule Them All](https://githubengineering.com/scripts-to-rule-them-all/) implementation, powered by [.devkit](https://github.com/bashup/.devkit). They should be run from within the project's root directory, using e.g. `script/test` to run tests, and so on. 4 | 5 | Please check the containing project's documentation for more details, or see the preceding links for more background or reference information. 6 | 7 | -------------------------------------------------------------------------------- /script/bootstrap: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | if ! [[ -f .dkrc ]]; then 4 | echo "Please run this script from the project root directory." >&2 5 | exit 64 # EX_USAGE 6 | fi 7 | 8 | if ! [[ -d .devkit ]]; then 9 | # Modify this line if you want to pin a particular .devkit revision: 10 | git clone -q --depth 1 https://github.com/bashup/.devkit 11 | fi 12 | 13 | exec ".devkit/dk" "$(basename "$0")" "$@" 14 | -------------------------------------------------------------------------------- /script/cibuild: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/clean: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/console: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/server: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/setup: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/test: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /script/update: -------------------------------------------------------------------------------- 1 | bootstrap -------------------------------------------------------------------------------- /specs/Data.cram.md: -------------------------------------------------------------------------------- 1 | ## Data merging 2 | 3 | ~~~shell 4 | $ source "$TESTDIR/../jqmd.md"; set +e 5 | 6 | # Null + any = any 7 | 8 | $ JSON '1'; RUN_JQ -n 9 | 1 10 | $ JSON '"x"'; RUN_JQ -n 11 | "x" 12 | $ JSON 'true'; RUN_JQ -n 13 | true 14 | $ JSON '{x:17}'; RUN_JQ -n -c 15 | {"x":17} 16 | $ JSON '[2]'; RUN_JQ -n -c 17 | [2] 18 | 19 | # Simple mappings 20 | $ JSON '{a:42, b:5}' 21 | $ JSON '{c:43, b:10}' 22 | $ RUN_JQ -n 23 | { 24 | "a": 42, 25 | "b": 10, 26 | "c": 43 27 | } 28 | 29 | # Nested mappings 30 | 31 | $ JSON '{x: {a: 42, b: 5}, y: "z"}' 32 | $ JSON '{q: {z: 19}, x: {b: 16, c: 27}}' 33 | $ RUN_JQ -n -c 34 | {"x":{"a":42,"b":16,"c":27},"y":"z","q":{"z":19}} 35 | 36 | # Array + Array = concatenation 37 | 38 | $ JSON '[1,2]' 39 | $ JSON '[3,4]' 40 | $ RUN_JQ -n -c 41 | [1,2,3,4] 42 | 43 | # Array + non-array = append 44 | $ JSON '[]' 45 | $ JSON '{}' 46 | $ RUN_JQ -n -c 47 | [{}] 48 | 49 | # Arrays inside mappings 50 | $ JSON '{a: [1,2]}' 51 | $ JSON '{a: [3]}' 52 | $ RUN_JQ -n -c 53 | {"a":[1,2,3]} 54 | ~~~ 55 | -------------------------------------------------------------------------------- /specs/JQ.cram.md: -------------------------------------------------------------------------------- 1 | ## Running JQ and Managing Filters+Options 2 | 3 | ### Filter/Option State 4 | 5 | Initially, there are no filters or options set: 6 | 7 | ````sh 8 | $ source "$TESTDIR/../jqmd.md"; set +e 9 | 10 | $ HAVE_FILTERS || echo nope 11 | nope 12 | 13 | $ echo "${JQ_OPTS[@]}" 14 | jq 15 | ```` 16 | 17 | But they can be added using `FILTER`, `JQ_OPTS`, `ARG`, and `ARGJSON`: 18 | 19 | ````sh 20 | $ FILTER '.' 21 | $ HAVE_FILTERS && echo yep 22 | yep 23 | $ ARG x "foo" 24 | $ ARGJSON y '{}' 25 | $ JQ_OPTS --slurpfile bar baz 26 | $ echo "${JQ_OPTS[@]}" 27 | jq --arg x foo --argjson y {} --slurpfile bar baz 28 | ```` 29 | 30 | And then reset with `CLEAR_FILTERS`: 31 | 32 | ````sh 33 | $ CLEAR_FILTERS 34 | $ HAVE_FILTERS || echo nope 35 | nope 36 | $ echo "${JQ_OPTS[@]}" 37 | jq 38 | ```` 39 | 40 | You can generate args with `ARGSTR` and `ARGVAL`: 41 | 42 | ~~~shell 43 | $ ARGSTR 'foo"bar'; echo $REPLY 44 | $JQMD_QA_1 45 | $ ARGSTR spammity; echo $REPLY 46 | $JQMD_QA_4 47 | $ ARGVAL true; echo $REPLY 48 | $JQMD_JA_7 49 | $ printf ' %q' "${JQ_OPTS[@]}"; echo 50 | jq --arg JQMD_QA_1 foo\"bar --arg JQMD_QA_4 spammity --argjson JQMD_JA_7 true 51 | $ RUN_JQ -n '$JQMD_JA_7' 52 | true 53 | ~~~ 54 | 55 | JSON quoting can be done with `JSON-QUOTE`, `JSON-LIST`, and `JSON-KV`, or via extra args to `FILTER` or `JSON`: 56 | 57 | ~~~shell 58 | $ JSON-QUOTE $'\n\r\t' $'\x01\x1f' "\\"; echo "${REPLY[@]}" 59 | "\n\r\t" "\u0001\u001f" "\\" 60 | 61 | $ JSON-LIST; echo "${REPLY[@]}" 62 | [] 63 | 64 | $ JSON-LIST foo bar baz; echo "${REPLY[@]}" 65 | ["foo", "bar", "baz"] 66 | 67 | $ JSON-KV; echo "${REPLY[@]}" 68 | {} 69 | 70 | $ JSON-KV foo=blue bar=baz spam; echo "${REPLY[@]}" 71 | {"foo": "blue", "bar": "baz", "spam": "spam"} 72 | 73 | $ FILTER 'foo(%s; %s)' bar 'baz"spam'; echo "$jqmd_filters" 74 | foo("bar"; "baz\"spam") 75 | 76 | $ CLEAR_FILTERS 77 | $ JSON '{%s: %s}' foo bar; echo "$jqmd_filters" 78 | jqmd_data({"foo": "bar"}) 79 | ~~~ 80 | 81 | If you're on bash 4, you can also `JSON-MAP` an associative array to a JSON object: 82 | 83 | ~~~shell 84 | $ if (( BASH_VERSINFO > 3 )); then 85 | > declare -A empty_map mymap=([foo]=42 [bar]=baz [bing]=boom) 86 | > JSON-MAP mymap; echo "${REPLY[@]}" 87 | > JSON-MAP emptymap; echo "${REPLY[@]}" 88 | > else # fake it for bash 3 89 | > echo '{"bing": "boom", "bar": "baz", "foo": "42"}' 90 | > echo '{}' 91 | > fi 92 | {"bing": "boom", "bar": "baz", "foo": "42"} 93 | {} 94 | ~~~ 95 | 96 | You can also `APPLY` a jq expression with jq variables bound to shell variables or values, either as strings or raw jq expressions: 97 | 98 | ~~~shell 99 | $ CLEAR_FILTERS 100 | $ foo='bar"baz' bar='27' 101 | 102 | $ APPLY . foo; echo "$jqmd_filters"; CLEAR_FILTERS 103 | "bar\"baz" as $foo 104 | 105 | $ APPLY 'bang($bar)'; echo "$jqmd_filters"; CLEAR_FILTERS 106 | ( bang($bar) ) 107 | 108 | $ APPLY 'bang($bar)' @bar; echo "$jqmd_filters"; CLEAR_FILTERS 109 | ( ("27"|fromjson) as $bar | bang($bar) ) 110 | 111 | $ APPLY . foo @bar baz=thingy @spam=54; echo "$jqmd_filters"; CLEAR_FILTERS 112 | "bar\"baz" as $foo | ("27"|fromjson) as $bar | "thingy" as $baz | ("54"|fromjson) as $spam 113 | 114 | # Longer values get passed as arguments instead of quoted: 115 | 116 | $ APPLY . foo="This is a relatively long string. It should be passed as an argument." 117 | $ APPLY . @bar='"This is a rather long JSON value. It should be passed as an argument."' 118 | $ echo "$jqmd_filters" 119 | $JQMD_QA_1 as $foo 120 | | $JQMD_JA_4 as $bar 121 | 122 | ~~~ 123 | 124 | ### Invoking JQ 125 | 126 | The `JQ_CMD` function adds the supplied args to `$JQOPTS` and combines them with the current imports, defines, and filters to generate a command line in `${REPLY[@]}`. It also resets the current filters and options. 127 | 128 | ````sh 129 | $ JQ() { JQ_CMD "$@" || return; printf ' %q' "${REPLY[@]}"; echo; } 130 | $ jqmd_defines= 131 | $ CLEAR_FILTERS 132 | 133 | # No filters specified, default to '.' 134 | 135 | $ JQ 136 | jq . 137 | $ JQ -c 138 | jq -c . 139 | $ JQ --slurpfile bar baz -L foo -- spim spam 140 | jq --slurpfile bar baz -L foo . spim spam 141 | 142 | # Filters given, use those: 143 | 144 | $ JQ .foo /bar # data file(s) after filter 145 | jq .foo /bar 146 | $ JQ --argfile x y --indent 3 .bar 147 | jq --argfile x y --indent 3 .bar 148 | $ JQ -c .[].baz 149 | jq -c .\[\].baz 150 | 151 | $ FILTER '.spam'; JQ_OPTS -L foo 152 | $ JQ 153 | jq -L foo .spam 154 | $ JQ -c # filter and opts are reset: 155 | jq -c . 156 | 157 | # Imports and defines go before filters, newline-separated: 158 | 159 | $ FILTER '.spam' 160 | $ DEFINE 'def x: 1;' 161 | $ IMPORTS 'import foo;' 162 | $ JQ -c 163 | jq -c $'import foo;\ndef x: 1;\n.spam' 164 | $ jqmd_defines= 165 | $ jqmd_imports= 166 | 167 | # Filters given by `-f` or `--fromfile` get read into the command line as part of the filter: 168 | 169 | $ echo '.filter' >foo.jq 170 | $ echo '.thing' >bar.jq 171 | $ JQ -f foo.jq -c --fromfile bar.jq 172 | jq -c $'.filter\n| .thing' 173 | 174 | # Unless the file doesn't exist or can't be read: 175 | $ JQ -f nosuch.file 176 | */jqmd.md: line *: nosuch.file: No such file or directory (glob) 177 | [69] 178 | 179 | # But if the total filter size is too long, it will use an inline file: 180 | $ JQ_SIZE_LIMIT=10 JQ -f foo.jq -c --fromfile bar.jq 181 | JQ_WITH_FILE 2 jq -c $'.filter\n| .thing' 182 | $ (function jq(){ echo jq "$@"; cat "$3"; }; JQ_WITH_FILE 2 jq -c $'.filter\n| .thing' x y z; ) 183 | jq -c -f /dev/fd/63 x y z 184 | .filter 185 | | .thing 186 | ```` 187 | 188 | ### Capturing JQ Output 189 | 190 | The `CALL_JQ` command is just like `RUN_JQ`, except that it captures the output in `REPLY`, while still clearing filters and such. It also returns jq's exit status. 191 | 192 | ````sh 193 | # No filter 194 | 195 | $ CALL_JQ -- <(echo '{}') 196 | $ echo "$REPLY" 197 | {} 198 | 199 | # With filter 200 | 201 | $ CALL_JQ .foo-2 <(echo '{"foo":42}') 202 | $ echo "$REPLY" 203 | 40 204 | 205 | # Exit levels 206 | $ CALL_JQ -n -e '[] | .[]' # force exit level 4 207 | [4] 208 | ```` 209 | 210 | -------------------------------------------------------------------------------- /specs/Smoke-Test.cram.md: -------------------------------------------------------------------------------- 1 | ## Basic check that jqmd handles jq and json blocks, const, JQ_OPTS, post-run RUN_JQ, etc.: 2 | 3 | $ cat <<'-' >test.md 4 | > ```json 5 | > { "x": "y" } 6 | > ``` 7 | > ```yaml ! const foo 8 | > a: \(env.FOO) - b 9 | > ``` 10 | > ```yaml @func mksite SITE="$1" WP_HOME="$2" 11 | > services: 12 | > \($SITE): 13 | > environment: 14 | > WP_HOME: \($WP_HOME) 15 | > ``` 16 | > ```shell 17 | > JQ_OPTS -n '. + foo' 18 | > mksite dev https://bashup.github.io/whatever 19 | > ``` 20 | > - 21 | $ FOO=bar $TESTDIR/../jqmd.md test.md 22 | { 23 | "x": "y", 24 | "services": { 25 | "dev": { 26 | "environment": { 27 | "WP_HOME": "https://bashup.github.io/whatever" 28 | } 29 | } 30 | }, 31 | "a": "bar - b" 32 | } 33 | -------------------------------------------------------------------------------- /specs/y2j.cram.md: -------------------------------------------------------------------------------- 1 | ## YAML Interpolation 2 | 3 | jqmd supports jq interpolation in YAML blocks, which requires changing the number of backslashes found before an open parenthesis. Since YAML escapes *all* backslashes in the input (to create valid JSON), this requires dealing with everything as doubled backslashes. In the case where the original number of backslashes was odd, a single backslash is removed, making the JSON invalid (and therefore suitable for a jq expression). In the case where the original number was even, *two* backslashes are removed, which removes one of the original backslashes (that was escaping the interpolation). 4 | 5 | So this YAML: 6 | 7 | ```yaml 8 | ultimate answer: \(40+2) 9 | not \\(a key: \\\\(or value 10 | ``` 11 | 12 | Should produce this output: 13 | 14 | ~~~shell 15 | $ "$TESTDIR/../jqmd.md" "$TESTDIR/$TESTFILE"