├── .gitignore ├── .npmignore ├── LICENSE.txt ├── README.md ├── expression.js ├── ndjson-cat ├── ndjson-filter ├── ndjson-join ├── ndjson-map ├── ndjson-reduce ├── ndjson-sort ├── ndjson-split ├── ndjson-top ├── output.js ├── package-lock.json ├── package.json ├── requires.js ├── resolve.js └── test ├── a.csv ├── b.csv ├── c.csv └── numbers.ndjson /.gitignore: -------------------------------------------------------------------------------- 1 | *.sublime-workspace 2 | .DS_Store 3 | node_modules 4 | npm-debug.log 5 | -------------------------------------------------------------------------------- /.npmignore: -------------------------------------------------------------------------------- 1 | *.sublime-* 2 | test/ 3 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2016, Michael Bostock 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * The name Michael Bostock may not be used to endorse or promote products 15 | derived from this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 20 | DISCLAIMED. IN NO EVENT SHALL MICHAEL BOSTOCK BE LIABLE FOR ANY DIRECT, 21 | INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 22 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY 24 | OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 25 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, 26 | EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ndjson-cli 2 | 3 | Unix-y tools for operating on [newline-delimited JSON](http://ndjson.org) streams. 4 | 5 | ``` 6 | npm install ndjson-cli 7 | ``` 8 | 9 | ## Command Line Reference 10 | 11 | * [Options](#options) 12 | * [ndjson-cat](#cat) - concatenate values to form a stream 13 | * [ndjson-filter](#filter) - filter values 14 | * [ndjson-map](#map) - transform values 15 | * [ndjson-reduce](#reduce) - reduce a stream of values to a single value 16 | * [ndjson-split](#split) - transform values to streams of values 17 | * [ndjson-join](#join) - join two streams of values into a single stream 18 | * [ndjson-sort](#sort) - sort a stream of values 19 | * [ndjson-top](#top) - select the top values from a stream 20 | 21 | ### Options 22 | 23 | All ndjson-cli commands support [--help](#_help) and [--version](#_version). Commands that take an expression also support [--require](#_require). 24 | 25 | # ndjson-command -h 26 |
# ndjson-command --help 27 | 28 | Output usage information. 29 | 30 | # ndjson-command -V 31 |
# ndjson-command --version 32 | 33 | Output the version number. 34 | 35 | # ndjson-command -r [name=]module 36 |
# ndjson-command --require [name=]module 37 | 38 | Requires the specified *module*, making it available for use in any expressions used by this command. The loaded module is available as the symbol *name*. If *name* is not specified, it defaults to *module*. (If *module* is not a valid identifier, you must specify a *name*.) For example, to [sort](#ndjson_sort) using [d3.ascending](https://github.com/d3/d3-array/blob/master/README.md#ascending) from [d3-array](https://github.com/d3/d3-array): 39 | 40 | ``` 41 | ndjson-sort -r d3=d3-array 'd3.ascending(a, b)' < values.ndjson 42 | ``` 43 | 44 | Or to require all of [d3](https://github.com/d3/d3): 45 | 46 | ``` 47 | ndjson-sort -r d3 'd3.ascending(a, b)' < values.ndjson 48 | ``` 49 | 50 | The required *module* is resolved relative to the [current working directory](https://nodejs.org/api/process.html#process_process_cwd). If the *module* is not found during normal resolution, the [global root](https://docs.npmjs.com/cli/root) is also searched, allowing you to require global modules from the command line. 51 | 52 | ### cat 53 | 54 | # ndjson-cat [files…] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-cat "Source") 55 | 56 | Sequentially concatenates one or more input *files* containing JSON into a single newline-delimited JSON on stdout. If *files* is not specified, it defaults to “-”, indicating stdin. This command is especially useful for converting pretty-printed JSON (that contains newlines) into newline-delimited JSON. For example, to print the binaries exported by this repository’s package.json: 57 | 58 | ``` 59 | ndjson-cat package.json | ndjson-split 'Object.keys(d.bin)' 60 | ``` 61 | 62 | ### filter 63 | 64 | # ndjson-filter [expression] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-filter "Source") 65 | 66 | Filters the newline-delimited JSON stream on stdin according to the specified *expression*: if the *expression* evaluates truthily for the given value *d* at the given zero-based index *i* in the stream, the resulting value is output to stdout; otherwise, it is ignored. If *expression* is not specified, it defaults to `true`. This program is much like [*array*.filter](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter). 67 | 68 | For example, given a stream of GeoJSON features from [shp2json](https://github.com/mbostock/shapefile/blob/master/README.md#shp2json), you can filter the stream to include only the multi-polygon features like so: 69 | 70 | ``` 71 | shp2json -n example.shp | ndjson-filter 'd.geometry.type === "MultiPolygon"' 72 | ``` 73 | 74 | Or, to skip every other feature: 75 | 76 | ``` 77 | shp2json -n example.shp | ndjson-filter 'i & 1' 78 | ``` 79 | 80 | Or to take a random 10% sample: 81 | 82 | ``` 83 | shp2json -n example.shp | ndjson-filter 'Math.random() < 0.1' 84 | ``` 85 | 86 | Side-effects during filter are allowed. For example, to delete a property: 87 | 88 | ``` 89 | shp2json -n example.shp | ndjson-filter 'delete d.properties.FID, true' 90 | ``` 91 | 92 | ### map 93 | 94 | # ndjson-map [expression] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-map "Source") 95 | 96 | Maps the newline-delimited JSON stream on stdin according to the specified *expression*: outputs the result of evaluating the *expression* for the given value *d* at the given zero-based index *i* in the stream. If *expression* is not specified, it defaults to `d`. This program is much like [*array*.map](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map). 97 | 98 | For example, given a stream of GeoJSON features from [shp2json](https://github.com/mbostock/shapefile/blob/master/README.md#shp2json), you can convert the stream to geometries like so: 99 | 100 | ``` 101 | shp2json -n example.shp | ndjson-map 'd.geometry' 102 | ``` 103 | 104 | Or you can extract the properties, and then convert to [tab-separated values](https://github.com/d3/d3-dsv): 105 | 106 | ``` 107 | shp2json -n example.shp | ndjson-map 'd.properties' | json2tsv -n > example.tsv 108 | ``` 109 | 110 | You can also add new properties to each object by assigning them and then returning the original object: 111 | 112 | ``` 113 | shp2json -n example.shp | ndjson-map 'd.properties.version = 2, d' | json2tsv -n > example.v2.tsv 114 | ``` 115 | 116 | ### reduce 117 | 118 | # ndjson-reduce [expression [initial]] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-reduce "Source") 119 | 120 | Reduces the newline-delimited JSON stream on stdin according to the specified *expression*. For each value in the input stream, evaluates the *expression* for the given value *d* at the given zero-based index *i* in the stream and the previous value *p*, which is initialized to *initial*. If *expression* and *initial* are not specified, they default to `p.push(d), p` and `[]`, respectively, merging all input values into a single array (the inverse of [ndjson-split](#ndjson_split)). Otherwise, if *initial* is not specified, the first time the *expression* is evaluated *p* will be equal to the first value in the stream (*i* = 0) and *d* will be equal to the second (*i* = 1). Outputs the last result when the stream ends. This program is much like [*array*.reduce](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce). 121 | 122 | For example, to count the number of values in a stream of GeoJSON features from [shp2json](https://github.com/mbostock/shapefile/blob/master/README.md#shp2json), like `wc -l`: 123 | 124 | ``` 125 | shp2json -n example.shp | ndjson-reduce 'p + 1' '0' 126 | ``` 127 | 128 | To merge a stream into a feature collection: 129 | 130 | ``` 131 | shp2json -n example.shp | ndjson-reduce 'p.features.push(d), p' '{type: "FeatureCollection", features: []}' 132 | ``` 133 | 134 | Or equivalently, in two steps: 135 | 136 | ``` 137 | shp2json -n example.shp | ndjson-reduce | ndjson-map '{type: "FeatureCollection", features: d}' 138 | ``` 139 | 140 | To convert a newline-delimited JSON stream of values to a JSON array, the inverse of [ndjson-split](#ndjson_split): 141 | 142 | ``` 143 | ndjson-reduce < values.ndjson > array.json 144 | ``` 145 | 146 | ### split 147 | 148 | # ndjson-split [expression] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-split "Source") 149 | 150 | Expands the newline-delimited JSON stream on stdin according to the specified *expression*: outputs the results of evaluating the *expression* for the given value *d* at the given zero-based index *i* in the stream. The result of evaluating the *expression* must be an array (though it may be the empty array if no values should be output for the given input). If *expression* is not specified, it defaults to `d`, which assumes that the input values are arrays. 151 | 152 | For example, given a single GeoJSON feature collection from [shp2json](https://github.com/mbostock/shapefile/blob/master/README.md#shp2json), you can convert a stream of features like so: 153 | 154 | ``` 155 | shp2json example.shp | ndjson-split 'd.features' 156 | ``` 157 | 158 | To convert a JSON array to a newline-delimited JSON stream of values, the inverse of [ndjson-reduce](#ndjson_reduce): 159 | 160 | ``` 161 | ndjson-split < array.json > values.ndjson 162 | ``` 163 | 164 | ### join 165 | 166 | # ndjson-join [expression₀ [expression₁]] file₀ file₁ [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-join "Source") 167 | 168 | [Joins](https://en.wikipedia.org/wiki/Join_\(SQL\)#Inner_join) the two newline-delimited JSON streams in *file₀* and *file₁* according to the specified key expressions *expression₀* and *expression₁*. For each value *d₀* at the zero-based index *i₀* in the stream *file₀*, the corresponding key is the result of evaluating the *expression₀*. Similarly, for each value *d₁* at the zero-based index *i₁* in the stream *file₁*, the corresponding key is the result of evaluating the *expression₁*. When both input streams end, for each distinct key, the cartesian product of corresponding values *d₀* and *d₁* are output as an array `[d₀, d₁]`. 169 | 170 | If *expression₀* is not specified, it defaults to `i`, joining the two streams by line number; in this case, the length of the output stream is the shorter of the two input streams. If *expression₁* is not specified, it defaults to *expression₀*. 171 | 172 | For example, consider the CSV file *a.csv*: 173 | 174 | ```csv 175 | name,color 176 | Fred,red 177 | Alice,green 178 | Bob,blue 179 | ``` 180 | 181 | And *b.csv*: 182 | 183 | ```csv 184 | name,number 185 | Fred,21 186 | Alice,42 187 | Bob,102 188 | ``` 189 | 190 | To merge these into a single stream by name using [csv2json](https://github.com/d3/d3-dsv/blob/master/README.md#command-line-reference): 191 | 192 | ``` 193 | ndjson-join 'd.name' <(csv2json -n a.csv) <(csv2json -n b.csv) 194 | ``` 195 | 196 | The resulting output is: 197 | 198 | ```json 199 | [{"name":"Fred","color":"red"},{"name":"Fred","number":"21"}] 200 | [{"name":"Alice","color":"green"},{"name":"Alice","number":"42"}] 201 | [{"name":"Bob","color":"blue"},{"name":"Bob","number":"102"}] 202 | ``` 203 | 204 | To consolidate the results into a single object, use [ndjson-map](#ndjson-map) and [Object.assign](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/assign): 205 | 206 | ``` 207 | ndjson-join 'd.name' <(csv2json -n a.csv) <(csv2json -n b.csv) | ndjson-map 'Object.assign(d[0], d[1])' 208 | ``` 209 | 210 | # ndjson-join --left
211 | # ndjson-join --right
212 | # ndjson-join --outer
213 | 214 | Specify the type of join: [left](https://en.wikipedia.org/wiki/Join_\(SQL\)#Left_outer_join), [right](https://en.wikipedia.org/wiki/Join_\(SQL\)#Right_outer_join), or [outer](https://en.wikipedia.org/wiki/Join_\(SQL\)#Full_outer_join). Empty values are output as `null`. If none of these arguments are specified, defaults to [inner](https://en.wikipedia.org/wiki/Join_\(SQL\)#Inner_join). 215 | 216 | ### sort 217 | 218 | # ndjson-sort [expression] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-sort "Source") 219 | 220 | Sorts the newline-delimited JSON stream on stdin according to the specified comparator *expression*. After reading the entire input stream, [sorts the array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort) of values with a comparator that evaluates the *expression* for two given values *a* and *b* from the input stream. If the resulting value is less than 0, then *a* appears before *b* in the output stream; if the value is greater than 0, then *a* appears after *b* in the output stream; any other value means that the partial order of *a* and *b* is undefined. If *expression* is not specified, it defaults to [ascending natural order](https://github.com/d3/d3-array/blob/master/src/ascending.js). 221 | 222 | For example, to sort a stream of GeoJSON features by their name property: 223 | 224 | ``` 225 | shp2json -n example.shp | ndjson-sort 'a.properties.name.localeCompare(b.properties.name)' 226 | ``` 227 | 228 | ### top 229 | 230 | # ndjson-top [expression] [<>](https://github.com/mbostock/ndjson-cli/blob/master/ndjson-top "Source") 231 | 232 | Selects the top *n* values (see [-n](#ndjson_top_n)) from the newline-delimited JSON stream on stdin according to the specified comparator *expression*. (This [selection algorithm](https://en.wikipedia.org/wiki/Selection_algorithm) is implemented using partial heap sort.) After reading the entire input stream, outputs the top *n* values in ascending order. As with [ndjson-sort](#ndjson_sort), the input values are compared by evaluating the *expression* for two given values *a* and *b* from the input stream. If the resulting value is less than 0, then *a* appears before *b* in the output stream; if the value is greater than 0, then *a* appears after *b* in the output stream; any other value means that the partial order of *a* and *b* is undefined. If *expression* is not specified, it defaults to [ascending natural order](https://github.com/d3/d3-array/blob/master/src/ascending.js). If the input stream has fewer than *n* values, this program is equivalent to [ndjson-sort](#ndjson_sort). 233 | 234 | For example, to output the GeoJSON feature with the largest size property: 235 | 236 | ``` 237 | shp2json -n example.shp | ndjson-top 'a.properties.size - b.properties.size' 238 | ``` 239 | 240 | This program is equivalent to `ndjson-sort expression | tail -n count`, except it is much more efficient if *n* is smaller than the size of the input stream. 241 | 242 | # ndjson-top -n count 243 | 244 | Specify the maximum number of output values. Defaults to 1. 245 | 246 | ## Recipes 247 | 248 | To count the number of values in a stream: 249 | 250 | ``` 251 | shp2json -n example.shp | wc -l 252 | ``` 253 | 254 | To reverse a stream: 255 | 256 | ``` 257 | shp2json -n example.shp | tail -r 258 | ``` 259 | 260 | To take the first 3 values in a stream: 261 | 262 | ``` 263 | shp2json -n example.shp | head -n 3 264 | ``` 265 | 266 | To take the last 3 values in a stream: 267 | 268 | ``` 269 | shp2json -n example.shp | tail -n 3 270 | ``` 271 | 272 | To take all but the first 3 values in a stream: 273 | 274 | ``` 275 | shp2json -n example.shp | tail -n +4 276 | ``` 277 | -------------------------------------------------------------------------------- /expression.js: -------------------------------------------------------------------------------- 1 | var acorn = require("acorn"), 2 | path = require("path"), 3 | vm = require("vm"); 4 | 5 | module.exports = function(value, name) { 6 | try { 7 | var node = acorn.parse("(" + value + ")", {preserveParens: true}); 8 | if (node.type !== "Program") throw new Error("Expected program"); 9 | if (node.body.length !== 1) throw new Error("Invalid expression"); 10 | if (node.body[0].type !== "ExpressionStatement") throw new Error("Expected expression statement"); 11 | if (node.body[0].expression.type !== "ParenthesizedExpression") throw new Error("Expected expression"); 12 | return new vm.Script("(" + value + ")"); 13 | } catch (error) { 14 | console.error(path.basename(process.argv[1]) + ":" + (name === undefined ? "expression" : name)); 15 | console.error(value); 16 | console.error("^"); 17 | console.error("SyntaxError: " + error.message); 18 | process.exit(1); 19 | } 20 | }; 21 | -------------------------------------------------------------------------------- /ndjson-cat: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var fs = require("fs"), 4 | commander = require("commander"), 5 | output = require("./output"); 6 | 7 | commander 8 | .version(require("./package.json").version) 9 | .usage("[options] [files…]") 10 | .description("Concatenate JSON values to form a newline-delimited JSON stream.") 11 | .parse(process.argv); 12 | 13 | if (commander.args.length < 1) { 14 | commander.args[0] = "-"; 15 | } 16 | 17 | var i = -1, n = commander.args.length, task = Promise.resolve(); 18 | while (++i < n) task = task.then(cat(commander.args[i])); 19 | task.catch(function(error) { console.error(error); }); 20 | 21 | function cat(input) { 22 | return new Promise(function(resolve, reject) { 23 | var data = [], stream = input === "-" ? process.stdin : fs.createReadStream(input); 24 | stream 25 | .on("data", function(d) { data.push(d); }) 26 | .on("error", reject) 27 | .on("end", function() { 28 | try { 29 | var d = JSON.parse(Buffer.concat(data)); 30 | } catch (e) { 31 | console.error((input === "-" ? "stdin" : input) + ": SyntaxError: " + e.message); 32 | process.exit(1); 33 | }; 34 | output(d); 35 | resolve(); 36 | }); 37 | }); 38 | } 39 | -------------------------------------------------------------------------------- /ndjson-filter: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression]") 13 | .description("Filter values in a newline-delimited JSON stream.") 14 | .option("-r, --require ", "require a module", requires, {d: undefined, i: -1}) 15 | .parse(process.argv); 16 | 17 | if (commander.args.length > 1) { 18 | commander.outputHelp(); 19 | process.exit(1); 20 | } 21 | 22 | if (commander.args.length < 1) { 23 | commander.args[0] = "true"; 24 | } 25 | 26 | var i = -1, 27 | sandbox = commander.require, 28 | filter = expression(commander.args[0]), 29 | context = new vm.createContext(sandbox); 30 | 31 | readline.createInterface({ 32 | input: process.stdin, 33 | output: null 34 | }).on("line", function(line) { 35 | sandbox.i = ++i; 36 | try { 37 | sandbox.d = JSON.parse(line); 38 | } catch (error) { 39 | console.error("stdin:" + (i + 1)); 40 | console.error(line); 41 | console.error("^"); 42 | console.error("SyntaxError: " + error.message); 43 | process.exit(1); 44 | } 45 | if (filter.runInContext(context)) { 46 | output(sandbox.d); 47 | } 48 | }); 49 | -------------------------------------------------------------------------------- /ndjson-join: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var fs = require("fs"), 4 | readline = require("readline"), 5 | vm = require("vm"), 6 | commander = require("commander"), 7 | requires = require("./requires"), 8 | expression = require("./expression"), 9 | output = require("./output"); 10 | 11 | commander 12 | .version(require("./package.json").version) 13 | .usage("[options] [expression₀ [expression₁]] ") 14 | .description("Join two newline-delimited JSON streams into a stream of pairs.") 15 | .option("-r, --require ", "require a module", requires, {d: undefined, i: -1}) 16 | .option("--left", "perform a left join", false) 17 | .option("--right", "perform a right join", false) 18 | .option("--outer", "perform an outer join", false) 19 | .parse(process.argv); 20 | 21 | if (commander.args.length < 2 || commander.args.length > 4) { 22 | commander.outputHelp(); 23 | process.exit(1); 24 | } 25 | 26 | if (commander.args.length < 4) { 27 | if (commander.args.length < 3) commander.args.splice(0, 0, "i"); 28 | commander.args.splice(1, 0, commander.args[0]); 29 | } 30 | 31 | if ([commander.left, commander.right, commander.outer].filter(Boolean).length > 1) { 32 | console.error(); 33 | console.error(" error: only one of --left, --right, --outer allowed") 34 | console.error(); 35 | process.exit(1); 36 | } 37 | 38 | var join = commander.left ? joinLeft 39 | : commander.right ? joinRight 40 | : commander.outer ? joinOuter 41 | : joinInner; 42 | 43 | var i0 = -1, 44 | i1 = -1, 45 | ii = 0, 46 | sandbox = commander.require, 47 | map = new Map, 48 | key0 = expression(commander.args[0]), 49 | key1 = expression(commander.args[1]), 50 | context = new vm.createContext(sandbox); 51 | 52 | readline.createInterface({ 53 | input: commander.args[2] === "-" ? process.stdin : fs.createReadStream(commander.args[2]), 54 | output: null 55 | }).on("line", function(line) { 56 | sandbox.i = ++i0; 57 | try { 58 | sandbox.d = JSON.parse(line); 59 | } catch (error) { 60 | console.error("stdin:" + (i0 + 1)); 61 | console.error(line); 62 | console.error("^"); 63 | console.error("SyntaxError: " + error.message); 64 | process.exit(1); 65 | } 66 | var k = key0.runInContext(context); 67 | if (map.has(k)) map.get(k)[0].push(sandbox.d); 68 | else map.set(k, [[sandbox.d], []]); 69 | }).on("close", function() { 70 | if ((ii |= 1) === 3) join(); 71 | }); 72 | 73 | readline.createInterface({ 74 | input: commander.args[3] === "-" ? process.stdin : fs.createReadStream(commander.args[3]), 75 | output: null 76 | }).on("line", function(line) { 77 | sandbox.i = ++i1; 78 | try { 79 | sandbox.d = JSON.parse(line); 80 | } catch (error) { 81 | console.error("stdin:" + (i1 + 1)); 82 | console.error(line); 83 | console.error("^"); 84 | console.error("SyntaxError: " + error.message); 85 | process.exit(1); 86 | } 87 | var k = key1.runInContext(context); 88 | if (map.has(k)) map.get(k)[1].push(sandbox.d); 89 | else map.set(k, [[], [sandbox.d]]); 90 | }).on("close", function() { 91 | if ((ii |= 2) === 3) join(); 92 | }); 93 | 94 | function joinInner() { 95 | map.forEach(function(value, key) { 96 | value[0].forEach(function(v0) { 97 | value[1].forEach(function(v1) { 98 | output([v0, v1]); 99 | }); 100 | }); 101 | }); 102 | } 103 | 104 | function joinLeft() { 105 | map.forEach(joinLeftValue); 106 | } 107 | 108 | function joinRight() { 109 | map.forEach(joinRightValue); 110 | } 111 | 112 | function joinOuter() { 113 | map.forEach(function(value, key) { 114 | if (value[0].length) joinLeftValue(value, key); 115 | else if (value[1].length) joinRightValue(value, key); 116 | }); 117 | } 118 | 119 | function joinLeftValue(value, key) { 120 | value[0].forEach(function(v0) { 121 | if (value[1].length) { 122 | value[1].forEach(function(v1) { 123 | output([v0, v1]); 124 | }); 125 | } else { 126 | output([v0, null]); 127 | } 128 | }); 129 | } 130 | 131 | function joinRightValue(value, key) { 132 | value[1].forEach(function(v1) { 133 | if (value[0].length) { 134 | value[0].forEach(function(v0) { 135 | output([v0, v1]); 136 | }); 137 | } else { 138 | output([null, v1]); 139 | } 140 | }); 141 | } 142 | -------------------------------------------------------------------------------- /ndjson-map: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression]") 13 | .description("Transform values in a newline-delimited JSON stream.") 14 | .option("-r, --require ", "require a module", requires, {d: undefined, i: -1}) 15 | .parse(process.argv); 16 | 17 | if (commander.args.length > 1) { 18 | commander.outputHelp(); 19 | process.exit(1); 20 | } 21 | 22 | if (commander.args.length < 1) { 23 | commander.args[0] = "d"; 24 | } 25 | 26 | var i = -1, 27 | sandbox = commander.require, 28 | map = expression(commander.args[0]), 29 | context = new vm.createContext(sandbox); 30 | 31 | readline.createInterface({ 32 | input: process.stdin, 33 | output: null 34 | }).on("line", function(line) { 35 | sandbox.i = ++i; 36 | try { 37 | sandbox.d = JSON.parse(line); 38 | } catch (error) { 39 | console.error("stdin:" + (i + 1)); 40 | console.error(line); 41 | console.error("^"); 42 | console.error("SyntaxError: " + error.message); 43 | process.exit(1); 44 | } 45 | output(map.runInContext(context)); 46 | }); 47 | -------------------------------------------------------------------------------- /ndjson-reduce: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression [initial]]") 13 | .description("Reduce a newline-delimited JSON stream to a single value.") 14 | .option("-r, --require ", "require a module", requires, {p: undefined, d: undefined, i: -1}) 15 | .parse(process.argv); 16 | 17 | if (commander.args.length > 2) { 18 | commander.outputHelp(); 19 | process.exit(1); 20 | } 21 | 22 | if (commander.args.length < 1) { 23 | commander.args[0] = "p.push(d), p"; 24 | commander.args[1] = "[]"; 25 | } 26 | 27 | var i = -1, 28 | sandbox = commander.require, 29 | unset = commander.args.length < 2 || (sandbox.p = expression(commander.args[1], "initial").runInNewContext(), false), 30 | reduce = expression(commander.args[0]), 31 | context = new vm.createContext(sandbox); 32 | 33 | readline.createInterface({ 34 | input: process.stdin, 35 | output: null 36 | }).on("line", function(line) { 37 | sandbox.i = ++i; 38 | try { 39 | sandbox.d = JSON.parse(line); 40 | } catch (error) { 41 | console.error("stdin:" + (i + 1)); 42 | console.error(line); 43 | console.error("^"); 44 | console.error("SyntaxError: " + error.message); 45 | process.exit(1); 46 | } 47 | sandbox.p = unset ? (unset = false, sandbox.d) : reduce.runInContext(context); 48 | }).on("close", function() { 49 | output(sandbox.p); 50 | }); 51 | -------------------------------------------------------------------------------- /ndjson-sort: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression]") 13 | .description("Sort values in a newline-delimited JSON stream.") 14 | .option("-r, --require ", "require a module", requires, {a: undefined, b: undefined}) 15 | .parse(process.argv); 16 | 17 | if (commander.args.length > 1) { 18 | commander.outputHelp(); 19 | process.exit(1); 20 | } 21 | 22 | if (commander.args.length < 1) { 23 | commander.args[0] = "a < b ? -1 : a > b ? 1 : a >= b ? 0 : NaN"; 24 | } 25 | 26 | var i = -1, 27 | results = [], 28 | sandbox = commander.require, 29 | compare = expression(commander.args[0]), 30 | context = new vm.createContext(sandbox); 31 | 32 | readline.createInterface({ 33 | input: process.stdin, 34 | output: null 35 | }).on("line", function(line) { 36 | ++i; 37 | try { 38 | results.push(JSON.parse(line)); 39 | } catch (error) { 40 | console.error("stdin:" + (i + 1)); 41 | console.error(line); 42 | console.error("^"); 43 | console.error("SyntaxError: " + error.message); 44 | process.exit(1); 45 | } 46 | }).on("close", function() { 47 | results 48 | .sort(function(a, b) { return sandbox.a = a, sandbox.b = b, compare.runInContext(context); }) 49 | .forEach(output); 50 | }); 51 | -------------------------------------------------------------------------------- /ndjson-split: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression]") 13 | .description("Transform values in a newline-delimited JSON stream into multiple values.") 14 | .option("-r, --require ", "require a module", requires, {d: undefined, i: -1}) 15 | .parse(process.argv); 16 | 17 | if (commander.args.length > 1) { 18 | commander.outputHelp(); 19 | process.exit(1); 20 | } 21 | 22 | if (commander.args.length < 1) { 23 | commander.args[0] = "d"; 24 | } 25 | 26 | var i = -1, 27 | sandbox = commander.require, 28 | map = expression(commander.args[0]), 29 | context = new vm.createContext(sandbox); 30 | 31 | readline.createInterface({ 32 | input: process.stdin, 33 | output: null 34 | }).on("line", function(line) { 35 | sandbox.i = ++i; 36 | try { 37 | sandbox.d = JSON.parse(line); 38 | } catch (error) { 39 | console.error("stdin:" + (i + 1)); 40 | console.error(line); 41 | console.error("^"); 42 | console.error("SyntaxError: " + error.message); 43 | process.exit(1); 44 | } 45 | var rows = map.runInContext(context); 46 | if (!Array.isArray(rows)) { 47 | console.error("stdin:" + (i + 1)); 48 | console.error(rows == null ? "null" : JSON.stringify(rows)); 49 | console.error("^"); 50 | console.error("TypeError: not array"); 51 | process.exit(1); 52 | } 53 | Array.prototype.forEach.call(rows, output); 54 | }); 55 | -------------------------------------------------------------------------------- /ndjson-top: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | var readline = require("readline"), 4 | vm = require("vm"), 5 | commander = require("commander"), 6 | requires = require("./requires"), 7 | expression = require("./expression"), 8 | output = require("./output"); 9 | 10 | commander 11 | .version(require("./package.json").version) 12 | .usage("[options] [expression]") 13 | .description("Select the top values from a newline-delimited JSON stream.") 14 | .option("-n ", "the maximum number of output values", 1) 15 | .option("-r, --require ", "require a module", requires, {a: undefined, b: undefined}) 16 | .parse(process.argv); 17 | 18 | if (commander.args.length > 1) { 19 | commander.outputHelp(); 20 | process.exit(1); 21 | } 22 | 23 | if (commander.args.length < 1) { 24 | commander.args[0] = "a < b ? -1 : a > b ? 1 : a >= b ? 0 : NaN"; 25 | } 26 | 27 | var i = -1, 28 | sandbox = commander.require, 29 | heap = new Heap(commander.n, function(a, b) { return sandbox.a = a, sandbox.b = b, compare.runInContext(context); }), 30 | compare = expression(commander.args[0]), 31 | context = new vm.createContext(sandbox); 32 | 33 | readline.createInterface({ 34 | input: process.stdin, 35 | output: null 36 | }).on("line", function(line) { 37 | ++i; 38 | try { 39 | heap.push(JSON.parse(line)); 40 | } catch (error) { 41 | console.error("stdin:" + (i + 1)); 42 | console.error(line); 43 | console.error("^"); 44 | console.error("SyntaxError: " + error.message); 45 | process.exit(1); 46 | } 47 | }).on("close", function() { 48 | var result 49 | while (!(result = heap.pop()).done) { 50 | output(result.value); 51 | } 52 | }); 53 | 54 | function Heap(k, compare) { 55 | this._array = []; 56 | this._n = 0; 57 | this._k = k; 58 | this._compare = compare; 59 | } 60 | 61 | Heap.prototype.push = function(object) { 62 | heap_up(this, this._array[this._n] = object, this._n++); 63 | if (this._array.length > this._k) this.pop(); 64 | }; 65 | 66 | Heap.prototype.pop = function(object) { 67 | if (this._n <= 0) return {done: true, value: undefined}; 68 | var removed = this._array[0], object; 69 | if (--this._n > 0) object = this._array[this._n], heap_down(this, this._array[0] = object, 0); 70 | return {done: false, value: removed}; 71 | }; 72 | 73 | function heap_up(heap, object, i) { 74 | while (i > 0) { 75 | var j = ((i + 1) >> 1) - 1, parent = heap._array[j]; 76 | if (heap._compare(object, parent) >= 0) break; 77 | heap._array[i] = parent; 78 | heap._array[i = j] = object; 79 | } 80 | } 81 | 82 | function heap_down(heap, object, i) { 83 | while (true) { 84 | var r = (i + 1) << 1, l = r - 1, j = i, child = heap._array[j]; 85 | if (l < heap._n && heap._compare(heap._array[l], child) < 0) child = heap._array[j = l]; 86 | if (r < heap._n && heap._compare(heap._array[r], child) < 0) child = heap._array[j = r]; 87 | if (j === i) break; 88 | heap._array[i] = child; 89 | heap._array[i = j] = object; 90 | } 91 | } 92 | -------------------------------------------------------------------------------- /output.js: -------------------------------------------------------------------------------- 1 | process.stdout.on("error", function(error) { 2 | if (error.code === "EPIPE" || error.errno === "EPIPE") { 3 | process.exit(0); 4 | } 5 | }); 6 | 7 | module.exports = function(value) { 8 | process.stdout.write(value == null ? "null\n" : JSON.stringify(value) + "\n"); 9 | }; 10 | -------------------------------------------------------------------------------- /package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ndjson-cli", 3 | "version": "0.3.1", 4 | "lockfileVersion": 1, 5 | "dependencies": { 6 | "acorn": { 7 | "version": "5.1.1", 8 | "resolved": "https://registry.npmjs.org/acorn/-/acorn-5.1.1.tgz", 9 | "integrity": "sha512-vOk6uEMctu0vQrvuSqFdJyqj1Q0S5VTDL79qtjo+DhRr+1mmaD+tluFSCZqhvi/JUhXSzoZN2BhtstaPEeE8cw==" 10 | }, 11 | "commander": { 12 | "version": "2.11.0", 13 | "resolved": "https://registry.npmjs.org/commander/-/commander-2.11.0.tgz", 14 | "integrity": "sha512-b0553uYA5YAEGgyYIGYROzKQ7X5RAqedkfjiZxwi0kL1g3bOaBNNZfYkzt/CL0umgD5wc9Jec2FbB98CjkMRvQ==" 15 | }, 16 | "path-parse": { 17 | "version": "1.0.5", 18 | "resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.5.tgz", 19 | "integrity": "sha1-PBrfhx6pzWyUMbbqK9dKD/BVxME=" 20 | }, 21 | "resolve": { 22 | "version": "1.3.3", 23 | "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.3.3.tgz", 24 | "integrity": "sha1-ZVkHw0aahoDcLeOidaj91paR8OU=" 25 | } 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ndjson-cli", 3 | "version": "0.3.1", 4 | "description": "Command line tools for operating on newline-delimited JSON streams.", 5 | "keywords": [ 6 | "ndjson", 7 | "filter", 8 | "map" 9 | ], 10 | "homepage": "https://github.com/mbostock/ndjson-cli", 11 | "license": "BSD-3-Clause", 12 | "author": { 13 | "name": "Mike Bostock", 14 | "url": "https://bost.ocks.org/mike" 15 | }, 16 | "repository": { 17 | "type": "git", 18 | "url": "http://github.com/mbostock/ndjson-cli.git" 19 | }, 20 | "bin": { 21 | "ndjson-cat": "ndjson-cat", 22 | "ndjson-filter": "ndjson-filter", 23 | "ndjson-join": "ndjson-join", 24 | "ndjson-map": "ndjson-map", 25 | "ndjson-reduce": "ndjson-reduce", 26 | "ndjson-sort": "ndjson-sort", 27 | "ndjson-split": "ndjson-split", 28 | "ndjson-top": "ndjson-top" 29 | }, 30 | "scripts": { 31 | "postpublish": "git push && git push --tags" 32 | }, 33 | "dependencies": { 34 | "acorn": "^5.1.1", 35 | "commander": "^2.11.0", 36 | "resolve": "^1.3.3" 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /requires.js: -------------------------------------------------------------------------------- 1 | var resolve = require("./resolve"); 2 | 3 | module.exports = function(module, sandbox) { 4 | var i = module.indexOf("="), name = module; 5 | if (i >= 0) name = module.slice(0, i), module = module.slice(i + 1); 6 | module = require(resolve(module)); 7 | if (sandbox[name]) Object.assign(sandbox[name], module); 8 | else sandbox[name] = module; 9 | return sandbox; 10 | }; 11 | -------------------------------------------------------------------------------- /resolve.js: -------------------------------------------------------------------------------- 1 | var child_process = require("child_process"), 2 | resolve = require("resolve"), 3 | options = {basedir: process.cwd(), paths: []}; 4 | 5 | module.exports = function(module) { 6 | try { 7 | return resolve.sync(module, options); // Attempt fast local resolve first. 8 | } catch (error) { 9 | if (!options.paths.length) { 10 | options.paths.push(child_process.execSync("npm root -g").toString().trim()); 11 | return resolve.sync(module, options); 12 | } else { 13 | throw error; 14 | } 15 | } 16 | }; 17 | -------------------------------------------------------------------------------- /test/a.csv: -------------------------------------------------------------------------------- 1 | name,color 2 | Fred,red 3 | Alice,green 4 | Bob,blue 5 | -------------------------------------------------------------------------------- /test/b.csv: -------------------------------------------------------------------------------- 1 | name,number 2 | Fred,21 3 | Alice,42 4 | Bob,102 5 | -------------------------------------------------------------------------------- /test/c.csv: -------------------------------------------------------------------------------- 1 | name,number 2 | Fred,42 3 | Barney,34 -------------------------------------------------------------------------------- /test/numbers.ndjson: -------------------------------------------------------------------------------- 1 | 0.8708564985150364 2 | 0.6282973483871181 3 | 0.941848612793901 4 | 0.14859169421600638 5 | 0.9999181560594017 6 | 0.3968767367470014 7 | 0.20371012722438175 8 | 0.9587241325087854 9 | 0.3942479465513997 10 | 0.8369493624421225 11 | 0.8936738362025676 12 | 0.19083839376771405 13 | 0.7974316018242735 14 | 0.06876498131515829 15 | 0.31140245651892484 16 | 0.6712151199144298 17 | 0.27630611837410757 18 | 0.07024482193459147 19 | 0.9533232922736004 20 | 0.6836214921749444 21 | 0.33035890991320715 22 | 0.300021643600789 23 | 0.3619732315522095 24 | 0.8889177135727768 25 | 0.0135483444266562 26 | 0.8829774179825169 27 | 0.11965210930463699 28 | 0.5950138613966398 29 | 0.08580672395012945 30 | 0.6470631209703241 31 | 0.5361872946814388 32 | 0.997327840286717 33 | 0.41952438656674174 34 | 0.23275768346158254 35 | 0.194062585796815 36 | 0.8547495788665811 37 | 0.7406677845464431 38 | 0.737990470694649 39 | 0.29492593761873853 40 | 0.8286296322845179 41 | 0.06273303630256044 42 | 0.3442407657387001 43 | 0.4858578245036098 44 | 0.29757084911002063 45 | 0.6404440948737637 46 | 0.3837127875169084 47 | 0.03356054567694988 48 | 0.8611259108248541 49 | 0.9873008495427522 50 | 0.3518102805888712 51 | 0.7962474583773096 52 | 0.9383121938554613 53 | 0.9490690587701376 54 | 0.17448304421612626 55 | 0.9649190760892725 56 | 0.9968625973928764 57 | 0.1575123061993542 58 | 0.591962303939316 59 | 0.4268484950797915 60 | 0.11704799523082254 61 | 0.24884962645810083 62 | 0.8800260457283289 63 | 0.6461462328197316 64 | 0.1905573286180653 65 | 0.08047385213752323 66 | 0.6603950802474836 67 | 0.19256913998864644 68 | 0.06866065477544048 69 | 0.977911044550319 70 | 0.23890072321459388 71 | 0.16338887009242464 72 | 0.4256788742334672 73 | 0.20930609299961223 74 | 0.12500293863432832 75 | 0.14190056509671956 76 | 0.6348176860532131 77 | 0.5461914692400491 78 | 0.18053627136600747 79 | 0.8470963644767187 80 | 0.6837470045314562 81 | 0.0019053260023955687 82 | 0.7130643678303361 83 | 0.7443331550250532 84 | 0.5610601860019833 85 | 0.8804856927222988 86 | 0.09330511173496969 87 | 0.9379317077171399 88 | 0.8588582986957094 89 | 0.30945346246064576 90 | 0.6289628574574031 91 | 0.016905412136452247 92 | 0.5071704003248041 93 | 0.6072093120578297 94 | 0.6707125302954746 95 | 0.7813567726764492 96 | 0.5885460334348609 97 | 0.9865334902296659 98 | 0.4452867416271975 99 | 0.7341139227720459 100 | 0.1964653301874124 101 | 0.43943889348533793 102 | 0.6219722793505724 103 | 0.6033004848184191 104 | 0.5994683669194465 105 | 0.40150957717398983 106 | 0.4618085715584819 107 | 0.5286754204159598 108 | 0.6637770615025989 109 | 0.24895855698247815 110 | 0.01796115952633115 111 | 0.9799741652050489 112 | 0.8221306794085728 113 | 0.9629846770592398 114 | 0.08533904594979247 115 | 0.9364925532972197 116 | 0.07387836101017564 117 | 0.9754248210046608 118 | 0.3848424472009 119 | 0.23858807622998013 120 | 0.0024836399315386615 121 | 0.9213958619355767 122 | 0.7672370045501646 123 | 0.6511587865291586 124 | 0.4675058828667398 125 | 0.6241209954191822 126 | 0.8167489678300641 127 | 0.5550772518431084 128 | 0.6818102191754447 129 | --------------------------------------------------------------------------------