├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── csv-to-json ├── dsv-to-json ├── generate-json-diff-results ├── generate-test-results ├── json-diff.go ├── json-empty.go ├── json-format ├── json-post ├── json-to-csv ├── json-to-dsv ├── json-to-newline ├── json-to-xml ├── json-to-yaml ├── newline-to-json ├── run-all-tests ├── run-json-diff-tests ├── test_data ├── 1.json ├── 2.json ├── a.json ├── b.json ├── empty.json ├── json-diff-tests.json ├── m1-m2-diff.json ├── m1.json ├── m2.json ├── null.json ├── s1.json ├── s2.json ├── s3.json └── tests.json ├── xml-to-json └── yaml-to-json /.gitignore: -------------------------------------------------------------------------------- 1 | json-diff 2 | json-empty 3 | *.swp 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Tyler Adams 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all : json-diff json-empty 2 | 3 | json-diff : json-diff.go 4 | go build json-diff.go 5 | 6 | json-empty : json-empty.go 7 | go build json-empty.go 8 | 9 | clean : 10 | rm json-diff json-empty 2> /dev/null || true 11 | 12 | install : all 13 | install csv-to-json /usr/local/bin 14 | install dsv-to-json /usr/local/bin 15 | install json-diff /usr/local/bin 16 | install json-empty /usr/local/bin 17 | install json-format /usr/local/bin 18 | install json-post /usr/local/bin 19 | install json-to-csv /usr/local/bin 20 | install json-to-dsv /usr/local/bin 21 | install json-to-newline /usr/local/bin 22 | install json-to-xml /usr/local/bin 23 | install json-to-yaml /usr/local/bin 24 | install newline-to-json /usr/local/bin 25 | install xml-to-json /usr/local/bin 26 | install yaml-to-json /usr/local/bin 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # JSON Toolkit 2 | 3 | A collection of CLI tools which make it easy to write pipelines processing various data files. Using these tools in conjunction with [jq](https://stedolan.github.io/jq/), you can write data processing prototypes in seconds! 4 | 5 | ## Example usecases 6 | ### 1 to 10 in json 7 | ```bash 8 | seq 10 | newline-to-json | jq 'map(tonumber)' 9 | ``` 10 | ### Select rows from a CSV where the 2nd and 3rd column are equal 11 | ```bash 12 | cat file.csv | csv-to-json | jq 'map(select(.[1] == .[2]))' | json-to-csv 13 | ``` 14 | ### Extract the difference between daily XML reports. 15 | ```bash 16 | cat yesterday.xml | json-to-xml | json-format > yesterday.json 17 | cat today.xml | json-to-xml | json-format > today.json 18 | json-diff yesterday.json today.json | json-format > difference.json 19 | ``` 20 | ### Extract the difference between daily CSV reports. 21 | ```bash 22 | cat yesterday.csv | json-to-csv | json-format > yesterday.json 23 | cat today.csv | json-to-csv | json-format > today.json 24 | json-diff yesterday.json today.json | json-format > difference.json 25 | ``` 26 | ### Testing framework for a web endpoint 27 | If the test passes, this will return exit code 0 and nothing on stdout 28 | If the test fails, this will return exit code 1 and the test difference as JSON on stdout. 29 | 30 | ```bash 31 | cat test-input.json | json-post https://your-web-server/api/endpoint | json-format > actual-test-output.json 32 | json-diff actual-test-output.json expected-test-output.json | json-format > test-difference.json 33 | test-difference.json | json-empty 34 | ``` 35 | 36 | 37 | ## Prerequsites 38 | 39 | * UNIX-like operating system 40 | * Bash 41 | * Go 42 | * Python3 43 | * [jq](https://stedolan.github.io/jq/) 44 | 45 | ## Getting Started 46 | 47 | ```bash 48 | git clone git@github.com:sisyphean-labs/json-toolkit.git 49 | make 50 | ./run-all-tests # On success this should return nothing 51 | sudo make install 52 | ``` 53 | 54 | ## Tools 55 | 56 | ### json-diff 57 | `json-diff` takes in two json files and returns the differences between the files as json. 58 | 59 | #### Output format 60 | The output is a json encoded list of difference objects describing the difference between two json files. 61 | If the files are equivalent, the output will be an empty json array. 62 | ##### Example 63 | Consider 64 | ```json 65 | [ 1, 2, 3 ] 66 | ``` 67 | and 68 | ```json 69 | [ 1, 2, 4 ] 70 | ``` 71 | Then the difference is 72 | ```json 73 | [{"leftValue":3,"path":[2],"rightValue":4}] 74 | ``` 75 | At path `[2]`, (index 2 element in the top level array), the left value is 3, but the right value is 4 76 | 77 | ##### Difference Object 78 | A difference object has a required `path`, an optional `leftValue`, and an optional `rightValue`. 79 | The `leftValue` and `rightValue` are the values of the json object at the path for the left file and right files respectively. 80 | If there is no `leftValue`, this means the path does not exist for the left file, and similarly for the `rightValue` and right file. For example, if the left is an array of length 1 and the right is an array of length 2, then path `[1]` (index 1 element) does not exist for the left but it does for the right. 81 | ###### Path 82 | The path is the location of a particular a json value nested within a larger json value. 83 | In `json-diff`, this is encoded as a json array of integers for array indicies and strings for object keys. 84 | 85 | ###### Example Path 86 | For: 87 | ```json 88 | [ 89 | 0, 90 | { 91 | "a": [ 92 | -1 93 | ], 94 | } 95 | ] 96 | ``` 97 | -1 can be found at `[1, "a", 0]`. 98 | This path should be read as take the whole array: 99 | Then take the index `1` element of top level array. 100 | Then take the value corresponding to the `"a"` key. 101 | Then take the index `0` element. 102 | This final value is the one found at `[1, "a", 0]`. 103 | 104 | ### json-empty 105 | #### Description 106 | `json-empty` helps you return the right error code in bash scripts. 107 | * If the input is an empty json array, `json-empty` returns exit code 0 and an empty stdout. 108 | * If the input is valid json but not an empty array, `json-empty` returns exit code 1 and the returns the inputted string to stdout. 109 | * If the input is not valid json, `json-empty` returns exit code 2 and throws an error message to stderr. 110 | #### Examples 111 | ##### Bash script checking if two json files are equivalent 112 | ```bash 113 | json-diff left.json right.json | json-empty 114 | ``` 115 | 116 | ##### Passing an empty JSON array into json-empty 117 | ```bash 118 | echo '[]' | json-empty 119 | ``` 120 | 121 | ##### Passing a non-empty JSON array into json-empty 122 | ```bash 123 | echo '[1]' | json-empty 124 | ``` 125 | 126 | ##### Passing a JSON string into json-empty 127 | ```bash 128 | echo '"non-empty array input"' | json-empty 129 | ``` 130 | 131 | ##### Passing non-JSON into json-empty 132 | ```bash 133 | echo 'this is not json' | json-empty 134 | ``` 135 | 136 | ### json-format 137 | #### Description 138 | `json-format` formats json on stdin to be pretty printed and sorts keys in objects alphabetically 139 | #### Example 140 | ```bash 141 | echo '[1, 2, 3]' | json-format 142 | json-format a.json 143 | ``` 144 | 145 | ### json-post 146 | #### Description 147 | `json-post` posts the json on stdin to the specified url 148 | #### Example 149 | ```bash 150 | echo '[1, 2, 3]' | json-post https://httpbin.org/post 151 | ``` 152 | 153 | ### json-to-csv 154 | #### Description 155 | `json-to-csv` takes a json array of array of strings from stdin and formats the data as a csv on stdout. 156 | #### Examples 157 | ```bash 158 | echo '[["Single cell"]]' | json-to-csv 159 | echo '[["Multiple", "cells", "but", "one", "row"]]' | json-to-csv 160 | echo '[["Multiple", "cells"], ["and"], ["multiple", "rows"]]' | json-to-csv 161 | ``` 162 | 163 | ### json-to-dsv 164 | #### Description 165 | `json-to-dsv` takes a json array of array of strings from stdin, and a delmiter as the first argument, and formats the data as a dsv with the specified delimiter on stdout. 166 | #### Examples 167 | ```bash 168 | echo '[["Single cell"]]' | json-to-dsv : 169 | echo '[["Multiple", "cells", "but", "one", "row"]]' | json-to-dsv : 170 | echo '[["Multiple", "cells"], ["and"], ["multiple", "rows"]]' | json-to-dsv : 171 | ``` 172 | 173 | ### json-to-newline 174 | #### Description 175 | `json-to-newline` takes a json array from stdin and formats the data as newline delimited strings on stdout. 176 | #### Examples 177 | ```bash 178 | echo '[1,2,3,4,true,"s"]' | json-to-newline 179 | ``` 180 | 181 | ### json-to-xml 182 | #### Description 183 | `json-to-xml` takes json from stdin and formats the data as xml on stdout. Only an object with a single key can be converted to xml. 184 | #### Examples 185 | ```bash 186 | echo '{"root": {"a": "b"}}' | json-to-xml 187 | ``` 188 | 189 | ### json-to-yaml 190 | #### Description 191 | `json-to-yaml` takes json from stdin and formats the data as yaml on stdout. 192 | #### Examples 193 | ```bash 194 | echo '{"a": 1, "b": 2}' | json-to-yaml 195 | ``` 196 | 197 | ### csv-to-json 198 | #### Description 199 | `csv-to-json` takes a csv from stdin and formats the data into a json array of array of strings. 200 | #### Examples 201 | ```bash 202 | echo Single cell | csv-to-json 203 | echo Multiple,cells,but,one,row | csv-to-json 204 | echo -e Multiple,cells\\nand\\nmultiple,rows | csv-to-json 205 | ``` 206 | 207 | ### dsv-to-json 208 | #### Description 209 | `dsv-to-json` takes a dsv file from stdin, the delimiter as the first argument, and formats the data into a json array of array of strings. 210 | #### Examples 211 | ```bash 212 | echo Single cell | dsv-to-json : 213 | echo Multiple:cells:but:one:row | dsv-to-json : 214 | echo -e Multiple:cells\\nand\\nmultiple:rows | dsv-to-json : 215 | ``` 216 | 217 | ### newline-to-json 218 | #### Description 219 | `newline-to-json` takes newline delimited strings from stdin and formats the data as a json array of strings on stdout. 220 | #### Examples 221 | ```bash 222 | seq 10 | newline-to-json 223 | ``` 224 | 225 | ### xml-to-json 226 | #### Description 227 | `xml-to-json` takes xml from stdin and formats the data as json on stdout. 228 | #### Examples 229 | ```bash 230 | echo 'b' | xml-to-json 231 | ``` 232 | 233 | ### yaml-to-json 234 | #### Description 235 | `yaml-to-json` takes yaml from stdin and formats the data as json on stdout. 236 | #### Examples 237 | ```bash 238 | echo 'a: b' | yaml-to-json 239 | ``` 240 | -------------------------------------------------------------------------------- /csv-to-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts csv from stdin to json on stdout") 10 | args = parser.parse_args() 11 | 12 | csv_reader = csv.reader(sys.stdin) 13 | print(json.dumps(list(csv_reader))) 14 | exit(0) 15 | 16 | if __name__ == "__main__": 17 | main() 18 | -------------------------------------------------------------------------------- /dsv-to-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts dsv from stdin to json on stdout") 10 | parser.add_argument('DELIMITER', help="The delimiter separating values" ) 11 | args = parser.parse_args() 12 | 13 | delimiter = args.DELIMITER 14 | lines = sys.stdin.readlines() 15 | cells = list(map(lambda l: l.strip("\n").split(delimiter), lines)) 16 | print(json.dumps(cells)) 17 | exit(0) 18 | 19 | if __name__ == "__main__": 20 | main() 21 | -------------------------------------------------------------------------------- /generate-json-diff-results: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import subprocess 5 | import sys 6 | 7 | def flatten(ll): 8 | a = [] 9 | for l in ll: 10 | a.extend(l) 11 | return a 12 | 13 | def run_command(command, stdin): 14 | p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 15 | output, _ = p.communicate(stdin.encode("ASCII")) 16 | return output.decode("ASCII"), p.returncode 17 | 18 | def run_test(test): 19 | expected_output = test["expectedOutput"] 20 | expected_return_code = test["expectedReturnCode"] 21 | actual_output, actual_return_code = run_command(test["command"], test["input"]) 22 | if expected_output == actual_output and expected_return_code == actual_return_code: 23 | return [] 24 | return [{ 25 | "actualOutput": actual_output, 26 | "actualReturnCode": actual_return_code, 27 | "command": test["command"], 28 | "expectedOutput": expected_output, 29 | "expectedReturnCode": expected_return_code, 30 | }] 31 | 32 | def main(): 33 | tests = json.load(sys.stdin) 34 | results = flatten(map(run_test, tests)) 35 | print(json.dumps(results)) 36 | if len(results) == 0: 37 | sys.exit(0) 38 | else: 39 | sys.exit(1) 40 | 41 | if __name__ == "__main__": 42 | main() 43 | -------------------------------------------------------------------------------- /generate-test-results: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import subprocess 5 | import sys 6 | 7 | def flatten(ll): 8 | a = [] 9 | for l in ll: 10 | a.extend(l) 11 | return a 12 | 13 | def run_command(command, stdin): 14 | p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 15 | output, _ = p.communicate(stdin.encode("ASCII")) 16 | return output.decode("ASCII"), p.returncode 17 | 18 | def run_test(test): 19 | actual_output, actual_return_code = run_command(test["command"], test["input"]) 20 | return [{ 21 | "command": test["command"], 22 | "input": test["input"], 23 | "output": actual_output, 24 | "returnCode": actual_return_code, 25 | }] 26 | 27 | def main(): 28 | tests = json.load(sys.stdin) 29 | results = flatten(map(run_test, tests)) 30 | print(json.dumps(results)) 31 | sys.exit(0) 32 | 33 | if __name__ == "__main__": 34 | main() 35 | -------------------------------------------------------------------------------- /json-diff.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import "encoding/json" 4 | import "errors" 5 | import "fmt" 6 | import "io/ioutil" 7 | import "os" 8 | import "reflect" 9 | 10 | // Standard mathematical min function 11 | func min(a int, b int) int { 12 | if a < b { 13 | return a 14 | } 15 | 16 | return b 17 | } 18 | 19 | // Compares two objects and returns a list of differences relative to the json path 20 | // If the objects are the same, an empty list is returned 21 | // If the objects are different, a list with a single element is returned 22 | func compare_simple(path []interface{}, simple1 interface{}, simple2 interface{}) []map[string]interface{} { 23 | if simple1 == simple2 { 24 | return []map[string]interface{}{} 25 | } else { 26 | return []map[string]interface{}{ 27 | { 28 | "path": path, 29 | "leftValue": simple1, 30 | "rightValue": simple2, 31 | }, 32 | } 33 | } 34 | } 35 | 36 | // Compares two slices index by index and returns a list of differences relative to the json path 37 | // If the slices are the same, an empty list is returned 38 | // If the slices differ at an index, a difference is returned specifying the value in each slice 39 | // If one slice is longer than another, a difference is returned for each index in one slice and not the other 40 | func compare_slice(path []interface{}, slice1 []interface{}, slice2 []interface{}) []map[string]interface{} { 41 | l := min(len(slice1), len(slice2)) 42 | m := []map[string]interface{}{} 43 | for i := 0; i < l; i++ { 44 | i1 := slice1[i] 45 | i2 := slice2[i] 46 | m = append(m, compare_object(append(path, i), i1, i2)...) 47 | } 48 | if len(slice1) > len(slice2) { 49 | for i := l; i < len(slice1); i++ { 50 | m = append(m, map[string]interface{}{ 51 | "path": append(path, i), 52 | "leftValue": slice1[i], 53 | }) 54 | } 55 | } else if len(slice2) > len(slice1) { 56 | for i := l; i < len(slice2); i++ { 57 | m = append(m, map[string]interface{}{ 58 | "path": append(path, i), 59 | "rightValue": slice2[i], 60 | }) 61 | } 62 | } 63 | return m 64 | } 65 | 66 | // Compares two maps key by key and returns a list of differences relative to the json path 67 | // If the maps are the same, an empty list is returned 68 | // If the maps differ at a key, a difference is returned specifying the value in each map 69 | // If one map has keys which are not in the another, a difference is returned for each key in one map and not the other 70 | func compare_map(path []interface{}, map1 map[string]interface{}, map2 map[string]interface{}) []map[string]interface{} { 71 | diff := []map[string]interface{}{} 72 | for key, _ := range map1 { 73 | _, keyInMap2 := map2[key] 74 | if keyInMap2 { 75 | diff = append(diff, compare_object(append(path, key), map1[key], map2[key])...) 76 | } else { 77 | diff = append(diff, map[string]interface{}{ 78 | "path": append(path, key), 79 | "leftValue": map1[key], 80 | }) 81 | } 82 | } 83 | for key, _ := range map2 { 84 | _, keyInMap1 := map1[key] 85 | if !keyInMap1 { 86 | diff = append(diff, map[string]interface{}{ 87 | "path": append(path, key), 88 | "rightValue": map2[key], 89 | }) 90 | } 91 | } 92 | return diff 93 | } 94 | 95 | // Compares two objects and returns a list of differences relative to the json path 96 | // If the objects are the same, an empty list is returned 97 | // If the objects are different types, returns a simple difference between the objects 98 | // If the objects are the same type, it 99 | func compare_object(path []interface{}, object1 interface{}, object2 interface{}) []map[string]interface{} { 100 | // nil does not have a reflection type kind, so it's easier to hardcode this special case 101 | if object1 == nil || object2 == nil { 102 | return compare_simple(path, object1, object2) 103 | } 104 | 105 | var type1 reflect.Kind 106 | var type2 reflect.Kind 107 | 108 | type1 = reflect.TypeOf(object1).Kind() 109 | type2 = reflect.TypeOf(object2).Kind() 110 | 111 | if type1 == type2 { 112 | if type1 == reflect.Float64 { 113 | return compare_simple(path, object1, object2) 114 | } else if type1 == reflect.Bool { 115 | return compare_simple(path, object1, object2) 116 | } else if type1 == reflect.String { 117 | return compare_simple(path, object1, object2) 118 | } else if type1 == reflect.Slice { 119 | return compare_slice(path, object1.([]interface{}), object2.([]interface{})) 120 | } else if type1 == reflect.Map { 121 | return compare_map(path, object1.(map[string]interface{}), object2.(map[string]interface{})) 122 | } else { 123 | panic(errors.New("Type not found: " + string(type1))) 124 | } 125 | } else { 126 | return compare_simple(path, object1, object2) 127 | } 128 | } 129 | 130 | func main() { 131 | if len(os.Args) != 3 { 132 | fmt.Println("Usage: json-diff LEFT RIGHT") 133 | fmt.Println("") 134 | fmt.Println("Computes the difference between two json files") 135 | fmt.Println("") 136 | fmt.Println("Options:") 137 | fmt.Println(" -h --help Show this screen") 138 | os.Exit(1) 139 | } 140 | 141 | file1, err := ioutil.ReadFile(os.Args[1]) 142 | if err != nil { 143 | panic(err) 144 | } 145 | file2, err := ioutil.ReadFile(os.Args[2]) 146 | if err != nil { 147 | panic(err) 148 | } 149 | 150 | var object1 interface{} 151 | var object2 interface{} 152 | if err := json.Unmarshal(file1, &object1); err != nil { 153 | panic(err) 154 | } 155 | 156 | if err := json.Unmarshal(file2, &object2); err != nil { 157 | panic(err) 158 | } 159 | 160 | diff := compare_object([]interface{}{}, object1, object2) 161 | 162 | output, _ := json.Marshal(diff) 163 | fmt.Printf("%v\n", string(output)) 164 | if len(diff) == 0 { 165 | os.Exit(0) 166 | } else { 167 | os.Exit(1) 168 | } 169 | } 170 | -------------------------------------------------------------------------------- /json-empty.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import "os" 4 | import "fmt" 5 | import "io/ioutil" 6 | import "encoding/json" 7 | 8 | func main() { 9 | if len(os.Args) != 1 { 10 | fmt.Println("Usage: json-empty") 11 | fmt.Println("") 12 | fmt.Println("Returns whether stdin is an empty json array") 13 | fmt.Println("") 14 | fmt.Println("Options:") 15 | fmt.Println(" -h --help Show this screen") 16 | os.Exit(1) 17 | } 18 | bytes, _ := ioutil.ReadAll(os.Stdin) 19 | 20 | var dat interface{} 21 | if err := json.Unmarshal(bytes, &dat); err != nil { 22 | panic(err) 23 | } 24 | 25 | ar, ok := dat.([]interface{}) 26 | if ok && len(ar) == 0 { 27 | os.Exit(0) 28 | } else { 29 | output, _ := json.Marshal(dat) 30 | fmt.Println(string(output)) 31 | os.Exit(1) 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /json-format: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | usage () { 4 | echo "Usage: json-format [FILE]" 5 | echo "" 6 | echo "Formats json file" 7 | echo "" 8 | echo "Example:" 9 | echo " cat file.json | json-format" 10 | echo " cat file.json | json-format -" 11 | echo " json-format file.json" 12 | exit 0 13 | } 14 | while getopts "h" opt; do 15 | case "$opt" in 16 | h) 17 | usage 18 | ;; 19 | esac 20 | done 21 | 22 | if [ "$#" -eq 0 ]; then 23 | jq -S '.' 24 | exit 0 25 | elif [ "$#" -eq 1 ]; then 26 | FILE="$1" 27 | if [ "$FILE" == "-" ]; then 28 | jq -S '.' 29 | else 30 | TMP_FILE="$(mktemp)" 31 | jq -S '.' $FILE > $TMP_FILE 32 | cp $TMP_FILE $FILE 33 | fi 34 | else 35 | usage 36 | fi 37 | -------------------------------------------------------------------------------- /json-post: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import json 5 | import sys 6 | import urllib 7 | import urllib.request 8 | 9 | def tuples_to_dict(tuples): 10 | return {i[0]: i[1] for i in tuples} 11 | 12 | def make_request(url, body): 13 | request = urllib.request.Request( 14 | url=url, 15 | data=json.dumps(body).encode("UTF-8"), 16 | headers={ 17 | "Content-Type": "application/json" 18 | }, 19 | method='POST') 20 | try: 21 | response = urllib.request.urlopen(request) 22 | response_dict = { 23 | "status": response.status, 24 | "body": response.read().decode("ASCII"), 25 | "headers": tuples_to_dict(response.getheaders()), 26 | } 27 | except urllib.error.URLError as e: 28 | response_dict = { 29 | "status": e.code, 30 | "body": e.reason, 31 | "headers": tuples_to_dict(e.headers.items()), 32 | } 33 | 34 | if response_dict["status"] < 400: 35 | exit_code = 0 36 | else: 37 | exit_code = 1 38 | 39 | return json.dumps(response_dict), exit_code 40 | 41 | def main(): 42 | parser = argparse.ArgumentParser(description="Posts the json in stdin to the URL") 43 | parser.add_argument('URL', help="The url to which to post the json") 44 | args = parser.parse_args() 45 | 46 | body = json.load(sys.stdin) 47 | stdout, exit_code = make_request(args.URL, body) 48 | 49 | print(stdout) 50 | exit(exit_code) 51 | 52 | if __name__ == "__main__": 53 | main() 54 | 55 | -------------------------------------------------------------------------------- /json-to-csv: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts json on stdin to csv on stdout") 10 | args = parser.parse_args() 11 | 12 | rows = json.load(sys.stdin) 13 | csv_writer = csv.writer(sys.stdout, dialect='unix') 14 | csv_writer.writerows(rows) 15 | exit(0) 16 | 17 | if __name__ == "__main__": 18 | main() 19 | -------------------------------------------------------------------------------- /json-to-dsv: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts json on stdin to dsv on stdout") 10 | args = parser.parse_args() 11 | 12 | rows = json.load(sys.stdin) 13 | delimiter = sys.argv[1] 14 | for row in rows: 15 | print(delimiter.join(row)) 16 | exit(0) 17 | 18 | if __name__ == "__main__": 19 | main() 20 | -------------------------------------------------------------------------------- /json-to-newline: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | while getopts "h" opt; do 4 | case "$opt" in 5 | h) 6 | echo "Usage: json-to-newline" 7 | echo "" 8 | echo "Converts a json array to newline delimited strings" 9 | echo "" 10 | echo "Example: cat file.json | json-to-newline" 11 | exit 0 12 | ;; 13 | esac 14 | done 15 | 16 | jq -r '.[]' 17 | -------------------------------------------------------------------------------- /json-to-xml: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import json 5 | import sys 6 | import xmltodict 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts json from stdin to xml on stdout") 10 | args = parser.parse_args() 11 | 12 | data = json.load(sys.stdin) 13 | if type(data) != dict or len(list(data)) != 1: 14 | raise Exception("Only an object with a single key can be converted to xml") 15 | print(xmltodict.unparse(data, pretty=True)) 16 | exit(0) 17 | 18 | if __name__ == "__main__": 19 | main() 20 | -------------------------------------------------------------------------------- /json-to-yaml: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | import yaml 8 | 9 | def main(): 10 | parser = argparse.ArgumentParser(description="Converts json from stdin to yaml on stdout") 11 | args = parser.parse_args() 12 | 13 | print(yaml.dump(json.load(sys.stdin), default_flow_style=False), end='') 14 | exit(0) 15 | 16 | if __name__ == "__main__": 17 | main() 18 | -------------------------------------------------------------------------------- /newline-to-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | while getopts "h" opt; do 4 | case "$opt" in 5 | h) 6 | echo "Usage: newline-to-json" 7 | echo "" 8 | echo "Converts newline delimited strings to a json array of strings" 9 | echo "" 10 | echo "Example: cat file.json | newline-to-json" 11 | exit 0 12 | ;; 13 | esac 14 | done 15 | 16 | jq --raw-input --slurp 'split("\n")' | jq '.[0:-1]' 17 | -------------------------------------------------------------------------------- /run-all-tests: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | # As the rest of the test program uses json diff, we cannot test json diff using json diff. 6 | ./run-json-diff-tests 7 | 8 | TEST_FILE=test_data/tests.json 9 | TMP_FILE=$(mktemp) 10 | cat ${TEST_FILE} | ./generate-test-results > ${TMP_FILE} 11 | ./json-diff ${TMP_FILE} ${TEST_FILE} | ./json-empty 12 | rm ${TMP_FILE} 13 | -------------------------------------------------------------------------------- /run-json-diff-tests: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | cat test_data/json-diff-tests.json | ./generate-json-diff-results | ./json-empty 6 | -------------------------------------------------------------------------------- /test_data/1.json: -------------------------------------------------------------------------------- 1 | 1 2 | -------------------------------------------------------------------------------- /test_data/2.json: -------------------------------------------------------------------------------- 1 | 2 2 | -------------------------------------------------------------------------------- /test_data/a.json: -------------------------------------------------------------------------------- 1 | "a" 2 | -------------------------------------------------------------------------------- /test_data/b.json: -------------------------------------------------------------------------------- 1 | "b" 2 | -------------------------------------------------------------------------------- /test_data/empty.json: -------------------------------------------------------------------------------- 1 | [] 2 | -------------------------------------------------------------------------------- /test_data/json-diff-tests.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "input": "", 4 | "expectedReturnCode": 1, 5 | "command": [ 6 | "./json-diff" 7 | ], 8 | "expectedOutput": "Usage: json-diff file1 file2\n" 9 | }, 10 | { 11 | "input": "", 12 | "expectedReturnCode": 1, 13 | "command": [ 14 | "./json-diff", 15 | "a.json" 16 | ], 17 | "expectedOutput": "Usage: json-diff file1 file2\n" 18 | }, 19 | { 20 | "input": "", 21 | "expectedReturnCode": 1, 22 | "command": [ 23 | "./json-diff", 24 | "a.json", 25 | "a.json", 26 | "a.json" 27 | ], 28 | "expectedOutput": "Usage: json-diff file1 file2\n" 29 | }, 30 | { 31 | "input": "", 32 | "expectedReturnCode": 2, 33 | "command": [ 34 | "./json-diff", 35 | "non_existant_file", 36 | "non_existant_file" 37 | ], 38 | "expectedOutput": "" 39 | }, 40 | { 41 | "input": "", 42 | "expectedReturnCode": 0, 43 | "command": [ 44 | "./json-diff", 45 | "test_data/null.json", 46 | "test_data/null.json" 47 | ], 48 | "expectedOutput": "[]\n" 49 | }, 50 | { 51 | "input": "", 52 | "expectedReturnCode": 0, 53 | "command": [ 54 | "./json-diff", 55 | "test_data/s1.json", 56 | "test_data/s1.json" 57 | ], 58 | "expectedOutput": "[]\n" 59 | }, 60 | { 61 | "input": "", 62 | "expectedReturnCode": 0, 63 | "command": [ 64 | "./json-diff", 65 | "test_data/s2.json", 66 | "test_data/s2.json" 67 | ], 68 | "expectedOutput": "[]\n" 69 | }, 70 | { 71 | "input": "", 72 | "expectedReturnCode": 1, 73 | "command": [ 74 | "./json-diff", 75 | "test_data/s2.json", 76 | "test_data/s3.json" 77 | ], 78 | "expectedOutput": "[{\"leftValue\":1,\"path\":[0],\"rightValue\":2}]\n" 79 | }, 80 | { 81 | "input": "", 82 | "expectedReturnCode": 0, 83 | "command": [ 84 | "./json-diff", 85 | "test_data/m1.json", 86 | "test_data/m1.json" 87 | ], 88 | "expectedOutput": "[]\n" 89 | }, 90 | { 91 | "input": "", 92 | "expectedReturnCode": 0, 93 | "command": [ 94 | "./json-diff", 95 | "test_data/m2.json", 96 | "test_data/m2.json" 97 | ], 98 | "expectedOutput": "[]\n" 99 | }, 100 | { 101 | "input": "", 102 | "expectedReturnCode": 1, 103 | "command": [ 104 | "./json-diff", 105 | "test_data/m1.json", 106 | "test_data/m2.json" 107 | ], 108 | "expectedOutput": "[{\"leftValue\":1,\"path\":[\"a\"],\"rightValue\":2},{\"path\":[\"b\"],\"rightValue\":1}]\n" 109 | }, 110 | { 111 | "input": "", 112 | "expectedReturnCode": 1, 113 | "command": [ 114 | "./json-diff", 115 | "test_data/a.json", 116 | "test_data/b.json" 117 | ], 118 | "expectedOutput": "[{\"leftValue\":\"a\",\"path\":[],\"rightValue\":\"b\"}]\n" 119 | } 120 | ] 121 | -------------------------------------------------------------------------------- /test_data/m1-m2-diff.json: -------------------------------------------------------------------------------- 1 | [{"leftValue":1,"path":["a"],"rightValue":2},{"path":["b"],"rightValue":1}] 2 | -------------------------------------------------------------------------------- /test_data/m1.json: -------------------------------------------------------------------------------- 1 | {"a": 1} 2 | -------------------------------------------------------------------------------- /test_data/m2.json: -------------------------------------------------------------------------------- 1 | {"a": 2, "b": 1} 2 | -------------------------------------------------------------------------------- /test_data/null.json: -------------------------------------------------------------------------------- 1 | null 2 | -------------------------------------------------------------------------------- /test_data/s1.json: -------------------------------------------------------------------------------- 1 | [] 2 | -------------------------------------------------------------------------------- /test_data/s2.json: -------------------------------------------------------------------------------- 1 | [1] 2 | -------------------------------------------------------------------------------- /test_data/s3.json: -------------------------------------------------------------------------------- 1 | [2] 2 | -------------------------------------------------------------------------------- /test_data/tests.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "command": [ 4 | "./json-empty" 5 | ], 6 | "input": "[]\n", 7 | "output": "", 8 | "returnCode": 0 9 | }, 10 | { 11 | "command": [ 12 | "./json-empty" 13 | ], 14 | "input": "{}\n", 15 | "output": "{}\n", 16 | "returnCode": 1 17 | }, 18 | { 19 | "command": [ 20 | "./json-empty" 21 | ], 22 | "input": "0\n", 23 | "output": "0\n", 24 | "returnCode": 1 25 | }, 26 | { 27 | "command": [ 28 | "./json-to-csv" 29 | ], 30 | "input": "[[\"single cell\"]]\n", 31 | "output": "\"single cell\"\n", 32 | "returnCode": 0 33 | }, 34 | { 35 | "command": [ 36 | "./json-format" 37 | ], 38 | "input": "{\"second\": 2, \"first\": 1}", 39 | "output": "{\n \"first\": 1,\n \"second\": 2\n}\n", 40 | "returnCode": 0 41 | }, 42 | { 43 | "command": [ 44 | "./json-to-csv" 45 | ], 46 | "input": "[[\"multiple\", \"cells\", \"single\", \"row\"]]\n", 47 | "output": "\"multiple\",\"cells\",\"single\",\"row\"\n", 48 | "returnCode": 0 49 | }, 50 | { 51 | "command": [ 52 | "./json-to-csv" 53 | ], 54 | "input": "[[\"multiple\", \"cells\"], [\"multiple \", \"rows\"]]\n", 55 | "output": "\"multiple\",\"cells\"\n\"multiple \",\"rows\"\n", 56 | "returnCode": 0 57 | }, 58 | { 59 | "command": [ 60 | "./json-to-dsv", 61 | "," 62 | ], 63 | "input": "[[\"single cell\"]]\n", 64 | "output": "single cell\n", 65 | "returnCode": 0 66 | }, 67 | { 68 | "command": [ 69 | "./json-to-dsv", 70 | "," 71 | ], 72 | "input": "[[\"multiple\", \"cells\", \"single\", \"row\"]]\n", 73 | "output": "multiple,cells,single,row\n", 74 | "returnCode": 0 75 | }, 76 | { 77 | "command": [ 78 | "./json-to-dsv", 79 | "," 80 | ], 81 | "input": "[[\"multiple\", \"cells\"], [\"multiple \", \"rows\"]]\n", 82 | "output": "multiple,cells\nmultiple ,rows\n", 83 | "returnCode": 0 84 | }, 85 | { 86 | "command": [ 87 | "./json-to-xml" 88 | ], 89 | "input": "{\"a\": 1}", 90 | "output": "\n1\n", 91 | "returnCode": 0 92 | }, 93 | { 94 | "command": [ 95 | "./json-to-yaml" 96 | ], 97 | "input": "{\"a\": 1, \"b\": 2}\n", 98 | "output": "a: 1\nb: 2\n", 99 | "returnCode": 0 100 | }, 101 | { 102 | "command": [ 103 | "./csv-to-json" 104 | ], 105 | "input": "Single cell\n", 106 | "output": "[[\"Single cell\"]]\n", 107 | "returnCode": 0 108 | }, 109 | { 110 | "command": [ 111 | "./csv-to-json" 112 | ], 113 | "input": "Multiple,cells,but,one,row\n", 114 | "output": "[[\"Multiple\", \"cells\", \"but\", \"one\", \"row\"]]\n", 115 | "returnCode": 0 116 | }, 117 | { 118 | "command": [ 119 | "./csv-to-json" 120 | ], 121 | "input": "Multiple,cells\nand\nmultiple,rows", 122 | "output": "[[\"Multiple\", \"cells\"], [\"and\"], [\"multiple\", \"rows\"]]\n", 123 | "returnCode": 0 124 | }, 125 | { 126 | "command": [ 127 | "./dsv-to-json", 128 | ":" 129 | ], 130 | "input": "Single cell\n", 131 | "output": "[[\"Single cell\"]]\n", 132 | "returnCode": 0 133 | }, 134 | { 135 | "command": [ 136 | "./dsv-to-json", 137 | ":" 138 | ], 139 | "input": "Multiple:cells:but:one:row\n", 140 | "output": "[[\"Multiple\", \"cells\", \"but\", \"one\", \"row\"]]\n", 141 | "returnCode": 0 142 | }, 143 | { 144 | "command": [ 145 | "./dsv-to-json", 146 | ";" 147 | ], 148 | "input": "Multiple cells\nand\nmultiple;rows", 149 | "output": "[[\"Multiple cells\"], [\"and\"], [\"multiple\", \"rows\"]]\n", 150 | "returnCode": 0 151 | }, 152 | { 153 | "command": [ 154 | "./xml-to-json" 155 | ], 156 | "input": "b", 157 | "output": "{\"a\": \"b\"}\n", 158 | "returnCode": 0 159 | }, 160 | { 161 | "command": [ 162 | "./yaml-to-json" 163 | ], 164 | "input": "a: b", 165 | "output": "{\"a\": \"b\"}\n", 166 | "returnCode": 0 167 | } 168 | ] 169 | -------------------------------------------------------------------------------- /xml-to-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import json 5 | import sys 6 | import xmltodict 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Converts xml from stdin to json on stdout") 10 | args = parser.parse_args() 11 | 12 | xml_string = sys.stdin.read() 13 | print(json.dumps(xmltodict.parse(xml_string))) 14 | exit(0) 15 | 16 | if __name__ == "__main__": 17 | main() 18 | -------------------------------------------------------------------------------- /yaml-to-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import csv 5 | import json 6 | import sys 7 | import yaml 8 | 9 | def main(): 10 | parser = argparse.ArgumentParser(description="Converts yaml from stdin to json on stdout") 11 | args = parser.parse_args() 12 | 13 | print(json.dumps(yaml.load(sys.stdin))) 14 | exit(0) 15 | 16 | if __name__ == "__main__": 17 | main() 18 | --------------------------------------------------------------------------------