├── LICENSE ├── README.md ├── collate-keys-get.jq ├── collate-keys-index.jq ├── collate-keys.jq ├── find-key.jq ├── har-get-req-postdata.jq ├── har-get-req.jq ├── har-get-res-json.jq ├── har-get-res.jq ├── har-getall-req-url.jq ├── har-getall-res-url-json.jq ├── har-getall-res-url.jq ├── har-scan-req-domains-report.jq ├── har-scan-req-domains-unique.jq ├── har-scan-req-postdata-full.jq ├── har-scan-req-postdata.jq ├── har-scan-req-referrer-report.jq ├── har-scan-req-referrer-unique.jq ├── har-scan-req-urls-report.jq ├── har-scan-req-urls-unique.jq ├── har-scan-res-json-full.jq ├── har-scan-res-json.jq ├── jjq.exe ├── jjq.py ├── scan-key-values.jq └── wildcards.bat /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 James Bach 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # jq_cookbook 2 | Here are my favorite JQ scripts for analysis and web-based testing. 3 | 4 | To learn about jq go [here](https://stedolan.github.io/jq/manual/). 5 | 6 | ## Installation 7 | 8 | Make sure you have installed JQ and that it is on the PATH. 9 | 10 | For Windows, clone the repo and set the PATH environment variable to point to it. 11 | 12 | JJQ is a Python program to conveniently serve these scripts and get help. I've included an EXE version. That way you can run the JJQ exe from the command-line, wherever you happen to be. 13 | 14 | ## Usage 15 | 16 | ``` 17 | usage: JJQ [-h] [--list [SCRIPT] | --listall] [script] [input] [json_file] 18 | 19 | A wrapper around JQ to serve complex scripts. 20 | 21 | positional arguments: 22 | script Script file to execute 23 | input Optional parameter for script 24 | json_file JSON file as target 25 | 26 | options: 27 | -h, --help show this help message and exit 28 | --list [SCRIPT] List all available scripts, or get help on specific script 29 | --listall List all available scripts with capsule descriptions 30 | 31 | Written by James Bach and Michael Bolton 32 | ``` 33 | 34 | Each file is commented. 35 | 36 | Contact me with any questions. 37 | -------------------------------------------------------------------------------- /collate-keys-get.jq: -------------------------------------------------------------------------------- 1 | # This script allows you to dump the values for a specific key listed in 2 | # the output from collate-keys-index.jq. Specify the index number of the 3 | # key that you want. EXAMPLE "jjq collate-keys-get 5 inputfile.json" 4 | 5 | # This function gets an element if it is a leaf node 6 | def grab(d): 7 | d 8 | | 9 | 10 | to_entries? 11 | | 12 | 13 | .[] 14 | | 15 | 16 | select( 17 | .value 18 | | 19 | 20 | type != "array" and type != "object" 21 | ); 22 | 23 | # Make an array 24 | [ 25 | 26 | # Crawl through the JSON 27 | recurse 28 | | 29 | 30 | # Find all the leaf nodes 31 | grab(.) 32 | | 33 | 34 | # Ignore scalars that are in arrays (we are only focusing on key value pairs in the JSON) 35 | select(.key | type == "string") 36 | ] 37 | | 38 | 39 | # Now take the array and collate it by key 40 | group_by(.key) 41 | | 42 | [ 43 | # for each different key... 44 | .[] 45 | | 46 | 47 | # make an object... 48 | { 49 | # that has the key name as the key... 50 | (.[0].key): 51 | ( 52 | # and all the unique values in an array attached to that key. 53 | [ 54 | .[] 55 | | 56 | 57 | .value 58 | ] 59 | | 60 | 61 | unique 62 | ) 63 | } 64 | ] 65 | | 66 | 67 | # Put it all into one JSON 68 | add 69 | | 70 | 71 | # Target the category the user wants 72 | keys[$myvar | tonumber] as $key 73 | | 74 | 75 | {($key):(.[$key])} -------------------------------------------------------------------------------- /collate-keys-index.jq: -------------------------------------------------------------------------------- 1 | # This script is like collate-keys, but creates an index of all the keys and displays a table of them. 2 | # To obtain the values, use the collate-keys-get script. 3 | 4 | # This function gets an element if it is a leaf node 5 | def grab(d): 6 | d 7 | | 8 | 9 | to_entries? 10 | | 11 | 12 | .[] 13 | | 14 | 15 | select( 16 | .value 17 | | 18 | 19 | type != "array" and type != "object" 20 | ); 21 | 22 | # Make an array 23 | [ 24 | 25 | # Crawl through the JSON 26 | recurse 27 | | 28 | 29 | # Find all the leaf nodes 30 | grab(.) 31 | | 32 | 33 | # Ignore scalars that are in arrays (we are only focusing on key value pairs in the JSON) 34 | select(.key | type == "string") 35 | ] 36 | | 37 | 38 | # Now take the array and collate it by key 39 | group_by(.key) 40 | | 41 | [ 42 | # for each different key... 43 | .[] 44 | | 45 | 46 | # make an object... 47 | { 48 | # that has the key name as the key... 49 | (.[0].key): 50 | ( 51 | # and all the unique values in an array attached to that key. 52 | [ 53 | .[] 54 | | 55 | 56 | .value 57 | ] 58 | | 59 | 60 | unique 61 | ) 62 | } 63 | ] 64 | | 65 | 66 | # Put it all into one JSON 67 | add 68 | | 69 | 70 | # Prepare to create a nice output table 71 | . as $original 72 | | 73 | 74 | # For each key get the length of the list of unique items in that category. 75 | keys 76 | | 77 | 78 | length as $len 79 | | 80 | 81 | # Now, from the top... 82 | $original 83 | | 84 | 85 | # ...iterate through the list of keys with an index number. 86 | range($len) as $index 87 | | 88 | 89 | keys[$index] as $key 90 | | 91 | 92 | .[$key] 93 | | 94 | 95 | # Emit table row 96 | [$index, $key, length] 97 | | 98 | 99 | @csv 100 | -------------------------------------------------------------------------------- /collate-keys.jq: -------------------------------------------------------------------------------- 1 | # This script take a JSON and walks through it to find all key value pairs where the value is a scalar or null. 2 | # It then makes a dictionary of all the unique values in the JSON for each of those keys. 3 | 4 | # This function gets an element if it is a leaf node 5 | def grab(d): 6 | d 7 | | 8 | 9 | to_entries? 10 | | 11 | 12 | .[] 13 | | 14 | 15 | select( 16 | .value 17 | | 18 | 19 | type != "array" and type != "object" 20 | ); 21 | 22 | # Make an array 23 | [ 24 | 25 | # Crawl through the JSON 26 | recurse 27 | | 28 | 29 | # Find all the leaf nodes 30 | grab(.) 31 | | 32 | 33 | # Ignore scalars that are in arrays (we are only focusing on key value pairs in the JSON) 34 | select(.key | type == "string") 35 | ] 36 | | 37 | 38 | # Now take the array and collate it by key 39 | group_by(.key) 40 | | 41 | [ 42 | # for each different key... 43 | .[] 44 | | 45 | 46 | # make an object... 47 | { 48 | # that has the key name as the key... 49 | (.[0].key): 50 | ( 51 | # and all the unique values in an array attached to that key. 52 | [ 53 | .[] 54 | | 55 | 56 | .value 57 | ] 58 | | 59 | 60 | unique 61 | ) 62 | } 63 | ] 64 | | 65 | 66 | add 67 | 68 | -------------------------------------------------------------------------------- /find-key.jq: -------------------------------------------------------------------------------- 1 | # This script looks for a particular key anywhere in the JSON and prints its path. 2 | # You have to pass in the key to look for from the command line. 3 | # EXAMPLE: jjq find-key "tag" input_file.json 4 | 5 | # Turn the JSON into a list of arrays, each one representing a path 6 | . as $original 7 | | 8 | 9 | paths 10 | | 11 | 12 | # if the last entry in the path array matches the key we are looking for, print that. 13 | select( 14 | .[-1]==$myvar 15 | ) 16 | | 17 | 18 | # make it look nice first by converting to CSV 19 | @csv 20 | | 21 | 22 | # prepend some commas to make the rest of this work... 23 | "," + . 24 | | 25 | 26 | # put brackets around numbers to indicate array indices 27 | gsub("(?<=[\b,])(?[0-9]+)(?=[\b,])";"[\(.x)]") 28 | | 29 | 30 | # If there are any keys that are pure numbers, put them in single quotes 31 | gsub("\"(?[0-9]+)\"";"'\(.x)'") 32 | | 33 | 34 | # commas to colons... 35 | gsub(","; ".") 36 | | 37 | 38 | # remove quotes 39 | gsub("\"";"") 40 | | 41 | 42 | # bring back the double quotes for keys that are numbers 43 | gsub("'(?[0-9]+)'";"\"\(.x)\"") 44 | -------------------------------------------------------------------------------- /har-get-req-postdata.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves a specific HTTP POST request based on an index passed in from the command line. 3 | # It outputs a pretty-printed version of the json, if it is one. Otherwise, just prints it plain. 4 | # 5 | 6 | # Take the variable from the command line and treat it as a number. 7 | # Then use it as the index to the array of entries in the HAR file. 8 | .log.entries[$myvar | tonumber] 9 | | 10 | 11 | # get the content 12 | .request.postData.text 13 | | 14 | 15 | # try to interpret the text content as a json 16 | # I use a try/catch here because often post requests are not jsons 17 | try(fromjson) catch (.) 18 | -------------------------------------------------------------------------------- /har-get-req.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves a specific HTTP request based on an index passed in from the command line. 3 | # 4 | 5 | # Take the variable from the command line and treat it as a number. 6 | # Then use it as the index to the array of entries in the HAR file. 7 | .log.entries[$myvar | tonumber] 8 | | 9 | 10 | # get the content 11 | .request 12 | 13 | -------------------------------------------------------------------------------- /har-get-res-json.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves a specific HTTP response based on an index passed in from the command line. 3 | 4 | # Take the variable from the command line and treat it as a number. 5 | # Then use it as the index to the array of entries in the HAR file. 6 | .log.entries[$myvar | tonumber] 7 | | 8 | 9 | # get the content 10 | .response.content.text 11 | | 12 | 13 | # parse it as a JSON 14 | # I don't use a try/catch here because I want to see the error. 15 | fromjson 16 | -------------------------------------------------------------------------------- /har-get-res.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves a specific HTTP response based on an index passed in from the command line. 3 | # 4 | 5 | # Take the variable from the command line and treat it as a number. 6 | # Then use it as the index to the array of entries in the HAR file. 7 | .log.entries[$myvar | tonumber] 8 | | 9 | 10 | # get the content 11 | .response 12 | 13 | -------------------------------------------------------------------------------- /har-getall-req-url.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves all HTTP requests that have a URL that begins with the supplied string 3 | # 4 | 5 | # Take the variable from the command line and treat it as a URL. 6 | [ 7 | .log.entries[].request 8 | | 9 | 10 | select( 11 | 12 | .url 13 | | 14 | 15 | startswith($myvar) 16 | ) 17 | ] -------------------------------------------------------------------------------- /har-getall-res-url-json.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves all HTTP requests that have a URL that begins with the supplied string 3 | # then it puts all the content that can be interpreted as JSON into one big JSON 4 | # 5 | 6 | # Take the variable from the command line and treat it as a URL. 7 | [ 8 | .log.entries[] 9 | | 10 | 11 | select( 12 | 13 | .request.url 14 | | 15 | 16 | startswith($myvar) 17 | ) 18 | 19 | | 20 | 21 | .response.content.text 22 | | 23 | 24 | fromjson? 25 | 26 | ] -------------------------------------------------------------------------------- /har-getall-res-url.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script retrieves all HTTP responses that have a URL that begins with the supplied string 3 | # 4 | 5 | # Take the variable from the command line and treat it as a URL. 6 | [ 7 | .log.entries[] 8 | | 9 | 10 | select( 11 | 12 | .request.url 13 | | 14 | 15 | startswith($myvar) 16 | ) 17 | 18 | | 19 | 20 | .response 21 | ] -------------------------------------------------------------------------------- /har-scan-req-domains-report.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and scans it for a list of unique domains (distinct from unique urls) 3 | # that were called. Each domain is followed by a list of numbers indicating the HAR file entry, 4 | # which can be retrieved using the har-get-res script. 5 | 6 | # save the original HAR file to a variable; we'll need that later. 7 | . as $original 8 | | 9 | 10 | # get the original entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | #create an array 19 | [ 20 | 21 | # for each entry, save the index number for later 22 | .[] as $index 23 | | 24 | 25 | #by going through the log and getting each entry 26 | $original.log.entries[$index] 27 | 28 | #and pulling the URL from each one 29 | .request.url 30 | 31 | #now we've got a list of URLs 32 | | 33 | 34 | # hack off everything after the domain 35 | capture("(?^http[s]?://.*?)[/?]") 36 | | 37 | 38 | #now we have a JSON object from which we can pull the domain 39 | [$index, .domain] 40 | 41 | ] 42 | | 43 | 44 | # save where we are 45 | . as $original 46 | | 47 | 48 | # make an array out of all the domains 49 | [ 50 | .[].[1] 51 | ] 52 | | 53 | 54 | # get rid of the duplicates 55 | unique 56 | | 57 | 58 | # iterate through each of the unique domains 59 | .[] as $key 60 | | 61 | 62 | # enclose in an array so that we can get the TSV, later 63 | [ 64 | 65 | # print the domain as the first column 66 | $key, 67 | 68 | # collect all the entry numbers into a text list 69 | ( 70 | 71 | # enclose in an array... 72 | [ 73 | # now start from the top... 74 | $original[] 75 | | 76 | 77 | # ...and search for any entry with the current domain... 78 | select(.[1] == $key) 79 | | 80 | 81 | # ...and grab that for our list. 82 | .[0] 83 | ] 84 | | 85 | 86 | # turn the collected list into a comma-separated list 87 | @csv 88 | ) 89 | ] 90 | | 91 | 92 | # turn the final array into a tab-separated list 93 | @tsv 94 | -------------------------------------------------------------------------------- /har-scan-req-domains-unique.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and scans it for a list of unique domains (distinct from unique urls) 3 | # that were called. 4 | 5 | # save the original HAR file to a variable; we'll need that later. 6 | . as $original 7 | | 8 | 9 | # get the original entries 10 | .log.entries 11 | | 12 | 13 | # get all the index numbers for the entries 14 | keys 15 | | 16 | 17 | #create an array 18 | [ 19 | 20 | # for each entry, save the index number for later 21 | .[] as $index 22 | | 23 | 24 | #by going through the log and getting each entry 25 | $original.log.entries[$index] 26 | 27 | #and pulling the URL from each one 28 | .request.url 29 | 30 | #now we've got a list of URLs 31 | | 32 | 33 | # hack off everything after the domain 34 | capture("(?^http[s]?://.*?)[/?]") 35 | | 36 | 37 | #now we have a JSON object from which we can pull the domain 38 | [$index, .domain] 39 | 40 | ] 41 | | 42 | 43 | # save where we are 44 | . as $original 45 | | 46 | 47 | # make an array out of all the domains 48 | [ 49 | .[].[1] 50 | ] 51 | | 52 | 53 | # get rid of the duplicates 54 | unique 55 | | 56 | 57 | # unpack the array so that it is a clean list 58 | .[] -------------------------------------------------------------------------------- /har-scan-req-postdata-full.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and tries to interpret the content of each POST request as a json. 3 | # It outputs the URL of each request, with an index number and pretty-printed version of the json. 4 | # 5 | 6 | # Save the original input, because we will need it, below... 7 | . as $original 8 | | 9 | 10 | # get all the entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | # for each entry, save the index number for later... 19 | .[] as $index 20 | | 21 | 22 | # ...then go back to the original input and pick out the entry for that index number 23 | $original.log.entries[$index] 24 | | 25 | 26 | # check if the content type include "json" or "text" 27 | # (the reason we check for text type is that in some webapps jsons are mislabeled as text type) 28 | select( 29 | .request.postData.text != null and .request.postData.text != "" 30 | ) 31 | | 32 | 33 | # now we are ready for output... 34 | # this outputs the entry index 35 | (["Entry", $index]| join(": ")), 36 | 37 | # url 38 | .request.url, 39 | 40 | # print the content 41 | .request.postData.text, 42 | 43 | # separator 44 | "\n=============================================" -------------------------------------------------------------------------------- /har-scan-req-postdata.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and tries to interpret the content of each POST request as a json. 3 | # It outputs the URL of each request, with an index number and pretty-printed version of the json. 4 | # 5 | 6 | # Save the original input, because we will need it, below... 7 | . as $original 8 | | 9 | 10 | # get all the entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | # for each entry, save the index number for later... 19 | .[] as $index 20 | | 21 | 22 | # ...then go back to the original input and pick out the entry for that index number 23 | $original.log.entries[$index] 24 | | 25 | 26 | # check if the content type include "json" or "text" 27 | # (the reason we check for text type is that in some webapps jsons are mislabeled as text type) 28 | 29 | select( 30 | .request.postData.text != null and .request.postData.text != "" 31 | ) 32 | | 33 | 34 | # now we are ready for output... 35 | # this outputs the entry index 36 | [ 37 | (["Entry", $index]| join(": ")), 38 | $original.log.entries[$index].request.url 39 | ] 40 | | 41 | 42 | # output as a TSV 43 | @tsv -------------------------------------------------------------------------------- /har-scan-req-referrer-report.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and finds each URL and referrer pair. 3 | # It outputs a TSV with index numbers and timestamps 4 | 5 | # Save the original input, because we will need it, below... 6 | . as $original 7 | | 8 | 9 | # get all the entries 10 | .log.entries 11 | | 12 | 13 | # get all the index numbers for the entries 14 | keys 15 | | 16 | 17 | # for each entry, save the index number for later... 18 | .[] as $index 19 | | 20 | 21 | # ...then go back to the original input and pick out the entry for that index number 22 | $original.log.entries[$index] 23 | | 24 | 25 | # create an array 26 | [ 27 | # output the index number and timestamp 28 | $index, 29 | .startedDateTime, 30 | 31 | # output URL 32 | .request.url, 33 | 34 | # output referrer if possible 35 | ( 36 | # go through headers 37 | .request.headers[] 38 | | 39 | 40 | # if the header name is "referer" (case insensitive)... 41 | select( 42 | .name 43 | | 44 | test("referer";"i") 45 | ) 46 | | 47 | 48 | # ...then output the referer 49 | .value 50 | ) 51 | 52 | # otherwise, just say N/A if there is no referrer 53 | // 54 | "N/A" 55 | 56 | ] 57 | | 58 | 59 | # dump it as tab-separated table 60 | @tsv 61 | -------------------------------------------------------------------------------- /har-scan-req-referrer-unique.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and finds each URL and referrer pair. 3 | # It outputs a list of all unique pairs 4 | 5 | .log.entries[] 6 | | 7 | 8 | # create an array 9 | [ 10 | 11 | # output URL 12 | .request.url, 13 | 14 | # output referrer if possible 15 | ( 16 | # go through headers 17 | .request.headers[] 18 | | 19 | 20 | # if the header name is "referer" (case insensitive)... 21 | select( 22 | .name 23 | | 24 | test("referer";"i") 25 | ) 26 | | 27 | 28 | # ...then output the referer 29 | .value 30 | ) 31 | 32 | # otherwise, just say N/A if there is no referrer 33 | // 34 | "N/A" 35 | 36 | ] 37 | | 38 | 39 | unique 40 | | 41 | 42 | # dump it as tab-separated table 43 | @tsv 44 | -------------------------------------------------------------------------------- /har-scan-req-urls-report.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and gets all the URLs visited. 3 | # It outputs the URL of each request, with a timestamp and index number. 4 | # 5 | 6 | # Save the original input, because we will need it, below... 7 | . as $original 8 | | 9 | 10 | # get all the entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | # for each entry, save the index number for later... 19 | .[] as $index 20 | | 21 | 22 | # ...then go back to the original input and pick out the entry for that index number 23 | $original.log.entries[$index] 24 | | 25 | 26 | # now we are ready for output... 27 | # this outputs the entry index, time, and url 28 | [ 29 | $index, 30 | .startedDateTime, 31 | .request.url 32 | ] 33 | | 34 | 35 | # output to a CSV so we can import it to Excel or something 36 | @csv 37 | -------------------------------------------------------------------------------- /har-scan-req-urls-unique.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and gets all the URLs visited. 3 | # It outputs a list of each unique URL. 4 | # 5 | 6 | # Save the original input, because we will need it, below... 7 | . as $original 8 | | 9 | 10 | # get all the entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | [ 19 | # for each entry, save the index number for later... 20 | .[] as $index 21 | | 22 | 23 | # ...then go back to the original input and pick out the entry for that index number 24 | 25 | $original.log.entries[$index] 26 | | 27 | 28 | # now we are ready for output... 29 | # we put this into a list so that we can run the next filter on it... 30 | 31 | .request.url 32 | ] 33 | | 34 | 35 | # list each URL only once 36 | unique 37 | | 38 | 39 | # print as a bare list 40 | .[] 41 | -------------------------------------------------------------------------------- /har-scan-res-json-full.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and finds each response that contains json or text mime type. 3 | # It outputs the URL of each request, with an index number and pretty-printed version of the 4 | # content interpreted as a json. 5 | # 6 | 7 | # Save the original input, because we will need it, below... 8 | . as $original 9 | | 10 | 11 | # get all the entries 12 | .log.entries 13 | | 14 | 15 | # get all the index numbers for the entries 16 | keys 17 | | 18 | 19 | # for each entry, save the index number for later... 20 | [ 21 | .[] as $index 22 | | 23 | 24 | # ...then go back to the original input and pick out the entry for that index number 25 | 26 | $original.log.entries[$index] 27 | | 28 | 29 | # check if the content type include "json" or "text" 30 | # (the reason we check for text type is that in some webapps jsons are mislabeled as text type) 31 | select( 32 | .response 33 | | select(.headers[] 34 | | select(.name | test("content-type";"i")) 35 | | select( 36 | (.value | contains("json")) 37 | or 38 | (.value | contains("text")) 39 | ) 40 | ) 41 | ) 42 | | 43 | 44 | # now we are ready for output... 45 | # this outputs the entry index 46 | {($index | tostring): 47 | { 48 | "url": .request.url, 49 | "content": 50 | 51 | # this tries to interpret the text content as a json 52 | (.response.content.text | try(fromjson) catch ("")) 53 | } 54 | } 55 | ] -------------------------------------------------------------------------------- /har-scan-res-json.jq: -------------------------------------------------------------------------------- 1 | # For HAR files only. 2 | # This script traverses a HAR file and finds each response that can be interpreted as a JSON. 3 | # It outputs the URL of each request, with an index number 4 | # 5 | 6 | # Save the original input, because we will need it, below... 7 | . as $original 8 | | 9 | 10 | # get all the entries 11 | .log.entries 12 | | 13 | 14 | # get all the index numbers for the entries 15 | keys 16 | | 17 | 18 | # for each entry, save the index number for later... 19 | .[] as $index 20 | | 21 | 22 | # ...then go back to the original input and pick out the entry for that index number 23 | $original.log.entries[$index] 24 | | 25 | 26 | # create an array 27 | [ 28 | 29 | # try to parse it as a json 30 | (.response.content.text 31 | | 32 | try( 33 | fromjson 34 | | 35 | 36 | # if that worked, output the index and URL of the lucky entry 37 | $index, 38 | $original.log.entries[$index].request.url 39 | ) 40 | ) 41 | ] 42 | | 43 | 44 | # delete any entries that are empty (because they failed the try) 45 | select ( length > 0 ) 46 | | 47 | 48 | # output as a TSV 49 | @tsv 50 | -------------------------------------------------------------------------------- /jjq.exe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/satisfice/jq_cookbook/99c61f81b2c482e3d8c9f3fd9c829544f06260ef/jjq.exe -------------------------------------------------------------------------------- /jjq.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import re 4 | import io 5 | import subprocess 6 | from colorama import init as colorama_init # type: ignore 7 | from colorama import Fore # type: ignore 8 | from colorama import Style # type: ignore 9 | import argparse 10 | from stat import S_ISFIFO 11 | import errno 12 | 13 | def get_executable_path(): 14 | if getattr(sys, 'frozen', False): 15 | return os.path.dirname(sys.executable) # Returns path to the .exe 16 | else: 17 | return os.path.dirname(os.path.abspath(__file__)) 18 | 19 | def get_jqfiles(): 20 | jjq_files = [] 21 | for filename in os.listdir(scriptdir): 22 | if filename.endswith(".jq"): 23 | jjq_files.append(filename) 24 | return jjq_files 25 | 26 | def fix_name(script): 27 | # if the user hasn't supplied ".jq", stick it on 28 | if script.endswith(".jq"): 29 | return script 30 | else: 31 | return script + ".jq" 32 | 33 | def display_help(script): 34 | 35 | print (f"\n{Fore.LIGHTBLUE_EX}{script.replace('.\\', '').replace(".jq", "")}{Style.RESET_ALL}\n") 36 | 37 | f = open(scriptdir + "/" + script, "r") 38 | 39 | help_text = f.readline().strip("\n") 40 | while re.match("^#", help_text): 41 | # indent the help text 42 | print(" ", help_text.replace("#", "")) 43 | help_text = f.readline().strip("\n") 44 | 45 | def process_command(script,target,input): 46 | if input: 47 | command = ["jq","-r","-f",script,"--arg","myvar",input,"-"] 48 | else: 49 | command = ["jq","-r","-f",script,"-"] 50 | 51 | data = target.read() 52 | target.close() 53 | subprocess.run(command, input=data, encoding="utf-8", shell=True) 54 | 55 | def parse_arguments(): 56 | """Parse command line arguments.""" 57 | parser = argparse.ArgumentParser( 58 | prog='JJQ', 59 | description='A wrapper around JQ to serve complex scripts.', 60 | epilog='Written by James Bach and Michael Bolton', 61 | formatter_class=argparse.RawDescriptionHelpFormatter, 62 | ) 63 | 64 | # Create a mutually exclusive group for --list and --listall 65 | list_group = parser.add_mutually_exclusive_group() 66 | list_group.add_argument( 67 | "--list", 68 | nargs="?", 69 | const=True, # Default value when --list is specified without an argument 70 | metavar="SCRIPT", 71 | help="List all available scripts, or get help on specific script" 72 | ) 73 | list_group.add_argument( 74 | "--listall", 75 | action="store_true", 76 | help="List all available scripts with capsule descriptions" 77 | ) 78 | 79 | # Add positional arguments that are only required if neither --list nor --listall is specified 80 | parser.add_argument( 81 | "script", 82 | nargs="?", # Make it optional initially 83 | help="Script file to execute" 84 | ) 85 | parser.add_argument( 86 | "input", 87 | nargs="?", # Make it optional initially 88 | help="Optional parameter for script" 89 | ) 90 | parser.add_argument( 91 | "json_file", 92 | nargs="?", # Make it optional initially 93 | help="JSON file as target" 94 | ) 95 | 96 | args = parser.parse_args() 97 | 98 | # Validate arguments 99 | if not args.list and not args.listall: 100 | # In normal mode, script and json_file are required 101 | if args.script is None: 102 | parser.error("Please specify a JQ script.") 103 | if S_ISFIFO(os.fstat(0).st_mode): 104 | args.json_file = io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8') 105 | else: 106 | if args.input is None: 107 | parser.error("Please specify a target JSON file.") 108 | else: 109 | if args.json_file is None: 110 | args.json_file = open(args.input,"r",encoding="utf8") 111 | args.input = None 112 | else: 113 | args.json_file = open(args.json_file,"r",encoding="utf8") 114 | 115 | return args 116 | 117 | colorama_init() 118 | scriptdir = get_executable_path() 119 | args = parse_arguments() 120 | 121 | 122 | if args.list is not None or args.listall is not None: 123 | jjq_files = get_jqfiles() 124 | 125 | if args.list is not None: 126 | if args.list is True: # --list specified without an argument 127 | print("Available jjq commands:") 128 | for file in jjq_files: 129 | print(f"\t{file.replace(".jq","")}") 130 | else: # --list specified with an argument 131 | display_help(fix_name(args.list)) 132 | 133 | elif args.listall: 134 | print("Listing all items with detailed information") 135 | for jq_file in jjq_files: 136 | display_help(jq_file) 137 | else: 138 | process_command(fix_name(scriptdir + "/" + args.script),args.json_file,args.input) 139 | 140 | 141 | 142 | -------------------------------------------------------------------------------- /scan-key-values.jq: -------------------------------------------------------------------------------- 1 | # This script traverses a any JSON and finds all value of a particular key you supply on the command line 2 | # 3 | 4 | [ 5 | paths as $path 6 | | 7 | 8 | select($path[-1]==$myvar) 9 | | 10 | 11 | getpath($path) 12 | ] 13 | | 14 | 15 | unique 16 | | 17 | 18 | .[] -------------------------------------------------------------------------------- /wildcards.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | rem This batch file takes a path name with wildcard characters as input and expands it 3 | rem to a list of explicit file names. This can be used to feed JQ multiple files at once 4 | rem more easily. 5 | rem 6 | rem usage: wildcards test*.json & jq "." %expanded_list% 7 | rem (replace "test*.json" with whatever you want) 8 | rem 9 | rem I got this from https://superuser.com/questions/460598/is-there-any-way-to-get-the-windows-cmd-shell-to-expand-wildcard-paths 10 | 11 | set expanded_list= 12 | for /f "tokens=*" %%F in ('dir /b /a:-d "%~1"') do call set expanded_list=%%expanded_list%% "%%F" 13 | --------------------------------------------------------------------------------