├── .gitattributes ├── .gitignore ├── CONTRIBUTING.md ├── Makefile ├── README.md ├── README.md.jinja ├── data ├── projects.toml └── sql-based.toml ├── macros.md.jinja ├── render-template.py ├── sql-based.md └── sql-based.md.jinja /.gitattributes: -------------------------------------------------------------------------------- 1 | Makefile linguist-vendored 2 | *.md.jinja linguist-vendored 3 | render-template.py linguist-vendored 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /attic 2 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | ## Do not edit `README.md` directly 4 | 5 | `README.md` is automatically generated from `README.md.jinja` and the data in [`data/projects.toml`](data/projects.toml). Do not edit `README.md` directly. To add a project or update a project's information, edit `data/projects.toml`. Edit `README.md.jinja` to change information not derived from `data/projects.toml`. 6 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: README.md sql-based.md 2 | 3 | README.md: README.md.jinja Makefile data/projects.toml macros.md.jinja render-template.py 4 | ./render-template.py README.md.jinja data/projects.toml > $@ 5 | 6 | sql-based.md: sql-based.md.jinja Makefile data/sql-based.toml macros.md.jinja render-template.py 7 | ./render-template.py sql-based.md.jinja data/sql-based.toml > $@ 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Structured text tools 2 | 3 | The following is a list of text-based file formats and command-line tools for manipulating each. 4 | 5 | 6 | ## Contents 7 | 8 | - [awk-like](#awk-like) 9 | - [awk](#awk) 10 | - [POSIX commands](#posix-commands) 11 | - [SQL-based tools](#sql-based-tools) 12 | - [Other tools](#other-tools) 13 | - [CSV](#csv) 14 | - [HTML](#html) 15 | - [JSON](#json) 16 | - [Markdown](#markdown) 17 | - [TOML](#toml) 18 | - [XML](#xml) 19 | - [YAML](#yaml) 20 | - [Configuration files](#configuration-files) 21 | - [.env](#env) 22 | - [/etc/hosts](#etchosts) 23 | - [INI](#ini) 24 | - [Multiple formats](#multiple-formats) 25 | - [Log files](#log-files) 26 | - [Multiformat tools](#multiformat-tools) 27 | - [Templating for structured text](#templating-for-structured-text) 28 | - [Extra: interactive TUIs](#extra-interactive-tuis) 29 | - [Extra: CLIs for single-file databases](#extra-clis-for-single-file-databases) 30 | - [License](#license) 31 | - [Disclosure](#disclosure) 32 | 33 | 34 | ## awk-like 35 | 36 | Tools that work with lines of fields separated by delimiters but do not necessarily support [CSV field quoting](https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules). 37 | 38 | ### awk 39 | 40 | AWK/awk is a programming language and a POSIX-standard command-line tool. (You will sometimes see "awk" used for the tool and "AWK" for the language. This document follows this convention. GNU Awk uses "Awk".) If you run Linux, macOS, or a BSD, you almost certainly have it installed. See below for Windows. 41 | 42 | - If you already know how to program, the nawk [man page](https://www.freebsd.org/cgi/man.cgi?query=nawk&sektion=1) is a great way to learn AWK quickly. What you learn from it will apply to other implementations on different platforms. Read it first if you feel overwhelmed by the sheer size of the [GNU Awk manual](https://www.gnu.org/software/gawk/manual/gawk.html). 43 | - [awk.info archive](https://web.archive.org/web/20160505033644/http://awk.info/) **—** an extensive resource on Awk. 44 | - ["AWK Vs NAWK Vs GAWK"](https://www.thegeekstuff.com/2011/06/awk-nawk-gawk/) **—** a comparison of features present in different implementations. 45 | - [busybox-w32](https://frippery.org/busybox/) includes a full implementation of POSIX awk and other tools like `sed` in a single Windows executable. 46 | - [frawk](https://github.com/ezrosent/frawk) is a Rust implementation of a language partially compatible with AWK that supports [parallelism](https://github.com/ezrosent/frawk/blob/master/info/parallelism.md) and CSV input and output. 47 | - [GNU Awk 5 binaries for Windows](https://sourceforge.net/projects/ezwinports/files/) by [EZWinPorts](https://www.gnu.org/software/emacs/manual/html_node/efaq-w32/EZWinPorts.html). 48 | - [GoAWK](https://github.com/benhoyt/goawk) is a cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows. 49 | 50 | ### POSIX commands 51 | 52 | - `comm` **—** Select the lines common to two sorted files or the lines contained in only one of them. (Manual: `man 1 comm` on your system, [GNU](https://linux.die.net/man/1/comm), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=comm&sektion=1).) 53 | - `cut` **—** Select portions of each line in one or more files. (Manual: `man 1 cut`, [GNU](https://linux.die.net/man/1/cut), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=cut&sektion=1).) 54 | - `grep` **—** Select the lines that match or do not match a pattern from one or more files. (Manual: `man 1 grep`, [GNU](https://linux.die.net/man/1/grep), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=grep&sektion=1).) 55 | - `join` **—** Take two files sorted by a common field and join their lines on the value of that field. Lines with values that do not appear in the other file are discarded. (Manual: `man 1 join`, [GNU](https://linux.die.net/man/1/join), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=join&sektion=1).) 56 | - `paste` **—** Combine several consecutive lines in a text file into one. (Manual: `man 1 paste`, [GNU](https://linux.die.net/man/1/paste), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=paste&sektion=1).) 57 | - `sort` **—** Sort lines by key fields. (Manual: `man 1 sort`, [GNU](https://linux.die.net/man/1/sort), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=sort&sektion=1).) 58 | - `uniq` **—** Find or remove repeated lines. (Manual: `man 1 uniq`, [GNU](https://linux.die.net/man/1/uniq), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=uniq&sektion=1).) 59 | 60 | ### Other tools 61 | 62 | - [csvquote](https://github.com/dbro/csvquote) **—** Transform CSV to and from a format processable with [awk-like](#awk-like) tools. 63 | - [GNU datamash](https://www.gnu.org/software/datamash/) **—** Perform statistical operations on text input. 64 | - [Hawk](https://github.com/gelisam/hawk) **—** Transform text from the command-line using Haskell expressions. 65 | - [pyp](https://github.com/hauntsaninja/pyp) **—** Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through [tomllib](https://docs.python.org/3.11/library/tomllib.html). 66 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 67 | - [vnlog](https://github.com/dkogan/vnlog/) **—** Process labelled tabular ASCII data using normal UNIX tools. Can plot data with gnuplot. 68 | 69 | 70 | ## CSV 71 | 72 | CSV, TSV, and other delimiter-separated value formats. Tools belong on this list if they support [field quoting](https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules). 73 | 74 | - [csv-nix-tools](https://github.com/mslusarz/csv-nix-tools) **—** List \*nix system information such as environment variables, files, processes, network connections, users as CSV. Manipulate and pretty-print CSV. Execute CSV rows as commands. 75 | - [csv2html](https://github.com/dbohdan/csv2html) **—** Convert CSV to HTML tables. 76 | - [csv2md](https://github.com/pstaender/csv2md) **—** Convert CSV to Markdown tables. 77 | - [csvfaker](https://github.com/pereorga/csvfaker) **—** Generate CSV files with fake data. Supports different types of fake data in different locales: names, cities, jobs, email addresses, and others. 78 | - [csvfix](https://github.com/jheusser/csvfix) **—** A multitool. Compare, filter, normalize, split, and validate CSV files. Reorder, remove, split, and merge fields. Convert data between fixed-width, multi-line, XML, and DSV format. Generate SQL statements. (Unofficial mirror.) 79 | - [csvkit](https://github.com/wireservice/csvkit) **—** csvkit is a suite of command-line tools for converting to and working with CSV: convert, clean, cut, grep, join, sort, stack, format, render, query, analyze, etc. 80 | - [csvquote](https://github.com/dbro/csvquote) **—** Transform CSV to and from a format processable with [awk-like](#awk-like) tools. 81 | - [csvtk](https://github.com/shenwei356/csvtk) **—** Search, sample, cut, join, transpose, and sort CSV/TSV files. Rename columns. Replace fields and generate new fiends from existing fields. Plot data as vector or raster histograms and box, line, and scatter plots. Convert CSV to Markdown. Convert XLSX to CSV. Split XLSX sheets. 82 | - [CSVtoTable](https://github.com/vividvilla/csvtotable) **—** Convert CSV to a searchable and sortable HTML table. 83 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 84 | - [eBay's TSV utilities](https://github.com/eBay/tsv-utils) **—** Filtering, statistics, sampling, joins and other operations on TSV files. High performance, especially good for large datasets. Written in D. 85 | - [emuto](http://kantord.github.io/emuto/) **—** CLI tool similar to jq. Create and manipulate CSV, TSV, and JSON. Can be compiled to JavaScript. 86 | - [frawk](https://github.com/ezrosent/frawk) **—** A Rust implementation of a language partially compatible with AWK that supports [parallelism](https://github.com/ezrosent/frawk/blob/master/info/parallelism.md) and CSV input and output. frawk is an awk-derived language with a CSV mode for input and for output. 87 | - [GoAWK](https://github.com/benhoyt/goawk) **—** A cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows. GoAWK is an awk implementation that adds a CSV mode for input and for output. 88 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 89 | - [jp (sgreben)](https://github.com/sgreben/jp) **—** Plot JSON and CSV data in the terminal. Supports different kinds of plots: bar charts, line charts, scatter plots, histograms, and heatmaps. 90 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 91 | - [MCMD (M-Command)](https://github.com/nysol/mcmd) **—** Select, sample, cut, join, sort, reformat, and generate CSV files. Contains a large set of commands. 92 | - [Miller](https://github.com/johnkerl/miller) **—** `sed`, `awk`, `cut`, `join` and `sort` for name-indexed data such as CSV and tabular JSON. 93 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 94 | - [pawk](https://github.com/alecthomas/pawk) **—** Process text with AWK-like patterns, but Python code. 95 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 96 | - [qsv](https://github.com/jqnatividad/qsv) **—** Index, slice, analyze, split, and join CSV files. A fork of xsv that adds subcommands and features. 97 | - [ReadStat](https://github.com/WizardMac/ReadStat) **—** Convert statistics package datasets between SAS (SAS7BDAT, XPORT), SPSS (POR, SAV, ZSAV), and Stata (DTA). Convert those formats to CSV and XLSX. Can be used as a C library with bindings for Julia, Python, and R. 98 | - [rows](https://github.com/turicas/rows) **—** A Python library with a [CLI](http://turicas.info/rows/cli/). Convert between a number of [file formats](http://turicas.info/rows/plugins/) for tabular data: CSV, XLS, XLSX, ODS, and others. Query the data (via SQLite). Combine tables. Generate schemas. 99 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 100 | - [scrubcsv](https://github.com/faradayio/scrubcsv) **—** Remove bad lines from a CSV file and normalize the rest. Written in Rust. 101 | - [Skeem](https://github.com/daq-tools/skeem) **—** Infer SQL DDL statements from tabular data. Supports CSV, JSON, JSON Lines, ODS, XLSX, and other formats. 102 | - [tab](http://tkatchev.bitbucket.io/tab/) **—** A non-Turing-complete statically typed programming language for data processing. An alternative to awk. 103 | - [teip](https://github.com/greymd/teip) **—** Select fields, character ranges, or regular expression matches from standard input. Replace them with the output of a command. 104 | - [tv](https://github.com/codechenx/tv) **—** View delimited files in the terminal. 105 | - [xsv](https://github.com/BurntSushi/xsv) **—** Index, slice, analyze, split, and join CSV files. 106 | - [zsv](https://github.com/liquidaty/zsv) **—** Slice, combine, reformat, flatten/unflatten CSV (TSV, DSV) files. Query them with SQL and jq filters. Convert between them, JSON, and SQLite 3. Also a C library. 107 | 108 | ### SQL-based tools 109 | 110 | See the [big comparison list](sql-based.md). It covers 111 | 112 | - AlaSQL CLI 113 | - csvq 114 | - csvsql 115 | - fsql 116 | - Musoq 117 | - q 118 | - RBQL 119 | - rows 120 | - Sqawk (dbohdan) 121 | - sqawk (tjunier) 122 | - Squawk 123 | - termsql 124 | - trdsql 125 | - textql 126 | 127 | 128 | ## HTML 129 | 130 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 131 | - [hred](https://github.com/danburzo/hred) **—** Query XML and HTML with a query language based on CSS selectors. 132 | - [html-xml-utils](https://www.w3.org/Tools/HTML-XML-utils/README) **—** A number of simple utilities (like `hxcopy`, `hxpipe`, `hxunent`, `hxselect`) for manipulating HTML and XML files from [W3C](https://www.w3.org/). Written in C, quite old-fashioned, but still relevant and maintained. 133 | - [htmlq](https://github.com/mgdm/htmlq) **—** Query HTML with CSS selectors. Can remove elements in the output. 134 | - [pup](https://github.com/EricChiang/pup) **—** Query HTML pages with CSS selectors. Static binaries available for releases. Inspired by [jq](#json). 135 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 136 | - [Saxon](http://saxon.sourceforge.net/) **—** Query XML and HTML data with [XPath](https://devhints.io/xpath). [Documentation](http://www.saxonica.com/documentation/#!using-xsl). 137 | - [Temme](https://github.com/shinima/temme) **—** Query HTML with CSS-like selectors to extract JSON. Temme extends CSS selectors with value capture patterns. 138 | - [tidy-html5](http://www.html-tidy.org/) **—** Validate, fix, and reformat HTML(5), XHTML, and XML documents. Convert HTML to XHTML. 139 | - [tq](https://github.com/plainas/tq) **—** Query HTML with CSS selectors. 140 | - [Xidel](http://www.videlibri.de/xidel.html) **—** Query or modify XML and HTML pages with XPath, XQuery 3, and CSS selectors. 141 | - [xml2](https://web.archive.org/web/20160719191401/http://ofb.net/~egnor/xml2/) **—** Convert XML and HTML to and from flat, greppable lists of "path=value" statements. [Source code mirror](https://github.com/clone/xml2). 142 | - [xpe](https://github.com/charmparticle/xpe) **—** Query HTML and XML with XPath expressions. 143 | 144 | 145 | ## JSON 146 | 147 | - [Cels](https://github.com/pacha/cels) **—** Patch JSON, TOML, and YAML with patches in the same format with some special values. Can be used as a Python library. 148 | - [clconf](https://github.com/pastdev/clconf) **—** Merge multiple config files and extract values from them using path string. Supports JSON and YAML. Can be used as a Go library. 149 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 150 | - [emuto](http://kantord.github.io/emuto/) **—** CLI tool similar to jq. Create and manipulate CSV, TSV, and JSON. Can be compiled to JavaScript. 151 | - [fastgron](https://github.com/adamritter/fastgron) **—** Convert JSON to and from GRON, a flat, greppable list of `path=value` statements. Much faster than the original gron on large files. 152 | - [ffs](https://github.com/mgree/ffs) **—** Mount JSON, TOML, and YAML as a Unix filesystem. 153 | - [fx](https://github.com/antonmedv/fx) **—** Run arbitrary JavaScript on JSON input. Standalone binaries available. 154 | - [gojq](https://github.com/itchyny/gojq) **—** A pure Go implementation of jq. Supports YAML input and output. 155 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 156 | - [gron](https://github.com/tomnomnom/gron) **—** Convert JSON to and from GRON, a flat, greppable list of `path=value` statements. 157 | - [jaq](https://github.com/01mf02/jaq) **—** A Rust implementation of jq with minor changes to the language to make it more predictable. 158 | - [JC](https://github.com/kellyjonbrazil/jc) **—** Convert the output of standard command-line tools to JSON. 159 | - [jello](https://github.com/kellyjonbrazil/jello) **—** Query JSON and [JSON Lines](http://jsonlines.org/) with Python code. Output the result in a line-based format suitable for creating Bash arrays. Generate a grep-able schema. 160 | - [jet](https://github.com/borkdude/jet) **—** Convert between JSON, YAML, Clojure's [edn](https://github.com/edn-format/edn), and [Transit](https://github.com/cognitect/transit-format). Transform them with Clojure code. 161 | - [jfq](https://github.com/blgm/jfq) **—** Query and transform JSON with the [JSONata](http://jsonata.org/) language. 162 | - [jj](https://github.com/tidwall/jj) **—** Query and modify values in JSON or JSON Lines with a key path. 163 | - [jl](https://github.com/chrisdone/jl) **—** Query and manipulate JSON using a tiny functional language. 164 | - [jo](https://github.com/jpmens/jo) **—** Create JSON objects from the shell. 165 | - [jp (jmespath)](https://github.com/jmespath/jp) **—** Query JSON with [JMESPath](http://jmespath.org/). 166 | - [jp (sgreben)](https://github.com/sgreben/jp) **—** Plot JSON and CSV data in the terminal. Supports different kinds of plots: bar charts, line charts, scatter plots, histograms, and heatmaps. 167 | - [jplot](https://github.com/rs/jplot) **—** Plot real-time JSON data in the terminal (works with terminals supporting graphic rendering). 168 | - [jq](http://stedolan.github.io/jq/manual/) **—** Create and manipulate JSON with a functional (as in "functional programming") [DSL](https://en.wikipedia.org/wiki/Domain-specific_language). Can convert JSON to other formats. 169 | - [jql](https://github.com/cube2222/jql) **—** Create and manipulate JSON with a Lisp-syntax DSL. 170 | - [jshon](http://kmkeen.com/jshon/) **—** Create and manipulate JSON using [getopt](https://en.wikipedia.org/wiki/Getopt)-style command-line options. 171 | - [json](https://github.com/trentm/json) **—** Run arbitrary JavaScript on JSON input. 172 | - [json-patch](https://github.com/evanphx/json-patch) **—** Apply [RFC 6902](https://tools.ietf.org/html/rfc6902) JSON Patches to JSON. The CLI tool is secondary to a Go library that also creates and applies [RFC 7386](https://tools.ietf.org/html/rfc7396) JSON merge patches. 173 | - [json-table](https://github.com/micha/json-table) **—** Convert nested JSON into CSV or TSV for processing in the shell. 174 | - [json.tool](https://docs.python.org/2/library/json.html) **—** Validate and pretty-print JSON. This module is part of the standard library of Python 2/3 and is likely to be available wherever Python is installed. ([Python 3 docs](https://docs.python.org/3/library/json.html).) 175 | - [json2](https://github.com/vi/json2) **—** Convert JSON to and from flat, greppable lists of "path=value" statements. Modeled after [xml2](#xml). 176 | - [jsonaxe](https://github.com/davvid/jsonaxe) **—** Create and manipulate JSON with a Python-based DSL. Inspired by jq. 177 | - [jsonwatch](https://github.com/dbohdan/jsonwatch) **—** Track changes in JSON data from the command line. Works like `watch -d`. 178 | - [jtbl](https://github.com/kellyjonbrazil/jtbl) **—** Format JSON or JSON Lines as a plain-text table. 179 | - [jtc](https://github.com/ldn-softdev/jtc) **—** Create, manipulate, search, validate JSON with path expressions. Can be used as a C++14 library. 180 | - [lobar](https://github.com/sodiumjoe/lobar) **—** Process JSON and explore it interactively with a wrapper for `lodash.chain()`. An alternative to jq with JavaScript syntax. 181 | - [madato](https://github.com/inosion/madato) **—** Convert ODS and XLSX spreadsheets to JSON, Markdown, and YAML. 182 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 183 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 184 | - [pyp](https://github.com/hauntsaninja/pyp) **—** Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through [tomllib](https://docs.python.org/3.11/library/tomllib.html). 185 | - [qpyson](https://github.com/mpkocher/qpyson) **—** Query and manipulate JSON with Python. 186 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 187 | - [query-json](https://github.com/davesnx/query-json) **—** A faster jq implementation written in Reason Native (OCaml). 188 | - [quicktype](https://github.com/quicktype/quicktype) **—** Infer the underlying model of the JSON and output as types for various programming languages or JSON Schema. CLI and [Web UI](https://app.quicktype.io). 189 | - [ramda-cli](https://github.com/raine/ramda-cli) **—** Manipulate JSON with the [Ramda](https://ramdajs.com/) functional library, and either LiveScript or JavaScript syntax. 190 | - [RecordStream](https://github.com/benbernard/RecordStream) **—** Create, manipulate, and output a stream of records, or JSON objects. Can retrieve records from an SQL database, MongoDB, Atom feeds, XML, and other sources. 191 | - [Remarshal](https://github.com/dbohdan/remarshal) **—** Convert between CBOR, JSON, MessagePack, TOML, and YAML. Validate each of the formats. Pretty-print JSON, TOML, and YAML. 192 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 193 | - [Skeem](https://github.com/daq-tools/skeem) **—** Infer SQL DDL statements from tabular data. Supports CSV, JSON, JSON Lines, ODS, XLSX, and other formats. 194 | - [validjson](http://github.com/martinlindhe/validjson) **—** Validate or pretty-print JSON. 195 | - [xml-to-json-fast](https://github.com/sinelaw/xml-to-json-fast) **—** Convert XML to JSON. Can handle very large XML files. 196 | - [xmljson](https://github.com/engali94/XMLJson) **—** Convert multiple and large XML files to JSON. Written in Swift. 197 | - [yaml-diff-patch](https://github.com/grantila/yaml-diff-patch) **—** Patch YAML with [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) JSON Patches. Generate a JSON Patch from two JSON documents or a YAML and a JSON document. Preserves style. Can be used as a TypeScript library. 198 | - [yamlpath](https://github.com/wwkimball/yamlpath) **—** Query, modify, diff, merge, and validate YAML and JSON with [YAML Paths](https://github.com/wwkimball/yamlpath/wiki/Segments-of-a-YAML-Path). Also a Python library. 199 | 200 | 201 | ## Markdown 202 | 203 | - [mdq](https://github.com/yshavit/mdq) **—** Select elements from Markdown documents using a syntax inspired by Markdown and jq. Match content with regular expressions. Output Markdown or JSON. 204 | 205 | 206 | ## TOML 207 | 208 | With a format converter like Remarshal you can use [JSON](#json) tools to process TOML and YAML, but make sure you do not lose data in the conversion. 209 | 210 | - [Cels](https://github.com/pacha/cels) **—** Patch JSON, TOML, and YAML with patches in the same format with some special values. Can be used as a Python library. 211 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 212 | - [ffs](https://github.com/mgree/ffs) **—** Mount JSON, TOML, and YAML as a Unix filesystem. 213 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 214 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 215 | - [pyp](https://github.com/hauntsaninja/pyp) **—** Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through [tomllib](https://docs.python.org/3.11/library/tomllib.html). 216 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 217 | - [Remarshal](https://github.com/dbohdan/remarshal) **—** Convert between CBOR, JSON, MessagePack, TOML, and YAML. Validate each of the formats. Pretty-print JSON, TOML, and YAML. 218 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 219 | - [taplo-cli](https://github.com/tamasfe/taplo) **—** Query, format, and validate (lint) TOML. 220 | - [validtoml](http://github.com/martinlindhe/validtoml) **—** Validate TOML. 221 | - [yq (kislyuk)](https://github.com/kislyuk/yq) **—** [jq](#json) wrapper for YAML, XML, and TOML. 222 | 223 | 224 | ## XML 225 | 226 | - [csvfix](https://github.com/jheusser/csvfix) **—** A multitool. Compare, filter, normalize, split, and validate CSV files. Reorder, remove, split, and merge fields. Convert data between fixed-width, multi-line, XML, and DSV format. Generate SQL statements. (Unofficial mirror.) 227 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 228 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 229 | - [hred](https://github.com/danburzo/hred) **—** Query XML and HTML with a query language based on CSS selectors. 230 | - [html-xml-utils](https://www.w3.org/Tools/HTML-XML-utils/README) **—** A number of simple utilities (like `hxcopy`, `hxpipe`, `hxunent`, `hxselect`) for manipulating HTML and XML files from [W3C](https://www.w3.org/). Written in C, quite old-fashioned, but still relevant and maintained. 231 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 232 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 233 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 234 | - [Saxon](http://saxon.sourceforge.net/) **—** Query XML and HTML data with [XPath](https://devhints.io/xpath). [Documentation](http://www.saxonica.com/documentation/#!using-xsl). 235 | - [sml2](https://github.com/JFLarvoire/libxml2) **—** Convert between XML and [SML](https://htmlpreview.github.io/?https://github.com/JFLarvoire/libxml2/blob/master/SML_presentation.htm), a simplified XML representation. 236 | - [tidy-html5](http://www.html-tidy.org/) **—** Validate, fix, and reformat HTML(5), XHTML, and XML documents. Convert HTML to XHTML. 237 | - [Xidel](http://www.videlibri.de/xidel.html) **—** Query or modify XML and HTML pages with XPath, XQuery 3, and CSS selectors. 238 | - [xml-to-json-fast](https://github.com/sinelaw/xml-to-json-fast) **—** Convert XML to JSON. Can handle very large XML files. 239 | - [xml2](https://web.archive.org/web/20160719191401/http://ofb.net/~egnor/xml2/) **—** Convert XML and HTML to and from flat, greppable lists of "path=value" statements. [Source code mirror](https://github.com/clone/xml2). 240 | - [xmljson](https://github.com/engali94/XMLJson) **—** Convert multiple and large XML files to JSON. Written in Swift. 241 | - [XMLLint](http://xmlsoft.org/xmllint.html) **—** Query (including XSLT), validate and reformat XML documents. 242 | - [XMLStarlet](http://xmlstar.sourceforge.net/overview.php) **—** Query, modify, and validate XML documents. 243 | - [xpe](https://github.com/charmparticle/xpe) **—** Query HTML and XML with XPath expressions. 244 | - [xq](https://github.com/kislyuk/yq) **—** [jq](#json) wrapper for XML documents. 245 | - [xsltproc](http://xmlsoft.org/XSLT/xsltproc2.html) **—** Transform XML documents using [XSLT](https://www.w3.org/TR/xslt) and [EXSLT](http://exslt.org). 246 | - [yq (kislyuk)](https://github.com/kislyuk/yq) **—** [jq](#json) wrapper for YAML, XML, and TOML. 247 | 248 | ### See also 249 | 250 | - ["Grep and Sed Equivalent for XML Command Line Processing"](http://stackoverflow.com/questions/91791/grep-and-sed-equivalent-for-xml-command-line-processing) on Stack Overflow. 251 | 252 | 253 | ## YAML 254 | 255 | - [Cels](https://github.com/pacha/cels) **—** Patch JSON, TOML, and YAML with patches in the same format with some special values. Can be used as a Python library. 256 | - [clconf](https://github.com/pastdev/clconf) **—** Merge multiple config files and extract values from them using path string. Supports JSON and YAML. Can be used as a Go library. 257 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 258 | - [dy](https://github.com/sampointer/dy) **—** Construct YAML from a directory tree. 259 | - [ffs](https://github.com/mgree/ffs) **—** Mount JSON, TOML, and YAML as a Unix filesystem. 260 | - [gojq](https://github.com/itchyny/gojq) **—** A pure Go implementation of jq. Supports YAML input and output. 261 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 262 | - [jet](https://github.com/borkdude/jet) **—** Convert between JSON, YAML, Clojure's [edn](https://github.com/edn-format/edn), and [Transit](https://github.com/cognitect/transit-format). Transform them with Clojure code. 263 | - [madato](https://github.com/inosion/madato) **—** Convert ODS and XLSX spreadsheets to JSON, Markdown, and YAML. 264 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 265 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 266 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 267 | - [Remarshal](https://github.com/dbohdan/remarshal) **—** Convert between CBOR, JSON, MessagePack, TOML, and YAML. Validate each of the formats. Pretty-print JSON, TOML, and YAML. 268 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 269 | - [shyaml](https://github.com/0k/shyaml) **—** Query YAML. Can output null-terminated strings for use in shell scripts. 270 | - [validyaml](http://github.com/martinlindhe/validyaml) **—** Validate or pretty-print YAML. 271 | - [yaml-diff-patch](https://github.com/grantila/yaml-diff-patch) **—** Patch YAML with [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) JSON Patches. Generate a JSON Patch from two JSON documents or a YAML and a JSON document. Preserves style. Can be used as a TypeScript library. 272 | - [yaml-tools](https://github.com/thecodingmachine/yaml-tools) **—** A set of CLI tools to manipulate YAML files (merge, delete, etc...) with comment preservation, based on [ruamel.yaml](http://yaml.readthedocs.io/en/latest/). 273 | - [yamlpath](https://github.com/wwkimball/yamlpath) **—** Query, modify, diff, merge, and validate YAML and JSON with [YAML Paths](https://github.com/wwkimball/yamlpath/wiki/Segments-of-a-YAML-Path). Also a Python library. 274 | - [yq (kislyuk)](https://github.com/kislyuk/yq) **—** [jq](#json) wrapper for YAML, XML, and TOML. 275 | - [yq (mikefarah)](https://github.com/mikefarah/yq) **—** Query, modify, and merge YAML. Convert to and from JSON. 276 | 277 | 278 | ## Configuration files 279 | 280 | ### .env 281 | 282 | - [dotenvx](https://github.com/dotenvx/dotenvx) 283 | - **Platform:** POSIX, Windows 284 | - **License:** BSD-3-Clause 285 | - **Description:** A CLI tool to manipulate, parse, and inject `.env` files as environment variables. 286 | 287 | ### /etc/hosts 288 | 289 | - [hostctl](https://github.com/guumaster/hostctl) **—** Add and remove entries in `/etc/hosts`. Disable (comment out) and enable (uncomment) entries. Idempotent. Preserves arbitrary comments above its section of the hosts file. Works with groups of entries called "profiles". 290 | - [hostess](https://github.com/cbednarski/hostess) **—** Add and remove entries in `/etc/hosts`. Disable (comment out) and enable (uncomment) entries. Check if a hostname exists. Reformat the hosts file. Convert the entries to JSON. Idempotent. Removes arbitrary comments. 291 | - [hosts](https://gitlab.com/dbohdan/hosts) **—** Add and remove entries in `/etc/hosts`. Change a hostname's IP address. Idempotent. Preserves arbitrary comments. Can be used as a Tcl library. 292 | 293 | ### INI 294 | 295 | - [cfget](https://packages.debian.org/source/buster/cfget) 296 | - **Platform:** Any with Python 2.6-2.7? 297 | - **License:** GPL-2.0-or-later 298 | - **Description:** Retrieve properties as shell script commands to set the corresponding variables (with `--dump exports`). Retrieve properties' values as plain text. Substitute values from an INI file in an Autoconf-style template. Supports plug-ins. Chokes on section names and keys with spaces. 299 | - [confget](https://devel.ringlet.net/textproc/confget/) 300 | - **Platform:** Free/Net/OpenBSD, Linux, likely others 301 | - **License:** BSD-2-Clause 302 | - **Description:** Retrieve properties and sections as shell script commands to set the corresponding variables. Retrieve properties' values as plain text. Check for existence of properties. List sections. Find values that match a pattern. Read-only. Has a C, Python, and Rust implementation. The Rust implementation can be installed with `cargo install confget`. 303 | - [crudini](https://github.com/pixelb/crudini/) 304 | - **Platform:** Any with Python 2.6–2.7 or 3.x 305 | - **License:** GPL-2.0 306 | - **Description:** Retrieve properties and sections as INI fragments or shell script commands to set the corresponding variables. Retrieve properties' values as plain text. Set properties. Remove properties and sections. Create empty sections. Merge INI files. Changes files in place. 307 | - [inicomp](https://github.com/JFLarvoire/SysToolsLib/blob/HEAD/C/SRC/inicomp.c) 308 | - **Platform:** Windows, POSIX 309 | - **License:** Apache-2.0 310 | - **Description:** Compare INI (and also Windows .reg) files. 311 | - [IniFile](http://www.horstmuc.de/wbat32.htm#inifile) 312 | - **Platform:** Windows (x86, x86-64), [MS-DOS](http://www.horstmuc.de/div.htm#inifile) 313 | - **License:** Closed-source freeware 314 | - **Description:** Retrieve properties and sections as batch file commands to set the corresponding variables. Set properties. Remove properties and sections. Changes files in place. 315 | - [initool](https://github.com/dbohdan/initool) 316 | - **Platform:** FreeBSD, Linux, Windows 317 | - **License:** MIT 318 | - **Description:** Retrieve properties and sections as INI fragments. Retrieve properties' values as plain text. Set properties. Check for existence of properties and sections. Remove properties and sections. Outputs the updated INI file. 319 | - [Nushell (`from ini`)](https://github.com/nushell/nushell) 320 | - **Platform:** Free/Net/OpenBSD, Linux, macOS, Windows 321 | - **License:** MIT 322 | - **Description:** Query and transform data with the Nushell language. 323 | - [qq (INI)](https://github.com/JFryy/qq) 324 | - **Platform:** Free/Net/OpenBSD, Linux, macOS, Windows 325 | - **License:** MIT 326 | - **Description:** Query and transform data with jq. Based on gojq. 327 | 328 | ### Multiple formats 329 | 330 | - [Augeas](http://augeas.net) **—** Query and modify [a number of file formats](http://augeas.net/stock_lenses.html). Not all of the formats are equally well supported by Augeas and for some only a limited subset of all valid files can be parsed. 331 | - [Elektra](http://libelektra.org) **—** Query and modify [configuration files](https://github.com/ElektraInitiative/libelektra/tree/master/src/plugins). Shares Augeas' limitations when it comes to application-specific configuration files (it uses the same lenses), but has better support for generic formats such as JSON and INI. 332 | 333 | 334 | ## Log files 335 | 336 | - [lnav](https://lnav.org) **—** Query and watch log files. Has batch and interactive mode. Supported formats include the Common Log Format, CUPS page_log, syslog, strace, and generic timestamped messages. Can perform SQL queries. 337 | - [Squawk](https://github.com/samuel/squawk) **—** Query Apache and Nginx log files. See the [SQL-based tool comparison](sql-based.md). 338 | 339 | 340 | ## Multiformat tools 341 | 342 | Tools that support multiple input formats. 343 | Programs that convert between only two formats in both directions are excluded. 344 | We only count JSON support that is separate from YAML. 345 | 346 | - [Augeas](http://augeas.net) **—** Query and modify [a number of file formats](http://augeas.net/stock_lenses.html). Not all of the formats are equally well supported by Augeas and for some only a limited subset of all valid files can be parsed. 347 | - [Cels](https://github.com/pacha/cels) **—** Patch JSON, TOML, and YAML with patches in the same format with some special values. Can be used as a Python library. 348 | - [clconf](https://github.com/pastdev/clconf) **—** Merge multiple config files and extract values from them using path string. Supports JSON and YAML. Can be used as a Go library. 349 | - [csvfix](https://github.com/jheusser/csvfix) **—** A multitool. Compare, filter, normalize, split, and validate CSV files. Reorder, remove, split, and merge fields. Convert data between fixed-width, multi-line, XML, and DSV format. Generate SQL statements. (Unofficial mirror.) 350 | - [csvtk](https://github.com/shenwei356/csvtk) **—** Search, sample, cut, join, transpose, and sort CSV/TSV files. Rename columns. Replace fields and generate new fiends from existing fields. Plot data as vector or raster histograms and box, line, and scatter plots. Convert CSV to Markdown. Convert XLSX to CSV. Split XLSX sheets. 351 | - [dasel](https://github.com/TomWright/dasel) **—** Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases. 352 | - [Elektra](http://libelektra.org) **—** Query and modify [configuration files](https://github.com/ElektraInitiative/libelektra/tree/master/src/plugins). Shares Augeas' limitations when it comes to application-specific configuration files (it uses the same lenses), but has better support for generic formats such as JSON and INI. 353 | - [emuto](http://kantord.github.io/emuto/) **—** CLI tool similar to jq. Create and manipulate CSV, TSV, and JSON. Can be compiled to JavaScript. 354 | - [ffs](https://github.com/mgree/ffs) **—** Mount JSON, TOML, and YAML as a Unix filesystem. 355 | - [frawk](https://github.com/ezrosent/frawk) **—** A Rust implementation of a language partially compatible with AWK that supports [parallelism](https://github.com/ezrosent/frawk/blob/master/info/parallelism.md) and CSV input and output. frawk is an awk-derived language with a CSV mode for input and for output. 356 | - [GoAWK](https://github.com/benhoyt/goawk) **—** A cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows. GoAWK is an awk implementation that adds a CSV mode for input and for output. 357 | - [gojq](https://github.com/itchyny/gojq) **—** A pure Go implementation of jq. Supports YAML input and output. 358 | - [Graphtage](https://github.com/trailofbits/graphtage) **—** Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library. 359 | - [hred](https://github.com/danburzo/hred) **—** Query XML and HTML with a query language based on CSS selectors. 360 | - [html-xml-utils](https://www.w3.org/Tools/HTML-XML-utils/README) **—** A number of simple utilities (like `hxcopy`, `hxpipe`, `hxunent`, `hxselect`) for manipulating HTML and XML files from [W3C](https://www.w3.org/). Written in C, quite old-fashioned, but still relevant and maintained. 361 | - [jet](https://github.com/borkdude/jet) **—** Convert between JSON, YAML, Clojure's [edn](https://github.com/edn-format/edn), and [Transit](https://github.com/cognitect/transit-format). Transform them with Clojure code. 362 | - [jp (sgreben)](https://github.com/sgreben/jp) **—** Plot JSON and CSV data in the terminal. Supports different kinds of plots: bar charts, line charts, scatter plots, histograms, and heatmaps. 363 | - [lnav](https://lnav.org) **—** Query and watch log files. Has batch and interactive mode. Supported formats include the Common Log Format, CUPS page_log, syslog, strace, and generic timestamped messages. Can perform SQL queries. 364 | - [madato](https://github.com/inosion/madato) **—** Convert ODS and XLSX spreadsheets to JSON, Markdown, and YAML. 365 | - [Mario](https://github.com/python-mario/mario) **—** Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code. 366 | - [Nushell](https://github.com/nushell/nushell) **—** A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats. 367 | - [pyp](https://github.com/hauntsaninja/pyp) **—** Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through [tomllib](https://docs.python.org/3.11/library/tomllib.html). 368 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 369 | - [RecordStream](https://github.com/benbernard/RecordStream) **—** Create, manipulate, and output a stream of records, or JSON objects. Can retrieve records from an SQL database, MongoDB, Atom feeds, XML, and other sources. 370 | - [ReadStat](https://github.com/WizardMac/ReadStat) **—** Convert statistics package datasets between SAS (SAS7BDAT, XPORT), SPSS (POR, SAV, ZSAV), and Stata (DTA). Convert those formats to CSV and XLSX. Can be used as a C library with bindings for Julia, Python, and R. 371 | - [Remarshal](https://github.com/dbohdan/remarshal) **—** Convert between CBOR, JSON, MessagePack, TOML, and YAML. Validate each of the formats. Pretty-print JSON, TOML, and YAML. 372 | - [rows](https://github.com/turicas/rows) **—** A Python library with a [CLI](http://turicas.info/rows/cli/). Convert between a number of [file formats](http://turicas.info/rows/plugins/) for tabular data: CSV, XLS, XLSX, ODS, and others. Query the data (via SQLite). Combine tables. Generate schemas. 373 | - [rq](https://github.com/dflemstr/rq) **—** Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text. 374 | - [Saxon](http://saxon.sourceforge.net/) **—** Query XML and HTML data with [XPath](https://devhints.io/xpath). [Documentation](http://www.saxonica.com/documentation/#!using-xsl). 375 | - [Skeem](https://github.com/daq-tools/skeem) **—** Infer SQL DDL statements from tabular data. Supports CSV, JSON, JSON Lines, ODS, XLSX, and other formats. 376 | - [tidy-html5](http://www.html-tidy.org/) **—** Validate, fix, and reformat HTML(5), XHTML, and XML documents. Convert HTML to XHTML. 377 | - [VisiData](https://github.com/saulpw/visidata) **—** Explore interactively data in TSV, CSV, XLS, XLSX, HDF5, JSON, and [other formats](http://visidata.org/man/#loaders). [Introduction](https://jsvine.github.io/intro-to-visidata/). 378 | - [Xidel](http://www.videlibri.de/xidel.html) **—** Query or modify XML and HTML pages with XPath, XQuery 3, and CSS selectors. 379 | - [xml2](https://web.archive.org/web/20160719191401/http://ofb.net/~egnor/xml2/) **—** Convert XML and HTML to and from flat, greppable lists of "path=value" statements. [Source code mirror](https://github.com/clone/xml2). 380 | - [xmljson](https://github.com/engali94/XMLJson) **—** Convert multiple and large XML files to JSON. Written in Swift. 381 | - [xpe](https://github.com/charmparticle/xpe) **—** Query HTML and XML with XPath expressions. 382 | - [yaml-diff-patch](https://github.com/grantila/yaml-diff-patch) **—** Patch YAML with [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) JSON Patches. Generate a JSON Patch from two JSON documents or a YAML and a JSON document. Preserves style. Can be used as a TypeScript library. 383 | - [yamlpath](https://github.com/wwkimball/yamlpath) **—** Query, modify, diff, merge, and validate YAML and JSON with [YAML Paths](https://github.com/wwkimball/yamlpath/wiki/Segments-of-a-YAML-Path). Also a Python library. 384 | - [yq (kislyuk)](https://github.com/kislyuk/yq) **—** [jq](#json) wrapper for YAML, XML, and TOML. 385 | - [zsv](https://github.com/liquidaty/zsv) **—** Slice, combine, reformat, flatten/unflatten CSV (TSV, DSV) files. Query them with SQL and jq filters. Convert between them, JSON, and SQLite 3. Also a C library. 386 | 387 | 388 | ## Templating for structured text 389 | 390 | Listed below are restricted programming language interpreters and templating tools that produce structured text output. They are generally intended to remove repetition in configuration files. They are distinct from unstructed templating tools like the `jinja2` CLI program, which should not be added to this table. 391 | 392 | - [CUE](https://github.com/cuelang/cue) 393 | - **Output format:** JSON 394 | - **Turing-complete:** No 395 | - **Syntax:** Extended JSON 396 | - **I/O:** ? 397 | - **Description:** A constraint language for JSON configuration data. Can generate and validates JSON. 398 | - [Dhall](https://dhall-lang.org/) 399 | - **Output format:** JSON, YAML 400 | - **Turing-complete:** No 401 | - **Syntax:** Haskell-inspired 402 | - **I/O:** Limited to importing libraries from files and HTTP(S) URLs (with protection against leaking your data to the server) 403 | - **Description:** A statically-typed functional configuration language. Has a standard formatting tool. 404 | - [jk](https://github.com/jkcfg/jk) 405 | - **Output format:** JSON, YAML, plain text 406 | - **Turing-complete:** Yes 407 | - **Syntax:** JavaScript 408 | - **I/O:** Disk I/O 409 | - **Description:** Generate configuration files using JavaScript (V8 VM). 410 | - [Jsonnet](https://jsonnet.org/) 411 | - **Output format:** JSON, INI, XML, YAML, plain text 412 | - **Turing-complete:** Yes 413 | - **Syntax:** Extended JSON 414 | - **I/O:** None 415 | - **Description:** A functional configuration language. Has a standard formatting tool. 416 | - [Nickel](https://nickel-lang.org/) 417 | - **Output format:** JSON, TOML, YAML 418 | - **Turing-complete:** Yes 419 | - **Syntax:** Inspired by ML and JSON 420 | - **I/O:** Limited input is to be implemented 421 | - **Description:** A gradually-typed functional configuration language with contracts. 422 | - [Pkl](https://pkl-lang.org/) 423 | - **Output format:** JSON, YAML, macOS property list, Java `.properties` 424 | - **Turing-complete:** Yes 425 | - **Syntax:** Swift-inspired 426 | - **I/O:** The CLI can read environment variables and files, `GET` HTTP(S) URLs. It can import modules from files and HTTP(S) URLs. 427 | - **Description:** A command-line tool, Java library, and build tool plugin. Can generate code for Go, Java, Kotlin, and Swift. ["Pkl vs. Other Config Languages"](https://pkl-lang.org/main/current/introduction/comparison.html#other-config-langs). 428 | - [rjsone](https://github.com/wryun/rjsone) 429 | - **Output format:** JSON, YAML 430 | - **Turing-complete:** No? 431 | - **Syntax:** Extended JSON 432 | - **I/O:** None 433 | - **Description:** A CLI tool for the [JSON-e](https://github.com/taskcluster/json-e) templating language. 434 | - [ytt](https://get-ytt.io/) 435 | - **Output format:** YAML 436 | - **Turing-complete:** No 437 | - **Syntax:** YAML/Python hybrid 438 | - **I/O:** None? 439 | - **Description:** A templating tool for YAML built upon the [Starlark](https://github.com/bazelbuild/starlark) configuration language. 440 | 441 | ### See also 442 | 443 | - [A comparison table of Nickel and other configuration languages](https://github.com/tweag/nickel/blob/master/RATIONALE.md#comparison-with-other-configuration-languages). 444 | 445 | 446 | ## Extra: interactive TUIs 447 | 448 | - [argrelay](https://github.com/argrelay/argrelay) **—** Implement tab completion for commands in Bash based on search of indexed data through a background server. 449 | - [jid](https://github.com/simeji/jid) **—** Explore JSON interactively with filtering queries like jq. 450 | - [jiq](https://github.com/fiatjaf/jiq) **—** Explore JSON interactively with jq. Requires jq. 451 | - [lobar](https://github.com/sodiumjoe/lobar) **—** Process JSON and explore it interactively with a wrapper for `lodash.chain()`. An alternative to jq with JavaScript syntax. 452 | - [otree](https://github.com/fioncat/otree) **—** Expore JSON, TOML, and YAML using a TUI tree widget. 453 | - [qq](https://github.com/JFryy/qq) **—** Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library. 454 | - [sc-im](https://github.com/andmarti1424/sc-im) **—** A Vim-like spreadsheet calculator for CSV and TSV files. 455 | - [VisiData](https://github.com/saulpw/visidata) **—** Explore interactively data in TSV, CSV, XLS, XLSX, HDF5, JSON, and [other formats](http://visidata.org/man/#loaders). [Introduction](https://jsvine.github.io/intro-to-visidata/). 456 | 457 | 458 | ## Extra: CLIs for single-file databases 459 | 460 | - [Firebird](https://firebirdsql.org/) 461 | - **Description:** Firebird is a FOSS database that can be used from a single file, like SQLite. "isql is a program that allows the user to issue arbitrary SQL commands". 462 | - **File format:** Binary 463 | - [Fsdb](https://www.isi.edu/~johnh/SOFTWARE/FSDB/perl-Fsdb-2.69_README.html) 464 | - **Description:** A flat-file database for shell scripting. 465 | - **File format:** Text-based, TSV with a header or "key: value" 466 | - [GNU Recutils](http://www.gnu.org/software/recutils/) 467 | - **Description:** "[A] set of tools and libraries to access human-editable, plain text databases called recfiles." 468 | - **File format:** Text-based, roughly "key: value" 469 | - [SDB](https://github.com/radare/sdb) 470 | - **Description:** "[A] simple string key/value database based on djb's cdb disk storage and supports JSON and arrays introspection." 471 | - **File format:** Binary 472 | - [sqlite3(1)](https://www.sqlite.org/cli.html) 473 | - **Description:** "[A] simple command-line utility [...] that allows the user to manually enter and execute SQL statements against an SQLite database." 474 | - **File format:** Binary 475 | 476 | 477 | ## License 478 | 479 | The contents of this document is licensed under the [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). By contributing you agree to release your contribution under this license. 480 | 481 | 482 | ## Disclosure 483 | 484 | [csv2html](https://github.com/dbohdan/csv2html), [hosts](https://gitlab.com/dbohdan/hosts), [Sqawk](https://github.com/dbohdan/sqawk), [jsonwatch](https://github.com/dbohdan/jsonwatch), [Remarshal](https://github.com/remarshal-project/remarshal), and [initool](https://github.com/dbohdan/initool) are developed by the curator of this document. 485 | -------------------------------------------------------------------------------- /README.md.jinja: -------------------------------------------------------------------------------- 1 | {% from "macros.md.jinja" import items -%} 2 | 3 | # Structured text tools 4 | 5 | The following is a list of text-based file formats and command-line tools for manipulating each. 6 | 7 | 8 | ## Contents 9 | 10 | - [awk-like](#awk-like) 11 | - [awk](#awk) 12 | - [POSIX commands](#posix-commands) 13 | - [SQL-based tools](#sql-based-tools) 14 | - [Other tools](#other-tools) 15 | - [CSV](#csv) 16 | - [HTML](#html) 17 | - [JSON](#json) 18 | - [Markdown](#markdown) 19 | - [TOML](#toml) 20 | - [XML](#xml) 21 | - [YAML](#yaml) 22 | - [Configuration files](#configuration-files) 23 | - [.env](#env) 24 | - [/etc/hosts](#etchosts) 25 | - [INI](#ini) 26 | - [Multiple formats](#multiple-formats) 27 | - [Log files](#log-files) 28 | - [Multiformat tools](#multiformat-tools) 29 | - [Templating for structured text](#templating-for-structured-text) 30 | - [Extra: interactive TUIs](#extra-interactive-tuis) 31 | - [Extra: CLIs for single-file databases](#extra-clis-for-single-file-databases) 32 | - [License](#license) 33 | - [Disclosure](#disclosure) 34 | 35 | 36 | ## awk-like 37 | 38 | Tools that work with lines of fields separated by delimiters but do not necessarily support [CSV field quoting](https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules). 39 | 40 | ### awk 41 | 42 | AWK/awk is a programming language and a POSIX-standard command-line tool. (You will sometimes see "awk" used for the tool and "AWK" for the language. This document follows this convention. GNU Awk uses "Awk".) If you run Linux, macOS, or a BSD, you almost certainly have it installed. See below for Windows. 43 | 44 | - If you already know how to program, the nawk [man page](https://www.freebsd.org/cgi/man.cgi?query=nawk&sektion=1) is a great way to learn AWK quickly. What you learn from it will apply to other implementations on different platforms. Read it first if you feel overwhelmed by the sheer size of the [GNU Awk manual](https://www.gnu.org/software/gawk/manual/gawk.html). 45 | - [awk.info archive](https://web.archive.org/web/20160505033644/http://awk.info/) **—** an extensive resource on Awk. 46 | - ["AWK Vs NAWK Vs GAWK"](https://www.thegeekstuff.com/2011/06/awk-nawk-gawk/) **—** a comparison of features present in different implementations. 47 | - [busybox-w32](https://frippery.org/busybox/) includes a full implementation of POSIX awk and other tools like `sed` in a single Windows executable. 48 | - [frawk](https://github.com/ezrosent/frawk) is a Rust implementation of a language partially compatible with AWK that supports [parallelism](https://github.com/ezrosent/frawk/blob/master/info/parallelism.md) and CSV input and output. 49 | - [GNU Awk 5 binaries for Windows](https://sourceforge.net/projects/ezwinports/files/) by [EZWinPorts](https://www.gnu.org/software/emacs/manual/html_node/efaq-w32/EZWinPorts.html). 50 | - [GoAWK](https://github.com/benhoyt/goawk) is a cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows. 51 | 52 | ### POSIX commands 53 | 54 | - `comm` **—** Select the lines common to two sorted files or the lines contained in only one of them. (Manual: `man 1 comm` on your system, [GNU](https://linux.die.net/man/1/comm), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=comm&sektion=1).) 55 | - `cut` **—** Select portions of each line in one or more files. (Manual: `man 1 cut`, [GNU](https://linux.die.net/man/1/cut), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=cut&sektion=1).) 56 | - `grep` **—** Select the lines that match or do not match a pattern from one or more files. (Manual: `man 1 grep`, [GNU](https://linux.die.net/man/1/grep), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=grep&sektion=1).) 57 | - `join` **—** Take two files sorted by a common field and join their lines on the value of that field. Lines with values that do not appear in the other file are discarded. (Manual: `man 1 join`, [GNU](https://linux.die.net/man/1/join), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=join&sektion=1).) 58 | - `paste` **—** Combine several consecutive lines in a text file into one. (Manual: `man 1 paste`, [GNU](https://linux.die.net/man/1/paste), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=paste&sektion=1).) 59 | - `sort` **—** Sort lines by key fields. (Manual: `man 1 sort`, [GNU](https://linux.die.net/man/1/sort), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=sort&sektion=1).) 60 | - `uniq` **—** Find or remove repeated lines. (Manual: `man 1 uniq`, [GNU](https://linux.die.net/man/1/uniq), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=uniq&sektion=1).) 61 | 62 | ### Other tools 63 | 64 | {{ items("awk-other") }} 65 | 66 | 67 | ## CSV 68 | 69 | CSV, TSV, and other delimiter-separated value formats. Tools belong on this list if they support [field quoting](https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules). 70 | 71 | {{ items("csv") }} 72 | 73 | ### SQL-based tools 74 | 75 | See the [big comparison list](sql-based.md). It covers 76 | 77 | - AlaSQL CLI 78 | - csvq 79 | - csvsql 80 | - fsql 81 | - Musoq 82 | - q 83 | - RBQL 84 | - rows 85 | - Sqawk (dbohdan) 86 | - sqawk (tjunier) 87 | - Squawk 88 | - termsql 89 | - trdsql 90 | - textql 91 | 92 | 93 | ## HTML 94 | 95 | {{ items("html") }} 96 | 97 | 98 | ## JSON 99 | 100 | {{ items("json") }} 101 | 102 | 103 | ## Markdown 104 | 105 | {{ items("markdown") }} 106 | 107 | 108 | ## TOML 109 | 110 | With a format converter like Remarshal you can use [JSON](#json) tools to process TOML and YAML, but make sure you do not lose data in the conversion. 111 | 112 | {{ items("toml") }} 113 | 114 | 115 | ## XML 116 | 117 | {{ items("xml") }} 118 | 119 | ### See also 120 | 121 | - ["Grep and Sed Equivalent for XML Command Line Processing"](http://stackoverflow.com/questions/91791/grep-and-sed-equivalent-for-xml-command-line-processing) on Stack Overflow. 122 | 123 | 124 | ## YAML 125 | 126 | {{ items("yaml") }} 127 | 128 | 129 | ## Configuration files 130 | 131 | ### .env 132 | 133 | {{ items(".env") }} 134 | 135 | ### /etc/hosts 136 | 137 | {{ items("hosts") }} 138 | 139 | ### INI 140 | 141 | {{ items("ini") }} 142 | 143 | ### Multiple formats 144 | 145 | {{ items("config") }} 146 | 147 | 148 | ## Log files 149 | 150 | {{ items("log") }} 151 | 152 | 153 | ## Multiformat tools 154 | 155 | Tools that support multiple input formats. 156 | Programs that convert between only two formats in both directions are excluded. 157 | We only count JSON support that is separate from YAML. 158 | 159 | {{ items("multiformat") }} 160 | 161 | 162 | ## Templating for structured text 163 | 164 | Listed below are restricted programming language interpreters and templating tools that produce structured text output. They are generally intended to remove repetition in configuration files. They are distinct from unstructed templating tools like the `jinja2` CLI program, which should not be added to this table. 165 | 166 | {{ items("template") }} 167 | 168 | ### See also 169 | 170 | - [A comparison table of Nickel and other configuration languages](https://github.com/tweag/nickel/blob/master/RATIONALE.md#comparison-with-other-configuration-languages). 171 | 172 | 173 | ## Extra: interactive TUIs 174 | 175 | {{ items("tui") }} 176 | 177 | 178 | ## Extra: CLIs for single-file databases 179 | 180 | {{ items("single-file-db") }} 181 | 182 | 183 | ## License 184 | 185 | The contents of this document is licensed under the [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). By contributing you agree to release your contribution under this license. 186 | 187 | 188 | ## Disclosure 189 | 190 | [csv2html](https://github.com/dbohdan/csv2html), [hosts](https://gitlab.com/dbohdan/hosts), [Sqawk](https://github.com/dbohdan/sqawk), [jsonwatch](https://github.com/dbohdan/jsonwatch), [Remarshal](https://github.com/remarshal-project/remarshal), and [initool](https://github.com/dbohdan/initool) are developed by the curator of this document. 191 | -------------------------------------------------------------------------------- /data/projects.toml: -------------------------------------------------------------------------------- 1 | ["\"AWK Vs NAWK Vs GAWK\""] 2 | url = "https://www.thegeekstuff.com/2011/06/awk-nawk-gawk/" 3 | descr = "A comparison of features present in different implementations." 4 | tags = ["awk"] 5 | 6 | [argrelay] 7 | url = "https://github.com/argrelay/argrelay" 8 | descr = "Implement tab completion for commands in Bash based on search of indexed data through a background server." 9 | tags = ["tui"] 10 | 11 | [Augeas] 12 | url = "http://augeas.net" 13 | descr = "Query and modify [a number of file formats](http://augeas.net/stock_lenses.html). Not all of the formats are equally well supported by Augeas and for some only a limited subset of all valid files can be parsed." 14 | tags = ["config", "multiformat"] 15 | 16 | ["awk.info archive"] 17 | url = "https://web.archive.org/web/20160505033644/http://awk.info/" 18 | descr = "an extensive resource on Awk." 19 | tags = ["awk"] 20 | 21 | [busybox-w32] 22 | url = "https://frippery.org/busybox/" 23 | descr = "includes a full implementation of POSIX awk and other tools like `sed` in a single Windows executable." 24 | tags = ["awk"] 25 | 26 | [Cels] 27 | url = "https://github.com/pacha/cels" 28 | descr = "Patch JSON, TOML, and YAML with patches in the same format with some special values. Can be used as a Python library." 29 | tags = ["json", "toml", "yaml", "multiformat"] 30 | 31 | [clconf] 32 | url = "https://github.com/pastdev/clconf" 33 | descr = "Merge multiple config files and extract values from them using path string. Supports JSON and YAML. Can be used as a Go library." 34 | tags = ["yaml", "json", "multiformat"] 35 | 36 | [csv-nix-tools] 37 | url = "https://github.com/mslusarz/csv-nix-tools" 38 | descr = "List \\*nix system information such as environment variables, files, processes, network connections, users as CSV. Manipulate and pretty-print CSV. Execute CSV rows as commands." 39 | tags = ["csv"] 40 | 41 | [csv2html] 42 | url = "https://github.com/dbohdan/csv2html" 43 | descr = "Convert CSV to HTML tables." 44 | tags = ["csv"] 45 | 46 | [csv2md] 47 | url = "https://github.com/pstaender/csv2md" 48 | descr = "Convert CSV to Markdown tables." 49 | tags = ["csv"] 50 | 51 | [csvfaker] 52 | url = "https://github.com/pereorga/csvfaker" 53 | descr = "Generate CSV files with fake data. Supports different types of fake data in different locales: names, cities, jobs, email addresses, and others." 54 | tags = ["csv"] 55 | 56 | [csvfix] 57 | url = "https://github.com/jheusser/csvfix" 58 | descr = "A multitool. Compare, filter, normalize, split, and validate CSV files. Reorder, remove, split, and merge fields. Convert data between fixed-width, multi-line, XML, and DSV format. Generate SQL statements. (Unofficial mirror.)" 59 | tags = ["csv", "xml", "multiformat"] 60 | 61 | [csvkit] 62 | url = "https://github.com/wireservice/csvkit" 63 | descr = "csvkit is a suite of command-line tools for converting to and working with CSV: convert, clean, cut, grep, join, sort, stack, format, render, query, analyze, etc." 64 | tags = ["csv"] 65 | 66 | [csvquote] 67 | url = "https://github.com/dbro/csvquote" 68 | descr = "Transform CSV to and from a format processable with [awk-like](#awk-like) tools." 69 | tags = ["csv", "awk-other"] 70 | 71 | [csvtk] 72 | url = "https://github.com/shenwei356/csvtk" 73 | descr = "Search, sample, cut, join, transpose, and sort CSV/TSV files. Rename columns. Replace fields and generate new fiends from existing fields. Plot data as vector or raster histograms and box, line, and scatter plots. Convert CSV to Markdown. Convert XLSX to CSV. Split XLSX sheets." 74 | tags = ["csv", "multiformat"] 75 | 76 | [CSVtoTable] 77 | url = "https://github.com/vividvilla/csvtotable" 78 | descr = "Convert CSV to a searchable and sortable HTML table." 79 | tags = ["csv"] 80 | 81 | [dasel] 82 | url = "https://github.com/TomWright/dasel" 83 | descr = "Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases." 84 | tags = ["csv", "xml", "yaml", "toml", "json", "multiformat"] 85 | 86 | [dy] 87 | url = "https://github.com/sampointer/dy" 88 | descr = "Construct YAML from a directory tree." 89 | tags = ["yaml"] 90 | 91 | ["eBay's TSV utilities"] 92 | url = "https://github.com/eBay/tsv-utils" 93 | descr = "Filtering, statistics, sampling, joins and other operations on TSV files. High performance, especially good for large datasets. Written in D." 94 | tags = ["csv"] 95 | 96 | [Elektra] 97 | url = "http://libelektra.org" 98 | descr = "Query and modify [configuration files](https://github.com/ElektraInitiative/libelektra/tree/master/src/plugins). Shares Augeas' limitations when it comes to application-specific configuration files (it uses the same lenses), but has better support for generic formats such as JSON and INI." 99 | tags = ["config", "multiformat"] 100 | 101 | [emuto] 102 | url = "http://kantord.github.io/emuto/" 103 | descr = "CLI tool similar to jq. Create and manipulate CSV, TSV, and JSON. Can be compiled to JavaScript." 104 | tags = ["csv", "json", "multiformat"] 105 | 106 | [fastgron] 107 | url = "https://github.com/adamritter/fastgron" 108 | descr = "Convert JSON to and from GRON, a flat, greppable list of `path=value` statements. Much faster than the original gron on large files." 109 | tags = ["json"] 110 | 111 | [ffs] 112 | url = "https://github.com/mgree/ffs" 113 | descr = "Mount JSON, TOML, and YAML as a Unix filesystem." 114 | tags = ["json", "toml", "yaml", "multiformat"] 115 | 116 | [frawk] 117 | url = "https://github.com/ezrosent/frawk" 118 | descr = "A Rust implementation of a language partially compatible with AWK that supports [parallelism](https://github.com/ezrosent/frawk/blob/master/info/parallelism.md) and CSV input and output. frawk is an awk-derived language with a CSV mode for input and for output." 119 | tags = ["csv", "awk", "multiformat"] 120 | 121 | [fx] 122 | url = "https://github.com/antonmedv/fx" 123 | descr = "Run arbitrary JavaScript on JSON input. Standalone binaries available." 124 | tags = ["json"] 125 | 126 | ["GNU datamash"] 127 | url = "https://www.gnu.org/software/datamash/" 128 | descr = "Perform statistical operations on text input." 129 | tags = ["awk-other"] 130 | 131 | [GoAWK] 132 | url = "https://github.com/benhoyt/goawk" 133 | descr = "A cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows. GoAWK is an awk implementation that adds a CSV mode for input and for output." 134 | tags = ["csv", "awk", "multiformat"] 135 | 136 | [gojq] 137 | url = "https://github.com/itchyny/gojq" 138 | descr = "A pure Go implementation of jq. Supports YAML input and output." 139 | tags = ["yaml", "json", "multiformat"] 140 | 141 | [Graphtage] 142 | url = "https://github.com/trailofbits/graphtage" 143 | descr = "Compare and merge tree-like structures semantically. Supports JSON, JSON5, XML, HTML, YAML, and CSV. Can be used as a Python library." 144 | tags = ["csv", "xml", "html", "yaml", "json", "multiformat"] 145 | 146 | [gron] 147 | url = "https://github.com/tomnomnom/gron" 148 | descr = "Convert JSON to and from GRON, a flat, greppable list of `path=value` statements." 149 | tags = ["json"] 150 | 151 | [Hawk] 152 | url = "https://github.com/gelisam/hawk" 153 | descr = "Transform text from the command-line using Haskell expressions." 154 | tags = ["awk-other"] 155 | 156 | [hostctl] 157 | url = "https://github.com/guumaster/hostctl" 158 | descr = "Add and remove entries in `/etc/hosts`. Disable (comment out) and enable (uncomment) entries. Idempotent. Preserves arbitrary comments above its section of the hosts file. Works with groups of entries called \"profiles\"." 159 | tags = ["hosts"] 160 | 161 | [hostess] 162 | url = "https://github.com/cbednarski/hostess" 163 | descr = "Add and remove entries in `/etc/hosts`. Disable (comment out) and enable (uncomment) entries. Check if a hostname exists. Reformat the hosts file. Convert the entries to JSON. Idempotent. Removes arbitrary comments." 164 | tags = ["hosts"] 165 | 166 | [hosts] 167 | url = "https://gitlab.com/dbohdan/hosts" 168 | descr = "Add and remove entries in `/etc/hosts`. Change a hostname's IP address. Idempotent. Preserves arbitrary comments. Can be used as a Tcl library." 169 | tags = ["hosts"] 170 | 171 | [hred] 172 | url = "https://github.com/danburzo/hred" 173 | descr = "Query XML and HTML with a query language based on CSS selectors." 174 | tags = ["xml", "html", "multiformat"] 175 | 176 | [html-xml-utils] 177 | url = "https://www.w3.org/Tools/HTML-XML-utils/README" 178 | descr = "A number of simple utilities (like `hxcopy`, `hxpipe`, `hxunent`, `hxselect`) for manipulating HTML and XML files from [W3C](https://www.w3.org/). Written in C, quite old-fashioned, but still relevant and maintained." 179 | tags = ["xml", "html", "multiformat"] 180 | 181 | [htmlq] 182 | url = "https://github.com/mgdm/htmlq" 183 | descr = "Query HTML with CSS selectors. Can remove elements in the output." 184 | tags = ["html"] 185 | 186 | [jaq] 187 | url = "https://github.com/01mf02/jaq" 188 | descr = "A Rust implementation of jq with minor changes to the language to make it more predictable." 189 | tags = ["json"] 190 | 191 | [JC] 192 | url = "https://github.com/kellyjonbrazil/jc" 193 | descr = "Convert the output of standard command-line tools to JSON." 194 | tags = ["json"] 195 | 196 | [jello] 197 | url = "https://github.com/kellyjonbrazil/jello" 198 | descr = "Query JSON and [JSON Lines](http://jsonlines.org/) with Python code. Output the result in a line-based format suitable for creating Bash arrays. Generate a grep-able schema." 199 | tags = ["json"] 200 | 201 | [jet] 202 | url = "https://github.com/borkdude/jet" 203 | descr = "Convert between JSON, YAML, Clojure's [edn](https://github.com/edn-format/edn), and [Transit](https://github.com/cognitect/transit-format). Transform them with Clojure code." 204 | tags = ["json", "yaml", "multiformat"] 205 | 206 | [jfq] 207 | url = "https://github.com/blgm/jfq" 208 | descr = "Query and transform JSON with the [JSONata](http://jsonata.org/) language." 209 | tags = ["json"] 210 | 211 | [jid] 212 | url = "https://github.com/simeji/jid" 213 | descr = "Explore JSON interactively with filtering queries like jq." 214 | tags = ["tui"] 215 | 216 | [jiq] 217 | url = "https://github.com/fiatjaf/jiq" 218 | descr = "Explore JSON interactively with jq. Requires jq." 219 | tags = ["tui"] 220 | 221 | [jj] 222 | url = "https://github.com/tidwall/jj" 223 | descr = "Query and modify values in JSON or JSON Lines with a key path." 224 | tags = ["json"] 225 | 226 | [jl] 227 | url = "https://github.com/chrisdone/jl" 228 | descr = "Query and manipulate JSON using a tiny functional language." 229 | tags = ["json"] 230 | 231 | [jo] 232 | url = "https://github.com/jpmens/jo" 233 | descr = "Create JSON objects from the shell." 234 | tags = ["json"] 235 | 236 | ["jp (jmespath)"] 237 | url = "https://github.com/jmespath/jp" 238 | descr = "Query JSON with [JMESPath](http://jmespath.org/)." 239 | tags = ["json"] 240 | 241 | ["jp (sgreben)"] 242 | url = "https://github.com/sgreben/jp" 243 | descr = "Plot JSON and CSV data in the terminal. Supports different kinds of plots: bar charts, line charts, scatter plots, histograms, and heatmaps." 244 | tags = ["csv", "json", "multiformat"] 245 | 246 | [jplot] 247 | url = "https://github.com/rs/jplot" 248 | descr = "Plot real-time JSON data in the terminal (works with terminals supporting graphic rendering)." 249 | tags = ["json"] 250 | 251 | [jq] 252 | url = "http://stedolan.github.io/jq/manual/" 253 | descr = "Create and manipulate JSON with a functional (as in \"functional programming\") [DSL](https://en.wikipedia.org/wiki/Domain-specific_language). Can convert JSON to other formats." 254 | tags = ["json"] 255 | 256 | [jql] 257 | url = "https://github.com/cube2222/jql" 258 | descr = "Create and manipulate JSON with a Lisp-syntax DSL." 259 | tags = ["json"] 260 | 261 | [jshon] 262 | url = "http://kmkeen.com/jshon/" 263 | descr = "Create and manipulate JSON using [getopt](https://en.wikipedia.org/wiki/Getopt)-style command-line options." 264 | tags = ["json"] 265 | 266 | [json] 267 | url = "https://github.com/trentm/json" 268 | descr = "Run arbitrary JavaScript on JSON input." 269 | tags = ["json"] 270 | 271 | [json-patch] 272 | url = "https://github.com/evanphx/json-patch" 273 | descr = "Apply [RFC 6902](https://tools.ietf.org/html/rfc6902) JSON Patches to JSON. The CLI tool is secondary to a Go library that also creates and applies [RFC 7386](https://tools.ietf.org/html/rfc7396) JSON merge patches." 274 | tags = ["json"] 275 | 276 | [json-table] 277 | url = "https://github.com/micha/json-table" 278 | descr = "Convert nested JSON into CSV or TSV for processing in the shell." 279 | tags = ["json"] 280 | 281 | ["json.tool"] 282 | url = "https://docs.python.org/2/library/json.html" 283 | descr = "Validate and pretty-print JSON. This module is part of the standard library of Python 2/3 and is likely to be available wherever Python is installed. ([Python 3 docs](https://docs.python.org/3/library/json.html).)" 284 | tags = ["json"] 285 | 286 | [json2] 287 | url = "https://github.com/vi/json2" 288 | descr = "Convert JSON to and from flat, greppable lists of \"path=value\" statements. Modeled after [xml2](#xml)." 289 | tags = ["json"] 290 | 291 | [jsonaxe] 292 | url = "https://github.com/davvid/jsonaxe" 293 | descr = "Create and manipulate JSON with a Python-based DSL. Inspired by jq." 294 | tags = ["json"] 295 | 296 | [jsonwatch] 297 | url = "https://github.com/dbohdan/jsonwatch" 298 | descr = "Track changes in JSON data from the command line. Works like `watch -d`." 299 | tags = ["json"] 300 | 301 | [jtbl] 302 | url = "https://github.com/kellyjonbrazil/jtbl" 303 | descr = "Format JSON or JSON Lines as a plain-text table." 304 | tags = ["json"] 305 | 306 | [jtc] 307 | url = "https://github.com/ldn-softdev/jtc" 308 | descr = "Create, manipulate, search, validate JSON with path expressions. Can be used as a C++14 library." 309 | tags = ["json"] 310 | 311 | [lnav] 312 | url = "https://lnav.org" 313 | descr = "Query and watch log files. Has batch and interactive mode. Supported formats include the Common Log Format, CUPS page_log, syslog, strace, and generic timestamped messages. Can perform SQL queries." 314 | tags = ["log", "multiformat"] 315 | 316 | [lobar] 317 | url = "https://github.com/sodiumjoe/lobar" 318 | descr = "Process JSON and explore it interactively with a wrapper for `lodash.chain()`. An alternative to jq with JavaScript syntax." 319 | tags = ["tui", "json"] 320 | 321 | [madato] 322 | url = "https://github.com/inosion/madato" 323 | descr = "Convert ODS and XLSX spreadsheets to JSON, Markdown, and YAML." 324 | tags = ["json", "yaml", "multiformat"] 325 | 326 | [Mario] 327 | url = "https://github.com/python-mario/mario" 328 | descr = "Manipulate and convert between CSV, JSON, YAML, TOML, and XML with Python code." 329 | tags = ["csv", "xml", "yaml", "toml", "json", "multiformat"] 330 | 331 | ["MCMD (M-Command)"] 332 | url = "https://github.com/nysol/mcmd" 333 | descr = "Select, sample, cut, join, sort, reformat, and generate CSV files. Contains a large set of commands." 334 | tags = ["csv"] 335 | 336 | [mdq] 337 | url = "https://github.com/yshavit/mdq" 338 | descr = "Select elements from Markdown documents using a syntax inspired by Markdown and jq. Match content with regular expressions. Output Markdown or JSON." 339 | tags = ["markdown"] 340 | 341 | [Miller] 342 | url = "https://github.com/johnkerl/miller" 343 | descr = "`sed`, `awk`, `cut`, `join` and `sort` for name-indexed data such as CSV and tabular JSON." 344 | tags = ["csv"] 345 | 346 | [Nushell] 347 | url = "https://github.com/nushell/nushell" 348 | descr = "A command shell. Can natively [load data](https://www.nushell.sh/book/loading_data.html) from CSV, INI, JSON, TOML, TSV, XML, YAML, and other formats." 349 | tags = ["csv", "json", "toml", "xml", "yaml", "multiformat"] 350 | 351 | [otree] 352 | url = "https://github.com/fioncat/otree" 353 | descr = "Expore JSON, TOML, and YAML using a TUI tree widget." 354 | tags = ["tui"] 355 | 356 | [pawk] 357 | url = "https://github.com/alecthomas/pawk" 358 | descr = "Process text with AWK-like patterns, but Python code." 359 | tags = ["csv"] 360 | 361 | [pup] 362 | url = "https://github.com/EricChiang/pup" 363 | descr = "Query HTML pages with CSS selectors. Static binaries available for releases. Inspired by [jq](#json)." 364 | tags = ["html"] 365 | 366 | [pyp] 367 | url = "https://github.com/hauntsaninja/pyp" 368 | descr = "Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through [tomllib](https://docs.python.org/3.11/library/tomllib.html)." 369 | tags = ["awk-other", "json", "toml", "multiformat"] 370 | 371 | [qpyson] 372 | url = "https://github.com/mpkocher/qpyson" 373 | descr = "Query and manipulate JSON with Python." 374 | tags = ["json"] 375 | 376 | [qq] 377 | url = "https://github.com/JFryy/qq" 378 | descr = "Query and manipulate data in a number of formats: CSV, GRON, HCL, HTML, INI, JSON, Proto definition language, Terraform, TOML, XML, YAML, and lines of text. Expore data interactively. Uses gojq as a library." 379 | tags = ["csv", "html", "json", "toml", "xml", "yaml", "multiformat", "tui"] 380 | 381 | [qsv] 382 | url = "https://github.com/jqnatividad/qsv" 383 | descr = "Index, slice, analyze, split, and join CSV files. A fork of xsv that adds subcommands and features." 384 | tags = ["csv"] 385 | 386 | [query-json] 387 | url = "https://github.com/davesnx/query-json" 388 | descr = "A faster jq implementation written in Reason Native (OCaml)." 389 | tags = ["json"] 390 | 391 | [quicktype] 392 | url = "https://github.com/quicktype/quicktype" 393 | descr = "Infer the underlying model of the JSON and output as types for various programming languages or JSON Schema. CLI and [Web UI](https://app.quicktype.io)." 394 | tags = ["json"] 395 | 396 | [ramda-cli] 397 | url = "https://github.com/raine/ramda-cli" 398 | descr = "Manipulate JSON with the [Ramda](https://ramdajs.com/) functional library, and either LiveScript or JavaScript syntax." 399 | tags = ["json"] 400 | 401 | [RecordStream] 402 | url = "https://github.com/benbernard/RecordStream" 403 | descr = "Create, manipulate, and output a stream of records, or JSON objects. Can retrieve records from an SQL database, MongoDB, Atom feeds, XML, and other sources." 404 | tags = ["json", "multiformat"] 405 | 406 | [ReadStat] 407 | url = "https://github.com/WizardMac/ReadStat" 408 | descr = "Convert statistics package datasets between SAS (SAS7BDAT, XPORT), SPSS (POR, SAV, ZSAV), and Stata (DTA). Convert those formats to CSV and XLSX. Can be used as a C library with bindings for Julia, Python, and R." 409 | tags = ["csv", "multiformat"] 410 | 411 | [Remarshal] 412 | url = "https://github.com/dbohdan/remarshal" 413 | descr = "Convert between CBOR, JSON, MessagePack, TOML, and YAML. Validate each of the formats. Pretty-print JSON, TOML, and YAML." 414 | tags = ["json", "yaml", "toml", "multiformat"] 415 | 416 | [rows] 417 | url = "https://github.com/turicas/rows" 418 | descr = "A Python library with a [CLI](http://turicas.info/rows/cli/). Convert between a number of [file formats](http://turicas.info/rows/plugins/) for tabular data: CSV, XLS, XLSX, ODS, and others. Query the data (via SQLite). Combine tables. Generate schemas." 419 | tags = ["csv", "multiformat"] 420 | 421 | [rq] 422 | url = "https://github.com/dflemstr/rq" 423 | descr = "Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text." 424 | tags = ["csv", "yaml", "toml", "json", "awk-other", "multiformat"] 425 | 426 | [Saxon] 427 | url = "http://saxon.sourceforge.net/" 428 | descr = "Query XML and HTML data with [XPath](https://devhints.io/xpath). [Documentation](http://www.saxonica.com/documentation/#!using-xsl)." 429 | tags = ["xml", "html", "multiformat"] 430 | 431 | [sc-im] 432 | url = "https://github.com/andmarti1424/sc-im" 433 | descr = "A Vim-like spreadsheet calculator for CSV and TSV files." 434 | tags = ["tui"] 435 | 436 | [scrubcsv] 437 | url = "https://github.com/faradayio/scrubcsv" 438 | descr = "Remove bad lines from a CSV file and normalize the rest. Written in Rust." 439 | tags = ["csv"] 440 | 441 | [shyaml] 442 | url = "https://github.com/0k/shyaml" 443 | descr = "Query YAML. Can output null-terminated strings for use in shell scripts." 444 | tags = ["yaml"] 445 | 446 | [Skeem] 447 | url = "https://github.com/daq-tools/skeem" 448 | descr = "Infer SQL DDL statements from tabular data. Supports CSV, JSON, JSON Lines, ODS, XLSX, and other formats." 449 | tags = ["csv", "json", "multiformat"] 450 | 451 | [sml2] 452 | url = "https://github.com/JFLarvoire/libxml2" 453 | descr = "Convert between XML and [SML](https://htmlpreview.github.io/?https://github.com/JFLarvoire/libxml2/blob/master/SML_presentation.htm), a simplified XML representation." 454 | tags = ["xml"] 455 | 456 | [Squawk] 457 | url = "https://github.com/samuel/squawk" 458 | descr = "Query Apache and Nginx log files. See the [SQL-based tool comparison](sql-based.md)." 459 | tags = ["log"] 460 | 461 | [tab] 462 | url = "http://tkatchev.bitbucket.io/tab/" 463 | descr = "A non-Turing-complete statically typed programming language for data processing. An alternative to awk." 464 | tags = ["csv"] 465 | 466 | [teip] 467 | url = "https://github.com/greymd/teip" 468 | descr = "Select fields, character ranges, or regular expression matches from standard input. Replace them with the output of a command." 469 | tags = ["csv"] 470 | 471 | [Temme] 472 | url = "https://github.com/shinima/temme" 473 | descr = "Query HTML with CSS-like selectors to extract JSON. Temme extends CSS selectors with value capture patterns." 474 | tags = ["html"] 475 | 476 | [taplo-cli] 477 | url = "https://github.com/tamasfe/taplo" 478 | descr = "Query, format, and validate (lint) TOML." 479 | tags = ["toml"] 480 | 481 | [tidy-html5] 482 | url = "http://www.html-tidy.org/" 483 | descr = "Validate, fix, and reformat HTML(5), XHTML, and XML documents. Convert HTML to XHTML." 484 | tags = ["xml", "html", "multiformat"] 485 | 486 | [tq] 487 | url = "https://github.com/plainas/tq" 488 | descr = "Query HTML with CSS selectors." 489 | tags = ["html"] 490 | 491 | [tv] 492 | url = "https://github.com/codechenx/tv" 493 | descr = "View delimited files in the terminal." 494 | tags = ["csv"] 495 | 496 | [validjson] 497 | url = "http://github.com/martinlindhe/validjson" 498 | descr = "Validate or pretty-print JSON." 499 | tags = ["json"] 500 | 501 | [validtoml] 502 | url = "http://github.com/martinlindhe/validtoml" 503 | descr = "Validate TOML." 504 | tags = ["toml"] 505 | 506 | [validyaml] 507 | url = "http://github.com/martinlindhe/validyaml" 508 | descr = "Validate or pretty-print YAML." 509 | tags = ["yaml"] 510 | 511 | [VisiData] 512 | url = "https://github.com/saulpw/visidata" 513 | descr = "Explore interactively data in TSV, CSV, XLS, XLSX, HDF5, JSON, and [other formats](http://visidata.org/man/#loaders). [Introduction](https://jsvine.github.io/intro-to-visidata/)." 514 | tags = ["tui", "multiformat"] 515 | 516 | [vnlog] 517 | url = "https://github.com/dkogan/vnlog/" 518 | descr = "Process labelled tabular ASCII data using normal UNIX tools. Can plot data with gnuplot." 519 | tags = ["awk-other"] 520 | 521 | [Xidel] 522 | url = "http://www.videlibri.de/xidel.html" 523 | descr = "Query or modify XML and HTML pages with XPath, XQuery 3, and CSS selectors." 524 | tags = ["xml", "html", "multiformat"] 525 | 526 | [xml-to-json-fast] 527 | url = "https://github.com/sinelaw/xml-to-json-fast" 528 | descr = "Convert XML to JSON. Can handle very large XML files." 529 | tags = ["json", "xml"] 530 | 531 | [xml2] 532 | url = "https://web.archive.org/web/20160719191401/http://ofb.net/~egnor/xml2/" 533 | descr = "Convert XML and HTML to and from flat, greppable lists of \"path=value\" statements. [Source code mirror](https://github.com/clone/xml2)." 534 | tags = ["xml", "html", "multiformat"] 535 | 536 | [xmljson] 537 | url = "https://github.com/engali94/XMLJson" 538 | descr = "Convert multiple and large XML files to JSON. Written in Swift." 539 | tags = ["json", "xml", "multiformat"] 540 | 541 | [XMLLint] 542 | url = "http://xmlsoft.org/xmllint.html" 543 | descr = "Query (including XSLT), validate and reformat XML documents." 544 | tags = ["xml"] 545 | 546 | [XMLStarlet] 547 | url = "http://xmlstar.sourceforge.net/overview.php" 548 | descr = "Query, modify, and validate XML documents." 549 | tags = ["xml"] 550 | 551 | [xpe] 552 | url = "https://github.com/charmparticle/xpe" 553 | descr = "Query HTML and XML with XPath expressions." 554 | tags = ["xml", "html", "multiformat"] 555 | 556 | [xq] 557 | url = "https://github.com/kislyuk/yq" 558 | descr = "[jq](#json) wrapper for XML documents." 559 | tags = ["xml"] 560 | 561 | [xsltproc] 562 | url = "http://xmlsoft.org/XSLT/xsltproc2.html" 563 | descr = "Transform XML documents using [XSLT](https://www.w3.org/TR/xslt) and [EXSLT](http://exslt.org)." 564 | tags = ["xml"] 565 | 566 | [xsv] 567 | url = "https://github.com/BurntSushi/xsv" 568 | descr = "Index, slice, analyze, split, and join CSV files." 569 | tags = ["csv"] 570 | 571 | [yaml-diff-patch] 572 | url = "https://github.com/grantila/yaml-diff-patch" 573 | descr = "Patch YAML with [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) JSON Patches. Generate a JSON Patch from two JSON documents or a YAML and a JSON document. Preserves style. Can be used as a TypeScript library." 574 | tags = ["json", "yaml", "multiformat"] 575 | 576 | [yaml-tools] 577 | url = "https://github.com/thecodingmachine/yaml-tools" 578 | descr = "A set of CLI tools to manipulate YAML files (merge, delete, etc...) with comment preservation, based on [ruamel.yaml](http://yaml.readthedocs.io/en/latest/)." 579 | tags = ["yaml"] 580 | 581 | [yamlpath] 582 | url = "https://github.com/wwkimball/yamlpath" 583 | descr = "Query, modify, diff, merge, and validate YAML and JSON with [YAML Paths](https://github.com/wwkimball/yamlpath/wiki/Segments-of-a-YAML-Path). Also a Python library." 584 | tags = ["json", "yaml", "multiformat"] 585 | 586 | ["yq (kislyuk)"] 587 | url = "https://github.com/kislyuk/yq" 588 | descr = "[jq](#json) wrapper for YAML, XML, and TOML." 589 | tags = ["toml", "xml", "yaml", "multiformat"] 590 | 591 | ["yq (mikefarah)"] 592 | url = "https://github.com/mikefarah/yq" 593 | descr = "Query, modify, and merge YAML. Convert to and from JSON." 594 | tags = ["yaml"] 595 | 596 | [zsv] 597 | url = "https://github.com/liquidaty/zsv" 598 | descr = "Slice, combine, reformat, flatten/unflatten CSV (TSV, DSV) files. Query them with SQL and jq filters. Convert between them, JSON, and SQLite 3. Also a C library." 599 | tags = ["csv", "multiformat"] 600 | 601 | # .env 602 | 603 | [dotenvx] 604 | url = "https://github.com/dotenvx/dotenvx" 605 | tags = [".env"] 606 | 607 | [dotenvx.props] 608 | Platform = "POSIX, Windows" 609 | License = "BSD-3-Clause" 610 | Description = "A CLI tool to manipulate, parse, and inject `.env` files as environment variables." 611 | 612 | # INI 613 | 614 | [cfget] 615 | url = "https://packages.debian.org/source/buster/cfget" 616 | tags = ["ini"] 617 | 618 | [cfget.props] 619 | Platform = "Any with Python 2.6-2.7?" 620 | License = "GPL-2.0-or-later" 621 | Description = "Retrieve properties as shell script commands to set the corresponding variables (with `--dump exports`). Retrieve properties' values as plain text. Substitute values from an INI file in an Autoconf-style template. Supports plug-ins. Chokes on section names and keys with spaces." 622 | 623 | [confget] 624 | url = "https://devel.ringlet.net/textproc/confget/" 625 | tags = ["ini"] 626 | 627 | [confget.props] 628 | Platform = "Free/Net/OpenBSD, Linux, likely others" 629 | License = "BSD-2-Clause" 630 | Description = "Retrieve properties and sections as shell script commands to set the corresponding variables. Retrieve properties' values as plain text. Check for existence of properties. List sections. Find values that match a pattern. Read-only. Has a C, Python, and Rust implementation. The Rust implementation can be installed with `cargo install confget`." 631 | 632 | [crudini] 633 | url = "https://github.com/pixelb/crudini/" 634 | tags = ["ini"] 635 | 636 | [crudini.props] 637 | Platform = "Any with Python 2.6–2.7 or 3.x" 638 | License = "GPL-2.0" 639 | Description = "Retrieve properties and sections as INI fragments or shell script commands to set the corresponding variables. Retrieve properties' values as plain text. Set properties. Remove properties and sections. Create empty sections. Merge INI files. Changes files in place." 640 | 641 | [inicomp] 642 | url = "https://github.com/JFLarvoire/SysToolsLib/blob/HEAD/C/SRC/inicomp.c" 643 | tags = ["ini"] 644 | 645 | [inicomp.props] 646 | Platform = "Windows, POSIX" 647 | License = "Apache-2.0" 648 | Description = "Compare INI (and also Windows .reg) files." 649 | 650 | [IniFile] 651 | url = "http://www.horstmuc.de/wbat32.htm#inifile" 652 | tags = ["ini"] 653 | 654 | [IniFile.props] 655 | Platform = "Windows (x86, x86-64), [MS-DOS](http://www.horstmuc.de/div.htm#inifile)" 656 | License = "Closed-source freeware" 657 | Description = "Retrieve properties and sections as batch file commands to set the corresponding variables. Set properties. Remove properties and sections. Changes files in place." 658 | 659 | [initool] 660 | url = "https://github.com/dbohdan/initool" 661 | tags = ["ini"] 662 | 663 | [initool.props] 664 | Platform = "FreeBSD, Linux, Windows" 665 | License = "MIT" 666 | Description = "Retrieve properties and sections as INI fragments. Retrieve properties' values as plain text. Set properties. Check for existence of properties and sections. Remove properties and sections. Outputs the updated INI file." 667 | 668 | ["Nushell (`from ini`)"] 669 | url = "https://github.com/nushell/nushell" 670 | tags = ["ini"] 671 | 672 | ["Nushell (`from ini`)".props] 673 | Platform = "Free/Net/OpenBSD, Linux, macOS, Windows" 674 | License = "MIT" 675 | Description = "Query and transform data with the Nushell language." 676 | 677 | ["qq (INI)"] 678 | url = "https://github.com/JFryy/qq" 679 | tags = ["ini"] 680 | 681 | ["qq (INI)".props] 682 | Platform = "Free/Net/OpenBSD, Linux, macOS, Windows" 683 | License = "MIT" 684 | Description = "Query and transform data with jq. Based on gojq." 685 | 686 | # Templating for structured text 687 | 688 | [CUE] 689 | url = "https://github.com/cuelang/cue" 690 | tags = ["template"] 691 | 692 | [CUE.props] 693 | "Output format" = "JSON" 694 | Turing-complete = "No" 695 | Syntax = "Extended JSON" 696 | "I/O" = "?" 697 | Description = "A constraint language for JSON configuration data. Can generate and validates JSON." 698 | 699 | [Dhall] 700 | url = "https://dhall-lang.org/" 701 | tags = ["template"] 702 | 703 | [Dhall.props] 704 | "Output format" = "JSON, YAML" 705 | Turing-complete = "No" 706 | Syntax = "Haskell-inspired" 707 | "I/O" = "Limited to importing libraries from files and HTTP(S) URLs (with protection against leaking your data to the server)" 708 | Description = "A statically-typed functional configuration language. Has a standard formatting tool." 709 | 710 | [jk] 711 | url = "https://github.com/jkcfg/jk" 712 | tags = ["template"] 713 | 714 | [jk.props] 715 | "Output format" = "JSON, YAML, plain text" 716 | Turing-complete = "Yes" 717 | Syntax = "JavaScript" 718 | "I/O" = "Disk I/O" 719 | Description = "Generate configuration files using JavaScript (V8 VM)." 720 | 721 | [Jsonnet] 722 | url = "https://jsonnet.org/" 723 | tags = ["template"] 724 | 725 | [Jsonnet.props] 726 | "Output format" = "JSON, INI, XML, YAML, plain text" 727 | Turing-complete = "Yes" 728 | Syntax = "Extended JSON" 729 | "I/O" = "None" 730 | Description = "A functional configuration language. Has a standard formatting tool." 731 | 732 | [Nickel] 733 | url = "https://nickel-lang.org/" 734 | tags = ["template"] 735 | 736 | [Nickel.props] 737 | "Output format" = "JSON, TOML, YAML" 738 | Turing-complete = "Yes" 739 | Syntax = "Inspired by ML and JSON" 740 | "I/O" = "Limited input is to be implemented" 741 | Description = "A gradually-typed functional configuration language with contracts." 742 | 743 | [Pkl] 744 | url = "https://pkl-lang.org/" 745 | tags = ["template"] 746 | 747 | [Pkl.props] 748 | "Output format" = "JSON, YAML, macOS property list, Java `.properties`" 749 | Turing-complete = "Yes" 750 | Syntax = "Swift-inspired" 751 | "I/O" = "The CLI can read environment variables and files, `GET` HTTP(S) URLs. It can import modules from files and HTTP(S) URLs." 752 | Description = "A command-line tool, Java library, and build tool plugin. Can generate code for Go, Java, Kotlin, and Swift. [\"Pkl vs. Other Config Languages\"](https://pkl-lang.org/main/current/introduction/comparison.html#other-config-langs)." 753 | 754 | [rjsone] 755 | url = "https://github.com/wryun/rjsone" 756 | tags = ["template"] 757 | 758 | [rjsone.props] 759 | "Output format" = "JSON, YAML" 760 | Turing-complete = "No?" 761 | Syntax = "Extended JSON" 762 | "I/O" = "None" 763 | Description = "A CLI tool for the [JSON-e](https://github.com/taskcluster/json-e) templating language." 764 | 765 | [ytt] 766 | url = "https://get-ytt.io/" 767 | tags = ["template"] 768 | 769 | [ytt.props] 770 | "Output format" = "YAML" 771 | Turing-complete = "No" 772 | Syntax = "YAML/Python hybrid" 773 | "I/O" = "None?" 774 | Description = "A templating tool for YAML built upon the [Starlark](https://github.com/bazelbuild/starlark) configuration language." 775 | 776 | # CLIs for single-file databases 777 | 778 | [Firebird] 779 | url = "https://firebirdsql.org/" 780 | tags = ["single-file-db"] 781 | 782 | [Firebird.props] 783 | Description = "Firebird is a FOSS database that can be used from a single file, like SQLite. \"isql is a program that allows the user to issue arbitrary SQL commands\"." 784 | "File format" = "Binary" 785 | 786 | [Fsdb] 787 | url = "https://www.isi.edu/~johnh/SOFTWARE/FSDB/perl-Fsdb-2.69_README.html" 788 | tags = ["single-file-db"] 789 | 790 | [Fsdb.props] 791 | Description = "A flat-file database for shell scripting." 792 | "File format" = "Text-based, TSV with a header or \"key: value\"" 793 | 794 | ["GNU Recutils"] 795 | url = "http://www.gnu.org/software/recutils/" 796 | tags = ["single-file-db"] 797 | 798 | ["GNU Recutils".props] 799 | Description = "\"[A] set of tools and libraries to access human-editable, plain text databases called recfiles.\"" 800 | "File format" = "Text-based, roughly \"key: value\"" 801 | 802 | [SDB] 803 | url = "https://github.com/radare/sdb" 804 | tags = ["single-file-db"] 805 | 806 | [SDB.props] 807 | Description = "\"[A] simple string key/value database based on djb's cdb disk storage and supports JSON and arrays introspection.\"" 808 | "File format" = "Binary" 809 | 810 | ["sqlite3(1)"] 811 | url = "https://www.sqlite.org/cli.html" 812 | tags = ["single-file-db"] 813 | 814 | ["sqlite3(1)".props] 815 | Description = "\"[A] simple command-line utility [...] that allows the user to manually enter and execute SQL statements against an SQLite database.\"" 816 | "File format" = "Binary" 817 | -------------------------------------------------------------------------------- /data/sql-based.toml: -------------------------------------------------------------------------------- 1 | ["AlaSQL CLI"] 2 | url = "https://github.com/agershun/alasql" 3 | tags = ["sql-based"] 4 | 5 | ["AlaSQL CLI".props] 6 | "Documentation link" = "" 7 | "Programming language" = "JavaScript" 8 | Database = "AlaSQL" 9 | "Column names from header row" = "yes, optional" 10 | "Custom character encoding" = "no" 11 | "Custom input field separator" = "yes, string" 12 | "Custom input record separator" = "no" 13 | "Custom output field separator" = "no" 14 | "Custom output record separator" = "no" 15 | JOINs = "yes" 16 | "Use as library" = "yes, JavaScript" 17 | "Input formats" = "lines, DSV, XLS, XLSX, HTML tables, JSON" 18 | "Output formats" = "lines, DSV, XLS, XLSX, HTML tables, JSON" 19 | "Custom table names" = "yes" 20 | "Custom column names" = "yes" 21 | "Keep database file" = "n/a" 22 | "Skip input fields" = "no" 23 | "Skip input records (lines)" = "no" 24 | "Merge input fields" = "no" 25 | "Database table customization" = "yes, can create custom table then import into it" 26 | "SQL dump" = "yes" 27 | Other = "" 28 | 29 | [csvq] 30 | url = "https://github.com/mithrandie/csvq" 31 | tags = ["sql-based"] 32 | 33 | [csvq.props] 34 | "Documentation link" = "" 35 | "Programming language" = "Go" 36 | Database = "custom SQL interpreter" 37 | "Column names from header row" = "yes, optional" 38 | "Custom character encoding" = "yes, input and output" 39 | "Custom input field separator" = "yes, character" 40 | "Custom input record separator" = "no" 41 | "Custom output field separator" = "yes" 42 | "Custom output record separator" = "no" 43 | JOINs = "yes" 44 | "Use as library" = "yes, Go" 45 | "Input formats" = "CSV, TSV, LTSV, fixed-width, JSON" 46 | "Output formats" = "CSV, TSV, LTSV, fixed-width, JSON, Markdown-style table, Org-mode, ASCII table" 47 | "Custom table names" = "yes" 48 | "Custom column names" = "yes" 49 | "Keep database file" = "n/a" 50 | "Skip input fields" = "no" 51 | "Skip input records (lines)" = "no" 52 | "Merge input fields" = "no" 53 | "Database table customization" = "yes, ALTER TABLE" 54 | "SQL dump" = "no" 55 | Other = "" 56 | 57 | [csvsql] 58 | url = "https://github.com/wireservice/csvkit" 59 | tags = ["sql-based"] 60 | 61 | [csvsql.props] 62 | "Documentation link" = "" 63 | "Programming language" = "Python" 64 | Database = "Firebird/MS SQL/MySQL/Oracle/PostgreSQL/SQLite 3/Sybase" 65 | "Column names from header row" = "yes, optional" 66 | "Custom character encoding" = "yes, input and output" 67 | "Custom input field separator" = "yes, string" 68 | "Custom input record separator" = "no" 69 | "Custom output field separator" = "yes" 70 | "Custom output record separator" = "no" 71 | JOINs = "yes" 72 | "Use as library" = "yes, Python" 73 | "Input formats" = "delimited without quotes, DSV, Excel, JSON, SQL, fixed-width, DBF, and others (separate converters)" 74 | "Output formats" = "delimited without quotes, DSV, JSON, Markdown-style table, SQL (separate converters)" 75 | "Custom table names" = "yes" 76 | "Custom column names" = "no" 77 | "Keep database file" = "yes" 78 | "Skip input fields" = "yes (separate tool)" 79 | "Skip input records (lines)" = "no" 80 | "Merge input fields" = "no?" 81 | "Database table customization" = "yes, UNIQUE constraints, database schema name, automatic column datatype or text" 82 | "SQL dump" = "yes" 83 | Other = "" 84 | 85 | [DuckDB] 86 | url = "https://github.com/duckdb/duckdb" 87 | tags = ["sql-based"] 88 | 89 | [DuckDB.props] 90 | "Documentation link" = "" 91 | "Programming language" = "C++" 92 | Database = "DuckDB" 93 | "Column names from header row" = "yes, optional" 94 | "Custom character encoding" = "no, only UTF-8" 95 | "Custom input field separator" = "yes, string" 96 | "Custom input record separator" = "no" 97 | "Custom output field separator" = "yes" 98 | "Custom output record separator" = "no" 99 | JOINs = "yes" 100 | "Use as library" = "yes, C, C++, JavaScript, Python, and other languages" 101 | "Input formats" = "DSV, Excel, JSON, SQL, Parquet" 102 | "Output formats" = "DSV, Excel, JSON, SQL, Parquet" 103 | "Custom table names" = "yes" 104 | "Custom column names" = "yes" 105 | "Keep database file" = "yes" 106 | "Skip input fields" = "yes" 107 | "Skip input records (lines)" = "yes" 108 | "Merge input fields" = "no" 109 | "Database table customization" = "yes, `CREATE TABLE`" 110 | "SQL dump" = "yes" 111 | Other = '["DuckDB as the New jq"](https://www.pgrs.net/2024/03/21/duckdb-as-the-new-jq/), Paul Gross (2024).' 112 | 113 | [fsql] 114 | url = "https://metacpan.org/release/App-fsql" 115 | tags = ["sql-based"] 116 | 117 | [fsql.props] 118 | "Documentation link" = "" 119 | "Programming language" = "Perl" 120 | Database = "custom SQL interpreter" 121 | "Column names from header row" = "yes, always" 122 | "Custom character encoding" = "no" 123 | "Custom input field separator" = "no" 124 | "Custom input record separator" = "no" 125 | "Custom output field separator" = "no" 126 | "Custom output record separator" = "no" 127 | JOINs = "yes" 128 | "Use as library" = "yes, Perl" 129 | "Input formats" = "CSV, TSV, LTSV, Perl, JSON, YAML" 130 | "Output formats" = "CSV, TSV, LTSV, Perl, JSON, YAML" 131 | "Custom table names" = "yes" 132 | "Custom column names" = "no" 133 | "Keep database file" = "no" 134 | "Skip input fields" = "no" 135 | "Skip input records (lines)" = "no" 136 | "Merge input fields" = "no" 137 | "Database table customization" = "no" 138 | "SQL dump" = "no" 139 | Other = "" 140 | 141 | [q] 142 | url = "https://github.com/harelba/q" 143 | tags = ["sql-based"] 144 | 145 | [q.props] 146 | "Documentation link" = "" 147 | "Programming language" = "Python" 148 | Database = "SQLite 3" 149 | "Column names from header row" = "yes, optional" 150 | "Custom character encoding" = "yes, input and output" 151 | "Custom input field separator" = "yes, string" 152 | "Custom input record separator" = "no" 153 | "Custom output field separator" = "yes" 154 | "Custom output record separator" = "no" 155 | JOINs = "yes" 156 | "Use as library" = "yes, Python" 157 | "Input formats" = "delimited without quotes, DSV" 158 | "Output formats" = "delimited without quotes, DSV, custom using Python formatting string" 159 | "Custom table names" = "no" 160 | "Custom column names" = "no" 161 | "Keep database file" = "yes" 162 | "Skip input fields" = "no" 163 | "Skip input records (lines)" = "no" 164 | "Merge input fields" = "no" 165 | "Database table customization" = "yes, automatic column datatype or text" 166 | "SQL dump" = "no" 167 | Other = "" 168 | 169 | [RBQL] 170 | url = "https://github.com/mechatroner/RBQL" 171 | tags = ["sql-based"] 172 | 173 | [RBQL.props] 174 | "Documentation link" = "" 175 | "Programming language" = "JavaScript, Python" 176 | Database = "custom SQL interpreter" 177 | "Column names from header row" = "yes, optional" 178 | "Custom character encoding" = "yes, input" 179 | "Custom input field separator" = "yes, string" 180 | "Custom input record separator" = "no" 181 | "Custom output field separator" = "yes" 182 | "Custom output record separator" = "no" 183 | JOINs = "yes" 184 | "Use as library" = "yes, JavaScript and Python" 185 | "Input formats" = "DSV" 186 | "Output formats" = "DSV" 187 | "Custom table names" = "no" 188 | "Custom column names" = "no" 189 | "Keep database file" = "n/a" 190 | "Skip input fields" = "no" 191 | "Skip input records (lines)" = "no" 192 | "Merge input fields" = "no" 193 | "Database table customization" = "no" 194 | "SQL dump" = "no" 195 | Other = "" 196 | 197 | [rows] 198 | url = "https://github.com/turicas/rows" 199 | tags = ["sql-based"] 200 | 201 | [rows.props] 202 | "Documentation link" = "" 203 | "Programming language" = "Python" 204 | Database = "SQLite 3" 205 | "Column names from header row" = "yes, always?" 206 | "Custom character encoding" = "no" 207 | "Custom input field separator" = "no" 208 | "Custom input record separator" = "no" 209 | "Custom output field separator" = "no" 210 | "Custom output record separator" = "no" 211 | JOINs = "no" 212 | "Use as library" = "yes, Python" 213 | "Input formats" = "CSV, JSON, XLS, XLSX, ODS, and others" 214 | "Output formats" = "CSV, JSON, XLS, XLSX, ODS, and others" 215 | "Custom table names" = "no" 216 | "Custom column names" = "no" 217 | "Keep database file" = "no" 218 | "Skip input fields" = "no" 219 | "Skip input records (lines)" = "no" 220 | "Merge input fields" = "no" 221 | "Database table customization" = "no" 222 | "SQL dump" = "no" 223 | Other = "" 224 | 225 | [Sqawk] 226 | url = "https://github.com/dbohdan/sqawk" 227 | tags = ["sql-based"] 228 | 229 | [Sqawk.props] 230 | "Documentation link" = "" 231 | "Programming language" = "Tcl" 232 | Database = "SQLite 3" 233 | "Column names from header row" = "yes, optional" 234 | "Custom character encoding" = "no" 235 | "Custom input field separator" = "yes, regexp, per-file" 236 | "Custom input record separator" = "yes, regexp, per-file" 237 | "Custom output field separator" = "yes" 238 | "Custom output record separator" = "yes" 239 | JOINs = "yes" 240 | "Use as library" = "yes, Tcl" 241 | "Input formats" = "delimited without quotes, DSV, Tcl" 242 | "Output formats" = "delimited without quotes, CSV, JSON, ASCII/Unicode table, Tcl" 243 | "Custom table names" = "yes" 244 | "Custom column names" = "yes" 245 | "Keep database file" = "yes" 246 | "Skip input fields" = "yes, any" 247 | "Skip input records (lines)" = "no" 248 | "Merge input fields" = "yes, any consecutive" 249 | "Database table customization" = "yes, column datatypes" 250 | "SQL dump" = "no" 251 | Other = "" 252 | 253 | [sqawk] 254 | url = "https://github.com/tjunier/sqawk" 255 | tags = ["sql-based"] 256 | 257 | [sqawk.props] 258 | "Documentation link" = "" 259 | "Programming language" = "C" 260 | Database = "SQLite 3" 261 | "Column names from header row" = "yes, optional" 262 | "Custom character encoding" = "no" 263 | "Custom input field separator" = "yes, string, per-file" 264 | "Custom input record separator" = "no" 265 | "Custom output field separator" = "no" 266 | "Custom output record separator" = "no" 267 | JOINs = "yes" 268 | "Use as library" = "no" 269 | "Input formats" = "DSV" 270 | "Output formats" = "CSV" 271 | "Custom table names" = "yes" 272 | "Custom column names" = "no" 273 | "Keep database file" = "yes" 274 | "Skip input fields" = "no" 275 | "Skip input records (lines)" = "yes, until regexp matches" 276 | "Merge input fields" = "no" 277 | "Database table customization" = "yes, primary key, indexes, foreign key constraints, automatic column datatype or text" 278 | "SQL dump" = "yes" 279 | Other = "chunked mode (read and process only N lines at a time)" 280 | 281 | [Squawk] 282 | url = "https://github.com/samuel/squawk" 283 | tags = ["sql-based"] 284 | 285 | [Squawk.props] 286 | "Documentation link" = "<>" 287 | "Programming language" = "Python" 288 | Database = "custom SQL interpreter" 289 | "Column names from header row" = "yes, always" 290 | "Custom character encoding" = "no" 291 | "Custom input field separator" = "no" 292 | "Custom input record separator" = "no" 293 | "Custom output field separator" = "no" 294 | "Custom output record separator" = "no" 295 | JOINs = "no" 296 | "Use as library" = "yes, Python" 297 | "Input formats" = "CSV, Apache and Nginx log files" 298 | "Output formats" = "table, CSV, JSON" 299 | "Custom table names" = "no" 300 | "Custom column names" = "no" 301 | "Keep database file" = "no" 302 | "Skip input fields" = "no" 303 | "Skip input records (lines)" = "no" 304 | "Merge input fields" = "no" 305 | "Database table customization" = "no" 306 | "SQL dump" = "yes" 307 | Other = "" 308 | 309 | [termsql] 310 | url = "https://github.com/tobimensch/termsql" 311 | tags = ["sql-based"] 312 | 313 | [termsql.props] 314 | "Documentation link" = "" 315 | "Programming language" = "Python" 316 | Database = "SQLite 3" 317 | "Column names from header row" = "yes, optional" 318 | "Custom character encoding" = "no" 319 | "Custom input field separator" = "yes, regexp" 320 | "Custom input record separator" = "no" 321 | "Custom output field separator" = "yes" 322 | "Custom output record separator" = "no" 323 | JOINs = "no" 324 | "Use as library" = "no" 325 | "Input formats" = "DSV, “vertical” DSV (lines as columns)" 326 | "Output formats" = "delimited without quotes, CSV, TSV, HTML, SQL, Tcl" 327 | "Custom table names" = "yes" 328 | "Custom column names" = "yes" 329 | "Keep database file" = "yes" 330 | "Skip input fields" = "no" 331 | "Skip input records (lines)" = "yes, N first and M last" 332 | "Merge input fields" = "yes, Nth to last" 333 | "Database table customization" = "yes, primary key" 334 | "SQL dump" = "yes" 335 | Other = "" 336 | 337 | [trdsql] 338 | url = "https://github.com/noborus/trdsql" 339 | tags = ["sql-based"] 340 | 341 | [trdsql.props] 342 | "Documentation link" = "" 343 | "Programming language" = "Go" 344 | Database = "MySQL/PostgreSQL/SQLite 3" 345 | "Column names from header row" = "yes, optional" 346 | "Custom character encoding" = "no" 347 | "Custom input field separator" = "yes, string" 348 | "Custom input record separator" = "no" 349 | "Custom output field separator" = "no" 350 | "Custom output record separator" = "no" 351 | JOINs = "yes" 352 | "Use as library" = "no" 353 | "Input formats" = "CSV, LTSV, JSON" 354 | "Output formats" = "delimited without quotes, CSV, LTSV, JSON, ASCII table, Markdown" 355 | "Custom table names" = "no" 356 | "Custom column names" = "no" 357 | "Keep database file" = "yes" 358 | "Skip input fields" = "no" 359 | "Skip input records (lines)" = "no" 360 | "Merge input fields" = "no" 361 | "Database table customization" = "no" 362 | "SQL dump" = "no" 363 | Other = "" 364 | 365 | [textql] 366 | url = "https://github.com/dinedal/textql" 367 | tags = ["sql-based"] 368 | 369 | [textql.props] 370 | "Documentation link" = "" 371 | "Programming language" = "Go" 372 | Database = "SQLite 3" 373 | "Column names from header row" = "yes, optional" 374 | "Custom character encoding" = "no" 375 | "Custom input field separator" = "yes, string" 376 | "Custom input record separator" = "no" 377 | "Custom output field separator" = "no" 378 | "Custom output record separator" = "no" 379 | JOINs = "no" 380 | "Use as library" = "no" 381 | "Input formats" = "DSV" 382 | "Output formats" = "DSV" 383 | "Custom table names" = "no" 384 | "Custom column names" = "no" 385 | "Keep database file" = "yes" 386 | "Skip input fields" = "no" 387 | "Skip input records (lines)" = "no" 388 | "Merge input fields" = "no" 389 | "Database table customization" = "no" 390 | "SQL dump" = "no" 391 | Other = "" 392 | 393 | [Musoq] 394 | url = "https://github.com/Puchaczov/Musoq" 395 | tags = ["sql-based"] 396 | 397 | [Musoq.props] 398 | "Documentation link" = "https://github.com/Puchaczov/Musoq" 399 | "Programming language" = "C#" 400 | Database = "custom SQL interpreter" 401 | "Column names from header row" = "yes, optional" 402 | "Custom character encoding" = "no" 403 | "Custom input field separator" = "depends on the data source" 404 | "Custom input record separator" = "no" 405 | "Custom output field separator" = "no" 406 | "Custom output record separator" = "no" 407 | JOINs = "yes" 408 | "Use as library" = "yes" 409 | "Input formats" = "various" 410 | "Output formats" = "CSV, JSON" 411 | "Custom table names" = "no" 412 | "Custom column names" = "no" 413 | "Keep database file" = "no" 414 | "Skip input fields" = "yes" 415 | "Skip input records (lines)" = "no" 416 | "Merge input fields" = "yes" 417 | "Database table customization" = "no" 418 | "SQL dump" = "no" 419 | Other = "" 420 | -------------------------------------------------------------------------------- /macros.md.jinja: -------------------------------------------------------------------------------- 1 | {% macro item(proj) %} 2 | - [{{ proj.name }}]({{ proj.url }}){% if proj.descr %} **—** {{ proj.descr }}{% endif %} 3 | {% if proj.props %} 4 | 5 | {% for prop, value in proj.props.items() %} 6 | - **{{ prop }}:** {{ value }}{% if not loop.last %} 7 | 8 | {% endif %} 9 | {% endfor %} 10 | {% endif %} 11 | {% endmacro %} 12 | 13 | {% macro items(tag) %} 14 | {% for proj in projs_with_tag(tag) %} 15 | {{ item(proj) }}{% if not loop.last %} 16 | 17 | {% endif %} 18 | {% endfor %} 19 | {% endmacro %} 20 | -------------------------------------------------------------------------------- /render-template.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env -S uv run --script --quiet 2 | # /// script 3 | # dependencies = [ 4 | # "click==8.*", 5 | # "Jinja2==3.*", 6 | # "tomli==2.*", 7 | # ] 8 | # requires-python = ">=3.8" 9 | # /// 10 | 11 | from __future__ import annotations 12 | 13 | from dataclasses import dataclass, field 14 | from itertools import starmap 15 | from pathlib import Path 16 | from typing import Any, Iterator 17 | 18 | import click 19 | import jinja2 as j2 20 | import tomli 21 | 22 | 23 | @dataclass 24 | class Project: 25 | name: str 26 | url: str 27 | tags: list[str] 28 | descr: str = "" 29 | props: dict[str, str] = field(default_factory=dict) 30 | 31 | @classmethod 32 | def from_dict(cls, name: str, d: dict[str, Any]): 33 | return cls(name=name, **d) 34 | 35 | 36 | @click.command() 37 | @click.argument("template-name", type=click.Path(exists=True, path_type=Path)) 38 | @click.argument("data-path", type=click.Path(exists=True, path_type=Path)) 39 | def main(template_name: Path, data_path: Path) -> None: 40 | data = tomli.loads(data_path.read_text()) 41 | projects = list(starmap(Project.from_dict, data.items())) 42 | 43 | def projs_with_tag(tag: str) -> list[Project]: 44 | return [proj for proj in projects if tag in proj.tags] 45 | 46 | env = j2.Environment( 47 | loader=j2.FileSystemLoader("."), 48 | autoescape=j2.select_autoescape(), 49 | lstrip_blocks=True, 50 | trim_blocks=True, 51 | ) 52 | 53 | env.globals["projs_with_tag"] = projs_with_tag 54 | 55 | template = env.get_template(str(template_name)) 56 | rendered = template.render(data) 57 | click.echo(rendered) 58 | 59 | 60 | if __name__ == "__main__": 61 | main() 62 | -------------------------------------------------------------------------------- /sql-based.md: -------------------------------------------------------------------------------- 1 | # SQL-based tools 2 | 3 | - [AlaSQL CLI](https://github.com/agershun/alasql) 4 | - **Documentation link:** 5 | - **Programming language:** JavaScript 6 | - **Database:** AlaSQL 7 | - **Column names from header row:** yes, optional 8 | - **Custom character encoding:** no 9 | - **Custom input field separator:** yes, string 10 | - **Custom input record separator:** no 11 | - **Custom output field separator:** no 12 | - **Custom output record separator:** no 13 | - **JOINs:** yes 14 | - **Use as library:** yes, JavaScript 15 | - **Input formats:** lines, DSV, XLS, XLSX, HTML tables, JSON 16 | - **Output formats:** lines, DSV, XLS, XLSX, HTML tables, JSON 17 | - **Custom table names:** yes 18 | - **Custom column names:** yes 19 | - **Keep database file:** n/a 20 | - **Skip input fields:** no 21 | - **Skip input records (lines):** no 22 | - **Merge input fields:** no 23 | - **Database table customization:** yes, can create custom table then import into it 24 | - **SQL dump:** yes 25 | - **Other:** 26 | - [csvq](https://github.com/mithrandie/csvq) 27 | - **Documentation link:** 28 | - **Programming language:** Go 29 | - **Database:** custom SQL interpreter 30 | - **Column names from header row:** yes, optional 31 | - **Custom character encoding:** yes, input and output 32 | - **Custom input field separator:** yes, character 33 | - **Custom input record separator:** no 34 | - **Custom output field separator:** yes 35 | - **Custom output record separator:** no 36 | - **JOINs:** yes 37 | - **Use as library:** yes, Go 38 | - **Input formats:** CSV, TSV, LTSV, fixed-width, JSON 39 | - **Output formats:** CSV, TSV, LTSV, fixed-width, JSON, Markdown-style table, Org-mode, ASCII table 40 | - **Custom table names:** yes 41 | - **Custom column names:** yes 42 | - **Keep database file:** n/a 43 | - **Skip input fields:** no 44 | - **Skip input records (lines):** no 45 | - **Merge input fields:** no 46 | - **Database table customization:** yes, ALTER TABLE 47 | - **SQL dump:** no 48 | - **Other:** 49 | - [csvsql](https://github.com/wireservice/csvkit) 50 | - **Documentation link:** 51 | - **Programming language:** Python 52 | - **Database:** Firebird/MS SQL/MySQL/Oracle/PostgreSQL/SQLite 3/Sybase 53 | - **Column names from header row:** yes, optional 54 | - **Custom character encoding:** yes, input and output 55 | - **Custom input field separator:** yes, string 56 | - **Custom input record separator:** no 57 | - **Custom output field separator:** yes 58 | - **Custom output record separator:** no 59 | - **JOINs:** yes 60 | - **Use as library:** yes, Python 61 | - **Input formats:** delimited without quotes, DSV, Excel, JSON, SQL, fixed-width, DBF, and others (separate converters) 62 | - **Output formats:** delimited without quotes, DSV, JSON, Markdown-style table, SQL (separate converters) 63 | - **Custom table names:** yes 64 | - **Custom column names:** no 65 | - **Keep database file:** yes 66 | - **Skip input fields:** yes (separate tool) 67 | - **Skip input records (lines):** no 68 | - **Merge input fields:** no? 69 | - **Database table customization:** yes, UNIQUE constraints, database schema name, automatic column datatype or text 70 | - **SQL dump:** yes 71 | - **Other:** 72 | - [DuckDB](https://github.com/duckdb/duckdb) 73 | - **Documentation link:** 74 | - **Programming language:** C++ 75 | - **Database:** DuckDB 76 | - **Column names from header row:** yes, optional 77 | - **Custom character encoding:** no, only UTF-8 78 | - **Custom input field separator:** yes, string 79 | - **Custom input record separator:** no 80 | - **Custom output field separator:** yes 81 | - **Custom output record separator:** no 82 | - **JOINs:** yes 83 | - **Use as library:** yes, C, C++, JavaScript, Python, and other languages 84 | - **Input formats:** DSV, Excel, JSON, SQL, Parquet 85 | - **Output formats:** DSV, Excel, JSON, SQL, Parquet 86 | - **Custom table names:** yes 87 | - **Custom column names:** yes 88 | - **Keep database file:** yes 89 | - **Skip input fields:** yes 90 | - **Skip input records (lines):** yes 91 | - **Merge input fields:** no 92 | - **Database table customization:** yes, `CREATE TABLE` 93 | - **SQL dump:** yes 94 | - **Other:** ["DuckDB as the New jq"](https://www.pgrs.net/2024/03/21/duckdb-as-the-new-jq/), Paul Gross (2024). 95 | - [fsql](https://metacpan.org/release/App-fsql) 96 | - **Documentation link:** 97 | - **Programming language:** Perl 98 | - **Database:** custom SQL interpreter 99 | - **Column names from header row:** yes, always 100 | - **Custom character encoding:** no 101 | - **Custom input field separator:** no 102 | - **Custom input record separator:** no 103 | - **Custom output field separator:** no 104 | - **Custom output record separator:** no 105 | - **JOINs:** yes 106 | - **Use as library:** yes, Perl 107 | - **Input formats:** CSV, TSV, LTSV, Perl, JSON, YAML 108 | - **Output formats:** CSV, TSV, LTSV, Perl, JSON, YAML 109 | - **Custom table names:** yes 110 | - **Custom column names:** no 111 | - **Keep database file:** no 112 | - **Skip input fields:** no 113 | - **Skip input records (lines):** no 114 | - **Merge input fields:** no 115 | - **Database table customization:** no 116 | - **SQL dump:** no 117 | - **Other:** 118 | - [q](https://github.com/harelba/q) 119 | - **Documentation link:** 120 | - **Programming language:** Python 121 | - **Database:** SQLite 3 122 | - **Column names from header row:** yes, optional 123 | - **Custom character encoding:** yes, input and output 124 | - **Custom input field separator:** yes, string 125 | - **Custom input record separator:** no 126 | - **Custom output field separator:** yes 127 | - **Custom output record separator:** no 128 | - **JOINs:** yes 129 | - **Use as library:** yes, Python 130 | - **Input formats:** delimited without quotes, DSV 131 | - **Output formats:** delimited without quotes, DSV, custom using Python formatting string 132 | - **Custom table names:** no 133 | - **Custom column names:** no 134 | - **Keep database file:** yes 135 | - **Skip input fields:** no 136 | - **Skip input records (lines):** no 137 | - **Merge input fields:** no 138 | - **Database table customization:** yes, automatic column datatype or text 139 | - **SQL dump:** no 140 | - **Other:** 141 | - [RBQL](https://github.com/mechatroner/RBQL) 142 | - **Documentation link:** 143 | - **Programming language:** JavaScript, Python 144 | - **Database:** custom SQL interpreter 145 | - **Column names from header row:** yes, optional 146 | - **Custom character encoding:** yes, input 147 | - **Custom input field separator:** yes, string 148 | - **Custom input record separator:** no 149 | - **Custom output field separator:** yes 150 | - **Custom output record separator:** no 151 | - **JOINs:** yes 152 | - **Use as library:** yes, JavaScript and Python 153 | - **Input formats:** DSV 154 | - **Output formats:** DSV 155 | - **Custom table names:** no 156 | - **Custom column names:** no 157 | - **Keep database file:** n/a 158 | - **Skip input fields:** no 159 | - **Skip input records (lines):** no 160 | - **Merge input fields:** no 161 | - **Database table customization:** no 162 | - **SQL dump:** no 163 | - **Other:** 164 | - [rows](https://github.com/turicas/rows) 165 | - **Documentation link:** 166 | - **Programming language:** Python 167 | - **Database:** SQLite 3 168 | - **Column names from header row:** yes, always? 169 | - **Custom character encoding:** no 170 | - **Custom input field separator:** no 171 | - **Custom input record separator:** no 172 | - **Custom output field separator:** no 173 | - **Custom output record separator:** no 174 | - **JOINs:** no 175 | - **Use as library:** yes, Python 176 | - **Input formats:** CSV, JSON, XLS, XLSX, ODS, and others 177 | - **Output formats:** CSV, JSON, XLS, XLSX, ODS, and others 178 | - **Custom table names:** no 179 | - **Custom column names:** no 180 | - **Keep database file:** no 181 | - **Skip input fields:** no 182 | - **Skip input records (lines):** no 183 | - **Merge input fields:** no 184 | - **Database table customization:** no 185 | - **SQL dump:** no 186 | - **Other:** 187 | - [Sqawk](https://github.com/dbohdan/sqawk) 188 | - **Documentation link:** 189 | - **Programming language:** Tcl 190 | - **Database:** SQLite 3 191 | - **Column names from header row:** yes, optional 192 | - **Custom character encoding:** no 193 | - **Custom input field separator:** yes, regexp, per-file 194 | - **Custom input record separator:** yes, regexp, per-file 195 | - **Custom output field separator:** yes 196 | - **Custom output record separator:** yes 197 | - **JOINs:** yes 198 | - **Use as library:** yes, Tcl 199 | - **Input formats:** delimited without quotes, DSV, Tcl 200 | - **Output formats:** delimited without quotes, CSV, JSON, ASCII/Unicode table, Tcl 201 | - **Custom table names:** yes 202 | - **Custom column names:** yes 203 | - **Keep database file:** yes 204 | - **Skip input fields:** yes, any 205 | - **Skip input records (lines):** no 206 | - **Merge input fields:** yes, any consecutive 207 | - **Database table customization:** yes, column datatypes 208 | - **SQL dump:** no 209 | - **Other:** 210 | - [sqawk](https://github.com/tjunier/sqawk) 211 | - **Documentation link:** 212 | - **Programming language:** C 213 | - **Database:** SQLite 3 214 | - **Column names from header row:** yes, optional 215 | - **Custom character encoding:** no 216 | - **Custom input field separator:** yes, string, per-file 217 | - **Custom input record separator:** no 218 | - **Custom output field separator:** no 219 | - **Custom output record separator:** no 220 | - **JOINs:** yes 221 | - **Use as library:** no 222 | - **Input formats:** DSV 223 | - **Output formats:** CSV 224 | - **Custom table names:** yes 225 | - **Custom column names:** no 226 | - **Keep database file:** yes 227 | - **Skip input fields:** no 228 | - **Skip input records (lines):** yes, until regexp matches 229 | - **Merge input fields:** no 230 | - **Database table customization:** yes, primary key, indexes, foreign key constraints, automatic column datatype or text 231 | - **SQL dump:** yes 232 | - **Other:** chunked mode (read and process only N lines at a time) 233 | - [Squawk](https://github.com/samuel/squawk) 234 | - **Documentation link:** <> 235 | - **Programming language:** Python 236 | - **Database:** custom SQL interpreter 237 | - **Column names from header row:** yes, always 238 | - **Custom character encoding:** no 239 | - **Custom input field separator:** no 240 | - **Custom input record separator:** no 241 | - **Custom output field separator:** no 242 | - **Custom output record separator:** no 243 | - **JOINs:** no 244 | - **Use as library:** yes, Python 245 | - **Input formats:** CSV, Apache and Nginx log files 246 | - **Output formats:** table, CSV, JSON 247 | - **Custom table names:** no 248 | - **Custom column names:** no 249 | - **Keep database file:** no 250 | - **Skip input fields:** no 251 | - **Skip input records (lines):** no 252 | - **Merge input fields:** no 253 | - **Database table customization:** no 254 | - **SQL dump:** yes 255 | - **Other:** 256 | - [termsql](https://github.com/tobimensch/termsql) 257 | - **Documentation link:** 258 | - **Programming language:** Python 259 | - **Database:** SQLite 3 260 | - **Column names from header row:** yes, optional 261 | - **Custom character encoding:** no 262 | - **Custom input field separator:** yes, regexp 263 | - **Custom input record separator:** no 264 | - **Custom output field separator:** yes 265 | - **Custom output record separator:** no 266 | - **JOINs:** no 267 | - **Use as library:** no 268 | - **Input formats:** DSV, “vertical” DSV (lines as columns) 269 | - **Output formats:** delimited without quotes, CSV, TSV, HTML, SQL, Tcl 270 | - **Custom table names:** yes 271 | - **Custom column names:** yes 272 | - **Keep database file:** yes 273 | - **Skip input fields:** no 274 | - **Skip input records (lines):** yes, N first and M last 275 | - **Merge input fields:** yes, Nth to last 276 | - **Database table customization:** yes, primary key 277 | - **SQL dump:** yes 278 | - **Other:** 279 | - [trdsql](https://github.com/noborus/trdsql) 280 | - **Documentation link:** 281 | - **Programming language:** Go 282 | - **Database:** MySQL/PostgreSQL/SQLite 3 283 | - **Column names from header row:** yes, optional 284 | - **Custom character encoding:** no 285 | - **Custom input field separator:** yes, string 286 | - **Custom input record separator:** no 287 | - **Custom output field separator:** no 288 | - **Custom output record separator:** no 289 | - **JOINs:** yes 290 | - **Use as library:** no 291 | - **Input formats:** CSV, LTSV, JSON 292 | - **Output formats:** delimited without quotes, CSV, LTSV, JSON, ASCII table, Markdown 293 | - **Custom table names:** no 294 | - **Custom column names:** no 295 | - **Keep database file:** yes 296 | - **Skip input fields:** no 297 | - **Skip input records (lines):** no 298 | - **Merge input fields:** no 299 | - **Database table customization:** no 300 | - **SQL dump:** no 301 | - **Other:** 302 | - [textql](https://github.com/dinedal/textql) 303 | - **Documentation link:** 304 | - **Programming language:** Go 305 | - **Database:** SQLite 3 306 | - **Column names from header row:** yes, optional 307 | - **Custom character encoding:** no 308 | - **Custom input field separator:** yes, string 309 | - **Custom input record separator:** no 310 | - **Custom output field separator:** no 311 | - **Custom output record separator:** no 312 | - **JOINs:** no 313 | - **Use as library:** no 314 | - **Input formats:** DSV 315 | - **Output formats:** DSV 316 | - **Custom table names:** no 317 | - **Custom column names:** no 318 | - **Keep database file:** yes 319 | - **Skip input fields:** no 320 | - **Skip input records (lines):** no 321 | - **Merge input fields:** no 322 | - **Database table customization:** no 323 | - **SQL dump:** no 324 | - **Other:** 325 | - [Musoq](https://github.com/Puchaczov/Musoq) 326 | - **Documentation link:** https://github.com/Puchaczov/Musoq 327 | - **Programming language:** C# 328 | - **Database:** custom SQL interpreter 329 | - **Column names from header row:** yes, optional 330 | - **Custom character encoding:** no 331 | - **Custom input field separator:** depends on the data source 332 | - **Custom input record separator:** no 333 | - **Custom output field separator:** no 334 | - **Custom output record separator:** no 335 | - **JOINs:** yes 336 | - **Use as library:** yes 337 | - **Input formats:** various 338 | - **Output formats:** CSV, JSON 339 | - **Custom table names:** no 340 | - **Custom column names:** no 341 | - **Keep database file:** no 342 | - **Skip input fields:** yes 343 | - **Skip input records (lines):** no 344 | - **Merge input fields:** yes 345 | - **Database table customization:** no 346 | - **SQL dump:** no 347 | - **Other:** 348 | -------------------------------------------------------------------------------- /sql-based.md.jinja: -------------------------------------------------------------------------------- 1 | {% from "macros.md.jinja" import items -%} 2 | 3 | # SQL-based tools 4 | 5 | {{ items("sql-based") }} 6 | --------------------------------------------------------------------------------