├── .github └── workflows │ └── run-tox.yaml ├── .gitignore ├── .readthedocs.yml ├── CHANGELOG.rst ├── LICENSE ├── README.rst ├── doc ├── Makefile └── source │ ├── api.rst │ ├── cli.rst │ ├── conf.py │ ├── index.rst │ ├── introduction.rst │ └── release.rst ├── gjson ├── __init__.py ├── _cli.py ├── _gjson.py ├── _protocols.py ├── exceptions.py └── py.typed ├── prospector.yaml ├── ruff.toml ├── setup.cfg ├── setup.py ├── tests ├── ruff.toml └── unit │ ├── __init__.py │ ├── test__cli.py │ └── test_init.py └── tox.ini /.github/workflows/run-tox.yaml: -------------------------------------------------------------------------------- 1 | name: Run tox 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | 9 | jobs: 10 | run-tox: 11 | runs-on: ubuntu-latest 12 | strategy: 13 | matrix: 14 | python: ["3.9", "3.10", "3.11", "3.12"] 15 | 16 | steps: 17 | - uses: actions/checkout@v3 18 | - name: Setup Python 19 | uses: actions/setup-python@v3 20 | with: 21 | python-version: ${{ matrix.python }} 22 | - name: Install tox and any other packages 23 | run: pip install tox 24 | - name: Run tox 25 | run: tox 26 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # Distribution / packaging 7 | /build/ 8 | /dist/ 9 | /.eggs/ 10 | /*.egg-info/ 11 | 12 | # Unit test / coverage reports 13 | /.tox/ 14 | /.coverage 15 | /.coverage.* 16 | /.pytest_cache/ 17 | 18 | # Sphinx documentation 19 | /doc/build/ 20 | 21 | # pyenv 22 | /.python-version 23 | 24 | # Environments 25 | /.venv/ 26 | 27 | # mypy 28 | /.mypy_cache/ 29 | 30 | # Vim 31 | *.sw? 32 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | build: 4 | os: "ubuntu-22.04" 5 | tools: 6 | python: "3" 7 | 8 | python: 9 | install: 10 | - method: pip 11 | path: . 12 | extra_requirements: 13 | - tests 14 | 15 | sphinx: 16 | configuration: "doc/source/conf.py" 17 | fail_on_warning: true 18 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Release Notes 2 | ============= 3 | 4 | `v1.0.0`_ (2023-01-24) 5 | ^^^^^^^^^^^^^^^^^^^^^^ 6 | 7 | With this release ``gjson-py`` has reached pretty much feature parity with `GJSON`_, with only some inherent minor 8 | caveats, differences and limitations due to the different programming languages and technologies used. With this 9 | release ``gjson-py`` can also be considered stable and production ready. 10 | 11 | Minor improvements 12 | """""""""""""""""" 13 | 14 | * Modifiers: add support for missing upstream GJSON modifiers: ``@tostr``, ``@fromstr``, ``@group`` and ``@join``. 15 | * gjson: make ``__version__`` more reliable: 16 | 17 | * Migrate the reading of the version from ``pkg_resources`` to ``importlib`` as it's more efficient and avoids 18 | an additional dependency. 19 | * If the version is not detectable, as a fallback, attempt to read the ``SETUPTOOLS_SCM_PRETEND_VERSION`` environment 20 | variable as documented in ``setuptools_scm``. 21 | 22 | Bug fixes 23 | """"""""" 24 | 25 | * I/O: allow control characters in all JSON inputs, loading the data with ``strict=False`` both for the JSON input 26 | and when parsing eventual JSON bits present in the query. 27 | * gjson: fix some corner cases: 28 | 29 | * In some corner cases gjson was not parsing correctly the query string. 30 | * Introduce a new ``GJSONInvalidSyntaxError`` exception to distinguish some special parsing errors. 31 | * Extract the check for sequence objects into a private static method. 32 | 33 | Miscellanea 34 | """"""""""" 35 | 36 | * setup.py: add ``python_requires`` to prevent the package to be installed in an unsupported version of Python. 37 | * setup.py: update classifiers. 38 | * setup.py: mark the package as typed according to PEP 561. 39 | * setup.py: mark Python 3.11 as officially supported. 40 | * tox.ini: add Python 3.11 in the test matrix of tox. 41 | * documentation: reduce the ``maxdepth`` property for the release notes page to not show all releases to avoid showing 42 | all the releases in the documentation home page. 43 | * documentation: clarify limits on multipaths keys, specifying that in gjson-py if a key of a multipath object is 44 | given, it must be specified as a JSON string (e.g. ``"key":query``) and that bare words (e.g. ``key:query``) are not 45 | accepted although they are in GJSON. 46 | 47 | `v0.4.0`_ (2022-11-12) 48 | ^^^^^^^^^^^^^^^^^^^^^^ 49 | 50 | New features 51 | """""""""""" 52 | 53 | * Query parsing: add support for the `GJSON Literals`_ feature. 54 | * Queries: add support for nested queries. There is no limit on the number of levels queries can be nested. 55 | 56 | Bug fixes 57 | """"""""" 58 | 59 | * Queries: fix bug on first element query without operator: 60 | 61 | * When performing a query for the first element ``#(...)`` and there is no match, an exception should be raised. 62 | * In the case of a key existence check when there is no operator (e.g. ``friends.#(nonexistent)``) an empty array was 63 | erroneously returned instead of raising an exception. 64 | 65 | Miscellanea 66 | """"""""""" 67 | 68 | * documentation: add missing method docstring. 69 | * documentation: add note to modifiers specifying that the ``@keys`` and ``@values`` modifiers are valid only if applied 70 | to a JSON object (mapping). 71 | * Query parsing: simplify internal method to find a matching parentheses for queries and multipaths. 72 | 73 | `v0.3.0`_ (2022-11-10) 74 | ^^^^^^^^^^^^^^^^^^^^^^ 75 | 76 | New features 77 | """""""""""" 78 | 79 | * Query parsing: add `GJSON multipaths`_ support. 80 | 81 | Bug fixes 82 | """"""""" 83 | 84 | * Query parsing: fix integer mapping keys 85 | 86 | * When after a hash ``#`` or a query that returns all items ``#(...)#``, if there is an integer key, return any 87 | matching key from the resulting items from the query if they match the integer key (as string, as per JSON 88 | specifications). 89 | 90 | `v0.2.1`_ (2022-10-25) 91 | ^^^^^^^^^^^^^^^^^^^^^^ 92 | 93 | Bug fixes 94 | """"""""" 95 | 96 | * Query parsing: fix modifier options: 97 | 98 | * Fix a bug that was failing to properly get the modifier name when there were options and the modifier was not the 99 | first path in the query. 100 | 101 | `v0.2.0`_ (2022-10-24) 102 | ^^^^^^^^^^^^^^^^^^^^^^ 103 | 104 | New features 105 | """""""""""" 106 | 107 | * Query parsing: fully rewrite of the query parser: 108 | 109 | * Until now the parsing was mostly relying on a couple of regular expressions with lookbehind assertions to take 110 | into account escaped characters. Although they were working fine and also allowed to sensibly speed up the first 111 | development of gjson-py, they also had two major limitations: 112 | 113 | * Could not work in all corner cases. 114 | * Prevented the implementation of the GJSON features still missing in gjson-py. 115 | 116 | * The parsing has been completely refactored using a more standard parser approach, that allows to fine-tune the 117 | parsing much more to cover all corner cases and also enables the development of GJSON features still missing. 118 | * There shouldn't be any difference for normal queries, but some corner cases might now return a proper error. 119 | * Introduced a new ``GJSONParseError`` for parser-specific errors, that inherits from GJSONError and also provides 120 | a graphic way to show where the parsing error occurred. Example output:: 121 | 122 | GJSONParseError: Invalid or unsupported query part `invalid`. 123 | Query: name.last.invalid 124 | -----------------^ 125 | 126 | Minor improvements 127 | """""""""""""""""" 128 | 129 | * Refactor code splitting it into multiple files: 130 | 131 | * Restructure the gjson code to split it into multiple files for ease of development and maintenance. 132 | * Keep all the split modules as private except the exceptions one, and re-export everything from the gjson module 133 | itself, both to keep backward compatibility and also for simplicity of use by the clients. 134 | 135 | * Custom modifiers: 136 | 137 | * Prevent to register custom modifiers with names that contains characters that are used by the GJSON grammair, 138 | raising a GJSONError exception. 139 | 140 | Miscellanea 141 | """"""""""" 142 | 143 | * README: clarify naming for nested queries, based on feedback from `issue #2`_. Also fix a typo. 144 | 145 | `v0.1.0`_ (2022-10-03) 146 | ^^^^^^^^^^^^^^^^^^^^^^ 147 | 148 | Minor improvements 149 | """""""""""""""""" 150 | 151 | * Modifiers: add ``@top_n`` modifier (not present in GJSON): 152 | 153 | * Add a ``@top_n`` modifier that optionally accepts as options the number of top common items to return: 154 | ``@top_n:{"n": 5}`` 155 | * If no options are provided all items are returned. 156 | * It requires a list of items as input and returns a dictionary with unique items as keys and the count of them as 157 | values. 158 | 159 | * Modifiers: add ``@sum_n`` modifier (not present in GJSON): 160 | 161 | * Add a ``@sum_n`` modifier that will work on a sequence of objects, grouping the items with the same value for a 162 | given grouping key and sum the values of a sum key for each of them. 163 | * The options are mandatory and must specify the key to use for grouping and the key to use for summing: 164 | ``{"group": "somekey", "sum": "anotherkey"}``. Optionally specifying the ``n`` parameter to just return the top N 165 | results based on the summed value: ``{"group": "somekey", "sum": "anotherkey", "n": 5}`` 166 | * It requires a list of objects as input and returns a dictionary with unique items as keys and the sum of their 167 | values as values. 168 | 169 | Bug fixes 170 | """"""""" 171 | 172 | * Output: fix unicode handling: 173 | 174 | * Fix the default behaviour ensuring non-ASCII characters are returned as-is. 175 | * Add a new modifier ``@ascii``, that when set will escape all non-ASCII characters. 176 | 177 | * CLI: fix encoding handling: 178 | 179 | * Use the ``surrogateescape`` Python mode when reading the input and back when printing the output to prevent 180 | failures when parsing the input and reducing the loss of data. 181 | 182 | Miscellanea 183 | """"""""""" 184 | 185 | * documentation: add mention to Debian packaging and the availability of Debian packages for the project. 186 | * Type hints: use native types when possible. Instead of importing from ``typing`` use directly the native types when 187 | they support the ``[]`` syntax added in Python 3.9. 188 | * documentation: refactor the modifiers documentation to clearly split the GJSON modifiers supported by gjson-py and 189 | the additional modifiers specific to gjson-py with more detailed explanation and example usage for the additional 190 | ones. 191 | * setup.py: mark project as Beta for this ``v0.1.0`` release and add an additional keyword for PyPI indexing. 192 | 193 | `v0.0.5`_ (2022-08-05) 194 | ^^^^^^^^^^^^^^^^^^^^^^ 195 | 196 | New features 197 | """""""""""" 198 | 199 | * Queries: add support for the tilde operator: 200 | 201 | * When performing queries on arrays, add support for the Go GJSON tilde operator to perform truthy-ness comparison. 202 | * The comparison is based on Python's definition of truthy-ness, hence the actual results might differ from the ones 203 | in the Go package. 204 | 205 | Minor improvements 206 | """""""""""""""""" 207 | 208 | * documentation: add man page for the gjson binary. 209 | 210 | `v0.0.4`_ (2022-06-11) 211 | ^^^^^^^^^^^^^^^^^^^^^^ 212 | 213 | New features 214 | """""""""""" 215 | 216 | * CLI: improve the JSON Lines support allowing to use the ``-l/--lines`` CLI argument and the special query prefix 217 | ``..`` syntax together to encapsulate each parsed line in an array to enable filtering using the Queries 218 | capabilities. 219 | 220 | Minor improvements 221 | """""""""""""""""" 222 | 223 | * CLI: the input file CLI argument is now optional, defaulting to read from stdin. The equivalent of passing ``-``. 224 | * Modifiers: add support for the upstream Go GJSON modifier ``@this``, that just returns the current object. 225 | 226 | Miscellanea 227 | """"""""""" 228 | 229 | * Documentation: add a section to with examples on how to use the CLI. 230 | * CLI: add a link at the bottom of the help message of the CLI to the online documentation. 231 | 232 | `v0.0.3`_ (2022-06-11) 233 | ^^^^^^^^^^^^^^^^^^^^^^ 234 | 235 | New features 236 | """""""""""" 237 | 238 | * Add CLI support for JSON Lines: 239 | 240 | * Add a ``-l/--lines`` CLI argument to specify that the input file/stream is made of one JSON per line. 241 | * When used, gjson applies the same query to all lines. 242 | * Based on the verbosity level the failing lines are completely ignored, an error message is printed to stderr or 243 | the execution is interrupted at the first error printing the full traceback. 244 | 245 | * Add CLI support for GJSON JSON Lines queries: 246 | 247 | * Add support for the GJSON queries that encapsulates a JSON Lines input in an array when the query starts with 248 | ``..`` so that they the data can be queries as if it was an array of objects in the CLI. 249 | 250 | * Add support for custom modifiers: 251 | 252 | * Add a ``ModifierProtocol`` to describe the interface that custom modifiers callable need to have. 253 | * Add a ``register_modifier()`` method in the ``GJSON`` class to register custom modifiers. 254 | * Allow to pass a dictionary of modifiers to the low-level ``GJSONObj`` class constructor. 255 | * Add a ``GJSONObj.builtin_modifiers()`` static method that returns a set with the names of the built-in modifiers. 256 | * Is not possible to register a custom modifier with the same name of a built-in modifier. 257 | * Clarify in the documentation that only JSON objects are accepted as modifier arguments. 258 | 259 | Bug fixes 260 | """"""""" 261 | 262 | * Query parsing: when using the queries GJSON syntax ``#(...)`` and ``#(...)#`` fix the return value in case of a key 263 | matching that doesn't match any element. 264 | 265 | * Query parsing fixes/improvements found with the Python fuzzing engine Atheris: 266 | 267 | * If any query parts between delimiters is empty error out with a specific message instead of hitting a generic 268 | ``IndexError``. 269 | * When a query has an integer index on a mapping object, in case the element is not present, raise a ``GJSONError`` 270 | instead of a ``KeyError`` one. 271 | * When the query has a wildcard matching, ensure that it's applied on a mapping object. Fail with a ``GJSONError`` 272 | otherwise. 273 | * Explicitly catch malformed modifier options and raise a ``GJSONError`` instead. 274 | * If the last part of the query is a ``#``, check that the object is actually a sequence like object and fail with 275 | a specific message if not. 276 | * Ensure all the conditions are valid before attempting to extract the inner element of a sequence like object. 277 | Ignore both non-mapping like objects inside the sequence or mapping like objects that don't have the specified key. 278 | * When parsing the query value as JSON catch the eventual decoding error to encapsulate it into a ``GJSONError`` one. 279 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` accept also an empty query to follow the same 280 | behaviour of the upstream Go GJSON. 281 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` follow closely the upstream behaviour of Go GJSON 282 | for all items queries ``#(..)#`` with regex matching. 283 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` fix the wildcard matching regular expression when 284 | using pattern matching. 285 | * Fix the regex to match keys in presence of wildcards escaping only the non-wildcards and ensuring to not 286 | double-escaping any already escaped wildcard. 287 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` ensure any exception raised while comparing 288 | incompatible objects is catched and raise as a GJSONError. 289 | 290 | Miscellanea 291 | """"""""""" 292 | 293 | * tests: when matching exception messages always escape the string or use raw strings to avoid false matchings. 294 | * pylint: remove unnecessary comments 295 | 296 | `v0.0.2`_ (2022-05-31) 297 | ^^^^^^^^^^^^^^^^^^^^^^ 298 | 299 | Bug fixes 300 | """"""""" 301 | 302 | * ``@sort`` modifier: fix the actual sorting. 303 | * tests: ensure that mapping-like objects are compared also in the order of their keys. 304 | 305 | Miscellanea 306 | """"""""""" 307 | 308 | * GitHub actions: add workflow to run tox. 309 | * GitHub actions: fix branch name for pushes 310 | * documentation: include also the ``@sort`` modifier that is not present in the GJSON project. 311 | * documentation: fix link to PyPI package. 312 | * documentation: add link to the generated docs. 313 | * documentation: fix section hierarchy and build. 314 | 315 | `v0.0.1`_ (2022-05-22) 316 | ^^^^^^^^^^^^^^^^^^^^^^ 317 | 318 | * Initial version. 319 | 320 | .. _`GJSON`: https://github.com/tidwall/gjson/ 321 | .. _`GJSON Literals`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#literals 322 | .. _`GJSON Multipaths`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#multipath 323 | 324 | .. _`issue #2`: https://github.com/volans-/gjson-py/issues/2 325 | 326 | .. _`v0.0.1`: https://github.com/volans-/gjson-py/releases/tag/v0.0.1 327 | .. _`v0.0.2`: https://github.com/volans-/gjson-py/releases/tag/v0.0.2 328 | .. _`v0.0.3`: https://github.com/volans-/gjson-py/releases/tag/v0.0.3 329 | .. _`v0.0.4`: https://github.com/volans-/gjson-py/releases/tag/v0.0.4 330 | .. _`v0.0.5`: https://github.com/volans-/gjson-py/releases/tag/v0.0.5 331 | .. _`v0.1.0`: https://github.com/volans-/gjson-py/releases/tag/v0.1.0 332 | .. _`v0.2.0`: https://github.com/volans-/gjson-py/releases/tag/v0.2.0 333 | .. _`v0.2.1`: https://github.com/volans-/gjson-py/releases/tag/v0.2.1 334 | .. _`v0.3.0`: https://github.com/volans-/gjson-py/releases/tag/v0.3.0 335 | .. _`v0.4.0`: https://github.com/volans-/gjson-py/releases/tag/v0.4.0 336 | .. _`v1.0.0`: https://github.com/volans-/gjson-py/releases/tag/v1.0.0 337 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | .. image:: https://github.com/volans-/gjson-py/actions/workflows/run-tox.yaml/badge.svg 2 | :alt: CI results 3 | :target: https://github.com/volans-/gjson-py/actions/workflows/run-tox.yaml 4 | 5 | Introduction 6 | ============ 7 | 8 | gjson-py is a Python package that provides a simple way to filter and extract data from JSON-like objects or JSON 9 | files, using the `GJSON`_ syntax. 10 | 11 | It is, compatibly with the language differences and with some limitation, the Python equivalent of the Go 12 | `GJSON`_ package. 13 | The main difference from GJSON is that gjson-py doesn't work directly with JSON strings but instead with 14 | JSON-like Python objects, that can either be the resulting object when calling ``json.load()`` or ``json.loads()``, 15 | or any Python object that is JSON-serializable. 16 | 17 | A detailed list of the GJSON features supported by gjson-py is provided below. 18 | 19 | See also the full `gjson-py documentation`_. 20 | 21 | Installation 22 | ------------ 23 | 24 | gjson-py is available on the `Python Package Index`_ (PyPI) and can be easily installed with:: 25 | 26 | pip install gjson 27 | 28 | It's also available as a Debian package (`python3-gjson`_) on Debian systems starting from Debian 12 (*bookworm*) and 29 | can be installed with:: 30 | 31 | apt-get install python3-gjson 32 | 33 | A ``.deb`` package for the current stable and unstable Debian versions is also available for download on the 34 | `releases page on GitHub`_. 35 | 36 | How to use the library 37 | ---------------------- 38 | 39 | gjson-py provides different ways to perform queries on JSON-like objects. 40 | 41 | ``gjson.get()`` 42 | ^^^^^^^^^^^^^^^ 43 | 44 | A quick accessor to GJSON functionalities exposed for simplicity of use. Particularly useful to perform a single 45 | query on a given object:: 46 | 47 | >>> import gjson 48 | >>> data = {'name': {'first': 'Tom', 'last': 'Anderson'}, 'age': 37} 49 | >>> gjson.get(data, 'name.first') 50 | 'Tom' 51 | 52 | It's also possible to make it return a JSON-encoded string and decide on failure if it should raise an exception 53 | or return `None`. See the full API documentation for more details. 54 | 55 | ``GJSON`` class 56 | ^^^^^^^^^^^^^^^ 57 | 58 | The ``GJSON`` class provides full access to the gjson-py API allowing to perform multiple queries on the same object:: 59 | 60 | >>> import gjson 61 | >>> data = {'name': {'first': 'Tom', 'last': 'Anderson'}, 'age': 37} 62 | >>> source = gjson.GJSON(data) 63 | >>> source.get('name.first') 64 | 'Tom' 65 | >>> str(source) 66 | '{"name": {"first": "Tom", "last": "Anderson"}, "age": 37}' 67 | >>> source.getj('name.first') 68 | '"Tom"' 69 | >>> name = source.get_gjson('name') 70 | >>> name.get('first') 71 | 'Tom' 72 | >>> name 73 | 74 | 75 | See the full API documentation for more details. 76 | 77 | How to use the CLI 78 | ------------------ 79 | 80 | gjson-py provides also a command line interface (CLI) for ease of use: 81 | 82 | .. code-block:: console 83 | 84 | $ echo '{"name": {"first": "Tom", "last": "Anderson"}, "age": 37}' > test.json 85 | $ cat test.json | gjson 'name.first' # Read from stdin 86 | "Tom" 87 | $ gjson test.json 'age' # Read from a file 88 | 37 89 | $ cat test.json | gjson - 'name.first' # Explicitely read from stdin 90 | "Tom" 91 | 92 | JSON Lines 93 | ^^^^^^^^^^ 94 | 95 | JSON Lines support in the CLI allows for different use cases. All the examples in this section operates on a 96 | ``test.json`` file generated with: 97 | 98 | .. code-block:: console 99 | 100 | $ echo -e '{"name": "Gilbert", "age": 61}\n{"name": "Alexa", "age": 34}\n{"name": "May", "age": 57}' > test.json 101 | 102 | Apply the same query to each line 103 | """"""""""""""""""""""""""""""""" 104 | 105 | Using the ``-l/--lines`` CLI argument, for each input line gjson-py applies the query and filters the data according 106 | to it. Lines are read one by one so there is no memory overhead for the processing. It can be used while tailing log 107 | files in JSON format for example. 108 | 109 | 110 | .. code-block:: console 111 | 112 | $ gjson --lines test.json 'age' 113 | 61 114 | 34 115 | 57 116 | $ tail -f log.json | gjson --lines 'bytes_sent' # Dummy example 117 | 118 | Encapsulate all lines in an array, then apply the query 119 | """"""""""""""""""""""""""""""""""""""""""""""""""""""" 120 | 121 | Using the special query prefix syntax ``..``, as described in GJSON's documentation for `JSON Lines`_, gjson-py will 122 | read all lines from the input and encapsulate them into an array. This approach has of course the memory overhead of 123 | loading the whole input to perform the query. 124 | 125 | .. code-block:: console 126 | 127 | $ gjson test.json '..#.name' 128 | ["Gilbert", "Alexa", "May"] 129 | 130 | Filter lines based on their values 131 | """""""""""""""""""""""""""""""""" 132 | 133 | Combining the ``-l/--lines`` CLI argument with the special query prefix ``..`` described above, it's possible to filter 134 | input lines based on their values. In this case gjson-py encapsulates each line in an array so that is possible to use 135 | the `Queries`_ GJSON syntax to filter them. As the ecapsulation is performed on each line, there is no memory overhead. 136 | Because technically when a line is filtered is because there was no match on the whole line query, the final exit code, 137 | if any line is filtered, will be ``1``. 138 | 139 | .. code-block:: console 140 | 141 | $ gjson --lines test.json '..#(age>40).name' 142 | "Gilbert" 143 | "May" 144 | 145 | Filter lines and apply query to the result 146 | """""""""""""""""""""""""""""""""""""""""" 147 | 148 | Combining the methods above is possible for example to filter/extract data from the lines first and then apply a query 149 | to the aggregated result. The memory overhead in this case is based on the amount of data resulting from the first 150 | filtering/extraction. 151 | 152 | .. code-block:: console 153 | 154 | $ gjson --lines test.json 'age' | gjson '..@sort' 155 | [34, 57, 61] 156 | $ gjson --lines test.json '..#(age>40).age' | gjson '..@sort' 157 | [57, 61] 158 | 159 | Query syntax 160 | ------------ 161 | 162 | For the generic query syntax refer to the original `GJSON Path Syntax`_ documentation. 163 | 164 | Supported GJSON features 165 | ^^^^^^^^^^^^^^^^^^^^^^^^ 166 | 167 | This is the list of GJSON features and how they are supported by gjson-py: 168 | 169 | 170 | +------------------------+------------------------+------------------------------------------------------+ 171 | | GJSON feature | Supported by gjson-py | Notes | 172 | +========================+========================+======================================================+ 173 | | `Path Structure`_ | YES | | 174 | +------------------------+------------------------+------------------------------------------------------+ 175 | | `Basic`_ | YES | | 176 | +------------------------+------------------------+------------------------------------------------------+ 177 | | `Wildcards`_ | YES | | 178 | +------------------------+------------------------+------------------------------------------------------+ 179 | | `Escape Character`_ | YES | | 180 | +------------------------+------------------------+------------------------------------------------------+ 181 | | `Arrays`_ | YES | | 182 | +------------------------+------------------------+------------------------------------------------------+ 183 | | `Queries`_ | YES | Using Python's operators [#]_ [#]_ | 184 | +------------------------+------------------------+------------------------------------------------------+ 185 | | `Dot vs Pipe`_ | YES | | 186 | +------------------------+------------------------+------------------------------------------------------+ 187 | | `Modifiers`_ | YES | See the table below for all the details | 188 | +------------------------+------------------------+------------------------------------------------------+ 189 | | `Modifier arguments`_ | YES | Only a JSON object is accepted as argument | 190 | +------------------------+------------------------+------------------------------------------------------+ 191 | | `Custom modifiers`_ | YES | Only a JSON object is accepted as argument [#]_ | 192 | +------------------------+------------------------+------------------------------------------------------+ 193 | | `Multipaths`_ | YES | Object keys, if specified, must be JSON strings [#]_ | 194 | +------------------------+------------------------+------------------------------------------------------+ 195 | | `Literals`_ | YES | Including infinite and NaN values [#]_ | 196 | +------------------------+------------------------+------------------------------------------------------+ 197 | | `JSON Lines`_ | YES | CLI support [#]_ [#]_ | 198 | +------------------------+------------------------+------------------------------------------------------+ 199 | 200 | .. [#] The queries matching is based on Python's operator and as such the results might be different than the ones from 201 | the Go GJSON package. In particular for the ``~`` operator that checks the truthy-ness of objects. 202 | .. [#] When using nested queries, only the outermost one controls whether to return only the first item or all items. 203 | .. [#] Custom modifiers names cannot contain reserved characters used by the GJSON grammar. 204 | .. [#] For example ``{"years":age}`` is valid while ``{years:age}`` is not, although that's valid in GJSON. 205 | .. [#] Those special cases are handled according to `Python's JSON documentation`_. 206 | .. [#] Both for applying the same query to each line using the ``-l/--lines`` argument and to automatically encapsulate 207 | the input lines in a list and apply the query to the list using the ``..`` special query prefix described in 208 | `JSON Lines`_. 209 | .. [#] Library support is not currently present because gjson-py accepts only Python objects, making it impossible to 210 | pass JSON Lines directly. The client is free to choose if calling gjson-py for each line or to encapsulate them in 211 | a list before calling gjson-py. 212 | 213 | This is the list of modifiers present in GJSON and how they are supported by gjson-py: 214 | 215 | +----------------+-----------------------+------------------------------------------+ 216 | | GJSON Modifier | Supported by gjson-py | Notes | 217 | +----------------+-----------------------+------------------------------------------+ 218 | | ``@reverse`` | YES | | 219 | +----------------+-----------------------+------------------------------------------+ 220 | | ``@ugly`` | YES | | 221 | +----------------+-----------------------+------------------------------------------+ 222 | | ``@pretty`` | PARTIALLY | The ``width`` argument is not supported | 223 | +----------------+-----------------------+------------------------------------------+ 224 | | ``@this`` | YES | | 225 | +----------------+-----------------------+------------------------------------------+ 226 | | ``@valid`` | YES | | 227 | +----------------+-----------------------+------------------------------------------+ 228 | | ``@flatten`` | YES | | 229 | +----------------+-----------------------+------------------------------------------+ 230 | | ``@join`` | PARTIALLY | Preserving duplicate keys not supported | 231 | +----------------+-----------------------+------------------------------------------+ 232 | | ``@keys`` | YES | Valid only on JSON objects (mappings) | 233 | +----------------+-----------------------+------------------------------------------+ 234 | | ``@values`` | YES | Valid only on JSON objects (mappings) | 235 | +----------------+-----------------------+------------------------------------------+ 236 | | ``@tostr`` | YES | | 237 | +----------------+-----------------------+------------------------------------------+ 238 | | ``@fromstr`` | YES | | 239 | +----------------+-----------------------+------------------------------------------+ 240 | | ``@group`` | YES | | 241 | +----------------+-----------------------+------------------------------------------+ 242 | 243 | 244 | Additional features 245 | ^^^^^^^^^^^^^^^^^^^ 246 | 247 | 248 | Additional modifiers 249 | """""""""""""""""""" 250 | 251 | This is the list of additional modifiers specific to gjson-py not present in GJSON: 252 | 253 | * ``@ascii``: escapes all non-ASCII characters when printing/returning the string representation of the object, 254 | ensuring that the output is made only of ASCII characters. It's implemented using the ``ensure_ascii`` arguments in 255 | the Python's ``json`` module. This modifier doesn't accept any arguments. 256 | * ``@sort``: sorts a mapping object by its keys or a sequence object by its values. This modifier doesn't accept any 257 | arguments. 258 | * ``@top_n``: given a sequence object groups the items in the sequence counting how many occurrences of each value are 259 | present. It returns a mapping object where the keys are the distinct values of the list and the values are the number 260 | of times the key was present in the list, ordered from the most common to the least common item. The items in the 261 | original sequence object must be Python hashable. This modifier accepts an optional argument ``n`` to return just the 262 | N items with the higher counts. When the ``n`` argument is not provided all items are returned. Example usage: 263 | 264 | .. code-block:: console 265 | 266 | $ echo '["a", "b", "c", "b", "c", "c"]' | gjson '@top_n' 267 | {"c": 3, "b": 2, "a": 1} 268 | $ echo '["a", "b", "c", "b", "c", "c"]' | gjson '@top_n:{"n":2}' 269 | {"c": 3, "b": 2} 270 | 271 | * ``@sum_n``: given a sequence of objects, groups the items in the sequence using a grouping key and sum the values of a 272 | sum key provided. It returns a mapping object where the keys are the distinct values of the grouping key and the 273 | values are the sums of all the values of the sum key for each distinct grouped key, ordered from the highest sum to 274 | the lowest. The values of the grouping key must be Python hashable. The values of the sum key must be integers or 275 | floats. This modifier required two mandatory arguments, ``group`` and ``sum`` that have as values the respective keys 276 | in the objects of the sequence. An optional ``n`` argument is also accepted to return just the top N items with the 277 | highest sum. Example usage: 278 | 279 | .. code-block:: console 280 | 281 | $ echo '[{"key": "a", "time": 1}, {"key": "b", "time": 2}, {"key": "c", "time": 3}, {"key": "a", "time": 4}]' > test.json 282 | $ gjson test.json '@sum_n:{"group": "key", "sum": "time"}' 283 | {"a": 5, "c": 3, "b": 2} 284 | $ gjson test.json '@sum_n:{"group": "key", "sum": "time", "n": 2}' 285 | {"a": 5, "c": 3} 286 | 287 | .. _`GJSON`: https://github.com/tidwall/gjson 288 | .. _`Python Package Index`: https://pypi.org/project/gjson/ 289 | .. _`GJSON Path Syntax`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md 290 | .. _`gjson-py documentation`: https://volans-.github.io/gjson-py/index.html 291 | .. _`releases page on GitHub`: https://github.com/volans-/gjson-py/releases 292 | .. _`Python's JSON documentation`: https://docs.python.org/3/library/json.html#infinite-and-nan-number-values 293 | .. _`python3-gjson`: https://packages.debian.org/sid/python3-gjson 294 | 295 | .. _`Path Structure`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#path-structure 296 | .. _`Basic`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#basic 297 | .. _`Wildcards`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#wildcards 298 | .. _`Escape Character`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#escape-character 299 | .. _`Arrays`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#arrays 300 | .. _`Queries`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#queries 301 | .. _`Dot vs Pipe`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#dot-vs-pipe 302 | .. _`Modifiers`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#modifiers 303 | .. _`Modifier arguments`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#modifiers 304 | .. _`Custom modifiers`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#custom-modifiers 305 | .. _`Multipaths`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#multipaths 306 | .. _`Literals`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#literals 307 | .. _`JSON Lines`: https://github.com/tidwall/gjson#json-lines 308 | -------------------------------------------------------------------------------- /doc/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = doc/source 9 | BUILDDIR = doc/build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /doc/source/api.rst: -------------------------------------------------------------------------------- 1 | API 2 | === 3 | 4 | .. automodule:: gjson 5 | -------------------------------------------------------------------------------- /doc/source/cli.rst: -------------------------------------------------------------------------------- 1 | CLI 2 | === 3 | 4 | .. argparse:: 5 | :module: gjson._cli 6 | :func: get_parser 7 | :prog: gjson 8 | :nodefault: 9 | 10 | -------------------------------------------------------------------------------- /doc/source/conf.py: -------------------------------------------------------------------------------- 1 | """Configuration file for the Sphinx documentation builder.""" 2 | # This file only contains a selection of the most common options. For a full 3 | # list see the documentation: 4 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 5 | 6 | import sys 7 | from pathlib import Path 8 | 9 | import sphinx_rtd_theme 10 | from pkg_resources import get_distribution 11 | 12 | # -- Path setup -------------------------------------------------------------- 13 | 14 | # If extensions (or modules to document with autodoc) are in another directory, 15 | # add these directories to sys.path here. If the directory is relative to the 16 | # documentation root, use os.path.abspath to make it absolute, like shown here. 17 | # 18 | # import os 19 | # import sys 20 | # sys.path.insert(0, os.path.abspath('.')) 21 | 22 | sys.path.insert(0, Path(__file__).resolve().parent.parent) 23 | 24 | # -- Project information ----------------------------------------------------- 25 | 26 | project = 'gjson' 27 | title = f'{project} Documentation' 28 | copyright = '2022, Riccardo Coccioli' 29 | author = 'Riccardo Coccioli' 30 | 31 | # The version info for the project you're documenting, acts as replacement for 32 | # |version| and |release|, also used in various other places throughout the 33 | # built documents. 34 | # 35 | # The full version, including alpha/beta/rc tags. 36 | release = get_distribution('gjson').version 37 | # The short X.Y version. 38 | version = release 39 | 40 | # -- General configuration --------------------------------------------------- 41 | 42 | # Add any Sphinx extension module names here, as strings. They can be 43 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 44 | # ones. 45 | extensions = [ 46 | 'sphinx.ext.autodoc', 47 | 'sphinx.ext.napoleon', 48 | 'sphinx.ext.intersphinx', 49 | 'sphinx.ext.todo', 50 | 'sphinx.ext.coverage', 51 | 'sphinx.ext.viewcode', 52 | 'sphinxarg.ext', 53 | 'sphinx_autodoc_typehints', 54 | ] 55 | 56 | # Add any paths that contain templates here, relative to this directory. 57 | templates_path = ['_templates'] 58 | 59 | # List of patterns, relative to source directory, that match files and 60 | # directories to ignore when looking for source files. 61 | # This pattern also affects html_static_path and html_extra_path. 62 | exclude_patterns = [] 63 | 64 | # -- Options for manual page output --------------------------------------- 65 | 66 | # One entry per manual page. List of tuples 67 | # (source start file, name, description, authors, manual section). 68 | man_pages = [ 69 | ('cli', 'gjson', 'filter and extract data from JSON-like files', [author], 1), 70 | ] 71 | 72 | # -- Options for HTML output ------------------------------------------------- 73 | 74 | # The theme to use for HTML and HTML Help pages. See the documentation for 75 | # a list of builtin themes. 76 | # 77 | html_theme = 'sphinx_rtd_theme' 78 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] 79 | 80 | # Add any paths that contain custom static files (such as style sheets) here, 81 | # relative to this directory. They are copied after the builtin static files, 82 | # so a file named "default.css" will overwrite the builtin "default.css". 83 | html_static_path = [] 84 | 85 | intersphinx_mapping = { 86 | 'python': ('https://docs.python.org/3/', None), 87 | } 88 | 89 | # Napoleon settings 90 | napoleon_google_docstring = True 91 | napoleon_numpy_docstring = False 92 | napoleon_include_init_with_doc = False 93 | napoleon_include_private_with_doc = False 94 | napoleon_include_special_with_doc = False 95 | napoleon_use_admonition_for_examples = False 96 | napoleon_use_admonition_for_notes = False 97 | napoleon_use_admonition_for_references = False 98 | napoleon_use_ivar = False 99 | napoleon_use_param = True 100 | napoleon_use_keyword = True 101 | napoleon_use_rtype = True 102 | napoleon_type_aliases = None 103 | napoleon_attr_annotations = True 104 | 105 | 106 | # Autodoc settings 107 | autodoc_default_options = { 108 | # Using None as value instead of True to support the version of Sphinx used in Buster 109 | 'members': None, 110 | 'member-order': 'bysource', 111 | 'show-inheritance': None, 112 | 'special-members': '__str__,__version__,__call__', 113 | } 114 | autoclass_content = 'both' 115 | 116 | -------------------------------------------------------------------------------- /doc/source/index.rst: -------------------------------------------------------------------------------- 1 | gjson-py |release| documentation 2 | ================================ 3 | 4 | .. toctree:: 5 | :maxdepth: 3 6 | 7 | introduction 8 | cli 9 | api 10 | 11 | 12 | .. toctree:: 13 | :maxdepth: 1 14 | 15 | release 16 | -------------------------------------------------------------------------------- /doc/source/introduction.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../../README.rst 2 | -------------------------------------------------------------------------------- /doc/source/release.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../../CHANGELOG.rst 2 | -------------------------------------------------------------------------------- /gjson/__init__.py: -------------------------------------------------------------------------------- 1 | """GJSON module.""" 2 | import json 3 | import os 4 | import re 5 | from importlib.metadata import PackageNotFoundError, version 6 | from typing import Any 7 | 8 | from gjson._gjson import MODIFIER_NAME_RESERVED_CHARS, GJSONObj 9 | from gjson._protocols import ModifierProtocol 10 | from gjson.exceptions import GJSONError, GJSONParseError 11 | 12 | # Explicit export of modules for the import * syntax, custom order to force the documentation order 13 | __all__ = ['get', 'GJSON', 'GJSONError', 'GJSONParseError', 'ModifierProtocol', 'GJSONObj', '__version__'] 14 | 15 | 16 | # TODO: use a proper type hint for obj once https://github.com/python/typing/issues/182 will be fixed 17 | def get(obj: Any, query: str, *, as_str: bool = False, quiet: bool = False) -> Any: 18 | """Quick accessor to GJSON functionalities exposed for simplicity of use. 19 | 20 | Examples: 21 | Import and directly use this quick helper for the simpler usage:: 22 | 23 | >>> import gjson 24 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]} 25 | >>> gjson.get(data, 'items.#.size') 26 | [1, 2] 27 | 28 | Arguments: 29 | obj: the object to query. It must be accessible in JSON-like fashion so it must be an object that can be 30 | converted to JSON. 31 | query: the query string to evaluate to extract the data from the object. 32 | as_str: if set to :py:data:`True` returns a JSON-encoded string, a Python object otherwise. 33 | quiet: on error, if set to :py:data:`True`, will raises an GJSONError exception. Otherwise returns 34 | :py:data:`None` on error. 35 | 36 | Return: 37 | the resulting object. 38 | 39 | """ 40 | gjson_obj = GJSON(obj) 41 | if as_str: 42 | return gjson_obj.getj(query, quiet=quiet) 43 | 44 | return gjson_obj.get(query, quiet=quiet) 45 | 46 | 47 | class GJSON: 48 | """The GJSON class to operate on JSON-like objects.""" 49 | 50 | def __init__(self, obj: Any): 51 | """Initialize the instance with the given object. 52 | 53 | Examples: 54 | Use the :py:class:`gjson.GJSON` class for more complex usage or to perform multiple queries on the same 55 | object:: 56 | 57 | >>> import gjson 58 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]} 59 | >>> gjson_obj = gjson.GJSON(data) 60 | 61 | Arguments: 62 | obj: the object to query. 63 | 64 | """ 65 | self._obj = obj 66 | self._custom_modifiers: dict[str, ModifierProtocol] = {} 67 | 68 | def __str__(self) -> str: 69 | """Return the current object as a JSON-encoded string. 70 | 71 | Examples: 72 | Converting to string a :py:class:`gjson.GJSON` object returns it as a JSON-encoded string:: 73 | 74 | >>> str(gjson_obj) 75 | '{"items": [{"name": "a", "size": 1}, {"name": "b", "size": 2}]}' 76 | 77 | Returns: 78 | the JSON-encoded string representing the instantiated object. 79 | 80 | """ 81 | return json.dumps(self._obj, ensure_ascii=False) 82 | 83 | def get(self, query: str, *, quiet: bool = False) -> Any: 84 | """Perform a query on the instantiated object and return the resulting object. 85 | 86 | Examples: 87 | Perform a query and get the resulting object:: 88 | 89 | >>> gjson_obj.get('items.#.size') 90 | [1, 2] 91 | 92 | Arguments: 93 | query: the GJSON query to apply to the object. 94 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in 95 | case of error. 96 | 97 | Raises: 98 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`. 99 | 100 | Returns: 101 | the resulting object or :py:data:`None` if the ``quiet`` parameter is :py:data:`True` and there was an 102 | error. 103 | 104 | """ 105 | try: 106 | return GJSONObj(self._obj, query, custom_modifiers=self._custom_modifiers).get() 107 | except GJSONError: 108 | if quiet: 109 | return None 110 | raise 111 | 112 | def getj(self, query: str, *, quiet: bool = False) -> str: 113 | """Perform a query on the instantiated object and return the resulting object as JSON-encoded string. 114 | 115 | Examples: 116 | Perform a query and get the resulting object as a JSON-encoded string:: 117 | 118 | >>> gjson_obj.getj('items.#.size') 119 | '[1, 2]' 120 | 121 | Arguments: 122 | query: the GJSON query to apply to the object. 123 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in 124 | case of error. 125 | 126 | Raises: 127 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`. 128 | 129 | Returns: 130 | the JSON-encoded string representing the resulting object or :py:data:`None` if the ``quiet`` parameter is 131 | :py:data:`True` and there was an error. 132 | 133 | """ 134 | try: 135 | return str(GJSONObj(self._obj, query, custom_modifiers=self._custom_modifiers)) 136 | except GJSONError: 137 | if quiet: 138 | return '' 139 | raise 140 | 141 | def get_gjson(self, query: str, *, quiet: bool = False) -> 'GJSON': 142 | """Perform a query on the instantiated object and return the resulting object as a GJSON instance. 143 | 144 | Examples: 145 | Perform a query and get the resulting object already encapsulated into a :py:class:`gjson.GJSON` object:: 146 | 147 | >>> sizes = gjson_obj.get_gjson('items.#.size') 148 | >>> str(sizes) 149 | '[1, 2]' 150 | >>> sizes.get('0') 151 | 1 152 | 153 | Arguments: 154 | query: the GJSON query to apply to the object. 155 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in 156 | case of error. 157 | 158 | Raises: 159 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`. 160 | 161 | Returns: 162 | the resulting object encapsulated as a :py:class:`gjson.GJSON` object or :py:data:`None` if the ``quiet`` 163 | parameter is :py:data:`True` and there was an error. 164 | 165 | """ 166 | return GJSON(self.get(query, quiet=quiet)) 167 | 168 | def register_modifier(self, name: str, func: ModifierProtocol) -> None: 169 | """Register a custom modifier. 170 | 171 | Examples: 172 | Register a custom modifier that sums all the numbers in a list: 173 | 174 | >>> def custom_sum(options, obj, *, last): 175 | ... # insert sanity checks code here 176 | ... return sum(obj) 177 | ... 178 | >>> gjson_obj.register_modifier('sum', custom_sum) 179 | >>> gjson_obj.get('items.#.size.@sum') 180 | 3 181 | 182 | Arguments: 183 | name: the modifier name. It will be called where ``@name`` is used in the query. If two custom modifiers 184 | are registered with the same name the last one will be used. 185 | func: the modifier code in the form of a callable object that adhere to the 186 | :py:class:`gjson.ModifierProtocol`. 187 | 188 | Raises: 189 | gjson.GJSONError: if the provided callable doesn't adhere to the :py:class:`gjson.ModifierProtocol`. 190 | 191 | """ 192 | # Escape the ] as they are inside a [...] block 193 | not_allowed_regex = ''.join(MODIFIER_NAME_RESERVED_CHARS).replace(']', r'\]') 194 | if re.search(fr'[{not_allowed_regex}]', name): 195 | not_allowed_string = ', '.join(f'`{i}`' for i in MODIFIER_NAME_RESERVED_CHARS) 196 | raise GJSONError(f'Unable to register modifier `{name}`, contains at least one not allowed character: ' 197 | f'{not_allowed_string}') 198 | 199 | if name in GJSONObj.builtin_modifiers(): 200 | raise GJSONError(f'Unable to register a modifier with the same name of the built-in modifier: @{name}.') 201 | 202 | if not isinstance(func, ModifierProtocol): 203 | raise GJSONError(f'The given func "{func}" for the custom modifier @{name} does not adhere ' 204 | 'to the gjson.ModifierProtocol.') 205 | 206 | self._custom_modifiers[name] = func 207 | 208 | 209 | try: 210 | __version__: str = version('gjson') 211 | """str: the version of the current gjson module.""" 212 | except PackageNotFoundError: # pragma: no cover - this should never happen during tests 213 | # Read the override from the environment, if present (like inside Debian build system) 214 | if 'SETUPTOOLS_SCM_PRETEND_VERSION' in os.environ: 215 | __version__ = os.environ['SETUPTOOLS_SCM_PRETEND_VERSION'] 216 | -------------------------------------------------------------------------------- /gjson/_cli.py: -------------------------------------------------------------------------------- 1 | """GJSON module.""" 2 | import argparse 3 | import json 4 | import sys 5 | from collections.abc import Sequence 6 | from typing import IO, Any, Optional 7 | 8 | from gjson import GJSONError, get 9 | 10 | 11 | def cli(argv: Optional[Sequence[str]] = None) -> int: 12 | """Command line entry point to run gjson as a CLI tool. 13 | 14 | Arguments: 15 | argv: a sequence of CLI arguments to parse. If not set they will be read from sys.argv. 16 | 17 | Returns: 18 | The CLI exit code to use. 19 | 20 | Raises: 21 | OSError: for system-related errors, including I/O failures. 22 | json.JSONDecodeError: when the input data is not a valid JSON. 23 | gjson.GJSONError: for any query-related error raised by gjson. 24 | 25 | """ 26 | parser = get_parser() 27 | args = parser.parse_args(argv) 28 | 29 | encapsulate = False 30 | if args.query.startswith('..'): 31 | args.query = args.query[2:] 32 | encapsulate = True 33 | 34 | # Use argparse.FileType here instead of putting it as type in the --file argument parsing, to allow to handle the 35 | # verbosity in case of error and make sure the file is always closed in case other arguments fail the validation. 36 | try: 37 | args.file = argparse.FileType(encoding='utf-8', errors='surrogateescape')(args.file) 38 | except (OSError, argparse.ArgumentTypeError) as ex: 39 | if args.verbose == 1: 40 | print(f'{ex.__class__.__name__}: {ex}', file=sys.stderr) 41 | elif args.verbose >= 2: # noqa: PLR2004 42 | raise 43 | 44 | return 1 45 | 46 | # Reconfigure __stdin__ and __stdout__ instead of stdin and stdout because the latters are TextIO and could not 47 | # have the reconfigure() method if re-assigned, while reconfigure() is part of TextIOWrapper. 48 | # See also: https://github.com/python/typeshed/pull/8171 49 | sys.__stdin__.reconfigure(errors='surrogateescape') 50 | sys.__stdout__.reconfigure(errors='surrogateescape') 51 | 52 | def _execute(line: str, file_obj: Optional[IO[Any]]) -> int: 53 | try: 54 | if encapsulate: 55 | if line: 56 | input_data = [json.loads(line, strict=False)] 57 | elif file_obj is not None: 58 | input_data = [] 59 | for input_line in file_obj: 60 | if input_line.strip(): 61 | input_data.append(json.loads(input_line, strict=False)) 62 | elif line: 63 | input_data = json.loads(line, strict=False) 64 | elif file_obj is not None: 65 | input_data = json.load(file_obj, strict=False) 66 | 67 | result = get(input_data, args.query, as_str=True) 68 | exit_code = 0 69 | except (json.JSONDecodeError, GJSONError) as ex: 70 | result = '' 71 | exit_code = 1 72 | if args.verbose == 1: 73 | print(f'{ex.__class__.__name__}: {ex}', file=sys.stderr) 74 | elif args.verbose >= 2: # noqa: PLR2004 75 | raise 76 | 77 | if result: 78 | print(result) 79 | 80 | return exit_code 81 | 82 | if args.lines: 83 | exit_code = 0 84 | for line in args.file: 85 | data = line.strip() 86 | if not data: 87 | continue 88 | ret = _execute(data, None) 89 | if ret > exit_code: 90 | exit_code = ret 91 | else: 92 | exit_code = _execute('', args.file) 93 | 94 | return exit_code 95 | 96 | 97 | def get_parser() -> argparse.ArgumentParser: 98 | """Get the CLI argument parser. 99 | 100 | Returns: 101 | the argument parser for the CLI. 102 | 103 | """ 104 | parser = argparse.ArgumentParser( 105 | prog='gjson', 106 | description=('A simple way to filter and extract data from JSON-like data structures. Python porting of the ' 107 | 'Go GJSON package.'), 108 | epilog='See also the full documentation available at https://volans-.github.io/gjson-py/index.html', 109 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 110 | ) 111 | parser.add_argument('-v', '--verbose', action='count', default=0, 112 | help=('Verbosity level. By default on error no output will be printed. Use -v to get the ' 113 | 'error message to stderr and -vv to get the full traceback.')) 114 | parser.add_argument('-l', '--lines', action='store_true', 115 | help='Treat the input as JSON Lines, parse each line and apply the query to each line.') 116 | # argparse.FileType is used later to parse this argument. 117 | parser.add_argument('file', default='-', nargs='?', 118 | help='Input JSON file to query. Reads from stdin if the argument is missing or set to "-".') 119 | parser.add_argument('query', help='A GJSON query to apply to the input data.') 120 | 121 | return parser 122 | -------------------------------------------------------------------------------- /gjson/_gjson.py: -------------------------------------------------------------------------------- 1 | """GJSON module.""" 2 | import json 3 | import operator 4 | import re 5 | from collections import Counter 6 | from collections.abc import Callable, Mapping, Sequence 7 | from dataclasses import dataclass 8 | from itertools import zip_longest 9 | from typing import Any, Optional, Type, TypeVar, Union 10 | 11 | from gjson._protocols import ModifierProtocol 12 | from gjson.exceptions import GJSONError, GJSONInvalidSyntaxError, GJSONParseError 13 | 14 | ESCAPE_CHARACTER = '\\' 15 | """str: The grammar escape character.""" 16 | DOT_DELIMITER = '.' 17 | """str: One of the available delimiters in the query grammar.""" 18 | PIPE_DELIMITER = '|' 19 | """str: One of the available delimiters in the query grammar.""" 20 | DELIMITERS = (DOT_DELIMITER, PIPE_DELIMITER) 21 | """tuple: All the available delimiters in the query grammar.""" 22 | MULTIPATHS_DELIMITERS = (*DELIMITERS, ']', '}', ',') 23 | """tuple: All the available delimiters in the query grammar.""" 24 | # Single character operators goes last to avoid mis-detection. 25 | QUERIES_OPERATORS = ('==~', '==', '!=', '<=', '>=', '!%', '=', '<', '>', '%') 26 | """tuple: The list of supported operators inside queries.""" 27 | MODIFIER_NAME_RESERVED_CHARS = ('"', ',', '.', '|', ':', '@', '{', '}', '[', ']', '(', ')') 28 | """tuple: The list of reserver characters not usable in a modifier's name.""" 29 | PARENTHESES_PAIRS = {'(': ')', ')': '(', '[': ']', ']': '[', '{': '}', '}': '{'} 30 | GJSONObjT = TypeVar('GJSONObjT', bound='GJSONObj') 31 | 32 | 33 | class NoResult: 34 | """A no result type to be passed around and be checked.""" 35 | 36 | 37 | @dataclass 38 | class BaseQueryPart: 39 | """Base dataclass class to represent a query part.""" 40 | 41 | start: int 42 | end: int 43 | part: str 44 | delimiter: str 45 | previous: Optional['BaseQueryPart'] 46 | is_last: bool 47 | 48 | def __str__(self) -> str: 49 | """Return the string representation of the part. 50 | 51 | Returns: 52 | The part property of the instance. 53 | 54 | """ 55 | return self.part 56 | 57 | 58 | class FieldQueryPart(BaseQueryPart): 59 | """Basic field path query part.""" 60 | 61 | 62 | class ArrayLenghtQueryPart(BaseQueryPart): 63 | """Hash query part, to get the size of an array.""" 64 | 65 | 66 | class ArrayIndexQueryPart(BaseQueryPart): 67 | """Integer query part to get an array index.""" 68 | 69 | @property 70 | def index(self) -> int: 71 | """Return the integer representation of the query part. 72 | 73 | Returns: 74 | the index as integer. 75 | 76 | """ 77 | return int(self.part) 78 | 79 | 80 | @dataclass 81 | class ArrayQueryQueryPart(BaseQueryPart): 82 | """Query part for array queries, with additional fields.""" 83 | 84 | field: str 85 | operator: str 86 | value: Union[str, 'ArrayQueryQueryPart'] 87 | first_only: bool 88 | 89 | 90 | @dataclass 91 | class ModifierQueryPart(BaseQueryPart): 92 | """Modifier query part.""" 93 | 94 | name: str 95 | options: dict[Any, Any] 96 | 97 | 98 | @dataclass 99 | class MultipathsItem: 100 | """Single multipaths query item.""" 101 | 102 | key: str 103 | values: list[BaseQueryPart] 104 | 105 | 106 | @dataclass 107 | class MultipathsObjectQueryPart(BaseQueryPart): 108 | """JSON object multipaths query part.""" 109 | 110 | parts: list[MultipathsItem] 111 | 112 | 113 | @dataclass 114 | class MultipathsArrayQueryPart(BaseQueryPart): 115 | """JSON object multipaths query part.""" 116 | 117 | parts: list[list[BaseQueryPart]] 118 | 119 | 120 | class LiteralQueryPart(BaseQueryPart): 121 | """Literal query part.""" 122 | 123 | 124 | class GJSONObj: 125 | """A low-level class to perform the GJSON query on a JSON-like object.""" 126 | 127 | def __init__(self, obj: Any, query: str, *, custom_modifiers: Optional[dict[str, ModifierProtocol]] = None): 128 | """Initialize the instance with the starting object and query. 129 | 130 | Examples: 131 | Client code should not need to instantiate this low-level class in normal circumastances:: 132 | 133 | >>> import gjson 134 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]} 135 | >>> gjson_obj = gjson.GJSONObj(data, 'items.#.size') 136 | 137 | Arguments: 138 | obj: the JSON-like object to query. 139 | query: the GJSON query to apply to the object. 140 | custom_modifiers: an optional dictionary with the custom modifiers to load. The dictionary keys are the 141 | names of the modifiers and the values are the callables with the modifier code that adhere to the 142 | :py:class:`gjson.ModifierProtocol` protocol. 143 | 144 | Raises: 145 | gjson.GJSONError: if any provided custom modifier overrides a built-in one or is not callable. 146 | 147 | """ 148 | self._obj = obj 149 | self._query = query 150 | if custom_modifiers is not None: 151 | if (intersection := self.builtin_modifiers().intersection(set(custom_modifiers.keys()))): 152 | raise GJSONError(f'Some provided custom_modifiers have the same name of built-in ones: {intersection}.') 153 | 154 | for name, modifier in custom_modifiers.items(): 155 | if not isinstance(modifier, ModifierProtocol): 156 | raise GJSONError(f'The given func "{modifier}" for the custom modifier @{name} does not adhere ' 157 | 'to the gjson.ModifierProtocol.') 158 | 159 | self._custom_modifiers = custom_modifiers if custom_modifiers else {} 160 | self._dump_params: dict[str, Any] = {'ensure_ascii': False} 161 | self._after_hash = False 162 | self._after_query_all = False 163 | 164 | @classmethod 165 | def builtin_modifiers(cls: Type[GJSONObjT]) -> set[str]: 166 | """Return the names of the built-in modifiers. 167 | 168 | Returns: 169 | the names of the built-in modifiers. 170 | 171 | """ 172 | prefix = '_apply_modifier_' 173 | return {modifier[len(prefix):] for modifier in dir(cls) if modifier.startswith(prefix)} 174 | 175 | def get(self) -> Any: 176 | """Perform the query and return the resulting object. 177 | 178 | Examples: 179 | Returns the resulting object:: 180 | 181 | >>> gjson_obj.get() 182 | [1, 2] 183 | 184 | Raises: 185 | gjson.GJSONError: on error. 186 | 187 | Returns: 188 | the resulting object. 189 | 190 | """ 191 | # Reset internal parameters 192 | self._dump_params = {'ensure_ascii': False} 193 | self._after_hash = False 194 | self._after_query_all = False 195 | 196 | if not self._query: 197 | raise GJSONError('Empty query.') 198 | 199 | obj = self._obj 200 | for part in self._parse(start=0, end=len(self._query) - 1): 201 | obj = self._parse_part(part, obj) 202 | 203 | return obj 204 | 205 | def __str__(self) -> str: 206 | """Return the JSON string representation of the object, based on the query parameters. 207 | 208 | Examples: 209 | Returns the resulting object as a JSON-encoded string:: 210 | 211 | >>> str(gjson_obj) 212 | '[1, 2]' 213 | 214 | Raises: 215 | gjson.GJSONError: on error. 216 | 217 | Returns: 218 | the JSON encoded string. 219 | 220 | """ 221 | obj = self.get() 222 | prefix = self._dump_params.pop('prefix', '') 223 | json_string = json.dumps(obj, **self._dump_params) 224 | 225 | if prefix: 226 | return '\n'.join(f'{prefix}{line}' for line in json_string.splitlines()) 227 | 228 | return json_string 229 | 230 | def _parse(self, *, start: int, end: int, max_end: int = 0, # noqa: PLR0912, PLR0913, PLR0915 231 | delimiter: str = '', in_multipaths: bool = False) -> list[BaseQueryPart]: 232 | """Parse the query. It will delegate to more specific parsers for each different feature. 233 | 234 | Arguments: 235 | start: the start position in the query. 236 | end: the end position in the query. 237 | max_end: an optional last position up to where a closing parentheses can be searched. 238 | delimiter: the optional delimiter before the query, if this is called on a multipaths. 239 | in_multipaths: whether the part to be parsed is inside a multipaths. 240 | 241 | Raises: 242 | gjson.GJSONParseError: on error. 243 | 244 | Returns: 245 | the resulting object. 246 | 247 | """ 248 | current: list[str] = [] 249 | current_start = -1 250 | parts: list[BaseQueryPart] = [] 251 | previous: Optional[BaseQueryPart] = None 252 | require_delimiter = False 253 | 254 | i = start 255 | while True: 256 | part: Optional[BaseQueryPart] = None 257 | # Get current and next character in the query 258 | if i == end: 259 | next_char = None 260 | elif i >= end: 261 | if parts and not current: 262 | parts[-1].is_last = True 263 | break 264 | else: 265 | next_char = self._query[i + 1] 266 | 267 | char = self._query[i] 268 | 269 | if char in DELIMITERS: 270 | if i == start: 271 | raise GJSONParseError('Invalid query starting with a path delimiter.', 272 | query=self._query, position=i) 273 | 274 | if next_char in DELIMITERS: 275 | raise GJSONParseError('Invalid query with two consecutive path delimiters.', 276 | query=self._query, position=i) 277 | if current: 278 | part = FieldQueryPart(start=current_start, end=i - 1, part=''.join(current), 279 | delimiter=delimiter, previous=previous, is_last=False) 280 | parts.append(part) 281 | previous = part 282 | current = [] 283 | current_start = -1 284 | 285 | delimiter = char 286 | require_delimiter = False 287 | if next_char is None: 288 | raise GJSONParseError('Delimiter at the end of the query.', query=self._query, position=i) 289 | 290 | i += 1 291 | continue 292 | 293 | if char == '@': 294 | part = self._parse_modifier_query_part( 295 | start=i, delimiter=delimiter, max_end=max_end, in_multipaths=in_multipaths) 296 | elif char == '#' and (next_char in DELIMITERS or next_char is None): 297 | part = ArrayLenghtQueryPart(start=i, end=i, part=char, delimiter=delimiter, is_last=next_char is None, 298 | previous=previous) 299 | elif char == '#' and next_char == '(': 300 | part = self._parse_array_query_query_part(start=i, delimiter=delimiter, max_end=max_end) 301 | elif re.match(r'[0-9]', char) and not current: 302 | part = self._parse_array_index_query_part(start=i, delimiter=delimiter, in_multipaths=in_multipaths) 303 | elif char == '{': 304 | part = self._parse_object_multipaths_query_part(start=i, delimiter=delimiter, max_end=max_end) 305 | require_delimiter = True 306 | elif char == '[': 307 | part = self._parse_array_multipaths_query_part(start=i, delimiter=delimiter, max_end=max_end) 308 | require_delimiter = True 309 | elif char == '!': 310 | part = self._parse_literal_query_part( 311 | start=i, delimiter=delimiter, max_end=max_end, in_multipaths=in_multipaths) 312 | elif in_multipaths and char == ',': 313 | i -= 1 314 | break 315 | elif in_multipaths and require_delimiter: 316 | raise GJSONInvalidSyntaxError('Missing separator after multipath.', query=self._query, position=i) 317 | 318 | if part: 319 | part.previous = previous 320 | parts.append(part) 321 | previous = part 322 | else: # Normal path, no special grammar 323 | if not current: 324 | current_start = i 325 | 326 | current.append(char) 327 | if char == ESCAPE_CHARACTER: 328 | i += 1 # Skip the escaped character 329 | if next_char is None: 330 | raise GJSONParseError('Escape character at the end of the query.', 331 | query=self._query, position=i) 332 | 333 | current.append(next_char) 334 | 335 | if part: 336 | i = part.end + 1 337 | else: 338 | i += 1 339 | 340 | if part is None and current: 341 | part = FieldQueryPart(start=current_start, end=i, part=''.join(current), 342 | delimiter=delimiter, previous=previous, is_last=True) 343 | parts.append(part) 344 | 345 | return parts 346 | 347 | @staticmethod 348 | def _is_sequence(obj: Any) -> bool: 349 | """Check if an object is a sequence but not a string or bytes object. 350 | 351 | Arguments: 352 | obj: the object to test. 353 | 354 | Returns: 355 | :py:data:`True` if the object is a sequence but not a string or bytes, :py:data:`False` otherwise. 356 | 357 | """ 358 | return isinstance(obj, Sequence) and not isinstance(obj, (str, bytes)) 359 | 360 | def _find_closing_parentheses(self, *, start: int, opening: str, suffix: str = '', # noqa: PLR0912 361 | max_end: int = 0) -> int: 362 | """Find the matching parentheses that closes the opening one looking for unbalance of the given character. 363 | 364 | Arguments: 365 | start: the index of the opening parentheses in the query. 366 | opening: the opening parentheses to look for imbalances. 367 | suffix: an optional suffix that can be present after the closing parentheses before reaching a delimiter or 368 | the end of the query. 369 | max_end: an optional last position up to where the parentheses can be found. 370 | 371 | Raises: 372 | gjson.GJSONParseError: if unable to find the closing parentheses or the parentheses are not balanced. 373 | 374 | Returns: 375 | the position of the closing parentheses if there is no suffix or the one of the last character of the 376 | suffix if present. 377 | 378 | """ 379 | closing = PARENTHESES_PAIRS[opening] 380 | opened = 0 381 | end = -1 382 | escaped = False 383 | in_string = False 384 | query = self._query[start:max_end + 1] if max_end else self._query[start:] 385 | 386 | for i, char in enumerate(query): 387 | if char == ESCAPE_CHARACTER: 388 | escaped = True 389 | continue 390 | 391 | if escaped: 392 | escaped = False 393 | continue 394 | 395 | if char == '"': 396 | in_string = not in_string 397 | continue 398 | 399 | if in_string: 400 | continue 401 | 402 | if char == opening: 403 | opened += 1 404 | elif char == closing: # noqa: SIM102 405 | if opened: 406 | opened -= 1 407 | if not opened: 408 | end = i 409 | break 410 | 411 | if opened or end < 0: 412 | raise GJSONParseError(f'Unbalanced parentheses `{opening}`, {opened} still opened.', 413 | query=self._query, position=start) 414 | 415 | if suffix and end + len(suffix) < len(query) and query[end + 1:end + len(suffix) + 1] == suffix: 416 | end += len(suffix) 417 | 418 | if end + 1 < len(query): 419 | delimiters = list(MULTIPATHS_DELIMITERS) if max_end else list(DELIMITERS) 420 | if opening == '(' and suffix == '#': # Nested queries 421 | delimiters.append(')') 422 | 423 | if (max_end and query[end + 1] not in delimiters) or (not max_end and query[end + 1] not in DELIMITERS): 424 | raise GJSONParseError('Expected delimiter or end of query after closing parenthesis.', 425 | query=self._query, position=start + end) 426 | 427 | return start + end 428 | 429 | def _parse_modifier_query_part(self, *, start: int, delimiter: str, max_end: int = 0, 430 | in_multipaths: bool = False) -> ModifierQueryPart: 431 | """Find the modifier end position in the query starting from a given point. 432 | 433 | Arguments: 434 | start: the index of the ``@`` symbol that starts a modifier in the query. 435 | delimiter: the delimiter before the modifier. 436 | max_end: an optional last position up to where the last character can be found. 437 | in_multipaths: whether the part to be parsed is inside a multipaths. 438 | 439 | Raises: 440 | gjson.GJSONParseError: on invalid modifier. 441 | 442 | Returns: 443 | the modifier query part object. 444 | 445 | """ 446 | end = start 447 | escaped = False 448 | delimiters = MULTIPATHS_DELIMITERS if in_multipaths else DELIMITERS 449 | options: dict[Any, Any] = {} 450 | query = self._query[start:max_end + 1] if max_end else self._query[start:] 451 | for i, char in enumerate(query): 452 | if char == ESCAPE_CHARACTER and not escaped: 453 | escaped = True 454 | continue 455 | 456 | if escaped: 457 | escaped = False 458 | continue 459 | 460 | if char == ':': 461 | name = self._query[start + 1:start + i] 462 | options_len, options = self._parse_modifier_options(start + i + 1) 463 | end = start + i + options_len 464 | break 465 | 466 | if char in delimiters: 467 | end = start + i - 1 468 | name = self._query[start + 1:start + i] 469 | break 470 | 471 | else: # End of query 472 | end = start + i 473 | name = self._query[start + 1:start + i + 1] 474 | 475 | name.replace(ESCAPE_CHARACTER, '') 476 | if not name: 477 | raise GJSONParseError('Got empty modifier name.', query=self._query, position=start) 478 | 479 | for char in MODIFIER_NAME_RESERVED_CHARS: 480 | if char in name: 481 | raise GJSONParseError(f'Invalid modifier name @{name}, the following characters are not allowed: ' 482 | f'{MODIFIER_NAME_RESERVED_CHARS}', query=self._query, position=start) 483 | 484 | return ModifierQueryPart(start=start, end=end, part=self._query[start:end + 1], delimiter=delimiter, 485 | name=name, options=options, is_last=False, previous=None) 486 | 487 | def _parse_modifier_options(self, start: int) -> tuple[int, dict[Any, Any]]: 488 | """Find the modifier options end position in the query starting from a given point. 489 | 490 | Arguments: 491 | start: the index of the ``:`` symbol that starts a modifier options. 492 | 493 | Raises: 494 | gjson.GJSONParseError: on invalid modifier options. 495 | 496 | Returns: 497 | the modifier options last character index in the query and the parsed options. 498 | 499 | """ 500 | if start >= len(self._query): 501 | raise GJSONParseError('Modifier with options separator `:` without any option.', 502 | query=self._query, position=start) 503 | 504 | if self._query[start] != '{': 505 | raise GJSONParseError('Expected JSON object `{...}` as modifier options.', 506 | query=self._query, position=start) 507 | 508 | query_parts = re.split(r'(? ArrayQueryQueryPart: 524 | """Parse an array query part starting from the given point. 525 | 526 | Arguments: 527 | start: the index of the ``#`` symbol that starts a ``#(...)`` or ``#(...)#`` query. 528 | delimiter: the delimiter before the modifier. 529 | max_end: an optional last position up to where the closing parentheses can be found. 530 | 531 | Raises: 532 | gjson.GJSONParseError: on invalid query. 533 | 534 | Returns: 535 | the array query part object. 536 | 537 | """ 538 | end = self._find_closing_parentheses(start=start, opening='(', suffix='#', max_end=max_end) 539 | part = self._query[start:end + 1] 540 | if part[-1] == '#': 541 | content_end = -2 542 | first_only = False 543 | else: 544 | content_end = -1 545 | first_only = True 546 | 547 | content = part[2: content_end] 548 | query_operator = '' 549 | key = '' 550 | value: Union[str, ArrayQueryQueryPart] = '' 551 | 552 | pattern = '|'.join(re.escape(op) for op in QUERIES_OPERATORS) 553 | match = re.search(fr'(? Optional[ArrayIndexQueryPart]: 578 | """Parse an array index query part. 579 | 580 | Arguments: 581 | start: the index of the start of the path in the query. 582 | delimiter: the delimiter before the query part. 583 | in_multipaths: whether the part to be parsed is inside a multipaths. 584 | 585 | Returns: 586 | the array index query object if the integer path is found, :py:const:`None` otherwise. 587 | 588 | """ 589 | subquery = self._query[start:] 590 | delimiters = MULTIPATHS_DELIMITERS if in_multipaths else DELIMITERS 591 | delimiters_match = '|'.join([re.escape(i) for i in delimiters]) 592 | match = re.search(fr'^([1-9][0-9]*|0)({delimiters_match}|$)', subquery) 593 | if not match: 594 | return None 595 | 596 | end = start + len(match.groups()[0]) - 1 597 | part = self._query[start:end + 1] 598 | 599 | return ArrayIndexQueryPart(start=start, end=end, part=part, delimiter=delimiter, is_last=False, previous=None) 600 | 601 | def _parse_object_multipaths_query_part( # noqa: PLR0912, PLR0915 602 | self, *, start: int, delimiter: str, max_end: int = 0) -> MultipathsObjectQueryPart: 603 | """Parse a multipaths object query part. 604 | 605 | Arguments: 606 | start: the index of the start of the path in the query. 607 | delimiter: the delimiter before the query part. 608 | max_end: an optional last position up to where the multipaths can extend. 609 | 610 | Returns: 611 | the multipaths object query part. 612 | 613 | Raises: 614 | gjson.GJSONParseError: on invalid query. 615 | 616 | """ 617 | end = self._find_closing_parentheses(start=start, opening='{', max_end=max_end) 618 | part = self._query[start:end + 1] 619 | parts = [] 620 | 621 | def _get_key(current_key: Optional[str], value: Optional[BaseQueryPart]) -> str: 622 | """Return the current key or the default value if not set. Allow for empty key as valid key. 623 | 624 | Arguments: 625 | current_key: the current key to evaluate. 626 | value: the current value from where to extract a key name if missing. 627 | 628 | """ 629 | if current_key is not None: 630 | return current_key 631 | 632 | if value and isinstance(value, (FieldQueryPart, ArrayIndexQueryPart, ModifierQueryPart)): 633 | return value.part 634 | 635 | return '_' 636 | 637 | new_item = True 638 | escaped = False 639 | key: Optional[str] = None 640 | key_start = 0 641 | value_start = 0 642 | skip_until = 0 643 | 644 | for i, char in enumerate(part[1:-1], start=1): 645 | if skip_until and i <= skip_until: 646 | if i == skip_until: 647 | skip_until = 0 648 | new_item = True 649 | 650 | continue 651 | 652 | if new_item: 653 | value_start = 0 654 | if char == '"': 655 | key_start = i 656 | new_item = False 657 | continue 658 | 659 | if char != ',': 660 | value_start = i 661 | new_item = False 662 | 663 | if key_start: 664 | if char == ESCAPE_CHARACTER and not escaped: 665 | escaped = True 666 | continue 667 | 668 | if escaped: 669 | escaped = False 670 | continue 671 | 672 | if char == '"': 673 | try: 674 | key = json.loads(part[key_start:i + 1], strict=False) 675 | except json.JSONDecodeError as ex: 676 | raise GJSONParseError(f'Failed to parse multipaths key {part[key_start:i + 1]}.', 677 | query=self._query, position=key_start) from ex 678 | key_start = 0 679 | continue 680 | 681 | if key is not None and not key_start and not value_start: 682 | if char == ':': 683 | value_start = i + 1 684 | continue 685 | 686 | raise GJSONParseError(f'Expected colon after multipaths item with key "{key}".', 687 | query=self._query, position=i) 688 | 689 | if value_start: 690 | try: 691 | values = self._parse( 692 | start=start + value_start, 693 | end=end - 1, 694 | max_end=max_end - 1 if max_end else end - 1, 695 | delimiter=delimiter, 696 | in_multipaths=True) 697 | except GJSONInvalidSyntaxError: 698 | raise 699 | except GJSONParseError: # In multipaths, paths that fails are silently suppressed 700 | values = [] 701 | 702 | if values: 703 | parts.append(MultipathsItem(key=_get_key(key, values[-1]), values=values)) 704 | skip_until = values[-1].end - start + 1 705 | else: 706 | skip_until = end - start 707 | 708 | new_item = True 709 | key = None 710 | key_start = 0 711 | value_start = 0 712 | continue 713 | 714 | return MultipathsObjectQueryPart(start=start, end=end, part=part, delimiter=delimiter, previous=None, 715 | is_last=False, parts=parts) 716 | 717 | def _parse_array_multipaths_query_part( 718 | self, *, start: int, delimiter: str, max_end: int = 0) -> MultipathsArrayQueryPart: 719 | """Parse a multipaths object query part. 720 | 721 | Arguments: 722 | start: the index of the start of the path in the query. 723 | delimiter: the delimiter before the query part. 724 | max_end: an optional last position up to where the multipaths can extend. 725 | 726 | Returns: 727 | the multipaths array query part. 728 | 729 | Raises: 730 | gjson.GJSONParseError: on invalid query. 731 | 732 | """ 733 | end = self._find_closing_parentheses(start=start, opening='[', max_end=max_end) 734 | part = self._query[start:end + 1] 735 | parts = [] 736 | skip_until = 0 737 | 738 | for i, _ in enumerate(part[1:-1], start=1): 739 | if skip_until and i <= skip_until: 740 | if i == skip_until: 741 | skip_until = 0 742 | 743 | continue 744 | 745 | try: 746 | values = self._parse( 747 | start=start + i, 748 | end=end - 1, 749 | max_end=max_end - 1 if max_end else end - 1, 750 | delimiter=delimiter, 751 | in_multipaths=True) 752 | except GJSONInvalidSyntaxError: 753 | raise 754 | except GJSONParseError: # In multipaths, paths that fails are silently suppressed 755 | values = [] 756 | 757 | if values: 758 | parts.append(values) 759 | skip_until = values[-1].end - start + 1 760 | else: 761 | skip_until = end - start 762 | 763 | return MultipathsArrayQueryPart(start=start, end=end, part=part, delimiter=delimiter, previous=None, 764 | is_last=False, parts=parts) 765 | 766 | def _parse_literal_query_part(self, *, start: int, delimiter: str, max_end: int = 0, 767 | in_multipaths: bool = False) -> LiteralQueryPart: 768 | """Parse a literal query part. 769 | 770 | Arguments: 771 | start: the index of the start of the path in the query. 772 | delimiter: the delimiter before the query part. 773 | max_end: an optional last position up to where the multipaths can extend. 774 | in_multipaths: whether the part to be parsed is inside a multipaths. 775 | 776 | Returns: 777 | the literal query part. 778 | 779 | Raises: 780 | gjson.GJSONParseError: on invalid query. 781 | 782 | """ 783 | end = -1 784 | begin = self._query[start + 1:start + 2] 785 | if begin in ('{', '['): 786 | end = self._find_closing_parentheses(start=start + 1, opening=begin, max_end=max_end) 787 | 788 | elif begin == '"': 789 | query = self._query[start + 2:max_end + 1] if max_end else self._query[start + 2:] 790 | match = re.search(r'(? Any: 824 | """Parse the given part of the full query. 825 | 826 | Arguments: 827 | part: the query part as already parsed. 828 | obj: the current object. 829 | in_multipaths: whether the part to be parsed is inside a multipaths. 830 | 831 | Raises: 832 | gjson.GJSONParseError: on invalid query. 833 | 834 | Returns: 835 | the result of the query. 836 | 837 | """ 838 | in_hash = False 839 | in_query_all = False 840 | ret: Any 841 | 842 | if isinstance(obj, NoResult): 843 | return obj 844 | 845 | if isinstance(part, ArrayLenghtQueryPart): 846 | in_hash = True 847 | if part.is_last: 848 | if part.delimiter == DOT_DELIMITER and (self._after_hash or self._after_query_all): 849 | ret = [] 850 | elif part.delimiter == PIPE_DELIMITER and isinstance(part.previous, ArrayLenghtQueryPart): 851 | raise GJSONParseError('The pipe delimiter cannot immediately follow the # element.', 852 | query=self._query, position=part.start) 853 | elif self._is_sequence(obj): 854 | ret = len(obj) 855 | else: 856 | raise GJSONParseError('Expected a sequence like object for query part # at the end of the query, ' 857 | f'got {type(obj)}.', query=self._query, position=part.start) 858 | else: 859 | ret = obj 860 | 861 | elif isinstance(part, ArrayQueryQueryPart): 862 | if not self._is_sequence(obj): 863 | raise GJSONParseError(f'Queries are supported only for sequence like objects, got {type(obj)}.', 864 | query=self._query, position=part.start) 865 | 866 | in_query_all = not part.first_only 867 | ret = self._parse_query(part, obj) 868 | 869 | elif isinstance(part, ModifierQueryPart): 870 | ret = self._apply_modifier(part, obj) 871 | 872 | elif isinstance(part, ArrayIndexQueryPart): 873 | if isinstance(obj, Mapping): # Integer object keys not supported by JSON 874 | if not in_multipaths and part.part not in obj: 875 | raise GJSONParseError(f'Mapping object does not have key `{part}`.', 876 | query=self._query, position=part.start) 877 | ret = obj.get(part.part, NoResult()) 878 | elif self._is_sequence(obj): 879 | if (self._after_hash or self._after_query_all) and part.delimiter == DOT_DELIMITER: 880 | # Skip non mapping items and items without the given key 881 | ret = [] 882 | for i in obj: 883 | if isinstance(i, Mapping) and part.part in i: 884 | ret.append(i[part.part]) 885 | elif self._is_sequence(i) and len(i): 886 | ret.append(i[int(part.part)]) 887 | elif (self._after_hash and part.delimiter == PIPE_DELIMITER 888 | and isinstance(part.previous, ArrayLenghtQueryPart)): 889 | raise GJSONParseError('Integer query part after a pipe delimiter on an sequence like object.', 890 | query=self._query, position=part.start) 891 | else: 892 | num = len(obj) 893 | if part.index >= num: 894 | raise GJSONParseError(f'Index `{part}` out of range for sequence object with {num} items in ' 895 | 'query.', query=self._query, position=part.start) 896 | ret = obj[part.index] 897 | else: 898 | raise GJSONParseError(f'Integer query part on unsupported object type {type(obj)}, expected a mapping ' 899 | 'or sequence like object.', query=self._query, position=part.start) 900 | 901 | elif isinstance(part, FieldQueryPart): 902 | if re.search(r'(? Any: 1042 | """Evaluate the return value of an inline query #(...) / #(...)# depending on first match or all matches. 1043 | 1044 | Arguments: 1045 | query: the query part. 1046 | obj: the current object. 1047 | 1048 | Raises: 1049 | gjson.GJSONParseError: if the query is for the first element and there are no matching items. 1050 | 1051 | Returns: 1052 | the result of the query. 1053 | 1054 | """ 1055 | if query.first_only: 1056 | if obj: 1057 | return obj[0] 1058 | 1059 | raise GJSONParseError('Query for first element does not match anything.', 1060 | query=self._query, position=query.start) 1061 | 1062 | return obj 1063 | 1064 | def _parse_query(self, query: ArrayQueryQueryPart, obj: Any) -> Any: # noqa: PLR0912, PLR0915 1065 | """Parse an inline query #(...) / #(...)#. 1066 | 1067 | Arguments: 1068 | query: the query part. 1069 | obj: the current object. 1070 | 1071 | Raises: 1072 | gjson.GJSONParseError: on invalid query. 1073 | 1074 | Returns: 1075 | the result of the query. 1076 | 1077 | """ 1078 | if isinstance(query.value, ArrayQueryQueryPart): 1079 | ret = [] 1080 | for i in obj: 1081 | nested_obj = None 1082 | if query.field: 1083 | if isinstance(i, Mapping) and query.field in i: 1084 | nested_obj = i[query.field] 1085 | elif self._is_sequence(i): 1086 | nested_obj = i 1087 | 1088 | if nested_obj is not None and self._parse_query(query.value, nested_obj): 1089 | ret.append(i) 1090 | 1091 | return self._evaluate_query_return_value(query, ret) 1092 | 1093 | if not query.operator: 1094 | return self._evaluate_query_return_value(query, [i for i in obj if query.field in i]) 1095 | 1096 | key = query.field.replace('\\', '') 1097 | try: 1098 | value = json.loads(query.value, strict=False) 1099 | except json.JSONDecodeError as ex: 1100 | position = query.start + len(query.field) + len(query.operator) 1101 | raise GJSONParseError(f'Invalid value `{query.value}` for the query key `{key}`.', 1102 | query=self._query, position=position) from ex 1103 | 1104 | if not key and query.first_only and obj and isinstance(obj[0], Mapping): 1105 | raise GJSONParseError('Query on mapping like objects require a key before the operator.', 1106 | query=self._query, position=query.start) 1107 | 1108 | oper: Callable[[Any, Any], bool] 1109 | if query.operator == '==~': 1110 | if value not in (True, False): 1111 | if query.first_only: 1112 | raise GJSONParseError(f'Queries ==~ operator requires a boolean value, got {type(value)} instead: ' 1113 | f'`{value}`.', query=self._query, position=query.start + len(query.field)) 1114 | 1115 | return [] 1116 | 1117 | def truthy_op(obj_a: Any, obj_b: bool) -> bool: # noqa: FBT001 1118 | truthy = operator.truth(obj_a) 1119 | if obj_b: 1120 | return truthy 1121 | return not truthy 1122 | 1123 | oper = truthy_op 1124 | elif query.operator in ('==', '='): 1125 | oper = operator.eq 1126 | elif query.operator == '!=': 1127 | oper = operator.ne 1128 | elif query.operator == '<': 1129 | oper = operator.lt 1130 | elif query.operator == '<=': 1131 | oper = operator.le 1132 | elif query.operator == '>': 1133 | oper = operator.gt 1134 | elif query.operator == '>=': 1135 | oper = operator.ge 1136 | elif query.operator in ('%', '!%'): 1137 | value = str(value).replace('*', '.*').replace('?', '.') 1138 | value = f'^{value}$' 1139 | if query.operator == '%': 1140 | def match_op(obj_a: Any, obj_b: Any) -> bool: 1141 | if not isinstance(obj_a, str): 1142 | return False 1143 | return re.match(obj_b, obj_a) is not None 1144 | oper = match_op 1145 | else: 1146 | def not_match_op(obj_a: Any, obj_b: Any) -> bool: 1147 | if not isinstance(obj_a, str): 1148 | return False 1149 | return re.match(obj_b, obj_a) is None 1150 | oper = not_match_op 1151 | 1152 | try: 1153 | if key: 1154 | if query.operator == '==~': # Consider missing keys as falsy according to GJSON docs. 1155 | ret = [i for i in obj if oper(i.get(key), value)] 1156 | else: 1157 | ret = [i for i in obj if key in i and oper(i[key], value)] 1158 | else: # Query on an array of non-objects, match them directly 1159 | ret = [i for i in obj if oper(i, value)] 1160 | except TypeError: 1161 | ret = [] 1162 | 1163 | return self._evaluate_query_return_value(query, ret) 1164 | 1165 | def _apply_modifier(self, modifier: ModifierQueryPart, obj: Any) -> Any: 1166 | """Apply a modifier. 1167 | 1168 | Arguments: 1169 | modifier: the modifier query part to parse. 1170 | obj: the current object before applying the modifier. 1171 | 1172 | Raises: 1173 | gjson.GJSONError: when the modifier raises an exception. 1174 | gjson.GJSONParseError: on unknown modifier. 1175 | 1176 | Returns: 1177 | the object modifier according to the modifier. 1178 | 1179 | """ 1180 | try: 1181 | modifier_func = getattr(self, f'_apply_modifier_{modifier.name}') 1182 | except AttributeError: 1183 | modifier_func = self._custom_modifiers.get(modifier.name) 1184 | if modifier_func is None: 1185 | raise GJSONParseError(f'Unknown modifier @{modifier.name}.', 1186 | query=self._query, position=modifier.start) from None 1187 | 1188 | try: 1189 | return modifier_func(modifier.options, obj, last=modifier.is_last) 1190 | except GJSONError: 1191 | raise 1192 | except Exception as ex: 1193 | raise GJSONError(f'Modifier @{modifier.name} raised an exception.') from ex 1194 | 1195 | def _apply_modifier_reverse(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1196 | """Apply the @reverse modifier. 1197 | 1198 | Arguments: 1199 | options: the eventual options for the modifier, currently unused. 1200 | obj: the current object to reverse. 1201 | last: whether this is the final part of the query. 1202 | 1203 | Returns: 1204 | the reversed object. If the object cannot be reversed is returned untouched. 1205 | 1206 | """ 1207 | del last # unused argument 1208 | if isinstance(obj, Mapping): 1209 | return {k: obj[k] for k in reversed(obj.keys())} 1210 | if self._is_sequence(obj): 1211 | return obj[::-1] 1212 | 1213 | return obj 1214 | 1215 | def _apply_modifier_keys(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1216 | """Apply the @keys modifier. 1217 | 1218 | Arguments: 1219 | options: the eventual options for the modifier, currently unused. 1220 | obj: the current object to get the keys from. 1221 | last: whether this is the final part of the query. 1222 | 1223 | Raises: 1224 | gjson.GJSONError: if the current object does not have a keys() method. 1225 | 1226 | Returns: 1227 | the current object keys as list. 1228 | 1229 | """ 1230 | del last # unused argument 1231 | try: 1232 | return list(obj.keys()) 1233 | except AttributeError as ex: 1234 | raise GJSONError('The current object does not have a keys() method.') from ex 1235 | 1236 | def _apply_modifier_values(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1237 | """Apply the @values modifier. 1238 | 1239 | Arguments: 1240 | options: the eventual options for the modifier, currently unused. 1241 | obj: the current object to get the values from. 1242 | last: whether this is the final part of the query. 1243 | 1244 | Raises: 1245 | gjson.GJSONError: if the current object does not have a values() method. 1246 | 1247 | Returns: 1248 | the current object values as list. 1249 | 1250 | """ 1251 | del last # unused argument 1252 | try: 1253 | return list(obj.values()) 1254 | except AttributeError as ex: 1255 | raise GJSONError('The current object does not have a values() method.') from ex 1256 | 1257 | def _apply_modifier_ugly(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1258 | """Apply the @ugly modifier to condense the output. 1259 | 1260 | Arguments: 1261 | options: the eventual options for the modifier, currently unused. 1262 | obj: the current object to uglyfy. 1263 | last: whether this is the final part of the query. 1264 | 1265 | Returns: 1266 | the current object, unmodified. 1267 | 1268 | """ 1269 | del last # unused argument 1270 | self._dump_params['separators'] = (',', ':') 1271 | self._dump_params['indent'] = None 1272 | return obj 1273 | 1274 | def _apply_modifier_pretty(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1275 | """Apply the @pretty modifier to pretty-print the output. 1276 | 1277 | Arguments: 1278 | options: the eventual options for the modifier. 1279 | obj: the current object to prettyfy. 1280 | last: whether this is the final part of the query. 1281 | 1282 | Returns: 1283 | the current object, unmodified. 1284 | 1285 | """ 1286 | del last # unused argument 1287 | self._dump_params['indent'] = options.get('indent', 2) 1288 | self._dump_params['sort_keys'] = options.get('sortKeys', False) 1289 | self._dump_params['prefix'] = options.get('prefix', '') 1290 | return obj 1291 | 1292 | def _apply_modifier_ascii(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1293 | """Apply the @ascii modifier to have all non-ASCII characters escaped when dumping the object. 1294 | 1295 | Arguments: 1296 | options: the eventual options for the modifier, currently unused. 1297 | obj: the current object to sort. 1298 | last: whether this is the final part of the query. 1299 | 1300 | Returns: 1301 | the current object, unmodified. 1302 | 1303 | """ 1304 | del last # unused argument 1305 | self._dump_params['ensure_ascii'] = True 1306 | return obj 1307 | 1308 | def _apply_modifier_sort(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1309 | """Apply the @sort modifier, sorts mapping and sequences. 1310 | 1311 | Arguments: 1312 | options: the eventual options for the modifier, currently unused. 1313 | obj: the current object to sort. 1314 | last: whether this is the final part of the query. 1315 | 1316 | Raises: 1317 | gjson.GJSONError: if the current object is not sortable. 1318 | 1319 | Returns: 1320 | the sorted object. 1321 | 1322 | """ 1323 | del last # unused argument 1324 | if isinstance(obj, Mapping): 1325 | return {k: obj[k] for k in sorted(obj.keys())} 1326 | if self._is_sequence(obj): 1327 | return sorted(obj) 1328 | 1329 | raise GJSONError(f'@sort modifier not supported for object of type {type(obj)}. ' 1330 | 'Expected a mapping or sequence like object.') 1331 | 1332 | def _apply_modifier_valid(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1333 | """Apply the @valid modifier, checking that the current object can be converted to JSON. 1334 | 1335 | Arguments: 1336 | options: the eventual options for the modifier, currently unused. 1337 | obj: the current element to validate. 1338 | last: whether this is the final part of the query. 1339 | 1340 | Raises: 1341 | gjson.GJSONError: if the current object cannot be converted to JSON. 1342 | 1343 | Returns: 1344 | the current object, unmodified. 1345 | 1346 | """ 1347 | del last # unused argument 1348 | try: 1349 | json.dumps(obj, **self._dump_params) 1350 | except Exception as ex: 1351 | raise GJSONError('The current object cannot be converted to JSON.') from ex 1352 | 1353 | return obj 1354 | 1355 | def _apply_modifier_this(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1356 | """Apply the @this modifier, that returns the current object. 1357 | 1358 | Arguments: 1359 | options: the eventual options for the modifier, currently unused. 1360 | obj: the current element to return. 1361 | last: whether this is the final part of the query. 1362 | 1363 | Returns: 1364 | the current object, unmodified. 1365 | 1366 | """ 1367 | del last # unused argument 1368 | return obj 1369 | 1370 | def _apply_modifier_fromstr(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1371 | """Apply the @fromstr modifier, converting a string to JSON, if valid. 1372 | 1373 | Arguments: 1374 | options: the eventual options for the modifier, currently unused. 1375 | obj: the current element from where to extract the JSON. 1376 | last: whether this is the final part of the query. 1377 | 1378 | Raises: 1379 | gjson.GJSONError: if the current object cannot be converted to JSON. 1380 | 1381 | Returns: 1382 | the parsed JSON. 1383 | 1384 | """ 1385 | del last # unused argument 1386 | if not isinstance(obj, (str, bytes)): 1387 | raise GJSONError(f'Modifier @fromstr got object of type {type(obj)} as input, expected string or bytes.') 1388 | 1389 | try: 1390 | return json.loads(obj, strict=False) 1391 | except Exception as ex: 1392 | raise GJSONError('The current @fromstr input object cannot be converted to JSON.') from ex 1393 | 1394 | def _apply_modifier_tostr(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1395 | """Apply the @tostr modifier, converting the current object to a JSON-encoded string, if valid. 1396 | 1397 | Arguments: 1398 | options: the eventual options for the modifier, currently unused. 1399 | obj: the current element from where to extract the JSON. 1400 | last: whether this is the final part of the query. 1401 | 1402 | Raises: 1403 | gjson.GJSONError: if the current object cannot be converted to a JSON-encoded string. 1404 | 1405 | Returns: 1406 | the JSON-encoded string. 1407 | 1408 | """ 1409 | del last # unused argument 1410 | try: 1411 | return json.dumps(obj, ensure_ascii=False) 1412 | except Exception as ex: 1413 | raise GJSONError('The current object cannot be converted to a JSON-encoded string for @tostr.') from ex 1414 | 1415 | def _apply_modifier_group(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1416 | """Apply the @group modifier, that groups a dictionary of lists in a list of dictionaries. 1417 | 1418 | Example input:: 1419 | 1420 | {"id": ["123", "456", "789"], "val": [2, 1]} 1421 | 1422 | Example output:: 1423 | 1424 | [{"id": "123", "val": 2}, {"id": "456", "val": 1}, {"id": "789"}] 1425 | 1426 | Arguments: 1427 | options: the eventual options for the modifier, currently unused. 1428 | obj: the current element to group. 1429 | last: whether this is the final part of the query. 1430 | 1431 | Raises: 1432 | gjson.GJSONError: if the current object is not a dictionary. 1433 | 1434 | Returns: 1435 | a list with the grouped objects or an empty list if the input has no lists as values. 1436 | 1437 | """ 1438 | del last # unused argument 1439 | if not isinstance(obj, Mapping): 1440 | raise GJSONError(f'Modifier @group got object of type {type(obj)} as input, expected dictionary.') 1441 | 1442 | # Skip all values that aren't lists: 1443 | obj = {k: v for k, v in obj.items() if self._is_sequence(v)} 1444 | # Fill missing values with NoResult to remove them afterwards 1445 | obj = [dict(zip_longest(obj.keys(), values)) for values in zip_longest(*obj.values(), fillvalue=NoResult())] 1446 | # Skip keys with value NoResult in each dictionary 1447 | return [{k: v for k, v in i.items() if not isinstance(v, NoResult)} for i in obj] 1448 | 1449 | def _apply_modifier_join(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1450 | """Apply the @join modifier, that joins a list of dictionaries into a single dictionary. 1451 | 1452 | Items in the sequence that are not dictionaries are skipped. 1453 | Differently from GJSON there is no support for duplicated keys as the can't exist in Python dictionaries. 1454 | Hence this modifier doesn't accept any option. 1455 | 1456 | Example input:: 1457 | 1458 | [{"first": "Tom", "age": 37}, {"age": 41}] 1459 | 1460 | Example output:: 1461 | 1462 | {"first": "Tom", "age":41} 1463 | 1464 | Arguments: 1465 | options: the eventual options for the modifier, currently unused. 1466 | obj: the current element to join. 1467 | last: whether this is the final part of the query. 1468 | 1469 | Returns: 1470 | the object untouched if the object is not a sequence, a dictionary with joined objects otherwise. 1471 | 1472 | """ 1473 | del last # unused argument 1474 | if not self._is_sequence(obj): 1475 | return obj 1476 | 1477 | ret: dict[Any, Any] = {} 1478 | for item in obj: 1479 | if isinstance(item, Mapping): 1480 | ret.update(item) 1481 | 1482 | return ret 1483 | 1484 | def _apply_modifier_top_n(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1485 | """Apply the @top_n modifier to find the most common values of a given field. 1486 | 1487 | Arguments: 1488 | options: the eventual modifier options. If not specified all items are returned. If specified it must 1489 | contain a 'n' key with the number of top N to return. 1490 | obj: the current object to extract the N most common items. 1491 | last: whether this is the final part of the query. 1492 | 1493 | Raises: 1494 | gjson.GJSONError: if the current object is not a sequence. 1495 | 1496 | Returns: 1497 | dict: a dictionary of unique items as keys and the count as value. 1498 | 1499 | """ 1500 | del last # unused argument 1501 | if not self._is_sequence(obj): 1502 | raise GJSONError(f'@top_n modifier not supported for object of type {type(obj)}. ' 1503 | 'Expected a sequence like object.') 1504 | 1505 | return dict(Counter(obj).most_common(options.get('n'))) 1506 | 1507 | def _apply_modifier_sum_n(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1508 | """Apply the @sum_n modifier that groups the values of a given key while summing the values of another key. 1509 | 1510 | The key used to sum must have numeric values. 1511 | 1512 | Arguments: 1513 | options: the modifier options. It must contain a 'group' key with the name of the field to use to group the 1514 | items as value and a 'sum' key with the name of the field to use to sum the values for each unique 1515 | grouped identifier. If a 'n' key is also provided, only the top N results are returned. If not 1516 | specified all items are returned. 1517 | obj: the current object to group and sum the top N values. 1518 | last: whether this is the final part of the query. 1519 | 1520 | Raises: 1521 | gjson.GJSONError: if the current object is not a sequence. 1522 | 1523 | Returns: 1524 | dict: a dictionary of unique items as keys and the sum as value. 1525 | 1526 | """ 1527 | del last # unused argument 1528 | if not self._is_sequence(obj): 1529 | raise GJSONError(f'@sum_n modifier not supported for object of type {type(obj)}. ' 1530 | 'Expected a sequence like object.') 1531 | 1532 | results: Counter[Any] = Counter() 1533 | for item in obj: 1534 | results[item[options['group']]] += item[options['sum']] 1535 | return dict(results.most_common(options.get('n'))) 1536 | 1537 | def _apply_modifier_flatten(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any: 1538 | """Apply the @flatten modifier. 1539 | 1540 | Arguments: 1541 | options: the eventual modifier options. 1542 | obj: the current object to flatten. 1543 | last: whether this is the final part of the query. 1544 | 1545 | Returns: 1546 | the modified object. 1547 | 1548 | """ 1549 | del last # unused argument 1550 | if not self._is_sequence(obj): 1551 | return obj 1552 | 1553 | return list(self._flatten_sequence(obj, deep=options.get('deep', False))) 1554 | 1555 | def _flatten_sequence(self, obj: Any, *, deep: bool = False) -> Any: 1556 | """Flatten nested sequences in the given object. 1557 | 1558 | Arguments: 1559 | obj: the current object to flatten 1560 | deep: if :py:data:`True` recursively flatten nested sequences. By default only the first level is 1561 | processed. 1562 | 1563 | Returns: 1564 | the flattened object if it was a flattable sequence, the given object itself otherwise. 1565 | 1566 | """ 1567 | for elem in obj: 1568 | if self._is_sequence(elem): 1569 | if deep: 1570 | yield from self._flatten_sequence(elem, deep=deep) 1571 | else: 1572 | yield from elem 1573 | else: 1574 | yield elem 1575 | -------------------------------------------------------------------------------- /gjson/_protocols.py: -------------------------------------------------------------------------------- 1 | """Typing protocols used by the gjson package.""" 2 | from typing import Any, Protocol, runtime_checkable 3 | 4 | 5 | @runtime_checkable 6 | class ModifierProtocol(Protocol): 7 | """Callback protocol for the custom modifiers.""" 8 | 9 | def __call__(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any: 10 | """To register a custom modifier a callable that adhere to this protocol must be provided. 11 | 12 | Examples: 13 | Register a custom modifier that sums all the numbers in a list: 14 | 15 | >>> import gjson 16 | >>> data = [1, 2, 3, 4, 5] 17 | >>> def custom_sum(options, obj, *, last): 18 | ... # insert sanity checks code here 19 | ... return sum(obj) 20 | ... 21 | >>> gjson_obj = gjson.GJSON(data) 22 | >>> gjson_obj.register_modifier('sum', custom_sum) 23 | >>> gjson_obj.get('@sum') 24 | 15 25 | 26 | Arguments: 27 | options: a dictionary of options. If no options are present in the query the callable will be called with 28 | an empty dictionary as options. The modifier can supports any number of options, or none. 29 | obj: the current object already modifier by any previous parts of the query. 30 | last: :py:data:`True` if the modifier is the last element in the query or :py:data:`False` otherwise. 31 | 32 | Raises: 33 | Exception: any exception that might be raised by the callable is catched by gjson and re-raised as a 34 | :py:class:`gjson.GJSONError` exception to ensure that the normal gjson behaviour is respected according 35 | to the selected verbosity (CLI) or ``quiet`` parameter (Python library). 36 | 37 | Returns: 38 | the resulting object after applying the modifier. 39 | 40 | """ 41 | -------------------------------------------------------------------------------- /gjson/exceptions.py: -------------------------------------------------------------------------------- 1 | """gjson custom exceptions module.""" 2 | from typing import Any 3 | 4 | 5 | class GJSONError(Exception): 6 | """Raised by the gjson module on error while performing queries or converting to JSON.""" 7 | 8 | 9 | class GJSONParseError(GJSONError): 10 | """Raised when there is an error parsing the query string, with nicer representation of the error.""" 11 | 12 | def __init__(self, *args: Any, query: str, position: int): 13 | """Initialize the exception with the additional data of the query part. 14 | 15 | Arguments: 16 | *args: all positional arguments like any regular exception. 17 | query: the full query that generated the parse error. 18 | position: the position in the query string where the parse error occurred. 19 | 20 | """ 21 | super().__init__(*args) 22 | self.query = query 23 | self.position = position 24 | 25 | def __str__(self) -> str: 26 | """Return a custom representation of the error. 27 | 28 | Returns: 29 | the whole query string with a clear indication on where the error occurred. 30 | 31 | """ 32 | default = super().__str__() 33 | line = '-' * (self.position + 7) # 7 is for the lenght of 'Query: ' 34 | return f'{default}\nQuery: {self.query}\n{line}^' 35 | 36 | 37 | class GJSONInvalidSyntaxError(GJSONParseError): 38 | """Raised when there is a query with an invalid syntax.""" 39 | -------------------------------------------------------------------------------- /gjson/py.typed: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/volans-/gjson-py/1cdc7d0633cdd7d21bd61c8d9cf1eab5d3b311fe/gjson/py.typed -------------------------------------------------------------------------------- /prospector.yaml: -------------------------------------------------------------------------------- 1 | strictness: high 2 | inherits: 3 | - strictness_high 4 | 5 | doc-warnings: true 6 | member-warnings: true 7 | test-warnings: true 8 | 9 | autodetect: false 10 | output-format: grouped 11 | 12 | ignore-paths: 13 | - doc/source/conf.py 14 | 15 | pyroma: 16 | run: true 17 | 18 | vulture: 19 | run: true 20 | -------------------------------------------------------------------------------- /ruff.toml: -------------------------------------------------------------------------------- 1 | target-version = 'py39' 2 | line-length = 120 3 | select = ['F', 'E', 'W', 'C90', 'B', 'I', 'N', 'D', 'YTT', 'ANN', 'S', 'FBT', 'A', 'C4', 'DTZ', 'T10', 'EXE', 'ISC', 4 | 'G', 'INP', 'T20', 'PT', 'Q', 'RSE', 'RET', 'SLF', 'SIM', 'TID', 'TCH', 'ARG', 'PTH', 'ERA', 'PL', 'TRY', 5 | 'RUF'] 6 | ignore = [ 7 | 'D203', 8 | 'D213', 9 | 'D406', 10 | 'D407', 11 | 'ANN101', 12 | 'ANN401', 13 | 'TRY003', 14 | 'C901', 15 | 'N999', 16 | ] 17 | 18 | [per-file-ignores] 19 | 'doc/source/conf.py' = ['A001', 'ERA001', 'INP001'] 20 | 'gjson/_cli.py' = ['T201'] 21 | 'setup.py' = ['EXE001'] 22 | 23 | [flake8-annotations] 24 | mypy-init-return = true 25 | 26 | [flake8-bandit] 27 | check-typed-exception = true 28 | 29 | [flake8-pytest-style] 30 | parametrize-values-type = 'tuple' 31 | raises-extend-require-match-for = [ 32 | 'GJSONError', 33 | 'GJSONParseError', 34 | 'GJSONInvalidSyntaxError', 35 | ] 36 | 37 | [flake8-quotes] 38 | inline-quotes = 'single' 39 | 40 | [pydocstyle] 41 | convention = 'google' 42 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [aliases] 2 | test = pytest 3 | 4 | [build_sphinx] 5 | project = gjson-py 6 | source-dir = doc/source 7 | build-dir = doc/build 8 | 9 | [mypy] 10 | disallow_untyped_calls = True 11 | disallow_untyped_defs = True 12 | disallow_incomplete_defs = True 13 | disallow_untyped_decorators = True 14 | no_implicit_optional = True 15 | warn_redundant_casts = True 16 | warn_unused_ignores = True 17 | warn_return_any = True 18 | warn_unreachable = True 19 | strict_equality = True 20 | strict = True 21 | show_error_context = True 22 | show_column_numbers = True 23 | show_error_codes = True 24 | pretty = True 25 | warn_incomplete_stub = True 26 | warn_unused_configs = True 27 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Package configuration.""" 3 | from pathlib import Path 4 | 5 | from setuptools import find_packages, setup 6 | 7 | # Extra dependencies 8 | extras_require = { 9 | # Test dependencies 10 | 'tests': [ 11 | 'bandit', 12 | 'flake8', 13 | 'flake8-import-order', 14 | 'mypy', 15 | 'pytest-cov', 16 | 'pytest-xdist', 17 | 'pytest', 18 | 'ruff', 19 | 'sphinx_rtd_theme>=1.0', 20 | 'sphinx-argparse', 21 | 'sphinx-autodoc-typehints', 22 | 'Sphinx', 23 | 'types-pkg_resources', 24 | ], 25 | 'prospector': [ 26 | 'prospector[with_everything]', 27 | 'pytest', 28 | ], 29 | } 30 | 31 | setup_requires = [ 32 | 'pytest-runner', 33 | 'setuptools_scm', 34 | ] 35 | 36 | setup( 37 | author='Riccardo Coccioli', 38 | author_email='volans-@users.noreply.github.com', 39 | classifiers=[ 40 | 'Development Status :: 5 - Production/Stable', 41 | 'Environment :: Console', 42 | 'Intended Audience :: Developers', 43 | 'Intended Audience :: End Users/Desktop', 44 | 'Intended Audience :: Information Technology', 45 | 'Intended Audience :: Science/Research', 46 | 'Intended Audience :: System Administrators', 47 | 'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)', 48 | 'Operating System :: OS Independent', 49 | 'Programming Language :: Python :: 3.9', 50 | 'Programming Language :: Python :: 3.10', 51 | 'Programming Language :: Python :: 3.11', 52 | 'Programming Language :: Python :: 3.12', 53 | 'Programming Language :: Python :: 3 :: Only', 54 | 'Topic :: Software Development :: Libraries :: Python Modules', 55 | 'Topic :: Text Processing', 56 | 'Topic :: Utilities', 57 | 'Typing :: Typed', 58 | ], 59 | description=('gjson-py is a Python package that provides a simple way to filter and extract data from JSON-like ' 60 | 'objects or JSON files, using the GJSON syntax.'), 61 | entry_points={ 62 | 'console_scripts': [ 63 | 'gjson = gjson._cli:cli', 64 | ], 65 | }, 66 | extras_require=extras_require, 67 | install_requires=[], 68 | keywords=['gjson', 'json'], 69 | license='GPLv3+', 70 | long_description=Path('README.rst').read_text(), 71 | long_description_content_type='text/x-rst', 72 | name='gjson', 73 | package_data={'gjson': ['py.typed']}, 74 | packages=find_packages(), 75 | platforms=['GNU/Linux', 'BSD', 'MacOSX'], 76 | python_requires='>=3.9', 77 | setup_requires=setup_requires, 78 | url='https://github.com/volans-/gjson-py', 79 | use_scm_version=True, 80 | zip_safe=False, 81 | ) 82 | -------------------------------------------------------------------------------- /tests/ruff.toml: -------------------------------------------------------------------------------- 1 | extend = "../ruff.toml" 2 | ignore = [ 3 | 'ANN', 4 | 'S101', 5 | 'PLR2004', 6 | ] 7 | 8 | -------------------------------------------------------------------------------- /tests/unit/__init__.py: -------------------------------------------------------------------------------- 1 | """gjson unit tests.""" 2 | -------------------------------------------------------------------------------- /tests/unit/test__cli.py: -------------------------------------------------------------------------------- 1 | """CLI test module.""" 2 | import argparse 3 | import io 4 | import json 5 | 6 | import pytest 7 | from gjson._cli import cli 8 | from gjson.exceptions import GJSONParseError 9 | 10 | from .test_init import INPUT_JSON, INPUT_LINES, INPUT_LINES_WITH_ERRORS 11 | 12 | 13 | def test_cli_help(capsys): 14 | """It should read the data from stdin and query it.""" 15 | with pytest.raises(SystemExit) as exc_info: 16 | cli(['--help']) 17 | 18 | assert exc_info.value.args[0] == 0 19 | captured = capsys.readouterr() 20 | assert 'usage: gjson' in captured.out 21 | assert 'See also the full documentation available at' in captured.out 22 | assert not captured.err 23 | 24 | 25 | def test_cli_stdin(monkeypatch, capsys): 26 | """It should read the data from stdin and query it.""" 27 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON)) 28 | ret = cli(['-', 'name.first']) 29 | assert ret == 0 30 | captured = capsys.readouterr() 31 | assert captured.out == '"Tom"\n' 32 | assert not captured.err 33 | 34 | 35 | def test_cli_file(tmp_path, capsys): 36 | """It should read the data from the provided file and query it.""" 37 | data_file = tmp_path / 'input.json' 38 | data_file.write_text(INPUT_JSON) 39 | ret = cli([str(data_file), 'name.first']) 40 | assert ret == 0 41 | captured = capsys.readouterr() 42 | assert captured.out == '"Tom"\n' 43 | assert not captured.err 44 | 45 | 46 | @pytest.mark.parametrize('query', ( 47 | 'a\tkey', 48 | '..0.a\tkey', 49 | )) 50 | def test_cli_file_with_control_chars(query, tmp_path, capsys): 51 | """It should read the data from the provided file and query it.""" 52 | data_file = tmp_path / 'input.json' 53 | data_file.write_text('{"a\tkey": "a\tvalue"}') 54 | ret = cli(['-vvv', str(data_file), query]) 55 | assert ret == 0 56 | captured = capsys.readouterr() 57 | assert captured.out == '"a\\tvalue"\n' 58 | assert not captured.err 59 | 60 | 61 | @pytest.mark.parametrize('query', ( 62 | 'a\tkey', 63 | '..0.a\tkey', 64 | )) 65 | @pytest.mark.parametrize('lines', (True, False)) 66 | def test_cli_stdin_with_control_chars(query, lines, monkeypatch, capsys): 67 | """It should read the data from stdin and query it.""" 68 | monkeypatch.setattr('sys.stdin', io.StringIO('{"a\tkey": "a\tvalue"}')) 69 | params = ['-vvv', '-', query] 70 | if lines: 71 | params.insert(0, '--lines') 72 | 73 | ret = cli(params) 74 | assert ret == 0 75 | captured = capsys.readouterr() 76 | assert captured.out == '"a\\tvalue"\n' 77 | assert not captured.err 78 | 79 | 80 | def test_cli_nonexistent_file(tmp_path, capsys): 81 | """It should exit with a failure exit code and no output.""" 82 | ret = cli([str(tmp_path / 'nonexistent.json'), 'name.first']) 83 | assert ret == 1 84 | captured = capsys.readouterr() 85 | assert not captured.out 86 | assert not captured.err 87 | 88 | 89 | def test_cli_nonexistent_file_verbosity_1(tmp_path, capsys): 90 | """It should exit with a failure exit code and print the error message.""" 91 | ret = cli(['-v', str(tmp_path / 'nonexistent.json'), 'name.first']) 92 | assert ret == 1 93 | captured = capsys.readouterr() 94 | assert not captured.out 95 | assert captured.err.startswith("ArgumentTypeError: can't open") 96 | assert 'nonexistent.json' in captured.err 97 | 98 | 99 | def test_cli_nonexistent_file_verbosity_2(tmp_path): 100 | """It should raise the exception and print the full traceback.""" 101 | with pytest.raises( 102 | argparse.ArgumentTypeError, match=r"can't open .*/nonexistent.json.* No such file or directory"): 103 | cli(['-vv', str(tmp_path / 'nonexistent.json'), 'name.first']) 104 | 105 | 106 | def test_cli_stdin_query_verbosity_1(monkeypatch, capsys): 107 | """It should exit with a failure exit code and print the error message.""" 108 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON)) 109 | ret = cli(['-v', '-', 'nonexistent']) 110 | assert ret == 1 111 | captured = capsys.readouterr() 112 | assert not captured.out 113 | assert captured.err == ('GJSONParseError: Mapping object does not have key `nonexistent`.\n' 114 | 'Query: nonexistent\n-------^\n') 115 | 116 | 117 | def test_cli_stdin_query_verbosity_2(monkeypatch): 118 | """It should exit with a failure exit code and print the full traceback.""" 119 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON)) 120 | with pytest.raises(GJSONParseError, match=r'Mapping object does not have key `nonexistent`.'): 121 | cli(['-vv', '-', 'nonexistent']) 122 | 123 | 124 | def test_cli_lines_ok(monkeypatch, capsys): 125 | """It should apply the same query to each line.""" 126 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES)) 127 | ret = cli(['--lines', '-', 'name']) 128 | assert ret == 0 129 | captured = capsys.readouterr() 130 | assert captured.out == '"Gilbert"\n"Alexa"\n"May"\n"Deloise"\n' 131 | assert not captured.err 132 | 133 | 134 | def test_cli_lines_failed_lines_verbosity_0(monkeypatch, capsys): 135 | """It should keep going with the other lines and just skip the failed line.""" 136 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 137 | ret = cli(['--lines', '-', 'name']) 138 | assert ret == 1 139 | captured = capsys.readouterr() 140 | assert captured.out == '"Gilbert"\n"Deloise"\n' 141 | assert not captured.err 142 | 143 | 144 | def test_cli_lines_failed_lines_verbosity_1(monkeypatch, capsys): 145 | """It should keep going with the other lines printing an error for the failed lines.""" 146 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 147 | ret = cli(['-v', '--lines', '-', 'name']) 148 | assert ret == 1 149 | captured = capsys.readouterr() 150 | assert captured.out == '"Gilbert"\n"Deloise"\n' 151 | assert captured.err.count('JSONDecodeError') == 2 152 | 153 | 154 | def test_cli_lines_failed_lines_verbosity_2(monkeypatch): 155 | """It should interrupt the processing and print the full traceback.""" 156 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 157 | with pytest.raises( 158 | json.decoder.JSONDecodeError, match=r'Expecting property name enclosed in double quotes'): 159 | cli(['-vv', '--lines', '-', 'name']) 160 | 161 | 162 | def test_cli_lines_double_dot_query(monkeypatch, capsys): 163 | """It should encapsulate each line in an array to allow queries.""" 164 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES)) 165 | ret = cli(['--lines', '..#(age>45).name']) 166 | assert ret == 1 167 | captured = capsys.readouterr() 168 | assert captured.out == '"Gilbert"\n"May"\n' 169 | assert not captured.err 170 | 171 | 172 | def test_cli_double_dot_query_ok(monkeypatch, capsys): 173 | """It should encapsulate the input in an array and apply the query to the array.""" 174 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES)) 175 | ret = cli(['-', '..#.name']) 176 | assert ret == 0 177 | captured = capsys.readouterr() 178 | assert captured.out == '["Gilbert", "Alexa", "May", "Deloise"]\n' 179 | assert not captured.err 180 | 181 | 182 | def test_cli_double_dot_query_failed_lines_verbosity_0(monkeypatch, capsys): 183 | """It should encapsulate the input in an array skipping failing lines.""" 184 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 185 | ret = cli(['-', '..#.name']) 186 | assert ret == 1 187 | captured = capsys.readouterr() 188 | assert not captured.out 189 | assert not captured.err 190 | 191 | 192 | def test_cli_double_dot_query_failed_lines_verbosity_1(monkeypatch, capsys): 193 | """It should encapsulate the input in an array skipping failing lines and printing an error for each failure.""" 194 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 195 | ret = cli(['-v', '-', '..#.name']) 196 | assert ret == 1 197 | captured = capsys.readouterr() 198 | assert not captured.out 199 | assert captured.err.startswith('JSONDecodeError: Expecting property name enclosed in double quotes') 200 | 201 | 202 | def test_cli_double_dot_query_failed_lines_verbosity_2(monkeypatch): 203 | """It should interrupt the execution at the first invalid line and exit printing the traceback.""" 204 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS)) 205 | with pytest.raises(json.decoder.JSONDecodeError, match=r'Expecting property name enclosed in double quotes'): 206 | cli(['-vv', '-', '..#.name']) 207 | -------------------------------------------------------------------------------- /tests/unit/test_init.py: -------------------------------------------------------------------------------- 1 | """GJSON test module.""" 2 | import json 3 | import re 4 | from collections.abc import Mapping 5 | from math import isnan 6 | 7 | import gjson 8 | import pytest 9 | from gjson._gjson import MODIFIER_NAME_RESERVED_CHARS 10 | 11 | INPUT_JSON = """ 12 | { 13 | "name": {"first": "Tom", "last": "Anderson"}, 14 | "age":37, 15 | "children": ["Sara","Alex","Jack"], 16 | "fav.movie": "Deer Hunter", 17 | "friends": [ 18 | {"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]}, 19 | {"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]}, 20 | {"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]} 21 | ] 22 | } 23 | """ 24 | INPUT_OBJECT = json.loads(INPUT_JSON) 25 | INPUT_LIST = json.loads(""" 26 | [ 27 | {"first": "Dale"}, 28 | {"first": "Jane"}, 29 | {"last": "Murphy"} 30 | ] 31 | """) 32 | INPUT_ESCAPE = json.loads(""" 33 | { 34 | "test": { 35 | "*":"valZ", 36 | "*v":"val0", 37 | "keyv*":"val1", 38 | "key*v":"val2", 39 | "keyv?":"val3", 40 | "key?v":"val4", 41 | "keyv.":"val5", 42 | "key.v":"val6", 43 | "keyk*":{"key?":"val7"}, 44 | "1key":"val8" 45 | } 46 | } 47 | """) 48 | # This json block is poorly formed on purpose. 49 | INPUT_BASIC = json.loads(""" 50 | {"age":100, "name2":{"here":"B\\\\\\"R"}, 51 | "noop":{"what is a wren?":"a bird"}, 52 | "happy":true,"immortal":false, 53 | "items":[1,2,3,{"tags":[1,2,3],"points":[[1,2],[3,4]]},4,5,6,7], 54 | "arr":["1",2,"3",{"hello":"world"},"4",5], 55 | "vals":[1,2,3],"name":{"first":"tom","last":null}, 56 | "created":"2014-05-16T08:28:06.989Z", 57 | "loggy":{ 58 | "programmers": [ 59 | { 60 | "firstName": "Brett", 61 | "lastName": "McLaughlin", 62 | "email": "aaaa", 63 | "tag": "good" 64 | }, 65 | { 66 | "firstName": "Jason", 67 | "lastName": "Hunter", 68 | "email": "bbbb", 69 | "tag": "bad" 70 | }, 71 | { 72 | "firstName": "Elliotte", 73 | "lastName": "Harold", 74 | "email": "cccc", 75 | "tag": "good" 76 | }, 77 | { 78 | "firstName": 1002.3, 79 | "age": 101 80 | } 81 | ] 82 | }, 83 | "lastly":{"end...ing":"soon","yay":"final"} 84 | } 85 | """) 86 | INPUT_LINES = """ 87 | {"name": "Gilbert", "age": 61} 88 | {"name": "Alexa", "age": 34} 89 | {"name": "May", "age": 57} 90 | {"name": "Deloise", "age": 44} 91 | """ 92 | INPUT_LINES_WITH_ERRORS = """ 93 | {"name": "Gilbert", "age": 61} 94 | {invalid 95 | {invalid 96 | {"name": "Deloise", "age": 44} 97 | """ 98 | INPUT_TRUTHINESS = json.loads(""" 99 | { 100 | "vals": [ 101 | { "a": 1, "b": true }, 102 | { "a": 2, "b": true }, 103 | { "a": 3, "b": false }, 104 | { "a": 4, "b": "0" }, 105 | { "a": 5, "b": 0 }, 106 | { "a": 6, "b": "1" }, 107 | { "a": 7, "b": 1 }, 108 | { "a": 8, "b": "true" }, 109 | { "a": 9, "b": false }, 110 | { "a": 10, "b": null }, 111 | { "a": 11 } 112 | ] 113 | } 114 | """) 115 | INPUT_SUM_N = json.loads(""" 116 | [ 117 | {"key": "a", "value": 1, "other": "value"}, 118 | {"key": "b", "value": 2}, 119 | {"key": "c", "value": 3, "other": "value"}, 120 | {"key": "a", "value": 7}, 121 | {"key": "b", "value": 1.5}, 122 | {"key": "d", "value": 4}, 123 | {"key": "c", "value": 9} 124 | ] 125 | """) 126 | INPUT_NESTED_QUERIES = json.loads(""" 127 | { 128 | "key": [ 129 | {"level1": [{"level2": [{"level3": [1, 2]}]}]}, 130 | {"level1": [{"level2": [{"level3": [2, 3]}]}]}, 131 | [[{"level3": [1, 2]}], [{"level3": [2, 3]}]], 132 | [[{"level3": [2, 3]}], [{"level3": [3, 4]}]], 133 | {"another": [{"level2": [{"level3": [2, 3]}]}]}, 134 | {"level1": [{"another": [{"level3": [2, 3]}]}]}, 135 | {"level1": [{"level2": [{"another": [2, 3]}]}]}, 136 | [[{"another": [2, 3]}], [{"another": [3, 4]}]], 137 | "spurious", 138 | 12.34, 139 | {"mixed": [[{"level4": [1, 2]}]]} 140 | ] 141 | } 142 | """) 143 | 144 | 145 | def compare_values(result, expected): 146 | """Compare results with the expected values ensuring same-order of keys for dictionaries.""" 147 | if isinstance(expected, float) and isnan(expected): 148 | assert isnan(result) 149 | return 150 | 151 | assert result == expected 152 | if isinstance(expected, Mapping): 153 | assert list(result.keys()) == list(expected.keys()) 154 | 155 | 156 | class TestObject: 157 | """Testing gjson with a basic input object.""" 158 | 159 | def setup_method(self): 160 | """Initialize the test instance.""" 161 | def upper(options, obj, *, last): 162 | """Change the case of the object returning its upper version. Custom modifier to be used in tests.""" 163 | del options 164 | del last 165 | if isinstance(obj, list): 166 | return [i.upper() for i in obj] 167 | 168 | return obj.upper() 169 | 170 | self.object = gjson.GJSON(INPUT_OBJECT) 171 | self.object.register_modifier('upper', upper) 172 | 173 | @pytest.mark.parametrize(('query', 'expected'), ( 174 | # Basic 175 | ('name.last', 'Anderson'), 176 | ('name.first', 'Tom'), 177 | ('age', 37), 178 | ('children', ['Sara', 'Alex', 'Jack']), 179 | ('children.0', 'Sara'), 180 | ('children.1', 'Alex'), 181 | ('friends.1', {'first': 'Roger', 'last': 'Craig', 'age': 68, 'nets': ['fb', 'tw']}), 182 | ('friends.1.first', 'Roger'), 183 | # Wildcards 184 | ('*.first', 'Tom'), 185 | ('?a??.first', 'Tom'), 186 | ('child*.2', 'Jack'), 187 | ('c?ildren.0', 'Sara'), 188 | # Escape characters 189 | (r'fav\.movie', 'Deer Hunter'), 190 | # Arrays 191 | ('friends.#', 3), 192 | ('friends.#.age', [44, 68, 47]), 193 | ('friends.#.first', ['Dale', 'Roger', 'Jane']), 194 | ('friends.#.nets.0', ['ig', 'fb', 'ig']), 195 | # Queries 196 | ('friends.#(last=="Murphy").first', 'Dale'), 197 | ('friends.#(last=="Murphy")#.first', ['Dale', 'Jane']), 198 | ('friends.#(=="Murphy")#', []), 199 | ('friends.#(=="Mu)(phy")#', []), 200 | ('friends.#(=="Mur\tphy")#', []), 201 | ('friends.#(age\\===44)#', []), 202 | ('friends.#(age>47)#.last', ['Craig']), 203 | ('friends.#(age>=47)#.last', ['Craig', 'Murphy']), 204 | ('friends.#(age<47)#.last', ['Murphy']), 205 | ('friends.#(age<=47)#.last', ['Murphy', 'Murphy']), 206 | ('friends.#(age==44)#.last', ['Murphy']), 207 | ('friends.#(age!=44)#.last', ['Craig', 'Murphy']), 208 | ('friends.#(first%"D*").last', 'Murphy'), 209 | ('friends.#(first!%"D*").last', 'Craig'), 210 | ('friends.#(first!%"D???").last', 'Craig'), 211 | ('friends.#(%0)#', []), 212 | ('friends.#(>40)#', []), 213 | ('children.#(!%"*a*")', 'Alex'), 214 | ('children.#(%"*a*")#', ['Sara', 'Jack']), 215 | # Nested queries 216 | ('friends.#(nets.#(=="fb"))#.first', ['Dale', 'Roger']), 217 | # Modifiers 218 | ('children.@reverse', ['Jack', 'Alex', 'Sara']), 219 | ('children.@reverse.0', 'Jack'), 220 | ('name.@reverse', {'last': 'Anderson', 'first': 'Tom'}), 221 | ('age.@reverse', 37), 222 | ('@keys', ['name', 'age', 'children', 'fav.movie', 'friends']), 223 | ('name.@values', ['Tom', 'Anderson']), 224 | ('age.@flatten', 37), 225 | ('@pretty:{"indent": 4}', INPUT_OBJECT), 226 | (r'fav\.movie.@pretty:{"indent": 4}', 'Deer Hunter'), 227 | ('name.@tostr', '{"first": "Tom", "last": "Anderson"}'), 228 | ('name.@join', {'first': 'Tom', 'last': 'Anderson'}), 229 | ('age.@join', 37), 230 | ('children.@join', {}), 231 | ('children.0.@join', 'Sara'), 232 | ('friends.@join', {'first': 'Jane', 'last': 'Murphy', 'age': 47, 'nets': ['ig', 'tw']}), 233 | # Dot vs Pipe 234 | ('friends.0.first', 'Dale'), 235 | ('friends|0.first', 'Dale'), 236 | ('friends.0|first', 'Dale'), 237 | ('friends|0|first', 'Dale'), 238 | ('friends|#', 3), 239 | ('friends.#(last="Murphy")#', 240 | [{'first': 'Dale', 'last': 'Murphy', 'age': 44, 'nets': ['ig', 'fb', 'tw']}, 241 | {'first': 'Jane', 'last': 'Murphy', 'age': 47, 'nets': ['ig', 'tw']}]), 242 | ('friends.#(last="Murphy")#.first', ['Dale', 'Jane']), 243 | ('friends.#(last="Murphy")#.0', []), 244 | ('friends.#(last="Murphy")#|0', {'first': 'Dale', 'last': 'Murphy', 'age': 44, 'nets': ['ig', 'fb', 'tw']}), 245 | ('friends.#(last="Murphy")#.#', []), 246 | ('friends.#(last="Murphy")#|#', 2), 247 | # Multipaths objects 248 | ('{}', {}), 249 | ('{.}', {}), 250 | ('{.invalid}', {}), 251 | ('{.invalid,}', {}), 252 | ('{age}', {'age': 37}), 253 | (r'{a\ge}', {r'a\ge': 37}), 254 | (r'{"a\\ge":age}', {r'a\ge': 37}), 255 | ('{"a\tb":age}', {'a\tb': 37}), 256 | ('{"key":age}', {'key': 37}), 257 | ('{age,age}', {'age': 37}), 258 | ('{age,"years":age}', {'age': 37, 'years': 37}), 259 | ('{"years":age,age}', {'years': 37, 'age': 37}), 260 | ('{age,name.first}', {'age': 37, 'first': 'Tom'}), 261 | ('{invalid,invalid.invalid,age}', {'age': 37}), 262 | ('{name.first,age,name.last}', {'first': 'Tom', 'age': 37, 'last': 'Anderson'}), 263 | ('{{age}}', {'_': {'age': 37}}), 264 | ('{{age},age}', {'_': {'age': 37}, 'age': 37}), 265 | ('friends.0.{age,nets.#(="ig")}', {'age': 44, '_': 'ig'}), 266 | ('friends.0.{age,nets.#(="ig"),invalid}', {'age': 44, '_': 'ig'}), 267 | ('friends.0.{age,nets.#(="ig")#}', {'age': 44, '_': ['ig']}), 268 | ('friends.#.{age,"key":first}', 269 | [{'age': 44, 'key': 'Dale'}, {'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]), 270 | ('friends.#(age>44)#.{age,"key":first}', [{'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]), 271 | ('friends.#(age>44)#.{age,"key":first,invalid}', [{'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]), 272 | (r'{age,name.first,fav\.movie}', {'age': 37, 'first': 'Tom', r'fav\.movie': 'Deer Hunter'}), 273 | ('{age,name.{"name":first,"surname":last},children.@sort}', 274 | {'age': 37, '_': {'name': 'Tom', 'surname': 'Anderson'}, '@sort': ['Alex', 'Jack', 'Sara']}), 275 | ('friends.{0.first,1.last,2.age}.@values', ['Dale', 'Craig', 47]), 276 | ('{friends.{"a":0.{nets.{0}}}}', {'_': {'a': {'_': {'0': 'ig'}}}}), 277 | ('{friends.{"a":0.{nets.{0,1}}}}', {'_': {'a': {'_': {'0': 'ig', '1': 'fb'}}}}), 278 | ('friends.#.{age,first|@upper}', 279 | [{'age': 44, '@upper': 'DALE'}, {'age': 68, '@upper': 'ROGER'}, {'age': 47, '@upper': 'JANE'}]), 280 | ('{friends.#.{age,"first":first|@upper}|0.first}', {'first': 'DALE'}), 281 | ('{"children":children|@upper,"name":name.first,"age":age}', 282 | {'children': ['SARA', 'ALEX', 'JACK'], 'name': 'Tom', 'age': 37}), 283 | ('friends.#.{age,"first":first.invalid}', [{'age': 44}, {'age': 68}, {'age': 47}]), 284 | # Multipaths arrays 285 | ('[]', []), 286 | ('[.]', []), 287 | ('[.invalid]', []), 288 | ('[.invalid,]', []), 289 | ('[age]', [37]), 290 | (r'[a\ge]', [37]), 291 | ('[age,age]', [37, 37]), 292 | ('[age,name.first]', [37, 'Tom']), 293 | ('[name.first,age,invalid,invalid.invalid,name.last]', ['Tom', 37, 'Anderson']), 294 | ('[[age]]', [[37]]), 295 | ('[[age],age]', [[37], 37]), 296 | ('friends.0.[age,nets.#(="ig")]', [44, 'ig']), 297 | ('friends.0.[age,nets.#(="ig"),invalid]', [44, 'ig']), 298 | ('friends.0.[age,nets.#(="ig")#]', [44, ['ig']]), 299 | ('friends.#.[age,first]', [[44, 'Dale'], [68, 'Roger'], [47, 'Jane']]), 300 | ('friends.#(age>44)#.[age,first]', [[68, 'Roger'], [47, 'Jane']]), 301 | ('friends.#(age>44)#.[age,invalid,invalid.invalid,first]', [[68, 'Roger'], [47, 'Jane']]), 302 | (r'[age,name.first,fav\.movie]', [37, 'Tom', 'Deer Hunter']), 303 | ('[age,name.[first,last],children.@sort]', [37, ['Tom', 'Anderson'], ['Alex', 'Jack', 'Sara']]), 304 | ('friends.[0.first,1.last,2.age]', ['Dale', 'Craig', 47]), 305 | ('[friends.[0.[nets.[0]]]]', [[[['ig']]]]), 306 | ('[friends.[0.[nets.[0,1]]]]', [[[['ig', 'fb']]]]), 307 | # Multipaths mixed 308 | ('[{}]', [{}]), 309 | ('{[]}', {'_': []}), 310 | ('[{},[],{}]', [{}, [], {}]), 311 | ('{"a":[]}', {'a': []}), 312 | ('[{age},{name.first}]', [{'age': 37}, {'first': 'Tom'}]), 313 | ('{friends.0.[age,nets.#(="ig")]}', {'_': [44, 'ig']}), 314 | ('{friends.0.[age,nets.#(="ig")],age}', {'_': [44, 'ig'], 'age': 37}), 315 | ('{friends.0.[invalid,nets.#(="ig")],age,invalid}', {'_': ['ig'], 'age': 37}), 316 | # Literals 317 | ('!true', True), 318 | ('!false', False), 319 | ('!null', None), 320 | ('!NaN', float('nan')), 321 | ('!Infinity', float('inf')), 322 | ('!-Infinity', float('-inf')), 323 | ('!"key"', 'key'), 324 | ('!"line \\"quotes\\""', 'line "quotes"'), 325 | ('!"a\tb"', 'a\tb'), 326 | ('!0', 0), 327 | ('!12', 12), 328 | ('!-12', -12), 329 | ('!12.34', 12.34), 330 | ('!12.34E2', 1234), 331 | ('!12.34E+2', 1234), 332 | ('!12.34e-2', 0.1234), 333 | ('!-12.34e-2', -0.1234), 334 | ('friends.#.!"value"', ['value', 'value', 'value']), 335 | ('friends.#.!invalid', []), 336 | ('friends.#|!"value"', 'value'), 337 | ('friends.#(age>45)#.!"value"', ['value', 'value']), 338 | ('name|!"value"', 'value'), 339 | ('!{}', {}), 340 | ('![]', []), 341 | ('!{"name":{"first":"Tom"}}.{name.first}.first', 'Tom'), 342 | ('{name.last,"key":!"value"}', {'last': 'Anderson', 'key': 'value'}), 343 | ('{name.last,"key":!{"a":"b"},"invalid"}', {'last': 'Anderson', 'key': {'a': 'b'}}), 344 | ('{name.last,"key":!{"c":"d"},!"valid"}', {'last': 'Anderson', 'key': {'c': 'd'}, '_': 'valid'}), 345 | ('[!true,!false,!null,!Infinity,!invalid,{"name":!"andy",name.last},+Infinity,!["value1","value2"]]', 346 | [True, False, None, float('inf'), {'name': 'andy', 'last': 'Anderson'}, ['value1', 'value2']]), 347 | ('[!12.34,!-12.34e-2,!true]', [12.34, -0.1234, True]), 348 | )) 349 | def test_get_ok(self, query, expected): 350 | """It should query the JSON object and return the expected result.""" 351 | compare_values(self.object.get(query), expected) 352 | 353 | @pytest.mark.parametrize(('query', 'error'), ( 354 | # Basic 355 | ('.', 'Invalid query starting with a path delimiter.'), 356 | ('|', 'Invalid query starting with a path delimiter.'), 357 | ('.name', 'Invalid query starting with a path delimiter.'), 358 | ('|age', 'Invalid query starting with a path delimiter.'), 359 | ('name..first', 'Invalid query with two consecutive path delimiters.'), 360 | ('name||first', 'Invalid query with two consecutive path delimiters.'), 361 | ('name.|first', 'Invalid query with two consecutive path delimiters.'), 362 | ('name|.first', 'Invalid query with two consecutive path delimiters.'), 363 | ('age.0', "Integer query part on unsupported object type "), 364 | ('friends.99', 'Index `99` out of range for sequence object with 3 items in query.'), 365 | ('name.nonexistent', 'Mapping object does not have key `nonexistent`.'), 366 | ('name.1', 'Mapping object does not have key `1`.'), 367 | ('children.invalid', 'Invalid or unsupported query part `invalid`.'), 368 | ('children.', 'Delimiter at the end of the query.'), 369 | ('children\\', 'Escape character at the end of the query.'), 370 | # Wildcards 371 | ('x*', 'No key matching pattern with wildcard `x*`'), 372 | ('??????????', 'No key matching pattern with wildcard `??????????`'), 373 | ('children.x*', "Wildcard matching key `x*` requires a mapping object, got instead."), 374 | ('(-?', 'No key matching pattern with wildcard `(-?`'), 375 | # Queries 376 | ('#', "Expected a sequence like object for query part # at the end of the query, got ."), 377 | ('#.invalid', 'Invalid or unsupported query part `invalid`.'), 378 | ('friends.#(=="Murphy")', 'Query on mapping like objects require a key before the operator.'), 379 | ('friends.#(last=={1: 2})', 'Invalid value `{1: 2}` for the query key `last`'), 380 | ('friends.#(invalid', 'Unbalanced parentheses `(`, 1 still opened.'), 381 | ('#(first)', 'Queries are supported only for sequence like objects'), 382 | ('friends.#(invalid)', 'Query for first element does not match anything.'), 383 | ('friends.#(last=="invalid")', 'Query for first element does not match anything.'), 384 | ('friends.#(first%"D?")', 'Query for first element does not match anything.'), 385 | ('friends.#(last=="Murphy")invalid', 'Expected delimiter or end of query after closing parenthesis.'), 386 | ('children.#()', 'Empty or invalid query.'), 387 | ('children.#()#', 'Empty or invalid query.'), 388 | ('friends.#.invalid.#()', 'Empty or invalid query.'), 389 | ('friends.#.invalid.#()#', 'Empty or invalid query.'), 390 | # Dot vs Pipe 391 | ('friends.#(last="Murphy")#|first', 'Invalid or unsupported query'), 392 | # Modifiers 393 | ('@', 'Got empty modifier name.'), 394 | ('friends.@', 'Got empty modifier name.'), 395 | ('friends.@pretty:', 'Modifier with options separator `:` without any option.'), 396 | ('friends.@pretty:{invalid', 'Unable to load modifier options.'), 397 | ('friends.@pretty:["invalid"]', 'Expected JSON object `{...}` as modifier options.'), 398 | ('friends.@invalid', 'Unknown modifier @invalid.'), 399 | ('friends.@in"valid', 'Invalid modifier name @in"valid, the following characters are not allowed'), 400 | # JSON Lines 401 | ('..name', 'Invalid query starting with a path delimiter.'), 402 | # Multipaths 403 | (r'{"a\ge":age}', r'Failed to parse multipaths key "a\ge"'), 404 | ('{"age",age}', 'Expected colon after multipaths item with key "age".'), 405 | ('{]', 'Unbalanced parentheses `{`, 1 still opened.'), 406 | ('{', 'Unbalanced parentheses `{`, 1 still opened.'), 407 | ('{}@pretty', 'Expected delimiter or end of query after closing parenthesis.'), 408 | ('[{age}}]', 'Missing separator after multipath.'), 409 | ('{[age]]}', 'Missing separator after multipath.'), 410 | ('[{age,name.first]},age]', 'Expected delimiter or end of query after closing parenthesis.'), 411 | # Literals 412 | ('!', 'Unable to load literal JSON'), 413 | ('name.!', 'Unable to load literal JSON'), 414 | ('!invalid', 'Unable to load literal JSON'), 415 | (r'!in\valid', 'Unable to load literal JSON'), 416 | ('!0.a', 'Invalid or unsupported query part `a`.'), 417 | ('!0.1ea', 'Invalid or unsupported query part `ea`.'), 418 | ('!-12.', 'Delimiter at the end of the query.'), 419 | ('!-12.e', 'Invalid or unsupported query part `e`.'), 420 | ('name.!invalid', 'Unable to load literal JSON'), 421 | ('!"invalid', 'Unable to find end of literal string.'), 422 | ('friends.#|!invalid', 'Unable to load literal JSON'), 423 | ('!{true,', 'Unbalanced parentheses `{`, 1 still opened.'), 424 | ('![true,', 'Unbalanced parentheses `[`, 1 still opened.'), 425 | ('!"value".invalid', 'Invalid or unsupported query part `invalid`.'), 426 | ('name.!"value"', 'Unable to load literal JSON: literal afer a dot delimiter.'), 427 | )) 428 | def test_get_parser_raise(self, query, error): 429 | """It should raise a GJSONParseError error with the expected message.""" 430 | with pytest.raises(gjson.GJSONParseError, match=fr'^{re.escape(error)}'): 431 | self.object.get(query) 432 | 433 | @pytest.mark.parametrize(('query', 'error'), ( 434 | # Basic 435 | ('', 'Empty query.'), 436 | # Modifiers 437 | ('children.@keys', 'The current object does not have a keys() method.'), 438 | ('children.@values', 'The current object does not have a values() method.'), 439 | ('age.@group', "Modifier @group got object of type as input, expected dictionary."), 440 | ('children.@group', "Modifier @group got object of type as input, expected dictionary."), 441 | )) 442 | def test_get_raise(self, query, error): 443 | """It should raise a GJSONError error with the expected message.""" 444 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'): 445 | self.object.get(query) 446 | 447 | 448 | class TestEscape: 449 | """Test gjson for all the escape sequences.""" 450 | 451 | def setup_method(self): 452 | """Initialize the test instance.""" 453 | self.escape = gjson.GJSON(INPUT_ESCAPE) 454 | 455 | @pytest.mark.parametrize(('query', 'expected'), ( 456 | (r'test.\*', 'valZ'), 457 | (r'test.\*v', 'val0'), 458 | (r'test.keyv\*', 'val1'), 459 | (r'test.key\*v', 'val2'), 460 | (r'test.keyv\?', 'val3'), 461 | (r'test.key\?v', 'val4'), 462 | (r'test.keyv\.', 'val5'), 463 | (r'test.key\.v', 'val6'), 464 | (r'test.keyk\*.key\?', 'val7'), 465 | ('test.1key', 'val8'), 466 | )) 467 | def test_get_ok(self, query, expected): 468 | """It should query the escape test JSON and return the expected result.""" 469 | assert self.escape.get(query, quiet=True) == expected 470 | 471 | 472 | class TestBasic: 473 | """Test gjson for basic queries.""" 474 | 475 | def setup_method(self): 476 | """Initialize the test instance.""" 477 | self.basic = gjson.GJSON(INPUT_BASIC) 478 | 479 | @pytest.mark.parametrize(('query', 'expected'), ( 480 | ('loggy.programmers.#(age=101).firstName', 1002.3), 481 | ('loggy.programmers.#(firstName != "Brett").firstName', 'Jason'), 482 | ('loggy.programmers.#(firstName % "Bre*").email', 'aaaa'), 483 | ('loggy.programmers.#(firstName !% "Bre*").email', 'bbbb'), 484 | ('loggy.programmers.#(firstName == "Brett").email', 'aaaa'), 485 | ('loggy.programmers.#.firstName', ['Brett', 'Jason', 'Elliotte', 1002.3]), 486 | ('loggy.programmers.#.asd', []), 487 | ('items.3.tags.#', 3), 488 | ('items.3.points.1.#', 2), 489 | ('items.#', 8), 490 | ('vals.#', 3), 491 | ('name2.here', r'B\"R'), 492 | ('arr.#', 6), 493 | ('arr.3.hello', 'world'), 494 | ('name.first', 'tom'), 495 | ('name.last', None), 496 | ('age', 100), 497 | ('happy', True), 498 | ('immortal', False), 499 | ('noop', {'what is a wren?': 'a bird'}), 500 | # Modifiers 501 | ('arr.@join', {'hello': 'world'}), 502 | )) 503 | def test_get_ok(self, query, expected): 504 | """It should query the basic test JSON and return the expected result.""" 505 | assert self.basic.get(query) == expected 506 | 507 | 508 | class TestList: 509 | """Test gjson queries on a list object.""" 510 | 511 | def setup_method(self): 512 | """Initialize the test instance.""" 513 | self.list = gjson.GJSON(INPUT_LIST) 514 | 515 | @pytest.mark.parametrize(('query', 'expected'), ( 516 | # Dot vs Pipe 517 | ('#.first', ['Dale', 'Jane']), 518 | ('#.first.#', []), 519 | ('#.first|#', 2), 520 | ('#.0', []), 521 | ('#.#', []), 522 | # Queries 523 | ('#(first)#', [{'first': 'Dale'}, {'first': 'Jane'}]), 524 | ('#(first)', {'first': 'Dale'}), 525 | ('#(last)#', [{'last': 'Murphy'}]), 526 | ('#(last)', {'last': 'Murphy'}), 527 | # Modifiers 528 | ('@join', {'first': 'Jane', 'last': 'Murphy'}), 529 | # Multipaths 530 | ('#.{first.@reverse}', [{'@reverse': 'Dale'}, {'@reverse': 'Jane'}, {}]), 531 | )) 532 | def test_get_ok(self, query, expected): 533 | """It should query the list test JSON and return the expected result.""" 534 | assert self.list.get(query, quiet=False) == expected 535 | 536 | @pytest.mark.parametrize(('query', 'error'), ( 537 | # Dot vs Pipe 538 | ('#|first', 'Invalid or unsupported query part `first`.'), 539 | ('#|0', 'Integer query part after a pipe delimiter on an sequence like object.'), 540 | ('#|#', 'The pipe delimiter cannot immediately follow the # element.'), 541 | )) 542 | def test_get_raise(self, query, error): 543 | """It should raise a GJSONError error with the expected message.""" 544 | with pytest.raises(gjson.GJSONParseError, match=fr'^{re.escape(error)}'): 545 | self.list.get(query) 546 | 547 | 548 | class TestFlattenModifier: 549 | """Test gjson @flatten modifier.""" 550 | 551 | def setup_method(self): 552 | """Initialize the test instance.""" 553 | self.list = gjson.GJSON(json.loads('[1, [2], [3, 4], [5, [6, 7]], [8, [9, [10, 11]]]]')) 554 | 555 | @pytest.mark.parametrize(('query', 'expected'), ( 556 | ('@flatten', [1, 2, 3, 4, 5, [6, 7], 8, [9, [10, 11]]]), 557 | ('@flatten:{"deep":true}', [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]), 558 | )) 559 | def test_get(self, query, expected): 560 | """It should correctly flatten the given object.""" 561 | assert self.list.get(query, quiet=True) == expected 562 | 563 | 564 | class TestTruthiness: 565 | """Testing gjson with an input object with truthy/falsy objects.""" 566 | 567 | def setup_method(self): 568 | """Initialize the test instance.""" 569 | self.object = gjson.GJSON(INPUT_TRUTHINESS) 570 | 571 | @pytest.mark.parametrize(('query', 'expected'), ( 572 | ('vals.#(b==~true).a', 1), 573 | ('vals.#(b==~true)#.a', [1, 2, 4, 6, 7, 8]), 574 | ('vals.#(b==~false).a', 3), 575 | ('vals.#(b==~false)#.a', [3, 5, 9, 10, 11]), 576 | ('vals.#(b==~"invalid")#', []), 577 | )) 578 | def test_get_ok(self, query, expected): 579 | """It should query the JSON object and return the expected result.""" 580 | compare_values(self.object.get(query), expected) 581 | 582 | @pytest.mark.parametrize(('query', 'error'), ( 583 | ('vals.#(b==~"invalid")', 584 | "Queries ==~ operator requires a boolean value, got instead: `invalid`"), 585 | )) 586 | def test_get_raise(self, query, error): 587 | """It should raise a GJSONError error with the expected message.""" 588 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'): 589 | self.object.get(query) 590 | 591 | 592 | class TestNestedQueries: 593 | """Testing gjson nested queries.""" 594 | 595 | def setup_method(self): 596 | """Initialize the test instance.""" 597 | self.object = gjson.GJSON(INPUT_NESTED_QUERIES) 598 | 599 | @pytest.mark.parametrize(('query', 'expected'), ( 600 | # Arrays of objects 601 | ('key.#(level1.#(level2.#(level3)))', INPUT_NESTED_QUERIES['key'][0]), 602 | ('key.#(level1.#(level2.#(level3)))#', INPUT_NESTED_QUERIES['key'][0:2]), 603 | ('key.#(level1.#(level2.#(level3.#(==0))))#', []), 604 | ('key.#(level1.#(level2.#(level3.#(=1))))', INPUT_NESTED_QUERIES['key'][0]), 605 | ('key.#(level1.#(level2.#(level3.#(=1)#)#)#)', INPUT_NESTED_QUERIES['key'][0]), 606 | ('key.#(level1.#(level2.#(level3.#(==1))))#', [INPUT_NESTED_QUERIES['key'][0]]), 607 | ('key.#(level1.#(level2.#(level3.#(==2))))', INPUT_NESTED_QUERIES['key'][0]), 608 | ('key.#(level1.#(level2.#(level3.#(=2))))#', INPUT_NESTED_QUERIES['key'][0:2]), 609 | # Arrays of arrays 610 | ('key.#(#(#(level3)))', INPUT_NESTED_QUERIES['key'][2]), 611 | ('key.#(#(#(level3)))#', INPUT_NESTED_QUERIES['key'][2:4]), 612 | ('key.#(#(#(level3.#(==0))))#', []), 613 | ('key.#(#(#(level3.#(==1))))', INPUT_NESTED_QUERIES['key'][2]), 614 | ('key.#(#(#(level3.#(==1)#)#)#)', INPUT_NESTED_QUERIES['key'][2]), 615 | ('key.#(#(#(level3.#(==1))))#', [INPUT_NESTED_QUERIES['key'][2]]), 616 | ('key.#(#(#(level3.#(==2))))', INPUT_NESTED_QUERIES['key'][2]), 617 | ('key.#(#(#(level3.#(==2))))#', INPUT_NESTED_QUERIES['key'][2:4]), 618 | ('key.#(#(#(level3.#(>=4))))', INPUT_NESTED_QUERIES['key'][3]), 619 | ('key.#(#(#(level3.#(>=4))))#', [INPUT_NESTED_QUERIES['key'][3]]), 620 | # Mixed 621 | ('key.#(mixed.#(#(level4)))', INPUT_NESTED_QUERIES['key'][-1]), 622 | ('key.#(mixed.#(#(level4)))#', [INPUT_NESTED_QUERIES['key'][-1]]), 623 | )) 624 | def test_get_ok(self, query, expected): 625 | """It should query the JSON object and return the expected result.""" 626 | compare_values(self.object.get(query), expected) 627 | 628 | @pytest.mark.parametrize(('query', 'error'), ( 629 | ('key.#(level1.#(level2.#(level3.#(==0))))', 'Query for first element does not match anything.'), 630 | ('key.#(#(#(level3.#(==0))))', 'Query for first element does not match anything.'), 631 | )) 632 | def test_get_raise(self, query, error): 633 | """It should raise a GJSONError error with the expected message.""" 634 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'): 635 | self.object.get(query) 636 | 637 | 638 | @pytest.mark.parametrize(('query', 'expected'), ( 639 | ('0.0', 'zero'), 640 | ('0|0', 'zero'), 641 | ('#.0', ['zero']), 642 | ('#.1', ['one', 'one']), 643 | ('#.9', []), 644 | ('#(0="zero")#|0', {'0': 'zero', '1': 'one'}), 645 | ('#(0="zero")#.1', ['one']), 646 | ('#(0="zero")#.9', []), 647 | ('#(0="invalid")#.1', []), 648 | )) 649 | def test_get_integer_mapping_keys_ok(query, expected): 650 | """It should return the expected result.""" 651 | obj = gjson.GJSON([{'0': 'zero', '1': 'one'}, {'1': 'one'}]) 652 | assert obj.get(query, quiet=True) == expected 653 | 654 | 655 | @pytest.mark.parametrize(('query', 'error'), ( 656 | ('0.1', 'Mapping object does not have key `1`.'), 657 | ('#|0', 'Integer query part after a pipe delimiter on an sequence like object.'), 658 | ('#|9', 'Integer query part after a pipe delimiter on an sequence like object.'), 659 | ('#(0="zero")#|1', 'Index `1` out of range for sequence object with 1 items in query.'), 660 | )) 661 | def test_get_integer_mapping_keys_raise(query, error): 662 | """It should return the expected result.""" 663 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'): 664 | 665 | gjson.GJSON([{'0': 'zero'}]).get(query) 666 | 667 | 668 | @pytest.mark.parametrize('modifier', ('@valid', '@this')) 669 | def test_get_modifier_unmodified_ok(modifier): 670 | """It should return the same object.""" 671 | obj = gjson.GJSON(INPUT_OBJECT) 672 | assert obj.get(modifier, quiet=True) == INPUT_OBJECT 673 | 674 | 675 | def test_get_modifier_valid_raise(): 676 | """It should return None if the object is invalid JSON and quiet is True.""" 677 | obj = gjson.GJSON({'invalid': {1, 2}}) 678 | assert obj.get('@valid', quiet=True) is None 679 | 680 | 681 | @pytest.mark.parametrize(('data', 'expected'), ( 682 | ('[3, 1, 5, 8, 2]', [1, 2, 3, 5, 8]), 683 | ('{"b": 2, "d": 4, "c": 3, "a": 1}', {'a': 1, 'b': 2, 'c': 3, 'd': 4}), 684 | ('"a string"', None), 685 | )) 686 | def test_get_modifier_sort(data, expected): 687 | """It should return the object sorted.""" 688 | obj = gjson.GJSON(json.loads(data)) 689 | compare_values(obj.get('@sort', quiet=True), expected) 690 | 691 | 692 | @pytest.mark.parametrize(('data', 'query', 'expected'), ( 693 | ({'a': '{"b": 25}'}, 'a.@fromstr', {'b': 25}), 694 | ({'a': '{"b": 25}'}, 'a.@fromstr.b', 25), 695 | )) 696 | def test_get_modifier_fromstr_ok(data, query, expected): 697 | """It should load the JSON-encoded string.""" 698 | obj = gjson.GJSON(data) 699 | assert obj.get(query, quiet=True) == expected 700 | 701 | 702 | @pytest.mark.parametrize(('query', 'error'), ( 703 | ('a.@fromstr', 'The current @fromstr input object cannot be converted to JSON.'), 704 | ('b.@fromstr', "Modifier @fromstr got object of type as input, expected string or bytes."), 705 | )) 706 | def test_get_modifier_fromstr_raise(query, error): 707 | """It should raise a GJSONError if the JSON-encoded string has invalid JSON.""" 708 | obj = gjson.GJSON({'a': '{"invalid: json"', 'b': {'not': 'a string'}}) 709 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'): 710 | obj.get(query) 711 | 712 | 713 | def test_get_modifier_tostr_raise(): 714 | """It should raise a GJSONError if the object cannot be JSON-encoded.""" 715 | obj = gjson.GJSON({'a': {1, 2, 3}}) # Python sets cannot be JSON-encoded 716 | match = re.escape('The current object cannot be converted to a JSON-encoded string for @tostr.') 717 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'): 718 | obj.get('a.@tostr') 719 | 720 | 721 | def test_get_modifier_group_ok(): 722 | """It should group the dict of lists into a list of dicts.""" 723 | obj = gjson.GJSON({ 724 | 'invalid1': 5, 725 | 'id': ['123', '456', '789'], 726 | 'val': [2, 1], 727 | 'invalid2': 'invalid', 728 | 'unit': ['ms', 's', 's', 'ms'], 729 | }) 730 | assert obj.get('@group') == [ 731 | {'id': '123', 'val': 2, 'unit': 'ms'}, 732 | {'id': '456', 'val': 1, 'unit': 's'}, 733 | {'id': '789', 'unit': 's'}, 734 | {'unit': 'ms'}, 735 | ] 736 | 737 | 738 | def test_get_modifier_group_empty(): 739 | """It should return an empty list if no values are lists or are empty.""" 740 | obj = gjson.GJSON({'invalid1': 5, 'invalid2': 'invalid', 'invalid3': {'a': 5}, 'id': []}) 741 | assert obj.get('@group') == [] 742 | 743 | 744 | def test_get_integer_index_on_mapping(): 745 | """It should access the integer as string key correctly.""" 746 | obj = gjson.GJSON(json.loads('{"1": 5, "11": 7}')) 747 | assert obj.get('1') == 5 748 | assert obj.get('11') == 7 749 | 750 | 751 | def test_module_get(): 752 | """It should return the queried object.""" 753 | assert gjson.get({'key': 'value'}, 'key') == 'value' 754 | 755 | 756 | def test_gjson_get_gjson_ok(): 757 | """It should return the queried object as a GJSON object.""" 758 | ret = gjson.GJSON(INPUT_OBJECT).get_gjson('children') 759 | assert isinstance(ret, gjson.GJSON) 760 | assert str(ret) == '["Sara", "Alex", "Jack"]' 761 | 762 | 763 | @pytest.mark.parametrize('kwargs', ({}, {'quiet': False})) 764 | def test_gjson_get_gjson_raise(kwargs): 765 | """It should raise a GJSONError if the quiet parameter is not passed or is False.""" 766 | with pytest.raises(gjson.GJSONError, match=r'^Mapping object does not have key `nonexistent`.'): 767 | gjson.GJSON(INPUT_OBJECT).get_gjson('nonexistent', **kwargs) 768 | 769 | 770 | @pytest.mark.parametrize(('data', 'num', 'expected'), ( 771 | # Valid data 772 | ('[1, 2, 3, 4, 5]', None, {1: 1, 2: 1, 3: 1, 4: 1, 5: 1}), 773 | ('[1, 2, 3, 4, 5]', 0, {}), 774 | ('[1, 2, 3, 4, 5]', 2, {1: 1, 2: 1}), 775 | ('[1, 1, 1, 1, 1]', None, {1: 5}), 776 | ('[1, 1, 1, 1, 1]', 1, {1: 5}), 777 | ('[1, 1, 1, 2, 2, 3]', None, {1: 3, 2: 2, 3: 1}), 778 | ('[1, 1, 1, 2, 2, 3, 3, 3, 3]', None, {3: 4, 1: 3, 2: 2}), 779 | ('[1, 1, 1, 2, 2, 3, 3, 3, 3]', 2, {3: 4, 1: 3}), 780 | # Invalid data 781 | ('{"key": "value"}', None, None), 782 | ('1', None, None), 783 | ('"value"', None, None), 784 | )) 785 | def test_get_modifier_top_n(data, num, expected): 786 | """It should return the top N common items.""" 787 | obj = gjson.GJSON(json.loads(data)) 788 | if num is not None: 789 | compare_values(obj.get(f'@top_n:{{"n": {num}}}', quiet=True), expected) 790 | else: 791 | compare_values(obj.get('@top_n', quiet=True), expected) 792 | 793 | 794 | @pytest.mark.parametrize(('num', 'expected'), ( 795 | (0, {}), 796 | (1, {'c': 12}), 797 | (2, {'c': 12, 'a': 8}), 798 | (3, {'c': 12, 'a': 8, 'd': 4}), 799 | (4, {'c': 12, 'a': 8, 'd': 4, 'b': 3.5}), 800 | (None, {'c': 12, 'a': 8, 'd': 4, 'b': 3.5}), 801 | )) 802 | def test_get_modifier_sum_n_valid(num, expected): 803 | """It should group and sum and return the top N items.""" 804 | obj = gjson.GJSON(INPUT_SUM_N) 805 | if num is not None: 806 | compare_values(obj.get(f'@sum_n:{{"group": "key", "sum": "value", "n": {num}}}', quiet=True), expected) 807 | else: 808 | compare_values(obj.get('@sum_n:{"group": "key", "sum": "value"}', quiet=True), expected) 809 | 810 | 811 | @pytest.mark.parametrize('data', ( 812 | '{"an": "object"}', 813 | '"a string"', 814 | '1', 815 | )) 816 | def test_get_modifier_sum_n_invalid_data(data): 817 | """It should raise a GJSONError if the input is invalid.""" 818 | obj = gjson.GJSON(json.loads(data)) 819 | with pytest.raises(gjson.GJSONError, match=r'^@sum_n modifier not supported for object of type'): 820 | obj.get('@sum_n:{"group": "key", "sum": "value"}') 821 | 822 | 823 | @pytest.mark.parametrize('options', ( 824 | '', 825 | ':{}', 826 | ':{"group": "invalid", "sum": "value"}', 827 | ':{"group": "key", "sum": "invalid"}', 828 | ':{"group": "key", "sum": "other"}', 829 | ':{"group": "other", "sum": "value"}', 830 | )) 831 | def test_get_modifier_sum_n_invalid_options(options): 832 | """It should raise a GJSONError if the options are invalid.""" 833 | obj = gjson.GJSON(INPUT_SUM_N) 834 | with pytest.raises(gjson.GJSONError, match=r'^Modifier @sum_n raised an exception'): 835 | obj.get(f'@sum_n{options}') 836 | 837 | 838 | class TestJSONOutput: 839 | """Test class for all JSON output functionalities.""" 840 | 841 | def setup_method(self): 842 | """Initialize the test instance.""" 843 | self.obj = {'key': 'value', 'hello world': '\u3053\u3093\u306b\u3061\u306f\u4e16\u754c'} 844 | self.query = 'key' 845 | self.value = '"value"' 846 | self.gjson = gjson.GJSON(self.obj) 847 | 848 | def test_module_get_as_str(self): 849 | """It should return the queried object as a JSON string.""" 850 | assert gjson.get(self.obj, self.query, as_str=True) == self.value 851 | assert gjson.get(self.obj, '', as_str=True, quiet=True) == '' 852 | 853 | def test_module_get_str(self): 854 | """It should return the string representation of the object.""" 855 | assert str(self.gjson) == '{"key": "value", "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"}' 856 | 857 | def test_gjson_getj(self): 858 | """It should return the queried object as a JSON string.""" 859 | assert self.gjson.getj(self.query) == self.value 860 | assert self.gjson.getj('', quiet=True) == '' 861 | 862 | def test_module_get_as_str_raise(self): 863 | """It should raise a GJSONError with the proper message on failure.""" 864 | with pytest.raises(gjson.GJSONError, match=r'^Empty query.'): 865 | gjson.get(self.obj, '', as_str=True) 866 | 867 | def test_gjson_get_as_str_raise(self): 868 | """It should raise a GJSONError with the proper message on failure.""" 869 | with pytest.raises(gjson.GJSONError, match=r'^Empty query.'): 870 | self.gjson.getj('') 871 | 872 | @pytest.mark.parametrize(('query', 'expected'), ( 873 | ('@pretty', '{\n "key": "value",\n "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'), 874 | ('@pretty:{"indent": 4}', 875 | '{\n "key": "value",\n "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'), 876 | ('@pretty:{"indent": "\t"}', 877 | '{\n\t"key": "value",\n\t"hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'), 878 | # Multipaths 879 | ('{key,"another":key}.@pretty', '{\n "key": "value",\n "another": "value"\n}'), 880 | )) 881 | def test_modifier_pretty(self, query, expected): 882 | """It should prettyfy the JSON string based on the parameters.""" 883 | assert self.gjson.getj(query) == expected 884 | 885 | def test_modifier_pretty_sort_keys_prefix(self): 886 | """It should prettyfy the JSON string and sort the keys.""" 887 | output = gjson.GJSON({'key2': 'value2', 'key1': 'value1'}).getj('@pretty:{"sortKeys": true, "prefix": "## "}') 888 | assert output == '## {\n## "key1": "value1",\n## "key2": "value2"\n## }' 889 | 890 | def test_modifier_ugly(self): 891 | """It should uglyfy the JSON string.""" 892 | assert gjson.get(self.obj, '@ugly', as_str=True) == ( 893 | '{"key":"value","hello world":"\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"}') 894 | 895 | def test_output_unicode(self): 896 | """It should return unicode characters as-is.""" 897 | assert gjson.get(self.obj, 'hello world', as_str=True) == '"\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"' 898 | 899 | def test_modifier_ascii(self): 900 | """It should escape all non-ASCII characters.""" 901 | assert gjson.get(self.obj, 'hello world.@ascii', as_str=True) == ( 902 | '"\\u3053\\u3093\\u306b\\u3061\\u306f\\u4e16\\u754c"') 903 | 904 | 905 | def custom_sum(options, obj, *, last): 906 | """Sum items in list. Custom modifier function to be used in tests.""" 907 | assert last is True 908 | assert options == {} 909 | if not isinstance(obj, list): 910 | raise TypeError('@sum can be used only on lists') 911 | 912 | return sum(obj) 913 | 914 | 915 | class TestCustomModifiers: 916 | """Test class for custom modifiers.""" 917 | 918 | def setup_method(self): 919 | """Initialize the test instance.""" 920 | self.valid_obj = [1, 2, 3, 4, 5] 921 | self.invalid_obj = 'invalid' 922 | self.query = '@sum' 923 | 924 | def test_gjson_register_modifier_ok(self): 925 | """It should register a valid modifier.""" 926 | obj = gjson.GJSON(self.valid_obj) 927 | obj.register_modifier('sum', custom_sum) 928 | assert obj.get(self.query) == 15 929 | 930 | def test_gjson_register_modifier_with_escape_ok(self): 931 | """It should register a valid modifier with escaped characters in the name.""" 932 | obj = gjson.GJSON(self.valid_obj) 933 | obj.register_modifier('sum\\=', custom_sum) 934 | assert obj.get('@sum\\=') == 15 935 | 936 | @pytest.mark.parametrize('char', MODIFIER_NAME_RESERVED_CHARS) 937 | def test_gjson_register_modifier_invalid_name(self, char): 938 | """It should raise a GJSONError if trying to register a modifier with a name with not allowed characters.""" 939 | obj = gjson.GJSON(self.valid_obj) 940 | name = fr'a{char}b' 941 | with pytest.raises( 942 | gjson.GJSONError, 943 | match=fr'^Unable to register modifier `{re.escape(name)}`, contains at least one not allowed'): 944 | obj.register_modifier(name, custom_sum) 945 | 946 | def test_gjson_register_modifier_override_builtin(self): 947 | """It should raise a GJSONError if trying to register a modifier with the same name of a built-in one.""" 948 | obj = gjson.GJSON(self.valid_obj) 949 | match = re.escape('Unable to register a modifier with the same name of the built-in modifier: @valid') 950 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'): 951 | obj.register_modifier('valid', custom_sum) 952 | 953 | def test_gjson_register_modifier_not_callable(self): 954 | """It should raise a GJSONError if trying to register a modifier that is not callable.""" 955 | obj = gjson.GJSON(self.valid_obj) 956 | match = re.escape('The given func "not-callable" for the custom modifier @sum does not adhere to the ' 957 | 'gjson.ModifierProtocol.') 958 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'): 959 | obj.register_modifier('sum', 'not-callable') 960 | 961 | def test_gjsonobj_custom_modifiers_ok(self): 962 | """It should register a valid modifier.""" 963 | obj = gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'sum': custom_sum}) 964 | assert obj.get() == 15 965 | 966 | def test_gjsonobj_custom_modifiers_raise(self): 967 | """It should encapsulate the modifier raised exception in a GJSONError.""" 968 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape("Modifier @sum raised an exception")}'): 969 | gjson.GJSONObj(self.invalid_obj, self.query, custom_modifiers={'sum': custom_sum}).get() 970 | 971 | def test_gjsonobj_custom_modifiers_override_builtin(self): 972 | """It should raise a GJSONError if passing custom modifiers that have the same name of a built-in one.""" 973 | with pytest.raises(gjson.GJSONError, 974 | match=r"^Some provided custom_modifiers have the same name of built-in ones: {'valid'}"): 975 | gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'valid': custom_sum}) 976 | 977 | def test_gjsoniobj_custom_modifiers_not_callable(self): 978 | """It should raise a GJSONError if passing custom modifiers that are not callable.""" 979 | match = re.escape('The given func "not-callable" for the custom modifier @sum does not adhere to the') 980 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'): 981 | gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'sum': 'not-callable'}) 982 | 983 | def test_gjsonobj_builtin_modifiers(self): 984 | """It should return a set with the names of the built-in modifiers.""" 985 | expected = {'ascii', 'flatten', 'fromstr', 'group', 'join', 'keys', 'pretty', 'reverse', 'sort', 'sum_n', 986 | 'this', 'top_n', 'tostr', 'valid', 'values', 'ugly'} 987 | assert gjson.GJSONObj.builtin_modifiers() == expected 988 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | minversion = 3.10.0 3 | envlist = py{39,310,311,312}-{ruff,unit,mypy,prospector,sphinx} 4 | skip_missing_interpreters = True 5 | 6 | 7 | [testenv] 8 | usedevelop = True 9 | download = True 10 | allowlist_externals = sed 11 | description = 12 | unit: Run unit tests 13 | bandit: Security-oriented static analyzer 14 | mypy: Static analyzer for type annotations 15 | prospector: Static analysis multi-tool 16 | ruff: Python linter with a lot of other functionalities 17 | sphinx: Build documentation and manpages 18 | py39: (Python 3.9) 19 | py310: (Python 3.10) 20 | py311: (Python 3.11) 21 | py312: (Python 3.12) 22 | commands = 23 | unit: py.test --strict-markers --cov-report=term-missing --cov=gjson tests/unit {posargs} 24 | mypy: mypy --show-error-codes gjson/ 25 | prospector: prospector --profile '{toxinidir}/prospector.yaml' --tool pyroma --tool vulture {posargs} {toxinidir} 26 | ruff: ruff check {posargs} {toxinidir} 27 | sphinx: sphinx-build -W -b html '{toxinidir}/doc/source/' '{toxinidir}/doc/build/html' 28 | sphinx: sphinx-build -W -b man '{toxinidir}/doc/source/' '{toxinidir}/doc/build/man' 29 | # Fix missing space after bold blocks in man page: https://github.com/ribozz/sphinx-argparse/issues/80 30 | sphinx: sed -i='' -e 's/^\.B/.B /' '{toxinidir}/doc/build/man/gjson.1' 31 | deps = 32 | # Use install_requires and the additional extras_require[tests/prospector] from setup.py 33 | prospector: .[prospector] 34 | !prospector: .[tests] 35 | --------------------------------------------------------------------------------