├── .github
└── workflows
│ └── run-tox.yaml
├── .gitignore
├── .readthedocs.yml
├── CHANGELOG.rst
├── LICENSE
├── README.rst
├── doc
├── Makefile
└── source
│ ├── api.rst
│ ├── cli.rst
│ ├── conf.py
│ ├── index.rst
│ ├── introduction.rst
│ └── release.rst
├── gjson
├── __init__.py
├── _cli.py
├── _gjson.py
├── _protocols.py
├── exceptions.py
└── py.typed
├── prospector.yaml
├── ruff.toml
├── setup.cfg
├── setup.py
├── tests
├── ruff.toml
└── unit
│ ├── __init__.py
│ ├── test__cli.py
│ └── test_init.py
└── tox.ini
/.github/workflows/run-tox.yaml:
--------------------------------------------------------------------------------
1 | name: Run tox
2 |
3 | on:
4 | push:
5 | branches:
6 | - main
7 | pull_request:
8 |
9 | jobs:
10 | run-tox:
11 | runs-on: ubuntu-latest
12 | strategy:
13 | matrix:
14 | python: ["3.9", "3.10", "3.11", "3.12"]
15 |
16 | steps:
17 | - uses: actions/checkout@v3
18 | - name: Setup Python
19 | uses: actions/setup-python@v3
20 | with:
21 | python-version: ${{ matrix.python }}
22 | - name: Install tox and any other packages
23 | run: pip install tox
24 | - name: Run tox
25 | run: tox
26 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | # Byte-compiled / optimized / DLL files
2 | __pycache__/
3 | *.py[cod]
4 | *$py.class
5 |
6 | # Distribution / packaging
7 | /build/
8 | /dist/
9 | /.eggs/
10 | /*.egg-info/
11 |
12 | # Unit test / coverage reports
13 | /.tox/
14 | /.coverage
15 | /.coverage.*
16 | /.pytest_cache/
17 |
18 | # Sphinx documentation
19 | /doc/build/
20 |
21 | # pyenv
22 | /.python-version
23 |
24 | # Environments
25 | /.venv/
26 |
27 | # mypy
28 | /.mypy_cache/
29 |
30 | # Vim
31 | *.sw?
32 |
--------------------------------------------------------------------------------
/.readthedocs.yml:
--------------------------------------------------------------------------------
1 | version: 2
2 |
3 | build:
4 | os: "ubuntu-22.04"
5 | tools:
6 | python: "3"
7 |
8 | python:
9 | install:
10 | - method: pip
11 | path: .
12 | extra_requirements:
13 | - tests
14 |
15 | sphinx:
16 | configuration: "doc/source/conf.py"
17 | fail_on_warning: true
18 |
--------------------------------------------------------------------------------
/CHANGELOG.rst:
--------------------------------------------------------------------------------
1 | Release Notes
2 | =============
3 |
4 | `v1.0.0`_ (2023-01-24)
5 | ^^^^^^^^^^^^^^^^^^^^^^
6 |
7 | With this release ``gjson-py`` has reached pretty much feature parity with `GJSON`_, with only some inherent minor
8 | caveats, differences and limitations due to the different programming languages and technologies used. With this
9 | release ``gjson-py`` can also be considered stable and production ready.
10 |
11 | Minor improvements
12 | """"""""""""""""""
13 |
14 | * Modifiers: add support for missing upstream GJSON modifiers: ``@tostr``, ``@fromstr``, ``@group`` and ``@join``.
15 | * gjson: make ``__version__`` more reliable:
16 |
17 | * Migrate the reading of the version from ``pkg_resources`` to ``importlib`` as it's more efficient and avoids
18 | an additional dependency.
19 | * If the version is not detectable, as a fallback, attempt to read the ``SETUPTOOLS_SCM_PRETEND_VERSION`` environment
20 | variable as documented in ``setuptools_scm``.
21 |
22 | Bug fixes
23 | """""""""
24 |
25 | * I/O: allow control characters in all JSON inputs, loading the data with ``strict=False`` both for the JSON input
26 | and when parsing eventual JSON bits present in the query.
27 | * gjson: fix some corner cases:
28 |
29 | * In some corner cases gjson was not parsing correctly the query string.
30 | * Introduce a new ``GJSONInvalidSyntaxError`` exception to distinguish some special parsing errors.
31 | * Extract the check for sequence objects into a private static method.
32 |
33 | Miscellanea
34 | """""""""""
35 |
36 | * setup.py: add ``python_requires`` to prevent the package to be installed in an unsupported version of Python.
37 | * setup.py: update classifiers.
38 | * setup.py: mark the package as typed according to PEP 561.
39 | * setup.py: mark Python 3.11 as officially supported.
40 | * tox.ini: add Python 3.11 in the test matrix of tox.
41 | * documentation: reduce the ``maxdepth`` property for the release notes page to not show all releases to avoid showing
42 | all the releases in the documentation home page.
43 | * documentation: clarify limits on multipaths keys, specifying that in gjson-py if a key of a multipath object is
44 | given, it must be specified as a JSON string (e.g. ``"key":query``) and that bare words (e.g. ``key:query``) are not
45 | accepted although they are in GJSON.
46 |
47 | `v0.4.0`_ (2022-11-12)
48 | ^^^^^^^^^^^^^^^^^^^^^^
49 |
50 | New features
51 | """"""""""""
52 |
53 | * Query parsing: add support for the `GJSON Literals`_ feature.
54 | * Queries: add support for nested queries. There is no limit on the number of levels queries can be nested.
55 |
56 | Bug fixes
57 | """""""""
58 |
59 | * Queries: fix bug on first element query without operator:
60 |
61 | * When performing a query for the first element ``#(...)`` and there is no match, an exception should be raised.
62 | * In the case of a key existence check when there is no operator (e.g. ``friends.#(nonexistent)``) an empty array was
63 | erroneously returned instead of raising an exception.
64 |
65 | Miscellanea
66 | """""""""""
67 |
68 | * documentation: add missing method docstring.
69 | * documentation: add note to modifiers specifying that the ``@keys`` and ``@values`` modifiers are valid only if applied
70 | to a JSON object (mapping).
71 | * Query parsing: simplify internal method to find a matching parentheses for queries and multipaths.
72 |
73 | `v0.3.0`_ (2022-11-10)
74 | ^^^^^^^^^^^^^^^^^^^^^^
75 |
76 | New features
77 | """"""""""""
78 |
79 | * Query parsing: add `GJSON multipaths`_ support.
80 |
81 | Bug fixes
82 | """""""""
83 |
84 | * Query parsing: fix integer mapping keys
85 |
86 | * When after a hash ``#`` or a query that returns all items ``#(...)#``, if there is an integer key, return any
87 | matching key from the resulting items from the query if they match the integer key (as string, as per JSON
88 | specifications).
89 |
90 | `v0.2.1`_ (2022-10-25)
91 | ^^^^^^^^^^^^^^^^^^^^^^
92 |
93 | Bug fixes
94 | """""""""
95 |
96 | * Query parsing: fix modifier options:
97 |
98 | * Fix a bug that was failing to properly get the modifier name when there were options and the modifier was not the
99 | first path in the query.
100 |
101 | `v0.2.0`_ (2022-10-24)
102 | ^^^^^^^^^^^^^^^^^^^^^^
103 |
104 | New features
105 | """"""""""""
106 |
107 | * Query parsing: fully rewrite of the query parser:
108 |
109 | * Until now the parsing was mostly relying on a couple of regular expressions with lookbehind assertions to take
110 | into account escaped characters. Although they were working fine and also allowed to sensibly speed up the first
111 | development of gjson-py, they also had two major limitations:
112 |
113 | * Could not work in all corner cases.
114 | * Prevented the implementation of the GJSON features still missing in gjson-py.
115 |
116 | * The parsing has been completely refactored using a more standard parser approach, that allows to fine-tune the
117 | parsing much more to cover all corner cases and also enables the development of GJSON features still missing.
118 | * There shouldn't be any difference for normal queries, but some corner cases might now return a proper error.
119 | * Introduced a new ``GJSONParseError`` for parser-specific errors, that inherits from GJSONError and also provides
120 | a graphic way to show where the parsing error occurred. Example output::
121 |
122 | GJSONParseError: Invalid or unsupported query part `invalid`.
123 | Query: name.last.invalid
124 | -----------------^
125 |
126 | Minor improvements
127 | """"""""""""""""""
128 |
129 | * Refactor code splitting it into multiple files:
130 |
131 | * Restructure the gjson code to split it into multiple files for ease of development and maintenance.
132 | * Keep all the split modules as private except the exceptions one, and re-export everything from the gjson module
133 | itself, both to keep backward compatibility and also for simplicity of use by the clients.
134 |
135 | * Custom modifiers:
136 |
137 | * Prevent to register custom modifiers with names that contains characters that are used by the GJSON grammair,
138 | raising a GJSONError exception.
139 |
140 | Miscellanea
141 | """""""""""
142 |
143 | * README: clarify naming for nested queries, based on feedback from `issue #2`_. Also fix a typo.
144 |
145 | `v0.1.0`_ (2022-10-03)
146 | ^^^^^^^^^^^^^^^^^^^^^^
147 |
148 | Minor improvements
149 | """"""""""""""""""
150 |
151 | * Modifiers: add ``@top_n`` modifier (not present in GJSON):
152 |
153 | * Add a ``@top_n`` modifier that optionally accepts as options the number of top common items to return:
154 | ``@top_n:{"n": 5}``
155 | * If no options are provided all items are returned.
156 | * It requires a list of items as input and returns a dictionary with unique items as keys and the count of them as
157 | values.
158 |
159 | * Modifiers: add ``@sum_n`` modifier (not present in GJSON):
160 |
161 | * Add a ``@sum_n`` modifier that will work on a sequence of objects, grouping the items with the same value for a
162 | given grouping key and sum the values of a sum key for each of them.
163 | * The options are mandatory and must specify the key to use for grouping and the key to use for summing:
164 | ``{"group": "somekey", "sum": "anotherkey"}``. Optionally specifying the ``n`` parameter to just return the top N
165 | results based on the summed value: ``{"group": "somekey", "sum": "anotherkey", "n": 5}``
166 | * It requires a list of objects as input and returns a dictionary with unique items as keys and the sum of their
167 | values as values.
168 |
169 | Bug fixes
170 | """""""""
171 |
172 | * Output: fix unicode handling:
173 |
174 | * Fix the default behaviour ensuring non-ASCII characters are returned as-is.
175 | * Add a new modifier ``@ascii``, that when set will escape all non-ASCII characters.
176 |
177 | * CLI: fix encoding handling:
178 |
179 | * Use the ``surrogateescape`` Python mode when reading the input and back when printing the output to prevent
180 | failures when parsing the input and reducing the loss of data.
181 |
182 | Miscellanea
183 | """""""""""
184 |
185 | * documentation: add mention to Debian packaging and the availability of Debian packages for the project.
186 | * Type hints: use native types when possible. Instead of importing from ``typing`` use directly the native types when
187 | they support the ``[]`` syntax added in Python 3.9.
188 | * documentation: refactor the modifiers documentation to clearly split the GJSON modifiers supported by gjson-py and
189 | the additional modifiers specific to gjson-py with more detailed explanation and example usage for the additional
190 | ones.
191 | * setup.py: mark project as Beta for this ``v0.1.0`` release and add an additional keyword for PyPI indexing.
192 |
193 | `v0.0.5`_ (2022-08-05)
194 | ^^^^^^^^^^^^^^^^^^^^^^
195 |
196 | New features
197 | """"""""""""
198 |
199 | * Queries: add support for the tilde operator:
200 |
201 | * When performing queries on arrays, add support for the Go GJSON tilde operator to perform truthy-ness comparison.
202 | * The comparison is based on Python's definition of truthy-ness, hence the actual results might differ from the ones
203 | in the Go package.
204 |
205 | Minor improvements
206 | """"""""""""""""""
207 |
208 | * documentation: add man page for the gjson binary.
209 |
210 | `v0.0.4`_ (2022-06-11)
211 | ^^^^^^^^^^^^^^^^^^^^^^
212 |
213 | New features
214 | """"""""""""
215 |
216 | * CLI: improve the JSON Lines support allowing to use the ``-l/--lines`` CLI argument and the special query prefix
217 | ``..`` syntax together to encapsulate each parsed line in an array to enable filtering using the Queries
218 | capabilities.
219 |
220 | Minor improvements
221 | """"""""""""""""""
222 |
223 | * CLI: the input file CLI argument is now optional, defaulting to read from stdin. The equivalent of passing ``-``.
224 | * Modifiers: add support for the upstream Go GJSON modifier ``@this``, that just returns the current object.
225 |
226 | Miscellanea
227 | """""""""""
228 |
229 | * Documentation: add a section to with examples on how to use the CLI.
230 | * CLI: add a link at the bottom of the help message of the CLI to the online documentation.
231 |
232 | `v0.0.3`_ (2022-06-11)
233 | ^^^^^^^^^^^^^^^^^^^^^^
234 |
235 | New features
236 | """"""""""""
237 |
238 | * Add CLI support for JSON Lines:
239 |
240 | * Add a ``-l/--lines`` CLI argument to specify that the input file/stream is made of one JSON per line.
241 | * When used, gjson applies the same query to all lines.
242 | * Based on the verbosity level the failing lines are completely ignored, an error message is printed to stderr or
243 | the execution is interrupted at the first error printing the full traceback.
244 |
245 | * Add CLI support for GJSON JSON Lines queries:
246 |
247 | * Add support for the GJSON queries that encapsulates a JSON Lines input in an array when the query starts with
248 | ``..`` so that they the data can be queries as if it was an array of objects in the CLI.
249 |
250 | * Add support for custom modifiers:
251 |
252 | * Add a ``ModifierProtocol`` to describe the interface that custom modifiers callable need to have.
253 | * Add a ``register_modifier()`` method in the ``GJSON`` class to register custom modifiers.
254 | * Allow to pass a dictionary of modifiers to the low-level ``GJSONObj`` class constructor.
255 | * Add a ``GJSONObj.builtin_modifiers()`` static method that returns a set with the names of the built-in modifiers.
256 | * Is not possible to register a custom modifier with the same name of a built-in modifier.
257 | * Clarify in the documentation that only JSON objects are accepted as modifier arguments.
258 |
259 | Bug fixes
260 | """""""""
261 |
262 | * Query parsing: when using the queries GJSON syntax ``#(...)`` and ``#(...)#`` fix the return value in case of a key
263 | matching that doesn't match any element.
264 |
265 | * Query parsing fixes/improvements found with the Python fuzzing engine Atheris:
266 |
267 | * If any query parts between delimiters is empty error out with a specific message instead of hitting a generic
268 | ``IndexError``.
269 | * When a query has an integer index on a mapping object, in case the element is not present, raise a ``GJSONError``
270 | instead of a ``KeyError`` one.
271 | * When the query has a wildcard matching, ensure that it's applied on a mapping object. Fail with a ``GJSONError``
272 | otherwise.
273 | * Explicitly catch malformed modifier options and raise a ``GJSONError`` instead.
274 | * If the last part of the query is a ``#``, check that the object is actually a sequence like object and fail with
275 | a specific message if not.
276 | * Ensure all the conditions are valid before attempting to extract the inner element of a sequence like object.
277 | Ignore both non-mapping like objects inside the sequence or mapping like objects that don't have the specified key.
278 | * When parsing the query value as JSON catch the eventual decoding error to encapsulate it into a ``GJSONError`` one.
279 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` accept also an empty query to follow the same
280 | behaviour of the upstream Go GJSON.
281 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` follow closely the upstream behaviour of Go GJSON
282 | for all items queries ``#(..)#`` with regex matching.
283 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` fix the wildcard matching regular expression when
284 | using pattern matching.
285 | * Fix the regex to match keys in presence of wildcards escaping only the non-wildcards and ensuring to not
286 | double-escaping any already escaped wildcard.
287 | * When using the queries GJSON syntax ``#(...)`` and ``#(...)#`` ensure any exception raised while comparing
288 | incompatible objects is catched and raise as a GJSONError.
289 |
290 | Miscellanea
291 | """""""""""
292 |
293 | * tests: when matching exception messages always escape the string or use raw strings to avoid false matchings.
294 | * pylint: remove unnecessary comments
295 |
296 | `v0.0.2`_ (2022-05-31)
297 | ^^^^^^^^^^^^^^^^^^^^^^
298 |
299 | Bug fixes
300 | """""""""
301 |
302 | * ``@sort`` modifier: fix the actual sorting.
303 | * tests: ensure that mapping-like objects are compared also in the order of their keys.
304 |
305 | Miscellanea
306 | """""""""""
307 |
308 | * GitHub actions: add workflow to run tox.
309 | * GitHub actions: fix branch name for pushes
310 | * documentation: include also the ``@sort`` modifier that is not present in the GJSON project.
311 | * documentation: fix link to PyPI package.
312 | * documentation: add link to the generated docs.
313 | * documentation: fix section hierarchy and build.
314 |
315 | `v0.0.1`_ (2022-05-22)
316 | ^^^^^^^^^^^^^^^^^^^^^^
317 |
318 | * Initial version.
319 |
320 | .. _`GJSON`: https://github.com/tidwall/gjson/
321 | .. _`GJSON Literals`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#literals
322 | .. _`GJSON Multipaths`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#multipath
323 |
324 | .. _`issue #2`: https://github.com/volans-/gjson-py/issues/2
325 |
326 | .. _`v0.0.1`: https://github.com/volans-/gjson-py/releases/tag/v0.0.1
327 | .. _`v0.0.2`: https://github.com/volans-/gjson-py/releases/tag/v0.0.2
328 | .. _`v0.0.3`: https://github.com/volans-/gjson-py/releases/tag/v0.0.3
329 | .. _`v0.0.4`: https://github.com/volans-/gjson-py/releases/tag/v0.0.4
330 | .. _`v0.0.5`: https://github.com/volans-/gjson-py/releases/tag/v0.0.5
331 | .. _`v0.1.0`: https://github.com/volans-/gjson-py/releases/tag/v0.1.0
332 | .. _`v0.2.0`: https://github.com/volans-/gjson-py/releases/tag/v0.2.0
333 | .. _`v0.2.1`: https://github.com/volans-/gjson-py/releases/tag/v0.2.1
334 | .. _`v0.3.0`: https://github.com/volans-/gjson-py/releases/tag/v0.3.0
335 | .. _`v0.4.0`: https://github.com/volans-/gjson-py/releases/tag/v0.4.0
336 | .. _`v1.0.0`: https://github.com/volans-/gjson-py/releases/tag/v1.0.0
337 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | GNU GENERAL PUBLIC LICENSE
2 | Version 3, 29 June 2007
3 |
4 | Copyright (C) 2007 Free Software Foundation, Inc.
5 | Everyone is permitted to copy and distribute verbatim copies
6 | of this license document, but changing it is not allowed.
7 |
8 | Preamble
9 |
10 | The GNU General Public License is a free, copyleft license for
11 | software and other kinds of works.
12 |
13 | The licenses for most software and other practical works are designed
14 | to take away your freedom to share and change the works. By contrast,
15 | the GNU General Public License is intended to guarantee your freedom to
16 | share and change all versions of a program--to make sure it remains free
17 | software for all its users. We, the Free Software Foundation, use the
18 | GNU General Public License for most of our software; it applies also to
19 | any other work released this way by its authors. You can apply it to
20 | your programs, too.
21 |
22 | When we speak of free software, we are referring to freedom, not
23 | price. Our General Public Licenses are designed to make sure that you
24 | have the freedom to distribute copies of free software (and charge for
25 | them if you wish), that you receive source code or can get it if you
26 | want it, that you can change the software or use pieces of it in new
27 | free programs, and that you know you can do these things.
28 |
29 | To protect your rights, we need to prevent others from denying you
30 | these rights or asking you to surrender the rights. Therefore, you have
31 | certain responsibilities if you distribute copies of the software, or if
32 | you modify it: responsibilities to respect the freedom of others.
33 |
34 | For example, if you distribute copies of such a program, whether
35 | gratis or for a fee, you must pass on to the recipients the same
36 | freedoms that you received. You must make sure that they, too, receive
37 | or can get the source code. And you must show them these terms so they
38 | know their rights.
39 |
40 | Developers that use the GNU GPL protect your rights with two steps:
41 | (1) assert copyright on the software, and (2) offer you this License
42 | giving you legal permission to copy, distribute and/or modify it.
43 |
44 | For the developers' and authors' protection, the GPL clearly explains
45 | that there is no warranty for this free software. For both users' and
46 | authors' sake, the GPL requires that modified versions be marked as
47 | changed, so that their problems will not be attributed erroneously to
48 | authors of previous versions.
49 |
50 | Some devices are designed to deny users access to install or run
51 | modified versions of the software inside them, although the manufacturer
52 | can do so. This is fundamentally incompatible with the aim of
53 | protecting users' freedom to change the software. The systematic
54 | pattern of such abuse occurs in the area of products for individuals to
55 | use, which is precisely where it is most unacceptable. Therefore, we
56 | have designed this version of the GPL to prohibit the practice for those
57 | products. If such problems arise substantially in other domains, we
58 | stand ready to extend this provision to those domains in future versions
59 | of the GPL, as needed to protect the freedom of users.
60 |
61 | Finally, every program is threatened constantly by software patents.
62 | States should not allow patents to restrict development and use of
63 | software on general-purpose computers, but in those that do, we wish to
64 | avoid the special danger that patents applied to a free program could
65 | make it effectively proprietary. To prevent this, the GPL assures that
66 | patents cannot be used to render the program non-free.
67 |
68 | The precise terms and conditions for copying, distribution and
69 | modification follow.
70 |
71 | TERMS AND CONDITIONS
72 |
73 | 0. Definitions.
74 |
75 | "This License" refers to version 3 of the GNU General Public License.
76 |
77 | "Copyright" also means copyright-like laws that apply to other kinds of
78 | works, such as semiconductor masks.
79 |
80 | "The Program" refers to any copyrightable work licensed under this
81 | License. Each licensee is addressed as "you". "Licensees" and
82 | "recipients" may be individuals or organizations.
83 |
84 | To "modify" a work means to copy from or adapt all or part of the work
85 | in a fashion requiring copyright permission, other than the making of an
86 | exact copy. The resulting work is called a "modified version" of the
87 | earlier work or a work "based on" the earlier work.
88 |
89 | A "covered work" means either the unmodified Program or a work based
90 | on the Program.
91 |
92 | To "propagate" a work means to do anything with it that, without
93 | permission, would make you directly or secondarily liable for
94 | infringement under applicable copyright law, except executing it on a
95 | computer or modifying a private copy. Propagation includes copying,
96 | distribution (with or without modification), making available to the
97 | public, and in some countries other activities as well.
98 |
99 | To "convey" a work means any kind of propagation that enables other
100 | parties to make or receive copies. Mere interaction with a user through
101 | a computer network, with no transfer of a copy, is not conveying.
102 |
103 | An interactive user interface displays "Appropriate Legal Notices"
104 | to the extent that it includes a convenient and prominently visible
105 | feature that (1) displays an appropriate copyright notice, and (2)
106 | tells the user that there is no warranty for the work (except to the
107 | extent that warranties are provided), that licensees may convey the
108 | work under this License, and how to view a copy of this License. If
109 | the interface presents a list of user commands or options, such as a
110 | menu, a prominent item in the list meets this criterion.
111 |
112 | 1. Source Code.
113 |
114 | The "source code" for a work means the preferred form of the work
115 | for making modifications to it. "Object code" means any non-source
116 | form of a work.
117 |
118 | A "Standard Interface" means an interface that either is an official
119 | standard defined by a recognized standards body, or, in the case of
120 | interfaces specified for a particular programming language, one that
121 | is widely used among developers working in that language.
122 |
123 | The "System Libraries" of an executable work include anything, other
124 | than the work as a whole, that (a) is included in the normal form of
125 | packaging a Major Component, but which is not part of that Major
126 | Component, and (b) serves only to enable use of the work with that
127 | Major Component, or to implement a Standard Interface for which an
128 | implementation is available to the public in source code form. A
129 | "Major Component", in this context, means a major essential component
130 | (kernel, window system, and so on) of the specific operating system
131 | (if any) on which the executable work runs, or a compiler used to
132 | produce the work, or an object code interpreter used to run it.
133 |
134 | The "Corresponding Source" for a work in object code form means all
135 | the source code needed to generate, install, and (for an executable
136 | work) run the object code and to modify the work, including scripts to
137 | control those activities. However, it does not include the work's
138 | System Libraries, or general-purpose tools or generally available free
139 | programs which are used unmodified in performing those activities but
140 | which are not part of the work. For example, Corresponding Source
141 | includes interface definition files associated with source files for
142 | the work, and the source code for shared libraries and dynamically
143 | linked subprograms that the work is specifically designed to require,
144 | such as by intimate data communication or control flow between those
145 | subprograms and other parts of the work.
146 |
147 | The Corresponding Source need not include anything that users
148 | can regenerate automatically from other parts of the Corresponding
149 | Source.
150 |
151 | The Corresponding Source for a work in source code form is that
152 | same work.
153 |
154 | 2. Basic Permissions.
155 |
156 | All rights granted under this License are granted for the term of
157 | copyright on the Program, and are irrevocable provided the stated
158 | conditions are met. This License explicitly affirms your unlimited
159 | permission to run the unmodified Program. The output from running a
160 | covered work is covered by this License only if the output, given its
161 | content, constitutes a covered work. This License acknowledges your
162 | rights of fair use or other equivalent, as provided by copyright law.
163 |
164 | You may make, run and propagate covered works that you do not
165 | convey, without conditions so long as your license otherwise remains
166 | in force. You may convey covered works to others for the sole purpose
167 | of having them make modifications exclusively for you, or provide you
168 | with facilities for running those works, provided that you comply with
169 | the terms of this License in conveying all material for which you do
170 | not control copyright. Those thus making or running the covered works
171 | for you must do so exclusively on your behalf, under your direction
172 | and control, on terms that prohibit them from making any copies of
173 | your copyrighted material outside their relationship with you.
174 |
175 | Conveying under any other circumstances is permitted solely under
176 | the conditions stated below. Sublicensing is not allowed; section 10
177 | makes it unnecessary.
178 |
179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
180 |
181 | No covered work shall be deemed part of an effective technological
182 | measure under any applicable law fulfilling obligations under article
183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or
184 | similar laws prohibiting or restricting circumvention of such
185 | measures.
186 |
187 | When you convey a covered work, you waive any legal power to forbid
188 | circumvention of technological measures to the extent such circumvention
189 | is effected by exercising rights under this License with respect to
190 | the covered work, and you disclaim any intention to limit operation or
191 | modification of the work as a means of enforcing, against the work's
192 | users, your or third parties' legal rights to forbid circumvention of
193 | technological measures.
194 |
195 | 4. Conveying Verbatim Copies.
196 |
197 | You may convey verbatim copies of the Program's source code as you
198 | receive it, in any medium, provided that you conspicuously and
199 | appropriately publish on each copy an appropriate copyright notice;
200 | keep intact all notices stating that this License and any
201 | non-permissive terms added in accord with section 7 apply to the code;
202 | keep intact all notices of the absence of any warranty; and give all
203 | recipients a copy of this License along with the Program.
204 |
205 | You may charge any price or no price for each copy that you convey,
206 | and you may offer support or warranty protection for a fee.
207 |
208 | 5. Conveying Modified Source Versions.
209 |
210 | You may convey a work based on the Program, or the modifications to
211 | produce it from the Program, in the form of source code under the
212 | terms of section 4, provided that you also meet all of these conditions:
213 |
214 | a) The work must carry prominent notices stating that you modified
215 | it, and giving a relevant date.
216 |
217 | b) The work must carry prominent notices stating that it is
218 | released under this License and any conditions added under section
219 | 7. This requirement modifies the requirement in section 4 to
220 | "keep intact all notices".
221 |
222 | c) You must license the entire work, as a whole, under this
223 | License to anyone who comes into possession of a copy. This
224 | License will therefore apply, along with any applicable section 7
225 | additional terms, to the whole of the work, and all its parts,
226 | regardless of how they are packaged. This License gives no
227 | permission to license the work in any other way, but it does not
228 | invalidate such permission if you have separately received it.
229 |
230 | d) If the work has interactive user interfaces, each must display
231 | Appropriate Legal Notices; however, if the Program has interactive
232 | interfaces that do not display Appropriate Legal Notices, your
233 | work need not make them do so.
234 |
235 | A compilation of a covered work with other separate and independent
236 | works, which are not by their nature extensions of the covered work,
237 | and which are not combined with it such as to form a larger program,
238 | in or on a volume of a storage or distribution medium, is called an
239 | "aggregate" if the compilation and its resulting copyright are not
240 | used to limit the access or legal rights of the compilation's users
241 | beyond what the individual works permit. Inclusion of a covered work
242 | in an aggregate does not cause this License to apply to the other
243 | parts of the aggregate.
244 |
245 | 6. Conveying Non-Source Forms.
246 |
247 | You may convey a covered work in object code form under the terms
248 | of sections 4 and 5, provided that you also convey the
249 | machine-readable Corresponding Source under the terms of this License,
250 | in one of these ways:
251 |
252 | a) Convey the object code in, or embodied in, a physical product
253 | (including a physical distribution medium), accompanied by the
254 | Corresponding Source fixed on a durable physical medium
255 | customarily used for software interchange.
256 |
257 | b) Convey the object code in, or embodied in, a physical product
258 | (including a physical distribution medium), accompanied by a
259 | written offer, valid for at least three years and valid for as
260 | long as you offer spare parts or customer support for that product
261 | model, to give anyone who possesses the object code either (1) a
262 | copy of the Corresponding Source for all the software in the
263 | product that is covered by this License, on a durable physical
264 | medium customarily used for software interchange, for a price no
265 | more than your reasonable cost of physically performing this
266 | conveying of source, or (2) access to copy the
267 | Corresponding Source from a network server at no charge.
268 |
269 | c) Convey individual copies of the object code with a copy of the
270 | written offer to provide the Corresponding Source. This
271 | alternative is allowed only occasionally and noncommercially, and
272 | only if you received the object code with such an offer, in accord
273 | with subsection 6b.
274 |
275 | d) Convey the object code by offering access from a designated
276 | place (gratis or for a charge), and offer equivalent access to the
277 | Corresponding Source in the same way through the same place at no
278 | further charge. You need not require recipients to copy the
279 | Corresponding Source along with the object code. If the place to
280 | copy the object code is a network server, the Corresponding Source
281 | may be on a different server (operated by you or a third party)
282 | that supports equivalent copying facilities, provided you maintain
283 | clear directions next to the object code saying where to find the
284 | Corresponding Source. Regardless of what server hosts the
285 | Corresponding Source, you remain obligated to ensure that it is
286 | available for as long as needed to satisfy these requirements.
287 |
288 | e) Convey the object code using peer-to-peer transmission, provided
289 | you inform other peers where the object code and Corresponding
290 | Source of the work are being offered to the general public at no
291 | charge under subsection 6d.
292 |
293 | A separable portion of the object code, whose source code is excluded
294 | from the Corresponding Source as a System Library, need not be
295 | included in conveying the object code work.
296 |
297 | A "User Product" is either (1) a "consumer product", which means any
298 | tangible personal property which is normally used for personal, family,
299 | or household purposes, or (2) anything designed or sold for incorporation
300 | into a dwelling. In determining whether a product is a consumer product,
301 | doubtful cases shall be resolved in favor of coverage. For a particular
302 | product received by a particular user, "normally used" refers to a
303 | typical or common use of that class of product, regardless of the status
304 | of the particular user or of the way in which the particular user
305 | actually uses, or expects or is expected to use, the product. A product
306 | is a consumer product regardless of whether the product has substantial
307 | commercial, industrial or non-consumer uses, unless such uses represent
308 | the only significant mode of use of the product.
309 |
310 | "Installation Information" for a User Product means any methods,
311 | procedures, authorization keys, or other information required to install
312 | and execute modified versions of a covered work in that User Product from
313 | a modified version of its Corresponding Source. The information must
314 | suffice to ensure that the continued functioning of the modified object
315 | code is in no case prevented or interfered with solely because
316 | modification has been made.
317 |
318 | If you convey an object code work under this section in, or with, or
319 | specifically for use in, a User Product, and the conveying occurs as
320 | part of a transaction in which the right of possession and use of the
321 | User Product is transferred to the recipient in perpetuity or for a
322 | fixed term (regardless of how the transaction is characterized), the
323 | Corresponding Source conveyed under this section must be accompanied
324 | by the Installation Information. But this requirement does not apply
325 | if neither you nor any third party retains the ability to install
326 | modified object code on the User Product (for example, the work has
327 | been installed in ROM).
328 |
329 | The requirement to provide Installation Information does not include a
330 | requirement to continue to provide support service, warranty, or updates
331 | for a work that has been modified or installed by the recipient, or for
332 | the User Product in which it has been modified or installed. Access to a
333 | network may be denied when the modification itself materially and
334 | adversely affects the operation of the network or violates the rules and
335 | protocols for communication across the network.
336 |
337 | Corresponding Source conveyed, and Installation Information provided,
338 | in accord with this section must be in a format that is publicly
339 | documented (and with an implementation available to the public in
340 | source code form), and must require no special password or key for
341 | unpacking, reading or copying.
342 |
343 | 7. Additional Terms.
344 |
345 | "Additional permissions" are terms that supplement the terms of this
346 | License by making exceptions from one or more of its conditions.
347 | Additional permissions that are applicable to the entire Program shall
348 | be treated as though they were included in this License, to the extent
349 | that they are valid under applicable law. If additional permissions
350 | apply only to part of the Program, that part may be used separately
351 | under those permissions, but the entire Program remains governed by
352 | this License without regard to the additional permissions.
353 |
354 | When you convey a copy of a covered work, you may at your option
355 | remove any additional permissions from that copy, or from any part of
356 | it. (Additional permissions may be written to require their own
357 | removal in certain cases when you modify the work.) You may place
358 | additional permissions on material, added by you to a covered work,
359 | for which you have or can give appropriate copyright permission.
360 |
361 | Notwithstanding any other provision of this License, for material you
362 | add to a covered work, you may (if authorized by the copyright holders of
363 | that material) supplement the terms of this License with terms:
364 |
365 | a) Disclaiming warranty or limiting liability differently from the
366 | terms of sections 15 and 16 of this License; or
367 |
368 | b) Requiring preservation of specified reasonable legal notices or
369 | author attributions in that material or in the Appropriate Legal
370 | Notices displayed by works containing it; or
371 |
372 | c) Prohibiting misrepresentation of the origin of that material, or
373 | requiring that modified versions of such material be marked in
374 | reasonable ways as different from the original version; or
375 |
376 | d) Limiting the use for publicity purposes of names of licensors or
377 | authors of the material; or
378 |
379 | e) Declining to grant rights under trademark law for use of some
380 | trade names, trademarks, or service marks; or
381 |
382 | f) Requiring indemnification of licensors and authors of that
383 | material by anyone who conveys the material (or modified versions of
384 | it) with contractual assumptions of liability to the recipient, for
385 | any liability that these contractual assumptions directly impose on
386 | those licensors and authors.
387 |
388 | All other non-permissive additional terms are considered "further
389 | restrictions" within the meaning of section 10. If the Program as you
390 | received it, or any part of it, contains a notice stating that it is
391 | governed by this License along with a term that is a further
392 | restriction, you may remove that term. If a license document contains
393 | a further restriction but permits relicensing or conveying under this
394 | License, you may add to a covered work material governed by the terms
395 | of that license document, provided that the further restriction does
396 | not survive such relicensing or conveying.
397 |
398 | If you add terms to a covered work in accord with this section, you
399 | must place, in the relevant source files, a statement of the
400 | additional terms that apply to those files, or a notice indicating
401 | where to find the applicable terms.
402 |
403 | Additional terms, permissive or non-permissive, may be stated in the
404 | form of a separately written license, or stated as exceptions;
405 | the above requirements apply either way.
406 |
407 | 8. Termination.
408 |
409 | You may not propagate or modify a covered work except as expressly
410 | provided under this License. Any attempt otherwise to propagate or
411 | modify it is void, and will automatically terminate your rights under
412 | this License (including any patent licenses granted under the third
413 | paragraph of section 11).
414 |
415 | However, if you cease all violation of this License, then your
416 | license from a particular copyright holder is reinstated (a)
417 | provisionally, unless and until the copyright holder explicitly and
418 | finally terminates your license, and (b) permanently, if the copyright
419 | holder fails to notify you of the violation by some reasonable means
420 | prior to 60 days after the cessation.
421 |
422 | Moreover, your license from a particular copyright holder is
423 | reinstated permanently if the copyright holder notifies you of the
424 | violation by some reasonable means, this is the first time you have
425 | received notice of violation of this License (for any work) from that
426 | copyright holder, and you cure the violation prior to 30 days after
427 | your receipt of the notice.
428 |
429 | Termination of your rights under this section does not terminate the
430 | licenses of parties who have received copies or rights from you under
431 | this License. If your rights have been terminated and not permanently
432 | reinstated, you do not qualify to receive new licenses for the same
433 | material under section 10.
434 |
435 | 9. Acceptance Not Required for Having Copies.
436 |
437 | You are not required to accept this License in order to receive or
438 | run a copy of the Program. Ancillary propagation of a covered work
439 | occurring solely as a consequence of using peer-to-peer transmission
440 | to receive a copy likewise does not require acceptance. However,
441 | nothing other than this License grants you permission to propagate or
442 | modify any covered work. These actions infringe copyright if you do
443 | not accept this License. Therefore, by modifying or propagating a
444 | covered work, you indicate your acceptance of this License to do so.
445 |
446 | 10. Automatic Licensing of Downstream Recipients.
447 |
448 | Each time you convey a covered work, the recipient automatically
449 | receives a license from the original licensors, to run, modify and
450 | propagate that work, subject to this License. You are not responsible
451 | for enforcing compliance by third parties with this License.
452 |
453 | An "entity transaction" is a transaction transferring control of an
454 | organization, or substantially all assets of one, or subdividing an
455 | organization, or merging organizations. If propagation of a covered
456 | work results from an entity transaction, each party to that
457 | transaction who receives a copy of the work also receives whatever
458 | licenses to the work the party's predecessor in interest had or could
459 | give under the previous paragraph, plus a right to possession of the
460 | Corresponding Source of the work from the predecessor in interest, if
461 | the predecessor has it or can get it with reasonable efforts.
462 |
463 | You may not impose any further restrictions on the exercise of the
464 | rights granted or affirmed under this License. For example, you may
465 | not impose a license fee, royalty, or other charge for exercise of
466 | rights granted under this License, and you may not initiate litigation
467 | (including a cross-claim or counterclaim in a lawsuit) alleging that
468 | any patent claim is infringed by making, using, selling, offering for
469 | sale, or importing the Program or any portion of it.
470 |
471 | 11. Patents.
472 |
473 | A "contributor" is a copyright holder who authorizes use under this
474 | License of the Program or a work on which the Program is based. The
475 | work thus licensed is called the contributor's "contributor version".
476 |
477 | A contributor's "essential patent claims" are all patent claims
478 | owned or controlled by the contributor, whether already acquired or
479 | hereafter acquired, that would be infringed by some manner, permitted
480 | by this License, of making, using, or selling its contributor version,
481 | but do not include claims that would be infringed only as a
482 | consequence of further modification of the contributor version. For
483 | purposes of this definition, "control" includes the right to grant
484 | patent sublicenses in a manner consistent with the requirements of
485 | this License.
486 |
487 | Each contributor grants you a non-exclusive, worldwide, royalty-free
488 | patent license under the contributor's essential patent claims, to
489 | make, use, sell, offer for sale, import and otherwise run, modify and
490 | propagate the contents of its contributor version.
491 |
492 | In the following three paragraphs, a "patent license" is any express
493 | agreement or commitment, however denominated, not to enforce a patent
494 | (such as an express permission to practice a patent or covenant not to
495 | sue for patent infringement). To "grant" such a patent license to a
496 | party means to make such an agreement or commitment not to enforce a
497 | patent against the party.
498 |
499 | If you convey a covered work, knowingly relying on a patent license,
500 | and the Corresponding Source of the work is not available for anyone
501 | to copy, free of charge and under the terms of this License, through a
502 | publicly available network server or other readily accessible means,
503 | then you must either (1) cause the Corresponding Source to be so
504 | available, or (2) arrange to deprive yourself of the benefit of the
505 | patent license for this particular work, or (3) arrange, in a manner
506 | consistent with the requirements of this License, to extend the patent
507 | license to downstream recipients. "Knowingly relying" means you have
508 | actual knowledge that, but for the patent license, your conveying the
509 | covered work in a country, or your recipient's use of the covered work
510 | in a country, would infringe one or more identifiable patents in that
511 | country that you have reason to believe are valid.
512 |
513 | If, pursuant to or in connection with a single transaction or
514 | arrangement, you convey, or propagate by procuring conveyance of, a
515 | covered work, and grant a patent license to some of the parties
516 | receiving the covered work authorizing them to use, propagate, modify
517 | or convey a specific copy of the covered work, then the patent license
518 | you grant is automatically extended to all recipients of the covered
519 | work and works based on it.
520 |
521 | A patent license is "discriminatory" if it does not include within
522 | the scope of its coverage, prohibits the exercise of, or is
523 | conditioned on the non-exercise of one or more of the rights that are
524 | specifically granted under this License. You may not convey a covered
525 | work if you are a party to an arrangement with a third party that is
526 | in the business of distributing software, under which you make payment
527 | to the third party based on the extent of your activity of conveying
528 | the work, and under which the third party grants, to any of the
529 | parties who would receive the covered work from you, a discriminatory
530 | patent license (a) in connection with copies of the covered work
531 | conveyed by you (or copies made from those copies), or (b) primarily
532 | for and in connection with specific products or compilations that
533 | contain the covered work, unless you entered into that arrangement,
534 | or that patent license was granted, prior to 28 March 2007.
535 |
536 | Nothing in this License shall be construed as excluding or limiting
537 | any implied license or other defenses to infringement that may
538 | otherwise be available to you under applicable patent law.
539 |
540 | 12. No Surrender of Others' Freedom.
541 |
542 | If conditions are imposed on you (whether by court order, agreement or
543 | otherwise) that contradict the conditions of this License, they do not
544 | excuse you from the conditions of this License. If you cannot convey a
545 | covered work so as to satisfy simultaneously your obligations under this
546 | License and any other pertinent obligations, then as a consequence you may
547 | not convey it at all. For example, if you agree to terms that obligate you
548 | to collect a royalty for further conveying from those to whom you convey
549 | the Program, the only way you could satisfy both those terms and this
550 | License would be to refrain entirely from conveying the Program.
551 |
552 | 13. Use with the GNU Affero General Public License.
553 |
554 | Notwithstanding any other provision of this License, you have
555 | permission to link or combine any covered work with a work licensed
556 | under version 3 of the GNU Affero General Public License into a single
557 | combined work, and to convey the resulting work. The terms of this
558 | License will continue to apply to the part which is the covered work,
559 | but the special requirements of the GNU Affero General Public License,
560 | section 13, concerning interaction through a network will apply to the
561 | combination as such.
562 |
563 | 14. Revised Versions of this License.
564 |
565 | The Free Software Foundation may publish revised and/or new versions of
566 | the GNU General Public License from time to time. Such new versions will
567 | be similar in spirit to the present version, but may differ in detail to
568 | address new problems or concerns.
569 |
570 | Each version is given a distinguishing version number. If the
571 | Program specifies that a certain numbered version of the GNU General
572 | Public License "or any later version" applies to it, you have the
573 | option of following the terms and conditions either of that numbered
574 | version or of any later version published by the Free Software
575 | Foundation. If the Program does not specify a version number of the
576 | GNU General Public License, you may choose any version ever published
577 | by the Free Software Foundation.
578 |
579 | If the Program specifies that a proxy can decide which future
580 | versions of the GNU General Public License can be used, that proxy's
581 | public statement of acceptance of a version permanently authorizes you
582 | to choose that version for the Program.
583 |
584 | Later license versions may give you additional or different
585 | permissions. However, no additional obligations are imposed on any
586 | author or copyright holder as a result of your choosing to follow a
587 | later version.
588 |
589 | 15. Disclaimer of Warranty.
590 |
591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
599 |
600 | 16. Limitation of Liability.
601 |
602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
610 | SUCH DAMAGES.
611 |
612 | 17. Interpretation of Sections 15 and 16.
613 |
614 | If the disclaimer of warranty and limitation of liability provided
615 | above cannot be given local legal effect according to their terms,
616 | reviewing courts shall apply local law that most closely approximates
617 | an absolute waiver of all civil liability in connection with the
618 | Program, unless a warranty or assumption of liability accompanies a
619 | copy of the Program in return for a fee.
620 |
621 | END OF TERMS AND CONDITIONS
622 |
623 | How to Apply These Terms to Your New Programs
624 |
625 | If you develop a new program, and you want it to be of the greatest
626 | possible use to the public, the best way to achieve this is to make it
627 | free software which everyone can redistribute and change under these terms.
628 |
629 | To do so, attach the following notices to the program. It is safest
630 | to attach them to the start of each source file to most effectively
631 | state the exclusion of warranty; and each file should have at least
632 | the "copyright" line and a pointer to where the full notice is found.
633 |
634 |
635 | Copyright (C)
636 |
637 | This program is free software: you can redistribute it and/or modify
638 | it under the terms of the GNU General Public License as published by
639 | the Free Software Foundation, either version 3 of the License, or
640 | (at your option) any later version.
641 |
642 | This program is distributed in the hope that it will be useful,
643 | but WITHOUT ANY WARRANTY; without even the implied warranty of
644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
645 | GNU General Public License for more details.
646 |
647 | You should have received a copy of the GNU General Public License
648 | along with this program. If not, see .
649 |
650 | Also add information on how to contact you by electronic and paper mail.
651 |
652 | If the program does terminal interaction, make it output a short
653 | notice like this when it starts in an interactive mode:
654 |
655 | Copyright (C)
656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657 | This is free software, and you are welcome to redistribute it
658 | under certain conditions; type `show c' for details.
659 |
660 | The hypothetical commands `show w' and `show c' should show the appropriate
661 | parts of the General Public License. Of course, your program's commands
662 | might be different; for a GUI interface, you would use an "about box".
663 |
664 | You should also get your employer (if you work as a programmer) or school,
665 | if any, to sign a "copyright disclaimer" for the program, if necessary.
666 | For more information on this, and how to apply and follow the GNU GPL, see
667 | .
668 |
669 | The GNU General Public License does not permit incorporating your program
670 | into proprietary programs. If your program is a subroutine library, you
671 | may consider it more useful to permit linking proprietary applications with
672 | the library. If this is what you want to do, use the GNU Lesser General
673 | Public License instead of this License. But first, please read
674 | .
675 |
--------------------------------------------------------------------------------
/README.rst:
--------------------------------------------------------------------------------
1 | .. image:: https://github.com/volans-/gjson-py/actions/workflows/run-tox.yaml/badge.svg
2 | :alt: CI results
3 | :target: https://github.com/volans-/gjson-py/actions/workflows/run-tox.yaml
4 |
5 | Introduction
6 | ============
7 |
8 | gjson-py is a Python package that provides a simple way to filter and extract data from JSON-like objects or JSON
9 | files, using the `GJSON`_ syntax.
10 |
11 | It is, compatibly with the language differences and with some limitation, the Python equivalent of the Go
12 | `GJSON`_ package.
13 | The main difference from GJSON is that gjson-py doesn't work directly with JSON strings but instead with
14 | JSON-like Python objects, that can either be the resulting object when calling ``json.load()`` or ``json.loads()``,
15 | or any Python object that is JSON-serializable.
16 |
17 | A detailed list of the GJSON features supported by gjson-py is provided below.
18 |
19 | See also the full `gjson-py documentation`_.
20 |
21 | Installation
22 | ------------
23 |
24 | gjson-py is available on the `Python Package Index`_ (PyPI) and can be easily installed with::
25 |
26 | pip install gjson
27 |
28 | It's also available as a Debian package (`python3-gjson`_) on Debian systems starting from Debian 12 (*bookworm*) and
29 | can be installed with::
30 |
31 | apt-get install python3-gjson
32 |
33 | A ``.deb`` package for the current stable and unstable Debian versions is also available for download on the
34 | `releases page on GitHub`_.
35 |
36 | How to use the library
37 | ----------------------
38 |
39 | gjson-py provides different ways to perform queries on JSON-like objects.
40 |
41 | ``gjson.get()``
42 | ^^^^^^^^^^^^^^^
43 |
44 | A quick accessor to GJSON functionalities exposed for simplicity of use. Particularly useful to perform a single
45 | query on a given object::
46 |
47 | >>> import gjson
48 | >>> data = {'name': {'first': 'Tom', 'last': 'Anderson'}, 'age': 37}
49 | >>> gjson.get(data, 'name.first')
50 | 'Tom'
51 |
52 | It's also possible to make it return a JSON-encoded string and decide on failure if it should raise an exception
53 | or return `None`. See the full API documentation for more details.
54 |
55 | ``GJSON`` class
56 | ^^^^^^^^^^^^^^^
57 |
58 | The ``GJSON`` class provides full access to the gjson-py API allowing to perform multiple queries on the same object::
59 |
60 | >>> import gjson
61 | >>> data = {'name': {'first': 'Tom', 'last': 'Anderson'}, 'age': 37}
62 | >>> source = gjson.GJSON(data)
63 | >>> source.get('name.first')
64 | 'Tom'
65 | >>> str(source)
66 | '{"name": {"first": "Tom", "last": "Anderson"}, "age": 37}'
67 | >>> source.getj('name.first')
68 | '"Tom"'
69 | >>> name = source.get_gjson('name')
70 | >>> name.get('first')
71 | 'Tom'
72 | >>> name
73 |
74 |
75 | See the full API documentation for more details.
76 |
77 | How to use the CLI
78 | ------------------
79 |
80 | gjson-py provides also a command line interface (CLI) for ease of use:
81 |
82 | .. code-block:: console
83 |
84 | $ echo '{"name": {"first": "Tom", "last": "Anderson"}, "age": 37}' > test.json
85 | $ cat test.json | gjson 'name.first' # Read from stdin
86 | "Tom"
87 | $ gjson test.json 'age' # Read from a file
88 | 37
89 | $ cat test.json | gjson - 'name.first' # Explicitely read from stdin
90 | "Tom"
91 |
92 | JSON Lines
93 | ^^^^^^^^^^
94 |
95 | JSON Lines support in the CLI allows for different use cases. All the examples in this section operates on a
96 | ``test.json`` file generated with:
97 |
98 | .. code-block:: console
99 |
100 | $ echo -e '{"name": "Gilbert", "age": 61}\n{"name": "Alexa", "age": 34}\n{"name": "May", "age": 57}' > test.json
101 |
102 | Apply the same query to each line
103 | """""""""""""""""""""""""""""""""
104 |
105 | Using the ``-l/--lines`` CLI argument, for each input line gjson-py applies the query and filters the data according
106 | to it. Lines are read one by one so there is no memory overhead for the processing. It can be used while tailing log
107 | files in JSON format for example.
108 |
109 |
110 | .. code-block:: console
111 |
112 | $ gjson --lines test.json 'age'
113 | 61
114 | 34
115 | 57
116 | $ tail -f log.json | gjson --lines 'bytes_sent' # Dummy example
117 |
118 | Encapsulate all lines in an array, then apply the query
119 | """""""""""""""""""""""""""""""""""""""""""""""""""""""
120 |
121 | Using the special query prefix syntax ``..``, as described in GJSON's documentation for `JSON Lines`_, gjson-py will
122 | read all lines from the input and encapsulate them into an array. This approach has of course the memory overhead of
123 | loading the whole input to perform the query.
124 |
125 | .. code-block:: console
126 |
127 | $ gjson test.json '..#.name'
128 | ["Gilbert", "Alexa", "May"]
129 |
130 | Filter lines based on their values
131 | """"""""""""""""""""""""""""""""""
132 |
133 | Combining the ``-l/--lines`` CLI argument with the special query prefix ``..`` described above, it's possible to filter
134 | input lines based on their values. In this case gjson-py encapsulates each line in an array so that is possible to use
135 | the `Queries`_ GJSON syntax to filter them. As the ecapsulation is performed on each line, there is no memory overhead.
136 | Because technically when a line is filtered is because there was no match on the whole line query, the final exit code,
137 | if any line is filtered, will be ``1``.
138 |
139 | .. code-block:: console
140 |
141 | $ gjson --lines test.json '..#(age>40).name'
142 | "Gilbert"
143 | "May"
144 |
145 | Filter lines and apply query to the result
146 | """"""""""""""""""""""""""""""""""""""""""
147 |
148 | Combining the methods above is possible for example to filter/extract data from the lines first and then apply a query
149 | to the aggregated result. The memory overhead in this case is based on the amount of data resulting from the first
150 | filtering/extraction.
151 |
152 | .. code-block:: console
153 |
154 | $ gjson --lines test.json 'age' | gjson '..@sort'
155 | [34, 57, 61]
156 | $ gjson --lines test.json '..#(age>40).age' | gjson '..@sort'
157 | [57, 61]
158 |
159 | Query syntax
160 | ------------
161 |
162 | For the generic query syntax refer to the original `GJSON Path Syntax`_ documentation.
163 |
164 | Supported GJSON features
165 | ^^^^^^^^^^^^^^^^^^^^^^^^
166 |
167 | This is the list of GJSON features and how they are supported by gjson-py:
168 |
169 |
170 | +------------------------+------------------------+------------------------------------------------------+
171 | | GJSON feature | Supported by gjson-py | Notes |
172 | +========================+========================+======================================================+
173 | | `Path Structure`_ | YES | |
174 | +------------------------+------------------------+------------------------------------------------------+
175 | | `Basic`_ | YES | |
176 | +------------------------+------------------------+------------------------------------------------------+
177 | | `Wildcards`_ | YES | |
178 | +------------------------+------------------------+------------------------------------------------------+
179 | | `Escape Character`_ | YES | |
180 | +------------------------+------------------------+------------------------------------------------------+
181 | | `Arrays`_ | YES | |
182 | +------------------------+------------------------+------------------------------------------------------+
183 | | `Queries`_ | YES | Using Python's operators [#]_ [#]_ |
184 | +------------------------+------------------------+------------------------------------------------------+
185 | | `Dot vs Pipe`_ | YES | |
186 | +------------------------+------------------------+------------------------------------------------------+
187 | | `Modifiers`_ | YES | See the table below for all the details |
188 | +------------------------+------------------------+------------------------------------------------------+
189 | | `Modifier arguments`_ | YES | Only a JSON object is accepted as argument |
190 | +------------------------+------------------------+------------------------------------------------------+
191 | | `Custom modifiers`_ | YES | Only a JSON object is accepted as argument [#]_ |
192 | +------------------------+------------------------+------------------------------------------------------+
193 | | `Multipaths`_ | YES | Object keys, if specified, must be JSON strings [#]_ |
194 | +------------------------+------------------------+------------------------------------------------------+
195 | | `Literals`_ | YES | Including infinite and NaN values [#]_ |
196 | +------------------------+------------------------+------------------------------------------------------+
197 | | `JSON Lines`_ | YES | CLI support [#]_ [#]_ |
198 | +------------------------+------------------------+------------------------------------------------------+
199 |
200 | .. [#] The queries matching is based on Python's operator and as such the results might be different than the ones from
201 | the Go GJSON package. In particular for the ``~`` operator that checks the truthy-ness of objects.
202 | .. [#] When using nested queries, only the outermost one controls whether to return only the first item or all items.
203 | .. [#] Custom modifiers names cannot contain reserved characters used by the GJSON grammar.
204 | .. [#] For example ``{"years":age}`` is valid while ``{years:age}`` is not, although that's valid in GJSON.
205 | .. [#] Those special cases are handled according to `Python's JSON documentation`_.
206 | .. [#] Both for applying the same query to each line using the ``-l/--lines`` argument and to automatically encapsulate
207 | the input lines in a list and apply the query to the list using the ``..`` special query prefix described in
208 | `JSON Lines`_.
209 | .. [#] Library support is not currently present because gjson-py accepts only Python objects, making it impossible to
210 | pass JSON Lines directly. The client is free to choose if calling gjson-py for each line or to encapsulate them in
211 | a list before calling gjson-py.
212 |
213 | This is the list of modifiers present in GJSON and how they are supported by gjson-py:
214 |
215 | +----------------+-----------------------+------------------------------------------+
216 | | GJSON Modifier | Supported by gjson-py | Notes |
217 | +----------------+-----------------------+------------------------------------------+
218 | | ``@reverse`` | YES | |
219 | +----------------+-----------------------+------------------------------------------+
220 | | ``@ugly`` | YES | |
221 | +----------------+-----------------------+------------------------------------------+
222 | | ``@pretty`` | PARTIALLY | The ``width`` argument is not supported |
223 | +----------------+-----------------------+------------------------------------------+
224 | | ``@this`` | YES | |
225 | +----------------+-----------------------+------------------------------------------+
226 | | ``@valid`` | YES | |
227 | +----------------+-----------------------+------------------------------------------+
228 | | ``@flatten`` | YES | |
229 | +----------------+-----------------------+------------------------------------------+
230 | | ``@join`` | PARTIALLY | Preserving duplicate keys not supported |
231 | +----------------+-----------------------+------------------------------------------+
232 | | ``@keys`` | YES | Valid only on JSON objects (mappings) |
233 | +----------------+-----------------------+------------------------------------------+
234 | | ``@values`` | YES | Valid only on JSON objects (mappings) |
235 | +----------------+-----------------------+------------------------------------------+
236 | | ``@tostr`` | YES | |
237 | +----------------+-----------------------+------------------------------------------+
238 | | ``@fromstr`` | YES | |
239 | +----------------+-----------------------+------------------------------------------+
240 | | ``@group`` | YES | |
241 | +----------------+-----------------------+------------------------------------------+
242 |
243 |
244 | Additional features
245 | ^^^^^^^^^^^^^^^^^^^
246 |
247 |
248 | Additional modifiers
249 | """"""""""""""""""""
250 |
251 | This is the list of additional modifiers specific to gjson-py not present in GJSON:
252 |
253 | * ``@ascii``: escapes all non-ASCII characters when printing/returning the string representation of the object,
254 | ensuring that the output is made only of ASCII characters. It's implemented using the ``ensure_ascii`` arguments in
255 | the Python's ``json`` module. This modifier doesn't accept any arguments.
256 | * ``@sort``: sorts a mapping object by its keys or a sequence object by its values. This modifier doesn't accept any
257 | arguments.
258 | * ``@top_n``: given a sequence object groups the items in the sequence counting how many occurrences of each value are
259 | present. It returns a mapping object where the keys are the distinct values of the list and the values are the number
260 | of times the key was present in the list, ordered from the most common to the least common item. The items in the
261 | original sequence object must be Python hashable. This modifier accepts an optional argument ``n`` to return just the
262 | N items with the higher counts. When the ``n`` argument is not provided all items are returned. Example usage:
263 |
264 | .. code-block:: console
265 |
266 | $ echo '["a", "b", "c", "b", "c", "c"]' | gjson '@top_n'
267 | {"c": 3, "b": 2, "a": 1}
268 | $ echo '["a", "b", "c", "b", "c", "c"]' | gjson '@top_n:{"n":2}'
269 | {"c": 3, "b": 2}
270 |
271 | * ``@sum_n``: given a sequence of objects, groups the items in the sequence using a grouping key and sum the values of a
272 | sum key provided. It returns a mapping object where the keys are the distinct values of the grouping key and the
273 | values are the sums of all the values of the sum key for each distinct grouped key, ordered from the highest sum to
274 | the lowest. The values of the grouping key must be Python hashable. The values of the sum key must be integers or
275 | floats. This modifier required two mandatory arguments, ``group`` and ``sum`` that have as values the respective keys
276 | in the objects of the sequence. An optional ``n`` argument is also accepted to return just the top N items with the
277 | highest sum. Example usage:
278 |
279 | .. code-block:: console
280 |
281 | $ echo '[{"key": "a", "time": 1}, {"key": "b", "time": 2}, {"key": "c", "time": 3}, {"key": "a", "time": 4}]' > test.json
282 | $ gjson test.json '@sum_n:{"group": "key", "sum": "time"}'
283 | {"a": 5, "c": 3, "b": 2}
284 | $ gjson test.json '@sum_n:{"group": "key", "sum": "time", "n": 2}'
285 | {"a": 5, "c": 3}
286 |
287 | .. _`GJSON`: https://github.com/tidwall/gjson
288 | .. _`Python Package Index`: https://pypi.org/project/gjson/
289 | .. _`GJSON Path Syntax`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md
290 | .. _`gjson-py documentation`: https://volans-.github.io/gjson-py/index.html
291 | .. _`releases page on GitHub`: https://github.com/volans-/gjson-py/releases
292 | .. _`Python's JSON documentation`: https://docs.python.org/3/library/json.html#infinite-and-nan-number-values
293 | .. _`python3-gjson`: https://packages.debian.org/sid/python3-gjson
294 |
295 | .. _`Path Structure`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#path-structure
296 | .. _`Basic`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#basic
297 | .. _`Wildcards`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#wildcards
298 | .. _`Escape Character`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#escape-character
299 | .. _`Arrays`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#arrays
300 | .. _`Queries`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#queries
301 | .. _`Dot vs Pipe`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#dot-vs-pipe
302 | .. _`Modifiers`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#modifiers
303 | .. _`Modifier arguments`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#modifiers
304 | .. _`Custom modifiers`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#custom-modifiers
305 | .. _`Multipaths`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#multipaths
306 | .. _`Literals`: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#literals
307 | .. _`JSON Lines`: https://github.com/tidwall/gjson#json-lines
308 |
--------------------------------------------------------------------------------
/doc/Makefile:
--------------------------------------------------------------------------------
1 | # Minimal makefile for Sphinx documentation
2 | #
3 |
4 | # You can set these variables from the command line, and also
5 | # from the environment for the first two.
6 | SPHINXOPTS ?=
7 | SPHINXBUILD ?= sphinx-build
8 | SOURCEDIR = doc/source
9 | BUILDDIR = doc/build
10 |
11 | # Put it first so that "make" without argument is like "make help".
12 | help:
13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
14 |
15 | .PHONY: help Makefile
16 |
17 | # Catch-all target: route all unknown targets to Sphinx using the new
18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
19 | %: Makefile
20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
21 |
--------------------------------------------------------------------------------
/doc/source/api.rst:
--------------------------------------------------------------------------------
1 | API
2 | ===
3 |
4 | .. automodule:: gjson
5 |
--------------------------------------------------------------------------------
/doc/source/cli.rst:
--------------------------------------------------------------------------------
1 | CLI
2 | ===
3 |
4 | .. argparse::
5 | :module: gjson._cli
6 | :func: get_parser
7 | :prog: gjson
8 | :nodefault:
9 |
10 |
--------------------------------------------------------------------------------
/doc/source/conf.py:
--------------------------------------------------------------------------------
1 | """Configuration file for the Sphinx documentation builder."""
2 | # This file only contains a selection of the most common options. For a full
3 | # list see the documentation:
4 | # https://www.sphinx-doc.org/en/master/usage/configuration.html
5 |
6 | import sys
7 | from pathlib import Path
8 |
9 | import sphinx_rtd_theme
10 | from pkg_resources import get_distribution
11 |
12 | # -- Path setup --------------------------------------------------------------
13 |
14 | # If extensions (or modules to document with autodoc) are in another directory,
15 | # add these directories to sys.path here. If the directory is relative to the
16 | # documentation root, use os.path.abspath to make it absolute, like shown here.
17 | #
18 | # import os
19 | # import sys
20 | # sys.path.insert(0, os.path.abspath('.'))
21 |
22 | sys.path.insert(0, Path(__file__).resolve().parent.parent)
23 |
24 | # -- Project information -----------------------------------------------------
25 |
26 | project = 'gjson'
27 | title = f'{project} Documentation'
28 | copyright = '2022, Riccardo Coccioli'
29 | author = 'Riccardo Coccioli'
30 |
31 | # The version info for the project you're documenting, acts as replacement for
32 | # |version| and |release|, also used in various other places throughout the
33 | # built documents.
34 | #
35 | # The full version, including alpha/beta/rc tags.
36 | release = get_distribution('gjson').version
37 | # The short X.Y version.
38 | version = release
39 |
40 | # -- General configuration ---------------------------------------------------
41 |
42 | # Add any Sphinx extension module names here, as strings. They can be
43 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
44 | # ones.
45 | extensions = [
46 | 'sphinx.ext.autodoc',
47 | 'sphinx.ext.napoleon',
48 | 'sphinx.ext.intersphinx',
49 | 'sphinx.ext.todo',
50 | 'sphinx.ext.coverage',
51 | 'sphinx.ext.viewcode',
52 | 'sphinxarg.ext',
53 | 'sphinx_autodoc_typehints',
54 | ]
55 |
56 | # Add any paths that contain templates here, relative to this directory.
57 | templates_path = ['_templates']
58 |
59 | # List of patterns, relative to source directory, that match files and
60 | # directories to ignore when looking for source files.
61 | # This pattern also affects html_static_path and html_extra_path.
62 | exclude_patterns = []
63 |
64 | # -- Options for manual page output ---------------------------------------
65 |
66 | # One entry per manual page. List of tuples
67 | # (source start file, name, description, authors, manual section).
68 | man_pages = [
69 | ('cli', 'gjson', 'filter and extract data from JSON-like files', [author], 1),
70 | ]
71 |
72 | # -- Options for HTML output -------------------------------------------------
73 |
74 | # The theme to use for HTML and HTML Help pages. See the documentation for
75 | # a list of builtin themes.
76 | #
77 | html_theme = 'sphinx_rtd_theme'
78 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
79 |
80 | # Add any paths that contain custom static files (such as style sheets) here,
81 | # relative to this directory. They are copied after the builtin static files,
82 | # so a file named "default.css" will overwrite the builtin "default.css".
83 | html_static_path = []
84 |
85 | intersphinx_mapping = {
86 | 'python': ('https://docs.python.org/3/', None),
87 | }
88 |
89 | # Napoleon settings
90 | napoleon_google_docstring = True
91 | napoleon_numpy_docstring = False
92 | napoleon_include_init_with_doc = False
93 | napoleon_include_private_with_doc = False
94 | napoleon_include_special_with_doc = False
95 | napoleon_use_admonition_for_examples = False
96 | napoleon_use_admonition_for_notes = False
97 | napoleon_use_admonition_for_references = False
98 | napoleon_use_ivar = False
99 | napoleon_use_param = True
100 | napoleon_use_keyword = True
101 | napoleon_use_rtype = True
102 | napoleon_type_aliases = None
103 | napoleon_attr_annotations = True
104 |
105 |
106 | # Autodoc settings
107 | autodoc_default_options = {
108 | # Using None as value instead of True to support the version of Sphinx used in Buster
109 | 'members': None,
110 | 'member-order': 'bysource',
111 | 'show-inheritance': None,
112 | 'special-members': '__str__,__version__,__call__',
113 | }
114 | autoclass_content = 'both'
115 |
116 |
--------------------------------------------------------------------------------
/doc/source/index.rst:
--------------------------------------------------------------------------------
1 | gjson-py |release| documentation
2 | ================================
3 |
4 | .. toctree::
5 | :maxdepth: 3
6 |
7 | introduction
8 | cli
9 | api
10 |
11 |
12 | .. toctree::
13 | :maxdepth: 1
14 |
15 | release
16 |
--------------------------------------------------------------------------------
/doc/source/introduction.rst:
--------------------------------------------------------------------------------
1 | .. include:: ../../README.rst
2 |
--------------------------------------------------------------------------------
/doc/source/release.rst:
--------------------------------------------------------------------------------
1 | .. include:: ../../CHANGELOG.rst
2 |
--------------------------------------------------------------------------------
/gjson/__init__.py:
--------------------------------------------------------------------------------
1 | """GJSON module."""
2 | import json
3 | import os
4 | import re
5 | from importlib.metadata import PackageNotFoundError, version
6 | from typing import Any
7 |
8 | from gjson._gjson import MODIFIER_NAME_RESERVED_CHARS, GJSONObj
9 | from gjson._protocols import ModifierProtocol
10 | from gjson.exceptions import GJSONError, GJSONParseError
11 |
12 | # Explicit export of modules for the import * syntax, custom order to force the documentation order
13 | __all__ = ['get', 'GJSON', 'GJSONError', 'GJSONParseError', 'ModifierProtocol', 'GJSONObj', '__version__']
14 |
15 |
16 | # TODO: use a proper type hint for obj once https://github.com/python/typing/issues/182 will be fixed
17 | def get(obj: Any, query: str, *, as_str: bool = False, quiet: bool = False) -> Any:
18 | """Quick accessor to GJSON functionalities exposed for simplicity of use.
19 |
20 | Examples:
21 | Import and directly use this quick helper for the simpler usage::
22 |
23 | >>> import gjson
24 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]}
25 | >>> gjson.get(data, 'items.#.size')
26 | [1, 2]
27 |
28 | Arguments:
29 | obj: the object to query. It must be accessible in JSON-like fashion so it must be an object that can be
30 | converted to JSON.
31 | query: the query string to evaluate to extract the data from the object.
32 | as_str: if set to :py:data:`True` returns a JSON-encoded string, a Python object otherwise.
33 | quiet: on error, if set to :py:data:`True`, will raises an GJSONError exception. Otherwise returns
34 | :py:data:`None` on error.
35 |
36 | Return:
37 | the resulting object.
38 |
39 | """
40 | gjson_obj = GJSON(obj)
41 | if as_str:
42 | return gjson_obj.getj(query, quiet=quiet)
43 |
44 | return gjson_obj.get(query, quiet=quiet)
45 |
46 |
47 | class GJSON:
48 | """The GJSON class to operate on JSON-like objects."""
49 |
50 | def __init__(self, obj: Any):
51 | """Initialize the instance with the given object.
52 |
53 | Examples:
54 | Use the :py:class:`gjson.GJSON` class for more complex usage or to perform multiple queries on the same
55 | object::
56 |
57 | >>> import gjson
58 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]}
59 | >>> gjson_obj = gjson.GJSON(data)
60 |
61 | Arguments:
62 | obj: the object to query.
63 |
64 | """
65 | self._obj = obj
66 | self._custom_modifiers: dict[str, ModifierProtocol] = {}
67 |
68 | def __str__(self) -> str:
69 | """Return the current object as a JSON-encoded string.
70 |
71 | Examples:
72 | Converting to string a :py:class:`gjson.GJSON` object returns it as a JSON-encoded string::
73 |
74 | >>> str(gjson_obj)
75 | '{"items": [{"name": "a", "size": 1}, {"name": "b", "size": 2}]}'
76 |
77 | Returns:
78 | the JSON-encoded string representing the instantiated object.
79 |
80 | """
81 | return json.dumps(self._obj, ensure_ascii=False)
82 |
83 | def get(self, query: str, *, quiet: bool = False) -> Any:
84 | """Perform a query on the instantiated object and return the resulting object.
85 |
86 | Examples:
87 | Perform a query and get the resulting object::
88 |
89 | >>> gjson_obj.get('items.#.size')
90 | [1, 2]
91 |
92 | Arguments:
93 | query: the GJSON query to apply to the object.
94 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in
95 | case of error.
96 |
97 | Raises:
98 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`.
99 |
100 | Returns:
101 | the resulting object or :py:data:`None` if the ``quiet`` parameter is :py:data:`True` and there was an
102 | error.
103 |
104 | """
105 | try:
106 | return GJSONObj(self._obj, query, custom_modifiers=self._custom_modifiers).get()
107 | except GJSONError:
108 | if quiet:
109 | return None
110 | raise
111 |
112 | def getj(self, query: str, *, quiet: bool = False) -> str:
113 | """Perform a query on the instantiated object and return the resulting object as JSON-encoded string.
114 |
115 | Examples:
116 | Perform a query and get the resulting object as a JSON-encoded string::
117 |
118 | >>> gjson_obj.getj('items.#.size')
119 | '[1, 2]'
120 |
121 | Arguments:
122 | query: the GJSON query to apply to the object.
123 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in
124 | case of error.
125 |
126 | Raises:
127 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`.
128 |
129 | Returns:
130 | the JSON-encoded string representing the resulting object or :py:data:`None` if the ``quiet`` parameter is
131 | :py:data:`True` and there was an error.
132 |
133 | """
134 | try:
135 | return str(GJSONObj(self._obj, query, custom_modifiers=self._custom_modifiers))
136 | except GJSONError:
137 | if quiet:
138 | return ''
139 | raise
140 |
141 | def get_gjson(self, query: str, *, quiet: bool = False) -> 'GJSON':
142 | """Perform a query on the instantiated object and return the resulting object as a GJSON instance.
143 |
144 | Examples:
145 | Perform a query and get the resulting object already encapsulated into a :py:class:`gjson.GJSON` object::
146 |
147 | >>> sizes = gjson_obj.get_gjson('items.#.size')
148 | >>> str(sizes)
149 | '[1, 2]'
150 | >>> sizes.get('0')
151 | 1
152 |
153 | Arguments:
154 | query: the GJSON query to apply to the object.
155 | quiet: wheter to raise a :py:class:`gjson.GJSONError` exception on error or just return :py:data:`None` in
156 | case of error.
157 |
158 | Raises:
159 | gjson.GJSONError: on error if the quiet parameter is not :py:data:`True`.
160 |
161 | Returns:
162 | the resulting object encapsulated as a :py:class:`gjson.GJSON` object or :py:data:`None` if the ``quiet``
163 | parameter is :py:data:`True` and there was an error.
164 |
165 | """
166 | return GJSON(self.get(query, quiet=quiet))
167 |
168 | def register_modifier(self, name: str, func: ModifierProtocol) -> None:
169 | """Register a custom modifier.
170 |
171 | Examples:
172 | Register a custom modifier that sums all the numbers in a list:
173 |
174 | >>> def custom_sum(options, obj, *, last):
175 | ... # insert sanity checks code here
176 | ... return sum(obj)
177 | ...
178 | >>> gjson_obj.register_modifier('sum', custom_sum)
179 | >>> gjson_obj.get('items.#.size.@sum')
180 | 3
181 |
182 | Arguments:
183 | name: the modifier name. It will be called where ``@name`` is used in the query. If two custom modifiers
184 | are registered with the same name the last one will be used.
185 | func: the modifier code in the form of a callable object that adhere to the
186 | :py:class:`gjson.ModifierProtocol`.
187 |
188 | Raises:
189 | gjson.GJSONError: if the provided callable doesn't adhere to the :py:class:`gjson.ModifierProtocol`.
190 |
191 | """
192 | # Escape the ] as they are inside a [...] block
193 | not_allowed_regex = ''.join(MODIFIER_NAME_RESERVED_CHARS).replace(']', r'\]')
194 | if re.search(fr'[{not_allowed_regex}]', name):
195 | not_allowed_string = ', '.join(f'`{i}`' for i in MODIFIER_NAME_RESERVED_CHARS)
196 | raise GJSONError(f'Unable to register modifier `{name}`, contains at least one not allowed character: '
197 | f'{not_allowed_string}')
198 |
199 | if name in GJSONObj.builtin_modifiers():
200 | raise GJSONError(f'Unable to register a modifier with the same name of the built-in modifier: @{name}.')
201 |
202 | if not isinstance(func, ModifierProtocol):
203 | raise GJSONError(f'The given func "{func}" for the custom modifier @{name} does not adhere '
204 | 'to the gjson.ModifierProtocol.')
205 |
206 | self._custom_modifiers[name] = func
207 |
208 |
209 | try:
210 | __version__: str = version('gjson')
211 | """str: the version of the current gjson module."""
212 | except PackageNotFoundError: # pragma: no cover - this should never happen during tests
213 | # Read the override from the environment, if present (like inside Debian build system)
214 | if 'SETUPTOOLS_SCM_PRETEND_VERSION' in os.environ:
215 | __version__ = os.environ['SETUPTOOLS_SCM_PRETEND_VERSION']
216 |
--------------------------------------------------------------------------------
/gjson/_cli.py:
--------------------------------------------------------------------------------
1 | """GJSON module."""
2 | import argparse
3 | import json
4 | import sys
5 | from collections.abc import Sequence
6 | from typing import IO, Any, Optional
7 |
8 | from gjson import GJSONError, get
9 |
10 |
11 | def cli(argv: Optional[Sequence[str]] = None) -> int:
12 | """Command line entry point to run gjson as a CLI tool.
13 |
14 | Arguments:
15 | argv: a sequence of CLI arguments to parse. If not set they will be read from sys.argv.
16 |
17 | Returns:
18 | The CLI exit code to use.
19 |
20 | Raises:
21 | OSError: for system-related errors, including I/O failures.
22 | json.JSONDecodeError: when the input data is not a valid JSON.
23 | gjson.GJSONError: for any query-related error raised by gjson.
24 |
25 | """
26 | parser = get_parser()
27 | args = parser.parse_args(argv)
28 |
29 | encapsulate = False
30 | if args.query.startswith('..'):
31 | args.query = args.query[2:]
32 | encapsulate = True
33 |
34 | # Use argparse.FileType here instead of putting it as type in the --file argument parsing, to allow to handle the
35 | # verbosity in case of error and make sure the file is always closed in case other arguments fail the validation.
36 | try:
37 | args.file = argparse.FileType(encoding='utf-8', errors='surrogateescape')(args.file)
38 | except (OSError, argparse.ArgumentTypeError) as ex:
39 | if args.verbose == 1:
40 | print(f'{ex.__class__.__name__}: {ex}', file=sys.stderr)
41 | elif args.verbose >= 2: # noqa: PLR2004
42 | raise
43 |
44 | return 1
45 |
46 | # Reconfigure __stdin__ and __stdout__ instead of stdin and stdout because the latters are TextIO and could not
47 | # have the reconfigure() method if re-assigned, while reconfigure() is part of TextIOWrapper.
48 | # See also: https://github.com/python/typeshed/pull/8171
49 | sys.__stdin__.reconfigure(errors='surrogateescape')
50 | sys.__stdout__.reconfigure(errors='surrogateescape')
51 |
52 | def _execute(line: str, file_obj: Optional[IO[Any]]) -> int:
53 | try:
54 | if encapsulate:
55 | if line:
56 | input_data = [json.loads(line, strict=False)]
57 | elif file_obj is not None:
58 | input_data = []
59 | for input_line in file_obj:
60 | if input_line.strip():
61 | input_data.append(json.loads(input_line, strict=False))
62 | elif line:
63 | input_data = json.loads(line, strict=False)
64 | elif file_obj is not None:
65 | input_data = json.load(file_obj, strict=False)
66 |
67 | result = get(input_data, args.query, as_str=True)
68 | exit_code = 0
69 | except (json.JSONDecodeError, GJSONError) as ex:
70 | result = ''
71 | exit_code = 1
72 | if args.verbose == 1:
73 | print(f'{ex.__class__.__name__}: {ex}', file=sys.stderr)
74 | elif args.verbose >= 2: # noqa: PLR2004
75 | raise
76 |
77 | if result:
78 | print(result)
79 |
80 | return exit_code
81 |
82 | if args.lines:
83 | exit_code = 0
84 | for line in args.file:
85 | data = line.strip()
86 | if not data:
87 | continue
88 | ret = _execute(data, None)
89 | if ret > exit_code:
90 | exit_code = ret
91 | else:
92 | exit_code = _execute('', args.file)
93 |
94 | return exit_code
95 |
96 |
97 | def get_parser() -> argparse.ArgumentParser:
98 | """Get the CLI argument parser.
99 |
100 | Returns:
101 | the argument parser for the CLI.
102 |
103 | """
104 | parser = argparse.ArgumentParser(
105 | prog='gjson',
106 | description=('A simple way to filter and extract data from JSON-like data structures. Python porting of the '
107 | 'Go GJSON package.'),
108 | epilog='See also the full documentation available at https://volans-.github.io/gjson-py/index.html',
109 | formatter_class=argparse.ArgumentDefaultsHelpFormatter,
110 | )
111 | parser.add_argument('-v', '--verbose', action='count', default=0,
112 | help=('Verbosity level. By default on error no output will be printed. Use -v to get the '
113 | 'error message to stderr and -vv to get the full traceback.'))
114 | parser.add_argument('-l', '--lines', action='store_true',
115 | help='Treat the input as JSON Lines, parse each line and apply the query to each line.')
116 | # argparse.FileType is used later to parse this argument.
117 | parser.add_argument('file', default='-', nargs='?',
118 | help='Input JSON file to query. Reads from stdin if the argument is missing or set to "-".')
119 | parser.add_argument('query', help='A GJSON query to apply to the input data.')
120 |
121 | return parser
122 |
--------------------------------------------------------------------------------
/gjson/_gjson.py:
--------------------------------------------------------------------------------
1 | """GJSON module."""
2 | import json
3 | import operator
4 | import re
5 | from collections import Counter
6 | from collections.abc import Callable, Mapping, Sequence
7 | from dataclasses import dataclass
8 | from itertools import zip_longest
9 | from typing import Any, Optional, Type, TypeVar, Union
10 |
11 | from gjson._protocols import ModifierProtocol
12 | from gjson.exceptions import GJSONError, GJSONInvalidSyntaxError, GJSONParseError
13 |
14 | ESCAPE_CHARACTER = '\\'
15 | """str: The grammar escape character."""
16 | DOT_DELIMITER = '.'
17 | """str: One of the available delimiters in the query grammar."""
18 | PIPE_DELIMITER = '|'
19 | """str: One of the available delimiters in the query grammar."""
20 | DELIMITERS = (DOT_DELIMITER, PIPE_DELIMITER)
21 | """tuple: All the available delimiters in the query grammar."""
22 | MULTIPATHS_DELIMITERS = (*DELIMITERS, ']', '}', ',')
23 | """tuple: All the available delimiters in the query grammar."""
24 | # Single character operators goes last to avoid mis-detection.
25 | QUERIES_OPERATORS = ('==~', '==', '!=', '<=', '>=', '!%', '=', '<', '>', '%')
26 | """tuple: The list of supported operators inside queries."""
27 | MODIFIER_NAME_RESERVED_CHARS = ('"', ',', '.', '|', ':', '@', '{', '}', '[', ']', '(', ')')
28 | """tuple: The list of reserver characters not usable in a modifier's name."""
29 | PARENTHESES_PAIRS = {'(': ')', ')': '(', '[': ']', ']': '[', '{': '}', '}': '{'}
30 | GJSONObjT = TypeVar('GJSONObjT', bound='GJSONObj')
31 |
32 |
33 | class NoResult:
34 | """A no result type to be passed around and be checked."""
35 |
36 |
37 | @dataclass
38 | class BaseQueryPart:
39 | """Base dataclass class to represent a query part."""
40 |
41 | start: int
42 | end: int
43 | part: str
44 | delimiter: str
45 | previous: Optional['BaseQueryPart']
46 | is_last: bool
47 |
48 | def __str__(self) -> str:
49 | """Return the string representation of the part.
50 |
51 | Returns:
52 | The part property of the instance.
53 |
54 | """
55 | return self.part
56 |
57 |
58 | class FieldQueryPart(BaseQueryPart):
59 | """Basic field path query part."""
60 |
61 |
62 | class ArrayLenghtQueryPart(BaseQueryPart):
63 | """Hash query part, to get the size of an array."""
64 |
65 |
66 | class ArrayIndexQueryPart(BaseQueryPart):
67 | """Integer query part to get an array index."""
68 |
69 | @property
70 | def index(self) -> int:
71 | """Return the integer representation of the query part.
72 |
73 | Returns:
74 | the index as integer.
75 |
76 | """
77 | return int(self.part)
78 |
79 |
80 | @dataclass
81 | class ArrayQueryQueryPart(BaseQueryPart):
82 | """Query part for array queries, with additional fields."""
83 |
84 | field: str
85 | operator: str
86 | value: Union[str, 'ArrayQueryQueryPart']
87 | first_only: bool
88 |
89 |
90 | @dataclass
91 | class ModifierQueryPart(BaseQueryPart):
92 | """Modifier query part."""
93 |
94 | name: str
95 | options: dict[Any, Any]
96 |
97 |
98 | @dataclass
99 | class MultipathsItem:
100 | """Single multipaths query item."""
101 |
102 | key: str
103 | values: list[BaseQueryPart]
104 |
105 |
106 | @dataclass
107 | class MultipathsObjectQueryPart(BaseQueryPart):
108 | """JSON object multipaths query part."""
109 |
110 | parts: list[MultipathsItem]
111 |
112 |
113 | @dataclass
114 | class MultipathsArrayQueryPart(BaseQueryPart):
115 | """JSON object multipaths query part."""
116 |
117 | parts: list[list[BaseQueryPart]]
118 |
119 |
120 | class LiteralQueryPart(BaseQueryPart):
121 | """Literal query part."""
122 |
123 |
124 | class GJSONObj:
125 | """A low-level class to perform the GJSON query on a JSON-like object."""
126 |
127 | def __init__(self, obj: Any, query: str, *, custom_modifiers: Optional[dict[str, ModifierProtocol]] = None):
128 | """Initialize the instance with the starting object and query.
129 |
130 | Examples:
131 | Client code should not need to instantiate this low-level class in normal circumastances::
132 |
133 | >>> import gjson
134 | >>> data = {'items': [{'name': 'a', 'size': 1}, {'name': 'b', 'size': 2}]}
135 | >>> gjson_obj = gjson.GJSONObj(data, 'items.#.size')
136 |
137 | Arguments:
138 | obj: the JSON-like object to query.
139 | query: the GJSON query to apply to the object.
140 | custom_modifiers: an optional dictionary with the custom modifiers to load. The dictionary keys are the
141 | names of the modifiers and the values are the callables with the modifier code that adhere to the
142 | :py:class:`gjson.ModifierProtocol` protocol.
143 |
144 | Raises:
145 | gjson.GJSONError: if any provided custom modifier overrides a built-in one or is not callable.
146 |
147 | """
148 | self._obj = obj
149 | self._query = query
150 | if custom_modifiers is not None:
151 | if (intersection := self.builtin_modifiers().intersection(set(custom_modifiers.keys()))):
152 | raise GJSONError(f'Some provided custom_modifiers have the same name of built-in ones: {intersection}.')
153 |
154 | for name, modifier in custom_modifiers.items():
155 | if not isinstance(modifier, ModifierProtocol):
156 | raise GJSONError(f'The given func "{modifier}" for the custom modifier @{name} does not adhere '
157 | 'to the gjson.ModifierProtocol.')
158 |
159 | self._custom_modifiers = custom_modifiers if custom_modifiers else {}
160 | self._dump_params: dict[str, Any] = {'ensure_ascii': False}
161 | self._after_hash = False
162 | self._after_query_all = False
163 |
164 | @classmethod
165 | def builtin_modifiers(cls: Type[GJSONObjT]) -> set[str]:
166 | """Return the names of the built-in modifiers.
167 |
168 | Returns:
169 | the names of the built-in modifiers.
170 |
171 | """
172 | prefix = '_apply_modifier_'
173 | return {modifier[len(prefix):] for modifier in dir(cls) if modifier.startswith(prefix)}
174 |
175 | def get(self) -> Any:
176 | """Perform the query and return the resulting object.
177 |
178 | Examples:
179 | Returns the resulting object::
180 |
181 | >>> gjson_obj.get()
182 | [1, 2]
183 |
184 | Raises:
185 | gjson.GJSONError: on error.
186 |
187 | Returns:
188 | the resulting object.
189 |
190 | """
191 | # Reset internal parameters
192 | self._dump_params = {'ensure_ascii': False}
193 | self._after_hash = False
194 | self._after_query_all = False
195 |
196 | if not self._query:
197 | raise GJSONError('Empty query.')
198 |
199 | obj = self._obj
200 | for part in self._parse(start=0, end=len(self._query) - 1):
201 | obj = self._parse_part(part, obj)
202 |
203 | return obj
204 |
205 | def __str__(self) -> str:
206 | """Return the JSON string representation of the object, based on the query parameters.
207 |
208 | Examples:
209 | Returns the resulting object as a JSON-encoded string::
210 |
211 | >>> str(gjson_obj)
212 | '[1, 2]'
213 |
214 | Raises:
215 | gjson.GJSONError: on error.
216 |
217 | Returns:
218 | the JSON encoded string.
219 |
220 | """
221 | obj = self.get()
222 | prefix = self._dump_params.pop('prefix', '')
223 | json_string = json.dumps(obj, **self._dump_params)
224 |
225 | if prefix:
226 | return '\n'.join(f'{prefix}{line}' for line in json_string.splitlines())
227 |
228 | return json_string
229 |
230 | def _parse(self, *, start: int, end: int, max_end: int = 0, # noqa: PLR0912, PLR0913, PLR0915
231 | delimiter: str = '', in_multipaths: bool = False) -> list[BaseQueryPart]:
232 | """Parse the query. It will delegate to more specific parsers for each different feature.
233 |
234 | Arguments:
235 | start: the start position in the query.
236 | end: the end position in the query.
237 | max_end: an optional last position up to where a closing parentheses can be searched.
238 | delimiter: the optional delimiter before the query, if this is called on a multipaths.
239 | in_multipaths: whether the part to be parsed is inside a multipaths.
240 |
241 | Raises:
242 | gjson.GJSONParseError: on error.
243 |
244 | Returns:
245 | the resulting object.
246 |
247 | """
248 | current: list[str] = []
249 | current_start = -1
250 | parts: list[BaseQueryPart] = []
251 | previous: Optional[BaseQueryPart] = None
252 | require_delimiter = False
253 |
254 | i = start
255 | while True:
256 | part: Optional[BaseQueryPart] = None
257 | # Get current and next character in the query
258 | if i == end:
259 | next_char = None
260 | elif i >= end:
261 | if parts and not current:
262 | parts[-1].is_last = True
263 | break
264 | else:
265 | next_char = self._query[i + 1]
266 |
267 | char = self._query[i]
268 |
269 | if char in DELIMITERS:
270 | if i == start:
271 | raise GJSONParseError('Invalid query starting with a path delimiter.',
272 | query=self._query, position=i)
273 |
274 | if next_char in DELIMITERS:
275 | raise GJSONParseError('Invalid query with two consecutive path delimiters.',
276 | query=self._query, position=i)
277 | if current:
278 | part = FieldQueryPart(start=current_start, end=i - 1, part=''.join(current),
279 | delimiter=delimiter, previous=previous, is_last=False)
280 | parts.append(part)
281 | previous = part
282 | current = []
283 | current_start = -1
284 |
285 | delimiter = char
286 | require_delimiter = False
287 | if next_char is None:
288 | raise GJSONParseError('Delimiter at the end of the query.', query=self._query, position=i)
289 |
290 | i += 1
291 | continue
292 |
293 | if char == '@':
294 | part = self._parse_modifier_query_part(
295 | start=i, delimiter=delimiter, max_end=max_end, in_multipaths=in_multipaths)
296 | elif char == '#' and (next_char in DELIMITERS or next_char is None):
297 | part = ArrayLenghtQueryPart(start=i, end=i, part=char, delimiter=delimiter, is_last=next_char is None,
298 | previous=previous)
299 | elif char == '#' and next_char == '(':
300 | part = self._parse_array_query_query_part(start=i, delimiter=delimiter, max_end=max_end)
301 | elif re.match(r'[0-9]', char) and not current:
302 | part = self._parse_array_index_query_part(start=i, delimiter=delimiter, in_multipaths=in_multipaths)
303 | elif char == '{':
304 | part = self._parse_object_multipaths_query_part(start=i, delimiter=delimiter, max_end=max_end)
305 | require_delimiter = True
306 | elif char == '[':
307 | part = self._parse_array_multipaths_query_part(start=i, delimiter=delimiter, max_end=max_end)
308 | require_delimiter = True
309 | elif char == '!':
310 | part = self._parse_literal_query_part(
311 | start=i, delimiter=delimiter, max_end=max_end, in_multipaths=in_multipaths)
312 | elif in_multipaths and char == ',':
313 | i -= 1
314 | break
315 | elif in_multipaths and require_delimiter:
316 | raise GJSONInvalidSyntaxError('Missing separator after multipath.', query=self._query, position=i)
317 |
318 | if part:
319 | part.previous = previous
320 | parts.append(part)
321 | previous = part
322 | else: # Normal path, no special grammar
323 | if not current:
324 | current_start = i
325 |
326 | current.append(char)
327 | if char == ESCAPE_CHARACTER:
328 | i += 1 # Skip the escaped character
329 | if next_char is None:
330 | raise GJSONParseError('Escape character at the end of the query.',
331 | query=self._query, position=i)
332 |
333 | current.append(next_char)
334 |
335 | if part:
336 | i = part.end + 1
337 | else:
338 | i += 1
339 |
340 | if part is None and current:
341 | part = FieldQueryPart(start=current_start, end=i, part=''.join(current),
342 | delimiter=delimiter, previous=previous, is_last=True)
343 | parts.append(part)
344 |
345 | return parts
346 |
347 | @staticmethod
348 | def _is_sequence(obj: Any) -> bool:
349 | """Check if an object is a sequence but not a string or bytes object.
350 |
351 | Arguments:
352 | obj: the object to test.
353 |
354 | Returns:
355 | :py:data:`True` if the object is a sequence but not a string or bytes, :py:data:`False` otherwise.
356 |
357 | """
358 | return isinstance(obj, Sequence) and not isinstance(obj, (str, bytes))
359 |
360 | def _find_closing_parentheses(self, *, start: int, opening: str, suffix: str = '', # noqa: PLR0912
361 | max_end: int = 0) -> int:
362 | """Find the matching parentheses that closes the opening one looking for unbalance of the given character.
363 |
364 | Arguments:
365 | start: the index of the opening parentheses in the query.
366 | opening: the opening parentheses to look for imbalances.
367 | suffix: an optional suffix that can be present after the closing parentheses before reaching a delimiter or
368 | the end of the query.
369 | max_end: an optional last position up to where the parentheses can be found.
370 |
371 | Raises:
372 | gjson.GJSONParseError: if unable to find the closing parentheses or the parentheses are not balanced.
373 |
374 | Returns:
375 | the position of the closing parentheses if there is no suffix or the one of the last character of the
376 | suffix if present.
377 |
378 | """
379 | closing = PARENTHESES_PAIRS[opening]
380 | opened = 0
381 | end = -1
382 | escaped = False
383 | in_string = False
384 | query = self._query[start:max_end + 1] if max_end else self._query[start:]
385 |
386 | for i, char in enumerate(query):
387 | if char == ESCAPE_CHARACTER:
388 | escaped = True
389 | continue
390 |
391 | if escaped:
392 | escaped = False
393 | continue
394 |
395 | if char == '"':
396 | in_string = not in_string
397 | continue
398 |
399 | if in_string:
400 | continue
401 |
402 | if char == opening:
403 | opened += 1
404 | elif char == closing: # noqa: SIM102
405 | if opened:
406 | opened -= 1
407 | if not opened:
408 | end = i
409 | break
410 |
411 | if opened or end < 0:
412 | raise GJSONParseError(f'Unbalanced parentheses `{opening}`, {opened} still opened.',
413 | query=self._query, position=start)
414 |
415 | if suffix and end + len(suffix) < len(query) and query[end + 1:end + len(suffix) + 1] == suffix:
416 | end += len(suffix)
417 |
418 | if end + 1 < len(query):
419 | delimiters = list(MULTIPATHS_DELIMITERS) if max_end else list(DELIMITERS)
420 | if opening == '(' and suffix == '#': # Nested queries
421 | delimiters.append(')')
422 |
423 | if (max_end and query[end + 1] not in delimiters) or (not max_end and query[end + 1] not in DELIMITERS):
424 | raise GJSONParseError('Expected delimiter or end of query after closing parenthesis.',
425 | query=self._query, position=start + end)
426 |
427 | return start + end
428 |
429 | def _parse_modifier_query_part(self, *, start: int, delimiter: str, max_end: int = 0,
430 | in_multipaths: bool = False) -> ModifierQueryPart:
431 | """Find the modifier end position in the query starting from a given point.
432 |
433 | Arguments:
434 | start: the index of the ``@`` symbol that starts a modifier in the query.
435 | delimiter: the delimiter before the modifier.
436 | max_end: an optional last position up to where the last character can be found.
437 | in_multipaths: whether the part to be parsed is inside a multipaths.
438 |
439 | Raises:
440 | gjson.GJSONParseError: on invalid modifier.
441 |
442 | Returns:
443 | the modifier query part object.
444 |
445 | """
446 | end = start
447 | escaped = False
448 | delimiters = MULTIPATHS_DELIMITERS if in_multipaths else DELIMITERS
449 | options: dict[Any, Any] = {}
450 | query = self._query[start:max_end + 1] if max_end else self._query[start:]
451 | for i, char in enumerate(query):
452 | if char == ESCAPE_CHARACTER and not escaped:
453 | escaped = True
454 | continue
455 |
456 | if escaped:
457 | escaped = False
458 | continue
459 |
460 | if char == ':':
461 | name = self._query[start + 1:start + i]
462 | options_len, options = self._parse_modifier_options(start + i + 1)
463 | end = start + i + options_len
464 | break
465 |
466 | if char in delimiters:
467 | end = start + i - 1
468 | name = self._query[start + 1:start + i]
469 | break
470 |
471 | else: # End of query
472 | end = start + i
473 | name = self._query[start + 1:start + i + 1]
474 |
475 | name.replace(ESCAPE_CHARACTER, '')
476 | if not name:
477 | raise GJSONParseError('Got empty modifier name.', query=self._query, position=start)
478 |
479 | for char in MODIFIER_NAME_RESERVED_CHARS:
480 | if char in name:
481 | raise GJSONParseError(f'Invalid modifier name @{name}, the following characters are not allowed: '
482 | f'{MODIFIER_NAME_RESERVED_CHARS}', query=self._query, position=start)
483 |
484 | return ModifierQueryPart(start=start, end=end, part=self._query[start:end + 1], delimiter=delimiter,
485 | name=name, options=options, is_last=False, previous=None)
486 |
487 | def _parse_modifier_options(self, start: int) -> tuple[int, dict[Any, Any]]:
488 | """Find the modifier options end position in the query starting from a given point.
489 |
490 | Arguments:
491 | start: the index of the ``:`` symbol that starts a modifier options.
492 |
493 | Raises:
494 | gjson.GJSONParseError: on invalid modifier options.
495 |
496 | Returns:
497 | the modifier options last character index in the query and the parsed options.
498 |
499 | """
500 | if start >= len(self._query):
501 | raise GJSONParseError('Modifier with options separator `:` without any option.',
502 | query=self._query, position=start)
503 |
504 | if self._query[start] != '{':
505 | raise GJSONParseError('Expected JSON object `{...}` as modifier options.',
506 | query=self._query, position=start)
507 |
508 | query_parts = re.split(r'(? ArrayQueryQueryPart:
524 | """Parse an array query part starting from the given point.
525 |
526 | Arguments:
527 | start: the index of the ``#`` symbol that starts a ``#(...)`` or ``#(...)#`` query.
528 | delimiter: the delimiter before the modifier.
529 | max_end: an optional last position up to where the closing parentheses can be found.
530 |
531 | Raises:
532 | gjson.GJSONParseError: on invalid query.
533 |
534 | Returns:
535 | the array query part object.
536 |
537 | """
538 | end = self._find_closing_parentheses(start=start, opening='(', suffix='#', max_end=max_end)
539 | part = self._query[start:end + 1]
540 | if part[-1] == '#':
541 | content_end = -2
542 | first_only = False
543 | else:
544 | content_end = -1
545 | first_only = True
546 |
547 | content = part[2: content_end]
548 | query_operator = ''
549 | key = ''
550 | value: Union[str, ArrayQueryQueryPart] = ''
551 |
552 | pattern = '|'.join(re.escape(op) for op in QUERIES_OPERATORS)
553 | match = re.search(fr'(? Optional[ArrayIndexQueryPart]:
578 | """Parse an array index query part.
579 |
580 | Arguments:
581 | start: the index of the start of the path in the query.
582 | delimiter: the delimiter before the query part.
583 | in_multipaths: whether the part to be parsed is inside a multipaths.
584 |
585 | Returns:
586 | the array index query object if the integer path is found, :py:const:`None` otherwise.
587 |
588 | """
589 | subquery = self._query[start:]
590 | delimiters = MULTIPATHS_DELIMITERS if in_multipaths else DELIMITERS
591 | delimiters_match = '|'.join([re.escape(i) for i in delimiters])
592 | match = re.search(fr'^([1-9][0-9]*|0)({delimiters_match}|$)', subquery)
593 | if not match:
594 | return None
595 |
596 | end = start + len(match.groups()[0]) - 1
597 | part = self._query[start:end + 1]
598 |
599 | return ArrayIndexQueryPart(start=start, end=end, part=part, delimiter=delimiter, is_last=False, previous=None)
600 |
601 | def _parse_object_multipaths_query_part( # noqa: PLR0912, PLR0915
602 | self, *, start: int, delimiter: str, max_end: int = 0) -> MultipathsObjectQueryPart:
603 | """Parse a multipaths object query part.
604 |
605 | Arguments:
606 | start: the index of the start of the path in the query.
607 | delimiter: the delimiter before the query part.
608 | max_end: an optional last position up to where the multipaths can extend.
609 |
610 | Returns:
611 | the multipaths object query part.
612 |
613 | Raises:
614 | gjson.GJSONParseError: on invalid query.
615 |
616 | """
617 | end = self._find_closing_parentheses(start=start, opening='{', max_end=max_end)
618 | part = self._query[start:end + 1]
619 | parts = []
620 |
621 | def _get_key(current_key: Optional[str], value: Optional[BaseQueryPart]) -> str:
622 | """Return the current key or the default value if not set. Allow for empty key as valid key.
623 |
624 | Arguments:
625 | current_key: the current key to evaluate.
626 | value: the current value from where to extract a key name if missing.
627 |
628 | """
629 | if current_key is not None:
630 | return current_key
631 |
632 | if value and isinstance(value, (FieldQueryPart, ArrayIndexQueryPart, ModifierQueryPart)):
633 | return value.part
634 |
635 | return '_'
636 |
637 | new_item = True
638 | escaped = False
639 | key: Optional[str] = None
640 | key_start = 0
641 | value_start = 0
642 | skip_until = 0
643 |
644 | for i, char in enumerate(part[1:-1], start=1):
645 | if skip_until and i <= skip_until:
646 | if i == skip_until:
647 | skip_until = 0
648 | new_item = True
649 |
650 | continue
651 |
652 | if new_item:
653 | value_start = 0
654 | if char == '"':
655 | key_start = i
656 | new_item = False
657 | continue
658 |
659 | if char != ',':
660 | value_start = i
661 | new_item = False
662 |
663 | if key_start:
664 | if char == ESCAPE_CHARACTER and not escaped:
665 | escaped = True
666 | continue
667 |
668 | if escaped:
669 | escaped = False
670 | continue
671 |
672 | if char == '"':
673 | try:
674 | key = json.loads(part[key_start:i + 1], strict=False)
675 | except json.JSONDecodeError as ex:
676 | raise GJSONParseError(f'Failed to parse multipaths key {part[key_start:i + 1]}.',
677 | query=self._query, position=key_start) from ex
678 | key_start = 0
679 | continue
680 |
681 | if key is not None and not key_start and not value_start:
682 | if char == ':':
683 | value_start = i + 1
684 | continue
685 |
686 | raise GJSONParseError(f'Expected colon after multipaths item with key "{key}".',
687 | query=self._query, position=i)
688 |
689 | if value_start:
690 | try:
691 | values = self._parse(
692 | start=start + value_start,
693 | end=end - 1,
694 | max_end=max_end - 1 if max_end else end - 1,
695 | delimiter=delimiter,
696 | in_multipaths=True)
697 | except GJSONInvalidSyntaxError:
698 | raise
699 | except GJSONParseError: # In multipaths, paths that fails are silently suppressed
700 | values = []
701 |
702 | if values:
703 | parts.append(MultipathsItem(key=_get_key(key, values[-1]), values=values))
704 | skip_until = values[-1].end - start + 1
705 | else:
706 | skip_until = end - start
707 |
708 | new_item = True
709 | key = None
710 | key_start = 0
711 | value_start = 0
712 | continue
713 |
714 | return MultipathsObjectQueryPart(start=start, end=end, part=part, delimiter=delimiter, previous=None,
715 | is_last=False, parts=parts)
716 |
717 | def _parse_array_multipaths_query_part(
718 | self, *, start: int, delimiter: str, max_end: int = 0) -> MultipathsArrayQueryPart:
719 | """Parse a multipaths object query part.
720 |
721 | Arguments:
722 | start: the index of the start of the path in the query.
723 | delimiter: the delimiter before the query part.
724 | max_end: an optional last position up to where the multipaths can extend.
725 |
726 | Returns:
727 | the multipaths array query part.
728 |
729 | Raises:
730 | gjson.GJSONParseError: on invalid query.
731 |
732 | """
733 | end = self._find_closing_parentheses(start=start, opening='[', max_end=max_end)
734 | part = self._query[start:end + 1]
735 | parts = []
736 | skip_until = 0
737 |
738 | for i, _ in enumerate(part[1:-1], start=1):
739 | if skip_until and i <= skip_until:
740 | if i == skip_until:
741 | skip_until = 0
742 |
743 | continue
744 |
745 | try:
746 | values = self._parse(
747 | start=start + i,
748 | end=end - 1,
749 | max_end=max_end - 1 if max_end else end - 1,
750 | delimiter=delimiter,
751 | in_multipaths=True)
752 | except GJSONInvalidSyntaxError:
753 | raise
754 | except GJSONParseError: # In multipaths, paths that fails are silently suppressed
755 | values = []
756 |
757 | if values:
758 | parts.append(values)
759 | skip_until = values[-1].end - start + 1
760 | else:
761 | skip_until = end - start
762 |
763 | return MultipathsArrayQueryPart(start=start, end=end, part=part, delimiter=delimiter, previous=None,
764 | is_last=False, parts=parts)
765 |
766 | def _parse_literal_query_part(self, *, start: int, delimiter: str, max_end: int = 0,
767 | in_multipaths: bool = False) -> LiteralQueryPart:
768 | """Parse a literal query part.
769 |
770 | Arguments:
771 | start: the index of the start of the path in the query.
772 | delimiter: the delimiter before the query part.
773 | max_end: an optional last position up to where the multipaths can extend.
774 | in_multipaths: whether the part to be parsed is inside a multipaths.
775 |
776 | Returns:
777 | the literal query part.
778 |
779 | Raises:
780 | gjson.GJSONParseError: on invalid query.
781 |
782 | """
783 | end = -1
784 | begin = self._query[start + 1:start + 2]
785 | if begin in ('{', '['):
786 | end = self._find_closing_parentheses(start=start + 1, opening=begin, max_end=max_end)
787 |
788 | elif begin == '"':
789 | query = self._query[start + 2:max_end + 1] if max_end else self._query[start + 2:]
790 | match = re.search(r'(? Any:
824 | """Parse the given part of the full query.
825 |
826 | Arguments:
827 | part: the query part as already parsed.
828 | obj: the current object.
829 | in_multipaths: whether the part to be parsed is inside a multipaths.
830 |
831 | Raises:
832 | gjson.GJSONParseError: on invalid query.
833 |
834 | Returns:
835 | the result of the query.
836 |
837 | """
838 | in_hash = False
839 | in_query_all = False
840 | ret: Any
841 |
842 | if isinstance(obj, NoResult):
843 | return obj
844 |
845 | if isinstance(part, ArrayLenghtQueryPart):
846 | in_hash = True
847 | if part.is_last:
848 | if part.delimiter == DOT_DELIMITER and (self._after_hash or self._after_query_all):
849 | ret = []
850 | elif part.delimiter == PIPE_DELIMITER and isinstance(part.previous, ArrayLenghtQueryPart):
851 | raise GJSONParseError('The pipe delimiter cannot immediately follow the # element.',
852 | query=self._query, position=part.start)
853 | elif self._is_sequence(obj):
854 | ret = len(obj)
855 | else:
856 | raise GJSONParseError('Expected a sequence like object for query part # at the end of the query, '
857 | f'got {type(obj)}.', query=self._query, position=part.start)
858 | else:
859 | ret = obj
860 |
861 | elif isinstance(part, ArrayQueryQueryPart):
862 | if not self._is_sequence(obj):
863 | raise GJSONParseError(f'Queries are supported only for sequence like objects, got {type(obj)}.',
864 | query=self._query, position=part.start)
865 |
866 | in_query_all = not part.first_only
867 | ret = self._parse_query(part, obj)
868 |
869 | elif isinstance(part, ModifierQueryPart):
870 | ret = self._apply_modifier(part, obj)
871 |
872 | elif isinstance(part, ArrayIndexQueryPart):
873 | if isinstance(obj, Mapping): # Integer object keys not supported by JSON
874 | if not in_multipaths and part.part not in obj:
875 | raise GJSONParseError(f'Mapping object does not have key `{part}`.',
876 | query=self._query, position=part.start)
877 | ret = obj.get(part.part, NoResult())
878 | elif self._is_sequence(obj):
879 | if (self._after_hash or self._after_query_all) and part.delimiter == DOT_DELIMITER:
880 | # Skip non mapping items and items without the given key
881 | ret = []
882 | for i in obj:
883 | if isinstance(i, Mapping) and part.part in i:
884 | ret.append(i[part.part])
885 | elif self._is_sequence(i) and len(i):
886 | ret.append(i[int(part.part)])
887 | elif (self._after_hash and part.delimiter == PIPE_DELIMITER
888 | and isinstance(part.previous, ArrayLenghtQueryPart)):
889 | raise GJSONParseError('Integer query part after a pipe delimiter on an sequence like object.',
890 | query=self._query, position=part.start)
891 | else:
892 | num = len(obj)
893 | if part.index >= num:
894 | raise GJSONParseError(f'Index `{part}` out of range for sequence object with {num} items in '
895 | 'query.', query=self._query, position=part.start)
896 | ret = obj[part.index]
897 | else:
898 | raise GJSONParseError(f'Integer query part on unsupported object type {type(obj)}, expected a mapping '
899 | 'or sequence like object.', query=self._query, position=part.start)
900 |
901 | elif isinstance(part, FieldQueryPart):
902 | if re.search(r'(? Any:
1042 | """Evaluate the return value of an inline query #(...) / #(...)# depending on first match or all matches.
1043 |
1044 | Arguments:
1045 | query: the query part.
1046 | obj: the current object.
1047 |
1048 | Raises:
1049 | gjson.GJSONParseError: if the query is for the first element and there are no matching items.
1050 |
1051 | Returns:
1052 | the result of the query.
1053 |
1054 | """
1055 | if query.first_only:
1056 | if obj:
1057 | return obj[0]
1058 |
1059 | raise GJSONParseError('Query for first element does not match anything.',
1060 | query=self._query, position=query.start)
1061 |
1062 | return obj
1063 |
1064 | def _parse_query(self, query: ArrayQueryQueryPart, obj: Any) -> Any: # noqa: PLR0912, PLR0915
1065 | """Parse an inline query #(...) / #(...)#.
1066 |
1067 | Arguments:
1068 | query: the query part.
1069 | obj: the current object.
1070 |
1071 | Raises:
1072 | gjson.GJSONParseError: on invalid query.
1073 |
1074 | Returns:
1075 | the result of the query.
1076 |
1077 | """
1078 | if isinstance(query.value, ArrayQueryQueryPart):
1079 | ret = []
1080 | for i in obj:
1081 | nested_obj = None
1082 | if query.field:
1083 | if isinstance(i, Mapping) and query.field in i:
1084 | nested_obj = i[query.field]
1085 | elif self._is_sequence(i):
1086 | nested_obj = i
1087 |
1088 | if nested_obj is not None and self._parse_query(query.value, nested_obj):
1089 | ret.append(i)
1090 |
1091 | return self._evaluate_query_return_value(query, ret)
1092 |
1093 | if not query.operator:
1094 | return self._evaluate_query_return_value(query, [i for i in obj if query.field in i])
1095 |
1096 | key = query.field.replace('\\', '')
1097 | try:
1098 | value = json.loads(query.value, strict=False)
1099 | except json.JSONDecodeError as ex:
1100 | position = query.start + len(query.field) + len(query.operator)
1101 | raise GJSONParseError(f'Invalid value `{query.value}` for the query key `{key}`.',
1102 | query=self._query, position=position) from ex
1103 |
1104 | if not key and query.first_only and obj and isinstance(obj[0], Mapping):
1105 | raise GJSONParseError('Query on mapping like objects require a key before the operator.',
1106 | query=self._query, position=query.start)
1107 |
1108 | oper: Callable[[Any, Any], bool]
1109 | if query.operator == '==~':
1110 | if value not in (True, False):
1111 | if query.first_only:
1112 | raise GJSONParseError(f'Queries ==~ operator requires a boolean value, got {type(value)} instead: '
1113 | f'`{value}`.', query=self._query, position=query.start + len(query.field))
1114 |
1115 | return []
1116 |
1117 | def truthy_op(obj_a: Any, obj_b: bool) -> bool: # noqa: FBT001
1118 | truthy = operator.truth(obj_a)
1119 | if obj_b:
1120 | return truthy
1121 | return not truthy
1122 |
1123 | oper = truthy_op
1124 | elif query.operator in ('==', '='):
1125 | oper = operator.eq
1126 | elif query.operator == '!=':
1127 | oper = operator.ne
1128 | elif query.operator == '<':
1129 | oper = operator.lt
1130 | elif query.operator == '<=':
1131 | oper = operator.le
1132 | elif query.operator == '>':
1133 | oper = operator.gt
1134 | elif query.operator == '>=':
1135 | oper = operator.ge
1136 | elif query.operator in ('%', '!%'):
1137 | value = str(value).replace('*', '.*').replace('?', '.')
1138 | value = f'^{value}$'
1139 | if query.operator == '%':
1140 | def match_op(obj_a: Any, obj_b: Any) -> bool:
1141 | if not isinstance(obj_a, str):
1142 | return False
1143 | return re.match(obj_b, obj_a) is not None
1144 | oper = match_op
1145 | else:
1146 | def not_match_op(obj_a: Any, obj_b: Any) -> bool:
1147 | if not isinstance(obj_a, str):
1148 | return False
1149 | return re.match(obj_b, obj_a) is None
1150 | oper = not_match_op
1151 |
1152 | try:
1153 | if key:
1154 | if query.operator == '==~': # Consider missing keys as falsy according to GJSON docs.
1155 | ret = [i for i in obj if oper(i.get(key), value)]
1156 | else:
1157 | ret = [i for i in obj if key in i and oper(i[key], value)]
1158 | else: # Query on an array of non-objects, match them directly
1159 | ret = [i for i in obj if oper(i, value)]
1160 | except TypeError:
1161 | ret = []
1162 |
1163 | return self._evaluate_query_return_value(query, ret)
1164 |
1165 | def _apply_modifier(self, modifier: ModifierQueryPart, obj: Any) -> Any:
1166 | """Apply a modifier.
1167 |
1168 | Arguments:
1169 | modifier: the modifier query part to parse.
1170 | obj: the current object before applying the modifier.
1171 |
1172 | Raises:
1173 | gjson.GJSONError: when the modifier raises an exception.
1174 | gjson.GJSONParseError: on unknown modifier.
1175 |
1176 | Returns:
1177 | the object modifier according to the modifier.
1178 |
1179 | """
1180 | try:
1181 | modifier_func = getattr(self, f'_apply_modifier_{modifier.name}')
1182 | except AttributeError:
1183 | modifier_func = self._custom_modifiers.get(modifier.name)
1184 | if modifier_func is None:
1185 | raise GJSONParseError(f'Unknown modifier @{modifier.name}.',
1186 | query=self._query, position=modifier.start) from None
1187 |
1188 | try:
1189 | return modifier_func(modifier.options, obj, last=modifier.is_last)
1190 | except GJSONError:
1191 | raise
1192 | except Exception as ex:
1193 | raise GJSONError(f'Modifier @{modifier.name} raised an exception.') from ex
1194 |
1195 | def _apply_modifier_reverse(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1196 | """Apply the @reverse modifier.
1197 |
1198 | Arguments:
1199 | options: the eventual options for the modifier, currently unused.
1200 | obj: the current object to reverse.
1201 | last: whether this is the final part of the query.
1202 |
1203 | Returns:
1204 | the reversed object. If the object cannot be reversed is returned untouched.
1205 |
1206 | """
1207 | del last # unused argument
1208 | if isinstance(obj, Mapping):
1209 | return {k: obj[k] for k in reversed(obj.keys())}
1210 | if self._is_sequence(obj):
1211 | return obj[::-1]
1212 |
1213 | return obj
1214 |
1215 | def _apply_modifier_keys(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1216 | """Apply the @keys modifier.
1217 |
1218 | Arguments:
1219 | options: the eventual options for the modifier, currently unused.
1220 | obj: the current object to get the keys from.
1221 | last: whether this is the final part of the query.
1222 |
1223 | Raises:
1224 | gjson.GJSONError: if the current object does not have a keys() method.
1225 |
1226 | Returns:
1227 | the current object keys as list.
1228 |
1229 | """
1230 | del last # unused argument
1231 | try:
1232 | return list(obj.keys())
1233 | except AttributeError as ex:
1234 | raise GJSONError('The current object does not have a keys() method.') from ex
1235 |
1236 | def _apply_modifier_values(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1237 | """Apply the @values modifier.
1238 |
1239 | Arguments:
1240 | options: the eventual options for the modifier, currently unused.
1241 | obj: the current object to get the values from.
1242 | last: whether this is the final part of the query.
1243 |
1244 | Raises:
1245 | gjson.GJSONError: if the current object does not have a values() method.
1246 |
1247 | Returns:
1248 | the current object values as list.
1249 |
1250 | """
1251 | del last # unused argument
1252 | try:
1253 | return list(obj.values())
1254 | except AttributeError as ex:
1255 | raise GJSONError('The current object does not have a values() method.') from ex
1256 |
1257 | def _apply_modifier_ugly(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1258 | """Apply the @ugly modifier to condense the output.
1259 |
1260 | Arguments:
1261 | options: the eventual options for the modifier, currently unused.
1262 | obj: the current object to uglyfy.
1263 | last: whether this is the final part of the query.
1264 |
1265 | Returns:
1266 | the current object, unmodified.
1267 |
1268 | """
1269 | del last # unused argument
1270 | self._dump_params['separators'] = (',', ':')
1271 | self._dump_params['indent'] = None
1272 | return obj
1273 |
1274 | def _apply_modifier_pretty(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1275 | """Apply the @pretty modifier to pretty-print the output.
1276 |
1277 | Arguments:
1278 | options: the eventual options for the modifier.
1279 | obj: the current object to prettyfy.
1280 | last: whether this is the final part of the query.
1281 |
1282 | Returns:
1283 | the current object, unmodified.
1284 |
1285 | """
1286 | del last # unused argument
1287 | self._dump_params['indent'] = options.get('indent', 2)
1288 | self._dump_params['sort_keys'] = options.get('sortKeys', False)
1289 | self._dump_params['prefix'] = options.get('prefix', '')
1290 | return obj
1291 |
1292 | def _apply_modifier_ascii(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1293 | """Apply the @ascii modifier to have all non-ASCII characters escaped when dumping the object.
1294 |
1295 | Arguments:
1296 | options: the eventual options for the modifier, currently unused.
1297 | obj: the current object to sort.
1298 | last: whether this is the final part of the query.
1299 |
1300 | Returns:
1301 | the current object, unmodified.
1302 |
1303 | """
1304 | del last # unused argument
1305 | self._dump_params['ensure_ascii'] = True
1306 | return obj
1307 |
1308 | def _apply_modifier_sort(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1309 | """Apply the @sort modifier, sorts mapping and sequences.
1310 |
1311 | Arguments:
1312 | options: the eventual options for the modifier, currently unused.
1313 | obj: the current object to sort.
1314 | last: whether this is the final part of the query.
1315 |
1316 | Raises:
1317 | gjson.GJSONError: if the current object is not sortable.
1318 |
1319 | Returns:
1320 | the sorted object.
1321 |
1322 | """
1323 | del last # unused argument
1324 | if isinstance(obj, Mapping):
1325 | return {k: obj[k] for k in sorted(obj.keys())}
1326 | if self._is_sequence(obj):
1327 | return sorted(obj)
1328 |
1329 | raise GJSONError(f'@sort modifier not supported for object of type {type(obj)}. '
1330 | 'Expected a mapping or sequence like object.')
1331 |
1332 | def _apply_modifier_valid(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1333 | """Apply the @valid modifier, checking that the current object can be converted to JSON.
1334 |
1335 | Arguments:
1336 | options: the eventual options for the modifier, currently unused.
1337 | obj: the current element to validate.
1338 | last: whether this is the final part of the query.
1339 |
1340 | Raises:
1341 | gjson.GJSONError: if the current object cannot be converted to JSON.
1342 |
1343 | Returns:
1344 | the current object, unmodified.
1345 |
1346 | """
1347 | del last # unused argument
1348 | try:
1349 | json.dumps(obj, **self._dump_params)
1350 | except Exception as ex:
1351 | raise GJSONError('The current object cannot be converted to JSON.') from ex
1352 |
1353 | return obj
1354 |
1355 | def _apply_modifier_this(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1356 | """Apply the @this modifier, that returns the current object.
1357 |
1358 | Arguments:
1359 | options: the eventual options for the modifier, currently unused.
1360 | obj: the current element to return.
1361 | last: whether this is the final part of the query.
1362 |
1363 | Returns:
1364 | the current object, unmodified.
1365 |
1366 | """
1367 | del last # unused argument
1368 | return obj
1369 |
1370 | def _apply_modifier_fromstr(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1371 | """Apply the @fromstr modifier, converting a string to JSON, if valid.
1372 |
1373 | Arguments:
1374 | options: the eventual options for the modifier, currently unused.
1375 | obj: the current element from where to extract the JSON.
1376 | last: whether this is the final part of the query.
1377 |
1378 | Raises:
1379 | gjson.GJSONError: if the current object cannot be converted to JSON.
1380 |
1381 | Returns:
1382 | the parsed JSON.
1383 |
1384 | """
1385 | del last # unused argument
1386 | if not isinstance(obj, (str, bytes)):
1387 | raise GJSONError(f'Modifier @fromstr got object of type {type(obj)} as input, expected string or bytes.')
1388 |
1389 | try:
1390 | return json.loads(obj, strict=False)
1391 | except Exception as ex:
1392 | raise GJSONError('The current @fromstr input object cannot be converted to JSON.') from ex
1393 |
1394 | def _apply_modifier_tostr(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1395 | """Apply the @tostr modifier, converting the current object to a JSON-encoded string, if valid.
1396 |
1397 | Arguments:
1398 | options: the eventual options for the modifier, currently unused.
1399 | obj: the current element from where to extract the JSON.
1400 | last: whether this is the final part of the query.
1401 |
1402 | Raises:
1403 | gjson.GJSONError: if the current object cannot be converted to a JSON-encoded string.
1404 |
1405 | Returns:
1406 | the JSON-encoded string.
1407 |
1408 | """
1409 | del last # unused argument
1410 | try:
1411 | return json.dumps(obj, ensure_ascii=False)
1412 | except Exception as ex:
1413 | raise GJSONError('The current object cannot be converted to a JSON-encoded string for @tostr.') from ex
1414 |
1415 | def _apply_modifier_group(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1416 | """Apply the @group modifier, that groups a dictionary of lists in a list of dictionaries.
1417 |
1418 | Example input::
1419 |
1420 | {"id": ["123", "456", "789"], "val": [2, 1]}
1421 |
1422 | Example output::
1423 |
1424 | [{"id": "123", "val": 2}, {"id": "456", "val": 1}, {"id": "789"}]
1425 |
1426 | Arguments:
1427 | options: the eventual options for the modifier, currently unused.
1428 | obj: the current element to group.
1429 | last: whether this is the final part of the query.
1430 |
1431 | Raises:
1432 | gjson.GJSONError: if the current object is not a dictionary.
1433 |
1434 | Returns:
1435 | a list with the grouped objects or an empty list if the input has no lists as values.
1436 |
1437 | """
1438 | del last # unused argument
1439 | if not isinstance(obj, Mapping):
1440 | raise GJSONError(f'Modifier @group got object of type {type(obj)} as input, expected dictionary.')
1441 |
1442 | # Skip all values that aren't lists:
1443 | obj = {k: v for k, v in obj.items() if self._is_sequence(v)}
1444 | # Fill missing values with NoResult to remove them afterwards
1445 | obj = [dict(zip_longest(obj.keys(), values)) for values in zip_longest(*obj.values(), fillvalue=NoResult())]
1446 | # Skip keys with value NoResult in each dictionary
1447 | return [{k: v for k, v in i.items() if not isinstance(v, NoResult)} for i in obj]
1448 |
1449 | def _apply_modifier_join(self, _options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1450 | """Apply the @join modifier, that joins a list of dictionaries into a single dictionary.
1451 |
1452 | Items in the sequence that are not dictionaries are skipped.
1453 | Differently from GJSON there is no support for duplicated keys as the can't exist in Python dictionaries.
1454 | Hence this modifier doesn't accept any option.
1455 |
1456 | Example input::
1457 |
1458 | [{"first": "Tom", "age": 37}, {"age": 41}]
1459 |
1460 | Example output::
1461 |
1462 | {"first": "Tom", "age":41}
1463 |
1464 | Arguments:
1465 | options: the eventual options for the modifier, currently unused.
1466 | obj: the current element to join.
1467 | last: whether this is the final part of the query.
1468 |
1469 | Returns:
1470 | the object untouched if the object is not a sequence, a dictionary with joined objects otherwise.
1471 |
1472 | """
1473 | del last # unused argument
1474 | if not self._is_sequence(obj):
1475 | return obj
1476 |
1477 | ret: dict[Any, Any] = {}
1478 | for item in obj:
1479 | if isinstance(item, Mapping):
1480 | ret.update(item)
1481 |
1482 | return ret
1483 |
1484 | def _apply_modifier_top_n(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1485 | """Apply the @top_n modifier to find the most common values of a given field.
1486 |
1487 | Arguments:
1488 | options: the eventual modifier options. If not specified all items are returned. If specified it must
1489 | contain a 'n' key with the number of top N to return.
1490 | obj: the current object to extract the N most common items.
1491 | last: whether this is the final part of the query.
1492 |
1493 | Raises:
1494 | gjson.GJSONError: if the current object is not a sequence.
1495 |
1496 | Returns:
1497 | dict: a dictionary of unique items as keys and the count as value.
1498 |
1499 | """
1500 | del last # unused argument
1501 | if not self._is_sequence(obj):
1502 | raise GJSONError(f'@top_n modifier not supported for object of type {type(obj)}. '
1503 | 'Expected a sequence like object.')
1504 |
1505 | return dict(Counter(obj).most_common(options.get('n')))
1506 |
1507 | def _apply_modifier_sum_n(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1508 | """Apply the @sum_n modifier that groups the values of a given key while summing the values of another key.
1509 |
1510 | The key used to sum must have numeric values.
1511 |
1512 | Arguments:
1513 | options: the modifier options. It must contain a 'group' key with the name of the field to use to group the
1514 | items as value and a 'sum' key with the name of the field to use to sum the values for each unique
1515 | grouped identifier. If a 'n' key is also provided, only the top N results are returned. If not
1516 | specified all items are returned.
1517 | obj: the current object to group and sum the top N values.
1518 | last: whether this is the final part of the query.
1519 |
1520 | Raises:
1521 | gjson.GJSONError: if the current object is not a sequence.
1522 |
1523 | Returns:
1524 | dict: a dictionary of unique items as keys and the sum as value.
1525 |
1526 | """
1527 | del last # unused argument
1528 | if not self._is_sequence(obj):
1529 | raise GJSONError(f'@sum_n modifier not supported for object of type {type(obj)}. '
1530 | 'Expected a sequence like object.')
1531 |
1532 | results: Counter[Any] = Counter()
1533 | for item in obj:
1534 | results[item[options['group']]] += item[options['sum']]
1535 | return dict(results.most_common(options.get('n')))
1536 |
1537 | def _apply_modifier_flatten(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any:
1538 | """Apply the @flatten modifier.
1539 |
1540 | Arguments:
1541 | options: the eventual modifier options.
1542 | obj: the current object to flatten.
1543 | last: whether this is the final part of the query.
1544 |
1545 | Returns:
1546 | the modified object.
1547 |
1548 | """
1549 | del last # unused argument
1550 | if not self._is_sequence(obj):
1551 | return obj
1552 |
1553 | return list(self._flatten_sequence(obj, deep=options.get('deep', False)))
1554 |
1555 | def _flatten_sequence(self, obj: Any, *, deep: bool = False) -> Any:
1556 | """Flatten nested sequences in the given object.
1557 |
1558 | Arguments:
1559 | obj: the current object to flatten
1560 | deep: if :py:data:`True` recursively flatten nested sequences. By default only the first level is
1561 | processed.
1562 |
1563 | Returns:
1564 | the flattened object if it was a flattable sequence, the given object itself otherwise.
1565 |
1566 | """
1567 | for elem in obj:
1568 | if self._is_sequence(elem):
1569 | if deep:
1570 | yield from self._flatten_sequence(elem, deep=deep)
1571 | else:
1572 | yield from elem
1573 | else:
1574 | yield elem
1575 |
--------------------------------------------------------------------------------
/gjson/_protocols.py:
--------------------------------------------------------------------------------
1 | """Typing protocols used by the gjson package."""
2 | from typing import Any, Protocol, runtime_checkable
3 |
4 |
5 | @runtime_checkable
6 | class ModifierProtocol(Protocol):
7 | """Callback protocol for the custom modifiers."""
8 |
9 | def __call__(self, options: dict[str, Any], obj: Any, *, last: bool) -> Any:
10 | """To register a custom modifier a callable that adhere to this protocol must be provided.
11 |
12 | Examples:
13 | Register a custom modifier that sums all the numbers in a list:
14 |
15 | >>> import gjson
16 | >>> data = [1, 2, 3, 4, 5]
17 | >>> def custom_sum(options, obj, *, last):
18 | ... # insert sanity checks code here
19 | ... return sum(obj)
20 | ...
21 | >>> gjson_obj = gjson.GJSON(data)
22 | >>> gjson_obj.register_modifier('sum', custom_sum)
23 | >>> gjson_obj.get('@sum')
24 | 15
25 |
26 | Arguments:
27 | options: a dictionary of options. If no options are present in the query the callable will be called with
28 | an empty dictionary as options. The modifier can supports any number of options, or none.
29 | obj: the current object already modifier by any previous parts of the query.
30 | last: :py:data:`True` if the modifier is the last element in the query or :py:data:`False` otherwise.
31 |
32 | Raises:
33 | Exception: any exception that might be raised by the callable is catched by gjson and re-raised as a
34 | :py:class:`gjson.GJSONError` exception to ensure that the normal gjson behaviour is respected according
35 | to the selected verbosity (CLI) or ``quiet`` parameter (Python library).
36 |
37 | Returns:
38 | the resulting object after applying the modifier.
39 |
40 | """
41 |
--------------------------------------------------------------------------------
/gjson/exceptions.py:
--------------------------------------------------------------------------------
1 | """gjson custom exceptions module."""
2 | from typing import Any
3 |
4 |
5 | class GJSONError(Exception):
6 | """Raised by the gjson module on error while performing queries or converting to JSON."""
7 |
8 |
9 | class GJSONParseError(GJSONError):
10 | """Raised when there is an error parsing the query string, with nicer representation of the error."""
11 |
12 | def __init__(self, *args: Any, query: str, position: int):
13 | """Initialize the exception with the additional data of the query part.
14 |
15 | Arguments:
16 | *args: all positional arguments like any regular exception.
17 | query: the full query that generated the parse error.
18 | position: the position in the query string where the parse error occurred.
19 |
20 | """
21 | super().__init__(*args)
22 | self.query = query
23 | self.position = position
24 |
25 | def __str__(self) -> str:
26 | """Return a custom representation of the error.
27 |
28 | Returns:
29 | the whole query string with a clear indication on where the error occurred.
30 |
31 | """
32 | default = super().__str__()
33 | line = '-' * (self.position + 7) # 7 is for the lenght of 'Query: '
34 | return f'{default}\nQuery: {self.query}\n{line}^'
35 |
36 |
37 | class GJSONInvalidSyntaxError(GJSONParseError):
38 | """Raised when there is a query with an invalid syntax."""
39 |
--------------------------------------------------------------------------------
/gjson/py.typed:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/volans-/gjson-py/1cdc7d0633cdd7d21bd61c8d9cf1eab5d3b311fe/gjson/py.typed
--------------------------------------------------------------------------------
/prospector.yaml:
--------------------------------------------------------------------------------
1 | strictness: high
2 | inherits:
3 | - strictness_high
4 |
5 | doc-warnings: true
6 | member-warnings: true
7 | test-warnings: true
8 |
9 | autodetect: false
10 | output-format: grouped
11 |
12 | ignore-paths:
13 | - doc/source/conf.py
14 |
15 | pyroma:
16 | run: true
17 |
18 | vulture:
19 | run: true
20 |
--------------------------------------------------------------------------------
/ruff.toml:
--------------------------------------------------------------------------------
1 | target-version = 'py39'
2 | line-length = 120
3 | select = ['F', 'E', 'W', 'C90', 'B', 'I', 'N', 'D', 'YTT', 'ANN', 'S', 'FBT', 'A', 'C4', 'DTZ', 'T10', 'EXE', 'ISC',
4 | 'G', 'INP', 'T20', 'PT', 'Q', 'RSE', 'RET', 'SLF', 'SIM', 'TID', 'TCH', 'ARG', 'PTH', 'ERA', 'PL', 'TRY',
5 | 'RUF']
6 | ignore = [
7 | 'D203',
8 | 'D213',
9 | 'D406',
10 | 'D407',
11 | 'ANN101',
12 | 'ANN401',
13 | 'TRY003',
14 | 'C901',
15 | 'N999',
16 | ]
17 |
18 | [per-file-ignores]
19 | 'doc/source/conf.py' = ['A001', 'ERA001', 'INP001']
20 | 'gjson/_cli.py' = ['T201']
21 | 'setup.py' = ['EXE001']
22 |
23 | [flake8-annotations]
24 | mypy-init-return = true
25 |
26 | [flake8-bandit]
27 | check-typed-exception = true
28 |
29 | [flake8-pytest-style]
30 | parametrize-values-type = 'tuple'
31 | raises-extend-require-match-for = [
32 | 'GJSONError',
33 | 'GJSONParseError',
34 | 'GJSONInvalidSyntaxError',
35 | ]
36 |
37 | [flake8-quotes]
38 | inline-quotes = 'single'
39 |
40 | [pydocstyle]
41 | convention = 'google'
42 |
--------------------------------------------------------------------------------
/setup.cfg:
--------------------------------------------------------------------------------
1 | [aliases]
2 | test = pytest
3 |
4 | [build_sphinx]
5 | project = gjson-py
6 | source-dir = doc/source
7 | build-dir = doc/build
8 |
9 | [mypy]
10 | disallow_untyped_calls = True
11 | disallow_untyped_defs = True
12 | disallow_incomplete_defs = True
13 | disallow_untyped_decorators = True
14 | no_implicit_optional = True
15 | warn_redundant_casts = True
16 | warn_unused_ignores = True
17 | warn_return_any = True
18 | warn_unreachable = True
19 | strict_equality = True
20 | strict = True
21 | show_error_context = True
22 | show_column_numbers = True
23 | show_error_codes = True
24 | pretty = True
25 | warn_incomplete_stub = True
26 | warn_unused_configs = True
27 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | """Package configuration."""
3 | from pathlib import Path
4 |
5 | from setuptools import find_packages, setup
6 |
7 | # Extra dependencies
8 | extras_require = {
9 | # Test dependencies
10 | 'tests': [
11 | 'bandit',
12 | 'flake8',
13 | 'flake8-import-order',
14 | 'mypy',
15 | 'pytest-cov',
16 | 'pytest-xdist',
17 | 'pytest',
18 | 'ruff',
19 | 'sphinx_rtd_theme>=1.0',
20 | 'sphinx-argparse',
21 | 'sphinx-autodoc-typehints',
22 | 'Sphinx',
23 | 'types-pkg_resources',
24 | ],
25 | 'prospector': [
26 | 'prospector[with_everything]',
27 | 'pytest',
28 | ],
29 | }
30 |
31 | setup_requires = [
32 | 'pytest-runner',
33 | 'setuptools_scm',
34 | ]
35 |
36 | setup(
37 | author='Riccardo Coccioli',
38 | author_email='volans-@users.noreply.github.com',
39 | classifiers=[
40 | 'Development Status :: 5 - Production/Stable',
41 | 'Environment :: Console',
42 | 'Intended Audience :: Developers',
43 | 'Intended Audience :: End Users/Desktop',
44 | 'Intended Audience :: Information Technology',
45 | 'Intended Audience :: Science/Research',
46 | 'Intended Audience :: System Administrators',
47 | 'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
48 | 'Operating System :: OS Independent',
49 | 'Programming Language :: Python :: 3.9',
50 | 'Programming Language :: Python :: 3.10',
51 | 'Programming Language :: Python :: 3.11',
52 | 'Programming Language :: Python :: 3.12',
53 | 'Programming Language :: Python :: 3 :: Only',
54 | 'Topic :: Software Development :: Libraries :: Python Modules',
55 | 'Topic :: Text Processing',
56 | 'Topic :: Utilities',
57 | 'Typing :: Typed',
58 | ],
59 | description=('gjson-py is a Python package that provides a simple way to filter and extract data from JSON-like '
60 | 'objects or JSON files, using the GJSON syntax.'),
61 | entry_points={
62 | 'console_scripts': [
63 | 'gjson = gjson._cli:cli',
64 | ],
65 | },
66 | extras_require=extras_require,
67 | install_requires=[],
68 | keywords=['gjson', 'json'],
69 | license='GPLv3+',
70 | long_description=Path('README.rst').read_text(),
71 | long_description_content_type='text/x-rst',
72 | name='gjson',
73 | package_data={'gjson': ['py.typed']},
74 | packages=find_packages(),
75 | platforms=['GNU/Linux', 'BSD', 'MacOSX'],
76 | python_requires='>=3.9',
77 | setup_requires=setup_requires,
78 | url='https://github.com/volans-/gjson-py',
79 | use_scm_version=True,
80 | zip_safe=False,
81 | )
82 |
--------------------------------------------------------------------------------
/tests/ruff.toml:
--------------------------------------------------------------------------------
1 | extend = "../ruff.toml"
2 | ignore = [
3 | 'ANN',
4 | 'S101',
5 | 'PLR2004',
6 | ]
7 |
8 |
--------------------------------------------------------------------------------
/tests/unit/__init__.py:
--------------------------------------------------------------------------------
1 | """gjson unit tests."""
2 |
--------------------------------------------------------------------------------
/tests/unit/test__cli.py:
--------------------------------------------------------------------------------
1 | """CLI test module."""
2 | import argparse
3 | import io
4 | import json
5 |
6 | import pytest
7 | from gjson._cli import cli
8 | from gjson.exceptions import GJSONParseError
9 |
10 | from .test_init import INPUT_JSON, INPUT_LINES, INPUT_LINES_WITH_ERRORS
11 |
12 |
13 | def test_cli_help(capsys):
14 | """It should read the data from stdin and query it."""
15 | with pytest.raises(SystemExit) as exc_info:
16 | cli(['--help'])
17 |
18 | assert exc_info.value.args[0] == 0
19 | captured = capsys.readouterr()
20 | assert 'usage: gjson' in captured.out
21 | assert 'See also the full documentation available at' in captured.out
22 | assert not captured.err
23 |
24 |
25 | def test_cli_stdin(monkeypatch, capsys):
26 | """It should read the data from stdin and query it."""
27 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON))
28 | ret = cli(['-', 'name.first'])
29 | assert ret == 0
30 | captured = capsys.readouterr()
31 | assert captured.out == '"Tom"\n'
32 | assert not captured.err
33 |
34 |
35 | def test_cli_file(tmp_path, capsys):
36 | """It should read the data from the provided file and query it."""
37 | data_file = tmp_path / 'input.json'
38 | data_file.write_text(INPUT_JSON)
39 | ret = cli([str(data_file), 'name.first'])
40 | assert ret == 0
41 | captured = capsys.readouterr()
42 | assert captured.out == '"Tom"\n'
43 | assert not captured.err
44 |
45 |
46 | @pytest.mark.parametrize('query', (
47 | 'a\tkey',
48 | '..0.a\tkey',
49 | ))
50 | def test_cli_file_with_control_chars(query, tmp_path, capsys):
51 | """It should read the data from the provided file and query it."""
52 | data_file = tmp_path / 'input.json'
53 | data_file.write_text('{"a\tkey": "a\tvalue"}')
54 | ret = cli(['-vvv', str(data_file), query])
55 | assert ret == 0
56 | captured = capsys.readouterr()
57 | assert captured.out == '"a\\tvalue"\n'
58 | assert not captured.err
59 |
60 |
61 | @pytest.mark.parametrize('query', (
62 | 'a\tkey',
63 | '..0.a\tkey',
64 | ))
65 | @pytest.mark.parametrize('lines', (True, False))
66 | def test_cli_stdin_with_control_chars(query, lines, monkeypatch, capsys):
67 | """It should read the data from stdin and query it."""
68 | monkeypatch.setattr('sys.stdin', io.StringIO('{"a\tkey": "a\tvalue"}'))
69 | params = ['-vvv', '-', query]
70 | if lines:
71 | params.insert(0, '--lines')
72 |
73 | ret = cli(params)
74 | assert ret == 0
75 | captured = capsys.readouterr()
76 | assert captured.out == '"a\\tvalue"\n'
77 | assert not captured.err
78 |
79 |
80 | def test_cli_nonexistent_file(tmp_path, capsys):
81 | """It should exit with a failure exit code and no output."""
82 | ret = cli([str(tmp_path / 'nonexistent.json'), 'name.first'])
83 | assert ret == 1
84 | captured = capsys.readouterr()
85 | assert not captured.out
86 | assert not captured.err
87 |
88 |
89 | def test_cli_nonexistent_file_verbosity_1(tmp_path, capsys):
90 | """It should exit with a failure exit code and print the error message."""
91 | ret = cli(['-v', str(tmp_path / 'nonexistent.json'), 'name.first'])
92 | assert ret == 1
93 | captured = capsys.readouterr()
94 | assert not captured.out
95 | assert captured.err.startswith("ArgumentTypeError: can't open")
96 | assert 'nonexistent.json' in captured.err
97 |
98 |
99 | def test_cli_nonexistent_file_verbosity_2(tmp_path):
100 | """It should raise the exception and print the full traceback."""
101 | with pytest.raises(
102 | argparse.ArgumentTypeError, match=r"can't open .*/nonexistent.json.* No such file or directory"):
103 | cli(['-vv', str(tmp_path / 'nonexistent.json'), 'name.first'])
104 |
105 |
106 | def test_cli_stdin_query_verbosity_1(monkeypatch, capsys):
107 | """It should exit with a failure exit code and print the error message."""
108 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON))
109 | ret = cli(['-v', '-', 'nonexistent'])
110 | assert ret == 1
111 | captured = capsys.readouterr()
112 | assert not captured.out
113 | assert captured.err == ('GJSONParseError: Mapping object does not have key `nonexistent`.\n'
114 | 'Query: nonexistent\n-------^\n')
115 |
116 |
117 | def test_cli_stdin_query_verbosity_2(monkeypatch):
118 | """It should exit with a failure exit code and print the full traceback."""
119 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_JSON))
120 | with pytest.raises(GJSONParseError, match=r'Mapping object does not have key `nonexistent`.'):
121 | cli(['-vv', '-', 'nonexistent'])
122 |
123 |
124 | def test_cli_lines_ok(monkeypatch, capsys):
125 | """It should apply the same query to each line."""
126 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES))
127 | ret = cli(['--lines', '-', 'name'])
128 | assert ret == 0
129 | captured = capsys.readouterr()
130 | assert captured.out == '"Gilbert"\n"Alexa"\n"May"\n"Deloise"\n'
131 | assert not captured.err
132 |
133 |
134 | def test_cli_lines_failed_lines_verbosity_0(monkeypatch, capsys):
135 | """It should keep going with the other lines and just skip the failed line."""
136 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
137 | ret = cli(['--lines', '-', 'name'])
138 | assert ret == 1
139 | captured = capsys.readouterr()
140 | assert captured.out == '"Gilbert"\n"Deloise"\n'
141 | assert not captured.err
142 |
143 |
144 | def test_cli_lines_failed_lines_verbosity_1(monkeypatch, capsys):
145 | """It should keep going with the other lines printing an error for the failed lines."""
146 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
147 | ret = cli(['-v', '--lines', '-', 'name'])
148 | assert ret == 1
149 | captured = capsys.readouterr()
150 | assert captured.out == '"Gilbert"\n"Deloise"\n'
151 | assert captured.err.count('JSONDecodeError') == 2
152 |
153 |
154 | def test_cli_lines_failed_lines_verbosity_2(monkeypatch):
155 | """It should interrupt the processing and print the full traceback."""
156 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
157 | with pytest.raises(
158 | json.decoder.JSONDecodeError, match=r'Expecting property name enclosed in double quotes'):
159 | cli(['-vv', '--lines', '-', 'name'])
160 |
161 |
162 | def test_cli_lines_double_dot_query(monkeypatch, capsys):
163 | """It should encapsulate each line in an array to allow queries."""
164 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES))
165 | ret = cli(['--lines', '..#(age>45).name'])
166 | assert ret == 1
167 | captured = capsys.readouterr()
168 | assert captured.out == '"Gilbert"\n"May"\n'
169 | assert not captured.err
170 |
171 |
172 | def test_cli_double_dot_query_ok(monkeypatch, capsys):
173 | """It should encapsulate the input in an array and apply the query to the array."""
174 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES))
175 | ret = cli(['-', '..#.name'])
176 | assert ret == 0
177 | captured = capsys.readouterr()
178 | assert captured.out == '["Gilbert", "Alexa", "May", "Deloise"]\n'
179 | assert not captured.err
180 |
181 |
182 | def test_cli_double_dot_query_failed_lines_verbosity_0(monkeypatch, capsys):
183 | """It should encapsulate the input in an array skipping failing lines."""
184 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
185 | ret = cli(['-', '..#.name'])
186 | assert ret == 1
187 | captured = capsys.readouterr()
188 | assert not captured.out
189 | assert not captured.err
190 |
191 |
192 | def test_cli_double_dot_query_failed_lines_verbosity_1(monkeypatch, capsys):
193 | """It should encapsulate the input in an array skipping failing lines and printing an error for each failure."""
194 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
195 | ret = cli(['-v', '-', '..#.name'])
196 | assert ret == 1
197 | captured = capsys.readouterr()
198 | assert not captured.out
199 | assert captured.err.startswith('JSONDecodeError: Expecting property name enclosed in double quotes')
200 |
201 |
202 | def test_cli_double_dot_query_failed_lines_verbosity_2(monkeypatch):
203 | """It should interrupt the execution at the first invalid line and exit printing the traceback."""
204 | monkeypatch.setattr('sys.stdin', io.StringIO(INPUT_LINES_WITH_ERRORS))
205 | with pytest.raises(json.decoder.JSONDecodeError, match=r'Expecting property name enclosed in double quotes'):
206 | cli(['-vv', '-', '..#.name'])
207 |
--------------------------------------------------------------------------------
/tests/unit/test_init.py:
--------------------------------------------------------------------------------
1 | """GJSON test module."""
2 | import json
3 | import re
4 | from collections.abc import Mapping
5 | from math import isnan
6 |
7 | import gjson
8 | import pytest
9 | from gjson._gjson import MODIFIER_NAME_RESERVED_CHARS
10 |
11 | INPUT_JSON = """
12 | {
13 | "name": {"first": "Tom", "last": "Anderson"},
14 | "age":37,
15 | "children": ["Sara","Alex","Jack"],
16 | "fav.movie": "Deer Hunter",
17 | "friends": [
18 | {"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]},
19 | {"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]},
20 | {"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]}
21 | ]
22 | }
23 | """
24 | INPUT_OBJECT = json.loads(INPUT_JSON)
25 | INPUT_LIST = json.loads("""
26 | [
27 | {"first": "Dale"},
28 | {"first": "Jane"},
29 | {"last": "Murphy"}
30 | ]
31 | """)
32 | INPUT_ESCAPE = json.loads("""
33 | {
34 | "test": {
35 | "*":"valZ",
36 | "*v":"val0",
37 | "keyv*":"val1",
38 | "key*v":"val2",
39 | "keyv?":"val3",
40 | "key?v":"val4",
41 | "keyv.":"val5",
42 | "key.v":"val6",
43 | "keyk*":{"key?":"val7"},
44 | "1key":"val8"
45 | }
46 | }
47 | """)
48 | # This json block is poorly formed on purpose.
49 | INPUT_BASIC = json.loads("""
50 | {"age":100, "name2":{"here":"B\\\\\\"R"},
51 | "noop":{"what is a wren?":"a bird"},
52 | "happy":true,"immortal":false,
53 | "items":[1,2,3,{"tags":[1,2,3],"points":[[1,2],[3,4]]},4,5,6,7],
54 | "arr":["1",2,"3",{"hello":"world"},"4",5],
55 | "vals":[1,2,3],"name":{"first":"tom","last":null},
56 | "created":"2014-05-16T08:28:06.989Z",
57 | "loggy":{
58 | "programmers": [
59 | {
60 | "firstName": "Brett",
61 | "lastName": "McLaughlin",
62 | "email": "aaaa",
63 | "tag": "good"
64 | },
65 | {
66 | "firstName": "Jason",
67 | "lastName": "Hunter",
68 | "email": "bbbb",
69 | "tag": "bad"
70 | },
71 | {
72 | "firstName": "Elliotte",
73 | "lastName": "Harold",
74 | "email": "cccc",
75 | "tag": "good"
76 | },
77 | {
78 | "firstName": 1002.3,
79 | "age": 101
80 | }
81 | ]
82 | },
83 | "lastly":{"end...ing":"soon","yay":"final"}
84 | }
85 | """)
86 | INPUT_LINES = """
87 | {"name": "Gilbert", "age": 61}
88 | {"name": "Alexa", "age": 34}
89 | {"name": "May", "age": 57}
90 | {"name": "Deloise", "age": 44}
91 | """
92 | INPUT_LINES_WITH_ERRORS = """
93 | {"name": "Gilbert", "age": 61}
94 | {invalid
95 | {invalid
96 | {"name": "Deloise", "age": 44}
97 | """
98 | INPUT_TRUTHINESS = json.loads("""
99 | {
100 | "vals": [
101 | { "a": 1, "b": true },
102 | { "a": 2, "b": true },
103 | { "a": 3, "b": false },
104 | { "a": 4, "b": "0" },
105 | { "a": 5, "b": 0 },
106 | { "a": 6, "b": "1" },
107 | { "a": 7, "b": 1 },
108 | { "a": 8, "b": "true" },
109 | { "a": 9, "b": false },
110 | { "a": 10, "b": null },
111 | { "a": 11 }
112 | ]
113 | }
114 | """)
115 | INPUT_SUM_N = json.loads("""
116 | [
117 | {"key": "a", "value": 1, "other": "value"},
118 | {"key": "b", "value": 2},
119 | {"key": "c", "value": 3, "other": "value"},
120 | {"key": "a", "value": 7},
121 | {"key": "b", "value": 1.5},
122 | {"key": "d", "value": 4},
123 | {"key": "c", "value": 9}
124 | ]
125 | """)
126 | INPUT_NESTED_QUERIES = json.loads("""
127 | {
128 | "key": [
129 | {"level1": [{"level2": [{"level3": [1, 2]}]}]},
130 | {"level1": [{"level2": [{"level3": [2, 3]}]}]},
131 | [[{"level3": [1, 2]}], [{"level3": [2, 3]}]],
132 | [[{"level3": [2, 3]}], [{"level3": [3, 4]}]],
133 | {"another": [{"level2": [{"level3": [2, 3]}]}]},
134 | {"level1": [{"another": [{"level3": [2, 3]}]}]},
135 | {"level1": [{"level2": [{"another": [2, 3]}]}]},
136 | [[{"another": [2, 3]}], [{"another": [3, 4]}]],
137 | "spurious",
138 | 12.34,
139 | {"mixed": [[{"level4": [1, 2]}]]}
140 | ]
141 | }
142 | """)
143 |
144 |
145 | def compare_values(result, expected):
146 | """Compare results with the expected values ensuring same-order of keys for dictionaries."""
147 | if isinstance(expected, float) and isnan(expected):
148 | assert isnan(result)
149 | return
150 |
151 | assert result == expected
152 | if isinstance(expected, Mapping):
153 | assert list(result.keys()) == list(expected.keys())
154 |
155 |
156 | class TestObject:
157 | """Testing gjson with a basic input object."""
158 |
159 | def setup_method(self):
160 | """Initialize the test instance."""
161 | def upper(options, obj, *, last):
162 | """Change the case of the object returning its upper version. Custom modifier to be used in tests."""
163 | del options
164 | del last
165 | if isinstance(obj, list):
166 | return [i.upper() for i in obj]
167 |
168 | return obj.upper()
169 |
170 | self.object = gjson.GJSON(INPUT_OBJECT)
171 | self.object.register_modifier('upper', upper)
172 |
173 | @pytest.mark.parametrize(('query', 'expected'), (
174 | # Basic
175 | ('name.last', 'Anderson'),
176 | ('name.first', 'Tom'),
177 | ('age', 37),
178 | ('children', ['Sara', 'Alex', 'Jack']),
179 | ('children.0', 'Sara'),
180 | ('children.1', 'Alex'),
181 | ('friends.1', {'first': 'Roger', 'last': 'Craig', 'age': 68, 'nets': ['fb', 'tw']}),
182 | ('friends.1.first', 'Roger'),
183 | # Wildcards
184 | ('*.first', 'Tom'),
185 | ('?a??.first', 'Tom'),
186 | ('child*.2', 'Jack'),
187 | ('c?ildren.0', 'Sara'),
188 | # Escape characters
189 | (r'fav\.movie', 'Deer Hunter'),
190 | # Arrays
191 | ('friends.#', 3),
192 | ('friends.#.age', [44, 68, 47]),
193 | ('friends.#.first', ['Dale', 'Roger', 'Jane']),
194 | ('friends.#.nets.0', ['ig', 'fb', 'ig']),
195 | # Queries
196 | ('friends.#(last=="Murphy").first', 'Dale'),
197 | ('friends.#(last=="Murphy")#.first', ['Dale', 'Jane']),
198 | ('friends.#(=="Murphy")#', []),
199 | ('friends.#(=="Mu)(phy")#', []),
200 | ('friends.#(=="Mur\tphy")#', []),
201 | ('friends.#(age\\===44)#', []),
202 | ('friends.#(age>47)#.last', ['Craig']),
203 | ('friends.#(age>=47)#.last', ['Craig', 'Murphy']),
204 | ('friends.#(age<47)#.last', ['Murphy']),
205 | ('friends.#(age<=47)#.last', ['Murphy', 'Murphy']),
206 | ('friends.#(age==44)#.last', ['Murphy']),
207 | ('friends.#(age!=44)#.last', ['Craig', 'Murphy']),
208 | ('friends.#(first%"D*").last', 'Murphy'),
209 | ('friends.#(first!%"D*").last', 'Craig'),
210 | ('friends.#(first!%"D???").last', 'Craig'),
211 | ('friends.#(%0)#', []),
212 | ('friends.#(>40)#', []),
213 | ('children.#(!%"*a*")', 'Alex'),
214 | ('children.#(%"*a*")#', ['Sara', 'Jack']),
215 | # Nested queries
216 | ('friends.#(nets.#(=="fb"))#.first', ['Dale', 'Roger']),
217 | # Modifiers
218 | ('children.@reverse', ['Jack', 'Alex', 'Sara']),
219 | ('children.@reverse.0', 'Jack'),
220 | ('name.@reverse', {'last': 'Anderson', 'first': 'Tom'}),
221 | ('age.@reverse', 37),
222 | ('@keys', ['name', 'age', 'children', 'fav.movie', 'friends']),
223 | ('name.@values', ['Tom', 'Anderson']),
224 | ('age.@flatten', 37),
225 | ('@pretty:{"indent": 4}', INPUT_OBJECT),
226 | (r'fav\.movie.@pretty:{"indent": 4}', 'Deer Hunter'),
227 | ('name.@tostr', '{"first": "Tom", "last": "Anderson"}'),
228 | ('name.@join', {'first': 'Tom', 'last': 'Anderson'}),
229 | ('age.@join', 37),
230 | ('children.@join', {}),
231 | ('children.0.@join', 'Sara'),
232 | ('friends.@join', {'first': 'Jane', 'last': 'Murphy', 'age': 47, 'nets': ['ig', 'tw']}),
233 | # Dot vs Pipe
234 | ('friends.0.first', 'Dale'),
235 | ('friends|0.first', 'Dale'),
236 | ('friends.0|first', 'Dale'),
237 | ('friends|0|first', 'Dale'),
238 | ('friends|#', 3),
239 | ('friends.#(last="Murphy")#',
240 | [{'first': 'Dale', 'last': 'Murphy', 'age': 44, 'nets': ['ig', 'fb', 'tw']},
241 | {'first': 'Jane', 'last': 'Murphy', 'age': 47, 'nets': ['ig', 'tw']}]),
242 | ('friends.#(last="Murphy")#.first', ['Dale', 'Jane']),
243 | ('friends.#(last="Murphy")#.0', []),
244 | ('friends.#(last="Murphy")#|0', {'first': 'Dale', 'last': 'Murphy', 'age': 44, 'nets': ['ig', 'fb', 'tw']}),
245 | ('friends.#(last="Murphy")#.#', []),
246 | ('friends.#(last="Murphy")#|#', 2),
247 | # Multipaths objects
248 | ('{}', {}),
249 | ('{.}', {}),
250 | ('{.invalid}', {}),
251 | ('{.invalid,}', {}),
252 | ('{age}', {'age': 37}),
253 | (r'{a\ge}', {r'a\ge': 37}),
254 | (r'{"a\\ge":age}', {r'a\ge': 37}),
255 | ('{"a\tb":age}', {'a\tb': 37}),
256 | ('{"key":age}', {'key': 37}),
257 | ('{age,age}', {'age': 37}),
258 | ('{age,"years":age}', {'age': 37, 'years': 37}),
259 | ('{"years":age,age}', {'years': 37, 'age': 37}),
260 | ('{age,name.first}', {'age': 37, 'first': 'Tom'}),
261 | ('{invalid,invalid.invalid,age}', {'age': 37}),
262 | ('{name.first,age,name.last}', {'first': 'Tom', 'age': 37, 'last': 'Anderson'}),
263 | ('{{age}}', {'_': {'age': 37}}),
264 | ('{{age},age}', {'_': {'age': 37}, 'age': 37}),
265 | ('friends.0.{age,nets.#(="ig")}', {'age': 44, '_': 'ig'}),
266 | ('friends.0.{age,nets.#(="ig"),invalid}', {'age': 44, '_': 'ig'}),
267 | ('friends.0.{age,nets.#(="ig")#}', {'age': 44, '_': ['ig']}),
268 | ('friends.#.{age,"key":first}',
269 | [{'age': 44, 'key': 'Dale'}, {'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]),
270 | ('friends.#(age>44)#.{age,"key":first}', [{'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]),
271 | ('friends.#(age>44)#.{age,"key":first,invalid}', [{'age': 68, 'key': 'Roger'}, {'age': 47, 'key': 'Jane'}]),
272 | (r'{age,name.first,fav\.movie}', {'age': 37, 'first': 'Tom', r'fav\.movie': 'Deer Hunter'}),
273 | ('{age,name.{"name":first,"surname":last},children.@sort}',
274 | {'age': 37, '_': {'name': 'Tom', 'surname': 'Anderson'}, '@sort': ['Alex', 'Jack', 'Sara']}),
275 | ('friends.{0.first,1.last,2.age}.@values', ['Dale', 'Craig', 47]),
276 | ('{friends.{"a":0.{nets.{0}}}}', {'_': {'a': {'_': {'0': 'ig'}}}}),
277 | ('{friends.{"a":0.{nets.{0,1}}}}', {'_': {'a': {'_': {'0': 'ig', '1': 'fb'}}}}),
278 | ('friends.#.{age,first|@upper}',
279 | [{'age': 44, '@upper': 'DALE'}, {'age': 68, '@upper': 'ROGER'}, {'age': 47, '@upper': 'JANE'}]),
280 | ('{friends.#.{age,"first":first|@upper}|0.first}', {'first': 'DALE'}),
281 | ('{"children":children|@upper,"name":name.first,"age":age}',
282 | {'children': ['SARA', 'ALEX', 'JACK'], 'name': 'Tom', 'age': 37}),
283 | ('friends.#.{age,"first":first.invalid}', [{'age': 44}, {'age': 68}, {'age': 47}]),
284 | # Multipaths arrays
285 | ('[]', []),
286 | ('[.]', []),
287 | ('[.invalid]', []),
288 | ('[.invalid,]', []),
289 | ('[age]', [37]),
290 | (r'[a\ge]', [37]),
291 | ('[age,age]', [37, 37]),
292 | ('[age,name.first]', [37, 'Tom']),
293 | ('[name.first,age,invalid,invalid.invalid,name.last]', ['Tom', 37, 'Anderson']),
294 | ('[[age]]', [[37]]),
295 | ('[[age],age]', [[37], 37]),
296 | ('friends.0.[age,nets.#(="ig")]', [44, 'ig']),
297 | ('friends.0.[age,nets.#(="ig"),invalid]', [44, 'ig']),
298 | ('friends.0.[age,nets.#(="ig")#]', [44, ['ig']]),
299 | ('friends.#.[age,first]', [[44, 'Dale'], [68, 'Roger'], [47, 'Jane']]),
300 | ('friends.#(age>44)#.[age,first]', [[68, 'Roger'], [47, 'Jane']]),
301 | ('friends.#(age>44)#.[age,invalid,invalid.invalid,first]', [[68, 'Roger'], [47, 'Jane']]),
302 | (r'[age,name.first,fav\.movie]', [37, 'Tom', 'Deer Hunter']),
303 | ('[age,name.[first,last],children.@sort]', [37, ['Tom', 'Anderson'], ['Alex', 'Jack', 'Sara']]),
304 | ('friends.[0.first,1.last,2.age]', ['Dale', 'Craig', 47]),
305 | ('[friends.[0.[nets.[0]]]]', [[[['ig']]]]),
306 | ('[friends.[0.[nets.[0,1]]]]', [[[['ig', 'fb']]]]),
307 | # Multipaths mixed
308 | ('[{}]', [{}]),
309 | ('{[]}', {'_': []}),
310 | ('[{},[],{}]', [{}, [], {}]),
311 | ('{"a":[]}', {'a': []}),
312 | ('[{age},{name.first}]', [{'age': 37}, {'first': 'Tom'}]),
313 | ('{friends.0.[age,nets.#(="ig")]}', {'_': [44, 'ig']}),
314 | ('{friends.0.[age,nets.#(="ig")],age}', {'_': [44, 'ig'], 'age': 37}),
315 | ('{friends.0.[invalid,nets.#(="ig")],age,invalid}', {'_': ['ig'], 'age': 37}),
316 | # Literals
317 | ('!true', True),
318 | ('!false', False),
319 | ('!null', None),
320 | ('!NaN', float('nan')),
321 | ('!Infinity', float('inf')),
322 | ('!-Infinity', float('-inf')),
323 | ('!"key"', 'key'),
324 | ('!"line \\"quotes\\""', 'line "quotes"'),
325 | ('!"a\tb"', 'a\tb'),
326 | ('!0', 0),
327 | ('!12', 12),
328 | ('!-12', -12),
329 | ('!12.34', 12.34),
330 | ('!12.34E2', 1234),
331 | ('!12.34E+2', 1234),
332 | ('!12.34e-2', 0.1234),
333 | ('!-12.34e-2', -0.1234),
334 | ('friends.#.!"value"', ['value', 'value', 'value']),
335 | ('friends.#.!invalid', []),
336 | ('friends.#|!"value"', 'value'),
337 | ('friends.#(age>45)#.!"value"', ['value', 'value']),
338 | ('name|!"value"', 'value'),
339 | ('!{}', {}),
340 | ('![]', []),
341 | ('!{"name":{"first":"Tom"}}.{name.first}.first', 'Tom'),
342 | ('{name.last,"key":!"value"}', {'last': 'Anderson', 'key': 'value'}),
343 | ('{name.last,"key":!{"a":"b"},"invalid"}', {'last': 'Anderson', 'key': {'a': 'b'}}),
344 | ('{name.last,"key":!{"c":"d"},!"valid"}', {'last': 'Anderson', 'key': {'c': 'd'}, '_': 'valid'}),
345 | ('[!true,!false,!null,!Infinity,!invalid,{"name":!"andy",name.last},+Infinity,!["value1","value2"]]',
346 | [True, False, None, float('inf'), {'name': 'andy', 'last': 'Anderson'}, ['value1', 'value2']]),
347 | ('[!12.34,!-12.34e-2,!true]', [12.34, -0.1234, True]),
348 | ))
349 | def test_get_ok(self, query, expected):
350 | """It should query the JSON object and return the expected result."""
351 | compare_values(self.object.get(query), expected)
352 |
353 | @pytest.mark.parametrize(('query', 'error'), (
354 | # Basic
355 | ('.', 'Invalid query starting with a path delimiter.'),
356 | ('|', 'Invalid query starting with a path delimiter.'),
357 | ('.name', 'Invalid query starting with a path delimiter.'),
358 | ('|age', 'Invalid query starting with a path delimiter.'),
359 | ('name..first', 'Invalid query with two consecutive path delimiters.'),
360 | ('name||first', 'Invalid query with two consecutive path delimiters.'),
361 | ('name.|first', 'Invalid query with two consecutive path delimiters.'),
362 | ('name|.first', 'Invalid query with two consecutive path delimiters.'),
363 | ('age.0', "Integer query part on unsupported object type "),
364 | ('friends.99', 'Index `99` out of range for sequence object with 3 items in query.'),
365 | ('name.nonexistent', 'Mapping object does not have key `nonexistent`.'),
366 | ('name.1', 'Mapping object does not have key `1`.'),
367 | ('children.invalid', 'Invalid or unsupported query part `invalid`.'),
368 | ('children.', 'Delimiter at the end of the query.'),
369 | ('children\\', 'Escape character at the end of the query.'),
370 | # Wildcards
371 | ('x*', 'No key matching pattern with wildcard `x*`'),
372 | ('??????????', 'No key matching pattern with wildcard `??????????`'),
373 | ('children.x*', "Wildcard matching key `x*` requires a mapping object, got instead."),
374 | ('(-?', 'No key matching pattern with wildcard `(-?`'),
375 | # Queries
376 | ('#', "Expected a sequence like object for query part # at the end of the query, got ."),
377 | ('#.invalid', 'Invalid or unsupported query part `invalid`.'),
378 | ('friends.#(=="Murphy")', 'Query on mapping like objects require a key before the operator.'),
379 | ('friends.#(last=={1: 2})', 'Invalid value `{1: 2}` for the query key `last`'),
380 | ('friends.#(invalid', 'Unbalanced parentheses `(`, 1 still opened.'),
381 | ('#(first)', 'Queries are supported only for sequence like objects'),
382 | ('friends.#(invalid)', 'Query for first element does not match anything.'),
383 | ('friends.#(last=="invalid")', 'Query for first element does not match anything.'),
384 | ('friends.#(first%"D?")', 'Query for first element does not match anything.'),
385 | ('friends.#(last=="Murphy")invalid', 'Expected delimiter or end of query after closing parenthesis.'),
386 | ('children.#()', 'Empty or invalid query.'),
387 | ('children.#()#', 'Empty or invalid query.'),
388 | ('friends.#.invalid.#()', 'Empty or invalid query.'),
389 | ('friends.#.invalid.#()#', 'Empty or invalid query.'),
390 | # Dot vs Pipe
391 | ('friends.#(last="Murphy")#|first', 'Invalid or unsupported query'),
392 | # Modifiers
393 | ('@', 'Got empty modifier name.'),
394 | ('friends.@', 'Got empty modifier name.'),
395 | ('friends.@pretty:', 'Modifier with options separator `:` without any option.'),
396 | ('friends.@pretty:{invalid', 'Unable to load modifier options.'),
397 | ('friends.@pretty:["invalid"]', 'Expected JSON object `{...}` as modifier options.'),
398 | ('friends.@invalid', 'Unknown modifier @invalid.'),
399 | ('friends.@in"valid', 'Invalid modifier name @in"valid, the following characters are not allowed'),
400 | # JSON Lines
401 | ('..name', 'Invalid query starting with a path delimiter.'),
402 | # Multipaths
403 | (r'{"a\ge":age}', r'Failed to parse multipaths key "a\ge"'),
404 | ('{"age",age}', 'Expected colon after multipaths item with key "age".'),
405 | ('{]', 'Unbalanced parentheses `{`, 1 still opened.'),
406 | ('{', 'Unbalanced parentheses `{`, 1 still opened.'),
407 | ('{}@pretty', 'Expected delimiter or end of query after closing parenthesis.'),
408 | ('[{age}}]', 'Missing separator after multipath.'),
409 | ('{[age]]}', 'Missing separator after multipath.'),
410 | ('[{age,name.first]},age]', 'Expected delimiter or end of query after closing parenthesis.'),
411 | # Literals
412 | ('!', 'Unable to load literal JSON'),
413 | ('name.!', 'Unable to load literal JSON'),
414 | ('!invalid', 'Unable to load literal JSON'),
415 | (r'!in\valid', 'Unable to load literal JSON'),
416 | ('!0.a', 'Invalid or unsupported query part `a`.'),
417 | ('!0.1ea', 'Invalid or unsupported query part `ea`.'),
418 | ('!-12.', 'Delimiter at the end of the query.'),
419 | ('!-12.e', 'Invalid or unsupported query part `e`.'),
420 | ('name.!invalid', 'Unable to load literal JSON'),
421 | ('!"invalid', 'Unable to find end of literal string.'),
422 | ('friends.#|!invalid', 'Unable to load literal JSON'),
423 | ('!{true,', 'Unbalanced parentheses `{`, 1 still opened.'),
424 | ('![true,', 'Unbalanced parentheses `[`, 1 still opened.'),
425 | ('!"value".invalid', 'Invalid or unsupported query part `invalid`.'),
426 | ('name.!"value"', 'Unable to load literal JSON: literal afer a dot delimiter.'),
427 | ))
428 | def test_get_parser_raise(self, query, error):
429 | """It should raise a GJSONParseError error with the expected message."""
430 | with pytest.raises(gjson.GJSONParseError, match=fr'^{re.escape(error)}'):
431 | self.object.get(query)
432 |
433 | @pytest.mark.parametrize(('query', 'error'), (
434 | # Basic
435 | ('', 'Empty query.'),
436 | # Modifiers
437 | ('children.@keys', 'The current object does not have a keys() method.'),
438 | ('children.@values', 'The current object does not have a values() method.'),
439 | ('age.@group', "Modifier @group got object of type as input, expected dictionary."),
440 | ('children.@group', "Modifier @group got object of type as input, expected dictionary."),
441 | ))
442 | def test_get_raise(self, query, error):
443 | """It should raise a GJSONError error with the expected message."""
444 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'):
445 | self.object.get(query)
446 |
447 |
448 | class TestEscape:
449 | """Test gjson for all the escape sequences."""
450 |
451 | def setup_method(self):
452 | """Initialize the test instance."""
453 | self.escape = gjson.GJSON(INPUT_ESCAPE)
454 |
455 | @pytest.mark.parametrize(('query', 'expected'), (
456 | (r'test.\*', 'valZ'),
457 | (r'test.\*v', 'val0'),
458 | (r'test.keyv\*', 'val1'),
459 | (r'test.key\*v', 'val2'),
460 | (r'test.keyv\?', 'val3'),
461 | (r'test.key\?v', 'val4'),
462 | (r'test.keyv\.', 'val5'),
463 | (r'test.key\.v', 'val6'),
464 | (r'test.keyk\*.key\?', 'val7'),
465 | ('test.1key', 'val8'),
466 | ))
467 | def test_get_ok(self, query, expected):
468 | """It should query the escape test JSON and return the expected result."""
469 | assert self.escape.get(query, quiet=True) == expected
470 |
471 |
472 | class TestBasic:
473 | """Test gjson for basic queries."""
474 |
475 | def setup_method(self):
476 | """Initialize the test instance."""
477 | self.basic = gjson.GJSON(INPUT_BASIC)
478 |
479 | @pytest.mark.parametrize(('query', 'expected'), (
480 | ('loggy.programmers.#(age=101).firstName', 1002.3),
481 | ('loggy.programmers.#(firstName != "Brett").firstName', 'Jason'),
482 | ('loggy.programmers.#(firstName % "Bre*").email', 'aaaa'),
483 | ('loggy.programmers.#(firstName !% "Bre*").email', 'bbbb'),
484 | ('loggy.programmers.#(firstName == "Brett").email', 'aaaa'),
485 | ('loggy.programmers.#.firstName', ['Brett', 'Jason', 'Elliotte', 1002.3]),
486 | ('loggy.programmers.#.asd', []),
487 | ('items.3.tags.#', 3),
488 | ('items.3.points.1.#', 2),
489 | ('items.#', 8),
490 | ('vals.#', 3),
491 | ('name2.here', r'B\"R'),
492 | ('arr.#', 6),
493 | ('arr.3.hello', 'world'),
494 | ('name.first', 'tom'),
495 | ('name.last', None),
496 | ('age', 100),
497 | ('happy', True),
498 | ('immortal', False),
499 | ('noop', {'what is a wren?': 'a bird'}),
500 | # Modifiers
501 | ('arr.@join', {'hello': 'world'}),
502 | ))
503 | def test_get_ok(self, query, expected):
504 | """It should query the basic test JSON and return the expected result."""
505 | assert self.basic.get(query) == expected
506 |
507 |
508 | class TestList:
509 | """Test gjson queries on a list object."""
510 |
511 | def setup_method(self):
512 | """Initialize the test instance."""
513 | self.list = gjson.GJSON(INPUT_LIST)
514 |
515 | @pytest.mark.parametrize(('query', 'expected'), (
516 | # Dot vs Pipe
517 | ('#.first', ['Dale', 'Jane']),
518 | ('#.first.#', []),
519 | ('#.first|#', 2),
520 | ('#.0', []),
521 | ('#.#', []),
522 | # Queries
523 | ('#(first)#', [{'first': 'Dale'}, {'first': 'Jane'}]),
524 | ('#(first)', {'first': 'Dale'}),
525 | ('#(last)#', [{'last': 'Murphy'}]),
526 | ('#(last)', {'last': 'Murphy'}),
527 | # Modifiers
528 | ('@join', {'first': 'Jane', 'last': 'Murphy'}),
529 | # Multipaths
530 | ('#.{first.@reverse}', [{'@reverse': 'Dale'}, {'@reverse': 'Jane'}, {}]),
531 | ))
532 | def test_get_ok(self, query, expected):
533 | """It should query the list test JSON and return the expected result."""
534 | assert self.list.get(query, quiet=False) == expected
535 |
536 | @pytest.mark.parametrize(('query', 'error'), (
537 | # Dot vs Pipe
538 | ('#|first', 'Invalid or unsupported query part `first`.'),
539 | ('#|0', 'Integer query part after a pipe delimiter on an sequence like object.'),
540 | ('#|#', 'The pipe delimiter cannot immediately follow the # element.'),
541 | ))
542 | def test_get_raise(self, query, error):
543 | """It should raise a GJSONError error with the expected message."""
544 | with pytest.raises(gjson.GJSONParseError, match=fr'^{re.escape(error)}'):
545 | self.list.get(query)
546 |
547 |
548 | class TestFlattenModifier:
549 | """Test gjson @flatten modifier."""
550 |
551 | def setup_method(self):
552 | """Initialize the test instance."""
553 | self.list = gjson.GJSON(json.loads('[1, [2], [3, 4], [5, [6, 7]], [8, [9, [10, 11]]]]'))
554 |
555 | @pytest.mark.parametrize(('query', 'expected'), (
556 | ('@flatten', [1, 2, 3, 4, 5, [6, 7], 8, [9, [10, 11]]]),
557 | ('@flatten:{"deep":true}', [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]),
558 | ))
559 | def test_get(self, query, expected):
560 | """It should correctly flatten the given object."""
561 | assert self.list.get(query, quiet=True) == expected
562 |
563 |
564 | class TestTruthiness:
565 | """Testing gjson with an input object with truthy/falsy objects."""
566 |
567 | def setup_method(self):
568 | """Initialize the test instance."""
569 | self.object = gjson.GJSON(INPUT_TRUTHINESS)
570 |
571 | @pytest.mark.parametrize(('query', 'expected'), (
572 | ('vals.#(b==~true).a', 1),
573 | ('vals.#(b==~true)#.a', [1, 2, 4, 6, 7, 8]),
574 | ('vals.#(b==~false).a', 3),
575 | ('vals.#(b==~false)#.a', [3, 5, 9, 10, 11]),
576 | ('vals.#(b==~"invalid")#', []),
577 | ))
578 | def test_get_ok(self, query, expected):
579 | """It should query the JSON object and return the expected result."""
580 | compare_values(self.object.get(query), expected)
581 |
582 | @pytest.mark.parametrize(('query', 'error'), (
583 | ('vals.#(b==~"invalid")',
584 | "Queries ==~ operator requires a boolean value, got instead: `invalid`"),
585 | ))
586 | def test_get_raise(self, query, error):
587 | """It should raise a GJSONError error with the expected message."""
588 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'):
589 | self.object.get(query)
590 |
591 |
592 | class TestNestedQueries:
593 | """Testing gjson nested queries."""
594 |
595 | def setup_method(self):
596 | """Initialize the test instance."""
597 | self.object = gjson.GJSON(INPUT_NESTED_QUERIES)
598 |
599 | @pytest.mark.parametrize(('query', 'expected'), (
600 | # Arrays of objects
601 | ('key.#(level1.#(level2.#(level3)))', INPUT_NESTED_QUERIES['key'][0]),
602 | ('key.#(level1.#(level2.#(level3)))#', INPUT_NESTED_QUERIES['key'][0:2]),
603 | ('key.#(level1.#(level2.#(level3.#(==0))))#', []),
604 | ('key.#(level1.#(level2.#(level3.#(=1))))', INPUT_NESTED_QUERIES['key'][0]),
605 | ('key.#(level1.#(level2.#(level3.#(=1)#)#)#)', INPUT_NESTED_QUERIES['key'][0]),
606 | ('key.#(level1.#(level2.#(level3.#(==1))))#', [INPUT_NESTED_QUERIES['key'][0]]),
607 | ('key.#(level1.#(level2.#(level3.#(==2))))', INPUT_NESTED_QUERIES['key'][0]),
608 | ('key.#(level1.#(level2.#(level3.#(=2))))#', INPUT_NESTED_QUERIES['key'][0:2]),
609 | # Arrays of arrays
610 | ('key.#(#(#(level3)))', INPUT_NESTED_QUERIES['key'][2]),
611 | ('key.#(#(#(level3)))#', INPUT_NESTED_QUERIES['key'][2:4]),
612 | ('key.#(#(#(level3.#(==0))))#', []),
613 | ('key.#(#(#(level3.#(==1))))', INPUT_NESTED_QUERIES['key'][2]),
614 | ('key.#(#(#(level3.#(==1)#)#)#)', INPUT_NESTED_QUERIES['key'][2]),
615 | ('key.#(#(#(level3.#(==1))))#', [INPUT_NESTED_QUERIES['key'][2]]),
616 | ('key.#(#(#(level3.#(==2))))', INPUT_NESTED_QUERIES['key'][2]),
617 | ('key.#(#(#(level3.#(==2))))#', INPUT_NESTED_QUERIES['key'][2:4]),
618 | ('key.#(#(#(level3.#(>=4))))', INPUT_NESTED_QUERIES['key'][3]),
619 | ('key.#(#(#(level3.#(>=4))))#', [INPUT_NESTED_QUERIES['key'][3]]),
620 | # Mixed
621 | ('key.#(mixed.#(#(level4)))', INPUT_NESTED_QUERIES['key'][-1]),
622 | ('key.#(mixed.#(#(level4)))#', [INPUT_NESTED_QUERIES['key'][-1]]),
623 | ))
624 | def test_get_ok(self, query, expected):
625 | """It should query the JSON object and return the expected result."""
626 | compare_values(self.object.get(query), expected)
627 |
628 | @pytest.mark.parametrize(('query', 'error'), (
629 | ('key.#(level1.#(level2.#(level3.#(==0))))', 'Query for first element does not match anything.'),
630 | ('key.#(#(#(level3.#(==0))))', 'Query for first element does not match anything.'),
631 | ))
632 | def test_get_raise(self, query, error):
633 | """It should raise a GJSONError error with the expected message."""
634 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'):
635 | self.object.get(query)
636 |
637 |
638 | @pytest.mark.parametrize(('query', 'expected'), (
639 | ('0.0', 'zero'),
640 | ('0|0', 'zero'),
641 | ('#.0', ['zero']),
642 | ('#.1', ['one', 'one']),
643 | ('#.9', []),
644 | ('#(0="zero")#|0', {'0': 'zero', '1': 'one'}),
645 | ('#(0="zero")#.1', ['one']),
646 | ('#(0="zero")#.9', []),
647 | ('#(0="invalid")#.1', []),
648 | ))
649 | def test_get_integer_mapping_keys_ok(query, expected):
650 | """It should return the expected result."""
651 | obj = gjson.GJSON([{'0': 'zero', '1': 'one'}, {'1': 'one'}])
652 | assert obj.get(query, quiet=True) == expected
653 |
654 |
655 | @pytest.mark.parametrize(('query', 'error'), (
656 | ('0.1', 'Mapping object does not have key `1`.'),
657 | ('#|0', 'Integer query part after a pipe delimiter on an sequence like object.'),
658 | ('#|9', 'Integer query part after a pipe delimiter on an sequence like object.'),
659 | ('#(0="zero")#|1', 'Index `1` out of range for sequence object with 1 items in query.'),
660 | ))
661 | def test_get_integer_mapping_keys_raise(query, error):
662 | """It should return the expected result."""
663 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'):
664 |
665 | gjson.GJSON([{'0': 'zero'}]).get(query)
666 |
667 |
668 | @pytest.mark.parametrize('modifier', ('@valid', '@this'))
669 | def test_get_modifier_unmodified_ok(modifier):
670 | """It should return the same object."""
671 | obj = gjson.GJSON(INPUT_OBJECT)
672 | assert obj.get(modifier, quiet=True) == INPUT_OBJECT
673 |
674 |
675 | def test_get_modifier_valid_raise():
676 | """It should return None if the object is invalid JSON and quiet is True."""
677 | obj = gjson.GJSON({'invalid': {1, 2}})
678 | assert obj.get('@valid', quiet=True) is None
679 |
680 |
681 | @pytest.mark.parametrize(('data', 'expected'), (
682 | ('[3, 1, 5, 8, 2]', [1, 2, 3, 5, 8]),
683 | ('{"b": 2, "d": 4, "c": 3, "a": 1}', {'a': 1, 'b': 2, 'c': 3, 'd': 4}),
684 | ('"a string"', None),
685 | ))
686 | def test_get_modifier_sort(data, expected):
687 | """It should return the object sorted."""
688 | obj = gjson.GJSON(json.loads(data))
689 | compare_values(obj.get('@sort', quiet=True), expected)
690 |
691 |
692 | @pytest.mark.parametrize(('data', 'query', 'expected'), (
693 | ({'a': '{"b": 25}'}, 'a.@fromstr', {'b': 25}),
694 | ({'a': '{"b": 25}'}, 'a.@fromstr.b', 25),
695 | ))
696 | def test_get_modifier_fromstr_ok(data, query, expected):
697 | """It should load the JSON-encoded string."""
698 | obj = gjson.GJSON(data)
699 | assert obj.get(query, quiet=True) == expected
700 |
701 |
702 | @pytest.mark.parametrize(('query', 'error'), (
703 | ('a.@fromstr', 'The current @fromstr input object cannot be converted to JSON.'),
704 | ('b.@fromstr', "Modifier @fromstr got object of type as input, expected string or bytes."),
705 | ))
706 | def test_get_modifier_fromstr_raise(query, error):
707 | """It should raise a GJSONError if the JSON-encoded string has invalid JSON."""
708 | obj = gjson.GJSON({'a': '{"invalid: json"', 'b': {'not': 'a string'}})
709 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape(error)}'):
710 | obj.get(query)
711 |
712 |
713 | def test_get_modifier_tostr_raise():
714 | """It should raise a GJSONError if the object cannot be JSON-encoded."""
715 | obj = gjson.GJSON({'a': {1, 2, 3}}) # Python sets cannot be JSON-encoded
716 | match = re.escape('The current object cannot be converted to a JSON-encoded string for @tostr.')
717 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'):
718 | obj.get('a.@tostr')
719 |
720 |
721 | def test_get_modifier_group_ok():
722 | """It should group the dict of lists into a list of dicts."""
723 | obj = gjson.GJSON({
724 | 'invalid1': 5,
725 | 'id': ['123', '456', '789'],
726 | 'val': [2, 1],
727 | 'invalid2': 'invalid',
728 | 'unit': ['ms', 's', 's', 'ms'],
729 | })
730 | assert obj.get('@group') == [
731 | {'id': '123', 'val': 2, 'unit': 'ms'},
732 | {'id': '456', 'val': 1, 'unit': 's'},
733 | {'id': '789', 'unit': 's'},
734 | {'unit': 'ms'},
735 | ]
736 |
737 |
738 | def test_get_modifier_group_empty():
739 | """It should return an empty list if no values are lists or are empty."""
740 | obj = gjson.GJSON({'invalid1': 5, 'invalid2': 'invalid', 'invalid3': {'a': 5}, 'id': []})
741 | assert obj.get('@group') == []
742 |
743 |
744 | def test_get_integer_index_on_mapping():
745 | """It should access the integer as string key correctly."""
746 | obj = gjson.GJSON(json.loads('{"1": 5, "11": 7}'))
747 | assert obj.get('1') == 5
748 | assert obj.get('11') == 7
749 |
750 |
751 | def test_module_get():
752 | """It should return the queried object."""
753 | assert gjson.get({'key': 'value'}, 'key') == 'value'
754 |
755 |
756 | def test_gjson_get_gjson_ok():
757 | """It should return the queried object as a GJSON object."""
758 | ret = gjson.GJSON(INPUT_OBJECT).get_gjson('children')
759 | assert isinstance(ret, gjson.GJSON)
760 | assert str(ret) == '["Sara", "Alex", "Jack"]'
761 |
762 |
763 | @pytest.mark.parametrize('kwargs', ({}, {'quiet': False}))
764 | def test_gjson_get_gjson_raise(kwargs):
765 | """It should raise a GJSONError if the quiet parameter is not passed or is False."""
766 | with pytest.raises(gjson.GJSONError, match=r'^Mapping object does not have key `nonexistent`.'):
767 | gjson.GJSON(INPUT_OBJECT).get_gjson('nonexistent', **kwargs)
768 |
769 |
770 | @pytest.mark.parametrize(('data', 'num', 'expected'), (
771 | # Valid data
772 | ('[1, 2, 3, 4, 5]', None, {1: 1, 2: 1, 3: 1, 4: 1, 5: 1}),
773 | ('[1, 2, 3, 4, 5]', 0, {}),
774 | ('[1, 2, 3, 4, 5]', 2, {1: 1, 2: 1}),
775 | ('[1, 1, 1, 1, 1]', None, {1: 5}),
776 | ('[1, 1, 1, 1, 1]', 1, {1: 5}),
777 | ('[1, 1, 1, 2, 2, 3]', None, {1: 3, 2: 2, 3: 1}),
778 | ('[1, 1, 1, 2, 2, 3, 3, 3, 3]', None, {3: 4, 1: 3, 2: 2}),
779 | ('[1, 1, 1, 2, 2, 3, 3, 3, 3]', 2, {3: 4, 1: 3}),
780 | # Invalid data
781 | ('{"key": "value"}', None, None),
782 | ('1', None, None),
783 | ('"value"', None, None),
784 | ))
785 | def test_get_modifier_top_n(data, num, expected):
786 | """It should return the top N common items."""
787 | obj = gjson.GJSON(json.loads(data))
788 | if num is not None:
789 | compare_values(obj.get(f'@top_n:{{"n": {num}}}', quiet=True), expected)
790 | else:
791 | compare_values(obj.get('@top_n', quiet=True), expected)
792 |
793 |
794 | @pytest.mark.parametrize(('num', 'expected'), (
795 | (0, {}),
796 | (1, {'c': 12}),
797 | (2, {'c': 12, 'a': 8}),
798 | (3, {'c': 12, 'a': 8, 'd': 4}),
799 | (4, {'c': 12, 'a': 8, 'd': 4, 'b': 3.5}),
800 | (None, {'c': 12, 'a': 8, 'd': 4, 'b': 3.5}),
801 | ))
802 | def test_get_modifier_sum_n_valid(num, expected):
803 | """It should group and sum and return the top N items."""
804 | obj = gjson.GJSON(INPUT_SUM_N)
805 | if num is not None:
806 | compare_values(obj.get(f'@sum_n:{{"group": "key", "sum": "value", "n": {num}}}', quiet=True), expected)
807 | else:
808 | compare_values(obj.get('@sum_n:{"group": "key", "sum": "value"}', quiet=True), expected)
809 |
810 |
811 | @pytest.mark.parametrize('data', (
812 | '{"an": "object"}',
813 | '"a string"',
814 | '1',
815 | ))
816 | def test_get_modifier_sum_n_invalid_data(data):
817 | """It should raise a GJSONError if the input is invalid."""
818 | obj = gjson.GJSON(json.loads(data))
819 | with pytest.raises(gjson.GJSONError, match=r'^@sum_n modifier not supported for object of type'):
820 | obj.get('@sum_n:{"group": "key", "sum": "value"}')
821 |
822 |
823 | @pytest.mark.parametrize('options', (
824 | '',
825 | ':{}',
826 | ':{"group": "invalid", "sum": "value"}',
827 | ':{"group": "key", "sum": "invalid"}',
828 | ':{"group": "key", "sum": "other"}',
829 | ':{"group": "other", "sum": "value"}',
830 | ))
831 | def test_get_modifier_sum_n_invalid_options(options):
832 | """It should raise a GJSONError if the options are invalid."""
833 | obj = gjson.GJSON(INPUT_SUM_N)
834 | with pytest.raises(gjson.GJSONError, match=r'^Modifier @sum_n raised an exception'):
835 | obj.get(f'@sum_n{options}')
836 |
837 |
838 | class TestJSONOutput:
839 | """Test class for all JSON output functionalities."""
840 |
841 | def setup_method(self):
842 | """Initialize the test instance."""
843 | self.obj = {'key': 'value', 'hello world': '\u3053\u3093\u306b\u3061\u306f\u4e16\u754c'}
844 | self.query = 'key'
845 | self.value = '"value"'
846 | self.gjson = gjson.GJSON(self.obj)
847 |
848 | def test_module_get_as_str(self):
849 | """It should return the queried object as a JSON string."""
850 | assert gjson.get(self.obj, self.query, as_str=True) == self.value
851 | assert gjson.get(self.obj, '', as_str=True, quiet=True) == ''
852 |
853 | def test_module_get_str(self):
854 | """It should return the string representation of the object."""
855 | assert str(self.gjson) == '{"key": "value", "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"}'
856 |
857 | def test_gjson_getj(self):
858 | """It should return the queried object as a JSON string."""
859 | assert self.gjson.getj(self.query) == self.value
860 | assert self.gjson.getj('', quiet=True) == ''
861 |
862 | def test_module_get_as_str_raise(self):
863 | """It should raise a GJSONError with the proper message on failure."""
864 | with pytest.raises(gjson.GJSONError, match=r'^Empty query.'):
865 | gjson.get(self.obj, '', as_str=True)
866 |
867 | def test_gjson_get_as_str_raise(self):
868 | """It should raise a GJSONError with the proper message on failure."""
869 | with pytest.raises(gjson.GJSONError, match=r'^Empty query.'):
870 | self.gjson.getj('')
871 |
872 | @pytest.mark.parametrize(('query', 'expected'), (
873 | ('@pretty', '{\n "key": "value",\n "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'),
874 | ('@pretty:{"indent": 4}',
875 | '{\n "key": "value",\n "hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'),
876 | ('@pretty:{"indent": "\t"}',
877 | '{\n\t"key": "value",\n\t"hello world": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"\n}'),
878 | # Multipaths
879 | ('{key,"another":key}.@pretty', '{\n "key": "value",\n "another": "value"\n}'),
880 | ))
881 | def test_modifier_pretty(self, query, expected):
882 | """It should prettyfy the JSON string based on the parameters."""
883 | assert self.gjson.getj(query) == expected
884 |
885 | def test_modifier_pretty_sort_keys_prefix(self):
886 | """It should prettyfy the JSON string and sort the keys."""
887 | output = gjson.GJSON({'key2': 'value2', 'key1': 'value1'}).getj('@pretty:{"sortKeys": true, "prefix": "## "}')
888 | assert output == '## {\n## "key1": "value1",\n## "key2": "value2"\n## }'
889 |
890 | def test_modifier_ugly(self):
891 | """It should uglyfy the JSON string."""
892 | assert gjson.get(self.obj, '@ugly', as_str=True) == (
893 | '{"key":"value","hello world":"\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"}')
894 |
895 | def test_output_unicode(self):
896 | """It should return unicode characters as-is."""
897 | assert gjson.get(self.obj, 'hello world', as_str=True) == '"\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"'
898 |
899 | def test_modifier_ascii(self):
900 | """It should escape all non-ASCII characters."""
901 | assert gjson.get(self.obj, 'hello world.@ascii', as_str=True) == (
902 | '"\\u3053\\u3093\\u306b\\u3061\\u306f\\u4e16\\u754c"')
903 |
904 |
905 | def custom_sum(options, obj, *, last):
906 | """Sum items in list. Custom modifier function to be used in tests."""
907 | assert last is True
908 | assert options == {}
909 | if not isinstance(obj, list):
910 | raise TypeError('@sum can be used only on lists')
911 |
912 | return sum(obj)
913 |
914 |
915 | class TestCustomModifiers:
916 | """Test class for custom modifiers."""
917 |
918 | def setup_method(self):
919 | """Initialize the test instance."""
920 | self.valid_obj = [1, 2, 3, 4, 5]
921 | self.invalid_obj = 'invalid'
922 | self.query = '@sum'
923 |
924 | def test_gjson_register_modifier_ok(self):
925 | """It should register a valid modifier."""
926 | obj = gjson.GJSON(self.valid_obj)
927 | obj.register_modifier('sum', custom_sum)
928 | assert obj.get(self.query) == 15
929 |
930 | def test_gjson_register_modifier_with_escape_ok(self):
931 | """It should register a valid modifier with escaped characters in the name."""
932 | obj = gjson.GJSON(self.valid_obj)
933 | obj.register_modifier('sum\\=', custom_sum)
934 | assert obj.get('@sum\\=') == 15
935 |
936 | @pytest.mark.parametrize('char', MODIFIER_NAME_RESERVED_CHARS)
937 | def test_gjson_register_modifier_invalid_name(self, char):
938 | """It should raise a GJSONError if trying to register a modifier with a name with not allowed characters."""
939 | obj = gjson.GJSON(self.valid_obj)
940 | name = fr'a{char}b'
941 | with pytest.raises(
942 | gjson.GJSONError,
943 | match=fr'^Unable to register modifier `{re.escape(name)}`, contains at least one not allowed'):
944 | obj.register_modifier(name, custom_sum)
945 |
946 | def test_gjson_register_modifier_override_builtin(self):
947 | """It should raise a GJSONError if trying to register a modifier with the same name of a built-in one."""
948 | obj = gjson.GJSON(self.valid_obj)
949 | match = re.escape('Unable to register a modifier with the same name of the built-in modifier: @valid')
950 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'):
951 | obj.register_modifier('valid', custom_sum)
952 |
953 | def test_gjson_register_modifier_not_callable(self):
954 | """It should raise a GJSONError if trying to register a modifier that is not callable."""
955 | obj = gjson.GJSON(self.valid_obj)
956 | match = re.escape('The given func "not-callable" for the custom modifier @sum does not adhere to the '
957 | 'gjson.ModifierProtocol.')
958 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'):
959 | obj.register_modifier('sum', 'not-callable')
960 |
961 | def test_gjsonobj_custom_modifiers_ok(self):
962 | """It should register a valid modifier."""
963 | obj = gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'sum': custom_sum})
964 | assert obj.get() == 15
965 |
966 | def test_gjsonobj_custom_modifiers_raise(self):
967 | """It should encapsulate the modifier raised exception in a GJSONError."""
968 | with pytest.raises(gjson.GJSONError, match=fr'^{re.escape("Modifier @sum raised an exception")}'):
969 | gjson.GJSONObj(self.invalid_obj, self.query, custom_modifiers={'sum': custom_sum}).get()
970 |
971 | def test_gjsonobj_custom_modifiers_override_builtin(self):
972 | """It should raise a GJSONError if passing custom modifiers that have the same name of a built-in one."""
973 | with pytest.raises(gjson.GJSONError,
974 | match=r"^Some provided custom_modifiers have the same name of built-in ones: {'valid'}"):
975 | gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'valid': custom_sum})
976 |
977 | def test_gjsoniobj_custom_modifiers_not_callable(self):
978 | """It should raise a GJSONError if passing custom modifiers that are not callable."""
979 | match = re.escape('The given func "not-callable" for the custom modifier @sum does not adhere to the')
980 | with pytest.raises(gjson.GJSONError, match=fr'^{match}'):
981 | gjson.GJSONObj(self.valid_obj, self.query, custom_modifiers={'sum': 'not-callable'})
982 |
983 | def test_gjsonobj_builtin_modifiers(self):
984 | """It should return a set with the names of the built-in modifiers."""
985 | expected = {'ascii', 'flatten', 'fromstr', 'group', 'join', 'keys', 'pretty', 'reverse', 'sort', 'sum_n',
986 | 'this', 'top_n', 'tostr', 'valid', 'values', 'ugly'}
987 | assert gjson.GJSONObj.builtin_modifiers() == expected
988 |
--------------------------------------------------------------------------------
/tox.ini:
--------------------------------------------------------------------------------
1 | [tox]
2 | minversion = 3.10.0
3 | envlist = py{39,310,311,312}-{ruff,unit,mypy,prospector,sphinx}
4 | skip_missing_interpreters = True
5 |
6 |
7 | [testenv]
8 | usedevelop = True
9 | download = True
10 | allowlist_externals = sed
11 | description =
12 | unit: Run unit tests
13 | bandit: Security-oriented static analyzer
14 | mypy: Static analyzer for type annotations
15 | prospector: Static analysis multi-tool
16 | ruff: Python linter with a lot of other functionalities
17 | sphinx: Build documentation and manpages
18 | py39: (Python 3.9)
19 | py310: (Python 3.10)
20 | py311: (Python 3.11)
21 | py312: (Python 3.12)
22 | commands =
23 | unit: py.test --strict-markers --cov-report=term-missing --cov=gjson tests/unit {posargs}
24 | mypy: mypy --show-error-codes gjson/
25 | prospector: prospector --profile '{toxinidir}/prospector.yaml' --tool pyroma --tool vulture {posargs} {toxinidir}
26 | ruff: ruff check {posargs} {toxinidir}
27 | sphinx: sphinx-build -W -b html '{toxinidir}/doc/source/' '{toxinidir}/doc/build/html'
28 | sphinx: sphinx-build -W -b man '{toxinidir}/doc/source/' '{toxinidir}/doc/build/man'
29 | # Fix missing space after bold blocks in man page: https://github.com/ribozz/sphinx-argparse/issues/80
30 | sphinx: sed -i='' -e 's/^\.B/.B /' '{toxinidir}/doc/build/man/gjson.1'
31 | deps =
32 | # Use install_requires and the additional extras_require[tests/prospector] from setup.py
33 | prospector: .[prospector]
34 | !prospector: .[tests]
35 |
--------------------------------------------------------------------------------