├── .github └── workflows │ └── ci.yml ├── .gitignore ├── LICENSE.apache-2.txt ├── LICENSE.mpl-2.txt ├── README.md ├── requirements-dev.in ├── requirements-dev.txt ├── requirements.in ├── requirements.txt ├── tests ├── test_compression.py ├── test_datetime.py ├── test_encode_decode.py ├── test_json.py ├── test_marshal.py ├── test_misc.py ├── test_re_by_construction.py ├── test_source_code.py ├── test_unicode.py └── test_zoneinfo.py └── tox.ini /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: stdlib-property-tests CI 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | check: 11 | runs-on: ubuntu-latest 12 | steps: 13 | - uses: actions/checkout@v2 14 | - name: Set up Python 3.8 15 | uses: actions/setup-python@v2 16 | with: 17 | python-version: 3.8 18 | - name: Install dependencies 19 | run: python -m pip install --upgrade pip setuptools tox 20 | - name: Run checks 21 | run: | 22 | python -m tox --recreate -e check 23 | git diff --exit-code 24 | test: 25 | runs-on: ubuntu-latest 26 | strategy: 27 | matrix: 28 | python-version: ["3.7", "3.7-dev", "3.8", "3.8-dev", "3.9-dev"] 29 | fail-fast: false 30 | steps: 31 | - uses: actions/checkout@v2 32 | - name: Set up Python ${{ matrix.python-version }} 33 | uses: actions/setup-python@v2 34 | with: 35 | python-version: ${{ matrix.python-version }} 36 | - name: Install dependencies 37 | run: python -m pip install --upgrade pip setuptools tox 38 | - name: Run tests 39 | run: python -m tox --recreate -e test 40 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /LICENSE.apache-2.txt: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | https://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | Copyright 2013-2018 Docker, Inc. 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | https://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. -------------------------------------------------------------------------------- /LICENSE.mpl-2.txt: -------------------------------------------------------------------------------- 1 | Mozilla Public License Version 2.0 2 | ================================== 3 | 4 | 1. Definitions 5 | -------------- 6 | 7 | 1.1. "Contributor" 8 | means each individual or legal entity that creates, contributes to 9 | the creation of, or owns Covered Software. 10 | 11 | 1.2. "Contributor Version" 12 | means the combination of the Contributions of others (if any) used 13 | by a Contributor and that particular Contributor's Contribution. 14 | 15 | 1.3. "Contribution" 16 | means Covered Software of a particular Contributor. 17 | 18 | 1.4. "Covered Software" 19 | means Source Code Form to which the initial Contributor has attached 20 | the notice in Exhibit A, the Executable Form of such Source Code 21 | Form, and Modifications of such Source Code Form, in each case 22 | including portions thereof. 23 | 24 | 1.5. "Incompatible With Secondary Licenses" 25 | means 26 | 27 | (a) that the initial Contributor has attached the notice described 28 | in Exhibit B to the Covered Software; or 29 | 30 | (b) that the Covered Software was made available under the terms of 31 | version 1.1 or earlier of the License, but not also under the 32 | terms of a Secondary License. 33 | 34 | 1.6. "Executable Form" 35 | means any form of the work other than Source Code Form. 36 | 37 | 1.7. "Larger Work" 38 | means a work that combines Covered Software with other material, in 39 | a separate file or files, that is not Covered Software. 40 | 41 | 1.8. "License" 42 | means this document. 43 | 44 | 1.9. "Licensable" 45 | means having the right to grant, to the maximum extent possible, 46 | whether at the time of the initial grant or subsequently, any and 47 | all of the rights conveyed by this License. 48 | 49 | 1.10. "Modifications" 50 | means any of the following: 51 | 52 | (a) any file in Source Code Form that results from an addition to, 53 | deletion from, or modification of the contents of Covered 54 | Software; or 55 | 56 | (b) any new file in Source Code Form that contains any Covered 57 | Software. 58 | 59 | 1.11. "Patent Claims" of a Contributor 60 | means any patent claim(s), including without limitation, method, 61 | process, and apparatus claims, in any patent Licensable by such 62 | Contributor that would be infringed, but for the grant of the 63 | License, by the making, using, selling, offering for sale, having 64 | made, import, or transfer of either its Contributions or its 65 | Contributor Version. 66 | 67 | 1.12. "Secondary License" 68 | means either the GNU General Public License, Version 2.0, the GNU 69 | Lesser General Public License, Version 2.1, the GNU Affero General 70 | Public License, Version 3.0, or any later versions of those 71 | licenses. 72 | 73 | 1.13. "Source Code Form" 74 | means the form of the work preferred for making modifications. 75 | 76 | 1.14. "You" (or "Your") 77 | means an individual or a legal entity exercising rights under this 78 | License. For legal entities, "You" includes any entity that 79 | controls, is controlled by, or is under common control with You. For 80 | purposes of this definition, "control" means (a) the power, direct 81 | or indirect, to cause the direction or management of such entity, 82 | whether by contract or otherwise, or (b) ownership of more than 83 | fifty percent (50%) of the outstanding shares or beneficial 84 | ownership of such entity. 85 | 86 | 2. License Grants and Conditions 87 | -------------------------------- 88 | 89 | 2.1. Grants 90 | 91 | Each Contributor hereby grants You a world-wide, royalty-free, 92 | non-exclusive license: 93 | 94 | (a) under intellectual property rights (other than patent or trademark) 95 | Licensable by such Contributor to use, reproduce, make available, 96 | modify, display, perform, distribute, and otherwise exploit its 97 | Contributions, either on an unmodified basis, with Modifications, or 98 | as part of a Larger Work; and 99 | 100 | (b) under Patent Claims of such Contributor to make, use, sell, offer 101 | for sale, have made, import, and otherwise transfer either its 102 | Contributions or its Contributor Version. 103 | 104 | 2.2. Effective Date 105 | 106 | The licenses granted in Section 2.1 with respect to any Contribution 107 | become effective for each Contribution on the date the Contributor first 108 | distributes such Contribution. 109 | 110 | 2.3. Limitations on Grant Scope 111 | 112 | The licenses granted in this Section 2 are the only rights granted under 113 | this License. No additional rights or licenses will be implied from the 114 | distribution or licensing of Covered Software under this License. 115 | Notwithstanding Section 2.1(b) above, no patent license is granted by a 116 | Contributor: 117 | 118 | (a) for any code that a Contributor has removed from Covered Software; 119 | or 120 | 121 | (b) for infringements caused by: (i) Your and any other third party's 122 | modifications of Covered Software, or (ii) the combination of its 123 | Contributions with other software (except as part of its Contributor 124 | Version); or 125 | 126 | (c) under Patent Claims infringed by Covered Software in the absence of 127 | its Contributions. 128 | 129 | This License does not grant any rights in the trademarks, service marks, 130 | or logos of any Contributor (except as may be necessary to comply with 131 | the notice requirements in Section 3.4). 132 | 133 | 2.4. Subsequent Licenses 134 | 135 | No Contributor makes additional grants as a result of Your choice to 136 | distribute the Covered Software under a subsequent version of this 137 | License (see Section 10.2) or under the terms of a Secondary License (if 138 | permitted under the terms of Section 3.3). 139 | 140 | 2.5. Representation 141 | 142 | Each Contributor represents that the Contributor believes its 143 | Contributions are its original creation(s) or it has sufficient rights 144 | to grant the rights to its Contributions conveyed by this License. 145 | 146 | 2.6. Fair Use 147 | 148 | This License is not intended to limit any rights You have under 149 | applicable copyright doctrines of fair use, fair dealing, or other 150 | equivalents. 151 | 152 | 2.7. Conditions 153 | 154 | Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted 155 | in Section 2.1. 156 | 157 | 3. Responsibilities 158 | ------------------- 159 | 160 | 3.1. Distribution of Source Form 161 | 162 | All distribution of Covered Software in Source Code Form, including any 163 | Modifications that You create or to which You contribute, must be under 164 | the terms of this License. You must inform recipients that the Source 165 | Code Form of the Covered Software is governed by the terms of this 166 | License, and how they can obtain a copy of this License. You may not 167 | attempt to alter or restrict the recipients' rights in the Source Code 168 | Form. 169 | 170 | 3.2. Distribution of Executable Form 171 | 172 | If You distribute Covered Software in Executable Form then: 173 | 174 | (a) such Covered Software must also be made available in Source Code 175 | Form, as described in Section 3.1, and You must inform recipients of 176 | the Executable Form how they can obtain a copy of such Source Code 177 | Form by reasonable means in a timely manner, at a charge no more 178 | than the cost of distribution to the recipient; and 179 | 180 | (b) You may distribute such Executable Form under the terms of this 181 | License, or sublicense it under different terms, provided that the 182 | license for the Executable Form does not attempt to limit or alter 183 | the recipients' rights in the Source Code Form under this License. 184 | 185 | 3.3. Distribution of a Larger Work 186 | 187 | You may create and distribute a Larger Work under terms of Your choice, 188 | provided that You also comply with the requirements of this License for 189 | the Covered Software. If the Larger Work is a combination of Covered 190 | Software with a work governed by one or more Secondary Licenses, and the 191 | Covered Software is not Incompatible With Secondary Licenses, this 192 | License permits You to additionally distribute such Covered Software 193 | under the terms of such Secondary License(s), so that the recipient of 194 | the Larger Work may, at their option, further distribute the Covered 195 | Software under the terms of either this License or such Secondary 196 | License(s). 197 | 198 | 3.4. Notices 199 | 200 | You may not remove or alter the substance of any license notices 201 | (including copyright notices, patent notices, disclaimers of warranty, 202 | or limitations of liability) contained within the Source Code Form of 203 | the Covered Software, except that You may alter any license notices to 204 | the extent required to remedy known factual inaccuracies. 205 | 206 | 3.5. Application of Additional Terms 207 | 208 | You may choose to offer, and to charge a fee for, warranty, support, 209 | indemnity or liability obligations to one or more recipients of Covered 210 | Software. However, You may do so only on Your own behalf, and not on 211 | behalf of any Contributor. You must make it absolutely clear that any 212 | such warranty, support, indemnity, or liability obligation is offered by 213 | You alone, and You hereby agree to indemnify every Contributor for any 214 | liability incurred by such Contributor as a result of warranty, support, 215 | indemnity or liability terms You offer. You may include additional 216 | disclaimers of warranty and limitations of liability specific to any 217 | jurisdiction. 218 | 219 | 4. Inability to Comply Due to Statute or Regulation 220 | --------------------------------------------------- 221 | 222 | If it is impossible for You to comply with any of the terms of this 223 | License with respect to some or all of the Covered Software due to 224 | statute, judicial order, or regulation then You must: (a) comply with 225 | the terms of this License to the maximum extent possible; and (b) 226 | describe the limitations and the code they affect. Such description must 227 | be placed in a text file included with all distributions of the Covered 228 | Software under this License. Except to the extent prohibited by statute 229 | or regulation, such description must be sufficiently detailed for a 230 | recipient of ordinary skill to be able to understand it. 231 | 232 | 5. Termination 233 | -------------- 234 | 235 | 5.1. The rights granted under this License will terminate automatically 236 | if You fail to comply with any of its terms. However, if You become 237 | compliant, then the rights granted under this License from a particular 238 | Contributor are reinstated (a) provisionally, unless and until such 239 | Contributor explicitly and finally terminates Your grants, and (b) on an 240 | ongoing basis, if such Contributor fails to notify You of the 241 | non-compliance by some reasonable means prior to 60 days after You have 242 | come back into compliance. Moreover, Your grants from a particular 243 | Contributor are reinstated on an ongoing basis if such Contributor 244 | notifies You of the non-compliance by some reasonable means, this is the 245 | first time You have received notice of non-compliance with this License 246 | from such Contributor, and You become compliant prior to 30 days after 247 | Your receipt of the notice. 248 | 249 | 5.2. If You initiate litigation against any entity by asserting a patent 250 | infringement claim (excluding declaratory judgment actions, 251 | counter-claims, and cross-claims) alleging that a Contributor Version 252 | directly or indirectly infringes any patent, then the rights granted to 253 | You by any and all Contributors for the Covered Software under Section 254 | 2.1 of this License shall terminate. 255 | 256 | 5.3. In the event of termination under Sections 5.1 or 5.2 above, all 257 | end user license agreements (excluding distributors and resellers) which 258 | have been validly granted by You or Your distributors under this License 259 | prior to termination shall survive termination. 260 | 261 | ************************************************************************ 262 | * * 263 | * 6. Disclaimer of Warranty * 264 | * ------------------------- * 265 | * * 266 | * Covered Software is provided under this License on an "as is" * 267 | * basis, without warranty of any kind, either expressed, implied, or * 268 | * statutory, including, without limitation, warranties that the * 269 | * Covered Software is free of defects, merchantable, fit for a * 270 | * particular purpose or non-infringing. The entire risk as to the * 271 | * quality and performance of the Covered Software is with You. * 272 | * Should any Covered Software prove defective in any respect, You * 273 | * (not any Contributor) assume the cost of any necessary servicing, * 274 | * repair, or correction. This disclaimer of warranty constitutes an * 275 | * essential part of this License. No use of any Covered Software is * 276 | * authorized under this License except under this disclaimer. * 277 | * * 278 | ************************************************************************ 279 | 280 | ************************************************************************ 281 | * * 282 | * 7. Limitation of Liability * 283 | * -------------------------- * 284 | * * 285 | * Under no circumstances and under no legal theory, whether tort * 286 | * (including negligence), contract, or otherwise, shall any * 287 | * Contributor, or anyone who distributes Covered Software as * 288 | * permitted above, be liable to You for any direct, indirect, * 289 | * special, incidental, or consequential damages of any character * 290 | * including, without limitation, damages for lost profits, loss of * 291 | * goodwill, work stoppage, computer failure or malfunction, or any * 292 | * and all other commercial damages or losses, even if such party * 293 | * shall have been informed of the possibility of such damages. This * 294 | * limitation of liability shall not apply to liability for death or * 295 | * personal injury resulting from such party's negligence to the * 296 | * extent applicable law prohibits such limitation. Some * 297 | * jurisdictions do not allow the exclusion or limitation of * 298 | * incidental or consequential damages, so this exclusion and * 299 | * limitation may not apply to You. * 300 | * * 301 | ************************************************************************ 302 | 303 | 8. Litigation 304 | ------------- 305 | 306 | Any litigation relating to this License may be brought only in the 307 | courts of a jurisdiction where the defendant maintains its principal 308 | place of business and such litigation shall be governed by laws of that 309 | jurisdiction, without reference to its conflict-of-law provisions. 310 | Nothing in this Section shall prevent a party's ability to bring 311 | cross-claims or counter-claims. 312 | 313 | 9. Miscellaneous 314 | ---------------- 315 | 316 | This License represents the complete agreement concerning the subject 317 | matter hereof. If any provision of this License is held to be 318 | unenforceable, such provision shall be reformed only to the extent 319 | necessary to make it enforceable. Any law or regulation which provides 320 | that the language of a contract shall be construed against the drafter 321 | shall not be used to construe this License against a Contributor. 322 | 323 | 10. Versions of the License 324 | --------------------------- 325 | 326 | 10.1. New Versions 327 | 328 | Mozilla Foundation is the license steward. Except as provided in Section 329 | 10.3, no one other than the license steward has the right to modify or 330 | publish new versions of this License. Each version will be given a 331 | distinguishing version number. 332 | 333 | 10.2. Effect of New Versions 334 | 335 | You may distribute the Covered Software under the terms of the version 336 | of the License under which You originally received the Covered Software, 337 | or under the terms of any subsequent version published by the license 338 | steward. 339 | 340 | 10.3. Modified Versions 341 | 342 | If you create software not governed by this License, and you want to 343 | create a new license for such software, you may create and use a 344 | modified version of this License if you rename the license and remove 345 | any references to the name of the license steward (except to note that 346 | such modified license differs from this License). 347 | 348 | 10.4. Distributing Source Code Form that is Incompatible With Secondary 349 | Licenses 350 | 351 | If You choose to distribute Source Code Form that is Incompatible With 352 | Secondary Licenses under the terms of this version of the License, the 353 | notice described in Exhibit B of this License must be attached. 354 | 355 | Exhibit A - Source Code Form License Notice 356 | ------------------------------------------- 357 | 358 | This Source Code Form is subject to the terms of the Mozilla Public 359 | License, v. 2.0. If a copy of the MPL was not distributed with this 360 | file, You can obtain one at http://mozilla.org/MPL/2.0/. 361 | 362 | If it is not possible or desirable to put the notice in a particular 363 | file, then You may include the notice in a location (such as a LICENSE 364 | file in a relevant directory) where a recipient would be likely to look 365 | for such a notice. 366 | 367 | You may add additional accurate notices of copyright ownership. 368 | 369 | Exhibit B - "Incompatible With Secondary Licenses" Notice 370 | --------------------------------------------------------- 371 | 372 | This Source Code Form is "Incompatible With Secondary Licenses", as 373 | defined by the Mozilla Public License, v. 2.0. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # stdlib-property-tests 2 | Property-based tests for the Python standard library (and builtins) 3 | 4 | 5 | ## Goal 6 | 7 | **Find and fix bugs in Python, *before* they ship to users.** 8 | 9 | CPython's existing test suite is good, but bugs still slip through occasionally. 10 | We think that using property-based testing tools - i.e. 11 | [Hypothesis](https://hypothesis.readthedocs.io/) - can help with this. 12 | They're no magic bullet, but computer-assisted testing techniques routinely 13 | try inputs that humans wouldn't think of (or bother trying), and 14 | [turn up bugs that humans missed](https://twitter.com/pganssle/status/1193371087968591872). 15 | 16 | Specifically, we [propose adding these tests to CPython's CI suite](https://github.com/python/steering-council/issues/65), 17 | and [gave a talk](https://us.pycon.org/2020/events/languagesummit/) at the 18 | [2020 Language Summit](https://pyfound.blogspot.com/2020/05/property-based-testing-for-python.html) 19 | to that effect. Doing so would mean: 20 | 21 | - New code can be developed with property-based tests - some already is, but 22 | [adding it to CPython CI would catch bugs](https://twitter.com/pganssle/status/1193371087968591872). 23 | - Tests for existing code can be _augumented_ with property-based tests where 24 | this seems particularly valuable - e.g. `tests/test_source_code.py` in this repo 25 | has discovered several bugs. 26 | - Property-based testing can be adopted incrementally. _Replacing_ existing 27 | tests is an explicit non-goal of this project. 28 | 29 | [PyPy already uses Hypothesis](https://github.com/Zac-HD/stdlib-property-tests/issues/7), 30 | and sharing as much of the test suite as possible between implementations would be great. 31 | How this would work depends largely on CPython's decisions, though. 32 | 33 | 34 | ## LICENSE 35 | By contributing to this repository, you agree to license the contributed 36 | code under user's choice of the Mozilla Public License Version 2.0, and 37 | the Apache License 2.0. 38 | 39 | This dual-licence is intended to make it as easy as possible for the tests 40 | in this repository to be used upstream by the CPython project, other 41 | implementations of Python, and the Hypothesis project and ecosystem. 42 | 43 | 44 | # Workflow 45 | To run the tests against the current version of Python: 46 | 47 | - `pip install -r requirements.txt` (or `hypothesis hypothesmith`) 48 | - `python -m unittest` 49 | 50 | For development, we use [`tox`](https://tox.readthedocs.io/en/latest/) 51 | to manage an extensive suite of auto-formatters and linters, so: 52 | 53 | - `pip install tox` 54 | - `tox` 55 | 56 | will set up a virtualenv for you, install everything, and finally run 57 | the formatters, linters, and test suite. 58 | 59 | 60 | ## Contributors 61 | 62 | - [Zac Hatfield-Dodds](https://zhd.dev) 63 | - [Paul Ganssle](https://ganssle.io) 64 | - [Carl Friedrich Bolz-Tereick](http://cfbolz.de/) 65 | 66 | 67 | ## Trophy Case 68 | Bugs found via this specific project: 69 | 70 | - [BPO-40661, a segfault in the new parser](https://bugs.python.org/issue40661), 71 | was given maximum priority and [blocked the planned release](https://discuss.python.org/t/all-hands-on-deck-the-release-of-python-3-9-0b1-is-currently-blocked/4217) 72 | of CPython 3.9 beta1. 73 | - [`OverflowError` in `binascii.crc_hqx()`](https://github.com/Zac-HD/stdlib-property-tests/pull/18#issuecomment-631426084) under PyPy 74 | - [BPO-38953](https://bugs.python.org/issue38953) `tokenize.tokenize` -> 75 | `tokenize.untokenize` does not round-trip as documented. 76 | Nor, for that matter, do the `tokenize`/`untokenize` functions in 77 | `lib2to3.pgen.tokenize`. 78 | - [PEP-615 (zoneinfo)](https://github.com/pganssle/zoneinfo/pull/32/commits/dc389beaaeaa702361fd186d8581da20dda807bb) 79 | `fold` detection failed for early transitions when the number of elapsed 80 | seconds is too large to fit in a C integer; and 81 | [a `fold` inconsistency](https://github.com/pganssle/zoneinfo/pull/41) 82 | where first offset handling was broken in the C extension. 83 | - [BPO-40668](https://bugs.python.org/issue40668), catastrophic loss of precision when attempting to round-trip YIQ-RGB-YIQ 84 | with the `colorsys` module - more than 10% of the possible range. (via #13) 85 | 86 | 87 | ## Further reading 88 | 89 | - Hypothesis' [website](https://hypothesis.works/), 90 | [documentation](https://hypothesis.readthedocs.io/), 91 | [GitHub repo](https://github.com/HypothesisWorks/hypothesis) 92 | - [Introductory articles](https://hypothesis.works/articles/intro/), 93 | [simple properties](https://fsharpforfunandprofit.com/posts/property-based-testing-2/), 94 | [metamorphic properties](https://www.hillelwayne.com/post/metamorphic-testing/) 95 | - Related thoughts from 96 | [hardware testing and verification](https://danluu.com/testing/), 97 | [testing a screencast editor](https://wickstrom.tech/programming/2019/03/02/property-based-testing-in-a-screencast-editor-introduction.html), 98 | [PBT in Erlang / Elixr](https://propertesting.com/toc.html), 99 | [testing C code using Hypothesis](https://engineering.backtrace.io/posts/2020-03-11-how-hard-is-it-to-guide-test-case-generators-with-branch-coverage-feedback/) 100 | - [`python-afl`](https://github.com/jwilk/python-afl) or 101 | [OSS-FUZZ](https://github.com/google/oss-fuzz) could work very nicely with 102 | [Hypothesis' fuzz support](https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers) 103 | - [`hypothesmith`](https://github.com/Zac-HD/hypothesmith) 104 | generates syntatically-valid but weird Python source code 105 | (e.g. [BPO-38953](https://bugs.python.org/issue38953) or 106 | [psf/black#970](https://github.com/psf/black/issues/970)). 107 | [Using a syntax tree](https://github.com/Zac-HD/hypothesmith/issues/2) 108 | for semantic validity is the logical next step. 109 | -------------------------------------------------------------------------------- /requirements-dev.in: -------------------------------------------------------------------------------- 1 | # Top-level dependencies for `tox -e check` 2 | flake8 3 | flake8-assertive 4 | flake8-bandit 5 | flake8-bugbear 6 | flake8-builtins 7 | flake8-comprehensions 8 | flake8-print 9 | pep8-naming 10 | shed 11 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile 3 | # To update, run: 4 | # 5 | # tox -e deps 6 | # 7 | appdirs==1.4.4 # via black 8 | attrs==19.3.0 # via black, flake8-bugbear 9 | autoflake==1.3.1 # via shed 10 | bandit==1.6.2 # via flake8-bandit 11 | black==19.10b0 # via shed 12 | click==7.1.2 # via black 13 | colorama==0.4.3 # via bandit 14 | flake8-assertive==1.2.1 # via -r requirements-dev.in 15 | flake8-bandit==2.1.2 # via -r requirements-dev.in 16 | flake8-bugbear==20.1.4 # via -r requirements-dev.in 17 | flake8-builtins==1.5.3 # via -r requirements-dev.in 18 | flake8-comprehensions==3.2.3 # via -r requirements-dev.in 19 | flake8-polyfill==1.0.2 # via flake8-bandit, pep8-naming 20 | flake8-print==3.1.4 # via -r requirements-dev.in 21 | flake8==3.8.3 # via -r requirements-dev.in, flake8-assertive, flake8-bandit, flake8-bugbear, flake8-builtins, flake8-comprehensions, flake8-polyfill, flake8-print 22 | gitdb==4.0.5 # via gitpython 23 | gitpython==3.1.7 # via bandit 24 | importlib-metadata==1.7.0 # via flake8, flake8-comprehensions, stevedore 25 | isort==5.4.2 # via shed 26 | mccabe==0.6.1 # via flake8 27 | pathspec==0.8.0 # via black 28 | pbr==5.4.5 # via stevedore 29 | pep8-naming==0.11.1 # via -r requirements-dev.in 30 | pycodestyle==2.6.0 # via flake8, flake8-bandit, flake8-print 31 | pyflakes==2.2.0 # via autoflake, flake8 32 | pyupgrade==2.7.2 # via shed 33 | pyyaml==5.3.1 # via bandit 34 | regex==2020.7.14 # via black 35 | shed==0.1.1 # via -r requirements-dev.in 36 | six==1.15.0 # via bandit, flake8-print 37 | smmap==3.0.4 # via gitdb 38 | stevedore==3.2.0 # via bandit 39 | tokenize-rt==4.0.0 # via pyupgrade 40 | toml==0.10.1 # via black 41 | typed-ast==1.4.1 # via black 42 | zipp==3.1.0 # via importlib-metadata 43 | -------------------------------------------------------------------------------- /requirements.in: -------------------------------------------------------------------------------- 1 | # Top-level dependencies for `tox -e test` 2 | hypothesis[dateutil] 3 | hypothesmith 4 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile 3 | # To update, run: 4 | # 5 | # tox -e deps 6 | # 7 | attrs==19.3.0 # via hypothesis 8 | hypothesis[dateutil]==5.26.0 # via -r requirements.in, hypothesmith 9 | hypothesmith==0.1.4 # via -r requirements.in 10 | lark-parser==0.9.0 # via hypothesmith 11 | libcst==0.3.10 # via hypothesmith 12 | mypy-extensions==0.4.3 # via typing-inspect 13 | python-dateutil==2.8.1 # via hypothesis 14 | pyyaml==5.3.1 # via libcst 15 | six==1.15.0 # via python-dateutil 16 | sortedcontainers==2.2.2 # via hypothesis 17 | typing-extensions==3.7.4.2 # via libcst, typing-inspect 18 | typing-inspect==0.6.0 # via libcst 19 | -------------------------------------------------------------------------------- /tests/test_compression.py: -------------------------------------------------------------------------------- 1 | import bz2 2 | import gzip 3 | import lzma 4 | import unittest 5 | import zlib 6 | 7 | from hypothesis import HealthCheck, given, settings, strategies as st 8 | 9 | no_health_checks = settings(suppress_health_check=HealthCheck.all()) 10 | 11 | 12 | @st.composite 13 | def lzma_filters(draw): 14 | """Generating filters options""" 15 | op_filters = [ 16 | lzma.FILTER_DELTA, 17 | lzma.FILTER_X86, 18 | lzma.FILTER_IA64, 19 | lzma.FILTER_ARM, 20 | lzma.FILTER_ARMTHUMB, 21 | lzma.FILTER_POWERPC, 22 | lzma.FILTER_SPARC, 23 | ] 24 | filter_ids = draw(st.lists(st.sampled_from(op_filters), max_size=3)) 25 | filter_ids.append(lzma.FILTER_LZMA2) 26 | # create filters options 27 | filters = [] 28 | for filter_ in filter_ids: 29 | lc = draw(st.integers(0, 4)) 30 | lp = draw(st.integers(0, 4 - lc)) 31 | mf = [lzma.MF_HC3, lzma.MF_HC4, lzma.MF_BT2, lzma.MF_BT3, lzma.MF_BT4] 32 | filters.append( 33 | { 34 | "id": filter_, 35 | "preset": draw(st.integers(0, 9)), 36 | "lc": lc, 37 | "lp": lp, 38 | "mode": draw(st.sampled_from([lzma.MODE_FAST, lzma.MODE_NORMAL])), 39 | "mf": draw(st.sampled_from(mf)), 40 | "depth": draw(st.integers(min_value=0)), 41 | } 42 | ) 43 | return filters 44 | 45 | 46 | class TestBz2(unittest.TestCase): 47 | @given(payload=st.binary(), compresslevel=st.integers(1, 9)) 48 | def test_bz2_round_trip(self, payload, compresslevel): 49 | result = bz2.decompress(bz2.compress(payload, compresslevel=compresslevel)) 50 | self.assertEqual(payload, result) 51 | 52 | @given(payloads=st.lists(st.binary()), compresslevel=st.integers(1, 9)) 53 | def test_bz2_incremental_compress_eq_oneshot(self, payloads, compresslevel): 54 | c = bz2.BZ2Compressor(compresslevel) 55 | compressed = b"".join(c.compress(p) for p in payloads) + c.flush() 56 | self.assertEqual(compressed, bz2.compress(b"".join(payloads), compresslevel)) 57 | 58 | @no_health_checks 59 | @given(payload=st.binary(), compresslevel=st.integers(1, 9), data=st.data()) 60 | def test_bz2_incremental_decompress_eq_oneshot(self, payload, compresslevel, data): 61 | compressed = bz2.compress(payload, compresslevel=compresslevel) 62 | d = bz2.BZ2Decompressor() 63 | 64 | output = [] 65 | while compressed: 66 | chunksize = data.draw(st.integers(int(not d.needs_input), len(compressed))) 67 | max_length = data.draw(st.integers(0, len(payload))) 68 | output.append(d.decompress(compressed[:chunksize], max_length=max_length)) 69 | self.assertLessEqual(len(output[-1]), max_length) 70 | compressed = compressed[chunksize:] 71 | if not d.eof: 72 | self.assertFalse(d.needs_input) 73 | output.append(d.decompress(b"")) 74 | 75 | self.assertEqual(payload, b"".join(output)) 76 | 77 | 78 | class TestGzip(unittest.TestCase): 79 | @given(payload=st.binary(), compresslevel=st.integers(0, 9)) 80 | def test_gzip_round_trip(self, payload, compresslevel): 81 | result = gzip.decompress(gzip.compress(payload, compresslevel=compresslevel)) 82 | self.assertEqual(payload, result) 83 | 84 | 85 | class TestLZMA(unittest.TestCase): 86 | # TODO: https://docs.python.org/3/library/lzma.html 87 | @given( 88 | payload=st.binary(), 89 | check=st.sampled_from( 90 | [lzma.CHECK_NONE, lzma.CHECK_CRC32, lzma.CHECK_CRC64, lzma.CHECK_SHA256] 91 | ), 92 | compresslevel=st.integers(0, 9), 93 | ) 94 | def test_lzma_round_trip_format_xz(self, payload, check, compresslevel): 95 | result = lzma.decompress( 96 | lzma.compress( 97 | payload, format=lzma.FORMAT_XZ, check=check, preset=compresslevel 98 | ) 99 | ) 100 | self.assertEqual(payload, result) 101 | 102 | @given( 103 | payload=st.binary(), compresslevel=st.integers(0, 9), 104 | ) 105 | def test_lzma_round_trip_format_alone(self, payload, compresslevel): 106 | result = lzma.decompress( 107 | lzma.compress(payload, format=lzma.FORMAT_ALONE, preset=compresslevel) 108 | ) 109 | self.assertEqual(payload, result) 110 | 111 | @unittest.skip(reason="LZMA filter strategy too general?") 112 | @given(payload=st.binary(), filters=lzma_filters()) 113 | def test_lzma_round_trip_format_raw(self, payload, filters): 114 | # This test is a stub from our attempt to write a round-trip test with 115 | # custom LZMA filters (from the strategy above). Ultimately we decided 116 | # to defer implementation to a future PR and merge what we had working. 117 | # TODO: work out what's happening here and fix it. 118 | compressed = lzma.compress(payload, format=lzma.FORMAT_RAW, filters=filters) 119 | self.assertEqual(payload, lzma.decompress(compressed)) 120 | 121 | 122 | class TestZlib(unittest.TestCase): 123 | # TODO: https://docs.python.org/3/library/zlib.html 124 | @given( 125 | payload=st.lists(st.binary(), min_size=1, max_size=2), 126 | checksum=st.just(1) | st.integers(), 127 | ) 128 | def test_adler32(self, payload, checksum): 129 | expected = zlib.adler32(b"".join(payload), checksum) 130 | for piece in payload: 131 | checksum = zlib.adler32(piece, checksum) 132 | self.assertIsInstance(checksum, int) 133 | self.assertLess(checksum, 2 ** 32) 134 | self.assertGreaterEqual(checksum, 0) 135 | self.assertEqual(expected, checksum) 136 | 137 | @given( 138 | payload=st.lists(st.binary(), min_size=1, max_size=2), 139 | checksum=st.just(1) | st.integers(), 140 | ) 141 | def test_crc32(self, payload, checksum): 142 | expected = zlib.crc32(b"".join(payload), checksum) 143 | for piece in payload: 144 | checksum = zlib.crc32(piece, checksum) 145 | self.assertIsInstance(checksum, int) 146 | self.assertLess(checksum, 2 ** 32) 147 | self.assertGreaterEqual(checksum, 0) 148 | self.assertEqual(expected, checksum) 149 | 150 | # TODO: tests for incremental compression, wbits, strategy, zdict 151 | @given( 152 | payload=st.binary(), 153 | level=st.just(-1) | st.integers(0, 9), 154 | bufsize=st.just(zlib.DEF_BUF_SIZE) | st.integers(0, zlib.DEF_BUF_SIZE), 155 | ) 156 | def test_compress_decompress_round_trip(self, payload, level, bufsize): 157 | x = zlib.compress(payload, level=level) 158 | self.assertEqual(payload, zlib.decompress(x, bufsize=bufsize)) 159 | -------------------------------------------------------------------------------- /tests/test_datetime.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from datetime import date, datetime, time, timezone 3 | 4 | from hypothesis import given, strategies as st 5 | from hypothesis.extra.dateutil import timezones as dateutil_timezones 6 | 7 | TIME_ZONES_STRATEGY = st.one_of( 8 | st.sampled_from([None, timezone.utc]), dateutil_timezones() 9 | ) 10 | 11 | 12 | class TestDatetime(unittest.TestCase): 13 | # Surrogate characters have been particularly problematic here in the past, 14 | # so we give them a boost by combining strategies pending an upstream 15 | # feature (https://github.com/HypothesisWorks/hypothesis/issues/1401) 16 | @unittest.skipIf(not hasattr(datetime, "fromisoformat"), reason="new in 3.7") 17 | @given( 18 | dt=st.datetimes(timezones=TIME_ZONES_STRATEGY), 19 | sep=st.characters() | st.characters(whitelist_categories=["Cs"]), 20 | ) 21 | def test_fromisoformat_auto(self, dt, sep): 22 | """Test isoformat with timespec="auto".""" 23 | dtstr = dt.isoformat(sep=sep, timespec="auto") 24 | dt_rt = datetime.fromisoformat(dtstr) 25 | self.assertEqual(dt, dt_rt) 26 | 27 | @unittest.skipIf(not hasattr(datetime, "fromisocalendar"), reason="new in 3.8") 28 | @given(dt=st.datetimes()) 29 | def test_fromisocalendar_date_datetime(self, dt): 30 | isocalendar = dt.isocalendar() 31 | dt_rt = datetime.fromisocalendar(*isocalendar) 32 | 33 | # Only the date portion of the datetime survives a round trip. 34 | d = dt.date() 35 | d_rt = dt_rt.date() 36 | 37 | self.assertEqual(d, d_rt) 38 | 39 | # .fromisocalendar() should always return a datetime at midnight 40 | t = time(0) 41 | t_rt = dt_rt.time() 42 | 43 | self.assertEqual(t, t_rt) 44 | 45 | # TODO: https://docs.python.org/3/library/datetime.html 46 | # e.g. round-trip for other serialization / deserialization pairs. 47 | # Use exhaustive testing rather than Hypothesis where possible! 48 | 49 | 50 | class TestDate(unittest.TestCase): 51 | @unittest.skipIf(not hasattr(datetime, "fromisocalendar"), reason="new in 3.8") 52 | @given(d=st.dates()) 53 | def test_fromisocalendar(self, d): 54 | isocalendar = d.isocalendar() 55 | d_rt = date.fromisocalendar(*isocalendar) 56 | 57 | self.assertEqual(d, d_rt) 58 | -------------------------------------------------------------------------------- /tests/test_encode_decode.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import binascii 3 | import binhex 4 | import colorsys 5 | import io 6 | import os 7 | import platform 8 | import plistlib 9 | import quopri 10 | import string 11 | import sys 12 | import unittest 13 | import uu 14 | from tempfile import TemporaryDirectory 15 | 16 | from hypothesis import HealthCheck, example, given, settings, strategies as st, target 17 | 18 | IS_PYPY = platform.python_implementation() == "PyPy" 19 | 20 | 21 | def add_padding(payload): 22 | """Add the expected padding for test_b85_encode_decode_round_trip.""" 23 | if len(payload) % 4 != 0: 24 | padding = b"\0" * ((-len(payload)) % 4) 25 | payload = payload + padding 26 | return payload 27 | 28 | 29 | @st.composite 30 | def altchars(draw): 31 | """Generate 'altchars' for base64 encoding. 32 | 33 | Via https://docs.python.org/3/library/base64.html#base64.b64encode : 34 | "Optional altchars must be a bytes-like object of at least length 2 35 | (additional characters are ignored) which specifies an alternative 36 | alphabet for the + and / characters." 37 | """ 38 | reserved_chars = (string.digits + string.ascii_letters + "=").encode("ascii") 39 | allowed_chars = st.sampled_from([n for n in range(256) if n not in reserved_chars]) 40 | return bytes(draw(st.lists(allowed_chars, min_size=2, max_size=2, unique=True))) 41 | 42 | 43 | class TestBase64(unittest.TestCase): 44 | @given( 45 | payload=st.binary(), 46 | altchars=st.none() | st.just(b"_-") | altchars(), 47 | validate=st.booleans(), 48 | ) 49 | def test_b64_encode_decode_round_trip(self, payload, altchars, validate): 50 | x = base64.b64encode(payload, altchars=altchars) 51 | self.assertEqual( 52 | payload, base64.b64decode(x, altchars=altchars, validate=validate) 53 | ) 54 | 55 | @given(payload=st.binary()) 56 | def test_standard_b64_encode_decode_round_trip(self, payload): 57 | x = base64.standard_b64encode(payload) 58 | self.assertEqual(payload, base64.standard_b64decode(x)) 59 | 60 | @given(payload=st.binary()) 61 | def test_urlsafe_b64_encode_decode_round_trip(self, payload): 62 | x = base64.urlsafe_b64encode(payload) 63 | self.assertEqual(payload, base64.urlsafe_b64decode(x)) 64 | 65 | @given( 66 | payload=st.binary(), 67 | casefold=st.booleans(), 68 | map01=(st.none() | st.binary(min_size=1, max_size=1)), 69 | ) 70 | def test_b32_encode_decode_round_trip(self, payload, casefold, map01): 71 | x = base64.b32encode(payload) 72 | self.assertEqual(payload, base64.b32decode(x, casefold=casefold, map01=map01)) 73 | 74 | @given(payload=st.binary(), casefold=st.booleans()) 75 | def test_b16_encode_decode_round_trip(self, payload, casefold): 76 | x = base64.b16encode(payload) 77 | self.assertEqual(payload, base64.b16decode(x, casefold=casefold)) 78 | 79 | @given( 80 | payload=st.binary(), 81 | foldspaces=st.booleans(), 82 | wrapcol=(st.just(0) | st.integers(0, 1000)), 83 | pad=st.booleans(), 84 | adobe=st.booleans(), 85 | ) 86 | def test_a85_encode_decode_round_trip( 87 | self, payload, foldspaces, wrapcol, pad, adobe 88 | ): 89 | x = base64.a85encode( 90 | payload, foldspaces=foldspaces, wrapcol=wrapcol, pad=pad, adobe=adobe 91 | ) 92 | # adding padding manually to payload, when pad is True and len(payload)%4!=0 93 | if pad: 94 | payload = add_padding(payload) 95 | self.assertEqual( 96 | payload, base64.a85decode(x, foldspaces=foldspaces, adobe=adobe) 97 | ) 98 | 99 | @given(payload=st.binary(), pad=st.booleans()) 100 | def test_b85_encode_decode_round_trip(self, payload, pad): 101 | x = base64.b85encode(payload, pad=pad) 102 | # adding padding manually to payload, when pad is True and len(payload)%4!=0 103 | if pad: 104 | payload = add_padding(payload) 105 | self.assertEqual(payload, base64.b85decode(x)) 106 | 107 | 108 | class TestBinASCII(unittest.TestCase): 109 | @given(payload=st.binary(max_size=45), backtick=st.booleans()) 110 | def test_b2a_uu_a2b_uu_round_trip(self, payload, backtick): 111 | if sys.version_info[:2] >= (3, 7): 112 | x = binascii.b2a_uu(payload, backtick=backtick) 113 | else: 114 | x = binascii.b2a_uu(payload) 115 | self.assertEqual(payload, binascii.a2b_uu(x)) 116 | 117 | @given(payload=st.binary(), newline=st.booleans()) 118 | def test_b2a_base64_a2b_base64_round_trip(self, payload, newline): 119 | x = binascii.b2a_base64(payload, newline=newline) 120 | self.assertEqual(payload, binascii.a2b_base64(x)) 121 | 122 | @given( 123 | payload=st.binary(), 124 | quotetabs=st.booleans(), 125 | istext=st.booleans(), 126 | header=st.booleans(), 127 | ) 128 | def test_b2a_qp_a2b_qp_round_trip(self, payload, quotetabs, istext, header): 129 | x = binascii.b2a_qp(payload, quotetabs=quotetabs, istext=istext, header=header) 130 | self.assertEqual(payload, binascii.a2b_qp(x, header=header)) 131 | 132 | @given(payload=st.binary()) 133 | def test_rlecode_hqx_rledecode_hqx_round_trip(self, payload): 134 | x = binascii.rlecode_hqx(payload) 135 | self.assertEqual(payload, binascii.rledecode_hqx(x)) 136 | 137 | @given(payload=st.binary()) 138 | def test_b2a_hqx_a2b_hqx_round_trip(self, payload): 139 | # assuming len(payload) as 3, since it throws exception: binascii.Incomplete, when length is not a multiple of 3 140 | if len(payload) % 3: 141 | with self.assertRaises(binascii.Incomplete): 142 | x = binascii.b2a_hqx(payload) 143 | binascii.a2b_hqx(x) 144 | payload += b"\x00" * (-len(payload) % 3) 145 | x = binascii.b2a_hqx(payload) 146 | res, _ = binascii.a2b_hqx(x) 147 | self.assertEqual(payload, res) 148 | 149 | @unittest.skipIf(IS_PYPY, "we found an overflow bug") 150 | @given(payload=st.binary(), value=st.just(0) | st.integers()) 151 | @example(payload=b"", value=2 ** 63) 152 | def test_crc_hqx(self, payload, value): 153 | crc = binascii.crc_hqx(payload, value) 154 | self.assertIsInstance(crc, int) 155 | 156 | @unittest.skipIf(IS_PYPY, "we found an overflow bug") 157 | @given( 158 | payload_piece_1=st.binary(), 159 | payload_piece_2=st.binary(), 160 | value=st.just(0) | st.integers(), 161 | ) 162 | def test_crc_hqx_two_pieces(self, payload_piece_1, payload_piece_2, value): 163 | combined_crc = binascii.crc_hqx(payload_piece_1 + payload_piece_2, value) 164 | crc_part1 = binascii.crc_hqx(payload_piece_1, value) 165 | crc = binascii.crc_hqx(payload_piece_2, crc_part1) 166 | self.assertEqual(combined_crc, crc) 167 | 168 | @given(payload=st.binary(), value=st.just(0) | st.integers()) 169 | def test_crc32(self, payload, value): 170 | crc = binascii.crc32(payload, value) 171 | self.assertIsInstance(crc, int) 172 | 173 | @given( 174 | payload_piece_1=st.binary(), 175 | payload_piece_2=st.binary(), 176 | value=st.just(0) | st.integers(), 177 | ) 178 | def test_crc32_two_part(self, payload_piece_1, payload_piece_2, value): 179 | combined_crc = binascii.crc32(payload_piece_1 + payload_piece_2, value) 180 | crc_part1 = binascii.crc32(payload_piece_1, value) 181 | crc = binascii.crc32(payload_piece_2, crc_part1) 182 | self.assertEqual(combined_crc, crc) 183 | 184 | @given(payload=st.binary()) 185 | def test_b2a_hex_a2b_hex_round_trip(self, payload): 186 | x = binascii.b2a_hex(payload) 187 | self.assertEqual(payload, binascii.a2b_hex(x)) 188 | 189 | @given(payload=st.binary()) 190 | def test_hexlify_unhexlify_round_trip(self, payload): 191 | x = binascii.hexlify(payload) 192 | self.assertEqual(payload, binascii.unhexlify(x)) 193 | 194 | 195 | class TestColorsys(unittest.TestCase): 196 | # Module documentation https://docs.python.org/3/library/colorsys.html 197 | 198 | def assertColorsValid(self, **colors): 199 | assert len(colors) == 3 # sanity-check 200 | # Our color assertion helper checks that each color is in the range 201 | # [0, 1], and that it approximately round-tripped. We also "target" 202 | # the difference, to maximise and report the largest error each run. 203 | for name, values in colors.items(): 204 | for v in values: 205 | self.assertGreaterEqual( 206 | v, 0 if name not in "iq" else -1, msg=f"color={name!r}" 207 | ) 208 | self.assertLessEqual(v, 1, msg=f"color={name!r}") 209 | target( 210 | abs(values[0] - values[1]), 211 | label=f"absolute difference in {name.upper()} values", 212 | ) 213 | self.assertAlmostEqual(*values, msg=f"color={name!r}") 214 | 215 | @given(r=st.floats(0, 1), g=st.floats(0, 1), b=st.floats(0, 1)) 216 | def test_rgb_yiq_round_trip(self, r, g, b): 217 | y, i, q = colorsys.rgb_to_yiq(r, g, b) 218 | r2, g2, b2 = colorsys.yiq_to_rgb(y, i, q) 219 | self.assertColorsValid(r=(r, r2), g=(g, g2), b=(b, b2)) 220 | 221 | # Allowed ranges for I and Q values are not documented in CPython 222 | # https://docs.python.org/3/library/colorsys.html - and code comments 223 | # note "I and Q ... covers a slightly larger range [than `[0, 1`]]". 224 | # We therefore follow https://en.wikipedia.org/wiki/YIQ#Preconditions 225 | @unittest.expectedFailure 226 | @given( 227 | y=st.floats(0.0, 1.0), 228 | i=st.floats(-0.5957, 0.5957), 229 | q=st.floats(-0.5226, 0.5226), 230 | ) 231 | def test_yiq_rgb_round_trip(self, y, i, q): 232 | r, g, b = colorsys.yiq_to_rgb(y, i, q) 233 | y2, i2, q2 = colorsys.rgb_to_yiq(r, g, b) 234 | self.assertColorsValid(y=(y, y2), i=(i, i2), q=(q, q2)) 235 | 236 | @unittest.expectedFailure 237 | @given(r=st.floats(0, 1), g=st.floats(0, 1), b=st.floats(0, 1)) 238 | @example(r=0.5714285714285715, g=0.0, b=2.2204460492503136e-16) 239 | def test_rgb_hls_round_trip(self, r, g, b): 240 | h, l, s = colorsys.rgb_to_hls(r, g, b) 241 | r2, g2, b2 = colorsys.hls_to_rgb(h, l, s) 242 | self.assertColorsValid(r=(r, r2), g=(g, g2), b=(b, b2)) 243 | 244 | h2, l2, s2 = colorsys.rgb_to_hls(r2, g2, b2) 245 | self.assertColorsValid(h=(h, h2), l=(l, l2), s=(s, s2)) 246 | 247 | @unittest.expectedFailure 248 | @given(r=st.floats(0, 1), g=st.floats(0, 1), b=st.floats(0, 1)) 249 | def test_rgb_hsv_round_trip(self, r, g, b): 250 | h, s, v = colorsys.rgb_to_hsv(r, g, b) 251 | r2, g2, b2 = colorsys.hsv_to_rgb(h, s, v) 252 | self.assertColorsValid(r=(r, r2), g=(g, g2), b=(b, b2)) 253 | 254 | h2, s2, v2 = colorsys.rgb_to_hls(r2, g2, b2) 255 | self.assertColorsValid(h=(h, h2), s=(s, s2), v=(v, v2)) 256 | 257 | 258 | text_strategy = st.text(alphabet=st.characters(blacklist_categories=["Cc", "Cs"])) 259 | 260 | plistlib_data = st.recursive( 261 | st.booleans() 262 | | st.binary() 263 | | st.datetimes().map(lambda d: d.replace(microsecond=0)) 264 | | st.integers(min_value=-1 << 63, max_value=1 << 64 - 1) 265 | | st.floats(allow_nan=False) 266 | | text_strategy, 267 | lambda sub_strategy: st.lists(sub_strategy) 268 | | st.dictionaries(text_strategy, sub_strategy), 269 | ) 270 | 271 | 272 | class TestPlistlib(unittest.TestCase): 273 | @settings(suppress_health_check=HealthCheck.all()) 274 | @given( 275 | payload=plistlib_data, 276 | fmt=st.sampled_from([plistlib.FMT_XML, plistlib.FMT_BINARY]), 277 | pass_format_arg=st.booleans(), 278 | ) 279 | def test_dumps_loads(self, payload, fmt, pass_format_arg): 280 | plist_dump = plistlib.dumps(payload, fmt=fmt) 281 | plist_load = plistlib.loads(plist_dump, fmt=fmt if pass_format_arg else None) 282 | self.assertEqual(payload, plist_load) 283 | 284 | 285 | class TestQuopri(unittest.TestCase): 286 | @unittest.expectedFailure 287 | @given(payload=st.binary(), quotetabs=st.booleans(), header=st.booleans()) 288 | @example(payload=b"\n\r\n", quotetabs=False, header=False) 289 | @example(payload=b"\r\n\n", quotetabs=False, header=False) 290 | def test_quopri_encode_decode_round_trip(self, payload, quotetabs, header): 291 | encoded = quopri.encodestring(payload, quotetabs=quotetabs, header=header) 292 | decoded = quopri.decodestring(encoded, header=header) 293 | self.assertEqual(payload, decoded) 294 | 295 | 296 | class TestBinhex(unittest.TestCase): 297 | @given(payload=st.binary()) 298 | def test_binhex_encode_decode(self, payload): 299 | with TemporaryDirectory() as dirname: 300 | input_file_name = os.path.join(dirname, "input.txt") 301 | encoded_file_name = os.path.join(dirname, "encoded.hqx") 302 | decoded_file_name = os.path.join(dirname, "decoded.txt") 303 | with open(input_file_name, "wb") as input_file: 304 | input_file.write(payload) 305 | binhex.binhex(input_file_name, encoded_file_name) 306 | binhex.hexbin(encoded_file_name, decoded_file_name) 307 | with open(decoded_file_name, "rb") as decoded_file: 308 | decoded_payload = decoded_file.read() 309 | assert payload == decoded_payload 310 | 311 | 312 | class TestUu(unittest.TestCase): 313 | @given( 314 | payload=st.binary(), 315 | name=st.none() | st.just("-") | st.just("0o666"), 316 | quiet=st.binary(), 317 | ) 318 | def test_uu_encode_decode(self, payload, name, quiet): 319 | input_file = io.BytesIO(payload) 320 | encoded_file = io.BytesIO() 321 | decoded_file = io.BytesIO() 322 | uu.encode(input_file, encoded_file, name=name) 323 | encoded_file.seek(0) 324 | uu.decode(encoded_file, out_file=decoded_file, quiet=quiet) 325 | decoded_payload = decoded_file.getvalue() 326 | assert payload == decoded_payload 327 | -------------------------------------------------------------------------------- /tests/test_json.py: -------------------------------------------------------------------------------- 1 | import json 2 | import unittest 3 | 4 | from hypothesis import given, strategies as st 5 | 6 | INDENTS = st.one_of( 7 | st.none(), st.integers(-2, 100), st.from_regex("[ \t\n\r]*", fullmatch=True) 8 | ) 9 | 10 | 11 | def jsondata(all_finite=False): 12 | # The JSON spec does not allow non-finite numbers; so Python's json module has 13 | # an option to reject such numbers. We therefore make it easy to avoid them. 14 | if all_finite: 15 | floats = st.floats(allow_infinity=False, allow_nan=False) 16 | else: 17 | floats = st.floats() 18 | return st.recursive( 19 | st.one_of(st.none(), st.booleans(), floats, st.text()), 20 | lambda children: st.lists(children) | st.dictionaries(st.text(), children), 21 | ) 22 | 23 | 24 | class TestJson(unittest.TestCase): 25 | @given( 26 | allow_nan=st.booleans(), 27 | ensure_ascii=st.booleans(), 28 | indent=INDENTS, 29 | obj=jsondata(all_finite=True), 30 | sort_keys=st.booleans(), 31 | ) 32 | def test_roundtrip_dumps_loads( 33 | self, allow_nan, ensure_ascii, indent, obj, sort_keys 34 | ): 35 | # For any self-equal JSON object, we can deserialise it to an equal object. 36 | # (regardless of how we vary string encoding, indentation, and sorting) 37 | self.assertEqual(obj, obj) 38 | deserialised = json.loads( 39 | json.dumps( 40 | obj, 41 | allow_nan=allow_nan, 42 | ensure_ascii=ensure_ascii, 43 | indent=indent, 44 | sort_keys=sort_keys, 45 | ) 46 | ) 47 | self.assertEqual(obj, deserialised) 48 | 49 | @given( 50 | ensure_ascii=st.booleans(), 51 | indent=INDENTS, 52 | obj=jsondata(), 53 | sort_keys=st.booleans(), 54 | ) 55 | def test_roundtrip_dumps_loads_dumps(self, ensure_ascii, indent, obj, sort_keys): 56 | # For any json object, even those with non-finite numbers and with all the 57 | # variations as above, object -> string -> object -> string should produce 58 | # identical strings. 59 | as_str = json.dumps( 60 | obj, ensure_ascii=ensure_ascii, indent=indent, sort_keys=sort_keys 61 | ) 62 | deserialised = json.loads(as_str) 63 | as_str2 = json.dumps( 64 | deserialised, ensure_ascii=ensure_ascii, indent=indent, sort_keys=sort_keys 65 | ) 66 | self.assertEqual(as_str, as_str2) 67 | -------------------------------------------------------------------------------- /tests/test_marshal.py: -------------------------------------------------------------------------------- 1 | import marshal 2 | import unittest 3 | 4 | from hypothesis import given, strategies as st 5 | 6 | simple_immutables = ( 7 | st.integers() 8 | | st.booleans() 9 | | st.floats(allow_nan=False) # NaNs compare unequal to themselves 10 | | st.complex_numbers(allow_nan=False) 11 | | st.just(None) 12 | | st.binary() 13 | | st.text() 14 | ) 15 | composite_immutables = st.recursive( 16 | simple_immutables, 17 | lambda children: st.tuples(children) | st.frozensets(children, max_size=1), 18 | max_leaves=20, 19 | ) 20 | marshallable_data = st.recursive( 21 | composite_immutables | st.sets(composite_immutables), 22 | lambda children: st.lists(children, max_size=20) 23 | | st.dictionaries(composite_immutables, children), 24 | max_leaves=20, 25 | ) 26 | 27 | 28 | class TestMarshal(unittest.TestCase): 29 | @given(dt=marshallable_data) 30 | def test_roundtrip(self, dt): 31 | b = marshal.dumps(dt) 32 | dt2 = marshal.loads(b) # noqa: S302 # using marshal is fine here 33 | self.assertEqual(dt, dt2) 34 | -------------------------------------------------------------------------------- /tests/test_misc.py: -------------------------------------------------------------------------------- 1 | import imghdr 2 | import sys 3 | import unittest 4 | 5 | from hypothesis import example, given, strategies as st 6 | 7 | 8 | class TestBuiltins(unittest.TestCase): 9 | @example(n=2 ** 63) 10 | @given(st.integers(min_value=0)) 11 | def test_len_of_range(self, n): 12 | seq = range(n) 13 | try: 14 | length = len(seq) 15 | except OverflowError: 16 | # len() internally casts to ssize_t, so we expect overflow here. 17 | self.assertGreater(n, sys.maxsize) 18 | else: 19 | self.assertEqual(length, n) 20 | 21 | 22 | class TestImghdr(unittest.TestCase): 23 | @given(st.binary()) 24 | def test_imghdr_what(self, payload): 25 | imghdr.what("", h=payload) 26 | -------------------------------------------------------------------------------- /tests/test_re_by_construction.py: -------------------------------------------------------------------------------- 1 | """ Test the re module by constructing random syntax trees of (a subset of) 2 | regular expressions. A tree can be used to generate matching or optionally 3 | non-matching strings. A tree can also be turned into re syntax.""" 4 | 5 | import re 6 | import sys 7 | import time 8 | import unittest 9 | from contextlib import contextmanager 10 | from functools import partial 11 | 12 | from hypothesis import given, reject, strategies as st 13 | from hypothesis.errors import InvalidArgument 14 | 15 | IS_PYPY = hasattr(sys, "pypy_version_info") 16 | 17 | special_characters = ".^$*+?{}\\[]-|()#=!" 18 | MAXREPEAT = 7 19 | 20 | 21 | class State: 22 | def __init__(self): 23 | self.groups = {} 24 | 25 | 26 | class CantGenerateNonMatching(Exception): 27 | pass 28 | 29 | 30 | class Re: 31 | """ abstract base class for regular expression syntax nodes """ 32 | 33 | def can_be_empty(self): 34 | """ Can self match the empty string? """ 35 | return False 36 | 37 | def matching_string(self, draw, state): 38 | """ use draw to generate a string that is known to match self """ 39 | raise NotImplementedError 40 | 41 | def non_matching_string(self, draw, state): 42 | """ try to use draw to generate a string that *doesn't* match self. Can 43 | fail by raising CantGenerateNonMatching """ 44 | raise NotImplementedError 45 | 46 | def build_re(self): 47 | """ Build the re syntax for self """ 48 | raise NotImplementedError 49 | 50 | 51 | class Char(Re): 52 | """ Matches a single character self.c""" 53 | 54 | def __init__(self, c): 55 | self.c = c 56 | 57 | def matching_string(self, draw, state): 58 | return self.c 59 | 60 | def non_matching_string(self, draw, state): 61 | return draw(st.characters(blacklist_characters=self.c)) 62 | 63 | def build_re(self): 64 | return self.c 65 | 66 | @staticmethod 67 | def make(draw): 68 | exp = draw(st.characters(blacklist_characters=special_characters)) 69 | return Char(exp) 70 | 71 | 72 | class CharClass(Re): 73 | """ Matches a (unicode) character category, positively or negatively """ 74 | 75 | def __init__(self, char, unicat, polarity_cat): 76 | self.char = char 77 | self.unicat = unicat 78 | self.polarity_cat = polarity_cat 79 | 80 | def matching_string(self, draw, state): 81 | if self.polarity_cat: 82 | return draw(st.characters(whitelist_categories=self.unicat)) 83 | return draw(st.characters(blacklist_categories=self.unicat)) 84 | 85 | def non_matching_string(self, draw, state): 86 | if self.polarity_cat: 87 | return draw(st.characters(blacklist_categories=self.unicat)) 88 | return draw(st.characters(whitelist_categories=self.unicat)) 89 | 90 | def build_re(self): 91 | return "\\" + self.char 92 | 93 | @staticmethod 94 | def make(draw): 95 | # XXX can add more 96 | return CharClass( 97 | *draw(st.sampled_from([("d", ["Nd"], True), ("D", ["Nd"], False)])) 98 | ) 99 | 100 | 101 | class Dot(Re): 102 | """ The regular expression '.', matches anything except a newline. """ 103 | 104 | def matching_string(self, draw, state): 105 | return draw(st.characters(blacklist_characters="\n")) 106 | 107 | def non_matching_string(self, draw, state): 108 | return "\n" 109 | 110 | def build_re(self): 111 | return "." 112 | 113 | @staticmethod 114 | def make(draw): 115 | return Dot() 116 | 117 | 118 | class Escape(Re): 119 | """ A regular expression escape. """ 120 | 121 | def __init__(self, c): 122 | self.c = c 123 | 124 | def matching_string(self, draw, state): 125 | return self.c 126 | 127 | def non_matching_string(self, draw, state): 128 | return draw(st.characters(blacklist_characters=self.c)) 129 | 130 | def build_re(self): 131 | return "\\" + self.c 132 | 133 | @staticmethod 134 | def make(draw): 135 | # XXX many more escapes 136 | c = draw(st.sampled_from(special_characters)) 137 | return Escape(c) 138 | 139 | 140 | class Charset(Re): 141 | """ A character set [...]. The elements are either single characters or 142 | ranges represented by (start, stop) tuples. """ 143 | 144 | def __init__(self, elements): 145 | # XXX character classes 146 | self.elements = elements 147 | 148 | def matching_string(self, draw, state): 149 | x = draw(st.sampled_from(self.elements)) 150 | if isinstance(x, tuple): 151 | return draw(st.characters(min_codepoint=ord(x[0]), max_codepoint=ord(x[1]))) 152 | return x 153 | 154 | def non_matching_string(self, draw, state): 155 | if not any(isinstance(x, tuple) for x in self.elements): 156 | # easy case, only chars 157 | return draw(st.characters(blacklist_characters=self.elements)) 158 | 159 | # Now, we *could* iterate through to get the set of all allowed characters, 160 | # but that would be pretty slow. Instead, we just get the highest and lowest 161 | # allowed codepoints and stay outside that interval, blacklisting individual 162 | # characters. If we allow both chr(0) and chr(maxunicode), just give up. 163 | chars = "".join(x for x in self.elements if not isinstance(x, tuple)) 164 | low = min(ord(x[0]) for x in self.elements if isinstance(x, tuple)) 165 | high = max(ord(x[1]) for x in self.elements if isinstance(x, tuple)) 166 | strat = st.nothing() 167 | if low > 0: 168 | strat |= st.characters(max_codepoint=low - 1, blacklist_characters=chars) 169 | if high < sys.maxunicode: 170 | strat |= st.characters(min_codepoint=high + 1, blacklist_characters=chars) 171 | try: 172 | return draw(strat) 173 | except InvalidArgument: 174 | reject() 175 | 176 | def build_re(self): 177 | res = [] 178 | for x in self.elements: 179 | if isinstance(x, tuple): 180 | res.append(f"{x[0]}-{x[1]}") 181 | else: 182 | res.append(x) 183 | return "[" + "".join(res) + "]" 184 | 185 | @staticmethod 186 | def make(draw): 187 | elements = draw(st.lists(Charset.charset_elements(), min_size=2, max_size=20)) 188 | return Charset(elements) 189 | 190 | @staticmethod 191 | @st.composite 192 | def charset_elements(draw): 193 | """ Generate an element of a character set element, either a single 194 | character or a character range, represented by a tuple. """ 195 | if draw(st.booleans()): 196 | # character 197 | return draw(st.characters(blacklist_characters="-^]\\")) 198 | else: 199 | start = draw(st.characters(blacklist_characters="-^]\\")) 200 | stop = draw( 201 | st.characters( 202 | blacklist_characters="-^]\\", min_codepoint=ord(start) + 1 203 | ) 204 | ) 205 | return start, stop 206 | 207 | 208 | class CharsetComplement(Re): 209 | """ An complemented character set [^...]""" 210 | 211 | def __init__(self, charset): 212 | assert isinstance(charset, Charset) 213 | self.charset = charset 214 | 215 | def matching_string(self, draw, state): 216 | return self.charset.non_matching_string(draw, state) 217 | 218 | def non_matching_string(self, draw, state): 219 | return self.charset.matching_string(draw, state) 220 | 221 | def build_re(self): 222 | charset = self.charset.build_re() 223 | assert charset.startswith("[") 224 | return f"[^{charset[1:-1]}]" 225 | 226 | @staticmethod 227 | def make(draw): 228 | return CharsetComplement(Charset.make(draw)) 229 | 230 | 231 | def re_simple(draw): 232 | """ Generate a "simple" regular expression, either '.', a single character, 233 | an escaped character a character category, a charset or its complement. """ 234 | cls = draw( 235 | st.sampled_from([Dot, Char, Escape, CharClass, Charset, CharsetComplement]) 236 | ) 237 | return cls.make(draw) 238 | 239 | 240 | class RecursiveRe(Re): 241 | """ Abstract base class for "recursive" Re nodes, ie nodes that build on 242 | top of other nodes. """ 243 | 244 | @staticmethod 245 | def make_with_base(base, draw): 246 | """ Factory function to construct a random instance of the current 247 | class. The nodes that this class builds upons are constructed using 248 | base, which has to be a function that takes draw, and returns an 249 | instance of (a subclass of) Re.""" 250 | raise NotImplementedError 251 | 252 | 253 | class Repetition(RecursiveRe): 254 | """ Abstract base class for "repetition"-like Re nodes. Can be either 255 | minimally matching (lazy) or maximally (greedy). """ 256 | 257 | # minimum number of repetitions of self.base 258 | # subclasses need to define that attribute, either on the class or instance 259 | istart = None 260 | # maximum number of repetitions of self.base 261 | # subclasses need to define that attribute, either on the class or instance 262 | istop = None 263 | 264 | def __init__(self, base, lazy): 265 | self.base = base 266 | self.lazy = lazy 267 | 268 | def can_be_empty(self): 269 | return self.base.can_be_empty() or self.istart == 0 270 | 271 | def build_re(self): 272 | return self._build_re() + "?" * self.lazy 273 | 274 | def _build_re(self): 275 | raise NotImplementedError 276 | 277 | def matching_string(self, draw, state): 278 | repetition = draw(st.integers(min_value=self.istart, max_value=self.istop)) 279 | res = [self.base.matching_string(draw, state) for i in range(repetition)] 280 | return "".join(res) 281 | 282 | def non_matching_string(self, draw, state): 283 | if self.can_be_empty() or self.base.can_be_empty(): 284 | raise CantGenerateNonMatching 285 | res = [self.base.matching_string(draw, state) for i in range(self.istart)] 286 | non_matching_pos = draw(st.integers(min_value=0, max_value=len(res) - 1)) 287 | res[non_matching_pos] = self.base.non_matching_string(draw, state) 288 | return "".join(res) 289 | 290 | @staticmethod 291 | def make_with_base(base, draw): 292 | b = base(draw) 293 | cls = draw( 294 | st.sampled_from( 295 | [Questionmark, Star, Plus, FixedNum, Start, Stop, StartStop] 296 | ) 297 | ) 298 | return cls.make_repetition(b, draw) 299 | 300 | 301 | class Questionmark(Repetition): 302 | istart = 0 303 | istop = 1 304 | 305 | def _build_re(self): 306 | return self.base.build_re() + "?" 307 | 308 | @staticmethod 309 | def make_repetition(base, draw): 310 | return Questionmark(base, draw(st.booleans())) 311 | 312 | 313 | class Star(Repetition): 314 | """ A Kleene-Star * regular expression, repeating the base expression 0 or 315 | more times. """ 316 | 317 | istart = 0 318 | istop = MAXREPEAT 319 | 320 | def _build_re(self): 321 | return self.base.build_re() + "*" 322 | 323 | @staticmethod 324 | def make_repetition(base, draw): 325 | return Star(base, draw(st.booleans())) 326 | 327 | 328 | class Plus(Repetition): 329 | """ A + regular expression repeating the base expression 1 or more 330 | times.""" 331 | 332 | istart = 1 333 | istop = MAXREPEAT 334 | 335 | def _build_re(self): 336 | return self.base.build_re() + "+" 337 | 338 | @staticmethod 339 | def make_repetition(base, draw): 340 | return Plus(base, draw(st.booleans())) 341 | 342 | 343 | class FixedNum(Repetition): 344 | """ Repeating the base expression a fixed number of times. """ 345 | 346 | def __init__(self, base, num, lazy): 347 | Repetition.__init__(self, base, lazy) 348 | self.istart = self.istop = num 349 | 350 | def _build_re(self): 351 | return f"{self.base.build_re()}{{{self.istart}}}" 352 | 353 | @staticmethod 354 | def make_repetition(base, draw): 355 | num = draw(st.integers(min_value=0, max_value=MAXREPEAT)) 356 | return FixedNum(base, num, draw(st.booleans())) 357 | 358 | 359 | class StartStop(Repetition): 360 | """ Repeating the base expression between istart and istop many times. """ 361 | 362 | def __init__(self, base, istart, istop, lazy): 363 | Repetition.__init__(self, base, lazy) 364 | self.istart = istart 365 | self.istop = istop 366 | 367 | def _build_re(self): 368 | return f"{self.base.build_re()}{{{self.istart},{self.istop}}}" 369 | 370 | @staticmethod 371 | def make_repetition(base, draw): 372 | start = draw(st.integers(min_value=0, max_value=MAXREPEAT)) 373 | stop = draw(st.integers(min_value=start, max_value=MAXREPEAT)) 374 | return StartStop(base, start, stop, draw(st.booleans())) 375 | 376 | 377 | class Stop(Repetition): 378 | """ Repeating the base expression between 0 and istop many times. """ 379 | 380 | istart = 0 381 | 382 | def __init__(self, base, istop, lazy): 383 | Repetition.__init__(self, base, lazy) 384 | self.istop = istop 385 | 386 | def _build_re(self): 387 | return f"{self.base.build_re()}{{,{self.istop}}}" 388 | 389 | @staticmethod 390 | def make_repetition(base, draw): 391 | stop = draw(st.integers(min_value=0, max_value=MAXREPEAT)) 392 | return Stop(base, stop, draw(st.booleans())) 393 | 394 | 395 | class Start(Repetition): 396 | """ Repeating the base expression at least istart many times. """ 397 | 398 | istop = MAXREPEAT 399 | 400 | def __init__(self, base, istart, lazy): 401 | Repetition.__init__(self, base, lazy) 402 | self.istart = istart 403 | 404 | def _build_re(self): 405 | return f"{self.base.build_re()}{{{self.istart},}}" 406 | 407 | @staticmethod 408 | def make_repetition(base, draw): 409 | start = draw(st.integers(min_value=0, max_value=MAXREPEAT)) 410 | return Start(base, start, draw(st.booleans())) 411 | 412 | 413 | class Sequence(RecursiveRe): 414 | """ A sequence of other regular expressions, which need to match one after 415 | the other. """ 416 | 417 | def __init__(self, bases): 418 | self.bases = bases 419 | 420 | def can_be_empty(self): 421 | return all(base.can_be_empty() for base in self.bases) 422 | 423 | def matching_string(self, draw, state): 424 | return "".join(b.matching_string(draw, state) for b in self.bases) 425 | 426 | def non_matching_string(self, draw, state): 427 | if self.can_be_empty(): 428 | raise CantGenerateNonMatching 429 | nonempty_positions = [ 430 | i for (i, b) in enumerate(self.bases) if not b.can_be_empty() 431 | ] 432 | res = [] 433 | for base_pos in nonempty_positions: 434 | res.append(self.bases[base_pos].non_matching_string(draw, state)) 435 | return "".join(res) 436 | 437 | def build_re(self): 438 | return "".join(b.build_re() for b in self.bases) 439 | 440 | @staticmethod 441 | def make_with_base(base, draw): 442 | return Sequence( 443 | [ 444 | base(draw) 445 | for i in range(draw(st.integers(min_value=2, max_value=MAXREPEAT))) 446 | ] 447 | ) 448 | 449 | 450 | class SequenceWithBackref(Sequence): 451 | """ Not really its own class, just a way to construct a sequence. Generate 452 | a random sequence and then turn one of the expressions into a named group, 453 | and add a reference to that group to the end of the sequence. """ 454 | 455 | @staticmethod 456 | def make_with_base(base, draw): 457 | sequence = Sequence.make_with_base(base, draw) 458 | bases = sequence.bases 459 | if IS_PYPY and sys.pypy_version_info < (7, 3, 1): 460 | # PyPy before 7.3.1 actually has broken group references! 461 | return Sequence(bases) 462 | # the following code would have found the bug in 463 | # https://foss.heptapod.net/pypy/pypy/commit/c83c263f9f00d18d48ef536947c9b61ca53e01a2 464 | group = draw(st.integers(min_value=0, max_value=len(bases) - 1)) 465 | # generate then caches a set across the run 466 | used_names = draw(st.shared(st.builds(set), key="group names")) 467 | # XXX a lot more characters are safe identifiers 468 | name = draw( 469 | st.text( 470 | min_size=1, 471 | alphabet=st.characters( 472 | whitelist_categories=["L", "Nl"], whitelist_characters="_" 473 | ), 474 | ).filter(lambda s: s not in used_names and s.isidentifier()) 475 | ) 476 | used_names.add(name) 477 | bases[group] = g = NamedGroup(bases[group], name) 478 | bases.append(GroupReference(g)) 479 | return sequence 480 | 481 | 482 | class NamedGroup(Re): 483 | """ Wrap the base expression into a named group with name. """ 484 | 485 | def __init__(self, base, name): 486 | self.base = base 487 | self.name = name 488 | 489 | def can_be_empty(self): 490 | return self.base.can_be_empty() 491 | 492 | def build_re(self): 493 | return f"(?P<{self.name}>{self.base.build_re()})" 494 | 495 | def matching_string(self, draw, state): 496 | res = self.base.matching_string(draw, state) 497 | state.groups[self] = res 498 | return res 499 | 500 | def non_matching_string(self, draw, state): 501 | res = self.base.non_matching_string(draw, state) 502 | state.groups[self] = res 503 | return res 504 | 505 | 506 | class GroupReference(Re): 507 | """ Backreference to a named group. """ 508 | 509 | def __init__(self, group): 510 | assert isinstance(group, NamedGroup) 511 | self.group = group 512 | 513 | def can_be_empty(self): 514 | return self.group.can_be_empty() 515 | 516 | def build_re(self): 517 | return f"(?P={self.group.name})" 518 | 519 | def matching_string(self, draw, state): 520 | return state.groups[self.group] 521 | 522 | def non_matching_string(self, draw, state): 523 | return state.groups[self.group] # doesn't matter, the group can't match 524 | 525 | 526 | class Disjunction(RecursiveRe): 527 | """ A disjunction of regular expressions, ie combining them with '|', where 528 | either of the base expressions has to match for the whole expression to 529 | match.""" 530 | 531 | def __init__(self, bases): 532 | self.bases = bases 533 | 534 | def can_be_empty(self): 535 | return any(base.can_be_empty() for base in self.bases) 536 | 537 | def matching_string(self, draw, state): 538 | base = draw(st.sampled_from(self.bases)) 539 | return base.matching_string(draw, state) 540 | 541 | def non_matching_string(self, draw, state): 542 | raise CantGenerateNonMatching 543 | 544 | def build_re(self): 545 | return "|".join(b.build_re() for b in self.bases) 546 | 547 | @staticmethod 548 | def make_with_base(base, draw): 549 | return Disjunction( 550 | [ 551 | base(draw) 552 | for i in range(draw(st.integers(min_value=2, max_value=MAXREPEAT))) 553 | ] 554 | ) 555 | 556 | 557 | # run some tests 558 | 559 | 560 | @contextmanager 561 | def assert_quick_not_quadratic(self): 562 | # Tests for timing can be brittle, but we think it's still worth checking 563 | # for pathologically slow (eg accidentally quadratic) performance issues. 564 | start = time.perf_counter() 565 | yield 566 | end = time.perf_counter() 567 | # On the machine this was developed, matching was always >= 500x faster 568 | self.assertLessEqual(end - start, 0.01) 569 | 570 | 571 | def _tag_test_for_distinct_hypothesis_database_key(maker): 572 | # Feel free to ignore this, it's a touch of magic with Hypothesis internals 573 | # which ensures that we have a separate database entry for each maker 574 | # despite reusing the test function. By design we don't actually *need* to 575 | # do this, but there's a small performance advantage if the tests fail 576 | # which could be useful if we run these on OSS-FUZZ one day. 577 | def inner(testfunc): 578 | key = getattr(maker, "func", maker).__qualname__.encode() 579 | testfunc._hypothesis_internal_add_digest = key 580 | return testfunc 581 | 582 | return inner 583 | 584 | 585 | def re_test(maker): 586 | """ Generate a test for the Re generating function maker. """ 587 | 588 | @given(data=st.data()) 589 | @_tag_test_for_distinct_hypothesis_database_key(maker) 590 | def test(self, data): 591 | draw = data.draw 592 | re_object = maker(draw) 593 | re_pattern = re_object.build_re() 594 | compiled = re.compile(re_pattern) 595 | 596 | # Sanity-check match on empty string is as we expect 597 | self.assertEqual(re_object.can_be_empty(), compiled.match("") is not None) 598 | 599 | # Check that a string expected to match does in fact match 600 | syes = re_object.matching_string(draw, State()) 601 | with assert_quick_not_quadratic(self): 602 | self.assertIsNotNone(compiled.match(syes)) 603 | 604 | # Check that, if we can generate a string that is not expected to match, 605 | # that string really doesn't match. 606 | try: 607 | sno = re_object.non_matching_string(draw, State()) 608 | except CantGenerateNonMatching: 609 | pass 610 | else: 611 | with assert_quick_not_quadratic(self): 612 | self.assertIsNone(compiled.match(sno)) 613 | 614 | return test 615 | 616 | 617 | class TestRe(unittest.TestCase): 618 | # Simple test cases 619 | test_char = re_test(Char.make) 620 | test_dots = re_test(Dot.make) 621 | test_escape = re_test(Escape.make) 622 | test_charclass = re_test(CharClass.make) 623 | test_charset = re_test(Charset.make) 624 | test_simple = re_test(re_simple) 625 | 626 | # Compound pattern types 627 | simple_repetition = partial(Repetition.make_with_base, re_simple) 628 | sequence_repetition = partial(Sequence.make_with_base, simple_repetition) 629 | backref_sequence = partial(SequenceWithBackref.make_with_base, simple_repetition) 630 | simple_disjunction = partial(Disjunction.make_with_base, re_simple) 631 | disjunction_sequence = partial(Disjunction.make_with_base, backref_sequence) 632 | 633 | # Tests for compound pattern types 634 | test_simple_repetition = re_test(simple_repetition) 635 | test_sequence_repetition = re_test(sequence_repetition) 636 | test_backref_sequence = re_test(backref_sequence) 637 | test_simple_disjunction = re_test(simple_disjunction) 638 | test_disjunction_sequence_repetition = re_test(disjunction_sequence) 639 | -------------------------------------------------------------------------------- /tests/test_source_code.py: -------------------------------------------------------------------------------- 1 | import ast 2 | import dis 3 | import io 4 | import lib2to3.pgen2.tokenize as lib2to3_tokenize 5 | import tokenize 6 | import unittest 7 | 8 | import hypothesmith 9 | from hypothesis import HealthCheck, example, given, reject, settings, strategies as st 10 | 11 | # Used to mark tests which generate arbitrary source code, 12 | # because that's a relatively expensive thing to get right. 13 | settings.register_profile( 14 | "slow", deadline=None, suppress_health_check=HealthCheck.all(), 15 | ) 16 | 17 | 18 | class TestAST(unittest.TestCase): 19 | @unittest.skipIf(not hasattr(ast, "unparse"), reason="new in 3.9") 20 | @given(source_code=hypothesmith.from_grammar()) 21 | @settings.get_profile("slow") 22 | def test_ast_unparse(self, source_code): 23 | """Unparsing always produces code which parses back to the same ast.""" 24 | first = ast.parse(source_code) 25 | unparsed = ast.unparse(first) 26 | second = ast.parse(unparsed) 27 | self.assertEqual(ast.dump(first), ast.dump(second)) 28 | 29 | 30 | class TestDis(unittest.TestCase): 31 | @given( 32 | code_object=hypothesmith.from_grammar().map( 33 | lambda s: compile(s, "", "exec") 34 | ), 35 | first_line=st.none() | st.integers(0, 100), 36 | current_offset=st.none() | st.integers(0, 100), 37 | ) 38 | @settings.get_profile("slow") 39 | def test_disassembly(self, code_object, first_line, current_offset): 40 | """Exercise the dis module.""" 41 | # TODO: what other properties of this module should we test? 42 | bcode = dis.Bytecode( 43 | code_object, first_line=first_line, current_offset=current_offset 44 | ) 45 | self.assertIs(code_object, bcode.codeobj) 46 | for inst in bcode: 47 | self.assertIsInstance(inst, dis.Instruction) 48 | 49 | 50 | def fixup(s): 51 | """Avoid known issues with tokenize() by editing the string.""" 52 | # Note that "passes after fixup" is a much weaker claim than "passes", 53 | # so ideally we would fix the tokenize bugs and then remove this. 54 | return "".join(x for x in s if x.isprintable()).strip().strip("\\").strip() + "\n" 55 | 56 | 57 | class TestLib2to3(unittest.TestCase): 58 | # TODO: https://docs.python.org/3/library/2to3.html 59 | 60 | @unittest.expectedFailure 61 | @example("#") 62 | @example("\n\\\n") 63 | @example("#\n\x0cpass#\n") 64 | @given(source_code=hypothesmith.from_grammar().map(fixup).filter(str.strip)) 65 | @settings.get_profile("slow") 66 | def test_lib2to3_tokenize_round_trip(self, source_code): 67 | tokens = [] 68 | 69 | def token_eater(*args): 70 | tokens.append(args) 71 | 72 | # Round-trip invariant for full input: 73 | # Untokenized source will match input source exactly 74 | lib2to3_tokenize.tokenize(io.StringIO(source_code).readline, token_eater) 75 | full = lib2to3_tokenize.untokenize(tokens) 76 | self.assertEqual(source_code, full) 77 | 78 | # Round-trip invariant for limited input: 79 | # Output text will tokenize the back to the input 80 | part_tokens = [t[:2] for t in tokens] 81 | partial = lib2to3_tokenize.untokenize(part_tokens) 82 | del tokens[:] 83 | lib2to3_tokenize.tokenize(io.StringIO(partial).readline, token_eater) 84 | self.assertEqual(part_tokens, [t[:2] for t in tokens]) 85 | 86 | 87 | class TestTokenize(unittest.TestCase): 88 | """Tests that the result of `untokenize` round-trips back to the same token stream, 89 | per https://docs.python.org/3/library/tokenize.html#tokenize.untokenize 90 | 91 | Unfortunately these tests demonstrate that it doesn't, and thus we have 92 | `@unittest.expectedFailure` decorators. 93 | """ 94 | 95 | @unittest.expectedFailure 96 | @example("#") 97 | @example("\n\\\n") 98 | @example("#\n\x0cpass#\n") 99 | @given(source_code=hypothesmith.from_grammar().map(fixup).filter(str.strip)) 100 | @settings.get_profile("slow") 101 | def test_tokenize_round_trip_bytes(self, source_code): 102 | try: 103 | source = source_code.encode("utf-8-sig") 104 | except UnicodeEncodeError: 105 | reject() 106 | tokens = list(tokenize.tokenize(io.BytesIO(source).readline)) 107 | # `outbytes` may have changed whitespace from `source` 108 | outbytes = tokenize.untokenize(tokens) 109 | output = list(tokenize.tokenize(io.BytesIO(outbytes).readline)) 110 | self.assertEqual( 111 | [(tt.type, tt.string) for tt in tokens], 112 | [(ot.type, ot.string) for ot in output], 113 | ) 114 | # It would be nice if the round-tripped string stabilised... 115 | # self.assertEqual(outbytes, tokenize.untokenize(output)) 116 | 117 | @unittest.expectedFailure 118 | @example("#") 119 | @example("\n\\\n") 120 | @example("#\n\x0cpass#\n") 121 | @given(source_code=hypothesmith.from_grammar().map(fixup).filter(str.strip)) 122 | @settings.get_profile("slow") 123 | def test_tokenize_round_trip_string(self, source_code): 124 | tokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline)) 125 | # `outstring` may have changed whitespace from `source_code` 126 | outstring = tokenize.untokenize(tokens) 127 | output = tokenize.generate_tokens(io.StringIO(outstring).readline) 128 | self.assertEqual( 129 | [(tt.type, tt.string) for tt in tokens], 130 | [(ot.type, ot.string) for ot in output], 131 | ) 132 | # It would be nice if the round-tripped string stabilised... 133 | # self.assertEqual(outstring, tokenize.untokenize(output)) 134 | -------------------------------------------------------------------------------- /tests/test_unicode.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from unicodedata import normalize 3 | 4 | from hypothesis import given, strategies as st 5 | 6 | # For every (n1, n2, n3) triple, applying n1 then n2 must be the same 7 | # as applying n3. 8 | # Reference: http://unicode.org/reports/tr15/#Design_Goals 9 | compositions = [ 10 | ("NFC", "NFC", "NFC"), 11 | ("NFC", "NFD", "NFD"), 12 | ("NFC", "NFKC", "NFKC"), 13 | ("NFC", "NFKD", "NFKD"), 14 | ("NFD", "NFC", "NFC"), 15 | ("NFD", "NFD", "NFD"), 16 | ("NFD", "NFKC", "NFKC"), 17 | ("NFD", "NFKD", "NFKD"), 18 | ("NFKC", "NFC", "NFKC"), 19 | ("NFKC", "NFD", "NFKD"), 20 | ("NFKC", "NFKC", "NFKC"), 21 | ("NFKC", "NFKD", "NFKD"), 22 | ("NFKD", "NFC", "NFKC"), 23 | ("NFKD", "NFD", "NFKD"), 24 | ("NFKD", "NFKC", "NFKC"), 25 | ("NFKD", "NFKD", "NFKD"), 26 | ] 27 | 28 | 29 | class TestUnicode(unittest.TestCase): 30 | @given(s=st.text(), comps=st.sampled_from(compositions)) 31 | def test_composition(self, s, comps): 32 | # see issues https://foss.heptapod.net/pypy/pypy/issues/2289 33 | # and https://bugs.python.org/issue26917 34 | norm1, norm2, norm3 = comps 35 | self.assertEqual(normalize(norm2, normalize(norm1, s)), normalize(norm3, s)) 36 | 37 | @given(u=st.text(), prefix=st.text(), suffix=st.text()) 38 | def test_find_index(self, u, prefix, suffix): 39 | s = prefix + u + suffix 40 | index = s.find(u) 41 | index2 = s.index(u) 42 | self.assertEqual(index, index2) 43 | 44 | # we check 0 <= index <= len(prefix) 45 | # whether u or a substring of it is in prefix or not, the highest index 46 | # we can find u at is len(prefix) 47 | self.assertLessEqual(0, index) 48 | self.assertLessEqual(index, len(prefix)) 49 | 50 | index = s.find(u, len(prefix), len(s) - len(suffix)) 51 | index2 = s.index(u, len(prefix), len(s) - len(suffix)) 52 | self.assertEqual(index, len(prefix)) 53 | self.assertEqual(index2, len(prefix)) 54 | 55 | @given(u=st.text(), prefix=st.text(), suffix=st.text()) 56 | def test_rfind(self, u, prefix, suffix): 57 | s = prefix + u + suffix 58 | index = s.rfind(u) 59 | index2 = s.rindex(u) 60 | self.assertEqual(index, index2) 61 | 62 | # we check len(prefix) <= index <= len(s) 63 | # as above: whether u or a substring of it is in suffix or not, the 64 | # lowest index we can find u at is len(prefix) 65 | self.assertLessEqual(len(prefix), index) 66 | self.assertLessEqual(index, len(s)) 67 | 68 | index = s.rfind(u, len(prefix), len(s) - len(suffix)) 69 | index2 = s.rindex(u, len(prefix), len(s) - len(suffix)) 70 | self.assertEqual(index, index2) 71 | self.assertEqual(index, len(prefix)) 72 | -------------------------------------------------------------------------------- /tests/test_zoneinfo.py: -------------------------------------------------------------------------------- 1 | import contextlib 2 | import datetime 3 | import os 4 | import pickle 5 | import unittest 6 | 7 | import hypothesis 8 | 9 | try: 10 | import zoneinfo 11 | except ImportError: 12 | # TODO: Switch this to a version check when a stable 3.9 is released 13 | raise unittest.SkipTest("zoneinfo module added in 3.9.0b2") 14 | 15 | import test.test_zoneinfo._support as test_support # isort:skip 16 | 17 | ZoneInfoTestBase = test_support.ZoneInfoTestBase 18 | 19 | py_zoneinfo, c_zoneinfo = test_support.get_modules() 20 | 21 | UTC = datetime.timezone.utc 22 | MIN_UTC = datetime.datetime.min.replace(tzinfo=UTC) 23 | MAX_UTC = datetime.datetime.max.replace(tzinfo=UTC) 24 | ZERO = datetime.timedelta(0) 25 | 26 | 27 | def _valid_keys(): 28 | """Get available time zones, including posix/ and right/ directories.""" 29 | from importlib import resources 30 | 31 | available_zones = sorted(zoneinfo.available_timezones()) 32 | TZPATH = zoneinfo.TZPATH # noqa: N806 33 | 34 | def valid_key(key): 35 | for root in TZPATH: 36 | key_file = os.path.join(root, key) 37 | if os.path.exists(key_file): 38 | return True 39 | 40 | components = key.split("/") 41 | package_name = ".".join(["tzdata.zoneinfo"] + components[:-1]) 42 | resource_name = components[-1] 43 | 44 | try: 45 | return resources.is_resource(package_name, resource_name) 46 | except ModuleNotFoundError: 47 | return False 48 | 49 | # This relies on the fact that dictionaries maintain insertion order — for 50 | # shrinking purposes, it is preferable to start with the standard version, 51 | # then move to the posix/ version, then to the right/ version. 52 | out_zones = {"": available_zones} 53 | for prefix in ["posix", "right"]: 54 | prefix_out = [] 55 | for key in available_zones: 56 | prefix_key = f"{prefix}/{key}" 57 | if valid_key(prefix_key): 58 | prefix_out.append(prefix_key) 59 | 60 | out_zones[prefix] = prefix_out 61 | 62 | output = [] 63 | for keys in out_zones.values(): 64 | output.extend(keys) 65 | 66 | return output 67 | 68 | 69 | VALID_KEYS = _valid_keys() 70 | if not VALID_KEYS: 71 | raise unittest.SkipTest("No time zone data available") 72 | 73 | 74 | def valid_keys(): 75 | return hypothesis.strategies.sampled_from(VALID_KEYS) 76 | 77 | 78 | class ZoneInfoTest(ZoneInfoTestBase): 79 | module = py_zoneinfo 80 | 81 | @hypothesis.given(key=valid_keys()) 82 | def test_str(self, key): 83 | zi = self.klass(key) 84 | self.assertEqual(str(zi), key) 85 | 86 | @hypothesis.given(key=valid_keys()) 87 | def test_key(self, key): 88 | zi = self.klass(key) 89 | 90 | self.assertEqual(zi.key, key) 91 | 92 | @hypothesis.given( 93 | dt=hypothesis.strategies.one_of( 94 | hypothesis.strategies.datetimes(), hypothesis.strategies.times() 95 | ) 96 | ) 97 | def test_utc(self, dt): 98 | zi = self.klass("UTC") 99 | dt_zi = dt.replace(tzinfo=zi) 100 | 101 | self.assertEqual(dt_zi.utcoffset(), ZERO) 102 | self.assertEqual(dt_zi.dst(), ZERO) 103 | self.assertEqual(dt_zi.tzname(), "UTC") 104 | 105 | 106 | class CZoneInfoTest(ZoneInfoTest): 107 | module = c_zoneinfo 108 | 109 | 110 | class ZoneInfoPickleTest(ZoneInfoTestBase): 111 | module = py_zoneinfo 112 | 113 | def setUp(self): 114 | with contextlib.ExitStack() as stack: 115 | stack.enter_context(test_support.set_zoneinfo_module(self.module)) 116 | self.addCleanup(stack.pop_all().close) 117 | 118 | super().setUp() 119 | 120 | @hypothesis.given(key=valid_keys()) 121 | def test_pickle_unpickle_cache(self, key): 122 | zi = self.klass(key) 123 | pkl_str = pickle.dumps(zi) 124 | zi_rt = pickle.loads(pkl_str) 125 | 126 | self.assertIs(zi, zi_rt) 127 | 128 | @hypothesis.given(key=valid_keys()) 129 | def test_pickle_unpickle_no_cache(self, key): 130 | zi = self.klass.no_cache(key) 131 | pkl_str = pickle.dumps(zi) 132 | zi_rt = pickle.loads(pkl_str) 133 | 134 | self.assertIsNot(zi, zi_rt) 135 | self.assertEqual(str(zi), str(zi_rt)) 136 | 137 | @hypothesis.given(key=valid_keys()) 138 | def test_pickle_unpickle_cache_multiple_rounds(self, key): 139 | """Test that pickle/unpickle is idempotent.""" 140 | zi_0 = self.klass(key) 141 | pkl_str_0 = pickle.dumps(zi_0) 142 | zi_1 = pickle.loads(pkl_str_0) 143 | pkl_str_1 = pickle.dumps(zi_1) 144 | zi_2 = pickle.loads(pkl_str_1) 145 | pkl_str_2 = pickle.dumps(zi_2) 146 | 147 | self.assertEqual(pkl_str_0, pkl_str_1) 148 | self.assertEqual(pkl_str_1, pkl_str_2) 149 | 150 | self.assertIs(zi_0, zi_1) 151 | self.assertIs(zi_0, zi_2) 152 | self.assertIs(zi_1, zi_2) 153 | 154 | @hypothesis.given(key=valid_keys()) 155 | def test_pickle_unpickle_no_cache_multiple_rounds(self, key): 156 | """Test that pickle/unpickle is idempotent.""" 157 | zi_cache = self.klass(key) 158 | 159 | zi_0 = self.klass.no_cache(key) 160 | pkl_str_0 = pickle.dumps(zi_0) 161 | zi_1 = pickle.loads(pkl_str_0) 162 | pkl_str_1 = pickle.dumps(zi_1) 163 | zi_2 = pickle.loads(pkl_str_1) 164 | pkl_str_2 = pickle.dumps(zi_2) 165 | 166 | self.assertEqual(pkl_str_0, pkl_str_1) 167 | self.assertEqual(pkl_str_1, pkl_str_2) 168 | 169 | self.assertIsNot(zi_0, zi_1) 170 | self.assertIsNot(zi_0, zi_2) 171 | self.assertIsNot(zi_1, zi_2) 172 | 173 | self.assertIsNot(zi_0, zi_cache) 174 | self.assertIsNot(zi_1, zi_cache) 175 | self.assertIsNot(zi_2, zi_cache) 176 | 177 | 178 | class CZoneInfoPickleTest(ZoneInfoPickleTest): 179 | module = c_zoneinfo 180 | 181 | 182 | class ZoneInfoCacheTest(ZoneInfoTestBase): 183 | module = py_zoneinfo 184 | 185 | @hypothesis.given(key=valid_keys()) 186 | def test_cache(self, key): 187 | zi_0 = self.klass(key) 188 | zi_1 = self.klass(key) 189 | 190 | self.assertIs(zi_0, zi_1) 191 | 192 | @hypothesis.given(key=valid_keys()) 193 | def test_no_cache(self, key): 194 | zi_0 = self.klass.no_cache(key) 195 | zi_1 = self.klass.no_cache(key) 196 | 197 | self.assertIsNot(zi_0, zi_1) 198 | 199 | 200 | class CZoneInfoCacheTest(ZoneInfoCacheTest): 201 | klass = c_zoneinfo.ZoneInfo 202 | 203 | 204 | class PythonCConsistencyTest(unittest.TestCase): 205 | """Tests that the C and Python versions do the same thing.""" 206 | 207 | def _is_ambiguous(self, dt): 208 | return dt.replace(fold=not dt.fold).utcoffset() == dt.utcoffset() 209 | 210 | @hypothesis.given(dt=hypothesis.strategies.datetimes(), key=valid_keys()) 211 | def test_same_str(self, dt, key): 212 | py_dt = dt.replace(tzinfo=py_zoneinfo.ZoneInfo(key)) 213 | c_dt = dt.replace(tzinfo=c_zoneinfo.ZoneInfo(key)) 214 | 215 | self.assertEqual(str(py_dt), str(c_dt)) 216 | 217 | @hypothesis.given(dt=hypothesis.strategies.datetimes(), key=valid_keys()) 218 | def test_same_offsets_and_names(self, dt, key): 219 | py_dt = dt.replace(tzinfo=py_zoneinfo.ZoneInfo(key)) 220 | c_dt = dt.replace(tzinfo=c_zoneinfo.ZoneInfo(key)) 221 | 222 | self.assertEqual(py_dt.tzname(), c_dt.tzname()) 223 | self.assertEqual(py_dt.utcoffset(), c_dt.utcoffset()) 224 | self.assertEqual(py_dt.dst(), c_dt.dst()) 225 | 226 | @hypothesis.given( 227 | dt=hypothesis.strategies.datetimes(timezones=hypothesis.strategies.just(UTC)), 228 | key=valid_keys(), 229 | ) 230 | @hypothesis.example(dt=MIN_UTC, key="Asia/Tokyo") 231 | @hypothesis.example(dt=MAX_UTC, key="Asia/Tokyo") 232 | @hypothesis.example(dt=MIN_UTC, key="America/New_York") 233 | @hypothesis.example(dt=MAX_UTC, key="America/New_York") 234 | @hypothesis.example( 235 | dt=datetime.datetime(2006, 10, 29, 5, 15, tzinfo=UTC), key="America/New_York", 236 | ) 237 | def test_same_from_utc(self, dt, key): 238 | py_zi = py_zoneinfo.ZoneInfo(key) 239 | c_zi = c_zoneinfo.ZoneInfo(key) 240 | 241 | # Convert to UTC: This can overflow, but we just care about consistency 242 | py_overflow_exc = None 243 | c_overflow_exc = None 244 | try: 245 | py_dt = dt.astimezone(py_zi) 246 | except OverflowError as e: 247 | py_overflow_exc = e 248 | 249 | try: 250 | c_dt = dt.astimezone(c_zi) 251 | except OverflowError as e: 252 | c_overflow_exc = e 253 | 254 | if (py_overflow_exc is not None) != (c_overflow_exc is not None): 255 | raise py_overflow_exc or c_overflow_exc # pragma: nocover 256 | 257 | if py_overflow_exc is not None: 258 | return # Consistently raises the same exception 259 | 260 | # PEP 495 says that an inter-zone comparison between ambiguous 261 | # datetimes is always False. 262 | if py_dt != c_dt: 263 | self.assertEqual( 264 | self._is_ambiguous(py_dt), self._is_ambiguous(c_dt), (py_dt, c_dt), 265 | ) 266 | 267 | self.assertEqual(py_dt.tzname(), c_dt.tzname()) 268 | self.assertEqual(py_dt.utcoffset(), c_dt.utcoffset()) 269 | self.assertEqual(py_dt.dst(), c_dt.dst()) 270 | 271 | @hypothesis.given(dt=hypothesis.strategies.datetimes(), key=valid_keys()) 272 | @hypothesis.example(dt=datetime.datetime.max, key="America/New_York") 273 | @hypothesis.example(dt=datetime.datetime.min, key="America/New_York") 274 | @hypothesis.example(dt=datetime.datetime.min, key="Asia/Tokyo") 275 | @hypothesis.example(dt=datetime.datetime.max, key="Asia/Tokyo") 276 | def test_same_to_utc(self, dt, key): 277 | py_dt = dt.replace(tzinfo=py_zoneinfo.ZoneInfo(key)) 278 | c_dt = dt.replace(tzinfo=c_zoneinfo.ZoneInfo(key)) 279 | 280 | # Convert from UTC: Overflow OK if it happens in both implementations 281 | py_overflow_exc = None 282 | c_overflow_exc = None 283 | try: 284 | py_utc = py_dt.astimezone(UTC) 285 | except OverflowError as e: 286 | py_overflow_exc = e 287 | 288 | try: 289 | c_utc = c_dt.astimezone(UTC) 290 | except OverflowError as e: 291 | c_overflow_exc = e 292 | 293 | if (py_overflow_exc is not None) != (c_overflow_exc is not None): 294 | raise py_overflow_exc or c_overflow_exc # pragma: nocover 295 | 296 | if py_overflow_exc is not None: 297 | return # Consistently raises the same exception 298 | 299 | self.assertEqual(py_utc, c_utc) 300 | 301 | @hypothesis.given(key=valid_keys()) 302 | def test_cross_module_pickle(self, key): 303 | py_zi = py_zoneinfo.ZoneInfo(key) 304 | c_zi = c_zoneinfo.ZoneInfo(key) 305 | 306 | with test_support.set_zoneinfo_module(py_zoneinfo): 307 | py_pkl = pickle.dumps(py_zi) 308 | 309 | with test_support.set_zoneinfo_module(c_zoneinfo): 310 | c_pkl = pickle.dumps(c_zi) 311 | 312 | with test_support.set_zoneinfo_module(c_zoneinfo): 313 | # Python → C 314 | py_to_c_zi = pickle.loads(py_pkl) 315 | self.assertIs(py_to_c_zi, c_zi) 316 | 317 | with test_support.set_zoneinfo_module(py_zoneinfo): 318 | # C → Python 319 | c_to_py_zi = pickle.loads(c_pkl) 320 | self.assertIs(c_to_py_zi, py_zi) 321 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # The test environment and commands 2 | [tox] 3 | envlist = check, test 4 | skipsdist = True 5 | 6 | [testenv:check] 7 | description = Runs all formatting tools then static analysis (quick) 8 | deps = 9 | --no-deps 10 | -r requirements-dev.txt 11 | commands = 12 | shed # combines autoflake, black, isort, and pyupgrade 13 | flake8 14 | 15 | [testenv:test] 16 | description = Run the tests 17 | deps = 18 | --no-deps 19 | -r requirements.txt 20 | commands = 21 | python -m unittest discover tests --catch {posargs: -v} 22 | 23 | [testenv:deps] 24 | description = Update pinned requirements 25 | deps = 26 | pip-tools 27 | setenv = 28 | CUSTOM_COMPILE_COMMAND = tox -e deps 29 | commands = 30 | pip-compile --quiet --upgrade --rebuild --output-file=requirements.txt requirements.in 31 | pip-compile --quiet --upgrade --rebuild --output-file=requirements-dev.txt requirements-dev.in 32 | 33 | 34 | # Settings for other tools 35 | [flake8] 36 | ignore = E501,W503,S101,S310,N802,S301,S403 37 | exclude = .*/,__pycache__ 38 | 39 | [isort] 40 | default_section = THIRDPARTY 41 | multi_line_output = 3 42 | include_trailing_comma = True 43 | force_grid_wrap = 0 44 | combine_as_imports = True 45 | line_length = 88 46 | --------------------------------------------------------------------------------