├── .gitignore ├── .travis.yml ├── CHANGELOG.md ├── LICENSE ├── MANIFEST.in ├── README.md ├── Screenshot.png ├── example ├── schema.json └── tests.json ├── jsonschema_test ├── __init__.py ├── schema-tests.json └── schema.json └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | dist 3 | *.egg-info/ 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.7" 4 | - "3.4" 5 | install: 6 | - pip install . 7 | script: 8 | - jsonschema-test jsonschema_test/schema.json jsonschema_test/schema-tests.json 9 | - jsonschema-test example/schema.json example/tests.json 10 | 11 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog for JSON Schema Test 2 | 3 | ## 1.1.0 4 | 5 | ### Enhancements 6 | 7 | - Added the ability to skip a test. 8 | 9 | ## 1.0.0 10 | 11 | Initial release. 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Kyle Fuller 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | 24 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include jsonschema_test/schema.json 2 | 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # JSON Schema Testing 2 | 3 | [![Build Status](http://img.shields.io/travis/kylef/jsonschema-test/master.svg?style=flat)](https://travis-ci.org/kylef/jsonschema-test) 4 | 5 | jsonschema-test is a tool for writing and running tests against a given 6 | JSON Schema. You can write a set of test-cases including example JSON 7 | values and if they should validate or not against the given JSON Schema. 8 | 9 | ![](Screenshot.png) 10 | 11 | ## Installation 12 | 13 | ``` 14 | pip install jsonschema-test 15 | ``` 16 | 17 | ## Usage 18 | 19 | To use jsonschema-test, you will need to write a collection of tests. 20 | The tests are decoupled from any language and are written in JSON. 21 | 22 | ### Example 23 | 24 | #### Test Suite 25 | 26 | A test suite is a collection of test's and test cases. See 27 | the [test suite structure](#test-suite-structure) for more information 28 | 29 | ```json 30 | [ 31 | { 32 | "description": "a person", 33 | "tests": [ 34 | { 35 | "description": "validates with a name", 36 | "data": { 37 | "name": "Kyle" 38 | }, 39 | "valid": true 40 | }, 41 | { 42 | "description": "fails validation without a name", 43 | "data": { 44 | }, 45 | "valid": false 46 | } 47 | ] 48 | } 49 | ] 50 | ``` 51 | 52 | #### Schema 53 | 54 | A JSON Schema to test. 55 | 56 | ```json 57 | { 58 | "type": "object", 59 | "properties": { 60 | "name": { 61 | "type": "string" 62 | } 63 | }, 64 | "required": [ "name" ] 65 | } 66 | ``` 67 | 68 | #### Running the tests 69 | 70 | ``` 71 | jsonschema-test example/schema.json example/tests.json 72 | ``` 73 | 74 | ## Test Suite Structure 75 | 76 | There is a [JSON Schema](jsonschema_test/schema.json) file for the test suite structure. 77 | 78 | ### Suite 79 | 80 | A test suite is a logical group of Test's. It's a simple JSON file with an 81 | array containing each test. 82 | 83 | ### Test 84 | 85 | #### Properties 86 | 87 | - `description` (string, required) - A short description of what this test tests. 88 | - `tests` (array[Case], required) - A collection of test cases. 89 | - `skip` (boolean, optional) - Whether to skip the test. This defaults to false. 90 | 91 | ### Case 92 | 93 | #### Properties 94 | 95 | - `description` (string, required) - The description of this test case. 96 | - `data` (any, required) - The JSON used against the JSON Schema. 97 | - `valid` (boolean, required) - If the `data` should validate or not. 98 | - `skip` (boolean, optional) - Whether to skip the case. This defaults to false. 99 | 100 | ## Credits 101 | 102 | This tool was built by [Kyle Fuller](http://fuller.li/) ([@kylefuller](https://twitter.com/kylefuller)). 103 | 104 | ## License 105 | 106 | jsonschema-test is released under the BSD license. See [LICENSE](LICENSE). 107 | 108 | -------------------------------------------------------------------------------- /Screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kylef-archive/jsonschema-test/1f19dbee75950f95ce379d7cd9b9da8396fc6562/Screenshot.png -------------------------------------------------------------------------------- /example/schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "object", 3 | "properties": { 4 | "name": { 5 | "type": "string" 6 | } 7 | }, 8 | "required": [ "name" ] 9 | } 10 | 11 | -------------------------------------------------------------------------------- /example/tests.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "description": "a person", 4 | "tests": [ 5 | { 6 | "description": "validates with a name", 7 | "data": { 8 | "name": "Kyle" 9 | }, 10 | "valid": true 11 | }, 12 | { 13 | "description": "fails validation without a name", 14 | "data": { 15 | }, 16 | "valid": false 17 | }, 18 | { 19 | "description": "fails validation when the name is not a string", 20 | "data": { 21 | "name": null 22 | }, 23 | "valid": false 24 | } 25 | ] 26 | } 27 | ] 28 | 29 | -------------------------------------------------------------------------------- /jsonschema_test/__init__.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import json 4 | from jsonschema import validate 5 | 6 | 7 | def load_json(path): 8 | with open(path) as fp: 9 | return json.load(fp) 10 | 11 | 12 | def load_json_suite(path): 13 | schema = test_schema() 14 | suite = load_json(path) 15 | 16 | try: 17 | validate(suite, schema) 18 | except Exception as e: 19 | print('{} is not a valid test file.'.format(path)) 20 | print(e) 21 | exit(2) 22 | 23 | return suite 24 | 25 | 26 | def print_ansi(code, text): 27 | print('\033[{}m{}\x1b[0m'.format(code, text)) 28 | 29 | 30 | print_bold = lambda text: print_ansi(1, text) 31 | print_green = lambda text: print_ansi(32, text) 32 | print_red = lambda text: print_ansi(31, text) 33 | print_yellow = lambda text: print_ansi(33, text) 34 | 35 | 36 | def test_schema(): 37 | path = os.path.join(os.path.dirname(__file__), 'schema.json') 38 | return load_json(path) 39 | 40 | 41 | def test(schema_path, suite_paths): 42 | schema = load_json(schema_path) 43 | suites = map(load_json_suite, suite_paths) 44 | 45 | passes = 0 46 | failures = 0 47 | skipped = 0 48 | 49 | for suite in suites: 50 | for case in suite: 51 | if case.get('skip', False): 52 | print_yellow('-> {} (skipped)'.format(case['description'])) 53 | 54 | for test in case['tests']: 55 | print_yellow(' -> {} (skipped)'.format(test['description'])) 56 | skipped += 1 57 | 58 | print('') 59 | continue 60 | 61 | print_bold('-> {}'.format(case['description'])) 62 | 63 | for test in case['tests']: 64 | if test.get('skip', False): 65 | skipped += 1 66 | print_yellow(' -> {} (skipped)'.format(test['description'])) 67 | continue 68 | 69 | success = True 70 | 71 | try: 72 | validate(test['data'], schema) 73 | except Exception as e: 74 | if test['valid']: 75 | success = False 76 | else: 77 | if not test['valid']: 78 | success = False 79 | 80 | if success: 81 | passes += 1 82 | print_green(' -> {}'.format(test['description'])) 83 | else: 84 | failures += 1 85 | print_red(' -> {}'.format(test['description'])) 86 | print(' Expected data to validate as: {}'.format(test['valid'])) 87 | print(' ' + json.dumps(test['data'])) 88 | print('') 89 | 90 | print('') 91 | 92 | if skipped > 0: 93 | print('{} passes, {} skipped, {} failures'.format(passes, skipped, failures)) 94 | else: 95 | print('{} passes, {} failures'.format(passes, failures)) 96 | 97 | if failures: 98 | exit(1) 99 | 100 | 101 | def usage(): 102 | print('Usage: {} [test suites, ...]'.format(sys.argv[0])) 103 | 104 | 105 | def validate_json(filename): 106 | if not os.path.isfile(filename): 107 | print('{} does not exist.'.format(filename)) 108 | exit(2) 109 | 110 | with open(filename) as fp: 111 | try: 112 | json.load(fp) 113 | except Exception as e: 114 | print('{} does not contain valid JSON.'.format(filename)) 115 | exit(2) 116 | 117 | 118 | def main(): 119 | if len(sys.argv) > 1: 120 | for filename in sys.argv[1:]: 121 | validate_json(filename) 122 | 123 | schema = sys.argv[1] 124 | suites = sys.argv[2:] 125 | 126 | if len(suites) > 0: 127 | test(schema, suites) 128 | else: 129 | usage() 130 | 131 | 132 | if __name__ == '__main__': 133 | main() 134 | 135 | -------------------------------------------------------------------------------- /jsonschema_test/schema-tests.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "description": "suite data type", 4 | "tests": [ 5 | { 6 | "description": "validates with an array type", 7 | "data": [], 8 | "valid": true 9 | }, 10 | { 11 | "description": "fails to validate with a null type", 12 | "data": null, 13 | "valid": false 14 | }, 15 | { 16 | "description": "fails to validate with a boolean type", 17 | "data": true, 18 | "valid": false 19 | }, 20 | { 21 | "description": "fails to validate with a number type", 22 | "data": 1, 23 | "valid": false 24 | }, 25 | { 26 | "description": "fails to validate with a string type", 27 | "data": "", 28 | "valid": false 29 | }, 30 | { 31 | "description": "fails to validate with an object type", 32 | "data": {}, 33 | "valid": false 34 | } 35 | ] 36 | }, 37 | { 38 | "description": "test data type", 39 | "tests": [ 40 | { 41 | "description": "validates with an object type", 42 | "data": [ 43 | { 44 | "description": "", 45 | "tests": [] 46 | } 47 | ], 48 | "valid": true 49 | }, 50 | { 51 | "description": "fails to validate with a null type", 52 | "data": [ 53 | { 54 | "description": "", 55 | "tests": [ null ] 56 | } 57 | ], 58 | "valid": false 59 | }, 60 | { 61 | "description": "fails to validate with a boolean type", 62 | "data": [ 63 | { 64 | "description": "", 65 | "tests": [ true ] 66 | } 67 | ], 68 | "valid": false 69 | }, 70 | { 71 | "description": "fails to validate with a number type", 72 | "data": [ 73 | { 74 | "description": "", 75 | "tests": [ 1 ] 76 | } 77 | ], 78 | "valid": false 79 | }, 80 | { 81 | "description": "fails to validate with a string type", 82 | "data": [ 83 | { 84 | "description": "", 85 | "tests": [ "" ] 86 | } 87 | ], 88 | "valid": false 89 | }, 90 | { 91 | "description": "fails to validate with an array type", 92 | "data": [ 93 | { 94 | "description": "", 95 | "tests": [ [] ] 96 | } 97 | ], 98 | "valid": false 99 | } 100 | ] 101 | }, 102 | { 103 | "description": "case data type", 104 | "tests": [ 105 | { 106 | "description": "validates with an object type", 107 | "data": [ 108 | { 109 | "description": "", 110 | "tests": [ 111 | { 112 | "description": "", 113 | "data": [], 114 | "valid": true 115 | } 116 | ] 117 | } 118 | ], 119 | "valid": true 120 | }, 121 | { 122 | "description": "fails to validate with a null type", 123 | "data": [ 124 | { 125 | "description": "", 126 | "tests": [ null ] 127 | } 128 | ], 129 | "valid": false 130 | }, 131 | { 132 | "description": "fails to validate with a boolean type", 133 | "data": [ 134 | { 135 | "description": "", 136 | "tests": [ true ] 137 | } 138 | ], 139 | "valid": false 140 | }, 141 | { 142 | "description": "fails to validate with a number type", 143 | "data": [ 144 | { 145 | "description": "", 146 | "tests": [ 1 ] 147 | } 148 | ], 149 | "valid": false 150 | }, 151 | { 152 | "description": "fails to validate with a string type", 153 | "data": [ 154 | { 155 | "description": "", 156 | "tests": [ "" ] 157 | } 158 | ], 159 | "valid": false 160 | }, 161 | { 162 | "description": "fails to validate with an array type", 163 | "data": [ 164 | { 165 | "description": "", 166 | "tests": [ [] ] 167 | } 168 | ], 169 | "valid": false 170 | } 171 | ] 172 | }, 173 | { 174 | "description": "test object keys", 175 | "tests": [ 176 | { 177 | "description": "fails to validate without a description", 178 | "data": [ 179 | { 180 | "tests": [] 181 | } 182 | ], 183 | "valid": false 184 | }, 185 | { 186 | "description": "fails to validate with a non-string description", 187 | "data": [ 188 | { 189 | "description": null, 190 | "tests": [] 191 | } 192 | ], 193 | "valid": false 194 | }, 195 | { 196 | "description": "fails to validate without any tests", 197 | "data": [ 198 | { 199 | "description": "" 200 | } 201 | ], 202 | "valid": false 203 | }, 204 | { 205 | "description": "fails to validate with a non-array tests", 206 | "data": [ 207 | { 208 | "description": "", 209 | "tests": null 210 | } 211 | ], 212 | "valid": false 213 | } 214 | ] 215 | }, 216 | { 217 | "description": "case object keys", 218 | "tests": [ 219 | { 220 | "description": "with a valid case", 221 | "data": [ 222 | { 223 | "description": "", 224 | "tests": [ 225 | { 226 | "description": "", 227 | "data": null, 228 | "valid": true 229 | } 230 | ] 231 | } 232 | ], 233 | "valid": true 234 | }, 235 | { 236 | "description": "fails to validate without a description", 237 | "data": [ 238 | { 239 | "description": "", 240 | "tests": [ 241 | { 242 | "data": null, 243 | "valid": true 244 | } 245 | ] 246 | } 247 | ], 248 | "valid": false 249 | }, 250 | { 251 | "description": "fails to validate with a non-string description", 252 | "data": [ 253 | { 254 | "description": "", 255 | "tests": [ 256 | { 257 | "description": null, 258 | "data": null, 259 | "valid": true 260 | } 261 | ] 262 | } 263 | ], 264 | "valid": false 265 | }, 266 | { 267 | "description": "fails to without data", 268 | "data": [ 269 | { 270 | "description": "", 271 | "tests": [ 272 | { 273 | "description": "", 274 | "valid": true 275 | } 276 | ] 277 | } 278 | ], 279 | "valid": false 280 | }, 281 | { 282 | "description": "fails to without valid", 283 | "data": [ 284 | { 285 | "description": "", 286 | "tests": [ 287 | { 288 | "description": "", 289 | "data": null 290 | } 291 | ] 292 | } 293 | ], 294 | "valid": false 295 | }, 296 | { 297 | "description": "fails without a boolean valid", 298 | "data": [ 299 | { 300 | "description": "", 301 | "tests": [ 302 | { 303 | "description": "", 304 | "data": null, 305 | "valid": null 306 | } 307 | ] 308 | } 309 | ], 310 | "valid": false 311 | } 312 | ] 313 | } 314 | ] 315 | -------------------------------------------------------------------------------- /jsonschema_test/schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "array", 3 | "items": { 4 | "$ref": "#/definitions/test" 5 | }, 6 | "definitions": { 7 | "test": { 8 | "type": "object", 9 | "properties": { 10 | "description": { 11 | "type": "string" 12 | }, 13 | "tests": { 14 | "type": "array", 15 | "items": { 16 | "$ref": "#/definitions/case" 17 | } 18 | }, 19 | "skip": { 20 | "type": "boolean" 21 | } 22 | }, 23 | "required": ["description", "tests"] 24 | }, 25 | "case": { 26 | "type": "object", 27 | "properties": { 28 | "description": { 29 | "type": "string" 30 | }, 31 | "data": { 32 | }, 33 | "valid": { 34 | "type": "boolean" 35 | }, 36 | "skip": { 37 | "type": "boolean" 38 | } 39 | }, 40 | "required": ["description", "data", "valid"] 41 | } 42 | } 43 | } 44 | 45 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | setup( 6 | name='jsonschema-test', 7 | version='1.1.0', 8 | description='A tool for writing and running tests against a given JSON Schema.', 9 | url='https://github.com/kylef/jsonschema-test', 10 | packages=['jsonschema_test'], 11 | package_data={ 12 | 'jsonschema_test': ['schema.json'], 13 | }, 14 | entry_points={ 15 | 'console_scripts': [ 16 | 'jsonschema-test = jsonschema_test:main', 17 | ] 18 | }, 19 | install_requires=[ 20 | 'jsonschema', 21 | ], 22 | author='Kyle Fuller', 23 | author_email='kyle@fuller.li', 24 | license='BSD', 25 | classifiers=( 26 | 'Development Status :: 5 - Production/Stable', 27 | 'Programming Language :: Python :: 2', 28 | 'Programming Language :: Python :: 2.7', 29 | 'Programming Language :: Python :: 3', 30 | 'Programming Language :: Python :: 3.2', 31 | 'Programming Language :: Python :: 3.3', 32 | 'Programming Language :: Python :: 3.4', 33 | 'License :: OSI Approved :: BSD License', 34 | ) 35 | ) 36 | 37 | --------------------------------------------------------------------------------