├── .gitignore ├── LICENSE ├── README.md ├── ansunit └── __init__.py ├── bar.lp ├── complex.yaml ├── foo.lp ├── setup.py └── simple.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.egg-info 3 | MANIFEST 4 | dist 5 | build 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (C) 2014 Adam M. Smith 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so, 8 | subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 15 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 16 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 17 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 18 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ansunit 2 | 3 | ## Overview 4 | 5 | **Ansunit** is a declarative testing framework for [answer set programming](https://en.wikipedia.org/wiki/Answer_set_programming) (ASP). In particular, it targets the ASP tools from the [Potassco](http://potassco.sourceforge.net/) project. Test suite descriptions are defined in [YAML](http://www.yaml.org/) syntax, and they are executed within Python's [unittest](https://docs.python.org/2/library/unittest.html) framework. 6 | 7 | ## Usage 8 | Install using `pip install ansunit`, then use the provided `ansunit` script on 9 | the command like: 10 | 11 | $ ansunit tests.yaml 12 | 13 | ## Concepts 14 | A test suite specification is a nested object definition in which important key names start with the word `Test`. Basic test specifications may define a **program** (a body of AnsProlog code), solver **arguments** (passed on the command line), and define an **expectation** for the result of running the program (regarding its satisfiability). Complex test specifications may make use of reusable **modules** described in a **definitions** specification. Tags and other test metadata may be included in the test name itself. 15 | 16 | ### Test Keywords 17 | - `Test ...` defines a subtest 18 | - `Program` defines a literal piece of code (childen override parents) 19 | - `Expect` defines an outcome (`SAT`, `UNSAT`, or `OPTIMAL`) 20 | - `Arguments` defines a list of string arguments 21 | - `Definitions` defines a *mapping* from module names to definitions 22 | - `Modules` defines a *list* of modules that will participate in the test 23 | 24 | ### Module Keywords 25 | A module may be defined as... 26 | 27 | - `some. inline. code.` (a YAML string) 28 | - `{filename: foo.lp}` 29 | - `{reference: another_module}` 30 | - `{group: [module_a, module_b]}` 31 | 32 | 33 | ## Syntax 34 | 35 | This simple specification defines two basic tests in canonical form (all details defined at the leaves of the specification). 36 | 37 | Test a implies b: 38 | Program: | 39 | b :- a. 40 | a. 41 | :- not b. 42 | Expect: SAT 43 | 44 | Test b not implies a: 45 | Program: | 46 | b :- a. 47 | b. 48 | :- not a. 49 | Expect: UNSAT 50 | 51 | This complex specification defines four test cases (three of which are grouped into a common suite). A `Definitions` section defines several modules with complex indirect references that are to be used later. In the second test (suite) several variations on a common test setup are defined concisely by means of inheriting details defined in the enclosing suite. Now shown, it is also possible to define all details including `Modules`, `Arguments`, and even `Program` at the top level for use by inheritance. 52 | 53 | Definitions: 54 | foo: {filename: foo.lp} 55 | bar: {filename: bar.lp} 56 | both: {group: [instance, encoding]} 57 | instance: {reference: bar} 58 | inline: | 59 | #const width = 3. 60 | dim(1..width). 61 | { p(X) } :- dim(X). 62 | 63 | 64 | Test twisted references: 65 | Definitions: 66 | encoding: {reference: foo} 67 | Modules: both 68 | Expect: SAT 69 | 70 | Test inline various: 71 | Modules: inline 72 | Expect: SAT 73 | 74 | Test small: 75 | Arguments: -c width=1 76 | 77 | Test medium: 78 | Arguments: -c width=3 79 | 80 | Test large: 81 | Arguments: -c width=5 82 | 83 | ## Command line arguments 84 | Run `ansunit --help` for more information. 85 | 86 | ## Working with large test suites 87 | 88 | Use the `-m` and `-n` (`--filter_match` and `--filter_nomatch`) arguments to 89 | select a focused subset of tests to run. A test is selected if it matches all 90 | positve conditions and no negative conditions. Conditions are checked by running 91 | `re.search` on the whole test name -- the strings with `::`. 92 | 93 | For the `demo_complex.yaml` suite above, we might select tests in the `various` 94 | sweet excluding the `large` test with a command like this (using `-l` to list 95 | tests matching the conditions rather than executing them): 96 | 97 | $ ansunit complex.yaml -m various -n large 98 | - Test complex.yaml :: inline various :: large 99 | * Test complex.yaml :: inline various :: medium 100 | * Test complex.yaml :: inline various :: small 101 | - Test complex.yaml :: twisted references 102 | 103 | ## Credits 104 | Created by Adam M. Smith (adam@adamsmith.as) 105 | -------------------------------------------------------------------------------- /ansunit/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ansunit.py: declarative unit testing for answer set programming""" 4 | 5 | from __future__ import print_function 6 | 7 | import argparse 8 | import unittest 9 | import subprocess 10 | import operator 11 | import yaml 12 | import re 13 | 14 | parser = argparse.ArgumentParser(description="AnsProlog unit testing") 15 | parser.add_argument("suite", help="test suite in YAML syntax") 16 | parser.add_argument("-c","--dump_canonical",action="store_true",help="dump test spec with all details pushed to leaves; exit") 17 | parser.add_argument("-l","--dump_list",action="store_true",help="dump transformed test names with filter matching; exit") 18 | parser.add_argument("-v","--verbosity", action="count", default=0) 19 | parser.add_argument("-s","--solver", default="clingo",help="which solver to use (default: clingo)") 20 | parser.add_argument("-o","--show_stdout",action="store_true") 21 | parser.add_argument("-e","--show_stderr",action="store_true") 22 | parser.add_argument("-x","--show_execution",action="store_true") 23 | parser.add_argument("-m","--filter_match",type=str,nargs="*",default=[],help="only run tests matching this regex") 24 | parser.add_argument("-n","--filter_nomatch",type=str,nargs="*",default=[],help="only run tests *not* matching this regex") 25 | parser.add_argument("-a","--solver_args",type=str,nargs="*",default=[]) 26 | 27 | def ensure_list(v): 28 | if type(v) == list: 29 | return v 30 | else: 31 | return [v] 32 | 33 | def reduce_contexts(parent, local): 34 | 35 | """Combine two test contexts into one. 36 | For value types of dict and list, the new context will aggregate the parent 37 | and local contexts. For other types, the value of the local context will 38 | replace the value of the parent (if any).""" 39 | 40 | context = {} 41 | 42 | for k,v in parent.items(): 43 | if type(v) == dict: 44 | d = v.copy() 45 | d.update(local.get(k,{})) 46 | context[k] = d 47 | elif type(v) == list: 48 | context[k] = v + ensure_list(local.get(k,[])) 49 | else: 50 | context[k] = local.get(k,v) 51 | 52 | for k in set(local.keys()) - set(parent.keys()): 53 | context[k] = local[k] 54 | 55 | return context 56 | 57 | def resolve_module(module, definitions): 58 | """Resolve (through indirections) the program contents of a module definition. 59 | The result is a list of program chunks.""" 60 | 61 | assert module in definitions, "No definition for module '%s'" % module 62 | 63 | d = definitions[module] 64 | if type(d) == dict: 65 | if 'filename' in d: 66 | with open(d['filename']) as f: 67 | return [f.read().strip()] 68 | elif 'reference' in d: 69 | return resolve_module(d['reference'], definitions) 70 | elif 'group' in d: 71 | return sum([resolve_module(m,definitions) for m in d['group']],[]) 72 | else: 73 | assert False 74 | else: 75 | assert type(d) == str 76 | return [d] 77 | 78 | def canonicalize_spec(spec, parent_context): 79 | """Push all context declarations to the leaves of a nested test specification.""" 80 | 81 | test_specs = {k:v for (k,v) in spec.items() if k.startswith("Test")} 82 | local_context = {k:v for (k,v) in spec.items() if not k.startswith("Test")} 83 | 84 | context = reduce_contexts(parent_context, local_context) 85 | 86 | if test_specs: 87 | return {k: canonicalize_spec(v, context) for (k,v) in test_specs.items()} 88 | else: 89 | program_chunks = sum([resolve_module(m,context['Definitions']) for m in context['Modules']],[]) + [context['Program']] 90 | test_spec = { 91 | 'Arguments': context['Arguments'], 92 | 'Program': "\n".join(program_chunks), 93 | 'Expect': context['Expect'], 94 | } 95 | return test_spec 96 | 97 | def flatten_spec(spec, prefix,joiner=" :: "): 98 | """Flatten a canonical specification with nesting into one without nesting. 99 | When building unique names, concatenate the given prefix to the local test 100 | name without the "Test " tag.""" 101 | 102 | if any(filter(operator.methodcaller("startswith","Test"),spec.keys())): 103 | flat_spec = {} 104 | for (k,v) in spec.items(): 105 | flat_spec.update(flatten_spec(v,prefix + joiner + k[5:])) 106 | return flat_spec 107 | else: 108 | return {"Test "+prefix: spec} 109 | 110 | 111 | 112 | class SolverTestCase(unittest.TestCase): 113 | 114 | def __init__(self, spec, args, description): 115 | self.spec = spec 116 | self.description = description 117 | self.args = args 118 | super(SolverTestCase,self).__init__() 119 | 120 | def __str__(self): 121 | return self.description 122 | 123 | def runTest(self): 124 | cmd = "%s %s %s" % (self.args.solver, ' '.join(self.args.solver_args), ' '.join(self.spec['Arguments'])) 125 | if self.args.show_execution: print("EXECUTING: ",cmd) 126 | proc = subprocess.Popen( 127 | cmd, 128 | shell=True, 129 | stdin=subprocess.PIPE, 130 | stdout=subprocess.PIPE, 131 | stderr=subprocess.PIPE) 132 | (out, err) = proc.communicate(self.spec['Program'].encode('utf8')) 133 | if self.args.show_stderr: print(err) 134 | if self.args.show_stdout: print(out) 135 | if self.spec['Expect'] == 'SAT': 136 | self.assertIn(proc.returncode,[10,30],msg='Expected SAT') 137 | elif self.spec['Expect'] == 'UNSAT': 138 | self.assertEqual(proc.returncode,20,msg='Expected UNSAT') 139 | elif self.spec['Expect'] == 'OPTIMAL': 140 | self.assertEqual(proc.returncode,30,msg='Expected OPTIMAL') 141 | else: 142 | assert False 143 | 144 | def main(): 145 | args = parser.parse_args() 146 | 147 | filename = args.suite 148 | with open(filename) as f: 149 | spec = yaml.load(f) 150 | 151 | initial_context = { 152 | 'Definitions': {}, 153 | 'Modules': [], 154 | 'Arguments': [], 155 | 'Program': '', 156 | 'Expect': 'SAT', 157 | } 158 | 159 | spec = canonicalize_spec(spec, initial_context) 160 | 161 | if args.dump_canonical: 162 | print(yaml.dump(spec)) 163 | return 164 | 165 | 166 | flat_spec = flatten_spec(spec,filename) 167 | 168 | matchers = [re.compile(m) for m in args.filter_match] 169 | nomatchers = [re.compile(n) for n in args.filter_nomatch] 170 | 171 | def selected(k): 172 | pos = all([m.search(k) for m in matchers]) 173 | neg = any([n.search(k) for n in nomatchers]) 174 | return all([m.search(k) for m in matchers]) \ 175 | and not any([n.search(k) for n in nomatchers]) 176 | 177 | if args.dump_list: 178 | print("\n".join([(" * " if selected(k) else " - ") + k for k in sorted(flat_spec.keys())])) 179 | return 180 | 181 | active_tests = filter(lambda t: selected(t[0]), sorted(flat_spec.items())) 182 | 183 | suite = unittest.TestSuite([SolverTestCase(v,args,k) for (k,v) in active_tests]) 184 | 185 | runner = unittest.TextTestRunner(verbosity=args.verbosity) 186 | result = runner.run(suite) 187 | return not result.wasSuccessful() 188 | 189 | if __name__ == "__main__": 190 | import sys 191 | sys.exit(main() or 0) 192 | -------------------------------------------------------------------------------- /bar.lp: -------------------------------------------------------------------------------- 1 | bar. 2 | -------------------------------------------------------------------------------- /complex.yaml: -------------------------------------------------------------------------------- 1 | Definitions: 2 | foo: {filename: foo.lp} 3 | bar: {filename: bar.lp} 4 | both: {group: [instance, encoding]} 5 | instance: {reference: bar} 6 | inline: | 7 | #const width = 3. 8 | dim(1..width). 9 | { p(X) } :- dim(X). 10 | 11 | Test twisted references: 12 | Definitions: 13 | encoding: {reference: foo} 14 | Modules: both 15 | Expect: SAT 16 | 17 | Test inline various: 18 | Modules: inline 19 | Expect: SAT 20 | 21 | Test small: 22 | Arguments: -c width=1 23 | 24 | Test medium: 25 | Arguments: -c width=3 26 | 27 | Test large: 28 | Arguments: -c width=5 29 | -------------------------------------------------------------------------------- /foo.lp: -------------------------------------------------------------------------------- 1 | foo. 2 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | setup(name='ansunit', 6 | version='0.1.7', 7 | description='Declarative unit testing for answer set programming projects', 8 | author='Adam M. Smith', 9 | author_email='adam@adamsmith.as', 10 | url='https://github.com/rndmcnlly/ansunit', 11 | license='MIT', 12 | packages=['ansunit'], 13 | install_requires=['PyYAML'], 14 | entry_points={'console_scripts': { 'ansunit = ansunit:main'}} 15 | ) 16 | -------------------------------------------------------------------------------- /simple.yaml: -------------------------------------------------------------------------------- 1 | Test a implies b: 2 | Program: | 3 | b :- a. 4 | a. 5 | :- not b. 6 | Expect: SAT 7 | 8 | Test b not implies a: 9 | Program: | 10 | b :- a. 11 | b. 12 | :- not a. 13 | Expect: UNSAT 14 | --------------------------------------------------------------------------------