├── .gitignore ├── .travis.yml ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.md ├── babbage ├── __init__.py ├── api.py ├── cube.py ├── exc.py ├── manager.py ├── model │ ├── __init__.py │ ├── aggregate.py │ ├── attribute.py │ ├── binding.py │ ├── concept.py │ ├── dimension.py │ ├── hierarchy.py │ ├── measure.py │ └── model.py ├── query │ ├── __init__.py │ ├── aggregates.py │ ├── cuts.py │ ├── drilldowns.py │ ├── fields.py │ ├── ordering.py │ ├── pagination.py │ └── parser.py ├── schema │ ├── model.json │ └── parser.ebnf ├── util.py ├── validation.py └── version.py ├── pylama.ini ├── setup.cfg ├── setup.py ├── tests ├── __init__.py ├── conftest.py ├── fixtures │ ├── cap_or_cur.csv │ ├── cofog1.csv │ ├── cra.csv │ └── models │ │ ├── cra.json │ │ ├── mexico.json │ │ └── simple_model.json ├── test_api.py ├── test_cube.py ├── test_manager.py ├── test_model.py ├── test_parser.py └── test_validation.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | *.egg-info 3 | *.pyc 4 | *.coverage 5 | env/* 6 | /.idea 7 | *.iml 8 | .tox 9 | .pytest_cache 10 | .cache 11 | .#* 12 | build/ 13 | dist/ 14 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | dist: xenial 3 | python: 4 | - "3.6" 5 | - "3.7" 6 | addons: 7 | postgresql: "9.6" 8 | 9 | env: 10 | global: 11 | - BABBAGE_TEST_DB=postgresql://postgres@/babbage 12 | matrix: 13 | - TOXENV="py${PYTHON_VERSION//./}" 14 | 15 | install: 16 | - pip install tox coveralls 17 | before_script: 18 | - psql -c 'create database babbage;' -U postgres 19 | script: 20 | - tox 21 | after_success: 22 | - coveralls 23 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2013, Open Knowledge Foundation, Friedrich Lindenberg, 2 | Gregor Aisch 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a 5 | copy of this software and associated documentation files (the 6 | "Software"), to deal in the Software without restriction, including 7 | without limitation the rights to use, copy, modify, merge, publish, 8 | distribute, sublicense, and/or sell copies of the Software, and to 9 | permit persons to whom the Software is furnished to do so, subject to 10 | the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included 13 | in all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 16 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 18 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 19 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 20 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 21 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | recursive-include babbage/schema * 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | 2 | test: 3 | tox 4 | 5 | install: env/bin/python 6 | 7 | env/bin/python: 8 | virtualenv env 9 | env/bin/pip install --upgrade pip 10 | env/bin/pip install -e . 11 | env/bin/pip install tox 12 | 13 | upload: install 14 | env/bin/python setup.py sdist bdist_wheel upload 15 | 16 | clean: 17 | rm -rf env 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Babbage Analytical Engine 2 | 3 | [![Gitter](https://img.shields.io/gitter/room/openspending/chat.svg)](https://gitter.im/openspending/chat) 4 | [![Build Status](https://travis-ci.org/openspending/babbage.svg?branch=master)](https://travis-ci.org/openspending/babbage) 5 | [![Coverage Status](https://coveralls.io/repos/openspending/babbage/badge.svg?branch=master&service=github)](https://coveralls.io/github/openspending/babbage?branch=master) 6 | 7 | ``babbage`` is a lightweight implementation of an OLAP-style database 8 | query tool for PostgreSQL. Given a database schema and a logical model 9 | of the data, it can be used to perform analytical queries against that 10 | data - programmatically or via a web API. 11 | 12 | It is heavily inspired by [Cubes](http://cubes.databrewery.org/) but 13 | has less ambitious goals, i.e. no pre-computation of aggregates, or 14 | multiple storage backends. 15 | 16 | ``babbage`` is not specific to government finances, and could easily be used e.g. for ReGENESIS, a project that makes German national statistics available via an API. The API functions by interpreting modelling metadata generated by the user (measures and dimensions). 17 | 18 | ## Installation and test 19 | 20 | ``babbage`` will normally included as a PyPI dependency, or installed via 21 | ``pip``: 22 | 23 | ```bash 24 | $ pip install babbage 25 | ``` 26 | 27 | People interested in contributing to the package should instead check out the 28 | source repository and then use the provided ``Makefile`` to install the 29 | library (this requires ``virtualenv`` to be installed): 30 | 31 | ```bash 32 | $ git clone https://github.com/openspending/babbage.git 33 | $ cd babbage 34 | $ make install 35 | $ pip install tox 36 | $ export BABBAGE_TEST_DB=postgresql://postgres@localhost:5432/postgres 37 | $ make test 38 | ``` 39 | 40 | ## Usage 41 | 42 | ``babbage`` is used to query a set of existing database tables, using an 43 | abstract, logical model to query them. A sample of a logical model can be 44 | found in ``tests/fixtures/models/cra.json``, and a JSON schema specifying 45 | the model is available in ``babbage/schema/model.json``. 46 | 47 | The central unit of ``babbage`` is a ``Cube``, i.e. a [OLAP cube](https://en.wikipedia.org/wiki/OLAP_cube) that uses the provided model metadata to construct queries 48 | against a database table. Additionally, the application supports managing 49 | multiple cubes at the same time via a ``CubeManager``, which can be 50 | subclassed to enable application-specific ways of defining cubes and where 51 | their metadata is stored. 52 | 53 | Futher, ``babbage`` includes a Flask Blueprint that can be used to expose 54 | a standard API via HTTP. This API is consumed by the JavaScript ``babbage.ui`` 55 | package and it is very closely modelled on the Cubes and OpenSpending HTTP 56 | APIs. 57 | 58 | ### Programmatic usage 59 | 60 | Let's assume you have an existing database table of procurement data and 61 | want to query it using ``babbage`` in a Python shell. A session might look 62 | like this: 63 | 64 | ```python 65 | import json 66 | from sqlalchemy import create_engine 67 | from babbage.cube import Cube 68 | from babbage.model import Measure 69 | 70 | engine = create_engine('postgresql://localhost/procurement') 71 | model = json.load(open('procurement_model.json', 'r')) 72 | 73 | cube = Cube(engine, 'procurement', model) 74 | facts = cube.facts(page_size=5) 75 | 76 | # There are 17201 rows in the table: 77 | assert facts['total_fact_count'] == 17201 78 | 79 | # There's a field called 'total_value': 80 | assert 'total_value' in facts['fields'] 81 | 82 | # We can get metadata about it: 83 | concept = cube.model['total_value'] 84 | assert isinstance(concept, Measure) 85 | assert concept.label == 'Total Value' 86 | 87 | # And there's some actual data: 88 | assert len(facts['data']) == 5 89 | fact_0 = facts['data'][0] 90 | assert 'total_value' in fact_0 91 | 92 | # For dimensions, we can get all the distinct values: 93 | members = cube.members('supplier', cut='year:2015', page_size=500) 94 | assert len(members['data']) <= 500 95 | assert members['total_member_count'] 96 | 97 | # And, finally, we can aggregate by specific dimensions: 98 | aggregate = cube.aggregate(aggregates='total_value.sum', 99 | drilldowns='supplier|authority' 100 | cut='year:2015|authority.country:GB', 101 | page_size=500) 102 | # This translates to: 103 | # Aggregate the procurement data by summing up the 'total_value' 104 | # for each unique pair of values in the 'supplier' and 'authority' 105 | # dimensions, and filter for only those entries where the 'year' 106 | # dimensions key attribute is '2015' and the 'authority' dimensions 107 | # 'country' attribute is 'GB'. Return the first 500 results. 108 | assert aggregate['total_cell_count'] 109 | assert len(aggregate['cells']) <= 500 110 | aggregate_0 = aggregate['cells'][0] 111 | assert 'total_value.sum' in aggregate_0 112 | 113 | # Note that these attribute names are made up for this example, they 114 | # should be reflected from the model: 115 | assert 'supplier.code' in aggregate_0 116 | assert 'supplier.label' in aggregate_0 117 | assert 'authority.code' in aggregate_0 118 | assert 'authority.label' in aggregate_0 119 | ``` 120 | 121 | ### Using the HTTP API 122 | 123 | The HTTP API for ``babbage`` is a simple Flask [Blueprint](http://flask.pocoo.org/docs/latest/blueprints/) used to expose a small set of calls that correspond to 124 | the cube functions listed above. To include it into an existing Flask 125 | application, you would need to create a ``CubeManager`` and then 126 | configure the API like this: 127 | 128 | ```python 129 | from flask import Flask 130 | from sqlalchemy import create_engine 131 | from babbage.manager import JSONCubeManager 132 | from babbage.api import configure_api 133 | 134 | app = Flask('demo') 135 | engine = 136 | models_directory = 'models/' 137 | manager = JSONCubeManager(engine, models_directory) 138 | blueprint = configure_api(app, manager) 139 | app.register_blueprint(blueprint, url_prefix='/api/babbage') 140 | 141 | app.run() 142 | ``` 143 | 144 | Of course, you can define your own ``CubeManager``, for example if 145 | you wish to retrieve model metadata from a database. 146 | 147 | When enabled, the API will expose a number of JSON(P) endpoints 148 | relative to the given ``url_prefix``: 149 | 150 | * ``/``, returns the system status and version. 151 | * ``/cubes``, returns a list of the available cubes (name only). 152 | * ``/cubes//model``, returns full metadata for a given 153 | cube (i.e. measures, dimensions, aggregates etc.) 154 | * ``/cubes//facts`` is used to return individual entries from 155 | the cube in a non-aggregated form. Supports filters (``cut``), a 156 | set of ``fields`` to return and a ``sort`` (``field_name:direction``), 157 | as well as ``page`` and ``page_size``. 158 | * ``/cubes//members`` is used to return the distinct set of 159 | values for a given dimension, e.g. all the suppliers mentioned in 160 | a procurement dataset. Supports filters (``cut``), a and a ``sort`` 161 | (``field_name:direction``), as well as ``page`` and ``page_size``. 162 | * ``/cubes//aggregate`` is the main endpoint for generating 163 | aggregate views of the data. Supports specifying the ``aggregates`` 164 | to include, the ``drilldowns`` to aggregate by, a set of filters 165 | (``cut``), a and a ``sort`` (``field_name:direction``), as well 166 | as ``page`` and ``page_size``. 167 | 168 | -------------------------------------------------------------------------------- /babbage/__init__.py: -------------------------------------------------------------------------------- 1 | """ Babbage, an OLAP-like, light-weight database analytical engine. """ 2 | 3 | 4 | from babbage.version import __version__ # noqa 5 | from babbage.manager import CubeManager, JSONCubeManager # noqa 6 | from babbage.api import configure_api # noqa 7 | from babbage.validation import validate_model # noqa 8 | from babbage.exc import BabbageException, QueryException, BindingException # noqa 9 | -------------------------------------------------------------------------------- /babbage/api.py: -------------------------------------------------------------------------------- 1 | # Flask web api 2 | # TODO: consider making this it's own Python package? 3 | from datetime import date 4 | from decimal import Decimal 5 | 6 | from werkzeug.exceptions import NotFound 7 | from flask import Blueprint, Response, request, current_app, json, url_for 8 | 9 | from babbage.exc import BabbageException 10 | 11 | map_is_class = type(map) == type 12 | 13 | blueprint = Blueprint('babbage_api', __name__) 14 | 15 | 16 | def configure_api(app, manager): 17 | """ Configure the current Flask app with an instance of ``CubeManager`` that 18 | will be used to load and query data. """ 19 | if not hasattr(app, 'extensions'): 20 | app.extensions = {} # pragma: nocover 21 | app.extensions['babbage'] = manager 22 | return blueprint 23 | 24 | 25 | def get_manager(): 26 | """ Try to locate a ``CubeManager`` on the Flask app which is currently 27 | processing a request. This will only work inside the request cycle. """ 28 | return current_app.extensions['babbage'] 29 | 30 | 31 | def get_cube(name): 32 | """ Load the named cube from the current registered ``CubeManager``. """ 33 | manager = get_manager() 34 | if not manager.has_cube(name): 35 | raise NotFound('No such cube: %r' % name) 36 | return manager.get_cube(name) 37 | 38 | 39 | class JSONEncoder(json.JSONEncoder): 40 | """ This encoder will serialize all entities that have a to_dict 41 | method by calling that method and serializing the result. """ 42 | 43 | def default(self, obj): 44 | if isinstance(obj, date): 45 | return obj.isoformat() 46 | if isinstance(obj, Decimal): 47 | return float(obj) 48 | if isinstance(obj, set): 49 | return [o for o in obj] 50 | if map_is_class and isinstance(obj, map): 51 | return [o for o in obj] 52 | if hasattr(obj, 'to_dict'): 53 | return obj.to_dict() 54 | return json.JSONEncoder.default(self, obj) 55 | 56 | 57 | def jsonify(obj, status=200, headers=None): 58 | """ Custom JSONificaton to support obj.to_dict protocol. """ 59 | data = JSONEncoder().encode(obj) 60 | if 'callback' in request.args: 61 | cb = request.args.get('callback') 62 | data = '%s && %s(%s)' % (cb, cb, data) 63 | return Response(data, headers=headers, status=status, 64 | mimetype='application/json') 65 | 66 | 67 | def create_csv_response(rows): 68 | def _generator(): 69 | convert_to_str = lambda value: str(value) if value is not None else '' # noqa 70 | columns = [] 71 | 72 | for index, row in enumerate(rows): 73 | if index == 0: 74 | columns = sorted(row.keys()) 75 | yield ','.join(columns) + '\n' 76 | 77 | data = [ 78 | convert_to_str(row.get(column)) 79 | for column in columns 80 | ] 81 | yield ','.join(data) + '\n' 82 | 83 | return Response( 84 | _generator(), 85 | mimetype='text/csv', 86 | headers={ 87 | 'Content-disposition': 'attachment; filename="data.csv"' 88 | } 89 | ) 90 | 91 | 92 | def url(*a, **kw): 93 | kw['_external'] = True 94 | return url_for(*a, **kw) 95 | 96 | 97 | @blueprint.errorhandler(BabbageException) 98 | def handle_error(exc): 99 | return jsonify({ 100 | 'status': 'error', 101 | 'message': exc.message, 102 | 'context': exc.context 103 | }, status=exc.http_equiv) 104 | 105 | 106 | @blueprint.route('/') 107 | def index(): 108 | """ General system status report :) """ 109 | from babbage import __version__, __doc__ 110 | return jsonify({ 111 | 'status': 'ok', 112 | 'api': 'babbage', 113 | 'message': __doc__, 114 | 'cubes_index_url': url('babbage_api.cubes'), 115 | 'version': __version__ 116 | }) 117 | 118 | 119 | @blueprint.route('/cubes/') 120 | def cubes(): 121 | """ Get a listing of all publicly available cubes. """ 122 | cubes = [] 123 | for cube in get_manager().list_cubes(): 124 | cubes.append({ 125 | 'name': cube 126 | }) 127 | return jsonify({ 128 | 'status': 'ok', 129 | 'data': cubes 130 | }) 131 | 132 | 133 | @blueprint.route('/cubes//model/') 134 | def model(name): 135 | """ Get the model for the specified cube. """ 136 | cube = get_cube(name) 137 | return jsonify({ 138 | 'status': 'ok', 139 | 'name': name, 140 | 'model': cube.model 141 | }) 142 | 143 | 144 | @blueprint.route('/cubes//aggregate/') 145 | def aggregate(name): 146 | """ Perform an aggregation request. """ 147 | cube = get_cube(name) 148 | result = cube.aggregate(aggregates=request.args.get('aggregates'), 149 | drilldowns=request.args.get('drilldown'), 150 | cuts=request.args.get('cut'), 151 | order=request.args.get('order'), 152 | page=request.args.get('page'), 153 | page_size=request.args.get('pagesize')) 154 | result['status'] = 'ok' 155 | 156 | if request.args.get('format', '').lower() == 'csv': 157 | return create_csv_response(result['cells']) 158 | else: 159 | return jsonify(result) 160 | 161 | 162 | @blueprint.route('/cubes//facts/') 163 | def facts(name): 164 | """ List the fact table entries in the current cube. This is the full 165 | materialized dataset. """ 166 | cube = get_cube(name) 167 | result = cube.facts(fields=request.args.get('fields'), 168 | cuts=request.args.get('cut'), 169 | order=request.args.get('order'), 170 | page=request.args.get('page'), 171 | page_size=request.args.get('pagesize')) 172 | result['status'] = 'ok' 173 | return jsonify(result) 174 | 175 | 176 | @blueprint.route('/cubes//members//') 177 | def members(name, ref): 178 | """ List the members of a specific dimension or the distinct values of a 179 | given attribute. """ 180 | cube = get_cube(name) 181 | result = cube.members(ref, cuts=request.args.get('cut'), 182 | order=request.args.get('order'), 183 | page=request.args.get('page'), 184 | page_size=request.args.get('pagesize')) 185 | result['status'] = 'ok' 186 | return jsonify(result) 187 | -------------------------------------------------------------------------------- /babbage/cube.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import MetaData 2 | from sqlalchemy.schema import Table 3 | from sqlalchemy.sql.expression import select 4 | from six import string_types 5 | 6 | from babbage.model import Model 7 | from babbage.model.dimension import Dimension 8 | from babbage.query import count_results, generate_results, first_result 9 | from babbage.query import Cuts, Drilldowns, Fields, Ordering, Aggregates 10 | from babbage.query import Pagination 11 | from babbage.exc import BindingException 12 | 13 | 14 | class Cube(object): 15 | """ A dataset that can be queried across a set of dimensions and measures. 16 | This functions as the central hub of functionality for accessing any 17 | data and queries. """ 18 | 19 | def __init__(self, engine, name, model, fact_table=None): 20 | self.name = name 21 | if not isinstance(model, Model): 22 | model = Model(model) 23 | self._tables = {} 24 | if fact_table is not None: 25 | self._tables[model.fact_table_name] = fact_table 26 | self.model = model 27 | self.engine = engine 28 | self.meta = MetaData(bind=engine) 29 | 30 | def _load_table(self, name): 31 | """ Reflect a given table from the database. """ 32 | table = self._tables.get(name, None) 33 | if table is not None: 34 | return table 35 | if not self.engine.has_table(name): 36 | raise BindingException('Table does not exist: %r' % name, 37 | table=name) 38 | table = Table(name, self.meta, autoload=True) 39 | self._tables[name] = table 40 | return table 41 | 42 | @property 43 | def fact_pk(self): 44 | """ Try to determine the primary key of the fact table for use in 45 | fact table counting. 46 | If more than one column exists, return the first column of the pk. 47 | """ 48 | keys = [c for c in self.fact_table.columns if c.primary_key] 49 | return keys[0] 50 | 51 | @property 52 | def fact_table(self): 53 | return self._load_table(self.model.fact_table_name) 54 | 55 | @property 56 | def is_postgresql(self): 57 | """ Enable postgresql-specific extensions. """ 58 | return 'postgresql' == self.engine.dialect.name 59 | 60 | def aggregate(self, aggregates=None, drilldowns=None, cuts=None, 61 | order=None, page=None, page_size=None, page_max=None): 62 | """Main aggregation function. This is used to compute a given set of 63 | aggregates, grouped by a given set of drilldown dimensions (i.e. 64 | dividers). The query can also be filtered and sorted. """ 65 | 66 | def prep(cuts, drilldowns=False, aggregates=False, columns=None): 67 | q = select(columns) 68 | bindings = [] 69 | cuts, q, bindings = Cuts(self).apply(q, bindings, cuts) 70 | 71 | attributes = None 72 | if drilldowns is not False: 73 | attributes, q, bindings = Drilldowns(self).apply( 74 | q, 75 | bindings, 76 | drilldowns 77 | ) 78 | 79 | if aggregates is not False: 80 | aggregates, q, bindings = Aggregates(self).apply( 81 | q, 82 | bindings, 83 | aggregates 84 | ) 85 | 86 | q = self.restrict_joins(q, bindings) 87 | return q, bindings, attributes, aggregates, cuts 88 | 89 | # Count 90 | count = count_results(self, prep(cuts, 91 | drilldowns=drilldowns, 92 | columns=[1])[0]) 93 | 94 | # Summary 95 | summary = first_result(self, prep(cuts, 96 | aggregates=aggregates)[0].limit(1)) 97 | 98 | # Results 99 | q, bindings, attributes, aggregates, cuts = \ 100 | prep(cuts, drilldowns=drilldowns, aggregates=aggregates) 101 | page, q = Pagination(self).apply(q, page, page_size, page_max) 102 | ordering, q, bindings = Ordering(self).apply(q, bindings, order) 103 | q = self.restrict_joins(q, bindings) 104 | 105 | cells = list(generate_results(self, q)) 106 | 107 | return { 108 | 'total_cell_count': count, 109 | 'cells': cells, 110 | 'summary': summary, 111 | 'cell': cuts, 112 | 'aggregates': aggregates, 113 | 'attributes': attributes, 114 | 'order': ordering, 115 | 'page': page['page'], 116 | 'page_size': page['page_size'] 117 | } 118 | 119 | def members(self, ref, cuts=None, order=None, page=None, page_size=None): 120 | """ List all the distinct members of the given reference, filtered and 121 | paginated. If the reference describes a dimension, all attributes are 122 | returned. """ 123 | def prep(cuts, ref, order, columns=None): 124 | q = select(columns=columns) 125 | bindings = [] 126 | cuts, q, bindings = Cuts(self).apply(q, bindings, cuts) 127 | fields, q, bindings = \ 128 | Fields(self).apply(q, bindings, ref, distinct=True) 129 | ordering, q, bindings = \ 130 | Ordering(self).apply(q, bindings, order, distinct=fields[0]) 131 | q = self.restrict_joins(q, bindings) 132 | return q, bindings, cuts, fields, ordering 133 | 134 | # Count 135 | count = count_results(self, prep(cuts, ref, order, [1])[0]) 136 | 137 | # Member list 138 | q, bindings, cuts, fields, ordering = prep(cuts, ref, order) 139 | page, q = Pagination(self).apply(q, page, page_size) 140 | q = self.restrict_joins(q, bindings) 141 | return { 142 | 'total_member_count': count, 143 | 'data': list(generate_results(self, q)), 144 | 'cell': cuts, 145 | 'fields': fields, 146 | 'order': ordering, 147 | 'page': page['page'], 148 | 'page_size': page['page_size'] 149 | } 150 | 151 | def facts(self, fields=None, cuts=None, order=None, page=None, 152 | page_size=None, page_max=None): 153 | """ List all facts in the cube, returning only the specified references 154 | if these are specified. """ 155 | 156 | def prep(cuts, columns=None): 157 | q = select(columns=columns).select_from(self.fact_table) 158 | bindings = [] 159 | _, q, bindings = Cuts(self).apply(q, bindings, cuts) 160 | q = self.restrict_joins(q, bindings) 161 | return q, bindings 162 | 163 | # Count 164 | count = count_results(self, prep(cuts, [1])[0]) 165 | 166 | # Facts 167 | q, bindings = prep(cuts) 168 | fields, q, bindings = Fields(self).apply(q, bindings, fields) 169 | ordering, q, bindings = Ordering(self).apply(q, bindings, order) 170 | page, q = Pagination(self).apply(q, page, page_size, page_max) 171 | q = self.restrict_joins(q, bindings) 172 | return { 173 | 'total_fact_count': count, 174 | 'data': list(generate_results(self, q)), 175 | 'cell': cuts, 176 | 'fields': fields, 177 | 'order': ordering, 178 | 'page': page['page'], 179 | 'page_size': page['page_size'] 180 | } 181 | 182 | def compute_cardinalities(self): 183 | """ This will count the number of distinct values for each dimension in 184 | the dataset and add that count to the model so that it can be used as a 185 | hint by UI components. """ 186 | for dimension in self.model.dimensions: 187 | result = self.members(dimension.ref, page_size=0) 188 | dimension.spec['cardinality'] = result.get('total_member_count') 189 | 190 | def restrict_joins(self, q, bindings): 191 | """ 192 | Restrict the joins across all tables referenced in the database 193 | query to those specified in the model for the relevant dimensions. 194 | If a single table is used for the query, no unnecessary joins are 195 | performed. If more than one table are referenced, this ensures 196 | their returned rows are connected via the fact table. 197 | """ 198 | if len(q.froms) == 1: 199 | return q 200 | else: 201 | for binding in bindings: 202 | if binding.table == self.fact_table: 203 | continue 204 | concept = self.model[binding.ref] 205 | if isinstance(concept, Dimension): 206 | dimension = concept 207 | else: 208 | dimension = concept.dimension 209 | dimension_table, key_col = dimension.key_attribute.bind(self) 210 | if binding.table != dimension_table: 211 | raise BindingException('Attributes must be of same table as ' 212 | 'as their dimension key') 213 | 214 | join_column_name = dimension.join_column_name 215 | if isinstance(join_column_name, string_types): 216 | try: 217 | join_column = self.fact_table.columns[join_column_name] 218 | except KeyError: 219 | raise BindingException("Join column '%s' for %r not in fact table." 220 | % (dimension.join_column_name, dimension)) 221 | else: 222 | if not isinstance(join_column_name, list) or len(join_column_name) != 2: 223 | raise BindingException("Join column '%s' for %r should be either a string or a 2-tuple." 224 | % (join_column_name, dimension)) 225 | try: 226 | join_column = self.fact_table.columns[join_column_name[0]] 227 | except KeyError: 228 | raise BindingException("Join column '%s' for %r not in fact table." 229 | % (dimension.join_column_name[0], dimension)) 230 | try: 231 | key_col = dimension_table.columns[join_column_name[1]] 232 | except KeyError: 233 | raise BindingException("Join column '%s' for %r not in dimension table." 234 | % (dimension.join_column_name[1], dimension)) 235 | 236 | q = q.where(join_column == key_col) 237 | return q 238 | 239 | def __repr__(self): 240 | return '" % self.ref 36 | 37 | def to_dict(self): 38 | data = self.spec.copy() 39 | data['ref'] = self.ref 40 | data['function'] = self.function 41 | if self.measure is not None: 42 | data['measure'] = self.measure.ref 43 | return data 44 | -------------------------------------------------------------------------------- /babbage/model/attribute.py: -------------------------------------------------------------------------------- 1 | from babbage.model.concept import Concept 2 | 3 | 4 | class Attribute(Concept): 5 | """ An attribute describes some concrete value stored in the data model. 6 | This value can either be stored directly on the facts table as a column, 7 | or introduced via a join. """ 8 | 9 | def __init__(self, dimension, name, spec): 10 | super(Attribute, self).__init__( 11 | dimension.model, name, spec, '%s.%s' % (dimension.hierarchy, name) 12 | ) 13 | self.dimension = dimension 14 | 15 | @property 16 | def ref(self): 17 | return '%s.%s' % (self.dimension.name, self.name) 18 | 19 | @property 20 | def datatype(self): 21 | return self.spec.get('type') 22 | 23 | def __repr__(self): 24 | return "" % self.ref 25 | 26 | def to_dict(self): 27 | data = self.spec.copy() 28 | data['ref'] = self.ref 29 | return data 30 | -------------------------------------------------------------------------------- /babbage/model/binding.py: -------------------------------------------------------------------------------- 1 | class Binding(object): 2 | def __init__(self, table, ref): 3 | self.table = table 4 | self.ref = ref 5 | 6 | def __unicode__(self): 7 | return u"" % (self.table.name, self.ref) 8 | 9 | def __repr__(self): 10 | return self.__unicode__() 11 | -------------------------------------------------------------------------------- /babbage/model/concept.py: -------------------------------------------------------------------------------- 1 | from babbage.exc import BindingException 2 | 3 | 4 | class Concept(object): 5 | """ A concept describes any branch of the model: dimensions, attributes, 6 | measures. """ 7 | 8 | def __init__(self, model, name, spec, alias=None): 9 | self.model = model 10 | self.name = name 11 | self.alias = name if alias is None else alias 12 | self.spec = spec 13 | self.label = spec.get('label', name) 14 | self.description = spec.get('description') 15 | self.column_name = spec.get('column') 16 | self._matched_ref = None 17 | 18 | @property 19 | def ref(self): 20 | """ A unique reference within the context of this model. """ 21 | return self.name 22 | 23 | @property 24 | def refs(self): 25 | """ Aliases for this model's ref. """ 26 | return [self.ref, self.alias] 27 | 28 | @property 29 | def matched_ref(self): 30 | return self.ref if self._matched_ref is None else self._matched_ref 31 | 32 | @property 33 | def datatype(self): 34 | """ 35 | String name of the type of the concept, ie string, integer or date, 36 | to be overridden by concrete subclasses. 37 | """ 38 | return None 39 | 40 | def match_ref(self, ref): 41 | """ Check if the ref matches one the concept's aliases. 42 | If so, mark the matched ref so that we use it as the column label. 43 | """ 44 | if ref in self.refs: 45 | self._matched_ref = ref 46 | return True 47 | return False 48 | 49 | def _physical_column(self, cube, column_name): 50 | """ Return the SQLAlchemy Column object matching a given, possibly 51 | qualified, column name (i.e.: 'table.column'). If no table is named, 52 | the fact table is assumed. """ 53 | table_name = self.model.fact_table_name 54 | if '.' in column_name: 55 | table_name, column_name = column_name.split('.', 1) 56 | table = cube._load_table(table_name) 57 | if column_name not in table.columns: 58 | raise BindingException('Column %r does not exist on table %r' % ( 59 | column_name, table_name), table=table_name, 60 | column=column_name) 61 | return table, table.columns[column_name] 62 | 63 | def bind(self, cube): 64 | """ Map a model reference to an physical column in the database. """ 65 | table, column = self._physical_column(cube, self.column_name) 66 | column = column.label(self.matched_ref) 67 | column.quote = True 68 | return table, column 69 | 70 | def __eq__(self, other): 71 | """ Test concept equality by means of references. """ 72 | if hasattr(other, 'ref'): 73 | return other.ref == self.ref 74 | return self.ref == other 75 | 76 | def __unicode__(self): 77 | return self.ref 78 | -------------------------------------------------------------------------------- /babbage/model/dimension.py: -------------------------------------------------------------------------------- 1 | from babbage.model.concept import Concept 2 | from babbage.model.attribute import Attribute 3 | from babbage.exc import BindingException 4 | 5 | 6 | class Dimension(Concept): 7 | """ A dimension is any property of an entry that can serve to describe 8 | it beyond its purely numeric ``Measures``. It is defined by several 9 | attributes, which contain actual values. """ 10 | 11 | def __init__(self, model, name, spec, hierarchy=None): 12 | super(Dimension, self).__init__(model, name, spec) 13 | self.hierarchy = hierarchy if hierarchy is not None else self.name 14 | self.join_column_name = spec.get('join_column') 15 | 16 | @property 17 | def attributes(self): 18 | key_attr_ref = self.spec.get('key_attribute') 19 | for name, attr in self.spec.get('attributes', {}).items(): 20 | if name == key_attr_ref: 21 | yield Attribute(self, name, attr) 22 | break 23 | for name, attr in self.spec.get('attributes', {}).items(): 24 | if name != key_attr_ref: 25 | yield Attribute(self, name, attr) 26 | 27 | @property 28 | def label_attribute(self): 29 | for attr in self.attributes: 30 | if attr.name == self.spec.get('label_attribute'): 31 | return attr 32 | return self.key_attribute 33 | 34 | @property 35 | def key_attribute(self): 36 | for attr in self.attributes: 37 | if attr.name == self.spec.get('key_attribute'): 38 | return attr 39 | raise BindingException("key_attribute '%s' not found in dimension '%s'" 40 | % (self.spec.get('key_attribute'), self.name)) 41 | 42 | @property 43 | def cardinality(self): 44 | """ Get the number of distinct values of the dimension. This is stored 45 | in the model as a denormalization which can be generated using a 46 | method on the ``Cube``. """ 47 | return self.spec.get('cardinality') 48 | 49 | @property 50 | def cardinality_class(self): 51 | """ Group the cardinality of the dimension into one of four buckets, 52 | from very small (less than 5) to very large (more than 1000). """ 53 | if self.cardinality: 54 | if self.cardinality > 1000: 55 | return 'high' 56 | if self.cardinality > 50: 57 | return 'medium' 58 | if self.cardinality > 7: 59 | return 'low' 60 | return 'tiny' 61 | 62 | @property 63 | def datatype(self): 64 | return self.key_attribute.datatype 65 | 66 | def bind(self, cube): 67 | """ When one column needs to match, use the key. """ 68 | return self.key_attribute.bind(cube) 69 | 70 | def __repr__(self): 71 | return "" % self.ref 72 | 73 | def to_dict(self): 74 | data = self.spec.copy() 75 | data['ref'] = self.ref 76 | data['label_attribute'] = self.label_attribute.name 77 | data['label_ref'] = self.label_attribute.ref 78 | data['key_attribute'] = self.key_attribute.name 79 | data['key_ref'] = self.key_attribute.ref 80 | data['cardinality_class'] = self.cardinality_class 81 | data['attributes'] = {a.name: a.to_dict() for a in self.attributes} 82 | data['hierarchy'] = self.hierarchy 83 | return data 84 | -------------------------------------------------------------------------------- /babbage/model/hierarchy.py: -------------------------------------------------------------------------------- 1 | 2 | class Hierarchy(object): 3 | """Represents a logical grouping and ordering of existing dimensions""" 4 | 5 | def __init__(self, name, hierarchy): 6 | self.name = name 7 | self.label = hierarchy.get('label', name) 8 | self.levels = hierarchy.get('levels', []) 9 | 10 | def to_dict(self): 11 | data = dict() 12 | data['ref'] = self.name 13 | data['label'] = self.label 14 | data['levels'] = self.levels[:] 15 | return data 16 | -------------------------------------------------------------------------------- /babbage/model/measure.py: -------------------------------------------------------------------------------- 1 | from babbage.model.concept import Concept 2 | 3 | 4 | class Measure(Concept): 5 | """ A value on the facts table that can be subject to aggregation, 6 | and is specific to this one fact. This would typically be some 7 | financial unit, i.e. the amount associated with the transaction or 8 | a specific portion thereof (i.e. co-financed amounts). """ 9 | 10 | def __init__(self, model, name, spec): 11 | super(Measure, self).__init__(model, name, spec) 12 | self.column_name = spec.get('column') 13 | self.aggregates = spec.get('aggregates', ['sum']) 14 | 15 | @property 16 | def datatype(self): 17 | return self.spec.get('type') 18 | 19 | def __repr__(self): 20 | return "" % self.ref 21 | 22 | def to_dict(self): 23 | data = self.spec.copy() 24 | data['ref'] = self.ref 25 | return data 26 | -------------------------------------------------------------------------------- /babbage/model/model.py: -------------------------------------------------------------------------------- 1 | from babbage.model.dimension import Dimension 2 | from babbage.model.hierarchy import Hierarchy 3 | from babbage.model.measure import Measure 4 | from babbage.model.aggregate import Aggregate 5 | 6 | 7 | class Model(object): 8 | """ The ``Model`` serves as an abstract representation of a cube, 9 | representing its measures, dimensions and attributes. """ 10 | 11 | def __init__(self, spec): 12 | """ Construct the in-memory object representation of this 13 | dataset's dimension and measures model. 14 | 15 | This is called upon initialization and deserialization of 16 | the dataset from the SQLAlchemy store. 17 | """ 18 | self.spec = spec 19 | 20 | @property 21 | def fact_table_name(self): 22 | return self.spec.get('fact_table') 23 | 24 | @property 25 | def dimensions(self): 26 | hierarchy_map = {} 27 | for h in self.hierarchies: 28 | for lvl in h.levels: 29 | hierarchy_map[lvl] = h.name 30 | for name, data in self.spec.get('dimensions', {}).items(): 31 | yield Dimension(self, name, data, hierarchy_map.get(name)) 32 | 33 | @property 34 | def measures(self): 35 | for name, data in self.spec.get('measures', {}).items(): 36 | yield Measure(self, name, data) 37 | 38 | @property 39 | def hierarchies(self): 40 | for name, data in self.spec.get('hierarchies', {}).items(): 41 | yield Hierarchy(name, data) 42 | 43 | @property 44 | def attributes(self): 45 | for dimension in self.dimensions: 46 | for attribute in dimension.attributes: 47 | yield attribute 48 | 49 | @property 50 | def aggregates(self): 51 | # TODO: nicer way than hard-coding this? 52 | yield Aggregate(self, 'Facts', 'count') 53 | 54 | for measure in self.measures: 55 | for function in measure.aggregates: 56 | yield Aggregate(self, measure.label, function, 57 | measure=measure) 58 | 59 | @property 60 | def concepts(self): 61 | """ Return all existing concepts, i.e. dimensions, measures and 62 | attributes within the model. """ 63 | for measure in self.measures: 64 | yield measure 65 | for aggregate in self.aggregates: 66 | yield aggregate 67 | for dimension in self.dimensions: 68 | yield dimension 69 | for attribute in dimension.attributes: 70 | yield attribute 71 | 72 | def match(self, ref): 73 | """ Get all concepts matching this ref. For a dimension, that is all 74 | its attributes, but not the dimension itself. """ 75 | try: 76 | concept = self[ref] 77 | if not isinstance(concept, Dimension): 78 | return [concept] 79 | return [a for a in concept.attributes] 80 | except KeyError: 81 | return [] 82 | 83 | @property 84 | def exists(self): 85 | """ Check if the model satisfies the basic conditions for being 86 | queried, i.e. at least one measure. """ 87 | return len(list(self.measures)) > 0 88 | 89 | def __getitem__(self, ref): 90 | """ Access a ref (dimension, attribute or measure) by ref. """ 91 | for concept in self.concepts: 92 | if concept.match_ref(ref): 93 | return concept 94 | raise KeyError() 95 | 96 | def __contains__(self, name): 97 | """ Check if the given ref exists within the model. """ 98 | try: 99 | self[name] 100 | return True 101 | except KeyError: 102 | return False 103 | 104 | def __repr__(self): 105 | return "" % self.fact_table_name 106 | 107 | def to_dict(self): 108 | data = self.spec.copy() 109 | data['measures'] = {m.name: m.to_dict() for m in self.measures} 110 | data['dimensions'] = {d.name: d.to_dict() for d in self.dimensions} 111 | data['aggregates'] = {a.ref: a.to_dict() for a in self.aggregates} 112 | data['hierarchies'] = {h.name: h.to_dict() for h in self.hierarchies} 113 | return data 114 | -------------------------------------------------------------------------------- /babbage/query/__init__.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import func 2 | from sqlalchemy.sql.expression import select 3 | 4 | from babbage.query.cuts import Cuts # noqa 5 | from babbage.query.fields import Fields # noqa 6 | from babbage.query.drilldowns import Drilldowns # noqa 7 | from babbage.query.aggregates import Aggregates # noqa 8 | from babbage.query.ordering import Ordering # noqa 9 | from babbage.query.pagination import Pagination # noqa 10 | 11 | 12 | def count_results(cube, q): 13 | """ Get the count of records matching the query. """ 14 | q = select(columns=[func.count(True)], from_obj=q.alias()) 15 | return cube.engine.execute(q).scalar() 16 | 17 | 18 | def generate_results(cube, q): 19 | """ Generate the resulting records for this query, applying pagination. 20 | Values will be returned by their reference. """ 21 | if q._limit is not None and q._limit < 1: 22 | return 23 | rp = cube.engine.execute(q) 24 | while True: 25 | row = rp.fetchone() 26 | if row is None: 27 | return 28 | yield dict(row.items()) 29 | 30 | 31 | def first_result(cube, q): 32 | for row in generate_results(cube, q): 33 | return row 34 | -------------------------------------------------------------------------------- /babbage/query/aggregates.py: -------------------------------------------------------------------------------- 1 | from babbage.query.parser import Parser 2 | from babbage.model.binding import Binding 3 | from babbage.exc import QueryException 4 | 5 | 6 | class Aggregates(Parser): 7 | """ Handle parser output for aggregate/drilldown specifications. """ 8 | start = "aggregates" 9 | 10 | def aggregate(self, ast): 11 | refs = [a.ref for a in self.cube.model.aggregates] 12 | if ast not in refs: 13 | raise QueryException('Invalid aggregate: %r' % ast) 14 | self.results.append(ast) 15 | 16 | def apply(self, q, bindings, aggregates): 17 | info = [] 18 | for aggregate in self.parse(aggregates): 19 | info.append(aggregate) 20 | table, column = self.cube.model[aggregate].bind(self.cube) 21 | bindings.append(Binding(table, aggregate)) 22 | q = q.column(column) 23 | 24 | if not len(self.results): 25 | # If no aggregates are specified, aggregate on all. 26 | for aggregate in self.cube.model.aggregates: 27 | info.append(aggregate.ref) 28 | table, column = aggregate.bind(self.cube) 29 | bindings.append(Binding(table, aggregate.ref)) 30 | q = q.column(column) 31 | return info, q, bindings 32 | -------------------------------------------------------------------------------- /babbage/query/cuts.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | 3 | import six 4 | 5 | from babbage.api import map_is_class 6 | from babbage.query.parser import Parser 7 | from babbage.model.binding import Binding 8 | from babbage.exc import QueryException 9 | 10 | 11 | class Cuts(Parser): 12 | """ Handle parser output for cuts. """ 13 | start = "cuts" 14 | 15 | def cut(self, ast): 16 | value = ast[2] 17 | if isinstance(value, six.string_types) and len(value.strip()) == 0: 18 | value = None 19 | # TODO: can you filter measures or aggregates? 20 | if ast[0] not in self.cube.model: 21 | raise QueryException('Invalid cut: %r' % ast[0]) 22 | self.results.append((ast[0], ast[1], value)) 23 | 24 | def _check_type(self, ref, value): 25 | """ 26 | Checks whether the type of the cut value matches the type of the 27 | concept being cut, and raises a QueryException if it doesn't match 28 | """ 29 | if isinstance(value, list): 30 | return [self._check_type(ref, val) for val in value] 31 | 32 | model_type = self.cube.model[ref].datatype 33 | if model_type is None: 34 | return 35 | query_type = self._api_type(value) 36 | if query_type == model_type: 37 | return 38 | else: 39 | raise QueryException("Invalid value %r parsed as type '%s' " 40 | "for cut %s of type '%s'" 41 | % (value, query_type, ref, model_type)) 42 | 43 | def _api_type(self, value): 44 | """ 45 | Returns the API type of the given value based on its python type. 46 | 47 | """ 48 | if isinstance(value, six.string_types): 49 | return 'string' 50 | elif isinstance(value, six.integer_types): 51 | return 'integer' 52 | elif type(value) is datetime.datetime: 53 | return 'date' 54 | 55 | def apply(self, q, bindings, cuts): 56 | """ Apply a set of filters, which can be given as a set of tuples in 57 | the form (ref, operator, value), or as a string in query form. If it 58 | is ``None``, no filter will be applied. """ 59 | info = [] 60 | for (ref, operator, value) in self.parse(cuts): 61 | if map_is_class and isinstance(value, map): 62 | value = list(value) 63 | self._check_type(ref, value) 64 | info.append({'ref': ref, 'operator': operator, 'value': value}) 65 | table, column = self.cube.model[ref].bind(self.cube) 66 | bindings.append(Binding(table, ref)) 67 | q = q.where(column.in_(value)) 68 | return info, q, bindings 69 | -------------------------------------------------------------------------------- /babbage/query/drilldowns.py: -------------------------------------------------------------------------------- 1 | from babbage.query.parser import Parser 2 | from babbage.model.binding import Binding 3 | from babbage.exc import QueryException 4 | 5 | 6 | class Drilldowns(Parser): 7 | """ Handle parser output for drilldowns. """ 8 | start = "drilldowns" 9 | 10 | def dimension(self, ast): 11 | refs = Parser.allrefs(self.cube.model.dimensions, 12 | self.cube.model.attributes) 13 | if ast not in refs: 14 | raise QueryException('Invalid drilldown: %r' % ast) 15 | if ast not in self.results: 16 | self.results.append(ast) 17 | 18 | def apply(self, q, bindings, drilldowns): 19 | """ Apply a set of grouping criteria and project them. """ 20 | info = [] 21 | for drilldown in self.parse(drilldowns): 22 | for attribute in self.cube.model.match(drilldown): 23 | info.append(attribute.ref) 24 | table, column = attribute.bind(self.cube) 25 | bindings.append(Binding(table, attribute.ref)) 26 | q = q.column(column) 27 | q = q.group_by(column) 28 | return info, q, bindings 29 | -------------------------------------------------------------------------------- /babbage/query/fields.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import func 2 | 3 | from babbage.query.parser import Parser 4 | from babbage.model.binding import Binding 5 | from babbage.exc import QueryException 6 | 7 | 8 | class Fields(Parser): 9 | """ Handle parser output for field specifications. """ 10 | start = "fields" 11 | 12 | def field(self, ast): 13 | refs = Parser.allrefs(self.cube.model.measures, 14 | self.cube.model.dimensions, 15 | self.cube.model.attributes) 16 | 17 | if ast not in refs: 18 | raise QueryException('Invalid field: %r' % ast) 19 | self.results.append(ast) 20 | 21 | def apply(self, q, bindings, fields, distinct=False): 22 | """ Define a set of fields to return for a non-aggregated query. """ 23 | info = [] 24 | 25 | group_by = None 26 | 27 | for field in self.parse(fields): 28 | for concept in self.cube.model.match(field): 29 | info.append(concept.ref) 30 | table, column = concept.bind(self.cube) 31 | bindings.append(Binding(table, concept.ref)) 32 | if distinct: 33 | if group_by is None: 34 | q = q.group_by(column) 35 | group_by = column 36 | else: 37 | min_column = func.max(column) 38 | min_column = min_column.label(column.name) 39 | column = min_column 40 | q = q.column(column) 41 | 42 | if not len(self.results): 43 | # If no fields are requested, return all available fields. 44 | for concept in list(self.cube.model.attributes) + \ 45 | list(self.cube.model.measures): 46 | info.append(concept.ref) 47 | table, column = concept.bind(self.cube) 48 | bindings.append(Binding(table, concept.ref)) 49 | q = q.column(column) 50 | 51 | return info, q, bindings 52 | -------------------------------------------------------------------------------- /babbage/query/ordering.py: -------------------------------------------------------------------------------- 1 | import six 2 | 3 | from babbage.query.parser import Parser 4 | from babbage.model.binding import Binding 5 | from babbage.exc import QueryException 6 | 7 | from sqlalchemy.sql.expression import asc, desc 8 | 9 | 10 | class Ordering(Parser): 11 | """ Handle parser output for sorting specifications, a tuple of a ref 12 | and a direction (which is 'asc' if unspecified). """ 13 | start = "ordering" 14 | 15 | def order(self, ast): 16 | if isinstance(ast, six.string_types): 17 | ref, direction = ast, 'asc' 18 | else: 19 | ref, direction = ast[0], ast[2] 20 | if ref not in self.cube.model: 21 | raise QueryException('Invalid sorting criterion: %r' % ast) 22 | self.results.append((ref, direction)) 23 | 24 | def apply(self, q, bindings, ordering, distinct=None): 25 | """ Sort on a set of field specifications of the type (ref, direction) 26 | in order of the submitted list. """ 27 | info = [] 28 | for (ref, direction) in self.parse(ordering): 29 | info.append((ref, direction)) 30 | table, column = self.cube.model[ref].bind(self.cube) 31 | if distinct is not None and distinct != ref: 32 | column = asc(ref) if direction == 'asc' else desc(ref) 33 | else: 34 | column = column.label(column.name) 35 | column = column.asc() if direction == 'asc' else column.desc() 36 | bindings.append(Binding(table, ref)) 37 | if self.cube.is_postgresql: 38 | column = column.nullslast() 39 | q = q.order_by(column) 40 | 41 | if not len(self.results): 42 | for column in q.columns: 43 | column = column.asc() 44 | if self.cube.is_postgresql: 45 | column = column.nullslast() 46 | q = q.order_by(column) 47 | return info, q, bindings 48 | -------------------------------------------------------------------------------- /babbage/query/pagination.py: -------------------------------------------------------------------------------- 1 | from babbage.query.parser import Parser 2 | from babbage.util import parse_int 3 | 4 | 5 | class Pagination(Parser): 6 | """ Handle pagination of results. Not actually using a parser. """ 7 | 8 | def apply(self, q, page, page_size, page_max=None, page_default=100): 9 | page_size = parse_int(page_size) 10 | if page_size is None: 11 | page_size = page_default 12 | if page_max is None: 13 | page_max = 10000 14 | page = max(1, parse_int(page, 0)) 15 | limit = max(0, min(page_max, page_size)) 16 | q = q.limit(limit) 17 | offset = (page - 1) * limit 18 | if offset > 0: 19 | q = q.offset(offset) 20 | return {'page': page, 'page_size': limit}, q 21 | -------------------------------------------------------------------------------- /babbage/query/parser.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | 4 | import grako 5 | import six 6 | import dateutil.parser 7 | from grako.exceptions import GrakoException 8 | 9 | from babbage.exc import QueryException 10 | from babbage.util import SCHEMA_PATH 11 | 12 | 13 | with open(os.path.join(SCHEMA_PATH, 'parser.ebnf'), 'rb') as fh: 14 | grammar = fh.read().decode('utf8') 15 | model = grako.genmodel("all", grammar) 16 | 17 | 18 | class Parser(object): 19 | """ Type casting for the basic primitives of the parser, e.g. strings, 20 | ints and dates. """ 21 | 22 | def __init__(self, cube): 23 | self.results = [] 24 | self.cube = cube 25 | self.bindings = [] 26 | 27 | def string_value(self, ast): 28 | text = ast[0] 29 | if text.startswith('"') and text.endswith('"'): 30 | return json.loads(text) 31 | return text 32 | 33 | def string_set(self, ast): 34 | return map(self.string_value, ast) 35 | 36 | def int_value(self, ast): 37 | return int(ast) 38 | 39 | def int_set(self, ast): 40 | return map(self.int_value, ast) 41 | 42 | def date_value(self, ast): 43 | return dateutil.parser.parse(ast).date() 44 | 45 | def date_set(self, ast): 46 | return map(self.date_value, ast) 47 | 48 | def parse(self, text): 49 | if isinstance(text, six.string_types): 50 | try: 51 | model.parse(text, start=self.start, semantics=self) 52 | return self.results 53 | except GrakoException as ge: 54 | raise QueryException(ge.message) 55 | elif text is None: 56 | text = [] 57 | return text 58 | 59 | @staticmethod 60 | def allrefs(*args): 61 | return [ 62 | ref 63 | for concept_list in args 64 | for concept in concept_list 65 | for ref in concept.refs 66 | ] 67 | -------------------------------------------------------------------------------- /babbage/schema/model.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-04/schema#", 3 | "title": "Babbage Model", 4 | "type": "object", 5 | "format": "valid_hierarchies", 6 | 7 | "properties": { 8 | "fact_table": {"$ref": "#/definitions/table"}, 9 | "measures": {"$ref": "#/definitions/measures"}, 10 | "dimensions": {"$ref": "#/definitions/dimensions"}, 11 | "hierarchies": {"$ref": "#/definitions/hierarchies"}, 12 | "joins": {"type": "array"} 13 | }, 14 | "required": ["measures", "dimensions", "fact_table"], 15 | "definitions": { 16 | "label": { 17 | "type": "string", 18 | "minLength": 2, 19 | "maxLength": 500 20 | }, 21 | "description": { 22 | "type": "string", 23 | "minLength": 0, 24 | "maxLength": 1000, 25 | "default": "" 26 | }, 27 | "column": { 28 | "type": "string", 29 | "minLength": 2, 30 | "maxLength": 250 31 | }, 32 | "table": { 33 | "type": "string", 34 | "minLength": 2, 35 | "maxLength": 250, 36 | "pattern": "^[a-zA-Z][a-zA-Z0-9_\\-]*[a-zA-Z0-9]$" 37 | }, 38 | "dimensions": { 39 | "type": "object", 40 | "patternProperties": { 41 | "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$": { 42 | "$ref": "#/definitions/dimension" 43 | } 44 | }, 45 | "additionalProperties": false 46 | }, 47 | "dimension": { 48 | "type": "object", 49 | "format": "attribute_exists", 50 | "properties": { 51 | "label": {"$ref": "#/definitions/label"}, 52 | "description": {"$ref": "#/definitions/description"}, 53 | "label_attribute": { 54 | "type": "string", 55 | "pattern": "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$" 56 | }, 57 | "key_attribute": { 58 | "type": "string", 59 | "pattern": "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$" 60 | }, 61 | "attributes": {"$ref": "#/definitions/attributes"} 62 | }, 63 | "required": ["label", "attributes", "key_attribute"] 64 | }, 65 | "attributes": { 66 | "type": "object", 67 | "minProperties": 1, 68 | "patternProperties": { 69 | "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$": { 70 | "$ref": "#/definitions/attribute" 71 | } 72 | }, 73 | "additionalProperties": false 74 | }, 75 | "attribute": { 76 | "type": "object", 77 | "properties": { 78 | "label": {"$ref": "#/definitions/label"}, 79 | "description": {"$ref": "#/definitions/description"}, 80 | "column": {"$ref": "#/definitions/column"} 81 | }, 82 | "required": ["label", "column"] 83 | }, 84 | "measures": { 85 | "type": "object", 86 | "minProperties": 1, 87 | "patternProperties": { 88 | "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$": { 89 | "$ref": "#/definitions/measure" 90 | } 91 | }, 92 | "additionalProperties": false 93 | }, 94 | "aggregates": { 95 | "type": "array", 96 | "items": { 97 | "type": "string", 98 | "enum": ["sum", "max", "min", "avg", "count"] 99 | }, 100 | "default": ["sum"], 101 | "minItems": 1, 102 | "uniqueItems": true 103 | }, 104 | "measure": { 105 | "type": "object", 106 | "properties": { 107 | "label": {"$ref": "#/definitions/label"}, 108 | "description": {"$ref": "#/definitions/description"}, 109 | "aggregates": {"$ref": "#/definitions/aggregates"}, 110 | "column": {"$ref": "#/definitions/column"} 111 | }, 112 | "required": ["label", "column"] 113 | }, 114 | "hierarchies": { 115 | "type": "object", 116 | "patternProperties": { 117 | "^[a-zA-Z][a-zA-Z0-9_]*[a-zA-Z0-9]$": { 118 | "$ref": "#/definitions/hierarchy" 119 | } 120 | }, 121 | "additionalProperties": false 122 | }, 123 | "hierarchy": { 124 | "type": "object", 125 | "properties": { 126 | "label": {"$ref": "#/definitions/label"}, 127 | "levels": {"$ref": "#/definitions/levels"} 128 | }, 129 | "required": ["levels"] 130 | }, 131 | "levels": { 132 | "type": "array", 133 | "items": {"$ref": "#/definitions/level"}, 134 | "minItems": 1, 135 | "uniqueItems": true 136 | }, 137 | "level": { 138 | "type": "string" 139 | } 140 | } 141 | } 142 | -------------------------------------------------------------------------------- /babbage/schema/parser.ebnf: -------------------------------------------------------------------------------- 1 | (* 2 | Description of the cuts/drilldown/ordering query structure. 3 | *) 4 | 5 | all = cuts | drilldowns | fields | ordering ; 6 | 7 | cuts = cut { '|' cut } ; 8 | cut = ref ':' value ; 9 | 10 | drilldowns = dimension { '|' dimension } ; 11 | dimension = ref ; 12 | 13 | fields = field { ',' field } ; 14 | field = ref ; 15 | 16 | aggregates = aggregate { '|' aggregate } ; 17 | aggregate = ref ; 18 | 19 | ordering = order { ',' order } ; 20 | order = ref [ ':' direction ] ; 21 | direction = 'asc' | 'desc' ; 22 | 23 | ref = ?/[A-Za-z0-9\._]*[A-Za-z0-9]/? ; 24 | 25 | value = date_set | int_set | string_set ; 26 | date_value = ?/[0-9]{4}-[0-9]{2}-[0-9]{2}/? ; 27 | date_set = ';'.{ >date_value } ; 28 | int_value = ?/[0-9]+/? !/[^0-9|;]+/ ; 29 | int_set = ';'.{ >int_value } ; 30 | string_value = escaped_string | {?/[^|]*/?} ; 31 | string_set = ';'.{ >string_value } ; 32 | escaped_string = ESCAPED_STRING ; 33 | ESCAPED_STRING = '"' @:{?/[^"\\\\]*/?|ESC} '"' ; 34 | ESC = ?/\\\\['"\\\\nrtbfv]/? | ?/\\\\u[a-fA-F0-9]{4}/? ; 35 | -------------------------------------------------------------------------------- /babbage/util.py: -------------------------------------------------------------------------------- 1 | import os 2 | import six 3 | 4 | SCHEMA_PATH = os.path.join(os.path.dirname(__file__), 'schema') 5 | 6 | 7 | def parse_int(text, fallback=None): 8 | """ Try to extract an integer from a string, return the fallback if that's 9 | not possible. """ 10 | try: 11 | if isinstance(text, six.integer_types): 12 | return text 13 | elif isinstance(text, six.string_types): 14 | return int(text) 15 | else: 16 | return fallback 17 | except ValueError: 18 | return fallback 19 | -------------------------------------------------------------------------------- /babbage/validation.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | from jsonschema import Draft4Validator, FormatChecker 4 | 5 | from babbage.util import SCHEMA_PATH 6 | 7 | checker = FormatChecker() 8 | 9 | 10 | @checker.checks('attribute_exists') 11 | def check_attribute_exists(instance): 12 | """ Additional check for the dimension model, to ensure that attributes 13 | given as the key and label attribute on the dimension exist. """ 14 | attributes = instance.get('attributes', {}).keys() 15 | if instance.get('key_attribute') not in attributes: 16 | return False 17 | label_attr = instance.get('label_attribute') 18 | if label_attr and label_attr not in attributes: 19 | return False 20 | return True 21 | 22 | 23 | @checker.checks('valid_hierarchies') 24 | def check_valid_hierarchies(instance): 25 | """ Additional check for the hierarchies model, to ensure that levels 26 | given are pointing to actual dimensions """ 27 | hierarchies = instance.get('hierarchies', {}).values() 28 | dimensions = set(instance.get('dimensions', {}).keys()) 29 | all_levels = set() 30 | for hierarcy in hierarchies: 31 | levels = set(hierarcy.get('levels', [])) 32 | if len(all_levels.intersection(levels)) > 0: 33 | # Dimension appears in two different hierarchies 34 | return False 35 | all_levels = all_levels.union(levels) 36 | if not dimensions.issuperset(levels): 37 | # Level which is not in a dimension 38 | return False 39 | return True 40 | 41 | 42 | def load_validator(name): 43 | """ Load the JSON Schema Draft 4 validator with the given name from the 44 | local schema directory. """ 45 | with open(os.path.join(SCHEMA_PATH, name)) as fh: 46 | schema = json.load(fh) 47 | Draft4Validator.check_schema(schema) 48 | return Draft4Validator(schema, format_checker=checker) 49 | 50 | 51 | def validate_model(model): 52 | validator = load_validator('model.json') 53 | validator.validate(model) 54 | -------------------------------------------------------------------------------- /babbage/version.py: -------------------------------------------------------------------------------- 1 | __version__ = '0.4.0' 2 | -------------------------------------------------------------------------------- /pylama.ini: -------------------------------------------------------------------------------- 1 | [pylama] 2 | linters = pyflakes,mccabe,pep8 3 | ignore = E128,E301 4 | 5 | [pylama:pep8] 6 | max_line_length = 120 7 | 8 | [pylama:mccabe] 9 | complexity = 36 10 | 11 | [pylama:*/__init__.py] 12 | ignore = W0611 13 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [tool:pytest] 2 | norecursedirs = .* *.egg build dist env 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | exec(open('babbage/version.py', 'r').read()) 4 | 5 | with open('README.md') as f: 6 | long_description = f.read() 7 | 8 | setup( 9 | name='babbage', 10 | version=__version__, 11 | description="A light-weight analytical engine for OLAP processing", 12 | long_description=long_description, 13 | long_description_content_type="text/markdown", 14 | classifiers=[ 15 | "Development Status :: 3 - Alpha", 16 | "Intended Audience :: Developers", 17 | "License :: OSI Approved :: MIT License", 18 | "Operating System :: OS Independent", 19 | 'Programming Language :: Python :: 2.7', 20 | 'Programming Language :: Python :: 3.4', 21 | 'Programming Language :: Python :: 3.5', 22 | 'Programming Language :: Python :: 3.6' 23 | ], 24 | keywords='sql sqlalchemy olap cubes analytics', 25 | author='Friedrich Lindenberg', 26 | author_email='friedrich@pudo.org', 27 | url='http://github.com/openspending/babbage', 28 | license='MIT', 29 | packages=find_packages(exclude=['ez_setup', 'examples', 'tests']), 30 | namespace_packages=[], 31 | include_package_data=True, 32 | package_data={ 33 | '': ['babbage/schema/model.json', 'babbage/schema/parser.ebnf'] 34 | }, 35 | zip_safe=False, 36 | install_requires=[ 37 | 'normality >= 0.2.2', 38 | 'PyYAML >= 3.10', 39 | 'six >= 1.7.3', 40 | 'flask >= 0.10.1', 41 | 'jsonschema >= 2.5.1', 42 | 'sqlalchemy >= 1.0', 43 | 'psycopg2 >= 2.6', 44 | 'grako == 3.10.1' # Versions > 3.10.1 break our tests 45 | ], 46 | tests_require=[ 47 | 'tox' 48 | ], 49 | test_suite='tests', 50 | entry_points={} 51 | ) 52 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openspending/babbage/9416105fd18dda13b06aaaeec0ce7abdd13d8453/tests/__init__.py -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import dateutil.parser 4 | import pytest 5 | import flask 6 | import unicodecsv 7 | import sqlalchemy 8 | 9 | import babbage.api 10 | import babbage.model 11 | import babbage.manager 12 | 13 | FIXTURE_PATH = os.path.join(os.path.dirname(__file__), 'fixtures') 14 | 15 | 16 | @pytest.fixture() 17 | def app(): 18 | app = flask.Flask('test') 19 | app.register_blueprint(babbage.api.blueprint, url_prefix='/bbg') 20 | app.config['DEBUG'] = True 21 | app.config['TESTING'] = True 22 | app.config['PRESERVE_CONTEXT_ON_EXCEPTION'] = True 23 | return app 24 | 25 | 26 | @pytest.fixture 27 | def fixtures_cube_manager(sqla_engine): 28 | path = os.path.join(FIXTURE_PATH, 'models') 29 | return babbage.manager.JSONCubeManager(sqla_engine, path) 30 | 31 | 32 | @pytest.fixture 33 | def load_api_fixtures(app, fixtures_cube_manager): 34 | return babbage.api.configure_api(app, fixtures_cube_manager) 35 | 36 | 37 | @pytest.fixture 38 | def simple_model_data(): 39 | return load_json_fixture('models/simple_model.json') 40 | 41 | 42 | @pytest.fixture 43 | def simple_model(simple_model_data): 44 | return babbage.model.Model(simple_model_data) 45 | 46 | 47 | @pytest.fixture() 48 | def cra_model(): 49 | return load_json_fixture('models/cra.json') 50 | 51 | 52 | @pytest.fixture() 53 | def cra_table(sqla_engine): 54 | return load_csv(sqla_engine, 'cra.csv') 55 | 56 | 57 | @pytest.fixture() 58 | def cap_or_cur_table(sqla_engine): 59 | return load_csv(sqla_engine, 'cap_or_cur.csv') 60 | 61 | @pytest.fixture() 62 | def cofog1_table(sqla_engine): 63 | return load_csv(sqla_engine, 'cofog1.csv') 64 | 65 | 66 | @pytest.fixture() 67 | def load_fixtures(cra_table, cap_or_cur_table, cofog1_table): 68 | pass 69 | 70 | 71 | @pytest.fixture() 72 | def sqla_engine(): 73 | DATABASE_URI = os.environ.get('BABBAGE_TEST_DB') 74 | assert DATABASE_URI, 'Set the envvar BABBAGE_TEST_DB to a PostgreSQL URI' 75 | engine = sqlalchemy.create_engine(DATABASE_URI) 76 | 77 | try: 78 | yield engine 79 | finally: 80 | meta = sqlalchemy.MetaData(bind=engine, reflect=True) 81 | meta.drop_all() 82 | 83 | 84 | def load_json_fixture(name): 85 | path = os.path.join(FIXTURE_PATH, name) 86 | with open(path, 'r') as fh: 87 | return json.load(fh) 88 | 89 | 90 | def load_csv(sqla_engine, file_name, table_name=None): 91 | table_name = table_name or os.path.basename(file_name).split('.')[0] 92 | path = os.path.join(FIXTURE_PATH, file_name) 93 | table = None 94 | with open(path, 'rb') as fh: 95 | for i, row in enumerate(unicodecsv.DictReader(fh)): 96 | if table is None: 97 | table = _create_table(sqla_engine, table_name, row.keys()) 98 | row['_id'] = str(i) 99 | stmt = table.insert(_convert_row(row)) 100 | sqla_engine.execute(stmt) 101 | return table 102 | 103 | 104 | def _create_table(engine, table_name, columns): 105 | meta = sqlalchemy.MetaData() 106 | meta.bind = engine 107 | 108 | if engine.has_table(table_name): 109 | table = sqlalchemy.schema.Table(table_name, meta, autoload=True) 110 | table.drop() 111 | 112 | table = sqlalchemy.schema.Table(table_name, meta) 113 | id_col = sqlalchemy.schema.Column('_id', sqlalchemy.types.Integer, primary_key=True) 114 | table.append_column(id_col) 115 | for (_, name, typ) in sorted(_column_specs(columns)): 116 | col = sqlalchemy.schema.Column(name, typ, primary_key=name in ('cap_or_cur', 'cofog1_name')) 117 | table.append_column(col) 118 | 119 | table.create(engine) 120 | return table 121 | 122 | 123 | def _column_specs(columns): 124 | TYPES = { 125 | 'string': sqlalchemy.types.Unicode, 126 | 'integer': sqlalchemy.types.Integer, 127 | 'bool': sqlalchemy.types.Boolean, 128 | 'float': sqlalchemy.types.Float, 129 | 'decimal': sqlalchemy.types.Float, 130 | 'date': sqlalchemy.types.Date 131 | } 132 | for column in columns: 133 | spec = column.rsplit(':', 1) 134 | typ = 'string' if len(spec) == 1 else spec[1] 135 | yield column, spec[0], TYPES[typ] 136 | 137 | 138 | def _convert_row(row): 139 | data = {} 140 | for (key, name, typ) in _column_specs(row.keys()): 141 | value = row.get(key) 142 | if not len(value.strip()): 143 | value = None 144 | elif typ == sqlalchemy.types.Integer: 145 | value = int(value) 146 | elif typ == sqlalchemy.types.Float: 147 | value = float(value) 148 | elif typ == sqlalchemy.types.Boolean: 149 | value = value.strip().lower() 150 | value = value in ['1', 'true', 'yes'] 151 | elif typ == sqlalchemy.types.Date: 152 | value = dateutil.parser.parse(value).date() 153 | data[name] = value 154 | return data 155 | -------------------------------------------------------------------------------- /tests/fixtures/cap_or_cur.csv: -------------------------------------------------------------------------------- 1 | code,label 2 | CAP,"Capital Expenditure" 3 | CUR,"Current Expenditure" 4 | -------------------------------------------------------------------------------- /tests/fixtures/cofog1.csv: -------------------------------------------------------------------------------- 1 | cofog1_change_date,cofog1_description,cofog1_label,cofog1_level,id,cofog1_taxonomy 2 | ,"Government outlays on social protection include expenditures on services and transfers provided to individual persons and households and expenditures on services provided on a collective basis. Expenditures on individual services and transfers are allocated to groups (10.1) through (10.7); expenditures on collective services are assigned to groups (10.8) and (10.9). 3 | Collective social protection services are concerned with matters such as formulation and administration of government policy; formulation and enforcement of legislation and standards for providing social protection; and applied research and experimental development into social protection affairs and services. 4 | The social protection functions and their definitions are based on the 1996 European System of integrated Social Protection Statistics (ESSPROS) of the Statistical Office of the European Communities (Eurostat). 5 | In ESSPROS, social protection includes health care, but this division does not include health care. Health care is covered by Division 07. Hence, medical goods and services provided to persons who receive the cash benefits and benefits in kind specified in groups (10.1) through (10.7) are classified under (07.1), (07.2) or (07.3) as appropriate.",Social protection,1,10,cofog 6 | ,,Economic affairs,1,4,cofog 7 | ,,Housing and community amenities,1,6,cofog 8 | ,,Public order and safety,1,3,cofog 9 | -------------------------------------------------------------------------------- /tests/fixtures/cra.csv: -------------------------------------------------------------------------------- 1 | amount:integer,cap_or_cur,cofog1_name,cofog2_change_date,cofog2_description,cofog2_label,cofog2_level,cofog2_name,cofog2_taxonomy,cofog3_change_date,cofog3_description,cofog3_label,cofog3_level,cofog3_name,cofog3_taxonomy,currency,from_description,from_label,from_name,name,pog_label,pog_name,pog_taxonomy,population2006:integer,region,time_from_year:integer,to_description,to_label,to_name 2 | 12100000,CUR,4,,,"General economic, commercial and labour affairs",2,4.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r1,P37 S121211 ADMIN COSTS OF MEASURES TO HELP UNEMPL PEOPLE MOVE FROM WELFARE T...,P37 S121211,pog,5116900,SCOTLAND,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 3 | 12100000,CUR,4,,,"General economic, commercial and labour affairs",2,4.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r2,P37 S121211 ADMIN COSTS OF MEASURES TO HELP UNEMPL PEOPLE MOVE FROM WELFARE T...,P37 S121211,pog,5116900,SCOTLAND,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 4 | 12100000,CUR,4,,,"General economic, commercial and labour affairs",2,4.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r3,P37 S121211 ADMIN COSTS OF MEASURES TO HELP UNEMPL PEOPLE MOVE FROM WELFARE T...,P37 S121211,pog,5116900,SCOTLAND,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 5 | 27500000,CUR,10,,,Sickness and disability,2,10.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r4,P37 S120812 Independent Living Fund,P37 S120812,pog,5366700,ENGLAND_West Midlands,2007,A dummy entity to be the recipient of final government spending,Society (the General Public),society 6 | 30400000,CUR,10,,,Sickness and disability,2,10.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r5,P37 S120812 Independent Living Fund,P37 S120812,pog,5366700,ENGLAND_West Midlands,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 7 | 31100000,CUR,10,,,Sickness and disability,2,10.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r6,P37 S120812 Independent Living Fund,P37 S120812,pog,5366700,ENGLAND_West Midlands,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 8 | 31800000,CUR,10,,,Sickness and disability,2,10.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r7,P37 S120812 Independent Living Fund,P37 S120812,pog,5366700,ENGLAND_West Midlands,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 9 | -100000,CUR,4,,,"General economic, commercial and labour affairs",2,4.1,cofog,,,,,,,GBP,,Department for Work and Pensions,Dept032,cra-r8,P37 S121216 ES Administration,P37 S121216,pog,5142400,ENGLAND_Yorkshire and The Humber,2003,A dummy entity to be the recipient of final government spending,Society (the General Public),society 10 | 2100000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 11 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 12 | - construction or operation of non-enterprise-type railway transport systems and facilities; 13 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 14 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 15 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 16 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r9,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2005,A dummy entity to be the recipient of final government spending,Society (the General Public),society 17 | 1600000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 18 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 19 | - construction or operation of non-enterprise-type railway transport systems and facilities; 20 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 21 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 22 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 23 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r10,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2006,A dummy entity to be the recipient of final government spending,Society (the General Public),society 24 | 2900000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 25 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 26 | - construction or operation of non-enterprise-type railway transport systems and facilities; 27 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 28 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 29 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 30 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r11,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2007,A dummy entity to be the recipient of final government spending,Society (the General Public),society 31 | 1300000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 32 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 33 | - construction or operation of non-enterprise-type railway transport systems and facilities; 34 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 35 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 36 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 37 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r12,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 38 | 800000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 39 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 40 | - construction or operation of non-enterprise-type railway transport systems and facilities; 41 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 42 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 43 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 44 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r13,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 45 | 500000,CUR,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 46 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 47 | - construction or operation of non-enterprise-type railway transport systems and facilities; 48 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 49 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 50 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 51 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r14,P07 S500003 RPROJ-Channel Tunnel Rail Link (CTRL),P07 S500003,pog,7512400,ENGLAND_London,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 52 | 22400000,CAP,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 53 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 54 | - construction or operation of non-enterprise-type railway transport systems and facilities; 55 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 56 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 57 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 58 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r15,P07 S500207 Government Owned Rolling Stock,P07 500207,pog,5142400,ENGLAND_Yorkshire and The Humber,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 59 | -22400000,CAP,4,,,Transport,2,4.5,cofog,,"- Administration of affairs and services concerning operation, use, construction or maintenance of railway transport systems and facilities (railway roadbeds, terminals, tunnels, bridges, embankments, cuttings, etc.); 60 | - supervision and regulation of railway users (rolling stock condition, roadbed stability, passenger safety, security of freight, etc.), of railway transport system operations (granting of franchises, approval of freight tariffs and passenger fares and of hours and frequency of service, etc.) and of railway construction and maintenance; 61 | - construction or operation of non-enterprise-type railway transport systems and facilities; 62 | - production and dissemination of general information, technical documentation and statistics on railway transport system operations and on railway construction activities; 63 | - grants, loans or subsidies to support the operation, construction, maintenance or upgrading of railway transport systems and facilities. 64 | Includes: long-line and interurban railway transport systems, urban rapid transit railway transport systems and street railway transport systems; acquisition and maintenance of rolling stock. 65 | Excludes: grants, loans and subsidies to rolling stock manufacturers (04.4.2); construction of noise embankments, hedges and other anti-noise facilities including the resurfacing of sections of railways with noise reducing surfaces (05.3.0).",Railway transport (CS),3,04.5.3,cofog,GBP,,Department for Transport,Dept004,cra-r16,P07 S500207 Government Owned Rolling Stock,P07 500207,pog,5142400,ENGLAND_Yorkshire and The Humber,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 66 | 100000,CAP,3,,,Law courts,2,3.3,cofog,,,,,,,GBP,,Ministry of Justice,Dept047,cra-r17,P13 S091105 CICA,P13 S091105,pog,5124100,ENGLAND_South West,2003,A dummy entity to be the recipient of final government spending,Society (the General Public),society 67 | 100000,CAP,3,,,Law courts,2,3.3,cofog,,,,,,,GBP,,Ministry of Justice,Dept047,cra-r18,P13 S091105 CICA,P13 S091105,pog,5124100,ENGLAND_South West,2007,A dummy entity to be the recipient of final government spending,Society (the General Public),society 68 | 100000,CAP,3,,,Law courts,2,3.3,cofog,,,,,,,GBP,,Ministry of Justice,Dept047,cra-r19,P13 S091105 CICA,P13 S091105,pog,5124100,ENGLAND_South West,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 69 | 100000,CAP,3,,,Law courts,2,3.3,cofog,,,,,,,GBP,,Ministry of Justice,Dept047,cra-r20,P13 S091105 CICA,P13 S091105,pog,5124100,ENGLAND_South West,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 70 | 100000,CAP,3,,,Law courts,2,3.3,cofog,,,,,,,GBP,,Ministry of Justice,Dept047,cra-r21,P13 S091105 CICA,P13 S091105,pog,5124100,ENGLAND_South West,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 71 | 1500000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r22,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2004,A dummy entity to be the recipient of final government spending,Society (the General Public),society 72 | 6900000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r23,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2005,A dummy entity to be the recipient of final government spending,Society (the General Public),society 73 | 100000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r24,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2006,A dummy entity to be the recipient of final government spending,Society (the General Public),society 74 | 300000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r25,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2007,A dummy entity to be the recipient of final government spending,Society (the General Public),society 75 | 100000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r26,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 76 | 15800000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r27,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 77 | 46000000,CUR,10,,,Family and children,2,10.4,cofog,,,,,,,GBP,,"Department for Children, Schools and Families",Dept022,cra-r28,P01 S100417 Children and Family - Misc Programmes,P01 S100417,pog,5124100,ENGLAND_South West,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 78 | -167700000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r29,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2003,A dummy entity to be the recipient of final government spending,Society (the General Public),society 79 | -118300000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r30,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2004,A dummy entity to be the recipient of final government spending,Society (the General Public),society 80 | -94700000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r31,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2005,A dummy entity to be the recipient of final government spending,Society (the General Public),society 81 | -84100000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r32,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2006,A dummy entity to be the recipient of final government spending,Society (the General Public),society 82 | -51100000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r33,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2007,A dummy entity to be the recipient of final government spending,Society (the General Public),society 83 | -20600000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r34,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2008,A dummy entity to be the recipient of final government spending,Society (the General Public),society 84 | -25000000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r35,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2009,A dummy entity to be the recipient of final government spending,Society (the General Public),society 85 | -47400000,CAP,6,,,,,,,,,,,,,GBP,,ENG_HRA,999,cra-r36,LA Dummy sprog 6. Housing and community amenities,999999,pog,5124100,ENGLAND_South West,2010,A dummy entity to be the recipient of final government spending,Society (the General Public),society 86 | -------------------------------------------------------------------------------- /tests/fixtures/models/cra.json: -------------------------------------------------------------------------------- 1 | { 2 | "fact_table": "cra", 3 | "dimensions": { 4 | "cap_or_cur": { 5 | "attributes": { 6 | "code": { 7 | "column": "cap_or_cur.code", 8 | "label": "Label", 9 | "type": "string" 10 | }, 11 | "label": { 12 | "column": "cap_or_cur.label", 13 | "label": "Label", 14 | "type": "string" 15 | } 16 | }, 17 | "key_attribute": "code", 18 | "description": "Central government, local government or public corporation", 19 | "label": "CG, LG or PC", 20 | "join_column": "cap_or_cur" 21 | }, 22 | "cofog1": { 23 | "attributes": { 24 | "change_date": { 25 | "column": "cofog1.cofog1_change_date", 26 | "label": "Change date", 27 | "type": "date" 28 | }, 29 | "description": { 30 | "column": "cofog1.cofog1_description", 31 | "label": "Description", 32 | "type": "string" 33 | }, 34 | "label": { 35 | "column": "cofog1.cofog1_label", 36 | "label": "Label", 37 | "type": "string" 38 | }, 39 | "name": { 40 | "column": "cofog1.id", 41 | "label": "Name", 42 | "type": "string" 43 | } 44 | }, 45 | "key_attribute": "name", 46 | "label_attribute": "label", 47 | "description": "Classification Of Function Of Government, level 1", 48 | "label": "COFOG level 1", 49 | "join_column": ["cofog1_name", "id"] 50 | }, 51 | "cofog2": { 52 | "attributes": { 53 | "change_date": { 54 | "column": "cofog2_change_date", 55 | "label": "Change date", 56 | "type": "date" 57 | }, 58 | "description": { 59 | "column": "cofog2_description", 60 | "label": "Description" 61 | }, 62 | "label": { 63 | "column": "cofog2_label", 64 | "label": "Label" 65 | }, 66 | "name": { 67 | "column": "cofog2_name", 68 | "label": "Name" 69 | } 70 | }, 71 | "key_attribute": "name", 72 | "label_attribute": "label", 73 | "description": "Classification Of Function Of Government, level 2", 74 | "label": "COFOG level 2" 75 | }, 76 | "cofog3": { 77 | "attributes": { 78 | "change_date": { 79 | "column": "cofog3_change_date", 80 | "label": "Date", 81 | "type": "date" 82 | }, 83 | "description": { 84 | "column": "cofog3_description", 85 | "label": "Description", 86 | "type": "string" 87 | }, 88 | "label": { 89 | "column": "cofog3_label", 90 | "label": "Label", 91 | "type": "string" 92 | }, 93 | "level": { 94 | "column": "cofog3_level", 95 | "label": "Level", 96 | "type": "string" 97 | }, 98 | "name": { 99 | "column": "cofog3_name", 100 | "label": "Name", 101 | "type": "integer" 102 | } 103 | }, 104 | "key_attribute": "name", 105 | "label_attribute": "label", 106 | "description": "Classification Of Function Of Government, level 3", 107 | "label": "COFOG level 3" 108 | }, 109 | "currency": { 110 | "attributes": { 111 | "currency": { 112 | "column": "currency", 113 | "label": "Currency", 114 | "type": "string" 115 | } 116 | }, 117 | "key_attribute": "currency", 118 | "label": "Currency" 119 | }, 120 | "from": { 121 | "attributes": { 122 | "description": { 123 | "column": "from_description", 124 | "label": "Description", 125 | "type": "string" 126 | }, 127 | "label": { 128 | "column": "from_label", 129 | "label": "Label", 130 | "type": "string" 131 | }, 132 | "name": { 133 | "column": "from_name", 134 | "label": "Name", 135 | "type": "string" 136 | } 137 | }, 138 | "key_attribute": "name", 139 | "label_attribute": "label", 140 | "description": "The entity that the money was paid from.", 141 | "label": "Paid by" 142 | }, 143 | "name": { 144 | "attributes": { 145 | "name": { 146 | "column": "name", 147 | "label": "Name", 148 | "type": "string" 149 | } 150 | }, 151 | "key_attribute": "name", 152 | "label": "Name" 153 | }, 154 | "pog": { 155 | "attributes": { 156 | "label": { 157 | "column": "pog_label", 158 | "label": "Label", 159 | "type": "string" 160 | }, 161 | "name": { 162 | "column": "pog_name", 163 | "label": "Name", 164 | "type": "string" 165 | } 166 | }, 167 | "key_attribute": "name", 168 | "label_attribute": "label", 169 | "label": "Programme Object Group" 170 | }, 171 | "population2006": { 172 | "attributes": { 173 | "population2006": { 174 | "column": "population2006", 175 | "label": "Count", 176 | "type": "integer" 177 | } 178 | }, 179 | "key_attribute": "population2006", 180 | "label": "Population in 2006" 181 | }, 182 | "region": { 183 | "attributes": { 184 | "region": { 185 | "column": "region", 186 | "label": "Label", 187 | "type": "string" 188 | } 189 | }, 190 | "key_attribute": "region", 191 | "label": "Region" 192 | }, 193 | "time": { 194 | "attributes": { 195 | "year": { 196 | "column": "time_from_year", 197 | "label": "Year", 198 | "type": "string" 199 | } 200 | }, 201 | "key_attribute": "year", 202 | "description": "The accounting period in which the spending happened", 203 | "label": "Tax year" 204 | }, 205 | "to": { 206 | "attributes": { 207 | "description": { 208 | "column": "to_description", 209 | "label": "Description", 210 | "type": "string" 211 | }, 212 | "label": { 213 | "column": "to_label", 214 | "label": "Label", 215 | "type": "string" 216 | }, 217 | "name": { 218 | "column": "to_name", 219 | "label": "Name", 220 | "type": "string" 221 | } 222 | }, 223 | "key_attribute": "name", 224 | "label_attribute": "label", 225 | "description": "The entity that the money was paid to", 226 | "label": "Paid to" 227 | } 228 | }, 229 | "measures": { 230 | "amount": { 231 | "column": "amount", 232 | "label": "Amount", 233 | "type": "integer" 234 | }, 235 | "total": { 236 | "column": "amount", 237 | "label": "Total amount", 238 | "type": "integer" 239 | } 240 | } 241 | } 242 | -------------------------------------------------------------------------------- /tests/fixtures/models/mexico.json: -------------------------------------------------------------------------------- 1 | { 2 | "dimensions": { 3 | "CICLO": { 4 | "attributes": { 5 | "CICLO": { 6 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_3.CICLO", 7 | "label": "Ciclo", 8 | "type": "integer" 9 | } 10 | }, 11 | "join_column": [ 12 | "date_id", 13 | "_fdp__id_" 14 | ], 15 | "key_attribute": "CICLO", 16 | "label": "Ciclo" 17 | }, 18 | "GPO_FUNCIONAL": { 19 | "attributes": { 20 | "DESC_GPO_FUNCIONAL": { 21 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.DESC_GPO_FUNCIONAL", 22 | "label": "DESC_GPO_FUNCIONAL", 23 | "type": "string" 24 | }, 25 | "GPO_FUNCIONAL": { 26 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.GPO_FUNCIONAL", 27 | "label": "Grupo Funcional", 28 | "type": "string" 29 | } 30 | }, 31 | "join_column": [ 32 | "functional_classification_id", 33 | "_fdp__id_" 34 | ], 35 | "key_attribute": "GPO_FUNCIONAL", 36 | "label": "Grupo Funcional", 37 | "label_attribute": "DESC_GPO_FUNCIONAL" 38 | }, 39 | "ID_AI": { 40 | "attributes": { 41 | "DESC_AI": { 42 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.DESC_AI", 43 | "label": "DESC_AI", 44 | "type": "string" 45 | }, 46 | "ID_AI": { 47 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.ID_AI", 48 | "label": "Actividad Institucional", 49 | "type": "string" 50 | } 51 | }, 52 | "join_column": [ 53 | "functional_classification_id", 54 | "_fdp__id_" 55 | ], 56 | "key_attribute": "ID_AI", 57 | "label": "Actividad Institucional", 58 | "label_attribute": "DESC_AI" 59 | }, 60 | "ID_CAPITULO": { 61 | "attributes": { 62 | "DESC_CAPITULO": { 63 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.DESC_CAPITULO", 64 | "label": "DESC_CAPITULO", 65 | "type": "string" 66 | }, 67 | "ID_CAPITULO": { 68 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.ID_CAPITULO", 69 | "label": "Cap\u00edtulo", 70 | "type": "string" 71 | } 72 | }, 73 | "join_column": [ 74 | "economic_classification_id", 75 | "_fdp__id_" 76 | ], 77 | "key_attribute": "ID_CAPITULO", 78 | "label": "Cap\u00edtulo", 79 | "label_attribute": "DESC_CAPITULO" 80 | }, 81 | "ID_CLAVE_CARTERA": { 82 | "attributes": { 83 | "ID_CLAVE_CARTERA": { 84 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_2.ID_CLAVE_CARTERA", 85 | "label": "Clave de Cartera", 86 | "type": "string" 87 | } 88 | }, 89 | "join_column": [ 90 | "budget_line_id_id", 91 | "_fdp__id_" 92 | ], 93 | "key_attribute": "ID_CLAVE_CARTERA", 94 | "label": "Clave de Cartera" 95 | }, 96 | "ID_CONCEPTO": { 97 | "attributes": { 98 | "DESC_CONCEPTO": { 99 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.DESC_CONCEPTO", 100 | "label": "DESC_CONCEPTO", 101 | "type": "string" 102 | }, 103 | "ID_CONCEPTO": { 104 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.ID_CONCEPTO", 105 | "label": "Concepto", 106 | "type": "string" 107 | } 108 | }, 109 | "join_column": [ 110 | "economic_classification_id", 111 | "_fdp__id_" 112 | ], 113 | "key_attribute": "ID_CONCEPTO", 114 | "label": "Concepto", 115 | "label_attribute": "DESC_CONCEPTO" 116 | }, 117 | "ID_ENTIDAD_FEDERATIVA": { 118 | "attributes": { 119 | "ENTIDAD_FEDERATIVA": { 120 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_8.ENTIDAD_FEDERATIVA", 121 | "label": "ENTIDAD_FEDERATIVA", 122 | "type": "string" 123 | }, 124 | "ID_ENTIDAD_FEDERATIVA": { 125 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_8.ID_ENTIDAD_FEDERATIVA", 126 | "label": "Entidad Federativa", 127 | "type": "string" 128 | } 129 | }, 130 | "join_column": [ 131 | "geo_source_id", 132 | "_fdp__id_" 133 | ], 134 | "key_attribute": "ID_ENTIDAD_FEDERATIVA", 135 | "label": "Entidad Federativa", 136 | "label_attribute": "ENTIDAD_FEDERATIVA" 137 | }, 138 | "ID_FF": { 139 | "attributes": { 140 | "DESC_FF": { 141 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_6.DESC_FF", 142 | "label": "DESC_FF", 143 | "type": "string" 144 | }, 145 | "ID_FF": { 146 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_6.ID_FF", 147 | "label": "Fuente de Financiamiento", 148 | "type": "string" 149 | } 150 | }, 151 | "join_column": [ 152 | "fin_source_id", 153 | "_fdp__id_" 154 | ], 155 | "key_attribute": "ID_FF", 156 | "label": "Fuente de Financiamiento", 157 | "label_attribute": "DESC_FF" 158 | }, 159 | "ID_FUNCION": { 160 | "attributes": { 161 | "DESC_FUNCION": { 162 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.DESC_FUNCION", 163 | "label": "DESC_FUNCION", 164 | "type": "string" 165 | }, 166 | "ID_FUNCION": { 167 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.ID_FUNCION", 168 | "label": "Funci\u00f3n", 169 | "type": "string" 170 | } 171 | }, 172 | "join_column": [ 173 | "functional_classification_id", 174 | "_fdp__id_" 175 | ], 176 | "key_attribute": "ID_FUNCION", 177 | "label": "Funci\u00f3n", 178 | "label_attribute": "DESC_FUNCION" 179 | }, 180 | "ID_MODALIDAD": { 181 | "attributes": { 182 | "DESC_MODALIDAD": { 183 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_0.DESC_MODALIDAD", 184 | "label": "DESC_MODALIDAD", 185 | "type": "string" 186 | }, 187 | "ID_MODALIDAD": { 188 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_0.ID_MODALIDAD", 189 | "label": "Modalidad", 190 | "type": "string" 191 | } 192 | }, 193 | "join_column": [ 194 | "activity_id", 195 | "_fdp__id_" 196 | ], 197 | "key_attribute": "ID_MODALIDAD", 198 | "label": "Modalidad", 199 | "label_attribute": "DESC_MODALIDAD" 200 | }, 201 | "ID_PARTIDA_ESPECIFICA": { 202 | "attributes": { 203 | "DESC_PARTIDA_ESPECIFICA": { 204 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.DESC_PARTIDA_ESPECIFICA", 205 | "label": "DESC_PARTIDA_ESPECIFICA", 206 | "type": "string" 207 | }, 208 | "ID_PARTIDA_ESPECIFICA": { 209 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.ID_PARTIDA_ESPECIFICA", 210 | "label": "Partida Espec\u00edfica", 211 | "type": "string" 212 | } 213 | }, 214 | "join_column": [ 215 | "economic_classification_id", 216 | "_fdp__id_" 217 | ], 218 | "key_attribute": "ID_PARTIDA_ESPECIFICA", 219 | "label": "Partida Espec\u00edfica", 220 | "label_attribute": "DESC_PARTIDA_ESPECIFICA" 221 | }, 222 | "ID_PARTIDA_GENERICA": { 223 | "attributes": { 224 | "DESC_PARTIDA_GENERICA": { 225 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.DESC_PARTIDA_GENERICA", 226 | "label": "DESC_PARTIDA_GENERICA", 227 | "type": "string" 228 | }, 229 | "ID_PARTIDA_GENERICA": { 230 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_4.ID_PARTIDA_GENERICA", 231 | "label": "Partida Gen\u00e9rica", 232 | "type": "string" 233 | } 234 | }, 235 | "join_column": [ 236 | "economic_classification_id", 237 | "_fdp__id_" 238 | ], 239 | "key_attribute": "ID_PARTIDA_GENERICA", 240 | "label": "Partida Gen\u00e9rica", 241 | "label_attribute": "DESC_PARTIDA_GENERICA" 242 | }, 243 | "ID_PP": { 244 | "attributes": { 245 | "DESC_PP": { 246 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_0.DESC_PP", 247 | "label": "DESC_PP", 248 | "type": "string" 249 | }, 250 | "ID_PP": { 251 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_0.ID_PP", 252 | "label": "Programa Presupuestario", 253 | "type": "string" 254 | } 255 | }, 256 | "join_column": [ 257 | "activity_id", 258 | "_fdp__id_" 259 | ], 260 | "key_attribute": "ID_PP", 261 | "label": "Programa Presupuestario", 262 | "label_attribute": "DESC_PP" 263 | }, 264 | "ID_RAMO": { 265 | "attributes": { 266 | "DESC_RAMO": { 267 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_1.DESC_RAMO", 268 | "label": "DESC_RAMO", 269 | "type": "string" 270 | }, 271 | "ID_RAMO": { 272 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_1.ID_RAMO", 273 | "label": "Ramo", 274 | "type": "string" 275 | } 276 | }, 277 | "join_column": [ 278 | "administrative_classification_id", 279 | "_fdp__id_" 280 | ], 281 | "key_attribute": "ID_RAMO", 282 | "label": "Ramo", 283 | "label_attribute": "DESC_RAMO" 284 | }, 285 | "ID_SUBFUNCION": { 286 | "attributes": { 287 | "DESC_SUBFUNCION": { 288 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.DESC_SUBFUNCION", 289 | "label": "DESC_SUBFUNCION", 290 | "type": "string" 291 | }, 292 | "ID_SUBFUNCION": { 293 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_7.ID_SUBFUNCION", 294 | "label": "Subfunci\u00f3n", 295 | "type": "string" 296 | } 297 | }, 298 | "join_column": [ 299 | "functional_classification_id", 300 | "_fdp__id_" 301 | ], 302 | "key_attribute": "ID_SUBFUNCION", 303 | "label": "Subfunci\u00f3n", 304 | "label_attribute": "DESC_SUBFUNCION" 305 | }, 306 | "ID_TIPOGASTO": { 307 | "attributes": { 308 | "DESC_TIPOGASTO": { 309 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_5.DESC_TIPOGASTO", 310 | "label": "DESC_TIPOGASTO", 311 | "type": "string" 312 | }, 313 | "ID_TIPOGASTO": { 314 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_5.ID_TIPOGASTO", 315 | "label": "Tipo de Gasto", 316 | "type": "string" 317 | } 318 | }, 319 | "join_column": [ 320 | "expenditure_type_id", 321 | "_fdp__id_" 322 | ], 323 | "key_attribute": "ID_TIPOGASTO", 324 | "label": "Tipo de Gasto", 325 | "label_attribute": "DESC_TIPOGASTO" 326 | }, 327 | "ID_UR": { 328 | "attributes": { 329 | "DESC_UR": { 330 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_1.DESC_UR", 331 | "label": "DESC_UR", 332 | "type": "string" 333 | }, 334 | "ID_UR": { 335 | "column": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd_1.ID_UR", 336 | "label": "Unidad Responsable", 337 | "type": "string" 338 | } 339 | }, 340 | "join_column": [ 341 | "administrative_classification_id", 342 | "_fdp__id_" 343 | ], 344 | "key_attribute": "ID_UR", 345 | "label": "Unidad Responsable", 346 | "label_attribute": "DESC_UR" 347 | } 348 | }, 349 | "fact_table": "667df60aa07c34260eae9b55b27787128e5d49d57370f3fd", 350 | "hierarchies": { 351 | "activity": { 352 | "label": "activity", 353 | "levels": [ 354 | "ID_MODALIDAD", 355 | "ID_PP" 356 | ] 357 | }, 358 | "administrative_classification": { 359 | "label": "administrative_classification", 360 | "levels": [ 361 | "ID_RAMO", 362 | "ID_UR" 363 | ] 364 | }, 365 | "budget_line_id": { 366 | "label": "budget_line_id", 367 | "levels": [ 368 | "ID_CLAVE_CARTERA" 369 | ] 370 | }, 371 | "date": { 372 | "label": "date", 373 | "levels": [ 374 | "CICLO" 375 | ] 376 | }, 377 | "economic_classification": { 378 | "label": "economic_classification", 379 | "levels": [ 380 | "ID_CAPITULO", 381 | "ID_CONCEPTO", 382 | "ID_PARTIDA_GENERICA", 383 | "ID_PARTIDA_ESPECIFICA" 384 | ] 385 | }, 386 | "expenditure_type": { 387 | "label": "expenditure_type", 388 | "levels": [ 389 | "ID_TIPOGASTO" 390 | ] 391 | }, 392 | "fin_source": { 393 | "label": "fin_source", 394 | "levels": [ 395 | "ID_FF" 396 | ] 397 | }, 398 | "functional_classification": { 399 | "label": "functional_classification", 400 | "levels": [ 401 | "GPO_FUNCIONAL", 402 | "ID_FUNCION", 403 | "ID_SUBFUNCION", 404 | "ID_AI" 405 | ] 406 | }, 407 | "geo_source": { 408 | "label": "geo_source", 409 | "levels": [ 410 | "ID_ENTIDAD_FEDERATIVA" 411 | ] 412 | } 413 | }, 414 | "measures": { 415 | "MONTO_ADEFAS": { 416 | "column": "MONTO_ADEFAS", 417 | "label": "ADEFAS", 418 | "type": "number" 419 | }, 420 | "MONTO_APROBADO": { 421 | "column": "MONTO_APROBADO", 422 | "label": "Aprobado", 423 | "type": "number" 424 | }, 425 | "MONTO_DEVENGADO": { 426 | "column": "MONTO_DEVENGADO", 427 | "label": "Devengado", 428 | "type": "number" 429 | }, 430 | "MONTO_EJERCICIO": { 431 | "column": "MONTO_EJERCICIO", 432 | "label": "Ejercicio", 433 | "type": "number" 434 | }, 435 | "MONTO_EJERCIDO": { 436 | "column": "MONTO_EJERCIDO", 437 | "label": "Ejercido", 438 | "type": "number" 439 | }, 440 | "MONTO_MODIFICADO": { 441 | "column": "MONTO_MODIFICADO", 442 | "label": "Modificado", 443 | "type": "number" 444 | }, 445 | "MONTO_PAGADO": { 446 | "column": "MONTO_PAGADO", 447 | "label": "Pagado", 448 | "type": "number" 449 | } 450 | } 451 | } -------------------------------------------------------------------------------- /tests/fixtures/models/simple_model.json: -------------------------------------------------------------------------------- 1 | { 2 | "fact_table": "simple", 3 | "dimensions": { 4 | "foo": { 5 | "label": "Foo", 6 | "key_attribute": "key", 7 | "label_attribute": "key", 8 | "attributes": { 9 | "key": {"label": "Key", "column": "foo_key", "type": "string"} 10 | } 11 | }, 12 | "bar": { 13 | "label": "Bar", 14 | "key_attribute": "key", 15 | "attributes": { 16 | "key": {"label": "Key", "column": "bar_key", "type": "string"} 17 | } 18 | }, 19 | "baz": { 20 | "label": "Baz", 21 | "key_attribute": "key", 22 | "attributes": { 23 | "key": {"label": "Key", "column": "bar_key", "type": "string"} 24 | } 25 | }, 26 | "waz": { 27 | "label": "Waz", 28 | "key_attribute": "key", 29 | "attributes": { 30 | "key": {"label": "Key", "column": "bar_key", "type": "string"} 31 | } 32 | } 33 | 34 | }, 35 | "measures": { 36 | "amount": { 37 | "label": "Amount in USD", 38 | "column": "amount", 39 | "type": "integer" 40 | } 41 | }, 42 | "hierarchies": { 43 | "bazwaz": { 44 | "label": "BazWaz", 45 | "levels": [ 46 | "baz", "waz" 47 | ] 48 | } 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /tests/test_api.py: -------------------------------------------------------------------------------- 1 | import os 2 | import io 3 | import csv 4 | import pytest 5 | from flask import url_for 6 | 7 | from babbage.manager import JSONCubeManager 8 | from babbage.api import configure_api 9 | 10 | 11 | @pytest.mark.usefixtures('load_api_fixtures') 12 | class TestCubeManager(object): 13 | def test_index(self, client): 14 | res = client.get(url_for('babbage_api.index')) 15 | assert res.status_code == 200, res 16 | assert res.json['api'] == 'babbage' 17 | 18 | def test_jsonp(self, client): 19 | res = client.get(url_for('babbage_api.index', callback='foo')) 20 | assert res.status_code == 200, res 21 | assert res.data.startswith(b'foo && foo('), res.data 22 | 23 | def test_list_cubes(self, client): 24 | res = client.get(url_for('babbage_api.cubes')) 25 | assert len(res.json['data']) == 3, res.json 26 | 27 | def test_get_model(self, client): 28 | res = client.get(url_for('babbage_api.model', name='cra')) 29 | assert len(res.json['model']['measures'].keys()) == 2, res.json 30 | assert res.json['name'] == 'cra', res.json 31 | 32 | def test_get_missing_model(self, client): 33 | res = client.get(url_for('babbage_api.model', name='crack')) 34 | assert res.status_code == 404, res 35 | 36 | def test_aggregate_missing(self, client): 37 | res = client.get(url_for('babbage_api.aggregate', name='crack')) 38 | assert res.status_code == 404, res 39 | 40 | @pytest.mark.usefixtures('load_fixtures') 41 | def test_aggregate_simple(self, client): 42 | res = client.get(url_for('babbage_api.aggregate', name='cra')) 43 | assert res.status_code == 200, res 44 | assert 'summary' in res.json, res.json 45 | assert 1 == len(res.json['cells']), res.json 46 | 47 | @pytest.mark.usefixtures('load_fixtures') 48 | def test_aggregate_returns_csv_format(self, client): 49 | url = url_for( 50 | 'babbage_api.aggregate', 51 | name='cra', 52 | drilldown='cofog1.label', 53 | order='cofog1.label', 54 | format='csv' 55 | ) 56 | res = client.get(url) 57 | 58 | assert res.status_code == 200 59 | data = res.get_data().decode('utf-8') 60 | rows = [row for row in csv.DictReader(io.StringIO(data))] 61 | expected_rows = [ 62 | { 63 | '_count': '12', 64 | 'amount.sum': '45400000', 65 | 'cofog1.label': 'Economic affairs', 66 | 'total.sum': '45400000' 67 | }, 68 | { 69 | '_count': '8', 70 | 'amount.sum': '-608900000', 71 | 'cofog1.label': 'Housing and community amenities', 72 | 'total.sum': '-608900000' 73 | }, 74 | { 75 | '_count': '5', 76 | 'amount.sum': '500000', 77 | 'cofog1.label': 'Public order and safety', 78 | 'total.sum': '500000' 79 | }, 80 | { 81 | '_count': '11', 82 | 'amount.sum': '191500000', 83 | 'cofog1.label': 'Social protection', 84 | 'total.sum': '191500000' 85 | } 86 | ] 87 | 88 | assert rows == expected_rows 89 | 90 | @pytest.mark.usefixtures('load_fixtures') 91 | def test_aggregate_drilldown(self, client): 92 | res = client.get(url_for('babbage_api.aggregate', name='cra', 93 | drilldown='cofog1')) 94 | assert res.status_code == 200, res 95 | assert 'summary' in res.json, res.json 96 | assert 4 == len(res.json['cells']), res.json 97 | 98 | @pytest.mark.usefixtures('load_fixtures') 99 | def test_aggregate_invalid_drilldown(self, client): 100 | res = client.get(url_for('babbage_api.aggregate', name='cra', 101 | drilldown='cofoxxxg1')) 102 | assert res.status_code == 400, res 103 | 104 | def test_facts_missing(self, client): 105 | res = client.get(url_for('babbage_api.facts', name='crack')) 106 | assert res.status_code == 404, res 107 | 108 | @pytest.mark.usefixtures('load_fixtures') 109 | def test_facts_simple(self, client): 110 | res = client.get(url_for('babbage_api.facts', name='cra')) 111 | assert res.status_code == 200, (res, res.get_data()) 112 | assert 'total_fact_count' in res.json, res.json 113 | assert 36 == len(res.json['data']), res.json 114 | 115 | @pytest.mark.usefixtures('load_fixtures') 116 | def test_facts_cut(self, client): 117 | res = client.get(url_for('babbage_api.facts', name='cra', 118 | cut='cofog1:"10"')) 119 | assert res.status_code == 200, (res, res.get_data()) 120 | assert 11 == len(res.json['data']), len(res.json['data']) 121 | 122 | def test_members_missing(self, client): 123 | res = client.get(url_for('babbage_api.members', name='cra', 124 | ref='codfss')) 125 | assert res.status_code == 400, res 126 | 127 | @pytest.mark.usefixtures('load_fixtures') 128 | def test_members_simple(self, client): 129 | res = client.get(url_for('babbage_api.members', name='cra', 130 | ref='cofog1')) 131 | assert res.status_code == 200, res 132 | assert 'total_member_count' in res.json, res.json 133 | assert 4 == len(res.json['data']), res.json 134 | -------------------------------------------------------------------------------- /tests/test_cube.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from babbage.cube import Cube 4 | from babbage.exc import BindingException, QueryException 5 | 6 | 7 | class TestCube(object): 8 | def test_table_exists(self, sqla_engine, cra_table): 9 | assert sqla_engine.has_table(cra_table.name) 10 | 11 | def test_table_load(self, cube, cra_table): 12 | table = cube._load_table(cra_table.name) 13 | assert table is not None 14 | assert 'cra' in repr(cube) 15 | 16 | def test_table_pk(self, cube): 17 | assert cube.fact_pk is not None 18 | 19 | def test_table_load_nonexist(self, cube): 20 | with pytest.raises(BindingException): 21 | cube._load_table('lalala') 22 | 23 | def test_dimension_column_nonexist(self, sqla_engine, cra_model): 24 | with pytest.raises(BindingException): 25 | model = cra_model.copy() 26 | model['dimensions']['cofog1']['attributes']['name']['column'] = 'lala' 27 | cube = Cube(sqla_engine, 'cra', model) 28 | cube.model['cofog1.name'].bind(cube) 29 | 30 | def test_star_column_nonexist(self, sqla_engine, cra_model): 31 | with pytest.raises(BindingException): 32 | model = cra_model.copy() 33 | model['dimensions']['cap_or_cur']['join_column'] = 'lala' 34 | cube = Cube(sqla_engine, 'cra', model) 35 | cube.facts() 36 | 37 | def test_attr_table_different_from_dimension_key(self, sqla_engine, cra_model): 38 | with pytest.raises(BindingException): 39 | model = cra_model.copy() 40 | model['dimensions']['cap_or_cur']['attributes']['code']['column'] = 'cap_or_cur' 41 | cube = Cube(sqla_engine, 'cra', model) 42 | cube.facts() 43 | 44 | def test_dimension_column_qualified(self, sqla_engine, cra_model, cra_table): 45 | model = cra_model.copy() 46 | name = 'cra.cofog1_name' 47 | model['dimensions']['cofog1']['attributes']['name']['column'] = name 48 | cube = Cube(sqla_engine, 'cra', model) 49 | cube.model['cofog1.name'].bind(cube) 50 | 51 | def test_facts_basic(self, cube): 52 | facts = cube.facts() 53 | assert facts['total_fact_count'] == 36 54 | assert len(facts['data']) == 36, len(facts['data']) 55 | row0 = facts['data'][0] 56 | assert 'cofog1.name' in row0, row0 57 | assert 'amount' in row0, row0 58 | assert 'amount.sum' not in row0, row0 59 | assert '_count' not in row0, row0 60 | 61 | def test_facts_basic_filter(self, cube): 62 | facts = cube.facts(cuts='cofog1:"4"') 63 | assert facts['total_fact_count'] == 12, facts 64 | assert len(facts['data']) == 12, len(facts['data']) 65 | 66 | def test_facts_set_filter(self, cube): 67 | facts = cube.facts(cuts='cofog1:"4";"10"') 68 | assert facts['total_fact_count'] == 23, facts 69 | assert len(facts['data']) == 23, len(facts['data']) 70 | 71 | def test_facts_star_filter(self, cube): 72 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"') 73 | assert facts['total_fact_count'] == 21, facts 74 | assert len(facts['data']) == 21, len(facts['data']) 75 | 76 | def test_facts_star_filter_and_facts_field(self, cube): 77 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"', 78 | fields='cofog1') 79 | assert facts['total_fact_count'] == 21, facts 80 | assert len(facts['data']) == 21, len(facts['data']) 81 | 82 | def test_facts_star_filter_and_field(self, cube): 83 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"', 84 | fields='cap_or_cur.code') 85 | assert facts['total_fact_count'] == 21, facts 86 | assert len(facts['data']) == 21, len(facts['data']) 87 | 88 | def test_facts_facts_filter_and_star_field(self, cube): 89 | facts = cube.facts(cuts='cofog1:"4"', 90 | fields='cap_or_cur.code') 91 | assert facts['total_fact_count'] == 12, facts 92 | assert len(facts['data']) == 12, len(facts['data']) 93 | 94 | def test_facts_star_filter(self, cube): 95 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"') 96 | assert facts['total_fact_count'] == 21, facts 97 | assert len(facts['data']) == 21, len(facts['data']) 98 | 99 | def test_facts_star_filter_and_facts_field(self, cube): 100 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"', 101 | fields='cofog1') 102 | assert facts['total_fact_count'] == 21, facts 103 | assert len(facts['data']) == 21, len(facts['data']) 104 | 105 | def test_facts_star_filter_and_field(self, cube): 106 | facts = cube.facts(cuts='cap_or_cur.label:"Current Expenditure"', 107 | fields='cap_or_cur.code') 108 | assert facts['total_fact_count'] == 21, facts 109 | assert len(facts['data']) == 21, len(facts['data']) 110 | 111 | def test_facts_facts_filter_and_star_field(self, cube): 112 | facts = cube.facts(cuts='cofog1:"4"', 113 | fields='cap_or_cur.code') 114 | assert facts['total_fact_count'] == 12, facts 115 | assert len(facts['data']) == 12, len(facts['data']) 116 | 117 | def test_facts_cut_type_error(self, cube): 118 | with pytest.raises(QueryException): 119 | cube.facts(cuts='cofog1:4') 120 | 121 | def test_facts_invalid_filter(self, cube): 122 | with pytest.raises(QueryException): 123 | cube.facts(cuts='cofogXX:"4"') 124 | 125 | def test_facts_basic_fields(self, cube): 126 | facts = cube.facts(fields='cofog1,cofog2') 127 | assert facts['total_fact_count'] == 36, facts['total_fact_count'] 128 | row0 = facts['data'][0] 129 | assert 'cofog1.name' in row0, row0 130 | assert 'amount' not in row0, row0 131 | assert 'amount.sum' not in row0, row0 132 | 133 | def test_facts_star_fields(self, cube): 134 | facts = cube.facts(fields='cofog1,cap_or_cur.label') 135 | assert facts['total_fact_count'] == 36, facts['total_fact_count'] 136 | row0 = facts['data'][0] 137 | assert 'cofog1.name' in row0, row0 138 | assert 'cap_or_cur.code' not in row0, row0 139 | assert 'cap_or_cur.label' in row0, row0 140 | 141 | def test_facts_invalid_field(self, cube): 142 | with pytest.raises(QueryException): 143 | cube.facts(fields='cofog1,schnasel') 144 | 145 | def test_facts_paginate(self, cube): 146 | facts = cube.facts(page_size=5) 147 | assert facts['total_fact_count'] == 36, facts['total_fact_count'] 148 | assert len(facts['data']) == 5, len(facts['data']) 149 | 150 | def test_facts_sort(self, cube): 151 | facts = cube.facts(order='amount:desc') 152 | assert facts['total_fact_count'] == 36, facts['total_fact_count'] 153 | facts = facts['data'] 154 | assert len(facts) == 36, len(facts['data']) 155 | assert max(facts, key=lambda f: f['amount']) == facts[0] 156 | assert min(facts, key=lambda f: f['amount']) == facts[-1] 157 | 158 | def test_members_basic(self, cube): 159 | members = cube.members('cofog1') 160 | assert members['total_member_count'] == 4, members['total_member_count'] 161 | assert len(members['data']) == 4, len(members['data']) 162 | row0 = members['data'][0] 163 | assert 'cofog1.name' in row0, row0 164 | assert 'amount' not in row0, row0 165 | 166 | def test_members_star_dimension(self, cube): 167 | members = cube.members('cap_or_cur', order='cap_or_cur.label:asc') 168 | assert members['total_member_count'] == 2, members['total_member_count'] 169 | assert len(members['data']) == 2, len(members['data']) 170 | assert 'cap_or_cur.code' in members['data'][0], members['data'][0] 171 | assert 'CAP' == members['data'][0]['cap_or_cur.code'], members['data'][0] 172 | 173 | def test_members_star_dimension_order(self, cube): 174 | members = cube.members('cap_or_cur', order='cap_or_cur.label:desc') 175 | assert 'CUR' == members['data'][0]['cap_or_cur.code'], members['data'][0] 176 | 177 | def test_members_paginate(self, cube): 178 | members = cube.members('cofog1', page_size=2) 179 | assert members['total_member_count'] == 4, members['total_member_count'] 180 | assert members['page'] == 1, members 181 | assert members['page_size'] == 2, members 182 | assert len(members['data']) == 2, len(members['data']) 183 | members2 = cube.members('cofog1', page_size=2, page=2) 184 | assert members2['page'] == 2, members2 185 | assert members2['page_size'] == 2, members2 186 | assert members2['data'] != members['data'], members2['data'] 187 | 188 | def test_aggregate_basic(self, cube): 189 | aggs = cube.aggregate(drilldowns='cofog1') 190 | assert aggs['total_cell_count'] == 4, aggs['total_cell_count'] 191 | assert len(aggs['cells']) == 4, len(aggs['data']) 192 | row0 = aggs['cells'][0] 193 | assert 'cofog1.name' in row0, row0 194 | assert 'amount.sum' in row0, row0 195 | assert 'amount' not in row0, row0 196 | 197 | def test_aggregate_summary(self, cube): 198 | sum_by = lambda arr, key: sum([element[key] for element in arr]) 199 | aggs = cube.aggregate(drilldowns='cofog1') 200 | cells = aggs['cells'] 201 | 202 | expected_summary = { 203 | '_count': sum_by(cells, '_count'), 204 | 'amount.sum': sum_by(cells, 'amount.sum'), 205 | 'total.sum': sum_by(cells, 'total.sum'), 206 | } 207 | 208 | assert aggs['summary'] == expected_summary 209 | 210 | def test_aggregate_star(self, cube): 211 | aggs = cube.aggregate(drilldowns='cap_or_cur', order='cap_or_cur') 212 | assert aggs['total_cell_count'] == 2, aggs 213 | assert len(aggs['cells']) == 2, len(aggs['data']) 214 | row0 = aggs['cells'][0] 215 | assert row0['cap_or_cur.code'] == 'CAP', row0 216 | assert row0['amount.sum'] == -608400000, row0 217 | assert row0['_count'] == 15, row0 218 | 219 | def test_aggregate_count_only(self, cube): 220 | aggs = cube.aggregate(drilldowns='cofog1', aggregates='_count') 221 | assert aggs['total_cell_count'] == 4, aggs['total_cell_count'] 222 | assert len(aggs['cells']) == 4, len(aggs['data']) 223 | assert '_count' in aggs['summary'], aggs['summary'] 224 | row0 = aggs['cells'][0] 225 | assert 'cofog1.name' in row0, row0 226 | assert 'amount.sum' not in row0, row0 227 | assert 'amount' not in row0, row0 228 | 229 | def test_aggregate_star_count_only(self, cube): 230 | aggs = cube.aggregate(drilldowns='cap_or_cur', 231 | order='cap_or_cur', 232 | aggregates='_count') 233 | assert aggs['total_cell_count'] == 2, aggs 234 | assert len(aggs['cells']) == 2, len(aggs['data']) 235 | row0 = aggs['cells'][0] 236 | assert row0['cap_or_cur.code'] == 'CAP', row0 237 | assert row0['_count'] == 15, row0 238 | 239 | def test_aggregate_empty(self, cube): 240 | aggs = cube.aggregate(drilldowns='cofog1', page_size=0) 241 | assert aggs['total_cell_count'] == 4, aggs['total_cell_count'] 242 | assert len(aggs['cells']) == 0, len(aggs['data']) 243 | 244 | def test_compute_cardinalities(self, cube): 245 | cofog = cube.model['cofog1'] 246 | assert cofog.cardinality is None 247 | assert cofog.cardinality_class is None 248 | cube.compute_cardinalities() 249 | assert cofog.cardinality == 4, cofog.cardinality 250 | assert cofog.cardinality_class == 'tiny', \ 251 | (cofog.cardinality, cofog.cardinality_class) 252 | 253 | 254 | @pytest.fixture() 255 | def cube(sqla_engine, cra_model, load_fixtures): 256 | return Cube(sqla_engine, 'cra', cra_model) 257 | -------------------------------------------------------------------------------- /tests/test_manager.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from babbage.cube import Cube 4 | from babbage.exc import BabbageException 5 | 6 | 7 | @pytest.mark.usefixtures('load_api_fixtures') 8 | class TestCubeManager(object): 9 | def test_list_cubes(self, fixtures_cube_manager): 10 | cubes = list(fixtures_cube_manager.list_cubes()) 11 | assert len(cubes) == 3, cubes 12 | 13 | def test_has_cube(self, fixtures_cube_manager): 14 | assert fixtures_cube_manager.has_cube('cra') 15 | assert not fixtures_cube_manager.has_cube('cro') 16 | 17 | def test_get_model(self, fixtures_cube_manager): 18 | model = fixtures_cube_manager.get_cube_model('cra') 19 | assert 'dimensions' in model 20 | 21 | def test_get_model_doesnt_exist(self, fixtures_cube_manager): 22 | with pytest.raises(BabbageException): 23 | fixtures_cube_manager.get_cube_model('cro') 24 | 25 | def test_get_cube(self, fixtures_cube_manager): 26 | cube = fixtures_cube_manager.get_cube('cra') 27 | assert isinstance(cube, Cube) 28 | 29 | def test_get_cube_doesnt_exist(self, fixtures_cube_manager): 30 | with pytest.raises(BabbageException): 31 | fixtures_cube_manager.get_cube('cro') 32 | -------------------------------------------------------------------------------- /tests/test_model.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | 4 | class TestModel(object): 5 | def test_model_concepts(self, simple_model): 6 | concepts = list(simple_model.concepts) 7 | assert len(concepts) == 11, len(concepts) 8 | 9 | def test_model_match(self, simple_model): 10 | concepts = list(simple_model.match('foo')) 11 | assert len(concepts) == 1, len(concepts) 12 | 13 | def test_model_match_invalid(self, simple_model): 14 | concepts = list(simple_model.match('fooxx')) 15 | assert len(concepts) == 0, len(concepts) 16 | 17 | def test_model_aggregates(self, simple_model): 18 | aggregates = list(simple_model.aggregates) 19 | assert len(aggregates) == 2, aggregates 20 | 21 | def test_model_fact_table(self, simple_model): 22 | assert simple_model.fact_table_name == 'simple' 23 | assert 'simple' in repr(simple_model), repr(simple_model) 24 | 25 | def test_model_hierarchies(self, simple_model): 26 | hierarchies = list(simple_model.hierarchies) 27 | assert len(hierarchies) == 1 28 | 29 | def test_model_dimension_hierarchies(self, simple_model): 30 | bar = simple_model.match('bar')[0] 31 | baz = simple_model.match('baz')[0] 32 | assert bar.ref.startswith('bar.') 33 | assert baz.alias.startswith('bazwaz.') 34 | 35 | def test_deref(self, simple_model): 36 | assert simple_model['foo'].name == 'foo' 37 | assert simple_model['foo.key'].name == 'key' 38 | assert simple_model['amount'].name == 'amount' 39 | assert 'amount' in simple_model 40 | assert 'amount.sum' in simple_model 41 | assert '_count' in simple_model 42 | assert 'yabba' not in simple_model 43 | assert 'foo.key' in simple_model 44 | 45 | def test_repr(self, simple_model): 46 | assert 'amount' in repr(simple_model['amount']) 47 | assert 'amount.sum' in repr(simple_model['amount.sum']) 48 | assert 'foo.key' in repr(simple_model['foo.key']) 49 | assert 'foo' in repr(simple_model['foo']) 50 | assert 'foo' in str(simple_model['foo']) 51 | assert simple_model['foo'] == 'foo' 52 | 53 | def test_to_dict(self, simple_model): 54 | data = simple_model.to_dict() 55 | assert 'measures' in data 56 | assert 'amount' in data['measures'] 57 | assert 'amount.sum' in data['aggregates'] 58 | assert 'ref' in data['measures']['amount'] 59 | assert 'dimensions' in data 60 | assert 'hierarchies' in data 61 | assert 'foo' in data['dimensions'] 62 | -------------------------------------------------------------------------------- /tests/test_parser.py: -------------------------------------------------------------------------------- 1 | from datetime import date 2 | import pytest 3 | 4 | from babbage.exc import QueryException 5 | from babbage.cube import Cube 6 | from babbage.query import Cuts, Ordering, Aggregates, Drilldowns 7 | 8 | 9 | class TestParser(object): 10 | 11 | def test_cuts_unquoted_string(self, cube): 12 | cuts = Cuts(cube).parse('foo:bar') 13 | assert len(cuts) == 1, cuts 14 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 15 | assert ('foo', ':', ['bar']) in cuts, cuts 16 | 17 | def test_cuts_quoted_string(self, cube): 18 | cuts = Cuts(cube).parse('foo:"bar lala"') 19 | assert len(cuts) == 1, cuts 20 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 21 | assert ('foo', ':', ['bar lala']) in cuts, cuts 22 | 23 | def test_cuts_string_set(self, cube): 24 | cuts = Cuts(cube).parse('foo:"bar";"lala"') 25 | assert len(cuts) == 1, cuts 26 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 27 | assert ('foo', ':', ['bar', 'lala']) in cuts, cuts 28 | 29 | def test_cuts_int_set(self, cube): 30 | cuts = Cuts(cube).parse('foo:3;22') 31 | assert len(cuts) == 1, cuts 32 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 33 | assert ('foo', ':', [3, 22]) in cuts, cuts 34 | 35 | def test_cuts_multiple(self, cube): 36 | cuts = Cuts(cube).parse('foo:bar|bar:5') 37 | assert len(cuts) == 2, cuts 38 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 39 | assert ('bar', ':', [5]) in list(cuts), cuts 40 | 41 | def test_cuts_multiple_int_first(self, cube): 42 | cuts = Cuts(cube).parse('bar:5|foo:bar') 43 | assert len(cuts) == 2, cuts 44 | cuts = [(c[0], c[1], list(c[2])) for c in cuts] 45 | assert ('bar', ':', [5]) in list(cuts), cuts 46 | 47 | def test_cuts_quotes(self, cube): 48 | cuts = Cuts(cube).parse('foo:"bar|lala"|bar:5') 49 | assert len(cuts) == 2, cuts 50 | 51 | def test_cuts_date(self, cube): 52 | cuts = Cuts(cube).parse('foo:2015-01-04') 53 | assert list(cuts[0][2]) == [date(2015, 1, 4)], cuts 54 | 55 | def test_cuts_date_set(self, cube): 56 | cuts = Cuts(cube).parse('foo:2015-01-04;2015-01-05') 57 | assert len(cuts) == 1, cuts 58 | assert list(cuts[0][2]) == [date(2015, 1, 4), date(2015, 1, 5)], cuts 59 | 60 | def test_cuts_int(self, cube): 61 | cuts = Cuts(cube).parse('foo:2015') 62 | assert list(cuts[0][2]) == [2015], cuts 63 | 64 | def test_cuts_int_prefixed_string(self, cube): 65 | cuts = Cuts(cube).parse('foo:2015M01') 66 | assert list(cuts[0][2]) == ['2015M01'], cuts 67 | 68 | def test_cuts_invalid(self, cube): 69 | with pytest.raises(QueryException): 70 | Cuts(cube).parse('f oo:2015-01-04') 71 | 72 | def test_null_filter(self, cube): 73 | cuts = Cuts(cube).parse(None) 74 | assert isinstance(cuts, list) 75 | assert not len(cuts) 76 | 77 | def test_order(self, cube): 78 | ordering = Ordering(cube).parse('foo:desc,bar') 79 | assert ordering[0][1] == "desc", ordering 80 | assert ordering[1][1] == "asc", ordering 81 | 82 | def test_order_invalid(self, cube): 83 | with pytest.raises(QueryException): 84 | Ordering(cube).parse('fooxx:desc') 85 | 86 | def test_drilldowns(self, cube): 87 | dd = Drilldowns(cube).parse('foo|bar') 88 | assert len(dd) == 2 89 | 90 | def test_drilldowns_invalid(self, cube): 91 | with pytest.raises(QueryException): 92 | Drilldowns(cube).parse('amount') 93 | 94 | def test_aggregates_invalid(self, cube): 95 | with pytest.raises(QueryException): 96 | Aggregates(cube).parse('amount') 97 | 98 | def test_aggregates_dimension(self, cube): 99 | with pytest.raises(QueryException): 100 | Aggregates(cube).parse('cofog1.name') 101 | 102 | def test_aggregates(self, cube): 103 | agg = Aggregates(cube).parse('amount.sum') 104 | assert len(agg) == 1 105 | agg = Aggregates(cube).parse('amount.sum|_count') 106 | assert len(agg) == 2 107 | 108 | 109 | @pytest.fixture 110 | def cube(sqla_engine, simple_model): 111 | return Cube(sqla_engine, 'simple', simple_model) 112 | -------------------------------------------------------------------------------- /tests/test_validation.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from jsonschema import ValidationError 3 | 4 | from babbage.validation import validate_model 5 | 6 | 7 | class TestValidation(object): 8 | 9 | def test_simple_model(self, simple_model_data): 10 | validate_model(simple_model_data) 11 | 12 | def test_invalid_fact_table(self, simple_model_data): 13 | with pytest.raises(ValidationError): 14 | model = simple_model_data 15 | model['fact_table'] = 'b....' 16 | validate_model(model) 17 | 18 | def test_no_fact_table(self, simple_model_data): 19 | with pytest.raises(ValidationError): 20 | model = simple_model_data 21 | del model['fact_table'] 22 | validate_model(model) 23 | 24 | def test_invalid_dimension_name(self, simple_model_data): 25 | with pytest.raises(ValidationError): 26 | model = simple_model_data 27 | model['dimensions']['goo fdj.'] = {'label': 'bar'} 28 | validate_model(model) 29 | 30 | def test_invalid_measure_name(self, simple_model_data): 31 | with pytest.raises(ValidationError): 32 | model = simple_model_data 33 | model['measures']['goo fdj.'] = {'label': 'bar'} 34 | validate_model(model) 35 | 36 | def test_no_measure(self, simple_model_data): 37 | with pytest.raises(ValidationError): 38 | model = simple_model_data 39 | model['measures'] = {} 40 | validate_model(model) 41 | 42 | def test_no_measure_label(self, simple_model_data): 43 | with pytest.raises(ValidationError): 44 | model = simple_model_data 45 | model['measures']['amount'] = {} 46 | validate_model(model) 47 | 48 | def test_invalid_aggregate(self, simple_model_data): 49 | with pytest.raises(ValidationError): 50 | model = simple_model_data 51 | model['measures']['amount']['aggregates'] = 'schnasel' 52 | validate_model(model) 53 | 54 | def test_invalid_aggregate_string(self, simple_model_data): 55 | with pytest.raises(ValidationError): 56 | model = simple_model_data 57 | model['measures']['amount']['aggregates'] = 'count' 58 | validate_model(model) 59 | 60 | def test_invalid_aggregate_string(self, simple_model_data): 61 | model = simple_model_data 62 | model['measures']['amount']['aggregates'] = ['count'] 63 | validate_model(model) 64 | 65 | def test_dimension_without_attributes(self, simple_model_data): 66 | with pytest.raises(ValidationError): 67 | model = simple_model_data 68 | model['dimensions']['foo']['attributes'] = {} 69 | validate_model(model) 70 | 71 | def test_dimension_without_key(self, simple_model_data): 72 | with pytest.raises(ValidationError): 73 | model = simple_model_data 74 | del model['dimensions']['foo']['key_attribute'] 75 | validate_model(model) 76 | 77 | def test_dimension_invalid_key(self, simple_model_data): 78 | with pytest.raises(ValidationError): 79 | model = simple_model_data 80 | model['dimensions']['foo']['key_attribute'] = 'lala' 81 | validate_model(model) 82 | 83 | def test_dimension_invalid_label(self, simple_model_data): 84 | with pytest.raises(ValidationError): 85 | model = simple_model_data 86 | model['dimensions']['foo']['label_attribute'] = 'lala' 87 | validate_model(model) 88 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | package = babbage 3 | envlist = 4 | py36 5 | py37 6 | lint 7 | skip_missing_interpreters = true 8 | 9 | [testenv] 10 | passenv = 11 | BABBAGE_TEST_DB 12 | deps = 13 | pytest 14 | pytest-cov 15 | pytest-flask 16 | coverage 17 | python-dateutil 18 | unicodecsv 19 | commands = 20 | pytest \ 21 | --cov {[tox]package} \ 22 | {posargs} 23 | 24 | 25 | [testenv:lint] 26 | deps = 27 | pylama 28 | commands = 29 | pylama {[tox]package} \ 30 | {posargs} --------------------------------------------------------------------------------