2 |
3 | # MongoFrames
4 |
5 | [](https://travis-ci.org/GetmeUK/MongoFrames)
6 | [](https://gitter.im/GetmeUK/MongoFrames?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
7 |
8 | MongoFrames is a fast unobtrusive MongoDB ODM for Python designed to fit into a workflow not dictate one. Documentation is available at [MongoFrames.com](http://mongoframes.com) and includes:
9 |
10 | - [A getting started guide](http://mongoframes.com/getting-started)
11 | - [Tutorials/Snippets](http://mongoframes.com/snippets)
12 | - [API documentation](http://mongoframes.com/api)
13 |
14 | ## Installation
15 |
16 | We recommend you use [virtualenv](https://virtualenv.pypa.io) or [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io) to create a virtual environment for your install. There are several options for installing MongoFrames:
17 |
18 | - `pip install MongoFrames` *(recommended)*
19 | - `easy_install MongoFrames`
20 | - Download the source and run `python setup.py install`
21 |
22 | ## Dependencies
23 |
24 | - [blinker](https://pythonhosted.org/blinker/) >= 1.4
25 | - [pymongo](https://api.mongodb.com) >= 3
26 | - [pytest](http://pytest.org/) >= 2.8.7 *(only for testing)*
27 |
28 | ## 10 second example
29 |
30 | ```Python
31 | from mongoframes import *
32 |
33 | # Define some document frames (a.k.a models)
34 | class Dragon(Frame):
35 | _fields = {'name', 'loot'}
36 |
37 | class Item(Frame):
38 | _fields = {'desc', 'value'}
39 |
40 | # Create a dragon and loot to boot
41 | Item(desc='Sock', value=1).insert()
42 | Item(desc='Diamond', value=100).insert()
43 | Dragon(name='Burt', loot=Item.many()).insert()
44 |
45 | # Have Burt boast about his loot
46 | burt = Dragon.one(Q.name == 'Burt', projection={'loot': {'$ref': Item}})
47 | for item in burt.loot:
48 | print('I have a {0.name} worth {0.value} crown'.format(item))
49 | ```
50 |
51 | ## Testing
52 |
53 | To test the library you'll need to be running a local instance of MongoDB on the standard port.
54 |
55 | To run the test suite: `py.test`
56 | To run the test suite on each supported version of Python: `tox`
57 |
58 | ## Helpful organizations
59 | MongoFrames is developed using a number of tools & services provided for free by nice folks at organizations committed to supporting open-source projects including [GitHub](https://github.com) and [Travis CI](https://travis-ci.org).
60 |
--------------------------------------------------------------------------------
/lib64:
--------------------------------------------------------------------------------
1 | lib
--------------------------------------------------------------------------------
/mongoframes/__init__.py:
--------------------------------------------------------------------------------
1 | from mongoframes.frames import *
2 | from mongoframes.pagination import *
3 | from mongoframes.queries import *
4 | from pymongo.collation import Collation
5 | from pymongo import (
6 | IndexModel,
7 | ASCENDING as ASC,
8 | DESCENDING as DESC,
9 | GEOSPHERE,
10 | TEXT
11 | )
--------------------------------------------------------------------------------
/mongoframes/factory/__init__.py:
--------------------------------------------------------------------------------
1 | import copy
2 |
3 | __all__ = ['Factory']
4 |
5 |
6 | class Factory:
7 | """
8 | The `Factory` class is responsible for production of fake data for.
9 | Production of fake data is a two stage process:
10 |
11 | Assembly (see `assemble`)
12 | : A `Quota` of documents is assembled based on a `Blueprint`.
13 |
14 | At this stage the documents contain a mixture of static and dynamic
15 | data. Dynamic data is data that will be transformed during population,
16 | for example a field might contain a value of `'now,tomorrow'` which on
17 | population will be converted to a date/time between now and tomorrow.
18 |
19 | Once assembled the generated documents are returned as a list and can be
20 | either used immediately to populate the database or saved out as a
21 | template for populating the database in future (for example when
22 | building a set of test data).
23 |
24 | Finishing and population (see `finish` and `populate`)
25 | : A database is populated based on a `Blueprint` and pre-assembled list of
26 | documents.
27 |
28 | During this stage dynamic data is converted to static data (this process
29 | is call finishing) and inserted into the database.
30 |
31 | Before populate inserts the finished documents into the database it
32 | converts each document into a Frame instance and calls the on_fake
33 | method against the Blueprint, after the documents are inserted into the
34 | database it calls the on_faked method against the Blueprint.
35 | """
36 |
37 | # Public methods
38 |
39 | def assemble(self, blueprint, quota):
40 | """Assemble a quota of documents"""
41 |
42 | # Reset the blueprint
43 | blueprint.reset()
44 |
45 | # Assemble the documents
46 | documents = []
47 | for i in range(0, int(quota)):
48 | documents.append(blueprint.assemble())
49 |
50 | return documents
51 |
52 | def finish(self, blueprint, documents):
53 | """Finish a list of pre-assembled documents"""
54 |
55 | # Reset the blueprint
56 | blueprint.reset()
57 |
58 | # Finish the documents
59 | finished = []
60 | for document in documents:
61 | finished.append(blueprint.finish(document))
62 |
63 | return finished
64 |
65 | def populate(self, blueprint, documents):
66 | """Populate the database with documents"""
67 |
68 | # Finish the documents
69 | documents = self.finish(blueprint, documents)
70 |
71 | # Convert the documents to frame instances
72 | frames = []
73 | for document in documents:
74 | # Separate out any meta fields
75 | meta_document = {}
76 | for field_name in blueprint._meta_fields:
77 | meta_document[field_name] = document[field_name]
78 | document.pop(field_name)
79 |
80 | # Initialize the frame
81 | frame = blueprint.get_frame_cls()(document)
82 |
83 | # Apply any meta fields
84 | for key, value in meta_document.items():
85 | setattr(frame, key, value)
86 |
87 | frames.append(frame)
88 |
89 | # Insert the documents
90 | blueprint.on_fake(frames)
91 | frames = blueprint.get_frame_cls().insert_many(frames)
92 | blueprint.on_faked(frames)
93 |
94 | return frames
95 |
96 | def reassemble(self, blueprint, fields, documents):
97 | """
98 | Reassemble the given set of fields for a list of pre-assembed documents.
99 |
100 | NOTE: Reassembly is done in place, since the data you send the method
101 | should be JSON type safe, if you need to retain the existing document
102 | it is recommended that you copy them using `copy.deepcopy`.
103 | """
104 |
105 | # Reset the blueprint
106 | blueprint.reset()
107 |
108 | # Reassemble the documents
109 | for document in documents:
110 | blueprint.reassemble(fields, document)
--------------------------------------------------------------------------------
/mongoframes/factory/blueprints.py:
--------------------------------------------------------------------------------
1 | from blinker import signal
2 | from collections import OrderedDict
3 |
4 | from mongoframes.factory.makers import Maker
5 |
6 | __all__ = ['Blueprint']
7 |
8 |
9 | class _BlueprintMeta(type):
10 | """
11 | Meta class for `Frame`s to ensure an `_id` is present in any defined set of
12 | fields.
13 | """
14 |
15 | def __new__(meta, name, bases, dct):
16 |
17 | # Collect all instructions for the blueprint. Each instruction is stored
18 | # as a key value pair in a dictionary where the name represents the
19 | # field the instruction refers to and the value is a `Maker` type for
20 | # generating a value for that field.
21 | dct['_instructions'] = dct.get('_instructions') or {}
22 |
23 | # Ensure the instructions are an ordered dictionary
24 | dct['_instructions'] = OrderedDict(dct['_instructions'])
25 |
26 | for k, v in dct.items():
27 | if isinstance(v, Maker):
28 | dct['_instructions'][k] = v
29 |
30 | # Check the blueprint has a frame class associated with it. The `Frame`
31 | # class will be used to insert the document into the database.
32 | assert '_frame_cls' in dct or len(bases) == 0, \
33 | 'No `_frame_cls` defined for the blueprint'
34 |
35 | # Check for a set of meta fields or define an empty set if there isn't
36 | # one. Meta-fields determine a set of one or more fields that should be
37 | # set when generating a document but which are not defined in the
38 | # assoicated `Frame` classes `_fields` attribute.
39 | if '_meta_fields' not in dct:
40 | dct['_meta_fields'] = set([])
41 |
42 | return super(_BlueprintMeta, meta).__new__(meta, name, bases, dct)
43 |
44 |
45 | class Blueprint(metaclass=_BlueprintMeta):
46 | """
47 | Blueprints provide the instructions for producing a fake document for a
48 | collection represented by a `Frame` class.
49 | """
50 |
51 | def __init__(self):
52 | assert False, \
53 | 'Blueprint classes should remain static and not be initialized'
54 |
55 | # Public methods
56 |
57 | @classmethod
58 | def get_frame_cls(cls):
59 | """Return the `Frame` class for the blueprint"""
60 | return cls._frame_cls
61 |
62 | @classmethod
63 | def get_instructions(cls):
64 | """Return the instructions for the blueprint"""
65 | return dict(cls._instructions)
66 |
67 | @classmethod
68 | def get_meta_fields(cls):
69 | """Return the meta-fields for the blueprint"""
70 | return cls._meta_fields
71 |
72 | # Factory methods
73 |
74 | @classmethod
75 | def assemble(cls):
76 | """Assemble a single document using the blueprint"""
77 | document = {}
78 | for field_name, maker in cls._instructions.items():
79 | with maker.target(document):
80 | document[field_name] = maker()
81 | return document
82 |
83 | @classmethod
84 | def finish(cls, document):
85 | """
86 | Take a assembled document and convert all assembled values to
87 | finished values.
88 | """
89 | target_document = {}
90 | document_copy = {}
91 | for field_name, value in document.items():
92 | maker = cls._instructions[field_name]
93 | target_document = document.copy()
94 | with maker.target(target_document):
95 | document_copy[field_name] = maker(value)
96 | target_document[field_name] = document_copy[field_name]
97 | return document_copy
98 |
99 | @classmethod
100 | def reassemble(cls, fields, document):
101 | """
102 | Take a previously assembled document and reassemble the given set of
103 | fields for it in place.
104 | """
105 | for field_name in cls._instructions:
106 | if field_name in fields:
107 | maker = cls._instructions[field_name]
108 | with maker.target(document):
109 | document[field_name] = maker()
110 |
111 | @classmethod
112 | def reset(cls):
113 | """
114 | Reset the blueprint. Blueprints are typically reset before being used to
115 | assemble a quota of documents. Resetting a Blueprint will in turn reset
116 | all the Maker instances defined as instructions for the Blueprint
117 | allowing internal counters and alike to be reset.
118 | """
119 | # Reset instructions
120 | for maker in cls._instructions.values():
121 | maker.reset()
122 |
123 | # Events
124 |
125 | @classmethod
126 | def on_fake(cls, frames):
127 | """Called before frames are inserted"""
128 |
129 | # By default the hook will simply trigger a `fake` event against the
130 | # frame. This allows the overriding method to control this behaviour.
131 | signal('fake').send(cls._frame_cls, frames=frames)
132 |
133 | @classmethod
134 | def on_faked(cls, frames):
135 | """Called after frames are inserted"""
136 |
137 | # By default the hook will simply trigger a `fake` event against the
138 | # frame. This allows the overriding method to control this behaviour.
139 | signal('faked').send(cls._frame_cls, frames=frames)
--------------------------------------------------------------------------------
/mongoframes/factory/makers/__init__.py:
--------------------------------------------------------------------------------
1 | import contextlib
2 | import random
3 |
4 | import faker
5 |
6 | from mongoframes.queries import Q
7 |
8 | __all__ = [
9 | 'DictOf',
10 | 'Faker',
11 | 'Lambda',
12 | 'ListOf',
13 | 'Reference',
14 | 'Static',
15 | 'SubFactory',
16 | 'Unique'
17 | ]
18 |
19 |
20 | class Maker:
21 | """
22 | A base class for all Maker classes.
23 | """
24 |
25 | def __init__(self):
26 |
27 | # The document the maker is assembling/finishing data for
28 | self._document = None
29 |
30 | def __call__(self, *args):
31 | if args:
32 | return self._finish(*args)
33 | return self._assemble()
34 |
35 | @property
36 | def document(self):
37 | # Return the target document
38 | return self._document
39 |
40 | def reset(self):
41 | """Reset the maker instance"""
42 | pass
43 |
44 | def _assemble(self):
45 | return None
46 |
47 | def _finish(self, value):
48 | return value
49 |
50 | @contextlib.contextmanager
51 | def target(self, document):
52 | self._document = document
53 | yield
54 | self._document = None
55 |
56 |
57 | class DictOf(Maker):
58 | """
59 | Make a dictionary of key/values where each value is a set value or generated
60 | using a maker.
61 | """
62 |
63 | def __init__(self, table):
64 | super().__init__()
65 |
66 | # The table of keyword arguments that will be used to generate the
67 | # dictionary.
68 | self._table = table
69 |
70 | def _assemble(self):
71 | table = {}
72 | for k, v in self._table.items():
73 | if isinstance(v, Maker):
74 | table[k] = v._assemble()
75 | else:
76 | table[k] = None
77 | return table
78 |
79 | def _finish(self, value):
80 | table = {}
81 | for k, v in self._table.items():
82 | if isinstance(v, Maker):
83 | table[k] = self._table[k]._finish(value[k])
84 | else:
85 | table[k] = self._table[k]
86 |
87 | return table
88 |
89 |
90 | class Faker(Maker):
91 | """
92 | Use any faker provider to generate a value (see
93 | http://fake-factory.readthedocs.io/)
94 | """
95 |
96 | default_locale = 'en_US'
97 |
98 | def __init__(self, provider, assembler=True, locale=None, **kwargs):
99 | super().__init__()
100 |
101 | # The provider that will be used to generate the value
102 | self._provider = provider
103 |
104 | # Flag indicating if the providers should be called in _assemble (True)
105 | # or _finish (False).
106 | self._assembler = assembler
107 |
108 | # The locale that will be used by the faker factory
109 | self._locale = locale or self.default_locale
110 |
111 | # The keyword arguments for the provider
112 | self._kwargs = kwargs
113 |
114 | def _assemble(self):
115 | if not self._assembler:
116 | return None
117 | provider = getattr(self.get_fake(self._locale), self._provider)
118 | return provider(**self._kwargs)
119 |
120 | def _finish(self, value):
121 | if self._assembler:
122 | return value
123 | provider = getattr(self.get_fake(self._locale), self._provider)
124 | return provider(**self._kwargs)
125 |
126 | @staticmethod
127 | def get_fake(locale=None):
128 | """Return a shared faker factory used to generate fake data"""
129 | if locale is None:
130 | locale = Faker.default_locale
131 |
132 | if not hasattr(Maker, '_fake_' + locale):
133 | Faker._fake = faker.Factory.create(locale)
134 | return Faker._fake
135 |
136 |
137 | class Lambda(Maker):
138 | """
139 | Use a function to generate a value.
140 | """
141 |
142 | def __init__(self, func, assembler=True, finisher=False):
143 | super().__init__()
144 |
145 | assert assembler or finisher, \
146 | 'Either `assembler` or `finisher` must be true for lambda'
147 |
148 | # The function to call
149 | self._func = func
150 |
151 | # Flag indicating if the lambda function should be called in _assemble
152 | self._assembler = assembler
153 |
154 | # Flag indicating if the lambda function should be called in _finish
155 | self._finisher = finisher
156 |
157 | def _assemble(self):
158 | if self._assembler:
159 | return self._func(self.document)
160 | return None
161 |
162 | def _finish(self, value):
163 | if self._finisher:
164 | return self._func(self.document, value)
165 | return value
166 |
167 |
168 | class ListOf(Maker):
169 | """
170 | Generate a list of values of the given quantity using the specified maker.
171 | """
172 |
173 | def __init__(self, maker, quantity, reset_maker=False):
174 | super().__init__()
175 |
176 | # The maker used to generate each value in the list
177 | self._maker = maker
178 |
179 | # The number of list items to generate
180 | self._quantity = quantity
181 |
182 | # A flag indicating if the maker should be reset each time the list is
183 | # generated.
184 | self._reset_maker = reset_maker
185 |
186 | def _assemble(self):
187 | quantity = int(self._quantity)
188 |
189 | if self._reset_maker:
190 | self._maker.reset()
191 |
192 | return [self._maker() for i in range(0, quantity)]
193 |
194 | def _finish(self, value):
195 |
196 | if self._reset_maker:
197 | self._maker.reset()
198 |
199 | with self._maker.target(self.document):
200 | return [self._maker(v) for v in value]
201 |
202 |
203 | class Static(Maker):
204 | """
205 | A maker that returns a fixed value.
206 | """
207 |
208 | def __init__(self, value, assembler=True):
209 | super().__init__()
210 |
211 | # The value to return
212 | self._value = value
213 |
214 | # Flag indicating if the value should be return in _assemble (True) or
215 | # _finish (False).
216 | self._assembler = assembler
217 |
218 | def _assemble(self):
219 | if self._assembler:
220 | return self._value
221 | return None
222 |
223 | def _finish(self, value):
224 | if self._assembler:
225 | return value
226 | return self._value
227 |
228 |
229 | class SubFactory(Maker):
230 | """
231 | A maker that generates sub-documents.
232 | """
233 |
234 | def __init__(self, blueprint):
235 | super().__init__()
236 |
237 | # The blueprint to produce
238 | self._blueprint = blueprint
239 |
240 | def reset(self):
241 | """Reset the blueprint for the maker"""
242 | self._blueprint.reset()
243 |
244 | def _assemble(self):
245 | return self._blueprint.assemble()
246 |
247 | def _finish(self, value):
248 | document = self._blueprint.finish(value)
249 |
250 | # Separate out any meta fields
251 | meta_document = {}
252 | for field_name in self._blueprint._meta_fields:
253 | meta_document[field_name] = document[field_name]
254 | document.pop(field_name)
255 |
256 | # Initialize the sub-frame
257 | sub_frame = self._blueprint.get_frame_cls()(document)
258 |
259 | # Apply any meta fields
260 | for key, value in meta_document.items():
261 | setattr(sub_frame, key, value)
262 |
263 | return sub_frame
264 |
265 |
266 | class Unique(Maker):
267 | """
268 | Ensure that unique values are generated by a maker.
269 | """
270 |
271 | def __init__(self,
272 | maker,
273 | exclude=None,
274 | assembler=True,
275 | max_attempts=1000
276 | ):
277 | super().__init__()
278 |
279 | # The maker that will generate values
280 | self._maker = maker
281 |
282 | # A set of existing values
283 | self._exclude = exclude or set([])
284 |
285 | # Flag indicating if the providers should be called in _assemble (True)
286 | # or _finish (False).
287 | self._assembler = assembler
288 |
289 | # A set of used values
290 | self._used_values = set(self._exclude)
291 |
292 | # The maximum number of attempts to generate unique data that should be
293 | # performed.
294 | self._max_attempts = max_attempts
295 |
296 | def reset(self):
297 | """Reset the set of used values"""
298 | self._used_values = set(self._exclude)
299 |
300 | def _get_unique(self, *args):
301 | """Generate a unique value using the assigned maker"""
302 |
303 | # Generate a unique values
304 | value = ''
305 | attempts = 0
306 | while True:
307 | attempts += 1
308 | value = self._maker(*args)
309 | if value not in self._used_values:
310 | break
311 |
312 | assert attempts < self._max_attempts, \
313 | 'Too many attempts to generate a unique value'
314 |
315 | # Add the value to the set of used values
316 | self._used_values.add(value)
317 |
318 | return value
319 |
320 | def _assemble(self):
321 | if not self._assembler:
322 | return self._maker()
323 | return self._get_unique()
324 |
325 | def _finish(self, value):
326 | if self._assembler:
327 | with self._maker.target(self.document):
328 | value = self._maker(value)
329 | return self._get_unique(value)
--------------------------------------------------------------------------------
/mongoframes/factory/makers/dates.py:
--------------------------------------------------------------------------------
1 | import datetime
2 | import math
3 | import random
4 | import re
5 |
6 | from mongoframes.factory.makers import Maker
7 |
8 | __all__ = [
9 | 'DateBetween'
10 | ]
11 |
12 |
13 | class DateBetween(Maker):
14 | """
15 | Return a date between two dates. Dates can be specified either as
16 | `datetime.date` instances or as strings of the form:
17 |
18 | "{yesterday|today|tomorrow}{+|-}{no_of_days}"
19 | """
20 |
21 | def __init__(self, min_date, max_date):
22 | super().__init__()
23 |
24 | # The date range between which a date will be selected
25 | self._min_date = min_date
26 | self._max_date = max_date
27 |
28 | def _assemble(self):
29 | min_date = self.parse_date(self._min_date)
30 | max_date = self.parse_date(self._max_date)
31 | seconds = random.randint(0, int((max_date - min_date).total_seconds()))
32 | return math.floor(seconds / 86400)
33 |
34 | def _finish(self, value):
35 | min_date = self.parse_date(self._min_date)
36 | return min_date + datetime.timedelta(seconds=value * 86400)
37 |
38 | @staticmethod
39 | def parse_date(d):
40 | if isinstance(d, datetime.date):
41 | return d
42 |
43 | # Parse the date string
44 | result = re.match(
45 | '(today|tomorrow|yesterday)((\-|\+)(\d+)){0,1}',
46 | d
47 | )
48 | assert result, 'Not a valid date string'
49 |
50 | # Determine the base date
51 | if result.groups()[0] == 'today':
52 | d = datetime.date.today()
53 |
54 | elif result.groups()[0] == 'tomorrow':
55 | d = datetime.date.today() + datetime.timedelta(days=1)
56 |
57 | elif result.groups()[0] == 'yesterday':
58 | d = datetime.date.today() - datetime.timedelta(days=1)
59 |
60 | # Add any offset
61 | if result.groups()[1]:
62 | op = result.groups()[2]
63 | days = int(result.groups()[3])
64 | if op == '+':
65 | d += datetime.timedelta(days=days)
66 | else:
67 | d -= datetime.timedelta(days=days)
68 |
69 | return d
--------------------------------------------------------------------------------
/mongoframes/factory/makers/images.py:
--------------------------------------------------------------------------------
1 | from urllib.parse import urlencode
2 |
3 | from mongoframes.factory.makers import Maker
4 |
5 | __all__ = [
6 | 'ImageURL'
7 | ]
8 |
9 |
10 | class ImageURL(Maker):
11 | """
12 | Return a fake image URL (by default we use the `fakeimg.pl` service).
13 | """
14 |
15 | def __init__(self,
16 | width,
17 | height,
18 | background='CCCCCC',
19 | foreground='8D8D8D',
20 | options=None,
21 | service_url='//fakeimg.pl',
22 | service_formatter=None
23 | ):
24 | super().__init__()
25 |
26 | # The size of the image to generate
27 | self._width = width
28 | self._height = height
29 |
30 | # The foreground/background colours for the image
31 | self._background = background
32 | self._foreground = foreground
33 |
34 | # A dictionary of options used when generating the image
35 | self._options = options
36 |
37 | # The service URL to use when calling the service
38 | self._service_url = service_url
39 |
40 | # A formatter function that can produce an image URL for the service
41 | self._service_formatter = service_formatter or \
42 | ImageURL._default_service_formatter
43 |
44 | def _assemble(self):
45 | return self._service_formatter(
46 | self._service_url,
47 | int(self._width),
48 | int(self._height),
49 | self._background,
50 | self._foreground,
51 | self._options
52 | )
53 |
54 | @staticmethod
55 | def _default_service_formatter(
56 | service_url,
57 | width,
58 | height,
59 | background,
60 | foreground,
61 | options
62 | ):
63 | """Generate an image URL for a service"""
64 |
65 | # Build the base URL
66 | image_tmp = '{service_url}/{width}x{height}/{background}/{foreground}/'
67 | image_url = image_tmp.format(
68 | service_url=service_url,
69 | width=width,
70 | height=height,
71 | background=background,
72 | foreground=foreground
73 | )
74 |
75 | # Add any options
76 | if options:
77 | image_url += '?' + urlencode(options)
78 |
79 | return image_url
80 |
--------------------------------------------------------------------------------
/mongoframes/factory/makers/numbers.py:
--------------------------------------------------------------------------------
1 | import random
2 |
3 | from mongoframes.factory.makers import Maker
4 |
5 | __all__ = [
6 | 'Counter'
7 | ]
8 |
9 |
10 | class Counter(Maker):
11 | """
12 | Generate a sequence of numbers.
13 | """
14 |
15 | def __init__(self, start_from=1, step=1):
16 | super().__init__()
17 |
18 | self._start_from = int(start_from)
19 | self._step = step
20 | self._counter = self._start_from
21 |
22 | def reset(self):
23 | self._counter = int(self._start_from)
24 |
25 | def _assemble(self):
26 | value = self._counter
27 | self._counter += int(self._step)
28 | return value
29 |
30 |
31 | class Float(Maker):
32 | """
33 | Generate a random float between two values.
34 | """
35 |
36 | def __init__(self, min_value, max_value):
37 | super().__init__()
38 |
39 | self._min_value = min_value
40 | self._max_value = max_value
41 |
42 | def _assemble(self):
43 | return random.uniform(float(self._min_value), float(self._max_value))
44 |
45 |
46 | class Int(Float):
47 | """
48 | Generate a random integer between two values.
49 | """
50 |
51 | def _assemble(self):
52 | return random.randint(int(self._min_value), int(self._max_value))
--------------------------------------------------------------------------------
/mongoframes/factory/makers/selections.py:
--------------------------------------------------------------------------------
1 | from functools import reduce
2 | import itertools
3 | import math
4 | import random
5 |
6 | from mongoframes.factory.makers import Maker
7 | from mongoframes import ASC
8 |
9 | __all__ = [
10 | 'Cycle',
11 | 'OneOf',
12 | 'RandomReference',
13 | 'SomeOf'
14 | ]
15 |
16 |
17 | class Cycle(Maker):
18 | """
19 | Pick the next item from a list of makers and/or values cycling through the
20 | list and repeating when we reach the end.
21 | """
22 |
23 | def __init__(self, items):
24 | super().__init__()
25 |
26 | # The list of makers/values to select from
27 | self._items = items
28 |
29 | # The index of the item that will be returned next
30 | self._item_index = 0
31 |
32 | def reset(self):
33 | """Reset the item index"""
34 | self._item_index = 0
35 |
36 | def _assemble(self):
37 | # Select the next item
38 | item_index = self._item_index
39 | item = self._items[item_index]
40 |
41 | # Move the index on 1 (and wrap it if we are at the end of the list)
42 | self._item_index += 1
43 | if self._item_index >= len(self._items):
44 | self._item_index = 0
45 |
46 | # Return the index and it's assembled value
47 | if isinstance(item, Maker):
48 | return [item_index, item._assemble()]
49 | return [item_index, None]
50 |
51 | def _finish(self, value):
52 | item = self._items[value[0]]
53 | if isinstance(item, Maker):
54 | with item.target(self.document):
55 | return item._finish(value[1])
56 | return item
57 |
58 |
59 | class OneOf(Maker):
60 | """
61 | Pick one item from a list of makers and/or values.
62 | """
63 |
64 | def __init__(self, items, weights=None):
65 | super().__init__()
66 |
67 | # The list of makers/values to select from
68 | self._items = items
69 |
70 | # The weighting to apply when selecting a maker/value
71 | self._weights = weights
72 |
73 | def _assemble(self):
74 | # Select an item
75 | item_index = 0
76 | if self._weights:
77 | item_index = self.weighted(self._weights)
78 | else:
79 | item_index = random.randint(0, len(self._items) - 1)
80 |
81 | # Return the index and it's assembled value
82 | item = self._items[item_index]
83 | if isinstance(item, Maker):
84 | return [item_index, item._assemble()]
85 | return [item_index, None]
86 |
87 | def _finish(self, value):
88 | item = self._items[value[0]]
89 | if isinstance(item, Maker):
90 | with item.target(self.document):
91 | return item._finish(value[1])
92 | return item
93 |
94 | @staticmethod
95 | def weighted(weights):
96 | """
97 | Return a random integer 0 <= N <= len(weights) - 1, where the weights
98 | determine the probability of each possible integer.
99 |
100 | Based on this StackOverflow post by Ned Batchelder:
101 |
102 | http://stackoverflow.com/questions/3679694/a-weighted-version-of-random-choice
103 | """
104 |
105 | # Convert weights to floats
106 | weights = [float(w) for w in weights]
107 |
108 | # Pick a value at random
109 | choice = random.uniform(0, sum(weights))
110 |
111 | # Find the value
112 | position = 0
113 | for i, weight in enumerate(weights):
114 | if position + weight >= choice:
115 | return i
116 | position += weight
117 |
118 |
119 | class RandomReference(Maker):
120 | """
121 | Pick a reference document at random from a collection (determined by the
122 | given frame_cls) optionally applying a constraint.
123 | """
124 |
125 | def __init__(self, frame_cls, constraint=None):
126 | super().__init__()
127 |
128 | # The frame class that will be used to select a reference with
129 | self._frame_cls = frame_cls
130 |
131 | # The constraint applied when select a reference document
132 | self._constraint = constraint or {}
133 |
134 | def _assemble(self):
135 | # Select a random float that will be used to select a reference document
136 | # by it's position.
137 | return random.random()
138 |
139 | def _finish(self, value):
140 | # Count the number of documents available to pick from
141 | total_documents = self._frame_cls.count(self._constraint)
142 |
143 | # Calculate the position of the document we've picked
144 | position = math.floor(total_documents * value)
145 |
146 | # Select the document
147 | document = self._frame_cls.one(
148 | self._constraint,
149 | limit=1,
150 | skip=position,
151 | sort=[('_id', ASC)],
152 | projection={'_id': True}
153 | )
154 |
155 | # Check the document was found
156 | if document:
157 | return document._id
158 |
159 | return None
160 |
161 |
162 | class SomeOf(Maker):
163 | """
164 | Pick one or more items from a list of makers and/or values.
165 | """
166 |
167 | def __init__(
168 | self,
169 | items,
170 | sample_size,
171 | weights=None,
172 | with_replacement=False
173 | ):
174 | super().__init__()
175 |
176 | # The list of makers/values to select from
177 | self._items = items
178 |
179 | # The number of items to pick
180 | self._sample_size = sample_size
181 |
182 | # The weighting to apply when selecting a maker/value
183 | self._weights = weights
184 |
185 | # A flag indicating if the same item can be selected from the list more
186 | # than once.
187 | self._with_replacement = with_replacement
188 |
189 | def _assemble(self):
190 | # Select some items
191 | sample_size = int(self._sample_size)
192 |
193 | sample_indexes = []
194 | if self._weights:
195 | sample_indexes = self.weighted(
196 | self._weights,
197 | sample_size,
198 | with_replacement=self._with_replacement
199 | )
200 | else:
201 | sample_range = range(0, len(self._items))
202 | if self._with_replacement:
203 | sample_indexes = [random.choice(sample_range) \
204 | for s in range(0, sample_size)]
205 | else:
206 | sample_indexes = random.sample(sample_range, sample_size)
207 |
208 | # Return the sampled indexes and their assembled values
209 | values = []
210 | for sample_index in sample_indexes:
211 | item = self._items[sample_index]
212 | if isinstance(item, Maker):
213 | values.append([sample_index, item._assemble()])
214 | else:
215 | values.append([sample_index, None])
216 |
217 | return values
218 |
219 | def _finish(self, value):
220 | values = []
221 | for sample in value:
222 | item = self._items[sample[0]]
223 | if isinstance(item, Maker):
224 | with item.target(self.document):
225 | values.append(item._finish(sample[1]))
226 | else:
227 | values.append(item)
228 | return values
229 |
230 | @staticmethod
231 | def p(i, sample_size, weights):
232 | """
233 | Given a weighted set and sample size return the probabilty that the
234 | weight `i` will be present in the sample.
235 |
236 | Created to test the output of the `SomeOf` maker class. The math was
237 | provided by Andy Blackshaw - thank you dad :)
238 | """
239 |
240 | # Determine the initial pick values
241 | weight_i = weights[i]
242 | weights_sum = sum(weights)
243 |
244 | # Build a list of weights that don't contain the weight `i`. This list will
245 | # be used to build the possible picks before weight `i`.
246 | other_weights = list(weights)
247 | del other_weights[i]
248 |
249 | # Calculate the probability
250 | probability_of_i = 0
251 | for picks in range(0, sample_size):
252 |
253 | # Build the list of possible permutations for this pick in the sample
254 | permutations = list(itertools.permutations(other_weights, picks))
255 |
256 | # Calculate the probability for this permutation
257 | permutation_probabilities = []
258 | for permutation in permutations:
259 |
260 | # Calculate the probability for each pick in the permutation
261 | pick_probabilities = []
262 | pick_weight_sum = weights_sum
263 |
264 | for pick in permutation:
265 | pick_probabilities.append(pick / pick_weight_sum)
266 |
267 | # Each time we pick we update the sum of the weight the next
268 | # pick is from.
269 | pick_weight_sum -= pick
270 |
271 | # Add the probability of picking i as the last pick
272 | pick_probabilities += [weight_i / pick_weight_sum]
273 |
274 | # Multiply all the probabilities for the permutation together
275 | permutation_probability = reduce(
276 | lambda x, y: x * y, pick_probabilities
277 | )
278 | permutation_probabilities.append(permutation_probability)
279 |
280 | # Add together all the probabilities for all permutations together
281 | probability_of_i += sum(permutation_probabilities)
282 |
283 | return probability_of_i
284 |
285 | @staticmethod
286 | def weighted(weights, sample_size, with_replacement=False):
287 | """
288 | Return a set of random integers 0 <= N <= len(weights) - 1, where the
289 | weights determine the probability of each possible integer in the set.
290 | """
291 | assert sample_size <= len(weights), "The sample size must be smaller \
292 | than or equal to the number of weights it's taken from."
293 |
294 | # Convert weights to floats
295 | weights = [float(w) for w in weights]
296 | weight_indexes = list(range(0, len(weights)))
297 |
298 | samples = []
299 | while len(samples) < sample_size:
300 | # Choice a weight
301 | sample = OneOf.weighted(weights)
302 |
303 | # Add the choosen weight to our samples
304 | samples.append(weight_indexes[sample])
305 |
306 | if not with_replacement:
307 | # Remove the weight from the list of weights we can select from
308 | del weights[sample]
309 | del weight_indexes[sample]
310 |
311 | return samples
312 |
313 |
314 |
--------------------------------------------------------------------------------
/mongoframes/factory/makers/text.py:
--------------------------------------------------------------------------------
1 | import random
2 | import re
3 | import string
4 |
5 | from mongoframes.factory.makers import Faker, Maker
6 |
7 | __all__ = [
8 | 'Code',
9 | 'Join',
10 | 'Lorem',
11 | 'Markov'
12 | 'Sequence'
13 | ]
14 |
15 |
16 | class Code(Maker):
17 | """
18 | Generate a random code of the given length using either the default_charset
19 | or optionally a custom charset.
20 | """
21 |
22 | # The default character set that codes are made from
23 | default_charset = string.ascii_uppercase + string.digits
24 |
25 | def __init__(self, length, charset=None):
26 | super().__init__()
27 |
28 | # If no charset is provided use the default
29 | if not charset:
30 | charset = self.default_charset
31 |
32 | # The length of the code that will be generated
33 | self._length = length
34 |
35 | # The character set the code will be made from
36 | self._charset = list(charset)
37 |
38 | def _assemble(self):
39 | return ''.join([
40 | random.choice(self._charset) for i in range(0, int(self._length))
41 | ])
42 |
43 |
44 | class Join(Maker):
45 | """
46 | Join the output of 2 or more items (makers and/or values) together with a
47 | seperator string.
48 | """
49 |
50 | def __init__(self, items, sep=' '):
51 | super().__init__()
52 |
53 | # The list of makers/values that will be joined
54 | self._items = items
55 |
56 | # The string used to join the items together
57 | self._sep = sep
58 |
59 | def _assemble(self):
60 | values = []
61 | for item in self._items:
62 | if isinstance(item, Maker):
63 | values.append(item._assemble())
64 | else:
65 | values.append(item)
66 | return values
67 |
68 | def _finish(self, value):
69 | parts = []
70 | for i, item in enumerate(self._items):
71 | if isinstance(item, Maker):
72 | parts.append(str(item._finish(value[i])))
73 | else:
74 | parts.append(str(item))
75 | return self._sep.join(parts)
76 |
77 |
78 | class Lorem(Maker):
79 | """
80 | Generate random amounts of lorem ipsum.
81 |
82 | To determine the amount of text generated the type of text structure to
83 | generate must be specified;
84 |
85 | - body,
86 | - paragraph,
87 | - sentence
88 |
89 | along with the quantity;
90 |
91 | - paragraphs in a body,
92 | - sentences in a paragraph,
93 | - words in a scentence.
94 | """
95 |
96 | def __init__(self, text_type, quantity):
97 | super().__init__()
98 |
99 | # The type of text structure to generate
100 | self._text_type = text_type
101 |
102 | # The quantity of text to generate
103 | self._quantity = quantity
104 |
105 | assert self._text_type in ['body', 'paragraph', 'sentence'], \
106 | 'Not a supported text type'
107 |
108 | def _assemble(self):
109 | quantity = int(self._quantity)
110 |
111 | if self._text_type == 'body':
112 | return '\n'.join(Faker.get_fake().paragraphs(nb=quantity))
113 |
114 | if self._text_type == 'paragraph':
115 | return Faker.get_fake().paragraph(
116 | nb_sentences=quantity,
117 | variable_nb_sentences=False
118 | )
119 |
120 | if self._text_type == 'sentence':
121 | return Faker.get_fake().sentence(
122 | nb_words=quantity,
123 | variable_nb_words=False
124 | )
125 |
126 |
127 | class Markov(Maker):
128 | """
129 | Generate random amounts of text using a Markov chain.
130 |
131 | To determine the amount of text generated the type of text structure to
132 | generate must be specified;
133 |
134 | - body,
135 | - paragraph,
136 | - sentence
137 |
138 | along with the quantity;
139 |
140 | - paragraphs in a body,
141 | - sentences in a paragraph,
142 | - words in a sentence.
143 |
144 | This code is heavily based on (lifted from) the code presented in this
145 | article by Shabda Raaj:
146 | http://agiliq.com/blog/2009/06/generating-pseudo-random-text-with-markov-chains-u/
147 | """
148 |
149 | # FIXME:
150 | # w1, w2 = w2, random.choice(db['freqs'][(w1, w2)])
151 | # KeyError: ('summer...."\n\nTHE', 'END')
152 |
153 | _dbs = {}
154 |
155 | def __init__(self, db, text_type, quantity):
156 | super().__init__()
157 |
158 | # The database to generate the text from
159 | self._db = db
160 |
161 | assert db in self.__class__._dbs, 'Word database does not exist'
162 |
163 | # The type of text structure to generate
164 | self._text_type = text_type
165 |
166 | # The quantity of text to generate
167 | self._quantity = quantity
168 |
169 | assert self._text_type in ['body', 'paragraph', 'sentence'], \
170 | 'Not a supported text type'
171 |
172 | # Public methds
173 |
174 | @property
175 | def database(self):
176 | """Return the selected word database"""
177 | return self.__class__._dbs[self._db]
178 |
179 | # Private methods
180 |
181 | def _assemble(self):
182 | quantity = int(self._quantity)
183 |
184 | if self._text_type == 'body':
185 | return self._body(quantity)
186 |
187 | if self._text_type == 'paragraph':
188 | return self._paragraph(quantity)
189 |
190 | if self._text_type == 'sentence':
191 | return self._sentence(quantity)
192 |
193 | def _body(self, paragraphs):
194 | """Generate a body of text"""
195 | body = []
196 | for i in range(paragraphs):
197 | paragraph = self._paragraph(random.randint(1, 10))
198 | body.append(paragraph)
199 |
200 | return '\n'.join(body)
201 |
202 | def _paragraph(self, sentences):
203 | """Generate a paragraph"""
204 | paragraph = []
205 | for i in range(sentences):
206 | sentence = self._sentence(random.randint(5, 16))
207 | paragraph.append(sentence)
208 |
209 | return ' '.join(paragraph)
210 |
211 | def _sentence(self, words):
212 | """Generate a sentence"""
213 | db = self.database
214 |
215 | # Generate 2 words to start a sentence with
216 | seed = random.randint(0, db['word_count'] - 3)
217 | seed_word, next_word = db['words'][seed], db['words'][seed + 1]
218 | w1, w2 = seed_word, next_word
219 |
220 | # Generate the complete sentence
221 | sentence = []
222 | for i in range(0, words - 1):
223 | sentence.append(w1)
224 | w1, w2 = w2, random.choice(db['freqs'][(w1, w2)])
225 | sentence.append(w2)
226 |
227 | # Make the sentence respectable
228 | sentence = ' '.join(sentence)
229 |
230 | # Capitalize the sentence
231 | sentence = sentence.capitalize()
232 |
233 | # Remove additional sentence ending puntuation
234 | sentence = sentence.replace('.', '')
235 | sentence = sentence.replace('!', '')
236 | sentence = sentence.replace('?', '')
237 | sentence = sentence.replace(':', '')
238 |
239 | # Remove quote tags
240 | sentence = sentence.replace('.', '')
241 | sentence = sentence.replace('!', '')
242 | sentence = sentence.replace('?', '')
243 | sentence = sentence.replace(':', '')
244 | sentence = sentence.replace('"', '')
245 |
246 | # If the last character is not an alphanumeric remove it
247 | sentence = re.sub('[^a-zA-Z0-9]$', '', sentence)
248 |
249 | # Remove excess space
250 | sentence = re.sub('\s+', ' ', sentence)
251 |
252 | # Add a full stop
253 | sentence += '.'
254 |
255 | return sentence
256 |
257 | @classmethod
258 | def init_word_db(cls, name, text):
259 | """Initialize a database of words for the maker with the given name"""
260 | # Prep the words
261 | text = text.replace('\n', ' ').replace('\r', ' ')
262 | words = [w.strip() for w in text.split(' ') if w.strip()]
263 |
264 | assert len(words) > 2, \
265 | 'Database text sources must contain 3 or more words.'
266 |
267 | # Build the database
268 | freqs = {}
269 | for i in range(len(words) - 2):
270 |
271 | # Create a triplet from the current word
272 | w1 = words[i]
273 | w2 = words[i + 1]
274 | w3 = words[i + 2]
275 |
276 | # Add the triplet to the database
277 | key = (w1, w2)
278 | if key in freqs:
279 | freqs[key].append(w3)
280 | else:
281 | freqs[key] = [w3]
282 |
283 | # Store the database so it can be used
284 | cls._dbs[name] = {
285 | 'freqs': freqs,
286 | 'words': words,
287 | 'word_count': len(words) - 2
288 | }
289 |
290 |
291 | class Sequence(Maker):
292 | """
293 | Generate a sequence of values where a number is inserted into a template.
294 | The template should specify an index value, for example:
295 |
296 | "prefix-{index}"
297 | """
298 |
299 | def __init__(self, template, start_from=1):
300 | super().__init__()
301 |
302 | self._template = template
303 | self._start_from = start_from
304 | self._index = start_from
305 |
306 | def reset(self):
307 | self._index = self._start_from
308 |
309 | def _assemble(self):
310 | value = self._template.format(index=self._index)
311 | self._index += 1
312 | return value
--------------------------------------------------------------------------------
/mongoframes/factory/quotas.py:
--------------------------------------------------------------------------------
1 | import random
2 |
3 | __all__ = [
4 | 'Gauss',
5 | 'Random'
6 | ]
7 |
8 |
9 | class Quota:
10 | """
11 | A base class for implementing variable quota (although the base class
12 | provides a fixed value and is no different than using an integer or float
13 | value).
14 |
15 | The Quota class can be safely used as an argument for `Factory`s and
16 | `Maker`s.
17 | """
18 |
19 | def __init__(self, quantity):
20 | self._quantity = quantity
21 |
22 | def __int__(self):
23 | return int(self._quantity)
24 |
25 | def __float__(self):
26 | return float(self._quantity)
27 |
28 |
29 | class Gauss(Quota):
30 | """
31 | Return a random quota using a Gaussian distribution.
32 | """
33 |
34 | def __init__(self, mu, sigma):
35 | self._mu = mu
36 | self._sigma = sigma
37 |
38 | def __int__(self):
39 | return int(float(self))
40 |
41 | def __float__(self):
42 | return random.gauss(self._mu, self._sigma)
43 |
44 |
45 | class Random(Quota):
46 | """
47 | Return a random quota between two values.
48 | """
49 |
50 | def __init__(self, min_quantity, max_quantity):
51 | self._min_quantity = min_quantity
52 | self._max_quantity = max_quantity
53 |
54 | def __int__(self):
55 | return random.randint(self._min_quantity, self._max_quantity)
56 |
57 | def __float__(self):
58 | return random.uniform(self._min_quantity, self._max_quantity)
--------------------------------------------------------------------------------
/mongoframes/pagination.py:
--------------------------------------------------------------------------------
1 | """
2 | Support for paginating frames.
3 | """
4 |
5 | from copy import deepcopy
6 | import math
7 |
8 | from mongoframes.queries import Condition, Group, to_refs
9 |
10 |
11 | __all__ = (
12 | # Exceptions
13 | 'InvalidPage',
14 |
15 | # Classes
16 | 'Page',
17 | 'Paginator'
18 | )
19 |
20 |
21 | class InvalidPage(Exception):
22 | """
23 | An error raised when an invalid page is requested.
24 | """
25 |
26 |
27 | class Page(object):
28 | """
29 | A class to represent one page of results.
30 | """
31 |
32 | def __init__(self, offset, number, items, next, prev):
33 |
34 | # The offset of the first result in the page from the first result of
35 | # the entire selection.
36 | self._offset = offset
37 |
38 | # The page number
39 | self._number = number
40 |
41 | # The results/frames for this page
42 | self._items = list(items)
43 |
44 | # The next and previous page numbers (if there is no next/previous page
45 | # the value will be None.
46 | self._next = next
47 | self._prev = prev
48 |
49 | def __getitem__(self, i):
50 | return self._items[i]
51 |
52 | def __iter__(self):
53 | for item in self._items:
54 | yield item
55 |
56 | def __len__(self):
57 | return len(self._items)
58 |
59 | # Read-only properties
60 |
61 | @property
62 | def items(self):
63 | """Return a list of results for the page"""
64 | return self._items
65 |
66 | @property
67 | def next(self):
68 | """
69 | Return the page number for the next page or None if there isn't one.
70 | """
71 | return self._next
72 |
73 | @property
74 | def number(self):
75 | """Return the page number"""
76 | return self._number
77 |
78 | @property
79 | def prev(self):
80 | """
81 | Return the page number for the previous page or None if there isn't one.
82 | """
83 | return self._prev
84 |
85 | # Public methods
86 |
87 | def offset(self, item):
88 | """Return the offset for an item in the page"""
89 | return self._offset + self.items.index(item)
90 |
91 |
92 | class Paginator(object):
93 | """
94 | A pagination class for slicing query results into pages. This class is
95 | designed to work with Frame classes.
96 | """
97 |
98 | def __init__(
99 | self,
100 | frame_cls,
101 | filter=None,
102 | per_page=20,
103 | orphans=0,
104 | **filter_args
105 | ):
106 |
107 | # The frame class results are being paginated for
108 | self._frame_cls = frame_cls
109 |
110 | # The filter applied when selecting results from the database (we
111 | # flattern the filter at this point which effectively deep copies.
112 | if isinstance(filter, (Condition, Group)):
113 | self._filter = filter.to_dict()
114 | else:
115 | self._filter = to_refs(filter)
116 |
117 | # Any additional filter arguments applied when selecting results such as
118 | # sort and projection,
119 | self._filter_args = filter_args
120 |
121 | # The number of results that will be displayed per page
122 | self._per_page = per_page
123 |
124 | # If a value is specified for orphans then the last page will be able to
125 | # hold the additional results (up to the value of orphans). This can
126 | # help prevent users being presented with pages contain only a few
127 | # results.
128 | self._orphans = orphans
129 |
130 | # Count the total results being paginated
131 | self._items_count = frame_cls.count(self._filter)
132 |
133 | # Calculated the number of pages
134 | total = self._items_count - orphans
135 | self._page_count = max(1, int(math.ceil(total / float(self._per_page))))
136 |
137 | # Create a list of page number that can be used to navigate the results
138 | self._page_numbers = range(1, self._page_count + 1)
139 |
140 | def __getitem__(self, page_number):
141 | if page_number not in self._page_numbers:
142 | raise InvalidPage(page_number, self.page_count)
143 |
144 | # Calculate the next and previous page numbers
145 | next = page_number + 1 if page_number + 1 in self._page_numbers else None
146 | prev = page_number - 1 if page_number - 1 in self._page_numbers else None
147 |
148 | # Select the items for the page
149 | filter_args = deepcopy(self._filter_args) or {}
150 | filter_args['skip'] = (page_number - 1) * self._per_page
151 | filter_args['limit'] = self._per_page
152 |
153 | # Check to see if we need to account for orphans
154 | if self.item_count - (page_number * self._per_page) <= self.orphans:
155 | filter_args['limit'] += self.orphans
156 |
157 | # Select the results for the page
158 | items = self._frame_cls.many(self._filter, **filter_args)
159 |
160 | # Build the page
161 | return Page(
162 | offset=filter_args['skip'],
163 | number=page_number,
164 | items=items,
165 | next=next,
166 | prev=prev
167 | )
168 |
169 | def __iter__(self):
170 | for page_number in self._page_numbers:
171 | yield self[page_number]
172 |
173 | # Read-only properties
174 |
175 | @property
176 | def item_count(self):
177 | """Return the total number of items being paginated"""
178 | return self._items_count
179 |
180 | @property
181 | def orphans(self):
182 | """
183 | Return the number of orphan results that will be allowed in the last
184 | page of results.
185 | """
186 | return self._orphans
187 |
188 | @property
189 | def page_count(self):
190 | """Return the total number of pages"""
191 | return self._page_count
192 |
193 | @property
194 | def page_numbers(self):
195 | """Return a list of page numbers"""
196 | return self._page_numbers
197 |
198 | @property
199 | def per_page(self):
200 | """
201 | Return the number of results per page (with the exception of orphans).
202 | """
203 | return self._per_page
--------------------------------------------------------------------------------
/mongoframes/queries.py:
--------------------------------------------------------------------------------
1 | """
2 | A set of helpers to simplify the creation of MongoDB queries.
3 | """
4 |
5 | import re
6 |
7 | from pymongo import (ASCENDING, DESCENDING)
8 |
9 | __all__ = [
10 | # Queries
11 | 'Q',
12 |
13 | # Operators
14 | 'All',
15 | 'ElemMatch',
16 | 'Exists',
17 | 'In',
18 | 'Not',
19 | 'NotIn',
20 | 'Size',
21 | 'Type',
22 |
23 | # Groups
24 | 'And',
25 | 'Or',
26 | 'Nor',
27 |
28 | # Sorting
29 | 'SortBy',
30 |
31 | # Utils
32 | 'to_refs'
33 | ]
34 |
35 |
36 | # Queries
37 |
38 | class Condition:
39 | """
40 | A query condition of the form `{path: {operator: value}}`.
41 | """
42 |
43 | def __init__(self, q, value, operator):
44 | self.q = q
45 | self.value = value
46 | self.operator = operator
47 |
48 | def to_dict(self):
49 | """Return a dictionary suitable for use with pymongo as a filter"""
50 | if self.operator == '$eq':
51 | return {self.q: self.value}
52 | if self.q is None:
53 | return {self.operator: self.value}
54 | return {self.q: {self.operator: self.value}}
55 |
56 |
57 | class QMeta(type):
58 | """
59 | Meta-class for query builder.
60 | """
61 |
62 | def __getattr__(self, name):
63 | return Q(name)
64 |
65 | def __getitem__(self, name):
66 | return Q(name)
67 |
68 | def __eq__(self, other):
69 | return Condition(None, other, '$eq')
70 |
71 | def __ge__(self, other):
72 | return Condition(None, other, '$gte')
73 |
74 | def __gt__(self, other):
75 | return Condition(None, other, '$gt')
76 |
77 | def __le__(self, other):
78 | return Condition(None, other, '$lte')
79 |
80 | def __lt__(self, other):
81 | return Condition(None, other, '$lt')
82 |
83 | def __ne__(self, other):
84 | return Condition(None, other, '$ne')
85 |
86 |
87 | class Q(metaclass=QMeta):
88 | """
89 | Start point for the query creation, the Q class is a special type of class
90 | that's typically initialized by appending an attribute, for example:
91 |
92 | Q.hit_points > 100
93 |
94 | """
95 |
96 | def __init__(self, path):
97 | self._path = path
98 |
99 | def __eq__(self, other):
100 | return Condition(self._path, other, '$eq')
101 |
102 | def __ge__(self, other):
103 | return Condition(self._path, other, '$gte')
104 |
105 | def __gt__(self, other):
106 | return Condition(self._path, other, '$gt')
107 |
108 | def __le__(self, other):
109 | return Condition(self._path, other, '$lte')
110 |
111 | def __lt__(self, other):
112 | return Condition(self._path, other, '$lt')
113 |
114 | def __ne__(self, other):
115 | return Condition(self._path, other, '$ne')
116 |
117 | def __getattr__(self, name):
118 | self._path = '{0}.{1}'.format(self._path, name)
119 | return self
120 |
121 | def __getitem__(self, name):
122 | self._path = '{0}.{1}'.format(self._path, name)
123 | return self
124 |
125 |
126 | # Operators
127 |
128 | def All(q, value):
129 | """
130 | The All operator selects documents where the value of the field is an list
131 | that contains all the specified elements.
132 | """
133 | return Condition(q._path, to_refs(value), '$all')
134 |
135 | def ElemMatch(q, *conditions):
136 | """
137 | The ElemMatch operator matches documents that contain an array field with at
138 | least one element that matches all the specified query criteria.
139 | """
140 | new_condition = {}
141 | for condition in conditions:
142 |
143 | if isinstance(condition, (Condition, Group)):
144 | condition = condition.to_dict()
145 |
146 | deep_merge(condition, new_condition)
147 |
148 | return Condition(q._path, new_condition, '$elemMatch')
149 |
150 | def Exists(q, value):
151 | """
152 | When exists is True, Exists matches the documents that contain the field,
153 | including documents where the field value is null. If exists is False, the
154 | query returns only the documents that do not contain the field.
155 | """
156 | return Condition(q._path, value, '$exists')
157 |
158 | def In(q, value):
159 | """
160 | The In operator selects the documents where the value of a field equals any
161 | value in the specified list.
162 | """
163 | return Condition(q._path, to_refs(value), '$in')
164 |
165 | def Not(condition):
166 | """
167 | Not performs a logical NOT operation on the specified condition and selects
168 | the documents that do not match. This includes documents that do not contain
169 | the field.
170 | """
171 | return Condition(
172 | condition.q,
173 | {condition.operator: condition.value},
174 | '$not'
175 | )
176 |
177 | def NotIn(q, value):
178 | """
179 | The NotIn operator selects documents where the field value is not in the
180 | specified list or the field does not exists.
181 | """
182 | return Condition(q._path, to_refs(value), '$nin')
183 |
184 | def Size(q, value):
185 | """
186 | The Size operator matches any list with the number of elements specified by
187 | size.
188 | """
189 | return Condition(q._path, value, '$size')
190 |
191 | def Type(q, value):
192 | """
193 | Type selects documents where the value of the field is an instance of the
194 | specified BSON type.
195 | """
196 | return Condition(q._path, value, '$type')
197 |
198 |
199 | # Groups
200 |
201 | class Group:
202 | """
203 | The Group class is used as a base class for operators that group together
204 | two or more conditions.
205 | """
206 |
207 | operator = ''
208 |
209 | def __init__(self, *conditions):
210 | self.conditions = conditions
211 |
212 | def to_dict(self):
213 | """Return a dictionary suitable for use with pymongo as a filter"""
214 | raw_conditions = []
215 | for condition in self.conditions:
216 | if isinstance(condition, (Condition, Group)):
217 | raw_conditions.append(condition.to_dict())
218 | else:
219 | raw_conditions.append(condition)
220 | return {self.operator: raw_conditions}
221 |
222 |
223 | class And(Group):
224 | """
225 | And performs a logical AND operation on a list of two or more conditions and
226 | selects the documents that satisfy all the conditions.
227 | """
228 |
229 | operator = '$and'
230 |
231 |
232 | class Or(Group):
233 | """
234 | The Or operator performs a logical OR operation on a list of two or more
235 | conditions and selects the documents that satisfy at least one of the
236 | conditions.
237 | """
238 |
239 | operator = '$or'
240 |
241 |
242 | class Nor(Group):
243 | """
244 | Nor performs a logical NOR operation on a list of one or more conditions and
245 | selects the documents that fail all the conditions.
246 | """
247 |
248 | operator = '$nor'
249 |
250 |
251 | # Sorting
252 |
253 | def SortBy(*qs):
254 | """Convert a list of Q objects into list of sort instructions"""
255 |
256 | sort = []
257 | for q in qs:
258 | if q._path.endswith('.desc'):
259 | sort.append((q._path[:-5], DESCENDING))
260 | else:
261 | sort.append((q._path, ASCENDING))
262 | return sort
263 |
264 |
265 | # Utils
266 |
267 | def deep_merge(source, dest):
268 | """
269 | Deep merges source dict into dest dict.
270 |
271 | This code was taken directly from the mongothon project:
272 | https://github.com/gamechanger/mongothon/tree/master/mongothon
273 | """
274 | for key, value in source.items():
275 | if key in dest:
276 | if isinstance(value, dict) and isinstance(dest[key], dict):
277 | deep_merge(value, dest[key])
278 | continue
279 | elif isinstance(value, list) and isinstance(dest[key], list):
280 | for item in value:
281 | if item not in dest[key]:
282 | dest[key].append(item)
283 | continue
284 | dest[key] = value
285 |
286 | def to_refs(value):
287 | """Convert all Frame instances within the given value to Ids"""
288 | from mongoframes.frames import Frame, SubFrame
289 |
290 | # Frame
291 | if isinstance(value, Frame):
292 | return value._id
293 |
294 | # SubFrame
295 | elif isinstance(value, SubFrame):
296 | return to_refs(value._document)
297 |
298 | # Lists
299 | elif isinstance(value, (list, set, tuple)):
300 | return [to_refs(v) for v in value]
301 |
302 | # Dictionaries
303 | elif isinstance(value, dict):
304 | return {k: to_refs(v) for k, v in value.items()}
305 |
306 | return value
307 |
--------------------------------------------------------------------------------
/performance/requirements.txt:
--------------------------------------------------------------------------------
1 | blinker==1.4
2 | fake-factory==0.5.8
3 | mongoengine==0.10.6
4 | MongoFrames==1.0.0
5 | pymongo==3.2.2
6 | python-dateutil==2.5.3
7 | six==1.10.0
--------------------------------------------------------------------------------
/performance/tests/__init__.py:
--------------------------------------------------------------------------------
1 | import random
2 | import time
3 |
4 | from faker import Faker
5 | from mongoengine import *
6 | from mongoframes import *
7 | from pymongo import MongoClient
8 |
9 | __all__ = [
10 | 'build_test_data',
11 | 'connect_to_db',
12 | 'time_it'
13 | ]
14 |
15 |
16 | class Company(Frame):
17 |
18 | _fields = {
19 | 'name',
20 | 'departments',
21 | 'address',
22 | 'tel',
23 | 'website_url'
24 | }
25 |
26 |
27 | class Department(SubFrame):
28 |
29 | _fields = {
30 | 'name',
31 | 'year_end',
32 | 'annual_budget'
33 | }
34 |
35 |
36 | class Employee(Frame):
37 |
38 | _fields = {
39 | 'first_name',
40 | 'last_name',
41 | 'dob',
42 | 'role',
43 | 'tel',
44 | 'email',
45 | 'annual_salary',
46 | 'ssn',
47 | 'company',
48 | 'department'
49 | }
50 |
51 |
52 | def build_test_data(
53 | total_companies=100,
54 | total_departments_per_company=10,
55 | total_employees_per_company=50
56 | ):
57 | """Add a context in which we can run the tests with the test data"""
58 |
59 | print('Building test data...')
60 |
61 | # Drop the database collections
62 | Company.get_collection().drop()
63 | Employee.get_collection().drop()
64 |
65 | # Create a faker object
66 | fake = Faker()
67 | fake.seed(11061979)
68 | random.seed(11061979)
69 |
70 | # Build companies
71 | companies = []
72 | for company_index in range(0, total_companies):
73 | company = Company(
74 | name=fake.company(),
75 | departments=[],
76 | address=fake.address(),
77 | tel=fake.phone_number(),
78 | website_url=fake.uri()
79 | )
80 |
81 | # Build departments
82 | for department_index in range(0, total_departments_per_company):
83 | department = Department(
84 | name=fake.word(),
85 | year_end=fake.date_time_this_year(False, True),
86 | annual_budget=fake.pyint()
87 | )
88 | company.departments.append(department)
89 |
90 | companies.append(company)
91 |
92 | Company.insert_many(companies)
93 |
94 | # Build employees
95 | for company in companies:
96 | for employee_index in range(0, total_employees_per_company):
97 | employee = Employee(
98 | first_name=fake.first_name(),
99 | last_name=fake.last_name(),
100 | dob=fake.date(),
101 | role=fake.job(),
102 | tel=fake.phone_number(),
103 | email=fake.email(),
104 | annual_salary=fake.pyint(),
105 | ssn=fake.ssn(),
106 | company=company,
107 | department=random.choice(company.departments)
108 | )
109 | employee.insert()
110 |
111 | def connect_to_db():
112 | """Connect to the database"""
113 |
114 | print('Connecting to database...')
115 |
116 | # Connect to database via MongoEngine
117 | connect('mongo_performance_tests')
118 |
119 | # Connect to database via MongoFrames
120 | Frame._client = MongoClient(
121 | 'mongodb://localhost:27017/mongo_performance_tests'
122 | )
123 |
124 | def time_it(func, calls, *args, **kwargs):
125 | """
126 | Call the given function X times and output the best, worst and average
127 | execution time.
128 | """
129 |
130 | # Call the function X times and record the times
131 | times = []
132 | for call_index in range(calls):
133 | start_time = time.time()
134 | func(*args, **kwargs)
135 | times.append(time.time() - start_time)
136 |
137 | # Get the best, worst and average times
138 | times.sort()
139 | best = times[0]
140 | worst = times[-1]
141 | average = sum(times) / float(len(times))
142 |
143 | print(
144 | '{func_name: <32} {best: 02.3f} {worst: 02.3f} {average: 02.3f}'.format(
145 | func_name=func.__name__,
146 | best=best,
147 | worst=worst,
148 | average=average
149 | )
150 | )
--------------------------------------------------------------------------------
/performance/tests/run_mongoengine.py:
--------------------------------------------------------------------------------
1 | from mongoengine import *
2 |
3 | from __init__ import build_test_data, connect_to_db, time_it
4 |
5 |
6 | # Define models
7 |
8 | class Company(Document):
9 |
10 | meta = {'collection': 'Company'}
11 |
12 | name = StringField()
13 | departments = EmbeddedDocumentListField('Department')
14 | address = StringField()
15 | tel = StringField()
16 | website_url = URLField()
17 |
18 |
19 | class Department(EmbeddedDocument):
20 |
21 | meta = {'collection': 'Department'}
22 |
23 | name = StringField()
24 | year_end = DateTimeField()
25 | annual_budget = IntField()
26 |
27 |
28 | class Employee(Document):
29 |
30 | meta = {'collection': 'Employee'}
31 |
32 | first_name = StringField()
33 | last_name = StringField()
34 | dob = StringField()
35 | role = StringField()
36 | tel = StringField()
37 | email = EmailField()
38 | annual_salary = IntField()
39 | ssn = StringField()
40 | company = ReferenceField(Company)
41 | department = StringField()
42 |
43 |
44 | # Define tests
45 |
46 | def test_flat_select():
47 | """Select all employees no dereferencing"""
48 | list(Employee.objects.no_dereference())
49 |
50 | def test_embedded_document_select():
51 | """Select all companies no dereferencing"""
52 | list(Company.objects.no_dereference())
53 |
54 | def test_full_select():
55 | """Select all employees and their referenced companies"""
56 | [e.company for e in Employee.objects]
57 |
58 | if __name__ == "__main__":
59 |
60 | # Connect to the database
61 | connect_to_db()
62 |
63 | # Build the test data
64 | build_test_data()
65 |
66 | # Run the tests
67 | #time_it(test_flat_select, 100)
68 | #time_it(test_embedded_document_select, 100)
69 | time_it(test_full_select, 100)
--------------------------------------------------------------------------------
/performance/tests/run_mongoframes.py:
--------------------------------------------------------------------------------
1 | from mongoframes import *
2 |
3 | from __init__ import build_test_data, connect_to_db, time_it
4 |
5 |
6 | # Define models
7 |
8 | class Company(Frame):
9 |
10 | _fields = {
11 | 'name',
12 | 'departments',
13 | 'address',
14 | 'tel',
15 | 'website_url'
16 | }
17 |
18 |
19 | class Department(SubFrame):
20 |
21 | _fields = {
22 | 'name',
23 | 'year_end',
24 | 'annual_budget'
25 | }
26 |
27 |
28 | class Employee(Frame):
29 |
30 | _fields = {
31 | 'first_name',
32 | 'last_name',
33 | 'dob',
34 | 'role',
35 | 'tel',
36 | 'email',
37 | 'annual_salary',
38 | 'ssn',
39 | 'company',
40 | 'department'
41 | }
42 |
43 |
44 | # Define tests
45 |
46 | def test_flat_select():
47 | """Select all employees no dereferencing"""
48 | Employee.many()
49 |
50 | def test_embedded_document_select():
51 | """Select all companies no dereferencing"""
52 | Company.many(projection={'departments': {'$sub': Department}})
53 |
54 | def test_full_select():
55 | """Select all employees and their referenced companies"""
56 | Employee.many(projection={
57 | 'company': {
58 | '$ref': Company,
59 | 'departments': {'$sub': Department}
60 | }
61 | })
62 |
63 |
64 | if __name__ == "__main__":
65 |
66 | # Connect to the database
67 | connect_to_db()
68 |
69 | # Build the test data
70 | build_test_data()
71 |
72 | # Run the tests
73 | #time_it(test_flat_select, 100)
74 | #time_it(test_embedded_document_select, 100)
75 | time_it(test_full_select, 100)
76 |
--------------------------------------------------------------------------------
/setup.cfg:
--------------------------------------------------------------------------------
1 | [metadata]
2 | description-file = README.md
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | """
2 | Setup instuctions for mongoframes.
3 | """
4 |
5 | # Always prefer setuptools over distutils
6 | from setuptools import setup, find_packages
7 |
8 | # To use a consistent encoding
9 | from codecs import open
10 | from os import path
11 |
12 | here = path.abspath(path.dirname(__file__))
13 |
14 | # Get the long description from the README file
15 | with open(path.join(here, 'README.md'), encoding='utf-8') as f:
16 | long_description = f.read()
17 |
18 | setup(
19 | name='MongoFrames',
20 |
21 | # Versions should comply with PEP440. For a discussion on single-sourcing
22 | # the version across setup.py and the project code, see
23 | # https://packaging.python.org/en/latest/single_source_version.html
24 | version='1.3.10',
25 | description='A fast unobtrusive MongoDB ODM for Python',
26 | long_description=long_description,
27 | long_description_content_type='text/markdown',
28 |
29 | # The project's main homepage.
30 | url='https://github.com/GetmeUK/MongoFrames',
31 |
32 | # Author details
33 | author='Anthony Blackshaw',
34 | author_email='ant@getme.co.uk',
35 |
36 | # Maintainer
37 | maintainer="Anthony Blackshaw",
38 | maintainer_email="ant@getme.co.uk",
39 |
40 | # Choose your license
41 | license='MIT',
42 |
43 | # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
44 | classifiers=[
45 | # How mature is this project? Common values are
46 | # 3 - Alpha
47 | # 4 - Beta
48 | # 5 - Production/Stable
49 | 'Development Status :: 4 - Beta',
50 |
51 | # Indicate who your project is intended for
52 | 'Intended Audience :: Developers',
53 | 'Topic :: Database',
54 |
55 | # Pick your license as you wish (should match "license" above)
56 | 'License :: OSI Approved :: MIT License',
57 |
58 | # Operating systems
59 | "Operating System :: MacOS :: MacOS X",
60 | "Operating System :: Microsoft :: Windows",
61 | "Operating System :: POSIX",
62 |
63 | # Specify the Python versions you support here. In particular, ensure
64 | # that you indicate whether you support Python 2, Python 3 or both.
65 | 'Programming Language :: Python :: 3.4',
66 | 'Programming Language :: Python :: 3.5'
67 | ],
68 |
69 | # What does your project relate to?
70 | keywords='database mongo mongodb odm pymongo',
71 |
72 | # You can just specify the packages manually here if your project is
73 | # simple. Or you can use find_packages().
74 | packages=find_packages(exclude=['contrib', 'docs', 'tests']),
75 |
76 | # Alternatively, if you want to distribute just a my_module.py, uncomment
77 | # this:
78 | # py_modules=["my_module"],
79 |
80 | # List run-time dependencies here. These will be installed by pip when
81 | # your project is installed. For an analysis of "install_requires" vs pip's
82 | # requirements files see:
83 | # https://packaging.python.org/en/latest/requirements.html
84 | install_requires=[
85 | 'blinker>=1.4',
86 | 'Faker>=0.7.18',
87 | 'pymongo>=3.9.0'
88 | ],
89 |
90 | # List additional groups of dependencies here (e.g. development
91 | # dependencies). You can install these using the following syntax,
92 | # for example:
93 | # $ pip install -e .[dev,test]
94 | extras_require={
95 | 'develop': ['pytest', 'pytest-mock', 'tox'],
96 | 'test': ['pytest', 'pytest-mock', 'tox']
97 | },
98 |
99 | # If there are data files included in your packages that need to be
100 | # installed, specify them here. If using Python 2.6 or less, then these
101 | # have to be included in MANIFEST.in as well.
102 | package_data={},
103 |
104 | # Although 'package_data' is the preferred approach, in some case you may
105 | # need to place data files outside of your packages. See:
106 | # http://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files # noqa
107 | # In this case, 'data_file' will be installed into '