├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.md ├── TODO.md ├── dabble ├── __init__.py ├── backends │ ├── __init__.py │ ├── fs.py │ └── mongodb.py └── util.py ├── distribute_setup.py ├── setup.py └── test ├── __init__.py └── test_backend.py /.gitignore: -------------------------------------------------------------------------------- 1 | .*.swp 2 | *.pyc 3 | dabble.egg-info 4 | *.egg 5 | distribute*.tar.gz 6 | dist 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011, Daniel Crosta 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, 8 | this list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | recursive-include dabble *.py 2 | recursive-include test *.py 3 | include distribute_setup.py 4 | include setup.py 5 | include LICENSE 6 | include README.md 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Dabble 2 | 3 | Dabble is a simple A/B testing framework for Python. Using dabble, you 4 | configure your tests in code, collect results, and analyze them later to 5 | make informed decisions about design changes, feature implementations, etc. 6 | 7 | You define an A/B test in dabble with class `ABTest`, which describes the 8 | test name, the names of each of the alternatives, and the set of steps the 9 | users will progress through during the test (in the simplest case, this is 10 | just two steps). You then define one or more `ABParameter`s, which contain 11 | the values you wish to vary for each alternative in the test. Each test can 12 | have one or more alternatives, though the most common case is to have 2 13 | (hence "A/B testing"). 14 | 15 | ## Example 16 | 17 | import dabble 18 | dabble.configure( 19 | CookieIdentityProvider('dabble_id'), 20 | FSResultStorage('/path/to/results.data') 21 | ) 22 | 23 | class Signup(page): 24 | path = '/signup' 25 | 26 | signup_button = ABTest('signup button', 27 | alternatives=['red', 'green'], 28 | steps=['show', 'signup']) 29 | button_color = ABParameter('signup button', ['#ff0000', '#00ff00']) 30 | 31 | def GET(self): 32 | self.signup_button.record('show') 33 | return render('index.html', button_color=self.button_color) 34 | 35 | def POST(self): 36 | self.signup_button.record('signup') 37 | return redirect('/account') 38 | 39 | Behind the scenes, dabble has used a cookie for each user on your site to 40 | assigne them each an *identity*, so that each user always ever sees the same 41 | *alternative*. Users may visit the homepage many times over many browsing 42 | sessions, but as long as they have the same cookie present in their browser, 43 | they will always see either the red or the green button, depending on which 44 | was chosen the first time the viewed the page. 45 | 46 | When a user signs up, the `record()` method of `ABTest` is called, to track 47 | the user's action. Later on, reports can be generated to determine whether 48 | the red or the green button induced more users to sign up. 49 | 50 | ## Configuring Dabble 51 | 52 | In addition to `ABTest` and `ABParameter`, dabble also needs an 53 | `IdentityProvider` and a `ResultsStorage`. Dabble provides several 54 | alternatives for each of these out of the box, and it is also 55 | straightforward to write your own. 56 | 57 | `IdentityProvider`s should make their best possible effort to always 58 | identify individuals, rather than browsing sessions (particularly if cookies 59 | are set to expire when the user closes his/her browser). If you are testing 60 | a feature that requires users to be logged in, then their username is a good 61 | choice for identity. 62 | 63 | `ResultsStorage` stores configuration and results of A/B tests, and provides 64 | some facilities for generating reports based on the stored results. Dabble 65 | provides several backends, including `MongoResultsStorage`, and 66 | `FSResultsStorage`. 67 | 68 | At this time it is not possible to configure different `IdentityProvider`s 69 | or `ResultsStorage`s for different tests within the same application. 70 | 71 | ## Reporting 72 | 73 | Dabble will also produce reports on all users who have taken part in an A/B 74 | test, by way of the `report()` method. The report is a dictionary which 75 | describes, for each alternative, how many users attempted and converted at 76 | each of the defined steps. For the above example, a report might look like: 77 | 78 | 79 | >>> storage = FSResultStorage('/path/to/results.data') 80 | >>> storage.report('signup button') 81 | { 82 | 'test_name': 'signup button', 83 | 'results': [ 84 | { 85 | 'alternative': 'red', 86 | 'funnel': [{ 87 | 'stage': ('show', 'signup'), 88 | 'attempted': 187, 89 | 'converted': 22, 90 | }], 91 | }, 92 | { 93 | 'alternative': 'green', 94 | 'funnel': [{ 95 | 'stage': ('show', 'signup'), 96 | 'attempted': 195 97 | 'converted': 18, 98 | }], 99 | } 100 | ], 101 | } 102 | 103 | The `funnel` key in each of the `results` entries will have one element 104 | fewer than the number of steps, since each entry describes the progression 105 | of users from one step to the next. 106 | 107 | -------------------------------------------------------------------------------- /TODO.md: -------------------------------------------------------------------------------- 1 | * add a `CookieIdentityProvider`? 2 | * add more reporting tools: 3 | * way to calculate significance 4 | * way to reformat report output into something easier to use in a table 5 | * way to run/print reports at command line? 6 | * add funnel report 7 | * add decorator-injected parameters? 8 | -------------------------------------------------------------------------------- /dabble/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011, Daniel Crosta 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # * Redistributions of source code must retain the above copyright notice, 8 | # this list of conditions and the following disclaimer. 9 | # 10 | # * Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | # POSSIBILITY OF SUCH DAMAGE. 25 | 26 | __all__ = ('configure', 'IdentityProvider', 'ResultStorage', 'ABTest', 'ABParameter') 27 | 28 | __version__ = '0.2.3' 29 | 30 | from datetime import datetime 31 | from hashlib import sha1 32 | import random 33 | 34 | class IdentityProvider(object): 35 | """:class:`IdentityProvider` is used to identify a user over 36 | a variety of sessions. It may use any means to do so, so long 37 | as (to the best of its ability) it returns the same identity 38 | for the same human being each time its :meth:`get_identity` 39 | method is called. 40 | """ 41 | 42 | def get_identity(self): 43 | """:meth:`get_identity` is always called with no arguments, 44 | and should return a hashable object identifying the user 45 | for whom A/B trials are currently being run. 46 | """ 47 | raise Exception('Not implemented. Use a sub-class of IdentityProvider') 48 | 49 | class ResultStorage(object): 50 | """:class:`ResultStorage` provides an interface for storing 51 | and retrieving A/B test results to a persistent medium, often 52 | a database or file on disk. 53 | 54 | The :meth:`record_action`, :meth:`has_action`, :meth:`set_alternative`, 55 | and :meth:`get_alternative` methods of this class will be called 56 | synchronously during usage of the framework (e.g. during web page loads), 57 | so care should be taken to ensure that they operate as efficiently as 58 | possible. 59 | """ 60 | 61 | def save_test(self, test_name, alternatives, steps): 62 | """Save an ABTest. 63 | 64 | Unlike the :meth:`record` method, this method should not save 65 | a new record when called with the same `test_name`. Instead, 66 | it should check if such a test already exists, and that it has 67 | the same set of alternatives, and raise if not. 68 | 69 | :Parameters: 70 | - `test_name`: the string name of the test, as set in 71 | :meth:`AB.__init__` 72 | - `alternatives`: a list of string names of the alternatives 73 | used by the :class:`ABTest` 74 | - `steps`: an ordered list of the steps the user will proceed 75 | through during the test (used for funnel analysis) 76 | """ 77 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 78 | 79 | def record(self, identity, test_name, alternative, action): 80 | """Save a user's action to the persistent medium. 81 | 82 | :Parameters: 83 | - `identity`: the hashed identity of the user, as returned 84 | by :meth:`IdentityProvider.get_identity` 85 | - `test_name`: the string name of the test, as set in 86 | :meth:`AB.__init__` 87 | - `alternative`: the postitive integer index of the alternative 88 | displayed to the user 89 | - `action`: the string name of the action the user took 90 | """ 91 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 92 | 93 | def has_action(self, identity, test_name, alternative, action): 94 | """Return `True` if the user with the given identity, has the given 95 | action recorded for the given test name and alternative, else `False`. 96 | 97 | :Parameters: 98 | - `identity`: the hashed identity of the user, as returned 99 | by :meth:`IdentityProvider.get_identity` 100 | - `test_name`: the string name of the test, as set in 101 | :meth:`AB.__init__` 102 | - `alternative`: the postitive integer index of the alternative 103 | displayed to the user 104 | - `action`: the name of an action 105 | """ 106 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 107 | 108 | def set_alternative(self, identity, test_name, alternative): 109 | """Record the given alternative for the user. 110 | 111 | :Parameters: 112 | - `identity`: the hashed identity of the user, as returned 113 | by :meth:`IdentityProvider.get_identity` 114 | - `test_name`: the string name of the test, as set in 115 | :meth:`AB.__init__` 116 | - `alternative`: the postitive integer index of the alternative 117 | displayed to the user 118 | """ 119 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 120 | 121 | def get_alternative(self, identity, test_name): 122 | """Return the alternative for the user, as previously set with 123 | :meth:`set_alternative`. Return `None` if no previous call for 124 | the given identity and test name has happened. 125 | 126 | :Parameters: 127 | - `identity`: the hashed identity of the user, as returned 128 | by :meth:`IdentityProvider.get_identity` 129 | - `test_name`: the string name of the test, as set in 130 | :meth:`AB.__init__` 131 | """ 132 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 133 | 134 | def report(self, test_name, a, b): 135 | """Return report data for the alternatives of a given test 136 | where users have either action `a` only, or actions `a` and 137 | `b`. Other actions, and duplicate or repeated actions are 138 | ignored. 139 | 140 | Action `a` is ordinarily a "start" action, for instance an 141 | action denoting "user was shown a page with an A/B test on it". 142 | Action `b` is ordinarily a "target" action, for instance an 143 | action denoting "user filled out the form being tested". 144 | 145 | The output is a dictionary in the following format: 146 | 147 | { test_name: "...", 148 | alternatives: ["...", "...", ...], 149 | results: [ 150 | { attempted: N, 151 | converted: M, 152 | }, ... 153 | ] 154 | } 155 | 156 | The dictionaries in the `results` array should be in the same 157 | order as the alternatives listed in the alternatives array, 158 | which need not be the same order as they are configured in 159 | the :class:`ABTest`. 160 | 161 | The values `N` and `M` within the result objects should count 162 | unique identities who attempted or completed the action. An 163 | attempt is defined as an identity with at least one recorded 164 | `a` action; a completion is defined as an identity with at 165 | least one recorded `a` action followed by (chronologically) 166 | at least one recorded `b` action. 167 | 168 | Implementation of the report is delegated to the storage 169 | class since dabble cannot know the most efficient way to 170 | query the underlying data store. 171 | 172 | :Parameters: 173 | - `test_name`: the string name of the test, as set in 174 | :meth:`AB.__init__` 175 | - `a`: a string identifying a start action 176 | - `b`: a string identifying a completion action 177 | """ 178 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 179 | 180 | def list_tests(self): 181 | """Return a list of string test names known.""" 182 | raise Exception('Not implemented. Use a sub-class of ResultStorage') 183 | 184 | 185 | def configure(identity_provider, result_storage): 186 | if not isinstance(identity_provider, IdentityProvider): 187 | raise Exception('identity_provider must extend IdentityProvider') 188 | if not isinstance(result_storage, ResultStorage): 189 | raise Exception('result_storage must extend ResultStorage') 190 | 191 | if AB._id_provider is not None or AB._storage is not None: 192 | raise Exception('configure called multiple times') 193 | 194 | AB._id_provider = identity_provider 195 | AB._storage = result_storage 196 | 197 | class AB(object): 198 | """TODO. 199 | """ 200 | 201 | # these are set by the configure() function 202 | _id_provider = None 203 | _storage = None 204 | 205 | # track the number of alternatives for each 206 | # named test; helps prevent errors where some 207 | # parameters have more alts than others 208 | __n_per_test = {} 209 | 210 | def __init__(self, test_name, alternatives): 211 | if test_name not in AB.__n_per_test: 212 | AB.__n_per_test[test_name] = len(alternatives) 213 | if len(alternatives) != AB.__n_per_test[test_name]: 214 | raise Exception('Wrong number of alternatives') 215 | 216 | self.test_name = test_name 217 | self.alternatives = alternatives 218 | 219 | @property 220 | def identity(self): 221 | return sha1(unicode(self._id_provider.get_identity())).hexdigest() 222 | 223 | @property 224 | def alternative(self): 225 | alternative = self._storage.get_alternative(self.identity, self.test_name) 226 | 227 | if alternative is None: 228 | alternative = random.randrange(len(self.alternatives)) 229 | self._storage.set_alternative(self.identity, self.test_name, alternative) 230 | 231 | return alternative 232 | 233 | class ABTest(AB): 234 | # can be added to a class definition to define information 235 | # about the AB test, as will be shown in the admin UI. 236 | # additionally, if the ABTest is assigned as a class attribute, 237 | # it contains some information about the state of the test 238 | # 239 | # class ShowAForm(app.page): 240 | # path = '/page/with/form' 241 | # abtest = ABTest('my_test', ['Complete Form', 'Brief Form'], ['Form Shown', 'Form Filled']) 242 | # formname = ABParameter('my_test', ['form_one', 'form_two']) 243 | # 244 | # def GET(self): 245 | # if abtest.completed: 246 | # raise web.seeother('/page/after/form/completion') 247 | # render('template.html', form=self.get_form(self.formname)) 248 | 249 | def __init__(self, test_name, alternatives, steps): 250 | super(ABTest, self).__init__(test_name, alternatives) 251 | self._storage.save_test(test_name, alternatives, steps) 252 | 253 | def record(self, action): 254 | self._storage.record( 255 | self.identity, 256 | self.test_name, 257 | self.alternative, 258 | action, 259 | ) 260 | 261 | class ABParameter(AB): 262 | # a descriptor object which can be used to vary parameters 263 | # in a class definition according to A/B testing rules. 264 | # 265 | # each viewer who views the given class will always be 266 | # consistently shown the Nth choice from among the 267 | # alternatives, even between different attributes in the 268 | # class, so long as the name is the same between them 269 | 270 | def __get__(self, instance, owner): 271 | return self.alternatives[self.alternative] 272 | 273 | -------------------------------------------------------------------------------- /dabble/backends/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011, Daniel Crosta 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # * Redistributions of source code must retain the above copyright notice, 8 | # this list of conditions and the following disclaimer. 9 | # 10 | # * Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | # POSSIBILITY OF SUCH DAMAGE. 25 | 26 | -------------------------------------------------------------------------------- /dabble/backends/fs.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011, Daniel Crosta 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # * Redistributions of source code must retain the above copyright notice, 8 | # this list of conditions and the following disclaimer. 9 | # 10 | # * Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | # POSSIBILITY OF SUCH DAMAGE. 25 | 26 | __all__ = ('FSResultStorage', ) 27 | 28 | from dabble import ResultStorage 29 | from dabble.util import * 30 | 31 | from os.path import exists, join, abspath 32 | from os import SEEK_END 33 | from lockfile import FileLock 34 | import json 35 | 36 | 37 | def find_lines(filename, **pattern): 38 | """Find a line (JSON-formatted) in the given file where 39 | all keys in `pattern` are present as keys in the line's 40 | JSON, and where their values equal the corresponding 41 | values in `pattern`. Additional keys in the line are 42 | ignored. If no matching line is found, or if the 43 | file does not exist, return None. 44 | """ 45 | if exists(filename): 46 | with file(filename, 'r') as fp: 47 | for line in fp: 48 | try: 49 | data = json.loads(line) 50 | except: 51 | continue 52 | matches = True 53 | for key, value in pattern.iteritems(): 54 | matches = matches and key in data and data[key] == value 55 | if matches: 56 | yield data 57 | 58 | def find_line(filename, **pattern): 59 | """Return the first line that would be found by 60 | :func:`find_lines`. 61 | """ 62 | for line in find_lines(filename, **pattern): 63 | return line 64 | return None 65 | 66 | lock = None 67 | def append_line(filename, **line): 68 | """Safely (i.e. with locking) append a line to 69 | the given file, serialized as JSON. 70 | """ 71 | global lock 72 | 73 | data = json.dumps(line, separators=(',', ':')) + '\n' 74 | with lock: 75 | with file(filename, 'a') as fp: 76 | fp.seek(0, SEEK_END) 77 | fp.write(data) 78 | 79 | 80 | class FSResultStorage(ResultStorage): 81 | 82 | def __init__(self, directory): 83 | """Set up storage in the filesystem for A/B test results. 84 | 85 | :Parameters: 86 | - `directory`: an existing directory in the filesystem where 87 | results can be stored. Several files with the ".dabble" 88 | extension will be created. 89 | - `namespace`: the name prefix used to name collections 90 | """ 91 | global lock 92 | 93 | self.directory = abspath(directory) 94 | 95 | if not exists(self.directory): 96 | raise Exception('directory "%s" does not exist' % self.directory) 97 | 98 | lock = FileLock(join(self.directory, 'lock.dabble')) 99 | 100 | self.tests_path = join(self.directory, 'tests.dabble') 101 | self.results_path = join(self.directory, 'results.dabble') 102 | self.alts_path = join(self.directory, 'alts.dabble') 103 | 104 | def save_test(self, test_name, alternatives, steps): 105 | existing = find_line(self.tests_path, t=test_name) 106 | if existing and (existing['a'] != alternatives or existing['s'] != steps): 107 | raise Exception( 108 | 'test "%s" already exists with different alternatives' % test_name) 109 | 110 | append_line(self.tests_path, t=test_name, a=alternatives, s=steps) 111 | 112 | def record(self, identity, test_name, alternative, action): 113 | append_line(self.results_path, 114 | i=identity, t=test_name, n=alternative, s=action) 115 | 116 | def has_action(self, identity, test_name, alternative, action): 117 | return find_line(self.results_path, i=identity, t=test_name, n=alternative, a=action) is not None 118 | 119 | def set_alternative(self, identity, test_name, alternative): 120 | existing = find_line(self.alts_path, i=identity, t=test_name) 121 | if existing and existing['n'] != alternative: 122 | raise Exception( 123 | 'different alternative already set for identity %s' % identity) 124 | 125 | append_line(self.alts_path, i=identity, t=test_name, n=alternative) 126 | 127 | def get_alternative(self, identity, test_name): 128 | existing = find_line(self.alts_path, i=identity, t=test_name) or {} 129 | return existing.get('n') 130 | 131 | def report(self, test_name): 132 | test = find_line(self.tests_path, t=test_name) 133 | if test is None: 134 | raise Exception('unknown test "%s"' % test_name) 135 | 136 | report = { 137 | 'test_name': test_name, 138 | 'results': [] 139 | } 140 | 141 | trials = sparsearray(int) 142 | maxstep = {} 143 | 144 | for result in find_lines(self.results_path, t=test_name): 145 | step = test['s'].index(result['s']) 146 | sofar = maxstep.get(result['i']) 147 | if sofar is None and step == 0 or sofar is not None and step == sofar + 1: 148 | trials[result['n']][step] += 1 149 | maxstep[result['i']] = step 150 | 151 | for i, alternative in enumerate(test['a']): 152 | funnel = [] 153 | alt = {'alternative': alternative, 'funnel': funnel} 154 | report['results'].append(alt) 155 | for s, stepspair in enumerate(pairwise(test['s'])): 156 | att = trials[i][s] 157 | con = trials[i][s + 1] 158 | funnel.append({ 159 | 'stage': stepspair, 160 | 'attempted': att, 161 | 'converted': con, 162 | }) 163 | 164 | return report 165 | 166 | def list_tests(self): 167 | """Return a list of string test names known.""" 168 | return [t['t'] for t in find_lines(self.tests_path)] 169 | 170 | -------------------------------------------------------------------------------- /dabble/backends/mongodb.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011, Daniel Crosta 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # * Redistributions of source code must retain the above copyright notice, 8 | # this list of conditions and the following disclaimer. 9 | # 10 | # * Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | # POSSIBILITY OF SUCH DAMAGE. 25 | 26 | __all__ = ('MongoResultStorage', ) 27 | 28 | from dabble import ResultStorage 29 | from dabble.util import * 30 | 31 | from datetime import datetime 32 | from random import randrange 33 | from bson.son import SON 34 | from pymongo import ASCENDING, DESCENDING 35 | from pymongo.database import Database 36 | from pymongo.errors import DuplicateKeyError 37 | 38 | class MongoResultStorage(ResultStorage): 39 | 40 | def __init__(self, database, namespace='dabble'): 41 | """Set up storage in MongoDB (using PyMongo) for A/B test results. 42 | Setup requires at least a :class:`pymongo.database.Database` instance, 43 | and optionally accepts a `namespace` parameter, which is used to 44 | generate collection names used for storage. Three collections will be 45 | used, named ".tests" and ".results". 46 | 47 | :Parameters: 48 | - `database`: a :class:`pymongo.database.Database` instance 49 | in which to create the two collections for result storage 50 | - `namespace`: the name prefix used to name collections 51 | """ 52 | if not isinstance(database, Database): 53 | raise Exception('"database" argument is not a pymongo.database.Database') 54 | 55 | self.namespace = namespace 56 | 57 | self.tests = database['%s.tests' % namespace] 58 | self.results = database['%s.results' % namespace] 59 | 60 | self.results.ensure_index([('t', ASCENDING), ('i', ASCENDING)]) 61 | 62 | def save_test(self, test_name, alternatives, steps): 63 | test = self.tests.find_one({'_id': test_name}) 64 | 65 | if test and test['a'] != alternatives: 66 | raise Exception('test "%s" already exists with different alternatives' % test_name) 67 | 68 | elif not test: 69 | self.tests.save({ 70 | '_id': test_name, 71 | 'a': alternatives, 72 | 's': steps, 73 | }, safe=True) 74 | 75 | def record(self, identity, test_name, alternative, action): 76 | self.results.update({ 77 | 'i': identity, 78 | 't': test_name, 79 | 'n': alternative}, 80 | {'$addToSet': {'s': action}}, 81 | upsert=True) 82 | 83 | def has_action(self, identity, test_name, alternative, action): 84 | return self.results.find_one({'i': identity, 't': test_name, 'n': alternative, 's': action}) is not None 85 | 86 | def set_alternative(self, identity, test_name, alternative): 87 | # XXX: possible race condition, but one will win, and 88 | # for A/B testing that's probably OK. 89 | result = self.results.find_one({'i': identity, 't': test_name}) 90 | if not result: 91 | self.results.save({'i': identity, 't': test_name, 'n': alternative, 's': []}) 92 | 93 | elif result and result['n'] != alternative: 94 | raise Exception('different alternative already set for identity %s' % identity) 95 | 96 | def get_alternative(self, identity, test_name): 97 | result = self.results.find_one({'i': identity, 't': test_name}) or {} 98 | return result.get('n') 99 | 100 | def report(self, test_name): 101 | test = self.tests.find_one({'_id': test_name}) 102 | if test is None: 103 | raise Exception('unknown test "%s"' % test_name) 104 | 105 | report = { 106 | 'test_name': test_name, 107 | 'results': [] 108 | } 109 | 110 | trials = sparsearray(int) 111 | 112 | for result in self.results.find({'t': test_name}): 113 | if result['s'] != test['s'][:len(result['s'])]: 114 | # invalid order of steps recorded 115 | continue 116 | 117 | for i in xrange(len(result['s'])): 118 | trials[result['n']][i] += 1 119 | 120 | for i, alternative in enumerate(test['a']): 121 | funnel = [] 122 | alt = {'alternative': alternative, 'funnel': funnel} 123 | report['results'].append(alt) 124 | for s, stepspair in enumerate(pairwise(test['s'])): 125 | att = trials[i][s] 126 | con = trials[i][s + 1] 127 | funnel.append({ 128 | 'stage': stepspair, 129 | 'attempted': att, 130 | 'converted': con, 131 | }) 132 | 133 | return report 134 | 135 | def list_tests(self): 136 | """Return a list of string test names known.""" 137 | return [t['_id'] for t in self.tests.find(fields=['_id'])] 138 | 139 | -------------------------------------------------------------------------------- /dabble/util.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011, Daniel Crosta 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # * Redistributions of source code must retain the above copyright notice, 8 | # this list of conditions and the following disclaimer. 9 | # 10 | # * Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 15 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 18 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 19 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 20 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 22 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 23 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24 | # POSSIBILITY OF SUCH DAMAGE. 25 | 26 | __all__ = ('pairwise', 'sparsearray') 27 | 28 | from itertools import tee, izip 29 | from collections import defaultdict 30 | 31 | def pairwise(iterable): 32 | # s => (s0,s1), (s1,s2), (s2, s3), ... 33 | a, b = tee(iterable) 34 | next(b, None) 35 | return izip(a, b) 36 | 37 | def sparsearray(ctor): 38 | # return a 2D "sparse array" (nested dicts) 39 | return defaultdict(lambda: defaultdict(int)) 40 | -------------------------------------------------------------------------------- /distribute_setup.py: -------------------------------------------------------------------------------- 1 | #!python 2 | """Bootstrap distribute installation 3 | 4 | If you want to use setuptools in your package's setup.py, just include this 5 | file in the same directory with it, and add this to the top of your setup.py:: 6 | 7 | from distribute_setup import use_setuptools 8 | use_setuptools() 9 | 10 | If you want to require a specific version of setuptools, set a download 11 | mirror, or use an alternate download directory, you can do so by supplying 12 | the appropriate options to ``use_setuptools()``. 13 | 14 | This file can also be run as a script to install or upgrade setuptools. 15 | """ 16 | import os 17 | import sys 18 | import time 19 | import fnmatch 20 | import tempfile 21 | import tarfile 22 | from distutils import log 23 | 24 | try: 25 | from site import USER_SITE 26 | except ImportError: 27 | USER_SITE = None 28 | 29 | try: 30 | import subprocess 31 | 32 | def _python_cmd(*args): 33 | args = (sys.executable,) + args 34 | return subprocess.call(args) == 0 35 | 36 | except ImportError: 37 | # will be used for python 2.3 38 | def _python_cmd(*args): 39 | args = (sys.executable,) + args 40 | # quoting arguments if windows 41 | if sys.platform == 'win32': 42 | def quote(arg): 43 | if ' ' in arg: 44 | return '"%s"' % arg 45 | return arg 46 | args = [quote(arg) for arg in args] 47 | return os.spawnl(os.P_WAIT, sys.executable, *args) == 0 48 | 49 | DEFAULT_VERSION = "0.6.24" 50 | DEFAULT_URL = "http://pypi.python.org/packages/source/d/distribute/" 51 | SETUPTOOLS_FAKED_VERSION = "0.6c11" 52 | 53 | SETUPTOOLS_PKG_INFO = """\ 54 | Metadata-Version: 1.0 55 | Name: setuptools 56 | Version: %s 57 | Summary: xxxx 58 | Home-page: xxx 59 | Author: xxx 60 | Author-email: xxx 61 | License: xxx 62 | Description: xxx 63 | """ % SETUPTOOLS_FAKED_VERSION 64 | 65 | 66 | def _install(tarball): 67 | # extracting the tarball 68 | tmpdir = tempfile.mkdtemp() 69 | log.warn('Extracting in %s', tmpdir) 70 | old_wd = os.getcwd() 71 | try: 72 | os.chdir(tmpdir) 73 | tar = tarfile.open(tarball) 74 | _extractall(tar) 75 | tar.close() 76 | 77 | # going in the directory 78 | subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) 79 | os.chdir(subdir) 80 | log.warn('Now working in %s', subdir) 81 | 82 | # installing 83 | log.warn('Installing Distribute') 84 | if not _python_cmd('setup.py', 'install'): 85 | log.warn('Something went wrong during the installation.') 86 | log.warn('See the error message above.') 87 | finally: 88 | os.chdir(old_wd) 89 | 90 | 91 | def _build_egg(egg, tarball, to_dir): 92 | # extracting the tarball 93 | tmpdir = tempfile.mkdtemp() 94 | log.warn('Extracting in %s', tmpdir) 95 | old_wd = os.getcwd() 96 | try: 97 | os.chdir(tmpdir) 98 | tar = tarfile.open(tarball) 99 | _extractall(tar) 100 | tar.close() 101 | 102 | # going in the directory 103 | subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) 104 | os.chdir(subdir) 105 | log.warn('Now working in %s', subdir) 106 | 107 | # building an egg 108 | log.warn('Building a Distribute egg in %s', to_dir) 109 | _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir) 110 | 111 | finally: 112 | os.chdir(old_wd) 113 | # returning the result 114 | log.warn(egg) 115 | if not os.path.exists(egg): 116 | raise IOError('Could not build the egg.') 117 | 118 | 119 | def _do_download(version, download_base, to_dir, download_delay): 120 | egg = os.path.join(to_dir, 'distribute-%s-py%d.%d.egg' 121 | % (version, sys.version_info[0], sys.version_info[1])) 122 | if not os.path.exists(egg): 123 | tarball = download_setuptools(version, download_base, 124 | to_dir, download_delay) 125 | _build_egg(egg, tarball, to_dir) 126 | sys.path.insert(0, egg) 127 | import setuptools 128 | setuptools.bootstrap_install_from = egg 129 | 130 | 131 | def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, 132 | to_dir=os.curdir, download_delay=15, no_fake=True): 133 | # making sure we use the absolute path 134 | to_dir = os.path.abspath(to_dir) 135 | was_imported = 'pkg_resources' in sys.modules or \ 136 | 'setuptools' in sys.modules 137 | try: 138 | try: 139 | import pkg_resources 140 | if not hasattr(pkg_resources, '_distribute'): 141 | if not no_fake: 142 | _fake_setuptools() 143 | raise ImportError 144 | except ImportError: 145 | return _do_download(version, download_base, to_dir, download_delay) 146 | try: 147 | pkg_resources.require("distribute>="+version) 148 | return 149 | except pkg_resources.VersionConflict: 150 | e = sys.exc_info()[1] 151 | if was_imported: 152 | sys.stderr.write( 153 | "The required version of distribute (>=%s) is not available,\n" 154 | "and can't be installed while this script is running. Please\n" 155 | "install a more recent version first, using\n" 156 | "'easy_install -U distribute'." 157 | "\n\n(Currently using %r)\n" % (version, e.args[0])) 158 | sys.exit(2) 159 | else: 160 | del pkg_resources, sys.modules['pkg_resources'] # reload ok 161 | return _do_download(version, download_base, to_dir, 162 | download_delay) 163 | except pkg_resources.DistributionNotFound: 164 | return _do_download(version, download_base, to_dir, 165 | download_delay) 166 | finally: 167 | if not no_fake: 168 | _create_fake_setuptools_pkg_info(to_dir) 169 | 170 | def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, 171 | to_dir=os.curdir, delay=15): 172 | """Download distribute from a specified location and return its filename 173 | 174 | `version` should be a valid distribute version number that is available 175 | as an egg for download under the `download_base` URL (which should end 176 | with a '/'). `to_dir` is the directory where the egg will be downloaded. 177 | `delay` is the number of seconds to pause before an actual download 178 | attempt. 179 | """ 180 | # making sure we use the absolute path 181 | to_dir = os.path.abspath(to_dir) 182 | try: 183 | from urllib.request import urlopen 184 | except ImportError: 185 | from urllib2 import urlopen 186 | tgz_name = "distribute-%s.tar.gz" % version 187 | url = download_base + tgz_name 188 | saveto = os.path.join(to_dir, tgz_name) 189 | src = dst = None 190 | if not os.path.exists(saveto): # Avoid repeated downloads 191 | try: 192 | log.warn("Downloading %s", url) 193 | src = urlopen(url) 194 | # Read/write all in one block, so we don't create a corrupt file 195 | # if the download is interrupted. 196 | data = src.read() 197 | dst = open(saveto, "wb") 198 | dst.write(data) 199 | finally: 200 | if src: 201 | src.close() 202 | if dst: 203 | dst.close() 204 | return os.path.realpath(saveto) 205 | 206 | def _no_sandbox(function): 207 | def __no_sandbox(*args, **kw): 208 | try: 209 | from setuptools.sandbox import DirectorySandbox 210 | if not hasattr(DirectorySandbox, '_old'): 211 | def violation(*args): 212 | pass 213 | DirectorySandbox._old = DirectorySandbox._violation 214 | DirectorySandbox._violation = violation 215 | patched = True 216 | else: 217 | patched = False 218 | except ImportError: 219 | patched = False 220 | 221 | try: 222 | return function(*args, **kw) 223 | finally: 224 | if patched: 225 | DirectorySandbox._violation = DirectorySandbox._old 226 | del DirectorySandbox._old 227 | 228 | return __no_sandbox 229 | 230 | def _patch_file(path, content): 231 | """Will backup the file then patch it""" 232 | existing_content = open(path).read() 233 | if existing_content == content: 234 | # already patched 235 | log.warn('Already patched.') 236 | return False 237 | log.warn('Patching...') 238 | _rename_path(path) 239 | f = open(path, 'w') 240 | try: 241 | f.write(content) 242 | finally: 243 | f.close() 244 | return True 245 | 246 | _patch_file = _no_sandbox(_patch_file) 247 | 248 | def _same_content(path, content): 249 | return open(path).read() == content 250 | 251 | def _rename_path(path): 252 | new_name = path + '.OLD.%s' % time.time() 253 | log.warn('Renaming %s into %s', path, new_name) 254 | os.rename(path, new_name) 255 | return new_name 256 | 257 | def _remove_flat_installation(placeholder): 258 | if not os.path.isdir(placeholder): 259 | log.warn('Unkown installation at %s', placeholder) 260 | return False 261 | found = False 262 | for file in os.listdir(placeholder): 263 | if fnmatch.fnmatch(file, 'setuptools*.egg-info'): 264 | found = True 265 | break 266 | if not found: 267 | log.warn('Could not locate setuptools*.egg-info') 268 | return 269 | 270 | log.warn('Removing elements out of the way...') 271 | pkg_info = os.path.join(placeholder, file) 272 | if os.path.isdir(pkg_info): 273 | patched = _patch_egg_dir(pkg_info) 274 | else: 275 | patched = _patch_file(pkg_info, SETUPTOOLS_PKG_INFO) 276 | 277 | if not patched: 278 | log.warn('%s already patched.', pkg_info) 279 | return False 280 | # now let's move the files out of the way 281 | for element in ('setuptools', 'pkg_resources.py', 'site.py'): 282 | element = os.path.join(placeholder, element) 283 | if os.path.exists(element): 284 | _rename_path(element) 285 | else: 286 | log.warn('Could not find the %s element of the ' 287 | 'Setuptools distribution', element) 288 | return True 289 | 290 | _remove_flat_installation = _no_sandbox(_remove_flat_installation) 291 | 292 | def _after_install(dist): 293 | log.warn('After install bootstrap.') 294 | placeholder = dist.get_command_obj('install').install_purelib 295 | _create_fake_setuptools_pkg_info(placeholder) 296 | 297 | def _create_fake_setuptools_pkg_info(placeholder): 298 | if not placeholder or not os.path.exists(placeholder): 299 | log.warn('Could not find the install location') 300 | return 301 | pyver = '%s.%s' % (sys.version_info[0], sys.version_info[1]) 302 | setuptools_file = 'setuptools-%s-py%s.egg-info' % \ 303 | (SETUPTOOLS_FAKED_VERSION, pyver) 304 | pkg_info = os.path.join(placeholder, setuptools_file) 305 | if os.path.exists(pkg_info): 306 | log.warn('%s already exists', pkg_info) 307 | return 308 | 309 | log.warn('Creating %s', pkg_info) 310 | f = open(pkg_info, 'w') 311 | try: 312 | f.write(SETUPTOOLS_PKG_INFO) 313 | finally: 314 | f.close() 315 | 316 | pth_file = os.path.join(placeholder, 'setuptools.pth') 317 | log.warn('Creating %s', pth_file) 318 | f = open(pth_file, 'w') 319 | try: 320 | f.write(os.path.join(os.curdir, setuptools_file)) 321 | finally: 322 | f.close() 323 | 324 | _create_fake_setuptools_pkg_info = _no_sandbox(_create_fake_setuptools_pkg_info) 325 | 326 | def _patch_egg_dir(path): 327 | # let's check if it's already patched 328 | pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') 329 | if os.path.exists(pkg_info): 330 | if _same_content(pkg_info, SETUPTOOLS_PKG_INFO): 331 | log.warn('%s already patched.', pkg_info) 332 | return False 333 | _rename_path(path) 334 | os.mkdir(path) 335 | os.mkdir(os.path.join(path, 'EGG-INFO')) 336 | pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') 337 | f = open(pkg_info, 'w') 338 | try: 339 | f.write(SETUPTOOLS_PKG_INFO) 340 | finally: 341 | f.close() 342 | return True 343 | 344 | _patch_egg_dir = _no_sandbox(_patch_egg_dir) 345 | 346 | def _before_install(): 347 | log.warn('Before install bootstrap.') 348 | _fake_setuptools() 349 | 350 | 351 | def _under_prefix(location): 352 | if 'install' not in sys.argv: 353 | return True 354 | args = sys.argv[sys.argv.index('install')+1:] 355 | for index, arg in enumerate(args): 356 | for option in ('--root', '--prefix'): 357 | if arg.startswith('%s=' % option): 358 | top_dir = arg.split('root=')[-1] 359 | return location.startswith(top_dir) 360 | elif arg == option: 361 | if len(args) > index: 362 | top_dir = args[index+1] 363 | return location.startswith(top_dir) 364 | if arg == '--user' and USER_SITE is not None: 365 | return location.startswith(USER_SITE) 366 | return True 367 | 368 | 369 | def _fake_setuptools(): 370 | log.warn('Scanning installed packages') 371 | try: 372 | import pkg_resources 373 | except ImportError: 374 | # we're cool 375 | log.warn('Setuptools or Distribute does not seem to be installed.') 376 | return 377 | ws = pkg_resources.working_set 378 | try: 379 | setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools', 380 | replacement=False)) 381 | except TypeError: 382 | # old distribute API 383 | setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools')) 384 | 385 | if setuptools_dist is None: 386 | log.warn('No setuptools distribution found') 387 | return 388 | # detecting if it was already faked 389 | setuptools_location = setuptools_dist.location 390 | log.warn('Setuptools installation detected at %s', setuptools_location) 391 | 392 | # if --root or --preix was provided, and if 393 | # setuptools is not located in them, we don't patch it 394 | if not _under_prefix(setuptools_location): 395 | log.warn('Not patching, --root or --prefix is installing Distribute' 396 | ' in another location') 397 | return 398 | 399 | # let's see if its an egg 400 | if not setuptools_location.endswith('.egg'): 401 | log.warn('Non-egg installation') 402 | res = _remove_flat_installation(setuptools_location) 403 | if not res: 404 | return 405 | else: 406 | log.warn('Egg installation') 407 | pkg_info = os.path.join(setuptools_location, 'EGG-INFO', 'PKG-INFO') 408 | if (os.path.exists(pkg_info) and 409 | _same_content(pkg_info, SETUPTOOLS_PKG_INFO)): 410 | log.warn('Already patched.') 411 | return 412 | log.warn('Patching...') 413 | # let's create a fake egg replacing setuptools one 414 | res = _patch_egg_dir(setuptools_location) 415 | if not res: 416 | return 417 | log.warn('Patched done.') 418 | _relaunch() 419 | 420 | 421 | def _relaunch(): 422 | log.warn('Relaunching...') 423 | # we have to relaunch the process 424 | # pip marker to avoid a relaunch bug 425 | if sys.argv[:3] == ['-c', 'install', '--single-version-externally-managed']: 426 | sys.argv[0] = 'setup.py' 427 | args = [sys.executable] + sys.argv 428 | sys.exit(subprocess.call(args)) 429 | 430 | 431 | def _extractall(self, path=".", members=None): 432 | """Extract all members from the archive to the current working 433 | directory and set owner, modification time and permissions on 434 | directories afterwards. `path' specifies a different directory 435 | to extract to. `members' is optional and must be a subset of the 436 | list returned by getmembers(). 437 | """ 438 | import copy 439 | import operator 440 | from tarfile import ExtractError 441 | directories = [] 442 | 443 | if members is None: 444 | members = self 445 | 446 | for tarinfo in members: 447 | if tarinfo.isdir(): 448 | # Extract directories with a safe mode. 449 | directories.append(tarinfo) 450 | tarinfo = copy.copy(tarinfo) 451 | tarinfo.mode = 448 # decimal for oct 0700 452 | self.extract(tarinfo, path) 453 | 454 | # Reverse sort directories. 455 | if sys.version_info < (2, 4): 456 | def sorter(dir1, dir2): 457 | return cmp(dir1.name, dir2.name) 458 | directories.sort(sorter) 459 | directories.reverse() 460 | else: 461 | directories.sort(key=operator.attrgetter('name'), reverse=True) 462 | 463 | # Set correct owner, mtime and filemode on directories. 464 | for tarinfo in directories: 465 | dirpath = os.path.join(path, tarinfo.name) 466 | try: 467 | self.chown(tarinfo, dirpath) 468 | self.utime(tarinfo, dirpath) 469 | self.chmod(tarinfo, dirpath) 470 | except ExtractError: 471 | e = sys.exc_info()[1] 472 | if self.errorlevel > 1: 473 | raise 474 | else: 475 | self._dbg(1, "tarfile: %s" % e) 476 | 477 | 478 | def main(argv, version=DEFAULT_VERSION): 479 | """Install or upgrade setuptools and EasyInstall""" 480 | tarball = download_setuptools() 481 | _install(tarball) 482 | 483 | 484 | if __name__ == '__main__': 485 | main(sys.argv[1:]) 486 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distribute_setup import use_setuptools 2 | use_setuptools() 3 | 4 | from setuptools import setup, find_packages 5 | from os.path import abspath, dirname, join 6 | 7 | from dabble import __version__ 8 | 9 | setup( 10 | name='dabble', 11 | version=__version__, 12 | description='Simple A/B testing framework', 13 | long_description=file(abspath(join(dirname(__file__), 'README.md'))).read(), 14 | license='BSD', 15 | classifiers=[ 16 | "Programming Language :: Python", 17 | "Programming Language :: Python :: 2.4", 18 | "Programming Language :: Python :: 2.5", 19 | "Programming Language :: Python :: 2.6", 20 | "Programming Language :: Python :: 2.7", 21 | ], 22 | author='Dan Crosta', 23 | author_email='dcrosta@late.am', 24 | url='https://github.com/dcrosta/dabble', 25 | keywords='python web abtest split ab a/b test', 26 | packages=find_packages(), 27 | tests_require=['nose'], 28 | test_suite='nose.collector', 29 | ) 30 | 31 | -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dcrosta/dabble/a671db61be7d2d5e83dbdd11b171fa9e3bf0dfb1/test/__init__.py -------------------------------------------------------------------------------- /test/test_backend.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | import dabble 4 | from dabble import * 5 | from dabble.backends.fs import * 6 | from dabble.backends.mongodb import * 7 | import pymongo 8 | 9 | from os import makedirs 10 | from os.path import dirname, exists, join 11 | from shutil import rmtree 12 | 13 | class MockIdentityProvider(IdentityProvider): 14 | 15 | def __init__(self): 16 | super(MockIdentityProvider, self).__init__() 17 | self.identity = None 18 | 19 | def get_identity(self): 20 | if self.identity is None: 21 | raise Exception('bad test, need to set identity') 22 | 23 | return self.identity 24 | 25 | class RandRange(object): 26 | def __init__(self): 27 | self.n = 0 28 | self.last = None 29 | 30 | def __call__(self, max): 31 | self.last = self.n % max 32 | self.n += 1 33 | return self.last 34 | 35 | 36 | def ReportTestFor(name, setUp_func, tearDown_func): 37 | def test_one(self): 38 | class T(object): 39 | abtest = ABTest('foobar', ['foo'], ['show', 'fill']) 40 | 41 | t = T() 42 | 43 | self.provider.identity = 1 44 | t.abtest.record('show') 45 | t.abtest.record('show') 46 | t.abtest.record('fill') 47 | 48 | self.provider.identity = 2 49 | t.abtest.record('show') 50 | t.abtest.record('fill') 51 | 52 | self.provider.identity = 3 53 | t.abtest.record('fill') 54 | 55 | self.provider.identity = 4 56 | t.abtest.record('show') 57 | 58 | report = self.storage.report('foobar') 59 | 60 | expected = { 61 | 'test_name': 'foobar', 62 | 'results': [{ 63 | 'alternative': 'foo', 64 | 'funnel': [{ 65 | 'stage': ('show', 'fill'), 66 | 'attempted': 3, 67 | 'converted': 2, 68 | }], 69 | }], 70 | } 71 | 72 | try: 73 | self.assertEquals(report, expected) 74 | except: 75 | from pprint import pprint 76 | pprint(report) 77 | pprint(expected) 78 | raise 79 | 80 | 81 | def test_two(self): 82 | class T(object): 83 | abtest = ABTest('foobar', ['foo', 'bar'], ['show', 'fill']) 84 | 85 | t = T() 86 | 87 | # foo 88 | self.provider.identity = 1 89 | t.abtest.record('show') 90 | t.abtest.record('show') 91 | t.abtest.record('fill') 92 | 93 | # bar 94 | self.provider.identity = 2 95 | t.abtest.record('show') 96 | t.abtest.record('fill') 97 | 98 | # foo 99 | self.provider.identity = 3 100 | t.abtest.record('fill') 101 | 102 | # bar 103 | self.provider.identity = 4 104 | t.abtest.record('show') 105 | 106 | report = self.storage.report('foobar') 107 | 108 | expected = { 109 | 'test_name': 'foobar', 110 | 'results': [ 111 | { 112 | 'alternative': 'foo', 113 | 'funnel': [{ 114 | 'stage': ('show', 'fill'), 115 | 'attempted': 1, 116 | 'converted': 1, 117 | }], 118 | }, 119 | { 120 | 'alternative': 'bar', 121 | 'funnel': [{ 122 | 'stage': ('show', 'fill'), 123 | 'attempted': 2, 124 | 'converted': 1, 125 | }], 126 | } 127 | ], 128 | } 129 | 130 | self.assertEquals(report, expected) 131 | 132 | def test_funnel(self): 133 | class T(object): 134 | abtest = ABTest('foobar', ['foo', 'bar'], ['a', 'b', 'c', 'd']) 135 | 136 | t = T() 137 | 138 | # foo 139 | self.provider.identity = 1 140 | t.abtest.record('a') 141 | t.abtest.record('b') 142 | 143 | # bar 144 | self.provider.identity = 2 145 | t.abtest.record('a') 146 | t.abtest.record('b') 147 | t.abtest.record('c') 148 | 149 | # foo 150 | self.provider.identity = 3 151 | t.abtest.record('a') 152 | t.abtest.record('b') 153 | t.abtest.record('c') 154 | 155 | # bar 156 | self.provider.identity = 4 157 | t.abtest.record('a') 158 | t.abtest.record('b') 159 | t.abtest.record('c') 160 | t.abtest.record('d') 161 | 162 | expected = { 163 | 'test_name': 'foobar', 164 | 'results': [ 165 | { 166 | 'alternative': 'foo', 167 | 'funnel': [ 168 | { 169 | 'stage': ('a', 'b'), 170 | 'attempted': 2, 171 | 'converted': 2, 172 | }, 173 | { 174 | 'stage': ('b', 'c'), 175 | 'attempted': 2, 176 | 'converted': 1, 177 | }, 178 | { 179 | 'stage': ('c', 'd'), 180 | 'attempted': 1, 181 | 'converted': 0, 182 | }, 183 | ], 184 | }, 185 | { 186 | 'alternative': 'bar', 187 | 'funnel': [ 188 | { 189 | 'stage': ('a', 'b'), 190 | 'attempted': 2, 191 | 'converted': 2, 192 | }, 193 | { 194 | 'stage': ('b', 'c'), 195 | 'attempted': 2, 196 | 'converted': 2, 197 | }, 198 | { 199 | 'stage': ('c', 'd'), 200 | 'attempted': 2, 201 | 'converted': 1, 202 | }, 203 | ], 204 | } 205 | ], 206 | } 207 | 208 | report = self.storage.report('foobar') 209 | self.assertEquals(report, expected) 210 | 211 | def test_list_tests(self): 212 | class T(object): 213 | first = ABTest('first', ['a', 'b'], ['a', 'b']) 214 | second = ABTest('second', ['a', 'b'], ['a', 'b']) 215 | 216 | T() 217 | 218 | tests = self.storage.list_tests() 219 | self.assertEquals(2, len(tests)) 220 | self.assertTrue('first' in tests, '"first" should be in tests') 221 | self.assertTrue('second' in tests, '"second" should be in tests') 222 | 223 | 224 | funcs = { 225 | 'test_one': test_one, 226 | 'test_two': test_two, 227 | 'test_funnel': test_funnel, 228 | 'test_list_tests': test_list_tests, 229 | } 230 | if setUp_func: 231 | funcs['setUp'] = setUp_func 232 | if tearDown_func: 233 | funcs['tearDown'] = tearDown_func 234 | 235 | return type(name, (unittest.TestCase, ), funcs) 236 | 237 | def generic_setUp(self): 238 | # also mock random.randrange with a callable 239 | # object which will tell us what the "random" 240 | # value was 241 | self.randrange = RandRange() 242 | dabble.random.randrange = self.randrange 243 | 244 | def mongo_setUp(self): 245 | generic_setUp(self) 246 | 247 | conn = pymongo.Connection() 248 | db = conn.dabble_test 249 | for collection in db.collection_names(): 250 | if collection.startswith('dabble'): 251 | db.drop_collection(collection) 252 | 253 | self.storage = MongoResultStorage(db) 254 | self.provider = MockIdentityProvider() 255 | configure(self.provider, self.storage) 256 | 257 | def fs_setUp(self): 258 | generic_setUp(self) 259 | 260 | here = dirname(__file__) 261 | storage_dir = join(here, 'storage') 262 | if exists(storage_dir): 263 | rmtree(storage_dir) 264 | makedirs(storage_dir) 265 | 266 | self.storage = FSResultStorage(storage_dir) 267 | self.provider = MockIdentityProvider() 268 | configure(self.provider, self.storage) 269 | 270 | 271 | def generic_tearDown(self): 272 | # pretend like the previous test never happened 273 | dabble.AB._id_provider = None 274 | dabble.AB._storage = None 275 | dabble.AB._AB__n_per_test = {} 276 | 277 | del self.storage 278 | del self.provider 279 | 280 | def mongo_tearDown(self): 281 | generic_tearDown(self) 282 | 283 | conn = pymongo.Connection() 284 | db = conn.dabble_test 285 | for collection in ('dabble.tests', 'dabble.results'): 286 | db.drop_collection(collection) 287 | 288 | def fs_tearDown(self): 289 | generic_tearDown(self) 290 | 291 | here = dirname(__file__) 292 | storage_dir = join(here, 'storage') 293 | if exists(storage_dir): 294 | rmtree(storage_dir) 295 | 296 | 297 | MongoReportTest = ReportTestFor('MongoReportTest', mongo_setUp, mongo_tearDown) 298 | FSReportTest = ReportTestFor('FSReportTest', fs_setUp, fs_tearDown) 299 | 300 | if __name__ == '__main__': 301 | unittest.main() 302 | 303 | --------------------------------------------------------------------------------