29 | JavaScript mode
30 |
31 |
32 |
81 |
82 |
90 |
91 |
92 | JavaScript mode supports several configuration options:
93 |
94 | json which will set the mode to expect JSON
95 | data rather than a JavaScript program.jsonld which will set the mode to expect
97 | JSON-LD linked data rather
98 | than a JavaScript program (demo).typescript which will activate additional
100 | syntax highlighting and some other things for TypeScript code
101 | (demo).statementIndent which (given a number) will
103 | determine the amount of indentation to use for statements
104 | continued on a new line.
106 |
107 |
108 | MIME types defined: text/javascript, application/json, application/ld+json, text/typescript, application/typescript.
109 |
110 |
--------------------------------------------------------------------------------
/planout-editor/static/js/mode/javascript/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
29 | JavaScript mode
30 |
31 |
32 |
81 |
82 |
90 |
91 |
92 | JavaScript mode supports several configuration options:
93 |
94 | json which will set the mode to expect JSON
95 | data rather than a JavaScript program.jsonld which will set the mode to expect
97 | JSON-LD linked data rather
98 | than a JavaScript program (demo).typescript which will activate additional
100 | syntax highlighting and some other things for TypeScript code
101 | (demo).statementIndent which (given a number) will
103 | determine the amount of indentation to use for statements
104 | continued on a new line.
106 |
107 |
108 | MIME types defined: text/javascript, application/json, application/ld+json, text/typescript, application/typescript.
109 |
110 |
--------------------------------------------------------------------------------
/python/docs/06-best-practices.md:
--------------------------------------------------------------------------------
1 | # Best practices
2 |
3 | PlanOut makes it easy to implement bug-free code that randomly assigns users (or other units) to parameters. The Experiment and Namespace classes are designed to reduce common errors in deploying and logging experiments. Here are a few tips for running experiments:
4 |
5 | * Use auto-exposure logging (link), which is enabled by default. Auto-exposure logging makes it easier to check that your assignment procedure is working correctly, increases the precision of your experiment, and reduces errors in downstream analysis.
6 | * Avoid changing an experiment while it is running. Instead, either run a follow-up experiment using a namespace (link), or create a new experiment with a different salt to re-randomize the units. These experiments should be analyzed separately from the original experiment.
7 | * Automate the analysis of your experiment. If you are running multiple related experiments, create a pipeline to automatically do the analysis.
8 |
9 |
10 | There are no hard and fast rules for what kinds of changes are actually a problem, but if you follow the best practices above, you should be in reasonable shape.
11 |
12 |
13 | ## Randomization failures
14 | Experiments are used to test the change of one or more parameters on some average outcome (e.g., messages sent, or clicks on a button). Differences can be safely attributed to a change in parameters if treatments are assigned to users completely at random.
15 |
16 | In practice, there are a number of common ways for two groups to not be equivalent (beyond random imbalance):
17 | - Some units from one group were previously in a different group, while users from the other group were not.
18 | - Some units in one group were recently added to the experiment.
19 | - There was a bug in the code for one group but not the other, and that bug recently got fixed.
20 |
21 | In these cases, we suggest that you launch a new experiment, either through the use of namespaces (links), or by re-assigning all of the units in your experiment. This can be done by simply changing the salt of your experiment:
22 |
23 | ```python
24 | class MyNewExperiment(MyOldExperiment):
25 | def set_experiment_properties(self):
26 | self.name = 'new_experiment_name'
27 | self.salt = 'new_experiment_salt'
28 | ```
29 |
30 |
31 | ### Unanticipated consequences from changing experiments
32 | Changes to experiment definitions will generally alter which parameters users are assigned to. For example, consider an experiment that manipulates the label of a button for sharing a link. The main outcome of interest is the effect of this text on how many links users share per day.
33 |
34 |
35 | ```python
36 | class SharingExperiment(SimpleExperiment):
37 | def assign(self, params, userid):
38 | params.button_text = UniformChoice(
39 | choices=['OK', 'Share', 'Share with friends'],
40 | unit=userid
41 | )
42 | ```
43 | Changing the variable name `button_text` changes the assignment, since it is used to salt (link) to assignment procedure (see `Experiment` intro document).
44 |
45 | Changing the number of choices for the `button_text` also affects users previously randomized into other conditions. For example, removing the 'Share' item from the `choices` list, will allocate some users who were previosuly in the 'Share' condition to the 'OK' and 'Share with friends group'. Their outcomes will now be a weighted average of the two, which may decrease the observed difference between 'OK' and 'Share with friends'.
46 |
47 | If an additional choice were added to `choices`, some percentage of each prior choice would be allocated to the new choice, whose outcome represents an average of all groups. Comparisons between users still in the old groups (the newly added parameters may be subject to greater novelty effects.
48 |
49 | ## Detecting problems
50 | If you suspect your experiment might have changed, check the `salt` and `checksum` fields of your log. If either of these items change, it is likely that your assignments have also changed mid-way through the experiment.
51 |
52 | ## Learn more
53 | [Designing and Deploying Online Field Experiments](http://www-personal.umich.edu/~ebakshy/planout.pdf). WWW 2014. Eytan Bakshy, Dean Eckles, Michael S. Bernstein.
54 | [Seven Pitfalls to Avoid when Running Controlled Experiments on the Web.](http://www.exp-platform.com/Documents/2009-ExPpitfalls.pdf) KDD 2009. Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham.
55 |
--------------------------------------------------------------------------------
/python/planout/ops/random.py:
--------------------------------------------------------------------------------
1 | import hashlib
2 | import six
3 | from .base import PlanOutOpSimple
4 |
5 |
6 | class PlanOutOpRandom(PlanOutOpSimple):
7 | LONG_SCALE = float(0xFFFFFFFFFFFFFFF)
8 |
9 | def getUnit(self, appended_unit=None):
10 | unit = self.getArgMixed('unit')
11 | if type(unit) is not list:
12 | unit = [unit]
13 | if appended_unit is not None:
14 | unit += [appended_unit]
15 | return unit
16 |
17 | def getHash(self, appended_unit=None):
18 | if 'full_salt' in self.args:
19 | full_salt = self.getArgString('full_salt') + '.' # do typechecking
20 | else:
21 | full_salt = '%s.%s%s' % (
22 | self.mapper.experiment_salt,
23 | self.getArgString('salt'),
24 | self.mapper.salt_sep)
25 |
26 | unit_str = '.'.join(map(str, self.getUnit(appended_unit)))
27 | hash_str = '%s%s' % (full_salt, unit_str)
28 | if not isinstance(hash_str, six.binary_type):
29 | hash_str = hash_str.encode("ascii")
30 | return int(hashlib.sha1(hash_str).hexdigest()[:15], 16)
31 |
32 | def getUniform(self, min_val=0.0, max_val=1.0, appended_unit=None):
33 | zero_to_one = self.getHash(appended_unit) / PlanOutOpRandom.LONG_SCALE
34 | return min_val + (max_val - min_val) * zero_to_one
35 |
36 |
37 | class RandomFloat(PlanOutOpRandom):
38 |
39 | def simpleExecute(self):
40 | min_val = self.getArgFloat('min')
41 | max_val = self.getArgFloat('max')
42 |
43 | return self.getUniform(min_val, max_val)
44 |
45 |
46 | class RandomInteger(PlanOutOpRandom):
47 |
48 | def simpleExecute(self):
49 | min_val = self.getArgInt('min')
50 | max_val = self.getArgInt('max')
51 |
52 | return min_val + self.getHash() % (max_val - min_val + 1)
53 |
54 |
55 | class BernoulliTrial(PlanOutOpRandom):
56 |
57 | def simpleExecute(self):
58 | p = self.getArgNumeric('p')
59 | assert p >= 0 and p <= 1.0, \
60 | '%s: p must be a number between 0.0 and 1.0, not %s!' \
61 | % (self.__class__, p)
62 |
63 | rand_val = self.getUniform(0.0, 1.0)
64 | return 1 if rand_val <= p else 0
65 |
66 |
67 | class BernoulliFilter(PlanOutOpRandom):
68 |
69 | def simpleExecute(self):
70 | p = self.getArgNumeric('p')
71 | values = self.getArgList('choices')
72 | assert p >= 0 and p <= 1.0, \
73 | '%s: p must be a number between 0.0 and 1.0, not %s!' \
74 | % (self.__class__, p)
75 |
76 | if len(values) == 0:
77 | return []
78 | return [i for i in values if self.getUniform(0.0, 1.0, i) <= p]
79 |
80 |
81 | class UniformChoice(PlanOutOpRandom):
82 |
83 | def simpleExecute(self):
84 | choices = self.getArgList('choices')
85 |
86 | if len(choices) == 0:
87 | return []
88 | rand_index = self.getHash() % len(choices)
89 | return choices[rand_index]
90 |
91 |
92 | class WeightedChoice(PlanOutOpRandom):
93 |
94 | def simpleExecute(self):
95 | choices = self.getArgList('choices')
96 | weights = self.getArgList('weights')
97 |
98 | if len(choices) == 0:
99 | return []
100 | cum_weights = dict(enumerate(weights))
101 | cum_sum = 0.0
102 | for index in cum_weights:
103 | cum_sum += cum_weights[index]
104 | cum_weights[index] = cum_sum
105 | stop_value = self.getUniform(0.0, cum_sum)
106 | for index in cum_weights:
107 | if stop_value <= cum_weights[index]:
108 | return choices[index]
109 |
110 |
111 | class BaseSample(PlanOutOpRandom):
112 |
113 | def copyChoices(self):
114 | return [x for x in self.getArgList('choices')]
115 |
116 | def getNumDraws(self, choices):
117 | if 'draws' in self.args:
118 | num_draws = self.getArgInt('draws')
119 | assert num_draws <= len(choices), \
120 | "%s: cannot make %s draws when only %s choices are available" \
121 | % (self.__class__, num_draws, len(choices))
122 | return num_draws
123 | else:
124 | return len(choices)
125 |
126 | class FastSample(BaseSample):
127 |
128 | def simpleExecute(self):
129 | choices = self.copyChoices()
130 | num_draws = self.getNumDraws(choices)
131 | stopping_point = len(choices) - num_draws
132 |
133 | for i in six.moves.range(len(choices) - 1, 0, -1):
134 | j = self.getHash(i) % (i + 1)
135 | choices[i], choices[j] = choices[j], choices[i]
136 | if stopping_point == i:
137 | return choices[i:]
138 | return choices[:num_draws]
139 |
140 | class Sample(BaseSample):
141 |
142 | def simpleExecute(self):
143 | choices = self.copyChoices()
144 | num_draws = self.getNumDraws(choices)
145 |
146 | for i in six.moves.range(len(choices) - 1, 0, -1):
147 | j = self.getHash(i) % (i + 1)
148 | choices[i], choices[j] = choices[j], choices[i]
149 | return choices[:num_draws]
150 |
--------------------------------------------------------------------------------
/planout-editor/js/components/PlanOutTesterBox.react.js:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 Facebook, Inc.
3 | *
4 | * @providesModule PlanOutTesterBox.react
5 | * @jsx React.DOM
6 | */
7 |
8 | var React = require('react');
9 | var ReactPropTypes = React.PropTypes;
10 |
11 | var PlanOutTesterActions = require('../actions/PlanOutTesterActions');
12 | var PlanOutTesterBoxForm = require('./PlanOutTesterBoxForm.react');
13 | var PlanOutTesterBoxOutput = require('./PlanOutTesterBoxOutput.react');
14 |
15 | var PlanOutEditorConstants = require('../constants/PlanOutEditorConstants');
16 | var TesterBoxTypes = PlanOutEditorConstants.TesterBoxTypes;
17 | var TesterStatusCodes = PlanOutEditorConstants.TesterStatusCodes;
18 |
19 | //var Bootstrap = require('react-bootstrap');
20 | //var Panel = Bootstrap.Panel;
21 |
22 |
23 | var PlanOutTesterBox = React.createClass({
24 | propTypes: {
25 | id: ReactPropTypes.string.isRequired,
26 | assertions: ReactPropTypes.object,
27 | errors: ReactPropTypes.array,
28 | inputs: ReactPropTypes.object,
29 | overrides: ReactPropTypes.object,
30 | results: ReactPropTypes.object,
31 | status: ReactPropTypes.string.isRequired,
32 | type: ReactPropTypes.string.isRequired
33 | },
34 |
35 | getInitialState: function() {
36 | return {expanded: this.props.type === TesterBoxTypes.PLAYGROUND};
37 | },
38 |
39 | _toggleExpand: function() {
40 | this.setState({expanded: !this.state.expanded});
41 | },
42 |
43 | _destroy: function() {
44 | PlanOutTesterActions.destroy(this.props.id);
45 | },
46 |
47 | render: function() {
48 | var titleBarSettings = this.getTitleBarStrings();
49 |
50 | var collapse_class = "panel-collapse collapse";
51 | collapse_class += this.state.expanded ? " in" : "";
52 |
53 | return (
54 | 
13 |
14 |
19 |