8 |
9 | **Using interactions to drive cell computation using _reactive cells_**
10 |
11 | 
12 |
13 | And more---see [here](https://github.com/yifanwu/midas-exp-pub) for tutorials. If you ran into problems, feel free to [open an issue](https://github.com/yifanwu/b2/issues/new/choose), or email Yifan directly at yifanwu@berkeley.edu, or [via Twitter](https://twitter.com/yifanwu).
14 |
15 | ## Installation
16 |
17 | ```sh
18 | pip install b2-ext
19 | ```
20 |
21 | ## Development
22 |
23 | To build your own JS bundle:
24 |
25 | ```sh
26 | pip install -r requirements.txt
27 | python setup.py develop
28 | jupyter nbextension install --py --symlink b2
29 | ```
30 |
31 | ```sh
32 | npm run install
33 | npm run watch
34 | ```
35 |
36 | ## Deployment
37 |
38 | ```sh
39 | npm run build
40 | python setup.py sdist
41 | twine upload dist/*
42 | ```
43 |
44 | You may need `pip install twine` if you do not have `twine` already; as well as a PyPi account and permissions.
45 |
46 | ## People
47 |
48 | B2 is developed by [Yifan](http://yifanwu.net/) at UC Berkeley, with advising from [Joe](https://www2.eecs.berkeley.edu/Faculty/Homepages/hellerstein.html), [Arvind](https://arvindsatya.com/), and others.
49 |
50 | If you are interested in using B2 or participating in our user study, please send Yifan a message at yifanwu@berkeley.edu.
51 |
--------------------------------------------------------------------------------
/b2/__init__.py:
--------------------------------------------------------------------------------
1 | from __future__ import absolute_import
2 | from warnings import warn
3 |
4 | from .b2 import B2
5 | from .algebra.dataframe import MidasDataFrame
6 |
7 | __all__ = ['B2', 'MidasDataFrame']
8 |
9 | __version__ = '0.0.5'
10 |
11 |
12 | def _jupyter_nbextension_paths():
13 | """Return metadata for the nbextension."""
14 | return [dict(
15 | section="notebook",
16 | # the path is relative to the `midas` directory
17 | src="static",
18 | # directory in the `nbextension/` namespace
19 | dest="b2",
20 | # _also_ in the `nbextension/` namespace
21 | require="b2/index")]
22 |
23 |
24 | def find_static_assets():
25 | warn("""To use the nbextension, you'll need to update
26 | the Jupyter notebook to version 4.2 or later.""")
27 | return []
28 |
--------------------------------------------------------------------------------
/b2/algebra/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yifanwu/b2/3dc11605a365a28ed07c8a2b253ff87fa43e549d/b2/algebra/__init__.py
--------------------------------------------------------------------------------
/b2/algebra/context.py:
--------------------------------------------------------------------------------
1 | from copy import deepcopy, copy
2 | from typing import List, Dict, Optional, cast, Tuple
3 | from collections import defaultdict
4 | # for development
5 | from IPython.core.debugger import set_trace
6 |
7 | from b2.util.errors import InternalLogicalError, debug_log
8 | from b2.state_types import DFName
9 | from b2.constants import ISDEBUG
10 |
11 | from .dataframe import RelationalOpType, MidasDataFrame, BaseOp, RelationalOp, DFInfo, VisualizedDFInfo, Where, JoinInfo, Select, create_predicate, Join
12 | from .selection import SelectionValue
13 |
14 | class Context(object):
15 | # dfs: Dict[DFName, DFInfo]
16 | join_info: Dict[Tuple[DFName, DFName], JoinInfo]
17 | # store it for easier df gen...
18 |
19 | def __init__(self, df_info_store: Dict[DFName, DFInfo], new_df_from_ops):
20 | """[summary]
21 |
22 | Arguments:
23 | get_dfs {Callable[[], List[MidasDataFrame]]} -- passed from state so that they share access to the up tp date state
24 | """
25 | self.df_info_store = df_info_store
26 | self.join_info = {}
27 | self.new_df_from_ops = new_df_from_ops
28 |
29 |
30 | def get_df(self, df_name: DFName) -> MidasDataFrame:
31 | found = self.df_info_store[df_name]
32 | if found:
33 | if isinstance(found, VisualizedDFInfo):
34 | return found.original_df
35 | else:
36 | return found.df
37 | else:
38 | raise InternalLogicalError(f"DF {df_name} not found")
39 |
40 |
41 | # takes in joinable columns on a left and right df, adds this information into our join_info dictionary
42 | # consider making it easier to enter
43 | def add_join_info(self, joins: JoinInfo):
44 | left_df = joins.left_df
45 | right_df = joins.right_df
46 | if left_df.df_name is not None and right_df.df_name is not None: # type: ignore
47 | self.join_info[(left_df.df_name, right_df.df_name)] = joins # type: ignore
48 | self.join_info[(right_df.df_name, left_df.df_name)] = joins.swap_left_right() # type: ignore
49 | else:
50 | raise InternalLogicalError("The DFs with join info should have df_names")
51 |
52 |
53 | # takes in a left_df and right_df, and returns all possible column names for them to be joined on
54 | # def get_join_info(self, left_df: MidasDataFrame, right_df: MidasDataFrame):
55 | # if left_df.df_name is not None and right_df.df_name is not None:
56 | # return
57 | # else:
58 | # raise InternalLogicalError("should have df_names")
59 |
60 |
61 | def apply_join_selection(self, join_info: JoinInfo, selections: List[SelectionValue]) -> RelationalOp:
62 | """
63 | Arguments:
64 | join_info {JoinInfo} -- Note that the left one is the "original base" and the right is the one joined
65 | selections {List[SelectionValue]} -- [description]
66 |
67 | Raises:
68 | NotImplementedError: if there are more than on ecolumn, do not handle for now.
69 |
70 | Returns:
71 | RelationalOp -- [description]
72 | """
73 | # if ISDEBUG: set_trace()
74 | # we know that join info must base baseop
75 | new_ops = cast(BaseOp, deepcopy(join_info.right_df._ops)) # type: ignore
76 | filtered_join_df = apply_non_join_selection(new_ops, selections)
77 | if len(join_info.columns) > 1:
78 | raise NotImplementedError("Currently not supported")
79 | base_col = join_info.columns[0].left_col.col_name
80 | join_col = join_info.columns[0].right_col.col_name
81 | index_column_df = self.new_df_from_ops(Select(join_col, filtered_join_df))
82 | # we must assign something
83 | selection_columns = "_".join([s.column.col_name for s in selections])
84 | index_column_df._suggested_df_name = f"{new_ops.df_name}_filtered_{selection_columns}"
85 | # then do the join
86 | # TODO: better to do the "in" operations
87 | new_base = deepcopy(join_info.left_df._ops) # type: ignore
88 | final_ops = Join(base_col, index_column_df, join_col, new_base)
89 | # if ISDEBUG: set_trace()
90 | return final_ops
91 |
92 | # get a bunch of bases and decide where the column comes from
93 | # we might have 2 here, and we need to decide which one to pick
94 | # based on the column names, this is easy if
95 | # Joins have a certain behaviors in datascience that
96 | # makes it hard to reason about which column things came from,
97 | # for now, let’s just give up. :(, maybe @shloak can take a stab? #TODO/FIXME
98 | def get_base_df_selection(self, s: SelectionValue) -> Optional[SelectionValue]:
99 | # look up df
100 | df = self.get_df(s.column.df_name)
101 | bases = find_all_baseops(df._ops)
102 |
103 | def find_base_with_column(bases: List[BaseOp]):
104 | for b in bases:
105 | a_df = self.get_df(b.df_name)
106 |
107 | # LEAKY abstraction --- indirectly depending on tables...
108 | if s.column.col_name in a_df.table.labels:
109 | # we are done
110 | return b
111 | base_op = find_base_with_column(bases)
112 | new_selection = deepcopy(s)
113 | if (base_op):
114 | new_selection.column.df_name = base_op.df_name
115 | return new_selection
116 | else:
117 | return None
118 | # raise InternalLogicalError(f"base selection should have been found for {s.column.df_name}filtered by columns {s.column.col_name}")
119 |
120 |
121 | def apply_selection(self, target_df: MidasDataFrame, selections: List[SelectionValue], is_union=False) -> Optional[MidasDataFrame]:
122 | if len(selections) == 0:
123 | return None
124 | # before we do any of that, just check to see if the filter is directly on the target_df itself?
125 | selections_on_base = map(self.get_base_df_selection, selections)
126 | selections_by_df = defaultdict(list)
127 | for s in selections_on_base:
128 | # note that if something is not found, we simply ignore it
129 | # this sometimes happens when we miscategorize.
130 | if s is not None:
131 | selections_by_df[s.column.df_name].append(s)
132 |
133 | if target_df.df_name in selections_by_df:
134 | raise InternalLogicalError(f"Shouldn't be using context to do the filter if the two DFs are the same, we got {target_df.df_name} as target, which is in {selections_by_df.keys()}")
135 |
136 | new_ops = target_df._ops
137 | # it doesn't really matter what order we apply these in
138 | for df_name in selections_by_df.keys():
139 | new_ops = self.apply_selection_from_single_df(new_ops, df_name, selections_by_df[df_name]) # type: ignore
140 | new_df = target_df.new_df_from_ops(new_ops) # type: ignore
141 | return new_df
142 |
143 |
144 | def find_joinable_base(self, current_bases: List[BaseOp], selection_base_df: DFName) -> Optional[JoinInfo]:
145 | """
146 | note that the current_base is left_df, and the base to join with is right_df
147 | """
148 | for b in current_bases:
149 | r = self.join_info.get((b.df_name, selection_base_df))
150 | if r is not None:
151 | return r
152 | return None
153 |
154 |
155 | def apply_selection_from_single_df(self, ops: RelationalOp, df_name: DFName, selections: List[SelectionValue]) -> RelationalOp:
156 | # here we can assume that all the selections have the same df
157 | bases = find_all_baseops(ops)
158 | # see if the selection list has anything in the bases
159 | non_join_base_list = list(filter(lambda b: b.df_name == df_name, bases))
160 | if len(non_join_base_list) > 0:
161 | non_join_base = non_join_base_list[0]
162 | local_base_df_name = non_join_base.df_name
163 | replacement_op = apply_non_join_selection(non_join_base, selections)
164 | else:
165 | # search for which one we can actually join with
166 | r = self.find_joinable_base(bases, df_name)
167 | if r:
168 | # it's always the right one (by construct)
169 | local_base_df_name = r.left_df.df_name
170 | replacement_op = self.apply_join_selection(r, selections)
171 | else:
172 | # NO OP
173 | if ISDEBUG:
174 | debug_log(f"No op for {df_name} selection because no join was found")
175 | return ops
176 |
177 | # 2. apply the replacement
178 | if replacement_op and local_base_df_name:
179 | return set_if_eq(deepcopy(ops), replacement_op, local_base_df_name)
180 | raise InternalLogicalError("Replacement Op is not set or the df_name is not set")
181 |
182 |
183 | ####################################
184 | ######## helper funcs ########
185 | ####################################
186 |
187 | def set_if_eq(original: RelationalOp, replacement: RelationalOp, df_name: DFName):
188 | should_return_replacement = False
189 | # note that we need the special casing because of the special case
190 | def _helper(op: RelationalOp, new_op: RelationalOp, parent_op: Optional[RelationalOp]):
191 | if (op.op_type == RelationalOpType.base):
192 | base_op = cast(BaseOp, op)
193 | if (base_op.df_name == df_name):
194 | # if parent_op is not defined, then we are literally replacing
195 | if parent_op is None:
196 | should_return_replacement = True
197 | return
198 | else:
199 | parent_op.child = new_op
200 | return
201 | elif (op.has_child()):
202 | return _helper(op.child, new_op, op)
203 | else:
204 | raise InternalLogicalError("Should either have child or be of base type")
205 |
206 | _helper(original, replacement, None)
207 | if should_return_replacement:
208 | return replacement
209 | else:
210 | return original
211 |
212 |
213 |
214 | def find_all_baseops(op: RelationalOp) -> List[BaseOp]:
215 | """takes the source op and returns all the baseops
216 | e.g. given that df and df2 are loaded in as external data,
217 | then the op representing `df.join("id", df2, "id")select(["sales"])`
218 | will return df and df2's respective `baseop`s.
219 |
220 | Arguments:
221 | op {RelationalOp} -- [description]
222 |
223 | Returns:
224 | List[BaseOp] -- [description]
225 | """
226 | if (op.op_type == RelationalOpType.base):
227 | base_op = cast(BaseOp, op)
228 | return [base_op]
229 | if (op.op_type == RelationalOpType.join):
230 | join_op = cast(Join, op)
231 | b1 = find_all_baseops(op.child)
232 | b2 = find_all_baseops(join_op.other._ops)
233 | return b1 + b2
234 | if (op.has_child()):
235 | return find_all_baseops(op.child)
236 | else:
237 | return []
238 |
239 |
240 | def apply_non_join_selection(ops: BaseOp, selections: List[SelectionValue]) -> RelationalOp:
241 | # it has to be BaseOp because it's used to generate the df to be replaced
242 | executable_predicates = list(map(create_predicate, selections))
243 | new_ops = copy(ops)
244 | for p in executable_predicates:
245 | new_ops = Where(p, new_ops)
246 | return new_ops
247 |
248 |
--------------------------------------------------------------------------------
/b2/algebra/data_types.py:
--------------------------------------------------------------------------------
1 | from typing import NewType, Dict, List, Any
2 |
3 | DFId = NewType('DFId', str)
4 |
5 | # Record = Dict[str, Any]
6 | # Relation = List[Record]
7 |
--------------------------------------------------------------------------------
/b2/algebra/selection.py:
--------------------------------------------------------------------------------
1 | from typing import List, cast, Set
2 | from enum import Enum
3 | import json
4 |
5 | from b2.util.utils import FG_BLUE, RESET_PRINT
6 | from b2.util.errors import InternalLogicalError
7 | from b2.state_types import DFName
8 |
9 | class ColumnRef(object):
10 | def __init__(self, col_name: str, df_name: DFName):
11 | self.col_name = col_name
12 | self.df_name = df_name
13 |
14 | def __eq__(self, other: 'ColumnRef'):
15 | if self.col_name != other.col_name:
16 | return False
17 | if self.df_name != other.df_name:
18 | return False
19 | return True
20 |
21 | def __repr__(self):
22 | return f"{{col_name: '{self.df_name}', df_name: '{self.col_name}'}}"
23 |
24 |
25 | class SelectionType(Enum):
26 | single_value = "single_value"
27 | numeric_range = "numeric_range"
28 | string_set = "string_set"
29 | empty = "empty"
30 |
31 |
32 | class SelectionValue(object):
33 | """
34 | column [ColumnRed]
35 | selection_type [SelectionType]
36 | """
37 | column: ColumnRef
38 | selection_type: SelectionType
39 | def __init__(self):
40 | raise InternalLogicalError("SelectionValue is abstract and should not be instantiated")
41 |
42 | def to_str(self):
43 | raise InternalLogicalError("SelectionValue is abstract and should not be instantiated")
44 |
45 |
46 | class EmptySelection(SelectionValue):
47 | def __init__(self, column: ColumnRef):
48 | self.column = column
49 | self.selection_type = SelectionType.empty
50 |
51 | def __eq__(self, other: SelectionValue):
52 | if other.selection_type != SelectionType.empty:
53 | return False
54 | if self.column != other.column:
55 | return False
56 | return True
57 |
58 | def to_str(self):
59 | raise InternalLogicalError("Should not try to make empty selections into strings")
60 |
61 | def __repr__(self):
62 | return f"{{column: {self.column}, val: {FG_BLUE}None{RESET_PRINT}, minVal: None, maxVal: None}}"
63 |
64 |
65 | class NumericRangeSelection(SelectionValue):
66 | def __init__(self, column: ColumnRef, minVal: float, maxVal: float):
67 | self.selection_type = SelectionType.numeric_range
68 | self.column = column
69 | self.minVal = minVal
70 | self.maxVal = maxVal
71 |
72 | def __eq__(self, other: SelectionValue):
73 | if other.selection_type != SelectionType.numeric_range:
74 | return False
75 | m_other = cast(NumericRangeSelection, other)
76 | if self.column != m_other.column:
77 | return False
78 | if self.minVal != m_other.minVal:
79 | return False
80 | if self.maxVal != m_other.maxVal:
81 | return False
82 | return True
83 |
84 | def __repr__(self):
85 | return f"{{column: {self.column}, minVal: {self.minVal}, maxVal: {self.maxVal}}}"
86 |
87 | def __str__(self) -> str:
88 | return self.__repr__()
89 |
90 | def to_str(self):
91 | return f'{{"{self.column.df_name}": {{"{self.column.col_name}": [{self.minVal}, {self.maxVal}]}}}}'
92 |
93 |
94 |
95 | class SetSelection(SelectionValue):
96 | def __init__(self, column: ColumnRef, val: Set):
97 | self.selection_type = SelectionType.string_set
98 | self.column = column
99 | self.val = val
100 |
101 | def __eq__(self, other: SelectionValue):
102 | if other.selection_type != SelectionType.string_set:
103 | return False
104 | if self.column != other.column:
105 | return False
106 |
107 | s_other = cast(SetSelection, other)
108 | # python has convenient set operations...
109 | if self.val != s_other.val:
110 | return False
111 | return True
112 |
113 | def __repr__(self):
114 | return f"{{column: {self.column}, val: {FG_BLUE}{self.val}{RESET_PRINT}}}"
115 |
116 | def __str__(self) -> str:
117 | return self.__repr__()
118 |
119 | def to_str(self):
120 | return f'{{"{self.column.df_name}": {{"{self.column.col_name}": {json.dumps(self.val)}}}}}'
121 |
122 |
123 | def diff_selection_value(new_selection: List[SelectionValue], old_selection: List[SelectionValue])-> List[SelectionValue]:
124 | """returns the difference between the values
125 | Arguments:
126 | new_selection {List[SelectionValue]} -- one selection
127 | old_selection {List[SelectionValue]} -- another selection
128 | Returns:
129 | returns
130 | - None if there are no changes
131 | - an empty selection if the selection is removed
132 | - all the new diffs as selections
133 | """
134 | def find_selection(a_selection: SelectionValue, selections: List[SelectionValue]):
135 | for s in selections:
136 | if s == a_selection:
137 | return True
138 | return False
139 |
140 | def find_df(df: ColumnRef, selections: List[SelectionValue]):
141 | for s in selections:
142 | if s.column == df:
143 | return True
144 | return False
145 |
146 | diff = []
147 | for s in new_selection:
148 | if not find_selection(s, old_selection):
149 | diff.append(s)
150 | for s in old_selection:
151 | if not find_df(s.column, new_selection):
152 | # this means that this item has been removed
153 | diff.append(EmptySelection(s.column))
154 | return diff
155 |
156 |
157 | def find_selections_with_df_name(current_selection: List[SelectionValue], df_name):
158 | r = []
159 | for s in current_selection:
160 | if s.column.df_name == df_name:
161 | r.append(s.column)
162 | return r
163 |
--------------------------------------------------------------------------------
/b2/b2_magic.py:
--------------------------------------------------------------------------------
1 | from IPython import get_ipython # type: ignore
2 | from IPython.core.magic import (Magics, magics_class, line_magic, cell_magic, line_cell_magic) # type: ignore
3 | from IPython.core.magic_arguments import (argument, magic_arguments, parse_argstring) # type: ignore
4 |
5 | from .ui_comm import UiComm
6 |
7 | @magics_class
8 | class B2Magic(Magics):
9 | ui_comm: UiComm
10 | def __init__(self, shell, ui_comm: UiComm):
11 | super(B2Magic, self).__init__(shell)
12 | self.ui_comm = ui_comm
13 |
14 | @cell_magic
15 | @magic_arguments()
16 | @argument('-disable', action="store_true", help="disable this reactive cell")
17 | @argument('-df', action='store', help='the name of the df, do not set this flag if you wish the cell to be ran for all interactions')
18 | # add another argument such that they can use append based
19 | # @argument('--append', action="store")
20 | def reactive(self, line: str, cell: str):
21 | args = parse_argstring(self.reactive, line)
22 | # do_append = "append" in args
23 | if args.disable:
24 | # self.ui_comm.send_debug_msg("disabled")
25 | self.ui_comm.remove_reactive_cell()
26 | # do NOT execute the cell
27 | return
28 | if args.df:
29 | # self.ui_comm.send_debug_msg(f"NEW cell magic with shell: {args.df} with cell: {cell}")
30 | self.ui_comm.add_reactive_cell(args.df)
31 | else:
32 | self.ui_comm.add_reactive_cell("")
33 |
34 | shell = get_ipython().get_ipython()
35 | shell.run_cell(cell)
--------------------------------------------------------------------------------
/b2/config.py:
--------------------------------------------------------------------------------
1 | class MidasConfig(object):
2 | def __init__(self, linked: bool):
3 | self.linked = linked
4 |
5 |
6 | IS_DEBUG = True
--------------------------------------------------------------------------------
/b2/constants.py:
--------------------------------------------------------------------------------
1 | ISDEBUG = False
2 | STUB_DISTRIBUTION_BIN = "10"
3 | MIDAS_CELL_COMM_NAME = "midas-cell-comm"
4 | MIDAS_RECOVERY_COMM_NAME = "midas-recovery-comm"
5 | DATA_SOURCE = "table"
6 | # the values will either be 0 (filtered) or 1 (original)
7 | IS_OVERVIEW_FIELD_NAME = "is_overview"
8 | COUNT_COL_NAME = "count"
9 | # to accomate the number of US states
10 | MAX_BINS = 100
11 | # note that max generated bins is much smaller than the allowed bins to make the chart we create a little nicer looking.
12 | MAX_GENERATED_BINS = 20
13 | MAX_DOTS = 10000
--------------------------------------------------------------------------------
/b2/context_types.py:
--------------------------------------------------------------------------------
1 | from typing import NamedTuple, List
2 |
3 | class JoinInfo(NamedTuple):
4 | dfs: List[str]
5 | join_colums: List[str]
--------------------------------------------------------------------------------
/b2/showme.py:
--------------------------------------------------------------------------------
1 | from functools import reduce
2 | # from midas.midas_algebra.dataframe import MidasDataFrame, RelationalOpType
3 | from typing import Optional, Dict, cast
4 | from typing_extensions import Literal
5 | from datascience.tables import Table
6 | from pandas.api.types import is_string_dtype, is_numeric_dtype, is_datetime64_any_dtype
7 | from IPython.core.debugger import set_trace
8 |
9 | from .util.errors import type_check_with_warning, InternalLogicalError
10 | from .vis_types import EncodingSpec
11 |
12 |
13 | def toggle_x_y(selection_dimensions: Literal["", "x", "y", "xy"]):
14 | if selection_dimensions == "x":
15 | return "y"
16 | elif selection_dimensions == "y":
17 | return "x"
18 | else:
19 | return selection_dimensions
20 |
21 |
22 | def infer_encoding_helper(df: Table, selectable, is_groupby: bool):
23 | """infers encoding, subject to more parameters, ideally we pass in all of the operations, but for now, we just need is_groupby, which affects the encoding choices.
24 |
25 | Arguments:
26 | df {Table} -- [description]
27 | selectable {[type]} -- [description]
28 | is_groupby {bool} -- whether the ops were groupby
29 | """
30 | df_len = len(df.columns)
31 | if df_len == 2:
32 | first_col = df.labels[0]
33 | second_col = df.labels[1]
34 | selection_dimensions = ""
35 | if len(selectable) == 2:
36 | selection_dimensions = "xy"
37 | elif len(selectable) == 1 and next(iter(selectable)) == first_col:
38 | selection_dimensions = "x"
39 | elif len(selectable) == 1 and next(iter(selectable)) == second_col:
40 | selection_dimensions = "y"
41 | elif len(selectable) == 0:
42 | selection_dimensions = ""
43 | # check if there was a groupby, special case
44 | if is_groupby:
45 | # then the results have to be ordinal
46 | # whether it's multiclick or brush would depend on whether the value is numeric
47 | # if its a groupby, can make the assum0ption that the first one is the ordinal value and the second one is the quantitative value
48 | selection_type = "brush"
49 | sort = ""
50 |
51 | if is_string_dtype(df[first_col]):
52 | selection_type = "multiclick"
53 | # we will arrange it such that it's the second one that's numeric
54 | sort = "-y"
55 |
56 | return EncodingSpec("bar", first_col, "ordinal", second_col, "quantitative", selection_type, selection_dimensions, sort)
57 | if is_string_dtype(df[first_col]) and is_numeric_dtype(df[second_col]):
58 | return EncodingSpec("bar", first_col, "ordinal", second_col, "quantitative", "multiclick", selection_dimensions)
59 | elif is_numeric_dtype(df[first_col]) and is_string_dtype(df[second_col]):
60 | selection_dimensions = toggle_x_y(selection_dimensions)
61 | return EncodingSpec("bar", second_col, "ordinal", first_col, "quantitative", "multiclick", selection_dimensions)
62 | elif is_numeric_dtype(df[first_col]) and is_numeric_dtype(df[second_col]):
63 | return EncodingSpec("circle", first_col, "quantitative", second_col, "quantitative", "brush", selection_dimensions)
64 | elif is_datetime64_any_dtype(df[first_col]) and is_numeric_dtype(df[second_col]):
65 | return EncodingSpec("line", first_col, "temporal", second_col, "quantitative", "brush", selection_dimensions)
66 | elif is_numeric_dtype(df[first_col]) and is_datetime64_any_dtype(df[second_col]):
67 | selection_dimensions = toggle_x_y(selection_dimensions)
68 | return EncodingSpec("line", second_col, "temporal", first_col, "quantitative", "brush", selection_dimensions)
69 | raise InternalLogicalError(f"Corner case in spec gen")
70 | else:
71 | raise InternalLogicalError(f"Midas only supports visualization of two dimensional data for now")
72 |
--------------------------------------------------------------------------------
/b2/state_types.py:
--------------------------------------------------------------------------------
1 | from enum import Enum
2 | from typing import NamedTuple, Callable, List, NewType, Union, Any, Optional, Dict, Tuple
3 |
4 |
5 | DFName = NewType('DFName', str)
6 |
--------------------------------------------------------------------------------
/b2/util/__init__.py:
--------------------------------------------------------------------------------
1 | # the util methods should not depend on anything outside of the util directory
--------------------------------------------------------------------------------
/b2/util/data_processing.py:
--------------------------------------------------------------------------------
1 | from datascience import Table
2 | import json
3 | import numpy as np
4 | from math import log10, pow, floor
5 | from pandas import notnull
6 | from typing import Tuple
7 | from IPython.core.debugger import set_trace
8 |
9 | from b2.constants import IS_OVERVIEW_FIELD_NAME, MAX_BINS, STUB_DISTRIBUTION_BIN, MAX_GENERATED_BINS
10 | from b2.vis_types import FilterLabelOptions, EncodingSpec
11 |
12 | from .errors import InternalLogicalError
13 | from .utils import sanitize_string_for_var_name
14 |
15 |
16 | def get_chart_title(df_name: str):
17 | # one level of indirection in case we need to change in the future
18 | return df_name
19 |
20 |
21 | DATE_HIERARCHY = [
22 | ("Y", "year"),
23 | ("M", "month"),
24 | ("D", "day")
25 | ]
26 |
27 |
28 | # B2DataFrame
29 | def static_vega_gen(encoding: EncodingSpec, df):
30 | records = dataframe_to_dict(df, FilterLabelOptions.none)
31 | # data = json.dumps(records)
32 | if encoding.mark == "bar":
33 | barSpec = {
34 | "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
35 | "height": 200,
36 | "data": {"values": records},
37 | "mark": "bar",
38 | "encoding": {
39 | "x": {
40 | "field": encoding.x,
41 | "type": encoding.x_type
42 | },
43 | "y": {
44 | "field": encoding.y,
45 | "type": encoding.y_type,
46 | },
47 | }
48 | }
49 | if encoding.sort != "":
50 | barSpec["encoding"]["x"]["sort"] = encoding.sort
51 | return barSpec
52 | elif encoding.mark == "circle":
53 | scatterSpec = {
54 | "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
55 | "data": { "values": records },
56 | "mark": {"type": "point", "tooltip": 1},
57 | "encoding": {
58 | "x": {
59 | "field": encoding.x,
60 | "type": encoding.x_type,
61 | "scale": {"zero": 0}
62 | },
63 | "y": {"field": encoding.y, "type": encoding.y_type},
64 | }
65 | }
66 | return scatterSpec
67 | elif encoding.mark == "line":
68 | lineSpec = {
69 | "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
70 | "data": { "values": records },
71 | "mark": "line",
72 | "encoding": {
73 | "x": {"field": encoding.x, "type": encoding.x_type},
74 | "y": {"field": encoding.y, "type": encoding.y_type},
75 | },
76 | }
77 | return lineSpec
78 | else:
79 | return None
80 |
81 |
82 | def get_basic_group_vis(new_name, df_name, col_name):
83 | return f"{new_name} = {df_name}.group('{col_name}')\n{new_name}.vis()"
84 |
85 |
86 | def create_binning_code(bound, col_name, df_name, new_name, midas_reference_name):
87 | bin_column_name = f"{col_name}_bin"
88 | # lambda n: int(n/5) * 5
89 | if bound < 1:
90 | round_num = -1 * floor(log10(bound))
91 | binning_lambda = f"lambda x: 'null' if {midas_reference_name}.np.isnan(x) else round(int(x/{bound}) * {bound}, {round_num})"
92 | else:
93 | binning_lambda = f"lambda x: 'null' if {midas_reference_name}.np.isnan(x) else int(x/{bound}) * {bound}"
94 | bin_transform = f"{df_name}['{bin_column_name}'] = {df_name}.apply({binning_lambda}, '{col_name}')"
95 | grouping_transform = get_basic_group_vis(new_name, df_name, bin_column_name)
96 | code = f"{bin_transform}\n{grouping_transform}"
97 | return code
98 |
99 |
100 | def try_parsing_date_time_level(ref, col_value, col_name, df_name):
101 | parsed = col_value.astype(f'datetime64[{ref[0]}]')
102 | count = len(np.unique(parsed))
103 | new_col_name = sanitize_string_for_var_name(f"{col_name}_{ref[1]}")
104 | if count > 1:
105 | new_column = f"{df_name}['{col_name}_{ref[1]}'] = {df_name}['{col_name}'].astype('datetime64[{ref[0]}]')"
106 | new_name = f"{df_name}_{new_col_name}_dist"
107 | if count > MAX_BINS:
108 | bound = snap_to_nice_number(count/MAX_BINS)
109 | binning_lambda = f"lambda x: 'null' if np.isnan(x) else int(x/{bound}) * {bound}"
110 | bin_column_name = f"{new_col_name}_bin"
111 | bin_transform = f"{df_name}['{bin_column_name}'] = {df_name}.apply({binning_lambda}, '{col_name}')"
112 | grouping = get_basic_group_vis(new_name, df_name, new_col_name)
113 | code = f"{new_column}\n{bin_transform}\n{grouping}"
114 | return code
115 | else:
116 | grouping = get_basic_group_vis(new_name, df_name, new_col_name)
117 | code = f"{new_column}\n{grouping}"
118 | return code
119 | else:
120 | return None
121 |
122 |
123 | # MidasDataFrame
124 | def get_datetime_distribution_code(col_name, df):
125 | col_value = df.table.column(col_name)
126 | for h in DATE_HIERARCHY:
127 | r = try_parsing_date_time_level(h, col_value, col_name, df.df_name)
128 | if r:
129 | return (r, True, "")
130 | return ("", False, "Cannot parse the date time column")
131 |
132 |
133 | def get_numeric_distribution_code(current_max_bins, unique_vals, col_name, df_name, new_name, reference_name) -> Tuple[str, bool, str]:
134 | d_max = unique_vals[-1]
135 | d_min = unique_vals[0]
136 | min_bucket_size = (d_max - d_min) / MAX_GENERATED_BINS
137 | # imports = "import numpy as np"
138 | try:
139 | bound = snap_to_nice_number(min_bucket_size)
140 | code = create_binning_code(bound, col_name, df_name, new_name, reference_name)
141 | return (code, True, "")
142 | except InternalLogicalError as e:
143 | # let's still given them a stub code
144 | code = create_binning_code(STUB_DISTRIBUTION_BIN, col_name, df_name, new_name, reference_name)
145 | return (f"# Please fix the following \n{code}", False, f"We were not able to create a distribution for column {col_name}, df {df_name}, because of error: {e}.")
146 |
147 |
148 | def snap_to_nice_number(n: float):
149 | if n == np.inf:
150 | raise InternalLogicalError("Should not have gotten infinity")
151 | if n <= 0:
152 | raise InternalLogicalError(f"Got {n}")
153 | if (n <= 1):
154 | zeroes = pow(10, abs(int(log10(n))) + 1)
155 | new_num = snap_to_nice_number(n * zeroes)
156 | return new_num/zeroes
157 | # if it's less than 1, make it as big as one and then call the same function and return
158 | if (n <= 2):
159 | return 2
160 | elif (n <= 5):
161 | return 5
162 | elif (n <= 10):
163 | return 10
164 | # bigger than 10, just zero out the digits
165 | zeroes = pow(10, int(log10(n)))
166 | return (int(n / zeroes) + 1) * zeroes
167 |
168 |
169 | # taken from ipyvega
170 | def sanitize_dataframe(df: Table):
171 | """Sanitize a DataFrame to prepare it for serialization.
172 |
173 | copied from the ipyvega project
174 | * Make a copy
175 | * Convert categoricals to strings.
176 | * Convert np.bool_ dtypes to Python bool objects
177 | * Convert np.int dtypes to Python int objects
178 | * Convert floats to objects and replace NaNs/infs with None.
179 | * Convert DateTime dtypes into appropriate string representations
180 | """
181 | import numpy as np
182 |
183 | if df is None:
184 | return None
185 | # raise InternalLogicalError("Cannot sanitize empty df")
186 |
187 | df = df.copy()
188 |
189 | def to_list_if_array(val):
190 | if isinstance(val, np.ndarray):
191 | return val.tolist()
192 | else:
193 | return val
194 |
195 | for col_name in df.labels:
196 | dtype = df.column(col_name).dtype
197 | if str(dtype) == 'category':
198 | # XXXX: work around bug in to_json for categorical types
199 | # https://github.com/pydata/pandas/issues/10778
200 | df[col_name] = df[col_name].astype(str)
201 | elif str(dtype) == 'bool':
202 | # convert numpy bools to objects; np.bool is not JSON serializable
203 | df[col_name] = df[col_name].astype(object)
204 | elif np.issubdtype(dtype, np.integer):
205 | # convert integers to objects; np.int is not JSON serializable
206 | df[col_name] = df[col_name].astype(object)
207 | elif np.issubdtype(dtype, np.floating):
208 | # For floats, convert to Python float: np.float is not JSON serializable
209 | # Also convert NaN/inf values to null, as they are not JSON serializable
210 | col = df[col_name]
211 | bad_values = np.isnan(col) | np.isinf(col)
212 | df[col_name] = np.where(bad_values, None, col).astype(object)
213 | # col.astype(object)[~bad_values]= None
214 | elif str(dtype).startswith('datetime'):
215 | # Convert datetimes to strings
216 | # astype(str) will choose the appropriate resolution
217 | new_column = df[col_name].astype(str)
218 | new_column[new_column == 'NaT'] = ''
219 | df[col_name] = new_column
220 | elif dtype == object:
221 | # Convert numpy arrays saved as objects to lists
222 | # Arrays are not JSON serializable
223 | col = np.vectorize(to_list_if_array)(df[col_name])
224 | df[col_name] = np.where(notnull(col), col, None).astype(object)
225 | return df
226 |
227 | # B2DataFrame
228 | def dataframe_to_dict(df, include_filter_label: FilterLabelOptions):
229 | """[summary]
230 |
231 | Keyword Arguments:
232 | include_filter_label {bool} -- whether we should insert another column indicating (default: {False})
233 | """
234 | clean_df = sanitize_dataframe(df.table)
235 | if clean_df is None:
236 | return []
237 |
238 | def s(x):
239 | k = {}
240 | for i, v in enumerate(x):
241 | k[clean_df.labels[i]] = v
242 | if include_filter_label != FilterLabelOptions.none:
243 | k[IS_OVERVIEW_FIELD_NAME] = include_filter_label.value
244 | return k
245 | return list(map(s, clean_df.rows))
246 |
--------------------------------------------------------------------------------
/b2/util/errors.py:
--------------------------------------------------------------------------------
1 | from typing import Any, Optional
2 | from b2.config import IS_DEBUG
3 |
4 | class bcolors:
5 | HEADER = '\033[95m'
6 | BLUE = '\033[94m'
7 | GREEN = '\033[92m'
8 | WARNING = '\033[93m'
9 | GREY = '\033[37m'
10 | FAIL = '\033[91m'
11 | ENDC = '\033[0m'
12 | BOLD = '\033[1m'
13 | UNDERLINE = '\033[4m'
14 |
15 |
16 | class MockComm(object):
17 | def __init__(self):
18 | pass
19 | def send(self, obj):
20 | print(bcolors.GREY + "sending", obj, bcolors.ENDC)
21 |
22 |
23 | class NullValueError(Exception):
24 | def __init__(self, message):
25 | super().__init__(message)
26 |
27 |
28 | class DebugException(Exception):
29 | def __init__(self, message):
30 | super().__init__(message)
31 |
32 |
33 | class WrongTypeError(Exception):
34 | def __init__(self, message):
35 | super().__init__(message)
36 |
37 |
38 | class DfNotFoundError(Exception):
39 | def __init__(self, message):
40 | super().__init__(message)
41 |
42 |
43 | class NotInRuntimeError(Exception):
44 | def __init__(self, message):
45 | super().__init__(message)
46 |
47 | class InternalLogicalError(Exception):
48 | def __init__(self, message):
49 | super().__init__(message)
50 |
51 |
52 | class UserError(Exception):
53 | def __init__(self, message):
54 | super().__init__(message)
55 |
56 |
57 | class TempDebuggingError(Exception):
58 | def __init__(self, message):
59 | super().__init__(message)
60 |
61 |
62 | class NotAllCaseHandledError(Exception):
63 | def __init__(self, message):
64 | super().__init__(message)
65 |
66 |
67 | def check_not_null(val: Any, err_msg: Optional[str]=None):
68 | if (val == None):
69 | raise NullValueError(err_msg)
70 |
71 |
72 | def type_check_with_warning(val: Any, t: Any):
73 | if not (isinstance(val, t)):
74 | err_msg = f"expected variable to be {t} but got {val} instead"
75 | raise WrongTypeError(err_msg)
76 |
77 |
78 | def report_error_to_user(msg: str):
79 | print(bcolors.WARNING + "[Warning] " + msg + bcolors.ENDC)
80 |
81 |
82 | def logging(function: str, msg: str):
83 | if IS_DEBUG:
84 | print(bcolors.GREEN + f"[{function}]\t\t" + msg + bcolors.ENDC)
85 |
86 |
87 | def debug_log(msg: str):
88 | if IS_DEBUG:
89 | print(bcolors.WARNING + msg + bcolors.ENDC)
90 |
--------------------------------------------------------------------------------
/b2/util/instructions.py:
--------------------------------------------------------------------------------
1 | HELP_INSTRUCTION = """
2 | # TODO
3 | """
--------------------------------------------------------------------------------
/b2/util/utils.py:
--------------------------------------------------------------------------------
1 | import random
2 | import string
3 | import codecs
4 | from os import path
5 | import traceback
6 | import ast
7 | from re import sub
8 | import requests
9 | from pathlib import Path
10 | import time
11 | import numpy as np
12 |
13 | from typing import Tuple, List
14 | from IPython import get_ipython # type: ignore
15 | from IPython.core.debugger import set_trace
16 |
17 | from b2.constants import ISDEBUG
18 | from b2.util.errors import UserError, InternalLogicalError
19 |
20 | FG_BLUE = "\x1b[34m";
21 | RESET_PRINT = "\x1b[0m";
22 | FG_PURPLE = "\x1b[035m"
23 |
24 | def plot(v, center, zoom_start, radius):
25 | import folium
26 | import folium.plugins
27 | locs = v.to_numpy()
28 | us_map = folium.Map(location=center, zoom_start = zoom_start)
29 | heatmap = folium.plugins.HeatMap(locs.tolist(), radius = radius)
30 | us_map.add_child(heatmap)
31 | return us_map
32 |
33 | def plot_heatmap(locs_df, zoom_start=12, radius=12):
34 | """Plots a heatmap using the Folium library
35 |
36 | Arguments:
37 | locs_df {MidasDatabFrame} -- Should contain lat, lon (in that order)
38 |
39 | Keyword Arguments:
40 | zoom_start {int} -- the higher the value, the more zoomed out (default: {12})
41 | radius {int} -- how to aggregate the heatmap (default: {12})
42 | """
43 | # basic data cleaning
44 | # compute the center
45 | center_lat = np.average(locs_df[locs_df.labels[0]])
46 | center_lon = np.average(locs_df[locs_df.labels[1]])
47 | if np.isnan(center_lat):
48 | filtered = locs_df.where(locs_df.labels[0], lambda x: not np.isnan(x))
49 | diff_len = len(locs_df) - len(filtered)
50 | center_lat = np.average(filtered[filtered.labels[0]])
51 | center_lon = np.average(filtered[filtered.labels[1]])
52 | print(f"{FG_PURPLE}[Notification] Filtered out {diff_len} NaN values.{RESET_PRINT}")
53 | return plot(filtered, [center_lat, center_lon], zoom_start, radius)
54 | else:
55 | return plot(locs_df, [center_lat, center_lon], zoom_start, radius)
56 |
57 |
58 |
59 |
60 | def fetch_and_cache(data_url, file, data_dir="data", force=False):
61 | """
62 | Download and cache a url and return the file object.
63 |
64 | data_url: the web address to download
65 | file: the file in which to save the results.
66 | data_dir: (default="data") the location to save the data
67 | force: if true the file is always re-downloaded
68 |
69 | return: The pathlib.Path object representing the file.
70 | """
71 |
72 | data_dir = Path(data_dir)
73 | data_dir.mkdir(exist_ok = True)
74 | file_path = data_dir / Path(file)
75 | # If the file already exists and we want to force a download then
76 | # delete the file first so that the creation date is correct.
77 | if force and file_path.exists():
78 | file_path.unlink()
79 | if force or not file_path.exists():
80 | print('Downloading...', end=' ')
81 | resp = requests.get(data_url)
82 | with file_path.open('wb') as f:
83 | f.write(resp.content)
84 | print('Done!')
85 | last_modified_time = time.ctime(file_path.stat().st_mtime)
86 | else:
87 | last_modified_time = time.ctime(file_path.stat().st_mtime)
88 | print("Using cached version that was downloaded (UTC):", last_modified_time)
89 | return file_path
90 |
91 |
92 | def isnotebook():
93 | try:
94 | shell = get_ipython().__class__.__name__
95 | if shell == 'ZMQInteractiveShell':
96 | return True # Jupyter notebook or qtconsole
97 | elif shell == 'TerminalInteractiveShell':
98 | return False # Terminal running IPython
99 | else:
100 | return False # Other type (?)
101 | except NameError:
102 | return False # Probably standard Python interpreter
103 |
104 |
105 | def red_print(m):
106 | print(f"\x1b[31m{m}\x1b[0m")
107 |
108 | LOG_SQL_SETUP_LOG = """
109 | CREATE TABLE log (
110 | session_id TEXT,
111 | action TEXT,
112 | seconds_since_start INTEGER,
113 | optional_metadata TEXT
114 | );
115 | """
116 |
117 | LOG_SQL_SETUP_SESSION = """
118 | CREATE TABLE session (
119 | user_id TEXT,
120 | task_id TEXT,
121 | session_id TEXT,
122 | start_time TEXT
123 | );
124 | """
125 |
126 |
127 | def abs_path(p: str):
128 | """Make path absolute."""
129 | return path.join(path.dirname(path.abspath(__file__)), p)
130 |
131 |
132 | def check_path(p: str):
133 | if not path.exists(p):
134 | raise UserWarning(f"The path you provided, {p} does not exists")
135 |
136 |
137 | def sanitize_string_for_var_name(p: str):
138 | return sub('[^0-9a-zA-Z]+', '_', p)
139 |
140 |
141 |
142 |
143 | def get_content(path):
144 | """Get content of file."""
145 | with codecs.open(abs_path(path), encoding='utf-8') as f:
146 | return f.read()
147 |
148 |
149 | def get_random_string(stringLength=10):
150 | """Generate a random string of fixed length """
151 | letters = string.ascii_lowercase
152 | return ''.join(random.choice(letters) for i in range(stringLength))
153 |
154 |
155 | def _get_first_target_from_prev_line(stack):
156 | try:
157 | prev_line = stack[-3]
158 | code = prev_line.splitlines()[1]
159 | body = ast.parse(code.strip()).body[0]
160 | first_target = body.targets[0] # type: ignore
161 | return first_target
162 | except:
163 | return None
164 |
165 |
166 | def find_tuple_name():
167 | try:
168 | stack = traceback.format_stack()
169 | first_target = _get_first_target_from_prev_line(stack)
170 | a = first_target.elts[0].id # type: ignore
171 | b = first_target.elts[1].id # type: ignore
172 | return a, b
173 | except:
174 | return None
175 |
176 |
177 | def find_name(throw_error=False):
178 | try:
179 | stack = traceback.format_stack()
180 | first_target = _get_first_target_from_prev_line(stack)
181 | a = first_target.id # type: ignore
182 | if throw_error and (a is None):
183 | raise InternalLogicalError("We did not get a name when expected!")
184 | return a
185 | except:
186 | if throw_error:
187 | raise UserError("We expect you to assign this compute to a variable")
188 | return None
189 |
190 |
191 | ifnone = lambda a, b: b if a is None else a
192 |
193 |
194 | def get_min_max_tuple_from_list(values: List[float]) -> Tuple[float, float]:
195 | """sets in place the array if the values are not min and max
196 |
197 | Arguments:
198 | x_value {List[int]} -- [description]
199 |
200 | Returns:
201 | returns the modifed array in place
202 | """
203 | return (min(values), max(values))
204 |
--------------------------------------------------------------------------------
/b2/vis_types.py:
--------------------------------------------------------------------------------
1 | from enum import Enum
2 | from b2.util.errors import UserError
3 | from typing import List, Any, Dict, Optional
4 | from typing_extensions import Literal
5 |
6 | class ChartType(Enum):
7 | bar_categorical = "bar_categorical"
8 | bar_linear = "bar_linear"
9 | scatter = "scatter"
10 | line = "line"
11 |
12 |
13 | # set to numerical values to save space.
14 | class FilterLabelOptions(Enum):
15 | filtered=False
16 | unfiltered=True
17 | none=2
18 |
19 |
20 | class Channel(Enum):
21 | x = "x"
22 | y = "y"
23 | color = "color"
24 |
25 |
26 | # class SelectionEvent(object):
27 | # def __init__(self, interaction_time: datetime, predicate: List[SelectionValue], df_name: DFName):
28 | # self.interaction_time = interaction_time
29 | # self.predicate = predicate
30 | # self.df_name = df_name
31 | # self.id = get_random_string(5)
32 |
33 | # def __repr__(self):
34 | # return f"df: {self.df_name}\n predicates: {self.predicate}"
35 |
36 |
37 | # basic stub for Vega typing
38 | VegaSpecType = Dict[str, Any]
39 |
40 | ENCODING_COUNT = 8
41 |
42 | class EncodingSpec(object):
43 | # note that this is synced with the vegaGen.ts file
44 | def __init__(self,
45 | mark: Literal["bar", "circle", "line"],
46 | x: str,
47 | x_type: Literal["ordinal", "quantitative", "temporal"],
48 | y: str,
49 | y_type: Literal["ordinal", "quantitative", "temporal"],
50 | selection_type: Literal["none", "multiclick", "brush"],
51 | selection_dimensions: Literal["", "x", "y", "xy"],
52 | sort: Literal["x", "y", "-y", "-x", ""] = ""
53 | ):
54 | """EncodingSpec object used for B2 to generate Vega-Lite specifications
55 |
56 | Arguments:
57 | mark {str} -- "bar" | "circle" | "line"
58 | x {str} -- column for x axis
59 | x_type {str} -- "ordinal" | "quantitative" | "temporal"
60 | y {str} -- column for y axis
61 | y_type {str} -- "ordinal" | "quantitative" | "temporal"
62 | selection_type {str} -- "none", "multiclick", "brush"
63 | selection_dimensions {str} -- "", "x", "y", "xy"
64 | sort optional{str} -- "", "x", "y", "-x", "-y"
65 | """
66 | self.mark = mark
67 | self.x = x
68 | self.x_type = x_type
69 | self.y = y
70 | self.y_type = y_type
71 | self.selection_dimensions = selection_dimensions
72 | self.selection_type = selection_type
73 | self.sort = sort
74 |
75 |
76 | def __eq__(self, other: 'EncodingSpec'):
77 | return self.to_json() == other.to_json()
78 |
79 |
80 | def __ne__(self, other: 'EncodingSpec'):
81 | return not self.__eq__(other)
82 |
83 |
84 | def __repr__(self):
85 | # FIXME: not sure why we have a "!r" here...
86 | # despite reading... https://stackoverflow.com/questions/38418070/what-does-r-do-in-str-and-repr
87 | return f"EncodingSpec({self.mark!r}, {self.x!r}, {self.x_type}, {self.y!r}, {self.y_type}, {self.selection_dimensions}, {self.selection_type!r}, {self.sort})"
88 |
89 | def to_hash(self):
90 | return f'{self.mark}_{self.x}_{self.x_type}_{self.y}_{self.y_type}_{self.selection_dimensions}_{self.selection_type}_{self.sort}'
91 |
92 |
93 | def to_args(self):
94 | return f'{{"mark"="{self.mark}", "x"="{self.x}", "x_type"="{self.x_type}", "y"="{self.y}", "y_type"="{self.y_type}", "selection_dimensions"="{self.selection_dimensions}", "selection_type"="{self.selection_type}", "sort"="{self.sort}"}}'
95 |
96 |
97 | def to_json(self):
98 | return f'{{"mark": "{self.mark}", "x": "{self.x}", "xType": "{self.x_type}", "y": "{self.y}", "yType": "{self.y_type}", "selectionDimensions": "{self.selection_dimensions}", "selectionType": "{self.selection_type}", "sort": "{self.sort}"}}'
99 |
--------------------------------------------------------------------------------
/docs/Tutorial.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Midas Tutorial\n",
8 | "\n",
9 | "Hello! Please follow the tutorial to learn the basics of Midas. Be sure to play around until you are comfortable. You will have about 20 minutes. Should you have any questions, please feel free to ask Yifan, who will be present during the entire session.\n",
10 | "\n",
11 | "## Introduction\n",
12 | "Midas is a Jupyter notebook library/extension that aids data exploration by providing relevant static visualizations. The key of Midas is that **the operations you perform in the interactive visualization space is also reflected in code space**---you will see what this means if you run the code cells below!\n",
13 | "\n",
14 | "## Dataframe Operations\n",
15 | "Midas is a special dataframe with syntax using that of the [data science module](http://data8.org/datascience/) from Data 8. Thw following are common operations that might be useful for querying:\n",
16 | "\n",
17 | "* SELECT: `df.select(['col_name', 'more_col_name'])` --- Note that columns are referenced as strings.\n",
18 | "* WHERER: `df.where('col_name', predicate)` -- the predicates are using lambda functions provided in the [`are`](http://data8.org/datascience/predicates.html) library, such as `are.above(8)` (as opposed to function overloading as seen in pandas, like `df[df['a']>8]`. If you wish to compare two columns, then you can use `.where('col1', preidcate, 'col2')`, such as `marbles.where(\"Price\", are.above, \"Amount\")`.\n",
19 | "* GROUP BY: `df.group('col_name', agg_fun)`, the default aggregation for a `group` is count, but you can also supple the aggregation by using existing aggregation functions such as Python's built in `sum`, `min`, `max` (or any of the `numpy` aggregation methods that work on arrays). The groupby operation is applied to all the columns that are not being grouped by on.\n",
20 | "* Apply general methods: `df.apply(map_fun, new_column_name)` -- for instance, if you want to derive a new column that was the original column plus 1, with the new column called \"incremented\", the function you can call is `df.apply(lambda x: x + 1, 'incremented')`.\n",
21 | "\n",
22 | "The following are useful for data modification:\n",
23 | "* `append_column(label, values)` appends a new column, note that values must be created via `make_array` (so that it's numpy compliant) \n",
24 | "* `append(array_of_new_values)` appends a new row\n",
25 | "\n",
26 | "Note that you can also access the columns as numpy arrays by using `df['col_name']`, which can be handy to use methods like `np.average(df['col_name'])`."
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "metadata": {},
32 | "source": [
33 | "## Initiate Midas\n",
34 | "Import the library and create an instance, `m = Midas()`, and we call the Midas runtime variable. Per a single notebook, you can only have one Midas instsance.\n",
35 | "Then you will see that a dashboard-like area pops up to the right. You will see three areas, one is that of the data (yellow pane), showing the dataframes with acommpanying columns, and the others are the charts."
36 | ]
37 | },
38 | {
39 | "cell_type": "code",
40 | "execution_count": 2,
41 | "metadata": {},
42 | "outputs": [],
43 | "source": [
44 | "from midas import Midas\n",
45 | "m = Midas()\n",
46 | "\n",
47 | "# other utility libraries\n",
48 | "import numpy as np\n",
49 | "from datascience import Table, make_array\n",
50 | "from datascience.predicates import are"
51 | ]
52 | },
53 | {
54 | "cell_type": "markdown",
55 | "metadata": {},
56 | "source": [
57 | "## Load data\n",
58 | "Midas takes in data from a few APIs, such as `from_df`, used below, which loads from pandas dataframe.\n",
59 | "Note that you can also use"
60 | ]
61 | },
62 | {
63 | "cell_type": "code",
64 | "execution_count": 3,
65 | "metadata": {},
66 | "outputs": [],
67 | "source": [
68 | "disaster_df = m.read_table('https://vega.github.io/vega-datasets/data/disasters.csv')"
69 | ]
70 | },
71 | {
72 | "cell_type": "markdown",
73 | "metadata": {},
74 | "source": [
75 | "## Seeing data\n",
76 | "\n",
77 | "Since a lot of basic visualization is highly predicatable, Midas attempts to visualize the basics for you directly. However, sometimes, you may want to change the encoding, which is also very easy to do in Midas---just specify `mark`, `x`, `y`, and if you have three columns, specify the third column for `color` or `size`."
78 | ]
79 | },
80 | {
81 | "cell_type": "code",
82 | "execution_count": 4,
83 | "metadata": {},
84 | "outputs": [],
85 | "source": [
86 | "# 🟡 05:15 PM 🟡\n",
87 | "disaster_df.append_column('Year_bin', disaster_df.apply(lambda x: int(x/20.0) * 20.0, 'Year'))\n",
88 | "Year_distribution = disaster_df.group('Year_bin')"
89 | ]
90 | },
91 | {
92 | "cell_type": "code",
93 | "execution_count": null,
94 | "metadata": {},
95 | "outputs": [],
96 | "source": [
97 | "# 🟡 04:04 PM 🟡\n",
98 | "disaster_df.append_column('Deaths_bin', disaster_df.apply(lambda x: int(x/200000.0) * 200000.0, 'Deaths'))\n",
99 | "Deaths_distribution = disaster_df.group('Deaths_bin')"
100 | ]
101 | },
102 | {
103 | "cell_type": "code",
104 | "execution_count": null,
105 | "metadata": {},
106 | "outputs": [],
107 | "source": [
108 | "# 🟡 03:55 PM 🟡\n",
109 | "disaster_df.append_column('Year_bin', disaster_df.apply(lambda x: int(x/20.0) * 20.0, 'Year'))\n",
110 | "Year_distribution = disaster_df.group('Year_bin')"
111 | ]
112 | },
113 | {
114 | "cell_type": "code",
115 | "execution_count": null,
116 | "metadata": {},
117 | "outputs": [],
118 | "source": [
119 | "Entity_distribution = disaster_df.group('Entity')"
120 | ]
121 | },
122 | {
123 | "cell_type": "markdown",
124 | "metadata": {},
125 | "source": [
126 | "## Getting distribution from clicking on the columns pane\n",
127 | "Go ahead and click on the columns. After you click, two effects take place:\n",
128 | "1. a cell will be created that contains dataframe calls that derives the new filtered values, as well as the visualization calls. You will see that they have color emoji such as 🟠, these are indicators for you to better visually navigate.\n",
129 | "2. a chart is created that visualizes the data created in the pane on the right hand side\n",
130 | "\n",
131 | "If the chart is the wrong encoding, or if the groupign query is inacurate, fell free to modify the code. You can click on the 📊icon to get the current definition to your clipboard. Paste the code to a cell, and the results will be reflected in the chart automatically."
132 | ]
133 | },
134 | {
135 | "cell_type": "code",
136 | "execution_count": null,
137 | "metadata": {},
138 | "outputs": [],
139 | "source": [
140 | "# 🟡 03:37 PM 🟡\n",
141 | "disaster_df.append_column('Year_bin', disaster_df.apply(lambda x: int(x/20.0) * 20.0, 'Year'))\n",
142 | "Year_distribution = disaster_df.group('Year_bin')"
143 | ]
144 | },
145 | {
146 | "cell_type": "code",
147 | "execution_count": null,
148 | "metadata": {},
149 | "outputs": [],
150 | "source": [
151 | "# 🟡 03:37 PM 🟡\n",
152 | "disaster_df.append_column('Deaths_bin', disaster_df.apply(lambda x: int(x/200000.0) * 200000.0, 'Deaths'))\n",
153 | "Deaths_distribution = disaster_df.group('Deaths_bin')"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "metadata": {},
159 | "source": [
160 | "## Accessing code with \"📋\"\n",
161 | "\n",
162 | "If you want to take the code with the selection apploed, click on the 📋icon and then the code will be in your clipboar --- use it however you want!"
163 | ]
164 | },
165 | {
166 | "cell_type": "markdown",
167 | "metadata": {},
168 | "source": [
169 | "## Snapshot of the current state with \"📷\"\n",
170 | "Clicking on 📷 will insert a new cell with the current chart you see."
171 | ]
172 | },
173 | {
174 | "cell_type": "markdown",
175 | "metadata": {},
176 | "source": [
177 | "## Making selections\n",
178 | "All the existing visualizations are equipped with the ability to **select**.\n",
179 | "\n",
180 | "* With scatter plots, you can **brush** select on both the x and y axis.\n",
181 | "* With bar charts, you can either brush to select the x axis items or click.\n",
182 | "* With line charts, you can brush to select a range on the x axis.\n",
183 | "\n",
184 | "When you perform a selection, you will observe two effects\n",
185 | "1. the charts will be filtered with the new data\n",
186 | "2. a cell will be generated with the selections you have made---the newly generated cells will keep on appending to the document based on the previous cell executed, and if you keep on interacting, the old interactions will be commented out and the new selection will be selected."
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "execution_count": null,
192 | "metadata": {},
193 | "outputs": [],
194 | "source": []
195 | },
196 | {
197 | "cell_type": "code",
198 | "execution_count": null,
199 | "metadata": {},
200 | "outputs": [],
201 | "source": [
202 | "# reset selections\n",
203 | "m.make_selections([])"
204 | ]
205 | },
206 | {
207 | "cell_type": "markdown",
208 | "metadata": {},
209 | "source": [
210 | "## Navigating selections\n",
211 | "\n",
212 | "You will see that your selections are shown in the selection pane (blue). You can rename and click on the selections to make the selections again."
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "metadata": {},
218 | "source": [
219 | "## Accessing selections programmatically\n",
220 | "\n",
221 | "Access selection in **predicate** form from the Midas runtime variable, `m` (you can assign it other names if you wish).\n",
222 | "- most recent selection: `m.current_selection`\n",
223 | "- all selections made in the past: `m.selection_history`\n",
224 | "\n",
225 | "Access selection results in **data** form, you have the following options:\n",
226 | "- access specific charts by the `To load data, use .from_file("path/to/your_data.csv"), the columns will show up to the right.
10 |For any dataframes, you can visualize it here with .vis().
12 |14 | All the loaded charts by default are augmented with interactivity by default. 15 | You can select a subset of the data by shift click with a bar chart, or shift-drag to draw the brush, with a scatter plot or line chart. 16 |
17 |19 | By default, your interactions are executed via a "log" in a code cell to the left. You can look at the code to get a sense of what you have interacted with. You can also execute the code by uncommenting the relevant selections. 20 |
21 |23 | To toggle this pane (restoring the traditional notebook view), click on Toggle {SYS_NAME}, from the menu bar on the top. Similarly, you can click Toggle Column Shelf to just hide the pane to the right. 24 |
25 |To resize, you can also drag the left edge of the the main (blue)pane---the resizer will be highlighted with a darker shade of blue when you hover over.
26 |Columns of loaded data will show here. You can click on the x symbol to the right to hide the column from view.
8 |You can click on the column to get the distribution of the data.
10 |selections will be shown here
7 |