16 |
17 |
18 |
19 |
20 |
21 | # Introduction
22 |
23 | This repository contains different [Jupyter Notebooks](https://jupyter.org) to demonstrate the capabilities of [getML](https://www.getml.com) in the realm of machine learning on relational data-sets in various domains. getML and its feature engineering algorithms ([FastProp](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-fastprop), [Multirel](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-multirel), [Relboost](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-relboost), [RelMT](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-relmt)), its [predictors](https://getml.com/latest/user_guide/concepts/predicting#using-getml) (LinearRegression, LogisticRegression, XGBoostClassifier, XGBoostRegressor) and its [hyperparameter optimizer](https://getml.com/latest/user_guide/concepts/hyperopt#hyperparameter-optimization) (RandomSearch, LatinHypercubeSearch, GaussianHyperparameterSearch), are benchmarked against competing tools in similar categories, like [featuretools](https://www.featuretools.com/), [tsfresh](https://tsfresh.com/), and [prophet](https://facebook.github.io/prophet/). While [FastProp](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-fastprop) usually outperforms the competition in terms of runtime and resource requirements, the more sophisticated algorithms ([Multirel](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-multirel), [Relboost](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-relboost), [RelMT](https://getml.com/latest/user_guide/concepts/feature_engineering/#feature-engineering-algorithms-relmt)), which are part of the [Enterprise edition](https://getml.com/latest/enterprise), often lead to even higher accuracy while maintaining low resource requirements. The demonstrations are done on publicly available data-sets, which are standardly used for such comparisons.
24 |
25 | # Table of Contents
26 |
27 | - [Introduction](#introduction)
28 | - [Table of Contents](#table-of-contents)
29 | - [Usage](#usage)
30 | - [Reading Online](#reading-online)
31 | - [Experimenting Locally](#experimenting-locally)
32 | - [Using Docker](#using-docker)
33 | - [On the Machine (Linux/x64 \& arm64)](#on-the-machine-linuxx64--arm64)
34 | - [Notebooks](#notebooks)
35 | - [Overview](#overview)
36 | - [Descriptions](#descriptions)
37 | - [Quick access by grouping by](#quick-access-by-grouping-by)
38 | - [Benchmarks](#benchmarks)
39 | - [FastProp Benchmarks](#fastprop-benchmarks)
40 | - [Further Benchmarks in the Relational Dataset Repository](#further-benchmarks-in-the-relational-dataset-repository)
41 |
42 | # Usage
43 |
44 | The provided notebooks can be checked and used in different ways.
45 |
46 | ## Reading Online
47 |
48 | As github renders the notebooks, they can each be viewed by just opening and scrolling through them. For convenience, the output of each cells execution is included.
49 |
50 | ## Experimenting Locally
51 |
52 | To experiment with the notebooks, such as playing with different pipelines and predictors, it is best to run them on a local machine. Linux users with an x64 architecture can choose from one of the options provided below. Soon, we will offer a simple, container-based solution compatible with all major systems (Windows, Mac) and will also support ARM-based architectures.
53 |
54 | ### Using Docker
55 |
56 | There are a `docker-compose.yml` and a `Dockerfile` for easy usage provided.
57 |
58 | Simply clone this repository and run the docker command to start the `notebooks` service. The image it depends on will be build if it is not already available.
59 |
60 | ```
61 | $ git clone https://github.com/getml/getml-demo.git
62 | $ docker compose up notebooks
63 | ```
64 |
65 | To open Jupyter Lab in the browser, look for the following lines in the output and copy-paste them in your browser:
66 |
67 | ```
68 | Or copy and paste one of these URLs:
69 |
70 | http://localhost:8888/lab?token=
71 | ```
72 |
73 | After the first `getml.engine.launch(...)` is executed and the Engine is started, the corresponding Monitor can be opened in the browser under
74 |
75 | ```
76 | http://localhost:1709/#/token/token
77 | ```
78 |
79 | > [!NOTE]
80 | > Using alternatives to [Docker Desktop](https://www.docker.com/products/docker-desktop) like
81 | > * [Podman](https://podman.io),
82 | > * [Podman Desktop](https://podman-desktop.io) or
83 | > * [Rancher Desktop](https://rancherdesktop.io) with a container engine like dockerd(moby) or containerd(nerdctl)
84 | >
85 | > allows bind-mounting the notebooks in a user-writeable way (this might need to be included: `userns_mode: keep-id`) instead of having to `COPY` them in. In combination with volume-binding `/home/user/.getML/logs` and `/home/user/.getML/projects`, runs and changes can be persisted across containers.
86 |
87 | ### On the Machine (Linux/x64 & arm64)
88 |
89 | Alternatively, getML and the notebooks can be run natively on the local Linux machine by having certain software installed, like Python and some Python libraries, Jupyter-Lab and the getML Engine. The [getML Python library](https://github.com/getml/getml-community) provides an Engine version without [Enterprise features](https://getml.com/latest/enterprise). In order to replicate Enterprise functionalities in the notebooks, you may obtain an [Enterprise trial version](https://getml.com/latest/enterprise/request-trial).
90 |
91 | The following commands will set up a Python environment with necessary Python libraries and the getML Enterprise trial version, and Jupyter-Lab
92 |
93 | ```
94 | $ git clone https://github.com/getml/getml-demo.git
95 | $ cd getml-demo
96 | $ pipx install hatch
97 | $ hatch env create
98 | $ hatch shell
99 | $ pip install -r requirements.txt
100 | $ jupyter-lab
101 | ```
102 |
103 | > [!TIP]
104 | > Install the [Enterprise trial version](https://getml.com/latest/enterprise/request-trial) via the [Install getML on Linux guide](https://getml.com/latest/install/packages/linux#install-getml-on-linux) to try the Enterprise features.
105 |
106 | With the last command, Jupyter-Lab should automatically open in the browser. If not, look for the following lines in the output and copy-paste it in your browser:
107 |
108 | ```
109 | Or copy and paste one of these URLs:
110 |
111 | http://localhost:8888/lab?token=
112 | ```
113 |
114 | After the first `getml.engine.launch(...)` is executed and the Engine is started, the corresponding Monitor can be opened in the browser under
115 |
116 | ```
117 | http://localhost:1709/#/token/token
118 | ```
119 |
120 | # Notebooks
121 |
122 | This repository contains various demonstrational projects to help getting started with relational learning and getML. They cover different aspects of the software, and can serve as documentation or as blueprints for own projects.
123 |
124 | Each project solves a typical data science problem in a specific domain. You
125 | can either choose a project by domain or by the underlying machine learning
126 | problem, e.g. binary classification on a time series or regression using a
127 | relational data scheme involving many tables.
128 |
129 | ## Overview
130 |
131 | | | Task | Data | Size | Domain |
132 | | ------------------------------------------------------------- | -------------- | ------------------------ | ------------------ | -------------- |
133 | | [AdventureWorks: Predicting customer churn][adventureworksnb] | Classification | Relational | 71 tables, 233 MB | Commerce |
134 | | [Air pollution prediction][airpollutionnb] | Regression | Multivariate time series | 1 table, 41k rows | Environment |
135 | | [Disease lethality prediction][atherosclerosisnb] | Classification | Relational | 3 tables, 22 MB | Health |
136 | | [Baseball (Lahman): Predicting salaries][baseballnb] | Regression | Relational | 25 tables, 74 MB | Sports |
137 | | [Expenditure categorization][consumerexpendituresnb] | Classification | Relational | 3 tables, 150 MB | E-commerce |
138 | | [CORA: Categorizing academic studies][coranb] | Classification | Relational | 3 tables, 4.6 MB | Academia |
139 | | [Traffic volume prediction (LA)][dodgersnb] | Regression | Multivariate time series | 1 table, 47k rows | Transportation |
140 | | [Formula 1 (ErgastF1): Predicting the winner][formula1nb] | Classification | Relational | 13 tables, 56 MB | Sports |
141 | | [IMDb: Predicting actors' gender][imdbnb] | Classification | Relational with text | 7 tables, 477.1 MB | Entertainment |
142 | | [Traffic volume prediction (I94)][interstate94nb] | Regression | Multivariate time series | 1 table, 24k rows | Transportation |
143 | | [Financial: Loan default prediction][loansnb] | Classification | Relational | 8 tables, 60 MB | Financial |
144 | | [MovieLens: Predicting users' gender][movielensnb] | Classification | Relational | 7 tables, 20 MB | Entertainment |
145 | | [Occupancy detection][occupancynb] | Classification | Multivariate time series | 1 table, 32k rows | Energy |
146 | | [Order cancellation][onlineretailnb] | Classification | Relational | 1 table, 398k rows | E-commerce |
147 | | [Predicting a force vector from sensor data][robotnb] | Regression | Multivariate time series | 1 table, 15k rows | Robotics |
148 | | [Seznam: Predicting the transaction volume][seznamnb] | Regression | Relational | 4 tables, 147 MB | E-commerce |
149 | | [SFScores: Predicting health check scores][sfscoresnb] | Regression | Relational | 3 tables, 9 MB | Restaurants |
150 | | [Stats: Predicting users' reputation][statsnb] | Regression | Relational | 8 tables, 658 MB | Internet |
151 |
152 | ## Descriptions
153 |
154 |
155 | Adventure Works - Predicting customer churn
156 |
157 | In the notebook, we demonstrate how getML can be used for a customer churn project using a synthetic dataset of a fictional company. We also benchmark getML against featuretools.
158 |
159 | AdventureWorks is a fictional company, that sells bicycles. It is used by Microsoft to showcase how its MS SQL Server can be used to manage business data. Since the dataset resembles a real-world customer database and it is open-source, we use it to showcase, how getML can be used for a classic customer churn project (real customer databases are not easily available for the purposes of showcasing and benchmarking, for reasons of data privacy).
160 |
161 | * Prediction type: Classification model
162 | * Domain: Customer loyalty
163 | * Prediction target: churn
164 | * Population size: 19704
165 |
166 | [> Open Notebook <](adventure_works.ipynb)
167 |
168 |
169 |
170 | Air Pollution - Why feature learning is better than simple propositionalization
171 |
172 | In the notebook we compare getML to featuretools and tsfresh, both of which are open-source libraries for feature engineering using propositionalization approaches. We showcase, that advanced algorithms featured in getML yield significantly better predictions on the dataset.
173 |
174 | The propositionalization methods usually work as follows:
175 |
176 | * Generate a large number of hard-coded features,
177 | * Use feature selection to pick a percentage of these features.
178 |
179 | By contrast, getML contains approaches for feature learning, which adapts machine learning approaches, such as decision trees or gradient boosting, to the problem of extracting features from relational data and time series.
180 |
181 | * Prediction type: Regression model
182 | * Domain: Air pollution
183 | * Prediction target: pm 2.5 concentration
184 | * Source data: Multivariate time series
185 | * Population size: 41757
186 |
187 | [> Open Notebook <](air_pollution.ipynb)
188 |
189 | [> Open FastProp Benchmark Notebook <](fastprop_benchmark/air_pollution_prop.ipynb)
190 |
191 |
192 |
193 | Atherosclerosis - Disease lethality prediction
194 |
195 | With the notebook we give a brief introduction to feature engineering on relational data with many columns. We discuss why feature engineering on such data is particularly challenging and what we can do to overcome these problems.
196 |
197 | Every column, that we have, can either be aggregated or it can be used for our conditions. That means, if we have n columns to aggregate, we can potentially build conditions for n other columns. In other words, the computational complexity is n\*n in the number of columns.
198 |
199 | Note, that this problem occurs regardless of whether you automate feature engineering or you do it by hand. The size of the search space is in the number of columns in either case, unless you can rule something out a-priori.
200 |
201 | An algorithm, that generates specific features, can only use columns for conditions, it is not allowed to aggregate columns – and it doesn't need to do so. That means, the computational complexity is linear instead of quadratic. For data sets with a large number of columns this can make all the difference in the world. For instance, if you have 100 columns, the size of the search space of the second approach is only 1% of the size of the search space of the first one.
202 |
203 | To illustrate the problem of dimensionality in predictive analytics on relational data, we use a longitudinal study of atherosclerosis patients. One of its defining features is, that it contains many columns, which makes it a good candidate to illustrate the problem discussed in the notebook.
204 |
205 | The way, these studies handle the large number of columns in the data set, is to divide the columns into subgroups and then handling each subgroup separately. Even though this is one way to overcome the curse of dimensionality, it is not a very satisfying approach. We would like to be able to handle a large number of columns at once.
206 |
207 | * Prediction type: Binary classification
208 | * Domain: Health
209 | * Prediction target: Mortality within one year
210 | * Source data: 146 columns in 2 tables, 22 MB
211 | * Population size: 28433
212 |
213 | [> Open Notebook <](atherosclerosis.ipynb)
214 |
215 |
216 |
217 | Baseball - Predicting players' salary
218 |
219 | In the notebook, we benchmark several of getML's feature learning algorithms against featuretools using a dataset related to baseball players' salary.
220 |
221 | In the late 1990s, the Oakland Athletics began focusing on the idea of sabermetrics, using statistical methods to identify undervalued baseball players. This was done to compensate for the fact, that the team had a significantly smaller budget than most other teams in its league. Under its general manager Billy Beane, the Oakland Athletics became the first team in over 100 years to win 20 consecutive games in a row, despite still being significantly disadvantaged in terms of its budget. After this remarkable success, the use of sabermetrics quickly became the norm in baseball. These events have been documented in a bestselling book and a movie, both called Moneyball.
222 |
223 | In the notebook we demonstrate, that relational learning can be used for sabermetrics. Specifically, we develop a model to predict players' salary using getML's statistical relational learning algorithms. Such predictions can be used to identify undervalued players.
224 |
225 | * Prediction type: Regression model
226 | * Domain: Sports
227 | * Prediction target: Salaries
228 | * Population size: 23111
229 |
230 | [> Open Notebook <](baseball.ipynb)
231 |
232 |
233 |
234 | Consumer Expenditures - Consumer expenditure categorization
235 |
236 | The notebook demonstrates how powerful a real relational learning algorithm can be. Based on a public-domain dataset on consumer behavior, we use a propostionalization algorithm to predict, whether purchases were made as a gift. We show, that with relational learning, we can get an AUC of over 90%. The generated features would have been impossible to build by hand or by using brute-force approaches.
237 |
238 | There are many subdomains of relational learning, but the most important one is extracting features from relational data: Most business data is relational, meaning that it is spread out over several relational tables. However, most machine learning algorithms require, that the data be presented in the form of a single flat table. So we need to extract features from our relational data. Some people also call this data wrangling.
239 |
240 | The Consumer Expenditure Data Set is a public domain data set provided by the American Bureau of Labor Statistics. It includes the diary entries, where American consumers are asked to keep diaries of the products they have purchased each month. These consumer goods are categorized using a six-digit classification system: the UCC. This system is hierarchical, meaning that every digit represents an increasingly granular category. The diaries also contain a flag that indicates whether the product was purchased as a gift. The challenge is to predict that flag using other information in the diary entries.
241 |
242 | * Prediction type: Classification model
243 | * Domain: Retail
244 | * Prediction target: If a purchase is a gift
245 | * Source data: Relational data set, 4 tables
246 | * Population size: 2.020.634
247 |
248 | [> Open Notebook <](consumer_expenditures.ipynb)
249 |
250 |
251 |
252 | CORA - Categorizing academic publications
253 |
254 | In the notebook, we compare getML against extant approaches in the relational learning literature on the CORA data set, which is often used for benchmarking. We demonstrate, how getML performs against the state of the art in the relational learning literature on this data set. Beyond the benchmarking aspects, this notebooks showcases getML's excellent capabilities in dealing with categorical data.
255 |
256 | CORA is a well-known benchmarking dataset in the academic literature on relational learning. The dataset contains 2708 scientific publications on machine learning. The papers are divided into 7 categories. The challenge is to predict the category of a paper based on the papers it cites, the papers it is cited by and keywords contained in the paper.
257 |
258 | * Prediction type: Classification model
259 | * Domain: Academia
260 | * Prediction target: The category of a paper
261 | * Population size: 2708
262 |
263 | [> Open Notebook <](cora.ipynb)
264 |
265 |
266 |
267 | Dodgers - Traffic volume prediction on LA's 101 North freeway
268 |
269 | In the notebook we demonstrate a time series application of getML. We benchmark our results against Facebook's Prophet and tsfresh.
270 |
271 | The data set features some particularly interesting characteristics common for time series, which classical models may struggle to deal with. Such characteristics are:
272 |
273 | * High frequency (every five minutes)
274 | * Dependence on irregular events (holidays, Dodgers games)
275 | * Strong and overlapping cycles (daily, weekly)
276 | * Anomalies
277 | * Multiple seasonalities
278 |
279 | To quote the maintainers of the data set:
280 |
281 | > This loop sensor data was collected for the Glendale on ramp for the 101 North freeway in Los Angeles. It is close enough to the stadium to see unusual traffic after a Dodgers game, but not so close and heavily used by game traffic so that the signal for the extra traffic is overly obvious.
282 |
283 | * Prediction type: Regression model
284 | * Domain: Transportation
285 | * Prediction target: traffic volume
286 | * Source data: Univariate time series
287 | * Population size: 47497
288 |
289 | [> Open Notebook <](dodgers.ipynb)
290 |
291 | [> Open FastProp Benchmark Notebook <](fastprop_benchmark/dodgers_prop.ipynb)
292 |
293 |
294 |
295 | Formula 1 - Predicting the winner of a race
296 |
297 | In the notebook we benchmark getML against featuretools to predict the winner of a Formula 1 race.
298 |
299 | We develop a prediction model for Formula 1 races, that allows us to predict the winner of a race before the race has started.
300 |
301 | We use a dataset of all Formula 1 races from 1950 to 2017. The dataset includes information such as the time taken in each lap, the time taken for pit stops, the performance in the qualifying rounds etc.
302 |
303 | * Prediction type: Classification model
304 | * Domain: Sports
305 | * Prediction target: Win
306 | * Population size: 31578
307 |
308 | [> Open Notebook <](formula1.ipynb)
309 |
310 |
311 |
312 | IMDB - Predicting actors' gender
313 |
314 | In the notebook, we demonstrate how getML can be applied to text fields. In relational databases, text fields are less structured and less standardized than categorical data, making it more difficult to extract useful information from them. When using a relational learning tool such as getML, we can easily generate simple features from text fields and leverage the information contained therein.
315 |
316 | As an example data set, we use the Internet Movie Database, which has been used by previous studies in the relational learning literature. This allows us to benchmark our approach to state-of-the-art algorithms in the relational learning literature.
317 |
318 | The data set contains about 800,000 actors. The goal is to predict the gender of said actors based on other information we have about them, such as the movies they have participated in and the roles they have played in these movies.
319 |
320 | * Prediction type: Classification model
321 | * Domain: Entertainment
322 | * Prediction target: The gender of an actor
323 | * Population size: 817718
324 |
325 | [> Open Notebook <](imdb.ipynb)
326 |
327 |
328 |
329 | Interstate 94 - Hourly traffic volume prediction on Interstate 94
330 |
331 | In the notebook, we demonstrate a time series application of getML. We predict the hourly traffic volume on I-94 westbound from Minneapolis-St Paul. We benchmark our results against Facebook's Prophet.
332 |
333 | The dataset features some particularly interesting characteristics common for time series, which classical models may struggle to deal with appropriately. Such characteristics are:
334 |
335 | * High frequency (hourly)
336 | * Dependence on irregular events (holidays)
337 | * Strong and overlapping cycles (daily, weekly)
338 | * Anomalies
339 | * Multiple seasonalities
340 |
341 |
342 |
343 | * Prediction type: Regression model
344 | * Domain: Transportation
345 | * Prediction target: Hourly traffic volume
346 | * Source data: Multivariate time series, 5 components
347 | * Population size: 24096
348 |
349 | [> Open Notebook <](interstate94.ipynb)
350 |
351 | [> Open FastProp Benchmark Notebook <](fastprop_benchmark/interstate94_prop.ipynb)
352 |
353 |
354 |
355 | Loans - Predicting the loan default risk of Czech bank customers
356 |
357 | The notebook demonstrates the application of our relational learning algorithm to predict if a customer of a bank will default on his loan. We train the predictor on customer metadata, transaction history, as well as other successful and unsuccessful loans.
358 |
359 | The notebook features a textbook example of predictive analytics applied to the financial sector. A loan is the lending of money to companies or individuals. Banks grant loans in exchange for the promise of repayment. Loan default is defined as the failure to meet this legal obligation, for example, when a home buyer fails to make a mortgage payment. A bank needs to estimate the risk it carries when granting loans to potentially non-performing customers.
360 |
361 | * Prediction type: Binary classification
362 | * Domain: Finance
363 | * Prediction target: Loan default
364 | * Source data: 8 tables, 78.8 MB
365 | * Population size: 682
366 |
367 | [> Open Notebook <](loans.ipynb)
368 |
369 |
370 |
371 | Movie Lens - Predicting a user's gender based on the movies they have watched
372 |
373 | In the notebook we apply getML to a dataset, that is often used for benchmarking in the relational learning literature: The MovieLens dataset.
374 |
375 | The MovieLens dataset is often used in the relational learning literature as a benchmark for newly developed algorithms. Following the tradition, we benchmark getML's own algorithms on this dataset as well. The task is to predict a user's gender based on the movies they have watched.
376 |
377 | * Prediction type: Classification model
378 | * Domain: Entertainment
379 | * Prediction target: The gender of a user
380 | * Population size: 6039
381 |
382 | [> Open Notebook <](movie_lens.ipynb)
383 |
384 |
385 |
386 | Occupancy - Occupancy detection
387 |
388 | The notebook demonstrates, how to apply getML to multivariate time series and how to use getML's high-level interface for hyperparameter tuning.
389 |
390 | Our use case is a public domain data set for predicting room occupancy from sensor data. Note, that this is not only a neat use case for machine learning algorithms, but a real-world application with tangible consequences: If room occupancy is known with sufficient certainty, it can be applied to the control systems of a building. Such as system can reduce the energy consumption by up to 50 %.
391 |
392 | Instead of creating features by merging and aggregating peripheral tables in a relational data model, for a (multivariate) time-series, we perform the same operations on the population table itself. This results in features like these:
393 |
394 | * Aggregations over time, such as the average value of some column for the last 3 days.
395 | * Seasonal effects, such as today is a Wednesday, so let's get the average value for the last four Wednesdays.
396 | * Lag variables, such as get the value of some column from two hours ago.
397 |
398 | Using getML's algorithms for relational learning, we extract all of these features automatically. Having created a flat table of such features, we then apply state-of-the-art machine learning algorithms, like xgboost.
399 |
400 | The present analysis is based on a public domain time series dataset. It is available in the UC Irvine Machine Learning Repository. The challenge is straightforward: We want to predict whether an office room is occupied at a given moment in time using sensor data. The data is measured about once a minute. Ground-truth occupancy was obtained from time-stamped pictures. The available columns are
401 |
402 | * Date, year-month-day hour:minute:second
403 | * Temperature, in Celsius
404 | * Relative Humidity, %
405 | * Light, in Lux
406 | * CO2, in ppm
407 | * Humidity Ratio, Derived quantity from temperature and relative humidity, in kgwater-vapor/kg-air
408 | * Occupancy, 0 or 1, 0 for not occupied, 1 for occupied status
409 |
410 |
411 |
412 | * Prediction type: Binary classification
413 | * Domain: Energy
414 | * Prediction target: Room occupancy
415 | * Source data: 1 table, 32k rows
416 | * Population size: 32k
417 |
418 | [> Open Notebook <](occupancy.ipynb)
419 |
420 | [> Open FastProp Benchmark Notebook <](fastprop_benchmark/occupancy_prop.ipynb)
421 |
422 |
423 |
424 | Online Retail - Predicting order cancellations
425 |
426 | The notebook demonstrate, how getML can be applied in an e-commerce context. We also show, that we can significantly improve our results by using getML's built-in hyperparameter tuning routines.
427 |
428 | The data set contains about 400,000 orders from a British online retailer. Each order consists of a product that has been ordered and a corresponding quantity. Several orders can be summarized onto a single invoice. The goal is to predict whether an order will be cancelled.
429 |
430 | Because the company mainly sells to other businesses, the cancellation rate is relatively low, namely 1.83%.
431 |
432 | * Prediction type: Classification model
433 | * Domain: E-commerce
434 | * Prediction target: Whether an order will be cancelled
435 | * Population size: 397925
436 |
437 | [> Open Notebook <](online_retail.ipynb)
438 |
439 |
440 |
441 | Robot - Feature engineering on sensor data - how to overcome feature explosion
442 |
443 | The purpose of this notebook is to illustrate, how we can overcome the feature explosion problem based on an example dataset involving sensor data.
444 |
445 | Every column that we have, can either be aggregated or it can be used for our conditions. That means, if we have n columns to aggregate, we can potentially build conditions for n other columns. In other words, the computational complexity is n\*n in the number of columns. Note, that this problem occurs regardless of whether you automate feature engineering or you do it by hand. The size of the search space is n\*n in the number of columns in either case, unless you can rule something out a-priori. This problem is known as feature explosion.
446 |
447 | An algorithm, that generates specific different features can only use columns for conditions, it is not allowed to aggregate columns – and it doesn't need to do so. That means, the computational complexity is linear instead of quadratic. For data sets with a large number of columns, this can make all the difference in the world. For instance, if you have 100 columns the size of the search space of the second approach is only 1% of the size of the search space of the first one.
448 |
449 | getML features an algorithm called Relboost, which generates features according to this principle and is therefore very suitable for data sets with many columns.
450 |
451 | To illustrate the problem, we use a data set related to robotics. When robots interact with humans, the most important thing is, that they don't hurt people. In order to prevent such accidents, the force vector on the robot's arm is measured. However, measuring the force vector is expensive. Therefore, we want consider an alternative approach, where we would like to predict the force vector based on other sensor data that are less costly to measure. To do so, we use machine learning. However, the data set contains measurements from almost 100 different sensors and we do not know which and how many sensors are relevant for predicting the force vector.
452 |
453 | * Prediction type: Regression
454 | * Domain: Robotics
455 | * Prediction target: The force vector on the robot's arm
456 | * Population size: 15001
457 |
458 | [> Open Notebook <](robot.ipynb)
459 |
460 | [> Open FastProp Benchmark Notebook <](fastprop_benchmark/robot_prop.ipynb)
461 |
462 |
463 |
464 | Seznam - Predicting the transaction volume
465 |
466 | Seznam is a Czech company with a scope similar to Google. The purpose of the notebook is to analyze data from Seznam's wallet, predicting the transaction volume.
467 |
468 | * Prediction type: Regression model
469 | * Domain: E-commerce
470 | * Prediction target: Transaction volume
471 | * Population size: 1,462,078
472 |
473 | [> Open Notebook <](seznam.ipynb)
474 |
475 |
476 |
477 | SFScores - Predicting the Results of Health Inspections of Restaurants
478 |
479 | In the notebook, we benchmark several of getML's feature learning algorithms against featuretools using the San Francisco Dept. of Public Health's database of eateries in San Francisco. These eateries are regularly inspected. The inspections often result in a score. The challenge is to predict the score resulting from an inspection.
480 |
481 | * Prediction type: Regression model
482 | * Domain: Health
483 | * Prediction target: Sales
484 | * Population size: 12887
485 |
486 | [> Open Notebook <](sfscores.ipynb)
487 |
488 |
489 |
490 | Stats - Predicting Users' Reputations
491 |
492 | In the notebook we use relational learning techniques to predict users' reputation on StackExchange. StatsExchange is a website similar to StackOverflow, but based on statistics and machine learning. Much like StackOverflow, it has a complicated system of calculating users' reputation.
493 |
494 | * Prediction type: Regression model
495 | * Domain: Internet
496 | * Prediction target: Reputation
497 | * Population size: 41793
498 |
499 | [> Open Notebook <](stats.ipynb)
500 |
501 |
502 | ## Quick access by grouping by
503 |
504 |
505 | Task
506 |
507 | -
508 | Classification
509 |
510 | * [Adventure Works][adventureworksnb]
511 | * [Atherosclerosis][atherosclerosisnb]
512 | * [Consumer Expenditures][consumerexpendituresnb]
513 | * [CORA][coranb]
514 | * [Formula 1][formula1nb]
515 | * [IMDB][imdbnb]
516 | * [Loans][loansnb]
517 | * [MovieLens][movielensnb]
518 | * [Occupancy][occupancynb]
519 | * [Online Retail][onlineretailnb]
520 |
521 |
522 | -
523 | Regression
524 |
525 | * [Air Pollution][airpollutionnb]
526 | * [Baseball][baseballnb]
527 | * [Dodgers][dodgersnb]
528 | * [Interstate 94][interstate94nb]
529 | * [Robot][robotnb]
530 | * [Seznam][seznamnb]
531 | * [SFScores][sfscoresnb]
532 | * [Stats][statsnb]
533 |
534 |
535 |
536 |
537 | Data
538 |
539 | -
540 | Relational
541 |
542 | * [Adventure Works][adventureworksnb]
543 | * [Atherosclerosis][atherosclerosisnb]
544 | * [Baseball][baseballnb]
545 | * [Consumer Expenditures][consumerexpendituresnb]
546 | * [CORA][coranb]
547 | * [Formula 1][formula1nb]
548 | * [Loans][loansnb]
549 | * [MovieLens][movielensnb]
550 | * [Online Retail][onlineretailnb]
551 | * [Seznam][seznamnb]
552 | * [SFScores][sfscoresnb]
553 | * [Stats][statsnb]
554 |
555 | -
556 | Relational With Text
557 |
558 | * [IMDB][imdbnb]
559 |
560 | -
561 | Multivariate Time Series
562 |
563 | * [Air Pollution][airpollutionnb]
564 | * [Dodgers][dodgersnb]
565 | * [Interstate 94][interstate94nb]
566 | * [Occupancy][occupancynb]
567 | * [Robot][robotnb]
568 |
569 |
570 |
571 |
572 | Domain
573 |
574 | -
575 | Academia
576 |
577 | * [CORA][coranb]
578 |
579 | -
580 | Commerce
581 |
582 | * [Adventure Works][adventureworksnb]
583 |
584 | -
585 | E-Commerce
586 |
587 | * [Consumer Expenditures][consumerexpendituresnb]
588 | * [Online Retail][onlineretailnb]
589 | * [Seznam][seznamnb]
590 |
591 | -
592 | Energy
593 |
594 | * [Occupancy][occupancynb]
595 |
596 | -
597 | Entertainment
598 |
599 | * [IMDB][imdbnb]
600 | * [MovieLens][movielensnb]
601 |
602 | -
603 | Environment
604 |
605 | * [Air Pollution][airpollutionnb]
606 |
607 | -
608 | Financial
609 |
610 | * [Loans][loansnb]
611 |
612 | -
613 | Health
614 |
615 | * [Atherosclerosis][atherosclerosisnb]
616 |
617 | -
618 | Internet
619 |
620 | * [Stats][statsnb]
621 |
622 | -
623 | Restaurants
624 |
625 | * [SFScores][sfscoresnb]
626 |
627 | -
628 | Robotics
629 |
630 | * [Robot][robotnb]
631 |
632 | -
633 | Sports
634 |
635 | * [Baseball][baseballnb]
636 | * [Formula 1][formula1nb]
637 |
638 | -
639 | Transportation
640 |
641 | * [Dodgers][dodgersnb]
642 | * [Interstate 94][interstate94nb]
643 |
644 |
645 |
646 |
647 | ## Benchmarks
648 |
649 | The following notebooks specifically show getML's performance compared to other approaches:
650 |
651 | > [!IMPORTANT]
652 | > The results are hardware, software and version-dependent and therefore may be different from your own experience.
653 | > However, usually getML's _FastProp_ is significantly faster than _featuretools_ and _tsfresh_ while consuming considerably less memory.
654 | > If this is not the case for you, or you see flaws or room for improvements, then please let us know!
655 |
656 | | | Benchmarks | Results | getML | other |
657 | | ------------------------------------------------------------- | ------------------------------------------------ | --------- | ------ | ------------------ |
658 | | [AdventureWorks: Predicting customer churn][adventureworksnb] | featuretools | AUC | 97.8% | featuretools 96.8% |
659 | | [Air pollution prediction][airpollutionnb] | featuretools, tsfresh | R-squared | 61.0% | next best 53.7% |
660 | | [Baseball (Lahman): Predicting salaries][baseballnb] | featuretools | R-squared | 83.7% | featuretools 78.0% |
661 | | [CORA: Categorizing academic studies][coranb] | Academic literature: RelF, LBP, EPRN, PRN, ACORA | Accuracy | 89.9% | next best 85.7% |
662 | | [Traffic volume prediction (LA)][dodgersnb] | Prophet (fbprophet), tsfresh | R-squared | 76% | next best 67% |
663 | | [Formula 1 (ErgastF1): Predicting the winner][formula1nb] | featuretools | AUC | 92.6% | featuretools 92.0% |
664 | | [IMDb: Predicting actors' gender][imdbnb] | Academic literature: RDN, Wordification, RPT | AUC | 91.34% | next best 86% |
665 | | [Traffic volume prediction (I94)][interstate94nb] | Prophet (fbprophet) | R-squared | 98.1% | prophet 83.3% |
666 | | [MovieLens: Predicting users' gender][movielensnb] | Academic literature: PRM, MBN | Accuracy | 81.6% | next best 69% |
667 | | [Occupancy detection][occupancynb] | Academic literature: Neural networks | AUC | 99.8% | next best 99.6% |
668 | | [Seznam: Predicting the transaction volume][seznamnb] | featuretools | R-squared | 78.2% | featuretools 63.2% |
669 | | [SFScores: Predicting health check scores][sfscoresnb] | featuretools | R-squared | 29.1% | featuretools 26.5% |
670 | | [Stats: Predicting users' reputation][statsnb] | featuretools | R-squared | 98.1% | featuretools 96.6% |
671 |
672 | ### FastProp Benchmarks
673 |
674 | The following notebooks specifically compare different implementations of propositionalization algorithms against getML's _FastProp_ (short for fast propositionalization).
675 |
676 |
"
550 | ],
551 | "text/plain": [
552 | "population\n",
553 | " subset name rows type\n",
554 | "0 test population 500 View\n",
555 | "1 train population 1708 View\n",
556 | "2 validation population 500 View\n",
557 | "\n",
558 | "peripheral\n",
559 | " alias name rows type \n",
560 | "0 cites cites 5429 DataFrame\n",
561 | "1 content content 49216 DataFrame\n",
562 | "2 paper population 2708 DataFrame"
563 | ]
564 | },
565 | "execution_count": 12,
566 | "metadata": {},
567 | "output_type": "execute_result"
568 | }
569 | ],
570 | "source": [
571 | "container = getml.data.Container(population=data_full, split=split)\n",
572 | "container.add(cites=cites, content=content, paper=paper)\n",
573 | "container.freeze()\n",
574 | "container"
575 | ]
576 | },
577 | {
578 | "cell_type": "markdown",
579 | "metadata": {},
580 | "source": [
581 | "### 2. Predictive modeling\n",
582 | "\n",
583 | "We loaded the data and defined the roles and units. Next, we create a getML pipeline for relational learning."
584 | ]
585 | },
586 | {
587 | "cell_type": "markdown",
588 | "metadata": {},
589 | "source": [
590 | "#### 2.1 Define relational model\n",
591 | "\n",
592 | "To get started with relational learning, we need to specify the data model. Even though the data set itself is quite simple with only three tables and six columns in total, the resulting data model is actually quite complicated.\n",
593 | "\n",
594 | "That is because the class label can be predicting using three different pieces of information:\n",
595 | "\n",
596 | "- The keywords used by the paper\n",
597 | "- The keywords used by papers it cites and by papers that cite the paper\n",
598 | "- The class label of papers it cites and by papers that cite the paper\n",
599 | "\n",
600 | "The main challenge here is that `cites` is used twice, once to connect the _cited_ papers and then to connect the _citing_ papers. To resolve this, we need two placeholders on `cites`."
601 | ]
602 | },
603 | {
604 | "cell_type": "code",
605 | "execution_count": 13,
606 | "metadata": {},
607 | "outputs": [
608 | {
609 | "data": {
610 | "text/html": [
611 | "
3 | 
4 | 
678 | 
