├── README.md
├── data
├── amazon20k.parquet
├── diamonds.csv
├── pageviews-20170120-180000.gz
└── zips.json
└── notebooks
├── Intro-DF-DS-SQL.json
├── RDD-Mem-Execution.json
├── SQL-DF-DS-Execution.json
├── Spark Machine Learning.json
└── Streaming.json
/README.md:
--------------------------------------------------------------------------------
1 | # Spark with Zeppelin
2 |
3 | ### Materials and setup instructions
4 |
5 | 1. Download Apache Zeppelin version 0.7 with "all" bundled interpreters. Although the following link looks like a direct link to the package, it actually opens the Apache download page, where you can choose a mirror: http://www.apache.org/dyn/closer.cgi/zeppelin/zeppelin-0.7.3/zeppelin-0.7.3-bin-all.tgz
6 | 2. Install ... which should be as simple as unzipping the archive you just downloaded. This version of Zeppelin includes an embedded install of Apache Spark 2.1, which is tested to work with the class notebooks.
7 | 3. Test it out!
8 | * `cd` into the folder where you unpacked Zeppelin
9 | * If you run `ls` you should see the `bin` folder
10 | * on ~~MacOS/~~ Linux, run `bin/zeppelin-daemon.sh start`
11 | * __UPDATE For MacOS USERS__:
12 | * There is a known bug that makes Zeppelin run extremely(!) slowly on MacOS (https://issues.apache.org/jira/browse/ZEPPELIN-2948) so, while it does work, I strongly recommend using Linux instead
13 | * __UPDATE FOR WINDOWS USERS__:
14 | * In theory, Zeppelin should run on Windows with minor changes (e.g., `zeppelin.cmd`), but your mileage may vary
15 | * Zeppelin claims support on Win 7 SP 1, but I have not tried that
16 | * I __did__ try it on Windows 10 (more or less latest Microsoft build) and it did *not* work out of the box
17 | * Of course, it's quite possible that with some monkeying around it can be made to work on Win 10
18 | * __HOWEVER__ my recommendation would be to install a Linux VM and run it there: free, easy, definitely works, and your "real" Spark work will almost certainly be on Linux anyway
19 | * Easiest path: Grab free VMWare or VirtualBox, install ubuntu-16.04.2-desktop-amd64, add a JRE with `sudo apt install openjdk-8-jre-headless` and then just unzip Zeppelin and you're all set.
20 | * Make sure you give this VM a good chunk of resources -- I'd recommend 8 GB RAM and 2 CPU cores if you have them
21 | * Point your browser to `localhost:8080` and you should see Zeppelin running!
22 | * occasionally it seems to require an extra few seconds, refreshes, or even clearing browser cache, moreso than other webapps, not sure why, but don't sweat it
23 | * Click "create new note" to start a new notebook
24 | * Give it a name (like "Test"); the default interpreter should say "Spark" and you can leave that as is, and click "Create Note"
25 | * In the first notebook cell, type in `1 + 1` and hit shift + enter
26 | * After a few seconds, you should see `res0: Int = 2` pop out
27 | * Just to make sure Spark can actually run a job, in the next cell, enter `sc.parallelize(1 to 100).sum` and hit shift + enter
28 | * You should see `res1: Double = 5050.0` pop out
29 | * Congrats, your Zeppelin + embedded Spark 2.1 is installed!
30 | 4. If you haven't yet cloned this github repo on your machine, go ahead and do that. If you really don't want to use git, you can just "Download as Zip" this whole repo and use that, and you'll be fine.
31 | 5. Move or copy the "data" folder from the repo into your Zeppelin folder. It should be at the same "level" as `bin`, `conf`, `interpreter`, etc. Why? This way, if you `cd` to the Zeppelin folder and start it with `bin/zeppelin-daemon.sh start`, inside the notebook environment, your data will be under `data/`.
32 | * This is for simplicity; of course you are welcome to `%sh cd ...` from inside Zeppelin, or hack the paths to the datasets, or anything else you wish -- I'm just trying to make it as simple and consistent for everyone as possible.
33 | * You can check if everything is in the right place by punching in `%sh ls data` from inside a notebook cell, and you should see the list of data files, with `amazon20k.parquet`, `diamonds.csv`, etc.
34 | 6. Make note (for yourself) of where you put the `notebooks` folder from this repo. You don't need to do anything with them yet, but in class, we will load notebooks into Zeppelin, so it will be handy to know where to find the files.
35 | * If you want, you can try it from the main Zeppelin page by clicking "Import Note" then "Choose a JSON here" and then using the file picker to find the notebook.
36 | * When long notebooks like these import, they will look weird at first -- like a bunch of blank cells. Just wait about 30 seconds for Zeppelin to catch up and render them, and you should be all set.
37 |
38 |
39 |
--------------------------------------------------------------------------------
/data/amazon20k.parquet:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/adbreind/spark-zeppelin-17-1/c31a80b29d4b7dc5dad8ef3060d78adf9d6a2b4f/data/amazon20k.parquet
--------------------------------------------------------------------------------
/data/pageviews-20170120-180000.gz:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/adbreind/spark-zeppelin-17-1/c31a80b29d4b7dc5dad8ef3060d78adf9d6a2b4f/data/pageviews-20170120-180000.gz
--------------------------------------------------------------------------------
/notebooks/RDD-Mem-Execution.json:
--------------------------------------------------------------------------------
1 | {"paragraphs":[{"user":"anonymous","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487456224733_-1880050817","id":"20170218-141704_979458167","dateCreated":"2017-02-18T14:17:04-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:12434","text":"%md # RDDs, Spark Memory, and Execution\n\n©2016, 2017 by Adam Breindel. All Rights Reserved.","dateUpdated":"2017-02-18T14:17:24-0800","dateFinished":"2017-02-18T14:17:24-0800","dateStarted":"2017-02-18T14:17:24-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
RDDs, Spark Memory, and Execution
\n
©2016, 2017 by Adam Breindel. All Rights Reserved.
\n
\n
Resilient Distributed Datasets (RDDs)
\n
\n
\n
Why a “new” computation approach?
\n
Lots of existing cluster computing models. Goal is to exploit availability of large amounts of RAM.
\n
Challenge: treating a cluster a large pool of memory and putting objects in it is not a new idea, but it is prone to many difficulties which have expensive mitigations.
\n
\n
Distributed computation = concurrent computation + guaranteed failures
\n
We can simplify distributed computation by addressing both of these elements:
\n
\n - Functional programming model for concurrency
\n - Metadata for reliability (but not for concurrency)
\n
\n
\n
Most of the big data world is (was) on Java-based tools (e.g., Hadoop); Scala provides a functional language with direct Java interop.
\n
Let’s look at Scala collections – can we make something that looks/behaves like Scala collections but transparently operates at cluster scale?
\n
it's also not a new or peculiar to Scala ... Java 8, C#, JavaScript, and other languages have had these patterns for a very long time)","user":"anonymous","dateUpdated":"2017-02-18T14:19:07-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487456310255_496682701","id":"20170218-141830_1787067081","dateCreated":"2017-02-18T14:18:30-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:12871","dateFinished":"2017-02-18T14:19:07-0800","dateStarted":"2017-02-18T14:19:07-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Here’s some Scala (note this is not Spark code, just plain Scala …
it’s also not a new or peculiar to Scala … Java 8, C#, JavaScript, and other languages have had these patterns for a very long time)
\n
(e.g., position in a collection, or partial aggregation totals), it's easy to provide a declarative parallel API:","dateUpdated":"2017-02-18T14:20:31-0800","dateFinished":"2017-02-18T14:20:31-0800","dateStarted":"2017-02-18T14:20:31-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Since this approach avoids side effects and does not require the user (programmer) to manage any state
(e.g., position in a collection, or partial aggregation totals), it’s easy to provide a declarative parallel API:
\n
some other mechanism for the actual parallelism. Scala uses regular JVM threads. This is similar to the Java 8 Streams API using the default Fork-Join threadpool.","dateUpdated":"2017-02-18T14:21:12-0800","dateFinished":"2017-02-18T14:21:12-0800","dateStarted":"2017-02-18T14:21:12-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Look, ma! No [visible] threads! … Of course underneath there are threads or processes or coroutines or
some other mechanism for the actual parallelism. Scala uses regular JVM threads. This is similar to the Java 8 Streams API using the default Fork-Join threadpool.
\n
\n
Now let’s take a look at how the Spark RDD API exposes similar functionality with similar syntax:
\n
(e.g., via RMI)? How can we fix things if (when!) we lose a node and all the data that was in RAM?\n\nWhat if we had a minimal metadata object that describes, deterministically, which how the dataset is distributed,
and how one step of the computation is derived from the parent.","dateUpdated":"2017-02-18T14:23:13-0800","dateFinished":"2017-02-18T14:23:13-0800","dateStarted":"2017-02-18T14:23:13-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Those functional patterns help with concurrency … but what about resiliency?
\n
I.e., how is this different from putting Scala collections in raw memory somewhere
(e.g., via RMI)? How can we fix things if (when!) we lose a node and all the data that was in RAM?
\n
What if we had a minimal metadata object that describes, deterministically, which how the dataset is distributed,
and how one step of the computation is derived from the parent.
\n
![](\"http://i.imgur.com/fQG6K3t.png\")
","user":"anonymous","dateUpdated":"2017-02-18T14:23:40-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487456574643_-1627735972","id":"20170218-142254_642582143","dateCreated":"2017-02-18T14:22:54-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:14053","dateFinished":"2017-02-18T14:23:40-0800","dateStarted":"2017-02-18T14:23:40-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Computation as a DAG
\n
\n
\n
Each RDD is a small piece of metadata describing a step of the computation
\n
\n - Parent(s)
\n - Partitions
\n - Compute
\n - (Locality Preference)
\n
\n
\n
Let’s grab a dataset and do some simple counting. The goal is to understand how Spark executes our jobs!
\n
\n
What is the parallelism of that job? Why?
\n
\n
Caching RDDs
\n
\n - Options for serialization, local disk cache, replication, etc.
\n - Caching, like computation, is always at the granularity of partitions!
\n - Why? That’s the level of detail about which we have metadata for recovery!
\n
\n
Let’s start with basic memory-only caching of raw Java/Scala objects:
\n
\n
RDD caching defaults to MEMORY_ONLY
\n
i.e., myRDD.cache is the same as myRDD.persist(MEMORY_ONLY)
\n
\n
Note that
1. DataFrame/Dataset default .cache as persist(MEMORY_AND_DISK) because the compressed encoded columnar cache is expensive to rebuild
2. Some streaming sources default to stronger caching (e.g., MEMORY_AND_DISK_SER_2)
\n
rdd.unpersist() or dataset.unpersist() – immediately remove from cache
\n
![](\"http://i.imgur.com/Di2SDiG.png\"/)
\n\nhttps://0x0fff.com/spark-memory-management/\n\nNote: these percentages are 1.6; they are slightly different in 2.x but the overall behavior is the same\n\nCache Eviction is due to\n\n* Least-recently-used (LRU) when space is needed for caching other data (per programmer requests)\n* or Spark needing memory for shuffle/execution","user":"anonymous","dateUpdated":"2017-02-18T14:43:38-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487457706060_69434633","id":"20170218-144146_1016781747","dateCreated":"2017-02-18T14:41:46-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:16203","dateFinished":"2017-02-18T14:43:38-0800","dateStarted":"2017-02-18T14:43:38-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Where does this memory pool fit into the executors (JVMs)? How does Spark allocate memory?
\n
\n
https://0x0fff.com/spark-memory-management/
\n
Note: these percentages are 1.6; they are slightly different in 2.x but the overall behavior is the same
\n
Cache Eviction is due to
\n
\n - Least-recently-used (LRU) when space is needed for caching other data (per programmer requests)
\n - or Spark needing memory for shuffle/execution
\n
\n
\n
RDD Job Execution In-Depth
\n
\n - Application
\n - Job
\n - Stage
\n - Task
\n
\n
![](\"http://i.imgur.com/Uqo6FJM.png\")
","user":"anonymous","dateUpdated":"2017-02-18T15:07:25-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487459154080_-829890468","id":"20170218-150554_1502543511","dateCreated":"2017-02-18T15:05:54-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:16557","dateFinished":"2017-02-18T15:07:25-0800","dateStarted":"2017-02-18T15:07:25-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
\n
![](\"http://i.imgur.com/2UuGJve.jpg\")
","dateUpdated":"2017-02-18T15:19:13-0800","dateFinished":"2017-02-18T15:19:13-0800","dateStarted":"2017-02-18T15:19:13-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
\n
\n
And, finally, to add up the word occurrences, we’ll reduceByKey
\n
\n
What does it look like if we increase parallelism in that last stage?
\n
\n
Let’s see how cached data appears in the execution UI
\n
\n
Take a look in the UI and look at the green dot in the DAG visualization, representing the cached RDD wordPairsWithDuplicates
\n
\n
What if we’ve performed a shuffle with a data set, and we need the output of that shuffle again?
\n
Spark will re-use the existing shuffle output files – this results in skipped stages.
\n
\n
If we need to perform a full sort, this involved a range partitioning, and so will generate an additional Spark job.
\n
\n
How Does the Shuffle Work?
\n
\n - Hash shuffle
\n - Sort shuffle
\n - “Tungsten-Sort”
\n
\n
![](\"http://i.imgur.com/chPvcqX.png\"/)
","user":"anonymous","dateUpdated":"2017-02-18T15:33:56-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487460826826_-1280240438","id":"20170218-153346_743694898","dateCreated":"2017-02-18T15:33:46-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:22364","dateFinished":"2017-02-18T15:33:56-0800","dateStarted":"2017-02-18T15:33:56-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Hash Shuffle
\n
\n
![](\"http://i.imgur.com/Welxbtq.png\")
","user":"anonymous","dateUpdated":"2017-02-18T15:34:08-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487460836778_2135037631","id":"20170218-153356_600904597","dateCreated":"2017-02-18T15:33:56-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:22438","dateFinished":"2017-02-18T15:34:08-0800","dateStarted":"2017-02-18T15:34:08-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Sort Shuffle
\n
\n
\n
SQL, DataFrame, and Dataset
\n
Catalyst, Tungsten, and Modern Spark Execution
\n
©2016, 2017 by Adam Breindel. All Rights Reserved.
\n
\n
Let’s take the list of page title words, and write it out to disk so we can easily play with it.
\n
\n
Ok, let’s start with a simple query, and take a look at the execution pattern
\n
\n
Why are there 2 stages there? (Recall that counting an RDD was just one stage and an output)
\n
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
\n
count is implemented as a “regular” aggregation, in this case with as data.groupBy().count(), with the actual result extracted for convenience
\n
In other words, each partition is counted, then the counts are merged.
\n
\n
Doing our word count with our Dataset, let’s look at the UI and understand the high-level execution: Jobs and Stages
\n
\n
As with RDDs, adding narrow operations doesn’t change the number of stages/tasks:
\n
\n
And sorting will require an extra job with a non-deterministic number of tasks (for the RangePartition)
\n
![](\"http://i.imgur.com/gRn5y6H.png\")
","dateUpdated":"2017-02-18T15:57:02-0800","dateFinished":"2017-02-18T15:57:02-0800","dateStarted":"2017-02-18T15:57:02-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Spark DataFrame/Dataset Execution Pipeline
\n
\n
\n
Ok, how can we ask for – and understand – the query plan?
\n
\n
Great info, but a little hard to read. Let’s look in the SQL UI!
\n
![](\"http://i.imgur.com/0ikv6OA.png\"/)
\n","user":"anonymous","dateUpdated":"2017-02-18T16:07:00-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487462306057_1025660935","id":"20170218-155826_2049509830","dateCreated":"2017-02-18T15:58:26-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:31115","dateFinished":"2017-02-18T16:07:00-0800","dateStarted":"2017-02-18T16:07:00-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
What does Catalyst Logical Optmizer do?
\n
(i.e., what is the difference between analyzed logical plan and optimized logical plan)
\n
Make a more performant plan that produces the same output
\n
Here is an example to discuss:
\n
\n
\n
Demo/Exploration of 2 Simple Optimizations:
\n
\n - Combine filters
\n - Push down predicate (filter), e.g., through most project operators
\n
\n
\n
If you want to take a closer look at the logical optimizer code and rules
\n
\n - Operators\n \n
\n - Expressions\n \n
\n - Transformation rules\n \n
\n
\n
Spark 2.x even has a user-facing public (though experimental) API for writing our own optimizations and dynamically adding them: http://blog.madhukaraphatak.com/introduction-to-spark-two-part-6/
\n
\n
Physical Planning and Execution
\n
\n - Physical planner
\n - Physical (SparkPlan) Optimizer
\n - Code Generation
\n
\n
![](\"http://i.imgur.com/zMGUCg5.png\")
\n
\n
\n__Additional optimizations...__\n\nWhat if the parquet file is partitioned by age? (or we're reading from a database with key columns and/or indices?)","user":"anonymous","dateUpdated":"2017-02-18T16:10:14-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487462997325_-837131775","id":"20170218-160957_1575272477","dateCreated":"2017-02-18T16:09:57-0800","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:31923","dateFinished":"2017-02-18T16:10:14-0800","dateStarted":"2017-02-18T16:10:14-0800","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
\n
\n
\n
Additional optimizations…
\n
What if the parquet file is partitioned by age? (or we’re reading from a database with key columns and/or indices?)
\n
\n
Why Whole-Stage Codegen?
\n
\n - Iterator/Volcano Model
\n - Codegen’ed Operators vs. Interpreted Queries
\n - Purpose-Built Code vs. General Framework (Freshman Paradox)\n
\n - Avoid virtual calls, page faults, leverage cache etc.
\n
\n \n
\n
\n
If you really want to dig in, look at the codegen design doc, framework, and basic physical operators…
\n
Filter is a good one to start with! look for case class FilterExec
\n
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
\n
HINTS:
\n
\n - IntelliJ has great, free Scala support and “Go To Declaration…” will help you learn your way around the code
\n - Building Spark in IntelliJ can be tricky, but if you build the repo using its own build script (easier), you can attach IntelliJ to the process via remote debugging.\n
\n - To enable remote debugging on the driver,
export SPARK_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \n
\n \n
\n
\n
Machine Learning with Spark
\n
©2016, 2017 by Adam Breindel. All Rights Reserved.
\n
Legacy API vs. Modern API
\n
\n - ML Pipelines\n
\n - DataFrame
\n - Transformer / Estimator / Pipeline
\n - CrossValidator / ParamGridBuilder / Evaluator
\n
\n \n - RDD API in maintenance mode, will be deprecated, then removed (~ 3.0?)\n
\n - Still contains some features not present in new API (e.g., SVD, covariance matrix)
\n
\n \n
\n
Example with (R/ggplot) Diamonds dataset
\n
\n - Data manipulation (DataFrame, Transformer, Estimator)
\n - Feature selection
\n - Model building (Pipeline)
\n - Evaluation (Evaluator)
\n - Tuning
\n - Crossvalidation (CrossValidator, ParamGridBuilder)
\n
\n
\n
We’ll look at the features in more detail … but right away we see we’ll have to do something about string-typed features.
\n
The price (label) is an integer, not a double. In many cases, an integer can be auto-widened to a double, but there may be some places we’ll have to watch out.
\n
Also, that “_c0” (a.k.a. the row number or row ID) … not only is it not a feature, but it can leak irrelevant data:
\n
\n
Now we need to do some processing with the categorical features: cut, color, and clarity.
\n
\n
In this example, we’ll build a Linear Regression model. For this type of model,
we will want to convert these categorical variables into a one-hot, or “dummy variable,” representation,
so we want to create a OneHotEncoder
\n
\n
The one-hot encoder takes a numeric value, so we need to convert the categorical values to numbers.
\n
We can do that with a StringIndexer
\n
\n
\n
This is a good time to think about how Transformers and Estimators work and gain some design intuition.
\n
First, we want to be clear what the StringIndexer and OneHotEncoder are supposed to do.
\n
Why might one be a transformer and the other an estimator? These two helpers could be built either as Transformers or Estimators. What factors would argue for/against using one pattern versus the other? Hint: there is a Spark JIRA to “fix” OneHotEncoder, perhaps in 2.2, to be an Estimator.
\n
Note that neither can be performed as a pure map operation. They both need to accumulate some bit of state, via a reduce, to then use in the map.
\n
EXTRA CREDIT: Try to locate these bits in the source code. Can you find where the “reduce” job is hidden in the OneHotEncoder (even though it’s a transformer)?
\n
\n
See if you can manually run these two helpers against the data.
\n
\n
Now let’s automate this work a bit: we’ll use functional collections to create our feature helpers, and a pipeline to wrap them
\n
\n
That looks pretty good. Next, we need to bring all of our features together into a single vector.
The VectorAssembler class, a Transformer (why?) does exactly that:
\n
\n
Let’s finish the pipeline by adding the Linear Regression algorithm
\n
\n
Now we’re ready to train and do an initial test
\n
\n
Looks plausible … but let’s get some numbers using the Evaluator class.
\n
Evaluator is useful for
\n
\n
\n
Ok … let’s see if we can wire Spark up to tune the model. In particular,
\n
\n - we’ll give Spark a set of hyperparamters to adjust, and some values to try
\n - let Spark build a series of tentative models using k-fold crossvalidation
\n - tell Spark what evaluator to use, so it knows which tentative models are better
\n
\n
\n
How different was the performance across the different parameter sets?
\n
\n
After training the CrossValidatorModel, cvModel.bestModel will contain a model trained on all of the training data using the best hyperparams.
\n
However, we could also train that “best model” ourselves:
\n
\n
Run the final model against the test set
\n
\n
We can examine the parameters of the final model:
\n
\n
One area where Spark ML really shines is in the pluggability of the different components.
Let’s swap in a Gradient Boosted Tree regressor and see if we can do better:
\n
\n
Not surprisingly, we did significantly better with the GBT model. See if you can add tuning and crossvalidation,
and improve the GBT model even further!
\n
\n
Spark Streaming
\n
©2016, 2017 by Adam Breindel. All Rights Reserved.
\n
\n
Structured Streaming vs. DStream Streaming
\n
\n - DStream (“classic”)\n
\n - Programming model and workarounds
\n - Operational models (receivers, receiverless)
\n - Fault-tolerance considerations
\n
\n \n - Structured Streaming (“streaming DataFrames”)\n
\n - Goals
\n - Advantages and Limitations
\n
\n \n
\n
Common Factors
\n
\n - High throughput
\n - Microbatch processing
\n - Not low latency (due to job scheduling)
\n - Ideal for bulk computation, not per-event isolated computation
\n
\n
![](\"http://i.imgur.com/Z2lp8Il.png\")
","user":"anonymous","dateUpdated":"2017-02-19T11:24:25-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487528858026_1923842115","id":"20170219-102738_2063400792","dateCreated":"2017-02-19T10:27:38-0800","dateStarted":"2017-02-19T11:24:25-0800","dateFinished":"2017-02-19T11:24:25-0800","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:41781","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
DStreams
\n
\n
\n
General Receiver-Based Model
\n
(run this next cell just to set up a data source; don’t worry about the code)
\n
\n
What does that source produce? Key-value data, where the key represents a user ID and the value a piece of sensor data (freeway speed).
\n
The records look like this:
\n
(101, 55)\n(110, 60)\n(103, 63)\n(101, 59)\n
\n
etc.
\n
\n
Our first streaming code:
\n
\n
Ok, that was better … can we add just a little more logic to the stream?
\n
\n
Nope … but what went wrong?
\n
Is it
1. a stream can only feed a single action or transformation? or
2. stream was already stopped?
\n
Per the error message, the problem is #2.
\n
We are allowed to attach any number of transformations and/or actions to streams, although there are performance implications.
\n
Why? each action will require one or more jobs, and the jobs overhead is a big factor in streaming performance. That’s why Spark does great with high throughput, provided the jobs are not too frequent (i.e., no low-latency).
\n
It's set to run for 30 seconds so you have time to look at the Spark Streaming GUI","user":"anonymous","dateUpdated":"2017-02-19T11:24:59-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487529509933_-1162077216","id":"20170219-103829_1746440337","dateCreated":"2017-02-19T10:38:29-0800","dateStarted":"2017-02-19T11:24:59-0800","dateFinished":"2017-02-19T11:24:59-0800","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:41793","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
It does work … how does the performance compare? Run the following cell with and without the comment block …
It’s set to run for 30 seconds so you have time to look at the Spark Streaming GUI
\n
![](\"http://i.imgur.com/GnTIQv9.png\")
\n\nTake a close look at the following cell. How is it different from the earlier counting code?\n\nIt uses the .count() stream transformation!","user":"anonymous","dateUpdated":"2017-02-19T11:25:28-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487529536316_-937737959","id":"20170219-103856_1195992770","dateCreated":"2017-02-19T10:38:56-0800","dateStarted":"2017-02-19T11:25:28-0800","dateFinished":"2017-02-19T11:25:28-0800","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:41795","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
DStream Transformations
\n
\n
Take a close look at the following cell. How is it different from the earlier counting code?
\n
It uses the .count() stream transformation!
\n
\n
Before going any further with the Streaming API, let’s take a quick look at a receiverless “advanced” source example, just so we can look for the similarities and differences.
\n
*Note: this code will not work unless you install a library (.jar). If you want an Uber-jar (i.e., JAR with dependencies), look for spark-streaming-kafka-XXXX
\n
Where XXXX matches your versions of Spark, Scala, and Kafka. If you’re running this with Zeppelin 0.7, that will be Spark 2.1, Scala 2.11, Kafka 0.8
\n
Even if you don’t want to wire up the libs to run this example, let’s take a quick look at the code. It’s not any more complex than what we have been doing, but with KafkaDirect, we get a ton of features that make our application more robust and easier to operate at the same time
\n
\n - Automatic parallelization across Kafka topic partitions
\n - No dedicated receiver cores
\n - Easy recovery in the case of brief failure, because Kafka keeps the records and we can just ask for them again (by offset range)\n
\n - This means we need to track offsets, but don’t need to journal the actual data
\n
\n \n
\n
\n
The DStream API includes additional operations like reduceByKey, union, and join.
If we need a DStream transformation which doesn’t exist in the API, we call dstream.transform(…)
and supply an arbitrary function that takes a RDD (for each batch) and returns a transformed RDD.
\n
\n
Stateful Streams
\n
So far, all of the streams we’ve used involve computation only on the current batch (in one or more streams).
\n
Some applications require more complex computation, such as calculating moving averages over several batched, or “sessionization” – collected all of the data for a user over an indefinite period of time until some business rule indicates the end of the “session” (e.g., user logout or idle time).
\n
To accommodate these scenarios, Spark has support for two different kinds of stateful streams.
\n
![](\"http://i.imgur.com/COBtkrV.png\")
","user":"anonymous","dateUpdated":"2017-02-19T11:25:58-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487530357001_2118884092","id":"20170219-105237_1084108607","dateCreated":"2017-02-19T10:52:37-0800","dateStarted":"2017-02-19T11:25:58-0800","dateFinished":"2017-02-19T11:25:58-0800","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:41801","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Windowed Streams
\n
\n - .window(…)
\n - “andWindow” methods: reduceByKeyAndWindow etc
\n
\n
\n
\n
Long-Term Stateful Streams
\n
\n - Collect data on per key (user, device, etc.) basis
\n - Any type of data
\n - Any type of collection structure
\n - (Almost) any business rules
\n
\n
updateStateByKey:
\n
\n
Combining Legacy Streaming with SQL/DataFrame
\n
Convert each batch into a DataFrame
\n
This example is stateless in the sense that the DataFrame is replaced anew for each batch – simple but limited
\n
\n
This approach can be useful; if we want to query over a larger (longer-term) collection of data … say, cumulative collection of records,
or hours of records, it usually a good idea to write the data out to another system. Depending on your requirements, this might be a filesystem (HDFS, S3, etc.) or a database that supports queries, like Cassandra.
\n
\n
What about all of the operational considerations?
\n
Fault Tolerance
\n
\n - Reliable Receivers
\n - Write-Ahead Log
\n - Data Checkpointing
\n
\n
High Availability
\n
\n - Driver Recovery
\n - Metadata Checkpointing
\n
\n
Exactly-Once End-to-End Message Processing
\n
\n
\n
Hmm…
\nWhat if we could combine streaming with DataFrames, optimized state tracking for aggregations, and simplified fault tolerance with end-to-end guarantees?
\n"}]}},{"text":"%md ### Structured Streaming\n\n* Fault tolerance\n* Available source/sink strategies\n* Incremental query optimization\n\n\"Easiest way to reason about streaming is to ... not reason about streaming\"\n\n* Treat sources as \"append-only\" tables\n* Read and manipulate with DataFrame API\n* Internal resilient state management for aggregations over time\n* Output to sinks","user":"anonymous","dateUpdated":"2017-02-19T11:26:24-0800","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1487530819506_-1232636217","id":"20170219-110019_800101349","dateCreated":"2017-02-19T11:00:19-0800","dateStarted":"2017-02-19T11:26:24-0800","dateFinished":"2017-02-19T11:26:24-0800","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:41811","results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"\n
Structured Streaming
\n
\n - Fault tolerance
\n - Available source/sink strategies
\n - Incremental query optimization
\n
\n
“Easiest way to reason about streaming is to … not reason about streaming”
\n
\n - Treat sources as “append-only” tables
\n - Read and manipulate with DataFrame API
\n - Internal resilient state management for aggregations over time
\n - Output to sinks
\n
\n
\n
To keep the environment simple for this class, and save time,
we’ll use a socket source and a memory sink to demonstrate Structured Streaming.
In production, it is essential to use fault-tolerant sources and sinks such as the Kafka source and the filesystem sink.
\n
\n
We’ll try something a little more practical.
\n
\n - Interpret the time as SQL timestamp (it was a raw string earlier)
\n - Aggregate over a time window
\n - Transform the stream by grouping by channel and time, then counting edits
\n - Specfiy a trigger interval
\n
\n