Sustainable Spark Development
47 |
48 | Sean McIntyre
49 | Software Architect
50 |
52 | The Problem
56 |Uncharted in a nutshell:
59 |-
60 |
- Data Science 61 |
- Machine Learning 62 |
- Data cleaning and enrichment 63 |
- Custom visual analytics software 64 |
66 | Visualization is a broad term these days, and it often involves a lot of data processing to create something useful. 67 |
68 |From a different perspective:
71 |-
72 |
- Data Scientists 73 |
- Computer Scientists 74 |
- Software Engineers 75 |
- Dev Ops 76 |
- ...and many more 77 |
Developing big, data-driven applications involves all of these folks!
79 |
85 | The informal workflow
86 | i.e. "what always happens"
87 |
88 | -
89 |
- Data collection (Dev Ops/Engineers) 90 |
- Exploratory data analysis (Data scientist) 91 |
- Application development (Engineers) 92 |
- Algorithm optimization (Data/computer scientists) 93 |
- Deployment/Scaling (Engineers/Dev Ops) 94 |
Issues:
98 |-
99 |
- Everyone ends up touching Spark 100 |
- Spark is hard to learn, and hard to use correctly 101 |
- Scripts written for data analysis are productionized due to time constraints 102 |
- Scripts are not sufficiently modular to re-use. Wheel reinventing ensues. 103 |
It became clear very quickly that this process is not sustainable.
105 |What is Sustainability?
110 |Priorities:
113 |-
114 |
- Retain the productivity/flexibility afforded by scripting 115 |
- Balance with the need for modularity 116 |
- Promote code reuse and composability
"Hey, how did you parse that twitter data?"
117 | - Reduce barriers to entry for new team members
people shouldn't have to be Spark experts to write efficient code
118 |
The Solution
124 |127 | Sparkpipe! 128 |
129 | $ ./spark-shell \
130 | --packages software.uncharted.sparkpipe:sparkpipe-core:0.9.7
131 |
132 | // import data
135 | val df = sqlContext.read
136 | .format("json")
137 | .load("tweets.json.gz")
138 |
139 | // add column containing words from tweet text
140 | import scala.reflect.runtime.universe._
141 | import org.apache.spark.sql.Column
142 | val wordsColumn = udf {
143 | (text: String) => text.trim.split("\\s+?")
144 | }(typeTag[Array[String]], typeTag[String])(new Column("text"))
145 | val df2 = df.withColumn("tweet_words", wordsColumn)
146 |
147 | // do other things (if you're not in tears)
148 |
149 | There are things which are unintuitive in the Spark API
150 | import software.uncharted.sparkpipe.Pipe
153 | import software.uncharted.sparkpipe.ops.core.{dataframe => dfo}
154 |
155 | val ingest = Pipe(sqlContext)
156 | .to(dfo.io.read(format = "json", path = "tweets.json.gz"))
157 | .to(dfo.addColumn(
158 | "tweet_words",
159 | (text: String) => text.trim.split("\\s+?"),
160 | "text"
161 | ))
162 | val df = ingest.run
163 |
164 | Sparkpipe aims to make them simpler and more semantically clear
165 | import software.uncharted.sparkpipe.Pipe
168 | import software.uncharted.sparkpipe.ops.community.{twitter => twops}
169 |
170 | val ingest = Pipe(sqlContext)
171 | .to(twops.io.read(format = "json", path = "tweets.json.gz"))
172 | .to(twops.tweet.hashtags("tweet_hashtags"))
173 |
174 | val df = ingest.run
175 |
176 | more importantly, we want to reduce boilerplate
in domain-specific situations
180 | The operation becomes the single modular unit that developers and data scientists can agree on. 181 |
182 | // operations are just simple scala functions!
183 | def myOperation(arg1: String, arg2: Int)(input: DataFrame): DataFrame = {
184 | /* do something */
185 | }
186 |
187 |
188 | They should each do one thing, and do it well
(with well-defined inputs and outputs)
189 |
©UNIX :)
190 |
Advantages for Uncharted
197 |-
198 |
- Pipelines are highly scriptable! 199 |
- Isolation from changes in the underlying Spark API
(i.e. RDD -> DataFrame -> Dataset)
200 | - Hide the uglier parts of Spark 201 |
- Pipelines separate scripts into modular, reusable bits without adding too much overhead 202 |
- Productionize scripts, optimize later 203 |
- Simple DSL encourages a specific format for functions, and separates structure from implementation details 204 |
Separating structure from implementation details:
208 | // linear
209 | val ingest = Pipe(red).to(blue).to(green).to(yellow)
210 | // branch
211 | val analysis1 = pipe.to(something).to(somethingElse)
212 | val analysis2 = pipe.to(anotherOp)
213 | // merge
214 | val output = Pipe(analysis1, analysis2).to(outputOp)
215 | //non-linear pipelines are cool!
216 |
217 | 218 | This makes even complex scripts easier to parse and share than the corresponding raw Spark code. 219 |
220 |The Bottom Line
225 |
228 | We're not trying to replace the underlying Spark API, so you generally won't find ops that wrap simple things like select().
229 |
Pipe(sqlContext)
231 | .to(twops.io.read(format = "json", path = "tweets.json.gz"))
232 | .to(_.select("text")) //you can use scala inlines anytime you want!
233 | .run
234 |
235 |
238 | In particular, we're not trying to replace the SparkML Pipeline, though we do have ops for using it with Sparkpipe.
239 |
We've released sparkpipe-core as open source, and it's available now from Maven central.
244 | We're also hard at work on domain-specific libraries which will also be open source. 245 |
246 |-
247 |
sparkpipe-twitter-ops
248 | sparkpipe-instagram-ops
249 | sparkpipe-gdelt-ops
250 | sparkpipe-salt-ops(visualization)
251 | - etc. 252 |
But I need your help!
256 |Creating sparkpipe has helped improve the productivity of developers within my company.
But it would be even more powerful if a community of domain-specific libraries developed around it!
258 | 261 |