| File | What's Illustrated |
|---|---|
| Simple.scala | 13 |
14 | A very simple example of structured streaming from a Kafka source, where the messages 15 | are produced directly via calls to a KafkaProducer. A streaming DataFrame is created from a 16 | single Kafka topic, and feeds all the data received to a streaming computation that outputs it to a console. 17 |Note that writing all the incremental data in each batch to output only makes sense because there is no 18 | aggregation performed. In subsequent examples with aggregation this will not be possible. 19 | |
20 |
| SimpleAggregation.scala | 23 |
24 | A streaming DataFrame is created from a single Kafka topic, an aggregating query is set up to count 25 | occurrences of each key, and the results are streamed to a console. Each batch results in the entire 26 | aggregation result to date being output. 27 | |
28 |
| SubscribeAndPublish.scala | 31 |
32 | Two Kafka topics are set up and a KafkaProducer is used to publish to the first topic. 33 | Then structured streaming is used to subscribe to that topic and publish a running aggregation to the 34 | second topic. Finally structured streaming is used to subscribe to the second topic and print the data received. 35 | 36 | |
37 |
| Foreach.scala | 41 |
42 | 43 | The 'foreach' operation allows arbitrary computations on the output data in way that is both 44 | partition-aware (computed on the executors and aware of which partition is being processed) and batch-aware 45 | (via a separate invocation for each partition/batch combination.) 46 | 47 | It is always used by passing the operation an object that implements the 'ForeachWriter' interface. In this 48 | example, the object doesn't do any "useful" work: instead it is set up to illustrate its slightly arcane state 49 | management by printing its arguments and state in each of the three overridden methods. 50 | 51 | Each instance of ForeachWriter is used for processing a sequence of partition/batch combinations, but at any point 52 | in time is is setup (via a single open() call) to process one partition/batch combination. Then it gets multiple 53 | process() calls, providing the the actual data for that partition and batch, and then a single close() call to 54 | signal that the partition/batch combination has been completely processed. 55 | 56 | |
57 |
| File | Purpose |
|---|---|
| util/DirectServerDemo.scala | 55 |Run this first as a Spark-free sanity check for embedded server and clients. | 56 |
| util/EmbeddedKafkaServer.scala | 59 |Starting and stopping an embedded Kafka server, and creating and modifying topics. | 60 |
| util/EmbeddedZookeeper.scala | 63 |Starting and stopping an embedded Zookeeper. | 64 |
| util/PartitionMapAnalyzer.scala | 67 |Support for understanding how subscribed Kafka topics and their Kafka partitions map to partitions in the 68 | RDD that is emitted by the Spark stream. | 69 |
| util/SimpleKafkaClient.scala | 72 |Directly connect to Kafka without using Spark. | 73 |
| util/SparkKafkaSink.scala | 76 |Support for publishing to Kafka topic in parallel from Spark. | 77 |
| util/TemporaryDirectories.scala | 80 |Support for creating and cleaning up temporary directories needed for Kafka broker, Zookeeper and 81 | Spark streaming. | 82 |
| File | What's Illustrated |
|---|---|
| SimpleStreaming.scala | 91 |Simple way to set up streaming from a Kafka topic. While this program also publishes to the topic, the publishing does not involve Spark | 92 |
| ExceptionPropagation.scala | 95 |Show how call to awaitTermination() throws propagated exceptions. | 96 |
| File | What's Illustrated |
|---|---|
| SimpleStreamingFromRDD.scala | 127 |Data is published by Spark from an RDD, but is repartitioned even through the publishing RDD and the topic have the same number of partitions. | 128 |
| SendWithDifferentPartitioning.scala | 131 |Send to a topic with different number of partitions. | 132 |
| ControlledPartitioning.scala | 135 |When publishing to the topic, explicitly assign each record to a partition. | 136 |
| AddPartitionsWhileStreaming.scala | 139 |Partitions can be added to a Kafka topic dynamically. This example shows that an existing stream 140 | will not see the data published to the new partitions, and only when the existing streaming context is terminated 141 | and a new stream is started from a new context will that data be delivered. 142 |143 | The topic is created with three partitions, and so each RDD the stream produces has three partitions as well, 144 | even after two more partitions are added to the topic. This is what's received after the first 500 records 145 | are published to the topic while it has only three partitions: 146 |147 | [1] *** got an RDD, size = 500 148 | [1] *** 3 partitions 149 | [1] *** partition size = 155 150 | [1] *** partition size = 173 151 | [1] *** partition size = 172 152 |153 | When two partitions are added and another 500 messages are published, this is what's received 154 | (note both the number of partitions and the number of messages): 155 |156 | [1] *** got an RDD, size = 288 157 | [1] *** 3 partitions 158 | [1] *** partition size = 98 159 | [1] *** partition size = 89 160 | [1] *** partition size = 101 161 |162 | 163 | When a new stream is subsequently created, the RDDs produced 164 | have five partitions, but only two of them contain data, as all the data has been drained from the initial three 165 | partitions of the topic, by the first stream. Now all 500 messages (288 + 212) from the second set have been delivered. 166 | 167 |168 | [2] *** got an RDD, size = 212 169 | [2] *** 5 partitions 170 | [2] *** partition size = 0 171 | [2] *** partition size = 0 172 | [2] *** partition size = 0 173 | [2] *** partition size = 112 174 | [2] *** partition size = 100 175 |176 | 177 | |
178 |
| File | What's Illustrated |
|---|---|
| MultipleConsumerGroups.scala | 187 |Two streams subscribing to the same topic via two consumer groups see all the same data. | 188 |
| MultipleStreams.scala | 191 |Two streams subscribing to the same topic via a single consumer group divide up the data. 192 | There's an interesting partitioning interaction here as the streams each get data from two fo the four topic 193 | partitions, and each produce RDDs with two partitions each. | 194 |
| MultipleTopics.scala | 197 |A single stream subscribing to the two topics receives data from both of them.
198 | The partitioning behavior here is quite interesting.
199 |
206 | *** got an RDD, size = 200 207 | *** 9 partitions 208 | *** partition 1 has 27 records 209 | *** rdd partition = 1, topic = foo, topic partition = 0, record count = 27. 210 | *** partition 2 has 15 records 211 | *** rdd partition = 2, topic = bar, topic partition = 1, record count = 15. 212 | *** partition 3 has 17 records 213 | *** rdd partition = 3, topic = bar, topic partition = 0, record count = 17. 214 | *** partition 4 has 39 records 215 | *** rdd partition = 4, topic = foo, topic partition = 1, record count = 39. 216 | *** partition 5 has 34 records 217 | *** rdd partition = 5, topic = foo, topic partition = 2, record count = 34. 218 | *** partition 6 has 11 records 219 | *** rdd partition = 6, topic = bar, topic partition = 3, record count = 11. 220 | *** partition 7 has 18 records 221 | *** rdd partition = 7, topic = bar, topic partition = 4, record count = 18. 222 | *** partition 8 has 20 records 223 | *** rdd partition = 8, topic = bar, topic partition = 2, record count = 20. 224 |225 | |
226 |
| Timestamp.scala | 229 |
230 | Record timestamps were introduced into Kafka 0.10 as described in 231 | KIP-32 232 | and 233 | KIP-33. 234 | 235 |This example sets up two different topics that handle timestamps differently -- topic A has the timestamp 236 | set by the broker when it receives the record, while topic B passes through the timestamp provided in the record 237 | (either programmatically when the record was created, as shown here, or otherwise automatically by the producer.) 238 | 239 |Since the record carries information about where its timestamp originates, its easy to subscribe to the two topics 240 | to create a single stream, and then examine the timestamp of every received record and its type. 241 | 242 |NOTE: The use of timestamps to filter topics in the broker, as introduced in Kafka 0.10.1, is blocked on 243 | SPARK-18057. 244 | |
245 |