![](\"//s3.amazonaws.com/datastaxtraining/developer-day/cassandra-day-studio-header.png\")
|
11 | 12 | 13 | 14 | Two atoms are walking down the street when one 15 | atom says to the other 16 | "Oh, my! I've lost an electron!" 17 | 18 | The second atom says"Are you sure" 19 | 20 | The first replies "I'm positive!" 21 | 22 | 23 | |
| 11 | |
12 | 13 | 14 | 15 | Arnold Swartzeneger and Sylvester Stallone are making a movie about 16 | the lives of the great composers. 17 | Stallone says "I want to be Mozart." 18 | Swartzeneger says: "In that case... I'll be Bach." 19 | 20 | |
23 |
![\"line\"](\"https://datastaxtraining.s3.amazonaws.com/developer-day/lineblue.png\" "\\"line\\"")
![](\"images/seinfeld.jpg\")
\n",
10 | "\n",
11 | "#### Real Dataset: http://eigentaste.berkeley.edu/dataset/ Dataset 2 \n",
12 | "#### Rate Jokes: http://eigentaste.berkeley.edu"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "## What are we trying to learn from this dataset?\n",
20 | "\n",
21 | "# QUESTION: Can Collaborative Filtering be used to find which jokes to recommend to our users?\n"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "%matplotlib inline\n",
31 | "import matplotlib.pyplot as plt"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "import pandas\n",
41 | "import cassandra\n",
42 | "import pyspark\n",
43 | "import re\n",
44 | "import os\n",
45 | "import matplotlib.pyplot as plt\n",
46 | "from IPython.display import IFrame\n",
47 | "from IPython.display import display, Markdown\n",
48 | "from pyspark.sql import SparkSession\n",
49 | "from pyspark.ml.evaluation import RegressionEvaluator\n",
50 | "from pyspark.ml.recommendation import ALS\n",
51 | "from pyspark.sql import Row"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "metadata": {},
57 | "source": [
58 | "#### Helper function to have nicer formatting of Spark DataFrames"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "execution_count": null,
64 | "metadata": {},
65 | "outputs": [],
66 | "source": [
67 | "#Helper for pretty formatting for Spark DataFrames\n",
68 | "def showDF(df, limitRows = 10, truncate = False):\n",
69 | " if(truncate):\n",
70 | " pandas.set_option('display.max_colwidth', 100)\n",
71 | " else:\n",
72 | " pandas.set_option('display.max_colwidth', -1)\n",
73 | " pandas.set_option('display.max_rows', limitRows)\n",
74 | " display(df.limit(limitRows).toPandas())\n",
75 | " pandas.reset_option('display.max_rows')"
76 | ]
77 | },
78 | {
79 | "cell_type": "markdown",
80 | "metadata": {},
81 | "source": [
82 | "## Creating Tables and Loading Tables"
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "metadata": {},
88 | "source": [
89 | "### Connect to Cassandra"
90 | ]
91 | },
92 | {
93 | "cell_type": "code",
94 | "execution_count": null,
95 | "metadata": {},
96 | "outputs": [],
97 | "source": [
98 | "from cassandra.cluster import Cluster\n",
99 | "\n",
100 | "cluster = Cluster(['dse'])\n",
101 | "session = cluster.connect()"
102 | ]
103 | },
104 | {
105 | "cell_type": "markdown",
106 | "metadata": {},
107 | "source": [
108 | "### Create Demo Keyspace "
109 | ]
110 | },
111 | {
112 | "cell_type": "code",
113 | "execution_count": null,
114 | "metadata": {},
115 | "outputs": [],
116 | "source": [
117 | "session.execute(\"\"\"\n",
118 | " CREATE KEYSPACE IF NOT EXISTS accelerate \n",
119 | " WITH REPLICATION = \n",
120 | " { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }\"\"\"\n",
121 | ")"
122 | ]
123 | },
124 | {
125 | "cell_type": "markdown",
126 | "metadata": {},
127 | "source": [
128 | "### Set keyspace "
129 | ]
130 | },
131 | {
132 | "cell_type": "code",
133 | "execution_count": null,
134 | "metadata": {},
135 | "outputs": [],
136 | "source": [
137 | "session.set_keyspace('accelerate')"
138 | ]
139 | },
140 | {
141 | "cell_type": "markdown",
142 | "metadata": {},
143 | "source": [
144 | "### Create table called jokes. Our PRIMARY will need to be a unique composite key (userid, jokeid). This will result in an even distribution of the data and allow for each row to be unique. Remember we will have to utilize that PRIMARY KEY in our WHERE clause in any of our CQL queries. "
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "execution_count": null,
150 | "metadata": {},
151 | "outputs": [],
152 | "source": [
153 | "query = \"CREATE TABLE IF NOT EXISTS jokes \\\n",
154 | " (userid int, jokeid int, rating float, \\\n",
155 | " PRIMARY KEY (userid, jokeid))\"\n",
156 | "session.execute(query)"
157 | ]
158 | },
159 | {
160 | "cell_type": "markdown",
161 | "metadata": {},
162 | "source": [
163 | "### What do these of these 3 columns represent: \n",
164 | "\n",
165 | "* **Column 1**: User id\n",
166 | "* **Column 2**: Joke id\n",
167 | "* **Column 3**: Rating of joke (-10.00 - 10.00) "
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "metadata": {},
173 | "source": [
174 | "### Load Jokes dataset from CSV file (jester_ratings3.csv)\n",
175 | "* This is a file I created from the *.dat file and I only have 10,000 rows -- dataset has over 1 million rows\n",
176 | "![](\"images/laughing.gif\")
\n",
177 | "\n",
178 | "#### Insert all the Joke Rating Data into the table `jokes`"
179 | ]
180 | },
181 | {
182 | "cell_type": "code",
183 | "execution_count": null,
184 | "metadata": {},
185 | "outputs": [],
186 | "source": [
187 | "fileName = 'data/jester_ratings3.csv'\n",
188 | "input_file = open(fileName, 'r')\n",
189 | "\n",
190 | "for line in input_file:\n",
191 | " jokeRow = line.split(',')\n",
192 | " query = \"INSERT INTO jokes (userid, jokeid, rating)\"\n",
193 | " \n",
194 | " query = query + \"VALUES (%s, %s, %s)\"\n",
195 | " \n",
196 | " session.execute(query, (int(jokeRow[0]), int(jokeRow[1]) , float(jokeRow[2]) ))"
197 | ]
198 | },
199 | {
200 | "cell_type": "markdown",
201 | "metadata": {},
202 | "source": [
203 | "#### Do a select * on joke_table WHERE userid = x to verify that data was loaded into the table"
204 | ]
205 | },
206 | {
207 | "cell_type": "code",
208 | "execution_count": null,
209 | "metadata": {
210 | "scrolled": true
211 | },
212 | "outputs": [],
213 | "source": [
214 | "query = 'SELECT * FROM jokes WHERE userid = 65'\n",
215 | "rows = session.execute(query)\n",
216 | "for row in rows:\n",
217 | " print (row.userid, row.jokeid, row.rating)"
218 | ]
219 | },
220 | {
221 | "cell_type": "markdown",
222 | "metadata": {},
223 | "source": [
224 | "![](\"images/sparklogo.png\")
"
225 | ]
226 | },
227 | {
228 | "cell_type": "markdown",
229 | "metadata": {},
230 | "source": [
231 | "### Finally time for Apache Spark! "
232 | ]
233 | },
234 | {
235 | "cell_type": "markdown",
236 | "metadata": {},
237 | "source": [
238 | "#### Create a spark session that is connected to Cassandra. From there load each table into a Spark Dataframe and take a count of the number of rows in each."
239 | ]
240 | },
241 | {
242 | "cell_type": "code",
243 | "execution_count": null,
244 | "metadata": {},
245 | "outputs": [],
246 | "source": [
247 | "spark = SparkSession.builder.appName('demo').master(\"local\").getOrCreate()\n",
248 | "\n",
249 | "jokeTable = spark.read.format(\"org.apache.spark.sql.cassandra\").options(table=\"jokes\", keyspace=\"accelerate\").load()\n",
250 | "\n",
251 | "print (\"Table Row Count: \")\n",
252 | "print (jokeTable.count())"
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {},
258 | "source": [
259 | "#### Split dataset into training and testing set "
260 | ]
261 | },
262 | {
263 | "cell_type": "code",
264 | "execution_count": null,
265 | "metadata": {},
266 | "outputs": [],
267 | "source": [
268 | "(training, test) = jokeTable.randomSplit([0.8, 0.2])\n",
269 | "\n",
270 | "training_df = training.withColumn(\"rating\", training.rating.cast('int'))\n",
271 | "testing_df = test.withColumn(\"rating\", test.rating.cast('int'))\n",
272 | "\n",
273 | "showDF(training_df)"
274 | ]
275 | },
276 | {
277 | "cell_type": "markdown",
278 | "metadata": {},
279 | "source": [
280 | "### Setup for CFliter with ALS\n",
281 | "\n",
282 | "https://spark.apache.org/docs/latest/ml-collaborative-filtering.html"
283 | ]
284 | },
285 | {
286 | "cell_type": "code",
287 | "execution_count": null,
288 | "metadata": {},
289 | "outputs": [],
290 | "source": [
291 | "als = ALS(maxIter=5, regParam=0.01, userCol=\"userid\", itemCol=\"jokeid\", ratingCol=\"rating\",\n",
292 | " coldStartStrategy=\"drop\")\n",
293 | "\n",
294 | "model = als.fit(training_df)"
295 | ]
296 | },
297 | {
298 | "cell_type": "code",
299 | "execution_count": null,
300 | "metadata": {},
301 | "outputs": [],
302 | "source": [
303 | "# Evaluate the model by computing the RMSE on the test data\n",
304 | "predictions = model.transform(testing_df)\n",
305 | "\n",
306 | "# Generate top 10 joke recommendations for each user\n",
307 | "userRecs = model.recommendForAllUsers(10)\n",
308 | "\n",
309 | "showDF(userRecs)\n",
310 | "\n",
311 | "# Generate top 10 user recommendations for each joke\n",
312 | "jokeRecs = model.recommendForAllItems(10)"
313 | ]
314 | },
315 | {
316 | "cell_type": "code",
317 | "execution_count": null,
318 | "metadata": {},
319 | "outputs": [],
320 | "source": [
321 | "showDF(userRecs.filter(userRecs.userid == 65))"
322 | ]
323 | },
324 | {
325 | "cell_type": "code",
326 | "execution_count": null,
327 | "metadata": {},
328 | "outputs": [],
329 | "source": [
330 | "IFrame(src='images/init94.html', width=700, height=200)"
331 | ]
332 | },
333 | {
334 | "cell_type": "code",
335 | "execution_count": null,
336 | "metadata": {},
337 | "outputs": [],
338 | "source": [
339 | "IFrame(src='images/init43.html', width=700, height=200)"
340 | ]
341 | }
342 | ],
343 | "metadata": {
344 | "kernelspec": {
345 | "display_name": "Python 3",
346 | "language": "python",
347 | "name": "python3"
348 | },
349 | "language_info": {
350 | "codemirror_mode": {
351 | "name": "ipython",
352 | "version": 3
353 | },
354 | "file_extension": ".py",
355 | "mimetype": "text/x-python",
356 | "name": "python",
357 | "nbconvert_exporter": "python",
358 | "pygments_lexer": "ipython3",
359 | "version": "3.7.3"
360 | }
361 | },
362 | "nbformat": 4,
363 | "nbformat_minor": 2
364 | }
365 |
--------------------------------------------------------------------------------
/jupyter/Random Forest.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Demo 3: Random Forest and DataStax Analytics\n",
8 | "------\n",
9 | "![](\"images/drinkWine.jpeg\")
\n",
10 | "\n",
11 | "\n",
12 | "#### Dataset: https://archive.ics.uci.edu/ml/datasets/Wine+Quality"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "## What are we trying to learn from this dataset? \n",
20 | "\n",
21 | "# QUESTION: Can Random Forest be used to classify a wine’s rating score by its attributes?"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "%matplotlib inline\n",
31 | "import matplotlib.pyplot as plt"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "import pandas\n",
41 | "import cassandra\n",
42 | "import pyspark\n",
43 | "import re\n",
44 | "import os\n",
45 | "import random\n",
46 | "from random import randint, randrange\n",
47 | "import matplotlib.pyplot as plt\n",
48 | "from IPython.display import display, Markdown\n",
49 | "from pyspark.sql import SparkSession\n",
50 | "from pyspark.ml.classification import RandomForestClassifier\n",
51 | "from pyspark.ml.feature import IndexToString, StringIndexer, VectorAssembler\n",
52 | "from pyspark.ml.evaluation import MulticlassClassificationEvaluator"
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {},
58 | "source": [
59 | "#### Helper function to have nicer formatting of Spark DataFrames"
60 | ]
61 | },
62 | {
63 | "cell_type": "code",
64 | "execution_count": null,
65 | "metadata": {},
66 | "outputs": [],
67 | "source": [
68 | "#Helper for pretty formatting for Spark DataFrames\n",
69 | "def showDF(df, limitRows = 5, truncate = True):\n",
70 | " if(truncate):\n",
71 | " pandas.set_option('display.max_colwidth', 50)\n",
72 | " else:\n",
73 | " pandas.set_option('display.max_colwidth', -1)\n",
74 | " pandas.set_option('display.max_rows', limitRows)\n",
75 | " display(df.limit(limitRows).toPandas())\n",
76 | " pandas.reset_option('display.max_rows')"
77 | ]
78 | },
79 | {
80 | "cell_type": "markdown",
81 | "metadata": {},
82 | "source": [
83 | "## Creating Tables and Loading Tables"
84 | ]
85 | },
86 | {
87 | "cell_type": "markdown",
88 | "metadata": {},
89 | "source": [
90 | "### Connect to Cassandra"
91 | ]
92 | },
93 | {
94 | "cell_type": "code",
95 | "execution_count": null,
96 | "metadata": {},
97 | "outputs": [],
98 | "source": [
99 | "from cassandra.cluster import Cluster\n",
100 | "\n",
101 | "cluster = Cluster(['dse'])\n",
102 | "session = cluster.connect()"
103 | ]
104 | },
105 | {
106 | "cell_type": "markdown",
107 | "metadata": {},
108 | "source": [
109 | "### Create Demo Keyspace "
110 | ]
111 | },
112 | {
113 | "cell_type": "code",
114 | "execution_count": null,
115 | "metadata": {},
116 | "outputs": [],
117 | "source": [
118 | "session.execute(\"\"\"\n",
119 | " CREATE KEYSPACE IF NOT EXISTS accelerate \n",
120 | " WITH REPLICATION = \n",
121 | " { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }\"\"\"\n",
122 | ")"
123 | ]
124 | },
125 | {
126 | "cell_type": "markdown",
127 | "metadata": {},
128 | "source": [
129 | "### Set keyspace "
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": null,
135 | "metadata": {},
136 | "outputs": [],
137 | "source": [
138 | "session.set_keyspace('accelerate')"
139 | ]
140 | },
141 | {
142 | "cell_type": "markdown",
143 | "metadata": {},
144 | "source": [
145 | "### Create table called `wines`. Our PRIMARY will be a unique key (wineid) we generate for each row. This will have two datasets \"white\" and \"red\""
146 | ]
147 | },
148 | {
149 | "cell_type": "code",
150 | "execution_count": null,
151 | "metadata": {},
152 | "outputs": [],
153 | "source": [
154 | "query = \"CREATE TABLE IF NOT EXISTS wines \\\n",
155 | " (wineid int, fixedAcidity float, volatileAcidity float, citricAcid float, sugar float, \\\n",
156 | " chlorides float, freeSulfur float, totalSulfur float, density float, ph float, \\\n",
157 | " sulphates float, alcohol float, quality float, \\\n",
158 | " PRIMARY KEY (wineid))\"\n",
159 | "session.execute(query)"
160 | ]
161 | },
162 | {
163 | "cell_type": "markdown",
164 | "metadata": {},
165 | "source": [
166 | "### What do these of these 12 columns represent: \n",
167 | "\n",
168 | "* **Fixed acidity**\n",
169 | "* **Volatile acidity**\n",
170 | "* **Citric Acid**\n",
171 | "* **Residual Sugar** \n",
172 | "* **Chlorides**\n",
173 | "* **Free sulfur dioxide** \n",
174 | "* **Total sulfur dioxide**\n",
175 | "* **Density** \n",
176 | "* **pH**\n",
177 | "* **Sulphates**\n",
178 | "* **Alcohol**\n",
179 | "* **Quality**"
180 | ]
181 | },
182 | {
183 | "cell_type": "markdown",
184 | "metadata": {},
185 | "source": [
186 | "### Load 2 Wine Dataset -- White and Red\n",
187 | "![](\"images/whiteAndRed.jpeg\")
"
188 | ]
189 | },
190 | {
191 | "cell_type": "markdown",
192 | "metadata": {},
193 | "source": [
194 | "### Load Wine datasets from CSV file (winequality-red.csv winequality-white.csv)\n",
195 | "* No clean up was requried! How nice :)\n",
196 | "\n",
197 | "#### Insert all the Wine Data into the DSE table `wines`"
198 | ]
199 | },
200 | {
201 | "cell_type": "code",
202 | "execution_count": null,
203 | "metadata": {},
204 | "outputs": [],
205 | "source": [
206 | "fileName = 'data/winequality-red.csv'\n",
207 | "input_file = open(fileName, 'r')\n",
208 | "i = 1\n",
209 | "for line in input_file:\n",
210 | " wineid = i\n",
211 | " row = line.split(';')\n",
212 | " \n",
213 | " query = \"INSERT INTO wines (wineid, fixedAcidity, volatileAcidity, citricAcid, sugar, \\\n",
214 | " chlorides, freeSulfur, totalSulfur, density, ph, \\\n",
215 | " sulphates, alcohol, quality)\"\n",
216 | " query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\"\n",
217 | " session.execute(query, (wineid, float(row[0]), float(row[1]), float(row[2]), float(row[3]), float(row[4]), float(row[5]), float(row[6]), float(row[7]), float(row[8]), float(row[9]), float(row[10]), float(row[11])))\n",
218 | " i = i + 1\n",
219 | "\n",
220 | "fileName = 'data/winequality-white.csv'\n",
221 | "input_file = open(fileName, 'r')\n",
222 | "\n",
223 | "for line in input_file:\n",
224 | " wineid = i\n",
225 | " row = line.split(';')\n",
226 | " \n",
227 | " query = \"INSERT INTO wines (wineid, fixedAcidity, volatileAcidity, citricAcid, sugar, \\\n",
228 | " chlorides, freeSulfur, totalSulfur, density, ph, \\\n",
229 | " sulphates, alcohol, quality)\"\n",
230 | " query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\"\n",
231 | " session.execute(query, (wineid, float(row[0]), float(row[1]), float(row[2]), float(row[3]), float(row[4]), float(row[5]), float(row[6]), float(row[7]), float(row[8]), float(row[9]), float(row[10]), float(row[11])))\n",
232 | " i = i + 1\n",
233 | " "
234 | ]
235 | },
236 | {
237 | "cell_type": "markdown",
238 | "metadata": {},
239 | "source": [
240 | "## Machine Learning with Apache Cassandra and Apache Spark\n",
241 | ""
242 | ]
243 | },
244 | {
245 | "cell_type": "markdown",
246 | "metadata": {},
247 | "source": [
248 | "#### Create a spark session that is connected to the database. From there load each table into a Spark Dataframe and take a count of the number of rows in each."
249 | ]
250 | },
251 | {
252 | "cell_type": "code",
253 | "execution_count": null,
254 | "metadata": {},
255 | "outputs": [],
256 | "source": [
257 | "spark = SparkSession.builder.appName('demo').master(\"local\").getOrCreate()\n",
258 | "\n",
259 | "\n",
260 | "wineDF = spark.read.format(\"org.apache.spark.sql.cassandra\").options(table=\"wines\", keyspace=\"accelerate\").load()\n",
261 | "\n",
262 | "print (\"Table Wine Row Count: \")\n",
263 | "print (wineDF.count())"
264 | ]
265 | },
266 | {
267 | "cell_type": "code",
268 | "execution_count": null,
269 | "metadata": {},
270 | "outputs": [],
271 | "source": [
272 | "showDF(wineDF)"
273 | ]
274 | },
275 | {
276 | "cell_type": "markdown",
277 | "metadata": {},
278 | "source": [
279 | "#### Let's filter out only wines that have been rated 6.0 or higher and create a new dataframe with that information "
280 | ]
281 | },
282 | {
283 | "cell_type": "code",
284 | "execution_count": null,
285 | "metadata": {},
286 | "outputs": [],
287 | "source": [
288 | "wine6DF = wineDF.filter(\"quality > 5\")\n",
289 | "showDF(wine6DF)"
290 | ]
291 | },
292 | {
293 | "cell_type": "markdown",
294 | "metadata": {},
295 | "source": [
296 | "#### Create Vector with all elements of the wine "
297 | ]
298 | },
299 | {
300 | "cell_type": "code",
301 | "execution_count": null,
302 | "metadata": {},
303 | "outputs": [],
304 | "source": [
305 | "assembler = VectorAssembler(\n",
306 | " inputCols=['alcohol', 'chlorides', 'citricacid', 'density', 'fixedacidity', 'ph', 'freesulfur', 'sugar', 'sulphates', 'totalsulfur', 'volatileacidity'],\n",
307 | " outputCol='features')\n",
308 | "\n",
309 | "trainingData = assembler.transform(wine6DF)\n",
310 | "\n",
311 | "labelIndexer = StringIndexer(inputCol=\"quality\", outputCol=\"label\", handleInvalid='keep')\n",
312 | "trainingData1 = labelIndexer.fit(trainingData).transform(trainingData)\n",
313 | "\n",
314 | "showDF(trainingData1)\n",
315 | "print(trainingData1.count())"
316 | ]
317 | },
318 | {
319 | "cell_type": "markdown",
320 | "metadata": {},
321 | "source": [
322 | "### We will be training a model with Random Forest, and because of this we need to split up our dataset in to a training and test set. Will split 80/20. "
323 | ]
324 | },
325 | {
326 | "cell_type": "code",
327 | "execution_count": null,
328 | "metadata": {},
329 | "outputs": [],
330 | "source": [
331 | "# Split the data into train and test\n",
332 | "splits = trainingData1.randomSplit([0.8, 0.2], 1234)\n",
333 | "train = splits[0]\n",
334 | "test = splits[1]\n",
335 | "\n",
336 | "print (\"Train Dataframe Row Count: \")\n",
337 | "print (train.count())\n",
338 | "print (\"Test Datafram Row Count: \")\n",
339 | "print (test.count())"
340 | ]
341 | },
342 | {
343 | "cell_type": "code",
344 | "execution_count": null,
345 | "metadata": {},
346 | "outputs": [],
347 | "source": [
348 | "rf = RandomForestClassifier(labelCol=\"label\", featuresCol=\"features\", numTrees=10)\n",
349 | "\n",
350 | "model = rf.fit(train)\n",
351 | "\n",
352 | "predictions = model.transform(test)\n",
353 | "#predictions.show()\n",
354 | "print (predictions.count())\n",
355 | "showDF(predictions)"
356 | ]
357 | },
358 | {
359 | "cell_type": "code",
360 | "execution_count": null,
361 | "metadata": {},
362 | "outputs": [],
363 | "source": [
364 | "showDF(predictions.select(\"quality\", \"label\", \"prediction\", \"probability\"))"
365 | ]
366 | },
367 | {
368 | "cell_type": "markdown",
369 | "metadata": {},
370 | "source": [
371 | "### We can now use the MutliclassClassifciationEvaluator to evalute the accurancy of our predictions. "
372 | ]
373 | },
374 | {
375 | "cell_type": "code",
376 | "execution_count": null,
377 | "metadata": {},
378 | "outputs": [],
379 | "source": [
380 | "# compute accuracy on the test set\n",
381 | "evaluator = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\",\n",
382 | " metricName=\"accuracy\")\n",
383 | "accuracy = evaluator.evaluate(predictions)\n",
384 | "print(\"Test set accuracy = \" + str(accuracy))"
385 | ]
386 | },
387 | {
388 | "cell_type": "code",
389 | "execution_count": null,
390 | "metadata": {
391 | "scrolled": true
392 | },
393 | "outputs": [],
394 | "source": [
395 | "session.execute(\"\"\"drop table wines\"\"\")"
396 | ]
397 | }
398 | ],
399 | "metadata": {
400 | "kernelspec": {
401 | "display_name": "Python 3",
402 | "language": "python",
403 | "name": "python3"
404 | },
405 | "language_info": {
406 | "codemirror_mode": {
407 | "name": "ipython",
408 | "version": 3
409 | },
410 | "file_extension": ".py",
411 | "mimetype": "text/x-python",
412 | "name": "python",
413 | "nbconvert_exporter": "python",
414 | "pygments_lexer": "ipython3",
415 | "version": "3.7.3"
416 | }
417 | },
418 | "nbformat": 4,
419 | "nbformat_minor": 2
420 | }
421 |
--------------------------------------------------------------------------------
/jupyter/Naivebayes.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Demo 2: Naive Bayes and DataStax Analytics\n",
8 | "------\n",
9 | "\n",
10 | "\n",
11 | "\n",
12 | "#### Dataset: https://archive.ics.uci.edu/ml/datasets/Wine+Quality"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "## What are we trying to learn from this dataset? \n",
20 | "\n",
21 | "# QUESTION: Can Naive Bayes be used to classify a wine’s rating score by its attributes?"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "%matplotlib inline\n",
31 | "import matplotlib.pyplot as plt"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "import pandas\n",
41 | "import cassandra\n",
42 | "import pyspark\n",
43 | "import re\n",
44 | "import os\n",
45 | "import random\n",
46 | "from random import randint, randrange\n",
47 | "import matplotlib.pyplot as plt\n",
48 | "from IPython.display import display, Markdown\n",
49 | "from pyspark.sql import SparkSession\n",
50 | "from pyspark.ml.classification import NaiveBayes\n",
51 | "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
52 | "from pyspark.ml.linalg import Vectors\n",
53 | "from pyspark.ml.feature import VectorAssembler\n",
54 | "from pyspark.ml.feature import StringIndexer"
55 | ]
56 | },
57 | {
58 | "cell_type": "markdown",
59 | "metadata": {},
60 | "source": [
61 | "#### Helper function to have nicer formatting of Spark DataFrames"
62 | ]
63 | },
64 | {
65 | "cell_type": "code",
66 | "execution_count": null,
67 | "metadata": {},
68 | "outputs": [],
69 | "source": [
70 | "#Helper for pretty formatting for Spark DataFrames\n",
71 | "def showDF(df, limitRows = 5, truncate = True):\n",
72 | " if(truncate):\n",
73 | " pandas.set_option('display.max_colwidth', 50)\n",
74 | " else:\n",
75 | " pandas.set_option('display.max_colwidth', -1)\n",
76 | " pandas.set_option('display.max_rows', limitRows)\n",
77 | " display(df.limit(limitRows).toPandas())\n",
78 | " pandas.reset_option('display.max_rows')"
79 | ]
80 | },
81 | {
82 | "cell_type": "markdown",
83 | "metadata": {},
84 | "source": [
85 | "![](\"images/dselogo.png\")
"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "metadata": {},
91 | "source": [
92 | "## Creating Tables and Loading Tables"
93 | ]
94 | },
95 | {
96 | "cell_type": "markdown",
97 | "metadata": {},
98 | "source": [
99 | "### Connect to Cassandra"
100 | ]
101 | },
102 | {
103 | "cell_type": "code",
104 | "execution_count": null,
105 | "metadata": {},
106 | "outputs": [],
107 | "source": [
108 | "from cassandra.cluster import Cluster\n",
109 | "\n",
110 | "cluster = Cluster(['dse'])\n",
111 | "session = cluster.connect()"
112 | ]
113 | },
114 | {
115 | "cell_type": "markdown",
116 | "metadata": {},
117 | "source": [
118 | "### Create Demo Keyspace "
119 | ]
120 | },
121 | {
122 | "cell_type": "code",
123 | "execution_count": null,
124 | "metadata": {
125 | "scrolled": true
126 | },
127 | "outputs": [],
128 | "source": [
129 | "session.execute(\"\"\"\n",
130 | " CREATE KEYSPACE IF NOT EXISTS accelerate \n",
131 | " WITH REPLICATION = \n",
132 | " { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }\"\"\"\n",
133 | ")"
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "### Set keyspace "
141 | ]
142 | },
143 | {
144 | "cell_type": "code",
145 | "execution_count": null,
146 | "metadata": {},
147 | "outputs": [],
148 | "source": [
149 | "session.set_keyspace('accelerate')"
150 | ]
151 | },
152 | {
153 | "cell_type": "markdown",
154 | "metadata": {},
155 | "source": [
156 | "### Create table called `wines`. Our PRIMARY will be a unique key (wineid) we generate for each row. This will have two datasets \"white\" and \"red\""
157 | ]
158 | },
159 | {
160 | "cell_type": "code",
161 | "execution_count": null,
162 | "metadata": {},
163 | "outputs": [],
164 | "source": [
165 | "query = \"CREATE TABLE IF NOT EXISTS wines \\\n",
166 | " (wineid int, fixedAcidity float, volatileAcidity float, citricAcid float, sugar float, \\\n",
167 | " chlorides float, freeSulfur float, totalSulfur float, density float, ph float, \\\n",
168 | " sulphates float, alcohol float, quality float, \\\n",
169 | " PRIMARY KEY (wineid))\"\n",
170 | "session.execute(query)"
171 | ]
172 | },
173 | {
174 | "cell_type": "markdown",
175 | "metadata": {},
176 | "source": [
177 | "### What do these of these 12 columns represent: \n",
178 | "\n",
179 | "* **Fixed acidity**\n",
180 | "* **Volatile acidity**\n",
181 | "* **Citric Acid**\n",
182 | "* **Residual Sugar** \n",
183 | "* **Chlorides**\n",
184 | "* **Free sulfur dioxide** \n",
185 | "* **Total sulfur dioxide**\n",
186 | "* **Density** \n",
187 | "* **pH**\n",
188 | "* **Sulphates**\n",
189 | "* **Alcohol**\n",
190 | "* **Quality**"
191 | ]
192 | },
193 | {
194 | "cell_type": "markdown",
195 | "metadata": {},
196 | "source": [
197 | "### Load 2 Wine Dataset -- White and Red\n",
198 | ""
199 | ]
200 | },
201 | {
202 | "cell_type": "markdown",
203 | "metadata": {},
204 | "source": [
205 | "### Load Wine datasets from CSV file (winequality-red.csv winequality-white.csv)\n",
206 | "* No clean up was requried! How nice :)\n",
207 | "\n",
208 | "#### Insert all the Wine Data into the DSE table `wines`"
209 | ]
210 | },
211 | {
212 | "cell_type": "code",
213 | "execution_count": null,
214 | "metadata": {},
215 | "outputs": [],
216 | "source": [
217 | "fileName = 'data/winequality-red.csv'\n",
218 | "input_file = open(fileName, 'r')\n",
219 | "i = 1\n",
220 | "for line in input_file:\n",
221 | " wineid = i\n",
222 | " row = line.split(';')\n",
223 | " \n",
224 | " query = \"INSERT INTO wines (wineid, fixedAcidity, volatileAcidity, citricAcid, sugar, \\\n",
225 | " chlorides, freeSulfur, totalSulfur, density, ph, \\\n",
226 | " sulphates, alcohol, quality)\"\n",
227 | " query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\"\n",
228 | " session.execute(query, (wineid, float(row[0]), float(row[1]), float(row[2]), float(row[3]), float(row[4]), float(row[5]), float(row[6]), float(row[7]), float(row[8]), float(row[9]), float(row[10]), float(row[11])))\n",
229 | " i = i + 1\n",
230 | "\n",
231 | "fileName = 'data/winequality-white.csv'\n",
232 | "input_file = open(fileName, 'r')\n",
233 | "\n",
234 | "for line in input_file:\n",
235 | " wineid = i\n",
236 | " row = line.split(';')\n",
237 | " \n",
238 | " query = \"INSERT INTO wines (wineid, fixedAcidity, volatileAcidity, citricAcid, sugar, \\\n",
239 | " chlorides, freeSulfur, totalSulfur, density, ph, \\\n",
240 | " sulphates, alcohol, quality)\"\n",
241 | " query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\"\n",
242 | " session.execute(query, (wineid, float(row[0]), float(row[1]), float(row[2]), float(row[3]), float(row[4]), float(row[5]), float(row[6]), float(row[7]), float(row[8]), float(row[9]), float(row[10]), float(row[11])))\n",
243 | " i = i + 1\n",
244 | " "
245 | ]
246 | },
247 | {
248 | "cell_type": "markdown",
249 | "metadata": {},
250 | "source": [
251 | "## Machine Learning with Apache Cassandra & Apache Spark\n",
252 | ""
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {},
258 | "source": [
259 | "#### Create a spark session that is connected to the database. From there load each table into a Spark Dataframe and take a count of the number of rows in each."
260 | ]
261 | },
262 | {
263 | "cell_type": "code",
264 | "execution_count": null,
265 | "metadata": {},
266 | "outputs": [],
267 | "source": [
268 | "spark = SparkSession.builder.appName('demo').master(\"local\").getOrCreate()\n",
269 | "\n",
270 | "wineDF = spark.read.format(\"org.apache.spark.sql.cassandra\").options(table=\"wines\", keyspace=\"accelerate\").load()\n",
271 | "\n",
272 | "print (\"Table Wine Row Count: \")\n",
273 | "print (wineDF.count())"
274 | ]
275 | },
276 | {
277 | "cell_type": "code",
278 | "execution_count": null,
279 | "metadata": {},
280 | "outputs": [],
281 | "source": [
282 | "showDF(wineDF)"
283 | ]
284 | },
285 | {
286 | "cell_type": "markdown",
287 | "metadata": {},
288 | "source": [
289 | "#### Let's filter out only wines that have been rated 6.0 or higher and create a new dataframe with that information "
290 | ]
291 | },
292 | {
293 | "cell_type": "code",
294 | "execution_count": null,
295 | "metadata": {},
296 | "outputs": [],
297 | "source": [
298 | "wine6DF = wineDF.filter(\"quality > 5\")\n",
299 | "showDF(wine6DF)"
300 | ]
301 | },
302 | {
303 | "cell_type": "markdown",
304 | "metadata": {},
305 | "source": [
306 | "#### Create Vector with all elements of the wine "
307 | ]
308 | },
309 | {
310 | "cell_type": "code",
311 | "execution_count": null,
312 | "metadata": {},
313 | "outputs": [],
314 | "source": [
315 | "assembler = VectorAssembler(\n",
316 | " inputCols=['alcohol', 'chlorides', 'citricacid', 'density', 'fixedacidity', 'ph', 'freesulfur', 'sugar', 'sulphates', 'totalsulfur', 'volatileacidity'],\n",
317 | " outputCol='features')\n",
318 | "\n",
319 | "trainingData = assembler.transform(wine6DF)\n",
320 | "\n",
321 | "labelIndexer = StringIndexer(inputCol=\"quality\", outputCol=\"label\", handleInvalid='keep')\n",
322 | "trainingData1 = labelIndexer.fit(trainingData).transform(trainingData)\n",
323 | "\n",
324 | "showDF(trainingData1)\n",
325 | "print(trainingData1.count())"
326 | ]
327 | },
328 | {
329 | "cell_type": "markdown",
330 | "metadata": {},
331 | "source": [
332 | "We need to split up our dataset in to a training and test set. Will split 80/20. "
333 | ]
334 | },
335 | {
336 | "cell_type": "code",
337 | "execution_count": null,
338 | "metadata": {},
339 | "outputs": [],
340 | "source": [
341 | "# Split the data into train and test\n",
342 | "splits = trainingData1.randomSplit([0.8, 0.2], 1234)\n",
343 | "train = splits[0]\n",
344 | "test = splits[1]\n",
345 | "\n",
346 | "print (\"Train Dataframe Row Count: \")\n",
347 | "print (train.count())\n",
348 | "print (\"Test Dataframe Row Count: \")\n",
349 | "print (test.count())"
350 | ]
351 | },
352 | {
353 | "cell_type": "markdown",
354 | "metadata": {},
355 | "source": [
356 | "### Now it's time to to use NaiveBayes. We will train the model, then use that model with out testing data to get our predictions. \n",
357 | "https://spark.apache.org/docs/2.2.0/ml-classification-regression.html#naive-bayes"
358 | ]
359 | },
360 | {
361 | "cell_type": "code",
362 | "execution_count": null,
363 | "metadata": {},
364 | "outputs": [],
365 | "source": [
366 | "nb = NaiveBayes(smoothing=1.0, modelType=\"multinomial\")\n",
367 | "\n",
368 | "# train the model\n",
369 | "model = nb.fit(train)\n",
370 | "\n",
371 | "predictions = model.transform(test)\n",
372 | "#predictions.show()\n",
373 | "print (predictions.count())\n",
374 | "showDF(predictions)"
375 | ]
376 | },
377 | {
378 | "cell_type": "code",
379 | "execution_count": null,
380 | "metadata": {},
381 | "outputs": [],
382 | "source": [
383 | "showDF(predictions.select(\"quality\", \"label\", \"prediction\", \"probability\"))"
384 | ]
385 | },
386 | {
387 | "cell_type": "markdown",
388 | "metadata": {},
389 | "source": [
390 | "### We can now use the MutliclassClassifciationEvaluator to evalute the accurancy of our predictions. "
391 | ]
392 | },
393 | {
394 | "cell_type": "code",
395 | "execution_count": null,
396 | "metadata": {},
397 | "outputs": [],
398 | "source": [
399 | "# compute accuracy on the test set\n",
400 | "evaluator = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\",\n",
401 | " metricName=\"accuracy\")\n",
402 | "accuracy = evaluator.evaluate(predictions)\n",
403 | "print(\"Test set accuracy = \" + str(accuracy))"
404 | ]
405 | }
406 | ],
407 | "metadata": {
408 | "kernelspec": {
409 | "display_name": "Python 3",
410 | "language": "python",
411 | "name": "python3"
412 | },
413 | "language_info": {
414 | "codemirror_mode": {
415 | "name": "ipython",
416 | "version": 3
417 | },
418 | "file_extension": ".py",
419 | "mimetype": "text/x-python",
420 | "name": "python",
421 | "nbconvert_exporter": "python",
422 | "pygments_lexer": "ipython3",
423 | "version": "3.7.3"
424 | }
425 | },
426 | "nbformat": 4,
427 | "nbformat_minor": 2
428 | }
429 |
--------------------------------------------------------------------------------
/jupyter/kmeans.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Demo 1: K-Means and DataStax Analytics\n",
8 | "------\n",
9 | "![](\"images/socialMedia.jpeg\")
\n",
10 | "\n",
11 | "\n",
12 | "#### Dataset: https://archive.ics.uci.edu/ml/datasets/Facebook+Live+Sellers+in+Thailand"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "## What are we trying to learn from this dataset? \n",
20 | "\n",
21 | "# QUESTION: Can K-Means be used to do social media analysis, can we group together different types of media by the reaction they received?\n"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "%matplotlib inline\n",
31 | "import matplotlib.pyplot as plt"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "import pandas\n",
41 | "import cassandra\n",
42 | "import pyspark\n",
43 | "import re\n",
44 | "import os\n",
45 | "import random\n",
46 | "from random import randint, randrange\n",
47 | "import matplotlib.pyplot as plt\n",
48 | "from IPython.display import display, Markdown\n",
49 | "from pyspark.sql import SparkSession\n",
50 | "from pyspark.ml.linalg import Vectors\n",
51 | "from pyspark.ml.feature import VectorAssembler\n",
52 | "from pyspark.ml.feature import StringIndexer\n",
53 | "from pyspark.ml.clustering import KMeans"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "metadata": {},
59 | "source": [
60 | "#### Helper function to have nicer formatting of Spark DataFrames"
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": null,
66 | "metadata": {},
67 | "outputs": [],
68 | "source": [
69 | "#Helper for pretty formatting for Spark DataFrames\n",
70 | "def showDF(df, limitRows = 5, truncate = True):\n",
71 | " if(truncate):\n",
72 | " pandas.set_option('display.max_colwidth', 50)\n",
73 | " else:\n",
74 | " pandas.set_option('display.max_colwidth', -1)\n",
75 | " pandas.set_option('display.max_rows', limitRows)\n",
76 | " display(df.limit(limitRows).toPandas())\n",
77 | " pandas.reset_option('display.max_rows')"
78 | ]
79 | },
80 | {
81 | "cell_type": "markdown",
82 | "metadata": {},
83 | "source": [
84 | ""
85 | ]
86 | },
87 | {
88 | "cell_type": "markdown",
89 | "metadata": {},
90 | "source": [
91 | "## Creating Tables and Loading Tables"
92 | ]
93 | },
94 | {
95 | "cell_type": "markdown",
96 | "metadata": {},
97 | "source": [
98 | "### Connect to Cassandra"
99 | ]
100 | },
101 | {
102 | "cell_type": "code",
103 | "execution_count": null,
104 | "metadata": {},
105 | "outputs": [],
106 | "source": [
107 | "from cassandra.cluster import Cluster\n",
108 | "\n",
109 | "cluster = Cluster(['dse'])\n",
110 | "session = cluster.connect()"
111 | ]
112 | },
113 | {
114 | "cell_type": "markdown",
115 | "metadata": {},
116 | "source": [
117 | "### Create Demo Keyspace "
118 | ]
119 | },
120 | {
121 | "cell_type": "code",
122 | "execution_count": null,
123 | "metadata": {},
124 | "outputs": [],
125 | "source": [
126 | "session.execute(\"\"\"\n",
127 | " CREATE KEYSPACE IF NOT EXISTS accelerate \n",
128 | " WITH REPLICATION = \n",
129 | " { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }\"\"\"\n",
130 | ")"
131 | ]
132 | },
133 | {
134 | "cell_type": "markdown",
135 | "metadata": {},
136 | "source": [
137 | "### Set keyspace "
138 | ]
139 | },
140 | {
141 | "cell_type": "code",
142 | "execution_count": null,
143 | "metadata": {},
144 | "outputs": [],
145 | "source": [
146 | "session.set_keyspace('accelerate')"
147 | ]
148 | },
149 | {
150 | "cell_type": "markdown",
151 | "metadata": {},
152 | "source": [
153 | "### Create table called `socialMedia`. Our PRIMARY will be a unique key (status_id) we generate for each row. "
154 | ]
155 | },
156 | {
157 | "cell_type": "code",
158 | "execution_count": null,
159 | "metadata": {},
160 | "outputs": [],
161 | "source": [
162 | "query = \"CREATE TABLE IF NOT EXISTS socialMedia \\\n",
163 | " (status_id int, social_type text, num_reactions int,\\\n",
164 | " num_comments int, num_shares int, num_likes int, num_loves int,\\\n",
165 | " num_wows int, num_hahas int, num_sads int, num_angrys int, \\\n",
166 | " PRIMARY KEY (status_id))\"\n",
167 | "session.execute(query)"
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "metadata": {},
173 | "source": [
174 | "### What do these of these 11 columns represent: \n",
175 | "\n",
176 | "* **Status_id**: Unique key created for each row\n",
177 | "\n",
178 | "* **Num Reactions**: \n",
179 | "\n",
180 | "* **Num Comments**:\n",
181 | "\n",
182 | "* **Num Shares**:\n",
183 | "\n",
184 | "* **Num Likes**:\n",
185 | "\n",
186 | "* **Num Loves**:\n",
187 | "\n",
188 | "* **Num Wows**:\n",
189 | "\n",
190 | "* **Num Hahas**:\n",
191 | "\n",
192 | "* **Num Sads**:\n",
193 | "\n",
194 | "* **Num Angrys**:\n",
195 | "\n",
196 | "* **Social Type**: Picture or Video\n"
197 | ]
198 | },
199 | {
200 | "cell_type": "markdown",
201 | "metadata": {},
202 | "source": [
203 | "### Load Social Media Dataset\n",
204 | "![](\"images/getTheLikes.png\")
"
205 | ]
206 | },
207 | {
208 | "cell_type": "markdown",
209 | "metadata": {},
210 | "source": [
211 | "### Load dataset from CSV file (socialMedia.csv)\n",
212 | "\n",
213 | "#### Insert all the Data into the Apache Cassandra table `socialmedia`"
214 | ]
215 | },
216 | {
217 | "cell_type": "code",
218 | "execution_count": null,
219 | "metadata": {},
220 | "outputs": [],
221 | "source": [
222 | "fileName = 'data/socialMedia.csv'\n",
223 | "input_file = open(fileName, 'r')\n",
224 | "i = 1\n",
225 | "for line in input_file:\n",
226 | " statusId = i\n",
227 | " row = line.split(',')\n",
228 | " \n",
229 | " if row[0] != \"photo\" and row[0] != \"video\":\n",
230 | " continue\n",
231 | "\n",
232 | " query = \"INSERT INTO socialmedia (status_id, social_type, num_reactions, num_comments, num_shares,\\\n",
233 | " num_likes, num_loves, num_wows, num_hahas, num_sads, num_angrys)\"\n",
234 | " query = query + \" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\"\n",
235 | " session.execute(query, (statusId, row[0], int(row[1]), int(row[2]), int(row[3]), int(row[4]), int(row[5]), int(row[6]), int(row[7]), int(row[8]), int(row[9])))\n",
236 | " i = i + 1\n",
237 | " "
238 | ]
239 | },
240 | {
241 | "cell_type": "markdown",
242 | "metadata": {},
243 | "source": [
244 | "## Machine Learning with Apache Spark\n",
245 | ""
246 | ]
247 | },
248 | {
249 | "cell_type": "markdown",
250 | "metadata": {},
251 | "source": [
252 | "#### Create a spark session that is connected to DSE. From there load each table into a Spark Dataframe and take a count of the number of rows in each."
253 | ]
254 | },
255 | {
256 | "cell_type": "code",
257 | "execution_count": null,
258 | "metadata": {},
259 | "outputs": [],
260 | "source": [
261 | "spark = SparkSession.builder.appName('demo').master(\"local\").getOrCreate()\n",
262 | "\n",
263 | "\n",
264 | "socialDF = spark.read.format(\"org.apache.spark.sql.cassandra\").options(table=\"socialmedia\", keyspace=\"accelerate\").load()\n",
265 | "\n",
266 | "print (\"Table Row Count: \")\n",
267 | "print (socialDF.count())"
268 | ]
269 | },
270 | {
271 | "cell_type": "code",
272 | "execution_count": null,
273 | "metadata": {},
274 | "outputs": [],
275 | "source": [
276 | "showDF(socialDF)"
277 | ]
278 | },
279 | {
280 | "cell_type": "markdown",
281 | "metadata": {},
282 | "source": [
283 | "#### When working with `STRING` data types you need to turn those `STRING` types into `FLOAT` types. Creating labels that K-MEANS and Apahce SPARK can understand."
284 | ]
285 | },
286 | {
287 | "cell_type": "code",
288 | "execution_count": null,
289 | "metadata": {},
290 | "outputs": [],
291 | "source": [
292 | "labelIndexer = StringIndexer(inputCol=\"social_type\", outputCol=\"label\", handleInvalid='keep')\n",
293 | "training = labelIndexer.fit(socialDF).transform(socialDF)\n",
294 | "\n",
295 | "showDF(training)\n"
296 | ]
297 | },
298 | {
299 | "cell_type": "code",
300 | "execution_count": null,
301 | "metadata": {},
302 | "outputs": [],
303 | "source": [
304 | "showDF(training.select(\"social_type\",\"label\"))"
305 | ]
306 | },
307 | {
308 | "cell_type": "code",
309 | "execution_count": null,
310 | "metadata": {},
311 | "outputs": [],
312 | "source": [
313 | "training.groupBy('social_type').count().show()"
314 | ]
315 | },
316 | {
317 | "cell_type": "markdown",
318 | "metadata": {},
319 | "source": [
320 | "## Let's visualize this data with a scatter plot \n",
321 | "### The x axis will be number of likes \n",
322 | "### The y axis will be number of comments\n",
323 | "### The color of the dot will be assigned based on its \"cluster\" Photo or Video\n",
324 | "\n",
325 | "Note: These attributes are what might be a strong attributes to finding clusters (Photo - Video)\n",
326 | "Note 1: Must move to a Pandas dataframe to do this visualization (be aware! This can't always be done as is, depends on your data size)"
327 | ]
328 | },
329 | {
330 | "cell_type": "code",
331 | "execution_count": null,
332 | "metadata": {},
333 | "outputs": [],
334 | "source": [
335 | "smPanda = training.toPandas()\n",
336 | "smPanda.plot.scatter(x = 'num_likes', y = 'num_comments', c= 'label', figsize=(12,8), colormap='viridis')"
337 | ]
338 | },
339 | {
340 | "cell_type": "markdown",
341 | "metadata": {},
342 | "source": [
343 | "### Two clusters here Yellow = Video and Purple = Pictures\n",
344 | "\n",
345 | "From what we can see from these two attributes Videos get less likes but more comments. Pictures get less comments but more likes. "
346 | ]
347 | },
348 | {
349 | "cell_type": "markdown",
350 | "metadata": {},
351 | "source": [
352 | "## Let's see if Kmeans can give us the same clustering\n",
353 | "\n",
354 | "## K-means clustering is a simple unsupervised learning algorithm that is used to solve clustering problems. Kmeans is very simple, but very powerful even on large datasets. It requires that all the input columns be vectorized. \n",
355 | "\n",
356 | "https://spark.apache.org/docs/latest/ml-features.html#vectorassembler"
357 | ]
358 | },
359 | {
360 | "cell_type": "code",
361 | "execution_count": null,
362 | "metadata": {},
363 | "outputs": [],
364 | "source": [
365 | "assembler = VectorAssembler(\n",
366 | " inputCols=[ 'num_likes', 'num_comments'],\n",
367 | " outputCol='features')\n",
368 | "\n",
369 | "trainingData = assembler.transform(training)"
370 | ]
371 | },
372 | {
373 | "cell_type": "markdown",
374 | "metadata": {},
375 | "source": [
376 | "### We need to set the K for KMeans which we will set at 2. One of the downsides of unsupervised learning is that we normally will not have clusteres predefinded (like we do in this case). Kmeans will happily split the data into as many clusters as you set. \n",
377 | "\n",
378 | "#### First we will generate the model and then make predictions based on that model "
379 | ]
380 | },
381 | {
382 | "cell_type": "code",
383 | "execution_count": null,
384 | "metadata": {},
385 | "outputs": [],
386 | "source": [
387 | "kmeans = KMeans().setK(2).setSeed(1)\n",
388 | "model = kmeans.fit(trainingData)\n",
389 | "\n",
390 | "# Make predictions\n",
391 | "predictions = model.transform(trainingData)\n",
392 | "\n",
393 | "showDF(predictions)"
394 | ]
395 | },
396 | {
397 | "cell_type": "markdown",
398 | "metadata": {},
399 | "source": [
400 | "## In this case because we are actually preforming surpervised learnings (since we do have the cluster labels) we can do some comparisions to see if our predictions are correct. \n",
401 | "\n",
402 | "## In this case I am just taking a look at the counts for each group. "
403 | ]
404 | },
405 | {
406 | "cell_type": "code",
407 | "execution_count": null,
408 | "metadata": {},
409 | "outputs": [],
410 | "source": [
411 | "predictions.groupBy('prediction').count().show()\n",
412 | "training.groupBy('social_type').count().show()"
413 | ]
414 | },
415 | {
416 | "cell_type": "markdown",
417 | "metadata": {},
418 | "source": [
419 | "## Let's create another scatter plot to see if this lines up with our orignal scatter plot. \n",
420 | "\n",
421 | "## Everything is the same except now our dots will represent the color of the prediction (instead of the orginal cluster)"
422 | ]
423 | },
424 | {
425 | "cell_type": "code",
426 | "execution_count": null,
427 | "metadata": {},
428 | "outputs": [],
429 | "source": [
430 | "car_df = predictions.toPandas()\n",
431 | "\n",
432 | "car_df.plot.scatter(x = 'num_likes', y = 'num_comments', c= 'prediction', figsize=(12,8), colormap='viridis')"
433 | ]
434 | },
435 | {
436 | "cell_type": "markdown",
437 | "metadata": {},
438 | "source": [
439 | "### Videos are represented in yellow and pictures are represnted in purple."
440 | ]
441 | },
442 | {
443 | "cell_type": "markdown",
444 | "metadata": {},
445 | "source": [
446 | "## KMeans struggles when you add many variables, so adding more variables is unlikely to help. "
447 | ]
448 | },
449 | {
450 | "cell_type": "markdown",
451 | "metadata": {},
452 | "source": [
453 | "# Remember Data Science and analytics is an iterative process! It's a science! Hypothesis, test, analysis, and loop again! "
454 | ]
455 | }
456 | ],
457 | "metadata": {
458 | "kernelspec": {
459 | "display_name": "Python 3",
460 | "language": "python",
461 | "name": "python3"
462 | },
463 | "language_info": {
464 | "codemirror_mode": {
465 | "name": "ipython",
466 | "version": 3
467 | },
468 | "file_extension": ".py",
469 | "mimetype": "text/x-python",
470 | "name": "python",
471 | "nbconvert_exporter": "python",
472 | "pygments_lexer": "ipython3",
473 | "version": "3.7.3"
474 | }
475 | },
476 | "nbformat": 4,
477 | "nbformat_minor": 2
478 | }
479 |
--------------------------------------------------------------------------------
/jupyter/FP-Growth.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Demo 4: FP-Growth and DataStax Analytics\n",
8 | "------\n",
9 | "![](\"images/pixarMovies.jpg\")
\n",
10 | "\n",
11 | "\n",
12 | "#### Dataset: https://grouplens.org/datasets/movielens/"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "## What are we trying to learn from this dataset? \n",
20 | "\n",
21 | "# QUESTION: Can FP-Growth be used to find which movies to recommend to our users?\n"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "%matplotlib inline\n",
31 | "import matplotlib.pyplot as plt"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "import pandas\n",
41 | "import cassandra\n",
42 | "import pyspark\n",
43 | "import re\n",
44 | "import os\n",
45 | "import random\n",
46 | "import matplotlib.pyplot as plt\n",
47 | "from random import randint, randrange\n",
48 | "from IPython.display import display, Markdown\n",
49 | "from pyspark.sql import SparkSession\n",
50 | "from pyspark.ml.fpm import FPGrowth\n",
51 | "from pyspark.sql import Row\n",
52 | "from pyspark.sql.functions import collect_set\n",
53 | "from pyspark.ml.feature import IndexToString, StringIndexer, VectorAssembler\n",
54 | "from pyspark.ml.evaluation import MulticlassClassificationEvaluator"
55 | ]
56 | },
57 | {
58 | "cell_type": "code",
59 | "execution_count": null,
60 | "metadata": {},
61 | "outputs": [],
62 | "source": [
63 | "#Helper for pretty formatting for Spark DataFrames\n",
64 | "def showDF(df, limitRows = 5, truncate = True):\n",
65 | " if(truncate):\n",
66 | " pandas.set_option('display.max_colwidth', 50)\n",
67 | " else:\n",
68 | " pandas.set_option('display.max_colwidth', -1)\n",
69 | " pandas.set_option('display.max_rows', limitRows)\n",
70 | " display(df.limit(limitRows).toPandas())\n",
71 | " pandas.reset_option('display.max_rows')"
72 | ]
73 | },
74 | {
75 | "cell_type": "markdown",
76 | "metadata": {},
77 | "source": [
78 | "#### Helper function to have nicer formatting of Spark DataFrames"
79 | ]
80 | },
81 | {
82 | "cell_type": "markdown",
83 | "metadata": {},
84 | "source": [
85 | "## Creating Tables and Loading Tables"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "metadata": {},
91 | "source": [
92 | ""
93 | ]
94 | },
95 | {
96 | "cell_type": "code",
97 | "execution_count": null,
98 | "metadata": {},
99 | "outputs": [],
100 | "source": [
101 | "from cassandra.cluster import Cluster\n",
102 | "\n",
103 | "cluster = Cluster(['dse'])\n",
104 | "session = cluster.connect()"
105 | ]
106 | },
107 | {
108 | "cell_type": "markdown",
109 | "metadata": {},
110 | "source": [
111 | "### Create Demo Keyspace "
112 | ]
113 | },
114 | {
115 | "cell_type": "code",
116 | "execution_count": null,
117 | "metadata": {},
118 | "outputs": [],
119 | "source": [
120 | "session.execute(\"\"\"\n",
121 | " CREATE KEYSPACE IF NOT EXISTS accelerate \n",
122 | " WITH REPLICATION = \n",
123 | " { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }\"\"\")"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "metadata": {},
129 | "source": [
130 | "### Set keyspace "
131 | ]
132 | },
133 | {
134 | "cell_type": "code",
135 | "execution_count": null,
136 | "metadata": {},
137 | "outputs": [],
138 | "source": [
139 | "session.set_keyspace('accelerate')"
140 | ]
141 | },
142 | {
143 | "cell_type": "markdown",
144 | "metadata": {},
145 | "source": [
146 | "### Create table called `movies`. Our PRIMARY will be a unique key (movieid) "
147 | ]
148 | },
149 | {
150 | "cell_type": "code",
151 | "execution_count": null,
152 | "metadata": {},
153 | "outputs": [],
154 | "source": [
155 | "query = \"CREATE TABLE IF NOT EXISTS movies \\\n",
156 | " (movieid int, title text, genres text, \\\n",
157 | " PRIMARY KEY (movieid))\"\n",
158 | "session.execute(query)"
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "metadata": {},
164 | "source": [
165 | "### Create table called `movieRatings`. Our PRIMARY key will be a compositite key (userid, movieid)."
166 | ]
167 | },
168 | {
169 | "cell_type": "code",
170 | "execution_count": null,
171 | "metadata": {},
172 | "outputs": [],
173 | "source": [
174 | "query = \"CREATE TABLE IF NOT EXISTS movieratings \\\n",
175 | " (userid int, movieid int, rating float, timestamp text, \\\n",
176 | " PRIMARY KEY (userid, movieid))\"\n",
177 | "session.execute(query)"
178 | ]
179 | },
180 | {
181 | "cell_type": "markdown",
182 | "metadata": {},
183 | "source": [
184 | "## Movies\n",
185 | "\n",
186 | "* **Movieid**\n",
187 | "* **Title**\n",
188 | "* **Genres**\n",
189 | "\n",
190 | "## Movie Ratings Table\n",
191 | "### What do these of these 4 columns represent:\n",
192 | "\n",
193 | "* **UserId**\n",
194 | "* **MovieId**\n",
195 | "* **Rating**\n",
196 | "* **Timestamp**"
197 | ]
198 | },
199 | {
200 | "cell_type": "markdown",
201 | "metadata": {},
202 | "source": [
203 | "### Load 2 Movie Dataset -- Movies and Movie Ratings Table\n",
204 | "![](\"images/bttf3.jpg\")
"
205 | ]
206 | },
207 | {
208 | "cell_type": "markdown",
209 | "metadata": {},
210 | "source": [
211 | "### Load Movie datasets from CSV file (rating_movies.csv, movies.csv)\n",
212 | "* No clean up was requried! How nice :)\n",
213 | "\n",
214 | "#### Insert all the Movie Data into the table `movies` and `movieratings`"
215 | ]
216 | },
217 | {
218 | "cell_type": "code",
219 | "execution_count": null,
220 | "metadata": {},
221 | "outputs": [],
222 | "source": [
223 | "fileName = 'data/ratings.csv'\n",
224 | "input_file = open(fileName, 'r')\n",
225 | "\n",
226 | "for line in input_file:\n",
227 | " row = line.split(',')\n",
228 | " \n",
229 | " query = \"INSERT INTO movieratings (userid, movieid, rating, timestamp)\"\n",
230 | " query = query + \" VALUES (%s, %s, %s, %s)\"\n",
231 | " session.execute(query, (int(row[0]), int(row[1]), float(row[2]), row[3]))"
232 | ]
233 | },
234 | {
235 | "cell_type": "code",
236 | "execution_count": null,
237 | "metadata": {},
238 | "outputs": [],
239 | "source": [
240 | "fileName = 'data/movies.csv'\n",
241 | "input_file = open(fileName, 'r')\n",
242 | "\n",
243 | "for line in input_file:\n",
244 | " row = line.split(',')\n",
245 | " \n",
246 | " query = \"INSERT INTO movies (movieid, title, genres)\"\n",
247 | " query = query + \" VALUES (%s, %s, %s)\"\n",
248 | " session.execute(query, (int(row[0]), row[1], row[2]))"
249 | ]
250 | },
251 | {
252 | "cell_type": "markdown",
253 | "metadata": {},
254 | "source": [
255 | "## Machine Learning with Apache Spark\n",
256 | ""
257 | ]
258 | },
259 | {
260 | "cell_type": "markdown",
261 | "metadata": {},
262 | "source": [
263 | "#### Create a spark session that is connected to cassandra. From there load each table into a Spark Dataframe and take a count of the number of rows in each."
264 | ]
265 | },
266 | {
267 | "cell_type": "code",
268 | "execution_count": null,
269 | "metadata": {},
270 | "outputs": [],
271 | "source": [
272 | "spark = SparkSession.builder.appName('demo').master(\"local\").getOrCreate()\n",
273 | "\n",
274 | "movieDF = spark.read.format(\"org.apache.spark.sql.cassandra\").options(table=\"movieratings\", keyspace=\"accelerate\").load()\n",
275 | "\n",
276 | "print (\"Table Row Count: \")\n",
277 | "print (movieDF.count())"
278 | ]
279 | },
280 | {
281 | "cell_type": "code",
282 | "execution_count": null,
283 | "metadata": {},
284 | "outputs": [],
285 | "source": [
286 | "showDF(movieDF)"
287 | ]
288 | },
289 | {
290 | "cell_type": "markdown",
291 | "metadata": {},
292 | "source": [
293 | "## This dataset is not in the format we need it to be. We need it to be more in a transaction format. Each user and the list of movies they have reviewed. "
294 | ]
295 | },
296 | {
297 | "cell_type": "markdown",
298 | "metadata": {},
299 | "source": [
300 | "#### Remove the timestamp column since we will not be using that"
301 | ]
302 | },
303 | {
304 | "cell_type": "code",
305 | "execution_count": null,
306 | "metadata": {},
307 | "outputs": [],
308 | "source": [
309 | "newMovieDF = movieDF.drop('timestamp')\n",
310 | "showDF(newMovieDF)"
311 | ]
312 | },
313 | {
314 | "cell_type": "markdown",
315 | "metadata": {},
316 | "source": [
317 | "#### Before we collect the set of movies for each user let's filter out any movies they rated below 3"
318 | ]
319 | },
320 | {
321 | "cell_type": "code",
322 | "execution_count": null,
323 | "metadata": {},
324 | "outputs": [],
325 | "source": [
326 | "newestMovies = newMovieDF.filter(\"rating > 3\")\n",
327 | "showDF(newestMovies)"
328 | ]
329 | },
330 | {
331 | "cell_type": "markdown",
332 | "metadata": {},
333 | "source": [
334 | "#### GroupBy the user id and create a collection set of all the movies they have rated and seen. "
335 | ]
336 | },
337 | {
338 | "cell_type": "code",
339 | "execution_count": null,
340 | "metadata": {},
341 | "outputs": [],
342 | "source": [
343 | "group_user = newestMovies.groupBy('userid').agg(collect_set('movieid').alias('moviesRated'))\n",
344 | "group_user.show()\n"
345 | ]
346 | },
347 | {
348 | "cell_type": "markdown",
349 | "metadata": {},
350 | "source": [
351 | "#### For FP-Growth the list needs to be a column named `items`"
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "execution_count": null,
357 | "metadata": {},
358 | "outputs": [],
359 | "source": [
360 | "df = group_user.withColumnRenamed(\"moviesRated\", \"items\")\n",
361 | "showDF(df)"
362 | ]
363 | },
364 | {
365 | "cell_type": "code",
366 | "execution_count": null,
367 | "metadata": {},
368 | "outputs": [],
369 | "source": [
370 | "df.select('userid').distinct().count()"
371 | ]
372 | },
373 | {
374 | "cell_type": "markdown",
375 | "metadata": {},
376 | "source": [
377 | "### FPGROWTH for Recommendations\n",
378 | "#### Use Apache Spark MLlib with FPGrowth to find Recommendation \n",
379 | "#### https://spark.apache.org/docs/latest/ml-frequent-pattern-mining.html\n",
380 | "#### https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.fpm.FPGrowth"
381 | ]
382 | },
383 | {
384 | "cell_type": "code",
385 | "execution_count": null,
386 | "metadata": {},
387 | "outputs": [],
388 | "source": [
389 | "fpGrowth = FPGrowth(itemsCol=\"items\", minSupport=0.1, minConfidence=0.2)\n",
390 | "model = fpGrowth.fit(df)\n",
391 | "recommendDF=model.transform(df)\n",
392 | "recommendDF.show()"
393 | ]
394 | },
395 | {
396 | "cell_type": "markdown",
397 | "metadata": {},
398 | "source": [
399 | "#### If you have watched these movies `antecedent` then you will like this movie `consquent`"
400 | ]
401 | },
402 | {
403 | "cell_type": "code",
404 | "execution_count": null,
405 | "metadata": {},
406 | "outputs": [],
407 | "source": [
408 | "# Display frequent itemsets.\n",
409 | "#model.freqItemsets.show()\n",
410 | "\n",
411 | "# Display generated association rules.\n",
412 | "dfAssociation = model.associationRules\n",
413 | "\n",
414 | "dfAssociation.show()"
415 | ]
416 | },
417 | {
418 | "cell_type": "code",
419 | "execution_count": null,
420 | "metadata": {},
421 | "outputs": [],
422 | "source": [
423 | "print(\"If you like these movies: \")\n",
424 | "print(list(dfAssociation.select('antecedent').first()))\n",
425 | "print(\"Then you will like this movie:\")\n",
426 | "print(list(dfAssociation.select('consequent').first()))\n",
427 | "\n",
428 | "movieYoulike = list(dfAssociation.select('antecedent').first())\n",
429 | "movieToRecommend=list(dfAssociation.select('consequent').first())"
430 | ]
431 | },
432 | {
433 | "cell_type": "markdown",
434 | "metadata": {},
435 | "source": [
436 | "#### Query database to get movie titles"
437 | ]
438 | },
439 | {
440 | "cell_type": "code",
441 | "execution_count": null,
442 | "metadata": {},
443 | "outputs": [],
444 | "source": [
445 | "query = \"select title from movies WHERE movieid=\"\n",
446 | "query = query + str(movieYoulike[0][0])\n",
447 | "\n",
448 | "rows = session.execute(query)\n",
449 | "print(rows)\n",
450 | "\n",
451 | "for user_row in rows:\n",
452 | " print (user_row.title)\n",
453 | "\n",
454 | "query = \"select title from movies WHERE movieid=\"\n",
455 | "query = query + str(movieYoulike[0][1])\n",
456 | "\n",
457 | "rows = session.execute(query)\n",
458 | "print(rows)\n",
459 | "\n",
460 | "for user_row in rows:\n",
461 | " print (user_row.title)"
462 | ]
463 | },
464 | {
465 | "cell_type": "markdown",
466 | "metadata": {},
467 | "source": [
468 | "#### Then you will like this movie ... "
469 | ]
470 | },
471 | {
472 | "cell_type": "code",
473 | "execution_count": null,
474 | "metadata": {},
475 | "outputs": [],
476 | "source": [
477 | "query = \"select title from movies WHERE movieid=\"\n",
478 | "query = query + str(movieToRecommend[0][0])\n",
479 | "\n",
480 | "rows = session.execute(query)\n",
481 | "print(rows)\n",
482 | "\n",
483 | "for user_row in rows:\n",
484 | " print (user_row.title)"
485 | ]
486 | },
487 | {
488 | "cell_type": "code",
489 | "execution_count": null,
490 | "metadata": {},
491 | "outputs": [],
492 | "source": [
493 | "session.execute(\"\"\"drop table movies\"\"\")\n",
494 | "session.execute(\"\"\"drop table movieratings\"\"\")"
495 | ]
496 | }
497 | ],
498 | "metadata": {
499 | "kernelspec": {
500 | "display_name": "Python 3",
501 | "language": "python",
502 | "name": "python3"
503 | },
504 | "language_info": {
505 | "codemirror_mode": {
506 | "name": "ipython",
507 | "version": 3
508 | },
509 | "file_extension": ".py",
510 | "mimetype": "text/x-python",
511 | "name": "python",
512 | "nbconvert_exporter": "python",
513 | "pygments_lexer": "ipython3",
514 | "version": "3.7.3"
515 | }
516 | },
517 | "nbformat": 4,
518 | "nbformat_minor": 2
519 | }
520 |
--------------------------------------------------------------------------------
/dse-studio/notebooks/3a0a2fc4-d7d7-40ba-9438-8d672da3c1d2:
--------------------------------------------------------------------------------
1 | json_notebook_v1 {"1":"07494f4b-83e1-4a0b-a12d-b8589166b006","10":"418ed742-cd61-4df8-abd8-07bc56a62e8d","11":"00 - Getting Started with Apache Cassandra™","12":{"1":1570518298,"2":431000000},"13":{"1":1570527776,"2":395000000},"14":false,"15":[{"1":"e3ccc89a-d2e4-4fc6-a032-ffe3e45bab15","10":4,"11":"\n
Welcome to the First Touch notebook on Cassandra. In this notebook we will:
\nThis notebook is a step-by-step tutorial.\n
Each step has a brief discussion section in a cell that describes the step, followed by an exercise in a second cell, where you get to try something out.\n
In some exercises, we'll tell you exactly what to do.\n
In others, we may let you figure it out on your own, because that's a lot more fun than always being told exactly what to do.
But don't worry. None of these exercises requires you to do anything as difficult as beat Deep Blue at chess. ![\"Smiley](\"https://s3.amazonaws.com/datastaxtraining/solution-days/cassandra-intro/smileyface.png\" "\\"Smiley")
\n
We're confident you'll be able to work though each of the exercises in a way that will be educational and fun.
Let's get started!
\n"},"11":4,"12":false},"16":true,"17":true,"18":{},"25":"CL.ONE"},{"1":"66bedac8-5c03-446b-ae58-44068714c842","10":4,"11":"On next bloc locate the `CL.ONE` on the top right end corner and click on the triangle to execute. This will create object we will need for later\n\nYou shoulg get message `Success - No Data Returned`","12":"markdown","13":{"1":"ef3db0dc-8aad-4abd-9b0f-90dff2c9cfd6","10":{"9":"On next bloc locate the CL.ONE on the top right end corner and click on the triangle to execute. This will create object we will need for later
You shoulg get message Success - No Data Returned
If you are seeing this notebook, that means that you have access to a cluster.\n
We have set up the cluster and have already installed some data on it for you.
Let's start by investigating the keyspaces that your cluster already has.\n
Remember, a “keyspace” is what the relational world calls a “database”.
To use Studio's tool to see the schemas, click on the button in the top-right corner of your browser window:\n
![\"schema](\"https://s3.amazonaws.com/datastaxtraining/developer-day/cassandra-intro/data-modeling/schema.png\" "\\"schema")
You see that you are connected to the Cassandra Day Cluster, but no schema is selected.\n
Click the Schema dropdown, select CQL and select the killrvideo schema.\n
![\"schema](\"https://s3.amazonaws.com/datastaxtraining/solution-days/cassandra-intro/FirstTouch/SchemaDropdown.png\" "\\"schema")
This should allow you to see the killrvideo schema as shown here:\n
![\"table](\"https://s3.amazonaws.com/datastaxtraining/solution-days/cassandra-intro/FirstTouch/TableNames.png\" "\\"table")
You can check out the users table by expandning it.\n
You can also see the columns in the table and its primary key by expanding those sections:\n
![\"expanding](\"https://s3.amazonaws.com/datastaxtraining/solution-days/cassandra-intro/FirstTouch/UserTable.png\" "\\"expanding")
In this step, let's see how we can look at the data in the users table.\n
We'll use a simple CQL command that retrieves all the data in the table.\n
Don't worry, Studio will paginate the results for us.
In the following cell, try this command: SELECT * FROM killrvideo.users;
Pro Tip: You would never want to execute this command in a true production environment because it would have significant performance implications. We use the command here for illustrative purposes only - to give you a gentle introduction to CQL.
\n"},"11":4,"12":false},"16":true,"17":true,"18":{},"25":"CL.ONE"},{"1":"b88cf06c-c0ae-4f73-9652-387d9f237e72","10":0,"11":"// Enter the SELECT command here followed by shift+enter or pressing the CL.ONE button:\n","12":"cql","16":true,"17":false,"25":"CL.ONE"},{"1":"127dbc38-9550-4a0b-9bdb-94ab5a31963a","10":4,"11":"![line](https://datastaxtraining.s3.amazonaws.com/developer-day/lineblue.png \"line\" )\n###
In the results from the previous step, you see that the columns correspond with the columns we saw when we investigated the killrvideo schema.\n
The first column, the index, of course isn't part of the schema, it's just the number of the column in the results.\n
You may have also noticed that CQL looks a lot like SQL.\n
This syntactical similarity makes it a little more intuitive to use CQL, assuming you are already familiar with SQL.\n
For example, if we don't want to get all the columns back, we can project only those columns we want, in the order we want - just like SQL.
Can you guess how you might select only the firstname, lastname and userid columns?\n
Try it in the next cell:
You may have noticed in the schema, that the userid column is the partition key.\n
A partition key is the key we use to retrieve a partition, which consists of one or more rows.\n
In the users table, the partition key corresponds to a single row, but in the comments_by_user table, a partition may consist of several rows - all the comments made by the specified user.
Let's try a query and see how that works.\n
Let's retrieve all the comments made by Corly Joddins (the first result in the previous query).
SPOILER ALERT: the rest of this cell explains how to do this, so if you think you want to try it on your own, skip to the next cell without expanding.
\n\nClick here for the explanation.
\n
Once again, CQL looks a lot like SQL - specifically, you can use a WHERE clause to identify the specific partition key.\n
The WHERE clause would be WHERE userid = <put the userid here> and goes right after the name of the table and before the final ;.\n
You don't add any quotes or anything around the userid value, just copy it from the previous query results and past it in the query.\n
It appears that Corly creates all the comments in Latin, but you get the idea.\n
Remember, a partition contains one or more rows, which allows you to create queries that can find all your results with a single access - and do it very quickly!
Now you have an intuitive feel for what Apache Cassandra™ can do.
\nLet's review what you have learned:
\nWOW! That's a great start!
\n