Welcome! This notebook will provide examples of using Db2 Magic Commands in a Jypyter Notebook in CPD and Watson Studio. Db2 magic commands are quite useful if you want to use a notebook for a sql editor. However to execute the prerequisite db2ipynb noteboo, you need to do some things a bit differently than you do in your own Jupyter environment. This notebook was created in a CPD 2.5 cluster and the examples connect to the DB2Warehouse database.
"
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {},
27 | "source": [
28 | "Db2 Magic commands make it easy to run SQL in cells in your Jupyter Notebook as is without any Python code. This notebook shows how to enable this feature in Cloud Pak for Data. Using these examples would probably allow you to use them in Watson Studio outside of CPD as well. I only show some basic Magic Commands here. For more information on Db2 Magic Commands and lots of example see https://github.com/IBM/db2-jupyter/"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {},
34 | "source": [
35 | "
Get the db2 magic commands notebook from github: db2.ipynb
"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "You must get and run the db2.iyndb notebook before using the magic commands. You can't run the db2.ipynb notebook from the directory shown above. Instead you need to reference the \"raw\" version as is done in the command below. I got that raw URL by going into directory shown in the above \"About\" description, then clicking on the db2.ipynb link, and then clicking the \"Raw\" button near that top of the notebook itself. Finally I copied the URL in the browser and pasted it in the wget command. "
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": null,
48 | "metadata": {},
49 | "outputs": [],
50 | "source": [
51 | "!wget https://raw.githubusercontent.com/IBM/db2-jupyter/master/db2.ipynb"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "metadata": {},
57 | "source": [
58 | "### Run the db2 magic comands notebook"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "execution_count": null,
64 | "metadata": {},
65 | "outputs": [],
66 | "source": [
67 | "%run db2.ipynb"
68 | ]
69 | },
70 | {
71 | "cell_type": "markdown",
72 | "metadata": {},
73 | "source": [
74 | "### Connect to a Db2 Warehouse or other Db2 database\n",
75 | "User change this string to specify your own database, database user_id and password, host and port. "
76 | ]
77 | },
78 | {
79 | "cell_type": "code",
80 | "execution_count": null,
81 | "metadata": {},
82 | "outputs": [],
83 | "source": [
84 | "%sql CONNECT TO bludb USER user1013 USING ppppppp HOST host.or.ip PORT 50000;"
85 | ]
86 | },
87 | {
88 | "cell_type": "markdown",
89 | "metadata": {},
90 | "source": [
91 | "##### Run a select command"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {},
98 | "outputs": [],
99 | "source": [
100 | "%sql select count(*) from syscat.indexes"
101 | ]
102 | },
103 | {
104 | "cell_type": "markdown",
105 | "metadata": {},
106 | "source": [
107 | "##### Terminate the connection to Db2W"
108 | ]
109 | },
110 | {
111 | "cell_type": "code",
112 | "execution_count": null,
113 | "metadata": {},
114 | "outputs": [],
115 | "source": [
116 | "%sql connect reset"
117 | ]
118 | }
119 | ],
120 | "metadata": {
121 | "anaconda-cloud": {},
122 | "kernelspec": {
123 | "display_name": "Python 3",
124 | "language": "python",
125 | "name": "python3"
126 | },
127 | "language_info": {
128 | "codemirror_mode": {
129 | "name": "ipython",
130 | "version": 3
131 | },
132 | "file_extension": ".py",
133 | "mimetype": "text/x-python",
134 | "name": "python",
135 | "nbconvert_exporter": "python",
136 | "pygments_lexer": "ipython3",
137 | "version": "3.7.3"
138 | }
139 | },
140 | "nbformat": 4,
141 | "nbformat_minor": 2
142 | }
143 |
--------------------------------------------------------------------------------
/Db2_11.1_Features/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_11.1_Features/Db2V11-ebook.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.1_Features/Db2V11-ebook.pdf
--------------------------------------------------------------------------------
/Db2_11.5_Features/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_11.5_Features/Db2V11-JSON-ebook.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/Db2V11-JSON-ebook.pdf
--------------------------------------------------------------------------------
/Db2_11.5_Features/Db2_11.5_JSON_05_Inserting_JSON_Data.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Storing JSON Documents in Db2\n",
8 | "Updated: 2019-09-14"
9 | ]
10 | },
11 | {
12 | "cell_type": "markdown",
13 | "metadata": {},
14 | "source": [
15 | "### Load Db2 Extensions and Connect to the Database\n",
16 | "The `connection` notebook contains the `CONNECT` statement which allows access to the `SAMPLE` database. If you need to modify the connection information, edit the `connection.ipynb` notebook."
17 | ]
18 | },
19 | {
20 | "cell_type": "code",
21 | "execution_count": null,
22 | "metadata": {},
23 | "outputs": [],
24 | "source": [
25 | "%run ../db2.ipynb\n",
26 | "%run ../connection.ipynb"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "metadata": {},
32 | "source": [
33 | "## Inserting and Retrieving JSON Documents\n",
34 | "Inserting a JSON value into a Db2 table can be done through a variety of methods including `LOAD`. In the previous section, the Db2 `IMPORT` command was used to move character JSON data into a table. If the Db2 column has been defined as a character field, you can use the `INSERT` statement without any additional modification."
35 | ]
36 | },
37 | {
38 | "cell_type": "code",
39 | "execution_count": null,
40 | "metadata": {},
41 | "outputs": [],
42 | "source": [
43 | "%%sql -q\n",
44 | "DROP TABLE CUSTOMERS;\n",
45 | "CREATE TABLE CUSTOMERS \n",
46 | "(\n",
47 | " CUSTOMER_ID INT,\n",
48 | " CUSTOMER_INFO VARCHAR(2000)\n",
49 | ");\n",
50 | "\n",
51 | "INSERT INTO CUSTOMERS VALUES \n",
52 | "(\n",
53 | " 1,\n",
54 | " '{\"customerid\": 100001,\n",
55 | " \"identity\": \n",
56 | " {\n",
57 | " \"firstname\": \"Kelly\",\n",
58 | " \"lastname\" : \"Gilmore\",\n",
59 | " \"birthdate\": \"1973-08-25\"\n",
60 | " }\n",
61 | " }'\n",
62 | ");"
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "metadata": {},
68 | "source": [
69 | "### JSON_TO_BSON and BSON_TO_JSON\n",
70 | "If you decide to store the data in binary format, you must use the `JSON_TO_BSON` function to convert the JSON into the proper format. You also have the option of using an external BSON library to convert the string and insert the value directly into the column (i.e. Db2 is not involved in the conversion). "
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": null,
76 | "metadata": {},
77 | "outputs": [],
78 | "source": [
79 | "%%sql -q\n",
80 | "DROP TABLE CUSTOMERS;\n",
81 | "CREATE TABLE CUSTOMERS \n",
82 | "(\n",
83 | " CUSTOMER_ID INT,\n",
84 | " CUSTOMER_INFO VARBINARY(2000)\n",
85 | ");\n",
86 | "\n",
87 | "INSERT INTO CUSTOMERS VALUES \n",
88 | "(\n",
89 | " 1,\n",
90 | " JSON_TO_BSON('{\"customerid\": 100001,\n",
91 | " \"identity\": \n",
92 | " {\n",
93 | " \"firstname\": \"Kelly\",\n",
94 | " \"lastname\" : \"Gilmore\",\n",
95 | " \"birthdate\": \"1973-08-25\"\n",
96 | " }\n",
97 | " }')\n",
98 | ");"
99 | ]
100 | },
101 | {
102 | "cell_type": "markdown",
103 | "metadata": {},
104 | "source": [
105 | "To retrieve an entire JSON document from a character field, you can use a standard `SELECT` statement. If the field is in BSON format, you must use the `BSON_TO_JSON` function to have it converted back into a readable format."
106 | ]
107 | },
108 | {
109 | "cell_type": "code",
110 | "execution_count": null,
111 | "metadata": {},
112 | "outputs": [],
113 | "source": [
114 | "%sql -j SELECT BSON_TO_JSON(CUSTOMER_INFO) FROM CUSTOMERS"
115 | ]
116 | },
117 | {
118 | "cell_type": "markdown",
119 | "metadata": {},
120 | "source": [
121 | "Retrieving the data requires the use of the `BSON_TO_JSON` function to convert it back to a text format."
122 | ]
123 | },
124 | {
125 | "cell_type": "markdown",
126 | "metadata": {},
127 | "source": [
128 | "### Invalid JSON Detection\n",
129 | "One of the advantages of using the new Db2 JSON functions is that you can store the data as either character (JSON) strings, or as binary (BSON) data. However, if you insert a document as a JSON character string, no checking will be done against the validity of the document until you attempt to use a JSON function against it. The following example attempts to retrieve the name field from a JSON document:"
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": null,
135 | "metadata": {},
136 | "outputs": [],
137 | "source": [
138 | "%sql VALUES JSON_VALUE('{\"name\": George}','$.name')"
139 | ]
140 | },
141 | {
142 | "cell_type": "markdown",
143 | "metadata": {},
144 | "source": [
145 | "From a JSON format perspective, this should fail as the value `George` is not quoted and is also not a valid number. Surprisingly, the result of the above statement will be the `NULL` value which will seem wrong at first until you realize that the default error handling clause for any ISO JSON statement is to return a `null` by default. \n",
146 | "\n",
147 | "\n",
148 | "\n",
149 | "If a document needs to be checked for validity during insert, then the `JSON_TO_BSON` function can be used. The following example uses the `VALUES` clause to generate an error on an invalid JSON document. "
150 | ]
151 | },
152 | {
153 | "cell_type": "code",
154 | "execution_count": null,
155 | "metadata": {},
156 | "outputs": [],
157 | "source": [
158 | "%sql VALUES JSON_TO_BSON('{\"name\": George}');"
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "metadata": {},
164 | "source": [
165 | "The Db2 `JSON_TO_BSON` function will check the structure of the JSON document to ensure it is in the proper format. You can write a simple function that can be used to check whether or not a character string is valid JSON:"
166 | ]
167 | },
168 | {
169 | "cell_type": "code",
170 | "execution_count": null,
171 | "metadata": {},
172 | "outputs": [],
173 | "source": [
174 | "%%sql -d\n",
175 | "CREATE OR REPLACE FUNCTION CHECK_JSON(JSON CLOB)\n",
176 | " RETURNS INTEGER\n",
177 | " CONTAINS SQL LANGUAGE SQL \n",
178 | " DETERMINISTIC\n",
179 | " NO EXTERNAL ACTION\n",
180 | "BEGIN\n",
181 | " DECLARE RC BOOLEAN;\n",
182 | " DECLARE EXIT HANDLER FOR SQLEXCEPTION RETURN(FALSE);\n",
183 | " SET RC = JSON_EXISTS(JSON,'$' ERROR ON ERROR);\n",
184 | " RETURN(TRUE);\n",
185 | "END"
186 | ]
187 | },
188 | {
189 | "cell_type": "markdown",
190 | "metadata": {},
191 | "source": [
192 | "The SQL to check the previous string would look like this:"
193 | ]
194 | },
195 | {
196 | "cell_type": "code",
197 | "execution_count": null,
198 | "metadata": {},
199 | "outputs": [],
200 | "source": [
201 | "%%sql\n",
202 | "VALUES \n",
203 | " CASE CHECK_JSON('{\"name\": George}')\n",
204 | " WHEN FALSE THEN 'Bad JSON' \n",
205 | " WHEN TRUE THEN 'Okay JSON' \n",
206 | " END;"
207 | ]
208 | },
209 | {
210 | "cell_type": "markdown",
211 | "metadata": {},
212 | "source": [
213 | "The function can be incorporated into a table definition as part of a check constraint."
214 | ]
215 | },
216 | {
217 | "cell_type": "code",
218 | "execution_count": null,
219 | "metadata": {},
220 | "outputs": [],
221 | "source": [
222 | "%%sql -q\n",
223 | "DROP TABLE TESTJSON;\n",
224 | "CREATE TABLE TESTJSON \n",
225 | "(\n",
226 | " JSON_IN VARCHAR(1000) CONSTRAINT CHECK_JSON CHECK(CHECK_JSON(JSON_IN))\n",
227 | ");"
228 | ]
229 | },
230 | {
231 | "cell_type": "markdown",
232 | "metadata": {},
233 | "source": [
234 | "Attempting to insert an invalid JSON document would result in the following error message being returned:"
235 | ]
236 | },
237 | {
238 | "cell_type": "code",
239 | "execution_count": null,
240 | "metadata": {},
241 | "outputs": [],
242 | "source": [
243 | "%sql INSERT INTO TESTJSON VALUES '{\"name\": George}';"
244 | ]
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {},
249 | "source": [
250 | "The user-defined function uses the `JSON_EXISTS` function. `JSON_EXISTS` allows you to check whether or not a valid JSON key exists within a document for the provided search path. You can use the result of this function to determine if the contents of a JSON document are consistent with your expectations and to decide whether or not to take further action or retrieve the value. You can also use this function to validate that the JSON document is properly formed."
251 | ]
252 | },
253 | {
254 | "cell_type": "markdown",
255 | "metadata": {},
256 | "source": [
257 | "### Summary\n",
258 | "JSON data can be stored in Db2 in either character or binary format. The actual storage format doesn't matter when using the JSON functions. If you want to store data in BSON format, you can use external BSON conversion libraries, or use the built-in `JSON_TO_BSON` function and the corresponding `BSON_TO_JSON` function to convert it back."
259 | ]
260 | },
261 | {
262 | "cell_type": "markdown",
263 | "metadata": {},
264 | "source": [
265 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
266 | ]
267 | }
268 | ],
269 | "metadata": {
270 | "kernelspec": {
271 | "display_name": "Python 3",
272 | "language": "python",
273 | "name": "python3"
274 | },
275 | "language_info": {
276 | "codemirror_mode": {
277 | "name": "ipython",
278 | "version": 3
279 | },
280 | "file_extension": ".py",
281 | "mimetype": "text/x-python",
282 | "name": "python",
283 | "nbconvert_exporter": "python",
284 | "pygments_lexer": "ipython3",
285 | "version": "3.7.3"
286 | }
287 | },
288 | "nbformat": 4,
289 | "nbformat_minor": 2
290 | }
291 |
--------------------------------------------------------------------------------
/Db2_11.5_Features/Db2_11.5_JSON_12_Unnesting_Arrays.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Advanced JSON: Unnesting JSON Arrays\n",
8 | "Updated: 2019-10-03"
9 | ]
10 | },
11 | {
12 | "cell_type": "markdown",
13 | "metadata": {},
14 | "source": [
15 | "## Unnesting Arrays\n",
16 | "One of the challenges of dealing with JSON objects is how to handle arrays of values. The relational model was never designed to deal with a column of data that could be an array so alternate techniques have to be used.\n",
17 | "The `JSON_QUERY` function can be used to retrieve the entire contents of an array, while `JSON_VALUE` or `JSON_TABLE` can extract the individual elements. However, what method is available to extract all of the elements of an array when the actual array size is unknown?\n",
18 | "\n",
19 | "For example, if we have the JSON array `[\"A\",\"B\",\"C\"]` and we want to have the elements returned from an SQL query in a result set like this:\n",
20 | "```\n",
21 | "RESULTS\n",
22 | "-------\n",
23 | "A\n",
24 | "B\n",
25 | "C\n",
26 | "```\n",
27 | "How would we do this?\n",
28 | "\n",
29 | "A complete implementation of the ISO SQL definition for `JSON_TABLE` would have that function handle this case by returning multiple rows with all the other row values duplicated but the Db2 implementation of `JSON_TABLE` is not yet at that stage of maturity and cannot handle this scenario. There is an older, proprietary Db2 JSON function (unfortunately) also called `JSON_TABLE` that is part of the `SYSTOOLS` schema that can be used to generate a simple result set where each row represents an element from the array, but this function does not return multiple values per row and is also not compliant with the ISO SQL JSON standard. \n",
30 | "\n",
31 | "So, in order to retrieve all the elements of an array as a series of independent values, we have to combine all three new ISO JSON functions (`JSON_EXISTS`, `JSON_VALUE`, `JSON_QUERY`) in a recursive SQL query to retrieve them. "
32 | ]
33 | },
34 | {
35 | "cell_type": "markdown",
36 | "metadata": {},
37 | "source": [
38 | "### Load Db2 Extensions and Connect to the Database\n",
39 | "The `connection` notebook contains the `CONNECT` statement which allows access to the `SAMPLE` database. If you need to modify the connection information, edit the `connection.ipynb` notebook."
40 | ]
41 | },
42 | {
43 | "cell_type": "code",
44 | "execution_count": null,
45 | "metadata": {},
46 | "outputs": [],
47 | "source": [
48 | "%run ../db2.ipynb\n",
49 | "%run ../connection.ipynb"
50 | ]
51 | },
52 | {
53 | "cell_type": "markdown",
54 | "metadata": {},
55 | "source": [
56 | "### Unnesting Simple JSON Arrays\n",
57 | "The first example uses the book document which contains a \"simple\" array field called formats. A simple array contains individual atomic values rather than complex objects. "
58 | ]
59 | },
60 | {
61 | "cell_type": "code",
62 | "execution_count": null,
63 | "metadata": {},
64 | "outputs": [],
65 | "source": [
66 | "book = {\n",
67 | " \"authors\": \n",
68 | " [\n",
69 | " {\"first_name\": \"Paul\", \"last_name\" : \"Bird\"},\n",
70 | " {\"first_name\": \"George\",\"last_name\" : \"Baklarz\"}\n",
71 | " ],\n",
72 | " \"foreword\": \n",
73 | " {\n",
74 | " \"primary\": {\"first_name\": \"Thomas\",\"last_name\" : \"Hronis\"}\n",
75 | " },\n",
76 | " \"formats\": [\"Hardcover\",\"Paperback\",\"eBook\",\"PDF\"]\n",
77 | "}"
78 | ]
79 | },
80 | {
81 | "cell_type": "markdown",
82 | "metadata": {},
83 | "source": [
84 | "The \"formats\" field has four values that need to be return as a list. The following SQL uses recursion to extract the values from the array. "
85 | ]
86 | },
87 | {
88 | "cell_type": "code",
89 | "execution_count": null,
90 | "metadata": {},
91 | "outputs": [],
92 | "source": [
93 | "%%sql\n",
94 | "WITH BOOKS(INFO) AS (VALUES :book),\n",
95 | "FORMATS(INDEX, JSON_PATH, BOOKTYPE) AS \n",
96 | "(\n",
97 | " SELECT \n",
98 | " 0, '$.formats[1]',JSON_VALUE(INFO,'$.formats[0]')\n",
99 | " FROM BOOKS \n",
100 | " WHERE JSON_EXISTS(INFO,'$.formats[0]') IS TRUE\n",
101 | " UNION ALL\n",
102 | " SELECT \n",
103 | " INDEX+1, \n",
104 | " '$.formats[' || TRIM(CHAR(INDEX + 2)) || ']',\n",
105 | " JSON_VALUE(INFO, JSON_PATH) \n",
106 | " FROM BOOKS, FORMATS\n",
107 | " WHERE JSON_EXISTS(INFO, JSON_PATH) IS TRUE\n",
108 | ")\n",
109 | "SELECT BOOKTYPE FROM FORMATS"
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "metadata": {},
115 | "source": [
116 | "The breakdown of the code is found below. The line numbers are shown below for reference. Note that the first line of code is not included `WITH BOOKS(INFO) AS (VALUES :book)` as it was used as a temporary table to run the SQL.\n",
117 | "```\n",
118 | "[ 1] WITH FORMATS(INDEX, JSON_PATH, BOOKTYPE) AS \n",
119 | "[ 2] (\n",
120 | "[ 3] SELECT \n",
121 | "[ 4] 0, '$.formats[1]',JSON_VALUE(INFO,'$.formats[0]')\n",
122 | "[ 5] FROM BOOKS \n",
123 | "[ 6] WHERE JSON_EXISTS(INFO,'$.formats[0]') IS TRUE\n",
124 | "[ 7] UNION ALL\n",
125 | "[ 8] SELECT \n",
126 | "[ 9] INDEX+1, \n",
127 | "[10] '$.formats[' || TRIM(CHAR(INDEX + 2)) || ']',\n",
128 | "[11] JSON_VALUE(INFO, JSON_PATH) \n",
129 | "[12] FROM BOOKS, FORMATS\n",
130 | "[13] WHERE JSON_EXISTS(INFO, JSON_PATH) IS TRUE\n",
131 | "[14] )\n",
132 | "[15] SELECT BOOKTYPE FROM FORMATS\n",
133 | "```\t"
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "`[1-14]` `WITH` Block\n",
141 | "\n",
142 | "The first section of code is used to initialize a recursive SQL block. Recursive SQL allows us to continually add rows to an answer set based on the results from a SQL statement that gets repeated multiple times.\n",
143 | "```\n",
144 | "[1] WITH FORMATS(INDEX, JSON_PATH, BOOKTYPE) AS\n",
145 | "```\n",
146 | "The common table expression used in this example is called `FORMATS` and contains three columns. The `INDEX` column is used to increment the array item we want to retrieve, the `JSON_PATH` is used as the path expression to find the next value, and `BOOKTYPE` is the value extracted from the array.\n",
147 | "\n",
148 | "`[3-5]` `SELECT` statement\n",
149 | "\n",
150 | "The first part of the `SELECT` statement is used to initialize the recursion by providing the first row of the result set.\n",
151 | "```\n",
152 | "[ 3] SELECT \n",
153 | "[ 4] 0, '$.formats[1]',JSON_VALUE(INFO,'$.formats[0]')\n",
154 | "[ 5] FROM BOOKS \n",
155 | "```\n",
156 | "The values are:\n",
157 | "* `INDEX = 0` – This is the first index value in an array\n",
158 | "* `JSON_PATH = '$.formats[1]'` – The path to the next array value\n",
159 | "* `BOOKTYPE = JSON_VALUE(INFO,'$.formats[0]')` – The first value in the formats array\n",
160 | "\n",
161 | "The `JSON_PATH` column is used as the path expression to find the next array value. This value could be placed directly in the SQL but since the expression is required twice, there is less likelihood of incorrect syntax! The `JSON_PATH` expression is always set to the next value that we need rather than the current one.\n",
162 | "\n",
163 | "`[6] WHERE JSON_EXISTS() IS TRUE`\n",
164 | "\n",
165 | "The `WHERE` clause is used to check whether or not the first value in the array exists. If it does not, then we return no results.\n",
166 | "```\n",
167 | "[ 6] WHERE JSON_EXISTS(INFO,'$.formats[0]') IS TRUE\n",
168 | "[ 7] UNION ALL\n",
169 | "```\n",
170 | "\n",
171 | "The `UNION ALL` is required to make the SQL recursive in nature. As the SQL executes, it will add more rows to the `FORMATS` table and then the new rows will be acted upon by this SQL block. \n",
172 | " \n",
173 | " `[8-12]` Get the remainder of the array values\n",
174 | " \n",
175 | "This block will continue to iterate as long as there are more array values.\n",
176 | "```\n",
177 | "[ 8] SELECT \n",
178 | "[ 9] INDEX+1, \n",
179 | "[10] '$.formats[' || TRIM(CHAR(INDEX + 2)) || ']',\n",
180 | "[11] JSON_VALUE(INFO, JSON_PATH) \n",
181 | "[12] FROM BOOKS, FORMATS\n",
182 | "```\n",
183 | "The `SELECT` statement increments the index number into the array, creates the next path expression, and retrieves the current array value.\n",
184 | "\n",
185 | "The `JSON_PATH` is generated as a character string:\n",
186 | "```\n",
187 | "[10] '$.formats[' || TRIM(CHAR(INDEX + 2)) || ']',\n",
188 | "```\n",
189 | "\n",
190 | "The first portion of the string is the path to the object, concatenated with the current index value plus 2 (always one ahead of the current index value).\n",
191 | "The tables that are accessed by the SQL are the `BOOKS` table (with the original JSON) and the `FORMATS` table – which is what we are building recursively.\n",
192 | "```\n",
193 | "[13] WHERE JSON_EXIST() IS TRUE\n",
194 | "```\n",
195 | "\n",
196 | "The `WHERE` clause is used to check whether or not the current value in the array exists. If it does not exist, then we stop the recursion. This is often referred to as the stop condition in the recursion loop.\n",
197 | "```\n",
198 | "[13] WHERE JSON_EXISTS(INFO, JSON_PATH) IS TRUE\n",
199 | "[15] Final SELECT statement\n",
200 | "```\n",
201 | "\n",
202 | "Once the recursion is done, we can retrieve the contents of the array. We refer to the `BOOKTYPE` column because that is the only value we are interested in, but if you select everything you will see the index values and path expressions that were generated as part of the SQL.\n",
203 | "```\n",
204 | "INDEX JSON_PATH BOOKTYPE\n",
205 | "----- ------------ --------\n",
206 | " 0 $.formats[1] Hardcover\n",
207 | " 1 $.formats[2] Paperback\n",
208 | " 2 $.formats[3] eBook\n",
209 | " 3 $.formats[4] PDF\n",
210 | "```"
211 | ]
212 | },
213 | {
214 | "cell_type": "markdown",
215 | "metadata": {},
216 | "source": [
217 | "## Summary\n",
218 | "While there is currently no single JSON function within Db2 to retrieve all array values, the combination of `JSON_EXISTS`, `JSON_VALUE`, `JSON_TABLE`, and `JSON_QUERY` can be combined with recursive SQL to extract array objects or individual values."
219 | ]
220 | },
221 | {
222 | "cell_type": "markdown",
223 | "metadata": {},
224 | "source": [
225 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
226 | ]
227 | }
228 | ],
229 | "metadata": {
230 | "kernelspec": {
231 | "display_name": "Python 3",
232 | "language": "python",
233 | "name": "python3"
234 | },
235 | "language_info": {
236 | "codemirror_mode": {
237 | "name": "ipython",
238 | "version": 3
239 | },
240 | "file_extension": ".py",
241 | "mimetype": "text/x-python",
242 | "name": "python",
243 | "nbconvert_exporter": "python",
244 | "pygments_lexer": "ipython3",
245 | "version": "3.6.8"
246 | }
247 | },
248 | "nbformat": 4,
249 | "nbformat_minor": 2
250 | }
251 |
--------------------------------------------------------------------------------
/Db2_11.5_Features/djia-symbols.csv:
--------------------------------------------------------------------------------
1 | Symbol,"Company Name"
2 | AAPL,"Apple Inc."
3 | AXP,"American Express Company"
4 | BA,"The Boeing Company"
5 | CAT,"Caterpillar Inc."
6 | CSCO,"Cisco Systems, Inc."
7 | CVX,"Chevron Corporation"
8 | DIS,"The Walt Disney Company"
9 | DWDP,DowDuPont Inc."
10 | GS,"The Goldman Sachs Group, Inc."
11 | HD,"The Home Depot, Inc."
12 | IBM,"International Business Machines Corporation"
13 | INTC,"Intel Corporation"
14 | JNJ,"Johnson & Johnson"
15 | JPM,"JPMorgan Chase & Co."
16 | KO,"The Coca-Cola Company"
17 | MCD,"McDonald's Corporation"
18 | MMM,"3M Company"
19 | MRK,"Merck & Co., Inc."
20 | MSFT,"Microsoft Corporation"
21 | NKE,"NIKE, Inc."
22 | PFE,"Pfizer Inc."
23 | PG,"The Procter & Gamble Company"
24 | TRV,"The Travelers Companies, Inc."
25 | UNH,"UnitedHealth Group Incorporated"
26 | UTX,"United Technologies Corporation"
27 | V,"Visa Inc."
28 | VZ,"Verizon Communications Inc."
29 | WBA,"Walgreens Boots Alliance, Inc."
30 | WMT,"Walmart Inc."
31 | XOM,Exxon Mobil Corporation
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/BSON2JSON.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/BSON2JSON.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/DEFAULT.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/DEFAULT.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/ERROR_CLAUSE.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/ERROR_CLAUSE.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON2BSON.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON2BSON.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_ARRAY.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_ARRAY.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_EXISTS copy.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_EXISTS copy.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_EXISTS.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_EXISTS.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_OBJECT.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_OBJECT.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_QUERY.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_QUERY.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_QUERY_KVE.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_QUERY_KVE.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_TABLE.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_TABLE.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/JSON_VALUE.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/JSON_VALUE.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/command-input.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/command-input.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/error-processing.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/error-processing.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/implicit-keyword.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/implicit-keyword.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/input.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/input.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/ja-full-select.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/ja-full-select.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/ja-json-expression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/ja-json-expression.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/ja-null-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/ja-null-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/ja-returning-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/ja-returning-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/je-error-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/je-error-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jo-key-expression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jo-key-expression.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jo-null-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jo-null-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jo-returning-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jo-returning-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jo-unique-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jo-unique-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jq-empty-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jq-empty-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jq-quotes-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jq-quotes-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jq-returning-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jq-returning-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jq-wrapper-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jq-wrapper-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/json-path-expression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/json-path-expression.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/json-pathexpression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/json-pathexpression.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/json_expression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/json_expression.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-formatted-empty-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-formatted-empty-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-formatted.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-formatted.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-quotes-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-quotes-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-regular-empty-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-regular-empty-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-regular.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-regular.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jt-wrapper-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jt-wrapper-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jv-empty-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jv-empty-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/jv-returning-clause.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/jv-returning-clause.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/multiple-choice.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/multiple-choice.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/required-parameters.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/required-parameters.png
--------------------------------------------------------------------------------
/Db2_11.5_Features/images/syntax-keywords.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_11.5_Features/images/syntax-keywords.png
--------------------------------------------------------------------------------
/Db2_BLU_Demonstration/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_BLU_Demonstration/CQW_Large_Reports.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# CQW - Large Reports\n",
8 | "Updated: 2019-10-16\n",
9 | "\n",
10 | "Large reports typically return more records. Due to the large number of records returned, total execution time of row versus columnar may be closer. The following reports are found in this notebook. \n",
11 | "\n",
12 | " - Report 3a1 - Customer Quantity of Sales 1999-2002 where Lifetime Quantity > 16,000\n",
13 | " - Report 3a2 - Customer Quantity of Sales 1999-2002\n",
14 | " - Report 3b1 - Sorted List Customers Quantity Sold for Year 1999\n",
15 | "\n",
16 | "### Load Db2 Extensions and Connect to the Database\n",
17 | "The `connection` notebook contains the `CONNECT` statement which allows access to the `SAMPLE` database. If you need to modify the connection information, edit the `connection.ipynb` notebook."
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": null,
23 | "metadata": {},
24 | "outputs": [],
25 | "source": [
26 | "%run ../db2.ipynb\n",
27 | "%run ../connection.ipynb"
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "metadata": {},
33 | "source": [
34 | "### Create a Function for Running the Queries\n",
35 | "This function is used to time the queries and return the total time taken. The parameter is the SQL that will be run and the function will execute against the row-based and col-based tables. "
36 | ]
37 | },
38 | {
39 | "cell_type": "code",
40 | "execution_count": null,
41 | "metadata": {},
42 | "outputs": [],
43 | "source": [
44 | "def runquery(sql):\n",
45 | " \n",
46 | " %sql -q SET CURRENT SCHEMA = CQWROW\n",
47 | " \n",
48 | " print(\"Starting execution of ROW query\") \n",
49 | " start_time = time.time()\n",
50 | "\n",
51 | " %sql -q {sql}\n",
52 | " \n",
53 | " end_time = time.time()\n",
54 | " row = end_time - start_time\n",
55 | " \n",
56 | " print(\"Row run time = {0}\".format(row))\n",
57 | " \n",
58 | " %sql -q SET CURRENT SCHEMA = CQWCOL\n",
59 | " \n",
60 | " print()\n",
61 | " print(\"Starting execution of COLUMNAR query\") \n",
62 | " \n",
63 | " start_time = time.time()\n",
64 | "\n",
65 | " results = %sql -q {query}\n",
66 | " \n",
67 | " end_time = time.time()\n",
68 | " col = end_time - start_time\n",
69 | " \n",
70 | " print(\"Column run time = {0}\".format(col)) \n",
71 | " \n",
72 | " pdisplay(results)\n",
73 | " \n",
74 | " %sql -bar VALUES ('Row',:row),('Column',:col)"
75 | ]
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {},
80 | "source": [
81 | "## Customer Quantity of Sales 1999-2002 where Lifetime Quantity > 16,000"
82 | ]
83 | },
84 | {
85 | "cell_type": "code",
86 | "execution_count": null,
87 | "metadata": {},
88 | "outputs": [],
89 | "source": [
90 | "query = '''\n",
91 | "WITH \n",
92 | "\"RSF_Query1\" AS \n",
93 | " (\n",
94 | " SELECT\n",
95 | " \"CUSTOMER\".\"C_CUSTOMER_SK\" AS \"Customers_C_CUSTOMER_SK\", \n",
96 | " SUM(CAST(\"STORE_SALES\".\"SS_QUANTITY\" AS DECIMAL(31,2))) AS \"Quantity__Store_Sales_\"\n",
97 | " FROM\n",
98 | " \"CUSTOMER\" \"CUSTOMER\"\n",
99 | " INNER JOIN \"STORE_SALES\" \"STORE_SALES\"\n",
100 | " ON \"CUSTOMER\".\"C_CUSTOMER_SK\" = \"STORE_SALES\".\"SS_CUSTOMER_SK\" \n",
101 | " GROUP BY \n",
102 | " \"CUSTOMER\".\"C_CUSTOMER_SK\"\n",
103 | " )\n",
104 | "SELECT\n",
105 | " \"RSF_Query1\".\"Customers_C_CUSTOMER_SK\" AS \"Customers_C_CUSTOMER_SK\", \n",
106 | " \"RSF_Query1\".\"Quantity__Store_Sales_\" AS \"Quantity__Store_Sales_\"\n",
107 | "FROM\n",
108 | " \"RSF_Query1\" \n",
109 | "WHERE \n",
110 | " \"RSF_Query1\".\"Quantity__Store_Sales_\" > 16000.0\n",
111 | "'''\n",
112 | "runquery(query)"
113 | ]
114 | },
115 | {
116 | "cell_type": "markdown",
117 | "metadata": {},
118 | "source": [
119 | "## Customer Quantity of Sales 1999-2002 "
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": null,
125 | "metadata": {},
126 | "outputs": [],
127 | "source": [
128 | "query = '''\n",
129 | "SELECT\n",
130 | " \"DATE_DIM\".\"D_YEAR\" AS \"Date_D_YEAR\", \n",
131 | " \"CUSTOMER\".\"C_CUSTOMER_SK\" AS \"Customers_C_CUSTOMER_SK\", \n",
132 | " SUM(CAST(\"STORE_SALES\".\"SS_QUANTITY\" AS DECIMAL(31,2))) AS \"Quantity__Store_Sales_\"\n",
133 | "FROM\n",
134 | " \"CUSTOMER\" \"CUSTOMER\"\n",
135 | " INNER JOIN \"STORE_SALES\" \"STORE_SALES\"\n",
136 | " ON \"CUSTOMER\".\"C_CUSTOMER_SK\" = \"STORE_SALES\".\"SS_CUSTOMER_SK\"\n",
137 | " INNER JOIN \"DATE_DIM\" \"DATE_DIM\"\n",
138 | " ON \"DATE_DIM\".\"D_DATE_SK\" = \"STORE_SALES\".\"SS_SOLD_DATE_SK\" \n",
139 | "WHERE \n",
140 | " \"CUSTOMER\".\"C_CUSTOMER_SK\" BETWEEN -99999 AND 11999979 \n",
141 | "GROUP BY \n",
142 | " \"DATE_DIM\".\"D_YEAR\", \n",
143 | " \"CUSTOMER\".\"C_CUSTOMER_SK\"\n",
144 | "'''\n",
145 | "runquery(query)"
146 | ]
147 | },
148 | {
149 | "cell_type": "markdown",
150 | "metadata": {},
151 | "source": [
152 | "## Sorted List Customers Quantity Sold for Year 1999"
153 | ]
154 | },
155 | {
156 | "cell_type": "code",
157 | "execution_count": null,
158 | "metadata": {},
159 | "outputs": [],
160 | "source": [
161 | "query = '''\n",
162 | "WITH \n",
163 | "\"SQ0_Query1\" AS \n",
164 | " (\n",
165 | " SELECT\n",
166 | " \"DATE_DIM\".\"D_YEAR\" AS \"Date_D_YEAR\", \n",
167 | " \"CUSTOMER\".\"C_CUSTOMER_SK\" AS \"Customers_C_CUSTOMER_SK\", \n",
168 | " SUM(CAST(\"STORE_SALES\".\"SS_QUANTITY\" AS DECIMAL(31,2))) AS \"Quantity__Store_Sales_\"\n",
169 | " FROM\n",
170 | " \"CUSTOMER\" \"CUSTOMER\"\n",
171 | " INNER JOIN \"STORE_SALES\" \"STORE_SALES\"\n",
172 | " ON \"CUSTOMER\".\"C_CUSTOMER_SK\" = \"STORE_SALES\".\"SS_CUSTOMER_SK\"\n",
173 | " INNER JOIN \"DATE_DIM\" \"DATE_DIM\"\n",
174 | " ON \"DATE_DIM\".\"D_DATE_SK\" = \"STORE_SALES\".\"SS_SOLD_DATE_SK\" \n",
175 | " WHERE \n",
176 | " \"DATE_DIM\".\"D_YEAR\" > 1899 AND \"DATE_DIM\".\"D_YEAR\" < 2000\n",
177 | " GROUP BY \n",
178 | " \"DATE_DIM\".\"D_YEAR\", \n",
179 | " \"CUSTOMER\".\"C_CUSTOMER_SK\"\n",
180 | " )\n",
181 | "SELECT\n",
182 | " \"SQ0_Query1\".\"Date_D_YEAR\" AS \"Date_D_YEAR\", \n",
183 | " \"SQ0_Query1\".\"Customers_C_CUSTOMER_SK\" AS \"Customers_C_CUSTOMER_SK\", \n",
184 | " \"SQ0_Query1\".\"Quantity__Store_Sales_\" AS \"Quantity__Store_Sales_\", \n",
185 | " RANK()\n",
186 | " OVER(\n",
187 | " ORDER BY\n",
188 | " \"SQ0_Query1\".\"Quantity__Store_Sales_\" DESC NULLS LAST\n",
189 | " ) AS \"rank0\"\n",
190 | "FROM\n",
191 | " \"SQ0_Query1\"\n",
192 | "'''\n",
193 | "runquery(query)"
194 | ]
195 | },
196 | {
197 | "cell_type": "markdown",
198 | "metadata": {},
199 | "source": [
200 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
201 | ]
202 | }
203 | ],
204 | "metadata": {
205 | "kernelspec": {
206 | "display_name": "Python 3",
207 | "language": "python",
208 | "name": "python3"
209 | },
210 | "language_info": {
211 | "codemirror_mode": {
212 | "name": "ipython",
213 | "version": 3
214 | },
215 | "file_extension": ".py",
216 | "mimetype": "text/x-python",
217 | "name": "python",
218 | "nbconvert_exporter": "python",
219 | "pygments_lexer": "ipython3",
220 | "version": "3.6.8"
221 | }
222 | },
223 | "nbformat": 4,
224 | "nbformat_minor": 2
225 | }
226 |
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/Change_Title.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_Jupyter_Tutorials/images/Change_Title.jpg
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/Edit_Menu.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_Jupyter_Tutorials/images/Edit_Menu.jpg
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/Jupyter_Files.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_Jupyter_Tutorials/images/Jupyter_Files.jpg
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/Jupyter_Toolbar.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_Jupyter_Tutorials/images/Jupyter_Toolbar.jpg
--------------------------------------------------------------------------------
/Db2_Jupyter_Tutorials/images/Menus.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/db2-jupyter/e14db5c4886761cb40cfdb52b03abca4fe3f9c01/Db2_Jupyter_Tutorials/images/Menus.jpg
--------------------------------------------------------------------------------
/Installation/.gitignore:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/Installation/Docker_Instructions.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Creating a Docker Container for Db2 and Jupyter Notebooks\n",
8 | "You can create a Docker image that will contain Jupyter Notebooks, the Db2 drivers, and all of the examples found in this Github repository.\n",
9 | "\n",
10 | "Assuming that you have Docker running on Windows, Mac, or Linux, the following commands will create a Docker image for you.\n",
11 | "\n",
12 | " * Download the `db2jupyter.docker` file (found in this directory) and place it into an **_empty_** directory of your choice\n",
13 | " \n",
14 | " \n",
15 | " * Open up a command window that is able to issue Docker commands (either use Kitematic CLI command line, or a terminal window on Mac or Linux)\n",
16 | " \n",
17 | " \n",
18 | " * Navigate to the directory that the `db2jupyter.docker` file is located (i.e. cd or chdir)\n",
19 | " \n",
20 | " \n",
21 | " * Issue the following command to create a Docker image:\n",
22 | "```Python\n",
23 | "docker build -t db2jupyter -f db2jupyter.docker . <- Note the period at the end\n",
24 | "```\n",
25 | "\n",
26 | "\n",
27 | " * Once the build is complete (there will be some warning messages with the `ibm-db` creation) you can now run the docker container with the following command.\n",
28 | "```Python\n",
29 | "docker run --name db2jupyter -d -p 8888:8888 db2jupyter \n",
30 | "```\n",
31 | "\n",
32 | "\n",
33 | " * If port 8888 is already in use on your system you will have to give it a different port number. For instance, the following command will map the host port 9999 to port 8888 inside the Docker container.\n",
34 | "```Python\n",
35 | "docker run --name db2jupyter -d -p 9999:8888 db2jupyter \n",
36 | "```\n",
37 | "\n",
38 | "\n",
39 | " * Use your favorite browser to navigate to `localhost:8888` and all of the Db2 notebooks will be listed and available for use."
40 | ]
41 | },
42 | {
43 | "cell_type": "markdown",
44 | "metadata": {},
45 | "source": [
46 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
47 | ]
48 | }
49 | ],
50 | "metadata": {
51 | "kernelspec": {
52 | "display_name": "Python 3",
53 | "language": "python",
54 | "name": "python3"
55 | },
56 | "language_info": {
57 | "codemirror_mode": {
58 | "name": "ipython",
59 | "version": 3
60 | },
61 | "file_extension": ".py",
62 | "mimetype": "text/x-python",
63 | "name": "python",
64 | "nbconvert_exporter": "python",
65 | "pygments_lexer": "ipython3",
66 | "version": "3.7.3"
67 | }
68 | },
69 | "nbformat": 4,
70 | "nbformat_minor": 2
71 | }
72 |
--------------------------------------------------------------------------------
/Installation/Installation_Instructions.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Installation Requirements\n",
8 | "\n",
9 | "There are three steps that you need to take in order to use Db2 Magic commands:\n",
10 | "* Install Jupyter Notebook\n",
11 | "* Install the Db2 Python drivers\n",
12 | "* Copy the Db2 Magic commands into your notebook directory\n",
13 | "\n",
14 | "## Jupyter Notebook\n",
15 | "You need to have Jupyter notebook installed on your system, which also needs Python to be installed on your system. If you already have Python available, odds are that you have Jupyter installed as well. The next set of commands will install Python on your platform and then install the Jupyter components. Note that Db2 magic commands will work on JupyterHub and JupyterLab, but some of the display functions (**grid**) are not supported in the JupyterLab environment. \n",
16 | "\n",
17 | "### Anaconda or Miniconda\n",
18 | "Anaconda is an Open Data Science Platform that is powered by Python http://www.continuum.io. The platform keeps track of packages and their dependencies for development with Jupyter notebooks and Python. This makes it very easy to install extensions for Python without having to manually install everything.\n",
19 | "\n",
20 | "Download the Anaconda or Miniconda package applicable to your platform. Miniconda creates the minimal system required for using Python and Jupyter, while Anaconda installs all major packages. There are two versions of Python - V2 or V3. While it doesn't matter which one you use for most notebooks, you may want to use V3 if you are writing Python code since that is the release that is being enhanced. \n",
21 | "\n",
22 | "After installing Anaconda, you should issue the following commands from a command line shell that will update and install components required by Db2 notebooks. In the Windows environment, the Anaconda installer will create a shortcut to a command line. Make sure you run it as an Administrative user. Similarly, if you are using Linux, you will need to use `sudo` in order to run these commands.\n",
23 | "```\n",
24 | "conda update conda - This will update the Anaconda distribution so you have the latest code\n",
25 | "conda install -y qgrid - Add components needed for displaying result sets \n",
26 | "easy_install ibm-db - Install the Db2 Python drivers\n",
27 | "```\n",
28 | "\n",
29 | "At this point your installation should have Jupyter available on your system. To start the notebook server you need to issue the following command:\n",
30 | "```\n",
31 | "jupyter notebook (opens browser)\n",
32 | "jupyter notebook --no-browser (runs as a service)\n",
33 | "```\n",
34 | "The first command will open up a browser window that displays your notebooks. You can click on one of these notebooks to see the contents. If no notebooks are available, you will need to move the files in the Github jupyter directory to a local folder on your system that the program can access.\n",
35 | "\n",
36 | "### Db2 Extensions\n",
37 | "\n",
38 | "If you have an existing Jupyter Notebook environment, you can add Db2 support to it by installing the `ibm-db` package. This package adds appropriate database commands to Python so that it can access Db2 data. The `ibm-db` package is not available as part of the Anaconda/Miniconda package so you need to use the `easy_install` command to install it.\n",
39 | "\n",
40 | "You may need administrator privileges (`sudo` or Windows Administrative User) to run this command.\n",
41 | "```\n",
42 | "easy_install ibm-db\n",
43 | "```\n",
44 | "\n",
45 | "Once you have installed the software you will need place the `db2.ipynb` notebook in the same directory where you want to issue Db2 commands. One line in your notebook should contain:\n",
46 | "```\n",
47 | "%run db2.ipynb\n",
48 | "```\n",
49 | "\n",
50 | "This will load the appropriate Db2 libraries and make the `%sql` and `%%sql` magic commands available to your notebook."
51 | ]
52 | },
53 | {
54 | "cell_type": "markdown",
55 | "metadata": {},
56 | "source": [
57 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
58 | ]
59 | }
60 | ],
61 | "metadata": {
62 | "kernelspec": {
63 | "display_name": "Python 3",
64 | "language": "python",
65 | "name": "python3"
66 | },
67 | "language_info": {
68 | "codemirror_mode": {
69 | "name": "ipython",
70 | "version": 3
71 | },
72 | "file_extension": ".py",
73 | "mimetype": "text/x-python",
74 | "name": "python",
75 | "nbconvert_exporter": "python",
76 | "pygments_lexer": "ipython3",
77 | "version": "3.7.3"
78 | }
79 | },
80 | "nbformat": 4,
81 | "nbformat_minor": 2
82 | }
83 |
--------------------------------------------------------------------------------
/Installation/db2jupyter.docker:
--------------------------------------------------------------------------------
1 | #
2 | # Jupyter notebooks for Db2 using Python 3.x format
3 | # Version 1.6
4 | # 2019-10-22
5 | # Author: George Baklarz
6 | #
7 |
8 | # Build image:
9 | # docker build -t db2jupyter -f db2jupyter.docker .
10 |
11 | # Run image:
12 | # docker run --name db2jupyter -d -p 8888:8888 db2jupyter
13 |
14 | FROM continuumio/anaconda3
15 |
16 | # 1. Update the distribution
17 | # 2. Update apt-get and install the gcc compiler
18 | # 3. easy_install the Db2 drivers (ibm-db)
19 | # 4. Remove temp files
20 |
21 | RUN conda update -y conda && \
22 | conda install -y -c conda-forge ipywidgets qgrid && \
23 | apt-get update && \
24 | apt-get install -y gcc && \
25 | easy_install ibm-db && \
26 | rm -rf /var/lib/apt/lists/*
27 |
28 | # Create the directory for Jupyter defaults and insert the settings into the file
29 | # 1. Allow connection without a password
30 | # 2. True deletion of files (bypasses a bug in Jupyter)
31 | # 3. Allow root to run Jupyter
32 | # 4. Clone the Db2-Samples Jupyter notebooks files into the /opt/notebooks directory
33 |
34 | RUN mkdir /root/.jupyter && \
35 | echo "c.NotebookApp.token = u''" >> ~/.jupyter/jupyter_notebook_config.py && \
36 | echo "c.FileContentsManager.delete_to_trash=False" >> ~/.jupyter/jupyter_notebook_config.py && \
37 | echo "c.NotebookApp.allow_root=True" >> ~/.jupyter/jupyter_notebook_config.py && \
38 | git clone git://github.com/DB2-Samples/jupyterlab /opt/notebooks
39 |
40 | #
41 | # Command to start Jupyter in the container
42 | # Updated --ip='*' to --ip='0.0.0.0'
43 | #
44 |
45 | CMD jupyter notebook --NotebookApp.token= --notebook-dir=/opt/notebooks --ip='0.0.0.0' --port=8888 --no-browser --allow-root
46 |
47 | ENV DESCRIPTION="Db2 Jupyter Notebook" \
48 | VERSION="1.6" \
49 | BUILD_DATE="2019-10-22"
50 |
51 | EXPOSE 8888
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Apache License
2 | Version 2.0, January 2004
3 | http://www.apache.org/licenses/
4 |
5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6 |
7 | 1. Definitions.
8 |
9 | "License" shall mean the terms and conditions for use, reproduction,
10 | and distribution as defined by Sections 1 through 9 of this document.
11 |
12 | "Licensor" shall mean the copyright owner or entity authorized by
13 | the copyright owner that is granting the License.
14 |
15 | "Legal Entity" shall mean the union of the acting entity and all
16 | other entities that control, are controlled by, or are under common
17 | control with that entity. For the purposes of this definition,
18 | "control" means (i) the power, direct or indirect, to cause the
19 | direction or management of such entity, whether by contract or
20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
21 | outstanding shares, or (iii) beneficial ownership of such entity.
22 |
23 | "You" (or "Your") shall mean an individual or Legal Entity
24 | exercising permissions granted by this License.
25 |
26 | "Source" form shall mean the preferred form for making modifications,
27 | including but not limited to software source code, documentation
28 | source, and configuration files.
29 |
30 | "Object" form shall mean any form resulting from mechanical
31 | transformation or translation of a Source form, including but
32 | not limited to compiled object code, generated documentation,
33 | and conversions to other media types.
34 |
35 | "Work" shall mean the work of authorship, whether in Source or
36 | Object form, made available under the License, as indicated by a
37 | copyright notice that is included in or attached to the work
38 | (an example is provided in the Appendix below).
39 |
40 | "Derivative Works" shall mean any work, whether in Source or Object
41 | form, that is based on (or derived from) the Work and for which the
42 | editorial revisions, annotations, elaborations, or other modifications
43 | represent, as a whole, an original work of authorship. For the purposes
44 | of this License, Derivative Works shall not include works that remain
45 | separable from, or merely link (or bind by name) to the interfaces of,
46 | the Work and Derivative Works thereof.
47 |
48 | "Contribution" shall mean any work of authorship, including
49 | the original version of the Work and any modifications or additions
50 | to that Work or Derivative Works thereof, that is intentionally
51 | submitted to Licensor for inclusion in the Work by the copyright owner
52 | or by an individual or Legal Entity authorized to submit on behalf of
53 | the copyright owner. For the purposes of this definition, "submitted"
54 | means any form of electronic, verbal, or written communication sent
55 | to the Licensor or its representatives, including but not limited to
56 | communication on electronic mailing lists, source code control systems,
57 | and issue tracking systems that are managed by, or on behalf of, the
58 | Licensor for the purpose of discussing and improving the Work, but
59 | excluding communication that is conspicuously marked or otherwise
60 | designated in writing by the copyright owner as "Not a Contribution."
61 |
62 | "Contributor" shall mean Licensor and any individual or Legal Entity
63 | on behalf of whom a Contribution has been received by Licensor and
64 | subsequently incorporated within the Work.
65 |
66 | 2. Grant of Copyright License. Subject to the terms and conditions of
67 | this License, each Contributor hereby grants to You a perpetual,
68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69 | copyright license to reproduce, prepare Derivative Works of,
70 | publicly display, publicly perform, sublicense, and distribute the
71 | Work and such Derivative Works in Source or Object form.
72 |
73 | 3. Grant of Patent License. Subject to the terms and conditions of
74 | this License, each Contributor hereby grants to You a perpetual,
75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76 | (except as stated in this section) patent license to make, have made,
77 | use, offer to sell, sell, import, and otherwise transfer the Work,
78 | where such license applies only to those patent claims licensable
79 | by such Contributor that are necessarily infringed by their
80 | Contribution(s) alone or by combination of their Contribution(s)
81 | with the Work to which such Contribution(s) was submitted. If You
82 | institute patent litigation against any entity (including a
83 | cross-claim or counterclaim in a lawsuit) alleging that the Work
84 | or a Contribution incorporated within the Work constitutes direct
85 | or contributory patent infringement, then any patent licenses
86 | granted to You under this License for that Work shall terminate
87 | as of the date such litigation is filed.
88 |
89 | 4. Redistribution. You may reproduce and distribute copies of the
90 | Work or Derivative Works thereof in any medium, with or without
91 | modifications, and in Source or Object form, provided that You
92 | meet the following conditions:
93 |
94 | (a) You must give any other recipients of the Work or
95 | Derivative Works a copy of this License; and
96 |
97 | (b) You must cause any modified files to carry prominent notices
98 | stating that You changed the files; and
99 |
100 | (c) You must retain, in the Source form of any Derivative Works
101 | that You distribute, all copyright, patent, trademark, and
102 | attribution notices from the Source form of the Work,
103 | excluding those notices that do not pertain to any part of
104 | the Derivative Works; and
105 |
106 | (d) If the Work includes a "NOTICE" text file as part of its
107 | distribution, then any Derivative Works that You distribute must
108 | include a readable copy of the attribution notices contained
109 | within such NOTICE file, excluding those notices that do not
110 | pertain to any part of the Derivative Works, in at least one
111 | of the following places: within a NOTICE text file distributed
112 | as part of the Derivative Works; within the Source form or
113 | documentation, if provided along with the Derivative Works; or,
114 | within a display generated by the Derivative Works, if and
115 | wherever such third-party notices normally appear. The contents
116 | of the NOTICE file are for informational purposes only and
117 | do not modify the License. You may add Your own attribution
118 | notices within Derivative Works that You distribute, alongside
119 | or as an addendum to the NOTICE text from the Work, provided
120 | that such additional attribution notices cannot be construed
121 | as modifying the License.
122 |
123 | You may add Your own copyright statement to Your modifications and
124 | may provide additional or different license terms and conditions
125 | for use, reproduction, or distribution of Your modifications, or
126 | for any such Derivative Works as a whole, provided Your use,
127 | reproduction, and distribution of the Work otherwise complies with
128 | the conditions stated in this License.
129 |
130 | 5. Submission of Contributions. Unless You explicitly state otherwise,
131 | any Contribution intentionally submitted for inclusion in the Work
132 | by You to the Licensor shall be under the terms and conditions of
133 | this License, without any additional terms or conditions.
134 | Notwithstanding the above, nothing herein shall supersede or modify
135 | the terms of any separate license agreement you may have executed
136 | with Licensor regarding such Contributions.
137 |
138 | 6. Trademarks. This License does not grant permission to use the trade
139 | names, trademarks, service marks, or product names of the Licensor,
140 | except as required for reasonable and customary use in describing the
141 | origin of the Work and reproducing the content of the NOTICE file.
142 |
143 | 7. Disclaimer of Warranty. Unless required by applicable law or
144 | agreed to in writing, Licensor provides the Work (and each
145 | Contributor provides its Contributions) on an "AS IS" BASIS,
146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 | implied, including, without limitation, any warranties or conditions
148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 | PARTICULAR PURPOSE. You are solely responsible for determining the
150 | appropriateness of using or redistributing the Work and assume any
151 | risks associated with Your exercise of permissions under this License.
152 |
153 | 8. Limitation of Liability. In no event and under no legal theory,
154 | whether in tort (including negligence), contract, or otherwise,
155 | unless required by applicable law (such as deliberate and grossly
156 | negligent acts) or agreed to in writing, shall any Contributor be
157 | liable to You for damages, including any direct, indirect, special,
158 | incidental, or consequential damages of any character arising as a
159 | result of this License or out of the use or inability to use the
160 | Work (including but not limited to damages for loss of goodwill,
161 | work stoppage, computer failure or malfunction, or any and all
162 | other commercial damages or losses), even if such Contributor
163 | has been advised of the possibility of such damages.
164 |
165 | 9. Accepting Warranty or Additional Liability. While redistributing
166 | the Work or Derivative Works thereof, You may choose to offer,
167 | and charge a fee for, acceptance of support, warranty, indemnity,
168 | or other liability obligations and/or rights consistent with this
169 | License. However, in accepting such obligations, You may act only
170 | on Your own behalf and on Your sole responsibility, not on behalf
171 | of any other Contributor, and only if You agree to indemnify,
172 | defend, and hold each Contributor harmless for any liability
173 | incurred by, or claims asserted against, such Contributor by reason
174 | of your accepting any such warranty or additional liability.
175 |
176 | END OF TERMS AND CONDITIONS
177 |
178 | APPENDIX: How to apply the Apache License to your work.
179 |
180 | To apply the Apache License to your work, attach the following
181 | boilerplate notice, with the fields enclosed by brackets "[]"
182 | replaced with your own identifying information. (Don't include
183 | the brackets!) The text should be enclosed in the appropriate
184 | comment syntax for the file format. We also recommend that a
185 | file or class name and description of purpose be included on the
186 | same "printed page" as the copyright notice for easier
187 | identification within third-party archives.
188 |
189 | Copyright [yyyy] [name of copyright owner]
190 |
191 | Licensed under the Apache License, Version 2.0 (the "License");
192 | you may not use this file except in compliance with the License.
193 | You may obtain a copy of the License at
194 |
195 | http://www.apache.org/licenses/LICENSE-2.0
196 |
197 | Unless required by applicable law or agreed to in writing, software
198 | distributed under the License is distributed on an "AS IS" BASIS,
199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 | See the License for the specific language governing permissions and
201 | limitations under the License.
202 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Db2 Magic Command for Jupyter Notebooks
2 |
3 | Jupyter notebooks include the ability to extend the syntax available within code blocks with a feature called [Magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) commands. Magic commands start with a percent sign `%` and provide a variety of features within a notebook environment, including modifying notebook behavior, executing OS commands, extending notebook functionality and with Db2 magic, a way of interacting with a Db2 database.
4 |
5 | Once you have loaded the Db2 magic commands into your notebook, you are able to query Db2 tables using standard SQL syntax:
6 |
7 | 
8 |
9 | Db2 magic commands provide a number of features that will simplify your use of Db2 databases, including:
10 |
11 | - Simplified connections to data sources
12 | - Ability to store result sets into Pandas dataframes
13 | - Create Db2 tables from Pandas dataframes and populate the tables with the data frame contents
14 | - Run the majority of DCL, DDL, and DML statements that are available in Db2
15 | - Create functions, stored procedures, and run a subset of administrative commands
16 | - Allow for parallel execution of SQL queries even on non-warehousing systems
17 | - And much more!
18 |
19 | ## Pre-requisites
20 |
21 | If you are running on a Jupyter notebook service (Watson Studio), you may already have some of these pre-requisites installed. You can check to see if the Db2 libraries are installed by running the following command in a Jupyter notebook code cell:
22 | ```
23 | import ibm_db
24 | ```
25 |
26 | If the command returns successfully, then you do not need to install `ibm_db` and can continue to the *Loading Db2 Magic Commands* section.
27 |
28 | If you have access to your Jupyter notebook environment, the Db2 Python client can be installed in one of three ways:
29 |
30 | - `python3 -m pip install ibm_db` or `pip install ibm_db`
31 | - `easy_install ibm_db`
32 | - `conda install ibm_db`
33 |
34 | Prior to running the installation you may want to run these commands to ensure the proper libraries are available for the Db2 drivers:
35 |
36 | - RHEL/CentOS `yum install python3-dev`
37 | - Ubuntu `apt-get install python3-dev`
38 |
39 | More detailed instructions can be found on the [Db2 Python Driver](https://github.com/ibmdb/python-ibmdb#inst) support page.
40 |
41 | ## Loading Db2 Magic Commands
42 |
43 | Once you have `ibm_db` installed, you will need to download the Db2 magic commands. The Db2 magic commands can be downloaded and placed directly to the directory that your Jupyter notebooks are stored in, or can be downloaded from within a Jupyter notebook.
44 |
45 | To load the Db2 magic commands into your notebook, run the following command in your Jupyter notebook:
46 | ```
47 | !wget https://raw.githubusercontent.com/IBM/db2-jupyter/master/db2.ipynb -O db2.ipynb
48 | ```
49 |
50 | Once you have loaded the Db2 magic commands into your notebook, you are able to query Db2 tables using standard SQL syntax:
51 | ```
52 | %run db2.ipynb
53 | %sql connect to sample
54 | %sql select * from employee
55 | ```
56 |
57 | # Support
58 |
59 | For any questions regarding the Db2 Magic commands, including any suggestions, general comments, or bug reports, please contact:
60 |
61 | * George Baklarz `baklarz@ca.ibm.com`
62 | * Phil Downey `phil.downey1@ibm.com`
63 |
64 | George & Phil
65 |
66 | ### Acknowledgements
67 |
68 | We would like to thank the following people who helped in early prototyping, testing, suggestions, and feedback on the Db2 Magic commands.
69 |
70 | * Peter Kohlmann
71 | * Dean Compher
72 |
73 | ## Db2 Magic Commands Help
74 |
75 | For more information on the Db2 Magic commands, refer the online [Db2 Magic Commands](https://ibm.github.io/db2-jupyter/) documentation. The
76 | section below is a summary of a few key features in the product.
77 |
78 | # DB2 Jupyter Notebook Extensions
79 |
80 | This code is imported as a Jupyter notebook extension in any notebooks you create with DB2 code in it. Place the following line of code in any notebook that you want to use these commands with:
81 |
82 | %run db2.ipynb
83 |
84 |
85 | This code defines a Jupyter/Python magic command called `%sql` which allows you to execute DB2 specific calls to
86 | the database. There are other packages available for manipulating databases, but this one has been specifically
87 | designed for demonstrating a number of the SQL features available in DB2.
88 |
89 | There are two ways of executing the `%sql` command. A single line SQL statement would use the
90 | line format of the magic command:
91 |
92 | %sql SELECT * FROM EMPLOYEE
93 |
94 | If you have a large block of sql then you would place the %%sql command at the beginning of the block and then
95 | place the SQL statements into the remainder of the block. Using this form of the `%%sql` statement means that the
96 | notebook cell can only contain SQL and no other statements.
97 |
98 | %%sql
99 | SELECT * FROM EMPLOYEE
100 | ORDER BY LASTNAME
101 |
102 | You can have multiple lines in the SQL block (`%%sql`). The default SQL delimiter is the semi-column (`;`).
103 | If you have scripts (triggers, procedures, functions) that use the semi-colon as part of the script, you
104 | will need to use the `-d` option to change the delimiter to an at "`@`" sign.
105 |
112 |
113 | The `%sql` command allows most DB2 commands to execute and has a special version of the CONNECT statement.
114 | A CONNECT by itself will attempt to reconnect to the database using previously used settings. If it cannot
115 | connect, it will prompt the user for additional information.
116 |
117 | The CONNECT command has the following format:
118 |
119 | %sql CONNECT TO <database> USER <userid> USING <password | ?> HOST <ip address> PORT <port number> [SSL <filename>]
120 |
121 | If you use a "`?`" for the password field, the system will prompt you for a password. This avoids typing the
122 | password as clear text on the screen. If a connection is not successful, the system will print the error
123 | message associated with the connect request.
124 |
125 | If the connection is successful, the parameters are saved on your system and will be used the next time you
126 | run a SQL statement, or when you issue the %sql CONNECT command with no parameters.
127 |
128 | In addition to the -d option, there are a several different options that you can specify at the beginning of
129 | the SQL:
130 |
131 | - `-d, -delim` - Change SQL delimiter to "`@`" from "`;`"
132 | - `-q, -quiet` - Quiet results - no messages returned from the function
133 | - `-r, -array` - Return the result set as an array of values instead of a dataframe
134 | - `-j` - Format the first character column of the result set as a JSON record
135 | - `-json` - Return result set as an array of JSON records
136 | - `-a, -all` - Return all rows in answer set and do not limit display
137 | - `-grid` - Display the results in a scrollable grid
138 | - `-e, -echo` - Any macro expansions are displayed in an output box
139 |
140 |
141 | You can pass python variables to the `%sql` command by using the `{}` braces with the name of the
142 | variable inbetween. Note that you will need to place proper punctuation around the variable in the event the
143 | SQL command requires it. For instance, the following example will find employee '000010' in the EMPLOYEE table.
144 |
145 | empno = '000010'
146 | %sql SELECT LASTNAME FROM EMPLOYEE WHERE EMPNO='{empno}'
147 |
148 |
149 | The other option is to use parameter markers. What you would need to do is use the name of the variable with a colon in front of it and the program will prepare the statement and then pass the variable to Db2 when the statement is executed. This allows you to create complex strings that might contain quote characters and other special characters and not have to worry about enclosing the string with the correct quotes. Note that you do not place the quotes around the variable even though it is a string.
150 |
151 |
152 | empno = '000020'
153 | %sql SELECT LASTNAME FROM EMPLOYEE WHERE EMPNO=:empno
154 |
155 |
156 | ## Development SQL
157 | The previous set of `%sql` and `%%sql` commands deals with SQL statements and commands that are run in an interactive manner. There is a class of SQL commands that are more suited to a development environment where code is iterated or requires changing input. The commands that are associated with this form of SQL are:
158 | - AUTOCOMMIT
159 | - COMMIT/ROLLBACK
160 | - PREPARE
161 | - EXECUTE
162 |
163 | Autocommit is the default manner in which SQL statements are executed. At the end of the successful completion of a statement, the results are committed to the database. There is no concept of a transaction where multiple DML/DDL statements are considered one transaction. The `AUTOCOMMIT` command allows you to turn autocommit `OFF` or `ON`. This means that the set of SQL commands run after the `AUTOCOMMIT OFF` command are executed are not committed to the database until a `COMMIT` or `ROLLBACK` command is issued.
164 |
165 | `COMMIT` (`WORK`) will finalize all transactions (`COMMIT`) to the database and `ROLLBACK` will undo all changes. If you issue a `SELECT` statement during the execution of your block, the results will reflect all of your changes. If you `ROLLBACK` the transaction, the changes will be lost.
166 |
167 | `PREPARE` is typically used in a situation where you want to repeatedly execute a SQL statement with different variables without incurring the SQL compilation overhead. For instance:
168 | ```
169 | x = %sql PREPARE SELECT LASTNAME FROM EMPLOYEE WHERE EMPNO=?
170 | for y in ['000010','000020','000030']:
171 | %sql execute :x using :y
172 | ```
173 | `EXECUTE` is used to execute a previously compiled statement.
174 |
175 | To retrieve the error codes that might be associated with any SQL call, the following variables are updated after every call:
176 |
177 | * SQLCODE
178 | * SQLSTATE
179 | * SQLERROR - Full error message retrieved from Db2
180 |
--------------------------------------------------------------------------------
/Table_of_Contents.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | ""
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "# Table of Contents\n",
15 | "Welcome to the Db2 Jupyter Notebook project. This project contains a number of folders that will let you explore the various functions and features of Db2 through the use of Jupyter notebooks. The Table of Contents below will link you to the various labs and software that are available within this project. Hopefully you find the material useful and we appreciate any feedback you may have on the tutorials or the software."
16 | ]
17 | },
18 | {
19 | "cell_type": "markdown",
20 | "metadata": {},
21 | "source": [
22 | "* [An Introduction to Jupyter Notebook and Python](#jupyter) \n",
23 | "\n",
24 | "* [Setting up a database connection](#connect)\n",
25 | "\n",
26 | "* [Using Db2 Magic commands](#magic)\n",
27 | "\n",
28 | "* [Db2 11.1 Features](#v11.1)\n",
29 | "\n",
30 | "* [Db2 11.5 Features and JSON Support](#v11.5)\n",
31 | "\n",
32 | "* [Installation Instructions](#install)"
33 | ]
34 | },
35 | {
36 | "cell_type": "markdown",
37 | "metadata": {},
38 | "source": [
39 | ""
40 | ]
41 | },
42 | {
43 | "cell_type": "markdown",
44 | "metadata": {},
45 | "source": [
46 | "## Introduction to Jupyter Notebook and Python\n",
47 | "\n",
48 | "If you are not familiar with the use of Jupyter notebooks or Python, the following notebooks will guide you through their usage.\n",
49 | "\n",
50 | "* [An Introduction to Jupyter Notebooks](./Db2_Jupyter_Tutorials/An_Introduction_to_Jupyter_Notebooks.ipynb)\n",
51 | "* [A Brief Introduction to Python](./Db2_Jupyter_Tutorials/An_Introduction_to_Python.ipynb)\n",
52 | "* [Db2 Python API Library and Examples](https://github.com/IBM/db2-python)"
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {},
58 | "source": [
59 | ""
60 | ]
61 | },
62 | {
63 | "cell_type": "markdown",
64 | "metadata": {},
65 | "source": [
66 | "## Connection Document\n",
67 | "\n",
68 | "All of the tutorials that are in this project require that you connect to Db2 before running any of the SQL examples. There is a connection document found in this project (`connection.ipynb`) that contains a single SQL statement:\n",
69 | "```sql\n",
70 | "%sql CONNECT TO SAMPLE USER DB2INST1 USING db2inst1\n",
71 | "```\n",
72 | "\n",
73 | "This is the default userid and password to connect to the `SAMPLE` Db2 database. If you have a different userid or require access to a different database, modify this one statement and execute it so that you do not to re-connect in any of the subsequent notebooks. \n",
74 | "\n",
75 | "* [Db2 Connection Document](connection.ipynb)"
76 | ]
77 | },
78 | {
79 | "cell_type": "markdown",
80 | "metadata": {},
81 | "source": [
82 | ""
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "metadata": {},
88 | "source": [
89 | "## Installation and Customization\n",
90 | "\n",
91 | "If you need to install Jupyter Notebook on your system, or want to create a Docker container that has Jupyter Notebook installed in it, review the following documents. The Docker container does not include Db2 in it, so you will still need access to a Db2 system. Instructions for creating a Db2 docker image are found on the Docker site (see the link below).\n",
92 | "\n",
93 | "* [Installing Jupyter Notebook and Db2 Extensions](./Installation/Installation_Instructions.ipynb)\n",
94 | "* [Using Docker to run Jupyter Notebook and Db2](./Installation/Docker_Instructions.ipynb)\n",
95 | "* [Installing Db2 in a Docker container](https://hub.docker.com/r/ibmcom/db2)"
96 | ]
97 | },
98 | {
99 | "cell_type": "markdown",
100 | "metadata": {},
101 | "source": [
102 | ""
103 | ]
104 | },
105 | {
106 | "cell_type": "markdown",
107 | "metadata": {},
108 | "source": [
109 | "## Db2 Magic Commands\n",
110 | "\n",
111 | "Db2 Magic commands allow you to access Db2 within a Jupyter notebook without having to use the underlying Python database drivers. These notebooks take you through the basics of using the Db2 magic commands as well as more complex SQL examples.\n",
112 | "\n",
113 | "* [An Introduction to Db2 Magic Commands](./Db2_Jupyter_Tutorials/Db2_Jupyter_Extensions_Tutorial.ipynb)\n",
114 | "* [Calling Stored Procedures in Db2](./Db2_Jupyter_Tutorials/Db2_Calling_Stored_Procedures.ipynb)\n",
115 | "* [Using Prepared Statements in your SQL](./Db2_Jupyter_Tutorials/Db2_Using_Prepared_Statements.ipynb)\n",
116 | "* [Writing Db2 Macros to Simplify your SQL](./Db2_Jupyter_Tutorials/Db2_Jupyter_Macros.ipynb)"
117 | ]
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "metadata": {},
122 | "source": [
123 | ""
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "metadata": {},
129 | "source": [
130 | "## Db2 Version 11.1 Features\n",
131 | "\n",
132 | "These set of notebooks cover some of the new SQL functionality that was delivered in Db2 11.1. Note that two of the notebooks cover the JSON functions that were added to Db2 in 10.5 and officially published in 11.1. These functions are used to support the MongoDB compatibility features in Db2 but are not part of the SQL standard. If you want to use the official JSON functions you should examine the notebooks in the Db2 11.5 section.\n",
133 | "\n",
134 | "* [Oracle, Open Source and Netezza SQL Compatibility Features](./Db2_11.1_Features/Db2_11.1_Compatibility_Features.ipynb)\n",
135 | "* [Regular Expressions](./Db2_11.1_Features/Db2_11.1_Regular_Expressions.ipynb)\n",
136 | "* [Row and Column Access Control](./Db2_11.1_Features/Db2_11.1_Row_and_Column_Access_Control.ipynb)\n",
137 | "* [Updated Statistical Functions](./Db2_11.1_Features/Db2_11.1_Statistical_Functions.ipynb)\n",
138 | "* [Time and Date Functions](./Db2_11.1_Features/Db2_11.1_Time_and_Date_Functions.ipynb)\n",
139 | "* [JSON Features (SYSTOOLS)](./Db2_11.1_Features/Db2_11.1_JSON_Features_SYSTOOLS.ipynb)\n",
140 | "* [Advanced Db2 JSON Techniques (SYSTOOLS)](./Db2_11.1_Features/Db2_11.1_Advanced_Db2_JSON_Techniques_SYSTOOLS.ipynb)\n",
141 | "* [Db2 Version 11 Highlights - **eBook**](./Db2_11.1_Features/Db2V11-ebook.pdf)"
142 | ]
143 | },
144 | {
145 | "cell_type": "markdown",
146 | "metadata": {},
147 | "source": [
148 | ""
149 | ]
150 | },
151 | {
152 | "cell_type": "markdown",
153 | "metadata": {},
154 | "source": [
155 | "## Db2 Version 11.5 Features\n",
156 | "\n",
157 | "These set of notebooks cover some of the new SQL functionality that was delivered in Db2 11.5. This set of notebooks include extensive information on the new ISO JSON functions. These functions were delivered late in the 11.1 release (FP4) but are being published here because of their importance.\n",
158 | "\n",
159 | "* [Oracle, Open Source and Netezza SQL Enhancements](./Db2_11.5_Features/Db2_11.5_SQL_Enhancements.ipynb)\n",
160 | "* [An Introduction to External Tables](./Db2_11.5_Features/Db2_11.5_External_Tables.ipynb)\n",
161 | "\n",
162 | "These notebooks on Db2 JSON are intended to be run in sequence. If you want a very quick overview of the functions, refer to Part 3 for a quick sample of what these JSON functions do.\n",
163 | "\n",
164 | "* [Part 1 - An Introduction to JSON](./Db2_11.5_Features/Db2_11.5_JSON_01_An_Introduction_to_JSON.ipynb)\n",
165 | "* [Part 2 - JSON Path Expressions](./Db2_11.5_Features/Db2_11.5_JSON_02_JSON_Path_Expressions.ipynb)\n",
166 | "* [Part 3 - Db2 ISO JSON Functions](./Db2_11.5_Features/Db2_11.5_JSON_03_Db2_ISO_JSON_Functions.ipynb)\n",
167 | "* [Part 4 - JSON Storage](./Db2_11.5_Features/Db2_11.5_JSON_04_JSON_Storage.ipynb)\n",
168 | "* [Part 5 - Inserting JSON Data](./Db2_11.5_Features/Db2_11.5_JSON_05_Inserting_JSON_Data.ipynb)\n",
169 | "* [Part 6 - JSON_EXISTS](./Db2_11.5_Features/Db2_11.5_JSON_06_JSON_EXISTS.ipynb)\n",
170 | "* [Part 7 - JSON_VALUE and JSON_QUERY](./Db2_11.5_Features/Db2_11.5_JSON_07_JSON_VALUE_and_JSON_QUERY.ipynb)\n",
171 | "* [Part 8 - JSON_TABLE](./Db2_11.5_Features/Db2_11.5_JSON_08_JSON_TABLE.ipynb)\n",
172 | "* [Part 9 - Publishing JSON Data](./Db2_11.5_Features/Db2_11.5_JSON_09_Publishing_JSON_Data.ipynb)\n",
173 | "* [Part 10 - Performance Optimization](./Db2_11.5_Features/Db2_11.5_JSON_010_Performance_Optimization.ipynb)\n",
174 | "* [Part 11 - Updating Documents](./Db2_11.5_Features/Db2_11.5_JSON_011_Updating_Documents.ipynb)\n",
175 | "* [Part 12 - Unnesting Arrays](./Db2_11.5_Features/Db2_11.5_JSON_012_Unnesting_Arrays.ipynb)\n",
176 | "* [Db2 Version 11 JSON Highlights **eBook**](./Db2_11.5_Features/Db2V11-JSON-ebook.pdf)"
177 | ]
178 | },
179 | {
180 | "cell_type": "markdown",
181 | "metadata": {},
182 | "source": [
183 | "## Db2 Columnar Technology\n",
184 | "This set of Jupyter notebooks demonstrate the performance advantages of using the DB2 BLU technology to run queries. The queries in these notebooks were generated by Cognos. The reports are divided into Small, Medium, Large and Dashboard reports with the classification based on the processing complexity and the amount of data returned. \n",
185 | "\n",
186 | "Each query is run against a row-organized and then a column-organized table. The results are returned and the performance of the queries are compared in the notebook.\n",
187 | "\n",
188 | "* [Small Reports](./Db2_BLU_Demonstration/CQW_Small_Reports.ipynb)\n",
189 | "* [Medium Report](./Db2_BLU_Demonstration/CQW_Medium_Reports.ipynb)\n",
190 | "* [Long Reports](./Db2_BLU_Demonstration/CQW_Large_Reports.ipynb)\n",
191 | "* [Dashboard Reports](./Db2_BLU_Demonstration/CQW_Dashboard_Reports.ipynb)"
192 | ]
193 | },
194 | {
195 | "cell_type": "markdown",
196 | "metadata": {},
197 | "source": [
198 | "## Digital Technical Engagement\n",
199 | "For more demos and technical content, visit our DTE site at [ibm.com/demos](https://ibm.com/demos)."
200 | ]
201 | },
202 | {
203 | "cell_type": "markdown",
204 | "metadata": {},
205 | "source": [
206 | "#### Questions and Comments: George Baklarz [baklarz@ca.ibm.com]"
207 | ]
208 | }
209 | ],
210 | "metadata": {
211 | "kernelspec": {
212 | "display_name": "Python 3",
213 | "language": "python",
214 | "name": "python3"
215 | },
216 | "language_info": {
217 | "codemirror_mode": {
218 | "name": "ipython",
219 | "version": 3
220 | },
221 | "file_extension": ".py",
222 | "mimetype": "text/x-python",
223 | "name": "python",
224 | "nbconvert_exporter": "python",
225 | "pygments_lexer": "ipython3",
226 | "version": "3.7.4"
227 | }
228 | },
229 | "nbformat": 4,
230 | "nbformat_minor": 2
231 | }
232 |
--------------------------------------------------------------------------------
/connection.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Db2 Connection Document"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "### Db2 Connection Statement\n",
15 | "This notebook contains the connect statement that will be used for connecting to Db2. The typical way of connecting to Db2 within a notebooks it to run the db2 notebook (`db2.ipynb`) and then issue the `%sql connect` statement:\n",
16 | "```sql\n",
17 | "%run db2.ipynb\n",
18 | "%sql connect to sample user ...\n",
19 | "```\n",
20 | "\n",
21 | "Rather than having to change the connect statement in every notebook, this one file can be changed and all of the other notebooks will use the value in here. Note that if you do reset a connection within a notebook, you will need to issue the `CONNECT` command again or run this notebook to re-connect.\n",
22 | "\n",
23 | "The `db2.ipynb` file is still used at the beginning of all notebooks to highlight the fact that we are using special code to allow Db2 commands to be issues from within Jupyter Notebooks."
24 | ]
25 | },
26 | {
27 | "cell_type": "code",
28 | "execution_count": null,
29 | "metadata": {},
30 | "outputs": [],
31 | "source": [
32 | "%sql CONNECT TO SAMPLE USER DB2INST1 USING db2inst1"
33 | ]
34 | },
35 | {
36 | "cell_type": "markdown",
37 | "metadata": {},
38 | "source": [
39 | "### Check that the EMPLOYEE and DEPARTMENT table exist\n",
40 | "A lot of the examples depend on these two tables existing in the database. These tables will be created for you if they don't already exist. Note that they will not overwrite the existing Db2 samples tables."
41 | ]
42 | },
43 | {
44 | "cell_type": "code",
45 | "execution_count": null,
46 | "metadata": {},
47 | "outputs": [],
48 | "source": [
49 | "%sql -sampledata"
50 | ]
51 | },
52 | {
53 | "cell_type": "markdown",
54 | "metadata": {},
55 | "source": [
56 | "#### Credits: IBM 2019, George Baklarz [baklarz@ca.ibm.com]"
57 | ]
58 | }
59 | ],
60 | "metadata": {
61 | "kernelspec": {
62 | "display_name": "Python 3",
63 | "language": "python",
64 | "name": "python3"
65 | },
66 | "language_info": {
67 | "codemirror_mode": {
68 | "name": "ipython",
69 | "version": 3
70 | },
71 | "file_extension": ".py",
72 | "mimetype": "text/x-python",
73 | "name": "python",
74 | "nbconvert_exporter": "python",
75 | "pygments_lexer": "ipython3",
76 | "version": "3.6.8"
77 | }
78 | },
79 | "nbformat": 4,
80 | "nbformat_minor": 2
81 | }
82 |
--------------------------------------------------------------------------------
/db2pot-legal.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
Disclaimer
4 |
Information, products, software, programs, and services are provided "as is," with no warranties or guarantees whatsoever. IBM expressly disclaims to the fullest extent permitted by law all express, implied, statutory, and other warranties, guarantees, or representations, including, without limitation, the warranties of merchantability, fitness for a particular purpose, and non-infringement of proprietary and intellectual property rights. Without
5 | limitation, IBM makes no warranty or guarantee that this web site will be uninterrupted, timely, secure, or error-free.
6 |
Test Data
7 |
The data used within this program has been derived from public sources, including the US Census Bureau (First names, Last names, Street names), and the US Postal Service (City, State, Zip code). The names, product names, credit cards, credit numbers in this program are fictitious. No identification with actual persons (living or deceased), places or products is intended or should be inferred.
8 |
Use of Personal Data
9 |
Clients are responsible for ensuring their own compliance with various laws and regulations, including the European Union General Data Protection Regulation. Clients are solely responsible for obtaining advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulations that may affect the clients' business and any actions the clients may need to take to comply with such laws and regulations.
10 |
This demonstration system is not designed to be used with any personal data and uploading of data is done at your own risk. There is no guarantee that at the termination of this proof of technology that the data will be completely erased and IBM takes no responsibility for data on
11 | this system. Use of this system is at your sole risk.
12 |
13 |
--------------------------------------------------------------------------------
/db2pot-refresh.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
Notebook Refresh
4 |
The Jupyter notebooks and lessons used in this lab are already found in the notebook directory. If you check the box below, the directory will be deleted and refreshed with the latest copy of the Db2 labs. If you hit cancel, nothing will happen.
5 |
Please make absolutely sure that you haven't placed anything of importance in the notebook directory because pressing OK will delete all of the files.
6 |
7 |
--------------------------------------------------------------------------------
/docs/command_options.md:
--------------------------------------------------------------------------------
1 | # Db2 Magic Options
2 |
3 | The previous section discussed options that are specific for `%sql` commands and are only valid during the execution of that statement. There are options available that impact the execution of the `%sql` statements and are discussed below.
4 |
5 | There are four options that can be set with the `%sql` command. These options are shown below with the default value shown in parentheses.
6 |
7 | * DISPLAY PANDAS | GRID (PANDAS)
8 |
9 | Display the results as a PANDAS dataframe (default) or as a scrollable GRID.
10 |
11 |
12 | * MAXROWS n (10)
13 |
14 | The maximum number of rows that will be displayed before summary information is shown. If the answer set is less than this number of rows, it will be completely shown on the screen. If the answer set is larger than this amount, only the first 5 rows and last 5 rows of the answer set will be displayed. If you want to display a very large answer set, you may want to consider using the grid option `-grid` to display the results in a scrollable table. If you really want to show all results, then setting `MAXROWS` to `-1` will return all output.
15 |
16 |
17 | * MAXGRID n (5)
18 |
19 | The maximum size of a grid display. When displaying a result set in a grid `-grid`, the default size of the display window is 5 rows. You can set this to a larger size so that more rows are shown on the screen. Note that the minimum size always remains at `5` which means that if the system is unable to display your maximum row size it will reduce the table display until it fits.
20 |
21 |
22 | * LIST
23 | Display the current settings.
24 |
25 |
26 | To set an option use either of the following commands:
27 | ```
28 | %sql SET OPTION value
29 | %sql OPTION setting value [setting value...]
30 | ```
31 |
32 | The `OPTION` command allows multiple options to be set with one command. When either command is executed, the current settings are displayed.
33 |
34 |
35 | For instance, to set the maximum display rows to 100:
36 | ```
37 | %sql SET MAXROWS 100
38 | ```
39 |
40 | ## Display Format
41 |
42 | The default display format for any data returned is Pandas. This example illustrates the standard output returned from a `SELECT` statement:
43 |
44 | 
45 |
46 | The Pandas output format will sometimes truncate the output (as shown above). You can adjust the number of rows that can be displayed from an answer set using the `MAXROWS` setting discussed below. This standard output may be sufficient for your requirements if you are only returning small answer sets (less than 100 rows to display). Otherwise, you may consider using the `-grid` option. The same answer set using a `-grid` display is shown below.
47 |
48 | 
49 |
50 | The advantage of the `-grid` option is that the entire answer set can be retrieved and reviewed in the scrollable window. This is not possible when using Pandas output. It is also not possible to display all rows in the Pandas dataframe due to memory limitations when writing to a Jupyter notebook cell. The grid control removes these restrictions by allowing the entire answer set to be viewed in a grid format.
51 |
52 | The `DISPLAY` option is used to set the default display format of an SQL answer set. The default setting is `PANDAS` which is the standard output format of a Pandas dataframe. To change the default display format use the `SET` command with the `DISPLAY` option:
53 | ```
54 | %sql SET DISPLAY [PANDAS | GRID]
55 | ```
56 |
57 | Setting this value to `GRID` will result in all answer sets being displayed using the interactive grid format.
58 | ```
59 | %sql SET DISPLAY GRID
60 | ```
61 |
62 | The display format is kept with the connection information. If you start up another session using the same connection information, the last display format will be used.
63 |
64 | ## Maximum Number of Rows
65 |
66 | The default number of rows that are displayed when using a Pandas dataframe is 10. An answer set will always display the first 5 rows and the last 5 rows of an answer set in a Jupyter notebook:
67 |
68 | 
69 |
70 | The missing rows are marked with a `...` in-between the top and bottom sections of the answer set. To increase the **default** display size, you can change the `MAXROWS` setting. The rows that are displayed depends on how many can fit within the `MAXROWS` settings. The rules are as follows:
71 |
72 | * If the number of rows is less than or equal to `MAXROWS`, then all rows will be displayed.
73 | * If the number of rows is greater than `MAXROWS` then only the top and bottom 5 rows are displayed.
74 | * If `MAXROWS` is too big for a Jupyter notebook cell, the cell will contain scroll bars allowing you to scroll through the results.
75 |
76 | The maximum value for `MAXROWS` is `100` to avoid memory issues with Jupyter notebooks.
77 |
78 | For the following examples, the `MAXROWS` setting is set to `5`.
79 | ```
80 | %sql SET MAXROWS 5
81 | ```
82 |
83 | The first example demonstrates when exactly 5 rows are returned from the `EMPLOYEE` table.
84 |
85 | 
86 |
87 | The next example demonstrates how the answer set is split when the number of rows exceeds 5.
88 |
89 | 
90 |
91 | If maximum value of `MAXROWS` is 100. As many rows possible will be displayed in the notebook. If the number of rows is too large, the cell will contain scrollbars so that you can see all rows that are returned.
92 |
93 | 
94 |
95 | If you want to scroll through large answer sets, you should consider using the `-grid` flag or set the `DISPLAY` option.
96 |
97 | ## Maximum Grid Size
98 |
99 | The grid control displays approximately 5 lines of data in a Jupyter notebook cell. The `MAXGRID` option can be used to change the number of rows that the grid control will display in a Jupyter notebook cell. The entire answer set can be reviewed using the scroll bars on the right side and bottom of the control.
100 |
101 | The following is the default view of the grid control.
102 |
103 | 
104 |
105 | To change the number of rows displayed, use the `MAXGRID` option:
106 | ```
107 | %sql SET MAXGRID 10
108 | ```
109 |
110 | The answer set will now display a minimum of 10 rows in the grid.
111 |
112 | 
113 |
114 | Note that the `MAXGRID` option does not affect the number of rows that are returned in an answer set.
115 |
116 | ## Thread Parallelism
117 |
118 | The Db2 magic commands have the ability to split a SQL statement into multiple threads to improve the time it takes to retrieve answer sets. This feature does not make Db2 more efficient - it provides a way to increase the utilization of Db2 and pipeline SQL statements so that less time is spent waiting for data to be retrieved. The section on parallelism explores this feature in more detail.
119 |
120 | The `THREADS` option tells Db2 the maximum of threads that it will be allowed to use when running the SQL. The SQL will be split (based on some range or key) and multithreaded into Db2. The `THREADS` value can be anything from zero (standard SQL call) to 12. The highest performance was found to occur when using 6 to 8 threads, but this will be entirely dependent on the SQL workload.
121 |
122 | To set the number of threads use the following syntax:
123 | ```
124 | %sql SET THREADS 0-12
125 | ```
126 |
127 | The `THREADS` option does not apply to standard SQL statements. This option will only be used when the `USING x SELECT...` syntax is detected in a `%sql` or `%%sql` block.
128 |
--------------------------------------------------------------------------------
/docs/connect.md:
--------------------------------------------------------------------------------
1 | # Connecting to Db2
2 |
3 | Before any SQL commands can be issued, a connection needs to be made to the Db2 database that you will be using. The connection can be done manually (through the use of the CONNECT command), or automatically when the first `%sql` command is issued.
4 |
5 | The Db2 magic command tracks whether a connection has occurred in the past and saves this information between notebooks and sessions. When you start up a notebook and issue a command, the program will reconnect to the database using your credentials from the last session. In the event that you have not connected before, the system will require that you execute the `CONNECT` command. To connect to a Db2 database you will need the following information:
6 |
7 | * Database name
8 | * Hostname
9 | * Db2 port number
10 | * Userid
11 | * Password
12 |
13 | You may also require two additional pieces of information:
14 |
15 | * SSL Security - Request that SSL connections be used with Db2 and optionally the location of the SSL file that contains required certificate information
16 | * SSL File - The certificate file (if required)
17 | * API Key - An API key that is provided to connect to Db2
18 | * Access Token - An access token that is provided to connect to Db2
19 | * DSN - A string that contains the ODBC connection string
20 |
21 | The information supplied will be stored in the directory that the notebooks are stored on. Once you have entered the information, the system will attempt to connect to the database for you, and then you can run all SQL scripts. More details on the `CONNECT `syntax will be found in a section below.
22 |
23 | If you have credentials available from Db2 on Cloud or CP4D, place the contents of the credentials into a variable and then use the `CONNECT CREDENTIALS ` syntax to connect to the database.
24 | ```
25 | db2blu = { "uid" : "xyz123456", ...}
26 | %sql CONNECT CREDENTIALS db2blu
27 | ```
28 |
29 | If the connection is successful using the credentials, the variable will be saved to disk so that you can connect from within another notebook using the same syntax.
30 |
31 | ## Connect Syntax
32 |
33 | The `CONNECT` command has four different options that are listed below. You must explicitly connect to a Db2 database for the first time. Subsequent connections are **not** required if you are connecting to the same database. When the Db2 magic command executes your SQL command, it checks to see whether you have already connected to the database. If not, it will check to see if there was a successful connection previously and will use those credentials to reconnect.
34 |
35 | You can also force a connection to the previous database by issuing the `CONNECT` command with no parameters.
36 |
37 | 
38 |
39 | ### Standard Connection
40 |
41 | The format of the `CONNECT` command is:
42 | ```
43 | %sql CONNECT TO database USER userid USING password | ?
44 | HOST ip_address PORT port_number
45 | [ SSL ] [ SSLFILE file ] [ APIKEY key ] [ ACCESSTOKEN token ]
46 | [ DNS "string" ]
47 | ```
48 |
49 | The required fields are:
50 |
51 | * Database - Database name you want to connect to
52 | * Hostname - `localhost` if Db2 is running on your own machine or the IP address or symbolic name of the server
53 | * PORT - The port to use for connecting to Db2. This is usually 50000.
54 |
55 | When connecting using an apikey or access token, you will need to provide the key value after the keyword. The SSL security settings are already preconfigured when using these settings.
56 |
57 | * APIKEY value - the value of the APIKEY
58 | * ACCESSTOKEN value - the value of the access token
59 |
60 | If not using APIKEY/ACCESSTOKEN, you will need to provide your credentials to connect:
61 |
62 | * Userid - The userid to use when connecting to the database.
63 | * Password - The password for the userid. Use a question mark `?` to be prompted to enter the password without displaying the text.
64 |
65 | There are two optional fields. The SSL option ensures that the transmission of the SQL and answer sets are encrypted "on the wire". SSL is typically required when using public networks to communicate to a Db2 database. For internal systems this may not be the case. If you are required to use SSL communication, you must specify `SSL` in the connection string. The use of an SSL port on Db2 (traditionally 50001) may require a certificate file.
66 |
67 | If your workstation has already been configured to use SSL communication, then only the `SSL` parameter is required. If you have been given access to a Db2 database and have been provided with an SSL certificate, then you will use the `SSLFILE` option. The certificate option requires the location and name of the file that contains the `SSL` certificate which is used for communicating with Db2. Normally this certificate would have been provided to you and placed onto your local file system.
68 |
69 | The format of the `SSLFILE` certificate option is `SSLFILE filename` where `filename` is the fully qualified location of the SSL file. You do not need to specify `SSL` as part of the connection string since it is assumed that you will be using `SSL` communication when you supply a certificate file.
70 |
71 | ```
72 | %sql CONNECT TO BLUDB USER BLUADMIN USING ? HOST 145.23.24.1 PORT 50001 SSLFILE /ssl/bludb.cert
73 | ```
74 |
75 | If you want to use an ODBC connection string, rather than using the CONNECT syntax, you can supply it by using the DSN option.
76 |
77 | * DSN "dsn string"
78 |
79 | The value in the `"dsn string"` will be passed to the connect routine without modification. A sample DSN string is found below:
80 |
81 | ```
82 | dsn = "DRIVER={IBM DB2 ODBC DRIVER};DATABASE=BLUDB;HOSTNAME=some.random.location.com;PORT=12345;PROTOCOL=TCPIP;ConnectTimeout=15;LONGDATACOMPAT=1;AUTHENTICATION=GSSplugin;SECURITY=SSL;APIKEY=lKJH896HUGY102938unad9eYtw;"
83 |
84 | %sql CONNECT DSN "{dsn}"
85 | ```
86 |
87 | Note how the DSN string must be surrounded in quotes in the `CONNECT` command.
88 |
89 | Details of all Db2 CLI settings can be found in the [Db2 ODBC/CLI documentation](https://www.ibm.com/docs/en/db2/11.5?topic=odbc-cliodbc-configuration-keywords).
90 |
91 |
92 | ### Passwords
93 |
94 | If you use a `?` for the password field, the system will prompt you for a password. This avoids typing the password as clear text on the screen. If a connection is not successful, the system will print the error message associated with the connect request.
95 |
96 | 
97 |
98 | ### Connection with Credentials
99 |
100 | The `CREDENTIALS` option allows you to use credentials that are supplied by Db2 on Cloud instances. The credentials can be supplied as a variable and if successful, the variable will be saved to disk for future use. If you create another notebook and use the identical syntax, if the variable is not defined, the contents on disk will be used as the credentials. You should assign the credentials to a variable that represents the database (or schema) that you are communicating with. Using familiar names makes it easier to remember the credentials when connecting.
101 |
102 | Credentials must be supplied as a JSON dictionary that contains the following information:
103 | ```
104 | {
105 | "database" : "bludb",
106 | "hostname" : "somesite-txn-blue.net",
107 | "port" : 50000,
108 | "pwd" : "somethingrandom",
109 | "uid" : "myuserid",
110 | "ssl" : "SSL",
111 | "sslfile" : "filename",
112 | "apikey" : "apikey",
113 | "accesstoken" : "accesstoken",
114 | "dsn" : "dsn"
115 | }
116 | ```
117 | Some of these fields are not necessary. For instance, the apikey, accesstoken, and dsn fields are only required if you are using these forms of connections.
118 |
119 | To use this as a connection document, assign this dictionary value to a variable:
120 | ```
121 | bludb = {
122 | "hostname": "somesite-txn-blue.net",
123 | "pwd": "somethingrandom",...
124 | }
125 | ```
126 |
127 | Then use the following syntax to connect to the database:
128 |
129 | ```
130 | %sql CONNECT CREDENTIALS bludb
131 | ```
132 |
133 | The system will attempt to connect to the database using the following process:
134 |
135 | 1. If the `bludb` variable exists in your Jupyter notebook, it will use the contents of that variable to connect to the database.
136 | 2. If `bludb` does not exist, the program will check the local directory that the notebook is in to find a file called `bludb.pickle` which contains the credentials.
137 | 3. If neither of the above exist the program will issue a connection error.
138 |
139 | If a connection was successfully made, the credentials are written to disk using the name of the variable as the file name and the file type being `pickle`. A `pickle` file is a Python format that allows structures and data types to be written to a file and easily read back in.
140 |
141 | Note that if you want to use the traditional `CONNECT` syntax, you can extract the values required from the credentials file. You will need the following:
142 |
143 | * `db` - Database name
144 | * `uid` - Connection userid
145 | * `pwd` - Connection password
146 | * `hostname` - Host
147 | * `port` - Port number
148 | * `SSL` - Add this keyword if you need to connect via SSL
149 |
150 | ### Closing a Connection
151 |
152 | `CONNECT CLOSE` will close the current connection, but will not reset the database parameters. This means that if you issue the `CONNECT` command again, the system will reconnect you to the database.
153 | ```
154 | %sql CONNECT CLOSE
155 | ```
156 |
157 | If you issue another SQL statement after closing the connection, the program will reconnect to last database you were connected to. If you don't want this behavior, use the `CONNECT RESET` command.
158 |
159 | ### Resetting a Connection
160 |
161 | `CONNECT RESET` will close the current connection and remove any information on the connection. You will need to issue a new `CONNECT `statement with the connection information.
162 | ```
163 | %sql CONNECT RESET
164 | ```
165 |
166 | Issuing a SQL statement after a `CONNECT RESET` will result in a connection error.
167 |
168 | ### Debugging Connection Strings
169 |
170 | If you need to see the connection string that is generated by the `CONNECT` command, use the `-e` flag as part of the connection:
171 | ```
172 | %sql -e connect to sample user db2inst1 using db2inst1
173 | DRIVER={IBM DB2 ODBC DRIVER};DATABASE=SAMPLE;HOSTNAME=localhost;PORT=50000;UID=db2inst1;PWD=db2inst1;PROTOCOL=TCPIP;ConnectTimeout=15;LONGDATACOMPAT=1;
174 | ```
175 |
176 | The program will **not** attempt to connect to the database, but will print the connection string that it was going to use. You can then modify this string to include any optional settings you require and then use the `DSN` option on the connect command to force the DSN value to be used instead of what the Db2 Magic command generates.
177 | ```
178 | dsn = "DRIVER={IBM DB2 ODBC DRIVER};UID=db2inst1;DATABASE=BLUDB;PWD=password;HOSTNAME=some.random.location.com;PORT=12345;"
179 |
180 | %sql CONNECT DSN "{dsn}"
181 | ```
--------------------------------------------------------------------------------
/docs/extramacros.md:
--------------------------------------------------------------------------------
1 | # Predefined SQL Macros
2 |
3 | There are four `%sql` commands that are implemented as macros. These commands are:
4 |
5 | * `SAMPLEDATA`
6 | * `LIST TABLES`
7 | * `DESCRIBE`
8 | * `SET