├── assets └── aa.png ├── module_demo ├── img │ └── reg.png └── my_module.py ├── final_assignment.md ├── aa_source_data.md ├── starter_code_parse_xw_5letter.ipynb ├── week03assignment.md ├── week04_starter.ipynb ├── intro_python.ipynb └── README.md /assets/aa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/visualizedata/data-structures/HEAD/assets/aa.png -------------------------------------------------------------------------------- /module_demo/img/reg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/visualizedata/data-structures/HEAD/module_demo/img/reg.png -------------------------------------------------------------------------------- /module_demo/my_module.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | '''This is my simple regression module''' 3 | 4 | class SimpleRegression(): 5 | '''Fits OLS simple regression''' 6 | 7 | def __init__(self): 8 | '''Initialize attributes''' 9 | self.intercept_ = False 10 | self.coef_ = False 11 | 12 | def get_b(self, X, Y): 13 | '''Calculate the coefficient / slope''' 14 | Xbar = np.average(X) 15 | Ybar = np.average(Y) 16 | X_minus_Xbar = np.subtract(X, [Xbar]) 17 | Y_minus_Ybar = np.subtract(Y, [Ybar]) 18 | X_minus_Xbar_squared = np.power(X_minus_Xbar, 2) 19 | sum_X_minus_Xbar_times_Y_minus_Ybar = np.vdot(X_minus_Xbar, Y_minus_Ybar) 20 | sum_X_minus_Xbar_squared = np.sum(X_minus_Xbar_squared) 21 | b = sum_X_minus_Xbar_times_Y_minus_Ybar / sum_X_minus_Xbar_squared 22 | return b 23 | 24 | def get_a(self, X, Y): 25 | '''Calculate the intercept''' 26 | Xbar = np.average(X) 27 | Ybar = np.average(Y) 28 | a = Ybar - self.get_b(X, Y) * Xbar 29 | return a 30 | 31 | def fit(self, X, Y): 32 | '''Fit the OLS model''' 33 | self.intercept_ = self.get_a(X, Y) 34 | self.coef_ = self.get_b(X, Y) 35 | 36 | def predict(self, X): 37 | '''Predict Y''' 38 | '''Return an array of the predicted values for X based on the model you fit''' 39 | return np.add([self.intercept_], np.multiply([self.coef_], X)) -------------------------------------------------------------------------------- /final_assignment.md: -------------------------------------------------------------------------------- 1 | # Final Assignment 2 | 3 | You will expand on the weekly assignments to scrape all ten zones of New York's AA Meeting List to capture, clean, and store all meetings in Manhattan (zones one through ten). The meetings [data should be read into DuckDB in an Observable Notebook and queried using SQL. You will use MapBox](https://observablehq.com/@aaronxhill/integrating-duckdb-user-inputs-and-mapbox) (or similar mapping services) to display relevant meetings as map markers with popups that show all relevant information about the meeting(s) at each marker. 4 | 5 | The ten "Meeting List Agenda" pages for Manhattan are available at: 6 | ``` 7 | https://parsons.nyc/aa/m01.html 8 | https://parsons.nyc/aa/m02.html 9 | https://parsons.nyc/aa/m03.html 10 | https://parsons.nyc/aa/m04.html 11 | https://parsons.nyc/aa/m05.html 12 | https://parsons.nyc/aa/m06.html 13 | https://parsons.nyc/aa/m07.html 14 | https://parsons.nyc/aa/m08.html 15 | https://parsons.nyc/aa/m09.html 16 | https://parsons.nyc/aa/m10.html 17 | ``` 18 | 19 | ### Some questions that may arise as you bind the data to the end-user interface: 20 | 21 | 1. What information does the end user need? How? Why? 22 | 2. From the data on AA's meeting list, which data is relevant for display in this project? How should it be displayed? 23 | 3. What does a map marker represent? A meeting group? A meeting? A location? 24 | 4. What is the minimum amount of data that can be queried to provide the necessary data for the visual representation? 25 | 26 | ### Checklist of Deliverables: 27 | 28 | 1. All work for the final assignment submission should be included in an Observable Notebook. To include other files (including code, documentation, context, etc.), link to them in your Observable Notebook. 29 | 2. The Observable Notebook should have a reasonable bare minimum of relevant user inputs for the parameters you wish to include in your map/visualization design. 30 | 3. The values for those user inputs should be passed to your DuckDB query. 31 | 4. The result of the DuckDB query should populate the map markers as well as any metadata that will be shown with each of the map markers. 32 | 5. Anyone arriving at your Observable Notebook should be able to find and adjust user inputs and get a refreshed map/visualization showing meetings and data for the meetings that match the criteria set by the user. 33 | 34 | ### Your work will be assessed on: 35 | 36 | 1. The integrity of the data 37 | 2. The integrity of the database 38 | 3. The efficiency of the queries and page load 39 | 4. The choices of data structures 40 | 5. The inclusion of relevant data 41 | 6. The coherence and organization of your code and repository 42 | 7. The method for binding the data to the visual representation -------------------------------------------------------------------------------- /aa_source_data.md: -------------------------------------------------------------------------------- 1 | # Source ("raw") data: 2 | 3 | ### *What is A.A.?* 4 | 5 | *Alcoholics Anonymous is a fellowship of people who come together to solve their drinking problem. It doesn’t cost anything to attend A.A. meetings. There are no age or education requirements to participate. Membership is open to anyone who wants to do something about their drinking problem.* 6 | 7 | *A.A.’s primary purpose is to help alcoholics to achieve sobriety.* 8 | 9 | Source: https://www.aa.org/what-is-aa 10 | 11 | ### AA meetings in Manhattan: 12 | 13 | https://parsons.nyc/aa/m01.html 14 | https://parsons.nyc/aa/m02.html 15 | https://parsons.nyc/aa/m03.html 16 | https://parsons.nyc/aa/m04.html 17 | https://parsons.nyc/aa/m05.html 18 | https://parsons.nyc/aa/m06.html 19 | https://parsons.nyc/aa/m07.html 20 | https://parsons.nyc/aa/m08.html 21 | https://parsons.nyc/aa/m09.html 22 | https://parsons.nyc/aa/m10.html 23 | 24 | ## Map of AA zones 25 | 26 | ![](https://github.com/visualizedata/data-structures/raw/master/assets/aa.png) 27 | 28 | ### raw data for parsing exercise: 29 | 30 | ``` 31 | ` 32 | 33 | 34 |

Union Baptist Church


35 | AA IT WORKS II - AA It Works II
36 | 240 West 145th Street, 37 |
(Betw 7th & 8th Avenues) NY 10039 38 |
39 |
40 | 41 |
42 | Thu=Altern.BB, Living Sober & Other A.A. Literature.
Fri.=Alternates between Step & Tradition. 43 |
44 | 45 | 46 | Wheelchair AccessWheelchair access 47 | 48 | 49 | 50 | 51 | 52 | 53 | Wednesdays From 10:00 AM to 11:00 AM
Meeting Type B = Beginners meeting 54 |
55 |
56 | 57 | Thursdays From 10:00 AM to 11:00 AM
Meeting Type BB = Big Book meeting 58 |
59 |
60 | 61 | Fridays From 10:00 AM to 11:00 AM
Meeting Type S = Step meeting 62 |
63 |
64 | 65 | Tuesdays From 10:00 AM to 11:00 AM
Meeting Type OD = Open Discussion meeting 66 |
67 |
68 | 69 | 70 | 71 | Get Directions 72 | 73 | 74 | ` 75 | ``` -------------------------------------------------------------------------------- /starter_code_parse_xw_5letter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": { 21 | "id": "P3zWHmLiDVdX" 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "# imports\n", 26 | "import urllib.request\n", 27 | "from bs4 import BeautifulSoup\n", 28 | "import re" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "source": [ 34 | " # assign the URL to variable\n", 35 | "url = \"https://nyxcrossword.com\"\n", 36 | "\n", 37 | "# open and read the the HTML\n", 38 | "text = urllib.request.urlopen(url).read()\n", 39 | "\n", 40 | "# parse the HTML in BeautifulSoup\n", 41 | "soup = BeautifulSoup(text, 'html.parser')" 42 | ], 43 | "metadata": { 44 | "id": "La5R4W-5Dhy8" 45 | }, 46 | "execution_count": 2, 47 | "outputs": [] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "source": [ 52 | "for clue in soup.find_all('h4'):\n", 53 | " print(clue)" 54 | ], 55 | "metadata": { 56 | "colab": { 57 | "base_uri": "https://localhost:8080/" 58 | }, 59 | "id": "PGXYROC5DkaE", 60 | "outputId": "b42f6d03-599a-41ef-bb21-a829a48d44e4" 61 | }, 62 | "execution_count": 3, 63 | "outputs": [ 64 | { 65 | "output_type": "stream", 66 | "name": "stdout", 67 | "text": [ 68 | "

\n", 69 | "\t1\tStuff in a party bag\t : \tSWAG\t

\n", 70 | "

\t5\tShade of black\t : \tJET\t

\n", 71 | "

\t19\tParadoxical answer to “What is ‘Golf’ in the NATO alphabet?”\t : \tGEE, I DON’T KNOW\t

\n", 72 | "

\t23\tEmcees’ deliveries\t : \tINTROS\t

\n", 73 | "

\t27\tParadoxical answer to “Can you say what ‘nyet’ is Russian for?”\t : \tNO, LET ME THINK\t

\n", 74 | "

\t33\tOverhead light?\t : \tHALO\t

\n", 75 | "

\t34\tArchitect who lived to be 102\t : \tPEI\t

\n", 76 | "

\t40\tSuffix with cyclo-\t : \t-TRON\t

\n", 77 | "

\t46\tComedian Aziz\t : \tANSARI\t

\n", 78 | "

\t50\tParadoxical answer to “What isle is located between Ireland and Great Britain?”\t : \tMAN, THAT’S HARD\t

\n", 79 | "

\t54\tView from a periscope\t : \tOCEAN\t

\n", 80 | "

\t59\tLa Campanella composer\t : \tLISZT\t

\n", 81 | "

\t61\tLondon transport, with “the”\t : \t… TUBE\t

\n", 82 | "

\t\t5\tWoman in a Lady Gaga song and album title (2016)\t : \tJOANNE\t

\n", 83 | "

\t7\tScuba need\t : \tTANK\t

\n", 84 | "

\t12\tHomer calls him “stupid Flanders”\t : \tNED\t

\n", 85 | "

\t14\tPart of a tape cassette\t : \tSPOOL\t

\n", 86 | "

\t20\tSinger known as the “Queen of Power Ballads”\t : \tDION\t

\n", 87 | "

\t21\tPhysics Nobelist Bohr\t : \tNIELS\t

\n", 88 | "

\t26\tS.N.L. routines\t : \tSKITS\t

\n", 89 | "

\t28\tBikini style\t : \tTHONG\t

\n", 90 | "

\t30\tStart or end for Alexa?\t : \tSCHWA\t

\n", 91 | "

\t37\tBaseball’s Gehrig\t : \tLOU\t

\n", 92 | "

\t41\tModern warfare concern, in brief\t : \tWMDS\t

\n", 93 | "

\t43\tDelivery class?\t : \tLAMAZE\t

\n", 94 | "

\t44\tMalia and Michelle, for two\t : \tOBAMAS\t

\n", 95 | "

\t53\tContacted privately on social media, informally\t : \tDM’ED\t

\n", 96 | "

\t55\tThe World Factbook org.\t : \tCIA\t

\n", 97 | "

\t56\tIs, to Ovid\t : \tEST\t

\n" 98 | ] 99 | } 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "source": [], 105 | "metadata": { 106 | "id": "ZjVwz--sDpjM" 107 | }, 108 | "execution_count": null, 109 | "outputs": [] 110 | } 111 | ] 112 | } -------------------------------------------------------------------------------- /week03assignment.md: -------------------------------------------------------------------------------- 1 | ``` 2 | 3 | 4 |

SRO Building


5 | A CHANCE TO LIVE AGAIN - A Chance To Live Again
6 | 109 West 129th Street, Basement, 7 |
(@ Lenox Avenue) NY 10027 8 |
9 |
10 | 11 | 12 | Wheelchair AccessWheelchair access 13 | 14 | 15 | 16 | 17 | 18 | 19 | Wednesdays From 6:00 PM to 7:00 PM
Meeting Type BB = Big Book meeting 20 |
21 |
22 | 23 | Mondays From 7:00 PM to 8:00 PM
Meeting Type B = Beginners meeting 24 |
25 |
26 | 27 | 28 | 29 | Get Directions 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 |

Union Baptist Church


38 | AA IT WORKS II - AA It Works II
39 | 240 West 145th Street, 40 |
(Betw 7th & 8th Avenues) NY 10039 41 |
42 |
43 | 44 |
45 | Thu=Altern.BB, Living Sober & Other A.A. Literature.
Fri.=Alternates between Step & Tradition. 46 |
47 | 48 | 49 | Wheelchair AccessWheelchair access 50 | 51 | 52 | 53 | 54 | 55 | 56 | Wednesdays From 10:00 AM to 11:00 AM
Meeting Type B = Beginners meeting 57 |
58 |
59 | 60 | Thursdays From 10:00 AM to 11:00 AM
Meeting Type BB = Big Book meeting 61 |
62 |
63 | 64 | Fridays From 10:00 AM to 11:00 AM
Meeting Type S = Step meeting 65 |
66 |
67 | 68 | Tuesdays From 10:00 AM to 11:00 AM
Meeting Type OD = Open Discussion meeting 69 |
70 |
71 | 72 | 73 | 74 | Get Directions 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 |

Our Lady of Lourdes Church


83 | ATTITUDE ADJUSTMENT - Attitude Adjustment
84 | 469 West 142nd Street, Basement, 85 |
(Betw Convent & Amsterdam Avenues) NY 10031 86 |
87 |
88 | 89 |
90 | Topic 1st Saturday. 91 |
92 | 93 | 94 | 95 | 96 | 97 | Saturdays From 7:30 AM to 8:30 AM
Meeting Type OD = Open Discussion meeting 98 |
99 |
100 | 101 | 102 | 103 | Get Directions 104 | 105 | 106 | ``` -------------------------------------------------------------------------------- /week04_starter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "code", 19 | "execution_count": 5, 20 | "metadata": { 21 | "id": "jbOhI5xnV7co" 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "m1 = \"\"\"\n", 26 | " \n", 27 | " \n", 28 | " \t

SRO Building


\n", 29 | "\t\t\t\t \t A CHANCE TO LIVE AGAIN - A Chance To Live Again
\n", 30 | "\t\t\t\t\t\t109 West 129th Street, Basement,\n", 31 | "\t\t\t\t\t\t
(@ Lenox Avenue) NY 10027\n", 32 | "\t\t\t\t\t\t
\n", 33 | "\t\t\t\t\t\t
\n", 34 | "\n", 35 | "\t\t\t\t\t\t\n", 36 | " \"WheelchairWheelchair access\n", 37 | " \n", 38 | "\n", 39 | "\n", 40 | " \n", 41 | " \n", 42 | "\n", 43 | "\t\t\t\t \t Wednesdays From 6:00 PM to 7:00 PM
Meeting Type BB = Big Book meeting\n", 44 | "\t\t\t \t\t\t
\n", 45 | " \t
\n", 46 | "\n", 47 | "\t\t\t\t \t Mondays From 7:00 PM to 8:00 PM
Meeting Type B = Beginners meeting\n", 48 | "\t\t\t \t\t\t
\n", 49 | " \t
\n", 50 | "\n", 51 | "\t\t\t\t\t\n", 52 | " \n", 53 | " \tGet Directions\n", 54 | " \n", 55 | " \n", 56 | "\"\"\"" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "source": [ 62 | "print(m1)" 63 | ], 64 | "metadata": { 65 | "colab": { 66 | "base_uri": "https://localhost:8080/" 67 | }, 68 | "id": "Vy7YBdSHWlBP", 69 | "outputId": "864790c3-086a-4b1f-f160-7ab79ded66dc" 70 | }, 71 | "execution_count": 6, 72 | "outputs": [ 73 | { 74 | "output_type": "stream", 75 | "name": "stdout", 76 | "text": [ 77 | "\n", 78 | " \n", 79 | " \n", 80 | " \t

SRO Building


\n", 81 | "\t\t\t\t \t A CHANCE TO LIVE AGAIN - A Chance To Live Again
\n", 82 | "\t\t\t\t\t\t109 West 129th Street, Basement, \n", 83 | "\t\t\t\t\t\t
(@ Lenox Avenue) NY 10027\n", 84 | "\t\t\t\t\t\t
\n", 85 | "\t\t\t\t\t\t
\n", 86 | " \n", 87 | "\t\t\t\t\t\t\n", 88 | " \"WheelchairWheelchair access\n", 89 | " \n", 90 | "\t\t\t \t\t\t\n", 91 | "\t\t\t\t\t\t\n", 92 | " \n", 93 | " \n", 94 | " \t \t\n", 95 | "\t\t\t\t \t Wednesdays From 6:00 PM to 7:00 PM
Meeting Type BB = Big Book meeting \n", 96 | "\t\t\t \t\t\t
\n", 97 | " \t
\n", 98 | " \t\n", 99 | "\t\t\t\t \t Mondays From 7:00 PM to 8:00 PM
Meeting Type B = Beginners meeting \n", 100 | "\t\t\t \t\t\t
\n", 101 | " \t
\n", 102 | " \t\t\n", 103 | "\t\t\t\t\t \n", 104 | " \n", 105 | " \tGet Directions\n", 106 | " \n", 107 | " \n", 108 | "\n" 109 | ] 110 | } 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "source": [ 116 | "m2 = \"\"\"\n", 117 | " \n", 118 | " \n", 119 | " \t

Union Baptist Church


\n", 120 | "\t\t\t\t \t AA IT WORKS II - AA It Works II
\n", 121 | "\t\t\t\t\t\t240 West 145th Street,\n", 122 | "\t\t\t\t\t\t
(Betw 7th & 8th Avenues) NY 10039\n", 123 | "\t\t\t\t\t\t
\n", 124 | "\t\t\t\t\t\t
\n", 125 | "\n", 126 | "
\n", 127 | " \tThu=Altern.BB, Living Sober & Other A.A. Literature.
Fri.=Alternates between Step & Tradition.\n", 128 | "
\n", 129 | "\n", 130 | "\t\t\t\t\t\t\n", 131 | " \"WheelchairWheelchair access\n", 132 | " \n", 133 | "\n", 134 | "\n", 135 | " \n", 136 | " \n", 137 | "\n", 138 | "\t\t\t\t \t Wednesdays From 10:00 AM to 11:00 AM
Meeting Type B = Beginners meeting\n", 139 | "\t\t\t \t\t\t
\n", 140 | " \t
\n", 141 | "\n", 142 | "\t\t\t\t \t Thursdays From 10:00 AM to 11:00 AM
Meeting Type BB = Big Book meeting\n", 143 | "\t\t\t \t\t\t
\n", 144 | " \t
\n", 145 | "\n", 146 | "\t\t\t\t \t Fridays From 10:00 AM to 11:00 AM
Meeting Type S = Step meeting\n", 147 | "\t\t\t \t\t\t
\n", 148 | " \t
\n", 149 | "\n", 150 | "\t\t\t\t \t Tuesdays From 10:00 AM to 11:00 AM
Meeting Type OD = Open Discussion meeting\n", 151 | "\t\t\t \t\t\t
\n", 152 | " \t
\n", 153 | "\n", 154 | "\t\t\t\t\t\n", 155 | " \n", 156 | " \tGet Directions\n", 157 | " \n", 158 | " \n", 159 | "\"\"\"" 160 | ], 161 | "metadata": { 162 | "id": "FLlPDaQqV_aY" 163 | }, 164 | "execution_count": 7, 165 | "outputs": [] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "source": [ 170 | "print(m2)" 171 | ], 172 | "metadata": { 173 | "colab": { 174 | "base_uri": "https://localhost:8080/" 175 | }, 176 | "id": "yZwy1fFEWoMY", 177 | "outputId": "92b5b6cc-20f7-4e32-b5c8-6ecb16f7e1c5" 178 | }, 179 | "execution_count": 8, 180 | "outputs": [ 181 | { 182 | "output_type": "stream", 183 | "name": "stdout", 184 | "text": [ 185 | "\n", 186 | " \n", 187 | " \n", 188 | " \t

Union Baptist Church


\n", 189 | "\t\t\t\t \t AA IT WORKS II - AA It Works II
\n", 190 | "\t\t\t\t\t\t240 West 145th Street, \n", 191 | "\t\t\t\t\t\t
(Betw 7th & 8th Avenues) NY 10039\n", 192 | "\t\t\t\t\t\t
\n", 193 | "\t\t\t\t\t\t
\n", 194 | " \n", 195 | "
\n", 196 | " \tThu=Altern.BB, Living Sober & Other A.A. Literature.
Fri.=Alternates between Step & Tradition. \n", 197 | "
\n", 198 | " \n", 199 | "\t\t\t\t\t\t\n", 200 | " \"WheelchairWheelchair access\n", 201 | " \n", 202 | "\t\t\t \t\t\t\n", 203 | "\t\t\t\t\t\t\n", 204 | " \n", 205 | " \n", 206 | " \t \t\n", 207 | "\t\t\t\t \t Wednesdays From 10:00 AM to 11:00 AM
Meeting Type B = Beginners meeting \n", 208 | "\t\t\t \t\t\t
\n", 209 | " \t
\n", 210 | " \t\n", 211 | "\t\t\t\t \t Thursdays From 10:00 AM to 11:00 AM
Meeting Type BB = Big Book meeting \n", 212 | "\t\t\t \t\t\t
\n", 213 | " \t
\n", 214 | " \t\n", 215 | "\t\t\t\t \t Fridays From 10:00 AM to 11:00 AM
Meeting Type S = Step meeting \n", 216 | "\t\t\t \t\t\t
\n", 217 | " \t
\n", 218 | " \t\n", 219 | "\t\t\t\t \t Tuesdays From 10:00 AM to 11:00 AM
Meeting Type OD = Open Discussion meeting \n", 220 | "\t\t\t \t\t\t
\n", 221 | " \t
\n", 222 | " \t\t\n", 223 | "\t\t\t\t\t \n", 224 | " \n", 225 | " \tGet Directions\n", 226 | " \n", 227 | " \n", 228 | "\n" 229 | ] 230 | } 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "source": [ 236 | "m3 = \"\"\"\n", 237 | " \n", 238 | " \n", 239 | " \t

Our Lady of Lourdes Church


\n", 240 | "\t\t\t\t \t ATTITUDE ADJUSTMENT - Attitude Adjustment
\n", 241 | "\t\t\t\t\t\t469 West 142nd Street, Basement,\n", 242 | "\t\t\t\t\t\t
(Betw Convent & Amsterdam Avenues) NY 10031\n", 243 | "\t\t\t\t\t\t
\n", 244 | "\t\t\t\t\t\t
\n", 245 | "\n", 246 | "
\n", 247 | " \tTopic 1st Saturday.\n", 248 | "
\n", 249 | "\n", 250 | "\n", 251 | " \n", 252 | " \n", 253 | "\n", 254 | "\t\t\t\t \t Saturdays From 7:30 AM to 8:30 AM
Meeting Type OD = Open Discussion meeting\n", 255 | "\t\t\t \t\t\t
\n", 256 | " \t
\n", 257 | "\n", 258 | "\t\t\t\t\t\n", 259 | " \n", 260 | " \tGet Directions\n", 261 | " \n", 262 | " \n", 263 | "\"\"\"" 264 | ], 265 | "metadata": { 266 | "id": "_SHedmHuWciy" 267 | }, 268 | "execution_count": 9, 269 | "outputs": [] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "source": [ 274 | "print(m3)" 275 | ], 276 | "metadata": { 277 | "colab": { 278 | "base_uri": "https://localhost:8080/" 279 | }, 280 | "id": "tZXbzI97Wjai", 281 | "outputId": "3e457ec9-4680-4701-8c6c-802931191036" 282 | }, 283 | "execution_count": 10, 284 | "outputs": [ 285 | { 286 | "output_type": "stream", 287 | "name": "stdout", 288 | "text": [ 289 | "\n", 290 | " \n", 291 | " \n", 292 | " \t

Our Lady of Lourdes Church


\n", 293 | "\t\t\t\t \t ATTITUDE ADJUSTMENT - Attitude Adjustment
\n", 294 | "\t\t\t\t\t\t469 West 142nd Street, Basement, \n", 295 | "\t\t\t\t\t\t
(Betw Convent & Amsterdam Avenues) NY 10031\n", 296 | "\t\t\t\t\t\t
\n", 297 | "\t\t\t\t\t\t
\n", 298 | " \n", 299 | "
\n", 300 | " \tTopic 1st Saturday. \n", 301 | "
\n", 302 | " \n", 303 | "\t\t\t\t\t\t\n", 304 | " \n", 305 | " \n", 306 | " \t \t\n", 307 | "\t\t\t\t \t Saturdays From 7:30 AM to 8:30 AM
Meeting Type OD = Open Discussion meeting \n", 308 | "\t\t\t \t\t\t
\n", 309 | " \t
\n", 310 | " \t\t\n", 311 | "\t\t\t\t\t \n", 312 | " \n", 313 | " \tGet Directions\n", 314 | " \n", 315 | " \n", 316 | "\n" 317 | ] 318 | } 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "source": [], 324 | "metadata": { 325 | "id": "q5S1BwCLWsEd" 326 | }, 327 | "execution_count": null, 328 | "outputs": [] 329 | } 330 | ] 331 | } -------------------------------------------------------------------------------- /intro_python.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# select examples from [Python Crash Course](https://nostarch.com/pythoncrashcourse)" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "print('hello, world!')" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "# 2) variables and simple data types" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "### variables" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "message = 'hello, world!'\n", 40 | "print(message)" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "![](img/namingusingvariables.jpeg \"Matthes p. 21\")" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "### avoiding name errors when using variables" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "print(mesage)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "### strings" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "```\n", 78 | "'I told my friend, \"Python is my favorite language!\"' \n", 79 | "\"The language 'Python' is named after Monty Python, not the snake.\" \n", 80 | "\"One of Python's strengths is its diverse and supportive community.\" \n", 81 | "```" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "### string methods" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "name = \"ada lovelace\"\n", 98 | "print(name.title())" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "### combining / concatenating strings" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "first_name = \"ada\"\n", 115 | "last_name = \"lovelace\"\n", 116 | "full_name = first_name + \" \" + last_name\n", 117 | "print(full_name)" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "*tip: using tab to autofill*" 125 | ] 126 | }, 127 | { 128 | "cell_type": "markdown", 129 | "metadata": {}, 130 | "source": [ 131 | "### adding whitespace" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": null, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "print(\"Python\")\n", 141 | "print(\"\\tPython\")\n", 142 | "print(\"Languages:\\nPython\\nJavascript\")" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "### removing whitespace" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "too_much_whitespace = ' python '\n", 159 | "print(\"*\" + too_much_whitespace + \"*\")\n", 160 | "print(\"*\" + too_much_whitespace.strip() + \"*\")" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "### numbers" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "# pp. 30-31" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "### combining strings and numbers" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "age = 42\n", 193 | "message = \"Happy \" + age + 1 + \"rd Birthday!\"" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": {}, 200 | "outputs": [], 201 | "source": [ 202 | "age = 42\n", 203 | "message = \"Happy \" + str(age + 1) + \"rd Birthday!\"\n", 204 | "print(message)" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "### comments" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": {}, 218 | "outputs": [], 219 | "source": [ 220 | "# Comments keep you sane" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "# 3) introducing lists" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "A list is a collection of items in a particular order. " 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "bicycles = ['trek', 'cannondale', 'redline', 'specialized']\n", 244 | "print(bicycles)\n", 245 | "print(bicycles[0]) # print the first item in the list" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "message = \"My first bicycle was a \" + bicycles[0].title() + \".\"\n", 255 | "print(message)" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": {}, 262 | "outputs": [], 263 | "source": [ 264 | "bicycles[0] = 'huffy'\n", 265 | "print(bicycles)" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "bicycles.append('trek')\n", 275 | "print(bicycles)" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "metadata": {}, 282 | "outputs": [], 283 | "source": [ 284 | "del bicycles[0]\n", 285 | "print(bicycles)" 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "execution_count": null, 291 | "metadata": {}, 292 | "outputs": [], 293 | "source": [ 294 | "last_bicycle = bicycles.pop()\n", 295 | "print(last_bicycle)\n", 296 | "print(bicycles)" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "print(len(bicycles))" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": null, 311 | "metadata": {}, 312 | "outputs": [], 313 | "source": [ 314 | "print(bicycles[3])" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": null, 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "print(bicycles[-1])" 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": {}, 329 | "source": [ 330 | "# 4) working with lists" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "### looping through a list" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "for bike in bicycles:\n", 347 | " print(bike)" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "for bike in bicycles:\n", 357 | " print(bike.title() + \" is a great bike.\")\n", 358 | " \n", 359 | "print(\"Those are all the great bikes I can think of.\")" 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "### indentation" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "metadata": {}, 373 | "outputs": [], 374 | "source": [ 375 | "print(\"this will work.\")\n", 376 | " print(\"why won't this work?\")" 377 | ] 378 | }, 379 | { 380 | "cell_type": "markdown", 381 | "metadata": {}, 382 | "source": [ 383 | "### numerical lists" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "metadata": {}, 390 | "outputs": [], 391 | "source": [ 392 | "for value in range(1, 5):\n", 393 | " print(value)" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [ 402 | "squares = []\n", 403 | "for i in range(1, 11):\n", 404 | " square = i**2\n", 405 | " squares.append(square)\n", 406 | " \n", 407 | "print(squares)" 408 | ] 409 | }, 410 | { 411 | "cell_type": "code", 412 | "execution_count": null, 413 | "metadata": {}, 414 | "outputs": [], 415 | "source": [ 416 | "# slice a list\n", 417 | "print(bicycles[0:2])" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "# 5) if statements" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "cars = ['audi', 'bmw', 'subaru', 'toyota']\n", 434 | "\n", 435 | "for car in cars:\n", 436 | " if car == 'bmw':\n", 437 | " print(car.upper())\n", 438 | " else: \n", 439 | " print(car.title())" 440 | ] 441 | }, 442 | { 443 | "cell_type": "markdown", 444 | "metadata": {}, 445 | "source": [ 446 | "### testing for equality" 447 | ] 448 | }, 449 | { 450 | "cell_type": "code", 451 | "execution_count": null, 452 | "metadata": {}, 453 | "outputs": [], 454 | "source": [ 455 | "print('bmw' == 'bmw')\n", 456 | "print('bmw' == 'BMW')\n", 457 | "print('bmw'.upper() == 'BMW')\n", 458 | "print(18 == 19)\n", 459 | "print(100 == 100)" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "### testing for inequality" 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": null, 472 | "metadata": {}, 473 | "outputs": [], 474 | "source": [ 475 | "# testing for inequality\n", 476 | "topping = 'mushrooms'\n", 477 | "\n", 478 | "if topping != 'anchovies':\n", 479 | " print(\"Hold the anchovies!\")" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "### testing multiple conditions" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": {}, 493 | "outputs": [], 494 | "source": [ 495 | "print(19 == 19 and 20 == 21)\n", 496 | "print(19 == 19 or 20 == 21)" 497 | ] 498 | }, 499 | { 500 | "cell_type": "markdown", 501 | "metadata": {}, 502 | "source": [ 503 | "### testing for inclusion in a list" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "execution_count": null, 509 | "metadata": {}, 510 | "outputs": [], 511 | "source": [ 512 | "print('redline' in bicycles)\n", 513 | "print('schwinn' in bicycles)\n", 514 | "print('schwinn' not in bicycles)" 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": null, 520 | "metadata": {}, 521 | "outputs": [], 522 | "source": [ 523 | "age = 12\n", 524 | "\n", 525 | "if age < 4:\n", 526 | " price = 0\n", 527 | "elif age < 18: \n", 528 | " price = 5\n", 529 | "\n", 530 | "else: \n", 531 | " price = 10\n", 532 | "\n", 533 | "print(\"Your admission cost is $\" + str(price) + \".00\")" 534 | ] 535 | }, 536 | { 537 | "cell_type": "markdown", 538 | "metadata": {}, 539 | "source": [ 540 | "# 6) dictionaries" 541 | ] 542 | }, 543 | { 544 | "cell_type": "code", 545 | "execution_count": null, 546 | "metadata": {}, 547 | "outputs": [], 548 | "source": [ 549 | "alien_0 = {'color': 'green', 'points': 5}\n", 550 | "print(alien_0)" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": null, 556 | "metadata": {}, 557 | "outputs": [], 558 | "source": [ 559 | "print(alien_0['color'])\n", 560 | "print(alien_0['points'])" 561 | ] 562 | }, 563 | { 564 | "cell_type": "code", 565 | "execution_count": null, 566 | "metadata": {}, 567 | "outputs": [], 568 | "source": [ 569 | "alien_0['favorite_food'] = 'cookies'\n", 570 | "print(alien_0)" 571 | ] 572 | }, 573 | { 574 | "cell_type": "code", 575 | "execution_count": null, 576 | "metadata": {}, 577 | "outputs": [], 578 | "source": [ 579 | "del alien_0['favorite_food']\n", 580 | "print(alien_0)" 581 | ] 582 | }, 583 | { 584 | "cell_type": "markdown", 585 | "metadata": {}, 586 | "source": [ 587 | "#### NESTING: \n", 588 | "* A list of dictionaries \n", 589 | "* A list in a dictionary \n", 590 | "* A dictionary in a dictionary" 591 | ] 592 | }, 593 | { 594 | "cell_type": "markdown", 595 | "metadata": {}, 596 | "source": [ 597 | "# 7) while loops" 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "metadata": {}, 603 | "source": [ 604 | "### avoiding infinite loops" 605 | ] 606 | }, 607 | { 608 | "cell_type": "code", 609 | "execution_count": null, 610 | "metadata": {}, 611 | "outputs": [], 612 | "source": [ 613 | "x = 1\n", 614 | "while x <= 5:\n", 615 | " print(x)\n", 616 | " x += 1" 617 | ] 618 | }, 619 | { 620 | "cell_type": "markdown", 621 | "metadata": {}, 622 | "source": [ 623 | "# 8) functions" 624 | ] 625 | }, 626 | { 627 | "cell_type": "code", 628 | "execution_count": null, 629 | "metadata": {}, 630 | "outputs": [], 631 | "source": [ 632 | "def greet_user():\n", 633 | " \"\"\"Display a simple greeting.\"\"\"\n", 634 | " print(\"Hello!\")\n", 635 | "\n", 636 | "greet_user()" 637 | ] 638 | }, 639 | { 640 | "cell_type": "markdown", 641 | "metadata": {}, 642 | "source": [ 643 | "### passing information to a function" 644 | ] 645 | }, 646 | { 647 | "cell_type": "code", 648 | "execution_count": null, 649 | "metadata": {}, 650 | "outputs": [], 651 | "source": [ 652 | "def greet_user(username):\n", 653 | " \"\"\"Display a simple greeting.\"\"\"\n", 654 | " print(\"Hello, \" + username.title() + \"!\")\n", 655 | "\n", 656 | "greet_user('jessie')" 657 | ] 658 | }, 659 | { 660 | "cell_type": "markdown", 661 | "metadata": {}, 662 | "source": [ 663 | "### positional and keyword arguments, default values" 664 | ] 665 | }, 666 | { 667 | "cell_type": "code", 668 | "execution_count": null, 669 | "metadata": {}, 670 | "outputs": [], 671 | "source": [ 672 | "def describe_pet(pet_name, animal_type='dog'):\n", 673 | " \"\"\"Display information about a pet.\"\"\"\n", 674 | " print(\"\\nI have a \" + animal_type + \".\")\n", 675 | " print(\"My \" + animal_type + \"'s name is \" + pet_name.title() + \".\")\n", 676 | " \n", 677 | "# A dog named Willie.\n", 678 | "describe_pet('willie')\n", 679 | "describe_pet(pet_name='willie')\n", 680 | "\n", 681 | "# A hamster named Harry.\n", 682 | "describe_pet('harry', 'hamster')\n", 683 | "describe_pet(pet_name='harry', animal_type='hamster')\n", 684 | "describe_pet(animal_type='hamster', pet_name='harry')\n" 685 | ] 686 | }, 687 | { 688 | "cell_type": "markdown", 689 | "metadata": {}, 690 | "source": [ 691 | "### return values" 692 | ] 693 | }, 694 | { 695 | "cell_type": "code", 696 | "execution_count": null, 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "def get_formatted_name(first_name, last_name, middle_name=''):\n", 701 | " \"\"\"Return a full name, neatly formatted.\"\"\"\n", 702 | " if middle_name:\n", 703 | " full_name = first_name + ' ' + middle_name + ' ' + last_name\n", 704 | " else:\n", 705 | " full_name = first_name + ' ' + last_name\n", 706 | " return full_name.title()\n", 707 | " \n", 708 | "musician = get_formatted_name('jimi', 'hendrix')\n", 709 | "print(musician)\n", 710 | "\n", 711 | "musician = get_formatted_name('john', 'hooker', 'lee')\n", 712 | "print(musician)" 713 | ] 714 | }, 715 | { 716 | "cell_type": "code", 717 | "execution_count": null, 718 | "metadata": {}, 719 | "outputs": [], 720 | "source": [] 721 | } 722 | ], 723 | "metadata": { 724 | "kernelspec": { 725 | "display_name": "Python 3", 726 | "language": "python", 727 | "name": "python3" 728 | }, 729 | "language_info": { 730 | "codemirror_mode": { 731 | "name": "ipython", 732 | "version": 3 733 | }, 734 | "file_extension": ".py", 735 | "mimetype": "text/x-python", 736 | "name": "python", 737 | "nbconvert_exporter": "python", 738 | "pygments_lexer": "ipython3", 739 | "version": "3.7.6" 740 | } 741 | }, 742 | "nbformat": 4, 743 | "nbformat_minor": 2 744 | } 745 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **Parsons School of Design, The New School 2 | School of Art, Media and Technology 3 | Master of Science, Data Visualization** 4 | 5 | # Data Structures 6 | 7 | ### PGDV 5110 Syllabus, Fall 2023 8 | 9 | Wednesdays, 12:10pm - 2:50pm 10 | 66 5th Ave | Room: 710 11 | 12 | **Faculty:** [Aaron Hill](https://aaronhill.io/) | Office Hours: Thursdays 3pm-4pm; contact info and Zoom info in Canvas 13 | 14 | I encourage use of the Canvas discussion board for collaboration and work on assignments, including the 10 weekly and 3 final assignments. All questions/issues about the assignments should be posted to the Canvas discussion board, building a shared knowledge base in the process. Questions/issues raised in office hours should also be posed here prior to the office hour meeting and the answer/resolution will be posted by the person who originally posed the question. 15 | 16 | **Course description:** Curating raw data presents issues of technology, speed, and efficiency, as well as broader ethical considerations of what it means to represent and make conclusions about groups and individuals from their data. This course covers the database, semi-structured data, and unstructured data. Students will gain familiarity with underlying data structures; techniques and tools, including acquisition, augmentation, and restructuring; data storage and aggregation; access to parallel and distributed computing; high-volume data, disparate sources, and performance; and streaming data, real time, and dynamic queries. 17 | 18 | **Prerequisites:** Basic knowledge of JavaScript, HTML/CSS, Linux command line, Git, and [GitHub](https://lab.github.com/githubtraining/introduction-to-github). 19 | 20 | ## Course Overview 21 | 22 | Duration | Description 23 | --- | ------- 24 | Week 1 | Introductions and overview of semester 25 | Weeks 2 - 4 | Fundamental data structures; modeling structured data 26 | Week 5 | Modeling semi-structured data 27 | Week 6 | The Query 28 | Weeks 7 - 9 | Physical sensors and IoT; data cleaning 29 | Week 10-11 | Interface design; serverless computing 30 | Weeks 12 - 14 | Reliability, scalability, maintainability, sustainability; Labs 31 | Week 15 | Final presentations (video) and reflections (Canvas) 32 | 33 | ## Course Outline 34 | 35 | ### August 30 (Week 1) 36 | 37 | Introduction to data structures. Time. Researching the end-user/consumer of data visualization and/or data application. 38 | 39 | #### Assignment: 40 | 41 | Canvas Discussion on researching the end-user/consumer of data visualization and/or data application. **Due Tuesday, September 5 by 11:59pm** 42 | 43 | #### Read: 44 | * Gitelman, Introduction and Color Plates 45 | 46 | ### September 6 (Week 2) 47 | 48 | The web as a database. "Raw data." 49 | 50 | #### Read: 51 | * Hills, Part One (pp. 17-42) 52 | * [Express/Node Introduction](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Introduction) (read carefully the "Node" introduction and quickly skim the "Express" introduction) 53 | 54 | **Weekly Assignment 1, due September 12 by 11:59pm:** 55 | You will study the structure of an HTML file and write a set of instructions (in narrative form, list form, and/or pseudo code) for parsing that HTML file. 56 | 57 | #### Documentation: 58 | 59 | * [What is `npm`?](https://docs.npmjs.com/getting-started/what-is-npm) 60 | * [Node got module](https://www.npmjs.com/package/got) 61 | * [Node fs module](https://nodejs.org/api/fs.html) 62 | 63 | ### September 13 (Week 3) 64 | 65 | Fundamental data structures. Data augmentation and integration. GitHub for open source collaboration. 66 | 67 | #### Read: 68 | 69 | * Gitelman, Chapter 5 70 | * Hills, Chapters 10, 11, and 12 71 | * [Javascript data types and structures](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures) 72 | * [Working with objects (JavaScript)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Working_with_Objects) 73 | 74 | **Weekly Assignment 2, due week 4** 75 | You will "get" some HTML files and then will parse one of these HTML files and log essential data to the console. 76 | 77 | #### Documentation: 78 | * [Node cheerio module](https://www.npmjs.com/package/cheerio) 79 | * [Introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) 80 | 81 | 82 | 83 | **Weekly Assignment 3:** 84 | Using data you parsed last week, you will interface with TAMU GeoServices to augment the data you collected. 85 | 86 | #### Documentation: 87 | 88 | * [TAMU GeoServices](http://geoservices.tamu.edu/) 89 | * [TAMU GeoServices Geocoding APIS](http://geoservices.tamu.edu/Services/Geocode/WebService/) 90 | * [Node Async module](http://caolan.github.io/async/) 91 | * [Tutorial: creating and using environment variables in Linux](https://www.digitalocean.com/community/tutorials/how-to-read-and-set-environmental-and-shell-variables-on-a-linux-vps). 92 | 93 | ### September 20 (Week 4) 94 | 95 | Data models I: SQL database. 96 | 97 | #### Read: 98 | 99 | * Gitelman, Chapter 2 100 | * Hills, Chapters 13, 14, 15, and 16 101 | 102 | **Weekly Assignment 4.1** 103 | You will design a schema and populate a representation of your database (in Excel) with the first three rows of your AA data. (2.5 points) 104 | 105 | #### Documentation: 106 | 107 | * [SQL tutorial](https://www.linkedin.com/learning/sql-essential-training-2) 108 | * [PostgreSQL data types](https://www.postgresql.org/docs/9.4/static/datatype.html) 109 | * [PostgreSQL documentation](https://www.postgresql.org/docs/9.4/static/) 110 | * [Node `pg` module](https://www.npmjs.com/package/pg) 111 | 112 | ### September 27 (Week 5) 113 | 114 | Data models I: NoSQL database. 115 | 116 | #### Read: 117 | * Gitelman, Chapter 6 118 | * Hills, Chapter 17 119 | 120 | **Weekly Assignment 5.1** 121 | You will design a (non)schema and populate a representation of your database (in Word) with the first three "documents" of your Process Blog data. (2.5 points) 122 | 123 | **Final Assignment 1 distributed:** Map of AA meetings in Manhattan. 124 | 125 | **Final Assignment 2 distributed:** "Process Blog," a source of semi-structured, qualitative data. 126 | 127 | ### October 4 (Week 6) 128 | 129 | [Observable Notebook for in-class exercise](https://observablehq.com/@aaronxhill/in-class-exercises-for-10-5) 130 | 131 | The Query. Indexes, joins, query optimization, and data restructuring. The E-R model. Data provision. 132 | 133 | #### Read: 134 | 135 | * Hills, Chapter 5 136 | * [Database as Symbolic Form](https://www.semanticscholar.org/paper/Database-as-Symbolic-Form-Manovich/d5d48898c764dc3949547354978815670c90ebc5), Lev Manovich 137 | * Gitelman, Data Flakes: An Afterword to "Raw Data" Is an Oxymoron 138 | 139 | **Weekly Assignments 4.2 and 5.2** 140 | 4.2: You will create a new managed PostgreSQL database, and write SQL code to populate your database with your AA data. (2.5 points) 141 | 142 | 5.2: You will create a new managed DynamoDB database, and write code to populate your database with your "Process Blog" data. (2.5 points) 143 | 144 | ### October 11 (Week 7) 145 | 146 | Data cleaning. Logging issues. 147 | 148 | Data cleaning workshop. 149 | 150 | **Weekly Assignment 6** 151 | You will write queries for both the SQL and NoSQL databases you have created, filtering, aggregating, and/or restructuring the data in the process. 152 | 153 | #### Read: 154 | 155 | * Gitelman, Chapter 4 156 | * [For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insights](https://nyti.ms/2jNUfHo), The New York Times, August 17, 2014 157 | 158 | #### Documentation: 159 | 160 | * [GitHub issues](https://guides.github.com/features/issues/) 161 | 162 | ### October 18 (Week 8) 163 | 164 | Sensors. Data gathering and surveillance. 165 | 166 | **Weekly Assignment 7** 167 | Finish parsing and cleaning the rest of the AA data in all ten Manhattan "zones", and update/replace the data in your Postgres table(s). 168 | 169 | #### Read: 170 | 171 | * Gitelman, Chapter 7 172 | 173 | #### Documentation: 174 | 175 | * Particle documentation [ [Photon](https://docs.particle.io/guide/getting-started/intro/photon/) | [Electron](https://docs.particle.io/guide/getting-started/intro/electron/) ] 176 | 177 | ### October 25 (Week 9) 178 | 179 | [slides] 180 | 181 | IoT. 182 | 183 | #### Read: 184 | 185 | * Gitelman, Chapter 3 186 | * Klepman, Chapter 2 187 | 188 | ### November 1 (Week 10) 189 | 190 | From database to visual representation. Interface design. Workshop by Jessie Han: How to create filters on the front end. 191 | 192 | **Weekly Assignment 8, due Tuesday 11/9 at 6:00pm:** 193 | Switching gears from previous assignments, you will use an IoT development board and sensors to collect and log data to your local computer. You will also sketch your first iteration of the design of an interactive visualization of the data you will collect from the sensor. 194 | 195 | **Final Assignment 3 distributed:** Surveilling your life. 196 | 197 | #### Read: 198 | 199 | * Gitelman, Chapter 1 200 | * [New York Times: Slump in Air Travel Hindered Weather Forecasting, Study Shows](https://www.nytimes.com/2020/10/29/climate/slump-in-air-travel-hindered-weather-forecasting-study-shows.html?action=click&module=Latest&pgtype=Homepage) 201 | 202 | ### November 8 (Week 11) 203 | 204 | Serverless computing. Workshop by Jessie Han: Use python + flask to host a website. 205 | 206 | **Weekly Assignment 9** 207 | You will begin writing sensor data to a database. 208 | 209 | #### Read: 210 | 211 | * [REDESIGNING DESIGN / JOS DE MUL](http://opendesignnow.org/index.html%3Fp=401.html) 212 | 213 | #### Documentation: 214 | 215 | * [What is Node.js and Express?](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Introduction) 216 | * [Express](http://expressjs.com/) 217 | * [Handlebars](http://handlebarsjs.com/) 218 | 219 | ### November 15 (Week 12) 220 | 221 | Discussion: Reliability. Lab. 222 | 223 | **Coding demo**: making a map in [mapbox](https://www.mapbox.com/) 224 | 225 | **Weekly Assignment 10** 226 | You will submit the final designs for the interfaces for each of your three data sources. 227 | 228 | #### Read: 229 | 230 | * Klepmann, Chapter 1. Read from the beginning through the "Reliability Section." 231 | * [How the internet works](https://edri.org/files/2012EDRiPapers/how_the_internet_works.pdf) 232 | 233 | ### November 22: (Week 13) 234 | 235 | One-on-one meetings via Zoom 236 | 237 | ### November 29 (Week 14) 238 | 239 | Discussion: Scalability. When do you need a database(s)? When do you not? Lab. 240 | 241 | #### Read: 242 | 243 | * Klepmann, Chapter 1. Read the "Scalability Section." 244 | * [MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat](https://research.google.com/archive/mapreduce.html) 245 | * [immens paper](http://vis.stanford.edu/papers/immens) 246 | 247 | ### December 6 (Week 15) 248 | 249 | Discussion: Maintainability. Lab. 250 | 251 | #### Read: 252 | 253 | * Klepmann, Chapter 1. Read the "Maintainability Section" and through the end of the chapter. 254 | * Gitelman, Chapter 8 255 | * [What is code?](https://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/) 256 | 257 | **Final Assignments due** 258 | 259 | In the final class meeting, you will present your work in groups, by theme. Group and theme assignments: 260 | 261 | * **Reliability** 262 | * **Scalability** 263 | * **Maintainability** 264 | 265 | Terminating and cleaning up your AWS resources and credentials. [**checklist**] 266 | 267 | ## Learning Outcomes 268 | 269 | By the successful completion of this course, students will be able to demonstrate: 270 | 271 | * Proficiency with various techniques to acquire, process, restructure, and analyze raw data. 272 | * Ability to program and work within structured and unstructured databases to deliver and exchange data. 273 | * Knowledge of methods for web development, hosting, and processing of data for visualizations, often from very large data sources. 274 | * An understanding of computer science principles that support computational efficiency and speed. 275 | * An understanding of the limitations and ethical considerations when using data to represent individuals and groups. 276 | 277 | ### Final Grade Calculation 278 | 279 | | Description | % | 280 | |:--|:--| 281 | | Engagement (Canvas and in-class discussions) | 10% | 282 | | Weekly Assignments (10 total) | 50% | 283 | | Final assignment | 40% | 284 | 285 | Weekly assignments are worth 5 points each, unless otherwise indicated. They are due 6:00pm the Tuesday after they are assigned, unless otherwise indicated. The quality of your [documentation](https://lab.github.com/githubtraining/communicating-using-markdown) is a part of the grade for every assignment, including all the weekly and final assignments. All documentation must be kept in your (well organized!) `data-structures` repository, using [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet). 286 | 287 | #### Late assignments: 288 | 289 | If an assignment is less than 48 hours late, the maximum possible grade is 75%. 290 | If an assignment is 48-96 hours late, the maximum possible grade is 50%. 291 | If an assignment is 96-144 hours late, the maximum possible grade is 25%. 292 | If an assignment is more than 144 hours late, no credit will be given. 293 | 294 | ### Required Reading 295 | 296 | *"[Raw Data" Is an Oxymoron](http://a.co/6bZnQzs)*, Lisa Gitelman, 2013. This book is available electronically through course reserves. 297 | *[NoSQL and SQL Data Modeling: Bringing Together Data, Semantics, and Software](http://a.co/6v7bJVs)*, Ted Hills, 2016. This book is not available through course reserves and should be purchased (~$27 on Amazon). 298 | *[Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems](http://a.co/dbjbtnU)*, Martin Kleppmann, 2017. This book is available electronically through course reserves. 299 | 300 | ### Recommended/Optional Reading 301 | 302 | *[Database Aesthetics : Art in the Age of Information Overflow](https://www.amazon.com/Database-Aesthetics-Information-Electronic-Mediations-dp-0816641196/dp/0816641196/ref=mt_other?_encoding=UTF8&me=&qid=1598487601)*, Victoria Vesna. This book is available electronically through course reserves. 303 | [The Stuff of Bits : An Essay on the Materialities of Information](https://www.amazon.com/Stuff-Bits-Essay-Materialities-Information/dp/0262036207/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=&sr=), Paul Dourish. This book is available electronically through course reserves. 304 | 305 | ### Office Hours (Zoom; no appointment needed) 306 | 307 | Thursdays, 3:00pm - 4:00pm 308 | 309 | ### Materials and Supplies 310 | 311 | In this course we will rely exclusively on open source software and we will work with the software by writing code: JavaScript, SQL, and some basic Linux. 312 | 313 | We will also rely on cloud-based services to provide infrastructure, platforms, and computational power: 314 | 315 | * [Amazon Web Services](https://aws.amazon.com/) (or similar) for computing infrastructure, platforms, and computational power. 316 | * [AWS Cloud 9](https://aws.amazon.com/cloud9/) for a working environment to use for prototyping and experimentation. 317 | 318 | #### Required: 319 | 320 | * A modern web browser. Preferred: [Google Chrome](https://www.google.com/chrome/). Acceptable: [Firefox](https://www.mozilla.org/en-US/firefox/). Not acceptable: Safari. 321 | * Bring your laptop to every class. 322 | * A (free) [GitHub](https://github.com/) account 323 | * It is also helpful to have a text editor installed locally. 324 | 325 | ### Resources 326 | 327 | The university provides many resources to help students achieve academic and artistic excellence. These resources include: 328 | 329 | * The University (and associated) Libraries: [http://library.newschool.edu](http://library.newschool.edu) 330 | * The University Learning Center: [http://www.newschool.edu/learning-center](http://www.newschool.edu/learning-center) 331 | * University Disabilities Service: [http://www.newschool.edu/student-disability-services/](http://www.newschool.edu/student-disability-services/) 332 | 333 | In keeping with the university's policy of providing equal access for students with disabilities, any student with a disability who needs academic accommodations must contact SDS. There are several ways for students to contact the office: via email at StudentDisability@newschool.edu, through the Starfish service catalog, or by calling the office at 212.229.5626. A self-ID form can also be completed on the SDS webpage at www.newschool.edu/student-disability-services. Once you contact the office, SDS staff will arrange an intake appointment to discuss your concerns and, if appropriate, provide you with accommodation notices to give to me. Please note that faculty will not work unilaterally with students to provide accommodations. If you inform me of a disability but do not provide any official notification, I must refer you to SDS. 334 | 335 | ### Making Center 336 | 337 | The [Making Center](https://makingcenter.parsons.edu/) is a constellation of shops, labs, and open workspaces that are situated across the New School to help students express their ideas in a variety of materials and methods. We have resources to help support woodworking, metalworking, ceramics and pottery work, photography and film, textiles, printmaking, 3D printing, manual and CNC machining, and more. A staff of technicians and student workers provide expertise and maintain the different shops and labs. Safety is a primary concern, so each area has policies for access, training, and etiquette with which students and faculty should be familiar. Many areas require specific orientations or trainings before access is granted. 338 | 339 | ### Grading Standards 340 | 341 | **A** Work of exceptional quality 342 | **A-** Work of high quality 343 | **B+** Very good work 344 | **B** Good work; satisfies course requirements 345 | Satisfactory completion of a course is considered to be a grade of B or higher. 346 | **B-** Below-average work 347 | **C+** Less than adequate work 348 | **C** Well below average work 349 | **C-** Poor work; lowest possible passing grade 350 | **F** Failure 351 | **GM** Grade missing for an individual 352 | 353 | *Grades of **D** are not used in graduate level courses.* 354 | 355 | Grade of **W**: 356 | The grade of **W** may be issued by the Office of the Registrar to a student who officially withdraws from a course within the applicable deadline. There is no academic penalty, but the grade will appear on the student transcript. 357 | 358 | Unofficial Withdrawal (Grade of **Z**): 359 | This grade is to be assigned to students who have **never attended or stopped attending** classes. Exceptions can be made if the student has completed enough work to warrant a grade (including a failing grade), and arrangements have been made with the instructor(s) and the Dean’s Office prior to grade submission. The Z grade does not calculate into the student’s GPA. Though a Z grade does not have a failing penalty it still carries a myriad of consequences for students on visas or receiving financial aid. Only issue the Z grade when a student meets the above criteria. 360 | 361 | Grades of **Incomplete**: 362 | The grade of **I**, or temporary incomplete, may be granted to a student under unusual and extenuating circumstances, such as when the student's academic life is interrupted by a medical or personal emergency. This mark is not given automatically but only upon the student's request and at the discretion of the instructor. A Request for Incomplete form must be completed and signed by student and instructor. The time allowed for completion of the work and removal of the **I** mark will be set by the instructor with the following limitations: 363 | 364 | > Work must be completed no later than one year following the end of the class. Grades of **I** not revised in the prescribed time will be recorded as a final grade of **N** by the Registrar’s Office. 365 | 366 | ### College, School, Program and Class Policies: 367 | 368 | A comprehensive overview of policy may be found under [Policies: A to Z](https://www.newschool.edu/about/university-resources/policies/). Students are also encouraged to consult the [Academic Catalog for Parsons](https://www.newschool.edu/provost/academic-catalogs/). 369 | 370 | ### Canvas 371 | 372 | Use of Canvas may be an important resource for this class. Students should check it for announcements before coming to class each week. 373 | 374 | ### Electronic Devices 375 | 376 | The use of electronic devices (phones, tablets, laptops, cameras, etc.) is permitted when the device is being used in relation to the course's work. All other uses are prohibited in the classroom and devices should be turned off before class starts. 377 | 378 | ### Responsibility 379 | 380 | Students are responsible for all assignments, even if they are absent. Late assignments, failure to complete the assignments for class discussion and/or critique, and lack of preparedness for in-class discussions, presentations and/or critiques will jeopardize your successful completion of this course. 381 | 382 | ### Active Participation and Attendance 383 | Class participation is an essential part of class and includes: keeping up with reading, assignments, projects, contributing meaningfully to class discussions, active participation in group work, and coming to class regularly and on time. 384 | 385 | Parsons' attendance guidelines were developed to encourage students’ success in all aspects of their academic programs. Full participation is essential to the successful completion of coursework and enhances the quality of the educational experience for all, particularly in courses where group work is integral; thus, Parsons promotes high levels of attendance. Students are expected to attend classes regularly and promptly and in compliance with the standards stated in this course syllabus. 386 | 387 | While attendance is just one aspect of active participation, absence from a significant portion of class time may prevent the successful attainment of course objectives. A significant portion of class time is generally defined as the equivalent of three weeks, or 20%, of class time. Lateness or early departure from class may be recorded as one full absence. Students may be asked to withdraw from a course if habitual absenteeism or tardiness has a negative impact on the class environment. 388 | 389 | I will assess each student’s performance against all of the assessment criteria in determining your final grade. 390 | 391 | ### Academic Honesty and Integrity 392 | 393 | Compromising your academic integrity may lead to serious consequences, including (but not limited to) one or more of the following: failure of the assignment, failure of the course, academic warning, disciplinary probation, suspension from the university, or dismissal from the university. 394 | 395 | Students are responsible for understanding the University’s policy on academic honesty and integrity and must make use of proper citations of sources for writing papers, creating, presenting, and performing their work, taking examinations, and doing research. It is the responsibility of students to learn the procedures specific to their discipline for correctly and appropriately differentiating their own work from that of others. The full text of the policy, including adjudication procedures, is found on the university website under [Policies: A to Z](https://www.newschool.edu/about/university-resources/policies/). Resources regarding what plagiarism is and how to avoid it can be found on the [Learning Center’s website](https://www.newschool.edu/learning-center/resources-workshops/). 396 | 397 | The New School views "academic honesty and integrity" as the duty of every member of an academic community to claim authorship for his or her own work and only for that work, and to recognize the contributions of others accurately and completely. This obligation is fundamental to the integrity of intellectual debate, and creative and academic pursuits. Academic honesty and integrity includes accurate use of quotations, as well as appropriate and explicit citation of sources in instances of paraphrasing and describing ideas, or reporting on research findings or any aspect of the work of others (including that of faculty members and other students). Academic dishonesty results from infractions of this "accurate use." The standards of academic honesty and integrity, and citation of sources, apply to all forms of academic work, including submissions of drafts of final papers or projects. All members of the University community are expected to conduct themselves in accord with the standards of academic honesty and integrity. Please see the complete policy in the Parsons Catalog. 398 | 399 | ### Intellectual Property Rights 400 | 401 | The New School (the "university") seeks to encourage creativity and invention among its faculty members and students. In doing so, the University affirms its traditional commitment to the personal ownership by its faculty members and students of Intellectual Property Rights in works they create. The complete policy governing Intellectual Property Rights may be seen on the [university website, on the Provost's page](https://www.newschool.edu/provost/accreditation-policies/). 402 | 403 | ### Student Course Ratings (Course Evaluations) 404 | 405 | During the last two weeks of the semester, students are asked to provide feedback for each of their courses through an online survey. They cannot view grades until providing feedback or officially declining to do so. Course evaluations are a vital space where students can speak about the learning experience. It is an important process which provides valuable data about the successful delivery and support of a course or topic to both the faculty and administrators. Instructors rely on course rating surveys for feedback on the course and teaching methods, so they can understand what aspects of the class are most successful in teaching students, and what aspects might be improved or changed in future. Without this information, it can be difficult for an instructor to reflect upon and improve teaching methods and course design. In addition, program/department chairs and other administrators review course surveys. Instructions are available online [here](https://docs.google.com/document/d/1JmK_0V6aNVCHAmGtflZ7lKHpNI1Q50yijLcHrrA_zRs/edit#heading=h.3dy6vkm). 406 | 407 | --------------------------------------------------------------------------------