![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "source": [
32 | "# Domain Frequency Dataset Exploration\n",
33 | "\n",
34 | "We're going to take a look at a few examples of how we can explore the Domain Frequency dataset. First we need to enter the URL for our Domain Frequency dataset. You can get this by right clicking the Download icon, and selecting \"Copy Link\"."
35 | ],
36 | "metadata": {
37 | "id": "vAyuRQ2PJIdc"
38 | }
39 | },
40 | {
41 | "cell_type": "code",
42 | "source": [
43 | "dataset = 'https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-06689/DomainFrequencyExtraction/domain-frequency.csv.gz?access=FUO54Y4B3J3GAZSDD6ML7DQENUN5BDOX' #@param {type:\"string\"}\n",
44 | "print(dataset)"
45 | ],
46 | "metadata": {
47 | "colab": {
48 | "base_uri": "https://localhost:8080/"
49 | },
50 | "cellView": "form",
51 | "id": "RfhJiesWVpAf",
52 | "outputId": "277fad1b-e6c8-4d4f-d2cb-11f38f484138"
53 | },
54 | "execution_count": 1,
55 | "outputs": [
56 | {
57 | "output_type": "stream",
58 | "name": "stdout",
59 | "text": [
60 | "https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-06689/DomainFrequencyExtraction/domain-frequency.csv.gz?access=FUO54Y4B3J3GAZSDD6ML7DQENUN5BDOX\n"
61 | ]
62 | }
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "source": [
68 | "# Environment\n",
69 | "\n",
70 | "Next, we'll set up our environment so we can load our Domain Frequency dataset into [pandas](https://pandas.pydata.org) and use [Altair](https://altair-viz.github.io/) for plots. Altair is useful for creating plots since they can be easily exported as a PNG or SVG."
71 | ],
72 | "metadata": {
73 | "id": "Z14F2cIWJVW0"
74 | }
75 | },
76 | {
77 | "cell_type": "code",
78 | "execution_count": 2,
79 | "metadata": {
80 | "id": "Chh6tt3HHF1s"
81 | },
82 | "outputs": [],
83 | "source": [
84 | "import pandas as pd\n",
85 | "import altair as alt"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "source": [
91 | "## Data Table Display\n",
92 | "\n",
93 | "Colab includes an extension that renders pandas DataFrames into interactive displays that can be filtered, sorted, and explored dynamically. This can be very useful for taking a look at what each DataFrame provides!\n",
94 | "\n",
95 | "Data table display for pandas DataFrames can be enabled by running:\n",
96 | "```python\n",
97 | "%load_ext google.colab.data_table\n",
98 | "```\n",
99 | "and disabled by running\n",
100 | "```python\n",
101 | "%unload_ext google.colab.data_table\n",
102 | "```"
103 | ],
104 | "metadata": {
105 | "id": "sH81XCf3I3xY"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "%load_ext google.colab.data_table"
112 | ],
113 | "metadata": {
114 | "id": "-qyCnbvBI7n6"
115 | },
116 | "execution_count": 3,
117 | "outputs": []
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "source": [
122 | "# Loading our ARCH Dataset as a DataFrame\n",
123 | "\n",
124 | "---\n",
125 | "\n",
126 | "\n",
127 | "Next, we'll create a pandas DataFrame from our dataset and show a preview of it using the Data Table Display."
128 | ],
129 | "metadata": {
130 | "id": "6prR7j1zI_D5"
131 | }
132 | },
133 | {
134 | "cell_type": "code",
135 | "source": [
136 | "domains = pd.read_csv(dataset, compression='gzip')\n",
137 | "domains"
138 | ],
139 | "metadata": {
140 | "colab": {
141 | "base_uri": "https://localhost:8080/",
142 | "height": 441
143 | },
144 | "id": "YL0LQaUNHRKx",
145 | "outputId": "498e4153-bb5a-427e-9eba-9a3aa10abdfe"
146 | },
147 | "execution_count": 4,
148 | "outputs": [
149 | {
150 | "output_type": "stream",
151 | "name": "stdout",
152 | "text": [
153 | "Warning: total number of rows (20407) exceeds max_rows (20000). Falling back to pandas display.\n"
154 | ]
155 | },
156 | {
157 | "output_type": "execute_result",
158 | "data": {
159 | "text/plain": [
160 | " domain count\n",
161 | "0 sld.cu 11002591\n",
162 | "1 cult.cu 5197445\n",
163 | "2 facebook.com 1985543\n",
164 | "3 icrt.cu 1936127\n",
165 | "4 cubava.cu 967903\n",
166 | "... ... ...\n",
167 | "20402 yardbarker.com 1\n",
168 | "20403 bufetetoro.com 1\n",
169 | "20404 understandingwar.org 1\n",
170 | "20405 headaches.org 1\n",
171 | "20406 nursite.com.ar 1\n",
172 | "\n",
173 | "[20407 rows x 2 columns]"
174 | ],
175 | "text/html": [
176 | "\n",
177 | " | \n", 197 | " | domain | \n", 198 | "count | \n", 199 | "
|---|---|---|
| 0 | \n", 204 | "sld.cu | \n", 205 | "11002591 | \n", 206 | "
| 1 | \n", 209 | "cult.cu | \n", 210 | "5197445 | \n", 211 | "
| 2 | \n", 214 | "facebook.com | \n", 215 | "1985543 | \n", 216 | "
| 3 | \n", 219 | "icrt.cu | \n", 220 | "1936127 | \n", 221 | "
| 4 | \n", 224 | "cubava.cu | \n", 225 | "967903 | \n", 226 | "
| ... | \n", 229 | "... | \n", 230 | "... | \n", 231 | "
| 20402 | \n", 234 | "yardbarker.com | \n", 235 | "1 | \n", 236 | "
| 20403 | \n", 239 | "bufetetoro.com | \n", 240 | "1 | \n", 241 | "
| 20404 | \n", 244 | "understandingwar.org | \n", 245 | "1 | \n", 246 | "
| 20405 | \n", 249 | "headaches.org | \n", 250 | "1 | \n", 251 | "
| 20406 | \n", 254 | "nursite.com.ar | \n", 255 | "1 | \n", 256 | "
20407 rows × 2 columns
\n", 260 | "| \n", 564 | " | domain | \n", 565 | "count | \n", 566 | "tld | \n", 567 | "
|---|---|---|---|
| 0 | \n", 572 | "sld.cu | \n", 573 | "11002591 | \n", 574 | "cu | \n", 575 | "
| 1 | \n", 578 | "cult.cu | \n", 579 | "5197445 | \n", 580 | "cu | \n", 581 | "
| 2 | \n", 584 | "facebook.com | \n", 585 | "1985543 | \n", 586 | "com | \n", 587 | "
| 3 | \n", 590 | "icrt.cu | \n", 591 | "1936127 | \n", 592 | "cu | \n", 593 | "
| 4 | \n", 596 | "cubava.cu | \n", 597 | "967903 | \n", 598 | "cu | \n", 599 | "
| ... | \n", 602 | "... | \n", 603 | "... | \n", 604 | "... | \n", 605 | "
| 20402 | \n", 608 | "yardbarker.com | \n", 609 | "1 | \n", 610 | "com | \n", 611 | "
| 20403 | \n", 614 | "bufetetoro.com | \n", 615 | "1 | \n", 616 | "com | \n", 617 | "
| 20404 | \n", 620 | "understandingwar.org | \n", 621 | "1 | \n", 622 | "org | \n", 623 | "
| 20405 | \n", 626 | "headaches.org | \n", 627 | "1 | \n", 628 | "org | \n", 629 | "
| 20406 | \n", 632 | "nursite.com.ar | \n", 633 | "1 | \n", 634 | "com.ar | \n", 635 | "
20407 rows × 3 columns
\n", 639 | ""
26 | ]
27 | },
28 | {
29 | "cell_type": "markdown",
30 | "source": [
31 | "# HTML Information Dataset Exploration\n",
32 | "\n",
33 | "We're going to take a look at a few examples of how we can explore the HTML Information dataset. \n",
34 | "\n",
35 | "The first thing we need to do is enter the URL for our HTML Information dataset in the cell below. You can get this by right clicking the Download icon, and selecting \"Copy Link\"."
36 | ],
37 | "metadata": {
38 | "id": "vAyuRQ2PJIdc"
39 | }
40 | },
41 | {
42 | "cell_type": "code",
43 | "source": [
44 | "dataset = 'https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-14462/TextFilesInformationExtraction/html-file-information.csv.gz?access=UCQ7VUUU4NDLKSGIPQD2R2WUGLOQXWPQ' #@param {type:\"string\"}\n",
45 | "print(dataset)"
46 | ],
47 | "metadata": {
48 | "colab": {
49 | "base_uri": "https://localhost:8080/"
50 | },
51 | "cellView": "form",
52 | "id": "RfhJiesWVpAf",
53 | "outputId": "507325b4-2e56-4aaf-bc7b-55b38fe221c8"
54 | },
55 | "execution_count": 1,
56 | "outputs": [
57 | {
58 | "output_type": "stream",
59 | "name": "stdout",
60 | "text": [
61 | "https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-14462/TextFilesInformationExtraction/html-file-information.csv.gz?access=UCQ7VUUU4NDLKSGIPQD2R2WUGLOQXWPQ\n"
62 | ]
63 | }
64 | ]
65 | },
66 | {
67 | "cell_type": "markdown",
68 | "source": [
69 | "## pandas\n",
70 | "\n",
71 | "Next, we'll setup our environment so we can load our HTML Information dataset into [pandas](https://pandas.pydata.org) DataFrames. If you're unfamiliar with DataFrames, but you've worked with spreadsheets before, you should quickly feel comfortable."
72 | ],
73 | "metadata": {
74 | "id": "Z14F2cIWJVW0"
75 | }
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": 2,
80 | "metadata": {
81 | "id": "Chh6tt3HHF1s"
82 | },
83 | "outputs": [],
84 | "source": [
85 | "import pandas as pd"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "source": [
91 | "## Data Table Display\n",
92 | "\n",
93 | "Colab includes an extension that renders pandas DataFrames into interactive displays that can be filtered, sorted, and explored dynamically. This can be very useful for taking a look at what each DataFrame provides, and doing some intital filtering!\n",
94 | "\n",
95 | "Data table display for pandas DataFrames can be enabled by running:\n",
96 | "```python\n",
97 | "%load_ext google.colab.data_table\n",
98 | "```\n",
99 | "and disabled by running\n",
100 | "```python\n",
101 | "%unload_ext google.colab.data_table\n",
102 | "```"
103 | ],
104 | "metadata": {
105 | "id": "sH81XCf3I3xY"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "%load_ext google.colab.data_table"
112 | ],
113 | "metadata": {
114 | "id": "-qyCnbvBI7n6"
115 | },
116 | "execution_count": 3,
117 | "outputs": []
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "source": [
122 | "## Loading our ARCH Dataset as a DataFrame\n",
123 | "\n",
124 | "---\n",
125 | "\n",
126 | "\n",
127 | "Next, we'll create pandas DataFrame from our dataset and show a preview of it using the Data Table Display."
128 | ],
129 | "metadata": {
130 | "id": "6prR7j1zI_D5"
131 | }
132 | },
133 | {
134 | "cell_type": "code",
135 | "source": [
136 | "html = pd.read_csv(dataset, compression='gzip')\n",
137 | "html"
138 | ],
139 | "metadata": {
140 | "id": "YL0LQaUNHRKx"
141 | },
142 | "execution_count": 4,
143 | "outputs": []
144 | },
145 | {
146 | "cell_type": "markdown",
147 | "metadata": {
148 | "id": "3HPwOCNAvqMe"
149 | },
150 | "source": [
151 | "# Data Analysis\n",
152 | "\n",
153 | "Now that we have all of our datasets loaded up, we can begin to work with them!"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "metadata": {
159 | "id": "J6Pkg0prv3BE"
160 | },
161 | "source": [
162 | "## Counting total files, and unique files\n",
163 | "\n",
164 | "Let's take a quick look at how to count items in DataFrames, and use total and unique files as an example to work with.\n",
165 | "\n",
166 | "It's definitely work checking out the [pandas documentation](https://pandas.pydata.org/docs/index.html). There are a lot of good examples available, along with a robust [API reference](https://pandas.pydata.org/docs/reference/index.html#api)."
167 | ]
168 | },
169 | {
170 | "cell_type": "markdown",
171 | "metadata": {
172 | "id": "DFX4Gl3wv7bi"
173 | },
174 | "source": [
175 | "\n",
176 | "### How many html files are in this collection?\n",
177 | "\n",
178 | "We can take our `html` variable and try a couple of functions to get the same answer.\n",
179 | "\n",
180 | "1. `len(html.index)`\n",
181 | " * Get the length of the DataFrame's index.\n",
182 | "2. `html.shape[0]`\n",
183 | " * Get the shape or dimensionality of the DataFrame, and take the first item in the tuple.\n",
184 | "3. `html.count()`\n",
185 | " * Count the number of rows for each column.\n",
186 | "\n"
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "metadata": {
192 | "id": "HTv8Oet3jiTH"
193 | },
194 | "source": [
195 | "len(html.index)"
196 | ],
197 | "execution_count": 5,
198 | "outputs": []
199 | },
200 | {
201 | "cell_type": "code",
202 | "metadata": {
203 | "id": "6rYEERnTjifk"
204 | },
205 | "source": [
206 | "html.shape[0]"
207 | ],
208 | "execution_count": 6,
209 | "outputs": []
210 | },
211 | {
212 | "cell_type": "code",
213 | "metadata": {
214 | "id": "bn-1v127aKIG"
215 | },
216 | "source": [
217 | "html.count()"
218 | ],
219 | "execution_count": 7,
220 | "outputs": []
221 | },
222 | {
223 | "cell_type": "markdown",
224 | "metadata": {
225 | "id": "38veKiPhwKo4"
226 | },
227 | "source": [
228 | "### How many unique html files are in the collection?\n",
229 | "\n",
230 | " We can see if an HTML file is unique or not by computing an [MD5 hash](https://en.wikipedia.org/wiki/MD5#MD5_hashes) of it, and comparing them. The exact same html file might have a filename of `example.html` or `foo.html`. If the hash is computed for each, we can see that even with different file names, they are actually the same html file. So, since we have both a `MD5` and `SHA1` hash column available in our DataFrame, we can just find the unique values, and count them!\n",
231 | "\n",
232 | "\n"
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "metadata": {
238 | "id": "WesM3kQowM5B"
239 | },
240 | "source": [
241 | "len(html.md5.unique())"
242 | ],
243 | "execution_count": 8,
244 | "outputs": []
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {
249 | "id": "ZIXkI0-1wWQf"
250 | },
251 | "source": [
252 | "### What are the top 10 most occurring html files in the collection?\n",
253 | "\n",
254 | "Here we can take advantage of [`value_counts()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.value_counts.html) to provide us with a list of MD5 hashes and their respective counts."
255 | ]
256 | },
257 | {
258 | "cell_type": "code",
259 | "metadata": {
260 | "id": "8Ts03OFyjPIM"
261 | },
262 | "source": [
263 | "html[\"md5\"].value_counts().head(10)"
264 | ],
265 | "execution_count": 9,
266 | "outputs": []
267 | },
268 | {
269 | "cell_type": "markdown",
270 | "metadata": {
271 | "id": "FG7pGZUEwlaI"
272 | },
273 | "source": [
274 | "\n",
275 | "### What's the information around all of the occurances of `d41d8cd98f00b204e9800998ecf8427e`?\n",
276 | "\n",
277 | "What, you mean you don't know what `d41d8cd98f00b204e9800998ecf8427e` means? \n",
278 | "\n",
279 | "Let's find those HTML files in the DataFrame. We can here see some of the filenames used, its dimensions, and its URL.\n"
280 | ]
281 | },
282 | {
283 | "cell_type": "code",
284 | "metadata": {
285 | "id": "msmmm65lkSIK"
286 | },
287 | "source": [
288 | "html.loc[html[\"md5\"] == \"d41d8cd98f00b204e9800998ecf8427e\"]"
289 | ],
290 | "execution_count": null,
291 | "outputs": []
292 | },
293 | {
294 | "cell_type": "markdown",
295 | "metadata": {
296 | "id": "GbLLZW2awzCv"
297 | },
298 | "source": [
299 | "### What are the top 10 most occuring filenames in the collection?\n",
300 | "\n",
301 | "Note that this is of course different than the MD5 results up above. Here we are focusing _just_ on filename. So `a16180790160.html ` for example, might actually be referring to different HTML files who happen to have the same name.\n",
302 | "\n",
303 | "Here we can use `value_counts()` again, but this time we'll create a variable for the top filenames so we can use it later.\n",
304 | "\n"
305 | ]
306 | },
307 | {
308 | "cell_type": "code",
309 | "metadata": {
310 | "id": "pQaw54ACkwdZ"
311 | },
312 | "source": [
313 | "top_filenames = html[\"filename\"].value_counts().head(10)\n",
314 | "top_filenames"
315 | ],
316 | "execution_count": null,
317 | "outputs": []
318 | },
319 | {
320 | "cell_type": "markdown",
321 | "metadata": {
322 | "id": "Z7F3re20BQRI"
323 | },
324 | "source": [
325 | "### Let's create our first graph!\n",
326 | "\n",
327 | "We'll first plot the data with the pandas [plot](https://pandas.pydata.org/docs/reference/api/pandas.Series.plot.html) functionality, and then with [Altair](https://altair-viz.github.io/)."
328 | ]
329 | },
330 | {
331 | "cell_type": "code",
332 | "metadata": {
333 | "id": "sRvlstfsBWEZ"
334 | },
335 | "source": [
336 | "top_filenames_chart = top_filenames.plot.bar(figsize=(25, 10))\n",
337 | "\n",
338 | "top_filenames_chart.set_title(\"Top Filenames\", fontsize=22)\n",
339 | "top_filenames_chart.set_xlabel(\"Filename\", fontsize=20)\n",
340 | "top_filenames_chart.set_ylabel(\"Count\", fontsize=20)"
341 | ],
342 | "execution_count": null,
343 | "outputs": []
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {
348 | "id": "pQgeOObvgLvK"
349 | },
350 | "source": [
351 | "Now let's setup [Altair](https://altair-viz.github.io/), and plot the data. Altair is useful for creating vizualizations since they can be easily exported as a PNG or SVG."
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "metadata": {
357 | "id": "Q7Z4J6qjWaVM"
358 | },
359 | "source": [
360 | "import altair as alt"
361 | ],
362 | "execution_count": null,
363 | "outputs": []
364 | },
365 | {
366 | "cell_type": "code",
367 | "metadata": {
368 | "id": "s0xwvILYWkgg"
369 | },
370 | "source": [
371 | "top_filenames_altair = (\n",
372 | " html[\"filename\"]\n",
373 | " .value_counts()\n",
374 | " .head(10)\n",
375 | " .rename_axis(\"Filename\")\n",
376 | " .reset_index(name=\"Count\")\n",
377 | ")\n",
378 | "\n",
379 | "filenames_bar = (\n",
380 | " alt.Chart(top_filenames_altair)\n",
381 | " .mark_bar()\n",
382 | " .encode(x=alt.X(\"Filename:O\", sort=\"-y\"), y=alt.Y(\"Count:Q\"))\n",
383 | ")\n",
384 | "\n",
385 | "filenames_rule = (\n",
386 | " alt.Chart(top_filenames_altair).mark_rule(color=\"red\").encode(y=\"mean(Count):Q\")\n",
387 | ")\n",
388 | "\n",
389 | "\n",
390 | "filenames_text = filenames_bar.mark_text(align=\"center\", baseline=\"bottom\").encode(\n",
391 | " text=\"Count:Q\"\n",
392 | ")\n",
393 | "\n",
394 | "(filenames_bar + filenames_rule + filenames_text).properties(\n",
395 | " width=1400, height=700, title=\"Top Filenames\"\n",
396 | ")"
397 | ],
398 | "execution_count": null,
399 | "outputs": []
400 | },
401 | {
402 | "cell_type": "markdown",
403 | "metadata": {
404 | "id": "BneaN9cgGoly"
405 | },
406 | "source": [
407 | "### How about a file format distribution?\n",
408 | "\n",
409 | "What _kind_ of html files are present? We can discover this by checking their \"media type\", or [MIME type](https://en.wikipedia.org/wiki/Media_type). \n",
410 | "\n",
411 | "\n",
412 | "\n",
413 | "\n"
414 | ]
415 | },
416 | {
417 | "cell_type": "code",
418 | "metadata": {
419 | "id": "RDd-J8D-GwDk"
420 | },
421 | "source": [
422 | "html_mime_types = (\n",
423 | " html[\"mime_type_tika\"]\n",
424 | " .value_counts()\n",
425 | " .head(5)\n",
426 | " .rename_axis(\"MIME Type\")\n",
427 | " .reset_index(name=\"Count\")\n",
428 | ")\n",
429 | "\n",
430 | "html_mimes_bar = (\n",
431 | " alt.Chart(html_mime_types)\n",
432 | " .mark_bar()\n",
433 | " .encode(x=alt.X(\"MIME Type:O\", sort=\"-y\"), y=alt.Y(\"Count:Q\"))\n",
434 | ")\n",
435 | "\n",
436 | "html_mime_rule = (\n",
437 | " alt.Chart(html_mime_types).mark_rule(color=\"red\").encode(y=\"mean(Count):Q\")\n",
438 | ")\n",
439 | "\n",
440 | "html_mime_text = html_mimes_bar.mark_text(align=\"center\", baseline=\"bottom\").encode(\n",
441 | " text=\"Count:Q\"\n",
442 | ")\n",
443 | "\n",
444 | "(html_mimes_bar + html_mime_rule + html_mime_text).properties(\n",
445 | " width=1400, height=700, title=\"HTML File Format Distribution\"\n",
446 | ")"
447 | ],
448 | "execution_count": null,
449 | "outputs": []
450 | },
451 | {
452 | "cell_type": "markdown",
453 | "metadata": {
454 | "id": "QUJR-jjqNxCL"
455 | },
456 | "source": [
457 | "### How do I get the actual html?\n",
458 | "\n",
459 | "...or, how do I get to the actual binary files described by each file format information derivative?\n",
460 | "\n",
461 | "There are a few options!\n",
462 | "\n",
463 | "1. `wget` or `curl` from the live URL, or a replay URL\n",
464 | " * Live web URL\n",
465 | " * `wget` or `curl` the value of the `url` column\n",
466 | " * Replay web URL\n",
467 | " * `wget` or `curl` the value of the `crawl_date` and `url` column using the following pattern:\n",
468 | " * `https://web.archive.org/web/` + `crawl_date` + `/` + `url`\n",
469 | " * https://web.archive.org/web/20120119124734/http://www.archive.org/images/glogo.png\n",
470 | " * `http://wayback.archive-it.org/14462/` + `crawl_date` + `/` + `url`\n",
471 | " * https://wayback.archive-it.org/14462/20210524212740/https://ruebot.net/visualization/elxn42/featured_hu33a17dfb90e2c5ed77f783db14a6e53a_5126291_550x0_resize_q90_box_2.png\n",
472 | "2. Use a scripting language, such as Python\n",
473 | " * Make use of the `url` and `filename` columns (and `crawl_date` if you want to use the replay URL)\n",
474 | " * `import requests`\n",
475 | " * `requests.get(url, allow_redirects=True)`\n",
476 | " * `open('filename', 'wb').write(r.content)`\n",
477 | "3. Use the [Archives Unleashed Toolkit](https://aut.docs.archivesunleashed.org/docs/extract-binary) (if you have access to the W/ARC files)."
478 | ]
479 | },
480 | {
481 | "cell_type": "markdown",
482 | "source": [
483 | "If you wanted to download the HTML files using the replay URL, below is a method for doing so.\n",
484 | "\n",
485 | "First, you'll want to setup a replay url base url. Here we'll use the Archive-It Wayback instance for the collection."
486 | ],
487 | "metadata": {
488 | "id": "8yoFE2xLAlwk"
489 | }
490 | },
491 | {
492 | "cell_type": "code",
493 | "source": [
494 | "wayback_url = 'http://wayback.archive-it.org/14462/'"
495 | ],
496 | "metadata": {
497 | "id": "RxrOHn_2AicZ"
498 | },
499 | "execution_count": null,
500 | "outputs": []
501 | },
502 | {
503 | "cell_type": "markdown",
504 | "source": [
505 | "Next we'll create a new column using a lambda function. If you're familiar working with spreadsheets, what we're doing here is basically concatenating some column values together and creating a new column."
506 | ],
507 | "metadata": {
508 | "id": "jCTmrEBGA-Yd"
509 | }
510 | },
511 | {
512 | "cell_type": "code",
513 | "source": [
514 | "html['replay_url'] = html.apply(lambda row: str(wayback_url + str(row['crawl_date']) + \"/\" + row['url']), axis=1)"
515 | ],
516 | "metadata": {
517 | "id": "p5HoWxkFA6C8"
518 | },
519 | "execution_count": null,
520 | "outputs": []
521 | },
522 | {
523 | "cell_type": "markdown",
524 | "source": [
525 | "Then we can export that new column we created out to a file, so we can use it with `wget` to download all the html files!"
526 | ],
527 | "metadata": {
528 | "id": "q87tSYyIBIiZ"
529 | }
530 | },
531 | {
532 | "cell_type": "code",
533 | "source": [
534 | "html['replay_url'].head().to_csv('14462_html_urls.txt', index=False, header=False)"
535 | ],
536 | "metadata": {
537 | "id": "wyhpsf9wBN1a"
538 | },
539 | "execution_count": null,
540 | "outputs": []
541 | },
542 | {
543 | "cell_type": "markdown",
544 | "source": [
545 | "Finally, we can pass the file to `wget` to use as a download list. You can also speed this process up using `xargs` or `parallel`."
546 | ],
547 | "metadata": {
548 | "id": "81a3q7feu5Ji"
549 | }
550 | },
551 | {
552 | "cell_type": "code",
553 | "source": [
554 | "!wget --random-wait -i 14462_html_urls.txt"
555 | ],
556 | "metadata": {
557 | "id": "D_p4qGKoBRaZ"
558 | },
559 | "execution_count": null,
560 | "outputs": []
561 | }
562 | ]
563 | }
--------------------------------------------------------------------------------
/assets/jupyter-shell.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/archivesunleashed/notebooks/8211b24ab6106e4da5f422b30cbc8b7bac83b4b6/assets/jupyter-shell.png
--------------------------------------------------------------------------------
/assets/juypter.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/archivesunleashed/notebooks/8211b24ab6106e4da5f422b30cbc8b7bac83b4b6/assets/juypter.png
--------------------------------------------------------------------------------
/assets/ny-schedule.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/archivesunleashed/notebooks/8211b24ab6106e4da5f422b30cbc8b7bac83b4b6/assets/ny-schedule.png
--------------------------------------------------------------------------------
/datathon-nyc/README.md:
--------------------------------------------------------------------------------
1 | # Archives Unleashed NYC Datathon
2 |
3 | Welcome to the Archives Unleashed NY Datathon! Below you will find a variety of resources that will be useful throughout our event.
4 |
5 | * [Requirements](#requirements)
6 | * [Schedule](#schedule)
7 | * [Team Collaboration Resources](#team-collaboration-resources)
8 | * [Homework Refresher](#homework-refresher)
9 | * [Additional/External Tools for Analysis](#additionalexternal-tools-for-analysis)
10 | * [Analysis Platforms](#analysis-platforms)
11 | * [Inspiration](#inspiration)
12 | * [Virtual Machines](#virtual-machines)
13 | * [Getting Started](#getting-started)
14 | * [Shell into assigned VM](#shell-into-assigned-vm)
15 | * [Start Juypter Notebook](#start-juypter-notebook)
16 | * [Start Spark with AUT](#start-spark-with-aut)
17 | * [Datasets](#datasets)
18 |
19 | ## Requirements
20 |
21 | - **Zoom** (login information will be provided via Slack/email).
22 | - **Slack** (datathon-ny-2020): our team will use Slack to communicate with participants. You can also use this platform to create a team channel (free) to collaborate (text + calls) during the datathon.
23 | - [Github Account](https://github.com/join) and [Google Account](https://accounts.google.com/signin/v2/identifier?service=g4np&passive=1209600&continue=https%3A%2F%2Fwww.google.com%2Fnonprofits%2Faccount%2Fsignup%3Flocality%3Dus&followup=https%3A%2F%2Fwww.google.com%2Fnonprofits%2Faccount%2Fsignup%3Flocality%3Dus&flowName=GlifWebSignIn&flowEntry=ServiceLogin) to work with our resources and your teams.
24 |
25 | - **Wifi + Space**: since we are all working remotely, find yourself a comfortable working space with a reliable internet connection.
26 |
27 | ## Schedule
28 |
29 | The datathon will be held March 26-27th (EST) online via (Zoom).
30 |
31 | The event will start off with introductions and a look at the tools you’ll be using. After a team formation activity, the majority of our time will be dedicated for group work. We will also be doing a few check-ins via Zoom to make sure everyone is doing fine. Final presentations will happen on Friday.
32 |
33 | 
34 |
35 | Unfortunately, because we can’t meet in person we won’t be holding a dinner/social, but feel free to use our hashtag **#hackarchives** in your social media posts.
36 |
37 | ## Team Collaboration Resources
38 |
39 | - **[#NY Datathon (Online) Resources](https://drive.google.com/drive/u/0/folders/1KdqBwpUTheYei_IjPaIXFxVLI5UH1_dL)**: This is a shared folder for participants, housing our introductory Google slides, additional resources, and where teams will save their final presentations.
40 | - **[Team Final Projects Folder](https://drive.google.com/drive/u/0/folders/138xKFQwHrjJEnruZ21lkK5UAxFPU2GLz)**: we have set up a folder where you can directly work on your presentations. We ask that you use Google Slides to help us with quick transitions between groups. Please name your slides using the following convention: AU-NY2020-TeamName.
41 | - **[Quick Guide to Setting up Colab Notebooks](https://youtu.be/JDXQRUp_Tx4)**: If you need a quick review of how to set up your Colab Notebook environment, we have a short tutorial.
42 | - **Project Examples**: Need a bit more inspiration before starting? Checkout projects from previous datathons: [Toronto](https://archivesunleashed.org/toronto/) | [Vancouver](https://archivesunleashed.org/vancouver/) | [Washington](https://archivesunleashed.org/washington/)
43 |
44 |
45 | ## Homework Refresher
46 |
47 | - **[Command Line/Terminal Tutorial](https://programminghistorian.org/en/lessons/intro-to-bash)**: For those new to command line, or if you'd like a refresher, check out this tutorial on the Programing Historian Introduction to the Bash Command Line by Ian Milligan and James Baker.
48 | - **[Archives Unleashed Toolkit Walkthrough](https://github.com/archivesunleashed/aut-docs/blob/master/current/toolkit-walkthrough.md)**: While we’ll be doing a lot of work in the notebooks during the datathon, it is good to have a conceptual understanding around working with data.
49 | - **Archives Unleashed Notebooks Reading**: Nick Ruest has been working on some new methods of working with derivatives created through the Archives Unleashed Cloud. To learn more about this development, check out our latest Medium post: [Cloud-hosted web archive data: The winding path to web archive collections as data](https://news.archivesunleashed.org/cloud-hosted-web-archive-data-the-winding-path-to-web-archive-collections-as-data-a2b3428701b7).
50 | - **[Notebooks Repository Walkthrough]()**: we suggest that you run through the two created for this event to become familiar with how the environment works and the types of analysis you can run. In each link click the “Open in Colab” button at the top of the page to launch and start exploring.
51 | - [parquet_pandas_stonewall.ipynb](https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_stonewall.ipynb)
52 | - [parquet_text_analyis_popline.ipynb](https://github.com/archivesunleashed/notebooks/blob/master/parquet_text_analyis_popline.ipynb)
53 |
54 | ## Additional/External Tools for Analysis
55 |
56 | - [AUK Tutorials](https://cloud.archivesunleashed.org/derivatives): provide basics on using a variety of external tools to work with dataset derivatives.
57 | - [An Introduction to Gephi (Beginner)](https://cloud.archivesunleashed.org/derivatives/basic-gephi)
58 | - [Network Graphing Archived Websites With Gephi (Intermediate)](https://cloud.archivesunleashed.org/derivatives/gephi)
59 | - [Grep - Filtering the Full-Text Derivative File](https://cloud.archivesunleashed.org/derivatives/text-filtering)
60 | - [Text Analysis Part One: Beyond the Keyword Search: Using AntConc](https://cloud.archivesunleashed.org/derivatives/text-antconc)
61 | - [Text Analysis Part Two: Sentiment Analysis With the Natural Language Toolkit](https://cloud.archivesunleashed.org/derivatives/text-sentiment)
62 | - [Voyant](https://voyant-tools.org)
63 |
64 | Also don’t forget about Excel (or Google spreadsheets)!
65 |
66 | ## Analysis Platforms
67 |
68 | We have a couple options for analysis platforms ([datasets are listed below](datathon-nyc#datasets)):
69 |
70 | - If you'd like to keep things relatively easy, and browser based, you're welcome to use [Google Colaboratory](https://colab.research.google.com). We have a couple options, in addition to the two mentioned above, available in the repo ([parquet_pandas_example.ipynb](https://github.com/archivesunleashed/notebooks/blob/master/parquet_pandas_example.ipynb), [parquet_pyspark_example.ipynb](https://github.com/archivesunleashed/notebooks/blob/master/parquet_pyspark_example.ipynb), and [parquet_text_analyis.ipynb](https://github.com/archivesunleashed/notebooks/blob/master/parquet_text_analyis.ipynb)) that can be used as starting points.
71 | - If you're comfortable with the command line, you're welcome to use one of the virtual machines provided by [Compute Canada](https://www.computecanada.ca/home/).
72 |
73 | The virtual machines, and the machine backing a given Google Colab notebook are roughly similar resource-wise. If you'd like to use a Compute Canada virtual machine, you'll have a bit more storage space, and you'll have much more control. You're welcome to install what you need to on the machines with `apt` to complete your project.
74 |
75 | ## Inspiration
76 |
77 | If you're looking for inspiration, check out the following notebook resources:
78 |
79 | - [GLAM Workbench](https://glam-workbench.github.io/)
80 | - [Awesome Jupyter GLAM](https://github.com/LibraryCarpentry/awesome-jupyter-glam)
81 | - [Getting started with ODate](https://o-date.github.io/support/notebooks-toc/)
82 | - [Jupyter notebooks for digital humanities](https://github.com/quinnanya/dh-jupyter)
83 | - [Spark NLP Workshop](https://github.com/JohnSnowLabs/spark-nlp-workshop)
84 | - [UW Interactive Data Lab, Data Visualization Curriculum](https://github.com/uwdata/visualization-curriculum)
85 | - [Sentiment Analysis with Pyspark](https://github.com/tthustla/setiment_analysis_pyspark/blob/master/Sentiment%20Analysis%20with%20PySpark.ipynb)
86 | - [Analysis of Car Accidents in Canada using PySpark](https://github.com/SinghGursimran/Analysis-of-Car-Accidents-in-Canada-using-PySpark/)
87 | - [Save Page Now Analysis](https://github.com/edsu/spn)
88 |
89 | If you'd like to add your notebook to the repo at the end of the datathon, we'd love to have it!
90 |
91 | ## Virtual Machines
92 |
93 | **c8-30gb-430gb**
94 |
95 | - 8 cores
96 | - 30G RAM
97 | - 11G `/`
98 | - 398G `/mnt`
99 | - Python 3.7.3 ([Anaconda](https://www.anaconda.com/distribution/))
100 | - findspark, wordcloud, spacy, pyarrow, jupyter, tldextract, jupyter_contrib_nbextensions, pyspark
101 | - feel free to `conda install` or `pip install` whatever else you need
102 | - OpenJDK 8
103 | - Spark 2.4.5
104 | - `SPARK_HOME=/home/ubuntu/spark`
105 | - jq
106 |
107 | Four machines available:
108 |
109 | - 206.167.181.146 (datathon1)
110 | - 206.167.182.14 (datathon2)
111 | - 206.167.181.104 (datathon3)
112 | - 206.167.181.105 (datathon4)
113 |
114 | **c8-45gb-430gb**
115 |
116 | - 8 Cores
117 | - 45G RAM
118 | - 11G `/`
119 | - 398G `/mnt`
120 | - Python 3.7.3 ([Anaconda](https://www.anaconda.com/distribution/))
121 | - findspark, wordcloud, spacy, pyarrow, jupyter, tldextract, jupyter_contrib_nbextensions, pyspark
122 | - feel free to `conda install` or `pip install` whatever else you need
123 | - OpenJDK 8
124 | - Spark 2.4.5
125 | - `SPARK_HOME=/home/ubuntu/spark`
126 | - jq
127 |
128 | One machine available:
129 |
130 | - 206.167.181.253 (datathon5)
131 |
132 | ## Getting Started
133 |
134 | Please note, the datathon hosts (Ian, Jimmy, Nick, and Sam) all use macOS or Linux variants. If you are on a Windows machine, things might be a little bit more difficult. If you have the [Windows SubSystem for Linux](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/) installed, you should be in a really good place.
135 |
136 | You may want to create a separate folder on your desktop or home directory to keep all of your datathon work in; this will also allow you to point terminal to one directory (folder).
137 |
138 | ### Shell into assigned VM
139 |
140 | Ian or Nick will provide you with key to access a virtual machine via ssh. You'll need to download or copy that key to your own machine, and apply the appropriate permissions to it. The permissions on the key should be `600`. You can do this with the following command on your own laptop before shelling in:
141 |
142 | ```bash
143 | chmod 600 /path/to/archives-hackathon.key
144 | ```
145 |
146 | Once you have the permissions set on the key, you can shell into your assigned datathon virtual machine with the provided key, and IP address:
147 |
148 | Example:
149 |
150 | ```bash
151 | ssh -L 8888:localhost:8888 -i ~/.ssh/archives-hackathon.key ubuntu@206.167.181.253
152 | ```
153 |
154 | ### Start Juypter Notebook
155 |
156 | ```bash
157 | cd /mnt/notebooks/datathon-nyc
158 | jupyter notebook --no-browser
159 | ```
160 |
161 | Click on the the localhost link to open up Juypter.
162 |
163 | 
164 |
165 | Select one of the two example notebooks.
166 |
167 | 
168 |
169 |
170 | ### Start Spark with AUT
171 |
172 | If you'd like to use Apache Spark and the Archives Unleashed Toolkit to analyze WARC/ARCs, you can get Spark started with the toolkit with the following command:
173 |
174 | ```bash
175 | ~/spark/bin/spark-shell --packages "io.archivesunleashed:aut:0.50.0"
176 | ```
177 |
178 | Documentation for the 0.50.0 release is available [here](https://github.com/archivesunleashed/aut-docs/tree/master/aut-0.50.0), and if you need a refresher on the datathon homework, it is available [here](https://github.com/archivesunleashed/aut-docs/blob/master/aut-0.50.0/toolkit-walkthrough.md).
179 |
180 | ## Datasets
181 |
182 | Scholarly derivatives created on cloud.archivesunleashed.org, and Parquet files should be downloaded to `/mnt/data`. If any team would like some WARC/ARC files from a collection, please work with Nick Ruest as soon as possible (`ruebot` in Slack).
183 |
184 | **Ivy Plus Libraries Confederation**
185 |
186 | - [National Statistical Offices and Central Banks Web Archive](https://zenodo.org/record/3633683)
187 | - [Contemporary Composers Web Archive (CCWA)](https://zenodo.org/record/3692559)
188 | - [#MeToo and the Women's Rights Movement in China Web Archive](https://zenodo.org/record/3633681)
189 | - [Geologic Field Trip Guidebooks Web Archive](https://zenodo.org/record/3666295)
190 | - [Literary Authors from Europe and Eurasia Web Archive](https://zenodo.org/record/3632728)
191 | - [Web Archive of Independent News Sites on Turkish Affairs](https://zenodo.org/record/3633234)
192 | - [State Elections Web Archive](https://zenodo.org/record/3635634)
193 | - [Brazilian Presidential Transition (2018) Web Archive](https://zenodo.org/record/3659692)
194 | - [Collaborative Architecture, Urbanism, and Sustainability Web Archive (CAUSEWAY)](https://zenodo.org/record/3674173)
195 | - [Global Webcomics Web Archive](https://zenodo.org/record/3633737)
196 | - [Queer Japan Web Archive](https://zenodo.org/record/3633284)
197 | - [Extreme Right Movements in Europe](https://zenodo.org/record/3633161)
198 | - [Latin American and Caribbean Contemporary Art Web Archive](https://zenodo.org/record/3633118)
199 | - [Popline and K4Health Web Archive](https://zenodo.org/record/3633022)
200 | - [Eastern Europe and Former Soviet Union Web Archive](https://zenodo.org/record/3633031)
201 | - [Independent Documentary Filmmakers from China, Hong Kong, and Taiwan Web Archive](https://zenodo.org/record/3632912)
202 |
203 | **Columbia University Libraries**
204 |
205 | - [General](https://zenodo.org/record/3633290)
206 | - [Resistance](https://zenodo.org/record/3660457)
207 | - [Stonewall 50 Commemoration](https://zenodo.org/record/3631347)
208 | - [Freely Accessible eJournals](https://zenodo.org/record/3633671)
209 | - [Avery Library Historic Preservation and Urban Planning](https://doi.org/10.5683/SP2/Z68EVJ)
210 | - [Rare Book and Manuscript Library](https://zenodo.org/record/3701593)
211 | - [Burke Library New York City Religions](https://zenodo.org/record/3701455)
212 |
213 | ## Sponsors + Special Thanks
214 |
215 | This event is possible thanks to the generous support from:
216 |
217 | [Andrew W. Mellon Foundation](https://mellon.org/), [Columbia University Libraries](https://library.columbia.edu), [Ivy Plus Libraries Confederation](https://library.columbia.edu/collections/web-archives/Ivy_Plus_Libraries.html), [Faculty of Arts](https://uwaterloo.ca/arts/) and [David R. Cheriton School of Computer Science](https://cs.uwaterloo.ca/) at the [University of Waterloo](https://uwaterloo.ca/), [York University Libraries](https://www.library.yorku.ca/web/), [Compute Canada](https://www.computecanada.ca/), and [Start Smart Labs](http://www.startsmartlabs.com/).
218 |
219 | We'd also like to say a special thanks to Columbia University and Ivy Plus libraries Confederation for for providing access to their collections!
220 |
--------------------------------------------------------------------------------
/geocities/geocities_domain_frequency.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "geocities-domain-frequency.ipynb",
7 | "provenance": [],
8 | "authorship_tag": "ABX9TyM2I/5HIhKxgXC4r24AYu/i",
9 | "include_colab_link": true
10 | },
11 | "kernelspec": {
12 | "name": "python3",
13 | "display_name": "Python 3"
14 | },
15 | "language_info": {
16 | "name": "python"
17 | }
18 | },
19 | "cells": [
20 | {
21 | "cell_type": "markdown",
22 | "metadata": {
23 | "id": "view-in-github",
24 | "colab_type": "text"
25 | },
26 | "source": [
27 | ""
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "source": [
33 | "# Working with Geocities Derivatives | Domain Frequency\n",
34 | "\n",
35 | "In this notebook we'll download some data from the [GeoCities Web Archive Collection Derivatives](https://archive.org/details/geocities-webarchive-collection-derivatives) to demonstrate a few examples of further exploration of web archive data."
36 | ],
37 | "metadata": {
38 | "id": "ZMfzPF61MeaJ"
39 | }
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "source": [
44 | "# Datasets\n",
45 | "\n",
46 | "First, we will need to download some derivative data from [GeoCities Web Archive Collection Derivatives](https://archive.org/details/geocities-webarchive-collection-derivatives)."
47 | ],
48 | "metadata": {
49 | "id": "SRt0p0ViMkSh"
50 | }
51 | },
52 | {
53 | "cell_type": "code",
54 | "source": [
55 | "%%capture\n",
56 | "\n",
57 | "!mkdir data\n",
58 | "\n",
59 | "!wget \"https://archive.org/download/geocities-webarchive-collection-derivatives/geocities-domain-frequency.csv.gz\" -P data"
60 | ],
61 | "metadata": {
62 | "id": "WO_U-V1sMu6N"
63 | },
64 | "execution_count": 1,
65 | "outputs": []
66 | },
67 | {
68 | "cell_type": "markdown",
69 | "source": [
70 | "Unzip the data."
71 | ],
72 | "metadata": {
73 | "id": "bGKddBwyNBvJ"
74 | }
75 | },
76 | {
77 | "cell_type": "code",
78 | "source": [
79 | "!gunzip data/*"
80 | ],
81 | "metadata": {
82 | "colab": {
83 | "base_uri": "https://localhost:8080/"
84 | },
85 | "id": "ycdyNiE8NC81",
86 | "outputId": "034e9302-44d7-4d0c-9d2a-c8947b884352"
87 | },
88 | "execution_count": 23,
89 | "outputs": [
90 | {
91 | "output_type": "stream",
92 | "name": "stdout",
93 | "text": [
94 | "gzip: data/geocities-domain-frequency.csv: unknown suffix -- ignored\n"
95 | ]
96 | }
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "source": [
102 | "Let's check our `data` directory, and make sure they've downloaded."
103 | ],
104 | "metadata": {
105 | "id": "YqJrkzNFNFnE"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "!ls -1 data"
112 | ],
113 | "metadata": {
114 | "colab": {
115 | "base_uri": "https://localhost:8080/"
116 | },
117 | "id": "fDEOuQHTNNyr",
118 | "outputId": "e0ad699b-2115-413e-cbc1-4f7b27a2f188"
119 | },
120 | "execution_count": 24,
121 | "outputs": [
122 | {
123 | "output_type": "stream",
124 | "name": "stdout",
125 | "text": [
126 | "geocities-domain-frequency.csv\n"
127 | ]
128 | }
129 | ]
130 | },
131 | {
132 | "cell_type": "markdown",
133 | "source": [
134 | "# Environment\n",
135 | "\n",
136 | "Next, we'll setup our environment so we can load our derivatives into [pandas](https://pandas.pydata.org), build charts with [Altair](https://altair-viz.github.io/), and use the [Data Table extension for Colab](https://colab.research.google.com/notebooks/data_table.ipynb)."
137 | ],
138 | "metadata": {
139 | "id": "2x2RuLJnOWT0"
140 | }
141 | },
142 | {
143 | "cell_type": "code",
144 | "source": [
145 | "import pandas as pd\n",
146 | "import altair as alt"
147 | ],
148 | "metadata": {
149 | "id": "9_Tz1e1FOZ_E"
150 | },
151 | "execution_count": 28,
152 | "outputs": []
153 | },
154 | {
155 | "cell_type": "code",
156 | "source": [
157 | "%load_ext google.colab.data_table"
158 | ],
159 | "metadata": {
160 | "id": "1q_8190bOgNn"
161 | },
162 | "execution_count": 19,
163 | "outputs": []
164 | },
165 | {
166 | "cell_type": "markdown",
167 | "source": [
168 | "## Let's take a look at the domain frequency derivative."
169 | ],
170 | "metadata": {
171 | "id": "H2nGI5pYOjTP"
172 | }
173 | },
174 | {
175 | "cell_type": "code",
176 | "source": [
177 | "domain_frequency = pd.read_csv(\"data/geocities-domain-frequency.csv\")\n",
178 | "domain_frequency"
179 | ],
180 | "metadata": {
181 | "colab": {
182 | "base_uri": "https://localhost:8080/",
183 | "height": 441
184 | },
185 | "id": "8jWQvOOpOoYh",
186 | "outputId": "ad264a43-1ef5-483b-a7ad-473ad5493153"
187 | },
188 | "execution_count": 25,
189 | "outputs": [
190 | {
191 | "output_type": "stream",
192 | "name": "stdout",
193 | "text": [
194 | "Warning: total number of rows (147923) exceeds max_rows (20000). Falling back to pandas display.\n"
195 | ]
196 | },
197 | {
198 | "output_type": "execute_result",
199 | "data": {
200 | "text/plain": [
201 | " domain count\n",
202 | "0 geocities.com 57922449\n",
203 | "1 yahoo.com 1110567\n",
204 | "2 amazon.com 87675\n",
205 | "3 myspace.com 67706\n",
206 | "4 bravenet.com 62904\n",
207 | "... ... ...\n",
208 | "147918 manosguardanapo.blogspot.com 1\n",
209 | "147919 bragaplasticos.com.ar 1\n",
210 | "147920 abed.org.br 1\n",
211 | "147921 mailearners.net 1\n",
212 | "147922 greensfirst.com 1\n",
213 | "\n",
214 | "[147923 rows x 2 columns]"
215 | ],
216 | "text/html": [
217 | "\n",
218 | " | \n", 238 | " | domain | \n", 239 | "count | \n", 240 | "
|---|---|---|
| 0 | \n", 245 | "geocities.com | \n", 246 | "57922449 | \n", 247 | "
| 1 | \n", 250 | "yahoo.com | \n", 251 | "1110567 | \n", 252 | "
| 2 | \n", 255 | "amazon.com | \n", 256 | "87675 | \n", 257 | "
| 3 | \n", 260 | "myspace.com | \n", 261 | "67706 | \n", 262 | "
| 4 | \n", 265 | "bravenet.com | \n", 266 | "62904 | \n", 267 | "
| ... | \n", 270 | "... | \n", 271 | "... | \n", 272 | "
| 147918 | \n", 275 | "manosguardanapo.blogspot.com | \n", 276 | "1 | \n", 277 | "
| 147919 | \n", 280 | "bragaplasticos.com.ar | \n", 281 | "1 | \n", 282 | "
| 147920 | \n", 285 | "abed.org.br | \n", 286 | "1 | \n", 287 | "
| 147921 | \n", 290 | "mailearners.net | \n", 291 | "1 | \n", 292 | "
| 147922 | \n", 295 | "greensfirst.com | \n", 296 | "1 | \n", 297 | "
147923 rows × 2 columns
\n", 301 | "| \n", 603 | " | domain | \n", 604 | "count | \n", 605 | "tld | \n", 606 | "
|---|---|---|---|
| 0 | \n", 611 | "geocities.com | \n", 612 | "57922449 | \n", 613 | "com | \n", 614 | "
| 1 | \n", 617 | "yahoo.com | \n", 618 | "1110567 | \n", 619 | "com | \n", 620 | "
| 2 | \n", 623 | "amazon.com | \n", 624 | "87675 | \n", 625 | "com | \n", 626 | "
| 3 | \n", 629 | "myspace.com | \n", 630 | "67706 | \n", 631 | "com | \n", 632 | "
| 4 | \n", 635 | "bravenet.com | \n", 636 | "62904 | \n", 637 | "com | \n", 638 | "
| ... | \n", 641 | "... | \n", 642 | "... | \n", 643 | "... | \n", 644 | "
| 147918 | \n", 647 | "manosguardanapo.blogspot.com | \n", 648 | "1 | \n", 649 | "com | \n", 650 | "
| 147919 | \n", 653 | "bragaplasticos.com.ar | \n", 654 | "1 | \n", 655 | "com.ar | \n", 656 | "
| 147920 | \n", 659 | "abed.org.br | \n", 660 | "1 | \n", 661 | "org.br | \n", 662 | "
| 147921 | \n", 665 | "mailearners.net | \n", 666 | "1 | \n", 667 | "net | \n", 668 | "
| 147922 | \n", 671 | "greensfirst.com | \n", 672 | "1 | \n", 673 | "com | \n", 674 | "
147923 rows × 3 columns
\n", 678 | "