![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
26 | ]
27 | },
28 | {
29 | "cell_type": "markdown",
30 | "source": [
31 | "# HTML Information Dataset Exploration\n",
32 | "\n",
33 | "We're going to take a look at a few examples of how we can explore the HTML Information dataset. \n",
34 | "\n",
35 | "The first thing we need to do is enter the URL for our HTML Information dataset in the cell below. You can get this by right clicking the Download icon, and selecting \"Copy Link\"."
36 | ],
37 | "metadata": {
38 | "id": "vAyuRQ2PJIdc"
39 | }
40 | },
41 | {
42 | "cell_type": "code",
43 | "source": [
44 | "dataset = 'https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-14462/TextFilesInformationExtraction/html-file-information.csv.gz?access=UCQ7VUUU4NDLKSGIPQD2R2WUGLOQXWPQ' #@param {type:\"string\"}\n",
45 | "print(dataset)"
46 | ],
47 | "metadata": {
48 | "colab": {
49 | "base_uri": "https://localhost:8080/"
50 | },
51 | "cellView": "form",
52 | "id": "RfhJiesWVpAf",
53 | "outputId": "507325b4-2e56-4aaf-bc7b-55b38fe221c8"
54 | },
55 | "execution_count": 1,
56 | "outputs": [
57 | {
58 | "output_type": "stream",
59 | "name": "stdout",
60 | "text": [
61 | "https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-14462/TextFilesInformationExtraction/html-file-information.csv.gz?access=UCQ7VUUU4NDLKSGIPQD2R2WUGLOQXWPQ\n"
62 | ]
63 | }
64 | ]
65 | },
66 | {
67 | "cell_type": "markdown",
68 | "source": [
69 | "## pandas\n",
70 | "\n",
71 | "Next, we'll setup our environment so we can load our HTML Information dataset into [pandas](https://pandas.pydata.org) DataFrames. If you're unfamiliar with DataFrames, but you've worked with spreadsheets before, you should quickly feel comfortable."
72 | ],
73 | "metadata": {
74 | "id": "Z14F2cIWJVW0"
75 | }
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": 2,
80 | "metadata": {
81 | "id": "Chh6tt3HHF1s"
82 | },
83 | "outputs": [],
84 | "source": [
85 | "import pandas as pd"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "source": [
91 | "## Data Table Display\n",
92 | "\n",
93 | "Colab includes an extension that renders pandas DataFrames into interactive displays that can be filtered, sorted, and explored dynamically. This can be very useful for taking a look at what each DataFrame provides, and doing some intital filtering!\n",
94 | "\n",
95 | "Data table display for pandas DataFrames can be enabled by running:\n",
96 | "```python\n",
97 | "%load_ext google.colab.data_table\n",
98 | "```\n",
99 | "and disabled by running\n",
100 | "```python\n",
101 | "%unload_ext google.colab.data_table\n",
102 | "```"
103 | ],
104 | "metadata": {
105 | "id": "sH81XCf3I3xY"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "%load_ext google.colab.data_table"
112 | ],
113 | "metadata": {
114 | "id": "-qyCnbvBI7n6"
115 | },
116 | "execution_count": 3,
117 | "outputs": []
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "source": [
122 | "## Loading our ARCH Dataset as a DataFrame\n",
123 | "\n",
124 | "---\n",
125 | "\n",
126 | "\n",
127 | "Next, we'll create pandas DataFrame from our dataset and show a preview of it using the Data Table Display."
128 | ],
129 | "metadata": {
130 | "id": "6prR7j1zI_D5"
131 | }
132 | },
133 | {
134 | "cell_type": "code",
135 | "source": [
136 | "html = pd.read_csv(dataset, compression='gzip')\n",
137 | "html"
138 | ],
139 | "metadata": {
140 | "id": "YL0LQaUNHRKx"
141 | },
142 | "execution_count": 4,
143 | "outputs": []
144 | },
145 | {
146 | "cell_type": "markdown",
147 | "metadata": {
148 | "id": "3HPwOCNAvqMe"
149 | },
150 | "source": [
151 | "# Data Analysis\n",
152 | "\n",
153 | "Now that we have all of our datasets loaded up, we can begin to work with them!"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "metadata": {
159 | "id": "J6Pkg0prv3BE"
160 | },
161 | "source": [
162 | "## Counting total files, and unique files\n",
163 | "\n",
164 | "Let's take a quick look at how to count items in DataFrames, and use total and unique files as an example to work with.\n",
165 | "\n",
166 | "It's definitely work checking out the [pandas documentation](https://pandas.pydata.org/docs/index.html). There are a lot of good examples available, along with a robust [API reference](https://pandas.pydata.org/docs/reference/index.html#api)."
167 | ]
168 | },
169 | {
170 | "cell_type": "markdown",
171 | "metadata": {
172 | "id": "DFX4Gl3wv7bi"
173 | },
174 | "source": [
175 | "\n",
176 | "### How many html files are in this collection?\n",
177 | "\n",
178 | "We can take our `html` variable and try a couple of functions to get the same answer.\n",
179 | "\n",
180 | "1. `len(html.index)`\n",
181 | " * Get the length of the DataFrame's index.\n",
182 | "2. `html.shape[0]`\n",
183 | " * Get the shape or dimensionality of the DataFrame, and take the first item in the tuple.\n",
184 | "3. `html.count()`\n",
185 | " * Count the number of rows for each column.\n",
186 | "\n"
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "metadata": {
192 | "id": "HTv8Oet3jiTH"
193 | },
194 | "source": [
195 | "len(html.index)"
196 | ],
197 | "execution_count": 5,
198 | "outputs": []
199 | },
200 | {
201 | "cell_type": "code",
202 | "metadata": {
203 | "id": "6rYEERnTjifk"
204 | },
205 | "source": [
206 | "html.shape[0]"
207 | ],
208 | "execution_count": 6,
209 | "outputs": []
210 | },
211 | {
212 | "cell_type": "code",
213 | "metadata": {
214 | "id": "bn-1v127aKIG"
215 | },
216 | "source": [
217 | "html.count()"
218 | ],
219 | "execution_count": 7,
220 | "outputs": []
221 | },
222 | {
223 | "cell_type": "markdown",
224 | "metadata": {
225 | "id": "38veKiPhwKo4"
226 | },
227 | "source": [
228 | "### How many unique html files are in the collection?\n",
229 | "\n",
230 | " We can see if an HTML file is unique or not by computing an [MD5 hash](https://en.wikipedia.org/wiki/MD5#MD5_hashes) of it, and comparing them. The exact same html file might have a filename of `example.html` or `foo.html`. If the hash is computed for each, we can see that even with different file names, they are actually the same html file. So, since we have both a `MD5` and `SHA1` hash column available in our DataFrame, we can just find the unique values, and count them!\n",
231 | "\n",
232 | "\n"
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "metadata": {
238 | "id": "WesM3kQowM5B"
239 | },
240 | "source": [
241 | "len(html.md5.unique())"
242 | ],
243 | "execution_count": 8,
244 | "outputs": []
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {
249 | "id": "ZIXkI0-1wWQf"
250 | },
251 | "source": [
252 | "### What are the top 10 most occurring html files in the collection?\n",
253 | "\n",
254 | "Here we can take advantage of [`value_counts()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.value_counts.html) to provide us with a list of MD5 hashes and their respective counts."
255 | ]
256 | },
257 | {
258 | "cell_type": "code",
259 | "metadata": {
260 | "id": "8Ts03OFyjPIM"
261 | },
262 | "source": [
263 | "html[\"md5\"].value_counts().head(10)"
264 | ],
265 | "execution_count": 9,
266 | "outputs": []
267 | },
268 | {
269 | "cell_type": "markdown",
270 | "metadata": {
271 | "id": "FG7pGZUEwlaI"
272 | },
273 | "source": [
274 | "\n",
275 | "### What's the information around all of the occurances of `d41d8cd98f00b204e9800998ecf8427e`?\n",
276 | "\n",
277 | "What, you mean you don't know what `d41d8cd98f00b204e9800998ecf8427e` means? \n",
278 | "\n",
279 | "Let's find those HTML files in the DataFrame. We can here see some of the filenames used, its dimensions, and its URL.\n"
280 | ]
281 | },
282 | {
283 | "cell_type": "code",
284 | "metadata": {
285 | "id": "msmmm65lkSIK"
286 | },
287 | "source": [
288 | "html.loc[html[\"md5\"] == \"d41d8cd98f00b204e9800998ecf8427e\"]"
289 | ],
290 | "execution_count": null,
291 | "outputs": []
292 | },
293 | {
294 | "cell_type": "markdown",
295 | "metadata": {
296 | "id": "GbLLZW2awzCv"
297 | },
298 | "source": [
299 | "### What are the top 10 most occuring filenames in the collection?\n",
300 | "\n",
301 | "Note that this is of course different than the MD5 results up above. Here we are focusing _just_ on filename. So `a16180790160.html ` for example, might actually be referring to different HTML files who happen to have the same name.\n",
302 | "\n",
303 | "Here we can use `value_counts()` again, but this time we'll create a variable for the top filenames so we can use it later.\n",
304 | "\n"
305 | ]
306 | },
307 | {
308 | "cell_type": "code",
309 | "metadata": {
310 | "id": "pQaw54ACkwdZ"
311 | },
312 | "source": [
313 | "top_filenames = html[\"filename\"].value_counts().head(10)\n",
314 | "top_filenames"
315 | ],
316 | "execution_count": null,
317 | "outputs": []
318 | },
319 | {
320 | "cell_type": "markdown",
321 | "metadata": {
322 | "id": "Z7F3re20BQRI"
323 | },
324 | "source": [
325 | "### Let's create our first graph!\n",
326 | "\n",
327 | "We'll first plot the data with the pandas [plot](https://pandas.pydata.org/docs/reference/api/pandas.Series.plot.html) functionality, and then with [Altair](https://altair-viz.github.io/)."
328 | ]
329 | },
330 | {
331 | "cell_type": "code",
332 | "metadata": {
333 | "id": "sRvlstfsBWEZ"
334 | },
335 | "source": [
336 | "top_filenames_chart = top_filenames.plot.bar(figsize=(25, 10))\n",
337 | "\n",
338 | "top_filenames_chart.set_title(\"Top Filenames\", fontsize=22)\n",
339 | "top_filenames_chart.set_xlabel(\"Filename\", fontsize=20)\n",
340 | "top_filenames_chart.set_ylabel(\"Count\", fontsize=20)"
341 | ],
342 | "execution_count": null,
343 | "outputs": []
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {
348 | "id": "pQgeOObvgLvK"
349 | },
350 | "source": [
351 | "Now let's setup [Altair](https://altair-viz.github.io/), and plot the data. Altair is useful for creating vizualizations since they can be easily exported as a PNG or SVG."
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "metadata": {
357 | "id": "Q7Z4J6qjWaVM"
358 | },
359 | "source": [
360 | "import altair as alt"
361 | ],
362 | "execution_count": null,
363 | "outputs": []
364 | },
365 | {
366 | "cell_type": "code",
367 | "metadata": {
368 | "id": "s0xwvILYWkgg"
369 | },
370 | "source": [
371 | "top_filenames_altair = (\n",
372 | " html[\"filename\"]\n",
373 | " .value_counts()\n",
374 | " .head(10)\n",
375 | " .rename_axis(\"Filename\")\n",
376 | " .reset_index(name=\"Count\")\n",
377 | ")\n",
378 | "\n",
379 | "filenames_bar = (\n",
380 | " alt.Chart(top_filenames_altair)\n",
381 | " .mark_bar()\n",
382 | " .encode(x=alt.X(\"Filename:O\", sort=\"-y\"), y=alt.Y(\"Count:Q\"))\n",
383 | ")\n",
384 | "\n",
385 | "filenames_rule = (\n",
386 | " alt.Chart(top_filenames_altair).mark_rule(color=\"red\").encode(y=\"mean(Count):Q\")\n",
387 | ")\n",
388 | "\n",
389 | "\n",
390 | "filenames_text = filenames_bar.mark_text(align=\"center\", baseline=\"bottom\").encode(\n",
391 | " text=\"Count:Q\"\n",
392 | ")\n",
393 | "\n",
394 | "(filenames_bar + filenames_rule + filenames_text).properties(\n",
395 | " width=1400, height=700, title=\"Top Filenames\"\n",
396 | ")"
397 | ],
398 | "execution_count": null,
399 | "outputs": []
400 | },
401 | {
402 | "cell_type": "markdown",
403 | "metadata": {
404 | "id": "BneaN9cgGoly"
405 | },
406 | "source": [
407 | "### How about a file format distribution?\n",
408 | "\n",
409 | "What _kind_ of html files are present? We can discover this by checking their \"media type\", or [MIME type](https://en.wikipedia.org/wiki/Media_type). \n",
410 | "\n",
411 | "\n",
412 | "\n",
413 | "\n"
414 | ]
415 | },
416 | {
417 | "cell_type": "code",
418 | "metadata": {
419 | "id": "RDd-J8D-GwDk"
420 | },
421 | "source": [
422 | "html_mime_types = (\n",
423 | " html[\"mime_type_tika\"]\n",
424 | " .value_counts()\n",
425 | " .head(5)\n",
426 | " .rename_axis(\"MIME Type\")\n",
427 | " .reset_index(name=\"Count\")\n",
428 | ")\n",
429 | "\n",
430 | "html_mimes_bar = (\n",
431 | " alt.Chart(html_mime_types)\n",
432 | " .mark_bar()\n",
433 | " .encode(x=alt.X(\"MIME Type:O\", sort=\"-y\"), y=alt.Y(\"Count:Q\"))\n",
434 | ")\n",
435 | "\n",
436 | "html_mime_rule = (\n",
437 | " alt.Chart(html_mime_types).mark_rule(color=\"red\").encode(y=\"mean(Count):Q\")\n",
438 | ")\n",
439 | "\n",
440 | "html_mime_text = html_mimes_bar.mark_text(align=\"center\", baseline=\"bottom\").encode(\n",
441 | " text=\"Count:Q\"\n",
442 | ")\n",
443 | "\n",
444 | "(html_mimes_bar + html_mime_rule + html_mime_text).properties(\n",
445 | " width=1400, height=700, title=\"HTML File Format Distribution\"\n",
446 | ")"
447 | ],
448 | "execution_count": null,
449 | "outputs": []
450 | },
451 | {
452 | "cell_type": "markdown",
453 | "metadata": {
454 | "id": "QUJR-jjqNxCL"
455 | },
456 | "source": [
457 | "### How do I get the actual html?\n",
458 | "\n",
459 | "...or, how do I get to the actual binary files described by each file format information derivative?\n",
460 | "\n",
461 | "There are a few options!\n",
462 | "\n",
463 | "1. `wget` or `curl` from the live URL, or a replay URL\n",
464 | " * Live web URL\n",
465 | " * `wget` or `curl` the value of the `url` column\n",
466 | " * Replay web URL\n",
467 | " * `wget` or `curl` the value of the `crawl_date` and `url` column using the following pattern:\n",
468 | " * `https://web.archive.org/web/` + `crawl_date` + `/` + `url`\n",
469 | " * https://web.archive.org/web/20120119124734/http://www.archive.org/images/glogo.png\n",
470 | " * `http://wayback.archive-it.org/14462/` + `crawl_date` + `/` + `url`\n",
471 | " * https://wayback.archive-it.org/14462/20210524212740/https://ruebot.net/visualization/elxn42/featured_hu33a17dfb90e2c5ed77f783db14a6e53a_5126291_550x0_resize_q90_box_2.png\n",
472 | "2. Use a scripting language, such as Python\n",
473 | " * Make use of the `url` and `filename` columns (and `crawl_date` if you want to use the replay URL)\n",
474 | " * `import requests`\n",
475 | " * `requests.get(url, allow_redirects=True)`\n",
476 | " * `open('filename', 'wb').write(r.content)`\n",
477 | "3. Use the [Archives Unleashed Toolkit](https://aut.docs.archivesunleashed.org/docs/extract-binary) (if you have access to the W/ARC files)."
478 | ]
479 | },
480 | {
481 | "cell_type": "markdown",
482 | "source": [
483 | "If you wanted to download the HTML files using the replay URL, below is a method for doing so.\n",
484 | "\n",
485 | "First, you'll want to setup a replay url base url. Here we'll use the Archive-It Wayback instance for the collection."
486 | ],
487 | "metadata": {
488 | "id": "8yoFE2xLAlwk"
489 | }
490 | },
491 | {
492 | "cell_type": "code",
493 | "source": [
494 | "wayback_url = 'http://wayback.archive-it.org/14462/'"
495 | ],
496 | "metadata": {
497 | "id": "RxrOHn_2AicZ"
498 | },
499 | "execution_count": null,
500 | "outputs": []
501 | },
502 | {
503 | "cell_type": "markdown",
504 | "source": [
505 | "Next we'll create a new column using a lambda function. If you're familiar working with spreadsheets, what we're doing here is basically concatenating some column values together and creating a new column."
506 | ],
507 | "metadata": {
508 | "id": "jCTmrEBGA-Yd"
509 | }
510 | },
511 | {
512 | "cell_type": "code",
513 | "source": [
514 | "html['replay_url'] = html.apply(lambda row: str(wayback_url + str(row['crawl_date']) + \"/\" + row['url']), axis=1)"
515 | ],
516 | "metadata": {
517 | "id": "p5HoWxkFA6C8"
518 | },
519 | "execution_count": null,
520 | "outputs": []
521 | },
522 | {
523 | "cell_type": "markdown",
524 | "source": [
525 | "Then we can export that new column we created out to a file, so we can use it with `wget` to download all the html files!"
526 | ],
527 | "metadata": {
528 | "id": "q87tSYyIBIiZ"
529 | }
530 | },
531 | {
532 | "cell_type": "code",
533 | "source": [
534 | "html['replay_url'].head().to_csv('14462_html_urls.txt', index=False, header=False)"
535 | ],
536 | "metadata": {
537 | "id": "wyhpsf9wBN1a"
538 | },
539 | "execution_count": null,
540 | "outputs": []
541 | },
542 | {
543 | "cell_type": "markdown",
544 | "source": [
545 | "Finally, we can pass the file to `wget` to use as a download list. You can also speed this process up using `xargs` or `parallel`."
546 | ],
547 | "metadata": {
548 | "id": "81a3q7feu5Ji"
549 | }
550 | },
551 | {
552 | "cell_type": "code",
553 | "source": [
554 | "!wget --random-wait -i 14462_html_urls.txt"
555 | ],
556 | "metadata": {
557 | "id": "D_p4qGKoBRaZ"
558 | },
559 | "execution_count": null,
560 | "outputs": []
561 | }
562 | ]
563 | }
--------------------------------------------------------------------------------
/arch/domain-frequency.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "domain-frequency.ipynb",
7 | "provenance": [],
8 | "include_colab_link": true
9 | },
10 | "kernelspec": {
11 | "name": "python3",
12 | "display_name": "Python 3"
13 | },
14 | "language_info": {
15 | "name": "python"
16 | }
17 | },
18 | "cells": [
19 | {
20 | "cell_type": "markdown",
21 | "metadata": {
22 | "id": "view-in-github",
23 | "colab_type": "text"
24 | },
25 | "source": [
26 | ""
27 | ]
28 | },
29 | {
30 | "cell_type": "markdown",
31 | "source": [
32 | "# Domain Frequency Dataset Exploration\n",
33 | "\n",
34 | "We're going to take a look at a few examples of how we can explore the Domain Frequency dataset. First we need to enter the URL for our Domain Frequency dataset. You can get this by right clicking the Download icon, and selecting \"Copy Link\"."
35 | ],
36 | "metadata": {
37 | "id": "vAyuRQ2PJIdc"
38 | }
39 | },
40 | {
41 | "cell_type": "code",
42 | "source": [
43 | "dataset = 'https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-06689/DomainFrequencyExtraction/domain-frequency.csv.gz?access=FUO54Y4B3J3GAZSDD6ML7DQENUN5BDOX' #@param {type:\"string\"}\n",
44 | "print(dataset)"
45 | ],
46 | "metadata": {
47 | "colab": {
48 | "base_uri": "https://localhost:8080/"
49 | },
50 | "cellView": "form",
51 | "id": "RfhJiesWVpAf",
52 | "outputId": "277fad1b-e6c8-4d4f-d2cb-11f38f484138"
53 | },
54 | "execution_count": 1,
55 | "outputs": [
56 | {
57 | "output_type": "stream",
58 | "name": "stdout",
59 | "text": [
60 | "https://webdata.archive-it.org/ait/files/download/ARCHIVEIT-06689/DomainFrequencyExtraction/domain-frequency.csv.gz?access=FUO54Y4B3J3GAZSDD6ML7DQENUN5BDOX\n"
61 | ]
62 | }
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "source": [
68 | "# Environment\n",
69 | "\n",
70 | "Next, we'll set up our environment so we can load our Domain Frequency dataset into [pandas](https://pandas.pydata.org) and use [Altair](https://altair-viz.github.io/) for plots. Altair is useful for creating plots since they can be easily exported as a PNG or SVG."
71 | ],
72 | "metadata": {
73 | "id": "Z14F2cIWJVW0"
74 | }
75 | },
76 | {
77 | "cell_type": "code",
78 | "execution_count": 2,
79 | "metadata": {
80 | "id": "Chh6tt3HHF1s"
81 | },
82 | "outputs": [],
83 | "source": [
84 | "import pandas as pd\n",
85 | "import altair as alt"
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "source": [
91 | "## Data Table Display\n",
92 | "\n",
93 | "Colab includes an extension that renders pandas DataFrames into interactive displays that can be filtered, sorted, and explored dynamically. This can be very useful for taking a look at what each DataFrame provides!\n",
94 | "\n",
95 | "Data table display for pandas DataFrames can be enabled by running:\n",
96 | "```python\n",
97 | "%load_ext google.colab.data_table\n",
98 | "```\n",
99 | "and disabled by running\n",
100 | "```python\n",
101 | "%unload_ext google.colab.data_table\n",
102 | "```"
103 | ],
104 | "metadata": {
105 | "id": "sH81XCf3I3xY"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "%load_ext google.colab.data_table"
112 | ],
113 | "metadata": {
114 | "id": "-qyCnbvBI7n6"
115 | },
116 | "execution_count": 3,
117 | "outputs": []
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "source": [
122 | "# Loading our ARCH Dataset as a DataFrame\n",
123 | "\n",
124 | "---\n",
125 | "\n",
126 | "\n",
127 | "Next, we'll create a pandas DataFrame from our dataset and show a preview of it using the Data Table Display."
128 | ],
129 | "metadata": {
130 | "id": "6prR7j1zI_D5"
131 | }
132 | },
133 | {
134 | "cell_type": "code",
135 | "source": [
136 | "domains = pd.read_csv(dataset, compression='gzip')\n",
137 | "domains"
138 | ],
139 | "metadata": {
140 | "colab": {
141 | "base_uri": "https://localhost:8080/",
142 | "height": 441
143 | },
144 | "id": "YL0LQaUNHRKx",
145 | "outputId": "498e4153-bb5a-427e-9eba-9a3aa10abdfe"
146 | },
147 | "execution_count": 4,
148 | "outputs": [
149 | {
150 | "output_type": "stream",
151 | "name": "stdout",
152 | "text": [
153 | "Warning: total number of rows (20407) exceeds max_rows (20000). Falling back to pandas display.\n"
154 | ]
155 | },
156 | {
157 | "output_type": "execute_result",
158 | "data": {
159 | "text/plain": [
160 | " domain count\n",
161 | "0 sld.cu 11002591\n",
162 | "1 cult.cu 5197445\n",
163 | "2 facebook.com 1985543\n",
164 | "3 icrt.cu 1936127\n",
165 | "4 cubava.cu 967903\n",
166 | "... ... ...\n",
167 | "20402 yardbarker.com 1\n",
168 | "20403 bufetetoro.com 1\n",
169 | "20404 understandingwar.org 1\n",
170 | "20405 headaches.org 1\n",
171 | "20406 nursite.com.ar 1\n",
172 | "\n",
173 | "[20407 rows x 2 columns]"
174 | ],
175 | "text/html": [
176 | "\n",
177 | " | \n", 197 | " | domain | \n", 198 | "count | \n", 199 | "
|---|---|---|
| 0 | \n", 204 | "sld.cu | \n", 205 | "11002591 | \n", 206 | "
| 1 | \n", 209 | "cult.cu | \n", 210 | "5197445 | \n", 211 | "
| 2 | \n", 214 | "facebook.com | \n", 215 | "1985543 | \n", 216 | "
| 3 | \n", 219 | "icrt.cu | \n", 220 | "1936127 | \n", 221 | "
| 4 | \n", 224 | "cubava.cu | \n", 225 | "967903 | \n", 226 | "
| ... | \n", 229 | "... | \n", 230 | "... | \n", 231 | "
| 20402 | \n", 234 | "yardbarker.com | \n", 235 | "1 | \n", 236 | "
| 20403 | \n", 239 | "bufetetoro.com | \n", 240 | "1 | \n", 241 | "
| 20404 | \n", 244 | "understandingwar.org | \n", 245 | "1 | \n", 246 | "
| 20405 | \n", 249 | "headaches.org | \n", 250 | "1 | \n", 251 | "
| 20406 | \n", 254 | "nursite.com.ar | \n", 255 | "1 | \n", 256 | "
20407 rows × 2 columns
\n", 260 | "| \n", 564 | " | domain | \n", 565 | "count | \n", 566 | "tld | \n", 567 | "
|---|---|---|---|
| 0 | \n", 572 | "sld.cu | \n", 573 | "11002591 | \n", 574 | "cu | \n", 575 | "
| 1 | \n", 578 | "cult.cu | \n", 579 | "5197445 | \n", 580 | "cu | \n", 581 | "
| 2 | \n", 584 | "facebook.com | \n", 585 | "1985543 | \n", 586 | "com | \n", 587 | "
| 3 | \n", 590 | "icrt.cu | \n", 591 | "1936127 | \n", 592 | "cu | \n", 593 | "
| 4 | \n", 596 | "cubava.cu | \n", 597 | "967903 | \n", 598 | "cu | \n", 599 | "
| ... | \n", 602 | "... | \n", 603 | "... | \n", 604 | "... | \n", 605 | "
| 20402 | \n", 608 | "yardbarker.com | \n", 609 | "1 | \n", 610 | "com | \n", 611 | "
| 20403 | \n", 614 | "bufetetoro.com | \n", 615 | "1 | \n", 616 | "com | \n", 617 | "
| 20404 | \n", 620 | "understandingwar.org | \n", 621 | "1 | \n", 622 | "org | \n", 623 | "
| 20405 | \n", 626 | "headaches.org | \n", 627 | "1 | \n", 628 | "org | \n", 629 | "
| 20406 | \n", 632 | "nursite.com.ar | \n", 633 | "1 | \n", 634 | "com.ar | \n", 635 | "
20407 rows × 3 columns
\n", 639 | ""
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "source": [
33 | "# Working with Geocities Derivatives | Domain Frequency\n",
34 | "\n",
35 | "In this notebook we'll download some data from the [GeoCities Web Archive Collection Derivatives](https://archive.org/details/geocities-webarchive-collection-derivatives) to demonstrate a few examples of further exploration of web archive data."
36 | ],
37 | "metadata": {
38 | "id": "ZMfzPF61MeaJ"
39 | }
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "source": [
44 | "# Datasets\n",
45 | "\n",
46 | "First, we will need to download some derivative data from [GeoCities Web Archive Collection Derivatives](https://archive.org/details/geocities-webarchive-collection-derivatives)."
47 | ],
48 | "metadata": {
49 | "id": "SRt0p0ViMkSh"
50 | }
51 | },
52 | {
53 | "cell_type": "code",
54 | "source": [
55 | "%%capture\n",
56 | "\n",
57 | "!mkdir data\n",
58 | "\n",
59 | "!wget \"https://archive.org/download/geocities-webarchive-collection-derivatives/geocities-domain-frequency.csv.gz\" -P data"
60 | ],
61 | "metadata": {
62 | "id": "WO_U-V1sMu6N"
63 | },
64 | "execution_count": 1,
65 | "outputs": []
66 | },
67 | {
68 | "cell_type": "markdown",
69 | "source": [
70 | "Unzip the data."
71 | ],
72 | "metadata": {
73 | "id": "bGKddBwyNBvJ"
74 | }
75 | },
76 | {
77 | "cell_type": "code",
78 | "source": [
79 | "!gunzip data/*"
80 | ],
81 | "metadata": {
82 | "colab": {
83 | "base_uri": "https://localhost:8080/"
84 | },
85 | "id": "ycdyNiE8NC81",
86 | "outputId": "034e9302-44d7-4d0c-9d2a-c8947b884352"
87 | },
88 | "execution_count": 23,
89 | "outputs": [
90 | {
91 | "output_type": "stream",
92 | "name": "stdout",
93 | "text": [
94 | "gzip: data/geocities-domain-frequency.csv: unknown suffix -- ignored\n"
95 | ]
96 | }
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "source": [
102 | "Let's check our `data` directory, and make sure they've downloaded."
103 | ],
104 | "metadata": {
105 | "id": "YqJrkzNFNFnE"
106 | }
107 | },
108 | {
109 | "cell_type": "code",
110 | "source": [
111 | "!ls -1 data"
112 | ],
113 | "metadata": {
114 | "colab": {
115 | "base_uri": "https://localhost:8080/"
116 | },
117 | "id": "fDEOuQHTNNyr",
118 | "outputId": "e0ad699b-2115-413e-cbc1-4f7b27a2f188"
119 | },
120 | "execution_count": 24,
121 | "outputs": [
122 | {
123 | "output_type": "stream",
124 | "name": "stdout",
125 | "text": [
126 | "geocities-domain-frequency.csv\n"
127 | ]
128 | }
129 | ]
130 | },
131 | {
132 | "cell_type": "markdown",
133 | "source": [
134 | "# Environment\n",
135 | "\n",
136 | "Next, we'll setup our environment so we can load our derivatives into [pandas](https://pandas.pydata.org), build charts with [Altair](https://altair-viz.github.io/), and use the [Data Table extension for Colab](https://colab.research.google.com/notebooks/data_table.ipynb)."
137 | ],
138 | "metadata": {
139 | "id": "2x2RuLJnOWT0"
140 | }
141 | },
142 | {
143 | "cell_type": "code",
144 | "source": [
145 | "import pandas as pd\n",
146 | "import altair as alt"
147 | ],
148 | "metadata": {
149 | "id": "9_Tz1e1FOZ_E"
150 | },
151 | "execution_count": 28,
152 | "outputs": []
153 | },
154 | {
155 | "cell_type": "code",
156 | "source": [
157 | "%load_ext google.colab.data_table"
158 | ],
159 | "metadata": {
160 | "id": "1q_8190bOgNn"
161 | },
162 | "execution_count": 19,
163 | "outputs": []
164 | },
165 | {
166 | "cell_type": "markdown",
167 | "source": [
168 | "## Let's take a look at the domain frequency derivative."
169 | ],
170 | "metadata": {
171 | "id": "H2nGI5pYOjTP"
172 | }
173 | },
174 | {
175 | "cell_type": "code",
176 | "source": [
177 | "domain_frequency = pd.read_csv(\"data/geocities-domain-frequency.csv\")\n",
178 | "domain_frequency"
179 | ],
180 | "metadata": {
181 | "colab": {
182 | "base_uri": "https://localhost:8080/",
183 | "height": 441
184 | },
185 | "id": "8jWQvOOpOoYh",
186 | "outputId": "ad264a43-1ef5-483b-a7ad-473ad5493153"
187 | },
188 | "execution_count": 25,
189 | "outputs": [
190 | {
191 | "output_type": "stream",
192 | "name": "stdout",
193 | "text": [
194 | "Warning: total number of rows (147923) exceeds max_rows (20000). Falling back to pandas display.\n"
195 | ]
196 | },
197 | {
198 | "output_type": "execute_result",
199 | "data": {
200 | "text/plain": [
201 | " domain count\n",
202 | "0 geocities.com 57922449\n",
203 | "1 yahoo.com 1110567\n",
204 | "2 amazon.com 87675\n",
205 | "3 myspace.com 67706\n",
206 | "4 bravenet.com 62904\n",
207 | "... ... ...\n",
208 | "147918 manosguardanapo.blogspot.com 1\n",
209 | "147919 bragaplasticos.com.ar 1\n",
210 | "147920 abed.org.br 1\n",
211 | "147921 mailearners.net 1\n",
212 | "147922 greensfirst.com 1\n",
213 | "\n",
214 | "[147923 rows x 2 columns]"
215 | ],
216 | "text/html": [
217 | "\n",
218 | " | \n", 238 | " | domain | \n", 239 | "count | \n", 240 | "
|---|---|---|
| 0 | \n", 245 | "geocities.com | \n", 246 | "57922449 | \n", 247 | "
| 1 | \n", 250 | "yahoo.com | \n", 251 | "1110567 | \n", 252 | "
| 2 | \n", 255 | "amazon.com | \n", 256 | "87675 | \n", 257 | "
| 3 | \n", 260 | "myspace.com | \n", 261 | "67706 | \n", 262 | "
| 4 | \n", 265 | "bravenet.com | \n", 266 | "62904 | \n", 267 | "
| ... | \n", 270 | "... | \n", 271 | "... | \n", 272 | "
| 147918 | \n", 275 | "manosguardanapo.blogspot.com | \n", 276 | "1 | \n", 277 | "
| 147919 | \n", 280 | "bragaplasticos.com.ar | \n", 281 | "1 | \n", 282 | "
| 147920 | \n", 285 | "abed.org.br | \n", 286 | "1 | \n", 287 | "
| 147921 | \n", 290 | "mailearners.net | \n", 291 | "1 | \n", 292 | "
| 147922 | \n", 295 | "greensfirst.com | \n", 296 | "1 | \n", 297 | "
147923 rows × 2 columns
\n", 301 | "| \n", 603 | " | domain | \n", 604 | "count | \n", 605 | "tld | \n", 606 | "
|---|---|---|---|
| 0 | \n", 611 | "geocities.com | \n", 612 | "57922449 | \n", 613 | "com | \n", 614 | "
| 1 | \n", 617 | "yahoo.com | \n", 618 | "1110567 | \n", 619 | "com | \n", 620 | "
| 2 | \n", 623 | "amazon.com | \n", 624 | "87675 | \n", 625 | "com | \n", 626 | "
| 3 | \n", 629 | "myspace.com | \n", 630 | "67706 | \n", 631 | "com | \n", 632 | "
| 4 | \n", 635 | "bravenet.com | \n", 636 | "62904 | \n", 637 | "com | \n", 638 | "
| ... | \n", 641 | "... | \n", 642 | "... | \n", 643 | "... | \n", 644 | "
| 147918 | \n", 647 | "manosguardanapo.blogspot.com | \n", 648 | "1 | \n", 649 | "com | \n", 650 | "
| 147919 | \n", 653 | "bragaplasticos.com.ar | \n", 654 | "1 | \n", 655 | "com.ar | \n", 656 | "
| 147920 | \n", 659 | "abed.org.br | \n", 660 | "1 | \n", 661 | "org.br | \n", 662 | "
| 147921 | \n", 665 | "mailearners.net | \n", 666 | "1 | \n", 667 | "net | \n", 668 | "
| 147922 | \n", 671 | "greensfirst.com | \n", 672 | "1 | \n", 673 | "com | \n", 674 | "
147923 rows × 3 columns
\n", 678 | "