├── web_scraping_maestría.pdf
├── LICENSE
├── README.md
└── Selenium_2023.ipynb


/web_scraping_maestría.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/GEJ1/web-scraping-python/HEAD/web_scraping_maestría.pdf


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2021 Gustavo Juantorena
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Web scraping con Python ⛏️
 2 | Recursos básicos para iniciarse en web scraping usando Python. 
 3 | 
 4 | 
 5 | Clase dictada por Gustavo Juantorena ([github](https://github.com/GEJ1) / [Linkedin](https://www.linkedin.com/in/gustavo-juantorena/)) como docente invitado en la materia <b>Text Mining</b> de la  [Maestría en Explotación de Datos y Descubrimiento de Conocimiento](http://datamining.dc.uba.ar/datamining/) de la Universidad de Buenos Aires.
 6 | El contenido aquí expuesto excede lo que se vió en a clase, pero me parece que está bueno que lo tengan como referencia para profundizar los contenidos.
 7 | * [Slides de la clase](https://docs.google.com/presentation/d/10-lc2Y6kMVHp7FO9v8ReZdY1MPwUlgxWIsSDePY0afg/edit?usp=sharing)
 8 | * [Primer Notebook de la clase (APIs + Beautiful Soup)](https://github.com/GEJ1/web-scraping-python/blob/main/web_scraping_maestria.ipynb)
 9 | * [Segunda Notebook de la clase (Selenium)](https://github.com/GEJ1/web-scraping-python/blob/main/Selenium_2023.ipynb)
10 | 
11 | **Update 2023**: Cree un curso para [freeCodeCamp](https://www.freecodecamp.org/espanol/news/aprende-web-scraping-con-python-y-beautiful-soup-en-espanol-curso-desde-cero/) que incluye la mayorìa del contenido de esta clase y pueden [verlo acá](https://github.com/GEJ1/web_scraping_freecodecamp)
12 | 
13 | # Recursos
14 | 
15 | * [Web general](#Web-general)
16 | * [Web scraping general](#Web-scraping-general)
17 | * [Beautiful Soup](#Beautiful-Soup)
18 | * [Selenium](#Selenium)
19 | * [Scrapy](#Scrapy)
20 | * [Libros](#Libros)
21 | 
22 | ## Web general
23 | Casi todo es de https://developer.mozilla.org/es/ , uno de los mejores lugares para buscar referencia sobre tecnologías web.
24 | 
25 | * [Listado de etiquetas HTML](https://developer.mozilla.org/es/docs/Web/HTML/Element)
26 | * [Material para CSS y listado de propiedades](https://developer.mozilla.org/es/docs/Web/HTML/Element)
27 | * [Material para JavaScript](https://developer.mozilla.org/es/docs/Web/JavaScript)
28 | * [Generalidades del protocolo HTTP](https://developer.mozilla.org/es/docs/Web/HTTP/Overview)
29 | * [HTTP status codes](https://gabicuesta.blogspot.com/2019/01/http-status-codes.html)
30 | * [Introducción al DOM](https://developer.mozilla.org/es/docs/Web/API/Document_Object_Model/Introduction)
31 | 
32 | ## Web scraping general
33 | 
34 | * [Contenido del curso <b>Scraping con Python</b>](https://github.com/institutohumai/cursos-python/tree/master/Scraping)
35 |   * Curso dictado por [Mathias Gatti](https://github.com/mathigatti), [Matías Grinberg](https://github.com/Cerebrock) y [Gustavo Juantorena](https://github.com/gej1) en [humai](https://www.ihum.ai/).
36 |    * Aprovechen [los videos](https://www.youtube.com/playlist?list=PLISuMnTdVU-xOHf3jEtiK1B_g5HFgXCb- ) si quieren ampliar sobre los temas vistos! 
37 | * [Clases 18 y 19 de la materia <b>Laboratorio de datos</b> (FCEyN, UBA. 1er cuatrimestre 2021)](http://materias.df.uba.ar/lda2021c1/171-2/)
38 |   * Videos de las clases, slides y Notebooks para interacción con APIs (clase 18) y web scraping (clase 19).
39 |   * Docentes a cargo: Enzo Tagliazucchi, Sebastián Pinto, Tomás Cicchini y Ariel Berardino.
40 | 
41 | ## Beautiful Soup
42 | 
43 | * [Documentación oficial](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
44 | * [Web Scraping con Python - Curso con Beautiful Soup](https://youtu.be/yKi9-BfbfzQ?si=ubomrRwjz5ziQrKq)
45 | * [Beautiful Soup: Build a Web Scraper With Python](https://realpython.com/beautiful-soup-web-scraper-python/)
46 | 
47 | 
48 | ## Selenium
49 | 
50 | * [Documentación NO oficial, pero útil](https://selenium-python.readthedocs.io/)
51 | * [Selenium FULL COURSE - Learn Selenium by creating a bot in 3 hours [2021]](https://youtu.be/6gxhcvrf2Jk)
52 | * [Web Scraping using Selenium and Python](https://www.scrapingbee.com/blog/selenium-python/)
53 | 
54 | ## Scrapy
55 | 
56 | * [Documentación oficial](https://docs.scrapy.org/en/latest/)
57 | * [Intro To Web Crawlers & Scraping With Scrapy](https://youtu.be/ALizgnSFTwQ)
58 | 
59 | 
60 | ## Libros
61 | 
62 | * [Web Scraping with Python, 2nd Edition](https://www.oreilly.com/library/view/web-scraping-with/9781491985564/)
63 |   * [Código asociado al libro](https://github.com/REMitchell/python-scraping)
64 | * [Hands-On Web Scraping with Python](https://www.amazon.com/Hands-Web-Scraping-Python-operations-ebook/dp/B07VFFYPGK)
65 | 
66 | 
67 | 


--------------------------------------------------------------------------------
/Selenium_2023.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |   "nbformat": 4,
  3 |   "nbformat_minor": 0,
  4 |   "metadata": {
  5 |     "colab": {
  6 |       "provenance": [],
  7 |       "authorship_tag": "ABX9TyO6mh0aVFJlNLXBMgAL8KKs",
  8 |       "include_colab_link": true
  9 |     },
 10 |     "kernelspec": {
 11 |       "name": "python3",
 12 |       "display_name": "Python 3"
 13 |     },
 14 |     "language_info": {
 15 |       "name": "python"
 16 |     }
 17 |   },
 18 |   "cells": [
 19 |     {
 20 |       "cell_type": "markdown",
 21 |       "metadata": {
 22 |         "id": "view-in-github",
 23 |         "colab_type": "text"
 24 |       },
 25 |       "source": [
 26 |         "<a href=\"https://colab.research.google.com/github/GEJ1/web-scraping-python/blob/main/Selenium_2023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
 27 |       ]
 28 |     },
 29 |     {
 30 |       "cell_type": "markdown",
 31 |       "source": [
 32 |         "#**Web scraping avanzado con Selenium**\n",
 33 |         "\n",
 34 |         "Selenium es\n",
 35 |         "* ### Selenium nos va a poder recorrer internet con un navegador sin interfaz gráfica, permitiéndonos hacer click, scroll, etc.\n",
 36 |         "\n",
 37 |         "* ### Usar Selenium dentro de Google Colab no es lo más común pero a fines didácticos resulta útil. No estoy seguro de que tan escalable sea.\n",
 38 |         "\n",
 39 |         "* Docs: https://selenium-python.readthedocs.io/\n",
 40 |         "\n",
 41 |         "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARwAAACxCAMAAAAh3/JWAAABklBMVEX////u7u4AegAAfQAAewAAeAAAfgCUlJTl5eUAAADy8vLe3t7b29vDw8Pi4uIAdgBAQEDAwMC2tratra2hoaHMzMwAcgCJiYlycnIiIiK5ubnV1dV4eHjw+PCFhYWdnZ0vLy+ZvXax3cxbnDBsbGxISEhkZGTk7NNZWVlQUFCpqakssDQ9PT0pKSkAjDZYmUHG2rRmsYS/49cbGxvy///w9eNGmUZxp03a8usAhiJtrW2Vu36Fs2u+0qY5m1w/kimCwZ8QjkGfyZ+55+kSEhImdx601rQAKAAANAAADgCpyI7D38ODtoDV6dXf8+0jhwAnny4rrDMAIAAJWBBpql04lkD//fI5jx0rIisNNxA4qz8QaBZZqWgvYjJLvlIARQBJg0xZw1+Ou45TcFI+UD0gKyFxyXUcfyJPqXh7u4yEw4Z7yn90sWuXvZaRzpOUy61rm2h0kHXk6cCIpobI151yozdXmVJRsFQ0PTNNYkweGB4/hThcmFaez7UPQAlLmTdDjgWIsFS5zo1fs4yU0cM0nmbYRXofAAAS1ElEQVR4nO2ci3/TRrbHndHL8vsZ+aHYSiPJkrFJaHCyzoME3xCKHeNQEptbWNimbCks3W5Zlu52l3Lbvff/vvOSLNsyIYGQbjq/T2mksTyWvjpz5szMkQIBJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiamX5VCxbSiqiVFksrlTCEZzWaLkUgsFgzOzITO+9zOWUGlJu/UazWtXs8v70DJWGhreTlhGLZlWaap53JqKZNJp1LJ7Izn29HYuZ34R1Amv2OXqqlCIZVKp6uZTFmSFKWkqjldNy3LNowE5KYhcvllzG5H1vS0wyclq+d69meqkL5jlJPRZLIA/0OAECKoKlIGqUwkUSmQnG7UjSr+uiQ3b4bP+RLOTCFrxyxANNkI9C8zQahYOBwpFovZbDQaTRJropSoqlXYtMq6ZgYDgZy8f+lJ7rwv4qxUkvUkRBMMhUIz0xSkioVjsXAkUswW0ulUIWPpBeOzO5cuXfq8eN5XcTbKyjZsThEfEm/RzEwsmiokC4Z89xLSE/M0P93eGnz6rsc2Dlbe+dgPJ3M5E02GCZVJLrAzj42L4glHo6p84xLRzYhv5XPdbndu6k/PC/FP3vU0FwW+efvkV/d+Ci5byWRxgsoYj7AjL56gJT+hbC41/UynsdcDgiAetq76//Y8EN4VTn/AifHfnfYiT6uCrEB/M5UKxELIRKBcPJhN2L61f8nVrUmvU+kBXoB0eOHQ/56fAE6gLcS/Of1VnlJluZqMBGOjWMLjijiieCCb7L3//mLI5r6cHa+5MeDAg0632xpMY3ASOIFu9/QXeVpJy4XoOJXIiMKxkQJMJziTkn//8KHL5tG94ETNbcA/JVvrT/1/+0RwzkNSvZD1sRCPYuUiKi8O6UA2ZfkPXz58eIeg2b9pzUzWvMAd50H/A+CksmM0iq4wm7RshZ1SB44q/+vLLx3L2ZdLfjX3uOsTZXO7R0dHu07/5YXThZ+03JbT6SCundbRkVM01+ngzUan43y9Szcr+JOOU3ED1fRhfPcInOK4IIqs/NXjXMwLJxax/rj29dcOnCdy1bfmHnd5rJfqr4tCPB4XxBbZH8KpLHLoE3BIY5k3s98EOlvwUABWiPktzcavoL/r8VkH6EY8jg+fn20GlraEuCBchkzWRfQ1cUpLPjmcCSZeOonHR60/Kh46waz2+NmfMJw72BVH/Wte4PjtkYK5N4LQXNt7PhCEK6NwlkQADp/tLXKAJwWL/PW2wL1aeyVy4DpGvAR4AmdobRs8R+AAsQXEV2srPLge2JjFW6LwIehI9fQYnKwrBEP/+97e0bfLmZhjXcGC/I8/Izg/PPzizqU7j/zbVAB1viJY83qdVQCuoAvt/wyE77xwKjzHkY0BL+JvLMKo5gHaqmyJ5NghHOADhxzc5rgV4Rra2uRF7gO0LARnqooRKf+Xtb2j1rdygdKJZeTvIZs/ff3DD5DN/s17ujSl5v4iEHnugesiNgXu/+jmAsAtzoGzyvP/Sz54AcA2gcPRO78EuGvHwuFoCAS/RjuBDR589yHgVKPT4aTkP6ytQdNp/fXHaAz7m5L81z8TOF/s32nKdqqkTKu6cQDgDQTNp3POmTujo5cAfDqEAw3HoRb4G2lF8Fh64/vUdb0VjlPUdjHBwz9AzDgJJ+oIbqfl+68Ine+1IhySR8z8t5ANhPPD/v6dG3KumFTfMubsPu8BTgRNdA0N2Hs5DvoFB7aHcNrUXPCFCrhdeeBscc13hrMpODVVqMG9n6R6JhmdpmxRkW9QOl8Z4WAx8XfM5tlzyGb/rqyEi6mc/bbq53YPOJETf4dOV2yuUL0RwZUhnA1ePHQ+2SJUhnACW6SlnBecwhiSJBWmY35O6Owd/UPP/vj4CKF59vwNZPOTXA0Xs2ndOOYXOj2M4oUgerU9hLPKeT9oTlrO+cHJlwvJaUJNK/GI0tl7LH+F/jx7drC1v9/8/F4yghqemTjuJyqcCBtUReCvzA0VGML5mRM+Hfvk1wMnhWaO/YTwZJM/3r3xz39hPN/Df2trayvNwf4T2c6iTj9atWrH/saAu3wbne64i6Rw1idHEb8aOFLKD0wqRfBEs1X5PqQD8RA9P2w2B/ehK8a+O5mxtGN/Y4A6nP5kxEzhbHocMtXb4PgEgWcIJ52aKkSnWLp1A9EhfF49edJsPpIVPP2eTBbKtjalZncGEGJBJwqjmTEDoXAa4rAfo5oOB3ZtTgDzMeAoU+CkUXkBLX5an0E6r7BWnkB9JmeKBE0hVbbr/hU3Bk4A3yYBMQznXAYEnBsEAt6N2NyYaAqcpbjTOhs98SPAqaZdpdxFK7cA0olqjyAdhOefEM2NW/dSDppUWjKmwFkVwGEHXW2bp2ErZIBje1gkXvNGyBWOEwmd7kH89dvh9EWRw4cs9cBHgFMawsmgZc6SUkYl1TQpR3RS8t0bCA/835P7sh3NUjQYTt6/4spA4AQY2fQETiSRcX9RgMMJ2NnBcfPhXGBk4MnDIemztRUgxLffDifQjovcytra1uz1dfAR4GTw0ma6WrJMVSqXJdW0dKlKhY0nKkGnTHRX1h00eFV0KpzA3DqaXhcAv+LMyfTXRRBHMxPNI9y+5t3pBzplIfArHbwPx+8OnAGJmZ0pCzQnARHG4Ri2MkunLOKzrymc2bgDJy58EDhqGVOQLDWKFvbQ2l6kaprlaibj4ClEc7cIm59kBbJx0KSrb4GDp7b29lrdiZIOdT1LR0fu0LnbgnHmrrPb2d29OrrZ2N11Kqq0aK27u7iVVuhf70Fzu7ud9+KCheCgBV7FTs+4K56hUDBjK+7SL2paxk+Izc0fy1kHDVlKVxLT4fzHC8KRIALJKIwuBoeSkE65XKZ4CikYC96/VU9HPWgQuwsOJ4fSlaxMiJjMTHCGrJmHUjbJrUB4oPFI8iPZSBVSo3kF5QsOR4dwFIuYSypnWWYpSugoOko6cawnpcqWYzOevJSLDWfZVKSyWUY8QhlbVRQlZ0sYTsSm+TiYT7WqDFN2yq4uPhzJJsZilQgNuxqaCRdKhioN8ZQzXpNxyqULDgcSUYziCJySVc3Zto7SJxW0DyMfyzTNnDKa4YWzvC46HEUxIrhZKbaCk9oky86V0AaSpBpmJhsJR7IZ3VJGyFx8OMjPUMsJlhJWrqSMyrQLMygyRP8KMHYmWYG/FTimotg0ygkVMznbwIBKWJCNGcZWhbPiQrGcNYqudLHhGLZaMtWQExvPxLJl3YDNiki3EJtQMCVJSfR5UDfPCM762trJU7daa2eZigDhGGZJTWSHATIEVCxbEBmUmkiiD8KmYVmGitLZI4bqZTMNDp0MHu5O/4xqAQjji5Qj88p+asd5cIapggiOkYPdUdg7fICDKymRK7kmVbIRKRtFQ6GyVfJqCpxNkczZ0IsQxeE1VAaiX3LKAsePlVbEJtXhnv8wcoMXhdcnv+h3lYSz91XVgqHOKJ50QlVVAxlOqGgQM7Kp6bwDnAovDu9pf4sDV9yP4O32W430gQNEUcACAKz4ZXYt8XHxDNMoIZyEkYB0zJqaHUlFDpUNNYdwwFFFLQelqjXU+EK6XnLwqFPhBN5ww9XqCieKTdeOVjngd7d94TRbSEcHIvD/UmV3erLq+wvBgbJzas5KmNWRXG0Y9+kYU9qEUaBl2wnUqYUkq6RioWnDqXDWPXNxbWgCrh31e6Lv3faF4yxYNFaBCM7S9/qKwkkYJsRjGgmrlI5QPKG0YZVcUjjVNEiK1aEgnGQ2G4lNPHcE25XrWRb4pgi26c7SlBnMt8LBiSYfPRFZWq7VagSPZeq6btoJQyVdF/QutkJnMhyhnegYnAyeK8yOpwUO3FtdAcLrVd5Zelh3l6nmOrstd37PgdPoughG4PQXSPYBPaDTwtN/XZxk2vWkmjac7blOy1N7t3sVzRS2TjQ/iOEgafV8Dfpmw7YNTUsSg7FrGE6oiB9OK0bCJMl2FE4dDsHKmXQhOpbEvsE72aRtoXm1LcTpNPsWR7vs9gDNG/NO2j6Gs7kicuLhax84aLEdm87q7OVAY1EQZiGqOSHehEUDIDpRQEUkk8f9dVo7wdOIx78LLEH3Hj+Jj8Jw6sv5mlWyNGJBhqGZxGBsrUSmMvKaptWwiSF6iRzxzxQObJCQTzUVHU23XQIizZhcAA8CFWdhoCKK2IYabwQgrhxAv00XrRY4rnswG4ddEyd84wOn/z/EFFdB8+oW7MHQrPqcKF7G9GnleBMvFG7h2mEvSfx4A5rdEs4YPymcupWGjjiQXK5RODWb+BmjRhxytWZ6pY/CMWzLhOPUdHI0iR06XmIiDQ7FIgv8dXr2/L/R30UAruDEtgXAf0rgiL34lU63s8GJwvYknMDPPEawyjefzz7Y7bRuu3A8a6YLOJ2nv8WDf6OSyoDHizwNjnuwJT7t7O6egA2Goxl4bj1UcujUVdKaNMOI4a5cw48xmja0HYRCH8JRE3WtZtimKlULY0nsB7QzbwvI6NsCWYqCHTlisemmXlV60LAIHJIZGPiF+t4xOL8AnAG2yg2zISkcWEazxioAB1TOHcC1f4PhiN449J3hwPaynCY+RtFQ+9HyFpnByNRt7H1CkRqBU5OSValkGV7TSdTz9RocqyrVwtijnpvkYmA7uEZOezuAF87xrX3D8Y6bmAfcbQzHyQyEm7gxjMF5Sbq5VU4cLitTOEuOk28D7Np6wwTxn3HtEA494oRwNK1uBEN0UK6aplrFe6GgnbBrOVwuacRwIqjHMi1d98JZzmsJKydVk2PdeQPeLNRFiNiA+gMeXdoLHltMhRvmAb4E+Mki6HP+i5b8QpY3p8EZhoMOnD7NqYPM0V9vustLARkVhHOKQADD0ZZVZ9DgdNiISN227XoK70laLVGroe1QUtNH4ezkNQM2q9TEoyGLuAVtxomNzOMuZZ14yE1BvO4ktRxgn+SFQzFMNisKx20fDpzABlkarhBvtQnEQ6d2eBKfYDiT+fTHKr2D4Gg7anBs3Qo2KtgxwaiHNLFiVUpH8OSFZVE4ORdOzdZLmcLEsyEkE5K0KjQOQskWbzgcHqMk5VmqOO54vHBeAHzNY3DmiT35woGN9in+Rdynwz/e2k8Np0jgaDuWd+AJR+VKHXfbdo06IDrbNaPWdN1rOjAOqCcsVUpPGA7Kob0eaPRI10PaFew1MKqXAr89lup2LJy/EQvzhYM6Q3joG3IjUBLPaO2ngxOo1QmdfF1PhWm7milmbMSG0DEKbksLhdWa6cLBdGDwqNmmn+GgeI+7vQQEei0bQLy9SXOPloSx3P8xOD7NCgZIuL/zh9NGAUGF4AtsgvEU7VPCkZYpHS2/XDPx8oNqafmaYVA4dkLLFSLIaIKQWcIkcBzT0et1Yjh+zwfP88LrDeD0LUs8+GSDJ7FshZsYYB3nkOfpmMwfTgP1RusC8bqT+YenhDOzAy+P4oFWAAV3cSjswEF4Epae02GYgzqtETh2vQ49jjLRVREcAIZetFWR9DfnOSNoVONDcw8cGO5NdOUvRDpW84cDY8PLgS06a0TjhfeHE1CXPXToQItGyg4cNF2BX2hBI+Rhs8ppddRVlVO+7yRAsxMi714KDH3daAOG+e701xyFw1M4sAGJ40Hgix4HiDVMgbMp8E/deZH14XiC1H5KOMGolZ+gMwYHvefDshw2psdyYHxsoEY15bEilH89vPmw/xYdB9QfcIDGua3eNoHDkYlVOP4mzxxhOMijNuCQggN02nUKHHQj3EEENB3q0/ptnA9+WjjJqpfOEM6o6XjpDOEYeczGv1ERHPxwfrTR86CqiDxoPm219gYgjq9pAU0Xbnc67QGH8pYDeEpI7CGJAPDgAf3mFDjI34NtZ6fS4/nmUav1vCfEr50eTjGZLpv1vEPHhWMcazmqXkMd1VvYBBrCyHP1q/FZD6pFlL4GJT7Fl70gXO/2ZgXPo9aVWYEKiA/ciZjV2dkhHDJlQY6OC54nzyuwdvJYIAbWiMddiidQMJqqSqqtedvWkJELaMJwdNOu52tkOD799UO7Le8T95XWyMOX3aODV6/2nJy1+YPvAo3nh73mivOVuRbVbsfjvJdareFeq/Xasz0y5O7s4dqJQ+t7DzyBislUtSyVYF+EX5AzoXq97jUrDMw2Elo+r1lqScqkoh/01UzTF6nORZFoMpWuVskrhdAyMOyhc+i9QujFQjZ5tRB5txBmt4yVrxu6IpUnprgunELBIkmfLTjJ66OZ2ulhyhLJIiiVFAnnwiWzFxwNERwheF7UUJx8bs/zIJbzMFY4+Ft/19mI3LelnPeJMDExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTH9ZvX/evinM53UJHMAAAAASUVORK5CYII=\">\n",
 42 |         "\n",
 43 |         "## Tenemos que instalarlo y configurarlo en Colab (en local es más fácil)"
 44 |       ],
 45 |       "metadata": {
 46 |         "id": "XPDa5GqBHOUe"
 47 |       }
 48 |     },
 49 |     {
 50 |       "cell_type": "code",
 51 |       "execution_count": null,
 52 |       "metadata": {
 53 |         "id": "fFqff2VjofiQ"
 54 |       },
 55 |       "outputs": [],
 56 |       "source": [
 57 |         "%%shell\n",
 58 |         "# Fuente: https://github.com/googlecolab/colabtools/issues/3347\n",
 59 |         "sudo apt -y update\n",
 60 |         "sudo apt install -y wget curl unzip\n",
 61 |         "wget http://archive.ubuntu.com/ubuntu/pool/main/libu/libu2f-host/libu2f-udev_1.1.4-1_all.deb\n",
 62 |         "dpkg -i libu2f-udev_1.1.4-1_all.deb\n",
 63 |         "wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb\n",
 64 |         "dpkg -i google-chrome-stable_current_amd64.deb\n",
 65 |         "CHROME_DRIVER_VERSION=`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`\n",
 66 |         "wget -N https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chromedriver-linux64.zip\n",
 67 |         "unzip -o /tmp/chromedriver-linux64.zip -d /tmp/\n",
 68 |         "chmod +x /tmp/chromedriver-linux64/chromedriver\n",
 69 |         "mv /tmp/chromedriver-linux64/chromedriver /usr/local/bin/chromedriver"
 70 |       ]
 71 |     },
 72 |     {
 73 |       "cell_type": "code",
 74 |       "source": [
 75 |         "!pip install selenium -q"
 76 |       ],
 77 |       "metadata": {
 78 |         "id": "rHwnvqpypFu9"
 79 |       },
 80 |       "execution_count": null,
 81 |       "outputs": []
 82 |     },
 83 |     {
 84 |       "cell_type": "code",
 85 |       "source": [
 86 |         "from selenium import webdriver\n",
 87 |         "from selenium.webdriver.chrome.options import Options\n",
 88 |         "from selenium.webdriver.common.by import By\n",
 89 |         "from selenium.webdriver.chrome.service import Service"
 90 |       ],
 91 |       "metadata": {
 92 |         "id": "1_7WNZZF3KWM"
 93 |       },
 94 |       "execution_count": null,
 95 |       "outputs": []
 96 |     },
 97 |     {
 98 |       "cell_type": "code",
 99 |       "source": [
100 |         "options = webdriver.ChromeOptions()\n",
101 |         "options.add_argument(\"--headless\")\n",
102 |         "options.add_argument('--disable-dev-shm-usage')\n",
103 |         "options.add_argument(\"--no-sandbox\")\n",
104 |         "\n",
105 |         "# instanciamos el driver\n",
106 |         "wd = webdriver.Chrome(options=options)\n",
107 |         "\n",
108 |         "# Tiempo  de espera ( si encuentra antes, no espera)\n",
109 |         "wd.implicitly_wait(20)\n",
110 |         "\n",
111 |         "# Hacemos el pedido a la URL\n",
112 |         "url = \"https://www.wikipedia.com\"\n",
113 |         "# Pedido HTTP\n",
114 |         "wd.get(url)\n",
115 |         "h1 = wd.find_element(By.CSS_SELECTOR, \"h1\")\n",
116 |         "print(f'h1 extraido de wikipedia: \\n\\n{h1.text}')"
117 |       ],
118 |       "metadata": {
119 |         "id": "_YvExg2YomO2"
120 |       },
121 |       "execution_count": null,
122 |       "outputs": []
123 |     },
124 |     {
125 |       "cell_type": "code",
126 |       "source": [
127 |         "search_input = wd.find_element(By.ID,'searchInput')\n",
128 |         "\n",
129 |         "# Envio el texto que quiero que ponga en el formulario\n",
130 |         "search_input.send_keys('Natural language processing')\n",
131 |         "wd.save_screenshot(\"1.png\")\n",
132 |         "\n",
133 |         "search_button = wd.find_element(By.XPATH,'//*[@id=\"search-form\"]/fieldset/button')\n",
134 |         "search_button.click()\n",
135 |         "wd.save_screenshot(\"2.png\")"
136 |       ],
137 |       "metadata": {
138 |         "id": "TFURhZIE-1B_"
139 |       },
140 |       "execution_count": null,
141 |       "outputs": []
142 |     },
143 |     {
144 |       "cell_type": "code",
145 |       "source": [
146 |         "# Imprimo el título de la página a la que se accedió\n",
147 |         "heading = wd.find_element(By.ID,\"firstHeading\")\n",
148 |         "body_content = wd.find_element(By.ID, \"bodyContent\")\n",
149 |         "print(f'Heading: \\n{heading.text}')\n",
150 |         "print(f'Content: \\n{body_content.text}')"
151 |       ],
152 |       "metadata": {
153 |         "id": "_XTVLqhg_G2p"
154 |       },
155 |       "execution_count": null,
156 |       "outputs": []
157 |     },
158 |     {
159 |       "cell_type": "markdown",
160 |       "source": [
161 |         "## **Caso de uso Nº 1: Scroll infinito**\n",
162 |         "\n",
163 |         "Existen páginas que no muestran todo el contenido a menos que vayamos hasta abajo (*scroll*). Esta acción dispara un evento de javascript que renderiza más HTML y por lo tanto vemos contenido nuevo.\n",
164 |         "\n",
165 |         "Podemos emular la acción de mediante Selenium."
166 |       ],
167 |       "metadata": {
168 |         "id": "YctCEElGxjyd"
169 |       }
170 |     },
171 |     {
172 |       "cell_type": "code",
173 |       "source": [
174 |         "# Configuramos el web driver\n",
175 |         "driver = webdriver.Chrome(options=options)\n",
176 |         "\n",
177 |         "# Hacemos el pedido a la URL\n",
178 |         "url = \"https://infinite-scroll.com/demo/full-page/\"\n",
179 |         "driver.get(url)\n",
180 |         "\n",
181 |         "# Busco todos los h2 (notar la sutileza del metodo elements en plural)\n",
182 |         "h2_list = driver.find_elements(By.CSS_SELECTOR, \"h2\")\n",
183 |         "for h2 in h2_list:\n",
184 |         "  print(h2.text)"
185 |       ],
186 |       "metadata": {
187 |         "id": "eaf8YGzPxlZh"
188 |       },
189 |       "execution_count": null,
190 |       "outputs": []
191 |     },
192 |     {
193 |       "cell_type": "code",
194 |       "source": [
195 |         "# Tomo un screenshot\n",
196 |         "driver.save_screenshot(f'infinite_page.screenshot.png')\n",
197 |         "\n",
198 |         "# Hago lo mismo que antes pero iterando 5 veces y pidiendole que scrollee hasta el final cada vez y saque un screenshot\n",
199 |         "for i in range(5):\n",
200 |         "  print(f'Iteracion numero {i}\\n\\n')\n",
201 |         "  # el metodo execute_script me permite ejecutar codigo de javascript, en este caso para ir al final de la pagina\n",
202 |         "  driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
203 |         "  driver.save_screenshot(f'infinite_page_{i}.screenshot.png')\n",
204 |         "  h2_list = driver.find_elements(By.CSS_SELECTOR, \"h2\")\n",
205 |         "  for h2 in h2_list:\n",
206 |         "    print(h2.text)\n",
207 |         "  print('\\n\\n')"
208 |       ],
209 |       "metadata": {
210 |         "id": "NQo_b719xl0b"
211 |       },
212 |       "execution_count": null,
213 |       "outputs": []
214 |     },
215 |     {
216 |       "cell_type": "markdown",
217 |       "source": [
218 |         "## **Caso de uso Nº 2: Páginas que usan JavaScript para mostrar el contenido de manera asíncrona**\n",
219 |         "\n",
220 |         "* Hay páginas que cuando hacemos un request a su URL no nos devuelve lo que esperamos. Sino bastante código de JavaScript (entre etiquetas `<script>`)\n",
221 |         "\n",
222 |         "* En esos casos hay que esperar un tiempo y/o interactuar con la pagina.\n",
223 |         "\n",
224 |         "Ejemplo: http://www.parrocchiemap.it/parrocchiemap/ricerca_pm.jsp?diocesi=Milano&idzonapastoraler=296\n"
225 |       ],
226 |       "metadata": {
227 |         "id": "5bxB-4hW1qWx"
228 |       }
229 |     },
230 |     {
231 |       "cell_type": "code",
232 |       "source": [
233 |         "from selenium.webdriver.support.ui import WebDriverWait\n",
234 |         "from selenium.webdriver.support.ui import Select\n",
235 |         "\n",
236 |         "idzonapastorale_value = []\n",
237 |         "idzonapastorale_text = []\n",
238 |         "\n",
239 |         "driver = webdriver.Chrome(options=options)\n",
240 |         "\n",
241 |         "url = \"http://www.parrocchiemap.it/parrocchiemap/ricerca_pm.jsp?diocesi=Milano&idzonapastoraler=296\"\n",
242 |         "\n",
243 |         "# Hacemos el pedido a la URL\n",
244 |         "driver.get(url)\n",
245 |         "\n",
246 |         "# Ejecuto codigo de javacript para ir al final de la pagina\n",
247 |         "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
248 |         "\n",
249 |         "# # Espero a que cargue la pagina\n",
250 |         "WebDriverWait(driver, timeout=5)\n",
251 |         "\n",
252 |         "\n",
253 |         "# El metodo Select nos permite acceder a las opciones en un menu desplegable (drop-down)\n",
254 |         "select = Select(driver.find_element(By.ID, 'idzonapastorale'))\n",
255 |         "for item in select.options:\n",
256 |         "    print(item.get_attribute('innerText'), item.get_attribute('value'))\n",
257 |         "    idzonapastorale_value.append(item.get_attribute('value'))\n",
258 |         "    idzonapastorale_text.append(item.get_attribute('innerText'))"
259 |       ],
260 |       "metadata": {
261 |         "id": "fzF3Ljwh1vVP"
262 |       },
263 |       "execution_count": null,
264 |       "outputs": []
265 |     },
266 |     {
267 |       "cell_type": "code",
268 |       "source": [
269 |         "pagesource_list = []\n",
270 |         "\n",
271 |         "diocesi = 'Milano'\n",
272 |         "for zona in idzonapastorale_value:\n",
273 |         "  print(f'Mirando la zona: {zona}' )\n",
274 |         "  # Configuramos el web driver\n",
275 |         "  driver = webdriver.Chrome(options=options)\n",
276 |         "  idzonapastorale = zona\n",
277 |         "  url = f\"http://www.parrocchiemap.it/parrocchiemap/ricerca_pm.jsp?diocesi={diocesi}&idzonapastoraler={idzonapastorale}\"\n",
278 |         "\n",
279 |         "  # Hacemos el pedido a la URL\n",
280 |         "  driver.get(url)\n",
281 |         "\n",
282 |         "  driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
283 |         "\n",
284 |         "  # # Espero a que cargue la pagina a la vieja usanza\n",
285 |         "  # time.sleep(10)\n",
286 |         "  WebDriverWait(driver, timeout=5)\n",
287 |         "\n",
288 |         "  # Cambio el driver para que este en el iframe\n",
289 |         "  driver.switch_to.frame(driver.find_element(By.TAG_NAME, \"iframe\"))\n",
290 |         "\n",
291 |         "  # Una manera de quedarse con el HTML\n",
292 |         "  html = driver.execute_script(\"return document.getElementsByTagName('html')[0].innerHTML\")\n",
293 |         "\n",
294 |         "  # Otra manera usando el atributo del web driver\n",
295 |         "  pageSource = driver.page_source\n",
296 |         "\n",
297 |         "  pagesource_list.append(pageSource)\n",
298 |         "\n",
299 |         "  driver.save_screenshot(f'screenshot_{zona}.png')"
300 |       ],
301 |       "metadata": {
302 |         "id": "awgw4zLZ2FDi"
303 |       },
304 |       "execution_count": null,
305 |       "outputs": []
306 |     },
307 |     {
308 |       "cell_type": "code",
309 |       "source": [
310 |         "# Proceso los mails\n",
311 |         "import re\n",
312 |         "\n",
313 |         "def procesar_mails(pagesource_list):\n",
314 |         "  mails_total = []\n",
315 |         "\n",
316 |         "  for zona in pagesource_list:\n",
317 |         "    mails = []\n",
318 |         "    pos = []\n",
319 |         "    # obteniendo los mails\n",
320 |         "    match = re.findall(r'(mailto:\\S+@\\S+)', zona)\n",
321 |         "\n",
322 |         "    for num, i in enumerate(match):\n",
323 |         "      pos.append(num)\n",
324 |         "      lista = i.split('+')\n",
325 |         "      mail = ''.join(lista[0][7:].replace(\"'\", \"\")+lista[1].replace(\"'\", \"\")+lista[2].replace(\"'\", \"\")).split(',')[0].split(';')[0]\n",
326 |         "      mails.append(mail)\n",
327 |         "\n",
328 |         "    mails_total.append(mails)\n",
329 |         "\n",
330 |         "  return mails_total\n",
331 |         "\n",
332 |         "mails_total = procesar_mails(pagesource_list)"
333 |       ],
334 |       "metadata": {
335 |         "id": "rRlVfbWZ2Ou7"
336 |       },
337 |       "execution_count": null,
338 |       "outputs": []
339 |     },
340 |     {
341 |       "cell_type": "code",
342 |       "source": [
343 |         "# Armo un Pandas Dataframe en base al diccionario\n",
344 |         "import pandas as pd\n",
345 |         "\n",
346 |         "diccionario_mails = dict()\n",
347 |         "df_list = []\n",
348 |         "for num, zona in enumerate(idzonapastorale_text[1:]):\n",
349 |         "   diccionario_mails[f'df{num+1}']= pd.DataFrame(mails_total[num+1], columns=[f'{zona}'])\n",
350 |         "   df_list.append(diccionario_mails[f'df{num+1}'])\n",
351 |         "\n",
352 |         "df_final = pd.concat(df_list, axis=1)\n",
353 |         "\n",
354 |         "\n",
355 |         "# Pasamos el dataframe a formato csv\n",
356 |         "df_final.to_csv('output.csv') # Si pasan como argumento un string con el nombre del archivo lo guarda. Ej: 'datos.csv'"
357 |       ],
358 |       "metadata": {
359 |         "id": "3G9CCs5v2Tzy"
360 |       },
361 |       "execution_count": null,
362 |       "outputs": []
363 |     },
364 |     {
365 |       "cell_type": "code",
366 |       "source": [
367 |         "driver.close()"
368 |       ],
369 |       "metadata": {
370 |         "id": "DytPeBoN2QYn"
371 |       },
372 |       "execution_count": null,
373 |       "outputs": []
374 |     },
375 |     {
376 |       "cell_type": "markdown",
377 |       "source": [
378 |         "### Las posibilidades son enormes!\n",
379 |         "\n",
380 |         "#### Básicamente podemos recorrer cualquier pagina web clickeando, scrolleando , completando formularios o lo que sea usando Python y con posibilidad de automatizarlo completamente.\n",
381 |         "\n",
382 |         "Algunos ejemplos más:\n",
383 |         "\n",
384 |         "```python\n",
385 |         "# Para ir al final de un documento: https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python\n",
386 |         "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
387 |         "```\n",
388 |         "Algunas formas de seleccionar elementos <br>\n",
389 |         "`<input type=\"text\" name=\"passwd\" id=\"passwd-id\" />`\n",
390 |         "\n",
391 |         "```python\n",
392 |         "# Diferentes formas de seleccionar un elemento\n",
393 |         "find_element(By.ID, \"passwd-id\")\n",
394 |         "find_element(By.NAME, \"passwd\")\n",
395 |         "find_element(By.XPATH, \"//input[@id='passwd-id']\")\n",
396 |         "find_element(By.CSS_SELECTOR, \"input#passwd-id\")\n",
397 |         "\n",
398 |         "# Y hay más!!\n",
399 |         "find_element(By.LINK_TEXT, \"link text\")\n",
400 |         "find_element(By.PARTIAL_LINK_TEXT, \"partial link text\")\n",
401 |         "find_element(By.TAG_NAME, \"tag name\")\n",
402 |         "find_element(By.CLASS_NAME, \"class name\")\n",
403 |         "```"
404 |       ],
405 |       "metadata": {
406 |         "id": "3y4X19PXxu2P"
407 |       }
408 |     },
409 |     {
410 |       "cell_type": "markdown",
411 |       "source": [
412 |         "# **Ideas? Sugerencias? Algo para charlar?**\n",
413 |         "\n",
414 |         "<img src='https://chandigarhx.com/wp-content/uploads/2017/04/coffee-ideas.jpg'>"
415 |       ],
416 |       "metadata": {
417 |         "id": "qQYvx32yHpWU"
418 |       }
419 |     }
420 |   ]
421 | }


--------------------------------------------------------------------------------