"
67 | ]
68 | },
69 | {
70 | "cell_type": "code",
71 | "execution_count": 2,
72 | "metadata": {
73 | "ExecuteTime": {
74 | "end_time": "2021-04-21T21:10:53.113870Z",
75 | "start_time": "2021-04-21T21:10:52.703140Z"
76 | }
77 | },
78 | "outputs": [
79 | {
80 | "data": {
81 | "text/plain": [
82 | "
"
138 | ]
139 | },
140 | {
141 | "cell_type": "markdown",
142 | "metadata": {},
143 | "source": [
144 | "## Step 3\n",
145 | "> **We collect the html from the website by adding `.text` to the end of the response variable.** "
146 | ]
147 | },
148 | {
149 | "cell_type": "code",
150 | "execution_count": 4,
151 | "metadata": {
152 | "ExecuteTime": {
153 | "end_time": "2021-04-21T21:10:53.371878Z",
154 | "start_time": "2021-04-21T21:10:53.368038Z"
155 | }
156 | },
157 | "outputs": [
158 | {
159 | "data": {
160 | "text/plain": [
161 | "'\\n\\n\\n\\t **We use `BeautifulSoup` to turn the html into something we can manipulate.**"
180 | ]
181 | },
182 | {
183 | "cell_type": "code",
184 | "execution_count": 5,
185 | "metadata": {
186 | "ExecuteTime": {
187 | "end_time": "2021-04-21T21:10:53.408867Z",
188 | "start_time": "2021-04-21T21:10:53.373516Z"
189 | }
190 | },
191 | "outputs": [
192 | {
193 | "data": {
194 | "text/plain": [
195 | "\n",
196 | "\n",
197 | "\n",
198 | "\n",
199 | "\n", 213 | "Login\n", 214 | "
\n", 215 | "
"
425 | ]
426 | },
427 | {
428 | "cell_type": "markdown",
429 | "metadata": {},
430 | "source": [
431 | "**Fortunately for us,** we do not have to read through every line of the html in order to web scrape. \n",
432 | "\n",
433 | "Modern day web browsers come equipped with tools that allow us to easily sift through this soupy text.\n",
434 | "\n",
435 | "\n",
436 | "## Step 4\n",
437 | ">**We open up the page we're trying to scrape in a new tab.** Click Here! 👀"
438 | ]
439 | },
440 | {
441 | "cell_type": "markdown",
442 | "metadata": {},
443 | "source": [
444 | "## Step 5\n",
445 | " > **We create a list of every `div` that has a `class` of \"quote\"**\n",
446 | "\n",
447 | "**In this instance,** every item we want to collect is divided into identically labeled containers.\n",
448 | "- A div with a class of 'quote'.\n",
449 | "\n",
450 | "Not all HTML is as well organized as this page, but HTML is basically just a bunch of different organizational containers that we use to divide up text and other forms of media. Figuring out the organizational structure of a website is the entire process for web scraping. "
451 | ]
452 | },
453 | {
454 | "cell_type": "code",
455 | "execution_count": 6,
456 | "metadata": {
457 | "ExecuteTime": {
458 | "end_time": "2021-04-21T21:10:53.415596Z",
459 | "start_time": "2021-04-21T21:10:53.410245Z"
460 | }
461 | },
462 | "outputs": [
463 | {
464 | "data": {
465 | "text/plain": [
466 | "10"
467 | ]
468 | },
469 | "execution_count": 6,
470 | "metadata": {},
471 | "output_type": "execute_result"
472 | }
473 | ],
474 | "source": [
475 | "quote_divs = soup.find_all('div', {'class': 'quote'})\n",
476 | "len(quote_divs)"
477 | ]
478 | },
479 | {
480 | "cell_type": "markdown",
481 | "metadata": {},
482 | "source": [
483 | "## Step 6\n",
484 | "> **To figure out how to grab all the datapoints from a quote, we isolate a single quote, and work through the code for a single `div`.**"
485 | ]
486 | },
487 | {
488 | "cell_type": "code",
489 | "execution_count": 7,
490 | "metadata": {
491 | "ExecuteTime": {
492 | "end_time": "2021-04-21T21:10:53.419751Z",
493 | "start_time": "2021-04-21T21:10:53.417109Z"
494 | }
495 | },
496 | "outputs": [],
497 | "source": [
498 | "first_quote = quote_divs[0]"
499 | ]
500 | },
501 | {
502 | "cell_type": "markdown",
503 | "metadata": {},
504 | "source": [
505 | "### First we grab the text of the quote"
506 | ]
507 | },
508 | {
509 | "cell_type": "code",
510 | "execution_count": 8,
511 | "metadata": {
512 | "ExecuteTime": {
513 | "end_time": "2021-04-21T21:10:53.426181Z",
514 | "start_time": "2021-04-21T21:10:53.422305Z"
515 | }
516 | },
517 | "outputs": [
518 | {
519 | "data": {
520 | "text/plain": [
521 | "'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'"
522 | ]
523 | },
524 | "execution_count": 8,
525 | "metadata": {},
526 | "output_type": "execute_result"
527 | }
528 | ],
529 | "source": [
530 | "text = first_quote.find('span', {'class':'text'})\n",
531 | "text = text.text\n",
532 | "text"
533 | ]
534 | },
535 | {
536 | "cell_type": "markdown",
537 | "metadata": {},
538 | "source": [
539 | "### Next we grab the author"
540 | ]
541 | },
542 | {
543 | "cell_type": "code",
544 | "execution_count": 9,
545 | "metadata": {
546 | "ExecuteTime": {
547 | "end_time": "2021-04-21T21:10:53.431920Z",
548 | "start_time": "2021-04-21T21:10:53.428068Z"
549 | }
550 | },
551 | "outputs": [
552 | {
553 | "data": {
554 | "text/plain": [
555 | "'Albert Einstein'"
556 | ]
557 | },
558 | "execution_count": 9,
559 | "metadata": {},
560 | "output_type": "execute_result"
561 | }
562 | ],
563 | "source": [
564 | "author = first_quote.find('small', {'class': 'author'})\n",
565 | "author_name = author.text\n",
566 | "author_name"
567 | ]
568 | },
569 | {
570 | "cell_type": "markdown",
571 | "metadata": {},
572 | "source": [
573 | "### Let's also grab the link that points to the author's bio!"
574 | ]
575 | },
576 | {
577 | "cell_type": "code",
578 | "execution_count": 10,
579 | "metadata": {
580 | "ExecuteTime": {
581 | "end_time": "2021-04-21T21:10:53.437215Z",
582 | "start_time": "2021-04-21T21:10:53.433319Z"
583 | }
584 | },
585 | "outputs": [
586 | {
587 | "data": {
588 | "text/plain": [
589 | "'http://quotes.toscrape.com/author/Albert-Einstein'"
590 | ]
591 | },
592 | "execution_count": 10,
593 | "metadata": {},
594 | "output_type": "execute_result"
595 | }
596 | ],
597 | "source": [
598 | "author_link = author.findNextSibling().attrs.get('href')\n",
599 | "author_link = url + author_link\n",
600 | "author_link"
601 | ]
602 | },
603 | {
604 | "cell_type": "markdown",
605 | "metadata": {},
606 | "source": [
607 | "### And finally, let's grab all of the tags for the quote"
608 | ]
609 | },
610 | {
611 | "cell_type": "code",
612 | "execution_count": 11,
613 | "metadata": {
614 | "ExecuteTime": {
615 | "end_time": "2021-04-21T21:10:53.443153Z",
616 | "start_time": "2021-04-21T21:10:53.438609Z"
617 | }
618 | },
619 | "outputs": [
620 | {
621 | "data": {
622 | "text/plain": [
623 | "['change', 'deep-thoughts', 'thinking', 'world']"
624 | ]
625 | },
626 | "execution_count": 11,
627 | "metadata": {},
628 | "output_type": "execute_result"
629 | }
630 | ],
631 | "source": [
632 | "tag_container = first_quote.find('div', {'class': 'tags'})\n",
633 | "tag_links = tag_container.find_all('a')\n",
634 | "\n",
635 | "tags = []\n",
636 | "for tag in tag_links:\n",
637 | " tags.append(tag.text)\n",
638 | " \n",
639 | "tags"
640 | ]
641 | },
642 | {
643 | "cell_type": "markdown",
644 | "metadata": {},
645 | "source": [
646 | "### Our data:"
647 | ]
648 | },
649 | {
650 | "cell_type": "code",
651 | "execution_count": 12,
652 | "metadata": {
653 | "ExecuteTime": {
654 | "end_time": "2021-04-21T21:10:53.448574Z",
655 | "start_time": "2021-04-21T21:10:53.444499Z"
656 | }
657 | },
658 | "outputs": [
659 | {
660 | "name": "stdout",
661 | "output_type": "stream",
662 | "text": [
663 | "text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\n",
664 | "author name: Albert Einstein\n",
665 | "author link: http://quotes.toscrape.com/author/Albert-Einstein\n",
666 | "tags: ['change', 'deep-thoughts', 'thinking', 'world']\n"
667 | ]
668 | }
669 | ],
670 | "source": [
671 | "print('text:', text)\n",
672 | "print('author name: ', author_name)\n",
673 | "print('author link: ', author_link)\n",
674 | "print('tags: ', tags)"
675 | ]
676 | },
677 | {
678 | "cell_type": "markdown",
679 | "metadata": {},
680 | "source": [
681 | "# Step 7\n",
682 | "> We create a function to make out code reusable.\n",
683 | "\n",
684 | "Now that we know how to collect this data from a quote div, we can compile the code into a [function](https://www.geeksforgeeks.org/functions-in-python/) called `quote_data`. This allows us to grab a quote div, feed it into the function like so...\n",
685 | "> `quote_data(quote_div)`\n",
686 | "\n",
687 | "...and receive all of the data from that div."
688 | ]
689 | },
690 | {
691 | "cell_type": "code",
692 | "execution_count": 13,
693 | "metadata": {
694 | "ExecuteTime": {
695 | "end_time": "2021-04-21T21:10:53.455650Z",
696 | "start_time": "2021-04-21T21:10:53.450078Z"
697 | }
698 | },
699 | "outputs": [],
700 | "source": [
701 | "def quote_data(quote_div):\n",
702 | " # Collect the quote\n",
703 | " text = quote_div.find('span', {'class':'text'})\n",
704 | " text = text.text\n",
705 | " \n",
706 | " # Collect the author name\n",
707 | " author = quote_div.find('small', {'class': 'author'})\n",
708 | " author_name = author.text\n",
709 | " \n",
710 | " # Collect author link\n",
711 | " author_link = author.findNextSibling().attrs.get('href')\n",
712 | " author_link = url + author_link\n",
713 | " \n",
714 | " # Collect tags\n",
715 | " tag_container = quote_div.find('div', {'class': 'tags'})\n",
716 | "\n",
717 | " tag_links = tag_container.find_all('a')\n",
718 | "\n",
719 | " tags = []\n",
720 | " for tag in tag_links:\n",
721 | " tags.append(tag.text)\n",
722 | " \n",
723 | " # Return data as a dictionary\n",
724 | " return {'author': author_name,\n",
725 | " 'text': text,\n",
726 | " 'author_link': author_link,\n",
727 | " 'tags': tags}"
728 | ]
729 | },
730 | {
731 | "cell_type": "markdown",
732 | "metadata": {},
733 | "source": [
734 | "### Let's test our fuction."
735 | ]
736 | },
737 | {
738 | "cell_type": "code",
739 | "execution_count": 14,
740 | "metadata": {
741 | "ExecuteTime": {
742 | "end_time": "2021-04-21T21:10:53.460722Z",
743 | "start_time": "2021-04-21T21:10:53.457107Z"
744 | }
745 | },
746 | "outputs": [
747 | {
748 | "data": {
749 | "text/plain": [
750 | "{'author': 'Thomas A. Edison',\n",
751 | " 'text': \"“I have not failed. I've just found 10,000 ways that won't work.”\",\n",
752 | " 'author_link': 'http://quotes.toscrape.com/author/Thomas-A-Edison',\n",
753 | " 'tags': ['edison', 'failure', 'inspirational', 'paraphrased']}"
754 | ]
755 | },
756 | "execution_count": 14,
757 | "metadata": {},
758 | "output_type": "execute_result"
759 | }
760 | ],
761 | "source": [
762 | "quote_data(quote_divs[7])"
763 | ]
764 | },
765 | {
766 | "cell_type": "markdown",
767 | "metadata": {},
768 | "source": [
769 | "Now we can collect the data from every quote on the first page with a simple [```for loop```](https://www.w3schools.com/python/python_for_loops.asp)!"
770 | ]
771 | },
772 | {
773 | "cell_type": "code",
774 | "execution_count": 15,
775 | "metadata": {
776 | "ExecuteTime": {
777 | "end_time": "2021-04-21T21:10:53.468852Z",
778 | "start_time": "2021-04-21T21:10:53.462096Z"
779 | },
780 | "scrolled": true
781 | },
782 | "outputs": [
783 | {
784 | "name": "stdout",
785 | "output_type": "stream",
786 | "text": [
787 | "10 quotes scraped!\n"
788 | ]
789 | },
790 | {
791 | "data": {
792 | "text/plain": [
793 | "[{'author': 'Albert Einstein',\n",
794 | " 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',\n",
795 | " 'author_link': 'http://quotes.toscrape.com/author/Albert-Einstein',\n",
796 | " 'tags': ['change', 'deep-thoughts', 'thinking', 'world']},\n",
797 | " {'author': 'J.K. Rowling',\n",
798 | " 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',\n",
799 | " 'author_link': 'http://quotes.toscrape.com/author/J-K-Rowling',\n",
800 | " 'tags': ['abilities', 'choices']}]"
801 | ]
802 | },
803 | "execution_count": 15,
804 | "metadata": {},
805 | "output_type": "execute_result"
806 | }
807 | ],
808 | "source": [
809 | "page_one_data = []\n",
810 | "for div in quote_divs:\n",
811 | " # Apply our function on each quote\n",
812 | " data_from_div = quote_data(div)\n",
813 | " page_one_data.append(data_from_div)\n",
814 | " \n",
815 | "print(len(page_one_data), 'quotes scraped!')\n",
816 | "page_one_data[:2]"
817 | ]
818 | },
819 | {
820 | "cell_type": "markdown",
821 | "metadata": {
822 | "ExecuteTime": {
823 | "end_time": "2020-07-03T18:55:14.762708Z",
824 | "start_time": "2020-07-03T18:55:14.758667Z"
825 | }
826 | },
827 | "source": [
828 | "# We just scraped an entire webpage!\n",
829 | "\n",
830 | ""
831 | ]
832 | },
833 | {
834 | "cell_type": "markdown",
835 | "metadata": {},
836 | "source": [
837 | "### Level Up: What if we wanted to scrape the quotes from *every* page?\n",
838 | "\n",
839 | "**Step 1:** The first thing we do is take the code from above that scraped the data for all of the quotes on page one, and move it into a function called `parse_page`."
840 | ]
841 | },
842 | {
843 | "cell_type": "code",
844 | "execution_count": 16,
845 | "metadata": {
846 | "ExecuteTime": {
847 | "end_time": "2021-04-21T21:10:53.473642Z",
848 | "start_time": "2021-04-21T21:10:53.470305Z"
849 | }
850 | },
851 | "outputs": [],
852 | "source": [
853 | "def scrape_page(quote_divs):\n",
854 | " data = []\n",
855 | " for div in quote_divs:\n",
856 | " div_data = quote_data(div)\n",
857 | " data.append(div_data)\n",
858 | " \n",
859 | " return data"
860 | ]
861 | },
862 | {
863 | "cell_type": "markdown",
864 | "metadata": {},
865 | "source": [
866 | "**Step 2:** We grab the code we used at the very beginning to collect the html and the list of divs from a web page."
867 | ]
868 | },
869 | {
870 | "cell_type": "code",
871 | "execution_count": 17,
872 | "metadata": {
873 | "ExecuteTime": {
874 | "end_time": "2021-04-21T21:10:53.601521Z",
875 | "start_time": "2021-04-21T21:10:53.475074Z"
876 | }
877 | },
878 | "outputs": [],
879 | "source": [
880 | "base_url = 'http://quotes.toscrape.com'\n",
881 | "response = requests.get(url)\n",
882 | "html = response.text\n",
883 | "soup = BeautifulSoup(html, 'lxml')\n",
884 | "quote_divs = soup.find_all('div', {'class': 'quote'})"
885 | ]
886 | },
887 | {
888 | "cell_type": "markdown",
889 | "metadata": {},
890 | "source": [
891 | "**Step 3:** We feed all of the `quote_divs` into our newly made `parse_page` function to grab all of the data from that page."
892 | ]
893 | },
894 | {
895 | "cell_type": "code",
896 | "execution_count": 18,
897 | "metadata": {
898 | "ExecuteTime": {
899 | "end_time": "2021-04-21T21:10:53.606735Z",
900 | "start_time": "2021-04-21T21:10:53.602884Z"
901 | }
902 | },
903 | "outputs": [],
904 | "source": [
905 | "data = scrape_page(quote_divs)"
906 | ]
907 | },
908 | {
909 | "cell_type": "code",
910 | "execution_count": 19,
911 | "metadata": {
912 | "ExecuteTime": {
913 | "end_time": "2021-04-21T21:10:53.611866Z",
914 | "start_time": "2021-04-21T21:10:53.608230Z"
915 | }
916 | },
917 | "outputs": [
918 | {
919 | "data": {
920 | "text/plain": [
921 | "[{'author': 'Albert Einstein',\n",
922 | " 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',\n",
923 | " 'author_link': 'http://quotes.toscrape.com/author/Albert-Einstein',\n",
924 | " 'tags': ['change', 'deep-thoughts', 'thinking', 'world']},\n",
925 | " {'author': 'J.K. Rowling',\n",
926 | " 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',\n",
927 | " 'author_link': 'http://quotes.toscrape.com/author/J-K-Rowling',\n",
928 | " 'tags': ['abilities', 'choices']}]"
929 | ]
930 | },
931 | "execution_count": 19,
932 | "metadata": {},
933 | "output_type": "execute_result"
934 | }
935 | ],
936 | "source": [
937 | "data[:2]"
938 | ]
939 | },
940 | {
941 | "cell_type": "markdown",
942 | "metadata": {},
943 | "source": [
944 | "**Step 4:** We check to see if there is a `Next Page` button at the bottom of the page.\n",
945 | "\n",
946 | "*This is requires multiple steps.*\n",
947 | "\n",
948 | "1. We grab the outer container called that has a class of `pager`."
949 | ]
950 | },
951 | {
952 | "cell_type": "code",
953 | "execution_count": 20,
954 | "metadata": {
955 | "ExecuteTime": {
956 | "end_time": "2021-04-21T21:10:53.616414Z",
957 | "start_time": "2021-04-21T21:10:53.613232Z"
958 | }
959 | },
960 | "outputs": [],
961 | "source": [
962 | "pager = soup.find('ul', {'class': 'pager'})"
963 | ]
964 | },
965 | {
966 | "cell_type": "markdown",
967 | "metadata": {},
968 | "source": [
969 | "If there is no pager element on the webpage, pager will be set to `None`.\n",
970 | "\n",
971 | "2. We use an if check to make sure a pager exists:"
972 | ]
973 | },
974 | {
975 | "cell_type": "code",
976 | "execution_count": 21,
977 | "metadata": {
978 | "ExecuteTime": {
979 | "end_time": "2021-04-21T21:10:53.620316Z",
980 | "start_time": "2021-04-21T21:10:53.617767Z"
981 | }
982 | },
983 | "outputs": [],
984 | "source": [
985 | "if pager:\n",
986 | " next_page = pager.find('li', {'class': 'next'})"
987 | ]
988 | },
989 | {
990 | "cell_type": "markdown",
991 | "metadata": {},
992 | "source": [
993 | "3. We then check to see if a `Next Page` button exists on the page. \n",
994 | "\n",
995 | " - Every page has a next button except the *last* page which only has a `Previous Page` button. Basically, we're checking to see if the `Next button` exists. It it does, we \"click\" it."
996 | ]
997 | },
998 | {
999 | "cell_type": "code",
1000 | "execution_count": 22,
1001 | "metadata": {
1002 | "ExecuteTime": {
1003 | "end_time": "2021-04-21T21:10:53.624558Z",
1004 | "start_time": "2021-04-21T21:10:53.621819Z"
1005 | }
1006 | },
1007 | "outputs": [],
1008 | "source": [
1009 | "if next_page:\n",
1010 | " next_page = next_page.findChild('a')\\\n",
1011 | " .attrs\\\n",
1012 | " .get('href')"
1013 | ]
1014 | },
1015 | {
1016 | "cell_type": "markdown",
1017 | "metadata": {},
1018 | "source": [
1019 | "With most webscraping tools, \"clicking a button\" means collecting the link inside the button and making a new request.\n",
1020 | "\n",
1021 | "If a link is pointing to a page on the same website, the links are usually just the forward slashes that need to be added to the base website url. This is called a `relative` link.\n",
1022 | "\n",
1023 | "**Step 5:** Collect the relative link that points to the next page, and add it to our base_url"
1024 | ]
1025 | },
1026 | {
1027 | "cell_type": "code",
1028 | "execution_count": 23,
1029 | "metadata": {
1030 | "ExecuteTime": {
1031 | "end_time": "2021-04-21T21:10:53.630467Z",
1032 | "start_time": "2021-04-21T21:10:53.627259Z"
1033 | }
1034 | },
1035 | "outputs": [
1036 | {
1037 | "data": {
1038 | "text/plain": [
1039 | "'http://quotes.toscrape.com/page/2/'"
1040 | ]
1041 | },
1042 | "execution_count": 23,
1043 | "metadata": {},
1044 | "output_type": "execute_result"
1045 | }
1046 | ],
1047 | "source": [
1048 | "next_page = url + next_page\n",
1049 | "next_page"
1050 | ]
1051 | },
1052 | {
1053 | "cell_type": "markdown",
1054 | "metadata": {},
1055 | "source": [
1056 | "**Step 6:** We repeat the exact same process for this new link!\n",
1057 | "\n",
1058 | "ie:\n",
1059 | "1. Make request using a url that points to the next page.\n",
1060 | "2. Scrape quote divs\n",
1061 | "3. Collect data from every quote div on that page\n",
1062 | "4. Find the `Next page` button.\n",
1063 | "5. Collect the url from the button\n",
1064 | "6. Repeat\n",
1065 | "\n",
1066 | "So how do we do this over and over again without repeating ourselves?\n",
1067 | "\n",
1068 | "The first step is compile all of these steps into a new function called `scrape_quotes`.\n",
1069 | "\n",
1070 | "The second step is, something called `recursion`. \n",
1071 | "\n",
1072 | "| \n", 1250 | " | author | \n", 1251 | "text | \n", 1252 | "author_link | \n", 1253 | "tags | \n", 1254 | "
|---|---|---|---|---|
| 0 | \n", 1259 | "Albert Einstein | \n", 1260 | "“The world as we have created it is a process ... | \n", 1261 | "http://quotes.toscrape.com/author/Albert-Einstein | \n", 1262 | "[change, deep-thoughts, thinking, world] | \n", 1263 | "
| 1 | \n", 1266 | "J.K. Rowling | \n", 1267 | "“It is our choices, Harry, that show what we t... | \n", 1268 | "http://quotes.toscrape.com/author/J-K-Rowling | \n", 1269 | "[abilities, choices] | \n", 1270 | "
| 2 | \n", 1273 | "Albert Einstein | \n", 1274 | "“There are only two ways to live your life. On... | \n", 1275 | "http://quotes.toscrape.com/author/Albert-Einstein | \n", 1276 | "[inspirational, life, live, miracle, miracles] | \n", 1277 | "
| 3 | \n", 1280 | "Jane Austen | \n", 1281 | "“The person, be it gentleman or lady, who has ... | \n", 1282 | "http://quotes.toscrape.com/author/Jane-Austen | \n", 1283 | "[aliteracy, books, classic, humor] | \n", 1284 | "
| 4 | \n", 1287 | "Marilyn Monroe | \n", 1288 | "“Imperfection is beauty, madness is genius and... | \n", 1289 | "http://quotes.toscrape.com/author/Marilyn-Monroe | \n", 1290 | "[be-yourself, inspirational] | \n", 1291 | "
48 |
49 |
50 | ```python
51 | url = 'http://quotes.toscrape.com'
52 | response = requests.get(url)
53 | response
54 | ```
55 |
56 |
57 |
58 |
59 |
83 |
84 | ## Step 3
85 | > **We collect the html from the website by adding `.text` to the end of the response variable.**
86 |
87 |
88 | ```python
89 | html = response.text
90 | html[:50]
91 | ```
92 |
93 |
94 |
95 |
96 | '\n\n\n\t **We use `BeautifulSoup` to turn the html into something we can manipulate.**
102 |
103 |
104 | ```python
105 | soup = BeautifulSoup(html, 'lxml')
106 | soup
107 | ```
108 |
109 |
110 |
111 |
112 |
113 |
114 |
115 |
116 | 130 | Login 131 |
132 |
329 |
330 | **Fortunately for us,** we do not have to read through every line of the html in order to web scrape.
331 |
332 | Modern day web browsers come equipped with tools that allow us to easily sift through this soupy text.
333 |
334 |
335 | ## Step 4
336 | >**We open up the page we're trying to scrape in a new tab.** Click Here! 👀
337 |
338 | ## Step 5
339 | > **We create a list of every `div` that has a `class` of "quote"**
340 |
341 | **In this instance,** every item we want to collect is divided into identically labeled containers.
342 | - A div with a class of 'quote'.
343 |
344 | Not all HTML is as well organized as this page, but HTML is basically just a bunch of different organizational containers that we use to divide up text and other forms of media. Figuring out the organizational structure of a website is the entire process for web scraping.
345 |
346 |
347 | ```python
348 | quote_divs = soup.find_all('div', {'class': 'quote'})
349 | len(quote_divs)
350 | ```
351 |
352 |
353 |
354 |
355 | 10
356 |
357 |
358 |
359 | ## Step 6
360 | > **To figure out how to grab all the datapoints from a quote, we isolate a single quote, and work through the code for a single `div`.**
361 |
362 |
363 | ```python
364 | first_quote = quote_divs[0]
365 | ```
366 |
367 | ### First we grab the text of the quote
368 |
369 |
370 | ```python
371 | text = first_quote.find('span', {'class':'text'})
372 | text = text.text
373 | text
374 | ```
375 |
376 |
377 |
378 |
379 | '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
380 |
381 |
382 |
383 | ### Next we grab the author
384 |
385 |
386 | ```python
387 | author = first_quote.find('small', {'class': 'author'})
388 | author_name = author.text
389 | author_name
390 | ```
391 |
392 |
393 |
394 |
395 | 'Albert Einstein'
396 |
397 |
398 |
399 | ### Let's also grab the link that points to the author's bio!
400 |
401 |
402 | ```python
403 | author_link = author.findNextSibling().attrs.get('href')
404 | author_link = url + author_link
405 | author_link
406 | ```
407 |
408 |
409 |
410 |
411 | 'http://quotes.toscrape.com/author/Albert-Einstein'
412 |
413 |
414 |
415 | ### And finally, let's grab all of the tags for the quote
416 |
417 |
418 | ```python
419 | tag_container = first_quote.find('div', {'class': 'tags'})
420 | tag_links = tag_container.find_all('a')
421 |
422 | tags = []
423 | for tag in tag_links:
424 | tags.append(tag.text)
425 |
426 | tags
427 | ```
428 |
429 |
430 |
431 |
432 | ['change', 'deep-thoughts', 'thinking', 'world']
433 |
434 |
435 |
436 | ### Our data:
437 |
438 |
439 | ```python
440 | print('text:', text)
441 | print('author name: ', author_name)
442 | print('author link: ', author_link)
443 | print('tags: ', tags)
444 | ```
445 |
446 | text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
447 | author name: Albert Einstein
448 | author link: http://quotes.toscrape.com/author/Albert-Einstein
449 | tags: ['change', 'deep-thoughts', 'thinking', 'world']
450 |
451 |
452 | # Step 7
453 | > We create a function to make out code reusable.
454 |
455 | Now that we know how to collect this data from a quote div, we can compile the code into a [function](https://www.geeksforgeeks.org/functions-in-python/) called `quote_data`. This allows us to grab a quote div, feed it into the function like so...
456 | > `quote_data(quote_div)`
457 |
458 | ...and receive all of the data from that div.
459 |
460 |
461 | ```python
462 | def quote_data(quote_div):
463 | # Collect the quote
464 | text = quote_div.find('span', {'class':'text'})
465 | text = text.text
466 |
467 | # Collect the author name
468 | author = quote_div.find('small', {'class': 'author'})
469 | author_name = author.text
470 |
471 | # Collect author link
472 | author_link = author.findNextSibling().attrs.get('href')
473 | author_link = url + author_link
474 |
475 | # Collect tags
476 | tag_container = quote_div.find('div', {'class': 'tags'})
477 |
478 | tag_links = tag_container.find_all('a')
479 |
480 | tags = []
481 | for tag in tag_links:
482 | tags.append(tag.text)
483 |
484 | # Return data as a dictionary
485 | return {'author': author_name,
486 | 'text': text,
487 | 'author_link': author_link,
488 | 'tags': tags}
489 | ```
490 |
491 | ### Let's test our fuction.
492 |
493 |
494 | ```python
495 | quote_data(quote_divs[7])
496 | ```
497 |
498 |
499 |
500 |
501 | {'author': 'Thomas A. Edison',
502 | 'text': "“I have not failed. I've just found 10,000 ways that won't work.”",
503 | 'author_link': 'http://quotes.toscrape.com/author/Thomas-A-Edison',
504 | 'tags': ['edison', 'failure', 'inspirational', 'paraphrased']}
505 |
506 |
507 |
508 | Now we can collect the data from every quote on the first page with a simple [```for loop```](https://www.w3schools.com/python/python_for_loops.asp)!
509 |
510 |
511 | ```python
512 | page_one_data = []
513 | for div in quote_divs:
514 | # Apply our function on each quote
515 | data_from_div = quote_data(div)
516 | page_one_data.append(data_from_div)
517 |
518 | print(len(page_one_data), 'quotes scraped!')
519 | page_one_data[:2]
520 | ```
521 |
522 | 10 quotes scraped!
523 |
524 |
525 |
526 |
527 |
528 | [{'author': 'Albert Einstein',
529 | 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
530 | 'author_link': 'http://quotes.toscrape.com/author/Albert-Einstein',
531 | 'tags': ['change', 'deep-thoughts', 'thinking', 'world']},
532 | {'author': 'J.K. Rowling',
533 | 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
534 | 'author_link': 'http://quotes.toscrape.com/author/J-K-Rowling',
535 | 'tags': ['abilities', 'choices']}]
536 |
537 |
538 |
539 | # We just scraped an entire webpage!
540 |
541 | 
542 |
543 | ### Level Up: What if we wanted to scrape the quotes from *every* page?
544 |
545 | **Step 1:** The first thing we do is take the code from above that scraped the data for all of the quotes on page one, and move it into a function called `parse_page`.
546 |
547 |
548 | ```python
549 | def scrape_page(quote_divs):
550 | data = []
551 | for div in quote_divs:
552 | div_data = quote_data(div)
553 | data.append(div_data)
554 |
555 | return data
556 | ```
557 |
558 | **Step 2:** We grab the code we used at the very beginning to collect the html and the list of divs from a web page.
559 |
560 |
561 | ```python
562 | base_url = 'http://quotes.toscrape.com'
563 | response = requests.get(url)
564 | html = response.text
565 | soup = BeautifulSoup(html, 'lxml')
566 | quote_divs = soup.find_all('div', {'class': 'quote'})
567 | ```
568 |
569 | **Step 3:** We feed all of the `quote_divs` into our newly made `parse_page` function to grab all of the data from that page.
570 |
571 |
572 | ```python
573 | data = scrape_page(quote_divs)
574 | ```
575 |
576 |
577 | ```python
578 | data[:2]
579 | ```
580 |
581 |
582 |
583 |
584 | [{'author': 'Albert Einstein',
585 | 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
586 | 'author_link': 'http://quotes.toscrape.com/author/Albert-Einstein',
587 | 'tags': ['change', 'deep-thoughts', 'thinking', 'world']},
588 | {'author': 'J.K. Rowling',
589 | 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
590 | 'author_link': 'http://quotes.toscrape.com/author/J-K-Rowling',
591 | 'tags': ['abilities', 'choices']}]
592 |
593 |
594 |
595 | **Step 4:** We check to see if there is a `Next Page` button at the bottom of the page.
596 |
597 | *This is requires multiple steps.*
598 |
599 | 1. We grab the outer container called that has a class of `pager`.
600 |
601 |
602 | ```python
603 | pager = soup.find('ul', {'class': 'pager'})
604 | ```
605 |
606 | If there is no pager element on the webpage, pager will be set to `None`.
607 |
608 | 2. We use an if check to make sure a pager exists:
609 |
610 |
611 | ```python
612 | if pager:
613 | next_page = pager.find('li', {'class': 'next'})
614 | ```
615 |
616 | 3. We then check to see if a `Next Page` button exists on the page.
617 |
618 | - Every page has a next button except the *last* page which only has a `Previous Page` button. Basically, we're checking to see if the `Next button` exists. It it does, we "click" it.
619 |
620 |
621 | ```python
622 | if next_page:
623 | next_page = next_page.findChild('a')\
624 | .attrs\
625 | .get('href')
626 | ```
627 |
628 | With most webscraping tools, "clicking a button" means collecting the link inside the button and making a new request.
629 |
630 | If a link is pointing to a page on the same website, the links are usually just the forward slashes that need to be added to the base website url. This is called a `relative` link.
631 |
632 | **Step 5:** Collect the relative link that points to the next page, and add it to our base_url
633 |
634 |
635 | ```python
636 | next_page = url + next_page
637 | next_page
638 | ```
639 |
640 |
641 |
642 |
643 | 'http://quotes.toscrape.com/page/2/'
644 |
645 |
646 |
647 | **Step 6:** We repeat the exact same process for this new link!
648 |
649 | ie:
650 | 1. Make request using a url that points to the next page.
651 | 2. Scrape quote divs
652 | 3. Collect data from every quote div on that page
653 | 4. Find the `Next page` button.
654 | 5. Collect the url from the button
655 | 6. Repeat
656 |
657 | So how do we do this over and over again without repeating ourselves?
658 |
659 | The first step is compile all of these steps into a new function called `scrape_quotes`.
660 |
661 | The second step is, something called `recursion`.
662 |
663 | | 779 | | author | 780 |text | 781 |author_link | 782 |tags | 783 |
|---|---|---|---|---|
| 0 | 788 |Albert Einstein | 789 |“The world as we have created it is a process ... | 790 |http://quotes.toscrape.com/author/Albert-Einstein | 791 |[change, deep-thoughts, thinking, world] | 792 |
| 1 | 795 |J.K. Rowling | 796 |“It is our choices, Harry, that show what we t... | 797 |http://quotes.toscrape.com/author/J-K-Rowling | 798 |[abilities, choices] | 799 |
| 2 | 802 |Albert Einstein | 803 |“There are only two ways to live your life. On... | 804 |http://quotes.toscrape.com/author/Albert-Einstein | 805 |[inspirational, life, live, miracle, miracles] | 806 |
| 3 | 809 |Jane Austen | 810 |“The person, be it gentleman or lady, who has ... | 811 |http://quotes.toscrape.com/author/Jane-Austen | 812 |[aliteracy, books, classic, humor] | 813 |
| 4 | 816 |Marilyn Monroe | 817 |“Imperfection is beauty, madness is genius and... | 818 |http://quotes.toscrape.com/author/Marilyn-Monroe | 819 |[be-yourself, inspirational] | 820 |