├── .gitignore
├── LICENSE
├── README.md
├── example_script
    ├── README.md
    ├── download_pictures_from_google.R
    ├── iframe_tutorial.Rmd
    ├── iframe_tutorial.md
    └── resources
    │   ├── README.md
    │   ├── turtle0.jpg
    │   ├── turtle1.jpg
    │   └── turtle2.jpg
└── resources
    ├── .gitkeep
    ├── functions_and_classes.png
    ├── placeholder.jpg
    └── rvest_parallel.md


/.gitignore:
--------------------------------------------------------------------------------
1 | .Rproj.user
2 | .Rhistory
3 | .RData
4 | .Ruserdata
5 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2019 Yifu Yan
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
   1 | # Web Scraping Reference: Cheat Sheet for Web Scraping using R
   2 | 
   3 | Inspired by Hartley Brody, this cheat sheet is about web scraping using [rvest](https://github.com/hadley/rvest),[httr](https://github.com/r-lib/httr) and [Rselenium](https://github.com/ropensci/RSelenium). It covers many topics in this [blog](https://blog.hartleybrody.com/web-scraping-cheat-sheet/).
   4 | 
   5 | While Hartley uses python's requests and beautifulsoup libraries, this cheat sheet covers the usage of httr and rvest. While rvest is good enough for many scraping tasks, httr is required for more advanced techniques. Usage of Rselenium(web driver) is also covered.
   6 | 
   7 | I also recommend the book [The Ultimate Guide to Web Scraping](https://blog.hartleybrody.com/guide-to-web-scraping/) by Hartley Brody. Though it uses Python libraries, the underlying logic of web scraping is the same. The same strategies can be applied using any languages including R.
   8 | 
   9 | Please post issues [here](https://github.com/yusuzech/r-web-scraping-cheat-sheet/issues) if you find any errors or have any recommendations.
  10 | 
  11 | # Table of Contents
  12 | 1. <a href="#rvest">Web Scraping using rvest and httr</a>
  13 |     1. <a href="#rvest1">Useful Libraries and Resources</a>
  14 |     2. <a href="#rvest2">Making Simple Requests</a>
  15 |     3. <a href="#rvest3">Inspecting Response</a>
  16 |     4. <a href="#rvest4">Extracting Elements from HTML</a>
  17 |     5. <a href="#rvest5">Storing Data in R</a>
  18 |         1. <a href="#rvest5.1">Storing Data as list</a>
  19 |         2. <a href="#rvest5.2">Storing Data as data.frame</a>
  20 |     6. <a href="#rvest6">Saving Data to disk</a>
  21 |         1. <a href="#rvest6.1">Saving Data to csv</a>
  22 |         2. <a href="#rvest6.2">Saving Data to SQLite Database</a>
  23 |     7. <a href="#rvest7">More Advanced Topics</a>
  24 |         1. <a href="#rvest7.1">Javascript Heavy Websites</a>
  25 |         2. <a href="#rvest7.2">Content Inside iFrames</a>
  26 |         3. <a href="#rvest7.3">Sessions and Cookies</a>
  27 |         4. <a href="#rvest7.4">Delays and Backing Off</a>
  28 |         5. <a href="#rvest7.5">Spoofing the User Agent</a>
  29 |         6. <a href="#rvest7.6">Using Proxy Servers</a>
  30 |         7. <a href="#rvest7.7">Setting Timeouts</a>
  31 |         8. <a href="#rvest7.8">Handling Network Errors</a>
  32 |         9. <a href="#rvest7.9">Downloading Files</a>
  33 |         10. <a href="#rvest7.10">Logins and Sessions</a>
  34 |         11. <a href="#rvest7.11">Web Scraping in Parallel</a>
  35 | 2. <a href="#rselenium">Web Scraping using Rselenium</a>
  36 |     1. <a href="#rselenium1">Why RSelenium</a>
  37 |        1. <a href="#rselenium1.1">Pros and Cons from Using RSelenium</a>
  38 |        2. <a href="#rselenium1.2">Useful Resources</a>
  39 |     2. <a href="#rselenium2">Interacting with the Web Driver In Rselenium</a>
  40 |        1. <a href="#rselenium2.0">How to Start</a>
  41 |        2. <a href="#rselenium2.1">Navigating to different URLs</a>
  42 |        3. <a href="#rselenium2.2">Simulating Scrolls, Clicks, Text Inputs, Logins, and Other Actions</a>
  43 |     3. <a href="#rselenium3">Extract Content from the Web Page</a>
  44 |        1. <a href="#rselenium3.1">Extracting Content using Rselenium</a>
  45 |        2. <a href="#rselenium3.2">Extracting Content using Parsed Page Source and `rvest`</a>
  46 |     4. <a href = "#rselneium4">miscellanea</a>
  47 |        1. <a href="#rselenium4.1">Javascript</a>
  48 |        2. <a href="#rselenium4.2">Iframe</a>
  49 | 3. <a href="#change">Change Log</a>
  50 | 
  51 | # 1. <a name="rvest">Web Scraping using rvest and httr</a>
  52 | ## 1.1. <a name="rvest1">Useful Libraries and Resources</a>
  53 | 
  54 | [rvest](https://github.com/hadley/rvest) is built upon the xml2 package and also accept config from the `httr` package. For the most part, we only need `rvest`. However, we need `httr` if we want to add extra configurations.
  55 | 
  56 | To install those two packages:
  57 | 
  58 | ```r
  59 | install.packages("rvest")
  60 | install.packages("httr")
  61 | ```
  62 | 
  63 | To load them:
  64 | 
  65 | ```r
  66 | require(rvest)
  67 | require(httr)
  68 | ```
  69 | 
  70 | There are many resources available online; these are what I found to be the most useful:
  71 | 
  72 | 1. [w3schools CSS selectors reference](https://www.w3schools.com/CSSref/css_selectors.asp) : if you forget CSS syntax, just check it here
  73 | 1. [w3schools XPATH reference](<https://www.w3schools.com/xml/xpath_intro.asp>): XPATH is an alternative in selecting elements on websites. It's harder to learn but it's more flexible and robust.
  74 | 1. [CSS Diner](https://flukeout.github.io/) : the easiest way to learn and understand CSS by playing games.
  75 | 1. [Chrome CSS selector plugin](https://selectorgadget.com/): a convenient tool to use for choosing CSS selector.
  76 | 1. [ChroPath](<https://chrome.google.com/webstore/detail/chropath/ljngjbnaijcbncmcnjfhigebomdlkcjo>): a very convenient tool for choosing XPATH.
  77 | 1. [Stack Overflow](https://stackoverflow.com/) : You can find answers to most of your problems, no matter it's web scraping, rvest or CSS.
  78 | 1. [Web Scraping Sandbox](http://toscrape.com/): Great place to test your web scraping skills.
  79 | 
  80 | **Functions and classes in rvest/httr:**
  81 | 
  82 | Sometimes you may get confused about all the functions and classes you have. You can review this image at the moment.
  83 | ![](resources/functions_and_classes.png)
  84 | 
  85 | \*\* Please notice that: somtimes response could be JSON or other formats instead of HTML. In those cases you need other functions to parse the content(e.g. jsonlite::fromJSON() to parse a JSON string to a list).
  86 | 
  87 | 
  88 | ## 1.2. <a name="rvest2">Making Simple Requests</a>
  89 | 
  90 | rvest provides two ways of making request: `read_html()` and `html_session()`  
  91 | `read_html()` can parse a HTML file or an url into xml document. `html_session()` is built on `GET()` from httr package and can accept configurations defined by httr package.  
  92 | 
  93 | Reading a url:
  94 | 
  95 | ```R
  96 | #making GET request andparse website into xml document
  97 | pagesource <- read_html("http://example.com/page")
  98 | 
  99 | #using html_session which creates a session and accept httr methods
 100 | my_session <- html_session("http://example.com/page")
 101 | #html_session is built upon httr, you can also get response with a session
 102 | response <- my_session$response
 103 | ```
 104 | 
 105 | Alternatively, GET and POST method are available in the httr package.
 106 | 
 107 | ```R
 108 | library(httr)
 109 | response <- GET("http://example.com/page")
 110 | #or
 111 | response <- POST("http://example.com/page",
 112 |     body = list(a=1,b=2))
 113 | ```
 114 | 
 115 | ## 1.3. <a name="rvest3">Inspecting Response</a>
 116 | 
 117 | Check status code:
 118 | 
 119 | ```R
 120 | status_code(my_session)
 121 | status_code(response)
 122 | ```
 123 | 
 124 | Get response and content:
 125 | 
 126 | ```R
 127 | #response
 128 | response <- my_session$response
 129 | #retrieve content as raw
 130 | content_raw <- content(my_session$response,as = "raw")
 131 | #retrieve content as text
 132 | content_text <- content(my_session$response,as = "text")
 133 | #retrieve content as parsed(parsed automatically)
 134 | content_parsed <- content(my_session$response,as = "parsed")
 135 | ```
 136 | 
 137 | \*\*note:
 138 | 
 139 | Content may be parsed incorrectly sometimes. For those situations, you can parse the content to text or raw and use other libraries or functions to parse it correctly.
 140 | 
 141 | Search for specific string:
 142 | 
 143 | ```R
 144 | library(stringr)
 145 | #regular expression can also be used here
 146 | if(str_detect(content_text,"blocked")){
 147 |     print("blocked from website")
 148 |     }
 149 | ```
 150 | 
 151 | check content type:
 152 | 
 153 | ```R
 154 | response$headers$`content-type`
 155 | ```
 156 | 
 157 | check html structure:
 158 | 
 159 | ```R
 160 | my_structure <- html_structure(content_parsed)
 161 | ```
 162 | 
 163 | ## 1.4. <a name="rvest4">Extracting Elements from HTML</a>
 164 | 
 165 | Using the regular expression to scrape HTML is not a very good idea, but it does have its usage like scraping all emails from websites, there is a detailed discussion about this topic on [stackoverflow](https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags).  
 166 | 
 167 | **Using rvest:**
 168 | 
 169 | I will scrape https://scrapethissite.com/ for demonstration, since it has static HTML:  
 170 | 
 171 | For the purpose of extracting elements, using `read_html()` or `html_session()` are both fine. When using `read_html()`, it returns a xml_document. When using `html_session()`, it creates a session and the response is included.
 172 | 
 173 | ```R
 174 | my_session <- html_session("https://scrapethissite.com/pages/simple/")
 175 | ```
 176 | 
 177 | Look for nodes:
 178 | 
 179 | ```R
 180 | my_nodes <- my_session %>% html_elements(".country")
 181 | ```
 182 | 
 183 | Look for attributes:
 184 | 
 185 | ```R
 186 | my_attributes <- my_session %>% html_elements(".country-capital") %>% html_attr("class")
 187 | ```
 188 | 
 189 | Look for texts:
 190 | 
 191 | ```R
 192 | my_texts <- my_session %>% html_elements(".country-capital") %>% html_text()
 193 | ```
 194 | 
 195 | ## 1.5. <a name="rvest5">Storing Data in R</a>
 196 | 
 197 | rvest can return a vector of elements or even table of elements, so it's easy to store it in R.
 198 | 
 199 | ### 1.5.1. <a name="rvest5.1">Storing Data as list</a>
 200 | 
 201 | Usually, rvest can return a vector, so it's very easy to store it.
 202 | 
 203 | ```R
 204 | my_texts <- my_session %>% html_elements(".country-capital") %>% html_text()
 205 | ```
 206 | 
 207 | ### 1.5.2. <a name="rvest5.2">Storing Data as data.frame</a>
 208 | 
 209 | We can concatenate vectors in a table or using `html_table()` to extract a HTML table directly into a data.frame.
 210 | 
 211 | ```R
 212 | my_country <- my_session %>% html_elements(".country-name") %>% html_text()
 213 | my_capitals <- my_session %>% html_elements(".country-capital") %>% html_text()
 214 | my_table <- data.frame(country = my_country, capital = my_capitals)
 215 | ```
 216 | 
 217 | ## 1.6. <a name="rvest6">Saving Data to disk</a>
 218 | ### 1.6.1. <a name="rvest6.1">Saving Data to csv</a>
 219 | 
 220 | If the data is already stored as a data.frame:
 221 | 
 222 | ```R
 223 | write.csv(my_table,file="my_table.csv")
 224 | ```
 225 | 
 226 | ### 1.6.2. <a name="rvest6.2">Saving Data to SQLite Database</a>
 227 | 
 228 | After creating the database "webscrape.db":
 229 | 
 230 | ```R
 231 | library(RSQLite)
 232 | connection <- dbConnect(SQLite(),"webscrape.db")
 233 | dbWriteTable(conn = connection,name = "country_capital",value = my_table)
 234 | dbDisconnect(conn = connection)
 235 | ```
 236 | 
 237 | ## 1.7. <a name="rvest7">More Advanced Topics</a>
 238 | 
 239 | <!-- 
 240 | rvest and httr package provides some handy functions, but they lack the function of:
 241 | 
 242 | + prevent errors from breaking loops
 243 | + auto retry requests
 244 | + keep the record of failed requests
 245 | 
 246 | So I wrote a wrapper function on html_session which can deal with the issues above and make web scraping scripts much more robust. The function and usage can be found in this [repository](https://github.com/yusuzech/r-web-scraping-template). 
 247 | -->
 248 | 
 249 | ### 1.7.1. <a name="rvest7.1">Javascript Heavy Websites</a>
 250 | 
 251 | For javascript heavy websites, there are three possible solutions:
 252 | 
 253 | 1. Execute javascript in R
 254 | 2. Use Developer tools(e.g. [Network in Chrome](https://developers.google.com/web/tools/chrome-devtools/network-performance/))
 255 | 3. Using Rselenium or other web drivers
 256 | 
 257 | There are pros and cons of each method:  
 258 | 1. Executing Javascript in R is the most difficult one since it requires some knowledge of Javascript, but it makes web-scraping javascript heavy websites possible with rvest.  
 259 | 2. Using Developer tools is not difficult. The pro is that you only need to learn some examples and you can then work on it by yourself. The con is that if the website structure gets more completed, it requires more knowledge of HTTP.
 260 | 3. The Rselenium is absolutely the easiest solution. The pro is it's easy to learn and use. The con is that it can be unstable sometimes and related resources are very limited online. In many situations, you may need to refer to python codes with selenium package.
 261 | 
 262 | #### 1.Execute javascript
 263 | 
 264 | I learned how to use Javascript with this [post](https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/). Since I'm not an expert in Javascript, I recommend you to search for other related resources online.
 265 | 
 266 | If you don't know Javascipt, then this method is unlikely to be suitable for you. Since Javascript isn't an easy subject, I don't recommend you to learn it if your only purpose is to use it to do web-scraping in R. The following two methods are much easier.
 267 | 
 268 | #### 2.Use Developer tools
 269 | 
 270 | I learned this trick from Hartley's blog; the following section is quoted from his [post](https://blog.hartleybrody.com/web-scraping-cheat-sheet/):
 271 | 
 272 | >Contrary to popular belief, you do not need any special tools to scrape websites that load their content via Javascript. For the information to get from their server and show up on a page in your browser, that information had to have been returned in an HTTP response somewhere.
 273 | >
 274 | >It usually means that you won’t be making an HTTP request to the page’s URL that you see at the top of your browser window, but instead you’ll need to find the URL of the AJAX request that’s going on in the background to fetch the data from the server and load it into the page.
 275 | >
 276 | >There’s not really an easy code snippet I can show here, but if you open the Chrome or Firefox Developer Tools, you can load the page, go to the “Network” tab and then look through the all of the requests that are being sent in the background to find the one that’s returning the data you’re looking for. Start by filtering the requests to only XHR or JS to make this easier.
 277 | >
 278 | >Once you find the AJAX request that returns the data you’re hoping to scrape, then you can make your scraper send requests to this URL, instead of to the parent page’s URL. If you’re lucky, the response will be encoded with JSON which is even easier to parse than HTML.
 279 | 
 280 | So, as Hartley said, basically, everything displayed on your browser must be sent to you through JSON, HTML or other formats. What you need to do is to capture this file. 
 281 | 
 282 | I answered a couple of questions on Stack Overflow about scraping JavaScript rendered content. You may have the same problems as in the posts. Please check out the answers, and hope you get the ideas of how to use developer tools.
 283 | 
 284 | https://stackoverflow.com/questions/50596714/how-to-scrap-a-jsp-page-in-r/50598032#50598032  
 285 | https://stackoverflow.com/questions/50765111/hover-pop-up-text-cannot-be-selected-for-rvest-in-r/50769875#50769875  
 286 | https://stackoverflow.com/questions/50900987/scraping-dl-dt-dd-html-data/50922733#50922733  
 287 | https://stackoverflow.com/questions/50693362/unable-to-extract-thorough-data-using-rvest/50730204#50730204  
 288 | https://stackoverflow.com/questions/50997094/trouble-reaching-a-css-node/50997121#50997121  
 289 | https://stackoverflow.com/questions/50262108/scraping-javascript-rendered-content-using-r/50263349#50263349  
 290 | 
 291 | #### 3.Using Rselenium or other web driver
 292 | 
 293 | Rselenium launches a Chrome/Firefox/IE browser where you can simulate human actions like clicking on links, scrolling up or down.  
 294 | 
 295 | It is a very convenient tool, and it renders JavaScript and Interactive content automatically, so you don't need to worry about the complex HTTP and AJAX stuff. However, there are also some limitations to it:  
 296 | 
 297 | 1. The first limitation is that:  it is very slow. Depending on the complexity of the websites, it could take seconds to render a single page while using httr/rvest takes less than one second. It is fine if you only want to scrape several hundred pages. However, if you want scrape thousands or ten thousands of pages, then the speed will become an issue.
 298 | 2. The second limitation is that: There are little online resources on Rselenium. In many situations, you can't find related posts on Stack Overflow that solve your problem. You may need to refer to Python/Java Selenium posts for answers, and sometimes answers can't be applied in R.
 299 | 
 300 | More detailed usage is explained in **Web Scraping using Rselenium**.
 301 | 
 302 | ### 1.7.2. <a name="rvest7.2">Content Inside iFrames</a>
 303 | 
 304 | Iframes are other websites embedded in the websites you are viewing as explained on [Wikipedia](https://en.wikipedia.org/wiki/HTML_element#Frames):
 305 | 
 306 | > Frames allow a visual HTML Browser window to be split into segments, each of which can show a different document. This can lower bandwidth use, as repeating parts of a layout can be used in one frame, while variable content is displayed in another. This may come at a certain usability cost, especially in non-visual user agents,[[51\]](https://en.wikipedia.org/wiki/HTML_element#cite_note-58) due to separate and independent documents (or websites) being displayed adjacent to each other and being allowed to interact with the same parent window. Because of this cost, frames (excluding the `<iframe>` element) are only allowed in HTML 4.01 Frame-set. Iframes can also hold documents on different servers. In this case, the interaction between windows is blocked by the browser. 
 307 | 
 308 | 
 309 | 
 310 | Therefore, to extract content in an iframe, you need to find the link to the iframe HTML.
 311 | 
 312 | ```R
 313 | #example script
 314 | link_to_iframe <- my_session("www.example.com") %>%
 315 |     html_element("css to locate the iframe") %>%
 316 |     html_attr("src")
 317 | #make another request to the iframe and use this session to extract information
 318 | iframe_session <- html_session(link_to_iframe)
 319 | ```
 320 | 
 321 | 
 322 | 
 323 | Here is a tutorial about iframes using [scrapethissite](https://blog.hartleybrody.com/web-scraping-cheat-sheet/#content-inside-iframes):
 324 | 
 325 | In this tutorial, we will get information embedded in an iframe: [**Tutorial Link**](example_script/iframe_tutorial.md)  
 326 | 
 327 | 
 328 | 
 329 | ### 1.7.3. <a name="rvest7.3">Sessions and Cookies</a>
 330 | 
 331 | `rvest::html_session()` creates a session automatically, you can use `jump_to()` and `follow_link` to navigate to other web pages using the same session.
 332 | 
 333 | 
 334 | 
 335 | ```R
 336 | library(rvest)
 337 | url1 <- "https://scrapethissite.com/"
 338 | url2 <- "https://scrapethissite.com/pages/simple/"
 339 | my_session <- html_session(url1)
 340 | my_session <- my_session %>% jump_to(url2) # navigate to another url
 341 | ```
 342 | 
 343 | 
 344 | 
 345 | you can check session history:
 346 | 
 347 | 
 348 | 
 349 | ```R
 350 | > session_history(my_session)
 351 |   https://scrapethissite.com/
 352 | - https://scrapethissite.com/pages/simple/
 353 | ```
 354 | 
 355 | 
 356 | 
 357 |  you can access the cookies:
 358 | 
 359 | ```R
 360 | library(httr)
 361 | cookies(my_session) #cookies are saved in a table
 362 | ```
 363 | 
 364 | 
 365 | 
 366 | ### 1.7.4. <a name="rvest7.4">Delays and Backing Off</a>
 367 | 
 368 | You can slow down your requests by pausing between requests:
 369 | 
 370 | 
 371 | 
 372 | ```R
 373 | library(httr)
 374 | for(my_url in my_urls){
 375 |     response <- httr::GET(my_url)
 376 |     #do something
 377 |     Sys.sleep(5) # sleep 5 seconds
 378 | }
 379 | ```
 380 | 
 381 | 
 382 | 
 383 | You can also decide how you wait by measuring how long the site took to respond. So why should you do it, as Hartley says:
 384 | 
 385 | > Some also recommend adding a backoff that’s proportional to how long the site took to respond to your request. That way if the site gets overwhelmed and starts to slow down, your code will automatically back off. 
 386 | 
 387 | 
 388 | 
 389 | ```R
 390 | library(httr)
 391 | for(my_url in my_urls){
 392 |     t0 <- Sys.time()
 393 |     response <- httr::GET(my_url)
 394 |     t1 <- Sys.time()
 395 |     #do something
 396 |     response_delay <- as.numeric(t1-t0)
 397 |     Sys.sleep(10*response_delay) # sleep 10 times longer than response_delay
 398 | }
 399 | 
 400 | ```
 401 | 
 402 | 
 403 | 
 404 | ### 1.7.5. <a name="rvest7.5">Spoofing the User Agent</a>
 405 | 
 406 | **First of all, what is a user agent?**
 407 | 
 408 | > In computing, a user agent is software (a software agent) that is acting on behalf of a user. One common use of the term refers to a web browser telling a website information about the browser and operating system. This allows the website to customize content for the capabilities of a particular device, but also raises privacy issues.
 409 | 
 410 | In short, a user agent is a string that identifies you, and you can search "most popular browser user agents" on google to get a rough idea.
 411 | 
 412 | **So why should I use spoof user agent?**  
 413 | 
 414 | * You want to make your scraper look like a real user instead of a script. Some websites even don't allow an uncommon user agent to access.
 415 | 
 416 | ```R
 417 | library(rvest)
 418 | my_session <- html_session("https://scrapethissite.com/")
 419 | # if you don't use custom user agent, your user agent will be something like:
 420 | # RUN: my_session$response$request$options$useragent
 421 | "libcurl/7.59.0 r-curl/3.2 httr/1.3.1"
 422 | ```
 423 | 
 424 | **So how can I spoof it?**
 425 | 
 426 | ```R
 427 | library(rvest)
 428 | library(httr)
 429 | #1.spoof it with common user agent
 430 | ua <- user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36")
 431 | seesion_with_ua <- html_session("https://scrapethissite.com/",ua)
 432 | #2.fill your contact information in user agent if you want the website owner to contact you
 433 | ua_string <- "Contact me at xyz123@gmail.com"
 434 | seesion_with_ua <- html_session("https://scrapethissite.com/",user_agent(ua_string))
 435 | ```
 436 | 
 437 | ### 1.7.6. <a name="rvest7.6">Using Proxy Servers</a>
 438 | 
 439 | Some servers automatically ban an IP if it sees abnormal traffic from that IP. The way to avoid it is to use proxies, so you can spread your requests among different IPs and reduce the chance of the server banning you. 
 440 | 
 441 | Though there are many free proxies available, paid ones are usually more reliable. One provider I use is "Proxy Bonanza."
 442 | 
 443 | Once you have free or paid proxies, you can use them when you make requests:
 444 | 
 445 | ```R
 446 | library(rvest)
 447 | library(httr)
 448 | my_proxy <- use_proxy(url="http://example.com",
 449 |                      user_name = "myusername",
 450 |                      password = "mypassword",
 451 |                      auth = "one of basic, digest, digest_ie, gssnegotiate, ntlm, any")
 452 | #use it in html_session(),POST() or GET()
 453 | my_session <- html_session("https://scrapethissite.com/",my_proxy)
 454 | my_response <- GET("https://scrapethissite.com/",my_proxy)
 455 | ```
 456 | 
 457 | ### 1.7.7. <a name="rvest7.7">Setting Timeouts</a>
 458 | 
 459 | Sometimes you may encounter slow connections and want to move to other jobs instead of waiting. You can set timeout if you don't receive a response.    
 460 | 
 461 | ```R
 462 | library(rvest)
 463 | library(httr)
 464 | my_session <- html_session("https://scrapethissite.com/",timeout(5)) # if you don't receive reponse within 5 seconds, it will throw an error
 465 | 
 466 | #you can use try() or tryCatch() to continue if the error occured
 467 | for(my_url in my_urls){
 468 |     try(GET(my_url,timeout(5)))
 469 | }
 470 | ```
 471 | 
 472 | For detailed usage of `try()` and `tryCatch()`, you can check the following posts:
 473 | 
 474 | https://stackoverflow.com/questions/12193779/how-to-write-trycatch-in-r
 475 | 
 476 | ### 1.7.8. <a name="rvest7.8">Handling Network Errors</a>
 477 | 
 478 | You can use `try()` or `tryCatch()` to handle unpredictable network issues.
 479 | 
 480 | You can retry if an error occurs. For more details, you can check the following posts:
 481 | 
 482 | https://stackoverflow.com/questions/20770497/how-to-retry-a-statement-on-error
 483 | 
 484 | ### 1.7.9. <a name="rvest7.9">Downloading Files</a>  
 485 | 
 486 | Most of the times, the purpose of web scrapping is to extract texts from web pages. But there are also scenarios that you want to save images,web pages or other files.
 487 | 
 488 | **Downloading files using `download.file()`**
 489 | 
 490 | `download.file()` is a built-in function in R and it's very easy to use. But it doesn't support extra configurations such as proxies or user agents.
 491 | 
 492 | ```R
 493 | #Download a web page
 494 | my_url <- "https://scrapethissite.com/"
 495 | download.file(url = my_url, # required: provide url 
 496 |               destfile = "test.html", # required: path and name for downloaded file
 497 |               quiet = T, # optional: whether to suppress messages
 498 |               mode = "w") #optional: default to "w", switch to other modes if file is garbled
 499 | 
 500 | #Download a image
 501 | image_url <- "https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/GreenSeaTurtle-2.jpg/200px-GreenSeaTurtle-2.jpg"
 502 | download.file(image_url,"turtle1.jpg",mode = "wb") #mode must be wb(binary) for images
 503 | ```
 504 | 
 505 | 
 506 | 
 507 | **Downloading files using `writeBin()`**
 508 | 
 509 | Another option is to use `writeBin()` function to download files. This function can write the binary content from  the response to a file.
 510 | 
 511 | ```R
 512 | library(httr)
 513 | #use writeBin() to write a html file
 514 | my_url <- "https://scrapethissite.com/"
 515 | response <- GET(my_url)
 516 | writeBin(response$content,"test1.html")
 517 | 
 518 | #downlaod images using `writeBin()`
 519 | image_url <- "https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/GreenSeaTurtle-2.jpg/200px-GreenSeaTurtle-2.jpg"
 520 | response <- GET(image_url)
 521 | writeBin(response$content,"turtle2.jpg")
 522 | ```
 523 | 
 524 | ### 1.7.10. <a name="rvest7.10">Logins and Sessions</a>
 525 | 
 526 | Sometimes you need to log in to access certain content. You can do it using rvest and httr. You can also save your cookies so you don't need to log in every time.
 527 | 
 528 | **Log in using rvest:**
 529 | 
 530 | ```R
 531 | library(rvest)
 532 | url <- "https://example.com/login"
 533 | my_session <- html_session(url) #Create a persistant session
 534 | unfilled_forms <- html_form(my_session) # find all forms in the web page
 535 | login_form <- unfilled_forms[[1]] # select the form you need to fill
 536 | filled_form <- set_values(login_form,username="myusername",password="mypassword") # fill the form
 537 | login_session <- submit_form(my_session,filled_form) # Now you're logged in
 538 | ```
 539 | 
 540 | There is also a built-in [demo](https://github.com/hadley/rvest/blob/master/demo/united.R) for logins in rvest.
 541 | 
 542 | **Using the session**
 543 | 
 544 | `html_session()` simulates a session in an html browser, you can navigate to other pages using the same session with same cookies.
 545 | 
 546 | To stay in the same session, you can use `jump_to()` and `follow_link()`:
 547 | 
 548 | ```R
 549 | library(rvest)
 550 | url <- "www.example.com"
 551 | my_session <- html_session(url) # create a session
 552 | my_session1 <- my_session %>% jump_to("www.example.com/page1") # go to another web page using the same session
 553 | my_session2 <- my_session %>% follow_link(css = "p a")
 554 | ```
 555 | 
 556 | 
 557 | 
 558 | You can also stay in the same session by using cookies: 
 559 | 
 560 | ```R
 561 | library(rvest)
 562 | library(httr)
 563 | url <- "http://ww.example.com"
 564 | my_session <- html_session(url) # create a session
 565 | my_cookies_table <- cookies(my_session) # get the cookies as a table
 566 | my_cookies <- my_cookies_table$value %>% setNames(my_cookies_table$name) # save the cookies as a named vector that you can use in your requests
 567 | new_session <- html_session(url,set_cookies(my_cookies)) # making requests using the same cookies/session
 568 | ```
 569 | 
 570 | ### 1.7.11. <a name="rvest7.11">Web Scraping in Parallel</a>
 571 | 
 572 | To speed up web scraping process, you may want to do it in parallel. To achieve this goal, you can use [`doParallel`](https://cran.r-project.org/web/packages/doParallel/index.html) package or [`Rcrawler`](https://github.com/salimk/Rcrawler) package. The later package has parallel web scraping as a built in feature, it also has many other great features and the developers are adding even more.   For this part, we are focusing on `doParallel`.
 573 | 
 574 | ```R
 575 | library(rvest)
 576 | library(doParallel)
 577 | #assume we have a list of urls
 578 | #urls <- c("url1","url2,...")
 579 | cl <- makeCluster(detectCores()-1) #using max cores - 1 for parallel processing
 580 | registerDoParallel(cl) #register cores
 581 | 
 582 | # use foreach() %dopar%{} to make requests and parese texts in parallel
 583 | result <- foreach(i = seq_along(urls),
 584 |                   .packages = "rvest",
 585 |                   .combine = "c",
 586 |                   .errorhandling='pass') %dopar% {
 587 |                       text <- html_session(urls[i]) %>% 
 588 |                           html_elements("css") %>%
 589 |     					  html_text()
 590 |                       return(text)
 591 |                   }
 592 | ```
 593 | 
 594 | In this [example](resources/rvest_parallel.md), it compares the speed between  `foreach` and `for` loop.
 595 | 
 596 | Also, you need to be aware of that: as you are making more concurrent requests, you are much more likely to get banned. So remember to use proxies or reduce the speed of requests. 
 597 | 
 598 | ****
 599 | 
 600 | # 2. <a name="rselenium">Web Scraping using Rselenium</a>
 601 | 
 602 | ## 2.1. <a name="rselenium1">Why RSelenium</a>
 603 | 
 604 | As mentioned in previous sections, in JavaScript heavy websites, the majority of content could be generated using JavaScript. When using a browser, it will make multiple requests for you while the original page is loading. However, if you use `httr` or `rvest`, only the original page will be loaded and JavaScript won't be executed. Hence some data is only available in browser.
 605 | 
 606 | 
 607 | 
 608 | [RSelenium](https://github.com/ropensci/RSelenium) makes use of [Selenium Web Driver](https://www.seleniumhq.org/projects/webdriver/) which simulates a browser(usually Chrome and Firefox) and renders web pages automatically, it will execute all the JavaScript codes for you so you will get a parsed page source instead of unparsed(raw) one.
 609 | 
 610 | ### 2.1.1 <a name="rselenium1.1">Pros and Cons from Using RSelenium</a>
 611 | 
 612 | **Pros:**
 613 | 
 614 | 1. Renders JavaScript generated content automatically:
 615 |    * One of the greatest headache from web scraping is that what you see in browser and what you get in response are different. By using RSelenium, you can avoid this issue.
 616 | 
 617 | **Cons:**
 618 | 
 619 | 1. It's very slow:
 620 |    * Since the browser loads everything on the web page(including photos, ads or even videos), it will be really slow comparing to making requests using `httr` or `rvest`.
 621 | 2. Lacking support:
 622 |    * Comparing to Selenium in Python, RSelenium doesn't have a large user base and hence lacks support. If you ever searched RSelenium related questions, you might already found out that: many solutions could be obsolete , you couldn't find related topics or the only solutions available were for Python.
 623 | 
 624 | ### 2.1.2 <a name="rselenium1.2">Useful Resources</a>
 625 | 
 626 | 1. [Rselenium Github Page](https://github.com/ropensci/RSelenium): many useful resources are already listed here.
 627 | 2. [http://toscrape.com/](http://toscrape.com/): A sandbox for testing your web scraping script.
 628 | 
 629 | ## 2.2 <a name="rselenium2">Interacting with the Web Driver In Rselenium</a>
 630 | 
 631 | ### 2.2.1. <a name="rselenium2.0">How to Start</a>
 632 | 
 633 | ```R
 634 | # Load the Library
 635 | library(RSelenium)
 636 | 
 637 | # start the server and browser(you can use other browsers here)
 638 | rD <- rsDriver(browser=c("firefox"))
 639 | 
 640 | driver <- rD[["client"]]
 641 | 
 642 | # navigate to an URL
 643 | driver$navigate("http://books.toscrape.com/")
 644 | 
 645 | #close the driver
 646 | driver$close()
 647 | 
 648 | #close the server
 649 | rD[["server"]]$stop()
 650 | ```
 651 | 
 652 | **Common Issues and Solutions when starting the server and browser** 
 653 | 
 654 | 1. Port already in use.
 655 |     - https://stackoverflow.com/questions/43991498/rselenium-server-signals-port-is-already-in-use.
 656 |     - https://stackoverflow.com/questions/53464640/launch-rselenium-browser-regardless-of-port-open-closed
 657 |     - Something the Java process behind RSelenium does not stop when you apply `rD[["server"]]$stop()`, thereby to ensure shutting down a RSelenium instance you may write:
 658 |     ```r
 659 |     # Kill Java process behind RSelenium
 660 |     system(paste0("Taskkill /F /T" ," /PID ", driver$server$process$get_pid()))
 661 |     ```
 662 |     - Try restart Rstudio.
 663 | 2. session not created: This version of ChromeDriver only supports Chrome verssion XX.
 664 |     - [Specify chrome version or install latest chrome driver](https://stackoverflow.com/questions/55201226/session-not-created-this-version-of-chromedriver-only-supports-chrome-version-7).
 665 | 3. Other issues:
 666 |     - Try reinstall/update RSelenium or wdman package.
 667 | 
 668 | 
 669 | ### 2.2.2. <a name = "rselenium2.1">Navigating to different URLs</a>
 670 | 
 671 | **Using `driver.navigate()`:**
 672 | 
 673 | ```
 674 | driver$navigate("http://books.toscrape.com/")
 675 | ```
 676 | 
 677 | **Simulating a click on the link:**
 678 | 
 679 | ```R
 680 | # navigate to an URL
 681 | driver$navigate("http://toscrape.com/")
 682 | 
 683 | # find the element
 684 | elements <- driver$findElements("a",using = "css")
 685 | 
 686 | # click the first link 
 687 | elements[[1]]$clickElement()
 688 | ```
 689 | 
 690 | **Go Back or Go Forward**
 691 | 
 692 | ```R
 693 | driver$goBack()
 694 | driver$goForward()
 695 | ```
 696 | 
 697 | ### 2.2.3. <a name = "rselenium2.2">Simulating Scrolls, Clicks, Text Inputs, Logins, and Other Actions</a>
 698 | 
 699 | **1. Simulating Scrolls:**
 700 | 
 701 | You can scroll up or down by sending the `pageUp` or `pageDown` keypress to the browser. 
 702 | 
 703 | *Scroll Down Once:*
 704 | 
 705 | ```R
 706 | driver$navigate("http://quotes.toscrape.com/scroll")
 707 | # find the webpage body
 708 | element <- driver$findElement("css", "body")
 709 | 
 710 | #scroll down once ----
 711 | element$sendKeysToElement(list(key = "page_down"))
 712 | ```
 713 | 
 714 | *Scroll Down Multiple Times:*
 715 | 
 716 | ```R
 717 | element <- driver$findElement("css", "body")
 718 | # Scroll down 10 times
 719 | for(i in 1:10){
 720 |     element$sendKeysToElement(list("key"="page_down"))
 721 |     # please make sure to sleep a couple of seconds to since it takes time to load contents
 722 |     Sys.sleep(2) 
 723 | }
 724 | ```
 725 | 
 726 | *Scroll Down Until the End(Not Recommended if There Are too Many Pages):*
 727 | 
 728 | ```r
 729 | element <- driver$findElement("css", "body")
 730 | flag <- TRUE
 731 | counter <- 0
 732 | n <- 5
 733 | while(flag){
 734 |     counter <- counter + 1
 735 |     #compare the pagesource every n(n=5) time, since sometimes one scroll down doesn't render new content
 736 |     for(i in 1:n){
 737 |         element$sendKeysToElement(list("key"="page_down"))
 738 |         Sys.sleep(2)
 739 |     }
 740 |     if(exists("pagesource")){
 741 |         if(pagesource == driver$getPageSource()[[1]]){
 742 |             flag <- FALSE
 743 |             writeLines(paste0("Scrolled down ",n*counter," times.\n"))
 744 |         } else {
 745 |             pagesource <- driver$getPageSource()[[1]]
 746 |         }
 747 |     } else {
 748 |         pagesource <- driver$getPageSource()[[1]]
 749 |     }
 750 | }
 751 | ```
 752 | 
 753 | 
 754 | 
 755 | **2. Simulating Clicks:**
 756 | 
 757 | *Click a Single Element by CSS:*
 758 | 
 759 | ```R
 760 | driver$navigate("http://quotes.toscrape.com/scroll")
 761 | 
 762 | #locate element using CSS(find the first match)
 763 | driver$navigate("https://scrapethissite.com/pages/ajax-javascript/#2011")
 764 | element <- driver$findElement(using = "css",".year-link")
 765 | element$clickElement()
 766 | ```
 767 | 
 768 | *Click a Single Element by Moving Mouse to Element:*
 769 | 
 770 | ```R
 771 | driver$navigate("https://scrapethissite.com/pages/ajax-javascript/#2011")
 772 | element <- driver$findElement(using = "css",".year-link")
 773 | driver$mouseMoveToLocation(webElement = element)
 774 | driver$click()
 775 | ```
 776 | 
 777 | *Click Multiple Items:*
 778 | 
 779 | ```R
 780 | driver$navigate("https://scrapethissite.com/pages/ajax-javascript/#2011")
 781 | elements <- driver$findElements(using = "css",".year-link")
 782 | for(element in elements){
 783 |     element$clickElement()
 784 |     Sys.sleep(2)
 785 | }
 786 | ```
 787 | 
 788 | *Go to Different URLs by Clicking Links:*
 789 | 
 790 | ```R
 791 | driver$navigate("http://books.toscrape.com/")
 792 | elements <- driver$findElements(using = "css",".nav-list ul a")
 793 | for(i in 1:length(elements)){
 794 |     elements[[i]]$clickElement()
 795 |     Sys.sleep(2)
 796 |     #do something and go back to previous page
 797 |     driver$goBack()
 798 |     Sys.sleep(1)
 799 |     # refresh the elements
 800 |     elements <- driver$findElements(using = "css",".nav-list ul a")
 801 | }
 802 | ```
 803 | 
 804 | **3. Simulating Text Input:**
 805 | 
 806 | *Enter Text and Search*
 807 | 
 808 | ```R
 809 | driver$navigate("https://www.google.com/")
 810 | #selcet input box
 811 | element <- driver$findElement(using = "css",'input[name="q"]')
 812 | #send text to input box. don't forget to use `list()` when sending text
 813 | element$sendKeysToElement(list("Web Scraping"))
 814 | #select search button
 815 | element <- driver$findElement(using = "css",'input[name="btnK"]')
 816 | element$clickElement()
 817 | ```
 818 | 
 819 | 
 820 | 
 821 | *Clear Input Box*
 822 | 
 823 | ```R
 824 | driver$navigate("https://www.google.com/")
 825 | #selcet input box
 826 | element <- driver$findElement(using = "css",'input[name="q"]')
 827 | element$sendKeysToElement(list("Web Scraping"))
 828 | #clear input box
 829 | element$clearElement()
 830 | ```
 831 | 
 832 | 
 833 | 
 834 | **4. Logins:**
 835 | 
 836 | Login is simple using RSelenium. Instead of doing post request, it's just a combination of sending texts to input boxes and click login button.
 837 | 
 838 | 
 839 | 
 840 | ```R
 841 | driver$navigate("http://quotes.toscrape.com/login")
 842 | #enter username
 843 | element <- driver$findElement(using = "css","#username")
 844 | element$sendKeysToElement(list("myusername"))
 845 | #enter password
 846 | element <- driver$findElement(using = "css","#password")
 847 | element$sendKeysToElement(list("mypassword"))
 848 | #click login button
 849 | element <- driver$findElement(using = "css", 'input[type="submit"]')
 850 | element$clickElement()
 851 | ```
 852 | 
 853 | 
 854 | 
 855 | 
 856 | 
 857 | **5. Simulating Key and Button Presses:**
 858 | 
 859 | You can also send keys to the browser, you can check all available keys by running `RSelenium::selKeys`.
 860 | 
 861 | *Available Keys:*
 862 | 
 863 | ```R
 864 |  [1] "null"         "cancel"       "help"         "backspace"    "tab"          "clear"        "return"       "enter"        "shift"        "control"     
 865 | [11] "alt"          "pause"        "escape"       "space"        "page_up"      "page_down"    "end"          "home"         "left_arrow"   "up_arrow"    
 866 | [21] "right_arrow"  "down_arrow"   "insert"       "delete"       "semicolon"    "equals"       "numpad_0"     "numpad_1"     "numpad_2"     "numpad_3"    
 867 | [31] "numpad_4"     "numpad_5"     "numpad_6"     "numpad_7"     "numpad_8"     "numpad_9"     "multiply"     "add"          "separator"    "subtract"    
 868 | [41] "decimal"      "divide"       "f1"           "f2"           "f3"           "f4"           "f5"           "f6"           "f7"           "f8"          
 869 | [51] "f9"           "f10"          "f11"          "f12"          "command_meta"
 870 | ```
 871 | 
 872 | 
 873 | 
 874 | *Send Key Combinations:*
 875 | 
 876 | ```R
 877 | driver$navigate("https://scrapethissite.com/pages/ajax-javascript/#2015")
 878 | Sys.sleep(2)
 879 | # Press Control+A to select the entire page
 880 | driver$findElement(using = "css","body")$sendKeysToElement(list(key="control","a"))
 881 | 
 882 | ```
 883 | 
 884 | 
 885 | 
 886 | 
 887 | ## 2.3. <a name="rselenium3">Extracting Texts from Web Pages</a>
 888 | 
 889 | ### 2.3.1. <a name="rselenium3.1">Extracting Content using Rselenium</a>
 890 | 
 891 | You can use `findElement()` or `findElements()` to extract single or multiple elements from page source.
 892 | 
 893 | **1.  Extract text from a single element:**
 894 | 
 895 | use `findElment()` method to select a single matching element, and use `getElementText()` method to extract text.
 896 | 
 897 | ```R
 898 | driver$navigate("https://scrapethissite.com/pages/simple/")
 899 | element <- driver$findElement(using = "css",".country-capital")
 900 | # get element text
 901 | element$getElementText()[[1]]
 902 | #get element attribute value(get the value for class)
 903 | element$getElementAttribute("class")[[1]]
 904 | ```
 905 | 
 906 | 
 907 | 
 908 | **2. Extract text from a list of elements:**
 909 | 
 910 | `RSelenium` doesn't support vectorized calculation.So you need to use for loops, apply or map(in `purrr` package) as alternative to get lists of items.
 911 | 
 912 | use `findElements()` method to select all matching elements, and use `getElementText()` method to extract text.
 913 | 
 914 | ```R
 915 | driver$navigate("https://scrapethissite.com/pages/simple/")
 916 | Sys.sleep(2)
 917 | elements <- driver$findElements(using = "css",".country-capital")
 918 | ```
 919 | 
 920 | 
 921 | 
 922 | *Use for loop to extract elements:*
 923 | 
 924 | ```R
 925 | texts <- c()
 926 | for(ele in elements){
 927 |     texts <- append(texts,ele$getElementText()[[1]])
 928 | }
 929 | ```
 930 | 
 931 | 
 932 | 
 933 | *Use apply functions to extract elements:*
 934 | 
 935 | ```R
 936 | # use lapply
 937 | texts_lapply <- lapply(elements,function(x) x$getElementText()[[1]])
 938 | # use sapply
 939 | texts_sapply <- sapply(elements,function(x) x$getElementText()[[1]])
 940 | ```
 941 | 
 942 | 
 943 | 
 944 | *Use map functions(using `purrr`) to extract elements:*
 945 | 
 946 | ```R
 947 | library(purrr)
 948 | # get result as a list
 949 | texts_purrr_map <- map(elements,~ .x$getElementText()[[1]])
 950 | # get result as a string vector
 951 | texts_purrr_map_chr <- map_chr(elements,~ .x$getElementText()[[1]])
 952 | ```
 953 | 
 954 |  
 955 | 
 956 | **3. Extract attribute(s) from a single element(s)**
 957 | 
 958 | The usage is the same as extracting text from element(s), the only difference it that you need to use `getElementAttribute()` method.
 959 | 
 960 | ```R
 961 | book_partial_links <- lapply(elements,function(x) x$getElementAttribute(attrName="href"))
 962 | ```
 963 | 
 964 | 
 965 | 
 966 | 
 967 | 
 968 | ### 2.3.2. <a name="rselenium3.2">Extracting Content using Parsed Page Source and `rvest`</a>
 969 | 
 970 | Since the browser will execute JavaScript codes by default, you can use `RSelenium` as a tool to get the parsed content from webpages. In this way, you can benefit from the vectorized calculation in `rvest`. 
 971 | 
 972 | Another benefit of using `rvest` is that it runs much faster while extracting information from page source comparing to using `getElementText()` or `getElementAttribute()` method.
 973 | 
 974 | ```R
 975 | library(rvest)
 976 | driver$navigate("https://scrapethissite.com/pages/ajax-javascript/#2015")
 977 | Sys.sleep(2)
 978 | # get parsed page source
 979 | parsed_pagesource <- driver$getPageSource()[[1]]
 980 | # using rvest to extract information
 981 | result <- read_html(parsed_pagesource) %>%
 982 |     html_elements(".film-title") %>%
 983 |     html_text()
 984 | ```
 985 | 
 986 | 
 987 | 
 988 | ## 2.4. <a name = "rselneium4">miscellanea</a>
 989 | 
 990 | ### 2.4.1. <a name="rselenium4.1">Javascript</a>
 991 | 
 992 | Congratulations, no more concerns about AJAX or JavaScript as the browser will parse everything for you!
 993 | 
 994 | But, of course, if you still need to run some javascript, then you can use `your_driver$executeScript()`
 995 | 
 996 | ### 2.4.2. <a name="rselenium4.2">Iframe</a>
 997 | 
 998 | Let's try to get all turtles family name in this [webpage](https://scrapethissite.com/pages/frames/), it doesn't work because the content is another webpage nested in the current one.
 999 | 
1000 | ```R
1001 | driver$navigate("https://scrapethissite.com/pages/frames/")
1002 | # select all turtles' name
1003 | elements <- driver$findElements(using = "css",".family-name")
1004 | print(elements)
1005 | > list()
1006 | ```
1007 | 
1008 | 
1009 | 
1010 | To read the content from inside iframe, we can use `switchToFrame` method.
1011 | 
1012 | > switchToFrame(Id)
1013 | > Change focus to another frame on the page. Id can be string|number|null|WebElement Object. If the Id is null, the server should switch to the page's default content.
1014 | 
1015 | ```R
1016 | driver$navigate("https://scrapethissite.com/pages/frames/")
1017 | # select iframe element
1018 | element <- driver$findElement(using = "css","#iframe")
1019 | #switch to the iframe
1020 | driver$switchToFrame(element)
1021 | 
1022 | elements <- driver$findElements(using = "css",".family-name")
1023 | for (ele in elements){
1024 |     print(ele$getElementText()[[1]])
1025 | }
1026 | #switch back to the default page
1027 | driver$switchToFrame()
1028 | ```
1029 | 
1030 | # 3 <a name = "change">Change Log</a>
1031 | 
1032 | - 2021-10-25: `html_node` changed to `html_element`, `html_nodes` changed to `html_elements`.
1033 |     - Thanks [@victorcaquilpan](https://github.com/victorcaquilpan) for this suggestion.
1034 |     - The reason for this chagne is mentioned in [rvest 1.0.0](https://www.tidyverse.org/blog/2021/03/rvest-1-0-0/)
1035 | 


--------------------------------------------------------------------------------
/example_script/README.md:
--------------------------------------------------------------------------------
 1 | # Exmple Scripts
 2 | 
 3 | This folder contains example scripts used in the cheatsheet and scripts provided by other contributors.
 4 | 
 5 | 
 6 | 
 7 | 
 8 | 
 9 | -------------------------------
10 | 
11 | I've been working on wer-scraping with R for a while. I'm active on [stackoverflow](https://stackoverflow.com/users/6037369/yifu-yan) and answered a bunch of web-scraping questions regarding on rvest and Rselenium. 
12 | 
13 | You can check for [my answers](https://stackoverflow.com/users/6037369/yifu-yan?tab=answers) and see if any solved your problem. Also, this folder contains some web-scraping scripts I've used.
14 | 
15 | 
16 | 


--------------------------------------------------------------------------------
/example_script/download_pictures_from_google.R:
--------------------------------------------------------------------------------
 1 | #This function is not vectorized, use for loop for multiple key words
 2 | #keyword_search: keyword
 3 | #image_num: return how many pictures, may return less than this number.
 4 | #scroll_times: how many times should scroll down,default 1
 5 | #css_searchbox: css selector for searchbox
 6 | #css_clicksearch: css selector for search button
 7 | #css_img_src: css selector for image source
 8 | #try_times: how many times to try to get missed images
 9 | download_google_img <- function(keyword_search,image_num = 50,scroll_times = 1,
10 |                                 css_searchbox = "#lst-ib",css_clicksearch = "#_fZl",css_img_src = ".rg_i",
11 |                                 try_times = 1){
12 |     #download pictures from google
13 |     #chenck if all packages installed
14 |     required_packages <- c("rvest","RSelenium","seleniumPipes","dplyr","stringr","magick")
15 |     for (packge in required_packages) {
16 |         if (packge %in% rownames(installed.packages())) {
17 |             require(packge, character.only = TRUE)
18 |         } else {
19 |             install.packages(packge)
20 |         }
21 |     }
22 |     #create tmp_img folder if doesn't exist, create one
23 |     if(!file.exists("tmp_img")){
24 |         dir.create("tmp_img")
25 |     }
26 |     #start server-----
27 |     rD <- rsDriver (browser = 'chrome',chromever = "latest",port = 4445L)
28 |     #open browser
29 |     remDr <- remoteDr(browserName = "chrome",port = 4445L)
30 |     #use google image search
31 |     web_url <- "https://www.google.com/search?q=fd&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjto6G9pKTZAhVdVWMKHZHWAr8Q_AUIDSgE&biw=1920&bih=900"
32 |     #go to url
33 |     remDr %>% go(web_url)
34 |     
35 |     #clear search box and clear it ------------
36 |     remDr %>% findElement(using = "css selector",value = css_searchbox) %>% elementClear() %>% elementSendKeys(keyword_search)
37 |     #click to search pictures
38 |     remDr %>% findElement(using = "css selector",value = css_clicksearch) %>% elementClick()
39 |     #Optional---------------
40 |     #scroll down to get more images
41 |     if(scroll_times == 0){
42 |         
43 |     } else {
44 |         for (i in 1:scroll_times){
45 |             remDr %>% executeScript("window.scrollTo(0,document.body.scrollHeight);")
46 |         }
47 |     }
48 |     #optional ends-----------------------
49 |     #after scrolling, wait a certain time for pictures to load
50 |     Sys.sleep(scroll_times*5)
51 |     #get picture source urls
52 |     img_href <- remDr %>% getPageSource() %>% html_elements(css_img_src) %>% html_attr("src")
53 |     #remove NAs in img_href
54 |     img_href <- img_href[!is.na(img_href)]
55 |     #how many images to return?
56 |     r_img_num <- min(image_num,length(img_href))
57 |     img_href <- img_href[1:r_img_num]
58 |     #save image one by one
59 |     missed_imgs <- numeric(0)
60 |     for(i in seq_along(img_href)){
61 |         tryCatch({
62 |             remDr %>% go(img_href[i])
63 |             #take screenshot
64 |             temp_img <- remDr %>% takeScreenshot(returnPNG = TRUE)
65 |             #render image
66 |             cropped_img <- image_trim(image_read(temp_img))
67 |             #save image in tmp_img file
68 |             image_write(cropped_img,path = str_c("tmp_img/",keyword_search,"_",i,".png"),format = "png")
69 |             message("process:",i,"/",r_img_num)
70 |             Sys.sleep(round(runif(1,min = 2, max = 4)))
71 |         },
72 |         error = function(x){
73 |             x <- i
74 |             missed_imgs <<- append(missed_imgs,x)
75 |             message(keyword_search," image ",x," missing")
76 |             Sys.sleep(10)
77 |             remDr <- remoteDr(browserName = "chrome",port = 4445L)
78 |             message("server restarted")
79 |         })
80 |         
81 |     }
82 |     remDr %>% deleteSession()
83 |     rD[["server"]]$stop() 
84 | }
85 | 
86 | 
87 | 
88 | 
89 | 
90 | 


--------------------------------------------------------------------------------
/example_script/iframe_tutorial.Rmd:
--------------------------------------------------------------------------------
 1 | ---
 2 | title: "Iframe Tutorial: Extract Information Inside Iframes"
 3 | author: "yusuzech"
 4 | date: "July 30, 2018"
 5 | output: github_document
 6 | ---
 7 | 
 8 | ```{r setup, include=FALSE}
 9 | knitr::opts_chunk$set(echo = TRUE)
10 | library(rvest)
11 | ```
12 | 
13 | In this tutorial, we are going to use this website: https://scrapethissite.com/pages/frames/ to extract information inside iframes.
14 | 
15 | ## 1 Normal method should fail
16 | 
17 | At first, let's extract each turtle's name as shown in the image below:
18 | 
19 | ![pic1](resources/turtle0.jpg)
20 | 
21 | Let's try the following codes:
22 | ```{r}
23 | library(rvest)
24 | my_url <- "https://scrapethissite.com/pages/frames/"
25 | turtle_names <- html_session(my_url) %>%
26 |     html_elements(".family-name") %>%
27 |     html_text()
28 | print(turtle_names)
29 | ```
30 | 
31 | It failed and returned nothing. The reason is that those information are actually from another HTML file and is imbedded in the current page you are reading. So you can't extract anything from current website.
32 | 
33 | 
34 | ## 2 Find the iframe
35 | 
36 | To extract turtles' name, we need to find the link to the iframe. Let's use Chrome Developer Tool to find the linke to iframe:  
37 | We can use ctrl+F and search for keyword "iframe" and here we find the link.  
38 | ![pic2](resources/turtle1.jpg)
39 | 
40 | Let's do this in R and this time it should succeed:
41 | 
42 | ```{r}
43 | library(rvest)
44 | library(stringr)
45 | my_url <- "https://scrapethissite.com/pages/frames/"
46 | #extract source as hown in the image above
47 | iframe_src <-  html_session(my_url) %>%
48 |     html_element("#iframe") %>%
49 |     html_attr("src")
50 | #get the url to that iframe
51 | iframe_url <- str_c("https://scrapethissite.com",iframe_src)
52 | #extract turtle names:
53 | turtle_names <- html_session(iframe_url) %>%
54 |     html_elements(".family-name") %>%
55 |     html_text()
56 | print(turtle_names)
57 | ```
58 | 
59 | 


--------------------------------------------------------------------------------
/example_script/iframe_tutorial.md:
--------------------------------------------------------------------------------
 1 | Iframe Tutorial: Extract Information Inside Iframes
 2 | ================
 3 | yusuzech
 4 | July 30, 2018
 5 | 
 6 | In this tutorial, we are going to use this website: <https://scrapethissite.com/pages/frames/> to extract information inside iframes.
 7 | 
 8 | 1 Normal method should fail
 9 | ---------------------------
10 | 
11 | At first, let's extract each turtle's name as shown in the image below:
12 | 
13 | ![pic1](resources/turtle0.jpg)
14 | 
15 | Let's try the following codes:
16 | 
17 | ``` r
18 | library(rvest)
19 | my_url <- "https://scrapethissite.com/pages/frames/"
20 | turtle_names <- html_session(my_url) %>%
21 |     html_elements(".family-name") %>%
22 |     html_text()
23 | print(turtle_names)
24 | ```
25 | 
26 |     ## character(0)
27 | 
28 | It failed and returned nothing. The reason is that those information are actually from another HTML file and is imbedded in the current page you are reading. So you can't extract anything from current website.
29 | 
30 | 2 Find the iframe
31 | -----------------
32 | 
33 | To extract turtles' name, we need to find the link to the iframe. Let's use Chrome Developer Tool to find the linke to iframe:
34 | We can use ctrl+F and search for keyword "iframe" and here we find the link.
35 | ![pic2](resources/turtle1.jpg)
36 | 
37 | Let's do this in R and this time it should succeed:
38 | 
39 | ``` r
40 | library(rvest)
41 | library(stringr)
42 | my_url <- "https://scrapethissite.com/pages/frames/"
43 | #extract source as hown in the image above
44 | iframe_src <-  html_session(my_url) %>%
45 |     html_element("#iframe") %>%
46 |     html_attr("src")
47 | #get the url to that iframe
48 | iframe_url <- str_c("https://scrapethissite.com",iframe_src)
49 | #extract turtle names:
50 | turtle_names <- html_session(iframe_url) %>%
51 |     html_elements(".family-name") %>%
52 |     html_text()
53 | print(turtle_names)
54 | ```
55 | 
56 |     ##  [1] "Carettochelyidae" "Cheloniidae"      "Chelydridae"     
57 |     ##  [4] "Dermatemydidae"   "Dermochelyidae"   "Emydidae"        
58 |     ##  [7] "Geoemydidae"      "Kinosternidae"    "Platysternidae"  
59 |     ## [10] "Testudinidae"     "Trionychidae"     "Chelidae"        
60 |     ## [13] "Pelomedusidae"    "Podocnemididae"
61 | 


--------------------------------------------------------------------------------
/example_script/resources/README.md:
--------------------------------------------------------------------------------
1 | resources used by markdown files.
2 | 


--------------------------------------------------------------------------------
/example_script/resources/turtle0.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yusuzech/r-web-scraping-cheat-sheet/58cb9bce6f749698c75ca86fa0e6cfc1f8d39e17/example_script/resources/turtle0.jpg


--------------------------------------------------------------------------------
/example_script/resources/turtle1.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yusuzech/r-web-scraping-cheat-sheet/58cb9bce6f749698c75ca86fa0e6cfc1f8d39e17/example_script/resources/turtle1.jpg


--------------------------------------------------------------------------------
/example_script/resources/turtle2.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yusuzech/r-web-scraping-cheat-sheet/58cb9bce6f749698c75ca86fa0e6cfc1f8d39e17/example_script/resources/turtle2.jpg


--------------------------------------------------------------------------------
/resources/.gitkeep:
--------------------------------------------------------------------------------
1 | 
2 | 


--------------------------------------------------------------------------------
/resources/functions_and_classes.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yusuzech/r-web-scraping-cheat-sheet/58cb9bce6f749698c75ca86fa0e6cfc1f8d39e17/resources/functions_and_classes.png


--------------------------------------------------------------------------------
/resources/placeholder.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yusuzech/r-web-scraping-cheat-sheet/58cb9bce6f749698c75ca86fa0e6cfc1f8d39e17/resources/placeholder.jpg


--------------------------------------------------------------------------------
/resources/rvest_parallel.md:
--------------------------------------------------------------------------------
 1 | # Speed Comparison between `for` and `foreach` loop
 2 | 
 3 | 
 4 | 
 5 | ## Prepare the data
 6 | 
 7 | Let's get a list of URLs to start with:
 8 | 
 9 | ```R
10 | library(rvest)
11 | library(stringr)
12 | library(doParallel)
13 | url_wiki <- "https://en.wikipedia.org/wiki/List_of_most_popular_websites"
14 | my_css <- "td:nth-child(2)"
15 | wikipedia <- html_session(url_wiki)
16 | 
17 | #get top 20 websites
18 | urls <- wikipedia %>%
19 |     html_elements(my_css) %>%
20 |     html_text() %>%
21 |     str_trim() %>%
22 |     .[1:20]
23 | ```
24 | 
25 | 
26 | 
27 | Let's display top twenty websites:
28 | 
29 | ```R
30 | urls
31 | ```
32 | 
33 | 
34 | 
35 | The result is:
36 | 
37 | ```R
38 |  [1] "google.com"    "youtube.com"   "facebook.com"  "baidu.com"     "wikipedia.org" "reddit.com"    "yahoo.com"     "qq.com"        "taobao.com"    "google.co.in" 
39 | [11] "amazon.com"    "tmall.com"     "twitter.com"   "sohu.com"      "instagram.com" "vk.com"        "live.com"      "jd.com"        "sina.com.cn"   "weibo.com"   
40 | ```
41 | 
42 | 
43 | 
44 | ## Making requests and compare speed
45 | 
46 | To compare the speed, we will extract title from each website.
47 | 
48 | **Using `foreach` loop for web scraping in parallel**
49 | 
50 | ```R
51 | cl <- makeCluster(detectCores()-1)
52 | registerDoParallel(cl)
53 | t1 <- Sys.time()
54 | result <- foreach(i = seq_along(urls),
55 |                   .packages = "rvest",
56 |                   .combine = "c",
57 |                   .errorhandling='pass') %dopar% {
58 |                       # get the header for each page
59 |                       title <- html_session(urls[i]) %>% 
60 |                           html_element("head") %>% 
61 |                           html_element("title") %>% 
62 |                           html_text()
63 |                       return(list(title))
64 |                   }
65 | t2 <- Sys.time()
66 | t_parallel <- t2 - t1
67 | ```
68 | 
69 | **Using `for` loop**
70 | 
71 | ```R
72 | result2 <- list()
73 | t1 <- Sys.time()
74 | for(url in urls){
75 |     title <- try(html_session(url) %>% 
76 |         html_element("head") %>% 
77 |         html_element("title") %>% 
78 |         html_text())
79 |     result2 <- append(result2,list(title))
80 | }
81 | t2 <- Sys.time()
82 | t_for <- t2 - t1
83 | ```
84 | 
85 | 
86 | 
87 | **Difference in speed between parallel and non-parallel**
88 | 
89 | ````R
90 | > data.frame(t_parallel,t_for)
91 |      t_parallel         t_for
92 | 1 5.357209 secs 20.48356 secs
93 | ````
94 | 
95 | 
96 | 
97 | In this example, `foreach` loop is about 4 times faster than `for` loop. The exact speed will differ depending on number of cores on your machine and network connection. But by using multiple cores, you can definitely speed up your requests.
98 | 


--------------------------------------------------------------------------------