├── book.bib ├── secret.png ├── 404.Rmd ├── Makefile ├── .gitignore ├── style.css ├── .github ├── issue_template.md ├── pull_request_template.md ├── CONTRIBUTING.md └── workflows │ └── bookdown.yml ├── 10-record-modes.Rmd ├── 12-logging.Rmd ├── 09-configuration.Rmd ├── 11-request-matching.Rmd ├── 14-escape-hatches.Rmd ├── http-testing.Rproj ├── CITATION.cff ├── _output.yml ├── matomo.html ├── 06-webmockr-utilities.Rmd ├── DESCRIPTION ├── preamble.tex ├── 05-webmockr-testing.Rmd ├── README.md ├── 18-session-info.Rmd ├── _bookdown.yml ├── packages.bib ├── CODE_OF_CONDUCT.md ├── 07-vcr.Rmd ├── topics-errors.Rmd ├── conclusion.Rmd ├── 15-cassettes.Rmd ├── 08-vcr-usage.Rmd ├── index.Rmd ├── 16-gotchas.Rmd ├── 13-security.Rmd ├── topics-real.Rmd ├── topics-contributing.Rmd ├── topics-cran.Rmd ├── 03-mocking.Rmd ├── 04-webmockr-stubs.Rmd ├── intro-general.Rmd ├── intro-graceful.Rmd ├── intro-pkgs.Rmd ├── LICENSE ├── wholegames-comparison.Rmd ├── wholegames-mocking.Rmd ├── wholegames-presser.Rmd ├── wholegames-httptest.Rmd ├── wholegames-httptest2.Rmd ├── topics-security.Rmd ├── wholegames-intro.Rmd └── wholegames-vcr.Rmd /book.bib: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /secret.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ropensci-books/http-testing/HEAD/secret.png -------------------------------------------------------------------------------- /404.Rmd: -------------------------------------------------------------------------------- 1 | Page not found. Use the table of contents or the search bar to find your way back. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | render: 2 | Rscript -e "bookdown::render_book('index.Rmd')" 3 | 4 | serve: 5 | Rscript -e "bookdown::serve_book(in_session = FALSE)" 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .Rproj.user 2 | .Rhistory 3 | .RData 4 | _publish.R 5 | _book 6 | _bookdown_files 7 | watch.R 8 | vcr.log 9 | docs 10 | rmd-fragments 11 | -------------------------------------------------------------------------------- /style.css: -------------------------------------------------------------------------------- 1 | /* bootstrap.min.css | file:///home/maelle/Documents/ropensci/http-testing/docs/libs/bootstrap-4.5.2/bootstrap.min.css */ 2 | 3 | .alert-info { 4 | /* color: #0c5460; */ 5 | color: #212529; 6 | } 7 | -------------------------------------------------------------------------------- /.github/issue_template.md: -------------------------------------------------------------------------------- 1 | 5 | -------------------------------------------------------------------------------- /10-record-modes.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Record modes {#record-modes} 11 | 12 | https://docs.ropensci.org/vcr/articles/debugging.html -------------------------------------------------------------------------------- /12-logging.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Debugging your tests that use vcr 11 | 12 | https://docs.ropensci.org/vcr/articles/debugging.html -------------------------------------------------------------------------------- /09-configuration.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # Configure vcr {#vcr-configuration} 10 | 11 | https://docs.ropensci.org/vcr/reference/vcr_configure.html -------------------------------------------------------------------------------- /11-request-matching.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Request matching {#request-matching} 11 | 12 | https://docs.ropensci.org/vcr/articles/debugging.html -------------------------------------------------------------------------------- /14-escape-hatches.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # Turning vcr on and off {#lightswitch} 10 | 11 | https://docs.ropensci.org/vcr/reference/lightswitch.html -------------------------------------------------------------------------------- /http-testing.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | 15 | BuildType: Makefile 16 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Description 4 | 5 | 6 | ## Related Issue 7 | 10 | 11 | -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | message: "If you use this book, please cite it as below." 3 | authors: 4 | - family-names: Chamberlain 5 | given-names: Scott 6 | orcid: https://orcid.org/0000-0003-1444-9135 7 | - family-names: Salmon 8 | given-names: Maëlle 9 | orcid: https://orcid.org/0000-0002-2815-0399 10 | title: HTTP testing in R 11 | version: 1.0.4 12 | date-released: 2024-02-02 13 | 14 | -------------------------------------------------------------------------------- /_output.yml: -------------------------------------------------------------------------------- 1 | bookdown::bs4_book: 2 | repo: 3 | base: https://github.com/ropensci-books/http-testing 4 | branch: main 5 | theme: 6 | primary: "#1f58a3" 7 | yellow: "#804600" 8 | h4-font-size: "1.1rem" 9 | includes: 10 | in_header: matomo.html 11 | css: style.css 12 | 13 | bookdown::pdf_book: 14 | includes: 15 | in_header: preamble.tex 16 | latex_engine: xelatex 17 | citation_package: natbib 18 | keep_tex: yes 19 | bookdown::epub_book: default 20 | -------------------------------------------------------------------------------- /matomo.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /06-webmockr-utilities.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # utilities {#webmockr-utilities} 10 | 11 | ```{r} 12 | library("webmockr") 13 | ``` 14 | 15 | ## Managing stubs {#webmockr-} 16 | 17 | - `enable()` 18 | - `enabled()` 19 | - `disable()` 20 | - `httr_mock()` 21 | 22 | ## Managing stubs {#webmockr-managing-stubs} 23 | 24 | - `stub_registry()` 25 | - `stub_registry_clear()` 26 | - `remove_request_stub()` 27 | 28 | ## Managing requests {#webmockr-managing-requests} 29 | 30 | - `request_registry()` 31 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Type: Book 2 | Package: httptestingbook 3 | Title: HTTP testing with R packages. 4 | Version: 0.3 5 | Authors@R: 6 | person(c("Scott", "myrmecocystus@gmail.com"), "Chamberlain", , c("aut", "cre")) 7 | URL: https://github.com/ropensci-books/http-testing 8 | BugReports: https://github.com/ropensci-books/http-testing/issues 9 | Depends: 10 | R (>= 3.4.3) 11 | Imports: 12 | attachment, 13 | bookdown (>= 0.22.15), 14 | crul, 15 | fs, 16 | gh, 17 | httptest, 18 | httptest2, 19 | httr, 20 | httr2, 21 | purrr, 22 | rhub, 23 | sessioninfo, 24 | testthat, 25 | urltools, 26 | usethis, 27 | vcr, 28 | webfakes, 29 | webmockr, 30 | xml2 31 | Suggests: 32 | bslib, 33 | downlit, 34 | sass, 35 | tinytex (>= 0.31) 36 | -------------------------------------------------------------------------------- /preamble.tex: -------------------------------------------------------------------------------- 1 | 2 | \usepackage{booktabs} 3 | \usepackage{mdframed} 4 | \usepackage{xcolor} 5 | \usepackage{hyperref} 6 | \usepackage[default]{sourcesanspro} 7 | \definecolor{roblue}{HTML}{6FAEF5} 8 | \definecolor{rolink}{HTML}{1F58A3} 9 | \hypersetup 10 | {colorlinks=true, 11 | linkcolor=rolink, 12 | urlcolor=rolink, 13 | filecolor=rolink, 14 | citecolor=rolink, 15 | allcolors=rolink 16 | } 17 | \renewcommand{\linethickness}{0.05em} 18 | \makeatletter 19 | \def\thm@space@setup{% 20 | \thm@preskip=8pt plus 2pt minus 4pt 21 | \thm@postskip=\thm@preskip 22 | } 23 | \makeatother 24 | \newenvironment{alert alert-dismissible alert-success} 25 | {\begin{mdframed}[linecolor=roblue,linewidth=2pt]} 26 | {\end{mdframed}} 27 | 28 | \newenvironment{alert} 29 | {\begin{mdframed}[linecolor=roblue,linewidth=2pt]} 30 | {\end{mdframed}} -------------------------------------------------------------------------------- /05-webmockr-testing.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # testing {#webmockr-testing} 10 | 11 | ```{r} 12 | library("webmockr") 13 | library("crul") 14 | library("testthat") 15 | 16 | stub_registry_clear() 17 | 18 | # make a stub 19 | stub_request("get", "https://httpbin.org/get") %>% 20 | to_return(body = "success!", status = 200) 21 | 22 | # check that it's in the stub registry 23 | stub_registry() 24 | 25 | # make the request 26 | z <- crul::HttpClient$new(url = "https://httpbin.org")$get("get") 27 | 28 | # run tests (nothing returned means it passed) 29 | expect_is(z, "HttpResponse") 30 | expect_equal(z$status_code, 200) 31 | expect_equal(z$parse("UTF-8"), "success!") 32 | ``` 33 | 34 | ```{r echo=FALSE} 35 | stub_registry_clear() 36 | ``` 37 | 38 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active) 2 | [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10608848.svg)](https://doi.org/10.5281/zenodo.10608848) 3 | 4 | HTTP testing in R book 5 | ====================== 6 | 7 | This book is meant to be a free, central reference for developers of R packages accessing web resources, to help them have a faster and more robust development. Our aim is to develop an useful guidance to go with the great recent tools that vcr, webmockr, httptest and webfakes are. 8 | 9 | ## Contributions 10 | 11 | Your feedback, documentation requests, fixes are welcome! 12 | See [contributing guide](.github/CONTRIBUTING.md). 13 | 14 | ## Code of Conduct 15 | 16 | See [rOpenSci Code of Conduct](ropensci.org/code-of-conduct). 17 | -------------------------------------------------------------------------------- /18-session-info.Rmd: -------------------------------------------------------------------------------- 1 | # Session info 2 | 3 | ```{r echo = FALSE} 4 | knitr::opts_chunk$set( 5 | comment = "#>" 6 | ) 7 | ``` 8 | 9 | 10 | ## Session info 11 | 12 | ```{r} 13 | library("magrittr") 14 | 15 | dependencies <- attachment::att_from_rmds(".") 16 | dependencies <- dependencies[!dependencies %in% c("attachment", "bookdown", "knitr")] 17 | 18 | sessioninfo::package_info( 19 | pkgs = dependencies 20 | ) %>% 21 | as.data.frame() %>% 22 | .[, c("package", "ondiskversion")] %>% 23 | knitr::kable() 24 | ``` 25 | 26 | None of crul, webmockr, vcr, httptest have compiled code, but an underlying dependency of all of them, `curl` does. See [curl's README](https://github.com/jeroen/curl/#installation) for installation instructions in case you run into curl related problems. webfakes has compiled code. 27 | 28 | ## Full session info 29 | 30 | Session info for this book 31 | 32 | ```{r} 33 | sessioninfo::session_info() 34 | ``` 35 | -------------------------------------------------------------------------------- /_bookdown.yml: -------------------------------------------------------------------------------- 1 | output_dir: "docs" 2 | repo: https://github.com/ropensci-books/http-testing 3 | language: 4 | ui: 5 | chapter_name: "Chapter " 6 | delete_merged_file: true 7 | rmd_files: ["index.Rmd", 8 | "intro-general.Rmd", "intro-graceful.Rmd", "intro-pkgs.Rmd", 9 | "wholegames-intro.Rmd", "wholegames-vcr.Rmd", "wholegames-httptest.Rmd", "wholegames-mocking.Rmd", 10 | "wholegames-httptest2.Rmd", "wholegames-presser.Rmd", "wholegames-comparison.Rmd", 11 | "topics-real.Rmd","topics-cran.Rmd","topics-security.Rmd","topics-errors.Rmd", 12 | "topics-contributing.Rmd", 13 | "conclusion.Rmd", 14 | "03-mocking.Rmd", "04-webmockr-stubs.Rmd", "05-webmockr-testing.Rmd", 15 | "06-webmockr-utilities.Rmd", "07-vcr.Rmd", "08-vcr-usage.Rmd", "09-configuration.Rmd", 16 | "10-record-modes.Rmd", "11-request-matching.Rmd", "12-logging.Rmd", "13-security.Rmd", 17 | "14-escape-hatches.Rmd", "15-cassettes.Rmd", "16-gotchas.Rmd", 18 | "18-session-info.Rmd", "404.Rmd"] 19 | -------------------------------------------------------------------------------- /packages.bib: -------------------------------------------------------------------------------- 1 | @Manual{R-base, 2 | title = {R: A Language and Environment for Statistical Computing}, 3 | author = {{R Core Team}}, 4 | organization = {R Foundation for Statistical Computing}, 5 | address = {Vienna, Austria}, 6 | year = {2017}, 7 | url = {https://www.R-project.org/}, 8 | } 9 | @Manual{R-bookdown, 10 | title = {bookdown: Authoring Books and Technical Documents with R Markdown}, 11 | author = {Yihui Xie}, 12 | year = {2017}, 13 | note = {R package version 0.5}, 14 | url = {https://CRAN.R-project.org/package=bookdown}, 15 | } 16 | @Manual{R-knitr, 17 | title = {knitr: A General-Purpose Package for Dynamic Report Generation in R}, 18 | author = {Yihui Xie}, 19 | year = {2017}, 20 | note = {R package version 1.17}, 21 | url = {https://CRAN.R-project.org/package=knitr}, 22 | } 23 | @Manual{R-rmarkdown, 24 | title = {rmarkdown: Dynamic Documents for R}, 25 | author = {JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang}, 26 | year = {2017}, 27 | note = {R package version 1.8}, 28 | url = {https://CRAN.R-project.org/package=rmarkdown}, 29 | } 30 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Code of Conduct 2 | 3 | As contributors and maintainers of this project, we pledge to respect all people who 4 | contribute through reporting issues, posting feature requests, updating documentation, 5 | submitting pull requests or patches, and other activities. 6 | 7 | We are committed to making participation in this project a harassment-free experience for 8 | everyone, regardless of level of experience, gender, gender identity and expression, 9 | sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion. 10 | 11 | Examples of unacceptable behavior by participants include the use of sexual language or 12 | imagery, derogatory comments or personal attacks, trolling, public or private harassment, 13 | insults, or other unprofessional conduct. 14 | 15 | Project maintainers have the right and responsibility to remove, edit, or reject comments, 16 | commits, code, wiki edits, issues, and other contributions that are not aligned to this 17 | Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed 18 | from the project team. 19 | 20 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by 21 | opening an issue or contacting one or more of the project maintainers. 22 | 23 | This Code of Conduct is adapted from the Contributor Covenant 24 | (http:contributor-covenant.org), version 1.0.0, available at 25 | http://contributor-covenant.org/version/1/0/0/ 26 | -------------------------------------------------------------------------------- /07-vcr.Rmd: -------------------------------------------------------------------------------- 1 | # (PART) vcr details {-} 2 | 3 | # Caching HTTP requests {#vcr-intro} 4 | 5 | Record HTTP calls and replay them 6 | 7 | ## Package documentation {#vcr-pkgdown} 8 | 9 | Check out for documentation on `vcr` functions. 10 | 11 | 12 | ## Design {#design} 13 | 14 | https://docs.ropensci.org/vcr/articles/design.html 15 | 16 | ## Basic usage {#vcr-basic-usage} 17 | 18 | https://docs.ropensci.org/vcr/articles/vcr.html 19 | ## vcr enabled testing {#vcr-enabled-testing} 20 | 21 | ### check vs. test {#check-vs-test} 22 | 23 | > TLDR: Run `devtools::test()` before running `devtools::check()` for recording your cassettes. 24 | 25 | When running tests or checks of your whole package, note that you'll get different results with 26 | `devtools::check()` (check button of RStudio build pane) vs. `devtools::test()` (test button of RStudio build pane). This arises because `devtools::check()` runs in a 27 | temporary directory and files created (vcr cassettes) are only in that temporary directory and 28 | thus don't persist after `devtools::check()` exits. 29 | 30 | However, `devtools::test()` does not run in a temporary directory, so files created (vcr 31 | cassettes) are in whatever directory you're running it in. 32 | 33 | Alternatively, you can run `devtools::test_file()` (or the "Run test" button in RStudio) to create your vcr cassettes one test file at a time. 34 | 35 | 36 | ### CI sites: GitHub Actions, Appveyor, etc. {#vcr-ci} 37 | 38 | Refer to [the security chapter](#vcr-security). 39 | -------------------------------------------------------------------------------- /topics-errors.Rmd: -------------------------------------------------------------------------------- 1 | # Faking HTTP errors {#errors-chapter} 2 | 3 | With HTTP testing you can test the behavior of your package in case of an API error without having to actually trigger an API error. 4 | This is important for testing your package's [gracefulness](#graceful) (informative error message for the user) and robustness (if you e.g. use retrying in case of API errors). 5 | 6 | ## How to test for API errors (e.g. 503) 7 | 8 | Different possibilities: 9 | 10 | * Use webmockr as in [our demo](#vcr). 11 | * [Edit a vcr cassette](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html); be careful to skip this test when vcr is off with `vcr::skip_if_vcr_is_off()`. 12 | * With httptest or httptest2, edit a mock file as in [our demo](#httptest2), or create it from scratch. 13 | * With webfakes, choose what to return, have a specific app for the test, see [our demo](#webfakes). 14 | 15 | ## How to test for sequence of responses (e.g. 503 then 200) 16 | 17 | Different possibilities: 18 | 19 | * Use [webmockr](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html#the-same-thing-with-webmockr-3). 20 | * [Edit a vcr cassette](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html#example-2-test-using-an-edited-cassette-with-a-503-then-a-200-1); be careful to skip this test when vcr is off with `vcr::skip_if_vcr_is_off()` 21 | * With httptest, this is [not easy yet](https://github.com/nealrichardson/httptest/issues/49) ([httptest2 issue](https://github.com/nealrichardson/httptest2/issues/18)) 22 | * With webfakes, follow [the docs](https://r-lib.github.io/webfakes/articles/how-to.html#how-do-i-test-a-sequence-of-requests-). Also have a specific app for the test as this is not the behavior you want in all your tests.` -------------------------------------------------------------------------------- /conclusion.Rmd: -------------------------------------------------------------------------------- 1 | # (PART) Conclusion {-} 2 | 3 | # Conclusion 4 | 5 | Once you get here you will have read about basic HTTP (testing) concepts in R, 6 | discovered five great packages in demos (httptest2, vcr&webmockr, httptest, webfakes), and 7 | dived into more advanced topics like security. 8 | 9 | What's next? Applying those tools in your package(s), of course! 10 | 11 | * Pick one or several HTTP testing package(s) for your package. Examples of combinations: 12 | * vcr for testing normal behavior, webmockr for testing behavior in case of web resource errors. 13 | * vcr or httptest2 for most tests, webfakes for more advanced things like OAuth2.0 flows or slow internet connection. 14 | 15 | * Read all the docs of the HTTP testing package(s) you choose -- a very worthy use of your time. For vcr and webmockr you can even stay here in this book and take advantage of the "vcr details" and "webmockr details" sections. 16 | 17 | * For more examples, you could also look at the reverse dependencies of the HTTP testing package(s) you use to see how they are used by other developers. 18 | 19 | * Follow developments of the HTTP testing package(s) you choose. As all five packages we presented are developed on GitHub, you could e.g. release-watch their repositories. They are also all distributed on CRAN, so you might use your usual channel for learning about CRAN updates. 20 | 21 | * Participate in the development of the HTTP testing package(s) you choose? Your bug reports, feature requests, contributions might be helpful. Make sure to read the contributing guide and to look at current activity in the repositories. 22 | 23 | * Report any feedback about this book, your experience HTTP testing, tips, etc. 24 | * in the [GitHub repository](https://github.com/ropensci-books/http-testing) of the book, 25 | * or in [rOpenSci forum](https://discuss.ropensci.org/). 26 | 27 | Happy HTTP testing! -------------------------------------------------------------------------------- /.github/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # CONTRIBUTING # 2 | 3 | ### Issues and Pull Requests 4 | 5 | If you are considering a pull request, you may want to open an issue first to discuss with the maintainer(s). 6 | 7 | ### Code contributions 8 | 9 | * Fork this repo to your GitHub account 10 | * Clone your version on your account down to your machine from your account, e.g,. `git clone https://github.com//http-testing.git` 11 | * Make sure to track progress upstream (i.e., on our version of `http-testing` at `ropensci-books/http-testing`) by doing `git remote add upstream https://github.com/ropensci-books/http-testing.git`. Before making changes make sure to pull changes in from upstream by doing either `git fetch upstream` then merge later or `git pull upstream` to fetch and merge in one step 12 | * Make your changes (bonus points for making changes on a new feature branch - see for how to contribute by branching, making changes, then submitting a pull request) 13 | * Push up to your account 14 | * Submit a pull request to home base (likely master branch, but check to make sure) at `ropensci-books/http-testing` 15 | 16 | ### Discussion forum 17 | 18 | Check out our [discussion forum](https://discuss.ropensci.org) if you think your issue requires a longer form discussion. 19 | 20 | ### Book deployment 21 | 22 | For commits to the repo (not forks), the book will be built and deployed by GitHub Actions. 23 | 24 | * **commits to master**: the book is built (gitbook, PDF, EPUB) and deployed via gh-pages. 25 | 26 | * **commits in a PR**: the book is built (gitbook, PDF, EPUB) and deployed to a Netlify website. 27 | 28 | * **commits in a PR from a fork**: the book is built (gitbook, PDF, EPUB) but only deployed if NETLIFY_SITE_ID and NETLIFY_AUTH_TOKEN are set in the fork repo settings (and it is not expected that contributors do that). 29 | 30 | Maëlle Salmon owns the Netlify website for this repo. To change the Netlify website, 31 | 32 | * create a new Netlify website from a local folder, get its site ID (`NETLIFY_SITE_ID`) via the site settings, 33 | and get a deploy token (`NETLIFY_AUTH_TOKEN`) for your account via the settings. 34 | * save NETLIFY_SITE_ID and NETLIFY_AUTH_TOKEN in the repo settings, secrets tab. 35 | -------------------------------------------------------------------------------- /15-cassettes.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Managing cassettes {#managing-cassettes} 11 | 12 | https://docs.ropensci.org/vcr/articles/vcr.html?q=editing#cassette-files 13 | 14 | Be aware when you add your cassettes to either `.gitignore` and/or 15 | `.Rbuildignore`. 16 | 17 | ## gitignore cassettes {#gitignore-cassettes} 18 | 19 | The [.gitignore][gitignorefile] file lets you tell [git][] what files to 20 | ignore - those files are not tracked by git and if you share the git 21 | repository to the public web, those files in the `.gitignore` file 22 | won't be shared in the public version. 23 | 24 | When using `vcr` you may want to include your cassettes in the 25 | `.gitignore` file. You may want to when your cassettes contain sensitive 26 | data that you don't want to have on the internet & dont want to hide 27 | with [filter_sensitive_data](#api-keys-security). 28 | 29 | You may want to have your cassettes included in your GitHub repo, both 30 | to be present when tests run on CI, and when others run your tests. 31 | 32 | There's no correct answer on whether to gitignore your cassettes. 33 | Think about security implications and whether you want CI and human 34 | contributors to use previously created cassettes or to create/use their 35 | own. 36 | 37 | ## Rbuildignore cassettes {#rbuildignore-cassettes} 38 | 39 | The [.Rbuildignore][Rbuildignore] file is used to tell R to ignore 40 | certain files/directories. 41 | 42 | There's not a clear use case for why you'd want to add vcr cassettes 43 | to your `.Rbuildignore` file, but if you do be aware that will affect 44 | your vcr enabled tests. 45 | 46 | ## sharing cassettes {#sharing-cassettes} 47 | 48 | Sometimes you may want to share or re-use cassettes across tests, 49 | for example to [reduce the size for package sources](#cran-preparedness) or 50 | to test different functionality of your package functions 51 | that make the same query under the hood. 52 | 53 | To do so, you can use the same cassette name for multiple `vcr::use_cassette()` 54 | calls. 55 | That way you can use the same cassette across multiple tests. 56 | 57 | ## deleting cassettes 58 | 59 | Removing a cassette is as easy as deleting in your file finder, 60 | or from the command line, or from within a text editor or an IDE (RStudio, Positron, etc). 61 | 62 | If you delete a cassette, on the next test run the cassette will 63 | be recorded again. 64 | 65 | If you do want to re-record a test to a cassette, instead of 66 | deleting the file you can toggle [record modes](#record-modes). 67 | 68 | 69 | 70 | [gitignorefile]: https://guide.freecodecamp.org/git/gitignore/ 71 | [Rbuildignore]: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#index-_002eRbuildignore-file 72 | -------------------------------------------------------------------------------- /08-vcr-usage.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # Advanced vcr usage {#vcr-usage} 10 | 11 | Now that we've covered basic `vcr` usage, it's time for some more advanced usage topics. 12 | 13 | ```{r} 14 | library("vcr") 15 | ``` 16 | 17 | ## Mocking writing to disk {#vcr-disk} 18 | 19 | If you have http requests for which you write the response to disk, then 20 | use `vcr_configure()` to set the `write_disk_path` option. See more about 21 | the [write_disk_path configuration option](#write-disk-path). 22 | 23 | Here, we create a temporary directory, then set the fixtures 24 | 25 | ```{r} 26 | tmpdir <- tempdir() 27 | vcr_configure( 28 | dir = file.path(tmpdir, "fixtures"), 29 | write_disk_path = file.path(tmpdir, "files") 30 | ) 31 | ``` 32 | 33 | Then pass a file path (that doesn't exist yet) to crul's `disk` parameter. 34 | `vcr` will take care of handling writing the response to that file in 35 | addition to the cassette. 36 | 37 | ```{r} 38 | library(crul) 39 | ## make a temp file 40 | f <- tempfile(fileext = ".json") 41 | ## make a request 42 | cas <- use_cassette("test_write_to_disk", { 43 | out <- HttpClient$new("https://httpbin.org/get")$get(disk = f) 44 | }) 45 | file.exists(out$content) 46 | out$parse() 47 | ``` 48 | 49 | This also works with `httr`. The only difference is that you write to disk 50 | with a function `httr::write_disk(path)` rather than a parameter. 51 | 52 | ::: {.alert .alert-dismissible .alert-info} 53 | Writing to disk with `{httr2}` does not yet work with `{vcr}` -- see 54 | ::: 55 | 56 | Note that when you write to disk when using `vcr`, the cassette is slightly 57 | changed. Instead of holding the http response body itself, the cassette 58 | has the file path with the response body. 59 | 60 | ```yaml 61 | http_interactions: 62 | - request: 63 | method: get 64 | uri: https://httpbin.org/get 65 | response: 66 | headers: 67 | status: HTTP/1.1 200 OK 68 | access-control-allow-credentials: 'true' 69 | body: 70 | encoding: UTF-8 71 | file: yes 72 | string: /private/var/folders/fc/n7g_vrvn0sx_st0p8lxb3ts40000gn/T/Rtmp5W4olr/files/file177e2e5d97ec.json 73 | ``` 74 | 75 | And the file has the response body that otherwise would have been in the `string` 76 | yaml field above: 77 | 78 | ```json 79 | { 80 | "args": {}, 81 | "headers": { 82 | "Accept": "application/json, text/xml, application/xml, */*", 83 | "Accept-Encoding": "gzip, deflate", 84 | "Host": "httpbin.org", 85 | "User-Agent": "libcurl/7.54.0 r-curl/4.3 crul/0.9.0" 86 | }, 87 | "origin": "24.21.229.59, 24.21.229.59", 88 | "url": "https://httpbin.org/get" 89 | } 90 | ``` 91 | 92 | ```{r echo=FALSE} 93 | invisible(vcr_configure_reset()) 94 | ``` 95 | -------------------------------------------------------------------------------- /index.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "HTTP testing in R" 3 | date: "`r Sys.Date()`" 4 | author: "Scott Chamberlain, Maëlle Salmon" 5 | site: bookdown::bookdown_site 6 | documentclass: book 7 | bibliography: [book.bib, packages.bib] 8 | biblio-style: apalike 9 | link-citations: yes 10 | github-repo: ropensci-books/http-testing 11 | description: "Best practice and tips for testing packages interfacing web resources." 12 | url: 'https\://books.ropensci.org/http-testing/' 13 | --- 14 | 15 | DOI 16 | 17 | # Preamble 18 | 19 | Are you working on an R package accessing resources on the web, be it a cat facts API, a scientific data source or your system for Customer relationship management? 20 | As with all other packages, appropriate unit testing can make your code more robust. 21 | The unit testing of a package interacting with web resources, however, brings special challenges: 22 | dependence of tests on a good internet connection, testing in the absence of authentication secrets, etc. 23 | Having tests fail due to resources being down or slow, during development or on CRAN, means a time loss for everyone involved (slower development, messages from CRAN). 24 | Although some packages accessing remote resources are well tested, there is a lack of resources around best practices. 25 | 26 | This book is meant to be a free, central reference for developers of R packages accessing web resources, to help them have a faster and more robust development. 27 | Our aim is to develop a useful guide to go with the great recent tools `{vcr}`, `{webmockr}`, `{httptest}`, `{httptest2}` and `{webfakes}`. 28 | 29 | We expect you to know [package development basics](https://r-pkgs.org/), and [git](https://happygitwithr.com/). 30 | 31 | _Note related to previous versions: this book was intended as a detailed guide to using a particular suite of packages for HTTP mocking and testing in R code and/or packages, namely those maintained by Scott Chamberlain (`{crul}`, `{webmockr}`, `{vcr}`), but its scope has been extended to generalize the explanation of concepts to similar packages._ 32 | 33 | You can also read the [PDF version](/http-testing/main.pdf) or [epub version](/http-testing/main.epub) of this book. 34 | 35 | _Thanks to contributors to the book: 36 | [Alex Whan](https://github.com/alexwhan), 37 | [Aurèle](https://github.com/eaurele), 38 | [Christophe Dervieux](https://github.com/cderv), 39 | [Daniel Possenriede](https://github.com/dpprdan), 40 | [Hugo Gruson](https://github.com/Bisaloo), 41 | [Jon Harmon](https://github.com/jonthegeek/), 42 | [Lluís Revilla Sancho](https://github.com/llrs), 43 | [Xavier A](https://github.com/xvrdm)._ 44 | 45 | ```{block, type = "alert alert-dismissible alert-success"} 46 | Project funded by [rOpenSci](https://ropensci.org) (Scott Chamberlain's work) & the [R Consortium](https://www.r-consortium.org/projects/awarded-projects/2020-group-1#HTTP+testing+in+R+Book) (Maëlle Salmon's work). 47 | ``` 48 | -------------------------------------------------------------------------------- /16-gotchas.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Gotchas {#gotchas} 11 | 12 | There's a few things to watch out for when using `vcr`. 13 | 14 | - **Security**: Don't put your secure API keys, tokens, etc. on the public web. See the [Security chapter](#security-chapter) and the [vcr security chapter](#vcr-security). 15 | - **API key issues**: Running vcr enabled tests in different contexts when API keys are used can have some rough edges. 16 | - **Dates**: Be careful when using dates in tests with `vcr`. e.g. if you generate todays date, and pass that in to a function in your package that uses that date for an HTTP request, the date will be different from the one in the matching cassette, causing a `vcr` failure. 17 | - **HTTP errors**: It's a good idea to test failure behavior of a web service in your test suite. Sometimes `vcr` can handle that and sometimes it cannot. Open any issues about this because ideally i think `vcr` could handle all cases of HTTP failures. 18 | - **Very large response bodies**: A few things about large response bodies. First, `vcr` may give you trouble with very large response bodies as we've see yaml parsing problems already. Second, large response bodies means large cassettes on disk - so just be aware of the file size if that's something that matters to you. Third, large response bodies will take longer to load into R, so you may still have a multi second test run even though the test is using a cached HTTP response. 19 | - **Encoding**: We haven't dealt with encoding much yet at all, so we're likely to run into encoding issues. One blunt instrument for this for now is to set `preserve_exact_body_bytes = TRUE` when running `vcr::use_cassette()` or `vcr::insert_cassette()`, which stores the response body as base64. 20 | - **devtools::check vs. devtools::test**: See (\@ref(check-vs-test)) 21 | - **ignored files**: See (\@ref(managing-cassettes)) 22 | 23 | ## Correct line identification {#line-identification} 24 | 25 | To get the actual lines where failures occur, you can wrap the `test_that` block in a `use_cassette()` block: 26 | 27 | ```r 28 | library(testthat) 29 | vcr::use_cassette("rl_citation", { 30 | test_that("my test", { 31 | aa <- rl_citation() 32 | 33 | expect_is(aa, "character") 34 | expect_match(aa, "IUCN") 35 | expect_match(aa, "www.iucnredlist.org") 36 | }) 37 | }) 38 | ``` 39 | 40 | OR put the `use_cassette()` block on the inside, but make sure to put `testthat` expectations outside of 41 | the `use_cassette()` block: 42 | 43 | ```r 44 | library(testthat) 45 | test_that("my test", { 46 | vcr::use_cassette("rl_citation", { 47 | aa <- rl_citation() 48 | }) 49 | 50 | expect_is(aa, "character") 51 | expect_match(aa, "IUCN") 52 | expect_match(aa, "www.iucnredlist.org") 53 | }) 54 | ``` 55 | 56 | Do not wrap the `use_cassette()` block inside your `test_that()` block with `testthat` expectations inside the `use_cassette()` block, as you'll only get the line number that the `use_cassette()` block starts on on failures. 57 | -------------------------------------------------------------------------------- /.github/workflows/bookdown.yml: -------------------------------------------------------------------------------- 1 | # Workflow derived from https://github.com/r-lib/actions/tree/master/examples 2 | # Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help 3 | on: 4 | push: 5 | branches: [main, master] 6 | pull_request: 7 | branches: [main, master] 8 | 9 | name: bookdown 10 | 11 | jobs: 12 | bookdown: 13 | runs-on: ubuntu-latest 14 | env: 15 | GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} 16 | isExtPR: ${{ github.event.pull_request.head.repo.fork == true }} 17 | steps: 18 | - uses: actions/checkout@v5 19 | 20 | - uses: r-lib/actions/setup-pandoc@v2 21 | 22 | - uses: r-lib/actions/setup-tinytex@v2 23 | 24 | - uses: r-lib/actions/setup-r@v2 25 | with: 26 | use-public-rspm: true 27 | 28 | - uses: r-lib/actions/setup-r-dependencies@v2 29 | 30 | - name: Cache bookdown results 31 | uses: actions/cache@v4 32 | with: 33 | path: _bookdown_files 34 | key: bookdown-2-${{ hashFiles('**/*Rmd') }} 35 | restore-keys: bookdown-2- 36 | 37 | - name: Configure Git user 38 | run: | 39 | git config --global user.name "$GITHUB_ACTOR" 40 | git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" 41 | 42 | - name: Build site 43 | run: Rscript -e 'options(knitr.duplicate.label = "allow");bookdown::render_book("index.Rmd", quiet = TRUE)' 44 | 45 | - name: Render book PDF 46 | run: Rscript -e 'options(knitr.duplicate.label = "allow");bookdown::render_book("index.Rmd", "bookdown::pdf_book", output_dir = "pdfbook")' 47 | 48 | - name: Render book EPUB 49 | run: Rscript -e 'options(knitr.duplicate.label = "allow");bookdown::render_book("index.Rmd", "bookdown::epub_book", output_dir = "epubbook")' 50 | 51 | - name: Move files around 52 | run: Rscript -e 'file.copy(from = "pdfbook/_main.pdf", to = "docs/main.pdf")' -e 'file.copy(from = "epubbook/_main.epub", to = "docs/main.epub")' 53 | 54 | - name: Deploy to GitHub Pages 55 | if: contains(env.isExtPR, 'false') 56 | id: gh-pages-deploy 57 | uses: JamesIves/github-pages-deploy-action@4.1.5 58 | with: 59 | branch: gh-pages 60 | folder: docs 61 | 62 | - name: Deploy to Netlify 63 | if: contains(env.isExtPR, 'false') 64 | id: netlify-deploy 65 | uses: nwtgck/actions-netlify@v1.2 66 | with: 67 | publish-dir: './docs' 68 | production-branch: main 69 | github-token: ${{ secrets.GITHUB_TOKEN }} 70 | deploy-message: 71 | 'Deploy from GHA: ${{ github.event.pull_request.title || github.event.head_commit.message }} (${{ github.sha }})' 72 | # these all default to 'true' 73 | enable-pull-request-comment: false 74 | enable-commit-comment: false 75 | # enable-commit-status: true 76 | #o verwrites-pull-request-comment: true 77 | env: 78 | NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} 79 | NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} 80 | timeout-minutes: 1 81 | -------------------------------------------------------------------------------- /13-security.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | 10 | # Security with vcr {#vcr-security} 11 | 12 | Refer to [the security chapter](#security-chapter) for more general guidance. 13 | 14 | ## Keeping secrets safe {#api-keys-security} 15 | 16 | To keep your secrets safe, you need to use parameters of `vcr::vcr_configure()` that tell vcr either where secrets are (and what to put in their place), or what secrets are (and what to put in their place). 17 | It is best if you know how secrets are used in requests: e.g. is the API key passed as a header or part of the query string? 18 | Maybe you will need different strategies for the different secrets (e.g. an OAuth2.0 access token will be set as Authorization header but an OAuth2.0 refresh token might be in a query string). 19 | 20 | In all cases, it is crucial to look at your cassettes before putting them on the public web, just to be sure you got the configuration right! 21 | 22 | ### If the secret is in a request header 23 | 24 | You can use `filter_request_headers`! 25 | 26 | There are different ways to use it. 27 | 28 | ```r 29 | # Remove one header from the cassettes 30 | vcr_configure( 31 | filter_request_headers = "Authorization" 32 | ) 33 | 34 | # Remove two headers from the cassettes 35 | vcr_configure( 36 | filter_request_headers = c("Authorization", "User-Agent") 37 | ) 38 | 39 | # Replace one header with a given string 40 | vcr_configure( 41 | filter_request_headers = list(Authorization = "<<>>") 42 | ) 43 | ``` 44 | 45 | ### If the secret is in a response header 46 | 47 | You can use `filter_response_headers` that works like `filter_request_headers`. 48 | 49 | ### If the secret is somewhere else 50 | 51 | In this case you need to tell vcr what the secret string is via `filter_sensitive_data`. 52 | Do not write the secret string directly in the configuration, that'd defeat the purpose of protecting it! 53 | Have the secret in an environment variable for instance and tell vcr to read it from there. 54 | 55 | The configuration parameter `filter_sensitive_data` accepts a named list. 56 | 57 | Each element in the list should be of the following format: 58 | 59 | `thing_to_replace_it_with = thing_to_replace` 60 | 61 | We replace all instances of `thing_to_replace` with `thing_to_replace_it_with`. 62 | 63 | Before recording (writing to a cassette) we do the replacement and then when 64 | reading from the cassette we do the reverse replacement to get back 65 | to the real data. 66 | 67 | ```r 68 | vcr_configure( 69 | filter_sensitive_data = list("<<>>" = Sys.getenv('API_KEY')) 70 | ) 71 | ``` 72 | 73 | You want to make the string that replaces your sensitive string something that 74 | won't be easily found elsewhere in the response body/headers/etc. 75 | 76 | ## API keys and tests run in varied contexts {#different-api-keys} 77 | 78 | * For real requests a real API key is needed. 79 | * For requests using cassettes a fake API key is needed to fool your package. That is 80 | why in our [demo of vcr](#vcr) we set a fake API key in a test setup file. 81 | 82 | ## Other security 83 | 84 | Let us know about any other security concerns! Surely there's things we haven't 85 | considered yet. 86 | -------------------------------------------------------------------------------- /topics-real.Rmd: -------------------------------------------------------------------------------- 1 | # (PART) Advanced Topics {-} 2 | 3 | # Making real requests {#real-requests-chapter} 4 | 5 | As touched upon in the Whole Games section, it's good to have _some_ tests against the real API. 6 | Indeed, the web resource can change. 7 | 8 | ## What can change? 9 | 10 | What can happen? 11 | 12 | * An API introducing rate-limiting; 13 | * A web resource disappearing; 14 | * etc. 15 | 16 | ## How to make real requests 17 | 18 | Maybe you can just run the same tests without using the mock files. 19 | 20 | * with vcr, this behavior is [one environment variable away](https://docs.ropensci.org/vcr/reference/lightswitch.html) (namely, `VCR_TURN_OFF`). 21 | * with httptest or httptest2 you can create the [same kind of behavior](https://enpiar.com/r/httptest/index.html#how-do-i-switch-between-mocking-and-real-requests). 22 | * with webfakes you can also [create that behavior](https://r-lib.github.io/webfakes/articles/how-to.html#how-to-make-sure-that-my-code-works-with-the-real-api-). 23 | 24 | Now this means assuming _all_ your tests work with real requests. 25 | 26 | * If a few tests won't work with real requests (say they have a fixture mimicking an [API error](#errors-chapter), or specific answer as if today were a given date) then you can skip these tests when mocking/faking the web service is off. With vcr this means using `vcr::skip_if_vcr_off()`; with httptest and webfakes you'd create your custom skipper. 27 | * If most tests won't work with real requests, then creating a different folder for tests making real requests makes sense. It might be less unit-y as you could view these tests as integration/contract tests. Maybe they could use [testthat's snapshot testing](https://testthat.r-lib.org/articles/snapshotting.html) (so you could view what's different in the API). 28 | 29 | ### When to make real requests? 30 | 31 | Locally, you might want to make real requests once in a while, in particular before a CRAN release. 32 | 33 | On continuous integration you have to learn how to trigger workflows and configure build matrices to e.g. 34 | 35 | * Have one build in your build matrix using real requests at each commit (this might be too much, see next section); 36 | * Have one scheduled workflow once a day or once a week using real requests. 37 | 38 | ## Why not make only or too many real requests? 39 | 40 | The reasons why you can't only make real requests in your tests are the [reasons why you are reading these book](#pkgs-testing-chapter): 41 | 42 | * they are slower; 43 | * you can't test for API errors; 44 | * etc. 45 | 46 | Now no matter what your setup is you don't want to make _too many_ real requests as it can be bad for the web resource and bad for you (e.g. using all your allowed requests!). 47 | Regarding allowed requests, if possible you could however increase them by requesting for some sort of special development account if such a thing exists for the API you are working with. 48 | 49 | ## A complement to real requests: API news! 50 | 51 | Running real requests is important to notice if something changes in the API (expected requests, responses). 52 | Now, you can and should also follow the news of the web resource you are using in case there is something in place. 53 | 54 | * Subscribe to the API newsletter if there's one; 55 | * Read API changelogs if they are public; 56 | * In particular, if the API is developed on GitHub/GitLab/etc. you could watch the repo or subscribe to releases, so that you might automatically get notified. 57 | -------------------------------------------------------------------------------- /topics-contributing.Rmd: -------------------------------------------------------------------------------- 1 | # Contributor friendliness {#contributor-friendliness} 2 | 3 | How do you make your package wrapping an HTTP resource contributor-friendly? 4 | 5 | rOpenSci has some general advice on [contributor-friendliness](https://devguide.ropensci.org/collaboration.html#friendlyfiles). 6 | 7 | Now, there are some more aspects when dealing with HTTP testing. 8 | 9 | ## Taking notes about encryption 10 | 11 | In your contributing guide, make sure you note how you e.g. created an encrypted token for the tests. Link to a script that one could run to re-create it. Good for future contributors including yourself! 12 | 13 | ## Providing a sandbox 14 | 15 | It might be very neat to provide a **sandbox**, even if just for yourself. 16 | 17 | 18 | * If interacting with say Twitter API you might want to create a Twitter account dedicated to this. 19 | 20 | ```{=html} 21 | 22 | ``` 23 | 24 | * If interacting with some sort of web platform you might want to create an account special for storing test data. 25 | 26 | * Some web APIs provide a test API key, a test account that one can request access to. 27 | 28 | Make sure to take notes on how to create / request access to a sandbox, in your contributing guide. 29 | 30 | ## Switching between accounts depending on the development mode 31 | 32 | Your package might have some behaviour to load a default token for instance, placed in an app dir. 33 | Now, for testing, you might want it to load another token, and you probably also want the token choice to be as automatic as possible. 34 | 35 | The [rtweet package](https://github.com/ropensci/rtweet) has such logic. 36 | 37 | * It [detects testing/dev mode](https://github.com/ropensci/rtweet/blob/f46bc98f9ac8433c7681d48ed778358bc22a552c/R/utils.R#L132). 38 | 39 | ```r 40 | is_testing <- function() { 41 | identical(Sys.getenv("TESTTHAT"), "true") 42 | } 43 | is_dev_mode <- function() { 44 | exists(".__DEVTOOLS__", .getNamespace("rtweet")) 45 | } 46 | ``` 47 | 48 | * If some environment variables are present it is able to [create a testing token](https://github.com/ropensci/rtweet/blob/270733c6bf46b2be794d7492d4a4e31d384db0b7/R/auth.R#L281). 49 | 50 | 51 | ```r 52 | rtweet_test <- function() { 53 | access_token <- Sys.getenv("RTWEET_ACCESS_TOKEN") 54 | access_secret <- Sys.getenv("RTWEET_ACCESS_SECRET") 55 | 56 | if (identical(access_token, "") || identical(access_secret, "")) { 57 | return() 58 | } 59 | 60 | rtweet_bot( 61 | "7rX1CfEYOjrtZenmBhjljPzO3", 62 | "rM3HOLDqmjWzr9UN4cvscchlkFprPNNg99zJJU5R8iYtpC0P0q", 63 | access_token, 64 | access_secret 65 | ) 66 | } 67 | ``` 68 | 69 | * The [testing token or a default token is loaded depending on the development mode](https://github.com/ropensci/rtweet/blob/270733c6bf46b2be794d7492d4a4e31d384db0b7/R/auth.R#L234). 70 | 71 | ## Documenting HTTP testing 72 | 73 | Contributors to the package might not be familiar with the HTTP testing package(s) you use (this is true of any non-trivial test setup). Make sure your contributing guide mentions pre-requisites and link to resources (maybe even this very book?). -------------------------------------------------------------------------------- /topics-cran.Rmd: -------------------------------------------------------------------------------- 1 | # CRAN- (and Bioconductor) preparedness for your tests {#cran-preparedness} 2 | 3 | There is no one right answer to how to manage your tests for CRAN, except that you 4 | do want a [clean check result on CRAN at all times](#graceful). 5 | This probably applies to Bioconductor too. 6 | The following is a 7 | discussion of the various considerations - which should give you enough 8 | information to make an educated decision. 9 | 10 | ## Running tests on CRAN? 11 | 12 | You can run vcr/httptest/httptest2/webfakes enabled tests on CRAN. 13 | CRAN is okay with files associated with tests, 14 | and so in general on CRAN you can run your tests that use cassettes, mock files or recorded responses on CRAN. 15 | Another aspect is the presence of dependencies: make sure the HTTP testing package you use is listed as `Suggests` dependency in DESCRIPTION! 16 | With webfakes this might mean also listing [optional dependencies](https://r-lib.github.io/webfakes/#optional-dependencies) in DESCRIPTION. 17 | With webfakes, your tests if run on CRAN should not assume the availability of a given port. 18 | 19 | When running HTTP tests on CRAN, be aware of a few things: 20 | 21 | - If your tests require any secret environment variables or R options (apart from the "foobar" ones used to fool your package when using a saved response), 22 | they won't be available on CRAN. In these cases you likely want to skip these 23 | tests with `testthat::skip_on_cran()`. 24 | - If your tests have cassettes, mock files or recorded responses with sensitive information in them, 25 | you probably do not want to have those cassettes on the internet, in which case 26 | you won't be running vcr enabled tests on CRAN either. In the case of sensitive 27 | information, you might want to [Rbuildignore](https://blog.r-hub.io/2020/05/20/rbuildignore/) the cassettes, mock files or recorded responses (and to gitignore them or make your package development repository private). 28 | - There is a maximal size for package sources so you will want your cassettes, mock files or recorded responses to not be too big. There are three ways to limit their size 29 | - Make requests that do not generate a huge response (e.g. tweak the time range); 30 | - Edit the recorded responses (why not even copy-paste responses from the API docs as those are often short) --- see [vcr docs about editing cassettes for pros and cons](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html); 31 | - Share [cassettes](#sharing-cassettes) / mock files / recorded responses between tests. 32 | 33 | Do [not *compress* cassettes, mock files or recorded responses](https://github.com/nealrichardson/httptest/issues/11#issuecomment-354699342): CRAN submissions are already compressed; compressed files will make git diffs hard to use. 34 | 35 | ## Skipping a few tests on CRAN? 36 | 37 | If you are worried at all about problems with HTTP tests on CRAN you can use 38 | `testthat::skip_on_cran()` to skip specific tests. 39 | Make sure your tests run somewhere else (on continuous integration) regularly! 40 | 41 | We'd recommend not running tests making real requests on CRAN, even when interacting with an API without authentication. 42 | 43 | ## Skipping all tests on CRAN? 44 | 45 | If you have a good continuous integration setup (several operating systems, scheduled runs, etc.) why not skip all tests on CRAN? 46 | 47 | ## Stress-test your package 48 | 49 | To stress-test your package before a CRAN submission, use `rhub::check_for_cran()` without passing any environment variable to the function, and use [WinBuilder](https://blog.r-hub.io/2020/04/01/win-builder/). -------------------------------------------------------------------------------- /03-mocking.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # (PART) webmockr details {-} 10 | 11 | # Mocking HTTP Requests {#mocking} 12 | 13 | The very very short version is: [webmockr][] helps you stub HTTP requests so you 14 | don't have to repeat yourself. 15 | 16 | ## Package documentation {#webmockr-pkgdown} 17 | 18 | Check out for documentation on `webmockr` functions. 19 | 20 | ## Features {#webmockr-features} 21 | 22 | * Stubbing HTTP requests at low http client lib level 23 | * Setting and verifying expectations on HTTP requests 24 | * Matching requests based on method, URI, headers and body 25 | * Support for `testthat` via [vcr][] 26 | * Can be used for testing or outside of a testing context 27 | 28 | ## How webmockr works in detail {#webmockr-detail} 29 | 30 | You tell `webmockr` what HTTP request you want to match against and if it sees a 31 | request matching your criteria it doesn't actually do the HTTP request. Instead, 32 | it gives back the same object you would have gotten back with a real request, but 33 | only with the bits it knows about. For example, we can't give back the actual 34 | data you'd get from a real HTTP request as the request wasn't performed. 35 | 36 | In addition, if you set an expectation of what `webmockr` should return, we 37 | return that. For example, if you expect a request to return a 418 error 38 | (I'm a Teapot), then that's what you'll get. 39 | 40 | **What you can match against** 41 | 42 | * HTTP method (required) 43 | 44 | Plus any single or combination of the following: 45 | 46 | * URI 47 | * Right now, we can match directly against URI's, and with regex URI patterns. 48 | Eventually, we will support RFC 6570 URI templates. 49 | * We normalize URI paths so that URL encoded things match 50 | URL un-encoded things (e.g. `hello world` to `hello%20world`) 51 | * Query parameters 52 | * We normalize query parameter values so that URL encoded things match 53 | URL un-encoded things (e.g. `message = hello world` to 54 | `message = hello%20world`) 55 | * Request headers 56 | * We normalize headers and treat all forms of same headers as equal. For 57 | example, the following two sets of headers are equal: 58 | * `list(H1 = "value1", content_length = 123, X_CuStOm_hEAder = "foo")` 59 | * `list(h1 = "value1", "Content-Length" = 123, "x-cuSTOM-HeAder" = "foo")` 60 | * Request body 61 | 62 | **Real HTTP requests** 63 | 64 | There's a few scenarios to think about when using `webmockr`: 65 | 66 | After doing 67 | 68 | ```r 69 | library(webmockr) 70 | ``` 71 | 72 | `webmockr` is loaded but not turned on. At this point `webmockr` doesn't 73 | change anything. 74 | 75 | Once you turn on `webmockr` like 76 | 77 | ```r 78 | webmockr::enable() 79 | ``` 80 | 81 | `webmockr` will now by default not allow real HTTP requests from the http 82 | libraries that adapters are loaded for (`crul`, `httr`, `httr2`). 83 | 84 | You can optionally allow real requests via `webmockr_allow_net_connect()`, and 85 | disallow real requests via `webmockr_disable_net_connect()`. You can check 86 | whether you are allowing real requests with `webmockr_net_connect_allowed()`. 87 | 88 | Certain kinds of real HTTP requests allowed: We don't suppoprt this yet, 89 | but you can allow localhost HTTP requests with the `allow_localhost` parameter 90 | in the `webmockr_configure()` function. 91 | 92 | **Storing actual HTTP responses** 93 | 94 | `webmockr` doesn't do that. Check out [vcr][] 95 | 96 | ## Basic usage {#webmockr-basic-usage} 97 | 98 | ```{r} 99 | library("webmockr") 100 | # enable webmockr 101 | webmockr::enable() 102 | ``` 103 | 104 | **Stubbed request based on uri only and with the default response** 105 | 106 | ```{r} 107 | stub_request("get", "https://httpbin.org/get") 108 | ``` 109 | 110 | ```{r} 111 | library("crul") 112 | x <- HttpClient$new(url = "https://httpbin.org") 113 | x$get('get') 114 | ``` 115 | 116 | 117 | [webmockr]: https://github.com/ropensci/webmockr 118 | [vcr]: https://github.com/ropensci/vcr 119 | -------------------------------------------------------------------------------- /04-webmockr-stubs.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # stubs {#webmockr-stubs} 10 | 11 | ```{r} 12 | library("webmockr") 13 | ``` 14 | 15 | set return objects 16 | 17 | ```{r} 18 | stub_request("get", "https://httpbin.org/get") %>% 19 | wi_th( 20 | query = list(hello = "world")) %>% 21 | to_return(status = 418) 22 | ``` 23 | 24 | ```{r} 25 | x$get('get', query = list(hello = "world")) 26 | ``` 27 | 28 | **Stubbing requests based on method, uri and query params** 29 | 30 | ```{r} 31 | stub_request("get", "https://httpbin.org/get") %>% 32 | wi_th(query = list(hello = "world"), 33 | headers = list('User-Agent' = 'libcurl/7.51.0 r-curl/2.6 crul/0.3.6', 34 | 'Accept-Encoding' = "gzip, deflate")) 35 | ``` 36 | 37 | ```{r} 38 | stub_registry() 39 | ``` 40 | 41 | ```{r} 42 | x <- HttpClient$new(url = "https://httpbin.org") 43 | x$get('get', query = list(hello = "world")) 44 | ``` 45 | 46 | **Stubbing requests and set expectation of a timeout** 47 | 48 | ```{r eval=FALSE} 49 | stub_request("post", "https://httpbin.org/post") %>% to_timeout() 50 | x <- HttpClient$new(url = "https://httpbin.org") 51 | x$post('post') 52 | #> Error: Request Timeout (HTTP 408). 53 | #> - The client did not produce a request within the time that the server was prepared 54 | #> to wait. The client MAY repeat the request without modifications at any later time. 55 | ``` 56 | 57 | **Stubbing requests and set HTTP error expectation** 58 | 59 | ```{r eval=FALSE} 60 | library(fauxpas) 61 | stub_request("get", "https://httpbin.org/get?a=b") %>% to_raise(HTTPBadRequest) 62 | x <- HttpClient$new(url = "https://httpbin.org") 63 | x$get('get', query = list(a = "b")) 64 | #> Error: Bad Request (HTTP 400). 65 | #> - The request could not be understood by the server due to malformed syntax. 66 | #> The client SHOULD NOT repeat the request without modifications. 67 | ``` 68 | 69 | ## Writing to disk {#webmockr-disk} 70 | 71 | 72 | There are two ways to deal with mocking writing to disk. First, you can create a file 73 | with the data you'd like in that file, then tell crul, httr, or httr2 where that file is. 74 | Second, you can simply give webmockr a file path (that doesn't exist yet) and some 75 | data, and webmockr can take care of putting the data in the file. 76 | 77 | 78 | ```{r echo=FALSE} 79 | stub_registry_clear() 80 | request_registry_clear() 81 | ``` 82 | 83 | Here's the first method, where you put data in a file as your mock, then pass the 84 | file as a connection (with `file()`) to `to_return()`. 85 | 86 | ```{r} 87 | ## make a temp file 88 | f <- tempfile(fileext = ".json") 89 | ## write something to the file 90 | cat("{\"hello\":\"world\"}\n", file = f) 91 | ## make the stub 92 | invisible(stub_request("get", "https://httpbin.org/get") %>% 93 | to_return(body = file(f))) 94 | ## make a request 95 | out <- HttpClient$new("https://httpbin.org/get")$get(disk = f) 96 | ## view stubbed file content 97 | readLines(file(f)) 98 | ``` 99 | 100 | With the second method, use `webmockr::mock_file()` to have `webmockr` handle file 101 | and contents. 102 | 103 | ```{r} 104 | g <- tempfile(fileext = ".json") 105 | ## make the stub 106 | invisible(stub_request("get", "https://httpbin.org/get?a=b") %>% 107 | to_return(body = mock_file(path = g, payload = "{\"hello\":\"mars\"}\n"))) 108 | ## make a request 109 | out <- crul::HttpClient$new("https://httpbin.org/get?a=b")$get(disk = g) 110 | ## view stubbed file content 111 | readLines(out$content) 112 | ``` 113 | 114 | `webmockr` also supports `httr::write_disk()`, here letting `webmockr` handle the 115 | mock file creation: 116 | 117 | ```{r} 118 | library(httr) 119 | httr_mock() 120 | ## make a temp file 121 | f <- tempfile(fileext = ".json") 122 | ## make the stub 123 | invisible(stub_request("get", "https://httpbin.org/get?cheese=swiss") %>% 124 | to_return( 125 | body = mock_file(path = f, payload = "{\"foo\": \"bar\"}"), 126 | headers = list('content-type' = "application/json") 127 | )) 128 | ## make a request 129 | out <- GET("https://httpbin.org/get?cheese=swiss", write_disk(f, TRUE)) 130 | ## view stubbed file content 131 | readLines(out$content) 132 | ``` 133 | 134 | ```{r cleanup, echo=FALSE} 135 | unlink(c(f, g)) 136 | httr_mock(FALSE) 137 | ``` 138 | -------------------------------------------------------------------------------- /intro-general.Rmd: -------------------------------------------------------------------------------- 1 | # (PART) Introduction {-} 2 | 3 | # HTTP in R 101 4 | 5 | ## What is HTTP? 6 | 7 | HTTP means HyperText Transport Protocol, but you were probably not just looking for a translation of the abbreviation. 8 | HTTP is a way for you to exchange information with a remote server. 9 | In your package, if information is going back and forth between the R session and the internet, you are using some sort of HTTP tooling. 10 | Your package is making _requests_ and receives _responses_. 11 | 12 | ### HTTP requests 13 | 14 | The HTTP request is what your package makes. 15 | It has a method (are you fetching information via `GET`? are you sending information via `POST`?), different parts of a URL (domain, endpoint, query string), and headers (containing for instance your secret identifiers). 16 | It can contain a body. For instance, you might be sending data as JSON. 17 | In that case one of the headers will describe the content. 18 | 19 | How do you know what request to make from your package? 20 | Hopefully you are interacting with a well documented web resource that will explain to you what methods are associated with what endpoints. 21 | 22 | ### HTTP responses 23 | 24 | The HTTP response is what the remote server provides, and what your package parses. 25 | A response has a status code indicating whether the request succeeded, response headers, and (optionally) a response body. 26 | 27 | Hopefully the documentation of the web API or web resource you are working with shows good examples of responses. 28 | In any case you'll find yourself experimenting with different requests to see what the response "looks like". 29 | 30 | ### More resources about HTTP 31 | 32 | How do you get started with interacting with HTTP in R? 33 | 34 | #### General HTTP resources 35 | 36 | * [Mozilla Developer Network docs about HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP) (recommended in the zine mentioned hereafter) 37 | * (_not free_) [Julia Evans' Zine "HTTP: Learn your browser's language!"](https://wizardzines.com/zines/http/) 38 | * The docs of the web API you are aiming to work with, and a search engine to understand the words that are new. 39 | 40 | #### HTTP with R 41 | 42 | * The docs of the R package you end up choosing! 43 | * Digging into the source code of another package that does similar things. 44 | 45 | 46 | ## HTTP requests in R: what package? 47 | 48 | In R, to interact with web resources, it is recommended to use `{curl}`; or its higher-level interfaces `{httr}` ([pronounced _hitter_ or _h-t-t-r_](https://community.rstudio.com/t/pronunciations-of-common-r-terms/1810/15)), `{httr2}` or `{crul}`. 49 | 50 | Do not use RCurl, because it is not actively maintained! 51 | 52 | When writing a package interacting with web resources, you will probably use `{httr2}`, `{httr}` or `{crul}`. 53 | 54 | * `{httr}` is the most popular and oldest of the three packages, and supports OAuth. 55 | `{httr}` docs feature a vignette called [Best practices for API packages](https://httr.r-lib.org/articles/api-packages.html) 56 | 57 | * `{httr2}` _"is a ground-up rewrite of httr that provides a pipeable API with an explicit request object that solves more problems felt by packages that wrap APIs (e.g. built-in rate-limiting, retries, OAuth, secure secrets, and more)"_ so it might be a good idea to adopt it rather than `{httr}` for a new package. It has a vignette about [Wrapping APIs](https://httr2.r-lib.org/articles/wrapping-apis.html). 58 | 59 | * `{crul}` does not support OAuth but it uses an object-oriented interface, which you might like. 60 | `{crul}` has a set of [clients, or ways to perform requests](https://docs.ropensci.org/crul/articles/choosing-a-client.html), that might be handy. `{crul}` also has a vignette about [API package best practices 61 | ](https://docs.ropensci.org/crul/articles/best-practices-api-packages.html). 62 | 63 | Below we will try to programmatically access the [status of GitHub](https://www.githubstatus.com/api/#status), the open-source platform provided by the company of the same name. 64 | We will access the same information with `{httr2}` and `{crul}` 65 | If you decide to try the low-level curl, feel free to contribute an example. 66 | The internet has enough examples for httr. 67 | 68 | ```{r} 69 | github_url <- "https://kctbh9vrtdwd.statuspage.io/api/v2/status.json" 70 | ``` 71 | 72 | The URL above leaves no doubt as to what format the data is provided in, JSON! 73 | 74 | Let's first use `{httr2}`. 75 | 76 | ```{r} 77 | library("magrittr") 78 | response <- httr2::request(github_url) %>% 79 | httr2::req_perform() 80 | 81 | # Check the response status 82 | httr2::resp_status(response) 83 | 84 | # Or in a package you'd write 85 | httr2::resp_check_status(response) 86 | 87 | # Parse the content 88 | httr2::resp_body_json(response) 89 | 90 | # In case you wonder, the format was obtained from a header 91 | httr2::resp_header(response, "content-type") 92 | ``` 93 | 94 | Now, the same with `{crul}`. 95 | 96 | ```{r} 97 | # Create a client and get a response 98 | client <- crul::HttpClient$new(github_url) 99 | response <- client$get() 100 | 101 | # Check the response status 102 | response$status_http() 103 | 104 | # Or in a package you'd write 105 | response$raise_for_status() 106 | 107 | # Parse the content 108 | response$parse() 109 | jsonlite::fromJSON(response$parse()) 110 | ``` 111 | 112 | Hopefully these very short snippets give you an idea of what syntax to expect when choosing one of these packages. 113 | 114 | Note that the choice of a package will constrain the HTTP testing tools you can use. 115 | However, the general ideas will remain the same. 116 | You could switch your package backend from, say, `{crul}` to `{httr}` _without changing your tests_, if your tests do not test too many specificities of internals. -------------------------------------------------------------------------------- /intro-graceful.Rmd: -------------------------------------------------------------------------------- 1 | # Graceful HTTP R packages {#graceful} 2 | 3 | Based on the previous chapter, your package interacting with a web resource has a dependency on `{curl}`, `{httr}`, `{httr2}` or `{crul}`. You have hopefully read the docs of the dependency you chose, including, in the case of `{httr}`, `{httr2}` and `{crul}`, the vignette about best practices for HTTP packages. Now, in this chapter we want to give more tips aimed at making your HTTP R package graceful, part of which you'll learn more about in this very book! 4 | 5 | **Why** write a *graceful* HTTP R package? First of all, graceful is a nice adjective. 💃🕺Second, graceful is the adjective used in [CRAN repository policy](https://cran.r-project.org/web/packages/policies.html) *"Packages which use Internet resources should fail gracefully with an informative message if the resource is not available or has changed (and not give a check warning nor error)."* Therefore, let's review how to make your R package graceful from this day forward, in success and in failure. 6 | 7 | ## Choose the HTTP resource wisely 8 | 9 | First of all, your life and the life of your package's users will be easier if the web service you're wrapping is well maintained and well documented. When you have a choice, try not to rely on a fragile web service. Moreover, if you can, try to communicate with the API providers (telling them about your package; reporting feature requests and bug reports in their preferred way). 10 | 11 | ## User-facing grace (how your package actually works) 12 | 13 | 0. If you can, do not request the API every time the user asks for something; cache data instead. No API call, no API call failure! 😉 See the R-hub blog post ["Caching the results of functions of your R package"](https://blog.r-hub.io/2021/07/30/cache/). To remember answers across sessions, see approaches presented in the R-hub blog post ["Persistent config and data for R packages"](). Caching behavior should be well documented for users, and there should probably be an expiration time for caches that's based on how often data is updated on the remote service. 14 | 15 | 1. Try to send correct requests by knowing what the API expects and validating user inputs; at the correct rate. 16 | * For instance, don't even try interacting with a web API requiring authentication if the user does not provide authentication information. 17 | * For limiting rate (not sending too many requests), automatically wait. If the API docs allow you to define an ideal or maximal rate, set the request rate in advance using the [ratelimitr](https://github.com/tarakc02/ratelimitr) package (or, with `{httr2}`, `httr2::req_throttle()`). 18 | 19 | 2. If there's a status API (a separate API indicating whether the web resource is up or down), use it. If it tells you the API is down, `stop()` (or `rlang::abort()`) with an informative error message. 20 | 21 | 3. If the API indicates an error, depending on the actual error, 22 | 23 | - If the *server* seems to be having issues, [re-try with an exponential back-off](). In `{httr2}` there is `httr2::req_retry()`. 24 | 25 | - Otherwise, [transform the error into a useful error](https://httr2.r-lib.org/articles/wrapping-apis.html#error-handling-1). 26 | 27 | - If you used retry and nothing was sent after the maximal number of retries, show an informative error message. 28 | 29 | That was it for aspects the user will care about. Now, what might be more problematic for your package's fate on CRAN are the automatic checks that happen there at submission and then [regularly](https://blog.r-hub.io/2019/04/25/r-devel-linux-x86-64-debian-clang/#cran-checks-101). 30 | 31 | ## Graceful vignettes and examples 32 | 33 | 4. [Pre-compute vignettes](https://blog.r-hub.io/2020/06/03/vignettes/#how-to-include-a-compute-intensive--authentication-dependent-vignette) in some way. Don't use them as tests; they are a showcase. Of course have a system to prevent them from going stale, maybe even simple reminders (potentially in the [unexported `release_questions()` function](https://devtools.r-lib.org/reference/release.html#details)). Don't let vignettes run on a system where a failure has bad consequences. 34 | 5. Don't run [examples](https://blog.r-hub.io/2020/01/27/examples/) on CRAN. Now, for a first submission, CRAN maintainers might complain if there is no example. In that case, you might want to add some minimal example, e.g. 35 | 36 | ```r 37 | if (crul::ok("some-url")) { 38 | my_fun() # some eg that uses some-url 39 | } 40 | ``` 41 | 42 | These two precautions ensure that CRAN checks won't end with some WARNINGs, e.g. because an example failed when the API was down. 43 | 44 | ## Graceful code 45 | 46 | For simplifying your own life and those of contributors, make sure to re-use code in your package by e.g. defining helper functions for making requests, handling responses etc. 47 | It will make it easier for you to support interactions with more parts of the web API. 48 | Writing DRY (don't repeat yourself) code means less lines of code to test, and less API calls to make or fake! 49 | 50 | Also, were you to export a function à la `gh::gh()`, you'll help users call any endpoint of the web API even if you haven't written any high-level helper for it yet. 51 | 52 | ## Graceful tests 53 | 54 | We're getting closer to the actual topic of this book! 55 | 56 | 6. Read the rest of this book! Your tests should ideally run without needing an actual internet connection nor the API being up. Your tests that do need to interact with the API should be skipped on CRAN. `testthat::skip_on_cran()` (or `skip_if_offline()` that skips if the test is run offline or on CRAN) will ensure that. 57 | 7. Do not only test "success" behavior! Test for the behavior of your package in case of API errors, which shall also be covered later in the book. 58 | 59 | ## Conclusion 60 | 61 | In summary, to have a graceful HTTP package, make the most of current best practice for the user interface; escape examples and vignettes on CRAN; make tests independent of actual HTTP requests. Do not forget CRAN's "graceful failure" policy is mostly about ensuring a clean R CMD check result on CRAN platforms (0 ERROR, 0 WARNING, 0 NOTE) even when the web service you're wrapping has some hiccups. 62 | -------------------------------------------------------------------------------- /intro-pkgs.Rmd: -------------------------------------------------------------------------------- 1 | ```{r echo = FALSE} 2 | knitr::opts_chunk$set( 3 | comment = "#>", 4 | warning = FALSE, 5 | message = FALSE 6 | ) 7 | ``` 8 | 9 | # Packages for HTTP testing {#pkgs-testing-chapter} 10 | 11 | A brief presentation of packages you'll "meet" again later in this book! 12 | 13 | ## Why do we need special packages for HTTP testing? 14 | 15 | Packages for HTTP testing are useful because there are challenges to HTTP testing. 16 | Packages for HTTP testing help you solve these challenges, rather than letting you solve them with some homegrown solutions (you can still choose to do that, of course). 17 | 18 | What are the challenges of HTTP testing? 19 | 20 | * Having tests depend on an internet connection is not ideal. 21 | * Having tests depend on having secrets for authentication at hand is not ideal. 22 | * Having tests for situations that are hard to trigger (e.g. the failure of a remote server) is tricky. 23 | 24 | ## webmockr {#webmockr} 25 | 26 | `{webmockr}`, maintained by Scott Chamberlain, is an R package to help you "mock" HTTP requests. What does "mock" mean? Mock refers to the fact that we're faking the response. Here is how it works: 27 | 28 | * You "stub" a request. That is, you set rules for what HTTP request you'd like to respond to with a fake response. E.g. a rule might be a method, or a URL. 29 | * You also can set rules for what fake response you'd like to respond with, if anything (if nothing, then we give you `NULL`). 30 | * Then you make HTTP requests, and those that match your stub i.e. set of rules will return what you requested be returned. 31 | * While `{webmockr}` is in use, real HTTP interactions are not allowed. Therefore you need to stub all possible HTTP requests happening via your code. You'll get error messages for HTTP requests not covered by any stub. 32 | * There is no recording interactions to disk at all, just mocked responses given as the user specifies in the R session. 33 | 34 | `{webmockr}` works with `{crul}`, `{httr}`, and `{httr2}`. 35 | 36 | `{webmockr}` is quite low-level and not the first tool you'll use directly in your day-to-day HTTP testing. 37 | You may never use it directly but if you use `{vcr}` it's one of its foundations. 38 | 39 | `{webmockr}` was inspired by the [Ruby webmock gem](https://github.com/bblimke/webmock). 40 | 41 | ## What is vcr? {#what-vcr} 42 | 43 | The short version is `{vcr}`, maintained by Scott Chamberlain, helps you stub HTTP requests so you don't have to repeat HTTP requests, mostly in your unit tests. 44 | It uses the power of `{webmockr}`, with a higher level interface. 45 | 46 | When using `{vcr}` in tests, the first time you run a test, the API response is stored in a YAML or JSON file. 47 | All subsequent runs of the test use that local file instead of really calling the API. 48 | Therefore tests work independently of an internet connection. 49 | 50 | `{vcr}` was inspired by the [Ruby vcr gem](https://relishapp.com/vcr/vcr/docs). 51 | 52 | `{vcr}` works for packages using `{httr}`, `{httr2}` or `{crul}`. 53 | 54 | ::: {.alert .alert-dismissible .alert-info} 55 | [Direct link to `{vcr}` (& `{webmockr}`) demo](#vcr) 56 | ::: 57 | 58 | ## What is httptest? 59 | 60 | `{httptest}`, maintained by Neal Richardson, uses mocked API responses (like `{vcr}`). 61 | It _"enables one to test all of the logic on the R sides of the API in your package without requiring access to the remote service."_ 62 | 63 | Contrary to `{vcr}`, `{httptest}` also lets you define mock files by hand (copying from API docs, or dumbing down real responses), whereas with `{vcr}` all mock files come from recording real interactions (although you can choose to [edit `{vcr}` mock files](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html) after recording). 64 | 65 | `{httptest}` works for packages using `{httr}`. 66 | 67 | ::: {.alert .alert-dismissible .alert-info} 68 | [Direct link to `{httptest}` demo](#httptest) 69 | ::: 70 | 71 | The differences and similarities between `{httptest}` and `{vcr}` will become clearer in the chapters where we provide the whole games for each of them. 72 | 73 | ## What is httptest2? 74 | 75 | `{httptest2}`, maintained by Neal Richardson, is like `{httptest}`, but for `{httr2}`. 76 | 77 | 78 | ::: {.alert .alert-dismissible .alert-info} 79 | [Direct link to `{httptest2}` demo](#httptest2) 80 | ::: 81 | 82 | ::: {.alert .alert-dismissible .alert-primary} 83 | With `{vcr}`, `{httptest}` and `{httptest2}` the tests will use some sort of fake API responses. 84 | 85 | In `{vcr}` they are called **fixtures** or **cassettes**. 86 | In `{httptest}` and `{httptest2}` they are called **mock files**. 87 | ::: 88 | 89 | ## What is webfakes? 90 | 91 | `{webfakes}`, maintained by Gábor Csárdi, provides an alternative (complementary?) tool for HTTP testing. 92 | It will let you fake a whole web service, potentially outputting responses from mock files you'll have created. 93 | It does not help with recording fake responses. 94 | Because it runs a fake web service, you can even interact with said web service in your browser or with curl in the command line. 95 | 96 | `{webfakes}` works with packages using any HTTP package (i.e. it works with `{curl}`, `{crul}`, `{httr}`, or `{httr2}`). 97 | 98 | ::: {.alert .alert-dismissible .alert-info} 99 | [Direct link to `{webfakes}` demo](#webfakes) 100 | ::: 101 | 102 | ## testthat 103 | 104 | `{testthat}`, maintained by Hadley Wickham, is not a package specifically for HTTP testing; it is a package for general-purpose unit testing of R packages. 105 | In this book we will assume that is what you use, because of its popularity. 106 | 107 | If you use an alternative like `{tinytest}`, 108 | 109 | * `{httptest}` won't work as it's specifically designed as a complement to `{testthat}`; 110 | 111 | * `{vcr}` [might](https://github.com/ropensci/vcr/issues/162) work; 112 | 113 | * `{webfakes}` can work. 114 | 115 | ## Conclusion 116 | 117 | Now that you have an idea of the tools we can use for HTTP testing, we'll now create a minimal package and then amend it in three versions tested with 118 | 119 | * `{vcr}` and `{webmockr}`; 120 | * `{httptest}`; 121 | * `{httptest2}`; 122 | * `{webfakes}`. 123 | 124 | Our minimal package will use `{httr}` (except for `{httptest2}`, where we'll use `{httr2}`). However, it will help you understand concepts even if you end up using `{crul}` or `{curl}`.[^limits] 125 | 126 | 127 | [^limits]: If you end up using `{crul}`, you can use `{vcr}` and `{webmockr}`; or `{webfakes}`; but not `{httptest}`. If you end up using `{curl}` you can only use `{webfakes}`. 128 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | CC0 1.0 Universal 2 | 3 | Statement of Purpose 4 | 5 | The laws of most jurisdictions throughout the world automatically confer 6 | exclusive Copyright and Related Rights (defined below) upon the creator and 7 | subsequent owner(s) (each and all, an "owner") of an original work of 8 | authorship and/or a database (each, a "Work"). 9 | 10 | Certain owners wish to permanently relinquish those rights to a Work for the 11 | purpose of contributing to a commons of creative, cultural and scientific 12 | works ("Commons") that the public can reliably and without fear of later 13 | claims of infringement build upon, modify, incorporate in other works, reuse 14 | and redistribute as freely as possible in any form whatsoever and for any 15 | purposes, including without limitation commercial purposes. These owners may 16 | contribute to the Commons to promote the ideal of a free culture and the 17 | further production of creative, cultural and scientific works, or to gain 18 | reputation or greater distribution for their Work in part through the use and 19 | efforts of others. 20 | 21 | For these and/or other purposes and motivations, and without any expectation 22 | of additional consideration or compensation, the person associating CC0 with a 23 | Work (the "Affirmer"), to the extent that he or she is an owner of Copyright 24 | and Related Rights in the Work, voluntarily elects to apply CC0 to the Work 25 | and publicly distribute the Work under its terms, with knowledge of his or her 26 | Copyright and Related Rights in the Work and the meaning and intended legal 27 | effect of CC0 on those rights. 28 | 29 | 1. Copyright and Related Rights. A Work made available under CC0 may be 30 | protected by copyright and related or neighboring rights ("Copyright and 31 | Related Rights"). Copyright and Related Rights include, but are not limited 32 | to, the following: 33 | 34 | i. the right to reproduce, adapt, distribute, perform, display, communicate, 35 | and translate a Work; 36 | 37 | ii. moral rights retained by the original author(s) and/or performer(s); 38 | 39 | iii. publicity and privacy rights pertaining to a person's image or likeness 40 | depicted in a Work; 41 | 42 | iv. rights protecting against unfair competition in regards to a Work, 43 | subject to the limitations in paragraph 4(a), below; 44 | 45 | v. rights protecting the extraction, dissemination, use and reuse of data in 46 | a Work; 47 | 48 | vi. database rights (such as those arising under Directive 96/9/EC of the 49 | European Parliament and of the Council of 11 March 1996 on the legal 50 | protection of databases, and under any national implementation thereof, 51 | including any amended or successor version of such directive); and 52 | 53 | vii. other similar, equivalent or corresponding rights throughout the world 54 | based on applicable law or treaty, and any national implementations thereof. 55 | 56 | 2. Waiver. To the greatest extent permitted by, but not in contravention of, 57 | applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and 58 | unconditionally waives, abandons, and surrenders all of Affirmer's Copyright 59 | and Related Rights and associated claims and causes of action, whether now 60 | known or unknown (including existing as well as future claims and causes of 61 | action), in the Work (i) in all territories worldwide, (ii) for the maximum 62 | duration provided by applicable law or treaty (including future time 63 | extensions), (iii) in any current or future medium and for any number of 64 | copies, and (iv) for any purpose whatsoever, including without limitation 65 | commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes 66 | the Waiver for the benefit of each member of the public at large and to the 67 | detriment of Affirmer's heirs and successors, fully intending that such Waiver 68 | shall not be subject to revocation, rescission, cancellation, termination, or 69 | any other legal or equitable action to disrupt the quiet enjoyment of the Work 70 | by the public as contemplated by Affirmer's express Statement of Purpose. 71 | 72 | 3. Public License Fallback. Should any part of the Waiver for any reason be 73 | judged legally invalid or ineffective under applicable law, then the Waiver 74 | shall be preserved to the maximum extent permitted taking into account 75 | Affirmer's express Statement of Purpose. In addition, to the extent the Waiver 76 | is so judged Affirmer hereby grants to each affected person a royalty-free, 77 | non transferable, non sublicensable, non exclusive, irrevocable and 78 | unconditional license to exercise Affirmer's Copyright and Related Rights in 79 | the Work (i) in all territories worldwide, (ii) for the maximum duration 80 | provided by applicable law or treaty (including future time extensions), (iii) 81 | in any current or future medium and for any number of copies, and (iv) for any 82 | purpose whatsoever, including without limitation commercial, advertising or 83 | promotional purposes (the "License"). The License shall be deemed effective as 84 | of the date CC0 was applied by Affirmer to the Work. Should any part of the 85 | License for any reason be judged legally invalid or ineffective under 86 | applicable law, such partial invalidity or ineffectiveness shall not 87 | invalidate the remainder of the License, and in such case Affirmer hereby 88 | affirms that he or she will not (i) exercise any of his or her remaining 89 | Copyright and Related Rights in the Work or (ii) assert any associated claims 90 | and causes of action with respect to the Work, in either case contrary to 91 | Affirmer's express Statement of Purpose. 92 | 93 | 4. Limitations and Disclaimers. 94 | 95 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 96 | surrendered, licensed or otherwise affected by this document. 97 | 98 | b. Affirmer offers the Work as-is and makes no representations or warranties 99 | of any kind concerning the Work, express, implied, statutory or otherwise, 100 | including without limitation warranties of title, merchantability, fitness 101 | for a particular purpose, non infringement, or the absence of latent or 102 | other defects, accuracy, or the present or absence of errors, whether or not 103 | discoverable, all to the greatest extent permissible under applicable law. 104 | 105 | c. Affirmer disclaims responsibility for clearing rights of other persons 106 | that may apply to the Work or any use thereof, including without limitation 107 | any person's Copyright and Related Rights in the Work. Further, Affirmer 108 | disclaims responsibility for obtaining any necessary consents, permissions 109 | or other rights required for any use of the Work. 110 | 111 | d. Affirmer understands and acknowledges that Creative Commons is not a 112 | party to this document and has no duty or obligation with respect to this 113 | CC0 or use of the Work. 114 | 115 | For more information, please see 116 | 117 | 118 | -------------------------------------------------------------------------------- /wholegames-comparison.Rmd: -------------------------------------------------------------------------------- 1 | # vcr (& webmockr), httptest, webfakes {#pkgs-comparison} 2 | 3 | We're now at a nice stage where we have made a demo of usage for each of the HTTP testing packages, in our exemplighratia package. 4 | Of course, the choice of strategy in the demo is a bit subjective, but we hope it showed the best of each tool. 5 | 6 | ::: {.alert .alert-dismissible .alert-primary} 7 | A first message that's important to us: if you're learning about HTTP testing and using it in a branch of your own package sounds daunting, create a minimal package for playing! 8 | ::: 9 | 10 | ## What HTTP client can you use (curl, httr, httr2, crul) 11 | 12 | * httptest only works with httr (the most popular HTTP R client); 13 | * vcr (& webmockr) works with httr, httr2, and crul (the three "high-level" HTTP R clients); 14 | * webfakes works with any R HTTP client, even base R if you wish. 15 | 16 | ## Sustainability of the packages 17 | 18 | All packages (vcr, webmockr, httptest, webfakes) are actively maintained. 19 | During the writing of this book, issues and pull requests were tackled rather quickly, and always in a very nice way. 20 | 21 | ## Test writing experience 22 | 23 | In all cases having HTTP tests, i.e. tests that work independently from any internet connection, depends on 24 | 25 | * setup, which is mainly adding a dependency on the HTTP testing packages in DESCRIPTION, and a setup or helper file; 26 | * providing responses from the API. 27 | 28 | The difference between packages, the _test writing experience_ depends on how you can provide responses from the API, both real ones and fake ones. 29 | 30 | With vcr and httptest for tests testing _normal behavior_, after set up (for which there is a helper function), testing is just a function away (`vcr::use_cassette()`, `httptest::with_mock_dir()`, `httptest::with_mock_api()`). 31 | Recording happens automatically during the first run of tests. 32 | You might also provide fake recorded response or dumb down the existing ones. 33 | For creating _API errors_, and API sequence of responses (e.g. 502 then 200), you end up either using webmockr, or amending mock files, see [vcr](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html) and httptest related docs.[^seq-httptest] 34 | 35 | [^seq-httptest]: Sequence of requests are not [supported smoothly yet by httptest](https://github.com/nealrichardson/httptest/issues/49). 36 | 37 | With webfakes you need to create an app. 38 | There could be one per test, per test file or for the whole test suite. 39 | It might seem like more overhead code but being able to share an app between different tests reduces this effort. 40 | You can test for an [API sequence of responses (e.g. 502 then 200)](https://r-lib.github.io/webfakes/dev/articles/how-to.html#how-do-i-test-a-sequence-of-requests-) by following an how-to. 41 | The one thing that's not supported in webfakes yet is a smooth workflow for recording responses, so at the time of writing you might need to write your own workflow for recording responses. 42 | 43 | **In general setup&test writing might be easier for packages with mocking (vcr and httptest) but you might be able to replicate more complex behavior with webfakes (such as an [OAuth dance](https://r-lib.github.io/webfakes/articles/oauth.html)).** 44 | 45 | ### The special case of secrets 46 | 47 | With webfakes as no authentication is needed at any point, you have less chance of exposing a secret. 48 | 49 | With httptest only the body of responses is saved, so unless it contains secrets, no further effort is needed. If you _need_ to redact mock files, see [the corresponding vignette](https://enpiar.com/r/httptest/articles/redacting.html). 50 | 51 | With vcr as all HTTP interactions, including request URLs and headers, are saved to disk, you will most often have to use the `filter_sensitive_data`, `filter_request_header` and/or `filter_response_header` arguments of `vcr::vcr_configure()`. 52 | 53 | ### How about making real requests 54 | 55 | In all three cases, switching back to real requests might be an environment variable away (turning vcr off, setting the URL of the real web service as URL to be connected to instead of a webfakes fake web service). 56 | However, your tests using fixed/fake responses / a fake web service might not work with real requests as you can't trigger an API error, and as you might test for specific values in your tests using mock files whereas the API returns something different every day. 57 | Therefore, and it's a challenge common to all three packages, you might need to choose to have _distinct_ tests as integration tests/[contract tests](https://www.martinfowler.com/bliki/ContractTest.html). 58 | See also our chapter about [making real requests](#real-requests-chapter). 59 | 60 | ## Test debugging experience 61 | 62 | Sadly sometimes one needs to run code from the tests in an interactive session, either to debug tests after making a code change, or to learn how to write HTTP tests. 63 | 64 | With webfakes, debugging works this way: load the helper or test file where 65 | 66 | * the app is created, 67 | * the environment variable connecting your package code to the fake web service is changed. 68 | 69 | Then run your code. To debug _webfakes apps_, follow the [guidance](https://r-lib.github.io/webfakes/dev/articles/how-to.html#how-can-i-debug-an-app-). 70 | 71 | With vcr, refer to the [debugging](https://docs.ropensci.org/vcr/articles/debugging.html) vignette: you'll have to load the helper file or source the setup file after making sure the paths use in it work both from `tests/testthat/` and the package root (see `?vcr::vcr_test_path`), and then use `vcr::inject_cassette()`; don't forget to run `vcr::eject_cassette()` afterwards. 72 | With webmockr debugging is quite natural, run the code that's in the test, in particular `webmockr::enable()` and `webmockr::disable()`. 73 | 74 | With httptest, the process is similar as with vcr except the key functions are 75 | 76 | * [`use_mock_api()`](https://enpiar.com/r/httptest/reference/use_mock_api.html) 77 | * [.mockPaths](https://enpiar.com/r/httptest/reference/mockPaths.html). 78 | 79 | ## Conclusion 80 | 81 | In this chapter we compared the three R packages that make HTTP testing easier. 82 | If you are still unsure which one to pick, first try packages out without commitment, in branches or so, but then choose one and [commit to your lock-in](https://vickiboykis.com/2019/02/10/commit-to-your-lock-in/). 83 | 84 | > "Every piece of code written in a given language or framework is a step away from any other language, and five more minutes you’ll have to spend migrating it to something else. That’s fine. You just have to decide what you’re willing to be locked into. 85 | > 86 | > (...) 87 | > 88 | > Code these days becomes obsolete so quickly, regardless of what’s chosen. By the time your needs change, by the time the latest framework is obsolete, all of the code will be rotten anyway 89 | > 90 | > (...) 91 | > 92 | > The most dangerous feature about these articles examining cloud lock-in is that they introduce a kind of paralysis into teams that result in applications never being completely fleshed out or finished." 93 | > 94 | > Vicki Boykis, "Commit to your lock-in". 95 | -------------------------------------------------------------------------------- /wholegames-mocking.Rmd: -------------------------------------------------------------------------------- 1 | # vcr and httptest {#mocking-pkgs-comparison} 2 | 3 | We have just followed very similar processes to add HTTP testing infrastructure involving mock files to exemplighratia 4 | 5 | * Adding a package as a Suggests dependency; 6 | * Creating a helper file that in particular loads this package before each test; 7 | * Tweaking tests, in some cases wrapping our tests into functions that allows to record API responses in mock files and to play them back from said mock files; in other cases (only with httptest), creating mock files ourselves. 8 | 9 | Now, there were a few differences. 10 | We won't end up advocating for one package in particular since both have their merits, but we do hope to help you differentiate the two packages. 11 | 12 | ## Setting up the infrastructure 13 | 14 | To set up the HTTP testing infrastructure, in one case you need to run `vcr::use_vcr()` and in another case you need to run `httptest::use_httptest()`. Not too hard to remember. 15 | 16 | ## Calling mock files 17 | 18 | As mentioned before, vcr and httptest both use mock files but they call them differently. 19 | 20 | In vcr they are called both **fixtures** and **cassettes**. 21 | In httptest they are called **mock files**. 22 | Note that fixtures is not as specific as cassettes and mock files: cassettes and mock files are fixtures, but anything (a csv file of input for instance) you use to consistently test your package is a fixture. 23 | 24 | ## Naming mock files 25 | 26 | With vcr the `use_cassette()` call needs to include a name that will be used to create the filename of the mock file. 27 | The help of `?use_cassette` explains some criteria for naming them, such as the fact that cassette names need to be unique. 28 | Now if you wrap your whole `test_that()` block in them you might just as well use a name similar to the test name, and you already make those meaningful, right? 29 | 30 | With httptest the mock filepaths are translated from requests according to several rules that incorporate the request method, URL, query parameters, and body. 31 | If you use `with_mock_dir()` you need a name for the directory under which the mock files are saved, and you can make it meaningful. 32 | 33 | Also note that with vcr one file can (but does not have to) contain several HTTP interactions (requests and responses) whereas with httptest one file contains one response only (and the filename helps matching it to a request). 34 | 35 | ## Matching requests 36 | 37 | With httptest as the mock file name includes everything that's potentially varying about a request, each mock file corresponds to one request only. 38 | 39 | With vcr, there are different possible [configurations for matching a request to a saved interaction](https://docs.ropensci.org/vcr/articles/request_matching.html) but by default you can mostly expect that one saved interaction corresponds to one request only. 40 | 41 | ## Handling secrets 42 | 43 | With vcr, since everything from the HTTP interactions is recorded, you always need to add some sort of configuration to be sure to wipe your API tokens from the mock files. 44 | 45 | With httptest, only responses are saved, and most often, only their bodies. 46 | Most often, responses don't contain secrets e.g. they don't contain your API token. 47 | If the response contains secrets, refer to httptest's article about ["Redacting sensitive information"](https://enpiar.com/r/httptest/articles/redacting.html). 48 | 49 | ## Recording, playing back 50 | 51 | When using mock files for testing, first you need to record responses in mock files; and then you want to use the mock files instead of real HTTP interactions (that's the whole point). 52 | 53 | With vcr, the recording vs playing back modes happen automatically depending on the existence of the cassette. If you write `vcr::use_cassette("blabla", )` and there's no cassette called blabla, vcr will create it. Note that if you change the HTTP interactions in the code block, you'll have to re-record the cassette which is as simple as deleting it then running the test. _Note that you can also change the way vcr behaves by looking into `?vcr::vcr_configure`'s "Cassette Options"._ 54 | 55 | With httptest, there is a lot of flexibility around how to record mock files. It is because httptest doesn't assume that every API mock came from a real request to a real server; maybe you copy some of the mocks directly from the API docs. 56 | 57 | **Note that nothing prevents you from editing vcr cassettes by hand, but you'll have to be careful not re-recording them by mistake.** 58 | 59 | ::: {.alert .alert-dismissible .alert-info} 60 | httptest flexiblity comes from [original design principles of httptest](https://github.com/nealrichardson/httptest/issues/40#issuecomment-708672654) 61 | 62 | > _"[httptest] doesn't assume that every API mock came from a real request to a real server, and it is designed so that you are able to see and modify test fixtures. 63 | Among the considerations:_ 64 | > 65 | > _1. In many cases, API responses contain way more content than is necessary to test your R code around them: 100 records when 2 will suffice, request metadata that you don't care about and can't meaningfully assert things about, and so on. In the interest of minimally reproducible examples, and of making tests readable, it often makes sense to take an actual API response and delete a lot of its content, or even to fabricate one entirely._ 66 | > 67 | > _2. And then it's good to keep that API mock fixed so you know exactly what is in it. If I re-recorded a Twitter API response of, say, the most recent 10 tweets with #rstats, the specific content will change every time I record it, so my tests can't say much about what is in the response without having to rewrite them every time too._ 68 | > 69 | > _3. Some conditions (rate limiting, server errors, e.g.) are difficult to test with real responses, but if you can hand-create a API mock with, say, a 503 response status code and test how your code handles it, you can have confidence of how your package will respond when that rare event happens with the real API._ 70 | > 71 | > _4. Re-recording all responses can make for a huge code diff, which can blow up your repository size and make code review harder."_ 72 | ::: 73 | 74 | Now, creating mock files by hand (or inventing some custom scripts to create them) involves more elbow grease, so it's a compromise. 75 | 76 | ## Testing for API errors 77 | 78 | In your test suite you probably want to check how things go if the server returns 502 or so, and you cannot trigger such a response to record it. 79 | 80 | With httptest, to test for API errors, you need to create one or several fake mock file(s). 81 | The easiest way to do that might be to use `httptest::with_mock_dir()` that will create mock files with the expected filenames and locations, that you can then tweak. 82 | Or reading the error message of `httptest::with_mock_ap()` helps you know where to create a mock file. 83 | 84 | With vcr, you either 85 | 86 | * use webmockr as we showed in our demo. On the one hand it's more compact than creating a fake mock file, on the other hand it's a way to test that's different from the vcr cassette. 87 | 88 | ```r 89 | test_that("gh_organizations errors when the API doesn't behave", { 90 | webmockr::enable() 91 | stub <- webmockr::stub_request("get", "https://api.github.com/organizations?since=1") 92 | webmockr::to_return(stub, status = 502) 93 | expect_error(gh_organizations(), "oops") 94 | webmockr::disable() 95 | }) 96 | ``` 97 | 98 | * or you edit a cassette by hand which would be similar to testing for API errors with httptest. If you did that, you'd need to skip the test when vcr is off, as when vcr is off real requests are made. For that you can use `vcr::skip_if_vcr_off()`. 99 | 100 | ## Conclusion 101 | 102 | Both vcr and httptest are similar packages in that they use mock files for allowing easier HTTP testing. 103 | They are a bit different in their design philosophy and features, which might help you choose one of them. 104 | 105 | And now, to make things even more complex, or fun, we shall explore a third HTTP testing package that does not _mock_ requests but instead spins up a local fake web service. -------------------------------------------------------------------------------- /wholegames-presser.Rmd: -------------------------------------------------------------------------------- 1 | # Use webfakes {#webfakes} 2 | 3 | In this chapter we aim at adding HTTP testing infrastructure to exemplighratia using webfakes. 4 | 5 | ## Setup 6 | 7 | Before working on all this, we need to install `{webfakes}`, with `install.packages("webfakes")`. 8 | 9 | Then, we need to add webfakes as a Suggests dependency of our package, potentially via running `usethis::use_package("webfakes", type = "Suggests")`. 10 | 11 | Last but not least, we create a setup file at `tests/testthat/setup.R`. 12 | When testthat runs tests, [files whose name starts with "setup" are always run first](https://testthat.r-lib.org/reference/test_dir.html#special-files). 13 | We need to ensure that we set up a fake API key when there is no API token around. 14 | Why? Because if you remember well, the code of our function `gh_organizations()` checks for the presence of a token. 15 | When using our own fake web service, we obviously don't really need a token but we still need to fool our own package in contexts where there is no token (e.g. in continuous integration checks for a fork of a GitHub repository). 16 | 17 | ```r 18 | if(!nzchar(Sys.getenv("REAL_REQUESTS"))) { 19 | Sys.setenv("GITHUB_PAT" = "foobar") 20 | } 21 | ``` 22 | 23 | The setup file could also load webfakes, but in our demo we will namespace webfakes functions instead. 24 | 25 | ## Actual testing 26 | 27 | With webfakes we will be spinning local fake web services that we will want our package to interact with instead of the real APIs. 28 | Therefore, we first need to amend the code of functions returning URLs to services to be able to change them via an environment variable. 29 | They become: 30 | 31 | ```r 32 | status_url <- function() { 33 | 34 | env_url <- Sys.getenv("EXEMPLIGHRATIA_GITHUB_STATUS_URL") 35 | 36 | if (nzchar(env_url)) { 37 | return(env_url) 38 | } 39 | 40 | "https://kctbh9vrtdwd.statuspage.io/api/v2/components.json" 41 | } 42 | ``` 43 | 44 | and 45 | 46 | ```r 47 | gh_v3_url <- function() { 48 | 49 | api_url <- Sys.getenv("EXEMPLIGHRATIA_GITHUB_API_URL") 50 | 51 | if (nzchar(api_url)) { 52 | return(api_url) 53 | } 54 | 55 | "https://api.github.com/" 56 | } 57 | ``` 58 | 59 | Having these two switches is crucial. 60 | 61 | Then, let's tweak our test of `gh_api_status()`. 62 | 63 | ```r 64 | test_that("gh_api_status() works", { 65 | if (!nzchar(Sys.getenv("REAL_REQUESTS"))) { 66 | app <- webfakes::new_app() 67 | app$get("/", function(req, res) { 68 | res$send_json( 69 | list( components = 70 | list( 71 | list( 72 | name = "API Requests", 73 | status = "operational" 74 | ) 75 | ) 76 | ), 77 | auto_unbox = TRUE 78 | ) 79 | }) 80 | web <- webfakes::local_app_process(app, start = TRUE) 81 | web$local_env(list(EXEMPLIGHRATIA_GITHUB_STATUS_URL = "{url}")) 82 | } 83 | 84 | testthat::expect_type(gh_api_status(), "character") 85 | }) 86 | ``` 87 | 88 | So what's happening here? 89 | 90 | * When we're not asking for requests to the real service to be made (`Sys.getenv("REAL_REQUESTS")`), we prepare a new app via `webfakes::new_app()`. It's a very simple one, that returns, for GET requests, a list corresponding to what we're used to getting out of the status API, except that a) it's much smaller and b) the "operational" status is hard-coded. 91 | * When then create a local app process via `webfakes::local_app_process(, start = TRUE)`. It will start right away thanks to `start=TRUE` but we could have chosen to start it later via calling e.g. `web$url()` (see `?webfakes::local_app_process`); and most importantly it will be stopped automatically after the test. No mess made! 92 | * We set the `EXEMPLIGHRATIA_GITHUB_STATUS_URL` variable to the URL of the local app process. This is what connects our code to our fake web service. 93 | 94 | It might seem like a lot of overhead code but 95 | 96 | * It means no real requests are made which is our ultimate goal. 97 | * We will get used to it. 98 | * We can write helper code in a testthat helper file to not repeat ourselves in further test files; there could even be an app shared between all test files depending on your package. 99 | 100 | Now, let's add a test for error behavior. 101 | This inspired us to change error behavior a bit with a slightly more specific error message i.e. `httr::stop_for_status(response)` became `httr::stop_for_status(response, task = "get API status, ouch!")`. 102 | 103 | ```r 104 | test_that("gh_api_status() errors when the API does not behave", { 105 | app <- webfakes::new_app() 106 | app$get("/", function(req, res) { 107 | res$send_status(502L) 108 | }) 109 | web <- webfakes::local_app_process(app, start = TRUE) 110 | web$local_env(list(EXEMPLIGHRATIA_GITHUB_STATUS_URL = "{url}")) 111 | testthat::expect_error(gh_api_status(), "ouch") 112 | }) 113 | ``` 114 | 115 | It's a similar process to the earlier test: 116 | 117 | * setting up a new app; 118 | * having it return something we chose, in this case a 502 status; 119 | * launching a local app process; 120 | * connecting our code to it via setting the `EXEMPLIGHRATIA_GITHUB_STATUS_URL` environment variable to the URL of the fake service; 121 | * test. 122 | 123 | Last but not least let's convert our test of `gh_organizations()`, 124 | 125 | ```r 126 | test_that("gh_organizations works", { 127 | 128 | if (!nzchar(Sys.getenv("REAL_REQUESTS"))) { 129 | app <- webfakes::new_app() 130 | app$get("/organizations", function(req, res) { 131 | res$send_json( 132 | jsonlite::read_json( 133 | testthat::test_path( 134 | file.path("responses", "organizations.json") 135 | ) 136 | ), 137 | auto_unbox = TRUE 138 | ) 139 | }) 140 | web <- webfakes::local_app_process(app, start = TRUE) 141 | web$local_env(list(EXEMPLIGHRATIA_GITHUB_API_URL = "{url}")) 142 | } 143 | 144 | testthat::expect_type(gh_organizations(), "character") 145 | }) 146 | ``` 147 | 148 | As before we 149 | 150 | * create a new app; 151 | * have it returned something we chose for a GET request of the `/organizations` endpoint. In this case, we have it return the content of a JSON file we created at `tests/testthat/responses/organizations.json` by copy-pasting a real response from the API; 152 | * launch a local app process; 153 | * set its URL as the `EXEMPLIGHRATIA_GITHUB_API_URL` environment variable; 154 | * test. 155 | 156 | ## Also testing for real interactions 157 | 158 | What if the API responses change? 159 | Hopefully we'd notice that thanks to following API news. 160 | However, sometimes web APIs change without any notice. 161 | Therefore it is important to run tests against the real web service once in a while. 162 | 163 | In our tests we have used the condition 164 | 165 | ```r 166 | if (!nzchar(Sys.getenv("REAL_REQUESTS"))) { 167 | ``` 168 | 169 | before launching the app and using its URL as URL for the service. 170 | So if our tests are generic enough, we can add a CI build where the environment variable `REAL_REQUESTS` is set to true. 171 | If they are not generic enough, we can use the[ approach exemplified in the chapter about httptest](#httptest-real). 172 | 173 | * set up a folder real-tests with tests interacting with the real web service; 174 | * add it to Rbuildignore; 175 | * in a CI build, delete tests/testthat and replace it with real-tests, before running R CMD check. 176 | 177 | ## Summary 178 | 179 | * We set up webfakes usage in our package exemplighratia by adding a dependency on webfakes and by adding a setup file to fool our own package that needs an API token. 180 | * We created and launched fake apps in our test files. 181 | 182 | Now, how do we make sure this works? 183 | 184 | * Turn off wifi, run the tests again. It works! Turn on wifi again. 185 | * Open .Renviron (`usethis::edit_r_environ()`), edit "GITHUB_PAT" into "byeGITHUB_PAT", re-start R, run the tests again. It works! Fix your "GITHUB_PAT" token in .Renviron. 186 | 187 | So we now have tests that no longer rely on an internet connection nor on having API credentials. 188 | 189 | For the full list of changes applied to exemplighratia in this chapter, see [the pull request diff on GitHub](https://github.com/ropensci-books/exemplighratia/pull/4/files). 190 | 191 | ::: {.alert .alert-dismissible .alert-primary} 192 | In the next chapter, we shall compare the three approaches to HTTP testing we've demo-ed. 193 | ::: 194 | -------------------------------------------------------------------------------- /wholegames-httptest.Rmd: -------------------------------------------------------------------------------- 1 | # Use httptest {#httptest} 2 | 3 | In this chapter we aim at adding HTTP testing infrastructure to exemplighratia using httptest. 4 | For this, we start from the initial state of exemplighratia again. 5 | Back to square one! 6 | 7 | ::: {.alert .alert-dismissible .alert-warning} 8 | Note that the `httptest::with_mock_dir()` function is only available in httptest version >= 4.0.0 (released on CRAN on 2021-02-01). 9 | ::: 10 | 11 | ::: {.alert .alert-dismissible .alert-info} 12 | [Corresponding pull request to exemplighratia](https://github.com/ropensci-books/exemplighratia/pull/9/files). 13 | Feel free to fork the repository to experiment yourself! 14 | ::: 15 | 16 | ## Setup 17 | 18 | Before working on all this, we need to install `{httptest}`. 19 | 20 | First, we need to run `httptest::use_httptest()` which has a few effects: 21 | 22 | * Adding httptest as a dependency to `DESCRIPTION`, under Suggests just like testthat. 23 | * Creating a setup file under `tests/testthat/setup`, 24 | 25 | ```r 26 | library(httptest) 27 | ``` 28 | 29 | When testthat runs tests, [files whose name starts with "setup" are always run first](https://testthat.r-lib.org/reference/test_dir.html#special-files). 30 | The setup file added by httptest loads httptest. 31 | 32 | We will tweak it a bit to fool our package into believing there is an API token around in contexts where there is not. 33 | Since tests will use recorded responses when we are not recording, we do not need an actual API token when not recording, but we need `gh_organizations()` to not stop because `Sys.getenv("GITHUB_PAT")` returns nothing. 34 | 35 | ```r 36 | library(httptest) 37 | 38 | # for contexts where the package needs to be fooled 39 | # (CRAN, forks) 40 | # this is ok because the package will used recorded responses 41 | # so no need for a real secret 42 | if (!nzchar(Sys.getenv("GITHUB_PAT"))) { 43 | Sys.setenv(GITHUB_PAT = "foobar") 44 | } 45 | ``` 46 | 47 | So this was just setup, now on to adapting our tests! 48 | 49 | ## Actual testing 50 | 51 | The key function will be `httptest::with_mock_dir("dir", {code-block})` which tells httptest to create mock files under `tests/testthat/dir` to store all API responses for API calls occurring in the code block. 52 | We are allowed to tweak the mock files by hand, and we will do that in some cases. 53 | 54 | Let's tweak the test file for `gh_status_api`, it becomes 55 | 56 | ```r 57 | with_mock_dir("gh_api_status", { 58 | test_that("gh_api_status() works", { 59 | expect_type(gh_api_status(), "character") 60 | expect_equal(gh_api_status(), "operational") 61 | }) 62 | }) 63 | ``` 64 | 65 | We only had to wrap the whole test in `httptest::with_mock_dir()`. 66 | 67 | If we run this test (in RStudio clicking on "Run test"), 68 | 69 | * the first time, httptest creates a mock file under `tests/testthat/gh_api_status/kctbh9vrtdwd.statuspage.io/api/v2/components.json.json` where it stores the API response. 70 | We however dumbed it down by hand, to 71 | 72 | ```json 73 | {"components":[{"name":"API Requests","status":"operational"}]} 74 | ``` 75 | 76 | * all the times after that, httptest simply uses the mock file instead of actually calling the API. 77 | 78 | Let's tweak our other test, of `gh_organizations()`. 79 | 80 | Here things get more exciting or complicated, as we also set out to adding a test of the error behavior. 81 | This inspired us to change error behavior a bit with a slightly more specific error message i.e. `httr::stop_for_status(response)` became `httr::stop_for_status(response, task = "get data from the API, oops")`. 82 | 83 | The test file ` tests/testthat/test-organizations.R` is now: 84 | 85 | ```r 86 | with_mock_dir("gh_organizations", { 87 | test_that("gh_organizations works", { 88 | expect_type(gh_organizations(), "character") 89 | }) 90 | }) 91 | 92 | with_mock_dir("gh_organizations_error", { 93 | test_that("gh_organizations errors if the API doesn't behave", { 94 | expect_error(gh_organizations()) 95 | }) 96 | }, 97 | simplify = FALSE) 98 | ``` 99 | 100 | The first test is similar to what we did for `gh_api_status()` except we didn't touch the mock file this time, out of laziness. 101 | In the second test there is more to unpack: how do we get a mock file corresponding to an error? 102 | 103 | * We first run the test as is. It fails because there is no error, which we expected. Note the `simplify = FALSE` that means the mock file also contains headers for the response. 104 | * We replaced `200L` with `502L` and removed the body, to end up with a very simple mock file under ` tests/testthat/gh_organizations_error/api.github.com/organizations-5377e8.R` 105 | 106 | ```r 107 | structure(list( 108 | url = "https://api.github.com/organizations?since=1", 109 | status_code = 502L, 110 | headers = NULL), 111 | class = "response") 112 | ``` 113 | 114 | * We re-run the tests. We got the expected error message. 115 | 116 | Without the HTTP testing infrastructure, testing for behavior of the package in case of API errors would be more difficult. 117 | 118 | Regarding our secret API token, since httptest doesn't save the requests, and since the responses don't contain the token, it is safe without our making any effort. 119 | 120 | ::: {.alert .alert-dismissible .alert-primary} 121 | In this demo we used `httptest::with_mock_dir()` but there are other ways to use httptest, e.g. using `httptest::with_mock_api()` that does not require naming a directory (you'd still need to use a separate directory for mocking the error response). 122 | 123 | Find out more in the [main httptest vignette](https://enpiar.com/r/httptest/articles/httptest.html). 124 | ::: 125 | 126 | ## Also testing for real interactions {#httptest-real} 127 | 128 | What if the API responses change? 129 | Hopefully we'd notice that thanks to following API news. 130 | However, sometimes web APIs change without any notice. 131 | Therefore it is important to run tests against the real web service once in a while. 132 | 133 | As with vcr we setup a [GitHub Actions workflow](https://github.com/ropensci-books/exemplighratia/blob/otherhttptestapproach/.github/workflows/R-CMD-check-schedule.yaml) that runs once a week with tests against the real web service. 134 | The difference is what and where these tests are. 135 | As some tests with custom made mock files can be more specific (e.g. testing for actual values, whereas the latest responses from the API will have different values), instead of turning off mock files usage, we use our old original tests that we put in a folder called `real-tests`. 136 | Most of the time `real-tests` is .Rbuildignored but in the scheduled run, before checking the package we replace the content of `tests` with `real-tests`. 137 | An alternative would be to use `testthat::test_dir()` on that directory but in case of failures we would not get artifacts as we do with `R CMD check` (at least not without further effort). 138 | 139 | Again, one could imagine other strategies, but in all cases it is important to keep checking the package against the real web service fairly regularly. 140 | 141 | ## Summary 142 | 143 | * We set up httptest usage in our package exemplighratia by running `use_httptest()` and tweaking the setup file to fool our own package that needs an API token. 144 | * We wrapped `test_that()` into `httptest::with_mock_dir()` and ran the tests a first time to generate mock files that hold all information about the API responses. In some cases we modified these mock files to make them smaller or to make them correspond to an API error. 145 | 146 | Now, how do we make sure this works? 147 | 148 | * Turn off wifi, run the tests again. It works! Turn on wifi again. 149 | * Open .Renviron (`usethis::edit_r_environ()`), edit "GITHUB_PAT" into "byeGITHUB_PAT", re-start R, run the tests again. It works! Fix your "GITHUB_PAT" token in .Renviron. 150 | 151 | So we now have tests that no longer rely on an internet connection nor on having API credentials. 152 | 153 | We also added a continuous integration workflow for having a build using real interactions once every week, as it is important to regularly make sure the package still works against the latest API responses. 154 | 155 | For the full list of changes applied to exemplighratia in this chapter, see [the pull request diff on GitHub](https://github.com/ropensci-books/exemplighratia/pull/9/files). 156 | 157 | ::: {.alert .alert-dismissible .alert-primary} 158 | How do we get there with yet another package? We'll try webfakes but first let's compare vcr and httptest as they both use mocking. 159 | ::: 160 | -------------------------------------------------------------------------------- /wholegames-httptest2.Rmd: -------------------------------------------------------------------------------- 1 | # Use httptest2 {#httptest2} 2 | 3 | In this chapter we aim at adding HTTP testing infrastructure to exemplighratia2 using httptest2. 4 | For this, we start from the initial state of exemplighratia2 again. Back to square one! 5 | 6 | 7 | ::: {.alert .alert-dismissible .alert-info} 8 | [Corresponding pull request to exemplighratia2](https://github.com/ropensci-books/exemplighratia2/pull/1/files) Feel free to fork the repository to experiment yourself! 9 | ::: 10 | 11 | ## Setup 12 | 13 | Before working on all this, we need to install `{httptest2}`. 14 | 15 | First, we need to run `httptest2::use_httptest2()` which has a few effects: 16 | 17 | * Adding httptest2 as a dependency to `DESCRIPTION`, under Suggests just like testthat. 18 | * Creating a setup file under `tests/testthat/setup`, 19 | 20 | ```r 21 | library(httptest2) 22 | ``` 23 | 24 | When testthat runs tests, [files whose name starts with "setup" are always run first](https://testthat.r-lib.org/reference/test_dir.html#special-files). 25 | The setup file added by httptest2 loads httptest2. 26 | 27 | We shall tweak it a bit to fool our package into believing there is an API token around in contexts where there is not. Since tests will use recorded responses when we are not recording, we do not need an actual API token when not recording, but we need `gh_organizations()` to not stop because `Sys.getenv("GITHUB_PAT")` returns nothing. 28 | 29 | ```r 30 | library(httptest2) 31 | 32 | # for contexts where the package needs to be fooled 33 | # (CRAN, forks) 34 | # this is ok because the package will used recorded responses 35 | # so no need for a real secret 36 | if (!nzchar(Sys.getenv("GITHUB_PAT"))) { 37 | Sys.setenv(GITHUB_PAT = "foobar") 38 | } 39 | ``` 40 | 41 | So this was just setup, now on to adapting our tests! 42 | 43 | ## Actual testing 44 | 45 | The key function will be `httptest2::with_mock_dir("dir", {code-block})` which tells httptest to create mock files under `tests/testthat/dir` to store all API responses for API calls occurring in the code block. 46 | We are allowed to tweak the mock files by hand, and we will do that in some cases. 47 | 48 | Let's tweak the test file for `gh_status_api`, it becomes 49 | 50 | ```r 51 | with_mock_dir("gh_api_status", { 52 | test_that("gh_api_status() works", { 53 | testthat::expect_type(gh_api_status(), "character") 54 | testthat::expect_equal(gh_api_status(), "operational") 55 | }) 56 | }) 57 | ``` 58 | 59 | We only had to wrap the whole test in `httptest2::with_mock_dir()`. 60 | 61 | If we run this test (in RStudio clicking on "Run test"), 62 | 63 | * the first time, httptest2 creates a mock file under `tests/testthat/gh_api_status/kctbh9vrtdwd.statuspage.io/api/v2/components.json.json` where it stores the API response. 64 | We however dumbed it down by hand, to 65 | 66 | ```json 67 | {"components":[{"name":"API Requests","status":"operational"}]} 68 | ``` 69 | 70 | * all the times after that, httptest2 simply uses the mock file instead of actually calling the API. 71 | 72 | Let's tweak our other test, of `gh_organizations()`. 73 | 74 | Here things get more exciting or complicated, as we also set out to adding a test of the error behavior. 75 | 76 | The test file ` tests/testthat/test-organizations.R` is now: 77 | 78 | ```r 79 | with_mock_dir("gh_organizations", { 80 | test_that("gh_organizations works", { 81 | testthat::expect_type(gh_organizations(), "character") 82 | }) 83 | }) 84 | 85 | with_mock_dir("gh_organizations_error", { 86 | test_that("gh_organizations errors if the API doesn't behave", { 87 | testthat::expect_snapshot_error(gh_organizations()) 88 | }) 89 | }, 90 | simplify = FALSE) 91 | ``` 92 | 93 | The first test is similar to what we did for `gh_api_status()` except we didn't touch the mock file this time, out of laziness. 94 | In the second test there is more to unpack: how do we get a mock file corresponding to an error? 95 | 96 | * We first run the test as is. It fails because there is no error, which we expected. Note the `simplify = FALSE` that means the mock file also contains headers for the response. 97 | * We replaced `200L` with `502L` and removed the body, to end up with a simpler mock file under ` tests/testthat/gh_organizations_error/api.github.com/organizations-5377e8.R` 98 | 99 | ```r 100 | structure(list(method = "GET", url = "https://api.github.com/organizations?since=1", 101 | status_code = 502L, headers = structure(list(server = "GitHub.com", 102 | date = "Thu, 17 Feb 2022 12:40:29 GMT", `content-type` = "application/json; charset=utf-8", 103 | `cache-control` = "private, max-age=60, s-maxage=60", 104 | vary = "Accept, Authorization, Cookie, X-GitHub-OTP", 105 | etag = "W/\"d56e867402a909d66653b6cb53d83286ba9a16eef993dc8f3cb64c43b66389f4\"", 106 | `x-oauth-scopes` = "gist, repo, user, workflow", `x-accepted-oauth-scopes` = "", 107 | `x-github-media-type` = "github.v3; format=json", link = "; rel=\"next\", ; rel=\"first\"", 108 | `x-ratelimit-limit` = "5000", `x-ratelimit-remaining` = "4986", 109 | `x-ratelimit-reset` = "1645104327", `x-ratelimit-used` = "14", 110 | `x-ratelimit-resource` = "core", `access-control-expose-headers` = "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset", 111 | `access-control-allow-origin` = "*", `strict-transport-security` = "max-age=31536000; includeSubdomains; preload", 112 | `x-frame-options` = "deny", `x-content-type-options` = "nosniff", 113 | `x-xss-protection` = "0", `referrer-policy` = "origin-when-cross-origin, strict-origin-when-cross-origin", 114 | `content-security-policy` = "default-src 'none'", vary = "Accept-Encoding, Accept, X-Requested-With", 115 | `content-encoding` = "gzip", `x-github-request-id` = "A4BA:12D5C:178438:211160:620E423C"), class = "httr2_headers"), 116 | body = charToRaw("")), class = "httr2_response") 117 | 118 | ``` 119 | * We re-run the tests. We got the expected error message. 120 | 121 | Without the HTTP testing infrastructure, testing for behavior of the package in case of API errors would be more difficult. 122 | 123 | Regarding our secret API token, since httptest2 doesn't save the requests[^save], and since the responses don't contain the token, it is safe without our making any effort. 124 | 125 | [^save]: `httr2_response` objects, unlike the equivalent in httr, don't include the request. 126 | 127 | ::: {.alert .alert-dismissible .alert-primary} 128 | In this demo we used `httptest2::with_mock_dir()` but there are other ways to use httptest2, e.g. using `httptest2::with_mock_api()` that does not require naming a directory (you'd still need to use a separate directory for mocking the error response). 129 | 130 | Find out more in the [main httptest2 vignette](https://enpiar.com/httptest2/articles/httptest2.html). 131 | ::: 132 | 133 | ## Also testing for real interactions {#httptest2-real} 134 | 135 | What if the API responses change? 136 | Hopefully we'd notice that thanks to following API news. 137 | However, sometimes web APIs change without any notice. 138 | Therefore it is important to run tests against the real web service once in a while. 139 | 140 | One could use the same strategy as the one [we demonstrated for httptest](#httptest-real) i.e. with a different test folder. 141 | 142 | Again, one could imagine other strategies, but in all cases it is important to keep checking the package against the real web service fairly regularly. 143 | 144 | ## Summary 145 | 146 | * We set up httptest2 usage in our package exemplighratia by running `use_httptest2()` and tweaking the setup file to fool our own package that needs an API token. 147 | * We wrapped `test_that()` into `httptest2::with_mock_dir()` and ran the tests a first time to generate mock files that hold all information about the API responses. In some cases we modified these mock files to make them smaller or to make them correspond to an API error. 148 | 149 | Now, how do we make sure this works? 150 | 151 | * Turn off wifi, run the tests again. It works! Turn on wifi again. 152 | * Open .Renviron (`usethis::edit_r_environ()`), edit "GITHUB_PAT" into "byeGITHUB_PAT", re-start R, run the tests again. It works! Fix your "GITHUB_PAT" token in .Renviron. 153 | 154 | So we now have tests that no longer rely on an internet connection nor on having API credentials. 155 | 156 | We also added a continuous integration workflow for having a build using real interactions once every week, as it is important to regularly make sure the package still works against the latest API responses. 157 | 158 | For the full list of changes applied to exemplighratia in this chapter, see [the pull request diff on GitHub](https://github.com/ropensci-books/exemplighratia/pull/9/files). 159 | 160 | ::: {.alert .alert-dismissible .alert-primary} 161 | How do we get there with yet another package? We'll try webfakes. 162 | ::: 163 | -------------------------------------------------------------------------------- /topics-security.Rmd: -------------------------------------------------------------------------------- 1 | # Security {#security-chapter} 2 | 3 | When developing a package that uses secrets (API keys, [OAuth](https://blog.r-hub.io/2021/01/25/oauth-2.0/) tokens) and produces them (OAuth tokens, sensitive data), 4 | 5 | * You want the secrets to be usable by you, collaborators and CI services, without being readable by anyone else; 6 | * You want tests and checks (e.g. vignette building) that use the secrets to be turned off in environments where secrets won't be available (CRAN, forks of your development repository). 7 | 8 | Your general attitude should be to think about: 9 | 10 | * what are my secrets (an API key, an OAuth2.0 access token and the refresh token, etc.) and where/how exactly are there used (in the query part of an URL? as a header? which header, Authentication or something else?) -- packages like httr or httr2 might abstract some of the complexity for you but you need to really know where secrets are used and could be leaked, 11 | * what could go wrong (e.g. your token ending up being published), 12 | * how to prevent that (save your unedited token outside of your package, make sure it is not printed in logs or present in package check artefacts), 13 | * how to fix mistakes (how do you deactivate a token and how do you check no one used it in the meantime). 14 | 15 | ## Managing secrets securely 16 | 17 | ### Follow best practice when developing your package 18 | 19 | This book is about testing but security starts with how you develop your package. 20 | To better protect your users' secret, 21 | 22 | * It might be best not to let users pass API keys as parameters. It's best to have them save them in `.Renviron` or e.g. using the [keyring package](https://github.com/r-lib/keyring). This way, API keys are not in scripts. The [opencage package](https://docs.ropensci.org/opencage/articles/opencage.html#authentication-1) might provide some inspiration. 23 | 24 | * If the API you are working with lets you pass keys either in the request headers or query string, prefer to use request headers. 25 | 26 | ### Share secrets with continuous integration services 27 | 28 | You need to share secrets with continuous integration services... for real requests only! 29 | For tests using vcr, httptest, httptest2 or webfakes, you at most need a fake secret, e.g. "foobar" as API key -- except for recording cassettes and mock files, but that is something you do locally. 30 | 31 | ::: {.alert .alert-dismissible .alert-primary} 32 | In GitHub repositories, when storing a new secret, do not save it with quotes. 33 | I.e. if your secret is "blabla", the field should contain `blabla`, not `"blabla"` nor `'blabla'`. 34 | ```{r secret, fig.alt="Screenshot of the interface for adding secrets in a GitHub repository, showing how the secret is stored without any quote."} 35 | knitr::include_graphics("secret.png") 36 | 37 | ``` 38 | ::: 39 | 40 | #### API keys 41 | 42 | For API keys, you can use something like GitHub repo secrets if you use GitHub Actions. 43 | Then for the secret to be accessible as environment variable from your workflow in GitHub Actions [as explained in gargle docs](https://gargle.r-lib.org/articles/articles/managing-tokens-securely.html#provide-environment-variable-to-other-services-1) you need to add a line like 44 | 45 | ```yaml 46 | env: 47 | PACKAGE_PASSWORD: ${{ secrets.PACKAGE_PASSWORD }} 48 | ``` 49 | 50 | #### More complex objects 51 | 52 | If your secret is an OAuth token, you might be able to re-create it from pieces, where the pieces are strings you can store as repo secrets much like what you'd do for an API key. 53 | E.g. if your secret is an OAuth token, the [actual secrets](https://blog.r-hub.io/2021/01/25/oauth-2.0/#what-are-your-oauth-20-secret-credentials) are the access token and refresh token. 54 | 55 | ```yaml 56 | env: 57 | ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }} 58 | REFRESH_TOKEN: ${{ secrets.REFRESH_TOKEN }} 59 | ``` 60 | 61 | Therefore you could re-create it using e.g. the `credentials` argument of `httr::oauth2.0_token()`. 62 | The re-creation using environment variables `Sys.getenv("ACCESS_TOKEN")` and `Sys.getenv("REFRESH_TOKEN")` would happen in a [testthat helper file](https://blog.r-hub.io/2020/11/18/testthat-utility-belt/). 63 | 64 | ### Secret files 65 | 66 | For files, you will need to use encryption and to store a text-version of the encryption key/passwords as GitHub repo secret if you use GitHub Actions. 67 | Read the documentation of the continuous integration service your are using to find out how secrets are protected and how you can use them in your builds. 68 | 69 | See [gargle vignette about securely managing tokens](https://gargle.r-lib.org/articles/articles/managing-tokens-securely.html). 70 | 71 | The approach is: 72 | 73 | * Create your OAuth token locally, either outside of your package folder, or inside of it if you really want to, but **gitignored and Rbuildignored**. 74 | * Encrypt it using e.g. the [user-friendly cyphr package](https://docs.ropensci.org/cyphr/). Save the code for this and for the step before in a file e.g. inst/secrets.R for when you need to re-create a token as even refresh tokens expire. 75 | * For encrypting you need some sort of password. You will want to save it securely as _text_ in your [user-level .Renviron](https://rstats.wtf/r-startup.html#renviron) and in your GitHub repo secrets (or equivalent secret place for other CI services). E.g. create a key via `sodium_key <- sodium::keygen()` and get its text equivalent via `sodium::bin2hex(sodium_key)`. E.g. the latter command might give me `e46b7faf296e3f0624e6240a6efafe3dfb17b92ae0089c7e51952934b60749f2` and I will save this in .Renviron 76 | 77 | ``` 78 | MEETUPR_PWD="e46b7faf296e3f0624e6240a6efafe3dfb17b92ae0089c7e51952934b60749f2" 79 | ``` 80 | 81 | Example of a script creating and encrypting an OAuth token (for the Meetup API). 82 | 83 | ```r 84 | # thanks Jenny Bryan https://github.com/r-lib/gargle/blob/4fcf142fde43d107c6a20f905052f24859133c30/R/secret.R 85 | 86 | token_path <- testthat::test_path(".meetup_token.rds") 87 | use_build_ignore(token_path) 88 | use_git_ignore(token_path) 89 | 90 | meetupr::meetup_auth( 91 | token = NULL, 92 | cache = TRUE, 93 | set_renv = FALSE, 94 | token_path = token_path 95 | ) 96 | 97 | # sodium_key <- sodium::keygen() 98 | # set_renv("MEETUPR_PWD" = sodium::bin2hex(sodium_key)) 99 | # set_renv being an internal function taken from rtweet 100 | # that saves something to .Renviron 101 | 102 | # get key from environment variable 103 | key <- cyphr::key_sodium(sodium::hex2bin(Sys.getenv("MEETUPR_PWD"))) 104 | 105 | cyphr::encrypt_file( 106 | token_path, 107 | key = key, 108 | dest = testthat::test_path("secret.rds") 109 | ) 110 | ``` 111 | 112 | * In tests you have a [setup / helper file](https://blog.r-hub.io/2020/11/18/testthat-utility-belt/#code-called-in-your-tests) with code like below. 113 | 114 | ```r 115 | key <- cyphr::key_sodium(sodium::hex2bin(Sys.getenv("MEETUPR_PWD"))) 116 | 117 | temptoken <- tempfile(fileext = ".rds") 118 | 119 | cyphr::decrypt_file( 120 | testthat::test_path("secret.rds"), 121 | key = key, 122 | dest = temptoken 123 | ) 124 | ``` 125 | 126 | Now what happens in contexts where `MEETUPR_PWD` is not available? 127 | Well there should be no tests using it! 128 | See [our chapter about making real requests](#real-requests-chapter). 129 | 130 | ### Do not store secrets in the cassettes, mock files, recorded responses 131 | 132 | * With vcr make sure to [configure vcr correctly](#vcr-security). 133 | * With httptest and httptest2 only the response body (and headers, but not by default) are recorded. If those contains secrets, refer to the documentation about [redacting sensitive information](https://enpiar.com/r/httptest/articles/redacting.html) ([for httptest2](https://enpiar.com/httptest2/articles/redacting.html)). 134 | * With webfakes you will be creating recorded responses yourself, make sure this process does not leak secrets. If you test something related to authentication, use fake secrets. 135 | 136 | If the API you are interacting with uses OAuth for instance, make sure you are not leaking access tokens nor _refresh tokens_. 137 | 138 | ### Escape tests that require secrets 139 | 140 | This all depends on your setup for testing [real requests](#real-requests-chapter). 141 | You have to be sure no test requiring secrets will be run on [CRAN](#cran-preparedness) for instance. 142 | 143 | ## Sensitive recorded responses? 144 | 145 | In that case you might want to gitignore the cassettes / mock files / recorded responses, 146 | and skip the tests using them on continuous integration (e.g. `testthat::skip_on_ci()` or something more involved). 147 | You'd also [Rbuildignore](https://blog.r-hub.io/2020/05/20/rbuildignore/) the cassettes / mock files / recorded responses, as you do not want to release them to CRAN. 148 | 149 | ## Further resources 150 | 151 | Some tools might help you detect leaks or prevent them. 152 | 153 | * [shhgit](https://github.com/eth0izzle/shhgit)'s goal is "Find secrets in your code. Secrets detection for your GitHub, GitLab and Bitbucket repositories". 154 | * [Yelp's detect-secret](https://github.com/Yelp/detect-secrets) is "An enterprise friendly way of detecting and preventing secrets in code.". 155 | * [git-secret](https://git-secret.io/) is a "bash tool to store your private data inside a git repo". -------------------------------------------------------------------------------- /wholegames-intro.Rmd: -------------------------------------------------------------------------------- 1 | # (PART) Whole Game(s) {-} 2 | 3 | # Introduction 4 | 5 | Similar to the [Whole Game chapter in the R packages book by Hadley Wickham and Jenny Bryan](https://r-pkgs.org/whole-game.html), we will go through how to add HTTP tests to a minimal package. 6 | However, we will do it _four_ times to present alternative approaches: with [vcr](#vcr), [httptest](#httptest), [httptest2](#httptest2), [webfakes](#webfakes). 7 | After that exercise, we will compare approaches: we will compare both packages that involve mocking i.e. [vcr vs. httptest](#mocking-pkgs-comparison); and all three HTTP packages in a [last chapter](#pkgs-comparison). 8 | The next section will then present single topics such as "how to deal with authentication" in further details. 9 | 10 | ## Our example packages 11 | 12 | Our minimal packages, [`exemplighratia`](https://github.com/maropensci-bookselle/exemplighratia) and [`exemplighratia2`](https://github.com/ropensci-books/exemplighratia2), access the GitHub status API and one endpoint of GitHub V3 REST API. 13 | They are named after the Latin phrase _exempli gratia_ that means "for instance", with an H for GH. 14 | If you really need to interact with GitHub V3 API, we recommend the [gh package](https://gh.r-lib.org/). 15 | We also recommend looking at the source of the gh package, and at the docs of GitHub V3 API, in particular about [authentication](https://developer.github.com/v3/#authentication). 16 | 17 | ::: {.alert .alert-dismissible .alert-info} 18 | Our example packages call web _APIs_ but the tools and concepts are applicable to packages wrapping any web resource, even poorly documented ones.[^undocumented] 19 | ::: 20 | 21 | GitHub V3 API works without authentication too, but at a lower rate. 22 | For the sake of having an example of a package _requiring_ authentication we will assume the API is *not* usable without authentication. 23 | Authentication is, here, the setting of a token in a HTTP header (so quite simple, compared to e.g. OAuth). 24 | 25 | GitHub Status API, on the contrary, does not necessitate authentication at all. 26 | 27 | So we will create two functions, one that works without authentication, one that works with authentication. 28 | 29 | How did we create the packages? 30 | You are obviously free to use your own favorite workflow tools, but below we share our workflow using the [usethis package](https://r-pkgs.org/whole-game.html). 31 | 32 | * We followed [usethis setup article](https://usethis.r-lib.org/articles/articles/usethis-setup.html). 33 | 34 | ### exemplighratia2 (httr2) 35 | 36 | Then we ran 37 | 38 | * `usethis::create_package("path/to/folder/exemplighratia2")` to create and open the package project; 39 | * `usethis::use_mit_license()` to add an MIT license; 40 | * `usethis::use_package("httr2")` to add a dependency on httr2; 41 | * `usethis::use_package("purrr")` to add a dependency on purrr; 42 | * `use_r("api-status.R")` to add the first function whose code is written below; 43 | 44 | ```r 45 | status_url <- function() { 46 | "https://kctbh9vrtdwd.statuspage.io/api/v2/components.json" 47 | } 48 | 49 | #' GitHub APIs status 50 | #' 51 | #' @description Get the status of requests to GitHub APIs 52 | #' 53 | #' @importFrom magrittr `%>%` 54 | #' 55 | #' @return A character vector, one of "operational", "degraded_performance", 56 | #' "partial_outage", or "major_outage." 57 | #' 58 | #' @details See details in https://www.githubstatus.com/api#components. 59 | #' @export 60 | #' 61 | #' @examples 62 | #' \dontrun{ 63 | #' gh_api_status() 64 | #' } 65 | gh_api_status <- function() { 66 | response <- status_url() %>% 67 | httr2::request() %>% 68 | httr2::req_perform() 69 | 70 | # Check status 71 | httr2::resp_check_status(response) 72 | 73 | # Parse the content 74 | content <- httr2::resp_body_json(response) 75 | 76 | # Extract the part about the API status 77 | components <- content$components 78 | api_status <- components[purrr::map_chr(components, "name") == "API Requests"][[1]] 79 | 80 | # Return status 81 | api_status$status 82 | 83 | } 84 | ``` 85 | 86 | * `use_test("api-status")` (and using [testthat latest edition](https://www.tidyverse.org/blog/2020/10/testthat-3-0-0/#3rd-edition) so setting `Config/testthat/edition: 3` in `DESCRIPTION`) to add a simple test whose code is below. 87 | 88 | ```r 89 | test_that("gh_api_status() works", { 90 | testthat::expect_type(gh_api_status(), "character") 91 | }) 92 | ``` 93 | 94 | * `use_r("organizations.R")` (and `usethis::use_package("cli")`) to add a second function. 95 | Note that an ideal version of this function would have some sort of callback in the retry, to call the `gh_api_status()` function (maybe with `httr2::req_retry()`'s `is_transient` argument). 96 | 97 | ```r 98 | gh_v3_url <- function() { 99 | "https://api.github.com/" 100 | } 101 | 102 | #' GitHub organizations 103 | #' 104 | #' @description Get logins of GitHub organizations. 105 | #' 106 | #' @param since The integer ID of the last organization that you've seen. 107 | #' 108 | #' @return A character vector of at most 30 elements. 109 | #' @export 110 | #' 111 | #' @details Refer to https://developer.github.com/v3/orgs/#list-organizations 112 | #' 113 | #' @examples 114 | #' \dontrun{ 115 | #' gh_organizations(since = 42) 116 | #' } 117 | gh_organizations <- function(since = 1) { 118 | 119 | if (!gh::gh_token_exists()) { 120 | cli::cli_abort("No token provided! {.url https://usethis.r-lib.org/articles/git-credentials.html}.") 121 | } 122 | 123 | token <- gh::gh_token() 124 | 125 | response <- httr2::request(gh_v3_url()) %>% 126 | httr2::req_url_path_append("organizations") %>% 127 | httr2::req_url_query(since = since) %>% 128 | httr2::req_headers("Authorization" = paste("token", token)) %>% 129 | httr2::req_retry(max_tries = 3, max_seconds = 120) %>% 130 | httr2::req_perform() 131 | 132 | content <- httr2::resp_body_json(response) 133 | 134 | purrr::map_chr(content, "login") 135 | 136 | } 137 | 138 | 139 | ``` 140 | 141 | * `use_test("organizations")` to add a simple test. 142 | 143 | ```r 144 | test_that("gh_organizations works", { 145 | expect_type(gh_organizations(), "character") 146 | }) 147 | 148 | ``` 149 | 150 | ### exemplighratia (httr) 151 | 152 | Then we ran 153 | 154 | * `usethis::create_package("path/to/folder/exemplighratia")` to create and open the package project; 155 | * `usethis::use_mit_license()` to add an MIT license; 156 | * `usethis::use_package("httr")` to add a dependency on httr; 157 | * `usethis::use_package("purrr")` to add a dependency on purrr; 158 | * `usethis::use_r("api-status.R")` to add the first function whose code is written below; 159 | 160 | ```r 161 | status_url <- function() { 162 | "https://kctbh9vrtdwd.statuspage.io/api/v2/components.json" 163 | } 164 | 165 | #' GitHub APIs status 166 | #' 167 | #' @description Get the status of requests to GitHub APIs 168 | #' 169 | #' @return A character vector, one of "operational", "degraded_performance", 170 | #' "partial_outage", or "major_outage." 171 | #' 172 | #' @details See details in https://www.githubstatus.com/api#components. 173 | #' @export 174 | #' 175 | #' @examples 176 | #' \dontrun{ 177 | #' gh_api_status() 178 | #' } 179 | gh_api_status <- function() { 180 | response <- httr::GET(status_url()) 181 | 182 | # Check status 183 | httr::stop_for_status(response) 184 | 185 | # Parse the content 186 | content <- httr::content(response) 187 | 188 | # Extract the part about the API status 189 | components <- content$components 190 | api_status <- components[purrr::map_chr(components, "name") == "API Requests"][[1]] 191 | 192 | # Return status 193 | api_status$status 194 | 195 | } 196 | ``` 197 | 198 | * `use_test("api-status")` (and using [testthat latest edition](https://www.tidyverse.org/blog/2020/10/testthat-3-0-0/#3rd-edition) so setting `Config/testthat/edition: 3` in `DESCRIPTION`) to add a simple test whose code is below. 199 | 200 | ```r 201 | test_that("gh_api_status() works", { 202 | testthat::expect_type(gh_api_status(), "character") 203 | }) 204 | ``` 205 | 206 | * `usethis::use_r("organizations.R")` to add a second function. Note that an ideal version of this function would have some sort of callback in the retry, to call the `gh_api_status()` function (which seems easier to implement with [crul's retry method](https://blog.r-hub.io/2020/04/07/retry-wheel/#retry-in-crul)). 207 | 208 | ```r 209 | gh_v3_url <- function() { 210 | "https://api.github.com/" 211 | } 212 | 213 | #' GitHub organizations 214 | #' 215 | #' @description Get logins of GitHub organizations. 216 | #' 217 | #' @param since The integer ID of the last organization that you've seen. 218 | #' 219 | #' @return A character vector of at most 30 elements. 220 | #' @export 221 | #' 222 | #' @details Refer to https://developer.github.com/v3/orgs/#list-organizations 223 | #' 224 | #' @examples 225 | #' \dontrun{ 226 | #' gh_organizations(since = 42) 227 | #' } 228 | gh_organizations <- function(since = 1) { 229 | url <- httr::modify_url( 230 | gh_v3_url(), 231 | path = "organizations", 232 | query = list(since = since) 233 | ) 234 | 235 | token <- Sys.getenv("GITHUB_PAT") 236 | 237 | if (!nchar(token)) { 238 | stop("No token provided! Set up the GITHUB_PAT environment variable please.") 239 | } 240 | 241 | response <- httr::RETRY( 242 | "GET", 243 | url, 244 | httr::add_headers("Authorization" = paste("token", token)) 245 | ) 246 | 247 | httr::stop_for_status(response) 248 | 249 | content <- httr::content(response) 250 | 251 | purrr::map_chr(content, "login") 252 | 253 | } 254 | 255 | ``` 256 | 257 | * `use_test("organizations")` to add a simple test. 258 | 259 | ```r 260 | test_that("gh_organizations works", { 261 | testthat::expect_type(gh_organizations(), "character") 262 | }) 263 | 264 | ``` 265 | 266 | ## Conclusion 267 | 268 | All good, now our package has 100% test coverage and passes R CMD Check (granted, our tests could be more thorough, but remember this is a minimal example). 269 | But what if we try working without a connection? 270 | In the following chapters, we'll add more robust testing infrastructure to this minimal package, and we will do that _four_ times to compare packages/approaches: once with [vcr](#vcr), once with [httptest](#httptest), once with [httptest2](#httptest2), and once with [webfakes](#webfakes). 271 | 272 | [^undocumented]: An interesting post to read about an R package wrapping an undocumented web API is ["One-Hour Package"](https://enpiar.com/2017/08/11/one-hour-package/) by Neal Richardson. -------------------------------------------------------------------------------- /wholegames-vcr.Rmd: -------------------------------------------------------------------------------- 1 | # Use vcr (& webmockr) {#vcr} 2 | 3 | In this chapter we aim at adding HTTP testing infrastructure to exemplighratia2 using [vcr](#what-vcr) (& [webmockr](#webmockr)). 4 | 5 | ::: {.alert .alert-dismissible .alert-info} 6 | [Corresponding pull request to exemplighratia2](https://github.com/ropensci-books/exemplighratia2/pull/2/files). Feel free to fork the repository to experiment yourself! 7 | ::: 8 | 9 | ## Setup 10 | 11 | Before working on all this, we need to install `{vcr}`. 12 | 13 | First, we need to run `usethis::use_package("vcr", type = "Suggests")` (in the exemplighratia directory) to add vcr as a dependency to `DESCRIPTION`, under Suggests just like testthat. 14 | 15 | Then we need to create a helper file using `usethis::use_test_helper("vcr")`. 16 | A new file is created under `tests/testthat/helper-vcr.R`. 17 | 18 | When testthat runs tests, [files whose name start with "helper" are always run first](https://testthat.r-lib.org/reference/test_dir.html#special-files). 19 | They are also loaded by `devtools::load_all()`, so the vcr setup is loaded when developing and testing interactively. 20 | See the table in the R-hub blog post ["Helper code and files for your testthat tests"](https://blog.r-hub.io/2020/11/18/testthat-utility-belt/#code-called-in-your-tests). 21 | 22 | We have to tweak the vcr setup for our needs. 23 | 24 | * We need to ensure that we set up a fake API key when there is no API token around. 25 | Why? Because if you remember well, the code of our function `gh_organizations()` checks for the presence of a token. 26 | With mock responses around, we don't need a token but we still need to fool our own package in contexts where there is no token (e.g. in continuous integration checks for a fork of a GitHub repository). 27 | 28 | * We do not need to tweak the configuration to prevent our API from leaking, because with vcr 2.0.0 and above, the `Authorization` header is never recorded. 29 | 30 | Below is the updated setup file saved under `tests/testthat/helper-vcr.R`. 31 | 32 | ```r 33 | vcr_dir <- vcr::vcr_test_path("_vcr") 34 | if (!gh::gh_token_exists()) { 35 | if (dir.exists(vcr_dir)) { 36 | # Fake API token to fool our package 37 | Sys.setenv("GITHUB_PAT" = "foobar") 38 | } else { 39 | # If there's no mock files nor API token, impossible to run tests 40 | stop("No API key nor cassettes, tests cannot be run.", call. = FALSE) 41 | } 42 | } 43 | 44 | ``` 45 | 46 | So this was just setup, now on to adapting our tests! 47 | 48 | ## Actual testing 49 | 50 | The most important function will be `vcr::local_cassette("cassette-informative-and-unique-name")` which tells vcr to create a mock file to store all API responses for API calls occurring in the current test (what's inside `test_that()`). 51 | The `vcr::local_cassette()` function might remind you of the `withr::local_` functions or of `rlang::local_options()`. 52 | 53 | Let's tweak the test for `gh_api_status`, it now becomes 54 | 55 | ```r 56 | test_that("gh_api_status() works", { 57 | vcr::local_cassette("gh_api_status") 58 | status <- gh_api_status() 59 | expect_type(status, "character") 60 | }) 61 | 62 | ``` 63 | 64 | If you had used `vcr::use_cassette()` instead of the newer `vcr::local_cassette()`, you might notice the code feels less nested! 65 | 66 | If we run this test, for instance through `devtools::test_active_file()`, 67 | 68 | * the first time, vcr creates a cassette (mock file) under `tests/testthat/_vcr/gh_api_status.yml` where it stores the API response 69 | It contains all the information related to requests and responses, headers included. 70 | 71 | ```yaml 72 | http_interactions: 73 | - request: 74 | method: GET 75 | uri: https://kctbh9vrtdwd.statuspage.io/api/v2/components.json 76 | response: 77 | status: 200 78 | headers: 79 | content-type: application/json; charset=utf-8 80 | date: Tue, 09 Sep 2025 10:09:28 GMT 81 | x-download-options: noopen 82 | x-permitted-cross-domain-policies: none 83 | referrer-policy: strict-origin-when-cross-origin 84 | x-statuspage-version: b27f95704d5835d5f1a3ccc341503d3f7f22f502 85 | strict-transport-security: max-age=259200 86 | x-statuspage-skip-logging: 'true' 87 | access-control-allow-origin: '*' 88 | cache-control: max-age=3, public 89 | x-pollinator-metadata-service: status-page-web-pages 90 | etag: W/"81603fa0d180d761ce3b3c3c02d68741" 91 | x-runtime: '0.049065' 92 | server: AtlassianEdge 93 | accept-ranges: bytes 94 | x-content-type-options: nosniff 95 | x-xss-protection: 1; mode=block 96 | atl-traceid: daa0d6503446464e9748ed3a8634ead0 97 | atl-request-id: daa0d650-3446-464e-9748-ed3a8634ead0 98 | report-to: '{"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], 99 | "group": "endpoint-1", "include_subdomains": true, "max_age": 600}' 100 | nel: '{"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, 101 | "report_to": "endpoint-1"}' 102 | content-encoding: br 103 | server-timing: atl-edge;dur=129,atl-edge-internal;dur=5,atl-edge-upstream;dur=125,atl-edge-pop;desc="aws-us-east-1" 104 | vary: Accept,Accept-Encoding 105 | x-cache: Miss from cloudfront 106 | via: 1.1 dd239fa6f06e5b3ae1437460dbc3d6a6.cloudfront.net (CloudFront) 107 | x-amz-cf-pop: MRS53-P3 108 | x-amz-cf-id: K63D5rR4HMl9WRamA6Yzh8R6SuF2vfDOuh8W-bgc1kUo-xGdrVZp9g== 109 | body: 110 | string: '{"page":{"id":"kctbh9vrtdwd","name":"GitHub","url":"https://www.githubstatus.com","time_zone":"Etc/UTC","updated_at":"2025-09-09T10:00:59.457Z"},"components":[{"id":"8l4ygp009s5s","name":"Git 111 | Operations","status":"operational","created_at":"2017-01-31T20:05:05.370Z","updated_at":"2025-08-21T06:58:30.069Z","position":1,"description":"Performance 112 | of git clones, pulls, pushes, and associated operations","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"4230lsnqdsld","name":"Webhooks","status":"operational","created_at":"2019-11-13T18:00:24.256Z","updated_at":"2025-08-05T16:14:01.686Z","position":2,"description":"Real 113 | time HTTP callbacks of user-generated and system events","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"0l2p9nhqnxpd","name":"Visit 114 | www.githubstatus.com for more information","status":"operational","created_at":"2018-12-05T19:39:40.838Z","updated_at":"2025-03-19T05:00:21.309Z","position":3,"description":null,"showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"brv1bkgrwx7q","name":"API 115 | Requests","status":"operational","created_at":"2017-01-31T20:01:46.621Z","updated_at":"2025-09-04T20:25:47.243Z","position":4,"description":"Requests 116 | for GitHub APIs","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"kr09ddfgbfsf","name":"Issues","status":"operational","created_at":"2017-01-31T20:01:46.638Z","updated_at":"2025-08-27T21:27:47.854Z","position":5,"description":"Requests 117 | for Issues on GitHub.com","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"hhtssxt0f5v2","name":"Pull 118 | Requests","status":"operational","created_at":"2020-09-02T15:39:06.329Z","updated_at":"2025-08-12T17:56:13.209Z","position":6,"description":"Requests 119 | for Pull Requests on GitHub.com","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"br0l2tvcx85d","name":"Actions","status":"operational","created_at":"2019-11-13T18:02:19.432Z","updated_at":"2025-08-21T18:13:02.491Z","position":7,"description":"Workflows, 120 | Compute and Orchestration for GitHub Actions","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"st3j38cctv9l","name":"Packages","status":"operational","created_at":"2019-11-13T18:02:40.064Z","updated_at":"2025-08-14T18:37:12.106Z","position":8,"description":"API 121 | requests and webhook delivery for GitHub Packages","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"vg70hn9s2tyj","name":"Pages","status":"operational","created_at":"2017-01-31T20:04:33.923Z","updated_at":"2025-06-17T20:03:36.357Z","position":9,"description":"Frontend 122 | application and API servers for Pages builds","showcase":false,"start_date":null,"group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"h2ftsgbw7kmk","name":"Codespaces","status":"operational","created_at":"2021-08-11T16:02:09.505Z","updated_at":"2025-07-21T09:47:56.207Z","position":10,"description":"Orchestration 123 | and Compute for GitHub Codespaces","showcase":false,"start_date":"2021-08-11","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"id":"pjmpxvq2cmr2","name":"Copilot","status":"operational","created_at":"2022-06-21T16:04:33.017Z","updated_at":"2025-08-27T21:36:28.924Z","position":11,"description":null,"showcase":false,"start_date":"2022-06-21","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false}]}' 124 | recorded_at: 2025-09-09 10:09:28 125 | recorded_with: VCR-vcr/2.0.0 126 | ``` 127 | 128 | * all the times after that, unless we delete the cassette (mock file), vcr simply uses the cassette instead of actually calling the API. 129 | 130 | Let's tweak our other test, of `gh_organizations()`. 131 | Here things get more exciting or complicated, as we also set out to adding a test of the error behavior. 132 | This inspired us to change error behavior a bit with a slightly more specific error message i.e. `httr::stop_for_status(response)` became `httr::stop_for_status(response, task = "get data from the API, oops")`. 133 | 134 | The test file ` tests/testthat/test-organizations.R` is now: 135 | 136 | ```r 137 | test_that("gh_organizations works", { 138 | vcr::local_cassette("gh_organizations") 139 | orgs <- gh_organizations() 140 | expect_type(orgs, "character") 141 | }) 142 | 143 | test_that("gh_organizations errors when the API doesn't behave", { 144 | webmockr::enable() 145 | stub <- webmockr::stub_request( 146 | "get", 147 | "https://api.github.com/organizations?since=1" 148 | ) 149 | webmockr::to_return(stub, status = 502) 150 | expect_snapshot(error = TRUE, gh_organizations()) 151 | webmockr::disable() 152 | }) 153 | 154 | 155 | ``` 156 | 157 | The first test is similar to what we did for `gh_api_status()`. 158 | In the second test there is more to unpack. 159 | 160 | * We enable the use of `{webmockr}` at the beginning with `webmockr::enable()`. Why webmockr? Because it can help mock a failure scenario. 161 | * We explicitly write that a request to `https://api.github.com/organizations?since=1` should return a status of 502. 162 | 163 | ```r 164 | stub <- webmockr::stub_request("get", "https://api.github.com/organizations?since=1") 165 | webmockr::to_return(stub, status = 502) 166 | ``` 167 | 168 | * We then test for the error message with `expect_snapshot(error = TRUE, gh_organizations())`. 169 | * We disable webmockr with `webmockr::disable()`. 170 | 171 | ::: {.alert .alert-dismissible .alert-primary} 172 | Instead of using webmockr for creating a fake API eror, we could have 173 | 174 | * recorded a normal cassette; 175 | * edited it to replace the status code. 176 | 177 | Read pros and cons of this approach in the vcr vignette [_Why and how edit your vcr cassettes?_](https://docs.ropensci.org/vcr/articles/cassette-manual-editing.html), especially if you don't find the webmockr approach enjoyable. 178 | ::: 179 | 180 | Without the HTTP testing infrastructure, testing for behavior of the package in case of API errors would be more difficult. 181 | However, we could resort to [testthat's mocking tools](https://testthat.r-lib.org/dev/reference/index.html#mocking). 182 | 183 | Regarding our secret API token, the first time we run the test file, vcr creates a cassette where we *do not see our token*. 184 | 185 | ## Also testing for real interactions 186 | 187 | What if the API responses change? 188 | Hopefully we'd notice that thanks to following API news. 189 | However, sometimes web APIs change without any notice. 190 | Therefore it is important to run tests against the real web service once in a while. 191 | 192 | The vcr package provides various methods to turn vcr use on and off to allow real requests i.e. ignoring mock files. 193 | See `?vcr::lightswitch`. 194 | 195 | In the case of exemplighratia2, we added a [GitHub Actions workflow](https://github.com/ropensci-books/exemplighratia2/blob/vcr/.github/workflows/real-requests.yaml) that will run on schedule once a week, for which one of the build has vcr turned off via the `VCR_TURN_OFF` environment variable. 196 | We chose to have one build with vcr turned on and otherwise the same configuration to make it easier to assess what broke in case of failure (if both builds fail, the web API is probably not the culprit). 197 | Compared to continuous integration builds where vcr is turned on, this one build needs to have access to a `GITHUB_PAT` secret environment variable. Furthermore, it is slower. 198 | 199 | One could imagine other strategies: 200 | 201 | * Always having one continuous integration build with vcr turned off but skipping it in contexts where there isn't any token (pull requests from forks for instance?); 202 | * Only running tests with vcr turned off locally once in a while. 203 | 204 | ## Summary 205 | 206 | * We set up vcr usage in our package exemplighratia2 by registering vcr as a dependency and creating a file to protect our secret API key and to fool our own package that needs an API token. 207 | * Inside `test_that()` blocks, we call `vcr::local_cassette()` and ran the tests a first time to generate mock files that hold all information about the API interactions. 208 | * In one of the tests, we used webmockr to create an environment where only fake requests are allowed. We defined that the request that `gh_organizations()` makes should get a 502 status. We were therefore able to test for the error message `gh_organizations()` returns in such cases. 209 | 210 | Now, how do we make sure this works? 211 | 212 | * Turn off wifi, run the tests again. It works! Turn on wifi again. 213 | * Open .Renviron (`usethis::edit_r_environ()`), edit "GITHUB_PAT" into "byeGITHUB_PAT", re-start R, run the tests again. It works! Fix your "GITHUB_PAT" token in .Renviron. (to store your credentials, you should actually use [gitcreds](https://usethis.r-lib.org/articles/git-credentials.html) for Git credentials, and [keyring](https://blog.r-hub.io/2024/02/28/key-advantages-of-using-keyring/) for other credentials). 214 | 215 | So we now have tests that no longer rely on an internet connection nor on having API credentials. 216 | 217 | We also added a continuous integration workflow for having a build using real interactions once every week, as it is important to regularly make sure the package still works against the latest API responses. 218 | 219 | For the full list of changes applied to exemplighratia in this chapter, see [the pull request diff on GitHub](https://github.com/ropensci-books/exemplighratia/pull/2/files). 220 | 221 | ::: {.alert .alert-dismissible .alert-primary} 222 | How do we get there with other packages? Let's try httptest in the next chapter! 223 | ::: 224 | 225 | ## PS: Where to put use_cassette() 226 | 227 | Where do we put the `vcr::use_cassette()` call? 228 | Well, as written in the manual page of that function, _There's a few ways to get correct line numbers for failed tests and one way to not get correct line numbers:_ 229 | What's correct? 230 | 231 | * Wrapping the whole `testthat::test_that()` call (do not do that if your test contains for instance `skip_on_cran()``); 232 | 233 | ```r 234 | vcr::use_cassette("thing", { 235 | testthat::test_that("thing", { 236 | lala <- get_foo() 237 | expect_true(lala) 238 | }) 239 | }) 240 | ``` 241 | 242 | * Wrapping a few lines inside `testthat::test_that()` **excluding the expectations `expect_blabla()`** 243 | 244 | ````r 245 | testthat::test_that("thing", { 246 | vcr::use_cassette("thing", { 247 | lala <- get_foo() 248 | }) 249 | expect_true(lala) 250 | }) 251 | ```` 252 | 253 | What's incorrect? 254 | 255 | ````r 256 | testthat::test_that("thing", { 257 | vcr::use_cassette("thing", { 258 | lala <- get_foo() 259 | expect_true(lala) 260 | }) 261 | }) 262 | ```` 263 | 264 | We used the solution of only wrapping the lines containing API calls in `vcr::use_cassette()`, but it is up to you to choose what you prefer. 265 | 266 | [^secrets]: However, if you change something related to handling secrets in your code or tests, please check again your new cassettes do not include secrets. 267 | --------------------------------------------------------------------------------