├── .Rbuildignore ├── .gitignore ├── CRAN-RELEASE ├── CRAN-SUBMISSION ├── DESCRIPTION ├── EIAapi.Rproj ├── LICENSE ├── LICENSE.md ├── NAMESPACE ├── NEWS.md ├── R ├── backfile.R ├── global_vars.R ├── metadata.R └── query.R ├── README.Rmd ├── README.md ├── docs ├── 404.html ├── LICENSE-text.html ├── LICENSE.html ├── articles │ ├── backfile.html │ ├── backfile_files │ │ └── figure-html │ │ │ └── unnamed-chunk-2-1.png │ ├── images │ │ ├── dashboard1.png │ │ ├── dashboard2.png │ │ └── register.png │ ├── index.html │ └── intro.html ├── authors.html ├── bootstrap-toc.css ├── bootstrap-toc.js ├── docsearch.css ├── docsearch.js ├── index.html ├── link.svg ├── news │ └── index.html ├── pkgdown.css ├── pkgdown.js ├── pkgdown.yml ├── reference │ ├── Rplot001.png │ ├── eia_backfill-1.png │ ├── eia_backfill.html │ ├── eia_get.html │ ├── eia_metadata.html │ ├── figures │ │ ├── EIA_API_browser.png │ │ ├── README-pressure-1.png │ │ └── README-unnamed-chunk-6-1.png │ └── index.html └── sitemap.xml ├── man ├── eia_backfill.Rd ├── eia_get.Rd ├── eia_metadata.Rd └── figures │ ├── EIA_API_browser.png │ └── README-unnamed-chunk-6-1.png └── vignettes ├── .gitignore ├── backfile.Rmd ├── images ├── dashboard1.png ├── dashboard2.png └── register.png └── intro.Rmd /.Rbuildignore: -------------------------------------------------------------------------------- 1 | ^.*\.Rproj$ 2 | ^\.Rproj\.user$ 3 | ^README\.Rmd$ 4 | ^LICENSE\.md$ 5 | docs 6 | ^CRAN-RELEASE$ 7 | vignettes 8 | ^vignettes/*.Rmd$ 9 | ^vignettes/images/*.png$ 10 | ^CRAN-SUBMISSION$ 11 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .Rproj.user 3 | inst/doc 4 | -------------------------------------------------------------------------------- /CRAN-RELEASE: -------------------------------------------------------------------------------- 1 | This package was submitted to CRAN on 2023-04-30. 2 | Once it is accepted, delete this file and tag the release (commit 4c5770f). 3 | -------------------------------------------------------------------------------- /CRAN-SUBMISSION: -------------------------------------------------------------------------------- 1 | Version: 0.1.2 2 | Date: 2023-08-13 06:24:47 UTC 3 | SHA: fc000a7e65d837595aee8d5107fb591082ba9cfd 4 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Package: EIAapi 2 | Type: Package 3 | Title: Query Data from the 'EIA' API 4 | Version: 0.1.2 5 | Authors@R: person(given = "Rami", 6 | family ="Krispin", 7 | email = "rami.krispin@gmail.com", 8 | role = c("aut", "cre")) 9 | Maintainer: Rami Krispin 10 | Description: Provides a function to query and extract data from the 'US Energy Information Administration' ('EIA') API V2 . The 'EIA' API provides a variety of information, in a time series format, about the energy sector in the US. The API is open, free, and requires an access key and registration at . 11 | License: MIT + file LICENSE 12 | Encoding: UTF-8 13 | LazyData: true 14 | RoxygenNote: 7.2.1 15 | Imports: 16 | data.table (>= 1.14.2), 17 | dplyr (>= 1.0.9), 18 | jsonlite (>= 1.8.2), 19 | lubridate (>= 1.8.0) 20 | Suggests: 21 | knitr, 22 | plotly (>= 4.10.0), 23 | rmarkdown 24 | SystemRequirements: The package required the jq command line library. Please check https://stedolan.github.io/jq/download/ for download instructions. 25 | URL: https://github.com/RamiKrispin/EIAapi 26 | BugReports: https://github.com/RamiKrispin/EIAapi/issues 27 | -------------------------------------------------------------------------------- /EIAapi.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | 15 | AutoAppendNewline: Yes 16 | StripTrailingWhitespace: Yes 17 | 18 | BuildType: Package 19 | PackageUseDevtools: Yes 20 | PackageInstallArgs: --no-multiarch --with-keep.source 21 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | YEAR: 2023 2 | COPYRIGHT HOLDER: EIAapi authors 3 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # MIT License 2 | 3 | Copyright (c) 2023 EIAapi authors 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /NAMESPACE: -------------------------------------------------------------------------------- 1 | # Generated by roxygen2: do not edit by hand 2 | 3 | export(eia_backfill) 4 | export(eia_get) 5 | export(eia_metadata) 6 | -------------------------------------------------------------------------------- /NEWS.md: -------------------------------------------------------------------------------- 1 | # EIAapi 0.1.2 2 | 3 | * Added a new function, `eia_backfill` to handle large data query from the API 4 | * Fixed issue with the `eia_get` function - `curl` commands failed to send query with brackets 5 | * Added new vignette with example of the `eia_backfill` function usage 6 | 7 | # EIAapi 0.1.1 8 | 9 | * Modified the `eia_get` function argument - rename the `api_url` argument to `api_path`, and dropping the end-point (`https://api.eia.gov/v2/`) from the query 10 | * Added new function - `eia_metadata` to query metadata information from the API 11 | * Added new vignettes - introduction the EIA API 12 | 13 | # EIAapi 0.1.0 14 | 15 | * Added the `eia_get` function to query data from EIA API 16 | * Added a `NEWS.md` file to track changes to the package 17 | -------------------------------------------------------------------------------- /R/backfile.R: -------------------------------------------------------------------------------- 1 | #' Pull a Large Number of Observations with a Sequential Query 2 | #' @description This function allows users to overcome the API's observation limit 3 | #' per query by breaking down the query into smaller sequential sub-queries and 4 | #' appending back the results. The main use case of this function is for backfilling 5 | #' hourly series. 6 | #' @param start defines the start time of the series, should use a POSIXt class 7 | #' for hourly series or Date format for non-hourly series (daily, monthly, etc.) 8 | #' @param end defines the end time of the series, should use a POSIXt class 9 | #' for hourly series or Date format for non-hourly series (daily, monthly, etc.) 10 | #' @param offset An integer, defines the number of observations limitation per query. 11 | #' In line with the API limitation of up to 5000 observations per query, the offset argument's upper limit is 5000 observations. 12 | #' @param api_key A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service 13 | #' @param api_path A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 14 | #' The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/ 15 | #' @param facets A list, optional, set the filtering argument (defined as 'facets' 16 | #' on the API header), following the structure of list(facet_name_1 = value_1, 17 | #' facet_name_2 = value_2) 18 | #' @details The function use start, end, and offset arguments to define a sequence 19 | #' of queries. 20 | #' @return A time series 21 | #' @export 22 | #' @examples 23 | #'\dontrun{ 24 | #' start <- as.POSIXlt("2018-06-19T00", tz = "UTC") 25 | #' end <- lubridate::floor_date(Sys.time()- lubridate::days(2), unit = "day") 26 | #' attr(end, "tzone") <- "UTC" 27 | #' offset <- 2000 28 | #' api_key <- Sys.getenv("eia_key") 29 | #' api_path <- "electricity/rto/region-sub-ba-data/data/" 30 | #' 31 | #' facets = list(parent = "NYIS", 32 | #' subba = "ZONA") 33 | #' 34 | #' df <- eia_backfill(start = start, 35 | #' end = end, 36 | #' offset = offset, 37 | #' api_key = api_key, 38 | #' api_path = api_path, 39 | #' facets = facets) 40 | #' 41 | #' at_y <- pretty(df$value)[c(2, 4, 6)] 42 | #' at_x <- seq.POSIXt(from = start, 43 | #' to = end, 44 | #' by = "2 years") 45 | #' plot(df$time, df$value, 46 | #' col = "#1f77b4", 47 | #' type = "l", 48 | #' frame.plot = FALSE, 49 | #' axes = FALSE, 50 | #' panel.first = abline(h = at_y, col = "grey80"), 51 | #' main = "NY Independent System Operator (West) - Hourly Generation of Electricity", 52 | #' xlab = "Source: https://www.eia.gov/", 53 | #' ylab = "MegaWatt/Hours") 54 | #' 55 | #' mtext(side =1, text = format(at_x, format = "%Y"), at = at_x, 56 | #' col = "grey20", line = 1, cex = 0.8) 57 | #' 58 | #' mtext(side =2, text = format(at_y, scientific = FALSE), at = at_y, 59 | #' col = "grey20", line = 1, cex = 0.8) 60 | #'} 61 | 62 | eia_backfill <- function(start, 63 | end, 64 | offset, 65 | api_key, 66 | api_path, 67 | facets){ 68 | 69 | # Error handling 70 | 71 | # Validating the start and end arguments 72 | if(!any(lubridate::is.POSIXt(start), 73 | lubridate::is.Date(start))){ 74 | stop("The start argument is not valid, the function suports POSIXlt, POSIXct, Date objects") 75 | } else if(!any(lubridate::is.POSIXt(end), 76 | lubridate::is.Date(end))){ 77 | stop("The end argument is not valid, the function suports POSIXlt, POSIXct, Date objects") 78 | } else if(!any(class(start) %in% class(end))){ 79 | stop("The class of the start argument is different from the end argument class") 80 | # Validate the offset argument 81 | } else if(!is.numeric(offset) || offset < 0 || offset %% 1 != 0){ 82 | stop("The offset argument is not valid, must be numeric") 83 | # Validate the api_key and api_path arguments 84 | } else if(!is.character(api_key)){ 85 | stop("The api_key argument is not valid") 86 | } else if(!is.character(api_path)){ 87 | stop("The api_path argument is not valid") 88 | # Validate the facets argument 89 | } else if(!is.null(facets) && !is.list(facets)){ 90 | stop("The facets argument must be a list object") 91 | } else if(offset > 5000){ 92 | message("The offset argument surpasses the API number of observations per call limit, setting it to 5000") 93 | offset <- 5000 94 | } 95 | 96 | 97 | # Classify the time stamp to either POSIXt or Date object 98 | if(lubridate::is.POSIXt(start)){ 99 | time_class <- "POSIXt" 100 | } else if(lubridate::is.Date(start)){ 101 | time_class <- "Date" 102 | } else { 103 | stop("Mismatch with start/end arguments class") 104 | } 105 | 106 | # Create a time/date vector 107 | 108 | if(time_class == "POSIXt"){ 109 | time_vec <- seq.POSIXt(from = start, to = end, by = paste(offset, "hour")) 110 | if(max(time_vec) < end){ 111 | time_vec <- c(time_vec, end) 112 | } 113 | } else if(time_class == "Date"){ 114 | time_vec <- seq.Date(from = start, to = end, by = "day") 115 | } 116 | 117 | 118 | time_vec_seq <- seq_along(time_vec)[-length(time_vec)] 119 | 120 | df <- lapply(time_vec_seq, function(i){ 121 | 122 | temp <- start_h <- end_h <- start_time <- end_time <- NULL 123 | s <- time_vec[i] 124 | if(time_class == "POSIXt"){ 125 | if(i < max(time_vec_seq)){ 126 | e <- time_vec[i + 1] - lubridate::hours(1) 127 | } else { 128 | e <- time_vec[i + 1] 129 | } 130 | 131 | start_h <- lubridate::hour(s) 132 | end_h <- lubridate::hour(e) 133 | if(start_h < 10){ 134 | start_time <- paste(substr(as.character(s), 1, 10), "T0", start_h, sep = "") 135 | } else { 136 | start_time <- paste(substr(as.character(s), 1, 10), "T", start_h, sep = "") 137 | } 138 | 139 | 140 | if(end_h < 10){ 141 | end_time <- paste(substr(as.character(e), 1, 10), "T0", end_h, sep = "") 142 | } else { 143 | end_time <- paste(substr(as.character(e), 1, 10), "T", end_h, sep = "") 144 | } 145 | 146 | temp <- EIAapi::eia_get(api_key = api_key, 147 | api_path = api_path, 148 | facets = facets, 149 | format = "data.frame", 150 | start = start_time, 151 | end = end_time, 152 | length = NULL, 153 | offset = NULL) |> 154 | dplyr::mutate(time = lubridate::ymd_h(period, tz = "UTC")) |> 155 | dplyr::select(-period) |> 156 | dplyr::select(time, dplyr::everything()) |> 157 | dplyr::arrange(time) 158 | 159 | 160 | 161 | } else if(time_class == "Date"){ 162 | e <- time_vec[i + 1] - lubridate::days(1) 163 | temp <- EIAapi::eia_get(api_key = api_key, 164 | api_path = api_path, 165 | facets = facets, 166 | format = "data.frame", 167 | start = s, 168 | end = e, 169 | length = NULL, 170 | offset = NULL) |> 171 | dplyr::mutate(time = lubridate::ymd_h(period, tz = "UTC")) |> 172 | dplyr::select(-period) |> 173 | dplyr::select(time, dplyr::everything()) |> 174 | dplyr::arrange(time) 175 | 176 | } 177 | 178 | names(temp) <- gsub(pattern = "-", replacement = "_", x = names(temp)) 179 | 180 | return(temp) 181 | }) |> 182 | dplyr::bind_rows() |> 183 | dplyr::arrange(time) 184 | 185 | 186 | 187 | 188 | return(df) 189 | 190 | } 191 | -------------------------------------------------------------------------------- /R/global_vars.R: -------------------------------------------------------------------------------- 1 | globalVariables( 2 | c("time", 3 | "period")) 4 | -------------------------------------------------------------------------------- /R/metadata.R: -------------------------------------------------------------------------------- 1 | #' Pull Metadata from EIA API 2 | #' @description Get data descriptions and metadata from the EIA API 3 | #' @details The function enables to explore the different data categories and available routes 4 | #' inline with the API dashboard (https://www.eia.gov/opendata/browser/) 5 | #' @param api_key A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service 6 | #' @param api_path A string, the API category/route path following the API endpoint (i.e., 'https://api.eia.gov/v2/') 7 | #' If set to NULL (default) or as empty string "" it returns the main categories available on the API. 8 | #' The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/ 9 | #' @return a list object with the series description and metadata 10 | #' @export 11 | #' @examples 12 | #'\dontrun{ 13 | #' electricity_metadata <- eia_metadata(api_key = Sys.getenv("eia_key"), 14 | #' api_path = "electricity") 15 | #' 16 | #' electricity_metadata$response$description 17 | #' electricity_metadata$response$id 18 | #' electricity_metadata$response$name 19 | #' electricity_metadata$response$routes 20 | #' 21 | #'} 22 | eia_metadata <- function(api_path = NULL, 23 | api_key){ 24 | 25 | # V2 end point 26 | end_point <- "https://api.eia.gov/v2/" 27 | 28 | if(missing(api_key)){ 29 | stop("The api_key argument is missing... \033[0;92m\xE2\x9D\x8C\033[0m\n") 30 | } else if(!is.character(api_key)){ 31 | stop("The api_key argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n") 32 | } else if(missing(api_path)){ 33 | stop(paste("The api_path argument is missing... \033[0;92m\xE2\x9D\x8C\033[0m\n", 34 | "Please check the API Dashboard for the API URL:\n", 35 | "https://www.eia.gov/opendata/browser/", sep = "")) 36 | } else if(!is.null(api_path) && !is.character(api_path)){ 37 | stop(paste("The api_path argument is not valid, must be a character object \033[0;92m\xE2\x9D\x8C\033[0m\n", 38 | "Please check the API Dashboard for the API URL:\n", 39 | "https://www.eia.gov/opendata/browser/", sep = "")) 40 | } 41 | 42 | 43 | if(is.null(api_path)){ 44 | path <- "" 45 | } else { 46 | path <- api_path 47 | } 48 | 49 | api_url <- paste(end_point, api_path, sep = "") 50 | query <- NULL 51 | query <- paste("curl '", 52 | api_url, 53 | "'?api_key=", 54 | api_key, 55 | sep = "") 56 | 57 | 58 | raw <- NULL 59 | 60 | tryCatch({ 61 | raw <- system(command = query, intern = TRUE)}, 62 | error = function(c) "Fail to pull the query, please check the error log", 63 | message = function(c) "message" 64 | ) 65 | 66 | if(is.null(raw) || !is.character(raw)){ 67 | stop(paste("Could not pull the metadata... \033[0;92m\xE2\x9D\x8C\033[0m\n", 68 | "Check the query parameters (e.g., api key, path, etc.) or the error log\n", sep = "")) 69 | } 70 | 71 | parsed_query <- NULL 72 | 73 | tryCatch({ 74 | parsed_query <- jsonlite::fromJSON(raw)}, 75 | error = function(c) "Fail to pull the query, please check the error log", 76 | message = function(c) "message" 77 | ) 78 | 79 | 80 | if(is.null(parsed_query) || !is.list(parsed_query)){ 81 | stop(paste("Could not parse the metadata JSON... \033[0;92m\xE2\x9D\x8C\033[0m\n", 82 | "Please check the query parameters (e.g., api key, path, etc.) or the error log\n", sep = "")) 83 | } 84 | 85 | 86 | meta <- parsed_query$response 87 | meta[["command"]] <- parsed_query$request$command 88 | 89 | attr(meta, "class") <- c(class(meta), "eia_metadata") 90 | 91 | return(meta) 92 | } 93 | -------------------------------------------------------------------------------- /R/query.R: -------------------------------------------------------------------------------- 1 | #' Query the EIA API 2 | #' @description Function to query and extract data from the EIA API v2 3 | #' @param api_key A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service 4 | #' @param api_path A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 5 | #' The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/ 6 | #' @param data A string, the metric type, by default uses 'value' (defined as 7 | #' 'data' on the API header) 8 | #' @param facets A list, optional, set the filtering argument (defined as 'facets' 9 | #' on the API header), following the structure of list(facet_name_1 = value_1, 10 | #' facet_name_2 = value_2) 11 | #' @param start A string, optional, set the starting date or time of the series 12 | #' using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series 13 | #' @param end A string, optional, set the ending date or time of the series 14 | #' using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series 15 | #' @param length An integer, optional, defines the length of the series, if set to 16 | #' NULL (default), will default to the API default value of 5000 observations per 17 | #' pull. The API enables a pull of up to 100K observations per call. If needed to 18 | #' pull more than the API limit per call, recommend to iterate the call with 19 | #' the use of the start, end and/or offset arguments 20 | #' @param offset An integer, optional, set the number of observations to offset 21 | #' from the default starting point of the series. If set to NULL (default), will default 22 | #' to the API default value of 0 23 | #' @param frequency A string, optional, define the API frequency argument 24 | #' (e.g., hourly, monthly, annual, etc.). If set to NULL (default), will default 25 | #' to the API default value 26 | #' @param format A string, defines the output of the return object to either 27 | #' "data.frame" (default) or "data.table" 28 | #' @return data.table/data.frame object 29 | #' @export 30 | #' @examples 31 | #'\dontrun{ 32 | #'# Required an EIA API key to send a query 33 | #' api_key <- "YOUR_API_KEY" 34 | #' 35 | #' df <- eia_get( 36 | #' api_key = api_key, 37 | #' api_path = "electricity/rto/fuel-type-data/data/", 38 | #' data = "value" 39 | #' ) 40 | #'} 41 | 42 | eia_get <- function(api_key, 43 | api_path, 44 | data = "value", 45 | facets = NULL, 46 | start = NULL, 47 | end = NULL, 48 | length = NULL, 49 | offset = NULL, 50 | frequency = NULL, 51 | format = "data.frame"){ 52 | # Error handling 53 | if(missing(api_key)){ 54 | stop("The api_key argument is missing... \033[0;92m\xE2\x9D\x8C\033[0m\n") 55 | } else if(!is.character(api_key)){ 56 | stop("The api_key argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n") 57 | } else if(missing(api_path)){ 58 | stop(paste("The api_path argument is missing... \033[0;92m\xE2\x9D\x8C\033[0m\n", 59 | "Please check the API Dashboard for the API URL:\n", 60 | "https://www.eia.gov/opendata/browser/", sep = "")) 61 | } else if(!is.character(api_path)){ 62 | stop(paste("The api_path argument is not valid, must be a character object \033[0;92m\xE2\x9D\x8C\033[0m\n", 63 | "Please check the API Dashboard for the API URL:\n", 64 | "https://www.eia.gov/opendata/browser/", sep = "")) 65 | } else if(missing(data) && !is.character(data)){ 66 | stop("The data argument is either missing or not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n") 67 | } else if(missing(facets) && !is.list(facets) && !is.null(facets)){ 68 | stop("The facets argument is either missing or not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n") 69 | } else if(!is.null(start) && !is.character(start)){ 70 | stop(paste("The start argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n", 71 | "Please use a character using the following format:\n", 72 | "Date: 'YYYY-MM-DD', for example start='2022-01-01'\n", 73 | "Time (Hourly): 'YYYY-MM-DDTHH', for example start='2022-01-01T01'\n",sep = "")) 74 | } else if(!is.null(end) && !is.character(end)){ 75 | stop(paste("The end argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n", 76 | "Please use a character using the following format:\n", 77 | "Date: 'YYYY-MM-DD', for example end='2022-01-01'\n", 78 | "Time (Hourly): 'YYYY-MM-DDTHH', for example end='2022-01-01T01'\n",sep = "")) 79 | } else if(!is.null(length) && !is.numeric(length) && length %% 1 != 0){ 80 | stop(paste("The length argument is not valid: \033[0;92m\xE2\x9D\x8C\033[0m\n", 81 | "Must be an integer number", sep = "")) 82 | } else if(!is.null(offset) && !is.numeric(offset) && offset %% 1 != 0){ 83 | stop(paste("The offset argument is not valid:\n", 84 | "Must be an integer number \033[0;92m\xE2\x9D\x8C\033[0m\n", sep = "")) 85 | } else if(!is.null(frequency) && !is.character(frequency)){ 86 | stop(paste("The frequency argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n", 87 | "Must be a character object", sep = "")) 88 | } else if(format != "data.table" && format != "data.frame"){ 89 | stop(paste("The format argument is not valid... \033[0;92m\xE2\x9D\x8C\033[0m\n", 90 | "Must be either 'data.frame' or 'data.table'", sep = "")) 91 | } 92 | 93 | if(substr(api_path, start = nchar(api_path), stop = nchar(api_path)) == "/"){ 94 | api_path <- substr(api_path, start = 1, stop = nchar(api_path) - 1) 95 | } 96 | 97 | if(is.null(start)){ 98 | s <- "" 99 | } else{ 100 | s <- paste("&start=", start, sep = "") 101 | } 102 | 103 | if(is.null(end)){ 104 | e <- "" 105 | } else { 106 | e <- paste("&end=", end, sep = "") 107 | } 108 | 109 | f <- "" 110 | if(!is.null(facets)){ 111 | for(i in names(facets)){ 112 | for(l in facets[[i]]){ 113 | f <- paste(f, 114 | sprintf("&facets[%s][]=%s", i, l), 115 | sep = "") 116 | } 117 | } 118 | } 119 | 120 | if(is.null(length)){ 121 | l <- "" 122 | } else { 123 | l <- paste("&length=", length, sep = "") 124 | } 125 | 126 | if(is.null(offset)){ 127 | o <- "" 128 | } else { 129 | o <- paste("&offset=", offset, sep = "") 130 | } 131 | 132 | if(is.null(frequency)){ 133 | q <- "" 134 | } else { 135 | q <- paste("&frequency=", frequency, sep = "") 136 | } 137 | 138 | api_url <- paste("https://api.eia.gov/v2/", api_path, sep = "") 139 | query <- NULL 140 | query <- paste("curl -g '", 141 | api_url, 142 | "?api_key=", 143 | api_key, 144 | "&data[]=", 145 | data, 146 | s, e, f, l, o, q, 147 | "' | jq -r ' .response.data | ( .[0] | keys_unsorted | map(ascii_upcase)), (.[] | [.[]]) | @csv'", 148 | sep = "" 149 | ) 150 | 151 | 152 | df <- NULL 153 | 154 | tryCatch({ 155 | df <- data.table::fread(cmd = query, 156 | header = TRUE)}, 157 | error = function(c) "error", 158 | warning = function(c) "warning", 159 | message = function(c) "message" 160 | ) 161 | 162 | 163 | if(is.null(df)){ 164 | stop(paste("Could not pull the data... \033[0;92m\xE2\x9D\x8C\033[0m\n", 165 | "Check the query parameters (e.g., api key, path, etc.) or the error log\n", sep = "")) 166 | } 167 | if(format == "data.frame"){ 168 | df <- as.data.frame(df) 169 | } 170 | 171 | names(df) <- tolower(names(df)) 172 | 173 | return(df) 174 | } 175 | -------------------------------------------------------------------------------- /README.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | output: github_document 3 | editor_options: 4 | markdown: 5 | wrap: 72 6 | --- 7 | 8 | 9 | 10 | ```{r, include = FALSE} 11 | knitr::opts_chunk$set( 12 | collapse = TRUE, 13 | comment = "#>", 14 | fig.path = "man/figures/README-", 15 | out.width = "100%" 16 | ) 17 | ``` 18 | 19 | # EIAapi 20 | 21 | 22 | [![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/EIAapi)](https://cran.r-project.org/package=EIAapi) 23 | [![lifecycle](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental) 24 | [![License: 25 | MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/license/mit/) 26 | 27 | 28 | 29 | The **EIAapi** package provides functions to query and pull tidy data from [EIA API 30 | v2](https://www.eia.gov/opendata/). 31 | 32 | Introduction to the EIAapi package available [here](https://ramikrispin.github.io/EIAapi/articles/intro.html). 33 | 34 | ## Requirments 35 | 36 | To pull data from the API using this package, you will need the following: 37 | 38 | - jq - The package uses [jq](https://stedolan.github.io/jq/) to parse the API output from JSON to tabular format. To download and install jq follow the 39 | instructions on the [download 40 | page](https://stedolan.github.io/jq/download/). 41 | - API key - To query the EIA API, you will need to register to the 42 | service to receive the API key. 43 | 44 | To register to to the API go to https://www.eia.gov/opendata/ and click the `Register` button, and follow the instruction. 45 | 46 | ## Installation 47 | 48 | Install the stable version from [CRAN]: 49 | ``` r 50 | install.packages("EIAapi") 51 | ``` 52 | 53 | Or, install the development version from [Github](https://github.com/RamiKrispin/EIAapi): 54 | 55 | ``` r 56 | # install.packages("devtools") 57 | devtools::install_github("RamiKrispin/EIAapi") 58 | ``` 59 | 60 | ## Query data 61 | 62 | A suggested workflow to query data from the EIA API with the `eia_get` 63 | function: 64 | 65 | - Go to the EIA API Dashboard 66 | [website](https://www.eia.gov/opendata/browser) 67 | - Select the API Route and define filters 68 | - Submit the query and extract the query information from the query 69 | metadata: 70 | - API URL 71 | - Header 72 | 73 | [![](man/figures/EIA_API_browser.png)](https://www.eia.gov/opendata/browser/) 74 | 75 | In the example above: 76 | 77 | - The API URL: `https://api.eia.gov/v2/electricity/rto/fuel-type-data/data/`, and 78 | - The query header: 79 | ```JSON 80 | { 81 | "frequency": "hourly", 82 | "data": [ 83 | "value" 84 | ], 85 | "facets": {}, 86 | "start": null, 87 | "end": null, 88 | "sort": [ 89 | { 90 | "column": "period", 91 | "direction": "desc" 92 | } 93 | ], 94 | "offset": 0, 95 | "length": 5000, 96 | "api-version": "2.0.3" 97 | } 98 | ``` 99 | 100 | Using the URL and header information, we can submit the GET request with the `eia_get` function: 101 | 102 | ```{r} 103 | library(EIAapi) 104 | 105 | # Pulling the API key from my renviron file 106 | api_key <- Sys.getenv("eia_key") 107 | 108 | df1 <- eia_get( 109 | api_key = api_key, 110 | api_path = "electricity/rto/fuel-type-data/data/", 111 | data = "value" 112 | ) 113 | 114 | nrow(df1) 115 | 116 | head(df1) 117 | ``` 118 | 119 | **Note:** The `api_path` argument defines by the query of the path that following the endpoint of the API - `https://api.eia.gov/v2/`. In the example above the API full URL is: 120 | 121 | ``` HTML 122 | https://api.eia.gov/v2/electricity/rto/fuel-type-data/data/ 123 | ``` 124 | 125 | Therefore, the query's path is set to `electricity/rto/fuel-type-data/data/`. 126 | 127 | 128 | The `eia_get` function leverages the [jq](https://stedolan.github.io/jq/) tool to parse the return JSON object from the API into CSV format and the [data.table](https://CRAN.R-project.org/package=data.table) package to read and parse the object into R. By default, the function returns a `data.frame` object, but you can use the `format` argument and set the output object as `data.table`: 129 | 130 | ```{r} 131 | df2 <- eia_get( 132 | api_key = api_key, 133 | api_path = "electricity/rto/fuel-type-data/data/", 134 | data = "value", 135 | format = "data.table" 136 | ) 137 | 138 | df2 139 | ``` 140 | 141 | If you wish to pull more than the `length` upper limit, you can use the `offset` to offset the query by limit and pull the next observations: 142 | 143 | 144 | ```{r} 145 | df3 <- eia_get( 146 | api_key = api_key, 147 | api_path = "electricity/rto/fuel-type-data/data/", 148 | data = "value", 149 | length = 5000, 150 | offset = 5000, 151 | format = "data.table" 152 | ) 153 | 154 | df3 155 | ``` 156 | 157 | You can narrow down your pull by using the `facets` argument and applying some filters. For example, in the example above, let's filter data by the `fuletype` field and select energy source as `Natural gas (NG)` and the region as `United States Lower 48 (US48)`, and then extract the header: 158 | 159 | ``` JSON 160 | { 161 | "frequency": "hourly", 162 | "data": [ 163 | "value" 164 | ], 165 | "facets": { 166 | "respondent": [ 167 | "US48" 168 | ], 169 | "fueltype": [ 170 | "NG" 171 | ] 172 | }, 173 | "start": null, 174 | "end": null, 175 | "sort": [ 176 | { 177 | "column": "period", 178 | "direction": "desc" 179 | } 180 | ], 181 | "offset": 0, 182 | "length": 5000, 183 | "api-version": "2.0.3" 184 | } 185 | ``` 186 | 187 | Updating the query with the `facets` information: 188 | 189 | ```{r} 190 | facets <- list(respondent = "US48", fueltype = "NG") 191 | 192 | df4 <- eia_get( 193 | api_key = api_key, 194 | api_path = "electricity/rto/fuel-type-data/data/", 195 | data = "value", 196 | length = 5000, 197 | format = "data.table", 198 | facets = facets 199 | ) 200 | 201 | df4 202 | 203 | unique(df4$fueltype) 204 | unique(df4$respondent) 205 | ``` 206 | 207 | 208 | Last but not least, you can set the starting and ending time of the query. For example, let's set a window between June 1st and October 1st, 2022: 209 | 210 | 211 | ```{r} 212 | df5 <- eia_get( 213 | api_key = api_key, 214 | api_path = "electricity/rto/fuel-type-data/data/", 215 | data = "value", 216 | length = 5000, 217 | format = "data.table", 218 | facets = facets, 219 | start = "2022-06-01T00", 220 | end = "2022-10-01T00" 221 | ) 222 | 223 | df5 224 | 225 | df5$time <- as.POSIXct(paste(substr(df5$period, start = 1, stop = 10)," ", 226 | substr(df5$period, start = 12, stop = 13), ":00:00", 227 | sep = "")) 228 | 229 | plot(x = df5$time, y = df5$value, 230 | main = "United States Lower 48 Hourly Electricity Generation by Natural Gas", 231 | col.main = "#457b9d", 232 | col = "#073b4c", 233 | sub = "Source: Form EIA-930 Product: Hourly Electric Grid Monitor", 234 | xlab = "", 235 | ylab = "Megawatt Hours", 236 | cex.main=1, 237 | cex.lab=1, 238 | cex.sub=0.8, 239 | frame=FALSE, 240 | type = "l") 241 | ``` 242 | 243 | 244 | 245 | ## API Resources 246 | 247 | * EIA API documentation: https://www.eia.gov/opendata/documentation.php 248 | * EIA APIv2 Webinar: https://www.youtube.com/watch?v=1VsSp1XG-Pg&t=1671s&ab_channel=EIAgov 249 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | # EIAapi 5 | 6 | 7 | 8 | [![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/EIAapi)](https://cran.r-project.org/package=EIAapi) 9 | [![lifecycle](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental) 10 | [![License: 11 | MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/license/mit/) 12 | 13 | 14 | The **EIAapi** package provides functions to query and pull tidy data 15 | from [EIA API v2](https://www.eia.gov/opendata/). 16 | 17 | Introduction to the EIAapi package available 18 | [here](https://ramikrispin.github.io/EIAapi/articles/intro.html). 19 | 20 | ## Requirments 21 | 22 | To pull data from the API using this package, you will need the 23 | following: 24 | 25 | - jq - The package uses [jq](https://stedolan.github.io/jq/) to parse 26 | the API output from JSON to tabular format. To download and install 27 | jq follow the instructions on the [download 28 | page](https://stedolan.github.io/jq/download/). 29 | - API key - To query the EIA API, you will need to register to the 30 | service to receive the API key. 31 | 32 | To register to to the API go to and 33 | click the `Register` button, and follow the instruction. 34 | 35 | ## Installation 36 | 37 | Install the stable version from \[CRAN\]: 38 | 39 | ``` r 40 | install.packages("EIAapi") 41 | ``` 42 | 43 | Or, install the development version from 44 | [Github](https://github.com/RamiKrispin/EIAapi): 45 | 46 | ``` r 47 | # install.packages("devtools") 48 | devtools::install_github("RamiKrispin/EIAapi") 49 | ``` 50 | 51 | ## Query data 52 | 53 | A suggested workflow to query data from the EIA API with the `eia_get` 54 | function: 55 | 56 | - Go to the EIA API Dashboard 57 | [website](https://www.eia.gov/opendata/browser) 58 | - Select the API Route and define filters 59 | - Submit the query and extract the query information from the query 60 | metadata: 61 | - API URL 62 | - Header 63 | 64 | [![](man/figures/EIA_API_browser.png)](https://www.eia.gov/opendata/browser/) 65 | 66 | In the example above: 67 | 68 | - The API URL: 69 | `https://api.eia.gov/v2/electricity/rto/fuel-type-data/data/`, and 70 | - The query header: 71 | 72 | ``` json 73 | { 74 | "frequency": "hourly", 75 | "data": [ 76 | "value" 77 | ], 78 | "facets": {}, 79 | "start": null, 80 | "end": null, 81 | "sort": [ 82 | { 83 | "column": "period", 84 | "direction": "desc" 85 | } 86 | ], 87 | "offset": 0, 88 | "length": 5000, 89 | "api-version": "2.0.3" 90 | } 91 | ``` 92 | 93 | Using the URL and header information, we can submit the GET request with 94 | the `eia_get` function: 95 | 96 | ``` r 97 | library(EIAapi) 98 | 99 | # Pulling the API key from my renviron file 100 | api_key <- Sys.getenv("eia_key") 101 | 102 | df1 <- eia_get( 103 | api_key = api_key, 104 | api_path = "electricity/rto/fuel-type-data/data/", 105 | data = "value" 106 | ) 107 | 108 | nrow(df1) 109 | #> [1] 5000 110 | 111 | head(df1) 112 | #> period respondent respondent-name fueltype 113 | #> 1 2019-09-02T06 FPC Duke Energy Florida, Inc. SUN 114 | #> 2 2019-09-03T04 PSEI Puget Sound Energy, Inc. WAT 115 | #> 3 2019-09-02T11 MIDA Mid-Atlantic NG 116 | #> 4 2019-09-02T18 NW Northwest OTH 117 | #> 5 2019-09-02T21 AECI Associated Electric Cooperative, Inc. NG 118 | #> 6 2019-09-02T12 CENT Central NUC 119 | #> type-name value value-units 120 | #> 1 Solar 0 megawatthours 121 | #> 2 Hydro 497 megawatthours 122 | #> 3 Natural gas 24868 megawatthours 123 | #> 4 Other 795 megawatthours 124 | #> 5 Natural gas 1551 megawatthours 125 | #> 6 Nuclear 2001 megawatthours 126 | ``` 127 | 128 | **Note:** The `api_path` argument defines by the query of the path that 129 | following the endpoint of the API - `https://api.eia.gov/v2/`. In the 130 | example above the API full URL is: 131 | 132 | ``` html 133 | https://api.eia.gov/v2/electricity/rto/fuel-type-data/data/ 134 | ``` 135 | 136 | Therefore, the query’s path is set to 137 | `electricity/rto/fuel-type-data/data/`. 138 | 139 | The `eia_get` function leverages the 140 | [jq](https://stedolan.github.io/jq/) tool to parse the return JSON 141 | object from the API into CSV format and the 142 | [data.table](https://CRAN.R-project.org/package=data.table) package to 143 | read and parse the object into R. By default, the function returns a 144 | `data.frame` object, but you can use the `format` argument and set the 145 | output object as `data.table`: 146 | 147 | ``` r 148 | df2 <- eia_get( 149 | api_key = api_key, 150 | api_path = "electricity/rto/fuel-type-data/data/", 151 | data = "value", 152 | format = "data.table" 153 | ) 154 | 155 | df2 156 | #> period respondent respondent-name fueltype 157 | #> 1: 2019-09-02T06 FPC Duke Energy Florida, Inc. SUN 158 | #> 2: 2019-09-03T04 PSEI Puget Sound Energy, Inc. WAT 159 | #> 3: 2019-09-02T11 MIDA Mid-Atlantic NG 160 | #> 4: 2019-09-02T18 NW Northwest OTH 161 | #> 5: 2019-09-02T21 AECI Associated Electric Cooperative, Inc. NG 162 | #> --- 163 | #> 4996: 2019-09-01T15 HST City of Homestead NG 164 | #> 4997: 2019-09-01T18 AEC PowerSouth Energy Cooperative WAT 165 | #> 4998: 2019-09-01T17 SC South Carolina Public Service Authority NG 166 | #> 4999: 2019-09-02T04 PSEI Puget Sound Energy, Inc. WND 167 | #> 5000: 2019-09-01T05 FPC Duke Energy Florida, Inc. OIL 168 | #> type-name value value-units 169 | #> 1: Solar 0 megawatthours 170 | #> 2: Hydro 497 megawatthours 171 | #> 3: Natural gas 24868 megawatthours 172 | #> 4: Other 795 megawatthours 173 | #> 5: Natural gas 1551 megawatthours 174 | #> --- 175 | #> 4996: Natural gas 0 megawatthours 176 | #> 4997: Hydro 0 megawatthours 177 | #> 4998: Natural gas 780 megawatthours 178 | #> 4999: Wind 239 megawatthours 179 | #> 5000: Petroleum 0 megawatthours 180 | ``` 181 | 182 | If you wish to pull more than the `length` upper limit, you can use the 183 | `offset` to offset the query by limit and pull the next observations: 184 | 185 | ``` r 186 | df3 <- eia_get( 187 | api_key = api_key, 188 | api_path = "electricity/rto/fuel-type-data/data/", 189 | data = "value", 190 | length = 5000, 191 | offset = 5000, 192 | format = "data.table" 193 | ) 194 | 195 | df3 196 | #> period respondent respondent-name 197 | #> 1: 2019-09-01T10 TAL City of Tallahassee 198 | #> 2: 2019-09-01T23 SE Southeast 199 | #> 3: 2019-09-01T16 SWPP Southwest Power Pool 200 | #> 4: 2019-09-01T09 PSCO Public Service Company of Colorado 201 | #> 5: 2019-09-01T23 CPLW Duke Energy Progress West 202 | #> --- 203 | #> 4996: 2019-09-08T13 SCEG Dominion Energy South Carolina, Inc. 204 | #> 4997: 2019-09-09T02 SC South Carolina Public Service Authority 205 | #> 4998: 2019-09-09T03 YAD Alcoa Power Generating, Inc. - Yadkin Division 206 | #> 4999: 2019-09-08T17 TIDC Turlock Irrigation District 207 | #> 5000: 2019-09-09T00 SE Southeast 208 | #> fueltype type-name value value-units 209 | #> 1: NG Natural gas 277 megawatthours 210 | #> 2: NG Natural gas 21452 megawatthours 211 | #> 3: COL Coal 13539 megawatthours 212 | #> 4: COL Coal 1835 megawatthours 213 | #> 5: WAT Hydro 32 megawatthours 214 | #> --- 215 | #> 4996: WAT Hydro 15 megawatthours 216 | #> 4997: SUN Solar 2 megawatthours 217 | #> 4998: WAT Hydro 1 megawatthours 218 | #> 4999: NG Natural gas 193 megawatthours 219 | #> 5000: OIL Petroleum 0 megawatthours 220 | ``` 221 | 222 | You can narrow down your pull by using the `facets` argument and 223 | applying some filters. For example, in the example above, let’s filter 224 | data by the `fuletype` field and select energy source as 225 | `Natural gas (NG)` and the region as `United States Lower 48 (US48)`, 226 | and then extract the header: 227 | 228 | ``` json 229 | { 230 | "frequency": "hourly", 231 | "data": [ 232 | "value" 233 | ], 234 | "facets": { 235 | "respondent": [ 236 | "US48" 237 | ], 238 | "fueltype": [ 239 | "NG" 240 | ] 241 | }, 242 | "start": null, 243 | "end": null, 244 | "sort": [ 245 | { 246 | "column": "period", 247 | "direction": "desc" 248 | } 249 | ], 250 | "offset": 0, 251 | "length": 5000, 252 | "api-version": "2.0.3" 253 | } 254 | ``` 255 | 256 | Updating the query with the `facets` information: 257 | 258 | ``` r 259 | facets <- list(respondent = "US48", fueltype = "NG") 260 | 261 | df4 <- eia_get( 262 | api_key = api_key, 263 | api_path = "electricity/rto/fuel-type-data/data/", 264 | data = "value", 265 | length = 5000, 266 | format = "data.table", 267 | facets = facets 268 | ) 269 | 270 | df4 271 | #> period respondent respondent-name fueltype type-name 272 | #> 1: 2019-09-03T02 US48 United States Lower 48 NG Natural gas 273 | #> 2: 2019-09-02T08 US48 United States Lower 48 NG Natural gas 274 | #> 3: 2019-09-02T06 US48 United States Lower 48 NG Natural gas 275 | #> 4: 2019-09-03T03 US48 United States Lower 48 NG Natural gas 276 | #> 5: 2019-09-02T16 US48 United States Lower 48 NG Natural gas 277 | #> --- 278 | #> 4996: 2018-11-09T20 US48 United States Lower 48 NG Natural gas 279 | #> 4997: 2018-11-09T21 US48 United States Lower 48 NG Natural gas 280 | #> 4998: 2018-11-07T11 US48 United States Lower 48 NG Natural gas 281 | #> 4999: 2018-11-07T12 US48 United States Lower 48 NG Natural gas 282 | #> 5000: 2018-11-07T13 US48 United States Lower 48 NG Natural gas 283 | #> value value-units 284 | #> 1: 238622 megawatthours 285 | #> 2: 149180 megawatthours 286 | #> 3: 164671 megawatthours 287 | #> 4: 217190 megawatthours 288 | #> 5: 206225 megawatthours 289 | #> --- 290 | #> 4996: 145412 megawatthours 291 | #> 4997: 147572 megawatthours 292 | #> 4998: 109269 megawatthours 293 | #> 4999: 123157 megawatthours 294 | #> 5000: 132755 megawatthours 295 | 296 | unique(df4$fueltype) 297 | #> [1] "NG" 298 | unique(df4$respondent) 299 | #> [1] "US48" 300 | ``` 301 | 302 | Last but not least, you can set the starting and ending time of the 303 | query. For example, let’s set a window between June 1st and October 1st, 304 | 2022: 305 | 306 | ``` r 307 | df5 <- eia_get( 308 | api_key = api_key, 309 | api_path = "electricity/rto/fuel-type-data/data/", 310 | data = "value", 311 | length = 5000, 312 | format = "data.table", 313 | facets = facets, 314 | start = "2022-06-01T00", 315 | end = "2022-10-01T00" 316 | ) 317 | 318 | df5 319 | #> period respondent respondent-name fueltype type-name 320 | #> 1: 2022-06-01T00 US48 United States Lower 48 NG Natural gas 321 | #> 2: 2022-06-01T01 US48 United States Lower 48 NG Natural gas 322 | #> 3: 2022-06-01T02 US48 United States Lower 48 NG Natural gas 323 | #> 4: 2022-06-01T03 US48 United States Lower 48 NG Natural gas 324 | #> 5: 2022-06-01T04 US48 United States Lower 48 NG Natural gas 325 | #> --- 326 | #> 2925: 2022-09-30T20 US48 United States Lower 48 NG Natural gas 327 | #> 2926: 2022-09-30T21 US48 United States Lower 48 NG Natural gas 328 | #> 2927: 2022-09-30T22 US48 United States Lower 48 NG Natural gas 329 | #> 2928: 2022-09-30T23 US48 United States Lower 48 NG Natural gas 330 | #> 2929: 2022-10-01T00 US48 United States Lower 48 NG Natural gas 331 | #> value value-units 332 | #> 1: 247460 megawatthours 333 | #> 2: 242340 megawatthours 334 | #> 3: 233394 megawatthours 335 | #> 4: 215728 megawatthours 336 | #> 5: 183732 megawatthours 337 | #> --- 338 | #> 2925: 186416 megawatthours 339 | #> 2926: 190630 megawatthours 340 | #> 2927: 196122 megawatthours 341 | #> 2928: 198929 megawatthours 342 | #> 2929: 200809 megawatthours 343 | 344 | df5$time <- as.POSIXct(paste(substr(df5$period, start = 1, stop = 10)," ", 345 | substr(df5$period, start = 12, stop = 13), ":00:00", 346 | sep = "")) 347 | 348 | plot(x = df5$time, y = df5$value, 349 | main = "United States Lower 48 Hourly Electricity Generation by Natural Gas", 350 | col.main = "#457b9d", 351 | col = "#073b4c", 352 | sub = "Source: Form EIA-930 Product: Hourly Electric Grid Monitor", 353 | xlab = "", 354 | ylab = "Megawatt Hours", 355 | cex.main=1, 356 | cex.lab=1, 357 | cex.sub=0.8, 358 | frame=FALSE, 359 | type = "l") 360 | ``` 361 | 362 | 363 | 364 | ## API Resources 365 | 366 | - EIA API documentation: 367 | 368 | - EIA APIv2 Webinar: 369 | 370 | -------------------------------------------------------------------------------- /docs/404.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Page not found (404) • EIAapi 9 | 10 | 11 | 12 | 13 | 14 | 15 | 19 | 20 | 21 | 22 | 23 |
24 |
76 | 77 | 78 | 79 | 80 |
81 |
82 |
83 |

Page not found (404)

84 |
85 | 86 | Content not found. Please use links in the navbar. 87 | 88 |
89 | 90 | 94 | 95 |
96 | 97 | 98 | 99 |
100 |

101 |

Developed by Rami Krispin.

102 |
103 | 104 |
105 |

106 |

Site built with pkgdown 107 | 2.0.1.

108 |
109 | 110 |
111 |
112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | -------------------------------------------------------------------------------- /docs/LICENSE-text.html: -------------------------------------------------------------------------------- 1 | 2 | License • EIAapi 6 | 7 | 8 |
9 |
53 | 54 | 55 | 56 |
57 |
58 |
59 |

License

60 |
61 | 62 | 
YEAR: 2023
63 | COPYRIGHT HOLDER: EIAapi authors
64 |
65 | 66 |
67 | 68 | 71 | 72 |
73 | 74 | 75 | 76 |
77 |

Developed by Rami Krispin.

78 |
79 | 80 |
81 |

Site built with pkgdown 82 | 2.0.1.

83 |
84 | 85 |
86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | -------------------------------------------------------------------------------- /docs/LICENSE.html: -------------------------------------------------------------------------------- 1 | 2 | MIT License • EIAapi 6 | 7 | 8 |
9 |
53 | 54 | 55 | 56 |
57 |
58 |
59 |

MIT License

60 |
61 | 62 |
63 | 64 |

Copyright (c) 2023 EIAapi authors

65 |

Permission is hereby granted, free of charge, to any person obtaining 66 | a copy of this software and associated documentation files (the 67 | “Software”), to deal in the Software without restriction, including 68 | without limitation the rights to use, copy, modify, merge, publish, 69 | distribute, sublicense, and/or sell copies of the Software, and to 70 | permit persons to whom the Software is furnished to do so, subject to 71 | the following conditions:

72 |

The above copyright notice and this permission notice shall be 73 | included in all copies or substantial portions of the Software.

74 |

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, 75 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 76 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 77 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 78 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 79 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 80 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

81 |
82 | 83 |
84 | 85 | 88 | 89 |
90 | 91 | 92 | 93 |
94 |

Developed by Rami Krispin.

95 |
96 | 97 |
98 |

Site built with pkgdown 99 | 2.0.1.

100 |
101 | 102 |
103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | -------------------------------------------------------------------------------- /docs/articles/backfile.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Working with Large Dataset • EIAapi 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 20 | 21 | 22 | 23 | 24 |
25 |
77 | 78 | 79 | 80 | 81 |
82 |
83 |
84 |

Working with Large Dataset

85 | 86 | 87 | Source: vignettes/backfile.Rmd 88 | 89 | 90 |
91 | 92 | 93 | 94 |

One of the limitations of the EIA API is the 5000 observations limit 95 | per call. This could be challenging if you are trying to pull hourly 96 | time series, which is roughly 26280 observations per year. The 97 | eia_backfill function solves this issue and removes the API 98 | number of observations per call limit. On the backend, the function 99 | splits the query into multiple sequential queries, pulls the data, and 100 | returns an append object.

101 |

For example, let’s pull hourly generation of electricity by New York 102 | Independent System Operator on sub-region West using the 103 | eia_backfill function:

104 | 
105 | library(EIAapi)
106 | 
107 |  start <- as.POSIXlt("2018-06-19T00", tz = "UTC")
108 |  end <- lubridate::floor_date(Sys.time()- lubridate::days(2), unit = "day")
109 |  attr(end, "tzone") <- "UTC"
110 |  offset <- 5000
111 |  api_key <- Sys.getenv("eia_key")
112 |  api_path <- "electricity/rto/region-sub-ba-data/data/"
113 | 
114 |  facets = list(parent = "NYIS",
115 |                subba = "ZONA")
116 |  
117 |   df <- eia_backfill(start = start,
118 |                 end = end,
119 |                 offset = offset,
120 |                 api_key = api_key,
121 |                 api_path = api_path,
122 |                 facets = facets)
123 |

As you can see below, the return series has more than 45,000 124 | observations:

125 | 
126 | head(df)
127 | #>                  time subba subba_name parent
128 | #> 1 2018-06-19 05:00:00  ZONA       West   NYIS
129 | #> 2 2018-06-19 06:00:00  ZONA       West   NYIS
130 | #> 3 2018-06-19 07:00:00  ZONA       West   NYIS
131 | #> 4 2018-06-19 08:00:00  ZONA       West   NYIS
132 | #> 5 2018-06-19 09:00:00  ZONA       West   NYIS
133 | #> 6 2018-06-19 10:00:00  ZONA       West   NYIS
134 | #>                            parent_name value   value_units
135 | #> 1 New York Independent System Operator  1848 megawatthours
136 | #> 2 New York Independent System Operator  1754 megawatthours
137 | #> 3 New York Independent System Operator  1699 megawatthours
138 | #> 4 New York Independent System Operator  1650 megawatthours
139 | #> 5 New York Independent System Operator  1640 megawatthours
140 | #> 6 New York Independent System Operator  1673 megawatthours
141 | 
142 | nrow(df)
143 | #> [1] 45074
144 | 
145 | at_y <- pretty(df$value)[c(2, 4, 6)]
146 |  at_x <- seq.POSIXt(from = start,
147 |                   to = end,
148 |                   by = "2 years")
149 |  plot(df$time, df$value,
150 |       col = "#1f77b4",
151 |       type = "l",
152 |       frame.plot = FALSE,
153 |       axes = FALSE,
154 |       panel.first = abline(h = at_y, col = "grey80"),
155 |       main = "NY Independent System Operator (West) - Hourly Generation of Electricity",
156 |       xlab = "Source: https://www.eia.gov/",
157 |       ylab = "MegaWatt/Hours")
158 | 
159 |  mtext(side =1, text = format(at_x, format = "%Y"), at = at_x,
160 |        col = "grey20", line = 1, cex = 0.8)
161 | 
162 |  mtext(side =2, text = format(at_y, scientific = FALSE), at = at_y,
163 |        col = "grey20", line = 1, cex = 0.8)
164 |

165 |
166 | 167 | 170 | 171 |
172 | 173 | 174 | 175 |
176 |

177 |

Developed by Rami Krispin.

178 |
179 | 180 |
181 |

182 |

Site built with pkgdown 183 | 2.0.1.

184 |
185 | 186 |
187 |
188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | -------------------------------------------------------------------------------- /docs/articles/backfile_files/figure-html/unnamed-chunk-2-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/articles/backfile_files/figure-html/unnamed-chunk-2-1.png -------------------------------------------------------------------------------- /docs/articles/images/dashboard1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/articles/images/dashboard1.png -------------------------------------------------------------------------------- /docs/articles/images/dashboard2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/articles/images/dashboard2.png -------------------------------------------------------------------------------- /docs/articles/images/register.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/articles/images/register.png -------------------------------------------------------------------------------- /docs/articles/index.html: -------------------------------------------------------------------------------- 1 | 2 | Articles • EIAapi 6 | 7 | 8 |
9 |
53 | 54 | 55 | 56 |
57 |
58 |
59 |

Articles

60 |
61 | 62 |
63 |

All vignettes

64 |

65 | 66 |
Working with Large Dataset
67 |
68 |
Introduction to the EIA API
69 |
70 |
71 |
72 |
73 | 74 | 75 |
76 |

Developed by Rami Krispin.

77 |
78 | 79 |
80 |

Site built with pkgdown 81 | 2.0.1.

82 |
83 | 84 |
85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /docs/articles/intro.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Introduction to the EIA API • EIAapi 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 20 | 21 | 22 | 23 | 24 |
25 |
77 | 78 | 79 | 80 | 81 |
82 |
83 |
84 |

Introduction to the EIA API

85 | 86 | 87 | Source: vignettes/intro.Rmd 88 | 89 | 90 |
91 | 92 | 93 | 94 |

The U.S. Energy Information 95 | Administration (EIA) hosts a variety of time series data describing 96 | the U.S. energy sector. The EIA data is open and accessible through an 97 | Application Programming Interface (API) for free. The 98 | EIAapi package provides functions to query and pull 99 | tidy data from the EIA API 100 | v2.

101 |
102 |

Prerequisites 103 |

104 |

To pull data from the API using this package, you will need the 105 | following:

106 |
    107 |
  • 108 | jq - The package uses jq to parse the API output 109 | from JSON to tabular format. To download and install jq follow the 110 | instructions on the download page.
    • 111 |
  • 112 | API key - To query the EIA API, you must register 113 | to the service to receive the API key.
    • 114 |
115 |

To register to the API, go to https://www.eia.gov/opendata/, click the 116 | Register button, and follow the instructions (see the 117 | screenshot below).

118 |


119 |

120 |
121 |
122 |

Getting started with the EIAapi 123 |

124 |

The EIAapi package provides the following functions:

125 |
    126 |
  • 127 | eia_metadata - Returns information and metadata from 128 | the API about the available categories and subcategories (routes)
    • 129 |
  • 130 | eia_get - Enables query data from the API
    • 131 |
132 |

A good place to start exploring the available categories and 133 | subcategories of the API is on the API Dashboard. For 134 | example, let’s explore the hourly demand for electricity in the 135 | California sub-region in the dashboard and use the metadata to set the 136 | query parameters. Let’s start by set the API Route to 137 | Electricity -> 138 | Electric Power Operrations (Daily And Hourly) -> 139 | Hourly Demand Subregion, as shown in the screenshot below. 140 | Once the data route is set, define the PARENT facet as 141 | CISO. This will filter the options in the next facet - 142 | SUBBA to the four operators in California. Let’s select 143 | CISO: (PGAE) Pacific Gas and Electric and save the 144 | selection:

145 |


146 |

147 |


148 |

Once finalize the routes,facets, and any other filters, you can 149 | submit the query and you should receive the API metadata. For the 150 | selection above, here is the expected respond:

151 |


152 |

153 |


154 |

Let’s look at the returned metadata above and use it to set the query 155 | to pull the data, starting with the API URL:

156 | 
https://api.eia.gov/v2/electricity/rto/region-sub-ba-data/data/?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000
157 |

The URL contains the following components:

158 |
    159 |
  • The API endpoint - https://api.eia.gov/v2/ 160 |
    • 161 |
  • The data path - rto/region-sub-ba-data/data/ 162 |
    • 163 |
  • The query - 164 | ?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000 165 |
    • 166 |
167 |

We will use the data path to set the api_path. To set 168 | the query parameters, you can use the embedded query in the URL by 169 | extracting the values of the different filters (frequency, facet, etc.) 170 | or use the query header on the right side:

171 | 

172 | X-Params: {
173 | "frequency": "hourly",
174 | "data": [
175 | "value"
176 | ],
177 | "facets": {
178 | "parent": [
179 | "CISO"
180 | ],
181 | "subba": [
182 | "PGAE"
183 | ]
184 | },
185 | "start": null,
186 | "end": null,
187 | "sort": [
188 | {
189 | "column": "period",
190 | "direction": "desc"
191 | }
192 | ],
193 | "offset": 0,
194 | "length": 5000
195 | }
196 |

We can now set the query by defining the frequency, 197 | data, and facets arguments as defined in the 198 | header above:

199 | 
200 | library(EIAapi)
201 | 
202 | df1 <- eia_get(api_key = Sys.getenv("eia_key"),
203 |                api_path = "electricity/rto/region-sub-ba-data/data/",
204 |                frequency = "hourly",
205 |                data = "value",
206 |                facets = list(parent = "CISO",
207 |                              subba = "PGAE"),
208 |                offset = 0,
209 |                length = 5000)
210 | 
211 | head(df1)
212 | #>          period subba               subba-name parent
213 | #> 1 2019-09-02T07  PGAE Pacific Gas and Electric   CISO
214 | #> 2 2019-09-01T11  PGAE Pacific Gas and Electric   CISO
215 | #> 3 2019-09-01T10  PGAE Pacific Gas and Electric   CISO
216 | #> 4 2019-09-02T04  PGAE Pacific Gas and Electric   CISO
217 | #> 5 2019-09-01T21  PGAE Pacific Gas and Electric   CISO
218 | #> 6 2019-09-02T05  PGAE Pacific Gas and Electric   CISO
219 | #>                              parent-name value   value-units
220 | #> 1 California Independent System Operator 12431 megawatthours
221 | #> 2 California Independent System Operator 10270 megawatthours
222 | #> 3 California Independent System Operator 10620 megawatthours
223 | #> 4 California Independent System Operator 15685 megawatthours
224 | #> 5 California Independent System Operator 12945 megawatthours
225 | #> 6 California Independent System Operator 14750 megawatthours
226 | 
227 | unique(df1$parent)
228 | #> [1] "CISO"
229 | unique(df1$subba)
230 | #> [1] "PGAE"
231 |

Note: The API limit the number of observations per a 232 | call to 5000. If you wish to pull more than 5000 observations you can 233 | iterate your call and set the start and end 234 | arguments as a rolling window function. Similarly, you can use the 235 | length and offset arguments to define the 236 | rolling function.

237 |
238 |
239 |

API metadata 240 |

241 |

The EIA API provides detailed information and metadata about the 242 | categories and time series available in the API. This enables extracting 243 | information programmatically about different data parameters. The 244 | eia_metadata enables query metadata from the API. The function has two 245 | arguments:

246 |
    247 |
  • 248 | api_path - the API category/route path following the 249 | API endpoint (i.e., https://api.eia.gov/v2/)
    • 250 |
  • 251 | api_key - the API key
    • 252 |
253 |
254 |

Category metadata 255 |

256 |

Querying metadata is done by sending a request with a specific route 257 | path, and the API, in response, returns the available sub-categories 258 | under that path. For example, setting the api_path argument 259 | empty or NULL returns a list with the API main 260 | categories:

261 | 
262 | main_route <- eia_metadata(api_path = "",
263 |                            api_key = Sys.getenv("eia_key"))
264 | 
265 | main_route$routes[, c("id", "name")]
266 | #>                   id                            name
267 | #> 1               coal                            Coal
268 | #> 2  crude-oil-imports               Crude Oil Imports
269 | #> 3        electricity                     Electricity
270 | #> 4      international                   International
271 | #> 5        natural-gas                     Natural Gas
272 | #> 6    nuclear-outages                 Nuclear Outages
273 | #> 7          petroleum                       Petroleum
274 | #> 8               seds State Energy Data System (SEDS)
275 | #> 9               steo       Short Term Energy Outlook
276 | #> 10 densified-biomass               Densified Biomass
277 | #> 11      total-energy                    Total Energy
278 | #> 12               aeo           Annual Energy Outlook
279 | #> 13               ieo    International Energy Outlook
280 | #> 14     co2-emissions             State CO2 Emissions
281 |

Similarly, we can continue and extract the sub-categories under the 282 | electricity category by setting the api_path argument to 283 | electricity:

284 | 
285 | main_route <- eia_metadata(api_path = "electricity",
286 |                            api_key = Sys.getenv("eia_key"))
287 | 
288 | main_route$routes[, c("id", "name")]
289 | #>                                id
290 | #> 1                    retail-sales
291 | #> 2 electric-power-operational-data
292 | #> 3                             rto
293 | #> 4      state-electricity-profiles
294 | #> 5    operating-generator-capacity
295 | #> 6                   facility-fuel
296 | #>                                                                         name
297 | #> 1                                    Electricity Sales to Ultimate Customers
298 | #> 2                             Electric Power Operations (Annual and Monthly)
299 | #> 3                               Electric Power Operations (Daily and Hourly)
300 | #> 4                                                        State Specific Data
301 | #> 5                                           Inventory of Operable Generators
302 | #> 6 Electric Power Operations for Individual Power Plants (Annual and Monthly)
303 |

Note: the number of sub-categories or routes varies 304 | between the different categories.

305 |
306 |
307 |

Series metadata 308 |

309 |

The last route in the API path defines the series available under 310 | those categories. For example, we used the above to pull the hourly 311 | demand for electricity by sub-region the following URL:

312 | 
https://api.eia.gov/v2/electricity/rto/region-sub-ba-data/data/?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000
313 |

In this case, the path is defined as - 314 | electricity/rto/region-sub-ba-data/, where:

315 |
    316 |
  • 317 | electricity is the main category
    • 318 |
  • 319 | rto - is the sub-category under electricity 320 | representing the Electric Power Operations (Daily and Hourly), and
    • 321 |
  • 322 | region-sub-ba-data - is the last sub-category in this 323 | route, representing the Hourly Demand by Sub-region
    • 324 |
325 |

You can notice that the last sub-category in this path - 326 | region-sub-ba-data ended with data to indicate 327 | to the user that this is the last route, and under it there is data. The 328 | metadata for the time series has a different structure with respect to 329 | the category metadata. It provides useful information such as: - 330 | Starting and ending time of the series - Available facets - Series 331 | Frequency

332 |

For example, let’s query for Demand by Sub-region metadata:

333 | 
334 | elec_sub_route <- eia_metadata(api_path = "electricity/rto/region-sub-ba-data/",
335 |                                api_key = Sys.getenv("eia_key"))
336 | 
337 | elec_sub_route 
338 | #> $id
339 | #> [1] "region-sub-ba-data"
340 | #> 
341 | #> $name
342 | #> [1] "Hourly Demand by Subregion"
343 | #> 
344 | #> $description
345 | #> [1] "Hourly demand by balancing authority subregion.  \n    Source: Form EIA-930\n    Product: Hourly Electric Grid Monitor"
346 | #> 
347 | #> $frequency
348 | #>             id                    alias
349 | #> 1       hourly             hourly (UTC)
350 | #> 2 local-hourly hourly (Local Time Zone)
351 | #>                                   description query               format
352 | #> 1   One data point for each hour in UTC time.     H    YYYY-MM-DD"T"HH24
353 | #> 2 One data point for each hour in local time.    LH YYYY-MM-DD"T"HH24TZH
354 | #> 
355 | #> $facets
356 | #>       id         description
357 | #> 1  subba           Subregion
358 | #> 2 parent Balancing Authority
359 | #> 
360 | #> $data
361 | #> $data$value
362 | #> $data$value$`aggregation-method`
363 | #> [1] "SUM"
364 | #> 
365 | #> $data$value$alias
366 | #> [1] "Demand"
367 | #> 
368 | #> $data$value$units
369 | #> [1] "megawatthours"
370 | #> 
371 | #> 
372 | #> 
373 | #> $startPeriod
374 | #> [1] "2018-06-19T05"
375 | #> 
376 | #> $endPeriod
377 | #> [1] "2023-08-11T07"
378 | #> 
379 | #> $defaultDateFormat
380 | #> [1] "YYYY-MM-DD\"T\"HH24"
381 | #> 
382 | #> $defaultFrequency
383 | #> [1] "hourly"
384 | #> 
385 | #> $command
386 | #> [1] "/v2/electricity/rto/region-sub-ba-data/"
387 | #> 
388 | #> attr(,"class")
389 | #> [1] "list"         "eia_metadata"
390 |

You can continue and query additional information on this series. For 391 | example, let’s extract the list of balancing authorities that are 392 | available for this series under the parent facet:

393 | 
394 |  eia_metadata(api_path = "electricity/rto/region-sub-ba-data/facet/parent",
395 |                                api_key = Sys.getenv("eia_key"))
396 | #> $totalFacets
397 | #> [1] 8
398 | #> 
399 | #> $facets
400 | #>     id                                           name
401 | #> 1 ISNE                                ISO New England
402 | #> 2 NYIS           New York Independent System Operator
403 | #> 3 SWPP                           Southwest Power Pool
404 | #> 4 ERCO    Electric Reliability Council of Texas, Inc.
405 | #> 5 MISO Midcontinent Independent System Operator, Inc.
406 | #> 6 CISO         California Independent System Operator
407 | #> 7  PNM           Public Service Company of New Mexico
408 | #> 8  PJM                       PJM Interconnection, LLC
409 | #>                                                   alias
410 | #> 1                                (ISNE) ISO New England
411 | #> 2           (NYIS) New York Independent System Operator
412 | #> 3                           (SWPP) Southwest Power Pool
413 | #> 4    (ERCO) Electric Reliability Council of Texas, Inc.
414 | #> 5 (MISO) Midcontinent Independent System Operator, Inc.
415 | #> 6         (CISO) California Independent System Operator
416 | #> 7            (PNM) Public Service Company of New Mexico
417 | #> 8                        (PJM) PJM Interconnection, LLC
418 | #> 
419 | #> $command
420 | #> [1] "/v2/electricity/rto/region-sub-ba-data/facet/parent/"
421 | #> 
422 | #> attr(,"class")
423 | #> [1] "list"         "eia_metadata"
424 |
425 |
426 |
427 | 428 | 433 | 434 |
435 | 436 | 437 | 438 |
439 |

440 |

Developed by Rami Krispin.

441 |
442 | 443 |
444 |

445 |

Site built with pkgdown 446 | 2.0.1.

447 |
448 | 449 |
450 |
451 | 452 | 453 | 454 | 455 | 456 | 457 | 458 | 459 | -------------------------------------------------------------------------------- /docs/authors.html: -------------------------------------------------------------------------------- 1 | 2 | Authors and Citation • EIAapi 6 | 7 | 8 |
9 |
53 | 54 | 55 | 56 |
57 |
58 |
59 |
60 |

Authors

61 |
62 | 63 | 64 |
  • 65 |

    Rami Krispin. Author, maintainer. 66 |

    67 |
    • 68 |
69 |
70 |
71 |

Citation

72 | Source: DESCRIPTION 73 |
74 |
75 | 76 | 77 |

Krispin R (2023). 78 | EIAapi: Query Data from the 'EIA' API. 79 | R package version 0.1.2, https://github.com/RamiKrispin/EIAapi. 80 |

81 | 
@Manual{,
 82 |   title = {EIAapi: Query Data from the 'EIA' API},
 83 |   author = {Rami Krispin},
 84 |   year = {2023},
 85 |   note = {R package version 0.1.2},
 86 |   url = {https://github.com/RamiKrispin/EIAapi},
 87 | }
88 | 89 |
90 | 91 |
92 | 93 | 94 | 95 |
96 |

Developed by Rami Krispin.

97 |
98 | 99 |
100 |

Site built with pkgdown 101 | 2.0.1.

102 |
103 | 104 |
105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | -------------------------------------------------------------------------------- /docs/bootstrap-toc.css: -------------------------------------------------------------------------------- 1 | /*! 2 | * Bootstrap Table of Contents v0.4.1 (http://afeld.github.io/bootstrap-toc/) 3 | * Copyright 2015 Aidan Feldman 4 | * Licensed under MIT (https://github.com/afeld/bootstrap-toc/blob/gh-pages/LICENSE.md) */ 5 | 6 | /* modified from https://github.com/twbs/bootstrap/blob/94b4076dd2efba9af71f0b18d4ee4b163aa9e0dd/docs/assets/css/src/docs.css#L548-L601 */ 7 | 8 | /* All levels of nav */ 9 | nav[data-toggle='toc'] .nav > li > a { 10 | display: block; 11 | padding: 4px 20px; 12 | font-size: 13px; 13 | font-weight: 500; 14 | color: #767676; 15 | } 16 | nav[data-toggle='toc'] .nav > li > a:hover, 17 | nav[data-toggle='toc'] .nav > li > a:focus { 18 | padding-left: 19px; 19 | color: #563d7c; 20 | text-decoration: none; 21 | background-color: transparent; 22 | border-left: 1px solid #563d7c; 23 | } 24 | nav[data-toggle='toc'] .nav > .active > a, 25 | nav[data-toggle='toc'] .nav > .active:hover > a, 26 | nav[data-toggle='toc'] .nav > .active:focus > a { 27 | padding-left: 18px; 28 | font-weight: bold; 29 | color: #563d7c; 30 | background-color: transparent; 31 | border-left: 2px solid #563d7c; 32 | } 33 | 34 | /* Nav: second level (shown on .active) */ 35 | nav[data-toggle='toc'] .nav .nav { 36 | display: none; /* Hide by default, but at >768px, show it */ 37 | padding-bottom: 10px; 38 | } 39 | nav[data-toggle='toc'] .nav .nav > li > a { 40 | padding-top: 1px; 41 | padding-bottom: 1px; 42 | padding-left: 30px; 43 | font-size: 12px; 44 | font-weight: normal; 45 | } 46 | nav[data-toggle='toc'] .nav .nav > li > a:hover, 47 | nav[data-toggle='toc'] .nav .nav > li > a:focus { 48 | padding-left: 29px; 49 | } 50 | nav[data-toggle='toc'] .nav .nav > .active > a, 51 | nav[data-toggle='toc'] .nav .nav > .active:hover > a, 52 | nav[data-toggle='toc'] .nav .nav > .active:focus > a { 53 | padding-left: 28px; 54 | font-weight: 500; 55 | } 56 | 57 | /* from https://github.com/twbs/bootstrap/blob/e38f066d8c203c3e032da0ff23cd2d6098ee2dd6/docs/assets/css/src/docs.css#L631-L634 */ 58 | nav[data-toggle='toc'] .nav > .active > ul { 59 | display: block; 60 | } 61 | -------------------------------------------------------------------------------- /docs/bootstrap-toc.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Bootstrap Table of Contents v0.4.1 (http://afeld.github.io/bootstrap-toc/) 3 | * Copyright 2015 Aidan Feldman 4 | * Licensed under MIT (https://github.com/afeld/bootstrap-toc/blob/gh-pages/LICENSE.md) */ 5 | (function() { 6 | 'use strict'; 7 | 8 | window.Toc = { 9 | helpers: { 10 | // return all matching elements in the set, or their descendants 11 | findOrFilter: function($el, selector) { 12 | // http://danielnouri.org/notes/2011/03/14/a-jquery-find-that-also-finds-the-root-element/ 13 | // http://stackoverflow.com/a/12731439/358804 14 | var $descendants = $el.find(selector); 15 | return $el.filter(selector).add($descendants).filter(':not([data-toc-skip])'); 16 | }, 17 | 18 | generateUniqueIdBase: function(el) { 19 | var text = $(el).text(); 20 | var anchor = text.trim().toLowerCase().replace(/[^A-Za-z0-9]+/g, '-'); 21 | return anchor || el.tagName.toLowerCase(); 22 | }, 23 | 24 | generateUniqueId: function(el) { 25 | var anchorBase = this.generateUniqueIdBase(el); 26 | for (var i = 0; ; i++) { 27 | var anchor = anchorBase; 28 | if (i > 0) { 29 | // add suffix 30 | anchor += '-' + i; 31 | } 32 | // check if ID already exists 33 | if (!document.getElementById(anchor)) { 34 | return anchor; 35 | } 36 | } 37 | }, 38 | 39 | generateAnchor: function(el) { 40 | if (el.id) { 41 | return el.id; 42 | } else { 43 | var anchor = this.generateUniqueId(el); 44 | el.id = anchor; 45 | return anchor; 46 | } 47 | }, 48 | 49 | createNavList: function() { 50 | return $('
    '); 51 | }, 52 | 53 | createChildNavList: function($parent) { 54 | var $childList = this.createNavList(); 55 | $parent.append($childList); 56 | return $childList; 57 | }, 58 | 59 | generateNavEl: function(anchor, text) { 60 | var $a = $(''); 61 | $a.attr('href', '#' + anchor); 62 | $a.text(text); 63 | var $li = $('
  • '); 64 | $li.append($a); 65 | return $li; 66 | }, 67 | 68 | generateNavItem: function(headingEl) { 69 | var anchor = this.generateAnchor(headingEl); 70 | var $heading = $(headingEl); 71 | var text = $heading.data('toc-text') || $heading.text(); 72 | return this.generateNavEl(anchor, text); 73 | }, 74 | 75 | // Find the first heading level (`

    `, then `

    `, etc.) that has more than one element. Defaults to 1 (for `

    `). 76 | getTopLevel: function($scope) { 77 | for (var i = 1; i <= 6; i++) { 78 | var $headings = this.findOrFilter($scope, 'h' + i); 79 | if ($headings.length > 1) { 80 | return i; 81 | } 82 | } 83 | 84 | return 1; 85 | }, 86 | 87 | // returns the elements for the top level, and the next below it 88 | getHeadings: function($scope, topLevel) { 89 | var topSelector = 'h' + topLevel; 90 | 91 | var secondaryLevel = topLevel + 1; 92 | var secondarySelector = 'h' + secondaryLevel; 93 | 94 | return this.findOrFilter($scope, topSelector + ',' + secondarySelector); 95 | }, 96 | 97 | getNavLevel: function(el) { 98 | return parseInt(el.tagName.charAt(1), 10); 99 | }, 100 | 101 | populateNav: function($topContext, topLevel, $headings) { 102 | var $context = $topContext; 103 | var $prevNav; 104 | 105 | var helpers = this; 106 | $headings.each(function(i, el) { 107 | var $newNav = helpers.generateNavItem(el); 108 | var navLevel = helpers.getNavLevel(el); 109 | 110 | // determine the proper $context 111 | if (navLevel === topLevel) { 112 | // use top level 113 | $context = $topContext; 114 | } else if ($prevNav && $context === $topContext) { 115 | // create a new level of the tree and switch to it 116 | $context = helpers.createChildNavList($prevNav); 117 | } // else use the current $context 118 | 119 | $context.append($newNav); 120 | 121 | $prevNav = $newNav; 122 | }); 123 | }, 124 | 125 | parseOps: function(arg) { 126 | var opts; 127 | if (arg.jquery) { 128 | opts = { 129 | $nav: arg 130 | }; 131 | } else { 132 | opts = arg; 133 | } 134 | opts.$scope = opts.$scope || $(document.body); 135 | return opts; 136 | } 137 | }, 138 | 139 | // accepts a jQuery object, or an options object 140 | init: function(opts) { 141 | opts = this.helpers.parseOps(opts); 142 | 143 | // ensure that the data attribute is in place for styling 144 | opts.$nav.attr('data-toggle', 'toc'); 145 | 146 | var $topContext = this.helpers.createChildNavList(opts.$nav); 147 | var topLevel = this.helpers.getTopLevel(opts.$scope); 148 | var $headings = this.helpers.getHeadings(opts.$scope, topLevel); 149 | this.helpers.populateNav($topContext, topLevel, $headings); 150 | } 151 | }; 152 | 153 | $(function() { 154 | $('nav[data-toggle="toc"]').each(function(i, el) { 155 | var $nav = $(el); 156 | Toc.init($nav); 157 | }); 158 | }); 159 | })(); 160 | -------------------------------------------------------------------------------- /docs/docsearch.css: -------------------------------------------------------------------------------- 1 | /* Docsearch -------------------------------------------------------------- */ 2 | /* 3 | Source: https://github.com/algolia/docsearch/ 4 | License: MIT 5 | */ 6 | 7 | .algolia-autocomplete { 8 | display: block; 9 | -webkit-box-flex: 1; 10 | -ms-flex: 1; 11 | flex: 1 12 | } 13 | 14 | .algolia-autocomplete .ds-dropdown-menu { 15 | width: 100%; 16 | min-width: none; 17 | max-width: none; 18 | padding: .75rem 0; 19 | background-color: #fff; 20 | background-clip: padding-box; 21 | border: 1px solid rgba(0, 0, 0, .1); 22 | box-shadow: 0 .5rem 1rem rgba(0, 0, 0, .175); 23 | } 24 | 25 | @media (min-width:768px) { 26 | .algolia-autocomplete .ds-dropdown-menu { 27 | width: 175% 28 | } 29 | } 30 | 31 | .algolia-autocomplete .ds-dropdown-menu::before { 32 | display: none 33 | } 34 | 35 | .algolia-autocomplete .ds-dropdown-menu [class^=ds-dataset-] { 36 | padding: 0; 37 | background-color: rgb(255,255,255); 38 | border: 0; 39 | max-height: 80vh; 40 | } 41 | 42 | .algolia-autocomplete .ds-dropdown-menu .ds-suggestions { 43 | margin-top: 0 44 | } 45 | 46 | .algolia-autocomplete .algolia-docsearch-suggestion { 47 | padding: 0; 48 | overflow: visible 49 | } 50 | 51 | .algolia-autocomplete .algolia-docsearch-suggestion--category-header { 52 | padding: .125rem 1rem; 53 | margin-top: 0; 54 | font-size: 1.3em; 55 | font-weight: 500; 56 | color: #00008B; 57 | border-bottom: 0 58 | } 59 | 60 | .algolia-autocomplete .algolia-docsearch-suggestion--wrapper { 61 | float: none; 62 | padding-top: 0 63 | } 64 | 65 | .algolia-autocomplete .algolia-docsearch-suggestion--subcategory-column { 66 | float: none; 67 | width: auto; 68 | padding: 0; 69 | text-align: left 70 | } 71 | 72 | .algolia-autocomplete .algolia-docsearch-suggestion--content { 73 | float: none; 74 | width: auto; 75 | padding: 0 76 | } 77 | 78 | .algolia-autocomplete .algolia-docsearch-suggestion--content::before { 79 | display: none 80 | } 81 | 82 | .algolia-autocomplete .ds-suggestion:not(:first-child) .algolia-docsearch-suggestion--category-header { 83 | padding-top: .75rem; 84 | margin-top: .75rem; 85 | border-top: 1px solid rgba(0, 0, 0, .1) 86 | } 87 | 88 | .algolia-autocomplete .ds-suggestion .algolia-docsearch-suggestion--subcategory-column { 89 | display: block; 90 | padding: .1rem 1rem; 91 | margin-bottom: 0.1; 92 | font-size: 1.0em; 93 | font-weight: 400 94 | /* display: none */ 95 | } 96 | 97 | .algolia-autocomplete .algolia-docsearch-suggestion--title { 98 | display: block; 99 | padding: .25rem 1rem; 100 | margin-bottom: 0; 101 | font-size: 0.9em; 102 | font-weight: 400 103 | } 104 | 105 | .algolia-autocomplete .algolia-docsearch-suggestion--text { 106 | padding: 0 1rem .5rem; 107 | margin-top: -.25rem; 108 | font-size: 0.8em; 109 | font-weight: 400; 110 | line-height: 1.25 111 | } 112 | 113 | .algolia-autocomplete .algolia-docsearch-footer { 114 | width: 110px; 115 | height: 20px; 116 | z-index: 3; 117 | margin-top: 10.66667px; 118 | float: right; 119 | font-size: 0; 120 | line-height: 0; 121 | } 122 | 123 | .algolia-autocomplete .algolia-docsearch-footer--logo { 124 | background-image: url("data:image/svg+xml;utf8,"); 125 | background-repeat: no-repeat; 126 | background-position: 50%; 127 | background-size: 100%; 128 | overflow: hidden; 129 | text-indent: -9000px; 130 | width: 100%; 131 | height: 100%; 132 | display: block; 133 | transform: translate(-8px); 134 | } 135 | 136 | .algolia-autocomplete .algolia-docsearch-suggestion--highlight { 137 | color: #FF8C00; 138 | background: rgba(232, 189, 54, 0.1) 139 | } 140 | 141 | 142 | .algolia-autocomplete .algolia-docsearch-suggestion--text .algolia-docsearch-suggestion--highlight { 143 | box-shadow: inset 0 -2px 0 0 rgba(105, 105, 105, .5) 144 | } 145 | 146 | .algolia-autocomplete .ds-suggestion.ds-cursor .algolia-docsearch-suggestion--content { 147 | background-color: rgba(192, 192, 192, .15) 148 | } 149 | -------------------------------------------------------------------------------- /docs/docsearch.js: -------------------------------------------------------------------------------- 1 | $(function() { 2 | 3 | // register a handler to move the focus to the search bar 4 | // upon pressing shift + "/" (i.e. "?") 5 | $(document).on('keydown', function(e) { 6 | if (e.shiftKey && e.keyCode == 191) { 7 | e.preventDefault(); 8 | $("#search-input").focus(); 9 | } 10 | }); 11 | 12 | $(document).ready(function() { 13 | // do keyword highlighting 14 | /* modified from https://jsfiddle.net/julmot/bL6bb5oo/ */ 15 | var mark = function() { 16 | 17 | var referrer = document.URL ; 18 | var paramKey = "q" ; 19 | 20 | if (referrer.indexOf("?") !== -1) { 21 | var qs = referrer.substr(referrer.indexOf('?') + 1); 22 | var qs_noanchor = qs.split('#')[0]; 23 | var qsa = qs_noanchor.split('&'); 24 | var keyword = ""; 25 | 26 | for (var i = 0; i < qsa.length; i++) { 27 | var currentParam = qsa[i].split('='); 28 | 29 | if (currentParam.length !== 2) { 30 | continue; 31 | } 32 | 33 | if (currentParam[0] == paramKey) { 34 | keyword = decodeURIComponent(currentParam[1].replace(/\+/g, "%20")); 35 | } 36 | } 37 | 38 | if (keyword !== "") { 39 | $(".contents").unmark({ 40 | done: function() { 41 | $(".contents").mark(keyword); 42 | } 43 | }); 44 | } 45 | } 46 | }; 47 | 48 | mark(); 49 | }); 50 | }); 51 | 52 | /* Search term highlighting ------------------------------*/ 53 | 54 | function matchedWords(hit) { 55 | var words = []; 56 | 57 | var hierarchy = hit._highlightResult.hierarchy; 58 | // loop to fetch from lvl0, lvl1, etc. 59 | for (var idx in hierarchy) { 60 | words = words.concat(hierarchy[idx].matchedWords); 61 | } 62 | 63 | var content = hit._highlightResult.content; 64 | if (content) { 65 | words = words.concat(content.matchedWords); 66 | } 67 | 68 | // return unique words 69 | var words_uniq = [...new Set(words)]; 70 | return words_uniq; 71 | } 72 | 73 | function updateHitURL(hit) { 74 | 75 | var words = matchedWords(hit); 76 | var url = ""; 77 | 78 | if (hit.anchor) { 79 | url = hit.url_without_anchor + '?q=' + escape(words.join(" ")) + '#' + hit.anchor; 80 | } else { 81 | url = hit.url + '?q=' + escape(words.join(" ")); 82 | } 83 | 84 | return url; 85 | } 86 | -------------------------------------------------------------------------------- /docs/link.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 5 | 8 | 12 | 13 | -------------------------------------------------------------------------------- /docs/news/index.html: -------------------------------------------------------------------------------- 1 | 2 | Changelog • EIAapi 6 | 7 | 8 |
    9 |
    53 | 54 | 55 | 56 |
    57 |
    58 |
    59 |

    Changelog

    60 | Source: NEWS.md 61 |
    62 | 63 |
    64 |

    EIAapi 0.1.2

    65 |
    • Added a new function, eia_backfill to handle large data 66 | query from the API
      • 67 |
    • Fixed issue with the eia_get function - 68 | curl commands failed to send query with brackets
      • 69 |
    • Added new vignette with example of the eia_backfill 70 | function usage
      • 71 |
    72 |
    73 |

    EIAapi 0.1.12023-04-30

    74 |
    • Modified the eia_get function argument - rename the 75 | api_url argument to api_path, and dropping the 76 | end-point (https://api.eia.gov/v2/) from the query
      • 77 |
    • Added new function - eia_metadata to query metadata 78 | information from the API
      • 79 |
    • Added new vignettes - introduction the EIA API
      • 80 |
    81 |
    82 |

    EIAapi 0.1.02023-04-12

    83 |
    • Added the eia_get function to query data from EIA 84 | API
      • 85 |
    • Added a NEWS.md file to track changes to the 86 | package
      • 87 |
    88 |
    89 | 90 | 93 | 94 |
    95 | 96 | 97 |
    98 |

    Developed by Rami Krispin.

    99 |
    100 | 101 |
    102 |

    Site built with pkgdown 103 | 2.0.1.

    104 |
    105 | 106 |
    107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | -------------------------------------------------------------------------------- /docs/pkgdown.css: -------------------------------------------------------------------------------- 1 | /* Sticky footer */ 2 | 3 | /** 4 | * Basic idea: https://philipwalton.github.io/solved-by-flexbox/demos/sticky-footer/ 5 | * Details: https://github.com/philipwalton/solved-by-flexbox/blob/master/assets/css/components/site.css 6 | * 7 | * .Site -> body > .container 8 | * .Site-content -> body > .container .row 9 | * .footer -> footer 10 | * 11 | * Key idea seems to be to ensure that .container and __all its parents__ 12 | * have height set to 100% 13 | * 14 | */ 15 | 16 | html, body { 17 | height: 100%; 18 | } 19 | 20 | body { 21 | position: relative; 22 | } 23 | 24 | body > .container { 25 | display: flex; 26 | height: 100%; 27 | flex-direction: column; 28 | } 29 | 30 | body > .container .row { 31 | flex: 1 0 auto; 32 | } 33 | 34 | footer { 35 | margin-top: 45px; 36 | padding: 35px 0 36px; 37 | border-top: 1px solid #e5e5e5; 38 | color: #666; 39 | display: flex; 40 | flex-shrink: 0; 41 | } 42 | footer p { 43 | margin-bottom: 0; 44 | } 45 | footer div { 46 | flex: 1; 47 | } 48 | footer .pkgdown { 49 | text-align: right; 50 | } 51 | footer p { 52 | margin-bottom: 0; 53 | } 54 | 55 | img.icon { 56 | float: right; 57 | } 58 | 59 | /* Ensure in-page images don't run outside their container */ 60 | .contents img { 61 | max-width: 100%; 62 | height: auto; 63 | } 64 | 65 | /* Fix bug in bootstrap (only seen in firefox) */ 66 | summary { 67 | display: list-item; 68 | } 69 | 70 | /* Typographic tweaking ---------------------------------*/ 71 | 72 | .contents .page-header { 73 | margin-top: calc(-60px + 1em); 74 | } 75 | 76 | dd { 77 | margin-left: 3em; 78 | } 79 | 80 | /* Section anchors ---------------------------------*/ 81 | 82 | a.anchor { 83 | display: none; 84 | margin-left: 5px; 85 | width: 20px; 86 | height: 20px; 87 | 88 | background-image: url(./link.svg); 89 | background-repeat: no-repeat; 90 | background-size: 20px 20px; 91 | background-position: center center; 92 | } 93 | 94 | h1:hover .anchor, 95 | h2:hover .anchor, 96 | h3:hover .anchor, 97 | h4:hover .anchor, 98 | h5:hover .anchor, 99 | h6:hover .anchor { 100 | display: inline-block; 101 | } 102 | 103 | /* Fixes for fixed navbar --------------------------*/ 104 | 105 | .contents h1, .contents h2, .contents h3, .contents h4 { 106 | padding-top: 60px; 107 | margin-top: -40px; 108 | } 109 | 110 | /* Navbar submenu --------------------------*/ 111 | 112 | .dropdown-submenu { 113 | position: relative; 114 | } 115 | 116 | .dropdown-submenu>.dropdown-menu { 117 | top: 0; 118 | left: 100%; 119 | margin-top: -6px; 120 | margin-left: -1px; 121 | border-radius: 0 6px 6px 6px; 122 | } 123 | 124 | .dropdown-submenu:hover>.dropdown-menu { 125 | display: block; 126 | } 127 | 128 | .dropdown-submenu>a:after { 129 | display: block; 130 | content: " "; 131 | float: right; 132 | width: 0; 133 | height: 0; 134 | border-color: transparent; 135 | border-style: solid; 136 | border-width: 5px 0 5px 5px; 137 | border-left-color: #cccccc; 138 | margin-top: 5px; 139 | margin-right: -10px; 140 | } 141 | 142 | .dropdown-submenu:hover>a:after { 143 | border-left-color: #ffffff; 144 | } 145 | 146 | .dropdown-submenu.pull-left { 147 | float: none; 148 | } 149 | 150 | .dropdown-submenu.pull-left>.dropdown-menu { 151 | left: -100%; 152 | margin-left: 10px; 153 | border-radius: 6px 0 6px 6px; 154 | } 155 | 156 | /* Sidebar --------------------------*/ 157 | 158 | #pkgdown-sidebar { 159 | margin-top: 30px; 160 | position: -webkit-sticky; 161 | position: sticky; 162 | top: 70px; 163 | } 164 | 165 | #pkgdown-sidebar h2 { 166 | font-size: 1.5em; 167 | margin-top: 1em; 168 | } 169 | 170 | #pkgdown-sidebar h2:first-child { 171 | margin-top: 0; 172 | } 173 | 174 | #pkgdown-sidebar .list-unstyled li { 175 | margin-bottom: 0.5em; 176 | } 177 | 178 | /* bootstrap-toc tweaks ------------------------------------------------------*/ 179 | 180 | /* All levels of nav */ 181 | 182 | nav[data-toggle='toc'] .nav > li > a { 183 | padding: 4px 20px 4px 6px; 184 | font-size: 1.5rem; 185 | font-weight: 400; 186 | color: inherit; 187 | } 188 | 189 | nav[data-toggle='toc'] .nav > li > a:hover, 190 | nav[data-toggle='toc'] .nav > li > a:focus { 191 | padding-left: 5px; 192 | color: inherit; 193 | border-left: 1px solid #878787; 194 | } 195 | 196 | nav[data-toggle='toc'] .nav > .active > a, 197 | nav[data-toggle='toc'] .nav > .active:hover > a, 198 | nav[data-toggle='toc'] .nav > .active:focus > a { 199 | padding-left: 5px; 200 | font-size: 1.5rem; 201 | font-weight: 400; 202 | color: inherit; 203 | border-left: 2px solid #878787; 204 | } 205 | 206 | /* Nav: second level (shown on .active) */ 207 | 208 | nav[data-toggle='toc'] .nav .nav { 209 | display: none; /* Hide by default, but at >768px, show it */ 210 | padding-bottom: 10px; 211 | } 212 | 213 | nav[data-toggle='toc'] .nav .nav > li > a { 214 | padding-left: 16px; 215 | font-size: 1.35rem; 216 | } 217 | 218 | nav[data-toggle='toc'] .nav .nav > li > a:hover, 219 | nav[data-toggle='toc'] .nav .nav > li > a:focus { 220 | padding-left: 15px; 221 | } 222 | 223 | nav[data-toggle='toc'] .nav .nav > .active > a, 224 | nav[data-toggle='toc'] .nav .nav > .active:hover > a, 225 | nav[data-toggle='toc'] .nav .nav > .active:focus > a { 226 | padding-left: 15px; 227 | font-weight: 500; 228 | font-size: 1.35rem; 229 | } 230 | 231 | /* orcid ------------------------------------------------------------------- */ 232 | 233 | .orcid { 234 | font-size: 16px; 235 | color: #A6CE39; 236 | /* margins are required by official ORCID trademark and display guidelines */ 237 | margin-left:4px; 238 | margin-right:4px; 239 | vertical-align: middle; 240 | } 241 | 242 | /* Reference index & topics ----------------------------------------------- */ 243 | 244 | .ref-index th {font-weight: normal;} 245 | 246 | .ref-index td {vertical-align: top; min-width: 100px} 247 | .ref-index .icon {width: 40px;} 248 | .ref-index .alias {width: 40%;} 249 | .ref-index-icons .alias {width: calc(40% - 40px);} 250 | .ref-index .title {width: 60%;} 251 | 252 | .ref-arguments th {text-align: right; padding-right: 10px;} 253 | .ref-arguments th, .ref-arguments td {vertical-align: top; min-width: 100px} 254 | .ref-arguments .name {width: 20%;} 255 | .ref-arguments .desc {width: 80%;} 256 | 257 | /* Nice scrolling for wide elements --------------------------------------- */ 258 | 259 | table { 260 | display: block; 261 | overflow: auto; 262 | } 263 | 264 | /* Syntax highlighting ---------------------------------------------------- */ 265 | 266 | pre, code, pre code { 267 | background-color: #f8f8f8; 268 | color: #333; 269 | } 270 | pre, pre code { 271 | white-space: pre-wrap; 272 | word-break: break-all; 273 | overflow-wrap: break-word; 274 | } 275 | 276 | pre { 277 | border: 1px solid #eee; 278 | } 279 | 280 | pre .img, pre .r-plt { 281 | margin: 5px 0; 282 | } 283 | 284 | pre .img img, pre .r-plt img { 285 | background-color: #fff; 286 | } 287 | 288 | code a, pre a { 289 | color: #375f84; 290 | } 291 | 292 | a.sourceLine:hover { 293 | text-decoration: none; 294 | } 295 | 296 | .fl {color: #1514b5;} 297 | .fu {color: #000000;} /* function */ 298 | .ch,.st {color: #036a07;} /* string */ 299 | .kw {color: #264D66;} /* keyword */ 300 | .co {color: #888888;} /* comment */ 301 | 302 | .error {font-weight: bolder;} 303 | .warning {font-weight: bolder;} 304 | 305 | /* Clipboard --------------------------*/ 306 | 307 | .hasCopyButton { 308 | position: relative; 309 | } 310 | 311 | .btn-copy-ex { 312 | position: absolute; 313 | right: 0; 314 | top: 0; 315 | visibility: hidden; 316 | } 317 | 318 | .hasCopyButton:hover button.btn-copy-ex { 319 | visibility: visible; 320 | } 321 | 322 | /* headroom.js ------------------------ */ 323 | 324 | .headroom { 325 | will-change: transform; 326 | transition: transform 200ms linear; 327 | } 328 | .headroom--pinned { 329 | transform: translateY(0%); 330 | } 331 | .headroom--unpinned { 332 | transform: translateY(-100%); 333 | } 334 | 335 | /* mark.js ----------------------------*/ 336 | 337 | mark { 338 | background-color: rgba(255, 255, 51, 0.5); 339 | border-bottom: 2px solid rgba(255, 153, 51, 0.3); 340 | padding: 1px; 341 | } 342 | 343 | /* vertical spacing after htmlwidgets */ 344 | .html-widget { 345 | margin-bottom: 10px; 346 | } 347 | 348 | /* fontawesome ------------------------ */ 349 | 350 | .fab { 351 | font-family: "Font Awesome 5 Brands" !important; 352 | } 353 | 354 | /* don't display links in code chunks when printing */ 355 | /* source: https://stackoverflow.com/a/10781533 */ 356 | @media print { 357 | code a:link:after, code a:visited:after { 358 | content: ""; 359 | } 360 | } 361 | 362 | /* Section anchors --------------------------------- 363 | Added in pandoc 2.11: https://github.com/jgm/pandoc-templates/commit/9904bf71 364 | */ 365 | 366 | div.csl-bib-body { } 367 | div.csl-entry { 368 | clear: both; 369 | } 370 | .hanging-indent div.csl-entry { 371 | margin-left:2em; 372 | text-indent:-2em; 373 | } 374 | div.csl-left-margin { 375 | min-width:2em; 376 | float:left; 377 | } 378 | div.csl-right-inline { 379 | margin-left:2em; 380 | padding-left:1em; 381 | } 382 | div.csl-indent { 383 | margin-left: 2em; 384 | } 385 | -------------------------------------------------------------------------------- /docs/pkgdown.js: -------------------------------------------------------------------------------- 1 | /* http://gregfranko.com/blog/jquery-best-practices/ */ 2 | (function($) { 3 | $(function() { 4 | 5 | $('.navbar-fixed-top').headroom(); 6 | 7 | $('body').css('padding-top', $('.navbar').height() + 10); 8 | $(window).resize(function(){ 9 | $('body').css('padding-top', $('.navbar').height() + 10); 10 | }); 11 | 12 | $('[data-toggle="tooltip"]').tooltip(); 13 | 14 | var cur_path = paths(location.pathname); 15 | var links = $("#navbar ul li a"); 16 | var max_length = -1; 17 | var pos = -1; 18 | for (var i = 0; i < links.length; i++) { 19 | if (links[i].getAttribute("href") === "#") 20 | continue; 21 | // Ignore external links 22 | if (links[i].host !== location.host) 23 | continue; 24 | 25 | var nav_path = paths(links[i].pathname); 26 | 27 | var length = prefix_length(nav_path, cur_path); 28 | if (length > max_length) { 29 | max_length = length; 30 | pos = i; 31 | } 32 | } 33 | 34 | // Add class to parent
  • , and enclosing
  • if in dropdown 35 | if (pos >= 0) { 36 | var menu_anchor = $(links[pos]); 37 | menu_anchor.parent().addClass("active"); 38 | menu_anchor.closest("li.dropdown").addClass("active"); 39 | } 40 | }); 41 | 42 | function paths(pathname) { 43 | var pieces = pathname.split("/"); 44 | pieces.shift(); // always starts with / 45 | 46 | var end = pieces[pieces.length - 1]; 47 | if (end === "index.html" || end === "") 48 | pieces.pop(); 49 | return(pieces); 50 | } 51 | 52 | // Returns -1 if not found 53 | function prefix_length(needle, haystack) { 54 | if (needle.length > haystack.length) 55 | return(-1); 56 | 57 | // Special case for length-0 haystack, since for loop won't run 58 | if (haystack.length === 0) { 59 | return(needle.length === 0 ? 0 : -1); 60 | } 61 | 62 | for (var i = 0; i < haystack.length; i++) { 63 | if (needle[i] != haystack[i]) 64 | return(i); 65 | } 66 | 67 | return(haystack.length); 68 | } 69 | 70 | /* Clipboard --------------------------*/ 71 | 72 | function changeTooltipMessage(element, msg) { 73 | var tooltipOriginalTitle=element.getAttribute('data-original-title'); 74 | element.setAttribute('data-original-title', msg); 75 | $(element).tooltip('show'); 76 | element.setAttribute('data-original-title', tooltipOriginalTitle); 77 | } 78 | 79 | if(ClipboardJS.isSupported()) { 80 | $(document).ready(function() { 81 | var copyButton = ""; 82 | 83 | $("div.sourceCode").addClass("hasCopyButton"); 84 | 85 | // Insert copy buttons: 86 | $(copyButton).prependTo(".hasCopyButton"); 87 | 88 | // Initialize tooltips: 89 | $('.btn-copy-ex').tooltip({container: 'body'}); 90 | 91 | // Initialize clipboard: 92 | var clipboardBtnCopies = new ClipboardJS('[data-clipboard-copy]', { 93 | text: function(trigger) { 94 | return trigger.parentNode.textContent.replace(/\n#>[^\n]*/g, ""); 95 | } 96 | }); 97 | 98 | clipboardBtnCopies.on('success', function(e) { 99 | changeTooltipMessage(e.trigger, 'Copied!'); 100 | e.clearSelection(); 101 | }); 102 | 103 | clipboardBtnCopies.on('error', function() { 104 | changeTooltipMessage(e.trigger,'Press Ctrl+C or Command+C to copy'); 105 | }); 106 | }); 107 | } 108 | })(window.jQuery || window.$) 109 | -------------------------------------------------------------------------------- /docs/pkgdown.yml: -------------------------------------------------------------------------------- 1 | pandoc: 3.1.1 2 | pkgdown: 2.0.1 3 | pkgdown_sha: ~ 4 | articles: 5 | backfile: backfile.html 6 | intro: intro.html 7 | last_built: 2023-08-12T14:22Z 8 | 9 | -------------------------------------------------------------------------------- /docs/reference/Rplot001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/reference/Rplot001.png -------------------------------------------------------------------------------- /docs/reference/eia_backfill-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/reference/eia_backfill-1.png -------------------------------------------------------------------------------- /docs/reference/eia_backfill.html: -------------------------------------------------------------------------------- 1 | 2 | Pull a Large Number of Observations with a Sequential Query — eia_backfill • EIAapi 9 | 10 | 11 |
    12 |
    56 | 57 | 58 | 59 |
    60 |
    61 |
    62 |

    Pull a Large Number of Observations with a Sequential Query

    63 | Source: R/backfile.R 64 | 65 |
    66 | 67 |
    68 |

    This function allows users to overcome the API's observation limit 69 | per query by breaking down the query into smaller sequential sub-queries and 70 | appending back the results. The main use case of this function is for backfilling 71 | hourly series.

    72 |
    73 | 74 |
    75 | 
    eia_backfill(start, end, offset, api_key, api_path, facets)
    76 |
    77 | 78 |
    79 |

    Arguments

    80 |
    start
    81 |

    defines the start time of the series, should use a POSIXt class 82 | for hourly series or Date format for non-hourly series (daily, monthly, etc.)

    83 |
    end
    84 |

    defines the end time of the series, should use a POSIXt class 85 | for hourly series or Date format for non-hourly series (daily, monthly, etc.)

    86 |
    offset
    87 |

    An integer, defines the number of observations limitation per query. 88 | In line with the API limitation of up to 5000 observations per query, the offset argument's upper limit is 5000 observations.

    89 |
    api_key
    90 |

    A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service

    91 |
    api_path
    92 |

    A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 93 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/

    94 |
    facets
    95 |

    A list, optional, set the filtering argument (defined as 'facets' 96 | on the API header), following the structure of list(facet_name_1 = value_1, 97 | facet_name_2 = value_2)

    98 |
    99 |
    100 |

    Value

    101 |

    A time series

    102 |
    103 |
    104 |

    Details

    105 |

    The function use start, end, and offset arguments to define a sequence 106 | of queries.

    107 |
    108 | 109 |
    110 |

    Examples

    111 | 
    if (FALSE) {
112 |  start <- as.POSIXlt("2018-06-19T00", tz = "UTC")
113 |  end <- lubridate::floor_date(Sys.time()- lubridate::days(2), unit = "day")
114 |  attr(end, "tzone") <- "UTC"
115 |  offset <- 2000
116 |  api_key <- Sys.getenv("eia_key")
117 |  api_path <- "electricity/rto/region-sub-ba-data/data/"
118 | 
119 |  facets = list(parent = "NYIS",
120 |                subba = "ZONA")
121 | 
122 |  df <- eia_backfill(start = start,
123 |                 end = end,
124 |                 offset = offset,
125 |                 api_key = api_key,
126 |                 api_path = api_path,
127 |                 facets = facets)
128 | 
129 |  at_y <- pretty(df$value)[c(2, 4, 6)]
130 |  at_x <- seq.POSIXt(from = start,
131 |                   to = end,
132 |                   by = "2 years")
133 |  plot(df$time, df$value,
134 |       col = "#1f77b4",
135 |       type = "l",
136 |       frame.plot = FALSE,
137 |       axes = FALSE,
138 |       panel.first = abline(h = at_y, col = "grey80"),
139 |       main = "NY Independent System Operator (West) - Hourly Generation of Electricity",
140 |       xlab = "Source: https://www.eia.gov/",
141 |       ylab = "MegaWatt/Hours")
142 | 
143 |  mtext(side =1, text = format(at_x, format = "%Y"), at = at_x,
144 |        col = "grey20", line = 1, cex = 0.8)
145 | 
146 |  mtext(side =2, text = format(at_y, scientific = FALSE), at = at_y,
147 |        col = "grey20", line = 1, cex = 0.8)
148 | }
149 |
    150 |
    151 |
    152 | 155 |
    156 | 157 | 158 |
    159 |

    Developed by Rami Krispin.

    160 |
    161 | 162 |
    163 |

    Site built with pkgdown 164 | 2.0.1.

    165 |
    166 | 167 |
    168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | -------------------------------------------------------------------------------- /docs/reference/eia_get.html: -------------------------------------------------------------------------------- 1 | 2 | Query the EIA API — eia_get • EIAapi 6 | 7 | 8 |
    9 |
    53 | 54 | 55 | 56 |
    57 |
    58 |
    59 |

    Query the EIA API

    60 | Source: R/query.R 61 | 62 |
    63 | 64 |
    65 |

    Function to query and extract data from the EIA API v2

    66 |
    67 | 68 |
    69 | 
    eia_get(
 70 |   api_key,
 71 |   api_path,
 72 |   data = "value",
 73 |   facets = NULL,
 74 |   start = NULL,
 75 |   end = NULL,
 76 |   length = NULL,
 77 |   offset = NULL,
 78 |   frequency = NULL,
 79 |   format = "data.frame"
 80 | )
    81 |
    82 | 83 |
    84 |

    Arguments

    85 |
    api_key
    86 |

    A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service

    87 |
    api_path
    88 |

    A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 89 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/

    90 |
    data
    91 |

    A string, the metric type, by default uses 'value' (defined as 92 | 'data' on the API header)

    93 |
    facets
    94 |

    A list, optional, set the filtering argument (defined as 'facets' 95 | on the API header), following the structure of list(facet_name_1 = value_1, 96 | facet_name_2 = value_2)

    97 |
    start
    98 |

    A string, optional, set the starting date or time of the series 99 | using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series

    100 |
    end
    101 |

    A string, optional, set the ending date or time of the series 102 | using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series

    103 |
    length
    104 |

    An integer, optional, defines the length of the series, if set to 105 | NULL (default), will default to the API default value of 5000 observations per 106 | pull. The API enables a pull of up to 100K observations per call. If needed to 107 | pull more than the API limit per call, recommend to iterate the call with 108 | the use of the start, end and/or offset arguments

    109 |
    offset
    110 |

    An integer, optional, set the number of observations to offset 111 | from the default starting point of the series. If set to NULL (default), will default 112 | to the API default value of 0

    113 |
    frequency
    114 |

    A string, optional, define the API frequency argument 115 | (e.g., hourly, monthly, annual, etc.). If set to NULL (default), will default 116 | to the API default value

    117 |
    format
    118 |

    A string, defines the output of the return object to either 119 | "data.frame" (default) or "data.table"

    120 |
    121 |
    122 |

    Value

    123 |

    data.table/data.frame object

    124 |
    125 | 126 |
    127 |

    Examples

    128 | 
    if (FALSE) {
129 | # Required an EIA API key to send a query
130 | api_key <- "YOUR_API_KEY"
131 | 
132 | df <- eia_get(
133 |   api_key = api_key,
134 |   api_path = "electricity/rto/fuel-type-data/data/",
135 |   data = "value"
136 | )
137 | }
138 |
    139 |
    140 |
    141 | 144 |
    145 | 146 | 147 |
    148 |

    Developed by Rami Krispin.

    149 |
    150 | 151 |
    152 |

    Site built with pkgdown 153 | 2.0.1.

    154 |
    155 | 156 |
    157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | -------------------------------------------------------------------------------- /docs/reference/eia_metadata.html: -------------------------------------------------------------------------------- 1 | 2 | Pull Metadata from EIA API — eia_metadata • EIAapi 6 | 7 | 8 |
    9 |
    53 | 54 | 55 | 56 |
    57 |
    58 |
    59 |

    Pull Metadata from EIA API

    60 | Source: R/metadata.R 61 | 62 |
    63 | 64 |
    65 |

    Get data descriptions and metadata from the EIA API

    66 |
    67 | 68 |
    69 | 
    eia_metadata(api_path = NULL, api_key)
    70 |
    71 | 72 |
    73 |

    Arguments

    74 |
    api_path
    75 |

    A string, the API category/route path following the API endpoint (i.e., 'https://api.eia.gov/v2/') 76 | If set to NULL (default) or as empty string "" it returns the main categories available on the API. 77 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/

    78 |
    api_key
    79 |

    A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service

    80 |
    81 |
    82 |

    Value

    83 |

    a list object with the series description and metadata

    84 |
    85 |
    86 |

    Details

    87 |

    The function enables to explore the different data categories and available routes 88 | inline with the API dashboard (https://www.eia.gov/opendata/browser/)

    89 |
    90 | 91 |
    92 |

    Examples

    93 | 
    if (FALSE) {
 94 | electricity_metadata <- eia_metadata(api_key = Sys.getenv("eia_key"),
 95 |                                      api_path = "electricity")
 96 | 
 97 | electricity_metadata$response$description
 98 | electricity_metadata$response$id
 99 | electricity_metadata$response$name
100 | electricity_metadata$response$routes
101 | 
102 | }
103 |
    104 |
    105 |
    106 | 109 |
    110 | 111 | 112 |
    113 |

    Developed by Rami Krispin.

    114 |
    115 | 116 |
    117 |

    Site built with pkgdown 118 | 2.0.1.

    119 |
    120 | 121 |
    122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | -------------------------------------------------------------------------------- /docs/reference/figures/EIA_API_browser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/reference/figures/EIA_API_browser.png -------------------------------------------------------------------------------- /docs/reference/figures/README-pressure-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/reference/figures/README-pressure-1.png -------------------------------------------------------------------------------- /docs/reference/figures/README-unnamed-chunk-6-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/docs/reference/figures/README-unnamed-chunk-6-1.png -------------------------------------------------------------------------------- /docs/reference/index.html: -------------------------------------------------------------------------------- 1 | 2 | Function reference • EIAapi 6 | 7 | 8 |
    9 |
    53 | 54 | 55 | 56 |
    57 |
    58 |
    59 |

    Reference

    60 |
    61 | 62 | 66 | 69 | 70 | 73 | 74 | 77 | 78 |
    63 |

    All functions

    64 |

    65 |
    67 |

    eia_backfill()

    68 |

    Pull a Large Number of Observations with a Sequential Query

    71 |

    eia_get()

    72 |

    Query the EIA API

    75 |

    eia_metadata()

    76 |

    Pull Metadata from EIA API

    79 | 80 | 83 |
    84 | 85 | 86 |
    87 |

    Developed by Rami Krispin.

    88 |
    89 | 90 |
    91 |

    Site built with pkgdown 92 | 2.0.1.

    93 |
    94 | 95 |
    96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | -------------------------------------------------------------------------------- /docs/sitemap.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | /404.html 5 | 6 | 7 | /LICENSE-text.html 8 | 9 | 10 | /LICENSE.html 11 | 12 | 13 | /articles/backfile.html 14 | 15 | 16 | /articles/index.html 17 | 18 | 19 | /articles/intro.html 20 | 21 | 22 | /authors.html 23 | 24 | 25 | /index.html 26 | 27 | 28 | /news/index.html 29 | 30 | 31 | /reference/eia_backfill.html 32 | 33 | 34 | /reference/eia_get.html 35 | 36 | 37 | /reference/eia_metadata.html 38 | 39 | 40 | /reference/index.html 41 | 42 | 43 | -------------------------------------------------------------------------------- /man/eia_backfill.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/backfile.R 3 | \name{eia_backfill} 4 | \alias{eia_backfill} 5 | \title{Pull a Large Number of Observations with a Sequential Query} 6 | \usage{ 7 | eia_backfill(start, end, offset, api_key, api_path, facets) 8 | } 9 | \arguments{ 10 | \item{start}{defines the start time of the series, should use a POSIXt class 11 | for hourly series or Date format for non-hourly series (daily, monthly, etc.)} 12 | 13 | \item{end}{defines the end time of the series, should use a POSIXt class 14 | for hourly series or Date format for non-hourly series (daily, monthly, etc.)} 15 | 16 | \item{offset}{An integer, defines the number of observations limitation per query. 17 | In line with the API limitation of up to 5000 observations per query, the offset argument's upper limit is 5000 observations.} 18 | 19 | \item{api_key}{A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service} 20 | 21 | \item{api_path}{A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 22 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/} 23 | 24 | \item{facets}{A list, optional, set the filtering argument (defined as 'facets' 25 | on the API header), following the structure of list(facet_name_1 = value_1, 26 | facet_name_2 = value_2)} 27 | } 28 | \value{ 29 | A time series 30 | } 31 | \description{ 32 | This function allows users to overcome the API's observation limit 33 | per query by breaking down the query into smaller sequential sub-queries and 34 | appending back the results. The main use case of this function is for backfilling 35 | hourly series. 36 | } 37 | \details{ 38 | The function use start, end, and offset arguments to define a sequence 39 | of queries. 40 | } 41 | \examples{ 42 | \dontrun{ 43 | start <- as.POSIXlt("2018-06-19T00", tz = "UTC") 44 | end <- lubridate::floor_date(Sys.time()- lubridate::days(2), unit = "day") 45 | attr(end, "tzone") <- "UTC" 46 | offset <- 2000 47 | api_key <- Sys.getenv("eia_key") 48 | api_path <- "electricity/rto/region-sub-ba-data/data/" 49 | 50 | facets = list(parent = "NYIS", 51 | subba = "ZONA") 52 | 53 | df <- eia_backfill(start = start, 54 | end = end, 55 | offset = offset, 56 | api_key = api_key, 57 | api_path = api_path, 58 | facets = facets) 59 | 60 | at_y <- pretty(df$value)[c(2, 4, 6)] 61 | at_x <- seq.POSIXt(from = start, 62 | to = end, 63 | by = "2 years") 64 | plot(df$time, df$value, 65 | col = "#1f77b4", 66 | type = "l", 67 | frame.plot = FALSE, 68 | axes = FALSE, 69 | panel.first = abline(h = at_y, col = "grey80"), 70 | main = "NY Independent System Operator (West) - Hourly Generation of Electricity", 71 | xlab = "Source: https://www.eia.gov/", 72 | ylab = "MegaWatt/Hours") 73 | 74 | mtext(side =1, text = format(at_x, format = "\%Y"), at = at_x, 75 | col = "grey20", line = 1, cex = 0.8) 76 | 77 | mtext(side =2, text = format(at_y, scientific = FALSE), at = at_y, 78 | col = "grey20", line = 1, cex = 0.8) 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /man/eia_get.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/query.R 3 | \name{eia_get} 4 | \alias{eia_get} 5 | \title{Query the EIA API} 6 | \usage{ 7 | eia_get( 8 | api_key, 9 | api_path, 10 | data = "value", 11 | facets = NULL, 12 | start = NULL, 13 | end = NULL, 14 | length = NULL, 15 | offset = NULL, 16 | frequency = NULL, 17 | format = "data.frame" 18 | ) 19 | } 20 | \arguments{ 21 | \item{api_key}{A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service} 22 | 23 | \item{api_path}{A string, the API path to follow the API endpoint https://api.eia.gov/v2/. 24 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/} 25 | 26 | \item{data}{A string, the metric type, by default uses 'value' (defined as 27 | 'data' on the API header)} 28 | 29 | \item{facets}{A list, optional, set the filtering argument (defined as 'facets' 30 | on the API header), following the structure of list(facet_name_1 = value_1, 31 | facet_name_2 = value_2)} 32 | 33 | \item{start}{A string, optional, set the starting date or time of the series 34 | using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series} 35 | 36 | \item{end}{A string, optional, set the ending date or time of the series 37 | using "YYYY-MM-DD" format for date and "YYYY-MM-DDTHH" format for hourly time series} 38 | 39 | \item{length}{An integer, optional, defines the length of the series, if set to 40 | NULL (default), will default to the API default value of 5000 observations per 41 | pull. The API enables a pull of up to 100K observations per call. If needed to 42 | pull more than the API limit per call, recommend to iterate the call with 43 | the use of the start, end and/or offset arguments} 44 | 45 | \item{offset}{An integer, optional, set the number of observations to offset 46 | from the default starting point of the series. If set to NULL (default), will default 47 | to the API default value of 0} 48 | 49 | \item{frequency}{A string, optional, define the API frequency argument 50 | (e.g., hourly, monthly, annual, etc.). If set to NULL (default), will default 51 | to the API default value} 52 | 53 | \item{format}{A string, defines the output of the return object to either 54 | "data.frame" (default) or "data.table"} 55 | } 56 | \value{ 57 | data.table/data.frame object 58 | } 59 | \description{ 60 | Function to query and extract data from the EIA API v2 61 | } 62 | \examples{ 63 | \dontrun{ 64 | # Required an EIA API key to send a query 65 | api_key <- "YOUR_API_KEY" 66 | 67 | df <- eia_get( 68 | api_key = api_key, 69 | api_path = "electricity/rto/fuel-type-data/data/", 70 | data = "value" 71 | ) 72 | } 73 | } 74 | -------------------------------------------------------------------------------- /man/eia_metadata.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/metadata.R 3 | \name{eia_metadata} 4 | \alias{eia_metadata} 5 | \title{Pull Metadata from EIA API} 6 | \usage{ 7 | eia_metadata(api_path = NULL, api_key) 8 | } 9 | \arguments{ 10 | \item{api_path}{A string, the API category/route path following the API endpoint (i.e., 'https://api.eia.gov/v2/') 11 | If set to NULL (default) or as empty string "" it returns the main categories available on the API. 12 | The path can be found on the EIA API dashboard, for more details see https://www.eia.gov/opendata/browser/} 13 | 14 | \item{api_key}{A string, EIA API key, see https://www.eia.gov/opendata/ for registration to the API service} 15 | } 16 | \value{ 17 | a list object with the series description and metadata 18 | } 19 | \description{ 20 | Get data descriptions and metadata from the EIA API 21 | } 22 | \details{ 23 | The function enables to explore the different data categories and available routes 24 | inline with the API dashboard (https://www.eia.gov/opendata/browser/) 25 | } 26 | \examples{ 27 | \dontrun{ 28 | electricity_metadata <- eia_metadata(api_key = Sys.getenv("eia_key"), 29 | api_path = "electricity") 30 | 31 | electricity_metadata$response$description 32 | electricity_metadata$response$id 33 | electricity_metadata$response$name 34 | electricity_metadata$response$routes 35 | 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /man/figures/EIA_API_browser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/man/figures/EIA_API_browser.png -------------------------------------------------------------------------------- /man/figures/README-unnamed-chunk-6-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/man/figures/README-unnamed-chunk-6-1.png -------------------------------------------------------------------------------- /vignettes/.gitignore: -------------------------------------------------------------------------------- 1 | *.html 2 | *.R 3 | -------------------------------------------------------------------------------- /vignettes/backfile.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Working with Large Dataset" 3 | output: rmarkdown::html_vignette 4 | vignette: > 5 | %\VignetteIndexEntry{backfile} 6 | %\VignetteEngine{knitr::rmarkdown} 7 | %\VignetteEncoding{UTF-8} 8 | --- 9 | 10 | ```{r, include = FALSE} 11 | knitr::opts_chunk$set( 12 | collapse = TRUE, 13 | comment = "#>" 14 | ) 15 | ``` 16 | 17 | One of the limitations of the EIA API is the 5000 observations limit per call. This could be challenging if you are trying to pull hourly time series, which is roughly 26280 observations per year. The `eia_backfill` function solves this issue and removes the API number of observations per call limit. On the backend, the function splits the query into multiple sequential queries, pulls the data, and returns an append object. 18 | 19 | For example, let's pull hourly generation of electricity by New York Independent System Operator on sub-region West using the `eia_backfill` function: 20 | 21 | 22 | ```{r setup} 23 | library(EIAapi) 24 | 25 | start <- as.POSIXlt("2018-06-19T00", tz = "UTC") 26 | end <- lubridate::floor_date(Sys.time()- lubridate::days(2), unit = "day") 27 | attr(end, "tzone") <- "UTC" 28 | offset <- 5000 29 | api_key <- Sys.getenv("eia_key") 30 | api_path <- "electricity/rto/region-sub-ba-data/data/" 31 | 32 | facets = list(parent = "NYIS", 33 | subba = "ZONA") 34 | 35 | df <- eia_backfill(start = start, 36 | end = end, 37 | offset = offset, 38 | api_key = api_key, 39 | api_path = api_path, 40 | facets = facets) 41 | ``` 42 | 43 | As you can see below, the return series has more than 45,000 observations: 44 | 45 | ```{r fig.height=5, fig.width=8} 46 | head(df) 47 | 48 | nrow(df) 49 | 50 | at_y <- pretty(df$value)[c(2, 4, 6)] 51 | at_x <- seq.POSIXt(from = start, 52 | to = end, 53 | by = "2 years") 54 | plot(df$time, df$value, 55 | col = "#1f77b4", 56 | type = "l", 57 | frame.plot = FALSE, 58 | axes = FALSE, 59 | panel.first = abline(h = at_y, col = "grey80"), 60 | main = "NY Independent System Operator (West) - Hourly Generation of Electricity", 61 | xlab = "Source: https://www.eia.gov/", 62 | ylab = "MegaWatt/Hours") 63 | 64 | mtext(side =1, text = format(at_x, format = "%Y"), at = at_x, 65 | col = "grey20", line = 1, cex = 0.8) 66 | 67 | mtext(side =2, text = format(at_y, scientific = FALSE), at = at_y, 68 | col = "grey20", line = 1, cex = 0.8) 69 | ``` 70 | 71 | 72 | -------------------------------------------------------------------------------- /vignettes/images/dashboard1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/vignettes/images/dashboard1.png -------------------------------------------------------------------------------- /vignettes/images/dashboard2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/vignettes/images/dashboard2.png -------------------------------------------------------------------------------- /vignettes/images/register.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RamiKrispin/EIAapi/bafd9125a1a9b2a32bd43feaa6a26b4901c8e905/vignettes/images/register.png -------------------------------------------------------------------------------- /vignettes/intro.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Introduction to the EIA API" 3 | output: rmarkdown::html_vignette 4 | vignette: > 5 | %\VignetteIndexEntry{introo} 6 | %\VignetteEngine{knitr::rmarkdown} 7 | %\VignetteEncoding{UTF-8} 8 | --- 9 | 10 | ```{r, include = FALSE} 11 | knitr::opts_chunk$set( 12 | collapse = TRUE, 13 | comment = "#>" 14 | ) 15 | ``` 16 | 17 | 18 | The [U.S. Energy Information Administration](https://www.eia.gov/) (EIA) hosts a variety of time series data describing the U.S. energy sector. The EIA data is open and accessible through an Application Programming Interface (API) for free. The **EIAapi** package provides functions to query and pull tidy data from the [EIA API 19 | v2](https://www.eia.gov/opendata/). 20 | 21 | 22 | ## Prerequisites 23 | 24 | To pull data from the API using this package, you will need the following: 25 | 26 | - **jq** - The package uses [jq](https://stedolan.github.io/jq/) to parse the API output from JSON to tabular format. To download and install jq follow the 27 | instructions on the [download 28 | page](https://stedolan.github.io/jq/download/). 29 | - **API key** - To query the EIA API, you must register to the 30 | service to receive the API key. 31 | 32 | 33 | To register to the API, go to https://www.eia.gov/opendata/, click the `Register` button, and follow the instructions (see the screenshot below). 34 | 35 |
    36 | 37 | 38 | 39 | 40 | 41 | ## Getting started with the EIAapi 42 | 43 | The EIAapi package provides the following functions: 44 | 45 | - `eia_metadata` - Returns information and metadata from the API about the available categories and subcategories (routes) 46 | - `eia_get` - Enables query data from the API 47 | 48 | A good place to start exploring the available categories and subcategories of the API is on the [API Dashboard](https://www.eia.gov/opendata/browser/). For example, let's explore the hourly demand for electricity in the California sub-region in the dashboard and use the metadata to set the query parameters. Let's start by set the `API Route` to `Electricity` -> `Electric Power Operrations (Daily And Hourly)` -> `Hourly Demand Subregion`, as shown in the screenshot below. Once the data route is set, define the `PARENT` facet as `CISO`. This will filter the options in the next facet - `SUBBA` to the four operators in California. Let's select `CISO: (PGAE) Pacific Gas and Electric` and save the selection: 49 | 50 | 51 |
    52 | 53 | 54 | 55 |
    56 | 57 | Once finalize the routes,facets, and any other filters, you can submit the query and you should receive the API metadata. For the selection above, here is the expected respond: 58 | 59 |
    60 | 61 | 62 | 63 |
    64 | 65 | Let's look at the returned metadata above and use it to set the query to pull the data, starting with the `API URL`: 66 | 67 | ``` HTML 68 | https://api.eia.gov/v2/electricity/rto/region-sub-ba-data/data/?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000 69 | ``` 70 | The URL contains the following components: 71 | 72 | - The API endpoint - `https://api.eia.gov/v2/` 73 | - The data path - `rto/region-sub-ba-data/data/` 74 | - The query - `?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000` 75 | 76 | We will use the data path to set the `api_path`. To set the query parameters, you can use the embedded query in the URL by extracting the values of the different filters (frequency, facet, etc.) or use the query header on the right side: 77 | 78 | ``` json 79 | 80 | X-Params: { 81 | "frequency": "hourly", 82 | "data": [ 83 | "value" 84 | ], 85 | "facets": { 86 | "parent": [ 87 | "CISO" 88 | ], 89 | "subba": [ 90 | "PGAE" 91 | ] 92 | }, 93 | "start": null, 94 | "end": null, 95 | "sort": [ 96 | { 97 | "column": "period", 98 | "direction": "desc" 99 | } 100 | ], 101 | "offset": 0, 102 | "length": 5000 103 | } 104 | ``` 105 | 106 | We can now set the query by defining the `frequency`, `data`, and `facets` arguments as defined in the header above: 107 | 108 | 109 | ```{r setup} 110 | library(EIAapi) 111 | 112 | df1 <- eia_get(api_key = Sys.getenv("eia_key"), 113 | api_path = "electricity/rto/region-sub-ba-data/data/", 114 | frequency = "hourly", 115 | data = "value", 116 | facets = list(parent = "CISO", 117 | subba = "PGAE"), 118 | offset = 0, 119 | length = 5000) 120 | 121 | head(df1) 122 | 123 | unique(df1$parent) 124 | unique(df1$subba) 125 | ``` 126 | 127 | 128 | **Note:** The API limit the number of observations per a call to 5000. If you wish to pull more than 5000 observations you can iterate your call and set the `start` and `end` arguments as a rolling window function. Similarly, you can use the `length` and `offset` arguments to define the rolling function. 129 | 130 | 131 | ## API metadata 132 | 133 | The EIA API provides detailed information and metadata about the categories and time series available in the API. This enables extracting information programmatically about different data parameters. The eia_metadata enables query metadata from the API. The function has two arguments: 134 | 135 | - `api_path` - the API category/route path following the API endpoint (i.e., `https://api.eia.gov/v2/`) 136 | - `api_key` - the API key 137 | 138 | ### Category metadata 139 | 140 | Querying metadata is done by sending a request with a specific route path, and the API, in response, returns the available sub-categories under that path. For example, setting the `api_path` argument empty or `NULL` returns a list with the API main categories: 141 | 142 | 143 | ```{r} 144 | main_route <- eia_metadata(api_path = "", 145 | api_key = Sys.getenv("eia_key")) 146 | 147 | main_route$routes[, c("id", "name")] 148 | ``` 149 | 150 | 151 | Similarly, we can continue and extract the sub-categories under the electricity category by setting the `api_path` argument to `electricity`: 152 | 153 | ```{r} 154 | main_route <- eia_metadata(api_path = "electricity", 155 | api_key = Sys.getenv("eia_key")) 156 | 157 | main_route$routes[, c("id", "name")] 158 | ``` 159 | 160 | **Note:** the number of sub-categories or routes varies between the different categories. 161 | 162 | ### Series metadata 163 | 164 | The last route in the API path defines the series available under those categories. For example, we used the above to pull the hourly demand for electricity by sub-region the following URL: 165 | ``` html 166 | https://api.eia.gov/v2/electricity/rto/region-sub-ba-data/data/?frequency=hourly&data[0]=value&facets[parent][]=CISO&facets[subba][]=PGAE&sort[0][column]=period&sort[0][direction]=desc&offset=0&length=5000 167 | ``` 168 | 169 | In this case, the path is defined as - `electricity/rto/region-sub-ba-data/`, where: 170 | 171 | - `electricity` is the main category 172 | - `rto` - is the sub-category under electricity representing the Electric Power Operations (Daily and Hourly), and 173 | - `region-sub-ba-data` - is the last sub-category in this route, representing the Hourly Demand by Sub-region 174 | 175 | You can notice that the last sub-category in this path - `region-sub-ba-data` ended with `data` to indicate to the user that this is the last route, and under it there is data. The metadata for the time series has a different structure with respect to the category metadata. It provides useful information such as: 176 | - Starting and ending time of the series 177 | - Available facets 178 | - Series Frequency 179 | 180 | For example, let's query for Demand by Sub-region metadata: 181 | 182 | 183 | ```{r} 184 | elec_sub_route <- eia_metadata(api_path = "electricity/rto/region-sub-ba-data/", 185 | api_key = Sys.getenv("eia_key")) 186 | 187 | elec_sub_route 188 | ``` 189 | 190 | You can continue and query additional information on this series. For example, let's extract the list of balancing authorities that are available for this series under the `parent` facet: 191 | 192 | ```{r} 193 | eia_metadata(api_path = "electricity/rto/region-sub-ba-data/facet/parent", 194 | api_key = Sys.getenv("eia_key")) 195 | ``` 196 | 197 | --------------------------------------------------------------------------------