TBD. This will contain examples of advanced usage options once the basic documentation is completed and stable.
40 | 41 | 42 | 43 | 44 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /inst/doc/bootstrap_devel_environment.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 |First ensure that you have cloned the repository into working version only your local machine, choosing one of:
78 |cd /dev
79 | git clone git@github.com:convergenceda/visdom.git
80 | git clone https://github.com/convergenceda/visdom.gitThen using that location as as your working directory (here we assume /dev/visdom), load requirements for package development and install from source.
setwd('/dev/visdom')
85 |
86 | install.packages(c("devtools", "roxygen2", "testthat", "knitr"))version check
88 |rstudioapi::isAvailable("0.99.149")for some reason, withr (a devtools dependency) will not install for RRO 3.2.2 with message that there is no 3.2.2 version of withr this is probably an ephemeral issue, but install from source for now
90 |devtools::install_github("jimhester/withr")install Wickham’s latest/src devtool support. Note that devtools can’t install itself on Windows (!!) https://github.com/hadley/devtools/issues/503 explains.
92 |devtools::install_github("hadley/devtools")Ensure that your getwd() is the visdom source dir and then call:
97 |devtools::document()
98 | devtools::build_vignettes()
99 | ?visdom
100 | ?MeterDataClass
101 | ?WeatherDataClassFollowed by installing the package from source, which should resolve and install all the package dependencies, with some exceptions, like for HDF5 support.
103 |devtools::install(build_vignettes = T)Test your install with a fresh R session:
105 |library(visdom)
106 | ?visdom
107 | browseVignettes(package='visdom')Documentation on loading MeterDataClass objects and exploring their data, functions and built-in features.
40 | 41 | 42 | 43 | 44 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /inst/doc/example_iterator_usage.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 |Examples of iterator function usage TBD. Please see vignette on feature extraction for a working example.
40 | 41 | 42 | 43 | 44 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /inst/doc/weather_data_objects.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 |Examples illustrating how to gather and work with weather data.
40 | 41 | 42 | 43 | 44 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /inst/feature_set_run_conf/example_feature_set.conf: -------------------------------------------------------------------------------- 1 | feature_set=example_feature_set 2 | feature_set_description="Features described in the example_feature_extraction vignette." 3 | run_name=test_run 4 | run_description="Just testing things out." 5 | -------------------------------------------------------------------------------- /inst/sql/db_connect_example.conf: -------------------------------------------------------------------------------- 1 | dbType=MySQL 2 | user=USERNAME 3 | password=PASSWORD 4 | dbname=visdom 5 | -------------------------------------------------------------------------------- /inst/sql/feature_runs.create.mysql.sql: -------------------------------------------------------------------------------- 1 | -- The SQL standard does not support several fundamental features, so table 2 | -- definitions need to be dialect-specific. This is the MySQL version of a table 3 | -- definition for feature run metadata. 4 | CREATE TABLE feature_runs ( 5 | -- AUTO_INCREMENT isn't part of the SQL standard. 6 | -- http://stackoverflow.com/questions/5823912/considering-the-different-implimentations-of-sequence-auto-increment-how-can-i 7 | -- mysql needs AUTO_INCREMENT 8 | -- psql needs the SERIAL datatype 9 | -- SQL Server needs IDENTITY(1,1) isntead of AUTO_INCREMENT 10 | -- Access needs AUTOINCREMENT 11 | -- Oracle needs a separate create sequence statement and a trigger: http://stackoverflow.com/questions/11296361/how-to-create-id-with-auto-increment-on-oracle 12 | id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, 13 | feature_set VARCHAR(20) NOT NULL UNIQUE, 14 | feature_set_description VARCHAR(250) NULL, 15 | run_name VARCHAR(40) NOT NULL UNIQUE, 16 | run_description VARCHAR(250) NULL, 17 | -- This only works with MySQL version 5.6.5 and above. See http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html 18 | create_time DATETIME NOT NULL, 19 | update_time DATETIME DEFAULT NULL 20 | COMMENT 'To be updated programmatically when features are written to the feature table.)', -- limit 255 chars 21 | CONSTRAINT feature_run UNIQUE (feature_set, run_name) 22 | ) 23 | COMMENT 'Tracking metadata for feature sets and runs.' -- limit 60 chars 24 | ; 25 | -------------------------------------------------------------------------------- /install/generic_create.sql: -------------------------------------------------------------------------------- 1 | /* 2 | SQLyog Community v12.2.4 (64 bit) 3 | MySQL - 10.1.13-MariaDB : Database - pgeres 4 | ********************************************************************* 5 | */ 6 | 7 | /* NOTE: this is obviously a create script for MYSQL/MariaDB, but any other SQL database can work just as well. 8 | The important thing is to ensure that R can connect to your preferred DB. 9 | See also the import script for more information on how the databse can be populated and how indices can be 10 | added to exsure adequate performance. 11 | */ 12 | 13 | /*!40101 SET NAMES utf8 */; 14 | 15 | /*!40101 SET SQL_MODE=''*/; 16 | 17 | /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; 18 | /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; 19 | /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; 20 | /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; 21 | CREATE DATABASE /*!32312 IF NOT EXISTS*/`visdom_data` /*!40100 DEFAULT CHARACTER SET latin1 */; 22 | 23 | USE `visdom_data`; 24 | 25 | /*Table structure for table `account` */ 26 | /* all customer acocunt associated data should go here 27 | add or subtract columns to reflect what you know about customers 28 | the main idea is that account data can be added directly to customer features based on meter or account id 29 | so if you want a feature that reflects account information you already have, put it in this table. 30 | */ 31 | 32 | DROP TABLE IF EXISTS `account`; 33 | 34 | CREATE TABLE `account` ( 35 | `id` int(11) NOT NULL AUTO_INCREMENT, 36 | `ACCOUNT_UUID` varchar(11) DEFAULT NULL, 37 | `METER_UUID` varchar(11) DEFAULT NULL, 38 | `ACCOUNT_START_DATE` datetime NOT NULL, # start of association of account_id with meter_id 39 | `ACCOUNT_END_DATE` datetime NOT NULL, # end of association of account_id with meter_id 40 | `zip5` varchar(5) DEFAULT NULL, 41 | `RATE_PLAN` varchar(6) DEFAULT NULL, 42 | `CLIMATE_ZONE` varchar(4) DEFAULT NULL, 43 | `PV_INDICATOR` char(1) DEFAULT NULL, 44 | # you might also have demographics, census data, site physical characteristics, income, etc. here 45 | PRIMARY KEY (`id`), 46 | KEY `zip5_meter_uuid_idx` (`METER_UUID`,`zip5`), 47 | KEY `account_uuid_idx` (`ACCOUNT_UUID`), 48 | KEY `meter_uuid_idx` (`METER_UUID`) 49 | ) ENGINE=InnoDB AUTO_INCREMENT=327676 DEFAULT CHARSET=latin1; 50 | 51 | 52 | /*Table structure for table `intervention` */ 53 | /* intervention data not required to run visdom, but useful for before/after studies 54 | add or subtract intervention data columns to match available intervention data */ 55 | DROP TABLE IF EXISTS `intervention`; 56 | 57 | CREATE TABLE `intervention` ( 58 | `id` int(11) NOT NULL AUTO_INCREMENT, 59 | `account_uuid` varchar(11) DEFAULT NULL, 60 | `meter_uuid` varchar(11) DEFAULT NULL, 61 | `INSTALL_DATE` date DEFAULT NULL, 62 | `MEASURE_TYPE` varchar(10) DEFAULT NULL, 63 | `MEASURE_DESC` varchar(80) DEFAULT NULL, 64 | `TECHNOLOGY_TYPE` varchar(30) DEFAULT NULL, 65 | `EE_PROGRAM_NAME` varchar(10) DEFAULT NULL, 66 | `EST_ANNUAL_KW_SAVINGS` float DEFAULT NULL, 67 | `EST_ANNUAL_KWH_SAVINGS` float DEFAULT NULL, 68 | `EST_ANNUAL_THM_SAVINGS` float DEFAULT NULL, 69 | `INCENTIVE_PAYMENT` float DEFAULT NULL, 70 | `TOTAL_PROJECT_COSTS` float DEFAULT NULL, 71 | PRIMARY KEY (`id`), 72 | KEY `account_uuid_idx` (`account_uuid`), 73 | KEY `meter_uuid_idx` (`meter_uuid`), 74 | KEY `install_idx` (`INSTALL_DATE`) 75 | ) ENGINE=MyISAM AUTO_INCREMENT=13477 DEFAULT CHARSET=latin1; 76 | 77 | 78 | 79 | /*Table structure for table `local_weather` */ 80 | /* this table matches the format of weather data provided by the NOAA data scraping GitHub project local-weather 81 | which keys off of zip code: http://github.com/sborgeson/local-weather 82 | */ 83 | 84 | DROP TABLE IF EXISTS `local_weather`; 85 | 86 | CREATE TABLE `local_weather` ( 87 | `id` int(6) unsigned NOT NULL AUTO_INCREMENT, 88 | `zip5` int(11) NOT NULL, 89 | `date` datetime NOT NULL, 90 | `TemperatureF` float(5,2) DEFAULT NULL, 91 | `DewpointF` float(5,2) DEFAULT NULL, 92 | `Pressure` float(5,2) DEFAULT NULL, 93 | `WindSpeed` float(5,2) DEFAULT NULL, 94 | `Humidity` float(5,2) DEFAULT NULL, 95 | `Clouds` varchar(10) DEFAULT NULL, 96 | `HourlyPrecip` float(5,2) DEFAULT NULL, 97 | `SolarRadiation` float(6,2) DEFAULT NULL, 98 | PRIMARY KEY (`id`), 99 | UNIQUE KEY `zip_date_idx` (`zip5`,`date`), 100 | KEY `zip_idx` (`zip5`), 101 | KEY `date_idx` (`date`) 102 | ) ENGINE=MyISAM AUTO_INCREMENT=1637609 DEFAULT CHARSET=latin1; 103 | 104 | /*Table structure for table `meter_data` */ 105 | /* hourly meter data uses 24 columns per row 15 minute interval data can use 96 cols in the same manner 106 | you can build a data source to pull data for each meter (i.e. all data from a given meter) or account 107 | i.e. for the period of time an individual account holder was behind a given meter. If you get the account_uuid 108 | correct over time, it is very simple for a data source to key by account rather than meter ids. 109 | */ 110 | 111 | DROP TABLE IF EXISTS `meter_data`; 112 | 113 | CREATE TABLE `meter_data` ( 114 | `meter_uuid` varchar(11) NOT NULL, 115 | `account_uuid` varchar(11) NOT NULL, 116 | `date` date NOT NULL, 117 | `zip5` varchar(5) DEFAULT NULL, 118 | `h1` int(11) DEFAULT NULL, # note that these are ints because they are cheaper to store than floats 119 | `h2` int(11) DEFAULT NULL, # and the assimption is that you can store Wh per interval period 120 | `h3` int(11) DEFAULT NULL, # converting to kWh/period on the way out of the db 121 | `h4` int(11) DEFAULT NULL, # thsi table can become HUGE, so optimizations like this are a good idea. 122 | `h5` int(11) DEFAULT NULL, 123 | `h6` int(11) DEFAULT NULL, 124 | `h7` int(11) DEFAULT NULL, 125 | `h8` int(11) DEFAULT NULL, 126 | `h9` int(11) DEFAULT NULL, 127 | `h10` int(11) DEFAULT NULL, 128 | `h11` int(11) DEFAULT NULL, 129 | `h12` int(11) DEFAULT NULL, 130 | `h13` int(11) DEFAULT NULL, 131 | `h14` int(11) DEFAULT NULL, 132 | `h15` int(11) DEFAULT NULL, 133 | `h16` int(11) DEFAULT NULL, 134 | `h17` int(11) DEFAULT NULL, 135 | `h18` int(11) DEFAULT NULL, 136 | `h19` int(11) DEFAULT NULL, 137 | `h20` int(11) DEFAULT NULL, 138 | `h21` int(11) DEFAULT NULL, 139 | `h22` int(11) DEFAULT NULL, 140 | `h23` int(11) DEFAULT NULL, 141 | `h24` int(11) DEFAULT NULL, 142 | PRIMARY KEY (`meter_uuid`,`date`), 143 | KEY `meter_uuid_idx` (`meter_uuid`), 144 | KEY `account_uuid_idx` (`account_uuid`), 145 | KEY `zip_Date_idx` (`date`,`zip5`), 146 | KEY `zip_idx` (`zip5`) 147 | ) ENGINE=InnoDB DEFAULT CHARSET=latin1; 148 | 149 | 150 | -------------------------------------------------------------------------------- /install/generic_insert.sql: -------------------------------------------------------------------------------- 1 | 2 | # under some configurations, this file has to be in the main MySQL data directory 3 | # ou can try to absolutely path to it, but if it doesn't work, see the MySQL docs on LOAD DATA 4 | LOAD DATA LOCAL INFILE 'intervention_data.csv' 5 | IGNORE # ignore duplicates of the primary key 6 | INTO TABLE intervention 7 | FIELDS TERMINATED BY ',' 8 | lines terminated by '\r\n' 9 | IGNORE 1 LINES 10 | ( ACCOUNT_UUID, 11 | METER_UUID, 12 | @INSTALL_DATE_v, # load date as a string variable so it can be parsed with a date format into a date 13 | MEASURE_TYPE, 14 | MEASURE_DESC, 15 | TECHNOLOGY_TYPE, 16 | EE_PROGRAM_NAME, 17 | @EST_ANNUAL_KW_SAVINGS_v, # load numbers as strings to handle nulls. 18 | @EST_ANNUAL_KWH_SAVINGS_v, 19 | @EST_ANNUAL_THM_SAVINGS_v, 20 | @INCENTIVE_PAYMENT_v, 21 | @TOTAL_PROJECT_COSTS_v, 22 | ) 23 | SET INSTALL_DATE = STR_TO_DATE( SUBSTRING_INDEX(@INSTALL_DATE_v, ' ', 1), '%m/%e/%Y'), # convert dates 24 | EST_ANNUAL_KW_SAVINGS = nullif(@EST_ANNUAL_KW_SAVINGS_v,''), # handle blanks as nulls 25 | EST_ANNUAL_KWH_SAVINGS = nullif(@EST_ANNUAL_KWH_SAVINGS_v,''), 26 | EST_ANNUAL_THM_SAVINGS = nullif(@EST_ANNUAL_THM_SAVINGS_v,''), 27 | INCENTIVE_PAYMENT = nullif(@INCENTIVE_PAYMENT_v,''), 28 | TOTAL_PROJECT_COSTS = nullif(@TOTAL_PROJECT_COSTS_v,''), 29 | ID = NULL # auto-increment 30 | ; 31 | 32 | 33 | LOAD DATA LOCAL INFILE 'account_data.csv' # this file has to be in the main data directory 34 | IGNORE # ignore duplicates of the primary key 35 | INTO TABLE account 36 | FIELDS TERMINATED BY ',' 37 | LINES TERMINATED BY '\r\n' 38 | IGNORE 1 LINES 39 | ( account_uuid, 40 | meter_uuid, 41 | @account_start_date_v, 42 | @account_end_date_v, 43 | zip5, 44 | rate_plan, 45 | climate_zone, 46 | @pv_indicator_v 47 | ) 48 | SET pv_indicator = NULLIF(@pv_indicator_v,''), 49 | account_start_date = STR_TO_DATE( SUBSTRING_INDEX(@account_start_date_v, ' ', 1), '%m/%e/%Y'), # convert dates 50 | account_end_date = STR_TO_DATE( SUBSTRING_INDEX(@account_end_date_v, ' ', 1), '%m/%e/%Y'), # convert dates 51 | ID = NULL 52 | ; 53 | 54 | 55 | /* assuming fields are meter_id, date, and 24 x interval readings per row */ 56 | load data local infile 'interval_data.txt' 57 | into TABLE meter_data 58 | FIELDS TERMINATED BY ',' 59 | IGNORE 1 LINES # ignore headers 60 | ( meter_uuid, 61 | # note here we assume the account id was not included with the meter data, but can be added below 62 | @usage_date, # one row per meter day 63 | @h1, @h2, @h3, @h4, @h5, @h6, 64 | @h7, @h8, @h9, @h10, @h11, @h12, 65 | @h13, @h14, @h15, @h16, @h17, @h18, 66 | @h19, @h20, @h21, @h22, @h23, @h24 67 | ) 68 | set date = STR_TO_DATE( SUBSTRING_INDEX(@usage_date, ' ', 1), '%m/%e/%Y'), 69 | h1 = round(@h1 * 1000), # add as ints in units of Wh/period to save disk space 70 | h2 = round(@h2 * 1000), # assuming raw data is kW/hour, we simply multiply by 1000 71 | h3 = round(@h3 * 1000), # and round into ints. 72 | h4 = round(@h4 * 1000), 73 | h5 = round(@h5 * 1000), 74 | h6 = round(@h6 * 1000), 75 | h7 = round(@h7 * 1000), 76 | h8 = round(@h8 * 1000), 77 | h9 = round(@h9 * 1000), 78 | h10 = round(@h10 * 1000), 79 | h11 = round(@h11 * 1000), 80 | h12 = round(@h12 * 1000), 81 | h13 = round(@h13 * 1000), 82 | h14 = round(@h14 * 1000), 83 | h15 = round(@h15 * 1000), 84 | h16 = round(@h16 * 1000), 85 | h17 = round(@h17 * 1000), 86 | h18 = round(@h18 * 1000), 87 | h19 = round(@h19 * 1000), 88 | h20 = round(@h20 * 1000), 89 | h21 = round(@h21 * 1000), 90 | h22 = round(@h22 * 1000), 91 | h23 = round(@h23 * 1000), 92 | h24 = round(@h24 * 1000), 93 | zip5 = NULL 94 | ; 95 | 96 | 97 | # pull zip code data from the account table 98 | UPDATE meter_data 99 | LEFT JOIN account ON meter_data.meter_uuid = account.meter_uuid 100 | SET meter_data.zip5 = account.zip5; 101 | 102 | # assuming meter_data doesnt have account ids, but the account table does, you can add them like this: 103 | UPDATE meter_data AS md 104 | LEFT JOIN account AS a ON md.meter_uuid = a.meter_uuid AND 105 | md.date >= a.ACCOUNT_START_DATE AND 106 | md.date <= a.ACCOUNT_END_DATE 107 | SET md.account_uuid = a.account_uuid; 108 | 109 | # get ready for weather data 110 | # use the output of this as the input into weatherDump.py from http://github.com/sborgeson/local-weather 111 | SELECT zip5, MIN(DATE), MAX(DATE) FROM meter_data GROUP BY zip5; 112 | # save results as zip_dates.csv 113 | # run with 3 stations averaged per zip and a preferred radius of less than 20 km 114 | # python weatherDump.py -i path/to/zip_dates.csv -o path/to/out/dir/weather_data.csv -n 3 -d 20 115 | 116 | 117 | /* load weather data in the format produced by local_weather's dumpWeather.py utility */ 118 | LOAD DATA LOCAL INFILE 'weather_data.csv' 119 | INTO TABLE local_weather 120 | FIELDS TERMINATED BY ',' 121 | LINES TERMINATED BY '\n' 122 | IGNORE 1 LINES 123 | (@dateStr, TemperatureF, DewpointF, Pressure, WindSpeed, Humidity, HourlyPrecip, zip5) 124 | SET `date` = STR_TO_DATE(@dateStr, '%Y-%m-%d %H:%i:%s'); 125 | 126 | -------------------------------------------------------------------------------- /install/generic_visdom_data_source.R: -------------------------------------------------------------------------------- 1 | require(RMySQL) # assuming data is in a MySQL database 2 | library(visdom) # includes generic database support from util-dbUtil.R for things like connection management 3 | 4 | MyDataSource = function(dbConfig='data_db.cfg', queryCache=paste(getwd(),'/','DATA_CACHE','/',sep='')){ 5 | obj = DataSource() 6 | obj$CACHE_DIR = queryCache 7 | obj$DB_CFG = dbCfg(dbConfig) 8 | obj$db = 'visdom_data' 9 | 10 | print(queryCache) 11 | #print(obj$DB_CFG) 12 | 13 | obj$accountTable = 'account' 14 | obj$meterTable = 'meter_data' 15 | obj$weatherTable = 'local_weather' 16 | obj$interventionTable = 'intervention' 17 | 18 | obj$geoColumnName = 'zip5' 19 | 20 | # utility function that returns a list of all the zipcodes in the data set 21 | obj$getGeocodes = function(useCache=F,forceRefresh=F) { 22 | query <- paste("select distinct zip5 from",obj$accountTable,'order by zip5') 23 | cacheFile = NULL 24 | if(useCache) { cacheFile='zipList.RData' } 25 | return(run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh)[[1]]) 26 | } 27 | 28 | obj$getIds = function(geocode=NULL,useCache=F,forceRefresh=F) { 29 | cacheFile = NULL 30 | if(is.null(geocode)) { 31 | if(useCache) { 32 | cacheFile = paste('meterids_all.RData',sep='') # only cache sp list for individual geos 33 | } 34 | # note that this query uses meter ids from the account table. You can also use the ids from the actual meter_table 35 | # if the account table is unreliable, but these queries will take longer. 36 | query <- paste("select distinct meter_uuid as id from",obj$accountTable,'order by id') 37 | } else { 38 | if(useCache) { 39 | cacheFile = paste('meterids_',geocode,'.RData',sep='') # only cache sp list for individual geos 40 | } 41 | query <- paste("select distinct meter_uuid as id from ",obj$account," where zip5='",geocode,"' order by id",sep='') 42 | } 43 | return(run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh,debug=T)[[1]]) 44 | } 45 | 46 | 47 | obj$getAllData = function(geocode,useCache=F,forceRefresh=F) { 48 | cacheFile = NULL 49 | if(useCache) { cacheFile=paste('meterData_',geocode,'.RData',sep='') } 50 | query = paste( 51 | # note that VISDOM expects the unique identifiers to be called 'id' and dates to be called 'dates' 52 | # in the data framse returned from here 53 | 'SELECT 54 | meter_uuid as id, zip5, date as dates, 55 | h1, h2, h3, h4, h5, h6, h7, h8, h9, h10,h11,h12, 56 | h13,h14,h15,h16,h17,h18,h19,h20,h21,h22,h23,h24 57 | FROM ',obj$meterTable, 58 | " where zip5 ='",geocode,"' ", 59 | ' ORDER BY id, dates',sep='') 60 | data = run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh, debug=F) 61 | # convert last 24 columns to kWh from Wh 62 | data[,c(-23:0) + ncol(data)] = data[,c(-23:0) + ncol(data)] / 1000 63 | return(data) 64 | } 65 | 66 | 67 | obj$getAccountData = function(useCache=F, forceRefresh=F) { 68 | cacheFile = NULL 69 | if(useCache) { cacheFile=paste('accountData.RData',sep='') } 70 | # note that with a select * statement, this generic function returns whatever you are able to cram into the acocunt table. 71 | # this allows for generic support for merging in account features with the other features by account id. 72 | # however, note that the column names can't be controlled by * queries, so all, including the ids will have their db names. 73 | query = 'SELECT * from account order by meter_uuid, acocunt_uuid' 74 | data = run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh, debug=F) 75 | return(data) 76 | } 77 | 78 | obj$getInterventionData = function(useCache=F,forceRefresh=F) { 79 | cacheFile = NULL 80 | if(useCache) { cacheFile=paste('interventionData.RData',sep='') } 81 | query = paste( 82 | 'SELECT * FROM ', obj$interventionTable, 83 | ' ORDER BY meter_uuid, account_uuid, install_date',sep='') 84 | data = run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh, debug=F) 85 | return(data) 86 | } 87 | 88 | obj$getMeterData = function(id,geocode=NULL) { 89 | # note that VISDOM expects the unique identifiers to be called 'id' and dates to be called 'dates' 90 | # in the data framse returned from here 91 | query = paste( 92 | 'SELECT 93 | meter_uuid as id, zip5, date as dates, 94 | h1, h2, h3, h4, h5, h6, h7, h8, h9, h10,h11,h12, 95 | h13,h14,h15,h16,h17,h18,h19,h20,h21,h22,h23,h24 96 | FROM ',obj$meterTable, 97 | " where meter_uuid ='",id,"'", 98 | ' ORDER BY dates', sep='') 99 | data = run.query(query, obj$DB_CFG, debug=T) 100 | # convert last 24 columns to kW from W 101 | data[,c(-23:0) + ncol(data)] = data[,c(-23:0) + ncol(data)] / 1000 102 | if( 'date' %in% names(data)) { 103 | names(data)[which(names(data) %in% 'date')] = 'dates' 104 | } 105 | return(data) 106 | } 107 | 108 | obj$getWeatherData = function(geocode,useCache=F,forceRefresh=F) { 109 | cacheFile = NULL 110 | if(useCache) { cacheFile=paste('weather_',geocode,'.RData',sep='') } 111 | query = paste( 112 | 'SELECT `date`, TemperatureF, Pressure, DewpointF, HourlyPrecip, WindSpeed 113 | FROM ',obj$weatherTable," where zip5 ='",geocode,"' ORDER BY DATE",sep='') 114 | data = run.query(query,DATA_SOURCE$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh) 115 | names(data) = tolower(names(data)) 116 | return(data) 117 | } 118 | 119 | obj$getGeoForId = function(id,useCache=F, forceRefresh=F) { 120 | query <- paste("select meter_uuid as id, zip5 from",obj$accountTable,' group by meter_uuid order by meter_uuid') 121 | cacheFile = NULL 122 | if(useCache) { cacheFile='idZipList.RData' } 123 | idZipData = run.query(query,obj$DB_CFG,cacheDir=obj$CACHE_DIR,cacheFile=cacheFile,forceRefresh=forceRefresh) 124 | return( idZipData[idZipData$id == id, 'zip5'] ) 125 | } 126 | 127 | class(obj) = append(class(obj),"MyDataSource") 128 | 129 | return(obj) 130 | } -------------------------------------------------------------------------------- /install/ubuntu_14.04_bash_requirements.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Download and install Microsoft R Open (MRO, formerly RRO): 4 | # See https://mran.microsoft.com/documents/rro/installation/#revorinst-lin 5 | 6 | # remove default version of R (if applicable) 7 | sudo apt-get remove r-base-core 8 | 9 | # get MRO 10 | cd /usr/local/src 11 | sudo wget https://mran.microsoft.com/install/mro/3.3.1/microsoft-r-open-3.3.1.tar.gz 12 | 13 | # extract and install 14 | sudo tar zxvf microsoft-r-open-3.3.1.tar.gz 15 | cd microsoft-r-open/ 16 | sudo ./install.sh 17 | #Choose to install MKL libraries and accept licenses 18 | #For unattended install, ./install.sh -a -s 19 | 20 | # back to /usr/local/src 21 | cd /usr/local/src 22 | 23 | #Install rstudio server for running browser-based R sessions: 24 | sudo apt-get install gdebi-core 25 | sudo wget https://download2.rstudio.org/rstudio-server-0.99.903-amd64.deb 26 | sudo gdebi rstudio-server-0.99.903-amd64.deb 27 | 28 | # knitr requires pandoc from more recent than the Ubuntu 14.04 packages 29 | # if necessary sudo apt-get remove pandoc pandoc-citeproc 30 | sudo wget https://github.com/jgm/pandoc/releases/download/1.17.2/pandoc-1.17.2-1-amd64.deb 31 | sudo gdebi pandoc-1.17.2-1-amd64.deb 32 | 33 | # install lib dependencies for R modules that will be used 34 | # by devtools, vignettes, or visodm 35 | sudo apt-get install zlib1g-dev libssl-dev libcurl4-openssl-dev 36 | sudo apt-get install gfortran 37 | sudo apt-get install libmariadbclient-dev 38 | 39 | 40 | # install devtools if you don't have it. 41 | # monitor this for errors related to missing dependencies 42 | sudo R -e 'install.packages(c("devtools"))' 43 | 44 | # install visdom from GitHub. 45 | # the unzip='internal' option is a fix for Ubuntu's devtools support 46 | # see discussion at https://github.com/RevolutionAnalytics/RRO/issues/37 47 | sudo R -e 'options(unzip = "internal"); devtools::install_github("convergenceda/visdom", build_vignettes=F )' 48 | 49 | # check if it works! 50 | R -e 'library(visdom);?visdom' 51 | 52 | # now get the vignettes up and running 53 | sudo R -e 'install.packages(c("knitr","rmarkdown","DBI"))' 54 | 55 | sudo R -e 'options(unzip = "internal"); devtools::install_github("convergenceda/visdom", build_vignettes=T, force=T)' 56 | 57 | -------------------------------------------------------------------------------- /man/CA_ZIP_CLIMATE.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/dataSetDocs.R 3 | \docType{data} 4 | \name{CA_ZIP_CLIMATE} 5 | \alias{CA_ZIP_CLIMATE} 6 | \title{A zip code to CA climate zone mapping} 7 | \format{A data frame with 1706 rows and 2 variables: 8 | \describe{ 9 | \item{ZIP.Code}{zip code, as a number} 10 | \item{Building.Climate.Zone}{the CA CEC climate zone the zip code is in} 11 | }} 12 | \source{ 13 | \url{http://CEC.web.site.somewhere/} 14 | } 15 | \usage{ 16 | data(CA_ZIP_CLIMATE) 17 | } 18 | \description{ 19 | A dataset containing columns with US zip codes and the corresponding CA CEC climate zones 20 | } 21 | \keyword{datasets} 22 | 23 | -------------------------------------------------------------------------------- /man/CENSUS_GAZ.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/dataSetDocs.R 3 | \docType{data} 4 | \name{CENSUS_GAZ} 5 | \alias{CENSUS_GAZ} 6 | \title{A summary of census statistics for each ZCTA in the census} 7 | \format{A data frame with 33120 rows and 9 variables: 8 | \describe{ 9 | \item{ZCTA}{zip code tabulation area as a 0 padded string} 10 | \item{POP10}{Population in 2010} 11 | \item{ALAND}{Land area in sqft?} 12 | \item{AWATER}{Water area in sqft?} 13 | \item{ALAND_SQMI}{Land area in sqare miles} 14 | \item{AWATER_SQMI}{Water area in square miles} 15 | \item{INTPTLAT}{Latitude of the area} 16 | \item{INTPTLONG}{Longitude of the area} 17 | }} 18 | \source{ 19 | \url{http://us.census.link/} 20 | } 21 | \usage{ 22 | data(CENSUS_GAZ) 23 | } 24 | \description{ 25 | A dataset containing summary census statistics for each ZCTA in the census 26 | } 27 | \keyword{datasets} 28 | 29 | -------------------------------------------------------------------------------- /man/DataSource.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dataSource.R 3 | \name{DataSource} 4 | \alias{DataSource} 5 | \title{Main S3 class that defines the data source interface} 6 | \usage{ 7 | DataSource() 8 | } 9 | \description{ 10 | The functions defined in the DataSource class must be implemented in the custom data source for each 11 | new data set, taking steps to convert data and normalize data access. It is good practice to ensure that 12 | custom data sources "extend" DataSource by using the \code{DataSource()} constructor as the line that creates the 13 | custom DataSource S3 objects, and then overriding functions as required. 14 | } 15 | \details{ 16 | \code{DataSource} Functions as a sort of interface definition and/or default implementation for 17 | all data source functions. It is good practice to have custom data sources extend this one because 18 | it provides default implementations for all functions and a handfull of useful utility functions. 19 | } 20 | \seealso{ 21 | \code{\link{TestData}}, \code{\link{sanityCheckDataSource}} 22 | } 23 | 24 | -------------------------------------------------------------------------------- /man/ERLE_ZIP_LOCATION.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/dataSetDocs.R 3 | \docType{data} 4 | \name{ERLE_ZIP_LOCATION} 5 | \alias{ERLE_ZIP_LOCATION} 6 | \title{Zipcode to location mapping} 7 | \format{A data frame with 43191 rows and 7 variables: 8 | \describe{ 9 | \item{zip}{zip code, as a number} 10 | \item{city}{city the zip code is in} 11 | \item{state}{two letter abbreviation for the state the zip code is in} 12 | \item{latitude}{latitude of the zip code center (post office?)} 13 | \item{longitude}{longitude of the zip code center (post office?)} 14 | \item{timezone}{timesone, as offste relative to GMT time} 15 | \item{dst}{indicator for participation in daylight savings time} 16 | }} 17 | \source{ 18 | \url{http://can.not.remember/} 19 | } 20 | \usage{ 21 | data(ERLE_ZIP_LOCATION) 22 | } 23 | \description{ 24 | A dataset containing columns with US zip codes and the corresponding locaiton information 25 | } 26 | \keyword{datasets} 27 | 28 | -------------------------------------------------------------------------------- /man/MeterDataClass.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/classes-customer.R 3 | \name{MeterDataClass} 4 | \alias{MeterDataClass} 5 | \title{Main S3 class that holds and normalizes VISDOM customer meter data} 6 | \usage{ 7 | MeterDataClass(id, geocode = NULL, weather = NULL, data = NULL, 8 | gasData = NULL, useCache = T, doSG = F, rawData = NULL) 9 | } 10 | \arguments{ 11 | \item{id}{Unique identifier used by data source to identify a customer meter. 12 | Even with numerical id's in a database, it is a good idea to make these 13 | character strings in R.} 14 | 15 | \item{geocode}{Primary geographic locator used by the data source. This is 16 | very often a zip code, but data sources can implement it as census block 17 | or any geographic category that covers all meters, with each meter 18 | associated with only one geocode. If it is not provided, the data source 19 | is used to query for the meter's location.} 20 | 21 | \item{weather}{Reference to instance of \link{WeatherData} object that provides 22 | weather data for the meter's location. The DATA_SOURCE is used to try to 23 | load this information for the relevant geocode if it is not provided.} 24 | 25 | \item{data}{Tabluar format dataframe of electric meter data for the meter identified 26 | by \code{id}, with each row covering all observations for a meter for a day, with 27 | columns including at least: id, date, and hourly or 15 minute readings.} 28 | 29 | \item{gasData}{Tabluar format dataframe of gas meter data, with each row covering all 30 | observations for a meter for a day, with columns including at least: id, date 31 | and daily readings. If this is not provided, the gas meter data is 32 | loaded using the data source if the data source supports gas data. 33 | \code{gasData} is typically passed in as an optimmization 34 | to avoid unnecessary database queries during bulk meter feature extraction.} 35 | 36 | \item{useCache}{Data caching option indicating to the data source whether cached 37 | data should be used to populate the class. Data sources implement their own 38 | caching, but the expectation is that they will hit the underlying data base 39 | once to get the data in the first place and then save it to a local cache from 40 | which it will be retrieved on subsequent calls. See \link{run.query} for details.} 41 | 42 | \item{doSG}{Option indicating whether expensive solar geometry calculations 43 | that rely on the \link{solaR} package should be performed. Off by default for 44 | performance reasons. See \link{solarGeom} for details.} 45 | 46 | \item{rawData}{Tabluar format dataframe of electric meter data, with each row covering all 47 | observations for a meter for a day, with columns including at least: id, date, 48 | and hourly or 15 minute readings. If this is not provided and \code{data} is not provided, 49 | the meter data is loaded using the data source. \code{rawData} is typically passed in as 50 | an optimization to avoid unnecessary database queries during bulk meter feature 51 | extraction.} 52 | } 53 | \description{ 54 | Data structures sand functions used to store, manipulate, and visualize customer 55 | meter data. This class provides structure and standardization to the representation 56 | of meter data in VISDOM, with support for passing in pre-loaded version of data 57 | that is otherwise expensive to load individually. It therefore enables feature 58 | functions, figures, etc. to all be written to draw upon standard structures and 59 | functions and centralizes work on performance. 60 | } 61 | \details{ 62 | MeterDataClass is often called within the iterator framework or using DATA_SOURCE$getMEterDataClass(id) for a data source. 63 | } 64 | \examples{ 65 | \dontrun{ 66 | DATA_SOURCE = TestData(10) 67 | cust = MeterDataClass(id='14') 68 | plot(cust) 69 | } 70 | } 71 | \seealso{ 72 | \code{\link{WeatherClass}} 73 | } 74 | 75 | -------------------------------------------------------------------------------- /man/TestData.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/testDataSource.R 3 | \name{TestData} 4 | \alias{TestData} 5 | \title{Implementation of a data source that generates synthetic data for testing and examples} 6 | \usage{ 7 | TestData(n = 100) 8 | } 9 | \arguments{ 10 | \item{n}{Number of meters for the data source to provide synthetic data for: determines 11 | the return values for functions like \code{DATA_SOURCE$getIds()} and and \code{DATA_SOURCE$getAllData()}} 12 | } 13 | \description{ 14 | Example implementation of the data source functions as a coherent data source. This one 15 | simply generates random synthetic data that conform to the formats required by \code{\link{MeterDataClass}} 16 | and \code{\link{WeatherClass}}, returning valid, but meaningless data for all calls. 17 | } 18 | \details{ 19 | \code{TestData} is instantiated with a certain number of meters to generate data for. It then generates 20 | random data for that number and consumes that data to support the rest of its functions. This class 21 | can be used for examples, learning how data sources work, and providing data for tests of feature algorithms 22 | and meter data related classes. 23 | } 24 | \examples{ 25 | \dontrun{ 26 | DATA_SOURCE = TestData(n=100) 27 | idList = DATA_SOURCE$getIds() 28 | dataMatrix = DATA_SOURCE$getAllData() 29 | } 30 | } 31 | \seealso{ 32 | \code{\link{DataSource}} 33 | } 34 | 35 | -------------------------------------------------------------------------------- /man/WeatherClass.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/classes-weather.R 3 | \name{WeatherClass} 4 | \alias{WeatherClass} 5 | \title{S3 class that holds and normalizes weather data} 6 | \usage{ 7 | WeatherClass(geocode, raw = NULL, doMeans = T, useCache = F, doSG = F) 8 | } 9 | \arguments{ 10 | \item{geocode}{Primary geographic locator used by the data source. This is 11 | very often a zip code, but data sources can implement it as census block 12 | or any geographic category that covers all meters, with time series 13 | observations of weather data associated with a geocode.} 14 | 15 | \item{doMeans}{Indicator of whether to calculate and retain daily and monthly 16 | mean values of observations. Disable when not in use for faster performance.} 17 | 18 | \item{useCache}{Data caching option indicating to the data source whether cached 19 | data should be used to populate the class. Data sources implement their own 20 | caching, but the expectation is that they will hit the underlying data base 21 | once to get the data in the first place and then save it to a local cache from 22 | which it will be retrieved on subsequent calls. See \link{run.query} for details.} 23 | 24 | \item{doSG}{Option indicating whether expensive solar geometry calculations 25 | that rely on the \link{solaR} package should be performed. Off by default for 26 | performance reasons. See \link{solarGeom} for details.} 27 | } 28 | \description{ 29 | Standard format and functions for loading, manipulating, and visualizing weather data in VISDOM. 30 | } 31 | \details{ 32 | \code{WeatherData} is compatible by default with the output (i.e. database table) of 33 | python weather data scraping code that draws upon NOAA's NOAA Quality Controlled 34 | Local Climatological Data (QCLCD) ftp server. See \code{http://github.com/sborgeson/local-weather} 35 | for code to download and convert weathre data into CSV files, the weather data zip files at 36 | \code{http://www.ncdc.noaa.gov/orders/qclcd/}, and the QCLCD homepage here 37 | \code{https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/quality-controlled-local-climatological-data-qclcd} 38 | } 39 | \section{Parameters}{ 40 | 41 | } 42 | \examples{ 43 | \dontrun{ 44 | DATA_SOURCE = TestData(10) 45 | weather = WeatherData(geocode='12601') 46 | plot(weather) 47 | } 48 | } 49 | \seealso{ 50 | \code{\link{MeterDataClass}} 51 | } 52 | 53 | -------------------------------------------------------------------------------- /man/ZIP_TO_ZCTA.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/dataSetDocs.R 3 | \docType{data} 4 | \name{ZIP_TO_ZCTA} 5 | \alias{ZIP_TO_ZCTA} 6 | \title{A zip code to census zip code tabulation area mapping} 7 | \format{A data frame with 41979 rows and 5 variables: 8 | \describe{ 9 | \item{ZIP}{zip code, as a 0 padded string} 10 | \item{ZIPType}{zip code type} 11 | \item{CityName}{Name of the city the zip code is in} 12 | \item{StateAbbr}{Two letter abbreviation for the state the zip code is in} 13 | \item{ZCTA}{the census zip code tabulation area (ZCTA) the zip code most overlaps with} 14 | }} 15 | \source{ 16 | \url{http://some.random.helpful.blog/} 17 | } 18 | \usage{ 19 | data(ZIP_TO_ZCTA) 20 | } 21 | \description{ 22 | A dataset containing columns with US zip codes and the corresponding ZCTA ids 23 | } 24 | \keyword{datasets} 25 | 26 | -------------------------------------------------------------------------------- /man/acs.fetch.and.cache.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-census.R 3 | \name{acs.fetch.and.cache} 4 | \alias{acs.fetch.and.cache} 5 | \title{Download census data, employing caching to avoid repeated downloads.} 6 | \usage{ 7 | acs.fetch.and.cache(...) 8 | } 9 | \description{ 10 | A thin caching wrapper around \code{acs.fetch(...)} in the acs package. 11 | } 12 | \examples{ 13 | acs.fetch.and.cache(...) 14 | } 15 | 16 | -------------------------------------------------------------------------------- /man/applyDateFilters.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/filter.R 3 | \name{applyDateFilters} 4 | \alias{applyDateFilters} 5 | \title{Date filter meter data} 6 | \usage{ 7 | applyDateFilters(df, filterRules = NULL, dateCol = "dates") 8 | } 9 | \arguments{ 10 | \item{df}{The data frame of data to filter} 11 | 12 | \item{filterRules}{A named list of filtering rules. Supported list entries include: 13 | 14 | \code{MOY} - list of months of the year to include, using 1 for Jan through 12 for Dec 15 | 16 | \code{DOW} - days of week to include, using 1 for Sun and 7 for Sat 17 | 18 | \code{start.date} - the first day of data to include: all dates before this date are excluded 19 | 20 | \code{end.date} - the last day of data to include: all dates after this date are excluded} 21 | 22 | \item{dateCol}{The name of the column in 'df' with dates in it.} 23 | } 24 | \description{ 25 | Utility function that filters raw meter data (one customer day per row) based on date criteria 26 | } 27 | 28 | -------------------------------------------------------------------------------- /man/basicFeatures.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/features-basic.R 3 | \name{basicFeatures} 4 | \alias{basicFeatures} 5 | \title{Implementation of several basic statistical features} 6 | \usage{ 7 | basicFeatures(meterData, ...) 8 | } 9 | \arguments{ 10 | \item{meterData}{An instance of \code{\link{MeterDataClass}} that provides the data used for all the implemented feature calculations.} 11 | } 12 | \description{ 13 | Function that implements a set of common statistical features that can be run on an instance of \code{\link{MeterDataClass}}. 14 | } 15 | \details{ 16 | \code{basicFeatures} is called by passing in an instance of a MeterDataClass, which provides the meter data in both linear series 17 | and day-per-row matrix formats. These data structures and some associated indices and intermediate data structures are 18 | re-used throughout this method, which tries to take advantage of the performance benefits of such re-use. 19 | This function is often called in the context of the \code{iterator.*} suite of functions, which can loop through large sets of 20 | meters, calling feature functions on each. 21 | } 22 | \examples{ 23 | \dontrun{ 24 | DATA_SOURCE = TestData(n=10) 25 | meterData = DATA_SOURCE$getMeterDataClass(DATA_SOURCE$getIds()[1]) 26 | features = basicFeatures(meterData) 27 | names(features) 28 | class(features) 29 | } 30 | } 31 | 32 | -------------------------------------------------------------------------------- /man/buildWhere.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{buildWhere} 4 | \alias{buildWhere} 5 | \title{Utility function that assembles a where clause from a list of named values. All values are 'AND'ed together. 6 | Does not support nested expressions or 'OR's, except if the values are length() > 1, they are put into an 7 | in() clause enumerating those values.} 8 | \usage{ 9 | buildWhere( config = list(a='foo', b=20, c=c('one','two','three'), d = 1:9) ) 10 | buildWhere( config = list(a='foo', b=20, c=c('one','two','three'), d = 1:9), 11 | other_clause = '(name = "jimmy" or name = "johnny")', 12 | suffix='limit 100' ) 13 | } 14 | \arguments{ 15 | \item{config}{List of named values to be included in the where clause. Supports caracter and numeric classes, 16 | using in() syntax for those that are length > 1.} 17 | 18 | \item{other_clause}{An arbitrary clause that is prepended to the list of clauses that are 'AND'ed together to form the where statement} 19 | 20 | \item{suffix}{Arbitrary text that is appended to the where clause after it has been generated. For example 'limit 100' or 'order by id'} 21 | } 22 | \details{ 23 | \code{buildWhere} is a utility function builds up a where clause using a passes list with named values. 24 | Values of different types require different handling with respect to quotation marks and valus that are arrays 25 | are turned into in( ... ) clauses. In the corner case where nothing is passed in, it returns empty string, 26 | so it should be friendly to concatenation to a select statement regardless of whether there are values. 27 | } 28 | 29 | -------------------------------------------------------------------------------- /man/cleanFeatureDF.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{cleanFeatureDF} 4 | \alias{cleanFeatureDF} 5 | \title{Clean up feature data in preparation for saving} 6 | \usage{ 7 | cleanFeatureDF(features, checkId = TRUE, checkGeo = TRUE) 8 | } 9 | \arguments{ 10 | \item{features}{The data frame of feature to be cleaned up} 11 | 12 | \item{checkId}{boolean indicating whether to enforce a check for an id column with an error message. This should 13 | be true when exporting features or other id matched data and false otherwise.} 14 | 15 | \item{checkGeo}{boolean indicating whether to enforce a check zip5 columns with a warning message. This should 16 | be true when exporting features that will be mapped.} 17 | } 18 | \value{ 19 | A copy of the original data frame that is cleaned up 20 | } 21 | \description{ 22 | This function renames data columns for export via fixNames(), converts factors 23 | into characters, and checks for id and zip5 colums 24 | } 25 | 26 | -------------------------------------------------------------------------------- /man/clearCons.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{clearCons} 4 | \alias{clearCons} 5 | \title{Close and clear all connections} 6 | \usage{ 7 | clearCons(cfg) 8 | } 9 | \arguments{ 10 | \item{cfg}{a named list of configuration options, often loaded from a flat configuration file using \code{\link{dbCfg}}.} 11 | } 12 | \description{ 13 | Utility function to close/clear all active db connections, which can sometimes get left open, especially when errors 14 | prevent connections from being closed. 15 | } 16 | 17 | -------------------------------------------------------------------------------- /man/clear_acs_cache.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-census.R 3 | \name{clear_acs_cache} 4 | \alias{clear_acs_cache} 5 | \title{Flush the cache directory that stores downloaded census data.} 6 | \usage{ 7 | clear_acs_cache() 8 | } 9 | \examples{ 10 | clear_acs_cache() 11 | } 12 | 13 | -------------------------------------------------------------------------------- /man/conf.dbCon.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{conf.dbCon} 4 | \alias{conf.dbCon} 5 | \title{Get a database connection} 6 | \usage{ 7 | conf.dbCon(cfg) 8 | } 9 | \arguments{ 10 | \item{cfg}{a named list of configuration options, often loaded from a flat configuration file using \code{\link{dbCfg}}.} 11 | } 12 | \description{ 13 | Utility function that opens a database connection using the standard \code{dbCfg} format for all configuration options. 14 | } 15 | \details{ 16 | \code{conf.dbCon} is called by passing in a names list that contains \code{dbType} providing the name of the desired database driver, 17 | which is passed directly into \code{\link{DBI::dbDriver}} to get an instance of the required driver. The rest of the parameters are 18 | driver specific, but typically include \code{host}, \code{user}, \code{password}, and \code{dbname}. These are passed as arguments 19 | into a call to \code{\link{DBI::dbConnect}}, using the named driver to get a database connection and return it. 20 | } 21 | \seealso{ 22 | \code{\link{dbDfg}}, \code{\link{run.query}} 23 | } 24 | 25 | -------------------------------------------------------------------------------- /man/ctree.boot.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{ctree.boot} 4 | \alias{ctree.boot} 5 | \title{bootstrap several runs of the ctree algorithm} 6 | \usage{ 7 | ctree.boot(df, nRun = 100, nSample = NULL, ctl = NULL, responseVar, 8 | ignoreCols) 9 | } 10 | \arguments{ 11 | \item{df}{data.frame with response variable to be explained and all explanatory variables for the ctree to run with} 12 | 13 | \item{nRun}{number of bootstrapping runs to perform} 14 | 15 | \item{nSample}{number of samples (with replacement) for each run} 16 | 17 | \item{ctl}{ctree control instance to be passed to ctree algorithm to control stopping criteria, etc.} 18 | 19 | \item{responseVar}{the name of the variable that the ctree is trying to explain} 20 | 21 | \item{ignoreCols}{an optional list of columns in df, but that should be excluded from the ctree modeling} 22 | } 23 | \description{ 24 | This function repeatedly runs the ctree algorithm on random sub-samples of the underlying data 25 | to provide a measure of robustness for the individual ctree results. 26 | } 27 | 28 | -------------------------------------------------------------------------------- /man/ctree.run.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{ctree.run} 4 | \alias{ctree.run} 5 | \title{run a ctree model} 6 | \usage{ 7 | ctree.run(df, fmla = NULL, sub = NULL, ctl = NULL, responseVar, 8 | ignoreCols, ...) 9 | } 10 | \arguments{ 11 | \item{df}{data.frame with response variable to be explained and all explanatory variables to be used in ctree classificaiton} 12 | 13 | \item{fmla}{formula, referring to values in the df, specifying the variables to be used to train the ctree model} 14 | 15 | \item{ctl}{partykit::ctree_control instance to specify ctree model parameters} 16 | 17 | \item{responseVar}{the name of the variable that the ctree is trying to explain} 18 | 19 | \item{ignoreCols}{an optional list of columns in df, but that should be excluded from the ctree modeling} 20 | 21 | \item{...}{additional arguments to be passed to partykit::ctree} 22 | } 23 | \description{ 24 | This function trains a partykit::ctree using data from a passed data.frame. 25 | } 26 | 27 | -------------------------------------------------------------------------------- /man/ctree.subscore.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{ctree.subscore} 4 | \alias{ctree.subscore} 5 | \title{fit a ctree with passed data and compute and return its spread score} 6 | \usage{ 7 | ctree.subscore(i, df, nSample = NULL, ctl = NULL, responseVar, ignoreCols) 8 | } 9 | \arguments{ 10 | \item{df}{data.frame with response variable to be explained and all explanatory variables to be used in ctree classificaiton} 11 | 12 | \item{nSample}{number of samples if hte spread score should be run on a subset of the data} 13 | 14 | \item{ctl}{ctree control to specify ctree model parameters} 15 | 16 | \item{responseVar}{the name of the variable that the ctree is trying to explain} 17 | 18 | \item{ignoreCols}{an optional list of columns in df, but that should be excluded from the ctree modeling} 19 | } 20 | \description{ 21 | This function trains a ctree using data from a passed data.frame and returns the resulting spread score. 22 | } 23 | 24 | -------------------------------------------------------------------------------- /man/datesToEpoch.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{datesToEpoch} 4 | \alias{datesToEpoch} 5 | \title{Convert POSIXct data frame columns to integer seconds since the epoch} 6 | \usage{ 7 | datesToEpoch(df) 8 | } 9 | \arguments{ 10 | \item{df}{Data frame of feature data} 11 | } 12 | \value{ 13 | A data frame identical to the one passed in, but with integer columns replacing POSIXct ones 14 | } 15 | \description{ 16 | Searches the data frame for columns with 'POSIXct' in their class values and 17 | converts them to integers representing seconds since the epoch 18 | } 19 | 20 | -------------------------------------------------------------------------------- /man/dbCfg.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{dbCfg} 4 | \alias{dbCfg} 5 | \title{Parse standard database configuration file format} 6 | \usage{ 7 | dbCfg(filePath) 8 | } 9 | \arguments{ 10 | \item{filePath}{The file path to the config file to be read in. Must be absolute or relative to the working directory from \code{getwd()}} 11 | } 12 | \description{ 13 | Read database config file with key=value format: 14 | \code{key=value 15 | key2=value2} 16 | etc. into a named list. 17 | Keys must include \code{dbType} passed in to \code{DBI::dbDriver} to load the appropriate driver (i.e. MySQL, PostgreSQL, SQLite, etc.) 18 | and any parameters like \code{host}, \code{user}, \code{password}, \code{dbname} that should be passed into the DBI::dbConnect function call for your driver. 19 | } 20 | \details{ 21 | The result of this function call is just a list with named entries. As long as the entries are valid, it can be generated in other ways. 22 | Note that the ability to specify a single file that contains database configuration information allows for 23 | multiple users to share a code based, but provide their personal credentials in their copy of the relevant cfg file. 24 | Due to the inevitability of code conflicts with these files where user names and passwords are different for each user 25 | and the fact that it is insecure and unwise to include database credentials in any version control system, this means 26 | that such dbCfg files should not be checked into version control. 27 | } 28 | \seealso{ 29 | \code{\link{conf.dbCon}}, \code{\link{run.query}} 30 | } 31 | 32 | -------------------------------------------------------------------------------- /man/exportData.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{exportData} 4 | \alias{exportData} 5 | \title{Export feature data into a selection of formats} 6 | \usage{ 7 | exportData(df, name, label = NA, format = "hdf5", checkId = TRUE, 8 | checkGeo = TRUE, ...) 9 | } 10 | \arguments{ 11 | \item{df}{Data frame of feature data to export} 12 | 13 | \item{name}{Primary name of export, meaning file name or database table name} 14 | 15 | \item{label}{Optional data label for export formats. For example if not NA, this would be the name 16 | of the data table within an hdf5 file or a suffix to the csv file name, as in \code{paste(name, label, sep='_')}} 17 | 18 | \item{format}{One of the supported formats for data export, currently 'hdf5', 'csv', or 'database'} 19 | 20 | \item{checkId}{boolean control over whether to error out with a \code{stop()} if an id column is not present} 21 | 22 | \item{checkGeo}{boolean control over whether to warn if a geographic field, \code{zip5} in this case, is not present.} 23 | 24 | \item{...}{Pass through parameters for specific export methods. For example, 25 | database export requires a conn object.} 26 | } 27 | \description{ 28 | Runs the export function for a given data format on feature data 29 | } 30 | 31 | -------------------------------------------------------------------------------- /man/exportFeatureAndShapeResults.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{exportFeatureAndShapeResults} 4 | \alias{exportFeatureAndShapeResults} 5 | \title{Export feature run and load shape results} 6 | \usage{ 7 | exportFeatureAndShapeResults(feature.data, shape.results.data = NULL, 8 | format = "hdf5", prefix = "", filePath = ".") 9 | } 10 | \arguments{ 11 | \item{feature.data}{File path to an RData file with feature data data frame or the data frame itself} 12 | 13 | \item{shape.results.data}{Optional file path to an RData file containing load shape clustering results 14 | or the results object itself. i.e. results from 15 | \code{visdomloadshape::shapeFeatures(visdomloadshape::shapeCategoryEncoding())}} 16 | 17 | \item{format}{Export data format - one of the ones supported by exportData()} 18 | 19 | \item{prefix}{Optional prefix to put n froun of all feature names} 20 | 21 | \item{filePath}{optional path to the directory where exported data should be written if the export type is a file. '.' by default.} 22 | } 23 | \description{ 24 | Loads feature data and load shape clustering data from RData files and 25 | saves them into the selected export format 26 | } 27 | 28 | -------------------------------------------------------------------------------- /man/exportShapes.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{exportShapes} 4 | \alias{exportShapes} 5 | \title{Save load shape results} 6 | \usage{ 7 | exportShapes(shape.results, prefix = "", format = "hdf5", filePath = ".") 8 | } 9 | \arguments{ 10 | \item{shape.results}{the shape feature results to export. These should be in the format returned by 11 | \code{visdomloadshape::shapeFeatures()}, as in 12 | \code{shapeFeatures(shapeCategoryEncoding(rawData=DATA_SOURCE$getAllData(), metaCols=1:4, encoding.dict=someDict))}} 13 | 14 | \item{prefix}{a prefix to apply to the feature column names} 15 | 16 | \item{format}{the data format for export. One of the values supported by the \code{format} paramater in \code{exportData()}} 17 | 18 | \item{filePath}{optional path to the location where exported files should be written (if applicable). Default is \code{getwd()}} 19 | } 20 | \description{ 21 | Exports standardized load shape clustering and assignment data into a 22 | corresponding set of exported data tables 23 | } 24 | 25 | -------------------------------------------------------------------------------- /man/fixNames.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{fixNames} 4 | \alias{fixNames} 5 | \title{Prepare data frame column names for export} 6 | \usage{ 7 | fixNames(df, prefix = "") 8 | } 9 | \arguments{ 10 | \item{df}{The data frame whose columns are to be renamed} 11 | 12 | \item{prefix}{An optional prefix to place in front of all column names} 13 | } 14 | \value{ 15 | A data frame identical to the one passed in, but with new column names. 16 | } 17 | \description{ 18 | Removes punctuation from data frame column names, replacing all with underscores 19 | and removing underscores that are repeated one after another 20 | } 21 | 22 | -------------------------------------------------------------------------------- /man/getRunId.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{getRunId} 4 | \alias{getRunId} 5 | \title{Get the database id of a feature_set/run combination} 6 | \usage{ 7 | getRunId(conn, runConfig) 8 | } 9 | \arguments{ 10 | \item{conn}{A database connection, usually obtained from \code{conf.dbCon} or \code{\link{DBI::dbConnect}}} 11 | 12 | \item{runConfig}{The run configuration file with key-value pairs of 13 | feature_set, feature_set_description, run_name and run_description. See 14 | 1inst/feature_set_run_conf/exampl_feature_set.conf1 for an example.} 15 | } 16 | \description{ 17 | Load user-specified run config file and return a unique numeric 18 | id from the feature_runs metadata table. Create the feature_runs table if 19 | it does not exist. 20 | } 21 | 22 | -------------------------------------------------------------------------------- /man/groupHist.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{groupHist} 4 | \alias{groupHist} 5 | \title{ctree classificaiton results plot for a binary response variable} 6 | \usage{ 7 | groupHist(df, df.ct, responseCol, title = "Group probabilities", 8 | idCol = "id", xlab = "Enrollment probability (\%)", normalize = FALSE, 9 | compareData = NULL, colors = NULL) 10 | } 11 | \arguments{ 12 | \item{df}{data.frame with response variable to be explained and all explanatory variables to be used in ctree classificaiton} 13 | 14 | \item{df.ct}{ctree already trained on data, that is used to classify the data points in each sub sample.} 15 | 16 | \item{responseCol}{the name of the variable that the ctree is trying to explain} 17 | 18 | \item{title}{figure title of the plot} 19 | 20 | \item{idCol}{column with data row identifiers in it, which is removed from the df} 21 | 22 | \item{xlab}{label for x-axis of the figure} 23 | 24 | \item{normalize}{boolean for whether the x-axis should be percentages compared to the sample mean or absolute percentages} 25 | 26 | \item{compareData}{optional data.fram of one additional set of bar values to be plotted as well.} 27 | 28 | \item{colors}{optional named list specifying the figure colors.} 29 | } 30 | \description{ 31 | plot a histogram of customer segments and enrollment percentages, based on the membership of ctree leaf nodes. 32 | The plot is a "histogram" whose x-axis is the average "True" response for each ctree leaf node and whose 33 | height is the number of customers in each corresponding group. Strong results have nodes with high membership 34 | far from the sample mean. 35 | } 36 | 37 | -------------------------------------------------------------------------------- /man/iterator.build.idx.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.build.idx} 4 | \alias{iterator.build.idx} 5 | \title{Optimize single meter lookup in a data.frame containing many meters} 6 | \usage{ 7 | iterator.build.idx(ctx) 8 | } 9 | \arguments{ 10 | \item{ctx}{The ctx environment that configures a feature run. This function is specifically looking 11 | to build an index for \code{ctx$RAW_DATA}, if present and saves the results as \code{ctx$idxLookup}. It allows 12 | for RAW_DATA to be set as a data.frame with data from a large number of meters loaded at once and in 13 | advance of the feature extraction runs to reduce runtimes.} 14 | } 15 | \description{ 16 | This function does a fast search for the first and last indices for each meter's data in a data.frame 17 | with many meters and caches the resulting indices. These indices can be used to very quickly access data 18 | for individual meters. 19 | } 20 | \details{ 21 | Standard methods of searching for all data for a given meterId would use boolean expressions like 22 | \code{ctx$RAW_DATA[ctx$RAW_DATA$meterId == 'some_id',]}. It turns out that this is pretty inefficient 23 | for large data.frames becaus it generates values for all rows and then does the necessary comparisons for 24 | all rows. Direct numerical indexing avoids this overhead and such indices can be quickly computed using 25 | \code{\link{match}} and the fact that all data for each meter must be returned from the data source in 26 | contigious rows. Finally, the constructor for a \code{MeterDataClass} checks for these \code{ctx$idxLookup} if 27 | \code{ctx$RAW_DATA} is found and uses them to pull the subset of data associated with the meterId passed in 28 | to that constructor. 29 | } 30 | 31 | -------------------------------------------------------------------------------- /man/iterator.callAll.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.callAll} 4 | \alias{iterator.callAll} 5 | \title{Run multiple feature algorithms specified in a list of functions} 6 | \usage{ 7 | iterator.callAll(meterDataClass, ctx, fnList, ...) 8 | } 9 | \arguments{ 10 | \item{meterDataClass}{The \code{\link{MeterDataClass}} object that contains the data to be analyzed.} 11 | 12 | \item{ctx}{The ctx environment that configures feature runs and provides a place to store and pass data across feature function calls.} 13 | 14 | \item{fnList}{The list of feature extraction functions to be run against the \code{\link{MeterDataClass}}.} 15 | 16 | \item{...}{Additional arguments that will be passed into the feature functions.} 17 | } 18 | \description{ 19 | This function passes the meterDataClass object into 20 | every feature function in the list found in the parameter \code{fnList}, 21 | concatenating all the results into a single list with named values. 22 | } 23 | 24 | -------------------------------------------------------------------------------- /man/iterator.callAllFromCtx.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.callAllFromCtx} 4 | \alias{iterator.callAllFromCtx} 5 | \title{Run feature algorithms specified in the ctx environment} 6 | \usage{ 7 | iterator.callAllFromCtx(meterDataClass, ctx, ...) 8 | } 9 | \arguments{ 10 | \item{meterDataClass}{The \code{\link{MeterDataClass}} object that contains the data to be analyzed.} 11 | 12 | \item{ctx}{The ctx environment that contains the list of functions to run under the name \code{fnVector} and 13 | configures feature runs and provides a place to store and pass data across feature function calls.} 14 | 15 | \item{...}{Additional arguments that will be passed into the feature functions.} 16 | } 17 | \description{ 18 | This function loads the meter data associated with the passed \code{meterId} and creates 19 | a MeterDataClass instance using that data. It then passes that meter data object into 20 | every feature function in the list found in the ctx environment under \code{fnVector}, 21 | concatenating all the results into a single list with named values. 22 | } 23 | 24 | -------------------------------------------------------------------------------- /man/iterator.iterateMeters.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.iterateMeters} 4 | \alias{iterator.iterateMeters} 5 | \title{Run features for a list of meter ids} 6 | \usage{ 7 | iterator.iterateMeters(meterList, custFn, ctx = new.env(), as_df = FALSE, 8 | ...) 9 | } 10 | \arguments{ 11 | \item{meterList}{The list of meterIds to use to instantiate a \code{MeterDataClass} objects} 12 | 13 | \item{custFn}{The feature extraction function to run on each instance of \code{MeterDataClass}. 14 | Can also be a list of feature functions, whose results will be \code{c()}'d together, so make sure the names of their 15 | return values are unique!} 16 | 17 | \item{ctx}{The ctx environment for the feature extraction.} 18 | 19 | \item{as_df}{Boolean (defaulted to FALSE) that determines whether \code{iterator.todf} is run on 20 | the results before returning them.} 21 | 22 | \item{...}{Additional arguments to be passed to the feature extraction function.} 23 | } 24 | \description{ 25 | Utility function that runs the configured set of feature functions on the data from a list of passed 26 | meter ids. This is useful for scripting feature extraction on an arbitrary number of meters. 27 | } 28 | 29 | -------------------------------------------------------------------------------- /man/iterator.iterateZip.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.iterateZip} 4 | \alias{iterator.iterateZip} 5 | \title{Iterate over zip codes, extracting features for meters in each} 6 | \usage{ 7 | iterator.iterateZip(zipList, custFn, cacheResults = FALSE, ctx = new.env(), 8 | clearCachedResultsFromRAM = FALSE, as_df = FALSE, ...) 9 | } 10 | \arguments{ 11 | \item{zipList}{List of all zip codes to iterate over} 12 | 13 | \item{custFn}{The feature function(s) to call on \code{MeterDataClass} instances from within each zip code 14 | If it is a list of feature functions, the results will be \code{c()}'d together, so make sure the names of their 15 | return values are unique!} 16 | 17 | \item{cacheResults}{A boolean flag that indicates whether feature results should be cached as RData. 18 | If true, the cached data will be looked up prior to feature extraction to bypass running 19 | features for the given zip code while returning the cached results. This can prevent duplicate processing 20 | and allow an interrupted batch process to resume processing where it left off. 21 | The cache directory is `getwd()` by default, but can be overridden using `ctx$resultsCache`.} 22 | 23 | \item{ctx}{The ctx environment that configures the feature run.} 24 | 25 | \item{clearCachedResultsFromRAM}{A boolean that instructs the code to drop previous results from RAM if caching 26 | is in effect. This allows for larer runs to be executed and cached without accumulating in RAM. The idea is 27 | that after the run, this function will be called again, with useCache=TRUE and this flag set to FALSE, which 28 | will simply load and concatenate results from disk to re-create the full set.} 29 | 30 | \item{as_df}{Boolean (defaulted to FALSE) that determines whether \code{iterator.todf} is run on 31 | the results before returning them.} 32 | 33 | \item{...}{Arguments to be passed into the feature function(s).} 34 | } 35 | \description{ 36 | Function that iterates through all passed zip codes, looks up local weather data (once) and 37 | looks up the list of meter ids for each and calls iterator.iterateMeters with those meter 38 | ids and the per-loaded weather data. This runs faster than calling ids individually, which 39 | can load similar weather data over and over. 40 | } 41 | 42 | -------------------------------------------------------------------------------- /man/iterator.runMeter.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.runMeter} 4 | \alias{iterator.runMeter} 5 | \title{Run features for a single meterId} 6 | \usage{ 7 | iterator.runMeter(meterId, custFn, ctx, ...) 8 | } 9 | \arguments{ 10 | \item{meterId}{The meterId to use to instantiate a \code{MeterDataClass} object} 11 | 12 | \item{custFn}{The feature extraction function(s) to run on the instance of \code{MeterDataClass}. 13 | If it is a list of feature functions, the results will be \code{c()}'d together, so make sure the names of their 14 | return values are unique!} 15 | 16 | \item{ctx}{The ctx environment for the feature extraction.} 17 | 18 | \item{...}{Additional arguments to be passed to the feature extraction function.} 19 | } 20 | \description{ 21 | Utility function that runs the configured set of feature functions on a single passed 22 | meter. This is useful for testing and feature development, but also as the function 23 | called by parallelizable methods like apply, and the *ply function of plyr to run feature 24 | extraction in on multiple cores of a computer. 25 | } 26 | 27 | -------------------------------------------------------------------------------- /man/iterator.runZip.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.runZip} 4 | \alias{iterator.runZip} 5 | \title{Iterate over all meters in a zip code, extracting features for each in a performance optimized manner} 6 | \usage{ 7 | iterator.runZip(zip, custFn, cacheResults = F, ctx = new.env(), 8 | as_df = FALSE, ...) 9 | } 10 | \arguments{ 11 | \item{zip}{The zip code to use for the weather, meter ids, and meter data lookups} 12 | 13 | \item{custFn}{The feature function(s) to call on \code{MeterDataClass} instances from within the zip code 14 | If it is a list of feature functions, the results will be \code{c()}'d together, so make sure the names of their 15 | return values are unique!} 16 | 17 | \item{cacheResults}{A boolean flag that indicates whether results should be cached as RData. 18 | If true, the cached data will be looked up prior to feature extraction to bypass running 19 | features for the given zip code while returning the cached results. This can prevent duplicate processing 20 | and allow an interrupted batch process to resume processing where it left off. 21 | The cache directory is `getwd()` by default, but can be overridden using `ctx$resultsCache`.} 22 | 23 | \item{ctx}{The ctx environment that configures the feature run.} 24 | 25 | \item{as_df}{Boolean (defaulted to FALSE) that determines whether \code{iterator.todf} is run on 26 | the results before returning them.} 27 | 28 | \item{...}{Arguments to be passed into the feature function(s).} 29 | } 30 | \description{ 31 | Function that looks up local weather and all meter data for the passed zip code 32 | (once) and caches them and then looks up all meter ids in the passed zip code 33 | calls iterator.iterateMeters with those meter ids and the pre-loaded weather and meter data. 34 | This runs faster than calling ids individually, which load individual meter data and similar 35 | weather data over and over. 36 | } 37 | 38 | -------------------------------------------------------------------------------- /man/iterator.todf.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{iterator.todf} 4 | \alias{iterator.todf} 5 | \title{Flatten feature data returned by iterator functions into a \code{data.frame}.} 6 | \usage{ 7 | iterator.todf(lofl) 8 | } 9 | \arguments{ 10 | \item{lofl}{The list of lists for feature values that should be flattened into a single data frame, with one row per meterId.} 11 | } 12 | \description{ 13 | Itertor functions like \code{iterator.iterateMeters} return lists of lists of derived features indexed by meterIds. 14 | This function flattens the scalar values in these lists into data.frame, with one row per meterId and columns for every named feature found. 15 | } 16 | \details{ 17 | This function ignores non-scalars, so diagnostic data and other complex data structures can be returned by feature algorithms without 18 | interfering with the task of creating clean vectors of features for each meter. The columns of the returned data.frame are 19 | a super set of all the features returned for every MeterDataClass with computed features. Missing features are given values of NA. 20 | } 21 | 22 | -------------------------------------------------------------------------------- /man/loadACS.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-census.R 3 | \name{loadACS} 4 | \alias{loadACS} 5 | \title{Retrieve census data from the American Community Survey.} 6 | \usage{ 7 | loadACS(filterErr = T, endyear = 2014, span = 5, geography = NULL, 8 | acs_vars_to_load = NULL) 9 | } 10 | \arguments{ 11 | \item{geography}{The geographic aggregations and search terms to use with the \code{acs.fetch} 12 | interface. This will default to all zip codes.} 13 | 14 | \item{acs_vars_to_load}{A dataframe to specify which variables to load and what to name the resultant columns. 15 | The dataframe should have columns "census_var" and "label" as characters. See example below. 16 | This will default to the variables specified in data-raw/census/census_vars.txt} 17 | 18 | \item{filterErr=T}{Remove the standard error columns from the results} 19 | 20 | \item{endyear=2014}{Specify which year of data to download. Currently, 2013 & 2014 are available.} 21 | 22 | \item{span=5}{Whether you want the 3-year or the 5-year estimates. See census website for more explanations.} 23 | } 24 | \description{ 25 | Download data from the Census API and cache it locally on disk. This is a 26 | thin wrapper around the \code{acs.fetch} function in the [acs 27 | package](https://cran.r-project.org/web/packages/acs/index.html). The results 28 | of acs.fetch() are converted from a fancy S4 class to a simple data frame. 29 | The geographical designations (zip code, state, census block, etc) are specified 30 | in the row names, and if zip code (actually ZCTA) is used for the geography, the 31 | numeric value is also available as a column. 32 | } 33 | \details{ 34 | Before using these functions, you will need to set a census API key. You can get one from 35 | http://api.census.gov/data/key_signup.html Once you have obtained a key, set it up like so: 36 | \code{ 37 | acs::api.key.install(key='PASTE_YOUR_KEY_HERE') 38 | } 39 | If you are a developer who frequently reinstalls packages, you may want to 40 | copy this into your .Rprofile 41 | 42 | The path to the cache directory is stored as an R option. You may customize 43 | it by creating a .Rprofile file in either your project or home directory. 44 | \code{ 45 | options("visdom.acs.cache"="/new/path/to/cache/dir") 46 | getOption("visdom.acs.cache") 47 | clear_acs_cache() 48 | } 49 | 50 | The default set of Data Profile variables are specified in 51 | raw-data/census/census_vars.txt. If you edit that list, don't forget to 52 | re-execute migrateData.R. You can look at what variables are available by 53 | going to 54 | \url{http://www.census.gov/data/developers/data-sets/acs-5year.2014.html}, 55 | choosing the appropriate year, searching the page for "ACS Data Profile API 56 | Variables", and clicking 57 | \href{http://api.census.gov/data/2014/acs5/profile/variables.html}{html}. As of 58 | November 11, 2016, the API only works reliably for 2013 and 2014. 59 | } 60 | \examples{ 61 | library(acs) 62 | # Load data by state 63 | loadACS(geography = acs::geo.make(state='*')) 64 | # Load data for 2 zip codes 65 | loadACS(geography = c(acs::geo.make(zip.code='94709'), acs::geo.make(zip.code='94710'))) 66 | # Load data for all tracts, restricted to California. 67 | loadACS(geography = acs::geo.make(state='CA', county='*', tract='*')) 68 | # Unfortunately, Zip codes can't be restricted to a single state like tracts can. 69 | 70 | # Selecting different variables 71 | acs_vars_to_load = data.frame( 72 | census_var = c("DP02_0015", "DP04_0047"), 73 | label = c("avg_hh_size", "owner_hh_size"), 74 | stringsAsFactors = F ) 75 | loadACS(acs_vars_to_load = acs_vars_to_load) 76 | } 77 | \seealso{ 78 | \code{\link{DataSource}} 79 | } 80 | 81 | -------------------------------------------------------------------------------- /man/mergeShapeFeatures.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{mergeShapeFeatures} 4 | \alias{mergeShapeFeatures} 5 | \title{Merge load shape features into feature data frame} 6 | \usage{ 7 | mergeShapeFeatures(features, shape.results) 8 | } 9 | \arguments{ 10 | \item{features}{Data frame of feature data} 11 | 12 | \item{shape.results}{Load shape clustering and assignment results object to pull features from} 13 | } 14 | \value{ 15 | A data frame identical to the one passed in, but with new laod shape feature columns 16 | } 17 | \description{ 18 | Pulls load shape features from a shape results object and appends them to 19 | an existing feature data frame 20 | } 21 | 22 | -------------------------------------------------------------------------------- /man/piecewise.regressor.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-regression.R 3 | \name{piecewise.regressor} 4 | \alias{piecewise.regressor} 5 | \title{Break a vector into piecewise regressor columns, using the specified breaks (aka bins)} 6 | \usage{ 7 | piecewise.regressor(bins, regressor, ...) 8 | } 9 | \arguments{ 10 | \item{bins}{- numerical arry whose values are the break points defining the piecewise regressors} 11 | 12 | \item{regressor}{The single array of regressor values to be broken out into piecewise columns.} 13 | 14 | \item{diverge}{- Defautl FALSE: Whether the first column contains the distance of 15 | the value from the bottom change point (rather than from 0)} 16 | } 17 | \description{ 18 | convienience reordering of \code{\link{regressor.piecewise}} putting the bins first for apply style calls... 19 | 20 | break a vector out for continuous piecewise regression (i.e. the fitted 21 | segments will join eachother at each end) into a matrix whose row 22 | totals are the original values, but whose columns divide the value across 23 | a set of bins, so 82 across bins with boundaries c(50,60,65,70,80,90) 24 | becomes the row 50,10,5,5,10,2,0, which sums to 82... 25 | This is very useful for finding rough change points in thermal response 26 | as is expected for buildings with clear setpoints 27 | } 28 | \details{ 29 | This can be used in an \code{apply} setting to run a parametric grid search of break points. 30 | 31 | TODO: This can create a column of zeros, which should break the regression 32 | so we might need to prune the columns when we're done and keep track of 33 | which bins are in play when comparing across regressions 34 | } 35 | \seealso{ 36 | \code{\link{regressor.piecewise}} for 'normal' syntax 37 | } 38 | 39 | -------------------------------------------------------------------------------- /man/reexports.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-base.R 3 | \docType{import} 4 | \name{reexports} 5 | \alias{\%>\%} 6 | \alias{reexports} 7 | \title{Objects exported from other packages} 8 | \description{ 9 | These objects are imported from other packages. Follow the links 10 | below to see their documentation. 11 | 12 | \describe{ 13 | \item{dplyr}{\code{\link[dplyr]{\%>\%}}} 14 | }} 15 | \keyword{internal} 16 | 17 | -------------------------------------------------------------------------------- /man/regressor.piecewise.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-regression.R 3 | \name{regressor.piecewise} 4 | \alias{regressor.piecewise} 5 | \title{Break a vector into piecewise regressor columns, using the specified breaks} 6 | \usage{ 7 | regressor.piecewise(regressor, bins, diverge = F) 8 | } 9 | \arguments{ 10 | \item{regressor}{The single array of regressor values to be broken out into piecewise columns.} 11 | 12 | \item{bins}{- numerical arry whose values are the break points defining the piecewise regressors} 13 | 14 | \item{diverge}{- Defautl FALSE: Whether the first column contains the distance of 15 | the value from the bottom change point (rather than from 0)} 16 | } 17 | \description{ 18 | break a vector out for continuous piecewise regression (i.e. the fitted 19 | segments will join eachother at each end) into a matrix whose row 20 | totals are the original values, but whose columns divide the value across 21 | a set of bins, so 82 across bins with boundaries c(50,60,65,70,80,90) 22 | becomes the row 50,10,5,5,10,2,0, which sums to 82... 23 | This is very useful for finding rough change points in thermal response 24 | as is expected for buildings with clear setpoints 25 | } 26 | \details{ 27 | TODO: This can create a column of zeros, which should break the regression 28 | so we might need to prune the columns when we're done and keep track of 29 | which bins are in play when comparing across regressions 30 | } 31 | \seealso{ 32 | \code{\link{piecewise.regressor}} for 'apply' syntax 33 | } 34 | 35 | -------------------------------------------------------------------------------- /man/regressor.split.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-regression.R 3 | \name{regressor.split} 4 | \alias{regressor.split} 5 | \title{Split a single column of values into multiple columns as determined by group membership} 6 | \usage{ 7 | regressor.split(regressor, membership = NULL) 8 | } 9 | \arguments{ 10 | \item{regressor}{The vector of data to be split} 11 | 12 | \item{membership}{A vector of the same length as the regressor that assigns group membership categorically. 13 | If the regressor is a factor, no membership assignment is required.} 14 | } 15 | \value{ 16 | A matrix of length(sort(unique(membership))) columns, that is all 17 | zeros except for the regressor values that match the membership values 18 | corresponding to the column. This supports regression with separate 19 | coefficints for each group defined by the membership 20 | for example, regressor.split(Tout,dates$hour) will return a matrix with 24 21 | columns, where the only non-zero entry per row contains the Tout value in 22 | the column corresponding to the hour of day it was recorded 23 | } 24 | \description{ 25 | Split a column of data by membership in discrete groups (as with factor levels) 26 | data into multiple columns containing just the values corresponding to the membership 27 | indicator associated with the column and zeros otherwise. 28 | } 29 | 30 | -------------------------------------------------------------------------------- /man/regressorDF.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-regression.R 3 | \name{regressorDF} 4 | \alias{regressorDF} 5 | \title{data.frame formatting of \code{\link{MeterDataClass}} data} 6 | \usage{ 7 | regressorDF(meterData, norm = FALSE, rm.na = FALSE) 8 | } 9 | \arguments{ 10 | \item{meterData}{The meter data class to be converted} 11 | } 12 | \description{ 13 | Given a MeterDataClass instance (meterData), regressorDF returns a data.frame consisting of a standard set of 14 | regressor columns suitable for passing into a call to lm. 15 | } 16 | 17 | -------------------------------------------------------------------------------- /man/rm.col.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-base.R 3 | \name{rm.col} 4 | \alias{rm.col} 5 | \title{remove named, index, or logical columns, if present, from a data.frame} 6 | \usage{ 7 | rm.col(df, cols) 8 | } 9 | 10 | -------------------------------------------------------------------------------- /man/run.query.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{run.query} 4 | \alias{run.query} 5 | \title{Utility function that runs and returns an arbitrary database query} 6 | \usage{ 7 | run.query(query, cfg, cacheDir = NULL, cacheFile = NULL, forceRefresh = F, 8 | debug = F) 9 | } 10 | \arguments{ 11 | \item{query}{a string representation of the SLQ query to be run.} 12 | 13 | \item{cfg}{a named list of connection configuration options, often loaded from a flat configuration file using \code{\link{dbCfg}}.} 14 | 15 | \item{cacheDir}{an optional directory path to a location where query results can be cached as RData for future use without querying the database again.} 16 | 17 | \item{cacheFile}{an optional file name that signals that the results of the query should be cached as RData using the passed cacheFile name. 18 | Note that if there already exists an RData file with that name in the cache directory, the contents will be returned without running the query. 19 | This it is imerative that the \code{cacheFile} names are unique to each unique query that should be run.} 20 | 21 | \item{forceRefresh}{an optional parameter for use when a \code{cacheFile} has been specified. If True (default is False), the query is run against 22 | the database with results overwriting any existing cached data. This is similar behavior to entering the cache directory and erasing the 23 | cached data before calling the \code{run.query} function.} 24 | 25 | \item{debug}{print out diagnostic information about the cache path, file, and status, along with the full text of any SQL query executed.} 26 | } 27 | \details{ 28 | \code{run.query} is a utility function that is very often used in \code{DataSource} implementations. It automatically connects to a database 29 | with configuration that can be read from a config file, runs queries, and returns results as data.frames. It also supports caching of query results 30 | in a query cache directory, using passed cacheFile names, which must be carefully managed by the data source author to ensure that the cacheFile names 31 | are truly unique to each unique query made. For example, the cache file for query to load meter data for an individual meter would need to include 32 | the meter's id (or similar) to ensure that it isn't mistaken for data from another meter already cached. The purpose of all this cacheing logic is, 33 | of course, to improve the performance of data retrieval for large sets of data where query times can significantly impact performance. Thus it is often good 34 | practice to load and cache data in larger chunks than individual meter data. 35 | } 36 | \seealso{ 37 | \code{\link{dbDfg}}, \code{\link{conf.dbCon}} 38 | } 39 | 40 | -------------------------------------------------------------------------------- /man/runDateFilterIfNeeded.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/iterator.R 3 | \name{runDateFilterIfNeeded} 4 | \alias{runDateFilterIfNeeded} 5 | \title{Run date filters on raw meter data} 6 | \usage{ 7 | runDateFilterIfNeeded(ctx) 8 | } 9 | \arguments{ 10 | \item{ctx}{Context object that is an R environment with named parameters} 11 | } 12 | \description{ 13 | Runs \code{applyDateFilters} if the appropriate values are found in the ctx and the data is not yet 14 | flagged as filtered. This function retrieves data from \code{ctx$RAW_DATA} and writes its results to the same field. 15 | } 16 | 17 | -------------------------------------------------------------------------------- /man/sanityCheckDataSource.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dataSource.R 3 | \name{sanityCheckDataSource} 4 | \alias{sanityCheckDataSource} 5 | \title{Function to exercise the core features and calls of a DataSource} 6 | \usage{ 7 | sanityCheckDataSource(DATA_SOURCE, useCache = FALSE) 8 | } 9 | \arguments{ 10 | \item{DATA_SOURCE}{The DataSource to be tested. In VISDOM, there is a convention to put the 11 | active \code{DataSource} into the global environment with the name DATA_SOURCE, so this code follows that 12 | naming convention, even though you are passing in the data source.} 13 | 14 | \item{useCache}{Boolean to control whether or not the DataSource should rely upon its data cache to 15 | retrieve data. If True and called with an empty cache, the function can be used to pre-populate the 16 | caches with id lists, geo code, meter data by geo code, and weather data by geo code.} 17 | } 18 | \description{ 19 | This function exercises the core functions of a \code{\link{DataSource}}, allowing DataSource authors to 20 | check if their DataSource meets the minimal obligations of data retrieval and formatting. 21 | It can also be used to pre-populate data caches before a feature run. 22 | } 23 | \details{ 24 | While these checks are not comprehensive, they represent the main functions exercised during a feature 25 | extraction run. DataSource authors should feel free to add functions to their DataSource and alter 26 | existing function signatures to meet their individual requirements. As long as this function still runs 27 | without errors, generic iterator functions should still work. The iterator relies on the ability to break 28 | ids and meter data down to geo code level sub-groups, to request all weather and meter data for a zip code, 29 | and to be able to instantiate a MeterDataClass object with the returned meter data and weather data. 30 | 31 | If this function no longer runs on your \code{DataSource}, then advanced users can write their own 32 | iteration methods that break down the feature extraction problem differently. 33 | } 34 | \seealso{ 35 | \code{\link{DataSource}} 36 | } 37 | 38 | -------------------------------------------------------------------------------- /man/showCons.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-dbUtil.R 3 | \name{showCons} 4 | \alias{showCons} 5 | \title{List all open database connections} 6 | \usage{ 7 | showCons(cfg) 8 | } 9 | \arguments{ 10 | \item{cfg}{parsed connection configuration data} 11 | } 12 | \description{ 13 | Use this to see whether too many connections are open 14 | } 15 | 16 | -------------------------------------------------------------------------------- /man/spread.boot.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{spread.boot} 4 | \alias{spread.boot} 5 | \title{bootstrap spread scores across several runs of the ctree algorithm} 6 | \usage{ 7 | spread.boot(df, df.ct, nRun = 100, nSample = NULL, responseVar, ignoreCols) 8 | } 9 | \arguments{ 10 | \item{df}{data.frame with response variable to be explained and all explanatory variables to be used in ctree classificaiton} 11 | 12 | \item{df.ct}{ctree already trained on data, that is used to classify the data points in each sub sample.} 13 | 14 | \item{nRun}{number of bootstrapping runs to perform} 15 | 16 | \item{nSample}{number of samples (with replacement) for each run} 17 | 18 | \item{responseVar}{the name of the variable that the ctree is trying to explain} 19 | 20 | \item{ignoreCols}{an optional list of columns in df, but that should be excluded from the ctree modeling} 21 | } 22 | \description{ 23 | This function repeatedly randomly sub-samples the passed data.frame to provide points for classificaiton 24 | by the passed ctree model and calculates the resulting spread score for classifications of each subset 25 | and returns a vector of the resulting scores. It is useful for determining the robustness of the spread score 26 | for a particular data set. 27 | } 28 | 29 | -------------------------------------------------------------------------------- /man/spreadScore.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-tree.R 3 | \name{spreadScore} 4 | \alias{spreadScore} 5 | \title{given a ctree model, compute its spread score} 6 | \usage{ 7 | spreadScore(df, df.ct, nSample = NULL, responseVar, ignoreCols = c()) 8 | } 9 | \arguments{ 10 | \item{df}{data.frame with response variable to be explained and all explanatory variables to be used in ctree classificaiton} 11 | 12 | \item{df.ct}{ctree already trained on data, that is used to classify the data points in each sub sample.} 13 | 14 | \item{nSample}{number of samples (with replacement) to use for the calculation} 15 | 16 | \item{responseVar}{the name of the variable that the ctree is trying to explain} 17 | 18 | \item{ignoreCols}{an optional list of columns in df, but that should be excluded from the ctree modeling} 19 | } 20 | \description{ 21 | This function computes the "spread score" for a sample of data, given a ctree explaining a binary variable. 22 | New data samples are "predicted" into their ctree nodes and the probability of a 'True' response value for 23 | their members is fed into a weighted average of the absolute distance from the sample mean across all dat asamples. 24 | The higher the score, the better the model has done at classifying people with divergent behaviors (higher or loewr than average). 25 | } 26 | 27 | -------------------------------------------------------------------------------- /man/visdom.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/visdom.R 3 | \docType{package} 4 | \name{visdom} 5 | \alias{visdom} 6 | \alias{visdom-package} 7 | \title{Smart Meter data analysis tools for R} 8 | \description{ 9 | Smart Meter data analysis tools for R 10 | } 11 | \section{Core functions}{ 12 | 13 | 14 | VISDOM relies most heavily on the following core functions. See the example vignettes for examples of their usage: 15 | 16 | \itemize{ 17 | \item \code{\link{DataSource}}: An S3 class that implements the set of standard data source function for your data. 18 | 19 | 20 | \item \code{\link[visdom]{MeterDataClass}}: S3 class that holds meter data in both vector time series and daily matrix formats, associated weather data, and several supporting functions. 21 | 22 | 23 | \item \code{\link{TestData}}: Example data source S3 class that implements all required data source functions and generates random synthetic data for use in testing and examples. 24 | 25 | 26 | \item \code{\link[visdom]{WeatherClass}}: S3 class that holds weather data and related functions. 27 | 28 | 29 | \item \code{\link{basicFeatures}}: Function that implements a full suite of "basic" feature calculations, which include annual, seasonal, monthly, hour of day averages and variances, and other simple statistics, like simple correlation between outside temperature and consumption. 30 | 31 | 32 | \item \code{\link{dbCfg}}: S3 class that can parse a database config from text file. This is used by the util-dbUtil.R file to connect to the configired database. 33 | 34 | 35 | \item \code{\link{run.query}}: Main function used to run SQL queries to load and cache data. 36 | 37 | 38 | \item \code{\link{iterator.callAllFromCtx}}: Function that iterates through all feature algorithms listed in the configuration ctx environment and returns a concatenated named list of all resulting features for a given MeterDataClass instance. 39 | 40 | 41 | \item \code{\link{iterator.iterateMeters}}: Function that iterates through all passed meter ids to instantiate MEterDataClass for each and call one or more feature extraction functions on each. This requires a properly configured data source, database connection (if applicable) and is further configured using fields in the ctx context object. 42 | 43 | 44 | \item \code{\link{iterator.iterateZip}}: Function that iterates through all passed zip codes, looks up local weather data (once) and looks up the list of meter ids for each and calls iterator.iterateMeters with those meter ids and the per-loaded weather data. This runs faster than calling ids individually, which can load similar weather data over and over. 45 | 46 | 47 | \item \code{\link{iterator.runMeter}}: Utility function that runs the configured set of featture functions on a single passed meter. This is useful for testing and feature development, but also as the function called by parallelizable methods like apply, and the *ply function of plyr. 48 | 49 | 50 | \item \code{\link{iterator.todf}}: The iterator.iterate* functions return lists of feature lists, indexed by meter id. This function converts all the scalar features in this data structure into a single data.frame, one row per meter id. 51 | 52 | } 53 | } 54 | 55 | -------------------------------------------------------------------------------- /man/writeCSVData.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{writeCSVData} 4 | \alias{writeCSVData} 5 | \title{Write feature data frame to a csv file} 6 | \usage{ 7 | writeCSVData(data, fName, label = NA, filePath = NA) 8 | } 9 | \arguments{ 10 | \item{data}{The feature data frame to be written} 11 | 12 | \item{fName}{The name of the csv file to write the data to} 13 | 14 | \item{label}{Unused, but present for compatibility with other write* fucntions} 15 | 16 | \item{filePath}{optional path to the location where exported files should be written (if applicable). Default is \code{getwd()}} 17 | } 18 | \description{ 19 | Write feature data frame to a csv file 20 | } 21 | 22 | -------------------------------------------------------------------------------- /man/writeDatabaseData.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{writeDatabaseData} 4 | \alias{writeDatabaseData} 5 | \title{Write feature data frame to a database} 6 | \usage{ 7 | writeDatabaseData(data, name = NULL, label = NULL, conn, overwrite = TRUE, 8 | runConfig) 9 | } 10 | \arguments{ 11 | \item{data}{The feature data frame to be written} 12 | 13 | \item{name}{Unused, but present for compatibility with other write* fucntions} 14 | 15 | \item{label}{Unused, but present for compatibility with other write* fucntions} 16 | 17 | \item{conn}{A DBI dbConnection object to the database that will host the table} 18 | 19 | \item{overwrite}{Boolean indicator for whether the data written should overwrite any existing table or append it} 20 | 21 | \item{runConfig}{Path to a run configuration file with names and descriptions 22 | of the feature set and run. See 23 | `inst/feature_set_run_conf/exampl_feature_set.conf` for an example.} 24 | } 25 | \description{ 26 | Write feature data frame to a database using a \code{\link{DBI::dbWriteTable}} call 27 | } 28 | 29 | -------------------------------------------------------------------------------- /man/writeH5Data.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/util-export.R 3 | \name{writeH5Data} 4 | \alias{writeH5Data} 5 | \title{Write feature data frame to an hdf5 file} 6 | \usage{ 7 | writeH5Data(data, fName, label, filePath = NA) 8 | } 9 | \arguments{ 10 | \item{data}{The feature data frame to be written} 11 | 12 | \item{fName}{The name of the hdf5 formatted file to write the data to} 13 | 14 | \item{label}{The name of the data table within the hdf5 file} 15 | 16 | \item{filePath}{optional path to the location where exported files should be written (if applicable). Default is \code{getwd()}} 17 | } 18 | \description{ 19 | Write feature data frame to an hdf5 file 20 | } 21 | 22 | -------------------------------------------------------------------------------- /tests/testthat.R: -------------------------------------------------------------------------------- 1 | library(testthat) 2 | library(visdom) 3 | 4 | test_check("VISDOM") 5 | -------------------------------------------------------------------------------- /tests/testthat/test_util-census.R: -------------------------------------------------------------------------------- 1 | library(visdom) 2 | library(acs) 3 | context("Integration test for merging census dataframes") 4 | 5 | check_acs_key = function() { 6 | if( ! acs::api.key.exists() ) { 7 | skip(paste( 8 | "A census API key has not been registered with the acs package. Skipping test.", 9 | "To install a census API key, go to http://api.census.gov/data/key_signup.html", 10 | "To register a census API key, do: acs::api.key.install(key='PASTE_YOUR_KEY_HERE')")) 11 | } 12 | } 13 | 14 | test_that("acs downloading & mergeCensus()", { 15 | check_acs_key() 16 | geography = c(acs::geo.make(zip.code='94709'), acs::geo.make(zip.code='94710')) 17 | censusStats = loadACS(geography = geography) 18 | df = data.frame("ZCTA" = as.numeric(censusStats$ZCTA), "fake_column" = as.numeric(censusStats$mean_fam_income) / 2.0) 19 | df = mergeCensus(df, censusStats=censusStats) 20 | expect_equal(df$fake_column, as.numeric(df$mean_fam_income) / 2.0) 21 | }) 22 | 23 | test_that("acs cache speedup", { 24 | check_acs_key() 25 | old_cache_path=R.cache::getCacheRootPath() 26 | cache_dir = file.path(tempdir(), "visdom_test_R.cache") 27 | dir.create(cache_dir, recursive=T, showWarnings=F) 28 | R.cache::setCacheRootPath(cache_dir) 29 | geography = c(acs::geo.make(zip.code='94709'), acs::geo.make(zip.code='94710')) 30 | time1 = system.time(loadACS(geography=geography)) 31 | time2 = system.time(loadACS(geography=geography)) 32 | expect_gt(time1['elapsed'], time2['elapsed']) 33 | # Clean up 34 | unlink("cache_dir", recursive=TRUE) 35 | R.cache::setCacheRootPath(old_cache_path) 36 | }) 37 | -------------------------------------------------------------------------------- /vignettes/FAQ.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Frequently Asked Questions" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{FAQ} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | This FAQ will grow with examples of how to resolve common issues or perform common tasks. 13 | 14 | *** 15 | 16 | **Q. When authoring a datasource customerID is my original customerID, ID is internal id for VISDOM. Should they always be in 'cust_id' and 'meter_id' format fount in TestDataSource? And what is the actual difference? Does this mean you can have different ID’s (for example different smart meters) per customer? Or how do I interpret these differences?** 17 | 18 | A. As the author of the data source, you get to choose what set of ids you want to key off of. You just need to be consistent. So if you are using meter ids, you know that the data is associated with the meter over time regardless of who was occupying the house, but if it is customer ids then the data will be some span of time from a specific meter. As long as the correct ids are associated with the meter data you want to use (i.e. getIds returns the ids that you can use to query for specific meter data), it will work. The actual values of the ids can be whatever you like, but you should ensure that they are character strings even if they are plausibly numerical. We have seen too many cases where there are leading zeros that get trimmed or the "number" of the id is too long for R's ints and the ids don't properly match up anymore. 19 | 20 | *** 21 | 22 | **Q. Time formats. How does visdom deal with different time resolutions between readings (1 min vs 1 h)?** 23 | 24 | A. VISDOM currently supports 15 minute and 1 hr readings, in the format of either 24 or 96 columns of data per day. Other timing is technically possible, but the official recommendation is using your data source to sample it down to one of these intervals for now. 25 | 26 | *** 27 | 28 | **Q. What timezone to use in input data?** 29 | 30 | A. Just be consistent. We usually try for the local timezone of the data, so that it is easier to have an intuitive feel for what you are seeing (i.e. 8am is breakfast time) and so it is more possible to sync with local weather data. 31 | 32 | *** 33 | 34 | **Q. How does VISDOM account for daylight saving time?** 35 | 36 | A. With the 24/96 column formats it ignores the 25th hour of fall back and treats the 24th hour of spring forward as missing data. Note that R is pretty picky about DST and crabby about dates in general. If you see unusual shifts in data timing around March or November in your plots, chances are good that DST is not being handled well. 37 | 38 | *** 39 | 40 | **Q. In the standard meter data format what hour does each column correspond to?** 41 | 42 | A. Your column headers don't have to match these, but referring to the TestData output, the first column, H1, is 12-1am (i.e. the average of consumption up to but excluding 1am) so H10 is 9-10am, and h24 is 11pm-12am. 43 | 44 | *** 45 | 46 | **Q. If we want to compare different datasets (for example conventional and high performance buildings), is there currently a way to assign and compare different groups of customers/buildings?** 47 | 48 | A. There isn't a built in idea of two groups, but if your groups are in the same data set, you can just add a custom feature that return each customer's group assignment as a feature and then you will be able to slice and dice later. If they are literally from different data sets, you will compute their features separately and maybe combine the feature data frames in R using \code{rbind()}. assuming you have an identical set features for both. Or you can use \code{plyr::rbind.fill()} to ensure that divergent feature sets are matched up. 49 | 50 | *** 51 | 52 | **Q. Data preparation: Are missing values as NA or 0?** 53 | 54 | A. Missing values should be NA's. 0 is (can be) a legitimate reading. 55 | 56 | *** 57 | 58 | **Q. What method to impute missing values (built in or in preprocess)?** 59 | 60 | A. That is up to you - if you want, your data source can impute values on the fly before returning meter data or you can do a batch process up front and write the results to a new database table that you will then read from with your data source. The latter will be preferable if you are worried CPU time during feature calculations, but this won't be a concern unless you have many ids worth of data. 61 | 62 | *** 63 | 64 | **Q. For what size of total dataset is a database recommended or required?** 65 | 66 | A. If you hoping to load data directly from CSVs or RData or something, you are only limited by the ram of your computer, so you should just try it and see. Thousands of customers should be no problem to load into memory together. 67 | 68 | *** 69 | 70 | **Q. Load Shape Clustering: Is the clustering profile library rich enough to expect it to perform well on customer data that includes also industrial customers and streetlighting.** 71 | 72 | A. First of all, the load shape clustering algorithms are protected by a patent and are therefore available separately from the rest of the VISDOM code. You need to ask for permission to work with that code, but non-commercial licensing is free. As for separating data for clustering, this depends on your intended use. Typically you would separate residential, commercial, industry, etc. and fit clusters separately. Even within commercial there is so much diversity of use that it can be good to do separate fits for each sub type (i.e. NAICS or other codes). 73 | 74 | *** 75 | 76 | **Q. Can we filter out different type of consumers (like residentials, street light, commercial buildings, etc.)** 77 | 78 | A. Yes. You can add an additional parameter to your datasource functions that return the lists of ids and meter data to restrict the ids to the ones you are interested in. i.e. if you implement the DATA_SOURCE functions to respond to such a parameter, you can call the functions with that parameter. 79 | 80 | *** 81 | 82 | **Q. DATA_CACHE: Our data is not in a database but loaded in RData files. Is there then still need to use the Cache functionality?** 83 | 84 | A. No the cache isn't necessary. In fact it is pretty tightly integrated into the database access code, so no DB means you probably don't need a cache. Also, the cache is just RData files, so you effectively are already using "cached" data... 85 | 86 | *** 87 | 88 | -------------------------------------------------------------------------------- /vignettes/advanced_usage.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Advanced Usage" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Advanced Usage} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | TBD. This will contain examples of advanced usage options once the basic documentation is completed and stable. 13 | -------------------------------------------------------------------------------- /vignettes/authoring_data_source.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Authoring Data Sources" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Authoring Data Sources} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | #Implementing your own data source 13 | 14 | In VISDOM, a data source is an adapter between your customer and meter data and the internal data formats used by VISDOM functions. To use VISDOM with your own data, you will need to author a DataSource object that maps between the formatting of your data and the data structures used by VISDOM by implementing all the relevant functions stubbed out by `DataSource()` in `util-dataSource.R`. This is the key step and main prerequisite for using VISDOM. You will typically need to set up data access (i.e. to a SQL database if applicable - see `util-dbUtil.R` - or figure out how you will be loading your data from disk or elsewhere), and write the code to perform the queries or other data access steps as appropriate to load, format, and return your data in the VISDOM standard format expected to come out of a data source. You can see the DataSource implemented for testing purposes in the file `testDataSource.R` in the R directory of the package. 15 | 16 | Setting the global variable `DATA_SOURCE = YourDataSource()` configures your data source for use by VISDOM (i.e. assign it to the global variable DATA_SOURCE). 17 | 18 | See the entry for the data parameter in the help for MeterDataClass: 19 | 20 | ```{r eval=F} 21 | library(visdom) 22 | ?MeterDataClass 23 | ``` 24 | 25 | 26 | The MeterDataClass object also does the weather data alignment. It matches and interpolates available weather data (from DATA_SOURCE$getWeatherData() ) to the dates associated with the meter data from getAllData. 27 | 28 | ##Data formats 29 | 30 | ```{r setup} 31 | library(visdom) 32 | DATA_SOURCE = TestData(100) 33 | ``` 34 | 35 | 1. Meter data for a single customer. Note that id (i.e. the VISDOM internal identifier for the meter), geocode, and dates (of type Date - without time) are required, as are 24 hourly or 96 15 minute meter observations per day. customerID (i.e. the owner of the meter) and other fields can be added, but are not required. 36 | 37 | ```{r customer_data_sample} 38 | custdata = DATA_SOURCE$getMeterData(id=1) 39 | head(custdata,2) 40 | dim(custdata) 41 | ``` 42 | 43 | 2. Meter data from multiple customers. 44 | 45 | ```{r meter_data_sample} 46 | # this is all data for a given geocode (i.e. zip code) 47 | geosample = DATA_SOURCE$getAllData( geocode='94305' ) 48 | head(geosample,2) 49 | dim(geosample) 50 | unique(geosample$id) 51 | ``` 52 | 53 | 3. Weather data: dates (required of type POSIXct, with time at whatever intervals observations are available in, ideally with hourly or less intervals), temperaturef (required), pressure, hourlyprecip, dewpoint. 54 | 55 | ```{r weather_attributes} 56 | weather = DATA_SOURCE$getWeatherData(geocode='94305') 57 | head(weather,2) 58 | dim(weather) 59 | ``` 60 | 61 | 4. Misc capabilities 62 | 63 | ```{r geo_attributes} 64 | DATA_SOURCE$getGeoForId('meter_1') 65 | class(DATA_SOURCE$getGeoForId('meter_1')) 66 | ``` 67 | 68 | ```{r ids_attributes} 69 | length(DATA_SOURCE$getIds()) # all the meter ids tracked by the data source 70 | class(DATA_SOURCE$getIds()) 71 | head(DATA_SOURCE$getIds()) 72 | ``` 73 | 74 | ##Testing your data source 75 | 76 | You can call `DATA_SOURCE$getMeterDataClass(id=123)` on your DataSource, replacing 123 with an appropriate id from your data set (i.e. using the provided default implementation of `getMeterDataClass()` and it will hit your data source for all relevant data and instantiate a MeterDataClass object with associated weather data and WeatherData class. Until that call succeeds, you will be getting errors that related to deficiencies in your DataSource, so it is a good guide to what else you need to implement. 77 | 78 | You can also exercise your data source with the function `sanityCheckDataSource()` 79 | 80 | ```{r sanityCheckDataSource, fig.width=6, fig.height=6} 81 | 82 | # runs a standard set of data checks 83 | # with chatty output 84 | sanityCheckDataSource(DATA_SOURCE) 85 | 86 | ``` 87 | 88 | 89 | 90 | Finally, you can probe specific data source functions with test code like this: 91 | 92 | ```{r exampleTestCode, eval=F} 93 | library(visdom) 94 | 95 | DATA_SOURCE = YourDataSource() 96 | 97 | # if your data source in configured to access a database 98 | DATA_SOURCE$run.query('select count(*) from meter_15min_user') 99 | 100 | DATA_SOURCE$run.query('select distinct zip5 from weather') 101 | 102 | # most important functions: 103 | # ------------------------- 104 | 105 | # primary geographic codes associated with meters, 106 | # typically a list of zip codes or census blocks 107 | geos = DATA_SOURCE$getGeocodes() # all geographic regions 108 | ids = DATA_SOURCE$getIds() # all known ids 109 | DATA_SOURCE$getIds(geos[1]) # all ids from the first geocoded location 110 | 111 | DATA_SOURCE$getAllData(geos[1]) # all meter data from the first geocoded region 112 | DATA_SOURCE$getGeoForId(ids[1]) # get the geo code for a specific meterId 113 | 114 | DATA_SOURCE$getMeterData(ids[1]) # returns meter data for a specific meterId 115 | md = DATA_SOURCE$getMeterDataClass(ids[1]) # returns a MeterDAtaClass object, with weather data, etc. 116 | 117 | DATA_SOURCE$getWeatherData(geos[1]) # data frame of tabular weather data for a geo location 118 | 119 | # these use DATA_SOURCE internally 120 | w = WeatherClass(geos[1],doMeans=F,useCache=F,doSG=F) 121 | md = MeterDataClass(ids[1],useCache=F) 122 | plot(md) 123 | 124 | # functions of secondary importance (you will likely know if you need these) 125 | # --------------------------------- 126 | DATA_SOURCE$getGeoCounts() 127 | # these can include census statistics and other suppliments to customer meter data 128 | DATA_SOURCE$getGeoMetaData(geos[1]) 129 | 130 | # gas data is optional 131 | DATA_SOURCE$getAllGasData() 132 | DATA_SOURCE$getGasMeterData(geo=geos[1]) 133 | DATA_SOURCE$getGasMeterData(id=ids[1]) 134 | 135 | ``` 136 | -------------------------------------------------------------------------------- /vignettes/bootstrap_devel_environment.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Developing the VISDOM Module" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Developing VISDOM} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | First ensure that you have cloned the repository into working version only your local machine, choosing one of: 13 | 14 | ```{bash eval=F} 15 | cd /dev 16 | git clone git@github.com:convergenceda/visdom.git 17 | git clone https://github.com/convergenceda/visdom.git 18 | ``` 19 | 20 | Then using that location as as your working directory (here we assume `/dev/visdom`), load requirements for package development and install from source. 21 | 22 | #Load requirements 23 | 24 | ```{r eval=F} 25 | setwd('/dev/visdom') 26 | 27 | install.packages(c("devtools", "roxygen2", "testthat", "knitr")) 28 | ``` 29 | 30 | version check 31 | 32 | ```{r eval=F} 33 | rstudioapi::isAvailable("0.99.149") 34 | ``` 35 | 36 | for some reason, withr (a devtools dependency) will not install for RRO 3.2.2 with message that there is no 3.2.2 version of withr 37 | this is probably an ephemeral issue, but install from source for now 38 | 39 | ```{r eval=F} 40 | devtools::install_github("jimhester/withr") 41 | ``` 42 | 43 | install Wickham's latest/src devtool support. Note that devtools can't install itself on Windows (!!) 44 | https://github.com/hadley/devtools/issues/503 explains. 45 | 46 | ```{r eval=F} 47 | devtools::install_github("hadley/devtools") 48 | ``` 49 | 50 | #Install VISDOM as a local package from your source 51 | 52 | Ensure that your getwd() is the visdom source dir and then call: 53 | 54 | ```{r eval=F} 55 | devtools::document() 56 | devtools::build_vignettes() 57 | ?visdom 58 | ?MeterDataClass 59 | ?WeatherDataClass 60 | 61 | ``` 62 | 63 | Followed by installing the package from source, which should resolve and install all the package dependencies, 64 | with some exceptions, like for HDF5 support. 65 | 66 | ```{r eval=F} 67 | devtools::install(build_vignettes = T) 68 | ``` 69 | 70 | 71 | Test your install with a fresh R session: 72 | ```{r eval=F} 73 | library(visdom) 74 | ?visdom 75 | browseVignettes(package='visdom') 76 | ``` 77 | -------------------------------------------------------------------------------- /vignettes/customer_data_objects.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Working With Meter Data" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Working With Meter Data} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | Documentation on loading MeterDataClass objects and exploring their data, functions and built-in features. 13 | -------------------------------------------------------------------------------- /vignettes/example_baseline.R: -------------------------------------------------------------------------------- 1 | #VISDOM_R_PATH = 'C:/dev/VISDOM-R/eneRgy/R' 2 | 3 | #DATASOURCE_PATH = '../../../SSSL_datasources/' 4 | 5 | #setwd( file.path(VISDOM_R_PATH) ) 6 | 7 | source('util-timer.R') # adds tic() and toc() functions 8 | source('iterator.R') # functions to iterate through collections of meters 9 | source('classes-customer.R') # loads MeterDataClass and WeatherDataClass 10 | 11 | 12 | #source(file.path(DATASOURCE_PATH,'/pgeResDataAccess.R')) # provides data source implementation for pge_res data 13 | 14 | #QUERY_CACHE = 'e:/dev/pge_collab/EnergyAnalytics/batch/QUERY_CACHE_STANFORD/' 15 | #DATA_SOURCE = PgeResData(dbConfig=file.path(DATASOURCE_PATH,'pge_res_DB.cfg'), queryCache=QUERY_CACHE) # Use PGE res data for analysis 16 | #DATA_SOURCE = PgeSmbData(dbConfig=file.path(DATASOURCE_PATH,'pge_smb_DB.cfg'), queryCache=QUERY_CACHE) # Use PGE res data for analysis 17 | 18 | 19 | source('baseline.R') # baselineing functions 20 | 21 | id = 34543252435432 22 | zip = 93304 23 | startTime = as.POSIXct('2013-01-01 20:00') 24 | endTime = as.POSIXct('2013-01-01 23:00') 25 | daysBefore = 100 26 | daysAfter = 10 27 | 28 | # 1. Load a MeterDataClass instance 29 | meterData = MeterDataClass(820735863,94610,useCache=T,doSG=F); plot(meterData,type='hourly',estimates=hourlyChangePoint(regressorDF(r),as.list(1:24),reweight=F)) # heat, no cooling 30 | meterData = MeterDataClass(553991005,93304,useCache=T,doSG=F); plot(meterData,type='hourly',estimates=hourlyChangePoint(regressorDF(r),as.list(1:24),reweight=F)) # very clear cooling 24x7 31 | 32 | # 2. Pass meter data into baseline function with linear regression 33 | result.regression = baseline.regression(meterData,startTime,endTime,daysBefore,daysAfter) 34 | 35 | # 3. Pass meter data into baseline function with averaging over past 10 days 36 | result.average = baseline.average(meterData,startTime,endTime,daysBefore,daysAfter) 37 | 38 | # 4. Pass meter data into baseline function with gaussian process 39 | result.gp = baseline.gp(meterData,startTime,endTime,daysBefore,daysAfter) 40 | 41 | # 5. compare and plot the results.... 42 | 43 | 44 | 45 | 46 | -------------------------------------------------------------------------------- /vignettes/example_census.R: -------------------------------------------------------------------------------- 1 | library(visdom) 2 | # use the local census data path 3 | a = data.frame(zippy=c(94611,93304)) # create a data frame with a column of zip codes. 4 | aPlus = mergeCensus(a,zipCol = "zippy") # uses zip column (aka "zippy" in this example) 5 | # to match ZCTA from census and return all the 6 | # ACS 2011 stats for the zips in question 7 | names(aPlus) # look at all the census data columns added 8 | aPlus # print everything out 9 | 10 | # see util-census.R for mode useful census data features. -------------------------------------------------------------------------------- /vignettes/example_iterator_usage.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Working With Samples from Many Meters" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Working With Samples from Many Meters} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | Examples of iterator function usage TBD. Please see vignette on feature extraction for a working example. 13 | ```{r eval=F} 14 | 15 | 16 | ``` 17 | -------------------------------------------------------------------------------- /vignettes/install_visdom.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Installing VISDOM" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Installing VISDOM} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | #Simplest approach 13 | 14 | There is support in R for installing and updating packages directly from their version control repositories. If you plan to use VISDOM without altering any of its source code, this is the preferred method of installation. 15 | 16 | ##Install VISDOM directly from GitHub source 17 | 18 | One useful feature of R documentation is its ability to present formatted examples of code usage and outputs, called vignettes. When installing from github, these vignettes are not built by default, largely due to their additional R module dependencies (and their runtimes in some cases). Still, it is highly recommended that you do build and refer to the vignettes. Note that any missing packages that R warns you about can likely be installed using the `install.packages` command. For example, if a package called `plyr` is not installed and you try to run code that depends on it, you will get a message like: `there is no package called ‘plyr’`. You can address this by calling `install.packages('plyr')` and repeat until your dependencies are all loading correctly. 19 | 20 | ```{r eval=F} 21 | install.packages(c("devtools")) 22 | 23 | # build_vignettes=T will build vignettes that serve as examples of usage 24 | # however, this will invoke other dependencies that might complicate things. 25 | # It can be set to F to just get the core code installed, but much of the documentation 26 | # effort to date has been in the form of vignettes. 27 | devtools::install_github("convergenceda/visdom", build_vignettes=T ) 28 | ``` 29 | 30 | ##Install VISDOM directly from local source 31 | 32 | If you are planning to read through, experiment with, or update the VISDOM source code itself, you will want a local copy of the repository on your machine. 33 | 34 | First ensure that you have cloned the repository into working version only your local machine, choosing one of: 35 | 36 | ```{bash eval=F} 37 | cd /dev 38 | git clone git@github.com:convergenceda/visdom.git 39 | git clone https://github.com/convergenceda/visdom.git 40 | ``` 41 | 42 | If you are working within a corporate firewall, it may be necessary to use a proxy server account to connect to GitHub. Here you will need to replace the user and pass with your username and password and `proxy.server` with either the name or ip address of your proxy server. The port `8080` may also be different for your specific configuration. 43 | 44 | ```{bash eval=F} 45 | git config --global http.proxy http://user:pass@proxy.server:8080 46 | git config --global https.proxy https://user:pass@proxy.server:8080 47 | ``` 48 | 49 | Then using that location as as your working directory (here we assume `/dev/visdom`), load requirements for package development and install from source. 50 | 51 | 52 | ```{r eval=F} 53 | 54 | install.packages(c("devtools")) 55 | 56 | setwd('/dev/visdom') 57 | 58 | devtools::install(build_vignettes = T) 59 | 60 | ``` 61 | 62 | ##Confirming that the package and documentation is in place 63 | Now check that you can load VISDOM and use it. 64 | 65 | ```{r eval=F} 66 | library(visdom) 67 | ``` 68 | 69 | Browse through the available vignettes. 70 | 71 | ```{r eval=F} 72 | # if you built them above, you can browse well formatted 73 | # code vignettes that provide example usage. 74 | browseVignettes('visdom') 75 | ``` 76 | 77 | Or the original/old school way. 78 | ```{r eval=F} 79 | # to list 80 | vignette(package='visdom') 81 | # to display a specific one as help 82 | vignette('example_feature_extraction',package='visdom') 83 | ``` 84 | -------------------------------------------------------------------------------- /vignettes/weather_data_objects.rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Working With Weather Data" 3 | author: "Sam Borgeson" 4 | date: "`r Sys.Date()`" 5 | output: rmarkdown::html_vignette 6 | vignette: > 7 | %\VignetteIndexEntry{Working With Weather Data} 8 | %\VignetteEngine{knitr::rmarkdown} 9 | %\VignetteEncoding{UTF-8} 10 | --- 11 | 12 | Examples illustrating how to gather and work with weather data. 13 | --------------------------------------------------------------------------------