├── LICENSE ├── deck ├── AEDES_PROJECT_2019OCTOBER.pptx ├── AEDES_PROJECT_VIDEO_25OCTOBER2019.pptx ├── Dengue Searches for sharing with Doc Ligot.xlsx ├── TEAM_AEDES.jpg ├── aedes_black.png ├── aedes_white.png ├── aedes_yellow.png ├── dengue_data.png ├── dengue_model.png ├── dengue_searches.png ├── maps-js.xlsx └── satellite.png ├── literature ├── 07785420 - Forecasting and Data Visualization of Dengue Spread in PH Visayas.pdf ├── 1630 Dr Rose Nani _MOH - Malaysia e-Dengue Operational Framework.pdf ├── 2018HoetalGDTPhPublished.pdf ├── Advances_in_using_Internet_searches_to_track_dengu.pdf ├── FAPAR_NDWI_92290K.pdf ├── Incubation-Dengue-pone.0050972.pdf ├── Stagnant Water Maps from Satellite Infrared Data.pdf ├── Using satellites to fight dengue » Manila Bulletin Technology.pdf ├── dengue-20191219 - WHO Report.pdf └── s12916-019-1396-4 - spatial mapping for wolbachia.pdf ├── maps ├── cotabato.html ├── dagupan.html ├── iloilo.html ├── mapbox-gl.css ├── mapbox-gl.js ├── qc.html └── tacloban.html ├── model ├── code │ ├── .ipynb_checkpoints │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ ├── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ └── pgm3_Time_Series_Modeling_v2-checkpoint.ipynb │ ├── ARMM │ │ ├── .ipynb_checkpoints │ │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ │ └── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ ├── pgm1_Data_Preparation.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis.ipynb │ │ └── pgm3_Time_Series_Modeling.ipynb │ ├── Eastern Visayas │ │ ├── .ipynb_checkpoints │ │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ │ └── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ ├── pgm1_Data_Preparation.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis.ipynb │ │ └── pgm3_Time_Series_Modeling.ipynb │ ├── NCR │ │ ├── .ipynb_checkpoints │ │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ │ └── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ ├── pgm1_Data_Preparation.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis.ipynb │ │ └── pgm3_Time_Series_Modeling.ipynb │ ├── PH │ │ ├── .ipynb_checkpoints │ │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ │ ├── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ │ ├── pgm3_Time_Series_Modeling_v2-checkpoint.ipynb │ │ │ ├── pgm3_Time_Series_Modeling_v3-checkpoint.ipynb │ │ │ └── pgm3_Time_Series_Modeling_v4-checkpoint.ipynb │ │ ├── pgm1_Data_Preparation.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis.ipynb │ │ └── pgm3_Time_Series_Modeling.ipynb │ └── Western Visayas │ │ ├── .ipynb_checkpoints │ │ ├── pgm1_Data_Preparation-checkpoint.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis-checkpoint.ipynb │ │ └── pgm3_Time_Series_Modeling-checkpoint.ipynb │ │ ├── pgm1_Data_Preparation.ipynb │ │ ├── pgm2_Exploratory_Data_Analysis.ipynb │ │ └── pgm3_Time_Series_Modeling.ipynb ├── datasets │ ├── Dengue_ARMM.xlsx │ ├── Dengue_ARMM_Clean.pickle │ ├── Dengue_ARMM_Diff.pickle │ ├── Dengue_ARMM_Fct.pickle │ ├── Dengue_ARMM_Fct.xlsx │ ├── Dengue_ARMM_Top3_Models.pickle │ ├── Dengue_EastVis_Clean.pickle │ ├── Dengue_EastVis_Diff.pickle │ ├── Dengue_EastVis_Fct.pickle │ ├── Dengue_EastVis_Fct.xlsx │ ├── Dengue_EastVis_Top3_Models.pickle │ ├── Dengue_Eastern_Visayas.xlsx │ ├── Dengue_NCR.xlsx │ ├── Dengue_NCR_Clean.pickle │ ├── Dengue_NCR_Diff.pickle │ ├── Dengue_NCR_Fct.pickle │ ├── Dengue_NCR_Fct.xlsx │ ├── Dengue_NCR_Top3_Models.pickle │ ├── Dengue_PH_Aggregated.xlsx │ ├── Dengue_PH_Clean.pickle │ ├── Dengue_PH_Diff.pickle │ ├── Dengue_PH_Fct.pickle │ ├── Dengue_PH_Fct.xlsx │ ├── Dengue_PH_Fct2.xlsx │ ├── Dengue_PH_Results.xlsx │ ├── Dengue_PH_Top3_Models.pickle │ ├── Dengue_WestVis_Clean.pickle │ ├── Dengue_WestVis_Diff.pickle │ ├── Dengue_WestVis_Fct.pickle │ ├── Dengue_WestVis_Fct.xlsx │ ├── Dengue_WestVis_Top3_Models.pickle │ ├── Dengue_Western_Visayas.xlsx │ ├── Model_Results.xlsx │ └── ~$Dengue_PH_Fct.xlsx ├── dengue_doh │ ├── 2013 Dengue Morbidity Week 52.pdf │ ├── 2014 Dengue Morbidity Week 52_0.pdf │ ├── 2015 Dengue Morbidity Week 52.pdf │ ├── 2016_Dengue_MW1-MW52.pdf │ ├── 2016_Dengue_W29_0.pdf │ ├── 2016_Dengue_W34.pdf │ ├── 2016_Dengue_W36.pdf │ ├── 2016_Dengue_W45.pdf │ ├── 2016_Dengue_W47_1.pdf │ ├── 2017 Dengue Morbidity Week 1-35.pdf │ ├── 2017_Dengue MW 1-MW44_0.pdf │ ├── 2017_Dengue_ MW1-MW26.pdf │ ├── 2017_Dengue_MW1-31.pdf │ ├── 2017_Dengue_MW1-MW18.pdf │ ├── 2017_Dengue_MW1-MW20.pdf │ ├── 2017_Dengue_MW1-MW22.pdf │ ├── 2017_Dengue_MW1-MW39.pdf │ ├── 2017_Dengue_MW1-MW48.pdf │ ├── 2017_Dengue_W1_10.pdf │ ├── 2017_Dengue_W1_11.pdf │ ├── 2017_Dengue_W1_13_0.pdf │ ├── 2017_Dengue_W1_5_0.pdf │ ├── 2017_Dengue_W1_9.pdf │ ├── 2018 Dengue Surveillance Report MW 1-10_5.pdf │ ├── 2018 Dengue Surveillance Report MW1-5_1.pdf │ ├── 2018 Dengue Surveillance Report MW1-6.pdf │ ├── 2018 Dengue Surveillance Report MW1-7.pdf │ ├── 2018 Dengue Surveillance Report MW1-8.pdf │ ├── 2018 Dengue Surveillance Report MW1-9_3.html │ ├── 2018_Monthly_Dengue_Report _N10.pdf │ ├── 2018_Monthly_Dengue_Report _N11.pdf │ ├── 2018_Monthly_Dengue_Report _N12.pdf │ ├── 2019 Dengue Monthly Report No. 7.pdf │ ├── 2019 Dengue Monthly Report No. 8.pdf │ ├── 2019_Monthly_Dengue_Report _N1.pdf │ ├── 2019_Monthly_Dengue_Report _N2.pdf │ ├── DENGUE MW12.pdf │ ├── DENGUE MW13.pdf │ ├── DENGUE MW14.pdf │ ├── DENGUE MW19.pdf │ ├── DENGUE MW21.pdf │ ├── DENGUE MW22.pdf │ ├── DENGUE MW23.pdf │ ├── DENGUE MW25.pdf │ ├── DENGUE MW26.pdf │ ├── DENGUE MW27.pdf │ ├── DENGUE MW28.pdf │ ├── DENGUE MW31.pdf │ ├── DENGUE MW33.pdf │ ├── DENGUE MW4.pdf │ ├── DENGUE MW5.pdf │ ├── DENGUE MW7.pdf │ ├── DENGUE38_compressed.pdf │ ├── DENGUE44_compressed.pdf │ ├── Dengue Monthly Report No. 3.pdf │ ├── Dengue Monthly Report No. 4.pdf │ ├── Dengue Monthly Report No. 5.pdf │ ├── Dengue Monthly Report No. 6.pdf │ ├── Dengue Monthly Report No.6.pdf │ ├── Dengue Monthly Report_MW1-MW30_2018_No.7.pdf │ ├── Dengue Montlhy Report No.4 (April).pdf │ ├── Dengue_Monthly_Report_No.5_2018_May.pdf │ ├── Monthly Dengue Report (MW13).pdf │ ├── dengue27.pdf │ ├── dengue28.pdf │ ├── dengue32.pdf │ ├── dengue33.pdf │ ├── dengueMw10.pdf │ ├── dengueMw13.pdf │ ├── dengueMw15.pdf │ ├── dengueMw18.pdf │ ├── dengueMw21.pdf │ ├── dengueMw23.pdf │ ├── dengueMw24.pdf │ ├── dengueMw26.compressed.pdf │ ├── dengueMw28.compressed.pdf │ ├── dengueMw6.pdf │ └── dengueMw9.pdf ├── google_trends │ ├── Dengue Searches for sharing with Doc Ligot.xlsx │ ├── dengue.csv │ ├── dengue_cure.csv │ ├── dengue_fever.csv │ ├── dengue_medicine.csv │ └── dengue_symptoms.csv ├── readme └── weather │ ├── Ambulong.csv │ ├── NAIA.csv │ ├── Sangley Point.csv │ └── Tayabas.csv ├── readme.md └── satellite ├── Using QGIS to Extract Dengue Hotspots.pdf ├── csv-js ├── cotabato_latlong1.xlsx ├── dagupan_csv.xlsx ├── iloilo_latlong1.xlsx ├── qc_latlong1.xlsx └── tacloban_latlong1.xlsx ├── csvs ├── Dagupan-2019-Dec.csv ├── QC-2018-Nov.csv ├── cotabato_latlong1.csv ├── iloilo_latlong1.csv └── tacloban_latlong1.csv ├── dagupan_csv.csv ├── downloadSatImage.py ├── images ├── GenSan 2020-02-01.jpg ├── Surigao 2019-11-08.jpg ├── Tagbilaran 2020-02-09.jpg ├── Tagum 2020-01-17.jpg ├── fapar-ndwi-formula.png ├── fapar.png ├── ndwi.png ├── qgis-extract-2.png └── qgis-extract.png ├── map_description.docx ├── processSatImage.py └── readme.md /deck/AEDES_PROJECT_2019OCTOBER.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/AEDES_PROJECT_2019OCTOBER.pptx -------------------------------------------------------------------------------- /deck/AEDES_PROJECT_VIDEO_25OCTOBER2019.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/AEDES_PROJECT_VIDEO_25OCTOBER2019.pptx -------------------------------------------------------------------------------- /deck/Dengue Searches for sharing with Doc Ligot.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/Dengue Searches for sharing with Doc Ligot.xlsx -------------------------------------------------------------------------------- /deck/TEAM_AEDES.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/TEAM_AEDES.jpg -------------------------------------------------------------------------------- /deck/aedes_black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/aedes_black.png -------------------------------------------------------------------------------- /deck/aedes_white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/aedes_white.png -------------------------------------------------------------------------------- /deck/aedes_yellow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/aedes_yellow.png -------------------------------------------------------------------------------- /deck/dengue_data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/dengue_data.png -------------------------------------------------------------------------------- /deck/dengue_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/dengue_model.png -------------------------------------------------------------------------------- /deck/dengue_searches.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/dengue_searches.png -------------------------------------------------------------------------------- /deck/maps-js.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/maps-js.xlsx -------------------------------------------------------------------------------- /deck/satellite.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/deck/satellite.png -------------------------------------------------------------------------------- /literature/07785420 - Forecasting and Data Visualization of Dengue Spread in PH Visayas.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/07785420 - Forecasting and Data Visualization of Dengue Spread in PH Visayas.pdf -------------------------------------------------------------------------------- /literature/1630 Dr Rose Nani _MOH - Malaysia e-Dengue Operational Framework.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/1630 Dr Rose Nani _MOH - Malaysia e-Dengue Operational Framework.pdf -------------------------------------------------------------------------------- /literature/2018HoetalGDTPhPublished.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/2018HoetalGDTPhPublished.pdf -------------------------------------------------------------------------------- /literature/Advances_in_using_Internet_searches_to_track_dengu.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/Advances_in_using_Internet_searches_to_track_dengu.pdf -------------------------------------------------------------------------------- /literature/FAPAR_NDWI_92290K.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/FAPAR_NDWI_92290K.pdf -------------------------------------------------------------------------------- /literature/Incubation-Dengue-pone.0050972.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/Incubation-Dengue-pone.0050972.pdf -------------------------------------------------------------------------------- /literature/Stagnant Water Maps from Satellite Infrared Data.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/Stagnant Water Maps from Satellite Infrared Data.pdf -------------------------------------------------------------------------------- /literature/Using satellites to fight dengue » Manila Bulletin Technology.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/Using satellites to fight dengue » Manila Bulletin Technology.pdf -------------------------------------------------------------------------------- /literature/dengue-20191219 - WHO Report.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/dengue-20191219 - WHO Report.pdf -------------------------------------------------------------------------------- /literature/s12916-019-1396-4 - spatial mapping for wolbachia.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/literature/s12916-019-1396-4 - spatial mapping for wolbachia.pdf -------------------------------------------------------------------------------- /model/code/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Aggregated.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 4\n", 43 | "MTD_Deaths 4\n", 44 | "Reg_Ave_Temp_NCR 0\n", 45 | "Reg_Ave_Rainfall_NCR 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Med 0\n", 50 | "GTrend_Dengue_Sym 0\n", 51 | "dtype: int64" 52 | ] 53 | }, 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "output_type": "execute_result" 57 | } 58 | ], 59 | "source": [ 60 | "# CHECK FOR MISSING VALUES\n", 61 | "raw_data_trunc.isna().sum()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stderr", 71 | "output_type": "stream", 72 | "text": [ 73 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 74 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 75 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 76 | "\n", 77 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 78 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 79 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 80 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 81 | "\n", 82 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 83 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 84 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 85 | "\n", 86 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 87 | " self._setitem_with_indexer(indexer, value)\n", 88 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 89 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 90 | "\n", 91 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 92 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 93 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 101 | "def seasonal_mean(ts, n, lr=0.7):\n", 102 | " \"\"\"\n", 103 | " Compute the mean of corresponding seasonal periods\n", 104 | " ts: 1D array-like of the time series\n", 105 | " n: Seasonal window length of the time series\n", 106 | " \"\"\"\n", 107 | " out = np.copy(ts)\n", 108 | " for i, val in enumerate(ts):\n", 109 | " if np.isnan(val):\n", 110 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 111 | " if np.isnan(np.nanmean(ts_seas)):\n", 112 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 113 | " out[i] = np.nanmean(ts_seas) * lr\n", 114 | " return out\n", 115 | "\n", 116 | "#--- Get the seasonal means\n", 117 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 118 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 119 | "\n", 120 | "#--- Replace the missing observations with the seasonal means\n", 121 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 123 | "\n", 124 | "#--- Drop the seasonal mean columns\n", 125 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 5, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 135 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 136 | "\n", 137 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 138 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 139 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 149 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Clean.pickle')" 150 | ] 151 | } 152 | ], 153 | "metadata": { 154 | "kernelspec": { 155 | "display_name": "Python 3", 156 | "language": "python", 157 | "name": "python3" 158 | }, 159 | "language_info": { 160 | "codemirror_mode": { 161 | "name": "ipython", 162 | "version": 3 163 | }, 164 | "file_extension": ".py", 165 | "mimetype": "text/x-python", 166 | "name": "python", 167 | "nbconvert_exporter": "python", 168 | "pygments_lexer": "ipython3", 169 | "version": "3.6.5" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /model/code/.ipynb_checkpoints/pgm3_Time_Series_Modeling-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "\n", 11 | "import pandas as pd\n", 12 | "import numpy as np\n", 13 | "import matplotlib.pyplot as plt\n", 14 | "%matplotlib inline\n", 15 | "\n", 16 | "#--- Import Statsmodels\n", 17 | "import statsmodels.api as sm\n", 18 | "\n", 19 | "#--- Import Sklearn\n", 20 | "from sklearn.linear_model import LinearRegression\n", 21 | "from scipy import stats" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "# READ PICKLE OF THE ORIGINAL AND DIFFERENCED SERIES\n", 31 | "Dengue_PH = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Clean.pickle')\n", 32 | "Dengue_PH_diff = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Diff.pickle')" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 3, 38 | "metadata": {}, 39 | "outputs": [ 40 | { 41 | "data": { 42 | "text/plain": [ 43 | "Index(['GTrend_Dengue', 'GTrend_Dengue_Fever', 'GTrend_Dengue_Cure',\n", 44 | " 'GTrend_Dengue_Med', 'GTrend_Dengue_Sym', 'Mort_Rate', 'MTD_Cases',\n", 45 | " 'Reg_Ave_Temp_NCR', 'Reg_Ave_Rainfall_NCR'],\n", 46 | " dtype='object')" 47 | ] 48 | }, 49 | "execution_count": 3, 50 | "metadata": {}, 51 | "output_type": "execute_result" 52 | } 53 | ], 54 | "source": [ 55 | "Dengue_PH_diff.columns" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 4, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "# CREATE LAGGED VERSIONS OF THE PREDICTORS (MAX OF A QUARTER)\n", 65 | "Dengue_PH_diff['Temp_L1'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(1)\n", 66 | "Dengue_PH_diff['Temp_L2'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(2)\n", 67 | "Dengue_PH_diff['Temp_L3'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(3)\n", 68 | "\n", 69 | "Dengue_PH_diff['Rain_L1'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(1)\n", 70 | "Dengue_PH_diff['Rain_L2'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(2)\n", 71 | "Dengue_PH_diff['Rain_L3'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(3)\n", 72 | "\n", 73 | "Dengue_PH_diff['GT_Dengue_L1'] = Dengue_PH_diff['GTrend_Dengue'].shift(1)\n", 74 | "Dengue_PH_diff['GT_Dengue_L2'] = Dengue_PH_diff['GTrend_Dengue'].shift(2)\n", 75 | "Dengue_PH_diff['GT_Dengue_L3'] = Dengue_PH_diff['GTrend_Dengue'].shift(3)\n", 76 | "\n", 77 | "Dengue_PH_diff['GT_DengueFvr_L1'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(1)\n", 78 | "Dengue_PH_diff['GT_DengueFvr_L2'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(2)\n", 79 | "Dengue_PH_diff['GT_DengueFvr_L3'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(3)\n", 80 | "\n", 81 | "Dengue_PH_diff['GT_DengueCure_L1'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(1)\n", 82 | "Dengue_PH_diff['GT_DengueCure_L2'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(2)\n", 83 | "Dengue_PH_diff['GT_DengueCure_L3'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(3)\n", 84 | "\n", 85 | "Dengue_PH_diff['GT_DengueMed_L1'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(1)\n", 86 | "Dengue_PH_diff['GT_DengueMed_L2'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(2)\n", 87 | "Dengue_PH_diff['GT_DengueMed_L3'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(3)\n", 88 | "\n", 89 | "Dengue_PH_diff['GT_DengueSym_L1'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(1)\n", 90 | "Dengue_PH_diff['GT_DengueSym_L2'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(2)\n", 91 | "Dengue_PH_diff['GT_DengueSym_L3'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(3)\n", 92 | "\n", 93 | "Dummies = pd.get_dummies(Dengue_PH_diff.index.month, prefix='m')\n", 94 | "Dengue_PH_diff = Dengue_PH_diff.reset_index()\n", 95 | "Dengue_PH_diff = Dengue_PH_diff.merge(Dummies, left_index=True, right_index=True)\n", 96 | "Dengue_PH_diff.set_index('Date', inplace=True)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 5, 102 | "metadata": {}, 103 | "outputs": [ 104 | { 105 | "name": "stdout", 106 | "output_type": "stream", 107 | "text": [ 108 | "(32, 42)\n", 109 | "(12, 42)\n" 110 | ] 111 | } 112 | ], 113 | "source": [ 114 | "# SPLIT SERIES TO TRAINING AND TEST SETS\n", 115 | "#--- Set 2018 as the test dataframe\n", 116 | "nobs = 12\n", 117 | "df_train, df_test = Dengue_PH_diff[0:-nobs], Dengue_PH_diff[-nobs:]\n", 118 | "df_train = df_train.dropna()\n", 119 | "df_test = df_test.dropna()\n", 120 | "\n", 121 | "# Check size\n", 122 | "print(df_train.shape) \n", 123 | "print(df_test.shape) \n" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 27, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "# PERFORM UNIVARIATE REGRESSION TO TRIM DOWN THE PREDICTORS\n", 133 | "predictor_col = df_train.columns[df_train.columns.str.contains(pat = '_L')]\n", 134 | "pvals = pd.DataFrame()\n", 135 | "for col in predictor_col:\n", 136 | " Y = df_train.MTD_Cases\n", 137 | " X = df_train[col]\n", 138 | " X2 = sm.add_constant(X)\n", 139 | " mod = sm.OLS(Y,X)\n", 140 | " fit = mod.fit()\n", 141 | " pval = fit.summary2().tables[1]['P>|t|']\n", 142 | " pval = pval.to_frame()\n", 143 | " #print(pval)\n", 144 | " pvals = pvals.append(pval)\n", 145 | " \n", 146 | "# RETAIN ONLY THE LAGGED PREDICTORS WITH SIGNIFICANT P-VALUES\n", 147 | "pvals = pvals[pvals['P>|t|'] <= 0.05].reset_index()\n", 148 | "pvals = pvals.rename(columns={'index':'Variable'})\n", 149 | "shortlist_predictor_col = pvals['Variable']\n", 150 | "dummy = pd.Series(df_train.columns[df_train.columns.str.contains(pat = 'm_')])\n", 151 | "#shortlist_predictor_col = shortlist_predictor_col.append(dummy)\n", 152 | "#X = df_train[shortlist_predictor_col].drop(columns=['m_1'],axis=1)\n", 153 | "X = df_train[shortlist_predictor_col]" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "print(type(X))\n", 163 | "print(type(Y))\n", 164 | "print(Y.axes)" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 28, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "name": "stdout", 174 | "output_type": "stream", 175 | "text": [ 176 | "Add Temp_L3 with p-value 0.0135999\n", 177 | "Add GT_DengueMed_L1 with p-value 0.0303546\n", 178 | "resulting features:\n", 179 | "['Temp_L3', 'GT_DengueMed_L1']\n" 180 | ] 181 | }, 182 | { 183 | "name": "stderr", 184 | "output_type": "stream", 185 | "text": [ 186 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:31: FutureWarning: 'argmin' is deprecated, use 'idxmin' instead. The behavior of 'argmin'\n", 187 | "will be corrected to return the positional minimum in the future.\n", 188 | "Use 'series.values.argmin' to get the position of the minimum now.\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "# PERFORM STEPWISE REGRESSION\n", 194 | "def stepwise_selection(X, y, \n", 195 | " initial_list=[], \n", 196 | " threshold_in=0.05, \n", 197 | " threshold_out = 0.10, \n", 198 | " verbose=True):\n", 199 | " \"\"\" Perform a forward-backward feature selection \n", 200 | " based on p-value from statsmodels.api.OLS\n", 201 | " Arguments:\n", 202 | " X - pandas.DataFrame with candidate features\n", 203 | " y - list-like with the target\n", 204 | " initial_list - list of features to start with (column names of X)\n", 205 | " threshold_in - include a feature if its p-value < threshold_in\n", 206 | " threshold_out - exclude a feature if its p-value > threshold_out\n", 207 | " verbose - whether to print the sequence of inclusions and exclusions\n", 208 | " Returns: list of selected features \n", 209 | " Always set threshold_in < threshold_out to avoid infinite looping.\n", 210 | " See https://en.wikipedia.org/wiki/Stepwise_regression for the details\n", 211 | " \"\"\"\n", 212 | " included = list(initial_list)\n", 213 | " while True:\n", 214 | " changed=False\n", 215 | " # forward step\n", 216 | " excluded = list(set(X.columns)-set(included))\n", 217 | " new_pval = pd.Series(index=excluded)\n", 218 | " for new_column in excluded:\n", 219 | " model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()\n", 220 | " new_pval[new_column] = model.pvalues[new_column]\n", 221 | " best_pval = new_pval.min()\n", 222 | " if best_pval < threshold_in:\n", 223 | " best_feature = new_pval.argmin()\n", 224 | " included.append(best_feature)\n", 225 | " changed=True\n", 226 | " if verbose:\n", 227 | " print('Add {:30} with p-value {:.6}'.format(best_feature, best_pval))\n", 228 | "\n", 229 | " # backward step\n", 230 | " model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()\n", 231 | " # use all coefs except intercept\n", 232 | " pvalues = model.pvalues.iloc[1:]\n", 233 | " worst_pval = pvalues.max() # null if pvalues is empty\n", 234 | " if worst_pval > threshold_out:\n", 235 | " changed=True\n", 236 | " worst_feature = pvalues.argmax()\n", 237 | " included.remove(worst_feature)\n", 238 | " if verbose:\n", 239 | " print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))\n", 240 | " if not changed:\n", 241 | " break\n", 242 | " return included\n", 243 | "\n", 244 | "Y = Y.tolist()\n", 245 | "result = stepwise_selection(X, Y)\n", 246 | "\n", 247 | "print('resulting features:')\n", 248 | "print(result)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": {}, 255 | "outputs": [], 256 | "source": [ 257 | "# PRINT THE RESULTS FOR THE FINAL MODEL\n", 258 | "#X = df_train[result]\n", 259 | "X2 = sm.add_constant(X)\n", 260 | "est = sm.OLS(Y, X)\n", 261 | "est2 = est.fit()\n", 262 | "print(est2.summary())" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 26, 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "name": "stdout", 272 | "output_type": "stream", 273 | "text": [ 274 | " OLS Regression Results \n", 275 | "==============================================================================\n", 276 | "Dep. Variable: y R-squared: 0.394\n", 277 | "Model: OLS Adj. R-squared: 0.329\n", 278 | "Method: Least Squares F-statistic: 6.071\n", 279 | "Date: Sun, 20 Oct 2019 Prob (F-statistic): 0.00257\n", 280 | "Time: 01:10:59 Log-Likelihood: -30.093\n", 281 | "No. Observations: 32 AIC: 68.19\n", 282 | "Df Residuals: 28 BIC: 74.05\n", 283 | "Df Model: 3 \n", 284 | "Covariance Type: nonrobust \n", 285 | "===================================================================================\n", 286 | " coef std err t P>|t| [0.025 0.975]\n", 287 | "-----------------------------------------------------------------------------------\n", 288 | "const 0.2936 0.124 2.364 0.025 0.039 0.548\n", 289 | "Temp_L3 7.9225 3.299 2.402 0.023 1.165 14.680\n", 290 | "GT_DengueMed_L1 2.5886 1.064 2.434 0.022 0.410 4.767\n", 291 | "m_11 -0.7942 0.402 -1.974 0.058 -1.618 0.030\n", 292 | "==============================================================================\n", 293 | "Omnibus: 5.227 Durbin-Watson: 2.703\n", 294 | "Prob(Omnibus): 0.073 Jarque-Bera (JB): 3.605\n", 295 | "Skew: 0.674 Prob(JB): 0.165\n", 296 | "Kurtosis: 3.943 Cond. No. 28.3\n", 297 | "==============================================================================\n", 298 | "\n", 299 | "Warnings:\n", 300 | "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "# PRINT THE RESULTS FOR THE FINAL MODEL\n", 306 | "X = df_train[['Temp_L3','GT_DengueMed_L1','m_11']]\n", 307 | "X2 = sm.add_constant(X)\n", 308 | "est = sm.OLS(Y, X2)\n", 309 | "est2 = est.fit()\n", 310 | "print(est2.summary())" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": null, 323 | "metadata": {}, 324 | "outputs": [], 325 | "source": [ 326 | "Y" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": null, 332 | "metadata": {}, 333 | "outputs": [], 334 | "source": [] 335 | } 336 | ], 337 | "metadata": { 338 | "kernelspec": { 339 | "display_name": "Python 3", 340 | "language": "python", 341 | "name": "python3" 342 | }, 343 | "language_info": { 344 | "codemirror_mode": { 345 | "name": "ipython", 346 | "version": 3 347 | }, 348 | "file_extension": ".py", 349 | "mimetype": "text/x-python", 350 | "name": "python", 351 | "nbconvert_exporter": "python", 352 | "pygments_lexer": "ipython3", 353 | "version": "3.6.5" 354 | } 355 | }, 356 | "nbformat": 4, 357 | "nbformat_minor": 2 358 | } 359 | -------------------------------------------------------------------------------- /model/code/ARMM/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 7\n", 43 | "MTD_Deaths 7\n", 44 | "Reg_Ave_Temp_ARMM 0\n", 45 | "Reg_Ave_Rainfall_ARMM 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Sym 0\n", 48 | "dtype: int64" 49 | ] 50 | }, 51 | "execution_count": 3, 52 | "metadata": {}, 53 | "output_type": "execute_result" 54 | }, 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "MTD_Cases 7\n", 59 | "MTD_Deaths 7\n", 60 | "Reg_Ave_Temp_ARMM 0\n", 61 | "Reg_Ave_Rainfall_ARMM 0\n", 62 | "GTrend_Dengue 0\n", 63 | "GTrend_Dengue_Sym 0\n", 64 | "dtype: int64" 65 | ] 66 | }, 67 | "execution_count": 3, 68 | "metadata": {}, 69 | "output_type": "execute_result" 70 | } 71 | ], 72 | "source": [ 73 | "# CHECK FOR MISSING VALUES\n", 74 | "raw_data_trunc.isna().sum()" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 4, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stderr", 84 | "output_type": "stream", 85 | "text": [ 86 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 87 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 88 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 89 | "\n", 90 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 91 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 92 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 93 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 96 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 97 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 98 | "\n", 99 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 100 | " self._setitem_with_indexer(indexer, value)\n", 101 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 102 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 103 | "\n", 104 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 105 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 106 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 107 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 108 | "\n", 109 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 110 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 111 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 112 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 113 | "\n", 114 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 115 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 116 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 117 | "\n", 118 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 119 | " self._setitem_with_indexer(indexer, value)\n", 120 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 121 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 122 | "\n", 123 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 124 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 125 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 126 | "\n", 127 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 128 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 129 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 130 | "\n", 131 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 132 | ] 133 | } 134 | ], 135 | "source": [ 136 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 137 | "def seasonal_mean(ts, n, lr=0.7):\n", 138 | " \"\"\"\n", 139 | " Compute the mean of corresponding seasonal periods\n", 140 | " ts: 1D array-like of the time series\n", 141 | " n: Seasonal window length of the time series\n", 142 | " \"\"\"\n", 143 | " out = np.copy(ts)\n", 144 | " for i, val in enumerate(ts):\n", 145 | " if np.isnan(val):\n", 146 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 147 | " if np.isnan(np.nanmean(ts_seas)):\n", 148 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 149 | " out[i] = np.nanmean(ts_seas) * lr\n", 150 | " return out\n", 151 | "\n", 152 | "#--- Get the seasonal means\n", 153 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 154 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 155 | "\n", 156 | "#--- Replace the missing observations with the seasonal means\n", 157 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 158 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 159 | "\n", 160 | "#--- Drop the seasonal mean columns\n", 161 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "data": { 171 | "text/plain": [ 172 | "MTD_Cases 0\n", 173 | "MTD_Deaths 0\n", 174 | "Reg_Ave_Temp_ARMM 0\n", 175 | "Reg_Ave_Rainfall_ARMM 0\n", 176 | "GTrend_Dengue 0\n", 177 | "GTrend_Dengue_Sym 0\n", 178 | "dtype: int64" 179 | ] 180 | }, 181 | "execution_count": 5, 182 | "metadata": {}, 183 | "output_type": "execute_result" 184 | }, 185 | { 186 | "data": { 187 | "text/plain": [ 188 | "MTD_Cases 0\n", 189 | "MTD_Deaths 0\n", 190 | "Reg_Ave_Temp_ARMM 0\n", 191 | "Reg_Ave_Rainfall_ARMM 0\n", 192 | "GTrend_Dengue 0\n", 193 | "GTrend_Dengue_Sym 0\n", 194 | "dtype: int64" 195 | ] 196 | }, 197 | "execution_count": 5, 198 | "metadata": {}, 199 | "output_type": "execute_result" 200 | } 201 | ], 202 | "source": [ 203 | "# CHECK FOR MISSING VALUES\n", 204 | "raw_data_trunc.isna().sum()" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 6, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 214 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 215 | "\n", 216 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 217 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 218 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 7, 224 | "metadata": {}, 225 | "outputs": [], 226 | "source": [ 227 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 228 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Clean.pickle')" 229 | ] 230 | } 231 | ], 232 | "metadata": { 233 | "kernelspec": { 234 | "display_name": "Python 3", 235 | "language": "python", 236 | "name": "python3" 237 | }, 238 | "language_info": { 239 | "codemirror_mode": { 240 | "name": "ipython", 241 | "version": 3 242 | }, 243 | "file_extension": ".py", 244 | "mimetype": "text/x-python", 245 | "name": "python", 246 | "nbconvert_exporter": "python", 247 | "pygments_lexer": "ipython3", 248 | "version": "3.6.5" 249 | } 250 | }, 251 | "nbformat": 4, 252 | "nbformat_minor": 2 253 | } 254 | -------------------------------------------------------------------------------- /model/code/ARMM/pgm1_Data_Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 7\n", 43 | "MTD_Deaths 7\n", 44 | "Reg_Ave_Temp_ARMM 0\n", 45 | "Reg_Ave_Rainfall_ARMM 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Sym 0\n", 48 | "dtype: int64" 49 | ] 50 | }, 51 | "execution_count": 3, 52 | "metadata": {}, 53 | "output_type": "execute_result" 54 | }, 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "MTD_Cases 7\n", 59 | "MTD_Deaths 7\n", 60 | "Reg_Ave_Temp_ARMM 0\n", 61 | "Reg_Ave_Rainfall_ARMM 0\n", 62 | "GTrend_Dengue 0\n", 63 | "GTrend_Dengue_Sym 0\n", 64 | "dtype: int64" 65 | ] 66 | }, 67 | "execution_count": 3, 68 | "metadata": {}, 69 | "output_type": "execute_result" 70 | } 71 | ], 72 | "source": [ 73 | "# CHECK FOR MISSING VALUES\n", 74 | "raw_data_trunc.isna().sum()" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 4, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stderr", 84 | "output_type": "stream", 85 | "text": [ 86 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 87 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 88 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 89 | "\n", 90 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 91 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 92 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 93 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 96 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 97 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 98 | "\n", 99 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 100 | " self._setitem_with_indexer(indexer, value)\n", 101 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 102 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 103 | "\n", 104 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 105 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 106 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 107 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 108 | "\n", 109 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 110 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 111 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 112 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 113 | "\n", 114 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 115 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 116 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 117 | "\n", 118 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 119 | " self._setitem_with_indexer(indexer, value)\n", 120 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 121 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 122 | "\n", 123 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 124 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 125 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 126 | "\n", 127 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 128 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 129 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 130 | "\n", 131 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 132 | ] 133 | } 134 | ], 135 | "source": [ 136 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 137 | "def seasonal_mean(ts, n, lr=0.7):\n", 138 | " \"\"\"\n", 139 | " Compute the mean of corresponding seasonal periods\n", 140 | " ts: 1D array-like of the time series\n", 141 | " n: Seasonal window length of the time series\n", 142 | " \"\"\"\n", 143 | " out = np.copy(ts)\n", 144 | " for i, val in enumerate(ts):\n", 145 | " if np.isnan(val):\n", 146 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 147 | " if np.isnan(np.nanmean(ts_seas)):\n", 148 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 149 | " out[i] = np.nanmean(ts_seas) * lr\n", 150 | " return out\n", 151 | "\n", 152 | "#--- Get the seasonal means\n", 153 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 154 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 155 | "\n", 156 | "#--- Replace the missing observations with the seasonal means\n", 157 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 158 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 159 | "\n", 160 | "#--- Drop the seasonal mean columns\n", 161 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "data": { 171 | "text/plain": [ 172 | "MTD_Cases 0\n", 173 | "MTD_Deaths 0\n", 174 | "Reg_Ave_Temp_ARMM 0\n", 175 | "Reg_Ave_Rainfall_ARMM 0\n", 176 | "GTrend_Dengue 0\n", 177 | "GTrend_Dengue_Sym 0\n", 178 | "dtype: int64" 179 | ] 180 | }, 181 | "execution_count": 5, 182 | "metadata": {}, 183 | "output_type": "execute_result" 184 | }, 185 | { 186 | "data": { 187 | "text/plain": [ 188 | "MTD_Cases 0\n", 189 | "MTD_Deaths 0\n", 190 | "Reg_Ave_Temp_ARMM 0\n", 191 | "Reg_Ave_Rainfall_ARMM 0\n", 192 | "GTrend_Dengue 0\n", 193 | "GTrend_Dengue_Sym 0\n", 194 | "dtype: int64" 195 | ] 196 | }, 197 | "execution_count": 5, 198 | "metadata": {}, 199 | "output_type": "execute_result" 200 | } 201 | ], 202 | "source": [ 203 | "# CHECK FOR MISSING VALUES\n", 204 | "raw_data_trunc.isna().sum()" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 6, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 214 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 215 | "\n", 216 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 217 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 218 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 7, 224 | "metadata": {}, 225 | "outputs": [], 226 | "source": [ 227 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 228 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Clean.pickle')" 229 | ] 230 | } 231 | ], 232 | "metadata": { 233 | "kernelspec": { 234 | "display_name": "Python 3", 235 | "language": "python", 236 | "name": "python3" 237 | }, 238 | "language_info": { 239 | "codemirror_mode": { 240 | "name": "ipython", 241 | "version": 3 242 | }, 243 | "file_extension": ".py", 244 | "mimetype": "text/x-python", 245 | "name": "python", 246 | "nbconvert_exporter": "python", 247 | "pygments_lexer": "ipython3", 248 | "version": "3.6.5" 249 | } 250 | }, 251 | "nbformat": 4, 252 | "nbformat_minor": 2 253 | } 254 | -------------------------------------------------------------------------------- /model/code/ARMM/pgm3_Time_Series_Modeling.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "\n", 11 | "import itertools\n", 12 | "import time\n", 13 | "from math import sqrt\n", 14 | "import pandas as pd\n", 15 | "import numpy as np\n", 16 | "import seaborn as sns\n", 17 | "import matplotlib.pyplot as plt\n", 18 | "%matplotlib inline\n", 19 | "\n", 20 | "#--- Import Statsmodels\n", 21 | "import statsmodels.api as sm\n", 22 | "\n", 23 | "#--- Import Sklearn\n", 24 | "from sklearn.linear_model import LinearRegression\n", 25 | "from sklearn.metrics import mean_squared_error\n", 26 | "from sklearn.metrics import mean_absolute_error\n", 27 | "\n", 28 | "#--- Importing tqdm for the progress bar\n", 29 | "from tqdm import tnrange, tqdm_notebook" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 2, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "# READ PICKLE OF THE ORIGINAL AND DIFFERENCED SERIES\n", 39 | "Dengue_ARMM = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Clean.pickle')\n", 40 | "Dengue_ARMM_diff = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Diff.pickle')" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 3, 46 | "metadata": {}, 47 | "outputs": [ 48 | { 49 | "data": { 50 | "text/plain": [ 51 | "Index(['MTD_Deaths', 'GTrend_Dengue', 'GTrend_Dengue_Sym', 'Mort_Rate',\n", 52 | " 'MTD_Cases', 'Reg_Ave_Temp_ARMM', 'Reg_Ave_Rainfall_ARMM'],\n", 53 | " dtype='object')" 54 | ] 55 | }, 56 | "execution_count": 3, 57 | "metadata": {}, 58 | "output_type": "execute_result" 59 | } 60 | ], 61 | "source": [ 62 | "Dengue_ARMM_diff.columns" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 4, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "# CREATE LAGGED VERSIONS OF THE PREDICTORS (MAX OF A QUARTER)\n", 72 | "Dengue_ARMM_diff['Temp_L1'] = Dengue_ARMM_diff['Reg_Ave_Temp_ARMM'].shift(1)\n", 73 | "Dengue_ARMM_diff['Temp_L2'] = Dengue_ARMM_diff['Reg_Ave_Temp_ARMM'].shift(2)\n", 74 | "Dengue_ARMM_diff['Temp_L3'] = Dengue_ARMM_diff['Reg_Ave_Temp_ARMM'].shift(3)\n", 75 | "\n", 76 | "Dengue_ARMM_diff['Rain_L1'] = Dengue_ARMM_diff['Reg_Ave_Rainfall_ARMM'].shift(1)\n", 77 | "Dengue_ARMM_diff['Rain_L2'] = Dengue_ARMM_diff['Reg_Ave_Rainfall_ARMM'].shift(2)\n", 78 | "Dengue_ARMM_diff['Rain_L3'] = Dengue_ARMM_diff['Reg_Ave_Rainfall_ARMM'].shift(3)\n", 79 | "\n", 80 | "Dengue_ARMM_diff['GT_Dengue_L1'] = Dengue_ARMM_diff['GTrend_Dengue'].shift(1)\n", 81 | "Dengue_ARMM_diff['GT_Dengue_L2'] = Dengue_ARMM_diff['GTrend_Dengue'].shift(2)\n", 82 | "Dengue_ARMM_diff['GT_Dengue_L3'] = Dengue_ARMM_diff['GTrend_Dengue'].shift(3)\n", 83 | "\n", 84 | "Dengue_ARMM_diff['GT_DengueSym_L1'] = Dengue_ARMM_diff['GTrend_Dengue_Sym'].shift(1)\n", 85 | "Dengue_ARMM_diff['GT_DengueSym_L2'] = Dengue_ARMM_diff['GTrend_Dengue_Sym'].shift(2)\n", 86 | "Dengue_ARMM_diff['GT_DengueSym_L3'] = Dengue_ARMM_diff['GTrend_Dengue_Sym'].shift(3)\n", 87 | "\n", 88 | "#Dengue_ARMM_diff['Cases_L1'] = Dengue_ARMM_diff['MTD_Cases'].shift(1)\n", 89 | "#Dengue_ARMM_diff['Cases_L2'] = Dengue_ARMM_diff['MTD_Cases'].shift(2)\n", 90 | "#Dengue_ARMM_diff['Cases_L3'] = Dengue_ARMM_diff['MTD_Cases'].shift(3)\n", 91 | "\n", 92 | "Dummies = pd.get_dummies(Dengue_ARMM_diff.index.month, prefix='m')\n", 93 | "Dengue_ARMM_diff = Dengue_ARMM_diff.reset_index()\n", 94 | "Dengue_ARMM_diff = Dengue_ARMM_diff.merge(Dummies, left_index=True, right_index=True)\n", 95 | "Dengue_ARMM_diff.set_index('Date', inplace=True)" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 5, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "name": "stdout", 105 | "output_type": "stream", 106 | "text": [ 107 | "(32, 31)\n", 108 | "(12, 31)\n" 109 | ] 110 | } 111 | ], 112 | "source": [ 113 | "# SPLIT SERIES TO TRAINING AND TEST SETS\n", 114 | "#--- Set 2018 as the test dataframe\n", 115 | "nobs = 12\n", 116 | "df_train, df_test = Dengue_ARMM_diff[0:-nobs], Dengue_ARMM_diff[-nobs:]\n", 117 | "df_train = df_train.dropna()\n", 118 | "df_test = df_test.dropna()\n", 119 | "\n", 120 | "# Check size\n", 121 | "print(df_train.shape) \n", 122 | "print(df_test.shape) \n" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 6, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "def fit_linear_reg(X,Y):\n", 132 | " #Fit linear regression model \n", 133 | " X2 = sm.add_constant(X)\n", 134 | " model_k = sm.OLS(Y,X2)\n", 135 | " fit = model_k.fit()\n", 136 | " pvalues = fit.pvalues\n", 137 | " worst_pval = pvalues.max() \n", 138 | " worst_feature = pvalues.argmax()\n", 139 | " Y_Pred_Test = fit.predict(sm.add_constant(df_test[list(X)]))\n", 140 | " \n", 141 | " if worst_feature == 'const':\n", 142 | " model_k = sm.OLS(Y,X)\n", 143 | " fit = model_k.fit()\n", 144 | " pvalues = fit.pvalues\n", 145 | " worst_pval = pvalues.max() \n", 146 | " worst_feature = pvalues.argmax()\n", 147 | " Y_Pred_Test = fit.predict(df_test[list(X)])\n", 148 | " \n", 149 | " pval = fit.pvalues.to_frame()\n", 150 | " features = list(pval.index)\n", 151 | " pvals = list(pval[0])\n", 152 | " sig = pval[pval[0]<=0.05]\n", 153 | " pct_sig = len(list(sig[0])) / len(list(pval[0]))\n", 154 | " rsq = fit.rsquared\n", 155 | " adjr = fit.rsquared_adj\n", 156 | " serial_corr = list(sm.stats.diagnostic.acorr_breusch_godfrey(fit, nlags=3))[3]\n", 157 | " het_arch = list(sm.stats.diagnostic.het_arch(fit.resid, maxlag=1))[3]\n", 158 | " normality = list(sm.stats.stattools.jarque_bera(fit.resid))[1]\n", 159 | " \n", 160 | " mae = mean_absolute_error(Y_True_Test,Y_Pred_Test)\n", 161 | " mse = mean_squared_error(Y_True_Test,Y_Pred_Test)\n", 162 | " rmse = sqrt(mean_squared_error(Y_True_Test,Y_Pred_Test))\n", 163 | " return features, pvals, pct_sig, rsq, adjr, serial_corr, het_arch, normality, mae, mse, rmse, worst_pval, worst_feature" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": 7, 169 | "metadata": {}, 170 | "outputs": [ 171 | { 172 | "name": "stderr", 173 | "output_type": "stream", 174 | "text": [ 175 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:8: FutureWarning: 'argmax' is deprecated, use 'idxmax' instead. The behavior of 'argmax'\n", 176 | "will be corrected to return the positional maximum in the future.\n", 177 | "Use 'series.values.argmax' to get the position of the maximum now.\n", 178 | " \n", 179 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:16: FutureWarning: 'argmax' is deprecated, use 'idxmax' instead. The behavior of 'argmax'\n", 180 | "will be corrected to return the positional maximum in the future.\n", 181 | "Use 'series.values.argmax' to get the position of the maximum now.\n", 182 | " app.launch_new_instance()\n" 183 | ] 184 | } 185 | ], 186 | "source": [ 187 | "# INITIALIZE VARIABLES\n", 188 | "X = df_train[df_train.columns[df_train.columns.str.contains(r'_L|m_')]]\n", 189 | "Y = df_train.MTD_Cases\n", 190 | "Y_True_Test = df_test.MTD_Cases\n", 191 | "threshold_out = 0.05\n", 192 | "\n", 193 | "remaining_features = list(X.columns.values)\n", 194 | "features = []\n", 195 | "R_squared_list, AdjR2_list, feature_list, pval_list = [],[],[],[]\n", 196 | "pct_sig_list = []\n", 197 | "num_features = []\n", 198 | "serial_corr_list = []\n", 199 | "het_arch_list = []\n", 200 | "norm_list = []\n", 201 | "mae_list, mse_list, rmse_list = [],[],[]\n", 202 | "\n", 203 | "# RUN BACKWARD STEPWISE REGRESSION \n", 204 | "#--- Remove predictors one at a time until there is no more p-value exceeding the threshold\n", 205 | "while True:\n", 206 | " changed = False\n", 207 | " tmp_result = fit_linear_reg(X[list(set(remaining_features))],Y) \n", 208 | " num_features.append(len(remaining_features)) \n", 209 | " feature_list.append(tmp_result[0])\n", 210 | " pval_list.append(tmp_result[1])\n", 211 | " pct_sig_list.append(tmp_result[2])\n", 212 | " R_squared_list.append(tmp_result[3])\n", 213 | " AdjR2_list.append(tmp_result[4])\n", 214 | " serial_corr_list.append(tmp_result[5])\n", 215 | " het_arch_list.append(tmp_result[6])\n", 216 | " norm_list.append(tmp_result[7])\n", 217 | " mae_list.append(tmp_result[8])\n", 218 | " mse_list.append(tmp_result[9])\n", 219 | " rmse_list.append(tmp_result[10])\n", 220 | " \n", 221 | " if tmp_result[11] > threshold_out:\n", 222 | " changed = True\n", 223 | " remaining_features.remove(tmp_result[12])\n", 224 | " \n", 225 | " if not changed:\n", 226 | " break" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 8, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "# STORE IN DATAFRAME\n", 236 | "subsets_df = pd.DataFrame({'num_features': num_features, 'features': feature_list, 'P>|t|': pval_list, 'pct_sig': pct_sig_list, \\\n", 237 | " 'rsq': R_squared_list, 'adj_rsq': AdjR2_list, 'serial_corr': serial_corr_list, 'het': het_arch_list, \\\n", 238 | " 'normality': norm_list, 'mae': mae_list, 'mse': mse_list, 'rmse': rmse_list})\n", 239 | "\n", 240 | "# RETAIN ONLY THE SUBSET MODELS WHICH PASSED THE DIAGNOSTIC TESTS\n", 241 | "subsets_df = subsets_df[(subsets_df['serial_corr'] > 0.05) & (subsets_df['het'] > 0.05) & (subsets_df['normality'] > 0.05)]\n", 242 | "\n", 243 | "# GET ONLY THE TOP 3 MODELS BY ADJUSTED R-SQUARED\n", 244 | "top3_subsets_df = subsets_df.nlargest(3,'adj_rsq').reset_index().drop(columns=['index'])\n", 245 | "top3_subsets_df.index += 1" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": 9, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "# RERUN THE TOP 3 MODELS AND STORE THE FORECASTS FOR BOTH TRAINING AND TEST SETS\n", 255 | "Y = df_train['MTD_Cases']\n", 256 | "\n", 257 | "for i in range(1,4):\n", 258 | " top = top3_subsets_df['features'][i]\n", 259 | " if top.count('const') == 0:\n", 260 | " X = df_train[top] \n", 261 | " reg = LinearRegression(fit_intercept=False)\n", 262 | "\n", 263 | " if top.count('const') > 0:\n", 264 | " top.remove('const')\n", 265 | " X = df_train[top] \n", 266 | " reg = LinearRegression(fit_intercept=True)\n", 267 | "\n", 268 | " reg.fit(X,Y)\n", 269 | " prediction = pd.DataFrame(reg.predict(X))\n", 270 | " prediction.columns = ['Pred_Model_'] \n", 271 | " prediction.columns += str(i)\n", 272 | " if i == 1:\n", 273 | " df_train2 = df_train.reset_index().join(prediction, how='inner')\n", 274 | " if i > 1:\n", 275 | " df_train2 = df_train2.reset_index().join(prediction, how='inner')\n", 276 | "\n", 277 | " prediction = pd.DataFrame(reg.predict(df_test[X.columns]))\n", 278 | " prediction.columns = ['Pred_Model_']\n", 279 | " prediction.columns += str(i)\n", 280 | " if i == 1:\n", 281 | " df_test2 = df_test.reset_index().join(prediction, how='inner')\n", 282 | " if i > 1:\n", 283 | " df_test2 = df_test2.reset_index().join(prediction, how='inner')" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 10, 289 | "metadata": {}, 290 | "outputs": [], 291 | "source": [ 292 | "# REVERT THE FORECAST TO THE ORIGINAL FORM FROM PERCENTAGE CHANGE\n", 293 | "Dengue_ARMM_Fct = df_train2.append(df_test2, ignore_index=True)\n", 294 | "Dengue_ARMM_Fct = Dengue_ARMM_Fct[['Date','Pred_Model_1','Pred_Model_2','Pred_Model_3']]\n", 295 | "Dengue_ARMM_Fct['Pred_Model_1'] += 1\n", 296 | "Dengue_ARMM_Fct['Pred_Model_2'] += 1\n", 297 | "Dengue_ARMM_Fct['Pred_Model_3'] += 1\n", 298 | "Dengue_ARMM_Fct = Dengue_ARMM_Fct.rename(columns={'Pred_Model_1': 'Pred_PctChg_1', 'Pred_Model_2': 'Pred_PctChg_2', \\\n", 299 | " 'Pred_Model_3': 'Pred_PctChg_3'})\n", 300 | "Dengue_ARMM_Fct.set_index('Date', inplace=True)\n", 301 | "\n", 302 | "Dengue_ARMM2 = Dengue_ARMM.merge(Dengue_ARMM_Fct,how='left',on='Date')\n", 303 | "Dengue_ARMM2['MTD_Cases_Fct_1'] = Dengue_ARMM2.MTD_Cases.shift(1) * Dengue_ARMM2['Pred_PctChg_1']\n", 304 | "Dengue_ARMM2['MTD_Cases_Fct_2'] = Dengue_ARMM2.MTD_Cases.shift(1) * Dengue_ARMM2['Pred_PctChg_2']\n", 305 | "Dengue_ARMM2['MTD_Cases_Fct_3'] = Dengue_ARMM2.MTD_Cases.shift(1) * Dengue_ARMM2['Pred_PctChg_3']\n", 306 | "Dengue_ARMM2 = Dengue_ARMM2.drop(columns=['Pred_PctChg_1','Pred_PctChg_2','Pred_PctChg_3'])\n" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 12, 312 | "metadata": {}, 313 | "outputs": [], 314 | "source": [ 315 | "# SAVE MODEL DATAFRAMES TO PICKLE\n", 316 | "top3_subsets_df.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Top3_Models.pickle')\n", 317 | "Dengue_ARMM2.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Fct.pickle')" 318 | ] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": 13, 323 | "metadata": {}, 324 | "outputs": [], 325 | "source": [ 326 | "# EXPORT PICKLES TO EXCEL \n", 327 | "writer = pd.ExcelWriter('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_ARMM_Fct.xlsx', engine='xlsxwriter')\n", 328 | "Dengue_ARMM2.to_excel(writer, sheet_name='Fct')\n", 329 | "top3_subsets_df.to_excel(writer, sheet_name='Model')\n", 330 | "writer.save()" 331 | ] 332 | } 333 | ], 334 | "metadata": { 335 | "kernelspec": { 336 | "display_name": "Python 3", 337 | "language": "python", 338 | "name": "python3" 339 | }, 340 | "language_info": { 341 | "codemirror_mode": { 342 | "name": "ipython", 343 | "version": 3 344 | }, 345 | "file_extension": ".py", 346 | "mimetype": "text/x-python", 347 | "name": "python", 348 | "nbconvert_exporter": "python", 349 | "pygments_lexer": "ipython3", 350 | "version": "3.6.5" 351 | } 352 | }, 353 | "nbformat": 4, 354 | "nbformat_minor": 2 355 | } 356 | -------------------------------------------------------------------------------- /model/code/Eastern Visayas/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_Eastern_Visayas.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 3\n", 43 | "MTD_Deaths 3\n", 44 | "Reg_Ave_Temp_EastVis 0\n", 45 | "Reg_Ave_Rainfall_EastVis 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Sym 0\n", 49 | "dtype: int64" 50 | ] 51 | }, 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "output_type": "execute_result" 55 | } 56 | ], 57 | "source": [ 58 | "# CHECK FOR MISSING VALUES\n", 59 | "raw_data_trunc.isna().sum()" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 4, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stderr", 69 | "output_type": "stream", 70 | "text": [ 71 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 72 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 73 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 74 | "\n", 75 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 76 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 77 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 78 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 79 | "\n", 80 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 81 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 82 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 83 | "\n", 84 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 85 | " self._setitem_with_indexer(indexer, value)\n", 86 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 87 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 88 | "\n", 89 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 90 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 91 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 92 | "\n", 93 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 94 | ] 95 | } 96 | ], 97 | "source": [ 98 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 99 | "def seasonal_mean(ts, n, lr=0.7):\n", 100 | " \"\"\"\n", 101 | " Compute the mean of corresponding seasonal periods\n", 102 | " ts: 1D array-like of the time series\n", 103 | " n: Seasonal window length of the time series\n", 104 | " \"\"\"\n", 105 | " out = np.copy(ts)\n", 106 | " for i, val in enumerate(ts):\n", 107 | " if np.isnan(val):\n", 108 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 109 | " if np.isnan(np.nanmean(ts_seas)):\n", 110 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 111 | " out[i] = np.nanmean(ts_seas) * lr\n", 112 | " return out\n", 113 | "\n", 114 | "#--- Get the seasonal means\n", 115 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 116 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 117 | "\n", 118 | "#--- Replace the missing observations with the seasonal means\n", 119 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 120 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 121 | "\n", 122 | "#--- Drop the seasonal mean columns\n", 123 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 5, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 133 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 134 | "\n", 135 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 136 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 137 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 147 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_EastVis_Clean.pickle')" 148 | ] 149 | } 150 | ], 151 | "metadata": { 152 | "kernelspec": { 153 | "display_name": "Python 3", 154 | "language": "python", 155 | "name": "python3" 156 | }, 157 | "language_info": { 158 | "codemirror_mode": { 159 | "name": "ipython", 160 | "version": 3 161 | }, 162 | "file_extension": ".py", 163 | "mimetype": "text/x-python", 164 | "name": "python", 165 | "nbconvert_exporter": "python", 166 | "pygments_lexer": "ipython3", 167 | "version": "3.6.5" 168 | } 169 | }, 170 | "nbformat": 4, 171 | "nbformat_minor": 2 172 | } 173 | -------------------------------------------------------------------------------- /model/code/Eastern Visayas/pgm1_Data_Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_Eastern_Visayas.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 3\n", 43 | "MTD_Deaths 3\n", 44 | "Reg_Ave_Temp_EastVis 0\n", 45 | "Reg_Ave_Rainfall_EastVis 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Sym 0\n", 49 | "dtype: int64" 50 | ] 51 | }, 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "output_type": "execute_result" 55 | } 56 | ], 57 | "source": [ 58 | "# CHECK FOR MISSING VALUES\n", 59 | "raw_data_trunc.isna().sum()" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 4, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stderr", 69 | "output_type": "stream", 70 | "text": [ 71 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 72 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 73 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 74 | "\n", 75 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 76 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 77 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 78 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 79 | "\n", 80 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 81 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 82 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 83 | "\n", 84 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 85 | " self._setitem_with_indexer(indexer, value)\n", 86 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 87 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 88 | "\n", 89 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 90 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 91 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 92 | "\n", 93 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 94 | ] 95 | } 96 | ], 97 | "source": [ 98 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 99 | "def seasonal_mean(ts, n, lr=0.7):\n", 100 | " \"\"\"\n", 101 | " Compute the mean of corresponding seasonal periods\n", 102 | " ts: 1D array-like of the time series\n", 103 | " n: Seasonal window length of the time series\n", 104 | " \"\"\"\n", 105 | " out = np.copy(ts)\n", 106 | " for i, val in enumerate(ts):\n", 107 | " if np.isnan(val):\n", 108 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 109 | " if np.isnan(np.nanmean(ts_seas)):\n", 110 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 111 | " out[i] = np.nanmean(ts_seas) * lr\n", 112 | " return out\n", 113 | "\n", 114 | "#--- Get the seasonal means\n", 115 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 116 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 117 | "\n", 118 | "#--- Replace the missing observations with the seasonal means\n", 119 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 120 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 121 | "\n", 122 | "#--- Drop the seasonal mean columns\n", 123 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 5, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 133 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 134 | "\n", 135 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 136 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 137 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 147 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_EastVis_Clean.pickle')" 148 | ] 149 | } 150 | ], 151 | "metadata": { 152 | "kernelspec": { 153 | "display_name": "Python 3", 154 | "language": "python", 155 | "name": "python3" 156 | }, 157 | "language_info": { 158 | "codemirror_mode": { 159 | "name": "ipython", 160 | "version": 3 161 | }, 162 | "file_extension": ".py", 163 | "mimetype": "text/x-python", 164 | "name": "python", 165 | "nbconvert_exporter": "python", 166 | "pygments_lexer": "ipython3", 167 | "version": "3.6.5" 168 | } 169 | }, 170 | "nbformat": 4, 171 | "nbformat_minor": 2 172 | } 173 | -------------------------------------------------------------------------------- /model/code/NCR/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_NCR.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 5\n", 43 | "MTD_Deaths 3\n", 44 | "Reg_Ave_Temp_NCR 0\n", 45 | "Reg_Ave_Rainfall_NCR 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Med 0\n", 50 | "GTrend_Dengue_Sym 0\n", 51 | "dtype: int64" 52 | ] 53 | }, 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "output_type": "execute_result" 57 | } 58 | ], 59 | "source": [ 60 | "# CHECK FOR MISSING VALUES\n", 61 | "raw_data_trunc.isna().sum()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stderr", 71 | "output_type": "stream", 72 | "text": [ 73 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 74 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 75 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 76 | "\n", 77 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 78 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 79 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 80 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 81 | "\n", 82 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 83 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 84 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 85 | "\n", 86 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 87 | " self._setitem_with_indexer(indexer, value)\n", 88 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 89 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 90 | "\n", 91 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 92 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 93 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 101 | "def seasonal_mean(ts, n, lr=0.7):\n", 102 | " \"\"\"\n", 103 | " Compute the mean of corresponding seasonal periods\n", 104 | " ts: 1D array-like of the time series\n", 105 | " n: Seasonal window length of the time series\n", 106 | " \"\"\"\n", 107 | " out = np.copy(ts)\n", 108 | " for i, val in enumerate(ts):\n", 109 | " if np.isnan(val):\n", 110 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 111 | " if np.isnan(np.nanmean(ts_seas)):\n", 112 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 113 | " out[i] = np.nanmean(ts_seas) * lr\n", 114 | " return out\n", 115 | "\n", 116 | "#--- Get the seasonal means\n", 117 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 118 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 119 | "\n", 120 | "#--- Replace the missing observations with the seasonal means\n", 121 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 123 | "\n", 124 | "#--- Drop the seasonal mean columns\n", 125 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 5, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 135 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 136 | "\n", 137 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 138 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 139 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 149 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_NCR_Clean.pickle')" 150 | ] 151 | } 152 | ], 153 | "metadata": { 154 | "kernelspec": { 155 | "display_name": "Python 3", 156 | "language": "python", 157 | "name": "python3" 158 | }, 159 | "language_info": { 160 | "codemirror_mode": { 161 | "name": "ipython", 162 | "version": 3 163 | }, 164 | "file_extension": ".py", 165 | "mimetype": "text/x-python", 166 | "name": "python", 167 | "nbconvert_exporter": "python", 168 | "pygments_lexer": "ipython3", 169 | "version": "3.6.5" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /model/code/NCR/pgm1_Data_Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_NCR.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 5\n", 43 | "MTD_Deaths 3\n", 44 | "Reg_Ave_Temp_NCR 0\n", 45 | "Reg_Ave_Rainfall_NCR 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Med 0\n", 50 | "GTrend_Dengue_Sym 0\n", 51 | "dtype: int64" 52 | ] 53 | }, 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "output_type": "execute_result" 57 | } 58 | ], 59 | "source": [ 60 | "# CHECK FOR MISSING VALUES\n", 61 | "raw_data_trunc.isna().sum()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stderr", 71 | "output_type": "stream", 72 | "text": [ 73 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 74 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 75 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 76 | "\n", 77 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 78 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 79 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 80 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 81 | "\n", 82 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 83 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 84 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 85 | "\n", 86 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 87 | " self._setitem_with_indexer(indexer, value)\n", 88 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 89 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 90 | "\n", 91 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 92 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 93 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 101 | "def seasonal_mean(ts, n, lr=0.7):\n", 102 | " \"\"\"\n", 103 | " Compute the mean of corresponding seasonal periods\n", 104 | " ts: 1D array-like of the time series\n", 105 | " n: Seasonal window length of the time series\n", 106 | " \"\"\"\n", 107 | " out = np.copy(ts)\n", 108 | " for i, val in enumerate(ts):\n", 109 | " if np.isnan(val):\n", 110 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 111 | " if np.isnan(np.nanmean(ts_seas)):\n", 112 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 113 | " out[i] = np.nanmean(ts_seas) * lr\n", 114 | " return out\n", 115 | "\n", 116 | "#--- Get the seasonal means\n", 117 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 118 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 119 | "\n", 120 | "#--- Replace the missing observations with the seasonal means\n", 121 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 123 | "\n", 124 | "#--- Drop the seasonal mean columns\n", 125 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 5, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 135 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 136 | "\n", 137 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 138 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 139 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 149 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_NCR_Clean.pickle')" 150 | ] 151 | } 152 | ], 153 | "metadata": { 154 | "kernelspec": { 155 | "display_name": "Python 3", 156 | "language": "python", 157 | "name": "python3" 158 | }, 159 | "language_info": { 160 | "codemirror_mode": { 161 | "name": "ipython", 162 | "version": 3 163 | }, 164 | "file_extension": ".py", 165 | "mimetype": "text/x-python", 166 | "name": "python", 167 | "nbconvert_exporter": "python", 168 | "pygments_lexer": "ipython3", 169 | "version": "3.6.5" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /model/code/PH/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Aggregated.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 4\n", 43 | "MTD_Deaths 4\n", 44 | "Reg_Ave_Temp_NCR 0\n", 45 | "Reg_Ave_Rainfall_NCR 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Med 0\n", 50 | "GTrend_Dengue_Sym 0\n", 51 | "dtype: int64" 52 | ] 53 | }, 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "output_type": "execute_result" 57 | } 58 | ], 59 | "source": [ 60 | "# CHECK FOR MISSING VALUES\n", 61 | "raw_data_trunc.isna().sum()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stderr", 71 | "output_type": "stream", 72 | "text": [ 73 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 74 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 75 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 76 | "\n", 77 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 78 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 79 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 80 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 81 | "\n", 82 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 83 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 84 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 85 | "\n", 86 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 87 | " self._setitem_with_indexer(indexer, value)\n", 88 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 89 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 90 | "\n", 91 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 92 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 93 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 101 | "def seasonal_mean(ts, n, lr=0.7):\n", 102 | " \"\"\"\n", 103 | " Compute the mean of corresponding seasonal periods\n", 104 | " ts: 1D array-like of the time series\n", 105 | " n: Seasonal window length of the time series\n", 106 | " \"\"\"\n", 107 | " out = np.copy(ts)\n", 108 | " for i, val in enumerate(ts):\n", 109 | " if np.isnan(val):\n", 110 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 111 | " if np.isnan(np.nanmean(ts_seas)):\n", 112 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 113 | " out[i] = np.nanmean(ts_seas) * lr\n", 114 | " return out\n", 115 | "\n", 116 | "#--- Get the seasonal means\n", 117 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 118 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 119 | "\n", 120 | "#--- Replace the missing observations with the seasonal means\n", 121 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 123 | "\n", 124 | "#--- Drop the seasonal mean columns\n", 125 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 5, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 135 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 136 | "\n", 137 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 138 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 139 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 149 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Clean.pickle')" 150 | ] 151 | } 152 | ], 153 | "metadata": { 154 | "kernelspec": { 155 | "display_name": "Python 3", 156 | "language": "python", 157 | "name": "python3" 158 | }, 159 | "language_info": { 160 | "codemirror_mode": { 161 | "name": "ipython", 162 | "version": 3 163 | }, 164 | "file_extension": ".py", 165 | "mimetype": "text/x-python", 166 | "name": "python", 167 | "nbconvert_exporter": "python", 168 | "pygments_lexer": "ipython3", 169 | "version": "3.6.5" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /model/code/PH/.ipynb_checkpoints/pgm3_Time_Series_Modeling-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "\n", 11 | "import pandas as pd\n", 12 | "import numpy as np\n", 13 | "import matplotlib.pyplot as plt\n", 14 | "%matplotlib inline\n", 15 | "\n", 16 | "#--- Import Statsmodels\n", 17 | "import statsmodels.api as sm\n", 18 | "\n", 19 | "#--- Import Sklearn\n", 20 | "from sklearn.linear_model import LinearRegression\n", 21 | "from scipy import stats" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "# READ PICKLE OF THE ORIGINAL AND DIFFERENCED SERIES\n", 31 | "Dengue_PH = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Clean.pickle')\n", 32 | "Dengue_PH_diff = pd.read_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Diff.pickle')" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 3, 38 | "metadata": {}, 39 | "outputs": [ 40 | { 41 | "data": { 42 | "text/plain": [ 43 | "Index(['GTrend_Dengue', 'GTrend_Dengue_Fever', 'GTrend_Dengue_Cure',\n", 44 | " 'GTrend_Dengue_Med', 'GTrend_Dengue_Sym', 'Mort_Rate', 'MTD_Cases',\n", 45 | " 'Reg_Ave_Temp_NCR', 'Reg_Ave_Rainfall_NCR'],\n", 46 | " dtype='object')" 47 | ] 48 | }, 49 | "execution_count": 3, 50 | "metadata": {}, 51 | "output_type": "execute_result" 52 | } 53 | ], 54 | "source": [ 55 | "Dengue_PH_diff.columns" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 4, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "# CREATE LAGGED VERSIONS OF THE PREDICTORS (MAX OF A QUARTER)\n", 65 | "Dengue_PH_diff['Temp_L1'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(1)\n", 66 | "Dengue_PH_diff['Temp_L2'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(2)\n", 67 | "Dengue_PH_diff['Temp_L3'] = Dengue_PH_diff['Reg_Ave_Temp_NCR'].shift(3)\n", 68 | "\n", 69 | "Dengue_PH_diff['Rain_L1'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(1)\n", 70 | "Dengue_PH_diff['Rain_L2'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(2)\n", 71 | "Dengue_PH_diff['Rain_L3'] = Dengue_PH_diff['Reg_Ave_Rainfall_NCR'].shift(3)\n", 72 | "\n", 73 | "Dengue_PH_diff['GT_Dengue_L1'] = Dengue_PH_diff['GTrend_Dengue'].shift(1)\n", 74 | "Dengue_PH_diff['GT_Dengue_L2'] = Dengue_PH_diff['GTrend_Dengue'].shift(2)\n", 75 | "Dengue_PH_diff['GT_Dengue_L3'] = Dengue_PH_diff['GTrend_Dengue'].shift(3)\n", 76 | "\n", 77 | "Dengue_PH_diff['GT_DengueFvr_L1'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(1)\n", 78 | "Dengue_PH_diff['GT_DengueFvr_L2'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(2)\n", 79 | "Dengue_PH_diff['GT_DengueFvr_L3'] = Dengue_PH_diff['GTrend_Dengue_Fever'].shift(3)\n", 80 | "\n", 81 | "Dengue_PH_diff['GT_DengueCure_L1'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(1)\n", 82 | "Dengue_PH_diff['GT_DengueCure_L2'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(2)\n", 83 | "Dengue_PH_diff['GT_DengueCure_L3'] = Dengue_PH_diff['GTrend_Dengue_Cure'].shift(3)\n", 84 | "\n", 85 | "Dengue_PH_diff['GT_DengueMed_L1'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(1)\n", 86 | "Dengue_PH_diff['GT_DengueMed_L2'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(2)\n", 87 | "Dengue_PH_diff['GT_DengueMed_L3'] = Dengue_PH_diff['GTrend_Dengue_Med'].shift(3)\n", 88 | "\n", 89 | "Dengue_PH_diff['GT_DengueSym_L1'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(1)\n", 90 | "Dengue_PH_diff['GT_DengueSym_L2'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(2)\n", 91 | "Dengue_PH_diff['GT_DengueSym_L3'] = Dengue_PH_diff['GTrend_Dengue_Sym'].shift(3)\n", 92 | "\n", 93 | "Dummies = pd.get_dummies(Dengue_PH_diff.index.month, prefix='m')\n", 94 | "Dengue_PH_diff = Dengue_PH_diff.reset_index()\n", 95 | "Dengue_PH_diff = Dengue_PH_diff.merge(Dummies, left_index=True, right_index=True)\n", 96 | "Dengue_PH_diff.set_index('Date', inplace=True)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 5, 102 | "metadata": {}, 103 | "outputs": [ 104 | { 105 | "name": "stdout", 106 | "output_type": "stream", 107 | "text": [ 108 | "(32, 42)\n", 109 | "(12, 42)\n" 110 | ] 111 | } 112 | ], 113 | "source": [ 114 | "# SPLIT SERIES TO TRAINING AND TEST SETS\n", 115 | "#--- Set 2018 as the test dataframe\n", 116 | "nobs = 12\n", 117 | "df_train, df_test = Dengue_PH_diff[0:-nobs], Dengue_PH_diff[-nobs:]\n", 118 | "df_train = df_train.dropna()\n", 119 | "df_test = df_test.dropna()\n", 120 | "\n", 121 | "# Check size\n", 122 | "print(df_train.shape) \n", 123 | "print(df_test.shape) \n" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 27, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "# PERFORM UNIVARIATE REGRESSION TO TRIM DOWN THE PREDICTORS\n", 133 | "predictor_col = df_train.columns[df_train.columns.str.contains(pat = '_L')]\n", 134 | "pvals = pd.DataFrame()\n", 135 | "for col in predictor_col:\n", 136 | " Y = df_train.MTD_Cases\n", 137 | " X = df_train[col]\n", 138 | " X2 = sm.add_constant(X)\n", 139 | " mod = sm.OLS(Y,X)\n", 140 | " fit = mod.fit()\n", 141 | " pval = fit.summary2().tables[1]['P>|t|']\n", 142 | " pval = pval.to_frame()\n", 143 | " #print(pval)\n", 144 | " pvals = pvals.append(pval)\n", 145 | " \n", 146 | "# RETAIN ONLY THE LAGGED PREDICTORS WITH SIGNIFICANT P-VALUES\n", 147 | "pvals = pvals[pvals['P>|t|'] <= 0.05].reset_index()\n", 148 | "pvals = pvals.rename(columns={'index':'Variable'})\n", 149 | "shortlist_predictor_col = pvals['Variable']\n", 150 | "dummy = pd.Series(df_train.columns[df_train.columns.str.contains(pat = 'm_')])\n", 151 | "#shortlist_predictor_col = shortlist_predictor_col.append(dummy)\n", 152 | "#X = df_train[shortlist_predictor_col].drop(columns=['m_1'],axis=1)\n", 153 | "X = df_train[shortlist_predictor_col]" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "print(type(X))\n", 163 | "print(type(Y))\n", 164 | "print(Y.axes)" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 28, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "name": "stdout", 174 | "output_type": "stream", 175 | "text": [ 176 | "Add Temp_L3 with p-value 0.0135999\n", 177 | "Add GT_DengueMed_L1 with p-value 0.0303546\n", 178 | "resulting features:\n", 179 | "['Temp_L3', 'GT_DengueMed_L1']\n" 180 | ] 181 | }, 182 | { 183 | "name": "stderr", 184 | "output_type": "stream", 185 | "text": [ 186 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:31: FutureWarning: 'argmin' is deprecated, use 'idxmin' instead. The behavior of 'argmin'\n", 187 | "will be corrected to return the positional minimum in the future.\n", 188 | "Use 'series.values.argmin' to get the position of the minimum now.\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "# PERFORM STEPWISE REGRESSION\n", 194 | "def stepwise_selection(X, y, \n", 195 | " initial_list=[], \n", 196 | " threshold_in=0.05, \n", 197 | " threshold_out = 0.10, \n", 198 | " verbose=True):\n", 199 | " \"\"\" Perform a forward-backward feature selection \n", 200 | " based on p-value from statsmodels.api.OLS\n", 201 | " Arguments:\n", 202 | " X - pandas.DataFrame with candidate features\n", 203 | " y - list-like with the target\n", 204 | " initial_list - list of features to start with (column names of X)\n", 205 | " threshold_in - include a feature if its p-value < threshold_in\n", 206 | " threshold_out - exclude a feature if its p-value > threshold_out\n", 207 | " verbose - whether to print the sequence of inclusions and exclusions\n", 208 | " Returns: list of selected features \n", 209 | " Always set threshold_in < threshold_out to avoid infinite looping.\n", 210 | " See https://en.wikipedia.org/wiki/Stepwise_regression for the details\n", 211 | " \"\"\"\n", 212 | " included = list(initial_list)\n", 213 | " while True:\n", 214 | " changed=False\n", 215 | " # forward step\n", 216 | " excluded = list(set(X.columns)-set(included))\n", 217 | " new_pval = pd.Series(index=excluded)\n", 218 | " for new_column in excluded:\n", 219 | " model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()\n", 220 | " new_pval[new_column] = model.pvalues[new_column]\n", 221 | " best_pval = new_pval.min()\n", 222 | " if best_pval < threshold_in:\n", 223 | " best_feature = new_pval.argmin()\n", 224 | " included.append(best_feature)\n", 225 | " changed=True\n", 226 | " if verbose:\n", 227 | " print('Add {:30} with p-value {:.6}'.format(best_feature, best_pval))\n", 228 | "\n", 229 | " # backward step\n", 230 | " model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()\n", 231 | " # use all coefs except intercept\n", 232 | " pvalues = model.pvalues.iloc[1:]\n", 233 | " worst_pval = pvalues.max() # null if pvalues is empty\n", 234 | " if worst_pval > threshold_out:\n", 235 | " changed=True\n", 236 | " worst_feature = pvalues.argmax()\n", 237 | " included.remove(worst_feature)\n", 238 | " if verbose:\n", 239 | " print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))\n", 240 | " if not changed:\n", 241 | " break\n", 242 | " return included\n", 243 | "\n", 244 | "Y = Y.tolist()\n", 245 | "result = stepwise_selection(X, Y)\n", 246 | "\n", 247 | "print('resulting features:')\n", 248 | "print(result)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": {}, 255 | "outputs": [], 256 | "source": [ 257 | "# PRINT THE RESULTS FOR THE FINAL MODEL\n", 258 | "#X = df_train[result]\n", 259 | "X2 = sm.add_constant(X)\n", 260 | "est = sm.OLS(Y, X)\n", 261 | "est2 = est.fit()\n", 262 | "print(est2.summary())" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 26, 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "name": "stdout", 272 | "output_type": "stream", 273 | "text": [ 274 | " OLS Regression Results \n", 275 | "==============================================================================\n", 276 | "Dep. Variable: y R-squared: 0.394\n", 277 | "Model: OLS Adj. R-squared: 0.329\n", 278 | "Method: Least Squares F-statistic: 6.071\n", 279 | "Date: Sun, 20 Oct 2019 Prob (F-statistic): 0.00257\n", 280 | "Time: 01:10:59 Log-Likelihood: -30.093\n", 281 | "No. Observations: 32 AIC: 68.19\n", 282 | "Df Residuals: 28 BIC: 74.05\n", 283 | "Df Model: 3 \n", 284 | "Covariance Type: nonrobust \n", 285 | "===================================================================================\n", 286 | " coef std err t P>|t| [0.025 0.975]\n", 287 | "-----------------------------------------------------------------------------------\n", 288 | "const 0.2936 0.124 2.364 0.025 0.039 0.548\n", 289 | "Temp_L3 7.9225 3.299 2.402 0.023 1.165 14.680\n", 290 | "GT_DengueMed_L1 2.5886 1.064 2.434 0.022 0.410 4.767\n", 291 | "m_11 -0.7942 0.402 -1.974 0.058 -1.618 0.030\n", 292 | "==============================================================================\n", 293 | "Omnibus: 5.227 Durbin-Watson: 2.703\n", 294 | "Prob(Omnibus): 0.073 Jarque-Bera (JB): 3.605\n", 295 | "Skew: 0.674 Prob(JB): 0.165\n", 296 | "Kurtosis: 3.943 Cond. No. 28.3\n", 297 | "==============================================================================\n", 298 | "\n", 299 | "Warnings:\n", 300 | "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "# PRINT THE RESULTS FOR THE FINAL MODEL\n", 306 | "X = df_train[['Temp_L3','GT_DengueMed_L1','m_11']]\n", 307 | "X2 = sm.add_constant(X)\n", 308 | "est = sm.OLS(Y, X2)\n", 309 | "est2 = est.fit()\n", 310 | "print(est2.summary())" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": null, 323 | "metadata": {}, 324 | "outputs": [], 325 | "source": [ 326 | "Y" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": null, 332 | "metadata": {}, 333 | "outputs": [], 334 | "source": [] 335 | } 336 | ], 337 | "metadata": { 338 | "kernelspec": { 339 | "display_name": "Python 3", 340 | "language": "python", 341 | "name": "python3" 342 | }, 343 | "language_info": { 344 | "codemirror_mode": { 345 | "name": "ipython", 346 | "version": 3 347 | }, 348 | "file_extension": ".py", 349 | "mimetype": "text/x-python", 350 | "name": "python", 351 | "nbconvert_exporter": "python", 352 | "pygments_lexer": "ipython3", 353 | "version": "3.6.5" 354 | } 355 | }, 356 | "nbformat": 4, 357 | "nbformat_minor": 2 358 | } 359 | -------------------------------------------------------------------------------- /model/code/PH/pgm1_Data_Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Aggregated.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 4\n", 43 | "MTD_Deaths 4\n", 44 | "Reg_Ave_Temp_NCR 0\n", 45 | "Reg_Ave_Rainfall_NCR 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Med 0\n", 50 | "GTrend_Dengue_Sym 0\n", 51 | "dtype: int64" 52 | ] 53 | }, 54 | "execution_count": 3, 55 | "metadata": {}, 56 | "output_type": "execute_result" 57 | } 58 | ], 59 | "source": [ 60 | "# CHECK FOR MISSING VALUES\n", 61 | "raw_data_trunc.isna().sum()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stderr", 71 | "output_type": "stream", 72 | "text": [ 73 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 74 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 75 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 76 | "\n", 77 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 78 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 79 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 80 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 81 | "\n", 82 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 83 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 84 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 85 | "\n", 86 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 87 | " self._setitem_with_indexer(indexer, value)\n", 88 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 89 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 90 | "\n", 91 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 92 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 93 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 94 | "\n", 95 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 101 | "def seasonal_mean(ts, n, lr=0.7):\n", 102 | " \"\"\"\n", 103 | " Compute the mean of corresponding seasonal periods\n", 104 | " ts: 1D array-like of the time series\n", 105 | " n: Seasonal window length of the time series\n", 106 | " \"\"\"\n", 107 | " out = np.copy(ts)\n", 108 | " for i, val in enumerate(ts):\n", 109 | " if np.isnan(val):\n", 110 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 111 | " if np.isnan(np.nanmean(ts_seas)):\n", 112 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 113 | " out[i] = np.nanmean(ts_seas) * lr\n", 114 | " return out\n", 115 | "\n", 116 | "#--- Get the seasonal means\n", 117 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 118 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 119 | "\n", 120 | "#--- Replace the missing observations with the seasonal means\n", 121 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 123 | "\n", 124 | "#--- Drop the seasonal mean columns\n", 125 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 5, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 135 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 136 | "\n", 137 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 138 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 139 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 149 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_PH_Clean.pickle')" 150 | ] 151 | } 152 | ], 153 | "metadata": { 154 | "kernelspec": { 155 | "display_name": "Python 3", 156 | "language": "python", 157 | "name": "python3" 158 | }, 159 | "language_info": { 160 | "codemirror_mode": { 161 | "name": "ipython", 162 | "version": 3 163 | }, 164 | "file_extension": ".py", 165 | "mimetype": "text/x-python", 166 | "name": "python", 167 | "nbconvert_exporter": "python", 168 | "pygments_lexer": "ipython3", 169 | "version": "3.6.5" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /model/code/Western Visayas/.ipynb_checkpoints/pgm1_Data_Preparation-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_Western_Visayas.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 6\n", 43 | "MTD_Deaths 6\n", 44 | "Reg_Ave_Temp_WestVis 0\n", 45 | "Reg_Ave_Rainfall_WestVis 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Sym 0\n", 50 | "dtype: int64" 51 | ] 52 | }, 53 | "execution_count": 3, 54 | "metadata": {}, 55 | "output_type": "execute_result" 56 | } 57 | ], 58 | "source": [ 59 | "# CHECK FOR MISSING VALUES\n", 60 | "raw_data_trunc.isna().sum()" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stderr", 70 | "output_type": "stream", 71 | "text": [ 72 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:12: RuntimeWarning: Mean of empty slice\n", 73 | " if sys.path[0] == '':\n", 74 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 75 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 76 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 77 | "\n", 78 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 79 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 80 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 81 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 82 | "\n", 83 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 84 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 85 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 86 | "\n", 87 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 88 | " self._setitem_with_indexer(indexer, value)\n", 89 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 90 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 91 | "\n", 92 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 93 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 94 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 95 | "\n", 96 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 102 | "def seasonal_mean(ts, n, lr=0.7):\n", 103 | " \"\"\"\n", 104 | " Compute the mean of corresponding seasonal periods\n", 105 | " ts: 1D array-like of the time series\n", 106 | " n: Seasonal window length of the time series\n", 107 | " \"\"\"\n", 108 | " out = np.copy(ts)\n", 109 | " for i, val in enumerate(ts):\n", 110 | " if np.isnan(val):\n", 111 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 112 | " if np.isnan(np.nanmean(ts_seas)):\n", 113 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 114 | " out[i] = np.nanmean(ts_seas) * lr\n", 115 | " return out\n", 116 | "\n", 117 | "#--- Get the seasonal means\n", 118 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 119 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 120 | "\n", 121 | "#--- Replace the missing observations with the seasonal means\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 123 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 124 | "\n", 125 | "#--- Drop the seasonal mean columns\n", 126 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 5, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 136 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 137 | "\n", 138 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 139 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 140 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 6, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 150 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_WestVis_Clean.pickle')" 151 | ] 152 | } 153 | ], 154 | "metadata": { 155 | "kernelspec": { 156 | "display_name": "Python 3", 157 | "language": "python", 158 | "name": "python3" 159 | }, 160 | "language_info": { 161 | "codemirror_mode": { 162 | "name": "ipython", 163 | "version": 3 164 | }, 165 | "file_extension": ".py", 166 | "mimetype": "text/x-python", 167 | "name": "python", 168 | "nbconvert_exporter": "python", 169 | "pygments_lexer": "ipython3", 170 | "version": "3.6.5" 171 | } 172 | }, 173 | "nbformat": 4, 174 | "nbformat_minor": 2 175 | } 176 | -------------------------------------------------------------------------------- /model/code/Western Visayas/pgm1_Data_Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# IMPORT LIBRARIES\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from dateutil.parser import parse \n", 13 | "import matplotlib as mpl\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "import seaborn as sns\n", 16 | "from scipy.interpolate import interp1d\n", 17 | "from sklearn.metrics import mean_squared_error" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# READ DATASET\n", 27 | "raw_data = pd.read_excel(\"C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_Western_Visayas.xlsx\")\n", 28 | "raw_data = raw_data.set_index('Date')\n", 29 | "\n", 30 | "# TRUNCATE DATA TO INCLUDE OBSERVATIONS UP TO 2018 ONLY\n", 31 | "raw_data_trunc = raw_data[raw_data.index <= '2018-12-31']" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": [ 42 | "MTD_Cases 6\n", 43 | "MTD_Deaths 6\n", 44 | "Reg_Ave_Temp_WestVis 0\n", 45 | "Reg_Ave_Rainfall_WestVis 0\n", 46 | "GTrend_Dengue 0\n", 47 | "GTrend_Dengue_Fever 0\n", 48 | "GTrend_Dengue_Cure 0\n", 49 | "GTrend_Dengue_Sym 0\n", 50 | "dtype: int64" 51 | ] 52 | }, 53 | "execution_count": 3, 54 | "metadata": {}, 55 | "output_type": "execute_result" 56 | } 57 | ], 58 | "source": [ 59 | "# CHECK FOR MISSING VALUES\n", 60 | "raw_data_trunc.isna().sum()" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stderr", 70 | "output_type": "stream", 71 | "text": [ 72 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:12: RuntimeWarning: Mean of empty slice\n", 73 | " if sys.path[0] == '':\n", 74 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: SettingWithCopyWarning: \n", 75 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 76 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 77 | "\n", 78 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 79 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:19: SettingWithCopyWarning: \n", 80 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 81 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 82 | "\n", 83 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 84 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:189: SettingWithCopyWarning: \n", 85 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 86 | "\n", 87 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 88 | " self._setitem_with_indexer(indexer, value)\n", 89 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:22: SettingWithCopyWarning: \n", 90 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 91 | "\n", 92 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n", 93 | "E:\\Users\\Claire\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:23: SettingWithCopyWarning: \n", 94 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 95 | "\n", 96 | "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "# TREAT THE MISSING VALUES USING SEASONAL MEAN\n", 102 | "def seasonal_mean(ts, n, lr=0.7):\n", 103 | " \"\"\"\n", 104 | " Compute the mean of corresponding seasonal periods\n", 105 | " ts: 1D array-like of the time series\n", 106 | " n: Seasonal window length of the time series\n", 107 | " \"\"\"\n", 108 | " out = np.copy(ts)\n", 109 | " for i, val in enumerate(ts):\n", 110 | " if np.isnan(val):\n", 111 | " ts_seas = ts[i-1::-n] # previous seasons only\n", 112 | " if np.isnan(np.nanmean(ts_seas)):\n", 113 | " ts_seas = np.concatenate([ts[i-1::-n], ts[i::n]]) # previous and forward\n", 114 | " out[i] = np.nanmean(ts_seas) * lr\n", 115 | " return out\n", 116 | "\n", 117 | "#--- Get the seasonal means\n", 118 | "raw_data_trunc['seas_mn_cases'] = seasonal_mean(raw_data_trunc.MTD_Cases, n=12, lr=1.25)\n", 119 | "raw_data_trunc['seas_mn_deaths'] = seasonal_mean(raw_data_trunc.MTD_Deaths, n=12, lr=1.25)\n", 120 | "\n", 121 | "#--- Replace the missing observations with the seasonal means\n", 122 | "raw_data_trunc.loc[raw_data_trunc['MTD_Cases'].isnull(),'MTD_Cases'] = raw_data_trunc['seas_mn_cases']\n", 123 | "raw_data_trunc.loc[raw_data_trunc['MTD_Deaths'].isnull(),'MTD_Deaths'] = raw_data_trunc['seas_mn_deaths']\n", 124 | "\n", 125 | "#--- Drop the seasonal mean columns\n", 126 | "raw_data_trunc = raw_data_trunc.drop(columns=['seas_mn_cases','seas_mn_deaths'])" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 5, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "# CREATE A NEW COLUMN FOR MORTALITY RATE\n", 136 | "raw_data_trunc['Mort_Rate'] = raw_data_trunc.MTD_Deaths / raw_data_trunc.MTD_Cases\n", 137 | "\n", 138 | "# TRANSFORM GOOGLE TREND COLUMNS TO PERCENTAGES\n", 139 | "gtrend_cols = [col for col in raw_data_trunc if col.startswith('GTrend')]\n", 140 | "raw_data_trunc[gtrend_cols]= raw_data_trunc[gtrend_cols].apply(lambda t: t / 100)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 6, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "# STORE UPDATED DATAFRAME TO PICKLE\n", 150 | "raw_data_trunc.to_pickle('C:/Users/Claire/Documents/GitHub/nasa_hack/model/datasets/Dengue_WestVis_Clean.pickle')" 151 | ] 152 | } 153 | ], 154 | "metadata": { 155 | "kernelspec": { 156 | "display_name": "Python 3", 157 | "language": "python", 158 | "name": "python3" 159 | }, 160 | "language_info": { 161 | "codemirror_mode": { 162 | "name": "ipython", 163 | "version": 3 164 | }, 165 | "file_extension": ".py", 166 | "mimetype": "text/x-python", 167 | "name": "python", 168 | "nbconvert_exporter": "python", 169 | "pygments_lexer": "ipython3", 170 | "version": "3.6.5" 171 | } 172 | }, 173 | "nbformat": 4, 174 | "nbformat_minor": 2 175 | } 176 | -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM_Clean.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM_Clean.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM_Diff.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM_Diff.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM_Fct.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM_Fct.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM_Fct.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM_Fct.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_ARMM_Top3_Models.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_ARMM_Top3_Models.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_EastVis_Clean.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_EastVis_Clean.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_EastVis_Diff.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_EastVis_Diff.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_EastVis_Fct.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_EastVis_Fct.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_EastVis_Fct.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_EastVis_Fct.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_EastVis_Top3_Models.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_EastVis_Top3_Models.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_Eastern_Visayas.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_Eastern_Visayas.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR_Clean.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR_Clean.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR_Diff.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR_Diff.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR_Fct.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR_Fct.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR_Fct.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR_Fct.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_NCR_Top3_Models.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_NCR_Top3_Models.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Aggregated.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Aggregated.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Clean.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Clean.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Diff.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Diff.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Fct.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Fct.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Fct.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Fct.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Fct2.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Fct2.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Results.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Results.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_PH_Top3_Models.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_PH_Top3_Models.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_WestVis_Clean.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_WestVis_Clean.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_WestVis_Diff.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_WestVis_Diff.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_WestVis_Fct.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_WestVis_Fct.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_WestVis_Fct.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_WestVis_Fct.xlsx -------------------------------------------------------------------------------- /model/datasets/Dengue_WestVis_Top3_Models.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_WestVis_Top3_Models.pickle -------------------------------------------------------------------------------- /model/datasets/Dengue_Western_Visayas.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Dengue_Western_Visayas.xlsx -------------------------------------------------------------------------------- /model/datasets/Model_Results.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/datasets/Model_Results.xlsx -------------------------------------------------------------------------------- /model/datasets/~$Dengue_PH_Fct.xlsx: -------------------------------------------------------------------------------- 1 | Claire Claire -------------------------------------------------------------------------------- /model/dengue_doh/2013 Dengue Morbidity Week 52.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2013 Dengue Morbidity Week 52.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2014 Dengue Morbidity Week 52_0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2014 Dengue Morbidity Week 52_0.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2015 Dengue Morbidity Week 52.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2015 Dengue Morbidity Week 52.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_MW1-MW52.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_MW1-MW52.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_W29_0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_W29_0.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_W34.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_W34.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_W36.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_W36.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_W45.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_W45.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2016_Dengue_W47_1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2016_Dengue_W47_1.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017 Dengue Morbidity Week 1-35.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017 Dengue Morbidity Week 1-35.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue MW 1-MW44_0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue MW 1-MW44_0.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_ MW1-MW26.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_ MW1-MW26.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-31.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-31.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-MW18.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-MW18.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-MW20.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-MW20.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-MW22.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-MW22.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-MW39.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-MW39.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_MW1-MW48.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_MW1-MW48.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_W1_10.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_W1_10.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_W1_11.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_W1_11.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_W1_13_0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_W1_13_0.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_W1_5_0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_W1_5_0.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2017_Dengue_W1_9.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2017_Dengue_W1_9.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018 Dengue Surveillance Report MW 1-10_5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018 Dengue Surveillance Report MW 1-10_5.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018 Dengue Surveillance Report MW1-5_1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018 Dengue Surveillance Report MW1-5_1.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018 Dengue Surveillance Report MW1-6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018 Dengue Surveillance Report MW1-6.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018 Dengue Surveillance Report MW1-7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018 Dengue Surveillance Report MW1-7.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018 Dengue Surveillance Report MW1-8.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018 Dengue Surveillance Report MW1-8.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018_Monthly_Dengue_Report _N10.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018_Monthly_Dengue_Report _N10.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018_Monthly_Dengue_Report _N11.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018_Monthly_Dengue_Report _N11.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2018_Monthly_Dengue_Report _N12.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2018_Monthly_Dengue_Report _N12.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2019 Dengue Monthly Report No. 7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2019 Dengue Monthly Report No. 7.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2019 Dengue Monthly Report No. 8.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2019 Dengue Monthly Report No. 8.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2019_Monthly_Dengue_Report _N1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2019_Monthly_Dengue_Report _N1.pdf -------------------------------------------------------------------------------- /model/dengue_doh/2019_Monthly_Dengue_Report _N2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/2019_Monthly_Dengue_Report _N2.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW12.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW12.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW13.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW13.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW14.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW14.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW19.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW19.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW21.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW21.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW22.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW22.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW23.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW23.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW25.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW25.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW26.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW26.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW27.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW27.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW28.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW28.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW31.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW31.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW33.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW33.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW4.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW5.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE MW7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE MW7.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE38_compressed.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE38_compressed.pdf -------------------------------------------------------------------------------- /model/dengue_doh/DENGUE44_compressed.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/DENGUE44_compressed.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report No. 3.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report No. 3.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report No. 4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report No. 4.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report No. 5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report No. 5.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report No. 6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report No. 6.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report No.6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report No.6.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Monthly Report_MW1-MW30_2018_No.7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Monthly Report_MW1-MW30_2018_No.7.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue Montlhy Report No.4 (April).pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue Montlhy Report No.4 (April).pdf -------------------------------------------------------------------------------- /model/dengue_doh/Dengue_Monthly_Report_No.5_2018_May.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Dengue_Monthly_Report_No.5_2018_May.pdf -------------------------------------------------------------------------------- /model/dengue_doh/Monthly Dengue Report (MW13).pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/Monthly Dengue Report (MW13).pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengue27.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengue27.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengue28.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengue28.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengue32.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengue32.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengue33.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengue33.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw10.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw10.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw13.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw13.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw15.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw15.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw18.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw18.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw21.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw21.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw23.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw23.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw24.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw24.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw26.compressed.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw26.compressed.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw28.compressed.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw28.compressed.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw6.pdf -------------------------------------------------------------------------------- /model/dengue_doh/dengueMw9.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/dengue_doh/dengueMw9.pdf -------------------------------------------------------------------------------- /model/google_trends/Dengue Searches for sharing with Doc Ligot.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/model/google_trends/Dengue Searches for sharing with Doc Ligot.xlsx -------------------------------------------------------------------------------- /model/google_trends/dengue.csv: -------------------------------------------------------------------------------- 1 | Category: All categories 2 | 3 | Week,dengue: (Philippines) 4 | 2014-08-31,15 5 | 2014-09-07,16 6 | 2014-09-14,14 7 | 2014-09-21,13 8 | 2014-09-28,13 9 | 2014-10-05,13 10 | 2014-10-12,16 11 | 2014-10-19,14 12 | 2014-10-26,12 13 | 2014-11-02,17 14 | 2014-11-09,15 15 | 2014-11-16,14 16 | 2014-11-23,14 17 | 2014-11-30,12 18 | 2014-12-07,11 19 | 2014-12-14,10 20 | 2014-12-21,9 21 | 2014-12-28,10 22 | 2015-01-04,12 23 | 2015-01-11,11 24 | 2015-01-18,11 25 | 2015-01-25,11 26 | 2015-02-01,10 27 | 2015-02-08,10 28 | 2015-02-15,11 29 | 2015-02-22,10 30 | 2015-03-01,9 31 | 2015-03-08,8 32 | 2015-03-15,7 33 | 2015-03-22,6 34 | 2015-03-29,5 35 | 2015-04-05,6 36 | 2015-04-12,6 37 | 2015-04-19,5 38 | 2015-04-26,6 39 | 2015-05-03,4 40 | 2015-05-10,5 41 | 2015-05-17,5 42 | 2015-05-24,7 43 | 2015-05-31,6 44 | 2015-06-07,6 45 | 2015-06-14,8 46 | 2015-06-21,10 47 | 2015-06-28,16 48 | 2015-07-05,15 49 | 2015-07-12,13 50 | 2015-07-19,16 51 | 2015-07-26,17 52 | 2015-08-02,18 53 | 2015-08-09,21 54 | 2015-08-16,19 55 | 2015-08-23,21 56 | 2015-08-30,22 57 | 2015-09-06,25 58 | 2015-09-13,25 59 | 2015-09-20,29 60 | 2015-09-27,39 61 | 2015-10-04,43 62 | 2015-10-11,35 63 | 2015-10-18,29 64 | 2015-10-25,29 65 | 2015-11-01,28 66 | 2015-11-08,25 67 | 2015-11-15,21 68 | 2015-11-22,23 69 | 2015-11-29,18 70 | 2015-12-06,17 71 | 2015-12-13,15 72 | 2015-12-20,12 73 | 2015-12-27,13 74 | 2016-01-03,19 75 | 2016-01-10,16 76 | 2016-01-17,17 77 | 2016-01-24,16 78 | 2016-01-31,13 79 | 2016-02-07,16 80 | 2016-02-14,14 81 | 2016-02-21,13 82 | 2016-02-28,11 83 | 2016-03-06,12 84 | 2016-03-13,11 85 | 2016-03-20,9 86 | 2016-03-27,9 87 | 2016-04-03,9 88 | 2016-04-10,7 89 | 2016-04-17,7 90 | 2016-04-24,7 91 | 2016-05-01,6 92 | 2016-05-08,5 93 | 2016-05-15,6 94 | 2016-05-22,6 95 | 2016-05-29,7 96 | 2016-06-05,8 97 | 2016-06-12,10 98 | 2016-06-19,13 99 | 2016-06-26,13 100 | 2016-07-03,15 101 | 2016-07-10,17 102 | 2016-07-17,20 103 | 2016-07-24,25 104 | 2016-07-31,23 105 | 2016-08-07,27 106 | 2016-08-14,28 107 | 2016-08-21,29 108 | 2016-08-28,32 109 | 2016-09-04,27 110 | 2016-09-11,23 111 | 2016-09-18,22 112 | 2016-09-25,20 113 | 2016-10-02,19 114 | 2016-10-09,21 115 | 2016-10-16,18 116 | 2016-10-23,17 117 | 2016-10-30,15 118 | 2016-11-06,17 119 | 2016-11-13,15 120 | 2016-11-20,16 121 | 2016-11-27,16 122 | 2016-12-04,14 123 | 2016-12-11,14 124 | 2016-12-18,10 125 | 2016-12-25,10 126 | 2017-01-01,11 127 | 2017-01-08,13 128 | 2017-01-15,12 129 | 2017-01-22,9 130 | 2017-01-29,11 131 | 2017-02-05,11 132 | 2017-02-12,9 133 | 2017-02-19,10 134 | 2017-02-26,9 135 | 2017-03-05,10 136 | 2017-03-12,9 137 | 2017-03-19,8 138 | 2017-03-26,7 139 | 2017-04-02,5 140 | 2017-04-09,5 141 | 2017-04-16,6 142 | 2017-04-23,6 143 | 2017-04-30,6 144 | 2017-05-07,5 145 | 2017-05-14,6 146 | 2017-05-21,6 147 | 2017-05-28,8 148 | 2017-06-04,8 149 | 2017-06-11,9 150 | 2017-06-18,11 151 | 2017-06-25,12 152 | 2017-07-02,11 153 | 2017-07-09,13 154 | 2017-07-16,15 155 | 2017-07-23,16 156 | 2017-07-30,17 157 | 2017-08-06,19 158 | 2017-08-13,18 159 | 2017-08-20,20 160 | 2017-08-27,16 161 | 2017-09-03,18 162 | 2017-09-10,15 163 | 2017-09-17,16 164 | 2017-09-24,15 165 | 2017-10-01,14 166 | 2017-10-08,14 167 | 2017-10-15,15 168 | 2017-10-22,11 169 | 2017-10-29,13 170 | 2017-11-05,13 171 | 2017-11-12,14 172 | 2017-11-19,13 173 | 2017-11-26,20 174 | 2017-12-03,35 175 | 2017-12-10,27 176 | 2017-12-17,15 177 | 2017-12-24,14 178 | 2017-12-31,17 179 | 2018-01-07,18 180 | 2018-01-14,17 181 | 2018-01-21,16 182 | 2018-01-28,18 183 | 2018-02-04,14 184 | 2018-02-11,12 185 | 2018-02-18,14 186 | 2018-02-25,13 187 | 2018-03-04,10 188 | 2018-03-11,10 189 | 2018-03-18,8 190 | 2018-03-25,5 191 | 2018-04-01,6 192 | 2018-04-08,7 193 | 2018-04-15,5 194 | 2018-04-22,5 195 | 2018-04-29,6 196 | 2018-05-06,6 197 | 2018-05-13,5 198 | 2018-05-20,7 199 | 2018-05-27,7 200 | 2018-06-03,9 201 | 2018-06-10,9 202 | 2018-06-17,9 203 | 2018-06-24,12 204 | 2018-07-01,14 205 | 2018-07-08,19 206 | 2018-07-15,21 207 | 2018-07-22,23 208 | 2018-07-29,22 209 | 2018-08-05,23 210 | 2018-08-12,24 211 | 2018-08-19,21 212 | 2018-08-26,21 213 | 2018-09-02,17 214 | 2018-09-09,19 215 | 2018-09-16,21 216 | 2018-09-23,19 217 | 2018-09-30,19 218 | 2018-10-07,18 219 | 2018-10-14,18 220 | 2018-10-21,17 221 | 2018-10-28,17 222 | 2018-11-04,18 223 | 2018-11-11,18 224 | 2018-11-18,18 225 | 2018-11-25,18 226 | 2018-12-02,17 227 | 2018-12-09,19 228 | 2018-12-16,16 229 | 2018-12-23,18 230 | 2018-12-30,23 231 | 2019-01-06,26 232 | 2019-01-13,26 233 | 2019-01-20,22 234 | 2019-01-27,23 235 | 2019-02-03,18 236 | 2019-02-10,18 237 | 2019-02-17,17 238 | 2019-02-24,16 239 | 2019-03-03,13 240 | 2019-03-10,13 241 | 2019-03-17,11 242 | 2019-03-24,8 243 | 2019-03-31,8 244 | 2019-04-07,8 245 | 2019-04-14,7 246 | 2019-04-21,7 247 | 2019-04-28,7 248 | 2019-05-05,7 249 | 2019-05-12,7 250 | 2019-05-19,8 251 | 2019-05-26,8 252 | 2019-06-02,10 253 | 2019-06-09,10 254 | 2019-06-16,14 255 | 2019-06-23,19 256 | 2019-06-30,24 257 | 2019-07-07,31 258 | 2019-07-14,54 259 | 2019-07-21,65 260 | 2019-07-28,64 261 | 2019-08-04,82 262 | 2019-08-11,81 263 | 2019-08-18,100 264 | -------------------------------------------------------------------------------- /model/google_trends/dengue_cure.csv: -------------------------------------------------------------------------------- 1 | Category: All categories 2 | 3 | Week,dengue cure: (Philippines) 4 | 2014-08-31,19 5 | 2014-09-07,26 6 | 2014-09-14,37 7 | 2014-09-21,26 8 | 2014-09-28,32 9 | 2014-10-05,13 10 | 2014-10-12,25 11 | 2014-10-19,13 12 | 2014-10-26,0 13 | 2014-11-02,39 14 | 2014-11-09,38 15 | 2014-11-16,13 16 | 2014-11-23,25 17 | 2014-11-30,25 18 | 2014-12-07,12 19 | 2014-12-14,27 20 | 2014-12-21,29 21 | 2014-12-28,14 22 | 2015-01-04,12 23 | 2015-01-11,36 24 | 2015-01-18,12 25 | 2015-01-25,17 26 | 2015-02-01,17 27 | 2015-02-08,12 28 | 2015-02-15,12 29 | 2015-02-22,34 30 | 2015-03-01,11 31 | 2015-03-08,17 32 | 2015-03-15,11 33 | 2015-03-22,0 34 | 2015-03-29,0 35 | 2015-04-05,0 36 | 2015-04-12,12 37 | 2015-04-19,0 38 | 2015-04-26,12 39 | 2015-05-03,0 40 | 2015-05-10,12 41 | 2015-05-17,12 42 | 2015-05-24,0 43 | 2015-05-31,11 44 | 2015-06-07,11 45 | 2015-06-14,11 46 | 2015-06-21,10 47 | 2015-06-28,21 48 | 2015-07-05,10 49 | 2015-07-12,21 50 | 2015-07-19,21 51 | 2015-07-26,42 52 | 2015-08-02,32 53 | 2015-08-09,26 54 | 2015-08-16,32 55 | 2015-08-23,21 56 | 2015-08-30,32 57 | 2015-09-06,42 58 | 2015-09-13,53 59 | 2015-09-20,52 60 | 2015-09-27,73 61 | 2015-10-04,41 62 | 2015-10-11,87 63 | 2015-10-18,43 64 | 2015-10-25,33 65 | 2015-11-01,43 66 | 2015-11-08,47 67 | 2015-11-15,47 68 | 2015-11-22,31 69 | 2015-11-29,21 70 | 2015-12-06,36 71 | 2015-12-13,22 72 | 2015-12-20,38 73 | 2015-12-27,49 74 | 2016-01-03,43 75 | 2016-01-10,53 76 | 2016-01-17,32 77 | 2016-01-24,26 78 | 2016-01-31,31 79 | 2016-02-07,0 80 | 2016-02-14,26 81 | 2016-02-21,15 82 | 2016-02-28,10 83 | 2016-03-06,10 84 | 2016-03-13,10 85 | 2016-03-20,17 86 | 2016-03-27,11 87 | 2016-04-03,27 88 | 2016-04-10,11 89 | 2016-04-17,11 90 | 2016-04-24,32 91 | 2016-05-01,21 92 | 2016-05-08,0 93 | 2016-05-15,11 94 | 2016-05-22,0 95 | 2016-05-29,11 96 | 2016-06-05,11 97 | 2016-06-12,26 98 | 2016-06-19,25 99 | 2016-06-26,35 100 | 2016-07-03,25 101 | 2016-07-10,21 102 | 2016-07-17,32 103 | 2016-07-24,32 104 | 2016-07-31,42 105 | 2016-08-07,42 106 | 2016-08-14,61 107 | 2016-08-21,33 108 | 2016-08-28,33 109 | 2016-09-04,43 110 | 2016-09-11,44 111 | 2016-09-18,59 112 | 2016-09-25,11 113 | 2016-10-02,37 114 | 2016-10-09,21 115 | 2016-10-16,26 116 | 2016-10-23,28 117 | 2016-10-30,35 118 | 2016-11-06,21 119 | 2016-11-13,21 120 | 2016-11-20,16 121 | 2016-11-27,10 122 | 2016-12-04,15 123 | 2016-12-11,32 124 | 2016-12-18,35 125 | 2016-12-25,18 126 | 2017-01-01,33 127 | 2017-01-08,20 128 | 2017-01-15,25 129 | 2017-01-22,10 130 | 2017-01-29,25 131 | 2017-02-05,20 132 | 2017-02-12,20 133 | 2017-02-19,10 134 | 2017-02-26,19 135 | 2017-03-05,28 136 | 2017-03-12,19 137 | 2017-03-19,10 138 | 2017-03-26,10 139 | 2017-04-02,0 140 | 2017-04-09,33 141 | 2017-04-16,0 142 | 2017-04-23,10 143 | 2017-04-30,0 144 | 2017-05-07,0 145 | 2017-05-14,11 146 | 2017-05-21,0 147 | 2017-05-28,0 148 | 2017-06-04,0 149 | 2017-06-11,16 150 | 2017-06-18,10 151 | 2017-06-25,11 152 | 2017-07-02,15 153 | 2017-07-09,10 154 | 2017-07-16,15 155 | 2017-07-23,20 156 | 2017-07-30,26 157 | 2017-08-06,15 158 | 2017-08-13,61 159 | 2017-08-20,15 160 | 2017-08-27,37 161 | 2017-09-03,36 162 | 2017-09-10,35 163 | 2017-09-17,30 164 | 2017-09-24,25 165 | 2017-10-01,15 166 | 2017-10-08,29 167 | 2017-10-15,25 168 | 2017-10-22,20 169 | 2017-10-29,49 170 | 2017-11-05,0 171 | 2017-11-12,20 172 | 2017-11-19,19 173 | 2017-11-26,19 174 | 2017-12-03,34 175 | 2017-12-10,30 176 | 2017-12-17,27 177 | 2017-12-24,23 178 | 2017-12-31,26 179 | 2018-01-07,19 180 | 2018-01-14,38 181 | 2018-01-21,28 182 | 2018-01-28,24 183 | 2018-02-04,28 184 | 2018-02-11,0 185 | 2018-02-18,10 186 | 2018-02-25,9 187 | 2018-03-04,9 188 | 2018-03-11,9 189 | 2018-03-18,10 190 | 2018-03-25,11 191 | 2018-04-01,10 192 | 2018-04-08,19 193 | 2018-04-15,10 194 | 2018-04-22,9 195 | 2018-04-29,10 196 | 2018-05-06,0 197 | 2018-05-13,10 198 | 2018-05-20,0 199 | 2018-05-27,9 200 | 2018-06-03,9 201 | 2018-06-10,27 202 | 2018-06-17,13 203 | 2018-06-24,17 204 | 2018-07-01,21 205 | 2018-07-08,21 206 | 2018-07-15,21 207 | 2018-07-22,60 208 | 2018-07-29,38 209 | 2018-08-05,38 210 | 2018-08-12,26 211 | 2018-08-19,35 212 | 2018-08-26,31 213 | 2018-09-02,17 214 | 2018-09-09,33 215 | 2018-09-16,9 216 | 2018-09-23,17 217 | 2018-09-30,26 218 | 2018-10-07,29 219 | 2018-10-14,26 220 | 2018-10-21,26 221 | 2018-10-28,44 222 | 2018-11-04,22 223 | 2018-11-11,21 224 | 2018-11-18,22 225 | 2018-11-25,18 226 | 2018-12-02,35 227 | 2018-12-09,22 228 | 2018-12-16,28 229 | 2018-12-23,20 230 | 2018-12-30,14 231 | 2019-01-06,21 232 | 2019-01-13,46 233 | 2019-01-20,37 234 | 2019-01-27,25 235 | 2019-02-03,24 236 | 2019-02-10,16 237 | 2019-02-17,32 238 | 2019-02-24,12 239 | 2019-03-03,32 240 | 2019-03-10,12 241 | 2019-03-17,12 242 | 2019-03-24,8 243 | 2019-03-31,18 244 | 2019-04-07,18 245 | 2019-04-14,14 246 | 2019-04-21,17 247 | 2019-04-28,17 248 | 2019-05-05,8 249 | 2019-05-12,8 250 | 2019-05-19,8 251 | 2019-05-26,12 252 | 2019-06-02,24 253 | 2019-06-09,54 254 | 2019-06-16,19 255 | 2019-06-23,30 256 | 2019-06-30,23 257 | 2019-07-07,41 258 | 2019-07-14,49 259 | 2019-07-21,54 260 | 2019-07-28,100 261 | 2019-08-04,80 262 | 2019-08-11,73 263 | 2019-08-18,95 264 | -------------------------------------------------------------------------------- /model/google_trends/dengue_fever.csv: -------------------------------------------------------------------------------- 1 | Category: All categories 2 | 3 | Week,dengue fever: (Philippines) 4 | 2014-08-31,36 5 | 2014-09-07,35 6 | 2014-09-14,39 7 | 2014-09-21,33 8 | 2014-09-28,29 9 | 2014-10-05,23 10 | 2014-10-12,26 11 | 2014-10-19,27 12 | 2014-10-26,28 13 | 2014-11-02,36 14 | 2014-11-09,47 15 | 2014-11-16,28 16 | 2014-11-23,35 17 | 2014-11-30,35 18 | 2014-12-07,29 19 | 2014-12-14,26 20 | 2014-12-21,12 21 | 2014-12-28,26 22 | 2015-01-04,37 23 | 2015-01-11,24 24 | 2015-01-18,27 25 | 2015-01-25,33 26 | 2015-02-01,21 27 | 2015-02-08,23 28 | 2015-02-15,27 29 | 2015-02-22,19 30 | 2015-03-01,19 31 | 2015-03-08,20 32 | 2015-03-15,12 33 | 2015-03-22,13 34 | 2015-03-29,12 35 | 2015-04-05,17 36 | 2015-04-12,12 37 | 2015-04-19,12 38 | 2015-04-26,13 39 | 2015-05-03,13 40 | 2015-05-10,16 41 | 2015-05-17,8 42 | 2015-05-24,21 43 | 2015-05-31,19 44 | 2015-06-07,13 45 | 2015-06-14,19 46 | 2015-06-21,20 47 | 2015-06-28,32 48 | 2015-07-05,35 49 | 2015-07-12,26 50 | 2015-07-19,45 51 | 2015-07-26,38 52 | 2015-08-02,31 53 | 2015-08-09,50 54 | 2015-08-16,45 55 | 2015-08-23,44 56 | 2015-08-30,55 57 | 2015-09-06,49 58 | 2015-09-13,61 59 | 2015-09-20,52 60 | 2015-09-27,76 61 | 2015-10-04,80 62 | 2015-10-11,68 63 | 2015-10-18,67 64 | 2015-10-25,56 65 | 2015-11-01,45 66 | 2015-11-08,58 67 | 2015-11-15,42 68 | 2015-11-22,46 69 | 2015-11-29,36 70 | 2015-12-06,30 71 | 2015-12-13,32 72 | 2015-12-20,21 73 | 2015-12-27,25 74 | 2016-01-03,27 75 | 2016-01-10,40 76 | 2016-01-17,34 77 | 2016-01-24,30 78 | 2016-01-31,22 79 | 2016-02-07,32 80 | 2016-02-14,20 81 | 2016-02-21,29 82 | 2016-02-28,24 83 | 2016-03-06,25 84 | 2016-03-13,17 85 | 2016-03-20,19 86 | 2016-03-27,13 87 | 2016-04-03,10 88 | 2016-04-10,12 89 | 2016-04-17,15 90 | 2016-04-24,9 91 | 2016-05-01,10 92 | 2016-05-08,10 93 | 2016-05-15,8 94 | 2016-05-22,12 95 | 2016-05-29,24 96 | 2016-06-05,16 97 | 2016-06-12,19 98 | 2016-06-19,14 99 | 2016-06-26,25 100 | 2016-07-03,30 101 | 2016-07-10,28 102 | 2016-07-17,41 103 | 2016-07-24,50 104 | 2016-07-31,46 105 | 2016-08-07,39 106 | 2016-08-14,46 107 | 2016-08-21,43 108 | 2016-08-28,54 109 | 2016-09-04,50 110 | 2016-09-11,47 111 | 2016-09-18,44 112 | 2016-09-25,39 113 | 2016-10-02,39 114 | 2016-10-09,32 115 | 2016-10-16,37 116 | 2016-10-23,30 117 | 2016-10-30,24 118 | 2016-11-06,27 119 | 2016-11-13,29 120 | 2016-11-20,31 121 | 2016-11-27,25 122 | 2016-12-04,23 123 | 2016-12-11,24 124 | 2016-12-18,18 125 | 2016-12-25,17 126 | 2017-01-01,25 127 | 2017-01-08,22 128 | 2017-01-15,24 129 | 2017-01-22,19 130 | 2017-01-29,13 131 | 2017-02-05,20 132 | 2017-02-12,20 133 | 2017-02-19,17 134 | 2017-02-26,18 135 | 2017-03-05,20 136 | 2017-03-12,22 137 | 2017-03-19,25 138 | 2017-03-26,17 139 | 2017-04-02,14 140 | 2017-04-09,15 141 | 2017-04-16,6 142 | 2017-04-23,12 143 | 2017-04-30,14 144 | 2017-05-07,11 145 | 2017-05-14,16 146 | 2017-05-21,10 147 | 2017-05-28,22 148 | 2017-06-04,12 149 | 2017-06-11,21 150 | 2017-06-18,17 151 | 2017-06-25,22 152 | 2017-07-02,19 153 | 2017-07-09,27 154 | 2017-07-16,29 155 | 2017-07-23,26 156 | 2017-07-30,34 157 | 2017-08-06,44 158 | 2017-08-13,35 159 | 2017-08-20,37 160 | 2017-08-27,27 161 | 2017-09-03,37 162 | 2017-09-10,32 163 | 2017-09-17,28 164 | 2017-09-24,28 165 | 2017-10-01,28 166 | 2017-10-08,32 167 | 2017-10-15,39 168 | 2017-10-22,18 169 | 2017-10-29,28 170 | 2017-11-05,24 171 | 2017-11-12,27 172 | 2017-11-19,28 173 | 2017-11-26,32 174 | 2017-12-03,37 175 | 2017-12-10,33 176 | 2017-12-17,20 177 | 2017-12-24,24 178 | 2017-12-31,37 179 | 2018-01-07,28 180 | 2018-01-14,17 181 | 2018-01-21,26 182 | 2018-01-28,13 183 | 2018-02-04,23 184 | 2018-02-11,20 185 | 2018-02-18,24 186 | 2018-02-25,23 187 | 2018-03-04,12 188 | 2018-03-11,19 189 | 2018-03-18,18 190 | 2018-03-25,9 191 | 2018-04-01,11 192 | 2018-04-08,18 193 | 2018-04-15,9 194 | 2018-04-22,12 195 | 2018-04-29,9 196 | 2018-05-06,11 197 | 2018-05-13,12 198 | 2018-05-20,15 199 | 2018-05-27,13 200 | 2018-06-03,16 201 | 2018-06-10,13 202 | 2018-06-17,15 203 | 2018-06-24,15 204 | 2018-07-01,27 205 | 2018-07-08,32 206 | 2018-07-15,34 207 | 2018-07-22,34 208 | 2018-07-29,38 209 | 2018-08-05,46 210 | 2018-08-12,37 211 | 2018-08-19,49 212 | 2018-08-26,27 213 | 2018-09-02,26 214 | 2018-09-09,34 215 | 2018-09-16,32 216 | 2018-09-23,37 217 | 2018-09-30,40 218 | 2018-10-07,26 219 | 2018-10-14,32 220 | 2018-10-21,34 221 | 2018-10-28,33 222 | 2018-11-04,46 223 | 2018-11-11,29 224 | 2018-11-18,28 225 | 2018-11-25,36 226 | 2018-12-02,33 227 | 2018-12-09,46 228 | 2018-12-16,34 229 | 2018-12-23,32 230 | 2018-12-30,47 231 | 2019-01-06,47 232 | 2019-01-13,48 233 | 2019-01-20,42 234 | 2019-01-27,40 235 | 2019-02-03,26 236 | 2019-02-10,28 237 | 2019-02-17,30 238 | 2019-02-24,33 239 | 2019-03-03,25 240 | 2019-03-10,25 241 | 2019-03-17,22 242 | 2019-03-24,22 243 | 2019-03-31,18 244 | 2019-04-07,15 245 | 2019-04-14,15 246 | 2019-04-21,21 247 | 2019-04-28,12 248 | 2019-05-05,13 249 | 2019-05-12,15 250 | 2019-05-19,18 251 | 2019-05-26,13 252 | 2019-06-02,14 253 | 2019-06-09,17 254 | 2019-06-16,25 255 | 2019-06-23,27 256 | 2019-06-30,36 257 | 2019-07-07,46 258 | 2019-07-14,73 259 | 2019-07-21,68 260 | 2019-07-28,81 261 | 2019-08-04,94 262 | 2019-08-11,94 263 | 2019-08-18,100 264 | -------------------------------------------------------------------------------- /model/google_trends/dengue_medicine.csv: -------------------------------------------------------------------------------- 1 | Category: All categories 2 | 3 | Week,dengue medicine: (Philippines) 4 | 2014-08-31,29 5 | 2014-09-07,19 6 | 2014-09-14,28 7 | 2014-09-21,20 8 | 2014-09-28,19 9 | 2014-10-05,0 10 | 2014-10-12,19 11 | 2014-10-19,19 12 | 2014-10-26,20 13 | 2014-11-02,39 14 | 2014-11-09,76 15 | 2014-11-16,38 16 | 2014-11-23,38 17 | 2014-11-30,19 18 | 2014-12-07,19 19 | 2014-12-14,20 20 | 2014-12-21,22 21 | 2014-12-28,0 22 | 2015-01-04,36 23 | 2015-01-11,18 24 | 2015-01-18,0 25 | 2015-01-25,18 26 | 2015-02-01,18 27 | 2015-02-08,0 28 | 2015-02-15,18 29 | 2015-02-22,17 30 | 2015-03-01,0 31 | 2015-03-08,0 32 | 2015-03-15,18 33 | 2015-03-22,0 34 | 2015-03-29,0 35 | 2015-04-05,18 36 | 2015-04-12,0 37 | 2015-04-19,18 38 | 2015-04-26,0 39 | 2015-05-03,18 40 | 2015-05-10,0 41 | 2015-05-17,18 42 | 2015-05-24,18 43 | 2015-05-31,17 44 | 2015-06-07,17 45 | 2015-06-14,16 46 | 2015-06-21,32 47 | 2015-06-28,40 48 | 2015-07-05,15 49 | 2015-07-12,40 50 | 2015-07-19,24 51 | 2015-07-26,32 52 | 2015-08-02,40 53 | 2015-08-09,32 54 | 2015-08-16,32 55 | 2015-08-23,33 56 | 2015-08-30,16 57 | 2015-09-06,71 58 | 2015-09-13,16 59 | 2015-09-20,48 60 | 2015-09-27,55 61 | 2015-10-04,40 62 | 2015-10-11,63 63 | 2015-10-18,49 64 | 2015-10-25,16 65 | 2015-11-01,49 66 | 2015-11-08,32 67 | 2015-11-15,24 68 | 2015-11-22,47 69 | 2015-11-29,16 70 | 2015-12-06,16 71 | 2015-12-13,51 72 | 2015-12-20,38 73 | 2015-12-27,28 74 | 2016-01-03,17 75 | 2016-01-10,40 76 | 2016-01-17,32 77 | 2016-01-24,31 78 | 2016-01-31,16 79 | 2016-02-07,47 80 | 2016-02-14,48 81 | 2016-02-21,31 82 | 2016-02-28,15 83 | 2016-03-06,46 84 | 2016-03-13,23 85 | 2016-03-20,0 86 | 2016-03-27,16 87 | 2016-04-03,16 88 | 2016-04-10,16 89 | 2016-04-17,0 90 | 2016-04-24,16 91 | 2016-05-01,16 92 | 2016-05-08,32 93 | 2016-05-15,0 94 | 2016-05-22,0 95 | 2016-05-29,0 96 | 2016-06-05,32 97 | 2016-06-12,16 98 | 2016-06-19,30 99 | 2016-06-26,15 100 | 2016-07-03,61 101 | 2016-07-10,16 102 | 2016-07-17,48 103 | 2016-07-24,32 104 | 2016-07-31,32 105 | 2016-08-07,40 106 | 2016-08-14,25 107 | 2016-08-21,100 108 | 2016-08-28,50 109 | 2016-09-04,16 110 | 2016-09-11,33 111 | 2016-09-18,49 112 | 2016-09-25,0 113 | 2016-10-02,24 114 | 2016-10-09,24 115 | 2016-10-16,63 116 | 2016-10-23,17 117 | 2016-10-30,35 118 | 2016-11-06,33 119 | 2016-11-13,40 120 | 2016-11-20,16 121 | 2016-11-27,24 122 | 2016-12-04,15 123 | 2016-12-11,16 124 | 2016-12-18,0 125 | 2016-12-25,0 126 | 2017-01-01,0 127 | 2017-01-08,15 128 | 2017-01-15,15 129 | 2017-01-22,15 130 | 2017-01-29,15 131 | 2017-02-05,0 132 | 2017-02-12,0 133 | 2017-02-19,0 134 | 2017-02-26,14 135 | 2017-03-05,21 136 | 2017-03-12,14 137 | 2017-03-19,15 138 | 2017-03-26,0 139 | 2017-04-02,0 140 | 2017-04-09,0 141 | 2017-04-16,16 142 | 2017-04-23,16 143 | 2017-04-30,0 144 | 2017-05-07,23 145 | 2017-05-14,0 146 | 2017-05-21,17 147 | 2017-05-28,0 148 | 2017-06-04,16 149 | 2017-06-11,17 150 | 2017-06-18,31 151 | 2017-06-25,16 152 | 2017-07-02,31 153 | 2017-07-09,0 154 | 2017-07-16,23 155 | 2017-07-23,23 156 | 2017-07-30,47 157 | 2017-08-06,31 158 | 2017-08-13,39 159 | 2017-08-20,31 160 | 2017-08-27,48 161 | 2017-09-03,31 162 | 2017-09-10,16 163 | 2017-09-17,0 164 | 2017-09-24,15 165 | 2017-10-01,15 166 | 2017-10-08,15 167 | 2017-10-15,15 168 | 2017-10-22,15 169 | 2017-10-29,16 170 | 2017-11-05,15 171 | 2017-11-12,15 172 | 2017-11-19,15 173 | 2017-11-26,0 174 | 2017-12-03,37 175 | 2017-12-10,15 176 | 2017-12-17,0 177 | 2017-12-24,26 178 | 2017-12-31,24 179 | 2018-01-07,15 180 | 2018-01-14,0 181 | 2018-01-21,0 182 | 2018-01-28,14 183 | 2018-02-04,21 184 | 2018-02-11,15 185 | 2018-02-18,29 186 | 2018-02-25,21 187 | 2018-03-04,14 188 | 2018-03-11,0 189 | 2018-03-18,0 190 | 2018-03-25,0 191 | 2018-04-01,30 192 | 2018-04-08,0 193 | 2018-04-15,29 194 | 2018-04-22,0 195 | 2018-04-29,15 196 | 2018-05-06,0 197 | 2018-05-13,0 198 | 2018-05-20,0 199 | 2018-05-27,0 200 | 2018-06-03,14 201 | 2018-06-10,0 202 | 2018-06-17,20 203 | 2018-06-24,13 204 | 2018-07-01,13 205 | 2018-07-08,25 206 | 2018-07-15,13 207 | 2018-07-22,13 208 | 2018-07-29,32 209 | 2018-08-05,26 210 | 2018-08-12,20 211 | 2018-08-19,27 212 | 2018-08-26,54 213 | 2018-09-02,45 214 | 2018-09-09,32 215 | 2018-09-16,26 216 | 2018-09-23,13 217 | 2018-09-30,13 218 | 2018-10-07,0 219 | 2018-10-14,13 220 | 2018-10-21,13 221 | 2018-10-28,15 222 | 2018-11-04,40 223 | 2018-11-11,13 224 | 2018-11-18,20 225 | 2018-11-25,13 226 | 2018-12-02,26 227 | 2018-12-09,20 228 | 2018-12-16,14 229 | 2018-12-23,23 230 | 2018-12-30,22 231 | 2019-01-06,19 232 | 2019-01-13,50 233 | 2019-01-20,25 234 | 2019-01-27,25 235 | 2019-02-03,31 236 | 2019-02-10,37 237 | 2019-02-17,12 238 | 2019-02-24,12 239 | 2019-03-03,24 240 | 2019-03-10,24 241 | 2019-03-17,12 242 | 2019-03-24,13 243 | 2019-03-31,13 244 | 2019-04-07,0 245 | 2019-04-14,14 246 | 2019-04-21,26 247 | 2019-04-28,13 248 | 2019-05-05,26 249 | 2019-05-12,0 250 | 2019-05-19,13 251 | 2019-05-26,13 252 | 2019-06-02,24 253 | 2019-06-09,12 254 | 2019-06-16,12 255 | 2019-06-23,35 256 | 2019-06-30,35 257 | 2019-07-07,63 258 | 2019-07-14,46 259 | 2019-07-21,47 260 | 2019-07-28,70 261 | 2019-08-04,58 262 | 2019-08-11,76 263 | 2019-08-18,80 264 | -------------------------------------------------------------------------------- /model/google_trends/dengue_symptoms.csv: -------------------------------------------------------------------------------- 1 | Category: All categories 2 | 3 | Week,dengue symptoms: (Philippines) 4 | 2014-08-31,22 5 | 2014-09-07,21 6 | 2014-09-14,19 7 | 2014-09-21,18 8 | 2014-09-28,13 9 | 2014-10-05,18 10 | 2014-10-12,23 11 | 2014-10-19,24 12 | 2014-10-26,28 13 | 2014-11-02,23 14 | 2014-11-09,24 15 | 2014-11-16,23 16 | 2014-11-23,18 17 | 2014-11-30,19 18 | 2014-12-07,19 19 | 2014-12-14,16 20 | 2014-12-21,22 21 | 2014-12-28,21 22 | 2015-01-04,26 23 | 2015-01-11,16 24 | 2015-01-18,14 25 | 2015-01-25,19 26 | 2015-02-01,20 27 | 2015-02-08,16 28 | 2015-02-15,7 29 | 2015-02-22,18 30 | 2015-03-01,13 31 | 2015-03-08,11 32 | 2015-03-15,14 33 | 2015-03-22,11 34 | 2015-03-29,8 35 | 2015-04-05,11 36 | 2015-04-12,10 37 | 2015-04-19,9 38 | 2015-04-26,5 39 | 2015-05-03,9 40 | 2015-05-10,6 41 | 2015-05-17,8 42 | 2015-05-24,7 43 | 2015-05-31,13 44 | 2015-06-07,9 45 | 2015-06-14,12 46 | 2015-06-21,14 47 | 2015-06-28,18 48 | 2015-07-05,23 49 | 2015-07-12,22 50 | 2015-07-19,27 51 | 2015-07-26,30 52 | 2015-08-02,30 53 | 2015-08-09,28 54 | 2015-08-16,27 55 | 2015-08-23,33 56 | 2015-08-30,33 57 | 2015-09-06,36 58 | 2015-09-13,45 59 | 2015-09-20,42 60 | 2015-09-27,61 61 | 2015-10-04,65 62 | 2015-10-11,49 63 | 2015-10-18,41 64 | 2015-10-25,42 65 | 2015-11-01,29 66 | 2015-11-08,31 67 | 2015-11-15,29 68 | 2015-11-22,31 69 | 2015-11-29,25 70 | 2015-12-06,21 71 | 2015-12-13,21 72 | 2015-12-20,19 73 | 2015-12-27,21 74 | 2016-01-03,28 75 | 2016-01-10,22 76 | 2016-01-17,22 77 | 2016-01-24,18 78 | 2016-01-31,18 79 | 2016-02-07,18 80 | 2016-02-14,21 81 | 2016-02-21,17 82 | 2016-02-28,12 83 | 2016-03-06,10 84 | 2016-03-13,14 85 | 2016-03-20,11 86 | 2016-03-27,12 87 | 2016-04-03,9 88 | 2016-04-10,7 89 | 2016-04-17,6 90 | 2016-04-24,11 91 | 2016-05-01,7 92 | 2016-05-08,8 93 | 2016-05-15,8 94 | 2016-05-22,13 95 | 2016-05-29,11 96 | 2016-06-05,12 97 | 2016-06-12,15 98 | 2016-06-19,16 99 | 2016-06-26,17 100 | 2016-07-03,20 101 | 2016-07-10,23 102 | 2016-07-17,32 103 | 2016-07-24,38 104 | 2016-07-31,31 105 | 2016-08-07,33 106 | 2016-08-14,43 107 | 2016-08-21,41 108 | 2016-08-28,45 109 | 2016-09-04,39 110 | 2016-09-11,40 111 | 2016-09-18,31 112 | 2016-09-25,33 113 | 2016-10-02,27 114 | 2016-10-09,21 115 | 2016-10-16,26 116 | 2016-10-23,24 117 | 2016-10-30,21 118 | 2016-11-06,16 119 | 2016-11-13,14 120 | 2016-11-20,13 121 | 2016-11-27,22 122 | 2016-12-04,17 123 | 2016-12-11,17 124 | 2016-12-18,15 125 | 2016-12-25,16 126 | 2017-01-01,16 127 | 2017-01-08,16 128 | 2017-01-15,13 129 | 2017-01-22,10 130 | 2017-01-29,13 131 | 2017-02-05,11 132 | 2017-02-12,14 133 | 2017-02-19,10 134 | 2017-02-26,11 135 | 2017-03-05,11 136 | 2017-03-12,6 137 | 2017-03-19,11 138 | 2017-03-26,9 139 | 2017-04-02,7 140 | 2017-04-09,6 141 | 2017-04-16,8 142 | 2017-04-23,5 143 | 2017-04-30,8 144 | 2017-05-07,6 145 | 2017-05-14,7 146 | 2017-05-21,5 147 | 2017-05-28,10 148 | 2017-06-04,8 149 | 2017-06-11,9 150 | 2017-06-18,14 151 | 2017-06-25,16 152 | 2017-07-02,12 153 | 2017-07-09,19 154 | 2017-07-16,24 155 | 2017-07-23,23 156 | 2017-07-30,22 157 | 2017-08-06,25 158 | 2017-08-13,19 159 | 2017-08-20,21 160 | 2017-08-27,19 161 | 2017-09-03,21 162 | 2017-09-10,16 163 | 2017-09-17,18 164 | 2017-09-24,21 165 | 2017-10-01,16 166 | 2017-10-08,20 167 | 2017-10-15,20 168 | 2017-10-22,16 169 | 2017-10-29,12 170 | 2017-11-05,16 171 | 2017-11-12,21 172 | 2017-11-19,15 173 | 2017-11-26,18 174 | 2017-12-03,25 175 | 2017-12-10,16 176 | 2017-12-17,22 177 | 2017-12-24,19 178 | 2017-12-31,22 179 | 2018-01-07,15 180 | 2018-01-14,18 181 | 2018-01-21,19 182 | 2018-01-28,14 183 | 2018-02-04,18 184 | 2018-02-11,16 185 | 2018-02-18,14 186 | 2018-02-25,10 187 | 2018-03-04,9 188 | 2018-03-11,12 189 | 2018-03-18,8 190 | 2018-03-25,10 191 | 2018-04-01,6 192 | 2018-04-08,9 193 | 2018-04-15,9 194 | 2018-04-22,5 195 | 2018-04-29,8 196 | 2018-05-06,5 197 | 2018-05-13,6 198 | 2018-05-20,8 199 | 2018-05-27,10 200 | 2018-06-03,10 201 | 2018-06-10,8 202 | 2018-06-17,12 203 | 2018-06-24,16 204 | 2018-07-01,17 205 | 2018-07-08,24 206 | 2018-07-15,24 207 | 2018-07-22,29 208 | 2018-07-29,33 209 | 2018-08-05,31 210 | 2018-08-12,27 211 | 2018-08-19,28 212 | 2018-08-26,27 213 | 2018-09-02,25 214 | 2018-09-09,24 215 | 2018-09-16,21 216 | 2018-09-23,25 217 | 2018-09-30,23 218 | 2018-10-07,25 219 | 2018-10-14,19 220 | 2018-10-21,20 221 | 2018-10-28,17 222 | 2018-11-04,20 223 | 2018-11-11,23 224 | 2018-11-18,21 225 | 2018-11-25,20 226 | 2018-12-02,25 227 | 2018-12-09,20 228 | 2018-12-16,21 229 | 2018-12-23,24 230 | 2018-12-30,37 231 | 2019-01-06,39 232 | 2019-01-13,38 233 | 2019-01-20,33 234 | 2019-01-27,33 235 | 2019-02-03,28 236 | 2019-02-10,22 237 | 2019-02-17,21 238 | 2019-02-24,17 239 | 2019-03-03,14 240 | 2019-03-10,14 241 | 2019-03-17,11 242 | 2019-03-24,14 243 | 2019-03-31,12 244 | 2019-04-07,11 245 | 2019-04-14,8 246 | 2019-04-21,9 247 | 2019-04-28,8 248 | 2019-05-05,7 249 | 2019-05-12,9 250 | 2019-05-19,10 251 | 2019-05-26,10 252 | 2019-06-02,13 253 | 2019-06-09,15 254 | 2019-06-16,17 255 | 2019-06-23,24 256 | 2019-06-30,27 257 | 2019-07-07,33 258 | 2019-07-14,50 259 | 2019-07-21,73 260 | 2019-07-28,81 261 | 2019-08-04,88 262 | 2019-08-11,100 263 | 2019-08-18,84 264 | -------------------------------------------------------------------------------- /model/readme: -------------------------------------------------------------------------------- 1 | This is the model folder. 2 | -------------------------------------------------------------------------------- /model/weather/Ambulong.csv: -------------------------------------------------------------------------------- 1 | YEAR,MONTH,TMEAN,RAINFALL 2 | 2014,1,24.6,0.5 3 | 2014,2,25.6,0 4 | 2014,3,27.2,28.1 5 | 2014,4,28.4,28.4 6 | 2014,5,29.8,154.6 7 | 2014,6,29,78.9 8 | 2014,7,27.7,406.7 9 | 2014,8,27.9,212.5 10 | 2014,9,27.5,321.5 11 | 2014,10,27.4,210.5 12 | 2014,11,27.3,43 13 | 2014,12,26.5,326.8 14 | 2015,1,25.3,50.8 15 | 2015,2,25.7,5 16 | 2015,3,26.6,15.2 17 | 2015,4,28.7,0 18 | 2015,5,29.4,11.4 19 | 2015,6,29.3,159.7 20 | 2015,7,28.1,309.1 21 | 2015,8,28.4,265.3 22 | 2015,9,28.4,243.8 23 | 2015,10,27.6,230.2 24 | 2015,11,27.8,43.4 25 | 2015,12,26.7,292.5 26 | 2016,1,26.7,18.7 27 | 2016,2,26.6,45.3 28 | 2016,3,27.8,10.2 29 | 2016,4,30,8.2 30 | 2016,5,30.4,101.4 31 | 2016,6,29.5,95.6 32 | 2016,7,28.9,225.5 33 | 2016,8,28.4,375.2 34 | 2016,9,28.4,167 35 | 2016,10,27.9,312.3 36 | 2016,11,27.5,162.1 37 | 2016,12,27.4,187.2 38 | 2017,1,26.3,87.1 39 | 2017,2,25.9,20.2 40 | 2017,3,27.3,19.4 41 | 2017,4,29.2,17.8 42 | 2017,5,30,126.6 43 | 2017,6,29.5,115.8 44 | 2017,7,28.4,254 45 | 2017,8,28.6,375 46 | 2017,9,28.5,584.8 47 | 2017,10,28,241.7 48 | 2017,11,28.2,189.7 49 | 2017,12,27,183.7 50 | 2018,1,26.5,40.1 51 | 2018,2,27.2,2.4 52 | 2018,3,27.6,4 53 | 2018,4,29.2,6.02 54 | 2018,5,29.8,131.5 55 | 2018,6,28.2,488.4 56 | 2018,7,27.7,596.2 57 | 2018,8,28.1,160.8 58 | 2018,9,27.8,255.6 59 | 2018,10,28.1,127.7 60 | 2018,11,27.8,20.3 61 | 2018,12,26.9,286.2 62 | -------------------------------------------------------------------------------- /model/weather/NAIA.csv: -------------------------------------------------------------------------------- 1 | YEAR,MONTH,TMEAN,RAINFALL 2 | 2014,1,26,0 3 | 2014,2,27.1,0 4 | 2014,3,28.4,7 5 | 2014,4,30.7,0 6 | 2014,5,31.8,64.5 7 | 2014,6,29.5,193.5 8 | 2014,7,28.2,380.5 9 | 2014,8,28.3,244 10 | 2014,9,28.1,424.5 11 | 2014,10,28.4,227 12 | 2014,11,28.4,34 13 | 2014,12,27.2,120.5 14 | 2015,1,26,24.7 15 | 2015,2,26.7,0.5 16 | 2015,3,27.9,7 17 | 2015,4,30.7,0 18 | 2015,5,31.1,32.5 19 | 2015,6,30.7,91 20 | 2015,7,28.7,297.5 21 | 2015,8,28.7,241 22 | 2015,9,25.9,268 23 | 2015,10,28.7,221 24 | 2015,11,29.3,14 25 | 2015,12,27.8,294.5 26 | 2016,1,28.1,0.5 27 | 2016,2,27.8,20.5 28 | 2016,3,29.5,0 29 | 2016,4,31.2,12 30 | 2016,5,31.5,47 31 | 2016,6,30,124.5 32 | 2016,7,29.4,268 33 | 2016,8,28.6,350.5 34 | 2016,9,28.9,137 35 | 2016,10,28.9,166.5 36 | 2016,11,28.3,98 37 | 2016,12,28.3,70 38 | 2017,1,27.1,38.95 39 | 2017,2,27.1,31.5 40 | 2017,3,28.6,3.5 41 | 2017,4,30.2,38 42 | 2017,5,31,70 43 | 2017,6,30.1,168.95 44 | 2017,7,29,272.5 45 | 2017,8,29.2,298.5 46 | 2017,9,29.2,324 47 | 2017,10,28.6,194.5 48 | 2017,11,28.7,114.5 49 | 2017,12,27.6,61.5 50 | 2018,1,27.6,25 51 | 2018,2,28,0 52 | 2018,3,28.6,28.5 53 | 2018,4,30.4,0 54 | 2018,5,31.8,4 55 | 2018,6,28.8,489.5 56 | 2018,7,27.8,470 57 | 2018,8,28.2,-999 58 | 2018,9,28.4,235 59 | 2018,10,29.2,80.5 60 | 2018,11,28.9,11.5 61 | 2018,12,27.7,138 62 | -------------------------------------------------------------------------------- /model/weather/Sangley Point.csv: -------------------------------------------------------------------------------- 1 | YEAR,MONTH,TMEAN,RAINFALL 2 | 2014,1,26.4,0.4 3 | 2014,2,27.7,3.3 4 | 2014,3,28.9,7.2 5 | 2014,4,30.7,1.2 6 | 2014,5,32.2,84.5 7 | 2014,6,29.9,321.4 8 | 2014,7,28.7,523.4 9 | 2014,8,29.3,296.8 10 | 2014,9,29,460.3 11 | 2014,10,29.1,212.9 12 | 2014,11,28.9,27.8 13 | 2014,12,27.7,163.2 14 | 2015,1,26.6,31.8 15 | 2015,2,27.4,1 16 | 2015,3,28.6,8.8 17 | 2015,4,30.6,0 18 | 2015,5,33.4,10.2 19 | 2015,6,31.2,166.8 20 | 2015,7,29.5,352.8 21 | 2015,8,29.7,312.6 22 | 2015,9,29.7,359.2 23 | 2015,10,29.4,103.6 24 | 2015,11,29.6,9.2 25 | 2015,12,28.1,294.7 26 | 2016,1,28.3,1.4 27 | 2016,2,27.8,80.6 28 | 2016,3,29.6,0.2 29 | 2016,4,31.4,20.2 30 | 2016,5,31.6,161.8 31 | 2016,6,30.3,182.4 32 | 2016,7,30,302.8 33 | 2016,8,28.9,775 34 | 2016,9,29.5,145.3 35 | 2016,10,29.3,189.36 36 | 2016,11,28.4,88.9 37 | 2016,12,28.4,89.8 38 | 2017,1,27.4,52.5 39 | 2017,2,27.3,5.4 40 | 2017,3,29,6.4 41 | 2017,4,30.5,39.3 42 | 2017,5,31.3,186.4 43 | 2017,6,30.7,105.1 44 | 2017,7,29.3,467.2 45 | 2017,8,29.8,323 46 | 2017,9,29.7,382.7 47 | 2017,10,29.1,196.1 48 | 2017,11,29.1,116.4 49 | 2017,12,28.2,57.1 50 | 2018,1,27.9,16.4 51 | 2018,2,28.5,0.8 52 | 2018,3,28.9,105.8 53 | 2018,4,30.8,0.2 54 | 2018,5,32,20 55 | 2018,6,29,723 56 | 2018,7,28.2,757.3 57 | 2018,8,28.9,427.2 58 | 2018,9,29.2,194.7 59 | 2018,10,30.1,72.8 60 | 2018,11,29.5,13.7 61 | 2018,12,28.1,132.9 62 | -------------------------------------------------------------------------------- /model/weather/Tayabas.csv: -------------------------------------------------------------------------------- 1 | YEAR,MONTH,TMEAN,RAINFALL 2 | 2014,1,22.6,30 3 | 2014,2,23.7,21.2 4 | 2014,3,26,91.1 5 | 2014,4,27.6,20.5 6 | 2014,5,29.1,83.6 7 | 2014,6,28.1,167.1 8 | 2014,7,27,337.4 9 | 2014,8,27.3,247.3 10 | 2014,9,26.2,293.1 11 | 2014,10,26.4,609.4 12 | 2014,11,26,332.2 13 | 2014,12,24.4,700.9 14 | 2015,1,23.6,124.2 15 | 2015,2,24.2,66.7 16 | 2015,3,25.5,59.8 17 | 2015,4,27,17.7 18 | 2015,5,28.4,19.2 19 | 2015,6,28.5,156.3 20 | 2015,7,27.6,123.3 21 | 2015,8,27.7,162.2 22 | 2015,9,27.8,306.6 23 | 2015,10,27.1,334.6 24 | 2015,11,26.6,443.9 25 | 2015,12,25.6,544.6 26 | 2016,1,25.2,113.8 27 | 2016,2,24.8,255.7 28 | 2016,3,26.4,60.9 29 | 2016,4,27.7,43 30 | 2016,5,29.1,67.9 31 | 2016,6,28.6,76.3 32 | 2016,7,28,219.5 33 | 2016,8,27.9,165.8 34 | 2016,9,28,75.4 35 | 2016,10,27.2,642.1 36 | 2016,11,26.5,697.4 37 | 2016,12,26,735.4 38 | 2017,1,25,416.2 39 | 2017,2,24.9,156.5 40 | 2017,3,26.2,81.2 41 | 2017,4,27.6,92.7 42 | 2017,5,29,176.9 43 | 2017,6,28.3,241.5 44 | 2017,7,26.7,310.3 45 | 2017,8,26.7,188.5 46 | 2017,9,25.6,480.2 47 | 2017,10,26.2,390.2 48 | 2017,11,26.7,787.2 49 | 2017,12,25.4,822.5 50 | 2018,1,25.2,279.3 51 | 2018,2,25.7,183.4 52 | 2018,3,26.3,129.3 53 | 2018,4,27.8,96.3 54 | 2018,5,28.8,153.9 55 | 2018,6,27.8,179.2 56 | 2018,7,27.2,264.6 57 | 2018,8,27.7,139.8 58 | 2018,9,27,269.4 59 | 2018,10,27.5,116.3 60 | 2018,11,26.8,263.5 61 | 2018,12,26,685.4 62 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Welcome to the AEDES Project 2 | 3 | ## Our Challenge 4 | 5 | AEDES is tackling the [SMASH YOUR SDGs](https://2019.spaceappschallenge.org/challenges/living-our-world/smash-your-sdgs/details) challenge under the [LIVING IN OUR WORLD](https://2019.spaceappschallenge.org/challenges/living-our-world/) category. Specifically we are tackling Goal 3: GOOD HEALTH AND WELL BEING under the UN Sustainable Development Goals. 6 | 7 | ## The Problem 8 | 9 | Dengue is now at epidemic levels in the Philippines with over 271,000 cases and more than 1,100 deaths as of August 2019. Data on dengue takes time to be manually gathered which hampers the health sector’s ability to deal with the threat. Dengue is spread between infected cases through the Aedes Aegypti mosquito and mosquitoes are known to breed in damp locations and stagnant water pools. 10 | 11 | ## Our Solution 12 | 13 | We propose an automated information portal that correlates dengue cases and deaths with real-time data from climate, google searches, and satellite maps, giving an advance indicator of when dengue will emerge and potential dengue hotspot locations. This portal is accessible publicly but is targeted towards public health and local government agencies to give them advanced notice of dengue outbreaks and help prioritize resources. 14 | 15 | The service relies on 3 data sets: 16 | * Satellite Data: Satellite imaging data from [Sentinel Online Copernicus](https://sentinel.esa.int/web/sentinel/sentinel-data-access) 17 | * Local Weather Data: Climate data from [DOST-PAGASA](http://bagong.pagasa.dost.gov.ph/climate/climatological-normals) 18 | * Google Data: [Search trends for 'dengue' and related terms](https://trends.google.com/trends/explore?date=today%205-y&geo=PH&q=dengue) 19 | 20 | To populate the information portal, we propose 3 models: 21 | * Predict dengue cases from climate and search data 22 | * Predict dengue deaths from dengue cases 23 | * Determine likely dengue hotspots by detecting stagnant water areas from satellite data 24 | 25 | By doing this we are addressing 2 key challenges for public health and local government officials: 26 | * Get ahead of the lagged delay of dengue reporting by using real-time information (e.g. climate, searches) to infer if dengue cases and deaths are about to spike. 27 | * Precisely anticipate areas that may be affected by dengue to prioritize health aid, supplies, and proactive fumigation to prevent the outbreaks. 28 | 29 | ## Related Literature 30 | 31 | We have conducted research and cite the following prior work that provide the foundation for this solution: 32 | * In Caro, et. al. (2016), *[Forecasting and Data Visualization of Dengue spread in the Philippine Visayas Island group](https://github.com/aedesproject/nasa_hack/blob/master/literature/07785420%20-%20Forecasting%20and%20Data%20Visualization%20of%20Dengue%20Spread%20in%20PH%20Visayas.pdf)*, researchers proposed a method to predict dengue cases in the Visayas regions through the use of an Artificial Neural Network (ANN) which predicted dengue cases (R 0.8) through the use of climate data (temperature and rainfall) and a lagged number of previous dengue cases. 33 | * In Yang, et. al. (2016), *[Advances in using Internet searches to track dengue](https://github.com/aedesproject/nasa_hack/blob/master/literature/Advances_in_using_Internet_searches_to_track_dengu.pdf)*, researchers used Google searches for 'dengue' and related keywords to predict dengue cases in Mexico, Brazil, Thailand, Singapore and Taiwan. 34 | * In Chua, Tan, et. al. (2018), *[Project Still Water](https://github.com/aedesproject/nasa_hack/blob/master/literature/Stagnant%20Water%20Maps%20from%20Satellite%20Infrared%20Data.pdf)*, researchers proposed a methodology to detect dengue hotspots using FAPAR (Vegetation) and NDWI (Water) areas. 35 | * In Chan, Johnson (2012), *[The Incubation Periods of Dengue Viruses](https://github.com/aedesproject/nasa_hack/blob/master/literature/Incubation-Dengue-pone.0050972.pdf)*, researchers establish the incubation periods for dengue. 95% of EIPs are between 5 and 33 days at 25C, and 2 and 15 days at30C, with means of 15 and 6.5 days, respectively which suggests that climate has an effect on the incubation of the disease. 36 | 37 | ## Objectives and Principles 38 | 39 | Our proposed solution intends to builds on this work by the following principles: 40 | * We test and propose independent and mixed models that combine Google searches and climate data to detect possible factors that affect dengue cases and deaths as well as provide an accurate forecast. 41 | * We provide the information on an easily navigable interactive web application accessible by the public to help disseminate crucial information and also assist in public health efforts to prevent and contain the disease. 42 | 43 | ![dengue-cycle](https://github.com/aedesproject/nasa_hack/blob/master/deck/dengue_data.png) 44 | 45 | Logically, the behavior captured by the datasets in our study is: 46 | 1. First, precipitation and temperature climate creates mosquito-breeding environments 47 | 2. Mosquitoes spread and get infected by existing dengue cases, thereby spreading the disease 48 | 3. New infections cause alarm which drives internet searches for dengue 49 | 4. Dengue cases result in deaths 50 | 5. Cases and deaths are reported to public health officials 51 | 52 | Therefore by detecting #1 and #3 we can address and hopefully disrupt the disease cycle before an epidemic spreads. 53 | 54 | ## Selecting Pilot Areas For Prototype 55 | 56 | 57 | 58 | For our initial prototype we decided to focus on four areas: 59 | 60 | * National Capital Region (Quezon City as a focus) 61 | * Eastern Visayas (Tacloban City as a focus) 62 | * Western Visayas (Iloilo City as a focus) 63 | * ARMM (Cotabato City as a focus) 64 | 65 | We selected the above areas due to availability of local weather station in these locations. We decided on NCR as a default and added the three other cities due to the observed spike in Google Searches for dengue related terms indicating possible prevailing public panic in these areas. 66 | 67 | Google Searches Spiking in July: 68 | ![searches](https://github.com/aedesproject/nasa_hack/blob/master/deck/dengue_searches.png) 69 | 70 | ## Predicting Dengue Cases and Deaths 71 | 72 | We propose to forecast dengue cases and deaths using a combination of: 73 | 74 | * Climate Data: Average monthly temperature readings from local weather stations 75 | * Climate Data: Average monthly rainfall (precipitation) readings from local weather stations 76 | * Google Data: Search index for 'dengue', 'dengue symptoms', 'dengue cure', and 'dengue medicine' 77 | * Lagged values for Climate factors, Google data, and actual cases and deaths 78 | 79 | ![model](https://github.com/aedesproject/nasa_hack/blob/master/deck/dengue_model.png) 80 | 81 | ## Surfacing Mosquito Hotspots From Satellite Data 82 | 83 | We propose calculating dengue hotspots from satellite images as the intersection of the following: 84 | 85 | * Fraction of Absorbed Photosynthetically Active Radiation (FAPAR) to identify areas with vegetation. 86 | * Normalized Difference Water Index (NDWI) to identify areas with water. 87 | 88 | Combining the two indices and testing for various thresholds we can approximate areas that have stagnant water which are breeding grounds for mosquitoes. 89 | 90 | ![satellite](https://github.com/aedesproject/nasa_hack/blob/master/deck/satellite.png) 91 | 92 | ## Results 93 | 94 | We visualized the resulting models on a web interface for easy navigation. Check the prototype [here](http://aedesproject.org). 95 | 96 | The preliminary run of our forecast model shows we are able to approximate the trendmonth-to-date dengue cases (R 0.82). We also visualized the dengue hotspots on an interactive map that can zoom down to street level for public health sector targeting. 97 | 98 | ## Contact Us 99 | 100 | AEDES was [2019 Global Winner](https://2019.spaceappschallenge.org/awards/) of the NASA Space Apps Challenge winning **Best Use Of Data**. 101 | 102 | ![alt text](https://github.com/docligot/aedesproject/blob/master/deck/TEAM_AEDES.jpg "Grand Prize") 103 | 104 | The AEDES team consists of the following: 105 | 106 | * [Dominic Ligot](https://www.linkedin.com/in/docligot/), Team Lead 107 | * [Claire Tayco](https://www.linkedin.com/in/claire-san-juan-tayco-81361828/), Statistical Models 108 | * [Jansen Lopez](https://www.linkedin.com/in/jansen-lopez/), Geospatial Models 109 | * [Mark Toledo](https://www.linkedin.com/in/toledomark/), Data Engineer 110 | 111 | We welcome collaborators. Contact us via Linked-IN, or email us at support@aedesproject.org. 112 | 113 | ## License 114 | 115 | This work is provided under the terms of Creative Commons Attribution Share Alike 4.0 International ([CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)) license. 116 | -------------------------------------------------------------------------------- /satellite/Using QGIS to Extract Dengue Hotspots.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/Using QGIS to Extract Dengue Hotspots.pdf -------------------------------------------------------------------------------- /satellite/csv-js/cotabato_latlong1.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/csv-js/cotabato_latlong1.xlsx -------------------------------------------------------------------------------- /satellite/csv-js/dagupan_csv.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/csv-js/dagupan_csv.xlsx -------------------------------------------------------------------------------- /satellite/csv-js/iloilo_latlong1.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/csv-js/iloilo_latlong1.xlsx -------------------------------------------------------------------------------- /satellite/csv-js/qc_latlong1.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/csv-js/qc_latlong1.xlsx -------------------------------------------------------------------------------- /satellite/csv-js/tacloban_latlong1.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/csv-js/tacloban_latlong1.xlsx -------------------------------------------------------------------------------- /satellite/downloadSatImage.py: -------------------------------------------------------------------------------- 1 | # Filename: downloadSatImage.py 2 | # v1.0: 02.15.2020 (rdgarcia) 3 | # This will download Sentinel 2A satellite image 4 | # of the specified city 5 | # (Catbalogan is selected as an example) 6 | 7 | import folium 8 | from sentinelsat import SentinelAPI 9 | import geopandas as gpd 10 | 11 | # Set your scihub user and password here 12 | user = 'yourusername' 13 | password = 'yourpassword' 14 | 15 | api = SentinelAPI(user, password, 'https://scihub.copernicus.eu/dhus') 16 | 17 | # Set to shapefile of the city here (must be in the root folder) 18 | nCatbalogan = gpd.read_file('Catbalogan.shp') 19 | 20 | # Set to lat and long coordinates of the city here (use Google maps) 21 | m = folium.Map([11.827248, 124.846270], zoom_start=11) 22 | folium.GeoJson(nCatbalogan).add_to(m) 23 | 24 | footprint = None 25 | for i in nCatbalogan['geometry']: 26 | footprint = i 27 | 28 | # Specify the date and processing level here (either Level-2A or Level-1C) 29 | # cloudcoverpercentage of 10% is selected 30 | products = api.query(footprint, 31 | date = ('20190101', '20200131'), 32 | platformname = 'Sentinel-2', 33 | processinglevel = 'Level-2A', 34 | cloudcoverpercentage = (0,10)) 35 | # Check if sat image is available, change query settings if 0 36 | len(products) 37 | 38 | # Sorting of available images based on cloud cover percentage 39 | products_gdf = api.to_geodataframe(products) 40 | products_gdf_sorted = products_gdf.sort_values(['cloudcoverpercentage'], ascending=[True]) 41 | products_gdf_sorted 42 | 43 | # Choosing of selected image with lowest cloud cover 44 | product_uuid=products_gdf_sorted.iloc[0]['uuid'] 45 | 46 | # Initiating download of satellite image 47 | # This may take a few minutes (or hours) depending on the time of the day 48 | api.download(product_uuid) 49 | 50 | # end 51 | -------------------------------------------------------------------------------- /satellite/images/GenSan 2020-02-01.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/GenSan 2020-02-01.jpg -------------------------------------------------------------------------------- /satellite/images/Surigao 2019-11-08.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/Surigao 2019-11-08.jpg -------------------------------------------------------------------------------- /satellite/images/Tagbilaran 2020-02-09.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/Tagbilaran 2020-02-09.jpg -------------------------------------------------------------------------------- /satellite/images/Tagum 2020-01-17.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/Tagum 2020-01-17.jpg -------------------------------------------------------------------------------- /satellite/images/fapar-ndwi-formula.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/fapar-ndwi-formula.png -------------------------------------------------------------------------------- /satellite/images/fapar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/fapar.png -------------------------------------------------------------------------------- /satellite/images/ndwi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/ndwi.png -------------------------------------------------------------------------------- /satellite/images/qgis-extract-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/qgis-extract-2.png -------------------------------------------------------------------------------- /satellite/images/qgis-extract.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/images/qgis-extract.png -------------------------------------------------------------------------------- /satellite/map_description.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/docligot/aedesproject/4cf47fc929560290ae09b6e086bd7aa2a7c3691e/satellite/map_description.docx -------------------------------------------------------------------------------- /satellite/processSatImage.py: -------------------------------------------------------------------------------- 1 | # Filename: processSatImage.py 2 | # v1.0: 02.15.2020 (rdgarcia) 3 | # This will process Sentinel 2A satellite image 4 | # of the specified city 5 | # (Angeles is selected as an example) 6 | 7 | import numpy as np 8 | import geopandas as gpd 9 | import pandas as pd 10 | import rasterio as rio 11 | from rasterio.plot import show 12 | import rasterio.mask 13 | import matplotlib.pyplot as plt 14 | 15 | # Open Bands with Rasterio: set image names here 16 | b4 = rio.open('T50PRB_20200128T021941_B04_10m.jp2') 17 | b3 = rio.open('T50PRB_20200128T021941_B03_10m.jp2') 18 | b8 = rio.open('T50PRB_20200128T021941_B08_10m.jp2') 19 | 20 | # Set to shapefile of the city here (must be in the root folder) 21 | nAngeles = gpd.read_file('Angeles.shp') 22 | nAngeles_proj = nAngeles.to_crs({'init': 'epsg:32650'}) 23 | 24 | # Read Red(b4), Green(b3) and NIR(b8) as arrays 25 | red = b4.read() 26 | green = b3.read() 27 | nir = b8.read() 28 | 29 | # Calculate FAPAR: ((nir-red)/(nir+red)*(1.25‐0.025)) 30 | fapar = (nir.astype(float) - red.astype(float))/(nir+red) * 1.225 31 | 32 | # Create the FAPAR image 33 | meta = b4.meta 34 | meta.update(driver='GTiff') 35 | meta.update(dtype=rasterio.float32) 36 | with rasterio.open('FAPAR.tif', 'w', **meta) as dst: 37 | dst.write(fapar.astype(rasterio.float32)) 38 | 39 | # Convert to FAPAR_masked image 40 | with rio.open("FAPAR.tif") as src: 41 | out_image, out_transform = rio.mask.mask(src, nAngeles_proj.geometry,crop=True) 42 | out_meta = src.meta.copy() 43 | out_meta.update({"driver": "GTiff", 44 | "height": out_image.shape[1], 45 | "width": out_image.shape[2], 46 | "transform": out_transform}) 47 | with rasterio.open("FAPAR_masked.tif", "w", **out_meta) as dest: 48 | dest.write(out_image) 49 | 50 | # Calculate NDWI: (green-nir)/(green+nir) 51 | ndwi = (green.astype(float) - nir.astype(float))/(green+nir) 52 | 53 | # Create the NDWI image 54 | meta = b3.meta 55 | meta.update(driver='GTiff') 56 | meta.update(dtype=rasterio.float32) 57 | with rasterio.open('NDWI.tif', 'w', **meta) as dst: 58 | dst.write(ndwi.astype(rasterio.float32)) 59 | 60 | # Convert to NDWI_masked image 61 | with rio.open("NDWI.tif") as src: 62 | out_image, out_transform = rio.mask.mask(src, nAngeles_proj.geometry,crop=True) 63 | out_meta = src.meta.copy() 64 | out_meta.update({"driver": "GTiff", 65 | "height": out_image.shape[1], 66 | "width": out_image.shape[2], 67 | "transform": out_transform}) 68 | with rasterio.open("NDWI_masked.tif", "w", **out_meta) as dest: 69 | dest.write(out_image) 70 | 71 | with rasterio.open('NDWI_masked.tif', 'r+') as ndwi: 72 | NDWI_masked = ndwi.read() # read all raster values 73 | 74 | with rasterio.open('FAPAR_masked.tif', 'r+') as fapar: 75 | FAPAR_masked = fapar.read() # read all raster values 76 | 77 | # Calculate Stagnant: (("NDWI_masked" > 0.xx) AND ("FAPAR_masked" > 0.xx)) * 1 78 | stagnant = np.logical_and(NDWI_masked > 0.05, FAPAR_masked > 0.05) 79 | 80 | # Create the Stagnant image 81 | meta = ndwi.meta.copy() 82 | meta.update(driver='GTiff') 83 | meta.update(dtype=rasterio.float32) 84 | with rasterio.open('Stagnant.tif', 'w', **meta) as dst: 85 | dst.write(stagnant.astype(rasterio.float32)) 86 | 87 | # Check Stagnant Image 88 | # if all black: no stagnant spots detected, then recompute stagnant 89 | # else: proceed to processing 90 | sw = rio.open(r"Stagnant.tif") 91 | fig, ax = plt.subplots(1, figsize=(20, 20)) 92 | show(sw, ax=ax) 93 | plt.show() 94 | 95 | # Convert stagnant image to vector layer: raster pixel to points 96 | # Generate joined layer: join vector layer by location 97 | # Save as csv output 98 | -------------------------------------------------------------------------------- /satellite/readme.md: -------------------------------------------------------------------------------- 1 | ## Drive containing satellite images extracted from Copernicus Sentinel Hub 2 | 3 | https://drive.google.com/drive/folders/1nyWbERLM_eoZFGu2vUAEAK7iIknRukTP 4 | 5 | ## Satellite Images extracted from Copernicus Sentinel Hub Service 6 | 7 | Check Sentinel [here](https://sentinel.esa.int/web/sentinel/sentinel-data-access). You can request for a free account [here](https://scihub.copernicus.eu/). 8 | 9 | Use QGIS to access the data. 10 | ![qgis](https://github.com/aedesproject/nasa_hack/blob/master/satellite/images/qgis-extract.png) 11 | 12 | Select Sentinel-2 Bands for 3, 4, 5, 8 for the extraction 13 | ![qgis-2](https://github.com/aedesproject/nasa_hack/blob/master/satellite/images/qgis-extract-2.png) 14 | 15 | ## FAPAR and NDWI Values 16 | 17 | Stagnant water (i.e. mosquito-breeding environments were derived using Fraction of Absorbed Photosynthetically Active Radiation (FAPAR) to derive a Vegetation Index and the Normalized Difference Water Index (NDWI). 18 | 19 | When a location hits a certain FAPAR and NDWI thresholds it indicates the possibility of a stagnant water pool. 20 | 21 | ## NDWI 22 | The NDWI index is most appropriate for water body mapping. The water body has strong absorbability and low radiation in the range from visible to infrared wavelengths. The index uses the green and Near Infra-red bands of remote sensing images based on this phenomenon. The NDWI can enhance the water information effectively in most cases. It is sensitive to built-up land and often results in over-estimated water bodies. https://www.sentinel-hub.com/eoproducts/ndwi-normalized-difference-water-index 23 | 24 | ## FAPAR 25 | The Fraction of Absorbed Photosynthetically Active Radiation (FAPAR, sometimes also noted fAPAR or fPAR) is the fraction of the incoming solar radiation in the Photosynthetically Active Radiation spectral region that is absorbed by a photosynthetic organism, typically describing the light absorption across an integrated plant canopy. This biophysical variable is directly related to the primary productivity of photosynthesis and some models use it to estimate the assimilation of carbon dioxide in vegetation. https://en.wikipedia.org/wiki/FAPAR 26 | 27 | ## Extraction 28 | 29 | FAPAR Extraction 30 | ![fapar](https://github.com/aedesproject/nasa_hack/blob/master/satellite/images/fapar.png) 31 | 32 | NDWI Extraction 33 | ![ndwi](https://github.com/aedesproject/nasa_hack/blob/master/satellite/images/ndwi.png) 34 | 35 | Where: 36 | * -1 to 0 - Bright surface with no vegetation or water content 37 | * +1 - represent water content 38 | 39 | Formula to derive stagnant water: 40 | is_pixel_stagnant_water(boolean) = ((NDWIpixel>0.2) AND (FAPARpixel<0.1)) * 1 41 | ![fapar-ndwi-formula](https://github.com/aedesproject/nasa_hack/blob/master/satellite/images/fapar-ndwi-formula.png) 42 | 43 | Variables: 44 | * Xgreen = X green band of visible light 45 | * Xnir = X near infra red band 46 | * B8 = Landsat8 47 | * B4 = Landsat4 48 | --------------------------------------------------------------------------------