├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts.ipynb ├── benchmark-deep-learning-models-for-multivariate-time-series.ipynb └── images ├── arima.png ├── deepar.png ├── mean.png ├── prophet.png └── seasonal.png /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts 2 | 3 | Sample Code for use of the Gluonts Python library in AWS Sagemaker Notebook Instance to benchmark popular time series forecast Algorithms, including 4 | * [ARIMA](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) 5 | * [Prophet](https://facebook.github.io/prophet/docs/quick_start.html) 6 | * [DeepAR](https://gluon-ts.mxnet.io/api/gluonts/gluonts.model.deepar.html) 7 | 8 | amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts.ipynb gives an example on how to compare forecast algorithms on a dataset by only using the Gluonts library. 9 | * The Jupyter notebook should be run in a AWS Sagemker Notebook Instance (ml.m5.4xlarge is recommended) 10 | * Pls use the conda_python3 kernel. 11 | 12 | The example charts below visualize the comparison of different algorithms on prediction the same time series. 13 | 14 | ![mean](images/mean.png) 15 | ![naive seasonal](images/seasonal.png) 16 | ![arima](images/arima.png) 17 | ![prophet](images/prophet.png) 18 | ![deepar](images/deepar.png) 19 | 20 | ## License 21 | 22 | This library is licensed under the MIT-0 License. See the LICENSE file. 23 | 24 | -------------------------------------------------------------------------------- /amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Contents\n", 8 | "\n", 9 | "1. [Background](#1)\n", 10 | "2. [Setup the Environment](#2)\n", 11 | "3. [Prepare the Data](#3)\n", 12 | "4. [Algorithm Comparison](#4)" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# 1. Background" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "Time series forecast is a very common problem in many real-world applications. A wide spectrum of algorithms have been proposed to solve this problem. However, it is usually difficult to benchmark different algorithms and compare their performance due to the various implementation of the algorithms. This notebook tries to show an example how to benchmark different time series forecast algorithms by only using the Glutonts library." 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": {}, 32 | "source": [ 33 | "# 2. Setup the Environment" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "## 2.1 install R for extenal r forecast algorithms" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "# intall the missing lib for R\n", 50 | "!sudo yum install libXt-1.1.4-6.1.9.amzn1.x86_64 -y" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": { 57 | "scrolled": true 58 | }, 59 | "outputs": [], 60 | "source": [ 61 | "# install python r interface\n", 62 | "!conda install -c r rpy2==2.9.4 --yes" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": { 69 | "scrolled": true 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "# install forecast R packages\n", 74 | "!R -e 'install.packages(c(\"forecast\", \"nnfor\"), repos=\"https://cloud.r-project.org\")'" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## 2.2 Install Python Pacakages" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "scrolled": true 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "# install Prophet python packages\n", 93 | "!conda install -c plotly plotly==3.10.0 --yes\n", 94 | "!conda install -c conda-forge fbprophet=0.5=py36he1b5a44_0 --yes" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "!conda install -c anaconda ujson=1.35=py36h14c3975_0 --yes" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "scrolled": true 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "# install gluonts\n", 115 | "!pip install gluonts==0.4.2" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "# install mxnet\n", 125 | "!pip install mxnet==1.4.1" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "## 2.3 import libraries" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": {}, 139 | "outputs": [], 140 | "source": [ 141 | "%matplotlib inline\n", 142 | "\n", 143 | "import matplotlib.pyplot as plt\n", 144 | "import numpy as np\n", 145 | "import os\n", 146 | "import pandas as pd\n", 147 | "import random\n", 148 | "import sys\n", 149 | "import zipfile\n", 150 | "import mxnet as mx\n", 151 | "\n", 152 | "from urllib.request import urlretrieve\n", 153 | "\n", 154 | "from gluonts.model.trivial.mean import MeanPredictor\n", 155 | "from gluonts.model.seasonal_naive import SeasonalNaivePredictor\n", 156 | "from gluonts.model.r_forecast import RForecastPredictor\n", 157 | "from gluonts.model.prophet import ProphetPredictor\n", 158 | "from gluonts.model.deepar import DeepAREstimator\n", 159 | "from gluonts.trainer import Trainer\n", 160 | "from gluonts.dataset.common import ListDataset\n", 161 | "from itertools import islice\n", 162 | "from gluonts.evaluation.backtest import make_evaluation_predictions\n", 163 | "from gluonts.evaluation import Evaluator" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "# fix some plot issues caused by Prophet model\n", 173 | "# pls refer to https://darektidwell.com/typeerror-float-argument-must-be-a-string-or-a-number-not-period-facebook-prophet-and-pandas/\n", 174 | "pd.plotting.register_matplotlib_converters()" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": null, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "# set random seeds for reproducibility\n", 184 | "np.random.seed(42)\n", 185 | "random.seed(42)\n", 186 | "mx.random.seed(42)" 187 | ] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "metadata": {}, 192 | "source": [ 193 | "# 3. Prepare the Data" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "metadata": {}, 199 | "source": [ 200 | "Import electricity dataset, we need to download the original data set of from the UCI data set repository." 201 | ] 202 | }, 203 | { 204 | "cell_type": "markdown", 205 | "metadata": {}, 206 | "source": [ 207 | "## 3.1 Download data" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "DATA_HOST = \"https://archive.ics.uci.edu\"\n", 217 | "DATA_PATH = \"/ml/machine-learning-databases/00321/\"\n", 218 | "ARCHIVE_NAME = \"LD2011_2014.txt.zip\"\n", 219 | "FILE_NAME = ARCHIVE_NAME[:-4]" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "def progress_report_hook(count, block_size, total_size):\n", 229 | " mb = int(count * block_size // 1e6)\n", 230 | " if count % 500 == 0:\n", 231 | " sys.stdout.write(\"\\r{} MB downloaded\".format(mb))\n", 232 | " sys.stdout.flush()\n", 233 | "\n", 234 | "if not os.path.isfile(FILE_NAME):\n", 235 | " print(\"downloading dataset (258MB), can take a few minutes depending on your connection\")\n", 236 | " urlretrieve(DATA_HOST + DATA_PATH + ARCHIVE_NAME, ARCHIVE_NAME, reporthook=progress_report_hook)\n", 237 | "\n", 238 | " print(\"\\nextracting data archive\")\n", 239 | " zip_ref = zipfile.ZipFile(ARCHIVE_NAME, 'r')\n", 240 | " zip_ref.extractall(\"./\")\n", 241 | " zip_ref.close()\n", 242 | "else:\n", 243 | " print(\"File found skipping download\")" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "data = pd.read_csv(FILE_NAME, sep=\";\", index_col=0, parse_dates=True, decimal=',')\n", 253 | "num_timeseries = data.shape[1]\n", 254 | "data_kw = data.resample('2H').sum() / 8\n", 255 | "timeseries = []\n", 256 | "for i in range(num_timeseries):\n", 257 | " timeseries.append(np.trim_zeros(data_kw.iloc[:,i], trim='f'))" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": {}, 264 | "outputs": [], 265 | "source": [ 266 | "fig, axs = plt.subplots(5, 2, figsize=(20, 20), sharex=True)\n", 267 | "axx = axs.ravel()\n", 268 | "for i in range(0, 10):\n", 269 | " timeseries[i].loc[\"2014-01-01\":\"2014-01-14\"].plot(ax=axx[i])\n", 270 | " axx[i].set_xlabel(\"date\") \n", 271 | " axx[i].set_ylabel(\"kW consumption\") \n", 272 | " axx[i].grid(which='minor', axis='x')" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "## 3.2 Train Test Data Split" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [ 288 | "def split_train_test_data(timeseries,\n", 289 | " start_dataset,\n", 290 | " end_training,\n", 291 | " num_test_windows):\n", 292 | " # create training data.\n", 293 | " training_data = [\n", 294 | " {\n", 295 | " \"start\": str(start_dataset),\n", 296 | " \"target\": ts[start_dataset:end_training - 1].tolist(), # We use -1, because pandas indexing includes the upper bound\n", 297 | " \"feat_static_cat\": [id]\n", 298 | " }\n", 299 | " for id, ts in enumerate(timeseries)\n", 300 | " ]\n", 301 | " \n", 302 | " # create testing data.\n", 303 | " test_data = [\n", 304 | " {\n", 305 | " \"start\": str(start_dataset),\n", 306 | " \"target\": ts[start_dataset:end_training + k * prediction_length].tolist(),\n", 307 | " \"feat_static_cat\": [id]\n", 308 | " }\n", 309 | " for k in range(1, num_test_windows + 1)\n", 310 | " for id, ts in enumerate(timeseries)\n", 311 | " ]\n", 312 | " \n", 313 | " return training_data, test_data" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "# we use 2 hour frequency for the time series\n", 323 | "freq = '2H'\n", 324 | "\n", 325 | "# we predict for 1 day\n", 326 | "prediction_length = 1 * 12\n", 327 | "\n", 328 | "# we also use 7 days as context length, this is the number of state updates accomplished before making predictions\n", 329 | "context_length = 7 * 12\n", 330 | "\n", 331 | "# The moving window for forecast\n", 332 | "num_test_windows = 1\n", 333 | "\n", 334 | "# training/test Split\n", 335 | "start_dataset = pd.Timestamp(\"2014-01-01 00:00:00\", freq=freq)\n", 336 | "end_training = pd.Timestamp(\"2014-09-01 00:00:00\", freq=freq)\n", 337 | "\n", 338 | "# number of time series selected\n", 339 | "n_timeseries = 50" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "metadata": {}, 346 | "outputs": [], 347 | "source": [ 348 | "training_data, test_data = split_train_test_data(timeseries[:n_timeseries],\n", 349 | " start_dataset,\n", 350 | " end_training,\n", 351 | " num_test_windows)" 352 | ] 353 | }, 354 | { 355 | "cell_type": "markdown", 356 | "metadata": {}, 357 | "source": [ 358 | "# 4. Algorithm Comparison" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "def train_and_test(training_data, \n", 368 | " test_data,\n", 369 | " freq,\n", 370 | " num_test_windows,\n", 371 | " model,\n", 372 | " train_per_ts=False,\n", 373 | " require_train=False\n", 374 | " ):\n", 375 | " forecasts = []\n", 376 | " tss = []\n", 377 | " # if training per time series is required.\n", 378 | " if train_per_ts:\n", 379 | " # iterate over the timeseries\n", 380 | " count = 0\n", 381 | " for training_ts in training_data:\n", 382 | " # get the training time series\n", 383 | " training_ts = ListDataset(\n", 384 | " [training_ts],\n", 385 | " freq = freq\n", 386 | " )\n", 387 | " # get the related testing time series\n", 388 | " test_tss = test_data[count*num_test_windows: (count+1)*num_test_windows]\n", 389 | "\n", 390 | " test_tss = ListDataset(\n", 391 | " test_tss,\n", 392 | " freq = freq\n", 393 | " )\n", 394 | " if require_train:\n", 395 | " predictor = model.train(training_data=training_ts)\n", 396 | " else:\n", 397 | " predictor = model\n", 398 | " \n", 399 | " forecast_it, ts_it = make_evaluation_predictions(test_tss, predictor=predictor, num_samples=100)\n", 400 | " forecasts.extend(list(forecast_it))\n", 401 | " tss.extend(list(ts_it))\n", 402 | " count += 1\n", 403 | " else:\n", 404 | " training_data = ListDataset(\n", 405 | " training_data,\n", 406 | " freq = freq\n", 407 | " )\n", 408 | " test_data = ListDataset(\n", 409 | " test_data,\n", 410 | " freq = freq\n", 411 | " )\n", 412 | " if require_train:\n", 413 | " predictor = model.train(training_data=training_data)\n", 414 | " else:\n", 415 | " predictor = model\n", 416 | " \n", 417 | " forecast_it, ts_it = make_evaluation_predictions(test_data, predictor=predictor, num_samples=100)\n", 418 | " forecasts.extend(list(forecast_it))\n", 419 | " tss.extend(list(ts_it))\n", 420 | " \n", 421 | " return forecasts, tss" 422 | ] 423 | }, 424 | { 425 | "cell_type": "markdown", 426 | "metadata": {}, 427 | "source": [ 428 | "## 4.1 Training and Testing" 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": {}, 434 | "source": [ 435 | "### Mean" 436 | ] 437 | }, 438 | { 439 | "cell_type": "code", 440 | "execution_count": null, 441 | "metadata": {}, 442 | "outputs": [], 443 | "source": [ 444 | "%%time\n", 445 | "mean= MeanPredictor(freq=freq, prediction_length=prediction_length,\n", 446 | " context_length=context_length)\n", 447 | "forecasts_mean, tss_mean = train_and_test(training_data, \n", 448 | " test_data,\n", 449 | " freq,\n", 450 | " num_test_windows,\n", 451 | " mean,\n", 452 | " train_per_ts=True,\n", 453 | " require_train=False\n", 454 | " )" 455 | ] 456 | }, 457 | { 458 | "cell_type": "markdown", 459 | "metadata": {}, 460 | "source": [ 461 | "### Seaonal Naive" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "%%time\n", 471 | "seasonal = SeasonalNaivePredictor(freq=freq,\n", 472 | " prediction_length=prediction_length,\n", 473 | " season_length=context_length)\n", 474 | "forecasts_seasonal, tss_seasonal = train_and_test(training_data, \n", 475 | " test_data,\n", 476 | " freq,\n", 477 | " num_test_windows,\n", 478 | " seasonal,\n", 479 | " train_per_ts=True,\n", 480 | " require_train=False\n", 481 | " )" 482 | ] 483 | }, 484 | { 485 | "cell_type": "markdown", 486 | "metadata": {}, 487 | "source": [ 488 | "### ARIMA" 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": null, 494 | "metadata": {}, 495 | "outputs": [], 496 | "source": [ 497 | "%%time\n", 498 | "arima = RForecastPredictor(freq=freq,\n", 499 | " prediction_length=prediction_length,\n", 500 | " method_name='arima')\n", 501 | "forecasts_arima, tss_arima = train_and_test(training_data, \n", 502 | " test_data,\n", 503 | " freq,\n", 504 | " num_test_windows,\n", 505 | " arima,\n", 506 | " train_per_ts=True,\n", 507 | " require_train=False\n", 508 | " )" 509 | ] 510 | }, 511 | { 512 | "cell_type": "markdown", 513 | "metadata": {}, 514 | "source": [ 515 | "### Prophet" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": {}, 522 | "outputs": [], 523 | "source": [ 524 | "%%time\n", 525 | "prophet = ProphetPredictor(freq, prediction_length)\n", 526 | "forecasts_prophet, tss_prophet = train_and_test(training_data, \n", 527 | " test_data,\n", 528 | " freq,\n", 529 | " num_test_windows,\n", 530 | " prophet,\n", 531 | " train_per_ts=True,\n", 532 | " require_train=False\n", 533 | " )" 534 | ] 535 | }, 536 | { 537 | "cell_type": "markdown", 538 | "metadata": {}, 539 | "source": [ 540 | "### DeepAR" 541 | ] 542 | }, 543 | { 544 | "cell_type": "code", 545 | "execution_count": null, 546 | "metadata": {}, 547 | "outputs": [], 548 | "source": [ 549 | "%%time\n", 550 | "deepar = DeepAREstimator(freq=freq,\n", 551 | " use_feat_static_cat=True,\n", 552 | " cardinality=[n_timeseries],\n", 553 | " prediction_length=prediction_length,\n", 554 | " trainer=Trainer(epochs=100),\n", 555 | " num_cells=40)\n", 556 | "forecasts_deepar, tss_deepar = train_and_test(training_data, \n", 557 | " test_data,\n", 558 | " freq,\n", 559 | " num_test_windows,\n", 560 | " deepar,\n", 561 | " train_per_ts=False,\n", 562 | " require_train=True\n", 563 | " )" 564 | ] 565 | }, 566 | { 567 | "cell_type": "markdown", 568 | "metadata": {}, 569 | "source": [ 570 | "## 4.2 Results" 571 | ] 572 | }, 573 | { 574 | "cell_type": "markdown", 575 | "metadata": {}, 576 | "source": [ 577 | "### Evaluation" 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "execution_count": null, 583 | "metadata": {}, 584 | "outputs": [], 585 | "source": [ 586 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 587 | "agg_metrics_mean, item_metrics_mean = evaluator(iter(tss_mean), iter(forecasts_mean), num_series=len(forecasts_mean))\n", 588 | "print(agg_metrics_mean)" 589 | ] 590 | }, 591 | { 592 | "cell_type": "code", 593 | "execution_count": null, 594 | "metadata": {}, 595 | "outputs": [], 596 | "source": [ 597 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 598 | "agg_metrics_seasonal, item_metrics_seasonal = evaluator(iter(tss_seasonal), \n", 599 | " iter(forecasts_seasonal), \n", 600 | " num_series=len(forecasts_seasonal))\n", 601 | "print(agg_metrics_seasonal)" 602 | ] 603 | }, 604 | { 605 | "cell_type": "code", 606 | "execution_count": null, 607 | "metadata": {}, 608 | "outputs": [], 609 | "source": [ 610 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 611 | "agg_metrics_arima, item_metrics_arima = evaluator(iter(tss_arima), \n", 612 | " iter(forecasts_arima), \n", 613 | " num_series=len(forecasts_arima))\n", 614 | "print(agg_metrics_arima)" 615 | ] 616 | }, 617 | { 618 | "cell_type": "code", 619 | "execution_count": null, 620 | "metadata": {}, 621 | "outputs": [], 622 | "source": [ 623 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 624 | "agg_metrics_prophet, item_metrics_prophet = evaluator(iter(tss_prophet), \n", 625 | " iter(forecasts_prophet), \n", 626 | " num_series=len(forecasts_prophet))\n", 627 | "print(agg_metrics_prophet)" 628 | ] 629 | }, 630 | { 631 | "cell_type": "code", 632 | "execution_count": null, 633 | "metadata": {}, 634 | "outputs": [], 635 | "source": [ 636 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 637 | "agg_metrics_deepar, item_metrics_deepar = evaluator(iter(tss_deepar), \n", 638 | " iter(forecasts_deepar), \n", 639 | " num_series=len(forecasts_deepar))\n", 640 | "print(agg_metrics_deepar)" 641 | ] 642 | }, 643 | { 644 | "cell_type": "code", 645 | "execution_count": null, 646 | "metadata": {}, 647 | "outputs": [], 648 | "source": [ 649 | "df_metrics = pd.concat(\n", 650 | " [\n", 651 | " pd.DataFrame.from_dict(agg_metrics_deepar, orient='index').rename(columns={0: \"DeepAR\"}),\n", 652 | " pd.DataFrame.from_dict(agg_metrics_prophet, orient='index').rename(columns={0: \"Prophet\"}),\n", 653 | " pd.DataFrame.from_dict(agg_metrics_arima, orient='index').rename(columns={0: \"ARIMA\"}),\n", 654 | " pd.DataFrame.from_dict(agg_metrics_seasonal, orient='index').rename(columns={0: \"Seasonal naive\"}),\n", 655 | " pd.DataFrame.from_dict(agg_metrics_mean, orient='index').rename(columns={0: \"Mean\"})], axis=1\n", 656 | ")\n", 657 | "df_metrics.loc[[\"MASE\", \"RMSE\", \"sMAPE\"]]" 658 | ] 659 | }, 660 | { 661 | "cell_type": "markdown", 662 | "metadata": {}, 663 | "source": [ 664 | "### Plot Example Forecast" 665 | ] 666 | }, 667 | { 668 | "cell_type": "code", 669 | "execution_count": null, 670 | "metadata": {}, 671 | "outputs": [], 672 | "source": [ 673 | "def plot_forecasts(tss, forecasts, past_length, start, stop, step, title):\n", 674 | " for target, forecast in islice(zip(tss, forecasts), start, stop, step):\n", 675 | " ax = target[-past_length:].plot(figsize=(12, 5), linewidth=2)\n", 676 | " forecast.plot(color='g')\n", 677 | " plt.title(title)\n", 678 | " plt.grid(which='both')\n", 679 | " plt.legend([\"observations\", \"median prediction\", \"90% confidence interval\", \"50% confidence interval\"])\n", 680 | " plt.show()" 681 | ] 682 | }, 683 | { 684 | "cell_type": "code", 685 | "execution_count": null, 686 | "metadata": {}, 687 | "outputs": [], 688 | "source": [ 689 | "start, stop, step = 10, 11, 1\n", 690 | "plot_forecasts(tss_mean, forecasts_mean, past_length=100, start=start, stop=stop, step=step, title=\"mean\")\n", 691 | "plot_forecasts(tss_seasonal, forecasts_seasonal, past_length=100, start=start, stop=stop, step=step, title=\"seasonal\")\n", 692 | "plot_forecasts(tss_arima, forecasts_arima, past_length=100, start=start, stop=stop, step=step, title=\"arima\")\n", 693 | "plot_forecasts(tss_prophet, forecasts_prophet, past_length=100, start=start, stop=stop, step=step, title=\"prophet\")\n", 694 | "plot_forecasts(tss_deepar, forecasts_deepar, past_length=100, start=start, stop=stop, step=step, title=\"deepar\")" 695 | ] 696 | } 697 | ], 698 | "metadata": { 699 | "kernelspec": { 700 | "display_name": "conda_python3", 701 | "language": "python", 702 | "name": "conda_python3" 703 | }, 704 | "language_info": { 705 | "codemirror_mode": { 706 | "name": "ipython", 707 | "version": 3 708 | }, 709 | "file_extension": ".py", 710 | "mimetype": "text/x-python", 711 | "name": "python", 712 | "nbconvert_exporter": "python", 713 | "pygments_lexer": "ipython3", 714 | "version": "3.6.7" 715 | } 716 | }, 717 | "nbformat": 4, 718 | "nbformat_minor": 4 719 | } 720 | -------------------------------------------------------------------------------- /benchmark-deep-learning-models-for-multivariate-time-series.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### Contents:\n", 8 | "1. [Background](#Background)\n", 9 | "2. [Environment Setup](#Environment-Setup)\n", 10 | " - [Install the required python packages](#Install-the-required-python-packages)\n", 11 | " - [Import the required libraries](#Import-the-required-libraries)\n", 12 | "3. [Download and prepare the data](#Download-and-prepare-the-data)\n", 13 | "4. [Scenario 1: Related time series are available in the forecast horizon](#Scenario-1:-Related-time-series-are-available-in-the-forecast-horizon)\n", 14 | "5. [Scenario 2: Related time series is not available in the forecast horizon](#Scenario-2:-Related-time-series-is-not-available-in-the-forecast-horizon)\n", 15 | "6. [Scenario 3: Model all the time series as target series](#Model-all-the-time-series-as-target-series)" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "## 1. Background\n", 23 | "\n", 24 | "Multivariate time series forecasting is a common problem and more recently deep learning models have been applied to time series forecasting. [GluonTS](https://ts.gluon.ai/index.html) is a deep learning toolkit for probabilistic modelling of time series. This notebook shows you different ways in which one can model a multivariate time series problem (time series with related variables) using different models that are implemented in GluonTS.\n", 25 | "The following models are explored in this notebook -\n", 26 | "- [DeepAR](https://ts.gluon.ai/api/gluonts/gluonts.model.deepar.html)\n", 27 | "- [Transformer](https://ts.gluon.ai/api/gluonts/gluonts.model.transformer.html)\n", 28 | "- [MQ-CNN](https://ts.gluon.ai/api/gluonts/gluonts.model.seq2seq.html)\n", 29 | "- [Temporal Fusion Transformer](https://ts.gluon.ai/api/gluonts/gluonts.model.tft.html)\n", 30 | "- [LSTNet](https://ts.gluon.ai/api/gluonts/gluonts.model.lstnet.html)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "## 2. Environment Setup\n", 38 | "\n", 39 | "Please run this notebook on an instance that has a GPU. (p2.xlarge or higher) " 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## 2.1 Install the required python packages" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "!pip uninstall -y mxnet" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "!pip install --upgrade mxnet~=1.7\n", 65 | "!pip install gluonts\n", 66 | "!pip install mxnet-cu102==1.7.0" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "## 2.2 Import the required libraries" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "import pandas as pd\n", 83 | "from gluonts.dataset.common import (\n", 84 | " CategoricalFeatureInfo,\n", 85 | " ListDataset,\n", 86 | " MetaData,\n", 87 | " TrainDatasets,\n", 88 | " load_datasets\n", 89 | ")\n", 90 | "\n", 91 | "from gluonts.dataset.field_names import FieldName\n", 92 | "from gluonts.model.deepar import DeepAREstimator\n", 93 | "from gluonts.model.transformer import TransformerEstimator\n", 94 | "from gluonts.model.lstnet import LSTNetEstimator\n", 95 | "from gluonts.model.seq2seq import MQCNNEstimator\n", 96 | "from gluonts.model.seq2seq import MQRNNEstimator\n", 97 | "from gluonts.model.tft import TemporalFusionTransformerEstimator\n", 98 | "from gluonts.evaluation.backtest import make_evaluation_predictions\n", 99 | "from gluonts.evaluation import Evaluator, MultivariateEvaluator\n", 100 | "from gluonts.mx.trainer import Trainer\n", 101 | "from gluonts.dataset.multivariate_grouper import MultivariateGrouper\n", 102 | "\n", 103 | "import mxnet as mx\n", 104 | "\n", 105 | "from itertools import islice\n", 106 | "import matplotlib.pyplot as plt\n", 107 | "%matplotlib inline" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "## 3. Download and prepare the data\n", 115 | "\n", 116 | "We use the [PM 2.5 dataset](https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data) from the UCI Machine Learning Repository.\n", 117 | "\n", 118 | "The dataset contains PM 2.5 data from the US Embassy in Beijing and is supplemented with meteorological data. The following columns are part of the data - \n", 119 | "- No: row number\n", 120 | "- year: year of data in this row\n", 121 | "- month: month of data in this row\n", 122 | "- day: day of data in this row\n", 123 | "- hour: hour of data in this row\n", 124 | "- pm2.5: PM2.5 concentration (ug/m^3)\n", 125 | "- DEWP: Dew Point (℃)\n", 126 | "- TEMP: Temperature (℃)\n", 127 | "- PRES: Pressure (hPa)\n", 128 | "- cbwd: Combined wind direction\n", 129 | "- Iws: Cumulated wind speed (m/s)\n", 130 | "- Is: Cumulated hours of snow\n", 131 | "- Ir: Cumulated hours of rain\n", 132 | "\n", 133 | "Given the above information, here is how the different features in the dataset are treated - \n", 134 | "- pm2.5 is the target variable. \n", 135 | "- Meteorological variables like 'TEMP', 'DEWP' and 'PRES' can be treated as related time series with real values.\n", 136 | "- 'cbwd' is a categorical variable and varies with time and can be treated as a dynamic categorical feature.\n", 137 | "\n", 138 | "There are different ways in which one can model multivariate time series problems depending on the availability of related time series features in the forecast horizon. This notebook illustrates them by assuming the presence or absence of the meteorological variables in the forecast horizon." 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "!wget https://archive.ics.uci.edu/ml/machine-learning-databases/00381/PRSA_data_2010.1.1-2014.12.31.csv" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "metadata": {}, 154 | "outputs": [], 155 | "source": [ 156 | "df = pd.read_csv(\"PRSA_data_2010.1.1-2014.12.31.csv\")\n", 157 | "\n", 158 | "#combine month,day,hour into a timestamp\n", 159 | "df['Timestamp'] = pd.to_datetime(df[['year', 'month', 'day', 'hour']])\n", 160 | "\n", 161 | "#set an ID to identify a time series\n", 162 | "df['id'] = 0\n", 163 | "\n", 164 | "#set the type of the categorical variable\n", 165 | "df[\"cbwd\"] = df[\"cbwd\"].astype('category')\n", 166 | "df[\"cbwd_cat\"] = df[\"cbwd\"].cat.codes" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "df.columns" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "## 4. Related time series are available in the forecast horizon\n", 183 | "\n", 184 | "In this section, you will assume that the meteorological variables (TEMP, DEWP, PRES) are available to the model in the forecast horizon. In real life, this could be from a weather prediction model or forecast.\n", 185 | "\n", 186 | "The following cells compare a DeepAR and Transformer in this particular scenario." 187 | ] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "metadata": {}, 192 | "source": [ 193 | "### 4.1 Prepare the training and testing dataset" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": {}, 200 | "outputs": [], 201 | "source": [ 202 | "forecast_length = 120\n", 203 | "num_backtest_windows = 2" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "training_data_list = []\n", 213 | "test_data_list = []\n", 214 | "\n", 215 | "for i in reversed(range(1, num_backtest_windows+1)):\n", 216 | " \n", 217 | " training_data = [\n", 218 | " {\n", 219 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 220 | " \"target\": df[\"pm2.5\"][:-forecast_length*i],\n", 221 | " \"feat_static_cat\": [0],\n", 222 | " \"feat_dynamic_real\": [df[\"TEMP\"][:-forecast_length*i],\n", 223 | " df[\"DEWP\"][:-forecast_length*i]],\n", 224 | " \"feat_dynamic_cat\": [df[\"cbwd_cat\"][:-forecast_length*i]]\n", 225 | " }\n", 226 | " ]\n", 227 | " \n", 228 | " # create testing data.\n", 229 | " test_data = [\n", 230 | " {\n", 231 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 232 | " \"target\": df[\"pm2.5\"][:-forecast_length*(i-1)] if i>1 else df[\"pm2.5\"][:],\n", 233 | " \"feat_static_cat\": [0],\n", 234 | " \"feat_dynamic_real\": [df[\"TEMP\"][:-forecast_length*(i-1)] if i>1 else df[\"TEMP\"][:],\n", 235 | " df[\"DEWP\"][:-forecast_length*(i-1)] if i>1 else df[\"DEWP\"][:]],\n", 236 | " \"feat_dynamic_cat\": [df[\"cbwd_cat\"][:-forecast_length*(i-1)] if i>1 else df[\"cbwd_cat\"][:]]\n", 237 | " }\n", 238 | " ]\n", 239 | "\n", 240 | " training_data_list.append(ListDataset(training_data, freq='1h'))\n", 241 | " test_data_list.append(ListDataset(test_data, freq='1h'))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": {}, 248 | "outputs": [], 249 | "source": [ 250 | "#function that takes a gluonTS estimator, trains the model and predicts it for every pair of train and test dataset\n", 251 | "def backtest(model,\n", 252 | " training_data_list,\n", 253 | " test_data_list,\n", 254 | " num_backtest_windows):\n", 255 | " \n", 256 | " forecasts = []\n", 257 | " obs = []\n", 258 | " \n", 259 | " #train a model for every backtest window\n", 260 | " for i in range(num_backtest_windows):\n", 261 | " predictor = model.train(training_data_list[i],\n", 262 | " force_reinit=True)\n", 263 | " forecast_it, ts_it = make_evaluation_predictions(test_data_list[i], \n", 264 | " predictor=predictor, \n", 265 | " num_samples=100)\n", 266 | " forecasts.extend(list(forecast_it))\n", 267 | " obs.extend(list(ts_it))\n", 268 | " return forecasts, obs" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": {}, 274 | "source": [ 275 | "#### DeepAR" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "metadata": { 282 | "jupyter": { 283 | "outputs_hidden": true 284 | } 285 | }, 286 | "outputs": [], 287 | "source": [ 288 | "deepar = DeepAREstimator(freq=\"1h\",\n", 289 | " use_feat_static_cat=True,\n", 290 | " use_feat_dynamic_real=True,\n", 291 | " cardinality=[1],\n", 292 | " prediction_length=forecast_length,\n", 293 | " trainer=Trainer(epochs=30, ctx = mx.context.gpu(0)),\n", 294 | " num_cells=40)\n", 295 | "forecast_deepar, obs_deepar = backtest(deepar,\n", 296 | " training_data_list,\n", 297 | " test_data_list,\n", 298 | " num_backtest_windows)" 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": {}, 304 | "source": [ 305 | "#### Transformer" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": null, 311 | "metadata": { 312 | "jupyter": { 313 | "outputs_hidden": true 314 | } 315 | }, 316 | "outputs": [], 317 | "source": [ 318 | "transformer = TransformerEstimator(freq=\"1h\",\n", 319 | " use_feat_dynamic_real=True,\n", 320 | " #context_length=168,\n", 321 | " prediction_length=forecast_length,\n", 322 | " trainer=Trainer(epochs=10,\n", 323 | " learning_rate=0.01,\n", 324 | " ctx = mx.context.gpu(0))\n", 325 | " )\n", 326 | "forecast_transformer, obs_transformer = backtest(transformer,\n", 327 | " training_data_list,\n", 328 | " test_data_list,\n", 329 | " num_backtest_windows)" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": {}, 335 | "source": [ 336 | "#### Evaluation" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": null, 342 | "metadata": {}, 343 | "outputs": [], 344 | "source": [ 345 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 346 | "agg_metrics_deepar, item_metrics_deepar = evaluator(iter(obs_deepar), \n", 347 | " iter(forecast_deepar), \n", 348 | " num_series=len(forecast_deepar))\n", 349 | "\n", 350 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 351 | "agg_metrics_transformer, item_metrics_transformer = evaluator(iter(obs_transformer),\n", 352 | " iter(forecast_transformer),\n", 353 | " num_series=len(forecast_transformer))\n" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "metadata": {}, 359 | "source": [ 360 | "#### Results" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": {}, 367 | "outputs": [], 368 | "source": [ 369 | "pd.concat([pd.DataFrame.from_dict(agg_metrics_deepar, orient='index').rename(columns={0: \"DeepAR\"}),\n", 370 | " pd.DataFrame.from_dict(agg_metrics_transformer, orient='index').rename(columns={0: \"Transformer\"})],\n", 371 | " axis=1)" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "metadata": {}, 378 | "outputs": [], 379 | "source": [ 380 | "def plot_forecasts(obs, forecasts, past_length, start, stop, step, title):\n", 381 | " for target, forecast in zip(obs, forecasts):\n", 382 | " ax = target[-past_length:].plot(figsize=(12, 5), linewidth=2)\n", 383 | " forecast.plot(color='g')\n", 384 | " plt.ylabel('PM2.5 concentration (ug/m^3)')\n", 385 | " plt.title(title)\n", 386 | " plt.grid(which='both')\n", 387 | " plt.legend([\"observations\", \"median prediction\", \"90% confidence interval\", \"50% confidence interval\"])\n", 388 | " plt.show()" 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": {}, 394 | "source": [ 395 | "The below charts plot the observed PM2.5 against the forecast from DeepAR and Transformer. Since, the model computes probabilistic forecasts, it is possible to draw a confidence interval around the median prediction. These charts show a 50% and 90% confidence interval." 396 | ] 397 | }, 398 | { 399 | "cell_type": "markdown", 400 | "metadata": {}, 401 | "source": [ 402 | "#### Plot Sample Forecast - DeepAR" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": null, 408 | "metadata": {}, 409 | "outputs": [], 410 | "source": [ 411 | "plot_forecasts(obs_deepar, forecast_deepar, 340, 0, 2, 1, 'deepar')" 412 | ] 413 | }, 414 | { 415 | "cell_type": "markdown", 416 | "metadata": {}, 417 | "source": [ 418 | "#### Plot Sample Forecast - Transformer" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": null, 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "plot_forecasts(obs_transformer, forecast_transformer, 340, 0, 2, 1, 'transformer')" 428 | ] 429 | }, 430 | { 431 | "cell_type": "markdown", 432 | "metadata": {}, 433 | "source": [ 434 | "## 5. Related time series is not available in the forecast horizon" 435 | ] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "metadata": {}, 440 | "source": [ 441 | "In this section, you will see how to train models when the related time series (meteorological features in this case) is not available in the forecast horizon. The meteorological variables are only present for the historical time period and can hence be used for training the model." 442 | ] 443 | }, 444 | { 445 | "cell_type": "markdown", 446 | "metadata": {}, 447 | "source": [ 448 | "Seq2Seq models like [MQ-CNN](https://ts.gluon.ai/api/gluonts/gluonts.model.seq2seq.html) which uses a CNN as an encoder and a MLP as a decoder can be used in this scenario.\n", 449 | "\n", 450 | "[Temporal Fusion Transformer](https://arxiv.org/abs/1912.09363) is another architecture that combines recurrent layers and attention layers to enable the usage of a mix of inputs like exogenous variables that are only observed historically and other static and dynamic covariates.\n", 451 | "\n", 452 | "We compare the above to models in this section." 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "### 5.1 Prepare the training and testing dataset to use 'past_feat_dynamic_real'" 460 | ] 461 | }, 462 | { 463 | "cell_type": "code", 464 | "execution_count": null, 465 | "metadata": {}, 466 | "outputs": [], 467 | "source": [ 468 | "training_data_list = []\n", 469 | "test_data_list = []\n", 470 | "\n", 471 | "for i in reversed(range(1, 3)):\n", 472 | " \n", 473 | " training_data = [\n", 474 | " {\n", 475 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 476 | " \"target\": df[\"pm2.5\"][:-forecast_length*i],\n", 477 | " \"past_feat_dynamic_real\": [df[\"TEMP\"][:-forecast_length*i],\n", 478 | " df[\"DEWP\"][:-forecast_length*i]],\n", 479 | " }\n", 480 | " ]\n", 481 | " \n", 482 | " # create testing data.\n", 483 | " test_data = [\n", 484 | " {\n", 485 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 486 | " \"target\": df[\"pm2.5\"][:-forecast_length*(i-1)] if i>1 else df[\"pm2.5\"][:],\n", 487 | " \"past_feat_dynamic_real\": [df[\"TEMP\"][:-forecast_length*(i-1)] if i>1 else df[\"TEMP\"][:],\n", 488 | " df[\"DEWP\"][:-forecast_length*(i-1)] if i>1 else df[\"DEWP\"][:]],\n", 489 | " }\n", 490 | " ]\n", 491 | "\n", 492 | " training_data_list.append(ListDataset(training_data, freq='1h'))\n", 493 | " test_data_list.append(ListDataset(test_data, freq='1h'))" 494 | ] 495 | }, 496 | { 497 | "cell_type": "markdown", 498 | "metadata": {}, 499 | "source": [ 500 | "#### MQ-CNN" 501 | ] 502 | }, 503 | { 504 | "cell_type": "code", 505 | "execution_count": null, 506 | "metadata": {}, 507 | "outputs": [], 508 | "source": [ 509 | "#At times, one can encounter exploding gradients and as a result the loss can become a NaN. \n", 510 | "#set hybridize=False. May be related to https://github.com/awslabs/gluon-ts/issues/833\n", 511 | "mqcnn = MQCNNEstimator(freq=\"1h\",\n", 512 | " use_past_feat_dynamic_real=True,\n", 513 | " prediction_length=forecast_length,\n", 514 | " trainer=Trainer(epochs=30,\n", 515 | " learning_rate=0.001,\n", 516 | " #clip_gradient=3,\n", 517 | " #batch_size=32,\n", 518 | " #num_batches_per_epoch=16,\n", 519 | " hybridize=False,\n", 520 | " ctx = mx.context.gpu(0)),\n", 521 | " \n", 522 | " )\n", 523 | "forecast_mqcnn, obs_mqcnn = backtest(mqcnn,\n", 524 | " training_data_list,\n", 525 | " test_data_list,\n", 526 | " num_backtest_windows)" 527 | ] 528 | }, 529 | { 530 | "cell_type": "markdown", 531 | "metadata": {}, 532 | "source": [ 533 | "#### Temporal Fusion Transformer" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "execution_count": null, 539 | "metadata": {}, 540 | "outputs": [], 541 | "source": [ 542 | "training_data_list = []\n", 543 | "test_data_list = []\n", 544 | "\n", 545 | "for i in reversed(range(1, 3)):\n", 546 | " \n", 547 | " training_data = [\n", 548 | " {\n", 549 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 550 | " \"target\": df[\"pm2.5\"][:-forecast_length*i],\n", 551 | " \"past_feat_dynamic_real_1\": df[\"TEMP\"][:-forecast_length*i],\n", 552 | " \"past_feat_dynamic_real_2\": df[\"DEWP\"][:-forecast_length*i],\n", 553 | " \"past_feat_dynamic_real_3\": df[\"Ir\"][:-forecast_length*i]\n", 554 | " }\n", 555 | " ]\n", 556 | " \n", 557 | " # create testing data.\n", 558 | " test_data = [\n", 559 | " {\n", 560 | " \"start\": df.iloc[0][\"Timestamp\"],\n", 561 | " \"target\": df[\"pm2.5\"][:-forecast_length*(i-1)] if i>1 else df[\"pm2.5\"][:],\n", 562 | " \"past_feat_dynamic_real_1\": df[\"TEMP\"][:-forecast_length*(i-1)] if i>1 else df[\"TEMP\"][:],\n", 563 | " \"past_feat_dynamic_real_2\": df[\"DEWP\"][:-forecast_length*(i-1)] if i>1 else df[\"DEWP\"][:],\n", 564 | " \"past_feat_dynamic_real_3\": df[\"Ir\"][:-forecast_length*(i-1)] if i>1 else df[\"Ir\"][:]\n", 565 | " }\n", 566 | " ]\n", 567 | "\n", 568 | " training_data_list.append(ListDataset(training_data, freq='1h'))\n", 569 | " test_data_list.append(ListDataset(test_data, freq='1h'))\n", 570 | "\n", 571 | "feat_past_dynamic_real = [\"past_feat_dynamic_real_1\", \"past_feat_dynamic_real_2\", \"past_feat_dynamic_real_3\"]" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "execution_count": null, 577 | "metadata": {}, 578 | "outputs": [], 579 | "source": [ 580 | "#https://github.com/awslabs/gluon-ts/issues/1075\n", 581 | "tft = TemporalFusionTransformerEstimator(freq = '1h',\n", 582 | " context_length=168,\n", 583 | " prediction_length = forecast_length,\n", 584 | " trainer=Trainer(epochs=30,\n", 585 | " learning_rate=0.001,\n", 586 | " ctx = mx.context.gpu(0)),\n", 587 | " hidden_dim=32,\n", 588 | " variable_dim=8,\n", 589 | " num_heads=4,\n", 590 | " num_outputs=3,\n", 591 | " num_instance_per_series=100,\n", 592 | " dropout_rate=0.1,\n", 593 | " dynamic_feature_dims={\n", 594 | " 'past_feat_dynamic_real_1': 1,\n", 595 | " 'past_feat_dynamic_real_2': 1,\n", 596 | " 'past_feat_dynamic_real_3': 1\n", 597 | " }, # dimensions of dynamic real features\n", 598 | " past_dynamic_features=feat_past_dynamic_real,\n", 599 | " )\n", 600 | "forecast_tft, obs_tft = backtest(tft,\n", 601 | " training_data_list,\n", 602 | " test_data_list,\n", 603 | " num_backtest_windows)" 604 | ] 605 | }, 606 | { 607 | "cell_type": "markdown", 608 | "metadata": {}, 609 | "source": [ 610 | "#### Evaluation" 611 | ] 612 | }, 613 | { 614 | "cell_type": "code", 615 | "execution_count": null, 616 | "metadata": {}, 617 | "outputs": [], 618 | "source": [ 619 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 620 | "agg_metrics_mqcnn, item_metrics_mqcnn = evaluator(iter(obs_mqcnn), \n", 621 | " iter(forecast_mqcnn), \n", 622 | " num_series=len(forecast_mqcnn))\n", 623 | "\n", 624 | "evaluator = Evaluator(quantiles=[0.5], seasonality=None)\n", 625 | "agg_metrics_tft, item_metrics_tft = evaluator(iter(obs_tft),\n", 626 | " iter(forecast_tft),\n", 627 | " num_series=len(forecast_tft))\n" 628 | ] 629 | }, 630 | { 631 | "cell_type": "markdown", 632 | "metadata": {}, 633 | "source": [ 634 | "#### Results" 635 | ] 636 | }, 637 | { 638 | "cell_type": "code", 639 | "execution_count": null, 640 | "metadata": {}, 641 | "outputs": [], 642 | "source": [ 643 | "pd.concat([pd.DataFrame.from_dict(agg_metrics_mqcnn, orient='index').rename(columns={0: \"MQ-CNN\"}),\n", 644 | " pd.DataFrame.from_dict(agg_metrics_tft, orient='index').rename(columns={0: \"TFT\"})],\n", 645 | " axis=1)" 646 | ] 647 | }, 648 | { 649 | "cell_type": "code", 650 | "execution_count": null, 651 | "metadata": {}, 652 | "outputs": [], 653 | "source": [ 654 | "# 'QuantileForecast.plot' plots all the quantiles as line plots.\n", 655 | "# This is some boiler plate code to plot an interval around the median \n", 656 | "# using the 10th and 90th quantile\n", 657 | "def plot_from_quantile_forecast(obs, past_length, lower_bound, upper_bound, forecasts):\n", 658 | " plt.figure(figsize=(12,6))\n", 659 | " plt.plot(obs[0][-forecast_length-past_length:], label='observed')\n", 660 | " plt.plot(obs[0][-forecast_length:].index, \n", 661 | " lower_bound,\n", 662 | " color='g',\n", 663 | " alpha=0.3,\n", 664 | " label='10th quantile')\n", 665 | " plt.plot(obs[0][-forecast_length:].index, \n", 666 | " forecasts,\n", 667 | " color='g',\n", 668 | " label='median prediction')\n", 669 | " plt.plot(obs[0][-forecast_length:].index, \n", 670 | " upper_bound,\n", 671 | " color='g',\n", 672 | " alpha=0.3,\n", 673 | " label='90th quantile')\n", 674 | " plt.fill_between(obs[0][-forecast_length:].index,\n", 675 | " lower_bound, \n", 676 | " upper_bound,\n", 677 | " color='g',\n", 678 | " alpha=0.3)\n", 679 | " plt.ylabel('PM2.5 concentration (ug/m^3)')\n", 680 | " \n", 681 | " plt.legend()\n", 682 | " plt.grid(which=\"both\")\n", 683 | " plt.show()" 684 | ] 685 | }, 686 | { 687 | "cell_type": "markdown", 688 | "metadata": {}, 689 | "source": [ 690 | "The two models illustrated in this section forecast quantiles. Hence to construct an interval, one needs to pick forecasts at different quantiles. The charts below use the 10th and the 90th quantile forecast to construct an interval." 691 | ] 692 | }, 693 | { 694 | "cell_type": "markdown", 695 | "metadata": {}, 696 | "source": [ 697 | "#### Plot Sample Forecast - MQCNN" 698 | ] 699 | }, 700 | { 701 | "cell_type": "code", 702 | "execution_count": null, 703 | "metadata": {}, 704 | "outputs": [], 705 | "source": [ 706 | "plot_from_quantile_forecast(obs_mqcnn, \n", 707 | " 100, \n", 708 | " forecast_mqcnn[0].forecast_array[0], \n", 709 | " forecast_mqcnn[0].forecast_array[8],\n", 710 | " forecast_mqcnn[0].forecast_array[4])" 711 | ] 712 | }, 713 | { 714 | "cell_type": "markdown", 715 | "metadata": {}, 716 | "source": [ 717 | "#### Plot Sample Forecast - Temporal Fusion Transformer (TFT)" 718 | ] 719 | }, 720 | { 721 | "cell_type": "code", 722 | "execution_count": null, 723 | "metadata": {}, 724 | "outputs": [], 725 | "source": [ 726 | "plot_from_quantile_forecast(obs_tft, \n", 727 | " 100, \n", 728 | " forecast_tft[0].forecast_array[1], \n", 729 | " forecast_tft[0].forecast_array[2],\n", 730 | " forecast_tft[0].forecast_array[0])" 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "## 6. Model all the time series as target series\n", 738 | "\n", 739 | "In this case, we forecast pm2.5 and the other meteorological features together as multivariate variables.\n", 740 | "\n", 741 | "Models like [LSTNet](https://ts.gluon.ai/api/gluonts/gluonts.model.lstnet.html) allow one to treat all the related time series in a multivariate fashion. One can train a model to forecast all the time series simultaneously.\n", 742 | "\n", 743 | "For this, the data needs to be prepared in a different way and the below cell does that." 744 | ] 745 | }, 746 | { 747 | "cell_type": "markdown", 748 | "metadata": {}, 749 | "source": [ 750 | "#### LSTNet" 751 | ] 752 | }, 753 | { 754 | "cell_type": "code", 755 | "execution_count": null, 756 | "metadata": { 757 | "jupyter": { 758 | "outputs_hidden": true 759 | } 760 | }, 761 | "outputs": [], 762 | "source": [ 763 | "train = df.transpose()\n", 764 | "train2 = train.to_numpy()\n", 765 | "\n", 766 | "target=train2[[5,6,7,8],:]\n", 767 | "\n", 768 | "#prediction_length=24\n", 769 | "\n", 770 | "start= [df.iloc[0][\"Timestamp\"] for _ in range(4)]\n", 771 | "\n", 772 | "\n", 773 | "train_ds = ListDataset([{FieldName.TARGET: target, \n", 774 | " FieldName.START: start\n", 775 | " } \n", 776 | " for (target, start) in zip(target[:, :-forecast_length], \n", 777 | " start)],\n", 778 | " freq='1h')\n", 779 | "\n", 780 | "test_ds = ListDataset([{FieldName.TARGET: target, \n", 781 | " FieldName.START: start\n", 782 | " }\n", 783 | " for (target, start) in zip(target[:, :],\n", 784 | " start)],\n", 785 | " freq='1h')\n", 786 | "\n", 787 | "\n", 788 | "lstnet_estimator=LSTNetEstimator(freq='1h', \n", 789 | " prediction_length=forecast_length, \n", 790 | " context_length=336, \n", 791 | " num_series=4, \n", 792 | " skip_size=10, \n", 793 | " ar_window=320, \n", 794 | " channels=80, \n", 795 | " trainer = Trainer(epochs=400,\n", 796 | " ctx = mx.context.gpu(0)), \n", 797 | " dropout_rate = 0.4, \n", 798 | " output_activation = 'sigmoid', \n", 799 | " rnn_cell_type = 'gru', \n", 800 | " rnn_num_cells = 100, \n", 801 | " rnn_num_layers = 6, \n", 802 | " skip_rnn_cell_type = 'gru', \n", 803 | " skip_rnn_num_layers = 3, \n", 804 | " skip_rnn_num_cells = 10, \n", 805 | " scaling = True)\n", 806 | "\n", 807 | "\n", 808 | "grouper_train = MultivariateGrouper(max_target_dim=4)\n", 809 | "\n", 810 | "train_ds = grouper_train(train_ds)\n", 811 | "\n", 812 | "lstnet_predictor = lstnet_estimator.train(train_ds)\n", 813 | "\n", 814 | "grouper_test = MultivariateGrouper(max_target_dim=4)\n", 815 | "\n", 816 | "test_ds = grouper_test(test_ds)\n", 817 | "\n", 818 | "forecast_lstnet, obs_lstnet = make_evaluation_predictions(test_ds,\n", 819 | " predictor=lstnet_predictor,\n", 820 | " num_samples=100)\n", 821 | "\n", 822 | "forecast_lstnet = list(forecast_lstnet)\n", 823 | "obs_lstnet = list(obs_lstnet)" 824 | ] 825 | }, 826 | { 827 | "cell_type": "markdown", 828 | "metadata": {}, 829 | "source": [ 830 | "#### Evaluation" 831 | ] 832 | }, 833 | { 834 | "cell_type": "code", 835 | "execution_count": null, 836 | "metadata": {}, 837 | "outputs": [], 838 | "source": [ 839 | "evaluator = MultivariateEvaluator(quantiles=[0.1, 0.5, 0.9])\n", 840 | "agg_metrics_lstnet, item_metrics_lstnet = evaluator(obs_lstnet, forecast_lstnet, num_series=len(test_ds))\n", 841 | "\n", 842 | "index_series_map = {0: \"pm 2.5\",\n", 843 | " 1: \"DEWP\",\n", 844 | " 2: \"TEMP\",\n", 845 | " 3: \"PRES\"}\n", 846 | "metrics_lstnet = []\n", 847 | "for i in range(4):\n", 848 | " metrics = [k for k in agg_metrics_lstnet.keys() if k.startswith(str(i))]\n", 849 | " metrics_lstnet.append(pd.DataFrame.from_dict({m[2:]:agg_metrics_lstnet[m] for m in metrics},\n", 850 | " orient='index').rename(columns={0: index_series_map[i]}))\n", 851 | "pd.concat(metrics_lstnet, axis=1) " 852 | ] 853 | }, 854 | { 855 | "cell_type": "markdown", 856 | "metadata": {}, 857 | "source": [ 858 | "#### Plot Sample Forecast - LSTNet" 859 | ] 860 | }, 861 | { 862 | "cell_type": "markdown", 863 | "metadata": {}, 864 | "source": [ 865 | "The below plots show the forecasts for each of the target time series as defined in the above cell. The results from the LSTNet model that is trained to forecast all the time series simultaneously is not great. One, probably needs to do a more thorough hyperparameter optimization. But this can be a good model to explore when one wants to build a single model to forecast multiple time series." 866 | ] 867 | }, 868 | { 869 | "cell_type": "code", 870 | "execution_count": null, 871 | "metadata": {}, 872 | "outputs": [], 873 | "source": [ 874 | "\n", 875 | "for x, y in zip(obs_lstnet, forecast_lstnet):\n", 876 | " for i in range(4):\n", 877 | " plt.figure(figsize=(12,6))\n", 878 | " plt.plot(x[i][-forecast_length-100:])\n", 879 | " median = y.copy_dim(i).quantile(0.5)\n", 880 | " y_10 = y.copy_dim(i).quantile(0.1)\n", 881 | " y_90 = y.copy_dim(i).quantile(0.9)\n", 882 | " #print(y_10)\n", 883 | " #print(y_90)\n", 884 | " plt.plot(x[i][-forecast_length:].index,\n", 885 | " median,\n", 886 | " color='g')\n", 887 | " plt.fill_between(x[i][-forecast_length:].index,\n", 888 | " y_10,\n", 889 | " y_90,\n", 890 | " color='g',\n", 891 | " alpha=0.3)\n", 892 | " plt.title(index_series_map[i])\n", 893 | " plt.show()" 894 | ] 895 | }, 896 | { 897 | "cell_type": "markdown", 898 | "metadata": {}, 899 | "source": [ 900 | "In conclusion, this notebook illustrates how one can use different deep learning models that are defined in [GluonTS](https://ts.gluon.ai/index.html). The choice of models depends on the nature of the covariates and exogenous variables that are present in the dataset. GluonTS provides a rich variety of recent deep learning based models for modelling time series and this notebook can be used to quickly benchmark some of them so that one can shortlist a couple for further experimentation and fine-tuning." 901 | ] 902 | } 903 | ], 904 | "metadata": { 905 | "kernelspec": { 906 | "display_name": "Python 3", 907 | "language": "python", 908 | "name": "python3" 909 | }, 910 | "language_info": { 911 | "codemirror_mode": { 912 | "name": "ipython", 913 | "version": 3 914 | }, 915 | "file_extension": ".py", 916 | "mimetype": "text/x-python", 917 | "name": "python", 918 | "nbconvert_exporter": "python", 919 | "pygments_lexer": "ipython3", 920 | "version": "3.8.3" 921 | } 922 | }, 923 | "nbformat": 4, 924 | "nbformat_minor": 5 925 | } 926 | -------------------------------------------------------------------------------- /images/arima.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts/292f0b29f616b370581ea5d79e9cb7243e3a3bdb/images/arima.png -------------------------------------------------------------------------------- /images/deepar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts/292f0b29f616b370581ea5d79e9cb7243e3a3bdb/images/deepar.png -------------------------------------------------------------------------------- /images/mean.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts/292f0b29f616b370581ea5d79e9cb7243e3a3bdb/images/mean.png -------------------------------------------------------------------------------- /images/prophet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts/292f0b29f616b370581ea5d79e9cb7243e3a3bdb/images/prophet.png -------------------------------------------------------------------------------- /images/seasonal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-forecast-algorithms-benchmark-using-gluonts/292f0b29f616b370581ea5d79e9cb7243e3a3bdb/images/seasonal.png --------------------------------------------------------------------------------