├── .ipynb_checkpoints ├── LRP Implementation-checkpoint.ipynb ├── LRP old working-checkpoint.ipynb ├── SHAP & LIME (Issues & Observations)-checkpoint.ipynb ├── SHAP & LIME Implementation dummy-checkpoint.ipynb └── SHAP & LIME Implementation-checkpoint.ipynb ├── README.md ├── images ├── LSTM Models.png ├── bidi.PNG ├── encdects_16_ts.PNG ├── encdects_1_ts.PNG ├── stackedlstmts_16_ts.PNG └── vanilla lstmts_1_ts.PNG ├── models ├── bidi_model.h5 ├── enc_dec_model_ts1.h5 ├── enc_dec_model_ts16.h5 ├── stacked_lstm_model_ts16_1.h5 ├── stacked_lstm_model_ts16_2.h5 └── vanilla_lstm_model_ts1.h5 ├── notebooks ├── .ipynb_checkpoints │ ├── LRP Implementation-checkpoint.ipynb │ └── SHAP & LIME Implementation-checkpoint.ipynb ├── LIME_and_SHAP_Implementation.ipynb └── LRP_Implementation.ipynb └── result plots ├── Bar plot - LRP relevance score for 1 instance.PNG ├── LRP relevance score for all instances.PNG ├── comp table new.PNG ├── shap_bidi_lstm.PNG ├── shap_enc_dec_ts1.png ├── shap_stacked_ lstm_ts16_1.PNG └── shap_vanilla_ts1.png /.ipynb_checkpoints/SHAP & LIME (Issues & Observations)-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "### Jithin Sasikumar\n", 8 | "\n", 9 | "This notebook addresses the major issues when implementing **feature attribution methods (SHAP and LIME)** for LSTM time-series model. It contains data pre-processing, model prediction, SHAP implementation for our model along with the issues and my observations. I have also added the comparison of our model with two other models trained on different datasets for different problems. " 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "import pandas as pd\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "import shap\n", 22 | "import tensorflow as tf" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "## Loading train and test data for pre-processing\n", 30 | "\n", 31 | "1. The data has **25 features**, each representing sensor information from a specific channel of the respective satellite. \n", 32 | "2. The data is one-hot encoded." 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 3, 38 | "metadata": {}, 39 | "outputs": [ 40 | { 41 | "name": "stdout", 42 | "output_type": "stream", 43 | "text": [ 44 | "(2880, 25)\n" 45 | ] 46 | }, 47 | { 48 | "data": { 49 | "text/html": [ 50 | "
\n", 51 | "\n", 64 | "\n", 65 | " \n", 66 | " \n", 67 | " \n", 68 | " \n", 69 | " \n", 70 | " \n", 71 | " \n", 72 | " \n", 73 | " \n", 74 | " \n", 75 | " \n", 76 | " \n", 77 | " \n", 78 | " \n", 79 | " \n", 80 | " \n", 81 | " \n", 82 | " \n", 83 | " \n", 84 | " \n", 85 | " \n", 86 | " \n", 87 | " \n", 88 | " \n", 89 | " \n", 90 | " \n", 91 | " \n", 92 | " \n", 93 | " \n", 94 | " \n", 95 | " \n", 96 | " \n", 97 | " \n", 98 | " \n", 99 | " \n", 100 | " \n", 101 | " \n", 102 | " \n", 103 | " \n", 104 | " \n", 105 | " \n", 106 | " \n", 107 | " \n", 108 | " \n", 109 | " \n", 110 | " \n", 111 | " \n", 112 | " \n", 113 | " \n", 114 | " \n", 115 | " \n", 116 | " \n", 117 | " \n", 118 | " \n", 119 | " \n", 120 | " \n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | " \n", 133 | " \n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | " \n", 204 | " \n", 205 | " \n", 206 | " \n", 207 | " \n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | " \n", 222 | " \n", 223 | " \n", 224 | " \n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | "
0123456789...15161718192021222324
00.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
10.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
20.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
30.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
40.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
..................................................................
28750.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
28760.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
28770.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
28780.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
28790.9990.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
\n", 358 | "

2880 rows × 25 columns

\n", 359 | "
" 360 | ], 361 | "text/plain": [ 362 | " 0 1 2 3 4 5 6 7 8 9 ... 15 16 17 \\\n", 363 | "0 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 364 | "1 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 365 | "2 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 366 | "3 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 367 | "4 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 368 | "... ... ... ... ... ... ... ... ... ... ... ... ... ... ... \n", 369 | "2875 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 370 | "2876 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 371 | "2877 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 372 | "2878 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 373 | "2879 0.999 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 374 | "\n", 375 | " 18 19 20 21 22 23 24 \n", 376 | "0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 377 | "1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 378 | "2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 379 | "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 380 | "4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 381 | "... ... ... ... ... ... ... ... \n", 382 | "2875 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 383 | "2876 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 384 | "2877 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 385 | "2878 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 386 | "2879 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 387 | "\n", 388 | "[2880 rows x 25 columns]" 389 | ] 390 | }, 391 | "execution_count": 3, 392 | "metadata": {}, 393 | "output_type": "execute_result" 394 | } 395 | ], 396 | "source": [ 397 | "#Train data\n", 398 | "train_arr = np.load('D:/DLR/telemanom/data/train/A-1.npy')\n", 399 | "data_train = pd.DataFrame(train_arr)\n", 400 | "print(data_train.shape)\n", 401 | "\n", 402 | "data_train" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 4, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "name": "stdout", 412 | "output_type": "stream", 413 | "text": [ 414 | "(8640, 25)\n" 415 | ] 416 | }, 417 | { 418 | "data": { 419 | "text/html": [ 420 | "
\n", 421 | "\n", 434 | "\n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | " \n", 448 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 452 | " \n", 453 | " \n", 454 | " \n", 455 | " \n", 456 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 460 | " \n", 461 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 497 | " \n", 498 | " \n", 499 | " \n", 500 | " \n", 501 | " \n", 502 | " \n", 503 | " \n", 504 | " \n", 505 | " \n", 506 | " \n", 507 | " \n", 508 | " \n", 509 | " \n", 510 | " \n", 511 | " \n", 512 | " \n", 513 | " \n", 514 | " \n", 515 | " \n", 516 | " \n", 517 | " \n", 518 | " \n", 519 | " \n", 520 | " \n", 521 | " \n", 522 | " \n", 523 | " \n", 524 | " \n", 525 | " \n", 526 | " \n", 527 | " \n", 528 | " \n", 529 | " \n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | " \n", 534 | " \n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 542 | " \n", 543 | " \n", 544 | " \n", 545 | " \n", 546 | " \n", 547 | " \n", 548 | " \n", 549 | " \n", 550 | " \n", 551 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | " \n", 717 | " \n", 718 | " \n", 719 | " \n", 720 | " \n", 721 | " \n", 722 | " \n", 723 | " \n", 724 | " \n", 725 | " \n", 726 | " \n", 727 | "
0123456789...15161718192021222324
01.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
11.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
21.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
31.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
41.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.01.01.00.00.0
..................................................................
86351.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
86361.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
86371.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
86381.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
86391.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
\n", 728 | "

8640 rows × 25 columns

\n", 729 | "
" 730 | ], 731 | "text/plain": [ 732 | " 0 1 2 3 4 5 6 7 8 9 ... 15 16 17 \\\n", 733 | "0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 734 | "1 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 735 | "2 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 736 | "3 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 737 | "4 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 738 | "... ... ... ... ... ... ... ... ... ... ... ... ... ... ... \n", 739 | "8635 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 740 | "8636 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 741 | "8637 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 742 | "8638 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 743 | "8639 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 \n", 744 | "\n", 745 | " 18 19 20 21 22 23 24 \n", 746 | "0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 747 | "1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 748 | "2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 749 | "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 750 | "4 0.0 0.0 0.0 1.0 1.0 0.0 0.0 \n", 751 | "... ... ... ... ... ... ... ... \n", 752 | "8635 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 753 | "8636 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 754 | "8637 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 755 | "8638 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 756 | "8639 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", 757 | "\n", 758 | "[8640 rows x 25 columns]" 759 | ] 760 | }, 761 | "execution_count": 4, 762 | "metadata": {}, 763 | "output_type": "execute_result" 764 | } 765 | ], 766 | "source": [ 767 | "#Test data\n", 768 | "test_arr = np.load('D:/DLR/telemanom/data/test/A-1.npy')\n", 769 | "test = pd.DataFrame(test_arr)\n", 770 | "print(test.shape)\n", 771 | "\n", 772 | "test" 773 | ] 774 | }, 775 | { 776 | "cell_type": "code", 777 | "execution_count": 34, 778 | "metadata": {}, 779 | "outputs": [ 780 | { 781 | "data": { 782 | "text/html": [ 783 | "
\n", 784 | "\n", 797 | "\n", 798 | " \n", 799 | " \n", 800 | " \n", 801 | " \n", 802 | " \n", 803 | " \n", 804 | " \n", 805 | " \n", 806 | " \n", 807 | " \n", 808 | " \n", 809 | " \n", 810 | " \n", 811 | " \n", 812 | " \n", 813 | " \n", 814 | " \n", 815 | " \n", 816 | " \n", 817 | " \n", 818 | " \n", 819 | " \n", 820 | " \n", 821 | " \n", 822 | " \n", 823 | " \n", 824 | " \n", 825 | " \n", 826 | " \n", 827 | " \n", 828 | " \n", 829 | " \n", 830 | " \n", 831 | " \n", 832 | " \n", 833 | " \n", 834 | " \n", 835 | " \n", 836 | " \n", 837 | " \n", 838 | " \n", 839 | " \n", 840 | " \n", 841 | " \n", 842 | " \n", 843 | " \n", 844 | " \n", 845 | " \n", 846 | " \n", 847 | " \n", 848 | " \n", 849 | " \n", 850 | "
0
00.976049
10.976191
20.976298
30.976378
40.976436
......
83750.976578
83760.976569
83770.976559
83780.976551
83790.976543
\n", 851 | "

8380 rows × 1 columns

\n", 852 | "
" 853 | ], 854 | "text/plain": [ 855 | " 0\n", 856 | "0 0.976049\n", 857 | "1 0.976191\n", 858 | "2 0.976298\n", 859 | "3 0.976378\n", 860 | "4 0.976436\n", 861 | "... ...\n", 862 | "8375 0.976578\n", 863 | "8376 0.976569\n", 864 | "8377 0.976559\n", 865 | "8378 0.976551\n", 866 | "8379 0.976543\n", 867 | "\n", 868 | "[8380 rows x 1 columns]" 869 | ] 870 | }, 871 | "execution_count": 34, 872 | "metadata": {}, 873 | "output_type": "execute_result" 874 | } 875 | ], 876 | "source": [ 877 | "y_hat = np.load('D:/DLR/telemanom/data/2018-05-19_15.00.10/y_hat/A-1.npy')\n", 878 | "y_hat_df = pd.DataFrame(y_hat)\n", 879 | "y_hat_df" 880 | ] 881 | }, 882 | { 883 | "cell_type": "markdown", 884 | "metadata": {}, 885 | "source": [ 886 | "## Loading pre-trained model (NASA)" 887 | ] 888 | }, 889 | { 890 | "cell_type": "code", 891 | "execution_count": 6, 892 | "metadata": {}, 893 | "outputs": [ 894 | { 895 | "name": "stdout", 896 | "output_type": "stream", 897 | "text": [ 898 | "2.0.0\n" 899 | ] 900 | } 901 | ], 902 | "source": [ 903 | "print(tf.__version__)\n", 904 | "model = tf.keras.models.load_model('D:/DLR/telemanom/data/2018-05-19_15.00.10/models/A-1.h5')" 905 | ] 906 | }, 907 | { 908 | "cell_type": "code", 909 | "execution_count": 8, 910 | "metadata": {}, 911 | "outputs": [ 912 | { 913 | "name": "stdout", 914 | "output_type": "stream", 915 | "text": [ 916 | "\n", 917 | "Model: \"sequential\"\n", 918 | "_________________________________________________________________\n", 919 | "Layer (type) Output Shape Param # \n", 920 | "=================================================================\n", 921 | "lstm_5 (LSTM) (None, None, 80) 33920 \n", 922 | "_________________________________________________________________\n", 923 | "dropout_5 (Dropout) (None, None, 80) 0 \n", 924 | "_________________________________________________________________\n", 925 | "lstm_6 (LSTM) (None, 80) 51520 \n", 926 | "_________________________________________________________________\n", 927 | "dropout_6 (Dropout) (None, 80) 0 \n", 928 | "_________________________________________________________________\n", 929 | "dense_3 (Dense) (None, 10) 810 \n", 930 | "_________________________________________________________________\n", 931 | "activation_3 (Activation) (None, 10) 0 \n", 932 | "=================================================================\n", 933 | "Total params: 86,250\n", 934 | "Trainable params: 86,250\n", 935 | "Non-trainable params: 0\n", 936 | "_________________________________________________________________\n" 937 | ] 938 | } 939 | ], 940 | "source": [ 941 | "print(model)\n", 942 | "#Displaying model configurations and properities\n", 943 | "model.summary()" 944 | ] 945 | }, 946 | { 947 | "cell_type": "markdown", 948 | "metadata": {}, 949 | "source": [ 950 | "## Data Pre-processing\n", 951 | "1. The data is pre-processed with respect to its time steps as batches. As LSTM accepts only 3D data as input, the actual data is transformed into three dimensional array in the format **(data_samples, n_timesteps, n_features). => (2620, 250, 25)**
\n", 952 | "3. During pre-processing, in order to restructure the data to represent like a supervised learning problem, previous time steps are used as input variables and the next time step is used as output variable which is called **sliding window technique for time-series**
\n", 953 | "4. Here, the number of previous timesteps is **250** as per config.yaml. Thus our data will be trained and model makes the predictions by processing each 250 timesteps data with 25 features, **ten times**. \n", 954 | "5. Thus the number of predictions will be 10. The output shape will be **(2620, 10) = > (data_samples, n_predictions).**\n", 955 | "\n", 956 | "
\n", 957 | "The same is demonstrated in the below code.\n", 958 | " " 959 | ] 960 | }, 961 | { 962 | "cell_type": "code", 963 | "execution_count": 9, 964 | "metadata": {}, 965 | "outputs": [ 966 | { 967 | "name": "stdout", 968 | "output_type": "stream", 969 | "text": [ 970 | "X: (2620, 250, 25)\n", 971 | "Y: (2620, 10)\n" 972 | ] 973 | } 974 | ], 975 | "source": [ 976 | "#train data\n", 977 | "train=False\n", 978 | "data = []\n", 979 | "for i in range(len(data_train) - 250 - 10):\n", 980 | " data.append(data_train[i:i + 250 + 10])\n", 981 | "data = np.array(data)\n", 982 | "\n", 983 | "assert len(data.shape) == 3\n", 984 | "\n", 985 | "if not train:\n", 986 | " X_train = data[:, :-10, :]\n", 987 | " y_train = data[:, -10:, 0]\n", 988 | "\n", 989 | "print(\"X:\",X_train.shape)\n", 990 | "print(\"Y:\",y_train.shape)" 991 | ] 992 | }, 993 | { 994 | "cell_type": "code", 995 | "execution_count": 15, 996 | "metadata": {}, 997 | "outputs": [], 998 | "source": [ 999 | "#For test data\n", 1000 | "train=False\n", 1001 | "data = []\n", 1002 | "for i in range(len(test_arr) - 250 - 10):\n", 1003 | " data.append(test_arr[i:i + 250 + 10])\n", 1004 | "data = np.array(data)\n", 1005 | "\n", 1006 | "assert len(data.shape) == 3\n", 1007 | "\n", 1008 | "if not train:\n", 1009 | " X_test = data[:, :-10, :]\n", 1010 | " y_test = data[:, -10:, 0]\n", 1011 | "\n" 1012 | ] 1013 | }, 1014 | { 1015 | "cell_type": "markdown", 1016 | "metadata": {}, 1017 | "source": [ 1018 | "## Model Prediction\n", 1019 | "1. As explained in the above cell, the model makes predictions with our pre-processed test data.\n", 1020 | "2. Before predicting the value, actual data is converted into **batches** and each batch is given into model seperately where the model uses previous batch information for current batch processsing and then predicts the output value.
\n", 1021 | "3. Thus each batch contains **260 data samples, 250 timesteps and 25 features** => **(260, 250, 25)**.\n", 1022 | "4. Our model outputs 10 predictions for 260 data samples as => **(260,10) predictions (y_hat)** for each batch given." 1023 | ] 1024 | }, 1025 | { 1026 | "cell_type": "code", 1027 | "execution_count": 12, 1028 | "metadata": {}, 1029 | "outputs": [], 1030 | "source": [ 1031 | "num_batches = int((y_test.shape[0] - 250)/ 70)\n", 1032 | "for i in range(0, num_batches + 1):\n", 1033 | " prior_idx = i * 70\n", 1034 | " idx = (i + 1) * 70\n", 1035 | " \n", 1036 | " if i + 1 == num_batches + 1:\n", 1037 | " idx = y_test.shape[0]\n", 1038 | " \n", 1039 | " X_test_batch = X_test[prior_idx:idx]\n", 1040 | " y_test_batch = y_test[prior_idx:idx]\n", 1041 | " y_hat_batch = model.predict(X_test_batch)" 1042 | ] 1043 | }, 1044 | { 1045 | "cell_type": "code", 1046 | "execution_count": 14, 1047 | "metadata": {}, 1048 | "outputs": [ 1049 | { 1050 | "name": "stdout", 1051 | "output_type": "stream", 1052 | "text": [ 1053 | "X batch in test data: (260, 250, 25)\n", 1054 | "Y batch in test data: (260, 10)\n", 1055 | "y_hat predictions from test data: (260, 10)\n" 1056 | ] 1057 | } 1058 | ], 1059 | "source": [ 1060 | "print(\"X batch in test data:\",X_test_batch.shape)\n", 1061 | "print(\"Y batch in test data:\",y_test_batch.shape)\n", 1062 | "print(\"y_hat predictions from test data:\",y_hat_batch.shape)" 1063 | ] 1064 | }, 1065 | { 1066 | "cell_type": "markdown", 1067 | "metadata": {}, 1068 | "source": [ 1069 | "## SHAP Implementation\n", 1070 | "\n", 1071 | "**Reference:**
\n", 1072 | "https://arxiv.org/pdf/1903.02407.pdf
\n", 1073 | "https://github.com/liuyilin950623/SHAP_on_Autoencoder" 1074 | ] 1075 | }, 1076 | { 1077 | "cell_type": "code", 1078 | "execution_count": 16, 1079 | "metadata": {}, 1080 | "outputs": [ 1081 | { 1082 | "data": { 1083 | "text/html": [ 1084 | "
\n", 1085 | "\n", 1098 | "\n", 1099 | " \n", 1100 | " \n", 1101 | " \n", 1102 | " \n", 1103 | " \n", 1104 | " \n", 1105 | " \n", 1106 | " \n", 1107 | " \n", 1108 | " \n", 1109 | " \n", 1110 | " \n", 1111 | " \n", 1112 | " \n", 1113 | " \n", 1114 | " \n", 1115 | " \n", 1116 | " \n", 1117 | " \n", 1118 | " \n", 1119 | " \n", 1120 | " \n", 1121 | " \n", 1122 | " \n", 1123 | " \n", 1124 | " \n", 1125 | " \n", 1126 | " \n", 1127 | " \n", 1128 | " \n", 1129 | "
0123456789
reconstruction_loss0.966080.9674890.9562030.9648190.9605380.9666760.9520930.966080.9663040.968249
\n", 1130 | "
" 1131 | ], 1132 | "text/plain": [ 1133 | " 0 1 2 3 4 \\\n", 1134 | "reconstruction_loss 0.96608 0.967489 0.956203 0.964819 0.960538 \n", 1135 | "\n", 1136 | " 5 6 7 8 9 \n", 1137 | "reconstruction_loss 0.966676 0.952093 0.96608 0.966304 0.968249 " 1138 | ] 1139 | }, 1140 | "execution_count": 16, 1141 | "metadata": {}, 1142 | "output_type": "execute_result" 1143 | } 1144 | ], 1145 | "source": [ 1146 | "#Picking a data point with largest reconstruction or prediction error\n", 1147 | "\n", 1148 | "X_reconstruction_standard = model.predict(X_test_batch)\n", 1149 | "rec_err = np.linalg.norm(y_test_batch - X_reconstruction_standard, axis = 1)\n", 1150 | "idx = list(rec_err).index(max(rec_err))\n", 1151 | "df = pd.DataFrame(data = X_reconstruction_standard[idx], columns = ['reconstruction_loss'])\n", 1152 | "df.T" 1153 | ] 1154 | }, 1155 | { 1156 | "cell_type": "code", 1157 | "execution_count": 17, 1158 | "metadata": {}, 1159 | "outputs": [ 1160 | { 1161 | "name": "stdout", 1162 | "output_type": "stream", 1163 | "text": [ 1164 | "Model Prediction output shape for test batch: (260, 10)\n", 1165 | "\n", 1166 | "Actual Value y:\n", 1167 | " [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", 1168 | "Predicted Value y_hat:\n", 1169 | " [0.9739516 0.9887368 0.9738245 0.99259615 0.9800745 0.98507845\n", 1170 | " 0.9748681 0.99089825 0.9805185 0.9821451 ]\n", 1171 | "\n", 1172 | "First five predictions:\n", 1173 | " [[0.9733607 0.9879205 0.9732397 0.9919983 0.9796158 0.9846846\n", 1174 | " 0.9742988 0.990171 0.9798947 0.9817468 ]\n", 1175 | " [0.9739516 0.9887368 0.9738245 0.99259615 0.9800745 0.98507845\n", 1176 | " 0.9748681 0.99089825 0.9805185 0.9821451 ]\n", 1177 | " [0.97446465 0.9894272 0.97432685 0.99309766 0.98047465 0.9854017\n", 1178 | " 0.9753439 0.99151456 0.98105055 0.9824883 ]\n", 1179 | " [0.9749013 0.9900011 0.9747505 0.9935121 0.9808161 0.9856611\n", 1180 | " 0.9757351 0.99203014 0.981495 0.9827764 ]\n", 1181 | " [0.97526526 0.99046975 0.9751019 0.9938499 0.9811022 0.9858655\n", 1182 | " 0.97605145 0.9924544 0.9818594 0.9830123 ]]\n", 1183 | "\n", 1184 | "First 5 Prediction (or) Re-construction error:\n", 1185 | " [0.06148452 0.0598148 0.05840042 0.05722382 0.05626203]\n" 1186 | ] 1187 | } 1188 | ], 1189 | "source": [ 1190 | "print(\"Model Prediction output shape for test batch:\",X_reconstruction_standard.shape)\n", 1191 | "print(\"\\nActual Value y:\\n\",y_test_batch[1])\n", 1192 | "print(\"Predicted Value y_hat:\\n\",X_reconstruction_standard[1])\n", 1193 | "print(\"\\nFirst five predictions:\\n\",X_reconstruction_standard[0:5])\n", 1194 | "print(\"\\nFirst 5 Prediction (or) Re-construction error:\\n\",rec_err[0:5])\n" 1195 | ] 1196 | }, 1197 | { 1198 | "cell_type": "code", 1199 | "execution_count": 18, 1200 | "metadata": {}, 1201 | "outputs": [ 1202 | { 1203 | "data": { 1204 | "text/html": [ 1205 | "
\n", 1206 | "\n", 1219 | "\n", 1220 | " \n", 1221 | " \n", 1222 | " \n", 1223 | " \n", 1224 | " \n", 1225 | " \n", 1226 | " \n", 1227 | " \n", 1228 | " \n", 1229 | " \n", 1230 | " \n", 1231 | " \n", 1232 | " \n", 1233 | " \n", 1234 | " \n", 1235 | " \n", 1236 | " \n", 1237 | " \n", 1238 | " \n", 1239 | " \n", 1240 | " \n", 1241 | " \n", 1242 | " \n", 1243 | " \n", 1244 | " \n", 1245 | " \n", 1246 | " \n", 1247 | " \n", 1248 | " \n", 1249 | " \n", 1250 | "
9158703426
reconstruction_loss0.9682490.9674890.9666760.9663040.966080.966080.9648190.9605380.9562030.952093
\n", 1251 | "
" 1252 | ], 1253 | "text/plain": [ 1254 | " 9 1 5 8 7 0 \\\n", 1255 | "reconstruction_loss 0.968249 0.967489 0.966676 0.966304 0.96608 0.96608 \n", 1256 | "\n", 1257 | " 3 4 2 6 \n", 1258 | "reconstruction_loss 0.964819 0.960538 0.956203 0.952093 " 1259 | ] 1260 | }, 1261 | "execution_count": 18, 1262 | "metadata": {}, 1263 | "output_type": "execute_result" 1264 | } 1265 | ], 1266 | "source": [ 1267 | "# Selecting top 5 features from the error list\n", 1268 | "def sort_by_absolute(df, index):\n", 1269 | " df_abs = df.apply(lambda x: abs(x))\n", 1270 | " df_abs = df_abs.sort_values('reconstruction_loss', ascending = False)\n", 1271 | " df = df.loc[df_abs.index,:]\n", 1272 | " return df\n", 1273 | "sort_by_absolute(df, idx).T" 1274 | ] 1275 | }, 1276 | { 1277 | "cell_type": "code", 1278 | "execution_count": 19, 1279 | "metadata": {}, 1280 | "outputs": [ 1281 | { 1282 | "data": { 1283 | "text/html": [ 1284 | "
\n", 1285 | "\n", 1298 | "\n", 1299 | " \n", 1300 | " \n", 1301 | " \n", 1302 | " \n", 1303 | " \n", 1304 | " \n", 1305 | " \n", 1306 | " \n", 1307 | " \n", 1308 | " \n", 1309 | " \n", 1310 | " \n", 1311 | " \n", 1312 | " \n", 1313 | " \n", 1314 | " \n", 1315 | " \n", 1316 | " \n", 1317 | " \n", 1318 | " \n", 1319 | "
91587
reconstruction_loss0.9682490.9674890.9666760.9663040.96608
\n", 1320 | "
" 1321 | ], 1322 | "text/plain": [ 1323 | " 9 1 5 8 7\n", 1324 | "reconstruction_loss 0.968249 0.967489 0.966676 0.966304 0.96608" 1325 | ] 1326 | }, 1327 | "execution_count": 19, 1328 | "metadata": {}, 1329 | "output_type": "execute_result" 1330 | } 1331 | ], 1332 | "source": [ 1333 | "top_5_features = sort_by_absolute(df, idx).iloc[:5,:]\n", 1334 | "top_5_features.T" 1335 | ] 1336 | }, 1337 | { 1338 | "cell_type": "code", 1339 | "execution_count": 20, 1340 | "metadata": {}, 1341 | "outputs": [ 1342 | { 1343 | "name": "stderr", 1344 | "output_type": "stream", 1345 | "text": [ 1346 | "Using 260 background data samples could cause slower run times. Consider using shap.sample(data, K) or shap.kmeans(data, K) to summarize the background as K samples.\n" 1347 | ] 1348 | }, 1349 | { 1350 | "data": { 1351 | "application/vnd.jupyter.widget-view+json": { 1352 | "model_id": "39122d36bbf5412bbff93857bd0135d0", 1353 | "version_major": 2, 1354 | "version_minor": 0 1355 | }, 1356 | "text/plain": [ 1357 | "HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=250.0), HTML(value='')))" 1358 | ] 1359 | }, 1360 | "metadata": {}, 1361 | "output_type": "display_data" 1362 | }, 1363 | { 1364 | "name": "stdout", 1365 | "output_type": "stream", 1366 | "text": [ 1367 | "\n" 1368 | ] 1369 | }, 1370 | { 1371 | "ename": "IndexError", 1372 | "evalue": "index 25 is out of bounds for axis 1 with size 25", 1373 | "output_type": "error", 1374 | "traceback": [ 1375 | "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", 1376 | "\u001b[1;31mIndexError\u001b[0m Traceback (most recent call last)", 1377 | "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mtop_5_features\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mexplainer_autoencoder\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mshap\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKernelExplainer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX_test_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mshap_values\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mexplainer_autoencoder\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshap_values\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_test_batch\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0midx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m", 1378 | "\u001b[1;32mC:\\ProgramData\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\shap\\explainers\\_kernel.py\u001b[0m in \u001b[0;36mshap_values\u001b[1;34m(self, X, **kwargs)\u001b[0m\n\u001b[0;32m 176\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkeep_index\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 177\u001b[0m \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mconvert_to_instance_with_index\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcolumn_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindex_value\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mi\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindex_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 178\u001b[1;33m \u001b[0mexplanations\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexplain\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 179\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 180\u001b[0m \u001b[1;31m# vector-output\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", 1379 | "\u001b[1;32mC:\\ProgramData\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\shap\\explainers\\_kernel.py\u001b[0m in \u001b[0;36mexplain\u001b[1;34m(self, incoming_instance, **kwargs)\u001b[0m\n\u001b[0;32m 197\u001b[0m \u001b[1;31m# convert incoming input to a standardized iml object\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 198\u001b[0m \u001b[0minstance\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mconvert_to_instance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mincoming_instance\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 199\u001b[1;33m \u001b[0mmatch_instance_to_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 200\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 201\u001b[0m \u001b[1;31m# find the feature groups we will test. If a feature does not change from its\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", 1380 | "\u001b[1;32mC:\\ProgramData\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\shap\\utils\\_legacy.py\u001b[0m in \u001b[0;36mmatch_instance_to_data\u001b[1;34m(instance, data)\u001b[0m\n\u001b[0;32m 85\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mDenseData\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 86\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 87\u001b[1;33m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;34m\"\"\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mgroup\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 88\u001b[0m \u001b[1;32massert\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 89\u001b[0m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", 1381 | "\u001b[1;32mC:\\ProgramData\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\shap\\utils\\_legacy.py\u001b[0m in \u001b[0;36m\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m 85\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mDenseData\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 86\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 87\u001b[1;33m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;34m\"\"\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mgroup\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 88\u001b[0m \u001b[1;32massert\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroup_display_values\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 89\u001b[0m \u001b[0minstance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", 1382 | "\u001b[1;31mIndexError\u001b[0m: index 25 is out of bounds for axis 1 with size 25" 1383 | ] 1384 | } 1385 | ], 1386 | "source": [ 1387 | "# using SHAP\n", 1388 | "for i in top_5_features.index:\n", 1389 | " explainer_autoencoder = shap.KernelExplainer(model.predict, X_test_batch)\n", 1390 | " shap_values = explainer_autoencoder.shap_values(X_test_batch[idx,:,:])" 1391 | ] 1392 | }, 1393 | { 1394 | "cell_type": "code", 1395 | "execution_count": 22, 1396 | "metadata": {}, 1397 | "outputs": [ 1398 | { 1399 | "name": "stdout", 1400 | "output_type": "stream", 1401 | "text": [ 1402 | "Model: \"sequential\"\n", 1403 | "_________________________________________________________________\n", 1404 | "Layer (type) Output Shape Param # \n", 1405 | "=================================================================\n", 1406 | "lstm_5 (LSTM) (None, None, 80) 33920 \n", 1407 | "_________________________________________________________________\n", 1408 | "dropout_5 (Dropout) (None, None, 80) 0 \n", 1409 | "_________________________________________________________________\n", 1410 | "lstm_6 (LSTM) (None, 80) 51520 \n", 1411 | "_________________________________________________________________\n", 1412 | "dropout_6 (Dropout) (None, 80) 0 \n", 1413 | "_________________________________________________________________\n", 1414 | "dense_3 (Dense) (None, 10) 810 \n", 1415 | "_________________________________________________________________\n", 1416 | "activation_3 (Activation) (None, 10) 0 \n", 1417 | "=================================================================\n", 1418 | "Total params: 86,250\n", 1419 | "Trainable params: 86,250\n", 1420 | "Non-trainable params: 0\n", 1421 | "_________________________________________________________________\n", 1422 | "\n", 1423 | "X test batch- (260, 250, 25)\n", 1424 | "\n", 1425 | "y test batch (260, 10)\n", 1426 | "\n", 1427 | "y hat batch (260, 10)\n" 1428 | ] 1429 | } 1430 | ], 1431 | "source": [ 1432 | "model.summary()\n", 1433 | "\n", 1434 | "print(\"\\nX test batch-\",X_test_batch.shape)\n", 1435 | "\n", 1436 | "print(\"\\ny test batch\",y_test_batch.shape)\n", 1437 | "\n", 1438 | "print(\"\\ny hat batch\",y_hat_batch.shape)" 1439 | ] 1440 | }, 1441 | { 1442 | "cell_type": "markdown", 1443 | "metadata": {}, 1444 | "source": [ 1445 | "### Issues with Feature Attribution Methods (SHAP & LIME):\n", 1446 | "\n", 1447 | "**My Observations:**\n", 1448 | "\n", 1449 | "1. As our LSTM network is designed to predict the values in 10 predictions for entire batches, it's input layer accepts data of shape (260, 250, 25) and output layer outputs predictions in shape (260, 10).
\n", 1450 | "2. Here, those 10 predictions are called **lag observations (or) lag features.** as we use previous timestep information.
\n", 1451 | "3. When we apply our model and data to any feature attribution methods like **SHAP, LIME** etc. they expect the features to be the same, only then these methods can assign importance scores to every feature and return the most contributing features for the prediction [In our case - features that contributed for anomaly]. \n", 1452 | "4. Thus, the ground rule is input and output features should be equal (i.e.) If input layer has **25 features**, then the output layer also should have 25 features. But there is no actual issue with our model as it uses 25 features to make predictions and the output **10 predictions** are just lag features and not actual features which creates a conflict.
\n", 1453 | "5. Thus SHAP, LIME assumes the features to be inconsistent as they don't match and throws the above error.
\n", 1454 | "6. To illustrate in simpler terms, the input shape of the network should be equal to it's output shape.
\n", 1455 | "\n", 1456 | "The above error applies to LIME as well." 1457 | ] 1458 | }, 1459 | { 1460 | "cell_type": "markdown", 1461 | "metadata": {}, 1462 | "source": [ 1463 | "## Comparison\n", 1464 | "To justify the observations, I have compared my implementaion with some other implementations as below." 1465 | ] 1466 | }, 1467 | { 1468 | "cell_type": "markdown", 1469 | "metadata": {}, 1470 | "source": [ 1471 | "### Our dataset (NASA)" 1472 | ] 1473 | }, 1474 | { 1475 | "cell_type": "code", 1476 | "execution_count": 24, 1477 | "metadata": {}, 1478 | "outputs": [ 1479 | { 1480 | "name": "stdout", 1481 | "output_type": "stream", 1482 | "text": [ 1483 | "X train - (2620, 250, 25)\n", 1484 | "y train - (2620, 10)\n", 1485 | "\n", 1486 | "X test batch- (260, 250, 25)\n", 1487 | "\n", 1488 | "y test batch (260, 10)\n", 1489 | "\n", 1490 | "y hat batch (260, 10)\n" 1491 | ] 1492 | } 1493 | ], 1494 | "source": [ 1495 | "print(\"X train - \",X_train.shape)\n", 1496 | "print(\"y train -\",y_train.shape)\n", 1497 | "\n", 1498 | "print(\"\\nX test batch-\",X_test_batch.shape)\n", 1499 | "\n", 1500 | "print(\"\\ny test batch\",y_test_batch.shape)\n", 1501 | "\n", 1502 | "print(\"\\ny hat batch\",y_hat_batch.shape)\n" 1503 | ] 1504 | }, 1505 | { 1506 | "cell_type": "markdown", 1507 | "metadata": {}, 1508 | "source": [ 1509 | "### 1. SHAP implementation using Auto encoders on generic numerical dataset\n", 1510 | "\n", 1511 | "1. The dataset used is **Boston Housing dataset**. It is not a time series dataset but the problem is anomaly detection using an auto encoder.
\n", 1512 | "2. For comparison, I have printed the dataset's shape along with their model summary.
\n", 1513 | "3. It has **13 features**.
\n", 1514 | "4. As it is a normal auto encoder, the input data is two dimensional. \n", 1515 | "\n", 1516 | "**Reference**
\n", 1517 | "https://github.com/liuyilin950623/SHAP_on_Autoencoder" 1518 | ] 1519 | }, 1520 | { 1521 | "cell_type": "code", 1522 | "execution_count": 31, 1523 | "metadata": {}, 1524 | "outputs": [ 1525 | { 1526 | "name": "stdout", 1527 | "output_type": "stream", 1528 | "text": [ 1529 | "Dataset shape: (506, 13)\n", 1530 | "X train: (379, 13)\n" 1531 | ] 1532 | }, 1533 | { 1534 | "data": { 1535 | "text/html": [ 1536 | "
\n", 1537 | "\n", 1550 | "\n", 1551 | " \n", 1552 | " \n", 1553 | " \n", 1554 | " \n", 1555 | " \n", 1556 | " \n", 1557 | " \n", 1558 | " \n", 1559 | " \n", 1560 | " \n", 1561 | " \n", 1562 | " \n", 1563 | " \n", 1564 | " \n", 1565 | " \n", 1566 | " \n", 1567 | " \n", 1568 | " \n", 1569 | " \n", 1570 | " \n", 1571 | " \n", 1572 | " \n", 1573 | " \n", 1574 | " \n", 1575 | " \n", 1576 | " \n", 1577 | " \n", 1578 | " \n", 1579 | " \n", 1580 | " \n", 1581 | " \n", 1582 | " \n", 1583 | " \n", 1584 | " \n", 1585 | " \n", 1586 | " \n", 1587 | " \n", 1588 | " \n", 1589 | " \n", 1590 | " \n", 1591 | " \n", 1592 | " \n", 1593 | " \n", 1594 | " \n", 1595 | " \n", 1596 | " \n", 1597 | " \n", 1598 | " \n", 1599 | " \n", 1600 | " \n", 1601 | " \n", 1602 | " \n", 1603 | " \n", 1604 | " \n", 1605 | " \n", 1606 | " \n", 1607 | " \n", 1608 | " \n", 1609 | " \n", 1610 | " \n", 1611 | " \n", 1612 | " \n", 1613 | " \n", 1614 | " \n", 1615 | " \n", 1616 | " \n", 1617 | " \n", 1618 | " \n", 1619 | " \n", 1620 | " \n", 1621 | " \n", 1622 | " \n", 1623 | " \n", 1624 | " \n", 1625 | " \n", 1626 | " \n", 1627 | " \n", 1628 | " \n", 1629 | " \n", 1630 | " \n", 1631 | " \n", 1632 | " \n", 1633 | " \n", 1634 | " \n", 1635 | " \n", 1636 | " \n", 1637 | " \n", 1638 | " \n", 1639 | " \n", 1640 | " \n", 1641 | " \n", 1642 | " \n", 1643 | " \n", 1644 | " \n", 1645 | " \n", 1646 | " \n", 1647 | " \n", 1648 | " \n", 1649 | " \n", 1650 | " \n", 1651 | " \n", 1652 | " \n", 1653 | " \n", 1654 | " \n", 1655 | " \n", 1656 | " \n", 1657 | " \n", 1658 | " \n", 1659 | " \n", 1660 | " \n", 1661 | " \n", 1662 | " \n", 1663 | " \n", 1664 | " \n", 1665 | " \n", 1666 | " \n", 1667 | " \n", 1668 | " \n", 1669 | " \n", 1670 | " \n", 1671 | " \n", 1672 | " \n", 1673 | " \n", 1674 | " \n", 1675 | " \n", 1676 | " \n", 1677 | " \n", 1678 | " \n", 1679 | " \n", 1680 | " \n", 1681 | " \n", 1682 | " \n", 1683 | " \n", 1684 | " \n", 1685 | " \n", 1686 | " \n", 1687 | " \n", 1688 | " \n", 1689 | " \n", 1690 | " \n", 1691 | " \n", 1692 | " \n", 1693 | " \n", 1694 | " \n", 1695 | " \n", 1696 | " \n", 1697 | " \n", 1698 | " \n", 1699 | " \n", 1700 | " \n", 1701 | " \n", 1702 | " \n", 1703 | " \n", 1704 | " \n", 1705 | " \n", 1706 | " \n", 1707 | " \n", 1708 | " \n", 1709 | " \n", 1710 | " \n", 1711 | " \n", 1712 | " \n", 1713 | " \n", 1714 | " \n", 1715 | " \n", 1716 | " \n", 1717 | " \n", 1718 | " \n", 1719 | " \n", 1720 | " \n", 1721 | " \n", 1722 | " \n", 1723 | " \n", 1724 | " \n", 1725 | " \n", 1726 | " \n", 1727 | " \n", 1728 | " \n", 1729 | " \n", 1730 | " \n", 1731 | " \n", 1732 | " \n", 1733 | " \n", 1734 | " \n", 1735 | " \n", 1736 | " \n", 1737 | " \n", 1738 | " \n", 1739 | " \n", 1740 | " \n", 1741 | " \n", 1742 | " \n", 1743 | " \n", 1744 | " \n", 1745 | " \n", 1746 | " \n", 1747 | "
CRIMZNINDUSCHASNOXRMAGEDISRADTAXPTRATIOBLSTAT
00.0063218.02.310.00.5386.57565.24.09001.0296.015.3396.904.98
10.027310.07.070.00.4696.42178.94.96712.0242.017.8396.909.14
20.027290.07.070.00.4697.18561.14.96712.0242.017.8392.834.03
30.032370.02.180.00.4586.99845.86.06223.0222.018.7394.632.94
40.069050.02.180.00.4587.14754.26.06223.0222.018.7396.905.33
..........................................
5010.062630.011.930.00.5736.59369.12.47861.0273.021.0391.999.67
5020.045270.011.930.00.5736.12076.72.28751.0273.021.0396.909.08
5030.060760.011.930.00.5736.97691.02.16751.0273.021.0396.905.64
5040.109590.011.930.00.5736.79489.32.38891.0273.021.0393.456.48
5050.047410.011.930.00.5736.03080.82.50501.0273.021.0396.907.88
\n", 1748 | "

506 rows × 13 columns

\n", 1749 | "
" 1750 | ], 1751 | "text/plain": [ 1752 | " CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX \\\n", 1753 | "0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0 \n", 1754 | "1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0 \n", 1755 | "2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0 \n", 1756 | "3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0 \n", 1757 | "4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0 \n", 1758 | ".. ... ... ... ... ... ... ... ... ... ... \n", 1759 | "501 0.06263 0.0 11.93 0.0 0.573 6.593 69.1 2.4786 1.0 273.0 \n", 1760 | "502 0.04527 0.0 11.93 0.0 0.573 6.120 76.7 2.2875 1.0 273.0 \n", 1761 | "503 0.06076 0.0 11.93 0.0 0.573 6.976 91.0 2.1675 1.0 273.0 \n", 1762 | "504 0.10959 0.0 11.93 0.0 0.573 6.794 89.3 2.3889 1.0 273.0 \n", 1763 | "505 0.04741 0.0 11.93 0.0 0.573 6.030 80.8 2.5050 1.0 273.0 \n", 1764 | "\n", 1765 | " PTRATIO B LSTAT \n", 1766 | "0 15.3 396.90 4.98 \n", 1767 | "1 17.8 396.90 9.14 \n", 1768 | "2 17.8 392.83 4.03 \n", 1769 | "3 18.7 394.63 2.94 \n", 1770 | "4 18.7 396.90 5.33 \n", 1771 | ".. ... ... ... \n", 1772 | "501 21.0 391.99 9.67 \n", 1773 | "502 21.0 396.90 9.08 \n", 1774 | "503 21.0 396.90 5.64 \n", 1775 | "504 21.0 393.45 6.48 \n", 1776 | "505 21.0 396.90 7.88 \n", 1777 | "\n", 1778 | "[506 rows x 13 columns]" 1779 | ] 1780 | }, 1781 | "execution_count": 31, 1782 | "metadata": {}, 1783 | "output_type": "execute_result" 1784 | } 1785 | ], 1786 | "source": [ 1787 | "from sklearn.preprocessing import StandardScaler\n", 1788 | "from sklearn.model_selection import train_test_split\n", 1789 | "\n", 1790 | "Xb, yb = shap.datasets.boston()\n", 1791 | "\n", 1792 | "std = StandardScaler()\n", 1793 | "X_standard = std.fit_transform(Xb)\n", 1794 | "X_trainn, X_testt = train_test_split(X_standard)\n", 1795 | "\n", 1796 | "print(\"Dataset shape:\",Xb.shape)\n", 1797 | "print(\"X train:\",X_trainn.shape)\n", 1798 | "Xb" 1799 | ] 1800 | }, 1801 | { 1802 | "cell_type": "markdown", 1803 | "metadata": {}, 1804 | "source": [ 1805 | "**Model summary for this implementation:**
\n", 1806 | "![title](images/SHAP_model_other.png)" 1807 | ] 1808 | }, 1809 | { 1810 | "cell_type": "markdown", 1811 | "metadata": {}, 1812 | "source": [ 1813 | "### 2. LIME Implementation using LSTM in Co2 dataset\n", 1814 | "\n", 1815 | "1. CO2 dataset has 2 features. This is a classification problem but data is time series.
\n", 1816 | "2. They have pre-processed the data with respect to 12 timesteps.
\n", 1817 | "3. Thus their input shape is **(2270, 12, 2)** and output shape is **(2270, 2)**.\n", 1818 | "\n", 1819 | "**Reference**
\n", 1820 | "https://github.com/marcotcr/lime/blob/master/doc/notebooks/Lime%20with%20Recurrent%20Neural%20Networks.ipynb" 1821 | ] 1822 | }, 1823 | { 1824 | "cell_type": "markdown", 1825 | "metadata": {}, 1826 | "source": [ 1827 | "![title](images/LIME.png)" 1828 | ] 1829 | }, 1830 | { 1831 | "cell_type": "markdown", 1832 | "metadata": {}, 1833 | "source": [ 1834 | "### Wrap-up:\n", 1835 | "\n", 1836 | "If we compare our model with the two above model summary and data shape, it is pretty evident that the input and output feature shapes should be the same.
\n", 1837 | "1. First implementation has 13 input features and output prediction also has 13 features.\n", 1838 | "2. Second implementation has 2 input features and output also has 2 features.\n", 1839 | "\n", 1840 | "From these obseravtions, we can also see that SHAP and LIME worked as expected with both datasets and models without any errors or exceptions as the input and output shapes are equal. " 1841 | ] 1842 | }, 1843 | { 1844 | "cell_type": "code", 1845 | "execution_count": null, 1846 | "metadata": {}, 1847 | "outputs": [], 1848 | "source": [] 1849 | } 1850 | ], 1851 | "metadata": { 1852 | "kernelspec": { 1853 | "display_name": "Python 3", 1854 | "language": "python", 1855 | "name": "python3" 1856 | }, 1857 | "language_info": { 1858 | "codemirror_mode": { 1859 | "name": "ipython", 1860 | "version": 3 1861 | }, 1862 | "file_extension": ".py", 1863 | "mimetype": "text/x-python", 1864 | "name": "python", 1865 | "nbconvert_exporter": "python", 1866 | "pygments_lexer": "ipython3", 1867 | "version": "3.7.9" 1868 | } 1869 | }, 1870 | "nbformat": 4, 1871 | "nbformat_minor": 4 1872 | } 1873 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Explaining deep LSTM models for detecting anomalies in time-series data (R&D Project) 2 | 3 | This research work focuses on comparing the existing approaches to explain the decisions of models trained using time-series data and proposing the best-fit method that generates `explanations` for a deep neural network. The proposed approach is used specifically for explaining `LSTM` networks for `anomaly detection` task in `time-series data (satellite telemetry data)`. 4 | 5 | ## Author 6 | 7 | - [@Jithin Sasikumar](https://www.github.com/Jithsaavvy) 8 | 9 | ## Acknowledgements 10 | 11 | - [@German Aerospace Center (DLR)](https://www.dlr.de/DE/Home/home_node.html) 12 | - [@Hochschule Bonn-Rhein-Sieg](https://www.h-brs.de/en) 13 | 14 | ## Languages and Tools 15 |

pythontensorflow numpy ipynb

16 | 17 | `Explainable AI (XAI)` methods are applied for `LSTM regression-based` anomaly detection models with different configurations `(simple/vanilla LSTM, encoder-decoder based LSTM, stacked LSTM and bi-directional LSTM)` for `satellite telemetry` data (time-series). These methods are compared based on the explanations obtained as a result of implementation as plots. A comparative analysis is performed by comparing the implemented methods based on generated results differentiating the methods fitting between differentiable and non-differentiable models. 18 | 19 | ## Why XAI? 20 | 21 | It serves to answer the `why` questions such as - _Why a specific input data is classified as anomaly?_ _What features made the model to classify a specific data as anomalous ones?_ and so on. The model should be clear and evident in detecting anomalies. Any failure will ruin the complete system, in our case - it can imperil the entire satellite if the anomaly detection mechanism is not properly done, hence it is necessary to explain the models. Explanations are generated for the users to better understand about the model prediction. To facilitate that, these `explainable AI (XAI)` methods are used. 22 | 23 | ## Problem Statement 24 | 25 | - The lack of `transparency` and `interpretability` in deep learning based models is a major drawback for understanding its decisions (or) predictions, making them as `black box` that restricts AI systems to be implemented in safety-critical applications in real-time with less supervision. Explainable AI is proposed to solve the black box problem. 26 | - Efficient anomaly detection systems should be employed to monitor the satellite telemetry channels that produces high volumes of `telemetry data`. Any failure in detecting anomaly could endanger the whole satellite. 27 | - Various methods (or) approaches are available for explaining and interpreting deep learning models, specifically for image and text data but a suitable method for a specific deep network architecture is not yet proposed for anomaly detection task using time series data. 28 | 29 | ## LSTM models used 30 | 31 | ![lstm](./images/LSTM%20Models.png) 32 | 33 | All LSTM model architectures are illustrated in [images](./images/) folder. For example, the architecture of `encoder-decoder LSTM` with $16$ timesteps is depicted as follows: 34 | 35 | ![encdec](./images/encdects_16_ts.PNG) 36 | 37 | ## XAI methods used 38 | - [LRP](http://www.heatmapping.org/) 39 | - [LIME](https://arxiv.org/abs/1602.04938) 40 | - [SHAP](https://shap.readthedocs.io/en/latest/index.html#)
41 | 42 | ## Notebooks 43 | 44 | [LRP implementation](./notebooks/LRP_Implementation.ipynb) describes the implementation of `Layer-wise Relevance Propogation (LRP)` which is a `gradient-based attribution method` for LSTM regression model trained on time-series data along with their issues and limitations. It contains data pre-processing, model summary and implementation of LRP method. My observations are documented. 45 | 46 | [LIME and SHAP implementation](./notebooks/LIME_and_SHAP_Implementation.ipynb) describes the implementation of `perturbation-based methods (SHAP and LIME)` for LSTM models with different architectures trained on time-series data along with their issues and limitations. It contains data pre-processing, model summary, model prediction, anomaly detection, SHAP and LIME implementation. All the models are developed from scratch using `tensorflow`, `keras` and trained except NASA model which is downloaded from [1] and based on [2]. The models are compared and my observations are documented. 47 | 48 | ## Results 49 | 50 | ### SHAP 51 | 52 | SHAP force plot for vanilla LSTM 53 | ![shap1](./result%20plots/shap_vanilla_ts1.png) 54 | 55 | SHAP force plot for bi-directional LSTM 56 | ![shap2](./result%20plots/shap_bidi_lstm.PNG) 57 | 58 | SHAP force plot for stacked LSTM 59 | ![shap3](./result%20plots/shap_stacked_%20lstm_ts16_1.PNG) 60 | 61 | SHAP force plot for encoder-decoder LSTM 62 | ![shap4](./result%20plots/shap_enc_dec_ts1.png) 63 | 64 | _In the above plot, it doesn't plot anything for encoder-decoder based LSTM due to the addition of flatten layers which resulted in the computation of zero SHAP values_. 65 | 66 | ### LRP 67 | 68 | LRP is implemented for the `bi-directional LSTM` model as the current LRP implementation [3] supports only bi-directional LSTMs. which is also a deficit. 69 | 70 | ![lrp1](./result%20plots/LRP%20relevance%20score%20for%20all%20instances.PNG) 71 | ![lrp2](./result%20plots/Bar%20plot%20-%20LRP%20relevance%20score%20for%201%20instance.PNG) 72 | 73 | ### LIME 74 | 75 | `Thows an exception!!!` LIME is proposed to generate explanations for any classifier. So, LIME API expects the input model to be a classifier (i.e.) the model should output probabilities with sigmoid (or) softmax output layer as activations. As our model is regression based, it has `ReLU` activation output layer, and does not output any probabilities. 76 | ![lime](./images/lime%20output.PNG) 77 | 78 | ## Comparative analysis 79 | 80 | ![comp](./result%20plots/comp%20table%20new.PNG) 81 | 82 | ## Conclusions and Take aways 83 | 84 | - Based on the comparative analysis and implementation, gradient based attribution methods serves to be the `best-fit approach` for LSTM regression based models. 85 | - Out of the perturbation based methods, `SHAP` fit for our models, as it generates `global explanations` and works for any models other than classifiers whereas `LIME` does not work for regression models. 86 | - But SHAP is not a best-fit approach for neural networks, because all the perturbation based methods are proposed to work for `non-differentiable models`. They do work for neural networks as well but they are not optimized. 87 | - SHAP performs very `slower` than LRP for LSTM network based on the [execution 88 | time](./result%20plots/comp%20table%20new.PNG). 89 | - Thus, `LSTM` network being a `differentiable model` (model with gradients) works best with gradient based methods. It can be concluded that `gradient-based methods` are the best fit methods exclusively for LSTM regression based neural networks. 90 | 91 | ## Future works 92 | - The research work can be extended to implement the other explainable methods like `integrated gradients`, `DeepLIFT` and so on, as they were not implemented in this work due to time constraints. 93 | - Current LRP implementation is supported only for bi-directional LSTM models. It can be extended to support multiple LSTM model architectures with additional functionalities. 94 | - `Optimal evaluation metrics` for evaluating the explanations generated by XAI methods for LSTM networks using time series data for anomaly detection is not proposed. 95 | 96 | ### _Note !!!_ 97 | 1. _As this is a research project, the implementation and other details are demonstrated using `jupyter notebooks`. Also, the code is `not refactored`._ 98 | 2. _This work was carried out between **June 2020 to March 2021**. All the tools and libraries used, implementation details were based on the ones available during that time frame._ 99 | 100 | ## Citation 101 | 102 | If you want to use this work, please cite: 103 | 104 | ``` 105 | title = {Explaining deep learning models for detecting anomalies in time-series data}, 106 | author = {Jithin Sasikumar}, 107 | month = "02", 108 | year = "2021", 109 | } 110 | ``` 111 | 112 | ## References 113 | 114 | [1] https://github.com/khundman/telemanom
115 | [2] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom.Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery amp; Data Mining, KDD ’18, page 387–395, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520. doi: 10.1145/3219819.3219845. URL https://doi.org/10.1145/3219819.3219845
116 | [3] AlexanderWarnecke. Layerwise Relevance Propagation for LSTMs, 2021. URL https://github.com/alewarne/Layerwise-Relevance-Propagation-for-LSTMs. Accessed on: 2021-02-06. [Online].
117 | [4] https://www.kaggle.com/datasets/vinayak123tyagi/bearing-dataset 118 | -------------------------------------------------------------------------------- /images/LSTM Models.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/LSTM Models.png -------------------------------------------------------------------------------- /images/bidi.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/bidi.PNG -------------------------------------------------------------------------------- /images/encdects_16_ts.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/encdects_16_ts.PNG -------------------------------------------------------------------------------- /images/encdects_1_ts.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/encdects_1_ts.PNG -------------------------------------------------------------------------------- /images/stackedlstmts_16_ts.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/stackedlstmts_16_ts.PNG -------------------------------------------------------------------------------- /images/vanilla lstmts_1_ts.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/images/vanilla lstmts_1_ts.PNG -------------------------------------------------------------------------------- /models/bidi_model.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/bidi_model.h5 -------------------------------------------------------------------------------- /models/enc_dec_model_ts1.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/enc_dec_model_ts1.h5 -------------------------------------------------------------------------------- /models/enc_dec_model_ts16.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/enc_dec_model_ts16.h5 -------------------------------------------------------------------------------- /models/stacked_lstm_model_ts16_1.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/stacked_lstm_model_ts16_1.h5 -------------------------------------------------------------------------------- /models/stacked_lstm_model_ts16_2.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/stacked_lstm_model_ts16_2.h5 -------------------------------------------------------------------------------- /models/vanilla_lstm_model_ts1.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/models/vanilla_lstm_model_ts1.h5 -------------------------------------------------------------------------------- /result plots/Bar plot - LRP relevance score for 1 instance.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/Bar plot - LRP relevance score for 1 instance.PNG -------------------------------------------------------------------------------- /result plots/LRP relevance score for all instances.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/LRP relevance score for all instances.PNG -------------------------------------------------------------------------------- /result plots/comp table new.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/comp table new.PNG -------------------------------------------------------------------------------- /result plots/shap_bidi_lstm.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/shap_bidi_lstm.PNG -------------------------------------------------------------------------------- /result plots/shap_enc_dec_ts1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/shap_enc_dec_ts1.png -------------------------------------------------------------------------------- /result plots/shap_stacked_ lstm_ts16_1.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/shap_stacked_ lstm_ts16_1.PNG -------------------------------------------------------------------------------- /result plots/shap_vanilla_ts1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jithsaavvy/Explaining-deep-learning-models-for-detecting-anomalies-in-time-series-data-RnD-project/d4362c1d6c6ec98412e872020ab26aa7d1283e04/result plots/shap_vanilla_ts1.png --------------------------------------------------------------------------------