├── LICENSE ├── README.md ├── Tatsuya_Hasegawa_msticpy_SANS_APAC_DFIR_SUMMIT_2023.pdf ├── analysis_outliers ├── README.md └── outliers_multi_dimension_PR.ipynb ├── analysis_rrcf_outliers ├── msticpy-if_outliers.ipynb └── msticpy-rrcf_outliers.ipynb ├── data └── buttercupgame_iplocation.csv ├── msticpy_light_tutorial.ipynb ├── qp_splunk_poc_bugfix ├── msticpy_qp_splunk.ipynb ├── msticpy_splunk_9_0_4_reader_bug.ipynb ├── msticpy_splunk_9_0_4_reader_merged.ipynb ├── msticpy_splunk_reader_bug.ipynb └── msticpy_splunk_reader_paging-test.csv ├── splunk_dsdl ├── README.md └── msticpy_powershell_ioc.ipynb └── timeseries_anomalies_stl └── msticpy_timeseries_anomalies_stl.ipynb /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 hackeT 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MSTICPy_utils 2 | 3 | - Mainly my MSTICpy practice repository. 4 | - Jupyter notebook files with useful msticpy execution's how-to history. 5 | 6 | | File or Directory | Short Desc | 7 | | ------------- | ------------- | 8 | | [msticpy_light_tutorial.ipynb](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/msticpy_light_tutorial.ipynb) | Tutorial to use msticpy along 'Quick Start Overview' | 9 | | [timeseries_anomalies_stl](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/timeseries_anomalies_stl) | Practice of timeseries_anomalies_stl function | 10 | | [splunk_dsdl](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/splunk_dsdl) | msticpy joint to Splunk DSDL example for powershell base64 hunting refered by my DFIR Summit PDF | 11 | | [qp_splunk_poc_bugfix](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/qp_splunk_poc_bugfix) | PoC notebooks and test csv of my [PR#657](https://github.com/microsoft/msticpy/pull/657) to msticpy | 12 | | [analysis_outliers](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/analysis_outliers) | PoC notebooks of my [PR#805](https://github.com/microsoft/msticpy/pull/805) to msticpy | 13 | | [analysis_rrcf_outliers](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/analysis_rrcf_outliers) | PoC notebooks of my [PR#846](https://github.com/microsoft/msticpy/pull/846) to msticpy, comparing IsolationForest and RobustRandomCutForest with same dataset | 14 | | [data](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/data) | sample data directory to be utilized by this repository's some notebook files | 15 | 16 | ### Presentation on SANS APAC DFIR Summit 2023 17 | 18 | [Practical msticpy use ~ rainbow bridge to SIEM for advanced threat hunting ~](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/d3b5e589ab4de714b430a5274a3378bde21a3aaf/Tatsuya_Hasegawa_msticpy_SANS_APAC_DFIR_SUMMIT_2023.pdf) 19 | 20 | 21 | -------------------------------------------------------------------------------- /Tatsuya_Hasegawa_msticpy_SANS_APAC_DFIR_SUMMIT_2023.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Tatsuya-hasegawa/MSTICPy_utils/c37a4fa3aeb1361f2ae07fb348139ae92b5de0b6/Tatsuya_Hasegawa_msticpy_SANS_APAC_DFIR_SUMMIT_2023.pdf -------------------------------------------------------------------------------- /analysis_outliers/README.md: -------------------------------------------------------------------------------- 1 | TBD 2 | -------------------------------------------------------------------------------- /qp_splunk_poc_bugfix/msticpy_qp_splunk.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "# PoC: SplunkGeneral.get_events_parameterized function will fetch all the Splunk records\n", 9 | "\n", 10 | "with my PR code https://github.com/microsoft/msticpy/pull/657\n", 11 | "\n", 12 | "\n", 13 | "Reference Splunk SDK python:\n", 14 | "- https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/howtousesplunkpython/howtodisplaysearchpython/#To-paginate-through-a-large-set-of-results\n", 15 | "- https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/howtousesplunkpython/howtorunsearchespython/\n", 16 | "\n", 17 | "- https://docs.splunk.com/DocumentationStatic/PythonSDK/1.7.2/client.html\n", 18 | "- https://docs.splunk.com/DocumentationStatic/PythonSDK/1.7.2/results.html\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stderr", 29 | "output_type": "stream", 30 | "text": [ 31 | "2023-04-18 13:40:04,642: WARNING - config validation error Missing or empty 'AzureSentinel' section (nbinit#697)\n", 32 | "2023-04-18 13:40:04,643: WARNING - Could not find msticpyconfig.yaml in standard search. (nbinit#710)\n" 33 | ] 34 | }, 35 | { 36 | "data": { 37 | "text/html": [ 38 | "

Notebook setup completed with some warnings.

" 39 | ], 40 | "text/plain": [ 41 | "" 42 | ] 43 | }, 44 | "metadata": {}, 45 | "output_type": "display_data" 46 | }, 47 | { 48 | "data": { 49 | "text/html": [ 50 | "

One or more configuration items were missing or set incorrectly.

" 51 | ], 52 | "text/plain": [ 53 | "" 54 | ] 55 | }, 56 | "metadata": {}, 57 | "output_type": "display_data" 58 | }, 59 | { 60 | "data": { 61 | "text/html": [ 62 | "

Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.

" 63 | ], 64 | "text/plain": [ 65 | "" 66 | ] 67 | }, 68 | "metadata": {}, 69 | "output_type": "display_data" 70 | }, 71 | { 72 | "data": { 73 | "text/html": [ 74 | "

This notebook may still run but with reduced functionality.

" 75 | ], 76 | "text/plain": [ 77 | "" 78 | ] 79 | }, 80 | "metadata": {}, 81 | "output_type": "display_data" 82 | } 83 | ], 84 | "source": [ 85 | "import msticpy as mp\n", 86 | "mp.init_notebook()" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 2, 92 | "metadata": {}, 93 | "outputs": [ 94 | { 95 | "name": "stdout", 96 | "output_type": "stream", 97 | "text": [ 98 | "connected\n" 99 | ] 100 | } 101 | ], 102 | "source": [ 103 | "splunk_prov = mp.QueryProvider(\"Splunk\")\n", 104 | "splunk_prov.connect()" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 3, 110 | "metadata": {}, 111 | "outputs": [ 112 | { 113 | "name": "stdout", 114 | "output_type": "stream", 115 | "text": [ 116 | "Query: get_events_parameterized\n", 117 | "Data source: Splunk\n", 118 | "Generic parameterized query from index/source\n", 119 | "\n", 120 | "Parameters\n", 121 | "----------\n", 122 | "add_query_items: str (optional)\n", 123 | " Additional query clauses\n", 124 | " (default value is: | head 100)\n", 125 | "end: datetime\n", 126 | " Query end time\n", 127 | "index: str (optional)\n", 128 | " Splunk index name\n", 129 | " (default value is: *)\n", 130 | "project_fields: str (optional)\n", 131 | " Project Field names\n", 132 | " (default value is: | table TimeCreated, host, EventID, EventDescripti...)\n", 133 | "source: str (optional)\n", 134 | " Splunk source type\n", 135 | " (default value is: *)\n", 136 | "start: datetime\n", 137 | " Query start time\n", 138 | "timeformat: str (optional)\n", 139 | " Datetime format to use in Splunk query\n", 140 | " (default value is: \"%Y-%m-%d %H:%M:%S.%6N\")\n", 141 | "Query:\n", 142 | " search index={index} source={source} timeformat={timeformat} earliest={start} latest={end} {project_fields} {add_query_items}\n" 143 | ] 144 | } 145 | ], 146 | "source": [ 147 | "splunk_prov.SplunkGeneral.get_events_parameterized('?')" 148 | ] 149 | }, 150 | { 151 | "attachments": {}, 152 | "cell_type": "markdown", 153 | "metadata": {}, 154 | "source": [ 155 | "## Test with botsv2 data" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 3, 161 | "metadata": {}, 162 | "outputs": [ 163 | { 164 | "data": { 165 | "text/plain": [ 166 | "' search index=botsv2 source=WinEventLog:Microsoft-Windows-Sysmon/Operational timeformat=\"%Y-%m-%d %H:%M:%S\" earliest=\"2017-08-25 00:00:00\" latest=\"2017-08-25 10:00:00\" | table TimeCreated, host, EventID, EventDescription, User, process, cmdline, Image, parent_process, ParentCommandLine, dest, Hashes | head 100'" 167 | ] 168 | }, 169 | "execution_count": 3, 170 | "metadata": {}, 171 | "output_type": "execute_result" 172 | } 173 | ], 174 | "source": [ 175 | "splunk_prov.SplunkGeneral.get_events_parameterized('print',\n", 176 | " index=\"botsv2\",\n", 177 | " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", 178 | " timeformat='\"%Y-%m-%d %H:%M:%S\"',\n", 179 | " start=\"2017-08-25 00:00:00\",\n", 180 | " end=\"2017-08-25 10:00:00\"\n", 181 | ")" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 4, 187 | "metadata": {}, 188 | "outputs": [ 189 | { 190 | "name": "stderr", 191 | "output_type": "stream", 192 | "text": [ 193 | "Waiting Splunk job to complete: 200.0it [00:03, 60.67it/s] " 194 | ] 195 | }, 196 | { 197 | "name": "stdout", 198 | "output_type": "stream", 199 | "text": [ 200 | "100.0% 11268 scanned 11268 matched 100 results\n", 201 | "Splunk job has Done!\n" 202 | ] 203 | }, 204 | { 205 | "name": "stderr", 206 | "output_type": "stream", 207 | "text": [ 208 | "\n" 209 | ] 210 | }, 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "Implicit parameter dump - 'paginate_width': 100 ,which means 100 records will be retrieved per one fetch.\n", 216 | " You can set paginate_width= to this function's option.\n" 217 | ] 218 | }, 219 | { 220 | "name": "stderr", 221 | "output_type": "stream", 222 | "text": [ 223 | "Waiting Splunk result to retrieve: 200it [00:00, 17219.06it/s] " 224 | ] 225 | }, 226 | { 227 | "name": "stdout", 228 | "output_type": "stream", 229 | "text": [ 230 | "Retrieved 100 results.\n" 231 | ] 232 | }, 233 | { 234 | "name": "stderr", 235 | "output_type": "stream", 236 | "text": [ 237 | "\n" 238 | ] 239 | }, 240 | { 241 | "data": { 242 | "text/plain": [ 243 | "100" 244 | ] 245 | }, 246 | "execution_count": 4, 247 | "metadata": {}, 248 | "output_type": "execute_result" 249 | } 250 | ], 251 | "source": [ 252 | "default_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 253 | " index=\"botsv2\",\n", 254 | " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", 255 | " start=\"2017-08-25 00:00:00.000000\",\n", 256 | " end=\"2017-08-25 10:00:00.000000\"\n", 257 | ")\n", 258 | "len(default_df) # 100 because of add_query_items = '| head 100' by default" 259 | ] 260 | }, 261 | { 262 | "attachments": {}, 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "### fetch unlimited records " 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": 10, 272 | "metadata": {}, 273 | "outputs": [ 274 | { 275 | "data": { 276 | "text/plain": [ 277 | "' search index=botsv2 source=WinEventLog:Microsoft-Windows-Sysmon/Operational timeformat=\"%Y-%m-%d %H:%M:%S.%6N\" earliest=\"2017-08-25 00:00:00\" latest=\"2017-08-25 10:00:00\" | table TimeCreated, host, EventID, EventDescription, User, process, cmdline, Image, parent_process, ParentCommandLine, dest, Hashes '" 278 | ] 279 | }, 280 | "execution_count": 10, 281 | "metadata": {}, 282 | "output_type": "execute_result" 283 | } 284 | ], 285 | "source": [ 286 | "splunk_prov.SplunkGeneral.get_events_parameterized('print',\n", 287 | " index=\"botsv2\",\n", 288 | " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", 289 | " start=\"2017-08-25 00:00:00.000000\",\n", 290 | " end=\"2017-08-25 10:00:00.000000\",\n", 291 | " add_query_items=''\n", 292 | ")" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 12, 298 | "metadata": {}, 299 | "outputs": [ 300 | { 301 | "name": "stderr", 302 | "output_type": "stream", 303 | "text": [ 304 | "Waiting Splunk job to complete: 200.0it [00:03, 60.83it/s] " 305 | ] 306 | }, 307 | { 308 | "name": "stdout", 309 | "output_type": "stream", 310 | "text": [ 311 | "100.0% 11268 scanned 11268 matched 11268 results\n", 312 | "Splunk job has Done!\n" 313 | ] 314 | }, 315 | { 316 | "name": "stderr", 317 | "output_type": "stream", 318 | "text": [ 319 | "\n" 320 | ] 321 | }, 322 | { 323 | "name": "stdout", 324 | "output_type": "stream", 325 | "text": [ 326 | "Implicit parameter dump - 'paginate_width': 100 ,which means 100 records will be retrieved per one fetch.\n", 327 | " You can set paginate_width= to this function's option.\n" 328 | ] 329 | }, 330 | { 331 | "name": "stderr", 332 | "output_type": "stream", 333 | "text": [ 334 | "Waiting Splunk result to retrieve: 22568it [00:01, 16391.93it/s] " 335 | ] 336 | }, 337 | { 338 | "name": "stdout", 339 | "output_type": "stream", 340 | "text": [ 341 | "Retrieved 11268 results.\n" 342 | ] 343 | }, 344 | { 345 | "name": "stderr", 346 | "output_type": "stream", 347 | "text": [ 348 | "\n" 349 | ] 350 | } 351 | ], 352 | "source": [ 353 | "result_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 354 | " index=\"botsv2\",\n", 355 | " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", 356 | " start=\"2017-08-25 00:00:00.000000\",\n", 357 | " end=\"2017-08-25 10:00:00.000000\",\n", 358 | " add_query_items='',\n", 359 | " count=0\n", 360 | ")" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": 13, 366 | "metadata": {}, 367 | "outputs": [ 368 | { 369 | "data": { 370 | "text/plain": [ 371 | "11268" 372 | ] 373 | }, 374 | "execution_count": 13, 375 | "metadata": {}, 376 | "output_type": "execute_result" 377 | } 378 | ], 379 | "source": [ 380 | "len(result_df)" 381 | ] 382 | }, 383 | { 384 | "attachments": {}, 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "## Test with original csv \"msticpy_splunk_reader_paging-test.csv\"" 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "execution_count": 14, 394 | "metadata": {}, 395 | "outputs": [ 396 | { 397 | "name": "stderr", 398 | "output_type": "stream", 399 | "text": [ 400 | "Waiting Splunk job to complete: 200.0it [00:03, 60.31it/s] " 401 | ] 402 | }, 403 | { 404 | "name": "stdout", 405 | "output_type": "stream", 406 | "text": [ 407 | "100.0% 100000 scanned 100000 matched 100000 results\n", 408 | "Splunk job has Done!\n" 409 | ] 410 | }, 411 | { 412 | "name": "stderr", 413 | "output_type": "stream", 414 | "text": [ 415 | "\n" 416 | ] 417 | }, 418 | { 419 | "name": "stdout", 420 | "output_type": "stream", 421 | "text": [ 422 | "Implicit parameter dump - 'paginate_width': 100 ,which means 100 records will be retrieved per one fetch.\n", 423 | " You can set paginate_width= to this function's option.\n" 424 | ] 425 | }, 426 | { 427 | "name": "stderr", 428 | "output_type": "stream", 429 | "text": [ 430 | "Waiting Splunk result to retrieve: 200000it [00:39, 5096.06it/s] \n" 431 | ] 432 | }, 433 | { 434 | "name": "stdout", 435 | "output_type": "stream", 436 | "text": [ 437 | "Retrieved 100000 results.\n" 438 | ] 439 | } 440 | ], 441 | "source": [ 442 | "# paginate_width = 100 by default\n", 443 | "result_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 444 | " index=\"msticpy\",\n", 445 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 446 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 447 | " add_query_items='',\n", 448 | " count=0\n", 449 | ")\n", 450 | "result_df['timestamp'] = result_df['timestamp'].astype('float')\n", 451 | "result_df['rownum'] = result_df['rownum'].astype('int')" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": 15, 457 | "metadata": {}, 458 | "outputs": [ 459 | { 460 | "data": { 461 | "text/plain": [ 462 | "100000" 463 | ] 464 | }, 465 | "execution_count": 15, 466 | "metadata": {}, 467 | "output_type": "execute_result" 468 | } 469 | ], 470 | "source": [ 471 | "len(result_df)" 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": 16, 477 | "metadata": {}, 478 | "outputs": [ 479 | { 480 | "data": { 481 | "text/plain": [ 482 | "array([ 1, 2, 3, ..., 99998, 99999, 100000])" 483 | ] 484 | }, 485 | "execution_count": 16, 486 | "metadata": {}, 487 | "output_type": "execute_result" 488 | } 489 | ], 490 | "source": [ 491 | "sort_df = result_df.sort_values('rownum')\n", 492 | "sort_df['rownum'].to_numpy()" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": 17, 498 | "metadata": {}, 499 | "outputs": [ 500 | { 501 | "data": { 502 | "text/html": [ 503 | "
\n", 504 | "\n", 517 | "\n", 518 | " \n", 519 | " \n", 520 | " \n", 521 | " \n", 522 | " \n", 523 | " \n", 524 | " \n", 525 | " \n", 526 | " \n", 527 | " \n", 528 | " \n", 529 | " \n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | " \n", 534 | " \n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 542 | " \n", 543 | " \n", 544 | " \n", 545 | " \n", 546 | " \n", 547 | " \n", 548 | " \n", 549 | " \n", 550 | " \n", 551 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | "
timestamprownumdescuuid4host
914281.681780e+091testing_rownum17230ab65-7622-4aea-8f89-e0cf94028e80hackeTlab.local
914271.681780e+092testing_rownum20673a921-400f-4f74-9955-2ebe3aa6b568hackeTlab.local
914261.681780e+093testing_rownum31b7d33b8-797f-4b19-978e-89d126d1736dhackeTlab.local
914251.681780e+094testing_rownum49b513862-7cb3-436b-b9b0-cee880d4c19bhackeTlab.local
914241.681780e+095testing_rownum596feff47-29db-4d78-a221-f96df595200fhackeTlab.local
..................
99121.681780e+0999996testing_rownum99996b051d150-b26d-4149-bd28-70f800229edehackeTlab.local
99111.681780e+0999997testing_rownum99997a2192da0-8262-43fb-9301-bd4780a9b499hackeTlab.local
99101.681780e+0999998testing_rownum9999857cd4cf6-b5e8-41dc-815e-870092c54caahackeTlab.local
99091.681780e+0999999testing_rownum99999a3a13916-89b8-4922-967e-ab680131ff39hackeTlab.local
124991.681780e+09100000testing_rownum1000009ac20835-91d1-42f5-8e43-6289ea79fc17hackeTlab.local
\n", 619 | "

100000 rows × 5 columns

\n", 620 | "
" 621 | ], 622 | "text/plain": [ 623 | " timestamp rownum desc \\\n", 624 | "91428 1.681780e+09 1 testing_rownum1 \n", 625 | "91427 1.681780e+09 2 testing_rownum2 \n", 626 | "91426 1.681780e+09 3 testing_rownum3 \n", 627 | "91425 1.681780e+09 4 testing_rownum4 \n", 628 | "91424 1.681780e+09 5 testing_rownum5 \n", 629 | "... ... ... ... \n", 630 | "9912 1.681780e+09 99996 testing_rownum99996 \n", 631 | "9911 1.681780e+09 99997 testing_rownum99997 \n", 632 | "9910 1.681780e+09 99998 testing_rownum99998 \n", 633 | "9909 1.681780e+09 99999 testing_rownum99999 \n", 634 | "12499 1.681780e+09 100000 testing_rownum100000 \n", 635 | "\n", 636 | " uuid4 host \n", 637 | "91428 7230ab65-7622-4aea-8f89-e0cf94028e80 hackeTlab.local \n", 638 | "91427 0673a921-400f-4f74-9955-2ebe3aa6b568 hackeTlab.local \n", 639 | "91426 1b7d33b8-797f-4b19-978e-89d126d1736d hackeTlab.local \n", 640 | "91425 9b513862-7cb3-436b-b9b0-cee880d4c19b hackeTlab.local \n", 641 | "91424 96feff47-29db-4d78-a221-f96df595200f hackeTlab.local \n", 642 | "... ... ... \n", 643 | "9912 b051d150-b26d-4149-bd28-70f800229ede hackeTlab.local \n", 644 | "9911 a2192da0-8262-43fb-9301-bd4780a9b499 hackeTlab.local \n", 645 | "9910 57cd4cf6-b5e8-41dc-815e-870092c54caa hackeTlab.local \n", 646 | "9909 a3a13916-89b8-4922-967e-ab680131ff39 hackeTlab.local \n", 647 | "12499 9ac20835-91d1-42f5-8e43-6289ea79fc17 hackeTlab.local \n", 648 | "\n", 649 | "[100000 rows x 5 columns]" 650 | ] 651 | }, 652 | "execution_count": 17, 653 | "metadata": {}, 654 | "output_type": "execute_result" 655 | } 656 | ], 657 | "source": [ 658 | "sort_df" 659 | ] 660 | }, 661 | { 662 | "attachments": {}, 663 | "cell_type": "markdown", 664 | "metadata": {}, 665 | "source": [ 666 | "OK, Fine.\n", 667 | "\n", 668 | "\n", 669 | "Next is test with paginate_width = 10000" 670 | ] 671 | }, 672 | { 673 | "cell_type": "code", 674 | "execution_count": 18, 675 | "metadata": {}, 676 | "outputs": [ 677 | { 678 | "name": "stderr", 679 | "output_type": "stream", 680 | "text": [ 681 | "Waiting Splunk job to complete: 200.0it [00:03, 60.55it/s] " 682 | ] 683 | }, 684 | { 685 | "name": "stdout", 686 | "output_type": "stream", 687 | "text": [ 688 | "100.0% 100000 scanned 100000 matched 100000 results\n", 689 | "Splunk job has Done!\n" 690 | ] 691 | }, 692 | { 693 | "name": "stderr", 694 | "output_type": "stream", 695 | "text": [ 696 | "\n" 697 | ] 698 | }, 699 | { 700 | "name": "stdout", 701 | "output_type": "stream", 702 | "text": [ 703 | "Implicit parameter dump - 'paginate_width': 10000 ,which means 10000 records will be retrieved per one fetch.\n", 704 | " You can set paginate_width= to this function's option.\n" 705 | ] 706 | }, 707 | { 708 | "name": "stderr", 709 | "output_type": "stream", 710 | "text": [ 711 | "Waiting Splunk result to retrieve: 200000it [00:00, 227110.13it/s] " 712 | ] 713 | }, 714 | { 715 | "name": "stdout", 716 | "output_type": "stream", 717 | "text": [ 718 | "Retrieved 100000 results.\n" 719 | ] 720 | }, 721 | { 722 | "name": "stderr", 723 | "output_type": "stream", 724 | "text": [ 725 | "\n" 726 | ] 727 | } 728 | ], 729 | "source": [ 730 | "# paginate_width = 10000 set to the option\n", 731 | "result_df2 = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 732 | " index=\"msticpy\",\n", 733 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 734 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 735 | " add_query_items='',\n", 736 | " count=0,\n", 737 | " paginate_width=10000,\n", 738 | ")\n", 739 | "result_df2['timestamp'] = result_df2['timestamp'].astype('float')\n", 740 | "result_df2['rownum'] = result_df2['rownum'].astype('int')" 741 | ] 742 | }, 743 | { 744 | "cell_type": "code", 745 | "execution_count": 19, 746 | "metadata": {}, 747 | "outputs": [ 748 | { 749 | "data": { 750 | "text/plain": [ 751 | "100000" 752 | ] 753 | }, 754 | "execution_count": 19, 755 | "metadata": {}, 756 | "output_type": "execute_result" 757 | } 758 | ], 759 | "source": [ 760 | "len(result_df2)" 761 | ] 762 | }, 763 | { 764 | "attachments": {}, 765 | "cell_type": "markdown", 766 | "metadata": {}, 767 | "source": [ 768 | "### Test with oneshot mode \n", 769 | "\n" 770 | ] 771 | }, 772 | { 773 | "cell_type": "code", 774 | "execution_count": 20, 775 | "metadata": {}, 776 | "outputs": [], 777 | "source": [ 778 | "result_df_oneshot = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 779 | " index=\"msticpy\",\n", 780 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 781 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 782 | " add_query_items='',\n", 783 | " oneshot=True,\n", 784 | " count=0,\n", 785 | ")\n", 786 | "result_df_oneshot['timestamp'] = result_df_oneshot['timestamp'].astype('float')\n", 787 | "result_df_oneshot['rownum'] = result_df_oneshot['rownum'].astype('int')" 788 | ] 789 | }, 790 | { 791 | "cell_type": "code", 792 | "execution_count": 21, 793 | "metadata": {}, 794 | "outputs": [ 795 | { 796 | "data": { 797 | "text/plain": [ 798 | "50000" 799 | ] 800 | }, 801 | "execution_count": 21, 802 | "metadata": {}, 803 | "output_type": "execute_result" 804 | } 805 | ], 806 | "source": [ 807 | "len(result_df_oneshot)" 808 | ] 809 | }, 810 | { 811 | "attachments": {}, 812 | "cell_type": "markdown", 813 | "metadata": {}, 814 | "source": [ 815 | "oneshot mode hits the splunk limit of maxresultrows (50000 by default) !\n", 816 | "\n", 817 | "Points to\n", 818 | "`service.confs[\"limits\"][\"restapi\"][\"maxresultrows\"]`\n", 819 | "\n", 820 | "It's along with my expect." 821 | ] 822 | }, 823 | { 824 | "cell_type": "markdown", 825 | "metadata": {}, 826 | "source": [] 827 | } 828 | ], 829 | "metadata": { 830 | "kernelspec": { 831 | "display_name": "msticpy", 832 | "language": "python", 833 | "name": "python3" 834 | }, 835 | "language_info": { 836 | "codemirror_mode": { 837 | "name": "ipython", 838 | "version": 3 839 | }, 840 | "file_extension": ".py", 841 | "mimetype": "text/x-python", 842 | "name": "python", 843 | "nbconvert_exporter": "python", 844 | "pygments_lexer": "ipython3", 845 | "version": "3.9.12" 846 | }, 847 | "orig_nbformat": 4 848 | }, 849 | "nbformat": 4, 850 | "nbformat_minor": 2 851 | } 852 | -------------------------------------------------------------------------------- /qp_splunk_poc_bugfix/msticpy_splunk_9_0_4_reader_bug.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "### Test for docker splunk latest 9.0.4 " 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 17, 14 | "metadata": {}, 15 | "outputs": [ 16 | { 17 | "name": "stdout", 18 | "output_type": "stream", 19 | "text": [ 20 | "\n", 21 | "\n", 22 | "\n", 23 | "\n", 24 | " splunkd\n", 25 | " https://localhost:8089/\n", 26 | " 2023-04-19T05:13:09+00:00\n", 27 | " \n", 28 | " \n", 29 | " Splunk\n", 30 | " \n", 31 | " \n", 32 | " rpc\n", 33 | " https://localhost:8089/rpc\n", 34 | " 1970-01-01T00:00:00+00:00\n", 35 | " \n", 36 | " \n", 37 | " \n", 38 | " services\n", 39 | " https://localhost:8089/services\n", 40 | " 1970-01-01T00:00:00+00:00\n", 41 | " \n", 42 | " \n", 43 | " \n", 44 | " servicesNS\n", 45 | " https://localhost:8089/servicesNS\n", 46 | " 1970-01-01T00:00:00+00:00\n", 47 | " \n", 48 | " \n", 49 | " \n", 50 | " static\n", 51 | " https://localhost:8089/static\n", 52 | " 1970-01-01T00:00:00+00:00\n", 53 | " \n", 54 | " \n", 55 | "\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "# check the splunk version via REST API\n", 61 | "!curl https://localhost:8089 -k\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 18, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stdout", 71 | "output_type": "stream", 72 | "text": [ 73 | "2.4.0\n" 74 | ] 75 | } 76 | ], 77 | "source": [ 78 | "import msticpy as mp\n", 79 | "print(mp.__version__)" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 19, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "name": "stderr", 89 | "output_type": "stream", 90 | "text": [ 91 | "2023-04-19 14:13:16,294: WARNING - config validation error Missing or empty 'AzureSentinel' section (nbinit#697)\n", 92 | "2023-04-19 14:13:16,294: WARNING - Could not find msticpyconfig.yaml in standard search. (nbinit#710)\n" 93 | ] 94 | }, 95 | { 96 | "data": { 97 | "text/html": [ 98 | "

Notebook setup completed with some warnings.

" 99 | ], 100 | "text/plain": [ 101 | "" 102 | ] 103 | }, 104 | "metadata": {}, 105 | "output_type": "display_data" 106 | }, 107 | { 108 | "data": { 109 | "text/html": [ 110 | "

One or more configuration items were missing or set incorrectly.

" 111 | ], 112 | "text/plain": [ 113 | "" 114 | ] 115 | }, 116 | "metadata": {}, 117 | "output_type": "display_data" 118 | }, 119 | { 120 | "data": { 121 | "text/html": [ 122 | "

Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.

" 123 | ], 124 | "text/plain": [ 125 | "" 126 | ] 127 | }, 128 | "metadata": {}, 129 | "output_type": "display_data" 130 | }, 131 | { 132 | "data": { 133 | "text/html": [ 134 | "

This notebook may still run but with reduced functionality.

" 135 | ], 136 | "text/plain": [ 137 | "" 138 | ] 139 | }, 140 | "metadata": {}, 141 | "output_type": "display_data" 142 | } 143 | ], 144 | "source": [ 145 | "mp.init_notebook()" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 20, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "connected\n" 158 | ] 159 | } 160 | ], 161 | "source": [ 162 | "splunk_prov = mp.QueryProvider(\"Splunk\")\n", 163 | "splunk_prov.connect()" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": 23, 169 | "metadata": {}, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "Query: get_events_parameterized\n", 176 | "Data source: Splunk\n", 177 | "Generic parameterized query from index/source\n", 178 | "\n", 179 | "Parameters\n", 180 | "----------\n", 181 | "add_query_items: str (optional)\n", 182 | " Additional query clauses\n", 183 | " (default value is: | head 100)\n", 184 | "end: datetime\n", 185 | " Query end time\n", 186 | "index: str (optional)\n", 187 | " Splunk index name\n", 188 | " (default value is: *)\n", 189 | "project_fields: str (optional)\n", 190 | " Project Field names\n", 191 | " (default value is: | table TimeCreated, host, EventID, EventDescripti...)\n", 192 | "source: str (optional)\n", 193 | " Splunk source type\n", 194 | " (default value is: *)\n", 195 | "start: datetime\n", 196 | " Query start time\n", 197 | "timeformat: str (optional)\n", 198 | " Datetime format to use in Splunk query\n", 199 | " (default value is: \"%Y-%m-%d %H:%M:%S.%6N\")\n", 200 | "Query:\n", 201 | " search index={index} source={source} timeformat={timeformat} earliest={start} latest={end} {project_fields} {add_query_items}\n" 202 | ] 203 | } 204 | ], 205 | "source": [ 206 | "splunk_prov.SplunkGeneral.get_events_parameterized('?')\n" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": 24, 212 | "metadata": {}, 213 | "outputs": [ 214 | { 215 | "data": { 216 | "text/plain": [ 217 | "' search index=msticpy source=msticpy_splunk_reader_paging-test.csv timeformat=\"%Y-%m-%d %H:%M:%S.%6N\" earliest=\"2023-04-17 00:00:00\" latest=\"2023-04-19 10:00:00\" | table timestamp,rownum, desc, uuid4, host '" 218 | ] 219 | }, 220 | "execution_count": 24, 221 | "metadata": {}, 222 | "output_type": "execute_result" 223 | } 224 | ], 225 | "source": [ 226 | "splunk_prov.SplunkGeneral.get_events_parameterized('print',\n", 227 | " index=\"msticpy\",\n", 228 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 229 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 230 | " start=\"2023-04-17 00:00:00.000000\",\n", 231 | " end=\"2023-04-19 10:00:00.000000\", \n", 232 | " add_query_items='',\n", 233 | " count=0\n", 234 | ")\n", 235 | "\n" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 25, 241 | "metadata": {}, 242 | "outputs": [ 243 | { 244 | "name": "stderr", 245 | "output_type": "stream", 246 | "text": [ 247 | "Waiting Splunk job to complete: 100%|██████████| 100.0/100 [00:01<00:00, 97.67it/s]\n", 248 | "/Users/hacket/.pyenv/versions/3.9.5/lib/python3.9/site-packages/msticpy/data/drivers/splunk_driver.py:234: DeprecationWarning: ResultsReader is a deprecated function. Use the JSONResultsReader function instead in conjuction with the 'output_mode' query param set to 'json'\n", 249 | " reader = sp_results.ResultsReader(query_job.results())\n" 250 | ] 251 | }, 252 | { 253 | "data": { 254 | "text/plain": [ 255 | "100" 256 | ] 257 | }, 258 | "execution_count": 25, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "\n", 265 | "result_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 266 | " index=\"msticpy\",\n", 267 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 268 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 269 | " start=\"2023-04-17 00:00:00.000000\",\n", 270 | " end=\"2023-04-19 10:00:00.000000\", \n", 271 | " add_query_items='',\n", 272 | " count=0\n", 273 | ")\n", 274 | "len(result_df)" 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": 26, 280 | "metadata": {}, 281 | "outputs": [ 282 | { 283 | "data": { 284 | "text/html": [ 285 | "
\n", 286 | "\n", 299 | "\n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | " \n", 381 | " \n", 382 | " \n", 383 | " \n", 384 | " \n", 385 | " \n", 386 | " \n", 387 | " \n", 388 | " \n", 389 | " \n", 390 | " \n", 391 | " \n", 392 | " \n", 393 | " \n", 394 | " \n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 415 | " \n", 416 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | " \n", 448 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 452 | " \n", 453 | " \n", 454 | " \n", 455 | " \n", 456 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 460 | " \n", 461 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 497 | " \n", 498 | " \n", 499 | " \n", 500 | " \n", 501 | " \n", 502 | " \n", 503 | " \n", 504 | " \n", 505 | " \n", 506 | " \n", 507 | " \n", 508 | " \n", 509 | " \n", 510 | " \n", 511 | " \n", 512 | " \n", 513 | " \n", 514 | " \n", 515 | " \n", 516 | " \n", 517 | " \n", 518 | " \n", 519 | " \n", 520 | " \n", 521 | " \n", 522 | " \n", 523 | " \n", 524 | " \n", 525 | " \n", 526 | " \n", 527 | " \n", 528 | " \n", 529 | " \n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | " \n", 534 | " \n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 542 | " \n", 543 | " \n", 544 | " \n", 545 | " \n", 546 | " \n", 547 | " \n", 548 | " \n", 549 | " \n", 550 | " \n", 551 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | " \n", 717 | " \n", 718 | " \n", 719 | " \n", 720 | " \n", 721 | " \n", 722 | " \n", 723 | " \n", 724 | " \n", 725 | " \n", 726 | " \n", 727 | " \n", 728 | " \n", 729 | " \n", 730 | " \n", 731 | " \n", 732 | " \n", 733 | " \n", 734 | " \n", 735 | " \n", 736 | " \n", 737 | " \n", 738 | " \n", 739 | " \n", 740 | " \n", 741 | " \n", 742 | " \n", 743 | " \n", 744 | " \n", 745 | " \n", 746 | " \n", 747 | " \n", 748 | " \n", 749 | " \n", 750 | " \n", 751 | " \n", 752 | " \n", 753 | " \n", 754 | " \n", 755 | " \n", 756 | " \n", 757 | " \n", 758 | " \n", 759 | " \n", 760 | " \n", 761 | " \n", 762 | " \n", 763 | " \n", 764 | " \n", 765 | " \n", 766 | " \n", 767 | " \n", 768 | " \n", 769 | " \n", 770 | " \n", 771 | " \n", 772 | " \n", 773 | " \n", 774 | " \n", 775 | " \n", 776 | " \n", 777 | " \n", 778 | " \n", 779 | " \n", 780 | " \n", 781 | " \n", 782 | " \n", 783 | " \n", 784 | " \n", 785 | " \n", 786 | " \n", 787 | " \n", 788 | " \n", 789 | " \n", 790 | " \n", 791 | " \n", 792 | " \n", 793 | " \n", 794 | " \n", 795 | " \n", 796 | " \n", 797 | " \n", 798 | " \n", 799 | " \n", 800 | " \n", 801 | " \n", 802 | " \n", 803 | " \n", 804 | " \n", 805 | " \n", 806 | " \n", 807 | " \n", 808 | " \n", 809 | " \n", 810 | " \n", 811 | " \n", 812 | " \n", 813 | " \n", 814 | " \n", 815 | " \n", 816 | " \n", 817 | " \n", 818 | " \n", 819 | " \n", 820 | " \n", 821 | " \n", 822 | " \n", 823 | " \n", 824 | " \n", 825 | " \n", 826 | " \n", 827 | " \n", 828 | " \n", 829 | " \n", 830 | " \n", 831 | " \n", 832 | " \n", 833 | " \n", 834 | " \n", 835 | " \n", 836 | " \n", 837 | " \n", 838 | " \n", 839 | " \n", 840 | " \n", 841 | " \n", 842 | " \n", 843 | " \n", 844 | " \n", 845 | " \n", 846 | " \n", 847 | " \n", 848 | " \n", 849 | " \n", 850 | " \n", 851 | " \n", 852 | " \n", 853 | " \n", 854 | " \n", 855 | " \n", 856 | " \n", 857 | " \n", 858 | " \n", 859 | " \n", 860 | " \n", 861 | " \n", 862 | " \n", 863 | " \n", 864 | " \n", 865 | " \n", 866 | " \n", 867 | " \n", 868 | " \n", 869 | " \n", 870 | " \n", 871 | " \n", 872 | " \n", 873 | " \n", 874 | " \n", 875 | " \n", 876 | " \n", 877 | " \n", 878 | " \n", 879 | " \n", 880 | " \n", 881 | " \n", 882 | " \n", 883 | " \n", 884 | " \n", 885 | " \n", 886 | " \n", 887 | " \n", 888 | " \n", 889 | " \n", 890 | " \n", 891 | " \n", 892 | " \n", 893 | " \n", 894 | " \n", 895 | " \n", 896 | " \n", 897 | " \n", 898 | " \n", 899 | " \n", 900 | " \n", 901 | " \n", 902 | " \n", 903 | " \n", 904 | " \n", 905 | " \n", 906 | " \n", 907 | " \n", 908 | " \n", 909 | " \n", 910 | " \n", 911 | " \n", 912 | " \n", 913 | " \n", 914 | " \n", 915 | " \n", 916 | " \n", 917 | " \n", 918 | " \n", 919 | " \n", 920 | " \n", 921 | " \n", 922 | " \n", 923 | " \n", 924 | " \n", 925 | " \n", 926 | " \n", 927 | " \n", 928 | " \n", 929 | " \n", 930 | " \n", 931 | " \n", 932 | " \n", 933 | " \n", 934 | " \n", 935 | " \n", 936 | " \n", 937 | " \n", 938 | " \n", 939 | " \n", 940 | " \n", 941 | " \n", 942 | " \n", 943 | " \n", 944 | " \n", 945 | " \n", 946 | " \n", 947 | " \n", 948 | " \n", 949 | " \n", 950 | " \n", 951 | " \n", 952 | " \n", 953 | " \n", 954 | " \n", 955 | " \n", 956 | " \n", 957 | " \n", 958 | " \n", 959 | " \n", 960 | " \n", 961 | " \n", 962 | " \n", 963 | " \n", 964 | " \n", 965 | " \n", 966 | " \n", 967 | " \n", 968 | " \n", 969 | " \n", 970 | " \n", 971 | " \n", 972 | " \n", 973 | " \n", 974 | " \n", 975 | " \n", 976 | " \n", 977 | " \n", 978 | " \n", 979 | " \n", 980 | " \n", 981 | " \n", 982 | " \n", 983 | " \n", 984 | " \n", 985 | " \n", 986 | " \n", 987 | " \n", 988 | " \n", 989 | " \n", 990 | " \n", 991 | " \n", 992 | " \n", 993 | " \n", 994 | " \n", 995 | " \n", 996 | " \n", 997 | " \n", 998 | " \n", 999 | " \n", 1000 | " \n", 1001 | " \n", 1002 | " \n", 1003 | " \n", 1004 | " \n", 1005 | " \n", 1006 | " \n", 1007 | " \n", 1008 | " \n", 1009 | " \n", 1010 | " \n", 1011 | " \n", 1012 | " \n", 1013 | " \n", 1014 | " \n", 1015 | " \n", 1016 | " \n", 1017 | " \n", 1018 | " \n", 1019 | " \n", 1020 | " \n", 1021 | " \n", 1022 | " \n", 1023 | " \n", 1024 | " \n", 1025 | " \n", 1026 | " \n", 1027 | " \n", 1028 | " \n", 1029 | " \n", 1030 | " \n", 1031 | " \n", 1032 | " \n", 1033 | " \n", 1034 | " \n", 1035 | " \n", 1036 | " \n", 1037 | " \n", 1038 | " \n", 1039 | " \n", 1040 | " \n", 1041 | " \n", 1042 | " \n", 1043 | " \n", 1044 | " \n", 1045 | " \n", 1046 | " \n", 1047 | " \n", 1048 | " \n", 1049 | " \n", 1050 | " \n", 1051 | " \n", 1052 | " \n", 1053 | " \n", 1054 | " \n", 1055 | " \n", 1056 | " \n", 1057 | " \n", 1058 | " \n", 1059 | " \n", 1060 | " \n", 1061 | " \n", 1062 | " \n", 1063 | " \n", 1064 | " \n", 1065 | " \n", 1066 | " \n", 1067 | " \n", 1068 | " \n", 1069 | " \n", 1070 | " \n", 1071 | " \n", 1072 | " \n", 1073 | " \n", 1074 | " \n", 1075 | " \n", 1076 | " \n", 1077 | " \n", 1078 | " \n", 1079 | " \n", 1080 | " \n", 1081 | " \n", 1082 | " \n", 1083 | " \n", 1084 | " \n", 1085 | " \n", 1086 | " \n", 1087 | " \n", 1088 | " \n", 1089 | " \n", 1090 | " \n", 1091 | " \n", 1092 | " \n", 1093 | " \n", 1094 | " \n", 1095 | " \n", 1096 | " \n", 1097 | " \n", 1098 | " \n", 1099 | " \n", 1100 | " \n", 1101 | " \n", 1102 | " \n", 1103 | " \n", 1104 | " \n", 1105 | " \n", 1106 | " \n", 1107 | " \n", 1108 | " \n", 1109 | " \n", 1110 | " \n", 1111 | " \n", 1112 | "
timestamprownumdescuuid4host
01681779752.40650891020testing_rownum91020e6b1e205-ad10-40a9-81da-6de2a48310bfb7ea75f8df3c
11681779752.40516791019testing_rownum910191fdf3802-c17e-4dbd-af9a-5a74beb6dac2b7ea75f8df3c
21681779752.40398391018testing_rownum9101863853418-0e82-453d-a816-d759fdb06f66b7ea75f8df3c
31681779752.40260691017testing_rownum91017f9410562-ef8a-48ce-8a20-0d700c57ceadb7ea75f8df3c
41681779752.40117691016testing_rownum91016a83544f4-3640-48ac-82fa-a69ef43c83f8b7ea75f8df3c
51681779752.39979391015testing_rownum91015abc1f653-9197-48af-9c51-de3ec4cbdf60b7ea75f8df3c
61681779752.39840391014testing_rownum9101473db45e8-498b-423a-99d4-93a8987f4383b7ea75f8df3c
71681779752.39714991013testing_rownum91013dcda1ec7-2aee-47e1-83f6-380f757ac274b7ea75f8df3c
81681779752.3957391012testing_rownum910125c78d913-36b7-4ea2-a8e1-d2f365c034bbb7ea75f8df3c
91681779752.39434591011testing_rownum91011a6089f12-9447-4098-8feb-2d1c90130ffdb7ea75f8df3c
101681779752.39295391010testing_rownum910109285c292-a454-41e8-a4e9-87b1ba27812eb7ea75f8df3c
111681779752.39152391009testing_rownum9100978d3b3cf-494e-450e-ae3b-5270e8c29a09b7ea75f8df3c
121681779752.39014191008testing_rownum91008ea055d48-7fa1-4233-b20b-d59dfdb6608eb7ea75f8df3c
131681779752.38875691007testing_rownum910075dd307fe-590e-4e34-b9c7-14fe264621b6b7ea75f8df3c
141681779752.38736891006testing_rownum910060bbc9c1d-e960-4669-9533-2eeb9f44368db7ea75f8df3c
151681779752.38598191005testing_rownum91005ea630193-d9e9-4b6f-ab53-2176ddcc58f6b7ea75f8df3c
161681779752.38458191004testing_rownum910041461a016-4a56-4888-9a3e-e78d49856c4ab7ea75f8df3c
171681779752.38317891003testing_rownum9100352317b49-f8cc-4218-aa35-a47ecd62e568b7ea75f8df3c
181681779752.38179491002testing_rownum910026799ef50-122c-4c73-bb70-bb55ac67de66b7ea75f8df3c
191681779752.38039391001testing_rownum910018c71a674-ca0c-4de9-bb39-2b246dac630fb7ea75f8df3c
201681779752.37899291000testing_rownum9100033c9cfb0-d264-44e9-b173-5020beb7b3feb7ea75f8df3c
211681779752.37760790999testing_rownum90999a535250e-3c1d-46c7-a1c3-cdebdfee7b24b7ea75f8df3c
221681779752.37622190998testing_rownum9099838b94fef-71b5-4172-99a8-2706ee362330b7ea75f8df3c
231681779752.37477790997testing_rownum9099711003d0b-e3a0-416c-af9a-eb8e1235f73eb7ea75f8df3c
241681779752.37347290996testing_rownum9099633860324-3eea-4839-9c5a-7cf166de2110b7ea75f8df3c
251681779752.37209790995testing_rownum90995ba436b9d-9082-4411-b603-658f810677aeb7ea75f8df3c
261681779752.37082290994testing_rownum909944a4dac63-c6bc-46a1-8ad8-43c0b68e9a74b7ea75f8df3c
271681779752.36940290993testing_rownum909938cec26a2-404e-43b7-8e1e-c672adb41cabb7ea75f8df3c
281681779752.36804590992testing_rownum909924e48d246-2a97-4892-bf7a-f9b6b0230ee7b7ea75f8df3c
291681779752.36681790991testing_rownum909916cc13611-669a-4ac6-a1a7-0f1ec2559d0eb7ea75f8df3c
301681779752.36542990990testing_rownum909906bbbafda-4bbe-40f0-b46a-11186cf12e7eb7ea75f8df3c
311681779752.36398890989testing_rownum90989d3aed9f7-bc7f-4d41-b4d2-0598a0af2deeb7ea75f8df3c
321681779752.36255490988testing_rownum909887068ee2c-25c2-44a1-bff3-f62c3edb2cd2b7ea75f8df3c
331681779752.36112690987testing_rownum90987ec051770-480e-4547-97c8-499cd48bafe6b7ea75f8df3c
341681779752.35968290986testing_rownum9098626c41209-b5a0-475c-8cc9-33e3ca0e9da9b7ea75f8df3c
351681779752.35829790985testing_rownum909859acc54f9-bec8-4569-b553-d0d428fb9d58b7ea75f8df3c
361681779752.3569290984testing_rownum90984958216ac-9583-4428-b1d8-f565596a833fb7ea75f8df3c
371681779752.35561490983testing_rownum90983f9128fe4-cd8c-4556-b660-a0ae9c542262b7ea75f8df3c
381681779752.35420890982testing_rownum90982460c77e6-dd96-4b42-be80-60a1e86427d5b7ea75f8df3c
391681779752.35280890981testing_rownum909810f240d33-23d5-4320-b7a4-073dc2b8f361b7ea75f8df3c
401681779752.35142490980testing_rownum90980687b894e-0961-4a9d-bfaf-ad7bac60468ab7ea75f8df3c
411681779752.35003890979testing_rownum90979970bf2e6-e344-4e1a-a8de-d289ecb85a36b7ea75f8df3c
421681779752.34862390978testing_rownum909782b9cf05a-f8c1-4da1-80f6-93204be6c2d8b7ea75f8df3c
431681779752.34720590977testing_rownum90977c9d59855-0363-45d4-8cff-c72d631e4b99b7ea75f8df3c
441681779752.34580590976testing_rownum909764c39cb21-a981-4802-bee3-04a44a0d48e3b7ea75f8df3c
451681779752.34441790975testing_rownum9097585214d7e-b933-4081-af84-10c0c5b8e37db7ea75f8df3c
461681779752.34303390974testing_rownum90974ec60e11b-8e08-437c-8c73-cb0e8bd4dc41b7ea75f8df3c
471681779752.34163390973testing_rownum90973e1224ee0-a8cb-46d4-ae9b-b9daab5e394fb7ea75f8df3c
481681779752.34022990972testing_rownum909728f7d1802-743d-4d4a-8a23-860e2613055fb7ea75f8df3c
491681779752.33883590971testing_rownum909713e96d2fb-e771-4b47-8e81-1cab2e9b6271b7ea75f8df3c
501681779752.33744890970testing_rownum9097077b6440d-c899-4ff1-9a7f-a8d77f485ed2b7ea75f8df3c
511681779752.33606590969testing_rownum90969a7d68e3d-a78f-407c-827d-362b789257b5b7ea75f8df3c
521681779752.33468390968testing_rownum9096878a0421c-99f6-40cd-a903-1ea72464b0ceb7ea75f8df3c
531681779752.33344690967testing_rownum90967505b326f-997d-446d-ad04-ceb0d98a8708b7ea75f8df3c
541681779752.33221690966testing_rownum90966adbfbba2-9415-4381-8c05-4d34bdcc914ab7ea75f8df3c
551681779752.33078990965testing_rownum909653517ba74-2e00-4bf2-a42b-20e0d764b811b7ea75f8df3c
561681779752.32940590964testing_rownum9096495f3c4a0-f0c6-44be-88ec-8cad8cc1fbf2b7ea75f8df3c
571681779752.32807790963testing_rownum90963ad693222-f6ee-4f45-8b37-6a78bc72a7d6b7ea75f8df3c
581681779752.32674890962testing_rownum9096201b6f4f2-db3f-48a9-b0ea-ded53ecc9bb0b7ea75f8df3c
591681779752.32538990961testing_rownum909612218a2ee-3838-4fed-83f3-f571d7330c1fb7ea75f8df3c
601681779752.3240190960testing_rownum909601fdeb5de-8c16-4e31-a19c-0ece227f75a0b7ea75f8df3c
611681779752.32284190959testing_rownum909593d4a5628-f467-4a54-8966-f30d057d1060b7ea75f8df3c
621681779752.32163690958testing_rownum909589d4cedfa-70db-4710-88f3-9b1f17238549b7ea75f8df3c
631681779752.32040390957testing_rownum90957eec23ddf-0d2e-41b3-b387-b1c7e09606b4b7ea75f8df3c
641681779752.31905490956testing_rownum9095601762a20-edb8-4531-8b47-feedbe22b565b7ea75f8df3c
651681779752.31763790955testing_rownum909555d9c83ea-5571-4248-a280-a084f21216e7b7ea75f8df3c
661681779752.31627490954testing_rownum9095460c1af52-657d-4c46-baa5-4df7d105264ab7ea75f8df3c
671681779752.31490690953testing_rownum9095312ab0219-f3a4-4581-a3e5-3a8703483a0db7ea75f8df3c
681681779752.31352790952testing_rownum909529b346382-e195-4985-828f-1aa4e6c4a109b7ea75f8df3c
691681779752.31231190951testing_rownum9095184d200f7-61d2-4ee6-a4ca-969c7056040ab7ea75f8df3c
701681779752.3110790950testing_rownum9095098ae2425-697f-46f5-8b80-0cb6d53e904ab7ea75f8df3c
711681779752.30963690949testing_rownum909498152b663-9cbe-44f2-a499-1d1c8328398eb7ea75f8df3c
721681779752.30825390948testing_rownum90948e16ceff5-f9d7-45f8-8aa9-702930a52d4bb7ea75f8df3c
731681779752.30683890947testing_rownum909476c821b9b-91d8-427e-9fc2-aa4b493b2fe5b7ea75f8df3c
741681779752.30543590946testing_rownum90946da396819-e679-4d53-bacc-a39e1c812996b7ea75f8df3c
751681779752.30403390945testing_rownum9094539df1d2b-77ce-40d0-8aa7-eaa58716089eb7ea75f8df3c
761681779752.30264990944testing_rownum9094432efbd2f-ffe9-460b-91af-b51fac459b1cb7ea75f8df3c
771681779752.30126890943testing_rownum9094392895835-df19-4737-8037-b86cb7d90fb3b7ea75f8df3c
781681779752.3001190942testing_rownum90942a00db8ab-393e-4c50-82c4-c1710cda362bb7ea75f8df3c
791681779752.29881490941testing_rownum9094139f9b0b8-b736-4bff-9ee4-6f1b183374a2b7ea75f8df3c
801681779752.29745290940testing_rownum9094088b32f13-b0f5-4343-95e1-f187c8096493b7ea75f8df3c
811681779752.29614690939testing_rownum909391533db32-9a9b-45cd-9cb6-0fc362e77f83b7ea75f8df3c
821681779752.29478690938testing_rownum90938e12176e7-96db-48c3-af63-18d35c437792b7ea75f8df3c
831681779752.29340390937testing_rownum9093738b765f9-2d39-40d0-bd1a-d5ac88a1ad65b7ea75f8df3c
841681779752.292290936testing_rownum9093616fad467-4fed-41b1-8023-8191415a1e77b7ea75f8df3c
851681779752.290890935testing_rownum90935b91fcd81-e115-4da5-b9cf-159df92fd4aeb7ea75f8df3c
861681779752.28933390934testing_rownum90934b37f9cda-cdeb-4eb7-a987-38acb30dca7bb7ea75f8df3c
871681779752.28794990933testing_rownum909331407d30e-829d-4532-89b5-19883a1b06b8b7ea75f8df3c
881681779752.28658690932testing_rownum9093274674bb6-092d-4063-a652-09dd7acb3614b7ea75f8df3c
891681779752.2852290931testing_rownum909317cb2e4d8-8ba1-439a-b5d1-8d350892b1d7b7ea75f8df3c
901681779752.28385190930testing_rownum909300ae0dac1-ee25-4132-b0b1-ba32ef035a86b7ea75f8df3c
911681779752.28242890929testing_rownum9092920d37721-a2f4-4a1e-856a-372fa0c494c8b7ea75f8df3c
921681779752.28110890928testing_rownum9092823bc07f9-5347-4e18-9859-f8c27c139807b7ea75f8df3c
931681779752.27981590927testing_rownum90927efc6ada0-ed11-497d-974d-c9d98af60bbfb7ea75f8df3c
941681779752.27849590926testing_rownum909262290da59-697e-418f-ab36-bedd2de3ce29b7ea75f8df3c
951681779752.27737490925testing_rownum90925df80d2d1-3cd2-42c3-a769-490d4b5f4d97b7ea75f8df3c
961681779752.27606590924testing_rownum90924db8ffe19-7d18-4529-9ed7-4484e4215e2bb7ea75f8df3c
971681779752.2747590923testing_rownum90923d069dacd-b1e1-4ffe-bec7-16b9345863cfb7ea75f8df3c
981681779752.27349690922testing_rownum909225aec83cf-ff0b-47c5-bc0f-2241a879e1aab7ea75f8df3c
991681779752.27217290921testing_rownum909214c1d9d50-1844-42e0-acd7-77aa5088b968b7ea75f8df3c
\n", 1113 | "
" 1114 | ], 1115 | "text/plain": [ 1116 | " timestamp rownum desc \\\n", 1117 | "0 1681779752.406508 91020 testing_rownum91020 \n", 1118 | "1 1681779752.405167 91019 testing_rownum91019 \n", 1119 | "2 1681779752.403983 91018 testing_rownum91018 \n", 1120 | "3 1681779752.402606 91017 testing_rownum91017 \n", 1121 | "4 1681779752.401176 91016 testing_rownum91016 \n", 1122 | "5 1681779752.399793 91015 testing_rownum91015 \n", 1123 | "6 1681779752.398403 91014 testing_rownum91014 \n", 1124 | "7 1681779752.397149 91013 testing_rownum91013 \n", 1125 | "8 1681779752.39573 91012 testing_rownum91012 \n", 1126 | "9 1681779752.394345 91011 testing_rownum91011 \n", 1127 | "10 1681779752.392953 91010 testing_rownum91010 \n", 1128 | "11 1681779752.391523 91009 testing_rownum91009 \n", 1129 | "12 1681779752.390141 91008 testing_rownum91008 \n", 1130 | "13 1681779752.388756 91007 testing_rownum91007 \n", 1131 | "14 1681779752.387368 91006 testing_rownum91006 \n", 1132 | "15 1681779752.385981 91005 testing_rownum91005 \n", 1133 | "16 1681779752.384581 91004 testing_rownum91004 \n", 1134 | "17 1681779752.383178 91003 testing_rownum91003 \n", 1135 | "18 1681779752.381794 91002 testing_rownum91002 \n", 1136 | "19 1681779752.380393 91001 testing_rownum91001 \n", 1137 | "20 1681779752.378992 91000 testing_rownum91000 \n", 1138 | "21 1681779752.377607 90999 testing_rownum90999 \n", 1139 | "22 1681779752.376221 90998 testing_rownum90998 \n", 1140 | "23 1681779752.374777 90997 testing_rownum90997 \n", 1141 | "24 1681779752.373472 90996 testing_rownum90996 \n", 1142 | "25 1681779752.372097 90995 testing_rownum90995 \n", 1143 | "26 1681779752.370822 90994 testing_rownum90994 \n", 1144 | "27 1681779752.369402 90993 testing_rownum90993 \n", 1145 | "28 1681779752.368045 90992 testing_rownum90992 \n", 1146 | "29 1681779752.366817 90991 testing_rownum90991 \n", 1147 | "30 1681779752.365429 90990 testing_rownum90990 \n", 1148 | "31 1681779752.363988 90989 testing_rownum90989 \n", 1149 | "32 1681779752.362554 90988 testing_rownum90988 \n", 1150 | "33 1681779752.361126 90987 testing_rownum90987 \n", 1151 | "34 1681779752.359682 90986 testing_rownum90986 \n", 1152 | "35 1681779752.358297 90985 testing_rownum90985 \n", 1153 | "36 1681779752.35692 90984 testing_rownum90984 \n", 1154 | "37 1681779752.355614 90983 testing_rownum90983 \n", 1155 | "38 1681779752.354208 90982 testing_rownum90982 \n", 1156 | "39 1681779752.352808 90981 testing_rownum90981 \n", 1157 | "40 1681779752.351424 90980 testing_rownum90980 \n", 1158 | "41 1681779752.350038 90979 testing_rownum90979 \n", 1159 | "42 1681779752.348623 90978 testing_rownum90978 \n", 1160 | "43 1681779752.347205 90977 testing_rownum90977 \n", 1161 | "44 1681779752.345805 90976 testing_rownum90976 \n", 1162 | "45 1681779752.344417 90975 testing_rownum90975 \n", 1163 | "46 1681779752.343033 90974 testing_rownum90974 \n", 1164 | "47 1681779752.341633 90973 testing_rownum90973 \n", 1165 | "48 1681779752.340229 90972 testing_rownum90972 \n", 1166 | "49 1681779752.338835 90971 testing_rownum90971 \n", 1167 | "50 1681779752.337448 90970 testing_rownum90970 \n", 1168 | "51 1681779752.336065 90969 testing_rownum90969 \n", 1169 | "52 1681779752.334683 90968 testing_rownum90968 \n", 1170 | "53 1681779752.333446 90967 testing_rownum90967 \n", 1171 | "54 1681779752.332216 90966 testing_rownum90966 \n", 1172 | "55 1681779752.330789 90965 testing_rownum90965 \n", 1173 | "56 1681779752.329405 90964 testing_rownum90964 \n", 1174 | "57 1681779752.328077 90963 testing_rownum90963 \n", 1175 | "58 1681779752.326748 90962 testing_rownum90962 \n", 1176 | "59 1681779752.325389 90961 testing_rownum90961 \n", 1177 | "60 1681779752.32401 90960 testing_rownum90960 \n", 1178 | "61 1681779752.322841 90959 testing_rownum90959 \n", 1179 | "62 1681779752.321636 90958 testing_rownum90958 \n", 1180 | "63 1681779752.320403 90957 testing_rownum90957 \n", 1181 | "64 1681779752.319054 90956 testing_rownum90956 \n", 1182 | "65 1681779752.317637 90955 testing_rownum90955 \n", 1183 | "66 1681779752.316274 90954 testing_rownum90954 \n", 1184 | "67 1681779752.314906 90953 testing_rownum90953 \n", 1185 | "68 1681779752.313527 90952 testing_rownum90952 \n", 1186 | "69 1681779752.312311 90951 testing_rownum90951 \n", 1187 | "70 1681779752.31107 90950 testing_rownum90950 \n", 1188 | "71 1681779752.309636 90949 testing_rownum90949 \n", 1189 | "72 1681779752.308253 90948 testing_rownum90948 \n", 1190 | "73 1681779752.306838 90947 testing_rownum90947 \n", 1191 | "74 1681779752.305435 90946 testing_rownum90946 \n", 1192 | "75 1681779752.304033 90945 testing_rownum90945 \n", 1193 | "76 1681779752.302649 90944 testing_rownum90944 \n", 1194 | "77 1681779752.301268 90943 testing_rownum90943 \n", 1195 | "78 1681779752.30011 90942 testing_rownum90942 \n", 1196 | "79 1681779752.298814 90941 testing_rownum90941 \n", 1197 | "80 1681779752.297452 90940 testing_rownum90940 \n", 1198 | "81 1681779752.296146 90939 testing_rownum90939 \n", 1199 | "82 1681779752.294786 90938 testing_rownum90938 \n", 1200 | "83 1681779752.293403 90937 testing_rownum90937 \n", 1201 | "84 1681779752.2922 90936 testing_rownum90936 \n", 1202 | "85 1681779752.2908 90935 testing_rownum90935 \n", 1203 | "86 1681779752.289333 90934 testing_rownum90934 \n", 1204 | "87 1681779752.287949 90933 testing_rownum90933 \n", 1205 | "88 1681779752.286586 90932 testing_rownum90932 \n", 1206 | "89 1681779752.28522 90931 testing_rownum90931 \n", 1207 | "90 1681779752.283851 90930 testing_rownum90930 \n", 1208 | "91 1681779752.282428 90929 testing_rownum90929 \n", 1209 | "92 1681779752.281108 90928 testing_rownum90928 \n", 1210 | "93 1681779752.279815 90927 testing_rownum90927 \n", 1211 | "94 1681779752.278495 90926 testing_rownum90926 \n", 1212 | "95 1681779752.277374 90925 testing_rownum90925 \n", 1213 | "96 1681779752.276065 90924 testing_rownum90924 \n", 1214 | "97 1681779752.27475 90923 testing_rownum90923 \n", 1215 | "98 1681779752.273496 90922 testing_rownum90922 \n", 1216 | "99 1681779752.272172 90921 testing_rownum90921 \n", 1217 | "\n", 1218 | " uuid4 host \n", 1219 | "0 e6b1e205-ad10-40a9-81da-6de2a48310bf b7ea75f8df3c \n", 1220 | "1 1fdf3802-c17e-4dbd-af9a-5a74beb6dac2 b7ea75f8df3c \n", 1221 | "2 63853418-0e82-453d-a816-d759fdb06f66 b7ea75f8df3c \n", 1222 | "3 f9410562-ef8a-48ce-8a20-0d700c57cead b7ea75f8df3c \n", 1223 | "4 a83544f4-3640-48ac-82fa-a69ef43c83f8 b7ea75f8df3c \n", 1224 | "5 abc1f653-9197-48af-9c51-de3ec4cbdf60 b7ea75f8df3c \n", 1225 | "6 73db45e8-498b-423a-99d4-93a8987f4383 b7ea75f8df3c \n", 1226 | "7 dcda1ec7-2aee-47e1-83f6-380f757ac274 b7ea75f8df3c \n", 1227 | "8 5c78d913-36b7-4ea2-a8e1-d2f365c034bb b7ea75f8df3c \n", 1228 | "9 a6089f12-9447-4098-8feb-2d1c90130ffd b7ea75f8df3c \n", 1229 | "10 9285c292-a454-41e8-a4e9-87b1ba27812e b7ea75f8df3c \n", 1230 | "11 78d3b3cf-494e-450e-ae3b-5270e8c29a09 b7ea75f8df3c \n", 1231 | "12 ea055d48-7fa1-4233-b20b-d59dfdb6608e b7ea75f8df3c \n", 1232 | "13 5dd307fe-590e-4e34-b9c7-14fe264621b6 b7ea75f8df3c \n", 1233 | "14 0bbc9c1d-e960-4669-9533-2eeb9f44368d b7ea75f8df3c \n", 1234 | "15 ea630193-d9e9-4b6f-ab53-2176ddcc58f6 b7ea75f8df3c \n", 1235 | "16 1461a016-4a56-4888-9a3e-e78d49856c4a b7ea75f8df3c \n", 1236 | "17 52317b49-f8cc-4218-aa35-a47ecd62e568 b7ea75f8df3c \n", 1237 | "18 6799ef50-122c-4c73-bb70-bb55ac67de66 b7ea75f8df3c \n", 1238 | "19 8c71a674-ca0c-4de9-bb39-2b246dac630f b7ea75f8df3c \n", 1239 | "20 33c9cfb0-d264-44e9-b173-5020beb7b3fe b7ea75f8df3c \n", 1240 | "21 a535250e-3c1d-46c7-a1c3-cdebdfee7b24 b7ea75f8df3c \n", 1241 | "22 38b94fef-71b5-4172-99a8-2706ee362330 b7ea75f8df3c \n", 1242 | "23 11003d0b-e3a0-416c-af9a-eb8e1235f73e b7ea75f8df3c \n", 1243 | "24 33860324-3eea-4839-9c5a-7cf166de2110 b7ea75f8df3c \n", 1244 | "25 ba436b9d-9082-4411-b603-658f810677ae b7ea75f8df3c \n", 1245 | "26 4a4dac63-c6bc-46a1-8ad8-43c0b68e9a74 b7ea75f8df3c \n", 1246 | "27 8cec26a2-404e-43b7-8e1e-c672adb41cab b7ea75f8df3c \n", 1247 | "28 4e48d246-2a97-4892-bf7a-f9b6b0230ee7 b7ea75f8df3c \n", 1248 | "29 6cc13611-669a-4ac6-a1a7-0f1ec2559d0e b7ea75f8df3c \n", 1249 | "30 6bbbafda-4bbe-40f0-b46a-11186cf12e7e b7ea75f8df3c \n", 1250 | "31 d3aed9f7-bc7f-4d41-b4d2-0598a0af2dee b7ea75f8df3c \n", 1251 | "32 7068ee2c-25c2-44a1-bff3-f62c3edb2cd2 b7ea75f8df3c \n", 1252 | "33 ec051770-480e-4547-97c8-499cd48bafe6 b7ea75f8df3c \n", 1253 | "34 26c41209-b5a0-475c-8cc9-33e3ca0e9da9 b7ea75f8df3c \n", 1254 | "35 9acc54f9-bec8-4569-b553-d0d428fb9d58 b7ea75f8df3c \n", 1255 | "36 958216ac-9583-4428-b1d8-f565596a833f b7ea75f8df3c \n", 1256 | "37 f9128fe4-cd8c-4556-b660-a0ae9c542262 b7ea75f8df3c \n", 1257 | "38 460c77e6-dd96-4b42-be80-60a1e86427d5 b7ea75f8df3c \n", 1258 | "39 0f240d33-23d5-4320-b7a4-073dc2b8f361 b7ea75f8df3c \n", 1259 | "40 687b894e-0961-4a9d-bfaf-ad7bac60468a b7ea75f8df3c \n", 1260 | "41 970bf2e6-e344-4e1a-a8de-d289ecb85a36 b7ea75f8df3c \n", 1261 | "42 2b9cf05a-f8c1-4da1-80f6-93204be6c2d8 b7ea75f8df3c \n", 1262 | "43 c9d59855-0363-45d4-8cff-c72d631e4b99 b7ea75f8df3c \n", 1263 | "44 4c39cb21-a981-4802-bee3-04a44a0d48e3 b7ea75f8df3c \n", 1264 | "45 85214d7e-b933-4081-af84-10c0c5b8e37d b7ea75f8df3c \n", 1265 | "46 ec60e11b-8e08-437c-8c73-cb0e8bd4dc41 b7ea75f8df3c \n", 1266 | "47 e1224ee0-a8cb-46d4-ae9b-b9daab5e394f b7ea75f8df3c \n", 1267 | "48 8f7d1802-743d-4d4a-8a23-860e2613055f b7ea75f8df3c \n", 1268 | "49 3e96d2fb-e771-4b47-8e81-1cab2e9b6271 b7ea75f8df3c \n", 1269 | "50 77b6440d-c899-4ff1-9a7f-a8d77f485ed2 b7ea75f8df3c \n", 1270 | "51 a7d68e3d-a78f-407c-827d-362b789257b5 b7ea75f8df3c \n", 1271 | "52 78a0421c-99f6-40cd-a903-1ea72464b0ce b7ea75f8df3c \n", 1272 | "53 505b326f-997d-446d-ad04-ceb0d98a8708 b7ea75f8df3c \n", 1273 | "54 adbfbba2-9415-4381-8c05-4d34bdcc914a b7ea75f8df3c \n", 1274 | "55 3517ba74-2e00-4bf2-a42b-20e0d764b811 b7ea75f8df3c \n", 1275 | "56 95f3c4a0-f0c6-44be-88ec-8cad8cc1fbf2 b7ea75f8df3c \n", 1276 | "57 ad693222-f6ee-4f45-8b37-6a78bc72a7d6 b7ea75f8df3c \n", 1277 | "58 01b6f4f2-db3f-48a9-b0ea-ded53ecc9bb0 b7ea75f8df3c \n", 1278 | "59 2218a2ee-3838-4fed-83f3-f571d7330c1f b7ea75f8df3c \n", 1279 | "60 1fdeb5de-8c16-4e31-a19c-0ece227f75a0 b7ea75f8df3c \n", 1280 | "61 3d4a5628-f467-4a54-8966-f30d057d1060 b7ea75f8df3c \n", 1281 | "62 9d4cedfa-70db-4710-88f3-9b1f17238549 b7ea75f8df3c \n", 1282 | "63 eec23ddf-0d2e-41b3-b387-b1c7e09606b4 b7ea75f8df3c \n", 1283 | "64 01762a20-edb8-4531-8b47-feedbe22b565 b7ea75f8df3c \n", 1284 | "65 5d9c83ea-5571-4248-a280-a084f21216e7 b7ea75f8df3c \n", 1285 | "66 60c1af52-657d-4c46-baa5-4df7d105264a b7ea75f8df3c \n", 1286 | "67 12ab0219-f3a4-4581-a3e5-3a8703483a0d b7ea75f8df3c \n", 1287 | "68 9b346382-e195-4985-828f-1aa4e6c4a109 b7ea75f8df3c \n", 1288 | "69 84d200f7-61d2-4ee6-a4ca-969c7056040a b7ea75f8df3c \n", 1289 | "70 98ae2425-697f-46f5-8b80-0cb6d53e904a b7ea75f8df3c \n", 1290 | "71 8152b663-9cbe-44f2-a499-1d1c8328398e b7ea75f8df3c \n", 1291 | "72 e16ceff5-f9d7-45f8-8aa9-702930a52d4b b7ea75f8df3c \n", 1292 | "73 6c821b9b-91d8-427e-9fc2-aa4b493b2fe5 b7ea75f8df3c \n", 1293 | "74 da396819-e679-4d53-bacc-a39e1c812996 b7ea75f8df3c \n", 1294 | "75 39df1d2b-77ce-40d0-8aa7-eaa58716089e b7ea75f8df3c \n", 1295 | "76 32efbd2f-ffe9-460b-91af-b51fac459b1c b7ea75f8df3c \n", 1296 | "77 92895835-df19-4737-8037-b86cb7d90fb3 b7ea75f8df3c \n", 1297 | "78 a00db8ab-393e-4c50-82c4-c1710cda362b b7ea75f8df3c \n", 1298 | "79 39f9b0b8-b736-4bff-9ee4-6f1b183374a2 b7ea75f8df3c \n", 1299 | "80 88b32f13-b0f5-4343-95e1-f187c8096493 b7ea75f8df3c \n", 1300 | "81 1533db32-9a9b-45cd-9cb6-0fc362e77f83 b7ea75f8df3c \n", 1301 | "82 e12176e7-96db-48c3-af63-18d35c437792 b7ea75f8df3c \n", 1302 | "83 38b765f9-2d39-40d0-bd1a-d5ac88a1ad65 b7ea75f8df3c \n", 1303 | "84 16fad467-4fed-41b1-8023-8191415a1e77 b7ea75f8df3c \n", 1304 | "85 b91fcd81-e115-4da5-b9cf-159df92fd4ae b7ea75f8df3c \n", 1305 | "86 b37f9cda-cdeb-4eb7-a987-38acb30dca7b b7ea75f8df3c \n", 1306 | "87 1407d30e-829d-4532-89b5-19883a1b06b8 b7ea75f8df3c \n", 1307 | "88 74674bb6-092d-4063-a652-09dd7acb3614 b7ea75f8df3c \n", 1308 | "89 7cb2e4d8-8ba1-439a-b5d1-8d350892b1d7 b7ea75f8df3c \n", 1309 | "90 0ae0dac1-ee25-4132-b0b1-ba32ef035a86 b7ea75f8df3c \n", 1310 | "91 20d37721-a2f4-4a1e-856a-372fa0c494c8 b7ea75f8df3c \n", 1311 | "92 23bc07f9-5347-4e18-9859-f8c27c139807 b7ea75f8df3c \n", 1312 | "93 efc6ada0-ed11-497d-974d-c9d98af60bbf b7ea75f8df3c \n", 1313 | "94 2290da59-697e-418f-ab36-bedd2de3ce29 b7ea75f8df3c \n", 1314 | "95 df80d2d1-3cd2-42c3-a769-490d4b5f4d97 b7ea75f8df3c \n", 1315 | "96 db8ffe19-7d18-4529-9ed7-4484e4215e2b b7ea75f8df3c \n", 1316 | "97 d069dacd-b1e1-4ffe-bec7-16b9345863cf b7ea75f8df3c \n", 1317 | "98 5aec83cf-ff0b-47c5-bc0f-2241a879e1aa b7ea75f8df3c \n", 1318 | "99 4c1d9d50-1844-42e0-acd7-77aa5088b968 b7ea75f8df3c " 1319 | ] 1320 | }, 1321 | "execution_count": 26, 1322 | "metadata": {}, 1323 | "output_type": "execute_result" 1324 | } 1325 | ], 1326 | "source": [ 1327 | "result_df" 1328 | ] 1329 | }, 1330 | { 1331 | "cell_type": "code", 1332 | "execution_count": null, 1333 | "metadata": {}, 1334 | "outputs": [], 1335 | "source": [] 1336 | } 1337 | ], 1338 | "metadata": { 1339 | "kernelspec": { 1340 | "display_name": "Python 3", 1341 | "language": "python", 1342 | "name": "python3" 1343 | }, 1344 | "language_info": { 1345 | "codemirror_mode": { 1346 | "name": "ipython", 1347 | "version": 3 1348 | }, 1349 | "file_extension": ".py", 1350 | "mimetype": "text/x-python", 1351 | "name": "python", 1352 | "nbconvert_exporter": "python", 1353 | "pygments_lexer": "ipython3", 1354 | "version": "3.9.5" 1355 | }, 1356 | "orig_nbformat": 4 1357 | }, 1358 | "nbformat": 4, 1359 | "nbformat_minor": 2 1360 | } 1361 | -------------------------------------------------------------------------------- /qp_splunk_poc_bugfix/msticpy_splunk_9_0_4_reader_merged.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "### Test for docker Splunk latest 9.0.4" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 13, 14 | "metadata": {}, 15 | "outputs": [ 16 | { 17 | "name": "stdout", 18 | "output_type": "stream", 19 | "text": [ 20 | "Tue Apr 25 05:41:04 JST 2023\n", 21 | "Python 3.9.5\n", 22 | "sys.version\n", 23 | " 3.9.5 (default, Jun 7 2021, 16:27:44) \n", 24 | "[Clang 12.0.5 (clang-1205.0.22.9)]\n" 25 | ] 26 | } 27 | ], 28 | "source": [ 29 | "!date\n", 30 | "!python -V # check if python version is later than 3.8\n", 31 | "\n", 32 | "import sys\n", 33 | "print(\"sys.version\\n\",sys.version)" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 14, 39 | "metadata": {}, 40 | "outputs": [ 41 | { 42 | "name": "stdout", 43 | "output_type": "stream", 44 | "text": [ 45 | "\n", 46 | "\n", 47 | "\n", 48 | "\n", 49 | " splunkd\n", 50 | " https://localhost:8089/\n", 51 | " 2023-04-24T20:41:06+00:00\n", 52 | " \n", 53 | " \n", 54 | " Splunk\n", 55 | " \n", 56 | " \n", 57 | " rpc\n", 58 | " https://localhost:8089/rpc\n", 59 | " 1970-01-01T00:00:00+00:00\n", 60 | " \n", 61 | " \n", 62 | " \n", 63 | " services\n", 64 | " https://localhost:8089/services\n", 65 | " 1970-01-01T00:00:00+00:00\n", 66 | " \n", 67 | " \n", 68 | " \n", 69 | " servicesNS\n", 70 | " https://localhost:8089/servicesNS\n", 71 | " 1970-01-01T00:00:00+00:00\n", 72 | " \n", 73 | " \n", 74 | " \n", 75 | " static\n", 76 | " https://localhost:8089/static\n", 77 | " 1970-01-01T00:00:00+00:00\n", 78 | " \n", 79 | " \n", 80 | "\n" 81 | ] 82 | } 83 | ], 84 | "source": [ 85 | "\n", 86 | "# check the splunk version via REST API\n", 87 | "!curl https://localhost:8089 -k\n" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 16, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "name": "stdout", 97 | "output_type": "stream", 98 | "text": [ 99 | "2.4.0\n" 100 | ] 101 | } 102 | ], 103 | "source": [ 104 | "import msticpy as mp\n", 105 | "print(mp.__version__)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 17, 111 | "metadata": {}, 112 | "outputs": [ 113 | { 114 | "name": "stderr", 115 | "output_type": "stream", 116 | "text": [ 117 | "2023-04-25 05:41:31,410: WARNING - config validation error Missing or empty 'AzureSentinel' section (nbinit#697)\n", 118 | "2023-04-25 05:41:31,410: WARNING - Could not find msticpyconfig.yaml in standard search. (nbinit#710)\n" 119 | ] 120 | }, 121 | { 122 | "data": { 123 | "text/html": [ 124 | "

Notebook setup completed with some warnings.

" 125 | ], 126 | "text/plain": [ 127 | "" 128 | ] 129 | }, 130 | "metadata": {}, 131 | "output_type": "display_data" 132 | }, 133 | { 134 | "data": { 135 | "text/html": [ 136 | "

One or more configuration items were missing or set incorrectly.

" 137 | ], 138 | "text/plain": [ 139 | "" 140 | ] 141 | }, 142 | "metadata": {}, 143 | "output_type": "display_data" 144 | }, 145 | { 146 | "data": { 147 | "text/html": [ 148 | "

Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.

" 149 | ], 150 | "text/plain": [ 151 | "" 152 | ] 153 | }, 154 | "metadata": {}, 155 | "output_type": "display_data" 156 | }, 157 | { 158 | "data": { 159 | "text/html": [ 160 | "

This notebook may still run but with reduced functionality.

" 161 | ], 162 | "text/plain": [ 163 | "" 164 | ] 165 | }, 166 | "metadata": {}, 167 | "output_type": "display_data" 168 | } 169 | ], 170 | "source": [ 171 | "mp.init_notebook()" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 18, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "name": "stdout", 181 | "output_type": "stream", 182 | "text": [ 183 | "connected\n" 184 | ] 185 | } 186 | ], 187 | "source": [ 188 | "splunk_prov = mp.QueryProvider(\"Splunk\")\n", 189 | "splunk_prov.connect()" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 19, 195 | "metadata": {}, 196 | "outputs": [ 197 | { 198 | "name": "stdout", 199 | "output_type": "stream", 200 | "text": [ 201 | "Query: get_events_parameterized\n", 202 | "Data source: Splunk\n", 203 | "Generic parameterized query from index/source\n", 204 | "\n", 205 | "Parameters\n", 206 | "----------\n", 207 | "add_query_items: str (optional)\n", 208 | " Additional query clauses\n", 209 | " (default value is: | head 100)\n", 210 | "end: datetime\n", 211 | " Query end time\n", 212 | "index: str (optional)\n", 213 | " Splunk index name\n", 214 | " (default value is: *)\n", 215 | "project_fields: str (optional)\n", 216 | " Project Field names\n", 217 | " (default value is: | table TimeCreated, host, EventID, EventDescripti...)\n", 218 | "source: str (optional)\n", 219 | " Splunk source type\n", 220 | " (default value is: *)\n", 221 | "start: datetime\n", 222 | " Query start time\n", 223 | "timeformat: str (optional)\n", 224 | " Datetime format to use in Splunk query\n", 225 | " (default value is: \"%Y-%m-%d %H:%M:%S.%6N\")\n", 226 | "Query:\n", 227 | " search index={index} source={source} timeformat={timeformat} earliest={start} latest={end} {project_fields} {add_query_items}\n" 228 | ] 229 | } 230 | ], 231 | "source": [ 232 | "splunk_prov.SplunkGeneral.get_events_parameterized('?')\n" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 20, 238 | "metadata": {}, 239 | "outputs": [ 240 | { 241 | "data": { 242 | "text/plain": [ 243 | "' search index=msticpy source=msticpy_splunk_reader_paging-test.csv timeformat=\"%Y-%m-%d %H:%M:%S.%6N\" earliest=\"2023-04-17 00:00:00\" latest=\"2023-04-19 10:00:00\" | table timestamp,rownum, desc, uuid4, host '" 244 | ] 245 | }, 246 | "execution_count": 20, 247 | "metadata": {}, 248 | "output_type": "execute_result" 249 | } 250 | ], 251 | "source": [ 252 | "splunk_prov.SplunkGeneral.get_events_parameterized('print',\n", 253 | " index=\"msticpy\",\n", 254 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 255 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 256 | " start=\"2023-04-17 00:00:00.000000\",\n", 257 | " end=\"2023-04-19 10:00:00.000000\", \n", 258 | " add_query_items='',\n", 259 | " count=0\n", 260 | ")\n", 261 | "\n" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 21, 267 | "metadata": {}, 268 | "outputs": [ 269 | { 270 | "name": "stderr", 271 | "output_type": "stream", 272 | "text": [ 273 | "Waiting Splunk job to complete: 200.0it [00:01, 195.71it/s] \n", 274 | "Waiting Splunk result to retrieve: 200000it [00:31, 6449.36it/s] \n" 275 | ] 276 | }, 277 | { 278 | "data": { 279 | "text/plain": [ 280 | "100000" 281 | ] 282 | }, 283 | "execution_count": 21, 284 | "metadata": {}, 285 | "output_type": "execute_result" 286 | } 287 | ], 288 | "source": [ 289 | "\n", 290 | "result_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 291 | " index=\"msticpy\",\n", 292 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 293 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 294 | " start=\"2023-04-17 00:00:00.000000\",\n", 295 | " end=\"2023-04-19 10:00:00.000000\", \n", 296 | " add_query_items='',\n", 297 | " count=0\n", 298 | ")\n", 299 | "len(result_df)" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 22, 305 | "metadata": {}, 306 | "outputs": [ 307 | { 308 | "name": "stderr", 309 | "output_type": "stream", 310 | "text": [ 311 | "Waiting Splunk job to complete: 200.0it [00:01, 195.50it/s] \n", 312 | "Waiting Splunk result to retrieve: 200000it [00:00, 253958.21it/s] \n" 313 | ] 314 | }, 315 | { 316 | "data": { 317 | "text/plain": [ 318 | "100000" 319 | ] 320 | }, 321 | "execution_count": 22, 322 | "metadata": {}, 323 | "output_type": "execute_result" 324 | } 325 | ], 326 | "source": [ 327 | "\n", 328 | "result_df = splunk_prov.SplunkGeneral.get_events_parameterized(\n", 329 | " index=\"msticpy\",\n", 330 | " source=\"msticpy_splunk_reader_paging-test.csv\",\n", 331 | " project_fields=\"| table timestamp,rownum, desc, uuid4, host\",\n", 332 | " start=\"2023-04-17 00:00:00.000000\",\n", 333 | " end=\"2023-04-19 10:00:00.000000\", \n", 334 | " add_query_items='',\n", 335 | " count=0,\n", 336 | " page_size=10000\n", 337 | ")\n", 338 | "len(result_df)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": {}, 345 | "outputs": [], 346 | "source": [] 347 | } 348 | ], 349 | "metadata": { 350 | "kernelspec": { 351 | "display_name": "Python 3", 352 | "language": "python", 353 | "name": "python3" 354 | }, 355 | "language_info": { 356 | "codemirror_mode": { 357 | "name": "ipython", 358 | "version": 3 359 | }, 360 | "file_extension": ".py", 361 | "mimetype": "text/x-python", 362 | "name": "python", 363 | "nbconvert_exporter": "python", 364 | "pygments_lexer": "ipython3", 365 | "version": "3.9.12" 366 | }, 367 | "orig_nbformat": 4 368 | }, 369 | "nbformat": 4, 370 | "nbformat_minor": 2 371 | } 372 | -------------------------------------------------------------------------------- /splunk_dsdl/README.md: -------------------------------------------------------------------------------- 1 | splunk_dsdl 2 | 3 | powershell_cmdline_usecase 4 | 5 | fitapply_result 6 | -------------------------------------------------------------------------------- /splunk_dsdl/msticpy_powershell_ioc.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "# [msticpy] ParentCommandLine powershell - base64 decode & ioc extraction" 9 | ] 10 | }, 11 | { 12 | "attachments": {}, 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Preparation for install msticpy\n", 17 | "\n", 18 | "Example of phdrieger/mltk-container-golden-image-cpu:5.1.0 container:\n", 19 | "\n", 20 | "$ mkdir /srv/app/model/customlib\n", 21 | "\n", 22 | "$ pip install msticpy -t /srv/app/model/customlib" 23 | ] 24 | }, 25 | { 26 | "attachments": {}, 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Stage 0 - import libraries\n", 31 | "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 23, 37 | "metadata": { 38 | "deletable": false, 39 | "name": "mltkc_import" 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "# this definition exposes all python module imports that should be available in all subsequent commands\n", 44 | "import json\n", 45 | "import numpy as np\n", 46 | "import pandas as pd\n", 47 | "import os\n", 48 | "import sys,os.path\n", 49 | "sys.path.insert(0,\"/srv/app/model/customlib\")\n", 50 | "import msticpy as mp\n", 51 | "\n", 52 | "# ...\n", 53 | "# global constants\n", 54 | "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", 55 | "\n", 56 | "# define the algorism name and model name\n", 57 | "algo_name = \"msticpy_powershell_ioc\"\n", 58 | "model_name = \"process_b64_iocs_enrich\"" 59 | ] 60 | }, 61 | { 62 | "attachments": {}, 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "## Stage 1 - get a data sample from Splunk\n", 67 | "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DSDL Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DSDL app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n", 68 | "\n", 69 | "### Option 1 - pull data from Splunk" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 25, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "# NOT EXPORTED\n", 79 | "from dsdlsupport import SplunkSearch as SplunkSearch\n", 80 | "search = SplunkSearch.SplunkSearch()\n", 81 | "df = search.as_df()" 82 | ] 83 | }, 84 | { 85 | "attachments": {}, 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "### Option 2 - push data from Splunk\n", 90 | "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4." 91 | ] 92 | }, 93 | { 94 | "attachments": {}, 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "index=botsv2 \"powershell\" \"-enc\" source=\"WinEventLog*Microsoft-Windows-Sysmon*Operational*\"
\n", 99 | "| where LIKE(ParentCommandLine,\"%powershell%-enc%\") | stats count by ParentCommandLine
\n", 100 | "| fit MLTKContainer mode=stage algo=msticpy_powershell_ioc ParentCommandLine into app:process_b64_iocs_enrich
" 101 | ] 102 | }, 103 | { 104 | "attachments": {}, 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"process_b64_iocs_enrich\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 2, 114 | "metadata": { 115 | "deletable": false, 116 | "name": "mltkc_stage" 117 | }, 118 | "outputs": [], 119 | "source": [ 120 | "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", 121 | "def stage(name):\n", 122 | " with open(\"data/\"+name+\".csv\", 'r') as f:\n", 123 | " df = pd.read_csv(f)\n", 124 | " with open(\"data/\"+name+\".json\", 'r') as f:\n", 125 | " param = json.load(f)\n", 126 | " return df, param" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 3, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", 136 | "df, param = stage(model_name)" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 30, 142 | "metadata": {}, 143 | "outputs": [ 144 | { 145 | "data": { 146 | "text/plain": [ 147 | "{'options': {'params': {'mode': 'stage', 'algo': 'msticpy_powershell_ioc'},\n", 148 | " 'args': ['ParentCommandLine'],\n", 149 | " 'feature_variables': ['ParentCommandLine'],\n", 150 | " 'model_name': 'process_b64_iocs_enrich',\n", 151 | " 'algo_name': 'MLTKContainer',\n", 152 | " 'mlspl_limits': {'handle_new_cat': 'default',\n", 153 | " 'max_distinct_cat_values': '100',\n", 154 | " 'max_distinct_cat_values_for_classifiers': '100',\n", 155 | " 'max_distinct_cat_values_for_scoring': '100',\n", 156 | " 'max_fit_time': '600',\n", 157 | " 'max_inputs': '100000',\n", 158 | " 'max_memory_usage_mb': '4000',\n", 159 | " 'max_model_size_mb': '30',\n", 160 | " 'max_score_time': '600',\n", 161 | " 'use_sampling': 'true'},\n", 162 | " 'kfold_cv': None},\n", 163 | " 'feature_variables': ['ParentCommandLine']}" 164 | ] 165 | }, 166 | "execution_count": 30, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "param" 173 | ] 174 | }, 175 | { 176 | "attachments": {}, 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "## Stage 2 - create and initialize a model" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 7, 186 | "metadata": { 187 | "deletable": false, 188 | "name": "mltkc_init" 189 | }, 190 | "outputs": [], 191 | "source": [ 192 | "# initialize your model\n", 193 | "# available inputs: data and parameters\n", 194 | "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", 195 | "def init(df,param):\n", 196 | " model = {}\n", 197 | " model['hyperparameter'] = 0\n", 198 | " model['desc'] = \"msticpy ioc enrichment from process command line\"\n", 199 | " return model" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "metadata": { 206 | "deletable": false, 207 | "name": "mltkc_apply" 208 | }, 209 | "outputs": [], 210 | "source": [] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 8, 215 | "metadata": { 216 | "scrolled": true 217 | }, 218 | "outputs": [ 219 | { 220 | "name": "stdout", 221 | "output_type": "stream", 222 | "text": [ 223 | "{'hyperparameter': 0, 'desc': 'msticpy ioc enrichment from process command line'}\n" 224 | ] 225 | } 226 | ], 227 | "source": [ 228 | "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", 229 | "model = init(df,param)\n", 230 | "print(model)" 231 | ] 232 | }, 233 | { 234 | "attachments": {}, 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "## fit the model" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 9, 244 | "metadata": { 245 | "deletable": false, 246 | "name": "mltkc_fit" 247 | }, 248 | "outputs": [], 249 | "source": [ 250 | "# train your model\n", 251 | "# returns a fit info json object and may modify the model object\n", 252 | "def fit(model,df,param):\n", 253 | " # model.fit()\n", 254 | " info = {\"message\": \"passthru model creating the model's json file\"}\n", 255 | " return info" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 10, 261 | "metadata": {}, 262 | "outputs": [ 263 | { 264 | "name": "stdout", 265 | "output_type": "stream", 266 | "text": [ 267 | "{'message': \"passthru model creating the model's json file\"}\n" 268 | ] 269 | } 270 | ], 271 | "source": [ 272 | "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", 273 | "print(fit(model,df,param))" 274 | ] 275 | }, 276 | { 277 | "attachments": {}, 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "## apply the model" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 13, 287 | "metadata": {}, 288 | "outputs": [], 289 | "source": [ 290 | "# NOT EXPORTED (just for debugging)\n", 291 | "def join_result(dec_df,ioc_input):\n", 292 | " for input_string, iocs in ioc_input.items():\n", 293 | " j=0\n", 294 | " while j < len(dec_df['decoded_string']):\n", 295 | " if dec_df['decode_validated'][j] == input_string:\n", 296 | " dec_df.at[dec_df.index[j], 'ioc_ipv4'] = \",\".join(iocs[\"ipv4\"])\n", 297 | " dec_df.at[dec_df.index[j], 'ioc_url'] = \",\".join(iocs[\"url\"])\n", 298 | " j+=1 \n", 299 | " return dec_df[[\"ParentCommandLine\",\"decode_validated\",\"ioc_ipv4\",\"ioc_url\"]]" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 13, 305 | "metadata": {}, 306 | "outputs": [], 307 | "source": [ 308 | "# NOT EXPORTED (just for debugging)\n", 309 | "def format_ioc(ioc_df):\n", 310 | " ioc_input = {}\n", 311 | " for row in ioc_df.itertuples():\n", 312 | " if not row.Input in ioc_input:\n", 313 | " ioc_input[row.Input] = { \"ipv4\":[],\"url\":[] }\n", 314 | " if row.IoCType==\"ipv4\":\n", 315 | " ioc_input[row.Input][\"ipv4\"].append(row.Observable)\n", 316 | " elif row.IoCType==\"url\":\n", 317 | " url = row.Observable\n", 318 | " elif row.IoCType==\"linux_path\" and row.Observable[0]==\"/\" and \".php\"in row.Observable.split(\";\")[0]:\n", 319 | " url_path = row.Observable.split(\";\")[0].rstrip(\"'\").rstrip('\"').rstrip()\n", 320 | " uri = url + url_path\n", 321 | " ioc_input[row.Input][\"url\"].append(uri)\n", 322 | " return ioc_input" 323 | ] 324 | }, 325 | { 326 | "cell_type": "code", 327 | "execution_count": 16, 328 | "metadata": {}, 329 | "outputs": [], 330 | "source": [ 331 | "# NOT EXPORTED\n", 332 | "def extract_ioc(dec_df):\n", 333 | " i=0\n", 334 | " while i < len(dec_df[\"decoded_string\"]):\n", 335 | " dec_df.at[dec_df.index[i], 'decode_validated'] = dec_df[\"decoded_string\"][i].replace(\"\\x00\",\"\")\n", 336 | " i+=1\n", 337 | " ioc_df = dec_df.mp.ioc_extract(columns=['decode_validated'],include_paths=True,\\\n", 338 | " ioc_types=['ipv4','ipv6','dns','url','email','windows_path','linux_path','md5_hash','sha1_hash','sha256_hash'])\n", 339 | " return ioc_df" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": 21, 345 | "metadata": { 346 | "name": "mltkc_apply" 347 | }, 348 | "outputs": [], 349 | "source": [ 350 | "# apply your model\n", 351 | "# returns the calculated results\n", 352 | "def apply(model,df,param): \n", 353 | " mp.init_notebook()\n", 354 | " \n", 355 | " # decode base64\n", 356 | " dec_df = mp.transform.base64unpack.unpack_df(data=df, column='ParentCommandLine')\n", 357 | " # remove nullbyte \\x00\n", 358 | " i=0\n", 359 | " while i < len(dec_df[\"decoded_string\"]):\n", 360 | " dec_df.at[dec_df.index[i], 'decode_validated'] = dec_df[\"decoded_string\"][i].replace(\"\\x00\",\"\")\n", 361 | " i+=1\n", 362 | " # extract ioc\n", 363 | " ioc_df = dec_df.mp.ioc_extract(columns=['decode_validated'],include_paths=True,\\\n", 364 | " ioc_types=['ipv4','ipv6','dns','url','email','windows_path','linux_path','md5_hash','sha1_hash','sha256_hash'])\n", 365 | " # format ioc to dict\n", 366 | " ioc_input = {}\n", 367 | " for row in ioc_df.itertuples():\n", 368 | " if not row.Input in ioc_input:\n", 369 | " ioc_input[row.Input] = { \"ipv4\":[],\"url\":[] }\n", 370 | " if row.IoCType==\"ipv4\":\n", 371 | " ioc_input[row.Input][\"ipv4\"].append(row.Observable)\n", 372 | " elif row.IoCType==\"url\":\n", 373 | " url = row.Observable\n", 374 | " elif row.IoCType==\"linux_path\" and row.Observable[0]==\"/\" and \".php\"in row.Observable.split(\";\")[0]:\n", 375 | " url_path = row.Observable.split(\";\")[0].rstrip(\"'\").rstrip('\"').rstrip()\n", 376 | " uri = url + url_path\n", 377 | " ioc_input[row.Input][\"url\"].append(uri) \n", 378 | " \n", 379 | " # join result\n", 380 | " for input_string, iocs in ioc_input.items():\n", 381 | " j=0\n", 382 | " while j < len(dec_df['decoded_string']):\n", 383 | " if dec_df['decode_validated'][j] == input_string:\n", 384 | " dec_df.at[dec_df.index[j], 'ioc_ipv4'] = \",\".join(iocs[\"ipv4\"])\n", 385 | " dec_df.at[dec_df.index[j], 'ioc_url'] = \",\".join(iocs[\"url\"])\n", 386 | " j+=1 \n", 387 | " dec_ioc_df = dec_df[[\"ParentCommandLine\",\"decode_validated\",\"ioc_ipv4\",\"ioc_url\"]]\n", 388 | "\n", 389 | " # ipwhois enrichment \n", 390 | " dec_ioc_enrich_df = dec_ioc_df.mp.whois(ip_column=\"ioc_ipv4\") \n", 391 | "\n", 392 | " return dec_ioc_enrich_df[[\"ParentCommandLine\",\"decode_validated\",\"ioc_ipv4\",\"ioc_url\",\"asn\",\"asn_country_code\",\"asn_description\"]] #dec_ioc_enrich_df_completed \n", 393 | " " 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [ 402 | "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", 403 | "print(apply(model,df,param))" 404 | ] 405 | }, 406 | { 407 | "attachments": {}, 408 | "cell_type": "markdown", 409 | "metadata": {}, 410 | "source": [ 411 | "## Stage 5 - save the model" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": 13, 417 | "metadata": { 418 | "deletable": false, 419 | "name": "mltkc_save" 420 | }, 421 | "outputs": [], 422 | "source": [ 423 | "# save model to name in expected convention \"_\"\n", 424 | "def save(model,name):\n", 425 | " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", 426 | " json.dump(model, file)\n", 427 | " return model" 428 | ] 429 | }, 430 | { 431 | "cell_type": "code", 432 | "execution_count": 14, 433 | "metadata": {}, 434 | "outputs": [ 435 | { 436 | "data": { 437 | "text/plain": [ 438 | "{'hyperparameter': 0,\n", 439 | " 'desc': 'msticpy ioc enrichment from process command line'}" 440 | ] 441 | }, 442 | "execution_count": 14, 443 | "metadata": {}, 444 | "output_type": "execute_result" 445 | } 446 | ], 447 | "source": [ 448 | "save(model,algo_name+\"_\"+model_name)" 449 | ] 450 | }, 451 | { 452 | "attachments": {}, 453 | "cell_type": "markdown", 454 | "metadata": {}, 455 | "source": [ 456 | "## Stage 6 - load the model" 457 | ] 458 | }, 459 | { 460 | "cell_type": "code", 461 | "execution_count": 15, 462 | "metadata": { 463 | "deletable": false, 464 | "name": "mltkc_load" 465 | }, 466 | "outputs": [], 467 | "source": [ 468 | "# load model from name in expected convention \"_\"\n", 469 | "def load(name):\n", 470 | " model = {}\n", 471 | " with open(MODEL_DIRECTORY + name + \".json\", 'r') as file:\n", 472 | " model = json.load(file)\n", 473 | " return model" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": 16, 479 | "metadata": {}, 480 | "outputs": [ 481 | { 482 | "data": { 483 | "text/plain": [ 484 | "{'hyperparameter': 0,\n", 485 | " 'desc': 'msticpy ioc enrichment from process command line'}" 486 | ] 487 | }, 488 | "execution_count": 16, 489 | "metadata": {}, 490 | "output_type": "execute_result" 491 | } 492 | ], 493 | "source": [ 494 | "load(algo_name+\"_\"+model_name)" 495 | ] 496 | }, 497 | { 498 | "attachments": {}, 499 | "cell_type": "markdown", 500 | "metadata": {}, 501 | "source": [ 502 | "## Stage 7 - provide a summary of the model" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 17, 508 | "metadata": { 509 | "deletable": false, 510 | "name": "mltkc_summary" 511 | }, 512 | "outputs": [], 513 | "source": [ 514 | "# return a model summary\n", 515 | "def summary(model=None):\n", 516 | " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} ,\"desc\": \"msticpy ioc enrichment from process command line\"}\n", 517 | " return returns" 518 | ] 519 | }, 520 | { 521 | "attachments": {}, 522 | "cell_type": "markdown", 523 | "metadata": {}, 524 | "source": [ 525 | "After implementing your fit, apply, save and load you can train your model:
\n", 526 | "\n", 527 | "index=botsv2 \"powershell\" \"-enc\" source=\"WinEventLog*Microsoft-Windows-Sysmon*Operational*\"
\n", 528 | "| where LIKE(ParentCommandLine,\"%powershell%-enc%\") | stats count by ParentCommandLine
\n", 529 | "| fit MLTKContainer algo=msticpy_powershell_ioc ParentCommandLine into app:process_b64_iocs_enrich
" 530 | ] 531 | }, 532 | { 533 | "attachments": {}, 534 | "cell_type": "markdown", 535 | "metadata": {}, 536 | "source": [ 537 | "Or apply your model:
\n", 538 | "\n", 539 | "index=botsv2 \"powershell\" \"-enc\" source=\"WinEventLog*Microsoft-Windows-Sysmon*Operational*\"
\n", 540 | "| where LIKE(ParentCommandLine,\"%powershell%-enc%\") | stats count by ParentCommandLine
\n", 541 | "| apply process_b64_iocs_enrich
" 542 | ] 543 | }, 544 | { 545 | "cell_type": "code", 546 | "execution_count": null, 547 | "metadata": {}, 548 | "outputs": [ 549 | { 550 | "data": { 551 | "text/plain": [ 552 | "{'version': {'numpy': '1.22.1', 'pandas': '1.5.3'},\n", 553 | " 'desc': 'msticpy ioc enrichment from process command line'}" 554 | ] 555 | }, 556 | "execution_count": 18, 557 | "metadata": {}, 558 | "output_type": "execute_result" 559 | } 560 | ], 561 | "source": [ 562 | "summary()" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": null, 568 | "metadata": {}, 569 | "outputs": [], 570 | "source": [] 571 | }, 572 | { 573 | "cell_type": "code", 574 | "execution_count": null, 575 | "metadata": {}, 576 | "outputs": [], 577 | "source": [] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": null, 582 | "metadata": {}, 583 | "outputs": [], 584 | "source": [] 585 | }, 586 | { 587 | "attachments": {}, 588 | "cell_type": "markdown", 589 | "metadata": {}, 590 | "source": [ 591 | "## Send data back to Splunk HEC\n", 592 | "When you configured the Splunk HEC Settings in the DSDL app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n", 593 | "\n", 594 | "### Use cases\n", 595 | "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n", 596 | "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n", 597 | "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n", 598 | "\n", 599 | "### Example" 600 | ] 601 | }, 602 | { 603 | "cell_type": "code", 604 | "execution_count": 19, 605 | "metadata": {}, 606 | "outputs": [], 607 | "source": [ 608 | "from dsdlsupport import SplunkHEC as SplunkHEC\n", 609 | "hec = SplunkHEC.SplunkHEC()" 610 | ] 611 | }, 612 | { 613 | "cell_type": "code", 614 | "execution_count": null, 615 | "metadata": {}, 616 | "outputs": [], 617 | "source": [ 618 | "# example to send 10 hello world events\n", 619 | "response = hec.send_hello_world(10)" 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": null, 625 | "metadata": {}, 626 | "outputs": [], 627 | "source": [ 628 | "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" 629 | ] 630 | }, 631 | { 632 | "cell_type": "code", 633 | "execution_count": 21, 634 | "metadata": {}, 635 | "outputs": [], 636 | "source": [ 637 | "# example to send a JSON object, e.g. to log some data\n", 638 | "from datetime import datetime\n", 639 | "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})" 640 | ] 641 | }, 642 | { 643 | "cell_type": "code", 644 | "execution_count": null, 645 | "metadata": {}, 646 | "outputs": [], 647 | "source": [ 648 | "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" 649 | ] 650 | }, 651 | { 652 | "attachments": {}, 653 | "cell_type": "markdown", 654 | "metadata": {}, 655 | "source": [ 656 | "## End of Stages\n", 657 | "All subsequent cells are not tagged and can be used for further freeform code" 658 | ] 659 | }, 660 | { 661 | "cell_type": "code", 662 | "execution_count": null, 663 | "metadata": {}, 664 | "outputs": [], 665 | "source": [] 666 | }, 667 | { 668 | "cell_type": "code", 669 | "execution_count": null, 670 | "metadata": {}, 671 | "outputs": [], 672 | "source": [] 673 | } 674 | ], 675 | "metadata": { 676 | "kernelspec": { 677 | "display_name": "Python 3 (ipykernel)", 678 | "language": "python", 679 | "name": "python3" 680 | }, 681 | "language_info": { 682 | "codemirror_mode": { 683 | "name": "ipython", 684 | "version": 3 685 | }, 686 | "file_extension": ".py", 687 | "mimetype": "text/x-python", 688 | "name": "python", 689 | "nbconvert_exporter": "python", 690 | "pygments_lexer": "ipython3", 691 | "version": "3.9.13" 692 | } 693 | }, 694 | "nbformat": 4, 695 | "nbformat_minor": 4 696 | } 697 | -------------------------------------------------------------------------------- /timeseries_anomalies_stl/msticpy_timeseries_anomalies_stl.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "## Anomary Detection with timeseries_anomalies_stl \n", 9 | "\n", 10 | "data: splunk buttercup game added iplocation result" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 22, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "name": "stdout", 20 | "output_type": "stream", 21 | "text": [ 22 | "Requirement already satisfied: statsmodels in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (0.12.1)\n", 23 | "Collecting statsmodels\n", 24 | " Downloading statsmodels-0.13.5-cp39-cp39-macosx_10_9_x86_64.whl (9.7 MB)\n", 25 | "\u001b[K |████████████████████████████████| 9.7 MB 11.8 MB/s eta 0:00:01\n", 26 | "\u001b[?25hRequirement already satisfied: pandas>=0.25 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from statsmodels) (1.4.2)\n", 27 | "Requirement already satisfied: numpy>=1.17 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from statsmodels) (1.22.3)\n", 28 | "Requirement already satisfied: packaging>=21.3 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from statsmodels) (21.3)\n", 29 | "Requirement already satisfied: scipy>=1.3 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from statsmodels) (1.10.1)\n", 30 | "Requirement already satisfied: patsy>=0.5.2 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from statsmodels) (0.5.3)\n", 31 | "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from packaging>=21.3->statsmodels) (3.0.4)\n", 32 | "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from pandas>=0.25->statsmodels) (2.8.2)\n", 33 | "Requirement already satisfied: pytz>=2020.1 in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from pandas>=0.25->statsmodels) (2021.3)\n", 34 | "Requirement already satisfied: six in /Users/hacket/opt/anaconda3/envs/msticpy/lib/python3.9/site-packages (from patsy>=0.5.2->statsmodels) (1.16.0)\n", 35 | "Installing collected packages: statsmodels\n", 36 | " Attempting uninstall: statsmodels\n", 37 | " Found existing installation: statsmodels 0.12.1\n", 38 | " Uninstalling statsmodels-0.12.1:\n", 39 | " Successfully uninstalled statsmodels-0.12.1\n", 40 | "Successfully installed statsmodels-0.13.5\n" 41 | ] 42 | } 43 | ], 44 | "source": [ 45 | "%pip install statsmodels --upgrade" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 51, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "name": "stdout", 55 | "output_type": "stream", 56 | "text": [ 57 | "Mon May 8 13:18:46 JST 2023\n", 58 | "Python 3.9.12\n", 59 | "sys.version\n", 60 | " 3.9.12 (main, Apr 5 2022, 01:53:17) \n", 61 | "[Clang 12.0.0 ]\n" 62 | ] 63 | } 64 | ], 65 | "source": [ 66 | "\n", 67 | "!date\n", 68 | "!python -V # check if python version is later than 3.8\n", 69 | "\n", 70 | "import sys\n", 71 | "print(\"sys.version\\n\",sys.version)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 52, 77 | "metadata": {}, 78 | "outputs": [ 79 | { 80 | "name": "stdout", 81 | "output_type": "stream", 82 | "text": [ 83 | "2.4.0\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "import msticpy\n", 89 | "print(msticpy.__version__)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "#mp.init_notebook()" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": 139, 104 | "metadata": {}, 105 | "outputs": [ 106 | { 107 | "data": { 108 | "text/html": [ 109 | "
\n", 110 | "\n", 123 | "\n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | " \n", 133 | " \n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | "
Countryactionbytesreq_timestatusdate_clockcount
0Netherlandsaddtocart274408/Sep/2022:07:19:242002022-09-08 071
1Netherlandspurchase173608/Sep/2022:07:19:262002022-09-08 071
2Netherlandspurchase98908/Sep/2022:07:19:252002022-09-08 071
3Netherlandsaddtocart158108/Sep/2022:07:19:242002022-09-08 071
4NetherlandsNaN74908/Sep/2022:07:19:242002022-09-08 071
\n", 189 | "
" 190 | ], 191 | "text/plain": [ 192 | " Country action bytes req_time status date_clock \\\n", 193 | "0 Netherlands addtocart 2744 08/Sep/2022:07:19:24 200 2022-09-08 07 \n", 194 | "1 Netherlands purchase 1736 08/Sep/2022:07:19:26 200 2022-09-08 07 \n", 195 | "2 Netherlands purchase 989 08/Sep/2022:07:19:25 200 2022-09-08 07 \n", 196 | "3 Netherlands addtocart 1581 08/Sep/2022:07:19:24 200 2022-09-08 07 \n", 197 | "4 Netherlands NaN 749 08/Sep/2022:07:19:24 200 2022-09-08 07 \n", 198 | "\n", 199 | " count \n", 200 | "0 1 \n", 201 | "1 1 \n", 202 | "2 1 \n", 203 | "3 1 \n", 204 | "4 1 " 205 | ] 206 | }, 207 | "execution_count": 139, 208 | "metadata": {}, 209 | "output_type": "execute_result" 210 | } 211 | ], 212 | "source": [ 213 | "import pandas as pd\n", 214 | "\n", 215 | "df = pd.read_csv(\n", 216 | " \"../data/buttercupgame_iplocation.csv\", infer_datetime_format=True,\n", 217 | " usecols=[\"req_time\",\"bytes\",\"Country\",\"status\",\"action\"]\n", 218 | " )\n", 219 | "df[\"date_clock\"] = pd.to_datetime(df[\"req_time\"], format=\"%d/%b/%Y:%H:%M:%S\", errors='coerce').dt.strftime(\"%Y-%m-%d %H\")\n", 220 | "df['count'] = 1\n", 221 | "df.head()" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 141, 227 | "metadata": {}, 228 | "outputs": [ 229 | { 230 | "data": { 231 | "text/html": [ 232 | "
\n", 233 | "\n", 246 | "\n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | "
date_clockactioncount
02022-09-01 18addtocart17
12022-09-01 18changequantity7
22022-09-01 18purchase17
32022-09-01 18remove9
42022-09-01 18view20
\n", 288 | "
" 289 | ], 290 | "text/plain": [ 291 | " date_clock action count\n", 292 | "0 2022-09-01 18 addtocart 17\n", 293 | "1 2022-09-01 18 changequantity 7\n", 294 | "2 2022-09-01 18 purchase 17\n", 295 | "3 2022-09-01 18 remove 9\n", 296 | "4 2022-09-01 18 view 20" 297 | ] 298 | }, 299 | "execution_count": 141, 300 | "metadata": {}, 301 | "output_type": "execute_result" 302 | } 303 | ], 304 | "source": [ 305 | "\n", 306 | "\n", 307 | "df_action = df.groupby(['date_clock','action'])['count'].sum().reset_index()\n", 308 | "df_action.head()\n" 309 | ] 310 | }, 311 | { 312 | "cell_type": "code", 313 | "execution_count": 142, 314 | "metadata": {}, 315 | "outputs": [ 316 | { 317 | "data": { 318 | "text/html": [ 319 | "
\n", 320 | "\n", 333 | "\n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | "
date_clockactioncount
02022-09-01 18addtocart17
52022-09-01 19addtocart40
102022-09-01 20addtocart26
152022-09-01 21addtocart46
202022-09-01 22addtocart24
\n", 375 | "
" 376 | ], 377 | "text/plain": [ 378 | " date_clock action count\n", 379 | "0 2022-09-01 18 addtocart 17\n", 380 | "5 2022-09-01 19 addtocart 40\n", 381 | "10 2022-09-01 20 addtocart 26\n", 382 | "15 2022-09-01 21 addtocart 46\n", 383 | "20 2022-09-01 22 addtocart 24" 384 | ] 385 | }, 386 | "execution_count": 142, 387 | "metadata": {}, 388 | "output_type": "execute_result" 389 | } 390 | ], 391 | "source": [ 392 | "df_action = df_action[df_action.action==\"addtocart\"]\n", 393 | "df_action.head()" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 143, 399 | "metadata": {}, 400 | "outputs": [ 401 | { 402 | "data": { 403 | "text/html": [ 404 | "
\n", 405 | "\n", 418 | "\n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | " \n", 448 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 452 | " \n", 453 | " \n", 454 | " \n", 455 | " \n", 456 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 460 | " \n", 461 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | "
date_clockcountresidualtrendseasonalweightsbaselinescoreanomalies
02022-09-01 1817-234-14119-0.4243890
12022-09-01 194013441380.2010910
22022-09-01 2026-334-4129-0.6328820
32022-09-01 214633481420.6180780
42022-09-01 2224-234-7126-0.4243890
\n", 496 | "
" 497 | ], 498 | "text/plain": [ 499 | " date_clock count residual trend seasonal weights baseline \\\n", 500 | "0 2022-09-01 18 17 -2 34 -14 1 19 \n", 501 | "1 2022-09-01 19 40 1 34 4 1 38 \n", 502 | "2 2022-09-01 20 26 -3 34 -4 1 29 \n", 503 | "3 2022-09-01 21 46 3 34 8 1 42 \n", 504 | "4 2022-09-01 22 24 -2 34 -7 1 26 \n", 505 | "\n", 506 | " score anomalies \n", 507 | "0 -0.424389 0 \n", 508 | "1 0.201091 0 \n", 509 | "2 -0.632882 0 \n", 510 | "3 0.618078 0 \n", 511 | "4 -0.424389 0 " 512 | ] 513 | }, 514 | "execution_count": 143, 515 | "metadata": {}, 516 | "output_type": "execute_result" 517 | } 518 | ], 519 | "source": [ 520 | "from msticpy.analysis.timeseries import timeseries_anomalies_stl\n", 521 | "df_count = df_action[['date_clock','count']]\n", 522 | "df_count = df_count.set_index('date_clock')\n", 523 | "output = timeseries_anomalies_stl(df_count)\n", 524 | "output.head()" 525 | ] 526 | }, 527 | { 528 | "cell_type": "code", 529 | "execution_count": 144, 530 | "metadata": {}, 531 | "outputs": [ 532 | { 533 | "data": { 534 | "text/html": [ 535 | "
\n", 536 | "\n", 549 | "\n", 550 | " \n", 551 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | "
date_clockcountresidualtrendseasonalweightsbaselinescoreanomalies
942022-09-05 1652153331363.1199981
\n", 579 | "
" 580 | ], 581 | "text/plain": [ 582 | " date_clock count residual trend seasonal weights baseline \\\n", 583 | "94 2022-09-05 16 52 15 33 3 1 36 \n", 584 | "\n", 585 | " score anomalies \n", 586 | "94 3.119998 1 " 587 | ] 588 | }, 589 | "execution_count": 144, 590 | "metadata": {}, 591 | "output_type": "execute_result" 592 | } 593 | ], 594 | "source": [ 595 | "output[output.anomalies == 1]" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": 146, 601 | "metadata": {}, 602 | "outputs": [ 603 | { 604 | "name": "stderr", 605 | "output_type": "stream", 606 | "text": [ 607 | "/var/folders/9j/5q4qwns11lv47mtj3ykbp5_40000gn/T/ipykernel_3437/3785166171.py:7: DeprecationWarning: Call to deprecated function (or staticmethod) display_timeseries_anomolies. (Will be removed in version 2.0.0. Please use display_timeseries_anomalies) -- Deprecated since version 1.7.0.\n", 608 | " timeseries_anomalies_plot = display_timeseries_anomolies(\n" 609 | ] 610 | }, 611 | { 612 | "data": { 613 | "text/html": [ 614 | "\n", 615 | "
\n", 616 | " \n", 617 | " Loading BokehJS ...\n", 618 | "
" 619 | ] 620 | }, 621 | "metadata": {}, 622 | "output_type": "display_data" 623 | }, 624 | { 625 | "data": { 626 | "application/javascript": "\n(function(root) {\n function now() {\n return new Date();\n }\n\n const force = true;\n\n if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n root._bokeh_onload_callbacks = [];\n root._bokeh_is_loading = undefined;\n }\n\n const JS_MIME_TYPE = 'application/javascript';\n const HTML_MIME_TYPE = 'text/html';\n const EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n const CLASS_NAME = 'output_bokeh rendered_html';\n\n /**\n * Render data to the DOM node\n */\n function render(props, node) {\n const script = document.createElement(\"script\");\n node.appendChild(script);\n }\n\n /**\n * Handle when an output is cleared or removed\n */\n function handleClearOutput(event, handle) {\n const cell = handle.cell;\n\n const id = cell.output_area._bokeh_element_id;\n const server_id = cell.output_area._bokeh_server_id;\n // Clean up Bokeh references\n if (id != null && id in Bokeh.index) {\n Bokeh.index[id].model.document.clear();\n delete Bokeh.index[id];\n }\n\n if (server_id !== undefined) {\n // Clean up Bokeh references\n const cmd_clean = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n cell.notebook.kernel.execute(cmd_clean, {\n iopub: {\n output: function(msg) {\n const id = msg.content.text.trim();\n if (id in Bokeh.index) {\n Bokeh.index[id].model.document.clear();\n delete Bokeh.index[id];\n }\n }\n }\n });\n // Destroy server and session\n const cmd_destroy = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n cell.notebook.kernel.execute(cmd_destroy);\n }\n }\n\n /**\n * Handle when a new output is added\n */\n function handleAddOutput(event, handle) {\n const output_area = handle.output_area;\n const output = handle.output;\n\n // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n return\n }\n\n const toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n\n if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n // store reference to embed id on output_area\n output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n }\n if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n const bk_div = document.createElement(\"div\");\n bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n const script_attrs = bk_div.children[0].attributes;\n for (let i = 0; i < script_attrs.length; i++) {\n toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n }\n // store reference to server id on output_area\n output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n }\n }\n\n function register_renderer(events, OutputArea) {\n\n function append_mime(data, metadata, element) {\n // create a DOM node to render to\n const toinsert = this.create_output_subarea(\n metadata,\n CLASS_NAME,\n EXEC_MIME_TYPE\n );\n this.keyboard_manager.register_events(toinsert);\n // Render to node\n const props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n render(props, toinsert[toinsert.length - 1]);\n element.append(toinsert);\n return toinsert\n }\n\n /* Handle when an output is cleared or removed */\n events.on('clear_output.CodeCell', handleClearOutput);\n events.on('delete.Cell', handleClearOutput);\n\n /* Handle when a new output is added */\n events.on('output_added.OutputArea', handleAddOutput);\n\n /**\n * Register the mime type and append_mime function with output_area\n */\n OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n /* Is output safe? */\n safe: true,\n /* Index of renderer in `output_area.display_order` */\n index: 0\n });\n }\n\n // register the mime type if in Jupyter Notebook environment and previously unregistered\n if (root.Jupyter !== undefined) {\n const events = require('base/js/events');\n const OutputArea = require('notebook/js/outputarea').OutputArea;\n\n if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n register_renderer(events, OutputArea);\n }\n }\n\n \n if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n root._bokeh_timeout = Date.now() + 5000;\n root._bokeh_failed_load = false;\n }\n\n const NB_LOAD_WARNING = {'data': {'text/html':\n \"
\\n\"+\n \"

\\n\"+\n \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n \"

\\n\"+\n \"
    \\n\"+\n \"
  • re-rerun `output_notebook()` to attempt to load from CDN again, or
    • \\n\"+\n \"
  • use INLINE resources instead, as so:
    • \\n\"+\n \"
\\n\"+\n \"\\n\"+\n \"from bokeh.resources import INLINE\\n\"+\n \"output_notebook(resources=INLINE)\\n\"+\n \"\\n\"+\n \"
\"}};\n\n function display_loaded() {\n const el = document.getElementById(\"8346\");\n if (el != null) {\n el.textContent = \"BokehJS is loading...\";\n }\n if (root.Bokeh !== undefined) {\n if (el != null) {\n el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n }\n } else if (Date.now() < root._bokeh_timeout) {\n setTimeout(display_loaded, 100)\n }\n }\n\n\n function run_callbacks() {\n try {\n root._bokeh_onload_callbacks.forEach(function(callback) {\n if (callback != null)\n callback();\n });\n } finally {\n delete root._bokeh_onload_callbacks\n }\n console.debug(\"Bokeh: all callbacks have finished\");\n }\n\n function load_libs(css_urls, js_urls, callback) {\n if (css_urls == null) css_urls = [];\n if (js_urls == null) js_urls = [];\n\n root._bokeh_onload_callbacks.push(callback);\n if (root._bokeh_is_loading > 0) {\n console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n return null;\n }\n if (js_urls == null || js_urls.length === 0) {\n run_callbacks();\n return null;\n }\n console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n function on_load() {\n root._bokeh_is_loading--;\n if (root._bokeh_is_loading === 0) {\n console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n run_callbacks()\n }\n }\n\n function on_error(url) {\n console.error(\"failed to load \" + url);\n }\n\n for (let i = 0; i < css_urls.length; i++) {\n const url = css_urls[i];\n const element = document.createElement(\"link\");\n element.onload = on_load;\n element.onerror = on_error.bind(null, url);\n element.rel = \"stylesheet\";\n element.type = \"text/css\";\n element.href = url;\n console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n document.body.appendChild(element);\n }\n\n for (let i = 0; i < js_urls.length; i++) {\n const url = js_urls[i];\n const element = document.createElement('script');\n element.onload = on_load;\n element.onerror = on_error.bind(null, url);\n element.async = false;\n element.src = url;\n console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n document.head.appendChild(element);\n }\n };\n\n function inject_raw_css(css) {\n const element = document.createElement(\"style\");\n element.appendChild(document.createTextNode(css));\n document.body.appendChild(element);\n }\n\n \n const js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.4.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-2.4.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-2.4.2.min.js\"];\n const css_urls = [];\n \n\n const inline_js = [\n function(Bokeh) {\n Bokeh.set_log_level(\"info\");\n },\n function(Bokeh) {\n \n \n }\n ];\n\n function run_inline_js() {\n \n if (root.Bokeh !== undefined || force === true) {\n \n for (let i = 0; i < inline_js.length; i++) {\n inline_js[i].call(root, root.Bokeh);\n }\n if (force === true) {\n display_loaded();\n }} else if (Date.now() < root._bokeh_timeout) {\n setTimeout(run_inline_js, 100);\n } else if (!root._bokeh_failed_load) {\n console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n root._bokeh_failed_load = true;\n } else if (force !== true) {\n const cell = $(document.getElementById(\"8346\")).parents('.cell').data().cell;\n cell.output_area.append_execute_result(NB_LOAD_WARNING)\n }\n\n }\n\n if (root._bokeh_is_loading === 0) {\n console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n run_inline_js();\n } else {\n load_libs(css_urls, js_urls, function() {\n console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n run_inline_js();\n });\n }\n}(window));", 627 | "application/vnd.bokehjs_load.v0+json": "" 628 | }, 629 | "metadata": {}, 630 | "output_type": "display_data" 631 | }, 632 | { 633 | "data": { 634 | "text/html": [ 635 | "\n", 636 | "\n", 637 | "\n", 638 | "\n", 639 | "\n", 640 | "\n", 641 | "
\n" 642 | ] 643 | }, 644 | "metadata": {}, 645 | "output_type": "display_data" 646 | }, 647 | { 648 | "data": { 649 | "application/javascript": "(function(root) {\n function embed_document(root) {\n \n const docs_json = {\"70465022-a210-4640-b276-2bf87def5374\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"children\":[{\"id\":\"8380\"},{\"id\":\"8532\"}]},\"id\":\"8563\",\"type\":\"Column\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"hatch_alpha\":{\"value\":0.1},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"navy\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8418\",\"type\":\"Circle\"},{\"attributes\":{\"months\":[0,6]},\"id\":\"8440\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"coordinates\":null,\"data_source\":{\"id\":\"8347\"},\"glyph\":{\"id\":\"8417\"},\"group\":null,\"hover_glyph\":null,\"muted_glyph\":{\"id\":\"8419\"},\"nonselection_glyph\":{\"id\":\"8418\"},\"view\":{\"id\":\"8421\"}},\"id\":\"8420\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"8441\",\"type\":\"YearsTicker\"},{\"attributes\":{\"mantissas\":[1,2,5],\"max_interval\":500.0,\"num_minor_ticks\":0},\"id\":\"8430\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{\"click_policy\":\"hide\",\"coordinates\":null,\"group\":null,\"items\":[{\"id\":\"8443\"},{\"id\":\"8470\"},{\"id\":\"8500\"}],\"location\":\"top_left\"},\"id\":\"8442\",\"type\":\"Legend\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.2},\"fill_color\":{\"value\":\"#1f77b4\"},\"hatch_alpha\":{\"value\":0.2},\"line_alpha\":{\"value\":0.2},\"line_color\":{\"value\":\"navy\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8419\",\"type\":\"Circle\"},{\"attributes\":{\"source\":{\"id\":\"8347\"}},\"id\":\"8421\",\"type\":\"CDSView\"},{\"attributes\":{\"label\":{\"value\":\"observed\"},\"renderers\":[{\"id\":\"8420\"}]},\"id\":\"8443\",\"type\":\"LegendItem\"},{\"attributes\":{\"end\":1662750720000.0,\"start\":1661964480000.0},\"id\":\"8535\",\"type\":\"Range1d\"},{\"attributes\":{\"below\":[{\"id\":\"8543\"},{\"id\":\"8548\"}],\"center\":[{\"id\":\"8546\"}],\"height\":120,\"renderers\":[{\"id\":\"8556\"}],\"title\":{\"id\":\"8533\"},\"toolbar\":{\"id\":\"8547\"},\"toolbar_location\":null,\"width\":1200,\"x_range\":{\"id\":\"8535\"},\"x_scale\":{\"id\":\"8539\"},\"y_range\":{\"id\":\"8537\"},\"y_scale\":{\"id\":\"8541\"}},\"id\":\"8532\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"callback\":null,\"formatters\":{\"@date_clock\":\"datetime\"},\"tooltips\":[[\"date_clock\",\"@date_clock{%F %T.%3N}\"],[\"count\",\"@count\"],[\"residual\",\"@residual\"],[\"trend\",\"@trend\"],[\"seasonal\",\"@seasonal\"],[\"weights\",\"@weights\"],[\"baseline\",\"@baseline\"],[\"score\",\"@score\"]]},\"id\":\"8348\",\"type\":\"HoverTool\"},{\"attributes\":{\"coordinates\":null,\"fill_alpha\":0.2,\"fill_color\":\"navy\",\"group\":null,\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[2,2],\"line_width\":0.5,\"syncable\":false},\"id\":\"8559\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"data\":{\"anomalies\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],\"baseline\":[19,38,29,42,26,29,39,40,32,54,35,23,30,27,36,27,34,25,33,33,34,43,35,40,26,38,31,39,30,33,37,43,36,54,35,28,30,28,34,26,33,28,33,34,33,46,38,41,33,39,34,36,33,36,35,45,38,52,33,30,28,26,29,23,29,28,29,30,28,43,36,39,36,36,33,31,34,35,31,46,39,49,29,31,25,24,24,21,26,28,27,28,26,44,36,39,38,36,32,32,34,32,34,50,40,50,29,33,29,26,24,22,29,29,26,29,27,44,34,38,34,36,31,31,33,28,35,54,38,50,28,31,30,25,22,22,32,29,24,28,29,43,33,39,31,38,32,33,34,25,40,60,39,52,29,33,35,27,23,24,38,30,23,29,32,43,30,40,27],\"count\":[17,40,26,46,24,26,46,48,28,55,35,24,29,29,36,30,29,25,31,31,38,47,37,40,25,34,35,32,35,38,33,26,44,52,40,26,38,27,34,26,39,29,38,39,33,38,39,42,39,43,35,42,34,33,31,54,38,59,24,27,21,24,34,18,35,29,29,29,23,52,26,41,44,37,38,28,28,44,27,54,32,41,36,44,20,30,16,23,22,28,23,27,26,39,52,35,34,37,32,28,42,38,34,44,49,54,31,29,33,23,28,22,19,34,32,26,31,48,33,44,38,29,24,32,40,22,46,42,44,49,21,31,32,25,18,24,29,23,24,37,24,47,33,38,36,44,38,36,28,27,34,71,33,54,33,34,34,28,26,24,46,33,22,25,35,40,29,40,23],\"date_clock\":{\"__ndarray__\":\"AABQW6MveEIAADjKpi94QgAAIDmqL3hCAAAIqK0veEIAAPAWsS94QgAA2IW0L3hCAADA9LcveEIAAKhjuy94QgAAkNK+L3hCAAB4QcIveEIAAGCwxS94QgAASB/JL3hCAAAwjswveEIAABj9zy94QgAAAGzTL3hCAADo2tYveEIAANBJ2i94QgAAuLjdL3hCAACgJ+EveEIAAIiW5C94QgAAcAXoL3hCAABYdOsveEIAAEDj7i94QgAAKFLyL3hCAAAQwfUveEIAAPgv+S94QgAA4J78L3hCAADIDQAweEIAALB8AzB4QgAAmOsGMHhCAACAWgoweEIAAGjJDTB4QgAAUDgRMHhCAAA4pxQweEIAACAWGDB4QgAACIUbMHhCAADw8x4weEIAANhiIjB4QgAAwNElMHhCAACoQCkweEIAAJCvLDB4QgAAeB4wMHhCAABgjTMweEIAAEj8NjB4QgAAMGs6MHhCAAAY2j0weEIAAABJQTB4QgAA6LdEMHhCAADQJkgweEIAALiVSzB4QgAAoARPMHhCAACIc1IweEIAAHDiVTB4QgAAWFFZMHhCAABAwFwweEIAACgvYDB4QgAAEJ5jMHhCAAD4DGcweEIAAOB7ajB4QgAAyOptMHhCAACwWXEweEIAAJjIdDB4QgAAgDd4MHhCAABopnsweEIAAFAVfzB4QgAAOISCMHhCAAAg84UweEIAAAhiiTB4QgAA8NCMMHhCAADYP5AweEIAAMCukzB4QgAAqB2XMHhCAACQjJoweEIAAHj7nTB4QgAAYGqhMHhCAABI2aQweEIAADBIqDB4QgAAGLerMHhCAAAAJq8weEIAAOiUsjB4QgAA0AO2MHhCAAC4crkweEIAAKDhvDB4QgAAiFDAMHhCAABwv8MweEIAAFguxzB4QgAAQJ3KMHhCAAAoDM4weEIAABB70TB4QgAA+OnUMHhCAADgWNgweEIAAMjH2zB4QgAAsDbfMHhCAACYpeIweEIAAIAU5jB4QgAAaIPpMHhCAABQ8uwweEIAADhh8DB4QgAAINDzMHhCAAAIP/cweEIAAPCt+jB4QgAA2Bz+MHhCAADAiwExeEIAAKj6BDF4QgAAkGkIMXhCAAB42AsxeEIAAGBHDzF4QgAASLYSMXhCAAAwJRYxeEIAABiUGTF4QgAAAAMdMXhCAADocSAxeEIAANDgIzF4QgAAuE8nMXhCAACgvioxeEIAAIgtLjF4QgAAcJwxMXhCAABYCzUxeEIAAEB6ODF4QgAAKOk7MXhCAAAQWD8xeEIAAPjGQjF4QgAA4DVGMXhCAADIpEkxeEIAALATTTF4QgAAmIJQMXhCAACA8VMxeEIAAGhgVzF4QgAAUM9aMXhCAAA4Pl4xeEIAACCtYTF4QgAACBxlMXhCAADwimgxeEIAANj5azF4QgAAwGhvMXhCAACo13IxeEIAAJBGdjF4QgAAeLV5MXhCAABgJH0xeEIAAEiTgDF4QgAAMAKEMXhCAAAYcYcxeEIAAADgijF4QgAA6E6OMXhCAADQvZExeEIAALgslTF4QgAAoJuYMXhCAACICpwxeEIAAHB5nzF4QgAAWOiiMXhCAABAV6YxeEIAACjGqTF4QgAAEDWtMXhCAAD4o7AxeEIAAOAStDF4QgAAyIG3MXhCAACw8LoxeEIAAJhfvjF4QgAAgM7BMXhCAABoPcUxeEIAAFCsyDF4QgAAOBvMMXhCAAAgis8xeEIAAAj50jF4QgAA8GfWMXhCAADY1tkxeEIAAMBF3TF4QgAAqLTgMXhCAACQI+QxeEI=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[169]},\"index\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168],\"residual\":[-2,1,-3,3,-2,-3,6,7,-4,0,0,0,-1,1,0,2,-5,0,-2,-2,3,3,1,0,-1,-4,3,-7,4,4,-4,-17,7,-2,4,-2,7,-1,0,0,5,0,4,4,0,-8,0,0,5,3,0,5,0,-3,-4,8,0,6,-9,-3,-7,-2,4,-5,5,0,0,-1,-5,8,-10,1,7,0,4,-3,-6,8,-4,7,-7,-8,6,12,-5,5,-8,1,-4,0,-4,-1,0,-5,15,-4,-4,0,0,-4,7,5,0,-6,8,3,1,-4,3,-3,3,0,-10,4,5,-3,3,3,-1,5,3,-7,-7,0,6,-6,10,-12,5,-1,-7,0,1,0,-4,1,-3,-6,0,8,-5,3,0,-1,4,5,5,2,-6,1,-6,10,-6,1,3,0,-1,0,2,0,7,2,-1,-4,2,-3,-1,0,-4],\"score\":{\"__ndarray__\":\"oh0zay8p278V3LoTW73JPximcd6RQOS/k+WeFkvH4z+iHTNrLynbvximcd6RQOS/tZWTyJzl8z9Zof/cmTv3P1+9SQeM7Oq/FCGw9LFRfr8UIbD0sVF+vxQhsPSxUX6/Jt4FM3aiy78V3LoTW73JPxQhsPSxUX6/mZyN26E22j9T6hAYQ8zwvxQhsPSxUX6/oh0zay8p27+iHTNrLynbv5PlnhZLx+M/k+WeFkvH4z8V3LoTW73JPxQhsPSxUX6/Jt4FM3aiy79fvUkHjOzqv5PlnhZLx+M/mgHpQD1497/b/HY/RXPqP9v8dj9Fc+o/X71JB4zs6r8Au5AGEGoMwFmh/9yZO/c/oh0zay8p27/b/HY/RXPqP6IdM2svKdu/WaH/3Jk79z8m3gUzdqLLvxQhsPSxUX6/FCGw9LFRfr8Riie0n4/wPxQhsPSxUX6/2/x2P0Vz6j/b/HY/RXPqPxQhsPSxUX6/Pg1VVTrO+r8UIbD0sVF+vxQhsPSxUX6/EYontJ+P8D+T5Z4WS8fjPxQhsPSxUX6/EYontJ+P8D8UIbD0sVF+vximcd6RQOS/X71JB4zs6r/8rGvxlpH6PxQhsPSxUX6/tZWTyJzl8z/iGMFpNyT+vximcd6RQOS/mgHpQD1497+iHTNrLynbv9v8dj9Fc+o/U+oQGEPM8L8Riie0n4/wPxQhsPSxUX6/FCGw9LFRfr8m3gUzdqLLv1PqEBhDzPC//Kxr8ZaR+j9DkhY/Gr0AwBXcuhNbvck/WaH/3Jk79z8UIbD0sVF+v9v8dj9Fc+o/GKZx3pFA5L/39XwsQCL0v/ysa/GWkfo/X71JB4zs6r9Zof/cmTv3P5oB6UA9ePe/Pg1VVTrO+r+1lZPInOXzP8btjaHF9ANAU+oQGEPM8L8Riie0n4/wPz4NVVU6zvq/Fdy6E1u9yT9fvUkHjOzqvxQhsPSxUX6/X71JB4zs6r8m3gUzdqLLvxQhsPSxUX6/U+oQGEPM8L87/y9AwfUIQF+9SQeM7Oq/X71JB4zs6r8UIbD0sVF+vxQhsPSxUX6/X71JB4zs6r9Zof/cmTv3PxGKJ7Sfj/A/FCGw9LFRfr/39XwsQCL0v/ysa/GWkfo/k+WeFkvH4z8V3LoTW73JP1+9SQeM7Oq/k+WeFkvH4z8YpnHekUDkv5PlnhZLx+M/FCGw9LFRfr9DkhY/Gr0AwNv8dj9Fc+o/EYontJ+P8D8YpnHekUDkv5PlnhZLx+M/k+WeFkvH4z8m3gUzdqLLvxGKJ7Sfj/A/k+WeFkvH4z+aAelAPXj3v5oB6UA9ePe/FCGw9LFRfr+1lZPInOXzP/f1fCxAIvS/IuIhjcieAEDmnYJTFxMEwBGKJ7Sfj/A/Jt4FM3aiy7+aAelAPXj3vxQhsPSxUX6/Fdy6E1u9yT8UIbD0sVF+v1+9SQeM7Oq/Fdy6E1u9yT8YpnHekUDkv/f1fCxAIvS/FCGw9LFRfr/8rGvxlpH6P1PqEBhDzPC/k+WeFkvH4z8UIbD0sVF+vybeBTN2osu/2/x2P0Vz6j8Riie0n4/wPxGKJ7Sfj/A/mZyN26E22j/39XwsQCL0vxXcuhNbvck/9/V8LEAi9L8i4iGNyJ4AQPf1fCxAIvS/Fdy6E1u9yT+T5Z4WS8fjPxQhsPSxUX6/Jt4FM3aiy78UIbD0sVF+v5mcjduhNto/FCGw9LFRfr9Zof/cmTv3P5mcjduhNto/Jt4FM3aiy79fvUkHjOzqv5mcjduhNto/GKZx3pFA5L8m3gUzdqLLvxQhsPSxUX6/X71JB4zs6r8=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[169]},\"seasonal\":[-14,4,-4,8,-7,-4,5,5,-1,20,1,-11,-3,-6,2,-7,0,-8,-1,0,0,9,1,5,-7,4,-2,4,-4,-1,2,8,0,19,0,-7,-5,-7,-1,-8,-2,-7,-2,-2,-2,9,2,5,-2,3,-1,1,-1,1,0,10,3,17,-1,-4,-6,-7,-5,-10,-4,-5,-3,-3,-4,10,3,5,3,3,0,-1,0,2,-1,13,6,16,-3,-1,-7,-8,-8,-11,-6,-4,-5,-4,-6,11,3,6,4,3,0,-1,1,0,1,17,6,17,-4,0,-4,-7,-9,-11,-4,-4,-7,-4,-5,10,1,5,1,3,-1,-1,0,-4,3,21,5,17,-4,0,-2,-7,-10,-10,-1,-4,-9,-5,-4,9,0,5,-2,4,-1,0,0,-8,5,25,5,18,-5,-1,0,-7,-11,-9,3,-4,-11,-5,-2,8,-4,5,-7],\"trend\":[34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,35,35,35,35,35,35,35,35,35,35,36,36,36,36,36,36,36,36,35,35,35,35,35,35,35,35,34,34,34,34,34,34,34,34,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,32,32,32,32,32,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,34,34,34,34,33,33,33,33,33,33,33,33,33,33,33,32,32,32,32,32,32,32,32,32,32,32,32,32,33,33,33,33,33,33,33,33,33,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34],\"weights\":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},\"selected\":{\"id\":\"8428\"},\"selection_policy\":{\"id\":\"8427\"}},\"id\":\"8347\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.2},\"fill_color\":{\"value\":\"blue\"},\"hatch_alpha\":{\"value\":0.2},\"hatch_color\":{\"value\":\"blue\"},\"line_alpha\":{\"value\":0.2},\"line_color\":{\"value\":\"blue\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"score\"}},\"id\":\"8555\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"8537\",\"type\":\"DataRange1d\"},{\"attributes\":{\"coordinates\":null,\"formatter\":{\"id\":\"8549\"},\"group\":null,\"major_label_policy\":{\"id\":\"8571\"},\"ticker\":{\"id\":\"8544\"}},\"id\":\"8543\",\"type\":\"DatetimeAxis\"},{\"attributes\":{},\"id\":\"8539\",\"type\":\"LinearScale\"},{\"attributes\":{\"coordinates\":null,\"group\":null,\"text\":\"Range Selector\"},\"id\":\"8533\",\"type\":\"Title\"},{\"attributes\":{\"num_minor_ticks\":5,\"tickers\":[{\"id\":\"8592\"},{\"id\":\"8593\"},{\"id\":\"8594\"},{\"id\":\"8595\"},{\"id\":\"8596\"},{\"id\":\"8597\"},{\"id\":\"8598\"},{\"id\":\"8599\"},{\"id\":\"8600\"},{\"id\":\"8601\"},{\"id\":\"8602\"},{\"id\":\"8603\"}]},\"id\":\"8544\",\"type\":\"DatetimeTicker\"},{\"attributes\":{\"axis\":{\"id\":\"8543\"},\"coordinates\":null,\"group\":null,\"ticker\":null},\"id\":\"8546\",\"type\":\"Grid\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"blue\"},\"hatch_alpha\":{\"value\":0.1},\"hatch_color\":{\"value\":\"blue\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"blue\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"score\"}},\"id\":\"8554\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"8541\",\"type\":\"LinearScale\"},{\"attributes\":{\"overlay\":{\"id\":\"8559\"},\"x_range\":{\"id\":\"8383\"},\"y_range\":null},\"id\":\"8558\",\"type\":\"RangeTool\"},{\"attributes\":{\"source\":{\"id\":\"8551\"}},\"id\":\"8557\",\"type\":\"CDSView\"},{\"attributes\":{\"bottom_units\":\"screen\",\"coordinates\":null,\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"group\":null,\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"8404\",\"type\":\"BoxAnnotation\"},{\"attributes\":{},\"id\":\"8589\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"axis_label\":\"date_clock\",\"coordinates\":null,\"formatter\":{\"id\":\"8412\"},\"group\":null,\"major_label_policy\":{\"id\":\"8426\"},\"ticker\":{\"id\":\"8392\"}},\"id\":\"8391\",\"type\":\"DatetimeAxis\"},{\"attributes\":{\"num_minor_ticks\":10,\"tickers\":[{\"id\":\"8430\"},{\"id\":\"8431\"},{\"id\":\"8432\"},{\"id\":\"8433\"},{\"id\":\"8434\"},{\"id\":\"8435\"},{\"id\":\"8436\"},{\"id\":\"8437\"},{\"id\":\"8438\"},{\"id\":\"8439\"},{\"id\":\"8440\"},{\"id\":\"8441\"}]},\"id\":\"8392\",\"type\":\"DatetimeTicker\"},{\"attributes\":{},\"id\":\"8424\",\"type\":\"AllLabels\"},{\"attributes\":{\"base\":60,\"mantissas\":[1,2,5,10,15,20,30],\"max_interval\":1800000.0,\"min_interval\":1000.0,\"num_minor_ticks\":0},\"id\":\"8593\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{},\"id\":\"8426\",\"type\":\"AllLabels\"},{\"attributes\":{\"dimensions\":\"width\"},\"id\":\"8403\",\"type\":\"PanTool\"},{\"attributes\":{\"dimensions\":\"width\"},\"id\":\"8399\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"base\":24,\"mantissas\":[1,2,4,6,8,12],\"max_interval\":43200000.0,\"min_interval\":3600000.0,\"num_minor_ticks\":0},\"id\":\"8594\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{},\"id\":\"8402\",\"type\":\"SaveTool\"},{\"attributes\":{\"axis_label\":\"count\",\"coordinates\":null,\"formatter\":{\"id\":\"8414\"},\"group\":null,\"major_label_policy\":{\"id\":\"8424\"},\"ticker\":{\"id\":\"8396\"}},\"id\":\"8395\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"8427\",\"type\":\"UnionRenderers\"},{\"attributes\":{},\"id\":\"8590\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"8428\",\"type\":\"Selection\"},{\"attributes\":{\"overlay\":{\"id\":\"8404\"}},\"id\":\"8400\",\"type\":\"BoxZoomTool\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"green\",\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"baseline\"}},\"id\":\"8446\",\"type\":\"Line\"},{\"attributes\":{\"axis\":{\"id\":\"8395\"},\"coordinates\":null,\"dimension\":1,\"grid_line_color\":null,\"group\":null,\"ticker\":null},\"id\":\"8398\",\"type\":\"Grid\"},{\"attributes\":{\"coordinates\":null,\"data_source\":{\"id\":\"8347\"},\"glyph\":{\"id\":\"8445\"},\"group\":null,\"hover_glyph\":null,\"muted_glyph\":{\"id\":\"8447\"},\"nonselection_glyph\":{\"id\":\"8446\"},\"view\":{\"id\":\"8449\"}},\"id\":\"8448\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"days\":[1,4,7,10,13,16,19,22,25,28]},\"id\":\"8596\",\"type\":\"DaysTicker\"},{\"attributes\":{},\"id\":\"8389\",\"type\":\"LinearScale\"},{\"attributes\":{\"source\":{\"id\":\"8347\"}},\"id\":\"8449\",\"type\":\"CDSView\"},{\"attributes\":{\"line_color\":\"green\",\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"baseline\"}},\"id\":\"8445\",\"type\":\"Line\"},{\"attributes\":{\"line_alpha\":0.2,\"line_color\":\"green\",\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"baseline\"}},\"id\":\"8447\",\"type\":\"Line\"},{\"attributes\":{\"data\":{\"baseline\":[36],\"count\":[52],\"date_clock\":{\"__ndarray__\":\"AACAFOYweEI=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1]},\"index\":[94],\"level_0\":[0],\"residual\":[15],\"score\":{\"__ndarray__\":\"O/8vQMH1CEA=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[1]},\"seasonal\":[3],\"trend\":[33],\"weights\":[1]},\"selected\":{\"id\":\"8486\"},\"selection_policy\":{\"id\":\"8485\"}},\"id\":\"8471\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"mantissas\":[1,2,5],\"max_interval\":500.0,\"num_minor_ticks\":0},\"id\":\"8592\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{\"below\":[{\"id\":\"8391\"}],\"center\":[{\"id\":\"8394\"},{\"id\":\"8398\"},{\"id\":\"8442\"}],\"height\":300,\"left\":[{\"id\":\"8395\"}],\"min_border_left\":50,\"renderers\":[{\"id\":\"8420\"},{\"id\":\"8448\"},{\"id\":\"8476\"}],\"title\":{\"id\":\"8381\"},\"toolbar\":{\"id\":\"8405\"},\"toolbar_location\":\"above\",\"width\":1200,\"x_range\":{\"id\":\"8383\"},\"x_scale\":{\"id\":\"8387\"},\"y_range\":{\"id\":\"8385\"},\"y_scale\":{\"id\":\"8389\"}},\"id\":\"8380\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"coordinates\":null,\"data_source\":{\"id\":\"8551\"},\"glyph\":{\"id\":\"8553\"},\"group\":null,\"hover_glyph\":null,\"muted_glyph\":{\"id\":\"8555\"},\"nonselection_glyph\":{\"id\":\"8554\"},\"view\":{\"id\":\"8557\"}},\"id\":\"8556\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"label\":{\"value\":\"baseline\"},\"renderers\":[{\"id\":\"8448\"}]},\"id\":\"8470\",\"type\":\"LegendItem\"},{\"attributes\":{\"days\":[1,8,15,22]},\"id\":\"8597\",\"type\":\"DaysTicker\"},{\"attributes\":{\"base\":60,\"mantissas\":[1,2,5,10,15,20,30],\"max_interval\":1800000.0,\"min_interval\":1000.0,\"num_minor_ticks\":0},\"id\":\"8431\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{\"base\":24,\"mantissas\":[1,2,4,6,8,12],\"max_interval\":43200000.0,\"min_interval\":3600000.0,\"num_minor_ticks\":0},\"id\":\"8432\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{\"days\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]},\"id\":\"8595\",\"type\":\"DaysTicker\"},{\"attributes\":{\"months\":[0,4,8]},\"id\":\"8439\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"navy\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8417\",\"type\":\"Circle\"},{\"attributes\":{\"months\":[0,1,2,3,4,5,6,7,8,9,10,11]},\"id\":\"8437\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"days\":[1,15]},\"id\":\"8436\",\"type\":\"DaysTicker\"},{\"attributes\":{},\"id\":\"8401\",\"type\":\"ResetTool\"},{\"attributes\":{\"days\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]},\"id\":\"8433\",\"type\":\"DaysTicker\"},{\"attributes\":{\"days\":[1,4,7,10,13,16,19,22,25,28]},\"id\":\"8434\",\"type\":\"DaysTicker\"},{\"attributes\":{\"days\":[1,8,15,22]},\"id\":\"8435\",\"type\":\"DaysTicker\"},{\"attributes\":{\"axis\":{\"id\":\"8391\"},\"coordinates\":null,\"grid_line_color\":null,\"group\":null,\"ticker\":null},\"id\":\"8394\",\"type\":\"Grid\"},{\"attributes\":{\"tools\":[{\"id\":\"8348\"},{\"id\":\"8399\"},{\"id\":\"8400\"},{\"id\":\"8401\"},{\"id\":\"8402\"},{\"id\":\"8403\"}]},\"id\":\"8405\",\"type\":\"Toolbar\"},{\"attributes\":{\"months\":[0,2,4,6,8,10]},\"id\":\"8438\",\"type\":\"MonthsTicker\"},{\"attributes\":{},\"id\":\"8396\",\"type\":\"BasicTicker\"},{\"attributes\":{\"months\":[0,1,2,3,4,5,6,7,8,9,10,11]},\"id\":\"8599\",\"type\":\"MonthsTicker\"},{\"attributes\":{},\"id\":\"8603\",\"type\":\"YearsTicker\"},{\"attributes\":{\"days\":[1,15]},\"id\":\"8598\",\"type\":\"DaysTicker\"},{\"attributes\":{\"months\":[0,6]},\"id\":\"8602\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"end\":1662690240000.0,\"start\":1662024960000.0},\"id\":\"8383\",\"type\":\"Range1d\"},{\"attributes\":{\"months\":[0,2,4,6,8,10]},\"id\":\"8600\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"align\":\"right\",\"coordinates\":null,\"group\":null,\"text\":\"Drag the middle or edges of the selection box to change the range in the main chart\",\"text_font_size\":\"10px\"},\"id\":\"8548\",\"type\":\"Title\"},{\"attributes\":{\"months\":[0,4,8]},\"id\":\"8601\",\"type\":\"MonthsTicker\"},{\"attributes\":{},\"id\":\"8486\",\"type\":\"Selection\"},{\"attributes\":{\"format\":\"00\"},\"id\":\"8414\",\"type\":\"NumeralTickFormatter\"},{\"attributes\":{\"label\":{\"value\":\"anomalies\"},\"renderers\":[{\"id\":\"8476\"}]},\"id\":\"8500\",\"type\":\"LegendItem\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.2},\"fill_color\":{\"value\":\"firebrick\"},\"hatch_color\":{\"value\":\"firebrick\"},\"line_color\":{\"value\":\"firebrick\"},\"marker\":{\"value\":\"circle_x\"},\"size\":{\"value\":12},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8473\",\"type\":\"Scatter\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.2},\"fill_color\":{\"value\":\"firebrick\"},\"hatch_alpha\":{\"value\":0.2},\"hatch_color\":{\"value\":\"firebrick\"},\"line_alpha\":{\"value\":0.2},\"line_color\":{\"value\":\"firebrick\"},\"marker\":{\"value\":\"circle_x\"},\"size\":{\"value\":12},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8475\",\"type\":\"Scatter\"},{\"attributes\":{\"coordinates\":null,\"group\":null,\"text\":\"Time Series Anomalies Visualization\"},\"id\":\"8381\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"8387\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"8485\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"source\":{\"id\":\"8471\"}},\"id\":\"8477\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"8571\",\"type\":\"AllLabels\"},{\"attributes\":{\"coordinates\":null,\"data_source\":{\"id\":\"8471\"},\"glyph\":{\"id\":\"8473\"},\"group\":null,\"hover_glyph\":null,\"muted_glyph\":{\"id\":\"8475\"},\"nonselection_glyph\":{\"id\":\"8474\"},\"view\":{\"id\":\"8477\"}},\"id\":\"8476\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"days\":[\"%m-%d %H:%M\"],\"hours\":[\"%H:%M:%S\"],\"milliseconds\":[\"%H:%M:%S.%3N\"],\"minutes\":[\"%H:%M:%S\"],\"seconds\":[\"%H:%M:%S\"]},\"id\":\"8412\",\"type\":\"DatetimeTickFormatter\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"firebrick\"},\"hatch_alpha\":{\"value\":0.1},\"hatch_color\":{\"value\":\"firebrick\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"firebrick\"},\"marker\":{\"value\":\"circle_x\"},\"size\":{\"value\":12},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"count\"}},\"id\":\"8474\",\"type\":\"Scatter\"},{\"attributes\":{\"days\":[\"%m-%d %H:%M\"],\"hours\":[\"%H:%M:%S\"],\"milliseconds\":[\"%H:%M:%S.%3N\"],\"minutes\":[\"%H:%M:%S\"],\"seconds\":[\"%H:%M:%S\"]},\"id\":\"8549\",\"type\":\"DatetimeTickFormatter\"},{\"attributes\":{\"fill_color\":{\"value\":\"blue\"},\"hatch_color\":{\"value\":\"blue\"},\"line_color\":{\"value\":\"blue\"},\"x\":{\"field\":\"date_clock\"},\"y\":{\"field\":\"score\"}},\"id\":\"8553\",\"type\":\"Circle\"},{\"attributes\":{\"data\":{\"anomalies\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],\"baseline\":[19,38,29,42,26,29,39,40,32,54,35,23,30,27,36,27,34,25,33,33,34,43,35,40,26,38,31,39,30,33,37,43,36,54,35,28,30,28,34,26,33,28,33,34,33,46,38,41,33,39,34,36,33,36,35,45,38,52,33,30,28,26,29,23,29,28,29,30,28,43,36,39,36,36,33,31,34,35,31,46,39,49,29,31,25,24,24,21,26,28,27,28,26,44,36,39,38,36,32,32,34,32,34,50,40,50,29,33,29,26,24,22,29,29,26,29,27,44,34,38,34,36,31,31,33,28,35,54,38,50,28,31,30,25,22,22,32,29,24,28,29,43,33,39,31,38,32,33,34,25,40,60,39,52,29,33,35,27,23,24,38,30,23,29,32,43,30,40,27],\"count\":[17,40,26,46,24,26,46,48,28,55,35,24,29,29,36,30,29,25,31,31,38,47,37,40,25,34,35,32,35,38,33,26,44,52,40,26,38,27,34,26,39,29,38,39,33,38,39,42,39,43,35,42,34,33,31,54,38,59,24,27,21,24,34,18,35,29,29,29,23,52,26,41,44,37,38,28,28,44,27,54,32,41,36,44,20,30,16,23,22,28,23,27,26,39,52,35,34,37,32,28,42,38,34,44,49,54,31,29,33,23,28,22,19,34,32,26,31,48,33,44,38,29,24,32,40,22,46,42,44,49,21,31,32,25,18,24,29,23,24,37,24,47,33,38,36,44,38,36,28,27,34,71,33,54,33,34,34,28,26,24,46,33,22,25,35,40,29,40,23],\"date_clock\":{\"__ndarray__\":\"AABQW6MveEIAADjKpi94QgAAIDmqL3hCAAAIqK0veEIAAPAWsS94QgAA2IW0L3hCAADA9LcveEIAAKhjuy94QgAAkNK+L3hCAAB4QcIveEIAAGCwxS94QgAASB/JL3hCAAAwjswveEIAABj9zy94QgAAAGzTL3hCAADo2tYveEIAANBJ2i94QgAAuLjdL3hCAACgJ+EveEIAAIiW5C94QgAAcAXoL3hCAABYdOsveEIAAEDj7i94QgAAKFLyL3hCAAAQwfUveEIAAPgv+S94QgAA4J78L3hCAADIDQAweEIAALB8AzB4QgAAmOsGMHhCAACAWgoweEIAAGjJDTB4QgAAUDgRMHhCAAA4pxQweEIAACAWGDB4QgAACIUbMHhCAADw8x4weEIAANhiIjB4QgAAwNElMHhCAACoQCkweEIAAJCvLDB4QgAAeB4wMHhCAABgjTMweEIAAEj8NjB4QgAAMGs6MHhCAAAY2j0weEIAAABJQTB4QgAA6LdEMHhCAADQJkgweEIAALiVSzB4QgAAoARPMHhCAACIc1IweEIAAHDiVTB4QgAAWFFZMHhCAABAwFwweEIAACgvYDB4QgAAEJ5jMHhCAAD4DGcweEIAAOB7ajB4QgAAyOptMHhCAACwWXEweEIAAJjIdDB4QgAAgDd4MHhCAABopnsweEIAAFAVfzB4QgAAOISCMHhCAAAg84UweEIAAAhiiTB4QgAA8NCMMHhCAADYP5AweEIAAMCukzB4QgAAqB2XMHhCAACQjJoweEIAAHj7nTB4QgAAYGqhMHhCAABI2aQweEIAADBIqDB4QgAAGLerMHhCAAAAJq8weEIAAOiUsjB4QgAA0AO2MHhCAAC4crkweEIAAKDhvDB4QgAAiFDAMHhCAABwv8MweEIAAFguxzB4QgAAQJ3KMHhCAAAoDM4weEIAABB70TB4QgAA+OnUMHhCAADgWNgweEIAAMjH2zB4QgAAsDbfMHhCAACYpeIweEIAAIAU5jB4QgAAaIPpMHhCAABQ8uwweEIAADhh8DB4QgAAINDzMHhCAAAIP/cweEIAAPCt+jB4QgAA2Bz+MHhCAADAiwExeEIAAKj6BDF4QgAAkGkIMXhCAAB42AsxeEIAAGBHDzF4QgAASLYSMXhCAAAwJRYxeEIAABiUGTF4QgAAAAMdMXhCAADocSAxeEIAANDgIzF4QgAAuE8nMXhCAACgvioxeEIAAIgtLjF4QgAAcJwxMXhCAABYCzUxeEIAAEB6ODF4QgAAKOk7MXhCAAAQWD8xeEIAAPjGQjF4QgAA4DVGMXhCAADIpEkxeEIAALATTTF4QgAAmIJQMXhCAACA8VMxeEIAAGhgVzF4QgAAUM9aMXhCAAA4Pl4xeEIAACCtYTF4QgAACBxlMXhCAADwimgxeEIAANj5azF4QgAAwGhvMXhCAACo13IxeEIAAJBGdjF4QgAAeLV5MXhCAABgJH0xeEIAAEiTgDF4QgAAMAKEMXhCAAAYcYcxeEIAAADgijF4QgAA6E6OMXhCAADQvZExeEIAALgslTF4QgAAoJuYMXhCAACICpwxeEIAAHB5nzF4QgAAWOiiMXhCAABAV6YxeEIAACjGqTF4QgAAEDWtMXhCAAD4o7AxeEIAAOAStDF4QgAAyIG3MXhCAACw8LoxeEIAAJhfvjF4QgAAgM7BMXhCAABoPcUxeEIAAFCsyDF4QgAAOBvMMXhCAAAgis8xeEIAAAj50jF4QgAA8GfWMXhCAADY1tkxeEIAAMBF3TF4QgAAqLTgMXhCAACQI+QxeEI=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[169]},\"index\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168],\"residual\":[-2,1,-3,3,-2,-3,6,7,-4,0,0,0,-1,1,0,2,-5,0,-2,-2,3,3,1,0,-1,-4,3,-7,4,4,-4,-17,7,-2,4,-2,7,-1,0,0,5,0,4,4,0,-8,0,0,5,3,0,5,0,-3,-4,8,0,6,-9,-3,-7,-2,4,-5,5,0,0,-1,-5,8,-10,1,7,0,4,-3,-6,8,-4,7,-7,-8,6,12,-5,5,-8,1,-4,0,-4,-1,0,-5,15,-4,-4,0,0,-4,7,5,0,-6,8,3,1,-4,3,-3,3,0,-10,4,5,-3,3,3,-1,5,3,-7,-7,0,6,-6,10,-12,5,-1,-7,0,1,0,-4,1,-3,-6,0,8,-5,3,0,-1,4,5,5,2,-6,1,-6,10,-6,1,3,0,-1,0,2,0,7,2,-1,-4,2,-3,-1,0,-4],\"score\":{\"__ndarray__\":\"oh0zay8p278V3LoTW73JPximcd6RQOS/k+WeFkvH4z+iHTNrLynbvximcd6RQOS/tZWTyJzl8z9Zof/cmTv3P1+9SQeM7Oq/FCGw9LFRfr8UIbD0sVF+vxQhsPSxUX6/Jt4FM3aiy78V3LoTW73JPxQhsPSxUX6/mZyN26E22j9T6hAYQ8zwvxQhsPSxUX6/oh0zay8p27+iHTNrLynbv5PlnhZLx+M/k+WeFkvH4z8V3LoTW73JPxQhsPSxUX6/Jt4FM3aiy79fvUkHjOzqv5PlnhZLx+M/mgHpQD1497/b/HY/RXPqP9v8dj9Fc+o/X71JB4zs6r8Au5AGEGoMwFmh/9yZO/c/oh0zay8p27/b/HY/RXPqP6IdM2svKdu/WaH/3Jk79z8m3gUzdqLLvxQhsPSxUX6/FCGw9LFRfr8Riie0n4/wPxQhsPSxUX6/2/x2P0Vz6j/b/HY/RXPqPxQhsPSxUX6/Pg1VVTrO+r8UIbD0sVF+vxQhsPSxUX6/EYontJ+P8D+T5Z4WS8fjPxQhsPSxUX6/EYontJ+P8D8UIbD0sVF+vximcd6RQOS/X71JB4zs6r/8rGvxlpH6PxQhsPSxUX6/tZWTyJzl8z/iGMFpNyT+vximcd6RQOS/mgHpQD1497+iHTNrLynbv9v8dj9Fc+o/U+oQGEPM8L8Riie0n4/wPxQhsPSxUX6/FCGw9LFRfr8m3gUzdqLLv1PqEBhDzPC//Kxr8ZaR+j9DkhY/Gr0AwBXcuhNbvck/WaH/3Jk79z8UIbD0sVF+v9v8dj9Fc+o/GKZx3pFA5L/39XwsQCL0v/ysa/GWkfo/X71JB4zs6r9Zof/cmTv3P5oB6UA9ePe/Pg1VVTrO+r+1lZPInOXzP8btjaHF9ANAU+oQGEPM8L8Riie0n4/wPz4NVVU6zvq/Fdy6E1u9yT9fvUkHjOzqvxQhsPSxUX6/X71JB4zs6r8m3gUzdqLLvxQhsPSxUX6/U+oQGEPM8L87/y9AwfUIQF+9SQeM7Oq/X71JB4zs6r8UIbD0sVF+vxQhsPSxUX6/X71JB4zs6r9Zof/cmTv3PxGKJ7Sfj/A/FCGw9LFRfr/39XwsQCL0v/ysa/GWkfo/k+WeFkvH4z8V3LoTW73JP1+9SQeM7Oq/k+WeFkvH4z8YpnHekUDkv5PlnhZLx+M/FCGw9LFRfr9DkhY/Gr0AwNv8dj9Fc+o/EYontJ+P8D8YpnHekUDkv5PlnhZLx+M/k+WeFkvH4z8m3gUzdqLLvxGKJ7Sfj/A/k+WeFkvH4z+aAelAPXj3v5oB6UA9ePe/FCGw9LFRfr+1lZPInOXzP/f1fCxAIvS/IuIhjcieAEDmnYJTFxMEwBGKJ7Sfj/A/Jt4FM3aiy7+aAelAPXj3vxQhsPSxUX6/Fdy6E1u9yT8UIbD0sVF+v1+9SQeM7Oq/Fdy6E1u9yT8YpnHekUDkv/f1fCxAIvS/FCGw9LFRfr/8rGvxlpH6P1PqEBhDzPC/k+WeFkvH4z8UIbD0sVF+vybeBTN2osu/2/x2P0Vz6j8Riie0n4/wPxGKJ7Sfj/A/mZyN26E22j/39XwsQCL0vxXcuhNbvck/9/V8LEAi9L8i4iGNyJ4AQPf1fCxAIvS/Fdy6E1u9yT+T5Z4WS8fjPxQhsPSxUX6/Jt4FM3aiy78UIbD0sVF+v5mcjduhNto/FCGw9LFRfr9Zof/cmTv3P5mcjduhNto/Jt4FM3aiy79fvUkHjOzqv5mcjduhNto/GKZx3pFA5L8m3gUzdqLLvxQhsPSxUX6/X71JB4zs6r8=\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[169]},\"seasonal\":[-14,4,-4,8,-7,-4,5,5,-1,20,1,-11,-3,-6,2,-7,0,-8,-1,0,0,9,1,5,-7,4,-2,4,-4,-1,2,8,0,19,0,-7,-5,-7,-1,-8,-2,-7,-2,-2,-2,9,2,5,-2,3,-1,1,-1,1,0,10,3,17,-1,-4,-6,-7,-5,-10,-4,-5,-3,-3,-4,10,3,5,3,3,0,-1,0,2,-1,13,6,16,-3,-1,-7,-8,-8,-11,-6,-4,-5,-4,-6,11,3,6,4,3,0,-1,1,0,1,17,6,17,-4,0,-4,-7,-9,-11,-4,-4,-7,-4,-5,10,1,5,1,3,-1,-1,0,-4,3,21,5,17,-4,0,-2,-7,-10,-10,-1,-4,-9,-5,-4,9,0,5,-2,4,-1,0,0,-8,5,25,5,18,-5,-1,0,-7,-11,-9,3,-4,-11,-5,-2,8,-4,5,-7],\"trend\":[34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,35,35,35,35,35,35,35,35,35,35,36,36,36,36,36,36,36,36,35,35,35,35,35,35,35,35,34,34,34,34,34,34,34,34,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,32,32,32,32,32,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,34,34,34,34,33,33,33,33,33,33,33,33,33,33,33,32,32,32,32,32,32,32,32,32,32,32,32,32,33,33,33,33,33,33,33,33,33,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34],\"weights\":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},\"selected\":{\"id\":\"8590\"},\"selection_policy\":{\"id\":\"8589\"}},\"id\":\"8551\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"active_multi\":{\"id\":\"8558\"},\"tools\":[{\"id\":\"8558\"}]},\"id\":\"8547\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"8385\",\"type\":\"DataRange1d\"}],\"root_ids\":[\"8563\"]},\"title\":\"Bokeh Application\",\"version\":\"2.4.2\"}};\n const render_items = [{\"docid\":\"70465022-a210-4640-b276-2bf87def5374\",\"root_ids\":[\"8563\"],\"roots\":{\"8563\":\"33e232ab-a5ae-4857-bd62-01520a0427ac\"}}];\n root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n\n }\n if (root.Bokeh !== undefined) {\n embed_document(root);\n } else {\n let attempts = 0;\n const timer = setInterval(function(root) {\n if (root.Bokeh !== undefined) {\n clearInterval(timer);\n embed_document(root);\n } else {\n attempts++;\n if (attempts > 100) {\n clearInterval(timer);\n console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n }\n }\n }, 10, root)\n }\n})(window);", 650 | "application/vnd.bokehjs_exec.v0+json": "" 651 | }, 652 | "metadata": { 653 | "application/vnd.bokehjs_exec.v0+json": { 654 | "id": "8563" 655 | } 656 | }, 657 | "output_type": "display_data" 658 | } 659 | ], 660 | "source": [ 661 | "from msticpy.nbtools.timeseries import display_timeseries_anomolies\n", 662 | "\n", 663 | "# display_timeseries_anomolies関数の使用上、事前に結果のタイムフィールドにdatetime型を適用し、かつ時系列順に並び替える\n", 664 | "output['date_clock'] = pd.to_datetime(output['date_clock'])\n", 665 | "output = output.sort_values(by='date_clock')\n", 666 | "\n", 667 | "timeseries_anomalies_plot = display_timeseries_anomolies(\n", 668 | " data=output, \n", 669 | " y='count',\n", 670 | " time_column='date_clock'\n", 671 | " )" 672 | ] 673 | }, 674 | { 675 | "attachments": {}, 676 | "cell_type": "markdown", 677 | "metadata": {}, 678 | "source": [] 679 | } 680 | ], 681 | "metadata": { 682 | "kernelspec": { 683 | "display_name": "Python 3", 684 | "language": "python", 685 | "name": "python3" 686 | }, 687 | "language_info": { 688 | "codemirror_mode": { 689 | "name": "ipython", 690 | "version": 3 691 | }, 692 | "file_extension": ".py", 693 | "mimetype": "text/x-python", 694 | "name": "python", 695 | "nbconvert_exporter": "python", 696 | "pygments_lexer": "ipython3", 697 | "version": "3.9.12" 698 | }, 699 | "orig_nbformat": 4 700 | }, 701 | "nbformat": 4, 702 | "nbformat_minor": 2 703 | } 704 | --------------------------------------------------------------------------------