├── .gitignore ├── LICENSE ├── README.md ├── checkpoints ├── checkpoint ├── seq2seq-4805.data-00000-of-00001 ├── seq2seq-4805.index ├── seq2seq-4805.meta └── seq2seq.meta ├── datasets ├── vulnbank_anomaly.txt └── vulnbank_train.txt ├── environment.yml ├── requirements.txt ├── seq2seq.ipynb ├── slides └── detecting_web_attacks_rnn.pdf └── utils ├── __init__.py ├── reader.py ├── utils.py ├── vocab.json └── vocab.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .idea 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Seq2Seq for Web Attack Detection 2 | This is the implementation of the Seq2Seq model for web attack detection. The Seq2Seq model is usually used in Neural Machine Translation. The main goal of this project is to demonstrate the relevance of the NLP approach for web security. 3 | 4 | 5 | The problem of web attack detection is considered in terms of anomaly detection. On the training step the model is given only benign HTTP requests. On the testing step the model determines whether a received request is anomalous or not. 6 | 7 | Check out our [slides](/slides/detecting_web_attacks_rnn.pdf) and a post at [AI Village](https://aivillage.org/posts/detecting-web-attacks-rnn/) ([DEFCON 26](https://www.defcon.org/)). 8 | 9 | ## Model 10 | The step-by-step solution is presented in [seq2seq.ipynb](seq2seq.ipynb) that contains the main stages such as a model initialization, training, validation, prediction and results. 11 | 12 | Unfortunately, github ui doesn't correctly visualize cell output with colored malicious parts of requests. So, we suggest to download the notebook or use this [link](https://nbviewer.jupyter.org/github/PositiveTechnologies/seq2seq-web-attack-detection/blob/master/seq2seq.ipynb) for correctly displaying cells outputs. 13 | 14 | 15 | ## Dataset 16 | The dataset contains data with 21991 benign and 1097 anomalous HTTP requests from a banking application. 17 | 18 | 19 | ## Running 20 | Please make sure that you have the same [requirements](requirements.txt) and python 2.7.* 21 | 22 | This repository contains environment.yml so it can be dockerized using [jupyter/repo2docker](https://github.com/jupyter/repo2docker). We have already dockerized it for you and you can run this playbook by 23 | 24 | ```bash 25 | docker run -it -p 8888:8888 montekki/seq2seq-web-attack-detection:latest jupyter notebook --ip=0.0.0.0 26 | ``` 27 | 28 | ## Authors 29 | 30 | * [Arseny Reutov](https://github.com/Raz0r) 31 | * [Fedor Sakharov](https://github.com/montekki) 32 | * [Irina Stepanyuk](https://github.com/idstep) 33 | * [Alexandra Murzina](https://github.com/amurzina) 34 | -------------------------------------------------------------------------------- /checkpoints/checkpoint: -------------------------------------------------------------------------------- 1 | model_checkpoint_path: "seq2seq-4805" 2 | all_model_checkpoint_paths: "seq2seq-4805" 3 | -------------------------------------------------------------------------------- /checkpoints/seq2seq-4805.data-00000-of-00001: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/checkpoints/seq2seq-4805.data-00000-of-00001 -------------------------------------------------------------------------------- /checkpoints/seq2seq-4805.index: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/checkpoints/seq2seq-4805.index -------------------------------------------------------------------------------- /checkpoints/seq2seq-4805.meta: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/checkpoints/seq2seq-4805.meta -------------------------------------------------------------------------------- /checkpoints/seq2seq.meta: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/checkpoints/seq2seq.meta -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | dependencies: 2 | - python=2.7.* 3 | - pip: 4 | - nteract_on_jupyter==1.8.1 5 | - colorama==0.3.9 6 | - numpy==1.14.5 7 | - scikit-learn==0.19.2 8 | - scipy==1.1.0 9 | - tensorflow==1.9.0 10 | - matplotlib==2.2.3 11 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | colorama==0.3.9 2 | numpy==1.14.5 3 | scikit-learn==0.19.2 4 | scipy==1.1.0 5 | tensorflow==1.9.0 6 | -------------------------------------------------------------------------------- /seq2seq.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from __future__ import print_function\n", 10 | "from __future__ import absolute_import\n", 11 | "from __future__ import division\n", 12 | "\n", 13 | "import os\n", 14 | "import numpy as np\n", 15 | "import tensorflow as tf\n", 16 | "import matplotlib.pyplot as plt\n", 17 | "import timeit\n", 18 | "\n", 19 | "from colorama import Fore\n", 20 | "from sklearn.metrics import auc, roc_curve, precision_score, recall_score\n", 21 | "\n", 22 | "from utils.vocab import Vocabulary\n", 23 | "from utils.reader import Data\n", 24 | "from utils.utils import print_progress, create_checkpoints_dir" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 3, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n", 34 | "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "Main model parameters:\n", 42 | "- *batch_size* - the number of samples in a batch\n", 43 | "- *embed_size* - the dimension of embedding space (should be less than vocabulary size)\n", 44 | "- *hidden_size* - the number of hidden states in lstm \n", 45 | "- *num_layers* - the number of lstm blocks\n", 46 | "- *checkpoints* - path to checkpoint directory\n", 47 | "- *std_factor* - the number of stds that is used for defining a model threshold\n", 48 | "- *dropout* - the probability that each element is kept\n", 49 | "- *vocab* - the Vocabulary object" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 4, 55 | "metadata": {}, 56 | "outputs": [ 57 | { 58 | "name": "stdout", 59 | "output_type": "stream", 60 | "text": [ 61 | "Downloaded 21991 samples\n" 62 | ] 63 | } 64 | ], 65 | "source": [ 66 | "params = {\n", 67 | " \"batch_size\": 128,\n", 68 | " \"embed_size\": 64,\n", 69 | " \"hidden_size\": 64,\n", 70 | " \"num_layers\": 2,\n", 71 | " \"checkpoints\": \"./checkpoints/\",\n", 72 | " \"std_factor\": 6.,\n", 73 | " \"dropout\": 0.7,\n", 74 | "}\n", 75 | "\n", 76 | "path_normal_data = \"datasets/vulnbank_train.txt\"\n", 77 | "path_anomaly_data = \"datasets/vulnbank_anomaly.txt\"\n", 78 | "\n", 79 | "create_checkpoints_dir(params[\"checkpoints\"])\n", 80 | "\n", 81 | "vocab = Vocabulary()\n", 82 | "params[\"vocab\"] = vocab\n", 83 | "\n", 84 | "d = Data(path_normal_data)" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": {}, 90 | "source": [ 91 | "# Model " 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "In this part of the code the Sequence-to-Sequence model for determining anomalies is defined. \n", 99 | "The same sequences are fed to the input and output of the model. So the model learns to reconstruct them. At the stage of training and validation, only valid samples are submitted to the model. The validation phase is needed in order to initialize the threshold value." 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 5, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "class Seq2Seq():\n", 109 | " def __init__(self, args):\n", 110 | " tf.reset_default_graph()\n", 111 | "\n", 112 | " self.batch_size = tf.placeholder(tf.int32, [], name='batch_size')\n", 113 | " self.max_seq_len = tf.placeholder(tf.int32, [], name='max_seq_len')\n", 114 | " self.inputs = tf.placeholder(tf.int32, [None, None], name='inputs')\n", 115 | " self.targets = tf.placeholder(tf.int32, [None, None], name='targets')\n", 116 | " self.lengths = tf.placeholder(tf.int32, [None, ], name='lengths')\n", 117 | " self.dropout = tf.placeholder(tf.float32, name='dropout')\n", 118 | " \n", 119 | " self.num_layers = args['num_layers']\n", 120 | " self.hidden_size = args['hidden_size']\n", 121 | " self.vocab = args['vocab']\n", 122 | "\n", 123 | " dec_input = self._process_decoder_input(\n", 124 | " self.targets,\n", 125 | " self.vocab.vocab,\n", 126 | " tf.to_int32(self.batch_size))\n", 127 | "\n", 128 | " vocab_size = len(self.vocab.vocab)\n", 129 | "\n", 130 | " # Embeddings for inputs\n", 131 | " embed_initializer = tf.random_uniform_initializer(-np.sqrt(3), np.sqrt(3))\n", 132 | "\n", 133 | " with tf.variable_scope('embedding'):\n", 134 | " embeds = tf.get_variable(\n", 135 | " 'embed_matrix',\n", 136 | " [vocab_size, args['embed_size']],\n", 137 | " initializer=embed_initializer,\n", 138 | " dtype=tf.float32)\n", 139 | "\n", 140 | " enc_embed_input = tf.nn.embedding_lookup(embeds, self.inputs)\n", 141 | " \n", 142 | " enc_state = self._encoder(enc_embed_input)\n", 143 | " \n", 144 | " # Embeddings for outputs\n", 145 | " with tf.variable_scope('embedding', reuse=True):\n", 146 | " dec_embed_input = tf.nn.embedding_lookup(embeds, dec_input)\n", 147 | "\n", 148 | " dec_outputs = self._decoder(enc_state, dec_embed_input)\n", 149 | "\n", 150 | " weight, bias = self._weight_and_bias(args['hidden_size'], vocab_size)\n", 151 | " outputs = tf.reshape(dec_outputs[0].rnn_output, [-1, args['hidden_size']])\n", 152 | " logits = tf.matmul(outputs, weight) + bias\n", 153 | "\n", 154 | " logits = tf.reshape(logits, [-1, self.max_seq_len, vocab_size], name='logits')\n", 155 | " self.probs = tf.nn.softmax(logits, name='probs')\n", 156 | " self.decoder_outputs = tf.argmax(logits, axis=2)\n", 157 | "\n", 158 | " self.cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(\n", 159 | " logits=logits,\n", 160 | " labels=self.targets,\n", 161 | " name='cross_entropy')\n", 162 | " self.batch_loss = tf.identity(tf.reduce_mean(self.cross_entropy, axis=1), name='batch_loss')\n", 163 | " self.loss = tf.reduce_mean(self.cross_entropy)\n", 164 | "\n", 165 | " self.train_optimizer = self._optimizer(self.loss)\n", 166 | "\n", 167 | " # Saver\n", 168 | " self.saver = tf.train.Saver()\n", 169 | " \n", 170 | " def _encoder(self, enc_embed_input):\n", 171 | " \"\"\"\n", 172 | " Adds an encoder to the model architecture.\n", 173 | " \"\"\"\n", 174 | " cells = [self._lstm_cell(self.hidden_size) for _ in range(self.num_layers)]\n", 175 | " multilstm = tf.contrib.rnn.MultiRNNCell(cells, state_is_tuple=True)\n", 176 | "\n", 177 | " _, enc_state = tf.nn.dynamic_rnn(\n", 178 | " multilstm,\n", 179 | " enc_embed_input,\n", 180 | " sequence_length=self.lengths,\n", 181 | " swap_memory=True,\n", 182 | " dtype=tf.float32)\n", 183 | " \n", 184 | " return enc_state\n", 185 | " \n", 186 | " def _decoder(self, enc_state, dec_embed_input):\n", 187 | " \"\"\"\n", 188 | " Adds a decoder to the model architecture.\n", 189 | " \"\"\"\n", 190 | " output_lengths = tf.ones([self.batch_size], tf.int32) * self.max_seq_len\n", 191 | " helper = tf.contrib.seq2seq.TrainingHelper(\n", 192 | " dec_embed_input,\n", 193 | " output_lengths,\n", 194 | " time_major=False)\n", 195 | "\n", 196 | " cells = [self._lstm_cell(self.hidden_size) for _ in range(self.num_layers)]\n", 197 | " dec_cell = tf.contrib.rnn.MultiRNNCell(cells, state_is_tuple=True)\n", 198 | "\n", 199 | " decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, helper, enc_state)\n", 200 | "\n", 201 | " dec_outputs = tf.contrib.seq2seq.dynamic_decode(\n", 202 | " decoder,\n", 203 | " output_time_major=False,\n", 204 | " impute_finished=True,\n", 205 | " maximum_iterations=self.max_seq_len, swap_memory=True)\n", 206 | " \n", 207 | " return dec_outputs\n", 208 | " \n", 209 | " def _optimizer(self, loss,):\n", 210 | " \"\"\"\n", 211 | " Optimizes weights given a loss. \n", 212 | " \"\"\"\n", 213 | " def _learning_rate_decay_fn(learning_rate, global_step):\n", 214 | " return tf.train.exponential_decay(learning_rate, global_step, decay_steps=10000, decay_rate=0.99)\n", 215 | "\n", 216 | " starting_lr = 0.001\n", 217 | " starting_global_step = tf.Variable(0, trainable=False)\n", 218 | " optimizer = tf.contrib.layers.optimize_loss(\n", 219 | " loss=loss,\n", 220 | " global_step=starting_global_step,\n", 221 | " learning_rate=starting_lr,\n", 222 | " optimizer=tf.train.AdamOptimizer,\n", 223 | " learning_rate_decay_fn=lambda lr, gs: _learning_rate_decay_fn(lr, gs),\n", 224 | " clip_gradients=5.0)\n", 225 | " \n", 226 | " return optimizer\n", 227 | " \n", 228 | " def _process_decoder_input(self, target_data, char_to_code, batch_size):\n", 229 | " \"\"\"\n", 230 | " Concatenates the to the begining of each batch.\n", 231 | " \"\"\"\n", 232 | " ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n", 233 | " dec_input = tf.concat([tf.fill([batch_size, 1], char_to_code['']), ending], 1)\n", 234 | "\n", 235 | " return dec_input\n", 236 | "\n", 237 | " def _lstm_cell(self, hidden_size):\n", 238 | " \"\"\"\n", 239 | " Returns LSTM cell with dropout.\n", 240 | " \"\"\"\n", 241 | " cell = tf.contrib.rnn.LSTMCell(\n", 242 | " hidden_size,\n", 243 | " initializer=tf.contrib.layers.xavier_initializer())\n", 244 | "\n", 245 | " cell = tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=self.dropout)\n", 246 | "\n", 247 | " return cell\n", 248 | "\n", 249 | " def _weight_and_bias(self, in_size, out_size):\n", 250 | " \"\"\"\n", 251 | " Initializes weights and biases.\n", 252 | " \"\"\"\n", 253 | " weight = tf.Variable(tf.truncated_normal([in_size, out_size], stddev=0.01))\n", 254 | " bias = tf.Variable(tf.constant(1., shape=[out_size]))\n", 255 | "\n", 256 | " return weight, bias\n" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "# Train" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 6, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "class Trainer():\n", 273 | "\n", 274 | " def __init__(self, batch_size, checkpoints_path, dropout):\n", 275 | " self.batch_size = batch_size\n", 276 | " self.checkpoints = checkpoints_path\n", 277 | " self.path_to_graph = checkpoints_path + 'seq2seq'\n", 278 | " self.dropout = dropout\n", 279 | "\n", 280 | " def train(self, model, train_data, train_size, num_steps, num_epochs, min_loss=0.3):\n", 281 | " \"\"\"\n", 282 | " Trains a given model architecture with given train data.\n", 283 | " \"\"\"\n", 284 | " tf.set_random_seed(1234)\n", 285 | " \n", 286 | " with tf.Session() as sess:\n", 287 | " sess.run(tf.global_variables_initializer())\n", 288 | " total_loss = []\n", 289 | " timings = []\n", 290 | " steps_per_epoch = int(train_size / self.batch_size)\n", 291 | " num_epoch = 1\n", 292 | " \n", 293 | " for step in range(1, num_steps):\n", 294 | " beg_t = timeit.default_timer()\n", 295 | " X, L = train_data.next()\n", 296 | " seq_len = np.max(L)\n", 297 | "\n", 298 | " # For anomaly detection problem we reconstruct input data, so\n", 299 | " # targets and inputs are identical.\n", 300 | " feed_dict = {\n", 301 | " model.inputs: X,\n", 302 | " model.targets: X,\n", 303 | " model.lengths: L,\n", 304 | " model.dropout: self.dropout,\n", 305 | " model.batch_size: self.batch_size,\n", 306 | " model.max_seq_len: seq_len}\n", 307 | " \n", 308 | " fetches = [model.loss, model.decoder_outputs, model.train_optimizer]\n", 309 | " step_loss, _, _ = sess.run(fetches, feed_dict)\n", 310 | "\n", 311 | " total_loss.append(step_loss)\n", 312 | " timings.append(timeit.default_timer() - beg_t)\n", 313 | "\n", 314 | " if step % steps_per_epoch == 0:\n", 315 | " num_epoch += 1\n", 316 | "\n", 317 | " if step % 200 == 0 or step == 1:\n", 318 | " print_progress(\n", 319 | " int(step / 200),\n", 320 | " num_epoch,\n", 321 | " np.mean(total_loss),\n", 322 | " np.mean(step_loss),\n", 323 | " np.sum(timings))\n", 324 | " timings = []\n", 325 | "\n", 326 | " if step == 1:\n", 327 | " _ = tf.train.export_meta_graph(filename=self.path_to_graph + '.meta')\n", 328 | " \n", 329 | " if np.mean(total_loss) < min_loss or num_epoch > num_epochs:\n", 330 | " model.saver.save(sess, self.path_to_graph, global_step=step)\n", 331 | " print(\"Training is finished.\")\n", 332 | " break\n" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 7, 338 | "metadata": {}, 339 | "outputs": [], 340 | "source": [ 341 | "model = Seq2Seq(params)\n", 342 | "t = Trainer(params[\"batch_size\"], params[\"checkpoints\"], params[\"dropout\"])" 343 | ] 344 | }, 345 | { 346 | "cell_type": "code", 347 | "execution_count": 8, 348 | "metadata": {}, 349 | "outputs": [ 350 | { 351 | "name": "stdout", 352 | "output_type": "stream", 353 | "text": [ 354 | "Step 0 (epoch 1), average_train_loss = 4.64057, step_loss = 4.64057, time_per_step = 4.530\n", 355 | "Step 1 (epoch 2), average_train_loss = 1.85948, step_loss = 1.33259, time_per_step = 806.258\n", 356 | "Step 2 (epoch 4), average_train_loss = 1.51688, step_loss = 1.06377, time_per_step = 800.767\n", 357 | "Step 3 (epoch 5), average_train_loss = 1.30170, step_loss = 0.76867, time_per_step = 785.980\n", 358 | "Step 4 (epoch 7), average_train_loss = 1.12864, step_loss = 0.50699, time_per_step = 784.316\n", 359 | "Step 5 (epoch 9), average_train_loss = 0.98843, step_loss = 0.36790, time_per_step = 788.007\n", 360 | "Step 6 (epoch 10), average_train_loss = 0.87675, step_loss = 0.29091, time_per_step = 787.017\n", 361 | "Step 7 (epoch 12), average_train_loss = 0.78760, step_loss = 0.23059, time_per_step = 786.724\n", 362 | "Step 8 (epoch 14), average_train_loss = 0.71524, step_loss = 0.20539, time_per_step = 673.019\n", 363 | "Step 9 (epoch 15), average_train_loss = 0.65551, step_loss = 0.16491, time_per_step = 646.233\n", 364 | "Step 10 (epoch 17), average_train_loss = 0.60534, step_loss = 0.15554, time_per_step = 647.465\n", 365 | "Step 11 (epoch 18), average_train_loss = 0.56261, step_loss = 0.12414, time_per_step = 646.088\n", 366 | "Step 12 (epoch 20), average_train_loss = 0.52579, step_loss = 0.10708, time_per_step = 646.132\n", 367 | "Step 13 (epoch 22), average_train_loss = 0.49373, step_loss = 0.11087, time_per_step = 647.348\n", 368 | "Step 14 (epoch 23), average_train_loss = 0.46557, step_loss = 0.09185, time_per_step = 645.979\n", 369 | "Step 15 (epoch 25), average_train_loss = 0.44061, step_loss = 0.09714, time_per_step = 646.337\n", 370 | "Step 16 (epoch 27), average_train_loss = 0.41833, step_loss = 0.07704, time_per_step = 647.477\n", 371 | "Step 17 (epoch 28), average_train_loss = 0.39833, step_loss = 0.07569, time_per_step = 645.866\n", 372 | "Step 18 (epoch 30), average_train_loss = 0.38026, step_loss = 0.06467, time_per_step = 647.254\n", 373 | "Step 19 (epoch 31), average_train_loss = 0.36386, step_loss = 0.06554, time_per_step = 646.087\n", 374 | "Step 20 (epoch 33), average_train_loss = 0.34891, step_loss = 0.06405, time_per_step = 645.754\n", 375 | "Step 21 (epoch 35), average_train_loss = 0.33520, step_loss = 0.06205, time_per_step = 647.425\n", 376 | "Step 22 (epoch 36), average_train_loss = 0.32261, step_loss = 0.05851, time_per_step = 646.370\n", 377 | "Step 23 (epoch 38), average_train_loss = 0.31098, step_loss = 0.04636, time_per_step = 646.329\n", 378 | "Step 24 (epoch 40), average_train_loss = 0.30022, step_loss = 0.05213, time_per_step = 647.241\n", 379 | "Training is finished.\n" 380 | ] 381 | } 382 | ], 383 | "source": [ 384 | "num_steps = 10 ** 6\n", 385 | "num_epochs = 60\n", 386 | "\n", 387 | "train_gen = d.train_generator(params[\"batch_size\"], num_epochs)\n", 388 | "train_size = d.train_size\n", 389 | "\n", 390 | "t.train(model, train_gen, train_size, num_steps, num_epochs)" 391 | ] 392 | }, 393 | { 394 | "cell_type": "markdown", 395 | "metadata": {}, 396 | "source": [ 397 | "# Parameters setting" 398 | ] 399 | }, 400 | { 401 | "cell_type": "markdown", 402 | "metadata": {}, 403 | "source": [ 404 | "In this part, the threshold setting is introduced. *Set_threshold* calculates the threshold value using *mean* and *std* of loss values of valid samples. \n", 405 | "\n", 406 | "At the testing stage, the model receives benign and anomalous samples.\n", 407 | "For each sample, the value of loss is calculated. If this value is greater than the threshold, then the request is considered anomalous.\n", 408 | "\n", 409 | "If you want to use special checkpoints without training a model, you can import a model from *params[\"checkpoint\"]* ." 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": 9, 415 | "metadata": {}, 416 | "outputs": [], 417 | "source": [ 418 | "class Predictor():\n", 419 | " def __init__(self, checkpoints_path, std_factor, vocab):\n", 420 | "\n", 421 | " self.threshold = 0.\n", 422 | " self.checkpoints = checkpoints_path\n", 423 | " self.path_to_graph = checkpoints_path + 'seq2seq'\n", 424 | " self.std_factor = std_factor\n", 425 | " self.vocab = vocab\n", 426 | " self.__load()\n", 427 | "\n", 428 | " def __load(self):\n", 429 | " \"\"\"\n", 430 | " Loads model from the checkpoint directory and sets models params. \n", 431 | " \"\"\"\n", 432 | " try:\n", 433 | " loaded_graph = tf.Graph()\n", 434 | " with loaded_graph.as_default():\n", 435 | " saver = tf.train.import_meta_graph(\n", 436 | " self.path_to_graph + '.meta')\n", 437 | "\n", 438 | " self.sess = tf.Session(graph=loaded_graph)\n", 439 | " saver.restore(self.sess, tf.train.latest_checkpoint(\n", 440 | " self.checkpoints))\n", 441 | "\n", 442 | " # loading model parameters\n", 443 | " self.inputs = loaded_graph.get_tensor_by_name('inputs:0')\n", 444 | " self.targets = loaded_graph.get_tensor_by_name('targets:0')\n", 445 | " self.lengths = loaded_graph.get_tensor_by_name('lengths:0')\n", 446 | " self.dropout = loaded_graph.get_tensor_by_name('dropout:0')\n", 447 | " self.batch_size_tensor = loaded_graph.get_tensor_by_name('batch_size:0')\n", 448 | " self.seq_len_tensor = loaded_graph.get_tensor_by_name('max_seq_len:0')\n", 449 | " self.get_batch_loss = loaded_graph.get_tensor_by_name('batch_loss:0')\n", 450 | " self.get_probabilities = loaded_graph.get_tensor_by_name('probs:0')\n", 451 | " self.get_logits = loaded_graph.get_tensor_by_name('logits:0')\n", 452 | " \n", 453 | " except Exception as e:\n", 454 | " raise ValueError('Unable to create model: {}'.format(e))\n", 455 | "\n", 456 | " def set_threshold(self, data_gen):\n", 457 | " \"\"\"\n", 458 | " Calculates threshold for anomaly detection.\n", 459 | " \"\"\"\n", 460 | " \n", 461 | " total_loss = []\n", 462 | " for seq, l in data_gen:\n", 463 | " batch_loss, _ = self._predict_for_request(seq, l)\n", 464 | " total_loss.extend(batch_loss)\n", 465 | "\n", 466 | " mean = np.mean(total_loss)\n", 467 | " std = np.std(total_loss)\n", 468 | " self.threshold = mean + self.std_factor * std\n", 469 | "\n", 470 | " print('Validation loss mean: ', mean)\n", 471 | " print('Validation loss std: ', std)\n", 472 | " print('Threshold for anomaly detection: ', self.threshold)\n", 473 | " \n", 474 | " return self.threshold\n", 475 | "\n", 476 | " def predict(self, data_gen, visual=True):\n", 477 | " \"\"\"\n", 478 | " Predicts probabilities and loss for given sequences.\n", 479 | " \"\"\"\n", 480 | " loss = []\n", 481 | " predictions = []\n", 482 | " num_displayed = 0\n", 483 | " \n", 484 | " for seq, l in data_gen:\n", 485 | " batch_loss, alphas = self._predict_for_request(seq, l)\n", 486 | " loss.extend(batch_loss)\n", 487 | " alphas = self._process_alphas(seq, alphas, 1)\n", 488 | " mask = np.array([l > self.threshold for l in batch_loss])\n", 489 | " final_pred = mask.astype(int)\n", 490 | " predictions.extend(final_pred)\n", 491 | " \n", 492 | " if visual and num_displayed < 10 and final_pred == [1]:\n", 493 | " print('\\n\\nPrediction: ', final_pred[0])\n", 494 | " print('Loss ', batch_loss[0])\n", 495 | " \n", 496 | " num_displayed += 1 \n", 497 | " self._visual(alphas, seq)\n", 498 | " \n", 499 | " return predictions, loss\n", 500 | "\n", 501 | " def _predict_for_request(self, X, l):\n", 502 | " \"\"\"\n", 503 | " Predicts probabilities and loss for given data. \n", 504 | " \"\"\"\n", 505 | " lengths = [l]\n", 506 | " max_seq_len = l\n", 507 | " feed_dict = {\n", 508 | " self.inputs: X,\n", 509 | " self.targets: X,\n", 510 | " self.lengths: lengths,\n", 511 | " self.dropout: 1.0,\n", 512 | " self.batch_size_tensor: 1,\n", 513 | " self.seq_len_tensor: max_seq_len}\n", 514 | "\n", 515 | " fetches = [self.get_batch_loss, self.get_probabilities]\n", 516 | " batch_loss, alphas = self.sess.run(fetches, feed_dict=feed_dict)\n", 517 | "\n", 518 | " return batch_loss, alphas\n", 519 | "\n", 520 | " def _process_alphas(self, X, alphas, batch_size):\n", 521 | " \"\"\"\n", 522 | " Counts numbers as probabilities for given data sample.\n", 523 | " \"\"\"\n", 524 | " processed_alphas = []\n", 525 | " for i in range(batch_size):\n", 526 | " probs = alphas[i]\n", 527 | " coefs = np.array([probs[j][X[i][j]] for j in range(len(X[i]))])\n", 528 | " coefs = coefs / coefs.max()\n", 529 | " processed_alphas.append(coefs)\n", 530 | " \n", 531 | " return processed_alphas\n", 532 | "\n", 533 | " def _visual(self, alphas, X):\n", 534 | " \"\"\"\n", 535 | " Colors sequence of malicious characters.\n", 536 | " \"\"\"\n", 537 | " for i, x in enumerate(X):\n", 538 | " coefs = alphas[i]\n", 539 | " tokens = self.vocab.int_to_string(x)\n", 540 | " \n", 541 | " for j in range(len(x)):\n", 542 | " token = tokens[j]\n", 543 | " if coefs[j] < 0.09:\n", 544 | " c = Fore.GREEN\n", 545 | " else:\n", 546 | " c = Fore.BLACK\n", 547 | " if token != '' and token != '':\n", 548 | " token = ''.join(c + token)\n", 549 | " print(token, end='')\n", 550 | " \n", 551 | " print(Fore.BLACK + '', end='')" 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": 10, 557 | "metadata": {}, 558 | "outputs": [ 559 | { 560 | "name": "stdout", 561 | "output_type": "stream", 562 | "text": [ 563 | "INFO:tensorflow:Restoring parameters from ./checkpoints/seq2seq-4805\n" 564 | ] 565 | } 566 | ], 567 | "source": [ 568 | "p = Predictor(params[\"checkpoints\"], params[\"std_factor\"], params[\"vocab\"])" 569 | ] 570 | }, 571 | { 572 | "cell_type": "code", 573 | "execution_count": 11, 574 | "metadata": {}, 575 | "outputs": [ 576 | { 577 | "name": "stdout", 578 | "output_type": "stream", 579 | "text": [ 580 | "Validation loss mean: 0.07533215\n", 581 | "Validation loss std: 0.108923055\n", 582 | "Threshold for anomaly detection: 0.7288704812526703\n" 583 | ] 584 | } 585 | ], 586 | "source": [ 587 | "val_gen = d.val_generator()\n", 588 | "threshold = p.set_threshold(val_gen)" 589 | ] 590 | }, 591 | { 592 | "cell_type": "markdown", 593 | "metadata": {}, 594 | "source": [ 595 | "### Benign samples " 596 | ] 597 | }, 598 | { 599 | "cell_type": "markdown", 600 | "metadata": {}, 601 | "source": [ 602 | "Here FP samples are showed and FP rate is computed." 603 | ] 604 | }, 605 | { 606 | "cell_type": "code", 607 | "execution_count": 12, 608 | "metadata": {}, 609 | "outputs": [ 610 | { 611 | "name": "stdout", 612 | "output_type": "stream", 613 | "text": [ 614 | "\n", 615 | "\n", 616 | "Prediction: 1\n", 617 | "Loss 0.946472\n", 618 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mP\u001b[32me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[32me\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[30mf\u001b[32m?\u001b[32md\u001b[32m7\u001b[32my\u001b[32mf\u001b[32m1\u001b[30mv\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 619 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 620 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 621 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 622 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 623 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 624 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mp\u001b[30me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[30mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[30me\u001b[32m.\u001b[32mc\u001b[30ms\u001b[32ms\u001b[30m\n", 625 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 626 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 627 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 628 | "\n", 629 | "Prediction: 1\n", 630 | "Loss 0.946472\n", 631 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mP\u001b[32me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[32me\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[30mf\u001b[32m?\u001b[32md\u001b[32m7\u001b[32my\u001b[32mf\u001b[32m1\u001b[30mv\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 632 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 633 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 634 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 635 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 636 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 637 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mp\u001b[30me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[30mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[30me\u001b[32m.\u001b[32mc\u001b[30ms\u001b[32ms\u001b[30m\n", 638 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 639 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 640 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 641 | "\n", 642 | "Prediction: 1\n", 643 | "Loss 1.1683424\n", 644 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mg\u001b[32ml\u001b[32my\u001b[32mp\u001b[32mh\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32ms\u001b[32m-\u001b[32mh\u001b[32ma\u001b[32ml\u001b[32mf\u001b[32ml\u001b[32mi\u001b[32mn\u001b[32mg\u001b[30ms\u001b[32m-\u001b[32mr\u001b[32me\u001b[32mg\u001b[32mu\u001b[32ml\u001b[32ma\u001b[32mr\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[32mf\u001b[32m2\u001b[30m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[30m/\u001b[32m1\u001b[30m.\u001b[30m1\u001b[30m\n", 645 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 646 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 647 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 648 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 649 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 650 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mb\u001b[30mo\u001b[32mo\u001b[30mt\u001b[32ms\u001b[32mt\u001b[32mr\u001b[32ma\u001b[32mp\u001b[32m.\u001b[32mm\u001b[32mi\u001b[32mn\u001b[32m.\u001b[32mc\u001b[32ms\u001b[32ms\u001b[32m\n", 651 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 652 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 653 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 654 | "\n", 655 | "Prediction: 1\n", 656 | "Loss 2.4811156\n", 657 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[32m/\u001b[32mg\u001b[32ml\u001b[32my\u001b[32mp\u001b[32mh\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32ms\u001b[32m-\u001b[32mh\u001b[32ma\u001b[32ml\u001b[32mf\u001b[32ml\u001b[32mi\u001b[32mn\u001b[32mg\u001b[30ms\u001b[32m-\u001b[32mr\u001b[32me\u001b[32mg\u001b[32mu\u001b[32ml\u001b[32ma\u001b[32mr\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[32mf\u001b[32m2\u001b[30m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 658 | "\u001b[32mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 659 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[32mU\u001b[32mb\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30mu\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m\n", 660 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[32mf\u001b[30mo\u001b[32mn\u001b[32mt\u001b[32m-\u001b[30mw\u001b[30mo\u001b[32mf\u001b[32mf\u001b[32m2\u001b[32m;\u001b[32mq\u001b[32m=\u001b[32m1\u001b[32m.\u001b[32m0\u001b[32m,\u001b[32ma\u001b[32mp\u001b[32mp\u001b[32ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m/\u001b[32mf\u001b[32mo\u001b[32mn\u001b[32mt\u001b[32m-\u001b[32mw\u001b[32mo\u001b[32mf\u001b[32mf\u001b[32m;\u001b[32mq\u001b[32m=\u001b[32m0\u001b[30m.\u001b[30m9\u001b[32m,\u001b[32m*\u001b[30m/\u001b[32m*\u001b[32m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m8\u001b[30m\n", 661 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[32mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m5\u001b[30m\n", 662 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[32mi\u001b[32md\u001b[32me\u001b[32mn\u001b[32mt\u001b[32mi\u001b[32mt\u001b[32my\u001b[32m\n", 663 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[32m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[32m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mb\u001b[30mo\u001b[32mo\u001b[30mt\u001b[32ms\u001b[32mt\u001b[32mr\u001b[32ma\u001b[32mp\u001b[32m.\u001b[32mm\u001b[32mi\u001b[32mn\u001b[32m.\u001b[32mc\u001b[32ms\u001b[32ms\u001b[32m\n", 664 | "\u001b[32mC\u001b[32mo\u001b[32mo\u001b[32mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[32mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mm\u001b[32ml\u001b[32ma\u001b[32mc\u001b[32ms\u001b[32m0\u001b[32mu\u001b[32mi\u001b[32mo\u001b[32mu\u001b[32m3\u001b[32m4\u001b[32m4\u001b[32mi\u001b[32m3\u001b[32mf\u001b[32ma\u001b[32m5\u001b[32m3\u001b[32ms\u001b[32m7\u001b[32mr\u001b[32ma\u001b[32mu\u001b[32mt\u001b[32m6\u001b[32m\n" 665 | ] 666 | }, 667 | { 668 | "name": "stdout", 669 | "output_type": "stream", 670 | "text": [ 671 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[32mv\u001b[30me\u001b[30m\n", 672 | "\n", 673 | "Prediction: 1\n", 674 | "Loss 0.946472\n", 675 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mP\u001b[32me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[32me\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[30mf\u001b[32m?\u001b[32md\u001b[32m7\u001b[32my\u001b[32mf\u001b[32m1\u001b[30mv\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 676 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 677 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 678 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 679 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 680 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 681 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mp\u001b[30me\u001b[32m-\u001b[32mi\u001b[32mc\u001b[30mo\u001b[30mn\u001b[32m-\u001b[32m7\u001b[32m-\u001b[32ms\u001b[32mt\u001b[32mr\u001b[30mo\u001b[32mk\u001b[30me\u001b[32m.\u001b[32mc\u001b[30ms\u001b[32ms\u001b[30m\n", 682 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 683 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 684 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 685 | "\n", 686 | "Prediction: 1\n", 687 | "Loss 1.1683424\n", 688 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mg\u001b[32ml\u001b[32my\u001b[32mp\u001b[32mh\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32ms\u001b[32m-\u001b[32mh\u001b[32ma\u001b[32ml\u001b[32mf\u001b[32ml\u001b[32mi\u001b[32mn\u001b[32mg\u001b[30ms\u001b[32m-\u001b[32mr\u001b[32me\u001b[32mg\u001b[32mu\u001b[32ml\u001b[32ma\u001b[32mr\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[32mf\u001b[32m2\u001b[30m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[30m/\u001b[32m1\u001b[30m.\u001b[30m1\u001b[30m\n", 689 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 690 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 691 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 692 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 693 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 694 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mb\u001b[30mo\u001b[32mo\u001b[30mt\u001b[32ms\u001b[32mt\u001b[32mr\u001b[32ma\u001b[32mp\u001b[32m.\u001b[32mm\u001b[32mi\u001b[32mn\u001b[32m.\u001b[32mc\u001b[32ms\u001b[32ms\u001b[32m\n", 695 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 696 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 697 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 698 | "\n", 699 | "Prediction: 1\n", 700 | "Loss 1.1683424\n", 701 | "\u001b[30mG\u001b[30mE\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mf\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30ms\u001b[30m/\u001b[32mg\u001b[32ml\u001b[32my\u001b[32mp\u001b[32mh\u001b[32mi\u001b[32mc\u001b[30mo\u001b[32mn\u001b[32ms\u001b[32m-\u001b[32mh\u001b[32ma\u001b[32ml\u001b[32mf\u001b[32ml\u001b[32mi\u001b[32mn\u001b[32mg\u001b[30ms\u001b[32m-\u001b[32mr\u001b[32me\u001b[32mg\u001b[32mu\u001b[32ml\u001b[32ma\u001b[32mr\u001b[32m.\u001b[32mw\u001b[32mo\u001b[32mf\u001b[32mf\u001b[32m2\u001b[30m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[30m/\u001b[32m1\u001b[30m.\u001b[30m1\u001b[30m\n", 702 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 703 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[30mv\u001b[30me\u001b[30m\n", 704 | "\u001b[32mO\u001b[32mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 705 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[30m4\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m2\u001b[30m8\u001b[30m2\u001b[30m.\u001b[30m1\u001b[30m8\u001b[30m6\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m\n", 706 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[32m*\u001b[30m/\u001b[30m*\u001b[32m\n", 707 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[32ma\u001b[32ms\u001b[32ms\u001b[30me\u001b[30mt\u001b[32ms\u001b[30m/\u001b[32mc\u001b[30ms\u001b[32ms\u001b[32m/\u001b[32mb\u001b[30mo\u001b[32mo\u001b[30mt\u001b[32ms\u001b[32mt\u001b[32mr\u001b[32ma\u001b[32mp\u001b[32m.\u001b[32mm\u001b[32mi\u001b[32mn\u001b[32m.\u001b[32mc\u001b[32ms\u001b[32ms\u001b[32m\n", 708 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 709 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[30m\n", 710 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mj\u001b[30m1\u001b[32mp\u001b[30ma\u001b[32mv\u001b[32mg\u001b[30ml\u001b[32mp\u001b[32m5\u001b[32mu\u001b[32me\u001b[32m3\u001b[30m0\u001b[30m2\u001b[30m6\u001b[32m6\u001b[32mc\u001b[30m0\u001b[32mj\u001b[30m8\u001b[30m8\u001b[30mg\u001b[30me\u001b[30md\u001b[30m3\u001b[30m0\u001b[30m\n", 711 | "\n", 712 | "Prediction: 1\n", 713 | "Loss 0.7708773\n", 714 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 715 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 716 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[32mU\u001b[32mb\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30mu\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m\n", 717 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[30m\n", 718 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m5\u001b[30m\n", 719 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 720 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30mt\u001b[30mr\u001b[30ma\u001b[30mn\u001b[30ms\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30ms\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m\n", 721 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m;\u001b[30m \u001b[30mc\u001b[30mh\u001b[30ma\u001b[30mr\u001b[30ms\u001b[30me\u001b[30mt\u001b[30m=\u001b[30mU\u001b[30mT\u001b[30mF\u001b[30m-\u001b[30m8\u001b[30m\n", 722 | "\u001b[32mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30m\n", 723 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30m1\u001b[32m1\u001b[32m2\u001b[30m\n", 724 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[30mc\u001b[32mh\u001b[32m1\u001b[32mu\u001b[32mp\u001b[32mt\u001b[32m4\u001b[30ml\u001b[32mk\u001b[32m9\u001b[30mo\u001b[32mn\u001b[30mm\u001b[30m6\u001b[32me\u001b[32m2\u001b[32m6\u001b[30ml\u001b[32mo\u001b[30m4\u001b[32ml\u001b[32mk\u001b[32mc\u001b[30m8\u001b[30m9\u001b[30m3\u001b[30m\n", 725 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[32mv\u001b[30me\u001b[30m\n", 726 | "\u001b[32m\n", 727 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[30m=\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[30mc\u001b[30mh\u001b[30me\u001b[30mc\u001b[30mk\u001b[30m&\u001b[30mf\u001b[30mi\u001b[30mr\u001b[30ms\u001b[30mt\u001b[30mn\u001b[30ma\u001b[30mm\u001b[30me\u001b[30m=\u001b[32m&\u001b[30ml\u001b[30ma\u001b[30ms\u001b[30mt\u001b[30mn\u001b[30ma\u001b[30mm\u001b[30me\u001b[30m=\u001b[32mP\u001b[32ml\u001b[30ma\u001b[32mn\u001b[32mt\u001b[30m&\u001b[32mc\u001b[30mr\u001b[30me\u001b[30md\u001b[30mi\u001b[30mt\u001b[30mc\u001b[30ma\u001b[30mr\u001b[30md\u001b[30m=\u001b[32m1\u001b[32m2\u001b[30m3\u001b[32m4\u001b[30m-\u001b[32m6\u001b[32m5\u001b[30m7\u001b[32m9\u001b[30m-\u001b[32m3\u001b[32m5\u001b[32m1\u001b[30m4\u001b[30m-\u001b[30m6\u001b[32m4\u001b[32m7\u001b[32m5\u001b[30m&\u001b[32mr\u001b[30me\u001b[30mc\u001b[30mi\u001b[30mp\u001b[30mi\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m=\u001b[30mD\u001b[30mE\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[32m6\u001b[32m5\u001b[32m6\u001b[32m5\u001b[32m6\u001b[30m7\u001b[32m4\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m8\u001b[32m3\u001b[32m8\u001b[32m3\u001b[32m8\u001b[30m" 728 | ] 729 | } 730 | ], 731 | "source": [ 732 | "test_gen = d.test_generator()\n", 733 | "valid_preds, valid_loss = p.predict(test_gen)" 734 | ] 735 | }, 736 | { 737 | "cell_type": "code", 738 | "execution_count": 13, 739 | "metadata": {}, 740 | "outputs": [ 741 | { 742 | "name": "stdout", 743 | "output_type": "stream", 744 | "text": [ 745 | "Number of FP: 8\n", 746 | "Number of samples: 2200\n", 747 | "FP rate: 0.0036\n" 748 | ] 749 | } 750 | ], 751 | "source": [ 752 | "print('Number of FP: ', np.sum(valid_preds))\n", 753 | "print('Number of samples: ', len(valid_preds))\n", 754 | "print('FP rate: {:.4f}'.format(np.sum(valid_preds) / len(valid_preds)))" 755 | ] 756 | }, 757 | { 758 | "cell_type": "markdown", 759 | "metadata": {}, 760 | "source": [ 761 | "### Anomalous samples " 762 | ] 763 | }, 764 | { 765 | "cell_type": "markdown", 766 | "metadata": {}, 767 | "source": [ 768 | "Here TP samples are showed and TP rate is computed." 769 | ] 770 | }, 771 | { 772 | "cell_type": "code", 773 | "execution_count": 14, 774 | "metadata": {}, 775 | "outputs": [ 776 | { 777 | "name": "stdout", 778 | "output_type": "stream", 779 | "text": [ 780 | "Downloaded 1097 samples\n", 781 | "\n", 782 | "\n", 783 | "Prediction: 1\n", 784 | "Loss 2.16642\n", 785 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 786 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 787 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[32mU\u001b[32mb\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30mu\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m\n", 788 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[30m\n", 789 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m5\u001b[30m\n", 790 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 791 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ml\u001b[30mo\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m\n", 792 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m;\u001b[30m \u001b[30mc\u001b[30mh\u001b[30ma\u001b[30mr\u001b[30ms\u001b[30me\u001b[30mt\u001b[30m=\u001b[30mU\u001b[30mT\u001b[30mF\u001b[30m-\u001b[30m8\u001b[30m\n", 793 | "\u001b[32mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30m\n", 794 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[32m2\u001b[32m0\u001b[32m9\u001b[30m\n", 795 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[32mk\u001b[32mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mm\u001b[32ml\u001b[30ma\u001b[32mc\u001b[32ms\u001b[32m0\u001b[32mu\u001b[32mi\u001b[32mo\u001b[32mu\u001b[32m3\u001b[32m4\u001b[32m4\u001b[32mi\u001b[32m3\u001b[32mf\u001b[32ma\u001b[32m5\u001b[32m3\u001b[32ms\u001b[32m7\u001b[32mr\u001b[32ma\u001b[32mu\u001b[32mt\u001b[32m6\u001b[32m\n", 796 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[32me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[32mv\u001b[30me\u001b[30m\n", 797 | "\u001b[32m\n", 798 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[30m=\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[32mc\u001b[32mr\u001b[30me\u001b[32ma\u001b[30mt\u001b[32me\u001b[32m&\u001b[32mu\u001b[32ms\u001b[30me\u001b[30mr\u001b[32mn\u001b[32ma\u001b[32mm\u001b[32me\u001b[30m=\u001b[32mJ\u001b[30ma\u001b[32mc\u001b[32mk\u001b[32m'\u001b[32m+\u001b[32ma\u001b[32mn\u001b[32md\u001b[32m+\u001b[32me\u001b[32mx\u001b[32mt\u001b[30mr\u001b[32ma\u001b[32mc\u001b[32mt\u001b[32mv\u001b[32ma\u001b[32ml\u001b[32mu\u001b[32me\u001b[32m(\u001b[32m0\u001b[32mx\u001b[32m0\u001b[32ma\u001b[32m,\u001b[32mc\u001b[32mo\u001b[32mn\u001b[32mc\u001b[32ma\u001b[32mt\u001b[32m(\u001b[30m0\u001b[32mx\u001b[32m0\u001b[32ma\u001b[32m,\u001b[32m(\u001b[32ms\u001b[32me\u001b[32ml\u001b[32me\u001b[32mc\u001b[32mt\u001b[32m \u001b[32mv\u001b[32me\u001b[32mr\u001b[32ms\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m(\u001b[32m)\u001b[32m)\u001b[32m)\u001b[32m)\u001b[32m+\u001b[32ma\u001b[32mn\u001b[32md\u001b[32m+\u001b[32m'\u001b[30m1\u001b[32m'\u001b[32m=\u001b[32m'\u001b[32m1\u001b[32m&\u001b[32mp\u001b[32ma\u001b[30ms\u001b[30ms\u001b[32mw\u001b[32mo\u001b[30mr\u001b[32md\u001b[30m=\u001b[32mp\u001b[32ma\u001b[32ms\u001b[32ms\u001b[32mw\u001b[32m0\u001b[32mr\u001b[32md\u001b[32m&\u001b[32mf\u001b[32mi\u001b[32mr\u001b[30ms\u001b[30mt\u001b[32mn\u001b[30ma\u001b[32mm\u001b[32me\u001b[30m=\u001b[32mf\u001b[32mi\u001b[32mr\u001b[32ms\u001b[32mt\u001b[32m&\u001b[32ml\u001b[32ma\u001b[32ms\u001b[32mt\u001b[32mn\u001b[32ma\u001b[32mm\u001b[32me\u001b[32m=\u001b[32ml\u001b[32ma\u001b[32ms\u001b[32mt\u001b[32m&\u001b[32mb\u001b[30mi\u001b[30mr\u001b[30mt\u001b[32mh\u001b[32md\u001b[30ma\u001b[32mt\u001b[30me\u001b[32m=\u001b[32m3\u001b[32m0\u001b[32m-\u001b[32m0\u001b[32m8\u001b[32m-\u001b[32m2\u001b[32m0\u001b[30m1\u001b[32m7\u001b[32m&\u001b[32me\u001b[32mm\u001b[32ma\u001b[32mi\u001b[32ml\u001b[32m=\u001b[32me\u001b[32me\u001b[32me\u001b[32m%\u001b[32m4\u001b[32m0\u001b[32mm\u001b[32ma\u001b[32mi\u001b[32ml\u001b[32m.\u001b[32mc\u001b[32mo\u001b[32mm\u001b[32m&\u001b[32mp\u001b[32mh\u001b[30mo\u001b[30mn\u001b[30me\u001b[32m=\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m&\u001b[32ma\u001b[32mc\u001b[32mc\u001b[32mo\u001b[32mu\u001b[30mn\u001b[30mt\u001b[30m=\u001b[32mD\u001b[32mE\u001b[32m4\u001b[32m4\u001b[32m4\u001b[32m0\u001b[32m4\u001b[32m4\u001b[32m1\u001b[32m9\u001b[32m5\u001b[30m6\u001b[32m9\u001b[32m7\u001b[32m5\u001b[32m0\u001b[32m5\u001b[32m5\u001b[32m3\u001b[32m3\u001b[32m4\u001b[30m0\u001b[32m&\u001b[30mc\u001b[30mr\u001b[30me\u001b[30md\u001b[30mi\u001b[30mt\u001b[30mc\u001b[30ma\u001b[30mr\u001b[30md\u001b[30m=\u001b[32m4\u001b[30m5\u001b[32m5\u001b[32m6\u001b[30m-\u001b[32m9\u001b[30m3\u001b[30m7\u001b[32m3\u001b[30m-\u001b[32m3\u001b[30m9\u001b[32m1\u001b[32m3\u001b[32m-\u001b[32m6\u001b[32m5\u001b[30m1\u001b[32m0\u001b[30m\n", 799 | "\n", 800 | "Prediction: 1\n", 801 | "Loss 2.2834342\n", 802 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 803 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 804 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[32mU\u001b[32mb\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30mu\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m\n", 805 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[30m\n", 806 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m5\u001b[30m\n", 807 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 808 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ml\u001b[30mo\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m\n", 809 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m\n", 810 | "\u001b[32mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30m\n", 811 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[32m7\u001b[30m6\u001b[30m\n", 812 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32mm\u001b[32ml\u001b[32ma\u001b[32mc\u001b[32ms\u001b[32m0\u001b[32mu\u001b[32mi\u001b[32mo\u001b[32mu\u001b[32m3\u001b[32m4\u001b[32m4\u001b[32mi\u001b[32m3\u001b[32mf\u001b[32ma\u001b[32m5\u001b[32m3\u001b[32ms\u001b[32m7\u001b[32mr\u001b[32ma\u001b[32mu\u001b[32mt\u001b[32m6\u001b[32m\n", 813 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[32me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[32mv\u001b[30me\u001b[30m\n", 814 | "\u001b[32m\n" 815 | ] 816 | }, 817 | { 818 | "name": "stdout", 819 | "output_type": "stream", 820 | "text": [ 821 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[30m=\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[32ml\u001b[30mo\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m&\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30mn\u001b[30ma\u001b[30mm\u001b[30me\u001b[30m=\u001b[32mn\u001b[32mo\u001b[32mn\u001b[32me\u001b[32m'\u001b[32m+\u001b[32mu\u001b[32mn\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m+\u001b[32ms\u001b[32me\u001b[32ml\u001b[32me\u001b[32mc\u001b[32mt\u001b[32m+\u001b[32m1\u001b[32m,\u001b[32m2\u001b[32m,\u001b[32ml\u001b[32mo\u001b[32mg\u001b[32mi\u001b[32mn\u001b[32m,\u001b[32mp\u001b[30ma\u001b[32ms\u001b[32ms\u001b[32mw\u001b[32mo\u001b[32mr\u001b[32md\u001b[32m,\u001b[32m5\u001b[32m,\u001b[32m6\u001b[32m,\u001b[32m7\u001b[32m,\u001b[32mN\u001b[32mU\u001b[32mL\u001b[32mL\u001b[32m,\u001b[32mN\u001b[32mU\u001b[32mL\u001b[32mL\u001b[32m,\u001b[32m1\u001b[32m0\u001b[32m,\u001b[32m1\u001b[32m1\u001b[32m,\u001b[32m1\u001b[32m2\u001b[32m,\u001b[32m1\u001b[32m3\u001b[32m,\u001b[32m1\u001b[32m4\u001b[32m,\u001b[32m1\u001b[32m5\u001b[32m,\u001b[32m1\u001b[32m6\u001b[32m,\u001b[32m1\u001b[32m7\u001b[32m+\u001b[32mf\u001b[32mr\u001b[32mo\u001b[32mm\u001b[32m+\u001b[32mu\u001b[32ms\u001b[32me\u001b[32mr\u001b[32ms\u001b[32m+\u001b[32ml\u001b[32mi\u001b[32mm\u001b[32mi\u001b[32mt\u001b[32m+\u001b[32m1\u001b[32m+\u001b[32m-\u001b[32m-\u001b[32m1\u001b[30m\n", 822 | "\n", 823 | "Prediction: 1\n", 824 | "Loss 0.9289187\n", 825 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[30m\n", 826 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m\n", 827 | "\u001b[32mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[32mU\u001b[32mb\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30mu\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m9\u001b[30m.\u001b[30m0\u001b[30m\n", 828 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[30m\n", 829 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m5\u001b[30m\n", 830 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[30m\n", 831 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30mt\u001b[30mr\u001b[30ma\u001b[30mn\u001b[30ms\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30ms\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m\n", 832 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m;\u001b[30m \u001b[30mc\u001b[30mh\u001b[30ma\u001b[30mr\u001b[30ms\u001b[30me\u001b[30mt\u001b[30m=\u001b[30mU\u001b[30mT\u001b[30mF\u001b[30m-\u001b[30m8\u001b[30m\n", 833 | "\u001b[32mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30m\n", 834 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m6\u001b[30m2\u001b[30m\n", 835 | "\u001b[30mC\u001b[30mo\u001b[30mo\u001b[30mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[30mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[30mc\u001b[32mh\u001b[32m1\u001b[32mu\u001b[32mp\u001b[32mt\u001b[32m4\u001b[32ml\u001b[32mk\u001b[32m9\u001b[30mo\u001b[32mn\u001b[30mm\u001b[30m6\u001b[32me\u001b[32m2\u001b[32m6\u001b[30ml\u001b[30mo\u001b[30m4\u001b[32ml\u001b[32mk\u001b[32mc\u001b[30m8\u001b[30m9\u001b[30m3\u001b[30m\n", 836 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[30me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[30mi\u001b[32mv\u001b[30me\u001b[30m\n", 837 | "\u001b[32m\n", 838 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[30m=\u001b[30mt\u001b[30mr\u001b[30ma\u001b[30mn\u001b[30ms\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[30ms\u001b[30me\u001b[30mn\u001b[30md\u001b[30m&\u001b[30ms\u001b[30me\u001b[30mn\u001b[30md\u001b[30me\u001b[30mr\u001b[30m=\u001b[30mD\u001b[30mE\u001b[30m0\u001b[30m0\u001b[30m0\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m2\u001b[30m2\u001b[30m2\u001b[30m2\u001b[30m2\u001b[30m3\u001b[30m3\u001b[30m3\u001b[30m3\u001b[30m3\u001b[30m&\u001b[30mr\u001b[30me\u001b[30mc\u001b[30mi\u001b[30mp\u001b[30mi\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m=\u001b[30mD\u001b[30mE\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[30m1\u001b[32m6\u001b[32m5\u001b[32m6\u001b[32m5\u001b[32m6\u001b[30m7\u001b[32m4\u001b[32m7\u001b[32m4\u001b[32m7\u001b[32m8\u001b[32m3\u001b[32m8\u001b[32m3\u001b[32m8\u001b[32m&\u001b[30mc\u001b[32mr\u001b[30me\u001b[30md\u001b[30mi\u001b[30mt\u001b[30mc\u001b[30ma\u001b[30mr\u001b[30md\u001b[30m=\u001b[32m1\u001b[32m2\u001b[30m3\u001b[32m4\u001b[32m-\u001b[32m6\u001b[32m5\u001b[30m7\u001b[32m9\u001b[30m-\u001b[30m3\u001b[32m5\u001b[32m1\u001b[32m4\u001b[32m-\u001b[30m6\u001b[32m4\u001b[32m7\u001b[32m5\u001b[30m&\u001b[30ma\u001b[30mm\u001b[30mo\u001b[30mu\u001b[30mn\u001b[30mt\u001b[30m=\u001b[32m2\u001b[30m&\u001b[30mc\u001b[30mo\u001b[30mm\u001b[30mm\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m=\u001b[32m<\u001b[32ms\u001b[32mv\u001b[32mg\u001b[32m \u001b[30mo\u001b[32mn\u001b[32ml\u001b[32mo\u001b[32ma\u001b[32md\u001b[32m=\u001b[30ma\u001b[32ml\u001b[32me\u001b[32mr\u001b[32mt\u001b[32m(\u001b[32m1\u001b[32m)\u001b[32m>\u001b[30m\n", 839 | "\n", 840 | "Prediction: 1\n", 841 | "Loss 8.67965\n", 842 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[32mb\u001b[30ma\u001b[32mn\u001b[32mk\u001b[32m/\u001b[32ma\u001b[30mp\u001b[30mi\u001b[32m.\u001b[30mp\u001b[32mh\u001b[30mp\u001b[32m?\u001b[32mx\u001b[32mm\u001b[32ml\u001b[32m \u001b[32mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[32m1\u001b[32m\n", 843 | "\u001b[32mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[32mv\u001b[32mu\u001b[32ml\u001b[32mn\u001b[32mb\u001b[32ma\u001b[32mn\u001b[32mk\u001b[30m.\u001b[32mc\u001b[32mo\u001b[32mm\u001b[32m\n", 844 | "\u001b[32mU\u001b[32ms\u001b[30me\u001b[30mr\u001b[32m-\u001b[32mA\u001b[32mg\u001b[30me\u001b[32mn\u001b[32mt\u001b[32m:\u001b[32m \u001b[32mM\u001b[30mo\u001b[32mz\u001b[32mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[32m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[30mW\u001b[30mi\u001b[30mn\u001b[30md\u001b[30mo\u001b[30mw\u001b[30ms\u001b[30m \u001b[30mN\u001b[30mT\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m;\u001b[30m \u001b[30mW\u001b[30mi\u001b[30mn\u001b[30m6\u001b[30m4\u001b[30m;\u001b[30m \u001b[30mx\u001b[30m6\u001b[30m4\u001b[32m;\u001b[30m \u001b[30mr\u001b[30mv\u001b[30m:\u001b[30m5\u001b[32m2\u001b[30m.\u001b[30m0\u001b[30m)\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m/\u001b[30m2\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m0\u001b[30m1\u001b[30m0\u001b[30m1\u001b[30m \u001b[30mF\u001b[30mi\u001b[30mr\u001b[30me\u001b[30mf\u001b[30mo\u001b[30mx\u001b[30m/\u001b[30m5\u001b[32m2\u001b[30m.\u001b[30m0\u001b[30m\n", 845 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[30m\n", 846 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[32mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[32mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[32mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[32mx\u001b[32mm\u001b[32ml\u001b[32m\n", 847 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[32m9\u001b[32m4\u001b[32m5\u001b[30m\n", 848 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[32mn\u001b[30m:\u001b[30m \u001b[32mc\u001b[30ml\u001b[30mo\u001b[32ms\u001b[30me\u001b[32m\n", 849 | "\u001b[32m\n" 850 | ] 851 | }, 852 | { 853 | "name": "stdout", 854 | "output_type": "stream", 855 | "text": [ 856 | "\u001b[32m<\u001b[32m?\u001b[32mx\u001b[32mm\u001b[32ml\u001b[32m \u001b[32mv\u001b[32me\u001b[32mr\u001b[32ms\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m=\u001b[32m\"\u001b[32m1\u001b[32m.\u001b[32m0\u001b[32m\"\u001b[32m \u001b[32me\u001b[32mn\u001b[32mc\u001b[32mo\u001b[32md\u001b[32mi\u001b[32mn\u001b[32mg\u001b[32m=\u001b[32m\"\u001b[32mu\u001b[32mt\u001b[30mf\u001b[32m-\u001b[30m8\u001b[32m\"\u001b[32m?\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mD\u001b[32mO\u001b[32mC\u001b[30mT\u001b[32mY\u001b[32mP\u001b[32mE\u001b[32m \u001b[32ml\u001b[30mo\u001b[32ml\u001b[32mz\u001b[32m \u001b[32m[\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[32m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m \u001b[32m\"\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mL\u001b[32mE\u001b[32mM\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32m \u001b[32ml\u001b[32mo\u001b[32mg\u001b[32mi\u001b[32mn\u001b[32m \u001b[32m(\u001b[32m#\u001b[32mP\u001b[32mC\u001b[32mD\u001b[32mA\u001b[32mT\u001b[32mA\u001b[32m)\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m1\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m1\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[32m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m2\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m3\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m4\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m5\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[30m \u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m7\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m6\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[32m \u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m7\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m<\u001b[32m!\u001b[32mE\u001b[32mN\u001b[32mT\u001b[32mI\u001b[32mT\u001b[32mY\u001b[32m \u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m9\u001b[30m \u001b[32m\"\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m&\u001b[32ml\u001b[32mo\u001b[32ml\u001b[32m8\u001b[32m;\u001b[32m\"\u001b[32m>\u001b[32m]\u001b[32m>\u001b[32m<\u001b[30ma\u001b[32mp\u001b[32mi\u001b[32m>\u001b[32m<\u001b[32mt\u001b[32my\u001b[32mp\u001b[32me\u001b[32m>\u001b[32mu\u001b[32ms\u001b[30me\u001b[32mr\u001b[32m<\u001b[32m/\u001b[32mt\u001b[32my\u001b[32mp\u001b[32me\u001b[32m>\u001b[32m<\u001b[32ma\u001b[32mc\u001b[32mt\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m>\u001b[32mf\u001b[32mo\u001b[32mr\u001b[32mg\u001b[32mo\u001b[32mt\u001b[32mp\u001b[32ma\u001b[32ms\u001b[32ms\u001b[32m<\u001b[32m/\u001b[32ma\u001b[32mc\u001b[32mt\u001b[32mi\u001b[32mo\u001b[32mn\u001b[32m>\u001b[32m<\u001b[32mu\u001b[32ms\u001b[32me\u001b[32mr\u001b[32mn\u001b[32ma\u001b[32mm\u001b[32me\u001b[32m>\u001b[32mj\u001b[32m.\u001b[32md\u001b[30mo\u001b[32me\u001b[32m<\u001b[32m/\u001b[32mu\u001b[32ms\u001b[32me\u001b[32mr\u001b[32mn\u001b[32ma\u001b[32mm\u001b[32me\u001b[32m>\u001b[32m<\u001b[32mp\u001b[32ma\u001b[32ms\u001b[32ms\u001b[32mw\u001b[30mo\u001b[32mr\u001b[32md\u001b[32m>\u001b[32m&\u001b[32ml\u001b[30mo\u001b[32ml\u001b[32m9\u001b[32m;\u001b[32m<\u001b[32m/\u001b[32mp\u001b[32ma\u001b[30ms\u001b[32ms\u001b[32mw\u001b[32mo\u001b[32mr\u001b[32md\u001b[32m>\u001b[32m<\u001b[32mc\u001b[32mo\u001b[32md\u001b[32me\u001b[32m>\u001b[32m0\u001b[32m0\u001b[32m0\u001b[32m<\u001b[32m/\u001b[32mc\u001b[32mo\u001b[32md\u001b[32me\u001b[32m>\u001b[32m<\u001b[32m/\u001b[32ma\u001b[32mp\u001b[32mi\u001b[32m>\u001b[30m\n", 857 | "\n", 858 | "Prediction: 1\n", 859 | "Loss 2.2545607\n", 860 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[32m\r", 861 | "\u001b[30m\n", 862 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\r", 863 | "\u001b[30m\n", 864 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[32me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[32mi\u001b[32mv\u001b[30me\u001b[32m\r", 865 | "\u001b[30m\n", 866 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30m1\u001b[32m3\u001b[32m0\u001b[32m\r", 867 | "\u001b[30m\n", 868 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[32m\r", 869 | "\u001b[30m\n", 870 | "\u001b[32mO\u001b[30mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\r", 871 | "\u001b[30m\n", 872 | "\u001b[30mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[32m\r", 873 | "\u001b[30m\n", 874 | "\u001b[30mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[32m6\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m3\u001b[32m5\u001b[32m9\u001b[30m.\u001b[30m1\u001b[30m3\u001b[32m9\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[32m\r", 875 | "\u001b[30m\n", 876 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m;\u001b[30m \u001b[30mc\u001b[30mh\u001b[30ma\u001b[30mr\u001b[30ms\u001b[30me\u001b[30mt\u001b[30m=\u001b[30mU\u001b[30mT\u001b[30mF\u001b[30m-\u001b[30m8\u001b[32m\r", 877 | "\u001b[30m\n", 878 | "\u001b[32mD\u001b[32mN\u001b[32mT\u001b[32m:\u001b[30m \u001b[30m1\u001b[32m\r", 879 | "\u001b[32m\n", 880 | "\u001b[32mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[32mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30mi\u001b[30mn\u001b[30mf\u001b[30mo\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[32m\r", 881 | "\u001b[30m\n", 882 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[32m\r", 883 | "\u001b[32m\n", 884 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[32m,\u001b[32mr\u001b[32mu\u001b[32m;\u001b[32mq\u001b[32m=\u001b[30m0\u001b[30m.\u001b[30m8\u001b[32m\r", 885 | "\u001b[32m\n", 886 | "\u001b[32mC\u001b[30mo\u001b[32mo\u001b[32mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[32mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32m7\u001b[32mb\u001b[32mg\u001b[32mg\u001b[32m5\u001b[32mt\u001b[32ma\u001b[32ml\u001b[32md\u001b[32m4\u001b[32ma\u001b[32me\u001b[32ma\u001b[32m8\u001b[32m0\u001b[32m7\u001b[32m9\u001b[32m0\u001b[32mi\u001b[32mt\u001b[32m7\u001b[32ms\u001b[32m1\u001b[32mc\u001b[32mc\u001b[32m1\u001b[32m\r", 887 | "\u001b[32m\n", 888 | "\u001b[32m\r", 889 | "\u001b[32m\n" 890 | ] 891 | }, 892 | { 893 | "name": "stdout", 894 | "output_type": "stream", 895 | "text": [ 896 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[32m=\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[32mi\u001b[32mn\u001b[32mf\u001b[32mo\u001b[32mu\u001b[32mp\u001b[32md\u001b[32ma\u001b[32mt\u001b[32me\u001b[32m&\u001b[32mf\u001b[30mi\u001b[32mr\u001b[32ms\u001b[32mt\u001b[32mn\u001b[30ma\u001b[32mm\u001b[30me\u001b[30m=\u001b[32mJ\u001b[32mo\u001b[32mh\u001b[32mn\u001b[32m&\u001b[30ml\u001b[30ma\u001b[30ms\u001b[30mt\u001b[30mn\u001b[30ma\u001b[30mm\u001b[30me\u001b[30m=\u001b[32mD\u001b[32mo\u001b[32me\u001b[32m&\u001b[32mp\u001b[32mh\u001b[32mo\u001b[32mn\u001b[32me\u001b[32m=\u001b[32m7\u001b[32m9\u001b[32m0\u001b[32m3\u001b[32m5\u001b[32m2\u001b[32m2\u001b[32m2\u001b[32m2\u001b[32m1\u001b[32m2\u001b[32m&\u001b[32me\u001b[30mm\u001b[32ma\u001b[32mi\u001b[32ml\u001b[30m=\u001b[32mj\u001b[32m.\u001b[32md\u001b[30mo\u001b[32me\u001b[32m%\u001b[32m4\u001b[32m0\u001b[32mv\u001b[32mu\u001b[32ml\u001b[32mn\u001b[32mb\u001b[32ma\u001b[32mn\u001b[32mk\u001b[32m.\u001b[32md\u001b[32me\u001b[32m&\u001b[32mb\u001b[32mi\u001b[32mr\u001b[30mt\u001b[32mh\u001b[32md\u001b[32ma\u001b[32mt\u001b[32me\u001b[32m=\u001b[32m1\u001b[32m9\u001b[32m8\u001b[32m4\u001b[30m-\u001b[32m0\u001b[32m4\u001b[32m-\u001b[32m0\u001b[32m4\u001b[32m&\u001b[32ma\u001b[32mb\u001b[32mo\u001b[32mu\u001b[30mt\u001b[32m=\u001b[32mH\u001b[32mi\u001b[32m!\u001b[30m\n", 897 | "\n", 898 | "Prediction: 1\n", 899 | "Loss 2.2545607\n", 900 | "\u001b[30mP\u001b[30mO\u001b[30mS\u001b[30mT\u001b[30m \u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[30ma\u001b[30mp\u001b[30mi\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[30m \u001b[30mH\u001b[30mT\u001b[30mT\u001b[30mP\u001b[30m/\u001b[30m1\u001b[30m.\u001b[30m1\u001b[32m\n", 901 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 902 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[32mk\u001b[30me\u001b[32me\u001b[30mp\u001b[32m-\u001b[30ma\u001b[30ml\u001b[32mi\u001b[32mv\u001b[30me\u001b[32m\n", 903 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30me\u001b[30mn\u001b[30mg\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30m1\u001b[32m3\u001b[32m0\u001b[32m\n", 904 | "\u001b[30mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mj\u001b[30ms\u001b[30mo\u001b[30mn\u001b[30m,\u001b[30m \u001b[30mt\u001b[30me\u001b[30mx\u001b[30mt\u001b[30m/\u001b[30mj\u001b[30ma\u001b[30mv\u001b[30ma\u001b[30ms\u001b[30mc\u001b[30mr\u001b[30mi\u001b[30mp\u001b[30mt\u001b[30m,\u001b[30m \u001b[30m*\u001b[30m/\u001b[30m*\u001b[30m;\u001b[30m \u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m1\u001b[32m\n", 905 | "\u001b[30mO\u001b[30mr\u001b[30mi\u001b[30mg\u001b[30mi\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 906 | "\u001b[30mX\u001b[30m-\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[30me\u001b[30md\u001b[30m-\u001b[30mW\u001b[30mi\u001b[30mt\u001b[30mh\u001b[30m:\u001b[30m \u001b[30mX\u001b[30mM\u001b[30mL\u001b[30mH\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30mR\u001b[30me\u001b[30mq\u001b[30mu\u001b[30me\u001b[30ms\u001b[30mt\u001b[32m\n", 907 | "\u001b[30mU\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m-\u001b[30mA\u001b[30mg\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m:\u001b[30m \u001b[30mM\u001b[30mo\u001b[30mz\u001b[30mi\u001b[30ml\u001b[30ml\u001b[30ma\u001b[30m/\u001b[30m5\u001b[30m.\u001b[30m0\u001b[30m \u001b[30m(\u001b[32mX\u001b[30m1\u001b[30m1\u001b[30m;\u001b[30m \u001b[30mL\u001b[30mi\u001b[30mn\u001b[30mu\u001b[30mx\u001b[30m \u001b[30mx\u001b[30m8\u001b[30m6\u001b[30m_\u001b[30m6\u001b[30m4\u001b[30m)\u001b[30m \u001b[30mA\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30me\u001b[30mW\u001b[30me\u001b[30mb\u001b[30mK\u001b[30mi\u001b[30mt\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[30m \u001b[30m(\u001b[30mK\u001b[30mH\u001b[30mT\u001b[30mM\u001b[30mL\u001b[30m,\u001b[30m \u001b[30ml\u001b[30mi\u001b[30mk\u001b[30me\u001b[30m \u001b[30mG\u001b[30me\u001b[30mc\u001b[30mk\u001b[30mo\u001b[30m)\u001b[30m \u001b[30mC\u001b[30mh\u001b[30mr\u001b[30mo\u001b[30mm\u001b[30me\u001b[30m/\u001b[30m6\u001b[32m6\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m3\u001b[30m3\u001b[32m5\u001b[32m9\u001b[30m.\u001b[30m1\u001b[30m3\u001b[32m9\u001b[30m \u001b[30mS\u001b[30ma\u001b[30mf\u001b[30ma\u001b[30mr\u001b[30mi\u001b[30m/\u001b[30m5\u001b[30m3\u001b[30m7\u001b[30m.\u001b[30m3\u001b[30m6\u001b[32m\n", 908 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[30mT\u001b[30my\u001b[30mp\u001b[30me\u001b[30m:\u001b[30m \u001b[30ma\u001b[30mp\u001b[30mp\u001b[30ml\u001b[30mi\u001b[30mc\u001b[30ma\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m/\u001b[30mx\u001b[30m-\u001b[30mw\u001b[30mw\u001b[30mw\u001b[30m-\u001b[30mf\u001b[30mo\u001b[30mr\u001b[30mm\u001b[30m-\u001b[30mu\u001b[30mr\u001b[30ml\u001b[30me\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30me\u001b[30md\u001b[30m;\u001b[30m \u001b[30mc\u001b[30mh\u001b[30ma\u001b[30mr\u001b[30ms\u001b[30me\u001b[30mt\u001b[30m=\u001b[30mU\u001b[30mT\u001b[30mF\u001b[30m-\u001b[30m8\u001b[32m\n", 909 | "\u001b[32mD\u001b[32mN\u001b[32mT\u001b[32m:\u001b[30m \u001b[30m1\u001b[32m\n", 910 | "\u001b[30mR\u001b[30me\u001b[30mf\u001b[30me\u001b[30mr\u001b[30me\u001b[30mr\u001b[30m:\u001b[30m \u001b[30mh\u001b[30mt\u001b[30mt\u001b[30mp\u001b[30m:\u001b[30m/\u001b[30m/\u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[30m/\u001b[30mv\u001b[30mu\u001b[30ml\u001b[30mn\u001b[30mb\u001b[30ma\u001b[30mn\u001b[30mk\u001b[30m/\u001b[30mo\u001b[30mn\u001b[30ml\u001b[30mi\u001b[30mn\u001b[30me\u001b[30m/\u001b[32mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30mi\u001b[30mn\u001b[30mf\u001b[30mo\u001b[30m.\u001b[30mp\u001b[30mh\u001b[30mp\u001b[32m\n", 911 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mE\u001b[30mn\u001b[30mc\u001b[30mo\u001b[30md\u001b[30mi\u001b[30mn\u001b[30mg\u001b[30m:\u001b[30m \u001b[30mg\u001b[30mz\u001b[30mi\u001b[30mp\u001b[30m,\u001b[30m \u001b[30md\u001b[30me\u001b[30mf\u001b[30ml\u001b[30ma\u001b[30mt\u001b[30me\u001b[32m\n", 912 | "\u001b[32mA\u001b[30mc\u001b[30mc\u001b[30me\u001b[30mp\u001b[30mt\u001b[30m-\u001b[30mL\u001b[30ma\u001b[30mn\u001b[30mg\u001b[30mu\u001b[30ma\u001b[30mg\u001b[30me\u001b[30m:\u001b[30m \u001b[30me\u001b[30mn\u001b[30m-\u001b[30mU\u001b[30mS\u001b[30m,\u001b[30me\u001b[30mn\u001b[30m;\u001b[30mq\u001b[30m=\u001b[30m0\u001b[30m.\u001b[30m9\u001b[32m,\u001b[32mr\u001b[32mu\u001b[32m;\u001b[32mq\u001b[32m=\u001b[30m0\u001b[30m.\u001b[30m8\u001b[32m\n", 913 | "\u001b[32mC\u001b[30mo\u001b[32mo\u001b[32mk\u001b[30mi\u001b[30me\u001b[30m:\u001b[30m \u001b[32mP\u001b[30mH\u001b[30mP\u001b[30mS\u001b[30mE\u001b[30mS\u001b[30mS\u001b[30mI\u001b[30mD\u001b[30m=\u001b[32m7\u001b[32mb\u001b[32mg\u001b[32mg\u001b[32m5\u001b[32mt\u001b[32ma\u001b[32ml\u001b[32md\u001b[32m4\u001b[32ma\u001b[32me\u001b[32ma\u001b[32m8\u001b[32m0\u001b[32m7\u001b[32m9\u001b[32m0\u001b[32mi\u001b[32mt\u001b[32m7\u001b[32ms\u001b[32m1\u001b[32mc\u001b[32mc\u001b[32m1\u001b[32m\n", 914 | "\u001b[32m\n", 915 | "\u001b[30mt\u001b[30my\u001b[30mp\u001b[30me\u001b[32m=\u001b[30mu\u001b[30ms\u001b[30me\u001b[30mr\u001b[30m&\u001b[30ma\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m=\u001b[32mi\u001b[32mn\u001b[32mf\u001b[32mo\u001b[32mu\u001b[32mp\u001b[32md\u001b[32ma\u001b[32mt\u001b[32me\u001b[32m&\u001b[32mf\u001b[30mi\u001b[32mr\u001b[32ms\u001b[32mt\u001b[32mn\u001b[30ma\u001b[32mm\u001b[30me\u001b[30m=\u001b[32mJ\u001b[32mo\u001b[32mh\u001b[32mn\u001b[32m&\u001b[30ml\u001b[30ma\u001b[30ms\u001b[30mt\u001b[30mn\u001b[30ma\u001b[30mm\u001b[30me\u001b[30m=\u001b[32mD\u001b[32mo\u001b[32me\u001b[32m&\u001b[32mp\u001b[32mh\u001b[32mo\u001b[32mn\u001b[32me\u001b[32m=\u001b[32m7\u001b[32m9\u001b[32m0\u001b[32m3\u001b[32m5\u001b[32m2\u001b[32m2\u001b[32m2\u001b[32m2\u001b[32m1\u001b[32m2\u001b[32m&\u001b[32me\u001b[30mm\u001b[32ma\u001b[32mi\u001b[32ml\u001b[30m=\u001b[32mj\u001b[32m.\u001b[32md\u001b[30mo\u001b[32me\u001b[32m%\u001b[32m4\u001b[32m0\u001b[32mv\u001b[32mu\u001b[32ml\u001b[32mn\u001b[32mb\u001b[32ma\u001b[32mn\u001b[32mk\u001b[32m.\u001b[32md\u001b[32me\u001b[32m&\u001b[32mb\u001b[32mi\u001b[32mr\u001b[30mt\u001b[32mh\u001b[32md\u001b[32ma\u001b[32mt\u001b[32me\u001b[32m=\u001b[32m1\u001b[32m9\u001b[32m8\u001b[32m4\u001b[30m-\u001b[32m0\u001b[32m4\u001b[32m-\u001b[32m0\u001b[32m4\u001b[32m&\u001b[32ma\u001b[32mb\u001b[32mo\u001b[32mu\u001b[30mt\u001b[32m=\u001b[32mH\u001b[32mi\u001b[32m!\u001b[30m\n", 916 | "\n", 917 | "Prediction: 1\n", 918 | "Loss 5.8993344\n", 919 | "\u001b[32mP\u001b[32mU\u001b[32mT\u001b[30m \u001b[32m/\u001b[32mc\u001b[32mv\u001b[32mp\u001b[32mh\u001b[32mq\u001b[32mf\u001b[32my\u001b[32mm\u001b[32ml\u001b[32mt\u001b[32m.\u001b[32mt\u001b[32mx\u001b[32mt\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[30mP\u001b[32m/\u001b[32m1\u001b[32m.\u001b[32m1\u001b[32m\n", 920 | "\u001b[30mH\u001b[30mo\u001b[32ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 921 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[32mt\u001b[30me\u001b[30mn\u001b[30mt\u001b[30m-\u001b[32mL\u001b[32me\u001b[32mn\u001b[30mg\u001b[32mt\u001b[32mh\u001b[30m:\u001b[30m \u001b[30m1\u001b[32m0\u001b[32m\n", 922 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mc\u001b[30ml\u001b[30mo\u001b[30ms\u001b[30me\u001b[32m\n", 923 | "\u001b[32m\n", 924 | "\u001b[32mm\u001b[32mb\u001b[32m4\u001b[32m7\u001b[32m0\u001b[32mo\u001b[32mr\u001b[32m8\u001b[32ms\u001b[32ml\u001b[30m\n", 925 | "\n", 926 | "Prediction: 1\n", 927 | "Loss 6.080594\n", 928 | "\u001b[32mT\u001b[32mR\u001b[32mA\u001b[32mC\u001b[32mE\u001b[32m \u001b[32m/\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[32m/\u001b[30m1\u001b[32m.\u001b[32m1\u001b[32m\n", 929 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 930 | "\u001b[32mC\u001b[30mo\u001b[32mo\u001b[32mk\u001b[32mi\u001b[30me\u001b[30m:\u001b[30m \u001b[32mw\u001b[32md\u001b[32m1\u001b[32m2\u001b[32mk\u001b[32mg\u001b[32mj\u001b[32m0\u001b[32mo\u001b[32m3\u001b[32m\n", 931 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[32mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mc\u001b[30ml\u001b[30mo\u001b[30ms\u001b[30me\u001b[30m\n", 932 | "\n", 933 | "Prediction: 1\n", 934 | "Loss 4.7534595\n", 935 | "\u001b[32mG\u001b[32mE\u001b[32mT\u001b[32m \u001b[30m/\u001b[32mc\u001b[32ml\u001b[32mi\u001b[32me\u001b[32mn\u001b[32mt\u001b[32ma\u001b[32mc\u001b[32mc\u001b[32me\u001b[32ms\u001b[30ms\u001b[32mp\u001b[32mo\u001b[32ml\u001b[32mi\u001b[32mc\u001b[32my\u001b[32m.\u001b[32mx\u001b[32mm\u001b[32ml\u001b[30m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[32m/\u001b[32m1\u001b[32m.\u001b[32m1\u001b[32m\n", 936 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[32m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 937 | "\u001b[30mC\u001b[30mo\u001b[30mn\u001b[30mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mc\u001b[30ml\u001b[30mo\u001b[30ms\u001b[30me\u001b[30m\n", 938 | "\n", 939 | "Prediction: 1\n", 940 | "Loss 5.9863434\n", 941 | "\u001b[32mD\u001b[30mE\u001b[32mB\u001b[32mU\u001b[32mG\u001b[30m \u001b[32m/\u001b[32mD\u001b[32me\u001b[32mf\u001b[32ma\u001b[32mu\u001b[32ml\u001b[32mt\u001b[32m.\u001b[32ma\u001b[32ms\u001b[32mp\u001b[32mx\u001b[32m \u001b[32mH\u001b[32mT\u001b[32mT\u001b[32mP\u001b[32m/\u001b[32m1\u001b[32m.\u001b[32m1\u001b[32m\n", 942 | "\u001b[30mH\u001b[30mo\u001b[30ms\u001b[30mt\u001b[30m:\u001b[30m \u001b[30m1\u001b[30m0\u001b[30m.\u001b[30m0\u001b[30m.\u001b[30m2\u001b[30m1\u001b[30m2\u001b[30m.\u001b[30m2\u001b[30m5\u001b[32m\n", 943 | "\u001b[32mC\u001b[30mo\u001b[32mm\u001b[32mm\u001b[32ma\u001b[30mn\u001b[32md\u001b[32m:\u001b[30m \u001b[32ms\u001b[32mt\u001b[32ma\u001b[32mr\u001b[32mt\u001b[32m-\u001b[32md\u001b[32me\u001b[32mb\u001b[32mu\u001b[32mg\u001b[32m\n", 944 | "\u001b[32mC\u001b[30mo\u001b[30mn\u001b[32mn\u001b[30me\u001b[30mc\u001b[30mt\u001b[30mi\u001b[30mo\u001b[30mn\u001b[30m:\u001b[30m \u001b[30mc\u001b[30ml\u001b[30mo\u001b[30ms\u001b[32me\u001b[30m" 945 | ] 946 | } 947 | ], 948 | "source": [ 949 | "pred_data = Data(path_anomaly_data, predict=True)\n", 950 | "pred_gen = pred_data.predict_generator()\n", 951 | "anomaly_preds, anomaly_loss = p.predict(pred_gen)" 952 | ] 953 | }, 954 | { 955 | "cell_type": "code", 956 | "execution_count": 15, 957 | "metadata": {}, 958 | "outputs": [ 959 | { 960 | "name": "stdout", 961 | "output_type": "stream", 962 | "text": [ 963 | "Number of TP: 1097\n", 964 | "Number of samples: 1097\n", 965 | "TP rate: 1.0000\n" 966 | ] 967 | } 968 | ], 969 | "source": [ 970 | "print('Number of TP: ', np.sum(anomaly_preds))\n", 971 | "print('Number of samples: ', len(anomaly_preds))\n", 972 | "print('TP rate: {:.4f}'.format(np.sum(anomaly_preds) / len(anomaly_preds)))" 973 | ] 974 | }, 975 | { 976 | "cell_type": "markdown", 977 | "metadata": {}, 978 | "source": [ 979 | "# Testing " 980 | ] 981 | }, 982 | { 983 | "cell_type": "markdown", 984 | "metadata": {}, 985 | "source": [ 986 | "To evaluate the results, let's compute metrics of quality: precision, recall, ROC-AUC." 987 | ] 988 | }, 989 | { 990 | "cell_type": "code", 991 | "execution_count": 16, 992 | "metadata": {}, 993 | "outputs": [], 994 | "source": [ 995 | "y_true = np.concatenate(([0] * len(valid_preds), [1] * len(anomaly_preds)), axis=0)\n", 996 | "preds = np.concatenate((valid_preds, anomaly_preds), axis=0)\n", 997 | "loss_pred = np.concatenate((valid_loss, anomaly_loss), axis=0)\n", 998 | "assert len(y_true)==len(loss_pred)" 999 | ] 1000 | }, 1001 | { 1002 | "cell_type": "code", 1003 | "execution_count": 17, 1004 | "metadata": {}, 1005 | "outputs": [ 1006 | { 1007 | "name": "stdout", 1008 | "output_type": "stream", 1009 | "text": [ 1010 | "Precision: 0.9928\n", 1011 | "Recall: 1.0000\n" 1012 | ] 1013 | } 1014 | ], 1015 | "source": [ 1016 | "precision = precision_score(y_true, preds)\n", 1017 | "recall = recall_score(y_true, preds)\n", 1018 | "print('Precision: {:.4f}'.format(precision))\n", 1019 | "print('Recall: {:.4f}'.format(recall))" 1020 | ] 1021 | }, 1022 | { 1023 | "cell_type": "code", 1024 | "execution_count": 18, 1025 | "metadata": {}, 1026 | "outputs": [ 1027 | { 1028 | "data": { 1029 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd4VNXWwOHfIvRmASwUBaSlkBAIXakCEQhFepEugkpTr6Li1St+1utFURAREFHpRVBBkCooLUonNEMLIE06JJBkfX/MEAOEyQQymZT1Pk8e5vR1hmTW7L3PWUdUFWOMMeZmsnk7AGOMMembJQpjjDEuWaIwxhjjkiUKY4wxLlmiMMYY45IlCmOMMS5ZojDGGOOSJQqT6YjIPhG5JCLnReQvEZkoIvmvW6eWiCwVkXMickZEvhcRv+vWKSgiH4nIAee+/nROF07bMzLGuyxRmMwqTFXzA5WAYODlqwtEpCawCJgLFAVKAZuAX0WktHOdnMASwB8IBQoCNYGTQDVPBS0i2T21b2NulSUKk6mp6l/AQhwJ46r3gUmq+rGqnlPVv1V1GLAGeMO5TjfgAaC1qm5X1XhVPaaqw1V1flLHEhF/EflZRP4WkaMi8opz/kQReSvRevVEJCrR9D4ReUlENgMXnK9nXrfvj0VkpPP1HSIyXkSOiMghEXlLRHxu860y5qYsUZhMTUSKA48Be5zTeYFawIwkVp8ONHK+fhT4SVXPu3mcAsBi4CccrZQyOFok7uoENAPuBKYCTZ37xJkE2gOTnetOBGKdxwgGGgN9UnAsY1LEEoXJrL4TkXPAQeAY8Lpz/t04fu+PJLHNEeDq+EOhm6xzM82Bv1T1Q1WNdrZU1qZg+5GqelBVL6nqfuAPoLVzWQPgoqquEZF7gabAYFW9oKrHgBFAxxQcy5gUsURhMqtWqloAqAdU4J8EcAqIB+5PYpv7gRPO1ydvss7NlAD+vKVIHQ5eNz0ZRysDoDP/tCYeBHIAR0TktIicBj4H7rmNYxvjkiUKk6mp6gocXTX/dU5fAFYD7ZJYvT3/dBctBpqISD43D3UQKH2TZReAvImm70sq1OumZwD1nF1nrfknURwEYoDCqnqn86egqvq7GacxKWaJwmQFHwGNRCTIOT0U6C4iA0WkgIjc5Rxsrgn8x7nO1zg+lGeJSAURySYihUTkFRFpmsQxfgDuF5HBIpLLud/qzmUbcYw53C0i9wGDkwtYVY8Dy4Evgb2qGuGcfwTHFVsfOi/fzSYiD4lI3Vt4X4xxiyUKk+k5P3QnAf92Tq8CmgCP4xiH2I9jUPhhVd3tXCcGx4D2DuBn4CywDkcX1g1jD6p6DsdAeBjwF7AbqO9c/DWOy2/34fiQn+Zm6JOdMUy+bn43ICewHUdX2kxS1k1mTIqIPbjIGGOMK9aiMMYY45IlCmOMMS5ZojDGGOOSJQpjjDEuZbgCZIULF9aSJUt6OwxjjMlQfv/99xOqWuRWts1wiaJkyZKEh4d7OwxjjMlQRGT/rW5rXU/GGGNcskRhjDHGJUsUxhhjXLJEYYwxxiVLFMYYY1yyRGGMMcYljyUKEZkgIsdEZOtNlouIjBSRPSKyWUQqeyoWY4wxt86TLYqJQKiL5Y8BZZ0/fYHPPBiLMcaYW+SxG+5U9RcRKelilZbAJHXUOV8jIneKyP3OB7N4xOS1B5i78ZCndm+MMenOuaMXKHCvuw9qTJo3xyiKce1zgqOc824gIn1FJFxEwo8fP37LB5y78RBr9/59y9sbY0xGEXP+MmsmbOGnN37l1IGzt7WvDFHCQ1XHAmMBQkJCbutJS9VL3c20p2qmSlzGGJPeqCrTp29jwGsLOH06mn//uy6vvPIIud6+9X16M1EcAkokmi7unGeMMeYWqCodO85i+vRtVK1alPHjW1Cx4r23vV9vJop5wLMiMhWoDpzx5PiEMcZkVqqKiCAiVK1alGrVijJoUA2yZ0+d0QWPJQoRmQLUAwqLSBTwOpADQFXHAPOBpsAe4CLQ01OxGGNMZvXnn3/z5JPfM2hQdVq2rMALL9RK9WN48qqnTsksV+AZTx3fGGMys7i4eD7+eC3Dhi0lRw4fLl684rFjZYjBbGOMMf/YuvUYvXvPY926Q4SFleOzz5pRrFhBjx3PEoUxxmQwGzYcITLyFFOmtKFDB39ExKPHs0RhjDEZwLp1h4iMPEXHjgF07RpI8+bluOuuPGlybCsKaIwx6djFi1d44YVF1Kw5njfeWE5sbDwikmZJAqxFYYwx6dayZXvp0+d7IiNP8dRTVXjvvUdT7ZLXlLBEYYwx6dCuXSdp2HASpUvfxbJl3alXr6TXYrFEYYwx6UhExHF8fYtQrlwhZsxox2OPlSVv3hxejcnGKIwxJh04fvwCnTrNIiDgMzZscBSpaNPGz+tJAqxFYYwxXqWqTJmylYEDF3D2bAxvvFEXf/97vB3WNSxRGGOMl6gq7drNYNasCGrUKM64cWHpLkmAJQpjjElziYv41apVgocffoABA6rh45M+RwPSZ1TGGJNJ7d59kvr1v+K773YA8NxzNRk8uEa6TRJgicIYY9JEbGw8//3vbwQGjmHjxr+Ijo71dkhus64nY4zxsM2bj9K79zzCww/TsmV5Ro9uRtGiBbwdltssURhjjIdt2vQXBw6cYfr0trRt6+fxIn6pzRKFMcZ4wOrVB9m79zSdO1eka9dAwsLKc+edub0d1i2xMQpjjElFFy5cZvDgn6hdewJvvfVLQhG/jJokwFoUxhiTahYvjuTJJ79n377TPPNMVd55p6FXivilNksUxhiTCnbtOknjxl9Tpszd/PJLDx555EFvh5RqLFEYY8xt2L79OH5+jiJ+s2a1JzS0DHnyeL8+U2rK+G0iY4zxgqNHz9O+/QwqVvyniF/r1r6ZLkmAtSiMMSZFVJVvvtnM4MELOX/+MsOH1ycgIP3VZ0pNliiMMcZNqkqbNtOZM2cHNWsWZ/z4Fvj6FvF2WB5nicIYY5KRuIhfnToPUq9eSZ55pmq6rs+UmrLGWRpjzC3aufMEdepMZM6cCAAGD67BwIHVs0ySAEsUxhiTpNjYeN59dxVBQWPYtu0YsbHx3g7Ja6zryRhjrrNx41/07j2PP/44Qps2vnz6aVPuuy+/t8PyGksUxhhznW3bjnHo0FlmzmxHmzZ+3g7H6yxRGGMM8NtvB9m79xRdugTSuXNFwsLKU7BgLm+HlS7YGIUxJks7f/4yAwcu4OGHJ/D226sSivhZkviHtSiMMVnWokV/0rfv9xw4cIZnn63G//1fg0xRxC+1WaIwxmRJO3eeIDT0G8qVK8TKlT2pXfsBb4eUbnk0dYpIqIjsFJE9IjI0ieUPiMgyEdkgIptFpKkn4zHGmC1bjgJQvnxh5szpwMaN/SxJJMNjiUJEfIBRwGOAH9BJRK6/fGAYMF1Vg4GOwGhPxWOMydqOHDlHmzbTCQoawx9/OIr4tWxZgdy5rWMlOZ58h6oBe1Q1EkBEpgItge2J1lGgoPP1HcBhD8ZjjMmCVJWvvtrEkCELuXTpCu+805DAwHu9HVaG4slEUQw4mGg6Cqh+3TpvAItEZACQD3g0qR2JSF+gL8ADD1gT0RjjHlWlVatpzJu3k4cffoBx48IoX76wt8PKcLw9vN8JmKiqxYGmwNcickNMqjpWVUNUNaRIkcxfqdEYc3vi4xUAEaFBg5KMGtWUFSt6WJK4RZ5MFIeAEommizvnJdYbmA6gqquB3ID9TxpjbllExHEeeeRLZs1y9HIPGlSDp5+uSrZs4uXIMi5PJor1QFkRKSUiOXEMVs+7bp0DQEMAEfHFkSiOezAmY0wmdeVKHG+/vZJKlT5nx44TqHo7oszDY2MUqhorIs8CCwEfYIKqbhORN4FwVZ0HPA98ISJDcAxs91C1/15jTMps2HCEXr3msXHjX7Rv78/IkaHce2/WLeKX2jx6XZiqzgfmXzfv34lebwdqezIGY0zmFxFxgr/+Os+cOR1o1aqCt8PJdOwCYmNMhrRy5X727TvNE08E0alTAGFh5ShQwOozeYK3r3oyxpgUOXcuhmee+ZE6dSby/vu/ERfnKOJnScJzLFEYYzKMBQt24+8/ms8+C2fw4OqsWdM7Sz2S1Fus68kYkyHs3HmCZs0m4+tbhN9+602NGsW9HVKWYYnCGJNuqSqbNx8lKOg+ypcvzLx5nWjUqDS5ctlHV1qyNpsxJl06fPgcjz8+neDgzxOK+DVvXs6ShBfYO26MSVdUlS+/3Mhzzy0kJiaO99571Ir4eZlbicJ5Z/UDqrrHw/EYY7IwVaVly6l8//0u6tR5kHHjwihbtpC3w8ryku16EpFmwBbgZ+d0JRGZ4+nAjDFZR+Iifo0aleazz5qxbFl3SxLphDtjFG/iKA9+GkBVNwJlPBmUMSbr2LbtGLVqjWfmTEcRvwEDqtOvX4gV8UtH3EkUV1T19HXzrB6TMea2XL4cx/DhKwgO/pw9e/62xJCOuTNGESEi7YFsIlIKGAis8WxYxpjMLDz8ML16zWXLlmN07BjAyJGhFCmSz9thmZtwp0XxLFAFiAdmAzHAIE8GZYzJ3Pbs+ZuTJy8xd25HpkxpY0kinXOnRdFEVV8CXro6Q0Qex5E0jDHGLStW7GP//jN06xZEhw7+NG9ejvz5c3o7LOMGd1oUw5KY92pqB2KMyZzOno2hf/8fqFfvK/7733+K+FmSyDhu2qIQkSZAKFBMRP6XaFFBHN1Qxhjj0o8/7qJfvx85fPgczz1Xg+HDG1gRvwzIVdfTMWArEA1sSzT/HDDUk0EZYzK+nTtPEBY2BT+/Isyc2Y7q1a2IX0Z100ShqhuADSLyrapGp2FMxpgMSlXZuPEvgoPvp3z5wnz/fScaNXqInDl9vB2auQ3utAGLichUEdksIruu/ng8MmNMhhIVdZaWLadSpcrYhCJ+zZqVsySRCbiTKCYCXwICPAZMB6Z5MCZjTAYSH6+MHfs7/v6jWbw4kg8/bExQkBXxy0zcSRR5VXUhgKr+qarDcCQMY0wWp6qEhU3hqad+oEqV+9mypT9DhtS0AetMxp37KGJEJBvwp4j0Aw4BBTwbljEmPYuLiydbNkFEeOyxMrRuXYHevYMRsTIcmZE7aX8IkA9H6Y7awJNAL08GZYxJv7ZuPUatWhOYNSsCgGefrUafPpUtSWRiybYoVHWt8+U54AkAESnmyaCMMenP5ctxvP32St5+eyV33JGb7NmteymrcJkoRKQqUAxYpaonRMQfRymPBoBdFG1MFrF+/SF69pzLtm3H6dKlIh99FErhwnm9HZZJI67uzH4HaANsAoaJyA/A08B7QL+0Cc8Ykx78+ecpzpyJ4YcfOtGsWTlvh2PSmKsWRUsgSFUvicjdwEGgoqpGpk1oxhhvWrp0LwcOnKFHj0p06OBPWFg58uWz+kxZkatOxmhVvQSgqn8DuyxJGJP5nT4dzZNPzqNhw0mMGLEmoYifJYmsy1WLorSIXC0lLkCpRNOo6uMejcwYk+bmzdtJ//4/8tdf53nxxVq88UY9uyfCuEwUba6b/tSTgRhjvGvnzhO0ajWVgIB7mDu3IyEhRb0dkkknXBUFXJKWgRhj0p6q8scfR6hSpSjlyxdm/vwuNGhQyuozmWtYm9KYLOrgwTOEhU2hatUvEor4hYaWsSRhbuDRRCEioSKyU0T2iEiSz7AQkfYisl1EtonIZE/GY4xxFPEbMyYcf//RLFu2jxEjmlgRP+OSO7WeABCRXKoak4L1fYBRQCMgClgvIvNUdXuidcoCLwO1VfWUiNzjfujGmJRSVZo1m8xPP+3h0UdLM3Zsc0qVusvbYZl0LtkWhYhUE5EtwG7ndJCIfOLGvqsBe1Q1UlUvA1Nx3JuR2JPAKFU9BaCqx1IUvTHGLXFx8agqIkJYWDnGj2/BokVdLUkYt7jT9TQSaA6cBFDVTUB9N7YrhuMmvauinPMSKweUE5FfRWSNiIS6sV9jTAps2vQX1aqNY+ZMR2P+6aer0quXVXo17nMnUWRT1f3XzYtLpeNnB8oC9YBOwBcicuf1K4lIXxEJF5Hw48ePp9KhjcncYmJiee21pYSEfMHBg2dskNrcMnfGKA6KSDVAneMOAwB3HoV6CCiRaLq4c15iUcBaVb0C7HU+YrUssD7xSqo6FhgLEBISom4c25gsbe3aKHr2nEtExAmeeCKQESOaUKiQFfEzt8adFkV/4DngAeAoUMM5LznrgbIiUkpEcgIdgXnXrfMdjtYEIlIYR1eUlQkx5jbt33+GCxeusGBBFyZNam1JwtwWd1oUsaraMaU7VtVYEXkWWAj4ABNUdZuIvAmEq+o857LGIrIdR3fWv1T1ZEqPZYyBxYsjOXDgDL16BdOunR/Nm5cjb94c3g7LZALuJIr1IrITmAbMVtVz7u5cVecD86+b9+9ErxVHa+U5d/dpjLnWqVOXeOGFRUyYsJGgoHvp3j0IH59sliRMqkm260lVHwLeAqoAW0TkOxFJcQvDGJP65syJwM9vNF99tYmhQ2uzZk0fK+JnUp1bv1Gq+puqDgQqA2eBbz0alTEmWTt2nKBNm+ncd19+1q17knfeeZTcud2+h9YYt7lzw11+EekiIt8D64DjQC2PR2aMuYGqsn694+LBChUK89NPXVm3rg+VK9/v5chMZuZOi2Irjiud3lfVMqr6vKqu9XBcxpjrHDhwhqZNJ1O9+riEIn6NGz9Ejhx2f4TxLHfaqaVVNd7jkRhjkhQfr3z22XqGDl2CqjJy5GNUqnSft8MyWchNE4WIfKiqzwOzROSGm9zsCXfGeJ6qEhr6DT//HEnjxg/x+efNKVnyhuIFxniUqxbFNOe/9mQ7Y9JYbGw8Pj6CiNC6dQW6dKlIt25BVp/JeMVNxyhUdZ3zpa+qLkn8A/imTXjGZD0bNhyhWrUvmDHDUcSvf/+qdO9eyZKE8Rp3BrN7JTGvd2oHYkxWFx0dy6uvLqFq1S84fPic3TBn0g1XYxQdcNRnKiUisxMtKgCc9nRgxmQlq1cfpGfPuezceZIePSrx4YeNufvuPN4OyxjA9RjFOhzPoCiO40l1V50DNngyKGOymoMHzxIdHcvChV1p3Pghb4djzDVumihUdS+wF1icduEYk3UsXLiHqKiz9O5dmXbt/AgLK0eePNbdZNKfm45RiMgK57+nROTvRD+nROTvtAvRmMzl778v0aPHd4SGfsuoUeuJi4tHRCxJmHTLVdfT1cedFk6LQIzJCmbN2s4zz8znxImLvPrqIwwbVseK+Jl0z1XX09W7sUsAh1X1sog8DAQC3+AoDmiMcdOOHSdo124GwcH389NPXe3uapNhuPNV5jscj0F9CPgSx6NKJ3s0KmMyCVVlzZoowFHE7+efn2Dt2j6WJEyG4k6iiHc+0/px4BNVHQIU82xYxmR8+/adpkmTb6hZc3xCEb+GDUuTPbt1NZmMxa1HoYpIO+AJoJVzno26GXMTcXHxjBq1nldeWYKIMGpUU2tBmAzNnUTRC3gaR5nxSBEpBUzxbFjGZEyOIn7fsnhxJKGhZfj88+Y88MAd3g7LmNuSbKJQ1a0iMhAoIyIVgD2q+n+eD82YjCNxEb+2bX3p1i2Qrl0DrT6TyRTcecLdI8AeYDwwAdglIrU9HZgxGcUffxwhJGQs06dvA+Cpp0J44gmr9GoyD3dG1UYATVW1tqrWApoBH3s2LGPSv0uXrjB06GKqVfuCo0cvkD9/Tm+HZIxHuDNGkVNVt1+dUNUIEbG/CJOl/frrAXr2nMvu3X/Tu3cwH3zQiLvusiJ+JnNyJ1H8ISJjcNxkB9AFKwposrgjR84TGxvP4sVP0LBhaW+HY4xHuZMo+gEDgRed0yuBTzwWkTHp1IIFu4mKOsuTT1ahTRtfmjcvR+7c7vwJGZOxufwtF5GKwEPAHFV9P21CMiZ9OXnyIkOGLOTrrzdTpcr99OoVjI9PNksSJstwVT32FRzlO7oAP4tIUk+6MybTUlWmT9+Gr+8opkzZyr//XYdff+1lRfxMluPqK1EXIFBVL4hIEWA+jstjjckSduw4QceOM6lSpSiLF7cgMPBeb4dkjFe4ShQxqnoBQFWPi4h9jTKZ3tUifjVrlsDXtwiLF3ejTp0HrT6TydJcJYrSiZ6VLcBDiZ+draqPezQyY9JYZOQp+vb9niVL9vL7732pXPl+GjQo5e2wjPE6V4mizXXTn3oyEGO8JS4unpEj1zJs2DJ8fITPPmtmRfyMScTVg4uWpGUgxniDqtK48TcsXbqXpk3LMmZMM0qUsCJ+xiRm1/eZLOnKlTiyZ8+GiNCxoz+9elWic+eKVp/JmCR4dIROREJFZKeI7BGRoS7WayMiKiIhnozHGID16w9RpcpYpk1zFPF78skqdOlilV6NuRm3E4WI5ErJjkXEBxgFPAb4AZ1ExC+J9QoAg4C1Kdm/MSl18eIV/vWvRdSoMZ6//77EnXfm9nZIxmQI7pQZryYiW4DdzukgEXGnhEc1HM+uiFTVy8BUoGUS6w0H3gOi3Q/bmJRZteoAQUFj+O9/V9OnTzDbtj1NaGgZb4dlTIbgTotiJNAcOAmgqpuA+m5sVww4mGg6iuuetS0ilYESqvqjqx2JSF8RCReR8OPHj7txaGOudfToeVSVpUu78fnnYdxxh7UmjHGXO4PZ2VR1/3X9t3G3e2DnDXz/A3okt66qjgXGAoSEhOjtHttkDT/+uItDh87Rt28V2rTxo3nzcuTKZddvGJNS7rQoDopINUBFxEdEBgO73NjuEFAi0XRx57yrCgABwHIR2QfUAObZgLa5XcePX6BLl9k0bz6FceP+IC4uHsCShDG3yJ1E0R94DngAOIrjA72/G9utB8qKSCnng446AvOuLlTVM6paWFVLqmpJYA3QQlXDU3gOxgCOeyKmTt2Kn99oZszYxhtv1GXVKiviZ8ztSvYrlqoew/EhnyKqGisizwILAR9ggqpuE5E3gXBVned6D8akzI4dJ+jceRZVqxZj/PgWBATc4+2QjMkUkk0UIvIFcMO4gKr2TW5bVZ2Po+ps4nn/vsm69ZLbnzHXi49XfvvtIA8//AC+vkVYurQ7jzzygLUijElF7vw1LQaWOH9+Be4BYjwZlDHu2LPnbxo2nMQjj3zJH38cAaBevZKWJIxJZe50PU1LPC0iXwOrPBaRMcmIi4tnxIg1vPbaMnLm9OGLL8IIDrYifsZ4yq1cBlIKsCe4GK9QVRo1+pply/bRokV5Ro9uSrFiBb0dljGZmjtjFKf4Z4wiG/A3cNO6TcZ4QuIifp07V+Spp6rQvr2/1WcyJg247MwVx19hEFDE+XOXqpZW1elpEZwxAOvWHSI4+HOmTNkKQJ8+lenQIcCShDFpxGWiUFUF5qtqnPPH7oo2aebixSs8//xCatYcz5kzMRQqlMfbIRmTJbkzRrFRRIJVdYPHozHG6Zdf9tOz51wiI0/Rr18V3nuvEQULpqiAsTEmldw0UYhIdlWNBYKB9SLyJ3ABx/OzVVUrp1GMJgs6ceIi2bIJy5d3p27dkt4Ox5gszVWLYh1QGWiRRrGYLG7evJ0cPnyOfv1CePxxX5o1K2v1mYxJB1z9FQqAqv6ZRrGYLOrYsQsMHLiAadO2Ua1aMZ58sjI+PtksSRiTTrj6SywiIs/dbKGq/s8D8ZgsRFX59tstDBr0E+fPX2b48Pq89FJtu7PamHTGVaLwAfLjbFkYk9p27DhBt25zqFGjOOPGtcDPr4i3QzLGJMFVojiiqm+mWSQmS4iPV1atOkCdOg/i61uE5ct7ULt2CWtFGJOOufrrtJaESVW7d5+kfv2vqFt3YkIRvzp1HrQkYUw65+ovtGGaRWEytdjYeN5//1cCA8ewadNfjB/fwor4GZOB3LTrSVX/TstATOakqjz66CRWrNhPq1YVGDWqKUWLFvB2WMaYFLDrD41HXL4cR44cjiJ+3boF8cwzVWnb1s/qMxmTAVnnsEl1q1cfpFKlMUyevAWAXr2CadfOKr0ak1FZojCp5vz5ywwe/BO1a0/gwoUr3Htvfm+HZIxJBdb1ZFLF8uX76NlzLvv2nebZZ6vy9tsNKVDAivgZkxlYojCp4vTpaHLl8mHlyp48/PAD3g7HGJOKLFGYWzZnTgR//XWe/v2r0qpVBZo2LUvOnD7eDssYk8psjMKk2NGj52nXbgaPPz6dSZM2ExcXD2BJwphMyhKFcZuqMmnSJnx9RzFv3k7+7/8a8MsvPezOamMyOet6Mm7bseMEPXvOpUaN4owf34IKFQp7OyRjTBqwRGFcio9XVqzYR/36pfD1LcKKFT2oVasE2bLZPRHGZBXWZ2BuaufOE9StO5EGDSYlFPF7+OEHLEkYk8VYojA3uHIljnfeWUlQ0Bi2bTvGxIktrYifMVmYdT2Za6gqDRtOYuXKA7Rt68cnnzzGfffZHdbGZGWWKAwAMTGx5Mzpg4jQs2clBg+uweOP+3o7LGNMOmBdT4Zffz1AUNAYvv3WUcSvZ89gSxLGmASWKLKwc+diGDBgPo888iXR0bHcf791MRljbuTRRCEioSKyU0T2iMjQJJY/JyLbRWSziCwRkQc9GY/5x9KlewkI+IxRo9YzYEA1tm59moYNS3s7LGNMOuSxMQoR8QFGAY2AKGC9iMxT1e2JVtsAhKjqRRHpD7wPdPBUTOYfZ8/GkDdvDlat6kWtWiW8HY4xJh3zZIuiGrBHVSNV9TIwFWiZeAVVXaaqF52Ta4DiHowny5s5czujRq0DoFWrCmze3M+ShDEmWZ5MFMWAg4mmo5zzbqY3sCCpBSLSV0TCRST8+PHjqRhi1nDkyDkef3wa7drNYPLkrQlF/HLksCJ+xpjkpYvBbBHpCoQAHyS1XFXHqmqIqoYUKVIkbYPLwFSVL7/cgJ/faBYs2MN77z3KihVWxM8YkzKevI/iEJC4X6O4c941RORR4FWgrqrGeDCeLCci4gR9+nxP7dolGDeuBeXKFfJ2SMaYDMiTXy3XA2VFpJSI5AQ6AvMSryAiwcDnQAtVPebBWLKMuLh4li7dC4CfXxFWrerJ8uU9LEkYY26ZxxKFqsYCzwILgQhguqpuE5E3RaSFc7UPgPzADBHLEkFLAAAVwklEQVTZKCLzbrI744aIiOPUqTORhg0nsWGDo4hfzZpW6dUYc3s8WsJDVecD86+b9+9Erx/15PGziitX4nj//V95881fyJ8/J19/3ZpKlayInzEmdVitpwxOVWnQYBKrVh2gfXt/PvnkMe65J5+3wzLGZCKWKDKo6OhYcuVyFPHr0yeY55+vSatWFbwdljEmE7LrJDOgX37ZT2DgZ3zzzWYAunevZEnCGOMxligykLNnY3j66R+pW3cisbHxlChxh7dDMsZkAdb1lEEsXhxJr15ziYo6y5AhNRg+vD758uX0dljGmCzAEkUGcfHiFQoUyMVvv/WmRg0riWWMSTuWKNIpVWXGjO0cPXqeAQOq06JFeZo2LUv27NZbaIxJW/apkw4dPnyO1q2n0aHDTKZN25ZQxM+ShDHGG+yTJx1RVcaP/wM/v1EsXPgnH3zQiOXLrYifMca7rOspHYmIOEHfvj/wyCMPMG5cC8qUudvbIRljjCUKb7taxK9Ro4fw8yvCr7/2olq1YlafyRiTblifhhdt23aM2rUn0LjxNwlF/GrUKG5JwhiTrlii8ILLl+N4880VBAd/zp9/nmLy5MetiJ8xJt2yrqc0pqrUqzeR1auj6Ny5Ih991IQiRayIX3p35coVoqKiiI6O9nYoxriUO3duihcvTo4cOVJtn5Yo0kjiIn79+oXw8ssPExZW3tthGTdFRUVRoEABSpYsiYh1DZr0SVU5efIkUVFRlCpVKtX2a11PaWD58n0EBIzm668dRfy6dQuyJJHBREdHU6hQIUsSJl0TEQoVKpTqLV9LFB505kw0Tz31PfXrfwXAgw9aEb+MzJKEyQg88XtqXU8esmjRn/TqNZcjR87zwgs1+c9/6pM3b+r1GRpjTFqxFoWHREfHctddeVi9ujcffNDYkoS5bT4+PlSqVImAgADCwsI4ffp0wrJt27bRoEEDypcvT9myZRk+fDiqmrB8wYIFhISE4OfnR3BwMM8//7w3TsGlDRs20Lt3b2+H4dI777xDmTJlKF++PAsXLkxynaVLl1K5cmUCAgLo3r07sbGxAJw6dYrWrVsTGBhItWrV2Lp1a8I2H3/8MQEBAfj7+/PRRx8lzN+0aRM1a9akYsWKhIWFcfbsWQC2bNlCjx49PHei11PVDPVTpUoVvVXtx/ym7cf8dsvbuxIfH6+TJ2/Wjz5anTDvypU4jxzLpL3t27d7OwTNly9fwutu3brpW2+9paqqFy9e1NKlS+vChQtVVfXChQsaGhqqn376qaqqbtmyRUuXLq0RERGqqhobG6ujR49O1diuXLly2/to27atbty4MU2PmRLbtm3TwMBAjY6O1sjISC1durTGxsZes05cXJwWL15cd+7cqaqqr732mo4bN05VVV944QV94403VFU1IiJCGzRooKqO/x9/f3+9cOGCXrlyRRs2bKi7d+9WVdWQkBBdvny5qqqOHz9ehw0blnCshg0b6v79+5OMNanfVyBcb/Fz17qeUkFU1Fn69/+RH37YRZ06DzJgQHWyZRMr4pdJ/ef7bWw/fDZV9+lXtCCvh/m7vX7NmjXZvNlxccTkyZOpXbs2jRs3BiBv3rx8+umn1KtXj2eeeYb333+fV199lQoVHE9B9PHxoX///jfs8/z58wwYMIDw8HBEhNdff502bdqQP39+zp8/D8DMmTP54YcfmDhxIj169CB37txs2LCB2rVrM3v2bDZu3Midd94JQNmyZVm1ahXZsmWjX79+HDhwAICPPvqI2rVrX3Psc+fOsXnzZoKCggBYt24dgwYNIjo6mjx58vDll19Svnx5Jk6cyOzZszl//jxxcXGsWLGCDz74gOnTpxMTE0Pr1q35z3/+A0CrVq04ePAg0dHRDBo0iL59+7r9/iZl7ty5dOzYkVy5clGqVCnKlCnDunXrqFmzZsI6J0+eJGfOnJQrVw6ARo0a8c4779C7d2+2b9/O0KFDAahQoQL79u3j6NGjREREUL16dfLmzQtA3bp1mT17Ni+++CK7du2iTp06Cftq0qQJw4cPByAsLIypU6fy4osv3tZ5ucM+yW5DfLzy+efh+PmNYsmSSP73v8YsXdrN7qw2HhUXF8eSJUto0aIF4Oh2qlKlyjXrPPTQQ5w/f56zZ8+ydevWG5YnZfjw4dxxxx1s2bKFzZs306BBg2S3iYqK4rfffuN///sfLVu2ZM6cOQCsXbuWBx98kHvvvZdBgwYxZMgQ1q9fz6xZs+jTp88N+wkPDycgICBhukKFCqxcuZINGzbw5ptv8sorryQs++OPP5g5cyYrVqxg0aJF7N69m3Xr1rFx40Z+//13fvnlFwAmTJjA77//Tnh4OCNHjuTkyZM3HHfIkCFUqlTphp933333hnUPHTpEiRIlEqaLFy/OoUOHrlmncOHCxMbGEh4eDjgS68GDBwEICgpi9uzZgCMR7t+/n6ioKAICAli5ciUnT57k4sWLzJ8/P2Ebf39/5s6dC8CMGTMS5gOEhISwcuXKm/7fpCZrUdyGHTtO8PTT86lXryRffBFG6dJ3eTskkwZS8s0/NV26dIlKlSpx6NAhfH19adSoUaruf/HixUydOjVh+q67kv99bteuHT4+PgB06NCBN998k549ezJ16lQ6dOiQsN/t27cnbHP27FnOnz9P/vz5E+YdOXKEIkWKJEyfOXOG7t27s3v3bkSEK1euJCxr1KgRd9/tKJi5aNEiFi1aRHBwMOBoFe3evZs6deowcuTIhMR18OBBdu/eTaFCha6Jf8SIEe69OW4SEaZOncqQIUOIiYmhcePGCe/P0KFDGTRoEJUqVaJixYoEBwfj4+ODr68vL730Eo0bNyZfvnxUqlQpYZsJEyYwcOBAhg8fTosWLciZ85+nWt5zzz0cPnw4VeO/GUsUKRQbG8+SJZE0aVIGP78irFnTm5CQonbppPG4PHnysHHjRi5evEiTJk0YNWoUAwcOxM/PL+Fb9FWRkZHkz5+fggUL4u/vz++//57QrZNSiX+3r78+P1++f6oK1KxZkz179nD8+HG+++47hg0bBkB8fDxr1qwhd+7cLs8t8b5fe+016tevz5w5c9i3bx/16tVL8piqyssvv8xTTz11zf6WL1/O4sWLWb16NXnz5qVevXpJ3lswZMgQli1bdsP8jh07JnQTXVWsWLFrvtFHRUVRrFixG7atWbNmwjf9RYsWsWvXLgAKFizIl19+mRB3qVKlKF26NAC9e/dOGMh/5ZVXKF7c8RTLChUqsGjRIgB27drFjz/+mHCcq91yacG6nlJgy5aj1Ko1ntDQbxOK+FWtWsyShElTefPmZeTIkXz44YfExsbSpUsXVq1axeLFiwFHy2PgwIEJfdf/+te/ePvttxM+sOLj4xkzZswN+23UqBGjRo1KmD516hQA9957LxEREcTHxyd8Q0+KiNC6dWuee+45fH19E769N27cmE8++SRhvY0bN96wra+vL3v27EmYPnPmTMKH8MSJE296zCZNmjBhwoSEMZRDhw5x7Ngxzpw5w1133UXevHnZsWMHa9asSXL7ESNGsHHjxht+rk8SAC1atGDq1KnExMSwd+9edu/eTbVq1W5Y79ixYwDExMTw3nvv0a9fPwBOnz7N5cuXARg3bhx16tShYMGC12xz4MABZs+eTefOna+ZHx8fz1tvvZWwL3AkjsTddZ5kicINMTGxvP76MipXHsu+faeZOrWNFfEzXhUcHExgYCBTpkwhT548zJ07l7feeovy5ctTsWJFqlatyrPPPgtAYGAgH330EZ06dcLX15eAgAAiIyNv2OewYcM4deoUAQEBBAUFJXzTfvfdd2nevDm1atXi/vvvdxlXhw4d+OabbxK6nQBGjhxJeHg4gYGB+Pn5JZmkKlSowJkzZzh37hwAL774Ii+//DLBwcEJl5cmpXHjxnTu3DnhEtK2bdty7tw5QkNDiY2NxdfXl6FDh1KjRo3k39Rk+Pv70759e/z8/AgNDWXUqFEJXURNmzZN6Ab64IMP8PX1JTAwkLCwsISxnoiICAICAihfvjwLFizg448/Tth3mzZt8PPzIywsjFGjRiVcEDBlyhTKlStHhQoVKFq0KD179kzYZtmyZTRr1uy2z8sdoomutc4IQkJC9OpAUUp1+Hw1ANOeqpnMmv9QVWrVmsCaNVF07RrIiBFNKFw47y0d32RcERER+Pr6ejuMTG3EiBEUKFAgycFuc62YmBjq1q3LqlWryJ79xhGEpH5fReR3VQ25leNZi+ImLl68gqoiIjzzTFV++KETX3/d2pKEMR7Sv39/cuXK5e0wMoQDBw7w7rvvJpkkPMESRRKWLIkkIGA0X321CYCuXQNp1qycl6MyJnPLnTs3TzzxhLfDyBDKli17zQC/p1miSOT06Wj69JnHo49+Tfbs2XjoIbvc1fwjo3XTmqzJE7+ndnms008/7aFXr7kcO3aBl16qzeuv1yVPHqvPZBxy587NyZMnrdS4SdfU+TwKV5ci3wpLFE5XrsRxzz35+P77TlSpUtTb4Zh0pnjx4kRFRXH8+HFvh2KMS1efcJeasmyiUFUmT97CsWMXGDKkJmFh5WnatCw+PtYbZ26UI0eOVH1imDEZiUc/FUUkVER2isgeEbnhDhYRySUi05zL14pISU/Gc9XBg2do3nwKXbvOYe7cncTHO/r0LEkYY8yNPPbJKCI+wCjgMcAP6CQiftet1hs4paplgBHAe56KB0DjlT0rDuLvP5rly/fx8cehLFliRfyMMcYVT36FrgbsUdVIVb0MTAVaXrdOS+Ar5+uZQEPx4Ejh2b8usGHaDqpXL87Wrf0ZOLC6tSKMMSYZnhyjKAYcTDQdBVS/2TqqGisiZ4BCwInEK4lIX+BqMfnzIrLzNuIqvHgxJ5y1uLKiwlz3/mYxWfn8s/K5g51/+VvdMEMMZqvqWGBsauxLRMJv9Tb2zMDOP+uef1Y+d7DzF5Fbq32EZ7ueDgElEk0Xd85Lch0RyQ7cAdz4dBFjjDFe48lEsR4oKyKlRCQn0BGYd90684DuztdtgaVqt78aY0y64rGuJ+eYw7PAQsAHmKCq20TkTRwP+Z4HjAe+FpE9wN84komnpUoXVgZm5591ZeVzBzv/Wz7/DFdm3BhjTNqya0ONMca4ZInCGGOMS5k2UaTX8iFpwY1zf05EtovIZhFZIiIPeiNOT0nu/BOt10ZEVEQy1SWT7py/iLR3/g5sE5HJaR2jJ7nx+/+AiCwTkQ3Ov4Gm3ojTE0RkgogcE5GtN1kuIjLS+d5sFpHKbu1YVTPdD47B8z+B0kBOYBPgd906TwNjnK87AtO8HXcannt9IK/zdf/Mcu7unr9zvQLAL8AaIMTbcafx/39ZYANwl3P6Hm/HncbnPxbo73ztB+zzdtypeP51gMrA1pssbwosAASoAax1Z7+ZtUWR7sqHpKFkz11Vl6nqRefkGhz3uGQW7vzfAwzHUVssOi2DSwPunP+TwChVPQWgqsfSOEZPcuf8FSjofH0HcDgN4/MoVf0FxxWkN9MSmKQOa4A7ReT+5PabWRNFUuVDit1sHVWNBa6WD8no3Dn3xHrj+IaRWSR7/s7mdglV/TEtA0sj7vz/lwPKicivIrJGRELTLDrPc+f83wC6ikgUMB8YkDahpQsp/XwAMkgJD+MZItIVCAHqejuWtCIi2YD/AT28HIo3ZcfR/VQPR2vyFxGpqKqnvRpV2ukETFTVD0WkJo57uQJUNd7bgaVXmbVFkZXLh7hz7ojIo8CrQAtVjUmj2NJCcudfAAgAlovIPhz9tPMy0YC2O///UcA8Vb2iqnuBXTgSR2bgzvn3BqYDqOpqIDeOgoFZgVufD9fLrIkiK5cPSfbcRSQY+BxHkshM/dOQzPmr6hlVLayqJVW1JI4xmhaqessF09IZd373v8PRmkBECuPoiopMyyA9yJ3zPwA0BBARXxyJIqs843Ye0M159VMN4IyqHkluo0zZ9aTpt3yIx7l57h8A+YEZzvH7A6rawmtBpyI3zz/TcvP8FwKNRWQ7EAf8S1UzQ2va3fN/HvhCRIbgGNjukUm+JCIiU3B8CSjsHIN5HcgBoKpjcIzJNAX2ABeBnm7tN5O8P8YYYzwks3Y9GWOMSSWWKIwxxrhkicIYY4xLliiMMca4ZInCGGOMS5YoTLojInEisjHRT0kX65a8WaXMFB5zubPi6CZnaYvyt7CPfiLSzfm6h4gUTbRsnIj4pXKc60WkkhvbDBaRvLd7bJN1WaIw6dElVa2U6GdfGh23i6oG4SgW+UFKN1bVMao6yTnZAyiaaFkfVd2eKlH+E+do3ItzMGCJwtwySxQmQ3C2HFaKyB/On1pJrOMvIuucrZDNIlLWOb9rovmfi4hPMof7BSjj3Lah87kFW5y1/nM5578r/zzT47/OeW+IyAsi0hZHDa1vncfM42wJhDhbHQkf7s6Wx6e3GOdqEhV0E5HPRCRcHM+Y+I9z3kAcCWuZiCxzzmssIqud7+MMEcmfzHFMFmeJwqRHeRJ1O81xzjsGNFLVykAHYGQS2/UDPlbVSjg+qKOcJRo6ALWd8+OALskcPwzYIiK5gYlAB1WtiKOSQX8RKQS0BvxVNRB4K/HGqjoTCMfxzb+Sql5KtHiWc9urOgBTbzHOUBzlOK56VVVDgECgrogEqupIHGW066tqfWfJjmHAo873Mhx4LpnjmCwuU5bwMBneJeeHZWI5gE+dffJxOOoTXW818KqIFAdmq+puEWkIVAHWO8uV5MGRdJLyrYhcAvbhKD1dHtirqrucy78CngE+xfEci/Ei8gPwg7snpqrHRSTSWWdnN1AB+NW535TEmRNHGZbE71N7EemL4+/6fhwP5dl83bY1nPN/dR4nJ473zZibskRhMoohwFEgCEdL+IYHDqnqZBFZCzQD5ovIUzie5PWVqr7sxjG6JC4OKCJ3J7WSs55QNRyF5doCzwINUnAuU4H2wA5gjqqqOD613Y4T+B3H+MQnwOMiUgp4AaiqqqdEZCKOYnfXE+BnVe2UgnhNFmddTyajuAM44nxmwBM4Cr5dQ0RKA5HO7pa5OLpglgBtReQe5zp3i/vPCN8JlBSRMs7pJ4AVzj79O1R1Po4EFpTEtudwlDRPyhwcTxrrhCNpkNI4nUXsXgNqiEgFHE9suwCcEZF7gcduEssaoPbVcxKRfCKSVOvMmASWKExGMRroLiKbcHTXXEhinfbAVhHZiOOZE5OcVxoNAxaJyGbgZxzdMslS1Wgc1TVniMgWIB4Yg+ND9wfn/laRdB//RGDM1cHs6/Z7CogAHlTVdc55KY7TOfbxIY7qr5twPAd7BzAZR3fWVWOBn0Rkmaoex3FF1hTncVbjeD+NuSmrHmuMMcYla1EYY4xxyRKFMcYYlyxRGGOMcckShTHGGJcsURhjjHHJEoUxxhiXLFEYY4xx6f8Btud2rBykvc4AAAAASUVORK5CYII=\n", 1030 | "text/plain": [ 1031 | "" 1032 | ] 1033 | }, 1034 | "metadata": {}, 1035 | "output_type": "display_data" 1036 | } 1037 | ], 1038 | "source": [ 1039 | "fpr, tpr, _ = roc_curve(y_true, loss_pred)\n", 1040 | "roc_auc = auc(fpr, tpr)\n", 1041 | "\n", 1042 | "plt.figure()\n", 1043 | "plt.plot(fpr, tpr, label='ROC curve (area = %0.4f)' % roc_auc)\n", 1044 | "plt.plot([0, 1], [0, 1], color='navy', linestyle='--')\n", 1045 | "plt.xlim([-0.05, 1.0])\n", 1046 | "plt.ylim([0.0, 1.05])\n", 1047 | "plt.xlabel('False Positive Rate')\n", 1048 | "plt.ylabel('True Positive Rate')\n", 1049 | "plt.title('ROC curve')\n", 1050 | "plt.legend(loc=\"lower right\")\n", 1051 | "plt.show()" 1052 | ] 1053 | }, 1054 | { 1055 | "cell_type": "code", 1056 | "execution_count": null, 1057 | "metadata": {}, 1058 | "outputs": [], 1059 | "source": [] 1060 | } 1061 | ], 1062 | "metadata": { 1063 | "kernelspec": { 1064 | "display_name": "Python 2", 1065 | "language": "python", 1066 | "name": "python2" 1067 | }, 1068 | "language_info": { 1069 | "codemirror_mode": { 1070 | "name": "ipython", 1071 | "version": 2 1072 | }, 1073 | "file_extension": ".py", 1074 | "mimetype": "text/x-python", 1075 | "name": "python", 1076 | "nbconvert_exporter": "python", 1077 | "pygments_lexer": "ipython2", 1078 | "version": "2.7.14" 1079 | } 1080 | }, 1081 | "nbformat": 4, 1082 | "nbformat_minor": 2 1083 | } 1084 | -------------------------------------------------------------------------------- /slides/detecting_web_attacks_rnn.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/slides/detecting_web_attacks_rnn.pdf -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PositiveTechnologies/seq2seq-web-attack-detection/6f18988a32334fbe49886b1fd29fa6fa2416bfe3/utils/__init__.py -------------------------------------------------------------------------------- /utils/reader.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import random 3 | import numpy as np 4 | 5 | from sklearn.model_selection import train_test_split 6 | 7 | from vocab import Vocabulary 8 | from utils import get_requests_from_file, batch_generator, one_by_one_generator 9 | 10 | 11 | class Reader(object): 12 | 13 | def __init__(self, data_path, vocab=Vocabulary()): 14 | self.vocab = vocab 15 | 16 | data = get_requests_from_file(data_path) 17 | print("Downloaded {} samples".format(len(data))) 18 | 19 | map_result = map(self._process_request, data) 20 | self.data = [x[0] for x in map_result] 21 | self.lengths = [x[1] for x in map_result] 22 | assert len(self.data) == len(self.lengths) 23 | 24 | def _process_request(self, req): 25 | """ 26 | Splits a request into lines and convert a string into ints. 27 | """ 28 | seq = self.vocab.string_to_int(req) 29 | l = len(seq) 30 | 31 | return seq, l 32 | 33 | 34 | class Data(Reader): 35 | 36 | def __init__(self, data_path, vocab=Vocabulary(), predict=False): 37 | """ 38 | Creates an object that gets data from a file. 39 | """ 40 | super(Data, self).__init__(data_path, vocab) 41 | 42 | if not predict: 43 | self._train_test_split() 44 | 45 | def _train_test_split(self): 46 | """ 47 | Train/val/test split for anomaly detection problem. 48 | """ 49 | # Shuffle requests 50 | data, lengths = self._shuffle(self.data, self.lengths) 51 | 52 | # Split into train/val/test 53 | X_train, X_test, l_train, l_test = train_test_split(data, lengths, test_size=0.1) 54 | X_train, X_val, l_train, l_val = train_test_split(X_train, l_train, test_size=0.2) 55 | 56 | self.X_train, self.l_train = X_train, l_train 57 | self.X_val, self.l_val = X_val, l_val 58 | self.X_test, self.l_test = X_test, l_test 59 | 60 | self.train_size = len(X_train) 61 | self.val_size = len(X_val) 62 | self.test_size = len(X_test) 63 | 64 | def _shuffle(self, data, lengths): 65 | temp = list(zip(data, lengths)) 66 | random.shuffle(temp) 67 | data, lengths = zip(*temp) 68 | 69 | return data, lengths 70 | 71 | def train_generator(self, batch_size, num_epochs): 72 | return batch_generator( 73 | self.X_train, 74 | self.l_train, 75 | num_epochs, 76 | batch_size, 77 | self.vocab) 78 | 79 | def val_generator(self): 80 | return one_by_one_generator( 81 | self.X_val, 82 | self.l_val, 83 | self.vocab) 84 | 85 | def test_generator(self): 86 | return one_by_one_generator( 87 | self.X_test, 88 | self.l_test, 89 | self.vocab) 90 | 91 | def predict_generator(self): 92 | return one_by_one_generator( 93 | self.data, 94 | self.lengths, 95 | self.vocab) 96 | -------------------------------------------------------------------------------- /utils/utils.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | from __future__ import division 3 | from __future__ import print_function 4 | 5 | import os 6 | import re 7 | 8 | import numpy as np 9 | 10 | HTTP_RE = re.compile(r"ST@RT.+?INFO\s+(.+?)\s+END", re.MULTILINE | re.DOTALL) 11 | 12 | 13 | def http_re(data): 14 | """ 15 | Extracts HTTP requests from raw data string in special logging format. 16 | 17 | Logging format `ST@RT\n%(asctime)s %(levelname)-8s\n%(message)s\nEND` 18 | where `message` is a required HTTP request bytes. 19 | """ 20 | return HTTP_RE.findall(data) 21 | 22 | 23 | def get_requests_from_file(path): 24 | """ 25 | Reads raw HTTP requests from given file. 26 | """ 27 | with open(path, 'r') as f: 28 | file_data = f.read() 29 | requests = http_re(file_data) 30 | return requests 31 | 32 | 33 | def batch_generator(inputs, lengths, num_epochs, batch_size, vocab): 34 | """ 35 | Generates a padded batch. 36 | """ 37 | i = 0 38 | input_size = len(inputs) 39 | for _ in range(num_epochs): 40 | while i + batch_size <= input_size: 41 | l = lengths[i:i + batch_size] 42 | padded = batch_padding(inputs[i:i + batch_size], l, vocab) 43 | yield padded, l 44 | i += batch_size 45 | i = 0 46 | 47 | 48 | def one_by_one_generator(inputs, lengths, vocab): 49 | """ 50 | Yields a sample. 51 | """ 52 | for i in range(len(inputs)): 53 | yield [inputs[i]], lengths[i] 54 | 55 | 56 | def batch_padding(inputs, lengths, vocab): 57 | """ 58 | Pads sequences to max sequence length in a batch. 59 | """ 60 | max_len = np.max(lengths) 61 | padded = [] 62 | for sample in inputs: 63 | padded.append( 64 | sample + ([vocab.vocab['']] * (max_len - len(sample)))) 65 | return padded 66 | 67 | 68 | def print_progress(step, epoch, loss, step_loss, time): 69 | """ 70 | Prints learning stage progress. 71 | """ 72 | msg = "Step {} (epoch {}), average_train_loss = {:.5f}, step_loss = {:.5f}, time_per_step = {:.3f}" 73 | msg = msg.format(step, epoch, loss, step_loss, time) 74 | print(msg) 75 | 76 | 77 | def create_checkpoints_dir(checkpoints_dir): 78 | """ 79 | Creates the checkpoints directory if it does not exist. 80 | """ 81 | if not os.path.exists(checkpoints_dir): 82 | os.makedirs(checkpoints_dir) 83 | -------------------------------------------------------------------------------- /utils/vocab.json: -------------------------------------------------------------------------------- 1 | {"\f": 103, "": 3, " ": 98, "$": 69, "(": 73, ",": 77, "0": 4, "4": 8, "": 2, "8": 12, "<": 83, "@": 87, "D": 43, "H": 47, "L": 51, "P": 55, "T": 59, "X": 63, "\\": 89, "`": 93, "d": 17, "h": 21, "l": 25, "p": 29, "t": 33, "x": 37, "|": 95, "\u000b": 102, "#": 68, "'": 72, "+": 76, "/": 80, "3": 7, "7": 11, ";": 82, "?": 86, "C": 42, "G": 46, "K": 50, "O": 54, "S": 58, "W": 62, "[": 88, "_": 92, "c": 16, "g": 20, "k": 24, "o": 28, "s": 32, "w": 36, "{": 94, "\n": 100, "": 1, "\"": 67, "&": 71, "*": 75, ".": 79, "2": 6, "6": 10, ":": 81, ">": 85, "B": 41, "F": 45, "J": 49, "N": 53, "R": 57, "V": 61, "Z": 65, "^": 91, "b": 15, "f": 19, "j": 23, "n": 27, "r": 31, "v": 35, "z": 39, "~": 97, "\t": 99, "\r": 101, "!": 66, "%": 70, ")": 74, "-": 78, "1": 5, "": 0, "5": 9, "9": 13, "=": 84, "A": 40, "E": 44, "I": 48, "M": 52, "Q": 56, "U": 60, "Y": 64, "]": 90, "a": 14, "e": 18, "i": 22, "m": 26, "q": 30, "u": 34, "y": 38, "}": 96} -------------------------------------------------------------------------------- /utils/vocab.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | from __future__ import division 3 | from __future__ import print_function 4 | 5 | import os 6 | import numpy as np 7 | import json 8 | 9 | 10 | class Vocabulary(): 11 | 12 | def __init__(self): 13 | self.vocab_file = os.path.dirname(os.path.abspath(__file__)) + '/vocab.json' 14 | with open(self.vocab_file, 'r') as f: 15 | self.vocab = json.load(f) 16 | 17 | self.reverse_vocab = {v: k for k, v in self.vocab.items()} 18 | 19 | def string_to_int(self, text): 20 | """ 21 | Converts a string into its character integer representation. 22 | """ 23 | try: 24 | text = text.decode('utf-8') 25 | characters = list(text) 26 | except Exception as e: 27 | characters = [''] 28 | 29 | characters.append('') 30 | char_ids = [self.vocab.get(char, self.vocab['']) 31 | for char in characters] 32 | 33 | return char_ids 34 | 35 | def int_to_string(self, char_ids): 36 | """ 37 | Decodes a list of integers into it's string representation. 38 | """ 39 | characters = [] 40 | for i in char_ids: 41 | characters.append(self.reverse_vocab[i].encode('utf-8')) 42 | 43 | return characters 44 | --------------------------------------------------------------------------------