├── README.md ├── MPC_RNN_CSTR_without Lyapunov constraints.ipynb ├── LMPC_RNN_CSTR.ipynb └── RNN_CSTR.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # RNN-based-MPC 2 | A CSTR example is used to illustrate the application of LMPC using RNN models to maintain the closed-loop state within the stability region. 3 | ## 1. Continuous Stireed Tank Reactor (CSTR) Example 4 | 5 | - Let us consider a second-order, exothermic, irreversible reaction from A to B 6 | 7 | 8 | 9 | 10 | 11 | 12 | - The First Principle equation for this system are as follows: 13 | 14 | ![fp](https://github.com/user-attachments/assets/d0da2dff-9f8f-428c-bc7a-e553df876acf) 15 | 16 | 17 | 18 | 19 |

20 | 21 | 22 | 23 | - Where, 24 | 25 | - 𝐶_𝐴: Concentration of reactant A (kmol/m3) 26 | - 𝑇: Temperature of the reactor (K) 27 | - 𝐶_𝐴0: Concentration of reactant A in the feed 28 | - 𝑄 : Heat input rate (kJ/h) 29 | - F: feed volumetric flow rate (m3/h) 30 | - 𝑇0: Inlet temperature (K) 31 | 32 | 33 | - The State and Manipulated variable for this system is: 34 | 35 | - States variables: 𝐱=[𝐶_𝐴−𝐶_𝐴𝑠, 𝑇 −𝑇_𝑠 ] 36 | - Control / Manipulated variables: 𝐮=[𝐶_𝐴0−𝐶_𝐴0𝑠, 𝑄 −𝑄_𝑠 ] 37 | 38 | 39 | - The generated dataset with the input and output will look like: 40 | 41 | ![image](https://github.com/GuoQWu/RNN-based-MPC/assets/85721266/e0e9f633-f1c6-4e22-922f-59ba22f60b0c) 42 | 43 | 44 | - The above data sample is for a dataset with n samples with 10 internal time-steps for each sample. 45 | 46 | - The code for generating the data set for RNN model is available [here](https://github.com/GuoQWu/RNN-based-MPC/blob/main/RNN_CSTR.ipynb) 47 | 48 | 49 | ## 2. Recurrent Neural Network (RNN) Structure 50 | 51 | - RNN are a class of neural networks that is powerful for modeling sequence data such as time series data. 52 | - It can handle sequential data and account for past information in the current prediction. 53 | 54 | - The RNN model input and output are as follows: 55 | - Input: System initial state variable 𝐱_𝟎(𝑡_𝑘), and control variables 𝐮(𝑡_𝑘). 56 | - Output: Future state dynamics 𝐱(𝑡_𝑘+Δ) are predicted for one sampling period ∆. 57 | 58 | 59 |

60 | 61 |

62 | 63 | ## 3. RNN-based MPC for CSTR 64 | - This repository contains two versions of our algorithm. 65 | 66 | - The first version of the MPC operates without Lyapunov constraints [Code](https://github.com/GuoQWu/RNN-based-MPC/blob/main/MPC_RNN_CSTR_without%20Lyapunov%20constraints.ipynb). 67 | 68 | - The second version incorporates Lyapunov-based constraints to ensure system stability and safety. To enhance computational efficiency, we made simplifications to the Lyapunov-based constraints in this code. Specifically, we use the constraint, dot_V(x,u) < C V instead of dot_V (x,u) < dot_V(x, \phi(x)) to make sure dot_V is negative, where C is a constant smaller than 0. [Code](https://github.com/GuoQWu/RNN-based-MPC/blob/main/LMPC_RNN_CSTR.ipynb) 69 | 70 | 71 | ## 4. Libraries and Tools used 72 | 73 | [Tensorflow](https://www.tensorflow.org/): Deep learning framework used for building and training neural networks. 74 | 75 | [Matplotlib](https://matplotlib.org/): Python plotting library for creating visualizations. 76 | 77 | [SciPy](https://www.scipy.org/): Open-source library used for scientific and technical computing. 78 | 79 | [scikit-learn](https://scikit-learn.org/): Machine learning library for Python used for data analysis and modeling. 80 | 81 | 82 | ## 5. Citation 83 | 84 | The demonstration of these examples are adapted from the following literature work: 85 | 86 | Wu, Z., Tran, A., Rincon, D., & Christofides, P. D. (2019). Machine learning‐based predictive control of nonlinear processes. Part I: theory. AIChE Journal, 65(11), e16729. 87 | 88 | Wu, Z., Tran, A., Rincon, D., & Christofides, P. D. (2019). Machine‐learning‐based predictive control of nonlinear processes. Part II: Computational implementation. AIChE Journal, 65(11), e16734. 89 | 90 | Additionally, you can find our paper [here]( https://doi-org.libproxy1.nus.edu.sg/10.1002/aic.16734) 91 | -------------------------------------------------------------------------------- /MPC_RNN_CSTR_without Lyapunov constraints.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "2024-03-04 14:49:11.474911: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA\n", 13 | "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", 14 | "2024-03-04 14:49:11.603091: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", 15 | "2024-03-04 14:49:11.606926: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n", 16 | "2024-03-04 14:49:11.606943: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n", 17 | "2024-03-04 14:49:12.131029: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", 18 | "2024-03-04 14:49:12.131060: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", 19 | "2024-03-04 14:49:12.131064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n", 20 | "/home/wulab/.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:20: UserWarning: Pandas requires version '2.7.3' or newer of 'numexpr' (version '2.7.1' currently installed).\n", 21 | " from pandas.core.computation.check import NUMEXPR_INSTALLED\n" 22 | ] 23 | } 24 | ], 25 | "source": [ 26 | "from __future__ import print_function\n", 27 | "from tensorflow.keras.models import model_from_json, load_model\n", 28 | "import numpy\n", 29 | "import numpy as np\n", 30 | "import time\n", 31 | "import os\n", 32 | "import math\n", 33 | "import pyipopt\n", 34 | "from numpy import *\n", 35 | "import tensorflow as tf\n", 36 | "import matplotlib.pyplot as plt" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "\n", 46 | "################################### Simulation time step ##################################\n", 47 | "delta = 0.01 # sampling time\n", 48 | "hc = 1e-4 # integration time step\n", 49 | "oper_time = 0.01 \n", 50 | "\n", 51 | "################################## Initial states #########################################\n", 52 | "CAi=-1.65\n", 53 | "Ti=72\n", 54 | "x1_nn=CAi\n", 55 | "x2_nn=Ti\n", 56 | "x1_record=[CAi]\n", 57 | "x2_record=[Ti]\n", 58 | "u1_record=[]\n", 59 | "u2_record=[]\n", 60 | "\n", 61 | "##################################### Constants ###########################################\n", 62 | "a=1060\n", 63 | "b=22\n", 64 | "d=0.52 # Lyapunov function V=x^T*P*x\n", 65 | "\n", 66 | "# CSTR PARAMETERS\n", 67 | "F=5\n", 68 | "V=1\n", 69 | "k0=8460000\n", 70 | "E=50000 # parametric drift #####E has the most effect on process dynamics######\n", 71 | "R=8.314\n", 72 | "T0=300\n", 73 | "Dh=-11500\n", 74 | "rho=1000\n", 75 | "sigma=1000\n", 76 | "cp=0.231\n", 77 | "Qs=0\n", 78 | "CA0s=4\n", 79 | "w1_std=2.5 #disturbance std\n", 80 | "w2_std=70 #disturbance std\n", 81 | "\n", 82 | "\n", 83 | "########################################### steady-state ###################################\n", 84 | "CAs= 1.9537\n", 85 | "Ts= 401.8727\n", 86 | "\n", 87 | "# state_ss=numpy.array([Ts, CAs])\n", 88 | "# input_ss=numpy.array([Qs, CA0s])\n", 89 | "state_ss=numpy.array([CAs,Ts])\n", 90 | "input_ss=numpy.array([CA0s,Qs])\n", 91 | "\n", 92 | "ROOT_FOLDER=os.getcwd()\n", 93 | "\n", 94 | "##################################### MPC Parameters ######################################\n", 95 | "\n", 96 | "NUM_MPC_ITERATION=10 # MPC TOTAL ITERATION\n", 97 | "\n", 98 | "\n", 99 | "NUM_OUTPUTS = 2 # Number of state variables (RNN output) \n", 100 | "NUM_INPUTS = 4 # Number of RNN input\n", 101 | "HORIZON = 2 ## MPC PREDICTION HORIZON (Depends on how many delta you want to predict with the RNN)\n", 102 | "\n", 103 | " \n", 104 | "NUM_IN_SEQUENCE = 10 # Number of integration time steps actually used in MPC (equal to timestep of ML model)\n", 105 | "\n", 106 | "NUM_MPC_INPUTS = 2*HORIZON # Number of MPC inputs to be optimized, 2 is the number of control inputs \n", 107 | "NUM_MPC_CONSTRAINTS = HORIZON # For each sampling time within prediction horizon, we have 1 constraint\n", 108 | "\n", 109 | "realtime_data = None\n", 110 | "\n", 111 | "setpoint=[0, 0]\n" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 3, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stderr", 121 | "output_type": "stream", 122 | "text": [ 123 | "2024-03-04 14:49:12.896807: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 124 | "2024-03-04 14:49:12.897072: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n", 125 | "2024-03-04 14:49:12.897116: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory\n", 126 | "2024-03-04 14:49:12.897146: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory\n", 127 | "2024-03-04 14:49:12.899139: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory\n", 128 | "2024-03-04 14:49:12.899191: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory\n", 129 | "2024-03-04 14:49:12.899214: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory\n", 130 | "2024-03-04 14:49:12.899222: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n", 131 | "Skipping registering GPU devices...\n", 132 | "2024-03-04 14:49:12.899492: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA\n", 133 | "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" 134 | ] 135 | }, 136 | { 137 | "name": "stdout", 138 | "output_type": "stream", 139 | "text": [ 140 | "\n" 141 | ] 142 | } 143 | ], 144 | "source": [ 145 | "# define scalers for both X and y\n", 146 | "X_mean = np.load('X_mean.npy')\n", 147 | "X_std = np.load('X_std.npy')\n", 148 | "y_mean = np.load('y_mean.npy')\n", 149 | "y_std = np.load('y_std.npy')\n", 150 | "\n", 151 | "model = load_model('model.h5')\n", 152 | "print(model)\n", 153 | "\n", 154 | "x1_mean= X_mean[0] # CA\n", 155 | "x1_std= X_std[0]\n", 156 | "x2_mean=X_mean[1] # T\n", 157 | "x2_std= X_std[1]\n", 158 | "u1_mean= X_mean[2] # CA0\n", 159 | "u1_std=X_std[2]\n", 160 | "u2_mean=X_mean[3] # Q\n", 161 | "u2_std=X_std[3]\n", 162 | "y1_mean=y_mean[0] # CA\n", 163 | "y1_std=y_std[0]\n", 164 | "y2_mean=y_mean[1] # T\n", 165 | "y2_std=y_std[1]\n", 166 | "\n", 167 | "state_mean = np.array([x1_mean, x2_mean]) # [CA_input, T_input]\n", 168 | "state_std = np.array([x1_std, x2_std])\n", 169 | "\n", 170 | "input_mean = np.array([u1_mean, u2_mean]) # [CA0, Q]\n", 171 | "input_std = np.array([u1_std, u2_std])\n", 172 | "\n", 173 | "output_mean = np.array([y1_mean, y2_mean]) # [CA_output, T_output]\n", 174 | "output_std = np.array([y1_std, y2_std])" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 4, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "def my_ens_prediction(num_horizon,my_rawdata,my_inputs):\n", 184 | " xx = [] \n", 185 | " nn_inputs = [] # NN input\n", 186 | " ensemble_output = numpy.zeros((num_horizon,NUM_OUTPUTS,NUM_IN_SEQUENCE))\n", 187 | " \n", 188 | " ensemble_output = ensemble_output.reshape(num_horizon,NUM_IN_SEQUENCE,NUM_OUTPUTS)\n", 189 | " x_test2 = my_rawdata[0:NUM_OUTPUTS].astype(float)\n", 190 | " x_test2= (x_test2-state_mean)/state_std # my_rawdata is normal value; needs to normalize before feeding into NN\n", 191 | "\n", 192 | " predict_output_normal=[[0 for i in range(NUM_OUTPUTS)] for j in range(NUM_IN_SEQUENCE)]\n", 193 | "\n", 194 | " for i_model in range(num_horizon): \n", 195 | " \n", 196 | " # ############################# get the normalized input for RNN #########################\n", 197 | " \n", 198 | " my_inputs_normalized = (my_inputs[2*i_model:2*(i_model+1)] - input_mean) / input_std # my_inputs is also normal value; needs to normalize before feeding into NN\n", 199 | " xx = numpy.concatenate((x_test2, my_inputs_normalized), axis=None).reshape((1, NUM_INPUTS)) # xx is the NN input\n", 200 | " xx = numpy.tile(xx, (NUM_IN_SEQUENCE, 1)) # duplicate #NUM_IN_SEQUENCE times\n", 201 | "\n", 202 | " nn_inputs = xx.reshape(1, NUM_IN_SEQUENCE, NUM_INPUTS) \n", 203 | " # ############## use machine learning model to predict the next sampling time ##############\n", 204 | " \n", 205 | " predict_output = model(nn_inputs).numpy()\n", 206 | " predict_output = predict_output.reshape(NUM_IN_SEQUENCE, NUM_OUTPUTS)\n", 207 | " predict_output = predict_output * output_std + output_mean # convert back to deviation form\n", 208 | "\n", 209 | " ####################### get the system state at the end of sampling time ################\n", 210 | " \n", 211 | " x_test2 = predict_output[-1, :]\n", 212 | "\n", 213 | "\n", 214 | " ###### transform the predicted states back to their original values in deviation form ######\n", 215 | " \n", 216 | " x_test2 = (x_test2 - state_mean) / state_std # normalize input for next prediction\n", 217 | "\n", 218 | " ensemble_output[i_model,:,:] = predict_output[:, :] # store the predicted states \n", 219 | "\n", 220 | " ########################################################################################\n", 221 | "\n", 222 | "\n", 223 | " return ensemble_output \n" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 5, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "def eval_f(x):\n", 233 | " '''\n", 234 | " define the objective function\n", 235 | " '''\n", 236 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 237 | " offset=0\n", 238 | " \n", 239 | " #### CALCULATE OUTLET CONC ###########\n", 240 | " df_ensemble_output = my_ens_prediction(num_horizon = HORIZON, my_rawdata=realtime_data, my_inputs=x)\n", 241 | "\n", 242 | " #### account for all intermediate steps ####\n", 243 | " for j in range (HORIZON):\n", 244 | " est_outlet_product = df_ensemble_output[j, :, 0:2]\n", 245 | " for i in range (int(NUM_IN_SEQUENCE)): \n", 246 | " \n", 247 | " offset = offset + (setpoint[0] - (est_outlet_product[i, 1])) ** 2.0 + (setpoint[1] - (est_outlet_product[i, 0])) ** 2.0 * 1000\n", 248 | "\n", 249 | "\n", 250 | "\n", 251 | " return offset/100\n", 252 | "\n", 253 | "def eval_grad_f(x):\n", 254 | " '''\n", 255 | " define the gradient of the objective function\n", 256 | " '''\n", 257 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 258 | " step = 1e-1 # we just have a small step\n", 259 | " objp=objm=0\n", 260 | " grad_f = [0]*NUM_MPC_INPUTS\n", 261 | " xpstep = [0]*NUM_MPC_INPUTS\n", 262 | " xmstep = [0]*NUM_MPC_INPUTS\n", 263 | " for i_mpc_input in range(NUM_MPC_INPUTS):\n", 264 | " xpstep=x.copy()\n", 265 | " xmstep=x.copy()\n", 266 | " # for each variables, we need to evaluate the derivative of the function with respect to that variable, This is why we have the for loop\n", 267 | " xpstep[i_mpc_input] = xpstep[i_mpc_input]+step \n", 268 | " xmstep[i_mpc_input] = xmstep[i_mpc_input]-step\n", 269 | "\n", 270 | " # Evaluate the objective function at xpstep and xmstep\n", 271 | " objp=eval_f(xpstep) # This function returns the value of the objective function evaluated with the variable x[i] is perturebed +step\n", 272 | " objm=eval_f(xmstep) # This function returns the value of the objective function evaluated with the variable x[i] is perturebed -step\n", 273 | " #print (\"obj \", objp, \" objm \", objm)\n", 274 | " grad_f[i_mpc_input] = (objp - objm) / (2 * step) # This evaluates the gradient of the objetive function with repect to the optimization variable x[i]\n", 275 | " #print(\"Gradient: \", grad_f)\n", 276 | " return array(grad_f)\n" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 6, 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "def eval_g(x):\n", 286 | " '''\n", 287 | " define the constraint function\n", 288 | " '''\n", 289 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 290 | " #### CALCULATE FLUID TEMPERATURE ALONG THE FIRST THREE SURFACES ###########\n", 291 | " \n", 292 | " CAd2=realtime_data[0]\n", 293 | " Td2=realtime_data[1]\n", 294 | "\n", 295 | " g=array([-5.0]*NUM_MPC_CONSTRAINTS) # g is the constraint (inequality) value; Initilize to be negative.\n", 296 | "\n", 297 | " return g\n", 298 | "\n", 299 | "nnzj = NUM_MPC_CONSTRAINTS*NUM_MPC_INPUTS\n", 300 | "\n", 301 | "\n", 302 | "def eval_jac_g(x, flag):\n", 303 | " '''\n", 304 | " define the Jacobian of the constraint function\n", 305 | " '''\n", 306 | " \n", 307 | " if flag:\n", 308 | " list_x = []\n", 309 | " list_y=[]\n", 310 | " for i in range(int(NUM_MPC_INPUTS / 2)):\n", 311 | " list_x = list_x + [i] * NUM_MPC_INPUTS\n", 312 | " list_y = list_y +list(range(0, int(NUM_MPC_INPUTS)))\n", 313 | "\n", 314 | " return (array(list_x),\n", 315 | " array(list_y))\n", 316 | "\n", 317 | "\n", 318 | " else:\n", 319 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 320 | " step = 1e-1 # we just have a small step\n", 321 | " gp=gm=numpy.zeros(NUM_MPC_CONSTRAINTS)\n", 322 | " xpstep=xmstep=numpy.zeros(NUM_MPC_INPUTS)\n", 323 | " jac_g = [[0]*int(NUM_MPC_INPUTS) for _ in range(NUM_MPC_CONSTRAINTS)]\n", 324 | "\n", 325 | " for i_mpc_input in range(NUM_MPC_INPUTS):\n", 326 | " xpstep=x.copy()\n", 327 | " xmstep=x.copy()\n", 328 | " # for each variables, we need to evaluate the derivative of the function with respect to that variable, This is why we have the for loop\n", 329 | " xpstep[i_mpc_input] += step \n", 330 | " xmstep[i_mpc_input] -= step\n", 331 | " gp=eval_g(xpstep)\n", 332 | " gm=eval_g(xmstep)\n", 333 | " for num_constraint in range(NUM_MPC_CONSTRAINTS):\n", 334 | " jac_g[num_constraint][i_mpc_input] = (gp[num_constraint] - gm[num_constraint]) / (2 * step)\n", 335 | " #print (\"in eval_jac_g_2:\")\n", 336 | " return array(jac_g)\n", 337 | "\n", 338 | "def apply_new(x):\n", 339 | " return True\n", 340 | "def print_variable(variable_name, value):\n", 341 | " for i in range(len(value)):\n", 342 | " print(\"{} {}\".format(variable_name + \"[\"+str(i)+\"] =\", value[i]))\n", 343 | "\n", 344 | "\n", 345 | "nnzh = NUM_MPC_INPUTS**2 ## number of nonzeros in the Hessian of the Lagrangian function" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": 7, 351 | "metadata": {}, 352 | "outputs": [ 353 | { 354 | "name": "stdout", 355 | "output_type": "stream", 356 | "text": [ 357 | "g_L [-2.e+19 -2.e+19] [0 0]\n" 358 | ] 359 | } 360 | ], 361 | "source": [ 362 | "nvar = NUM_MPC_INPUTS ## number of variables\n", 363 | "x_lower=[0]* nvar ## lower bound of the variables\n", 364 | "x_upper=[0]* nvar ## upper bound of the variables \n", 365 | "for i in range(int(HORIZON)):\n", 366 | " x_lower[2*i]= -3.5 \n", 367 | " x_lower[2 * i+1] = -5e5\n", 368 | " x_upper[2 * i] = 3.5\n", 369 | " x_upper[2 * i + 1] = 5e5\n", 370 | "x_L = array(x_lower) #array([-3.5,-5e5])\n", 371 | "x_U = array(x_upper) #array([3.5, 5e5])\n", 372 | "\n", 373 | "### DEFINE THE UPPER BOUND AND LOWER BOUND OF THE CONSTRAINT ###\n", 374 | "ncon = NUM_MPC_CONSTRAINTS ## number of constraints\n", 375 | "g_L = array([-2e19]*HORIZON) ## lower bound of the constraints\n", 376 | "g_U = array([0]*HORIZON) ## upper bound of the constraints\n", 377 | "\n", 378 | "print (\"g_L\", g_L, g_U)\n" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": 8, 384 | "metadata": {}, 385 | "outputs": [ 386 | { 387 | "name": "stdout", 388 | "output_type": "stream", 389 | "text": [ 390 | "Num Iteratin: 0\n", 391 | "[PyIPOPT] Ipopt will use Hessian approximation.\n", 392 | "\n", 393 | "[PyIPOPT] Problem created\n", 394 | "Going to call solve\n", 395 | "x0 = [0. 0. 0. 0.]\n", 396 | "\n", 397 | "******************************************************************************\n", 398 | "This program contains Ipopt, a library for large-scale nonlinear optimization.\n", 399 | " Ipopt is released as open source code under the Eclipse Public License (EPL).\n", 400 | " For more information visit https://github.com/coin-or/Ipopt\n", 401 | "******************************************************************************\n", 402 | "\n", 403 | "Solution of the primal variables, x\n", 404 | "x[0] = 3.5000000349985347\n", 405 | "x[1] = -0.8601783386446912\n", 406 | "x[2] = 3.5000000349975773\n", 407 | "x[3] = -0.9449021993056471\n", 408 | "status= -1\n", 409 | "Objective value\n", 410 | "f(x*) = 1251.1258619520822\n", 411 | "Control action=: 3.5000000349985347 -0.8601783386446912\n", 412 | "Real model output x1 x2 in deviation form: -1.3491245181524205 66.01608363831458\n", 413 | "Num Iteratin: 1\n", 414 | "[PyIPOPT] Ipopt will use Hessian approximation.\n", 415 | "\n", 416 | "[PyIPOPT] Problem created\n", 417 | "Going to call solve\n", 418 | "x0 = [ 3.50000003 -0.9449022 3.50000003 -0.9449022 ]\n" 419 | ] 420 | } 421 | ], 422 | "source": [ 423 | "for main_iteration in range(NUM_MPC_ITERATION):\n", 424 | " print (\"Num Iteratin: \", main_iteration)\n", 425 | "\n", 426 | " rawdata = numpy.array([CAi, Ti])\n", 427 | " \n", 428 | " realtime_data=rawdata\n", 429 | "\n", 430 | " nlp = pyipopt.create(nvar, x_L, x_U, ncon, g_L, g_U, nnzj, nnzh, eval_f, eval_grad_f, eval_g, eval_jac_g)\n", 431 | "\n", 432 | " if main_iteration ==0 :\n", 433 | " x0 = array([0.0]*int(NUM_MPC_INPUTS)) # initial guess\n", 434 | " \n", 435 | " else:\n", 436 | " x0=x # use the previous solution as the initial guess\n", 437 | " x0[0:-2]=x[2:] # shift the previous solution to the left by 2\n", 438 | " x0[-2:]=x[-2:] # keep the last two elements unchanged\n", 439 | " x_record=x\n", 440 | "\n", 441 | " \"\"\"\n", 442 | " print x0\n", 443 | " print nvar, ncon, nnzj\n", 444 | " print x_L, x_U\n", 445 | " print g_L, g_U\n", 446 | " print eval_f(x0)\n", 447 | " print eval_grad_f(x0)\n", 448 | " print eval_g(x0)\n", 449 | " a = eval_jac_g(x0, True)\n", 450 | " print \"a = \", a[1], a[0]\n", 451 | " print eval_jac_g(x0, False)\n", 452 | " print eval_h(x0, pi0, 1.0, False)\n", 453 | " print eval_h(x0, pi0, 1.0, True)\n", 454 | " \"\"\"\n", 455 | "\n", 456 | " \"\"\" You CAd2 set Ipopt options by calling nlp.num_option, nlp.str_option\n", 457 | " or nlp.int_option. For instance, to set the tolarance by calling\n", 458 | "\n", 459 | " nlp.num_option('tol', 1e-8)\n", 460 | "\n", 461 | " For a complete list of Ipopt options, refer to\n", 462 | "\n", 463 | " http://www.coin-or.org/Ipopt/documentation/node59.html\n", 464 | "\n", 465 | " Note that Ipopt distinguishs between Int, Num, and Str options, yet sometimes\n", 466 | " does not explicitly tell you which option is which. If you are not sure about\n", 467 | " the option's type, just try it in PyIpopt. If you try to set one type of\n", 468 | " option using the wrong function, Pyipopt will remind you of it. \"\"\"\n", 469 | "\n", 470 | " nlp.int_option('max_iter', 1000) # maximum number of iterations\n", 471 | " nlp.num_option('tol', 1e-5) # convergence tolerance\n", 472 | " nlp.int_option('print_level', 2) # print out the process\n", 473 | " print(\"Going to call solve\") # solve the problem\n", 474 | " print(\"x0 = {}\".format(x0)) # initial guess\n", 475 | " x, zl, zu, constraint_multipliers, obj, status = nlp.solve(x0) \n", 476 | "\n", 477 | " nlp.close()\n", 478 | "\n", 479 | " print(\"Solution of the primal variables, x\")\n", 480 | " print_variable(\"x\", x)\n", 481 | " print (\"status=\", status)\n", 482 | "\n", 483 | " print(\"Objective value\")\n", 484 | " print(\"f(x*) = {}\".format(obj))\n", 485 | " print (\"Control action=: \", x[0], x[1])\n", 486 | "\n", 487 | " x1=CAi \n", 488 | " x2=Ti \n", 489 | "\n", 490 | " # w is the disturbance\n", 491 | " # w1 =numpy.random.normal(0, w1_std, 1)\n", 492 | " # w2 =numpy.random.normal(0, w2_std, 1)\n", 493 | " # if w1>w1_std:\n", 494 | " # w1=w1_std\n", 495 | " # if w1<-w1_std:\n", 496 | " # w1=-w1_std\n", 497 | " # if w2>w2_std:\n", 498 | " # w2=w2_std\n", 499 | " # if w2>w2_std:\n", 500 | " # w2=w2_std\n", 501 | " #print (numpy.asscalar(w1))\n", 502 | " #print (numpy.asscalar(w2))\n", 503 | " for kk in range (int(delta/hc)): # apply the control action for real model\n", 504 | "\n", 505 | "\n", 506 | " x1_new = x1 + hc * ((F / V) * (x[0] - x1) -\n", 507 | " k0 * ((numpy.exp(-E / (R * (x2 + Ts)))*(x1 + CAs) * (x1 + CAs))\n", 508 | " - numpy.exp(-E / (R * Ts)) * CAs * CAs))\n", 509 | "\n", 510 | " x2_new = x2 + hc * (((F / V) * (-x2) + (-Dh / (sigma * cp)) *\n", 511 | " (k0 * ((numpy.exp(-E / (R * (x2 + Ts))) * (x1 + CAs) * (x1 + CAs)) -\n", 512 | " numpy.exp(-E / (R * Ts)) * CAs * CAs)) + (x[1] / (sigma * cp * V))))\n", 513 | "\n", 514 | "\n", 515 | "\n", 516 | " x1 = x1_new\n", 517 | " x2 = x2_new\n", 518 | "\n", 519 | " # if (kk%5==1):\n", 520 | " # x1_record.append(x1)\n", 521 | " # x2_record.append(x2)\n", 522 | " # u1_record.append(x[1])\n", 523 | " # u2_record.append(x[0])\n", 524 | "\n", 525 | " CAi=x1\n", 526 | " Ti=x2\n", 527 | "\n", 528 | "\n", 529 | "\n", 530 | "\n", 531 | " print('Real model output x1 x2 in deviation form: ', x1, x2)\n", 532 | "\n", 533 | " x1_record.append(x1)\n", 534 | " x2_record.append(x2)\n", 535 | " u1_record.append(x[0])\n", 536 | " u2_record.append(x[1])\n", 537 | "\n", 538 | "print (\"x1_record: \",x1_record)\n", 539 | "print (\"x2_record: \",x2_record)\n", 540 | "\n", 541 | "print (\"u1_record: \",u1_record)\n", 542 | "print (\"u2_record: \",u2_record)\n", 543 | "\n", 544 | "# numpy.savetxt(\"x1.txt\", x1_record, fmt=\"%f\", delimiter=\" \")\n", 545 | "# numpy.savetxt(\"x2.txt\", x2_record, fmt=\"%f\", delimiter=\" \")\n", 546 | "\n", 547 | "\n", 548 | "# numpy.savetxt(\"u1.txt\", u1_record, fmt=\"%f\", delimiter=\" \")\n", 549 | "# numpy.savetxt(\"u2.txt\", u2_record, fmt=\"%f\", delimiter=\" \")\n", 550 | "\n" 551 | ] 552 | } 553 | ], 554 | "metadata": { 555 | "kernelspec": { 556 | "display_name": "Python 3", 557 | "language": "python", 558 | "name": "python3" 559 | }, 560 | "language_info": { 561 | "codemirror_mode": { 562 | "name": "ipython", 563 | "version": 3 564 | }, 565 | "file_extension": ".py", 566 | "mimetype": "text/x-python", 567 | "name": "python", 568 | "nbconvert_exporter": "python", 569 | "pygments_lexer": "ipython3", 570 | "version": "3.8.10" 571 | } 572 | }, 573 | "nbformat": 4, 574 | "nbformat_minor": 2 575 | } 576 | -------------------------------------------------------------------------------- /LMPC_RNN_CSTR.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "2024-03-04 14:49:11.474911: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA\n", 13 | "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", 14 | "2024-03-04 14:49:11.603091: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", 15 | "2024-03-04 14:49:11.606926: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n", 16 | "2024-03-04 14:49:11.606943: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n", 17 | "2024-03-04 14:49:12.131029: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", 18 | "2024-03-04 14:49:12.131060: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", 19 | "2024-03-04 14:49:12.131064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n", 20 | "/home/wulab/.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:20: UserWarning: Pandas requires version '2.7.3' or newer of 'numexpr' (version '2.7.1' currently installed).\n", 21 | " from pandas.core.computation.check import NUMEXPR_INSTALLED\n" 22 | ] 23 | } 24 | ], 25 | "source": [ 26 | "from __future__ import print_function\n", 27 | "from tensorflow.keras.models import model_from_json, load_model\n", 28 | "import numpy\n", 29 | "import numpy as np\n", 30 | "import time\n", 31 | "import os\n", 32 | "import math\n", 33 | "import pyipopt\n", 34 | "from numpy import *\n", 35 | "import tensorflow as tf\n", 36 | "import matplotlib.pyplot as plt" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "\n", 46 | "################################### Simulation time step ##################################\n", 47 | "delta = 0.01 # sampling time\n", 48 | "hc = 1e-4 # integration time step\n", 49 | "oper_time = 0.01 \n", 50 | "\n", 51 | "################################## Initial states #########################################\n", 52 | "CAi=-1.65\n", 53 | "Ti=72\n", 54 | "x1_nn=CAi\n", 55 | "x2_nn=Ti\n", 56 | "x1_record=[CAi]\n", 57 | "x2_record=[Ti]\n", 58 | "u1_record=[]\n", 59 | "u2_record=[]\n", 60 | "\n", 61 | "##################################### Constants ###########################################\n", 62 | "a=1060\n", 63 | "b=22\n", 64 | "d=0.52 # Lyapunov function V=x^T*P*x\n", 65 | "\n", 66 | "# CSTR PARAMETERS\n", 67 | "F=5\n", 68 | "V=1\n", 69 | "k0=8460000\n", 70 | "E=50000 # parametric drift #####E has the most effect on process dynamics######\n", 71 | "R=8.314\n", 72 | "T0=300\n", 73 | "Dh=-11500\n", 74 | "rho=1000\n", 75 | "sigma=1000\n", 76 | "cp=0.231\n", 77 | "Qs=0\n", 78 | "CA0s=4\n", 79 | "w1_std=2.5 #disturbance std\n", 80 | "w2_std=70 #disturbance std\n", 81 | "\n", 82 | "\n", 83 | "########################################### steady-state ###################################\n", 84 | "CAs= 1.9537\n", 85 | "Ts= 401.8727\n", 86 | "\n", 87 | "# state_ss=numpy.array([Ts, CAs])\n", 88 | "# input_ss=numpy.array([Qs, CA0s])\n", 89 | "state_ss=numpy.array([CAs,Ts])\n", 90 | "input_ss=numpy.array([CA0s,Qs])\n", 91 | "\n", 92 | "ROOT_FOLDER=os.getcwd()\n", 93 | "\n", 94 | "##################################### MPC Parameters ######################################\n", 95 | "\n", 96 | "NUM_MPC_ITERATION=10 # MPC TOTAL ITERATION\n", 97 | "\n", 98 | "\n", 99 | "NUM_OUTPUTS = 2 # Number of state variables (RNN output) \n", 100 | "NUM_INPUTS = 4 # Number of RNN input\n", 101 | "HORIZON = 2 ## MPC PREDICTION HORIZON (Depends on how many delta you want to predict with the RNN)\n", 102 | "\n", 103 | " \n", 104 | "NUM_IN_SEQUENCE = 10 # Number of integration time steps actually used in MPC (equal to timestep of ML model)\n", 105 | "\n", 106 | "NUM_MPC_INPUTS = 2*HORIZON # Number of MPC inputs to be optimized, 2 is the number of control inputs \n", 107 | "NUM_MPC_CONSTRAINTS = HORIZON # For each sampling time within prediction horizon, we have 1 constraint\n", 108 | "\n", 109 | "realtime_data = None\n", 110 | "\n", 111 | "setpoint=[0, 0]\n" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 3, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stderr", 121 | "output_type": "stream", 122 | "text": [ 123 | "2024-03-04 14:49:12.896807: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 124 | "2024-03-04 14:49:12.897072: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n", 125 | "2024-03-04 14:49:12.897116: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory\n", 126 | "2024-03-04 14:49:12.897146: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory\n", 127 | "2024-03-04 14:49:12.899139: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory\n", 128 | "2024-03-04 14:49:12.899191: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory\n", 129 | "2024-03-04 14:49:12.899214: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory\n", 130 | "2024-03-04 14:49:12.899222: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n", 131 | "Skipping registering GPU devices...\n", 132 | "2024-03-04 14:49:12.899492: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA\n", 133 | "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" 134 | ] 135 | }, 136 | { 137 | "name": "stdout", 138 | "output_type": "stream", 139 | "text": [ 140 | "\n" 141 | ] 142 | } 143 | ], 144 | "source": [ 145 | "# define scalers for both X and y\n", 146 | "X_mean = np.load('X_mean.npy')\n", 147 | "X_std = np.load('X_std.npy')\n", 148 | "y_mean = np.load('y_mean.npy')\n", 149 | "y_std = np.load('y_std.npy')\n", 150 | "\n", 151 | "model = load_model('model.h5')\n", 152 | "print(model)\n", 153 | "\n", 154 | "x1_mean= X_mean[0] # CA\n", 155 | "x1_std= X_std[0]\n", 156 | "x2_mean=X_mean[1] # T\n", 157 | "x2_std= X_std[1]\n", 158 | "u1_mean= X_mean[2] # CA0\n", 159 | "u1_std=X_std[2]\n", 160 | "u2_mean=X_mean[3] # Q\n", 161 | "u2_std=X_std[3]\n", 162 | "y1_mean=y_mean[0] # CA\n", 163 | "y1_std=y_std[0]\n", 164 | "y2_mean=y_mean[1] # T\n", 165 | "y2_std=y_std[1]\n", 166 | "\n", 167 | "state_mean = np.array([x1_mean, x2_mean]) # [CA_input, T_input]\n", 168 | "state_std = np.array([x1_std, x2_std])\n", 169 | "\n", 170 | "input_mean = np.array([u1_mean, u2_mean]) # [CA0, Q]\n", 171 | "input_std = np.array([u1_std, u2_std])\n", 172 | "\n", 173 | "output_mean = np.array([y1_mean, y2_mean]) # [CA_output, T_output]\n", 174 | "output_std = np.array([y1_std, y2_std])" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 4, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "def my_ens_prediction(num_horizon,my_rawdata,my_inputs):\n", 184 | " xx = [] \n", 185 | " nn_inputs = [] # NN input\n", 186 | " ensemble_output = numpy.zeros((num_horizon,NUM_OUTPUTS,NUM_IN_SEQUENCE))\n", 187 | " \n", 188 | " ensemble_output = ensemble_output.reshape(num_horizon,NUM_IN_SEQUENCE,NUM_OUTPUTS)\n", 189 | " x_test2 = my_rawdata[0:NUM_OUTPUTS].astype(float)\n", 190 | " x_test2= (x_test2-state_mean)/state_std # my_rawdata is normal value; needs to normalize before feeding into NN\n", 191 | "\n", 192 | " predict_output_normal=[[0 for i in range(NUM_OUTPUTS)] for j in range(NUM_IN_SEQUENCE)]\n", 193 | "\n", 194 | " for i_model in range(num_horizon): \n", 195 | " \n", 196 | " # ############################# get the normalized input for RNN #########################\n", 197 | " \n", 198 | " my_inputs_normalized = (my_inputs[2*i_model:2*(i_model+1)] - input_mean) / input_std # my_inputs is also normal value; needs to normalize before feeding into NN\n", 199 | " xx = numpy.concatenate((x_test2, my_inputs_normalized), axis=None).reshape((1, NUM_INPUTS)) # xx is the NN input\n", 200 | " xx = numpy.tile(xx, (NUM_IN_SEQUENCE, 1)) # duplicate #NUM_IN_SEQUENCE times\n", 201 | "\n", 202 | " nn_inputs = xx.reshape(1, NUM_IN_SEQUENCE, NUM_INPUTS) \n", 203 | " # ############## use machine learning model to predict the next sampling time ##############\n", 204 | " \n", 205 | " predict_output = model(nn_inputs).numpy()\n", 206 | " predict_output = predict_output.reshape(NUM_IN_SEQUENCE, NUM_OUTPUTS)\n", 207 | " predict_output = predict_output * output_std + output_mean # convert back to deviation form\n", 208 | "\n", 209 | " ####################### get the system state at the end of sampling time ################\n", 210 | " \n", 211 | " x_test2 = predict_output[-1, :]\n", 212 | "\n", 213 | "\n", 214 | " ###### transform the predicted states back to their original values in deviation form ######\n", 215 | " \n", 216 | " x_test2 = (x_test2 - state_mean) / state_std # normalize input for next prediction\n", 217 | "\n", 218 | " ensemble_output[i_model,:,:] = predict_output[:, :] # store the predicted states \n", 219 | "\n", 220 | " ########################################################################################\n", 221 | "\n", 222 | "\n", 223 | " return ensemble_output \n" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 5, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "def eval_f(x):\n", 233 | " '''\n", 234 | " define the objective function\n", 235 | " '''\n", 236 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 237 | " offset=0\n", 238 | " \n", 239 | " #### CALCULATE OUTLET CONC ###########\n", 240 | " df_ensemble_output = my_ens_prediction(num_horizon = HORIZON, my_rawdata=realtime_data, my_inputs=x)\n", 241 | "\n", 242 | " #### account for all intermediate steps ####\n", 243 | " for j in range (HORIZON):\n", 244 | " est_outlet_product = df_ensemble_output[j, :, 0:2]\n", 245 | " for i in range (int(NUM_IN_SEQUENCE)): \n", 246 | " \n", 247 | " offset = offset + (setpoint[0] - (est_outlet_product[i, 1])) ** 2.0 + (setpoint[1] - (est_outlet_product[i, 0])) ** 2.0 * 1000\n", 248 | "\n", 249 | "\n", 250 | "\n", 251 | " return offset/100\n", 252 | "\n", 253 | "def eval_grad_f(x):\n", 254 | " '''\n", 255 | " define the gradient of the objective function\n", 256 | " '''\n", 257 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 258 | " step = 1e-1 # we just have a small step\n", 259 | " objp=objm=0\n", 260 | " grad_f = [0]*NUM_MPC_INPUTS\n", 261 | " xpstep = [0]*NUM_MPC_INPUTS\n", 262 | " xmstep = [0]*NUM_MPC_INPUTS\n", 263 | " for i_mpc_input in range(NUM_MPC_INPUTS):\n", 264 | " xpstep=x.copy()\n", 265 | " xmstep=x.copy()\n", 266 | " # for each variables, we need to evaluate the derivative of the function with respect to that variable, This is why we have the for loop\n", 267 | " xpstep[i_mpc_input] = xpstep[i_mpc_input]+step \n", 268 | " xmstep[i_mpc_input] = xmstep[i_mpc_input]-step\n", 269 | "\n", 270 | " # Evaluate the objective function at xpstep and xmstep\n", 271 | " objp=eval_f(xpstep) # This function returns the value of the objective function evaluated with the variable x[i] is perturebed +step\n", 272 | " objm=eval_f(xmstep) # This function returns the value of the objective function evaluated with the variable x[i] is perturebed -step\n", 273 | " #print (\"obj \", objp, \" objm \", objm)\n", 274 | " grad_f[i_mpc_input] = (objp - objm) / (2 * step) # This evaluates the gradient of the objetive function with repect to the optimization variable x[i]\n", 275 | " #print(\"Gradient: \", grad_f)\n", 276 | " return array(grad_f)\n" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 6, 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "def eval_g(x):\n", 286 | " '''\n", 287 | " define the constraint function\n", 288 | " '''\n", 289 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 290 | "\n", 291 | " CAd2=realtime_data[0] ## current CA\n", 292 | " Td2=realtime_data[1] ## current T\n", 293 | " g=array([-5.0]*NUM_MPC_CONSTRAINTS) # g is the constraint (inequality) value; Initilize to be negative.\n", 294 | "\n", 295 | "\n", 296 | " if ((a*CAd2**2+d*Td2**2+2*b*CAd2*Td2-2)> 0): # (If V > \\rho_min=2)\n", 297 | " \n", 298 | " df_ensemble_output3 = my_ens_prediction(num_horizon=1,my_rawdata=realtime_data, my_inputs=x)\n", 299 | " \n", 300 | " est_outlet = df_ensemble_output3[-1, -1, 0:2].reshape(1,2)\n", 301 | " \n", 302 | " # calculate the derivative of the Lyapunov function V(x,u) with respect to x\n", 303 | " dot_V1=(2*a * CAd2 + 2*b * Td2)*((est_outlet[0][0])-CAd2)/(0.01)+\\\n", 304 | " (2*d*Td2 + 2*b * CAd2)*((est_outlet[0][1])-Td2)/(0.01) \n", 305 | " \n", 306 | "\n", 307 | " '''\n", 308 | " This corresponds to the constraint: dot_V (x,u) < dot_V(x, \\phi(x))\n", 309 | " To simplify calculation, we use the constraint, dot_V(x,u) < -15 V to make sure dot_V is negative.\n", 310 | " '''\n", 311 | " vv=a*CAd2**2+d*Td2**2+2*b*CAd2*Td2 # calculate the derivative of the Lyapunov function V(x,\\phi(x))\n", 312 | " g[0]=dot_V1+15*abs(vv/100) \n", 313 | "\n", 314 | "\n", 315 | " else:\n", 316 | " df_ensemble_output2 = my_ens_prediction(num_horizon=int(NUM_MPC_INPUTS / 2), my_rawdata=realtime_data,\n", 317 | " my_inputs=x)\n", 318 | " \n", 319 | " ##### only account for the last point #####\n", 320 | " for j in range(int(NUM_MPC_INPUTS / 2)):\n", 321 | " est_outlet_product2 = df_ensemble_output2[j, -1, 0:2] # only account for the last point\n", 322 | " \n", 323 | " g[j]= d * (est_outlet_product2[1]) ** 2+ 2 * b * (est_outlet_product2[1])*(est_outlet_product2[0]) + \\\n", 324 | " a*(est_outlet_product2[0]) ** 2 -2\n", 325 | " # this corresponds to the constraint: V(x,u) < 2 (\\rho_min or \\rho_nn in different papers)\n", 326 | "\n", 327 | " return g\n", 328 | "\n", 329 | "nnzj = NUM_MPC_CONSTRAINTS*NUM_MPC_INPUTS ## number of nonzeros in the Jacobian of the constraint function\n", 330 | "\n", 331 | "\n", 332 | "def eval_jac_g(x, flag):\n", 333 | " '''\n", 334 | " define the Jacobian of the constraint function\n", 335 | " '''\n", 336 | " \n", 337 | " if flag:\n", 338 | " list_x = []\n", 339 | " list_y=[]\n", 340 | " for i in range(int(NUM_MPC_INPUTS / 2)):\n", 341 | " list_x = list_x + [i] * NUM_MPC_INPUTS\n", 342 | " list_y = list_y +list(range(0, int(NUM_MPC_INPUTS)))\n", 343 | "\n", 344 | " return (array(list_x),\n", 345 | " array(list_y))\n", 346 | "\n", 347 | "\n", 348 | " else:\n", 349 | " assert len(x) == int(NUM_MPC_INPUTS)\n", 350 | " step = 1e-1 # we just have a small step\n", 351 | " gp=gm=numpy.zeros(NUM_MPC_CONSTRAINTS)\n", 352 | " xpstep=xmstep=numpy.zeros(NUM_MPC_INPUTS)\n", 353 | " jac_g = [[0]*int(NUM_MPC_INPUTS) for _ in range(NUM_MPC_CONSTRAINTS)]\n", 354 | "\n", 355 | " for i_mpc_input in range(NUM_MPC_INPUTS):\n", 356 | " xpstep=x.copy()\n", 357 | " xmstep=x.copy()\n", 358 | " # for each variables, we need to evaluate the derivative of the function with respect to that variable, This is why we have the for loop\n", 359 | " xpstep[i_mpc_input] += step \n", 360 | " xmstep[i_mpc_input] -= step\n", 361 | " gp=eval_g(xpstep)\n", 362 | " gm=eval_g(xmstep)\n", 363 | " for num_constraint in range(NUM_MPC_CONSTRAINTS):\n", 364 | " jac_g[num_constraint][i_mpc_input] = (gp[num_constraint] - gm[num_constraint]) / (2 * step)\n", 365 | " #print (\"in eval_jac_g_2:\")\n", 366 | " return array(jac_g)\n", 367 | "\n", 368 | "def apply_new(x):\n", 369 | " return True\n", 370 | "def print_variable(variable_name, value):\n", 371 | " for i in range(len(value)):\n", 372 | " print(\"{} {}\".format(variable_name + \"[\"+str(i)+\"] =\", value[i]))\n", 373 | "\n", 374 | "\n", 375 | "nnzh = NUM_MPC_INPUTS**2 ## number of nonzeros in the Hessian of the Lagrangian function" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 7, 381 | "metadata": {}, 382 | "outputs": [ 383 | { 384 | "name": "stdout", 385 | "output_type": "stream", 386 | "text": [ 387 | "g_L [-2.e+19 -2.e+19] [0 0]\n" 388 | ] 389 | } 390 | ], 391 | "source": [ 392 | "nvar = NUM_MPC_INPUTS ## number of variables\n", 393 | "x_lower=[0]* nvar ## lower bound of the variables\n", 394 | "x_upper=[0]* nvar ## upper bound of the variables \n", 395 | "for i in range(int(HORIZON)):\n", 396 | " x_lower[2*i]= -3.5 \n", 397 | " x_lower[2 * i+1] = -5e5\n", 398 | " x_upper[2 * i] = 3.5\n", 399 | " x_upper[2 * i + 1] = 5e5\n", 400 | "x_L = array(x_lower) #array([-3.5,-5e5])\n", 401 | "x_U = array(x_upper) #array([3.5, 5e5])\n", 402 | "\n", 403 | "### DEFINE THE UPPER BOUND AND LOWER BOUND OF THE CONSTRAINT ###\n", 404 | "ncon = NUM_MPC_CONSTRAINTS ## number of constraints\n", 405 | "g_L = array([-2e19]*HORIZON) ## lower bound of the constraints\n", 406 | "g_U = array([0]*HORIZON) ## upper bound of the constraints\n", 407 | "\n", 408 | "print (\"g_L\", g_L, g_U)\n" 409 | ] 410 | }, 411 | { 412 | "cell_type": "code", 413 | "execution_count": 8, 414 | "metadata": {}, 415 | "outputs": [ 416 | { 417 | "name": "stdout", 418 | "output_type": "stream", 419 | "text": [ 420 | "Num Iteratin: 0\n", 421 | "[PyIPOPT] Ipopt will use Hessian approximation.\n", 422 | "\n", 423 | "[PyIPOPT] Problem created\n", 424 | "Going to call solve\n", 425 | "x0 = [0. 0. 0. 0.]\n", 426 | "\n", 427 | "******************************************************************************\n", 428 | "This program contains Ipopt, a library for large-scale nonlinear optimization.\n", 429 | " Ipopt is released as open source code under the Eclipse Public License (EPL).\n", 430 | " For more information visit https://github.com/coin-or/Ipopt\n", 431 | "******************************************************************************\n", 432 | "\n", 433 | "Solution of the primal variables, x\n", 434 | "x[0] = 3.5000000349985347\n", 435 | "x[1] = -0.8601783386446912\n", 436 | "x[2] = 3.5000000349975773\n", 437 | "x[3] = -0.9449021993056471\n", 438 | "status= -1\n", 439 | "Objective value\n", 440 | "f(x*) = 1251.1258619520822\n", 441 | "Control action=: 3.5000000349985347 -0.8601783386446912\n", 442 | "Real model output x1 x2 in deviation form: -1.3491245181524205 66.01608363831458\n", 443 | "Num Iteratin: 1\n", 444 | "[PyIPOPT] Ipopt will use Hessian approximation.\n", 445 | "\n", 446 | "[PyIPOPT] Problem created\n", 447 | "Going to call solve\n", 448 | "x0 = [ 3.50000003 -0.9449022 3.50000003 -0.9449022 ]\n" 449 | ] 450 | } 451 | ], 452 | "source": [ 453 | "for main_iteration in range(NUM_MPC_ITERATION):\n", 454 | " print (\"Num Iteratin: \", main_iteration)\n", 455 | "\n", 456 | " rawdata = numpy.array([CAi, Ti])\n", 457 | " \n", 458 | " realtime_data=rawdata\n", 459 | "\n", 460 | " nlp = pyipopt.create(nvar, x_L, x_U, ncon, g_L, g_U, nnzj, nnzh, eval_f, eval_grad_f, eval_g, eval_jac_g)\n", 461 | "\n", 462 | " if main_iteration ==0 :\n", 463 | " x0 = array([0.0]*int(NUM_MPC_INPUTS)) # initial guess\n", 464 | " \n", 465 | " else:\n", 466 | " x0=x # use the previous solution as the initial guess\n", 467 | " x0[0:-2]=x[2:] # shift the previous solution to the left by 2\n", 468 | " x0[-2:]=x[-2:] # keep the last two elements unchanged\n", 469 | " x_record=x\n", 470 | "\n", 471 | " \"\"\"\n", 472 | " print x0\n", 473 | " print nvar, ncon, nnzj\n", 474 | " print x_L, x_U\n", 475 | " print g_L, g_U\n", 476 | " print eval_f(x0)\n", 477 | " print eval_grad_f(x0)\n", 478 | " print eval_g(x0)\n", 479 | " a = eval_jac_g(x0, True)\n", 480 | " print \"a = \", a[1], a[0]\n", 481 | " print eval_jac_g(x0, False)\n", 482 | " print eval_h(x0, pi0, 1.0, False)\n", 483 | " print eval_h(x0, pi0, 1.0, True)\n", 484 | " \"\"\"\n", 485 | "\n", 486 | " \"\"\" You CAd2 set Ipopt options by calling nlp.num_option, nlp.str_option\n", 487 | " or nlp.int_option. For instance, to set the tolarance by calling\n", 488 | "\n", 489 | " nlp.num_option('tol', 1e-8)\n", 490 | "\n", 491 | " For a complete list of Ipopt options, refer to\n", 492 | "\n", 493 | " http://www.coin-or.org/Ipopt/documentation/node59.html\n", 494 | "\n", 495 | " Note that Ipopt distinguishs between Int, Num, and Str options, yet sometimes\n", 496 | " does not explicitly tell you which option is which. If you are not sure about\n", 497 | " the option's type, just try it in PyIpopt. If you try to set one type of\n", 498 | " option using the wrong function, Pyipopt will remind you of it. \"\"\"\n", 499 | "\n", 500 | " nlp.int_option('max_iter', 1000) # maximum number of iterations\n", 501 | " nlp.num_option('tol', 1e-5) # convergence tolerance\n", 502 | " nlp.int_option('print_level', 2) # print out the process\n", 503 | " print(\"Going to call solve\") # solve the problem\n", 504 | " print(\"x0 = {}\".format(x0)) # initial guess\n", 505 | " x, zl, zu, constraint_multipliers, obj, status = nlp.solve(x0) \n", 506 | "\n", 507 | " nlp.close()\n", 508 | "\n", 509 | " print(\"Solution of the primal variables, x\")\n", 510 | " print_variable(\"x\", x)\n", 511 | " print (\"status=\", status)\n", 512 | "\n", 513 | " print(\"Objective value\")\n", 514 | " print(\"f(x*) = {}\".format(obj))\n", 515 | " print (\"Control action=: \", x[0], x[1])\n", 516 | "\n", 517 | " x1=CAi \n", 518 | " x2=Ti \n", 519 | "\n", 520 | " # w is the disturbance\n", 521 | " # w1 =numpy.random.normal(0, w1_std, 1)\n", 522 | " # w2 =numpy.random.normal(0, w2_std, 1)\n", 523 | " # if w1>w1_std:\n", 524 | " # w1=w1_std\n", 525 | " # if w1<-w1_std:\n", 526 | " # w1=-w1_std\n", 527 | " # if w2>w2_std:\n", 528 | " # w2=w2_std\n", 529 | " # if w2>w2_std:\n", 530 | " # w2=w2_std\n", 531 | " #print (numpy.asscalar(w1))\n", 532 | " #print (numpy.asscalar(w2))\n", 533 | " for kk in range (int(delta/hc)): # apply the control action for real model\n", 534 | "\n", 535 | "\n", 536 | " x1_new = x1 + hc * ((F / V) * (x[0] - x1) -\n", 537 | " k0 * ((numpy.exp(-E / (R * (x2 + Ts)))*(x1 + CAs) * (x1 + CAs))\n", 538 | " - numpy.exp(-E / (R * Ts)) * CAs * CAs))\n", 539 | "\n", 540 | " x2_new = x2 + hc * (((F / V) * (-x2) + (-Dh / (sigma * cp)) *\n", 541 | " (k0 * ((numpy.exp(-E / (R * (x2 + Ts))) * (x1 + CAs) * (x1 + CAs)) -\n", 542 | " numpy.exp(-E / (R * Ts)) * CAs * CAs)) + (x[1] / (sigma * cp * V))))\n", 543 | "\n", 544 | "\n", 545 | "\n", 546 | " x1 = x1_new\n", 547 | " x2 = x2_new\n", 548 | "\n", 549 | " # if (kk%5==1):\n", 550 | " # x1_record.append(x1)\n", 551 | " # x2_record.append(x2)\n", 552 | " # u1_record.append(x[1])\n", 553 | " # u2_record.append(x[0])\n", 554 | "\n", 555 | " CAi=x1\n", 556 | " Ti=x2\n", 557 | "\n", 558 | "\n", 559 | "\n", 560 | "\n", 561 | " print('Real model output x1 x2 in deviation form: ', x1, x2)\n", 562 | "\n", 563 | " x1_record.append(x1)\n", 564 | " x2_record.append(x2)\n", 565 | " u1_record.append(x[0])\n", 566 | " u2_record.append(x[1])\n", 567 | "\n", 568 | "print (\"x1_record: \",x1_record)\n", 569 | "print (\"x2_record: \",x2_record)\n", 570 | "\n", 571 | "print (\"u1_record: \",u1_record)\n", 572 | "print (\"u2_record: \",u2_record)\n", 573 | "\n", 574 | "# numpy.savetxt(\"x1.txt\", x1_record, fmt=\"%f\", delimiter=\" \")\n", 575 | "# numpy.savetxt(\"x2.txt\", x2_record, fmt=\"%f\", delimiter=\" \")\n", 576 | "\n", 577 | "\n", 578 | "# numpy.savetxt(\"u1.txt\", u1_record, fmt=\"%f\", delimiter=\" \")\n", 579 | "# numpy.savetxt(\"u2.txt\", u2_record, fmt=\"%f\", delimiter=\" \")\n", 580 | "\n" 581 | ] 582 | } 583 | ], 584 | "metadata": { 585 | "kernelspec": { 586 | "display_name": "Python 3", 587 | "language": "python", 588 | "name": "python3" 589 | }, 590 | "language_info": { 591 | "codemirror_mode": { 592 | "name": "ipython", 593 | "version": 3 594 | }, 595 | "file_extension": ".py", 596 | "mimetype": "text/x-python", 597 | "name": "python", 598 | "nbconvert_exporter": "python", 599 | "pygments_lexer": "ipython3", 600 | "version": "3.8.10" 601 | } 602 | }, 603 | "nbformat": 4, 604 | "nbformat_minor": 2 605 | } 606 | -------------------------------------------------------------------------------- /RNN_CSTR.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "2024-02-21 10:26:35.417428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "# Import all necessary packages\n", 18 | "\n", 19 | "import numpy as np\n", 20 | "import matplotlib.pyplot as plt\n", 21 | "from sklearn import preprocessing\n", 22 | "from sklearn.model_selection import train_test_split\n", 23 | "from sklearn.metrics import mean_absolute_percentage_error\n", 24 | "from tensorflow.keras.models import Sequential\n", 25 | "from tensorflow.keras.layers import Dense, SimpleRNN, Input, Activation, Dropout, LSTM\n", 26 | "from tensorflow.keras import backend as K\n", 27 | "from tensorflow.keras.optimizers import Adam,SGD\n", 28 | "import tensorflow as tf\n", 29 | "from tensorflow.keras.models import Model,load_model" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 2, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "# specifying constant parameters\n", 39 | "\n", 40 | "T_0 = 300\n", 41 | "V = 1\n", 42 | "k_0 = 8.46*(np.power(10,6))\n", 43 | "C_p = 0.231\n", 44 | "rho_L = 1000\n", 45 | "Q_s = 0.0\n", 46 | "T_s = 402\n", 47 | "F = 5\n", 48 | "E = 5*(np.power(10,4))\n", 49 | "delta_H = -1.15*(np.power(10,4))\n", 50 | "R = 8.314\n", 51 | "C_A0s = 4\n", 52 | "C_As = 1.95\n", 53 | "t_final = 0.01\n", 54 | "t_step = 1e-4\n", 55 | "P = np.array([[1060, 22], [22, 0.52]])" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 3, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "# generating inputs and initial states for CSTR, all expressed in deviation form\n", 65 | "\n", 66 | "u1_list = np.linspace(-3.5, 3.5, 30, endpoint=True)\n", 67 | "u2_list = np.linspace(-5e5, 5e5, 30, endpoint=True)\n", 68 | "T_initial = np.linspace(300, 600, 50, endpoint=True) - T_s\n", 69 | "CA_initial = np.linspace(0, 6, 50, endpoint=True) - C_As" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "outputs": [ 77 | { 78 | "name": "stdout", 79 | "output_type": "stream", 80 | "text": [ 81 | "number of initial conditions: 190\n", 82 | "shape of x_deviation is (190, 2)\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "# sieve out initial states that lie outside of stability region\n", 88 | "\n", 89 | "T_start = list()\n", 90 | "CA_start = list()\n", 91 | "\n", 92 | "for T in T_initial:\n", 93 | " for CA in CA_initial:\n", 94 | " x = np.array([CA, T])\n", 95 | " if x @ P @ x < 372:\n", 96 | " CA_start.append(CA)\n", 97 | " T_start.append(T)\n", 98 | "print(\"number of initial conditions: {}\".format(len(CA_start)))\n", 99 | "\n", 100 | "# convert to np.arrays\n", 101 | "CA_start = np.array([CA_start])\n", 102 | "T_start = np.array([T_start])\n", 103 | "x_deviation = np.concatenate((CA_start.T, T_start.T), axis=1) # every row is a pair of initial states within stability region\n", 104 | "print(\"shape of x_deviation is {}\".format(x_deviation.shape))" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 5, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "# Open-loop simulations of the first-principles model of CSTR\n", 114 | "\n", 115 | "def CSTR_simulation(F, V, C_A0, k_0, E, R, T_0, delta_H, rho_L, C_p, Q, t_final, t_step, C_A_initial, T_initial):\n", 116 | " \"\"\"\n", 117 | " simulating CSTR using forward Euler method\n", 118 | " \"\"\"\n", 119 | " \n", 120 | " C_A_list = list() # evolution of CA over time\n", 121 | " T_list = list() # evolution of T over time\n", 122 | " \n", 123 | " C_A = C_A_initial + C_As\n", 124 | " T = T_initial + T_s\n", 125 | " \n", 126 | " for i in range(int(t_final / t_step)):\n", 127 | " dCAdt = F / V * (C_A0 - C_A) - k_0 * np.exp(-E / (R * T)) * C_A**2\n", 128 | " dTdt = F / V * (T_0 - T) - delta_H / (rho_L * C_p) * k_0 * np.exp(-E / (R * T)) * C_A**2 + Q / (rho_L * C_p * V)\n", 129 | " \n", 130 | " C_A += dCAdt * t_step\n", 131 | " T += dTdt * t_step\n", 132 | "\n", 133 | " if (i+1)% 10 == 0:\n", 134 | " C_A_list.append(C_A - C_As) # in deviation form\n", 135 | " T_list.append(T - T_s) # in deviation form \n", 136 | " \n", 137 | " \n", 138 | " return C_A_list, T_list" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 6, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "# get X and y data for training and testing\n", 148 | "\n", 149 | "CA_output = list()\n", 150 | "T_output = list()\n", 151 | "CA_input = list()\n", 152 | "T_input = list()\n", 153 | "CA0_input = list()\n", 154 | "Q_input = list()\n", 155 | "\n", 156 | "for u1 in u1_list:\n", 157 | " C_A0 = u1 + C_A0s\n", 158 | " \n", 159 | " for u2 in u2_list:\n", 160 | " Q = u2 + Q_s\n", 161 | " \n", 162 | " for C_A_initial, T_initial in x_deviation:\n", 163 | " CA0_input.append(u1)\n", 164 | " Q_input.append(u2)\n", 165 | " CA_input.append(C_A_initial)\n", 166 | " T_input.append(T_initial)\n", 167 | " \n", 168 | " C_A_list, T_list = \\\n", 169 | " CSTR_simulation(F, V, C_A0, k_0, E, R, T_0, delta_H, rho_L, C_p, Q, t_final, t_step, C_A_initial, T_initial)\n", 170 | " CA_output.append(C_A_list)\n", 171 | " T_output.append(T_list)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 7, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "name": "stdout", 181 | "output_type": "stream", 182 | "text": [ 183 | "RNN_input shape is (171000, 10, 4)\n", 184 | "[ 1.35612245e+00 -7.13877551e+01 -3.50000000e+00 -5.00000000e+05]\n", 185 | "[ 1.35612245e+00 -7.13877551e+01 -3.50000000e+00 -5.00000000e+05]\n" 186 | ] 187 | } 188 | ], 189 | "source": [ 190 | "# collate input for RNN\n", 191 | "\n", 192 | "CA0_input = np.array(CA0_input)\n", 193 | "CA0_input = CA0_input.reshape(-1,1,1)\n", 194 | "\n", 195 | "Q_input = np.array(Q_input)\n", 196 | "Q_input = Q_input.reshape(-1,1,1)\n", 197 | "\n", 198 | "CA_input = np.array(CA_input)\n", 199 | "CA_input = CA_input.reshape(-1,1,1)\n", 200 | "\n", 201 | "T_input = np.array(T_input)\n", 202 | "T_input = T_input.reshape(-1,1,1)\n", 203 | "\n", 204 | "RNN_input = np.concatenate((CA_input, T_input, CA0_input, Q_input), axis=2)\n", 205 | "\n", 206 | "\"\"\"\n", 207 | " the input to RNN is in the shape [number of samples x timestep x variables], and the input variables are same for every\n", 208 | " time step, not sure if my treatment here is correct\n", 209 | "\"\"\"\n", 210 | "RNN_input = RNN_input.repeat(10, axis=1) # 10 time steps in this example\n", 211 | "print(\"RNN_input shape is {}\".format(RNN_input.shape))\n", 212 | "\n", 213 | "# checking the input is duplicated 10 times for each time step\n", 214 | "print(RNN_input[0, 0])\n", 215 | "print(RNN_input[0, 1])" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": 8, 221 | "metadata": {}, 222 | "outputs": [ 223 | { 224 | "name": "stdout", 225 | "output_type": "stream", 226 | "text": [ 227 | "RNN_output shape is (171000, 10, 2)\n", 228 | "RNN_output shape is (171000, 10, 2)\n" 229 | ] 230 | } 231 | ], 232 | "source": [ 233 | "# collate output for RNN\n", 234 | "\n", 235 | "CA_output = np.array(CA_output)\n", 236 | "CA_output = CA_output.reshape(-1, 10, 1)\n", 237 | "\n", 238 | "T_output = np.array(T_output)\n", 239 | "T_output = T_output.reshape(-1, 10, 1)\n", 240 | "\n", 241 | "RNN_output = np.concatenate((CA_output, T_output), axis=2)\n", 242 | "print(\"RNN_output shape is {}\".format(RNN_output.shape)) # output shape: number of samples x timestep x variables\n", 243 | "\n", 244 | "# checking output\n", 245 | "print('RNN_output shape is',RNN_output.shape)" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": 9, 251 | "metadata": {}, 252 | "outputs": [ 253 | { 254 | "name": "stdout", 255 | "output_type": "stream", 256 | "text": [ 257 | "[ 7.25026853e-03 -3.35123523e-01 -5.10593517e-17 0.00000000e+00]\n", 258 | "[7.01467140e-01 1.41521192e+03 4.36494253e+00 8.90804598e+10]\n", 259 | "[ 0.01304949 -0.62300591]\n", 260 | "[6.86987919e-01 1.48200017e+03]\n", 261 | "[-0.72869851 1.20347524 -0.98204119 -0.51990416]\n", 262 | "[-0.72869851 1.20347524 -0.98204119 -0.51990416]\n" 263 | ] 264 | } 265 | ], 266 | "source": [ 267 | "# Normalization\n", 268 | "\n", 269 | "scaler_X = preprocessing.StandardScaler().fit(RNN_input.reshape(-1, 4))\n", 270 | "scaler_y = preprocessing.StandardScaler().fit(RNN_output.reshape(-1, 2))\n", 271 | "\n", 272 | "print(scaler_X.mean_)\n", 273 | "print(scaler_X.var_)\n", 274 | "print(scaler_y.mean_)\n", 275 | "print(scaler_y.var_)\n", 276 | "\n", 277 | "RNN_input = scaler_X.transform(RNN_input.reshape(-1, 4)).reshape(-1,10,4)\n", 278 | "RNN_output = scaler_y.transform(RNN_output.reshape(-1, 2)).reshape(-1,10,2)\n", 279 | "\n", 280 | "# split into train and test sets\n", 281 | "X_train, X_test, y_train, y_test = train_test_split(RNN_input, RNN_output, test_size=0.3, random_state=123)\n", 282 | "\n", 283 | "# checking X_train\n", 284 | "print(X_train[0, 0])\n", 285 | "print(X_train[0, 1])" 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "execution_count": 10, 291 | "metadata": {}, 292 | "outputs": [], 293 | "source": [ 294 | "np.save('X_train.npy',X_train)\n", 295 | "np.save('X_test.npy',X_test)\n", 296 | "np.save('y_train.npy',y_train)\n", 297 | "np.save('y_test.npy',y_test)\n", 298 | "np.save('X_mean.npy',scaler_X.mean_)\n", 299 | "np.save('X_std.npy',np.sqrt(scaler_X.var_))\n", 300 | "np.save('y_mean.npy',scaler_y.mean_)\n", 301 | "np.save('y_std.npy',np.sqrt(scaler_y.var_))" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": 11, 307 | "metadata": {}, 308 | "outputs": [ 309 | { 310 | "name": "stderr", 311 | "output_type": "stream", 312 | "text": [ 313 | "2024-02-21 10:28:25.422320: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n", 314 | "2024-02-21 10:28:25.423406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1\n", 315 | "2024-02-21 10:28:25.451859: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 316 | "2024-02-21 10:28:25.451941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: \n", 317 | "pciBusID: 0000:01:00.0 name: NVIDIA RTX A5000 computeCapability: 8.6\n", 318 | "coreClock: 1.695GHz coreCount: 64 deviceMemorySize: 23.68GiB deviceMemoryBandwidth: 715.34GiB/s\n", 319 | "2024-02-21 10:28:25.451957: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n", 320 | "2024-02-21 10:28:25.453190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11\n", 321 | "2024-02-21 10:28:25.453219: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11\n", 322 | "2024-02-21 10:28:25.453795: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10\n", 323 | "2024-02-21 10:28:25.453942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10\n", 324 | "2024-02-21 10:28:25.456665: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10\n", 325 | "2024-02-21 10:28:25.457253: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11\n", 326 | "2024-02-21 10:28:25.457621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8\n", 327 | "2024-02-21 10:28:25.458038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 328 | "2024-02-21 10:28:25.458373: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 329 | "2024-02-21 10:28:25.458436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0\n", 330 | "2024-02-21 10:28:25.459526: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX512F\n", 331 | "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", 332 | "2024-02-21 10:28:25.460518: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n", 333 | "2024-02-21 10:28:25.460669: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 334 | "2024-02-21 10:28:25.460757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: \n", 335 | "pciBusID: 0000:01:00.0 name: NVIDIA RTX A5000 computeCapability: 8.6\n", 336 | "coreClock: 1.695GHz coreCount: 64 deviceMemorySize: 23.68GiB deviceMemoryBandwidth: 715.34GiB/s\n", 337 | "2024-02-21 10:28:25.460799: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n", 338 | "2024-02-21 10:28:25.460819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11\n", 339 | "2024-02-21 10:28:25.460828: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11\n", 340 | "2024-02-21 10:28:25.460837: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10\n", 341 | "2024-02-21 10:28:25.460846: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10\n", 342 | "2024-02-21 10:28:25.460856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10\n", 343 | "2024-02-21 10:28:25.460865: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11\n", 344 | "2024-02-21 10:28:25.460873: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8\n", 345 | "2024-02-21 10:28:25.460916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 346 | "2024-02-21 10:28:25.460990: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 347 | "2024-02-21 10:28:25.461037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0\n", 348 | "2024-02-21 10:28:25.461060: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n", 349 | "2024-02-21 10:28:25.843337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:\n", 350 | "2024-02-21 10:28:25.843393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 \n", 351 | "2024-02-21 10:28:25.843398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N \n", 352 | "2024-02-21 10:28:25.843739: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 353 | "2024-02-21 10:28:25.843884: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 354 | "2024-02-21 10:28:25.843962: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 355 | "2024-02-21 10:28:25.844037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22449 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6)\n", 356 | "2024-02-21 10:28:26.251018: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)\n", 357 | "2024-02-21 10:28:26.335765: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2496000000 Hz\n" 358 | ] 359 | }, 360 | { 361 | "name": "stdout", 362 | "output_type": "stream", 363 | "text": [ 364 | "Epoch 1/100\n" 365 | ] 366 | }, 367 | { 368 | "name": "stderr", 369 | "output_type": "stream", 370 | "text": [ 371 | "2024-02-21 10:28:28.464707: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11\n", 372 | "2024-02-21 10:28:29.070909: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11\n", 373 | "2024-02-21 10:28:29.238823: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8\n", 374 | "2024-02-21 10:28:51.294480: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.\n" 375 | ] 376 | }, 377 | { 378 | "name": "stdout", 379 | "output_type": "stream", 380 | "text": [ 381 | "421/421 - 27s - loss: 0.0840 - val_loss: 0.0031\n", 382 | "Epoch 2/100\n", 383 | "421/421 - 1s - loss: 0.0018 - val_loss: 8.7862e-04\n", 384 | "Epoch 3/100\n", 385 | "421/421 - 1s - loss: 4.6085e-04 - val_loss: 1.7768e-04\n", 386 | "Epoch 4/100\n", 387 | "421/421 - 1s - loss: 8.0227e-05 - val_loss: 3.4171e-05\n", 388 | "Epoch 5/100\n", 389 | "421/421 - 1s - loss: 2.3749e-05 - val_loss: 1.7404e-05\n", 390 | "Epoch 6/100\n", 391 | "421/421 - 1s - loss: 1.5054e-05 - val_loss: 1.1524e-05\n", 392 | "Epoch 7/100\n", 393 | "421/421 - 1s - loss: 1.3127e-05 - val_loss: 1.0504e-05\n", 394 | "Epoch 8/100\n", 395 | "421/421 - 1s - loss: 1.3166e-05 - val_loss: 7.6837e-06\n", 396 | "Epoch 9/100\n", 397 | "421/421 - 1s - loss: 9.4243e-06 - val_loss: 6.6791e-06\n", 398 | "Epoch 10/100\n", 399 | "421/421 - 1s - loss: 1.2481e-05 - val_loss: 6.4639e-06\n", 400 | "Epoch 11/100\n", 401 | "421/421 - 1s - loss: 9.4851e-06 - val_loss: 6.6650e-06\n", 402 | "Epoch 12/100\n", 403 | "421/421 - 1s - loss: 1.2279e-05 - val_loss: 2.5351e-05\n", 404 | "Epoch 13/100\n", 405 | "421/421 - 1s - loss: 9.6027e-06 - val_loss: 5.1697e-06\n", 406 | "Epoch 14/100\n", 407 | "421/421 - 1s - loss: 1.1595e-05 - val_loss: 8.2628e-06\n", 408 | "Epoch 15/100\n", 409 | "421/421 - 1s - loss: 1.0947e-05 - val_loss: 1.4353e-05\n", 410 | "Epoch 16/100\n", 411 | "421/421 - 1s - loss: 1.0239e-05 - val_loss: 5.7932e-06\n", 412 | "Epoch 17/100\n", 413 | "421/421 - 1s - loss: 1.0529e-05 - val_loss: 3.1893e-06\n", 414 | "Epoch 18/100\n", 415 | "421/421 - 1s - loss: 1.1624e-05 - val_loss: 2.0867e-05\n", 416 | "Epoch 19/100\n", 417 | "421/421 - 1s - loss: 8.1228e-06 - val_loss: 4.4792e-06\n", 418 | "Epoch 20/100\n", 419 | "421/421 - 1s - loss: 1.0536e-05 - val_loss: 3.0476e-06\n", 420 | "Epoch 21/100\n", 421 | "421/421 - 1s - loss: 1.1288e-05 - val_loss: 4.4886e-06\n", 422 | "Epoch 22/100\n", 423 | "421/421 - 1s - loss: 5.0512e-06 - val_loss: 1.6076e-05\n", 424 | "Epoch 23/100\n", 425 | "421/421 - 1s - loss: 1.0438e-05 - val_loss: 7.2526e-06\n", 426 | "Epoch 24/100\n", 427 | "421/421 - 1s - loss: 8.5285e-06 - val_loss: 2.8927e-05\n", 428 | "Epoch 25/100\n", 429 | "421/421 - 1s - loss: 7.5739e-06 - val_loss: 7.3439e-06\n", 430 | "Epoch 26/100\n", 431 | "421/421 - 1s - loss: 8.3979e-06 - val_loss: 2.1963e-06\n", 432 | "Epoch 27/100\n", 433 | "421/421 - 1s - loss: 9.2899e-06 - val_loss: 2.3667e-06\n", 434 | "Epoch 28/100\n", 435 | "421/421 - 1s - loss: 6.1636e-06 - val_loss: 3.3234e-06\n", 436 | "Epoch 29/100\n", 437 | "421/421 - 1s - loss: 9.2320e-06 - val_loss: 9.9937e-06\n", 438 | "Epoch 30/100\n", 439 | "421/421 - 1s - loss: 6.1414e-06 - val_loss: 5.1030e-06\n", 440 | "Epoch 31/100\n", 441 | "421/421 - 1s - loss: 8.4749e-06 - val_loss: 1.1783e-05\n", 442 | "Epoch 32/100\n", 443 | "421/421 - 1s - loss: 7.7641e-06 - val_loss: 1.8456e-06\n", 444 | "Epoch 33/100\n", 445 | "421/421 - 1s - loss: 9.0241e-06 - val_loss: 2.2181e-06\n", 446 | "Epoch 34/100\n", 447 | "421/421 - 1s - loss: 5.1764e-06 - val_loss: 3.2333e-06\n", 448 | "Epoch 35/100\n", 449 | "421/421 - 1s - loss: 4.5075e-06 - val_loss: 3.1991e-06\n", 450 | "Epoch 36/100\n", 451 | "421/421 - 1s - loss: 6.9000e-06 - val_loss: 1.6357e-06\n", 452 | "Epoch 37/100\n", 453 | "421/421 - 1s - loss: 7.1700e-06 - val_loss: 1.5716e-05\n", 454 | "Epoch 38/100\n", 455 | "421/421 - 1s - loss: 5.8307e-06 - val_loss: 1.6306e-05\n", 456 | "Epoch 39/100\n", 457 | "421/421 - 1s - loss: 5.9178e-06 - val_loss: 1.8610e-06\n", 458 | "Epoch 40/100\n", 459 | "421/421 - 1s - loss: 7.7533e-06 - val_loss: 1.7409e-06\n", 460 | "Epoch 41/100\n", 461 | "421/421 - 1s - loss: 4.4593e-06 - val_loss: 2.0611e-06\n", 462 | "Epoch 42/100\n", 463 | "421/421 - 1s - loss: 6.8187e-06 - val_loss: 1.5721e-06\n", 464 | "Epoch 43/100\n", 465 | "421/421 - 1s - loss: 6.3038e-06 - val_loss: 2.6926e-06\n", 466 | "Epoch 44/100\n", 467 | "421/421 - 1s - loss: 5.1890e-06 - val_loss: 3.9038e-06\n", 468 | "Epoch 45/100\n", 469 | "421/421 - 1s - loss: 5.5451e-06 - val_loss: 2.0774e-06\n", 470 | "Epoch 46/100\n", 471 | "421/421 - 1s - loss: 5.5083e-06 - val_loss: 1.9519e-06\n", 472 | "Epoch 47/100\n", 473 | "421/421 - 1s - loss: 5.1549e-06 - val_loss: 3.1790e-06\n", 474 | "Epoch 48/100\n", 475 | "421/421 - 1s - loss: 5.7650e-06 - val_loss: 1.3267e-06\n", 476 | "Epoch 49/100\n", 477 | "421/421 - 1s - loss: 5.0594e-06 - val_loss: 1.0482e-06\n", 478 | "Epoch 50/100\n", 479 | "421/421 - 1s - loss: 7.1688e-06 - val_loss: 3.4804e-06\n", 480 | "Epoch 51/100\n", 481 | "421/421 - 1s - loss: 3.8364e-06 - val_loss: 4.1716e-06\n", 482 | "Epoch 52/100\n", 483 | "421/421 - 1s - loss: 6.0176e-06 - val_loss: 5.1703e-06\n", 484 | "Epoch 53/100\n", 485 | "421/421 - 1s - loss: 3.2911e-06 - val_loss: 6.3235e-06\n", 486 | "Epoch 54/100\n", 487 | "421/421 - 1s - loss: 5.3345e-06 - val_loss: 1.1659e-05\n", 488 | "Epoch 55/100\n", 489 | "421/421 - 1s - loss: 5.2957e-06 - val_loss: 8.7990e-06\n", 490 | "Epoch 56/100\n", 491 | "421/421 - 1s - loss: 5.1285e-06 - val_loss: 8.0515e-07\n", 492 | "Epoch 57/100\n", 493 | "421/421 - 1s - loss: 4.6393e-06 - val_loss: 6.4288e-06\n", 494 | "Epoch 58/100\n", 495 | "421/421 - 1s - loss: 7.1009e-06 - val_loss: 1.1350e-05\n", 496 | "Epoch 59/100\n", 497 | "421/421 - 1s - loss: 2.1043e-06 - val_loss: 3.3751e-06\n", 498 | "Epoch 60/100\n", 499 | "421/421 - 1s - loss: 3.9583e-06 - val_loss: 4.3719e-06\n", 500 | "Epoch 61/100\n", 501 | "421/421 - 1s - loss: 8.2017e-06 - val_loss: 1.7088e-06\n", 502 | "Epoch 62/100\n", 503 | "421/421 - 1s - loss: 1.5410e-06 - val_loss: 4.3479e-06\n", 504 | "Epoch 63/100\n", 505 | "421/421 - 1s - loss: 4.5081e-06 - val_loss: 7.9819e-07\n", 506 | "Epoch 64/100\n", 507 | "421/421 - 1s - loss: 4.3019e-06 - val_loss: 6.7453e-06\n", 508 | "Epoch 65/100\n", 509 | "421/421 - 1s - loss: 4.4400e-06 - val_loss: 2.4986e-06\n", 510 | "Epoch 66/100\n", 511 | "421/421 - 1s - loss: 4.1741e-06 - val_loss: 1.7548e-06\n", 512 | "Epoch 67/100\n", 513 | "421/421 - 1s - loss: 4.3357e-06 - val_loss: 7.0194e-07\n", 514 | "Epoch 68/100\n", 515 | "421/421 - 1s - loss: 3.6766e-06 - val_loss: 1.5335e-06\n", 516 | "Epoch 69/100\n", 517 | "421/421 - 1s - loss: 3.4976e-06 - val_loss: 5.5209e-06\n", 518 | "Epoch 70/100\n", 519 | "421/421 - 1s - loss: 5.2310e-06 - val_loss: 3.8231e-06\n", 520 | "Epoch 71/100\n", 521 | "421/421 - 1s - loss: 4.3236e-06 - val_loss: 1.2795e-06\n", 522 | "Epoch 72/100\n", 523 | "421/421 - 1s - loss: 3.4877e-06 - val_loss: 1.6783e-06\n", 524 | "Epoch 73/100\n", 525 | "421/421 - 1s - loss: 4.5582e-06 - val_loss: 2.7497e-06\n", 526 | "Epoch 74/100\n", 527 | "421/421 - 1s - loss: 3.4199e-06 - val_loss: 3.1909e-06\n", 528 | "Epoch 75/100\n", 529 | "421/421 - 1s - loss: 4.0955e-06 - val_loss: 3.7174e-06\n", 530 | "Epoch 76/100\n", 531 | "421/421 - 1s - loss: 3.3441e-06 - val_loss: 3.9447e-06\n", 532 | "Epoch 77/100\n", 533 | "421/421 - 1s - loss: 4.0914e-06 - val_loss: 7.8676e-06\n", 534 | "Epoch 78/100\n", 535 | "421/421 - 1s - loss: 3.8378e-06 - val_loss: 4.6919e-06\n", 536 | "Epoch 79/100\n", 537 | "421/421 - 1s - loss: 4.5851e-06 - val_loss: 5.4041e-06\n", 538 | "Epoch 80/100\n", 539 | "421/421 - 1s - loss: 2.2709e-06 - val_loss: 4.1691e-06\n", 540 | "Epoch 81/100\n", 541 | "421/421 - 1s - loss: 4.5990e-06 - val_loss: 2.6995e-06\n", 542 | "Epoch 82/100\n", 543 | "421/421 - 1s - loss: 3.7309e-06 - val_loss: 1.6635e-06\n", 544 | "Epoch 83/100\n", 545 | "421/421 - 1s - loss: 2.6284e-06 - val_loss: 6.7241e-06\n", 546 | "Epoch 84/100\n", 547 | "421/421 - 1s - loss: 7.2892e-06 - val_loss: 4.4365e-07\n", 548 | "Epoch 85/100\n", 549 | "421/421 - 1s - loss: 1.0918e-06 - val_loss: 1.0316e-06\n", 550 | "Epoch 86/100\n", 551 | "421/421 - 1s - loss: 3.1322e-06 - val_loss: 3.2079e-06\n", 552 | "Epoch 87/100\n", 553 | "421/421 - 1s - loss: 3.6172e-06 - val_loss: 4.6684e-07\n", 554 | "Epoch 88/100\n", 555 | "421/421 - 1s - loss: 4.2615e-06 - val_loss: 1.3923e-06\n", 556 | "Epoch 89/100\n", 557 | "421/421 - 1s - loss: 5.2540e-06 - val_loss: 6.7662e-07\n", 558 | "Epoch 90/100\n", 559 | "421/421 - 1s - loss: 1.1922e-06 - val_loss: 7.1169e-07\n", 560 | "Epoch 91/100\n", 561 | "421/421 - 1s - loss: 3.9915e-06 - val_loss: 1.5073e-06\n", 562 | "Epoch 92/100\n", 563 | "421/421 - 1s - loss: 4.4626e-06 - val_loss: 6.0079e-07\n", 564 | "Epoch 93/100\n", 565 | "421/421 - 1s - loss: 4.6675e-06 - val_loss: 1.1668e-06\n", 566 | "Epoch 94/100\n", 567 | "421/421 - 1s - loss: 8.4509e-07 - val_loss: 2.6283e-06\n", 568 | "Epoch 95/100\n", 569 | "421/421 - 1s - loss: 4.1223e-06 - val_loss: 3.1129e-06\n", 570 | "Epoch 96/100\n", 571 | "421/421 - 1s - loss: 2.6247e-06 - val_loss: 1.0943e-06\n", 572 | "Epoch 97/100\n", 573 | "421/421 - 1s - loss: 3.6573e-06 - val_loss: 2.8297e-06\n", 574 | "Epoch 98/100\n", 575 | "421/421 - 1s - loss: 3.6310e-06 - val_loss: 8.7850e-07\n", 576 | "Epoch 99/100\n", 577 | "421/421 - 1s - loss: 2.9179e-06 - val_loss: 1.3847e-06\n", 578 | "Epoch 100/100\n", 579 | "421/421 - 1s - loss: 4.2174e-06 - val_loss: 1.4105e-06\n" 580 | ] 581 | }, 582 | { 583 | "data": { 584 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAlNElEQVR4nO3df5wWdb338dd7fwgpiApoCRJ4/BXKbxCVNNTOSdIjapiSqWT562SlZoaVyrG8z/HISW9P6IkyNbPAW8ubEqPbX6lpKqBpKJxQMdfUEBUhQtndz/3HzHUx18Xssix7ccHu+/l4wF4z852Zz8zsXu/rO3NdcykiMDMzK1dT7QLMzGzr5IAwM7NcDggzM8vlgDAzs1wOCDMzy+WAMDOzXA4I2yIk3SPp9I5uW02Slkn6eAWWG5L2Sh//t6RL29K2Hes5RdJv2ltnK8sdL6mho5drW15dtQuwrZek1ZnB7YH3gKZ0+OyIuK2ty4qICZVo29lFxDkdsRxJA4GXgPqIaEyXfRvQ5mNoXY8DwloUET0KjyUtA74QEfeWt5NUV3jSMbPOw6eYbJMVTiFI+rqk14GbJO0s6VeSlkt6O33cPzPPg5K+kD6eIukRSdPTti9JmtDOtoMkPSRplaR7Jc2Q9JMW6m5Ljd+W9Lt0eb+R1Ccz/VRJL0taIembreyfsZJel1SbGXe8pGfSxwdKekzSO5Jek/Q9Sdu1sKybJX0nM/y1dJ6/SDqjrO3Rkp6S9K6kVyRNy0x+KP35jqTVkg4u7NvM/IdIelLSyvTnIW3dN62R9JF0/nckLZJ0bGbaJyU9ly7zVUkXpeP7pMfnHUlvSXpYkp+vtjDvcGuvDwK7AB8GziL5XbopHR4A/B34XivzjwWWAH2A/wBulKR2tP0p8ATQG5gGnNrKOttS42eAzwG7AtsBhSeswcAN6fJ3T9fXnxwR8TjwN+CIsuX+NH3cBFyQbs/BwJHAv7RSN2kNR6X1/COwN1B+/eNvwGnATsDRwLmSjkunHZb+3CkiekTEY2XL3gW4G7gu3bbvAndL6l22DRvsm43UXA/8EvhNOt+XgNsk7Zs2uZHkdGVP4ADg/nT8V4EGoC+wG/ANwPcF2sIcENZezcDlEfFeRPw9IlZExJ0RsSYiVgFXAh9rZf6XI+IHEdEE3AJ8iOSJoM1tJQ0AxgCXRcT7EfEIMKelFbaxxpsi4n8i4u/A7cDwdPwk4FcR8VBEvAdcmu6DlvwMmAwgqSfwyXQcEbEgIn4fEY0RsQz4fk4deT6d1vfHiPgbSSBmt+/BiHg2Ipoj4pl0fW1ZLiSB8qeIuDWt62fAYuCfM21a2jetOQjoAfx7eozuB35Fum+AdcBgSTtGxNsRsTAz/kPAhyNiXUQ8HL5x3BbngLD2Wh4RawsDkraX9P30FMy7JKc0dsqeZinzeuFBRKxJH/bYxLa7A29lxgG80lLBbazx9czjNZmads8uO32CXtHSukh6CydI6gacACyMiJfTOvZJT5+8ntbxv0h6ExtTUgPwctn2jZX0QHoKbSVwThuXW1j2y2XjXgb6ZYZb2jcbrTkismGaXe6nSMLzZUm/lXRwOv5qYCnwG0kvSprats2wjuSAsPYqfzX3VWBfYGxE7Mj6UxotnTbqCK8Bu0jaPjNuj1bab06Nr2WXna6zd0uNI+I5kifCCZSeXoLkVNViYO+0jm+0pwaS02RZPyXpQe0REb2A/84sd2Ovvv9CcuotawDwahvq2thy9yi7flBcbkQ8GRETSU4/3UXSMyEiVkXEVyNiT+BY4EJJR25mLbaJHBDWUXqSnNN/Jz2ffXmlV5i+Ip8PTJO0Xfrq859bmWVzarwDOEbSR9MLylew8b+fnwJfIQmi/1NWx7vAakn7Aee2sYbbgSmSBqcBVV5/T5Ie1VpJB5IEU8FyklNie7aw7LnAPpI+I6lO0knAYJLTQZvjcZLexsWS6iWNJzlGs9JjdoqkXhGxjmSfNANIOkbSXum1ppUk121aO6VnFeCAsI5yLfAB4E3g98Cvt9B6TyG50LsC+A4wm+TzGnmupZ01RsQi4IskT/qvAW+TXERtTeEawP0R8WZm/EUkT96rgB+kNbelhnvSbbif5PTL/WVN/gW4QtIq4DLSV+PpvGtIrrn8Ln1n0EFly14BHEPSy1oBXAwcU1b3JouI90kCYQLJfr8eOC0iFqdNTgWWpafaziE5npBchL8XWA08BlwfEQ9sTi226eTrPtaZSJoNLI6IivdgzDo79yBsmyZpjKR/kFSTvg10Ism5bDPbTP4ktW3rPgj8nOSCcQNwbkQ8Vd2SzDoHn2IyM7NcPsVkZma5Os0ppj59+sTAgQOrXYaZ2TZlwYIFb0ZE37xpnSYgBg4cyPz586tdhpnZNkVS+Sfoi3yKyczMcjkgzMwslwPCzMxydZprEGa25a1bt46GhgbWrl278cZWVd27d6d///7U19e3eR4HhJm1W0NDAz179mTgwIG0/H1PVm0RwYoVK2hoaGDQoEFtns+nmMys3dauXUvv3r0dDls5SfTu3XuTe3oOCDPbLA6HbUN7jpMDoqEBLrsM/ud/ql2JmdlWxQHx+uvw7W87IMy2QStWrGD48OEMHz6cD37wg/Tr1684/P7777c67/z58/nyl7+80XUccsghHVLrgw8+yDHHHNMhy9pSfJG6Lt0FjY3VrcPMNlnv3r15+umnAZg2bRo9evTgoosuKk5vbGykri7/aW706NGMHj16o+t49NFHO6TWbZF7EA4Is05lypQpnHPOOYwdO5aLL76YJ554goMPPpgRI0ZwyCGHsGTJEqD0Ff20adM444wzGD9+PHvuuSfXXXddcXk9evQoth8/fjyTJk1iv/3245RTTqFwN+y5c+ey3377MWrUKL785S9vtKfw1ltvcdxxxzF06FAOOuggnnnmGQB++9vfFntAI0aMYNWqVbz22mscdthhDB8+nAMOOICHH364w/dZS9yDcECYdYzzz4f01XyHGT4crr12k2draGjg0Ucfpba2lnfffZeHH36Yuro67r33Xr7xjW9w5513bjDP4sWLeeCBB1i1ahX77rsv55577gafGXjqqadYtGgRu+++O+PGjeN3v/sdo0eP5uyzz+ahhx5i0KBBTJ48eaP1XX755YwYMYK77rqL+++/n9NOO42nn36a6dOnM2PGDMaNG8fq1avp3r07M2fO5BOf+ATf/OY3aWpqYs2aNZu8P9rLAeGAMOt0TjzxRGprawFYuXIlp59+On/605+QxLp163LnOfroo+nWrRvdunVj11135Y033qB///4lbQ488MDiuOHDh7Ns2TJ69OjBnnvuWfx8weTJk5k5c2ar9T3yyCPFkDriiCNYsWIF7777LuPGjePCCy/klFNO4YQTTqB///6MGTOGM844g3Xr1nHccccxfPjwzdk1m8QB4YAw6xjteKVfKTvssEPx8aWXXsrhhx/OL37xC5YtW8b48eNz5+nWrVvxcW1tLY05zwltabM5pk6dytFHH83cuXMZN24c8+bN47DDDuOhhx7i7rvvZsqUKVx44YWcdtppHbrellT0GoSkoyQtkbRU0tSc6d0kzU6nPy5pYDq+XtItkp6V9LykSypWpAPCrFNbuXIl/fr1A+Dmm2/u8OXvu+++vPjiiyxbtgyA2bNnb3SeQw89lNtuuw1Irm306dOHHXfckRdeeIEhQ4bw9a9/nTFjxrB48WJefvlldtttN84880y+8IUvsHDhwg7fhpZULCAk1QIzgAnAYGCypMFlzT4PvB0RewHXAFel408EukXEEGAUcHYhPDqcA8KsU7v44ou55JJLGDFiRIe/4gf4wAc+wPXXX89RRx3FqFGj6NmzJ7169Wp1nmnTprFgwQKGDh3K1KlTueWWWwC49tprOeCAAxg6dCj19fVMmDCBBx98kGHDhjFixAhmz57NV77ylQ7fhpZU7DupJR0MTIuIT6TDlwBExL9l2sxL2zwmqQ54HegLnAx8Bjge6AU8BhwUEW+1tL7Ro0dHu74w6M03oW9f+K//gvPO2/T5zbqw559/no985CPVLqPqVq9eTY8ePYgIvvjFL7L33ntzwQUXVLusDeQdL0kLIiL3/b6VPMXUD3glM9yQjsttExGNwEqgN3AH8DfgNeDPwPTWwmGzuAdhZpvpBz/4AcOHD2f//fdn5cqVnH322dUuqUNsrRepDwSagN2BnYGHJd0bES9mG0k6CzgLYMCAAe1bkwPCzDbTBRdcsFX2GDZXJXsQrwJ7ZIb7p+Ny26SnmHoBK0hOL/06ItZFxF+B3wEbdIEiYmZEjI6I0X375n7n9sY5IMzMclUyIJ4E9pY0SNJ2JNcV5pS1mQOcnj6eBNwfyUWRPwNHAEjaATgIWFyRKh0QZma5KhYQ6TWF84B5wPPA7RGxSNIVko5Nm90I9Ja0FLgQKLwVdgbQQ9IikqC5KSKeqUih6YdpaOHDM2ZmXVVFr0FExFxgbtm4yzKP15K8pbV8vtV54ytCSkLCPQgzsxK+WR9Afb0DwmwbdPjhhzNv3ryScddeey3nnntui/OMHz+ewlviP/nJT/LOO+9s0GbatGlMnz691XXfddddPPfcc8Xhyy67jHvvvXcTqs+3Nd0W3AEByXUIB4TZNmfy5MnMmjWrZNysWbPadMM8SO7CutNOO7Vr3eUBccUVV/Dxj3+8XcvaWjkgwAFhto2aNGkSd999d/HLgZYtW8Zf/vIXDj30UM4991xGjx7N/vvvz+WXX547/8CBA3nzzTcBuPLKK9lnn3346Ec/WrwlOCSfcRgzZgzDhg3jU5/6FGvWrOHRRx9lzpw5fO1rX2P48OG88MILTJkyhTvuuAOA++67jxEjRjBkyBDOOOMM3nvvveL6Lr/8ckaOHMmQIUNYvLj1995U+7bgW+vnILYsB4TZZqvG3b532WUXDjzwQO655x4mTpzIrFmz+PSnP40krrzySnbZZReampo48sgjeeaZZxg6dGjuchYsWMCsWbN4+umnaWxsZOTIkYwaNQqAE044gTPPPBOAb33rW9x444186Utf4thjj+WYY45h0qRJJctau3YtU6ZM4b777mOfffbhtNNO44YbbuD8888HoE+fPixcuJDrr7+e6dOn88Mf/rDF7av2bcHdgwAHhNk2LHuaKXt66fbbb2fkyJGMGDGCRYsWlZwOKvfwww9z/PHHs/3227Pjjjty7LHHFqf98Y9/5NBDD2XIkCHcdtttLFq0qNV6lixZwqBBg9hnn30AOP3003nooYeK00844QQARo0aVbzBX0seeeQRTj31VCD/tuDXXXcd77zzDnV1dYwZM4abbrqJadOm8eyzz9KzZ89Wl90W7kGAA8KsA1Trbt8TJ07kggsuYOHChaxZs4ZRo0bx0ksvMX36dJ588kl23nlnpkyZwtq1a9u1/ClTpnDXXXcxbNgwbr75Zh588MHNqrdwy/DNuV34lrotuHsQ4IAw24b16NGDww8/nDPOOKPYe3j33XfZYYcd6NWrF2+88Qb33HNPq8s47LDDuOuuu/j73//OqlWr+OUvf1mctmrVKj70oQ+xbt264i26AXr27MmqVas2WNa+++7LsmXLWLp0KQC33norH/vYx9q1bdW+Lbh7EOCAMNvGTZ48meOPP754qqlwe+z99tuPPfbYg3HjxrU6/8iRIznppJMYNmwYu+66K2PGjClO+/a3v83YsWPp27cvY8eOLYbCySefzJlnnsl1111XvDgN0L17d2666SZOPPFEGhsbGTNmDOecc067tqvwXdlDhw5l++23L7kt+AMPPEBNTQ37778/EyZMYNasWVx99dXU19fTo0cPfvzjH7drnVkVu933ltbu230DfOQjMHQotOGLPsxsPd/ue9uyNd3ue9vhHoSZ2QYcEOCAMDPL4YAAB4TZZugsp6k7u/YcJwcEOCDM2ql79+6sWLHCIbGViwhWrFhB9+7dN2k+v4sJkoDw7b7NNln//v1paGhg+fLl1S7FNqJ79+70799/k+ZxQIB7EGbtVF9fz6BBg6pdhlWITzGBA8LMLIcDAvx9EGZmORwQ4B6EmVkOBwQ4IMzMcjggwAFhZpbDAQEOCDOzHA4IcECYmeVwQIADwswshwMCHBBmZjkcEOCAMDPL4YAAB4SZWQ4HBDggzMxyOCDAd3M1M8vhgAD3IMzMcjggYH1A+EtPzMyKHBCQBARAc3N16zAz24o4ICC53Tf4NJOZWYYDAtb3IBwQZmZFDghwQJiZ5XBAgAPCzCyHAwIcEGZmORwQ4IAwM8vhgAAHhJlZjooGhKSjJC2RtFTS1Jzp3STNTqc/LmlgZtpQSY9JWiTpWUndK1aoA8LMbAMVCwhJtcAMYAIwGJgsaXBZs88Db0fEXsA1wFXpvHXAT4BzImJ/YDxQuZslOSDMzDZQyR7EgcDSiHgxIt4HZgETy9pMBG5JH98BHClJwD8Bz0TEHwAiYkVENFWsUgeEmdkGKhkQ/YBXMsMN6bjcNhHRCKwEegP7ACFpnqSFki6uYJ0OCDOzHHXVLqAFdcBHgTHAGuA+SQsi4r5sI0lnAWcBDBgwYDPWlu4G3/LbzKyokj2IV4E9MsP903G5bdLrDr2AFSS9jYci4s2IWAPMBUaWryAiZkbE6IgY3bdv3/ZX6h6EmdkGKhkQTwJ7SxokaTvgZGBOWZs5wOnp40nA/RERwDxgiKTt0+D4GPBcxSp1QJiZbaBip5giolHSeSRP9rXAjyJikaQrgPkRMQe4EbhV0lLgLZIQISLelvRdkpAJYG5E3F2pWh0QZmYbqug1iIiYS3J6KDvusszjtcCJLcz7E5K3ulaeb/dtZrYBf5Ia3IMwM8vhgAAHhJlZDgcEOCDMzHI4IMABYWaWwwEBDggzsxwOCHBAmJnlcECAA8LMLIcDAhwQZmY5HBDggDAzy+GAAAeEmVkOBwT4dt9mZjkcEOAehJlZDgcEOCDMzHI4IMABYWaWwwEBUFub/HRAmJkVOSAApKQX4YAwMytyQBQ4IMzMSjggChwQZmYlHBAFDggzsxIOiAIHhJlZCQdEgQPCzKyEA6LAAWFmVsIBUeCAMDMr4YAocECYmZVwQBQ4IMzMSjggCurqfLtvM7MMB0SBexBmZiUcEAUOCDOzEg6IAgeEmVkJB0RBfb0DwswswwFR4B6EmVmJNgWEpK9I2lGJGyUtlPRPlS5ui3JAmJmVaGsP4oyIeBf4J2Bn4FTg3ytWVTU4IMzMSrQ1IJT+/CRwa0QsyozrHBwQZmYl2hoQCyT9hiQg5knqCTRXrqwqcECYmZWoa2O7zwPDgRcjYo2kXYDPVayqanBAmJmVaGsP4mBgSUS8I+mzwLeAlZUrqwocEGZmJdoaEDcAayQNA74KvAD8uGJVVYMDwsysRFsDojEiApgIfC8iZgA9K1dWFTggzMxKtDUgVkm6hOTtrXdLqgHqNzaTpKMkLZG0VNLUnOndJM1Opz8uaWDZ9AGSVku6qI11tp8DwsysRFsD4iTgPZLPQ7wO9Aeubm0GSbXADGACMBiYLGlwWbPPA29HxF7ANcBVZdO/C9zTxho3j2/3bWZWok0BkYbCbUAvSccAayNiY9cgDgSWRsSLEfE+MIvkFFXWROCW9PEdwJGSBCDpOOAlYFFbatxs7kGYmZVo6602Pg08AZwIfBp4XNKkjczWD3glM9yQjsttExGNJO+M6i2pB/B14F83UtdZkuZLmr98+fK2bErLHBBmZiXa+jmIbwJjIuKvAJL6AveSvOqvhGnANRGxOu1Q5IqImcBMgNGjR8dmrdEBYWZWoq0BUVMIh9QKNt77eBXYIzPcPx2X16ZBUh3QK132WGCSpP8AdgKaJa2NiO+1sd5N54AwMyvR1oD4taR5wM/S4ZOAuRuZ50lgb0mDSILgZOAzZW3mAKcDjwGTgPvTt9MeWmggaRqwuqLhAOu/DyICWum1mJl1FW0KiIj4mqRPAePSUTMj4hcbmadR0nnAPKAW+FFELJJ0BTA/IuYANwK3SloKvEUSItVRl+6K5maora1aGWZmW4u29iCIiDuBOzdl4RExl7KeRkRclnm8luTCd2vLmLYp62y3QkA0NjogzMzYSEBIWgXkXfwVEBGxY0WqqoZsQHTrVt1azMy2Aq0GRER0rttptCYbEGZm5u+kLnJAmJmVcEAUOCDMzEo4IAocEGZmJRwQBQ4IM7MSDogCB4SZWQkHREEhIHzLbzMzwAGxnnsQZmYlHBAFDggzsxIOiAIHhJlZCQdEgQPCzKyEA6Kgvj756YAwMwMcEOu5B2FmVsIBUeCAMDMr4YAocECYmZVwQBQ4IMzMSjggChwQZmYlHBAFDggzsxIOiAIHhJlZCQdEgQPCzKyEA6LAAWFmVsIBUeDbfZuZlXBAFLgHYWZWwgFR4IAwMyvhgChwQJiZlXBAFDggzMxKOCAKfLtvM7MSDogC9yDMzEo4IApq0l3hgDAzAxwQ60lJL8IBYWYGOCBKOSDMzIocEFkOCDOzIgdElgPCzKzIAZHlgDAzK3JAZDkgzMyKHBBZDggzsyIHRFZdnW/3bWaWqmhASDpK0hJJSyVNzZneTdLsdPrjkgam4/9R0gJJz6Y/j6hknUXuQZiZFVUsICTVAjOACcBgYLKkwWXNPg+8HRF7AdcAV6Xj3wT+OSKGAKcDt1aqzhIOCDOzokr2IA4ElkbEixHxPjALmFjWZiJwS/r4DuBISYqIpyLiL+n4RcAHJHWrYK0JB4SZWVElA6If8EpmuCEdl9smIhqBlUDvsjafAhZGxHvlK5B0lqT5kuYvX7588yt2QJiZFW3VF6kl7U9y2unsvOkRMTMiRkfE6L59+27+CuvrHRBmZqlKBsSrwB6Z4f7puNw2kuqAXsCKdLg/8AvgtIh4oYJ1rucehJlZUSUD4klgb0mDJG0HnAzMKWszh+QiNMAk4P6ICEk7AXcDUyPidxWssZQDwsysqGIBkV5TOA+YBzwP3B4RiyRdIenYtNmNQG9JS4ELgcJbYc8D9gIuk/R0+m/XStVa5IAwMyuqq+TCI2IuMLds3GWZx2uBE3Pm+w7wnUrWlquuDtau3eKrNTPbGm3VF6m3OPcgzMyKHBBZDggzsyIHRJYDwsysyAGR5YAwMytyQGQ5IMzMihwQWb7dt5lZkQMiyz0IM7MiB0SWA8LMrMgBkeWAMDMrckBkOSDMzIocEFm+3beZWZEDIss9CDOzIgdElgPCzKzIAZFVCIiIaldiZlZ1DoisuvTu583N1a3DzGwr4IDIKgSETzOZmTkgSjggzMyKHBBZDggzsyIHRJYDwsysyAGRVQgI39HVzMwBUcI9CDOzIgdElgPCzKzIAZHlgDAzK3JAZDkgzMyKHBBZDggzsyIHRFZ9ffLTAWFm5oAo4R6EmVmRAyLLAWFmVuSAyHJAmJkVOSCyHBBmZkUOiCwHhJlZkQMiywFhZlbkgACeeir9llEHhJlZUZcPiPvug5EjYfZsHBBmZhldPiDGj08C4qKL4G/vpx+U8+2+zcwcELW1cN118Oqr8G8/2i0Z6R6EmZkDAmDcOPjsZ+Hqm3rzAnvC4sXVLsnMrOocEKmrroLttoMLd7sNrrgCbrih2iWZmVVVRQNC0lGSlkhaKmlqzvRukman0x+XNDAz7ZJ0/BJJn6hknQC77w6XXirmvHEQX9vzTt7+l2/AjBmVXq2Z2VarYgEhqRaYAUwABgOTJQ0ua/Z54O2I2Au4BrgqnXcwcDKwP3AUcH26vIo6/3yYMgX+86Xj+Yf6P3P1ectY9NGzWfkf3ydeeDF9L6yZWdegqNCTnqSDgWkR8Yl0+BKAiPi3TJt5aZvHJNUBrwN9ganZttl2La1v9OjRMX/+/A6p/Q9/gEu+3sw989bn5w6sZgf+Rp2aqFUzNQoE1KgZsX4fqmxZUnbahvu6mRqaoobGqKMZUacm6miiVk1EurRARCj5ma5DRMm6RdBMDY1RSyN1iCgup4bm3O1spib5FzUEKlleY9Tm1lRDc3H9eQo1Jj9VrE1EcVygpH6aqSnOQbF97o7MriNEEzU0UUtEacNCjbU0l+z7lmpti9z62tiuWF9hVGy47uw+K19e9nds/e9B6f4FqEl/HzILza9L+eOz+7Gw30p/rzfcl9nfz3J587Z1f5fvv7z9kq2zNdk6ivssHVVT9jvSUn0t/a63R1v3QUtaqmXCAQ3854Lx7VumtCAiRudNq2vXEtumH/BKZrgBGNtSm4holLQS6J2O/33ZvP3KVyDpLOAsgAEDBnRY4cOGwdxf1/CHPyTXqxuefpOGxxv4+8r3aXy/mab3m2luCgJozv5Nlj1ZRfG/wo/8P6S6miZqFdQoaArR2FxLY3MNUukfQw2BFESIZkRzur7CH1GtmovLCqCpuYbG5pqSP+TywKqlOXliiWR6c9qprK9poq6mCRHJcqKWxqgprjtCaIPNCQiQCk9qhScf1geb1v+JNIdoipoNn7Ta8PdYq+ZiUGf3d3MkgdsULXWOC5Vk1teK8qkt/XmX/uGXDm2wDpWNT/eZ0uMLlLwgKOzzQjAX9mHhZ2H/NqW/M9n1RKj4M7PEtMLSFxyF363yLWhxD0U2TPK3PjZo38KCWggBKP2dKYwt/1vLX2o2aNYvJ/tipZma4j7Jbs/6+bTB301u+aXFtyp73JO2Zc8bOXUUamlpPXsM2LzgaUklA6LiImImMBOSHkRHL3/YsOQfJ/UB+nT04s3MtmqVvEj9KrBHZrh/Oi63TXqKqRewoo3zmplZBVUyIJ4E9pY0SNJ2JBed55S1mQOcnj6eBNwfyUWROcDJ6bucBgF7A09UsFYzMytTsVNM6TWF84B5QC3wo4hYJOkKYH5EzAFuBG6VtBR4iyRESNvdDjwHNAJfjIimStVqZmYbqti7mLa0jnwXk5lZV9Hau5j8SWozM8vlgDAzs1wOCDMzy+WAMDOzXJ3mIrWk5cDLmzBLH+DNCpWzNeuK290Vtxm65nZ3xW2GzdvuD0dE37wJnSYgNpWk+S1due/MuuJ2d8Vthq653V1xm6Fy2+1TTGZmlssBYWZmubpyQMysdgFV0hW3uytuM3TN7e6K2wwV2u4uew3CzMxa15V7EGZm1goHhJmZ5eqSASHpKElLJC2VNLXa9VSCpD0kPSDpOUmLJH0lHb+LpP8n6U/pz52rXWslSKqV9JSkX6XDgyQ9nh7z2ekt6DsNSTtJukPSYknPSzq4KxxrSRekv99/lPQzSd0747GW9CNJf5X0x8y43OOrxHXp9j8jaWR719vlAkJSLTADmAAMBiZLGlzdqiqiEfhqRAwGDgK+mG7nVOC+iNgbuC8d7oy+AjyfGb4KuCYi9gLeBj5flaoq538Dv46I/YBhJNveqY+1pH7Al4HREXEAydcKnEznPNY3A0eVjWvp+E4g+Q6dvUm+kvmG9q60ywUEcCCwNCJejIj3gVnAxCrX1OEi4rWIWJg+XkXyhNGPZFtvSZvdAhxXlQIrSFJ/4Gjgh+mwgCOAO9ImnWq7JfUCDiP5fhUi4v2IeIcucKxJvtPmA+k3Um4PvEYnPNYR8RDJd+ZktXR8JwI/jsTvgZ0kfag96+2KAdEPeCUz3JCO67QkDQRGAI8Du0XEa+mk14HdqlVXBV0LXAw0p8O9gXciojEd7mzHfBCwHLgpPa32Q0k70MmPdUS8CkwH/kwSDCuBBXTuY53V0vHtsOe4rhgQXYqkHsCdwPkR8W52Wvr1rp3qfc6SjgH+GhELql3LFlQHjARuiIgRwN8oO53USY/1ziSvlgcBuwM7sOFpmC6hUse3KwbEq8AemeH+6bhOR1I9STjcFhE/T0e/Uehupj//Wq36KmQccKykZSSnD48gOT+/U3oaAjrfMW8AGiLi8XT4DpLA6OzH+uPASxGxPCLWAT8nOf6d+VhntXR8O+w5risGxJPA3uk7HbYjuag1p8o1dbj0vPuNwPMR8d3MpDnA6enj04H/u6Vrq6SIuCQi+kfEQJJje39EnAI8AExKm3Wq7Y6I14FXJO2bjjqS5PvcO/WxJjm1dJCk7dPf98J2d9pjXaal4zsHOC19N9NBwMrMqahN0iU/SS3pkyTnqWuBH0XEldWtqONJ+ijwMPAs68/Ff4PkOsTtwACS26N/OiLKL351CpLGAxdFxDGS9iTpUewCPAV8NiLeq2J5HUrScJKL8tsBLwKfI3kB2KmPtaR/BU4iedfeU8AXSM63d6pjLelnwHiS23q/AVwO3EXO8U3D8nskp9vWAJ+LiPntWm9XDAgzM9u4rniKyczM2sABYWZmuRwQZmaWywFhZma5HBBmZpbLAWG2FZA0vnDnWbOthQPCzMxyOSDMNoGkz0p6QtLTkr6ffu/EaknXpN9LcJ+kvmnb4ZJ+n96T/xeZ+/XvJeleSX+QtFDSP6SL75H5Tofb0g88mVWNA8KsjSR9hORTu+MiYjjQBJxCcpO4+RGxP/Bbkk+5AvwY+HpEDCX5RHth/G3AjIgYBhxCcidSSO64ez7J95TsSXJfIbOqqdt4EzNLHQmMAp5MX9x/gOQGac3A7LTNT4Cfp9/RsFNE/DYdfwvwfyT1BPpFxC8AImItQLq8JyKiIR1+GhgIPFLxrTJrgQPCrO0E3BIRl5SMlC4ta9fe+9dk7xfUhP8+rcp8isms7e4DJknaFYrfCfxhkr+jwt1DPwM8EhErgbclHZqOPxX4bfrtfg2SjkuX0U3S9ltyI8zayq9QzNooIp6T9C3gN5JqgHXAF0m+oOfAdNpfSa5TQHIL5v9OA6Bwh1VIwuL7kq5Il3HiFtwMszbz3VzNNpOk1RHRo9p1mHU0n2IyM7Nc7kGYmVku9yDMzCyXA8LMzHI5IMzMLJcDwszMcjkgzMws1/8HKN19TxlyhqoAAAAASUVORK5CYII=", 585 | "text/plain": [ 586 | "

" 587 | ] 588 | }, 589 | "metadata": { 590 | "needs_background": "light" 591 | }, 592 | "output_type": "display_data" 593 | } 594 | ], 595 | "source": [ 596 | "X_train = np.load('X_train.npy')\n", 597 | "X_test = np.load('X_test.npy')\n", 598 | "y_train = np.load('y_train.npy')\n", 599 | "y_test = np.load('y_test.npy')\n", 600 | "\n", 601 | "\n", 602 | "model = Sequential()\n", 603 | "model.add(LSTM(64, activation='tanh', return_sequences=True))\n", 604 | "model.add(LSTM(32, activation='tanh', return_sequences=True))\n", 605 | "model.add(Dense(2, activation='linear'))\n", 606 | "model.compile(optimizer='adam', loss='mean_squared_error')\n", 607 | "history = model.fit(X_train, y_train, epochs=100, batch_size=256, validation_split=0.1, verbose=2)\n", 608 | "\n", 609 | "\n", 610 | "loss = history.history['loss']\n", 611 | "val_loss = history.history['val_loss']\n", 612 | "epochs = range(1, len(loss) + 1)\n", 613 | "plt.plot(epochs, loss, 'r', label='Training loss')\n", 614 | "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n", 615 | "plt.title('Training and validation loss')\n", 616 | "plt.xlabel(\"epoch\")\n", 617 | "plt.ylabel(\"loss\")\n", 618 | "plt.legend()\n", 619 | "plt.show()" 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": 12, 625 | "metadata": {}, 626 | "outputs": [ 627 | { 628 | "name": "stdout", 629 | "output_type": "stream", 630 | "text": [ 631 | "1604/1604 [==============================] - 2s 1ms/step - loss: 1.4797e-06\n", 632 | "1.479692400607746e-06\n" 633 | ] 634 | } 635 | ], 636 | "source": [ 637 | "# Evaluation on Test Data Set\n", 638 | "print(model.evaluate(X_test,y_test))" 639 | ] 640 | }, 641 | { 642 | "cell_type": "code", 643 | "execution_count": 13, 644 | "metadata": {}, 645 | "outputs": [], 646 | "source": [ 647 | "model.save('model.h5')" 648 | ] 649 | }, 650 | { 651 | "cell_type": "code", 652 | "execution_count": 25, 653 | "metadata": {}, 654 | "outputs": [ 655 | { 656 | "name": "stderr", 657 | "output_type": "stream", 658 | "text": [ 659 | "/tmp/ipykernel_2210252/1511572212.py:20: RuntimeWarning: invalid value encountered in sqrt\n", 660 | " sqrt = np.sqrt(-2688000 * i**2 + 15772800000)\n" 661 | ] 662 | }, 663 | { 664 | "data": { 665 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnUAAAJRCAYAAAAjykF0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAACC9UlEQVR4nOzddXhW5QPG8e/zvutiwRjdPXpjjAYVxSAFRUUwEPNnByp2J7aIiIIigiKCnXSP7m4GjI1mLM/vjw0JybHtvHF/rmvX9tZ5bw6D3TvnPM9jLMtCRERERNybw+4AIiIiInLhVOpEREREPIBKnYiIiIgHUKkTERER8QAqdSIiIiIeQKVORERExAPYXuqMMcOMMbuMMUuPuy/SGPOnMWZN/ueI/PuNMeY9Y8xaY8xiY0wT+5KLiIiIuA7bSx3wBdDxpPsGAH9bllUD+Dv/NsDlQI38j/7Ax8WUUURERMSl2V7qLMuaAqSddHcXYHj+18OBrsfdP8LKMwsIN8aUKZagIiIiIi7Mx+4ApxFjWVZy/tc7gJj8r8sBW4573tb8+5KPuw9jTH/yjuQRHBwcV7t27aJNKyIiIlII5s2bt9uyrOiCvNZVS92/LMuyjDHntZaZZVlDgCEA8fHxVlJSUpFkExERESlMxphNBX2t7adfT2Pn0dOq+Z935d+/Dahw3PPK598nIiIi4tVctdRNAPrmf90XGH/c/X3yR8EmAvuOO00rIiIi4rVsP/1qjBkFtANKGmO2As8ArwJjjDG3ApuAa/Kf/gtwBbAWOAzcXOyBRURERFyQ7aXOsqzrTvPQxad4rgXcXbSJRERE5HSysrLYunUrR44csTuKWwsICKB8+fL4+voW2jZtL3UiIiLiPrZu3UpoaCiVK1fGGGN3HLdkWRapqals3bqVKlWqFNp2XfWaOhEREXFBR44cISoqSoXuAhhjiIqKKvSjnSp1IiIicl5U6C5cUexDlToRERERD6Br6kRERMRtpKamcvHFeWMpd+zYgdPpJDo6bwGGOXPm4Ofnd8bXT5o0CT8/P1q0aFHkWYubSp2IiIi4jaioKBYuXAjAs88+S0hICA8//PA5v37SpEmEhIR4ZKnT6VcRERFxa/PmzaNt27bExcVx2WWXkZycty7Be++9R926dWnQoAG9evVi48aNDB48mEGDBtGoUSOmTp1qc/LCpSN1IiIiUiBFNVzifBZ8tyyL//3vf4wfP57o6GhGjx7Nk08+ybBhw3j11VfZsGED/v7+7N27l/DwcO64447zPrrnLlTqRERExG1lZGSwdOlSOnToAEBOTg5lypQBoEGDBtxwww107dqVrl272piyeKjUiYiISIGczxG1omJZFrGxscycOfM/j/38889MmTKFH3/8kZdeeoklS5bYkLD46Jo6ERERcVv+/v6kpKT8W+qysrJYtmwZubm5bNmyhfbt2/Paa6+xb98+Dh48SGhoKAcOHLA5ddFQqRMRERG35XA4+O6773jsscdo2LAhjRo1YsaMGeTk5NC7d2/q169P48aNuffeewkPD6dTp06MGzfOIwdKGMtyhYOnRSc+Pt5KSkqyO4aIiIhHWLFiBXXq1LE7hkc41b40xsyzLCu+INvTkToRERERD6BSJyIiIuIBVOpEREREPIBKnYiIiIgHUKkTERER8QAqdSIiIiIeQKVORERE3IrT6aRRo0bUq1ePnj17cvjw4QJv66abbuK7774DoF+/fixfvvy0z500aRIzZsw47/eoXLkyu3fvLnDGc6VSJyIiIm4lMDCQhQsXsnTpUvz8/Bg8ePAJj2dnZxdou0OHDqVu3bqnfbygpa64qNSJiIiI22rdujVr165l0qRJtG7dms6dO1O3bl1ycnJ45JFHaNq0KQ0aNOCTTz4B8taKveeee6hVqxaXXHIJu3bt+ndb7dq14+iCBb/99htNmjShYcOGXHzxxWzcuJHBgwczaNCgf1ejSElJ4eqrr6Zp06Y0bdqU6dOnA5Camsqll15KbGws/fr1o7gWelCpExERkQIzxmCMOeG+Tp06YYzhxx9//Pe+IUOGYIyhf//+/963fft2jDGULVu2QO+dnZ3Nr7/+Sv369QGYP38+7777LqtXr+azzz6jRIkSzJ07l7lz5/Lpp5+yYcMGxo0bx6pVq1i+fDkjRow45ZG3lJQUbrvtNsaOHcuiRYv49ttvqVy5MnfccQcPPPAACxcupHXr1tx333088MADzJ07l7Fjx9KvXz8AnnvuOVq1asWyZcvo1q0bmzdvLtCf73z5FMu7iIiIiBSS9PR0GjVqBOQdqbv11luZMWMGCQkJVKlSBYA//viDxYsX/3u93L59+1izZg1Tpkzhuuuuw+l0UrZsWS666KL/bH/WrFm0adPm321FRkaeMsdff/11wjV4+/fv5+DBg0yZMoXvv/8egCuvvJKIiIhC+7OfiUqdiIiIFNipTi0ef4TuqP79+59wlA6gbNmyBTo1efSaupMFBwefkOv999/nsssuO+E5v/zyy3m/3+nk5uYya9YsAgICCm2bF0KnX0VERMTjXHbZZXz88cdkZWUBsHr1ag4dOkSbNm0YPXo0OTk5JCcnM3HixP+8NjExkSlTprBhwwYA0tLSAAgNDeXAgQP/Pu/SSy/l/fff//f20aLZpk0bvv76awB+/fVX9uzZUyR/xpOp1ImIiIjH6devH3Xr1qVJkybUq1eP22+/nezsbLp160aNGjWoW7cuffr0oXnz5v95bXR0NEOGDKF79+40bNiQa6+9Fsi7VnDcuHH/DpR47733SEpKokGDBtStW/ffUbjPPPMMU6ZMITY2lu+//56KFSsWy5/ZFNeIDLvEx8dbR0eyiIiIyIVZsWIFderUsTuGRzjVvjTGzLMsK74g29OROhEREREPoFInIiIi4gFU6kREROS8ePqlW8WhKPahSp2IiIics4CAAFJTU1XsLoBlWaSmphb6VCiap05ERETOWfny5dm6dSspKSl2R3FrAQEBlC9fvlC3qVInIiIi58zX1/fflRbEtej0q4iIiIgHUKkTERER8QAqdSIiIiIeQKVORERExAOo1ImIiIh4AJU6EREREQ+gUiciIiLiAVTqRERERDyASp2IiIiIB1CpExEREfEAKnUiIiIiHkClTkRERMQDqNSJiIiIeACVOhEREREPoFInIiIi4gFU6kREREQ8gEqdiIiIiAdQqRMRERHxACp1IiIiIh5ApU5ERETEA6jUiYiIiHgAlToRERERD6BSJyIiIuIBfOwOIOfGsiy2ph5i2ZY0NqUcJHnPYdIOZnAkM5utaYcoHR5IcIAvQf4+RIUEEB0WQHSJQMpHBVOlVBglgvzs/iOIiIhIEVKpc3FpB4/wU9Jm/l6ylR170wHw93VSJjyIyFB/okL8yMjOoWxkMJYFhzKyWLwpldQDGeRa1r/biQr1p0qpMOqUC6dexUhql48gwNdp1x9LRERECplLljpjTC1g9HF3VQWeBsKB24CU/PufsCzrl+JNVzyycnIZNXUt381cR2Z2Lo2qlOTaltWpXzGS8lHBGGPO+PqcXIvUA0fYsvsg63ftZ8POA6zfuZ+vpqzBApwOQ40yJWhUOYqEGqWoXS4Cp+PM2xQRERHXZazjjua4ImOME9gGNANuBg5alvXmub4+Pj7eSkpKKqp4RWLPwQyeHj2X1dv30S62LH3a1qRcVHChbPvgkSyWb9nD0s1pLNmcxspte8m1LEIDfYmvFk1C9VIk1ChFSIBvobyfiIiInDtjzDzLsuIL8lqXPFJ3kouBdZZlbTrb0SlPsP9wJg+PmEnK/iMM7NGE1nXKFOr2QwJ8SaiRV9wgr+TNW5fC3LUpzF23i4lLt+PrdBBXtSRt6pYhsVYMwf4qeCIiIq7OHUpdL2DUcbfvMcb0AZKAhyzL2nPyC4wx/YH+ABUrViyWkIXBsixe/WEhO/em80rvZtSvGFnk7xkS4Evb2LK0jS1LrmWxattepqxIZsryZGat2YWv00FC9Wja1StHYs1S+PnoOjwRERFX5NKnX40xfsB2INayrJ3GmBhgN2ABLwBlLMu65UzbcKfTr5OWbueVcQu45/JYOsVXtjVLrmWxYusepizPK3hpBzMIDfTlonrluKxReaqVLmFrPhEREU/kyadfLwfmW5a1E+DoZwBjzKfAT3YFK2yWZfHl5NVUiwnjyrhKdsfBYQyxFSKJrRBJ/w51WbhhN78v3MIv8zczfu5GqpcO49KG5WlfvxxhgZouRURExG6uXuqu47hTr8aYMpZlJeff7AYstSVVEVi6ZQ9b0w7xcOeGOFzs2kGnwxBXLZq4atHsT89k4tLt/LFwCx/9vpyhf6/konrl6Ny0ko7eiYiI2MhlS50xJhjoANx+3N2vG2MakXf6deNJj7m1uWt34XQYWtUpbXeUMwoL9KNL08p0aVqZdTv28dO8zfy9ZBu/LdxCbIUIOsdXpmWd0vg6tViJiIhIcXLpa+oKg7tcU/fQ8Jlk5+Ty7i0t7Y5y3g6kZ/Hnoi1MSNpE8p7DRIb4c0WTinSKr0R4sL/d8URERNyGJ19T5zW2px0ioXopu2MUSGigL90Tq9K1WRXmrUthwtyNjJyyhtHT19GhYXl6JFYttHn2RERE5NRU6lxAdk4uew5mUDIswO4oF8RhDE2rl6Jp9VJsTT3I2Fkb+HPRVn6dv5kWtWLo2aIadcpH2B1TRETEI6nUuYCsnFws8Ki1WMtHhXDflfXp07YmE+ZuZELSJqav2klshQh6Nq9Gs5qlXG5AiIiIiDtTqXMB/17W6IEdJyLEn77ta3FNy2r8vnAL38/awLNjkqgcHcr1ravTqk4ZrTkrIiJSCFTqXICfT95I0YzMHJuTFJ1APx+6JlShU3wlJi9LZtS0tbz8/QIqllzDda2q0za2rMqdiIjIBdC8Ey7Ax+kgNNCXPYcy7I5S5JwOBxfVL8fg29vwRPfGOIzhtR8W0v/jyfy5aCs5ubl2RxQREXFLOlLnIiJD/Ek94Pml7iinw9A2tiyt65ZhxsodjJy6ljcnLGLk1DX0almNSxqUx0dz3YmIiJwzlToXUSEqhA27Dtgdo9g5jKFVnTK0rF2aWat3MXLqGgb9tIRvpq+jT9uatKtXVgMqREREzoEOhbiIKjFhbE87xJHMbLuj2MIYQ/NaMbx/a0ueuzaeAF8nr/2wkLuGTGXW6p14+iTZIiIiF0qlzkVULRWKBazbud/uKLYyxpBYM4aP+rdmQLdGZGTn8MzoJB74fAYLN+62O56IiIjLUqlzEbEVIwFYvCnN5iSuwWEM7euV49M72nL/VfVJOXCEx76czYCvZrNy216744mIiLgclToXUSLIjyqlQlm0MdXuKC7Fx+ng8sYV+fzudtx+aV3W79zPfcOm8/yYJLamHrQ7noiIiMtQqXMhDStHsWxLGhlZnjtfXUH5+Tjp3qwKX9zTnj5tazJ/w276D57CB78uZa8XTAUjIiJyNip1LqRp9VJkZucyb32K3VFcVpC/Dze0qcHnd7fn8sYV+HneZm7+YBLfTFurMiwiIl5Npc6FNKwcRUiAD9NX7rA7isuLCPHnf1fU55M72tCgchSfT1zFLR9N4s9FW8nVSFkREfFCKnUuxNfpoHnN0sxavZOsHK2scC4qlgzhuWvjeaNPIpHB/rw5YRH3fDqNBRs0UlZERLyLSp2LaVWnNAePZDNvnU7Bno8GlaJ499aWPNa1EQePZDHgq9k8NWoOm3drMIWIiHgHlToXE18tmohgf35fuMXuKG7HYQwX1S/H0Lva0u/i2izdsoc7PpnCJ38u59CRLLvjiYiIFCmVOhfj43RwSYNyzFq9i7SDR+yO45b8fJz0bFGNYXe1o0PD8oybtYGbP5zErws2k5Or6+1ERMQzqdS5oMsaVSDXsvhr8Ta7o7i1iBB/HriqAe/3a0X5qGDe+WkJ9342jaWbNcGziIh4HpU6F1ShZAj1Kkbyy3wdWSoMNcqU4K2+zRnQrRF7D2Xy0PCZvPL9AnbtS7c7moiISKFRqXNRXZpWJnnPYWav2Wl3FI9g8pcd++yutlzfujozVu2g38eTGTlljea3ExERj6BS56Ja1o4hpkQg38/aYHcUjxLg50PfdrX49M62JFSPZsTk1dw2eDKzVqs8i4iIe1Opc1FOh4OuCZVZsjmN1dv32h3H45QOD2Jgjzheu7EZ/j5OnhmdxDPfzGXHnsN2RxMRESkQlToXdlnjCgT5+TBWR+uKTKPKJfm4f2v6XVKbhRtTuW3wZL6euobMbJ2SFRER96JS58KC/X25Iq4iU5ZvZ1vqIbvjeCwfp4Oezasx9K62NKtRiuGTVnPHJ1M1AbSIiLgVlToX1yOxKr5OB19PW2N3FI8XHRbIwB5xvHR9AhYWT3w9hxe/m0/Kfo2SFRER16dS5+IiQvy5Mr4S/yzZpqN1xSS+WjSf3N6GPm1rMnvNTvp9NJlvZ64jW+vxioiIC1OpcwM9m+cdrRs1ba3dUbyGn4+TG9rU4NM72tKwchRD/1rJXZ9O1cTFIiLislTq3EBkSABXxlfi7yVbdbSumJWOCOL5Xk159pp4jmTm8NDwmbz78xIOpGstWRERcS0qdW7imubV8PVxMnzSKrujeKXmtWIYckcbrk6swm8LNnPbx5OZvGw7lqUVP0RExDWo1LmJiBB/eiRWZfLyZFZu22t3HK8U4OdD/w51ee/WVpQMC+Dl7xfw9Ogkdu7V3HYiImI/lTo30qN5VcKD/Rj61wodIbJRjTIlePeWFtzeoQ6LN6Zy2+ApjJ21npxcDaQQERH7qNS5kSB/H25sW5Mlm9OYvWaX3XG8mtPhoHtiVYbc0YaGlaMY8ucK7v1sOmuS99kdTUREvJRKnZvp2KgC5aOCGfrXCh0ZcgEx4UE8f208T17dhLSDGdz72TQ++XM56ZnZdkcTEREvo1LnZnycDm69qDZbUg/xy/wtdscRwBhDm7pl+PTOtnRsXJHvZ22g/+ApzF2ro6kiIlJ8VOrcUPNaMTSoFMnwSavYfzjT7jiSLyTAl/uurM/bNzUnwNfJwFFzeWP8Qvan6+9IRESKnkqdGzLGcHfHehw6ks0XmuLE5cRWiOTD21pxfevqTFy6nf4fT2HaimS7Y4mIiIdTqXNTlUuF0iWhMr/M26yL812Qn4+Tvu1q8f6trYgK9eeF7+bz4nfz2XMww+5oIiLioVTq3NiNbWpQItiPD39bSq6mOHFJ1UqH8e4tLbm5fS1mrd7JbYMn88+SbZqSRkRECp1KnRsLDvCl38V1WLF1L38v3mZ3HDkNH6eDXq2q89FtrSgfFcxrPyzkmdFJ7N5/xO5oIiLiQVTq3NzFDcpRp3w4Q/9eofVIXVzF6FDe6tuCOy6ty8KNqdw2eDK/Ltiso3YiIlIoVOrcnMMY7ulYj/2Hs/js7xV2x5GzcDoM3ZpV4ZPb21CjTAne+WkJA0bOZsceLTUmIiIXRqXOA1QvU4LuiVX4dcEWFm9KtTuOnIMyEUG81rsZ911Zn9Xb9tH/kymMn7tR10aKiEiBqdR5iBvb1KB0eCDv/ryEzOwcu+PIOTDGcEWTinxyRxsaVIrko9+W8fhXs9mxV0ftRETk/KnUeYgAPx/uvaI+W1MPMWraWrvjyHkoVSKQF3o15YGr6rN6+z7u+GSKrrUTEZHzplLnQeKqRXNx/XKMmb6OjbsO2B1HzoMxho6NKzL49tbUKhvOOz8t4alv5mqErIiInDOVOg/Tv0Mdgvx9eOfnxbo+yw3FhAfxSu9m3N0xlsWb0rj9k8n8tXirjtqJiMhZqdR5mPBgf/p3qMuKrXuZMHej3XGkABzG0LlpZT7u35pK0aG8MX4Rz42Zp9UoRETkjFTqPNAlDcqRUD2aYX+vZGvqQbvjSAGViwzmjT7Nue2SOiStS6H/4MlMWa41ZEVE5NRU6jyQMYb7r2qAr4+DtyYsJidXp+7cldNh6NG8Kh/d1orSEUG8NHY+L4+dz/7DmXZHExERF6NS56GiQgO467JYlm/dw/ez19sdRy5QxehQ3rm5BX3b1WT6yh30HzyFWat32h1LRERciEqdB7uofjla1Iph+MTVbErRaFh353Q4uL51Dd67tRURIf48MzqJQT8tJj0z2+5oIiLiAlTqPJgxhnuvqE+gn5M3JywiJzfX7khSCKqVDuO9W1tybYtq/L5gC3cOmcryrXvsjiUiIjZTqfNwESH+3HN5PVZv38fo6evsjiOFxNfp4JaLa/NG3+bk5lo89MUMhk9aRXaOiruIiLdSqfMCbWPL0qZuGUZOWcO6HfvsjiOFqH7FSD6+vTUX1y/P11PX8sDnM9iyWyOeRUS8kUqdl7jn8nqEBfnx6riFHMnS2rCeJNjfl4e7NGRgjyYk7z3M3Z9O5cekjZqwWETEy6jUeYkSQX483KUhm3cfZOhfK+yOI0WgdZ0yfHJ7G+pXiuKDX5fx1DdzST2gZcZERLyFSp0XiasaTffEKvyYtEnTYXioqNAAXryuKXd3jGXRxlTu+GQK01fusDuWiIgUA5cudcaYjcaYJcaYhcaYpPz7Io0xfxpj1uR/jrA7pzu5uX0tqsaE8faPi0k7qKM4nsjkLzP24W2tiQkP4vlv5/HWhEUcysiyO5qIiBQhly51+dpbltXIsqz4/NsDgL8ty6oB/J1/W86Rn4+TAd0akZ6ZzVsTFpOr6648VsWSIbxzcwuub1WdvxZv5c4hU1m6Oc3uWCIiUkTcodSdrAswPP/r4UBX+6K4p0rRofTvkLee6IS5G+2OI0XIx+mgb/tavNm3OQ5jeGTETEZMWq05C0VEPJCrlzoL+MMYM88Y0z//vhjLso6uar4DiDn5RcaY/saYJGNMUkpKSnFldStXxVWiWY1SDP1rJRt27rc7jhSx2AqRfHRb3tQnI6eu4eHhs9ix97DdsUREpBC5eqlrZVlWE+By4G5jTJvjH7Ty5mz4z/lDy7KGWJYVb1lWfHR0dDFFdS/GGB7s1ICQAF9eGbdA05x4gSB/Hx7u0pAB3RqxMeUAdw6ZyqSl2+2OJSIihcSlS51lWdvyP+8CxgEJwE5jTBmA/M+77Evo3sKD/Xmka0M2pxzk49+W2R1Hikn7euX4uH9rKkeH8sq4Bbw5fhGHM7R+rIiIu3PZUmeMCTbGhB79GrgUWApMAPrmP60vMN6ehJ4hrmo0vVpV57eFW/h78Va740gxKR0exJt9E+ndpgZ/L9nKXZ9OZeW2vXbHEhGRC+CypY68a+WmGWMWAXOAny3L+g14FehgjFkDXJJ/Wy7AjW1rUK9iJO/9spTNWmLKazgdDm5sW5M3+jQnJ9fiwS9mMHr6WnJyNSJaRMQdGU9fSig+Pt5KSkqyO4bL273/CHd9OpXIEH/evaUl/r5OuyNJMTp4JIt3f17ClOXJNKwcxaNdGlEyLOC8t7MUeBGoAzxT2CFFRLyAMWbecdO4nRdXPlInxahkWACPdGnIhl0HGPzHcrvjSDELCfDlie6NebBTA1Zt28sdQ6YwowArUaxLTWX0uHF8P3lyEaQUEZEzUamTfzWtXoprW1Tjl/mbmbh0m91xpJgZY7isUQU+vK0VpcODeO7bebz785LzGhldKioKunUjoG3bIkwqIiKnolInJ+jbviaxFSJ49+clbE3V9XXeqHxUCINubkHP5lX5Zf5m/jd0GuvPcS7DoydsM4ounoiInIZKnZzA6XAwoFtjfJwOXhq7gAzNX+eVfJ0O+l1Sh1duaMbBI1nc+9l0fpq3ibNdg+vMzoZt2zi4VSOpRUSKm0qd/EepEoE82qUR63fu54Nfl571B7l4riZVS/Jx/9Y0qBzF+78s5aWxCzh0JOu0z9+9aROUL8+mdu2KL6SIiAAqdXIaCTVKcX3r6vyxaCu/LthidxyxUXiwPy9e15RbL67N9JU7uOvTqazavveUzw0NCIBy5TAx/1m9T0REiphKnZxW7zY1iasWzUe/LTvtD3HxDg5juKZFNd66qTm5Fjz4+QzGzlr/n6O4lcuVg61bCZs+3aakIiLeS6VOTsvpMAzo2ojIEH9e/G4++w5n2h1JbFa3fAQf3daaZjVKMeTPFTw9OumE74uj/6Hk2hNPRMSrqdTJGYUF+TGwRxP2HMzg1XELtNqAEBroy1M947irYywL1u/mriFTWbI5DYCjK8j62hdPRMRrqdTJWdUsG87dl8cyf/1uvpy82u444gKMMXRpWpl3bm6Bn6+DR0fM5Oupa1i/YSPUr8/+Ll3sjigi4nV87A4g7uHyxhVZuXUvo6atpXa5cBJr6kJ4T5ADXMiCcNXLlODDfq15/5clDJ+0morRAfht3EZ2dvbZXywiIoVKR+rknN19eSw1ypTg9R8Wsi3tkN1x5AItAOqRt17rhQjy9+HRro14sFMDkvdk0vypkVT76MtCSCgiIudDpU7OmZ+Pk4E9muBwGF74dh5HMnU0xp29BawELgZWXeC2ji4x9nC/VmSGBVNp2k6G/b2SnFwNmRARKS4qdXJeSocHMaBbYzbuOsBbPy7WxMRubChwCbALaH/oED8uWHDB24yMDmX2rS051LgCo2es49EvZ7N7/5EL3q6IiJydSp2ct/hq0dxycW2mLE9m9PR1dseRAgoAfgBaHDpE8uWX06VtW36aO/eCtjl/8WJyBz5BSvoqHuvaiLXJ+7jr06nMX7+7MCKLiMgZqNRJgfRsXpV2sWX5YuIqZq/ZaXccKaBg4OeAACLLlsUKC+PusDCSL2B7ixctgtdf5+D48VxUvxzv39qS8GA/nhg5m68mr9aUOCIiRUilTgrEGMMDnRpQrXQYr45byObdB+2OJAUU7nSy4ssviZ09m821anEJkFLAbUU3agQvv0y1nj0BqBgdynu3tOTiBuX4csoaBo6aw95DGYUVXUREjqNSJwUW4OvkmWvi8XU6eG50EgfPsNC7uLZSvr5MKleOWGA50Oynn1i8adN5byesfn14/HFij5unLsDPh4c7N+SBq+qzdHMad306laX5kxWLiEjhUamTC1KqRCBP9Ywjee9hXnOlFScyUmFix7zPck5KAn8BZf/8kw1du5J4xRVsO3Dg2BPOYZ/uzf8cftL9xhg6Nq7IOze3xN/XySMjZvHtjHUaaCMiUohU6uSC1a8Yyd0dY5mzNoXhEy90coxCsv4LSP4d1g+3O4lbKQ383rQpvrVqkd6lC9cGB/PvjITnsE9XJCXB7NkEHF8Gj1OtdBgf9GtFy9oxDP17Jc+OTuJAuo7wiogUBpU6KRRXxlXiiiYVGT1jHROXbrM3jGXBykF5X68alHdbzlm98HAWzJlD+ZdfZrrDQRfgyDnu078eewwSEzk4c+Zptx/s78uTVzfhrsvqkrQuhbuHTmXV9r2F/wcREfEyKnVSaO7qGEtshQgG/biYtcn77AuSMhWy8t8/cy+kTLMvi5uKDQ7mbyAG+PvAAWIfvZ+U/fnH7M6wT521a0NcHHUrVTrj9o0xdEmowls3Ncey4KEvZjJh7kadjhURuQAqdVJofJ0OnuoRR1iQH8+OSSLtoE2Tzq58B7LzC0j2oWNHmOS81AT+Bnx792b9m+/ReExtso3zjPs06MMPISmJxFq1zuk9apeL4MPbWtG4akk+/G0ZL3+/gEMZOh0rIlIQxtN/M46Pj7eSkpLsjuFV1ibv48HhM6lSKpTXb0zE3/dClow/i8ldYNuEE+9z+EFu5ulvA5TrDG3HF10ud3bSPv3xSB26fRFBzhdf8uumu+iY/Psp96lVrjPBbceTDuwDws7jLXMti29nrOeLiasoExHEUz2aUCXmfLYgIuIZjDHzLMuKL8hrdaROCl31MiV4rGsjVm7by9tFvZRYw5chqCI4Ao7dd3KBO6HgBUBwpbzXyamdtE87Baxg2pM5DN31Ul6hg1Pu07R6z5Oek0Mo51foABzGcG3Larx+YzPSM7O5b9h0/l68tVD+OCIi3kKlTopEy9qlueWiWkxatp2RU9cW3RuFx8JVy6F8Z3AGnfm5ziAo3wWuXJb3Ojm1U+zTxNTZ3Lp+2H+fe9w+HT1nO/j7w3XXFfit61eK4sPbWlGzbDivj1/EB78uJSsnt8DbExHxJip1UmSuaVGNDg3K8+Xk1Uxatr3o3sgnGFqNhiZvgcOftSHV+CemPckBpY89x+Gf93irb/KeL2d20j49pZP26dLkZMjNJSLswk6bRoYE8GrvZlydWIUfkzbxyPCZpOxPv6Btioh4A5U6KTLGGO69sh71Kkby1oRFrNy2t2jfMKIJOPwZlHUJF0e+xdvlbzv2mNMfIuOK9v09Uf4+PSWnP1ZkHD8BFlD+llvg8GGuevXVC35bH6eD/h3qMvDqJmxMOcDdn05j4cbdF7xdERFPplInRcrPx8nTPeOIDPHn2dFJ7NpXhEdc0pLAymLCBzOgSRMOT5sPmLzHcrPyHpfzk79P85j807HH9ukjDl86AQOBVQABATSMiCi0t29dtwzv3dKSEkF+PP7VbMZoFQoRkdNSqZMiVyLIj+d7NSUjO4dnRieRnpldNG+0ayrkpEOVqtCgAbUbXg1BFfIu5M9Jz3tczs/RfeoIgOCK0GLkCfu05Y5/cAIvA//kv+TcJjM5dxWjQ3n3lpa0rF2Gz/5eyQvfzde0JyIip6BSJ8WiUnQoT17dhI279vPquIVFs0Zs6mwwToI++AgWLeKii28+dsG/ceY9Lucnf5/+O8CkQtcT9mm3NR/yJWAsi63Ll0P37tTILfyBDUH+Pjx5dWP6d6jDzFU7uXfodDbuOvVSZCIi3kqlTopNfLVo7rgsllmrd/L5PysL/w1K1MFKGMLWoLIAlIdjF/wnDIGw2oX/np6uRJ28fXf8AJOT9ul1wP1r18Kll8K4cTz46adFEsUYw9WJVXn9xmYcysjm3mHT7V+STkTEhWjyYSl2H/y6lB+TNnH/VfW5vHHFQt32XssiAgg2hgP8e/WXFLFxGRl0//JL+PFH+OYbhgYGcmsRvl/qgSO8NHY+y7bsoWtCZfpdUgdfp35HFRH3p8mHxa3ceVld4qpF897PS5m3LqVQt/3PokUQFIR1+eUqdMVosb8/9OtHmx9+gMBAbgO+LsL3iwoN4PUbE+maUJkf5mzksS9nkXrApmXpRERchEqdFDunw8HAq5tQuVQoL343n3U79hfattfu2gVHjuCXpQvpi9O8/M+3G8NLgJWbS+///Y/bP/nkxCdmpMLEjnmfL5CP08Gdl8XyeLfGrN2xn7s/ncbiTRe+XRERd6VSJ7YI8vfh+V7xBPn78PQ3cwttctnyHTrA/v20GTWqULYnZ5ebm8vft98OH3xA49xcngB6T5mC9cEHDHnwQb5OOe5o7PovIPl3WD+80N6/Xb2yvHdLS4L9fXjsy9mMm71B056IiFdSqRPbRIcF8sJ1TTmckc1To+YWyjQVu4yB0FAqRkcXQkI5FzNWr+bwkCGY116jliPvv5QR7drR8qWX4IcfuCU6mr8ALAtWDsp70apBebcLSeVSobzXryXNapRi8B/LeWP8IjKycgpt+yIi7kClTmxVNSaMJ3s0YVPKQV4au4DsC1znc1f+51IXHk3O0froaPjwQ6o+8MC//6EYYOoTT3BXhw5kAF2Af3YlQda+vCdk7oWUaYWaI9jfl6eviePGtjX5e8k2HvxiBjv3Hi7U9xARcWUqdWK7+GrR3HdlPeatS+H9X5de0KmzSe++C717s3/69EJMKGcyPyoK7rqL3g8+eML9BngfuBk4vHgxlyRczRvL86abIfvQsaN2hchhDL3b1OC5a+PZvucw//tsupYXExGvoVInLqFj44pc16o6vy3YwjfT1xV4OxsnTYKRI7F27Ci8cHJGk/M/tz3hzi7wtcHxteHTUU7qfPQk1uYtPLm4FgvDGwIWbP8ZvjYnfkzuUiiZEmvG8P6tR5cXm8P3s9brOjsR8XgqdeIy+rarSft6Zfli4ir+WVKwSWVLPfYYDB9O04SEQk4np7JkyxYWvvMOPkuXknj8Aw1fhqCK4AjAaeWysM2vNHr9LgI//4QjzoC85+RmHnu+IwCCK+W9rpCUjwrh3Vta0rxmKT75cwWv/7CQI7rOTkQ8mEqduAxjDA92akD9ipG8/eNilhRgeoqcxETo04faFSoUQUI52ZDff4cHHqDEs88SePwD4bHHlhNzBuFHDrMqDGXW1ItIPHm5NmfQsWXIwmMLNV+Qvw8De8bRt11NJi7dzkNfzGCHrrMTEQ+lUicuxc/HydPXxBETHsizY+axeffB83r9nvzPEYUfTU4hJTYWbryR5p06/ffBo8uJNXkLHP7452ZSZ/9Jy8M5/PMeP34ZskLmMIbrW9fguV7xJO85zP+GTmPhBl1nJyKeR6VOXE5YoB8vXZeAj9Mw8Os557VSQMrnn8O33xKWo9NsxWFB8+YwYgSP9O17+idFNMkrb6fi9IfIuKIJd5JmNWJ4/9ZWhAf78/jI2YzVdXYi4mFU6sQllY4I4oVeTdl3OJOBo+Zy6MjZ57DLyMkh85Zb4JprCDVaJKyorQVWA+FAizM9MS0JrKN/fybvdOvRRdxys/IeLyblooJ595aWtKhVmiF/ruA1XWcnIh5EpU5cVs2y4Qzs0YRNKQd4/tt5ZGaf+YdvWnY23HILvjfcgNOhb+2i9sYvv8DMmXTIycHnTE/cNRVy0vMHQ1SEFiMhqELe7Zz0vMeLUZC/DwN7NOGm9rWYtHQ7D34+gx17dJ2diLg//eQTl9a0eike7NSAhRtTeXP8InLPcLosw98fPvuMMl99VYwJvZNlWXx1333QogU1Z80685NTZ4NxHhsMUaHrsUEUxpn3eDEzxnBdq+q8cF1Tdu47zD2fTWP+el1nJyLuTaVOXN4lDcpz68W1mbw8mSF/rjjtdVAH8j+HFl80r5WWlcWRq66CJk24JzHxzE8uUQcShpw4GOLoIIqEIRBWu+gDn0bT6qV479ZWRIUE8OTXWjdWRNyb8fT/wOLj462kpOK7ZkeKhmVZDP5jOT/M2Ui/S2rTs3m1/zxnSmYmbffto2loKHMCAmxI6T2+Bm4AWgKFu9iXPQ5nZPPG+IXMWLWTyxqV557L6+Hn47Q7loh4IWPMPMuy4gvyWh2pE7dgjOH2S+vSpm4Zhv61kr8Xb/3Pc+bOnAmlSrG6QwcbEnqXMfmfr7U1ReEJ8vfhqZ5x3NC6Br8v3MpjX84m7eC5j7oWEXEFKnXiNhzG8EiXhjSsHMVbPy5m3rqUEx5Pz82FyEgCIjRLXVFalZzMzz/9BEeOcLXdYQqRwxj6tKvJwKubsG7nfv732XTWJO+zO5aIyDlTqRO34ufj5JmecVQsGcIL38074YdutfbtITWVdhMm2JjQg2WkwsSOPD/yC7I7daJUv36UtTtTEWhdtwyDbmqOwxge/GIGk5ZutzuSiMg5UakTtxMc4MtL1ycQFujHwFFz2J52CID0/McDT/9SuRDrv4Dk31lWwh8aN+aqU60i4SGqlS7B+7e2pGbZcF4Zt4Bh/6w848hrERFXoFInbikqNICXrk8gN9fiia/nsPdQxr+lTkMkioBlwcpBWEClq+rhP28eL11zjd2pilR4sD+v9m7GFU0qMnr6Op4dncShjLNPgi0iYheVOnFbFUqG8HyvpqQdOMLAUXOZOWoMtGjB8nfesTua50mZCln7MMD4qVeTsnsmpb1g1Q5fp4P7rqzPPZfXI2ldCvcPm8G2/CPDIiKuRqVO3Fqd8hEM7BHH+p372b0eHHOTOLx5s92xPM/KdyA7v8xkHyJ0xZu2xiluneIr8coNzdh7KIN7P5uuiYpFxCVpnjrxCP8s2cZrPyxkZ0lfWl1SlQ9qVLc7kvua3AW2nTTYxOEHuZmnvw1QrjO0HV/0+Wy0Y89hnh2TxKaUA/TvUJeuCZUxXnDEUkSKj+apE693Uf1yVLisLjG7sziw4pBWBbgQDV+GoIp5a7MedXKBO6HgBUBwpbzXebjSEUEMurkFzWvGMPiP5Qz6afFZ1yQWESkuKnXiMd5PqEK3NjXYsWgrn/51+uXE5CzCY4+tzeoMOvNznUHH1nQNjy2efDYL9PNhYM84erfRRMUi4lpcstQZYyoYYyYaY5YbY5YZY+7Lv/9ZY8w2Y8zC/I8r7M4qriMQuL1NDTo3rcTYWRsYPX2d3ZHc19G1WZu8BQ7/Uz/H4Z/3+PFrunoJhzHc2LYmA3scm6h43Q5NVCwi9nLJUgdkAw9ZllUXSATuNsbUzX9skGVZjfI/frEvorgiYwx3XhZL+3pl+XziKn6Zr0ETFySiyelLndMfIuOKN4+LaV0nb6JiAzzwxUxmrNxhdyQR8WIuWeosy0q2LGt+/tcHgBVAOXtTibtwGMPDnRuSUD2a935ewpTlyXZHcl9pSWAdnZvN5J+OzR8YkJuV97iXq1a6BO/d2pIqpUJ5/tt5jJ6+Tqf+RcQWLlnqjmeMqQw0Bmbn33WPMWaxMWaYMeaUi3waY/obY5KMMUkpKSmneop4OB+ngyd7xFG3QgSvjVvwn3Vi5Rztmgo56fmDISpCi5EQVCHvdk563uNCZEgAr9+YSNvYsgz7ZyVvTlikARQiUuxcutQZY0KAscD9lmXtBz4GqgGNgGTgrVO9zrKsIZZlxVuWFR8dHV1cccXFBPg6eb5XUyqUDOG5b+exYuseuyO5n9TZYJzHBkNU6HpsEIVx5j0uAPj7OhnQrRF92tbkr8XbGPDVbPYeyrA7loh4EZctdcYYX/IK3UjLsr4HsCxrp2VZOZZl5QKfAgl2ZhTXFxLgy8s3JBAZ4s/AUXPZuOuA3ZHcS4k6kDDkxMEQRwdRJAyBsNr25nMxxhhuaFODJ69uwtrkfdw7bLq+50Sk2Ljk5MMmbzbP4UCaZVn3H3d/GcuykvO/fgBoZllWrzNtS5MPC+RNGvvAFzMwBt7u24LSEWeZqkPkAq3evpdnRieRnpnN490b06xGjN2RRMQNeOLkwy2BG4GLTpq+5HVjzBJjzGKgPfCArSnFbZSOCOKVG5qRkZXLgJGzST2gecWkaNUsG877t7aiXGQwz45O4vtZ6zWAQkSKlEseqStMOlInx1u5bQ8DvppNdFggb/RJJDz4NNN1iBSSI5nZvDF+EdNW7uDyxhW4+/J6+Dpd9fdpEbGbJx6pEykStctF8HyvpuzYe5gnv57DwSNZZ3+RyAUI8PPhyR5NuK5VdX5dsIUnRs5m/+HMs79QROQ8qdSJ12lQKYqne8axcdcBnho1lyOZ2XZHEg/nMIab2tfisa6NWLF1L/cOm87m3QftjiUiHkalTrxS0+qlGNC9MSu37eGZMUmaU0yKxUX1y/F6n0TSM7O5f9h0zZ8oIoVKpU68Vus6ZXiwU0MWbkjlpe/mk52Ta3ck8QJ1y0fw3i0tKVUikIGj5vJj0ka7I4mIh1CpE6/WoWF57rm8HrPW7OL1HxaSk+vZA4fENcSEB/H2TS1IqB7NB78uY/Afy/W9JyIXzMfuACJ26xRfiSOZ2Qz9eyWBfj7cd1V9HMbYHUs8XJC/D09fE8+nf61g3OwNJO85zIBujQj003/LIlIwOlInAvRsUY3rW1fnt4Vb+OSP5ZpPTIqF02G449K63N0xljlrdvLw8JmaQ1FECkylTiRfn7Y16d6sCj/M2cjwSavtjiNepHPTyjx3bVO2pR3i3mHTWbdjv92RRMQNqdSJ5DPG0L9DHS5vXIFR09Yyevo6uyOJF0moUYq3+rYACx4aPoM5a3bZHUlE3IxKnchxjDH874r6tK9XlmH/rGTC3I12RxIvUq10GO/e0pJykcE8M3quvv9E5Lyo1ImcxOkwPNy5IS1qxfDhb8v4dcFmuyOJFykZFsCbfZuTUL0UH/62jI9/X6aRsSJyTlTqRE7Bx+ng8e6NaVo9mnd/WsKfi7baHUm8SKBf3sjYbvnXeD4/Jol0rXwiImehUidyGn4+Tp7uGUejKiV5+8dFTFy6ze5I4kVOGBm7dpdGxorIWanUiZyBn4+TZ6+Np17FSF7/YRFTlyfbHUm8jEbGisi5UqkTOYsAXyfP92pKnfLhvDJuATNW7bA7kniZf0fGopGxInJ6KnUi5yDQz4cXrmtKjTIleOm7+fqhKsWuWukw3jtuZOx4jYwVkZOo1Imco2B/X166PoEqMWE8/+085q1LsTuSeJmo0GMjYz/6TWvGisiJVOpEzkNIgC8v35BAhZIhPDsmiYUbdtsdSbzM0ZGxXRMqM272Bl4aO5+MrBy7Y4mIC1CpEzlPYYF+vNq7GWUjgnl6dBJLNqfZHUm8jNNhuPOyWG6/tC4zVu7gsa9mse9wpt2xRMRmKnUiBVAiKK/YlQoL4KlRc1i+dY/dkcQLdW9WhSd7NGHdjv088PkMtqcdsjuSiNhIpU6kgCJC/HntxkQiQvx58us5rNq+1+5I4oVa1ynDq72bcSA9k/s/n8HKbfoFQ8RbqdSJXICo0ABe651IWKAvT4yczdrkfXZHEi8UWyGSQTe3IMjfh0dHzNK0OyJeSqVO5AKVKhHIazcmEuTvy4CRszU5rNiifFQI79zcgsqlwnh+zDxNeSLihVTqRApB6fAgXr8xkQBfJ499NUvFTmwRHuzP630SSawZw0e/LePTv1aQa2nKExFvoVInUkjKRKjYif0CfJ081TOOTvGV+G7mel75fgGZ2ZryRMQbqNSJFKKykcG80af5ccVO19hJ8XM6DHd3jOW2S+owZXkyA76azf50TXki4ulU6kQKWZmIoOOK3WwVO7GFMYYezavyRPfGrN6+jwc+n8GOPYftjiUiRUilTqQIqNiJq2gbW5ZXejdj76FM7vt8Oqs19Y6Ix1KpEykiR4tdoJ8Pj32l6U7EPvUr5k154u/r5OERs5i1eqfdkUSkCKjUiRSho4MnAv18GKB57MRGFUuG8O7NLalUMoTnxiTxy/zNdkcSkUKmUidSxMpEBPGGip24gIgQf97ok0h8tWje/XkJX05ejaUpT0Q8hkqdSDEofVyx06lYsVOAnw/PXBPPpQ3L89WUNbz78xJycnPtjiUihUClTqSYHC12Qf4qdmIvH6eDBzs14LpW1fl1wRae/3Y+R7I0l52Iu1OpEylGR4tdsIqd2MwYw03ta3HP5bHMXr2TASOmsf+PrpCRanc0ESkglTqRYlY6IojX+xwrdmtU7MRGneIrM7BHE9bu2M+D8zuyc8lIuyOJSAGp1InYoHT4sWI34KtZrNLcYWKjVrVL80qlt9iTE8H9f0axXvMqirgllToRm5QOD+KNPomEBvox4KvZLNuSZnck8VYpU6nvN4+3yj6GgxweGj6NRRt1GlbE3ajUidgoJr/YRQT788TIOSzZpB+kYoOV70D2ISr7b2JQuYcp6bOHJ7+ew+Rl2+1OJiLnwXj6HEXx8fFWUlKS3TFEzij1wBEGfDWbnXsP81yvpjSuUtLuSOKpJneBbRNOvM/hB7mZ/97cb0Xy7PbHWH6kDneWHEKXEj9Buc7QdnwxhxXxPsaYeZZlxRfktTpSJ+ICokIDeKNPImUjg3n6m7kkrUuxO5J4qoYvQ1BFcAQcu++4QgcQZtJ4pcxTNA+ezUe77+Cz/fdiNXipmIOKyPlSqRNxEeHB/rx+YyIVS4bw7Ogkrc8pRSM8Fq5aDuU7gzPotE/zd2QysOw7XFl2FWNSLuXNKdlk52iSYhFXplIn4kLCgvx4tXciVWJCef7beUxbkWx3JPFEPsHQajQ0eQsc/mwLLMvLdR/niyp9jz3H4Y8z7k3+d8sD9G1Xk78Wb+Pp0UmkZ2bbl1tEzkilTsTFhAb68uoNzahVNpyXxi5gki5Wl6IS0QQc/izJKs2TmV15x3Hlscec/hAZhzGG61vX4P6r6rNgfQqPjpjF3kMZ9mUWkdNSqRNxQcEBvrx0fQKxFSJ4bdwC/lq81e5I4onSksDKYsqMPdCsGdvf/xIweY/lZuU9nu/yxhV55pp4NqYc4KEvZrJrX7o9mUXktFTqRFxUkL8PL17XlAaVo3hz/CJ+X7jF7kjiaXZNhZx0okoGQXw8flXqQlCFvEEUOel5jx8nsWYMr9zQjD2HMnjg8xlsSjlgU3ARORWVOhEXFuDnw/PXNiWuWjRv/7iYH5M22R1JPEnqbDBOWl3VFebOpewLrx4bRGGceY+fpF7FSN7s25xcy+Kh4TNZsXVP8ecWkVNSqRNxcf6+Tp65Jo7EGqX44NeljJu9we5I4ilK1IGEIQQ0ehGADDg2iCJhCITVPuXLqsaE8fZNLQgJ8OWxr2ZrCh4RF6HJh0XcRFZOLq9+v4BpK3fQ7+La9GxRze5I4iFWAbWBmvlfn6u0g0d48uu5bE45wCNdGtGuXtmiCSjiRYp88mFjTEtjTHD+172NMW8bYyoV5A1FpGB8nQ6euLox7WLLMvTvlXw5eTWe/kuZFI+0TZugTBnWN2hwXq+LDAngzT6J1CkfwavjFjBh7saiCSgi5+RcT79+DBw2xjQEHgLWASOKLJWInJLT4eDRro24rFF5vpqyhiF/rXCvYpeRChM75n0WlxHucMCOHeTsOf/r446O1G5WM4YPf1umXzZEbHSupS7byvtX2gX4wLKsD4HQooslIqfjdBjuv6oBXRMq8/2sDbz3y1Jy3eWH6PovIPl3WD/c7iRynBply2J27MBaupSsArze39fJ0z2b0KFh3i8bH/62jJxcN/meFPEgPuf4vAPGmMeB3kAbY4wD8C26WCJyJg5juOPSugT4Ovlm+jqOZGbzcJeGOB0uPPbJsmDloLyvVw2C2g+AMfZmEgB8nE6iYmLYDaQCpQuwDafDwUOdGlAiyI/vZq5n/+FMHunaCF+nC39PiniYc/3Xdi15A6NutSxrB1AeeKPIUonIWRljuPmi2tzcvhb/LN3Oi9/NJzM7x+5Yp5cyFbL25X2duRdSptkaR04Uk//5QlYcNsZw2yV16HdxbSYvT+aZb+ZqWTGRYnROpc6yrB2WZb1tWdbU/NubLcv695o6Y8zMogooImfWq1V17rqsLjNW7eTZ0UkcyXLRYrfyHcg+lPd19qFjR+3EJWS+/Tb06MHsZcsueFs9W1TjwU4NWLBhNwO+ms3+w5mFkFBEzqawjosHFNJ2RKQAuiRU+feH6JNfz+FQRkGujCpEk7vA1+bEj+0/A0evs7Lybp/8nMld7Ezt1Q5PmgRjx7J49epC2d5ljSrwVM841u3Yz0PDtayYSHEorFKnK2JFbHZZowo81q0xK7buyTs6km7j0ZGGL0NQxbzlpo7KPSnP8bcdARBcKe91Yos2994Lo0YRkJBQaNtsUas0L9+QwO4DR3jwixls2X2w0LYtIv+lK1hFPEi72LI83TOODTsP8MjwWaQdPGJPkPDYY8tNOYPO/FxnEJTvAlcuy3ud2KLtJZdAr17sLVeuULfboFIUb9yYSGZ2Lg8Nn8m6HfsKdfsickxhlToNYRNxEYk1Y3jhuqbs2HuYh4fPsu+019Hlppq8BQ7/Uz/H4Z/3eKtv8p4vtqmY/7koFqGrXqYEb/Vtjp+Pg0dGzGLZlrQieBcROWOpM8b8cY7bubEQsohIIWlcpSQv35DA3kMZPDR8JtvSDtkXJqLJ6Uud0x8i44o3j5xSmYMH4dtvWfj550Wy/QolQ3j7phaEB/vz+Mg5zFuv9WJFCtvZjtRFn8tGLMtaWghZzpkxpqMxZpUxZq0xZkBxvreIu4itEMlrNybmzWE3fCYbdx2wJ0haElhHB26Y/NOx+Qf3c7PyHhfbRR44ANdcQ9ojj1BUJ+1LlQjkrb7NKRsRxDPfJDFtRXIRvZOIdzpbqSthjOl+uo9iSXgSY4wT+BC4HKgLXGeMqWtHFhFXV6NMCd7s2xyAR0bMZE2yDdcz7ZoKOen5gyEqQouREFQh73ZOet7jYrsKpUsT3LUr3HILq7KLbm65iBB/3ujTnOplwnhp7Hz+XLS1yN5LxNuctdQBVwGdTvFxVdFGO60EYK1lWesty8oEviFv+TIROYVK0aG81bc5gX4+PDpiFos3FfO6q6mzwTiPDYao0JWMq5bzVrMhtLpoMtk6UucSjDG0HzcOXn+dNT7nuthQwYQG+vLKDc1oWLkkb05YxPg5RXEln4j3OVup22RZ1i2WZd18io9biiXhf5UDthx3e2v+ff8yxvQ3xiQZY5JSUnTdhkjZyGDeuqk5JcMCePLrOcxafSHrBpynEnUgYcgJgyF8fIJ5Y6aT6W368XrG5cWXRc7o6CmPC59++OwC/Xx4vlc8LWrF8NHvy/l66hosd1nDWMRFna3UueWoVsuyhliWFW9ZVnx09DldFiji8aLDAnmzb3MqlwrluTHz+HtxMZ32avczVDvxd0An0GDePFizhrf+ydJEly6ijmXBli3MWLKkWN7Pz8fJwB5NuKRBOYZPWs2nf61QsRO5AGcrda44qnUbUOG42+Xz7xORsygR5MdrvRNpUCmS18fbe9pryMMPE/7ll6S99x4/25ZCjnfkn3+gYkWm3H57sb2n0+Hgoc4N6dy0EmNnbeCdn5eQk6tiJ1IQZyx1xT2q9RzNBWoYY6oYY/yAXsAEmzOJuI0gfx9euK7pv6e9vpq82pajI5XLlOHp3r3Bx4eX0LI0ruCqhg0hMpIjpUpRnJPgOIzhrstiub5VdX5bsIXXxi0gKye3GBOIeAa3W1HCsqxs4B7gd2AFMMayrOK4BETEYxw97dWhYXm+nLKGwX8sJ9eGYncbEAnMOnyY8br+1XblS5ak3u7d8MMPFPdv9MYY+ravRb9LajN5eTLPj0niSFZOMacQcW/nXeqMMU2KIsj5sCzrF8uyalqWVc2yrJfsziPijpwOBw92akC3ZlX4Yc5G3hy/iOxiPjoSAlz5++9QpQp3P/posb63nFoTk3cp9Tyb3r9n82rcd2V95q5NYeDXcziUkXX2F4kIULAjdUMLPYWI2MJhDLd3qEPfdjX5e8k2XvhuPpnZxXt05P4aNSA1le2rVjGzCOdHk3MTD2BZTN+zx7YMVzSpyIDujVm+dQ+PfTmb/Yczbcsi4k4KUurcckSsiJyaMYbrW9fg7o6xzFq9kyeL+ehIk6pV6Tt/PkyfzutFPD+anF2pVaugbFnGtm1ra452sWV55po4Nu46wKNfzmLPwQxb84i4g4KUuucKPYWI2K5z08o81rURy7bkHR3Ze6j4foi+2qAB/sbwAxT7tVxyoksrVcIMHEjmiBHFOljiVJrViOGF65qyfc9hHh4xk937i2oBMxHPcN6lzrKsH4ogh4i4gIvql+OZa+LYlHKAh4fPZNe+9GJ539JAP4Ddu7nnp5+K5T3l1CICAnjp7rsZ3qiRS5yWaVylJC9fn0DagQweHjGTHXsP2x1JxGWdsdQZY3QuRMTLNKsRw8vXJ5B6MIMHv5jB1tSDxfK+t+7eDVWrMrlHD2Yma6F3Oz1O3iSlQXYHyVevYiSv3tiMA+lZPDR8JttS7T6GKOKaznakbk6xpBARl1K/UhRv3JhIZnYuD34xkzXJ+4r8PRuXLEnFSy6B9u1552DxFElxH7XKhvP6jYlkZefy8IiZbNx1wO5IIi7HI5cJE5ELV71MCd6+qTn+vk4eHTGLhRt3F/l7/jRqFObXXxlXowbFtIiZuJFqpcN4s08iAI9+OYu1xfDLhog7MWeaSd4YsxV4+3SPW5Z12sdcRXx8vJWUlGR3DBG3lbI/nSdGziF5z2Ee69qI1nXLFOn79QJGA/cC7xbpO4m72pZ2iAFfzeZwRhYvXZ9A7XIRdkcSKTTGmHmWZcUX5LVnO1LnJG9+0NDTfIiIh4sOC+Stm5pTo0wJXho7n5/mbSrS93sCYONGPvzf/1iRmlqk7yXuqVxkMG/2SSQ00I8BX81mySZ9n4jA2Y/Uzbcsy/YVJC6EjtSJFI4jWTm8PHY+s9fsonebGvRuUwNjiuYKjdJXXcXOn3+m+cCBzHjhhSJ5D3F/qQeO8NiXs9i1L51nro0nrmq03ZFELlhRHqnTNXUiAkCAr5Nnronj0obl+WrKGt7/dSk5uUWzXuxLTzwBvXuz6IYb0DEYOZ2o0ADe7NucclEhPPNNErNW77Q7koitzlbqLi6WFCLiFo6uF3tNi2r8PG8zL40tmmXFbm3Rgsu+/JLDtWvrujo5o/Bgf167sRlVYkJ5/tt5TFmu6XDEe52x1FmWlVZcQUTEPRhjuPXi2tzeoQ7TV+7IW1bsSOEvKzYw//N7wN4zXCYiEhbox6u9m1G7XDivfD+fvxZr7LR4p4IsEyYiQvfEqv8uK/bwiFmkHijcJZxaAfHLl7OvRw96vflmoW7bm/wBfLpnj90xilywvy8vX59Ag0pRvDl+Eb8u2Gx3JJFip1InIgV2Uf1yPN+rKdvTDvHgFzPYlla4M/1fvWULjB3Ln++/z76cwj/N6+nSgOv37aP/b79xK9i+lmtRC/Dz4fleTYmvHs07Py3h5yIeqS3ialTqROSCxFeL5rUbEzmckc0Dn88o1NUnHr30Uiq98Qa5s2YxxOkstO16iwjgrs2b8b/mGoYB8cC87OxTPzkjFSZ2zPvsxvx9nTzdM46EGqV475elTJi70e5IIsVGpU5ELljtcuG8fVML/H2dPDJiJvPXF87qEw5j+Pjhh6FsWd4EtJT7+THA8/Xrk+R0UhdYefAgTZs0oed775F78nWK67+A5N9h/XAbkhYuPx8nT/VoQvOaMXz42zLGzd5gdySRYqFSJyKFokLJEAbd1IKYEkE8NWoOk5dtL5TtdiTvCNMuYFBKSqFs09vUA+YC7caNw1qyhO+GDKFLRsax6WIsC1YOyvt61aC8227Oz8fJkz2a0LJ2aQb/sZyxs9bbHUmkyKnUiUihKRmWN29YrXLhvPL9AsYXwqkvA9yzaxe0a8cz8fHsz8y84G16oyBg4o038uC33xIyZgw/BQTQEJgEkDIVsvJPm2fuhZRpdsUsVL5OB090b0zrOmUY8ucKvp2xzu5IIkVKpU5EClVooC+v3NCMZjVj+Oi3ZQz7ZyVnWrnmXPQuWRL/lBRy9u/n5aVLCympd3qrRw+W1q1LC2Ab0P6JJ2j53CD2HMofvZx96NhROw/g43TwePdGtIsty9C/V/LNtLV2RxIpMmdcJswTaJkwEXvk5Obywa/L+GX+Zi6uX44HOjXA11nw3yPfWbqUBypUoHyJEqwF/AsvqneZ3AW2TSDbOHko6l7eu/J9yM2lwcTh/JnyIKUyUsDhB7knHREt1xnajrcncyHIyc3lzfGL+Gfpdvq0rckNbWrYHUnklC5kmTCfwg4jIgJ5q0/ce0U9osMCGD5pNWkHM3iqZxOC/X0LtL1769XjM2Ap8AVweyFm9SoNX4Y9C/E5sot3dw+i9lsNefxIFwLrVifiz/z57I4vdI4ACIzJe50bczocPNylEQ6HYcTk1eTkWtzYtujWLxaxg47UiUiR+33hFt75aQlVSoXywnVNiQoNKNB2xgDX5uYS/csvbOzYkSAf/V5aINmHYNYtsO0nyDlMml8Eh3yCqXD4pJUYnEFQrhMkfgY+wfZkLWQ5uRbv/ryY3xdu5bpW1enbrqaKnbiUCzlSp2vqRKTIXdaoAs/3imdb2iHu/3wGm1MOFGg7VwOhPXuS0qkT94wcWbghvYlPMLQaDU3eAoc/kZl7/lvoHP55j7f6xmMKHYDTYbj/qgZc3rgCo6at5fN/Vl3wNZ8irkKlTkSKRdPqpXizb3OysnN54IuZLN18/ktLO4EbOneGcuX40d+fwl9x1stENMkrb6fi9IfIuOLNU0wcxnDvlfW5Kq4io2esY+jfFz6YR8QVqNSJSLGpUaYE79zcgvAgPwZ8NZtpK5LPexvv3HADNdatY3evXnxZBBm9SloSWEersck73Ur+qcjcrLzHPZTDGO65vB5dmlbmu5nr+eTPFSp24vZU6kSkWJWOCOLtm1tQvUwYL343n/Fzzm+2f38fH572zzu69BLoaN2F2DUVctLzBkMEV4QWIyGoQt7tnPS8xz2YMYY7L6tLt2ZVGDd7A4P/WK5iJ25NpU5Eil2JID9e7Z1IYs0YPvp9OUP/WvHfZavOoBdQPTOT9UOG8NB4951mw07ZwLr0ZDBOKN8FrlwGFbrCVcuhfOe8+1Nn2x2zyBljuL1DHbo1q8IPczYyREfsxI2p1ImILQJ8nTzVM46r4iry7cz1vP7DQjKzc87ptT7AJd9+C7ffzuBHH+XI6Rapl1NaB1QHLmv1NTkJn544GOLoIIqEIRBW286YxeZoseuaUJnvZ2/QNXbitjSliYjYyrIsxsxYx7B/VtGochRP94wjOODsc9mlZ2VRsls3DvfuzefXXMNNDv2Oeq6ygZrABmAs0N3eOC7Dsiw+/G0ZPyZtomfzqtx6cW1NdyLFTlOaiIjbMsZwbcvqPNKlIUs2p/HQ8Jns3n/krK8L9PXlo59+gl69eNnhQMfqzp0PcL9lwV9/cf+zz9odx2UYY7i7Y+y/R48/n6jpTsS9qNSJiEu4pEF5XriuKTv3pnP/59PZuOvsc9ndAFQD1gBf64fveemSlgadOrHluecYu3IlZKTCxI55n72YMYa7L6/HFU0qMnr6OoZPWq1iJ25DpU5EXEZc1Wje7JtITq7Fg1/MYOHG3Wd8vg8wICMDXnmF21u00LV156FSVBTNHnkEXnyRb6OjYf0XkPw7rB9udzTbOYzhf1fUo2P+BMVfTl5jdySRc6Jr6kTE5ezce5iBo+ayPe0Q91/VgA4Ny5/2uUeyswmrU4estWt5YMIE3u7UqRiTurf15A2Y8LMstv7SgJL7lkJQeeiyGXQtGbmWxTs/5S0pdmObGvRuW9PuSOIFdE2diHiUmPAgBt3cgnoVI3lzwiK+mnz6U2ABPj787/334Y8/+PGqq3Rt3XmoCnQEMozh8/Jd8u7M3Asp02xM5Tocxvz7S8WXU9YwcoqO2IlrU6kTEZcUEuDLi9cn0KFB3g/UNycsIisn95TPfa1jR6p16MBaY/i6mHO6u1uPHIEvvuClDzZgAWQfgpWD7I7lMhzG8MBVDbikQTlGTF7NqGlr7Y4kclo+dgcQETkdX6eDhzo3oExEECMmryZl/xGe7hlHyElTnvgATwE3Ac/s3EmPiAiC/PxsSOwGJneBbRP+vXlJhgPzUCj79uzj08aN6R++ALb/DF+fdPq1XGdo650TPTsdhgc7NcSy4IuJq3AYuLZldbtjifyHjtSJiEszxnBDmxo80qUhyzan8cDnM9ix9/B/nncDUPK999hYpQp3D9fF/qfV8GUIqpi3FBhQwj+XDre3gqFDmdT69rzn5GYee74jAIIr5b3Oizkdhoc6N6R9vbIM+2cV385YZ3ckkf9QqRMRt3BJg/K8dEMCqQeOcP+wGazevveEx32Aa0qVgvR0vl+4UGvCnk547LGlwJxBAHycuJyHm6fx1KZ3TnyuM+jYEmLhscWf1cU4HYZHujSkXWxZhv69krGz1tsdSeQEGv0qIm5lU8oBnho1l72HM3mie2MSa8b8+1hmbi41Fi9mc6NGfAbcYl9M97BmMMy7H3Iz/vuYwx/i3oEadxR3KpeXk5vLK98vYOqKHdxzeSyd4ivbHUk8iEa/iojXqBQdyju3tKBSyRCeG5PE+Lkb/33Mz+Hg5UaNAHgRdLTubCKa5JW3U3H6Q2Rc8eZxE06HgwHdGpNYoxQf/LqM3xdusTuSCKBSJyJuKDIkgDf6JNKsRgwf/baMwX8sJyc376xDL6A2sGHdOh77809bc7q8tCSwjlZfk386Nn+ARG5W3uNySj5OB0/2aEJc1ZIM+nEx/yzZZnckEZU6EXFPAX4+PNUzjq4JlRk3ewMvfjePI1k5OIE+CxdCrVq816cP+9LT7Y7qunZNhZz0/MEQFaHFSAiqkHc7Jz3vcTktPx8nT18TT/1KkbwxfhFTVyTbHUm8nEqdiLgtp8Nw52Wx3HFpXWau2smjI2ax52AGjzRsSEBcHDmXX87QQ4fsjum6UmeDcR4bDFGh67FBFMaZ97icUYCvk+d7NaV2uXBe/X4Bs9fstDuSeDENlBARjzBj5Q5eHbeA8BB/XuzVlGnhAdzg60t5YC1wmivHvNukK6HC1VDtFENK1g2DLWOh3c/Fn8sNHTqSxYCvZrNh1wGe79WUJlVL2h1J3NSFDJRQqRMRj7Fy216eGT2X7JxcnugRx01VSrIU+AC42+5w4vH2p2fy6IhZbE87xEvXJ1C/UpTdkcQNafSriAhQu1w4797SkqjQAJ76eg695m+G+fN55MYbSdVpWCliYYF+vNq7GTHhQTz1zVyWb91jdyTxMip1IuJRSocHMejmFjSuUpIpPy+h7vvfkz7ya+777ju7o4kXCA/259XezQgP9mfg13NYk7zP7kjiRVTqRMTjBPv78nyveDo3rUT5qs1p9O4EZvW6nly7g4lXiAoN4PUbEwkJ8OXxkbPZsHO/3ZHES6jUiYhHcjoc3N2xHndeVpdS++Ci4bNI3a/pTaR4lCoRyKu9m+Hn42DAyNls3n3Q7kjiBVTqRMSjdU2owgvXNiVlz2HuGzZdp8Ok2JSNDObV3okADPhqFsl7DtucSDydSp2IeLyEGqV4+6bmOB0OHho+k+krd9gdSbxExZIhvNY7kczsXB4fOZvUA0fsjiQeTKVORLxClZgw3r2lBVVKhfLCt/MYM2Mdnj6lk7iGyqVCeen6BPYeymDAV7PZdzjT7kjioVTqRMRrRIbkXcDeum4ZPvt7JYN+WkxWjoZPSNGrVTac565tSvKewwz8eg6HMrLO/iKR86RSJyJexd/XyePdG3N9q+r8vnArT349h/3pOnIiRa9h5SgG9mjCup37eeabJDKycuyOJB5GpU5EvI7DGPq2r8UjXRqyfMseHhg2g22pmpxYil5izRge7dKIpZvTePG7eTpSLIVKpU5EvNYlDcrzau9m7E/P5L7Pp7N4U6rdkcQLtKtXlnuvrM+ctSm88cNCcnJ1bacUDpU6EfFq9SpG8u4tLQkP8uPxr2bz+8ItdkcSL3BFk4r0u7g2k5cn894vSzRoRwqFSp2IeL2ykcG8c0tL6leK4u0fF/PJn8t19ESKXM8W1biuVXV+W7CFT/9aoWInF8zlSp0x5g1jzEpjzGJjzDhjTHj+/ZWNMenGmIX5H4NtjioiHiQkwJeXrm9Kl6aV+X7WBp4ZPZdDRzRCUYpW33Y16dy0EmNnbWDUtLV2xxE353KlDvgTqGdZVgNgNfD4cY+tsyyrUf7HHfbEExFP5XQ4uKtjLPdeUY/563dz/+cz2JamARRSdIwx3HlZLJc0KMfwSasZP2eD3ZHEjblcqbMs6w/LsrLzb84CytuZR0S8z5VxlXjlhmbsOZTBfcOms3DDbrsjiQdzGMODnRrQslYMH/2+nD8XbbU7krgplyt1J7kF+PW421WMMQuMMZONMa1P9yJjTH9jTJIxJiklJaXoU4qIx2lYOYr3bmlJRLA/j4+cw49Jm+yOJB7M6XAwoHtjmlQtyds/LmKGlrKTAjB2XJhpjPkLKH2Kh560LGt8/nOeBOKB7pZlWcYYfyDEsqxUY0wc8AMQa1nW/jO9V3x8vJWUlFS4fwAR8RqHMrJ49fsFzFmbQqf4StxxaV18nK7++7C4qyOZ2Tz21WzW7djPKzckUL9SlN2RpJgZY+ZZlhVfoNe64mgbY8xNwO3AxZZlHT7NcyYBD1uWdcbGplInIhcqJ9di2D8r+W7mehpVjuLJHk0IC/SzO5Z4qP2HM3lo+Ex2HzjCm32aU610mN2RpBhdSKlzuV83jTEdgUeBzscXOmNMtDHGmf91VaAGsN6elCLiTZwOw22X1OGhzg1YtmUP9w2bzuaUA3bHEleXkQoTO+Z9Pg9hQX68dH0CQf4+PPn1HJL3nPLYhsh/uFypAz4AQoE/T5q6pA2w2BizEPgOuMOyrDSbMoqIF7q0YQVeu7EZhzOyue/zGcxdu8vuSOLK1n8Byb/D+uHn/dJSJQJ55foEsnNzeXzkbNIOHin8fOJxXPL0a2HS6VcRKWw79x7m2THz2LhrP7d1qEu3hMoYY+yOJa7EsuCHCpC+DYLKQ5fNUIDvkZXb9vDol7MpHxnMG30SCQ7wLYKw4ko86vSriIiriwkP4u2bmtO8Zgyf/LGcd35aooXZ5UQpUyFrX97XmXshZVqBNlO7XARP9WjCxpQDPDsmiczsnMLLKB5HpU5EpAAC/XwY2DOO61tV57eFW3jsy1nsPZRhdyxxFSvfgez8iauzD8HKQQXeVNPqpXi4c0MWb0rjle8XkJOrXyDk1HT6VUTkAk1cuo23f1xMeLA/z/SMo3qZEnZHkuI0uQtsm3DifQ4/yM1kUqm2NEidT6SVAbmZJz6nXGdoO/6c3+aHORv4+PfldGxcgfuvrK9T/h5Kp19FRGzUvl453urbnFzL4sEvZjBx6Ta7I0lxavgyBFUER8Cx+3Iz2e8TyqVNxhD1UiXKjErkk+r9SQ4onfe84Ep5rzsPXROqcF2r6vy2YAvDJ60u5D+EeAKVOhGRQlCzbDgf3NqK6mVK8Oq4hQz9awU5uZ59JkTyhcfCVcuhfGdwBv17987AGCoPew2WLGXHHos7Ej6hbPdkml+xiP+VHsW68Njzfqu+7WpyeeMKjJq2lnGztU6snEilTkSkkESE+PPajYlc0aQi385cz9PfzOVAepbdsaQ4+ARDq9HQ5C1w+ANQ48BaVpd9m0WfRDDwphJ02vYTAbnZzNofxAc7dvFZAd7GGMP/rqhPy9qlGfzHch0VlhOo1ImIFCJfp4P7rqzPvVfUY8GG3dw7bBqbNFGx94ho8m+pO6pByB5eCPiJCTNuYPeeRXyUlcW17drRo4Bv4XQYBnRrRP2Kkbw5fhELN+y+8NziEVTqRESKwJVxlXj9xkQOZ2Rz/7AZzFy10+5IUhzSksA6enTW5J+OzR/QkJtFcNoc7qxShW9KlKDJBbyNn4+TZ66Jp1xUMM99O4/1O8+4DLp4CZU6EZEiUq9iJO/f2opyUcE8OyaJkVPWkOvhMw54vV1TISc9fzBERWgxEoIq5N3OSc97vJCEBvry4nUJBPn58NSouezal15o2xb3pFInIlKESpUI5K2+zbm4fjlGTF7Ni9/NJz0z2+5YUlRSZ4NxQvkucOUyqND12CAK48x7vBCVKhHIi9c15XBmNgNHzdE1nF5OpU5EpIj5+zp5pEtD+neow8xVO7h/2Ay2px2yO5YUhRJ1IGEItPomb/AEHBtEkTAEwmoX+ltWiQnjmWvi2JZ6iOe06oRX0+TDIiLFaN76FF4euwCAJ69uQpOqJW1OJJ5i4tJtvDpuIW3qluHx7o1xaHJit6TJh0VE3ERc1Wjev7UlUaH+PPn1bL6ftR5P/+Vaikf7euXod3FtpixP5tM/V9gdR2ygUiciUszKRgbzzs0taV4zhk/+XMEb4xeRkaVTZnLhejSvSteEynw/ewPfz1pvdxwpZip1IiI2CPL3YWDPOPq0rcnfS7bx4Bcz2LH3sN2xxM0ZY+jfoS6tapfmkz9XMHnZdrsjSTFSqRMRsYnDGG5oU4Pnro0nec9h/jd0GvPWp9gdS9yc02F4rFsjYitE8Mb4RSzelGp3JCkmKnUiIjZLrBnD+7e2IiLEn4Ffz2H09HW6zk4uiJ+Pk2evjadMRBDPjk5is1Y18QoqdSIiLqBcVDDv3tKSVnXKMOyflbz43XwOZ2g+Oym4sEA/XryuKX4+TgZ+M5c9BzPsjiRFTKVORMRFBPr58ET3xtx2SR1mrNrBfcOms2X3QbtjiRuLCQ/i2Wvj2Xswg2fHJGlAjodTqRMRcSHGGHo0r8orNzRj3+FM7v1sOjNW7rA7lrix2uXCeaxbY1Zt28sb4xdqqToPplInIuKCGlUpyQf9WlG+ZN6C7V9MXEVOrn4YS8G0rF2afpfUYeqKHXz+zyq740gRUakTEXFRR9eN7dioAqOmreXpb+ayPz3T7ljipq5OrMJVcRUZM2Mdv8zfbHccKQIqdSIiLszPx8kDnRpw35X1WbhhN/8bOo11O/bbHUvckDGGuzrGEl8tmvd/WarpczyQSp2IiBu4oklF3uzbnOwciwc+n84/S7bZHUnckNPh4ImrG1MpOoQXv5vPxl2a6sSTqNSJiLiJOuUj+KBfK2qWDee1Hxby8e/LyM7JtTuWuJlgf1+e79WUAF8nT30zl7SDR+yOJIVEpU5ExI1EhPjzau9mdGtWhR/mbOTRL2eRekA/lOX8lCoRyPO9mrLvcCbPfJPEEU114hFU6kRE3IyP08Edl9ZlQLdGrN2xn7s/ncaijVoKSs5PjTIleKJ7Y9Yk7+P1cQs01YkHUKkTEXFT7euV471bWhIc4MOAr2Yxevo6/WCW85JYM4bbL63L9FU7GT5RU524O5U6ERE3VrlUKO/f2urf5cWeGzOPg0ey7I4lbqRrQmUub1yBb6av0wAcN6dSJyLi5oL885YXu+PSusxdu4t7hk5j3Y59dscSN2GM4e7L69GgUiRv/7iYldv22B1JCkilTkTEAxhj6NasCm/0SSQrO5f7P5/B7wu32B1L3ISv08HAHnGUDAvg2dHz2LUv3e5IUgAqdSIiHiS2QiQf3taKuhUiePvHxbz94yIt4i7npESQH89eE09GVg7Pjk7iSGa23ZHkPKnUiYh4mPBgf16+vhnXtarO7wu38sDnM9iedsjuWOIGKpcK5fHujdmwaz9vjF+kgTduRqVORMQDOR2Gm9rX4vle8ezcl849Q6cxY9UOu2OJG0ioUYp+l9Rh2sodfDl5td1x5Dyo1ImIeLBmNWL4sF8rykQE8dyYeXz290pycrUKhZxZ92ZVuKxReb6eupZJS7fbHUfOkUqdiIiHKx0RxKCbW3BFk4qMmbGOAV/N1tJQckbGGO65vB6xFSJ468dFrNy21+5Icg5U6kREvICfj5P7rqzPw50bsmrbXq1CIWfl5+Pk6Z5xRIT489yYJC1H5wZU6kREvEiHhuV555aWBPnlrULx9dQ1uhheTis82J/nronnUEY2L3w3j8xsjaR2ZSp1IiJepmpMGO/3a0WbumUZPmk1T349h72HMuyOJS6qSkwYD3VqwIqte/n49+V2x5EzUKkTEfFCQf4+DOjWiPuurM+STWncOWQqizfpdKycWtvYslzTohq/zN/ML/M32x1HTkOlTkTESxljuKJJRd69pSWBfj489qVOx8rp3dS+FnHVovnw16Us25Jmdxw5BZU6EREvV610GB8cdzp2oE7Hyik4HYbHuzUmukQgL343XwMnXJBKnYiInHA6dvGmNO76dCpLdDpWThIa6Muz18RzOCObF77VwAlXo1InIiLA8adjWxDg68OjX87im2lrdTpWTlC5VCgPd27Iim17+fC3ZVj6/nAZKnUiInKCaqVL8H6/lrSuU4bPJ65i4Ki5Oh0rJ2hdtwy9WlbjtwVb+FkDJ1yGSp2IiPxHsL8vj3dvzL1X1GPxxtS807GbdXG8HNOnXS2aVo/m49+WaeCEi1CpExGRUzLGcGVcJd69pQX+vk4eHaHTsXKM02EYkD9w4uWxC3Q01wWo1ImIyBlVK12CD/q1onWd0nw+cRVPjJyjtWMFgJAAX57q0YT96Zm8Mm4BObkq/HZSqRMRkbM6ejr2vivrs2xLGnd8MpW5a3fZHUtcQLXSJbjn8nos3JDKV5NX2x3Hq6nUiYjIOTk6Ovb9W1sREezPwFFz+fSvFWTl5NodTWx2WaMKXNaoPF9PW8vsNTvtjuO1VOpEROS8VC4Vynu3tuTKuIp8N3M9D34xg+1ph+yOJTa7u2M9qsWE8foPi9ix97DdcbySSp2IiJw3f18n915Rn4E9mrA97RB3fzqNiUu32R1LbOTv6+SpnnFYlsWL383XxMQ2UKkTEZECa12nDB/d1poqMaG8Om4hb01YxJHMbLtjiU3KRATxSJdGrEnex8e/L7c7jtdRqRMRkQsSEx7EG30Sub5Vdf5ctJW7h05j3Y59dscSmzSvFcM1Larxy/zN/Lloq91xvIpKnYiIXDCnw0Hf9rV49cZmpGdmc9+wGYyfu1FLSHmpm9rXpEGlSN7/ZQmbUg7YHcdrqNSJiEihaVS5JB/d1prGVUvy0W/LeG7MPPYfzrQ7lhQzp8PBgG6NCfDz4eWxC8jI0vV1xUGlTkREClV4sD/PXxvPHZfWJWldCnd+OpXFm1LtjiXFLCo0gEe7NmJjygEG/6Hr64qDSp2IiBQ6YwzdmlVh0M0t8PfJW2Ls839Wkq057bxKfLXof6+vm7xsu91xPJ5KnYiIFJkaZUrw4W2tuKxxBb6Zvo4HPp/BtlTNaedN+rarSZ3y4bzz8xKS92j+uqLkcqXOGPOsMWabMWZh/scVxz32uDFmrTFmlTHmMjtziojIuQn08+GBqxrwVI8mJO89zJ2fTuXXBZs1iMJL+DgdPN6tMQ4DL4+drxVIipDLlbp8gyzLapT/8QuAMaYu0AuIBToCHxljnHaGFBGRc9eqThkG92+Td9TmpyW88K0GUXiLmPAgHriqAauT9zHsn5V2x/FYrlrqTqUL8I1lWRmWZW0A1gIJNmcSEZHzUDIsgFduaMZtl9Rh9ppd3DFkCvPX77Y7lhSDVnXK0Cm+Et/P2qD1YYuIq5a6e4wxi40xw4wxEfn3lQO2HPecrfn3/Ycxpr8xJskYk5SSklLUWUVE5Dw4jKFH86q8d2tLgv19eXzkbIb8uVzLSnmB/h3qUC0mjLcmLCbt4BG743gcW0qdMeYvY8zSU3x0AT4GqgGNgGTgrfPdvmVZQyzLircsKz46Orpww4uISKGoVroE7/drRaf4SoydtYH7hs3QRLWuLCMVJnbM+1xAfj5OBnRrRHpmNm//uFjXVRYyW0qdZVmXWJZV7xQf4y3L2mlZVo5lWbnApxw7xboNqHDcZsrn3yciIm4qwNfJPZfX47lr40k9cIR7hk7jxyStROGS1n8Byb/D+uEXtJmK0aHcdkkd5q5N4cekTYWTTQAXPP1qjClz3M1uwNL8rycAvYwx/saYKkANYE5x5xMRkcKXWDOGwbe3pkGlKD74dRlPj05iz8EMu2PJUZYFKwflfb1qUN7tC9ApvhLx1aL59K8VbNbR2ULjcqUOeN0Ys8QYsxhoDzwAYFnWMmAMsBz4DbjbsixdgCEi4iEiQwJ48bqm3HVZXRas383tn0xhxqoddscSgJSpkLUv7+vMvZAy7YI2Z4zhoc4NCPTz4bUfFmqak0LicqXOsqwbLcuqb1lWA8uyOluWlXzcYy9ZllXNsqxalmX9amdOEREpfMYYuiRU4YN+rSgZGsBzY+bx1oRFHMrIsjuad1v5DmTnTxqdfejYUbsLEBkSwP1X1mftjv18OWn1BW9PwHj6dQvx8fFWUlKS3TFEROQ8ZeXkMnLKGkZPX0t0WCAPd2lIg0pRdsfyfJO7wLYJJ97n8IPczNPfBijXGdqOP++3G/TTYn5fsIU3+iRSX3+/GGPmWZYVX5DXutyROhEREQBfp4Ob2tfirZta4HQaHh0xS1OfFIeGL0NQRXAEHLvv5AJ3QsELgOBKea8rgDsurUuZyCBeH7+IQ0d0RPZCqNSJiIhLq1s+go9va82VcRUZO2sD9wydxtrkfXbH8lzhsXDVcijfGZxBZJ9p8SZnEJTvAlcuy3tdAQT6+fBol0bs3p/OkL9WFDC0gEqdiIi4gQA/H/53RX1evK4pB9KzuG/YdEZNW0tOri6wLxI+wdBqNFaTt7g2bjh3Nv2Iw87AE5/j8Icmb0Grb/KefwHqlI/g6sSq/LZgC0nrtGhAQanUiYiI22havRSf3NGGlrVL88XEVTw0fCbb0g7ZHctjPbjYj+8vHsDwnTVJDixz4oNOf4iMK7T36tOuJhVLhjDop8U6DVtAKnUiIuJWwgL9eOLqJjzerTFbdh/kziFT+WneJk1YXMjWAO+P+AG2buXyEQ9R7eCGvNOtmLwn5GZBWuENRPTzcfJQ54akHTjCkD91GrYgVOpERMQttatXlk9ub0u9ChG8/8tSBo6ay+79Wk+0MGQBvYGc78bS9JXbGNNuJQRXhBYjIahC3uCInHTYNbVQ37d2uXB6NK/Gbwu3MHftrkLdtjdQqRMREbdVMiyAl65P4J7LY1myOY3+gyfzx6ItOmp3gV4kb8mmClkp/FF9LM6KXfMGQ1ToemwQhXFC6uxCf+8b29agYskQ3vl5iU7DnieVOhERcWvGGDrFV2Zw/9ZUiQnjrQmLeXp0ko7aFdCLY8bw/D33wOHDjFj9HuGN3zhxMET+IAoShkBY7UJ/f52GLThNPiwiIh4j17KYMHcjw/5eia+PgzsujeWSBuUwxtgdzS3sTk+ndJUq5OzcyeUjRvDLjTfalmXoXyv4duZ63vSySYk1+bCIiAjgMIauCVX4+PY2VIoO5c0Ji3hmdBKpB3TU7lw8GxhIzq+/EnX//Yzt3dvWLL3b1CAmPJB3fl6iCafPkUqdiIh4nHKRwbzZtzm3X1qXhRt203/wZP5avFXX2p3B38CHgE/jxvw1aBCBNh/dDPDz4X+X12Nr6iHGTF9naxZ3oVInIiIeyWEM3ZtV4eP+eUft3hi/iGd11O6UFq5dS+/ZeYMengEa2ZrmmKbVS9EutizfTF/H5t0H7Y7j8lTqRETEo5WLCuaNPs3p36EO8zfspv/gKfyto3b/ysnJ4fKbbmJHixZUGTOGAXYHOskdl9bF39fBez8vIVd/Z2ekUiciIh7P6TBcnViVj25rTYWSwbw+fhHPjpmno3bAXzk57GjZEsqW5ctLLsHH7kAniQjxp98ldViyOY0/Fm6xO45L0+hXERHxKjm5FuNmb2D4pFX4Oh3c1qEOHRtV8MoRsgeBBsAGYODBg7wQEmJzolPLtSweGTGLzSkHGHZ3e0IDfe2OVGQ0+lVEROQcOR2GHs2rMrh/G6qVDuOdn5bw2FezvW4N2ZycHB7PymIDedfQPe2ihQ7yro+867JYDh7JYsTkVXbHcVkqdSIi4pXKRQXz2o2J3HdlfdYk7+OOT6bw7cx15OTm2h2tWDz0/vt80LQpjoUL+Qxw9WNf1UqHcWVcJX5K2sSGnfvtjuOSVOpERMRrOYzhiiYV+fSOtjSpGs3Qv1Zy37AZrNvh2aUhPSeHwZ99BosW0W3LFprYHegc9WlXk+AAXz76fZkGupyCSp2IiHi9kmEBPHtNHE90b0zK/nT+99k0vpi4ymMnvX3b6SRj5kxKDR/Ol5062R3nnIUF+nFT+1os3pTGlOXJdsdxOSp1IiIi5K0h2za2LJ/e0Zb29coyatpa7hoylaWb0+yOVqhWAy8AhIQwqk8fAm3Oc74ub1yRajFhDPlrBUeyPLN0F5RKnYiIyHHCgvx4pEsjXro+gYzsXB4aPpMPfl3K4Yxsu6NdsP0HDtB58GAysrPpC1xkd6ACcDoMd3aMZff+I4ybvcHuOC5FpU5EROQU4qtFM+SONnRNqMxPSZu47ePJTF+5w62v5br+xRdZdeed+Pfrx5t2h7kA9StGklgzhjEz1rHvcKbdcVyGSp2IiMhpBPr5cOdlsQy6uQWhgb48/+08nh2dxK596XZHO28WsL5fP+jYkcfuuYeSdge6QLdeVIsjmdl8PXWN3VFchiYfFhEROQfZObmMm72BL6eswQA3tq1Jt2aVcTrc5/jIfuBz4F7AE6Zafuenxfy5aCtD72pHmYggu+MUCk0+LCIiUsR8nA56tqjGp3e0oUHlKD79awX/Gzqdldv22h3tnIUB9+EZhQ7yirXT6eCLiZqQGFTqREREzktMeBDPXxvPwB5N2Hs4g/uHTeeDX5dy6EiW3dG8TlRoAFc3q8KkZdtZrwmJVepERETOlzGG1nXK8OmdbencNG8gRb+PJzNlebJbD6RwR90TqxLk78PIKbq2TqVORESkgIL9fbmrYyzv3dqSyBB/Xho7n6e+mct2L1tH1k6hgb50TajMtJU7vH75MJU6ERGRC1SzbDjv3dqS2y+ty9LNafQfPIURk1aToclxi0W3ZlUI8vNh5NS1dkexlUqdiIhIIXA6HHRvVoWhd7ajZe3SjJy6htsGT2bGKvee284dhAX60SWhMtNWJLNx1wG749hGpU5ERKQQlQwL4PHujXn9xkQCfJ08N2YeT30zl206JVukujergr+vk+9mrrc7im1U6kRERIpAw8pRfHRba/p3qMOyzXu4ffAUhk9cpfVKi0hYkB+XNarAxKXbSD1wxO44tlCpExERKSI+TgdXJ1Zl6F1taV2nNF9PW0v/jyczw82XG3NVXRMqk5Nr8WPSJruj2EKlTkREpIhFhQbwWLfGvNEnkUA/H577dh4DR81ly+6DdkfzKGUjg2leK4af5m3yyiOiKnUiIiLFpEGlKD68rRW3X1qX5Vv3cPsnUxj8x3IOpGvi4sLSPbEqB9Kz+GfJNrujFDuVOhERkWLk48wbJTvsrnZc2rA8P8zewC0fTuTHpI3k5ObaHc/t1asQQaXoEH5bsMXuKMVOpU5ERMQGESH+3H9VAz68rRWVS4Xywa/LuHPIVOatT7E7mlszxtCxcUVWbd/rdUuHqdSJiIjYqFrpErx+YyJP94wjMzuXJ0bO4Zlv5rI1VdfbFdTF9cvh63Tw+0LvOlqnUiciImIzYwwta5dmyB1tuPXi2izelMbtg6fwyZ/LOXhE19udrxJBfjSvFcPfS7aRneM9p7RV6kRERFyEn4+Ta1pU47O723JJg/KMm7WBmz+YyLjZG8jM9r7RnBeifWxZDqRnsWhTqt1Rio1KnYiIiIuJDAnggU4N+KBfK6qVLsHgP5Zz28eTmbR0O7ma3+6cxFWLJtDPybQVO+yOUmxU6kRERFxU9TIleLV3M16+PoEgf19eGbeA+z6bzsKNu+2O5vL8fZ00qxHD9JU7vGZUsUqdiIiIi4urFs0H/VrxcOeG7DmUwWNfzuapUXPY4GWjO89Xi1ox7Ducyert++yOUixU6kRERNyA02Ho0LA8w+5uR7+La7Nsyx7uHDKVtyYsYte+dLvjuaRGVUoCsGCDdxzZVKkTERFxI34+Tnq2qMYX/2tP98QqTFy6nVs+nMTHvy9jz8EMu+O5lBJBflSLCWPhRu8YLKFSJyIi4obCAv3o36Eun93VlosblGPC3E30/WAin/29kv2HM+2O5zIaVo5ixdY9XnFdnUqdiIiIG4sJD+KBqxow9M62tKwVw7cz1tH3g4l8NXk1hzI0x1310mFkZueyZfchu6MUOZU6ERERD1AuKpjHujVm8O1taFKlJF9OWUPf9ycyevo6jmRm2x3PNtVKlwBg3Q7PHyzhY3cAERERKTyVS4XyVM841iTvY8SkVQz7ZyVjZ62ne7MqdGpaiWB/X7sjFqvyUcE4DGxN8/wjdSp1IiIiHqhGmRK8cF0Cy7fu4eupa/h84iq+nbmOrglV6JJQmbBAP7sjFgsfp4OIEH927z9id5Qip1InIiLiweqWj+DF6xJYk7yPr6eu4aspaxg7az2d4itzdWIVwoP97Y5Y5KJCAkj1gpHBKnUiIiJeoEaZEjxzTTwbdu5n1LS1fDtjHePnbODyJhXp3qwKMeFBdkcsMv6+TrK8YO1clToREREvUiUmjCeubsKNbQ8yevo6JszdxIS5m2hTtww9mlelRpkSdkcsdE6nITPL86c0UakTERHxQhVKhvBwl4b0aVeTH+Zs4Nf5W5i0bDsNK0fRI7Eq8dWjcRhjd8xCcSQzh2B/z688nv8nFBERkdMqVSKQ/h3qckPrGvyyYDM/zN7IU9/MpWLJELonVqF9vXIE+DrtjnlB9h7KoGyE555ePkrz1ImIiAjBAb70bF6N4f9rz2NdG+HrdPDOT0u44Z2/+OSP5WxLdc8pQdIzs9m5N51ykcF2RylyOlInIiIi//JxOriofjna1yvL0s1p/Ji0ifFzN/L97A3EVYumU1wlEmqUwulwj1Ozq7btxQKqe+C1gidTqRMREZH/MMZQv1IU9StFkXrgCL8u2MIv8zfx7JgkYkoE0qFheTo0KE9pFz+tOXVFMv4+DhpVjrI7SpFzuVJnjBkN1Mq/GQ7stSyrkTGmMrACWJX/2CzLsu4o/oQiIiLeJSo0gN5tatCrZTVmrt7JL/M3M3JK3px3DStH0aFBeVrXKU2An2vViv3pmfy9ZBstartetqLgcn9Cy7KuPfq1MeYt4PjF2tZZltWo2EOJiIgIPk4HreuUoXWdMuzal85fi7fyx6KtvDlhER/9toxWdUrTNrYsjSpH4eO0/7L9EZNWcyQzh14tq9sdpVi4XKk7yhhjgGuAi+zOIiIiIicqVSKQ61vX4LpW1Vm6OY3fF21l2ood/LFoK6GBvrSsXZo2dcvQqHIUTkfxF7zJy7bzY9ImujerQuVSocX+/nYwlmXZneGUjDFtgLcty4rPv10ZWAasBvYDAy3Lmnqa1/YH+gNUrFgxbtOmTcWSWURExJtlZucwb91upizfzszVO0nPzCE00Jf4atEkVC9FXLVoSgQV/Zqzfy3eyqAfF1OrXDiv9m6Gn4/7TMlijJl3tPuc92vtKHXGmL+A0qd46EnLssbnP+djYK1lWW/l3/YHQizLSjXGxAE/ALGWZe0/03vFx8dbSUlJhZpfREREziwjK4d561KYvmoHSetS2HsoEwPULhdO4yolqVcxkjrlIwgqxEmBd+8/wtC/VzBx6XYaVIrk2WviCQ7wLbTtFwe3K3VnY4zxAbYBcZZlbT3NcyYBD1uWdcbGplInIiJir1zLYk3yPuau2cWctSmsSd5LrgUOY6heOoxa5cKpGhNG1ZhQKkeHnteghkMZWSzemMakZduZtiIZYww9W1Sld5satpz2vVAXUupc9Zq6S4CVxxc6Y0w0kGZZVo4xpipQA1hvV0ARERE5Nw5jqFU2nFplw+ndtiaHM7JZsXUPSzensXRLGn8v3saPmXmXShkgMtSfUiUCKRUWSIlgPwJ9fTiUkcWhjGxiSgSSnpnD7gNH2LL7INvSDpGTaxES4MNV8ZXollDF5adZKSquWup6AaNOuq8N8LwxJgvIBe6wLCut2JOJiIjIBQny9yGuWjRx1aKBvCN5O/ems2HnfjbsOsCOvYfZtS+d1cn7OJCeRXpmNjm5eWcWHQYC/XyICPGnQlQILWuXplGVKOqWj3Cra+eKgkuefi1MOv0qIiLi/g5lZJGVnUuJID/yJsjwTJ54+lVERETkX8H+vuBvdwrX5n5XEIqIiIjIf6jUiYiIiHgAlToRERERD6BSJyIiIuIBVOpEREREPIBKnYiIiIgHUKkTERER8QAqdSIiIiIeQKVORERExAOo1ImIiIh4AJU6EREREQ+gUiciIiLiAVTqRERERDyASp2IiIiIB1CpExEREfEAKnUiIiIiHkClTkRERMQDqNSJiIiIeACVOhEREREPoFInIiIi4gFU6kREREQ8gEqdiIiIiAdQqRMRERHxACp1IiIiIh5ApU5ERETEA6jUiYiIiHgAlToRERERD6BSJyIiIuIBVOpEREREPIBKnYiIiIgHUKkTERER8QAqdSIiIiIeQKVORERExAOo1ImIiIh4AJU6EREREQ+gUiciIiLiAVTqRERERDyASp2IiIiIB1CpExEREfEAKnUiIiIiHkClTkRERMQDqNSJiIiIeACVOhEREREPoFInIiIi4gFU6kREREQ8gEqdiIiIiAdQqRMRERHxACp1IiIiIh5ApU5ERETEA6jUiYiIiHgAlToRERERD6BSJyIiIuIBVOpEREREPIBKnYiIiIgHUKkTERER8QAqdSIiIiIeQKVORERExAOo1ImIiIh4ANtKnTGmpzFmmTEm1xgTf9Jjjxtj1hpjVhljLjvu/o759601xgwo/tQiIiIirsnOI3VLge7AlOPvNMbUBXoBsUBH4CNjjNMY4wQ+BC4H6gLX5T9XRERExOv52PXGlmWtADDGnPxQF+Aby7IygA3GmLVAQv5jay3LWp//um/yn7u8eBKLiIiIuC7bSt0ZlANmHXd7a/59AFtOur/ZqTZgjOkP9M+/mWGMWVrYIT1ASWC33SFckPbLqWm//Jf2yalpv5ya9supab/8V62CvrBIS50x5i+g9CkeetKyrPFF9b6WZQ0BhuRnSLIsK/4sL/E62i+npv1yatov/6V9cmraL6em/XJq2i//ZYxJKuhri7TUWZZ1SQFetg2ocNzt8vn3cYb7RURERLyaK05pMgHoZYzxN8ZUAWoAc4C5QA1jTBVjjB95gykm2JhTRERExGXYdk2dMaYb8D4QDfxsjFloWdZllmUtM8aMIW8ARDZwt2VZOfmvuQf4HXACwyzLWnYObzWkaP4Ebk/75dS0X05N++W/tE9OTfvl1LRfTk375b8KvE+MZVmFGUREREREbOCKp19FRERE5Dyp1ImIiIh4AI8rdcaYN4wxK40xi40x44wx4ad5nlctOXamZdlOet5GY8wSY8zCCxlW7S7OY7942/dLpDHmT2PMmvzPEad5Xk7+98pCY4xHDlw62999/qCu0fmPzzbGVLYhZrE7h/1ykzEm5bjvj3525CxOxphhxphdp5sb1eR5L3+fLTbGNCnujHY4h/3Szhiz77jvlaeLO2NxM8ZUMMZMNMYsz/8ZdN8pnnPe3y8eV+qAP4F6lmU1AFYDj5/8BC9dcuyUy7KdRnvLshp5ydxBZ90vXvr9MgD427KsGsDf+bdPJT3/e6WRZVmdiy9e8TjHv/tbgT2WZVUHBgGvFW/K4nce/yZGH/f9MbRYQ9rjC/KWtzydy8mb0aEGeRPkf1wMmVzBF5x5vwBMPe575fliyGS3bOAhy7LqAonA3af4N3Te3y8eV+osy/rDsqzs/JuzyJvP7mQJ5C85ZllWJnB0yTGPZVnWCsuyVtmdw9Wc437xuu8X8v58w/O/Hg50tS+Krc7l7/74ffUdcLE5xfqHHsYb/02clWVZU4C0MzylCzDCyjMLCDfGlCmedPY5h/3idSzLSrYsa37+1weAFRxbPeuo8/5+8bhSd5JbgF9PcX85/rvk2Mk701tZwB/GmHn5y62Jd36/xFiWlZz/9Q4g5jTPCzDGJBljZhljuhZPtGJ1Ln/3/z4n/xfKfUBUsaSzz7n+m7g6/7TRd8aYCqd43Nt44/8l56q5MWaRMeZXY0ys3WGKU/4lG42B2Sc9dN7fL6649utZmXNYfswY8yR5hzdHFmc2O53LfjkHrSzL2maMKQX8aYxZmf9bltsqpP3icc60X46/YVmWZYw53dxHlfK/X6oC/xhjlliWta6ws4pb+hEYZVlWhjHmdvKOZl5kcyZxTfPJ+7/koDHmCuAH8k45ejxjTAgwFrjfsqz9F7o9tyx1Z1t+zBhzE3AVcLF16on4zrQUmdsq4LJsJ29jW/7nXcaYceSdZnHrUlcI+8Xrvl+MMTuNMWUsy0rOP9y/6zTbOPr9st4YM4m83zY9qdSdy9/90edsNcb4ACWA1OKJZ5uz7hfLso7fB0OB14shl6vzyP9LLtTxZcayrF+MMR8ZY0palrXbzlxFzRjjS16hG2lZ1veneMp5f7943OlXY0xH4FGgs2VZh0/zNC05dgrGmGBjTOjRr4FLyRtI4O288ftlAtA3/+u+wH+OaBpjIowx/vlflwRakrcSjCc5l7/74/dVD+Cf0/wy6UnOul9OuvanM3nXDHm7CUCf/FGNicC+4y5z8FrGmNJHr0M1xiSQ1008+hej/D/vZ8AKy7LePs3Tzv/7xbIsj/oA1pJ3Dnph/sfg/PvLAr8c97wryBsdu46803C2Zy/i/dKNvPPxGcBO4PeT9wtQFViU/7FM+8Wrv1+iyBv1ugb4C4jMvz8eGJr/dQtgSf73yxLgVrtzF9G++M/fPfA8eb84AgQA3+b/3zMHqGp3ZhfZL6/k/z+yCJgI/2/v/kHsKsIwjD9v1E6M2KQQBBPUdYkBTSAgBgzEwliIiphAOkFSqKQSRS03qWyCoAhWUbMgms5CGzFNQCRxNSriYiL4DywEG0PUz+JM4LJxuXd3s1mZfX7lnJn7DYdzL9+dOTPD1Fr3+Srck+PAz8DF9rvyJHAQONiuh2HV8Hz7zuxY6z7/T+7L0yPPying3rXu81W4J/cxvMM+N5Kv7F3p8+IxYZIkSR3obvpVkiRpPTKpkyRJ6oBJnSRJUgdM6iRJkjpgUidJktQBkzpJkqQOmNRJ6k7bzHQ2yXw7x/iDJLePaXMoyZ9JNi4z5oPtHNyvkpxO8soEbc4kmV1OPElayKROUlfaTu0ngI+raktVbQdeADaNabqf4aSER5cRcyvwKnCgqqYZNmn+bkybO4FrgF3tBBdJWhGTOkm92Q1crKrXLxVU1edVdXKxBkm2ANcDLzEkd0v1HDBTVd+0eH9X1Wtj2uwHjgEfAg+P9OXZNto35yiepKW4dq07IElX2FbgsyW22QfMAieBO5Jsqqpflxhz7HTrAk8ADwBTwDPAO638eeDWqrqQ5MYlfqakdcyROkkaRs1mq+of4D3g8dUMlmQH8FtV/cBwxu7dSW5ql+eAt5McAP5azX5I6otJnaTenAW2T1o5yV3AbcBHSc4xjNpdNgWbZKYtbDiz0pjt86davHngBuCxdu0hhkO87wE+TeKMiqSJpKrWug+SdMW0hRKngDer6o1Wtg3Y+F/v1SU5DPxRVUdGyr4H7q+q8xPG3Aa8D+ytqm+TbACeGn2vb6TuBuA8sLOqfmplu4GXgT3ALVV1Lsl1rd50Vf0++R2QtF45UiepKzX8U30E2NO2NDkLHAF+WaTJPobVsqNOtPJJY84Bh4DjSb4GvgQ2L1J9F/DjpYSu+QSYBm4G3kryBXAaOGpCJ2lSjtRJkiR1wJE6SZKkDvgCrqR1oS2IOLag+EJV7VzFmC9y+Urad6tqZrViSlq/nH6VJEnqgNOvkiRJHTCpkyRJ6oBJnSRJUgdM6iRJkjrwL3bqgv87IvJnAAAAAElFTkSuQmCC", 666 | "text/plain": [ 667 | "

" 668 | ] 669 | }, 670 | "metadata": { 671 | "needs_background": "light" 672 | }, 673 | "output_type": "display_data" 674 | } 675 | ], 676 | "source": [ 677 | "\n", 678 | "model = load_model('model.h5')\n", 679 | "X_test = np.load('X_test.npy')\n", 680 | "y_test = np.load('y_test.npy')\n", 681 | "Input_mean = np.load('X_mean.npy')\n", 682 | "Input_std = np.load('X_std.npy')\n", 683 | "Output_mean = np.load('y_mean.npy')\n", 684 | "Output_std = np.load('y_std.npy')\n", 685 | "\n", 686 | "\"\"\"\n", 687 | " equation for the stability ellipse is 1060x^2 + 44xy + 0.52y^2 - 372 = 0\n", 688 | "\"\"\"\n", 689 | "# prepare x and y coordinates for plotting the stability region\n", 690 | "y = np.linspace(-100, 100, 100000, endpoint=True)\n", 691 | "\n", 692 | "x_upper = list()\n", 693 | "x_lower = list()\n", 694 | "y_plot = list()\n", 695 | "\n", 696 | "for i in y:\n", 697 | " sqrt = np.sqrt(-2688000 * i**2 + 15772800000)\n", 698 | " if sqrt >= 0:\n", 699 | " y_plot.append(i)\n", 700 | " x_upper.append((-4400 * i + sqrt) / 212000)\n", 701 | " x_lower.append((-4400 * i - sqrt) / 212000)\n", 702 | " pass\n", 703 | " pass\n", 704 | "\n", 705 | "\n", 706 | "plt.figure(figsize=(10,10))\n", 707 | "\n", 708 | "# plot the first 10 samples and their trajectories\n", 709 | "y_predict = model.predict(X_test)\n", 710 | "y_predict = y_predict * Output_std + Output_mean\n", 711 | "y_test = y_test * Output_std + Output_mean\n", 712 | "X_plot = X_test * Input_std + Input_mean\n", 713 | "\n", 714 | "\n", 715 | "for i in range(10):\n", 716 | " if i == 0: # only add label to 1 data point\n", 717 | " plt.plot(X_plot[i, 0, 0], X_plot[i, 0, 1], marker=\"*\", markersize=15,color='orange')\n", 718 | " plt.plot(y_test[i, :, 0], y_test[i, :, 1], color='cyan', lw=2, label='Test')\n", 719 | " plt.plot(y_predict[i, :, 0], y_predict[i, :, 1], color='black', lw=2, ls=':', label='Predicted')\n", 720 | " else:\n", 721 | " plt.plot(X_plot[i, 0, 0], X_plot[i, 0, 1], marker=\"*\", markersize=15,color='orange')\n", 722 | " plt.plot(y_test[i, :, 0], y_test[i, :, 1], color='cyan', lw=2)\n", 723 | " plt.plot(y_predict[i, :, 0], y_predict[i, :, 1], color='black', lw=2, ls=':')\n", 724 | "\n", 725 | " \n", 726 | "# plot stability region \n", 727 | "plt.plot(x_lower, y_plot, color='steelblue')\n", 728 | "plt.plot(x_upper, y_plot, color='steelblue')\n", 729 | "plt.ylim([-100, 100])\n", 730 | "plt.xlim([-2, 2])\n", 731 | "\n", 732 | "plt.xlabel(\"C_A - C_As\")\n", 733 | "plt.ylabel(\"T - T_s\")\n", 734 | "plt.legend()\n", 735 | "plt.show() \n", 736 | " " 737 | ] 738 | } 739 | ], 740 | "metadata": { 741 | "kernelspec": { 742 | "display_name": "Python 3", 743 | "language": "python", 744 | "name": "python3" 745 | }, 746 | "language_info": { 747 | "codemirror_mode": { 748 | "name": "ipython", 749 | "version": 3 750 | }, 751 | "file_extension": ".py", 752 | "mimetype": "text/x-python", 753 | "name": "python", 754 | "nbconvert_exporter": "python", 755 | "pygments_lexer": "ipython3", 756 | "version": "3.8.10" 757 | } 758 | }, 759 | "nbformat": 4, 760 | "nbformat_minor": 2 761 | } 762 | --------------------------------------------------------------------------------