├── Figures ├── LCCDE_Overview.jpg ├── MTH-IDS_Overview.png ├── README.md └── Tree-based_IDS_Overview.jpg ├── LCCDE_IDS_GlobeCom22.ipynb ├── LCCDE_IDS_GlobeCom22_paper.pdf ├── LICENSE ├── MTH_IDS_IoTJ.ipynb ├── MTH_IDS_paper.pdf ├── README.md ├── Tree-based_IDS_GlobeCom19.ipynb ├── Tree-based_IDS_paper.pdf └── data ├── CICIDS2017_sample.csv ├── CICIDS2017_sample_km.csv └── README.md /Figures/LCCDE_Overview.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/Figures/LCCDE_Overview.jpg -------------------------------------------------------------------------------- /Figures/MTH-IDS_Overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/Figures/MTH-IDS_Overview.png -------------------------------------------------------------------------------- /Figures/README.md: -------------------------------------------------------------------------------- 1 | # The figures used in this GitHub repository 2 | -------------------------------------------------------------------------------- /Figures/Tree-based_IDS_Overview.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/Figures/Tree-based_IDS_Overview.jpg -------------------------------------------------------------------------------- /LCCDE_IDS_GlobeCom22.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles\n", 8 | "This is the code for the paper entitled \"**LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles**\" accepted in 2022 IEEE Global Communications Conference (GLOBECOM). \n", 9 | "Authors: Li Yang (lyang339@uwo.ca), Abdallah Shami (Abdallah.Shami@uwo.ca), Gary Stevens, and Stephen de Rusett \n", 10 | "Organization: The Optimized Computing and Communications (OC2) Lab, ECE Department, Western University, Ontario, Canada; S2E Technologies, St. Jacobs, Ontario, Canada \n", 11 | "\n", 12 | "If you find this repository useful in your research, please cite: \n", 13 | "L. Yang, A. Shami, G. Stevens, and S. DeRusett, “LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles,\" in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6." 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "## Import libraries" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 1, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "import warnings\n", 30 | "warnings.filterwarnings(\"ignore\")" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 2, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "import pandas as pd\n", 40 | "import numpy as np\n", 41 | "import matplotlib.pyplot as plt\n", 42 | "import seaborn as sns\n", 43 | "from sklearn.model_selection import train_test_split\n", 44 | "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n", 45 | "import lightgbm as lgb\n", 46 | "import catboost as cbt\n", 47 | "import xgboost as xgb\n", 48 | "import time\n", 49 | "from river import stream\n", 50 | "from statistics import mode" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "## Read the sampled CICIDS2017 dataset\n", 58 | "The CICIDS2017 dataset is publicly available at: https://www.unb.ca/cic/datasets/ids-2017.html \n", 59 | "Due to the large size of this dataset, the sampled subsets of CICIDS2017 is used. The subsets are in the \"data\" folder. \n", 60 | "If you want to use this code on other datasets (e.g., CAN-intrusion dataset), just change the dataset name and follow the same steps. The models in this code are generic models that can be used in any intrusion detection/network traffic datasets." 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 3, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "df = pd.read_csv(\"./data/CICIDS2017_sample_km.csv\")" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "outputs": [ 77 | { 78 | "data": { 79 | "text/plain": [ 80 | "0 18225\n", 81 | "3 3042\n", 82 | "6 2180\n", 83 | "1 1966\n", 84 | "5 1255\n", 85 | "2 96\n", 86 | "4 36\n", 87 | "Name: Label, dtype: int64" 88 | ] 89 | }, 90 | "execution_count": 4, 91 | "metadata": {}, 92 | "output_type": "execute_result" 93 | } 94 | ], 95 | "source": [ 96 | "df.Label.value_counts()" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "**Corresponding Attack Types:** \n", 104 | "0 BENIGN   18225 \n", 105 | "3 DoS     3042 \n", 106 | "6 WebAttack   2180 \n", 107 | "1 Bot     1966 \n", 108 | "5 PortScan   1255 \n", 109 | "2 BruteForce   96 \n", 110 | "4 Infiltration   36 " 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "## Split train set and test set" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 5, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "X = df.drop(['Label'],axis=1)\n", 127 | "y = df['Label']\n", 128 | "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, random_state = 0) #shuffle=False" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": { 134 | "collapsed": true 135 | }, 136 | "source": [ 137 | "## SMOTE to solve class-imbalance" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "data": { 147 | "text/plain": [ 148 | "0 14569\n", 149 | "3 2430\n", 150 | "6 1728\n", 151 | "1 1579\n", 152 | "5 1024\n", 153 | "2 82\n", 154 | "4 28\n", 155 | "Name: Label, dtype: int64" 156 | ] 157 | }, 158 | "execution_count": 6, 159 | "metadata": {}, 160 | "output_type": "execute_result" 161 | } 162 | ], 163 | "source": [ 164 | "pd.Series(y_train).value_counts()" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 7, 170 | "metadata": {}, 171 | "outputs": [], 172 | "source": [ 173 | "from imblearn.over_sampling import SMOTE\n", 174 | "smote=SMOTE(n_jobs=-1,sampling_strategy={2:1000,4:1000})" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 8, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "X_train, y_train = smote.fit_resample(X_train, y_train)" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": 9, 189 | "metadata": {}, 190 | "outputs": [ 191 | { 192 | "data": { 193 | "text/plain": [ 194 | "0 14569\n", 195 | "3 2430\n", 196 | "6 1728\n", 197 | "1 1579\n", 198 | "5 1024\n", 199 | "4 1000\n", 200 | "2 1000\n", 201 | "Name: Label, dtype: int64" 202 | ] 203 | }, 204 | "execution_count": 9, 205 | "metadata": {}, 206 | "output_type": "execute_result" 207 | } 208 | ], 209 | "source": [ 210 | "pd.Series(y_train).value_counts()" 211 | ] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "metadata": {}, 216 | "source": [ 217 | "## Machine Learning (ML) model training\n", 218 | "### Training three base learners: LightGBM, XGBoost, CatBoost" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 10, 224 | "metadata": {}, 225 | "outputs": [ 226 | { 227 | "name": "stdout", 228 | "output_type": "stream", 229 | "text": [ 230 | " precision recall f1-score support\n", 231 | "\n", 232 | " 0 1.00 1.00 1.00 3656\n", 233 | " 1 0.99 0.99 0.99 387\n", 234 | " 2 1.00 1.00 1.00 14\n", 235 | " 3 1.00 1.00 1.00 612\n", 236 | " 4 1.00 0.75 0.86 8\n", 237 | " 5 0.99 1.00 0.99 231\n", 238 | " 6 1.00 1.00 1.00 452\n", 239 | "\n", 240 | " accuracy 1.00 5360\n", 241 | " macro avg 1.00 0.96 0.98 5360\n", 242 | "weighted avg 1.00 1.00 1.00 5360\n", 243 | "\n", 244 | "Accuracy of LightGBM: 0.9970149253731343\n", 245 | "Precision of LightGBM: 0.9970231077536348\n", 246 | "Recall of LightGBM: 0.9970149253731343\n", 247 | "Average F1 of LightGBM: 0.9969877746104384\n", 248 | "F1 of LightGBM for each type of attack: [0.99795054 0.99092088 1. 0.997543 0.85714286 0.99354839\n", 249 | " 0.99778271]\n" 250 | ] 251 | }, 252 | { 253 | "data": { 254 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw+ElEQVR4nO3dd3wUdf7H8ddnkyBSpAgCSVAQsBdAilgQW8CCcBZURNEfJxZUUE/ELiqn3lk5PRUFARvEghRRQAQRTiFIJ/SipFCUooBiyuf3x05gxWySXWZ2l93P08c82Hx3dt4zi/nwnfYdUVWMMcb8lS/aK2CMMbHKCqQxxgRhBdIYY4KwAmmMMUFYgTTGmCCsQBpjTBDJ0V6BMonYNUjGRIOqhPOxgp/Whvw7m1Ln6LCyIiGmC2TBljVRy06p24TklNSo5RcW5CVsfmFBHoDlRznfxHiBNMYcZIqLor0GrrICaYxxjxZHew1cZQXSGOOeYiuQxhhTKrUepDHGBGE9SGOMCSLOepB2obgxxj3FRaFP5RCRyiIyR0QWishSERnotA8XkXUissCZmjvtIiKDRWS1iCwSkZYBy+opIqucqWd52daDNMa4x5se5B7gPFXdKSIpwEwR+dx57z5V/Wi/+S8CmjlTW+A1oK2I1AYeA1oBCnwvIuNUdVuwYOtBGmPcU1wc+lQO9dvp/JjiTGXdsdMFGOl87jugpog0ADoCU1R1q1MUpwCdysq2AmmMcY1qcciTiPQWkbkBU+/9lysiSSKyANiMv8jNdt4a5OxGvygihzhtacCGgI/nOG3B2oOyXWxjjHvCOIutqkOAIeXMUwQ0F5GawBgROQl4ANgIVHI+fz/wRMgrUAbrQRpj3KPFoU+hLF51OzAN6KSq+c5u9B7gbaCNM1su0DDgY+lOW7D2oKxAGmPc481Z7LpOzxERORS4EFjuHFdERAToCixxPjIOuME5m306sENV84FJQIaI1BKRWkCG0xbUQVUg9+z5g2v+3pfLe95Ol+tu4ZW33gFAVXn5jeFccs3f6dy9N+9+OBaAOfMWcXrGFVzRsw9X9OzDa8PeAyB/0xZuuuN+LruuN12uu4V3Mj91bR3fHPI8eTkLWTB/qmvLDFXHjA4sXTKD5dkz6X9fH8uPoEMOOYRvZ03g+7lTWLjgKx579N6I5kf7u/eoB9kAmCYii4As/McgJwDvichiYDFQB3jKmX8isBZYDbwJ3A6gqluBJ51lZAFPOG1BSSw/9nX/seVUld9++50qVQ6loLCQG277BwP63sLaHzYwZ94iBj10Dz6fj5+3befwWjWZM28Rwz/4mP/+e+Cflrvlp61s+XkrJxzblF27dtOt110MfvoRmjQ+au884Q53dvZZbdm5cxdvv/0yzVucH+aWhz/cmM/nY9nSb+h08bXk5OTz3bcT6XH97SxbtuqgyT+Q4b6inQ9QtWoVdu3aTXJyMjOmj+Huex5j9px5nue7+Xcf7niQe5ZNC7mgHHL8uTE7HqRnPUgROU5E7ncu2BzsvD7+AJdJlSqHAlBYWEhhYSEiwugxn3HbTd3x+fybc3itmmUup26d2pxwbFPA/z/z0Uc1ZNOWnw9k1fb6ZuZstm7b7sqywtGmdQvWrFnPunU/UlBQQGbmWC7r3NHyI2jXrt0ApKQkk5ySQqQ6IbGw7V5c5hNNnhRIEbkfGAUIMMeZBPhARAYcyLKLioq4omcf2l96Le1at+CUE49jQ24+n0/9mm7/dxe33vsIP2zYd9x14ZJlXN7zdm699xFWr/3hL8vLzd/EslVrOOXEYw9ktWJGalp9NuTsG/A0Jzef1NT6lh9BPp+PuVmTyc9dxNSpM5iTNT8iubGw7V6fpIk0r3qQvYDWqvqMqr7rTM/gP8vU60AWnJSUxMcjXmXqmHdYnL2SVWvX80dBAYdUqkTmsMFc0bkTj/zzRQBOOLYJUz4ewScj/kv3Kzpz1wN/vgJg9+7fuPuhp7j/rluoVrXqgayWMXsVFxfTqnUGRzVuRetWLTgxTv7xrRDrQVZIMVDaAZQGzntBBV40+tbID4LOd1j1arRpeQozv5tL/bp1uOCcMwG44JwzWLlmHQDVqlbdu0ve/ow2FBYWsm37DgAKCgvp99BTXJJxLhd2ODP0LYxRebkbaZi+76tPT2tAXt5Gy4+CHTt+YfrXs+iY0SEiebGw7apFIU+xzKsC2Q+YKiKfi8gQZ/oCmAr0LeuDqjpEVVupaqu/33Dtn97bum07v/zqv+Po9z17+DZrPo2Pash57dsxZ95CALLmL+aohv6L43/6eeve4z+Ls1dQrErNGoehqjz69EscfVRDel5zuasbHm1ZcxfQtGljGjVqSEpKCt26dWH8hMmWHyF16tSmRo3DAKhcuTIXnN+eFSsi82ylaG87EHe72J7cSaOqX4jIMfh3qUtu5ckFsvQA/snY8vM2HnrqOYqKi9FipeN5Z9PhzLa0POVE7h/4L94Z/SlVDq3MwAH9AJg8bSajx3xGUnISlStV4t8DByAizFu4hPFfTKVZk0Zc0dN/KUTfW3rS/ow2ZaRXzLvvvMo57dtRp05t1q+dy8AnnuPt4aMOeLkVVVRURN9+DzPxs/dJ8vkYPmI02dkrLT9CGjSox7ChL5GU5MPn8/HRR+P5bOKXEcmO9rYDMb/LHKqD6jKfSLKnGtpTDRM6P8zLfH7//tOQf2crn9Y1Zi/zsXuxjTHusacaGmNMEDF+TDFUViCNMe6Js2OQViCNMe6xHqQxxgRhPUhjjAnCCqQxxpQu1u+MCZUVSGOMe6wHaYwxQcTZSZqDakRxY4yJJOtBGmPcY7vYxhgTRJztYluBNMa4x3qQxhgThPUgIyelbpOo5pcMO2X5lp+I+WGxHqQxxgRhBTJyoj1g7HF1W0Utf/mWuVHf/oQeMNbyw2O72MYYE4T1II0xJgjrQRpjTBDWgzTGmCDirAdp92IbY9xTXBz6VA4RqSwic0RkoYgsFZGBTntjEZktIqtFZLSIVHLaD3F+Xu283yhgWQ847StEpGN52VYgjTHu8aBAAnuA81T1VKA50ElETgeeBV5U1abANqCXM38vYJvT/qIzHyJyAnANcCLQCfiviCSVFWwF0hjjHtXQp3IXqaqqO50fU5xJgfOAj5z2EUBX53UX52ec988XEXHaR6nqHlVdB6wG2pSVbQXSGOMeb3qQiEiSiCwANgNTgDXAdlUtdGbJAdKc12nABgDn/R3A4YHtpXymVFYgjTHuCaNAikhvEZkbMPXef7GqWqSqzYF0/L2+4yKxOXYW2xjjnjDOYqvqEGBIBefdLiLTgHZATRFJdnqJ6UCuM1su0BDIEZFkoAbwc0B7icDPlMp6kMaYmCYidUWkpvP6UOBCYBkwDbjSma0nMNZ5Pc75Gef9r1RVnfZrnLPcjYFmwJyysq0HaYxxjzcXijcARjhnnH1ApqpOEJFsYJSIPAXMB4Y68w8F3hGR1cBW/GeuUdWlIpIJZAOFQB8t5zGMcdmDXL3yO+bP+5K5WZP57tuJnmRUOqQSmV8M59Np7zF+xmju7O8/bHL62a35+Mt3GPPVe7w3/k2ObJwOwI23dmfCN6MZO/193v7ov6Sm1/dkvQA6ZnRg6ZIZLM+eSf/7+niWY/mxlR0L+R6dxV6kqi1U9RRVPUlVn3Da16pqG1VtqqpXqeoep/135+emzvtrA5Y1SFWbqOqxqvp5edlxWSABLrjwKlq1zuD0dhd7svw/9vzBjVfcRtdzr+Nv53XnrHPbceppJ/H4v+7nvtse4W/nXceETyZx293+S7OWLV7BlRk30KVDdyZNmMo/Hr3Lk/Xy+XwMfnkQl3buwcmnnsvVV3fl+OObeZJl+bGTHQv5gGdnsaMlbgtkJOze9RsAySnJJKcko6qoQrXqVQGoXr0amzduAWD2rO/5/bc9ACycu5j6qUd4sk5tWrdgzZr1rFv3IwUFBWRmjuWyzuXeMGD5B3l2LOQDViAPBqrK5xM/YPZ3n/P3Xtd5luPz+Rjz1XvMyp7M/76ezaJ5S3n47qcY8v5LTF8wgcuuuoghg0f85XNXXteFGVP/58k6pabVZ0POvvH8cnLzSU31bnfe8mMjOxbyAf9Z7FCnGBaVAikiN3m5/HPO/Rtt2nbi0s49uO22Gzn7rLae5BQXF/O3866jw6mXcEqLE2l2XBN63tqd3t370aH5pXwyajwDnuj3p890vvIiTjz1eIa++o4n62RMNGmxhjzFsmj1IAcGeyPwotHi4l1hLTwvbyMAW7b8zNixn9O6dfOwllNRv/6yk9mzvufs89tx3InNWDRvKQCffzqFFq1P2Ttfu/ZtuLXfTdx+w70U/FHgybrk5W6kYfq+kajT0xrs/T4iIZHzE3nb97Jd7IoRkUVBpsVAvWCfU9UhqtpKVVv5fFVDzq1S5VCqVau69/WFF5zD0qUrwt6OYGodXpPqh1UD4JDKh3DGOW1Yu3I91atXo9HRRwJwxjltWbtqPQDHn3QMA597gNuvv5etP21zfX1KZM1dQNOmjWnUqCEpKSl069aF8RMme5Zn+bGRHQv5QNztYnt5HWQ9oCP+UTYCCeDNATigXr26fPSh/3Ko5OQkRo36lEmTp7ueU7deHZ75z+MkJfkQ8fHFuC+ZPmUmj9w7iMHDnqVYi/ll+6882O9JAO57vC9Vqh7KS0OfASA/ZyO333Cv6+tVVFRE334PM/Gz90ny+Rg+YjTZ2Stdz7H82MqOhXwAYnyXOVSiFbgOKawFiwwF3lbVmaW8976qdi9vGcmV0qL2bdtDu+yhXQmdryrhfHb3f24P+Xe2yp3/DSsrEjzrQapqrzLeK7c4GmMOQjF+TDFUdquhMcY9Hu2RRosVSGOMe6wHaYwxQcTZSRorkMYY98T4ZTuhsgJpjHFPnPUg4/JebGOMcYP1II0xrlE7SWOMMUHE2S62FUhjjHvsJI0xxgRhPUhjjAnCjkEaY0wQ1oM0xpgg7BikMcYEYT3IyCkZFy9alm+ZG9X8aG+/5Sd2fjjsOsgIStQBYxM9PyYGjLX88FgP0hhjgrACaYwxQdhJGmOMCSLOepA2mo8xxjVarCFP5RGRhiIyTUSyRWSpiPR12h8XkVwRWeBMFwd85gERWS0iK0SkY0B7J6dttYgMKC/bepDGGPd404MsBO5V1XkiUh34XkSmOO+9qKrPBc4sIicA1wAnAqnAlyJyjPP2q8CFQA6QJSLjVDU7WLAVSGOMezy4zEdV84F85/WvIrIMSCvjI12AUaq6B1gnIquBNs57q1V1LYCIjHLmDVogbRfbGHPQEJFGQAtgttN0h4gsEpFhIlLLaUsDNgR8LMdpC9YelBVIY4x7ijXkSUR6i8jcgKl3aYsWkWrAx0A/Vf0FeA1oAjTH38N83u3NsV1sY4x7wjgGqapDgCFlzSMiKfiL43uq+onzuU0B778JTHB+zAUaBnw83WmjjPZSWQ/SGOMaVQ15Ko+ICDAUWKaqLwS0NwiY7W/AEuf1OOAaETlERBoDzYA5QBbQTEQai0gl/CdyxpWVbT1IY4x7vDmLfSZwPbBYRBY4bQ8C14pIc0CB9cAtAKq6VEQy8Z98KQT6qGoRgIjcAUwCkoBhqrq0rGArkMYY93hQIFV1JiClvDWxjM8MAgaV0j6xrM/tzwqkMcY1Fbnw+2ASl8cgO2Z0YOmSGSzPnkn/+/rEff6bQ54nL2chC+ZP/ct7d/e7hcI/cjn88FqlfNIbifb9x0p2LOSHcxY7lsVdgfT5fAx+eRCXdu7Byaeey9VXd+X445vFdf7IkZlccul1f2lPT0/lwgva88MPOZ7mB0rE7z8WsmMhH4DiMKYYFncFsk3rFqxZs551636koKCAzMyxXNa5Y/kfPIjzv5k5m63btv+l/fnnHmfAg4MqdKbQLYn4/cdCdizkgzf3YkeTZwVSRI4TkfOdizsD2zt5lQmQmlafDTn7BvzMyc0nNbW+l5ExlV+ic+cMcnPzWbQo6F1Unoj29kczP5G3fS/bxS6fiNwFjAXuBJaISJeAt//pRabZ59BDK/PA/Xfy+MDnyp/ZGDfF2S62V2exbwZOU9Wdzr2TH4lII1V9mdJP1+/l3GbUG0CSauDzVQ0pOC93Iw3T9w1Vn57WgLy8jSGufviinQ/QpEkjGjU6knlz/QOepKc3IGv2JNqdeQmbNm3xNDva2x/N/ETe9hKxvsscKq92sX2quhNAVdcDHYCLROQFyimQqjpEVVupaqtQiyNA1twFNG3amEaNGpKSkkK3bl0YP2Fy6FsQpmjnAyxZspzU9FNpeszpND3mdHJy8mndtqPnxRGiv/3RzE/kbd/LepAVsklEmqvqAgCnJ3kpMAw42aNMAIqKiujb72EmfvY+ST4fw0eMJjt7pZeRUc9/951XOad9O+rUqc36tXMZ+MRzvD18lKeZwSTi9x8L2bGQD/HXgxQvznCKSDpQqKp/6d+LyJmqOqsiy0mulBa1bzuRnyoY7fxYeapfQuerlrmnF8zWLueE/Dtbe+zXYWVFgic9SFUNeuFdRYujMebgE2fP7Iq/6yCNMcYtdi+2McY9cdaDtAJpjHFNvO1iW4E0xrjHCqQxxpTOepDGGBOEFUhjjAnCCqQxxgQT3vXlMcsKpDHGNdaDNMaYILTYepDGGFMq60EaY0wQYY5xEbOsQBpjXGM9SGOMCcKOQUZQybh4lm/5ln9wiOADNCMipgtkog4YGyv5HdLOj0r29NypQIIPWBsD+eGwHqQxxgQRbwXSBsw1xsQ0EWkoItNEJFtElopIX6e9tohMEZFVzp+1nHYRkcEislpEFolIy4Bl9XTmXyUiPcvLtgJpjHGNauhTBRQC96rqCcDpQB8ROQEYAExV1WbAVOdngIuAZs7UG3gN/AUVeAxoC7QBHispqsFYgTTGuEaLJeSp3GWq5qvqPOf1r8AyIA3oAoxwZhsBdHVedwFGqt93QE0RaQB0BKao6lZV3QZMATqVlW3HII0xrvH6QnERaQS0AGYD9VQ133lrI1DPeZ0GbAj4WI7TFqw9qHJ7kCJST0SGisjnzs8niEiv8jfFGJNotDj0SUR6i8jcgKl3acsWkWrAx0A/Vf3lT7n+51e7fpFRRXaxhwOTgJJrDlYC/dxeEWPMwa9YJeRJVYeoaquAacj+yxWRFPzF8T1V/cRp3uTsOuP8udlpzwUaBnw83WkL1h5URQpkHVXNxHnahKoWAkUV+JwxJsGoSshTeUREgKHAMlV9IeCtcUDJmeiewNiA9hucs9mnAzucXfFJQIaI1HJOzmQ4bUFV5BjkLhE5HKf7WhJYgc8ZYxKMR9dBnglcDywWkQVO24PAM0Cmc8jvB6Cb895E4GJgNbAbuAlAVbeKyJNAljPfE6q6tazgihTIe/BX5CYiMguoC1xZse0yxiQSL241VNWZQLDK+5fbvZzjkX2CLGsYMKyi2eUWSFWdJyLnAMc6K7lCVQsqGmCMSRzxdidNuQVSRG7Yr6mliKCqIz1aJ2PMQao4AceDbB3wujL+Lu08wAqkMeZP4m3A3HLPYqvqnQHTzUBLoJr3qxaeQw45hG9nTeD7uVNYuOArHnv03oivQ8eMDixdMoPl2TPpf1+ph0IO+vxqh1Vl4BuPMnL6MEZMG8oJLY+nes3qPPf+s7z7zXCee/9ZqtXw/29SrUY1nnzrcYZOGcJrE16h8bGNPFmnEtH8/hPh774sHt1qGDXh3Gq4C2js9oq4Zc+ePVyQ0Y3TWl3Iaa0y6JjRgbZtWpb/QZf4fD4GvzyISzv34ORTz+Xqq7ty/PHN4i7/joF9mDM9ixs6/B+9Mm7hx9U/0r3PNcybNZ8eZ9/IvFnz6d7nGgB63Nmd1UvX0OvC3jzd91nuGHi76+tTIprff6L83ZclnOsgY1lF7qQZLyLjnGkCsAIYU4HPtRGR1s7rE0TkHhG5+MBXuXy7du0GICUlmeSUFDSC/0y1ad2CNWvWs27djxQUFJCZOZbLOneMq/yq1atyatuT+eyDzwEoLChk5y+7ODPjDL74cDIAX3w4mbM6ngnAUc2OYt6s+QD8uGYD9dPrU6tOTVfXqUQ0v/9E+LsvjxfXQUZTRXqQzwHPO9PTQHtVHVDWB0TkMWAw8JqIPA28AlQFBojIQwe2yuXz+XzMzZpMfu4ipk6dwZys+V5H7pWaVp8NOfsGHM3JzSc1tX5c5TdoWJ/tW3cw4IX7ePOL17nv3/dQ+dDK1K5Ti62b/ZeVbd28ldp1/AOlrMleQ/uLzgbguObHUj+9HnUb1HV1nUpE8/tPhL/78iTULraIJAGPq+rXzjRLVXMqsNwr8V/c2R7/9UhdVfVJ/KNpXH2gK12e4uJiWrXO4KjGrWjdqgUnnnis15EJJSk5iWNOasbYd8Zzc6db+W3373t3pwOV9Nzff3UU1Q6ryluTXufym7qyaslqiovi7OlOBoi/Xewyz2KrapGIFItIDVUN5e6ZQlUtAnaLyJqSG8tV9TcRKfM3w7lRvTeAJNXA56saQuyf7djxC9O/nuU/cL10RdjLCUVe7kYapu8bKj89rQF5eRsjkh2p/C35W9iSv4Vl85cD8PVnM+je51q2/rSN2kfU9vcej6jNtp+3A7B7526evfe5vZ8f9e275P2YX9qiD1g0v/9E+LsvT6zvMoeqIrvYO/Hf4jPUGaV3sIgMLuczf4hIFef1aSWNIlID557uYAJvXA+nONapU5saNQ4DoHLlylxwfntWrFgT8nLClTV3AU2bNqZRo4akpKTQrVsXxk+YHFf5W7dsY3PeFhoenQ7AaWe15IdVP/C/Kd/S6aoMADpdlcGsyf8D/Ge8k1P8/xZf0v1iFs5ezO6du11dpxLR/P4T4e8+0VTkOshPnClQeUcO2qvqHgDVPz0pN4V9N5d7okGDegwb+hJJST58Ph8ffTSezyZ+6WXknxQVFdG338NM/Ox9knw+ho8YTXb2yrjLH/zIKzz8nwdIrpRC/g/5PHPvv/GJj8def5iLr+nEppzNPH7bkwAc2fRIHnjpflSV9SvX869/PO/6+pSI5vefKH/3ZYn1XeZQSXlneEWkr6q+XF6bF5IrpUXtEG4sPFUw2vn2VMMEzg9zX/m71MtD/p09Pe+TmK2qFdnFLq3Hd6PL62GMiQMJc5JGRK4FugONRWRcwFvVgTKHCDLGJKZ4O0lT1jHI/wH5QB3810CW+BVY5OVKGWMOTvF28VbQAqmqP+AfhLJdWQsQkW9Vtcx5jDGJQYMO23hwcuOphpVdWIYxJg4Ux/idMaFyo0DG2VdijAlXsfUgjTGmdPG2i12R0XzudJ4AFnQWF9fHGHMQKw5jimUVuQ6yHpAlIpki0sl5BGOg6z1YL2PMQUiRkKdYVpERxR8GmuF/Lu2NwCoR+aeINHHeX+LpGhpjDhqJ2IMseYziRmcqBGoBH4nIvzxcN2PMQSbeCmRFnmrYF7gB+Al4C7hPVQtExAesAvp7u4rGmINFrO8yh6oiZ7FrA5c7F47vparFInKpN6tljDkYxdljscsvkKr6WBnvLXN3dYwxB7N4uw4ynKcaGmNMQojpC8VLxsWz/OgoGZcxWqK9/YmeH454u60upgtktAeMtfwEHjDW8sPixVlpERkGXApsVtWTnLbHgZuBLc5sD6rqROe9B4BeQBFwl6pOcto7AS8DScBbqvpMedkxXSCNMQeX4r/cR+KK4fgfHT1yv/YXVfW5wAYROQG4BjgRSAW+FJFjnLdfBS4EcvDf/DJOVbPLCrYCaYxxjRe72Ko6Q0QaVXD2LsAo55lY60RkNdDGeW+1qq4FEJFRzrxlFkg7SWOMcU2ELxS/Q0QWiciwgPEi0oANAfPkOG3B2stkBdIY45piCX0Skd4iMjdg6l2BqNeAJkBz/E8+8ORRmbaLbYxxTTjXQarqEGBIiJ/ZVPJaRN4EJjg/5gINA2ZNd9oooz0o60EaY1yjYUzhEJEGAT/+DSgZNGcccI2IHCIijfEPtDMHyAKaiUhjEamE/0RO4MMIS2U9SGOMa7y41VBEPgA6AHVEJAd4DOggIs3x19j1wC0AqrpURDLxn3wpBPqoapGznDuASfgv8xmmqkvLy7YCaYxxjRfXQarqtaU0Dy1j/kHAoFLaJwITQ8m2AmmMcY3dSWOMMUEk3Gg+xhhTUbE+AG6orEAaY1xjBdIYY4LQONvFjrvrIN8c8jx5OQtZMD86Q3VFOx+gY0YHli6ZwfLsmfS/r0/C5deocRijRw1hyeKvWbxoOqe3PS1i2dHe9mjnx9szaeKuQI4cmckll16XsPk+n4/BLw/i0s49OPnUc7n66q4cf3yzhMkHePGFJ5g0aRonnXwOLU+7kGXLV0UkN9rbHu38eBSxAiki+w9V5IlvZs5m67btkYiKyfw2rVuwZs161q37kYKCAjIzx3JZ544Jk3/YYdU5+6y2DHv7AwAKCgrYseOXiGRHe9ujnQ/Wg6wQERm33zQeuLzkZy8yjV9qWn025Owb8DQnN5/U1PoJk9+48ZH89NPPDH3rRbLmTOKN1/9NlSqHRiQ72tse7XyI3K2GkeJVDzId+AV4Af8oG88Dvwa8NsYTyUlJtGhxMm+8MZLWbTqya9du7u9/R7RXK2GEM5pPLPOqQLYCvgceAnao6nTgN1X9WlW/LuuDgUMfFRfv8mj14lde7kYapu8bqj89rQF5eRsTJj8nN5+cnHzmZM0H4JNPPqNF85Mjkh3tbY92PtgudoWoarGqvgjcBDwkIq9QwUuKVHWIqrZS1VY+X1UvVi+uZc1dQNOmjWnUqCEpKSl069aF8RMmJ0z+pk1byMnJ45hjmgBw3nlnsWzZyohkR3vbo50P8VcgPb0OUlVzgKtE5BL8u9yee/edVzmnfTvq1KnN+rVzGfjEc7w9fFQkomMiv6ioiL79HmbiZ++T5PMxfMRosrMjUyBiIR+g792PMHLEf6hUKYV1636k19/viUhutLc92vkQ+8cUQyWqsbtJyZXSorZyifxUwWjnx8pT/RI6X8O75PtfR/UI+Xe2/w/vxuyRSLuTxhjjmljfZQ6VFUhjjGtid380PFYgjTGuKY6zEmkF0hjjGtvFNsaYIOKr/2gF0hjjIutBGmNMELF+62CorEAaY1xjJ2mMMSaI+CqPcThgrjHGuMV6kMYY19hJGmOMCcKOQRpjTBDxVR6tQBpjXGS72BFUMuyT5Vu+5R8c4m0X285iG2Nc48VDu0RkmIhsFpElAW21RWSKiKxy/qzltIuIDBaR1SKySERaBnympzP/KhHpWZHtiekeZKIOGJvo+TExYCzQoMbxUcnP37EMiP72h8OjXezhwCtA4KOjBwBTVfUZERng/Hw/cBHQzJnaAq8BbUWkNvAY/udlKfC9iIxT1W1lBVsP0hjjGg3jv3KXqToD2LpfcxdghPN6BNA1oH2k+n0H1BSRBkBHYIqqbnWK4hSgU3nZViCNMa4J56FdgU8ydabeFYiqp6r5zuuNQD3ndRqwIWC+HKctWHuZYnoX2xhzcAnnJI2qDgGGhJupqioinpwdsh6kMcY1XpykCWKTs+uM8+dmpz0XaBgwX7rTFqy9TFYgjTGuKUZDnsI0Dig5E90TGBvQfoNzNvt0YIezKz4JyBCRWs4Z7wynrUy2i22McY0XZ7FF5AOgA1BHRHLwn41+BsgUkV7AD0A3Z/aJwMXAamA3cBOAqm4VkSeBLGe+J1R1/xM/f2EF0hjjmoqclQ55marXBnnr/FLmVaBPkOUMA4aFkm0F0hjjGrvV0BhjgvCiBxlNdpLGGGOCsB6kMcY1tottjDFBFGt87WJbgTTGuCa+ymOcHoPsmNGBpUtmsDx7Jv3vK/WMv+VbfthS0+rz0fi3+fq78Uz/dhx/v7UHAP0fupOps8Yw5ZtPGPXJm9SrXxeAps0aM37y+6zftIBb77jJ9fUJFO3vPoIXikeEaAx3iZMrpYW8cj6fj2VLv6HTxdeSk5PPd99OpMf1t7Ns2aqQlhPucF+Wf+D5BzLcmZv5wYY7O6JeHerVr8vihcuoWq0Kk6Z/xP9ddyd5eRvZ+esuAHrd0oNjjm3C/fcM5PA6tUlvmMpFl5zP9u2/8Porb5eZH+5wZ27+3aMqIX3Ice1RXUP+nf3gh0/DyoqEiPQgReQsEblHRDK8zmrTugVr1qxn3bofKSgoIDNzLJd17uh1rOUnUP7mTT+xeKG/iO3auZtVK9dSv8ERe4sjQJUqh1LS+fj5p60snL+EgsJCV9djf9H+7iG80XximScFUkTmBLy+Gf9gl9WBx5zBLT2TmlafDTn7BvzMyc0nNbW+l5GWn8D56UemcvLJxzPv+0UADHi4L3OXTOXyqy7l3//8j2e5pYn2dw/xt4vtVQ8yJeB1b+BCVR2I/wbx6zzKNCaiqlStwtCRL/Pog0/v7T0+89TLtDrpfD75cAI39U68/9W9GDA3mrwqkD5n1IzD8R/n3AKgqruAMvczAgfPLC7eVdaspcrL3UjD9H3HbtLTGpCXtzHk5YTL8hMjPzk5maEjX+KTDycwcfyXf3n/kw8ncEnnC13PLUu0v3uwXeyKqgF8D8wFageM21YNKPOArKoOUdVWqtrK56sacnDW3AU0bdqYRo0akpKSQrduXRg/YXIYmxAey0+M/BdeeZJVK9fyxqsj9rY1Pvqova87Xnweq1etdT23LNH+7gFUNeQplnlyHaSqNgryVjHwNy8ySxQVFdG338NM/Ox9knw+ho8YTXb2Si8jLT/B8tuc3pKrrulC9tIVTPnmEwCefuIlul9/OU2aNqZYi8nZkMf9dw8EoO4RdfhiWibVq1ejWIu5+bbrOef0zn86qeOGaH/3EH+PfY27y3zckshPFYx2vj3VMAaeahjmZT6dj7w05N/Z8T9OiNnLfOxOGmOMa2L9pEuorEAaY1wTb7vYViCNMa6J5UN24bACaYxxTaxfthMqK5DGGNfE2zHIuBzNxxhj3GA9SGOMa+wkjTHGBGEnaYwxJgjrQRpjTBDxdpLGCqQxxjX20C5jjAkivsqjFUhjjIvi7RikXQdpjHGNV49cEJH1IrJYRBaIyFynrbaITBGRVc6ftZx2EZHBIrJaRBaJSMtwtyeme5Alw05ZvuVHQ8mwY9ES7e0Ph8eX+Zyrqj8F/DwAmKqqzzjPuhoA3A9cBDRzprbAa86fIbMepDHGNRF+aFcXoGRI9xFA14D2ker3HVCz5KkGoYrpHmSiDhib6PmxMmButPNPrdcuKvkLN30b9mc9vMxHgckiosAbqjoEqKeq+c77G4F6zus0YEPAZ3OctnxCFNMF0hhzcAlnF1tEeuN/+mmJIU4BDHSWquaKyBHAFBFZvl+uOsXTVVYgjTGuCWeX2SmG+xfE/efJdf7cLCJjgDbAJhFpoKr5zi70Zmf2XKBhwMfTnbaQ2TFIY4xrvHiqoYhUFZHqJa+BDGAJMA7o6czWExjrvB4H3OCczT4d2BGwKx4S60EaY1zj0XWQ9YAxIgL+mvW+qn4hIllApoj0An4AujnzTwQuBlYDu4Gbwg22AmmMcY0XJ2lUdS1waintPwPnl9KuQB83sm0X2xhjgrAepDHGNTZYhTHGBGHDnRljTBDWgzTGmCCsB2mMMUFYD9IYY4KwHqQxxgQRbz3IuLsOMj09lS8nf8iihdNYuOAr7ryjV8TXoWNGB5YumcHy7Jn0v8+V61Ut/yDJj2S2z+dj9JTh/OedfwPwxMsPMXHOR4z+cjijvxzOsSc22zvv/U/dzfhvM/nwq5Ecd/Ixnq2ThvFfLIu7HmRhYSH39R/I/AVLqFatKnNmf8GXU2ewbNmqiOT7fD4GvzyIThdfS05OPt99O5HxEyZbfgLkRzr7upu7sXbVeqpVr7q37YUnXuXLCdP+NN9Z57fjyKPT6dyuGye3PJGHn72PHhff7Mk6qRZ7stxo8aQHKSJtReQw5/WhIjJQRMaLyLMiUsOLzBIbN25m/oIlAOzcuYvly1eRllrfy8g/adO6BWvWrGfduh8pKCggM3Msl3XuaPkJkB/J7CMa1OXsC85gzHvjy5333I5nMz7zCwAWz1tK9cOqUeeIwz1ZrwgPmOs5r3axh+G/SRzgZaAG8KzT9rZHmX9x1FHpND/1JGbPmR+pSFLT6rMhZ99Q+Tm5+aRGsEBbfvTyI5nd/8l+vPjkqxTv12O7c0BvPvxqJP8YeBcplVIAfzHdlLdp7zyb8rdwRIO6nqyXF6P5RJNXBdKnqoXO61aq2k9VZ6rqQODosj4oIr1FZK6IzC0u3hX2ClStWoXM0W9yzz8e49dfd4a9HGNiTfsLz2DrT9tYtmjFn9oHD3qdLmddS/dOvahR6zD+744eEV8360FWzBIRKRliaKGItAIQkWOAgrI+qKpDVLWVqrby+aqWNWtQycnJfDj6TT74YAyffvp5WMsIV17uRhqm7xuqPz2tAXl5Gy0/AfIjld289Sl0yDiLiVkf8+zrT9D6zNP45yuP8dPmnwEo+KOAsaM+46QWJwCwOX8L9VLr7f18vQZ12Zy/xfX1AutBVtTfgXNEZA1wAvCtiKwF3nTe89SbQ55n2fLVvPRymYMUeyJr7gKaNm1Mo0YNSUlJoVu3LoyfMNnyEyA/UtmD//k6GS27cnHrK7j/1kfJmvU9D94x8E/HFc/t1J7Vy9cCMH3yTDp36wTAyS1PZOevu/YWU7cVq4Y8xTJPzmKr6g7gRudETWMnJ0dVN5X9yQN35hmtub7HlSxanM3cLP//nI888gyff/GV19EAFBUV0bffw0z87H2SfD6GjxhNdvbKiGRbfnTzo73tT//3cWodXhMRYcWSVTzZ/18AfPPl/zjr/HZM+O5Dfv/tdx7tN8izdYj1y3ZCJbHcxU2ulBa1lUvkpwpGOz9WnioY7fyoPtVQVcL5bL0ax4X8O7tpx/KwsiIh7i4UN8YYt8TdheLGmOiJ9bPSobICaYxxTSwfsguHFUhjjGti/ax0qKxAGmNcYz1IY4wJwo5BGmNMENaDNMaYIOwYpDHGBBFvd9JYgTTGuMZ6kMYYE0S8HYO0Ww2NMa7x6pk0ItJJRFaIyGoRGeDxZuxlPUhjjGu86EGKSBLwKnAhkANkicg4Vc12PWw/1oM0xrjGowFz2wCrVXWtqv4BjAK6eLohjpjuQZYM+2T5lp+I+Qs3fRvV/HB4dAQyDdgQ8HMO0NabqD+L6QIZ7ph0JUSkt6pGfljxKGdbvuVHK7/wj9yQf2dFpDfQO6BpSDS/u0Dxvovdu/xZ4jLb8i0/2vkVFvgcKmfavzjmAg0Dfk532jwX7wXSGHPwywKaiUhjEakEXAOMi0RwbO9iG2MSnqoWisgdwCQgCRimqksjkR3vBTKaxzGifQzF8i0/bqjqRGBipHNj+qFdxhgTTXYM0hhjgojLAhmt25Kc7GEisllElkQyNyC/oYhME5FsEVkqIn0jnF9ZROaIyEInf2Ak8511SBKR+SIyIdLZTv56EVksIgtEZG6Es2uKyEcislxElolIdJ4dGyfibhfbuS1pJQG3JQHXRuK2JCe/PbATGKmqJ0Uic7/8BkADVZ0nItWB74GuEdx+Aaqq6k4RSQFmAn1V9btI5DvrcA/QCjhMVS+NVG5A/nqglar+FIXsEcA3qvqWc8a3iqpuj/R6xIt47EFG7bYkAFWdAWyNVF4p+fmqOs95/SuwDP+dCJHKV1Xd6fyY4kwR+1dYRNKBS4C3IpUZK0SkBtAeGAqgqn9YcTww8VggS7stKWIFIpaISCOgBTA7wrlJIrIA2AxMUdVI5r8E9AeKI5i5PwUmi8j3zl0ikdIY2AK87RxieEtEqkYwP+7EY4E0gIhUAz4G+qnqL5HMVtUiVW2O/46HNiISkUMNInIpsFlVv49EXhnOUtWWwEVAH+ewSyQkAy2B11S1BbALiOgx+HgTjwUyarclxQrn2N/HwHuq+km01sPZvZsGdIpQ5JnAZc4xwFHAeSLyboSy91LVXOfPzcAY/Id9IiEHyAnosX+Ev2CaMMVjgYzabUmxwDlJMhRYpqovRCG/rojUdF4fiv9k2fJIZKvqA6qarqqN8P+9f6WqPSKRXUJEqjonx3B2bzOAiFzRoKobgQ0icqzTdD4QkZNz8Sru7qSJ5m1JACLyAdABqCMiOcBjqjo0Uvn4e1HXA4ud44AADzp3IkRCA2CEczWBD8hU1ahcbhMl9YAx/n+nSAbeV9UvIph/J/Ce0zlYC9wUwey4E3eX+RhjjFvicRfbGGNcYQXSGGOCsAJpjDFBWIE0xpggrEAaY0wQViCNMSYIK5AmpojIjSLySrTXwxiwAmkixLlw3JiDihVIUyoReUJE+gX8PKi0wXdFpIOIzBCRz5xBil8XEZ/z3k4ReV5EFgLtRKSHM5juAhF5o6RoishNIrJSRObgvxPImJhgBdIEMwy4AcApeNcAwQZ+aIP/FrcTgCbA5U57VWC2qp4K/AxcDZzpjPRTBFznDPA7EH9hPMtZhjExIe7uxTbuUNX1IvKziLTAf3/xfFX9Ocjsc1R1Ley9F/0s/CPJFOEfVQj8AyecBmQ59ykfin+8yLbAdFXd4nx+NHCMN1tlTGisQJqyvAXcCNTH36MMZv8b+kt+/l1Vi5zXAoxQ1QcCZxSRrge+msZ4w3axTVnG4B/LsTX+0ZGCaeMML+fDvxs9s5R5pgJXisgRACJSW0SOwj/a+TkicrgzjuVVrm6BMQfAepAmKFX9Q0SmAdsDeoKlyQJeAZriHyB3TCnLyhaRh/E/isAHFAB9VPU7EXkc+BbYDixwdSOMOQA23JkJyilk84CrVHVVkHk6AP+IxtMDjfGa7WKbUonICcBqYGqw4mhMvLMepKkQETkZeGe/5j2q2jYa62NMJFiBNMaYIGwX2xhjgrACaYwxQViBNMaYIKxAGmNMEFYgjTEmiP8HmUspzJEWQkgAAAAASUVORK5CYII=\n", 255 | "text/plain": [ 256 | "
" 257 | ] 258 | }, 259 | "metadata": { 260 | "needs_background": "light" 261 | }, 262 | "output_type": "display_data" 263 | }, 264 | { 265 | "name": "stdout", 266 | "output_type": "stream", 267 | "text": [ 268 | "Wall time: 2.7 s\n" 269 | ] 270 | } 271 | ], 272 | "source": [ 273 | "%%time\n", 274 | "# Train the LightGBM algorithm\n", 275 | "import lightgbm as lgb\n", 276 | "lg = lgb.LGBMClassifier()\n", 277 | "lg.fit(X_train, y_train)\n", 278 | "y_pred = lg.predict(X_test)\n", 279 | "print(classification_report(y_test,y_pred))\n", 280 | "print(\"Accuracy of LightGBM: \"+ str(accuracy_score(y_test, y_pred)))\n", 281 | "print(\"Precision of LightGBM: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n", 282 | "print(\"Recall of LightGBM: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n", 283 | "print(\"Average F1 of LightGBM: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n", 284 | "print(\"F1 of LightGBM for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n", 285 | "lg_f1=f1_score(y_test, y_pred, average=None)\n", 286 | "\n", 287 | "# Plot the confusion matrix\n", 288 | "cm=confusion_matrix(y_test,y_pred)\n", 289 | "f,ax=plt.subplots(figsize=(5,5))\n", 290 | "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n", 291 | "plt.xlabel(\"y_pred\")\n", 292 | "plt.ylabel(\"y_true\")\n", 293 | "plt.show()" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 11, 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | " precision recall f1-score support\n", 306 | "\n", 307 | " 0 1.00 1.00 1.00 3656\n", 308 | " 1 1.00 0.99 0.99 387\n", 309 | " 2 1.00 1.00 1.00 14\n", 310 | " 3 1.00 1.00 1.00 612\n", 311 | " 4 1.00 0.75 0.86 8\n", 312 | " 5 0.99 1.00 0.99 231\n", 313 | " 6 1.00 1.00 1.00 452\n", 314 | "\n", 315 | " accuracy 1.00 5360\n", 316 | " macro avg 1.00 0.96 0.98 5360\n", 317 | "weighted avg 1.00 1.00 1.00 5360\n", 318 | "\n", 319 | "Accuracy of XGBoost: 0.9975746268656717\n", 320 | "Precision of XGBoost: 0.9975807247413017\n", 321 | "Recall of XGBoost: 0.9975746268656717\n", 322 | "Average F1 of XGBoost: 0.9975482770384609\n", 323 | "F1 of XGBoost for each type of attack: [0.99836021 0.99351492 1. 0.99836334 0.85714286 0.99137931\n", 324 | " 0.99889258]\n" 325 | ] 326 | }, 327 | { 328 | "data": { 329 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvd0lEQVR4nO3dfZyVc/7H8dfnNBNKIqWmmSgVQlK6QbYSKpTCKtKKRbs2FPuTkKVsu3bXzcrdilK5q1DbLUkiWTTpvindp7nphgrlbmbO5/fHuaaOzDUz57iuc86c83nu43p0zvdc1/W+zmn7+F5330tUFWOMMb8UiPcGGGNMorICaYwxLqxAGmOMCyuQxhjjwgqkMca4sAJpjDEu0uK9AWUSsWuQjIkHVYlmscIvN0X8bza99olRZcVCQhfIwl0b45adXqcxaen145ZfVJifsvlFhfkAlh/nfJPgBdIYU8kEi+O9BZ6yAmmM8Y4G470FnrICaYzxTtAKpDHGlEqtB2mMMS6sB2mMMS6SrAdpF4obY7wTLI58KoeIHC4ii0RkuYisFpHhTvs4EdksIsuc6UynXURklIhsEJEVItIqbF39RWS9M/UvL9t6kMYY7/jTg/wR6Kyq+0QkHVgoIm85n92lqm8cMv/FQFNnagc8C7QTkVrAA0BrQIHPRGS6qu5xC7YepDHGO8Fg5FM5NGSf8zbdmcq6Y6cnMMFZ7hPgaBHJALoCc1V1t1MU5wLdysq2AmmM8YxqMOJJRAaIyOKwacCh6xWRKiKyDNhJqMh96nw00tmNflxEDnPaMoFtYYvnOm1u7a5sF9sY450ozmKr6mhgdDnzFANnisjRwFQROR24B9gOVHWWvxsYEfEGlMF6kMYY72gw8imS1avuBeYD3VS1wNmN/hF4EWjrzJYHNAhbLMtpc2t3ZQXSGOMdf85i13F6jojIEcBFwFrnuCIiIkAvYJWzyHTgOuds9tnA16paAMwBuojIMSJyDNDFaXNVqQrkjz/+xNU3DeKK/n+i57V/4KkXXgJAVXniuXFcevVN9Og7gJdfnwbAoiUrOLvLlVzZfyBX9h/Is2NfKXM9XunapROrVy1gbc5Chtw10NN1W35i5x922GF8/NFMPls8l+XL3uOBv/w5pvnx/u196kFmAPNFZAWQTegY5EzgFRFZCawEagN/deafDWwCNgDPA38CUNXdwEPOOrKBEU6bK0nkx74eOracqvL99z9QrdoRFBYVcd0t/8fQQX9g09ZtLFqygpH33UkgEOCrPXs59pijWbRkBeNee5Nn/jWciqynxenNDswT7XBngUCANas/pNsl15CbW8AnH8+m3+/+xJo16yNaT7TDjSVD/q8Z7ive+QDVq1dj//7vSEtLY8H7U7njzgf4dNES3/O9/LuPdjzIH9fMj7igHNbs/IQdD9K3HqSInCIidzsXbI5yXjcrf8ky10m1akcAUFRURFFRESLCpKmzuOWGvgQCoa9z7DFHR7UeL7Rt05KNG7ewefMXFBYWMnnyNC7r0dWTdVt+4ucD7N//HQDp6WmkpacTq05IInx3Py7ziSdfCqSI3A1MBARY5EwCvCYiQ3/NuouLi7my/0A6dL+Gc9q05IzTTmFbXgFvzfuA3r+/nT/++X62bjt43HX5qjVc0f9P/PHP97Nh09Yy1+OF+pn12JZ7cMDR3LwC6tev58m6LT/x8yHUk1uc/Q4FeSuYN28Bi7KXxiQ3Eb673ydpYs2vHuSNQBtVfVhVX3amhwmdZbrx16y4SpUqvDn+aeZNfYmVOetYv2kLPxUWcljVqkweO4ore3Tj/r89DsCpJzdm7pvjmTL+Gfpe2YPb7xlR5nqM8UIwGKR1my6c0Kg1bVq35LTTTo73JsWO9SArJAiUdgAlw/nMVfhFoy9MeM11vqNqHEnbVmew8JPF1KtTmws7tgfgwo7nsm7jZgCOrF79wK50h3PbUlRUxJ69X7uuxwv5edtpkHXwq2dlZpCfv92TdVt+4ueH+/rrb3j/g4/o2qVTTPIS4burFkc8JTK/CuRgYJ6IvCUio53pbWAeMKisBVV1tKq2VtXWN113zc8+271nL998G7rj6Icff+Tj7KU0OqEBnTucw6IlywHIXrqSExqELo7/8qvdB47/rMz5nKAqR9c8ynU9XshevIwmTRrRsGED0tPT6d27JzNmvuPJui0/8fNr165FzZpHAXD44Ydz4QUd+Pzz2DxbKd7fHUi6XWxf7qRR1bdF5CRCu9Qlt/LkAdn6K/6TseurPdz310coDgbRoNK182/o1L4drc44jbuH/5OXJv2XakcczvChgwF4Z/5CJk2dRZW0KhxetSr/Gj4UEXFdjxeKi4sZNHgYs2e9SpVAgHHjJ5GTs86TdVt+4udnZNRl7Jh/U6VKgEAgwBtvzGDW7Hdjkh3v7w4k/C5zpCrVZT6xZE81tKcapnR+lJf5/PDZfyP+N3v4Wb0S9jIfuxfbGOMde6qhMca4SPBjipGyAmmM8U6SHYO0AmmM8Y71II0xxoX1II0xxoUVSGOMKV2i3xkTKSuQxhjvWA/SGGNcJNlJmko1orgxxsSS9SCNMd6xXWxjjHGRZLvYViCNMd6xHqQxxriwHmTspNdpHNf8kmGnLN/yUzE/KtaDNMYYF1YgYyfeA8aeUqd13PLX7loc9++f0gPGWn50bBfbGGNcWA/SGGNcWA/SGGNcWA/SGGNcJFkP0u7FNsZ4JxiMfCqHiBwuIotEZLmIrBaR4U57IxH5VEQ2iMgkEanqtB/mvN/gfN4wbF33OO2fi0jX8rKtQBpjvONDgQR+BDqragvgTKCbiJwN/AN4XFWbAHuAG535bwT2OO2PO/MhIqcCVwOnAd2AZ0SkSlnBViCNMd5RjXwqd5WqqrrPeZvuTAp0Bt5w2scDvZzXPZ33OJ9fICLitE9U1R9VdTOwAWhbVrYVSGOMd/zpQSIiVURkGbATmAtsBPaqapEzSy6Q6bzOBLYBOJ9/DRwb3l7KMqWyAmmM8U4UBVJEBojI4rBpwKGrVdViVT0TyCLU6zslFl/HzmIbY7wTxVlsVR0NjK7gvHtFZD5wDnC0iKQ5vcQsIM+ZLQ9oAOSKSBpQE/gqrL1E+DKlsh6kMSahiUgdETnaeX0EcBGwBpgP/NaZrT8wzXk93XmP8/l7qqpO+9XOWe5GQFNgUVnZ1oM0xnjHnwvFM4DxzhnnADBZVWeKSA4wUUT+CiwFxjjzjwFeEpENwG5CZ65R1dUiMhnIAYqAgVrOYxiTtgcZCATIXjSHaVPHlz9zFKoeVpXJb4/jv/NfYcaCSdw2JHTY5OzftOHNd19i6nuv8MqM5zm+URYAffpfwfT3XzvQ3vikRr5sF0DXLp1YvWoBa3MWMuSugb7lWH5iZSdCvk9nsVeoaktVPUNVT1fVEU77JlVtq6pNVPUqVf3Raf/Bed/E+XxT2LpGqmpjVT1ZVd8qLztpC+Ttt93E2rXrfVv/Tz/+xPVX3kKv86/l8s59Oe/8c2hx1uk8+M+7ueuW+7m887XMnDKHW+4IXZo18805XNbpGi7vfC0vPPUSQ0fc4ct2BQIBRj0xku49+tG8xfn06dOLZs2a+pJl+YmTnQj5gG9nseMlKQtkZmYGl1x8AWPHvuZrznf7vwcgLT2NtPQ0VBVVOLJGdQBq1DiSndt3AbB/3/4Dy1Wrdjhagf9yRqNtm5Zs3LiFzZu/oLCwkMmTp3FZj3JvGLD8Sp6dCPlA0hXIpDwG+dijwxl6z1+pUeNIX3MCgQBvvvsSxzfK4tWxr7NiyWqG3fFXRr/6b3744Uf2fbufPhf//sD8fX9/Fdf/sS/p6elcf8UtvmxT/cx6bMs9OJ5fbl4Bbdu09CXL8hMnOxHyAbsX2wsicoNf6770kgvZufNLlixd6VfEAcFgkMs7X0unFpdyRsvTaHpKY/r/sS8D+g6m05ndmTJxBkNHDD4w/6tjX6dL28t59KEnueXO37uv2JhKSoMa8ZTI4rWLPdztg/CLRoPB/W6zuTr33Nb06N6FDes+4ZWXn+H889szftyoX7Wx5fn2m318+tFn/OaCczjltKasWLIagLf+O5eWbc74xfyzpr7DBRd38mVb8vO20yDr4EjUWZkZ5Odv9yXL8hMnOxHygaTbxfatQIrICpdpJVDXbTlVHa2qrVW1dSBQPeLc+4Y9TMMTW9PkpLO5tt+fmD//I/pff/uv+SqlOubYo6lxVGgX/rDDD+Pcjm3ZtG4LNWocScMTjwfg3I7t2LR+CwAnNDp4fWqni85j66YvPN8mgOzFy2jSpBENGzYgPT2d3r17MmPmO75kWX7iZCdCPhDaxY50SmB+HoOsC3QlNMpGOAH+52NuTNSpW5uHn3yQKlUCiAR4e/q7vD93Iff/eSSjxv6DoAb5Zu+33Dv4IQCuvbE353RoS1FREd/s/Yaht7l2on+V4uJiBg0exuxZr1IlEGDc+Enk5KzzJcvyEyc7EfIBSPBd5kiJX2dTRWQM8KKqLizls1dVtW9560irmhm3X9se2mUP7UrpfFWJZtnvnvxTxP9mq932TFRZseBbD1JVbyzjs3KLozGmEkrwY4qRSsrLfIwxceLTHmm8WIE0xnjHepDGGOMiyU7SWIE0xngnwS/biZQVSGOMd5KsB5mUg1UYY4wXrAdpjPGM2kkaY4xxkWS72FYgjTHesZM0xhjjwnqQxhjjwo5BGmOMC+tBGmOMCzsGaYwxLqwHGTsl4+LFy9pdi+OaH+/vb/mpnR8Nuw4yhlJ1wNhUz0+IAWMtPzrWgzTGGBdWII0xxoWdpDHGGBdJ1oO00XyMMZ7RoEY8lUdEGojIfBHJEZHVIjLIaX9QRPJEZJkzXRK2zD0iskFEPheRrmHt3Zy2DSIytLxs60EaY7zjTw+yCPizqi4RkRrAZyIy1/nscVV9JHxmETkVuBo4DagPvCsiJzkfPw1cBOQC2SIyXVVz3IKtQBpjvOPDZT6qWgAUOK+/FZE1QGYZi/QEJqrqj8BmEdkAtHU+26CqmwBEZKIzr2uBtF1sY0ylISINgZbAp07TrSKyQkTGisgxTlsmsC1ssVynza3dlRVIY4x3ghrxJCIDRGRx2DSgtFWLyJHAm8BgVf0GeBZoDJxJqIf5qNdfx3axjTHeieIYpKqOBkaXNY+IpBMqjq+o6hRnuR1hnz8PzHTe5gENwhbPctooo71U1oM0xnhGVSOeyiMiAowB1qjqY2HtGWGzXQ6scl5PB64WkcNEpBHQFFgEZANNRaSRiFQldCJnelnZ1oM0xnjHn7PY7YHfAStFZJnTdi9wjYicCSiwBfgDgKquFpHJhE6+FAEDVbUYQERuBeYAVYCxqrq6rGArkMYY7/hQIFV1ISClfDS7jGVGAiNLaZ9d1nKHsgJpjPFMRS78rkyS8hhk1y6dWL1qAWtzFjLkroFJn//86EfJz13OsqXzfvHZHYP/QNFPeRx77DGlLOmPVPv9EyU7EfKjOYudyJKuQAYCAUY9MZLuPfrRvMX59OnTi2bNmiZ1/oQJk7m0+7W/aM/Kqs9FF3Zg69ZcX/PDpeLvnwjZiZAPQDCKKYElXYFs26YlGzduYfPmLygsLGTy5Glc1qNr+QtW4vwPF37K7j17f9H+6CMPMvTekRU6U+iVVPz9EyE7EfLBn3ux48m3Aikip4jIBc7FneHt3fzKBKifWY9tuQcH/MzNK6B+/Xp+RiZUfokePbqQl1fAihWud1H5It7fP575qfzdD7Bd7PKJyO3ANOA2YJWI9Az7+G9+ZJqDjjjicO65+zYeHP5I+TMb46Uk28X26yz2zcBZqrrPuXfyDRFpqKpPUPrp+gOc24wGAEiVmgQC1SMKzs/bToOsg0PVZ2VmkJ+/PcLNj1688wEaN25Iw4bHs2RxaMCTrKwMsj+dwzntL2XHjl2+Zsf7+8czP5W/e4lE32WOlF+72AFV3QegqluATsDFIvIY5RRIVR2tqq1VtXWkxREge/EymjRpRMOGDUhPT6d3757MmPlO5N8gSvHOB1i1ai31s1rQ5KSzaXLS2eTmFtCmXVffiyPE//vHMz+Vv/sB1oOskB0icqaqLgNwepLdgbFAc58yASguLmbQ4GHMnvUqVQIBxo2fRE7OOj8j457/8ktP07HDOdSuXYstmxYzfMQjvDhuoq+ZblLx90+E7ETIh+TrQYofZzhFJAsoUtVf9O9FpL2qflSR9aRVzYzbr53KTxWMd36iPNUvpfNVy9zTc7O7Z8eI/83WmvZBVFmx4EsPUlVdL7yraHE0xlQ+SfbMruS7DtIYY7xi92IbY7yTZD1IK5DGGM8k2y62FUhjjHesQBpjTOmsB2mMMS6sQBpjjAsrkMYY4ya668sTlhVIY4xnrAdpjDEuNGg9SGOMKZX1II0xxkWUY1wkLCuQxhjPWA/SGGNc2DHIGCoZF8/yLd/yK4cYPkAzJhK6QKbqgLGJkt8p84K4ZL+fNw9I8QFrEyA/GtaDNMYYF8lWIG3AXGNMQhORBiIyX0RyRGS1iAxy2muJyFwRWe/8eYzTLiIySkQ2iMgKEWkVtq7+zvzrRaR/edlWII0xnlGNfKqAIuDPqnoqcDYwUEROBYYC81S1KTDPeQ9wMdDUmQYAz0KooAIPAO2AtsADJUXVjRVIY4xnNCgRT+WuU7VAVZc4r78F1gCZQE9gvDPbeKCX87onMEFDPgGOFpEMoCswV1V3q+oeYC7QraxsOwZpjPGM3xeKi0hDoCXwKVBXVQucj7YDdZ3XmcC2sMVynTa3dlfl9iBFpK6IjBGRt5z3p4rIjeV/FWNMqtFg5JOIDBCRxWHTgNLWLSJHAm8Cg1X1m5/lhp5f7flFRhXZxR4HzAFKrjlYBwz2ekOMMZVfUCXiSVVHq2rrsGn0oesVkXRCxfEVVZ3iNO9wdp1x/tzptOcBDcIWz3La3NpdVaRA1lbVyThPm1DVIqC4AssZY1KMqkQ8lUdEBBgDrFHVx8I+mg6UnInuD0wLa7/OOZt9NvC1sys+B+giIsc4J2e6OG2uKnIMcr+IHIvTfS0JrMByxpgU49N1kO2B3wErRWSZ03Yv8DAw2TnktxXo7Xw2G7gE2AB8B9wAoKq7ReQhINuZb4Sq7i4ruCIF8k5CFbmxiHwE1AF+W7HvZYxJJX7caqiqCwG3yvuL272c45EDXdY1Fhhb0exyC6SqLhGRjsDJzkZ+rqqFFQ0wxqSOZLuTptwCKSLXHdLUSkRQ1Qk+bZMxppIKpuB4kG3CXh9OqEu7BLACaYz5mWQbMLfcs9iqelvYdDPQCjjS/02LTlZWfd5953VWLJ/P8mXvcdutsb9ks2uXTqxetYC1OQsZcleph0Iqff6RR1Vn+HN/YcL7Yxk/fwyntmpGx0s78OK8F3jvi3c4+YyTfjZ/34HX8MrC8Uz44EXadGztyzaViOfvnwp/92Xx6VbDuInmVsP9QCOvN8QrRUVF3DVkOGe0OJ/25/Xglluup1mzpjHLDwQCjHpiJN179KN5i/Pp06dXUubfOnwgi97P5rpOv+fGLn/giw1fsPnzLfzl5gdZ8enKn817QtPj6dyzE9d3vokh/e5h8MjbCQT8ucs1nr9/qvzdlyWa6yATWUXupJkhItOdaSbwOTC1Asu1FZE2zutTReROEbnk129y2bZv38nSZasA2LdvP2vXriezfj2/Yw9o26YlGzduYfPmLygsLGTy5Glc1qNrUuVXr1GdFu2aM+u1twAoKixi3zf7+WLDF2zblPuL+dt3ac97096n8KdCtm/bTt6WfE4582RPt6lEPH//VPi7L48f10HGU0WOQT4S9roI2Kqqv/xXEEZEHiA0okaaiMwlNHrGfGCoiLRU1ZHRbnAkTjghizNbnM6ni5bGIg6A+pn12JZ7cMDR3LwC2rZpmVT5GQ3qsXf31wx97C4an9qYdSvX8eRfnuGH738odf46GceSs2TNgfe7tu+iTkZtT7epRDx//1T4uy9Pou8yR6rMHqSIVAEeVNUPnOmj8oqj47eELu7sQOh6pF6q+hCh0TT6/NqNrojq1asxedLz3Pl/D/Dtt/tiEZkyqqRV4aTTmzLtpRnc3O2PfP/dD/QdeHW8N8skgJTaxVbVYiAoIjUjXG+Rqhar6nfAxpIby1X1e5xbFt2E37geDO6PMDYkLS2N1yc9z2uvTeW//30rqnVEKz9vOw2yDg6Vn5WZQX7+9qTK31Wwi10Fu1izdC0AH8xaQNPm7se6dhV8RZ2M4w68r1OvDrsKvvR0m0rE8/dPhb/78iTbLnZFjpTvI3SLzxhnlN5RIjKqnGV+EpFqzuuzShqdQltmgQy/cT0QqF6Bzful50c/ypq1G/j3E7+459132YuX0aRJIxo2bEB6ejq9e/dkxsx3kip/96497MzfRYMTswA467xWbF2/1XX+/839H517diK9ajr1GtQjq1Ema5d97uk2lYjn758Kf/eppiLHIKc4U7jyjjR0UNUfAVR/9qTcdA7eXO6L9ue24Xf9fsuKlTkszg79n+P++x/mrbff8zP2gOLiYgYNHsbsWa9SJRBg3PhJ5OSsi0l2LPNH3f8Uw568h7Sq6RRsLeDhP/+L87q1Z9BDt1KzVk3+Pn4kG1ZvZEi/oWxZt5X3Z3zAuPfGUFxczL+HjSIY9OcByvH8/VPl774sib7LHCnRco6qisggVX2ivDY/pFXNjNsh30R4qmC88+2phimcH+W+7yf1r4j43+zZ+VMStqpWZBe7tB7f9R5vhzEmCSTbSRrXXWwRuQboCzQSkelhH9UAyhwiyBiTmhL9pEukyjoG+T+gAKgNPBrW/i2wws+NMsZUTv4cWY4f1wKpqlsJDUJ5TlkrEJGPVbXMeYwxqUFdh22snLx4quHhHqzDGJMEgkl2J40XBTLJfhJjTLSC1oM0xpjSJdsudkVG87nNeQKY6ywebo8xphILRjElsopcB1kXyBaRySLSzXkEY7jf+bBdxphKSJGIp0RWkRHFhwFNCT2X9npgvYj8TUQaO5+v8nULjTGVRir2IEseo7jdmYqAY4A3ROSfPm6bMaaSSbYCWZGnGg4CrgO+BF4A7lLVQhEJAOuBIf5uojGmskj0XeZIVeQsdi3gCufC8QNUNSgi3f3ZLGNMZZRkj8Uuv0Cq6gNlfLbG7TNjTOpJtusg/Xm0nDHGJIGEvlC8ZFw8y4+PknEZ4yXe3z/V86ORbLfVJXSBjPeAsZafwgPGWn5U/DgrLSJjge7ATlU93Wl7ELgZ2OXMdq+qznY+uwe4ESgGblfVOU57N+AJoArwgqo+XF52QhdIY0zlEvzFfSSeGAc8BUw4pP1xVQ1/LDUicipwNXAaUB94V0ROcj5+GrgIyCV088t0Vc0pK9gKpDHGM37sYqvqAhFpWMHZewITnWdibRaRDUBb57MNqroJQEQmOvOWWSDtJI0xxjMxvlD8VhFZISJjw8aLyAS2hc2T67S5tZfJCqQxxjNBiXwSkQEisjhsGlCBqGeBxsCZhJ588GiZc0fJdrGNMZ6J5jpIVR0NRPQQe1XdUfJaRJ4HZjpv84AGYbNmOW2U0e7KepDGGM9oFFM0RCQj7O3lQMmgOdOBq0XkMBFpRGignUVANtBURBqJSFVCJ3LCH0ZYKutBGmM848ethiLyGtAJqC0iucADQCcROZNQjd0C/AFAVVeLyGRCJ1+KgIGqWuys51ZgDqHLfMaq6urysq1AGmM848d1kKp6TSnNY8qYfyQwspT22cDsSLKtQBpjPGN30hhjjIuUG83HGGMqKtEHwI2UFUhjjGesQBpjjAtNsl3spLsO8vnRj5Kfu5xlS+MzVFe88wG6dunE6lULWJuzkCF3DUy5/Jo1j2LSxNGsWvkBK1e8z9ntzopZdry/e7zzk+2ZNElXICdMmMyl3a9N2fxAIMCoJ0bSvUc/mrc4nz59etGsWdOUyQd4/LERzJkzn9Obd6TVWRexZu36mOTG+7vHOz8ZxaxAisihQxX54sOFn7J7z95YRCVkfts2Ldm4cQubN39BYWEhkydP47IeXVMm/6ijavCb89ox9sXXACgsLOTrr7+JSXa8v3u888F6kBUiItMPmWYAV5S89yPThNTPrMe23IMDnubmFVC/fr2UyW/U6Hi+/PIrxrzwONmL5vDcf/5FtWpHxCQ73t893vkQu1sNY8WvHmQW8A3wGKFRNh4Fvg17bYwv0qpUoWXL5jz33ATatO3K/v3fcfeQW+O9WSkjmtF8EplfBbI18BlwH/C1qr4PfK+qH6jqB2UtGD70UTC436fNS175edtpkHVwqP6szAzy87enTH5uXgG5uQUsyl4KwJQps2h5ZvOYZMf7u8c7H2wXu0JUNaiqjwM3APeJyFNU8JIiVR2tqq1VtXUgUN2PzUtq2YuX0aRJIxo2bEB6ejq9e/dkxsx3UiZ/x45d5Obmc9JJjQHo3Pk81qxZF5PseH/3eOdD8hVIX6+DVNVc4CoRuZTQLrfvXn7paTp2OIfatWuxZdNiho94hBfHTYxFdELkFxcXM2jwMGbPepUqgQDjxk8iJyc2BSIR8gEG3XE/E8Y/SdWq6Wze/AU33nRnTHLj/d3jnQ+Jf0wxUqKauF8prWpm3DYulZ8qGO/8RHmqX0rna3SXfP/zhH4R/5sdsvXlhD0SaXfSGGM8k+i7zJGyAmmM8Uzi7o9GxwqkMcYzwSQrkVYgjTGesV1sY4xxkVz9RyuQxhgPWQ/SGGNcJPqtg5GyAmmM8YydpDHGGBfJVR6TcMBcY4zxivUgjTGesZM0xhjjwo5BGmOMi+Qqj1YgjTEesl3sGCoZ9snyLd/yK4dk28W2s9jGGM/48dAuERkrIjtFZFVYWy0RmSsi650/j3HaRURGicgGEVkhIq3ClunvzL9eRPpX5PskdA8yVQeMTfX8hBgwFsio2Swu+QVfrwHi//2j4dMu9jjgKSD80dFDgXmq+rCIDHXe3w1cDDR1pnbAs0A7EakFPEDoeVkKfCYi01V1T1nB1oM0xnhGo/hfuetUXQDsPqS5JzDeeT0e6BXWPkFDPgGOFpEMoCswV1V3O0VxLtCtvGwrkMYYz0Tz0K7wJ5k604AKRNVV1QLn9XagrvM6E9gWNl+u0+bWXqaE3sU2xlQu0ZykUdXRwOhoM1VVRcSXs0PWgzTGeMaPkzQudji7zjh/7nTa84AGYfNlOW1u7WWyAmmM8UwQjXiK0nSg5Ex0f2BaWPt1ztnss4GvnV3xOUAXETnGOePdxWkrk+1iG2M848dZbBF5DegE1BaRXEJnox8GJovIjcBWoLcz+2zgEmAD8B1wA4Cq7haRh4BsZ74RqnroiZ9fsAJpjPFMRc5KR7xO1WtcPrqglHkVGOiynrHA2EiyrUAaYzxjtxoaY4wLP3qQ8WQnaYwxxoX1II0xnrFdbGOMcRHU5NrFtgJpjPFMcpXHJDwG+fzoR8nPXc6ypfPitg1du3Ri9aoFrM1ZyJC7Sr3iwPIrcX79zHq8MeNFPvhkBu9/PJ2b/tgPgCH33ca8j6Yy98MpTJzyPHXr1TmwzEP/uJf/LXmbeR9NpXkL/0YJivdvH8MLxWMi6QrkhAmTubT7tXHLDwQCjHpiJN179KN5i/Pp06cXzZo1tfwkyi8qKmL4sH/S8eweXHrR1Vx/U19OOrkxz4waywXtL+ei31zB3DkfcOeQPwHQ+aIOnHjiCZzbqht3DXqAhx99wNPtKRHv3x78Gc0nnmJSIEXkPBG5U0S6+J314cJP2b1nr98xrtq2acnGjVvYvPkLCgsLmTx5Gpf16Gr5SZS/c8eXrFweGrNx/77vWL9uE/UyjmPft/sPzFOt2hGoczyu2yWdeX1i6E64JYtXcFTNGhxXt7an2wTx/+0hutF8EpkvBVJEFoW9vpnQYJc1gAecwS2TVv3MemzLPTjgaG5eAfXr17P8JM3POr4+zZs3Y8lnKwAYOmwQi1fN44qruvOvvz0JQL2M48jP235gmYL8HWRk1C11fb9GvH97sF3sikoPez0AuEhVhxO6QTx++7/GeKha9WqMmfAEf7n37wd6jw//9Qlan34BU16fyQ0DUu//6raLXcH1OqNmHAuIqu4CUNX9QFFZC4YPnhkM7i9r1oSUn7edBlkHh8rPyswgP397GUtYfmXMT0tLY8yEfzPl9ZnMnvHuLz6f8vpMLu1xEQDbC3ZSP/NgTy6jfl0KCnZ4vk3x/u3BdrErqibwGbAYqBU2btuRgJS1oKqOVtXWqto6EKju0+b5J3vxMpo0aUTDhg1IT0+nd++ezJj5juUnWf5jTz3E+nWbeO7p8QfaGp14woHXXS/pzIb1mwCY89Z7XHV1TwBatT6Db7/5lp07vvR8m+L92wOoasRTIvPlOkhVbejyURC43I/MEi+/9DQdO5xD7dq12LJpMcNHPMKL4yb6GfkzxcXFDBo8jNmzXqVKIMC48ZPIyVln+UmU3/bsVlx1dU9yVn/O3A+nAPD3Ef+m7++uoHGTRgQ1SO62fO6+YzgA895ZwAUXdeDjpW/z/Xc/cMfA+zzdnhLx/u0h+R77KolcwdOqZsZt41L5qYLxzrenGibAUw1Vy9zTc9Pj+O4R/5ud8cXMqLJiwe6kMcZ4JtFPukTKCqQxxjPJtottBdIY45lEPmQXDSuQxhjPJPplO5GyAmmM8UyyHYNMusEqjDHGK9aDNMZ4xk7SGGOMCztJY4wxLqwHaYwxLpLtJI0VSGOMZ+yhXcYY4yK5yqMVSGOMh5LtGKRdB2mM8Yxfj1wQkS0islJElonIYqetlojMFZH1zp/HOO0iIqNEZIOIrBCRVtF+n4TuQZYMO2X5lh8PJcOOxUu8v380fL7M53xVDR9peCgwT1Ufdp51NRS4G7gYaOpM7YBnnT8jZj1IY4xnYvzQrp5AyZDu44FeYe0TNOQT4OiSpxpEKqF7kKk6YGyq5yfKgLnxzm9R95y45C/f8XHUy/p4mY8C74iIAs+p6migrqoWOJ9vB0oeFZkJbAtbNtdpKyBCCV0gjTGVSzS72CIygNDTT0uMdgpguPNUNU9EjgPmisjaQ3LVKZ6esgJpjPFMNLvMTjE8tCAeOk+e8+dOEZkKtAV2iEiGqhY4u9A7ndnzgAZhi2c5bRGzY5DGGM/48VRDEakuIjVKXgNdgFXAdKC/M1t/YJrzejpwnXM2+2zg67Bd8YhYD9IY4xmfroOsC0wVEQjVrFdV9W0RyQYmi8iNwFagtzP/bOASYAPwHXBDtMFWII0xnvHjJI2qbgJalNL+FXBBKe0KDPQi23axjTHGhfUgjTGescEqjDHGhQ13ZowxLqwHaYwxLqwHaYwxLqwHaYwxLqwHaYwxLpKtB5l010E+P/pR8nOXs2zpvLhtQ9cunVi9agFrcxYy5C5Prle1/EqSH8vsQCDApLnjePKlfwEw4on7mL3oDSa9O45J747j5NOaAtCwyQlMmDma7K3vc90t1/i6TRrF/xJZ0hXICRMmc2n3a+OWHwgEGPXESLr36EfzFufTp08vmjVravkpkB/r7Gtv7s2m9Vt+1vbYiKfpc+H19Lnwej5fvR6Ab/Z+wz+GPc74Z1/zbVtKqAYjnhKZLwVSRNqJyFHO6yNEZLiIzBCRf4hITT8yS3y48FN279nrZ0SZ2rZpycaNW9i8+QsKCwuZPHkal/XoavkpkB/L7OMy6vCbC89l6iszyp1395d7WL1sDUVFRb5sS7gYD5jrO796kGMJ3SQO8ARQE/iH0/aiT5kJoX5mPbblHhwqPzevgPr161l+CuTHMnvIQ4N5/KGnCR7SA7tt6ABef28C/zf8dtKrpvuSXRY/RvOJJ78KZEBVS/5z1VpVB6vqQlUdDpxY1oIiMkBEFovI4mBwv0+bZ0zl1eGic9n95R7WrPj8Z+2jRv6HnuddQ99uN1LzmKP4/a39Yr5t1oOsmFUiUjLE0HIRaQ0gIicBhWUtqKqjVbW1qrYOBKr7tHn+yc/bToOsg0P1Z2VmkJ+/3fJTID9W2We2OYNOXc5jdvab/OM/I2jT/iz+9tQDfLnzKwAKfypk2sRZnN7yVM+zy2M9yIq5CegoIhuBU4GPRWQT8LzzWdLKXryMJk0a0bBhA9LT0+nduyczZr5j+SmQH6vsUX/7D11a9eKSNldy9x//QvZHn3HvrcOpfdyxB+Y5v1sHNqzd5Hl2eYKqEU+JzJfrIFX1a+B650RNIycnV1V3+JEX7uWXnqZjh3OoXbsWWzYtZviIR3hx3ES/Yw8oLi5m0OBhzJ71KlUCAcaNn0ROzjrLT4H8eH/3vz/zIMccezQiwuer1vPQkH8CcGydWrw2ZyzVa1QnGAzS7+Y+XN6hL/v3fVfOGiOX6JftREoSuYubVjUzbhuXyk8VjHd+ojxVMN75cX2qoapEs2zdmqdE/G92x9dro8qKhaS7DtIYY7xitxoaYzyT6GelI2UF0hjjmUQ+ZBcNK5DGGM8k+lnpSFmBNMZ4xnqQxhjjwo5BGmOMC+tBGmOMCzsGaYwxLpLtThorkMYYz1gP0hhjXCTbMUi71dAY4xm/nkkjIt1E5HMR2SAiQ33+GgdYD9IY4xk/epAiUgV4GrgIyAWyRWS6quZ4HnYI60EaYzzj04C5bYENqrpJVX8CJgI9ff0ijoTuQZYM+2T5lp+K+ct3fBzX/Gj4dAQyE9gW9j4XaOdP1M8ldIGMdky6EiIyQFVHe7U5lSXb8i0/XvlFP+VF/G9WRAYAA8KaRsfztwuX7LvYA8qfJSmzLd/y451fYeHPoXKmQ4tjHtAg7H2W0+a7ZC+QxpjKLxtoKiKNRKQqcDUwPRbBib2LbYxJeapaJCK3AnOAKsBYVV0di+xkL5DxPI4R72Molm/5SUNVZwOzY52b0A/tMsaYeLJjkMYY4yIpC2S8bktysseKyE4RWRXL3LD8BiIyX0RyRGS1iAyKcf7hIrJIRJY7+cNjme9sQxURWSoiM2Od7eRvEZGVIrJMRBbHOPtoEXlDRNaKyBoRic+zY5NE0u1iO7clrSPstiTgmljcluTkdwD2ARNU9fRYZB6SnwFkqOoSEakBfAb0iuH3F6C6qu4TkXRgITBIVT+JRb6zDXcCrYGjVLV7rHLD8rcArVX1yzhkjwc+VNUXnDO+1VR1b6y3I1kkYw8ybrclAajqAmB3rPJKyS9Q1SXO62+BNYTuRIhVvqrqPudtujPF7L/CIpIFXAq8EKvMRCEiNYEOwBgAVf3JiuOvk4wFsrTbkmJWIBKJiDQEWgKfxji3iogsA3YCc1U1lvn/BoYAwRhmHkqBd0TkM+cukVhpBOwCXnQOMbwgItVjmJ90krFAGkBEjgTeBAar6jexzFbVYlU9k9AdD21FJCaHGkSkO7BTVT+LRV4ZzlPVVsDFwEDnsEsspAGtgGdVtSWwH4jpMfhkk4wFMm63JSUK59jfm8ArqjolXtvh7N7NB7rFKLI9cJlzDHAi0FlEXo5R9gGqmuf8uROYSuiwTyzkArlhPfY3CBVME6VkLJBxuy0pETgnScYAa1T1sTjk1xGRo53XRxA6WbY2Ftmqeo+qZqlqQ0J/7++par9YZJcQkerOyTGc3dsuQEyuaFDV7cA2ETnZaboAiMnJuWSVdHfSxPO2JAAReQ3oBNQWkVzgAVUdE6t8Qr2o3wErneOAAPc6dyLEQgYw3rmaIABMVtW4XG4TJ3WBqaH/TpEGvKqqb8cw/zbgFadzsAm4IYbZSSfpLvMxxhivJOMutjHGeMIKpDHGuLACaYwxLqxAGmOMCyuQxhjjwgqkMca4sAJpEoqIXC8iT8V7O4wBK5AmRpwLx42pVKxAmlKJyAgRGRz2fmRpg++KSCcRWSAis5xBiv8jIgHns30i8qiILAfOEZF+zmC6y0TkuZKiKSI3iMg6EVlE6E4gYxKCFUjjZixwHYBT8K4G3AZ+aEvoFrdTgcbAFU57deBTVW0BfAX0Ado7I/0UA9c6A/wOJ1QYz3PWYUxCSLp7sY03VHWLiHwlIi0J3V+8VFW/cpl9kapuggP3op9HaCSZYkKjCkFo4ISzgGznPuUjCI0X2Q54X1V3OctPAk7y51sZExkrkKYsLwDXA/UI9SjdHHpDf8n7H1S12HktwHhVvSd8RhHp9es30xh/2C62KctUQmM5tiE0OpKbts7wcgFCu9ELS5lnHvBbETkOQERqicgJhEY77ygixzrjWF7l6Tcw5lewHqRxpao/ich8YG9YT7A02cBTQBNCA+ROLWVdOSIyjNCjCAJAITBQVT8RkQeBj4G9wDJPv4Qxv4INd2ZcOYVsCXCVqq53macT8H/xeHqgMX6zXWxTKhE5FdgAzHMrjsYkO+tBmgoRkebAS4c0/6iq7eKxPcbEghVIY4xxYbvYxhjjwgqkMca4sAJpjDEurEAaY4wLK5DGGOPi/wHZC7AwKOoHIQAAAABJRU5ErkJggg==\n", 330 | "text/plain": [ 331 | "
" 332 | ] 333 | }, 334 | "metadata": { 335 | "needs_background": "light" 336 | }, 337 | "output_type": "display_data" 338 | }, 339 | { 340 | "name": "stdout", 341 | "output_type": "stream", 342 | "text": [ 343 | "Wall time: 7.91 s\n" 344 | ] 345 | } 346 | ], 347 | "source": [ 348 | "%%time\n", 349 | "# Train the XGBoost algorithm\n", 350 | "import xgboost as xgb\n", 351 | "xg = xgb.XGBClassifier()\n", 352 | "\n", 353 | "X_train_x = X_train.values\n", 354 | "X_test_x = X_test.values\n", 355 | "\n", 356 | "xg.fit(X_train_x, y_train)\n", 357 | "\n", 358 | "y_pred = xg.predict(X_test_x)\n", 359 | "print(classification_report(y_test,y_pred))\n", 360 | "print(\"Accuracy of XGBoost: \"+ str(accuracy_score(y_test, y_pred)))\n", 361 | "print(\"Precision of XGBoost: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n", 362 | "print(\"Recall of XGBoost: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n", 363 | "print(\"Average F1 of XGBoost: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n", 364 | "print(\"F1 of XGBoost for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n", 365 | "xg_f1=f1_score(y_test, y_pred, average=None)\n", 366 | "\n", 367 | "# Plot the confusion matrix\n", 368 | "cm=confusion_matrix(y_test,y_pred)\n", 369 | "f,ax=plt.subplots(figsize=(5,5))\n", 370 | "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n", 371 | "plt.xlabel(\"y_pred\")\n", 372 | "plt.ylabel(\"y_true\")\n", 373 | "plt.show()" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": 12, 379 | "metadata": { 380 | "scrolled": false 381 | }, 382 | "outputs": [ 383 | { 384 | "name": "stdout", 385 | "output_type": "stream", 386 | "text": [ 387 | " precision recall f1-score support\n", 388 | "\n", 389 | " 0 1.00 1.00 1.00 3656\n", 390 | " 1 0.99 0.99 0.99 387\n", 391 | " 2 1.00 1.00 1.00 14\n", 392 | " 3 1.00 1.00 1.00 612\n", 393 | " 4 1.00 0.75 0.86 8\n", 394 | " 5 0.99 1.00 0.99 231\n", 395 | " 6 1.00 0.99 0.99 452\n", 396 | "\n", 397 | " accuracy 1.00 5360\n", 398 | " macro avg 1.00 0.96 0.98 5360\n", 399 | "weighted avg 1.00 1.00 1.00 5360\n", 400 | "\n", 401 | "Accuracy of CatBoost: 0.996455223880597\n", 402 | "Precision of CatBoost: 0.9964590935743635\n", 403 | "Recall of CatBoost: 0.996455223880597\n", 404 | "Average F1 of CatBoost: 0.9964290021228678\n", 405 | "F1 of CatBoost for each type of attack: [0.99781241 0.99222798 1. 0.99591837 0.85714286 0.99137931\n", 406 | " 0.9944629 ]\n" 407 | ] 408 | }, 409 | { 410 | "data": { 411 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAE+CAYAAADvb4nvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxzElEQVR4nO3deXxU9fX/8dcZElQWUYRCSFAoYN2QRRYRBYQKqCBoK7igaG2pFi1oC6LVIlr6pa2oULU/QVY3CAqyiAIiilgFAoQtILuShUVRBKyY5fz+mJswQibJDPfODDPn6eM+mHzm3nnfO5jD526fK6qKMcaYE/mivQLGGBOrrEAaY0wQViCNMSYIK5DGGBOEFUhjjAnCCqQxxgSRFO0VKJOIXYNkTDSoSjiL5X+1I+Tf2eRaPw8rKxJiukDm798etezk2o1ISq4XtfyC/NyEzS/IzwWw/CjnmxgvkMaYU0xRYbTXwFVWII0x7tGiaK+Bq6xAGmPcU2QF0hhjSqXWgzTGmCDirAdp10EaY9yjRaFP5RCR00VkhYisFZGNIjLCaZ8sIjtFJNOZmjvtIiJjRWSbiKwTkZYBn9VfRLY6U//ysq0HaYxxjzdnsY8CnVX1sIgkA8tE5F3nvSGq+uZx818LNHGmtsB/gLYiUhMYDrQCFFglInNU9ZtgwdaDNMa4x4MepPoddn5MdqayLkjvBUx1lvsMOEtEUoBuwCJVPeAUxUVA97KyrUAaY9xTVBT6VAEiUklEMoF9+Ivccuetkc5u9LMicprTlgrsDlg822kL1h6UFUhjjGtUi0KeRGSAiGQETANO/FwtVNXmQBrQRkQuAR4BLgBaAzWBh93eHiuQxhj3hNGDVNVxqtoqYBoX7ONV9VtgCdBdVfOc3eijwCSgjTNbDlA/YLE0py1Ye1BWII0x7vHmLHZtETnLeX0GcA2w2TmuiIgI0BvY4CwyB7jTOZt9OXBQVfOABUBXETlbRM4GujptQdlZbGOMe7w5i50CTBGRSvg7demqOk9EPhCR2oAAmcC9zvzzgeuAbcD3wN0AqnpARJ4CVjrzPamqB8oKPqV6kEeP/sgtvx3ETf3/QK/bf8/zL78CgKoy5qXJXH/Lb+l52wBenTH7J8ut3/Q5zTpcz8IlH5e0PfPiBHr3u5fe/e7l3fc/cm0dx48bTW72WjLXLHbtM0PVrWsnNm5YyuasZQwdMjCi2Ym+/aeddhqffjKPVRmLWJv5AcP/+qeIZael1eP9hTNYt3YJazM/4IH774lYdglvzmKvU9UWqnqpql6iqk867Z1VtanT1q/4TLez2z1QVRs572cEfNZEVW3sTJPKyz6lepCVKyczcewoqlQ5g/yCAu68789cdXkrdnyxmz37vmLu6+Pw+Xx8/c23JcsUFhby7IuTuKJ1ybWifPTfFWR9vp03J7/Aj/n53H3/UK5q14pqVaue9DpOnZrOiy9OYtKkMSf9WeHw+XyMHTOS7tfdSnZ2Hp99Op+58xayadPWiOQn+vYfPXqUX3btw5Ej35OUlMTSD2fx3ntLWL5itefZBQUFDBk6gjWZG6hWrSorlr/H+4uXRmzb45FnPUgRuUBEHnauaB/rvL7wJD+TKlXOAPz/MxQUFCAiTJ/1DvfdfRs+n39zzjn7rJJlXn9zDtd0ak/NgLbtO7+kVfNLSEqqRJUzTuf8xg1Z9tmqk1m1Eh8vW86BgAIdaW1at2D79l3s3Pkl+fn5pKfP5oae3SKWn+jbD3DkyPcAJCcnkZScTKSePb9nzz7WZPoPwx0+fITNm7eSWq9uRLJLeHSZT7R4UiBF5GFgGv5jAyucSYA3RGTYyXx2YWEhv+o/kA49bqVd6xZcevEF7M7J493FH9HnN3/k3j89zhe7/Sem9u7/isVL/0vfG6//yWf8onFDli1fxf9++IFvvj3IytXr2LNv/8msVsyol1qX3dnHBjzNzsmjXqR/SaIoFrbf5/ORsXIheTnrWLx4KStWroloPsB556XRvNklLF8R4WwPdrGjyatd7HuAi1U1P7BRRJ4BNgKjwv3gSpUq8daUF/ju0GEGPfIUW3fs4sf8fE6rXJn0iWNZ9OEnPP73Z5n6n6f5x5iXePC+35T0LIu1b3sZGzZvod/v/8TZZ9Wg2cUXUMl3Sh2ONTGsqKiIVq27UqPGmbw1YwIXX/wLNm78PGL5VatWIX36eB7683AOHTpc/gJuivEeYai8qgpFQGnjxac47wUVeNHoy1PfCDrfmdWr0ablpSz7LIO6tWvxy47tAfhlxyvYsn0nABs3b2XI8FF0/VV/Fn64jL89/QKLl/4XgN/3v5W3przAy2P+jgLn1S/zgvpTRm7OHuqnHfvq01JTyM3dE8U1iqxY2v6DB7/jw48+oVvXThHLTEpKYsb08bzxxizefvvd8hdwmWphyFMs86oHORhYLCJbOXZrz7lAY+D+shZ0LhIdByc+AOjAN9+SlJTEmdWr8cPRo3y6cg2/6XcznTu0Y8XqtaTVq8vKNetLit2CNyeXLPuXv42mY/s2dOlwBYWFhRw6fISzapzJ59t2smXbTq547M/ubHmUrczIpHHjhjRoUJ+cnD306dOLO+6M7JncaIr29teqVZP8/AIOHvyO008/nV926cC/nn4xYvnjx41m0+ZtPDcm6LXW3orxXeZQeVIgVfU9ETkf/5XtxV2zHGClnsQ/Gfu//oa//O1pCouK0CKlW+er6NS+LS0vvZiHR/yTV6a/TZUzTmfEsMFlfk5BQSF3/sFfEKtVqcKovw4hKalSuKv1E6++8gIdO7SjVq2a7NqRwYgnn2bS5GmufHZFFBYWMmjwY8x/53Uq+XxMnjKdrKwtEctP9O1PSanDxAnPUamSD5/Px5tvzuWd+e9HJLv9Fa25o9+vWbc+i4yVCwF4/PFRvPveBxHJB+JuF1sidYYtHOE8QtIt9lRDe6phQueH+djXH1a9HfLv7OmX9bbHvhpjEoA91dAYY4KwY5DGGBNEnB2DtAJpjHGP9SCNMSYI60EaY0wQViCNMaZ0sX5nTKisQBpj3GM9SGOMCSLOTtLYEDbGGBOE9SCNMe6xXWxjjAkiznaxrUAaY9xjPUhjjAnCepCRk1y7UVTzi4edsnzLT8T8sFgP0hhjgrACGTnRHjD2gtqtopa/eX9G1Lc/oQeMtfzw2C62McYEYT1IY4wJwnqQxhgThPUgjTEmiDjrQdq92MYY9xQVhT6VQ0ROF5EVIrJWRDaKyAinvaGILBeRbSIyXUQqO+2nOT9vc95vEPBZjzjtn4tIt/KyrUAaY9zjQYEEjgKdVbUZ0BzoLiKXA/8AnlXVxsA3wD3O/PcA3zjtzzrzISIXAbcAFwPdgRdFpFJZwVYgjTHuUQ19KvcjVVX1sPNjsjMp0Bl402mfAvR2XvdyfsZ5v4uIiNM+TVWPqupOYBvQpqxsK5DGGPeE0YMUkQEikhEwDTj+Y0WkkohkAvuARcB24FtVLXBmyQZSndepwG4A5/2DwDmB7aUsUyo7SWOMcU8YZ7FVdRwwrpx5CoHmInIWMAu4IJzVC5X1II0x7tGi0KdQPl71W2AJ0A44S0SKO3lpQI7zOgeoD+C8XwP4OrC9lGVKZQXSGBPTRKS203NERM4ArgE24S+Uv3Zm6w/Mdl7PcX7Gef8DVVWn/RbnLHdDoAmwoqxs28U2xrjHmwvFU4ApzhlnH5CuqvNEJAuYJiJ/A9YAE5z5JwCviMg24AD+M9eo6kYRSQeygAJgoJbzGMa47UH6fD5WrljA7FlTyp85DJVPq0z6e5N5e8lrzF06nQeG+o8rX35Va956/xVmffAar80dz7kN0wDo2/8m5nz4Rkl7o/MberJeAN26dmLjhqVszlrG0CEDPcux/NjKjoV8j85ir1PVFqp6qapeoqpPOu07VLWNqjZW1ZtV9ajT/oPzc2Pn/R0BnzVSVRup6i9U9d3ysuO2QP7xgd+yefNWzz7/x6M/ctev7qP31bdzY+fbuPLqdjS77BKe+OfDDLnvcW7sfDvzZi7gvgf9l2bNe2sBN3S6lRs7387Lz7/CsCcf9GS9fD4fY8eMpEfPfjRtdjV9+/bmwgubeJJl+bGTHQv5gFfXQUZNXBbI1NQUrru2CxMnvuFpzvdH/gdAUnISSclJqCqqUK16VQCqV6/Gvj37AThy+EjJclWqnI5W4F/OcLRp3YLt23exc+eX5Ofnk54+mxt6lnvDgOWf4tmxkA/EXYGMy2OQz4wewbBH/kb16tU8zfH5fLz1/iuc2zCN1yfOYN3qjTz24N8Y9/pz/PDDUQ4fOkLfa39TMv9tv7mZu+69jeTkZO666T5P1qleal12Zx8bzy87J482rVt4kmX5sZMdC/mA3YvtBhG526vPvv66X7Jv31esXrPeq4gSRUVF3Nj5djo1u55LW1xMkwsa0f/e2xhw22A6Ne/BzGlzGfbk4JL5X584g65tbmT0U//mvod+E/yDjTlFaZGGPMWyaO1ijwj2RuBV9UVFR4LNFtQVV7SiZ4+ubNvyGa+9+iJXX92eKZPHntTKlufQd4dZ/skqrurSjgsubsK61RsBePftRbRofekJ878zayFdru3kybrk5uyhftqxkajTUlPIzd3jSZblx052LOQDcbeL7VmBFJF1Qab1QJ1gy6nqOFVtpaqtfL6qIef+5bFRNPh5Kxqffzm39/sDS5Z8Qv+7/ngym1Kqs885i+pn+nfhTzv9NK7o2IYdW3ZRvXo1Gvz8XACu6NiWHVt3AXBew2PXp3a65kq+2PGl6+sEsDIjk8aNG9KgQX2Sk5Pp06cXc+ct9CTL8mMnOxbyAc8vFI80L49B1gG64R9lI5AA//UwNyJq16nFqH8/QaVKPkR8vDfnfT5ctIzH/zSSsRP/QZEW8d23h3h08FMA3H5PH9p1aENBQQHfffsdwx4I2ok+KYWFhQwa/Bjz33mdSj4fk6dMJytriydZlh872bGQD0CM7zKHSrw6myoiE4BJqrqslPdeV9XbyvuMpMqpUfu27aFd9tCuhM5XlXCW/f7ffwj5d7bKAy+GlRUJnvUgVfWeMt4rtzgaY05BMX5MMVRxeZmPMSZKPNojjRYrkMYY91gP0hhjgoizkzRWII0x7onxy3ZCZQXSGOOeOOtBxuVgFcYY4wbrQRpjXKN2ksYYY4KIs11sK5DGGPfYSRpjjAnCepDGGBOEHYM0xpggrAdpjDFB2DFIY4wJwnqQkVM8Ll60bN6fEdX8aG+/5Sd2fjjsOsgIStQBYxM9PyYGjLX88FgP0hhjgrACaYwxQdhJGmOMCSLOepA2mo8xxjVapCFP5RGR+iKyRESyRGSjiAxy2p8QkRwRyXSm6wKWeUREtonI5yLSLaC9u9O2TUSGlZdtPUhjjHu86UEWAH9S1dUiUh1YJSKLnPeeVdWnA2cWkYuAW4CLgXrA+yJyvvP2C8A1QDawUkTmqGpWsGArkMYY93hwmY+q5gF5zutDIrIJSC1jkV7ANFU9CuwUkW1AG+e9baq6A0BEpjnzBi2QtottjDlliEgDoAWw3Gm6X0TWichEETnbaUsFdgcslu20BWsPygqkMcY9RRryJCIDRCQjYBpQ2keLSDXgLWCwqn4H/AdoBDTH38Mc7fbm2C62McY9YRyDVNVxwLiy5hGRZPzF8TVVnekstzfg/fHAPOfHHKB+wOJpThtltJfKepDGGNeoashTeUREgAnAJlV9JqA9JWC2G4ENzus5wC0icpqINASaACuAlUATEWkoIpXxn8iZU1a29SCNMe7x5ix2e+AOYL2IZDptjwK3ikhzQIFdwO8BVHWjiKTjP/lSAAxU1UIAEbkfWABUAiaq6saygq1AGmPc40GBVNVlgJTy1vwylhkJjCylfX5Zyx3PCqQxxjUVufD7VBKXxyC7de3Exg1L2Zy1jKFDBsZ9/vhxo8nNXkvmmsUnvPfg4N9T8GMO55xzdilLeiPRvv9YyY6F/HDOYseyuCuQPp+PsWNG0qNnP5o2u5q+fXtz4YVN4jp/6tR0ru9x+wntaWn1uOaXHfjii2xP8wMl4vcfC9mxkA9AURhTDIu7AtmmdQu2b9/Fzp1fkp+fT3r6bG7o2a38BU/h/I+XLefAN9+e0D766ScY9ujICp0pdEsifv+xkB0L+eDNvdjR5FmBFJELRKSLc3FnYHt3rzIB6qXWZXf2sQE/s3PyqFevrpeRMZVfrGfPruTk5LFuXdC7qDwR7e2PZn4ib3sJ28Uun4j8EZgNPABsEJFeAW//3YtMc8wZZ5zOIw8/wBMjni5/ZmPcFGe72F6dxf4dcJmqHnbunXxTRBqo6hhKP11fwrnNaACAVKqBz1c1pODcnD3UTzs2VH1aagq5uXtCXP3wRTsfoFGjBjRocC6rM/wDnqSlpbBy+QLatb+evXv3e5od7e2PZn4ib3uxWN9lDpVXu9g+VT0MoKq7gE7AtSLyDOUUSFUdp6qtVLVVqMURYGVGJo0bN6RBg/okJyfTp08v5s5bGPoWhCna+QAbNmymXlozGp9/OY3Pv5zs7Dxat+3meXGE6G9/NPMTedtLWA+yQvaKSHNVzQRwepI9gIlAU48yASgsLGTQ4MeY/87rVPL5mDxlOllZW7yMjHr+q6+8QMcO7ahVqya7dmQw4smnmTR5mqeZwSTi9x8L2bGQD/HXgxQvznCKSBpQoKon9O9FpL2qflKRz0mqnBq1bzuRnyoY7fxYeapfQuerlrmnF8yBXh1D/p2tOfujsLIiwZMepKoGvfCuosXRGHPqibNndsXfdZDGGOMWuxfbGOOeOOtBWoE0xrgm3naxrUAaY9xjBdIYY0pnPUhjjAnCCqQxxgRhBdIYY4IJ7/rymGUF0hjjGutBGmNMEFpkPUhjjCmV9SCNMSaIMMe4iFlWII0xrrEepDHGBGHHICOoeFw8y7d8yz81RPABmhER0wUyUQeMjZX8TqldopL9Yc5iIMEHrI2B/HBYD9IYY4KItwJpA+YaY2KaiNQXkSUikiUiG0VkkNNeU0QWichW58+znXYRkbEisk1E1olIy4DP6u/Mv1VE+peXbQXSGOMa1dCnCigA/qSqFwGXAwNF5CJgGLBYVZsAi52fAa4FmjjTAOA/4C+owHCgLdAGGF5cVIOxAmmMcY0WSchTuZ+pmqeqq53Xh4BNQCrQC5jizDYF6O287gVMVb/PgLNEJAXoBixS1QOq+g2wCOheVrYdgzTGuMbrC8VFpAHQAlgO1FHVPOetPUAd53UqsDtgsWynLVh7UOX2IEWkjohMEJF3nZ8vEpF7yt8UY0yi0aLQJxEZICIZAdOA0j5bRKoBbwGDVfW7n+T6n1/t+kVGFdnFngwsAIqvOdgCDHZ7RYwxp74ilZAnVR2nqq0CpnHHf66IJOMvjq+p6kynea+z64zz5z6nPQeoH7B4mtMWrD2oihTIWqqajvO0CVUtAAorsJwxJsGoSshTeUREgAnAJlV9JuCtOUDxmej+wOyA9juds9mXAwedXfEFQFcROds5OdPVaQuqIscgj4jIOTjd1+LACixnjEkwHl0H2R64A1gvIplO26PAKCDdOeT3BdDHeW8+cB2wDfgeuBtAVQ+IyFPASme+J1X1QFnBFSmQD+GvyI1E5BOgNvDrim2XMSaReHGroaouA4JV3hNu93KORw4M8lkTgYkVzS63QKrqahHpCPzCWcnPVTW/ogHGmMQRb3fSlFsgReTO45paigiqOtWjdTLGnKKKEnA8yNYBr0/H36VdDViBNMb8RLwNmFvuWWxVfSBg+h3QEqjm/aqFJy2tHu8vnMG6tUtYm/kBD9wf+Us2u3XtxMYNS9mctYyhQ0o9FHLK51c7syojXvorUz+cyJQlE7io5YV0vL4Dkxa/zAdfLuQXl57/k/lvG3grry2bwtSPJtG6YytP1qlYNL//RPi7L4tHtxpGTTi3Gh4BGrq9Im4pKChgyNARXNrsatpf2ZP77ruLCy9sErF8n8/H2DEj6dGzH02bXU3fvr3jMv/+EQNZ8eFK7uz0G+7p+nu+3PYlOz/fxV9/9wTrlq//ybznNTmXzr06cVfn3zK03yMMHvlHfD5v7nKN5vefKH/3ZQnnOshYVpE7aeaKyBxnmgd8DsyqwHJtRKS18/oiEXlIRK47+VUu2549+1iTuQGAw4ePsHnzVlLr1fU6tkSb1i3Yvn0XO3d+SX5+Punps7mhZ7e4yq9avSrN2jblnTfeBaAgv4DD3x3hy21fsntH9gnzt+/ang9mf0j+j/ns2b2HnF25XND8F66uU7Fofv+J8HdfHi+ug4ymihyDfDrgdQHwhaqe+FsQQESG4x9RI0lEFuEfPWMJMExEWqjqyHBXOBTnnZdG82aXsHzFmkjEAVAvtS67s48NOJqdk0eb1i3iKj+lfl2+PXCQYc8ModFFjdiyfgv//uuL/PC/H0qdv3bKOWSt3lTy8/49+6mdUsvVdSoWze8/Ef7uyxPru8yhKrMHKSKVgCdU9SNn+qS84uj4Nf6LOzvgvx6pt6o+hX80jb4nu9IVUbVqFdKnj+ehPw/n0KHDkYhMGJWSKnH+JU2Y/cpcftf9Xv73/Q/cNvCWaK+WiQEJtYutqoVAkYjUCPFzC1S1UFW/B7YX31iuqv/DuWUxmMAb14uKjoQY65eUlMSM6eN5441ZvP32u2F9Rrhyc/ZQP+3YUPlpqSnk5u6Jq/z9efvZn7efTWs2A/DRO0tp0jT4sa79eV9TO+VnJT/Xrlub/XlfubpOxaL5/SfC33154m0XuyJHyg/jv8VngjNK71gRGVvOMj+KSBXn9WXFjU6hLbNABt647vNVrcDqnWj8uNFs2ryN58accM+751ZmZNK4cUMaNKhPcnIyffr0Yu68hXGVf2D/N+zL3U/9n6cBcNmVLfli6xdB5//vov/SuVcnkisnU7d+XdIaprI583NX16lYNL//RPi7TzQVOQY505kClXekoYOqHgVQ/cmTcpM5dnO5J9pf0Zo7+v2adeuzyFjp/5/j8cdH8e57H3gZW6KwsJBBgx9j/juvU8nnY/KU6WRlbYlIdiTzxz7+PI/9+xGSKieT90Ueo/70L67s3p5BT91PjZo1+L8pI9m2cTtD+w1j15Yv+HDuR0z+YAKFhYU899hYioq8eYByNL//RPm7L0us7zKHSrSco6oiMkhVx5TX5oWkyqlRO+QbC08VjHa+PdUwgfPD3Pf9rN5NIf/OXp47M2arakV2sUvr8d3l8noYY+JAvJ2kCbqLLSK3ArcBDUVkTsBb1YEyhwgyxiSmWD/pEqqyjkH+F8gDagGjA9oPAeu8XCljzKnJmyPL0RO0QKrqF/gHoWxX1geIyKeqWuY8xpjEoEGHbTw1ufFUw9Nd+AxjTBwoirM7adwokHH2lRhjwlVkPUhjjCldvO1iV2Q0nwecJ4AFncXF9THGnMKKwphiWUWug6wDrBSRdBHp7jyCMdAdHqyXMeYUpEjIUyyryIjijwFN8D+X9i5gq4j8XUQaOe9v8HQNjTGnjETsQRY/RnGPMxUAZwNvisg/PVw3Y8wpJt4KZEWeajgIuBP4CngZGKKq+SLiA7YCQ71dRWPMqSLWd5lDVZGz2DWBm5wLx0uoapGI9PBmtYwxp6I4eyx2+QVSVYeX8d6mYO8ZYxJPvF0H6c2j5YwxJg7E9IXixePiWX50FI/LGC3R3v5Ezw9HvN1WF9MFMtoDxlp+Ag8Ya/lh8eKstIhMBHoA+1T1EqftCeB3wH5ntkdVdb7z3iPAPUAh8EdVXeC0dwfGAJWAl1V1VHnZMV0gjTGnlqIT7iNxxWTgeWDqce3PqmrgY6kRkYuAW4CLgXrA+yJyvvP2C8A1QDb+m1/mqGpWWcFWII0xrvFiF1tVl4pIgwrO3guY5jwTa6eIbAPaOO9tU9UdACIyzZm3zAJpJ2mMMa6J8IXi94vIOhGZGDBeRCqwO2CebKctWHuZrEAaY1xTJKFPIjJARDICpgEViPoP0Ahojv/JB6PLnDtMtottjHFNONdBquo4IKSH2Kvq3uLXIjIemOf8mAPUD5g1zWmjjPagrAdpjHGNhjGFQ0RSAn68ESgeNGcOcIuInCYiDfEPtLMCWAk0EZGGIlIZ/4mcwIcRlsp6kMYY13hxq6GIvAF0AmqJSDYwHOgkIs3x19hdwO8BVHWjiKTjP/lSAAxU1ULnc+4HFuC/zGeiqm4sL9sKpDHGNV5cB6mqt5bSPKGM+UcCI0tpnw/MDyXbCqQxxjV2J40xxgSRcKP5GGNMRcX6ALihsgJpjHGNFUhjjAlC42wXO+6ugxw/bjS52WvJXBOdobqinQ/QrWsnNm5YyuasZQwdMjDh8mvUOJPp08axYf1HrF/3IZe3vSxi2dHe9mjnx9szaeKuQE6dms71PW5P2Hyfz8fYMSPp0bMfTZtdTd++vbnwwiYJkw/w7DNPsmDBEi5p2pGWl13Dps1bI5Ib7W2Pdn48iliBFJHjhyryxMfLlnPgm28jERWT+W1at2D79l3s3Pkl+fn5pKfP5oae3RIm/8wzq3PVlW2ZOOkNAPLz8zl48LuIZEd726OdD9aDrBARmXPcNBe4qfhnLzKNX73UuuzOPjbgaXZOHvXq1U2Y/IYNz+Wrr75mwsvPsnLFAl76f/+iSpUzIpId7W2Pdj5E7lbDSPGqB5kGfAc8g3+UjdHAoYDXxngiqVIlWrRoyksvTaV1m24cOfI9Dw+9P9qrlTDCGc0nlnlVIFsBq4C/AAdV9UPgf6r6kap+VNaCgUMfFRUd8Wj14lduzh7qpx0bqj8tNYXc3D0Jk5+dk0d2dh4rVq4BYObMd2jRvGlEsqO97dHOB9vFrhBVLVLVZ4G7gb+IyPNU8JIiVR2nqq1UtZXPV9WL1YtrKzMyady4IQ0a1Cc5OZk+fXoxd97ChMnfu3c/2dm5nH9+IwA6d76STZu2RCQ72tse7XyIvwLp6XWQqpoN3Cwi1+Pf5fbcq6+8QMcO7ahVqya7dmQw4smnmTR5WiSiYyK/sLCQQYMfY/47r1PJ52PylOlkZUWmQMRCPsCgBx9n6pR/U7lyMjt3fsk9v30oIrnR3vZo50PsH1MMlajG7iYlVU6N2sol8lMFo50fK0/1S+h8De+S73+e1y/k39mhX7was0ci7U4aY4xrYn2XOVRWII0xrond/dHwWIE0xrimKM5KpBVIY4xrbBfbGGOCiK/+oxVIY4yLrAdpjDFBxPqtg6GyAmmMcY2dpDHGmCDiqzzG4YC5xhjjFutBGmNcYydpjDEmCDsGaYwxQcRXebQCaYxxke1iR1DxsE+Wb/mWf2qIt11sO4ttjHGNFw/tEpGJIrJPRDYEtNUUkUUistX582ynXURkrIhsE5F1ItIyYJn+zvxbRaR/RbYnpnuQiTpgbKLnx8SAsUBKjQujkp93cBMQ/e0Ph0e72JOB54HAR0cPAxar6igRGeb8/DBwLdDEmdoC/wHaikhNYDj+52UpsEpE5qjqN2UFWw/SGOMaDeO/cj9TdSlw4LjmXsAU5/UUoHdA+1T1+ww4S0RSgG7AIlU94BTFRUD38rKtQBpjXBPOQ7sCn2TqTAMqEFVHVfOc13uAOs7rVGB3wHzZTluw9jLF9C62MebUEs5JGlUdB4wLN1NVVUQ8OTtkPUhjjGu8OEkTxF5n1xnnz31Oew5QP2C+NKctWHuZrEAaY1xThIY8hWkOUHwmuj8wO6D9Tuds9uXAQWdXfAHQVUTOds54d3XaymS72MYY13hxFltE3gA6AbVEJBv/2ehRQLqI3AN8AfRxZp8PXAdsA74H7gZQ1QMi8hSw0pnvSVU9/sTPCaxAGmNcU5Gz0iF/puqtQd7qUsq8CgwM8jkTgYmhZFuBNMa4xm41NMaYILzoQUaTnaQxxpggrAdpjHGN7WIbY0wQRRpfu9hWII0xromv8hiHxyDHjxtNbvZaMtcsjto6dOvaiY0blrI5axlDh5R6xYHln8L59VLr8ubcSXz02Vw+/HQOv723HwBD//IAiz+ZxaKPZzJt5njq1K1dssxT/3iU/65+j8WfzKJpM+9GCYr2dx/BC8UjIu4K5NSp6Vzf4/ao5ft8PsaOGUmPnv1o2uxq+vbtzYUXNrH8OMovKChgxGP/pOPlPbn+mlu467e3cf4vGvHi2Il0aX8j11x1E4sWfMRDQ/8AQOdrOvDzn5/HFS27M2TQcEaNHu7q+hSL9ncP3ozmE00RKZAicqWIPCQiXb3O+njZcg58863XMUG1ad2C7dt3sXPnl+Tn55OePpsbenaz/DjK37f3K9av9Y/ZeOTw92zdsoO6KT/j8KEjJfNUqXIG6hyP635dZ2ZM898JtzpjHWfWqM7P6tRydZ0g+t89hDeaTyzzpECKyIqA17/DP9hldWC4M7hl3KqXWpfd2ccGHM3OyaNevbqWH6f5aefWo2nTC1m9ah0Awx4bRMaGxdx0cw/+9fd/A1A35Wfk5uwpWSYvdy8pKXVK/byTEe3vHmwXu6KSA14PAK5R1RH4bxCP3v6vMS6qUrUKE6aO4a+P/l9J73HU38bQ6pIuzJwxj7sHJN7/6raLXcHPdUbNOAcQVd0PoKpHgIKyFgwcPLOo6EhZs8ak3Jw91E87NlR+WmoKubl7yljC8k/F/KSkJCZMfY6ZM+Yxf+77J7w/c8Y8ru95DQB78vZRL/VYTy6lXh3y8va6vk7R/u7BdrErqgawCsgAagaM21YNkLIWVNVxqtpKVVv5fFU9Wj3vrMzIpHHjhjRoUJ/k5GT69OnF3HkLLT/O8p95/im2btnBSy9MKWlr+PPzSl53u64z27buAGDBux9w8y29AGjZ6lIOfXeIfXu/cn2dov3dA6hqyFMs8+Q6SFVtEOStIuBGLzKLvfrKC3Ts0I5atWqya0cGI558mkmTp3kZ+ROFhYUMGvwY8995nUo+H5OnTCcra4vlx1F+m8tbcvMtvcja+DmLPp4JwP89+Ry33XETjRo3pEiLyN6dy8MPjgBg8cKldLmmA5+ueY//ff8DDw78i6vrUyza3z3E32NfJZYreFLl1KitXCI/VTDa+fZUwxh4qqFqmXt6wfQ8t0fIv7Nzv5wXVlYk2J00xhjXxPpJl1BZgTTGuCbedrGtQBpjXBPLh+zCYQXSGOOaWL9sJ1RWII0xrom3Y5BxN1iFMca4xXqQxhjX2EkaY4wJwk7SGGNMENaDNMaYIOLtJI0VSGOMa+yhXcYYE0R8lUcrkMYYF8XbMUi7DtIY4xqvHrkgIrtEZL2IZIpIhtNWU0QWichW58+znXYRkbEisk1E1olIy3C3J6Z7kMXDTlm+5UdD8bBj0RLt7Q+Hx5f5XK2qgSMNDwMWq+oo51lXw4CHgWuBJs7UFviP82fIrAdpjHFNhB/a1QsoHtJ9CtA7oH2q+n0GnFX8VINQxXQPMlEHjE30/FgZMDfa+c3qtItK/tq9n4a9rIeX+SiwUEQUeElVxwF1VDXPeX8PUPyoyFRgd8Cy2U5bHiGK6QJpjDm1hLOLLSID8D/9tNg4pwAGulJVc0TkZ8AiEdl8XK46xdNVViCNMa4JZ5fZKYbHF8Tj58lx/twnIrOANsBeEUlR1TxnF3qfM3sOUD9g8TSnLWR2DNIY4xovnmooIlVFpHrxa6ArsAGYA/R3ZusPzHZezwHudM5mXw4cDNgVD4n1II0xrvHoOsg6wCwRAX/Nel1V3xORlUC6iNwDfAH0ceafD1wHbAO+B+4ON9gKpDHGNV6cpFHVHUCzUtq/BrqU0q7AQDeybRfbGGOCsB6kMcY1NliFMcYEYcOdGWNMENaDNMaYIKwHaYwxQVgP0hhjgrAepDHGBBFvPci4uw5y/LjR5GavJXPN4qitQ7eundi4YSmbs5YxdIgr16ueMvlpafV4f+EM1q1dwtrMD3jg/nsimg/R3f5IZvt8PqYvmsy/X/nXT9of/tuDfLr9/ZKfU9LqMm7GWGZ8MJWXZz7Pz1Jqe7ZOGsZ/sSzuCuTUqelc3+P2qOX7fD7GjhlJj579aNrsavr27c2FFzZJmPyCggKGDB3Bpc2upv2VPbnvvrsSZvsjnX377/qwY+uun7Rd1OwCzqxR/SdtDw2/n7kz3uXmzncybvQkBj16n2frpFoU8hTLPCmQItJWRM50Xp8hIiNEZK6I/ENEaniRWezjZcs58M23XkaUqU3rFmzfvoudO78kPz+f9PTZ3NCzW8Lk79mzjzWZGwA4fPgImzdvJbVe3YjlR3P7I5n9s5TaXPXLK5j12tySNp/Px0N/HcizT73wk3kbnd+AFctWAbDik1V06n6VJ+sEER8w13Ne9SAn4r9JHGAMUAP4h9M2yaPMmFAvtS67s48NlZ+dk0e9CBaIaOcHOu+8NJo3u4TlK9ZELDOa2x/J7KFPDebZp16gKKAHdstvfs2HC5bx1b6vfzLv5xu30eW6TgB0ua4j1apXpcbZZ3qyXl6M5hNNXhVIn6oWOK9bqepgVV2mqiOAn5e1oIgMEJEMEckoKjri0eoZr1WtWoX06eN56M/DOXTocLRXJ650uOYKDnz1DZvWfV7SVrtOLbr2vJo3Jrx5wvzPjHieVu2aM33RZC5r14K9ufsoKvRm1zbeepBencXeICJ3q+okYK2ItFLVDBE5H8gva8HAwTOTKqfG9rdXitycPdRPOzZUf1pqCrm5exImHyApKYkZ08fzxhuzePvtdyOaHc3tj1R289aX0qnrlVzZpR2nnVaZqtWqMvOjV/nxx3zmfpYOwOlnnM7cT9Pp2a4P+/d+xUP3PArAGVXO4JfXd+LQd978oxXrPcJQedWD/C3QUUS2AxcBn4rIDmC8817cWpmRSePGDWnQoD7Jycn06dOLufMWJkw++K8k2LR5G8+NKXOQaE9Ec/sjlT327/+Pri17c13rX/HwvX9l5SeruOqC7nS5tCfXtf4V17X+FT/87wd6tvMPj3hWzRo4Yylyzx/v5O1p81xfp2JFqiFPscyTHqSqHgTuck7UNHRyslV1rxd5gV595QU6dmhHrVo12bUjgxFPPs2kydO8ji1RWFjIoMGPMf+d16nk8zF5ynSysrYkTH77K1pzR79fs259Fhkr/cXh8cdH8e57H0QkP5rbH+3vPphWV7Tkj4/eC6qs+iyTvz8y2rOsWL9sJ1QSy13iaO5iJ/JTBaOdHytPFYx2flSfaqgq4Sxbp8YFIf/O7j24OaysSIi76yCNMcYtdquhMcY1sX5WOlRWII0xronlQ3bhsAJpjHFNrJ+VDpUVSGOMa6wHaYwxQdgxSGOMCcJ6kMYYE4QdgzTGmCDi7U4aK5DGGNdYD9IYY4KIt2OQdquhMcY1Xj2TRkS6i8jnIrJNRIZ5vBklrAdpjHGNFz1IEakEvABcA2QDK0VkjqpmuR52HOtBGmNc49EjF9oA21R1h6r+CEwDenm6IY6Y7kEWD/tk+ZafiPlr934a1fxweHQEMhXYHfBzNtDWm6ifiukCGe6YdMVEZIDzCIeIi2a25Vt+tPILfswJ+XdWRAYAAwKaxkXzuwsU77vYA8qfJS6zLd/yo51fYao6TlVbBUzHF8ccoH7Az2lOm+fivUAaY059K4EmItJQRCoDtwBzIhEc27vYxpiEp6oFInI/sACoBExU1Y2RyI73AhnN4xjRPoZi+ZYfN1R1PjA/0rkx/dAuY4yJJjsGaYwxQcRlgYzWbUlO9kQR2SciGyKZG5BfX0SWiEiWiGwUkUERzj9dRFaIyFonf0Qk8511qCQia0RkXqSznfxdIrJeRDJFJCPC2WeJyJsisllENolIdJ4dGyfibhfbuS1pCwG3JQG3RuK2JCe/A3AYmKqql0Qi87j8FCBFVVeLSHVgFdA7gtsvQFVVPSwiycAyYJCqfhaJfGcdHgJaAWeqao9I5Qbk7wJaqepXUcieAnysqi87Z3yrqOq3kV6PeBGPPcio3ZYEoKpLgQORyislP09VVzuvDwGb8N+JEKl8VdXDzo/JzhSxf4VFJA24Hng5UpmxQkRqAB2ACQCq+qMVx5MTjwWytNuSIlYgYomINABaAMsjnFtJRDKBfcAiVY1k/nPAUKAogpnHU2ChiKxy7hKJlIbAfmCSc4jhZRGpGsH8uBOPBdIAIlINeAsYrKrfRTJbVQtVtTn+Ox7aiEhEDjWISA9gn6quikReGa5U1ZbAtcBA57BLJCQBLYH/qGoL4AgQ0WPw8SYeC2TUbkuKFc6xv7eA11R1ZrTWw9m9WwJ0j1Bke+AG5xjgNKCziLwaoewSqprj/LkPmIX/sE8kZAPZAT32N/EXTBOmeCyQUbstKRY4J0kmAJtU9Zko5NcWkbOc12fgP1m2ORLZqvqIqqapagP8f+8fqGq/SGQXE5GqzskxnN3brkBErmhQ1T3AbhH5hdPUBYjIybl4FXd30kTztiQAEXkD6ATUEpFsYLiqTohUPv5e1B3Aeuc4IMCjzp0IkZACTHGuJvAB6aoalcttoqQOMMv/7xRJwOuq+l4E8x8AXnM6BzuAuyOYHXfi7jIfY4xxSzzuYhtjjCusQBpjTBBWII0xJggrkMYYE4QVSGOMCcIKpDHGBGEF0sQUEblLRJ6P9noYA1YgTYQ4F44bc0qxAmlKJSJPisjggJ9Hljb4roh0EpGlIvKOM0jx/xMRn/PeYREZLSJrgXYi0s8ZTDdTRF4qLpoicreIbBGRFfjvBDImJliBNMFMBO4EcAreLUCwgR/a4L/F7SKgEXCT014VWK6qzYCvgb5Ae2ekn0LgdmeA3xH4C+OVzmcYExPi7l5s4w5V3SUiX4tIC/z3F69R1a+DzL5CVXdAyb3oV+IfSaYQ/6hC4B844TJgpXOf8hn4x4tsC3yoqvud5acD53uzVcaExgqkKcvLwF1AXfw9ymCOv6G/+OcfVLXQeS3AFFV9JHBGEel98qtpjDdsF9uUZRb+sRxb4x8dKZg2zvByPvy70ctKmWcx8GsR+RmAiNQUkfPwj3beUUTOccaxvNnVLTDmJFgP0gSlqj+KyBLg24CeYGlWAs8DjfEPkDurlM/KEpHH8D+KwAfkAwNV9TMReQL4FPgWyHR1I4w5CTbcmQnKKWSrgZtVdWuQeToBf47G0wON8ZrtYptSichFwDZgcbDiaEy8sx6kqRARaQq8clzzUVVtG431MSYSrEAaY0wQtottjDFBWIE0xpggrEAaY0wQViCNMSYIK5DGGBPE/weHNF6Zh5rd/wAAAABJRU5ErkJggg==\n", 412 | "text/plain": [ 413 | "
" 414 | ] 415 | }, 416 | "metadata": { 417 | "needs_background": "light" 418 | }, 419 | "output_type": "display_data" 420 | }, 421 | { 422 | "name": "stdout", 423 | "output_type": "stream", 424 | "text": [ 425 | "Wall time: 49.3 s\n" 426 | ] 427 | } 428 | ], 429 | "source": [ 430 | "%%time\n", 431 | "# Train the CatBoost algorithm\n", 432 | "import catboost as cbt\n", 433 | "cb = cbt.CatBoostClassifier(verbose=0,boosting_type='Plain')\n", 434 | "#cb = cbt.CatBoostClassifier()\n", 435 | "\n", 436 | "cb.fit(X_train, y_train)\n", 437 | "y_pred = cb.predict(X_test)\n", 438 | "print(classification_report(y_test,y_pred))\n", 439 | "print(\"Accuracy of CatBoost: \"+ str(accuracy_score(y_test, y_pred)))\n", 440 | "print(\"Precision of CatBoost: \"+ str(precision_score(y_test, y_pred, average='weighted')))\n", 441 | "print(\"Recall of CatBoost: \"+ str(recall_score(y_test, y_pred, average='weighted')))\n", 442 | "print(\"Average F1 of CatBoost: \"+ str(f1_score(y_test, y_pred, average='weighted')))\n", 443 | "print(\"F1 of CatBoost for each type of attack: \"+ str(f1_score(y_test, y_pred, average=None)))\n", 444 | "cb_f1=f1_score(y_test, y_pred, average=None)\n", 445 | "\n", 446 | "# Plot the confusion matrix\n", 447 | "cm=confusion_matrix(y_test,y_pred)\n", 448 | "f,ax=plt.subplots(figsize=(5,5))\n", 449 | "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n", 450 | "plt.xlabel(\"y_pred\")\n", 451 | "plt.ylabel(\"y_true\")\n", 452 | "plt.show()" 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "## Proposed ensemble model: Leader Class and Confidence Decision Ensemble (LCCDE)" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "LCCDE aims to achieve optimal model performance by identifying the best-performing base ML model with the highest prediction confidence for each class. " 467 | ] 468 | }, 469 | { 470 | "cell_type": "markdown", 471 | "metadata": {}, 472 | "source": [ 473 | "### Find the best-performing (leading) model for each type of attack among the three ML models" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": 13, 479 | "metadata": {}, 480 | "outputs": [], 481 | "source": [ 482 | "# Leading model list for each class\n", 483 | "model=[]\n", 484 | "for i in range(len(lg_f1)):\n", 485 | " if max(lg_f1[i],xg_f1[i],cb_f1[i]) == lg_f1[i]:\n", 486 | " model.append(lg)\n", 487 | " elif max(lg_f1[i],xg_f1[i],cb_f1[i]) == xg_f1[i]:\n", 488 | " model.append(xg)\n", 489 | " else:\n", 490 | " model.append(cb)" 491 | ] 492 | }, 493 | { 494 | "cell_type": "code", 495 | "execution_count": 14, 496 | "metadata": { 497 | "scrolled": false 498 | }, 499 | "outputs": [ 500 | { 501 | "data": { 502 | "text/plain": [ 503 | "[XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", 504 | " colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n", 505 | " importance_type='gain', interaction_constraints='',\n", 506 | " learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n", 507 | " min_child_weight=1, missing=nan, monotone_constraints='()',\n", 508 | " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", 509 | " objective='multi:softprob', random_state=0, reg_alpha=0,\n", 510 | " reg_lambda=1, scale_pos_weight=None, subsample=1,\n", 511 | " tree_method='exact', validate_parameters=1, verbosity=None),\n", 512 | " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", 513 | " colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n", 514 | " importance_type='gain', interaction_constraints='',\n", 515 | " learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n", 516 | " min_child_weight=1, missing=nan, monotone_constraints='()',\n", 517 | " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", 518 | " objective='multi:softprob', random_state=0, reg_alpha=0,\n", 519 | " reg_lambda=1, scale_pos_weight=None, subsample=1,\n", 520 | " tree_method='exact', validate_parameters=1, verbosity=None),\n", 521 | " LGBMClassifier(),\n", 522 | " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", 523 | " colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n", 524 | " importance_type='gain', interaction_constraints='',\n", 525 | " learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n", 526 | " min_child_weight=1, missing=nan, monotone_constraints='()',\n", 527 | " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", 528 | " objective='multi:softprob', random_state=0, reg_alpha=0,\n", 529 | " reg_lambda=1, scale_pos_weight=None, subsample=1,\n", 530 | " tree_method='exact', validate_parameters=1, verbosity=None),\n", 531 | " LGBMClassifier(),\n", 532 | " LGBMClassifier(),\n", 533 | " XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", 534 | " colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n", 535 | " importance_type='gain', interaction_constraints='',\n", 536 | " learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n", 537 | " min_child_weight=1, missing=nan, monotone_constraints='()',\n", 538 | " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", 539 | " objective='multi:softprob', random_state=0, reg_alpha=0,\n", 540 | " reg_lambda=1, scale_pos_weight=None, subsample=1,\n", 541 | " tree_method='exact', validate_parameters=1, verbosity=None)]" 542 | ] 543 | }, 544 | "execution_count": 14, 545 | "metadata": {}, 546 | "output_type": "execute_result" 547 | } 548 | ], 549 | "source": [ 550 | "model" 551 | ] 552 | }, 553 | { 554 | "cell_type": "markdown", 555 | "metadata": {}, 556 | "source": [ 557 | "**Leading Model for Each Type of Attack:** \n", 558 | "0 BENIGN:   XGBClassifier \n", 559 | "1 Bot:     XGBClassifier \n", 560 | "2 BruteForce:   LGBMClassifier \n", 561 | "3 DoS:     XGBClassifier \n", 562 | "4 Infiltration:   LGBMClassifier \n", 563 | "5 PortScan:   LGBMClassifier \n", 564 | "6 WebAttack:   XGBClassifier " 565 | ] 566 | }, 567 | { 568 | "cell_type": "markdown", 569 | "metadata": { 570 | "collapsed": true 571 | }, 572 | "source": [ 573 | "## LCCDE Prediction" 574 | ] 575 | }, 576 | { 577 | "cell_type": "code", 578 | "execution_count": 15, 579 | "metadata": {}, 580 | "outputs": [], 581 | "source": [ 582 | "def LCCDE(X_test, y_test, m1, m2, m3):\n", 583 | " i = 0\n", 584 | " t = []\n", 585 | " m = []\n", 586 | " yt = []\n", 587 | " yp = []\n", 588 | " l = []\n", 589 | " pred_l = []\n", 590 | " pro_l = []\n", 591 | "\n", 592 | " # For each class (normal or a type of attack), find the leader model\n", 593 | " for xi, yi in stream.iter_pandas(X_test, y_test):\n", 594 | "\n", 595 | " xi2=np.array(list(xi.values()))\n", 596 | " y_pred1 = m1.predict(xi2.reshape(1, -1)) # model 1 (LightGBM) makes a prediction on text sample xi\n", 597 | " y_pred1 = int(y_pred1[0])\n", 598 | " y_pred2 = m2.predict(xi2.reshape(1, -1)) # model 2 (XGBoost) makes a prediction on text sample xi\n", 599 | " y_pred2 = int(y_pred2[0])\n", 600 | " y_pred3 = m3.predict(xi2.reshape(1, -1)) # model 3 (Catboost) makes a prediction on text sample xi\n", 601 | " y_pred3 = int(y_pred3[0])\n", 602 | "\n", 603 | " p1 = m1.predict_proba(xi2.reshape(1, -1)) # The prediction probability (confidence) list of model 1 \n", 604 | " p2 = m2.predict_proba(xi2.reshape(1, -1)) # The prediction probability (confidence) list of model 2 \n", 605 | " p3 = m3.predict_proba(xi2.reshape(1, -1)) # The prediction probability (confidence) list of model 3 \n", 606 | "\n", 607 | " # Find the highest prediction probability among all classes for each ML model\n", 608 | " y_pred_p1 = np.max(p1)\n", 609 | " y_pred_p2 = np.max(p2)\n", 610 | " y_pred_p3 = np.max(p3)\n", 611 | "\n", 612 | " if y_pred1 == y_pred2 == y_pred3: # If the predicted classes of all the three models are the same\n", 613 | " y_pred = y_pred1 # Use this predicted class as the final predicted class\n", 614 | "\n", 615 | " elif y_pred1 != y_pred2 != y_pred3: # If the predicted classes of all the three models are different\n", 616 | " # For each prediction model, check if the predicted class’s original ML model is the same as its leader model\n", 617 | " if model[y_pred1]==m1: # If they are the same and the leading model is model 1 (LightGBM)\n", 618 | " l.append(m1)\n", 619 | " pred_l.append(y_pred1) # Save the predicted class\n", 620 | " pro_l.append(y_pred_p1) # Save the confidence\n", 621 | "\n", 622 | " if model[y_pred2]==m2: # If they are the same and the leading model is model 2 (XGBoost)\n", 623 | " l.append(m2)\n", 624 | " pred_l.append(y_pred2)\n", 625 | " pro_l.append(y_pred_p2)\n", 626 | "\n", 627 | " if model[y_pred3]==m3: # If they are the same and the leading model is model 3 (CatBoost)\n", 628 | " l.append(m3)\n", 629 | " pred_l.append(y_pred3)\n", 630 | " pro_l.append(y_pred_p3)\n", 631 | "\n", 632 | " if len(l)==0: # Avoid empty probability list\n", 633 | " pro_l=[y_pred_p1,y_pred_p2,y_pred_p3]\n", 634 | "\n", 635 | " elif len(l)==1: # If only one pair of the original model and the leader model for each predicted class is the same\n", 636 | " y_pred=pred_l[0] # Use the predicted class of the leader model as the final prediction class\n", 637 | "\n", 638 | " else: # If no pair or multiple pairs of the original prediction model and the leader model for each predicted class are the same\n", 639 | " max_p = max(pro_l) # Find the highest confidence\n", 640 | " \n", 641 | " # Use the predicted class with the highest confidence as the final prediction class\n", 642 | " if max_p == y_pred_p1:\n", 643 | " y_pred = y_pred1\n", 644 | " elif max_p == y_pred_p2:\n", 645 | " y_pred = y_pred2\n", 646 | " else:\n", 647 | " y_pred = y_pred3 \n", 648 | " \n", 649 | " else: # If two predicted classes are the same and the other one is different\n", 650 | " n = mode([y_pred1,y_pred2,y_pred3]) # Find the predicted class with the majority vote\n", 651 | " y_pred = model[n].predict(xi2.reshape(1, -1)) # Use the predicted class of the leader model as the final prediction class\n", 652 | " y_pred = int(y_pred[0]) \n", 653 | "\n", 654 | " yt.append(yi)\n", 655 | " yp.append(y_pred) # Save the predicted classes for all tested samples\n", 656 | " return yt, yp" 657 | ] 658 | }, 659 | { 660 | "cell_type": "code", 661 | "execution_count": 16, 662 | "metadata": {}, 663 | "outputs": [ 664 | { 665 | "name": "stdout", 666 | "output_type": "stream", 667 | "text": [ 668 | "Wall time: 59.1 s\n" 669 | ] 670 | } 671 | ], 672 | "source": [ 673 | "%%time\n", 674 | "# Implementing LCCDE\n", 675 | "yt, yp = LCCDE(X_test, y_test, m1 = lg, m2 = xg, m3 = cb)" 676 | ] 677 | }, 678 | { 679 | "cell_type": "code", 680 | "execution_count": 17, 681 | "metadata": {}, 682 | "outputs": [ 683 | { 684 | "name": "stdout", 685 | "output_type": "stream", 686 | "text": [ 687 | "Accuracy of LCCDE: 0.9975746268656717\n", 688 | "Precision of LCCDE: 0.9975807247413017\n", 689 | "Recall of LCCDE: 0.9975746268656717\n", 690 | "Average F1 of LCCDE: 0.9975482770384609\n", 691 | "F1 of LCCDE for each type of attack: [0.99836021 0.99351492 1. 0.99836334 0.85714286 0.99137931\n", 692 | " 0.99889258]\n" 693 | ] 694 | } 695 | ], 696 | "source": [ 697 | "# The performance of the proposed lCCDE model\n", 698 | "print(\"Accuracy of LCCDE: \"+ str(accuracy_score(yt, yp)))\n", 699 | "print(\"Precision of LCCDE: \"+ str(precision_score(yt, yp, average='weighted')))\n", 700 | "print(\"Recall of LCCDE: \"+ str(recall_score(yt, yp, average='weighted')))\n", 701 | "print(\"Average F1 of LCCDE: \"+ str(f1_score(yt, yp, average='weighted')))\n", 702 | "print(\"F1 of LCCDE for each type of attack: \"+ str(f1_score(yt, yp, average=None)))" 703 | ] 704 | }, 705 | { 706 | "cell_type": "code", 707 | "execution_count": 18, 708 | "metadata": { 709 | "scrolled": true 710 | }, 711 | "outputs": [ 712 | { 713 | "name": "stdout", 714 | "output_type": "stream", 715 | "text": [ 716 | "F1 of LightGBM for each type of attack: [0.99795054 0.99092088 1. 0.997543 0.85714286 0.99354839\n", 717 | " 0.99778271]\n", 718 | "F1 of XGBoost for each type of attack: [0.99836021 0.99351492 1. 0.99836334 0.85714286 0.99137931\n", 719 | " 0.99889258]\n", 720 | "F1 of CatBoost for each type of attack: [0.99781241 0.99222798 1. 0.99591837 0.85714286 0.99137931\n", 721 | " 0.9944629 ]\n" 722 | ] 723 | } 724 | ], 725 | "source": [ 726 | "# Comparison: The F1-scores for each base model\n", 727 | "print(\"F1 of LightGBM for each type of attack: \"+ str(lg_f1))\n", 728 | "print(\"F1 of XGBoost for each type of attack: \"+ str(xg_f1))\n", 729 | "print(\"F1 of CatBoost for each type of attack: \"+ str(cb_f1))" 730 | ] 731 | }, 732 | { 733 | "cell_type": "markdown", 734 | "metadata": { 735 | "collapsed": true 736 | }, 737 | "source": [ 738 | "**Conclusion**: The performance (F1-score) of the proposed LCCDE ensemble model on each type of attack detection is higher than any base ML model." 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": null, 744 | "metadata": { 745 | "collapsed": true 746 | }, 747 | "outputs": [], 748 | "source": [] 749 | } 750 | ], 751 | "metadata": { 752 | "anaconda-cloud": {}, 753 | "kernelspec": { 754 | "display_name": "Python 3", 755 | "language": "python", 756 | "name": "python3" 757 | }, 758 | "language_info": { 759 | "codemirror_mode": { 760 | "name": "ipython", 761 | "version": 3 762 | }, 763 | "file_extension": ".py", 764 | "mimetype": "text/x-python", 765 | "name": "python", 766 | "nbconvert_exporter": "python", 767 | "pygments_lexer": "ipython3", 768 | "version": "3.6.8" 769 | } 770 | }, 771 | "nbformat": 4, 772 | "nbformat_minor": 2 773 | } 774 | -------------------------------------------------------------------------------- /LCCDE_IDS_GlobeCom22_paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/LCCDE_IDS_GlobeCom22_paper.pdf -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Western OC2 Lab 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MTH_IDS_paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/MTH_IDS_paper.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Intrusion-Detection-System-Using-Machine-Learning 2 | 3 | This repository contains the code for the project "IDS-ML: Intrusion Detection System Development Using Machine Learning". The code and proposed Intrusion Detection System (IDSs) are general models that can be used in any IDS and anomaly detection applications. In this project, three papers have been published: 4 | * L. Yang, A. Moubayed, I. Hamieh and A. Shami, "[Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles](https://arxiv.org/pdf/1910.08635.pdf)," in 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1-6, doi: 10.1109/GLOBECOM38437.2019.9013892. 5 | * L. Yang, A. Moubayed, and A. Shami, “[MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles](https://arxiv.org/pdf/2105.13289.pdf),” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 616-632, Jan.1, 2022, doi: 10.1109/JIOT.2021.3084796. 6 | * L. Yang, A. Shami, G. Stevens, and S. DeRusett, “[LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles](https://arxiv.org/pdf/2208.03399.pdf)," in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6, doi: 10.1109/GLOBECOM48099.2022.10001280. 7 | 8 | 9 | The code introduction of this repository is publicly available at: 10 | * L. Yang, and A. Shami, “[IDS-ML: An open source code for Intrusion Detection System development using Machine Learning](https://www.sciencedirect.com/science/article/pii/S2665963822001300)," Software Impacts, vol. 14, pp. 1-4, 2022, doi: 10.1016/j.simpa.2022.100446. 11 | 12 | This repository proposed three **intrusion detection systems** by implementing many **machine learning** algorithms, including tree-based algorithms (**decision tree, random forest, XGBoost, LightGBM, CatBoost etc.**), unsupervised learning algorithms (**k-means**), ensemble learning algorithms (**stacking, proposed LCCDE**), and hyperparameter optimization techniques (**Bayesian optimization**)**. 13 | 14 | - Another **intrusion detection system development code** using **convolutional neural networks (CNNs)** and **transfer learning** techniques can be found in: [Intrusion-Detection-System-Using-CNN-and-Transfer-Learning](https://github.com/Western-OC2-Lab/Intrusion-Detection-System-Using-CNN-and-Transfer-Learning) 15 | 16 | - A comprehensive **hyperparameter optimization** tutorial code can be found in: [Hyperparameter-Optimization-of-Machine-Learning-Algorithms](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms) 17 | 18 | 19 | ## Paper Abstract 20 | ### Paper 1: Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles 21 |   The use of autonomous vehicles (AVs) is a promising technology in Intelligent Transportation Systems (ITSs) to improve safety and driving efficiency. Vehicle-to-everything (V2X) technology enables communication among vehicles and other infrastructures. However, AVs and Internet of Vehicles (IoV) are vulnerable to different types of cyber-attacks such as denial of service, spoofing, and sniffing attacks. An intelligent IDS is proposed in this paper for network attack detection that can be applied to not only Controller Area Network (CAN) bus of AVs but also on general IoVs. The proposed IDS utilizes tree-based ML algorithms including decision tree (DT), random forest (RF), extra trees (ET), and Extreme Gradient Boosting (XGBoost). The results from the implementation of the proposed intrusion detection system on standard data sets indicate that the system has the ability to identify various cyber-attacks in the AV networks. Furthermore, the proposed ensemble learning and feature selection approaches enable the proposed system to achieve high detection rate and low computational cost simultaneously. 22 | 23 | **

Figure 1: The overview of the tree-based IDS model.

** 24 |

25 | 26 |

27 | 28 | ### Paper 2: MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles 29 |   Modern vehicles, including connected vehicles and autonomous vehicles, nowadays involve many electronic control units connected through intra-vehicle networks to implement various functionalities and perform actions. Modern vehicles are also connected to external networks through vehicle-to-everything technologies, enabling their communications with other vehicles, infrastructures, and smart devices. However, the improving functionality and connectivity of modern vehicles also increase their vulnerabilities to cyber-attacks targeting both intra-vehicle and external networks due to the large attack surfaces. To secure vehicular networks, many researchers have focused on developing intrusion detection systems (IDSs) that capitalize on machine learning methods to detect malicious cyber-attacks. In this paper, the vulnerabilities of intra-vehicle and external networks are discussed, and a multi-tiered hybrid IDS that incorporates a signature-based IDS and an anomaly-based IDS is proposed to detect both known and unknown attacks on vehicular networks. Experimental results illustrate that the proposed system can accurately detect various types of known attacks on the CAN-intrusion-dataset representing the intra-vehicle network data and the CICIDS2017 dataset illustrating the external vehicular network data. 30 |   The proposed MTH-IDS framework consists of two traditional ML stages (data pre-processing and feature engineering) and four tiers of learning models: 31 | 1. Four tree-based supervised learners — decision tree (DT), random forest (RF), extra trees (ET), and extreme gradient boosting (XGBoost) — used as multi-class classifiers for known attack detection; 32 | 2. A stacking ensemble model and a Bayesian optimization with tree Parzen estimator (BO-TPE) method for supervised learner optimization; 33 | 3. A cluster labeling (CL) k-means used as an unsupervised learner for zero-day attack detection; 34 | 4. Two biased classifiers and a Bayesian optimization with Gaussian process (BO-GP) method for unsupervised learner optimization. 35 | 36 | **

Figure 2: The overview of the MTH-IDS model.

** 37 |

38 | 39 |

40 | 41 | 42 | ### Paper 3: LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles 43 |   Modern vehicles, including autonomous vehicles and connected vehicles, have adopted an increasing variety of functionalities through connections and communications with other vehicles, smart devices, and infrastructures. However, the growing connectivity of the Internet of Vehicles (IoV) also increases the vulnerabilities to network attacks. To protect IoV systems against cyber threats, Intrusion Detection Systems (IDSs) that can identify malicious cyber-attacks have been developed using Machine Learning (ML) approaches. To accurately detect various types of attacks in IoV networks, we propose a novel ensemble IDS framework named Leader Class and Confidence Decision Ensemble (LCCDE). It is constructed by determining the best-performing ML model among three advanced ML algorithms (XGBoost, LightGBM, and CatBoost) for every class or type of attack. The class leader models with their prediction confidence values are then utilized to make accurate decisions regarding the detection of various types of cyber-attacks. Experiments on two public IoV security datasets (Car-Hacking and CICIDS2017 datasets) demonstrate the effectiveness of the proposed LCCDE for intrusion detection on both intra-vehicle and external networks. 44 | 45 | **

Figure 3: The overview of the LCCCDE IDS model.

** 46 |

47 | 48 |

49 | 50 | 51 | ## Implementation 52 | ### Dataset 53 | CICIDS2017 dataset, a popular network traffic dataset for intrusion detection problems 54 | * Publicly available at: https://www.unb.ca/cic/datasets/ids-2017.html 55 | * For the purpose of displaying the experimental results in Jupyter Notebook, the sampled subsets of CICIDS2017 is used in the sample code. The subsets are in the "data" folder. 56 | 57 | CAN-intrusion dataset, a benchmark network security dataset for intra-vehicle intrusion detection 58 | * Publicly available at: https://ocslab.hksecurity.net/Datasets/CAN-intrusion-dataset 59 | * Can be processed using the same code 60 | 61 | ### Code 62 | * [Tree-based_IDS_GlobeCom19.ipynb](https://github.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/blob/main/Tree-based_IDS_GlobeCom19.ipynb): code for the paper "Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles" 63 | * [MTH_IDS_IoTJ.ipynb](https://github.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/blob/main/MTH_IDS_IoTJ.ipynb): code for the paper "MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles" 64 | * [LCCDE_IDS_GlobeCom22.ipynb](https://github.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/blob/main/LCCDE_IDS_GlobeCom22.ipynb): code for the paper "LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles" 65 | 66 | ### Machine Learning Algorithms 67 | * Decision tree (DT) 68 | * Random forest (RF) 69 | * Extra trees (ET) 70 | * XGBoost 71 | * LightGBM 72 | * CatBoost 73 | * Stacking 74 | * K-means 75 | 76 | ### Hyperparameter Optimization Methods 77 | * Bayesian Optimization with Gaussian Processes (BO-GP) 78 | * Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE) 79 | 80 | If you are interested in hyperparameter tuning of machine learning algorithms, please see the code in the following link: 81 | https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms 82 | 83 | ### Requirements & Libraries 84 | * Python 3.6+ 85 | * [scikit-learn](https://scikit-learn.org/stable/) 86 | * [Xgboost](https://xgboost.readthedocs.io/en/latest/python/python_intro.html) 87 | * [lightgbm](https://lightgbm.readthedocs.io/en/v3.3.2/Python-Intro.html) 88 | * [catboost](https://xgboost.readthedocs.io/en/latest/python/python_intro.html) 89 | * [FCBF](https://github.com/SantiagoEG/FCBF_module) 90 | * [scikit-optimize](https://github.com/scikit-optimize/scikit-optimize) 91 | * [hyperopt](https://github.com/hyperopt/hyperopt) 92 | * [River](https://riverml.xyz/dev/) 93 | 94 | ## Contact-Info 95 | Please feel free to contact us for any questions or cooperation opportunities. We will be happy to help. 96 | * Email: [liyanghart@gmail.com](mailto:liyanghart@gmail.com) 97 | * GitHub: [LiYangHart](https://github.com/LiYangHart) and [Western OC2 Lab](https://github.com/Western-OC2-Lab/) 98 | * LinkedIn: [Li Yang](https://www.linkedin.com/in/li-yang-phd-65a190176/) 99 | * Google Scholar: [Li Yang](https://scholar.google.com.eg/citations?user=XEfM7bIAAAAJ&hl=en) and [OC2 Lab](https://scholar.google.com.eg/citations?user=oiebNboAAAAJ&hl=en) 100 | 101 | ## Citation 102 | If you find this repository useful in your research, please cite one of the following two articles as: 103 | 104 | L. Yang, A. Moubayed, I. Hamieh and A. Shami, "Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles," 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1-6, doi: 10.1109/GLOBECOM38437.2019.9013892. 105 | ``` 106 | @INPROCEEDINGS{9013892, 107 | author={Yang, Li and Moubayed, Abdallah and Hamieh, Ismail and Shami, Abdallah}, 108 | booktitle={2019 IEEE Global Communications Conference (GLOBECOM)}, 109 | title={Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles}, 110 | year={2019}, 111 | pages={1-6}, 112 | doi={10.1109/GLOBECOM38437.2019.9013892} 113 | } 114 | ``` 115 | 116 | L. Yang, A. Moubayed, and A. Shami, “MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 616-632, Jan.1, 2022, doi: 10.1109/JIOT.2021.3084796. 117 | ``` 118 | @ARTICLE{9443234, 119 | author={Yang, Li and Moubayed, Abdallah and Shami, Abdallah}, 120 | journal={IEEE Internet of Things Journal}, 121 | title={MTH-IDS: A Multitiered Hybrid Intrusion Detection System for Internet of Vehicles}, 122 | year={2022}, 123 | volume={9}, 124 | number={1}, 125 | pages={616-632}, 126 | doi={10.1109/JIOT.2021.3084796}} 127 | ``` 128 | 129 | L. Yang, A. Shami, G. Stevens, and S. DeRusett, “LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles," in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6, doi: 10.1109/GLOBECOM48099.2022.10001280. 130 | ``` 131 | @INPROCEEDINGS{10001280, 132 | author={Yang, Li and Shami, Abdallah and Stevens, Gary and de Rusett, Stephen}, 133 | booktitle={GLOBECOM 2022 - 2022 IEEE Global Communications Conference}, 134 | title={LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles}, 135 | year={2022}, 136 | pages={3545-3550}, 137 | doi={10.1109/GLOBECOM48099.2022.10001280}} 138 | ``` 139 | -------------------------------------------------------------------------------- /Tree-based_IDS_paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Western-OC2-Lab/Intrusion-Detection-System-Using-Machine-Learning/1b8b79d1711a97fee33d83639b3ae4e19f980ba6/Tree-based_IDS_paper.pdf -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- 1 | # The sampled datasets used for the experiments in the sample code 2 | 3 | **CICIDS2017_sample.csv**: The randomly sampled subset of CICIDS2017 4 | **CICIDS2017_sample_km.csv**: The subset of CICIDS2017 sampled by k-means clustering 5 | --------------------------------------------------------------------------------