├── Data ├── X_val_3bin.txt ├── X_val_3bin_15epoch.txt ├── sampletext.txt ├── y_val_3bin.txt └── y_val_3bin_15epoch.txt ├── JupyterNotebooks ├── Step 0 - Text - Exploration.ipynb ├── Step 0+Text+-+Exploration.ipynb ├── Step 1 - Text Preprocessing Filter Source Data - 3 Class Model.ipynb ├── Step 2 - Text Preprocessing Clean Text - for 3 Class Model.ipynb ├── Step 3 - Run 1DCNN Stock Text Classification to 3 Classes.ipynb └── Step 4 - Model Evaluation 3 Classes Reporting.ipynb ├── PythonScripts ├── 1DCNN_MODEL_StockText_2Class.py └── 1DCNN_MODEL_StockText_3Class.py ├── README.md └── images ├── 1D CNN.png ├── ModelSummary.png ├── Model_Training_accuracyandloss.png └── modelarchitecture.png /Data/y_val_3bin.txt: -------------------------------------------------------------------------------- 1 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 2 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 3 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 4 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 5 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 6 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 7 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 8 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 9 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 10 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 11 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 12 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 13 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 14 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 15 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 16 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 17 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 18 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 19 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 20 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 21 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 22 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 23 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 24 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 25 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 26 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 27 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 28 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 29 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 30 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 31 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 32 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 33 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 34 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 35 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 36 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 37 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 38 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 39 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 40 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 41 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 42 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 43 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 44 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 45 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 46 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 47 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 48 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 49 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 50 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 51 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 52 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 53 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 54 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 55 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 56 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 57 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 58 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 59 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 60 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 61 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 62 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 63 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 64 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 65 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 66 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 67 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 68 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 69 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 70 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 71 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 72 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 73 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 74 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 75 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 76 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 77 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 78 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 79 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 80 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 81 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 82 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 83 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 84 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 85 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 86 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 87 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 88 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 89 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 90 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 91 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 92 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 93 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 94 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 95 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 96 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 97 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 98 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 99 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 100 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 101 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 102 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 103 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 104 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 105 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 106 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 107 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 108 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 109 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 110 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 111 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 112 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 113 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 114 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 115 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 116 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 117 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 118 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 119 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 120 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 121 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 122 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 123 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 124 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 125 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 126 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 127 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 128 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 129 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 130 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 131 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 132 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 133 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 134 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 135 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 136 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 137 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 138 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 139 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 140 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 141 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 142 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 143 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 144 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 145 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 146 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 147 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 148 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 149 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 150 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 151 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 152 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 153 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 154 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 155 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 156 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 157 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 158 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 159 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 160 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 161 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 162 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 163 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 164 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 165 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 166 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 167 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 168 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 169 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 170 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 171 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 172 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 173 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 174 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 175 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 176 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 177 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 178 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 179 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 180 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 181 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 182 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 183 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 184 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 185 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 186 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 187 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 188 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 189 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 190 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 191 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 192 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 193 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 194 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 195 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 196 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 197 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 198 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 199 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 200 | -------------------------------------------------------------------------------- /Data/y_val_3bin_15epoch.txt: -------------------------------------------------------------------------------- 1 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 2 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 3 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 4 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 5 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 6 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 7 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 8 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 9 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 10 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 11 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 12 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 13 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 14 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 15 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 16 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 17 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 18 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 19 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 20 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 21 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 22 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 23 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 24 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 25 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 26 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 27 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 28 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 29 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 30 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 31 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 32 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 33 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 34 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 35 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 36 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 37 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 38 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 39 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 40 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 41 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 42 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 43 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 44 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 45 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 46 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 47 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 48 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 49 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 50 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 51 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 52 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 53 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 54 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 55 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 56 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 57 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 58 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 59 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 60 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 61 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 62 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 63 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 64 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 65 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 66 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 67 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 68 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 69 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 70 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 71 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 72 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 73 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 74 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 75 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 76 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 77 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 78 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 79 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 80 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 81 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 82 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 83 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 84 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 85 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 86 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 87 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 88 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 89 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 90 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 91 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 92 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 93 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 94 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 95 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 96 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 97 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 98 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 99 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 100 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 101 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 102 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 103 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 104 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 105 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 106 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 107 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 108 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 109 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 110 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 111 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 112 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 113 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 114 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 115 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 116 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 117 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 118 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 119 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 120 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 121 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 122 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 123 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 124 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 125 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 126 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 127 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 128 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 129 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 130 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 131 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 132 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 133 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 134 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 135 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 136 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 137 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 138 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 139 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 140 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 141 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 142 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 143 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 144 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 145 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 146 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 147 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 148 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 149 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 150 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 151 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 152 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 153 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 154 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 155 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 156 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 157 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 158 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 159 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 160 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 161 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 162 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 163 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 164 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 165 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 166 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 167 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 168 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 169 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 170 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 171 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 172 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 173 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 174 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 175 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 176 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 177 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 178 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 179 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 180 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 181 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 182 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 183 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 184 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 185 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 186 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 187 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 188 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 189 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 190 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 191 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 192 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 193 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 194 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 195 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 196 | 1.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00 197 | 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00 198 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 199 | 0.000000000000000000e+00,1.000000000000000000e+00,0.000000000000000000e+00 200 | -------------------------------------------------------------------------------- /JupyterNotebooks/Step 3 - Run 1DCNN Stock Text Classification to 3 Classes.ipynb: -------------------------------------------------------------------------------- 1 | 2 | { 3 | "cells": [ 4 | { 5 | "cell_type": "code", 6 | "execution_count": 1, 7 | "metadata": { 8 | "scrolled": true 9 | }, 10 | "outputs": [ 11 | { 12 | "name": "stderr", 13 | "output_type": "stream", 14 | "text": [ 15 | "Using Theano backend.\n", 16 | "C:\\Users\\pattyry\\AppData\\local\\AmlWorkbench\\Python\\lib\\site-packages\\gensim\\utils.py:862: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n", 17 | " warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n" 18 | ] 19 | } 20 | ], 21 | "source": [ 22 | "#############################################################################################################################################\n", 23 | "#\n", 24 | "# Stock future performance classification based on text\n", 25 | "#\n", 26 | "# Approach:\n", 27 | "#\n", 28 | "# Build on top of it a 1D convolutional neural network, ending in a softmax output over 3 even categories.\n", 29 | "# Use word Glove word vectors for large English text corpus as inputs model\n", 30 | "#\n", 31 | "# Steps\n", 32 | "# 1) After cleaning, we convert all text samples in the dataset into sequences of word indices. In this case, a \"word index\" would simply be an integer ID for the word. \n", 33 | "# 2) We consider the top 350,000 most commonly occuring words in the dataset\n", 34 | "# 3) We truncate the sequences to a maximum length of 25,000 words.\n", 35 | "# 5) We [repare an \"embedding matrix\" which will contain at index i the embedding vector for the word of index i in our word index.\n", 36 | "# 6) Then, we load this embedding matrix into a Keras Embedding layer, set to be frozen (its weights, the embedding vectors, will not be updated during training).\n", 37 | "#\n", 38 | "###############################################################################################################################################\n", 39 | "\n", 40 | "# import libraries\n", 41 | "from __future__ import print_function\n", 42 | "import numpy as np\n", 43 | "from six.moves import zip\n", 44 | "import json\n", 45 | "import warnings\n", 46 | "import pandas as pd\n", 47 | "from pandas import DataFrame \n", 48 | "import pickle\n", 49 | "import re\n", 50 | "import sys \n", 51 | "import azureml\n", 52 | "import string\n", 53 | "from scipy import stats\n", 54 | "import pip\n", 55 | "import keras\n", 56 | "from keras.preprocessing.sequence import pad_sequences\n", 57 | "from keras.utils.np_utils import to_categorical\n", 58 | "from keras.preprocessing.text import Tokenizer \n", 59 | "from keras.preprocessing import sequence\n", 60 | "from keras.models import Sequential\n", 61 | "from keras.layers import Dense, Embedding\n", 62 | "from keras.layers.core import Dropout\n", 63 | "from keras.layers import LSTM\n", 64 | "from keras.layers import Dense, Input, Flatten \n", 65 | "from keras.layers import Conv1D, MaxPooling1D, Embedding \n", 66 | "from keras.models import Model \n", 67 | "from keras.utils import plot_model\n", 68 | "from keras.utils.vis_utils import model_to_dot\n", 69 | "from keras.models import load_model\n", 70 | "from keras.applications import imagenet_utils\n", 71 | "from keras.layers import Embedding\n", 72 | "from keras.utils.np_utils import to_categorical\n", 73 | "from keras import initializers\n", 74 | "from keras import optimizers\n", 75 | "from keras.callbacks import EarlyStopping\n", 76 | "from keras.layers.normalization import BatchNormalization\n", 77 | "from keras.layers import initializers \n", 78 | "from keras.layers import regularizers \n", 79 | "from keras.layers import constraints \n", 80 | "from keras.layers import Activation\n", 81 | "from keras.layers.advanced_activations import PReLU\n", 82 | "from keras.layers.advanced_activations import LeakyReLU\n", 83 | "from keras.layers.advanced_activations import ELU\n", 84 | "from keras.constraints import max_norm\n", 85 | "import keras.backend as K\n", 86 | "import os\n", 87 | "import tempfile \n", 88 | "import logging\n", 89 | "import gensim\n", 90 | "from gensim.models import Phrases, phrases\n", 91 | "from gensim.models.phrases import Phraser\n", 92 | "from gensim.models import Word2Vec as wv\n", 93 | "import nltk\n", 94 | "from nltk.corpus import stopwords\n", 95 | "from gensim import corpora, models, similarities\n", 96 | "from IPython.display import SVG\n", 97 | "import cloudpickle\n", 98 | "import csv\n", 99 | "import mkl\n", 100 | "import matplotlib.pyplot as plt\n", 101 | "import h5py\n", 102 | "from keras.models import load_model\n", 103 | "import re\n", 104 | "import io\n", 105 | "from os.path import dirname, join\n", 106 | "import regex\n", 107 | "import graphviz\n", 108 | "import pydotplus\n", 109 | "import pyparsing\n", 110 | "from keras.utils import plot_model\n", 111 | "\n", 112 | "\n" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 2, 118 | "metadata": { 119 | "scrolled": true 120 | }, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "Data unpickled\n", 127 | "Review the unique labels [0, 2, 1]\n", 128 | "Categories (3, int64): [0 < 1 < 2]\n", 129 | "(916, 3)\n" 130 | ] 131 | }, 132 | { 133 | "data": { 134 | "text/html": [ 135 | "
\n", 136 | "\n", 149 | "\n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | "
countReturn3Bin_4WeeksCleanText
131729550acad overview we are biopharmaceutical company...
133747412acad company overview we are biopharmaceutical...
\n", 173 | "
" 174 | ], 175 | "text/plain": [ 176 | " count Return3Bin_4Weeks \\\n", 177 | "131 72955 0 \n", 178 | "133 74741 2 \n", 179 | "\n", 180 | " CleanText \n", 181 | "131 acad overview we are biopharmaceutical company... \n", 182 | "133 acad company overview we are biopharmaceutical... " 183 | ] 184 | }, 185 | "execution_count": 2, 186 | "metadata": {}, 187 | "output_type": "execute_result" 188 | } 189 | ], 190 | "source": [ 191 | "\n", 192 | "##########################################\n", 193 | "# Get Previously Organized Stock Data\n", 194 | "##########################################\n", 195 | "\n", 196 | "os.chdir('C:\\\\users\\\\pattyry\\\\documents\\\\AzureML\\\\NextAgenda_CodeStory\\\\NextAgenda_CodeStory')\n", 197 | "\n", 198 | "with open('biotechcleaned.pkl', 'rb') as f:\n", 199 | " data = pickle.load(f, encoding='utf8')\n", 200 | " print(\"Data unpickled\")\n", 201 | "data = pd.DataFrame(data)\n", 202 | "thedata = data\n", 203 | "\n", 204 | "np.random.seed(1337) # for reproducibility\n", 205 | "#################################\n", 206 | "#If necessary, convert categories\n", 207 | "#################################\n", 208 | "#thedata['ReturnQuantile'] = thedata['ReturnQuantile'].map({0:0,1:1,2:1,3:1,4:2})\n", 209 | "print('Review the unique labels',thedata['Return3Bin_4Weeks'].unique())\n", 210 | "\n", 211 | "##########################################\n", 212 | "# clean up the text in the data with regex\n", 213 | "##########################################\n", 214 | "#Most clean up already done in pre-processing script in a jupyter notebook.\n", 215 | "thedata['fulltext'] = thedata['fulltext'].str.encode('utf-8')\n", 216 | "thedata['fulltext'] = thedata['fulltext'].str.lower()\n", 217 | "\n", 218 | "def clean_text(row):\n", 219 | " text = str(row['fulltext'])\n", 220 | "\n", 221 | " # Remove newline characters\n", 222 | " cleantext = text.replace('\\r\\n', ' ')\n", 223 | "\n", 224 | " # Convert HTML punctuation chaaracters\n", 225 | " cleantext = cleantext.replace('.', '')\n", 226 | "\n", 227 | " #remove non alpha characters and specific noise\n", 228 | " #cleantext = re.sub(r'\\d+', '',cleantext)\n", 229 | " cleantext = re.sub(r'^b','',cleantext)\n", 230 | " cleantext = re.sub(r'[^\\w]',' ',cleantext)\n", 231 | "\n", 232 | " #remove specific noise\n", 233 | " cleantext = cleantext.translate(str.maketrans({'‘':' ','’':' '}))\n", 234 | " cleantext = cleantext.translate(str.maketrans({',':' ',',':' '}))\n", 235 | " cleantext = cleantext.translate(str.maketrans({'\"':' ','%':' '}))\n", 236 | "\n", 237 | " #remove punctuation\n", 238 | " punctpattern = re.compile('[%s]' % re.escape(string.punctuation))\n", 239 | " cleanttext = re.sub(punctpattern,'', cleantext)\n", 240 | "\n", 241 | " #remove single letter word\n", 242 | " cleantext = re.sub('(\\\\b[A-Za-z] \\\\b|\\\\b [A-Za-z]\\\\b)', '', cleantext) \n", 243 | "\n", 244 | " # Remove extra spaces\n", 245 | " cleantext = re.sub('\\s+', ' ', cleantext).strip()\n", 246 | "\n", 247 | " return cleantext\n", 248 | "\n", 249 | "#apply regex fixes to the input text column\n", 250 | "thedata['CleanText'] = thedata.apply(clean_text, axis=1)\n", 251 | "justcleandocs=thedata.drop(['fulltext'], axis=1)\n", 252 | "\n", 253 | "#save a cleaned copy to inspect\n", 254 | "justcleandocs.to_csv('C:\\\\glove\\cleaneddata2.tsv', sep='\\t', encoding='utf-8')\n", 255 | "print(justcleandocs.shape)\n", 256 | "justcleandocs.head(2)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": 3, 262 | "metadata": { 263 | "scrolled": true 264 | }, 265 | "outputs": [ 266 | { 267 | "name": "stdout", 268 | "output_type": "stream", 269 | "text": [ 270 | "post regex justcleandocs 131 acad overview we are biopharmaceutical company...\n", 271 | "133 acad company overview we are biopharmaceutical...\n", 272 | "134 acad company overview we are biopharmaceutical...\n", 273 | "135 acad company overview we are biopharmaceutical...\n", 274 | "136 acad overview we are biopharmaceutical company...\n", 275 | "137 acad overview we are biopharmaceutical company...\n", 276 | "138 acad overview we are biopharmaceutical company...\n", 277 | "139 acad overview we are biopharmaceutical company...\n", 278 | "140 acad overview we are biopharmaceutical company...\n", 279 | "181 achn overview we are biopharmaceutical company...\n", 280 | "Name: CleanText, dtype: object\n", 281 | "head of just labels Return3Bin_4Weeks\n", 282 | "131 0\n", 283 | "133 2\n", 284 | "134 0\n", 285 | "135 0\n", 286 | "136 2\n", 287 | " Return3Bin_4Weeks\n", 288 | "131 0\n", 289 | "133 2\n", 290 | "134 0\n", 291 | "135 0\n", 292 | "136 2\n", 293 | " Return3Bin_4Weeks\n", 294 | "65850 0\n", 295 | "65851 1\n", 296 | "65852 0\n", 297 | "65853 0\n", 298 | "65854 2\n", 299 | "[0, 2, 1]\n", 300 | "Categories (3, int64): [0 < 1 < 2]\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "\n", 306 | "\n", 307 | "################################\n", 308 | "# Convert labels to categorical\n", 309 | "################################\n", 310 | "\n", 311 | "justcleandocs=thedata.drop(['fulltext','Return3Bin_4Weeks'], axis=1)\n", 312 | "justcleandocs = justcleandocs['CleanText']\n", 313 | "print('post regex justcleandocs',justcleandocs.head(10))\n", 314 | "\n", 315 | "justlabels=thedata.drop(['fulltext','CleanText'], axis=1)\n", 316 | "justlabels=pd.DataFrame(justlabels['Return3Bin_4Weeks'])\n", 317 | "print('head of just labels',justlabels.head(5))\n", 318 | "print(justlabels.head())\n", 319 | "print(justlabels.tail())\n", 320 | "print(justlabels['Return3Bin_4Weeks'].unique())\n", 321 | "\n", 322 | "\n", 323 | "###################################################\n", 324 | "# Set Global Vars\n", 325 | "####################################################\n", 326 | "\n", 327 | "MAX_SEQUENCE_LENGTH = 10000\n", 328 | "MAX_NB_WORDS = 400000\n", 329 | "EMBEDDING_DIM = 300\n", 330 | "VALIDATION_SPLIT = 0.2\n", 331 | "LEARNING_RATE = .00011\n", 332 | "BATCH_SIZE = 32\n", 333 | "DROPOUT_RATE = 0.45\n", 334 | "INNERLAYER_DROPOUT_RATE = 0.15\n", 335 | "np.random.seed(2032)\n", 336 | "\n", 337 | "#change directory to write results\n", 338 | "os.chdir('C:\\\\')\n", 339 | "BASE_DIR = '.'\n", 340 | "GLOVE_DIR = BASE_DIR + '/glove/'\n", 341 | "\n" 342 | ] 343 | }, 344 | { 345 | "cell_type": "code", 346 | "execution_count": 4, 347 | "metadata": { 348 | "scrolled": false 349 | }, 350 | "outputs": [ 351 | { 352 | "name": "stdout", 353 | "output_type": "stream", 354 | "text": [ 355 | "labels_index {0: 0, 1: 1, 2: 2}\n", 356 | "Found 23887 unique tokens\n", 357 | "Shape of data tensor: (916, 10000)\n", 358 | "Shape of label tensor: (916, 3)\n", 359 | "length of y_val 183\n", 360 | "shape of y_val (183, 3)\n", 361 | "length of X_val 183\n", 362 | "shape of X_val (183, 10000)\n", 363 | "test and training sets saved to disk for later evaluation\n" 364 | ] 365 | } 366 | ], 367 | "source": [ 368 | "\n", 369 | "######################################################\n", 370 | "# Format our text samples and labels for use in Keras\n", 371 | "######################################################\n", 372 | "# Then we can format our text samples and labels into tensors that can be fed into a neural network. \n", 373 | "# Here we tokenize our source 'justcleandocs'\n", 374 | "# note that the values here are ultimately indexes to the actual words\n", 375 | "\n", 376 | "#convert text format\n", 377 | "justcleandocslist = justcleandocs.values\n", 378 | "justcleandocslist[6]\n", 379 | "labels = justlabels.values\n", 380 | "labels_index = {}\n", 381 | "#labels_index = {0:0,1:1,2:2,3:3,4:4}\n", 382 | "labels_index = {0:0,1:1,2:2}\n", 383 | "print('labels_index', labels_index)\n", 384 | "\n", 385 | "#tokenize the text\n", 386 | "tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n", 387 | "tokenizer.fit_on_texts(justcleandocslist) #tokenizer.fit_on_texts(texts)\n", 388 | "sequences = tokenizer.texts_to_sequences(justcleandocslist) #sequences = tokenizer.texts_to_sequences(texts)\n", 389 | "word_index = tokenizer.word_index #word_index = tokenizer.word_index\n", 390 | "print('Found {} unique tokens'.format(len(word_index)))\n", 391 | "#print('sequences first', sequences[0])\n", 392 | "\n", 393 | "#Pad sequences so that they all have the same length in a batch of input data \n", 394 | "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH, padding='pre', truncating='pre')\n", 395 | "sequences = None\n", 396 | "texts = None\n", 397 | "\n", 398 | "\n", 399 | "##################################################\n", 400 | "#build label array from target y label in data set\n", 401 | "##################################################\n", 402 | "labels = to_categorical(np.asarray(labels))\n", 403 | "print('Shape of data tensor: ', data.shape)\n", 404 | "print('Shape of label tensor: ', labels.shape)\n", 405 | "\n", 406 | "indices = np.arange(data.shape[0])\n", 407 | "np.random.shuffle(indices)\n", 408 | "data = data[indices]\n", 409 | "labels = labels[indices]\n", 410 | "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n", 411 | "\n", 412 | "X_train = data[:-nb_validation_samples]\n", 413 | "y_train = labels[:-nb_validation_samples]\n", 414 | "X_val = data[-nb_validation_samples:]\n", 415 | "y_val = labels[-nb_validation_samples:]\n", 416 | "\n", 417 | "print('length of y_val',len(y_val))\n", 418 | "print('shape of y_val',y_val.shape)\n", 419 | "print('length of X_val',len(X_val))\n", 420 | "print('shape of X_val',X_val.shape)\n", 421 | "\n", 422 | "os.chdir('C:\\\\glove\\\\nextagenda')\n", 423 | "\n", 424 | "#from itertools import islice\n", 425 | "#head = list(islice(y_val, 6))\n", 426 | "#print('head of yval',head)\n", 427 | "\n", 428 | "\n", 429 | "\n", 430 | "#####################################\n", 431 | "# Save Validation Set for Evaluation\n", 432 | "####################################\n", 433 | "np.savetxt('y_val_3bin.txt', y_val, delimiter=',')\n", 434 | "np.savetxt('X_val_3bin.txt', X_val, fmt='%s', delimiter=',')\n", 435 | "print('test and training sets saved to disk for later evaluation')" 436 | ] 437 | }, 438 | { 439 | "cell_type": "code", 440 | "execution_count": 5, 441 | "metadata": { 442 | "scrolled": true 443 | }, 444 | "outputs": [ 445 | { 446 | "name": "stdout", 447 | "output_type": "stream", 448 | "text": [ 449 | "Loading word vectors to prepare the embedding layer...\n", 450 | "C:\\glove\\nextagenda\n", 451 | "Loading Glove Model...\n", 452 | " 0 1 2 3 4 5 6 7 \\\n", 453 | "0 the 0.046560 0.213180 -0.007436 -0.458540 -0.035639 0.236430 -0.288360 \n", 454 | "1 , -0.255390 -0.257230 0.131690 -0.042688 0.218170 -0.022702 -0.178540 \n", 455 | "2 . -0.125590 0.013630 0.103060 -0.101230 0.098128 0.136270 -0.107210 \n", 456 | "3 of -0.076947 -0.021211 0.212710 -0.722320 -0.139880 -0.122340 -0.175210 \n", 457 | "4 to -0.257560 -0.057132 -0.671900 -0.380820 -0.364210 -0.082155 -0.010955 \n", 458 | "\n", 459 | " 8 9 ... 291 292 293 294 \\\n", 460 | "0 0.215210 -0.134860 ... -0.013064 -0.296860 -0.079913 0.195000 \n", 461 | "1 0.107560 0.058936 ... 0.075968 -0.014359 -0.073794 0.221760 \n", 462 | "2 0.236970 0.328700 ... 0.060148 -0.156190 -0.119490 0.234450 \n", 463 | "3 0.121370 -0.070866 ... -0.366730 -0.386030 0.302900 0.015747 \n", 464 | "4 -0.082047 0.460560 ... -0.012806 -0.597070 0.317340 -0.252670 \n", 465 | "\n", 466 | " 295 296 297 298 299 300 \n", 467 | "0 0.031549 0.285060 -0.087461 0.009061 -0.209890 0.053913 \n", 468 | "1 0.146520 0.566860 0.053307 -0.232900 -0.122260 0.354990 \n", 469 | "2 0.081367 0.246180 -0.152420 -0.342240 -0.022394 0.136840 \n", 470 | "3 0.340360 0.478410 0.068617 0.183510 -0.291830 -0.046533 \n", 471 | "4 0.543840 0.063007 -0.049795 -0.160430 0.046744 -0.070621 \n", 472 | "\n", 473 | "[5 rows x 301 columns]\n", 474 | "shape of glove model (400000, 301)\n", 475 | "wordkeys type of file \n", 476 | "Found 400000 word vectors.\n", 477 | "Building Embedding Matrix...\n" 478 | ] 479 | } 480 | ], 481 | "source": [ 482 | "\n", 483 | "########################################\n", 484 | "# Preparing the embedding layer\n", 485 | "########################################\n", 486 | "\n", 487 | "#load in word vectors from glove reference global English data set\n", 488 | "# https://nlp.stanford.edu/projects/glove/\n", 489 | "# see more reference links at bottom\n", 490 | "\n", 491 | "print('Loading word vectors to prepare the embedding layer...')\n", 492 | "print(os.getcwd())\n", 493 | "\n", 494 | "embeddings_index = {}\n", 495 | "print('Loading Glove Model...')\n", 496 | "gloveFile = 'C:\\\\glove\\\\glove6B300d.txt'\n", 497 | "words = pd.read_table(gloveFile, sep=\" \", header=None, quoting=csv.QUOTE_NONE)\n", 498 | "\n", 499 | "print(words.head(5))\n", 500 | "print('shape of glove model',words.shape)\n", 501 | "\n", 502 | "wordkeys=words.iloc[:,0]\n", 503 | "print('wordkeys type of file', type(wordkeys))\n", 504 | "words2 = words.rename(columns={ words.columns[0]: \"words\" })\n", 505 | "words2['words'].apply(str)\n", 506 | "#print(words2.dtypes)\n", 507 | "\n", 508 | "embeddings_index = words2.set_index('words').T.to_dict('list')\n", 509 | "\n", 510 | "#print(dict(list(embeddings_index.items())[0:2]))\n", 511 | "print('Found {} word vectors.'.format(len(embeddings_index)))\n", 512 | "#usage of pandas function dataFrame.to_dict(outtype='dict') outtype : str {‘dict’, ‘list’, ‘series’}\n", 513 | "\n", 514 | "\n", 515 | "#################################\n", 516 | "#Build the embedding matrix\n", 517 | "#################################\n", 518 | "\n", 519 | "print('Building Embedding Matrix...')\n", 520 | "embedding_matrix = np.zeros((len(word_index) + 1, EMBEDDING_DIM))\n", 521 | "for word, i in word_index.items():\n", 522 | " embedding_vector = embeddings_index.get(word)\n", 523 | " if embedding_vector is not None:\n", 524 | " # words not found in embedding index will be all-zeros.\n", 525 | " embedding_matrix[i] = embedding_vector\n", 526 | "\n", 527 | "embedding_layer = Embedding(len(word_index) + 1,\n", 528 | " EMBEDDING_DIM,\n", 529 | " weights=[embedding_matrix],\n", 530 | " input_length=MAX_SEQUENCE_LENGTH,\n", 531 | " trainable=False)\n" 532 | ] 533 | }, 534 | { 535 | "cell_type": "code", 536 | "execution_count": 6, 537 | "metadata": { 538 | "scrolled": true 539 | }, 540 | "outputs": [ 541 | { 542 | "name": "stdout", 543 | "output_type": "stream", 544 | "text": [ 545 | "Train 1D Convnet with global maxpooling\n", 546 | "Shape of training data sample tensor: (733, 10000)\n", 547 | "Shape of training label tensor: (733, 3)\n" 548 | ] 549 | } 550 | ], 551 | "source": [ 552 | "\n", 553 | "##############################################\n", 554 | "#Training a 1D convnet\n", 555 | "##############################################\n", 556 | "\n", 557 | "print('Train 1D Convnet with global maxpooling')\n", 558 | "print('Shape of training data sample tensor: ', X_train.shape)\n", 559 | "print('Shape of training label tensor: ', y_train.shape)\n", 560 | "\n", 561 | "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n", 562 | "\n", 563 | "\n", 564 | "embedded_sequences = embedding_layer(sequence_input)\n", 565 | "\n", 566 | "x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(embedded_sequences)\n", 567 | "x = MaxPooling1D(5)(x)\n", 568 | "x = Dropout(INNERLAYER_DROPOUT_RATE)(x)\n", 569 | "\n", 570 | "x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x)\n", 571 | "x = MaxPooling1D(5)(x)\n", 572 | "x = Dropout(INNERLAYER_DROPOUT_RATE)(x)\n", 573 | "\n", 574 | "x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x)\n", 575 | "x = MaxPooling1D(35)(x) # global max pooling\n", 576 | "\n", 577 | "x = Flatten()(x)\n", 578 | "x = Dense(100, activation='elu', kernel_initializer='lecun_uniform')(x) # best initializers: #glorot_normal #VarianceScaling #lecun_uniform\n", 579 | "x = Dropout(DROPOUT_RATE)(x)\n", 580 | "\n", 581 | "preds = Dense(len(labels_index), activation='softmax')(x) #no initialization in output layer\n", 582 | "\n", 583 | "\n", 584 | "model = Model(sequence_input, preds)\n" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": 7, 590 | "metadata": { 591 | "scrolled": true 592 | }, 593 | "outputs": [ 594 | { 595 | "name": "stdout", 596 | "output_type": "stream", 597 | "text": [ 598 | "Train on 733 samples, validate on 183 samples\n", 599 | "Epoch 1/24\n", 600 | "733/733 [==============================] - 330s - loss: 1.2609 - acc: 0.3302 - val_loss: 1.1046 - val_acc: 0.3497\n", 601 | "Epoch 2/24\n", 602 | "733/733 [==============================] - 293s - loss: 1.1536 - acc: 0.3834 - val_loss: 1.0895 - val_acc: 0.4153\n", 603 | "Epoch 3/24\n", 604 | "733/733 [==============================] - 288s - loss: 1.1175 - acc: 0.3874 - val_loss: 1.0802 - val_acc: 0.3607\n", 605 | "Epoch 4/24\n", 606 | "733/733 [==============================] - 324s - loss: 1.0857 - acc: 0.4120 - val_loss: 1.0744 - val_acc: 0.3825\n", 607 | "Epoch 5/24\n", 608 | "733/733 [==============================] - 331s - loss: 1.0675 - acc: 0.4256 - val_loss: 1.0710 - val_acc: 0.3934\n", 609 | "Epoch 6/24\n", 610 | "733/733 [==============================] - 338s - loss: 1.0693 - acc: 0.4338 - val_loss: 1.0857 - val_acc: 0.4153\n", 611 | "Epoch 7/24\n", 612 | "733/733 [==============================] - 331s - loss: 1.0571 - acc: 0.4447 - val_loss: 1.0910 - val_acc: 0.3825\n", 613 | "Epoch 8/24\n", 614 | "733/733 [==============================] - 320s - loss: 1.0301 - acc: 0.4666 - val_loss: 1.0686 - val_acc: 0.3825\n", 615 | "Epoch 9/24\n", 616 | "733/733 [==============================] - 314s - loss: 1.0330 - acc: 0.4638 - val_loss: 1.1050 - val_acc: 0.3552\n", 617 | "Epoch 10/24\n", 618 | "733/733 [==============================] - 313s - loss: 0.9989 - acc: 0.4980 - val_loss: 1.0557 - val_acc: 0.4153\n", 619 | "Epoch 11/24\n", 620 | "733/733 [==============================] - 334s - loss: 0.9991 - acc: 0.4925 - val_loss: 1.0921 - val_acc: 0.4153\n", 621 | "Epoch 12/24\n", 622 | "733/733 [==============================] - 326s - loss: 0.9626 - acc: 0.5321 - val_loss: 1.0480 - val_acc: 0.3880\n", 623 | "Epoch 13/24\n", 624 | "733/733 [==============================] - 327s - loss: 0.9396 - acc: 0.5375 - val_loss: 1.0565 - val_acc: 0.3825\n", 625 | "Epoch 14/24\n", 626 | "733/733 [==============================] - 371s - loss: 0.9308 - acc: 0.5321 - val_loss: 1.0501 - val_acc: 0.4536\n", 627 | "Epoch 15/24\n", 628 | "733/733 [==============================] - 342s - loss: 0.9116 - acc: 0.5498 - val_loss: 1.0406 - val_acc: 0.4536\n", 629 | "Epoch 16/24\n", 630 | "733/733 [==============================] - 304s - loss: 0.9010 - acc: 0.5498 - val_loss: 1.0208 - val_acc: 0.4481\n", 631 | "Epoch 17/24\n", 632 | "733/733 [==============================] - 316s - loss: 0.8857 - acc: 0.5607 - val_loss: 1.0177 - val_acc: 0.4372\n", 633 | "Epoch 18/24\n", 634 | "733/733 [==============================] - 313s - loss: 0.8485 - acc: 0.5948 - val_loss: 1.0381 - val_acc: 0.4262\n", 635 | "Epoch 19/24\n", 636 | "733/733 [==============================] - 316s - loss: 0.8242 - acc: 0.6139 - val_loss: 1.0133 - val_acc: 0.4645\n", 637 | "Epoch 20/24\n", 638 | "733/733 [==============================] - 310s - loss: 0.8399 - acc: 0.6085 - val_loss: 1.0197 - val_acc: 0.4590\n", 639 | "Epoch 21/24\n", 640 | "733/733 [==============================] - 311s - loss: 0.7924 - acc: 0.6235 - val_loss: 1.0351 - val_acc: 0.4536\n", 641 | "Epoch 22/24\n", 642 | "733/733 [==============================] - 311s - loss: 0.7821 - acc: 0.6289 - val_loss: 1.0552 - val_acc: 0.4426\n", 643 | "Epoch 23/24\n", 644 | "733/733 [==============================] - 310s - loss: 0.7717 - acc: 0.6453 - val_loss: 1.0199 - val_acc: 0.4645\n", 645 | "Epoch 24/24\n", 646 | "733/733 [==============================] - 309s - loss: 0.7434 - acc: 0.6535 - val_loss: 1.0339 - val_acc: 0.5027\n" 647 | ] 648 | }, 649 | { 650 | "data": { 651 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsnXd4XMXVh9+jVe/NklUsW+6944IN\n2IDBNr05QGihGAgECIEAyQcEEhLSCBAI3UBCKA692xhsDLjbuPeuLlmyulbS7s73x1xJa1u2V9au\nVmXe57mP7r1zy7nX6/ndOWfmjCilMBgMBoMBIMDfBhgMBoOh/WBEwWAwGAyNGFEwGAwGQyNGFAwG\ng8HQiBEFg8FgMDRiRMFgMBgMjRhRMHQpROQ1EfmDh8fuFZEzfW2TwdCeMKJgMBgMhkaMKBgMHRAR\nCfS3DYbOiREFQ7vDctvcKyLrRaRKRF4RkWQR+UJEKkRkgYjEuR1/vohsEpFSEVkkIoPcykaJyBrr\nvHeA0MPuda6IrLXOXSIiwz208RwR+VFEykUkS0R+d1j5ZOt6pVb5ddb+MBH5u4jsE5EyEfne2jdF\nRLKbeQ9nWuu/E5F3ReQNESkHrhORcSKy1LpHnog8IyLBbucPEZGvRKRERApE5Dci0l1EqkUkwe24\nMSJSJCJBnjy7oXNjRMHQXrkEmAb0B84DvgB+AySif7d3AIhIf+At4C6gG/A58ImIBFsV5IfAf4B4\n4H/WdbHOHQ3MAW4GEoAXgI9FJMQD+6qAa4BY4BzgVhG50LpuhmXvPy2bRgJrrfP+BowBTrZs+jXg\n8vCdXAC8a93zv4AT+KX1TiYCZwA/t2yIAhYAXwKpQF/ga6VUPrAImOV23auAt5VS9R7aYejEGFEw\ntFf+qZQqUErlAN8By5VSPyqlaoEPgFHWcT8BPlNKfWVVan8DwtCV7gQgCHhSKVWvlHoXWOl2j5uA\nF5RSy5VSTqXU60Ctdd4xUUotUkptUEq5lFLr0cJ0mlX8U2CBUuot677FSqm1IhIAXA/cqZTKse65\nxHomT1iqlPrQumeNUmq1UmqZUsqhlNqLFrUGG84F8pVSf1dK2ZVSFUqp5VbZ62ghQERswBVo4TQY\njCgY2i0Fbus1zWxHWuupwL6GAqWUC8gC0qyyHHVo1sd9bus9gV9Z7pdSESkFeljnHRMRGS8iCy23\nSxlwC/qLHesau5o5LRHtvmquzBOyDrOhv4h8KiL5lkvpjx7YAPARMFhEeqNbY2VKqRUnaJOhk2FE\nwdDRyUVX7gCIiKArxBwgD0iz9jWQ4baeBTymlIp1W8KVUm95cN83gY+BHkqpGOB5oOE+WUCfZs45\nANiPUlYFhLs9hw3tenLn8JTGzwFbgX5KqWi0e+14NqCUsgNz0S2aqzGtBIMbRhQMHZ25wDkicoYV\nKP0V2gW0BFgKOIA7RCRQRC4Gxrmd+xJwi/XVLyISYQWQozy4bxRQopSyi8g44Eq3sv8CZ4rILOu+\nCSIy0mrFzAGeEJFUEbGJyEQrhrEdCLXuHwT8H3C82EYUUA5UishA4Fa3sk+B7iJyl4iEiEiUiIx3\nK/83cB1wPvCGB89r6CIYUTB0aJRS29D+8X+iv8TPA85TStUppeqAi9GV30F0/OF9t3NXoeMKz1jl\nO61jPeHnwKMiUgE8hBanhuvuB2aiBaoEHWQeYRXfA2xAxzZKgD8DAUqpMuuaL6NbOVXAIb2RmuEe\ntBhVoAXuHTcbKtCuofOAfGAHMNWt/Ad0gHuNFY8wGAAQM8mOwdA1EZFvgDeVUi/72xZD+8GIgsHQ\nBRGRk4Cv0DGRCn/bY2g/GPeRwdDFEJHX0WMY7jKCYDgc01IwGAwGQyOmpWAwGAyGRjpcUq3ExETV\nq1cvf5thMBgMHYrVq1cfUEodPvblCDqcKPTq1YtVq1b52wyDwWDoUIjIvuMfZdxHBoPBYHDDiILB\nYDB0ENqiY1CHcx8ZDAZDZ6fO4WLPgSq2FVSwPb9C/y2oYNbYHtw2ta9P790pRKG+vp7s7Gzsdru/\nTfE5oaGhpKenExRk5kMxGDo6Tpdif0k12/J1pd8gAnsOVOFw6VaBLUDonRjB0LQY+nSL8LlNnUIU\nsrOziYqKolevXhyaELNzoZSiuLiY7OxsMjMz/W2OwWBoIbUOJ99uK+KrzQVsyS9nR0EltY6mOZYy\n4sPpnxzFWUOS6Z8cxYDuUWQmRhASaGszGzuFKNjt9k4vCAAiQkJCAkVFRf42xWAweEi908WSXcV8\nsi6XeZvyqbA7iA0PYlhaDFdP6En/7lEMSI6ib1IkESH+r5L9b4GX6OyC0EBXeU6DoSPjcilW7C3h\nk3W5fLExn5KqOqJCAjl7aHfOG5HKyX0SCLK1z34+nUYUDAaDwZ8opVibVcon6/L4bEMuBeW1hAXZ\nOHNwMucNT+HU/t0IDWo7N9CJYkTBC5SWlvLmm2/y85//vEXnzZw5kzfffJPY2FgfWWYwGHyJy6XY\nkl/Op+vz+GRdLtkHawi2BTBlQDfOG5HKGYOSCA/uWNVsx7K2nVJaWsq//vWvI0TB6XRisx39y+Dz\nzz/3tWkGg+EEUUpRVlNPbqmd3NIa8spqyC2z1kvt5JbVkF9mx+FS2AKEyX0TuevM/kwbnExMWMft\nHWhEwQvcf//97Nq1i5EjRxIUFERkZCQpKSmsXbuWzZs3c+GFF5KVlYXdbufOO+9k9uzZQFPKjsrK\nSmbMmMHkyZNZsmQJaWlpfPTRR4SFhfn5yQyGrkF+mZ2vNuezIaeMPKvizy21U1PvPOS4IJvQPSaU\nlJgwxvaMIzU2jF6JEZwxMImEyOPNntox6HSi8Mgnm9icW+7Vaw5Ojebh84Yctfzxxx9n48aNrF27\nlkWLFnHOOeewcePGxm6jc+bMIT4+npqaGk466SQuueQSEhISDrnGjh07eOutt3jppZeYNWsW7733\nHldddZVXn8NgMDSxq6iSeZvymbepgHVZpQAkRoaQFhfGgO5RTBmQREpMKGmxYaTEhpEaE0piZAgB\nAZ27s0enE4X2wLhx4w4ZR/D000/zwQcfAJCVlcWOHTuOEIXMzExGjhwJwJgxY9i7d2+b2WswdAWU\nUqzPLrOEIJ9dRVUADE+P4d6zB3D2kGT6JkX52Ur/0+lE4Vhf9G1FRETTqMNFixaxYMECli5dSnh4\nOFOmTGl25HVISFPT02azUVNT0ya2GgztlaySat5ZmcXnG/IIDbKRGqvdNqmxYaTGhpIaG0ZKTCjJ\n0aFH7d5Z73SxYk8J8zblM39TAfnldmwBwvjMeK6Z2Itpg5NJjTVuWnc6nSj4g6ioKCoqmp/VsKys\njLi4OMLDw9m6dSvLli1rY+sMho5DvdPF11sKeWvFfhbvKEKASX0TCbIFkH2whpV7D1JWU3/IOQEC\nSVGhpMSGkhqjBSM5OpTNeeV8vaWQspp6QoMCOLVfN+4dMoAzBiURGx7snwfsAPhUFERkOvAUYANe\nVko93swxs4DfAQpYp5S60pc2+YKEhAQmTZrE0KFDCQsLIzk5ubFs+vTpPP/88wwfPpwBAwYwYcIE\nP1pqMLRPsg/qVsE7K7MorKile3Qod5zej1kn9SDtsC/5qloHeWU15JTaySt16xFUVsOWvHIWbCmg\n1uEiJiyIMwYlcfaQ7pzarxthwe1/jEB7wGdzNIuIDdgOTAOygZXAFUqpzW7H9APmAqcrpQ6KSJJS\nqvBY1x07dqw6fJKdLVu2MGjQIG8/Qrulqz2voXPicLr4Zmshb67Yz7fbdeqWqQOSuGJcBlMHdCPw\nBEf8KqUora4nKjTwhK/RGRGR1Uqpscc7zpcthXHATqXUbsugt4ELgM1ux9wEPKuUOghwPEEwGAwd\nn+yD1cxdmcU7q7IoKK8lOTqEX0zty0/GZRzRKjgRRIS4COMeOlF8KQppQJbbdjYw/rBj+gOIyA9o\nF9PvlFJfHn4hEZkNzAbIyMjwibEGg8G3bMuv4PEvtrDIahWc1r8bv78gg9MHJpkv+naEL0Whuc68\nh/uqAoF+wBQgHfhORIYqpUoPOUmpF4EXQbuPvG+qwWDwFUop/r10H499voXIkEBun9qXWWN70CM+\n3N+mGZrBl6KQDfRw204Hcps5ZplSqh7YIyLb0CKx0od2GQyGNuJAZS2/fnc932wtZOqAbvz1shEk\ndpKRv50VX4rCSqCfiGQCOcDlwOE9iz4ErgBeE5FEtDtptw9tMhgMbcSibYXc87/1lNvreeT8IVwz\nsadJ/d4B8JkoKKUcInI7MA8dL5ijlNokIo8Cq5RSH1tlZ4nIZsAJ3KuUKvaVTQaDwffY65385ctt\nzPlhDwOSo3jjxnEM7B7tb7MMHuLT6I5S6nOlVH+lVB+l1GPWvocsQUBp7lZKDVZKDVNKve1Le3xF\nQ5bUE+HJJ5+kurrayxYZDP5hR0EFFz77A3N+2MO1E3vy0e2TjCB0MEzI3wsYUTB0dZRS/GfZPs79\n5/cUVdQy57qxPHLB0A4xqYzhUEyaCy/gnjp72rRpJCUlMXfuXGpra7nooot45JFHqKqqYtasWWRn\nZ+N0OnnwwQcpKCggNzeXqVOnkpiYyMKFC/39KAZDiympquPX765nwZYCTuvfjb9eNpykqFB/m2U4\nQTqfKHxxP+Rv8O41uw+DGUdk6GjEPXX2/Pnzeffdd1mxYgVKKc4//3wWL15MUVERqampfPbZZ4DO\niRQTE8MTTzzBwoULSUxM9K7NBkMb8N2OIu6eu46y6noePHcwPzu5V6dPLd3ZMe4jLzN//nzmz5/P\nqFGjGD16NFu3bmXHjh0MGzaMBQsWcN999/Hdd98RExPjb1MNhhPGXu/ksc82c/UrK4gNC+LD2yZx\nw+RMIwidAI9aCiLyHjAH+EIp5fKtSa3kGF/0bYFSigceeICbb775iLLVq1fz+eef88ADD3DWWWfx\n0EMP+cFCg6HlFFfWsmZ/Kav2lbB670HW55RR53Bx1YQMfjtzsEk214nw1H30HPAz4GkR+R/wmlJq\nq+/M6li4p84+++yzefDBB/npT39KZGQkOTk5BAUF4XA4iI+P56qrriIyMpLXXnvtkHON+8jQXnC5\nFLsPVLJq70FW79PL7gN6QpogmzA0LYZrJvTkjEHJTOyTcJyrGToaHomCUmoBsEBEYtCDzb4SkSzg\nJeANa0Ryl8U9dfaMGTO48sormThxIgCRkZG88cYb7Ny5k3vvvZeAgACCgoJ47rnnAJg9ezYzZswg\nJSXFBJoNraaq1nHEvMLHQynYXVTJKksA1uw/SGm1/i8dFx7EmJ5xXDo2nbE94xmeHmN6FHVyPE6d\nLSIJwFXA1eh0Ff8FJgPDlFJTfGXg4ZjU2V3veQ3Hptbh5Osthby3OptF24twuk48PVjvbhGM7RnH\n2J7xjO4ZR59uEWYUcifBq6mzReR9YCDwH+A8pVSeVfSOiKw6+pkGg8EXKKVYl13Ge6uz+XhdLmU1\n9SRHh3Dj5EzS41qefrp7TBhjesYRb1JOd3k8jSk8o5T6prkCT5THYDB4h/wyOx/8mMN7a7LZWVhJ\nSGAAZw/pziVj0pncNxGb6f1jaCWeisIgEVnTkNJaROLQs6id2DBeH6CU6hLNXF/NlGfwDvZ6J/ll\ndnLLash1my6yps5BckwoabFhpMToCefTYsOIDQ867u/WXu9k3qZ83luTw/c7inApGNszjj9dPIxz\nhqcQHRrURk9n6Ap4Kgo3KaWebdiwps68CWgXohAaGkpxcTEJCQmdWhiUUhQXFxMaakaL+ouqWgdb\n8ysa5wTOLW2YH9hOXlkNByrrjjgnISKYsGAbBeV26p2HinpoUIA12bwWipTYMNJiQ0mJCSMwQPhk\nfS6frsujotZBWmwYt03ty8Wj08lMjGirRzZ0MTwVhQAREWV9plrzL7cb52N6ejrZ2dkUFRX52xSf\nExoaSnp6ur/N6JJszCnjxtdXkV9ub9wXGRJIqlWJD02LJjUmjJTYMFJjQkmNDaN7TGhjbx2XS3Gg\nqpY8S0hyyxpaElpcFu8oorCiFvfGYFiQjRnDunPp6HQm9E4wg8MMPsdTUZgHzBWR59Gzp90CHDFt\npr8ICgoiMzPT32YYOjGfb8jj7rlriQ8P5vmrRpOZGElKbGiLXDcBAUJSVChJUaGM6BHb7DF1DhcF\n5XbyyuxU2OsZ3zuByJDOl43G0H7x9Nd2H3AzcCt6ms35wMu+MspgaC8opXj66538Y8F2RmfE8sLV\nY+kW5buZw4IDA+gRH26mqjT4DU8Hr7nQo5qf8605BkP7oabOyT3vruOz9XlcPDqNP108jJBAM3DL\n0LnxdJxCP+BPwGCgMcqplOrtI7sMBr+SX2Zn9n9WsSGnjPtnDOTmU3t36k4MBkMDnrqPXgUeBv4B\nTEXnQTL/QwydknVZpdz071VU1Tp46eqxnDk42d8mGQxthqeps8OUUl+j02LsU0r9Djj9eCeJyHQR\n2SYiO0Xk/mbKrxORIhFZay03tsx8g8G7fLIul1kvLCU4MID3fn6yEQRDl8PTloJdRAKAHSJyO5AD\nJB3rBKvb6rPANCAbWCkiHyulNh926DtKqdtbaLfB4FVcLsWTC7bz9Dc7GdcrnueuGk1CpO8CygZD\ni6ksgrA4sPm2N5qnLYW7gHDgDmAMOjHetcc5ZxywUym1WylVB7wNXHCihhoMvqK6zsFtb67h6W92\nMmtsOm/cON4IgqF9kbUSnp8MC//g81sdV3KsL/5ZSql7gUp0PMET0oAst+1sYHwzx10iIqcC24Ff\nKqWymjnGYDgqOwoqeHHxbiJCAo8YFZwUFUKg7ejfPnllNdz4+iq25JXzf+cM4obJmSagbGhfrPk3\nfPYriE6FYZf5/HbHFQWllFNExriPaPaQ5v5nHX7+J8BbSqlaEbkFeJ1mYhUiMhuYDZCRkdECEwyd\nGaUUbyzfzx8+3UxggBAgQkWt45BjbAFCclQIKbFN+YYahEOA3364kZo6J69cexJTBx7TI2owtC2O\nOpj3AKx8GfqcDpe8AuHxPr+tp86pH4GPrFnXqhp2KqXeP8Y52UAPt+109DwMjSilit02XwL+3NyF\nlFIvAi+Cnk/BQ5sNnZiSqjp+/e56Fmwp4NT+3fjbZcNJigqlwl5PXpmdnNIa8krth+Qn2phTxvzN\nBdQ5mmaUzYgP580bx9MvOcqPT2MwHEZlIcy9BvYvhUl3whkPQ0DbjJHxVBTigWIO/YpXwLFEYSXQ\nT0Qy0YHpy4Er3Q8QkRS3uRnOB7Z4aI+hC/PdjiLunruOsup6Hjx3MD87uVdjTqCo0CCiQoPof5RK\nXilFcVUdeaV2DlTWMrpnHDFhJsuooR2RsxrevgpqDurWwbBL2/T2no5o9jSO4H6Ow+qpNA+wAXOU\nUptE5FFglVLqY+AOETkfcAAlwHUtvY+h61DrcPK3edt46bs99E2K5PWfjWNwanSLriEiJEaGkGgC\nyYb2yI//hU9/CVHJcMN8SBne5iZ4NB2niLzKkfEAlFLX+8KoY9HcdJyGzs/OwkrufPtHNuWWc9WE\nDH47czBhwSblhKGT4KyH+f8Hy5+HzFPh0tcgIsGrt/DqdJzAp27rocBFHBYfMBh8gVKKt1Zk8ein\nmwgLsvHSNWOZZgaUGToTVQdg7rWw73uYeDuc+YjPxyIcC0/dR++5b4vIW8ACn1hkMFgcrKrj/vfX\nM29TAZP7JvL3WSNIjjYTDBk6Eblr4Z2roKoILnoRRvzE3xZ53FI4nH6A6Rtq8BlLdh7g7rnrKK6q\n5bcz9fgBM8GMoVOx7h345A4IT4Tr50HqSH9bBHieJbWCQ2MK+eg5FgyGI3C6FPM25fPyd7vZWVhJ\nVGgQkSGBRIXqJTI0qHE9KiTQ6jEUaB0TxKLthby4eDeZiRG8fO0khqbF+PuRDP7C5YKdX4HYICYN\notMgtGWdC9odtZWw8I+w7FnoORkuew0iu/nbqkY8dR+ZTtyG42Kvd/K/1dm8/N1u9hVX0zMhnAtH\npVFV66TCXk+F3cGByjr2HKiiwu6gotZxyJgBd64Yl8GD5w4iPNjMOtZlsZfDBzfDts8P3R8SowUi\nJl2LREwaxPSw1tP1yN/Adti7rLIQlr+gB6PZS2HcbDj7j2BrX12iPW0pXAR8o5Qqs7ZjgSlKqQ99\naZyhY1BSVcd/lu7j9aV7KamqY0SPWO6bPpCzh3THdhyXT63DqQXC7qDS7qDCXk9UaBDD0k3roEtz\nYAe8fSUU79IVZ9oYKMvWS3lO03rOaqguPvL8iCQtEIcIhtt6ZDIEeJr6rbXPshOW/hPWvgXOOhh4\njh6Q1mNc29y/hXj6GfawUuqDhg2lVKmIPAwYUejC7C+u5uXvdzN3VRb2ehdnDExi9qm9GZcZ73H+\noJBAGyGRNjNu4HBcTtj5NdRVtPBEgYhuTa6W9vjFfDy2fQnv36S/oK/5CDJPOfbxddVQngvlllCU\n5UBZlhaPou2w8xuorzr0nIAgiE6B6HQ38UiHhL5agEK84BzJWgk/PAlbPwNbMIy8Aib+AhL7tv7a\nPsRTUWhOUk27vouyPruUFxbv5osNedgChAtHpjH71N4mVYQ3WfhH+O5vrb9ORFKTQMT0OHI9MrnN\n0iccF5cLvvs7LHwMug+Dy/8LsR70ZwkO1xXt0SpbpbS7psxqYZRnu63nQNYy2JQLLitvlgRA8hDo\nMQEyJkCP8RDbo/lrN/cMO+bBD0/pFBWhsXDKr2D8zRDZMXJreTp4bQ5Qip4fQQG/AOKUUtf51Lpm\nMIPX/INSikXbi3jx290s3V1MVEggV07I4GcnZ9I9xnQT9So5a+DlM2HoJXDqPS07V7m07/pwN0vD\nel3loccHBEJUKiQN1O6MHhP0l3JwuPeexxNqK+DDW2HLJzBsFpz3VNva4HJBZQEUboL9yyFrOWSv\namphRKc1vZ+M8ZA87NCxBI5aWD8XljwNB7Zr0Z14G4y6GkIi2+45joGng9c8FYUI4EHgTGvXfOAx\npVTV0c/yDUYU2galFLuKqliz7yCr9pWwfE8J+4qr6R4dyg2TM7l8XA+iQttXgKxT4KiFF07VQdaf\nL4WwWO9dWymwlx0pGGXZkLcODmzTxwUEQvfh+gs5Y7yuCKNTvGfH4RTv0vGDA9th2u91Zdoe0pc7\nHVCwUQtE1nItFuXZuiwoXItnxgSwhejgcWW+buGcfCcMubDdBZC9KgrtCSMKvsFe72R9dhmr9x1k\n9b4SVu87yMHqegBiw4MYnRHHOcNSOG9EKsGBbRSg64os+B18/w/46bvQb1rb3ru6BLJXwv5luhLM\nWQ0Ouy6LybAEYryuCJMGe8fttGMBvHe9dtlc+ir0mdr6a/qSsmzr/azQbqf8Dbp11nsqTLpD/20P\ngtYM3m4pfAVcppQqtbbjgLeVUme32tIWYkTBOxRV1LJ6Xwmr9h5k9f6DbMwpo96pfwu9u0UwJiOO\nsb3iGNMznt6JEWbgWFuQvRpeORNGXgkXPOtva3Q+//wNuvJrEIrKAl0WHAXpY5t87uljWxacVUqL\n39ePav/95f+FuF4+eQyfUlups5l6GnPwI94WhR+VUqOOt68tMKLQMuocLvYWV7Etv4LtBRVsy69g\nS345WSU1AAQHBjAiPYYxPeMZ2zOO0T3jiI8I9rPVXZB6O7xwCtRVabdRaDvskqsUlO6zfO7W13LB\nJkA1E5wdp/3qzX0111XBR7fBpg9gyMVwwTMQHNHmj9PV8HZCPJeIZCil9lsX70UzWVMN/sPpUmSV\nVLOtoILt+RX6b0EFew5UNbYAbAFCZmIEw9NiuWZCL8b0imNoaoxxB7UHFj6mfepXvd8+BQF0BR/X\nSy8NOXrsZZbLyfK7r30TVr6ky6JSm2ISDcHZsix4+6dQtEUnfpt0Z7t1t3RVPBWF3wLfi8i31vap\nWNNjGvxDYYWdb7cVsXxPCdvyK9hRWIG9vml0cI/4MAYkR3HmoGQGdI+if3IUvbtFEBLYTrofGprI\nWgFLn4HR10LfM/xtTcsIjYG+Z+oF3IKzls99/3LdIgAdnBWbHjT20/81nWNoV3gcaBaRJLQQrEWn\nzy5USi32oW3N0lXdRw6ni7VZpSzaVsSi7YVszCkHICEimMGp0fRPjmJAchT9u0fRLymSiBAzjKRD\nUF8Dz0/WvY5uXdLx8/o0R1l2U++dqiI440GI7+1vq7ocXnUficiNwJ3oeZbXAhOApRw6PafByxRV\n1PLt9iIWbSvkux0HKKupxxYgjM6I5d6zB3Ba/24MSY32ePSwoR3yzR+geCdc/WHnFASwRgyn63EX\nhnaPp5+TdwInAcuUUlNFZCDwiO/MaifUVekugiOvhFTfx9SdLsXarIO6NbCtiA05ZQB0iwph2uBk\npgzoxil9uxETfoL9nxsyTm79VA8QOl76gI5CeS7Mf1D7t1uCBMDAmXqAkT9G9e5fBkufhbHXt/+u\nmIYug6e9j1YqpU4SkbXAeKVUrYisVUq1eQLwNnUffXQb/PgGhCfAjQt81uQtLLfzr0W7+HBtDqXV\n9QQIjM6IY8qAbkwZkMTglOjWdQl11MHGd+GHp3WAT2y6b/XE2+D0ByGoA49IdtTCqzOhcDN0G9iy\nc2vL9Vd68jCY/qe2Fcm6anh+kk6tcOsS7+TaMRiOgbd7H2VbmVE/BL4SkYN4MB2niEwHngJswMtK\nqcePctylwP+Ak5RS7SNgsPE9LQgjr4Jtn8Ebl2phCI/32i1Kqup4/ttd/HvpXhxOxTnDUzhzUDKn\n9EskNtwL3ULt5bD6NVj2HFTkQtIQPbtT/7O122LpM7BzAVz0QruZ4KPFfPFryFkFs/4Dg89v2blK\nweYPYf5D8Pq5MOg8PaI2PtM3trrzze+hZDdc+4kRBEO7osUjmkXkNCAG+FIpVXeM42zAdmAakA2s\nBK5QSm0+7Lgo4DMgGLj9eKLQJi2Fg3vh+VP0l+fPPtc5UP59gXYhXfNRq7+sy2rqefm73cz5fg/V\n9U4uGpnGnWf2o2eCl/pql+fB8udg1av6azjzVN31r88Zh3b/2/k1fHQ7VBXCaffD5F/6dW7YFrP6\ndT1z1eS74cyHT/w69TVaIL/7B7jqdQvqlF/5rrLet0S3bk66Ec7xQtI7g8ED/J7mQkQmAr9rGPUs\nIg8AKKX+dNhxT6Lne74HuMf2n+hhAAAgAElEQVTvouCsh1dnQNE2uOV7iOup9298D969HoZcBJfM\nOaFc7FW1Dl5bspcXvt1Fud3BOcNSuOvMft7LLlq4FZb8E9a/A8oJgy/UQ++PFQ+pOQif3wsb/qdz\nuVz0AiT28449viR7lf536nWK7t7ojZhAeS4seATWv62zh57xEIy40rt59+uq4LlJgIJbfmg3ydIM\nnR9vu49OhDQgy207GxjvfoCIjAJ6KKU+FZGjpoMUkdlY4yIyMnw8NfSix/VgnEvnNAkC6J4TpVmw\n4GGdznfaox5f0l7v5I1l+/jXol2UVNVx5qAkfjmtP0NSvTBISSmdoveHp2D7lxAYBmN/BhN+7pkb\nJCwOLnkZBsyEz+7WLaRpj+qv2LaahKSlVBbCO1dDVIq23VtB4uhUuPgFGHcTfHm/jimteBGm/xl6\nTvTOPRY8Agf3wHWfGUEwtEt8KQrNRUYbmyUiEgD8A7jueBdSSr0IvAi6peAl+45kz2Kdz33U1c13\nn5t0px7m/8NTENsTTrrhmJerdTiZuzKLf36zk8KKWk7pl8jd0/ozKiOudXbW2yFvre69suUT7VMP\nT4Apv9GVeURCy6859GLImAgf/wK+uFdPgXjBszrnfnvCWQ//u063cG6Y79UYTyPpY+GGr3Tr6auH\n4dXpuoU47VHP8vsfjT3fwYoXYPwt0Guy9+w1GLyI39xHIhID7AIaErx3B0qA84/lQvKZ+6iqWPcG\nCY6Em789ei4Wp0On+d35FVzxtg7aHobD6eL9NTk89fUOckprOKlXHL86awATep9AZQ1QWWSl77VG\niOat1dP6gY57jLtJuzm8kX9eKVj9Ksz7rZ6d6py/wbDL2k8qgi8fgGX/gotfguGzfH+/uir9EfDD\nU3r75F/A+Fu1GLXkndRWwnMn626wt/5gcv0Y2pz2EFMIRAeazwBy0IHmK5VSm45y/CL8FVNQCt66\nAnZ9rXsYpYw44hCnS1FaXUdxVR0lB0sY+OXlRFTs4e0hz7OZPhRX1lJcVUdxZS1FFbVU1TkZkR7D\nr84awCn9Ej0fYOZy6bz27ul5S3brMluwjg80pC9OHweR3bz4Itwo3qUnPclaDoMvgHP+cWItEG+y\n/n/w/o26Up7RbEc231GapcesbHxXbwdHNk0UH5N26LSO0dbfoLCm8z/7Fax8RXdc6Hly29puMNAO\nRMEyYibwJLpL6hyl1GMi8iiwSin18WHHLsJforD8Re0ymf44TLi1cXd+mZ073/6RXUWVlFTV4XJ7\nVd04yAchDxOMg+sD/4QjKp2EyGASIkKIjwhmct9EzhiU5JkY1Nt1cHjrp1oI7KV6f3jioROdpIxo\n2zEFLqf+Ql74R/1l3H96y1sM3QbpGEdr5wrOWw+vnAVpo3UPMH9NYJKzGvYtdZvW0ZrasarwyGPD\nE7RQRCbDjvkw4TaY/se2t9lgoJ2Igi/wuijkb4SXTofeU+DKdxorvbKaen7ywlKySqo5f2QqiZEh\nJEQEkxAZQkJkMImRISTV7CbmrXOR6FS4fl7LZ8mqOai/Hpe/oCuV+D7Qa1JT+uH43u3DbZO/AT69\nW8dTWoJy6Vw3cZlw1h9g4Dkn9jzVJfDiFB1PuPnb9jnXraPWms3sKPMAh8XpiXPaeppLg8HCiIIn\n1FXrysZeqkeVRiQCOkB87ZwVrN53kFevG8fkfolHv8aexfCfi3UlftX7EOjBoLPSLO0XX/26ngO2\nzxk6iJ15avsQAW+ycwF8+RvtEss8VbfGkod4fr7LCf+9FPZ+Dz/7QgeBDQZDi/FUFNppn8M2Yt4D\nOof9xS82CoLLpbh77jqW7S7hb5eNOLYggK7oLngG9n6ne+4cS2TzN8B7N8FTI3TrYNC5eizE1e9D\n79M6nyCATo986w8w46/aBfT8ZPj0l1B1wLPzFz4Gu76BmX81gmAwtAEdaPiql9n0oU4BMeku7TpC\nT1b/+88289n6PH4zcyAXjPSwO+aIy6F0v67A4nrC1N80lSkFe77Vvvld3+gA5fhbdOyiA0zh5xVs\nQTB+Ngy7VI8DWfkybHgPptwHJ9109NbVlk90F+HR18KY69rUZIOhq9I13Uel+/UXa0JfHQuwgpYv\nfLuLP32xlesnZfLguYNalpJaKfj4dp0v6YJnYfjlOq/Okqchbx1EJMGEW3RGzLBWjlPo6BRuhXm/\n0b29EvrCWY/prr3u77tom471dBug3UatDVQbDF0cE1M4Gk4HvHaOnlv2lsWNmU/fX5PN3XPXce7w\nFJ6+fNSJZSV11sN/L9OupKhUKNsPCf103/bhP+nY2Ui9jVK6R8683+hMpX1Oh7P/BEkDdSK/l07X\nsZ7Z37a/AXQGQwekPaS5aJ8s/ovu+3/xy42CsHh7Eb9+dz0Teyfw91kjTjxNtS0IZv0b3rhEp16Y\n8Tj0n9F+00X4ExHdOug9VbuTFj2uB3eddIMOxJfshms/NoJgMLQxXUsU9n4Pi/+qR/8OvwyAjTll\n3PrGavomRfLCNWNaP4dxaDTc+JUXjO0iBAbDxJ/rltTCx7RAKJfupWRSQRgMbU7XEYXqEnh/tu4z\nP/MvAOwvrua6V1cQGx7M69ePIzrUTwOiDHq09LlP6JZC/sa2SWFhMBiOoOuIwq5vdDfIG+ZBSBTF\nlbVcM2c5Dpfi7evHkRxt/P3tguQhLRvHYDAYvErXEYVhl+qcM9GpVNU6uP61leSV2XnzpvH0TTIp\njA0GgwG62uC16FTqnS5ue3MNG3LKeObK0Yzp6YPUywaDwdBB6TotBfTgtAfe38CibUX86eJhTBuc\n7G+TDAaDoV3RpVoKf5+/nXdXZ3PnGf24YpyPZ3AzGAyGDkiXEYWP1ubwzMKdXDGuB3ed2QHmIDYY\nDAY/0GXcR1MGJHHH6X2544x+LUtfYTAYDF2ILiMKMWFB3H3WAH+bYTAYDO2aLuM+MhgMBsPxMaJg\nMBgMhkY6XJZUESkCWjgvZCOJgIezu3R6zLvQmPegMe9B05nfQ0+lVLfjHdThRKE1iMgqT1LHdgXM\nu9CY96Ax70Fj3oNxHxkMBoPBDSMKBoPBYGikq4nCi/42oB1h3oXGvAeNeQ+aLv8eulRMwWBoDSLy\nGpCtlPo/D47dC9yolFrQmusYDG1NV2spGAwGg+EYGFEwGAwGQyNdRhREZLqIbBORnSJyv7/t8Rci\nsldENojIWhFZ5W97vI31fPeKyHoRqRKRV0QkWUS+EJF6EakTkc1ux18pIpUi4hSREhEZ71Y2SkTW\niEiFiLwDhB52r3Ot91gqIktEZPgJ2nyT9bssEZGPRSTV2i8i8g8RKRSRMuuZhlplM0Vks2Vbjojc\n04L7zbGuudFt3++s66y1lpkn8iwdCRHpISILRWSLiGwSkTut/fEi8pWI7LD+xvnb1jZFKdXpF8AG\n7AJ6A8HAOmCwv+3y07vYCyT62w4fP98yIBlIAwqBNcAo4AxgBVBgHdsfqANeBoKAz4GD1m8kGD1I\n8pdW2aVAPfAH69zR1rXHW7+va617h7jZceZRbHzN7TqnowdLjQZCgH8Ci62ys4HVQCwgwCAgxSrL\nA06x1uOA0S14R6da99votu93wD3+/vdr499KSsN7A6KA7cBg4C/A/db++4E/+9vWtly6SkthHLBT\nKbVbKVUHvA1c4GebDL7jn0qpAqVUDvAdsFwp9aNS6mvgMyDMOu4ngB14UClVD9wIRAInAxPQYvCk\nUqpeKfUusNLtHjcBLyilliulnEqp14Fa67yW8FNgjlJqjVKqFngAmCgivdAiFAUMRHcK2aKUyrPO\nqwcGi0i0UuqgUmqNpzdUSi0GSlpoZ6dDKZXX8N6UUhXAFvSHxAXA69ZhrwMX+sdC/9BVRCENyHLb\nzrb2dUUUMF9EVovIbH8b4yMK3NZrDtu20/S7TwWCGypapVQu+os8zSrLUdbnooV7epWewK8s11Gp\niJQCPazzWkKq+3WVUpVAMZCmlPoGeAZ4FigQkRdFJNo69BJgJrBPRL4VkYktvG9z3G65qOZ0NZeJ\nJcKjgOVAsttvIg9I8p9lbU9XEYXmJlDoqn1xJymlRgMzgNtE5FR/G+RHcnH7PyB6oo0AIAftnkmT\nQyffcJ+uLwt4TCkV67aEK6XeOgEberrZEAEkWDaglHpaKTUGGIJ2d91r7V+plLoAXWF9CMxt4X0P\n5zmgDzAS/ex/b+X1OgwiEgm8B9yllCr3tz3+pquIQjb6K66BdPR/xi6H9TWMUqoQ+ADtWuuqzAUC\nReQyEQlC+9UdwBJgqbV+h4gEisjFHPquXgJuEZHxVkA4QkTOEZGoFtrwJvAzERkpIiHAH9Hurr0i\ncpJ1/SCgCt3KcYpIsIj8VERiLLdXOeA88dcAlrvNqZRyWc/WJX4X1rt9D/ivUup9a3eBiKRY5Sno\n2FGXoauIwkqgn4hkikgwcDnwsZ9tanOsiiuqYR04C9h47LM6L0qpbejfwTPoYO/VwH+UUnVW7Oli\n4Dp08PknwPtu565CxxWescp3Wse21IavgQfRFVMe+mv9cqs4Gl1BH0S7mIqBv1llVwN7RaQcuAW4\nqqX3dqehErS4iC7wu7Baga8AW5RST7gVfYzuOID196O2ts2fdJkRzVYXuyfRPUXmKKUe87NJbY6I\n9Ea3DkDPuvdmV3oPIvIWMAWdHrkAeJgm10sGsB+4TCnVqYOwR3kPU9CuI4XuOXWzW1C7UyIik9Ed\nETYALmv3b9BxhS71m3Cny4iCwWAwGI5PV3EfGQwGg8EDjCgYDAaDoREjCgaDwWBoJNDfBrSUxMRE\n1atXL3+bYTAYDB2K1atXH1AezNHc4UShV69erFrV6fK4GQwGg08RkX3HP8q4jwwGg8HgRpcRhTqH\ni4/W5mC64BoMBsPR6TKi8P6abO58ey3vr8nxtykGg8HQbulwMYXmqK+vJzs7G7vdftRjhkfCvy9O\no76ukA0bywi0dUw9DA0NJT09naCgIH+bYjAYOiGdQhSys7OJioqiV69eHJrU8lDqHC52FFYQEmij\nT7eIYx7bHlFKUVxcTHZ2NpmZmf42x2AwdEI65ufyYdjtdhISEo5byQcHBpAWG0Z1nYPCito2ss57\niAgJCQnHbBEZDAZDa+gUogB4/NUfGx5MXHgwheV2qmodPrbK+3S01o3BYOhYdBpRaAmpsaEEBQaQ\ndbAap8t1/BMMBoOhi9AlRcEWEECPuHDqHS5yS1vviiktLeVf//pXi8+bOXMmpaWlrb6/wWAweIsu\nKQoAESGBJEWHcrC6jtLqulZd62ii4HQeezKszz//nNjY2Fbd22AwGLyJz3oficgc4FygUCk1tJny\nnwL3WZuVwK1KqXWtve8jn2xic67n06zW1DtRShEWFMjR3PWDU6N5+LwhR73G/fffz65duxg5ciRB\nQUFERkaSkpLC2rVr2bx5MxdeeCFZWVnY7XbuvPNOZs+eDTSl7KisrGTGjBlMnjyZJUuWkJaWxkcf\nfURYWFiLnt1gMBhaiy9bCq8B049Rvgc4TSk1HPg98KIPbTkqoYEBKKDWceJT3D7++OP06dOHtWvX\n8te//pUVK1bw2GOPsXnzZgDmzJnD6tWrWbVqFU8//TTFxcVHXGPHjh3cdtttbNq0idjYWN57770T\ntsdgMBhOFJ+1FJRSi0Wk1zHKl7htLgPSvXHfY33RH42SqjqyD1bTPSaUpKjQVtswbty4Q8YRPP30\n03zwgZ4FMysrix07dpCQkHDIOZmZmYwcORKAMWPGsHfv3lbbYTAYDC2lvQxeuwH44miFIjIbmA2Q\nkZHh9ZvHhQdRYQ+ioKyWyJBAwoNb91oiIiIa1xctWsSCBQtYunQp4eHhTJkypdlxBiEhIY3rNpuN\nmpqaVtlgMBgMJ4LfA80iMhUtCvcd7Ril1ItKqbFKqbHduh03HfiJ2EBabBiBNiGrpAanq2VJ86Ki\noqioqGi2rKysjLi4OMLDw9m6dSvLli3zhskGg8HgE/zaUhCR4cDLwAyl1JGO9jYk0Ka7qe4+UEle\nWQ3pceEen5uQkMCkSZMYOnQoYWFhJCcnN5ZNnz6d559/nuHDhzNgwAAmTJjgC/MNBoPBK4gvU0lb\nMYVPj9L7KAP4BrjmsPjCMRk7dqw6fJKdLVu2MGjQoNYZa5FXVkNRRS09EyKICWufSee8+bwGg6Fr\nICKrlVJjj3ecL7ukvgVMARJFJBt4GAgCUEo9DzwEJAD/slI3ODwx2NckR4dSaXeQc7Ca8OAogjpo\nNlWDwWA4EXzZ++iK45TfCNzoq/ufKAEi9IgPZ2dhJVkl1WQmdrxsqgaDwXCimM/gZggNspESE0pl\nrYMDla0b7WwwGAwdCSMKRyE+Ipjo0CDyy+2UVteZaTwNBkOXwIjCURAR0uPCCAkMYH9JNdsLKiip\nqsVlxMFgMHRijCgcg0BbAP2SIumZEE6ACNkHa9iWX0FRRW2LxzIYDAZDR8CIwnEQEWLCgumbFElm\nYgTBgQHkldWwNb+cgnI7DqfrhFNnAzz55JNUV1d72WqDwWA4MYwoeIiIEBUaRJ9ukfTpFklEcCAF\n5Xa25lewfX8+zxpRMBgMnYD2kvvIe3xxP+Rv8O41uw+DGY83bkaEBBIREkhNvZOiilru/b/fsGvn\nLoYMG87ZZ00jpXt35s6dS21tLRdddBGPPPIIVVVVzJo1i+zsbJxOJw8++CAFBQXk5uYydepUEhMT\nWbhwoXftNhgMhhbS+UShDQkLspERH86TT/yVC87byttffseSRV/z7bxP+fb7JYQG2Tj//PNZvHgx\nRUVFpKam8tlnnwE6J1JMTAxPPPEECxcuJDEx0c9PYzAYDJ1RFNy+6NuKkEAbgbYABnaP4tlli/l2\n4QJGjx6NiFBbU8W27ds57dRTueeee7jvvvs499xzOeWUU9rcToPBYDgenU8UjobLAU4HBIZw1CnW\nWkmQLYCIYBu//c0DzLrqeoqr6qh1OAkMCCA6Ioily1ewYP48HnjgAc466yweeughn9jRoam3g70U\norr72xKDoUvSdQLNtRVQtAXy18OB7VCWDdUlUF8DrRx74J46++yzz+b1114jVOrpnxxJSG0p1WXF\nbNqxl+xyJ5OnX8Rtd9zFmjVrjji3S1NbAT88BU8Og78PgM9+BXbPp1U1GAzeoeu0FIIjIDYD6quh\nrgaqioEiXSYBEBgGweEQFAZB4RAY6nGLwj119owZM7jyyiuZOHEiAJGRkbzxxhvk79nDL66dhQsh\nMDCQ3/35HxRV1HLDTTcxY8YMUlJSumaguboElj8Py1/QLYTeUyH+XFj5Cmz7Es79B/Q/y99WGgxd\nBp+mzvYFXkudrRQ47Fok6mugrhocNaBcutxdKGwhYAuGwCD9N+DEtdSlFGU19RRX1lFd5yBAhNjw\nIBIiggnzcMY3r6XOdjnBXgY1B6Gm1PrbsJQctl0KCX2g31nQ53QIi23dvcvzYOkzsOpVqK+CgefC\nKXdD2hhdnrUSPr4dirbCsFkw/XGISDj2NT3FXg6rXoE1/4aUkTDtEf3BYDB0YvyeOrvdI2K1CsKa\n9h0uFPXVUF3cJBSN59rAZgmELRgCg5vWG0TjKK2MABHiwoOJCw+mps5BcWUdpdX1lFTVERZka+zu\nGhGsg9deobIQ8tbpJX89FGyCqgNaEDjGR0FIjK78w+IgJAq2fwnr3tLPnzER+p+tl8T+nsdpSvZo\nN9Ha/2pRGnYpTP4lJB0mcj1OgpsXw3dPwHd/h11fw4y/wNBLTjwmVFUMy5+DFS/qZ+8xHrZ9Ads+\nh0l3wqS79EeAwdCF6botBU9RygpS1x26OOrAWa/XlfOwk0SLRoDNEohAsAW6bVt/rcVBAAdrnJTb\n66mpczbmVwoJtBERYiMiOJCIEBvBgbZjP69SULq/qfLPWwd566Eyv+mYuEzoPhSiUnRl37jEH7od\nGqNtdsflhOyVsH0e7JgPBRv1/tieTQLRczIEhR5pW+EW+P4fsOFd/R5G/lRXxPGZx/83KNisWw05\nq6H/dDjnCYhJO/55DZTnwpJnYPWrWugHnQeT74a00VCaBQseho3vQXQaTHu0dcJjMLRTPG0pdBpR\nGDhwoP/mPXA5mgTCYYmGy3HY4mxGPNwRCAhESQAuBCeCwyXUK8GpBBcBiAQQYLOxLyuHlIq1dItP\nIACXHqyXt07/tZdalwuAbgOh+3BIGQ4pI6D7MFzB0TiV8s7kQaVZWhx2zIfd32r3W1A49J6i3Uz9\nz4aKPP21v/VTCIqAsT+DibdDdErL7uVy6tjD17/XYjrtERjzMwg4xnMU79KtknVv6fOHz9KtgaSB\nRx67bwl8cZ8W0x4TdNfm1FEts9FgaMd0KVHYs2cPUVFRJCQktO8JcZRLV04NItGcaLhc+q9ygXKh\nlAvl0tvK5eRgVT0Vu1eRueyBxsvWEcT+wF7sCuzLjoBMtkpvtqseVLmCqXW4qHfqpc7hwmEl8use\nHUrfpEj6JkXSJymSvt30emJk8Im9w/oa2PMd7JgH2+dD2f6mstAYGH+LXsLjW/cOS/bAJ3fAnsXQ\ncxKc9zQk9j30mIJNulWy8T0ICIJRV+lWSVzPY1/b5YQf34CvH9Vuw1FXwRkPQWRS62w2GNoBXUoU\n6uvryc7Oxm63+8mqtiM0JBgJDGbDvny27Mtnb3EVB4JSsQUGExQYQLBNCLIFEBwYQJBNL4fvA8gq\nqWZnUSW7CiupqmtqwcSEBdHPEgt3wUiLDSMgwEOxUEoHiHfM1zGWUVfpmIS3UEpX3vN+q2NAUx+A\nib+AvLU6/rDtcwiOhLHXw8TbWj7mwV4G3/5Ft0yCwuG0X8O4m3XsyGDooHQpUTCcOEop8srs7Cys\nZGdhJTsKtVDsLKqkpKpp1rmwIBsT+yRwwchUpg1OJtzDnlI+pSJfj2fY+ilEJEFVoY6HjL8Vxt3U\n+lbJgR1aeHbMg/g+MP1P2iV2NOzlUJ4DZTlQnq1jGQ3rtZUQngAR3XQvqohuEJ7YtN2w3lw8xtA+\nObgP9nyre9KNuPz4LVE/Y0TB0GpKquoaxWJbfjlfbS4gt8xOWJCNs4Ykc+HINCb3S/ROfKI1bP5I\nj2voN03HGUIivXv9HV/Blw9A8Q7oeyYMOl/HSsqsir9BCOoOG4QoARDZHaJTdUupulj3+qo+oONO\nzREcCRGJWiQikyCmh+4u676ExZlAuD+oLtFuy92L9HJwT1OZ2GDoxdpN2X2Yvyw8JkYUDF7H5VKs\n3FvCh2tz+XxDHmU19cRHBHPOsBQuHJXK6Iy49h3TaQ3Oet2VddGfobYMEF1pR6fpnlDRaUeuR3XX\nvdAORymoLbcEohiqivR6VdGh25WFujfZ4WITHHWkULgvrW0hGTR11ZC1rEkE8tYDSr//XpN1h4re\nU/RHyLLnYPVrUFcJfc6AyXdBr1PalXgbUTD4lDqHi2+3F/HR2hy+2lxArcNFelwYF4xM5YKRafRP\n9mIMoT1hL9cD+6JS2ybGoJTuUVa6v/nl4L4jRSO+N2SeBpmn6iWiHWTgLdkNuxbqv5mn6cq0vcVo\nnA7di2/3Qi0CWct1iy4gSI9p6W3ZnTr6yO7aoAd5rnxFx6KqivRxk+/SAzMDbG38MEfiVVEQkTuB\nV4EK4GVgFHC/Ump+aw1tKUYU2h+VtQ7mbczno3W5fL+jCJeCQSnRXDAylVE9YrE7XNjrndjrndTU\nWX/rXdRY+xr2620XESE2rhiXwfjM+M7b8vAWh4tGyW7dvXbvD01ikTxMi0Pv06Dnyd4N+h+N6hLt\nb9+9SItB6T69PyBQ97YLiYEB02HwBXqEvPsg0rbA5YSibbpzQu6PkLtWd+l21Ojy7sMs8ZoKPSfq\nNDmeUl+ju0H/8LR2McX3gZN/ASOu8DxmpJSOmRVs1HYVbIT8jTDySi00J4C3RWGdUmqEiJwN3AY8\nCLyqlBp9Qta1AiMK7Zuiilo+XZ/LR2tzWZtVetzjQwIDCAu2ERZkI9Ra8spqKK2uZ2haNDed0puZ\nw1L8H7foaDgdurLb861e9i8HZ632faeN0QKReSqkj/NOcNtRC/sbXC0LdSXb4GrJPEVXrn2mavfW\n7kWw+WPdQcBequMo/c7SAtFvWssqYE9wOXWngdwfm0Qgf4MeyAj6/t2HQ+pISB+rxcAbrSuXE7Z8\nDN8/qe8bkQQTboGxNxyaJsZRBwe26UrfXQSqi5uOicnQg06HXaZjFyeAt0VhvVJquIg8BSxSSn0g\nIj8qpdp8dI8RhY7DvuIq9pdUN1b4YcHWX2sJCQxotpurvd7J+2tyePn73ewuqiIlJpTrTu7F5eMy\niAlrxkdvOD71NZC1whKJxZCzRo+HCQzVrpHE/lZCyIbFSgx5yL7wQ5NGVhboVsDuRbp14qjRLYH0\nkyx/+1QtQM25WkDHafYs1hXnlk91AD4wDPqdCYMv1EIRGn38Z1NK+/KrDjTFZqqK9Cj6vLU6FlBf\npY8NCtcDOVNG6sGJqSMhoa9v3TtK6ef84SmdriU4Ckb8RPdIK9ioWyyuen1sYKhO+ZI8VLdWkodC\n8pDW5xrD+6LwKpAGZAIjABtaHMa01tCWYkSh6+ByKRZtL+SlxXtYuruYiGAbs07qwfWTMukRb3IU\ntQp7ma7I9yzWS3mODqw6a1t+rcT+TS2BnpM8q8gPx+mA/Ut1T7Itn+jULLZgHbQdOFOLTWNAviEo\n77buaGaMUlB4UwsgdZQWgsR+/vXv563X4rDpA90FufvQJgHoPky7mo4moq3E26IQAIwEdiulSkUk\nHkhXSq0/xjlzgHOBQqXU0GbKBXgKmAlUA9cppdYczxYjCl2TjTllvPL9Hj5Zl4tLKaYP7c4Nk3sz\npmecv03rXLicTckg66u1UNTX6C/thv11VllwpHZBtSQPlUc2uCB7hRaIzR/rcR4N2EJ0r6/GMR/d\ntKsnIvHQ7fBE3RW4HQR4m8Xp8FnlfzS8LQqTgLVKqSoRuQoYDTyllNp3jHNOBSqBfx9FFGYCv0CL\nwnjreuOPZ4sRha5Nfn2INKgAABCLSURBVJmd15bs5c3l+yi3OxiVEctNp/Tm7CHdsXk64trQcVBK\nT4plC9IVfnBku+rm2ZHwekwB7TYaDvwHeAW4WCl12nHO6wV8ehRReAHtgnrL2t4GTFFK5R3rmkYU\nDABVtQ7+tyqLOT/sZX9JNcnRISRGhqBUUzJw9992w6qySpXSdcsp/bpx0ym96R5jRhIbOjfenk/B\noZRSInIB+ov+FRG5tnUmkgZkuW1nW/uOEAURmQ3MBsjIMJOhGCAiJJDrJmVy9cRefLW5gI/X5VDn\naJj3Qho/Jhu+KUVArC0RvVTXOXltyV7+s3Qfl4xJ55bTetMzwcs9XwyGDoanolAhIg8AVwOniIgN\naG03kObagM02W5RSLwIvgm4ptPK+hk6ELUCYPrQ704e2MOmdRVZJNS8s3sXcVdm8s3I/549I5dYp\nfRnQvZMOvjMYjoOnnb9/AtQC1yul8tFf9H9t5b2zgR5u2+lAbiuvaTC0iB7x4fzhwmF8/+up3HhK\n7/9v796D4yrPO45/H0mW5NVl5dX9LhvZRr4IA7ULMQnG2GAnBEOBJqTQhGmHTIvTpKEZaIZOGDrp\ndJqBJjOlgXIZcEJIQzC3DAVitbGxwRjj+CIsMLZkSau7tUiWtLrr6R/neJF8FbZWK2mfz4xnd8+e\nPX72zJF+Ou97zvvy1sEWrv/pNu7etJt947jPwpiZZlyh4AbBc4BXRG4A+lR10wX+368CfymOK4DO\nc/UnGBMuWamJ/PDLZey4bzXfvXY+79UE2PDoDu548j3ePdLOePrejJkJxtvR/Oc4ZwZ/wGn2+SLw\nA1X97Vk+8zywCsgAWoAf4TY5qepj7iWp/wGsw7kk9S5VPWcPsnU0m8nQ3T/EcztreeLtGo5193NZ\nURobV5dyzcIsG3rDTEsTPswFsFZVW93XmcAWVb3kgiv9nCwUzGTqGxzmhd31PLa1moaOXspyU/nW\nF4pZfXE2mSkJkS7PmHGb6FA4oKpLR72OAfaNXjZZLBRMJAwOj/DK3kZ+/ofDHGnrQQSWFaaxpiyb\ntYuymZ+VbGcQZkqb6FD4Cc49Cs+7i74G7FfV+y6oyvNgoWAiSVU52HSciqpWtlS1sN/fCUChbzZr\nyrJZU5bNirk+G8DPTDkTPp+CiNwCrMTpU9imqi9dWInnx0LBTCUtx/uoqGqloqqF7YeP0T80Qkpi\nHFcvyGTtomxWLcjC67FB/Ezk2SQ7xkyy4MAQOw63s+VgCxUftXCse4DYGGF5yRyuKs2g0Och1zub\nvLREslMT7WzCTKoJCQUR6eL0N5QJoKp6HsMhXhgLBTMdjIwo+/wdbKlqoaKqlY+ax86OFiOQlZJI\nbloieWmzyU+bTa7XeZ7nBocvKd76KcyEsTMFY6aQ4MAQjR19NHb00tTZS8Oo5yeW94eG6XB44mO5\nvHgOV8xL54p56ZQXeO3swpy3iR77yBhzATzxcZRmJVOalXza91WVQM8ATZ19NHT00tjRS82xHnbV\nBPjJmx+724jlT0p8XDHPx5Xz0lma7yXOQsJMMAsFY6YAESE9OYH05ASW5HvHvNfe3c+umgDvVrez\ns7qdf3vDCYmk+FiWz/WFziSW5KVaSJgLZs1Hxkwzx7r7ea86wM7qdt6tbudwazcAyQlxLC+Zw8Kc\nVErSPRSleyhJTyInNfG0056a6GLNR8bMUBnJCXylPJevlOcC0NbVz073LGJXTYDth48xOPzZH3vx\ncTEU+TwU+zwUpydRnO5x/yVRMGe29VOYMSwUjJnmMlMS+OoleXz1kjwAhkeUxo5eatuD1AZ6nMd2\n5/GdI+30Dg6HPhsbI+SlJbIgK4Ul+V7KC7wszfeSlWqTDkUrCwVjZpjYGKHQ56HQ5+EqMsa8p6q0\ndfVTGwhy9FgPdYEgNcd6+Li5i//9uDU0Q112agJL870szU9jaUEqS/K9ZKVYUEQDCwVjooiIkJWa\nSFZqIstLfGPe6+kf4mDTcfb7O6ls6ORAQycVH30WFDmpiWPOJi4tSiPNEx+Bb2HCyULBGAM4U5wu\nL/GNCYvu/iEONh7nQEMnB/wdblC0oAoJcTHcvqKIb189j1zv7AhWbiaSXX1kjPlcuvuHqGzoZPMe\nP5v3NCACt15eyN+uuohCnyfS5ZkzsDuajTFhVx8I8tjWI7yw28+wKjdfms8915QyNyMp0qWZk1go\nGGMmTXNnH49vO8Kv3qtjcHiEG8rz2Li6lAXZKZEuzbgsFIwxk66tq58n367mFztrCQ4Ms35JDhtX\nl7I4z3vuD5uwslAwxkTMpz0DPL2jhmd2HKWrf4g1ZVlsXD2fZYVpkS4talkoGGMirrN3kE3vHOWp\nHTV0BAdZWZrOTcvyWVOWzZwku5x1MlkoGGOmjO7+IX65s5ZN7xylsbOP2BjhT+f6WLckh+sW5ZDj\nnbgb41QV/6e9ZKYkkDgrdsK2O91ZKBhjphxV5UBDJ29UNvPGh81Ut/UAcGlRGusW53D94hxKPseV\nS6pK8/E+9tV3st/fwX6/83i8b4g8byIPbVjCmkXZ4fo604qFgjFmyjvc2hUKiMqG4wBcnJPC9Ytz\nWLckh4tzUsbMPhfoGWCfv4P9J0KgoZO2rn4A4mKEhTkplBd4WZCdwvO76jjU0s26xTk8eOPiCT0b\nmY4sFIwx00p9IMhbB1t4s7KZ92sDqEJxuodrFmbR2tXHfn8n/k97ARCBizKTKS/wUp7vpbwwjUW5\nqWOaiwaGRnhyezU/2/IJs2Jj+IfrFnDnlSXERukw4hYKxphpq62rny1VLbxR2cw7R46R402kvCDN\nCYCCNJbkp5KSOGtc26prD/LAK5VsO9RGeYGXf7l56SkTGUWDKREKIrIO+BkQCzypqv960vtFwLNA\nmrvO/ar6+tm2aaFgTHQZGdELniRIVXltfxMPvXaQQE8/d62cy/fXLiApIXqGfxtvKIRtdg0RiQUe\nBdYDi4DbRWTRSas9APxGVS8Fvg78Z7jqMcZMTxMxa5yIcOMleVTcezW3ryjiqe01rH1kK2992DwB\nFc4s4ZxyaQVwWFWrVXUA+DWw4aR1FEh1n3uBxjDWY4yJct7Zs/jxzUt58W+uJCVxFnf/4gPu3rSb\nxo7eSJc2ZYQzFPKB+lGv/e6y0R4E7hARP/A68J3TbUhE7haR3SKyu62tLRy1GmOiyOXFPn73d1dx\n//qL2fZJG2sf2crT22sYHplefazhELY+BRG5DbheVf/afX0nsEJVvzNqne+7NTwsIlcCTwFLVHXk\nTNu1PgVjzESqDwR54OVKth5qIyM5gXmZSRT7PBT5PBS5c1kX+TzM8cwac3nsdDPePoVw9rL4gcJR\nrws4tXnor4B1AKr6rogkAhlAaxjrMsaYkEKfh2fuWs4blc38vqqF+kCQrYfaaHXvfzghJSGOQp+H\n4vRRgeFzAiMvLZG42HA2vEyecIbC+8B8EZkLNOB0JH/jpHXqgGuBZ0SkDEgErH3IGDOpRIT1S3NZ\nvzQ3tKx3YJj6T4PUtgepbe+hPhCkNhDk45YuKqpaGRj+rEEjLkbInzObIjc0in1JFLnhUZzuwRM/\nfa5yClulqjokIhuBN3EuN31aVT8UkYeA3ar6KnAv8ISI/D1Op/O3dLrdOGGMmZFmx8eyIDvltHNC\nDI8oLcf7qG0PUhfocYIjEKQ+EOS1fU109g6OWT8jOcENC+cMY25GElcvyJySc1zbzWvGGDPBOoOD\n1LphURcIUtcepDbQQ117kKbjfahCfGwM15Zl8WeXFbBqYSazwtz8NBX6FIwxJip5PbMo96RRXnDq\n/BF9g8Mcaunilb2NvLK3gf+pbCY9KZ4bl+Vxy2UFLM5LjWiHtp0pGGNMhAwOj7DtUBsv7vGz5aDT\nT7EwO4VbLs/npmX5ZKVO3CB+U2KYi3CwUDDGzEQdwQFe29/E5j1+/ljXQYzAlxZkcstlBaxdlH3B\nc0NYKBhjzDR1pK2bzXv8vLSngcbOPlIS47ihPJfbVxSdtklqPKxPwRhjpqmLMpP5wfUXc+/ahbxb\n3c6Le/y8/MdGMpITzjsUxstCwRhjpqiYGGFlaQYrSzP45w1DDA2Hv2XHQsEYY6aByRrme2bcl22M\nMWZCWCgYY4wJmXZXH4lIG1B7nh/PAI5NYDnTme0Lh+0Hh+0Hx0zeD8WqmnmulaZdKFwIEdk9nkuy\nooHtC4ftB4ftB4ftB2s+MsYYM4qFgjHGmJBoC4X/inQBU4jtC4ftB4ftB0fU74eo6lMwxhhzdtF2\npmCMMeYsLBSMMcaERE0oiMg6EflYRA6LyP2RridSROSoiBwQkb0iElXDzYrI0yLSKiKVo5b5ROT3\nIvKJ+zgnkjVOhjPshwdFpME9LvaKyJcjWeNkEJFCEfk/EakSkQ9F5Lvu8qg7JkaLilAQkVjgUWA9\nsAi4XUQWRbaqiLpGVZdF4fXYzwDrTlp2P1ChqvOBCvf1TPcMp+4HgH93j4tlqvr6JNcUCUPAvapa\nBlwB3OP+XojGYyIkKkIBWAEcVtVqVR0Afg1siHBNZpKp6jYgcNLiDcCz7vNngZsmtagIOMN+iDqq\n2qSqe9znXUAVkE8UHhOjRUso5AP1o1773WXRSIG3ROQDEbk70sVMAdmq2gTOLwkgK8L1RNJGEdnv\nNi9FVZOJiJQAlwLvEeXHRLSEwulmwY7Wa3FXquplOE1p94jIlyJdkJkSfg5cBCwDmoCHI1vO5BGR\nZOBF4HuqejzS9URatISCHygc9boAaIxQLRGlqo3uYyvwEk7TWjRrEZFcAPexNcL1RISqtqjqsKqO\nAE8QJceFiMzCCYTnVHWzuziqj4loCYX3gfkiMldE4oGvA69GuKZJJyJJIpJy4jlwHVB59k/NeK8C\n33SffxN4JYK1RMyJX4Kum4mC40JEBHgKqFLVR0a9FdXHRNTc0exeYvdTIBZ4WlV/HOGSJp2IzMM5\nOwBn1r1fRdN+EJHngVU4wyO3AD8CXgZ+AxQBdcBtqjqjO2HPsB9W4TQdKXAU+PaJdvWZSkSuAt4G\nDgAj7uIf4vQrRNUxMVrUhIIxxphzi5bmI2OMMeNgoWCMMSbEQsEYY0yIhYIxxpgQCwVjjDEhFgrG\nTCIRWSUiv4t0HcaciYWCMcaYEAsFY05DRO4QkV3u3AKPi0isiHSLyMMiskdEKkQk0113mYjsdAeT\ne+nEYHIiUioiW0Rkn/uZi9zNJ4vIb0XkIxF5zr2z1pgpwULBmJOISBnwNZzBA5cBw8BfAEnAHndA\nwa04dwIDbALuU9VynLtjTyx/DnhUVS8BvoAz0Bw4o3F+D2duj3nAyrB/KWPGKS7SBRgzBV0LXA68\n7/4RPxtnULQR4L/ddX4JbBYRL5Cmqlvd5c8CL7hjTOWr6ksAqtoH4G5vl6r63dd7gRJge/i/ljHn\nZqFgzKkEeFZV/3HMQpF/Omm9s40Rc7Ymof5Rz4exn0MzhVjzkTGnqgBuFZEsCM3ZW4zz83Kru843\ngO2q2gl8KiJfdJffCWx1x+X3i8hN7jYSRMQzqd/CmPNgf6EYcxJVPSgiD+DMUBcDDAL3AD3AYhH5\nAOjE6XcAZ3jlx9xf+tXAXe7yO4HHReQhdxu3TeLXMOa82CipxoyTiHSranKk6zAmnKz5yBhjTIid\nKRhjjAmxMwVjjDEhFgrGGGNCLBSMMcaEWCgYY4wJsVAwxhgT8v9CwYZPXqCR3wAAAABJRU5ErkJg\ngg==\n", 652 | "text/plain": [ 653 | "" 654 | ] 655 | }, 656 | "metadata": {}, 657 | "output_type": "display_data" 658 | } 659 | ], 660 | "source": [ 661 | "################################\n", 662 | "#Compile model, set optimizers\n", 663 | "################################ \n", 664 | "\n", 665 | "adam = optimizers.Adam(lr=LEARNING_RATE, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0, clipvalue=0.5)#, clipnorm=1.)\n", 666 | "rmsprop = optimizers.RMSprop(lr=LEARNING_RATE, rho=0.9, epsilon=1e-08, decay=0.00)\n", 667 | "\n", 668 | "model.compile(loss='categorical_crossentropy',\n", 669 | " optimizer= adam,\n", 670 | " metrics=['accuracy'])\n", 671 | "from keras.callbacks import History \n", 672 | "history = History()\n", 673 | "\n", 674 | "early_stopping = EarlyStopping(monitor='val_loss', patience=30)\n", 675 | "\n", 676 | "history = model.fit(X_train, y_train,\n", 677 | " batch_size=32,\n", 678 | " epochs=24,\n", 679 | " validation_data=(X_val, y_val), callbacks=[early_stopping, history])\n", 680 | "\n", 681 | "\n", 682 | "##############################\n", 683 | "# Save Model and Plots\n", 684 | "##############################\n", 685 | "model.save('C:\\\\glove\\\\StockText_3Level3EvenClass_modelNov2_8pm.h5')\n", 686 | " \n", 687 | "import matplotlib.pyplot as plt \n", 688 | "plt.figure(1) \n", 689 | "\n", 690 | "# summarize history for accuracy \n", 691 | "plt.subplot(211) \n", 692 | "plt.plot(history.history['acc']) \n", 693 | "plt.plot(history.history['val_acc']) \n", 694 | "plt.title('model accuracy') \n", 695 | "plt.ylabel('accuracy') \n", 696 | "plt.xlabel('epoch') \n", 697 | "plt.legend(['train', 'test'], loc='upper left') \n", 698 | " \n", 699 | "# summarize history for loss \n", 700 | "plt.subplot(212) \n", 701 | "plt.plot(history.history['loss']) \n", 702 | "plt.plot(history.history['val_loss']) \n", 703 | "plt.title('model loss') \n", 704 | "plt.ylabel('loss') \n", 705 | "plt.xlabel('epoch') \n", 706 | "plt.legend(['train', 'test'], loc='upper left') \n", 707 | "plt.show() \n", 708 | "\n", 709 | "#plot_model(model, to_file='C:\\\\glove\\stocktext_model3class.png')\n", 710 | "\n", 711 | "#from IPython.display import SVG\n", 712 | "#from keras.utils.vis_utils import model_to_dot\n", 713 | "#SVG(model_to_dot(model).create(prog='dot', format='svg'))\n", 714 | "\n", 715 | "##############################\n", 716 | "# More helpful links\n", 717 | "##############################\n", 718 | "\n", 719 | "#We can also test how well we would have performed by not using pre-trained word embeddings, \n", 720 | "#but instead initializing our Embedding layer from scratch and learning its weights during training. \n", 721 | "\n", 722 | "#https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html\n", 723 | "#https://arxiv.org/abs/1603.03827 \n", 724 | "#https://nlp.stanford.edu/projects/glove/ \n", 725 | "#https://stackoverflow.com/questions/37793118/load-pretrained-glove-vectors-in-python\n", 726 | "#https://stackoverflow.com/questions/27139908/load-precomputed-vectors-gensim?rq=1\n", 727 | "#https://stackoverflow.com/questions/14415741/numpy-array-vs-asarray" 728 | ] 729 | }, 730 | { 731 | "cell_type": "code", 732 | "execution_count": null, 733 | "metadata": { 734 | "scrolled": true 735 | }, 736 | "outputs": [], 737 | "source": [] 738 | } 739 | ], 740 | "metadata": { 741 | "kernelspec": { 742 | "display_name": "Python 3", 743 | "language": "python", 744 | "name": "python3" 745 | }, 746 | "language_info": { 747 | "codemirror_mode": { 748 | "name": "ipython", 749 | "version": 3 750 | }, 751 | "file_extension": ".py", 752 | "mimetype": "text/x-python", 753 | "name": "python", 754 | "nbconvert_exporter": "python", 755 | "pygments_lexer": "ipython3", 756 | "version": "3.5.2" 757 | } 758 | }, 759 | "nbformat": 4, 760 | "nbformat_minor": 2 761 | } 762 | -------------------------------------------------------------------------------- /JupyterNotebooks/Step 4 - Model Evaluation 3 Classes Reporting.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "Using Theano backend.\n", 13 | "C:\\Users\\pattyry\\AppData\\local\\AmlWorkbench\\Python\\lib\\site-packages\\gensim\\utils.py:862: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n", 14 | " warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n" 15 | ] 16 | } 17 | ], 18 | "source": [ 19 | "#############################################################################################################################################\n", 20 | "#\n", 21 | "# Stock future performance classification based on text\n", 22 | "# Evaluation of Model \n", 23 | "###############################################################################################################################################\n", 24 | "\n", 25 | "# import libraries\n", 26 | "from __future__ import print_function\n", 27 | "import numpy as np\n", 28 | "from six.moves import zip\n", 29 | "import json\n", 30 | "import warnings\n", 31 | "import pandas as pd\n", 32 | "from pandas import DataFrame \n", 33 | "import pickle\n", 34 | "import re\n", 35 | "import sys \n", 36 | "import azureml\n", 37 | "import string\n", 38 | "from scipy import stats\n", 39 | "import pip\n", 40 | "import keras\n", 41 | "from keras.preprocessing.sequence import pad_sequences\n", 42 | "from keras.utils.np_utils import to_categorical\n", 43 | "from keras.preprocessing.text import Tokenizer \n", 44 | "from keras.preprocessing import sequence\n", 45 | "from keras.models import Sequential\n", 46 | "from keras.layers import Dense, Embedding\n", 47 | "from keras.layers.core import Dropout\n", 48 | "from keras.layers import LSTM\n", 49 | "from keras.layers import Dense, Input, Flatten \n", 50 | "from keras.layers import Conv1D, MaxPooling1D, Embedding \n", 51 | "from keras.models import Model \n", 52 | "from keras.utils import plot_model\n", 53 | "from keras.utils.vis_utils import model_to_dot\n", 54 | "from keras.models import load_model\n", 55 | "from keras.applications import imagenet_utils\n", 56 | "from keras.layers import Embedding\n", 57 | "from keras.utils.np_utils import to_categorical\n", 58 | "from keras import initializers\n", 59 | "from keras import optimizers\n", 60 | "from keras.callbacks import EarlyStopping\n", 61 | "from keras.layers.normalization import BatchNormalization\n", 62 | "from keras.layers import initializers \n", 63 | "from keras.layers import regularizers \n", 64 | "from keras.layers import constraints \n", 65 | "from keras.layers.advanced_activations import PReLU\n", 66 | "from keras.layers.advanced_activations import LeakyReLU\n", 67 | "from keras.constraints import max_norm\n", 68 | "import keras.backend as K\n", 69 | "import os\n", 70 | "import tempfile \n", 71 | "import logging\n", 72 | "import gensim\n", 73 | "from gensim.models import Phrases, phrases\n", 74 | "from gensim.models.phrases import Phraser\n", 75 | "from gensim.models import Word2Vec as wv\n", 76 | "import nltk\n", 77 | "from nltk.corpus import stopwords\n", 78 | "from gensim import corpora, models, similarities\n", 79 | "from IPython.display import SVG\n", 80 | "import cloudpickle\n", 81 | "import csv\n", 82 | "import mkl\n", 83 | "import matplotlib.pyplot as plt\n", 84 | "import h5py\n", 85 | "from keras.models import load_model\n", 86 | "import re\n", 87 | "import io\n", 88 | "from os.path import dirname, join\n", 89 | "import regex\n", 90 | "import graphviz\n", 91 | "import pydotplus\n", 92 | "import pyparsing\n", 93 | "from keras.utils import plot_model\n", 94 | "\n", 95 | "\n" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 2, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "\n", 105 | "####################################################\n", 106 | "# Set Global Vars\n", 107 | "####################################################\n", 108 | "\n", 109 | "MAX_SEQUENCE_LENGTH = 10000\n", 110 | "MAX_NB_WORDS = 400000\n", 111 | "EMBEDDING_DIM = 300\n", 112 | "VALIDATION_SPLIT = 0.2\n", 113 | "BATCH_SIZE = 32\n", 114 | "\n", 115 | "np.random.seed(2032)" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 3, 121 | "metadata": {}, 122 | "outputs": [ 123 | { 124 | "name": "stdout", 125 | "output_type": "stream", 126 | "text": [ 127 | "shape of y_val (183, 3)\n", 128 | "shape of X_val (183, 10000)\n", 129 | "yval data type int32\n", 130 | "xval data type int32\n", 131 | "[array([0, 0, 1]), array([0, 0, 1])]\n", 132 | "[array([ 0, 0, 0, ..., 1063, 278, 1612])]\n" 133 | ] 134 | } 135 | ], 136 | "source": [ 137 | "##################################################\n", 138 | "#load in train and test set\n", 139 | "##################################################\n", 140 | "import os\n", 141 | "os.chdir('C:\\\\glove\\\\nextagenda\\\\24epoch')\n", 142 | "\n", 143 | "#double check\n", 144 | "y_val = np.loadtxt(\"y_val_3bin.txt\", delimiter=',', dtype = int) # use ';' as column separator instead of whitespace\n", 145 | "X_val = np.loadtxt(\"X_val_3bin.txt\", delimiter=',', dtype = int) \n", 146 | "\n", 147 | "print('shape of y_val',y_val.shape)\n", 148 | "print('shape of X_val',X_val.shape)\n", 149 | "\n", 150 | "print('yval data type',y_val.dtype)\n", 151 | "print('xval data type',X_val.dtype)\n", 152 | " \n", 153 | "from itertools import islice\n", 154 | "head = list(islice(y_val, 2))\n", 155 | "print(head)\n", 156 | "\n", 157 | "from itertools import islice\n", 158 | "head = list(islice(X_val, 1))\n", 159 | "print(head)\n", 160 | "#Reformat label content\n", 161 | "\n", 162 | "#labels = to_categorical(np.asarray(labels))" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 4, 168 | "metadata": {}, 169 | "outputs": [ 170 | { 171 | "name": "stdout", 172 | "output_type": "stream", 173 | "text": [ 174 | "model loaded\n" 175 | ] 176 | } 177 | ], 178 | "source": [ 179 | "#######################\n", 180 | "# Load Model\n", 181 | "#######################\n", 182 | "\n", 183 | "model = load_model('C:\\\\glove\\\\nextagenda\\\\24epoch\\\\StockText_3Level3EvenClass_modelNov2_8pm.h5') \n", 184 | "history = model\n", 185 | "print('model loaded')" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 5, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "def predict_classes(self, x, verbose=1):\n", 195 | " if proba.shape[-1] > 1:\n", 196 | " return proba.argmax(axis=-1)\n", 197 | " else:\n", 198 | " return (proba > 0.5).astype('int32')\n", 199 | "type(X_val)\n", 200 | "\n", 201 | "prediction = model.predict(np.array(X_val))\n", 202 | "#predict_class = predict_classes(prediction)\n", 203 | "\n", 204 | "#https://stackoverflow.com/questions/37891954/keras-how-do-i-predict-after-i-trained-a-model\n", 205 | "#model.predict() expects the first parameter to be a numpy array. You supply a list, which does not have the shape attribute a numpy array has.\n", 206 | "#prediction = model.predict(np.array(test_seq))" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": 6, 212 | "metadata": {}, 213 | "outputs": [ 214 | { 215 | "name": "stdout", 216 | "output_type": "stream", 217 | "text": [ 218 | "model loaded\n", 219 | "183/183 [==============================] - 30s \n" 220 | ] 221 | } 222 | ], 223 | "source": [ 224 | "prediction.shape\n", 225 | "history = model\n", 226 | "print('model loaded')\n", 227 | "score, acc = model.evaluate(X_val, y_val, batch_size=50)\n" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 7, 233 | "metadata": {}, 234 | "outputs": [ 235 | { 236 | "name": "stdout", 237 | "output_type": "stream", 238 | "text": [ 239 | "Test score: 1.03387291249\n", 240 | "Test accuracy: 0.502732243369\n" 241 | ] 242 | } 243 | ], 244 | "source": [ 245 | "print('Test score:', score)\n", 246 | "print('Test accuracy:', acc)" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 8, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "name": "stdout", 256 | "output_type": "stream", 257 | "text": [ 258 | "_________________________________________________________________\n", 259 | "Layer (type) Output Shape Param # \n", 260 | "=================================================================\n", 261 | "input_1 (InputLayer) (None, 10000) 0 \n", 262 | "_________________________________________________________________\n", 263 | "embedding_1 (Embedding) (None, 10000, 300) 7166400 \n", 264 | "_________________________________________________________________\n", 265 | "conv1d_1 (Conv1D) (None, 9996, 128) 192128 \n", 266 | "_________________________________________________________________\n", 267 | "max_pooling1d_1 (MaxPooling1 (None, 1999, 128) 0 \n", 268 | "_________________________________________________________________\n", 269 | "dropout_1 (Dropout) (None, 1999, 128) 0 \n", 270 | "_________________________________________________________________\n", 271 | "conv1d_2 (Conv1D) (None, 1995, 128) 82048 \n", 272 | "_________________________________________________________________\n", 273 | "max_pooling1d_2 (MaxPooling1 (None, 399, 128) 0 \n", 274 | "_________________________________________________________________\n", 275 | "dropout_2 (Dropout) (None, 399, 128) 0 \n", 276 | "_________________________________________________________________\n", 277 | "conv1d_3 (Conv1D) (None, 395, 128) 82048 \n", 278 | "_________________________________________________________________\n", 279 | "max_pooling1d_3 (MaxPooling1 (None, 11, 128) 0 \n", 280 | "_________________________________________________________________\n", 281 | "flatten_1 (Flatten) (None, 1408) 0 \n", 282 | "_________________________________________________________________\n", 283 | "dense_1 (Dense) (None, 100) 140900 \n", 284 | "_________________________________________________________________\n", 285 | "dropout_3 (Dropout) (None, 100) 0 \n", 286 | "_________________________________________________________________\n", 287 | "dense_2 (Dense) (None, 3) 303 \n", 288 | "=================================================================\n", 289 | "Total params: 7,663,827\n", 290 | "Trainable params: 497,427\n", 291 | "Non-trainable params: 7,166,400\n", 292 | "_________________________________________________________________\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "#######################################\n", 298 | "# Plot the model summary\n", 299 | "#######################################\n", 300 | "\n", 301 | "from keras.models import load_model\n", 302 | "# returns a compiled model\n", 303 | "# identical to the previous one\n", 304 | "\n", 305 | "model.summary()\n" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": 9, 311 | "metadata": {}, 312 | "outputs": [ 313 | { 314 | "name": "stdout", 315 | "output_type": "stream", 316 | "text": [ 317 | "Prediction [[ 0.60299897 0.02783478 0.36916623]\n", 318 | " [ 0.46770996 0.01239048 0.51989955]\n", 319 | " [ 0.46939662 0.12456405 0.40603933]\n", 320 | " [ 0.0628856 0.8633914 0.07372297]\n", 321 | " [ 0.70326638 0.05797603 0.23875761]\n", 322 | " [ 0.59860665 0.0770886 0.32430479]\n", 323 | " [ 0.37346879 0.08498149 0.54154974]\n", 324 | " [ 0.17527041 0.62540859 0.199321 ]\n", 325 | " [ 0.4665738 0.13735861 0.39606759]\n", 326 | " [ 0.08514553 0.79618055 0.11867391]\n", 327 | " [ 0.42242661 0.25151208 0.32606131]\n", 328 | " [ 0.12611417 0.72044301 0.15344279]\n", 329 | " [ 0.14903413 0.73043209 0.12053379]\n", 330 | " [ 0.39039862 0.10495906 0.50464231]\n", 331 | " [ 0.35217601 0.30299756 0.34482643]\n", 332 | " [ 0.4665738 0.13735861 0.39606759]\n", 333 | " [ 0.46770909 0.02654023 0.50575066]\n", 334 | " [ 0.34450069 0.28776178 0.36773756]\n", 335 | " [ 0.43264279 0.02500514 0.54235208]\n", 336 | " [ 0.07004065 0.82702613 0.10293324]\n", 337 | " [ 0.46179762 0.08026155 0.45794082]\n", 338 | " [ 0.51733589 0.05633128 0.42633283]\n", 339 | " [ 0.57519263 0.07393355 0.35087383]\n", 340 | " [ 0.44667706 0.17529279 0.37803015]\n", 341 | " [ 0.11937422 0.70633811 0.17428765]\n", 342 | " [ 0.40361729 0.14150363 0.45487908]\n", 343 | " [ 0.21613279 0.4208065 0.36306071]\n", 344 | " [ 0.4665738 0.13735861 0.39606759]\n", 345 | " [ 0.452645 0.04496452 0.5023905 ]\n", 346 | " [ 0.45421156 0.06204712 0.48374131]\n", 347 | " [ 0.44364905 0.03718776 0.51916319]\n", 348 | " [ 0.29144272 0.32931802 0.37923926]\n", 349 | " [ 0.46277821 0.01837069 0.5188511 ]\n", 350 | " [ 0.48267859 0.03770601 0.47961539]\n", 351 | " [ 0.53030503 0.1514855 0.31820947]\n", 352 | " [ 0.23063813 0.51607746 0.25328439]\n", 353 | " [ 0.34898257 0.29123628 0.35978115]\n", 354 | " [ 0.54017258 0.02036776 0.43945965]\n", 355 | " [ 0.30582371 0.31297088 0.38120538]\n", 356 | " [ 0.20344214 0.40226004 0.39429784]\n", 357 | " [ 0.43161568 0.12716676 0.44121757]\n", 358 | " [ 0.47095811 0.12117136 0.40787053]\n", 359 | " [ 0.42091623 0.19208066 0.38700312]\n", 360 | " [ 0.58862025 0.07601111 0.33536863]\n", 361 | " [ 0.4657554 0.16524281 0.36900178]\n", 362 | " [ 0.49727497 0.08295182 0.41977322]\n", 363 | " [ 0.10306041 0.76639616 0.13054346]\n", 364 | " [ 0.31572932 0.47703689 0.2072338 ]\n", 365 | " [ 0.41081586 0.11093593 0.47824821]\n", 366 | " [ 0.47522232 0.12616254 0.39861515]\n", 367 | " [ 0.5275988 0.09859334 0.37380788]\n", 368 | " [ 0.41266727 0.08510612 0.50222659]\n", 369 | " [ 0.37701482 0.05200946 0.57097572]\n", 370 | " [ 0.46554187 0.16730174 0.36715639]\n", 371 | " [ 0.18800572 0.60255551 0.20943879]\n", 372 | " [ 0.16422257 0.63237506 0.20340236]\n", 373 | " [ 0.28420705 0.09384562 0.62194735]\n", 374 | " [ 0.42901951 0.15547951 0.41550097]\n", 375 | " [ 0.46446392 0.03332587 0.5022102 ]\n", 376 | " [ 0.15007201 0.65483403 0.19509399]\n", 377 | " [ 0.26363406 0.33644333 0.39992261]\n", 378 | " [ 0.4665738 0.13735861 0.39606759]\n", 379 | " [ 0.58551508 0.0454276 0.3690573 ]\n", 380 | " [ 0.4665738 0.13735861 0.39606759]\n", 381 | " [ 0.4665738 0.13735861 0.39606759]\n", 382 | " [ 0.4665738 0.13735861 0.39606759]\n", 383 | " [ 0.31754553 0.1548299 0.52762455]\n", 384 | " [ 0.3646346 0.13597551 0.49938992]\n", 385 | " [ 0.44396541 0.15561369 0.4004209 ]\n", 386 | " [ 0.53981853 0.12262181 0.33755967]\n", 387 | " [ 0.4665738 0.13735861 0.39606759]\n", 388 | " [ 0.31311208 0.33908516 0.34780276]\n", 389 | " [ 0.3019056 0.30487448 0.39321992]\n", 390 | " [ 0.48491621 0.01837749 0.49670628]\n", 391 | " [ 0.49474895 0.13655685 0.36869419]\n", 392 | " [ 0.48935714 0.07876917 0.43187368]\n", 393 | " [ 0.35819802 0.16399305 0.47780895]\n", 394 | " [ 0.46246624 0.08740985 0.45012391]\n", 395 | " [ 0.12286267 0.74513632 0.132001 ]\n", 396 | " [ 0.33868682 0.2734147 0.38789847]\n", 397 | " [ 0.37417278 0.34339771 0.28242952]\n", 398 | " [ 0.26085693 0.45219672 0.28694636]\n", 399 | " [ 0.4665738 0.13735861 0.39606759]\n", 400 | " [ 0.4665738 0.13735861 0.39606759]\n", 401 | " [ 0.4665738 0.13735861 0.39606759]\n", 402 | " [ 0.05054875 0.85916674 0.09028453]\n", 403 | " [ 0.23007289 0.45616841 0.3137587 ]\n", 404 | " [ 0.4665738 0.13735861 0.39606759]\n", 405 | " [ 0.42563197 0.1963986 0.37796941]\n", 406 | " [ 0.47333771 0.09401057 0.4326517 ]\n", 407 | " [ 0.5491783 0.02041293 0.43040875]\n", 408 | " [ 0.30602634 0.21317507 0.4807986 ]\n", 409 | " [ 0.59495962 0.12993681 0.27510354]\n", 410 | " [ 0.37515137 0.09291959 0.53192902]\n", 411 | " [ 0.50698256 0.05204119 0.44097623]\n", 412 | " [ 0.39413545 0.09537414 0.51049042]\n", 413 | " [ 0.32044554 0.13157222 0.54798228]\n", 414 | " [ 0.22032522 0.54409868 0.23557611]\n", 415 | " [ 0.4665738 0.13735861 0.39606759]\n", 416 | " [ 0.46521366 0.13864134 0.39614502]\n", 417 | " [ 0.50945681 0.02989419 0.46064898]\n", 418 | " [ 0.44190782 0.12995666 0.42813551]\n", 419 | " [ 0.24478719 0.51695919 0.23825359]\n", 420 | " [ 0.30041552 0.15919617 0.54038835]\n", 421 | " [ 0.5802905 0.02489477 0.39481473]\n", 422 | " [ 0.29458958 0.2913017 0.41410872]\n", 423 | " [ 0.4665738 0.13735861 0.39606759]\n", 424 | " [ 0.34216848 0.08232628 0.57550526]\n", 425 | " [ 0.25935474 0.43187302 0.30877224]\n", 426 | " [ 0.23491099 0.26621357 0.49887544]\n", 427 | " [ 0.21168354 0.56294787 0.22536859]\n", 428 | " [ 0.25282598 0.48996717 0.25720689]\n", 429 | " [ 0.4665738 0.13735861 0.39606759]\n", 430 | " [ 0.01431965 0.96704686 0.01863352]\n", 431 | " [ 0.46099803 0.158087 0.38091496]\n", 432 | " [ 0.22340927 0.54999691 0.22659381]\n", 433 | " [ 0.09754586 0.80863082 0.09382334]\n", 434 | " [ 0.4665738 0.13735861 0.39606759]\n", 435 | " [ 0.23208055 0.35675776 0.41116169]\n", 436 | " [ 0.42301393 0.10305042 0.47393566]\n", 437 | " [ 0.35002917 0.24552996 0.40444085]\n", 438 | " [ 0.31192398 0.07734481 0.61073118]\n", 439 | " [ 0.38423079 0.05041759 0.56535161]\n", 440 | " [ 0.03353249 0.90471816 0.06174935]\n", 441 | " [ 0.05659912 0.88762707 0.05577382]\n", 442 | " [ 0.235851 0.50962126 0.25452775]\n", 443 | " [ 0.38930121 0.05944649 0.55125231]\n", 444 | " [ 0.4665738 0.13735861 0.39606759]\n", 445 | " [ 0.29538968 0.41019636 0.29441392]\n", 446 | " [ 0.4665738 0.13735861 0.39606759]\n", 447 | " [ 0.26269528 0.25173974 0.48556498]\n", 448 | " [ 0.43715301 0.04330873 0.51953822]\n", 449 | " [ 0.4379856 0.18162702 0.38038737]\n", 450 | " [ 0.47882476 0.04107263 0.4801026 ]\n", 451 | " [ 0.57012689 0.02440714 0.40546593]\n", 452 | " [ 0.44706309 0.03195711 0.52097982]\n", 453 | " [ 0.41668108 0.19445698 0.38886195]\n", 454 | " [ 0.34573105 0.18963075 0.4646382 ]\n", 455 | " [ 0.33697206 0.13795936 0.52506858]\n", 456 | " [ 0.4840053 0.10219728 0.41379741]\n", 457 | " [ 0.43246323 0.13964675 0.42789 ]\n", 458 | " [ 0.53119338 0.06646411 0.40234253]\n", 459 | " [ 0.58462 0.06153217 0.3538478 ]\n", 460 | " [ 0.03810813 0.90610737 0.05578449]\n", 461 | " [ 0.33561036 0.13042352 0.53396612]\n", 462 | " [ 0.62929219 0.07593569 0.29477212]\n", 463 | " [ 0.4665738 0.13735861 0.39606759]\n", 464 | " [ 0.4665738 0.13735861 0.39606759]\n", 465 | " [ 0.4673616 0.1545091 0.37812927]\n", 466 | " [ 0.36537096 0.24865231 0.38597673]\n", 467 | " [ 0.21997865 0.41420817 0.36581317]\n", 468 | " [ 0.61102688 0.06276071 0.32621244]\n", 469 | " [ 0.32390884 0.24842249 0.42766869]\n", 470 | " [ 0.4665738 0.13735861 0.39606759]\n", 471 | " [ 0.50885373 0.13504741 0.35609886]\n", 472 | " [ 0.49062636 0.18852042 0.3208532 ]\n", 473 | " [ 0.33956149 0.09712241 0.56331611]\n", 474 | " [ 0.21786022 0.22522183 0.55691797]\n", 475 | " [ 0.13955827 0.60070211 0.25973961]\n", 476 | " [ 0.23732822 0.43911263 0.32355917]\n", 477 | " [ 0.5778203 0.01443868 0.40774104]\n", 478 | " [ 0.4665738 0.13735861 0.39606759]\n", 479 | " [ 0.12884988 0.76318592 0.10796422]\n", 480 | " [ 0.32109544 0.07848836 0.60041618]\n", 481 | " [ 0.41925964 0.20532952 0.37541085]\n", 482 | " [ 0.31256393 0.16301358 0.52442253]\n", 483 | " [ 0.37799886 0.08057527 0.54142588]\n", 484 | " [ 0.35097313 0.09822193 0.55080491]\n", 485 | " [ 0.36460248 0.0671491 0.56824839]\n", 486 | " [ 0.49066481 0.03813173 0.47120348]\n", 487 | " [ 0.4184173 0.04254701 0.53903568]\n", 488 | " [ 0.44721672 0.14903522 0.40374804]\n", 489 | " [ 0.23512945 0.21657352 0.54829705]\n", 490 | " [ 0.4665738 0.13735861 0.39606759]\n", 491 | " [ 0.44349152 0.18796439 0.3685441 ]\n", 492 | " [ 0.5419786 0.02103475 0.43698663]\n", 493 | " [ 0.55246741 0.03337465 0.41415793]\n", 494 | " [ 0.49978092 0.03294394 0.46727511]\n", 495 | " [ 0.45043409 0.18216302 0.36740291]\n", 496 | " [ 0.46574453 0.13973719 0.39451832]\n", 497 | " [ 0.20662741 0.49180555 0.30156702]\n", 498 | " [ 0.08763801 0.80226952 0.11009248]\n", 499 | " [ 0.17849751 0.46278679 0.35871568]]\n", 500 | "Labels [[0 0 1]\n", 501 | " [0 0 1]\n", 502 | " [0 0 1]\n", 503 | " [0 1 0]\n", 504 | " [0 1 0]\n", 505 | " [0 1 0]\n", 506 | " [1 0 0]\n", 507 | " [0 1 0]\n", 508 | " [0 1 0]\n", 509 | " [0 1 0]\n", 510 | " [1 0 0]\n", 511 | " [0 1 0]\n", 512 | " [0 1 0]\n", 513 | " [0 1 0]\n", 514 | " [1 0 0]\n", 515 | " [0 0 1]\n", 516 | " [0 0 1]\n", 517 | " [1 0 0]\n", 518 | " [0 0 1]\n", 519 | " [0 1 0]\n", 520 | " [1 0 0]\n", 521 | " [1 0 0]\n", 522 | " [1 0 0]\n", 523 | " [1 0 0]\n", 524 | " [0 1 0]\n", 525 | " [0 0 1]\n", 526 | " [0 0 1]\n", 527 | " [0 0 1]\n", 528 | " [0 0 1]\n", 529 | " [1 0 0]\n", 530 | " [0 1 0]\n", 531 | " [0 0 1]\n", 532 | " [1 0 0]\n", 533 | " [1 0 0]\n", 534 | " [1 0 0]\n", 535 | " [0 1 0]\n", 536 | " [0 1 0]\n", 537 | " [1 0 0]\n", 538 | " [1 0 0]\n", 539 | " [0 1 0]\n", 540 | " [0 0 1]\n", 541 | " [1 0 0]\n", 542 | " [1 0 0]\n", 543 | " [1 0 0]\n", 544 | " [0 0 1]\n", 545 | " [0 0 1]\n", 546 | " [0 1 0]\n", 547 | " [0 1 0]\n", 548 | " [0 0 1]\n", 549 | " [1 0 0]\n", 550 | " [1 0 0]\n", 551 | " [0 0 1]\n", 552 | " [1 0 0]\n", 553 | " [0 0 1]\n", 554 | " [0 1 0]\n", 555 | " [0 0 1]\n", 556 | " [1 0 0]\n", 557 | " [1 0 0]\n", 558 | " [0 0 1]\n", 559 | " [1 0 0]\n", 560 | " [0 0 1]\n", 561 | " [0 1 0]\n", 562 | " [1 0 0]\n", 563 | " [1 0 0]\n", 564 | " [0 0 1]\n", 565 | " [1 0 0]\n", 566 | " [0 1 0]\n", 567 | " [0 0 1]\n", 568 | " [1 0 0]\n", 569 | " [0 0 1]\n", 570 | " [0 0 1]\n", 571 | " [0 1 0]\n", 572 | " [0 1 0]\n", 573 | " [0 0 1]\n", 574 | " [0 0 1]\n", 575 | " [1 0 0]\n", 576 | " [0 0 1]\n", 577 | " [0 0 1]\n", 578 | " [0 1 0]\n", 579 | " [1 0 0]\n", 580 | " [1 0 0]\n", 581 | " [0 0 1]\n", 582 | " [0 1 0]\n", 583 | " [0 0 1]\n", 584 | " [0 1 0]\n", 585 | " [0 1 0]\n", 586 | " [0 0 1]\n", 587 | " [0 0 1]\n", 588 | " [0 1 0]\n", 589 | " [0 0 1]\n", 590 | " [0 0 1]\n", 591 | " [1 0 0]\n", 592 | " [0 0 1]\n", 593 | " [1 0 0]\n", 594 | " [0 1 0]\n", 595 | " [1 0 0]\n", 596 | " [0 0 1]\n", 597 | " [0 1 0]\n", 598 | " [0 0 1]\n", 599 | " [0 1 0]\n", 600 | " [1 0 0]\n", 601 | " [1 0 0]\n", 602 | " [0 1 0]\n", 603 | " [1 0 0]\n", 604 | " [1 0 0]\n", 605 | " [1 0 0]\n", 606 | " [0 1 0]\n", 607 | " [0 0 1]\n", 608 | " [0 1 0]\n", 609 | " [0 0 1]\n", 610 | " [0 1 0]\n", 611 | " [0 1 0]\n", 612 | " [1 0 0]\n", 613 | " [0 1 0]\n", 614 | " [0 1 0]\n", 615 | " [0 0 1]\n", 616 | " [0 1 0]\n", 617 | " [1 0 0]\n", 618 | " [1 0 0]\n", 619 | " [0 1 0]\n", 620 | " [0 0 1]\n", 621 | " [0 1 0]\n", 622 | " [0 0 1]\n", 623 | " [0 1 0]\n", 624 | " [0 1 0]\n", 625 | " [0 1 0]\n", 626 | " [0 1 0]\n", 627 | " [0 1 0]\n", 628 | " [0 1 0]\n", 629 | " [0 0 1]\n", 630 | " [0 0 1]\n", 631 | " [1 0 0]\n", 632 | " [0 1 0]\n", 633 | " [0 1 0]\n", 634 | " [0 0 1]\n", 635 | " [0 1 0]\n", 636 | " [0 0 1]\n", 637 | " [0 1 0]\n", 638 | " [0 0 1]\n", 639 | " [1 0 0]\n", 640 | " [0 1 0]\n", 641 | " [1 0 0]\n", 642 | " [1 0 0]\n", 643 | " [0 1 0]\n", 644 | " [1 0 0]\n", 645 | " [0 1 0]\n", 646 | " [0 1 0]\n", 647 | " [1 0 0]\n", 648 | " [0 0 1]\n", 649 | " [0 0 1]\n", 650 | " [0 0 1]\n", 651 | " [1 0 0]\n", 652 | " [0 1 0]\n", 653 | " [1 0 0]\n", 654 | " [1 0 0]\n", 655 | " [0 0 1]\n", 656 | " [1 0 0]\n", 657 | " [0 0 1]\n", 658 | " [1 0 0]\n", 659 | " [1 0 0]\n", 660 | " [0 0 1]\n", 661 | " [1 0 0]\n", 662 | " [0 1 0]\n", 663 | " [1 0 0]\n", 664 | " [1 0 0]\n", 665 | " [0 1 0]\n", 666 | " [0 0 1]\n", 667 | " [0 1 0]\n", 668 | " [1 0 0]\n", 669 | " [1 0 0]\n", 670 | " [1 0 0]\n", 671 | " [0 0 1]\n", 672 | " [0 0 1]\n", 673 | " [0 0 1]\n", 674 | " [1 0 0]\n", 675 | " [1 0 0]\n", 676 | " [0 0 1]\n", 677 | " [0 0 1]\n", 678 | " [0 1 0]\n", 679 | " [0 1 0]\n", 680 | " [0 0 1]\n", 681 | " [0 1 0]\n", 682 | " [0 1 0]]\n" 683 | ] 684 | } 685 | ], 686 | "source": [ 687 | "print('Prediction',prediction)\n", 688 | "print('Labels',y_val)" 689 | ] 690 | }, 691 | { 692 | "cell_type": "code", 693 | "execution_count": 10, 694 | "metadata": {}, 695 | "outputs": [ 696 | { 697 | "name": "stdout", 698 | "output_type": "stream", 699 | "text": [ 700 | "(183, 10000)\n", 701 | "(183, 3)\n" 702 | ] 703 | } 704 | ], 705 | "source": [ 706 | "from keras.utils import np_utils\n", 707 | "print(X_val.shape)\n", 708 | "print(y_val.shape)" 709 | ] 710 | }, 711 | { 712 | "cell_type": "code", 713 | "execution_count": 11, 714 | "metadata": {}, 715 | "outputs": [], 716 | "source": [ 717 | "#create np array with assigned predicted classes\n", 718 | "def predict_classes(self, x, verbose=1):\n", 719 | " proba = self.predict(x, verbose=verbose)\n", 720 | " if proba.shape[-1] > 1:\n", 721 | " return proba.argmax(axis=-1)\n", 722 | " else:\n", 723 | " return (proba > 0.5).astype('int32')\n", 724 | "\n", 725 | "def probas_to_classes(prediction):\n", 726 | " if len(prediction.shape) > 1 and prediction.shape[1] > 1:\n", 727 | " return categorical_probas_to_classes(prediction)\n", 728 | " return np.array([1 if p > 0.5 else 0 for p in prediction])\n", 729 | "\n", 730 | "def categorical_probas_to_classes(prediction):\n", 731 | " return np.argmax(prediction, axis=1)\n", 732 | "\n", 733 | "def labels_to_classes(y_val):\n", 734 | " if len(y_val.shape) > 1 and y_val.shape[1] > 1:\n", 735 | " return categorical_labels_to_classes(y_val)\n", 736 | " return np.array([1 if p > 0.5 else 0 for p in y_val])\n", 737 | "\n", 738 | "def categorical_labels_to_classes(y_val):\n", 739 | " return np.argmax(y_val, axis=1)" 740 | ] 741 | }, 742 | { 743 | "cell_type": "code", 744 | "execution_count": 12, 745 | "metadata": {}, 746 | "outputs": [ 747 | { 748 | "name": "stdout", 749 | "output_type": "stream", 750 | "text": [ 751 | "[0 2 0 1 0 0 2 1 0 1 0 1 1 2 0 0 2 2 2 1 0 0 0 0 1 2 1 0 2 2 2 2 2 0 0 1 2\n", 752 | " 0 2 1 2 0 0 0 0 0 1 1 2 0 0 2 2 0 1 1 2 0 2 1 2 0 0 0 0 0 2 2 0 0 0 2 2 2\n", 753 | " 0 0 2 0 1 2 0 1 0 0 0 1 1 0 0 0 0 2 0 2 0 2 2 1 0 0 0 0 1 2 0 2 0 2 1 2 1\n", 754 | " 1 0 1 0 1 1 0 2 2 2 2 2 1 1 1 2 0 1 0 2 2 0 2 0 2 0 2 2 0 0 0 0 1 2 0 0 0\n", 755 | " 0 2 1 0 2 0 0 0 2 2 1 1 0 0 1 2 0 2 2 2 2 0 2 0 2 0 0 0 0 0 0 0 1 1 1]\n", 756 | "[2 2 2 1 1 1 0 1 1 1 0 1 1 1 0 2 2 0 2 1 0 0 0 0 1 2 2 2 2 0 1 2 0 0 0 1 1\n", 757 | " 0 0 1 2 0 0 0 2 2 1 1 2 0 0 2 0 2 1 2 0 0 2 0 2 1 0 0 2 0 1 2 0 2 2 1 1 2\n", 758 | " 2 0 2 2 1 0 0 2 1 2 1 1 2 2 1 2 2 0 2 0 1 0 2 1 2 1 0 0 1 0 0 0 1 2 1 2 1\n", 759 | " 1 0 1 1 2 1 0 0 1 2 1 2 1 1 1 1 1 1 2 2 0 1 1 2 1 2 1 2 0 1 0 0 1 0 1 1 0\n", 760 | " 2 2 2 0 1 0 0 2 0 2 0 0 2 0 1 0 0 1 2 1 0 0 0 2 2 2 0 0 2 2 1 1 2 1 1]\n" 761 | ] 762 | } 763 | ], 764 | "source": [ 765 | "from tempfile import TemporaryFile\n", 766 | "pred_classes=probas_to_classes(prediction)\n", 767 | "print(pred_classes)\n", 768 | "y_classes=labels_to_classes(y_val)\n", 769 | "print(y_classes)" 770 | ] 771 | }, 772 | { 773 | "cell_type": "code", 774 | "execution_count": 13, 775 | "metadata": {}, 776 | "outputs": [ 777 | { 778 | "name": "stdout", 779 | "output_type": "stream", 780 | "text": [ 781 | "Confusion matrix, without normalization\n", 782 | "[[38 3 20]\n", 783 | " [18 29 15]\n", 784 | " [28 7 25]]\n", 785 | "Normalized confusion matrix\n", 786 | "[[ 0.62 0.05 0.33]\n", 787 | " [ 0.29 0.47 0.24]\n", 788 | " [ 0.47 0.12 0.42]]\n" 789 | ] 790 | }, 791 | { 792 | "data": { 793 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEmCAYAAAD1FIKpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcFNW5//HPd4ZV2VRQEVEMKq4B\nd69LNO4xi8vVxH2JETXRX1ySqzHmxi0uidHcxCRGr17NgnGJRqIm7ooYdwQFERUURZBVZB9geH5/\nVA0248x0D3RP1cx837zqRXdV9amnu3qePnXq1ClFBGZmlqjKOgAzszxxUjQzK+CkaGZWwEnRzKyA\nk6KZWQEnRTOzAm0yKUrqKukfkj6VdM8alHO8pEfLGVtWJO0taUJetidpgKSQ1KGlYmotJL0v6YD0\n8cWS/rcC27hJ0k/KXW5boCz7KUo6Djgf2AqYD4wGfhYRI9ew3BOBc4A9ImL5Ggeac5IC2CIi3s06\nlsZIeh/4TkQ8nj4fALwHdCz3PpJ0OzAlIi4pZ7ktpf5nVYbyTknL26sc5bV1mdUUJZ0P/Aq4CtgA\n2AT4HXBYGYrfFHi7PSTEUrg2Vjn+bNugiGjxCegJLACObmKdziRJc2o6/QronC7bF5gCXADMAKYB\np6bLLgOWAsvSbZwGXAr8uaDsAUAAHdLnpwCTSGqr7wHHF8wfWfC6PYCXgU/T//coWPY0cAXwXFrO\no0DvRt5bXfz/VRD/4cChwNvAHODigvV3BZ4H5qbr3gh0SpeNSN/LwvT9fqug/AuBj4E/1c1LXzMw\n3caO6fONgFnAviXsuzuAC9LH/dJtfzd9vnlarupt70/ACmBxGuN/FeyDk4EP0u3/uMT9v8p+SedF\nuv2h6b5fmm7rH428jwDOBN4BPgF+y2dHTlXAJcDkdP/8EehZ77tzWhr3iIJ5pwIfpuWdCewCvJ7u\ntxsLtj0QeBKYnb7vvwC9Cpa/DxyQPr6U9Lub7vcFBdNy4NJ02UXARJLv3pvAEen8rYElQG36mrnp\n/NuBKwu2eTrwbrr/hgMblfJZtcUpq6R4SLpDOzSxzuXAC8D6QB/g38AV6bJ909dfDnQkSSaLgHXq\nf5EaeV73Je4ArA3MAwaly/oC29b/4wPWTb8QJ6avOzZ9vl66/On0S7kl0DV9fk0j760u/v9O4z8d\nmAkMA7oD26Zf5C+k6+8E7J5udwAwHji3fkJooPxrSZJLVwqSVMEfwXhgLeAR4LoS9923SRMNcFz6\nnu8qWPZAQQyF23uf9A+93j64JY1vMFADbF3C/l+5Xxr6DKj3B9/I+wjgQaAXyVHKTOCQgvfxLvAF\noBtwH/CnenH/keS707Vg3k1AF+CgdP/9PY2/H0ly3SctY3PgwHTf9CFJrL9q6LOi3ne3YJ0hacw7\npM+PJvlxqyL5YVwI9G3i81r5GQH7kSTnHdOYfgOMKOWzaotTVofP6wGzounD2+OByyNiRkTMJKkB\nnliwfFm6fFlEPEzyKzhoNeNZAWwnqWtETIuIcQ2s81XgnYj4U0Qsj4g7gbeArxes838R8XZELAbu\nJvniNmYZSfvpMuCvQG/gfyJifrr9ccAXASLi1Yh4Id3u+8AfgH1KeE8/jYiaNJ5VRMQtJL/8L5L8\nEPy4SHl1ngH2llQFfAn4ObBnumyfdHlzXBYRiyNiDDCGJDlC8f1fDtdExNyI+AB4is/21/HA9REx\nKSIWAD8Cjql3qHxpRCys99leERFLIuJRkqR0Zxr/R8CzwA4AEfFuRDyW7puZwPUU358rSepDknDP\niYjX0jLviYipEbEiIu4i2be7lljk8cBtETEqImrS9/sfabtvncY+qzYnq6Q4G+hdpD1mI5LDlzqT\n03kry6iXVBeR/Ko3S0QsJPllPROYJukhSVuVEE9dTP0Knn/cjHhmR0Rt+rjuD2t6wfLFda+XtKWk\nByV9LGkeSTts7ybKBpgZEUuKrHMLsB3wm/SPoaiImEjyAzQE2JukBjFV0iBWLyk29pkV2//l0Jxt\ndyBp+67zYQPl1d9/je3P9SX9VdJH6f78M8X3J+lrOwL3AsMi4q8F80+SNFrSXElzSfZrSWVS7/2m\nPwSzWf3vdquWVVJ8nuTw4vAm1plKcsKkzibpvNWxkOQwsc6GhQsj4pGIOJCkxvQWSbIoFk9dTB+t\nZkzN8XuSuLaIiB7AxSTtdk1psluBpG4k7XS3ApdKWrcZ8TwDHEXSrvlR+vwkYB2SHgTNjqcBTe3/\nVfanpFX252psq5RtL2fVJLcm27g6ff0X0/15AsX3Z53fkLQbrjyzLmlTku/s2STNOb2AsQVlFot1\nlfcraW2So7mW+G7nTiZJMSI+JWlP+62kwyWtJamjpK9I+nm62p3AJZL6SOqdrv/n1dzkaOBLkjaR\n1JPk8AAASRtI+kb6RaghqQXVNlDGw8CWko6T1EHSt4BtSGpKldadpN1zQVqLPave8ukk7V/N8T/A\nqxHxHeAhkvYwACRdKunpJl77DMkf4Ij0+dMkXaBGFtR+62tujE3t/zHAtpKGSOpC0u62JttqaNvn\nSdos/fG4iqTdtFy9GbqTnvSQ1A/4YSkvknQGSW38uIhYUbBobZLENzNd71SSmmKd6cDGkjo1UvQw\n4NT08+xM8n5fTJtq2p3MuuRExPUkfRQvIdmZH5L8of09XeVK4BWSs3dvAKPSeauzrceAu9KyXmXV\nRFZFchZ7KsmZt32A7zZQxmzga+m6s0nOoH4tImatTkzN9AOSkxrzSWoEd9VbfilwR3ro9M1ihUk6\njORk15nprPOBHSUdnz7vT3IWvTHPkPxh1yXFkSQ1txGNviKpHV2SxviDYjHSxP6PiLdJTsQ8TtJ2\nVr9f663ANum2/k7z3UZyxnwESW+EJSRJv1wuIzmp8SnJD9J9Jb7uWJJkP1XSgnS6OCLeBH5JcgQ2\nHdieVfffkyRt1B9L+tz3NSKeAH4C/I2kd8NA4JjVeWNtQaadty2fJI0G9k9/CMzaFSdFM7MCbfLa\nZzOz1eWkaGZWwEnRzKxAri5mV4euoU7dsw6jVdlh602yDqHVmfJpsT7tVt+8GR+xeN4npfalLEl1\nj00jln/uYqtGxeKZj0TEIeWMoSH5SoqdutN5UNEeJVbguRdvzDqEVudHD7+VdQitzrDzjyp7mbF8\ncbP+3peM/m2pV+iskVwlRTNrTwTKXwuek6KZZUOAynpEXhZOimaWHdcUzczqCKqqsw7ic5wUzSw7\nPnw2M0sJHz6bmX1Grimama3CNUUzswKuKZqZ1XHnbTOzz7jztplZPa4pmpnVEVS787aZWcL9FM3M\n6slhm2L+0rSZtRPp2edSp6ZKkrpIeknSGEnjJF2Wzr9d0nuSRqfTkGJRuaZoZtkpX02xBtgvIhZI\n6giMlPTPdNkPI+LeUgtyUjSz7JSpTTGSezUvSJ92TKfVun+zD5/NLBtS8yboLemVgmnoqsWpWtJo\nYAbwWES8mC76maTXJd0gqXOxsFxTNLPsNK+mOCsidm5sYUTUAkMk9QLul7Qd8CPgY6ATcDNwIXB5\nUxtxTdHMstO8mmJJImIu8DRwSERMi0QN8H/ArsVe76RoZhkp69nnPmkNEUldgQOAtyT1TecJOBwY\nWywqHz6bWTZEOW9H0Be4Q1I1SWXv7oh4UNKTkvqkWxsNnFmsICdFM8tI+UbJiYjXgR0amL9fc8ty\nUjSz7OTwihYnRTPLjq99NjMr4JqimVlKHnnbzGxVrimamX1GTor517lTBx6/9Vw6depAh+pq7n/8\nNa686WH23XVLrjr3CKqqxMJFNZz+0z8x6cNZWYebO0uWLOGAL3+JpTU1LK9dzhFHHsVPfnpZ1mHl\nTq8uHThux7706NyBCHh+8lxGvPcJa3Ws4qSd+7Fu147MWbyMO175iMXLVmQdbkUkt2hxUsy9mqXL\nOWTor1m4eCkdOlTx5G3n8+hzb/Lri4/h6PP+wIT3pjP06L256DuHMPSnf8463Nzp3Lkz/3rsSbp1\n68ayZcvYb5+9OOjgr7Db7rtnHVqurIhg+LgZTPm0hs7VVZy/zwAmzFzIrpv05J2ZC3ni3Tnsv/m6\n7L/5ejw4fmbW4VaGhKrylxTz18qZAwsXLwWgY4dqOnSoJiKICHqs3QWAHt27Mm3mp1mGmFuS6Nat\nGwDLli1j+bJluawNZG1eTS1TPq0BoKZ2BdPn19Czawe227AbL3+YfLde/vBTtu/bLcswK05SyVNL\ncU2xAVVV4t/DLmRg/z784a4RvDx2Mt+9fBj3/+a7LKlZyryFS9jnpF9mHWZu1dbWsseuOzFx4ruc\ncdb32HW33bIOKdfW6dqRjXt2YfInS+jeuQPzamqBJHF269S2/0Tz+INZ0ZqipEMkTZD0rqSLKrmt\nclqxItj9mGvY/OBL2Hm7TdlmYF/OOf7LHHHO79j8kJ/wpwde4NoLjsw6zNyqrq7mxVdH8+77U3jl\n5ZcYN7boNfjtVqdqceou/bh/3HRqlrfNtsOm5LGmWLGkmF6Y/VvgK8A2wLGStqnU9irh0wWLGfHK\nOxy85zZsv2U/Xh47GYB7Hx3F7oM3yzi6/OvVqxdf2mdfHn30X1mHkktVglN36cerUz7ljWnJoNHz\na5bTo3MySEKPztUsWLo8yxArS82cWkgla4q7Au9GxKSIWAr8FTisgtsri97rdKNnt64AdOnckf12\nG8Rb702nR7eubL7J+gDst/tWTHhvepZh5tbMmTOZO3cuAIsXL+bJJx5n0KCtMo4qn44Z0pfp85fy\nzKRPVs4b+/ECdunfE4Bd+vdk7McLGnt5qydKryW2lTbFfsCHBc+nAJ9rXEqHFE+GFe+YfaPyhr17\ncMvlJ1JdVUVVlfjbY6P457Nj+d4Vw7jzuu+wIlYwd95izrjUZ54b8vG0aZz+7ZOpra1lRazgP4/6\nJod+9WtZh5U7m63blV3692TqvCX8YJ8BADw0fiZPvDObk3fux26b9OKTtEtOW5bHNsVKJsWG3u3n\nbiQTETeTDBNO1Vrrr9aNZspp7DtT+Y9jr/3c/OFPvc7wp17PIKLWZfsvfpEXXnkt6zBy7705izlv\n+FsNLvv98x82OL8tam9JcQrQv+D5xsDUCm7PzFqZPCbFSrYpvgxsIWkzSZ2AY4DhFdyembUmOT3R\nUrGaYkQsl3Q28AhQDdwWEeMqtT0za12EqKrK3/UjFe0ZGhEPAw9Xchtm1nrl8fC5bXeXN7N8y19O\ndFI0s4zINUUzs1U4KZqZFXBSNDNL1V3mlzf5Ox9uZu1HmfopSuoi6SVJYySNk3RZOn8zSS9KekfS\nXWmf6SY5KZpZNlTWocNqgP0iYjAwBDhE0u7AtcANEbEF8AlwWrGCnBTNLDPlSoqRqBtSqGM6BbAf\ncG86/w7g8GIxOSmaWWZUpZInoLekVwqmoauUJVVLGg3MAB4DJgJzI6JuUMopJKN3NcknWswsM808\n0TIrInZubGFE1AJDJPUC7ge2bmi1YhtxUjSzTFRq8NiImCvpaWB3oJekDmltsaSRunz4bGaZKVeb\noqQ+aQ0RSV2BA4DxwFPAUelqJwMPFIvJNUUzy0wZa4p9gTvSe0NVAXdHxIOS3gT+KulK4DXg1mIF\nOSmaWXbKlBMj4nVghwbmTyK5X1TJnBTNLDN5vKLFSdHMsuFRcszMPiMghznRSdHMsiKqqvKXFZ0U\nzSwzPnw2M6sjHz6bma0k8OGzmVkh1xTNzAq4TdHMrI7bFM3MPpP0U8xfVnRSNLOM5PPGVU6KZpaZ\nHOZEJ0Uzy4jcJcfMbCW3KZqZ1ZPDnOikaGbZcU3RzKxADnNivpLiRv034Jzrz8s6jFZl2wsfzjqE\nVueqEwZnHUKrM7xTdfkL9SCzZmaf8SCzZmarcOdtM7NV5DAnOimaWUbcedvM7DN57bxdlXUAZtZ+\nSSp5KlJOf0lPSRovaZyk76fzL5X0kaTR6XRosZhcUzSzzJSxorgcuCAiRknqDrwq6bF02Q0RcV2p\nBTkpmllmynX4HBHTgGnp4/mSxgP9VqcsHz6bWTbSkbdLnUouVhoA7AC8mM46W9Lrkm6TtE6x1zsp\nmlkmROntiWmNsrekVwqmoZ8rU+oG/A04NyLmAb8HBgJDSGqSvywWlw+fzSwzzTx6nhUROzdeljqS\nJMS/RMR9ABExvWD5LcCDxTbipGhmmakqU5uikqrkrcD4iLi+YH7ftL0R4AhgbLGynBTNLDNlPPu8\nJ3Ai8Iak0em8i4FjJQ0BAngfOKNYQU6KZpYJCarLdEVLRIwk6Q9eX7OHkXJSNLPM5PGKlkaToqQe\nTb0wPbNjZrbacpgTm6wpjiM5Di8Mu+55AJtUMC4za+NE0i0nbxpNihHRvyUDMbP2J4eD5JTWeVvS\nMZIuTh9vLGmnyoZlZm1eMzput2TbY9GkKOlG4Mskp7sBFgE3VTIoM2sfKnGZ35oq5ezzHhGxo6TX\nACJijqROFY7LzNo4Ub7O2+VUSlJcJqmK5OQKktYDVlQ0KjNrF3KYE0tqU/wtyfWEfSRdBowErq1o\nVGbWLuSxTbFoTTEi/ijpVeCAdNbREVH0+kEzs6aU84qWcir1ipZqYBnJIbSHGzOzsshfSizt7POP\ngTuBjYCNgWGSflTpwMys7WuVh8/ACcBOEbEIQNLPgFeBqysZmJm1bcnZ56yj+LxSkuLkeut1ACZV\nJhwzazdauAZYqqYGhLiBpA1xETBO0iPp84NIzkCbma2RHObEJmuKdWeYxwEPFcx/oXLhmFl70qpq\nihFxa0sGYmbtS6ttU5Q0EPgZsA3QpW5+RGxZwbgyc8/PL+KtF56kW6/1OO+2fwIw9d03uf+Gn7B8\n6VKqqqs5/PuX0X/rwRlHmh99e3XhumMH07t7Z1ZEcNcLH3L7s++zVd/uXHHUdqzduQNT5izi/L+M\nYUHN8qzDzY2bLr2AUc8+To91e3PdPU8AcM9Nv+TJ+4fRY531ADjm7AvZYa/9swyzolpVTbHA7cCV\nwHXAV4BTacOX+e108JHscfgJ3H3ND1fO++cfruWAk/4fg3bbh7deeJqHb76WM24YlmGU+bK8Nrhq\n+HjGfTSPtTtX88B5ezHy7Vlc/c3tufofb/HSpDkctevGnP7lzbjhX+9kHW5u7PP1ozn4W6fw2/8+\nd5X5hx5/Ol8/6cyMomo5ElTnMCmW0hF7rYh4BCAiJkbEJSSj5rRJXxi8K1179Fp1psSSRQsAWLJw\nPj3W2yCDyPJr5vwaxn2UDMS+sKaWd6cvYIOeXdhs/bV5adIcAJ57exYHb79hlmHmztY77c7aPXsV\nX7ENa62j5NSktw+cKOlM4CNg/cqGlS9f/94l3HrhqTx809XEiuCs39yddUi51W+drmzbrwdjJs/l\nnY8XcMC26/P4uBl85Yt96dura9bhtQqP3HU7zz54L1/YZjAnnP8TutX/kW5D8nj4XEpN8TygG/D/\nSG4jeDrw7WIvknSbpBmSWv110i8MH8bXvvtjfnTXSL72vYv523W+oKcha3Wq5ncn78gVD7zJgprl\nXHjX65yw56Y8cO6erN2lmmW1bbbVpWwOPPokfj38Oa7566P06r0+f77+iqxDqqg81hSLJsWIeDEi\n5kfEBxFxYkR8IyKeK6Hs24FD1jjCHHj10fvYbu+DAdh+n0P58K0xGUeUPx2qxG9P2ZEHRk3l0Tem\nAzBpxkJOufllDvvVc/xj1DQ+mL0o4yjzr9d6faiqrqaqqor9jjyOd8eNLv6iVkqIKpU+tZSmOm/f\nTzqGYkMi4simCo6IEZIGrHZkOdJjvQ2YNOZFBg7ZnYmvPU/vfgOyDil3rvnW9kycvoDbRry3ct56\n3Toxe8FSJDj7wIEMe/6DDCNsHT6ZOZ11+iRt1i8/+S/6DxyUcUQV1MI1wFI11aZ4Y0sEIGkoMBSg\n1wYbtcQmm3TnFecyacyLLPz0E6765p4ceMr3+c8LfsY/bryC2tpaOnbqzBEX/CzrMHNlp83W4Yid\nN+atqfP4x/l7AfDLhycwoM/anLDnpgA88sbH3PvSlCzDzJ1f/+h7vPnq88yfO4fvHrIzR515AW++\n8jyT3x6HEH026s93fnxN1mFWVLnaFCX1B/4IbEjSO+bmiPgfSesCdwEDgPeBb0bEJ02WFdFoZbAc\ngQ4AHoyI7UpZf+NB28c5N/29YvG0RTc/OCHrEFqdq05wH9Pmuvj4Q5n45piy1uvW33y7+NYv7il5\n/RuP3ObViNi5oWWS+gJ9I2KUpO4kg9YcDpwCzImIayRdBKwTERc2tR2PjWhmmRDlGzosIqZFxKj0\n8XxgPNAPOAy4I13tDpJE2aRSB5k1Myu7Slzmlx6h7gC8CGwQEdMgSZySinYnLLmmKKlzMwO7E3ge\nGCRpiqTTmvN6M2vb6m5HUOoE9Jb0SsE09PNlqhvJPaXOjYh5qxNXKdc+7wrcCvQENpE0GPhORJzT\n1Osi4tjVCcjM2o9m1hRnNdamCCCpI0lC/EtE3JfOni6pb1pL7AvMKBpTCYH8GvgaMBsgIsbQhi/z\nM7OWU67O2+lVd7cC4yPi+oJFw4GT08cnAw8Ui6mUNsWqiJhcr6GztoTXmZk1Khk6rGyNinsCJwJv\nSKrr8X4xcA1wd9p89wFwdLGCSkmKH6aH0CGpGjgHeHu1wjYzK1Cu7i8RMZLGbw7YrLHXSkmKZ5Ec\nQm8CTAceT+eZma2R1nZFCwARMQM4pgViMbN2RC18TXOpSjn7fAsNXAMdEZ87HW5m1hw5zIklHT4/\nXvC4C3AE8GFlwjGz9qRV3qMlIu4qfC7pT8BjFYvIzNoFQV2n7FxZncv8NgM2LXcgZtbOqJXWFCV9\nwmdtilXAHOCiSgZlZu2DGu1Fk50mk2LaS3wwyX1ZAFZEJccaM7N2I6/3fW6y72SaAO+PiNp0ckI0\ns7KpUulTi8VUwjovSdqx4pGYWbtTrvEUy6mpe7R0iIjlwF7A6ZImAgtJar0REU6UZrba8nr43FSb\n4kvAjpQwUq2ZWbO1whtXCSAiJrZQLGbWzrS2y/z6SDq/sYX1xiwzM2uW1nj4XA10o/HheMzM1oCo\nbmU1xWkRcXmLRWJm7UpyN7+so/i8om2KZmYV0Qov82vWaLVmZs3Vqk60RMSclgzEzNqX1nj4bGZW\nUa2qpmhmVmk5zIlOimaWDVG+u/mVk5OimWVDtOhAD6VyUjSzzOQvJTopmllGBLm8oiWPh/Rm1k5I\npU/Fy9JtkmZIGlsw71JJH0kanU6HFivHSdHMMlL6ALMltj3eDhzSwPwbImJIOj1crBAfPptZJsp9\n9jkiRkgasKbluKZoZplpZk2xt6RXCqahJW7mbEmvp4fX6xRb2UnRzDKjZkzArIjYuWC6uYRN/B4Y\nCAwBpgG/LPaCXB0+z55Xwx8f90DfzfHkxftlHUKrc+pfRmUdQqszc0FN+QttgX6KETF95eakW4AH\ni73GNUUzy0Rdm2Kp02ptQ+pb8PQIYGxj69bJVU3RzNqXctYUJd0J7EvS9jgF+Cmwr6QhQADvA2cU\nK8dJ0cwyU85BZiPi2AZm39rccpwUzSwTyeFz/q5ocVI0s8zk8Co/J0Uzy4qQa4pmZp9xTdHMLOU2\nRTOzQiWOftPSnBTNLDNOimZmBXyixcwsJcrbebtcnBTNLDO+77OZWQEfPpuZpXz4bGa2Cl/RYmb2\nGfdTNDNbVQ5zopOimWUjaVPMX1p0UjSzzOQvJTopmlmWcpgVnRTNLDM+fDYzK5C/lOikaGZZymFW\ndFI0s0wIX+ZnZvYZd942M1tVDnMiVVkHYGbtmJoxFStKuk3SDEljC+atK+kxSe+k/69TrBwnRTPL\niJr1rwS3A4fUm3cR8EREbAE8kT5vkpOimWVGKn0qJiJGAHPqzT4MuCN9fAdweLFy3KZYz4Y9O3P1\nUduzXrdORMA9L0/hz89/wFZ9u/Pf39iazh2rWL4iuHL4eN6YMi/rcHNn4jtvc/bpJ658/uH773He\nRT/htDPPyTCq/OnTrRMXHbQF667VkQh4cOx07hszjZN3689Xt12fuYuXA3Drvyfz4uS5GUdbGSUe\nFRfqLemVguc3R8TNRV6zQURMA4iIaZLWL7YRJ8V6lq8Ifv7PCYyfOp+1OlVzz/d25/l3Z3P+wVvw\nu6cmMfLtWey9ZW/OP3hLTr31leIFtjMDt9iSfz79IgC1tbXstv1ADv7qNzKOKn9qVwQ3Pfs+78xc\nSNeOVdx0zGBe/TBJfve+No27X5uacYQtQ807/TwrInauVCx1nBTrmTV/KbPmLwVg0dJaJs1cyPo9\nOgPQrXM1AN27dGDm/JrMYmwtnhvxFJsO2IyN+2+adSi5M2fRMuYsWgbA4mUr+OCTxfReu1PGUbW8\nFuiSM11S37SW2BeYUewFTopN2KhXF7bu253Xp3zKNQ9N4OZTduQHhwyiqgqO/8NLWYeXe/+4/x6+\nceQ3sw4j9zbo3pnN+6zN+OkL2G6jHhw+eEMO3LoPb09fwO9Hvs+CmtqsQ6yYFuiSMxw4Gbgm/f+B\nYi+o2IkWSf0lPSVpvKRxkr5fqW1VwlqdqvnVcUO45qEJLKyp5Vu7bsy1D0/ggF+M4NqHJnDFEdtm\nHWKuLV26lMf/9RCHfuPIrEPJtS4dq7jsq4P43Yj3WLS0luGvf8wJd4xi6LAxzF60jLP2GpB1iJXT\nnO44pXXJuRN4HhgkaYqk00iS4YGS3gEOTJ83qZJnn5cDF0TE1sDuwPckbVPB7ZVNhyrxq+MG89CY\naTz+ZlLbPmzHjXhsXPL4kbHT2X7jnlmGmHtPP/4I231xCH3W3yDrUHKrukpcduggHp8wk2cnJidN\nP1m8jBUBATw0djpbbdg92yArrJxdciLi2IjoGxEdI2LjiLg1ImZHxP4RsUX6f/2z059TsaQYEdMi\nYlT6eD4wHuhXqe2V0+VHbsukGQu547nJK+fNmFfDLpsl/T53+8K6TJ69KKvwWoXh993N133o3KQf\n7j+QD+Ys5t7Xpq2ct+5aHVc+3nvgurzXhr9norxdcsqlRdoUJQ0AdgBebIntrYkdN+3FYTtsxISP\n5/O3s3cH4FePvsulf3+Ti766FR2qRM3yFVz693EZR5pfixctYuQzT3LV9TdmHUpubde3OwdtvT4T\nZy3k5mMHA0n3m/0G9WZg77UFt+R2AAAHuUlEQVQJYPq8Gq5/cmK2gVZYHi/zq3hSlNQN+BtwbkR8\nrmOfpKHAUICOPYp2Iaq4UZPnsu2PH21w2Td/90ILR9M6dV1rLUa/81HWYeTa2Gnz2e/X//7c/Lba\nJ7FROcyKFb2iRVJHkoT4l4i4r6F1IuLmiNg5InauXsvtdGbtSZkv8yuLitUUlfTKvBUYHxHXV2o7\nZtZ6VbWzmuKewInAfpJGp9OhFdyembU2ZeySUy4VqylGxEhy2WJgZnngkbfNzAp55G0zs1XlMCc6\nKZpZhnKYFZ0UzSwjLdvVplROimaWGbcpmpmlWrinTcmcFM0sOznMik6KZpaZqhwePzspmllm8pcS\nnRTNLCvuvG1mVl/+sqKTopllom7k7bxxUjSzzOQwJzopmll2XFM0Myvgy/zMzArlLyc6KZpZdnKY\nE50UzSwbUnmvaJH0PjAfqAWWR8TOq1OOk6KZZaf8VcUvR8SsNSnASdHMMpPHw+eK3vfZzKwpUukT\n0FvSKwXT0HrFBfCopFcbWFYy1xTNLCPNHnl7VpF2wj0jYqqk9YHHJL0VESOaG5VrimaWibrL/JpR\nU2xSRExN/58B3A/sujpxOSmaWasnaW1J3eseAwcBY1enLB8+m1lmytgjZwPgfiUFdgCGRcS/Vqcg\nJ0Uzy0y5LvOLiEnA4HKU5aRoZplIOm9nHcXnOSmaWXacFM3MPuNRcszMCng8RTOzAjnMiU6KZpah\nHGZFJ0Uzy0we2xQVEVnHsJKkmcDkrONoQG9gjYYjaof8ma2evH5um0ZEn3IWKOlfJO+3VLMi4pBy\nxtCQXCXFvJL0yuoOWNle+TNbPf7csudrn83MCjgpmpkVcFIszc1ZB9AK+TNbPf7cMuY2RTOzAq4p\nmpkVcFI0MyvgpGhmVsBJsRGSBkn6D0kdJVVnHU9r4c+qeSRtLmlnSZ2zjsUSPtHSAElHAlcBH6XT\nK8DtETEv08ByTNKWEfF2+rg6ImqzjinvJH2N5Hs2G/gY+GndZ2jZcU2xHkkdgW8Bp0XE/sADQH/g\nvyT1yDS4nEr/uEdLGgYQEbWuMTZN0h7AdcDJEfFl4BPgomyjMnBSbEwPYIv08f3Ag0An4DgpjyPA\nZSe9c9rZwLnAUkl/BifGEl0TEa+lj38KrOvD6Ow5KdYTEcuA64EjJe0dESuAkcBoYK9Mg8uhiFgI\nfBsYBvwA6FKYGLOMLedeBO6Dle2wnYFNSX6QkbRedqG1b06KDXsWeBQ4UdKXIqI2IoYBG1GmO4a1\nJRExNSIWRMQs4Ayga11ilLSjpK2yjTB/0u9UXRu1gLnAnIiYKel44EpJXbOLsP3yeIoNiIglkv4C\nBPCj9I+6huTestMyDS7nImK2pDOAX0h6C6gGvpxxWLkWEcuBBZI+lHQ1yY3cT4mIxRmH1i45KTYi\nIj6RdAvwJkntZwlwQkRMzzay/IuIWZJeB74CHBgRU7KOKc/SduqOwN7p//tHxDvZRtV+uUtOCdI2\nn0jbF60ISesAdwMXRMTrWcfTWkg6BXg5IsZlHUt75qRoFSGpS0QsyTqO1kSSwn+QmXNSNDMr4LPP\nZmYFnBTNzAo4KZqZFXBSNDMr4KTYRkiqlTRa0lhJ90haaw3K2lfSg+njb0hqdKACSb0kfXc1tnGp\npB+UOr/eOrdLOqoZ2xogaWxzY7T2yUmx7VgcEUMiYjtgKXBm4UIlmr2/I2J4RFzTxCq9gGYnRbO8\nclJsm54FNk9rSOMl/Q4YBfSXdJCk5yWNSmuU3QAkHSLpLUkjgSPrCpJ0iqQb08cbSLpf0ph02gO4\nBhiY1lJ/ka73Q0kvS3pd0mUFZf1Y0gRJjwODir0JSaen5YyR9Ld6td8DJD0r6e106DIkVUv6RcG2\nz1jTD9LaHyfFNkZSB5LL695IZw0C/hgROwALgUuAAyJiR5LBc8+X1AW4Bfg6yaVmGzZS/K+BZyJi\nMLAjMI5kDMCJaS31h5IOIhl2bVdgCLCTpC9J2gk4BtiBJOnuUsLbuS8idkm3Nx44rWDZAGAf4KvA\nTel7OA34NCJ2Scs/XdJmJWzHbCVf+9x2dJU0On38LHAryag+kyPihXT+7sA2wHPpsJCdgOeBrYD3\n6q63TUe4GdrANvYDToKVw4J9ml7SV+igdKobJ7AbSZLsDtwfEYvSbQwv4T1tJ+lKkkP0bsAjBcvu\nTi+7fEfSpPQ9HAR8saC9sWe6bY9mbSVzUmw7FkfEkMIZaeJbWDgLeCwijq233hCSEYHKQcDVEfGH\nets4dzW2cTtweESMSa8L3rdgWf2yIt32ORFRmDyRNKCZ27V2zIfP7csLwJ6SNgeQtJakLYG3gM0k\nDUzXO7aR1z8BnJW+tjq9PcN8klpgnUeAbxe0VfaTtD4wAjhCUldJ3UkO1YvpDkxTcouI4+stO1pS\nVRrzF4AJ6bbPStdH0pZKRgY3K5lriu1IOoDpKcCd+mzY+0si4m1JQ4GHJM0iGWl8uwaK+D5ws6TT\ngFrgrIh4XtJzaZeXf6btilsDz6c11QUkQ66NknQXyQjmk0kO8Yv5CckI1ZNJ2kgLk+8E4BmSMS7P\nTMfA/F+StsZR6XBcM4HDS/t0zBIeEMLMrIAPn83MCjgpmpkVcFI0MyvgpGhmVsBJ0cysgJOimVkB\nJ0UzswL/H9wPSREg0wwsAAAAAElFTkSuQmCC\n", 794 | "text/plain": [ 795 | "" 796 | ] 797 | }, 798 | "metadata": {}, 799 | "output_type": "display_data" 800 | }, 801 | { 802 | "data": { 803 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEmCAYAAAAA6gkZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8FPX9x/HXOwnhvq9wgxwWRBEE\n6lmtJ1Y8atV6S0Wp/mq19arWVluqrdbWqlWreNR63wcCFq0WrTeIiHLKKeEIdzgEAuHz+2MmYRN2\nkw3uZjbJ58ljH+zsfPc7n9lsPvl+vzPzHZkZzjnndpcVdQDOOZepPEE651wCniCdcy4BT5DOOZeA\nJ0jnnEvAE6RzziXgCbIGkPQ7SU+Ez7tK2iQpO8XbWCTp6FTWmcQ2L5VUEO5P629RzyZJe6UytqhI\nmiHpiKjjcAFPkJQmhwJJjWNeu0jSpAjDisvMvjazJmZWHHUs34akesAdwLHh/qzZ07rC9y9IXXSp\nJ+lRSTdXVs7M9jGzSdUQkkuCJ8hdcoArvm0lCvjnWrn2QANgRtSBZAJJOVHH4Hbnv8i73A5cLalF\nvJWSDpY0WVJh+P/BMesmSbpF0vvAN8Be4Ws3S/og7AK+Jqm1pCclbQjr6B5Tx12SloTrPpV0WII4\nuksySTmSDgrrLnlslbQoLJcl6TpJ8yWtkfScpFYx9ZwnaXG47oaKPhhJDSX9NSxfKOk9SQ3DdSeF\n3cL14T73jXnfIklXS5oevu9ZSQ0k9QHmhMXWS3o7dr/Kfa4Xhc97SXonrGe1pGdjypmkXuHz5pIe\nk7QqjPc3JX+wJI0IY/+LpHWSFko6voL9XiTpmjD+zZIeltRe0uuSNkr6j6SWMeWfl7QijPFdSfuE\nr48CzgGuLfkuxNT/K0nTgc3hz7R0qEPSBEl/jan/WUmPVPSzcilmZnX+ASwCjgZeAm4OX7sImBQ+\nbwWsA84jaGmeFS63DtdPAr4G9gnX1wtfmwf0BJoDM4G54XZygMeAf8bEcC7QOlx3FbACaBCu+x3w\nRPi8O2BATrl9KNnmn8LlXwAfAZ2B+sADwNPhun7AJuB74bo7gB3A0Qk+n3vDujsB2cDB4fv6AJuB\nY8LtXxvuc27M5/oJ0DH8DGcBl8Tbj3j7FW7zovD508ANBH/UGwCHxpQzoFf4/DHgVaBpWOdcYGS4\nbgSwHbg43I9LgWWAKvhefETQ2u0ErASmAgPD/X8buCmm/IXhdusDdwLTYtY9SvjdKlf/NKAL0DD2\nuxg+zwu3eSRBgl0ANI3696UuPSIPIBMe7EqQ/YFCoC1lE+R5wCfl3vMhMCJ8PgkYXW79JOCGmOW/\nAq/HLJ8Y+wsUJ6Z1wIDw+e+oPEH+AxgPZIXLs4CjYtZ3CJNDDnAj8EzMusZAEXESZJiQtpTEUm7d\nb4HnypVdChwR87meG7P+z8D98fYj3n5RNkE+BowBOseJw4BeBElvG9AvZt1PY36OI4B5Mesahe/N\nq+B7cU7M8ovAP2KWfw68kuC9LcK6m4fLjxI/QV4Y77sYs3wqsARYTcwfBX9Uz8O72DHM7EtgHHBd\nuVUdgcXlXltM0KoosSROlQUxz7fEWW5SsiDpKkmzwu7ZeoJWZ5tk4pb0U+AI4Gwz2xm+3A14Oez6\nridImMUEraGOsfGa2WYg0UGSNgQttvlx1pX5XMJtL6Hs57Ii5vk3xOxzFV0LCPgk7NJfmCDWXMr+\nrMr/nErjMbNvwqcVxZTUz1BStqRbwyGNDQSJriSmisT73sQaR5D455jZe5WUdSnmCXJ3NxF0wWJ/\nqZYRJJxYXQlaSyX2eFqkcLzxV8AZQEsza0HQklWS7/0DcLKZFcasWgIcb2YtYh4NzGwpsJygW1dS\nRyOC7n08q4GtBEMF5ZX5XCQprHdpnLKV2Rz+3yjmtbySJ2a2wswuNrOOBK3C+0rGHcvFup2yP6vy\nP6d0ORs4maAn0pygRQy7foaJvh+VfW9uIfjj1kHSWd8yRldFniDLMbN5wLPA5TEvTwD6SDo7HEj/\nMcE43rgUbbYpwRjgKiBH0o1As8reJKlLGOv5Zja33Or7gVskdQvLtpV0crjuBWC4pEMl5QKjSfBd\nCFuFjwB3SOoYtpQOklQfeA44QdJRCk7buYqgi/tBlfY+2M4qgkR2briNC4lJypJOl9Q5XFxHkFiK\ny9VRHMZ0i6Sm4b5fCTxR1Xj2QFOCfV9DkOT/WG59AVClczUlfQ/4CXB++Pi7pE4Vv8ulkifI+EYT\njMsBYME5esMJEsAagu7ecDNbnaLtTQReJzigsJigxVZZ1wvgKIJW1gvadSS75LSZu4CxwBuSNhIc\nbPhuuD8zgJ8BTxG0JtcB+RVs52rgC2AysBa4jWCscw7BwaW/E7TeTgRONLOiJPe7vIuBawg+430o\nm2iHAB9L2hTu1xVmtjBOHT8naI0uAN4L97E6jvw+RvCzW0pwQO6jcusfBvqFQx6vVFaZpGZhnZeZ\n2dKwe/0w8M+wpe6qgcKBYOecc+V4C9I55xLwBOmccwl4gnTOuQQ8QTrnXAIZdYG8chqacptGHUaN\nMrBv16hDqHHyC7dGHUKNs2HlUrZsWJfSo+fZzbqZ7diSdHnbsmqimQ1LZQyVyawEmduU+nufEXUY\nNcr7H98TdQg1zvUTZkcdQo3z1JWnpbxO27GlSr/vW6fdm9SVZamUUQnSOVeXCDJ8ZkBPkM65aAjI\n8HPePUE656LjLUjnnItHkJXSWyulXGanb+dc7SYl/6i0Kg2TNEfSPEnlpywsKXOGpJnhlHlPVVan\ntyCdc9EQKetiK7jL570Es9vnA5MljTWzmTFlegPXA4eY2TpJ7Sqr11uQzrmIVKH1WHkLcijBbPEL\nwtmkniGYnzPWxcC9ZrYOwMxWVlapJ0jnXHSUlfyjYp0oO0VgPmUnvYbgHkp9JL0v6SNJlZ507l1s\n51x0qnaaTxtJU2KWx5jZmJKa4pQvP5djDtCb4PYknYH/SepvZusTbdATpHMuIlU+UXy1mQ1OsC6f\nmNuIECTAZXHKfGRm24GFkuYQJMzJiTboXWznXDRKThRPzRjkZKC3pB7hbUTOJJh5PtYrwPcBJLUh\n6HIvqKhSb0E656KToqPYZrZD0mUEty/JBh4xsxmSRgNTzGxsuO5YSTMJ7md0TXg7lYQ8QTrnIiLI\nTt2J4mY2geAGe7Gv3Rjz3Ahu4nZlsnV6gnTORSOF50GmiydI51x0fLIK55yLx6c7c865xLwF6Zxz\nCXgL0jnn4khylp4oeYJ0zkXHW5DOOZeAtyCdcy4eP4rtnHPxiYy/5YInSOdcRLwF6ZxzifkYpHPO\nJeAtSOecS8BbkM45F4d8DNI55xLzFqRzzsUnT5CZ7ZiD+/KXa04jOyuLR1/5gL/8883dyvzomIHc\ncMkPMIMv5i5lxK8fZb8+nbj7hjNp2rgBxcU7+fPDE3nhjakR7EH1e2Piv7n6yisoLi5mxIUXcc21\n15VZv23bNkb+5Hw+m/oprVq15omnnqVb9+4sXrSI/fftS58+ewMw9LsH8vf77o9iF6rdd9o25of7\ntkMSHy9ez1vz1pZZf3C3FhzSowVmsG3HTp77fAUFm4ro2qIBZwzIKy03cc5qvlixqbrDT4vgljSe\nIDNWVpa487ozOOHSe1hasJ73nryGce98wewFK0rL9OzalqsvPJYjR9zB+o1baNuyCQDfbN3OyN8+\nxvyvV9GhbXPef/Ja3vxgFoWbtkS1O9WiuLiYX1z+M8a//iadOnfm0AOHMHz4SfTt16+0zKOPPEzL\nFi2ZMXsezz37DDf8+lc88dSzAOzVsycffzotqvAjIeBH+7Xn/g+XsH7Ldn75ve58uWITBZuKSst8\nunQDHywO7j66T/smnNy/HWM+ymf5xm3c8e4idho0q5/N1Uf0YEbBPHaWv6FpTSShrMxOkJk9Qppm\nQ/p3Z/6S1SxauobtO4p5fuJUhh+xX5kyF/7wYB547l3WbwwS36p1wV/veV+vZP7XqwBYvqqQVes2\n0qZVk+rdgQhM/uQTevbsRY+99iI3N5fTf3wm4157tUyZca+9yjnnXQDAqT86jUlvv0VwO5C6qWvL\nBqzeXMSab7ZTbPDZ0g30zyv7Xdm2Y2fp89wcld7ReXuxlSbDnOza9+sqKelHFOp0C7Jju+bkF6wr\nXV5asI6h/buXKdO7WzsA3v7nL8nOyuLmBybw5gezypQZvE83cnNyWLBkddpjjtqyZUvp3HnX7Yc7\nderMJ598vHuZLkGZnJwcmjVvzpo1wc3jFi1cyIGDB9K0WTNuGn0zhx56WPUFH5EWDeqxfsuO0uXC\nrTvo2rLhbuUO6d6CI3q2IjtL3PfB16Wvd23RgLP270DLRvV4cuqy2tF6DNXpLrakYcBdBLdhfMjM\nbk3n9qpK7P7DKf/dy87OplfXdhx78V10ateStx75BQec9sfSrnRem2Y8fPP5XHzj43WilRRvH8t/\nyROVyevQgbkLvqZ169ZM/fRTzjjtFKZ+PoNmzZqlLd6MFeer8v6i9by/aD2DOjXj2D5teOqz5QB8\nvX4rt01aSLsmuZw9sAOzVm5mRy3JkpmeINPWZpeUDdwLHA/0A86S1K/id1WvpSvX07l9y9LlTu1b\nsmxV4W5lXps0nR07drJ42RrmLlpJr65tAWjauAEv3X0pv793HJ98sag6Q49Mp06dyc9fUrq8dGk+\nHTt23L3MkqDMjh072FBYSKtWrahfvz6tW7cGYNABB7DXXj35au7c6gs+Iuu3bqdFw11tkeYNcijc\nuj1h+XhdcICVm4ooKt5Jh6b10xJntVMVHxFI56DGUGCemS0wsyLgGeDkNG6vyqbMWEyvrm3p1rE1\n9XKyOf24QYyfNL1Mmdf++zmHD+kDQOsWjendrR0Ll66hXk42z/71Yp4a9zEv/eezKMKPxOAhQ5g3\n7ysWLVxIUVERzz/7DCcMP6lMmROGn8STj/8LgJdefIHDv38kkli1ahXFxcUALFywgHnzvqLHXntV\n+z5UtyXrt9K2cS6tGtUjWzCwUzNmFJQ9Et2mcb3S5/3aN2H15uAATqtG9Sg5jtGyYQ7tmuSydkvi\n5FqTiOTHH2vjGGQnYEnMcj7w3fKFJI0CRgFQr3oPchQX7+SXtz3Ha/f9jOws8a9XP2LWghX89tIT\nmDrza8a/8wVvfjCLow/qy9QXb6C42Pj1na+wtnAzZ/5gCIcO6kWrFo0596QDARh14+NMn7u0Wveh\nuuXk5PC3u+7hxBOOo7i4mAtGXEi/ffZh9O9uZNABgxl+4kmMuHAkF444j32+04uWLVvx+JPPAPDe\n/97lD7+/kZzsHLKzs/n7vffTqlWriPco/XYavPhFAT89sAtZgo+/LmTFxiKG7d2GJeu3MqNgE4f1\naEmfNo0pNuOb7cWl3eu9WjXkqF6dKTbDDF6YXsDmouKI9yh1Mr2LrXSNm0k6HTjOzC4Kl88DhprZ\nzxO9J6tRO6u/9xlpiae2Wjf5nqhDqHGunzA76hBqnKeuPI2CeV+mNJvltN7Lmv3g5qTLr3vinE/N\nbHAqY6hMOluQ+UCXmOXOwLI0bs85V8NkegsynWOQk4HeknpIygXOBMamcXvOuZqkBhykSVsL0sx2\nSLoMmEhwms8jZjYjXdtzztUsQmRlZfbJ72k9D9LMJgAT0rkN51zNleld7Dp9JY1zLmKZnR89QTrn\nIqLMb0Fm9gCAc65WS+WJ4pKGSZojaZ6k6+KsHyFplaRp4eOiyur0FqRzLjKpakHGXNp8DMEphpMl\njTWzmeWKPmtmlyVbr7cgnXORSPGlhmm5tNkTpHMuOlU7D7KNpCkxj1ExNcW7tLlTnC3+SNJ0SS9I\n6hJnfRnexXbORaPqB2lWV3CpYbyKyl9H/RrwtJltk3QJ8C/gyIo26C1I51xkUtjFrvTSZjNbY2bb\nwsUHgQMqq9QTpHMuMspS0o9KVHpps6QOMYsnAWVvDRCHd7Gdc5FJ1VHsRJc2SxoNTDGzscDlkk4C\ndgBrgRGV1esJ0jkXiVRPhBvv0mYzuzHm+fXA9VWp0xOkcy4ymX4ljSdI51xkPEE651wimZ0fPUE6\n56LjLUjnnIunBszm4wnSORcJARmeHz1BOueiIrIqPwE8Up4gnXOR8S62c87FI+9iO+dcXALvYjvn\nXCLegnTOuQR8DNI55+LxMUjnnIsvOA8yszOkJ0jnXERSO91ZOniCdM5FJsPzoydI51xE5Kf5OOdc\nXD4G6ZxzFcjw/OgJ0jkXHW9BOudcAhmeHzMrQXbq0p4r7rwy6jBqlAE3/DvqEGqcm8/aL+oQapyx\nudmpr9QnzHXOufh8wlznnEvITxR3zrmEMjw/eoJ0zkXETxR3zrn4/ERx55yrgCdI55xLIMPzoydI\n51x0vAXpnHPx1IAZxbOiDsA5VzcpPA8y2Uel9UnDJM2RNE/SdRWUO02SSRpcWZ2eIJ1zkZGSf1Rc\nj7KBe4HjgX7AWZL6xSnXFLgc+DiZ+DxBOucikyUl/ajEUGCemS0wsyLgGeDkOOX+APwZ2JpUfFXZ\nGeecS6UqtiDbSJoS8xgVU1UnYEnMcn74Wsy2NBDoYmbjko3PD9I45yIhQXbVrqRZbWaJxg3jVWS7\ntqUs4G/AiKps0BOkcy4yKTzNJx/oErPcGVgWs9wU6A9MCreZB4yVdJKZTUlUacIEKalZRdGY2YYk\ngnbOuYRSeJrPZKC3pB7AUuBM4OySlWZWCLTZtV1NAq6uKDlCxS3IGQRN1NhdKFk2oGvV4nfOuV1E\ncKpPKpjZDkmXAROBbOARM5shaTQwxczG7km9CROkmXVJtM4551IhlZP5mNkEYEK5125MUPaIZOpM\n6ii2pDMl/Tp83lnSAcm8zznnEqrCSeJRXZJYaYKUdA/wfeC88KVvgPvTGZRzrm5I1Yni6ZLMUeyD\nzWyQpM8AzGytpNw0x+Wcq+UEyZwAHqlkEuT28BwiA5DUGtiZ1qicc3VChufHpMYg7wVeBNpK+j3w\nHnBbWqNyztUJmT4GWWkL0swek/QpcHT40ulm9mV6w3LO1XZ7cCVNtUv2SppsYDtBN9uv33bOpURm\np8fkjmLfADwNdCS4fOcpSdenOzDnXO1X47vYwLnAAWb2DYCkW4BPgT+lMzDnXO0WHMWOOoqKJZMg\nF5crlwMsSE84zrk6I8KWYbIqmqzibwRjjt8AMyRNDJePJTiS7Zxz30qG58cKW5AlR6pnAONjXv8o\nfeE45+qSGtuCNLOHqzMQ51zdUivGICX1BG4huBFOg5LXzaxPGuOqNrM/foex9/yBncXFDD3hxxx5\nziVl1r/z3MN8Mv45srKzadKiFWdcexst84KZ3Mc/cBuzPvwvAEeffxn7Hzm82uOPwmF92nDDSX3J\nEjw/OZ8HJy2MW+64fdtz97kD+dHdH/Dl0g2cuH8HRh7eo3T93nlN+eHdHzB7+cbqCj0y097/L/+8\n/UZ27tzJUaecxSkXXlZm/bjHH+Ctl58mOyeHZi1bcelNd9C2Y+fS9d9s2sgvTz2CoUcOY+R1t1R3\n+GmT6S3IZM5pfBT4J0HCPx54juCGODXezuJiXr7rd4y87RGu/tdEpr39GgWLvipTplPvflzxwCtc\n9cgE9j38eMY/cCsAsz78L0vnzuCXD43j8n+8xDvPPMjWzbX/Fz1LcOMp/bjokSmccMd7DB/QgZ7t\nGu9WrnFuNucd3I1pX68vfe21acs55a4POOWuD7j22eksXbelTiTHncXFPHzrDfz6nif424v/5f1/\nv0L+/LllynT/Tn9uffJ1/vLcfzjwqBN44q6by6x/9r7b6XfAgdUZdtpJkC0l/YhCMgmykZlNBDCz\n+Wb2G4LZfWq8r2d/TptO3WjdsSs59XLZ/8jhzHj/P2XK9Bp4ELkNGgLQrd/+rF+1AoCCxV+x14Ch\nZOfkkNuwER169WXOJ+9W+z5Ut/26tGDxmm/IX7uF7cXG+M9XcFS/9ruVu+K43jz0zkK2bY9/2f4J\nAzow7vPl6Q43I8z78jPyunSnfedu5NTL5eDjTmbypIllyvQfcgj1Gwbfs977HcDagl2fzYKZ0ylc\ns4oBB32vWuOuDpk+m08yCXKbgnbwfEmXSDoRaJfmuKrFhlUFtGjboXS5eds8ClcVJCz/yfjn+c7Q\nwwHo0LMvsz95h6KtW9i8fi3zP/uI9Str/y98++b1WbF+S+lyQeFW2jevX6ZM345NyWvegEmzVyWs\n5wcDOjB+Wu3/vADWrlxB6/YdS5dbt+/A2vAPbTxvv/I0+x8StEF27tzJY3eM5txf/ibtcUahNpwo\n/kugCcHNtm8BmgMXVvYmSY8Aw4GVZtb/2wSZLrbrpmelEv0cPn3jFfLnfMGldz0FwN5DDiN/9nTu\n+dnpNGnRim77DCQrOzud4WaEuLeOi/kYJbh+eF+uf/6LhHXs16U5W4qK+apgU+oDzEBxv2cJLrJ7\nd/yLLJj5Ob976EUA3njuXww89Eja5HWKW76my/AhyKQmq/g4fLqRXZPmJuNR4B7gsaqHVT2at81j\n/apdrZjCVSto1mb37uLcKe/z9hP3celdT5GTu6u1dNR5P+Oo834GwJN/+AVtOndPe8xRW1G4jbwW\nDUuX2zdvwMoN20qXG9fPoU9eEx4bNRSAtk1z+ceIQVz66FS+XBrc5+2EAR0YX0e61wCt23VgTcGu\nG+ytKVhOy7a7f8+mf/QuLz98N7976EXqhd+zudM/ZdZnH/PGc/9i65bN7Ni+nQYNG3POFb+utvjT\nRajmzgcp6WWI86cvZGanVlSxmb0rqfseR1YNuuy9H6vzF7F2+RKatWnPtLfHcfZv/lamzNKvZvDi\nHb/hoj8/QpOWpTdFY2dxMVs2baBx85Ysmz+b5fNn0+f6v1T3LlS7L/IL6d66EZ1bNqRgw1ZOGJDH\nVc9ML12/aesODhz9dunyY6OG8ufxs0uTowTD9svjnPs/3q3u2qrnPvuz/OuFrFz6Na3a5fHBxFe5\n/E/3limzcPaXPHjLdfz6nido3mrX9+zyP95T+nzS2GeZP3N6rUiOAEQ4tpisilqQ91SwLmUkjQJG\nAbSIGaepDtk5OZxyxU08eM0Idu7cydDjTyOvRx8mPvI3Ou+9L/sccjTj/nErRVs28/hNPwegZfuO\n/OSPYyjesYP7Lj8TgAaNmnDWDXeQnVP7bzNevNMY/epMHho5mOws8eLkfOYVbOLyY3rxZX4hb89K\nPO4IMKRHK1YUbiV/7ZYKy9Um2Tk5XPirm7nl/85m586dfP/kH9Ol5948e9/t9Ow3gMFHHMsTf/sD\nW7/ZzB3X/hSANnmd+NVdj0YbeDXI9NN8ZJawkfjtKw9akOOSHYPssve+dsWYV9MWT2304IS5lRdy\nZdx81n5Rh1DjXHf28cyf+XlKs1m7Xv3tx7c/n3T5e07t96mZDU5lDJWp/U0e51xGEpnfgvQE6ZyL\nTKZfapj07OCS6ldeqkz5p4EPgb0l5UsaWdXgnHO1V8ktF5J9RCGZa7GHAg8TnP/YVdIA4CIz+3lF\n7zOzs1ITonOutqoNLci7CU74XgNgZp9TSy41dM5FK9MvNUxmDDLLzBaXG0wtTlM8zrk6IpjuLLOb\nkMkkyCVhN9skZQM/B/zcEufct5bpt0hNJkFeStDN7goUAP8JX3POuW8lwxuQSV2LvRI4sxpicc7V\nIVINvha7hKQHiXNNtpmNSktEzrk6I8PzY1JDAP8B3gof7xPMBbmtwnc451wSspT8ozKShkmaI2me\npOvirL9E0heSpkl6T1K/yupMpov9bLmNPA68WXm4zjmXmCBlJ4CHB5DvBY4B8oHJksaa2cyYYk+Z\n2f1h+ZOAO4BhFdW7JweRegDd9uB9zjm3SxVaj0nk0aHAPDNbYGZFBPfNOjm2gJltiFlsTAXTOZZI\nZgxyXUxFWcBaYLfmq3POVVWimdUTaCNpSszyGDMbEz7vBCyJWZcPfHe37Uk/A64EcoEjK9tghQky\nvBfNAGBp+NJOS+f8aM65OmMP7ou9uoLpzuLeDWS3F8zuBe6VdDbwG+CCijZYYRc7TIYvm1lx+PDk\n6JxLmRR2sfOBLjHLnYFlCcpC0AU/pdL4Kt0sfCJpUBLlnHOuSlJ4V8PJQG9JPSTlEpy7PbbctnrH\nLJ4AfFVZpRXdkybHzHYAhwIXS5oPbCZoypqZedJ0zu2xPehiJ2RmOyRdBkwEsoFHzGyGpNHAFDMb\nC1wm6WhgO7COSrrXUPEY5CfAIJJohjrnXJWleJYeM5sATCj32o0xz6+oap0VJUiFlc6vaqXOOZeM\nmnypYVtJVyZaaWZ3pCEe51wdkcoudrpUlCCzgSbEP3zunHPfksiuwS3I5WY2utoicc7VKcFdDaOO\nomKVjkE651xaJDkJRZQqSpBHVVsUzrk6qcYepDGztdUZiHOubqnpXWznnEurGtuCdM65dMvw/OgJ\n0jkXDVE77mronHOpJ5KZhCJSniCdc5HJ7PToCdI5FxFBjb6Sxjnn0irD86MnSOdcVJKaCDdSniCd\nc5Hwo9jOOVcBb0E651wCmZ0eMyxBrircxoMT5kYdRo0y8Zojog6hxrng8U+jDqHGWblxW+or9fMg\nnXMuPh+DdM65CngL0jnnEqjJE+Y651zaBF3szM6QniCdc5HJ8B62J0jnXFSEvAXpnHPxeQvSOefi\n8DFI55xLRN6CdM65hDxBOudcAn6Qxjnn4hCZf6J4pl8K6ZyrxbKkpB+VkTRM0hxJ8yRdF2f9lZJm\nSpou6S1J3SqNbw/3yznnvjVV4V+F9UjZwL3A8UA/4CxJ/coV+wwYbGb7AS8Af64sPk+QzrlIlHSx\nk31UYigwz8wWmFkR8AxwcmwBM/uvmX0TLn4EdK6sUk+QzrmIVKX9KIA2kqbEPEbFVNYJWBKznB++\nlshI4PXKIvSDNM65aFT9PMjVZjY4cW27sbgFpXOBwcDhlW3QE6RzLjIpPIidD3SJWe4MLNtte9LR\nwA3A4WZW6TTpniCdc5EIxiBTliInA70l9QCWAmcCZ5fZnjQQeAAYZmYrk6nUxyCdc5FRFR4VMbMd\nwGXARGAW8JyZzZA0WtJJYbHbgSbA85KmSRpbWXzegnTORSeFfWwzmwBMKPfajTHPj65qnZ4gnXOR\nSWEXOy08QTrnIpPZ6dETpHPMHZi3AAANnklEQVQuShmeIT1BOuciERx8yewM6QnSORcNnzDXOecS\ny/D86AnSORehDM+QniCdcxHx274651xCmT4GWecvNTysTxv+ffVhvHHNYVx8RI+E5Y7btz1zbhtG\n/07NADhx/w68csXBpY9ZfzqO73RoWl1hR+qdt97gyAP344gh+/CPu27fbf3HH7zH8CMPoldeEyaM\nfan09ZlffM6pxx/OsYcOYtjhQxj38vPVGXakhnZrweMXDOLJnwzi7CGJZ+E6vHdr3vnlIezdvgkA\ng7s2Z8zZA/jnefsz5uwBDOzSvLpCTruqXGYYVR6t0y3ILMGNp/TjJw9NpqBwKy9cdhBvz1zJ/JWb\ny5RrnJvNeQd3Y9rX60tfe23acl6bthyAPnlNuO/8QcxevrFa449CcXExN173Cx5/fjx5HTtx8rGH\ncvSw4fTeu29pmU6du3D738fw4H13lnlvg0aN+Os9D9OjZy8KVizjxKMO4XtHHkOz5i2qezeqVZbg\nF0fuxVUvzWDVxiIeOHsA789fy+K1W8qUa1gvmx/t34EZMd+jwi07uP7VWazZXESP1o24/dR+nPbg\nlOrehbRRhjch63QLcr8uLVi85hvy125he7Ex/vMVHNWv/W7lrjiuNw+9s5Bt23fGreeEAR0Y9/ny\ndIebET6fOplu3XvStXsPcnNzOfGU03nz9XFlynTu2o2+++xLlsp+vfbq2ZsePXsB0D6vI63btmXN\n6tXVFntU+uY1Zen6rSwv3MaOncbbc1ZxaM9Wu5UbeXBXnp6ylKIdu75nX63azJrNRQAsXPMNudlZ\n1MvO7KRSFVLyjyjU6QTZvnl9Vqzf9Ve8oHAr7ZvXL1Omb8em5DVvwKTZqxLW84MBHRg/rW4kyBXL\nl9Gh066Z6vM6dmLF8qVVrmfa1MlsLyqiW4+9UhleRmrTJJeVG4tKl1dtKqJNk7Lfs95tG9OuaS4f\nLlyXsJ7De7fmq1Wb2V4cdx7YGinTu9hpS5CSukj6r6RZkmZIuiJd29pTcacgjvnuSXD98L7cNn5O\nwjr269KcLUXFfFWwKfUBZiCz3X85q9pNWrliOVf+30huv/sBsrJq/9/ouJ9OzOco4GeH9+C+dxcl\nrKN764b89NBu/PU/81MdXnRqwCBkOscgdwBXmdlUSU2BTyW9aWYz07jNKllRuI28Fg1Ll9s3b8DK\nDbsmGW5cP4c+eU14bNRQANo2zeUfIwZx6aNT+XLpBiDoXo+vI91rgA4dO7F8aX7p8oplS2mf1zHp\n92/cuIELzz6Vq66/iYGDv5uOEDPOqk1FtGuaW7rctkkuqzfvalE2ys2mR5tG3HlafwBaNc7ljyf1\n5ddjZzGnYBNtm+Ry84l9+ePEr1hWuLXa40+nOnuaj5ktB5aHzzdKmkVwE52MSZBf5BfSvXUjOrds\nSMGGrZwwII+rnpleun7T1h0cOPrt0uXHRg3lz+NnlyZHCYbtl8c5939c7bFHZb+Bg1m0cB5LFi+i\nfYeOvPbK89x1/6NJvbeoqIhLLvgxp55xNiec/KP0BppBZq/YSOeWDclrVp/Vm4o4cu+2/OH1Xb2S\nzUXFnHz/J6XLd57Wn3/8bxFzCjbRpH42t57SjzHvLebLZbXrIKDI/NN8quUotqTuwEAgozJJ8U5j\n9KszeWjkYLKzxIuT85lXsInLj+nFl/mFvD0r8bgjwJAerVhRuJX8ckcja7OcnBx+/6e/cf4ZJ7Jz\nZzGnn3UBfb7TjztuHc2++w/imGHD+fyzKVxywY8pLFzPW29M4M4/38wb701l/Ksv8smH77Fu7Vpe\neOYJAP7y9zH023dAxHuVXsUGd769gL+cug9ZggkzVrJozRYuPKgrsws28cGCtQnf+8MBHejUogHn\nf7cz5383GPu9+qWZrN+yvbrCT6sMz48o3phSSjcgNQHeAW4xs5firB8FjALIadbugL0u+Vda46lt\nJl5zRNQh1DgXPP5p1CHUOFPvGMnGJbNTms/6Dxhkz//7f0mX79exyacV3NUwLdI6Qi6pHvAi8GS8\n5AhgZmPMbLCZDc5uWHtOgnXOVa6K98WudmnrYis4tPkwMMvM7kjXdpxzNVdWhvex09mCPAQ4Dzgy\nvIPYNEk/SOP2nHM1TV09zcfM3iPzx2CdcxHxGcWdcy4Rn1HcOecSy/D86AnSORehDM+QniCdcxHx\nGcWdcy4hH4N0zrk4opzGLFmeIJ1z0cnwDOkJ0jkXmawM72N7gnTORSaz02Mdv+WCcy5CVbgfTTIN\nTUnDJM2RNE/SdXHWf0/SVEk7JJ2WTIieIJ1zEUrNxdiSsoF7geOBfsBZkvqVK/Y1MAJ4KtnovIvt\nnItEimcUHwrMM7MFAJKeAU4m5g4GZrYoXBf/9qRxeAvSOReZFE7m0wlYErOcH772rXgL0jkXmSq2\nINtImhKzPMbMxpRUFaf8t75dgidI51xkqnip4eoKbrmQD3SJWe4MLNvTuEp4F9s5F53U9bEnA70l\n9ZCUC5wJjP224XmCdM5FJlX50cx2AJcBE4FZwHNmNkPSaEknAUgaIikfOB14QNKMyuLzLrZzLhJS\naq+kMbMJwIRyr90Y83wyQdc7aZ4gnXPRyfBLaTxBOucik+H50ROkcy46GT5XhSdI51xUfEZx55yL\nK8WXGqaFn+bjnHMJeAvSOReZTG9BeoJ0zkXGxyCdcy6O4ETxqKOomCdI51x0PEE651x83sV2zrkE\n/CCNc84lkOH50ROkcy5CGZ4hPUE65yKT6WOQMvvWt21IGUmrgMVRxxFHG2B11EHUMP6Z7ZlM/dy6\nmVnbVFYo6d8E+5us1WY2LJUxVCajEmSmkjSlgnthuDj8M9sz/rllFr8W2znnEvAE6ZxzCXiCTM6Y\nyou4cvwz2zP+uWUQH4N0zrkEvAXpnHMJeIJ0zrkEPEE651wCniATkLS3pIMk1ZOUHXU8NYV/VlUj\nqZekwZLqRx2L250fpIlD0qnAH4Gl4WMK8KiZbYg0sAwmqY+ZzQ2fZ5tZcdQxZTpJwwm+Z2uAFcBN\nJZ+hywzegixHUj3gx8BIMzsKeBXoAlwrqVmkwWWo8Bd9mqSnAMys2FuSFZN0MPAX4AIz+z6wDrgu\n2qhceZ4g42sG9A6fvwyMA3KBs6VMn8GueklqDFwG/AIokvQEeJJM0q1m9ln4/CaglXe1M4snyHLM\nbDtwB3CqpMPMbCfwHjANODTS4DKQmW0GLgSeAq4GGsQmyShjy3AfAy9B6bhtfaAbwR9nJLWOLjRX\nwhNkfP8D3gDOk/Q9Mys2s6eAjsCAaEPLPGa2zMw2mdlq4KdAw5IkKWmQpO9EG2HmCb9TJWPaAtYD\na81slaRzgJslNYwuQgc+H2RcZrZV0pOAAdeHv+DbgPbA8kiDy3BmtkbST4HbJc0GsoHvRxxWRjOz\nHcAmSUsk/Qk4FhhhZlsiDq3O8wSZgJmtk/QgMJOgVbQVONfMCqKNLPOZ2WpJ04HjgWPMLD/qmDJZ\nOK5dDzgs/P8oM/sq2qgc+Gk+SQnHiCwcj3SVkNQSeA64ysymRx1PTSFpBDDZzGZEHYsLeIJ0aSGp\ngZltjTqOmkSSzH8hM4onSOecS8CPYjvnXAKeIJ1zLgFPkM45l4AnSOecS8ATZC0hqVjSNElfSnpe\nUqNvUdcRksaFz0+SlHASBUktJP3fHmzjd5KuTvb1cmUelXRaFbbVXdKXVY3ROU+QtccWM9vfzPoD\nRcAlsSsVqPLP28zGmtmtFRRpAVQ5QTpXE3iCrJ3+B/QKW06zJN0HTAW6SDpW0oeSpoYtzSYAkoZJ\nmi3pPeDUkookjZB0T/i8vaSXJX0ePg4GbgV6hq3X28Ny10iaLGm6pN/H1HWDpDmS/gPsXdlOSLo4\nrOdzSS+WaxUfLel/kuaG060hKVvS7THb/um3/SBd3eYJspaRlENwid8X4Ut7A4+Z2UBgM/Ab4Ggz\nG0QwEfCVkhoADwInElzulpeg+ruBd8xsADAImEEwh+H8sPV6jaRjCaaKGwrsDxwg6XuSDgDOBAYS\nJOAhSezOS2Y2JNzeLGBkzLruwOHACcD94T6MBArNbEhY/8WSeiSxHefi8muxa4+GkqaFz/8HPEww\n+9BiM/sofP1AoB/wfjitZS7wIfAdYGHJ9b/hTDyj4mzjSOB8KJ3KrDC8rDDWseGjZJ7DJgQJsynw\nspl9E25jbBL71F/SzQTd+CbAxJh1z4WXfn4laUG4D8cC+8WMTzYPt+2zdLs94gmy9thiZvvHvhAm\nwc2xLwFvmtlZ5crtTzBzUSoI+JOZPVBuG7/Yg208CpxiZp+H1ykfEbOufF0WbvvnZhabSJHUvYrb\ndQ7wLnZd8xFwiKReAJIaSeoDzAZ6SOoZljsrwfvfAi4N35sd3oJiI0HrsMRE4MKYsc1OktoB7wI/\nlNRQUlOC7nxlmgLLFdwG45xy606XlBXGvBcwJ9z2pWF5JPVRMOO5c3vEW5B1SDgZ6wjgae2a2v83\nZjZX0ihgvKTVBDOo949TxRXAGEkjgWLgUjP7UNL74Wk0r4fjkH2BD8MW7CaCaeKmSnqWYGb2xQTD\nAJX5LcHM24sJxlRjE/Ec4B2COTovCefwfIhgbHJqOIXYKuCU5D4d53bnk1U451wC3sV2zrkEPEE6\n51wCniCdcy4BT5DOOZeAJ0jnnEvAE6RzziXgCdI55xL4f1X0+gSdaxewAAAAAElFTkSuQmCC\n", 804 | "text/plain": [ 805 | "" 806 | ] 807 | }, 808 | "metadata": {}, 809 | "output_type": "display_data" 810 | }, 811 | { 812 | "name": "stdout", 813 | "output_type": "stream", 814 | "text": [ 815 | "Confusion Matrix:\n", 816 | "\n", 817 | "Predicted 0 1 2 __all__\n", 818 | "Actual \n", 819 | "0 38 3 20 61\n", 820 | "1 18 29 15 62\n", 821 | "2 28 7 25 60\n", 822 | "__all__ 84 39 60 183\n", 823 | "\n", 824 | "\n", 825 | "Overall Statistics:\n", 826 | "\n", 827 | "Accuracy: 0.502732240437\n", 828 | "95% CI: (0.42803368164557776, 0.57734086516071881)\n", 829 | "No Information Rate: ToDo\n", 830 | "P-Value [Acc > NIR]: 0.133021698692\n", 831 | "Kappa: 0.25479930192\n", 832 | "Mcnemar's Test P-Value: ToDo\n", 833 | "\n", 834 | "\n", 835 | "Class Statistics:\n", 836 | "\n", 837 | "Classes 0 1 2\n", 838 | "Population 183 183 183\n", 839 | "P: Condition positive 61 62 60\n", 840 | "N: Condition negative 122 121 123\n", 841 | "Test outcome positive 84 39 60\n", 842 | "Test outcome negative 99 144 123\n", 843 | "TP: True Positive 38 29 25\n", 844 | "TN: True Negative 76 111 88\n", 845 | "FP: False Positive 46 10 35\n", 846 | "FN: False Negative 23 33 35\n", 847 | "TPR: (Sensitivity, hit rate, recall) 0.622951 0.467742 0.416667\n", 848 | "TNR=SPC: (Specificity) 0.622951 0.917355 0.715447\n", 849 | "PPV: Pos Pred Value (Precision) 0.452381 0.74359 0.416667\n", 850 | "NPV: Neg Pred Value 0.767677 0.770833 0.715447\n", 851 | "FPR: False-out 0.377049 0.0826446 0.284553\n", 852 | "FDR: False Discovery Rate 0.547619 0.25641 0.583333\n", 853 | "FNR: Miss Rate 0.377049 0.532258 0.583333\n", 854 | "ACC: Accuracy 0.622951 0.765027 0.617486\n", 855 | "F1 score 0.524138 0.574257 0.416667\n", 856 | "MCC: Matthews correlation coefficient 0.232621 0.445088 0.132114\n", 857 | "Informedness 0.245902 0.385097 0.132114\n", 858 | "Markedness 0.220058 0.514423 0.132114\n", 859 | "Prevalence 0.333333 0.338798 0.327869\n", 860 | "LR+: Positive likelihood ratio 1.65217 5.65968 1.46429\n", 861 | "LR-: Negative likelihood ratio 0.605263 0.580209 0.815341\n", 862 | "DOR: Diagnostic odds ratio 2.72968 9.75455 1.79592\n", 863 | "FOR: False omission rate 0.232323 0.229167 0.284553\n" 864 | ] 865 | } 866 | ], 867 | "source": [ 868 | "from pandas_confusion import ConfusionMatrix\n", 869 | "from six.moves import range\n", 870 | "from six.moves import zip\n", 871 | "import itertools\n", 872 | "confusion_matrix = ConfusionMatrix(y_classes, pred_classes)\n", 873 | "\n", 874 | "from sklearn.metrics import confusion_matrix\n", 875 | "cm = confusion_matrix\n", 876 | "def plot_confusion_matrix(cm, classes,\n", 877 | " normalize=False,\n", 878 | " title='Confusion matrix',\n", 879 | " cmap=plt.cm.Blues):\n", 880 | " \"\"\"\n", 881 | " This function prints and plots the confusion matrix.\n", 882 | " Normalization can be applied by setting `normalize=True`.\n", 883 | " \"\"\"\n", 884 | " if normalize:\n", 885 | " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", 886 | " print(\"Normalized confusion matrix\")\n", 887 | " else:\n", 888 | " print('Confusion matrix, without normalization')\n", 889 | "\n", 890 | " print(cm)\n", 891 | "\n", 892 | " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", 893 | " plt.title(title)\n", 894 | " plt.colorbar()\n", 895 | " tick_marks = np.arange(len(classes))\n", 896 | " plt.xticks(tick_marks, classes, rotation=45)\n", 897 | " plt.yticks(tick_marks, classes)\n", 898 | "\n", 899 | " fmt = '.2f' if normalize else 'd'\n", 900 | " thresh = cm.max() / 2.\n", 901 | " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", 902 | " plt.text(j, i, format(cm[i, j], fmt),\n", 903 | " horizontalalignment=\"center\",\n", 904 | " color=\"white\" if cm[i, j] > thresh else \"black\")\n", 905 | "\n", 906 | " plt.tight_layout()\n", 907 | " plt.ylabel('True label')\n", 908 | " plt.xlabel('Predicted label')\n", 909 | "\n", 910 | "# Compute confusion matrix\n", 911 | "cnf_matrix = confusion_matrix(y_classes, pred_classes)\n", 912 | "np.set_printoptions(precision=2)\n", 913 | "\n", 914 | "# Plot non-normalized confusion matrix\n", 915 | "plt.figure()\n", 916 | "plot_confusion_matrix(cnf_matrix, classes=[0,1,2],\n", 917 | " title='Confusion matrix, without normalization')\n", 918 | "\n", 919 | "# Plot normalized confusion matrix\n", 920 | "plt.figure()\n", 921 | "plot_confusion_matrix(cnf_matrix, classes=[0,1,2], normalize=True,\n", 922 | " title='Normalized confusion matrix')\n", 923 | "\n", 924 | "plt.show()\n", 925 | "from pandas_ml import ConfusionMatrix\n", 926 | "ConfusionMatrix = ConfusionMatrix(y_classes, pred_classes)\n", 927 | "import matplotlib.pyplot as plt\n", 928 | "ConfusionMatrix.print_stats()\n" 929 | ] 930 | }, 931 | { 932 | "cell_type": "code", 933 | "execution_count": null, 934 | "metadata": {}, 935 | "outputs": [], 936 | "source": [] 937 | }, 938 | { 939 | "cell_type": "code", 940 | "execution_count": null, 941 | "metadata": {}, 942 | "outputs": [], 943 | "source": [] 944 | }, 945 | { 946 | "cell_type": "code", 947 | "execution_count": null, 948 | "metadata": {}, 949 | "outputs": [], 950 | "source": [] 951 | } 952 | ], 953 | "metadata": { 954 | "kernelspec": { 955 | "display_name": "Python 3", 956 | "language": "python", 957 | "name": "python3" 958 | }, 959 | "language_info": { 960 | "codemirror_mode": { 961 | "name": "ipython", 962 | "version": 3 963 | }, 964 | "file_extension": ".py", 965 | "mimetype": "text/x-python", 966 | "name": "python", 967 | "nbconvert_exporter": "python", 968 | "pygments_lexer": "ipython3", 969 | "version": "3.5.2" 970 | } 971 | }, 972 | "nbformat": 4, 973 | "nbformat_minor": 2 974 | } 975 | -------------------------------------------------------------------------------- /PythonScripts/1DCNN_MODEL_StockText_2Class.py: -------------------------------------------------------------------------------- 1 | ############################################################################################################################################# 2 | # 3 | # Stock future performance classification based on text 4 | # 5 | # Approach: 6 | # 7 | # Build on top of it a 1D convolutional neural network, ending in a softmax output over 3 even categories. 8 | # Use word Glove word vectors for large English text corpus as inputs model 9 | # 10 | # Steps 11 | # 1) After cleaning, we convert all text samples in the dataset into sequences of word indices. In this case, a "word index" would simply be an integer ID for the word. 12 | # 2) We consider the top 350,000 most commonly occuring words in the dataset 13 | # 3) We truncate the sequences to a maximum length of 25,000 words. 14 | # 5) We [repare an "embedding matrix" which will contain at index i the embedding vector for the word of index i in our word index. 15 | # 6) Then, we load this embedding matrix into a Keras Embedding layer, set to be frozen (its weights, the embedding vectors, will not be updated during training). 16 | # 17 | ############################################################################################################################################### 18 | 19 | # import libraries 20 | from __future__ import print_function 21 | import numpy as np 22 | from six.moves import zip 23 | import json 24 | import warnings 25 | import pandas as pd 26 | from pandas import DataFrame 27 | import pickle 28 | import re 29 | import sys 30 | import azureml 31 | import string 32 | from scipy import stats 33 | import pip 34 | import keras 35 | from keras.preprocessing.sequence import pad_sequences 36 | from keras.utils.np_utils import to_categorical 37 | from keras.preprocessing.text import Tokenizer 38 | from keras.preprocessing import sequence 39 | from keras.models import Sequential 40 | from keras.layers import Dense, Embedding 41 | from keras.layers.core import Dropout 42 | from keras.layers import LSTM 43 | from keras.layers import Dense, Input, Flatten 44 | from keras.layers import Conv1D, MaxPooling1D, Embedding 45 | from keras.models import Model 46 | from keras.utils import plot_model 47 | from keras.utils.vis_utils import model_to_dot 48 | from keras.models import load_model 49 | from keras.applications import imagenet_utils 50 | from keras.layers import Embedding 51 | from keras.utils.np_utils import to_categorical 52 | from keras import initializers 53 | from keras import optimizers 54 | from keras.callbacks import EarlyStopping 55 | from keras.layers.normalization import BatchNormalization 56 | from keras.layers import initializers 57 | from keras.layers import regularizers 58 | from keras.layers import constraints 59 | from keras.layers import Activation 60 | from keras.layers.advanced_activations import PReLU 61 | from keras.layers.advanced_activations import LeakyReLU 62 | from keras.layers.advanced_activations import ELU 63 | from keras.constraints import max_norm 64 | import keras.backend as K 65 | import os 66 | import tempfile 67 | import logging 68 | import gensim 69 | from gensim.models import Phrases, phrases 70 | from gensim.models.phrases import Phraser 71 | from gensim.models import Word2Vec as wv 72 | import nltk 73 | from nltk.corpus import stopwords 74 | from gensim import corpora, models, similarities 75 | from IPython.display import SVG 76 | import cloudpickle 77 | import csv 78 | import mkl 79 | import matplotlib.pyplot as plt 80 | import h5py 81 | from keras.models import load_model 82 | import re 83 | import io 84 | from os.path import dirname, join 85 | import regex 86 | import graphviz 87 | import pydotplus 88 | import pyparsing 89 | from keras.utils import plot_model 90 | 91 | 92 | ########################################## 93 | # Get Previously Organized Stock Data 94 | ########################################## 95 | 96 | os.chdir('C:\\users\\pattyry\\documents\\AzureML\\NextAgenda_CodeStory\\NextAgenda_CodeStory') 97 | 98 | with open('biotechcleaned2bin.pkl', 'rb') as f: 99 | data = pickle.load(f, encoding='utf8') 100 | print("Data unpickled") 101 | data = pd.DataFrame(data) 102 | thedata = data 103 | 104 | np.random.seed(1337) # for reproducibility 105 | ################################# 106 | #If necessary, convert categories 107 | ################################# 108 | #thedata['ReturnQuantile'] = thedata['ReturnQuantile'].map({0:0,1:1,2:1,3:1,4:2}) 109 | print('Review the unique labels',thedata['Return2Bin_4Weeks'].unique()) 110 | 111 | ########################################## 112 | # clean up the text in the data with regex 113 | ########################################## 114 | #Most clean up already done in pre-processing script in a jupyter notebook. 115 | thedata['fulltext'] = thedata['fulltext'].str.encode('utf-8') 116 | thedata['fulltext'] = thedata['fulltext'].str.lower() 117 | 118 | def clean_text(row): 119 | text = str(row['fulltext']) 120 | 121 | # Remove newline characters 122 | cleantext = text.replace('\r\n', ' ') 123 | 124 | # Convert HTML punctuation chaaracters 125 | cleantext = cleantext.replace('.', '') 126 | 127 | #remove non alpha characters and specific noise 128 | #cleantext = re.sub(r'\d+', '',cleantext) 129 | cleantext = re.sub(r'^b','',cleantext) 130 | cleantext = re.sub(r'[^\w]',' ',cleantext) 131 | 132 | #remove specific noise 133 | cleantext = cleantext.translate(str.maketrans({'‘':' ','’':' '})) 134 | cleantext = cleantext.translate(str.maketrans({',':' ',',':' '})) 135 | cleantext = cleantext.translate(str.maketrans({'"':' ','%':' '})) 136 | 137 | #remove punctuation 138 | punctpattern = re.compile('[%s]' % re.escape(string.punctuation)) 139 | cleanttext = re.sub(punctpattern,'', cleantext) 140 | 141 | #remove single letter word 142 | cleantext = re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '', cleantext) 143 | 144 | # Remove extra spaces 145 | cleantext = re.sub('\s+', ' ', cleantext).strip() 146 | 147 | return cleantext 148 | 149 | #apply regex fixes to the input text column 150 | thedata['CleanText'] = thedata.apply(clean_text, axis=1) 151 | justcleandocs=thedata.drop(['fulltext'], axis=1) 152 | #save a cleaned copy to inspect 153 | justcleandocs.to_csv('C:\\glove\cleaneddata2.tsv', sep='\t', encoding='utf-8') 154 | 155 | 156 | ################################ 157 | # Convert labels to categorical 158 | ################################ 159 | 160 | justcleandocs=thedata.drop(['fulltext','Return2Bin_4Weeks'], axis=1) 161 | justcleandocs = justcleandocs['CleanText'] 162 | print('post regex justcleandocs',justcleandocs.head(10)) 163 | 164 | justlabels=thedata.drop(['fulltext','CleanText'], axis=1) 165 | justlabels=pd.DataFrame(justlabels['Return2Bin_4Weeks']) 166 | print('head of just labels',justlabels.head(5)) 167 | print(justlabels.head()) 168 | print(justlabels.tail()) 169 | print(justlabels['Return2Bin_4Weeks'].unique()) 170 | 171 | 172 | #################################################### 173 | # Set Global Vars 174 | #################################################### 175 | 176 | MAX_SEQUENCE_LENGTH = 10000 177 | MAX_NB_WORDS = 400000 178 | EMBEDDING_DIM = 300 179 | VALIDATION_SPLIT = 0.2 180 | LEARNING_RATE = .0003 181 | BATCH_SIZE = 128 182 | DROPOUT_RATE = 0.4 183 | np.random.seed(2032) 184 | 185 | #change directory to write results 186 | os.chdir('C:\\') 187 | BASE_DIR = '.' 188 | GLOVE_DIR = BASE_DIR + '/glove/' 189 | 190 | 191 | ###################################################### 192 | # Format our text samples and labels for use in Keras 193 | ###################################################### 194 | # Then we can format our text samples and labels into tensors that can be fed into a neural network. 195 | # Here we tokenize our source 'justcleandocs' 196 | # note that the values here are ultimately indexes to the actual words 197 | 198 | #convert text format 199 | justcleandocslist = justcleandocs.values 200 | justcleandocslist[6] 201 | labels = justlabels.values 202 | labels_index = {} 203 | #labels_index = {0:0,1:1,2:2,3:3,4:4} 204 | #labels_index = {0:0,1:1,2:2} 205 | labels_index = {0:0,1:1} 206 | print('labels_index', labels_index) 207 | 208 | #tokenize the text 209 | tokenizer = Tokenizer(num_words=MAX_NB_WORDS) 210 | tokenizer.fit_on_texts(justcleandocslist) #tokenizer.fit_on_texts(texts) 211 | sequences = tokenizer.texts_to_sequences(justcleandocslist) #sequences = tokenizer.texts_to_sequences(texts) 212 | word_index = tokenizer.word_index #word_index = tokenizer.word_index 213 | print('Found {} unique tokens'.format(len(word_index))) 214 | print('sequences first', sequences[0]) 215 | 216 | #Pad sequences so that they all have the same length in a batch of input data 217 | data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH, padding='pre', truncating='pre') 218 | sequences = None 219 | texts = None 220 | 221 | 222 | ################################################## 223 | #build label array from target y label in data set 224 | ################################################## 225 | labels = to_categorical(np.asarray(labels)) 226 | print('Shape of data tensor: ', data.shape) 227 | print('Shape of label tensor: ', labels.shape) 228 | 229 | indices = np.arange(data.shape[0]) 230 | np.random.shuffle(indices) 231 | data = data[indices] 232 | labels = labels[indices] 233 | nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0]) 234 | 235 | X_train = data[:-nb_validation_samples] 236 | y_train = labels[:-nb_validation_samples] 237 | X_val = data[-nb_validation_samples:] 238 | y_val = labels[-nb_validation_samples:] 239 | 240 | print('length of y_val',len(y_val)) 241 | print('shape of y_val',y_val.shape) 242 | print('length of X_val',len(X_val)) 243 | print('shape of X_val',X_val.shape) 244 | 245 | os.chdir('C:\\glove\\nextagenda') 246 | 247 | #from itertools import islice 248 | #head = list(islice(y_val, 6)) 249 | #print('head of yval',head) 250 | 251 | 252 | 253 | ##################################### 254 | # Save Validation Set for Evaluation 255 | #################################### 256 | np.savetxt('y_val_2bin.txt', y_val, delimiter=',') 257 | np.savetxt('X_val_2bin.txt', X_val, fmt='%s', delimiter=',') 258 | print('test and training sets saved to disk for later evaluation') 259 | 260 | 261 | ######################################## 262 | # Preparing the embedding layer 263 | ######################################## 264 | 265 | #load in word vectors from glove reference global English data set 266 | # https://nlp.stanford.edu/projects/glove/ 267 | # see more reference links at bottom 268 | 269 | print('Loading word vectors to prepare the embedding layer...') 270 | print(os.getcwd()) 271 | 272 | embeddings_index = {} 273 | print('Loading Glove Model...') 274 | gloveFile = 'C:\\glove\\glove6B300d.txt' 275 | words = pd.read_table(gloveFile, sep=" ", header=None, quoting=csv.QUOTE_NONE) 276 | 277 | print(words.head(5)) 278 | print('shape of glove model',words.shape) 279 | 280 | wordkeys=words.iloc[:,0] 281 | print('wordkeys type of file', type(wordkeys)) 282 | words2 = words.rename(columns={ words.columns[0]: "words" }) 283 | words2['words'].apply(str) 284 | #print(words2.dtypes) 285 | 286 | embeddings_index = words2.set_index('words').T.to_dict('list') 287 | 288 | #print(dict(list(embeddings_index.items())[0:2])) 289 | print('Found {} word vectors.'.format(len(embeddings_index))) 290 | #usage of pandas function dataFrame.to_dict(outtype='dict') outtype : str {‘dict’, ‘list’, ‘series’} 291 | 292 | 293 | ################################# 294 | #Build the embedding matrix 295 | ################################# 296 | 297 | print('Building Embedding Matrix...') 298 | embedding_matrix = np.zeros((len(word_index) + 1, EMBEDDING_DIM)) 299 | for word, i in word_index.items(): 300 | embedding_vector = embeddings_index.get(word) 301 | if embedding_vector is not None: 302 | # words not found in embedding index will be all-zeros. 303 | embedding_matrix[i] = embedding_vector 304 | 305 | embedding_layer = Embedding(len(word_index) + 1, 306 | EMBEDDING_DIM, 307 | weights=[embedding_matrix], 308 | input_length=MAX_SEQUENCE_LENGTH, 309 | trainable=False) 310 | 311 | 312 | ############################################## 313 | #Training a 1D convnet 314 | ############################################## 315 | 316 | print('Train 1D Convnet with global maxpooling') 317 | print('Shape of training data sample tensor: ', X_train.shape) 318 | print('Shape of training label tensor: ', y_train.shape) 319 | 320 | sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32') 321 | 322 | 323 | embedded_sequences = embedding_layer(sequence_input) 324 | 325 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(embedded_sequences) 326 | #x = BatchNormalization(axis=-1)(x) 327 | x = MaxPooling1D(5)(x) 328 | 329 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x) 330 | #x = BatchNormalization(axis=-1)(x) 331 | x = MaxPooling1D(5)(x) 332 | 333 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x) 334 | #x = BatchNormalization(axis=-1)(x) 335 | x = MaxPooling1D(35)(x) # global max pooling 336 | 337 | x = Flatten()(x) 338 | x = Dense(100, activation='elu', kernel_initializer='lecun_uniform')(x) # best initializers: #VarianceScaling #lecun_uniform 339 | #x = BatchNormalization(axis=-1)(x) 340 | x = Dropout(DROPOUT_RATE)(x) 341 | 342 | preds = Dense(len(labels_index), activation='softmax')(x) #no initialization in output layer 343 | 344 | 345 | model = Model(sequence_input, preds) 346 | ################################ 347 | #Compile model, set optimizers 348 | ################################ 349 | 350 | adam = optimizers.Adam(lr=LEARNING_RATE, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0, clipvalue=0.5)#, clipnorm=1.) 351 | rmsprop = optimizers.RMSprop(lr=LEARNING_RATE, rho=0.9, epsilon=1e-08, decay=0.00) 352 | 353 | model.compile(loss='categorical_crossentropy', 354 | optimizer= adam, 355 | metrics=['accuracy']) 356 | from keras.callbacks import History 357 | history = History() 358 | 359 | early_stopping = EarlyStopping(monitor='val_loss', patience=3) 360 | 361 | history = model.fit(X_train, y_train, 362 | batch_size=BATCH_SIZE, 363 | epochs=6, 364 | validation_data=(X_val, y_val), callbacks=[early_stopping, history]) 365 | 366 | 367 | ############################## 368 | # Save Model and Plots 369 | ############################## 370 | model.save('C:\\glove\\StockText_2EvenClass_model.h5') 371 | 372 | import matplotlib.pyplot as plt 373 | plt.figure(1) 374 | 375 | # summarize history for accuracy 376 | plt.subplot(211) 377 | plt.plot(history.history['acc']) 378 | plt.plot(history.history['val_acc']) 379 | plt.title('model accuracy') 380 | plt.ylabel('accuracy') 381 | plt.xlabel('epoch') 382 | plt.legend(['train', 'test'], loc='upper left') 383 | 384 | # summarize history for loss 385 | plt.subplot(212) 386 | plt.plot(history.history['loss']) 387 | plt.plot(history.history['val_loss']) 388 | plt.title('model loss') 389 | plt.ylabel('loss') 390 | plt.xlabel('epoch') 391 | plt.legend(['train', 'test'], loc='upper left') 392 | plt.show() 393 | 394 | #plot_model(model, to_file='C:\\glove\stocktext_model3class.png') 395 | 396 | #from IPython.display import SVG 397 | #from keras.utils.vis_utils import model_to_dot 398 | #SVG(model_to_dot(model).create(prog='dot', format='svg')) 399 | 400 | ############################## 401 | # More helpful links 402 | ############################## 403 | 404 | #We can also test how well we would have performed by not using pre-trained word embeddings, 405 | #but instead initializing our Embedding layer from scratch and learning its weights during training. 406 | 407 | #https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html 408 | #https://arxiv.org/abs/1603.03827 409 | #https://nlp.stanford.edu/projects/glove/ 410 | #https://stackoverflow.com/questions/37793118/load-pretrained-glove-vectors-in-python 411 | #https://stackoverflow.com/questions/27139908/load-precomputed-vectors-gensim?rq=1 412 | #https://stackoverflow.com/questions/14415741/numpy-array-vs-asarray -------------------------------------------------------------------------------- /PythonScripts/1DCNN_MODEL_StockText_3Class.py: -------------------------------------------------------------------------------- 1 | ############################################################################################################################################# 2 | # 3 | # Stock future performance classification based on text 4 | # 5 | # Approach: 6 | # 7 | # Build on top of it a 1D convolutional neural network, ending in a softmax output over 3 even categories. 8 | # Use word Glove word vectors for large English text corpus as inputs model 9 | # 10 | # Steps 11 | # 1) After cleaning, we convert all text samples in the dataset into sequences of word indices. In this case, a "word index" would simply be an integer ID for the word. 12 | # 2) We consider the top 350,000 most commonly occuring words in the dataset 13 | # 3) We truncate the sequences to a maximum length of 25,000 words. 14 | # 5) We [repare an "embedding matrix" which will contain at index i the embedding vector for the word of index i in our word index. 15 | # 6) Then, we load this embedding matrix into a Keras Embedding layer, set to be frozen (its weights, the embedding vectors, will not be updated during training). 16 | # 17 | ############################################################################################################################################### 18 | 19 | # import libraries 20 | from __future__ import print_function 21 | import numpy as np 22 | from six.moves import zip 23 | import json 24 | import warnings 25 | import pandas as pd 26 | from pandas import DataFrame 27 | import pickle 28 | import re 29 | import sys 30 | import azureml 31 | import string 32 | from scipy import stats 33 | import pip 34 | import keras 35 | from keras.preprocessing.sequence import pad_sequences 36 | from keras.utils.np_utils import to_categorical 37 | from keras.preprocessing.text import Tokenizer 38 | from keras.preprocessing import sequence 39 | from keras.models import Sequential 40 | from keras.layers import Dense, Embedding 41 | from keras.layers.core import Dropout 42 | from keras.layers import LSTM 43 | from keras.layers import Dense, Input, Flatten 44 | from keras.layers import Conv1D, MaxPooling1D, Embedding 45 | from keras.models import Model 46 | from keras.utils import plot_model 47 | from keras.utils.vis_utils import model_to_dot 48 | from keras.models import load_model 49 | from keras.applications import imagenet_utils 50 | from keras.layers import Embedding 51 | from keras.utils.np_utils import to_categorical 52 | from keras import initializers 53 | from keras import optimizers 54 | from keras.callbacks import EarlyStopping 55 | from keras.layers.normalization import BatchNormalization 56 | from keras.layers import initializers 57 | from keras.layers import regularizers 58 | from keras.layers import constraints 59 | from keras.layers import Activation 60 | from keras.layers.advanced_activations import PReLU 61 | from keras.layers.advanced_activations import LeakyReLU 62 | from keras.layers.advanced_activations import ELU 63 | from keras.constraints import max_norm 64 | import keras.backend as K 65 | import os 66 | import tempfile 67 | import logging 68 | import gensim 69 | from gensim.models import Phrases, phrases 70 | from gensim.models.phrases import Phraser 71 | from gensim.models import Word2Vec as wv 72 | import nltk 73 | from nltk.corpus import stopwords 74 | from gensim import corpora, models, similarities 75 | from IPython.display import SVG 76 | import cloudpickle 77 | import csv 78 | import mkl 79 | import matplotlib.pyplot as plt 80 | import h5py 81 | from keras.models import load_model 82 | import re 83 | import io 84 | from os.path import dirname, join 85 | import regex 86 | import graphviz 87 | import pydotplus 88 | import pyparsing 89 | from keras.utils import plot_model 90 | 91 | 92 | ########################################## 93 | # Get Previously Organized Stock Data 94 | ########################################## 95 | 96 | os.chdir('C:\\users\\pattyry\\documents\\AzureML\\NextAgenda_CodeStory\\NextAgenda_CodeStory') 97 | 98 | with open('biotechcleaned.pkl', 'rb') as f: 99 | data = pickle.load(f, encoding='utf8') 100 | print("Data unpickled") 101 | data = pd.DataFrame(data) 102 | thedata = data 103 | 104 | np.random.seed(1337) # for reproducibility 105 | ################################# 106 | #If necessary, convert categories 107 | ################################# 108 | #thedata['ReturnQuantile'] = thedata['ReturnQuantile'].map({0:0,1:1,2:1,3:1,4:2}) 109 | print('Review the unique labels',thedata['Return3Bin_4Weeks'].unique()) 110 | 111 | ########################################## 112 | # clean up the text in the data with regex 113 | ########################################## 114 | #Most clean up already done in pre-processing script in a jupyter notebook. 115 | thedata['fulltext'] = thedata['fulltext'].str.encode('utf-8') 116 | thedata['fulltext'] = thedata['fulltext'].str.lower() 117 | 118 | def clean_text(row): 119 | text = str(row['fulltext']) 120 | 121 | # Remove newline characters 122 | cleantext = text.replace('\r\n', ' ') 123 | 124 | # Convert HTML punctuation chaaracters 125 | cleantext = cleantext.replace('.', '') 126 | 127 | #remove non alpha characters and specific noise 128 | #cleantext = re.sub(r'\d+', '',cleantext) 129 | cleantext = re.sub(r'^b','',cleantext) 130 | cleantext = re.sub(r'[^\w]',' ',cleantext) 131 | 132 | #remove specific noise 133 | cleantext = cleantext.translate(str.maketrans({'‘':' ','’':' '})) 134 | cleantext = cleantext.translate(str.maketrans({',':' ',',':' '})) 135 | cleantext = cleantext.translate(str.maketrans({'"':' ','%':' '})) 136 | 137 | #remove punctuation 138 | punctpattern = re.compile('[%s]' % re.escape(string.punctuation)) 139 | cleanttext = re.sub(punctpattern,'', cleantext) 140 | 141 | #remove single letter word 142 | cleantext = re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '', cleantext) 143 | 144 | # Remove extra spaces 145 | cleantext = re.sub('\s+', ' ', cleantext).strip() 146 | 147 | return cleantext 148 | 149 | #apply regex fixes to the input text column 150 | thedata['CleanText'] = thedata.apply(clean_text, axis=1) 151 | justcleandocs=thedata.drop(['fulltext'], axis=1) 152 | #save a cleaned copy to inspect 153 | justcleandocs.to_csv('C:\\glove\cleaneddata2.tsv', sep='\t', encoding='utf-8') 154 | 155 | 156 | ################################ 157 | # Convert labels to categorical 158 | ################################ 159 | 160 | justcleandocs=thedata.drop(['fulltext','Return3Bin_4Weeks'], axis=1) 161 | justcleandocs = justcleandocs['CleanText'] 162 | print('post regex justcleandocs',justcleandocs.head(10)) 163 | 164 | justlabels=thedata.drop(['fulltext','CleanText'], axis=1) 165 | justlabels=pd.DataFrame(justlabels['Return3Bin_4Weeks']) 166 | print('head of just labels',justlabels.head(5)) 167 | print(justlabels.head()) 168 | print(justlabels.tail()) 169 | print(justlabels['Return3Bin_4Weeks'].unique()) 170 | 171 | 172 | #################################################### 173 | # Set Global Vars 174 | #################################################### 175 | 176 | MAX_SEQUENCE_LENGTH = 10000 177 | MAX_NB_WORDS = 400000 178 | EMBEDDING_DIM = 300 179 | VALIDATION_SPLIT = 0.2 180 | LEARNING_RATE = .0003 181 | BATCH_SIZE = 128 182 | DROPOUT_RATE = 0.4 183 | np.random.seed(2032) 184 | 185 | #change directory to write results 186 | os.chdir('C:\\') 187 | BASE_DIR = '.' 188 | GLOVE_DIR = BASE_DIR + '/glove/' 189 | 190 | 191 | ###################################################### 192 | # Format our text samples and labels for use in Keras 193 | ###################################################### 194 | # Then we can format our text samples and labels into tensors that can be fed into a neural network. 195 | # Here we tokenize our source 'justcleandocs' 196 | # note that the values here are ultimately indexes to the actual words 197 | 198 | #convert text format 199 | justcleandocslist = justcleandocs.values 200 | justcleandocslist[6] 201 | labels = justlabels.values 202 | labels_index = {} 203 | #labels_index = {0:0,1:1,2:2,3:3,4:4} 204 | labels_index = {0:0,1:1,2:2} 205 | print('labels_index', labels_index) 206 | 207 | #tokenize the text 208 | tokenizer = Tokenizer(num_words=MAX_NB_WORDS) 209 | tokenizer.fit_on_texts(justcleandocslist) #tokenizer.fit_on_texts(texts) 210 | sequences = tokenizer.texts_to_sequences(justcleandocslist) #sequences = tokenizer.texts_to_sequences(texts) 211 | word_index = tokenizer.word_index #word_index = tokenizer.word_index 212 | print('Found {} unique tokens'.format(len(word_index))) 213 | print('sequences first', sequences[0]) 214 | 215 | #Pad sequences so that they all have the same length in a batch of input data 216 | data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH, padding='pre', truncating='pre') 217 | sequences = None 218 | texts = None 219 | 220 | 221 | ################################################## 222 | #build label array from target y label in data set 223 | ################################################## 224 | labels = to_categorical(np.asarray(labels)) 225 | print('Shape of data tensor: ', data.shape) 226 | print('Shape of label tensor: ', labels.shape) 227 | 228 | indices = np.arange(data.shape[0]) 229 | np.random.shuffle(indices) 230 | data = data[indices] 231 | labels = labels[indices] 232 | nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0]) 233 | 234 | X_train = data[:-nb_validation_samples] 235 | y_train = labels[:-nb_validation_samples] 236 | X_val = data[-nb_validation_samples:] 237 | y_val = labels[-nb_validation_samples:] 238 | 239 | print('length of y_val',len(y_val)) 240 | print('shape of y_val',y_val.shape) 241 | print('length of X_val',len(X_val)) 242 | print('shape of X_val',X_val.shape) 243 | 244 | os.chdir('C:\\glove\\nextagenda') 245 | 246 | #from itertools import islice 247 | #head = list(islice(y_val, 6)) 248 | #print('head of yval',head) 249 | 250 | 251 | 252 | ##################################### 253 | # Save Validation Set for Evaluation 254 | #################################### 255 | np.savetxt('y_val_3bin.txt', y_val, delimiter=',') 256 | np.savetxt('X_val_3bin.txt', X_val, fmt='%s', delimiter=',') 257 | print('test and training sets saved to disk for later evaluation') 258 | 259 | 260 | ######################################## 261 | # Preparing the embedding layer 262 | ######################################## 263 | 264 | #load in word vectors from glove reference global English data set 265 | # https://nlp.stanford.edu/projects/glove/ 266 | # see more reference links at bottom 267 | 268 | print('Loading word vectors to prepare the embedding layer...') 269 | print(os.getcwd()) 270 | 271 | embeddings_index = {} 272 | print('Loading Glove Model...') 273 | gloveFile = 'C:\\glove\\glove6B300d.txt' 274 | words = pd.read_table(gloveFile, sep=" ", header=None, quoting=csv.QUOTE_NONE) 275 | 276 | print(words.head(5)) 277 | print('shape of glove model',words.shape) 278 | 279 | wordkeys=words.iloc[:,0] 280 | print('wordkeys type of file', type(wordkeys)) 281 | words2 = words.rename(columns={ words.columns[0]: "words" }) 282 | words2['words'].apply(str) 283 | #print(words2.dtypes) 284 | 285 | embeddings_index = words2.set_index('words').T.to_dict('list') 286 | 287 | #print(dict(list(embeddings_index.items())[0:2])) 288 | print('Found {} word vectors.'.format(len(embeddings_index))) 289 | #usage of pandas function dataFrame.to_dict(outtype='dict') outtype : str {‘dict’, ‘list’, ‘series’} 290 | 291 | 292 | ################################# 293 | #Build the embedding matrix 294 | ################################# 295 | 296 | print('Building Embedding Matrix...') 297 | embedding_matrix = np.zeros((len(word_index) + 1, EMBEDDING_DIM)) 298 | for word, i in word_index.items(): 299 | embedding_vector = embeddings_index.get(word) 300 | if embedding_vector is not None: 301 | # words not found in embedding index will be all-zeros. 302 | embedding_matrix[i] = embedding_vector 303 | 304 | embedding_layer = Embedding(len(word_index) + 1, 305 | EMBEDDING_DIM, 306 | weights=[embedding_matrix], 307 | input_length=MAX_SEQUENCE_LENGTH, 308 | trainable=False) 309 | 310 | 311 | ############################################## 312 | #Training a 1D convnet 313 | ############################################## 314 | 315 | print('Train 1D Convnet with global maxpooling') 316 | print('Shape of training data sample tensor: ', X_train.shape) 317 | print('Shape of training label tensor: ', y_train.shape) 318 | 319 | sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32') 320 | 321 | 322 | embedded_sequences = embedding_layer(sequence_input) 323 | 324 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(embedded_sequences) 325 | #x = BatchNormalization(axis=-1)(x) 326 | x = MaxPooling1D(5)(x) 327 | 328 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x) 329 | #x = BatchNormalization(axis=-1)(x) 330 | x = MaxPooling1D(5)(x) 331 | 332 | x = Conv1D(128, 5, activation='elu', kernel_initializer='lecun_uniform')(x) 333 | #x = BatchNormalization(axis=-1)(x) 334 | x = MaxPooling1D(35)(x) # global max pooling 335 | 336 | x = Flatten()(x) 337 | x = Dense(100, activation='elu', kernel_initializer='lecun_uniform')(x) # best initializers: #VarianceScaling #lecun_uniform 338 | #x = BatchNormalization(axis=-1)(x) 339 | x = Dropout(DROPOUT_RATE)(x) 340 | 341 | preds = Dense(len(labels_index), activation='softmax')(x) #no initialization in output layer 342 | 343 | 344 | model = Model(sequence_input, preds) 345 | ################################ 346 | #Compile model, set optimizers 347 | ################################ 348 | 349 | adam = optimizers.Adam(lr=LEARNING_RATE, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0, clipvalue=0.5)#, clipnorm=1.) 350 | rmsprop = optimizers.RMSprop(lr=LEARNING_RATE, rho=0.9, epsilon=1e-08, decay=0.00) 351 | 352 | model.compile(loss='categorical_crossentropy', 353 | optimizer= adam, 354 | metrics=['accuracy']) 355 | from keras.callbacks import History 356 | history = History() 357 | 358 | early_stopping = EarlyStopping(monitor='val_loss', patience=3) 359 | 360 | history = model.fit(X_train, y_train, 361 | batch_size=BATCH_SIZE, 362 | epochs=6, 363 | validation_data=(X_val, y_val), callbacks=[early_stopping, history]) 364 | 365 | 366 | ############################## 367 | # Save Model and Plots 368 | ############################## 369 | model.save('C:\\glove\\StockText_3Level3EvenClass_v1model3.h5') 370 | 371 | import matplotlib.pyplot as plt 372 | plt.figure(1) 373 | 374 | # summarize history for accuracy 375 | plt.subplot(211) 376 | plt.plot(history.history['acc']) 377 | plt.plot(history.history['val_acc']) 378 | plt.title('model accuracy') 379 | plt.ylabel('accuracy') 380 | plt.xlabel('epoch') 381 | plt.legend(['train', 'test'], loc='upper left') 382 | 383 | # summarize history for loss 384 | plt.subplot(212) 385 | plt.plot(history.history['loss']) 386 | plt.plot(history.history['val_loss']) 387 | plt.title('model loss') 388 | plt.ylabel('loss') 389 | plt.xlabel('epoch') 390 | plt.legend(['train', 'test'], loc='upper left') 391 | plt.show() 392 | 393 | #plot_model(model, to_file='C:\\glove\stocktext_model3class.png') 394 | 395 | #from IPython.display import SVG 396 | #from keras.utils.vis_utils import model_to_dot 397 | #SVG(model_to_dot(model).create(prog='dot', format='svg')) 398 | 399 | ############################## 400 | # More helpful links 401 | ############################## 402 | 403 | #We can also test how well we would have performed by not using pre-trained word embeddings, 404 | #but instead initializing our Embedding layer from scratch and learning its weights during training. 405 | 406 | #https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html 407 | #https://arxiv.org/abs/1603.03827 408 | #https://nlp.stanford.edu/projects/glove/ 409 | #https://stackoverflow.com/questions/37793118/load-pretrained-glove-vectors-in-python 410 | #https://stackoverflow.com/questions/27139908/load-precomputed-vectors-gensim?rq=1 411 | #https://stackoverflow.com/questions/14415741/numpy-array-vs-asarray -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Stock Performance Classification with a 1D CNN, Keras and Azure ML Workbench 3 | 4 | ## Overview 5 | We recently worked with a financial services partner to develop a model to predict the future stock market performance of public companies in categories where they invest. The goal was to use select text narrative sections from publicly available earnings release documents to predict and alert their analytics to investment opportunities and risks. We developed a deep learning model using a one-dimensional convolutional neural network (a 1D CNN) based on text extracted from public financial statements from these companies to make these predictions. We used Azure Machine Learning Workbench to explore the data and develop the model. We modeled using the Keras deep learning Python framework with a Theano backend. The results demonstrate how a deep learning model trained on text in earnings releases and other sources could provide a valuable signal to the investment decision maker. 6 | 7 | This initial result suggests that that deep learning models trained on text in earnings releases and other sources could prove a viable mechanism to improve the quality of the information available to the person making the investment decision, particularly in avoiding investment losses. While the model needs to be improved with more samples, refinements of domain-specific vocabulary and text augmentation, this model suggests that providing this signal as another decision input for investmennt analyst would improve the efficiency of the firm’s analysis work. 8 | 9 | The history of model training and testing is below, trained for 24 epochs. 10 | ![alt text](https://github.com/SingingData/StockPerformanceClassification/blob/master/images/Model_Training_accuracyandloss.png) 11 | 12 | ## Model Architecture 13 | This graphic depicts the model architecture. 14 | ![alt text](https://github.com/SingingData/StockPerformanceClassification/blob/master/images/modelarchitecture.png) 15 | 16 | This graphic gives you a summary of this model at each level. 17 | ![alt text](https://github.com/SingingData/StockPerformanceClassification/blob/master/images/ModelSummary.png) 18 | 19 | ## Azure ML Workbench 20 | We built the solution on the Azure ML Workbench python environment. We found the following installs and upgrades were required. 21 | 22 | Installs and Upgrades Required (Order is Important) 23 | - conda install mkl-service 24 | - conda install m2w64-toolchain 25 | - conda install -c anaconda libpython=2.0 26 | - pip install numpy==1.13.0 27 | - pip install keras #Remember to change back0end to theano in the backend folder w/in keras, init file 28 | - pip install theano 29 | - pip install gensim 30 | - pip install nltk 31 | - pip install cloudpickle 32 | - pip install --upgrade pandas 33 | - pip install --upgrade gensim 34 | - pip install importio 35 | - pip install matplotlib 36 | - pip install netwrokx 37 | - pip install h5py 38 | - pip install pyparsing==1.5.7 39 | - pip install graphviz 40 | - Install graphviz binaries linked in this issue https://stackoverflow.com/questions/36886711/keras-runtimeerror-failed-to-import-pydot-after-installing-graphviz-and-pyd 41 | - pip install pydotplus 42 | - In vis_utils.py comment out def _check_pydot(): lines 19 through 28 43 | 44 | 45 | -------------------------------------------------------------------------------- /images/1D CNN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SingingData/StockPerformanceClassification/2c0000be1ad0b9dca71981240c45f56a08736b89/images/1D CNN.png -------------------------------------------------------------------------------- /images/ModelSummary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SingingData/StockPerformanceClassification/2c0000be1ad0b9dca71981240c45f56a08736b89/images/ModelSummary.png -------------------------------------------------------------------------------- /images/Model_Training_accuracyandloss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SingingData/StockPerformanceClassification/2c0000be1ad0b9dca71981240c45f56a08736b89/images/Model_Training_accuracyandloss.png -------------------------------------------------------------------------------- /images/modelarchitecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SingingData/StockPerformanceClassification/2c0000be1ad0b9dca71981240c45f56a08736b89/images/modelarchitecture.png --------------------------------------------------------------------------------