├── .gitattributes ├── irisTestData.txt ├── .gitignore ├── README.md ├── irisTrainData.txt └── nn_backprop.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | 4 | # Custom for Visual Studio 5 | *.cs diff=csharp 6 | 7 | # Standard to msysgit 8 | *.doc diff=astextplain 9 | *.DOC diff=astextplain 10 | *.docx diff=astextplain 11 | *.DOCX diff=astextplain 12 | *.dot diff=astextplain 13 | *.DOT diff=astextplain 14 | *.pdf diff=astextplain 15 | *.PDF diff=astextplain 16 | *.rtf diff=astextplain 17 | *.RTF diff=astextplain 18 | -------------------------------------------------------------------------------- /irisTestData.txt: -------------------------------------------------------------------------------- 1 | 5.0,3.5,1.3,0.3,1,0,0 2 | 4.5,2.3,1.3,0.3,1,0,0 3 | 4.4,3.2,1.3,0.2,1,0,0 4 | 5.0,3.5,1.6,0.6,1,0,0 5 | 5.1,3.8,1.9,0.4,1,0,0 6 | 4.8,3.0,1.4,0.3,1,0,0 7 | 5.1,3.8,1.6,0.2,1,0,0 8 | 4.6,3.2,1.4,0.2,1,0,0 9 | 5.3,3.7,1.5,0.2,1,0,0 10 | 5.0,3.3,1.4,0.2,1,0,0 11 | 5.5,2.6,4.4,1.2,0,1,0 12 | 6.1,3.0,4.6,1.4,0,1,0 13 | 5.8,2.6,4.0,1.2,0,1,0 14 | 5.0,2.3,3.3,1.0,0,1,0 15 | 5.6,2.7,4.2,1.3,0,1,0 16 | 5.7,3.0,4.2,1.2,0,1,0 17 | 5.7,2.9,4.2,1.3,0,1,0 18 | 6.2,2.9,4.3,1.3,0,1,0 19 | 5.1,2.5,3.0,1.1,0,1,0 20 | 5.7,2.8,4.1,1.3,0,1,0 21 | 6.7,3.1,5.6,2.4,0,0,1 22 | 6.9,3.1,5.1,2.3,0,0,1 23 | 5.8,2.7,5.1,1.9,0,0,1 24 | 6.8,3.2,5.9,2.3,0,0,1 25 | 6.7,3.3,5.7,2.5,0,0,1 26 | 6.7,3.0,5.2,2.3,0,0,1 27 | 6.3,2.5,5.0,1.9,0,0,1 28 | 6.5,3.0,5.2,2.0,0,0,1 29 | 6.2,3.4,5.4,2.3,0,0,1 30 | 5.9,3.0,5.1,1.8,0,0,1 31 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Windows image file caches 2 | Thumbs.db 3 | ehthumbs.db 4 | 5 | # Folder config file 6 | Desktop.ini 7 | 8 | # Recycle Bin used on file shares 9 | $RECYCLE.BIN/ 10 | 11 | # Windows Installer files 12 | *.cab 13 | *.msi 14 | *.msm 15 | *.msp 16 | 17 | # Windows shortcuts 18 | *.lnk 19 | 20 | # ========================= 21 | # Operating System Files 22 | # ========================= 23 | 24 | # OSX 25 | # ========================= 26 | 27 | .DS_Store 28 | .AppleDouble 29 | .LSOverride 30 | 31 | # Thumbnails 32 | ._* 33 | 34 | # Files that might appear in the root of a volume 35 | .DocumentRevisions-V100 36 | .fseventsd 37 | .Spotlight-V100 38 | .TemporaryItems 39 | .Trashes 40 | .VolumeIcon.icns 41 | 42 | # Directories potentially created on remote AFP share 43 | .AppleDB 44 | .AppleDesktop 45 | Network Trash Folder 46 | Temporary Items 47 | .apdisk 48 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # IrisData 2 | ## Iris Data Example Python Numpy 3 | ### https://blogs.msdn.microsoft.com/uk_faculty_connection/2017/07/04/how-to-implement-the-backpropagation-using-python-and-numpy/ 4 | 5 | ### Azure Jupyter Notebook Demo Interactive Experiment https://notebooks.azure.com/n/ln6ojzL3dZY/notebooks/IrisDataDemo.ipynb 6 | 7 | The Iris Data Set has over 150 item records. Each item has four numeric predictor variables (often called features): sepal length and width, and petal length and width, followed by the species ("setosa," "versicolor" or "virginica"). 8 | 9 | - The demo program uses 1-of-N label encoding, so setosa = (1,0,0) and versicolor = (0,1,0) and virginica = (0,0,1). The goal is to predict species from sepal and petal length and width. 10 | 11 | - The 150-item dataset has 50 setosa items, followed by 50 versicolor, followed by 50 virginica. Before writing the demo program, There is a 120-item file of training data (using the first 30 of each species) and a 30-item file of test data (the leftover 10 of each species). 12 | 13 | - The demo program creates a simple neural network with four input nodes (one for each feature), five hidden processing nodes (the number of hidden nodes is a free parameter and must be determined by trial and error), and three output nodes (corresponding to encoded species). The demo loaded the training and test data into two matrices. 14 | 15 | - The back-propagation algorithm is iterative and you must supply a maximum number of iterations (50 in the demo) and a learning rate (0.050) that controls how much each weight and bias value changes in each iteration. Small learning rate values lead to slow but steady training. Large learning rates lead to quicker training at the risk of overshooting good weight and bias values. The max-iteration and leaning rate are free parameters. 16 | 17 | - The demo displays the value of the mean squared error, every 10 iterations during training. As you'll see shortly, there are two error types that are commonly used with back-propagation, and the choice of error type affects the back-propagation implementation. After training completed, the demo computed the classification accuracy of the resulting model on the training data (0.9333 = 112 out of 120 correct) and on the test data (0.9667 = 29 out of 30 correct). The classification accuracy on a set of test data is a very rough approximation of the accuracy you'd expect to see on new, previously unseen data. 18 | -------------------------------------------------------------------------------- /irisTrainData.txt: -------------------------------------------------------------------------------- 1 | 5.1,3.5,1.4,0.2,1,0,0 2 | 4.9,3.0,1.4,0.2,1,0,0 3 | 4.7,3.2,1.3,0.2,1,0,0 4 | 4.6,3.1,1.5,0.2,1,0,0 5 | 5.0,3.6,1.4,0.2,1,0,0 6 | 5.4,3.9,1.7,0.4,1,0,0 7 | 4.6,3.4,1.4,0.3,1,0,0 8 | 5.0,3.4,1.5,0.2,1,0,0 9 | 4.4,2.9,1.4,0.2,1,0,0 10 | 4.9,3.1,1.5,0.1,1,0,0 11 | 5.4,3.7,1.5,0.2,1,0,0 12 | 4.8,3.4,1.6,0.2,1,0,0 13 | 4.8,3.0,1.4,0.1,1,0,0 14 | 4.3,3.0,1.1,0.1,1,0,0 15 | 5.8,4.0,1.2,0.2,1,0,0 16 | 5.7,4.4,1.5,0.4,1,0,0 17 | 5.4,3.9,1.3,0.4,1,0,0 18 | 5.1,3.5,1.4,0.3,1,0,0 19 | 5.7,3.8,1.7,0.3,1,0,0 20 | 5.1,3.8,1.5,0.3,1,0,0 21 | 5.4,3.4,1.7,0.2,1,0,0 22 | 5.1,3.7,1.5,0.4,1,0,0 23 | 4.6,3.6,1.0,0.2,1,0,0 24 | 5.1,3.3,1.7,0.5,1,0,0 25 | 4.8,3.4,1.9,0.2,1,0,0 26 | 5.0,3.0,1.6,0.2,1,0,0 27 | 5.0,3.4,1.6,0.4,1,0,0 28 | 5.2,3.5,1.5,0.2,1,0,0 29 | 5.2,3.4,1.4,0.2,1,0,0 30 | 4.7,3.2,1.6,0.2,1,0,0 31 | 4.8,3.1,1.6,0.2,1,0,0 32 | 5.4,3.4,1.5,0.4,1,0,0 33 | 5.2,4.1,1.5,0.1,1,0,0 34 | 5.5,4.2,1.4,0.2,1,0,0 35 | 4.9,3.1,1.5,0.2,1,0,0 36 | 5.0,3.2,1.2,0.2,1,0,0 37 | 5.5,3.5,1.3,0.2,1,0,0 38 | 4.9,3.6,1.4,0.1,1,0,0 39 | 4.4,3.0,1.3,0.2,1,0,0 40 | 5.1,3.4,1.5,0.2,1,0,0 41 | 7.0,3.2,4.7,1.4,0,1,0 42 | 6.4,3.2,4.5,1.5,0,1,0 43 | 6.9,3.1,4.9,1.5,0,1,0 44 | 5.5,2.3,4.0,1.3,0,1,0 45 | 6.5,2.8,4.6,1.5,0,1,0 46 | 5.7,2.8,4.5,1.3,0,1,0 47 | 6.3,3.3,4.7,1.6,0,1,0 48 | 4.9,2.4,3.3,1.0,0,1,0 49 | 6.6,2.9,4.6,1.3,0,1,0 50 | 5.2,2.7,3.9,1.4,0,1,0 51 | 5.0,2.0,3.5,1.0,0,1,0 52 | 5.9,3.0,4.2,1.5,0,1,0 53 | 6.0,2.2,4.0,1.0,0,1,0 54 | 6.1,2.9,4.7,1.4,0,1,0 55 | 5.6,2.9,3.6,1.3,0,1,0 56 | 6.7,3.1,4.4,1.4,0,1,0 57 | 5.6,3.0,4.5,1.5,0,1,0 58 | 5.8,2.7,4.1,1.0,0,1,0 59 | 6.2,2.2,4.5,1.5,0,1,0 60 | 5.6,2.5,3.9,1.1,0,1,0 61 | 5.9,3.2,4.8,1.8,0,1,0 62 | 6.1,2.8,4.0,1.3,0,1,0 63 | 6.3,2.5,4.9,1.5,0,1,0 64 | 6.1,2.8,4.7,1.2,0,1,0 65 | 6.4,2.9,4.3,1.3,0,1,0 66 | 6.6,3.0,4.4,1.4,0,1,0 67 | 6.8,2.8,4.8,1.4,0,1,0 68 | 6.7,3.0,5.0,1.7,0,1,0 69 | 6.0,2.9,4.5,1.5,0,1,0 70 | 5.7,2.6,3.5,1.0,0,1,0 71 | 5.5,2.4,3.8,1.1,0,1,0 72 | 5.5,2.4,3.7,1.0,0,1,0 73 | 5.8,2.7,3.9,1.2,0,1,0 74 | 6.0,2.7,5.1,1.6,0,1,0 75 | 5.4,3.0,4.5,1.5,0,1,0 76 | 6.0,3.4,4.5,1.6,0,1,0 77 | 6.7,3.1,4.7,1.5,0,1,0 78 | 6.3,2.3,4.4,1.3,0,1,0 79 | 5.6,3.0,4.1,1.3,0,1,0 80 | 5.5,2.5,4.0,1.3,0,1,0 81 | 6.3,3.3,6.0,2.5,0,0,1 82 | 5.8,2.7,5.1,1.9,0,0,1 83 | 7.1,3.0,5.9,2.1,0,0,1 84 | 6.3,2.9,5.6,1.8,0,0,1 85 | 6.5,3.0,5.8,2.2,0,0,1 86 | 7.6,3.0,6.6,2.1,0,0,1 87 | 4.9,2.5,4.5,1.7,0,0,1 88 | 7.3,2.9,6.3,1.8,0,0,1 89 | 6.7,2.5,5.8,1.8,0,0,1 90 | 7.2,3.6,6.1,2.5,0,0,1 91 | 6.5,3.2,5.1,2.0,0,0,1 92 | 6.4,2.7,5.3,1.9,0,0,1 93 | 6.8,3.0,5.5,2.1,0,0,1 94 | 5.7,2.5,5.0,2.0,0,0,1 95 | 5.8,2.8,5.1,2.4,0,0,1 96 | 6.4,3.2,5.3,2.3,0,0,1 97 | 6.5,3.0,5.5,1.8,0,0,1 98 | 7.7,3.8,6.7,2.2,0,0,1 99 | 7.7,2.6,6.9,2.3,0,0,1 100 | 6.0,2.2,5.0,1.5,0,0,1 101 | 6.9,3.2,5.7,2.3,0,0,1 102 | 5.6,2.8,4.9,2.0,0,0,1 103 | 7.7,2.8,6.7,2.0,0,0,1 104 | 6.3,2.7,4.9,1.8,0,0,1 105 | 6.7,3.3,5.7,2.1,0,0,1 106 | 7.2,3.2,6.0,1.8,0,0,1 107 | 6.2,2.8,4.8,1.8,0,0,1 108 | 6.1,3.0,4.9,1.8,0,0,1 109 | 6.4,2.8,5.6,2.1,0,0,1 110 | 7.2,3.0,5.8,1.6,0,0,1 111 | 7.4,2.8,6.1,1.9,0,0,1 112 | 7.9,3.8,6.4,2.0,0,0,1 113 | 6.4,2.8,5.6,2.2,0,0,1 114 | 6.3,2.8,5.1,1.5,0,0,1 115 | 6.1,2.6,5.6,1.4,0,0,1 116 | 7.7,3.0,6.1,2.3,0,0,1 117 | 6.3,3.4,5.6,2.4,0,0,1 118 | 6.4,3.1,5.5,1.8,0,0,1 119 | 6.0,3.0,4.8,1.8,0,0,1 120 | 6.9,3.1,5.4,2.1,0,0,1 121 | -------------------------------------------------------------------------------- /nn_backprop.py: -------------------------------------------------------------------------------- 1 | # nn_backprop.py 2 | # Python 3.x 3 | 4 | import numpy as np 5 | import random 6 | import math 7 | import sys 8 | 9 | # helper functions 10 | 11 | def loadFile(df): 12 | # load a comma-delimited text file into an np matrix 13 | resultList = [] 14 | f = open(df, 'r') 15 | for line in f: 16 | line = line.rstrip('\n') # "1.0,2.0,3.0" 17 | sVals = line.split(',') # ["1.0", "2.0, "3.0"] 18 | fVals = list(map(np.float32, sVals)) # [1.0, 2.0, 3.0] 19 | resultList.append(fVals) # [[1.0, 2.0, 3.0] , [4.0, 5.0, 6.0]] 20 | f.close() 21 | return np.asarray(resultList, dtype=np.float32) # not necessary 22 | # end loadFile 23 | 24 | def showVector(v, dec): 25 | fmt = "%." + str(dec) + "f" # like %.4f 26 | for i in range(len(v)): 27 | x = v[i] 28 | if x >= 0.0: print(' ', end='') 29 | print(fmt % x + ' ', end='') 30 | print('') 31 | 32 | def showMatrix(m, dec): 33 | fmt = "%." + str(dec) + "f" # like %.4f 34 | for i in range(len(m)): 35 | for j in range(len(m[i])): 36 | x = m[i,j] 37 | if x >= 0.0: print(' ', end='') 38 | print(fmt % x + ' ', end='') 39 | print('') 40 | 41 | def showMatrixPartial(m, numRows, dec, indices): 42 | fmt = "%." + str(dec) + "f" # like %.4f 43 | lastRow = len(m) - 1 44 | width = len(str(lastRow)) 45 | for i in range(numRows): 46 | if indices == True: 47 | print("[", end='') 48 | print(str(i).rjust(width), end='') 49 | print("] ", end='') 50 | 51 | for j in range(len(m[i])): 52 | x = m[i,j] 53 | if x >= 0.0: print(' ', end='') 54 | print(fmt % x + ' ', end='') 55 | print('') 56 | print(" . . . ") 57 | 58 | if indices == True: 59 | print("[", end='') 60 | print(str(lastRow).rjust(width), end='') 61 | print("] ", end='') 62 | for j in range(len(m[lastRow])): 63 | x = m[lastRow,j] 64 | if x >= 0.0: print(' ', end='') 65 | print(fmt % x + ' ', end='') 66 | print('') 67 | 68 | # ----- 69 | 70 | class NeuralNetwork: 71 | 72 | def __init__(self, numInput, numHidden, numOutput, seed): 73 | self.ni = numInput 74 | self.nh = numHidden 75 | self.no = numOutput 76 | 77 | self.iNodes = np.zeros(shape=[self.ni], dtype=np.float32) 78 | self.hNodes = np.zeros(shape=[self.nh], dtype=np.float32) 79 | self.oNodes = np.zeros(shape=[self.no], dtype=np.float32) 80 | 81 | self.ihWeights = np.zeros(shape=[self.ni,self.nh], dtype=np.float32) 82 | self.hoWeights = np.zeros(shape=[self.nh,self.no], dtype=np.float32) 83 | 84 | self.hBiases = np.zeros(shape=[self.nh], dtype=np.float32) 85 | self.oBiases = np.zeros(shape=[self.no], dtype=np.float32) 86 | 87 | self.rnd = random.Random(seed) # allows multiple instances 88 | self.initializeWeights() 89 | 90 | def setWeights(self, weights): 91 | if len(weights) != self.totalWeights(self.ni, self.nh, self.no): 92 | print("Warning: len(weights) error in setWeights()") 93 | 94 | idx = 0 95 | for i in range(self.ni): 96 | for j in range(self.nh): 97 | self.ihWeights[i,j] = weights[idx] 98 | idx += 1 99 | 100 | for j in range(self.nh): 101 | self.hBiases[j] = weights[idx] 102 | idx += 1 103 | 104 | for j in range(self.nh): 105 | for k in range(self.no): 106 | self.hoWeights[j,k] = weights[idx] 107 | idx += 1 108 | 109 | for k in range(self.no): 110 | self.oBiases[k] = weights[idx] 111 | idx += 1 112 | 113 | def getWeights(self): 114 | tw = self.totalWeights(self.ni, self.nh, self.no) 115 | result = np.zeros(shape=[tw], dtype=np.float32) 116 | idx = 0 # points into result 117 | 118 | for i in range(self.ni): 119 | for j in range(self.nh): 120 | result[idx] = self.ihWeights[i,j] 121 | idx += 1 122 | 123 | for j in range(self.nh): 124 | result[idx] = self.hBiases[j] 125 | idx += 1 126 | 127 | for j in range(self.nh): 128 | for k in range(self.no): 129 | result[idx] = self.hoWeights[j,k] 130 | idx += 1 131 | 132 | for k in range(self.no): 133 | result[idx] = self.oBiases[k] 134 | idx += 1 135 | 136 | return result 137 | 138 | def initializeWeights(self): 139 | numWts = self.totalWeights(self.ni, self.nh, self.no) 140 | wts = np.zeros(shape=[numWts], dtype=np.float32) 141 | lo = -0.01; hi = 0.01 142 | for idx in range(len(wts)): 143 | wts[idx] = (hi - lo) * self.rnd.random() + lo 144 | self.setWeights(wts) 145 | 146 | def computeOutputs(self, xValues): 147 | hSums = np.zeros(shape=[self.nh], dtype=np.float32) 148 | oSums = np.zeros(shape=[self.no], dtype=np.float32) 149 | 150 | for i in range(self.ni): 151 | self.iNodes[i] = xValues[i] 152 | 153 | for j in range(self.nh): 154 | for i in range(self.ni): 155 | hSums[j] += self.iNodes[i] * self.ihWeights[i,j] 156 | 157 | for j in range(self.nh): 158 | hSums[j] += self.hBiases[j] 159 | 160 | for j in range(self.nh): 161 | self.hNodes[j] = self.hypertan(hSums[j]) 162 | 163 | for k in range(self.no): 164 | for j in range(self.nh): 165 | oSums[k] += self.hNodes[j] * self.hoWeights[j,k] 166 | 167 | for k in range(self.no): 168 | oSums[k] += self.oBiases[k] 169 | 170 | softOut = self.softmax(oSums) 171 | for k in range(self.no): 172 | self.oNodes[k] = softOut[k] 173 | 174 | result = np.zeros(shape=self.no, dtype=np.float32) 175 | for k in range(self.no): 176 | result[k] = self.oNodes[k] 177 | 178 | return result 179 | 180 | def train(self, trainData, maxEpochs, learnRate): 181 | hoGrads = np.zeros(shape=[self.nh, self.no], dtype=np.float32) # hidden-to-output weights gradients 182 | obGrads = np.zeros(shape=[self.no], dtype=np.float32) # output node biases gradients 183 | ihGrads = np.zeros(shape=[self.ni, self.nh], dtype=np.float32) # input-to-hidden weights gradients 184 | hbGrads = np.zeros(shape=[self.nh], dtype=np.float32) # hidden biases gradients 185 | 186 | oSignals = np.zeros(shape=[self.no], dtype=np.float32) # output signals: gradients w/o assoc. input terms 187 | hSignals = np.zeros(shape=[self.nh], dtype=np.float32) # hidden signals: gradients w/o assoc. input terms 188 | 189 | epoch = 0 190 | x_values = np.zeros(shape=[self.ni], dtype=np.float32) 191 | t_values = np.zeros(shape=[self.no], dtype=np.float32) 192 | numTrainItems = len(trainData) 193 | indices = np.arange(numTrainItems) # [0, 1, 2, . . n-1] # rnd.shuffle(v) 194 | 195 | while epoch < maxEpochs: 196 | self.rnd.shuffle(indices) # scramble order of training items 197 | for ii in range(numTrainItems): 198 | idx = indices[ii] 199 | 200 | for j in range(self.ni): 201 | x_values[j] = trainData[idx, j] # get the input values 202 | for j in range(self.no): 203 | t_values[j] = trainData[idx, j+self.ni] # get the target values 204 | self.computeOutputs(x_values) # results stored internally 205 | 206 | # 1. compute output node signals 207 | for k in range(self.no): 208 | derivative = (1 - self.oNodes[k]) * self.oNodes[k] # softmax 209 | oSignals[k] = derivative * (self.oNodes[k] - t_values[k]) # E=(t-o)^2 do E'=(o-t) 210 | 211 | # 2. compute hidden-to-output weight gradients using output signals 212 | for j in range(self.nh): 213 | for k in range(self.no): 214 | hoGrads[j, k] = oSignals[k] * self.hNodes[j] 215 | 216 | # 3. compute output node bias gradients using output signals 217 | for k in range(self.no): 218 | obGrads[k] = oSignals[k] * 1.0 # 1.0 dummy input can be dropped 219 | 220 | # 4. compute hidden node signals 221 | for j in range(self.nh): 222 | sum = 0.0 223 | for k in range(self.no): 224 | sum += oSignals[k] * self.hoWeights[j,k] 225 | derivative = (1 - self.hNodes[j]) * (1 + self.hNodes[j]) # tanh activation 226 | hSignals[j] = derivative * sum 227 | 228 | # 5 compute input-to-hidden weight gradients using hidden signals 229 | for i in range(self.ni): 230 | for j in range(self.nh): 231 | ihGrads[i, j] = hSignals[j] * self.iNodes[i] 232 | 233 | # 6. compute hidden node bias gradients using hidden signals 234 | for j in range(self.nh): 235 | hbGrads[j] = hSignals[j] * 1.0 # 1.0 dummy input can be dropped 236 | 237 | # update weights and biases using the gradients 238 | 239 | # 1. update input-to-hidden weights 240 | for i in range(self.ni): 241 | for j in range(self.nh): 242 | delta = -1.0 * learnRate * ihGrads[i,j] 243 | self.ihWeights[i, j] += delta 244 | 245 | # 2. update hidden node biases 246 | for j in range(self.nh): 247 | delta = -1.0 * learnRate * hbGrads[j] 248 | self.hBiases[j] += delta 249 | 250 | # 3. update hidden-to-output weights 251 | for j in range(self.nh): 252 | for k in range(self.no): 253 | delta = -1.0 * learnRate * hoGrads[j,k] 254 | self.hoWeights[j, k] += delta 255 | 256 | # 4. update output node biases 257 | for k in range(self.no): 258 | delta = -1.0 * learnRate * obGrads[k] 259 | self.oBiases[k] += delta 260 | 261 | epoch += 1 262 | 263 | if epoch % 10 == 0: 264 | mse = self.meanSquaredError(trainData) 265 | print("epoch = " + str(epoch) + " ms error = %0.4f " % mse) 266 | # end while 267 | 268 | result = self.getWeights() 269 | return result 270 | # end train 271 | 272 | def accuracy(self, tdata): # train or test data matrix 273 | num_correct = 0; num_wrong = 0 274 | x_values = np.zeros(shape=[self.ni], dtype=np.float32) 275 | t_values = np.zeros(shape=[self.no], dtype=np.float32) 276 | 277 | for i in range(len(tdata)): # walk thru each data item 278 | for j in range(self.ni): # peel off input values from curr data row 279 | x_values[j] = tdata[i,j] 280 | for j in range(self.no): # peel off tareget values from curr data row 281 | t_values[j] = tdata[i, j+self.ni] 282 | 283 | y_values = self.computeOutputs(x_values) # computed output values) 284 | max_index = np.argmax(y_values) # index of largest output value 285 | 286 | if abs(t_values[max_index] - 1.0) < 1.0e-5: 287 | num_correct += 1 288 | else: 289 | num_wrong += 1 290 | 291 | return (num_correct * 1.0) / (num_correct + num_wrong) 292 | 293 | def meanSquaredError(self, tdata): # on train or test data matrix 294 | sumSquaredError = 0.0 295 | x_values = np.zeros(shape=[self.ni], dtype=np.float32) 296 | t_values = np.zeros(shape=[self.no], dtype=np.float32) 297 | 298 | for ii in range(len(tdata)): # walk thru each data item 299 | for jj in range(self.ni): # peel off input values from curr data row 300 | x_values[jj] = tdata[ii, jj] 301 | for jj in range(self.no): # peel off tareget values from curr data row 302 | t_values[jj] = tdata[ii, jj+self.ni] 303 | 304 | y_values = self.computeOutputs(x_values) # computed output values 305 | 306 | for j in range(self.no): 307 | err = t_values[j] - y_values[j] 308 | sumSquaredError += err * err # (t-o)^2 309 | 310 | return sumSquaredError / len(tdata) 311 | 312 | @staticmethod 313 | def hypertan(x): 314 | if x < -20.0: 315 | return -1.0 316 | elif x > 20.0: 317 | return 1.0 318 | else: 319 | return math.tanh(x) 320 | 321 | @staticmethod 322 | def softmax(oSums): 323 | result = np.zeros(shape=[len(oSums)], dtype=np.float32) 324 | m = max(oSums) 325 | divisor = 0.0 326 | for k in range(len(oSums)): 327 | divisor += math.exp(oSums[k] - m) 328 | for k in range(len(result)): 329 | result[k] = math.exp(oSums[k] - m) / divisor 330 | return result 331 | 332 | @staticmethod 333 | def totalWeights(nInput, nHidden, nOutput): 334 | tw = (nInput * nHidden) + (nHidden * nOutput) + nHidden + nOutput 335 | return tw 336 | 337 | # end class NeuralNetwork 338 | 339 | def main(): 340 | print("\nBegin NN back-propagation demo \n") 341 | pv = sys.version 342 | npv = np.version.version 343 | print("Using Python version " + str(pv) + 344 | "\n and NumPy version " + str(npv)) 345 | 346 | numInput = 4 347 | numHidden = 5 348 | numOutput = 3 349 | print("\nCreating a %d-%d-%d neural network " % 350 | (numInput, numHidden, numOutput) ) 351 | nn = NeuralNetwork(numInput, numHidden, numOutput, seed=3) 352 | 353 | print("\nLoading Iris training and test data ") 354 | trainDataPath = "irisTrainData.txt" 355 | trainDataMatrix = loadFile(trainDataPath) 356 | print("\nTest data: ") 357 | showMatrixPartial(trainDataMatrix, 4, 1, True) 358 | testDataPath = "irisTestData.txt" 359 | testDataMatrix = loadFile(testDataPath) 360 | 361 | maxEpochs = 50 362 | learnRate = 0.05 363 | print("\nSetting maxEpochs = " + str(maxEpochs)) 364 | print("Setting learning rate = %0.3f " % learnRate) 365 | print("\nStarting training") 366 | nn.train(trainDataMatrix, maxEpochs, learnRate) 367 | print("Training complete") 368 | 369 | accTrain = nn.accuracy(trainDataMatrix) 370 | accTest = nn.accuracy(testDataMatrix) 371 | 372 | print("\nAccuracy on 120-item train data = %0.4f " % accTrain) 373 | print("Accuracy on 30-item test data = %0.4f " % accTest) 374 | 375 | print("\nEnd demo \n") 376 | 377 | if __name__ == "__main__": 378 | main() 379 | 380 | # end script 381 | 382 | --------------------------------------------------------------------------------