├── .gitignore ├── CMakeLists.txt ├── README.html ├── README.md ├── README.pdf ├── data ├── testdata.txt └── traindata.txt ├── img └── net-info.png ├── index.html ├── lib ├── Config.h ├── Net.cpp ├── Net.h ├── Utils.cpp └── Utils.h ├── main.cpp └── updatePages.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | 3 | cmake-build-debug 4 | 5 | test 6 | test.dSYM/ 7 | -------------------------------------------------------------------------------- /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8.12) 2 | project(BPNN) 3 | 4 | set(CMAKE_CXX_STANDARD 11) 5 | 6 | add_executable(BPNN main.cpp lib/Utils.cpp lib/Utils.h lib/Config.h lib/Net.cpp lib/Net.h) 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Back-Propagation-Neural-Network 2 | 3 | [![BPNN](https://badgen.net/badge/github/BPNN?icon&label=GitHub)](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network) [![C++](https://img.shields.io/badge/support-C%2B%2B11%20or%20later-blue?style=flat&logo=cplusplus)](https://github.com/topics/cpp) [![CMake](https://img.shields.io/badge/support-v2.8.12%20or%20later-blue?style=flat&logo=cmake)](https://cmake.org/) [![update](https://img.shields.io/github/last-commit/GavinTechStudio/Back-Propagation-Neural-Network)](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network/commits) [![pages-build-deployment](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network/actions/workflows/pages/pages-build-deployment) 4 | 5 | [本项目](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network)是对项目 [**GavinTechStudio/bpnn_with_cpp**](https://github.com/GavinTechStudio/bpnn_with_cpp) 的代码重构，基于C++实现基础BP神经网络，有助于深入理解BP神经网络原理。 6 | 7 | ## 项目结构 8 | 9 | ``` 10 | . 11 | ├── CMakeLists.txt 12 | ├── data 13 | │   ├── testdata.txt 14 | │   └── traindata.txt 15 | ├── lib 16 | │   ├── Config.h 17 | │   ├── Net.cpp 18 | │   ├── Net.h 19 | │   ├── Utils.cpp 20 | │   └── Utils.h 21 | └── main.cpp 22 | ``` 23 | 24 | #### 主要框架 25 | 26 | - Net：网络具体实现 27 | - Config：网络参数设置 28 | - Utils：工具类 29 | - 数据加载 30 | - 激活函数 31 | - main：网络具体应用 32 | 33 | ## 训练原理 34 | 35 | > 具体公式推导请看视频讲解[彻底搞懂BP神经网络 理论推导+代码实现（C++）](https://www.bilibili.com/video/BV1Y64y1z7jM?p=1) 36 | 37 | #### 注意：本部分文档包含大量数学公式，由于GitHub markdown不支持数学公式渲染，推荐以下阅读方式： 38 | 39 | 1. 如果您使用的是Chrome、Edge、Firefox等浏览器，可以安装插件[MathJax Plugin for Github](https://chrome.google.com/webstore/detail/mathjax-plugin-for-github/ioemnmodlmafdkllaclgeombjnmnbima)（需要网络能够访问chrome web store）。 40 | 2. 使用[**PDF**](https://gavintechstudio.github.io/Back-Propagation-Neural-Network/README.pdf)的方式进行阅读。 41 | 2. 使用[**预渲染的静态网页**](https://gavintechstudio.github.io/Back-Propagation-Neural-Network/README.html)进行阅读（**推荐**）。 42 | 2. 按`.`键或[点击链接](https://github.dev/GavinTechStudio/Back-Propagation-Neural-Network)进入GitHub在线IDE预览`README.md`文件。 43 | 44 | ### 0. 神经网络结构图 45 | 46 | ![](img/net-info.png) 47 | 48 | ### 1. Forward（前向传播） 49 | 50 | #### 1.1 输入层向隐藏层传播 51 | 52 | $$ 53 | h_j = \sigma( \sum_i x_i w_{ij} - \beta_j ) 54 | $$ 55 | 56 | 其中$h_j$为第$j$个隐藏层节点的值，$x_i$为第$i$个输入层节点的值，$w_{ij}$为第$i$个输入层节点到第$j$个隐藏层节点的权重，$\beta_j$为第$j$个隐藏层节点偏置值，$\sigma(x)$为**Sigmoid**激活函数，后续也将继续用这个表达，其表达式如下 57 | $$ 58 | \sigma(x) = \frac{1}{1+e^{-x}} 59 | $$ 60 | 本项目中的代码实现如下： 61 | 62 | ```C++ 63 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 64 | double sum = 0; 65 | for (size_t i = 0; i < Config::INNODE; ++i) { 66 | sum += inputLayer[i]->value * 67 | inputLayer[i]->weight[j]; 68 | } 69 | sum -= hiddenLayer[j]->bias; 70 | 71 | hiddenLayer[j]->value = Utils::sigmoid(sum); 72 | } 73 | ``` 74 | 75 | #### 1.2 隐藏层向输出层传播 76 | 77 | $$ 78 | \hat{y_k} = \sigma( \sum_j h_j v_{jk} - \lambda_k ) 79 | $$ 80 | 81 | 其中$\hat{y_k}$为第$k$个输出层节点的值（预测值），$h_j$为第$j$个隐藏层节点的值，$v_{jk}$为第$j$个隐藏层节点到第$k$个输出层节点的权重，$\lambda_k$为第$k$个输出层节点的偏置值，$\sigma(x)$为激活函数。 82 | 83 | 本项目中的代码实现如下： 84 | 85 | ```C++ 86 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 87 | double sum = 0; 88 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 89 | sum += hiddenLayer[j]->value * 90 | hiddenLayer[j]->weight[k]; 91 | } 92 | sum -= outputLayer[k]->bias; 93 | 94 | outputLayer[k]->value = Utils::sigmoid(sum); 95 | } 96 | ``` 97 | 98 | ### 2. 计算损失函数（Loss Function） 99 | 100 | 损失函数定义如下： 101 | 102 | $$ 103 | Loss = \frac{1}{2}\sum_k ( y_k - \hat{y_k} )^2 104 | $$ 105 | 106 | 其中$y_k$为第$k$个输出层节点的目标值（真实值），$\hat{y_k}$为第$k$个输出层节点的值（预测值）。 107 | 108 | 本项目中的代码实现如下： 109 | 110 | ```C++ 111 | double loss = 0.f; 112 | 113 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 114 | double tmp = std::fabs(outputLayer[k]->value - label[k]); 115 | los += tmp * tmp / 2; 116 | } 117 | ``` 118 | 119 | ### 3. Backward（反向传播） 120 | 121 | 利用梯度下降法进行优化。 122 | 123 | #### 3.1 计算$\Delta \lambda_k$（输出层节点偏置值的修正值） 124 | 125 | 其计算公式如下（激活函数为Sigmoid时）： 126 | 127 | $$ 128 | \Delta \lambda_k = - \eta (y_k - \hat{y_k}) \hat{y_k} (1 - \hat{y_k}) 129 | $$ 130 | 131 | 其中$\eta$为学习率（其余变量上方已出现过不再进行标注）。 132 | 133 | 本项目中的代码实现如下： 134 | 135 | ```C++ 136 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 137 | double bias_delta = 138 | -(label[k] - outputLayer[k]->value) * 139 | outputLayer[k]->value * 140 | (1.0 - outputLayer[k]->value); 141 | 142 | outputLayer[k]->bias_delta += bias_delta; 143 | } 144 | ``` 145 | 146 | #### 3.2 计算$\Delta v_{jk}$（隐藏层节点到输出层节点权重的修正值） 147 | 148 | 其计算公式如下（激活函数为Sigmoid时）： 149 | 150 | $$ 151 | \Delta v_{jk} = \eta ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) h_j 152 | $$ 153 | 154 | 其中$h_j$为第$j$个隐藏层节点的值（其余变量上方已出现过不再进行标注）。 155 | 156 | 本项目中的代码实现如下： 157 | 158 | ```C++ 159 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 160 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 161 | double weight_delta = 162 | (label[k] - outputLayer[k]->value) * 163 | outputLayer[k]->value * 164 | (1.0 - outputLayer[k]->value) * 165 | hiddenLayer[j]->value; 166 | 167 | hiddenLayer[j]->weight_delta[k] += weight_delta; 168 | } 169 | } 170 | ``` 171 | 172 | #### 3.3 计算$\Delta \beta_j$（隐藏层节点偏置值的修正值） 173 | 174 | 其计算公式如下（激活函数为Sigmoid时）： 175 | 176 | $$ 177 | \Delta \beta_j = - \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) 178 | $$ 179 | 180 | 其中$v_{jk}$为第$j$个隐藏层节点到第$k$个输出层节点的权重（其余变量上方已出现过不再进行标注）。 181 | 182 | 本项目中的代码实现如下： 183 | 184 | ```C++ 185 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 186 | double bias_delta = 0.f; 187 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 188 | bias_delta += 189 | -(label[k] - outputLayer[k]->value) * 190 | outputLayer[k]->value * 191 | (1.0 - outputLayer[k]->value) * 192 | hiddenLayer[j]->weight[k]; 193 | } 194 | bias_delta *= 195 | hiddenLayer[j]->value * 196 | (1.0 - hiddenLayer[j]->value); 197 | 198 | hiddenLayer[j]->bias_delta += bias_delta; 199 | } 200 | ``` 201 | 202 | #### 3.4 计算$\Delta w_{ij}$（输入层节点到隐藏层节点权重的修正值） 203 | 204 | 其计算公式如下（激活函数为Sigmoid时）： 205 | 206 | $$ 207 | \Delta w_{ij} = \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) x_i 208 | $$ 209 | 210 | 其中$x_i$为第$i$个输入层节点的值（其余变量上方已出现过不再进行标注）。 211 | 212 | 本项目中的代码实现如下： 213 | 214 | ```C++ 215 | for (size_t i = 0; i < Config::INNODE; ++i) { 216 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 217 | double weight_delta = 0.f; 218 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 219 | weight_delta += 220 | (label[k] - outputLayer[k]->value) * 221 | outputLayer[k]->value * 222 | (1.0 - outputLayer[k]->value) * 223 | hiddenLayer[j]->weight[k]; 224 | } 225 | weight_delta *= 226 | hiddenLayer[j]->value * 227 | (1.0 - hiddenLayer[j]->value) * 228 | inputLayer[i]->value; 229 | 230 | inputLayer[i]->weight_delta[j] += weight_delta; 231 | } 232 | } 233 | ``` 234 | -------------------------------------------------------------------------------- /README.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GavinTechStudio/Back-Propagation-Neural-Network/4a4224e4eb28570e043a99f02ce8cd34c319e658/README.pdf -------------------------------------------------------------------------------- /data/testdata.txt: -------------------------------------------------------------------------------- 1 | 0.111 0.112 2 | 0.001 0.999 3 | 0.123 0.345 4 | 0.123 0.456 5 | 0.123 0.789 6 | 0.234 0.567 7 | 0.234 0.678 8 | 0.387 0.401 9 | 0.616 0.717 10 | 0.701 0.919 -------------------------------------------------------------------------------- /data/traindata.txt: -------------------------------------------------------------------------------- 1 | 0 0 0 2 | 0 1 1 3 | 1 0 1 4 | 1 1 0 5 | 0.8 0.8 0 6 | 0.6 0.6 0 7 | 0.4 0.4 0 8 | 0.2 0.2 0 9 | 1.0 0.8 1 10 | 1.0 0.6 1 11 | 1.0 0.4 1 12 | 1.0 0.2 1 13 | 0.8 0.6 1 14 | 0.6 0.4 1 15 | 0.4 0.2 1 16 | 0.2 0 1 17 | 0.999 0.666 1 18 | 0.666 0.333 1 19 | 0.333 0 1 20 | 0.8 0.4 1 21 | 0.4 0 1 22 | 0 0.123 1 23 | 0.12 0.23 1 24 | 0.23 0.34 1 25 | 0.34 0.45 1 26 | 0.45 0.56 1 27 | 0.56 0.67 1 28 | 0.67 0.78 1 29 | 0.78 0.89 1 30 | 0.89 0.99 1 -------------------------------------------------------------------------------- /img/net-info.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GavinTechStudio/Back-Propagation-Neural-Network/4a4224e4eb28570e043a99f02ce8cd34c319e658/img/net-info.png -------------------------------------------------------------------------------- /index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /lib/Config.h: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #pragma once 8 | 9 | using std::size_t; 10 | 11 | namespace Config { 12 | const size_t INNODE = 2; 13 | const size_t HIDENODE = 4; 14 | const size_t OUTNODE = 1; 15 | 16 | const double lr = 0.8; 17 | const double threshold = 1e-4; 18 | const size_t max_epoch = 1e6; 19 | } 20 | -------------------------------------------------------------------------------- /lib/Net.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #include "Net.h" 8 | #include "Utils.h" 9 | #include 10 | 11 | Net::Net() { 12 | std::mt19937 rd; 13 | rd.seed(std::random_device()()); 14 | 15 | std::uniform_real_distribution distribution(-1, 1); 16 | 17 | /** 18 | * Initialize input layer 19 | */ 20 | for (size_t i = 0; i < Config::INNODE; ++i) { 21 | inputLayer[i] = new Node(Config::HIDENODE); 22 | 23 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 24 | 25 | // Initialize 'weight'(the weight value) 26 | // from the i-th node in the input layer to the j-th node in the hidden layer 27 | inputLayer[i]->weight[j] = distribution(rd); 28 | 29 | // Initialize 'weight_delta'(the weight correction value) 30 | // from the i-th node in the input layer to the j-th node in the hidden layer 31 | inputLayer[i]->weight_delta[j] = 0.f; 32 | } 33 | } 34 | 35 | /** 36 | * Initialize hidden layer 37 | */ 38 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 39 | hiddenLayer[j] = new Node(Config::OUTNODE); 40 | 41 | // Initialize 'bias'(the bias value) 42 | // of the j-th node in the hidden layer 43 | hiddenLayer[j]->bias = distribution(rd); 44 | 45 | // Initialize 'bias_delta'(the bias correction value) 46 | // of the j-th node in the hidden layer 47 | hiddenLayer[j]->bias_delta = 0.f; 48 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 49 | 50 | // Initialize 'weight'(the weight value) 51 | // from the j-th node in the hidden layer to the k-th node in the output layer 52 | hiddenLayer[j]->weight[k] = distribution(rd); 53 | 54 | // Initialize 'weight_delta'(the weight correction value) 55 | // from the j-th node in the hidden layer to the k-th node in the output layer 56 | hiddenLayer[j]->weight_delta[k] = 0.f; 57 | } 58 | } 59 | 60 | // Initialize output layer 61 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 62 | outputLayer[k] = new Node(0); 63 | 64 | // Initialize 'bias'(the bias value) 65 | // of the k-th node in the output layer 66 | outputLayer[k]->bias = distribution(rd); 67 | 68 | // Initialize 'bias_delta'(the bias correction value) 69 | // of the k-th node in the output layer 70 | outputLayer[k]->bias_delta = 0.f; 71 | } 72 | } 73 | 74 | void Net::grad_zero() { 75 | 76 | // Clear 'weight_delta'(the weight correction value) 77 | // of all nodes in the input layer 78 | for (auto &nodeOfInputLayer: inputLayer) { 79 | nodeOfInputLayer->weight_delta.assign(nodeOfInputLayer->weight_delta.size(), 0.f); 80 | } 81 | 82 | // Clear 'weight_delta'(the weight correction value) and 'bias_delta'(the bias correction value) 83 | // of all nodes in the hidden layer 84 | for (auto &nodeOfHiddenLayer: hiddenLayer) { 85 | nodeOfHiddenLayer->bias_delta = 0.f; 86 | nodeOfHiddenLayer->weight_delta.assign(nodeOfHiddenLayer->weight_delta.size(), 0.f); 87 | } 88 | 89 | // Clear 'bias_delta'(the bias correction value) 90 | // of all nodes in the hidden layer 91 | for (auto &nodeOfOutputLayer: outputLayer) { 92 | nodeOfOutputLayer->bias_delta = 0.f; 93 | } 94 | } 95 | 96 | void Net::forward() { 97 | 98 | /** 99 | * The input layer propagate forward to the hidden layer. 100 | * MathJax formula: h_j = \sigma( \sum_i x_i w_{ij} - \beta_j ) 101 | */ 102 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 103 | double sum = 0; 104 | for (size_t i = 0; i < Config::INNODE; ++i) { 105 | sum += inputLayer[i]->value * inputLayer[i]->weight[j]; 106 | } 107 | sum -= hiddenLayer[j]->bias; 108 | 109 | hiddenLayer[j]->value = Utils::sigmoid(sum); 110 | } 111 | 112 | /** 113 | * The hidden layer propagate forward to the output layer. 114 | * MathJax formula: \hat{y_k} = \sigma( \sum_j h_j v_{jk} - \lambda_k ) 115 | */ 116 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 117 | double sum = 0; 118 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 119 | sum += hiddenLayer[j]->value * hiddenLayer[j]->weight[k]; 120 | } 121 | sum -= outputLayer[k]->bias; 122 | 123 | outputLayer[k]->value = Utils::sigmoid(sum); 124 | } 125 | } 126 | 127 | double Net::calculateLoss(const vector &label) { 128 | double loss = 0.f; 129 | 130 | /** 131 | * MathJax formula: Loss = \frac{1}{2}\sum_k ( y_k - \hat{y_k} )^2 132 | */ 133 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 134 | double tmp = std::fabs(outputLayer[k]->value - label[k]); 135 | loss += tmp * tmp / 2; 136 | } 137 | 138 | return loss; 139 | } 140 | 141 | void Net::backward(const vector &label) { 142 | 143 | /** 144 | * Calculate 'bias_delta'(the bias correction value) 145 | * of the k-th node in the output layer 146 | * MathJax formula: \Delta \lambda_k = - \eta (y_k - \hat{y_k}) \hat{y_k} (1 - \hat{y_k}) 147 | */ 148 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 149 | double bias_delta = 150 | -(label[k] - outputLayer[k]->value) 151 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value); 152 | 153 | outputLayer[k]->bias_delta += bias_delta; 154 | } 155 | 156 | /** 157 | * Calculate 'weight_delta'(the weight correction value) 158 | * from the j-th node in the hidden layer to the k-th node in the output layer 159 | * MathJax formula: \Delta v_{jk} = \eta ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) h_j 160 | */ 161 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 162 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 163 | double weight_delta = 164 | (label[k] - outputLayer[k]->value) 165 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value) 166 | * hiddenLayer[j]->value; 167 | 168 | hiddenLayer[j]->weight_delta[k] += weight_delta; 169 | } 170 | } 171 | 172 | /** 173 | * Calculate 'bias_delta'(the bias correction value) 174 | * of the j-th node in the hidden layer 175 | * MathJax formula: \Delta \beta_j = - \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) 176 | */ 177 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 178 | double bias_delta = 0.f; 179 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 180 | bias_delta += 181 | -(label[k] - outputLayer[k]->value) 182 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value) 183 | * hiddenLayer[j]->weight[k]; 184 | } 185 | bias_delta *= 186 | hiddenLayer[j]->value * (1.0 - hiddenLayer[j]->value); 187 | 188 | hiddenLayer[j]->bias_delta += bias_delta; 189 | } 190 | 191 | /** 192 | * Calculate 'weight_delta'(the weight correction value) 193 | * from the i-th node in the input layer to the j-th node in the hidden layer 194 | * MathJax formula: \Delta w_{ij} = \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) x_i 195 | */ 196 | for (size_t i = 0; i < Config::INNODE; ++i) { 197 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 198 | double weight_delta = 0.f; 199 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 200 | weight_delta += 201 | (label[k] - outputLayer[k]->value) 202 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value) 203 | * hiddenLayer[j]->weight[k]; 204 | } 205 | weight_delta *= 206 | hiddenLayer[j]->value * (1.0 - hiddenLayer[j]->value) 207 | * inputLayer[i]->value; 208 | 209 | inputLayer[i]->weight_delta[j] += weight_delta; 210 | } 211 | } 212 | } 213 | 214 | bool Net::train(const vector &trainDataSet) { 215 | for (size_t epoch = 0; epoch <= Config::max_epoch; ++epoch) { 216 | 217 | grad_zero(); 218 | 219 | double max_loss = 0.f; 220 | 221 | for (const Sample &trainSample: trainDataSet) { 222 | 223 | // Load trainSample's feature into the network 224 | for (size_t i = 0; i < Config::INNODE; ++i) 225 | inputLayer[i]->value = trainSample.feature[i]; 226 | 227 | // Forward propagation 228 | forward(); 229 | 230 | // Calculate 'loss' 231 | double loss = calculateLoss(trainSample.label); 232 | max_loss = std::max(max_loss, loss); 233 | 234 | // Back propagation 235 | backward(trainSample.label); 236 | 237 | } 238 | 239 | // Deciding whether to stop training 240 | if (max_loss < Config::threshold) { 241 | printf("Training SUCCESS in %lu epochs.\n", epoch); 242 | printf("Final maximum error(loss): %lf\n", max_loss); 243 | return true; 244 | } else if (epoch % 5000 == 0) { 245 | printf("#epoch %-7lu - max_loss: %lf\n", epoch, max_loss); 246 | } 247 | 248 | // Revise 'weight' and 'bias' of each node 249 | revise(trainDataSet.size()); 250 | } 251 | 252 | printf("Failed within %lu epoch.", Config::max_epoch); 253 | 254 | return false; 255 | } 256 | 257 | void Net::revise(size_t batch_size) { 258 | 259 | auto batch_size_double = (double) batch_size; 260 | 261 | for (size_t i = 0; i < Config::INNODE; ++i) { 262 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 263 | 264 | /** 265 | * Revise 'weight' according to 'weight_delta'(the weight correction value) 266 | * from the i-th node in the input layer to the j-th node in the hidden layer 267 | */ 268 | inputLayer[i]->weight[j] += 269 | Config::lr * inputLayer[i]->weight_delta[j] / batch_size_double; 270 | 271 | } 272 | } 273 | 274 | for (size_t j = 0; j < Config::HIDENODE; ++j) { 275 | 276 | /** 277 | * Revise 'bias' according to 'bias_delta'(the bias correction value) 278 | * of the j-th node in the hidden layer 279 | */ 280 | hiddenLayer[j]->bias += 281 | Config::lr * hiddenLayer[j]->bias_delta / batch_size_double; 282 | 283 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 284 | 285 | /** 286 | * Revise 'weight' according to 'weight_delta'(the weight correction value) 287 | * from the j-th node in the hidden layer to the k-th node in the output layer 288 | */ 289 | hiddenLayer[j]->weight[k] += 290 | Config::lr * hiddenLayer[j]->weight_delta[k] / batch_size_double; 291 | 292 | } 293 | } 294 | 295 | for (size_t k = 0; k < Config::OUTNODE; ++k) { 296 | 297 | /** 298 | * Revise 'bias' according to 'bias_weight'(the bias correction value) 299 | * of the k-th node in the output layer 300 | */ 301 | outputLayer[k]->bias += 302 | Config::lr * outputLayer[k]->bias_delta / batch_size_double; 303 | 304 | } 305 | } 306 | 307 | Sample Net::predict(const vector &feature) { 308 | 309 | // load sample into the network 310 | for (size_t i = 0; i < Config::INNODE; ++i) 311 | inputLayer[i]->value = feature[i]; 312 | 313 | forward(); 314 | 315 | vector label(Config::OUTNODE); 316 | for (size_t k = 0; k < Config::OUTNODE; ++k) 317 | label[k] = outputLayer[k]->value; 318 | 319 | Sample pred = Sample(feature, label); 320 | return pred; 321 | } 322 | 323 | vector Net::predict(const vector &predictDataSet) { 324 | vector predSet; 325 | 326 | for (auto &sample: predictDataSet) { 327 | Sample pred = predict(sample.feature); 328 | predSet.push_back(pred); 329 | } 330 | 331 | return predSet; 332 | } 333 | 334 | Node::Node(size_t nextLayerSize) { 335 | weight.resize(nextLayerSize); 336 | weight_delta.resize(nextLayerSize); 337 | } 338 | 339 | Sample::Sample() = default; 340 | 341 | Sample::Sample(const vector &feature, const vector &label) { 342 | this->feature = feature; 343 | this->label = label; 344 | } 345 | 346 | void Sample::display() { 347 | printf("input : "); 348 | for (auto &x: feature) printf("%lf ", x); 349 | puts(""); 350 | printf("output: "); 351 | for (auto &y: label) printf("%lf ", y); 352 | puts(""); 353 | } 354 | -------------------------------------------------------------------------------- /lib/Net.h: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #pragma once 8 | 9 | #include 10 | #include "Config.h" 11 | 12 | using std::vector; 13 | 14 | struct Sample { 15 | vector feature, label; 16 | 17 | Sample(); 18 | 19 | Sample(const vector &feature, const vector &label); 20 | 21 | void display(); 22 | }; 23 | 24 | struct Node { 25 | double value{}, bias{}, bias_delta{}; 26 | vector weight, weight_delta; 27 | 28 | explicit Node(size_t nextLayerSize); 29 | }; 30 | 31 | class Net { 32 | private: 33 | Node *inputLayer[Config::INNODE]{}; 34 | Node *hiddenLayer[Config::HIDENODE]{}; 35 | Node *outputLayer[Config::OUTNODE]{}; 36 | 37 | /** 38 | * Clear all gradient accumulation 39 | * 40 | * Set 'weight_delta'(the weight correction value) and 41 | * 'bias_delta'(the bias correction value) to 0 of nodes 42 | */ 43 | void grad_zero(); 44 | 45 | /** 46 | * Forward propagation 47 | */ 48 | void forward(); 49 | 50 | /** 51 | * Calculate the value of Loss Function 52 | * @param label the label of sample (vector / numeric) 53 | * @return loss 54 | */ 55 | double calculateLoss(const vector &label); 56 | 57 | /** 58 | * Back propagation 59 | * @param label label of sample (vector / numeric) 60 | */ 61 | void backward(const vector &label); 62 | 63 | /** 64 | * Revise 'weight' and 'bias according to 65 | * 'weight_delta'(the weight correction value) and 66 | * 'bias_weight'(the bias correction value) 67 | * @param batch_size 68 | */ 69 | void revise(size_t batch_size); 70 | 71 | public: 72 | 73 | Net(); 74 | 75 | /** 76 | * Training network with training data 77 | * @param trainDataSet The sample set 78 | * @return Training convergence 79 | */ 80 | bool train(const vector &trainDataSet); 81 | 82 | /** 83 | * Using network to predict sample 84 | * @param feature The feature of sample (vector) 85 | * @return Sample with 'feature' and 'label'(predicted) 86 | */ 87 | Sample predict(const vector &feature); 88 | 89 | /** 90 | * Using network to predict the sample set 91 | * @param predictDataSet The sample set 92 | * @return The sample set, in which each sample has 'feature' and 'label'(predicted) 93 | */ 94 | vector predict(const vector &predictDataSet); 95 | 96 | }; 97 | -------------------------------------------------------------------------------- /lib/Utils.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #include 8 | #include 9 | #include "Utils.h" 10 | #include "Config.h" 11 | 12 | #if defined(WIN64) || defined(_WIN64) || defined(WIN32) || defined(_WIN32) 13 | #include 14 | #else 15 | #include 16 | #endif 17 | 18 | vector Utils::getFileData(const string &filename) { 19 | vector res; 20 | 21 | std::ifstream in(filename); 22 | if (in.is_open()) { 23 | while (!in.eof()) { 24 | double val; 25 | in >> val; 26 | res.push_back(val); 27 | } 28 | in.close(); 29 | } else { 30 | // Data file not found 31 | printf("[ERROR] '%s' not found.\n", filename.c_str()); 32 | // Prompt the correct storage path of data file 33 | char path[256]; 34 | getcwd(path, sizeof(path)); 35 | printf("Please check the path '%s' is relative to '%s'.\n", filename.c_str(), path); 36 | exit(1); 37 | } 38 | 39 | return res; 40 | } 41 | 42 | vector Utils::getTrainData(const string &filename) { 43 | vector trainDataSet; 44 | 45 | vector buffer = getFileData(filename); 46 | 47 | for (size_t i = 0; i < buffer.size(); i += Config::INNODE + Config::OUTNODE) { 48 | Sample trainSample; 49 | // Read in training sample 'feature' 50 | for (size_t j = 0; j < Config::INNODE; ++j) 51 | trainSample.feature.push_back(buffer[i + j]); 52 | // Read in training sample 'label' 53 | for (size_t k = 0; k < Config::OUTNODE; ++k) 54 | trainSample.label.push_back(buffer[i + Config::INNODE + k]); 55 | // Add samples to the 'trainDataSet' 56 | trainDataSet.push_back(trainSample); 57 | } 58 | return trainDataSet; 59 | } 60 | 61 | vector Utils::getTestData(const string &filename) { 62 | vector testDataSet; 63 | 64 | vector buffer = getFileData(filename); 65 | 66 | for (size_t i = 0; i < buffer.size(); i += Config::INNODE) { 67 | Sample testSample; 68 | // Read in test sample 'feature' 69 | for (size_t j = 0; j < Config::INNODE; ++j) 70 | testSample.feature.push_back(buffer[i + j]); 71 | // Add samples to the 'testDataSet' 72 | testDataSet.push_back(testSample); 73 | } 74 | 75 | return testDataSet; 76 | } 77 | -------------------------------------------------------------------------------- /lib/Utils.h: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #pragma once 8 | 9 | #include 10 | #include 11 | #include 12 | #include "Net.h" 13 | 14 | using std::vector; 15 | using std::string; 16 | 17 | namespace Utils { 18 | static double sigmoid(double x) { 19 | return 1.0 / (1.0 + std::exp(-x)); 20 | } 21 | 22 | vector getFileData(const string &filename); 23 | 24 | vector getTrainData(const string &filename); 25 | 26 | vector getTestData(const string &filename); 27 | } 28 | -------------------------------------------------------------------------------- /main.cpp: -------------------------------------------------------------------------------- 1 | /** 2 | * @author Gavin 3 | * @date 2022/2/10 4 | * @Email gavinsun0921@foxmail.com 5 | */ 6 | 7 | #include 8 | #include "lib/Net.h" 9 | #include "lib/Utils.h" 10 | 11 | using std::cout; 12 | using std::endl; 13 | 14 | int main(int argc, char *argv[]) { 15 | 16 | // Create neural network object 17 | Net net; 18 | 19 | // Read training data 20 | const vector trainDataSet = Utils::getTrainData("../data/traindata.txt"); 21 | 22 | // Training neural network 23 | net.train(trainDataSet); 24 | 25 | // Prediction of samples using neural network 26 | const vector testDataSet = Utils::getTestData("../data/testdata.txt"); 27 | vector predSet = net.predict(testDataSet); 28 | for (auto &pred: predSet) { 29 | pred.display(); 30 | } 31 | 32 | return 0; 33 | } 34 | -------------------------------------------------------------------------------- /updatePages.sh: -------------------------------------------------------------------------------- 1 | git checkout pages 2 | git merge master 3 | git push 4 | git checkout master 5 | --------------------------------------------------------------------------------