├── .gitignore
├── CMakeLists.txt
├── README.html
├── README.md
├── README.pdf
├── data
├── testdata.txt
└── traindata.txt
├── img
└── net-info.png
├── index.html
├── lib
├── Config.h
├── Net.cpp
├── Net.h
├── Utils.cpp
└── Utils.h
├── main.cpp
└── updatePages.sh
/.gitignore:
--------------------------------------------------------------------------------
1 | .idea
2 |
3 | cmake-build-debug
4 |
5 | test
6 | test.dSYM/
7 |
--------------------------------------------------------------------------------
/CMakeLists.txt:
--------------------------------------------------------------------------------
1 | cmake_minimum_required(VERSION 2.8.12)
2 | project(BPNN)
3 |
4 | set(CMAKE_CXX_STANDARD 11)
5 |
6 | add_executable(BPNN main.cpp lib/Utils.cpp lib/Utils.h lib/Config.h lib/Net.cpp lib/Net.h)
7 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Back-Propagation-Neural-Network
2 |
3 | [](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network) [](https://github.com/topics/cpp) [](https://cmake.org/) [](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network/commits) [](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network/actions/workflows/pages/pages-build-deployment)
4 |
5 | [本项目](https://github.com/GavinTechStudio/Back-Propagation-Neural-Network)是对项目 [**GavinTechStudio/bpnn_with_cpp**](https://github.com/GavinTechStudio/bpnn_with_cpp) 的代码重构,基于C++实现基础BP神经网络,有助于深入理解BP神经网络原理。
6 |
7 | ## 项目结构
8 |
9 | ```
10 | .
11 | ├── CMakeLists.txt
12 | ├── data
13 | │ ├── testdata.txt
14 | │ └── traindata.txt
15 | ├── lib
16 | │ ├── Config.h
17 | │ ├── Net.cpp
18 | │ ├── Net.h
19 | │ ├── Utils.cpp
20 | │ └── Utils.h
21 | └── main.cpp
22 | ```
23 |
24 | #### 主要框架
25 |
26 | - Net:网络具体实现
27 | - Config:网络参数设置
28 | - Utils:工具类
29 | - 数据加载
30 | - 激活函数
31 | - main:网络具体应用
32 |
33 | ## 训练原理
34 |
35 | > 具体公式推导请看视频讲解[彻底搞懂BP神经网络 理论推导+代码实现(C++)](https://www.bilibili.com/video/BV1Y64y1z7jM?p=1)
36 |
37 | #### 注意:本部分文档包含大量数学公式,由于GitHub markdown不支持数学公式渲染,推荐以下阅读方式:
38 |
39 | 1. 如果您使用的是Chrome、Edge、Firefox等浏览器,可以安装插件[MathJax Plugin for Github](https://chrome.google.com/webstore/detail/mathjax-plugin-for-github/ioemnmodlmafdkllaclgeombjnmnbima)(需要网络能够访问chrome web store)。
40 | 2. 使用[**PDF**](https://gavintechstudio.github.io/Back-Propagation-Neural-Network/README.pdf)的方式进行阅读。
41 | 2. 使用[**预渲染的静态网页**](https://gavintechstudio.github.io/Back-Propagation-Neural-Network/README.html)进行阅读(**推荐**)。
42 | 2. 按`.`键或[点击链接](https://github.dev/GavinTechStudio/Back-Propagation-Neural-Network)进入GitHub在线IDE预览`README.md`文件。
43 |
44 | ### 0. 神经网络结构图
45 |
46 | 
47 |
48 | ### 1. Forward(前向传播)
49 |
50 | #### 1.1 输入层向隐藏层传播
51 |
52 | $$
53 | h_j = \sigma( \sum_i x_i w_{ij} - \beta_j )
54 | $$
55 |
56 | 其中$h_j$为第$j$个隐藏层节点的值,$x_i$为第$i$个输入层节点的值,$w_{ij}$为第$i$个输入层节点到第$j$个隐藏层节点的权重,$\beta_j$为第$j$个隐藏层节点偏置值,$\sigma(x)$为**Sigmoid**激活函数,后续也将继续用这个表达,其表达式如下
57 | $$
58 | \sigma(x) = \frac{1}{1+e^{-x}}
59 | $$
60 | 本项目中的代码实现如下:
61 |
62 | ```C++
63 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
64 | double sum = 0;
65 | for (size_t i = 0; i < Config::INNODE; ++i) {
66 | sum += inputLayer[i]->value *
67 | inputLayer[i]->weight[j];
68 | }
69 | sum -= hiddenLayer[j]->bias;
70 |
71 | hiddenLayer[j]->value = Utils::sigmoid(sum);
72 | }
73 | ```
74 |
75 | #### 1.2 隐藏层向输出层传播
76 |
77 | $$
78 | \hat{y_k} = \sigma( \sum_j h_j v_{jk} - \lambda_k )
79 | $$
80 |
81 | 其中$\hat{y_k}$为第$k$个输出层节点的值(预测值),$h_j$为第$j$个隐藏层节点的值,$v_{jk}$为第$j$个隐藏层节点到第$k$个输出层节点的权重,$\lambda_k$为第$k$个输出层节点的偏置值,$\sigma(x)$为激活函数。
82 |
83 | 本项目中的代码实现如下:
84 |
85 | ```C++
86 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
87 | double sum = 0;
88 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
89 | sum += hiddenLayer[j]->value *
90 | hiddenLayer[j]->weight[k];
91 | }
92 | sum -= outputLayer[k]->bias;
93 |
94 | outputLayer[k]->value = Utils::sigmoid(sum);
95 | }
96 | ```
97 |
98 | ### 2. 计算损失函数(Loss Function)
99 |
100 | 损失函数定义如下:
101 |
102 | $$
103 | Loss = \frac{1}{2}\sum_k ( y_k - \hat{y_k} )^2
104 | $$
105 |
106 | 其中$y_k$为第$k$个输出层节点的目标值(真实值),$\hat{y_k}$为第$k$个输出层节点的值(预测值)。
107 |
108 | 本项目中的代码实现如下:
109 |
110 | ```C++
111 | double loss = 0.f;
112 |
113 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
114 | double tmp = std::fabs(outputLayer[k]->value - label[k]);
115 | los += tmp * tmp / 2;
116 | }
117 | ```
118 |
119 | ### 3. Backward(反向传播)
120 |
121 | 利用梯度下降法进行优化。
122 |
123 | #### 3.1 计算$\Delta \lambda_k$(输出层节点偏置值的修正值)
124 |
125 | 其计算公式如下(激活函数为Sigmoid时):
126 |
127 | $$
128 | \Delta \lambda_k = - \eta (y_k - \hat{y_k}) \hat{y_k} (1 - \hat{y_k})
129 | $$
130 |
131 | 其中$\eta$为学习率(其余变量上方已出现过不再进行标注)。
132 |
133 | 本项目中的代码实现如下:
134 |
135 | ```C++
136 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
137 | double bias_delta =
138 | -(label[k] - outputLayer[k]->value) *
139 | outputLayer[k]->value *
140 | (1.0 - outputLayer[k]->value);
141 |
142 | outputLayer[k]->bias_delta += bias_delta;
143 | }
144 | ```
145 |
146 | #### 3.2 计算$\Delta v_{jk}$(隐藏层节点到输出层节点权重的修正值)
147 |
148 | 其计算公式如下(激活函数为Sigmoid时):
149 |
150 | $$
151 | \Delta v_{jk} = \eta ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) h_j
152 | $$
153 |
154 | 其中$h_j$为第$j$个隐藏层节点的值(其余变量上方已出现过不再进行标注)。
155 |
156 | 本项目中的代码实现如下:
157 |
158 | ```C++
159 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
160 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
161 | double weight_delta =
162 | (label[k] - outputLayer[k]->value) *
163 | outputLayer[k]->value *
164 | (1.0 - outputLayer[k]->value) *
165 | hiddenLayer[j]->value;
166 |
167 | hiddenLayer[j]->weight_delta[k] += weight_delta;
168 | }
169 | }
170 | ```
171 |
172 | #### 3.3 计算$\Delta \beta_j$(隐藏层节点偏置值的修正值)
173 |
174 | 其计算公式如下(激活函数为Sigmoid时):
175 |
176 | $$
177 | \Delta \beta_j = - \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j )
178 | $$
179 |
180 | 其中$v_{jk}$为第$j$个隐藏层节点到第$k$个输出层节点的权重(其余变量上方已出现过不再进行标注)。
181 |
182 | 本项目中的代码实现如下:
183 |
184 | ```C++
185 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
186 | double bias_delta = 0.f;
187 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
188 | bias_delta +=
189 | -(label[k] - outputLayer[k]->value) *
190 | outputLayer[k]->value *
191 | (1.0 - outputLayer[k]->value) *
192 | hiddenLayer[j]->weight[k];
193 | }
194 | bias_delta *=
195 | hiddenLayer[j]->value *
196 | (1.0 - hiddenLayer[j]->value);
197 |
198 | hiddenLayer[j]->bias_delta += bias_delta;
199 | }
200 | ```
201 |
202 | #### 3.4 计算$\Delta w_{ij}$(输入层节点到隐藏层节点权重的修正值)
203 |
204 | 其计算公式如下(激活函数为Sigmoid时):
205 |
206 | $$
207 | \Delta w_{ij} = \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) x_i
208 | $$
209 |
210 | 其中$x_i$为第$i$个输入层节点的值(其余变量上方已出现过不再进行标注)。
211 |
212 | 本项目中的代码实现如下:
213 |
214 | ```C++
215 | for (size_t i = 0; i < Config::INNODE; ++i) {
216 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
217 | double weight_delta = 0.f;
218 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
219 | weight_delta +=
220 | (label[k] - outputLayer[k]->value) *
221 | outputLayer[k]->value *
222 | (1.0 - outputLayer[k]->value) *
223 | hiddenLayer[j]->weight[k];
224 | }
225 | weight_delta *=
226 | hiddenLayer[j]->value *
227 | (1.0 - hiddenLayer[j]->value) *
228 | inputLayer[i]->value;
229 |
230 | inputLayer[i]->weight_delta[j] += weight_delta;
231 | }
232 | }
233 | ```
234 |
--------------------------------------------------------------------------------
/README.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/GavinTechStudio/Back-Propagation-Neural-Network/4a4224e4eb28570e043a99f02ce8cd34c319e658/README.pdf
--------------------------------------------------------------------------------
/data/testdata.txt:
--------------------------------------------------------------------------------
1 | 0.111 0.112
2 | 0.001 0.999
3 | 0.123 0.345
4 | 0.123 0.456
5 | 0.123 0.789
6 | 0.234 0.567
7 | 0.234 0.678
8 | 0.387 0.401
9 | 0.616 0.717
10 | 0.701 0.919
--------------------------------------------------------------------------------
/data/traindata.txt:
--------------------------------------------------------------------------------
1 | 0 0 0
2 | 0 1 1
3 | 1 0 1
4 | 1 1 0
5 | 0.8 0.8 0
6 | 0.6 0.6 0
7 | 0.4 0.4 0
8 | 0.2 0.2 0
9 | 1.0 0.8 1
10 | 1.0 0.6 1
11 | 1.0 0.4 1
12 | 1.0 0.2 1
13 | 0.8 0.6 1
14 | 0.6 0.4 1
15 | 0.4 0.2 1
16 | 0.2 0 1
17 | 0.999 0.666 1
18 | 0.666 0.333 1
19 | 0.333 0 1
20 | 0.8 0.4 1
21 | 0.4 0 1
22 | 0 0.123 1
23 | 0.12 0.23 1
24 | 0.23 0.34 1
25 | 0.34 0.45 1
26 | 0.45 0.56 1
27 | 0.56 0.67 1
28 | 0.67 0.78 1
29 | 0.78 0.89 1
30 | 0.89 0.99 1
--------------------------------------------------------------------------------
/img/net-info.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/GavinTechStudio/Back-Propagation-Neural-Network/4a4224e4eb28570e043a99f02ce8cd34c319e658/img/net-info.png
--------------------------------------------------------------------------------
/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/lib/Config.h:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #pragma once
8 |
9 | using std::size_t;
10 |
11 | namespace Config {
12 | const size_t INNODE = 2;
13 | const size_t HIDENODE = 4;
14 | const size_t OUTNODE = 1;
15 |
16 | const double lr = 0.8;
17 | const double threshold = 1e-4;
18 | const size_t max_epoch = 1e6;
19 | }
20 |
--------------------------------------------------------------------------------
/lib/Net.cpp:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #include "Net.h"
8 | #include "Utils.h"
9 | #include
10 |
11 | Net::Net() {
12 | std::mt19937 rd;
13 | rd.seed(std::random_device()());
14 |
15 | std::uniform_real_distribution distribution(-1, 1);
16 |
17 | /**
18 | * Initialize input layer
19 | */
20 | for (size_t i = 0; i < Config::INNODE; ++i) {
21 | inputLayer[i] = new Node(Config::HIDENODE);
22 |
23 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
24 |
25 | // Initialize 'weight'(the weight value)
26 | // from the i-th node in the input layer to the j-th node in the hidden layer
27 | inputLayer[i]->weight[j] = distribution(rd);
28 |
29 | // Initialize 'weight_delta'(the weight correction value)
30 | // from the i-th node in the input layer to the j-th node in the hidden layer
31 | inputLayer[i]->weight_delta[j] = 0.f;
32 | }
33 | }
34 |
35 | /**
36 | * Initialize hidden layer
37 | */
38 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
39 | hiddenLayer[j] = new Node(Config::OUTNODE);
40 |
41 | // Initialize 'bias'(the bias value)
42 | // of the j-th node in the hidden layer
43 | hiddenLayer[j]->bias = distribution(rd);
44 |
45 | // Initialize 'bias_delta'(the bias correction value)
46 | // of the j-th node in the hidden layer
47 | hiddenLayer[j]->bias_delta = 0.f;
48 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
49 |
50 | // Initialize 'weight'(the weight value)
51 | // from the j-th node in the hidden layer to the k-th node in the output layer
52 | hiddenLayer[j]->weight[k] = distribution(rd);
53 |
54 | // Initialize 'weight_delta'(the weight correction value)
55 | // from the j-th node in the hidden layer to the k-th node in the output layer
56 | hiddenLayer[j]->weight_delta[k] = 0.f;
57 | }
58 | }
59 |
60 | // Initialize output layer
61 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
62 | outputLayer[k] = new Node(0);
63 |
64 | // Initialize 'bias'(the bias value)
65 | // of the k-th node in the output layer
66 | outputLayer[k]->bias = distribution(rd);
67 |
68 | // Initialize 'bias_delta'(the bias correction value)
69 | // of the k-th node in the output layer
70 | outputLayer[k]->bias_delta = 0.f;
71 | }
72 | }
73 |
74 | void Net::grad_zero() {
75 |
76 | // Clear 'weight_delta'(the weight correction value)
77 | // of all nodes in the input layer
78 | for (auto &nodeOfInputLayer: inputLayer) {
79 | nodeOfInputLayer->weight_delta.assign(nodeOfInputLayer->weight_delta.size(), 0.f);
80 | }
81 |
82 | // Clear 'weight_delta'(the weight correction value) and 'bias_delta'(the bias correction value)
83 | // of all nodes in the hidden layer
84 | for (auto &nodeOfHiddenLayer: hiddenLayer) {
85 | nodeOfHiddenLayer->bias_delta = 0.f;
86 | nodeOfHiddenLayer->weight_delta.assign(nodeOfHiddenLayer->weight_delta.size(), 0.f);
87 | }
88 |
89 | // Clear 'bias_delta'(the bias correction value)
90 | // of all nodes in the hidden layer
91 | for (auto &nodeOfOutputLayer: outputLayer) {
92 | nodeOfOutputLayer->bias_delta = 0.f;
93 | }
94 | }
95 |
96 | void Net::forward() {
97 |
98 | /**
99 | * The input layer propagate forward to the hidden layer.
100 | * MathJax formula: h_j = \sigma( \sum_i x_i w_{ij} - \beta_j )
101 | */
102 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
103 | double sum = 0;
104 | for (size_t i = 0; i < Config::INNODE; ++i) {
105 | sum += inputLayer[i]->value * inputLayer[i]->weight[j];
106 | }
107 | sum -= hiddenLayer[j]->bias;
108 |
109 | hiddenLayer[j]->value = Utils::sigmoid(sum);
110 | }
111 |
112 | /**
113 | * The hidden layer propagate forward to the output layer.
114 | * MathJax formula: \hat{y_k} = \sigma( \sum_j h_j v_{jk} - \lambda_k )
115 | */
116 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
117 | double sum = 0;
118 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
119 | sum += hiddenLayer[j]->value * hiddenLayer[j]->weight[k];
120 | }
121 | sum -= outputLayer[k]->bias;
122 |
123 | outputLayer[k]->value = Utils::sigmoid(sum);
124 | }
125 | }
126 |
127 | double Net::calculateLoss(const vector &label) {
128 | double loss = 0.f;
129 |
130 | /**
131 | * MathJax formula: Loss = \frac{1}{2}\sum_k ( y_k - \hat{y_k} )^2
132 | */
133 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
134 | double tmp = std::fabs(outputLayer[k]->value - label[k]);
135 | loss += tmp * tmp / 2;
136 | }
137 |
138 | return loss;
139 | }
140 |
141 | void Net::backward(const vector &label) {
142 |
143 | /**
144 | * Calculate 'bias_delta'(the bias correction value)
145 | * of the k-th node in the output layer
146 | * MathJax formula: \Delta \lambda_k = - \eta (y_k - \hat{y_k}) \hat{y_k} (1 - \hat{y_k})
147 | */
148 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
149 | double bias_delta =
150 | -(label[k] - outputLayer[k]->value)
151 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value);
152 |
153 | outputLayer[k]->bias_delta += bias_delta;
154 | }
155 |
156 | /**
157 | * Calculate 'weight_delta'(the weight correction value)
158 | * from the j-th node in the hidden layer to the k-th node in the output layer
159 | * MathJax formula: \Delta v_{jk} = \eta ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) h_j
160 | */
161 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
162 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
163 | double weight_delta =
164 | (label[k] - outputLayer[k]->value)
165 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value)
166 | * hiddenLayer[j]->value;
167 |
168 | hiddenLayer[j]->weight_delta[k] += weight_delta;
169 | }
170 | }
171 |
172 | /**
173 | * Calculate 'bias_delta'(the bias correction value)
174 | * of the j-th node in the hidden layer
175 | * MathJax formula: \Delta \beta_j = - \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j )
176 | */
177 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
178 | double bias_delta = 0.f;
179 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
180 | bias_delta +=
181 | -(label[k] - outputLayer[k]->value)
182 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value)
183 | * hiddenLayer[j]->weight[k];
184 | }
185 | bias_delta *=
186 | hiddenLayer[j]->value * (1.0 - hiddenLayer[j]->value);
187 |
188 | hiddenLayer[j]->bias_delta += bias_delta;
189 | }
190 |
191 | /**
192 | * Calculate 'weight_delta'(the weight correction value)
193 | * from the i-th node in the input layer to the j-th node in the hidden layer
194 | * MathJax formula: \Delta w_{ij} = \eta \sum_k ( y_k - \hat{y_k} ) \hat{y_k} ( 1 - \hat{y_k} ) v_{jk} h_j ( 1 - h_j ) x_i
195 | */
196 | for (size_t i = 0; i < Config::INNODE; ++i) {
197 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
198 | double weight_delta = 0.f;
199 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
200 | weight_delta +=
201 | (label[k] - outputLayer[k]->value)
202 | * outputLayer[k]->value * (1.0 - outputLayer[k]->value)
203 | * hiddenLayer[j]->weight[k];
204 | }
205 | weight_delta *=
206 | hiddenLayer[j]->value * (1.0 - hiddenLayer[j]->value)
207 | * inputLayer[i]->value;
208 |
209 | inputLayer[i]->weight_delta[j] += weight_delta;
210 | }
211 | }
212 | }
213 |
214 | bool Net::train(const vector &trainDataSet) {
215 | for (size_t epoch = 0; epoch <= Config::max_epoch; ++epoch) {
216 |
217 | grad_zero();
218 |
219 | double max_loss = 0.f;
220 |
221 | for (const Sample &trainSample: trainDataSet) {
222 |
223 | // Load trainSample's feature into the network
224 | for (size_t i = 0; i < Config::INNODE; ++i)
225 | inputLayer[i]->value = trainSample.feature[i];
226 |
227 | // Forward propagation
228 | forward();
229 |
230 | // Calculate 'loss'
231 | double loss = calculateLoss(trainSample.label);
232 | max_loss = std::max(max_loss, loss);
233 |
234 | // Back propagation
235 | backward(trainSample.label);
236 |
237 | }
238 |
239 | // Deciding whether to stop training
240 | if (max_loss < Config::threshold) {
241 | printf("Training SUCCESS in %lu epochs.\n", epoch);
242 | printf("Final maximum error(loss): %lf\n", max_loss);
243 | return true;
244 | } else if (epoch % 5000 == 0) {
245 | printf("#epoch %-7lu - max_loss: %lf\n", epoch, max_loss);
246 | }
247 |
248 | // Revise 'weight' and 'bias' of each node
249 | revise(trainDataSet.size());
250 | }
251 |
252 | printf("Failed within %lu epoch.", Config::max_epoch);
253 |
254 | return false;
255 | }
256 |
257 | void Net::revise(size_t batch_size) {
258 |
259 | auto batch_size_double = (double) batch_size;
260 |
261 | for (size_t i = 0; i < Config::INNODE; ++i) {
262 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
263 |
264 | /**
265 | * Revise 'weight' according to 'weight_delta'(the weight correction value)
266 | * from the i-th node in the input layer to the j-th node in the hidden layer
267 | */
268 | inputLayer[i]->weight[j] +=
269 | Config::lr * inputLayer[i]->weight_delta[j] / batch_size_double;
270 |
271 | }
272 | }
273 |
274 | for (size_t j = 0; j < Config::HIDENODE; ++j) {
275 |
276 | /**
277 | * Revise 'bias' according to 'bias_delta'(the bias correction value)
278 | * of the j-th node in the hidden layer
279 | */
280 | hiddenLayer[j]->bias +=
281 | Config::lr * hiddenLayer[j]->bias_delta / batch_size_double;
282 |
283 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
284 |
285 | /**
286 | * Revise 'weight' according to 'weight_delta'(the weight correction value)
287 | * from the j-th node in the hidden layer to the k-th node in the output layer
288 | */
289 | hiddenLayer[j]->weight[k] +=
290 | Config::lr * hiddenLayer[j]->weight_delta[k] / batch_size_double;
291 |
292 | }
293 | }
294 |
295 | for (size_t k = 0; k < Config::OUTNODE; ++k) {
296 |
297 | /**
298 | * Revise 'bias' according to 'bias_weight'(the bias correction value)
299 | * of the k-th node in the output layer
300 | */
301 | outputLayer[k]->bias +=
302 | Config::lr * outputLayer[k]->bias_delta / batch_size_double;
303 |
304 | }
305 | }
306 |
307 | Sample Net::predict(const vector &feature) {
308 |
309 | // load sample into the network
310 | for (size_t i = 0; i < Config::INNODE; ++i)
311 | inputLayer[i]->value = feature[i];
312 |
313 | forward();
314 |
315 | vector label(Config::OUTNODE);
316 | for (size_t k = 0; k < Config::OUTNODE; ++k)
317 | label[k] = outputLayer[k]->value;
318 |
319 | Sample pred = Sample(feature, label);
320 | return pred;
321 | }
322 |
323 | vector Net::predict(const vector &predictDataSet) {
324 | vector predSet;
325 |
326 | for (auto &sample: predictDataSet) {
327 | Sample pred = predict(sample.feature);
328 | predSet.push_back(pred);
329 | }
330 |
331 | return predSet;
332 | }
333 |
334 | Node::Node(size_t nextLayerSize) {
335 | weight.resize(nextLayerSize);
336 | weight_delta.resize(nextLayerSize);
337 | }
338 |
339 | Sample::Sample() = default;
340 |
341 | Sample::Sample(const vector &feature, const vector &label) {
342 | this->feature = feature;
343 | this->label = label;
344 | }
345 |
346 | void Sample::display() {
347 | printf("input : ");
348 | for (auto &x: feature) printf("%lf ", x);
349 | puts("");
350 | printf("output: ");
351 | for (auto &y: label) printf("%lf ", y);
352 | puts("");
353 | }
354 |
--------------------------------------------------------------------------------
/lib/Net.h:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #pragma once
8 |
9 | #include
10 | #include "Config.h"
11 |
12 | using std::vector;
13 |
14 | struct Sample {
15 | vector feature, label;
16 |
17 | Sample();
18 |
19 | Sample(const vector &feature, const vector &label);
20 |
21 | void display();
22 | };
23 |
24 | struct Node {
25 | double value{}, bias{}, bias_delta{};
26 | vector weight, weight_delta;
27 |
28 | explicit Node(size_t nextLayerSize);
29 | };
30 |
31 | class Net {
32 | private:
33 | Node *inputLayer[Config::INNODE]{};
34 | Node *hiddenLayer[Config::HIDENODE]{};
35 | Node *outputLayer[Config::OUTNODE]{};
36 |
37 | /**
38 | * Clear all gradient accumulation
39 | *
40 | * Set 'weight_delta'(the weight correction value) and
41 | * 'bias_delta'(the bias correction value) to 0 of nodes
42 | */
43 | void grad_zero();
44 |
45 | /**
46 | * Forward propagation
47 | */
48 | void forward();
49 |
50 | /**
51 | * Calculate the value of Loss Function
52 | * @param label the label of sample (vector / numeric)
53 | * @return loss
54 | */
55 | double calculateLoss(const vector &label);
56 |
57 | /**
58 | * Back propagation
59 | * @param label label of sample (vector / numeric)
60 | */
61 | void backward(const vector &label);
62 |
63 | /**
64 | * Revise 'weight' and 'bias according to
65 | * 'weight_delta'(the weight correction value) and
66 | * 'bias_weight'(the bias correction value)
67 | * @param batch_size
68 | */
69 | void revise(size_t batch_size);
70 |
71 | public:
72 |
73 | Net();
74 |
75 | /**
76 | * Training network with training data
77 | * @param trainDataSet The sample set
78 | * @return Training convergence
79 | */
80 | bool train(const vector &trainDataSet);
81 |
82 | /**
83 | * Using network to predict sample
84 | * @param feature The feature of sample (vector)
85 | * @return Sample with 'feature' and 'label'(predicted)
86 | */
87 | Sample predict(const vector &feature);
88 |
89 | /**
90 | * Using network to predict the sample set
91 | * @param predictDataSet The sample set
92 | * @return The sample set, in which each sample has 'feature' and 'label'(predicted)
93 | */
94 | vector predict(const vector &predictDataSet);
95 |
96 | };
97 |
--------------------------------------------------------------------------------
/lib/Utils.cpp:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #include
8 | #include
9 | #include "Utils.h"
10 | #include "Config.h"
11 |
12 | #if defined(WIN64) || defined(_WIN64) || defined(WIN32) || defined(_WIN32)
13 | #include
14 | #else
15 | #include
16 | #endif
17 |
18 | vector Utils::getFileData(const string &filename) {
19 | vector res;
20 |
21 | std::ifstream in(filename);
22 | if (in.is_open()) {
23 | while (!in.eof()) {
24 | double val;
25 | in >> val;
26 | res.push_back(val);
27 | }
28 | in.close();
29 | } else {
30 | // Data file not found
31 | printf("[ERROR] '%s' not found.\n", filename.c_str());
32 | // Prompt the correct storage path of data file
33 | char path[256];
34 | getcwd(path, sizeof(path));
35 | printf("Please check the path '%s' is relative to '%s'.\n", filename.c_str(), path);
36 | exit(1);
37 | }
38 |
39 | return res;
40 | }
41 |
42 | vector Utils::getTrainData(const string &filename) {
43 | vector trainDataSet;
44 |
45 | vector buffer = getFileData(filename);
46 |
47 | for (size_t i = 0; i < buffer.size(); i += Config::INNODE + Config::OUTNODE) {
48 | Sample trainSample;
49 | // Read in training sample 'feature'
50 | for (size_t j = 0; j < Config::INNODE; ++j)
51 | trainSample.feature.push_back(buffer[i + j]);
52 | // Read in training sample 'label'
53 | for (size_t k = 0; k < Config::OUTNODE; ++k)
54 | trainSample.label.push_back(buffer[i + Config::INNODE + k]);
55 | // Add samples to the 'trainDataSet'
56 | trainDataSet.push_back(trainSample);
57 | }
58 | return trainDataSet;
59 | }
60 |
61 | vector Utils::getTestData(const string &filename) {
62 | vector testDataSet;
63 |
64 | vector buffer = getFileData(filename);
65 |
66 | for (size_t i = 0; i < buffer.size(); i += Config::INNODE) {
67 | Sample testSample;
68 | // Read in test sample 'feature'
69 | for (size_t j = 0; j < Config::INNODE; ++j)
70 | testSample.feature.push_back(buffer[i + j]);
71 | // Add samples to the 'testDataSet'
72 | testDataSet.push_back(testSample);
73 | }
74 |
75 | return testDataSet;
76 | }
77 |
--------------------------------------------------------------------------------
/lib/Utils.h:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #pragma once
8 |
9 | #include
10 | #include
11 | #include
12 | #include "Net.h"
13 |
14 | using std::vector;
15 | using std::string;
16 |
17 | namespace Utils {
18 | static double sigmoid(double x) {
19 | return 1.0 / (1.0 + std::exp(-x));
20 | }
21 |
22 | vector getFileData(const string &filename);
23 |
24 | vector getTrainData(const string &filename);
25 |
26 | vector getTestData(const string &filename);
27 | }
28 |
--------------------------------------------------------------------------------
/main.cpp:
--------------------------------------------------------------------------------
1 | /**
2 | * @author Gavin
3 | * @date 2022/2/10
4 | * @Email gavinsun0921@foxmail.com
5 | */
6 |
7 | #include
8 | #include "lib/Net.h"
9 | #include "lib/Utils.h"
10 |
11 | using std::cout;
12 | using std::endl;
13 |
14 | int main(int argc, char *argv[]) {
15 |
16 | // Create neural network object
17 | Net net;
18 |
19 | // Read training data
20 | const vector trainDataSet = Utils::getTrainData("../data/traindata.txt");
21 |
22 | // Training neural network
23 | net.train(trainDataSet);
24 |
25 | // Prediction of samples using neural network
26 | const vector testDataSet = Utils::getTestData("../data/testdata.txt");
27 | vector predSet = net.predict(testDataSet);
28 | for (auto &pred: predSet) {
29 | pred.display();
30 | }
31 |
32 | return 0;
33 | }
34 |
--------------------------------------------------------------------------------
/updatePages.sh:
--------------------------------------------------------------------------------
1 | git checkout pages
2 | git merge master
3 | git push
4 | git checkout master
5 |
--------------------------------------------------------------------------------