35 | Activation functions are non-linear functions used in the Activation layer. 36 |
37 |41 | Leaky relu: ActivationFunctions.leaky_relu 42 |
43 |44 | Tanh: ActivationFunctions.tanh 45 |
46 |35 | Cost functions are functions that gives a metric to tell how bad the network is. 36 | It allow the network to know its flaws for example during backpropagation. 37 |
38 |42 | Mean Squared Error: CostFunctions.mse 43 |
44 |37 | This is the documentation for the module. 38 |
39 |40 | You can learn how to create a neural networks here: > Creating a network. 41 |
42 |35 | The Activation layer takes all the inputs and applies the activation function on each one. 36 |
37 |38 | It allows the network to learn non-linear functions, so without an Activation layer the network would only be able to learn linear functions. 39 | So it's an important part of the network. 40 |
41 |45 | To create a layer we first need a neural network to create the layer in, as seen in the XOR example. 46 |
47 |48 | We can then call the add_layer method on the neural network and create the layer that we want as the parameter for the function. 49 | In our case we are going to create an Activation layer. 50 |
51 |52 | We only need one parameter, it is the activation function we chose for this layer. 53 | A commonly used one is the leaky relu function, as the derivative is easy to compute and the derivative does not fades out as for a sigmoid for example. 54 | Other activation functions can be found in here. 55 |
56 |57 | model.add_layer(nn.Activation.new(.leaky_relu)) 58 |
59 |35 | A neural network is made out of layers. The dense layer is also called the fully connected layer or just known as a 'normal' layer. 36 | It is composed of weights connecting each input with every output, and then it adds the biases to each output. 37 |
38 |39 | The Dense layer is complementary with the Activation layer as the dense layer is the part that will train and improve and the activation layer 40 | produces non-linearity or said in another way: it allows the network to learn many more things. 41 |
42 |46 | To create a layer we first need a neural network to create the layer in, as seen in the XOR example. 47 |
48 |49 | We can then call the add_layer method on the neural network and create the layer that we want as the parameter for the function. 50 | In our case we are going to create a Dense layer. 51 |
52 |53 | The first two arguments are for the number of inputs and the number of outputs of the layer, because as said earlier each input is connected with every output and there is a bias for each output. 54 | The third ad fourth arguments are the range for initialisation of weights and biases respectively. In this example for weights, they can be initialized between -0.7 to 0.7. 55 |
56 |57 | model.add_layer(nn.Dense.new(2, 3, 0.7, 0.65)) 58 |
59 |35 | You need to have V installed, you can verify that by running v run .. 36 | You also need to have the vsl module installed, you can install it with v install vsl. 37 | Then you need to clone the module's repo and create a v file in it that we are going to use. 38 |
39 |43 | We are going to create a neural network to train a neural network that will be able to perform the XOR logic gate. The result can be found here. 44 |
45 |46 | First we are going to import the neural network module and create a main function. 47 | Then we create the neural network that we are going to train. 48 | The 0 is the seed for the random weights and biases to be able to get the same neural network at each run. 49 |
50 |51 | import neural_networks as nn 52 | 53 | fn main() { 54 | mut model := nn.NeuralNetwork.new(0) 55 | } 56 |
57 |58 | Then we add the layers that we want our network to have. 59 | We need our network to have 2 inputs and 1 output to match the XOR gate. 60 | So we will first add a Dense layer with 2 inputs and 3 outputs, 3 is arbitrary but works well. 61 | The two numbers after the number of inputs/outputs is the range for the initialisation of random weights and biases. 62 |
63 |64 | Then an Activation layer, the Dense and Activation layers are complementary so we will add one Activation per Dense layer. 65 | The Activation function that we will use for this layer is leaky relu, as it is convenient. 66 | We add a second Dense layer with 3 input and 1 output and the Activation layer that goes with it. 67 |
68 |69 | model.add_layer(nn.Dense.new(2, 3, 0.7, 0.65)) 70 | model.add_layer(nn.Activation.new(.leaky_relu)) 71 | model.add_layer(nn.Dense.new(3, 1, 0.6, 0.65)) 72 | model.add_layer(nn.Activation.new(.leaky_relu)) 73 |
74 |75 | Then we need to create the parametters for the training. 76 | The learning rate, momentum, number of epochs are found by trial and error and these work well. 77 | The Cost function that we will use is the Mean Squared Error (MSE). 78 |
79 |80 | We then add the dataset that the network will use for it's training. 81 | And same for the testing, in a real example the test data is unseen during the training to be able to see how well the networks does in an unseen situation 82 | but as we have only 4 different possible inputs we can not show unseen data to the network so we will use the same data. 83 |
84 |85 | The neural newtork will print it's performance every print_interval epochs. 86 | For the test parameters, every training_interval epochs it will run the test dataset and print the results from the print_startth element of the test dataset to the print_endth one. 87 |
88 |89 | training_parameters := nn.BackpropTrainingParams{ 90 | learning_rate: 0.37 91 | momentum: 0.9 92 | nb_epochs: 300 93 | print_interval: 25 94 | cost_function: .mse // mean squared error 95 | training: nn.Dataset { 96 | inputs: [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]] 97 | expected_outputs: [[0.0], [1.0], [1.0], [0.0]] 98 | } 99 | test: nn.Dataset { 100 | inputs: [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]] 101 | expected_outputs: [[0.0], [1.0], [1.0], [0.0]] 102 | } 103 | test_params: nn.TestParams{ 104 | print_start: 0 105 | print_end: 3 106 | training_interval: 100 107 | } 108 | } 109 |
110 |111 | Now it's the time to train the network! 112 |
113 |114 | model.train(training_parameters) 115 |
116 |117 | We can also save the model by adding that to the end of the program: 118 |
119 |120 | model.save_model('saveXOR') 121 |
122 |123 | And to load a model (to use it or to train it further) you just need to create an empty model like we did at the start and then do: 124 |
125 |126 | model.load_model('saveXOR') 127 |
128 |129 | There it is, we can just run the program and it will train! 130 |
131 |35 | Backpropagation is a commonly used training algorithm for neural networks. 36 |
37 |41 | We need to specify different parameters to be able to make the neural network train successfully. 42 |
43 |44 | The learning_rate is a factor that controls the speed of the adjustements of the weights and biases. If it is high the training can be faster but it can overshoot too. 45 | If it is low the learning will take more time but can be more precise and get closer to the solution more safely. 46 |
47 |48 | The momentum is a factor that controls how much of the precedent adjustement is applied with the new adjustement to the weights and biases. 49 | It allows the network to converge a lot faster in a lot of cases as the modification will get more and more momentum towards the solution and less towards noise from the data. 50 |
51 |52 | The nb_epochs is how many epochs the learning will last. An epoch is finished when the neural network has seen all the dataset one time. 53 | It will apply the modifications calculated with backpropagation on each item of the dataset at the end of each epoch. 54 |
55 |56 | classifier tells the training algorithm if it needs to keep track of the accuracy of the classification and other convenient things for classifiers. 57 |
58 |59 | The neural newtork will print it's performance every print_interval epochs. 60 |
61 |62 | The cost_function is the cost function that you choose for this training. 63 |
64 |65 | training and test are the dataset that are used respectively for the training and the tests. 66 |
67 |68 | test_params allows finer control over frequency of the tests and what is displayed during the tests. 69 |
70 |71 | struct BackpropTrainingParams { 72 | learning_rate f64 73 | momentum f64 74 | nb_epochs int 75 | classifier bool 76 | print_interval int 77 | cost_function CostFunctions 78 | training Dataset 79 | test Dataset 80 | test_params TestParams 81 | } 82 |
83 |35 | Backpropagation is a commonly used training algorithm for neural networks. 36 |
37 |41 | We need to specify different parameters to be able to make the neural network train successfully. 42 |
43 |44 | The learning_rate is a factor that controls the speed of the adjustements of the weights and biases. If it is high the training can be faster but it can overshoot too. 45 | If it is low the learning will take more time but can be more precise and get closer to the solution more safely. 46 |
47 |48 | The momentum is a factor that controls how much of the precedent adjustement is applied with the new adjustement to the weights and biases. 49 | It allows the network to converge a lot faster in a lot of cases as the modification will get more and more momentum towards the solution and less towards noise from the data. 50 |
51 |52 | The batch_size is how many items of a dataset will be used to create each batch. 53 | A batch is a collection of items that the neural network will go through and then apply the changes calculated during these items's backpropagation. 54 | And then it will go through the next batch and repeat. 55 |
56 |57 | The nb_epochs is how many epochs the learning will last. An epoch is finished when the neural network has seen all the dataset one time. 58 |
59 |60 | classifier tells the training algorithm if it needs to keep track of the accuracy of the classification and other convenient things for classifiers. 61 |
62 |63 | The neural newtork will print it's performance every print_batch_interval batchs on every print_interval epochs. 64 |
65 |66 | The cost_function is the cost function that you choose for this training. 67 |
68 |69 | training and test are the dataset that are used respectively for the training and the tests. 70 |
71 |72 | test_params allows finer control over frequency of the tests and what is displayed during the tests. 73 |
74 |75 | pub struct MinibatchesBackpropTrainingParams { 76 | learning_rate f64 77 | momentum f64 78 | batch_size int = 1 79 | nb_epochs int 80 | classifier bool 81 | print_interval int 82 | print_batch_interval int 83 | cost_function CostFunctions 84 | training Dataset 85 | test Dataset 86 | test_params TestParams 87 | } 88 |
89 |