├── DEEP.exe ├── Thumbs.db ├── V2_MANUAL.pdf ├── cudart64_65.dll ├── 9781484235904.jpg ├── errata.md ├── README.md ├── Contributing.md ├── V1 Source ├── README.TXT ├── GENERATIVE.CPP ├── RBM_THR1.CPP ├── MLFN_CUDA.CPP ├── SVDCMP.CPP └── MLFN_THR.CPP ├── LICENSE.txt └── V2 Source ├── MRFFT_P.TXT ├── CUDA_GRAD.TXT ├── SERIES.TXT ├── MRFFT.TXT ├── MRFFT_K.TXT ├── SVDCMP.TXT └── THREADED_GRAD.TXT /DEEP.exe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Apress/deep-belief-nets-vol-1/HEAD/DEEP.exe -------------------------------------------------------------------------------- /Thumbs.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Apress/deep-belief-nets-vol-1/HEAD/Thumbs.db -------------------------------------------------------------------------------- /V2_MANUAL.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Apress/deep-belief-nets-vol-1/HEAD/V2_MANUAL.pdf -------------------------------------------------------------------------------- /cudart64_65.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Apress/deep-belief-nets-vol-1/HEAD/cudart64_65.dll -------------------------------------------------------------------------------- /9781484235904.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Apress/deep-belief-nets-vol-1/HEAD/9781484235904.jpg -------------------------------------------------------------------------------- /errata.md: -------------------------------------------------------------------------------- 1 | # Errata for *Book Title* 2 | 3 | On **page xx** [Summary of error]: 4 | 5 | Details of error here. Highlight key pieces in **bold**. 6 | 7 | *** 8 | 9 | On **page xx** [Summary of error]: 10 | 11 | Details of error here. Highlight key pieces in **bold**. 12 | 13 | *** -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Apress Source Code 2 | 3 | This repository accompanies [*Deep Belief Nets in C++ and CUDA C: Volume 1*](http://www.apress.com/9781484235904) by Timothy Masters (Apress, 2018). 4 | 5 | [comment]: #cover 6 | ![Cover image](9781484235904.jpg) 7 | 8 | Download the files as a zip using the green button, or clone the repository to your machine using Git. 9 | 10 | ## Releases 11 | 12 | Release v1.0 corresponds to the code in the published book, without corrections or updates. 13 | 14 | ## Contributions 15 | 16 | See the file Contributing.md for more information on how you can contribute to this repository. -------------------------------------------------------------------------------- /Contributing.md: -------------------------------------------------------------------------------- 1 | # Contributing to Apress Source Code 2 | 3 | Copyright for Apress source code belongs to the author(s). However, under fair use you are encouraged to fork and contribute minor corrections and updates for the benefit of the author(s) and other readers. 4 | 5 | ## How to Contribute 6 | 7 | 1. Make sure you have a GitHub account. 8 | 2. Fork the repository for the relevant book. 9 | 3. Create a new branch on which to make your change, e.g. 10 | `git checkout -b my_code_contribution` 11 | 4. Commit your change. Include a commit message describing the correction. Please note that if your commit message is not clear, the correction will not be accepted. 12 | 5. Submit a pull request. 13 | 14 | Thank you for your contribution! -------------------------------------------------------------------------------- /V1 Source/README.TXT: -------------------------------------------------------------------------------- 1 | The code in the book has been simplified by the removal 2 | of memory allocation, error checking, et cetera, 3 | in order to provide the clearest presentation. 4 | 5 | Most users would like examples of reasonable ways 6 | to handle these issues. Also, few readers would want 7 | to type in those long subroutines. 8 | 9 | For these reasons I am making available mostly or 10 | entirely complete subroutines in this download. 11 | 12 | However, because these routines are part of a large program, 13 | they necessarily contain numerous references to external 14 | routines. These references must be replaced by the user 15 | with routines appropriate to the program the user 16 | is constructing. This should not be difficult, as these 17 | external references are just to things like memory allocation 18 | and checking if the user has pressed the ESCape key. 19 | 20 | Also, this code includes numerous references to a Model class 21 | whose declaration is not supplied. This is because the 22 | complete declaration is large and complex. But the 23 | references to Model members are all straightforward, 24 | so the reader should have no difficulty adapting these 25 | references to his/her own Model class. -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Freeware License, some rights reserved 2 | 3 | Copyright (c) 2018 Timothy Masters 4 | 5 | Permission is hereby granted, free of charge, to anyone obtaining a copy 6 | of this software and associated documentation files (the "Software"), 7 | to work with the Software within the limits of freeware distribution and fair use. 8 | This includes the rights to use, copy, and modify the Software for personal use. 9 | Users are also allowed and encouraged to submit corrections and modifications 10 | to the Software for the benefit of other users. 11 | 12 | It is not allowed to reuse, modify, or redistribute the Software for 13 | commercial use in any way, or for a user’s educational materials such as books 14 | or blog articles without prior permission from the copyright holder. 15 | 16 | The above copyright notice and this permission notice need to be included 17 | in all copies or substantial portions of the software. 18 | 19 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | AUTHORS OR COPYRIGHT HOLDERS OR APRESS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | SOFTWARE. 26 | 27 | 28 | -------------------------------------------------------------------------------- /V2 Source/MRFFT_P.TXT: -------------------------------------------------------------------------------- 1 | /******************************************************************************/ 2 | /* */ 3 | /* MRFFT_P - This contains the 'permute' routine called from MRFFT. */ 4 | /* */ 5 | /******************************************************************************/ 6 | 7 | void permute ( double *real , double *imag , int ntot , int npts , 8 | int nspan , int inc , int n_facs , int n_sq_facs , double *work1 , 9 | double *work2 , int *index , int *factors , int max_factor ) 10 | { 11 | int i, j, k, ibase, ip, span_inc, sqfac_index, fac_sum, offset, lfm1 ; 12 | int inner_span, inner_span_m1, tot_pts, jmp, jump, jump_save, nearp, farp ; 13 | int this_fac, other_fac, current_index, cycle, limit, which_index ; 14 | double temp, *wptr1, *wptr2 ; 15 | 16 | index[0] = inner_span = inc * nspan ; 17 | inner_span_m1 = inner_span - 1 ; 18 | tot_pts = inc * ntot ; 19 | jump = inner_span / npts ; 20 | current_index = 0 ; 21 | 22 | /* 23 | Do the square factors 24 | */ 25 | 26 | if (n_sq_facs) { 27 | i = 0 ; 28 | j = 2 * n_sq_facs ; 29 | if (j >= n_facs) 30 | --j ; 31 | index[j+1] = jump ; 32 | for (;;) { 33 | index[i+1] = index[i] / factors[i] ; 34 | index[j] = index[j+1] * factors[i] ; 35 | if (++i >= --j) 36 | break ; 37 | } 38 | nearp = jump ; 39 | farp = span_inc = index[1] ; 40 | jump_save = index[j+1] ; 41 | which_index = 1 ; 42 | 43 | if (npts != ntot) { 44 | mv_permute: 45 | limit = nearp + jump ; // Permutations for multivariate transform 46 | while (nearp < limit) { 47 | temp = real[nearp] ; 48 | real[nearp] = real[farp] ; 49 | real[farp] = temp ; 50 | temp = imag[nearp] ; 51 | imag[nearp] = imag[farp] ; 52 | imag[farp] = temp ; 53 | nearp += inc ; 54 | farp += inc ; 55 | } 56 | 57 | k = inner_span - jump ; 58 | nearp += k ; 59 | farp += k ; 60 | if (nearp >= tot_pts-1) { 61 | nearp += jump - tot_pts ; 62 | farp += span_inc - tot_pts ; 63 | if (farp < inner_span_m1) 64 | goto mv_permute ; 65 | } 66 | else 67 | goto mv_permute ; 68 | 69 | for (;;) { 70 | ++which_index ; 71 | farp += index[which_index] - index[which_index-2] ; 72 | if (farp >= index[which_index-1]) 73 | continue ; 74 | which_index = 1 ; 75 | for (;;) { 76 | if (nearp < farp) 77 | goto mv_permute ; 78 | nearp += jump ; 79 | farp += span_inc ; 80 | if (farp >= inner_span_m1) 81 | break ; 82 | } 83 | 84 | if (nearp >= inner_span_m1) 85 | break ; 86 | } 87 | } 88 | 89 | /* 90 | Permutation for single-value transform 91 | */ 92 | 93 | else { 94 | permute_loop: 95 | for (;;) { 96 | temp = real[nearp] ; 97 | real[nearp] = real[farp] ; 98 | real[farp] = temp ; 99 | temp = imag[nearp] ; 100 | imag[nearp] = imag[farp] ; 101 | imag[farp] = temp ; 102 | nearp += inc ; 103 | farp += span_inc ; 104 | if (farp >= inner_span_m1) 105 | break ; 106 | } 107 | 108 | for (;;) { 109 | ++which_index ; 110 | farp += index[which_index] - index[which_index-2] ; 111 | if (farp >= index[which_index-1]) 112 | continue ; 113 | which_index = 1 ; 114 | for (;;) { 115 | if (nearp >= farp) { 116 | nearp += inc ; 117 | farp += span_inc ; 118 | if (farp >= inner_span_m1) 119 | break ; 120 | } 121 | else 122 | goto permute_loop ; 123 | } 124 | if (nearp >= inner_span_m1) 125 | break ; 126 | } 127 | } 128 | 129 | jump = jump_save ; 130 | } 131 | 132 | /* 133 | The square factors are done. If that is all of them, we are done. 134 | */ 135 | 136 | if (2*n_sq_facs+1 >= n_facs) 137 | return ; 138 | 139 | /* 140 | Permutations for square-free factors 141 | */ 142 | 143 | sqfac_index = index[n_sq_facs] ; 144 | k = n_facs - n_sq_facs ; 145 | factors[k] = 1 ; 146 | while (--k > n_sq_facs) 147 | factors[k-1] *= factors[k] ; 148 | 149 | other_fac = factors[n_sq_facs++] ; 150 | lfm1 = other_fac - 1 ; 151 | k = n_sq_facs ; 152 | this_fac = factors[k] ; 153 | fac_sum = 0 ; 154 | 155 | for (i=0 ; i= other_fac) { 158 | fac_sum -= other_fac ; 159 | other_fac = this_fac ; 160 | this_fac = factors[++k] ; 161 | continue ; 162 | } 163 | other_fac = factors[n_sq_facs-1] ; 164 | k = n_sq_facs ; 165 | this_fac = factors[k] ; 166 | index[i++] = fac_sum ; 167 | } 168 | 169 | /* 170 | Determine permutation cycles > 1 171 | */ 172 | 173 | which_index = 0 ; 174 | for (;;) { 175 | while (index[which_index] < 0) 176 | ++which_index ; 177 | cycle = index[which_index++] ; 178 | if (cycle != which_index) { 179 | for (;;) { 180 | i = cycle - 1 ; 181 | cycle = index[i] ; 182 | index[i] = -cycle ; 183 | if (cycle == which_index) 184 | break ; 185 | } 186 | current_index = cycle ; 187 | continue ; 188 | } 189 | index[which_index-1] = -which_index ; 190 | if (which_index == lfm1) 191 | break ; 192 | } 193 | 194 | max_factor *= inc ; 195 | 196 | /* 197 | Reorder 198 | */ 199 | 200 | which_index = current_index ; 201 | tot_pts -= sqfac_index ; 202 | 203 | while (tot_pts >= 0) { 204 | while (index[--which_index] < 0) ; 205 | jmp = jump ; 206 | offset = tot_pts - inc ; 207 | 208 | for (;;) { 209 | inner_span = jmp ; 210 | if (inner_span > max_factor) 211 | inner_span = max_factor ; 212 | jmp -= inner_span ; 213 | cycle = index[which_index] ; 214 | ibase = offset + jump * cycle + jmp ; 215 | ip = ibase + inner_span ; 216 | wptr1 = work1 ; 217 | wptr2 = work2 ; 218 | for (;;) { 219 | *wptr1++ = real[ip] ; 220 | *wptr2++ = imag[ip] ; 221 | ip -= inc ; 222 | if (ip == ibase) 223 | break ; 224 | } 225 | for (;;) { 226 | ip = ibase + inner_span ; 227 | k = jump * (cycle + index[cycle-1]) ; 228 | wptr1 = real - k ; 229 | wptr2 = imag - k ; 230 | for (;;) { 231 | real[ip] = wptr1[ip] ; 232 | imag[ip] = wptr2[ip] ; 233 | ip -= inc ; 234 | if (ip == ibase) 235 | break ; 236 | } 237 | ibase -= k ; 238 | cycle = -index[cycle-1] ; 239 | if (cycle == which_index+1) 240 | break ; 241 | } 242 | ip = ibase + inner_span ; 243 | wptr1 = work1 ; 244 | wptr2 = work2 ; 245 | for (;;) { 246 | real[ip] = *wptr1++ ; 247 | imag[ip] = *wptr2++ ; 248 | ip -= inc ; 249 | if (ip == ibase) 250 | break ; 251 | } 252 | if (! jmp) 253 | break ; 254 | } 255 | if (which_index) 256 | continue ; 257 | 258 | which_index = current_index ; 259 | tot_pts -= sqfac_index ; 260 | if (tot_pts < 0) 261 | break ; 262 | } 263 | return ; 264 | } 265 | 266 | -------------------------------------------------------------------------------- /V2 Source/CUDA_GRAD.TXT: -------------------------------------------------------------------------------- 1 | /* 2 | -------------------------------------------------------------------------------- 3 | 4 | gradient_cuda - Compute the gradient for the entire training set 5 | 6 | -------------------------------------------------------------------------------- 7 | */ 8 | 9 | double CpxAuto::gradient_cuda ( 10 | int nc , // Number of cases 11 | int nin , // Number of (possibly complex) inputs 12 | double *input , // Nc by max_neurons input matrix 13 | int nout , // Number of (possibly complex) outputs 14 | double *target , // Nc by nout target matrix, or autoencoding if NULL 15 | int n_layers , // Number of layers, including output (Number of hidden is one less than this) 16 | int *nhid , // Number of hidden neurons in each layer 17 | int n_weights , // Total (actual) number of weights, including final layer and bias 18 | double *weights[] , // Weight matrices for layers 19 | int use_final_layer_weights , // Use final_layer_weights (vs last weight layer)? 20 | double *grad // Concatenated gradient vector, which is computed here 21 | ) 22 | { 23 | int i, k, n, ilayer, ineuron, ivar, ret_val, ibatch, n_in_batch, n_subsets, istart, istop, n_done, max_batch ; 24 | int n_prior, gradlen, nin_this_layer, timer, n_last_layer_weights, mult ; 25 | double mse, wpen, *wptr, *gptr, *last_layer_weights ; 26 | char msg[256] ; 27 | 28 | assert ( n_layers >= 2 ) ; // Use CUDA only if at least one hidden layer 29 | 30 | mult = is_complex ? 2 : 1 ; 31 | 32 | if (use_final_layer_weights) { // Full CpxAuto model 33 | assert ( target != NULL ) ; 34 | last_layer_weights = final_layer_weights ; 35 | n_last_layer_weights = n_final_layer_weights ; // Per output, not total; If complex, this is actual 36 | } 37 | 38 | else { // Greedily training an autoencoder 39 | assert ( target == NULL ) ; // which may or may not be complex 40 | last_layer_weights = weights[n_layers-1] ; 41 | n_last_layer_weights = mult * (nhid[n_layers-2] + 1) ; 42 | } 43 | 44 | // Setup pointers to gradient for each layer 45 | gptr = grad ; 46 | 47 | for (ilayer=0 ; ilayer