├── .github └── README.md ├── .gitignore ├── .gitmodules ├── dataset └── train.sh ├── lib └── kann-master │ ├── .gitignore │ ├── LICENSE.txt │ ├── Makefile │ ├── README.md │ ├── dna │ └── README.md │ ├── doc │ ├── 01user.md │ ├── 02dev.md │ ├── 11math.tex │ ├── README.md │ └── images │ │ ├── autodiff.png │ │ ├── matmul1.png │ │ ├── matmul2.png │ │ ├── mlp.png │ │ ├── rnn-unroll.png │ │ └── rnn.png │ ├── examples │ ├── README.md │ ├── ae.c │ ├── inspect.c │ ├── keras │ │ ├── mlp.py │ │ └── rnn-bit.py │ ├── mlp.c │ ├── mnist-cnn.c │ ├── rnn-bit.c │ ├── tensorflow │ │ └── mlp.py │ ├── textgen.c │ ├── tiny-dnn │ │ ├── Makefile │ │ └── mlp.cpp │ └── vae.c │ ├── kann.c │ ├── kann.h │ ├── kann_extra │ ├── kann_data.c │ ├── kann_data.h │ └── kseq.h │ ├── kautodiff.c │ └── kautodiff.h ├── models ├── cnn.model ├── mlp.model └── rnn.model └── src └── brain ├── build.sh ├── cnn_train.c ├── dataset.c ├── dataset.h ├── guess.c ├── mlp_train.c ├── norm.c ├── norm.h └── rnn_train.c /.github/README.md: -------------------------------------------------------------------------------- 1 | # C Keyword Spotting 2 | No C++, no dependency hell. Suitable for embedded devices. 3 | 4 | ### Demo 5 | Default models pretrained on 0-9 words: zero one two three four five six seven eight nine. 6 | 7 | ~$ arecord -f S16_LE -c1 -r16000 -d1 test.wav 8 | ~$ aplay test.wav 9 | ~$ dataset/dataset/google_speech_commands/src/features/build.sh 10 | ~$ src/brain/build.sh 11 | ~$ alias fe=dataset/dataset/google_speech_commands/bin/fe 12 | ~$ fe test.wav | bin/guess models/mlp.model 13 | ~$ fe test.wav | bin/guess models/cnn.model 14 | ~$ fe test.wav | bin/guess models/rnn.model 15 | 16 | ### Training 17 | See [google speech commands dataset](https://github.com/42io/dataset/tree/master/google_speech_commands#custom-words) for available words. 18 | 19 | ~$ apt install gcc lrzip wget 20 | ~$ wget https://github.com/42io/dataset/releases/download/v1.0/0-9up.lrz -O /tmp/0-9up.lrz 21 | ~$ lrunzip /tmp/0-9up.lrz -o /tmp/0-9up.data # md5 87fc2460c7b6cd3dcca6807e9de78833 22 | ~$ dataset/train.sh /tmp/0-9up.data 49 13 12 # inputs height, inputs width, outputs 23 | 24 | It takes some time, be patient. Finally you'll see confusion matrix. 25 | 26 | MLP confusion matrix... 27 | zero | 0.91 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 | 603 28 | one | 0.00 0.92 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.02 0.05 0.00 | 575 29 | two | 0.01 0.00 0.89 0.01 0.01 0.00 0.00 0.01 0.00 0.00 0.05 0.01 | 564 30 | three | 0.00 0.00 0.01 0.92 0.00 0.01 0.00 0.01 0.02 0.00 0.03 0.01 | 548 31 | four | 0.00 0.01 0.01 0.00 0.89 0.00 0.00 0.00 0.00 0.00 0.07 0.00 | 605 32 | five | 0.00 0.01 0.00 0.01 0.00 0.84 0.00 0.01 0.01 0.02 0.08 0.00 | 607 33 | six | 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.01 0.01 | 462 34 | seven | 0.01 0.00 0.01 0.01 0.00 0.00 0.01 0.93 0.00 0.00 0.03 0.00 | 574 35 | eight | 0.00 0.00 0.01 0.03 0.00 0.00 0.01 0.00 0.91 0.00 0.03 0.01 | 547 36 | nine | 0.00 0.03 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.86 0.08 0.01 | 596 37 | #unk# | 0.01 0.03 0.02 0.04 0.03 0.03 0.01 0.02 0.02 0.03 0.76 0.01 | 730 38 | #pub# | 0.00 0.00 0.00 0.00 0.01 0.00 0.01 0.01 0.00 0.00 0.01 0.95 | 730 39 | MLP guessed wrong 773... 40 | 41 | CNN confusion matrix... 42 | zero | 0.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 | 603 43 | one | 0.00 0.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.06 0.00 | 575 44 | two | 0.01 0.00 0.95 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.02 0.00 | 564 45 | three | 0.00 0.00 0.01 0.94 0.00 0.00 0.00 0.00 0.01 0.00 0.03 0.00 | 548 46 | four | 0.00 0.00 0.00 0.00 0.94 0.00 0.00 0.00 0.00 0.00 0.05 0.00 | 605 47 | five | 0.00 0.00 0.00 0.00 0.00 0.95 0.00 0.00 0.00 0.00 0.04 0.00 | 607 48 | six | 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 | 462 49 | seven | 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.01 0.00 | 574 50 | eight | 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.01 0.00 | 547 51 | nine | 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.94 0.05 0.00 | 596 52 | #unk# | 0.00 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.95 0.01 | 730 53 | #pub# | 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 | 730 54 | CNN guessed wrong 291... 55 | 56 | RNN confusion matrix... 57 | zero | 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.01 0.00 | 603 58 | one | 0.00 0.95 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.01 0.03 0.00 | 575 59 | two | 0.01 0.00 0.96 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.01 0.00 | 564 60 | three | 0.00 0.00 0.00 0.94 0.00 0.00 0.00 0.00 0.02 0.00 0.02 0.00 | 548 61 | four | 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.00 0.02 0.00 | 605 62 | five | 0.00 0.00 0.00 0.00 0.01 0.98 0.00 0.00 0.00 0.00 0.00 0.00 | 607 63 | six | 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.01 0.00 | 462 64 | seven | 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.98 0.00 0.00 0.01 0.00 | 574 65 | eight | 0.00 0.00 0.00 0.01 0.00 0.01 0.00 0.00 0.97 0.00 0.01 0.00 | 547 66 | nine | 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.97 0.01 0.01 | 596 67 | #unk# | 0.01 0.02 0.00 0.01 0.02 0.01 0.00 0.01 0.01 0.01 0.92 0.01 | 730 68 | #pub# | 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.98 | 730 69 | RNN guessed wrong 254... 70 | 71 | ### Heap Memory Usage 72 | Some magic numbers to know before stepping into embedded world. 73 | 74 | ~$ valgrind dataset/dataset/google_speech_commands/bin/fe test.wav # 606,416 bytes allocated 75 | ~$ fe test.wav | valgrind bin/guess models/mlp.model # 622,768 bytes allocated 76 | ~$ fe test.wav | valgrind bin/guess models/cnn.model # 2,445,100 bytes allocated 77 | ~$ fe test.wav | valgrind bin/guess models/rnn.model # 403,772 bytes allocated 78 | 79 | See [ESP32](https://github.com/42io/esp32_kws) example. 80 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | bin/ 2 | models/rnn-*.model -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "dataset/dataset"] 2 | path = dataset/dataset 3 | url = https://github.com/42io/dataset.git 4 | -------------------------------------------------------------------------------- /dataset/train.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | set -u 5 | 6 | cd "`dirname "${BASH_SOURCE[0]}"`" 7 | 8 | export LC_ALL=C 9 | 10 | readonly DATASET_FILE_NAME=$1 11 | readonly DATASET_NUM_OUTPUT=$4 12 | 13 | bash ./../src/brain/build.sh 14 | 15 | do_confusion_matrix() { 16 | local model=$1 17 | local i 18 | for i in `seq ${DATASET_NUM_OUTPUT}` ; do 19 | awk -v m="${DATASET_NUM_OUTPUT}" '$1 >= m' "${DATASET_FILE_NAME}" \ 20 | | awk -v i="${i}" -v m="${DATASET_NUM_OUTPUT}" '$1 == i - 1 + m || $1 == i - 1 + 2*m' \ 21 | | awk '{for(i=2;i<=NF;i++){if(i>2)printf " ";printf $i} print ""}' \ 22 | | ./../bin/guess "./../models/${model}" \ 23 | | awk '{m=$1;j=1;for(i=j;i<=NF;i++)if($i>m){m=$i;j=i;} for(i=1;i<=NF;i++){if(i>1)printf " ";printf "%d", i==j} print ""}' \ 24 | | awk '{for(i=1;i<=NF;i++)sum[i]+=$i} END {for(j=1;j1)printf " ";printf "%.2f", sum[j]/NR} print " | " NR}' 25 | done 26 | } 27 | 28 | do_validation() { 29 | local model=$1 30 | local i 31 | for i in `seq ${DATASET_NUM_OUTPUT}` ; do 32 | awk -v m="${DATASET_NUM_OUTPUT}" '$1 >= m' "${DATASET_FILE_NAME}" \ 33 | | awk -v i="${i}" -v m="${DATASET_NUM_OUTPUT}" '$1 == i - 1 + m || $1 == i - 1 + 2*m' \ 34 | | awk '{for(i=2;i<=NF;i++){if(i>2)printf " ";printf $i} print ""}' \ 35 | | ./../bin/guess "./../models/${model}" \ 36 | | awk -v x="${i}" '{m=$1;j=1;for(i=j;i<=NF;i++)if($i>m){m=$i;j=i;} if(j!=x)print x}' 37 | done 38 | } 39 | 40 | leave_only_the_best_model() { 41 | local pattern="./../models/${1}" 42 | local best_model="./../models/${2}" 43 | local best_score 44 | local best_score_has_value= 45 | local score 46 | local model 47 | 48 | rm "${best_model}" 49 | 50 | for model in `ls -tr ${pattern}` 51 | do 52 | score=`do_validation "${model}" | wc -l` 53 | echo "${model} ${score}" 54 | if ((best_score_has_value == 0 || best_score > score)); then 55 | best_score_has_value=1 56 | best_score=${score} 57 | cp "${model}" "${best_model}" 58 | fi 59 | done 60 | echo "Best model score is ${best_score}" 61 | rm ${pattern} 62 | } 63 | 64 | echo 'MLP training...' 65 | rm ./../models/mlp.model 66 | ./../bin/mlp_train "${@}" 67 | echo 'MLP confusion matrix...' 68 | do_confusion_matrix 'mlp.model' 69 | echo "MLP guessed wrong `do_validation 'mlp.model' | wc -l`..." 70 | 71 | echo 'CNN training...' 72 | rm ./../models/cnn.model 73 | ./../bin/cnn_train "${@}" 74 | echo 'CNN confusion matrix...' 75 | do_confusion_matrix 'cnn.model' 76 | echo "CNN guessed wrong `do_validation 'cnn.model' | wc -l`..." 77 | 78 | echo 'RNN training...' 79 | rm -f ./../models/rnn-epoch-*.model 80 | ./../bin/rnn_train "${@}" 81 | leave_only_the_best_model 'rnn-epoch-*.model' 'rnn.model' 82 | echo 'RNN confusion matrix...' 83 | do_confusion_matrix 'rnn.model' 84 | echo "RNN guessed wrong `do_validation 'rnn.model' | wc -l`..." -------------------------------------------------------------------------------- /lib/kann-master/.gitignore: -------------------------------------------------------------------------------- 1 | .*.swp 2 | *.o 3 | *.a 4 | *.kan 5 | *.dSYM 6 | a.out 7 | -------------------------------------------------------------------------------- /lib/kann-master/LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2018-2019 Dana-Farber Cancer Institute 4 | 2016-2018 Broad Institute 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining 7 | a copy of this software and associated documentation files (the 8 | "Software"), to deal in the Software without restriction, including 9 | without limitation the rights to use, copy, modify, merge, publish, 10 | distribute, sublicense, and/or sell copies of the Software, and to 11 | permit persons to whom the Software is furnished to do so, subject to 12 | the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be 15 | included in all copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 18 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 19 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 20 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 21 | BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 22 | ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 23 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 24 | SOFTWARE. 25 | -------------------------------------------------------------------------------- /lib/kann-master/Makefile: -------------------------------------------------------------------------------- 1 | CC= gcc 2 | CFLAGS= -g -Wall -Wextra -Wc++-compat -O2 3 | CFLAGS_LIB= #-ansi -pedantic -Wno-long-long # ANSI C does not have inline which affects performance a little bit 4 | CPPFLAGS= -DHAVE_PTHREAD 5 | INCLUDES= -I. 6 | EXE= examples/mlp examples/mnist-cnn examples/inspect examples/textgen examples/rnn-bit \ 7 | examples/ae examples/vae 8 | LIBS= -lpthread -lz -lm 9 | 10 | ifdef CBLAS 11 | CPPFLAGS+=-DHAVE_CBLAS 12 | INCLUDES+=-I$(CBLAS)/include 13 | LIBS=-fopenmp -pthread -L$(CBLAS)/lib -lopenblas -lz -lm 14 | endif 15 | 16 | .SUFFIXES:.c .o 17 | .PHONY:all clean depend 18 | 19 | .c.o: 20 | $(CC) -c $(CFLAGS) $(INCLUDES) $(CPPFLAGS) $< -o $@ 21 | 22 | all:kautodiff.o kann.o kann_extra/kann_data.o $(EXE) 23 | 24 | kautodiff.o:kautodiff.c 25 | $(CC) -c $(CFLAGS) $(CFLAGS_LIB) $(INCLUDES) $(CPPFLAGS) -o $@ $< 26 | 27 | kann.o:kann.c 28 | $(CC) -c $(CFLAGS) $(CFLAGS_LIB) $(INCLUDES) $(CPPFLAGS) -o $@ $< 29 | 30 | kann_extra/kann_data.o:kann_extra/kann_data.c 31 | $(CC) -c $(CFLAGS) -DHAVE_ZLIB $< -o $@ 32 | 33 | examples/mlp:examples/mlp.o kautodiff.o kann.o kann_extra/kann_data.o 34 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 35 | 36 | examples/ae:examples/ae.o kautodiff.o kann.o kann_extra/kann_data.o 37 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 38 | 39 | examples/vae:examples/vae.o kautodiff.o kann.o kann_extra/kann_data.o 40 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 41 | 42 | examples/textgen:examples/textgen.o kautodiff.o kann.o 43 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 44 | 45 | examples/rnn-bit:examples/rnn-bit.o kautodiff.o kann.o 46 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 47 | 48 | examples/inspect:examples/inspect.o kautodiff.o kann.o 49 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 50 | 51 | examples/mnist-cnn:examples/mnist-cnn.o kautodiff.o kann.o kann_extra/kann_data.o 52 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 53 | 54 | clean: 55 | rm -fr *.o */*.o a.out */a.out *.a *.dSYM */*.dSYM $(EXE) 56 | 57 | depend: 58 | (LC_ALL=C; export LC_ALL; makedepend -Y -- $(CFLAGS) $(DFLAGS) -- *.c kann_extra/*.c examples/*.c) 59 | 60 | # DO NOT DELETE 61 | 62 | kann.o: kann.h kautodiff.h 63 | kautodiff.o: kautodiff.h 64 | kann_extra/kann_data.o: kann_extra/kseq.h kann_extra/kann_data.h 65 | examples/ae.o: kann.h kautodiff.h kann_extra/kann_data.h 66 | examples/inspect.o: kann.h kautodiff.h 67 | examples/mlp.o: kann.h kautodiff.h kann_extra/kann_data.h 68 | examples/mnist-cnn.o: kann_extra/kann_data.h kann.h kautodiff.h 69 | examples/rnn-bit.o: kann.h kautodiff.h 70 | examples/textgen.o: kann.h kautodiff.h 71 | examples/vae.o: kann.h kautodiff.h kann_extra/kann_data.h 72 | -------------------------------------------------------------------------------- /lib/kann-master/README.md: -------------------------------------------------------------------------------- 1 | ## Getting Started 2 | ```sh 3 | # acquire source code and compile 4 | git clone https://github.com/attractivechaos/kann 5 | cd kann; make 6 | # learn unsigned addition (30000 samples; numbers within 10000) 7 | seq 30000 | awk -v m=10000 '{a=int(m*rand());b=int(m*rand());print a,b,a+b}' \ 8 | | ./examples/rnn-bit -m7 -o add.kan - 9 | # apply the model (output 1138429, the sum of the two numbers) 10 | echo 400958 737471 | ./examples/rnn-bit -Ai add.kan - 11 | ``` 12 | 13 | ## Introduction 14 | 15 | KANN is a standalone and lightweight library in C for constructing and training 16 | small to medium artificial neural networks such as [multi-layer 17 | perceptrons][mlp], [convolutional neural networks][cnn] and [recurrent neural 18 | networks][rnn] (including [LSTM][lstm] and [GRU][gru]). It implements 19 | graph-based reverse-mode [automatic differentiation][ad] and allows to build 20 | topologically complex neural networks with recurrence, shared weights and 21 | multiple inputs/outputs/costs. In comparison to mainstream deep learning 22 | frameworks such as [TensorFlow][tf], KANN is not as scalable, but it is close 23 | in flexibility, has a much smaller code base and only depends on the standard C 24 | library. In comparison to other lightweight frameworks such as [tiny-dnn][td], 25 | KANN is still smaller, times faster and much more versatile, supporting RNN, 26 | VAE and non-standard neural networks that may fail these lightweight 27 | frameworks. 28 | 29 | KANN could be potentially useful when you want to experiment small to medium 30 | neural networks in C/C++, to deploy no-so-large models without worrying about 31 | [dependency hell][dh], or to learn the internals of deep learning libraries. 32 | 33 | ### Features 34 | 35 | * Flexible. Model construction by building a computational graph with 36 | operators. Support RNNs, weight sharing and multiple inputs/outputs. 37 | 38 | * Efficient. Reasonably optimized matrix product and convolution. Support 39 | mini-batching and effective multi-threading. Sometimes faster than mainstream 40 | frameworks in their CPU-only mode. 41 | 42 | * Small and portable. As of now, KANN has less than 4000 lines of code in four 43 | source code files, with no non-standard dependencies by default. Compatible with 44 | ANSI C compilers. 45 | 46 | ### Limitations 47 | 48 | * CPU only. As such, KANN is **not** intended for training huge neural 49 | networks. 50 | 51 | * Lack of some common operators and architectures such as batch normalization. 52 | 53 | * Verbose APIs for training RNNs. 54 | 55 | ## Installation 56 | 57 | The KANN library is composed of four files: `kautodiff.{h,c}` and `kann.{h,c}`. 58 | You are encouraged to include these files in your source code tree. No 59 | installation is needed. To compile examples: 60 | ```sh 61 | make 62 | ``` 63 | This generates a few executables in the [examples](examples) directory. 64 | 65 | ## Documentations 66 | 67 | Comments in the header files briefly explain the APIs. More documentations can 68 | be found in the [doc](doc) directory. Examples using the library are in the 69 | [examples](examples) directory. 70 | 71 | ### A tour of basic KANN APIs 72 | 73 | Working with neural networks usually involves three steps: model construction, 74 | training and prediction. We can use layer APIs to build a simple model: 75 | ```c 76 | kann_t *ann; 77 | kad_node_t *t; 78 | t = kann_layer_input(784); // for MNIST 79 | t = kad_relu(kann_layer_dense(t, 64)); // a 64-neuron hidden layer with ReLU activation 80 | t = kann_layer_cost(t, 10, KANN_C_CEM); // softmax output + multi-class cross-entropy cost 81 | ann = kann_new(t, 0); // compile the network and collate variables 82 | ``` 83 | For this simple feedforward model with one input and one output, we can train 84 | it with: 85 | ```c 86 | int n; // number of training samples 87 | float **x; // model input, of size n * 784 88 | float **y; // model output, of size n * 10 89 | // fill in x and y here and then call: 90 | kann_train_fnn1(ann, 0.001f, 64, 25, 10, 0.1f, n, x, y); 91 | ``` 92 | We can save the model to a file with `kann_save()` or use it to classify a 93 | MNIST image: 94 | ```c 95 | float *x; // of size 784 96 | const float *y; // this will point to an array of size 10 97 | // fill in x here and then call: 98 | y = kann_apply1(ann, x); 99 | ``` 100 | 101 | Working with complex models requires to use low-level APIs. Please see 102 | [01user.md](doc/01user.md) for details. 103 | 104 | ### A complete example 105 | 106 | This example learns to count the number of "1" bits in an integer (i.e. 107 | popcount): 108 | ```c 109 | // to compile and run: gcc -O2 this-prog.c kann.c kautodiff.c -lm && ./a.out 110 | #include 111 | #include 112 | #include "kann.h" 113 | 114 | int main(void) 115 | { 116 | int i, k, max_bit = 20, n_samples = 30000, mask = (1<