├── .gitignore ├── README.md ├── notes_loglikelihood ├── README.md ├── latex-image-1.png ├── latex-image-10.png ├── latex-image-11.png ├── latex-image-2.png ├── latex-image-3.png ├── latex-image-4.png ├── latex-image-5.png ├── latex-image-6.png ├── latex-image-7.png ├── latex-image-8.png └── latex-image-9.png ├── notes_lse ├── README.md ├── latex-image-1.png ├── latex-image-2.png ├── latex-image-3.png ├── latex-image-4.png ├── latex-image-5.png ├── latex-image-6.png ├── latex-image-7.png ├── latex-image-9.png ├── sinus.png └── training.png ├── requirements.txt ├── sparse-coding ├── Readme.md ├── cnn_ae_sparsity.py ├── cnn_sparsity.py ├── results-ae.jpg ├── results-cnn.jpg ├── results-fc.jpg ├── sparse-coding-equation.jpg ├── sparsecoding.png └── sparsity.py ├── tf-architecture ├── .gitignore ├── README.md └── archi │ ├── README │ ├── hpsearch │ ├── hyperband.py │ └── randomsearch.py │ ├── main.py │ └── models │ ├── __init__.py │ └── basic_model.py ├── tf-freeze ├── freeze.py ├── img │ ├── Screen Shot 2016-11-21 at 17.04.36.png │ └── Screen Shot 2016-11-21 at 17.11.41.png ├── load.py ├── server.py └── tiny_model.py ├── tf-mut-control ├── README.md ├── anim1.gif ├── anim1.m4v ├── anim2.gif ├── anim2.m4v ├── animation.key ├── animation2.key └── dyn_array.py ├── tf-queues ├── README.md ├── enqueue.py ├── lorem_ipsum1.txt ├── lorem_ipsum2.txt ├── ptb_producer.py ├── reader_test.py └── test_batching.py ├── tf-save-load ├── README.md ├── embedding.py ├── lorem.txt ├── results │ └── .gitkeep └── sc1.png ├── tf-shape └── Readme.md └── tf-uat ├── README.md ├── train.sh ├── universal.py └── universal_mnist.py /.gitignore: -------------------------------------------------------------------------------- 1 | **/env 2 | **/*data* 3 | **/results/* 4 | gist-medium/* 5 | .DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # blog 2 | Technical blog repo of metaflow 3 | -------------------------------------------------------------------------------- /notes_loglikelihood/README.md: -------------------------------------------------------------------------------- 1 | # ML notes: Why the log-likelihood? 2 | 3 | ## TL, DR! 4 | ### EN 5 | Here is my first note on the machine learning subject about the reasons behind the use of the log-likelihood. I work out the equations from the definition of the likelihood to the log-likelihood removing any untold assumptions between each line of calculus. 6 | Bonus: I add some remarks about SGD and MAP. 7 | 8 | ### FR 9 | Voici ma première note sur le Machine Learning à propos des raisons d'usages du log-likelihood. Je refais le chemin de pensé qui mène de la définition du likelihood aux raisons de l'utilisation du log-likelihood en supprimant tout non dis pour chaque ligne de calcul. 10 | En bonus, je fais quelques remarques sur les raisons du SGD et du MAP. 11 | 12 | 13 | ## Latex 14 | ``` 15 | \begin{align} 16 | \underset{\theta}{\text{ argmax }} L(\theta, m | \mathcal{D}) &= \underset{\theta}{\text{ argmax }} p(\mathcal{D}|\theta, m) \\ 17 | &= \underset{\theta}{\text{ argmax }} p_{D_1, \ldots, D_n}(d_1, \ldots, d_n | \theta,m) \\ 18 | &\stackrel{\text{independence}}{=} \underset{\theta}{\text{ argmax }} \prod_i p_{D_i}(d_i | \theta, m) \\ 19 | &\stackrel{\text{identically distributed}}{=} \underset{\theta}{\text{ argmax }} \prod_i p(d_i | \theta, m) \\ 20 | &= \underset{\theta}{\text{ argmax }} \sum_i \log(p(d_i | \theta, m)) \\ 21 | &= \underset{\theta}{\text{ argmax }} \frac{1}{n} \sum_i \log(p(d_i | \theta, m)) \\ 22 | &= \underset{\theta}{\text{ argmax }} E_{d \sim U_{\mathcal{D}}}[log(p(d|\theta,m))] 23 | \end{align} 24 | ``` 25 | 26 | ``` 27 | P(\mathcal{D}|\theta,m) = P(\mathcal{D}|\theta,\eta_1, \ldots, \eta_K, f_m) 28 | ``` -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-1.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-10.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-11.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-2.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-3.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-4.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-5.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-6.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-7.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-8.png -------------------------------------------------------------------------------- /notes_loglikelihood/latex-image-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_loglikelihood/latex-image-9.png -------------------------------------------------------------------------------- /notes_lse/README.md: -------------------------------------------------------------------------------- 1 | # ML notes: why the Least Square Error? 2 | 3 | ## TL, DR! 4 | ### EN 5 | A second ML note exploring how the Least Square Error emerge from supervised learning and the additive white Gaussian noise model: 6 | - I start from the end of my last note and explore the impact of supervised learning on the math 7 | - I explore the impact of the additive white gaussian noise model 8 | - Bonus: I make a simple experiment (using TF) on a noisy regression example showing that if you respect the assumptions, you indeed retrieve the original function on a given interval 9 | 10 | ### FR 11 | Une seconde note ML ou j'explore comment la "least square error" émerge de l'apprentissage supervisé et du model de bruit blanc Gaussien additif: 12 | - Je repars de ma dernière note et j'explore l'impact de l'apprentissage supervisé sur les maths. 13 | - J'explore ensuite l'impact du model de bruit blanc Gaussien additif 14 | - Bonus: Je fais une experimentation simple (utilisant TF) sur une regression bruitée, montrant que tant que l'on respect exactement les postulats probabilistic, on reconstruit bien la fonction d'origine sur un interval donné. 15 | 16 | 17 | ## Latex 18 | ``` 19 | \begin{equation*} 20 | \underset{\theta}{\text{ argmax }} L(\theta, m | \mathcal{D}) \stackrel{\text{i.i.d.}}{=} \underset{\theta}{\text{ argmax }} \sum_{d^{(i)} \in \mathcal{D}} \log(p(d^{(i)} | \theta, m)) \tag{1} 21 | \end{equation*} 22 | With: 23 | \begin{itemize} 24 | \item $ \theta $ the model parameters (a random variable) 25 | \item $ m $ the model hypothesis (a random variable) 26 | \end{itemize} 27 | 28 | ``` 29 | 30 | ``` 31 | \begin{align} 32 | p(d^{(i)} | \theta, m) &= p(x^{(i)}, y^{(i)} | \theta, m) \tag{2} \\ 33 | &= p(y^{(i)} | x^{(i)}, \theta, m) \times p(x^{(i)} | \theta, m) \tag{3} \\ 34 | &= p(y^{(i)} | x^{(i)}, \theta, m) \times p(x^{(i)}) \tag{4} 35 | \end{align} 36 | ``` 37 | 38 | ``` 39 | \underset{\theta}{\text{ argmax }} \sum_i \log(p(d^{(i)} | \theta, m)) &= \underset{\theta}{\text{ argmax }} \sum_i \log(p(y^{(i)} | x^{(i)}, \theta, m)) \tag{5} 40 | ``` 41 | 42 | ``` 43 | \forall i, y^{(i)}= h(x^{(i)}) + z^{(i)} \text{ In term of random variables: } Y = h(X) + Z 44 | ``` 45 | 46 | ``` 47 | \begin{align} 48 | &\forall i, z^{(i)} \sim \mathcal{N}(0, \sigma^2) \Rightarrow p(z^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(z^{(i)})^2}{2\sigma^2}} \tag{6} \\ 49 | &\forall i \neq j \text{, } p(z^{(i)} | z^{(j)}) = p(z^{(i)}) \text{ and } \forall i \text{, } p(z^{(i)} | x^{(i)}) = p(z^{(i)}) \tag{7} 50 | \end{align} 51 | ``` 52 | 53 | ``` 54 | \forall i \text{, } y^{(i)} \approx h_{\theta, m}(x^{(i)}) + z^{(i)} 55 | ``` 56 | 57 | ``` 58 | Given that: $ Y = h_{\theta, m}(X) + Z $ and $ d = \{x, y\} $ we have: 59 | \begin{align} 60 | p(Y=y | X=x, \theta, m) &= p(Z= y - h_{\theta, m}(X) | X=x, \theta, m) \tag{8} \\ 61 | &= p(Z= y - h_{\theta, m}(x)) \tag{9} 62 | \end{align} 63 | \begin{align} 64 | \Leftrightarrow &p(Y=y | x, \theta, m) = \frac{1}{\sqrt{2\pi}\sigma} e^{- \frac{(y - h_{\theta, m}(x))^2}{2\sigma^2}} \tag{10} \\ 65 | \Leftrightarrow &log(p(Y=y | x, \theta, m)) = \log\frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{\sigma^2} \frac{1}{2} (y - h_{\theta, m}(x))^2 \tag{11} \\ 66 | \Rightarrow &\underset{\theta}{\text{ argmax }} \sum_i \log(p(d | \theta, m)) = \underset{\theta}{\text{ argmin }} \frac{1}{2}\sum_i(y - h_{\theta, m}(x))^2 \tag{12} 67 | \end{align} 68 | ``` 69 | 70 | ``` 71 | p(y^{(i)}|x^{(i)}, \theta, m) \neq p(y^{(i)}|x^{(i)}; \theta, m) 72 | ``` 73 | -------------------------------------------------------------------------------- /notes_lse/latex-image-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-1.png -------------------------------------------------------------------------------- /notes_lse/latex-image-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-2.png -------------------------------------------------------------------------------- /notes_lse/latex-image-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-3.png -------------------------------------------------------------------------------- /notes_lse/latex-image-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-4.png -------------------------------------------------------------------------------- /notes_lse/latex-image-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-5.png -------------------------------------------------------------------------------- /notes_lse/latex-image-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-6.png -------------------------------------------------------------------------------- /notes_lse/latex-image-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-7.png -------------------------------------------------------------------------------- /notes_lse/latex-image-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/latex-image-9.png -------------------------------------------------------------------------------- /notes_lse/sinus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/sinus.png -------------------------------------------------------------------------------- /notes_lse/training.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/notes_lse/training.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | appdirs==1.4.0 2 | bleach==1.5.0 3 | cycler==0.10.0 4 | html5lib==0.9999999 5 | Markdown==2.6.9 6 | matplotlib==2.0.0 7 | numpy==1.13.1 8 | packaging==16.8 9 | protobuf==3.4.0 10 | pyparsing==2.1.10 11 | python-dateutil==2.6.0 12 | pytz==2016.10 13 | six==1.10.0 14 | tensorflow==1.3.0 15 | tensorflow-tensorboard==0.1.5 16 | Werkzeug==0.12.2 17 | -------------------------------------------------------------------------------- /sparse-coding/Readme.md: -------------------------------------------------------------------------------- 1 | # Sparse coding: A simple exploration 2 | 3 | *Disclaimer:* I don't own a Ph.D in machine learning, yet I'm deeply passionate about the field and try to learn as much as I can about it. 4 | 5 | While I'm learning a new subject, I found out that it was a very valuable exercise to write a comprehensive guide about my study. So I'm now sharing my progress looking for useful feedback while helping others reach understanding faster. 6 | 7 | I've been spending the last few days to understand what is sparse coding, Here are my results! 8 | 9 | We won't dive hard in any theoretical math, this article is trying to explore the sparse coding notion and see their impact on neural networks: I prefer to experiment with a notion and implement some concrete hypothesis than just playin with the math, i believe that in ML, concrete experience seems more relevant than pure theory. 10 | 11 | **Please, if you find any errors, contact me: I'll be glad to hear from you :)** 12 | 13 | **The reader should already have some understanding of what is a neural networks before reading this** 14 | 15 | ## What is Sparse coding? 16 | 17 | ### The mathematical standpoint 18 | Sparse coding is the study of algorithms which aim to learn a useful **sparse representation** of any given data. Each datum will then be encoded as a *sparse code*: 19 | - The algorithm only needs input data to learn the sparse representation. This is very useful since you can apply it directly on any kind of data, it is called unsupervised learning 20 | - It will automatically find the representation without loosing any information (As if one could automatically reveals the intrinsic atoms of one's data). 21 | 22 | To do so, sparse coding algorithms try to satisfy two constraints at the same time: 23 | - For a given datum as a vector **x**, it will try to learn a "useful" sparse representation as a vector **h** 24 | - For each representation as a vector **h**, it will try to learn a basis **D** to reconstruct the original datum as a vector **x**. 25 | 26 | The mathematical representation of the general objective function for this problem speaks for itself: 27 | ![General equation of the sparse coding algorithm](sparse-coding-equation.jpg "Sparse coding equation") 28 | where: 29 | - **N** is the number of datum in the data 30 | - **x_k** is the **k** given vector of the data 31 | - **h_k** is the sparse representation of **x_k** 32 | - **D** D is a the decoder matrix (more on that later) 33 | - **lambda** is the coefficient of sparsity 34 | - **C** is a given constant 35 | 36 | The general form looks like we are trying to nest two optimization problem (The double "min" expression). I prefer to see this as two different optimization problem competing against each other to find the best middle ground possible: 37 | - The first "min" is acting only on the left side of the sum trying to minimize the **reconstruction loss** of the input by tweaking **D**. 38 | - The second "min" tries to promote sparsity by minimizing the L1-norm of the sparse representation **h** 39 | Put simply, we are just trying to resolve a problem (in this case, reconstruction) while using the least possible amount of resource we can to store our data. 40 | 41 | **Note on the constraint on D rows:** 42 | 43 | If you don't apply any constraint on D rows you can actually minimize **h** as much as you want while making **D** big enough to compensate. We want to avoid this behaviour as we are looking for vector containing actual zero value with only as few as possible non-zeros and big values. 44 | 45 | The value of C doesn't really matter, in our case, we will choose **C = 1** 46 | 47 | 48 | #### The vector space interpretation: 49 | If you try to find a vector space to plot a representation of your data, you need to find a basis of vectors. 50 | 51 | Given a number of dimensions, **sparse coding** tries to learn an **over-complete basis** to represent data efficiently. To do so, you must have provided at first enough dimensions to learn this **over-complete basis**. 52 | 53 | In real life, you just give more (or same amount) dimensions than the number of dimensions the original data are encoded in. For example, for images of size 28x28 with only 1 channel (gray scale), our space containing our input data has 28x28x1=784 dimension 54 | 55 | **Why would you need it to be over-complete?** 56 | 57 | An over-complete basis means redundancy in your basis, and vectors (while training) will be able to "compete" to represent data more efficiently. Also you are assured, you won't need all dimensions to represent any data point: In an over-complete basis, you always can set some dimensions to 0. 58 | 59 | Features will have multiple representation in the obtained basis, but by adding the sparsity constraint you can enforce a unique representation as each feature will get encoded with the most sparse representation. 60 | 61 | ### The biological standpoint 62 | From the biological standpoint the notion of sparse code (and sparse coding) comes after the more general notion of neural code (and neural coding). 63 | 64 | *Wikipedia says:* 65 | ``` 66 | Neural coding is concerned with characterizing the relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among the electrical activity of the neurons in the ensemble. 67 | ``` 68 | Let's say you have N binary (for simplicity) neurons (which can be biological or virtual). Basically: 69 | - You will feed some inputs to your neural networks which will give you an output in returns. 70 | - To compute this output, the neural network will strongly activate some neurons while some others won't activate at all. 71 | - The observation of those activations for the a given input is what we call the neural code and the quantity of activations on some data is called the **average activity ratio** 72 | - Training your neurons to reach an acceptable neural code is what we call neural coding. 73 | 74 | Now that we know what a neural code is we can start to guess what neural code could looks like. You could have only 4 big cases: 75 | - No neuron activate at all 76 | - Only one neuron gets activated 77 | - Less than half of the neurons get activated 78 | - Half of the neurons get activated 79 | 80 | Let's put aside the case where no neurons activate at all as this case is pretty dull. 81 | 82 | ### Local code 83 | At one extreme of low average activity ratio are local codes: Only one neurons gets activated per feature of the stimulus. No overlap between neurons is possible: one neuron can't react to different features. 84 | 85 | Note that a complex stimulus might contain multiple feature (multiple neurons can fire at the same time for a complex stimulus) 86 | 87 | ### Dense code 88 | The other extreme corresponds to dense code: half of the neurons get activated per feature. 89 | 90 | Why half (average activity ratio of 0.5)? 91 | 92 | Remember we are looking at neurons in the ensemble, for any given distribution of activated neurons with an activity ratio > 0.5, you could swap activated ones with the inactivated ones without losing information in the ensemble. Remember that all neurons are used to encode information, to the opposite of sparse code, one neurons doesn't hold any specific information in itself. 93 | 94 | ### Sparse code 95 | Everything in between those two extremes is what we call sparse code. We take into account only a subset of neurons to encode information. 96 | 97 | Sparce code can be seen as a good compromise between balancing all the different characteristics of neural codes: computation power, fault tolerance, interference, complexity ... . This is why in an evolutionary perspective, it make sens that the brain works only with sparse code. 98 | 99 | *More on that in the why section.* 100 | 101 | ### Linking the two fields 102 | **How does neural code relate to the mathematical definition of sparse coding?** 103 | 104 | This two fields can be linked thanks to neural networks: You can interpret a neural network in terms of projections of information into vector spaces: Each layer of neurons will receive a representation of the data (in a vector space) from the previous layer and project it to another vector space. 105 | 106 | **If you interpret a layer as a data projection, what can be interpreted as the basis of the vector space?** 107 | 108 | It's the neurons per layer: each neurons would represent a vector. if a neuron is activated in a layer: it means that you need this dimension to encode your data. The less neurons is activated per projection, the closer you are from a sparse representation of your data and a sparse code in the biological standpoint. 109 | 110 | Choosing the number of neurons per layer is then equivalent as choosing the number of dimension of your over-complete basis which will represent your data. It is the same idea as word embedding when you choose the number of dimensions of the word vector space to encode your words for example. 111 | 112 | *Remember the mathematical equation* 113 | 114 | You can notice the presence of 2 unknown variables in the equation: **D** and **h**. In neural networks: 115 | - **h = E . x** (I avoid writing the bias terms for the sake of simplicity). With **E** a matrix 116 | 117 | You can interpret the neural network weights as the weights of an encoder (trying to create the most sparse representation of the data it cans) and the neural network weight D as the weights of the decoder (Trying to reproduce the original content as best as it cans). Also it happens to be what we call an auto-encoder. 118 | 119 | *Remark:* 120 | How much neurons? you said. It seems impossible to tell without experiment, this is why we use cross validation to optimize hyper-parameters. 121 | If the number of neurons (and so, dimensions) is greater than the intrinsic vector space of the data, then it means that you can infer an over-complete basis and so, you can achieve sparse coding in your neural network. If you don't have enough neurons, you will compress your data with some loss, getting worst results when you try to reproduce the data. 122 | 123 | Why not using as much neurons as we can? Because this is just too expensive and terribly inefficient. 124 | 125 | ## Why having a sparse representation of data can be useful? 126 | ### Intuitions 127 | A good sparse coding algorithm would be able to find a very good over-complete basis of vectors representing the inherent structure of the data. By "very good", I mean "meaningful" for a human: 128 | 129 | It could be compared to: 130 | - Decomposing all the matters in the universe in a composition of atoms 131 | - Decomposing music in a composition of just a few notes 132 | - Decomposing a whole language in just a few letters 133 | And yet, from those basic components, you can generate all the music or all the matters or all the words in a language back. 134 | 135 | Having an algorithm, which could infer those basic components just by looking at the data would be very useful, because it would imply that we could pretrain a neural network to learn *always useful features* for any future task that you would need to solve. 136 | 137 | Also sparse code, brings a loads of interesting characteristics for the brain which can be apply to computers too and are often overlooked. 138 | 139 | #### Interference 140 | This scheme of neural code avoids the risk of unwanted interference between neurons: One neurons can't encode two features at the same time. 141 | 142 | in other schemes (see dense/sparse code): one neuron might react to multiple feature, if it happens that the 2 feature appears at the same time in the stimuli, one could imagine the emergence of a new signal from a neurons reacting to both information which could be misleading. 143 | 144 | #### Parallelism 145 | Since not all neurons are used at the same time and interference are unlikely in sparse code, neurons can work asynchronously and parallelise the computation of information. 146 | 147 | It is important to note that parallelism is closely linked to interference, in a dense code, since all neurons are used to encode any piece of information, trying to parallelise computation leads to a great risk of interference. 148 | 149 | #### Fault tolerance 150 | Having multiple neurons firing for the same stimulus can help one to recover from an error coming from a neuron. Local code is especially sensible to errors, and dense code can always recover. 151 | 152 | #### Memory capacity 153 | In local code, since one neurons represent a feature you can encode only as much as neurons. 154 | 155 | If you think about neurons as a binary entity (1 or 0), we have two state. It means that for dense code we have actually **2^N** different combination of neurons to choose from for each feature of the input data which is A LOT! 156 | 157 | In sparse code, if we set **d** to be the average density of neurons activation per stimulus, we then have **2^d** different combination this time. It's interesting because since **d** can be a lot smaller than **N**, we have room for other characteristics. 158 | 159 | #### Complexity And Decoding 160 | Obviously, the more neurons you need to observe to detect what feature has been encoded, the more complex it will be to decode information. 161 | 162 | #### Sum up 163 | (N: total numbers of neurons, d: average density of activations, we simplify the behaviour of neurons as activated or not) 164 | 165 | | | Local code | Sparse code | Dense code | 166 | |------------|------------|-------------|------------| 167 | | Interference | None | Unlikely | Likely | 168 | | Parallelism | Possible | Possible | Dangerous | 169 | | Decoding | easy | Average | Complex | 170 | | Fault tolerant | None | Ok | Very good | 171 | | Memory Capacity | N | 2^(d) | 2^N | 172 | | Complexity | None | low | Very high | 173 | | Computation power or energy | very low | low | high | 174 | 175 | As you can see sparse code seems to be the perfect middle ground between having enough computation power/memory/stability and using as less as possible energy to handle the task at hand. 176 | 177 | 178 | 179 | ## How to use it in neural networks? 180 | The simplest known usage of combining neural networks and sparse coding is in sparse auto-encoder: It is a neural network that will try to mimic the identity function while under some constraint of sparsity in the hidden layers or the objective function. 181 | 182 | After learning the concept, I wanted to try some experiments that I've never seen around. My idea was to use the ReLU activation combined with a L1 norm in the objective functions to promote sparse neurons activations: Let's see (with code and TensorFlow) how far can we go with this idea. 183 | 184 | ### Some experiments with TensorFlow 185 | We will now try to validate our previous statement with some very simple neural networks. 186 | Multiple questions here: 187 | - Can we reach (at all) local code on the MNIST dataset? 188 | - Can we reach sparse code while keeping a good accuracy? 189 | - Does all neurons get activated through the complete dataset? 190 | 191 | Experiments: 192 | - A simple fully connected neural network used as a classifier of digits on MNIST, i add the sparsity constraint with l1-norm directly on the hidden layer 193 | - I will test with different coefficient of sparsity, **especially 0**, given us a baseline 194 | - Let's move forward with a CNN, doing the same experiment as before (l1-norm on the activations of the feature map this time). Since we are trying to classify images, we should see an improvement. 195 | - Finally, I will add a reconstruction layer and loss to our CNN architecture and train them jointly to see if any improvement happen 196 | 197 | 198 | #### Fully connected neural network 199 | I will do this experiment with only one hidden layer of 784 neurons because our input data is a 28x28 image so I'm sure I can reencode the data in as many dimensions as the original space. Let's see how sparse we can get! 200 | 201 | **All code files can be found in the repo** 202 | 203 | First we load the MNIST dataset and prepare our placeholder for training/testing: 204 | ```python 205 | import time, os 206 | 207 | import tensorflow as tf 208 | from tensorflow.examples.tutorials.mnist import input_data 209 | 210 | dir = os.path.dirname(os.path.realpath(__file__)) 211 | mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 212 | 213 | # Fully connected model 214 | # Number of parameters: (784 * 784 + 784) + (784 * 10 + 10) = 615440 + 7850 = 623290 215 | # Dimensionality: R^784 -> R^784 -> R^10 216 | 217 | # Placeholder 218 | x = tf.placeholder(tf.float32, shape=[None, 784]) 219 | y_true = tf.placeholder(tf.float32, shape=[None, 10]) 220 | 221 | sparsity_constraint = tf.placeholder(tf.float32) 222 | ``` 223 | We add our neural layer and calculate the average density of activations: 224 | ```python 225 | # Variables 226 | with tf.variable_scope('NeuralLayer'): 227 | W = tf.get_variable('W', shape=[784, 784], initializer=tf.random_normal_initializer(stddev=1e-1)) 228 | b = tf.get_variable('b', shape=[784], initializer=tf.constant_initializer(0.1)) 229 | 230 | z = tf.matmul(x, W) + b 231 | a = tf.nn.relu(z) 232 | 233 | # We graph the average density of neurons activation 234 | average_density = tf.reduce_mean(tf.reduce_sum(tf.cast((a > 0), tf.float32), axis=[1])) 235 | tf.summary.scalar('AverageDensity', average_density) 236 | ``` 237 | 238 | We add the softmax layer 239 | ```python 240 | with tf.variable_scope('SoftmaxLayer'): 241 | W_s = tf.get_variable('W_s', shape=[784, 10], initializer=tf.random_normal_initializer(stddev=1e-1)) 242 | b_s = tf.get_variable('b_s', shape=[10], initializer=tf.constant_initializer(0.1)) 243 | 244 | out = tf.matmul(a, W_s) + b_s 245 | y = tf.nn.softmax(out) 246 | ``` 247 | 248 | We finish with the loss: note that we add our sparsity constraint on activations 249 | ``` 250 | with tf.variable_scope('Loss'): 251 | epsilon = 1e-7 # After some training, y can be 0 on some classes which lead to NaN 252 | cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y + epsilon), axis=[1])) 253 | # We add our sparsity constraint on the activations 254 | loss = cross_entropy + sparsity_constraint * tf.reduce_sum(a) 255 | 256 | tf.summary.scalar('loss', loss) # Graph the loss 257 | ``` 258 | 259 | Now we are going to merge all the so far defined summaries and only then we will calculate accuracy and create its summary. 260 | We do that in this order because it is convenient and we avoid graphing the accuracy every iteration 261 | ```python 262 | summaries = tf.summary.merge_all() # This is convenient 263 | 264 | with tf.variable_scope('Accuracy'): 265 | correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_true, 1)) 266 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 267 | acc_summary = tf.summary.scalar('accuracy', accuracy) 268 | ``` 269 | Now we finish with the training/testing part 270 | ```python 271 | # Training 272 | adam = tf.train.AdamOptimizer(learning_rate=1e-3) 273 | train_op = adam.minimize(loss) 274 | sess = None 275 | # We iterate over different sparsity constraint 276 | for sc in [0, 1e-4, 5e-4, 1e-3, 2.7e-3]: 277 | result_folder = dir + '/results/' + str(int(time.time())) + '-fc-sc' + str(sc) 278 | with tf.Session() as sess: 279 | sess.run(tf.global_variables_initializer()) 280 | sw = tf.summary.FileWriter(result_folder, sess.graph) 281 | 282 | for i in range(20000): 283 | batch = mnist.train.next_batch(100) 284 | current_loss, summary, _ = sess.run([loss, summaries, train_op], feed_dict={ 285 | x: batch[0], 286 | y_true: batch[1], 287 | sparsity_constraint: sc 288 | }) 289 | sw.add_summary(summary, i + 1) 290 | 291 | if (i + 1) % 100 == 0: 292 | acc, acc_sum = sess.run([accuracy, acc_summary], feed_dict={ 293 | x: mnist.test.images, 294 | y_true: mnist.test.labels 295 | }) 296 | sw.add_summary(acc_sum, i + 1) 297 | print('batch: %d, loss: %f, accuracy: %f' % (i + 1, current_loss, acc)) 298 | ``` 299 | 300 | *Since we are talking about images, I've done another experiment using a convolutional neural network and 200 feature maps (Remember that the weight are shared between feature map in convolutional neural net, we are just looking for a local feature everywhere in the image)* 301 | 302 | Et voila! 303 | Let's launch our little experiment (Do it yourself to see graphs with TensorBoard) 304 | Here is the summary for (20000 iterations): 305 | 306 | Fully connected neural net (N = 784 neurons): 307 | 308 | ![Fully connected neural net results](results-fc.jpg "Fully connected neural net results") 309 | 310 | **Some remarks:** 311 | - Without sparsity constraint the networks converge under a dense network (<~0.5 average activity ratio) 312 | - We almost reach local code with an AVR(average activity ratio) of 1.67, for 89% accuracy. Clearly better than a random monkey! 313 | - Without loosing accuracy, we've been able to reach an AVR of 5.4 instead of 156.3 with a good sparsity parameter which is only 0.6% of activated neurons. 314 | - It's hard to see but we reached overfitting with the sparsity constraint. The absolute best score would have been 0.981 accuracy for an AVR of 28.19 (Sparsity constraint of 1e-05) 315 | 316 | #### Convolutionnal neural network 317 | Convolutionnal neural net (N = 39200, note that neurons are spatially distributed): 318 | 319 | ![Convolutionnal neural net results](results-cnn.jpg "Convolutionnal neural net results") 320 | 321 | **Some remarks:** 322 | - We are very close to the capacity of the fully connecter neural network. 323 | - Without loosing accuracy, we've been able to reach an AVR of 146.2 instead of 10932 with a good sparsity parameter which is only 0.3% of activated neurons. 324 | 325 | Finally I venture in the auto-encoder area to see if my idea makes more sense there. 326 | 327 | #### Convolutionnal Sparse auto-encoder 328 | Convolutionnal neural net jointly trained with a sparse auto-encoder (N = 39200): 329 | 330 | ![Autoencoder results](results-ae.jpg "Autoencoder results") 331 | 332 | **Some remarks:** 333 | - We actually **accelerated**(in terms of iteration, not time) and **improved** the learning of the classifier with sparsity. That's an interesting result! 334 | - The auto-encoder got worse with sparsity, until you quantize again real values to integer, In this case, the auto-encoder didn't change much (Average squared errors per pixels < 1). 335 | 336 | ## Conclusion 337 | It was very interesting to explore sparse coding and finally test the concept with some very simple but concrete example. TensorFlow is definitely very efficient for fast prototyping, having access to every variables and operations allows one to customize any process very easily. 338 | 339 | I don't think this work would lead to anything interesting in terms of neural networks, but I believe it clearly shows the concept of sparse coding in this field. 340 | 341 | Also, imposing the sparsity constraint on ReLU activations made me think about what we call: [lateral inhibition](https://en.wikipedia.org/wiki/Lateral_inhibition) in our brain. Since the sparsity constraint is applied on the global network activations, it resonate with the description found in the wikipedia article: 342 | 343 | *An often under-appreciated point is that although lateral inhibition is visualised in a spatial sense, it is also thought to exist in what is known as "lateral inhibition across abstract dimensions." This refers to lateral inhibition between neurons that are not adjacent in a spatial sense, but in terms of modality of stimulus. This phenomenon is thought to aid in colour discrimination* 344 | 345 | Who knows, maybe this is an interesting journey to pursue? 346 | 347 | **P.S.: If you find any mistakes of incoherences in this writeup, please contact me: morgan@explee.com, thank you!** 348 | 349 | ## Running the experiment using nvidia-docker 350 | Run experiments (using GPU): 351 | 352 | `nvidia-docker run --name sparsity --rm -v ~/gist/sparsity:/sparsity gcr.io/tensorflow/tensorflow:0.10.0-gpu python /sparsity/sparsity.py && python /sparsity/cnn_sparsity.py && python cnn_ae_sparsity.py` 353 | 354 | Look at tensorboard: 355 | 356 | `nvidia-docker run --name tensorboard -it -d -p 6006:6006 -v ~/gist/sparsity:/sparsity gcr.io/tensorflow/tensorflow:0.10.0-gpu tensorboard --logdir /sparsity/results --reload_interval 30` 357 | 358 | 359 | ## References 360 | - https://en.wikipedia.org/wiki/Neural_coding 361 | - https://en.wikipedia.org/wiki/Neural_coding#Sparse_coding 362 | - http://ufldl.stanford.edu/wiki/index.php/Sparse_Coding 363 | - http://deeplearning.stanford.edu/wiki/index.php/Sparse_Coding:_Autoencoder_Interpretation 364 | - http://www.scholarpedia.org/article/Sparse_coding 365 | - http://www.scholarpedia.org/article/Donald_Olding_Hebb 366 | - http://www.scholarpedia.org/article/Oja_learning_rule 367 | - https://www.youtube.com/user/hugolarochelle/search?query=sparse 368 | -------------------------------------------------------------------------------- /sparse-coding/cnn_ae_sparsity.py: -------------------------------------------------------------------------------- 1 | import time, os 2 | 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | 6 | # The quality of the conv2d_transpose reconstruction operation has been recently beaten by the subpixel operation 7 | # Taken from the subpixel paper: https://github.com/Tetrachrome/subpixel/blob/master/subpixel.py 8 | #### 9 | def _phase_shift(I, r): 10 | bsize, a, b, c = I.get_shape().as_list() 11 | bsize = tf.shape(I)[0] # I've made a minor change to the original implementation to enable Dimension(None) for the batch dim 12 | X = tf.reshape(I, (bsize, a, b, r, r)) 13 | X = tf.transpose(X, (0, 1, 2, 4, 3)) # bsize, a, b, 1, 1 14 | X = tf.split(X, a, 1) # a, [bsize, b, r, r] 15 | X = tf.concat([tf.squeeze(x) for x in X], 2) # bsize, b, a*r, r 16 | X = tf.split(X, b, 1) # b, [bsize, a*r, r] 17 | X = tf.concat([tf.squeeze(x) for x in X], 2) # bsize, a*r, b*r 18 | return tf.reshape(X, (bsize, a*r, b*r, 1)) 19 | 20 | 21 | def PS(X, r, color=False): 22 | if color: 23 | Xc = tf.split(X, 3, 3) 24 | X = tf.concat([_phase_shift(x, r) for x in Xc], 3) 25 | else: 26 | X = _phase_shift(X, r) 27 | return X 28 | #### 29 | 30 | dir = os.path.dirname(os.path.realpath(__file__)) 31 | mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 32 | 33 | # Convolutionnel model 34 | # We want to have approximatively the same reprensatitivity between neural layers 35 | # and the Softmax layer 36 | # Number of parameters: (3 * 3 * 1 * 200) + (14 * 14 * 200 * 10) = 393800 37 | # Dimensionality: R^784 -> R^39200 -> R_10, 39200 neurons but only 200 hundred features per spatial point 38 | 39 | # Placeholder 40 | x = tf.placeholder(tf.float32, shape=[None, 784]) 41 | y_true = tf.placeholder(tf.float32, shape=[None, 10]) 42 | 43 | sparsity_constraint = tf.placeholder(tf.float32) 44 | 45 | x_img = tf.reshape(x, [-1, 28, 28, 1]) 46 | with tf.variable_scope('NeuralLayer'): 47 | # We would like 200 feature map, remember that weights are shared inside each feature map 48 | # The only difference with FC layers, is that we look for a precise feature everywhere in the image 49 | W1 = tf.get_variable('W1', shape=[3, 3, 1, 200], initializer=tf.random_normal_initializer(stddev=1e-1)) 50 | b1 = tf.get_variable('b1', shape=[200], initializer=tf.constant_initializer(0.1)) 51 | 52 | z1 = tf.nn.conv2d(x_img, W1, strides=[1, 2, 2, 1], padding='SAME') + b1 53 | a = tf.nn.relu(z1) 54 | 55 | # We graph the average density of neurons activation 56 | average_density = tf.reduce_mean(tf.reduce_sum(tf.cast((a > 0), tf.float32), axis=[1, 2, 3])) 57 | tf.summary.scalar('AverageDensity', average_density) 58 | 59 | a_vec_size = 14 * 14 * 200 60 | 61 | with tf.variable_scope('SoftmaxLayer'): 62 | a_vec = tf.reshape(a, [-1, a_vec_size]) 63 | 64 | W_s = tf.get_variable('W_s', shape=[a_vec_size, 10], initializer=tf.random_normal_initializer(stddev=1e-1)) 65 | b_s = tf.get_variable('b_s', shape=[10], initializer=tf.constant_initializer(0.1)) 66 | 67 | out = tf.matmul(a_vec, W_s) + b_s 68 | y = tf.nn.softmax(out) 69 | 70 | # The conv2d_transpose decoder is not cool anymore 71 | # with tf.variable_scope('Decoder'): 72 | # W_decoder = tf.get_variable('W_decoder', shape=[3, 3, 1, 200], initializer=tf.random_normal_initializer(stddev=1e-1)) 73 | # b_decoder = tf.get_variable('b_decoder', shape=[1], initializer=tf.constant_initializer(0.1)) 74 | 75 | # # We force the decoder weights (9) of each feature (200) to stay in the 1-norm area 76 | # W_decoder = tf.clip_by_norm(W_decoder, 1, axes=[3]) 77 | 78 | # z_decoder = tf.nn.conv2d_transpose(a, W_decoder, output_shape=[tf.shape(x)[0], 28, 28, 1], strides=[1, 2, 2, 1], padding='SAME') + b1 79 | 80 | # The phaseshift decoder is a lot cooler! 81 | with tf.variable_scope('Decoder_phaseshift'): 82 | # a is a 14x14x200 feature map, we want to end up with a 28x28x1 image, so we need to scale down to 14x14x4 feature map 83 | # before applying the phase shift trick with a ratio of 2 84 | W_decoder_ps = tf.get_variable('W1', shape=[3, 3, 200, 4], initializer=tf.random_normal_initializer(stddev=1e-1)) 85 | b_decoder_ps = tf.get_variable('b1', shape=[4], initializer=tf.constant_initializer(0.1)) 86 | 87 | # We force the decoder weights (9) of each feature (200) to stay in the 1-norm area 88 | W_decoder_ps = tf.clip_by_norm(W_decoder_ps, 1, axes=[3]) 89 | 90 | z_ps = tf.nn.conv2d(a, W_decoder_ps, strides=[1, 1, 1, 1], padding='SAME') + b_decoder_ps 91 | z_decoder_ps = PS(z_ps, 2) 92 | 93 | with tf.variable_scope('Loss'): 94 | epsilon = 1e-7 # After some training, y can be 0 on some classes which lead to NaN 95 | loss_classifier = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y + epsilon), axis=[1])) 96 | # loss_ae = tf.reduce_mean(tf.reduce_sum(tf.square(x_img - z_decoder), axis=[1, 2, 3])) 97 | loss_ae_ps = tf.reduce_mean(tf.reduce_sum(tf.square(x_img - z_decoder_ps), axis=[1, 2, 3])) 98 | loss_sc = sparsity_constraint * tf.reduce_sum(a) 99 | loss = loss_classifier + loss_ae_ps + loss_sc 100 | 101 | # tf.summary.scalar('loss_ae', loss_ae) 102 | tf.summary.scalar('loss_ae_ps', loss_ae_ps) 103 | tf.summary.scalar('loss_classifier', loss_classifier) 104 | tf.summary.scalar('loss_sc', loss_sc) 105 | tf.summary.scalar('loss', loss) 106 | 107 | 108 | # We merge summaries before the accuracy summary to avoid 109 | # graphing the accuracy with training data 110 | summaries = tf.summary.merge_all() 111 | 112 | with tf.variable_scope('Accuracy'): 113 | correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_true, 1)) 114 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 115 | acc_summary = tf.summary.scalar('accuracy', accuracy) 116 | 117 | # Let's see the actual tensorflow reconstruction 118 | input_img_summary = tf.summary.image("input", x_img, 1) 119 | rec_img_summary_ps = tf.summary.image("reconstruction", z_decoder_ps, 1) 120 | 121 | 122 | # Training 123 | adam_train = tf.train.AdamOptimizer(learning_rate=1e-3) 124 | train_op = adam_train.minimize(loss) 125 | sess = None 126 | # We iterate over different sparsity constraint 127 | for sc in [0, 1e-5, 5e-5, 1e-4, 5e-4]: 128 | result_folder = dir + '/results/' + str(int(time.time())) + '-ae-sc' + str(sc) 129 | with tf.Session() as sess: 130 | sess.run(tf.global_variables_initializer()) 131 | sw = tf.summary.FileWriter(result_folder, sess.graph) 132 | 133 | for i in range(20000): 134 | batch = mnist.train.next_batch(100) 135 | current_loss, summary, _ = sess.run([loss, summaries, train_op], feed_dict={ 136 | x: batch[0], 137 | y_true: batch[1], 138 | sparsity_constraint: sc 139 | }) 140 | sw.add_summary(summary, i + 1) 141 | 142 | if (i + 1) % 100 == 0: 143 | acc, acc_sum, input_img_sum, rec_img_sum = sess.run([accuracy, acc_summary, input_img_summary, rec_img_summary_ps], feed_dict={ 144 | x: mnist.test.images, 145 | y_true: mnist.test.labels 146 | }) 147 | sw.add_summary(acc_sum, i + 1) 148 | sw.add_summary(input_img_sum, i + 1) 149 | sw.add_summary(rec_img_sum, i + 1) 150 | print('batch: %d, loss: %f, accuracy: %f' % (i + 1, current_loss, acc)) 151 | -------------------------------------------------------------------------------- /sparse-coding/cnn_sparsity.py: -------------------------------------------------------------------------------- 1 | import time, os 2 | 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | 6 | dir = os.path.dirname(os.path.realpath(__file__)) 7 | mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 8 | 9 | # Convolutionnel model 10 | # We want to have approximatively the same reprensatitivity between neural layers 11 | # and the Softmax layer 12 | # Number of parameters: (3 * 3 * 1 * 784) + (14 * 14 * 200 * 10) = 393800 13 | # Dimensionality: R^784 -> R^39200 -> R_10, 39200 neurons but only 200 hundred features per spatial point 14 | 15 | # Placeholder 16 | x = tf.placeholder(tf.float32, shape=[None, 784]) 17 | y_true = tf.placeholder(tf.float32, shape=[None, 10]) 18 | 19 | sparsity_constraint = tf.placeholder(tf.float32) 20 | 21 | x_img = tf.reshape(x, [-1, 28, 28, 1]) 22 | with tf.variable_scope('NeuralLayer'): 23 | # We would like 200 feature map, remember that weights are shared inside each feature map 24 | # The only difference with FC layers, is that we look for a precise feature everywhere in the image 25 | W1 = tf.get_variable('W1', shape=[3, 3, 1, 200], initializer=tf.random_normal_initializer(stddev=1e-1)) 26 | b1 = tf.get_variable('b1', shape=[200], initializer=tf.constant_initializer(0.1)) 27 | 28 | z1 = tf.nn.conv2d(x_img, W1, strides=[1, 2, 2, 1], padding='SAME') + b1 29 | a = tf.nn.relu(z1) 30 | 31 | # We graph the average density of neurons activation 32 | average_density = tf.reduce_mean(tf.reduce_sum(tf.cast((a > 0), tf.float32), axis=[1, 2, 3])) 33 | tf.summary.scalar('AverageDensity', average_density) 34 | 35 | a_vec_size = 14 * 14 * 200 36 | 37 | with tf.variable_scope('SoftmaxLayer'): 38 | a_vec = tf.reshape(a, [-1, a_vec_size]) 39 | 40 | W_s = tf.get_variable('W_s', shape=[a_vec_size, 10], initializer=tf.random_normal_initializer(stddev=1e-1)) 41 | b_s = tf.get_variable('b_s', shape=[10], initializer=tf.constant_initializer(0.1)) 42 | 43 | out = tf.matmul(a_vec, W_s) + b_s 44 | y = tf.nn.softmax(out) 45 | 46 | with tf.variable_scope('Loss'): 47 | epsilon = 1e-7 # After some training, y can be 0 on some classes which lead to NaN 48 | cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y + epsilon), axis=[1])) 49 | # We add our sparsity constraint on the activations 50 | # loss = cross_entropy + sparsity_constraint * (tf.reduce_sum(a1) + tf.reduce_sum(a2) + tf.reduce_sum(a)) 51 | loss = cross_entropy + sparsity_constraint * tf.reduce_sum(a) 52 | 53 | tf.summary.scalar('loss', loss) # Graph the loss 54 | 55 | # We merge summaries before the accuracy summary to avoid 56 | # graphing the accuracy with training data 57 | summaries = tf.summary.merge_all() 58 | 59 | with tf.variable_scope('Accuracy'): 60 | correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_true, 1)) 61 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 62 | acc_summary = tf.summary.scalar('accuracy', accuracy) 63 | 64 | 65 | # Training 66 | adam_train = tf.train.AdamOptimizer(learning_rate=1e-3) 67 | train_op = adam_train.minimize(loss) 68 | sess = None 69 | # We iterate over different sparsity constraint 70 | for sc in [0, 1e-5, 5e-5, 1e-4, 5e-4]: 71 | result_folder = dir + '/results/' + str(int(time.time())) + '-cnn-sc' + str(sc) 72 | with tf.Session() as sess: 73 | sess.run(tf.global_variables_initializer()) 74 | sw = tf.summary.FileWriter(result_folder, sess.graph) 75 | 76 | for i in range(20000): 77 | batch = mnist.train.next_batch(100) 78 | current_loss, summary, _ = sess.run([loss, summaries, train_op], feed_dict={ 79 | x: batch[0], 80 | y_true: batch[1], 81 | sparsity_constraint: sc 82 | }) 83 | sw.add_summary(summary, i + 1) 84 | 85 | if (i + 1) % 100 == 0: 86 | acc, acc_sum = sess.run([accuracy, acc_summary], feed_dict={ 87 | x: mnist.test.images, 88 | y_true: mnist.test.labels 89 | }) 90 | sw.add_summary(acc_sum, i + 1) 91 | print('batch: %d, loss: %f, accuracy: %f' % (i + 1, current_loss, acc)) -------------------------------------------------------------------------------- /sparse-coding/results-ae.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/sparse-coding/results-ae.jpg -------------------------------------------------------------------------------- /sparse-coding/results-cnn.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/sparse-coding/results-cnn.jpg -------------------------------------------------------------------------------- /sparse-coding/results-fc.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/sparse-coding/results-fc.jpg -------------------------------------------------------------------------------- /sparse-coding/sparse-coding-equation.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/sparse-coding/sparse-coding-equation.jpg -------------------------------------------------------------------------------- /sparse-coding/sparsecoding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/sparse-coding/sparsecoding.png -------------------------------------------------------------------------------- /sparse-coding/sparsity.py: -------------------------------------------------------------------------------- 1 | import time, os 2 | 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | 6 | dir = os.path.dirname(os.path.realpath(__file__)) 7 | mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 8 | 9 | # Fully connected model 10 | # Number of parameters: (784 * 784 + 784) + (784 * 10 + 10) = 615440 + 7850 = 623290 11 | # Dimensionality: R^784 -> R^784 -> R^10 12 | 13 | # Placeholder 14 | x = tf.placeholder(tf.float32, shape=[None, 784]) 15 | y_true = tf.placeholder(tf.float32, shape=[None, 10]) 16 | 17 | sparsity_constraint = tf.placeholder(tf.float32) 18 | 19 | # Variables 20 | with tf.variable_scope('NeuralLayer'): 21 | W = tf.get_variable('W', shape=[784, 784], initializer=tf.random_normal_initializer(stddev=1e-1)) 22 | b = tf.get_variable('b', shape=[784], initializer=tf.constant_initializer(0.1)) 23 | 24 | z = tf.matmul(x, W) + b 25 | a = tf.nn.relu(z) 26 | 27 | # We graph the average density of neurons activation 28 | average_density = tf.reduce_mean(tf.reduce_sum(tf.cast((a > 0), tf.float32), axis=[1])) 29 | tf.summary.scalar('AverageDensity', average_density) 30 | 31 | with tf.variable_scope('SoftmaxLayer'): 32 | W_s = tf.get_variable('W_s', shape=[784, 10], initializer=tf.random_normal_initializer(stddev=1e-1)) 33 | b_s = tf.get_variable('b_s', shape=[10], initializer=tf.constant_initializer(0.1)) 34 | 35 | out = tf.matmul(a, W_s) + b_s 36 | y = tf.nn.softmax(out) 37 | 38 | with tf.variable_scope('Loss'): 39 | epsilon = 1e-7 # After some training, y can be 0 on some classes which lead to NaN 40 | cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y + epsilon), axis=[1])) 41 | # We add our sparsity constraint on the activations 42 | loss = cross_entropy + sparsity_constraint * tf.reduce_sum(a) 43 | 44 | tf.summary.scalar('loss', loss) # Graph the loss 45 | 46 | # We merge summaries before the accuracy summary to avoid 47 | # graphing the accuracy with training data 48 | summaries = tf.summary.merge_all() 49 | 50 | with tf.variable_scope('Accuracy'): 51 | correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_true, 1)) 52 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 53 | acc_summary = tf.summary.scalar('accuracy', accuracy) 54 | 55 | # Training 56 | adam = tf.train.AdamOptimizer(learning_rate=1e-3) 57 | train_op = adam.minimize(loss) 58 | sess = None 59 | # We iterate over different sparsity constraint 60 | for sc in [0, 1e-4, 5e-4, 1e-3, 2.7e-3]: 61 | result_folder = dir + '/results/' + str(int(time.time())) + '-fc-sc' + str(sc) 62 | with tf.Session() as sess: 63 | sess.run(tf.global_variables_initializer()) 64 | sw = tf.summary.FileWriter(result_folder, sess.graph) 65 | 66 | for i in range(20000): 67 | batch = mnist.train.next_batch(100) 68 | current_loss, summary, _ = sess.run([loss, summaries, train_op], feed_dict={ 69 | x: batch[0], 70 | y_true: batch[1], 71 | sparsity_constraint: sc 72 | }) 73 | sw.add_summary(summary, i + 1) 74 | 75 | if (i + 1) % 100 == 0: 76 | acc, acc_sum = sess.run([accuracy, acc_summary], feed_dict={ 77 | x: mnist.test.images, 78 | y_true: mnist.test.labels 79 | }) 80 | sw.add_summary(acc_sum, i + 1) 81 | print('batch: %d, loss: %f, accuracy: %f' % (i + 1, current_loss, acc)) -------------------------------------------------------------------------------- /tf-architecture/.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-architecture/.gitignore -------------------------------------------------------------------------------- /tf-architecture/README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow: A proposal of good practices for files, folders and models architecture 2 | 3 | ## TL, DR! 4 | ### EN 5 | - I list all the different tasks one will have to do when doing ML 6 | - I show a common folder structure that I believe handles all possible use cases nicely 7 | - I show a basic Model class, easily extendable that structure a lot of possible kinds of models 8 | - I describe how to build a good "shell API" for easy iterations. 9 | Bonus: Some TF code linked to the subject. 10 | 11 | ### FR 12 | - Je liste l'ensemble des tâches à effectuer quand on fait du machine learning 13 | - Je présente une architecture de dossier qui, je pense, permet de gérer l'ensemble de ces usages. 14 | - Je présente une classe Modèle de base, facilement "extensible", qui structure la manière de concevoir de nombreux types de modeles. 15 | - Enfin, je décris comment construire une bonne "API shell" pour faciliter les itérations de recherches. 16 | Bonus: Du code TF divers et varié associé au sujet. 17 | -------------------------------------------------------------------------------- /tf-architecture/archi/README: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-architecture/archi/README -------------------------------------------------------------------------------- /tf-architecture/archi/hpsearch/hyperband.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-architecture/archi/hpsearch/hyperband.py -------------------------------------------------------------------------------- /tf-architecture/archi/hpsearch/randomsearch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-architecture/archi/hpsearch/randomsearch.py -------------------------------------------------------------------------------- /tf-architecture/archi/main.py: -------------------------------------------------------------------------------- 1 | import os, json 2 | import tensorflow as tf 3 | 4 | # See the __init__ script in the models folder 5 | # `make_models` is a helper function to load any models you have 6 | from models import make_models 7 | from hpsearch import hyperband, randomsearch 8 | 9 | # I personally always like to make my paths absolute 10 | # to be independent from where the python binary is called 11 | dir = os.path.dirname(os.path.realpath(__file__)) 12 | 13 | # I won't dig into TF interaction with the shell, feel free to explore the documentation 14 | flags = tf.app.flags 15 | 16 | 17 | # Hyper-parameters search configuration 18 | flags.DEFINE_boolean('fullsearch', False, 'Perform a full search of hyperparameter space ex:(hyperband -> lr search -> hyperband with best lr)') 19 | flags.DEFINE_boolean('dry_run', False, 'Perform a dry_run (testing purpose)') 20 | flags.DEFINE_integer('nb_process', 4, 'Number of parallel process to perform a HP search') 21 | 22 | # fixed_params is a trick I use to be able to fix some parameters inside the model random function 23 | # For example, one might want to explore different models fixing the learning rate, see the basic_model get_random_config function 24 | flags.DEFINE_string('fixed_params', "{}", 'JSON inputs to fix some params in a HP search, ex: \'{"lr": 0.001}\'') 25 | 26 | 27 | # Agent configuration 28 | flags.DEFINE_string('model_name', 'DQNAgent', 'Unique name of the model') 29 | flags.DEFINE_boolean('best', False, 'Force to use the best known configuration') 30 | flags.DEFINE_float('initial_mean', 0., 'Initial mean for NN') 31 | flags.DEFINE_float('initial_stddev', 1e-2, 'Initial standard deviation for NN') 32 | flags.DEFINE_float('lr', 1e-3, 'The learning rate of SGD') 33 | flags.DEFINE_float('nb_units', 20, 'Number of hidden units in Deep learning agents') 34 | 35 | 36 | # Environment configuration 37 | flags.DEFINE_boolean('debug', False, 'Debug mode') 38 | flags.DEFINE_integer('max_iter', 2000, 'Number of training step') 39 | flags.DEFINE_boolean('infer', False, 'Load an agent for playing') 40 | 41 | # This is very important for TensorBoard 42 | # each model will end up in its own unique folder using time module 43 | # Obviously one can also choose to name the output folder 44 | flags.DEFINE_string('result_dir', dir + '/results/' + flags.FLAGS.model_name + '/' + str(int(time.time())), 'Name of the directory to store/log the model (if it exists, the model will be loaded from it)') 45 | 46 | # Another important point, you must provide an access to the random seed 47 | # to be able to fully reproduce an experiment 48 | flags.DEFINE_integer('random_seed', random.randint(0, sys.maxsize), 'Value of random seed') 49 | 50 | def main(_): 51 | config = flags.FLAGS.__flags.copy() 52 | # fixed_params must be a string to be passed in the shell, let's use JSON 53 | config["fixed_params"] = json.loads(config["fixed_params"]) 54 | 55 | if config['fullsearch']: 56 | # Some code for HP search ... 57 | else: 58 | model = make_model(config) 59 | 60 | if config['infer']: 61 | # Some code for inference ... 62 | else: 63 | # Some code for training ... 64 | 65 | 66 | if __name__ == '__main__': 67 | tf.app.run() -------------------------------------------------------------------------------- /tf-architecture/archi/models/__init__.py: -------------------------------------------------------------------------------- 1 | from models.basic_model import BasicModel 2 | from agents.other_model import SomeOtherModel 3 | 4 | __all__ = [ 5 | "BasicModel", 6 | "SomeOtherModel" 7 | ] 8 | 9 | def make_model(config, env): 10 | if config['model_name'] in __all__: 11 | return globals()[config['model_name']](config, env) 12 | else: 13 | raise Exception('The model name %s does not exist' % config['model_name']) 14 | 15 | def get_model_class(config): 16 | if config['model_name'] in __all__: 17 | return globals()[config['model_name']] 18 | else: 19 | raise Exception('The model name %s does not exist' % config['model_name']) -------------------------------------------------------------------------------- /tf-architecture/archi/models/basic_model.py: -------------------------------------------------------------------------------- 1 | import os, copy 2 | import tensorflow as tf 3 | 4 | class BasicAgent(object): 5 | # To build your model, you only to pass a "configuration" which is a dictionary 6 | def __init__(self, config): 7 | # I like to keep the best HP found so far inside the model itself 8 | # This is a mechanism to load the best HP and override the configuration 9 | if config['best']: 10 | config.update(self.get_best_config(config['env_name'])) 11 | 12 | # I make a `deepcopy` of the configuration before using it 13 | # to avoid any potential mutation when I iterate asynchronously over configurations 14 | self.config = copy.deepcopy(config) 15 | 16 | if config['debug']: # This is a personal check i like to do 17 | print('config', self.config) 18 | 19 | # When working with NN, one usually initialize randomly 20 | # and you want to be able to reproduce your initialization so make sure 21 | # you store the random seed and actually use it in your TF graph (tf.set_random_seed() for example) 22 | self.random_seed = self.config['random_seed'] 23 | 24 | # All models share some basics hyper parameters, this is the section where we 25 | # copy them into the model 26 | self.result_dir = self.config['result_dir'] 27 | self.max_iter = self.config['max_iter'] 28 | self.lr = self.config['lr'] 29 | self.nb_units = self.config['nb_units'] 30 | # etc. 31 | 32 | # Now the child Model needs some custom parameters, to avoid any 33 | # inheritance hell with the __init__ function, the model 34 | # will override this function completely 35 | self.set_agent_props() 36 | 37 | # Again, child Model should provide its own build_grap function 38 | self.graph = self.build_graph(tf.Graph()) 39 | 40 | # Any operations that should be in the graph but are common to all models 41 | # can be added this way, here 42 | with self.graph.as_default(): 43 | self.saver = tf.train.Saver( 44 | max_to_keep=50, 45 | ) 46 | 47 | # Add all the other common code for the initialization here 48 | gpu_options = tf.GPUOptions(allow_growth=True) 49 | sessConfig = tf.ConfigProto(gpu_options=gpu_options) 50 | self.sess = tf.Session(config=sessConfig, graph=self.graph) 51 | self.sw = tf.summary.FileWriter(self.result_dir, self.sess.graph) 52 | 53 | # This function is not always common to all models, that's why it's again 54 | # separated from the __init__ one 55 | self.init() 56 | 57 | # At the end of this function, you want your model to be ready! 58 | 59 | def set_agent_props(self): 60 | # This function is here to be overriden completely. 61 | # When you look at your model, you want to know exactly which custom options it needs. 62 | pass 63 | 64 | def get_best_config(self): 65 | # This function is here to be overriden completely. 66 | # It returns a dictionary used to update the initial configuration (see __init__) 67 | return {} 68 | 69 | @staticmethod 70 | def get_random_config(fixed_params={}): 71 | # Why static? Because you want to be able to pass this function to other processes 72 | # so they can independently generate random configuration of the current model 73 | raise Exception('The get_random_config function must be overriden by the agent') 74 | 75 | def build_graph(self, graph): 76 | raise Exception('The build_graph function must be overriden by the agent') 77 | 78 | def infer(self): 79 | raise Exception('The infer function must be overriden by the agent') 80 | 81 | def learn_from_epoch(self): 82 | # I like to separate the function to train per epoch and the function to train globally 83 | raise Exception('The learn_from_epoch function must be overriden by the agent') 84 | 85 | def train(self, save_every=1): 86 | # This function is usually common to all your models, Here is an example: 87 | for epoch_id in range(0, self.max_iter): 88 | self.learn_from_epoch() 89 | 90 | # If you don't want to save during training, you can just pass a negative number 91 | if save_every > 0 and epoch_id % save_every == 0: 92 | self.save() 93 | 94 | def save(self): 95 | # This function is usually common to all your models, Here is an example: 96 | global_step_t = tf.train.get_global_step(self.graph) 97 | global_step, episode_id = self.sess.run([global_step_t, self.episode_id]) 98 | if self.config['debug']: 99 | print('Saving to %s with global_step %d' % (self.result_dir, global_step)) 100 | self.saver.save(self.sess, self.result_dir + '/agent-ep_' + str(episode_id), global_step) 101 | 102 | # I always keep the configuration that 103 | if not os.path.isfile(self.result_dir + '/config.json'): 104 | config = self.config 105 | if 'phi' in config: 106 | del config['phi'] 107 | with open(self.result_dir + '/config.json', 'w') as f: 108 | json.dump(self.config, f) 109 | 110 | 111 | def init(self): 112 | # This function is usually common to all your models 113 | # but making separate than the __init__ function allows it to be overidden cleanly 114 | # this is an example of such a function 115 | checkpoint = tf.train.get_checkpoint_state(self.result_dir) 116 | if checkpoint is None: 117 | self.sess.run(self.init_op) 118 | else: 119 | 120 | if self.config['debug']: 121 | print('Loading the model from folder: %s' % self.result_dir) 122 | self.saver.restore(self.sess, checkpoint.model_checkpoint_path) 123 | 124 | def infer(self): 125 | # This function is usually common to all your models 126 | pass 127 | -------------------------------------------------------------------------------- /tf-freeze/freeze.py: -------------------------------------------------------------------------------- 1 | import os, argparse 2 | 3 | import tensorflow as tf 4 | from tensorflow.python.framework import graph_util 5 | 6 | dir = os.path.dirname(os.path.realpath(__file__)) 7 | 8 | def freeze_graph(model_folder): 9 | # We retrieve our checkpoint fullpath 10 | checkpoint = tf.train.get_checkpoint_state(model_folder) 11 | input_checkpoint = checkpoint.model_checkpoint_path 12 | 13 | # We precise the file fullname of our freezed graph 14 | absolute_model_folder = "/".join(input_checkpoint.split('/')[:-1]) 15 | output_graph = absolute_model_folder + "/frozen_model.pb" 16 | 17 | # Before exporting our graph, we need to precise what is our output node 18 | # NOTE: this variables is plural, because you can have multiple output nodes 19 | output_node_names = "Accuracy/predictions" 20 | 21 | # We clear the devices, to allow TensorFlow to control on the loading where it wants operations to be calculated 22 | clear_devices = True 23 | 24 | # We import the meta graph and retrive a Saver 25 | saver = tf.train.import_meta_graph(input_checkpoint + '.meta', clear_devices=clear_devices) 26 | 27 | # We retrieve the protobuf graph definition 28 | graph = tf.get_default_graph() 29 | input_graph_def = graph.as_graph_def() 30 | 31 | # We start a session and restore the graph weights 32 | with tf.Session() as sess: 33 | saver.restore(sess, input_checkpoint) 34 | 35 | # We use a built-in TF helper to export variables to constant 36 | output_graph_def = graph_util.convert_variables_to_constants( 37 | sess, 38 | input_graph_def, 39 | output_node_names.split(",") # We split on comma for convenience 40 | ) 41 | 42 | # Finally we serialize and dump the output graph to the filesystem 43 | with tf.gfile.GFile(output_graph, "wb") as f: 44 | f.write(output_graph_def.SerializeToString()) 45 | print("%d ops in the final graph." % len(output_graph_def.node)) 46 | 47 | 48 | if __name__ == '__main__': 49 | parser = argparse.ArgumentParser() 50 | parser.add_argument("--model_folder", type=str, help="Model folder to export") 51 | args = parser.parse_args() 52 | 53 | freeze_graph(args.model_folder) -------------------------------------------------------------------------------- /tf-freeze/img/Screen Shot 2016-11-21 at 17.04.36.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-freeze/img/Screen Shot 2016-11-21 at 17.04.36.png -------------------------------------------------------------------------------- /tf-freeze/img/Screen Shot 2016-11-21 at 17.11.41.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-freeze/img/Screen Shot 2016-11-21 at 17.11.41.png -------------------------------------------------------------------------------- /tf-freeze/load.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | import tensorflow as tf 4 | 5 | def load_graph(frozen_graph_filename): 6 | # We parse the graph_def file 7 | with tf.gfile.GFile(frozen_graph_filename, "rb") as f: 8 | graph_def = tf.GraphDef() 9 | graph_def.ParseFromString(f.read()) 10 | 11 | # We load the graph_def in the default graph 12 | with tf.Graph().as_default() as graph: 13 | tf.import_graph_def( 14 | graph_def, 15 | input_map=None, 16 | return_elements=None, 17 | name="prefix", 18 | op_dict=None, 19 | producer_op_list=None 20 | ) 21 | return graph 22 | 23 | if __name__ == '__main__': 24 | parser = argparse.ArgumentParser() 25 | parser.add_argument("--frozen_model_filename", default="results/frozen_model.pb", type=str, help="Frozen model file to import") 26 | args = parser.parse_args() 27 | 28 | graph = load_graph(args.frozen_model_filename) 29 | 30 | # We can list operations 31 | for op in graph.get_operations(): 32 | print(op.name) 33 | # prefix/Placeholder/inputs_placeholder 34 | # ... 35 | # prefix/Accuracy/predictions 36 | x = graph.get_tensor_by_name('prefix/Placeholder/inputs_placeholder:0') 37 | y = graph.get_tensor_by_name('prefix/Accuracy/predictions:0') 38 | 39 | with tf.Session(graph=graph) as sess: 40 | y_out = sess.run(y, feed_dict={ 41 | x: [[3, 5, 7, 4, 5, 1, 1, 1, 1, 1]] # < 45 42 | }) 43 | print(y_out) # [[ False ]] Yay! -------------------------------------------------------------------------------- /tf-freeze/server.py: -------------------------------------------------------------------------------- 1 | import json, argparse, time 2 | 3 | import tensorflow as tf 4 | from load import load_graph 5 | 6 | from flask import Flask, request 7 | from flask_cors import CORS 8 | 9 | ################################################## 10 | # API part 11 | ################################################## 12 | app = Flask(__name__) 13 | cors = CORS(app) 14 | @app.route("/api/predict", methods=['POST']) 15 | def predict(): 16 | start = time.time() 17 | 18 | data = request.data.decode("utf-8") 19 | if data == "": 20 | params = request.form 21 | x_in = json.loads(params['x']) 22 | else: 23 | params = json.loads(data) 24 | x_in = params['x'] 25 | 26 | ################################################## 27 | # Tensorflow part 28 | ################################################## 29 | y_out = persistent_sess.run(y, feed_dict={ 30 | x: x_in 31 | # x: [[3, 5, 7, 4, 5, 1, 1, 1, 1, 1]] # < 45 32 | }) 33 | ################################################## 34 | # END Tensorflow part 35 | ################################################## 36 | 37 | json_data = json.dumps({'y': y_out.tolist()}) 38 | print("Time spent handling the request: %f" % (time.time() - start)) 39 | 40 | return json_data 41 | ################################################## 42 | # END API part 43 | ################################################## 44 | 45 | if __name__ == "__main__": 46 | parser = argparse.ArgumentParser() 47 | parser.add_argument("--frozen_model_filename", default="results/frozen_model.pb", type=str, help="Frozen model file to import") 48 | parser.add_argument("--gpu_memory", default=.2, type=float, help="GPU memory per process") 49 | args = parser.parse_args() 50 | 51 | ################################################## 52 | # Tensorflow part 53 | ################################################## 54 | print('Loading the model') 55 | graph = load_graph(args.frozen_model_filename) 56 | x = graph.get_tensor_by_name('prefix/Placeholder/inputs_placeholder:0') 57 | y = graph.get_tensor_by_name('prefix/Accuracy/predictions:0') 58 | 59 | print('Starting Session, setting the GPU memory usage to %f' % args.gpu_memory) 60 | gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=args.gpu_memory) 61 | sess_config = tf.ConfigProto(gpu_options=gpu_options) 62 | persistent_sess = tf.Session(graph=graph, config=sess_config) 63 | ################################################## 64 | # END Tensorflow part 65 | ################################################## 66 | 67 | print('Starting the API') 68 | app.run() 69 | -------------------------------------------------------------------------------- /tf-freeze/tiny_model.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | 4 | 5 | with tf.variable_scope('Placeholder'): 6 | inputs_placeholder = tf.placeholder(tf.float32, name='inputs_placeholder', shape=[None, 10]) 7 | labels_placeholder = tf.placeholder(tf.float32, name='labels_placeholder', shape=[None, 1]) 8 | 9 | with tf.variable_scope('NN'): 10 | W1 = tf.get_variable('W1', shape=[10, 1], initializer=tf.random_normal_initializer(stddev=1e-1)) 11 | b1 = tf.get_variable('b1', shape=[1], initializer=tf.constant_initializer(0.1)) 12 | W2 = tf.get_variable('W2', shape=[10, 1], initializer=tf.random_normal_initializer(stddev=1e-1)) 13 | b2 = tf.get_variable('b2', shape=[1], initializer=tf.constant_initializer(0.1)) 14 | 15 | a = tf.nn.relu(tf.matmul(inputs_placeholder, W1) + b1) 16 | a2 = tf.nn.relu(tf.matmul(inputs_placeholder, W2) + b2) 17 | 18 | y = tf.divide(tf.add(a, a2), 2) 19 | 20 | with tf.variable_scope('Loss'): 21 | loss = tf.reduce_sum(tf.square(y - labels_placeholder) / 2) 22 | 23 | with tf.variable_scope('Accuracy'): 24 | predictions = tf.greater(y, 0.5, name="predictions") 25 | correct_predictions = tf.equal(predictions, tf.cast(labels_placeholder, tf.bool), name="correct_predictions") 26 | accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32)) 27 | 28 | 29 | adam = tf.train.AdamOptimizer(learning_rate=1e-3) 30 | train_op = adam.minimize(loss) 31 | 32 | # generate_data 33 | inputs = np.random.choice(10, size=[10000, 10]) 34 | labels = (np.sum(inputs, axis=1) > 45).reshape(-1, 1).astype(np.float32) 35 | print('inputs.shape:', inputs.shape) 36 | print('labels.shape:', labels.shape) 37 | 38 | 39 | test_inputs = np.random.choice(10, size=[100, 10]) 40 | test_labels = (np.sum(test_inputs, axis=1) > 45).reshape(-1, 1).astype(np.float32) 41 | print('test_inputs.shape:', test_inputs.shape) 42 | print('test_labels.shape:', test_labels.shape) 43 | 44 | batch_size = 32 45 | epochs = 10 46 | 47 | batches = [] 48 | print("%d items in batch of %d gives us %d full batches and %d batches of %d items" % ( 49 | len(inputs), 50 | batch_size, 51 | len(inputs) // batch_size, 52 | batch_size - len(inputs) // batch_size, 53 | len(inputs) - (len(inputs) // batch_size) * 32) 54 | ) 55 | for i in range(len(inputs) // batch_size): 56 | batch = [ inputs[batch_size*i:batch_size*i+batch_size], labels[batch_size*i:batch_size*i+batch_size] ] 57 | batches.append(list(batch)) 58 | if (i + 1) * batch_size < len(inputs): 59 | batch = [ inputs[batch_size*(i + 1):],labels[batch_size*(i + 1):] ] 60 | batches.append(list(batch)) 61 | print("Number of batches: %d" % len(batches)) 62 | print("Size of full batch: %d" % len(batches[0])) 63 | print("Size if final batch: %d" % len(batches[-1])) 64 | 65 | global_count = 0 66 | with tf.Session() as sess: 67 | sess.run(tf.global_variables_initializer()) 68 | for i in range(epochs): 69 | for batch in batches: 70 | # print(batch[0].shape, batch[1].shape) 71 | train_loss , _= sess.run([loss, train_op], feed_dict={ 72 | inputs_placeholder: batch[0], 73 | labels_placeholder: batch[1] 74 | }) 75 | # print('train_loss: %d' % train_loss) 76 | 77 | if global_count % 100 == 0: 78 | acc = sess.run(accuracy, feed_dict={ 79 | inputs_placeholder: test_inputs, 80 | labels_placeholder: test_labels 81 | }) 82 | print('accuracy: %f' % acc) 83 | global_count += 1 84 | 85 | acc = sess.run(accuracy, feed_dict={ 86 | inputs_placeholder: test_inputs, 87 | labels_placeholder: test_labels 88 | }) 89 | print("final accuracy: %f" % acc) 90 | 91 | saver = tf.train.Saver() 92 | last_chkp = saver.save(sess, 'results/graph.chkp') 93 | 94 | for op in tf.get_default_graph().get_operations(): 95 | print(op.name) 96 | -------------------------------------------------------------------------------- /tf-mut-control/README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow: Mutating variables and control flow 2 | 3 | ## TL, DR! 4 | ### EN 5 | - I explore the different ways to manually mutate Variables in TF (content and shape) 6 | - I explore how to construct control flow like an "if statement" 7 | - I end up and showing a weird TF behaviour when you mix those 8 | - Bonus: a first try at animated GIF :D 9 | 10 | ### FR 11 | - J'explore les différents moyens de modifier une Variable manuellement (contenu et "forme") 12 | - J'explore comment construire des flux de données (condition if/else) 13 | - Je finis sur un cas étrange de TF quand on utilise les deux sujets précédents en même temps 14 | - En bonus: une première tentative de GIF animé :D 15 | -------------------------------------------------------------------------------- /tf-mut-control/anim1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/anim1.gif -------------------------------------------------------------------------------- /tf-mut-control/anim1.m4v: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/anim1.m4v -------------------------------------------------------------------------------- /tf-mut-control/anim2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/anim2.gif -------------------------------------------------------------------------------- /tf-mut-control/anim2.m4v: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/anim2.m4v -------------------------------------------------------------------------------- /tf-mut-control/animation.key: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/animation.key -------------------------------------------------------------------------------- /tf-mut-control/animation2.key: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-mut-control/animation2.key -------------------------------------------------------------------------------- /tf-mut-control/dyn_array.py: -------------------------------------------------------------------------------- 1 | import os 2 | import tensorflow as tf 3 | 4 | dir = os.path.dirname(os.path.realpath(__file__)) 5 | 6 | embed = tf.get_variable( 7 | "embed", 8 | shape=[0, 1], 9 | dtype=tf.float32, 10 | validate_shape=False # This shape will evolve, so we need to remove any TensorFlow optim here 11 | ) 12 | word_dict = tf.Variable( 13 | initial_value=[], 14 | name='word_dict', 15 | dtype=tf.string, 16 | validate_shape=False, 17 | trainable=False 18 | ) 19 | textfile = tf.placeholder(tf.string) 20 | 21 | # Update word dict 22 | splitted_textfile = tf.string_split(textfile, " ") 23 | tmp_word_dict = tf.concat([word_dict, splitted_textfile.values], 0) 24 | tmp_word_dict, word_idx, word_count = tf.unique_with_counts(tmp_word_dict) 25 | assign_word_dict = tf.assign(word_dict, tmp_word_dict, validate_shape=False) 26 | with tf.control_dependencies([assign_word_dict]): 27 | word_dict_value = word_dict.read_value() 28 | missing_nb_dim = tf.shape(word_dict_value)[0] - tf.shape(embed)[0] 29 | missing_nb_dim = tf.Print(missing_nb_dim, data=[missing_nb_dim, word_dict_value], message="missing_nb_dim, word_dict:", summarize=10) 30 | 31 | # Update embed 32 | def update_embed_func(): 33 | new_columns = tf.random_normal([missing_nb_dim, 1], mean=-1, stddev=4) 34 | new_embed = tf.concat([embed, new_columns], 0) 35 | assign_op = tf.assign(embed, new_embed, validate_shape=False) 36 | return assign_op 37 | 38 | should_update_embed = tf.less(0, missing_nb_dim) 39 | assign_to_embed = tf.cond(should_update_embed, update_embed_func, lambda: embed) 40 | with tf.control_dependencies([assign_to_embed]): 41 | # outputs = tf.identity(outputs) 42 | embed_value = embed.read_value() 43 | word_embed = tf.embedding_lookup_sparse(embed_value, word_idx) 44 | 45 | persistent_sess = tf.Session() 46 | persistent_sess.run(tf.global_variables_initializer()) 47 | 48 | persistent_sess.run(assign_to_embed, feed_dict={ 49 | textfile: ["that is cool! that is awesome!"] 50 | }) 51 | print(persistent_sess.run(tf.trainable_variables()[0])) 52 | persistent_sess.run(assign_to_embed, feed_dict={ 53 | textfile: ["this is cool! that was crazy"] 54 | }) 55 | print(persistent_sess.run(tf.trainable_variables()[0])) 56 | 57 | 58 | 59 | tf.summary.FileWriter(dir, persistent_sess.graph).flush() 60 | 61 | # More discussion on https://github.com/tensorflow/tensorflow/issues/7782 -------------------------------------------------------------------------------- /tf-queues/README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow: How to optimise your input pipeline with queues and multi-threading 2 | 3 | ## TL, DR! 4 | ### EN 5 | - I implement a first basic neural net using the "feed_dict" system 6 | - I show step by step how to use queues in TF 7 | - I update the first example with a queue implementation and show that i train 33% faster the neural net 8 | - bonus: Some more code on queues on my Github in the references section 9 | 10 | ### FR 11 | - J'implémente un premier réseau de neurones basique utilisant le système "feed_dict" 12 | - Je montre étape par étape comment utiliser les queues dans TF 13 | - Je mets à jour le code du premier exemple et montre que j'entraine mon réseau de neurones 33% plus vite 14 | - En bonus: du code en plus sur mon Github dans la section Références de l'article -------------------------------------------------------------------------------- /tf-queues/enqueue.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | q = tf.FIFOQueue(capacity=3, dtypes=tf.float32) 4 | 5 | x_input_data = tf.random_normal([3], mean=-1, stddev=4) 6 | enqueue_many_op = q.enqueue_many(x_input_data) # <- x1 - x2 -x3 | 7 | enqueue_op = q.enqueue(x_input_data) # <- [x1, x2, x3] - void - void | 8 | 9 | x_input_data2 = tf.random_normal([3, 1], mean=-1, stddev=4) 10 | enqueue_many_op2 = q.enqueue_many(x_input_data2) # <- [x1] - [x2] - [x3] | 11 | enqueue_op2 = q.enqueue(x_input_data2) # <- [ [x1], [x2], [x3] ] - void - void | 12 | 13 | dequeue_op = q.dequeue() 14 | 15 | with tf.Session() as sess: 16 | sess.run(enqueue_many_op) 17 | print(sess.run(dequeue_op), sess.run(dequeue_op), sess.run(dequeue_op)) 18 | 19 | sess.run(enqueue_op) 20 | print(sess.run(dequeue_op)) 21 | 22 | sess.run(enqueue_many_op2) 23 | print(sess.run(dequeue_op), sess.run(dequeue_op), sess.run(dequeue_op)) 24 | 25 | sess.run(enqueue_op2) 26 | print(sess.run(dequeue_op)) -------------------------------------------------------------------------------- /tf-queues/lorem_ipsum1.txt: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec ac ex ullamcorper, rutrum neque in, tristique eros. Nulla ac molestie diam, ac mattis neque. Sed ut lectus non odio imperdiet efficitur ac eget elit. Sed a mi non tellus condimentum semper. Sed dolor mi, gravida non orci mollis, tempor elementum metus. Nullam non ex sed sapien placerat fermentum sit amet ac odio. Sed mollis, odio ullamcorper varius condimentum, dui nisi porta ante, nec rhoncus orci turpis a magna. Aenean nec velit id odio faucibus ornare ac a mi. Morbi pharetra auctor nibh, in hendrerit felis viverra quis. Praesent pellentesque neque hendrerit felis venenatis, quis commodo lorem fermentum. Aenean mattis quam eu ante condimentum ultricies. -------------------------------------------------------------------------------- /tf-queues/lorem_ipsum2.txt: -------------------------------------------------------------------------------- 1 | Pellentesque tempus est nec tristique porttitor. Ut vitae metus rutrum, dictum lectus eu, fringilla erat. Vivamus dignissim lobortis urna in rutrum. Duis lobortis ligula quis aliquet sagittis. Fusce ac faucibus tellus, in varius nibh. Nunc mi turpis, pellentesque vel ligula id, condimentum fermentum sapien. Pellentesque dapibus quam a ipsum tincidunt, quis sodales nunc ultrices. Duis fermentum luctus tortor eget fermentum. Vivamus iaculis magna nisi, ac vulputate elit interdum vitae. Praesent id mauris pharetra, mollis diam et, lacinia nisl. Vestibulum feugiat dignissim tortor eget luctus. In ante turpis, egestas vitae aliquet id, posuere nec sem. Sed arcu arcu, porta scelerisque tempus euismod, condimentum eget erat. -------------------------------------------------------------------------------- /tf-queues/ptb_producer.py: -------------------------------------------------------------------------------- 1 | import os, time 2 | import tensorflow as tf 3 | 4 | # We are fucked, the models folder has been kicked out of tensorflow 1.0 deps 5 | # I can't import this handy reader anymore 6 | # (For anyone interested in the source code you can find it here: https://github.com/tensorflow/models/blob/master/tutorials/rnn/ptb/reader.py) 7 | # from tensorflow.models.rnn.ptb import reader 8 | 9 | dir = os.path.dirname(os.path.realpath(__file__)) 10 | 11 | # ain't cool, but hey! Let's rebuild one and even better 12 | # Let's make it to create an online RNN which adapt automatically to unseen words 13 | # crazy shit you say ? Not so much my friend 14 | 15 | # Let's say some backend is pulling some text files fom somewhere and dumpt them in the current folder 16 | # We will use a regexp to scan them 17 | filenames = tf.train.match_filenames_once(dir + '/*.txt') # One minor thing, this op needs to be initialized 18 | # And then we will build a queue to load them once at a time 19 | # Again: multiple threads, no waiting 20 | filenames = tf.Print(filenames, [filenames], message="filename_queue: ") 21 | filename_queue = tf.train.string_input_producer(filenames) # Here we build a queue 22 | reader = tf.WholeFileReader() 23 | key, value = reader.read(filename_queue) # Key holds the filename and value the content 24 | 25 | # So now, we have an async queue to load and read our text files from the filesystem 26 | 27 | # What we want now, is an other async job to prepare all the text files content 28 | # in a good batched format for our GPU 29 | # Remember the only goal of queues is to avoid starving the GPU 30 | # The bottleneck could very well be the environment around the GPU and the GPU itself 31 | 32 | # We will train with each batch containing ... 33 | batch_size = 5 # ... 5 sequence of ... 34 | seq_length = 5 # ... 5 words 35 | # This leads us to find the epoch_size 36 | splitted_textfile = tf.string_split([value], " ") 37 | text_length = tf.size(splitted_textfile.values) 38 | batch_len = text_length // batch_size 39 | text = tf.reshape(splitted_textfile.values[:batch_size * batch_len], [batch_size, batch_len]) 40 | epoch_size = (batch_len - 1) // seq_length 41 | range_q = tf.train.range_input_producer(epoch_size, shuffle=False) 42 | index = range_q.dequeue() 43 | x = text[:, index * seq_length:(index + 1) * seq_length] 44 | y = text[:, index * seq_length + 1:(index + 1) * seq_length + 1] 45 | with tf.Session() as sess: 46 | # We initialize Variables 47 | # This is when "match_filenames_once" run the regexp 48 | sess.run(tf.global_variables_initializer()) 49 | 50 | coord = tf.train.Coordinator() 51 | threads = tf.train.start_queue_runners(coord=coord) 52 | 53 | for i in range(1): 54 | print(sess.run([x, y])) 55 | # print("File %s beginning with %s" % (sess.run(key), str(sess.run(value)[:30]) + "...") ) 56 | # print("File %s beginning with %s" % (sess.run(key), str(sess.run(value)[:30]) + "...") ) 57 | # print("File %s beginning with %s" % (sess.run(key), str(sess.run(value)[:30]) + "...") ) 58 | # print("File %s beginning with %s" % (sess.run(key), str(sess.run(value)[:30]) + "...") ) 59 | 60 | coord.request_stop() 61 | coord.join(threads) 62 | -------------------------------------------------------------------------------- /tf-queues/reader_test.py: -------------------------------------------------------------------------------- 1 | import os, time 2 | import tensorflow as tf 3 | 4 | dir = os.path.dirname(os.path.realpath(__file__)) 5 | 6 | filenames = tf.train.match_filenames_once(dir + '/*.txt') 7 | filename_queue = tf.train.string_input_producer(filenames) 8 | reader = tf.WholeFileReader() 9 | key, value = reader.read(filename_queue) 10 | 11 | splitted_textfile = tf.string_split([value], " ") 12 | value_size = tf.size(splitted_textfile.values) 13 | 14 | with tf.Session() as sess: 15 | sess.run(tf.global_variables_initializer()) 16 | 17 | coord = tf.train.Coordinator() 18 | threads = tf.train.start_queue_runners(coord=coord) 19 | 20 | print(sess.run(value_size)) 21 | 22 | coord.request_stop() 23 | coord.join(threads) -------------------------------------------------------------------------------- /tf-queues/test_batching.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import tensorflow as tf 4 | 5 | dir = os.path.dirname(os.path.realpath(__file__)) 6 | 7 | # We simulates some raw inputs data 8 | # let's say we receive 100 batches, each containing 50 elements 9 | x_inputs_data = tf.random_normal([2], mean=0, stddev=1) 10 | # q = tf.FIFOQueue(capacity=10, dtypes=tf.float32) 11 | # enqueue_op = q.enqueue_many(x_inputs_data) 12 | 13 | # input = q.dequeue() 14 | 15 | # numberOfThreads = 1 16 | # qr = tf.train.QueueRunner(q, [enqueue_op] * numberOfThreads) 17 | 18 | batch_input = tf.train.batch( 19 | [x_inputs_data], 20 | batch_size=3, 21 | num_threads=1, 22 | capacity=32, 23 | enqueue_many=False, 24 | shapes=None, 25 | dynamic_pad=False, 26 | allow_smaller_final_batch=True 27 | ) 28 | with tf.Session() as sess: 29 | coord = tf.train.Coordinator() 30 | threads = tf.train.start_queue_runners(coord=coord) 31 | # threads = qr.create_threads(sess, coord=coord, start=True) 32 | 33 | print(sess.run([x_inputs_data, batch_input])) 34 | coord.request_stop() 35 | coord.join(threads) -------------------------------------------------------------------------------- /tf-save-load/README.md: -------------------------------------------------------------------------------- 1 | # Tensorflow: How to freeze a model and serve it with a python API 2 | 3 | ## TL, DR! 4 | ### EN 5 | 6 | ### FR 7 | -------------------------------------------------------------------------------- /tf-save-load/embedding.py: -------------------------------------------------------------------------------- 1 | import os, time 2 | import tensorflow as tf 3 | 4 | dir = os.path.dirname(os.path.realpath(__file__)) 5 | results_dir = dir + "/results/" + str(int(time.time())) 6 | 7 | ### UTILS ### 8 | def batch_text(corpus, batch_size, seq_length): 9 | if seq_length >= len(corpus): 10 | raise Error("seq_length >= len(corpus): %d>=%d" % (seq_length, len(corpus))) 11 | 12 | seqs = [corpus[i:i+seq_length] for i in range(len(corpus) - seq_length)] 13 | ys = [corpus[i:i+1] for i in range(seq_length, len(corpus))] 14 | for i in range(0, len(seqs), batch_size): 15 | x = seqs[i:i+batch_size] 16 | y = ys[i:i+batch_size] 17 | 18 | yield x, y 19 | 20 | 21 | # Let's take a usual usecase you might have: 22 | # You want to train a NN on a NLP task to do predictive coding on a corpus using an embedding 23 | # This is an unsupervised task, one can see this as a way to evaluate the capcity of model in terms of pure memorisation 24 | 25 | # We load our corpus 26 | with open("lorem.txt", 'r') as f: 27 | text = f.read() 28 | corpus = text.split() 29 | corpus_length = len(corpus) 30 | # We build our mapping between token ids and tokens 31 | tokens = set(corpus) 32 | word_to_id_dict = { word:i for i, word in enumerate(tokens) } 33 | id_to_word_dict = { i:word for word,i in word_to_id_dict.items() } 34 | id_corpus = [ word_to_id_dict[word] for word in corpus ] 35 | nb_token = len(tokens) 36 | 37 | 38 | # We will train this embedding with predictive coding 39 | # The input of our model is a number "seq_length" of precedent words ids 40 | # and the output is the id of next word 41 | seq_length = 5 42 | with tf.variable_scope("placeholder"): 43 | x = tf.placeholder(tf.int32, shape=[None, seq_length], name="x") 44 | y_true = tf.placeholder(tf.int32, shape=[None, 1], name="y_true") 45 | 46 | 47 | # We create an embedding 48 | # We choose in how many dimensions we want to embed our word vectors 49 | dim_embedding = 30 50 | with tf.variable_scope("embedding"): 51 | # Let's build our embedding, we intialize it with s impel random normal centerd on 0 with a small variance 52 | embedding = tf.get_variable("embedding", shape=[nb_token, dim_embedding], initializer=tf.random_normal_initializer(0., 1e-3)) 53 | # Then we retrieve the context vector 54 | context_vec = tf.nn.embedding_lookup(embedding, x, name="lookup") # Dim: bs x seq_length x dim_embedding 55 | context_vec = tf.reshape(context_vec, [tf.shape(x)[0], seq_length * dim_embedding]) 56 | 57 | # We build a Neural net to predict the next word vector 58 | with tf.variable_scope("1layer"): 59 | # We use the context vector to predict the next word vector inside the embedding 60 | W1 = tf.get_variable("W1", dtype=tf.float32, shape=[seq_length * dim_embedding, dim_embedding]) 61 | b1 = tf.get_variable("b1", dtype=tf.float32, shape=[dim_embedding], initializer=tf.constant_initializer(.1)) 62 | h1 = tf.nn.relu(tf.matmul(context_vec, W1) + b1) 63 | 64 | W2 = tf.get_variable("W2", dtype=tf.float32, shape=[dim_embedding, dim_embedding]) 65 | b2 = tf.get_variable("b2", dtype=tf.float32, shape=[dim_embedding], initializer=tf.constant_initializer(.1)) 66 | y_vector = tf.matmul(h1, W2) + b2 # Dim: bs x dim_embeddiing 67 | 68 | # Now we calculated the dot product of the current words with all other words vectors 69 | z = tf.matmul(y_vector, tf.transpose(embedding)) # Dim: bs x nb_token 70 | 71 | with tf.variable_scope("loss"): 72 | y_true_reshaped = tf.reshape(y_true, [-1]) 73 | losses = tf.nn.sparse_softmax_cross_entropy_with_logits(None, y_true_reshaped, z) 74 | loss_op = tf.reduce_mean(losses) 75 | 76 | tf.summary.scalar('loss', loss_op) 77 | 78 | with tf.variable_scope("Accuracy"): 79 | a = tf.nn.softmax(z) 80 | predictions = tf.cast(tf.argmax(a, 1, name="predictions"), tf.int32) 81 | correct_predictions = tf.equal(predictions, y_true_reshaped) 82 | nb_predicted_words = tf.shape(predictions)[0] 83 | nb_wrong_predictions = nb_predicted_words - tf.reduce_sum(tf.cast(correct_predictions, tf.int32)) 84 | accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32), name="accuracy") 85 | 86 | with tf.variable_scope('Optimizer'): 87 | global_step_t = tf.Variable(0, name="global_step", trainable=False) 88 | lr = tf.train.exponential_decay(3e-2, global_step_t, 500, 0.5, staircase=True) 89 | adam = tf.train.AdamOptimizer(lr) 90 | tf.summary.scalar('lr', lr) 91 | 92 | train_op = adam.minimize(loss_op, global_step=global_step_t, name="train_op") 93 | 94 | summaries = tf.summary.merge_all() 95 | 96 | # We build a Saver in the default graph handling all existing Variables 97 | saver = tf.train.Saver() 98 | 99 | nb_epochs = 500 100 | with tf.Session() as sess: # The Session handles the default graph too 101 | sess.run(tf.global_variables_initializer()) 102 | sw = tf.summary.FileWriter(results_dir, sess.graph) 103 | 104 | for i in range(nb_epochs): 105 | input_gen = batch_text(id_corpus, corpus_length // 6, seq_length) 106 | for x_batch, y_true_batch in input_gen: 107 | to_compute = [train_op, loss_op, global_step_t, summaries] 108 | feed_dict = { 109 | x: x_batch, 110 | y_true: y_true_batch 111 | } 112 | _, loss, global_step, summaries_metric = sess.run(to_compute, feed_dict=feed_dict) 113 | 114 | # We add a data point in our "events..." file 115 | sw.add_summary(summaries_metric, global_step) 116 | 117 | if (i + 1) % 250 == 0: 118 | # We save our model 119 | saver.save(sess, results_dir + '/model', global_step=i + 1) 120 | 121 | # We compute the final accuracy 122 | val_gen = batch_text(id_corpus, corpus_length, seq_length) 123 | for x_batch, y_batch in val_gen: 124 | feed_dict = { 125 | x: x_batch, 126 | y_true: y_batch 127 | } 128 | acc, nb_preds, nb_w_pred = sess.run([accuracy, nb_predicted_words, nb_wrong_predictions], feed_dict=feed_dict) 129 | print('Final accuracy: %f' % acc) 130 | print('%d mispredicted words of %d predictions' % (nb_w_pred, nb_preds)) 131 | 132 | -------------------------------------------------------------------------------- /tf-save-load/lorem.txt: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum pulvinar leo nec pulvinar. Sed et rhoncus turpis. Nunc vitae nisl elementum, facilisis libero eu, elementum augue. Sed et lectus tellus. Interdum et malesuada fames ac ante ipsum primis in faucibus. Proin pellentesque lacus sit amet ante luctus cursus. Nunc ut lacus eget arcu varius facilisis non eget magna. Nullam vestibulum velit non lorem gravida, vel bibendum ligula suscipit. Donec aliquet aliquet dignissim. Integer vestibulum elementum mi, et auctor erat egestas vitae. 2 | 3 | Morbi vitae suscipit magna. Pellentesque auctor nisi et turpis scelerisque laoreet. Etiam nec hendrerit ipsum. Nulla ut nibh sit amet elit eleifend tempus. Aenean mi ligula, aliquam venenatis viverra at, suscipit id dui. Etiam auctor nec neque sit amet convallis. Suspendisse quis diam eros. 4 | 5 | Praesent quis porta tortor, at tincidunt justo. Curabitur nec tortor consectetur, porta ligula sit amet, consequat nibh. Curabitur blandit purus at blandit feugiat. Nullam aliquam eget tellus quis maximus. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Donec et imperdiet risus. Morbi sapien enim, finibus at erat sit amet, dictum rutrum libero. Suspendisse fermentum consequat est, eu dapibus orci aliquam at. Phasellus eu nisl nibh. Phasellus in aliquam magna. 6 | 7 | Integer interdum dui nec lacus varius volutpat. Fusce volutpat, odio nec euismod bibendum, risus odio ultricies mi, vel aliquam quam sapien eget tortor. Maecenas et tempus felis. Fusce blandit eleifend metus vel feugiat. Proin sed tristique orci. Sed vel tincidunt mi. Nunc lobortis arcu quis enim feugiat congue. Ut quis arcu nisl. Ut dui lorem, scelerisque at suscipit at, congue vel mi. Integer feugiat turpis ut elementum suscipit. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. 8 | 9 | Nulla tincidunt pellentesque egestas. Phasellus leo ex, scelerisque mollis eleifend ac, commodo vel arcu. Morbi in auctor mauris. Vestibulum id lobortis mi. Etiam scelerisque tempus augue, id scelerisque sapien euismod eu. Integer euismod, neque ut egestas hendrerit, est urna feugiat felis, ut maximus arcu arcu sed nisl. Suspendisse facilisis aliquam ante eu consequat. Ut vitae magna enim. Sed in ipsum vitae turpis lacinia faucibus. 10 | 11 | Sed augue ligula, tempor sed nisl eget, vehicula faucibus nunc. Nullam feugiat at tortor ut malesuada. In ullamcorper, lorem quis scelerisque mattis, magna purus convallis tortor, eu laoreet ligula ligula ut mauris. Sed odio ex, tempus ut vestibulum a, commodo sit amet orci. Donec et sodales nibh, ac gravida leo. Etiam vel urna vel ipsum aliquam luctus. Quisque bibendum felis ac urna maximus sagittis. Nulla eget tortor eu nisl malesuada varius a et tellus. Suspendisse sed posuere neque. 12 | 13 | Morbi sed tortor in tellus commodo fermentum eu ultricies elit. Sed nec consectetur enim, egestas consequat ligula. Sed massa ante, vulputate nec tortor nec, porttitor lobortis augue. Integer pulvinar, dui id dapibus commodo, sapien nisi scelerisque lacus, eget porttitor orci lorem vitae massa. Donec a nibh at justo elementum faucibus. Suspendisse lacinia risus rhoncus dui elementum, sit amet volutpat ligula venenatis. Vivamus ut pharetra risus. Sed et augue sit amet nisi varius tempus. Proin elit purus, iaculis ac imperdiet pretium, posuere vel purus. Aliquam faucibus, diam id pretium sollicitudin, nulla urna aliquam sem, sed feugiat ipsum eros vitae nisl. In ornare, justo ac bibendum mattis, lorem mauris sagittis tellus, eget vehicula tellus orci vel leo. Vivamus malesuada arcu eget luctus imperdiet. Morbi laoreet venenatis felis ac molestie. Maecenas in accumsan turpis. Sed nec laoreet metus, sit amet auctor urna. 14 | 15 | Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed et nibh urna. Nam ac massa at nisl suscipit rutrum. Aenean vehicula lectus ex, a pretium nunc luctus at. Nullam blandit est id purus pharetra ornare. Sed et tortor et felis vestibulum imperdiet quis vel diam. Nunc vehicula nec quam bibendum eleifend. Proin ornare nulla sit amet lacinia luctus. Fusce ultricies vitae enim quis pretium. Aliquam venenatis ligula sit amet nisi venenatis, non viverra justo vehicula. Phasellus eu sollicitudin erat, at blandit ante. Suspendisse vel elit vel eros venenatis rhoncus eget quis urna. Duis dictum, erat ac sodales accumsan, tellus erat egestas eros, in sagittis eros ligula in lectus. 16 | 17 | Proin suscipit augue non mattis suscipit. Ut eleifend tempor urna, non blandit velit aliquet ac. Quisque id dictum mi. Aliquam mi quam, ullamcorper eget elit et, feugiat cursus erat. Cras mattis tellus sed eleifend laoreet. Etiam pellentesque eros ipsum, non hendrerit purus dignissim sed. Quisque dignissim neque sem, id scelerisque velit condimentum eu. 18 | 19 | Sed accumsan nisl in odio consectetur, at hendrerit elit molestie. Donec id gravida ante, sed suscipit nunc. Nunc id vehicula est, id tempor tellus. Sed vel maximus libero, sodales sodales arcu. Nam dictum lacus ipsum, vitae tempus est vulputate iaculis. Nam tincidunt arcu eget felis fermentum, placerat ultricies quam vehicula. Sed suscipit facilisis mauris, quis pretium augue semper eu. Fusce in tempus quam, vel facilisis urna. Donec tempor nunc metus, et fringilla ipsum ultrices in. Nam libero sapien, faucibus id felis sed, elementum porta neque. Aenean vel magna a nisl commodo varius a id ipsum. Maecenas quis tortor varius, finibus nisi at, ornare tortor. Duis sed felis enim. Phasellus eget lacus posuere justo tempor interdum. Maecenas ex ante, sodales ac molestie sed, mollis non metus. Donec tortor sem, volutpat id orci eget, semper porta elit. 20 | 21 | Curabitur non elit id nunc ornare mollis. Proin ac purus vitae sem consectetur aliquet. Morbi sed felis vitae tellus mattis hendrerit congue in velit. Cras vitae vulputate est. Aenean eu eros ex. Nam eget libero velit. Vestibulum pharetra sollicitudin mi, pretium rhoncus ipsum faucibus ut. Duis tincidunt erat nec eros tincidunt lobortis. Nunc ac tellus eget nunc imperdiet lobortis ac a diam. Sed facilisis libero vitae rhoncus pulvinar. Vestibulum aliquet posuere lectus, aliquet malesuada purus porttitor eget. Donec at lobortis massa. Praesent consectetur rutrum purus, at consectetur nisi ultricies ut. In vehicula elit neque, vel consectetur purus porta vel. Nulla fermentum felis commodo velit gravida sagittis sed aliquam nisi. 22 | 23 | Curabitur vel urna et leo scelerisque maximus. Praesent mattis, massa nec lobortis tempor, nulla tortor blandit mi, id malesuada urna velit sit amet elit. Aenean fringilla vel tellus eget facilisis. Proin eget lorem in quam consectetur facilisis eget eu odio. Quisque pulvinar consectetur nisi nec venenatis. Praesent ut semper nisi. Vivamus est nulla, semper sed massa vitae, placerat eleifend purus. Phasellus vel risus vitae mi pharetra dictum. Mauris maximus consequat faucibus. Pellentesque est velit, semper et ante vel, commodo ullamcorper arcu. 24 | 25 | Maecenas placerat lorem nec auctor sollicitudin. Aliquam erat volutpat. Donec gravida pretium augue, in tristique lectus rutrum ut. Aliquam aliquam, neque quis consequat eleifend, ex ex tincidunt dui, feugiat faucibus metus leo ac odio. Integer vel vulputate sem. Phasellus ullamcorper sapien risus, sit amet pulvinar sapien semper non. Aliquam rutrum maximus tellus, at consequat dolor. 26 | 27 | Phasellus convallis, est interdum auctor dignissim, felis libero tempor urna, in maximus mi sapien eget dolor. Nulla facilisi. Etiam viverra sollicitudin neque, vel semper dui porta ut. Sed viverra massa diam, nec feugiat massa vehicula non. Curabitur tempor arcu eu arcu venenatis vestibulum. In ligula mauris, laoreet et tortor non, fringilla blandit ante. Aenean sit amet orci est. Suspendisse vel felis at elit tristique commodo. Nullam rutrum faucibus velit, id porttitor nisi vehicula eget. 28 | 29 | Integer tristique, orci sit amet molestie dictum, magna justo lacinia libero, faucibus accumsan velit neque eget elit. Maecenas diam purus, eleifend vitae aliquet ut, lacinia eu lectus. Praesent mi nunc, porta suscipit viverra eget, auctor id magna. Praesent non urna libero. Praesent pharetra nisi neque, eget pulvinar lectus porttitor porta. Nunc auctor odio vitae posuere lacinia. Phasellus eu tortor venenatis, maximus libero vel, porta mauris. Curabitur et arcu nec dolor imperdiet auctor. In id eros laoreet, vehicula eros non, elementum urna. Curabitur semper mollis dolor, vel ultrices augue ullamcorper id. Integer eget auctor lacus. Vestibulum accumsan odio at ipsum feugiat, nec dignissim velit suscipit. Phasellus tincidunt, quam at mollis pharetra, dui risus congue tortor, at tristique dui neque vitae velit. Vivamus et commodo purus. Aliquam eu fringilla leo, vitae facilisis velit. Etiam eget tellus vel leo interdum rutrum. 30 | 31 | Suspendisse tempor sem a felis fringilla accumsan. Proin rhoncus tincidunt risus, nec cursus enim malesuada sit amet. Quisque sem arcu, vulputate vel condimentum nec, porta non turpis. Aenean tincidunt nisl non faucibus hendrerit. Donec cursus at urna quis mollis. Fusce suscipit risus id fringilla rutrum. Quisque lectus velit, feugiat et mauris ac, condimentum molestie elit. In sollicitudin lorem quis congue eleifend. Cras id euismod magna. In fermentum porta fermentum. Etiam mollis viverra efficitur. Aenean efficitur faucibus pellentesque. Vivamus ullamcorper vitae ex eu pharetra. Aenean maximus, dui eu luctus sollicitudin, quam nibh malesuada massa, quis pulvinar nulla nibh id sapien. Curabitur eget quam nec augue mattis scelerisque. Phasellus feugiat purus vel nisl facilisis rhoncus. 32 | 33 | Vivamus sit amet mollis metus. Aenean mollis neque at urna scelerisque pulvinar. Morbi varius nibh augue, eget sollicitudin nunc interdum sed. Interdum et malesuada fames ac ante ipsum primis in faucibus. Sed ligula velit, consequat condimentum justo id, elementum porttitor massa. Maecenas feugiat accumsan erat vel sagittis. Praesent eu lorem interdum, maximus eros et, laoreet velit. Morbi at accumsan mi. 34 | 35 | Vivamus dictum et odio eget scelerisque. Cras dictum, leo eget ultrices vehicula, lectus augue venenatis nunc, eu convallis velit nisi malesuada massa. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Duis vehicula purus vel mauris dictum, eget blandit nulla dictum. Sed eu erat imperdiet, facilisis turpis et, imperdiet urna. Duis iaculis pretium leo, non imperdiet urna cursus at. Vestibulum rutrum urna odio, et volutpat sem volutpat nec. Interdum et malesuada fames ac ante ipsum primis in faucibus. 36 | 37 | Proin molestie sollicitudin nulla vel venenatis. Vestibulum vel auctor urna, sit amet aliquam velit. Pellentesque feugiat erat ante, ac elementum lacus feugiat a. Praesent eu placerat erat, quis mattis lectus. Etiam sit amet molestie velit. Maecenas ante augue, aliquam in tincidunt ac, accumsan nec felis. Vivamus vel volutpat orci, ac consequat neque. Duis porta mi nec volutpat tempor. Vivamus rhoncus neque ipsum. 38 | 39 | Vivamus pharetra augue metus, vel condimentum augue facilisis sit amet. Fusce non nibh varius, porta ante sed, vehicula ipsum. Aenean bibendum erat leo, at egestas magna ultricies nec. Aenean vel risus a metus commodo venenatis. In hac habitasse platea dictumst. Aenean dolor neque, lacinia varius sollicitudin vel, dignissim vitae felis. Nulla facilisi. 40 | 41 | Phasellus venenatis nunc ut nunc consectetur sagittis. Donec a velit a elit lobortis suscipit. Cras accumsan sit amet nunc tempus molestie. Vestibulum at dolor vel felis luctus bibendum. Quisque porttitor, augue id ornare iaculis, purus nisi vehicula ligula, vel pellentesque nisl ante at mauris. Mauris eros risus, lacinia non est quis, tempor tempor velit. Praesent nec suscipit sapien, a rhoncus dui. Proin dolor lorem, ullamcorper eget mi a, ultricies aliquet ligula. Suspendisse potenti. Phasellus semper magna vel odio porta facilisis. Mauris et eros non mi semper sodales. Phasellus vehicula dolor vel neque suscipit volutpat. Nam nec massa lorem. Nullam efficitur cursus placerat. Nunc dapibus accumsan luctus. Aliquam erat volutpat. 42 | 43 | Quisque rhoncus orci nec aliquet volutpat. Sed non metus vitae mi tincidunt posuere. Interdum et malesuada fames ac ante ipsum primis in faucibus. Donec ullamcorper convallis sapien, eu fermentum leo sollicitudin eu. In imperdiet mollis nisl, nec consequat justo lacinia id. Curabitur ac ipsum vel enim pretium egestas in sit amet nisi. Nam porttitor, ipsum ac sollicitudin sodales, nulla tellus sodales lacus, vel eleifend mauris ex id purus. Sed id cursus quam. Fusce volutpat ligula turpis, a blandit dolor commodo eget. Proin in velit ac ipsum ultrices pellentesque. Fusce quam erat, lobortis et faucibus non, tincidunt non ipsum. 44 | 45 | Cras faucibus convallis felis ac pharetra. Sed nibh metus, consectetur semper tempor eu, vestibulum vel justo. In hac habitasse platea dictumst. Curabitur condimentum eros tellus, quis condimentum sapien interdum eu. Nunc sagittis dictum odio. Ut rhoncus ex eros, vel ultricies massa sollicitudin et. Pellentesque mattis lobortis dui, sit amet rhoncus erat facilisis ac. 46 | 47 | Nulla pretium porta metus, sed commodo velit tincidunt at. In dapibus neque at nisi ultrices feugiat. Fusce a arcu eget sapien pulvinar elementum quis eu sapien. Sed gravida massa lacinia dui lacinia, sit amet euismod nisl porttitor. Nulla facilisi. Integer non tincidunt ex. Nunc a euismod quam, dapibus egestas sapien. Mauris maximus semper magna, ut porttitor ligula. Sed cursus vulputate turpis, a ultricies purus. Proin semper enim et nisl elementum, quis vehicula tortor gravida. Maecenas vehicula accumsan risus rhoncus sollicitudin. Praesent lobortis velit vel enim rutrum, in aliquet nulla euismod. Ut eu porta augue. 48 | 49 | Cras ornare commodo urna, id lobortis lorem scelerisque facilisis. Nunc ut molestie diam. Suspendisse a tortor sem. Aenean varius lacus vitae nibh porttitor auctor. Cras consequat ipsum eget massa ullamcorper, eget aliquet ante congue. Vestibulum id semper purus. Nam rhoncus leo ac sapien aliquam, in rhoncus ligula blandit. 50 | 51 | Nulla facilisi. Suspendisse non rhoncus leo. Nam eu purus sollicitudin, lobortis ex at, consectetur eros. Morbi pulvinar leo feugiat, malesuada quam sit amet, aliquet mi. Donec malesuada, est vel ullamcorper ultrices, magna mauris ultrices nisl, vitae pretium nisl tellus consequat mi. Pellentesque quis consectetur tortor, eu commodo odio. Quisque in sollicitudin mauris. Ut laoreet leo vel est interdum accumsan. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Curabitur sollicitudin, mi sed tincidunt maximus, erat turpis cursus sapien, et accumsan nibh erat a libero. Nulla velit augue, fermentum eget est eget, consequat sollicitudin sem. Interdum et malesuada fames ac ante ipsum primis in faucibus. 52 | 53 | Nunc enim lacus, egestas malesuada orci vitae, condimentum bibendum quam. Sed semper, ligula id fermentum placerat, turpis magna condimentum ante, id placerat lorem magna nec leo. Vestibulum laoreet elit ac enim congue, quis pulvinar tellus luctus. Quisque laoreet nulla at turpis congue lacinia. Vestibulum dignissim libero quis hendrerit fermentum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Proin scelerisque velit sit amet condimentum elementum. Integer mollis felis ut rutrum ullamcorper. Quisque sit amet lacus non augue eleifend aliquam nec lobortis dui. Maecenas eros dolor, fermentum in eros eget, imperdiet malesuada nunc. Vivamus tempus semper nibh, sed vehicula risus placerat vel. Nullam sed molestie nisi, a pellentesque est. Proin ut commodo sapien. 54 | 55 | Vivamus a suscipit lectus, eget malesuada dolor. Integer laoreet aliquam turpis, non mollis magna rutrum eu. Phasellus ut nibh eros. Mauris sit amet magna vel metus ornare dictum. Cras sit amet enim eget magna tincidunt cursus id quis dolor. Curabitur ornare, nunc non lacinia tristique, mi nibh suscipit dolor, ac vulputate ipsum elit quis turpis. Nulla ante sem, euismod non accumsan in, rutrum vel diam. Etiam vel ligula ullamcorper, sodales dui sed, pellentesque ipsum. Quisque eget dignissim est. Cras nec nisi et diam lobortis elementum. Ut fermentum sodales libero, a varius diam pretium sit amet. Aenean vel blandit eros. Proin volutpat quis quam tincidunt suscipit. Etiam convallis, arcu id facilisis auctor, dui sem semper lacus, ac cursus massa leo sollicitudin sapien. Aliquam feugiat eget leo id vestibulum. 56 | 57 | Donec venenatis nisl nibh, id iaculis erat tincidunt non. Donec lobortis et diam vitae ultricies. Phasellus vitae efficitur metus, non blandit nibh. Mauris id leo libero. Sed et maximus leo. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Mauris nec risus sit amet erat lacinia sagittis. Donec tempus lacus eget neque facilisis varius non a dolor. Nulla vehicula laoreet sapien vel iaculis. Sed hendrerit, augue sit amet rutrum commodo, nisl quam molestie augue, sed accumsan urna lacus ac enim. 58 | 59 | Nunc non mauris eget nisi congue cursus. Integer fermentum mauris eros, ut feugiat massa tempus quis. Fusce at nulla ultrices neque rutrum hendrerit vitae vel odio. Sed at urna posuere, convallis magna non, cursus justo. In non justo at lacus sollicitudin finibus. Curabitur at placerat massa. Quisque nec mi libero. Sed condimentum justo ut diam convallis, laoreet mollis orci consequat. Donec sagittis ipsum at eleifend bibendum. Vestibulum quis aliquet ipsum. 60 | 61 | Nullam tristique imperdiet finibus. Duis eu tortor tincidunt dui pellentesque vestibulum eu nec massa. Praesent consectetur, sapien vitae tincidunt iaculis, turpis arcu efficitur magna, in fermentum nunc mi pellentesque diam. Donec sit amet cursus odio, a condimentum tellus. Integer feugiat, tellus id suscipit accumsan, dui arcu molestie diam, eget mattis massa eros nec massa. Praesent sit amet neque ac ex imperdiet sodales at eget nulla. Maecenas euismod tellus non tempor elementum. Integer efficitur rhoncus eros in commodo. Nulla blandit suscipit eleifend. Aliquam laoreet rutrum arcu, molestie tristique arcu tincidunt eu. Nullam fringilla nisi placerat libero accumsan sagittis. Fusce fermentum ultrices orci, vel ullamcorper tellus malesuada lobortis. Sed ut rutrum dui. Phasellus quis velit enim. Praesent a lectus et mauris sagittis tincidunt. Sed interdum tempus mauris sed venenatis. 62 | 63 | Proin finibus et leo sed varius. Morbi sit amet luctus dolor, vitae tincidunt orci. Nam fringilla suscipit nulla et luctus. Cras quis tortor in lectus ultricies vulputate nec et orci. Mauris at hendrerit lectus, eu molestie nisl. Proin ut cursus nisi, a vulputate purus. Aenean dictum sem est, vel vulputate risus fringilla in. 64 | 65 | Pellentesque hendrerit ut purus eget facilisis. Morbi accumsan gravida turpis. Nam non imperdiet velit. Integer ipsum dolor, iaculis id varius non, pellentesque in mi. Proin faucibus ut nibh ac consectetur. Etiam ullamcorper gravida arcu sit amet sodales. Phasellus eget libero in tortor faucibus vehicula. Integer viverra urna turpis, at ultricies risus posuere id. Etiam luctus finibus elementum. Nulla fringilla aliquet ante, et lobortis enim placerat eu. 66 | 67 | Aliquam erat volutpat. Duis pellentesque rutrum orci, eu sodales neque venenatis ut. Integer ac volutpat libero. Aenean congue consequat ultrices. Vestibulum eget nibh ornare, scelerisque ligula at, iaculis purus. Phasellus posuere dignissim elit, at auctor velit malesuada et. Suspendisse id lacus quis dui tristique scelerisque. Vivamus feugiat fringilla diam. Etiam nec lectus elit. Donec pellentesque convallis pretium. Sed sed odio vestibulum, pulvinar sem in, semper dui. Aenean sed mauris a erat laoreet bibendum a non sem. Nunc nec sem purus. Proin tincidunt tincidunt feugiat. Aenean lorem elit, pretium a erat ac, ultrices ullamcorper dolor. Sed tincidunt tortor sed scelerisque varius. 68 | 69 | Sed ornare, mi varius sagittis sagittis, ligula diam rhoncus ligula, vel euismod orci nibh elementum sem. Curabitur sed neque at elit ultricies ultrices at eu nisi. Proin tempor neque sed elit finibus, a commodo eros faucibus. Praesent luctus a justo vel condimentum. Suspendisse blandit nisl sollicitudin sem vulputate vehicula. Fusce sit amet convallis enim. Integer vel molestie ante, a vehicula augue. Mauris fringilla tincidunt risus in blandit. Nullam quis blandit lacus. Sed dictum lectus est, non tincidunt metus dapibus a. Quisque eleifend euismod erat in faucibus. Aenean non bibendum augue, in pulvinar lorem. Cras consectetur ex vel neque tincidunt posuere in nec diam. Integer sed mauris vitae diam lobortis euismod vitae quis magna. 70 | 71 | In hac habitasse platea dictumst. Vivamus a mi metus. Nulla iaculis leo ut purus maximus maximus. Proin tincidunt imperdiet diam, nec pellentesque dolor lacinia eu. Praesent augue massa, ultrices et tellus eu, vulputate accumsan arcu. Curabitur facilisis, felis mattis euismod dictum, felis metus venenatis erat, nec ornare lectus nulla vel nibh. Ut vestibulum dignissim tellus, ullamcorper consequat eros blandit quis. Praesent sed neque in elit imperdiet bibendum et lobortis purus. Sed sed libero leo. Nullam tristique est eu dolor elementum vulputate. Quisque vestibulum faucibus elit, eget pharetra nibh pellentesque et. Quisque aliquam ut purus ut laoreet. Ut vel vehicula eros, ac lacinia lacus. Praesent quis auctor orci, in scelerisque arcu. Sed iaculis aliquet nibh, sit amet elementum augue efficitur id. Nam sed aliquet metus. 72 | 73 | Mauris ultrices mattis mattis. Mauris orci mauris, blandit non augue sit amet, commodo cursus nisi. Cras sed posuere justo. Sed sodales lacinia dictum. Sed vel sapien at arcu volutpat maximus. Sed finibus ac libero et lacinia. Interdum et malesuada fames ac ante ipsum primis in faucibus. Phasellus ipsum nibh, blandit et vestibulum non, sagittis eu lorem. 74 | 75 | Aliquam eu ullamcorper erat, et tincidunt magna. Maecenas urna neque, varius nec ligula ac, luctus cursus quam. Quisque vestibulum, leo ac auctor interdum, arcu nulla vestibulum dolor, eu pretium elit leo posuere lacus. Aenean eget tincidunt turpis. Aliquam erat volutpat. Sed sed iaculis eros, nec vestibulum metus. Duis libero tellus, feugiat in tortor vel, porttitor suscipit orci. Pellentesque vulputate, orci at dapibus sagittis, metus dolor faucibus nulla, ac rutrum nulla odio ac massa. Mauris ac laoreet neque. In vitae odio luctus, congue ex id, blandit eros. Integer porta venenatis purus vulputate bibendum. Sed ligula magna, imperdiet a tristique a, pharetra non enim. Ut pretium risus id sem lacinia, nec scelerisque erat interdum. Ut quis nisl a libero venenatis dictum ac vitae orci. Fusce imperdiet neque urna, in pharetra dui convallis in. Quisque neque enim, posuere in libero sed, condimentum scelerisque erat. 76 | 77 | Praesent magna nisi, ultricies viverra felis a, suscipit pellentesque purus. Pellentesque massa quam, scelerisque eget felis quis, blandit sollicitudin nisl. Nunc pellentesque scelerisque enim, non elementum ex auctor vitae. Mauris vel orci a metus eleifend porttitor quis sit amet metus. Phasellus faucibus tortor sed diam ullamcorper, a molestie libero semper. Integer vel molestie lacus. Nunc bibendum blandit augue, at egestas lectus sagittis eu. Fusce dictum ipsum vel porta ornare. Curabitur vitae odio non odio hendrerit euismod. Duis eu elit nisl. Sed bibendum orci vitae placerat venenatis. Fusce vestibulum sem vel urna sollicitudin gravida. In dapibus ex quis sodales mattis. 78 | 79 | Quisque vehicula ligula vel neque sagittis volutpat. In pharetra lorem eu ante euismod, nec interdum nisi convallis. In elit lacus, vestibulum vulputate magna nec, tincidunt porttitor diam. Sed tristique, urna semper feugiat efficitur, massa turpis rhoncus magna, et euismod libero tortor nec sem. Etiam varius molestie malesuada. Pellentesque vehicula lacinia turpis in convallis. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Quisque porta nec urna a dignissim. Phasellus diam turpis, sagittis quis fermentum in, dapibus sed orci. Donec vestibulum metus hendrerit, ultrices est sollicitudin, commodo eros. Donec in nulla ut velit gravida elementum eu vitae velit. 80 | 81 | Pellentesque vehicula risus eget elit aliquam elementum. Sed vulputate urna lacus. Nullam aliquam bibendum elit, vel accumsan metus iaculis in. Praesent at congue velit. Donec sed rutrum nisl. Nullam euismod, sem at fermentum dapibus, mauris est facilisis dui, eget mollis augue lectus in libero. Suspendisse potenti. Vivamus eget ante vitae risus pretium iaculis. Nulla at risus dictum, sollicitudin leo eu, efficitur nunc. Aliquam fringilla dolor justo, non mollis mi lobortis ultrices. Maecenas ultricies, neque vel tempor consectetur, lectus sem rhoncus leo, cursus auctor urna urna non ante. Morbi nec nulla molestie, ultricies mauris id, ornare est. Interdum et malesuada fames ac ante ipsum primis in faucibus. 82 | 83 | Vivamus congue porta accumsan. Curabitur in leo sit amet risus scelerisque fringilla eu ut mi. Sed ut pellentesque lorem. Praesent varius lectus placerat sodales sollicitudin. Praesent id tincidunt lectus. Etiam vulputate efficitur justo sed rutrum. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Donec varius tempus nunc, in luctus mi euismod eget. Sed ac commodo ex. Vivamus aliquet nibh urna. 84 | 85 | Fusce sit amet felis ut nulla iaculis eleifend id non quam. Ut ut laoreet diam. Aenean aliquet lacus odio, a luctus lorem ullamcorper nec. Praesent id turpis lobortis, lobortis lorem sed, molestie leo. Pellentesque nec nibh faucibus, bibendum libero mollis, convallis risus. Vivamus feugiat vestibulum mi a egestas. Aliquam commodo est ac lectus sagittis, at luctus metus condimentum. Quisque luctus lectus lorem, a lobortis lorem condimentum et. Quisque ex leo, dapibus vel risus nec, dignissim elementum lectus. Duis eget sem accumsan, gravida libero at, molestie diam. Duis molestie dapibus erat. Donec a tincidunt purus. Phasellus scelerisque sagittis libero in iaculis. 86 | 87 | Mauris eget laoreet ipsum. Aliquam ut velit convallis risus interdum euismod eget a dolor. Proin auctor nibh ac diam aliquet auctor. Cras enim diam, lacinia sed libero non, rutrum maximus augue. Suspendisse eu varius sapien. Nam neque libero, vestibulum et dignissim quis, vestibulum vitae purus. Nunc mollis sit amet erat at semper. Nullam tempor molestie urna, quis egestas magna ornare in. Nulla vitae interdum tellus. Proin mollis, enim sed ullamcorper interdum, ante nunc ornare ante, ut molestie lacus nibh vel elit. 88 | 89 | Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nulla lacinia congue quam et mattis. Sed iaculis maximus suscipit. Fusce in pellentesque urna. In non tellus sapien. Integer a felis scelerisque, sodales augue sed, cursus erat. Morbi eu tempus odio. Fusce eu semper neque. Vivamus sed tellus at nisl viverra faucibus. Aliquam in nulla ullamcorper, tempor augue nec, vehicula augue. Nunc pellentesque est elit, eu ullamcorper sapien laoreet id. Sed hendrerit pulvinar vehicula. 90 | 91 | Mauris cursus varius mollis. Morbi non augue porttitor lectus ullamcorper porttitor sit amet sed tortor. Curabitur convallis auctor condimentum. Curabitur dictum, tellus sit amet gravida ornare, lacus elit varius nibh, nec lobortis arcu tellus nec elit. Mauris tincidunt ultricies diam tincidunt efficitur. Etiam molestie posuere ante ut imperdiet. Morbi eleifend arcu eget justo elementum, non porttitor orci ullamcorper. Ut vitae ultrices quam, sit amet eleifend ex. Pellentesque sagittis vel mi eget consequat. Proin ac velit magna. 92 | 93 | Donec vel purus risus. Maecenas accumsan dapibus ultrices. Vestibulum at arcu quis massa ultrices pellentesque a in turpis. Cras quis neque vel lorem tincidunt congue. Mauris aliquet libero pellentesque dui malesuada, at accumsan nulla posuere. Nullam posuere viverra volutpat. Sed molestie vitae orci sit amet vestibulum. Pellentesque quis mattis enim. Donec eget justo non ipsum facilisis auctor. Sed ex velit, facilisis id bibendum vulputate, mattis eget quam. Proin laoreet velit ut viverra facilisis. Integer vel urna sed odio facilisis cursus id ac diam. Vestibulum consectetur, erat at congue dignissim, enim ex ultricies eros, id tempus ex nulla in sapien. Phasellus vehicula vestibulum est nec malesuada. 94 | 95 | Proin at pharetra nisl. Suspendisse sed augue eu metus placerat porta vitae non quam. Nam efficitur metus sed libero laoreet, id porttitor orci accumsan. Proin vitae dictum elit. Quisque non massa dui. Pellentesque sit amet dignissim risus. Sed lacinia tellus et nulla congue, eu condimentum tortor fermentum. Duis ipsum ante, vehicula aliquam pretium sed, mollis ac arcu. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Nulla lobortis aliquet ipsum, id pulvinar urna rhoncus eu. Nunc at accumsan elit. Nunc tristique lobortis mi. 96 | 97 | Nam vitae justo id ex porta accumsan. Nunc molestie leo quis odio convallis, quis varius lectus elementum. Morbi commodo nisl at mi fermentum, a porttitor libero lobortis. Pellentesque id vulputate turpis, sed tempus eros. Nulla consequat, orci sed commodo tincidunt, quam lacus interdum lacus, eget porta dolor nunc ut sapien. Praesent vitae justo ultricies, facilisis mauris a, facilisis velit. Phasellus ultricies molestie neque eget fringilla. 98 | 99 | Curabitur volutpat nibh non mi vestibulum, vitae faucibus est vestibulum. Aliquam erat volutpat. Proin bibendum tellus eu enim aliquet, a gravida sapien porta. Integer luctus viverra velit a pretium. In gravida dapibus sapien, in tincidunt nibh interdum vel. Aliquam tempus tincidunt est nec venenatis. Sed mollis purus quis placerat mattis. Sed facilisis quam et elit pharetra tincidunt. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In faucibus iaculis faucibus. Donec posuere aliquam lacus, vitae feugiat lorem elementum et. Phasellus sed sapien ac est feugiat porta. Donec ante est, blandit et nulla at, bibendum placerat arcu. Nam dui nisi, pellentesque ut tempor et, viverra eu massa. 100 | 101 | Mauris semper, turpis sit amet placerat euismod, turpis enim finibus lectus, viverra dapibus purus nisl in purus. Vivamus lacus magna, finibus id porta id, sollicitudin eget sapien. Nunc nec sem id enim pretium eleifend ut eu metus. Nam pharetra finibus dui a finibus. Cras euismod tellus nec lectus ullamcorper, convallis malesuada diam ornare. Duis ac enim et nibh lobortis vulputate. Aliquam non urna non eros tempor posuere. Duis faucibus odio vel urna euismod pellentesque. Cras dapibus, nisl vitae dignissim consectetur, eros leo viverra libero, sit amet lobortis metus erat a quam. 102 | 103 | Nullam quis tempus dui. Phasellus tincidunt mi eget placerat commodo. Donec tincidunt nulla tellus, sed fermentum orci viverra ut. Morbi blandit a nisl sit amet lacinia. Nulla quis orci a ante consequat molestie. Integer gravida tristique magna vitae ultricies. Integer suscipit feugiat enim, non luctus mi finibus eget. Proin viverra ut justo in laoreet. Aenean ut purus aliquam, rhoncus lectus non, condimentum sapien. Cras porttitor ut mi eu lacinia. Vestibulum maximus dolor arcu, ut accumsan nisi lacinia rutrum. Fusce ut urna rutrum, gravida augue at, feugiat nisl. 104 | 105 | Duis nec laoreet odio, ut pretium tellus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec sed dui viverra, condimentum nibh in, bibendum dolor. Donec sit amet ipsum et augue ultricies semper non tincidunt nulla. Pellentesque dictum commodo ipsum a bibendum. Praesent dignissim iaculis augue, id dapibus nulla molestie non. Sed molestie convallis augue, a posuere nulla pulvinar eu. Praesent venenatis, odio non viverra suscipit, risus ligula placerat metus, non feugiat metus mauris sed magna. Aenean sed erat at justo semper bibendum ut at nunc. 106 | 107 | Pellentesque a urna vel lectus rhoncus imperdiet nec ac metus. Nullam rutrum placerat purus quis pharetra. In accumsan ex odio, vel maximus tortor rhoncus ut. Ut laoreet turpis sit amet blandit consectetur. Curabitur ultricies facilisis posuere. Mauris nec lorem ornare, mattis est ut, condimentum felis. Pellentesque tempor dui a maximus tempus. 108 | 109 | Suspendisse ultrices sed neque vitae scelerisque. Nam euismod mi eget orci condimentum, ac consectetur ante lobortis. Integer et rutrum purus, quis aliquam nunc. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Pellentesque elementum elementum orci, eu luctus tortor suscipit a. Sed mollis metus tincidunt, consequat turpis eu, dictum leo. Nam at bibendum elit. Praesent facilisis mauris a ipsum efficitur commodo. Duis finibus sagittis cursus. Curabitur mattis justo tellus, commodo eleifend diam blandit eu. Proin et interdum metus. Etiam ullamcorper venenatis fermentum. Donec sodales, sapien sit amet iaculis venenatis, lacus enim cursus sem, et pulvinar magna lectus ac leo. 110 | 111 | Aenean vitae mauris eu urna euismod vehicula sed ac libero. Pellentesque neque ex, suscipit ut mattis non, consectetur at urna. Morbi faucibus diam ante, ac consequat leo blandit a. Integer tincidunt convallis blandit. Quisque eu arcu at purus finibus commodo non et nisi. Pellentesque sed rutrum diam, non dictum velit. In sed volutpat sem. Proin dignissim lacinia eleifend. Nam sed lacus est. Aenean nec odio sem. Praesent scelerisque molestie nibh vitae volutpat. Vestibulum tempus magna ac tortor tincidunt varius. 112 | 113 | Ut tempus malesuada nunc ac tincidunt. Phasellus fermentum augue ipsum. Cras justo nibh, viverra sed hendrerit nec, interdum eget metus. Sed quis tempor turpis, vel faucibus dui. Fusce id odio fringilla, tincidunt neque vitae, fermentum leo. Donec auctor feugiat neque non placerat. Praesent a rutrum mi. Maecenas facilisis tellus in nisi dictum convallis. -------------------------------------------------------------------------------- /tf-save-load/results/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-save-load/results/.gitkeep -------------------------------------------------------------------------------- /tf-save-load/sc1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/metaflow-ai/blog/6a5c5dc3ffb7b61747dba5f4d3e8b818f2845a08/tf-save-load/sc1.png -------------------------------------------------------------------------------- /tf-shape/Readme.md: -------------------------------------------------------------------------------- 1 | # Shapes and dynamic dimensions in TensorFlow 2 | 3 | ## TL;DR 4 | ### EN 5 | 6 | ### FR -------------------------------------------------------------------------------- /tf-uat/README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow howto: a universal approximator inside a neural net 2 | 3 | ## TL, DR! 4 | ### EN 5 | - I implement a first universal approximator with TesnorFlow and i train it on a sinus function (I show that it actually works) 6 | - I use it inside a bigger neural networks to classify the MNIST dataset 7 | - I display the learnt activation functions 8 | - I show that whatever the learnt activation function is, i get consistently the accuracy 0.98 on the test set 9 | - bonus: all the code is open-source 10 | 11 | ### FR 12 | - J'implémente une fonction d'approximation universelle avec Tensorflow et je l'entraine sur une fonction Sinus (je montre que ça marche :) ) 13 | - Je l'intègre à l'intérieur d'un réseau de neurones plus larges pour classifier le dataset MNIST 14 | - Je présentes les fonctions d'activation ainsi apprises 15 | - Je montre que quelquesoit la fonction d'activation obtenu, j'obtiens toujours la même "accuracy" de 0.98 sur les données de test 16 | - En bonus, tous le code est open-source et commenté -------------------------------------------------------------------------------- /tf-uat/train.sh: -------------------------------------------------------------------------------- 1 | for i in {1..10}; do 2 | python3 universal_mnist.py 3 | done -------------------------------------------------------------------------------- /tf-uat/universal.py: -------------------------------------------------------------------------------- 1 | import time, os, argparse, io 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | 6 | import matplotlib 7 | matplotlib.use('Agg') 8 | import matplotlib.pyplot as plt 9 | 10 | dir = os.path.dirname(os.path.realpath(__file__)) 11 | 12 | # Note: elu is not bounded, yet it works 13 | def univAprox(x, N=50, phi=tf.nn.elu, reuse=False): # First trick: the reuse capacity 14 | with tf.variable_scope('UniversalApproximator', reuse=reuse): 15 | x = tf.expand_dims(x, -1) 16 | 17 | # Second trick: using convolutions! 18 | aW_1 = tf.get_variable('aW_1', shape=[1, 1, N], initializer=tf.random_normal_initializer(stddev=.1)) 19 | ab_1 = tf.get_variable('ab_1', shape=[N], initializer=tf.constant_initializer(0.)) 20 | z = tf.nn.conv1d(x, aW_1, stride=1, padding='SAME') + ab_1 21 | a = phi(z) 22 | 23 | aW_2 = tf.get_variable('aW_2', shape=[1, N, 1], initializer=tf.random_normal_initializer(stddev=.1)) 24 | z = tf.nn.conv1d(a, aW_2, stride=1, padding='SAME') 25 | 26 | out = tf.squeeze(z, [-1]) 27 | return out 28 | 29 | def func_to_approx(x): 30 | return tf.sin(x) 31 | 32 | 33 | if __name__ == '__main__': 34 | parser = argparse.ArgumentParser() 35 | parser.add_argument("--nb_neurons", default=50, type=int, help="Number of neurons") 36 | args = parser.parse_args() 37 | 38 | with tf.variable_scope('Graph') as scope: 39 | # Graph 40 | x = tf.placeholder(tf.float32, shape=[None, 1], name="x") 41 | y_true = func_to_approx(x) 42 | y = univAprox(x, args.nb_neurons) 43 | loss = tf.reduce_mean(tf.square(y - y_true)) 44 | loss_summary_t = tf.summary.scalar('loss', loss) 45 | adam = tf.train.AdamOptimizer(learning_rate=1e-2) 46 | train_op = adam.minimize(loss) 47 | 48 | # Plot graph 49 | img_strbuf_plh = tf.placeholder(tf.string, shape=[]) 50 | my_img = tf.image.decode_png(img_strbuf_plh, 4) 51 | img_summary_t = tf.summary.image('img', tf.expand_dims(my_img, 0)) 52 | 53 | saver = tf.train.Saver() 54 | with tf.Session() as sess: 55 | result_folder = dir + '/results/' + str(int(time.time())) 56 | sw = tf.summary.FileWriter(result_folder, sess.graph) 57 | 58 | print('Training our universal approximator') 59 | sess.run(tf.global_variables_initializer()) 60 | for i in range(2000): 61 | x_in = np.random.uniform(-10, 10, [100000, 1]) 62 | 63 | current_loss, loss_summary, _ = sess.run([loss, loss_summary_t, train_op], feed_dict={ 64 | x: x_in 65 | }) 66 | sw.add_summary(loss_summary, i + 1) 67 | 68 | if (i + 1) % 100 == 0: 69 | print('batch: %d, loss: %f' % (i + 1, current_loss)) 70 | 71 | print('Plotting graphs') 72 | inputs = np.array([ [(i - 1000) / 100] for i in range(2000) ]) 73 | y_true_res, y_res = sess.run([y_true, y], feed_dict={ 74 | x: inputs 75 | }) 76 | plt.figure(1) 77 | plt.subplot(211) 78 | plt.plot(inputs, y_true_res.flatten()) 79 | plt.subplot(212) 80 | plt.plot(inputs, y_res) 81 | imgdata = io.BytesIO() 82 | plt.savefig(imgdata, format='png') 83 | imgdata.seek(0) 84 | img_summary = sess.run(img_summary_t, feed_dict={ 85 | img_strbuf_plh: imgdata.getvalue() 86 | }) 87 | sw.add_summary(img_summary, i + 1) 88 | plt.clf() 89 | 90 | # Saving the graph 91 | saver.save(sess, result_folder + '/data.chkp') 92 | -------------------------------------------------------------------------------- /tf-uat/universal_mnist.py: -------------------------------------------------------------------------------- 1 | import time, os, argparse, io 2 | 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | import numpy as np 6 | 7 | import matplotlib 8 | matplotlib.use('Agg') 9 | import matplotlib.pyplot as plt 10 | 11 | from universal import univAprox 12 | 13 | dir = os.path.dirname(os.path.realpath(__file__)) 14 | mnist = input_data.read_data_sets('MNIST_data', one_hot=True) 15 | 16 | # HyperParam 17 | parser = argparse.ArgumentParser() 18 | parser.add_argument("--nb_neurons", default=50, type=int, help="Number of neurons") 19 | args = parser.parse_args() 20 | 21 | with tf.variable_scope('Graph') as scope: 22 | # Graph 23 | x = tf.placeholder(tf.float32, shape=[None, 784], name="x") 24 | y_true = tf.placeholder(tf.float32, shape=[None, 10], name="y_true") 25 | 26 | W1 = tf.get_variable('W1', shape=[784, 200], initializer=tf.random_normal_initializer(stddev=1e-1)) 27 | b1 = tf.get_variable('b1', shape=[200], initializer=tf.constant_initializer(0.1)) 28 | z = tf.matmul(x, W1) + b1 29 | a = univAprox(z, args.nb_neurons) 30 | 31 | W2 = tf.get_variable('W2', shape=[200, 50], initializer=tf.random_normal_initializer(stddev=1e-1)) 32 | b2 = tf.get_variable('b2', shape=[50], initializer=tf.constant_initializer(0.1)) 33 | z = tf.matmul(a, W2) + b2 34 | a = univAprox(z, args.nb_neurons, reuse=True) 35 | 36 | W_s = tf.get_variable('W_s', shape=[50, 10], initializer=tf.random_normal_initializer(stddev=1e-1)) 37 | b_s = tf.get_variable('b_s', shape=[10], initializer=tf.constant_initializer(0.1)) 38 | logits = tf.matmul(a, W_s) + b_s 39 | 40 | loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(None, y_true, logits)) 41 | tf.summary.scalar('loss', loss) # Graph the loss 42 | adam = tf.train.AdamOptimizer(learning_rate=1e-3) 43 | train_op = adam.minimize(loss) 44 | 45 | # We merge summaries before the accuracy summary to avoid 46 | # graphing the accuracy with training data 47 | summaries = tf.summary.merge_all() 48 | 49 | correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(y_true, 1)) 50 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 51 | acc_summary = tf.summary.scalar('accuracy', accuracy) 52 | 53 | # Plot graph 54 | plot_x = tf.placeholder(tf.float32, shape=[None, 1], name="plot_x") 55 | plot_y = univAprox(plot_x, args.nb_neurons, reuse=True) 56 | img_strbuf_plh = tf.placeholder(tf.string, shape=[]) 57 | my_img = tf.image.decode_png(img_strbuf_plh, 4) 58 | img_summary = tf.summary.image( 59 | 'matplotlib_graph' 60 | , tf.expand_dims(my_img, 0) 61 | ) 62 | 63 | saver = tf.train.Saver() 64 | with tf.Session() as sess: 65 | result_folder = dir + '/results/' + str(int(time.time())) 66 | sess.run(tf.global_variables_initializer()) 67 | sw = tf.summary.FileWriter(result_folder, sess.graph) 68 | 69 | print('Training') 70 | for i in range(20000): 71 | batch = mnist.train.next_batch(200) 72 | current_loss, summary, _ = sess.run([loss, summaries, train_op], feed_dict={ 73 | x: batch[0], 74 | y_true: batch[1] 75 | }) 76 | sw.add_summary(summary, i + 1) 77 | 78 | if (i + 1) % 1000 == 0: 79 | acc, acc_sum = sess.run([accuracy, acc_summary], feed_dict={ 80 | x: mnist.test.images, 81 | y_true: mnist.test.labels 82 | }) 83 | sw.add_summary(acc_sum, i + 1) 84 | print('batch: %d, loss: %f, accuracy: %f' % (i + 1, current_loss, acc)) 85 | 86 | print('Plotting approximated function graph') 87 | inputs = np.array([ [(i - 500) / 100] for i in range(1000) ]) 88 | plot_y_res = sess.run(plot_y, feed_dict={ 89 | plot_x: inputs 90 | }) 91 | plt.figure(1) 92 | plt.plot(inputs, plot_y_res) 93 | imgdata = io.BytesIO() 94 | plt.savefig(imgdata, format='png') 95 | imgdata.seek(0) 96 | plot_img_summary = sess.run(img_summary, feed_dict={ 97 | img_strbuf_plh: imgdata.getvalue() 98 | }) 99 | sw.add_summary(plot_img_summary, i + 1) 100 | plt.clf() 101 | 102 | # Saving the graph 103 | saver.save(sess, result_folder + '/data.chkp') --------------------------------------------------------------------------------