├── README.md
└── SNE.ipynb


/README.md:
--------------------------------------------------------------------------------
 1 | # requirements
 2 | tensorflow = 1.2 (on gpu)
 3 | matplotlib
 4 | numpy
 5 | 
 6 | # SNE.ipynb
 7 | Derive the derivative of cost function in SNE  
 8 | [https://www.cs.nyu.edu/~roweis/papers/sne_final.pdf](https://www.cs.nyu.edu/~roweis/papers/sne_final.pdf)
 9 | 
10 | # t-SNE.ipynb
11 | Implementation with tensorflow on **MNIST_data**  
12 | [http://www.cs.toronto.edu/~hinton/absps/tsne.pdf](http://www.cs.toronto.edu/~hinton/absps/tsne.pdf)
13 | 
14 | 
15 | 


--------------------------------------------------------------------------------
/SNE.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# SNE"
  8 |    ]
  9 |   },
 10 |   {
 11 |    "cell_type": "markdown",
 12 |    "metadata": {},
 13 |    "source": [
 14 |     "$$ C=\\sum_i KL(P_i | Q_i) = \\sum_i \\sum_j p_{j|i}\\log\\frac{p_{j|i}}{q_{j|i}}$$\n",
 15 |     "\n",
 16 |     "$$q_{j|i} = \\frac{\\exp(-||y_i-y_j||^2)}{\\sum_{k}\\exp(-||y_i-y_k||^2}$$"
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "markdown",
 21 |    "metadata": {},
 22 |    "source": [
 23 |     "## 1.\n",
 24 |     "$$z_{j,i}=z_{i,j}=\\exp(-||y_i-y_j||)^2$$\n",
 25 |     "$$\\frac{\\partial z_{i,j}}{\\partial y_i}=-2(y_i-y_j)z_{i,j}$$"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "markdown",
 30 |    "metadata": {},
 31 |    "source": [
 32 |     "## 2.\n",
 33 |     "\n",
 34 |     "### 2.1\n",
 35 |     "\n",
 36 |     "$$q_{k|j}=\\frac{z_{k,j}}{\\sum_t z_{j,t}}$$\n",
 37 |     "$$\\frac{\\partial }{\\partial y_i}\\log q_{k|j}=2q_{i|j}(y_i-y_j)$$\n",
 38 |     "\n",
 39 |     "### 2.2\n",
 40 |     "$$q_{i|j}=\\frac{z_{i,j}}{\\sum_t z_{j,t}}$$\n",
 41 |     "$$\\frac{\\partial }{\\partial y_i}\\log q_{i|j}=2(y_i-y_j)(q_{i|j}-1)$$\n",
 42 |     "\n",
 43 |     "### 2.3\n",
 44 |     "$$q_{j|i}=\\frac{z_{i,j}}{\\sum_t z_{i,t}}$$\n",
 45 |     "$$\\frac{\\partial }{\\partial y_i}\\log q_{j|i}=-2\\big{(}(y_i-y_j)+\\sum_t (y_i-y_t)q_{t|i}\\big{)}$$"
 46 |    ]
 47 |   },
 48 |   {
 49 |    "cell_type": "markdown",
 50 |    "metadata": {},
 51 |    "source": [
 52 |     "## 3.\n",
 53 |     "$$ \\frac{\\partial C}{\\partial y_i}= -\\frac{\\partial }{\\partial y_i} \\sum_j \\sum_k p_{k|j}\\log q_{k|j}$$\n",
 54 |     "$$=-\\frac{\\partial }{\\partial y_i} \\sum_{j, k\\neq i} p_{k|j}\\log q_{k|j}-\\frac{\\partial }{\\partial y_i} \\sum_{j} p_{i|j}\\log q_{i|j}-\\frac{\\partial }{\\partial y_i} \\sum_{j} p_{j|i}\\log q_{j|i}$$\n",
 55 |     "\n",
 56 |     "\n",
 57 |     "\n",
 58 |     "$$=-\\sum_{j, k\\neq i} 2p_{k|j}q_{i|j}(y_i-y_j)\n",
 59 |     "-\\sum_j 2p_{i|j}(q_{i,j}-1)(y_i-y_j)\n",
 60 |     "+\\sum_j 2p_{j|i}\\big{(}(y_i-y_j)-\\sum_t (y_i-y_t)q_{t|i}\\big{)}$$\n",
 61 |     "\n",
 62 |     "$$=-\\sum_{j \\neq i} 2(1-p_{i|j})q_{i|j}(y_i-y_j)\n",
 63 |     "-\\sum_j 2p_{i|j}(q_{i,j}-1)(y_i-y_j)\n",
 64 |     "+\\sum_j 2p_{j|i}(y_i-y_j)-\\sum_j\\sum_t p_{j|i}(y_i-y_t)q_{t|i}$$\n",
 65 |     "\n",
 66 |     "$$=-\\sum_{j} 2q_{i|j}(y_i-y_j)\n",
 67 |     "+\\sum_j 2p_{i|j}(y_i-y_j)\n",
 68 |     "+\\sum_j 2p_{j|i}(y_i-y_j)-\\sum_t\\sum_j p_{j|i}(y_i-y_t)q_{t|i}$$\n",
 69 |     "\n",
 70 |     "$$=-\\sum_{j} 2q_{i|j}(y_i-y_j)\n",
 71 |     "+\\sum_j 2p_{i|j}(y_i-y_j)\n",
 72 |     "+\\sum_j 2p_{j|i}(y_i-y_j)-\\sum_t(y_i-y_t)q_{t|i}$$\n",
 73 |     "\n",
 74 |     "$$=-\\sum_{j} 2q_{i|j}(y_i-y_j)\n",
 75 |     "+\\sum_j 2p_{i|j}(y_i-y_j)\n",
 76 |     "+\\sum_j 2p_{j|i}(y_i-y_j)-\\sum_j(y_i-y_j)q_{j|i}$$\n",
 77 |     "\n",
 78 |     "$$=-2\\sum_{j} \\big{(}p_{i|j}-q_{i|j}+p_{j|i}-q_{j|i}\\big{)}(y_i-y_j)$$"
 79 |    ]
 80 |   }
 81 |  ],
 82 |  "metadata": {
 83 |   "kernelspec": {
 84 |    "display_name": "Python 3",
 85 |    "language": "python",
 86 |    "name": "python3"
 87 |   },
 88 |   "language_info": {
 89 |    "codemirror_mode": {
 90 |     "name": "ipython",
 91 |     "version": 3
 92 |    },
 93 |    "file_extension": ".py",
 94 |    "mimetype": "text/x-python",
 95 |    "name": "python",
 96 |    "nbconvert_exporter": "python",
 97 |    "pygments_lexer": "ipython3",
 98 |    "version": "3.6.1"
 99 |   }
100 |  },
101 |  "nbformat": 4,
102 |  "nbformat_minor": 2
103 | }
104 | 


--------------------------------------------------------------------------------