├── .DS_Store ├── .ipynb_checkpoints └── RankNet-checkpoint.ipynb ├── README.md ├── RankNet.ipynb ├── assets └── siamese-net.png ├── base.png └── meta.png /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eggie5/RankNet/c814f6ab0fd6bc638c224c4a967a9a455891b3a5/.DS_Store -------------------------------------------------------------------------------- /.ipynb_checkpoints/RankNet-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# RankNet\n", 8 | "\n", 9 | "This is the generic implementation of RankNet on a synthetic dataset.\n", 10 | "\n", 11 | "This model is applied to hotel imags in the other notebook.\n", 12 | "\n", 13 | "![alt text](assets/siamese-net.png \"Title\")" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 1, 19 | "metadata": {}, 20 | "outputs": [ 21 | { 22 | "name": "stderr", 23 | "output_type": "stream", 24 | "text": [ 25 | "Using TensorFlow backend.\n" 26 | ] 27 | }, 28 | { 29 | "name": "stdout", 30 | "output_type": "stream", 31 | "text": [ 32 | "Populating the interactive namespace from numpy and matplotlib\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "import numpy as np\n", 38 | "\n", 39 | "import keras\n", 40 | "from keras import backend\n", 41 | "from keras.layers import Activation, Add, Dense, Input, Lambda, Dropout, Subtract\n", 42 | "from keras.models import Model, Sequential\n", 43 | "from keras.utils import plot_model\n", 44 | "%pylab inline\n", 45 | "\n", 46 | "INPUT_DIM = 1536" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "## Base Network\n", 54 | "\n", 55 | "This is a scoring function (regression) that takes an images vector as input and outputs a scalar number. The Siamese model create two of these base networks to for higher and lower ranking images respectively." 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 2, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "def create_base_network(input_dim):\n", 65 | " '''Base network to be shared (eq. to feature extraction).\n", 66 | " '''\n", 67 | " seq = Sequential()\n", 68 | " seq.add(Dense(INPUT_DIM, input_shape=(input_dim,), activation='relu'))\n", 69 | " seq.add(Dropout(0.1))\n", 70 | " seq.add(Dense(64, activation='relu'))\n", 71 | " seq.add(Dropout(0.1))\n", 72 | " seq.add(Dense(32, activation='relu'))\n", 73 | " seq.add(Dense(1))\n", 74 | " return seq" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## Siamse Archecture\n", 82 | "\n", 83 | "We create two instances of the base archecture and take the difference between their scalar outputs. We then pass the difference between a sigmoid function so that we can use binary-cross-entropy for the differntiable loss function." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 3, 89 | "metadata": { 90 | "scrolled": false 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "def create_meta_network(input_dim, base_network):\n", 95 | " input_a = Input(shape=(input_dim,))\n", 96 | " input_b = Input(shape=(input_dim,))\n", 97 | "\n", 98 | " rel_score = base_network(input_a)\n", 99 | " irr_score = base_network(input_b)\n", 100 | "\n", 101 | " # subtract scores\n", 102 | " diff = Subtract()([rel_score, irr_score])\n", 103 | "\n", 104 | " # Pass difference through sigmoid function.\n", 105 | " prob = Activation(\"sigmoid\")(diff)\n", 106 | "\n", 107 | " # Build model.\n", 108 | " model = Model(inputs = [input_a, input_b], outputs = prob)\n", 109 | " model.compile(optimizer = \"adam\", loss = \"binary_crossentropy\")\n", 110 | "\n", 111 | " return model" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "__________________________________________________________________________________________________\n", 124 | "Layer (type) Output Shape Param # Connected to \n", 125 | "==================================================================================================\n", 126 | "input_7 (InputLayer) (None, 1536) 0 \n", 127 | "__________________________________________________________________________________________________\n", 128 | "input_8 (InputLayer) (None, 1536) 0 \n", 129 | "__________________________________________________________________________________________________\n", 130 | "sequential_4 (Sequential) (None, 1) 2461313 input_7[0][0] \n", 131 | " input_8[0][0] \n", 132 | "__________________________________________________________________________________________________\n", 133 | "subtract_4 (Subtract) (None, 1) 0 sequential_4[1][0] \n", 134 | " sequential_4[2][0] \n", 135 | "__________________________________________________________________________________________________\n", 136 | "activation_4 (Activation) (None, 1) 0 subtract_4[0][0] \n", 137 | "==================================================================================================\n", 138 | "Total params: 2,461,313\n", 139 | "Trainable params: 2,461,313\n", 140 | "Non-trainable params: 0\n", 141 | "__________________________________________________________________________________________________\n" 142 | ] 143 | } 144 | ], 145 | "source": [ 146 | "base_network = create_base_network(INPUT_DIM)\n", 147 | "model = create_meta_network(INPUT_DIM, base_network)\n", 148 | "model.summary()\n", 149 | "\n", 150 | "plot_model(base_network, to_file='base.png')\n", 151 | "plot_model(model, to_file='meta.png')" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "### Base Network\n", 159 | "\n", 160 | "This can be any (scoring) function which outputs a scalar:\n", 161 | "![base](base.png)\n", 162 | "\n", 163 | "### Meta Network\n", 164 | "\n", 165 | "This takes an instance of the base network and evalutes the pair on it:\n", 166 | "![meta](meta.png)" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "## Synthetic Dataset\n", 174 | "\n", 175 | "We generate two N dimensional datasets from an uniform distribution where one distribution is shift by a constant. This can be thought of as a good image and a less good image that we want to learn to differentiate." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 28, 181 | "metadata": {}, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "text/plain": [ 186 | "(0.0023731553517547066, -0.00052067296140614276)" 187 | ] 188 | }, 189 | "execution_count": 28, 190 | "metadata": {}, 191 | "output_type": "execute_result" 192 | } 193 | ], 194 | "source": [ 195 | "# Fake data.\n", 196 | "N = 1000\n", 197 | "SPLIT=750\n", 198 | "X_1 = 2* np.random.randn(N, INPUT_DIM) # notice this data is shifted by 2\n", 199 | "X_2 = np.random.randn(N, INPUT_DIM)\n", 200 | "\n", 201 | "X_1_train = X_1[0:SPLIT,:]\n", 202 | "X_1_test = X_1[SPLIT:N,:]\n", 203 | "\n", 204 | "X_2_train = X_2[0:SPLIT,:]\n", 205 | "X_2_test = X_2[SPLIT:N,:]\n", 206 | "\n", 207 | "\n", 208 | "y = np.ones((X_1.shape[0], 1))\n", 209 | "\n", 210 | "y_train = y[0:SPLIT,:]\n", 211 | "y_test = y[SPLIT:N,:]\n", 212 | "\n", 213 | "np.mean(X_1_train), np.mean(X_2_train)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": 29, 219 | "metadata": { 220 | "scrolled": false 221 | }, 222 | "outputs": [ 223 | { 224 | "name": "stdout", 225 | "output_type": "stream", 226 | "text": [ 227 | "Train on 750 samples, validate on 250 samples\n", 228 | "Epoch 1/10\n", 229 | "750/750 [==============================] - 7s 9ms/step - loss: 0.0460 - val_loss: 1.1323e-04\n", 230 | "Epoch 2/10\n", 231 | "750/750 [==============================] - 5s 7ms/step - loss: 0.0045 - val_loss: 3.0593e-04\n", 232 | "Epoch 3/10\n", 233 | "750/750 [==============================] - 6s 8ms/step - loss: 8.4689e-04 - val_loss: 8.8093e-04\n", 234 | "Epoch 00003: early stopping\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "# Train model.\n", 240 | "\n", 241 | "es=keras.callbacks.EarlyStopping(monitor='val_loss',\n", 242 | " min_delta=0,\n", 243 | " patience=2,\n", 244 | " verbose=1, mode='auto')\n", 245 | "\n", 246 | "NUM_EPOCHS = 10\n", 247 | "BATCH_SIZE = 10\n", 248 | "history = model.fit([X_1_train, X_2_train], y_train, \n", 249 | " validation_data=([X_1_test, X_2_test], y_test), \n", 250 | " batch_size = BATCH_SIZE, epochs = NUM_EPOCHS, verbose = 1, callbacks=[es])" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": 33, 256 | "metadata": {}, 257 | "outputs": [ 258 | { 259 | "data": { 260 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXd//H3NzsJSSAL+5IgWGRRkIBrW1u1olbRahUV\nBaVF29rapz5UfdRWbftUf+2j1qUqCgWXiora0qp1qdrNjYQdQQkEJCBLAiSQkJDl/v0xJziEJDOB\nTE4y+byuay7OnHOfmU8Ok/PNue8z55hzDhERkZbE+B1AREQ6PhULEREJScVCRERCUrEQEZGQVCxE\nRCQkFQsREQlJxULkCJnZXDP7ZZhtN5jZGUf6OiLtTcVCRERCUrEQEZGQVCykS/C6f2aa2XIzqzCz\n2WbW28xeM7M9ZvaWmfUMan++ma0ys91m9q6ZHRO0bKyZLfbWew5IavRe3zSzpd6675nZsYeZ+btm\nVmhmO81soZn18+abmd1nZtvNrNzMVpjZKG/ZOWb2sZdts5n992FtMJFGVCykK7kIOBM4GjgPeA34\nHyCbwO/CjwDM7GjgWeDH3rJXgb+YWYKZJQB/Ap4CMoAXvNfFW3csMAe4FsgEHgMWmllia4Ka2deB\nXwOXAH2BjcB8b/E3gK94P0e616bUWzYbuNY5lwqMAt5uzfuKNEfFQrqSB51z25xzm4F/AR8655Y4\n56qAl4GxXrtLgVecc28652qA3wLdgJOBE4F44H7nXI1zbgGwKOg9ZgCPOec+dM7VOefmAdXeeq1x\nBTDHObfYOVcN3AKcZGY5QA2QCgwHzDm32jn3ubdeDTDCzNKcc7ucc4tb+b4iTVKxkK5kW9D0viae\nd/em+xH4Sx4A51w9sAno7y3b7A6+AufGoOnBwI1eF9RuM9sNDPTWa43GGfYSOHro75x7G3gIeBjY\nbmazzCzNa3oRcA6w0cz+YWYntfJ9RZqkYiFyqC0EdvpAYIyAwA5/M/A50N+b12BQ0PQm4FfOuR5B\nj2Tn3LNHmCGFQLfWZgDn3APOuXHACALdUTO9+Yucc5OAXgS6y55v5fuKNEnFQuRQzwPnmtnpZhYP\n3EigK+k94H2gFviRmcWb2beACUHrPg5cZ2YneAPRKWZ2rpmltjLDs8DVZjbGG+/4XwLdZhvMbLz3\n+vFABVAF1HtjKleYWbrXfVYO1B/BdhA5QMVCpBHn3CfAFOBBoITAYPh5zrn9zrn9wLeAacBOAuMb\nLwWtmw98l0A30S6g0Gvb2gxvAbcDLxI4mjkKmOwtTiNQlHYR6KoqBX7jLbsS2GBm5cB1BMY+RI6Y\n6eZHIiISio4sREQkJBULEREJScVCRERCUrEQEZGQ4vwO0FaysrJcTk6O3zFERDqVgoKCEudcdqh2\nUVMscnJyyM/P9zuGiEinYmYbQ7dSN5SIiIRBxUJEREJSsRARkZCiZsxCRORw1NTUUFxcTFVVld9R\nIiopKYkBAwYQHx9/WOurWIhIl1ZcXExqaio5OTkcfDHh6OGco7S0lOLiYnJzcw/rNdQNJSJdWlVV\nFZmZmVFbKADMjMzMzCM6elKxEJEuL5oLRYMj/Rm7fLGoqK7ljoWrKNtX43cUEZEOq8sXizVby3n6\ng4384JnF1NTpPjEi0r52797N73//+1avd84557B79+4IJGpaly8W4wZn8L/fGs2/C0v4+cJV6P4e\nItKemisWtbW1La736quv0qNHj0jFOoTOhgIuyRtIUUkFj7y7jiFZKXzny0P8jiQiXcTNN9/MunXr\nGDNmDPHx8SQlJdGzZ0/WrFnDp59+ygUXXMCmTZuoqqrihhtuYMaMGcAXlzjau3cvZ599Nqeeeirv\nvfce/fv3589//jPdunVr05wqFp6Z3/gSG0oq+NWrqxmUkcw3RvbxO5KItLM7/7KKj7eUt+lrjuiX\nxs/PG9ns8rvvvpuVK1eydOlS3n33Xc4991xWrlx54BTXOXPmkJGRwb59+xg/fjwXXXQRmZmZB73G\n2rVrefbZZ3n88ce55JJLePHFF5kyZUqb/hxdvhuqQUyMce8lYzi2fzo3zF/Kys1lfkcSkS5owoQJ\nB30X4oEHHuC4447jxBNPZNOmTaxdu/aQdXJzcxkzZgwA48aNY8OGDW2eS0cWQbolxPL41DwufPg9\nps9bxJ9+cAp909v2UE5EOq6WjgDaS0pKyoHpd999l7feeov333+f5ORkTjvttCa/K5GYmHhgOjY2\nln379rV5Lh1ZNNIrNYnZ0/KoqK5j+tx8KqpbHmQSETkSqamp7Nmzp8llZWVl9OzZk+TkZNasWcMH\nH3zQzum+oGLRhOF90njw8rGs2VrODfOXUFevM6REJDIyMzM55ZRTGDVqFDNnzjxo2cSJE6mtreWY\nY47h5ptv5sQTT/QpJVi0nCqal5fn2vrmR0++v4Gf/XkV00/N5fZvjmjT1xaRjmH16tUcc8wxfsdo\nF039rGZW4JzLC7WuxixacNVJOazfUcHsfxeRk5XClScO9juSiIgvVCxCuP2bI9hYWsEdC1cxKCOZ\nrx4d8la1IiJRR2MWIcTGGA9efjzDenXn+mcW8+m2pgeiRESimYpFGLonxjF72niSEmK5+g+L2LGn\n2u9IIiLtSsUiTP17dOOJq/IorahmxlP5VNXU+R1JRKTdqFi0wnEDe3D/pWNY8tlu/vuFZdTrlFoR\n6SJULFpp4qi+3Hz2cP66/HPue+tTv+OISCd3uJcoB7j//vuprKxs40RNU7E4DNd+ZQiX5g3kwbcL\nebGg2O84ItKJdZZioVNnD4OZ8YsLRrFpVyU3v7ScAT27ccKQzNAriog0EnyJ8jPPPJNevXrx/PPP\nU11dzYUXXsidd95JRUUFl1xyCcXFxdTV1XH77bezbds2tmzZwte+9jWysrJ45513IppTxeIwJcTF\n8MgV47jwkf9w7dMFvPz9U8jNSgm9ooh0XK/dDFtXtO1r9hkNZ9/d7OLgS5S/8cYbLFiwgI8++gjn\nHOeffz7//Oc/2bFjB/369eOVV14BAteMSk9P59577+Wdd94hKyurbTM3Qd1QRyA9OZ4/TBuPAdfM\nXcTuyv1+RxKRTuyNN97gjTfeYOzYsRx//PGsWbOGtWvXMnr0aN58801uuukm/vWvf5Gent7u2XRk\ncYQGZ6Yw66o8rnj8Q659qoCnpp9AQpxqsEin1MIRQHtwznHLLbdw7bXXHrJs8eLFvPrqq9x2222c\nfvrp/OxnP2vXbNqrtYHxORncc/FoPizaya0vr9B9vEUkbMGXKD/rrLOYM2cOe/fuBWDz5s1s376d\nLVu2kJyczJQpU5g5cyaLFy8+ZN1I05FFG7lw7ACKSip54O9ryc1O4funDfU7koh0AsGXKD/77LO5\n/PLLOemkkwDo3r07Tz/9NIWFhcycOZOYmBji4+N55JFHAJgxYwYTJ06kX79+ER/gjuglys1sIvA7\nIBZ4wjl3d6PlicCTwDigFLjUObchaPkg4GPgDufcb1t6r0hcory1nHPcMH8pC5dt4fdXHM85o/v6\nmkdEQtMlysO7RHnEuqHMLBZ4GDgbGAFcZmaNbwoxHdjlnBsK3Afc02j5vcBrkcrY1syM/3fxsRw/\nqAf/9dxSlm7a7XckEZE2EckxiwlAoXNuvXNuPzAfmNSozSRgnje9ADjdzAzAzC4AioBVEczY5pLi\nY3n8qjx6pSXynXn5FO9qny/MiIhEUiSLRX9gU9DzYm9ek22cc7VAGZBpZt2Bm4A7W3oDM5thZvlm\nlr9jx442C36kMrsnMmfqeKprA/fx3lNV43ckEWlBVzgp5Uh/xo56NtQdwH3Oub0tNXLOzXLO5Tnn\n8rKzO9ZNiYb1TuWRK8ZRuGMv1/9xCbV19X5HEpEmJCUlUVpaGtUFwzlHaWkpSUlJh/0akTwbajMw\nMOj5AG9eU22KzSwOSCcw0H0CcLGZ/T+gB1BvZlXOuYcimLfNnTosi19eMIpbXlrBnX/5mLsmjcTr\nZRORDmLAgAEUFxfTkXonIiEpKYkBAwYc9vqRLBaLgGFmlkugKEwGLm/UZiEwFXgfuBh42wXK+5cb\nGpjZHcDezlYoGlw2YRBFJRXM+ud6hmSncPUpuX5HEpEg8fHx5Obq9zKUiBUL51ytmV0PvE7g1Nk5\nzrlVZnYXkO+cWwjMBp4ys0JgJ4GCEnVumjicopIKfvHXjxmcmczXh/f2O5KISKtE9HsW7akjfM+i\nJZX7a/n2o++zoaSCF647mRH90vyOJCLi//cs5GDJCXHMnjqe1KR4ps9bxPbyKr8jiYiETcWiHfVJ\nT+KJqXmU7ath+rx8KvfX+h1JRCQsKhbtbFT/dB6YPJaVW8r4r+eW6j7eItIpqFj44IwRvbnt3BG8\nvmob97y+xu84IiIh6aqzPrnmlByKSvby2D/Wk5uZwuQJg/yOJCLSLBULn5gZd5w3ko2lldz2p5UM\nzEjmlKGRvzWiiMjhUDeUj+JiY3j4iuMZkp3CdU8XULi9fW5iIiLSWioWPktLimf21PEkxsVwzdx8\nSvdW+x1JROQQKhYdwMCMZGZdlcfW8iqufaqA6to6vyOJiBxExaKDOH5QT/7v28eRv3EXNy1YHtVX\nwBSRzkcD3B3Iecf1Y2NpBb9941Nys7pzwxnD/I4kIgKoWHQ4P/jaUNaXVHDfW5+Sk5XMpDGN7xcl\nItL+1A3VwZgZv/7WaCbkZjDzheXkb9jpdyQRERWLjigxLpbHpoyjX48kZjxVwGeluo+3iPhLxaKD\n6pmSwJxp46mrd1w99yPK9uk+3iLiHxWLDmxIdnceu3Icn+2s5PvPFFCj+3iLiE9ULDq4E4dk8r8X\njuY/haXc/qeVOqVWRHyhs6E6gW/nDWRDaQUPv7OOIdkpzPjKUX5HEpEuRsWik7jxzC9RVFLBr19b\nw+DMFM4a2cfvSCLShagbqpOIiTHuvWQMxw7owY/nL2VFcZnfkUSkC1Gx6ESS4mN5/KpxZKQkMH3e\nIj4v2+d3JBHpIlQsOpleqUnMnpZH5f46rpmbT0W17uMtIpGnYtEJDe+TxkOXj+WTreX86Nkl1Ok+\n3iISYSoWndRpX+rFneeP5O9rtvOrV1b7HUdEopzOhurErjwph/UlFcz5TxG5WclceVKO35FEJEqp\nWHRyt507go2lldzxl48ZmJHMaV/q5XckEYlC6obq5GJjjAcuG8vRvVO5/o9L+GSr7uMtIm1PxSIK\ndE+MY/bUPJITYrlm7iK276nyO5KIRBkViyjRr0c3Zk8dT2lFNTOeLKCqRvfxFpG2o2IRRUYPSOf+\nS8eyrHg3Nz6/jHqdUisibUTFIspMHNWHmycO55UVn3Pvm5/6HUdEooTOhopCM74yhKKSCh56p5Cc\nrBQuHjfA70gi0snpyCIKmRm/uGAUJx+VyS0vLeeD9aV+RxKRTk7FIkrFx8bwyBXjGJSRzLVPFbB+\nx16/I4lIJxbRYmFmE83sEzMrNLObm1ieaGbPecs/NLMcb/4EM1vqPZaZ2YWRzBmt0pPjmTNtPLEx\nxvR5+eyq2O93JBHppCJWLMwsFngYOBsYAVxmZiMaNZsO7HLODQXuA+7x5q8E8pxzY4CJwGNmpvGV\nwzA4M4VZV45j8659XPt0AftrdR9vEWm9SB5ZTAAKnXPrnXP7gfnApEZtJgHzvOkFwOlmZs65Sudc\nw7W3kwCdA3oE8nIy+M23j+Wjop3c8tIK3cdbRFotksWiP7Ap6HmxN6/JNl5xKAMyAczsBDNbBawA\nrgsqHgeY2Qwzyzez/B07dkTgR4gek8b058dnDOPFxcX8/t11fscRkU6mww5wO+c+dM6NBMYDt5hZ\nUhNtZjnn8pxzednZ2e0fspO54fRhTBrTj9+8/gl/Xb7F7zgi0olEslhsBgYGPR/gzWuyjTcmkQ4c\ndJ6nc241sBcYFbGkXYSZcc9FxzJucE9ufH4ZSz7b5XckEekkIlksFgHDzCzXzBKAycDCRm0WAlO9\n6YuBt51zzlsnDsDMBgPDgQ0RzNplJMXHMuvKcfRKS+S7T+azaWel35FEpBOIWLHwxhiuB14HVgPP\nO+dWmdldZna+12w2kGlmhcBPgIbTa08FlpnZUuBl4PvOuZJIZe1qMrsn8odp46murWf6vEWUV9X4\nHUlEOjiLljNj8vLyXH5+vt8xOpX/FJYwdc5HnDw0izlT84iL7bBDWCISIWZW4JzLC9VOe4cu7JSh\nWfziglH889Md3PGXVTqlVkSapS+6dXGXTRjEhpIKHvvneoZkdeeaU3P9jiQiHZCKhXDTxOEUlVTw\ni1c+ZlBGMmeM6O13JBHpYNQNJcTEGPdPHsOofun8aP4SVm0p8zuSiHQwKhYCQHJCHE9MzSO9WzzT\n5+azrVz38RaRL6hYyAG905J4Ymoe5VU1TJ+3iMr9h1xhRUS6KBULOcjIfuk8eNlYPt5Szo/nL9V9\nvEUEULGQJpx+TG9uO3cEb3y8jXv+tsbvOCLSAehsKGnS1afkUOSdUpuTlcJlEwb5HUlEfKRiIU0y\nM35+3gg+21nJ7X9aycCeyZw6LMvvWCLiE3VDSbPiYmN46PKxHJXdne89U0Dh9j1+RxIRn6hYSItS\nk+KZPS2PxLgYrp67iNK91X5HEhEfqFhISAN6JvP4VXlsL69mxlMFVNXU+R1JRNqZioWEZeygntx7\nyRgKNu7ipwuW66KDIl2MioWE7dxj+zLzrC+xcNkW7n9rrd9xRKQd6WwoaZXvn3YU63dU8Lu/ryU3\nK4ULxvb3O5KItAMdWUirmBm//tZoTsjN4KcLlpO/YaffkUSkHahYSKslxMXw6JRx9O/ZjRlPFbCx\ntMLvSCISYWEVCzO7wczSLGC2mS02s29EOpx0XD1TEpgzbTz1znHN3EWUVeo+3iLRLNwji2ucc+XA\nN4CewJXA3RFLJZ1CblYKj04Zx2c7K/neMwXU1NX7HUlEIiTcYmHev+cATznnVgXNky7sxCGZ/Ppb\nx/LeulJue3mlTqkViVLhng1VYGZvALnALWaWCujPSAHg4nED2FBSwUPvFDIkO4Vrv3qU35FEpI2F\nWyymA2OA9c65SjPLAK6OXCzpbH5y5tEUlVZw99/WMDgzmYmj+vodSUTaULjdUCcBnzjndpvZFOA2\nQDdqlgNiYoz/+/ZxHDegBz9+binLi3f7HUlE2lC4xeIRoNLMjgNuBNYBT0YslXRKSfGxPH5VHpkp\niUyfl8+W3fv8jiQibSTcYlHrAiOXk4CHnHMPA6mRiyWdVXZqInOmjWff/jqumbuIvdW6j7dINAi3\nWOwxs1sInDL7ipnFAPGRiyWd2Zf6pPLwFcezdvtefvTsEup0H2+RTi/cYnEpUE3g+xZbgQHAbyKW\nSjq9rx6dzR3nj+TtNdv55Ssf+x1HRI5QWMXCKxDPAOlm9k2gyjmnMQtp0ZUnDuaaU3L5w3828OT7\nG/yOIyJHINzLfVwCfAR8G7gE+NDMLo5kMIkOt557DKcP78UdC1fx7ifb/Y4jIocp3G6oW4Hxzrmp\nzrmrgAnA7ZGLJdEiNsZ44LKxDO+TxvV/XMKareV+RxKRwxBusYhxzgX/WVjainWli0tJjGP2tDyS\nE2KZPjef7Xuq/I4kIq0U7g7/b2b2uplNM7NpwCvAq5GLJdGmb3o3Zk8dz86K/Xz3yQL27dd9vEU6\nk3AHuGcCs4Bjvccs59xNkQwm0Wf0gHTunzyG5cW7ufGFpdTrlFqRTiPsriTn3IvOuZ94j5fDWcfM\nJprZJ2ZWaGY3N7E80cye85Z/aGY53vwzzazAzFZ4/3493JzSsZ01sg+3nD2cV1ds5bdvfOJ3HBEJ\nU4sXEjSzPUBTf/4Z4JxzaS2sGws8DJwJFAOLzGyhcy74pPvpwC7n3FAzmwzcQ+A7HSXAec65LWY2\nCngd0M2eo8R3vzyEopIKfv/uOnKyUrgkb6DfkUQkhBaLhXPuSC7pMQEodM6tBzCz+QQuFxJcLCYB\nd3jTC4CHzMycc0uC2qwCuplZonOu+gjySAdhZtw1aRSf7azk1pdXMLBnMicdlel3LBFpQSTPaOoP\nbAp6XsyhRwcH2jjnaglcybbxXuMiYHFThcLMZphZvpnl79ixo82CS+TFx8bw+yvGMSgjmeueLmD9\njr1+RxKRFnTo01/NbCSBrqlrm1runJvlnMtzzuVlZ2e3bzg5Yund4vnDtAnExhjXzF3Eror9fkcS\nkWZEslhsBoI7owd485psY2ZxQDqB73BgZgOAl4GrnHPrIphTfDQoM5nHrxrHlrIqrn2qgOpanVIr\n0hFFslgsAoaZWa6ZJQCTgYWN2iwEpnrTFwNvO+ecmfUg8F2Om51z/4lgRukAxg3O4DcXH8tHG3Zy\ny0srdB9vkQ4oYsXCG4O4nsCZTKuB551zq8zsLjM732s2G8g0s0LgJ0DD6bXXA0OBn5nZUu/RK1JZ\nxX+TxvTnv844mpcWb+bhdwr9jiMijVi0/BWXl5fn8vPz/Y4hR8A5x0+eX8bLSzbz4GVjOe+4fn5H\nEol6ZlbgnMsL1a5DD3BL12Jm3H3RaMbn9OTGF5ZRsHGX35FExKNiIR1KYlwsj12ZR5+0JGY8mc+m\nnZV+RxIRVCykA8pISWDOtPHsr6vnmrmLKK+q8TuSSJenYiEd0tBe3Xl0yjiKSir4wTOLqa2r9zuS\nSJemYiEd1ilDs/jlBaP419oSfr5wlU6pFfFRi9eGEvHb5AmDKCqt4LF/rGdIdnemn5rrdySRLknF\nQjq8m84azsaSSn75yscMzkjmjBG9/Y4k0uWoG0o6vJgY475LxzC6fzo/mr+ElZvL/I4k0uWoWEin\n0C0hlieuyiO9WzzfmZfP1jLdx1ukPalYSKfRKy2J2VPHs6eqhunzFlFRXet3JJEuQ8VCOpUR/dJ4\n8PKxrP68nBvmL6VO9/EWaRcqFtLpfH14b27/5gjeWr2Nu19b7XcckS5BZ0NJpzTt5ByKSip4/F9F\n5GZ15/ITBvkdSSSqqVhIp2Rm/OybI9hYWsntf17JoIxkTh2W5XcskailbijptOJiY3jo8rEMze7O\n954pYO22PX5HEolaKhbSqaUmxTN7Wh6JcbFcM28RJXur/Y4kEpVULKTTG9AzmSem5rG9vJoZT+ZT\nVaP7eIu0NRULiQpjBvbgvkvHsPiz3cxcsFwXHRRpYyoWEjXOGd2Xn078En9ZtoX73lrrdxyRqKKz\noSSqfO+rR1G0o4IH/r6W3KxkLhw7wO9IIlFBRxYSVcyMX104mhOHZHDTghV8VLTT70giUUHFQqJO\nQlwMj04Zx4Ce3bj2qXw2lFT4HUmk01OxkKjUIzmB2dPG44Br5i6irFL38RY5EioWErVys1J4bMo4\nNu2q5LqnC9hfq/t4ixwuFQuJaicMyeTubx3L++tLuf1PK3VKrchh0tlQEvUuGjeADaUVPPh2IbnZ\nKVz31aP8jiTS6ahYSJfwX2ccTVFJBXe/toaczGQmjurrdySRTkXdUNIlxMQYv/32cYwd1IMfP7eU\nZZt2+x1JpFNRsZAuIyk+lllX5pHVPZHvPJnP5t37/I4k0mmoWEiXkp2ayJxp46naX8f0uYvYU6VT\nakXCoWIhXc7RvVN5+IrjWbt9Lz98dgm1dTqlViQUFQvpkr5ydDZ3nD+Sdz/ZwS9f0X28RULR2VDS\nZV154mA2lFQw+99F5GalMPXkHL8jiXRYKhbSpf3POcewsbSCO/+yikEZyXxteC+/I4l0SBHthjKz\niWb2iZkVmtnNTSxPNLPnvOUfmlmONz/TzN4xs71m9lAkM0rXFhtj/G7yWIb3SeP6Py5m9eflfkcS\n6ZAiVizMLBZ4GDgbGAFcZmYjGjWbDuxyzg0F7gPu8eZXAbcD/x2pfCINUhLjmD0tj+5JcUyfu4jt\ne6r8jiTS4UTyyGICUOicW++c2w/MByY1ajMJmOdNLwBONzNzzlU45/5NoGiIRFzf9G7MnjqeXZU1\nfHdePvv26z7eIsEiWSz6A5uCnhd785ps45yrBcqAzHDfwMxmmFm+meXv2LHjCONKVzeqfzq/mzyG\n5ZvL+MnzS6mv10UHRRp06lNnnXOznHN5zrm87Oxsv+NIFPjGyD7ces4xvLZyK7954xO/44h0GJE8\nG2ozMDDo+QBvXlNtis0sDkgHSiOYSSSk6afmsr6kgkfeXUduZgqXjB8YeiWRKBfJI4tFwDAzyzWz\nBGAysLBRm4XAVG/6YuBtpxsOiM/MjDvPH8mXh2XxPy+v4L11JX5HEvFdxIqFNwZxPfA6sBp43jm3\nyszuMrPzvWazgUwzKwR+Ahw4vdbMNgD3AtPMrLiJM6lEIiY+NoaHLj+enKwUrnuqgHU79vodScRX\nFi1/yOfl5bn8/Hy/Y0iU2bSzkgse/g/dk+J4+funkJGS4HckkTZlZgXOubxQ7Tr1ALdIpA3MSGbW\nVeP4vKyK654qoLpWp9RK16RiIRLCuMEZ/ObiY/low05ueXGF7uMtXZKuDSUShklj+rOxtJJ73/yU\n3KwUfnj6ML8jibQrFQuRMP3w60MpKqng/978lMFZKZx/XD+/I4m0G3VDiYTJzLj7otGMz+nJf7+w\njIKNu/yOJNJuVCxEWiExLpbHrsyjb3oSM57MZ9POSr8jibQLFQuRVspISWDOtPHU1NVz9dxFlO3T\nfbwl+qlYiByGo7K78+iV49hQUsH1f1xMje7jLVFOxULkMJ18VBa/unAU/1pbws8XrtIptRLVdDaU\nyBG4dPwgikoqefQf6xiSlcJ3vjzE70giEaFiIXKEfnrWl9hQUsGvXl3N4MwUzhzR2+9IIm1O3VAi\nRygmxrjv0jGM7p/Oj55dwsrNZX5HEmlzKhYibaBbQixPXJVHz+R4ps9bxNYy3RFYoouKhUgb6ZWW\nxOxp49lbVcv0eYuoqK71O5JIm1GxEGlDx/RN46HLj2f15+XcMH8pdbqPt0QJFQuRNva14b34+Xkj\neWv1Nn796mq/44i0CZ0NJRIBU0/Ooaikgif+XUROVgpTThzsdySRI6JiIRIht517DBtKK/j5wlV8\nVLSTvulJ9E4LPPqkJ9IrNYleaYkkxsX6HVUkJBULkQiJi43hwcvG8tMFy1myaRd/W1XN/tpDLwuS\nkZJAr9SE85PjAAANzUlEQVRE+qQn0Ts1id7pSfROS6RP2hfFJTMlgZgY8+GnEAlQsRCJoNSkeB6Z\nMg4A5xy7K2vYWl7FtvIqtpdXH5gOPKpZtaWckr3VNL5ySFyMkZ2a6BWPQCHplZYUVFAS6Z2eRGpi\nHGYqKtL2VCxE2omZ0TMlgZ4pCRzTN63ZdrV19ezYW8228mq2llWxfU+gmGwtq2b7nirW76jg/XWl\nlFcdemput/hY+qQnfXGkElxMvOKSnZpIUry6vqR1VCxEOpi42Bj6pnejb3o3GNh8u8r9tQcdnTQ+\nUlny2W62llc12fXVMzn+oELS1JFKZvdEYtX1JR4VC5FOKjkhjpysOHKyUppt45yjbF9D11d1oJCU\nVbFtzxdHKqs/D3R9Nf5KSGyMkd090L3VO+hIpfFRS1qSur66AhULkShmZvRITqBHcgLD+zTfrrau\nntKK/WwtO3gMpeFIZUNpBR8W7WzyRk9J8TGNjkwSg45akrxl6vrq7FQsRIS42JgDO/eW7Ntf542h\nBArJ9vKqQIHZU822siqWFe9ma1kV1U10ffVIjv/ibK+GwfpGRy1Z6vrqsFQsRCRs3RJiGZyZwuDM\nlru+yvfVel1dhx6pbC+v4tOte9i+p+qQrq8Yg+zUL8ZQmhpP6ZOWRFo3dX21NxULEWlTZkZ6cjzp\nyfEc3Tu12XZ19Y7SvdUHxlMaH6ls2lnJog072V3ZdNdX77TA91J6BX8npdGRirq+2o6KhYj4IjbG\n6OUdNbSkqqaO7eXVTR6pbCuvYuXmMt5avY2qmkO7vtK7xTcaQzn0SCWrewJxsbpMXigqFiLSoSXF\nxzIoM5lBmcnNtnHOUV5VGzgyCT7zK+hIpXB7Cdv3VB9yJeAYg6zuhxaUA0cq3vP0bvFduutLxUJE\nOj0zI71bPOnd4hkWquuropptZYFi0tD11dANVryrkoKNO9nVRNdXQlxME0cmiQed9dU7LYluCdHZ\n9aViISJdRmyMBS7gmJrEaNKbbVdVU8eOPV8UlG3l1UFHLVV8vKWct1dvZ19N3SHrpibFHXRdr95p\nid636r+YzuqeSHwn6/pSsRARaSQpPpaBGckMzGi562tPtdf15R2pbNvjfenRO1JZvy7Q9VXbqOvL\nDnR9HXqkEjye0jO543R9qViIiBwGMyMtKZ60pHiG9mq+66u+3lFasf+LMZRGRyqbd1ex+LPd7KzY\nf8i6CbEx9E6Np39aHH1T4+mTGk/v7rH0TomjV/c4spNjyE6JpVtyKqT1jeSPq2IhIp2Ec1BfB/W1\nLTzqoK7m4OcHpmuamFcLdU28Rlu1r6shpr6O7PpasutrGXVgeaOMybW4pFrqa2sOeu0YV4dVOagC\ntje/aQpSv864G1+O6OaPaLEws4nA74BY4Ann3N2NlicCTwLjgFLgUufcBm/ZLcB0oA74kXPu9Uhm\nFelU6uvBhdoxHulOsYnXaGpH12L7Zt6zrpn3bKm9O3R8oF3FxLXwiA38Gxt/8POGR1wixKS02N5i\n4oht6fW9tlV1Rvl+R3m1o6zasavKkdxrSMR//IgVCzOLBR4GzgSKgUVmttA593FQs+nALufcUDOb\nDNwDXGpmI4DJwEigH/CWmR3tnN+flghwDlx94F+Cpl299zx4uqEdYbYLfj0XZruGZYTZrvF0Z/85\nCLNdwzLCe+1wd4zh7kQbsvrCgnZyjXeM8YfuKIN2dMTEQXw3SExtecfb5Os3syOObWbHHXb7hjZN\nvGdDDosJDDR0AEneo1c7v28kjywmAIXOufUAZjYfmAQEF4tJwB3e9ALgIQuM5kwC5jvnqoEiMyv0\nXu/9Nk+5bRW8cDW+7JxwzYSSiLIYwAK//AemG3YGQdMHnodq17CMZtpZMztRb8cZ7k631TvFZnbc\nTe0Um23f1KNzncUjbSOSxaI/sCnoeTFwQnNtnHO1ZlYGZHrzP2i0bv/Gb2BmM4AZAIMGDTq8lHFJ\n0Gt4mDsQWrGjiWkIGeaOxg5d1my7FtYJma+p12jLHWYH/TkOeh8Raa1OPcDtnJsFzALIy8s7vD/T\nM4+CS55sy1giIlEnkseTmzn4Pl8DvHlNtjGzOCCdwEB3OOuKiEg7iWSxWAQMM7NcM0sgMGC9sFGb\nhcBUb/pi4G3nnPPmTzazRDPLBYYBH0Uwq4iItCBi3VDeGMT1wOsETp2d45xbZWZ3AfnOuYXAbOAp\nbwB7J4GCgtfueQKD4bXAD6LyTCgRkU7CnIuOM3Ly8vJcfn6+3zFERDoVMytwzuWFaqdz4EREJCQV\nCxERCUnFQkREQlKxEBGRkKJmgNvMdgAbj+AlsoCSNorTlpSrdZSrdZSrdaIx12DnXHaoRlFTLI6U\nmeWHc0ZAe1Ou1lGu1lGu1unKudQNJSIiIalYiIhISCoWX5jld4BmKFfrKFfrKFfrdNlcGrMQEZGQ\ndGQhIiIhqViIiEhIUV8szGyimX1iZoVmdnMTyxPN7Dlv+YdmlhO07BZv/idmdlY75/qJmX1sZsvN\n7O9mNjhoWZ2ZLfUejS/7Hulc08xsR9D7fydo2VQzW+s9pjZeN8K57gvK9KmZ7Q5aFsntNcfMtpvZ\nymaWm5k94OVebmbHBy2L5PYKlesKL88KM3vPzI4LWrbBm7/UzNr06pxh5DrNzMqC/r9+FrSsxc9A\nhHPNDMq00vtMZXjLIrm9BprZO96+YJWZ3dBEm/b5jDnnovZB4NLo64AhQAKwDBjRqM33gUe96cnA\nc970CK99IpDrvU5sO+b6GpDsTX+vIZf3fK+P22sa8FAT62YA671/e3rTPdsrV6P2PyRwSfyIbi/v\ntb8CHA+sbGb5OcBrgAEnAh9GenuFmevkhvcDzm7I5T3fAGT5tL1OA/56pJ+Bts7VqO15BO690x7b\nqy9wvDedCnzaxO9ku3zGov3IYgJQ6Jxb75zbD8wHJjVqMwmY500vAE43M/Pmz3fOVTvnioBC7/Xa\nJZdz7h3nXKX39AMCdwuMtHC2V3POAt50zu10zu0C3gQm+pTrMuDZNnrvFjnn/kngXizNmQQ86QI+\nAHqYWV8iu71C5nLOvee9L7Tf5yuc7dWcI/lstnWu9vx8fe6cW+xN7wFWA/0bNWuXz1i0F4v+wKag\n58UcuqEPtHHO1QJlQGaY60YyV7DpBP5yaJBkZvlm9oGZXdBGmVqT6yLvcHeBmTXc/rZDbC+vuy4X\neDtodqS2Vziayx7J7dVajT9fDnjDzArMbIYPeU4ys2Vm9pqZjfTmdYjtZWbJBHa4LwbNbpftZYEu\n8rHAh40WtctnLGJ3ypO2YWZTgDzgq0GzBzvnNpvZEOBtM1vhnFvXTpH+AjzrnKs2s2sJHJV9vZ3e\nOxyTgQXu4Dsr+rm9OjQz+xqBYnFq0OxTve3VC3jTzNZ4f3m3h8UE/r/2mtk5wJ8I3Fa5ozgP+I9z\nLvgoJOLby8y6EyhQP3bOlbfla4cr2o8sNgMDg54P8OY12cbM4oB0oDTMdSOZCzM7A7gVON85V90w\n3zm32ft3PfAugb822iWXc640KMsTwLhw141kriCTadRFEMHtFY7mskdye4XFzI4l8H84yTlX2jA/\naHttB16m7bpfQ3LOlTvn9nrTrwLxZpZFB9henpY+XxHZXmYWT6BQPOOce6mJJu3zGYvEoExHeRA4\nclpPoFuiYVBsZKM2P+DgAe7nvemRHDzAvZ62G+AOJ9dYAgN6wxrN7wkketNZwFraaKAvzFx9g6Yv\nBD5wXwymFXn5enrTGe2Vy2s3nMBgo7XH9gp6jxyaH7A9l4MHHz+K9PYKM9cgAuNwJzeanwKkBk2/\nB0xsx1x9Gv7/COx0P/O2XVifgUjl8panExjXSGmv7eX97E8C97fQpl0+Y222oTvqg8CZAp8S2PHe\n6s27i8Bf6wBJwAveL85HwJCgdW/11vsEOLudc70FbAOWeo+F3vyTgRXeL8sKYHo75/o1sMp7/3eA\n4UHrXuNtx0Lg6vbM5T2/A7i70XqR3l7PAp8DNQT6hKcD1wHXecsNeNjLvQLIa6ftFSrXE8CuoM9X\nvjd/iLetlnn/z7e2c67rgz5fHxBUzJr6DLRXLq/NNAInvQSvF+ntdSqBMZHlQf9X5/jxGdPlPkRE\nJKRoH7MQEZE2oGIhIiIhqViIiEhIKhYiIhKSioWIiISkYiHSAXhXW/2r3zlEmqNiISIiIalYiLSC\nmU0xs4+8exc8ZmaxZrbXu5/GKgvceyTbazvGu3jhcjN72cx6evOHmtlb3sXyFpvZUd7Ld/cuzrjG\nzJ7xrn4s0iGoWIiEycyOAS4FTnHOjQHqgCsIXOYh3zk3EvgH8HNvlSeBm5xzxxL4Zm3D/GeAh51z\nxxH4hvnn3vyxwI8J3EtlCHBKxH8okTDpqrMi4TudwIUTF3l/9HcDtgP1wHNem6eBl8wsHejhnPuH\nN38e8IKZpQL9nXMvAzjnqgC81/vIOVfsPV9K4FpF/478jyUSmoqFSPgMmOecu+WgmWa3N2p3uNfQ\nqQ6arkO/n9KBqBtKJHx/By727luAmWV4N1uKAS722lwO/Ns5VwbsMrMve/OvBP7hAnc7K264CZMF\n7gGf3K4/hchh0F8uImFyzn1sZrcRuCtaDIErlP4AqAAmeMu2ExjXAJgKPOoVg/XA1d78K4HHzOwu\n7zW+3Y4/hshh0VVnRY6Qme11znX3O4dIJKkbSkREQtKRhYiIhKQjCxERCUnFQkREQlKxEBGRkFQs\nREQkJBULEREJ6f8Ds6UvfTZ1pJkAAAAASUVORK5CYII=\n", 261 | "text/plain": [ 262 | "" 263 | ] 264 | }, 265 | "metadata": {}, 266 | "output_type": "display_data" 267 | } 268 | ], 269 | "source": [ 270 | "# summarize history for loss\n", 271 | "plt.plot(history.history['loss'])\n", 272 | "plt.plot(history.history['val_loss'])\n", 273 | "plt.title('model loss')\n", 274 | "plt.ylabel('loss')\n", 275 | "plt.xlabel('epoch')\n", 276 | "plt.legend(['train', 'test'], loc='upper right')\n", 277 | "plt.show()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 31, 283 | "metadata": { 284 | "scrolled": false 285 | }, 286 | "outputs": [ 287 | { 288 | "data": { 289 | "text/plain": [ 290 | "(46.19725, 22.895014)" 291 | ] 292 | }, 293 | "execution_count": 31, 294 | "metadata": {}, 295 | "output_type": "execute_result" 296 | } 297 | ], 298 | "source": [ 299 | "#the mean score of the high-rank distribution should be higher\n", 300 | "np.mean(base_network.predict(X_1_test)), np.mean(base_network.predict(X_2_test))" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": 32, 306 | "metadata": {}, 307 | "outputs": [ 308 | { 309 | "data": { 310 | "text/plain": [ 311 | "1.0" 312 | ] 313 | }, 314 | "execution_count": 32, 315 | "metadata": {}, 316 | "output_type": "execute_result" 317 | } 318 | ], 319 | "source": [ 320 | "#AUC\n", 321 | "(np.sum(base_network.predict(X_1_test) > base_network.predict(X_2_test))+0.0) / X_1_test.shape[0]" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [] 330 | } 331 | ], 332 | "metadata": { 333 | "kernelspec": { 334 | "display_name": "Python 2", 335 | "language": "python", 336 | "name": "python2" 337 | }, 338 | "language_info": { 339 | "codemirror_mode": { 340 | "name": "ipython", 341 | "version": 2 342 | }, 343 | "file_extension": ".py", 344 | "mimetype": "text/x-python", 345 | "name": "python", 346 | "nbconvert_exporter": "python", 347 | "pygments_lexer": "ipython2", 348 | "version": "2.7.13" 349 | } 350 | }, 351 | "nbformat": 4, 352 | "nbformat_minor": 2 353 | } 354 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RankNet 2 | Learning to Rank from Pair-wise data 3 | 4 | 5 | 6 | Given a set of items, there is value in being able to rank them from best to worst. A common way to do this is to have a scoring function and then pasing the items in the set through the scoring function, then sorting the scores to give an overall rank. There is a way to learn the scoring function by using pairwise data: in this scenario, we take a pair where one item is higher ranked than the other and then we pass it through a scoring function and try to maximize the difference between the two scores. If we are able to consitently differentiate between the pairs, then we will have been able to learn a scoring function. 7 | 8 | I was originally exposed to this technique in the Learning to rank [1] paper by Burges. This paper uses pair-wise data and a Siamese Archecture to learn a scoring function. 9 | 10 | For the full code see the github repo: [https://github.com/eggie5/RankNet/blob/master/RankNet.ipynb](https://github.com/eggie5/RankNet/blob/master/RankNet.ipynb) 11 | 12 | ## Archecture 13 | 14 | ![siamese-net](https://eggie5_production.s3.amazonaws.com/static/201346.jpg) 15 | 16 | ***Figure 1: Siamese Nework*** - Scores between pairs are maximized using binary cross-entropy 17 | 18 | ### Base Network/Scoring Function 19 | 20 | This is scoring function is a regression which takes an embedding (some vector describing the item to be scored/ranked) and outputs a single neuron: the score of the item. The network is composed of two of these, one for each item in the pair. 21 | 22 | In Figure 1 above, the base network is a Deep CNN w/ the traditional softmax removed from the end and replaced w/ a single neuron. The inputs A & B are of course images. 23 | 24 | ```python 25 | def create_base_network(input_dim): 26 | seq = Sequential() 27 | seq.add(Dense(input_dim, input_shape=(input_dim,), activation='relu')) 28 | seq.add(Dropout(0.1)) 29 | seq.add(Dense(64, activation='relu')) 30 | seq.add(Dropout(0.1)) 31 | seq.add(Dense(32, activation='relu')) 32 | seq.add(Dense(1)) 33 | return seq 34 | ``` 35 | 36 | 37 | 38 | ### Meta Network 39 | 40 | The Meta Network takes the outputs of the base network and takes the difference. The cost function then returns a proporital positive on how big the differnece is between the two pairs and a negative feedback if the difference is negative (meaning the ranking was incorrect). 41 | 42 | ```python 43 | def create_meta_network(input_dim, base_network): 44 | input_a = Input(shape=(input_dim,)) 45 | input_b = Input(shape=(input_dim,)) 46 | 47 | rel_score = base_network(input_a) 48 | irr_score = base_network(input_b) 49 | 50 | # subtract scores 51 | diff = Subtract()([rel_score, irr_score]) 52 | 53 | # Pass difference through sigmoid function. 54 | prob = Activation("sigmoid")(diff) 55 | 56 | # Build model. 57 | model = Model(inputs = [input_a, input_b], outputs = prob) 58 | model.compile(optimizer = "adam", loss = "binary_crossentropy") 59 | 60 | return model 61 | ``` 62 | 63 | 64 | 65 | ![meta](https://eggie5_production.s3.amazonaws.com/static/201341.jpg) 66 | 67 | ***Figure 2: Siamese Network diagram*** - The Sequential block in the diagram is any type of function which takes the input vector and outputs a scalar. In this example it's a Neural Network which is learnt. We then subtract the two scores from the pair and pass that to the Activation Sigmoid which is then used in the cost function. 68 | 69 | ### Data & Cost Function 70 | 71 | We can use the binary cross-entropy cost function w/ the pairwise data. The label for an instance is a number between 1-0 where 0 means that the right item is 100% more important .5 means the pair is equal and 1 means the left item is 100% more important. The binary cross entropy then compare the sigmoid of the predicted score (probability) w/ the actual probability which then gives feedback to the gradient descent backprop routine. 72 | 73 | $$ 74 | C_{ij} = - \hat{P_{ij}}o_{ij} + \log(1+e^{o_{ij}}) 75 | $$ 76 | 77 | where $o_{ij}$ is the difference of the output from the scoring function for both items of the pair $i$ and $j$. 78 | 79 | ![learning_to_rank_pdf__page_4_of_8_](https://eggie5_production.s3.amazonaws.com/static/201345.jpg) 80 | 81 | Figure 3: Example values of the cost function for 3 different class labels. For example if the training example is a `1`, this means the left item is higher ranked. Therefore $C_{ij}$ is very small when the difference is positive large and vary large when the difference is negative giving feedback to the backprop. 82 | 83 | ## Case Study: Ranking Search Results 84 | 85 | This was the original use-case explored in the paper: how to more effectivly sort search results. They looked at past examples of click stream data and quries and learnt a scoring function to help sort the results. 86 | 87 | 88 | 89 | 90 | 91 | ## Case Study: Image Selection 92 | 93 | A common use-case for these types of networks is to automate the selection of an image from a set of images in order to choose the best one by some criteria. The training data would be annotated pairs where the labels would be which image is better according to the criteria. For example, the best quality image. Expedia and trip advisor did something like this to help automate and scale up the default image selection process of a hotel's landing page. For example, show a nice picture of the view or room then the toilet as the first image the user sees. 94 | 95 | 96 | 97 | 98 | 99 | [1] Burges, Chris, et al. "Learning to rank using gradient descent." *Proceedings of the 22nd international conference on Machine learning*. ACM, 2005. 100 | -------------------------------------------------------------------------------- /RankNet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# RankNet\n", 8 | "\n", 9 | "This is the generic implementation of RankNet on a synthetic dataset.\n", 10 | "\n", 11 | "This model is applied to hotel imags in the other notebook.\n", 12 | "\n", 13 | "![alt text](assets/siamese-net.png \"Title\")" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 1, 19 | "metadata": {}, 20 | "outputs": [ 21 | { 22 | "name": "stderr", 23 | "output_type": "stream", 24 | "text": [ 25 | "Using TensorFlow backend.\n" 26 | ] 27 | }, 28 | { 29 | "name": "stdout", 30 | "output_type": "stream", 31 | "text": [ 32 | "Populating the interactive namespace from numpy and matplotlib\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "import numpy as np\n", 38 | "\n", 39 | "import keras\n", 40 | "from keras import backend\n", 41 | "from keras.layers import Activation, Add, Dense, Input, Lambda, Dropout, Subtract\n", 42 | "from keras.models import Model, Sequential\n", 43 | "from keras.utils import plot_model\n", 44 | "%pylab inline\n", 45 | "\n", 46 | "INPUT_DIM = 1536" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "## Base Network\n", 54 | "\n", 55 | "This is a scoring function (regression) that takes an images vector as input and outputs a scalar number. The Siamese model create two of these base networks to for higher and lower ranking images respectively." 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 2, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "def create_base_network(input_dim):\n", 65 | " '''Base network to be shared (eq. to feature extraction).\n", 66 | " '''\n", 67 | " seq = Sequential()\n", 68 | " seq.add(Dense(INPUT_DIM, input_shape=(input_dim,), activation='relu'))\n", 69 | " seq.add(Dropout(0.1))\n", 70 | " seq.add(Dense(64, activation='relu'))\n", 71 | " seq.add(Dropout(0.1))\n", 72 | " seq.add(Dense(32, activation='relu'))\n", 73 | " seq.add(Dense(1))\n", 74 | " return seq" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## Siamse Archecture\n", 82 | "\n", 83 | "We create two instances of the base archecture and take the difference between their scalar outputs. We then pass the difference between a sigmoid function so that we can use binary-cross-entropy for the differntiable loss function." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 3, 89 | "metadata": { 90 | "scrolled": false 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "def create_meta_network(input_dim, base_network):\n", 95 | " input_a = Input(shape=(input_dim,))\n", 96 | " input_b = Input(shape=(input_dim,))\n", 97 | "\n", 98 | " rel_score = base_network(input_a)\n", 99 | " irr_score = base_network(input_b)\n", 100 | "\n", 101 | " # subtract scores\n", 102 | " diff = Subtract()([rel_score, irr_score])\n", 103 | "\n", 104 | " # Pass difference through sigmoid function.\n", 105 | " prob = Activation(\"sigmoid\")(diff)\n", 106 | "\n", 107 | " # Build model.\n", 108 | " model = Model(inputs = [input_a, input_b], outputs = prob)\n", 109 | " model.compile(optimizer = \"adam\", loss = \"binary_crossentropy\")\n", 110 | "\n", 111 | " return model" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "__________________________________________________________________________________________________\n", 124 | "Layer (type) Output Shape Param # Connected to \n", 125 | "==================================================================================================\n", 126 | "input_7 (InputLayer) (None, 1536) 0 \n", 127 | "__________________________________________________________________________________________________\n", 128 | "input_8 (InputLayer) (None, 1536) 0 \n", 129 | "__________________________________________________________________________________________________\n", 130 | "sequential_4 (Sequential) (None, 1) 2461313 input_7[0][0] \n", 131 | " input_8[0][0] \n", 132 | "__________________________________________________________________________________________________\n", 133 | "subtract_4 (Subtract) (None, 1) 0 sequential_4[1][0] \n", 134 | " sequential_4[2][0] \n", 135 | "__________________________________________________________________________________________________\n", 136 | "activation_4 (Activation) (None, 1) 0 subtract_4[0][0] \n", 137 | "==================================================================================================\n", 138 | "Total params: 2,461,313\n", 139 | "Trainable params: 2,461,313\n", 140 | "Non-trainable params: 0\n", 141 | "__________________________________________________________________________________________________\n" 142 | ] 143 | } 144 | ], 145 | "source": [ 146 | "base_network = create_base_network(INPUT_DIM)\n", 147 | "model = create_meta_network(INPUT_DIM, base_network)\n", 148 | "model.summary()\n", 149 | "\n", 150 | "plot_model(base_network, to_file='base.png')\n", 151 | "plot_model(model, to_file='meta.png')" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "### Base Network\n", 159 | "\n", 160 | "This can be any (scoring) function which outputs a scalar:\n", 161 | "![base](base.png)\n", 162 | "\n", 163 | "### Meta Network\n", 164 | "\n", 165 | "This takes an instance of the base network and evalutes the pair on it:\n", 166 | "![meta](meta.png)" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "## Synthetic Dataset\n", 174 | "\n", 175 | "We generate two N dimensional datasets from an uniform distribution where one distribution is shift by a constant. This can be thought of as a good image and a less good image that we want to learn to differentiate." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 28, 181 | "metadata": {}, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "text/plain": [ 186 | "(0.0023731553517547066, -0.00052067296140614276)" 187 | ] 188 | }, 189 | "execution_count": 28, 190 | "metadata": {}, 191 | "output_type": "execute_result" 192 | } 193 | ], 194 | "source": [ 195 | "# Fake data.\n", 196 | "N = 1000\n", 197 | "SPLIT=750\n", 198 | "X_1 = 2* np.random.randn(N, INPUT_DIM) # notice this data is shifted by 2\n", 199 | "X_2 = np.random.randn(N, INPUT_DIM)\n", 200 | "\n", 201 | "X_1_train = X_1[0:SPLIT,:]\n", 202 | "X_1_test = X_1[SPLIT:N,:]\n", 203 | "\n", 204 | "X_2_train = X_2[0:SPLIT,:]\n", 205 | "X_2_test = X_2[SPLIT:N,:]\n", 206 | "\n", 207 | "\n", 208 | "y = np.ones((X_1.shape[0], 1))\n", 209 | "\n", 210 | "y_train = y[0:SPLIT,:]\n", 211 | "y_test = y[SPLIT:N,:]\n", 212 | "\n", 213 | "np.mean(X_1_train), np.mean(X_2_train)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": 29, 219 | "metadata": { 220 | "scrolled": false 221 | }, 222 | "outputs": [ 223 | { 224 | "name": "stdout", 225 | "output_type": "stream", 226 | "text": [ 227 | "Train on 750 samples, validate on 250 samples\n", 228 | "Epoch 1/10\n", 229 | "750/750 [==============================] - 7s 9ms/step - loss: 0.0460 - val_loss: 1.1323e-04\n", 230 | "Epoch 2/10\n", 231 | "750/750 [==============================] - 5s 7ms/step - loss: 0.0045 - val_loss: 3.0593e-04\n", 232 | "Epoch 3/10\n", 233 | "750/750 [==============================] - 6s 8ms/step - loss: 8.4689e-04 - val_loss: 8.8093e-04\n", 234 | "Epoch 00003: early stopping\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "# Train model.\n", 240 | "\n", 241 | "es=keras.callbacks.EarlyStopping(monitor='val_loss',\n", 242 | " min_delta=0,\n", 243 | " patience=2,\n", 244 | " verbose=1, mode='auto')\n", 245 | "\n", 246 | "NUM_EPOCHS = 10\n", 247 | "BATCH_SIZE = 10\n", 248 | "history = model.fit([X_1_train, X_2_train], y_train, \n", 249 | " validation_data=([X_1_test, X_2_test], y_test), \n", 250 | " batch_size = BATCH_SIZE, epochs = NUM_EPOCHS, verbose = 1, callbacks=[es])" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": 33, 256 | "metadata": {}, 257 | "outputs": [ 258 | { 259 | "data": { 260 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXd//H3NzsJSSAL+5IgWGRRkIBrW1u1olbRahUV\nBaVF29rapz5UfdRWbftUf+2j1qUqCgWXiora0qp1qdrNjYQdQQkEJCBLAiSQkJDl/v0xJziEJDOB\nTE4y+byuay7OnHOfmU8Ok/PNue8z55hzDhERkZbE+B1AREQ6PhULEREJScVCRERCUrEQEZGQVCxE\nRCQkFQsREQlJxULkCJnZXDP7ZZhtN5jZGUf6OiLtTcVCRERCUrEQEZGQVCykS/C6f2aa2XIzqzCz\n2WbW28xeM7M9ZvaWmfUMan++ma0ys91m9q6ZHRO0bKyZLfbWew5IavRe3zSzpd6675nZsYeZ+btm\nVmhmO81soZn18+abmd1nZtvNrNzMVpjZKG/ZOWb2sZdts5n992FtMJFGVCykK7kIOBM4GjgPeA34\nHyCbwO/CjwDM7GjgWeDH3rJXgb+YWYKZJQB/Ap4CMoAXvNfFW3csMAe4FsgEHgMWmllia4Ka2deB\nXwOXAH2BjcB8b/E3gK94P0e616bUWzYbuNY5lwqMAt5uzfuKNEfFQrqSB51z25xzm4F/AR8655Y4\n56qAl4GxXrtLgVecc28652qA3wLdgJOBE4F44H7nXI1zbgGwKOg9ZgCPOec+dM7VOefmAdXeeq1x\nBTDHObfYOVcN3AKcZGY5QA2QCgwHzDm32jn3ubdeDTDCzNKcc7ucc4tb+b4iTVKxkK5kW9D0viae\nd/em+xH4Sx4A51w9sAno7y3b7A6+AufGoOnBwI1eF9RuM9sNDPTWa43GGfYSOHro75x7G3gIeBjY\nbmazzCzNa3oRcA6w0cz+YWYntfJ9RZqkYiFyqC0EdvpAYIyAwA5/M/A50N+b12BQ0PQm4FfOuR5B\nj2Tn3LNHmCGFQLfWZgDn3APOuXHACALdUTO9+Yucc5OAXgS6y55v5fuKNEnFQuRQzwPnmtnpZhYP\n3EigK+k94H2gFviRmcWb2beACUHrPg5cZ2YneAPRKWZ2rpmltjLDs8DVZjbGG+/4XwLdZhvMbLz3\n+vFABVAF1HtjKleYWbrXfVYO1B/BdhA5QMVCpBHn3CfAFOBBoITAYPh5zrn9zrn9wLeAacBOAuMb\nLwWtmw98l0A30S6g0Gvb2gxvAbcDLxI4mjkKmOwtTiNQlHYR6KoqBX7jLbsS2GBm5cB1BMY+RI6Y\n6eZHIiISio4sREQkJBULEREJScVCRERCUrEQEZGQ4vwO0FaysrJcTk6O3zFERDqVgoKCEudcdqh2\nUVMscnJyyM/P9zuGiEinYmYbQ7dSN5SIiIRBxUJEREJSsRARkZCiZsxCRORw1NTUUFxcTFVVld9R\nIiopKYkBAwYQHx9/WOurWIhIl1ZcXExqaio5OTkcfDHh6OGco7S0lOLiYnJzcw/rNdQNJSJdWlVV\nFZmZmVFbKADMjMzMzCM6elKxEJEuL5oLRYMj/Rm7fLGoqK7ljoWrKNtX43cUEZEOq8sXizVby3n6\ng4384JnF1NTpPjEi0r52797N73//+1avd84557B79+4IJGpaly8W4wZn8L/fGs2/C0v4+cJV6P4e\nItKemisWtbW1La736quv0qNHj0jFOoTOhgIuyRtIUUkFj7y7jiFZKXzny0P8jiQiXcTNN9/MunXr\nGDNmDPHx8SQlJdGzZ0/WrFnDp59+ygUXXMCmTZuoqqrihhtuYMaMGcAXlzjau3cvZ599Nqeeeirv\nvfce/fv3589//jPdunVr05wqFp6Z3/gSG0oq+NWrqxmUkcw3RvbxO5KItLM7/7KKj7eUt+lrjuiX\nxs/PG9ns8rvvvpuVK1eydOlS3n33Xc4991xWrlx54BTXOXPmkJGRwb59+xg/fjwXXXQRmZmZB73G\n2rVrefbZZ3n88ce55JJLePHFF5kyZUqb/hxdvhuqQUyMce8lYzi2fzo3zF/Kys1lfkcSkS5owoQJ\nB30X4oEHHuC4447jxBNPZNOmTaxdu/aQdXJzcxkzZgwA48aNY8OGDW2eS0cWQbolxPL41DwufPg9\nps9bxJ9+cAp909v2UE5EOq6WjgDaS0pKyoHpd999l7feeov333+f5ORkTjvttCa/K5GYmHhgOjY2\nln379rV5Lh1ZNNIrNYnZ0/KoqK5j+tx8KqpbHmQSETkSqamp7Nmzp8llZWVl9OzZk+TkZNasWcMH\nH3zQzum+oGLRhOF90njw8rGs2VrODfOXUFevM6REJDIyMzM55ZRTGDVqFDNnzjxo2cSJE6mtreWY\nY47h5ptv5sQTT/QpJVi0nCqal5fn2vrmR0++v4Gf/XkV00/N5fZvjmjT1xaRjmH16tUcc8wxfsdo\nF039rGZW4JzLC7WuxixacNVJOazfUcHsfxeRk5XClScO9juSiIgvVCxCuP2bI9hYWsEdC1cxKCOZ\nrx4d8la1IiJRR2MWIcTGGA9efjzDenXn+mcW8+m2pgeiRESimYpFGLonxjF72niSEmK5+g+L2LGn\n2u9IIiLtSsUiTP17dOOJq/IorahmxlP5VNXU+R1JRKTdqFi0wnEDe3D/pWNY8tlu/vuFZdTrlFoR\n6SJULFpp4qi+3Hz2cP66/HPue+tTv+OISCd3uJcoB7j//vuprKxs40RNU7E4DNd+ZQiX5g3kwbcL\nebGg2O84ItKJdZZioVNnD4OZ8YsLRrFpVyU3v7ScAT27ccKQzNAriog0EnyJ8jPPPJNevXrx/PPP\nU11dzYUXXsidd95JRUUFl1xyCcXFxdTV1XH77bezbds2tmzZwte+9jWysrJ45513IppTxeIwJcTF\n8MgV47jwkf9w7dMFvPz9U8jNSgm9ooh0XK/dDFtXtO1r9hkNZ9/d7OLgS5S/8cYbLFiwgI8++gjn\nHOeffz7//Oc/2bFjB/369eOVV14BAteMSk9P59577+Wdd94hKyurbTM3Qd1QRyA9OZ4/TBuPAdfM\nXcTuyv1+RxKRTuyNN97gjTfeYOzYsRx//PGsWbOGtWvXMnr0aN58801uuukm/vWvf5Gent7u2XRk\ncYQGZ6Yw66o8rnj8Q659qoCnpp9AQpxqsEin1MIRQHtwznHLLbdw7bXXHrJs8eLFvPrqq9x2222c\nfvrp/OxnP2vXbNqrtYHxORncc/FoPizaya0vr9B9vEUkbMGXKD/rrLOYM2cOe/fuBWDz5s1s376d\nLVu2kJyczJQpU5g5cyaLFy8+ZN1I05FFG7lw7ACKSip54O9ryc1O4funDfU7koh0AsGXKD/77LO5\n/PLLOemkkwDo3r07Tz/9NIWFhcycOZOYmBji4+N55JFHAJgxYwYTJ06kX79+ER/gjuglys1sIvA7\nIBZ4wjl3d6PlicCTwDigFLjUObchaPkg4GPgDufcb1t6r0hcory1nHPcMH8pC5dt4fdXHM85o/v6\nmkdEQtMlysO7RHnEuqHMLBZ4GDgbGAFcZmaNbwoxHdjlnBsK3Afc02j5vcBrkcrY1syM/3fxsRw/\nqAf/9dxSlm7a7XckEZE2EckxiwlAoXNuvXNuPzAfmNSozSRgnje9ADjdzAzAzC4AioBVEczY5pLi\nY3n8qjx6pSXynXn5FO9qny/MiIhEUiSLRX9gU9DzYm9ek22cc7VAGZBpZt2Bm4A7W3oDM5thZvlm\nlr9jx442C36kMrsnMmfqeKprA/fx3lNV43ckEWlBVzgp5Uh/xo56NtQdwH3Oub0tNXLOzXLO5Tnn\n8rKzO9ZNiYb1TuWRK8ZRuGMv1/9xCbV19X5HEpEmJCUlUVpaGtUFwzlHaWkpSUlJh/0akTwbajMw\nMOj5AG9eU22KzSwOSCcw0H0CcLGZ/T+gB1BvZlXOuYcimLfNnTosi19eMIpbXlrBnX/5mLsmjcTr\nZRORDmLAgAEUFxfTkXonIiEpKYkBAwYc9vqRLBaLgGFmlkugKEwGLm/UZiEwFXgfuBh42wXK+5cb\nGpjZHcDezlYoGlw2YRBFJRXM+ud6hmSncPUpuX5HEpEg8fHx5Obq9zKUiBUL51ytmV0PvE7g1Nk5\nzrlVZnYXkO+cWwjMBp4ys0JgJ4GCEnVumjicopIKfvHXjxmcmczXh/f2O5KISKtE9HsW7akjfM+i\nJZX7a/n2o++zoaSCF647mRH90vyOJCLi//cs5GDJCXHMnjqe1KR4ps9bxPbyKr8jiYiETcWiHfVJ\nT+KJqXmU7ath+rx8KvfX+h1JRCQsKhbtbFT/dB6YPJaVW8r4r+eW6j7eItIpqFj44IwRvbnt3BG8\nvmob97y+xu84IiIh6aqzPrnmlByKSvby2D/Wk5uZwuQJg/yOJCLSLBULn5gZd5w3ko2lldz2p5UM\nzEjmlKGRvzWiiMjhUDeUj+JiY3j4iuMZkp3CdU8XULi9fW5iIiLSWioWPktLimf21PEkxsVwzdx8\nSvdW+x1JROQQKhYdwMCMZGZdlcfW8iqufaqA6to6vyOJiBxExaKDOH5QT/7v28eRv3EXNy1YHtVX\nwBSRzkcD3B3Iecf1Y2NpBb9941Nys7pzwxnD/I4kIgKoWHQ4P/jaUNaXVHDfW5+Sk5XMpDGN7xcl\nItL+1A3VwZgZv/7WaCbkZjDzheXkb9jpdyQRERWLjigxLpbHpoyjX48kZjxVwGeluo+3iPhLxaKD\n6pmSwJxp46mrd1w99yPK9uk+3iLiHxWLDmxIdnceu3Icn+2s5PvPFFCj+3iLiE9ULDq4E4dk8r8X\njuY/haXc/qeVOqVWRHyhs6E6gW/nDWRDaQUPv7OOIdkpzPjKUX5HEpEuRsWik7jxzC9RVFLBr19b\nw+DMFM4a2cfvSCLShagbqpOIiTHuvWQMxw7owY/nL2VFcZnfkUSkC1Gx6ESS4mN5/KpxZKQkMH3e\nIj4v2+d3JBHpIlQsOpleqUnMnpZH5f46rpmbT0W17uMtIpGnYtEJDe+TxkOXj+WTreX86Nkl1Ok+\n3iISYSoWndRpX+rFneeP5O9rtvOrV1b7HUdEopzOhurErjwph/UlFcz5TxG5WclceVKO35FEJEqp\nWHRyt507go2lldzxl48ZmJHMaV/q5XckEYlC6obq5GJjjAcuG8vRvVO5/o9L+GSr7uMtIm1PxSIK\ndE+MY/bUPJITYrlm7iK276nyO5KIRBkViyjRr0c3Zk8dT2lFNTOeLKCqRvfxFpG2o2IRRUYPSOf+\nS8eyrHg3Nz6/jHqdUisibUTFIspMHNWHmycO55UVn3Pvm5/6HUdEooTOhopCM74yhKKSCh56p5Cc\nrBQuHjfA70gi0snpyCIKmRm/uGAUJx+VyS0vLeeD9aV+RxKRTk7FIkrFx8bwyBXjGJSRzLVPFbB+\nx16/I4lIJxbRYmFmE83sEzMrNLObm1ieaGbPecs/NLMcb/4EM1vqPZaZ2YWRzBmt0pPjmTNtPLEx\nxvR5+eyq2O93JBHppCJWLMwsFngYOBsYAVxmZiMaNZsO7HLODQXuA+7x5q8E8pxzY4CJwGNmpvGV\nwzA4M4VZV45j8659XPt0AftrdR9vEWm9SB5ZTAAKnXPrnXP7gfnApEZtJgHzvOkFwOlmZs65Sudc\nw7W3kwCdA3oE8nIy+M23j+Wjop3c8tIK3cdbRFotksWiP7Ap6HmxN6/JNl5xKAMyAczsBDNbBawA\nrgsqHgeY2Qwzyzez/B07dkTgR4gek8b058dnDOPFxcX8/t11fscRkU6mww5wO+c+dM6NBMYDt5hZ\nUhNtZjnn8pxzednZ2e0fspO54fRhTBrTj9+8/gl/Xb7F7zgi0olEslhsBgYGPR/gzWuyjTcmkQ4c\ndJ6nc241sBcYFbGkXYSZcc9FxzJucE9ufH4ZSz7b5XckEekkIlksFgHDzCzXzBKAycDCRm0WAlO9\n6YuBt51zzlsnDsDMBgPDgQ0RzNplJMXHMuvKcfRKS+S7T+azaWel35FEpBOIWLHwxhiuB14HVgPP\nO+dWmdldZna+12w2kGlmhcBPgIbTa08FlpnZUuBl4PvOuZJIZe1qMrsn8odp46murWf6vEWUV9X4\nHUlEOjiLljNj8vLyXH5+vt8xOpX/FJYwdc5HnDw0izlT84iL7bBDWCISIWZW4JzLC9VOe4cu7JSh\nWfziglH889Md3PGXVTqlVkSapS+6dXGXTRjEhpIKHvvneoZkdeeaU3P9jiQiHZCKhXDTxOEUlVTw\ni1c+ZlBGMmeM6O13JBHpYNQNJcTEGPdPHsOofun8aP4SVm0p8zuSiHQwKhYCQHJCHE9MzSO9WzzT\n5+azrVz38RaRL6hYyAG905J4Ymoe5VU1TJ+3iMr9h1xhRUS6KBULOcjIfuk8eNlYPt5Szo/nL9V9\nvEUEULGQJpx+TG9uO3cEb3y8jXv+tsbvOCLSAehsKGnS1afkUOSdUpuTlcJlEwb5HUlEfKRiIU0y\nM35+3gg+21nJ7X9aycCeyZw6LMvvWCLiE3VDSbPiYmN46PKxHJXdne89U0Dh9j1+RxIRn6hYSItS\nk+KZPS2PxLgYrp67iNK91X5HEhEfqFhISAN6JvP4VXlsL69mxlMFVNXU+R1JRNqZioWEZeygntx7\nyRgKNu7ipwuW66KDIl2MioWE7dxj+zLzrC+xcNkW7n9rrd9xRKQd6WwoaZXvn3YU63dU8Lu/ryU3\nK4ULxvb3O5KItAMdWUirmBm//tZoTsjN4KcLlpO/YaffkUSkHahYSKslxMXw6JRx9O/ZjRlPFbCx\ntMLvSCISYWEVCzO7wczSLGC2mS02s29EOpx0XD1TEpgzbTz1znHN3EWUVeo+3iLRLNwji2ucc+XA\nN4CewJXA3RFLJZ1CblYKj04Zx2c7K/neMwXU1NX7HUlEIiTcYmHev+cATznnVgXNky7sxCGZ/Ppb\nx/LeulJue3mlTqkViVLhng1VYGZvALnALWaWCujPSAHg4nED2FBSwUPvFDIkO4Vrv3qU35FEpI2F\nWyymA2OA9c65SjPLAK6OXCzpbH5y5tEUlVZw99/WMDgzmYmj+vodSUTaULjdUCcBnzjndpvZFOA2\nQDdqlgNiYoz/+/ZxHDegBz9+binLi3f7HUlE2lC4xeIRoNLMjgNuBNYBT0YslXRKSfGxPH5VHpkp\niUyfl8+W3fv8jiQibSTcYlHrAiOXk4CHnHMPA6mRiyWdVXZqInOmjWff/jqumbuIvdW6j7dINAi3\nWOwxs1sInDL7ipnFAPGRiyWd2Zf6pPLwFcezdvtefvTsEup0H2+RTi/cYnEpUE3g+xZbgQHAbyKW\nSjq9rx6dzR3nj+TtNdv55Ssf+x1HRI5QWMXCKxDPAOlm9k2gyjmnMQtp0ZUnDuaaU3L5w3828OT7\nG/yOIyJHINzLfVwCfAR8G7gE+NDMLo5kMIkOt557DKcP78UdC1fx7ifb/Y4jIocp3G6oW4Hxzrmp\nzrmrgAnA7ZGLJdEiNsZ44LKxDO+TxvV/XMKareV+RxKRwxBusYhxzgX/WVjainWli0tJjGP2tDyS\nE2KZPjef7Xuq/I4kIq0U7g7/b2b2uplNM7NpwCvAq5GLJdGmb3o3Zk8dz86K/Xz3yQL27dd9vEU6\nk3AHuGcCs4Bjvccs59xNkQwm0Wf0gHTunzyG5cW7ufGFpdTrlFqRTiPsriTn3IvOuZ94j5fDWcfM\nJprZJ2ZWaGY3N7E80cye85Z/aGY53vwzzazAzFZ4/3493JzSsZ01sg+3nD2cV1ds5bdvfOJ3HBEJ\nU4sXEjSzPUBTf/4Z4JxzaS2sGws8DJwJFAOLzGyhcy74pPvpwC7n3FAzmwzcQ+A7HSXAec65LWY2\nCngd0M2eo8R3vzyEopIKfv/uOnKyUrgkb6DfkUQkhBaLhXPuSC7pMQEodM6tBzCz+QQuFxJcLCYB\nd3jTC4CHzMycc0uC2qwCuplZonOu+gjySAdhZtw1aRSf7azk1pdXMLBnMicdlel3LBFpQSTPaOoP\nbAp6XsyhRwcH2jjnaglcybbxXuMiYHFThcLMZphZvpnl79ixo82CS+TFx8bw+yvGMSgjmeueLmD9\njr1+RxKRFnTo01/NbCSBrqlrm1runJvlnMtzzuVlZ2e3bzg5Yund4vnDtAnExhjXzF3Eror9fkcS\nkWZEslhsBoI7owd485psY2ZxQDqB73BgZgOAl4GrnHPrIphTfDQoM5nHrxrHlrIqrn2qgOpanVIr\n0hFFslgsAoaZWa6ZJQCTgYWN2iwEpnrTFwNvO+ecmfUg8F2Om51z/4lgRukAxg3O4DcXH8tHG3Zy\ny0srdB9vkQ4oYsXCG4O4nsCZTKuB551zq8zsLjM732s2G8g0s0LgJ0DD6bXXA0OBn5nZUu/RK1JZ\nxX+TxvTnv844mpcWb+bhdwr9jiMijVi0/BWXl5fn8vPz/Y4hR8A5x0+eX8bLSzbz4GVjOe+4fn5H\nEol6ZlbgnMsL1a5DD3BL12Jm3H3RaMbn9OTGF5ZRsHGX35FExKNiIR1KYlwsj12ZR5+0JGY8mc+m\nnZV+RxIRVCykA8pISWDOtPHsr6vnmrmLKK+q8TuSSJenYiEd0tBe3Xl0yjiKSir4wTOLqa2r9zuS\nSJemYiEd1ilDs/jlBaP419oSfr5wlU6pFfFRi9eGEvHb5AmDKCqt4LF/rGdIdnemn5rrdySRLknF\nQjq8m84azsaSSn75yscMzkjmjBG9/Y4k0uWoG0o6vJgY475LxzC6fzo/mr+ElZvL/I4k0uWoWEin\n0C0hlieuyiO9WzzfmZfP1jLdx1ukPalYSKfRKy2J2VPHs6eqhunzFlFRXet3JJEuQ8VCOpUR/dJ4\n8PKxrP68nBvmL6VO9/EWaRcqFtLpfH14b27/5gjeWr2Nu19b7XcckS5BZ0NJpzTt5ByKSip4/F9F\n5GZ15/ITBvkdSSSqqVhIp2Rm/OybI9hYWsntf17JoIxkTh2W5XcskailbijptOJiY3jo8rEMze7O\n954pYO22PX5HEolaKhbSqaUmxTN7Wh6JcbFcM28RJXur/Y4kEpVULKTTG9AzmSem5rG9vJoZT+ZT\nVaP7eIu0NRULiQpjBvbgvkvHsPiz3cxcsFwXHRRpYyoWEjXOGd2Xn078En9ZtoX73lrrdxyRqKKz\noSSqfO+rR1G0o4IH/r6W3KxkLhw7wO9IIlFBRxYSVcyMX104mhOHZHDTghV8VLTT70giUUHFQqJO\nQlwMj04Zx4Ce3bj2qXw2lFT4HUmk01OxkKjUIzmB2dPG44Br5i6irFL38RY5EioWErVys1J4bMo4\nNu2q5LqnC9hfq/t4ixwuFQuJaicMyeTubx3L++tLuf1PK3VKrchh0tlQEvUuGjeADaUVPPh2IbnZ\nKVz31aP8jiTS6ahYSJfwX2ccTVFJBXe/toaczGQmjurrdySRTkXdUNIlxMQYv/32cYwd1IMfP7eU\nZZt2+x1JpFNRsZAuIyk+lllX5pHVPZHvPJnP5t37/I4k0mmoWEiXkp2ayJxp46naX8f0uYvYU6VT\nakXCoWIhXc7RvVN5+IrjWbt9Lz98dgm1dTqlViQUFQvpkr5ydDZ3nD+Sdz/ZwS9f0X28RULR2VDS\nZV154mA2lFQw+99F5GalMPXkHL8jiXRYKhbSpf3POcewsbSCO/+yikEZyXxteC+/I4l0SBHthjKz\niWb2iZkVmtnNTSxPNLPnvOUfmlmONz/TzN4xs71m9lAkM0rXFhtj/G7yWIb3SeP6Py5m9eflfkcS\n6ZAiVizMLBZ4GDgbGAFcZmYjGjWbDuxyzg0F7gPu8eZXAbcD/x2pfCINUhLjmD0tj+5JcUyfu4jt\ne6r8jiTS4UTyyGICUOicW++c2w/MByY1ajMJmOdNLwBONzNzzlU45/5NoGiIRFzf9G7MnjqeXZU1\nfHdePvv26z7eIsEiWSz6A5uCnhd785ps45yrBcqAzHDfwMxmmFm+meXv2LHjCONKVzeqfzq/mzyG\n5ZvL+MnzS6mv10UHRRp06lNnnXOznHN5zrm87Oxsv+NIFPjGyD7ces4xvLZyK7954xO/44h0GJE8\nG2ozMDDo+QBvXlNtis0sDkgHSiOYSSSk6afmsr6kgkfeXUduZgqXjB8YeiWRKBfJI4tFwDAzyzWz\nBGAysLBRm4XAVG/6YuBtpxsOiM/MjDvPH8mXh2XxPy+v4L11JX5HEvFdxIqFNwZxPfA6sBp43jm3\nyszuMrPzvWazgUwzKwR+Ahw4vdbMNgD3AtPMrLiJM6lEIiY+NoaHLj+enKwUrnuqgHU79vodScRX\nFi1/yOfl5bn8/Hy/Y0iU2bSzkgse/g/dk+J4+funkJGS4HckkTZlZgXOubxQ7Tr1ALdIpA3MSGbW\nVeP4vKyK654qoLpWp9RK16RiIRLCuMEZ/ObiY/low05ueXGF7uMtXZKuDSUShklj+rOxtJJ73/yU\n3KwUfnj6ML8jibQrFQuRMP3w60MpKqng/978lMFZKZx/XD+/I4m0G3VDiYTJzLj7otGMz+nJf7+w\njIKNu/yOJNJuVCxEWiExLpbHrsyjb3oSM57MZ9POSr8jibQLFQuRVspISWDOtPHU1NVz9dxFlO3T\nfbwl+qlYiByGo7K78+iV49hQUsH1f1xMje7jLVFOxULkMJ18VBa/unAU/1pbws8XrtIptRLVdDaU\nyBG4dPwgikoqefQf6xiSlcJ3vjzE70giEaFiIXKEfnrWl9hQUsGvXl3N4MwUzhzR2+9IIm1O3VAi\nRygmxrjv0jGM7p/Oj55dwsrNZX5HEmlzKhYibaBbQixPXJVHz+R4ps9bxNYy3RFYoouKhUgb6ZWW\nxOxp49lbVcv0eYuoqK71O5JIm1GxEGlDx/RN46HLj2f15+XcMH8pdbqPt0QJFQuRNva14b34+Xkj\neWv1Nn796mq/44i0CZ0NJRIBU0/Ooaikgif+XUROVgpTThzsdySRI6JiIRIht517DBtKK/j5wlV8\nVLSTvulJ9E4LPPqkJ9IrNYleaYkkxsX6HVUkJBULkQiJi43hwcvG8tMFy1myaRd/W1XN/tpDLwuS\nkZJAr9SE85PjAAANzUlEQVRE+qQn0Ts1id7pSfROS6RP2hfFJTMlgZgY8+GnEAlQsRCJoNSkeB6Z\nMg4A5xy7K2vYWl7FtvIqtpdXH5gOPKpZtaWckr3VNL5ySFyMkZ2a6BWPQCHplZYUVFAS6Z2eRGpi\nHGYqKtL2VCxE2omZ0TMlgZ4pCRzTN63ZdrV19ezYW8228mq2llWxfU+gmGwtq2b7nirW76jg/XWl\nlFcdemput/hY+qQnfXGkElxMvOKSnZpIUry6vqR1VCxEOpi42Bj6pnejb3o3GNh8u8r9tQcdnTQ+\nUlny2W62llc12fXVMzn+oELS1JFKZvdEYtX1JR4VC5FOKjkhjpysOHKyUppt45yjbF9D11d1oJCU\nVbFtzxdHKqs/D3R9Nf5KSGyMkd090L3VO+hIpfFRS1qSur66AhULkShmZvRITqBHcgLD+zTfrrau\nntKK/WwtO3gMpeFIZUNpBR8W7WzyRk9J8TGNjkwSg45akrxl6vrq7FQsRIS42JgDO/eW7Ntf542h\nBArJ9vKqQIHZU822siqWFe9ma1kV1U10ffVIjv/ibK+GwfpGRy1Z6vrqsFQsRCRs3RJiGZyZwuDM\nlru+yvfVel1dhx6pbC+v4tOte9i+p+qQrq8Yg+zUL8ZQmhpP6ZOWRFo3dX21NxULEWlTZkZ6cjzp\nyfEc3Tu12XZ19Y7SvdUHxlMaH6ls2lnJog072V3ZdNdX77TA91J6BX8npdGRirq+2o6KhYj4IjbG\n6OUdNbSkqqaO7eXVTR6pbCuvYuXmMt5avY2qmkO7vtK7xTcaQzn0SCWrewJxsbpMXigqFiLSoSXF\nxzIoM5lBmcnNtnHOUV5VGzgyCT7zK+hIpXB7Cdv3VB9yJeAYg6zuhxaUA0cq3vP0bvFduutLxUJE\nOj0zI71bPOnd4hkWquuropptZYFi0tD11dANVryrkoKNO9nVRNdXQlxME0cmiQed9dU7LYluCdHZ\n9aViISJdRmyMBS7gmJrEaNKbbVdVU8eOPV8UlG3l1UFHLVV8vKWct1dvZ19N3SHrpibFHXRdr95p\nid636r+YzuqeSHwn6/pSsRARaSQpPpaBGckMzGi562tPtdf15R2pbNvjfenRO1JZvy7Q9VXbqOvL\nDnR9HXqkEjye0jO543R9qViIiBwGMyMtKZ60pHiG9mq+66u+3lFasf+LMZRGRyqbd1ex+LPd7KzY\nf8i6CbEx9E6Np39aHH1T4+mTGk/v7rH0TomjV/c4spNjyE6JpVtyKqT1jeSPq2IhIp2Ec1BfB/W1\nLTzqoK7m4OcHpmuamFcLdU28Rlu1r6shpr6O7PpasutrGXVgeaOMybW4pFrqa2sOeu0YV4dVOagC\ntje/aQpSv864G1+O6OaPaLEws4nA74BY4Ann3N2NlicCTwLjgFLgUufcBm/ZLcB0oA74kXPu9Uhm\nFelU6uvBhdoxHulOsYnXaGpH12L7Zt6zrpn3bKm9O3R8oF3FxLXwiA38Gxt/8POGR1wixKS02N5i\n4oht6fW9tlV1Rvl+R3m1o6zasavKkdxrSMR//IgVCzOLBR4GzgSKgUVmttA593FQs+nALufcUDOb\nDNwDXGpmI4DJwEigH/CWmR3tnN+flghwDlx94F+Cpl299zx4uqEdYbYLfj0XZruGZYTZrvF0Z/85\nCLNdwzLCe+1wd4zh7kQbsvrCgnZyjXeM8YfuKIN2dMTEQXw3SExtecfb5Os3syOObWbHHXb7hjZN\nvGdDDosJDDR0AEneo1c7v28kjywmAIXOufUAZjYfmAQEF4tJwB3e9ALgIQuM5kwC5jvnqoEiMyv0\nXu/9Nk+5bRW8cDW+7JxwzYSSiLIYwAK//AemG3YGQdMHnodq17CMZtpZMztRb8cZ7k631TvFZnbc\nTe0Um23f1KNzncUjbSOSxaI/sCnoeTFwQnNtnHO1ZlYGZHrzP2i0bv/Gb2BmM4AZAIMGDTq8lHFJ\n0Gt4mDsQWrGjiWkIGeaOxg5d1my7FtYJma+p12jLHWYH/TkOeh8Raa1OPcDtnJsFzALIy8s7vD/T\nM4+CS55sy1giIlEnkseTmzn4Pl8DvHlNtjGzOCCdwEB3OOuKiEg7iWSxWAQMM7NcM0sgMGC9sFGb\nhcBUb/pi4G3nnPPmTzazRDPLBYYBH0Uwq4iItCBi3VDeGMT1wOsETp2d45xbZWZ3AfnOuYXAbOAp\nbwB7J4GCgtfueQKD4bXAD6LyTCgRkU7CnIuOM3Ly8vJcfn6+3zFERDoVMytwzuWFaqdz4EREJCQV\nCxERCUnFQkREQlKxEBGRkKJmgNvMdgAbj+AlsoCSNorTlpSrdZSrdZSrdaIx12DnXHaoRlFTLI6U\nmeWHc0ZAe1Ou1lGu1lGu1unKudQNJSIiIalYiIhISCoWX5jld4BmKFfrKFfrKFfrdNlcGrMQEZGQ\ndGQhIiIhqViIiEhIUV8szGyimX1iZoVmdnMTyxPN7Dlv+YdmlhO07BZv/idmdlY75/qJmX1sZsvN\n7O9mNjhoWZ2ZLfUejS/7Hulc08xsR9D7fydo2VQzW+s9pjZeN8K57gvK9KmZ7Q5aFsntNcfMtpvZ\nymaWm5k94OVebmbHBy2L5PYKlesKL88KM3vPzI4LWrbBm7/UzNr06pxh5DrNzMqC/r9+FrSsxc9A\nhHPNDMq00vtMZXjLIrm9BprZO96+YJWZ3dBEm/b5jDnnovZB4NLo64AhQAKwDBjRqM33gUe96cnA\nc970CK99IpDrvU5sO+b6GpDsTX+vIZf3fK+P22sa8FAT62YA671/e3rTPdsrV6P2PyRwSfyIbi/v\ntb8CHA+sbGb5OcBrgAEnAh9GenuFmevkhvcDzm7I5T3fAGT5tL1OA/56pJ+Bts7VqO15BO690x7b\nqy9wvDedCnzaxO9ku3zGov3IYgJQ6Jxb75zbD8wHJjVqMwmY500vAE43M/Pmz3fOVTvnioBC7/Xa\nJZdz7h3nXKX39AMCdwuMtHC2V3POAt50zu10zu0C3gQm+pTrMuDZNnrvFjnn/kngXizNmQQ86QI+\nAHqYWV8iu71C5nLOvee9L7Tf5yuc7dWcI/lstnWu9vx8fe6cW+xN7wFWA/0bNWuXz1i0F4v+wKag\n58UcuqEPtHHO1QJlQGaY60YyV7DpBP5yaJBkZvlm9oGZXdBGmVqT6yLvcHeBmTXc/rZDbC+vuy4X\neDtodqS2Vziayx7J7dVajT9fDnjDzArMbIYPeU4ys2Vm9pqZjfTmdYjtZWbJBHa4LwbNbpftZYEu\n8rHAh40WtctnLGJ3ypO2YWZTgDzgq0GzBzvnNpvZEOBtM1vhnFvXTpH+AjzrnKs2s2sJHJV9vZ3e\nOxyTgQXu4Dsr+rm9OjQz+xqBYnFq0OxTve3VC3jTzNZ4f3m3h8UE/r/2mtk5wJ8I3Fa5ozgP+I9z\nLvgoJOLby8y6EyhQP3bOlbfla4cr2o8sNgMDg54P8OY12cbM4oB0oDTMdSOZCzM7A7gVON85V90w\n3zm32ft3PfAugb822iWXc640KMsTwLhw141kriCTadRFEMHtFY7mskdye4XFzI4l8H84yTlX2jA/\naHttB16m7bpfQ3LOlTvn9nrTrwLxZpZFB9henpY+XxHZXmYWT6BQPOOce6mJJu3zGYvEoExHeRA4\nclpPoFuiYVBsZKM2P+DgAe7nvemRHDzAvZ62G+AOJ9dYAgN6wxrN7wkketNZwFraaKAvzFx9g6Yv\nBD5wXwymFXn5enrTGe2Vy2s3nMBgo7XH9gp6jxyaH7A9l4MHHz+K9PYKM9cgAuNwJzeanwKkBk2/\nB0xsx1x9Gv7/COx0P/O2XVifgUjl8panExjXSGmv7eX97E8C97fQpl0+Y222oTvqg8CZAp8S2PHe\n6s27i8Bf6wBJwAveL85HwJCgdW/11vsEOLudc70FbAOWeo+F3vyTgRXeL8sKYHo75/o1sMp7/3eA\n4UHrXuNtx0Lg6vbM5T2/A7i70XqR3l7PAp8DNQT6hKcD1wHXecsNeNjLvQLIa6ftFSrXE8CuoM9X\nvjd/iLetlnn/z7e2c67rgz5fHxBUzJr6DLRXLq/NNAInvQSvF+ntdSqBMZHlQf9X5/jxGdPlPkRE\nJKRoH7MQEZE2oGIhIiIhqViIiEhIKhYiIhKSioWIiISkYiHSAXhXW/2r3zlEmqNiISIiIalYiLSC\nmU0xs4+8exc8ZmaxZrbXu5/GKgvceyTbazvGu3jhcjN72cx6evOHmtlb3sXyFpvZUd7Ld/cuzrjG\nzJ7xrn4s0iGoWIiEycyOAS4FTnHOjQHqgCsIXOYh3zk3EvgH8HNvlSeBm5xzxxL4Zm3D/GeAh51z\nxxH4hvnn3vyxwI8J3EtlCHBKxH8okTDpqrMi4TudwIUTF3l/9HcDtgP1wHNem6eBl8wsHejhnPuH\nN38e8IKZpQL9nXMvAzjnqgC81/vIOVfsPV9K4FpF/478jyUSmoqFSPgMmOecu+WgmWa3N2p3uNfQ\nqQ6arkO/n9KBqBtKJHx/By727luAmWV4N1uKAS722lwO/Ns5VwbsMrMve/OvBP7hAnc7K264CZMF\n7gGf3K4/hchh0F8uImFyzn1sZrcRuCtaDIErlP4AqAAmeMu2ExjXAJgKPOoVg/XA1d78K4HHzOwu\n7zW+3Y4/hshh0VVnRY6Qme11znX3O4dIJKkbSkREQtKRhYiIhKQjCxERCUnFQkREQlKxEBGRkFQs\nREQkJBULEREJ6f8Ds6UvfTZ1pJkAAAAASUVORK5CYII=\n", 261 | "text/plain": [ 262 | "" 263 | ] 264 | }, 265 | "metadata": {}, 266 | "output_type": "display_data" 267 | } 268 | ], 269 | "source": [ 270 | "# summarize history for loss\n", 271 | "plt.plot(history.history['loss'])\n", 272 | "plt.plot(history.history['val_loss'])\n", 273 | "plt.title('model loss')\n", 274 | "plt.ylabel('loss')\n", 275 | "plt.xlabel('epoch')\n", 276 | "plt.legend(['train', 'test'], loc='upper right')\n", 277 | "plt.show()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 31, 283 | "metadata": { 284 | "scrolled": false 285 | }, 286 | "outputs": [ 287 | { 288 | "data": { 289 | "text/plain": [ 290 | "(46.19725, 22.895014)" 291 | ] 292 | }, 293 | "execution_count": 31, 294 | "metadata": {}, 295 | "output_type": "execute_result" 296 | } 297 | ], 298 | "source": [ 299 | "#the mean score of the high-rank distribution should be higher\n", 300 | "np.mean(base_network.predict(X_1_test)), np.mean(base_network.predict(X_2_test))" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": 32, 306 | "metadata": {}, 307 | "outputs": [ 308 | { 309 | "data": { 310 | "text/plain": [ 311 | "1.0" 312 | ] 313 | }, 314 | "execution_count": 32, 315 | "metadata": {}, 316 | "output_type": "execute_result" 317 | } 318 | ], 319 | "source": [ 320 | "#AUC\n", 321 | "(np.sum(base_network.predict(X_1_test) > base_network.predict(X_2_test))+0.0) / X_1_test.shape[0]" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [] 330 | } 331 | ], 332 | "metadata": { 333 | "kernelspec": { 334 | "display_name": "Python 2", 335 | "language": "python", 336 | "name": "python2" 337 | }, 338 | "language_info": { 339 | "codemirror_mode": { 340 | "name": "ipython", 341 | "version": 2 342 | }, 343 | "file_extension": ".py", 344 | "mimetype": "text/x-python", 345 | "name": "python", 346 | "nbconvert_exporter": "python", 347 | "pygments_lexer": "ipython2", 348 | "version": "2.7.13" 349 | } 350 | }, 351 | "nbformat": 4, 352 | "nbformat_minor": 2 353 | } 354 | -------------------------------------------------------------------------------- /assets/siamese-net.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eggie5/RankNet/c814f6ab0fd6bc638c224c4a967a9a455891b3a5/assets/siamese-net.png -------------------------------------------------------------------------------- /base.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eggie5/RankNet/c814f6ab0fd6bc638c224c4a967a9a455891b3a5/base.png -------------------------------------------------------------------------------- /meta.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eggie5/RankNet/c814f6ab0fd6bc638c224c4a967a9a455891b3a5/meta.png --------------------------------------------------------------------------------