├── .gitignore ├── CONTRIBUTING.md ├── notebooks ├── README.md ├── pytorch_mine.ipynb ├── comparison.ipynb ├── pytorch_mine_vs_nce.ipynb └── pytorch_mine_vs_nwj.ipynb └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints/ 2 | .DS_Store 3 | __pycache__ 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## CONTRIBUTING 2 | You can contribute to this repository in the following ways: 3 | 4 | 1. Add relevant research papers 5 | 2. Code implementations in any framework 6 | 3. Links to other blog posts or videos 7 | 4. New ideas and approaches in the field 8 | 9 | Just open a PR with required changes for review. 10 | -------------------------------------------------------------------------------- /notebooks/README.md: -------------------------------------------------------------------------------- 1 | ## Implementation Details 2 | 3 | The notebooks in this folder implement estimation of Mutual Information using different lower-bound objective functions described in the papers mentioned in the repository. 4 | 5 | Some of these techniques include the Mutual Information Neural Estimation (MINE), Nguyen Wainwright Jordan (NWJ) loss, Jensen Shannon Divergence (JS) loss, Noise Contrastive Estimation(NCE) loss, etc. The neural network is trained using PyTorch and the calculated MI estimates are plotted against the MI obtained using the traditional method (the basic definition of MI). 6 | 7 | These lower bounds have been compared in the analysis done by Poole et al. in [On variational lower bounds of mutual information](http://bayesiandeeplearning.org/2018/papers/136.pdf) and this paper was recently presented at NeurIPS, 2018. 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Information Theory in Deep Learning 2 | This repository contains implementations (mostly in PyTorch), relevant resources and lessons related to information theory of deep learning. The aim is to have a single source for all the information on the topic. 3 | 4 | ## Resources 5 | 6 | ### Videos and Talks 7 | 8 | - [Stanford Seminar - Information Theory of Deep Learning, Dr. Naftly Tishby, 2018](https://www.youtube.com/watch?v=XL07WEc2TRI) 9 | - [Information Theory of Deep Learning (Yandex) - Naftali Tishby, 2017](https://www.youtube.com/watch?v=dPhsU0bu4LY) 10 | 11 | ### Research Papers 12 | 13 | [1] [Naftali Tishby and Noga Zaslavsky. “Deep learning and the information bottleneck principle” IEEE Information Theory Workshop (ITW), 2015](https://arxiv.org/pdf/1503.02406.pdf) 14 | 15 | [2] [Ravid Schwartz-Ziv and Naftali Tishby. “Opening the Black Box of Deep Neural Networks via Information” ICRI-CI, 2017](https://arxiv.org/pdf/1703.00810.pdf) 16 | 17 | [3] [Naftali Tishby, Fernando C. Pereira, and William Bialek. "The information bottleneck method"](https://arxiv.org/pdf/physics/0004057.pdf) 18 | 19 | [4] [Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, R Devon Hjelm, "Mutual Information Neural Estimation" ICML, 2018](https://arxiv.org/abs/1801.04062) 20 | 21 | [5] [Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, George Tucker1, "On variational lower bounds of mutual information", NeurIPS, 2018](http://bayesiandeeplearning.org/2018/papers/136.pdf) 22 | 23 | [6] [R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio, "Learning deep representations by mutual information estimation and maximization", [stat.ML] 3 Oct 2018](https://arxiv.org/abs/1808.06670) 24 | 25 | ### Blog posts and Articles 26 | 27 | - [Lilian Weng’s blog post: Anatomize Deep Learning using Information Theory, Sep 2017](https://lilianweng.github.io/lil-log/2017/09/28/anatomize-deep-learning-with-information-theory.html) 28 | - [My own blog post on Information Theory of Deep Learning, Dec 2018](https://adityashrm21.github.io/Information-Theory-In-Deep-Learning/) 29 | -------------------------------------------------------------------------------- /notebooks/pytorch_mine.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "The code is adapted from [Masanori Yamada's repository](https://github.com/MasanoriYamada/Mine_pytorch) on MINE implementation." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": { 14 | "colab": {}, 15 | "colab_type": "code", 16 | "id": "BS2h2kglwPa_" 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "import torch\n", 21 | "from torch.autograd import Variable\n", 22 | "import torch.nn as nn\n", 23 | "import torch.nn.functional as F\n", 24 | "from torch.utils.data import Dataset, DataLoader\n", 25 | "import numpy as np\n", 26 | "import matplotlib.pyplot as plt\n", 27 | "from tqdm import tqdm" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": { 34 | "colab": {}, 35 | "colab_type": "code", 36 | "id": "Ze6Kk6SsQKiE" 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "# data\n", 41 | "np.random.seed(1234)\n", 42 | "var = 0.2\n", 43 | "def func(x):\n", 44 | " return x\n", 45 | "\n", 46 | "def gen_x():\n", 47 | " return np.sign(np.random.normal(0.,1.,[data_size,1]))\n", 48 | "\n", 49 | "def gen_y(x):\n", 50 | " return func(x)+np.random.normal(0.,np.sqrt(var),[data_size,1])" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 3, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "np.random.seed(1234)\n", 60 | "data_size = 1000000\n", 61 | "x = gen_x()\n", 62 | "y = gen_y(x)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 4, 68 | "metadata": { 69 | "colab": { 70 | "base_uri": "https://localhost:8080/", 71 | "height": 34 72 | }, 73 | "colab_type": "code", 74 | "id": "C_wM4ty1Qldx", 75 | "outputId": "2892779e-b4dc-4f29-9858-28afb707d11d" 76 | }, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "Mutual information calculated through traditional method is: 0.6584537102332939\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "x = gen_x()\n", 88 | "y = gen_y(x)\n", 89 | "p_y_x = np.exp(-(y - x)**2 / (2 * var))\n", 90 | "p_y_x_minus = np.exp(-(y + 1)**2 / (2 * var))\n", 91 | "p_y_x_plus = np.exp(-(y - 1)**2 / (2 * var))\n", 92 | "mi = np.average(np.log(p_y_x / (0.5 * p_y_x_minus + 0.5 * p_y_x_plus)))\n", 93 | "#mi = mutual_information(x, y)\n", 94 | "print(\"Mutual information calculated through traditional method is:\", mi)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 5, 100 | "metadata": { 101 | "colab": {}, 102 | "colab_type": "code", 103 | "id": "Y9qfiZYqS-0R" 104 | }, 105 | "outputs": [ 106 | { 107 | "name": "stderr", 108 | "output_type": "stream", 109 | "text": [ 110 | "100%|██████████| 500/500 [00:05<00:00, 84.49it/s]\n" 111 | ] 112 | } 113 | ], 114 | "source": [ 115 | "np.random.seed(1234)\n", 116 | "H=10\n", 117 | "n_epoch = 500\n", 118 | "data_size = 20000\n", 119 | "\n", 120 | "class Net(nn.Module):\n", 121 | " def __init__(self):\n", 122 | " super(Net, self).__init__()\n", 123 | " self.fc1 = nn.Linear(1, H)\n", 124 | " self.fc2 = nn.Linear(1, H)\n", 125 | " self.fc3 = nn.Linear(H, 1)\n", 126 | "\n", 127 | " def forward(self, x, y):\n", 128 | " h1 = F.relu(self.fc1(x)+self.fc2(y))\n", 129 | " h2 = self.fc3(h1)\n", 130 | " return h2 \n", 131 | "\n", 132 | "model = Net()\n", 133 | "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n", 134 | "plot_loss = []\n", 135 | "for epoch in tqdm(range(n_epoch)):\n", 136 | " x_sample=gen_x()\n", 137 | " y_sample=gen_y(x_sample)\n", 138 | " y_shuffle=np.random.permutation(y_sample)\n", 139 | " \n", 140 | " x_sample = Variable(torch.from_numpy(x_sample).type(torch.FloatTensor), requires_grad = True)\n", 141 | " y_sample = Variable(torch.from_numpy(y_sample).type(torch.FloatTensor), requires_grad = True)\n", 142 | " y_shuffle = Variable(torch.from_numpy(y_shuffle).type(torch.FloatTensor), requires_grad = True) \n", 143 | " \n", 144 | " pred_xy = model(x_sample, y_sample)\n", 145 | " pred_x_y = model(x_sample, y_shuffle)\n", 146 | "\n", 147 | " ret = torch.mean(pred_xy) - torch.log(torch.mean(torch.exp(pred_x_y)))\n", 148 | " loss = - ret # maximize\n", 149 | " plot_loss.append(loss.data.numpy())\n", 150 | " model.zero_grad()\n", 151 | " loss.backward()\n", 152 | " optimizer.step()" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 6, 158 | "metadata": {}, 159 | "outputs": [ 160 | { 161 | "data": { 162 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4XNWB/vHv0ahbxbaKLVuW5W4LVxCmGDCEZkqAJEAwSSCbQhpJNoX8SEJJSGOBTVgSsoQQfpslhYQS7IAphhgw4I57kYss22pW72Xa2T9mNJZkybaskUaaeT/Po0dzZ47uPWckvffMufeea6y1iIhIZIkKdQVERGTwKfxFRCKQwl9EJAIp/EVEIpDCX0QkAin8RUQikMJfRCQCKfxFRCKQwl9EJAJFh7oCvUlPT7e5ubmhroaIyLCyadOmKmttxsnKDdnwz83NZePGjaGuhojIsGKMOXQq5TTsIyISgRT+IiIRSOEvIhKBFP4iIhFI4S8iEoEU/iIiEUjhLyISgRT+IhIWrLW8sKmYpnZ3qKsyLCj8RSQsbC+p5zvPbeXel3aEuirDgsJfRIal+hYXHq8NLNe1uAAorGoetDq0Oj1sPVI3aNsLJoW/yDBhre0Sdj05UtPC2wUVp7Q+j9fS5vKctNyyLSVc/5v3OHgaoWqtpd198m20Oj2nVK6D0+1l3gNvcE+nXv7RhjYASuta+eazm2loc1Hd1M5dz22lttnZ67rcHi/f+ftWdpU2nPL2i6qaqW9xcedfPuT6x9+n3r/jAV+bre36e3J7vLg83uPW8+z6w/z7s5tPebvBpPAXGcJWbC/j9Z3llNa1cvbP3uQLf9xwXJllW0p46LU9AHz0N+/x2f+/gd++vZ8n3jnQpdz7+6vYU+4LuBc2FTPlByuY9+M3cLq9/PKNAioa2yipa6WwsinwM7tKG/j+i9vZWlzPix8W91jHdrenS9i1uTx889nNrC2s5r/e2seMe15jf0UTzd3G4h96bQ8vbS4BYNZ9r/Gxxz844Xvh9nhZsb2Mq/5rdaAdf11/mMdX7Wfxw6t49M19AFQ2trNsSyl/WnuIR94o4LlNxfxp7SHe2VvZ43qLqpt54cNirn5sNe/6y7y7t5LNh2sDZVqdvja9sKmYbcV1XPzI29yzbAdv7fHtaPd3es+eeKeQxQ+/3WVn8qVnNnHNY6tpc3l4+PU9fPXPm3hmTRF3v7idl7aUsqe8gfyfrmT9wZrjdhwDxQzWhvoqPz/fnvbEbq/eDeXbg1shGTReaylvaGNsSjwWMECUMX1ax4HKJhxRhty0EX3evtPjpdXpITUhBgCLpay+DQOkJ8VhDERHReH2eimqbiF7VALx0Y7AzxfXttDq8jAtM5ma5naanR4mjErE5fVS2+yktsVFbloiVU1ORsQ5GJkQG/jZVpeb8vp2xqbGEx1l2OQPoNy0ERRV+3reeVkpxEZHERcdxb6KJmr8vdpzJo1m3cGaLm05a+IoYqKisNjAa93LJcdF09juZmxKPOX+3vM5k0bj9lo2HToWgImxDvKyUrBATJSv3+i1lvVFNYxNiSc3bQQlda0cqW0BYFRiLK1OD23+Hn1cdBQLJozCYimtawuU61yfGWOSGZnoe98NXX/nB6ubA737U5GRFEdDm4t297Ee95zxqYyIjcbl8bKrrIEpGUl4vJbd/p1JUlw0rU4PHn8unpkziugow5HaFsrqfdsenRhLTUvXTxKT00fQ5vaSNiKWfUebaHN7cBjD7PGpOKIMH/p/jx3vsQE6J298jCPwKSzKGPIWLCLphkdOua2dGWM2WWvzT1ZuyM7qKcNfTYuTuOgoRsT27c+svKGNwzUtRBnfP110lGHBhFEAeKzF6faSEOPo9ecb211UNrUDnHL417Q4aWpzkzM6kS1H6vBay+T0EXi8lsTYaA7X+ILqUE0L8TEO5mePpLi2laqmdqK77WSK61oBmGwteyt8PcIxKfHsKmsI/IO3uzy0uDyMTIgJhP/RxrbA0MrRxq4h1+I81mveVeYLqlljUwLBD1BS23pcuzYdqiVndCJpI47tYLrvIBr9PfLyTsG65UgdMY5jAwOxjihanB62HKnD7bUkxjqYOHoETv9QRnlDG8nx0RTXtRAf7aDN7aG53R14HaDd7aWisY3y+jZaOg03bS+uDzwuONoI+Hay2aMSqGhop93tIXtUItX+32lPJoxKDOxMOlT2UL6ysR1HqqG+1UWry8Oh6mZcHl8Mj0yIoa7VdVz5yqZ22lweYhxRuDzeLsEfE2VweS0Vje00tbspr2/Day1ZqfFUNLaztbjr8YCO9zg/dzTbi+to8++YOg+/ea3laGMbSb22NjjCM/yvejDUNRDgzLtfAaDowWt6LbOxqAan28v5U9Nxe7zct3wnO5vr2eqs5/78PH78z10A7L/tKqIdUXz72c0s21LKuh9cSmZyHD9fsZs52SO5bt44Supaueu5rXxQVB1Y/8HPXg1AU7ub6KgoEmIdrDlQzYrtZdx7bR6x0VHc+9IOninwzYK77OZF3Pz4+74fLvV9i3GYQEAA4ITvTpzOIwV7AZgXn8qyf7uA5zcV89hb+zjs9IVQzIFjP3eeN401zdXMGJOMMbCn3BdyiThYu/RS9h1t5BP/vSawiZljkwNlAPJiU6hPdFFSdyzgL4xL50Nq+c+b5/HlP30IvQ31l8PIxBjqnF2D7d5r8/jJy77398Jp6azeV9WljZ194YJJPPXeQQCiowxup4WmrmXwjwr94fZ8XtlWxov+IZ1pmUns8+8EO97TLnoYjk9tjSG63VDt37ndMWcyTx0s5CsXT+HaueP4yp820dTu5rFbFhAX42BczkhefreQmmYnv3u3sMu6unzSqYKUpmhmj0/lA2d1l20/9NG5fO+FbV0rcvTYw69dMoVn1hyioc3N5IwRpMTH8MhN8/j237ewrdMOLDrK8MEXP8IH64/wqzf3Bp5/4tNn+n5PQNHnrqFkWxlf+8uHgde/f9VM/rzuMEdqW/hi2mR+0MNbFUzhGf4yaH6xYjfv7qtixTcuwHQamml1nvjgnbWWn72ym6feO8jIxBjW3H0pu8sb+Mu6w4EyHcEP8PAbBVQ3OVm2xZced7+wjZvyJ/D71b5Aand5uOv5bv+4wPkP/otzJo1mxY5y4qKj+OedF7D092sByB6VwB0XTeaFTmPZv/7X/uPW0RHgnz0/l//5oAiAR97w/VNfecYYXt95lBn3vIrL46Xz8djOO4w1hdVcMDWdP33hHF7YVMx3ntuKMdDi9DD3R2902V5mchwrvnEhX392M9uL6zlc08KusgY+dU4Of+70/qzeV8X5U9KYknGsj/jFCyexbEspyfHRHKg8doC240yYPT9Zwh/eO8jDrxdwzZysQPj/4uNzWLnrKFMykpgwOpHxIxO445mNvF3gGwO/as5YnnrvIJPTR/CPry5i3gNd69whOT6ahZNGB4Y5AL575Qxy00Zw5aPvctH0DJ66LZ/nNh3ht6sO8NjS+ZTWtfH1v/oOel6RN4YLp2dw70s7cEQZlt+5iC8/s4kn/YE+Y2wKs7JSeOs7F+PxWmKjj306+dLiKVhrWballJvys4mPcfDw6wVcO29cIPw/cWY2L3xYzAcHquluyZyxXcL/u1dMZ+WuoyyZnUV0lOHmsyewp6yRt/ZUsGhKOj+5YTYAnzx7Qpfwv2pOFpkp8Xzq3Bz+vvFIYId9Rd5Yxo9M4Np5WQBcMzeL3PQLeGPnURbPyODMnFF8afEUrv/Ne10+DQ2U8BzzlwGxoaiGZ9Yc4lefnI8jytDm8jDz3tcAePPbi1l3sJoJoxK57en1PHVbPl/4X9/vb/0PL+WdgkpuPCsbYwxrC6v587rD/HPrsW5gSnw02aMSA0Map2P+hJFsOclpd8lx0bS5PWQmx+OIMoxJiWNDUS33fzSPX63cS0ObmygDPZ1UU/TgNVhr+dPaQ+yvaOL6BePJy0rhmsdWB4L21nNy+OeW0sBQyq4HrsTlsdQ2OxmbGk98jIODVc1c8sjb3HXlDDxeiwEmZYygptnJfct2MjIxhi33XQHAh4dr+fhvfQdC//Omeby15ygrtpfz/atm8otX93DPNbP41DkTmXWf7/dw4OdX44gyrD9Yw82/W8OsrBR+8fE53PD4+yzMHc3fv3weAC6PlxhHFE+tLuTcyWnMHp96XHu9XsvvVxfyi1f3sP1HV7C/ookpmUmkxMewqqCCdwoqAztDgB9cPZPbzsslPsZBc7ub9UU1TM1IIntUAsYYdpbWM2NMMtGO488z6dgh3npODrcuzOHaX7/H5y+YxL3X5rHowX8FAnTTPZeRlhR3wt9xZx6vZfPhWm58Yg2jR8Sy6Z7LOPtnb1LV5OSrF0+hzeXl6fcPBn6/aw5U893ntlJS18qW+y5nZGJsl/X9Zd1hfvCP7SxdOIFffHxuYBsrd5UzKjGWR9/cx29uXdCljv/z/kFmj08lP3c01tounaSe/G3DYZwey2fOnXjK7ezsVMf8Ff5yyi5+eBVF1S3846vnsyBnFG/uOhoI+O7mjE9le0nX3su1c7NweyzbS+qpa3HS3MOng1GJMXztkqn89JXdgedGJsZQ1+Li5a9fwL//bQv/8Ym53LdsBztLG7hu3jiW+3cim+65jL+uPxzolXeXHB9NY5ubm/OzWZAziu+/eOykgBe+ch41zS7uen4rV+SN4R+bS3B5fOP+HeeNn2j4atWeCmpbnHxswXiMMWwvricnLTFw0Li77cX15I1LwRF1LAjqWpzMf2Bl4BMCQHO7mzPufx2A1d+7hMyUOI7Wt5OTlsiRmhbGjUzAEWV4fWc5c7NTyUpNAKC22cmCn6zkd585iyvyxvDSlhIumpbRp+AE3yc0t9d2Gf/vsK6wmk8+uZbL88YwJSOJb146jYTY3o/FnEir08NPX9nFty+fTlpSHG8XVHDu5DTiYxxc++vV7ChpYPX3LmHC6MQ+r7uqqZ38n77JtMwkVn57MeX1bby4uZgvXTQFR5ShuLYFp9vLZP8nKI/XUt3UTmZK/HHranN5uH/ZTr588RQmpff9ZILBoPCX41Q2tpOaENPlozL4Du6lJsQwMiGGlbuPckXeGOJjHMT5y339r5u5eEYmf/ygiO0l9Xzrsul887Jp3P3CNl7ZVsa187L46/ojfarLj687g4/MzOSxt/Zx1ZyxvPhhCTPHJnPdvPHkpCWS6z9eAPDiV89nSnoSqYnHgrSisY1VeypYMjuLQ9XN1DQ7uXhGJgAvbyvlzr9sJsrAlWeMpdnpISbK8G+LJvHDl7bzh9vzyUyJDwy3zBybzEtfW0R8p4PINzz+PluO1PG7z5yFtZZxIxOYmz2yb2/4aVi9r5I541O79DiXbSlh5a6j/HrpgpP2GgeTtZZn1h7i2rnjGD0i9uQ/cJoOVTez5kA1tyzMOe11/O+aIi6enklOWt93HsONwl+6OFLTwoUPrWJqZhIrvnEhh2uamZSehMvjZea9rxFl4Jq547oMxdx15QwWT8/g2l+/d9z6PrZgPKsKKjh3UhpPfOYsSuta+eXKvTy/6dj4eawjis+cN5E/+A8UfuGCSSzbWkplYztvfnsxUzN7P59h8cOrOFTtO3C68Z7LSO9jj7Wzzh+1Oz9+u6CCQ9Ut3H5+7nE/U1TVzL3LdvCbpWd22emIDHUK/wiyqqCCxjY3180b1+PrawurueXJtYHlxdMzeGdvJbOyUrj1nJxe50KZlZXCBVPTAgdVAS7PG0N0lOHVHeUA/L8lM/nKxVMAqGl2cuZPVrJ04QTuXjKLulYnibHRnP2zN/m3Rbnc/9EzaHG62V/RdNJedH2Li4Y2F7HRUYzp4eO3iPRM4R9BOoZIRsQ6+NF1Z3BT/gSstawprObcSWnc9fy2Lme0nEhirIOt91/B957fxj/8p+p1tvJbFzFtTHJgm3/6/DlcMC098HpVUztJcdFdhlCKqpoZNzLhuOEmEQm+Uw1//TcOY16vZcmj7waWm50efvrKbt7fX8XL28q49ffreHJ1YZfL1Du88a2LiPYfbEyJP3bG79VzsohxRDEv+9jZHzn+g2z3XDOLaWOSAd85yQCzx6d0WW96UlyX4AfITR+h4BcZYoJynr8xZgnwX4ADeMpae9xVVsaYm4Ef4buqeau19tZgbDuSbTpc2+VCIID6VhefempdYPnBV/d0ef2j88aRkRTH9DHJLJk9lpe3lfHEp88i2hFFZnIcY1N9QyyfOCsbt9fy8TOzaXV5uH/ZDm46a0JgPXdcNJnPnDeRxD5evSsiQ0O/h32MMQ5gL3A5vmv8NgBLrbW7OpWZBvwd+Ii1ttYYk2mtPeHUgxr2Obm7ntvKc50OsE4YnUBu2oiuV2p20/nc5aZ2N+0uT59P/xORoWswh30WAvuttYXWWifwLHB9tzJfBB631tYCnCz45eSeWVPEc5uKue28iTz6yfkAOIzh6c+ezRvfuui48h3nk3c+7zwpLlrBLxKhgvGZfTzQ+STvYuCcbmWmAxhj3sc3NPQja+1r3VdkjLkDuAMgJ+f0z+kNR51PUWx3e3jo9QImZ4zgm5dOC8w2GBVliHFEMa3TKZSFP7+awqpmUhNiOFzTMqTOExeR0AlG+PeUJt3HkqKBacDFQDaw2hgz21rb5Vp8a+2TwJPgG/YJQt3CQmObi3N//hbfvXIGSxfmcMPj79PY5ubXS32XkUf7p9f93KJJABhj+OHVs8hMiSMqygTOp89IVi9fRHyCEf7FwIROy9kcP3dfMbDWWusCDhpjCvDtDI6/M4Uc58UPS2h2evjxP3cFJjv7zLkTuXBaBgCpiTHHTT3wxYsmD3o9RWT4CMaY/wZgmjFmkjEmFrgFWN6tzEvAJQDGmHR8w0CFyElZa/njmqIuz115xhh+csPsLvPCiIj0Rb97/tZatzHmTuB1fOP5T1trdxpjHgA2WmuX+1+7whizC/AAd1lrj59TVQKO1LRw29PrAzf3+PnH5jA103fXoTMnDvwcMyIS3nSF7xDUMR1vh3uumcXnFk0iSj19ETkJ3cZxGOuYVuGRm+Zx0bT0HqeWFRHpD4X/EFLf4sJjLav2VHDNnCxuPCs71FUSkTCl8B9CLnzoXzS0+e4AtSBH4/oiMnA029YQsWJ7WSD4AfKyUk5QWkSkfxT+Q0B9i4uv/vnDLs/NVPiLyABS+A8BHxzoOhHblxZPHtDb4omIaMw/xOpbXdy3fGdg+Y+fW8ji6RkhrJGIRAKFfwg1tLm4b9kOKhvbufuqmZTWtXLu5NGhrpaIRACFfwg9/FoBy7b4pkG65ewJgXn2RUQGmsb8Q6ihzRV4rOAXkcGk8A+h+lZf+M/w3xdXRGSwKPxDpKSulW3F9Vw6M5Nldy4KdXVEJMJozD8EXt1exlf85/WfMS6F+BhHiGskIpFG4R8Cj7xRQHpSLN+7ciZL5owNdXVEJAIp/AfRhqIaCiubOFDZzHcun87NZ084+Q+JiAwAhf8guumJY3P0z85ODWFNRCTS6YDvIGl3e7oszx6n8BeR0FH4D5IdJfWBx2fmjCQjOS6EtRGRSKdhn0GyoagWgD/cns/5U9JDXBsRiXQK/0GysaiGyRkjuHTWmFBXRUREwz6Dweu1bDxUS/7EUaGuiogIoPAfFAcqm6hrcZGfqxk7RWRoUPgPgqffLwLgbIW/iAwRCv8Btu9oI39df5jF0zPITUsMdXVERACF/4Byebzct8x3l64Hrj8DY0yIayQi4qPwH0CvbCtjTWE1ADmj1esXkaFD4T+A9h5tBOD3t+Wr1y8iQ4rCf4C8tLmE3759gNy0RC7P07n9IjK0KPwHyP3LfWP9zU7PSUqKiAw+hf8A8HgtHq8F4KEb54a4NiIix1P4D4ADlU00tbv55c3zuGRGZqirIyJyHIX/ANh3tAmAGWN1Y3YRGZoU/gNgf0UTxsCUjKRQV0VEpEcK/wGwr6KR7FEJujG7iAxZCv8BsKu0gRljUkJdDRGRXin8g6y6qZ3CqmbO0vTNIjKEKfyDrGM6h/xchb+IDF1BCX9jzBJjTIExZr8x5u4TlLvRGGONMfnB2O5Q9PiqA0xMS2T+hJGhroqISK/6Hf7GGAfwOHAVkAcsNcbk9VAuGfgGsK6/2xyqqpra2V3WwKfPmUiMQx+qRGToCkZCLQT2W2sLrbVO4Fng+h7K/QR4CGgLwjaHpIJy30RuZ4zTwV4RGdqCEf7jgSOdlov9zwUYYxYAE6y1L59oRcaYO4wxG40xGysrK4NQtcG1u6wB0MVdIjL0BSP8e5qr2AZeNCYK+BXwnZOtyFr7pLU231qbn5GREYSqDZ7aZidPvFNIbloiaUlxoa6OiMgJBSP8i4EJnZazgdJOy8nAbOBtY0wRcC6wPNwO+n5woJqqpnYe/IQmchORoS8Y4b8BmGaMmWSMiQVuAZZ3vGitrbfWpltrc621ucBa4Dpr7cYgbHvI2FpcR6wjijNzdIqniAx9/Q5/a60buBN4HdgN/N1au9MY84Ax5rr+rn+42HqkjlnjUoiN1lk+IjL0RQdjJdbaFcCKbs/d10vZi4OxzaHEWkvB0Uaump0V6qqIiJwSdVODoLrZSV2Li6mZmsVTRIYHhX8Q7K/wzd8/TeEvIsOEwj8I1hzwzeczbYzCX0SGB4V/P7W5PPx+dSGX540hKzUh1NURETklCv9+KihvpMXp4eMLxp+8sIjIEKHw76edpb4pHc4YlxrimoiInDqFfz/tLK0nOT6aCaM15CMiw4fCv592ljaQl5WCMT1NcSQiMjQp/PvB7fGyp7xBQz4iMuwo/PuhsKqZNpdX8/eLyLCj8O+H9QdrAJifo1s2isjwovDvh/f2VTEuNZ7J6SNCXRURkT5R+J+mdreH9/dXcdH0DB3sFZFhR+F/mt7bV0Vju5sls8eGuioiIn2m8D9N6w7WEBsdxflT0kNdFRGRPlP4n6bdZQ1MH5Okm7eIyLCk5DpNe8obmTFGp3iKyPCk8D8N1U3tVDa2MysrOdRVERE5LQr/01BQ3gjAzLHq+YvI8KTwPw27O8JfPX8RGaYU/qdhV2kD6UlxpCfFhboqIiKnReHfRy1ON2/sKmfR1LRQV0VE5LQp/Pvo3b2VNLa5ueXsnFBXRUTktCn8+2hHSQOOKMMCTeYmIsOYwr+PdpU1MDUjifgYR6irIiJy2hT+fWCtZUdJPXmav19EhjmFfx8U17ZS0diuIR8RGfYU/n2w8ZDv5i35E0eHuCYiIv2j8O+DLYfrSIx1MGOsLu4SkeFN4d8He8obmTE2GUeUbt4iIsObwv8UWWvZU96o+XxEJCwo/E9RSV0r9a0uzeQpImFB4X+K3tlbCcD5UzStg4gMfwr/U7RqTyUTRicwJSMp1FUREek3hf8p2l5Sx9kTR2OMDvaKyPCn8D8FVU3tHG1o15W9IhI2FP6nYFdpAwB5WQp/EQkPQQl/Y8wSY0yBMWa/MebuHl7/tjFmlzFmmzHmLWPMxGBsd7BsPFRLlIEzxqeGuioiIkHR7/A3xjiAx4GrgDxgqTEmr1uxzUC+tXYu8DzwUH+3O5jWHqhm9vhUUhNiQl0VEZGgCEbPfyGw31pbaK11As8C13cuYK1dZa1t8S+uBbKDsN1B4fZ42VJcx8JczecjIuEjGOE/HjjSabnY/1xvPg+8GoTtDorDNS043V7N5yMiYSU6COvo6dxH22NBYz4N5AOLe3n9DuAOgJycoXGbxP0VTQBMG6PwF5HwEYyefzEwodNyNlDavZAx5jLgh8B11tr2nlZkrX3SWptvrc3PyMgIQtX6b3+lL/ynZIwIcU1ERIInGOG/AZhmjJlkjIkFbgGWdy5gjFkA/A5f8FcEYZuDwuO1vLS5hBljkkmO18FeEQkf/Q5/a60buBN4HdgN/N1au9MY84Ax5jp/sYeBJOA5Y8wWY8zyXlY3pGwrrmPv0SbuuGhyqKsiIhJUwRjzx1q7AljR7bn7Oj2+LBjbGWy7ynwXdy2cpDN9RCS86ArfE9hd1kByfDTZoxJCXRURkaBS+J/A9pIGZo1N0WRuIhJ2FP69qGxsZ1txHYumpoe6KiIiQafw78WqggqshcvzxoS6KiIiQafw78XawmrSRsTqto0iEpYU/r3YUFTD2bm6eYuIhCeFfw9K61o5UtOqUzxFJGwp/HuwoagG0Pn9IhK+FP49+GB/NUlx0czSnbtEJEwp/Ltpd3t4bWc5l87KxBGl8X4RCU8K/27WFdZQ3+riunnjQl0VEZEBo/Dv5oMD1cQ4DOdNSQt1VUREBozCv5s1B6qYP2EkibFBmfNORGRIUvh30tDmYntJPedNVq9fRMKbwr+T9YU1eC2cN0Xz+YhIeFP4d/LytlKS46JZkDMy1FURERlQCn+/mmYnK7aX8/EzxxMf4wh1dUREBpTC3++FTcU4PV5uPWdiqKsiIjLgFP5+K3aUMS87lRljNYuniIQ/hT/Q2OZiW3E9F07LCHVVREQGhcIfeH9/FR6v5fypOsVTRCKDwh9YtqWU9KRYFuZqFk8RiQwRH/71rS7e2l3BtXPHEe2I+LdDRCJExKfdazvKcHq83LBgfKirIiIyaCI+/F/aXEpuWiLzslNDXRURkUET0eFfXt/G2oPVXD9/vO7VKyIRJaLDf/nWEqxFQz4iEnEiOvxf2lzKvOxUJqWPCHVVREQGVcSG/76jjewqa+D6+er1i0jkidjwf2lLCVEGrp2XFeqqiIgMuogMf2sty7eWcsG0DDKT40NdHRGRQReR4b+rrIEjNa1cM2dsqKsiIhISERn+K3cdxRi4dNaYUFdFRCQkIjb8z8oZRXpSXKirIiISEhEX/iV1rewsbeDyPPX6RSRyRVz4r95bCcBHZmaGuCYiIqETceH/7r5KxqbEMzUzKdRVEREJmaCEvzFmiTGmwBiz3xhzdw+vxxlj/uZ/fZ0xJjcY2+0rj9fy3r4qLpqerrl8RCSi9Tv8jTEO4HHgKiAPWGqMyetW7PNArbV2KvAr4D/6u93Tsa24joY2t27XKCIRLxg9/4XAfmttobXWCTwLXN+tzPXAH/2PnwcuNSHoen9woBqA86fodo0iEtmCEf7jgSOdlov9z/VYxloYI4NeAAAIZ0lEQVTrBuqBQU/gDUU1TM1MIk2neIpIhAtG+PfUg7enUQZjzB3GmI3GmI2VlZVBqNoxHq9lU1EtCyfpPr0iIsEI/2JgQqflbKC0tzLGmGggFajpviJr7ZPW2nxrbX5GRnDH5XeXNdDY7tZN2kVECE74bwCmGWMmGWNigVuA5d3KLAdu9z++EfiXtfa4nv9A2lDk29ecrZ6/iAjR/V2BtdZtjLkTeB1wAE9ba3caYx4ANlprlwN/AJ4xxuzH1+O/pb/b7asPD9eRlRrP+JEJg71pEZEhp9/hD2CtXQGs6PbcfZ0etwE3BWNbp2tHST1zdZN2EREgQq7wbWhzcbCqmTnjFf4iIhAh4b+zpAGA2Qp/EREgQsJ/R0k9gHr+IiJ+ERH+20vqGZcar4u7RET8IiL8d5TUa8hHRKSTsA//xjYXhTrYKyLSRdiH/85S/8FeneYpIhIQ9uG/qyP8xyn8RUQ6hH34H6hsIjUhhvSk2FBXRURkyAj78C+sbGZyxgjduUtEpJOwD/8DlU1MTtf9ekVEOgvr8G9sc1HR2M7kjBGhroqIyJAS1uF/sKoZgCkZ6vmLiHQW1uFfWNkR/ur5i4h0Fubh30SUgZy0xFBXRURkSAnr8D9Q1cyE0YnERTtCXRURkSElrMO/sLKZyeka8hER6S5sw9/rtRysamKyDvaKiBwnbMO/tL6VNpdXp3mKiPQgbMP/UHULAJM07CMicpywDf/SulYAxo9MCHFNRESGnrAN/7L6NgDGpsaHuCYiIkNPWId/elKsTvMUEelBGId/q3r9IiK9CN/wr2sjK1Xj/SIiPQnL8LfWcqS2RQd7RUR6EZbhX9XkpMXpYaLm9BER6VFYhv/hGt9sngp/EZGehWX4d1zglTNaF3iJiPQkLMP/SI3vAq/sURrzFxHpSViGf2VTGyMTY4iP0Tn+IiI9Ccvwr2p0kpEUF+pqiIgMWWEZ/pVN7WQkK/xFRHoTnuHfqPAXETmRsAt/a60v/DXsIyLSq7AL/2anh1aXRz1/EZETCLvwd7m9fHTeOGZlpYS6KiIiQ1a/wt8YM9oYs9IYs8//fVQPZeYbY9YYY3YaY7YZYz7Zn22ezKgRsfx66QIump4xkJsRERnW+tvzvxt4y1o7DXjLv9xdC3CbtfYMYAnwqDFmZD+3KyIi/dDf8L8e+KP/8R+BG7oXsNbutdbu8z8uBSoAdctFREKov+E/xlpbBuD/nnmiwsaYhUAscKCf2xURkX6IPlkBY8ybwNgeXvphXzZkjMkCngFut9Z6eylzB3AHQE5OTl9WLyIifXDS8LfWXtbba8aYo8aYLGttmT/cK3oplwK8AtxjrV17gm09CTwJkJ+fb09WNxEROT39HfZZDtzuf3w7sKx7AWNMLPAP4H+ttc/1c3siIhIE/Q3/B4HLjTH7gMv9yxhj8o0xT/nL3AxcBHzWGLPF/zW/n9sVEZF+MNYOzdGV/Px8u3HjxlBXQ0RkWDHGbLLW5p+03FANf2NMJXCoH6tIB6qCVJ3hQm2ODGpzZDjdNk+01p70dPohG/79ZYzZeCp7v3CiNkcGtTkyDHSbw25uHxEROTmFv4hIBArn8H8y1BUIAbU5MqjNkWFA2xy2Y/4iItK7cO75i4hIL8Iu/I0xS4wxBcaY/caYnqaYHpaMMU8bYyqMMTs6Pdfj/RSMz2P+92CbMebM0NX89BljJhhjVhljdvvvB/FN//Nh225jTLwxZr0xZqu/zT/2Pz/JGLPO3+a/+a+cxxgT51/e7389N5T17w9jjMMYs9kY87J/OazbbIwpMsZs91/4utH/3KD9bYdV+BtjHMDjwFVAHrDUGJMX2loFzf/gux9CZ73dT+EqYJr/6w7gvwepjsHmBr5jrZ0FnAt8zf/7DOd2twMfsdbOA+YDS4wx5wL/AfzK3+Za4PP+8p8Haq21U4Ff+csNV98EdndajoQ2X2Ktnd/plM7B+9u21obNF3Ae8Hqn5e8D3w91vYLYvlxgR6flAiDL/zgLKPA//h2wtKdyw/kL39xRl0dKu4FE4EPgHHwX+0T7nw/8nQOvA+f5H0f7y5lQ1/002prtD7uPAC8DJgLaXASkd3tu0P62w6rnD4wHjnRaLvY/F656u59C2L0P/o/2C4B1hHm7/cMfW/DNkrsS3/0v6qy1bn+Rzu0KtNn/ej2QNrg1DopHge8BHdO9pxH+bbbAG8aYTf7p7GEQ/7ZPOqXzMGN6eC4ST2cKq/fBGJMEvAD8u7W2wZiemucr2sNzw67d1loPMN9/u9N/ALN6Kub/PuzbbIy5Fqiw1m4yxlzc8XQPRcOmzX6LrLWlxphMYKUxZs8Jyga9zeHW8y8GJnRazgZKQ1SXwXDUfx+FjpvldNxPIWzeB2NMDL7g/7O19kX/02HfbgBrbR3wNr7jHSONMR2dtc7tCrTZ/3oqUDO4Ne23RcB1xpgi4Fl8Qz+PEt5txvpua4u1tgLfTn4hg/i3HW7hvwGY5j9LIBa4Bd89B8JVb/dTWA7c5j9D4FygvuOj5HBifF38PwC7rbW/7PRS2LbbGJPh7/FjjEkALsN3EHQVcKO/WPc2d7wXNwL/sv5B4eHCWvt9a222tTYX3//sv6y1nyKM22yMGWGMSe54DFwB7GAw/7ZDfdBjAA6iXA3sxTdO+sNQ1yeI7forUAa48PUCPo9vnPMtYJ//+2h/WYPvrKcDwHYgP9T1P802X4Dvo+02YIv/6+pwbjcwF9jsb/MO4D7/85OB9cB+4Dkgzv98vH95v//1yaFuQz/bfzHwcri32d+2rf6vnR1ZNZh/27rCV0QkAoXbsI+IiJwChb+ISARS+IuIRCCFv4hIBFL4i4hEIIW/iEgEUviLiEQghb+ISAT6Pxbc4FjWYsdTAAAAAElFTkSuQmCC\n", 163 | "text/plain": [ 164 | "
" 165 | ] 166 | }, 167 | "metadata": {}, 168 | "output_type": "display_data" 169 | } 170 | ], 171 | "source": [ 172 | "plot_x = np.arange(len(plot_loss))\n", 173 | "plot_y = np.array(plot_loss).reshape(-1,)\n", 174 | "\n", 175 | "plt.plot(plot_x, -plot_y)\n", 176 | "plt.plot(plot_x, mi*np.ones(len(plot_loss)))\n", 177 | "plt.show()" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": null, 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [] 186 | } 187 | ], 188 | "metadata": { 189 | "accelerator": "GPU", 190 | "colab": { 191 | "name": "pytorch_mine.ipynb", 192 | "provenance": [], 193 | "version": "0.3.2" 194 | }, 195 | "kernelspec": { 196 | "display_name": "Python 3", 197 | "language": "python", 198 | "name": "python3" 199 | }, 200 | "language_info": { 201 | "codemirror_mode": { 202 | "name": "ipython", 203 | "version": 3 204 | }, 205 | "file_extension": ".py", 206 | "mimetype": "text/x-python", 207 | "name": "python", 208 | "nbconvert_exporter": "python", 209 | "pygments_lexer": "ipython3", 210 | "version": "3.6.5" 211 | } 212 | }, 213 | "nbformat": 4, 214 | "nbformat_minor": 1 215 | } 216 | -------------------------------------------------------------------------------- /notebooks/comparison.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "This notebook compares most of the objective functions mentioned in the recent paper at NEurIPS 2018 [On variational lower bounds of mutual information](http://bayesiandeeplearning.org/2018/papers/136.pdf) by [Ben Poole](https://cs.stanford.edu/~poole/) et al. to calculate Mutual Information using an implementation in PyTorch." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import torch\n", 17 | "from torch.autograd import Variable\n", 18 | "import torch.nn as nn\n", 19 | "import torch.nn.functional as F\n", 20 | "from torch.utils.data import Dataset, DataLoader\n", 21 | "import numpy as np\n", 22 | "import matplotlib.pyplot as plt\n", 23 | "from tqdm import tqdm" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 2, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "# data\n", 33 | "np.random.seed(1234)\n", 34 | "var = 0.2\n", 35 | "def func(x):\n", 36 | " return x\n", 37 | "\n", 38 | "def gen_x():\n", 39 | " return np.sign(np.random.normal(0.,1.,[data_size,1]))\n", 40 | "\n", 41 | "def gen_y(x):\n", 42 | " return func(x)+np.random.normal(0.,np.sqrt(var),[data_size,1])" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "np.random.seed(1234)\n", 52 | "data_size = 1000000\n", 53 | "x = gen_x()\n", 54 | "y = gen_y(x)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 4, 60 | "metadata": {}, 61 | "outputs": [ 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "Mutual information calculated through traditional method is: 0.6584537102332939\n" 67 | ] 68 | } 69 | ], 70 | "source": [ 71 | "x = gen_x()\n", 72 | "y = gen_y(x)\n", 73 | "p_y_x = np.exp(-(y - x)**2 / (2 * var))\n", 74 | "p_y_x_minus = np.exp(-(y + 1)**2 / (2 * var))\n", 75 | "p_y_x_plus = np.exp(-(y - 1)**2 / (2 * var))\n", 76 | "mi = np.average(np.log(p_y_x / (0.5 * p_y_x_minus + 0.5 * p_y_x_plus)))\n", 77 | "#mi = mutual_information(x, y)\n", 78 | "print(\"Mutual information calculated through traditional method is:\", mi)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 5, 84 | "metadata": {}, 85 | "outputs": [ 86 | { 87 | "name": "stderr", 88 | "output_type": "stream", 89 | "text": [ 90 | "100%|██████████| 500/500 [00:11<00:00, 43.71it/s]\n" 91 | ] 92 | } 93 | ], 94 | "source": [ 95 | "np.random.seed(1234)\n", 96 | "H=10\n", 97 | "n_epoch = 500\n", 98 | "data_size = 20000\n", 99 | "e = 2.71828\n", 100 | "\n", 101 | "class Net(nn.Module):\n", 102 | " def __init__(self):\n", 103 | " super(Net, self).__init__()\n", 104 | " self.fc1 = nn.Linear(1, H)\n", 105 | " self.fc2 = nn.Linear(1, H)\n", 106 | " self.fc3 = nn.Linear(H, 1)\n", 107 | "\n", 108 | " def forward(self, x, y):\n", 109 | " h1 = F.relu(self.fc1(x)+self.fc2(y))\n", 110 | " h2 = self.fc3(h1)\n", 111 | " return h2 \n", 112 | "\n", 113 | "model = Net()\n", 114 | "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n", 115 | "\n", 116 | "plot_loss_nwj = []\n", 117 | "plot_loss_jsd = []\n", 118 | "plot_loss_mine = []\n", 119 | "plot_loss_nce = []\n", 120 | "\n", 121 | "\n", 122 | "for epoch in tqdm(range(n_epoch)):\n", 123 | " x_sample=gen_x()\n", 124 | " y_sample=gen_y(x_sample)\n", 125 | " y_shuffle=np.random.permutation(y_sample)\n", 126 | " \n", 127 | " x_sample = Variable(torch.from_numpy(x_sample).type(torch.FloatTensor), requires_grad = True)\n", 128 | " y_sample = Variable(torch.from_numpy(y_sample).type(torch.FloatTensor), requires_grad = True)\n", 129 | " y_shuffle = Variable(torch.from_numpy(y_shuffle).type(torch.FloatTensor), requires_grad = True) \n", 130 | " \n", 131 | " pred_xy = model(x_sample, y_sample)\n", 132 | " pred_x_y = model(x_sample, y_shuffle)\n", 133 | "\n", 134 | " # Nguyen Wainwright Jordan (NWJ)\n", 135 | " ret_nwj = torch.mean(pred_xy) - (1/e) * torch.mean(torch.exp(pred_x_y))\n", 136 | " # Jensen Shannon Divergence (JS) loss\n", 137 | " ret_jsd = torch.mean(-torch.log(1 + torch.exp(-pred_xy))) - torch.mean(torch.log(1 + torch.exp(pred_x_y))) + e - 1\n", 138 | " #ret_jsd = torch.mean(torch.log(1 + pred_xy)) - torch.mean(torch.log(pred_x_y))\n", 139 | " # Mutual Information Neural Estimation (MINE)/ Donsker Varadhan (DV) loss\n", 140 | " ret_mine = torch.mean(pred_xy) - torch.log(torch.mean(torch.exp(pred_x_y)))\n", 141 | " # Noise Contrastive Estimation(NCE) loss\n", 142 | " #ret_nce = torch.mean(pred_xy - torch.log(1 + pred_xy + torch.sum(torch.exp(pred_x_y))))\n", 143 | " ret_nce = torch.mean(pred_xy - torch.mean(torch.log(torch.sum(torch.exp(pred_x_y)))))\n", 144 | " \n", 145 | " loss_nwj = - ret_nwj # maximize\n", 146 | " loss_jsd = - ret_jsd\n", 147 | " loss_mine = - ret_mine # maximize\n", 148 | " loss_nce = - ret_nce # maximize\n", 149 | " \n", 150 | " plot_loss_mine.append(loss_mine.data.numpy())\n", 151 | " plot_loss_jsd.append(loss_jsd.data.numpy())\n", 152 | " plot_loss_nwj.append(loss_nwj.data.numpy())\n", 153 | " plot_loss_nce.append(loss_nce.data.numpy())\n", 154 | " \n", 155 | " model.zero_grad()\n", 156 | " \n", 157 | " loss_nwj.backward(retain_graph=True)\n", 158 | " loss_jsd.backward(retain_graph=True)\n", 159 | " loss_mine.backward(retain_graph=True)\n", 160 | " loss_nce.backward(retain_graph=True)\n", 161 | " \n", 162 | " optimizer.step()" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 6, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "plot_x = np.arange(len(plot_loss_mine))\n", 172 | "plot_y_nwj = np.array(plot_loss_nwj).reshape(-1,)\n", 173 | "plot_y_jsd = np.array(plot_loss_jsd).reshape(-1,)\n", 174 | "plot_y_mine = np.array(plot_loss_mine).reshape(-1,)\n", 175 | "plot_y_nce = np.array(plot_loss_nce).reshape(-1,) - np.log(data_size)" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 7, 181 | "metadata": {}, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd8FMX7wPHPXHpCCiGhpRBK6KFGIkVApIMUQQF/IoiADZWiqF8QEUVFQcQuCorSm4CAIiAgHULvLZQklHRIb/f8/riQAgmkXECSeb9e97rb3ZnZuYM8Nze7+6wSETRN07TSxXC/O6Bpmqbdezr4a5qmlUI6+GuappVCOvhrmqaVQjr4a5qmlUI6+GuappVCOvhrmqaVQjr4a5qmlUI6+GuappVClve7A3lxc3MTHx+f+90NTdO0B8q+ffsiRMT9buX+s8Hfx8eHwMDA+90NTdO0B4pS6mJ+yulpH03TtFJIB39N07RSSAd/TdO0UkgHf03TtFJIB39N07RSSAd/TdO0UkgHf03TtFJIB39N00oGMcKvw+FG6P3uyQNBB39N00qGfXNh0I/wyiP3uycPBB38NU27s9RECNp8v3txuxlPwInVWcuXT5mej1yBPyeZfgmIEfb8bHq+k78/hviwgvdh27cwoAqkpxS8LsDZjbDh08LVLSoR+U8+mjZtKppW6qUmiKQlm17v+knk6Mrby4SfFDm+yvR6zUSR1xuJRJwWCTues1x8hEhyrOn1iTUiPz4r8nFn0/LhZVn7uVW7siIgsuKd/Pf7n89FIs+KrH5P5OV6IqmJt5dZ/Z7IgQWmt/m0t6SMb3n3dg8tERnTVCRkr6lPFois/J/I4OqS3t3dtO7mY/4IkW/7m17Pe1lk39zc27y4M6vO8T9M646tFLm4PatMcqwY32khsn6KxJ1eL+kNbESmdM2qd2R5ZtHUuS9KeCdrWbHlk6z6n/cUeSvA9Dn8Okxkdm+Rfz8VscqoH7JX5PEKpn+XuLC7fw53AARKPmKsMpX97/H395dC5/b58224esS8HdJKN0kDoxEsrHOuT0sCS9uc6xIiIC0ZnDzAmGoaFVo5AGJavnEVXLwg9hrYOIC1Y1ZdYxokREKZCqbd7tkCVgbExxvD4fOIjUL5NwODJRisIOQonIsw1W3TFrZszuqyrUIFtAaUacX2zVDGEhq2ylEOzzIQEgc13CAhEVJSoV4LMKbA9h1wc9Bc1hLqB5jehyHjc5B0OLQT3FzAsz4SchSCI1EpAh5lkJgEVHxGA27WpnYxwuUTcCY8o9+tYcu/ptd1PcG5IljZgbLI+bkGH4agqLv8Q2URVytUUjokZI36xb8eysEdSb4BBw4gtauBGDEcPm8q4GQBsemQERaNzf0xpKeSev4YVuFpAKS6WWAVkZ5jX2k1y3PjRhhSrhKup6+gUkEsQTWojTE9DcOhs6aClR3gcvztfXUwZH1OCujUC/rPy/d7zU4ptU9E/O9aTgd/rUQK2gcGA/g0zmcFMQVpCxsIOw3hEeDbAIzpYGEJe/eClQHKO4GNLVSsDeFn4XiIKWC518hq6mZgbdMWDm2DmDRTgDsTCJcTTNs8HOBKPLjZQJ3mpnXRl5Cz51EJGX+TLpamukBKZVusLydl9baMAVW7DgQey1rn44K6EJPzXRlA1agA5bxgZ8bfk5WC1Lv83dsqsFCQEZDEElQaYKcgSRBbRVoFRywNVqigSFOdKs7IpetgiSn4kfm1k6WSPRKXjIpNv3VLTm7WUN4dQq9BqhFq1YDjZyA5935HVoZyl+/cJIA4GVDeniRfj8AmOCHHtjRHsIzNWT69rAUW0aa+ijWoW2Z3jLZgSAKxAZWc9Z7j3MHhes7yYgCVEd/3e0PDCLDI2YWsvtTywfKdY7lvvIv8Bv/7Pr2T10NP+zz4Lt+4LDeSbhS84pkNIlUsJOnUOjG++ZDIRx2ztkWeFdk7J2s5JV4kPTVrOeaCyE/PZv0cz82NUJGdP2Utr58iRm+DGCsr03RJ9qkDkJTH3W5bJ3vniNFgep3auoypnXUfifSqlFWmvk3W635et7dx8/HvlyJ/TsqxzuiSs0xyfQtJd8u5ztjETox2yLev5dHuXR6zR2S9Dml/S9tVLXMs/z0k63Wa253bfeUt5Fy3rOWLnXNuTw2wu2P9qDYWYrRCjAoxOuTctvZ5JOBdZMuzyKmeSMvxSON3ESYiE8cgX2d8FkbbrDpBXfP3ebzy9p23f/2WvSTXML2+1gbZMgjxnYBcfixnubTySOcfmsjnI3Ou7z0uo292pv4O+8BFUqqY1l3qhLz+FrLz/5DUysj1Xi4F/7vJQD6nfcwetM310ME/f4KigmT8xvHyxKIn5KXVL8ny46a5x/iUeBn11ygJiwuTSzGXZMDSATJl2xTZdH6TXIi+IN/u+VauxV3L0ZbRaJSwuDBZfHRxjqAdcj1E4pLjJDoxWrZc2JKj/Ki/RkmHXzuI0Wi8ra09A5CF75TLs++xybGStu07kS0zTCtSE0VG+EnaI2VuCziSGC2h57dJ2pOVTcsbp0rC5YOS7m2QlJdqS3LoPrnxXC4B9n8tRFb8TxLb2EvswxaScuOypDc0BeWo1eNNffUyZAWmQVXz/ONPd719XXRAxh90BXXX4BLbzk4s30N+ePX2AJUjeLgj9pOs5URPJL5+1vo9A24vG/oYMvb7ZpnLX440Bbuz3XJv23oCMuIt5FJHpP5UD1Pg6Yh4fOggncchdSYghgmm4PRNti+V0T80k+iHkHkvIPYTkBSf3Ns/1huZv/8Xeft/NpLuikQ1NwXrShOQqIeRt940td1kAvLV64jte0jbd8kMghc7Id+OdxUBSaqJuL2XtU1Axn1USY5eOypMNLVzMTpY9l/eL/MPz5ePNn8njuO7y+QxyItTPKTnONOXxFcT3DI/1yYTsv3bNUNSPbOWlxxbIhsGZy1PmF5bvn/VVGfG2pHy15m/5J9Bpm2neithItLh1w7SY5KDhLdCjvQxbft0NDJj1wypPLWivDkWSaqOnHurjjARWTLWRULWjpdN5zdJUFSQ+H7pK/7vIp1nt5aPt34sFT6rIOd6W8rVrtaFjgk6+JcCe0L2ZP4RZH98t/c7mbFrRq7bsj+qTK8iWy5skRazWsgzy5+Rvov7Zm7z/dJX9l3eJ+vPrRcmIkNXDpWBywcKE5F1Z9eJiMiUbVOk63jk/TeQ51Y8JxWnVpRJmydJkx+ayNagjZl/RH+c+kOa/NBEYhJjRETkh8AfpM70qjn+0Dr81Fy++CTvwHvrw2iBJDbMCtpGmzuUtcz22iOrTqKfQaSWtQjIqqFIWoU779NmAlLmPWThS8iG55CxYxG7CciJXhltG5BFLyHhj5hGrTEPmcrbvYfUnewovIf0WNBDVmyaJOllkSmjkE9HIZPGIC+/bXoISFplZO6hucJEpNP4rP0v+7K5TB2FxDRDnvvUQ4I7IkPeQVYcXSwp3si+fsgra16R8p+Vl5HTa4qAHO6LVHvXNJoM/MRf+i3pJ5vOb5J5h+dJZHy01HwXafilr+wK3iXPLH9GNpzbIG+vf1ta/9xaFh1dJH6THeWT0cikdcul+/weMu/wPJmxa4aEx16R+R+b9nGjCRLvZwryTESiEqLkZPhJGbFmhCSmJsrio79L5WmV5WrsVVl+fLnU+qqWvLHuDdl/eb/YfGAjcw/NlXXr3xUBmTqttqw88Isc74V8+H1zmX94vsTXM73/NZ82kMSMA8fTtn8hDuMflQ9XH5PUtHQ5GhojKw+GSpW3VkvAp3Pk1LXLMmnzJGEi0uf9oSIgyb7IP0H/yJPjkMD+SJ3PKgoTkdFjkV3f9JCz127I/ENLpNYE0y+I7Ze2i8snLqImKjkdcVqi4pJl9NyX5UY7W9m+fJx4fV5Fdl06IK1+6ixMRD76Z6I88p6VMBEJigqSt9e/LUxEJvwzQYbN2S1l/zdMNp0+KYv2XJJ3lh+WxJQ0WXEgRJ6d+7OEx4eLiEi6MV1WnVgpa06vKXRcyG/wL5lz/iVMujGdj7Z+RK/avfCr4EdYfBjv/vMuM/fPzFHO1c4VK4MV1+Kv3bXN99u+z3ub38t1W6/avVh9ejWda3QmMiGSnSE7AbATqGeEQAuo7FiZy7GXkYmmOp4ToIehDEfS45i3G97xh3kfZbQ3Dv4XBJ1qwoSAkXiu/ZrHN6Zhe6Jgn8O1xxQVNgobnoNH14HFZYhtDI4HTNtXftuaen/+S40/bq8rdpBWHqwuwsWuil1VhH7fZW3/7eeubL+wiSl/JrKvLjz6q2l+NqQ9eG6A2Kbwv0kjqOtel041OrHk2BKOhR/jhaYv0HleZx4x2DOq6TC+Cj/M+qD1tK/SluiUWBysHZjVYxbl7MoRmRiJh6MHqcZUPv53Ms29W/LZjs/oXL0zw5sO552/3+Tln39gabMqTHj7AouOLqKOex28ejfEZRdsOb6SaQd+ZPXp1ex8fifNZzWnnnt9DrywH7dPXbmREsfRF88TFWekWRUP/v21L77dP2ZT6F6eafAMp64m4GxnhZerPaExiYxaeJDdlzeyZdQgVu1Pon8zL8o75jx4/fGWr/jf5tfwSlzM2I6NGNHOl7R0I5YWBr7YNoV6P7/N7w8H8OXz25i2YxotvVtS08Wf/jN38dpjvlR1c+CZn3bTq7EH7/eoh8GguJ6YioO1BSPmH8DL1Zpx3fwIuRFC7a9r83P3lfSs9wgfbf2IF5q+QCXHSmxaPQHnzT+zq9sGdpyL4a3Otdl9PpIP15j+A7Wq4ca2sxG3/Zv/8txD/Lz9FJtOXea7qP5cebQPQ59eyveB3zNx80T+GfQPn/3zN+2rdmHcctMB2d6NPejxUDLvb5rKuy1mUKuSLSuPHGN4i5Y0/WA9N5LSmDc0gP/7aXfmfozE0SFgL9O7TiAw+AJjFh3j+eb+nI+8zr6IpSwb9CZNJ+0CoHuDSqw+fAWAl9tW59vN5wBY8UpL3lt5lId8XKnq7sCAh7wxGG47YpIvpfuAbwlx6folytmVY9rOabkG6ocqP8TsnrM5Hn6cfkv70aNWD8a2GMvvJ39n2s5pACx/ajkbz2/km73fAPBzz5+pWKYinWt0pv2v7dl4fiOvNnuVr/Z8BcDCPgvpV78fI/8ayYzdMwDwcPQgNDaUXachYD7M/WsMA3dNY0gSzPok977HNYQyh3KuM5YzPasYUOlkHUTMJvRlL14uG8zKyVnrZoyESCt4YvQBfM7+ywfXT3Jt32+Mu6BIHf4Jmxa9gm8cBMyIoOl3DXn/fCiDvgFxgOOd4YYtGAQCH23DU79toUdbeO/JBTR+dQBup2BtN0gZuZgz0ed4Z+M7PFXvKQb+s5zu36WxedVrVKjWFoO9J2XK1sfDxe629xoWH4ajtSN2VqZtIoJSpj/c+OQ04pLTqOBky287L7D9bCQzBjTCxtKCvReiqOxil9lmUHgsE7a8zZiWQwm67M7+S9G81LY6Y/4azOGTS5g75BopaUYquCZQrWw1Ptqwiu83JDOt78O0rGlFmtGC52Yd41x4PINb+FC9fBkaeDhT0dmW8o42VH1nLQDT+zVk8pqTRMQlA/B2l9p88udJ/DyceadrbWZsOMO58DiGPVKNj/88mfk+3crYMOyRqkz7+zQf9qqPR4VLtPq5FdXsu/JKw89BKbo3qMSkP46z7WwE3RpUAmBNRrDr6leRKuUc+G7zOTxc7AiNScxc72BtyZJ9IQB086tEm1ruPNnUkzFLDrF8fyifP9WQD9ecICo+/+fTWxgU6UZTfEvnOtVcy/Nh74b4eThTxsaSH/4N4rN1p+7YRpf6Ffnz6FX8q5Ql8GI0kPuXTTV3B1a80pJJfxxnacb7uKlD3QqsP55zQObhYse1G0mkGXOPv/OHBtCihlu+32t2Ovg/wI6HH+f5Vc+zK2RX5jorgxWpxlQA2lRpw+jmo3m85uMopRAR5h2ZR1ffrrjauQIw+d/JeDp5MqjRIAAGrxjMnENzSJ+QjkGZru2LTIgkKjEK33K+OQIWUUEcS0vkhdUv0MqrBZP2H2RTeRs6fLoOw8lUGNOUv4ZPpOEbfai4KQVVNWO0eCTrbJSbpIIBdS3jjJFyChVp+v+Wsvo9rNq8ROzCUTj1mwpB28HnYYxOHmwI2kD7wX0xbDOdevHRiqGsDD/Mrud3ZfYxe39Xn17NjeQbPO33NKcjT/Pk4qc4evUQ50dfRKGo/U1tWldpzeR2k+m1sBdT2k9hgN8APt3+KW+tfwsUXBp5CQ8nD/Zf2U/DCg1ZffpPPl7Yix9fOEX9ijXw/3A90Qmp/DOmDa4O1rjYW3M2LI5Fey/xbHMfvFztAQi7kcT8PZeIiEvmiSaePPHtDgA2jG5D+8+3ZH4uo9rXZPqG01R0smXnO+0AMoPz+G51Mke1AGXtLfjsqVoM/eU4YAqOg1r48NU/Z9h6JisIOdpaEpt0y7cp4OpgzedPNWTwz3tz/w+XD0/5e7I4MCuoOVhbMKCZF9N3zsQ+vQUWlMlR3q2MDRFxydhaGQioWo5jl69zPTGV1HShRfVy7DgXmaPcrRxtLHn1sRp8tNb05fN0gDdLA0N4rE55/jx6FYAv+jVi5KKDeJa1481OtXh94UGqlLPnYmTOU2huDb6ONpYEVCvHhhM5A/Lk3vUZ9/vRO34ON9tXCraOfTTzS/Ozdado5OVC4MVoald05OTVWBxtLalf2ZmdGWdD9WpUmRUHTackrRvZmv/7aXfme3e0teTZ5lWISUhl3u5LDG1VlfHd696xL3m5p8FfKdUZmAFYAD+JyG3jQaXUU8BEQIBDIvL0ndosjcE/LD6MdGM6fRb3YWfITnrW6omXkxdNKjVhgN8AbC1tORlxEl9XXywMFndvMJs0YxpJKXGUsXXJuUGM8EYzqFsfGrWHjs9CVMb/id9eBHtH6PPZ7Q1WNsBlIwytBT9mjA7Xvg/dJkJ1S+jTBFxdoVEr6DwevCzg2CVYMQlOHoH3N95+fnx2iVGQdB0cK9253C2SUtOp9e5aalSO44NuHWhR3Y00Yxqp6amZI/OwG0m4lbHBYFCkpKfwx/FdNKnUNHME/sys3QRHJZBuNBIWm8KcIc0YNHtP5j6a+bgypJUPby07wvXEVDxc7Hi7S20EeG3BgcxyBgU3B3Zl7a2ISUylnMPtwa66uwONvcveNmK8GUQA6lZyIiQ6gRu3BPfsI9LsrC0NpKSZvnSd7ay4nmgaOEx7siFjlph+ki19sTl9v9+Jo60lYzvV4o9DVzh+5QajOtSkU70KtJqyKbO9ne+0o82nm0lJNzKhe10mrT5+2z5bVC9HfQ9nOtWryP6L0Uxea/oCm9C9Lq4O1oxcdBC3Mjb8O7Ytnb/YyqWoBPaMe4ze3+zI/BUwoJk3LaqX49WMz7Fd7fJExiVzKOQ6ADP6N6Kiky0u9tbUquhIWGwSlgYDZe2tuBiZgEdZO/p8t4PO9Ssy4CFvgiLiiYpPYdivplgyqWc9lu0L4VDIdRysLXj50RqZo/9zH3Wlx9fbOHb5BpWcbbly3TSY+aBXfd5dcZRGXi48XK0c3285R30PJ1a/mpVGYvn+EEYvNn2uPwxsyrebz/FEYw/qVXai7/emadN/xrSh3TTTAODCJ91ITksnJiEVe2sLbK0ssLIwDcoGztrNtRtJ/D2qzW2fcX7kN/gX+QbuSikL4BugAxAC7FVKrRKR49nK+ALvAC1FJFopVb6o+y1JUtNTaf1L6xwj/e+6fceL/i/eVra2W+1C7CAByy/7UWb6WtP5028Mgg9nQ6d60KI1fL4P2AfMyVnvm7lgc8uXzKzB4OAE/b80Lbd5NGtb1/dgRTLUbw/V22Xum/4z4bXx4FgZBn6fry6fj7ehqltVPl9/mhuJqfyvax2sLQ3sOBfBvN2XeLqZNy1ruHE9IRVrSwN21hYYjcKnf51CYeDcZSee/nE373Spzamrsbzfsx5YwU9bg5i89gR9m3jy8RN+DPxpH7vPxwKbcwTM7N7ICJbjutZh8toT7LkQxZ4LUXi72vN+j3p8v+VcZrDKzigwpY8f7606RnRCKq895suo9r6sOnSZ1xcexNrCQPu65Vl75CrnwuNpVtWVPedNFzHNHxaAfxVXfthyjmnrT3P8yg0+6u2Ht6s9oxYfJDw2Gbcy1swY0BhnOyv6fLuDZ5pXoW1NdzzL2rHldDiDf95LVTcHXmhdjbeXm6576d6wEg42lhy/coPG3mXZ/b/HcHWwxsrCwMDmPqQbBYuMuebx3eoQHJXA+z3rA7D29UdwsrXEwcaSSauPY2Wh+P3llvx9/BpfbjxDtwaV+L+AKoDpi8vW2oJzYXH0bFSZlHQj1hYG3uxUE3trS5a+1Jy956Mp72jL3KEBzN52npjEVEa298XF3oqhrapSzb0MfZp6MOQX0y8Wd0cbAqqWo6Jz1mAg+zEKHzcHAFaNaJW5rqmDNeGxpi/bh6u58mxzH54JqEJoTCIeLnYYDIpfd14gNikNC4Pp/aQZjdhbWxIZl0xkfApeZe05cy2WF9pU51xYHN9vOUfL6jmnZJ5o4omDjSWztp7nEV83OtWrCJgGIzdVcy9D7YqOtKnlDoCNpQUVnG4fxH3Qsz5lHaxvW29uRR75K6WaAxNFpFPG8jsAIvJxtjKfAqdF5Kf8tlsaRv4iQnxqPK//+TqzD87OXO/t7M2JV05gb2Vvnh190QtGrcxfWX972HYFxj8GU2/5/Me1hA/+BWUw5TNZeAkubIMqLQvdNaNRMIpgmTHqWbYvhPl7LrHvYjTzhwbwdMaBtUHNqzCxRz1GLDjAmsNXUAqWvtiC1xYcICYhhbe71ObqjSS+2XQuz3292q4Gv2y/QGyyafT8Qutq/PBvUOZ2e2sLDEpRvXwZDgXH3Fb/wifdiIxLZsySQ/h5OPP6Y75YWhhISk2n9rt/AfB+j3r0bFSZZ2btpqtfJV5uW4PgqATSjELVjOAUm5TK83MCebtLbZp4lyUkOoGo+BTqV3Zm9OKD9G/mzcPVTAdIriem0n/mLuytLVgw7GGsLU2f09XrSZSxtaSMTe7jt9ikVJ78ficTe9QzBZzPNjOqvS+DW1Yt0L9PXnacjcDHzYHKLnYkpKTx686LPNfSBxvLvH+Rxial4mhrVeB9bTh+jZ93nGfmQH8c8ni/d7PpVBgNPV1wzSWoJqSkYRTy/CyzS05LZ/zvR3mpbXWquZe5a3mAlQdD8S3vSN3KTgXud2Hcs2kfpVRfoLOIDM1YHggEiMiIbGVWAKeBlpimhiaKyF+5tDUcGA7g7e3d9OLFi0Xq23/dkmNLeGrpUwBUKlOJmuVq8vFjH9OoYqPMKYp8mTMM1q2HyzHQsy2MWgHBe6DJw/DnHBj5Cmy/5dLFjVOh0xuQBjxTFa5dh8Y1YNhHUOMx2D8Pnn4OmlSCD+fA56/BJ39DGdOIhqQYOLAYmg8v0mfwwm+BXIxM4M/XH+FSVAJtPtucuW3i43V5f/Vxbv4X7VK/IruCIqnv4cyh4JjbpkBuyj5dAtDY24UDl7KC+YJhD/PGkkOExiTiaGvJupGtqZwx5XPzy+jdlcdYsOcSTwd4M3/3JZpWKcuyl1rk+T4W7w0mODoh8wvhv8ZolEKfPaI9WO5l8H8S6HRL8G8mIq9mK7MaSAWeAjyBrUB9Ebl9eJWhpI/8Y5Nj6b+sP2vPmA7yZT8QWyDTHoc3Vudc19gOqpeDpSG3lx/dFGrXhWG/wvResHUPzNwEbrUK8S7u7Njl65RzsMnxMz2709di6TjdlNPlh4FNSUxJZ+SigwxpWZXZ289nnhb3Wd8GxCSkZs4hj+9Wh+iEFL7ZdA4rC8Xa1x7hcMh1lDLNETvaWrF8fwhvLj0MwNnJXQBYf/wa5Z1saVqlLBci4vnh3yD6NPHA38f1tr6JCKExiXiWtSc4KgFHW0tc7Iv/p7imFdU9m/PHNM/vlW3ZE7g1y0YIsEtEUoHzSqlTgC9Q+NMPHmC/HvqVwSsGIwj13Oux7KllhQv8gb/mDPxVLeF8GhxIhAO5BH6AMTOhchPT61ErYFTBd5sfIkK3L7dhb23B8UmdATgXHsf58Hja163Ayas36PzF1szy/1t+BM+ydthbWzCuWx3+PHol83xoL1d7nvQvRwNPZ3aci+SJJp6kG4UVBy4zvlsdfCs44lvBMcf++zTxJDE1nVY13DJH4l38KmVu93Fz4OMn/PLsv1IKz7L2mfvXtJLGHMF/L+CrlKoKhAL9gVvP5FkBDAB+UUq5ATWBIEqZuYfnsjd0L7MPzkYQWnm3YnK7ydQqzKg7/hq8PiLnOmsD/Ps1rPwZpu0zrWvuCJWd4N/LEC5QsUHR38gdJKWms+HENZp4lwUgISWdkOgENp0M4/P1p4lOSKWMjSVxGfPug1v40L+ZF8/9vJdDIdd5orEHFgaVeaYFZAXfgGrlCMiYCwfY/na7PPthMCiebe5TDO9Q00qGIgd/EUlTSo0A1mGaz58tIseUUpMwXWa8KmNbR6XUcSAdeFNEIou67wfJmcgzDPx9YOby8ZePU8e9TuEb7FoLdsTChEeg9ePQfqxp/SOvQKuXYJoFlFew44Zp/ZWDcGarKRWwGc3bfZE/j1xlUs96BEcnEhQex/t/HKeiU9ZUT/ZTBsF08U2fJp68/Gh1qmccNFv5SktWHrzM0wHeAPTz92JRYDBf9GuU64VVmqYVjb7I6x7pMq8LG4I20LBCQ0Y+PJJnGjyT/8pihOm9oesL4NsBenvBH9egUzlYcdZ09k1fX5jwKTxkuqiLC9vAzgUq1Dfr+8h+4DAlzUjN8X/esXzbWu5sPmXK2z7i0RqM6lAz81TCu+0n1Wi849kjmqbd7l7O+Wt3ceDKAf46+xcfPvoh41qPK3gD27+HMatMDw9LCE0zjep/2wY3L9r645Z8Pj6tbm+niBbtvcQ7y4+wb3wHyjpYZ141+dpjvly7nsSiwGAAOtatwIBm3qQbhYerl+Op73cysr0vHTPOfc4Pg0FhU8AL2TRNyz898i9GKekpTN0xlXH/jMPawprzr5+nsmPlgjfUvQKsyXZ/0UfKwJbrphH/PdTyk38IjUmkW4NKPN+qKp//fZrzEfH8O/bAgckEAAAgAElEQVRRLAyKmIQUftt5kUdrl6e+h/M97ZumaSZ65H+fRSdGM3b9WH468BMGZWDL4C35D/zpKTB7KExdCGcybok0sS34NYOd/8BrX97zwL8rKDLzEvw1h69kJut6uW31zGkcF3trXn3M9572S9O0wtHBvxikGdNoPqs5pyJNOUP2D99Pw4oN89/AvBEw/Les5U86w+hlYGUPT5i5s3m4mbY33SiMmL8/M5nWspeaU97Rll7fbCcyPoUOdSvcmw5pmmZWOvgXg/Xn1mcG/qkdphYs8AMsy0hIv3cO1OsOdrdfhFSc4pLT6DLjXzxd7HEtY50Z+F9/zJemVUx9WfVqK9YevkJDT5c7NaVp2n+UDv5mFpscy5TtU3CycSLsjTBsLG3yVzEhHA79DtFXYN1VeK0R+D9bvJ3NMHvbeco72VDW3ppP/jzJkVBTBsXgqMQc5QKqZX0JebjYMax1tXvSP03TzE8HfzNKTU/l0TmPsv/Kfj567KP8B/60JHC4JdFpnyHm72AuwmKTbkvPW7eSE32beuLvU5akVCMNvZw5dTWWBnqUr2klhg7+ZmIUI+9uepd9V/axoM8C+tfvn//Ke25JpVzFAlq+YN4O5uHWHPKf9m3Ak009s27skkEHfk0rWXTwN5MZu2YwZfsU+tTpQ796/fJXKSkG/vkCXsl2z8Kp3eHFWWBRvEnE9l2MZuzSQ5wLj8fPw5nWNd0Y/kh1nO0LnnJX07QHjw7+ZpCSnsKHWz+kc43OLHlyyW2j5jx91AM+yEhuNrkDNGwBXScU22mcoxYdJCYhhfZ1K/D+H8czb1zy+mO+tNdn7WhaqaKDfxGlpKfQbk47ohKjeK3Za/kP/ABnst2v4O21Zs+7k11kXDK/HwgFYNOpcBp4OvNqO19cHaxpWqVsse1X07T/Jh38i+j7wO/ZHrydPnX60KF6h/xVSkuCNe9Dxo2d+bB9sQb+CxHxtJ26GYBq7g5Udy/DZ30b6Pz0mlaK6eBfBOHx4Xzw7we0q9quYNM9X/Yz5ekB0w3Qx60vvk4CG05k5f35ZXAzvMvp/PSaVtrp4F9I2y9tp8/iPsQmx/JZh8/yF/h3/QRXzsBnGTdgaeYA/Z8v3o4CW89EZL72LKvTI2uapoN/oYgIr//1Okoptg/ZTpNKTfJRyQjNh2UtLxgB/b8qvk5mSEpNZ1dQJAOaeTOiXQ19H1dN0wAd/AssNjmWHgt7sO/KPn56/CeaVm6av4pbv8m53O1t83cOSEhJ49lZe3B3tOFiZAIP+ZQlOc1I5/oV9U1RNE3LpIN/AX215ys2X9hMZcfKDGw48O4Vbpr5OdgB/k4w5mVw9DB7385ci2XurosEXozOXHf8yg0aejrzcLV7mx9I07T/Nh38CyA2OZbvAr/Dr7wfG57dgHV+L8RKTYCVF2BgbfjhhNn7lZyWzsqDlxm79DAA7etUIPBiFAkp6Yxs78vgFj76jliapuWgg38BfLT1I0JvhLKwz0LK35qL507WfQJxQIeuxdKvPt/t4GjoDeysLPjt+WY08S5LYmo6FgaFrZUO+pqm3U4H/3zaf2U/X+35ij51+9DSu2X+Kl3YBr07wsGM7Jhthpq9XweDYzgaarpJe51Kjvj7mKZ3HGz0P62maXm7t7eDekAlpibSY0EPbC1tebf1u/mrFHcVJg7OCvyLRoJ7HbP263pCKs/8tDtzubyjrVnb1zSt5NLDw3z4es/XhMaGsmXwFhpUaJC/Sj3qwKYY0+t5L8NT083er1nbgohLTmP+sADOhcfTuQA3SNc0rXTTwf8u0oxpTNk+hc41OtO6Suv8VQo7lhX47YCnv7lj8cLYfjaC77cE8XjDyrSo7kaL6m5m34emaSWXnva5i10hu4hMjOT5xgW4EndbRn7+TuXgj0/N3icRYdzvR/BytWNSj3pmb1/TtJJPj/zvIDE1kTf+fgMLZUH7au3zX3HXZrAGVgaBjZPZ+7V0XwgXIhOY0sePsg46OZumaQWng/8d/H7yd3aH7mZK+ym42ObzTlYhe2HBfvB3LJbAHxyVwNvLj1DVzYEufpXM3r6maaWDnvbJg1GMzDowCw9HD95o8Ub+KqWnQOdH4EY6zPja7H1KSTPywerjGBTMHxaAk62+65amaYVjluCvlOqslDqllDqrlMozaY1Sqq9SSpRS/ubYb3GasWsG/5z/hzdbvIkhv3fW2vQFHEuGb14E/2fN2p+YhBSe/GEnfx+/xluda1PJWefp0TSt8Io87aOUsgC+AToAIcBepdQqETl+SzlH4DVg9+2t/LdEJUYxftN4utfszmsBr+Wv0sXt0O9tcAKe+MCs/QkKj6P7V9tISElnSh8/+j3kbdb2NU0rfcwx8m8GnBWRIBFJARYCPXMp9wHwKZBkhn0Wq9kHZpOQmsCHj35YgPvxDoVYgTXfgL15T7t8b9UxFPDjs/468GuaZhbmCP4eQHC25ZCMdZmUUo0BLxFZbYb9FauLMReZuHki7au1p2HFhvmrZEyDJadgQDVo9bJZ+/PX0atsPRPB6+196aBvsq5pmpmYI/jnNjSWzI1KGYDpwJi7NqTUcKVUoFIqMDw83AxdK7gp26eQakxlVo9Z+asgRhheD6IFOnQ0a18OBccwevFBGnq58GxzH7O2rWla6WaO4B8CeGVb9gQuZ1t2BOoDm5VSF4CHgVW5HfQVkZki4i8i/u7u7mboWsFcib3C7AOzGdRwEN7O+Zxe2T8fZp02vW437M5lCyDsRhIDftxFWXtrfhzYVGfn1DTNrMwR/PcCvkqpqkopa6A/sOrmRhG5LiJuIuIjIj7ALqCHiASaYd9mExQdRL+l/UgzpjG25dj8V/x1qum3z+GlUDkft3PMh+CoBNp8tpmElHTmDg2gvJNO2KZpmnkV+WwfEUlTSo0A1gEWwGwROaaUmgQEisiqO7dw/6Ub0+mxoAdno84yp9ccarjWyF/F1ARYcBj6eIBfH7P1Z86OCySmpvPKo9Wp6uZgtnY1TdNuMssVviKyFlh7y7oJeZRta459FpZRjEQnRpNmTMPB2oEdwTuYvHUyx8KPseTJJfSt2zd/DcWHwautIFxg0BCz9S8lzcjvB0LpUr8ib3aqbbZ2NU3Tsis16R0uxlzk/S3vM/fwXFKNqTm2Ods4M7vH7PwHfoCPn4Cfz8A7zaHbRLP0MSw2iVUHLxMZn8JT/l53r6BpmlZIpSL4bwzaSLf53QAY0ngIddzqYFAGopOiCfAIoJV3KxysCzi9smwvPFYWPtphtn4O+3Ufh4JjqOrmwCO+OkWzpmnFp8QH/1MRp+izuA81XGvw5//9iZezGUbUJ9fCyRQYkdu1bIWz72IUh4JN9wD4aZA/lhY67ZKmacWnREcYoxh5buVzWBgsWPP0GvME/hOroY7pVwQ98pnw7S7CYpPo891OAFaNaEl19zJmaVfTNC0vJXrkP3PfTHaG7GROrzlUcalinkYXZNycpbkjeDUrcnOp6UbeXHIYgGlPNqSBZz5TR2uaphVBiR35p6anMnnrZB7xfoSBDQaar+H1ByDAAbbHmKW5DcevseV0OO92r0ufpp5maVPTNO1uSmzwX3xsMSE3Qnir5Vv5T852N9u/h11x0PsRyG+a5zv488gVXpq3H4Bnm5vpl4mmaVo+lNjgP2P3DGq71aaLbxfzNTrrc3BRMGKOWZpbtj8EgEk962GlD/BqmnYPlciIcyriFHsv72V4k+H5vxHL3aQlwcqz0K4SOJQvcnOztp1nw4kwnmvpo5O2aZp2z5XI4L/w6EIUin71+5mv0f+rCVECHdsWuanZ287zwerjdKhbgVEdaha9b5qmaQVU4s72EREWHF1AG582VHasbJ5Gw0/AsmDoVQmG/FikphJT0pn69yna1nLn66cbY2Ops3VqmnbvlbiR//mY85yNOkv/ev3N1+jcdyAdmPQVWNkXupn1x68xcNZuElLSGd66mg78mqbdNyVu5F+tbDWuvnEVW0szpkH+bi20dCxS5s7UdCPjfj9CWGwy3fwqEVC1nPn6p2maVkAlLvgDuJnzHrpXDsKZVHipa5GaWXP4CmGxyfz4rL++HaOmafddiZv2MbsDK03PTR8rdBPpRuHLf85Qq4Ijj9Uu+plCmqZpRaWD/51EnMpK19yod6GaSEhJo9uXWwkKj+f19r4YDGa64EzTNK0ISuS0j9msm2F67uoOTgVPvXA09Drdv9oGQLcGlehcr6I5e6dpmlZoOvjfyZZN4AysCilwVRHh281nAehYtwLfPG2e+/tqmqaZg572ycupP2H+SWjvARbWBa7+7eZzrD1ylSeaeDDzWf9i6KCmaVrh6eCfl49GgAJmrChw1cSUdGb+G4SfhzOTe/mZv2+apmlFpIN/buKuwsIgeLY+eBR81P77gVCuJ6bybve62FnrC7k0Tfvv0cE/N5u/hRSgz+BCVV+49xJ1KjnxkE9Zs3ZL0zTNXHTwz83S+WAHtBxW4KoXIuI5HHKdXo0qm+8+ApqmaWamg/+tDi2GOefgxSZg41SgqulG4cW5+7A0KLr6VSqmDmqaphWdPtXzVr98AtbA+MUFrro4MJiTV2OZ0scPL9fCJ4DTNE0rbnrkn50YYdlh6FIRXKsXqOruoEjeWX6ECk429GrsUUwd1DRNMw8d/LO7uAOC06F9qwJVExGmbziNk60lv7/cUqdq1jTtP08H/+zWfWd6bp3/O4CJCJPXnGBXUBRjOtaisotdMXVO0zTNfMwS/JVSnZVSp5RSZ5VSb+eyfbRS6rhS6rBSaqNSqoo59mtWh5fB/xZAE3uo3yvf1Y6G3uCnbefpWLcC/xfgXYwd1DRNM58iB3+llAXwDdAFqAsMUErVvaXYAcBfRBoAS4FPi7pfsxv7gunw98IVYMj/cfAFey9hbWngs74NsbTQP6Q0TXswmCNaNQPOikiQiKQAC4Ge2QuIyCYRSchY3AUUPEVmcUpNgG2R8FR98O2Q72pHQ6+zaG8wTzb1xNneqhg7qGmaZl7mCP4eQHC25ZCMdXl5Hvgztw1KqeFKqUClVGB4eLgZupZPu2ZDPNAm/4H/QkQ8Q37ZSwVHG8Z0rFV8fdM0TSsG5gj+uV3GKrkWVOoZwB/4LLftIjJTRPxFxN/d3d0MXcunbz4FJ6Dj6HxXmbb+NAkp6fwypBmuDgXP+qlpmnY/mSP4hwBe2ZY9gcu3FlJKtQfGAT1EJNkM+zWPC9tgaTAMfwic8nd+ftiNJP48coWn/L2oWcGxmDuoaZpmfuYI/nsBX6VUVaWUNdAfWJW9gFKqMfADpsAfZoZ9ms/koabfLq99le8qv+26SJpRGNj8v3fSkqZpWn4UOfiLSBowAlgHnAAWi8gxpdQkpVSPjGKfAWWAJUqpg0qpVXk0d2+d3QizTsGrTcArIF9VDofE8N3mc3Tzq0RVN4di7qCmaVrxMEtuHxFZC6y9Zd2EbK/bm2M/ZvfbJNPz6O/zVTw+OY2Jq47hbGfFx330TVo0TXtwld4T08UIc3dAu7Lg+VC+qnyx4TQHg2OY8HhdnGz1qZ2apj24Sm/w3/YtBKXBwD75Kh4Zl8yivcG0r1OBno104jZN0x5spTf4f/oBuCro88Fdi4oIQ38N5EZSGk/rFA6appUApTP4n90Iq8Pg1dZQpuJdi288EcaBSzG8270ubWuVvwcd1DRNK16l82YuP40DC2DY1DsWExG2nA7n1QUHqF3RUSdu0zStxCh9wT8lDn7eA90rgof/HYtuOhXGkF8CAXi/Rz1srXSefk3TSobSN+2zfiqECTw/9I7FriekMn39GQA61K1As6qu96J3mqZp90TpG/kvnQfOQMc371hs8trjnLhyg+/+rwld9M3YNU0rYUrXyD81AVaeg8ergo1TnsUuRSawZF8Ig1v46MCvaVqJVLqC/6YZEC3Qt3+eRYxGYdr6UxiUYugj1e5h5zRN0+6d0hX8l/1qyjDU8Y08iywODGblwcuMeLQGFZ1t713fNE3T7qHSE/zTkuD3U9DdG+xyP3grInyz+SyNvV0Y2d73HndQ0zTt3ik9wX/rtxAu0KdvnkUuRSUQHJVI78YeKJXbPWo0TdNKhtIT/Jf9AnZAl7fyLLLmyBUAWlR3uzd90jRNu09KR/A3psHyY9C1Mjjknp7hUmQCX2w4Q7va5anurvP0a5pWspWO4L93DlwxwhO98izyzaazGBR81NtPT/lomlbilY7gv2mJ6bnDK7luFhE2nQqjQ92K+gwfTdNKhVIS/AOhvg2418118+lrcYTFJtOqRrl73DFN07T7o+QH/9QE2BYJbWvmWWTb2QgAWvm636teaZqm3VclP/jv/Q0SgLad8iyy7Uw4Vd0c8HCxu3f90jRNu49KfvDfvMz03Cb3LJ4paUZ2n4+iVQ19eqemaaVHKQj++8DPFtxq5br5wKVoElLSaamDv6ZppUjJDv6pCbA96o7z/TvORWJQ0Ly6PtiraVrpUbKD/8Elpvn+Vu3yLHIgOIaaFRxxtrO6d/3SNE27z0p28N+5yvTcPPcUziLCoeAYGnm53MNOaZqm3X8lO/jv2gceBvAKyHXz+Yh4riem0thbB39N00qXkh38d4bCw3nfievApRgAGnmVvVc90jRN+08wyz18lVKdgRmABfCTiHxyy3Yb4FegKRAJ9BORC+bYd56uHoYLaTCicZ5FDgbH4GBtQY3yZYq1K5qm3V1qaiohISEkJSXd7648EGxtbfH09MTKqnDHK4sc/JVSFsA3QAcgBNirlFolIsezFXseiBaRGkqp/sAUoF9R931HexaZngO65FnkYHAMDb1csDDoRG6adr+FhITg6OiIj4+PTq54FyJCZGQkISEhVK1atVBtmGPapxlwVkSCRCQFWAj0vKVMT2BOxuulwGOquP91D+wABTR6ItfNSanpnLhyQx/s1bT/iKSkJMqVK6cDfz4opShXrlyRfiWZI/h7AMHZlkMy1uVaRkTSgOtA8Z5Yf+Ak+FpBmYq5bj4aep00o+jgr2n/ITrw519RPytzBP/ceiCFKINSarhSKlApFRgeHl60Xh0Ih8a5B34wTfkANNJn+miaVgqZI/iHAF7Zlj2By3mVUUpZAs5A1K0NichMEfEXEX939yJk2Iw6C5fSoXG9PIscCI7Bw8WO8o46f7+maaWPOYL/XsBXKVVVKWUN9AdW3VJmFTAo43Vf4B8RuW3kbzYHV5qeG7fOu8ilGD3q1zQtB6UUAwcOzFxOS0vD3d2d7t27A/DLL78wYsSIPOtPnDgRe3t7wsLCMteVKWM6m3DUqFF88cUXmes7derE0KFZCSfHjBnD559/zoULF6hfv77Z3lNeihz8M+bwRwDrgBPAYhE5ppSapJTqkVFsFlBOKXUWGA28XdT93tGBzabnRj1y3RyblEpoTCJ1KzkVazc0TXuwODg4cPToURITEwFYv349Hh63HsK8Mzc3N6ZNm3bb+hYtWrBjxw4AjEYjERERHDt2LHP7jh07aNmyZRF6XzBmOc9fRNYCa29ZNyHb6yTgSXPsK18OHIXKBiif+7TPufB4AHz1+f2a9p/0/h/HOH75hlnbrFvZifcez3sq+KYuXbqwZs0a+vbty4IFCxgwYABbt27N936GDBnCL7/8wltvvYWrq2vm+pYtWzJq1CgAjh07Rv369bly5QrR0dHY29tz4sQJGjduzOXLt86aF4+SeYXvgSvQOO8UzWeuxQLoi7s0TbtN//79WbhwIUlJSRw+fJiAgNzTw+SlTJkyDBkyhBkzZuRYX7lyZSwtLbl06RI7duygefPmBAQEsHPnTgIDA2nQoAHW1tbmfCt3ZJaR/39KQgScTIYn8k7jfDYsDmsLA96u9vewY5qm5Vd+RujFpUGDBly4cIEFCxbQtWvXQrXx2muv0ahRI8aMGZNjfcuWLdmxYwc7duxg9OjRhIaGsmPHDpydnWnRooU5up9vJW/kH3sV+leBtrnP94Np2qdKOXssLUre29c0reh69OjBG2+8wYABAwpV38XFhaeffppvv/02x/qb8/5Hjhyhfv36PPzww+zcufOez/dDSRz5V6gP8y7csUhQRJye79c0LU9DhgzB2dkZPz8/Nm/eXKg2Ro8ezUMPPURaWlrmupYtWzJt2jSqVauGhYUFrq6uxMTEcOzYMX788Ucz9T5/St3QNzXdyKXIBKq56+CvaVruPD09ef3114vUhpubG7179yY5OTlznZ+fHxERETz88MM51jk7O+PmZjpOmZaWho2NTZH2nR+qOE+3Lwp/f38JDAw0e7tB4XG0m7aFz/o24El/r7tX0DTtnjhx4gR16tS5392471auXMm8efNYvHjxXcvm9pkppfaJiP/d6pa8aZ+7CMo4zVOP/DVN+6+ZMGECK1eu5Jdffin2fZW6aZ+giDgAqrs73OeeaJr2oJo8eTKNGjXK8Zg8eXKR2500aRKHDh2iceO870NiLqVy5O/qYI2L/b07n1bTtJJl3LhxjBs37n53o0hK38g/PJ6qbnrUr2la6Vb6gn9EPNV08Nc0rZQrVcH/RlIqEXHJ+mCvpmmlXqkK/lln+uiRv6ZptzNHSuepU6cCsGvXLgICAmjUqBF16tRh4sSJmW24u7vTuHFjfH196dSpU2a2z3upVB3wDQrXZ/pompa37Cmd7ezsCpXS+aZBgwaxePFiGjZsSHp6OqdOncrc1q9fP77++msANm3axBNPPMGmTZvu6XUOpW7kb2FQeLvq4K9pWu5upnQGMlM6F0ZYWBiVKlUCwMLCgrp16+Za7tFHH2X48OHMnDmzcB0upNI18o+Iw6usHdaWpeo7T9MeOCP/GsnBqwfN2majio34ovMXdy3Xv39/Jk2aRPfu3Tl8+DBDhgwpUD7/m0aNGkWtWrVo27YtnTt3ZtCgQdja5n7b2CZNmvDDDz8UeB9FUaqiYFB4vD7Yq2naHZkjpTOYrtYNDAykY8eOzJ8/n86dO+dZ9n6k2Sk1I3+jUTgfEU/LGnnf5EXTtP+G/IzQi9PNlM6bN28mMjKy0O1Ur16dl156iWHDhuHu7p5nWwcOHLjneY1KTfAPi00mOc2ITzl9AxdN0+7MHCmd16xZQ9euXVFKcebMGSwsLHBxcbmt3JYtW5g5cyabNm0qYq8LptQE/4uRptM8vcvpg72apt1ZYVM6Z0/H/NtvvzFq1Cjs7e2xtLRk3rx5WFhYALBo0SK2bdtGQkICVatWZdmyZXrkX1wuRSUA6Fs3apqWp7i4uNvWtW3blrZt2wIwePBgBg8enGf9Y8eOZd6OceHChbmWuVsb90qpOeAbHJWAQYGHi9397oqmaSWQn58fBoOBjh073u+u5EupGflfjEqgkrM+zVPTtKKbPHkyS5YsybGuf//+D1Smz1IT/C9FJVBFH+zVNM0MdErnB0hwVIKe79c0TctQKoJ/XHIaEXEpeOngr2maBpSS4B+ccaaPnvbRNE0zKVLwV0q5KqXWK6XOZDyXzaVMI6XUTqXUMaXUYaVUv6LsszD0aZ6apmk5FXXk/zawUUR8gY0Zy7dKAJ4VkXpAZ+ALpdTtl7kVo0uRGSN/nc1T07Q7MEc+f6UUZ8+ezVw3ffp0lFIEBgYC4OPjQ0REROb+xowZk1l26tSpmXn/J06ciIeHR46bxMfExJjtvRY1+PcE5mS8ngP0urWAiJwWkTMZry8DYYB7EfdbIJeiEnCytcTZ3upe7lbTtAdM9nz+QKHy+fv5+eW4wGvp0qV5pnO2sbFh+fLlmV8Gtxo1ahQHDx7MfOSWHqKwihr8K4jIFYCM5/J3KqyUagZYA+eKuN8CuRiVQBWd1kHTtHwoaj7/Xr16sXLlSgCCgoJwdnbG3T338a6lpSXDhw9n+vTpRet0Idz1PH+l1AagYi6bCnSSq1KqEvAbMEhEjHmUGQ4MB/D29i5I83cUEpVA7UqOZmtP07TiNhIwbz5/aAQUfz5/JycnvLy8OHr0KCtXrqRfv378/PPPeZZ/5ZVXaNCgAWPHjr1t2/Tp05k7dy4AZcuWNWvyt7sGfxFpn9c2pdQ1pVQlEbmSEdzD8ijnBKwBxovIrjvsayYwE8Df398sCa5FhNCYRB6rc8cfJZqmaYB58vn379+fhQsXsm7dOjZu3HjH4O/k5MSzzz7Ll19+iZ1dzvQzo0aN4o033ihUH+6mqFf4rgIGAZ9kPK+8tYBSyhr4HfhVRJbcur24RcSlkJxm1Dl9NO2B8mDn83/88cd588038ff3x8nJ6a7lR44cSZMmTXjuuecK091CKeqc/ydAB6XUGaBDxjJKKX+l1E8ZZZ4CWgODlVIHMx6NirjffAuNMR248SirT/PUNC1/hgwZwoQJE/Dz8ytUfTs7O6ZMmZLvFBCurq489dRTzJo1q1D7K4wijfxFJBJ4LJf1gcDQjNdzgblF2U9RXL4Z/PXIX9O0fCpsPv/s+vfvX6DyY8aM4euvv86xLvucP8CKFSvw8fEpUr9uUvfj3pH54e/vLzfPiy2KH/8NYvLaExx6ryPOdvpUT037rzpx4sQ9v6HJgy63z0wptU9E/O9Wt8SndwiNScTRxlIHfk3TtGxKfErnkOhEPMrqKR9N08wnt3z+Tz755AOV5rnEB//QmEQ9369pmlnpfP4PgNDoBCrr4K9pmpZDiQ7+sUmp3EhK09M+mqZptyjRwf9yTBKgT/PUNE27VYkO/qExplTOeuSvaZqWU8kO/tGmC7w89chf07R8MEc+f3t7e8LCstKclSlTJvP11atX6d+/P9WrV6du3bp07dqV06dPc+HCBezs7HLk7v/111+L4R1mKdFn+4TEJGJtYcCtjM397oqmaQ+A7Pn87ezsCpXP383NjWnTpjFlypQc60WE3r17M2jQoMx8/wcPHuTatWt4eXlRvXp1Dh40dybTvJXo4B8anUjl/2/v/mOrKu84jr+/QKFohVqxQFcdiIxhw7wFMoIAVSQAAAmgSURBVDSKdtUNMNvAhC1TM5xladhYQDAZGhPmwj9uuqGdc6lmC/6xbLpMUkMyfgyp8S9FBEoNMCGpG7MtrIMyZ3pD4bs/7tOuYEt7f/Reeu7nldzce577nN7ne7l8+/Q5535PcSGjRlmuhyIiyXj0Uch0IozF4LnBC8b11PNfvnx5bz3/ZEo619TUsGXLFjZs2EBJSUlv+549eygoKGDVqlV9hpQoc9bS0jL0ODIk2ss+Z/QFLxFJTk855q6uLpqamliwYEFS+xcVFVFTU8Pzzz9/UXtzczPz5s0bcL/jx49ftOyTzC+cVER+5n/3F7J6xUgRyYQhzNCHSybq+a9Zs4ZYLHbR9XkHk+1ln8jO/OPd5zn5n7hm/iKStJ56/slewrFHcXExDz74IC+++GJvW0VFBfv27cvUENMW2eTf1qlz/EUkNenW8wdYv3499fX1dHd3A1BdXU08Hufll1/u7bN3717eeuuttMebisgm/3+qjr+IpCgT9fwnTZrE/fffTzweBxKnkW7dupVdu3YxY8YMKioqeOqppygrKwM+u+ZfV1eXdhyXE9l6/q+/f4L1rx3kzcfu5qbriwbfQURySvX8k6d6/v1oO5tY9pkysTDHIxERufJE9myf9s4urikcw1VjIxuiiOSI6vlfwdrOdjFlgmb9IpJ5qud/BWs7G9eSj4jIACKb/Ns7u5ismb+ISL8imfzPX3BOfRLXso+IyAAimfw7Polz/oIzWcs+IiL9imTy7z3NUzN/EUlCuvX8jx49SlVVFbFYjNmzZ1NbW8uOHTt6v7hVVFTErFmziMVirFixgsbGRiZOnEhlZSWzZs3irrvuYtu2bcMeJ0T0bJ+e0g6TJ6iOv4gMXbr1/NesWcO6detYunQpAIcOHWLOnDksWrQIgKqqKp599lnmz098B6uxsZGFCxf2JvwDBw6wbNkyxo8fzz333JPh6C4WyeTfrpm/yMj2l8eh7VBmf+aUObDk6UG7pVPPv7W1lfLy8t7tZGsDxWIxNm7cyAsvvDDsyT+yyz6jRxnX6QpeIpKkdOr5r1u3jurqapYsWcLmzZs5c+ZM0q8/d+5cjhw5kvR+yUpr5m9mJcCrwDSgBfi2u58eoO8E4DCw1d0HXjTLgLbOOKXXjGO0ruAlMjINYYY+XNKp5//II4+waNEitm/fTkNDA/X19Rw8eJBx44Y+Ec1WvbV0Z/6PA7vdfSawO2wPZBOQldql7Wd1jr+IpC6dev5lZWXU1NTQ0NDAmDFjaG5uTmr//fv3Z6XAXbrJfynwSnj8CrCsv05mNg+YDOxM8/WGRKUdRCQdqdbz3759O+fOnQOgra2Njo6OpA4YNzU1sWnTJlavXp3U66Yi3QO+k929FcDdW82s9NIOZjYK+AXwXWB4j2AE7Z1d3HnzpGy8lIhEUKr1/Hfu3MnatWspLExMPp955hmmTJly2X3efvttKisr+fTTTyktLaWurm7YD/bCEOr5m9lfgf5G/yTwirsX9+l72t2vvWT/HwFXufvPzex7wPyB1vzNrBaoBbjxxhvnffTRR8nEAsB/491U/GQHGxZ/kR9UzUh6fxHJDdXzT1469fwHnfm7+70DPWdm7WY2Ncz6pwIn++l2O7DQzH4IFAFjzewTd//M8QF3fwl4CRIXcxlsbP2Jd1/gG7eWUVE2IZXdRUTyQrrLPm8ADwNPh/uGSzu4+0M9j/vM/C93YDgtJVeP5VcPVA7XjxcRUT1/Ekn/NTNbCfwd+BaAmc0HVrn799P8+SIiV5wo1PNPK/m7ewf9HMR19/eAzyR+d98CbEnnNUUkutwdM30/ZyjS/T5AJL/hKyIjT2FhIR0dHVn7ktNI5u50dHT0nlWUikjW9hGRkae8vJwTJ05w6tSpXA9lRCgsLLyojlCylPxF5IpQUFDA9OnTcz2MvKFlHxGRPKTkLyKSh5T8RUTy0KDlHXLFzE4Bydd3+L9JwL8yNJyRQjHnB8WcH1KN+fPufv1gna7Y5J8uM3tvKPUtokQx5wfFnB+GO2Yt+4iI5CElfxGRPBTl5P9SrgeQA4o5Pyjm/DCsMUd2zV9ERAYW5Zm/iIgMIHLJ38wWm9lRMztmZsN23YBsM7PfmdlJM2vu01ZiZrvM7MNwf21oNzOrC+9Bk5nNzd3IU2dmN5jZHjM7bGYfmNna0B7ZuM2s0MzeNbODIeafhvbpZvZOiPlVMxsb2seF7WPh+Wm5HH86zGy0me03s21hO9Ixm1mLmR0yswNm9l5oy9pnO1LJ38xGA78GlgC3AA+Y2S25HVXGbAEWX9L2OLDb3WcCu8M2JOKfGW61wG+yNMZM6wYec/fZwG3A6vDvGeW440C1u98KxIDFZnYb8DNgc4j5NLAy9F8JnHb3m4HNod9ItRY43Gc7H2L+irvH+pzSmb3PtrtH5kbikpE7+mw/ATyR63FlML5pQHOf7aPA1PB4KnA0PK4HHuiv30i+kbhS3FfzJW7gKuB9YAGJL/uMCe29n3NgB3B7eDwm9LNcjz2FWMtDsqsGtgGWBzG3AJMuacvaZztSM3/gc8A/+myfCG1RNdndWwHCfWloj9z7EP60rwTeIeJxh+WPAySuib0LOA6ccffu0KVvXL0xh+c7geuyO+KMeA74MXAhbF9H9GN2YKeZ7TOz2tCWtc921Eo693cJoHw8nSlS74OZFQF/Bh5197OXudJTJOJ29/NAzMyKga3A7P66hfsRH7OZfR046e77zKyqp7mfrpGJObjD3T82s1Jgl5kduUzfjMcctZn/CeCGPtvlwMc5Gks2tJvZVIBwfzK0R+Z9MLMCEon/9+7+emiOfNwA7n4GaCRxvKPYzHoma33j6o05PD8R+Hd2R5q2O4BvmlkL8EcSSz/PEe2YcfePw/1JEr/kv0wWP9tRS/57gZnhLIGxwHeAN3I8puH0BvBwePwwiTXxnvYV4QyB24DOnj8lRxJLTPF/Cxx291/2eSqycZvZ9WHGj5mNB+4lcRB0D7A8dLs05p73YjnwpodF4ZHC3Z9w93J3n0bi/+yb7v4QEY7ZzK42s2t6HgNfA5rJ5mc71wc9huEgyn3A30iskz6Z6/FkMK4/AK3AORKzgJUk1jl3Ax+G+5LQ10ic9XQcOATMz/X4U4z5ThJ/2jYBB8LtvijHDXwJ2B9ibgY2hvabgHeBY8CfgHGhvTBsHwvP35TrGNKMvwrYFvWYQ2wHw+2DnlyVzc+2vuErIpKHorbsIyIiQ6DkLyKSh5T8RUTykJK/iEgeUvIXEclDSv4iInlIyV9EJA8p+YuI5KH/AUMIF8lcummoAAAAAElFTkSuQmCC\n", 186 | "text/plain": [ 187 | "
" 188 | ] 189 | }, 190 | "metadata": {}, 191 | "output_type": "display_data" 192 | } 193 | ], 194 | "source": [ 195 | "# plotting all together\n", 196 | "plt.plot(plot_x, -plot_y_nwj)\n", 197 | "plt.plot(plot_x, -plot_y_jsd, c = \"green\")\n", 198 | "plt.plot(plot_x, -plot_y_mine, c = \"yellow\")\n", 199 | "plt.plot(plot_x, -plot_y_nce, c = \"red\")\n", 200 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 201 | "plt.legend([\"MI_NWJ\", \"MI_JSD\", \"MI_MINE\", \"MI_NCE\", \"MI_STD\"])\n", 202 | "plt.show()" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": {}, 208 | "source": [ 209 | "MINE and NCE almost coincide and hence, we don't see the yellow line." 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [] 218 | } 219 | ], 220 | "metadata": { 221 | "kernelspec": { 222 | "display_name": "Python 3", 223 | "language": "python", 224 | "name": "python3" 225 | }, 226 | "language_info": { 227 | "codemirror_mode": { 228 | "name": "ipython", 229 | "version": 3 230 | }, 231 | "file_extension": ".py", 232 | "mimetype": "text/x-python", 233 | "name": "python", 234 | "nbconvert_exporter": "python", 235 | "pygments_lexer": "ipython3", 236 | "version": "3.6.5" 237 | } 238 | }, 239 | "nbformat": 4, 240 | "nbformat_minor": 2 241 | } 242 | -------------------------------------------------------------------------------- /notebooks/pytorch_mine_vs_nce.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "The code is adapted from [Masanori Yamada's repository](https://github.com/MasanoriYamada/Mine_pytorch) on MINE implementation." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": { 14 | "colab": {}, 15 | "colab_type": "code", 16 | "id": "BS2h2kglwPa_" 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "import torch\n", 21 | "from torch.autograd import Variable\n", 22 | "import torch.nn as nn\n", 23 | "import torch.nn.functional as F\n", 24 | "from torch.utils.data import Dataset, DataLoader\n", 25 | "import numpy as np\n", 26 | "import matplotlib.pyplot as plt\n", 27 | "from tqdm import tqdm" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 3, 33 | "metadata": { 34 | "colab": {}, 35 | "colab_type": "code", 36 | "id": "Ze6Kk6SsQKiE" 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "# data\n", 41 | "np.random.seed(1234)\n", 42 | "var = 0.2\n", 43 | "def func(x):\n", 44 | " return x\n", 45 | "\n", 46 | "def gen_x():\n", 47 | " return np.sign(np.random.normal(0.,1.,[data_size,1]))\n", 48 | "\n", 49 | "def gen_y(x):\n", 50 | " return func(x)+np.random.normal(0.,np.sqrt(var),[data_size,1])" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 4, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "np.random.seed(1234)\n", 60 | "data_size = 1000000\n", 61 | "x = gen_x()\n", 62 | "y = gen_y(x)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 5, 68 | "metadata": { 69 | "colab": { 70 | "base_uri": "https://localhost:8080/", 71 | "height": 34 72 | }, 73 | "colab_type": "code", 74 | "id": "C_wM4ty1Qldx", 75 | "outputId": "2892779e-b4dc-4f29-9858-28afb707d11d" 76 | }, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "Mutual information calculated through traditional method is: 0.6584537102332939\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "x = gen_x()\n", 88 | "y = gen_y(x)\n", 89 | "p_y_x = np.exp(-(y - x)**2 / (2 * var))\n", 90 | "p_y_x_minus = np.exp(-(y + 1)**2 / (2 * var))\n", 91 | "p_y_x_plus = np.exp(-(y - 1)**2 / (2 * var))\n", 92 | "mi = np.average(np.log(p_y_x / (0.5 * p_y_x_minus + 0.5 * p_y_x_plus)))\n", 93 | "#mi = mutual_information(x, y)\n", 94 | "print(\"Mutual information calculated through traditional method is:\", mi)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 6, 100 | "metadata": {}, 101 | "outputs": [ 102 | { 103 | "name": "stderr", 104 | "output_type": "stream", 105 | "text": [ 106 | "100%|██████████| 500/500 [00:07<00:00, 66.77it/s]\n" 107 | ] 108 | } 109 | ], 110 | "source": [ 111 | "np.random.seed(1234)\n", 112 | "H=10\n", 113 | "n_epoch = 500\n", 114 | "data_size = 20000\n", 115 | "\n", 116 | "class Net(nn.Module):\n", 117 | " def __init__(self):\n", 118 | " super(Net, self).__init__()\n", 119 | " self.fc1 = nn.Linear(1, H)\n", 120 | " self.fc2 = nn.Linear(1, H)\n", 121 | " self.fc3 = nn.Linear(H, 1)\n", 122 | "\n", 123 | " def forward(self, x, y):\n", 124 | " h1 = F.relu(self.fc1(x)+self.fc2(y))\n", 125 | " h2 = self.fc3(h1)\n", 126 | " return h2 \n", 127 | "\n", 128 | "model = Net()\n", 129 | "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n", 130 | "plot_loss_mine = []\n", 131 | "plot_loss_nce = []\n", 132 | "for epoch in tqdm(range(n_epoch)):\n", 133 | " x_sample=gen_x()\n", 134 | " y_sample=gen_y(x_sample)\n", 135 | " y_shuffle=np.random.permutation(y_sample)\n", 136 | " \n", 137 | " x_sample = Variable(torch.from_numpy(x_sample).type(torch.FloatTensor), requires_grad = True)\n", 138 | " y_sample = Variable(torch.from_numpy(y_sample).type(torch.FloatTensor), requires_grad = True)\n", 139 | " y_shuffle = Variable(torch.from_numpy(y_shuffle).type(torch.FloatTensor), requires_grad = True) \n", 140 | " \n", 141 | " pred_xy = model(x_sample, y_sample)\n", 142 | " pred_x_y = model(x_sample, y_shuffle)\n", 143 | "\n", 144 | " # Noise Contrastive Estimation(NCE) loss\n", 145 | " #ret_nce = torch.mean(pred_xy - torch.log(1 + pred_xy + torch.sum(torch.exp(pred_x_y))))\n", 146 | " ret_nce = torch.mean(pred_xy - torch.mean(torch.log(torch.sum(torch.exp(pred_x_y)))))\n", 147 | " # Mutual Information Neural Estimation (MINE) loss\n", 148 | " ret_mine = torch.mean(pred_xy) - torch.log(torch.mean(torch.exp(pred_x_y)))\n", 149 | " loss_mine = - ret_mine # maximize\n", 150 | " loss_nce = - ret_nce # maximize\n", 151 | " plot_loss_mine.append(loss_mine.data.numpy())\n", 152 | " plot_loss_nce.append(loss_nce.data.numpy())\n", 153 | " model.zero_grad()\n", 154 | " loss_mine.backward(retain_graph=True)\n", 155 | " loss_nce.backward(retain_graph=True)\n", 156 | " optimizer.step()" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 7, 162 | "metadata": { 163 | "colab": {}, 164 | "colab_type": "code", 165 | "id": "0HJFmVYJQ4uf" 166 | }, 167 | "outputs": [], 168 | "source": [ 169 | "plot_x = np.arange(len(plot_loss_mine))\n", 170 | "plot_y_mine = np.array(plot_loss_mine).reshape(-1,)\n", 171 | "plot_y_nce = np.array(plot_loss_nce).reshape(-1,) - np.log(data_size)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "Plotting one after the other as the overlap is too sharp!" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 8, 184 | "metadata": { 185 | "colab": { 186 | "base_uri": "https://localhost:8080/", 187 | "height": 349 188 | }, 189 | "colab_type": "code", 190 | "id": "1-ezg88MRVEH", 191 | "outputId": "94ee42a7-9e19-4296-eff7-57e5077860c9" 192 | }, 193 | "outputs": [ 194 | { 195 | "data": { 196 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VPW9//HXJ/u+JxCSAAHCEgRBA6IIiIqiVmmtbaGtVa8W26vXar2tWq/+rPW22t7r1kutuNVaW7VW24i4IG4gi4R9hyQEEhLIvq+T+f7+mGGYhASQOckkM5/n48Ejc875cr7fM4T3fOd7vuccMcaglFLKvwR4uwFKKaX6n4a/Ukr5IQ1/pZTyQxr+SinlhzT8lVLKD2n4K6WUH9LwV0opP6Thr5RSfkjDXyml/FCQtxvQm6SkJDNy5EhvN0MppQaVjRs3Vhpjkk9VzpLwF5H5wFNAIPC8MebRbtufAOY6FyOAFGNM3Mn2OXLkSPLy8qxonlJK+Q0ROXg65TwOfxEJBJYA84ASYIOI5Bpjdh0rY4y5y638fwBTPa1XKaXUmbNizH86kG+MKTTGtAOvAQtOUn4R8DcL6lVKKXWGrAj/NKDYbbnEue4EIjICyAQ+tqBepZRSZ8iK8Jce1vV2n+iFwJvGmM4edySyWETyRCSvoqLCgqYppZTqiRXhXwJkuC2nA6W9lF3ISYZ8jDFLjTE5xpic5ORTnqxWSil1hqwI/w1AlohkikgIjoDP7V5IRMYB8cBaC+pUSinlAY/D3xhjA24HPgB2A28YY3aKyMMico1b0UXAa0YfHaaUUl5nyTx/Y8xyYHm3dQ92W37IirqUUqonxhje3nyYyyYOJSp0wF6/OmDo7R2UUj5h++E6fvrGVh745w5vN2VQ0PBXSvmEhlYbAAcqm/qtTmMMrR09Tl4c8DT8lVInVdHQxj83H6aj0+7tprgYY/jVsl1sPFjjWlfZ2AZAc7uNV9cfpNNuMMawoaiaU51q/Ofmw9Q2t3/ldjyxYh/jH3j/jD8ANh+q4Y0Nxacu2Ac0/JUaJPKKqtl3tOGE9e02O3UtHQA8v6qQK59aRXl9K2V1Lb3uq665g4/3HOX5VYWAI4R6C/dFz63jzte3kLultxncJ/pkTzmVjW3kFVXzfx/v7zF81xRUkl/uOJ4ln+SfVgiW1bXwl3UHqWhs44XVB/jmM2uwddpZtq2UzYdqAdh3tJH7397BPzaW8Jd1B/nWH9fy3o4jve7zcG0Ld76+hSkPr6DK+QHS0WnH1u39+HDnEQ5VNdPUZuOa/1vN0s8LePrjfACKqo5/2/hkTzn3v729ywfCs58V8Mgyxx1vNh+qIXdrKZWNbXzjD2v4+T+20dDawT1vbqOkpvmU74FVZKBOvsnJyTFnfGO39+6FI9utbZBSPTAYpNt1jnYMGAiQnq5/BLsxvW7rbtvhWgIDhPT4CHaX1RMaFMDUjHjX9g67na3FtdgNTB+ZwLoDVa5tApyXmehaLqhoJCw4kLS4cDYdqqHdGW4TUmPYXVZPRnwEAQI2uyEjPgKDIb+8kaomR484MTKErJToE9p4pL6ViJBAYsKCabN1cqi6maqmdpKjQmlu76Sp3UZ8RAhJUSEkRoa63oMvi6oBmJGZ6Gr3DLf2dlff2sGusnoAhsaEcaS+9aTvXWpsGA2tNhrbbESFBtHRaWdSWixBAQHYjaG4ppnU2HDabJ3sLHXsNykyhKiwYA7XtBAUKExOj0UQ6lo62H2knkARhsWFUVzTgnD8ataxKVFEhgYRHBjAniP11LfaiA0PZtxQx/v15QHHsWalRLG/vBGAwACh0+7YQ0p0KOUNbcSFBxMZGsSw8dMJvPKxkx5fb0RkozEm51Tl9JS4GnDsGMrr20iJCcXWaRCB4ICv9iV1S0ktQQHCWcNiv3L9LR02GlptpESHAY6AL6529KLjI4MJChDCg4No7rCxraSOCakxxIYFu/7+rtJ6mtpsnJeZyMHqJupaOpicFkdTu42iqiaa2jqZkBrNwapmEiJDGBYb7vq7dS0dFFU1kR4fTlBAAM3tjt5jo3M8u81mp7immcjQIOIigtlxuA6bM0Ds3S6sN0BpXQsp0aEEBAgVzl5tQmSIK/gBdjsDtbnd5gr65OhQ7HbjWgaob+mgrqWDNlsnkaFBRIYE0dLRSVFVE4EiTBuZwN6jDa42u3+TqGlup6a5ncTMUFptnWwprnVtc2/LniP1DIkNIzIkiJDAAAwGu4EA6TqWf6rgB6ht7qDF2ftubHO8f/WtNhIiQqhr7aCsrpWQwABCg47/blU2tVPpPOYOu+M8QnN7p6tn32kM5Q1trvf3mIY2G/vKG0mOCqWh1UZEcCB1LR0cqmru8kF/LPhHJUVS6HY8x/ZZ29JBbUsHgfWtDDvlEXrGN8P/ikdPXUb1ub+uP8So5EhmjOq9N9eTF1cV8si7u3nkvLP4r3/uQAQO/OYqwBFQFQ1tjEiM7PXvry2oYtFz6wAouumq06pzxa6jjtki88Yy4d53Abhp5kjKG9pYNG04339hvaOg864jRY9exT1/28w77aVcm5DG49+ZAoDdbvjaLxyzntddewlzfrMSgFUL5nL5k5+7gjGtNpzDDS1MT0rgjZvOB+C/393Fc6sOOCrods7yG8PSePvoYcdCuePH04umcsfeza4yExti2Nle3/UvHoFvpKbxbzMzWfh/qx3rHCM9hAcHusIRAPch74KuuzlneBybDtXCoePrfnzRaDrthqWFjh1eEzaM3MZS7p43ls3FtXy8p5zuxtREcaSulcZ22/GV+W4F2oE6uGB0IpeNHcJD7ziGSv74/XN5sHQH9hCICQ+isMLxBp2XmUBKTBgPXZ3NuY981LUy5/FEhAS63vehdWHcMjmTsrpWXjh0ANxGg265MJPnVx/osotZIUmsOlgJwPTMBEcP3u19uvPSLP6y7iCVFc6Vjg4+r1w/nRdWH+DTvcdvU/ONqWm8vdnxb1h0x1X86A9fON5TQASCAoSOTsdHygPp2dx8wrtnLR3zV33mF29vZ+HSdScts/9og6vnaeu0c99b23l3exngGMsGMAbX1+P73trOnN99SrWzd/bK2iLW5Dv+c1Y3tXPH3za7gr83W4treSh3p6tn+lDuTn745zyeXrmfvUeOj6m/9EUR724r47a/bjphHy+vKeKdrY4x8H3OcetX1h1k8i8/dJWZ4Qx+wBX8/zYzk7njkjlc6/gmseVQLQermli9v/J48AOXTxzSpb5Nh2oYP7TrkMuSj/OJiwjmTzdNA3ANXXT39ubDXO0M/rFDolzrf33tWa7XP75otOv1kJhQrpw0tMs+5p91fPmG80cA8MynBSz9vNC1Ptf5fswYnUhs+PFvQt89bzhZKY5688sbuXzi8X2lRId2qSc7NYbs1BjyDtbw8LJdBDg7zU+v3E95Qxu3zMrk/Z/M5rkf5HDfFeN5bfEMfr9oKolRoWx6YB6f/eyiE47fve1H6lt55N3dvJF34vmFy8/qeswhgQGs2l/pWv52TgbDYh3fBq+cNJRnrz+X2+eOOaFzkxIdygWjk1g4LaPL+jsvzeqy/PvvnsOMUQkAXD15GB/eNYffL5pKUlSI6/9EX/LNnr/qN7/7YA+r86v4120zu6w/1ewHYwxPrdzPkx/tZ1hsGKvuuZjdZQ387cvjXctX1x9/JsVHu49S2djGv5wnHZ/6aB/fPW8ED/xrJwD/vG0mN7z4pevE5zE3vfQlF2Yl8/qGQ8SFh/Dnm6fzrWfX0m6zMzoliutnjGDZtjJX+edWFdLdsX0umDLMVf//y3XUe9XkVN7dVkbmfe8SHBDQZQjDXXN7J1OHx/Hg1dm8saGYT/ZWIOIY8pjzu0+7lB0SE8oz3zuXu/++ldLaFtYfqOZgVTM3zRzJHrcPp71HG5iVlUR6fIRr3b9um8kLqw9gN6bLcR2z/I5ZrCus5qmV+7h4vOMDJjo0iP+4eAzXnZtOUlSoK7if/ayA37y3B4DZY5P59fI9fCcng19cNYGX1/b8vJCz0mKYlBbLuSPiXb3cRdOGc8/l4/nmH9dwz/zxzMsewh2XjOGTPeXcODOTvKJqrvuj464vIxIjmDsuhZ//YxuRIYF8/vO5XPfHta6x/jHJUYQEBTAvewjQ9QMyITKEhMgQ/uuqCUwcFovdGF5eU8Scscm8tekwQ2JCeePW85nzu09d00Ld5YyIJykqhMpGR8fi5X+bzs7SOuaMTSY8xHGuZNX+Cv61pZSU6DDXh9ji2aNYubuctPhw8ssbuX7GCAIDhEsnDOGn88by+Ip95IyIZ0RiJPdfOYExzg/gtLhwXlt8PjtL6xiZGElkaBCZSZG8kVes4a8GlqP1rawrrGLBFMcdu9tsnSz5xDE+UFzdTE1zOxEhgdz/9g7uu3KC6+81tHawvaSOC8YkAVDe0Mo/Nx/myY/2A1Ba18ptr24iNLjrF9GCiuNjH7e+srHLtpfXHmTFrqOu5W8+s4aI4MAT2vzJ3go+cfvq/cM/57m+UbyZV0xUaCCVjW3ceWkWf/ikgDc3lvR6/E8tnMrj357C/364l4KKRm6bO4bRyVGsL6yisrGd9k47s7KSuvQWNz0wj6rGNqqb2hnt7P1OznCch7h19mg67XaGxoYzbkg0e47U88i7u2m32QkIEJ74zhQ2Hqzhm8+sAWBKRhzXTk3jrc2H+a+rJvDIu7uZMSqRYXFhrvomp8fy9KKp5BVVs2xbGRNSY1jy3alc/L+fcVn2EIICA7gwK4kLsxz/Fr9fNJXJ6bFEhAQxOvn4twKAW+eMZnxqDI+9t4fMpEh2/PJyIoIDCQgQVt8zl3e2lvHY+3tc5X9z7SQWTR8OwHenD2f+WUNJijres//op3Ncr0ckRnLjzEwAckYm8MR3zuau17eSHB3qen9umTWKxKhQWpxDNqOTI13tPplbZo1yvZ45Jsl1fqHdZmdEYiQzRiWwrrCaV285j5iwYL717BpaO+yICJ//fC65W0p5auV+pg6P4/zRXXv115zt6ABEhx2Pzsnpcez85eWU1rXwRl4JP3J+iwoKDOCOS7L4+pQ04iIdH6g/nD2K7iZ2Oy/1wNeyCe/hd9lqvjnbR/WJ619Yz6r9lXz2s4sYkRjJp3vLufGlDT2WnT02mc/3OUJ3Ulos2w/X8T/fOpsAgT+vPdjlhF93549KZG1h1Qnrbzh/BC+vPch9V4znpS+KOFLfyrDYMErrHCf/PrhzNv/+6sYuHxo9mZWVxLSRCTy+Yp9r3fM/yKG4pplfvrOLnBHx5LnNHz+m6NGezx/Y7Ya/rD/I4ZoW/vPycQSK8MxnBeSMiOe8Xs53vLe9jIvGpRAecvw/eXlDK9P/eyVZKVGscAZlfWsHkx9yDCV9/rO5pMaFUd3UzpCYMPYcqWd0chTBgQG8vuEQ0zMTyUxynAupamzj3Ec+4ulFU7nm7GF8kV/JlIw4Ii287cG6wioWLl3HlZOGMndcCteek05gwOnNYuqutaOT//lgL/9xcRaxEcGsL6zinBHxBAcGcMVTq9hdVs/a+y4m1e3k+Ok69l6MHRLFh3fNobWjkw1F1czKctw5uL61g85OQ3xkyCn3ZYzhw11HuXBMkqXvpZVOd7aPhr+iuLqZiJBAYsODKa1tJSUmlDC3nscfPytgcnosv1q2m91l9Tzy9bP4/owR/Hr5bl5cfYCMhIivfFXlzRdmcuGYJG760wYuGJ3ImoIq4iKC+cUVE/j2tAxGOk+6Ajz4tWwiQwP5zrThrC+sYurweLaW1LL080LuvDSLLw9UU9PcwU/njaWupYPrX1jPtpK6E+q889IsnvxoP8//IIf0hHDmP7nKte3ju+cwKjmK/PIGIkOD+N7z6ymsaOLZ68/l5TVFzBiVyB2XZJ2wT6u99MUBLh6f0uWE9strHOceXr91BnKaU0T7g91ueGH1Ab5xTlqXHr7VDlY1saagyvWt4ky8su4gF41NJiMh4tSFBzkNf3WCBUu+YNqIeO6/agJtNjthwYHYOu2Muf89RidHcsMFI3nQOYYO8NwPcjg7PZbpv3acuIwMCaSpvZPEyBB+cmkWb24sITw4kNdvPZ8nVuzjqZX7T1p/Wly460TnSzdOY+74FMDR66tsbOsydu0e/h/9dA5jUroOSZxMXUsHy7eXYbMbxiRH0dRmo6PTzvyzhrLjcD2T0h1fs+95cxvrDlRx6+zRLJqe0SVY/7GxhLv/vpUVd80ma8iJc9uVGqg0/P1IcXUzxTXNVDS0cfXkYQR0++pd19zB3X/fwke7HVPvbp09ipfWFPHjOaOZlBbLLX/u+X2eOSaR+WeldrlR1piUKDrtxtXTXzx7FL9wju9vKa7l60u+IC0unLvmjWVNfiU3z8rkqqdXkxIdyoq75vDu9jL+8Gk+794xq8uMkO7e3lxCrvPE2n9/4yyCAq2fmHbsd7+33nRdcwexEb23UamBSMPfT7ifEAS4be5ofnb5eMAxlhkTFsxDuTv505oiVxkRiAh29OLdrzg8tm3FXXP43vPriI8IITQogK1uQyhPLZzC1ZOHMco5l/3xb5/Nteeku7Y/+1kBOSPjOXeEYwqb3W743Yd7uXZqmvagleoHpxv+Os9/EGvt6OwS/AAvri5ibUEVG4qqmfzQh7y/4wjrnZeWH2MM/Ofl4xChS/CD40NhTEoUV08exp4jDWwtqeNYx/i2uaNZMCWNgAAhI8Fx4m1st0C/dc5oV/ADBAQI98wfr8Gv1ABjSfiLyHwR2Ssi+SJyby9lvi0iu0Rkp4j81Yp6/d2mHmaktHR0sui5dXzLOW/6N+/tPmHOcGhQANedm+66rcCi6Rlclj2E2PBgHrtuMgDjU2MAuPGCkTxwVTYAU9zuKfP8D6axaHrGCRceKaUGB4/nKolIILAEmIfjYe4bRCTXGLPLrUwWcB8w0xhTIyIpntar4P2dXe9UOG5INHu73fXxYFXXuwTecmEmQ2LCiA4L5qaZI/lsXwV3XJJ1whS6BVOGMSo5kqkZcRgD41OjOd9t2uK4odH85trJFh+RUqq/WNHznw7kG2MKjTHtwGvAgm5lfggsMcbUABhjTrzph/pKfrVsF39ee5Dv5GSw5LvnuNb//Ufn88NZmb3+vX+fO8Z1ockts0bxys3n9Th3OjgwgHOGxyMiBAQIF4xOGlDTDJVSnrEi/NMA9xtllDjXuRsLjBWRL0RknYjMt6Bev9Vus/P6hmJGJEbwiysnkBR1/OKUaSMTuPuyca7lmy/MJDjweGjHnWSGjVLKf1hxiVpP3cHuU4iCgCzgIiAdWCUiZxljulzmKSKLgcUAw4ef+QUdvqbTbrjr9S0snJbBuSPjmfXYJzS22fjfb59NbESw60pD43zbj12gFRsezANfy+aBr2WTV1TNmoKqE6aBKqX8kxXhXwK4374uHej+yJ8SYJ0xpgM4ICJ7cXwYdLk3gDFmKbAUHFM9LWibT9h40PHkn9ytpQyNCaO8oY0pGXHMdl6enjUkiumZCdwz/3iPf+19FxMadPwq3ZyRCeSMTDhh30op/2TFsM8GIEtEMkUkBFgI5HYr809gLoCIJOEYBjrx9omqR8duHQyOW9JOSovl7X+/wHVfmNCgQN649fwuUyxTY8NJOI17lSil/JPHPX9jjE1Ebgc+AAKBF40xO0XkYSDPGJPr3HaZiOwCOoGfGWNOvHOXOkFpbQtv5BVz0bhkslNjWH+gmocXTNSTr0opj+gVvgOU3W54blUhSz8vpKWjkw/unO0XN6VSSnlGn+E7yK0pqHI9SOO3103W4FdKWUrDfwDKL2/gBy86nhn7+LfP5htTu8+cVUopz2j4D0Af7jqK3cCLN+a4HrWnlFJW0hu7DUBfHqhmTEqUBr9Sqs9o+A8wxhi2FNeSMyL+1IWVUuoMafgPMNVN7dQ2d+gtkJVSfUrH/AeQtzeXsGpfJQCjkiNPUVoppc6chv8ActfrW12vxySf/jNrlVLqq9JhnwFqWNyJt1lWSimraPgPEN97fp3r9V9uPo9AvfumUqoPafgPAOX1rXyR77jV0VMLp3BhVpKXW6SU8nUa/gPAztLjz9hNigr1YkuUUv5Cw38A2HTI8SD2a84exrk6v18p1Q90to+X1bV08Kc1Rcwem8zTi6Z6uzlKKT+hPX8v23m4joZWG7dc2PtD15VSymoa/l7U2tHJX788BMCYFJ3Xr5TqP5aEv4jMF5G9IpIvIvf2sP1GEakQkS3OP7dYUe9g94dPC1i2rQyAoTFhXm6NUsqfeDzmLyKBwBJgHo4HtW8QkVxjzK5uRV83xtzuaX2+pLKxzfU6QOf1K6X6kRU9/+lAvjGm0BjTDrwGLLBgvz6vqc0GwH1XjPdyS5RS/saK8E8Dit2WS5zruvumiGwTkTdFJMOCege9wzUtzBiVwK1zRnu7KUopP2NF+Pc0XtH9qfDvACONMZOBj4CXe9yRyGIRyRORvIqKCguaNrCV1rboPXyUUl5hRfiXAO49+XSg1L2AMabKGHNsgPs54NyedmSMWWqMyTHG5CQnJ1vQtIHr073llNW3MiJBb92slOp/VoT/BiBLRDJFJARYCOS6FxCRVLfFa4DdFtQ7aG06VMNNf9rAqKRIbrhghLebo5TyQx7P9jHG2ETkduADIBB40RizU0QeBvKMMbnAHSJyDWADqoEbPa13MPvzmiKiQoL4x48vIC4ixNvNUUr5IUtu72CMWQ4s77buQbfX9wH3WVGXL1idX8m8iUM0+JVSXqNX+PazysY2KhvbyU6N8XZTlFJ+TG/s1o9Kapp5f8cRAMYP1fBXSnmPhn8/uurp1dS1dAAwPjXay61RSvkzHfbpJ8YYV/AHBYg+tEUp5VUa/v2kpKbF9fqlm6Z5sSVKKaXDPv1ma0ktAO/cfiGT0mO93BqllL/Tnn8/2VpcS0hQAOOG6li/Usr7NPz7ydaSOiYOiyEkSN9ypZT3aRL1A1unne0ldZydHuftpiilFKDh3y/yKxpp6ejk7Awd61dKDQwa/v1gXUEVgPb8lVIDhoZ/H6tuaufR9/cwOT2WkYl6+2al1MCg4d/H1hRU0tph55fXTNTn9CqlBgwN/z6050g9t/91MyIwKU3H+5VSA4eGfx9ak+8Y6188exRBgfpWK6UGDk2kPlRQ0UhcRDD3zh/v7aYopVQXGv595PUNh3h1/SGGRIchomP9SqmBxZLwF5H5IrJXRPJF5N6TlLtORIyI5FhR70D29Mp8AJKi9WldSqmBx+PwF5FAYAlwBZANLBKR7B7KRQN3AOs9rXOga7fZqWpqY0xKFI99c7K3m6OUUiewouc/Hcg3xhQaY9qB14AFPZT7FfBboNWCOge0fUcbaO2wc+elWaTHR3i7OUopdQIrwj8NKHZbLnGucxGRqUCGMWbZyXYkIotFJE9E8ioqKixomnfsO9oA6KMalVIDlxXh39PZTOPaKBIAPAHcfaodGWOWGmNyjDE5ycnJFjTNO/YebSAkMICRidrrV0oNTFaEfwmQ4bacDpS6LUcDZwGfikgRMAPI9eWTvnvKGhiVHKlz+5VSA5YV6bQByBKRTBEJARYCucc2GmPqjDFJxpiRxpiRwDrgGmNMngV1DzjtNjt5RdWcMyLe201RSqleeRz+xhgbcDvwAbAbeMMYs1NEHhaRazzd/2CTV1RNU3snF40dvMNWSinfZ8kzfI0xy4Hl3dY92EvZi6yoc6B6fvUB4iKCmTkmydtNUUqpXumgtIXqWjr4eE853z9vBJGhlnyuKqVUn9Dwt9C2kloAZoxK9HJLlFLq5DT8LbShqAaASel6+2al1MCm4W+Rkppm/vhZAbOykogND/Z2c5RS6qQ0/C3yyd4K2m2OJ3YppdRAp+FvkS/2V5IWF05mkj6nVyk18Gn4W2RzcQ3TMxP03v1KqUFBw98ClY1tHK1vY+IwvZGbUmpw0PC3wK7SegCyUzX8lVKDg4a/h+x2w/98uJew4AAmDtMpnkqpwUHD30N7jzawraSO+6+cQGyETvFUSg0OGv4e2lBUDcBF41K83BKllDp9Gv4eWn+gmtTYMNLjw73dFKWUOm0a/h4wxrDhQLVO8VRKDToa/h4oqmqmvKGN6ZkJ3m6KUkp9JRr+Hsjd4nha5ewsfXCLUmpwsST8RWS+iOwVkXwRubeH7T8Ske0iskVEVotIthX1elNzu41X1x9kVlYSGQn6oHal1ODicfiLSCCwBLgCyAYW9RDufzXGTDLGTAF+Czzuab3e9saGYsob2vjJJVnebopSSn1lVvT8pwP5xphCY0w78BqwwL2AMabebTESMBbU61XrCqsZnhBBzkgd71dKDT5WPGswDSh2Wy4BzuteSERuA34KhAAX97QjEVkMLAYYPny4BU3rG8YYNh2q4fzR+sQupdTgZEXPv6c5jif07I0xS4wxo4F7gP/qaUfGmKXGmBxjTE5y8sA9iVpa10p5QxvnDI/3dlOUUuqMWBH+JUCG23I6UHqS8q8BX7egXq/ZfMjxuEYNf6XUYGVF+G8AskQkU0RCgIVArnsBEXE/K3oVsN+Cer3m493lhAUHMD412ttNUUqpM+LxmL8xxiYitwMfAIHAi8aYnSLyMJBnjMkFbheRS4EOoAa4wdN6vWVtQRVvbT7M1yanEhyol0kopQYnK074YoxZDizvtu5Bt9c/saKegWBtQSUi8JtrJ3m7KUopdca06/oVbS6uZfzQGKLD9PbNSqnBS8P/K2jt6GTzoVrOGR7n7aYopZRHNPy/gs/2VdDYZuOyiUO93RSllPKIhv9X8Pm+CqJDg7hAL+5SSg1yGv5fwbaSOialx+osH6XUoKcpdppaOzrZc6Seyek63q+UGvw0/E/TM58W0NFpmJ2V5O2mKKWUxzT8T9M/NpUwd1wyF4zR8FdKDX4a/qehuLqZkpoW5owduDebU0qpr0LD/zR8kV8JwAyd5aOU8hEa/qfh3e1ljEiMYNwQvZGbUso3aPifQrvNzrrCKi7LHoJIT48uUEqpwUfD/xQKKxvp6DSclRbr7aYopZRlNPxPYU9ZAwATUmO83BKllLKOhv9JNLbZePS9PYQEBZCZFOnt5iillGU0/E9iTX4lR+pb+fmW4uiBAAANbklEQVTl4/SWDkopn2JJoonIfBHZKyL5InJvD9t/KiK7RGSbiKwUkRFW1NvXNhfXEhwofH/GoGiuUkqdNo/DX0QCgSXAFUA2sEhEsrsV2wzkGGMmA28Cv/W03r5mjGFtQRUTUmMICw70dnOUUspSVvT8pwP5xphCY0w78BqwwL2AMeYTY0yzc3EdkG5BvX1qTUEVW4pruXZqmrebopRSlrMi/NOAYrflEue63twMvGdBvX3qzY0lxIYHs+i84d5uilJKWc6KB7j3dOWT6bGgyPeBHGBOL9sXA4sBhg/3Xuh2dNpZsesoV04aSmiQDvkopXyPFT3/EiDDbTkdKO1eSEQuBe4HrjHGtPW0I2PMUmNMjjEmJznZezdR21ZSR2ObjbnjUrzWBqWU6ktWhP8GIEtEMkUkBFgI5LoXEJGpwLM4gr/cgjr71PoDVQBMz0zwckuUUqpveBz+xhgbcDvwAbAbeMMYs1NEHhaRa5zFfgdEAX8XkS0iktvL7rzObje8tekwk9JiSYwK9XZzlFKqT1gx5o8xZjmwvNu6B91eX2pFPf1hTUEV+eWNPPmdKd5uilJK9Rm9bLWbZdtKiQwJZP5ZQ73dFKWU6jMa/m6MMXy8p5y541P0wi6llE/T8HdTUtNCeUObnuhVSvk8DX83Gw/WAHDO8Hgvt0QppfqWhr+TMYZX1h0kNTaM8UP1cY1KKd+m4e/01qbDbDxYw+0XjyFIb9+slPJxmnJOf/vyEOOHRrNomt7LRynl+zT8gTZbJ9sO1zErK4mAAH1Iu1LK92n447iXT7vNzrkjdJaPUso/aPgDy7aWEhIUwAVjEr3dFKWU6hd+H/7tNjvvbCtjXvYQYsKCvd0cpZTqF34f/p/tq6C6qV2f2KWU8it+H/7LtpUSHxHM7LHee36AUkr1N78O/zZbJx/vLuey7KEE69x+pZQf8evE+yK/koY2G/Mn6R08lVL+xa/D/73tR4gOC2Lm6CRvN0UppfqVJeEvIvNFZK+I5IvIvT1sny0im0TEJiLXWVGnpzo67azYfZRLJwwhJMivPwOVUn7I49QTkUBgCXAFkA0sEpHsbsUOATcCf/W0PqusK6yitrlDH9qilPJLVjzGcTqQb4wpBBCR14AFwK5jBYwxRc5tdgvqs8R7O44QERLIHJ3lo5TyQ1aMd6QBxW7LJc51A1an3fDhziPMHadP7FJK+Scrwr+nO6GZM9qRyGIRyRORvIqKCg+b1bu8omoqG9u5Qmf5KKX8lBXhXwJkuC2nA6VnsiNjzFJjTI4xJic5ue+GY97bcYTQoADmjkvpszqUUmogsyL8NwBZIpIpIiHAQiDXgv32Cbvd8MHOI8wem0xkqBWnPJRSavDxOPyNMTbgduADYDfwhjFmp4g8LCLXAIjINBEpAb4FPCsiOz2t90xtLamlrK6VK3SWj1LKj1nS9TXGLAeWd1v3oNvrDTiGg7zu/R1HCA4ULpkwxNtNUUopr/Grq5uMMSzfUcYFo5OIDdfbNyul/Jdfhf/O0nqKq1t0yEcp5ff8Kvw/2+eYPnpptg75KKX8m1+F/7rCKsYPjSYpKtTbTVFKKa/ym/Bvt9nJK6phxih9Tq9SSvlN+G8/XEtLRyczRiV4uylKKeV1fhP+awuqADgvU3v+SinlN+G/rrCa8UOjiY8M8XZTlFLK6/wi/NttdvIOVut4v1JKOflF+O8oraO1w855mTrer5RS4Cfhv7O0HoDJGXFebolSSg0MfhH+u0rriIsIZlhsmLebopRSA4JfhP/O0nomDotBpKfnziillP/x+fDv6LSz50gD2akx3m6KUkoNGD4f/oUVTbTb7EwcFuvtpiil1IDh8+G/s7QOgInDtOevlFLHWBL+IjJfRPaKSL6I3NvD9lARed25fb2IjLSi3tORX95IUICQmRTZX1UqpdSA53H4i0ggsAS4AsgGFolIdrdiNwM1xpgxwBPAY57We7oOVDYxPDGCoECf/5KjlFKnzYpEnA7kG2MKjTHtwGvAgm5lFgAvO1+/CVwi/TT15kBlE6O016+UUl1YEf5pQLHbcolzXY9lnA98rwP6/F4LdrvhQGWTDvkopVQ3VoR/Tz14cwZlEJHFIpInInkVFRUeN6y0roU2m53MpCiP96WUUr7EivAvATLcltOB0t7KiEgQEAtUd9+RMWapMSbHGJOTnJzsccMOVDYBaM9fKaW6sSL8NwBZIpIpIiHAQiC3W5lc4Abn6+uAj40xJ/T8rXYs/Ecna/grpZS7IE93YIyxicjtwAdAIPCiMWaniDwM5BljcoEXgFdEJB9Hj3+hp/WejsKKJiJDAkmO1mf2KqWUO4/DH8AYsxxY3m3dg26vW4FvWVHXV1Fc3UxGQoTe00cppbrx6cnvh2tbSIsL93YzlFJqwPHp8C+ra2WYhr9SSp3AZ8O/qc1GXUuHhr9SSvXAZ8O/rK4FgGFx+gAXpZTqzmfD/3BtK4D2/JVSqgc+G/6ltcd6/hr+SinVnc+Gf1ltCwECQ3SOv1JKncBnw/9wbStDYsL0Vs5KKdUDn03G0toWHfJRSqle+Gz4l9W1kBqrM32UUqonPhn+xhhK61r16l6llOqFT4Z/VVM77Ta7DvsopVQvfDL8j03z1GEfpZTqmU+Hv/b8lVKqZz4a/np1r1JKnYxPhn95QxvBgUJ8RLC3m6KUUgOST4Z/VWMbiZGh+hAXpZTqhUfhLyIJIrJCRPY7f8b3Uu59EakVkWWe1He6qpraSYoO6Y+qlFJqUPK0538vsNIYkwWsdC735HfA9R7WddoqnT1/pZRSPfM0/BcALztfvwx8vadCxpiVQIOHdZ22qsZ2EqO056+UUr3xNPyHGGPKAJw/UzzZmYgsFpE8EcmrqKg4o30YY6hsbCMpSnv+SinVm6BTFRCRj4ChPWy63+rGGGOWAksBcnJyzJnso6m9kzabncRI7fkrpVRvThn+xphLe9smIkdFJNUYUyYiqUC5pa07Ax02O1efPYwJqTHebopSSg1Ypwz/U8gFbgAedf78l8ct8lB8ZAi/XzTV281QSqkBzdMx/0eBeSKyH5jnXEZEckTk+WOFRGQV8HfgEhEpEZHLPaxXKaWUBzzq+RtjqoBLelifB9zitjzLk3qUUkpZyyev8FVKKXVyGv5KKeWHNPyVUsoPafgrpZQf0vBXSik/pOGvlFJ+SIw5o7so9DkRqQAOerCLJKDSouYMFnrM/kGP2T+c6TGPMMYkn6rQgA1/T4lInjEmx9vt6E96zP5Bj9k/9PUx67CPUkr5IQ1/pZTyQ74c/ku93QAv0GP2D3rM/qFPj9lnx/yVUkr1zpd7/koppXrhc+EvIvNFZK+I5ItIbw+UH3RE5EURKReRHW7rEkRkhYjsd/6Md64XEXna+R5sE5FzvNfyMyciGSLyiYjsFpGdIvIT53qfPW4RCRORL0Vkq/OYf+lcnyki653H/LqIhDjXhzqX853bR3qz/Z4QkUAR2Swiy5zLPn3MIlIkIttFZIuI5DnX9dvvtk+Fv4gEAkuAK4BsYJGIZHu3VZb5EzC/27p7gZXGmCxgpXMZHMef5fyzGHimn9poNRtwtzFmAjADuM357+nLx90GXGyMORuYAswXkRnAY8ATzmOuAW52lr8ZqDHGjAGecJYbrH4C7HZb9odjnmuMmeI2pbP/freNMT7zBzgf+MBt+T7gPm+3y8LjGwnscFveC6Q6X6cCe52vnwUW9VRuMP/B8aS4ef5y3EAEsAk4D8fFPkHO9a7fc+AD4Hzn6yBnOfF228/gWNOdYXcxsAwQPzjmIiCp27p++932qZ4/kAYUuy2XONf5qiHGmDIA588U53qfex+cX+2nAuvx8eN2Dn9swfFM7BVAAVBrjLE5i7gfl+uYndvrgMT+bbElngR+Dtidy4n4/jEb4EMR2Sgii53r+u1329Nn+A400sM6f5zO5FPvg4hEAf8A7jTG1Iv0dHiOoj2sG3THbYzpBKaISBzwNjChp2LOn4P+mEXka0C5MWajiFx0bHUPRX3mmJ1mGmNKRSQFWCEie05S1vJj9rWefwmQ4bacDpR6qS394aiIpAI4f5Y71/vM+yAiwTiC/1VjzFvO1T5/3ADGmFrgUxznO+JE5Fhnzf24XMfs3B4LVPdvSz02E7hGRIqA13AM/TyJbx8zxphS589yHB/y0+nH321fC/8NQJZzlkAIsBDI9XKb+lIucIPz9Q04xsSPrf+Bc4bADKDu2FfJwUQcXfwXgN3GmMfdNvnscYtIsrPHj4iEA5fiOAn6CXCds1j3Yz72XlwHfGycg8KDhTHmPmNMujFmJI7/sx8bY76HDx+ziESKSPSx18BlwA7683fb2yc9+uAkypXAPhzjpPd7uz0WHtffgDKgA0cv4GYc45wrgf3OnwnOsoJj1lMBsB3I8Xb7z/CYL8Tx1XYbsMX550pfPm5gMrDZecw7gAed60cBXwL5wN+BUOf6MOdyvnP7KG8fg4fHfxGwzNeP2XlsW51/dh7Lqv783dYrfJVSyg/52rCPUkqp06Dhr5RSfkjDXyml/JCGv1JK+SENf6WU8kMa/kop5Yc0/JVSyg9p+CullB/6//vbA0eZIYdRAAAAAElFTkSuQmCC\n", 197 | "text/plain": [ 198 | "
" 199 | ] 200 | }, 201 | "metadata": {}, 202 | "output_type": "display_data" 203 | } 204 | ], 205 | "source": [ 206 | "# plotting MI using MINE\n", 207 | "plt.plot(plot_x, -plot_y_mine)\n", 208 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 209 | "plt.show()" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 9, 215 | "metadata": {}, 216 | "outputs": [ 217 | { 218 | "data": { 219 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmUVOWd//H3VxA04gIKBFmEGKKiIsQWdYyKC4o4EWdiRvDndg4Jk0xQfxodNZmjDp4kLnHJJLgw6gQNggvxR0eJJmxGVLAbQdaALVu3MHQrICBLN/T398dTZVV3V0PTdbtvd9XndU6de+9zn7rP91ZXf+upp+5i7o6IiOSXg+IOQEREmp+Sv4hIHlLyFxHJQ0r+IiJ5SMlfRCQPKfmLiOQhJX8RkTyk5C8ikoeU/EVE8lDbuAOozzHHHOO9e/eOOwwRkVZl/vz5n7l75/3ViyT5m9lQ4DdAG+AZd3+g1vrHgAsSi18Durj7UfvaZu/evSkuLo4iPBGRvGFmaxtSL+vkb2ZtgHHAEKAMKDKzQndflqzj7rem1b8JGJhtuyIi0nhRjPkPAkrcfZW7VwKTgeH7qD8SmBRBuyIi0khRJP/uQGnaclmirA4zOw7oA8yMoF0REWmkKJK/ZSir7zrRI4BX3X1vxg2ZjTazYjMrrqioiCA0ERHJJIrkXwb0TFvuAayvp+4I9jHk4+7j3b3A3Qs6d97vj9UiItJIUST/IqCvmfUxs3aEBF9Yu5KZnQB0BN6PoE0REclC1snf3fcAY4C3gOXAy+6+1MzGmtkVaVVHApNdtw4TEYldJMf5u/s0YFqtsntqLd8XRVsiIhm5wx/+AFdeCYcfHnc0LZ4u7yAiuWH+fLj+evi3f4s7klZByV9EcsMXX4Tpxx83X5vusHNn87UXISV/Edm3jRth4kSoqoo7khR3uO02eD/t+JGNG8N0+3Z4+mnYuzfUmzMnTPdl4kTYtOnA47j3Xvja1xr/ATBvHjz3XOOemyUlf5HW4t13YenSuuWVlbBlS5h/9FEYOBA2bICysvq3tXkzvPEGPPZYWJ47t/7kfsEFcO21MOkATsyfNg3Ky0PMv/hF5uQ7axYsXx7mf/nLhiXBsjJ46qmQ6B97DP7hH2DPHnj55bAPEF6jH/0IJkyAJ5+Ec8+FKVPq3+a6dWH/jj4akucXVVWF7aabOhVWrQofLmecAb/+Ndx/f1hXUlJz33/0I9ixI1X28MPw05+G+blzYfLk8PqcdRaMGgVbt8IPfgBrG3RZnkhYSz34pqCgwBt7Ybf//NNSlq3fGnFEIg3kDjhYPX0rr65/XW3zi6FtWziuN3z0ERxyCJx5Zmr9nir44AOorobvnAtvv51aZwbnnZdaXrECDj0UevWCeXNh1+5QftppYdt9+sBBB4XE16dP2Iflfw9JCqBLFzjppLoxfvopdOgARx4Ju3eHBFleDl//Ony5HbZtD4m1a1dInr/jDn/7W5g///xU3OefX/9r8cUXsHBhmO/RY98fbgA9e4bnbN0KRxwOlVVQUABt2oT2V68O29m1CxYsCM/p2gWOOBLWrIF27UJ9CB+uH30UnturV3iuWepD7eST4fAO4TmLFof6HTvCqaeG9cl97dcPliUue9a2beoD5thusH5DeJ0OO4x+Z53KvVeeuu/9q4eZzXf3gv3VU89fWh53WL8+TCsrQ4I7UEVFsODDxrW/Y0foOadbvTo8tm1N9eh2fBmSVrLXnbRwIbwzJ8yvWhV+iITQY/xoIcx5N2xnwQIoLa353C1bQuwVFaF3vv1L2JJIYBAS1Zo18Pnn4fWZ/yFU7YG91XV71+5h+3v3hPn//d+wDzt3pBI/hKQG8OWX8MknoSe8axd8uSOV+CHEs2Vz2M727aFs587Q6128OCwvXpx6Tvo3ic8/TyW9XbtSyRDC3zhpyZIw/JJetjdxQYCVK1Nl+0v8yTaTr9vWbaHdzZtT+1JaGl7n9LY2loffDKqqwuvxxZbwXky+Rnv3wv8m3hvpr/fWrTB3Hqz8OHzgHHZYaGPVJ+HvlZR8DU44oeY3i/UbUjGvWxfabGru3iIfp59+uksr9/TT7rNnH/jzHnnEHdyffDJMzVLrtm93LynZ9/NnzQrPg4a3OXWq+z33hPnkc2+5xf1f/sX9r39NlaVvd8SIMH/ddant7N2bqlNWlppftcr9sMNSy716hel556We+9Of1m0n+bj22rplkybVXB44sP7nFhfXLU+PZ3+Ps8+uW3bXXe63355aHjkyTO+/3/3yyzNv56ST3A8/fP/tXXih+3/9V2p5yhT3bt3cu3Z1P+GEVPn554e/Q3l5/dtK388ePdwffdT91lvr1rvttrplQ4ak5s87r+76++5z79Klbvlf/uJ+2WX1/w3da76mZu4HH5xafuyxhr93awGKvQE5tkGJOI6Hkn8OaEgCXrrU/aOPwnxVlfvo0e5nnRWe9/jjqW3s2RPqXHNNWC4vD8vjxrnPmBHmKypSCWhfbX/wgftNN7lXVoblm29O1V+8uO4/cseOdcvSE9O3vx2288QT9Se2ZAK69daaibFdu/BhVvsD5p/+qeby8ce7n3pqzbJTTnHv1Mn9z3+uP/HVfpx8cmr+hRdS83fdlZo/9lj373+/5vMefjg1f9NN+27jnXdqJrp//Vf3fv1SyzfckJrv1q3mcwcMCI/27d0POig8wP2008L0wQfdd+8OH9YPPeReXZ36u1ZUhNeydjzXX1+37MgjM8edvtyuXc3l3//evWfPMH/VVe6vvRbes1dfXbNet27h/TplSs3y9Njc3deudR88OCyPHOm+cqX7Sy+FD5Mbbzygf7V0Sv7SPH72M/dBg+qW79hR841eW3V16DVB+Ifas6du7/TEE1Pzr72W+iYA7mPG1EzU8+ZlTtLDhoVe1Mknu597boirffuw7oknQixdu6bq33hj/Umt9gcLpP7xzeomi9qPs88O7T33XOo5meode2z4BnHttaFnmyy/5Za6dS+5xH358tRyUVGIs3ZCSj6qqsIHzXe+475lSyg74ojwjervf3ffvDn1N3roodTzFi0K01Gj3Hftqn8fv/1t9507a/6tiovdN20Kvf6pU8O2S0rCB6i7+5w5qbrf+577s8+G+Q4dwod8376p9YWF+39PPvJI+PY3fbr7lVe6T5yYel0zfTgkH9XVNXvxs2aFbS1b5r5mTVif7HzcfHOqvaIi9699LewfhG8+7uG1vv/+UHbOOaHs1792nzatZrwLFrhv25ZaHjIkfItrJCV/id6nn7q/+GJqOT0JrF4d/gmWLQsJa9681Lovvkj1zt3dN2yomVjA/Z//OfWP1dBHshcG7m3ahCS2v+dccklq/owzUr3f++7bf/J2Dx9Sd98deuZFReGfNj1hpG8fQm906VL3t99237gxbCOZSO+8Mwz1PPZYSMiPPhrKjz469Vq9915qWxMnhiEmSA2N/fKXIXGnJzD3VEI97TT3FSvC/JVX1v2bTp6872G0N98MPfFdu9y3bg0fSu6h1/rAAzX3dfz41PP27k3tb0Mk/w4/+Unq9bn33rCue/ewfOKJ4YPlQCXfi8nXNdnbnjEjfDAdemjq77t9e/jwOe64zG396U+h7n/8R83yvXvDB8Q996S+USZ98kn4oG2oJUvCMGEjKflL9C6+OLxlksliX8MNl16amj/99DD9/e/dn3/e/cwz951kL7ggc3lyuOGhh1IJIf0DYPHimt8W6nsMGeI+dmzNssJC99/8xr/qpdWX/DPZu9f9d79zv+OO8I+/d6/7L34REn59Xn3V/csva5atXx/a6dcvVZbsnUNIIpWV4UPYPSTJZKJ55pkwbJCUHAOfNCksT59es3cZhdmzQxtXXeX+P/+TGpprjJ07w5j7pk1h+e23U/uWHPIpLW3ctpOvxcknh+UdO8KYfNKWLe6ff96wbVVXh2+hUb+WEVLyl4ZbvTr8g1RVhQSzY0fN9Q8+6D5zpnv//uEt8+STofz2293btq35tbyhj1tvdX/jjTB/0UVh2qlT6HW516z7+OMhubmHpLB7dxifHT48fGX+zW9SP9Zu3uxeUJC5zeQwU2FhqneZfKxYEZ6/bFlIMskfFV97Lfz4OHZsk/8Z3D3sS+2e+G9/G4as0se3W4K9e8MwxoH08BujpKTmt4rGeOKJrHrTrYmSv9Q1aFDoXVVXpxJ8VZV/9ZX6d7+rmRCnTk31RiGMwYJ7586hbkFBGOJxD1/R95fwk0e4QEj8STt3hq/M6dKft3z5ge3n5s0hWTzxRBi3LSwMPe3q6vA1P2nUqPBD6tNP102sEyaEtpcuPbC2RWKm5J9PVq8OPfOJE1Njsuk2bXL/7ndTyfSOO8KPnvfeGxJjfcn6ootCAk0vO+kk929+M7V8++2pdpJjq8cdF4Z4rrsu9MwhHAGxaVNItL171/xhMZMXXgg/1o4aFT6gmkJ19b5708khCJFWRMk/X6T/IAjh6JukZIJNP5QRwlEmyV58+iF4yXXLl4cx9f79w4+i6euTHzDJ5eefrxnPQw+5v/tuannv3nAYoXrQIs2ioclfZ/i2Zrt2hWubpHv88XDNlDlzwunlf/xjzVP+IaTtX/winJ6ePOMw6bDD4MQT4eqrYdGicLapJW7T/LOfwTXXhEsA9OkTyk45pebz77ijZkwHHQS/+lU4rV1EWoxIkr+ZDTWzFWZWYmZ31VPnX8xsmZktNbMXo2g37733Xt2yHTvgwgvDxawA/v3fU6emJx1yCNx4Y7hGCcAPfxhugNGxIzz7bCjr3z9Mb7opdfGv9GvKFBaG553auOuPiEi8sr6Tl5m1AcYBQwg3cy8ys0J3X5ZWpy9wN3COu282sy7ZtiuEXn26U04J10ZJ98knNZdvuw2OPRaOOAJuvhnefBPuuSdc4CrdNdeEbwCDBoVvCv37w+DBNdsaPz6yXRGR5hVFz38QUOLuq9y9EpgMDK9V54fAOHffDODu5Uh2brsNxo0Ll4N9+eVU+TvvpC4dm8ndd6fW33Yb/OUvdRM/wMEHh56+WRi6ueCC1PCPiLR6UST/7kD6pQnLEmXpvgV8y8zeNbO5ZjY0gnbzV2UlPPMMHH98uE54166pdd/5Tuoa4wC33houM5vUqVPzxSkiLVYUyT9Td9BrLbcF+gKDgZHAM2Z2VJ0NmY02s2IzK65I3lRBwmVkr7kGZs4M10vv3Ru2bQs3k+jYMXWzak+87IceGqYdO4abe+zeHX4AHjs29OJFJO9lPeZP6On3TFvuAdS+GHUZMNfdq4DVZraC8GFQlF7J3ccD4yHczCWC2HLDe++FuyhNmgTdu4drzZ95JlxySVjfr1+4acevfpV6Tmlp+GE36ZxzwkNEhGiSfxHQ18z6AJ8CI4BratX5f4Qe/+/N7BjCMNCqCNrOD5Mnp+Y//RROPz3cuzQ5Bt++fd3DOTON44uIJGSd/N19j5mNAd4C2gDPuftSMxtLONmgMLHuEjNbBuwF7nD3z7NtOy+UloZ7m152GQwYEJL8uHH68VVEspKT9/DNCdXV8Mgj4QfdHTvC7fGSJ1aJiNSjoffwjWLYR5rCzJnhBC0IPX8lfhGJkJJ/S7R8OVx6aZh//nm49tp44xGRnKPk3xJNnRqGfV5/HS6/PO5oRCQH6aDvluhvfwuHbyrxi0gTUfJvadxh3ry6V+sUEYmQkn9L89lnsGkTnHxy3JGISA7TmH9L8oc/hAutAZxwQryxiEhOU/JvSa67LjV/4onxxSEiOU/DPi1V8kYrIiJNQMm/pbjootT89OnQpk18sYhIzlPybwk2bAhn9AJMnFjzg0BEpAko+bcECxak5tNvzCIi0kSU/FuC998P05Ejdc19EWkWOtonblu2wG9/G67l8+KLcUcjInlCPf+4ffghfPFFuJm6iEgzUfKP086dMH58mD/ppHhjEZG8EknyN7OhZrbCzErM7K4M6280swozW5h4/CCKdlu9Bx6Al14K8927xxuLiOSVrMf8zawNMA4YQrhRe5GZFbr7slpVX3L3Mdm2l1M2bkzNH6QvYSLSfKLIOIOAEndf5e6VwGRgeATbzX3btoXpww/HG4eI5J0okn93oDRtuSxRVtv3zGyRmb1qZj0jaLf1W7sWBg+G22+POxIRyTNRJH/LUFb7rvB/Anq7e39gOjAh44bMRptZsZkVV1RURBBaC7duna7hIyKxiCL5lwHpPfkewPr0Cu7+ubvvTiz+N3B6pg25+3h3L3D3gs6dO0cQWgv25ptQVgbHHx93JCKSh6JI/kVAXzPrY2btgBFAYXoFM+uWtngFsDyCdluvuXNh2LBwzf4x+g1cRJpf1kf7uPseMxsDvAW0AZ5z96VmNhYodvdC4GYzuwLYA2wCbsy23Vbtd7+Dww+Hd9+FTp3ijkZE8pC51x6ebxkKCgq8uLg47jCaRteuMHQoTMj404eISKOZ2Xx3L9hfPR1c3tzKy8NjwIC4IxGRPKYLuzWntWthypQwf+qp8cYiInlNyb85DRwImzeH+f79441FRPKahn2ai3sq8bdtC126xBuPiOQ1Jf/msnZtan7atPjiEBFBwz7N54MPwrS4GE7PeI6biEizUc+/uXzwAbRvrx96RaRFUPJvLh98EH7wbdcu7khERJT8m8WePTB/PgwaFHckIiKAkn/zWL4cduyAM86IOxIREUDJv3nMnh2m6vmLSAuh5N/UPvsM7rwTCgrgm9+MOxoREUDJv+nNnAk7d4Yreeo+vSLSQigbNaXFi+Hqq8FMx/aLSIui5N+UZs4M09tvD5d0EBFpIZT8m9Lf/x5u1vLgg3FHIiJSg5J/U3n2WXjqKTj22DDsIyLSgkSS/M1sqJmtMLMSM7trH/WuMjM3s/3eZabVGzs2TLt2jTcOEZEMsk7+ZtYGGAdcBvQDRppZvwz1DgduBuZl22aLV1kJFRVw0knhG4CISAsTRc9/EFDi7qvcvRKYDAzPUO9+4CFgVwRttmxLloTDO++7D447Lu5oRETqiCL5dwdK05bLEmVfMbOBQE93f31fGzKz0WZWbGbFFRUVEYQWkyVLwlR36xKRFiqK5J/p10z/aqXZQcBjwE/3tyF3H+/uBe5e0Llz5whCi8nSpeHqnTqjV0RaqCiSfxnQM225B7A+bflw4BRgtpmtAc4CCnP6R99Fi+DEE3Vsv4i0WFEk/yKgr5n1MbN2wAigMLnS3b9w92Pcvbe79wbmAle4e3EEbbc8lZUwZw6cfXbckYiI1Cvr5O/ue4AxwFvAcuBld19qZmPN7Ipst9/qzJkD27fDZZfFHYmISL0iGZdw92nAtFpl99RTd3AUbbZYjz4azuq96KK4IxERqZfO8I3Sli3wxhvw4x9Dhw5xRyMiUi8l/ygVFYXp4MGxhiEisj9K/lGaMydMC3L3QCYRyQ1K/lFZuzZcvfOSS+Coo+KORkRkn5T8ozJtGuzeDb/9bdyRiIjsl5J/VKZPh169oG/fuCMREdkvJf+ozJ0L552na/eLSKug5B+F8nJYvx4GDow7EhGRBlHyj8LChWE6YEC8cYiINJCSf7aqq+HnP4dDD1XPX0RaDV12MltLlkBxMYwbBx07xh2NiEiDqOefrXfeCdNhw+KNQ0TkACj5Z+tvf4MePXS7RhFpVZT8s+Eeev46xFNEWhkl/2yUlMCGDSH5i4i0Ikr+2Zg0KUwvuSTeOEREDlAkyd/MhprZCjMrMbO7Mqz/kZktNrOFZjbHzPpF0W6svvwSnnoKhgyBPn3ijkZE5IBkfainmbUBxgFDCDdzLzKzQndfllbtRXd/KlH/CuBRYGi2bcfquefCkM8rr8QdiYjIAYui5z8IKHH3Ve5eCUwGhqdXcPetaYuHAR5Bu/GaPRu+8Q0455y4IxEROWBRnOTVHShNWy4Dzqxdycx+AtwGtAMuzLQhMxsNjAbo1atXBKE1EXd4/3244IK4IxERaZQoev6ZjnGs07N393HufjxwJ/AfmTbk7uPdvcDdCzp37hxBaE2ktDQM+Zx9dtyRiIg0ShTJvwzombbcA1i/j/qTgSsjaDc+778fpkr+ItJKRZH8i4C+ZtbHzNoBI4DC9Apmln6Hk8uBjyNoNz5vvBEu5Na/f9yRiIg0StZj/u6+x8zGAG8BbYDn3H2pmY0Fit29EBhjZhcDVcBm4IZs243N7Nnwwgtw9dVw8MFxRyMi0iiRXNXT3acB02qV3ZM2f0sU7bQIM2eGSzmMHx93JCIijaYzfA/UvHlhuOeII+KORESk0ZT8D8SuXeHHXv3QKyKtnJL/gXjzTdi2Da5s3QcriYgo+R+It94Kwz0XZjxHTUSk1VDyPxBFRVBQoKN8RKTVU/JvqF27YNEiOOOMuCMREcmakn9DPfAAVFXp2v0ikhOU/BtqwoRwk3aN94tIDlDyb4g1a8JjaOu+BYGISJKSf0NMnx6mgwfHGoaISFSU/BvilVfg+OPhlFPijkREJBJK/vtTWQmzZoUTuyzTrQtERFofJf/9WbEiHOVz+ulxRyIiEhkl//1ZtChMde1+EckhSv77sm0b3HkntG8P3/pW3NGIiEQmkuv556yZM+HTT+GRR3RJBxHJKZH0/M1sqJmtMLMSM7srw/rbzGyZmS0ysxlmdlwU7Ta5uXND0v/xj+OOREQkUlknfzNrA4wDLgP6ASPNrF+taguAAnfvD7wKPJRtu03OPRzlc9pp4X69IiI5JIqe/yCgxN1XuXslMBkYnl7B3We5+47E4lygRwTtNq0ZM8Jdu66/Pu5IREQiF0Xy7w6Upi2XJcrqMwr4cwTtNq0JE6BjRxg9Ou5IREQiF8UPvpnOfPKMFc2uBQqA8+tZPxoYDdCrV68IQmukqiqYOhW+//1wpI+ISI6JoudfBvRMW+4BrK9dycwuBn4OXOHuuzNtyN3Hu3uBuxd07tw5gtAaqbg4HOY5bFh8MYiINKEokn8R0NfM+phZO2AEUJhewcwGAk8TEn95BG02rbffDtPzzos3DhGRJpJ18nf3PcAY4C1gOfCyuy81s7FmdkWi2sNAB+AVM1toZoX1bC5+1dXw/PPhcg5xfvsQEWlCkZzk5e7TgGm1yu5Jm784inaaxcyZsHw5/OEPcUciItJkdHmH2l56CTp0gO99L+5IRESajJJ/Ond44w24/HI45JC4oxERaTJK/unWroUNG+Dcc+OORESkSSn5p3v33TA9++x44xARaWJK/knu8MQT0KOHrt0vIjlPl3ROev55eO89eOopaKuXRURym3r+Sf/936HH/8Mfxh2JiEiTU/IH2L07XNJhyBA4SC+JiOQ+ZTqAoqLwAXDOOXFHIiLSLJT8IZzY1b49XHhh3JGIiDQLJf/KSpg8GYYPhyOPjDsaEZFmoeT/5pvw2Wdw3XVxRyIi0myU/F96CY4+Gi69NO5IRESaTX4n/9274fXX4cor4eCD445GRKTZ5Hfynz4dtm7VFTxFJO/kd/KfMiX8yHvRRXFHIiLSrCJJ/mY21MxWmFmJmd2VYf15Zvahme0xs6uiaDNryZu0f/e70K5d3NGIiDSrrJO/mbUBxgGXAf2AkWbWr1a1dcCNwIvZtheZ2bNh0yYN+YhIXoriCmaDgBJ3XwVgZpOB4cCyZAV3X5NYVx1Be9GYMgUOO0xH+YhIXopi2Kc7UJq2XJYoa7n27oXXXoNhw+DQQ+OORkSk2UWR/C1DmTdqQ2ajzazYzIorKiqyDGsf3n0Xysvhqpbx84OISHOLIvmXAT3TlnsA6xuzIXcf7+4F7l7QuXPnCEKrx5Qp4R69w4Y1XRsiIi1YFMm/COhrZn3MrB0wAiiMYLtNo7oa/vjHMNbfoUPc0YiIxCLr5O/ue4AxwFvAcuBld19qZmPN7AoAMzvDzMqA7wNPm9nSbNtttKIiKCvTUT4iktciuV+hu08DptUquydtvogwHBS/KVPCpRy++924IxERiU1+neHrDq++Gs7oPeqouKMREYlNfiX/hQth9WoN+YhI3suv5P/nP4fpFVfEG4eISMzyK/nPng2nngpdusQdiYhIrPIn+VdWhpO7Bg+OOxIRkdjlT/IvLoYdO5T8RUTIp+Q/a1aYnn9+vHGIiLQA+ZP8Z8+G/v3D/XpFRPJcfiR/jfeLiNSQH8n/ww9h504477y4IxERaRHyI/kvWBCmZ5wRbxwiIi1E/iT/Tp2gZ8/91xURyQP5k/wHDgTLdN8ZEZH8k/vJv6oKFi+GAQPijkREpMXI/eS/YgXs3h16/iIiAuRD8k/+2KvkLyLylUiSv5kNNbMVZlZiZndlWN/ezF5KrJ9nZr2jaLdBli+Htm3hW99qtiZFRFq6rJO/mbUBxgGXAf2AkWbWr1a1UcBmd/8m8BjwYLbtNtjKlXD88eEDQEREgGh6/oOAEndf5e6VwGRgeK06w4EJiflXgYvMmunQm5Ur1esXEakliuTfHShNWy5LlGWsk7jh+xdA019kp7oaPv5YyV9EpJYokn+mHrw3og5mNtrMis2suKKiIvvISkth1y4lfxGRWqJI/mVA+qmzPYD19dUxs7bAkcCm2hty9/HuXuDuBZ07d84+spUrw1TJX0SkhiiSfxHQ18z6mFk7YARQWKtOIXBDYv4qYKa71+n5Ry6Z/E84ocmbEhFpTbI+BMbd95jZGOAtoA3wnLsvNbOxQLG7FwLPAi+YWQmhxz8i23YbZOVK6NABvv71ZmlORKS1iOT4R3efBkyrVXZP2vwu4PtRtHVAVq2Cb3xD1/QREaklt8/wXbcOevWKOwoRkRYnt5N/aamSv4hIBrmb/Ldtg82blfxFRDLI3eRfmjjvTDdwERGpI/eTv3r+IiJ15G7yX7cuTJX8RUTqyN3kX1oKBx0Exx4bdyQiIi1O7ib/detC4telnEVE6sjt5K8hHxGRjHI3+ZeW6kgfEZF65Gbyd9cJXiIi+5Cbyb+iAnbvVvIXEalHbib/5GGeGvYREckot5O/ev4iIhnlZvLXpR1ERPYpN5P/hg1w8MFwdNPfI15EpDXKzeRfXg5duugmLiIi9cgq+ZtZJzP7q5l9nJh2rKfem2a2xcxez6a9Bisvh65dm6UpEZHWKNue/13ADHfvC8xILGfyMHBdlm013MaNoecvIiIDiRcUAAAFE0lEQVQZZZv8hwMTEvMTgCszVXL3GcC2LNtquOSwj4iIZJRt8u/q7hsAEtOsMq6ZjTazYjMrrqioaNxG3JX8RUT2Y7+XvDSz6cDXM6z6edTBuPt4YDxAQUGBN2oj27fDrl1K/iIi+7Df5O/uF9e3zsw2mlk3d99gZt2A8kija4zKShgxAk47Le5IRERarGwvdl8I3AA8kJhOzTqibB19NEyaFHcUIiItWrZj/g8AQ8zsY2BIYhkzKzCzZ5KVzOwd4BXgIjMrM7NLs2xXRESykFXP390/By7KUF4M/CBt+dxs2hERkWjl5hm+IiKyT0r+IiJ5SMlfRCQPKfmLiOQhJX8RkTyk5C8ikofMvXFXUWhqZlYBrM1iE8cAn0UUTmuhfc4P2uf80Nh9Ps7dO++vUotN/tkys2J3L4g7juakfc4P2uf80NT7rGEfEZE8pOQvIpKHcjn5j487gBhon/OD9jk/NOk+5+yYv4iI1C+Xe/4iIlKPnEv+ZjbUzFaYWYmZ1XdD+VbHzJ4zs3IzW5JW1snM/mpmHyemHRPlZmb/lXgNFpnZt+OLvPHMrKeZzTKz5Wa21MxuSZTn7H6b2SFm9oGZfZTY5/9MlPcxs3mJfX7JzNolytsnlksS63vHGX82zKyNmS0ws9cTyzm9z2a2xswWm9lCMytOlDXbezunkr+ZtQHGAZcB/YCRZtYv3qgi83tgaK2yu4AZ7t4XmJFYhrD/fROP0cCTzRRj1PYAP3X3k4CzgJ8k/p65vN+7gQvd/TRgADDUzM4CHgQeS+zzZmBUov4oYLO7fxN4LFGvtboFWJ62nA/7fIG7D0g7pLP53tvunjMP4GzgrbTlu4G7444rwv3rDSxJW14BdEvMdwNWJOafBkZmqteaH4Q7xQ3Jl/0GvgZ8CJxJONmnbaL8q/c58BZwdmK+baKexR17I/a1RyLZXQi8Dlge7PMa4JhaZc323s6pnj/QHShNWy5LlOWqru6+ASAxTd61Pudeh8RX+4HAPHJ8vxPDHwsJ98T+K/AJsMXd9ySqpO/XV/ucWP8FcHTzRhyJx4F/B6oTy0eT+/vswF/MbL6ZjU6UNdt7O9t7+LY0lqEsHw9nyqnXwcw6AFOA/+vuW80y7V6omqGs1e23u+8FBpjZUcBrwEmZqiWmrX6fzewfgXJ3n29mg5PFGarmzD4nnOPu682sC/BXM/v7PupGvs+51vMvA3qmLfcA1scUS3PYaGbdABLT8kR5zrwOZnYwIfFPdPc/Jopzfr8B3H0LMJvwe8dRZpbsrKXv11f7nFh/JLCpeSPN2jnAFWa2BphMGPp5nNzeZ9x9fWJaTviQH0QzvrdzLfkXAX0TRwm0A0YAhTHH1JQKgRsS8zcQxsST5dcnjhA4C/gi+VWyNbHQxX8WWO7uj6atytn9NrPOiR4/ZnYocDHhR9BZwFWJarX3OflaXAXM9MSgcGvh7ne7ew937034n53p7v+HHN5nMzvMzA5PzgOXAEtozvd23D96NMGPKMOAlYRx0p/HHU+E+zUJ2ABUEXoBowjjnDOAjxPTTom6Rjjq6RNgMVAQd/yN3OfvEL7aLgIWJh7Dcnm/gf7AgsQ+LwHuSZR/A/gAKAFeAdonyg9JLJck1n8j7n3Icv8HA6/n+j4n9u2jxGNpMlc153tbZ/iKiOShXBv2ERGRBlDyFxHJQ0r+IiJ5SMlfRCQPKfmLiOQhJX8RkTyk5C8ikoeU/EVE8tD/B67qjiP8mJVfAAAAAElFTkSuQmCC\n", 220 | "text/plain": [ 221 | "
" 222 | ] 223 | }, 224 | "metadata": {}, 225 | "output_type": "display_data" 226 | } 227 | ], 228 | "source": [ 229 | "# plotting MI using NCE\n", 230 | "plt.plot(plot_x, -plot_y_nce, c = \"red\")\n", 231 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 232 | "plt.show()" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 10, 238 | "metadata": {}, 239 | "outputs": [ 240 | { 241 | "data": { 242 | "text/plain": [ 243 | "True" 244 | ] 245 | }, 246 | "execution_count": 10, 247 | "metadata": {}, 248 | "output_type": "execute_result" 249 | } 250 | ], 251 | "source": [ 252 | "np.allclose(plot_y_mine,plot_y_nce, atol=1e-04)" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": 11, 258 | "metadata": {}, 259 | "outputs": [ 260 | { 261 | "data": { 262 | "text/plain": [ 263 | "False" 264 | ] 265 | }, 266 | "execution_count": 11, 267 | "metadata": {}, 268 | "output_type": "execute_result" 269 | } 270 | ], 271 | "source": [ 272 | "np.allclose(plot_y_mine,plot_y_nce, atol=1e-05)" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "We see that the mutual information obtained from both the methods is almost equal and close to 1e-4 tolerance! Therefore, both the methods are consistent." 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [] 288 | } 289 | ], 290 | "metadata": { 291 | "accelerator": "GPU", 292 | "colab": { 293 | "name": "pytorch_mine.ipynb", 294 | "provenance": [], 295 | "version": "0.3.2" 296 | }, 297 | "kernelspec": { 298 | "display_name": "Python 3", 299 | "language": "python", 300 | "name": "python3" 301 | }, 302 | "language_info": { 303 | "codemirror_mode": { 304 | "name": "ipython", 305 | "version": 3 306 | }, 307 | "file_extension": ".py", 308 | "mimetype": "text/x-python", 309 | "name": "python", 310 | "nbconvert_exporter": "python", 311 | "pygments_lexer": "ipython3", 312 | "version": "3.6.5" 313 | } 314 | }, 315 | "nbformat": 4, 316 | "nbformat_minor": 1 317 | } 318 | -------------------------------------------------------------------------------- /notebooks/pytorch_mine_vs_nwj.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Using the fixed constant `e`, we get $I_{NWJ}$ from Nguyen et al. (2010) (also known as MINE-f from Belghazi et al. (2018))." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import torch\n", 17 | "from torch.autograd import Variable\n", 18 | "import torch.nn as nn\n", 19 | "import torch.nn.functional as F\n", 20 | "from torch.utils.data import Dataset, DataLoader\n", 21 | "import numpy as np\n", 22 | "import matplotlib.pyplot as plt\n", 23 | "from tqdm import tqdm" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 2, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "# data\n", 33 | "np.random.seed(1234)\n", 34 | "var = 0.2\n", 35 | "def func(x):\n", 36 | " return x\n", 37 | "\n", 38 | "def gen_x():\n", 39 | " return np.sign(np.random.normal(0.,1.,[data_size,1]))\n", 40 | "\n", 41 | "def gen_y(x):\n", 42 | " return func(x)+np.random.normal(0.,np.sqrt(var),[data_size,1])" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "np.random.seed(1234)\n", 52 | "data_size = 1000000\n", 53 | "x = gen_x()\n", 54 | "y = gen_y(x)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 4, 60 | "metadata": {}, 61 | "outputs": [ 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "Mutual information calculated through traditional method is: 0.6584537102332939\n" 67 | ] 68 | } 69 | ], 70 | "source": [ 71 | "x = gen_x()\n", 72 | "y = gen_y(x)\n", 73 | "p_y_x = np.exp(-(y - x)**2 / (2 * var))\n", 74 | "p_y_x_minus = np.exp(-(y + 1)**2 / (2 * var))\n", 75 | "p_y_x_plus = np.exp(-(y - 1)**2 / (2 * var))\n", 76 | "mi = np.average(np.log(p_y_x / (0.5 * p_y_x_minus + 0.5 * p_y_x_plus)))\n", 77 | "#mi = mutual_information(x, y)\n", 78 | "print(\"Mutual information calculated through traditional method is:\", mi)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 5, 84 | "metadata": {}, 85 | "outputs": [ 86 | { 87 | "name": "stderr", 88 | "output_type": "stream", 89 | "text": [ 90 | "100%|██████████| 500/500 [00:07<00:00, 67.17it/s]\n" 91 | ] 92 | } 93 | ], 94 | "source": [ 95 | "np.random.seed(1234)\n", 96 | "H=10\n", 97 | "n_epoch = 500\n", 98 | "data_size = 20000\n", 99 | "\n", 100 | "class Net(nn.Module):\n", 101 | " def __init__(self):\n", 102 | " super(Net, self).__init__()\n", 103 | " self.fc1 = nn.Linear(1, H)\n", 104 | " self.fc2 = nn.Linear(1, H)\n", 105 | " self.fc3 = nn.Linear(H, 1)\n", 106 | "\n", 107 | " def forward(self, x, y):\n", 108 | " h1 = F.relu(self.fc1(x)+self.fc2(y))\n", 109 | " h2 = self.fc3(h1)\n", 110 | " return h2 \n", 111 | "\n", 112 | "model = Net()\n", 113 | "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n", 114 | "plot_loss_mine = []\n", 115 | "plot_loss_nwj = []\n", 116 | "for epoch in tqdm(range(n_epoch)):\n", 117 | " x_sample=gen_x()\n", 118 | " y_sample=gen_y(x_sample)\n", 119 | " y_shuffle=np.random.permutation(y_sample)\n", 120 | " \n", 121 | " x_sample = Variable(torch.from_numpy(x_sample).type(torch.FloatTensor), requires_grad = True)\n", 122 | " y_sample = Variable(torch.from_numpy(y_sample).type(torch.FloatTensor), requires_grad = True)\n", 123 | " y_shuffle = Variable(torch.from_numpy(y_shuffle).type(torch.FloatTensor), requires_grad = True) \n", 124 | " \n", 125 | " pred_xy = model(x_sample, y_sample)\n", 126 | " pred_x_y = model(x_sample, y_shuffle)\n", 127 | "\n", 128 | " # Nguyen Wainwright Jordan (NWJ) loss\n", 129 | " ret_nwj = torch.mean(pred_xy) - (1/2.71828) * torch.mean(torch.exp(pred_x_y))\n", 130 | " # Mutual Information Neural Estimation (MINE) loss\n", 131 | " ret_mine = torch.mean(pred_xy) - torch.log(torch.mean(torch.exp(pred_x_y)))\n", 132 | " loss_mine = - ret_mine # maximize\n", 133 | " loss_nwj = - ret_nwj # maximize\n", 134 | " plot_loss_mine.append(loss_mine.data.numpy())\n", 135 | " plot_loss_nwj.append(loss_nwj.data.numpy())\n", 136 | " model.zero_grad()\n", 137 | " loss_mine.backward(retain_graph=True)\n", 138 | " loss_nwj.backward(retain_graph=True)\n", 139 | " optimizer.step()" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "plot_x = np.arange(len(plot_loss_mine))\n", 149 | "plot_y_mine = np.array(plot_loss_mine).reshape(-1,)\n", 150 | "plot_y_nwj = np.array(plot_loss_nwj).reshape(-1,)" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 7, 156 | "metadata": {}, 157 | "outputs": [ 158 | { 159 | "data": { 160 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VOW9x/HPL/u+h7AkkLApYREkgor7iru9tQpaq62t7a3aVntt9WrV0mqtt9VWS1utWq17a1tFRQFxqaIgQWVfEkIgIfu+TiYz89w/ZjJMJgMZIGGYmd/79eLFnDPPzPzOJPnOM895zjlijEEppVRoiQh0AUoppYaehrtSSoUgDXellApBGu5KKRWCNNyVUioEabgrpVQI0nBXSqkQpOGulFIhSMNdKaVCUFSgXjgrK8vk5+cH6uWVUioorVu3rsEYkz1YO7/CXUTmA78HIoEnjTEPet3/CHCmazEBGGGMSTvQc+bn51NcXOzPyyullHIRkd3+tBs03EUkElgMnAtUAmtFZIkxZktfG2PMrR7tbwFmHXTFSimlhow/Y+5zgFJjTJkxxgq8DFx2gPYLgZeGojillFKHxp9wHwNUeCxXutYNICLjgALgvcMvTSml1KHyJ9zFx7r9nSd4AfCqMcbu84lEbhSRYhEprq+v97dGpZRSB8mfcK8E8jyWc4Gq/bRdwAGGZIwxTxhjiowxRdnZg+7sVUopdYj8Cfe1wCQRKRCRGJwBvsS7kYgcA6QDnw5tiUoppQ7WoOFujLEBNwPLgK3A340xm0VkkYhc6tF0IfCy0Us7KaVUwPk1z90YsxRY6rXuHq/l+4auLKWU6q+m1cL6yhbOnzoy0KUEBT39gFIqKFz71Bq++9w6LL0+52soLxruSqmgUFrfAUBdW0+AKwkOGu5KhbldDZ2sr2gJdBn91LRaeOidbfTaHe510RHOuKps6XL33h0Og91x4N18xhh6bIfW23/2k3KWba45pMcCtHb3YvPYhiNJw12pAOqbf9DSZWXx+6XUtVkGtKlrt2C1OQNi8l1v88DSrbR0WXEMEmrvbKqhoqnrgG1qWi2c+ZsPuGzxKvdrHEzdH5c0UN3afcC2vXYHj64sod3SO+jztnRZ6eyx8et3tvHHD3bynx3O42HaLb1ERjgPubn6L2s49mfvAHDLS18w94GVB3zOpz7exTF3v0Nr1+CvD9DZY+OZVbuw9Nq5d8lmvvvcugE1en+gdFvteM8lcTgMx/18OT/950aMMYP+vIaahrs64iqauvh8T/MhP94Yw2tf7KXbOnRjrx09tgF/nN1WO+UNnQPaflnRQmePDQCrzdHvcZZe+wHHhI0x7vZVLd0U3LmUtzdWc/drm/i/Zdt56bOKfu131LYz5/6V/Gb5dlq7e7HaHTzxnzJmLlrBK8X72jZ3Wvn2s2upabXQ0NHD5Lve5nvPr+OmFz/nsZUlzLhvGQ6H4blPy90fIE2dVn75lvsUUWzc2+qz5j2N+z4gjDE8+VEZU+9dRkuXla8/tYaTfvUeW6vb+vVQt9W0kX/HW6yvaOHNDVU8vGIHj71Xut/3BcDuMMxctIIrH/8UcR06+dzq3Zz60HtMv2853V7va0ePjbc2VtPQ0cMnpQ3sqG33+bzLN9cCcNyi5TR1Wum1O1j8fmm/sP9gex1n//YDdtZ38MOXv+C+N7bw5EdlA57LZncw54GVLPzLavfPsb69hyn3vMNfV5WzpaqNe1/fxE9eXc/qskYA/vl5JY+9V8rpv3n/iAa8BGrmYlFRkTmks0K+fQfUbBz6gtRBsdod7KzvYEJ2EjGRB9dHWL3L+Us/e1w6GytbmZyTTFKs/2efbrP0sqW6jZyUOAoyE/16jMEgroOtyxo6SImPJisxFnB+dd5a08bYjAQAclLiiBRhV0MnDR09zB6XToQrbWwOB8W7m8lIiGFiThKf7WpidGo8eRnx7G3uprKlm+jICMZnJbK9tp0ZuakkRO/btq3VbdgchvzMBLp67exq6CQ9IQZLr53uXjsZiTFkJMaQmRhDZ4+dTVXOwE2OjWJcZqJ7GSArKZbx2YlEINS0WShv7CQ7KZb0xBh30EVFCDZXoEzOSWZHbTspcdEUjkpha00brd37Ai4vPYFRaXFEeByUXttuYVdDJ1NHpRAfE8mW6ja6XB+qk0YkUVLX4W6bmx5PbloCPXY7W6ra6LE5GJkSR2xUBLubukiMjWLKyGSiIgb+vjiM4bPyJr9+ln3GZSSw2+ubyYkFmYDz205Nm4XpY1LZWddBQ6cVgBHJsdS1O8fs+94Hg2Hd7mZsDkNGYgzNnVYMECGCw5WPJ+Rn0G7pJToywv0hmJMcS0FWEtVt3exu7CJCIDU+hpaugY+PFMFuDLFREaQnxJA/9US4oN/Jdf0mIuuMMUWDtQvY+dxVcGjutpIQE0lsZGS/9bVtFlq7e6lts5CXnrDfx3f32omLjkBw/nL39XgBOiw2rHYHu5s6mToqFYDq1m4SYqJIjY/e7/Ntq3EGV4+fsybsxvDFnmbGpMWTEh9NXXsPde09lOL8cGp2/eHvcQWFzWHITYunobMHuzF0We3uDx9Lr7N32mbppavH+fpVrd0kx0VR2eIcnui1O9juCte27l4SoqMwGLbXtNPqGprYXN3mri8qQrC4xoSbOq00dVqxZiS4QxRARNhZvy9IARo6emjqtHJcbio2h7Ou+o4emrus7jY2j57i3uZud+17W7r6BXt0hNDQ0cPeli5iIiPJy4gnIzGGatc2tXT30mNz0GW1kxgTRafV1u/xAJ09drqsNjZ4fANo6rQS5RpO6eyxUby7maTYKMZnJRIdGYHFZscYiIr0dZYTp9T4aHptDrq8ft7ewQ7OD4kIERo7rHRZnR+YVruDxJhIuqx2d7D3vQ89djubKtvc71NLV6/73CoOj47vtpo22i02RqbEAZAWH01tew924/w5ONtDc5eV9ATnB7Tnz8vueq4em4OaNgv5+93aoRN84X6In3ZqcH/+cCe56fFcPGM0AHVtFuY8sJJpY1J485ZT+7V9dcUOfr+yhJumTuD284/1+Xxbqtq48NGPuPG08fzvhVO47eUveH3HvjNXPDR3Bj95dQOj4uP49JtnY7U5OOnutwEof/AivtjTzHVPf8a/vn8yE0ck89yn5fzs9c3ux589fgRPXX8Cxhi6e+0kxDh/nV9dV8mo1DjmTcyiudPKWb/9gGZLL6mt0Vxz7Fj+uGvnviL3Dqz7rIwRREUIyy3Or/O/mDmNa08cx53/2sgH2+uotrrGxcv3PWZ0WxxtYuPWcyfz63e2ucevvz2hgLsvLmT1zkYW/mW1u70I9GVHUXI6xU3NpCdE0+waKjg1OYut1W2cOjWbdouNd7fW+nyPASJ3CVNGJbPJ2tZv/dVzx/Limj1kJ8dS394DVo87vZ7uvMIclm/xWNkFU0ensM3S7hxfrnOujo2KYN3t5zLt3mXQ6FWIFfBnv6wVrp+ez/LNNVS1Ot/LR646jlvL1vP8DXNZ/H4pn5Y1Mm1MCmMzErj9/GNpb+9hweP7Dn6PiYzA6hoGSnAFN0B+YwJ3XHAst76y3jmM4xpZOWfKCKpbLWyu6v8eFbQkssviHHq79sRxPLe6/6nSH7nqOG59Zf2+964B8jLieePmU5i/aAW4vmxcPGMUq8uaaOjo4b9PnsCt50zmB79+z/1hcvKETD7Zue8N+/y0c8nw4606HDrmfpTrtTu46cXP2Vjpezy0j8NhBvTsDtaDb2/j5he/cC8v3VgNQGldB7sbO1myvop3t9Tyj+IKd6+mb8zZeyzx5c/2cOGjHwHw3KfOPxjvMd2fvLoBgOpWC/e/tYXLF69y37dscw0/f2MLbRYbS9ZX8/0X1vULdnAOcZz12w/41dvbKLxnGU99vIt3t9TyP/9YzzVPrqHN0svGva3uwDTGDPjj7jNvYiaTc5IAeG9bHcu31BIdKYxMieNnr23ihTW7eemzPVS3DtzhCVDVauG2cydzwykF5KbFu9c/+fEuFj6xul+wL7psKk9fd4J7B2Hxbuf+h8tm7jvZ6kclDTR0WJk1No2clFj3+rjogX+ydodh0942Tp6QyZf3nAs4Pzx6XN8yfnDWRK6eO5a5BRnces5k7rm4kL9+84R+z5GW4Pym9P0zJvCVWc46Nle1DdhxOG1M6oAhtGU/Os29buGcPP72rTnu+y49bnS/ti9+Zy5j0uJ55pNyqlotjEp19oQffHsbANPHpPLSjSfy568fzxPXFvHHa2ZTkJXInIIMSu6/gOtPzgfgroumuJ8zIWbft8ryxi5uevGLAePzyXHRzBrb//pBURHOoTeA6EjhK8fve/9/cfk03rzlFE6bNPAcWBdOH0VaQgzjMvd9Y52Rm0rRuHQAMhNjiImK4H8vnEKy63255axJPPutOfz80qmA829quAVfzz2MGGN4fvVu3tpQzVsbqrn+5HzuvaQQkf5fYZdurOZ//rGeLqudh688jv86Ptd9X2NHD3f8ayO/uGyae8bB+Oykfq+xtbqd8dn7xq6tNgcxURF86JqpYOl1cPr/fdDvNb822/kaDR1Wznn4Q0amxvH8DXOx9Dr49t/Wsqp0Xy+lu9fON57+rN+OOW9/+WiX+3ZMVAS3vvKl+w/00ZUlAEwZlcJWj+GMvl7fE/XO7tkLa3b3+0N/a0M1GyqdXckfnzuZ367Y4d4mbyfkZ/CjcyazobKF/37+c8ZmJPA/509mfUUri97cwl3/3jTgMcflpbH46ll8vqeF5LgoTp2YBUCTa1jkohmj2FHTTm2bhYVzxvLSZ3sA55j+mceOYMcvL+BHr3zJG+ud32YWzhnLM5+U93uN6WNSqWpxbucZx2Tz56/P5q0N1dy3ZDPtPTb+98JjeWCpMxhPm5xNWkIMr980j7joSD4ubeCfn8P03DSuPSm/3/MaY7jzgmP5lStUrz0xn5K6Dm44pYBtNe38+4uBX2luOKWAC6Y5jw699LjRLFlfxVdmjeGYkcn85mszeOaTcu69ZCpx0ZG8+r2TSE+MYUJ2EgvnjHV/uI3NSGBSThJ7W7r5xknjuGJ2Lpf+YRW1rrnrqa4PmfnTRg14/ejICH583mRS4qJYMCePLqudvIx4HnhrKwB/uHoWZfWdPLxix4DHNnT08P0zCnl+9R73ule+exIbKlu4Zu4453vicbLb2WPTKRyd4vw556ay3qNzdWWR8zyKz31rLlc/uZrK5m5Gpsbz4/NGsLm6lfMKne/R5bPGcNGMUWypauO4POcHy/isRO5dspnSug7mFAxv313D/Sj29KpyfvHmvtkMz3xSzqmTsjh7Sk6/dt9/4XP37YdX7OCD7fWkJURzz8WF/O7dElZsqWX2uHR37yg1Ppqicek8unAWq0obuPG5dTy2cN/FszZVtTJjTCpry5s5JifZPX7s6c0Nzl59XwjsrO9k+n3LKcpPZ1VpI/MmZvYL+L4pbd85tYCclDh+6fqD7DMjN5X0hBg+3FHP766a6d6mMWnx7G3pZnx2In//7olMv2/5ft+vsnpnL+wXl0/j0ZUl3PmvfTver5uXz18/KXePUa/3+iYU7dopPCM3jY9/eqb7A3RWXjpZybH87LVNfOOkcVw9dyyf7Wpi+eZabj//GHLTE8j12ufwyJUzeemzPTy2YBYREfs+iNeUNVLW0MmIZGdPPDJCGOnqlSfHRjE5J4micemcPCGTsZmJLN9cQ+HoFDa5vvF8c14BcdGRfHV2Lj/9p/Nbz+Uzx1DdauGvq8q5eIYzEPuCZNKIJE6dlMXknOQB75WI8N3TJ+Awzp7m9NxU/v39eQDMmxjLzgcuZML/9jvjCD+7uNB9+9GFs3jU43dm/rRR/QK5KH9fcJ00IdN9e1RqPDNy01hT1sRNZ04kNmrfN5Hbzz9mQJ3ekuOiue08Z7v/PmMCAL9Zth1w7mCek5/hDvfnb5hLj83ODc8WMzIljkk5ybx5yylc/NjHjM1IYPa4dGa7etveRrq+UQD87Vtzae3uZcn6vURECBNcnaOxmQm8e9vp/L24ggunjSQqMoKPfnJWv+eJjoxw/zzA+fv84H9N7/eeDJfgmy0TRr72509YW95/yuC5hTn86Zrj+XxPCxsqW7jqhLz9Bt4Vs3PZUdvOhv0M6bz4nbm89sVe/l5c2W99VIRw/1em8dN/buT3C2ZywbRRPLxiB3/+cKfP5/EmAhvvO58dte1YbQ7W7W5mc1UrYzMSue3cyVS3dvf7JvDubaczMtU5q6Kxw0pWUgx3v7aJvS3dPPCV6Xznb8X84OxJXDh9FB+XNPD1p9a4HxsXHYGl18HU0SnuIZcvfnYu9y/dyqvr9m1X+YMXUd7QyVsbqzllYhaXuYaA/nD1LG5+8Qv+/PXjffYW+9gdxj2Mcqjuf2sLf/loF5/ccRajXUM3a8oaueqJ1UQIlP3qIp+P67U7KKvv5JiR+0L6PzvqeW71bp64djZWu4OKpm4mjkjy+fhDddZvPqCsoZM3bzkFhzHMyD3gZZEPaN3uJtZXtPKtUwqw9Npp6rQyOi0eYwwFdzo/RMof9L39gznvkQ/ZUdvBm7ecwrQxqazb3YwIHD/WGdwfbK9j9rh0kuOc3wraLb1ERUQQHxM54Lle/3IvL6zZw8vfObHfB/PRxN/ZMhruRwGHw1De2NlvuGRjZSuX/OHjfu0iI2TAGOiL357L1U+u6bcuJjKCcwtzWLrJ2bv2/hH/16wx/OuLvfzy8mk89l6J+ysxwCkTs/i4tMG9vPzW09w9v8l3vY3V7mBmXhpfVrRwwbSRvL3JefReZmIMVxTl8viHZYxKjePTO8/e7/a2WXqZ4fGBdLB/1Cu21LKmrJGGjh5+dM5kHl1ZwndPn8D6yhZqWy3ccvYkNlS2cOkfnAH+2MJZXOI19pt/x1vu195W08YxOckDhruGms3uoLyxq18IG2O4+7VNzJuYxYXT9//hEgg1rRbWljcNeO+G2lMf76IgK4Gzjs0ZvLEPGytbeXjFdv587WxiowYGdqjRcA8ii98v5bfLt/Ofn5xJbnoC5Q2dnPGbDwB45psncP1f1wJwVVFevwNXAL45L5+/rirvt+7C6SM5f+pIfvjylwNe64Vvz+Wk8ZkU3vsOgvTb8ZSWEM3qO8/mo5IGvvM3589m+y/nu/9gnlm1i/ve2MLyW08jNiqCnJQ4Ll+8im017bz349MZl5nIK2srGJeZwDzX+LMvxhhuevFzlm6s4YT8dP7xvZMP+j3zx+tf7mX2uPQBwyYAyzfX0NLVy5Un5Pl4pFJHL53nfhSqa7eQlRjL06t28ciKHZw9JYdFl03lyY/KcBhYubWO607O54U1ztklV88dyxnHjCArKYaGDivzp41kfWWLe543MCDYz5+aw/dOn0Cix4yG8VmJlLlmBRwzMpmICHHP1/b00U/OJC46knkT940HevaErp9XwIXTRzEiZd945OPXzubVdZXkZyYSESFcPXfsoO+DiPDHa2ZjjBnW3rLn7BNv5+lpY1WI03A/AuwOw5L1e7n1lfX86JxJ/O5d5+yPJeur+E9JPS2uqXr3LtnM0o3VrNnVxLEjk3ngK9PdjwfnLIt3fnQaj7jmmPvy+LXOD3SHw3DTmRM4LjeNUyZl0WW1k5kY4w7T+y4p5N2tdSyYk+ee/tg3Jtk3XzwvI37A83sGO8C4zER+fN7gO8J8Ge5hEKXCmYb7MOo7x8ifPtjpDuO+YP/uaeN5Y30VVa0Wrpk7ltioSJ5etYs1u5qYNTaN31+1byZC3zB7RmIMgPvozUuOG43d4SA7KZZnP+1/8EVEhPQ7uKgvsPtcP6+A6+cVAM5Doz2P3ANYf+957iMLlVLBR8N9CG2paiM/K8EdpHMfWElkhJCfOXDM95vzCrjprIl09djd066Oy0vljfVV/PbKmf0Ov7/roinc/domMpOc4d43/3behEwWzHEOg7y1sYYri3I5FBf42JG3v8P/lVLBQXeoDpGKpi5Ofeh9rpk7lvtdwyl9MzK8JcVGsfG+8w5rWKK0rp0J2Uk6tKFUmPF3h6pfpx8Qkfkisl1ESkXkjv20uVJEtojIZhF58WALDnavf+k8mKe8sZO6Nguldb5PPwqQlRRz2KE8ccTwT91TSgWvQYdlRCQSWAycC1QCa0VkiTFmi0ebScCdwDxjTLOIjBiugo8mnT02bA5Danw0/ylxzg2vaXWebMvTjNzUfgcSnZA/3KcMUkqFO3/G3OcApcaYMgAReRm4DNji0eY7wGJjTDOAMaZuqAs9mjgchpMeXOk++Kf0/gvcJ/baWT/w4g5TRqZw7yWFWHod5GclkhynuzqUUsPLn5QZA3geOVMJzPVqMxlARFYBkcB9xph3hqTCo9CqnQ39jupcX9lKd6/dfR6U+OhI98FBX5udyz2XFPabd66UUsPNnzF3XwO73ntho4BJwBnAQuBJERlwIgoRuVFEikWkuL7e99n5gsHKrf2/mHz1T58A8L3TxwNw1rEjuPnMiQDcPv8YDXal1BHnT+pUAp7HaOcCVT7arDbG9AK7RGQ7zrBf69nIGPME8AQ4Z8scatGBtr9zMV97Uj4XTh9FYmwUcdGR3HBKAemuuelKKXUk+RPua4FJIlKA87o1C4Crvdq8hrPH/oyIZOEcphl4ddkQ4RnuC+eMJT0hmuljnJeJy0zad2EFDXalVKAMGu7GGJuI3Awswzme/rQxZrOILAKKjTFLXPedJyJbADtwuzHG+yJcIaGxo4eaNgvfPqWATqudn108ZcDRn0opFWh+pZIxZimw1GvdPR63DXCb619I2tvSzfOrd/OnD5znNL9s5him56YGuCqllPJNu5x+uurxT6l0XT3+oa/O0GBXSh3V9ALZfuix2d3BHiHoOcCVUkc9DXc/9F3DMicllr9+c84grZVSKvB0WGYQJbXtfPVPnwLw6vdOJi9j4BkelVLqaKM990EsenPfWRbGpA28eIVSSh2NNNwH0dljc98+Wq+GrpRS3jTcD8Bmd7Ctpp2YyAgeXThr8AcopdRRQsfcD2BHbQddVju/XzCTS48bHehylFLKb9pzP4BPdjrP0T4rLz3AlSil1MHRnvt+PLJiB79fWUJWUix5GbojVSkVXLTn7oMxhqc+3sXU0Sk8+60T9HJ2Sqmgo+HuQ2VzNx09Nq6ZO46po/U0A0qp4KPh7sO2GufFrY8dlRzgSpRS6tBouPuwrboNgGNyNNyVUsFJw93L5qpWnvx4F+MyE/TyeEqpoKXh7uWSxz6mtbuXsXoOGaVUENNw9+JwXdk1JT46sIUopdRh0HEHDza7g6gIIS0hmnsvLgx0OUopdci05+6hutWCzWG4/fxjGJESF+hylFLqkPkV7iIyX0S2i0ipiNzh4/7rRaReRL50/fv20Jc6/HbUOqdAjstMDHAlSil1eAYdlhGRSGAxcC5QCawVkSXGmC1eTV8xxtw8DDUeMUs31pAcF8XMvLRAl6KUUofFn577HKDUGFNmjLECLwOXDW9ZgfHhjnrOmZJDXHRkoEtRSqnD4k+4jwEqPJYrXeu8fVVENojIqyISdFeQ7rbaaejoYeKIpECXopRSh82fcPd11izjtfwGkG+MmQG8Czzr84lEbhSRYhEprq+vP7hKh9nelm5AL6WnlAoN/oR7JeDZE88FqjwbGGMajTE9rsW/ALN9PZEx5gljTJExpig7O/tQ6h02lc1dAOSma7grpYKfP+G+FpgkIgUiEgMsAJZ4NhCRUR6LlwJbh67EI+Mf6yoBGKPhrpQKAYPOljHG2ETkZmAZEAk8bYzZLCKLgGJjzBLgByJyKWADmoDrh7HmIVfR1MVbG6oBGJGs89uVUsHPryNUjTFLgaVe6+7xuH0ncOfQlnbklNQ557c/d8McIiP0whxKqeCnR6gCZfWdAHphDqVUyNBwB3bWd5KWEE1GYkygS1FKqSGh4Q6U1LYzMVvntyulQkfYh7vV5mDD3laO01MOKKVCSNiH+5bqNqw2B8ePTQ90KUopNWTCPtx31nUAUDg6JcCVKKXU0An7cG/ocB5Ym50cG+BKlFJq6Gi4d/QQFx1BYoyeCVIpFTo03DusZCXFIqIHLymlQoeGe0cPWUk6JKOUCi1hHe4rttTyUUkD6QnRgS5FKaWGVFiH+6MrSwBo7uoNcCVKKTW0wjrc01w99nsuKQxwJUopNbTCOtxr2yycV5ijBzAppUJOWId7TauFUal6/nalVOgJ23Dvstpos9jI0XBXSoWgsA333Y3Oa6Zqz10pFYrCNtzf2lBNhMDJE7ICXYpSSg25sA33ldvqOHF8Jjkp2nNXSoWesAx3q81BaV07M/Uc7kqpEOVXuIvIfBHZLiKlInLHAdpdISJGRIqGrsShV1LXTq/dMGWUnuZXKRWaBg13EYkEFgMXAIXAQhEZcNSPiCQDPwDWDHWRQ21bdTuAhrtSKmT503OfA5QaY8qMMVbgZeAyH+1+ATwEWIawvmFR2dwNQF5GfIArUUqp4eFPuI8BKjyWK13r3ERkFpBnjHnzQE8kIjeKSLGIFNfX1x90sUOlqqWb7ORYYqP0HO5KqdDkT7j7OtG5cd8pEgE8Avx4sCcyxjxhjCkyxhRlZ2f7X+UQq2rtZnSa9tqVUqHLn3CvBPI8lnOBKo/lZGAa8IGIlAMnAkuO5p2qe1u6ydVwV0qFMH/CfS0wSUQKRCQGWAAs6bvTGNNqjMkyxuQbY/KB1cClxpjiYan4MNkdhqqWbj0yVSkV0gYNd2OMDbgZWAZsBf5ujNksIotE5NLhLnCobatpw9LrYNqY1ECXopRSwybKn0bGmKXAUq919+yn7RmHX9bw+WxXEwBzCjICXIlSSg2fsDtCdUNlKyNT4nSHqlIqpIVduO9q6GTCiMRAl6GUUsMq7MK9vLGTcZka7kqp0BZW4d7SZaWlq5cCDXelVIgLq3Dvu0DHuMyEAFeilFLDK6zCvbbNedqbUam6M1UpFdrCKtzr2nsAyE6ODXAlSik1vMIu3EUgMykm0KUopdSwCqtwr2/vISMhhujIsNpspVQYCquUq2+36JCMUioshFW417b1aLgrpcJC2IS7ze6gpK6dCdlJgS5FKaWGXdiEe2mFD7UGAAAODklEQVR9B5ZeBzPz0gJdilJKDbuwCff1FS0AzMjVU/0qpUJf2IT7JzsbyUqKpSBLTz2glAp9YRHuxhhWlTZwysRMRHxdElYppUJLWIR7U6eVhg4rM3J1vF0pFR7CItwbO62AnnZAKRU+wiLcGzqc55TJStJwV0qFB7/CXUTmi8h2ESkVkTt83P89EdkoIl+KyMciUjj0pR66hg5nzz1LzymjlAoTg4a7iEQCi4ELgEJgoY/wftEYM90YMxN4CHh4yCs9DI2unnum9tyVUmHCn577HKDUGFNmjLECLwOXeTYwxrR5LCYCZuhKPHyNHVYiI4S0+OhAl6KUUkdElB9txgAVHsuVwFzvRiJyE3AbEAOcNSTVDZGGjh4yEmOIiNBpkEqp8OBPz91XIg7omRtjFhtjJgA/Be72+UQiN4pIsYgU19fXH1ylh6G61cIInSmjlAoj/oR7JZDnsZwLVB2g/cvA5b7uMMY8YYwpMsYUZWdn+1/lYapo7mJshl43VSkVPvwJ97XAJBEpEJEYYAGwxLOBiEzyWLwIKBm6Eg+Pw2GobOrWcFdKhZVBx9yNMTYRuRlYBkQCTxtjNovIIqDYGLMEuFlEzgF6gWbguuEs+mDUtfdgtTvI03BXSoURf3aoYoxZCiz1WnePx+0fDnFdQ2ZPUxeAhrtSKqyE/BGqFa5w12EZpVQ4Cflw39PUhQiMTosLdClKKXXEhHy4VzR1MSoljtioyECXopRSR0zoh3tzl463K6XCTsiHe3mjhrtSKvyEdLi3W3qpb+9hQnZSoEtRSqkjKqTDvay+E4Dx2XrdVKVUeAnpcN9Z3wGgPXelVNgJ6XAvb3ROg9Q57kqpcBPS4V7baiErKZaYqJDeTKWUGiCkU6+mzcLIFD14SSkVfkI63GvbLORouCulwlBIh3tNm4WRqXqRDqVU+AnZcLf02mnp6tVhGaVUWArZcK9tswDosIxSKiyFbLjXtDrDfVRqfIArUUqpIy90w93Vc9cxd6VUOArdcG/VYRmlVPgK3XBvs5AYE0lyXHSgS1FKqSPOr3AXkfkisl1ESkXkDh/33yYiW0Rkg4isFJFxQ1/qwalts5CTqr12pVR4GjTcRSQSWAxcABQCC0Wk0KvZF0CRMWYG8Crw0FAXerD2NnczJk13piqlwpM/Pfc5QKkxpswYYwVeBi7zbGCMed8Y0+VaXA3kDm2ZB29PU5eeMEwpFbb8CfcxQIXHcqVr3f7cALx9OEUdrjZLL81dvRruSqmwFeVHG/GxzvhsKPJ1oAg4fT/33wjcCDB27Fg/Szx4FU3OLxEa7kqpcOVPz70SyPNYzgWqvBuJyDnAXcClxpgeX09kjHnCGFNkjCnKzs4+lHr90hfueu1UpVS48ifc1wKTRKRARGKABcASzwYiMgt4HGew1w19mQdnT1+4p2u4K6XC06DhboyxATcDy4CtwN+NMZtFZJGIXOpq9n9AEvAPEflSRJbs5+mOiIqmbpLjokhN0DnuSqnw5M+YO8aYpcBSr3X3eNw+Z4jrOiwVzV3aa1dKhbWQPEK1oqmLvAyd466UCl8hF+7GGCqbu3WmjFIqrIVcuNe399Bjc+hMGaVUWAu5cK9o1pkySikVeuHe1A2gY+5KqbAWguHu7Lnnas9dKRXGQi/cm7vITo4lLjoy0KUopVTAhF64N3WTl65DMkqp8BZ64d7cpTNllFJhL6TC3WZ3UN1q0ZkySqmwF1LhXtViwe4wOlNGKRX2Qircyxo6ACjISgpwJUopFVihFe71nQCMz04McCVKKRVYIRXuO+s7SImLIjMxJtClKKVUQIVUuO9q6GR8dhIivq4MqJRS4SOkwr28oZPxWToko5RSIRPull47Va0WxmVquCulVMiEe991U/OzdI67UkqFTLiXNzhnyuRrz10ppUIn3Hc3unruGu5KKeVfuIvIfBHZLiKlInKHj/tPE5HPRcQmIlcMfZmD29XYSXpCNKkJ0YF4eaWUOqoMGu4iEgksBi4ACoGFIlLo1WwPcD3w4lAX6K/djZ26M1UppVz86bnPAUqNMWXGGCvwMnCZZwNjTLkxZgPgGIYa/VLe0EWBToNUSinAv3AfA1R4LFe61h00EblRRIpFpLi+vv5QnsKnbqudqtZuHW9XSikXf8Ld1+Ge5lBezBjzhDGmyBhTlJ2dfShP4dPO+g6Mgck5esIwpZQC/8K9EsjzWM4FqoannEOzo7YdgEka7kopBfgX7muBSSJSICIxwAJgyfCWdXB21HYQHSm6Q1UppVwGDXdjjA24GVgGbAX+bozZLCKLRORSABE5QUQqga8Bj4vI5uEs2ltJbTvjs5KIjgyZaftKKXVYovxpZIxZCiz1WnePx+21OIdrAqKkroMZuamBenmllDrqBH1Xt9tqp6K5i0kjkgNdilJKHTWCPtxL63SmjFJKeQv6cN83U0Z77kop1Sfow72kzjlTJj9TT/WrlFJ9gj/cXTNlonSmjFJKuQV9Iu6oa9eDl5RSyktQh3uX1UZFUzeTdbxdKaX6CepwL63rAHSmjFJKeQvqcC+pdYb7RJ3jrpRS/QR1uO+oaycmMkJnyiillJegDveS2g7GZyfqTBmllPIS1Km4o7ZdD15SSikfgjbcu6w2Kpu7mTRCd6YqpZS3oA33svpOACZquCul1ABBG+47650zZSZka7grpZS34A33ug4iBPKzdKaMUkp5C9pwL63vYGxGArFRkYEuRSmljjpBG+476zp1vF0ppfYjKMPdZnewq6FTx9uVUmo//Ap3EZkvIttFpFRE7vBxf6yIvOK6f42I5A91oZ4qm7ux2h1M0J67Ukr5NGi4i0gksBi4ACgEFopIoVezG4BmY8xE4BHg10NdqKe+E4Zpz10ppXzzp+c+Byg1xpQZY6zAy8BlXm0uA5513X4VOFtEZOjK7K9vGuREDXellPLJn3AfA1R4LFe61vlsY4yxAa1A5lAU6MvO+g6ykmJJTYgerpdQSqmg5k+4++qBm0Nog4jcKCLFIlJcX1/vT30+ldZ1MCE78ZAfr5RSoc6fcK8E8jyWc4Gq/bURkSggFWjyfiJjzBPGmCJjTFF2dvYhFWyMYWe9ToNUSqkD8Sfc1wKTRKRARGKABcASrzZLgOtct68A3jPGDOi5D4WGDiut3b26M1UppQ4garAGxhibiNwMLAMigaeNMZtFZBFQbIxZAjwFPCcipTh77AuGq2D3OWW0566UUvs1aLgDGGOWAku91t3jcdsCfG1oS/OtbxqkDssopdT+Bd0RqiOSYzm3MIdRKXGBLkUppY5afvXcjybnTR3JeVNHBroMpZQ6qgVdz10ppdTgNNyVUioEabgrpVQI0nBXSqkQpOGulFIhSMNdKaVCkIa7UkqFIA13pZQKQTJM5/ca/IVF6oHdh/jwLKBhCMsJBrrN4UG3OTwczjaPM8YMelrdgIX74RCRYmNMUaDrOJJ0m8ODbnN4OBLbrMMySikVgjTclVIqBAVruD8R6AICQLc5POg2h4dh3+agHHNXSil1YMHac1dKKXUAQRfuIjJfRLaLSKmI3BHoeoaKiDwtInUissljXYaIrBCREtf/6a71IiKPut6DDSJyfOAqP3Qikici74vIVhHZLCI/dK0P2e0WkTgR+UxE1ru2+eeu9QUissa1za+4rleMiMS6lktd9+cHsv5DJSKRIvKFiLzpWg7p7QUQkXIR2SgiX4pIsWvdEfvdDqpwF5FIYDFwAVAILBSRwsBWNWSeAeZ7rbsDWGmMmQSsdC2Dc/snuf7dCPzpCNU41GzAj40xU4ATgZtcP89Q3u4e4CxjzHHATGC+iJwI/Bp4xLXNzcANrvY3AM3GmInAI652weiHwFaP5VDf3j5nGmNmekx7PHK/28aYoPkHnAQs81i+E7gz0HUN4fblA5s8lrcDo1y3RwHbXbcfBxb6ahfM/4DXgXPDZbuBBOBzYC7OA1qiXOvdv+c4L0x/kut2lKudBLr2g9zOXFeQnQW8CUgob6/HdpcDWV7rjtjvdlD13IExQIXHcqVrXajKMcZUA7j+H+FaH3Lvg+vr9yxgDSG+3a4hii+BOmAFsBNoMcbYXE08t8u9za77W4HMI1vxYfsd8BPA4VrOJLS3t48BlovIOhG50bXuiP1uB9s1VMXHunCc7hNS74OIJAH/BH5kjGkT8bV5zqY+1gXddhtj7MBMEUkD/g1M8dXM9X9Qb7OIXAzUGWPWicgZfat9NA2J7fUyzxhTJSIjgBUisu0AbYd8u4Ot514J5Hks5wJVAarlSKgVkVEArv/rXOtD5n0QkWicwf6CMeZfrtUhv90AxpgW4AOc+xvSRKSvs+W5Xe5tdt2fCjQd2UoPyzzgUhEpB17GOTTzO0J3e92MMVWu/+twfojP4Qj+bgdbuK8FJrn2tMcAC4AlAa5pOC0BrnPdvg7nmHTf+m+49rCfCLT2fdULJuLsoj8FbDXGPOxxV8hut4hku3rsiEg8cA7OHY3vA1e4mnlvc997cQXwnnENygYDY8ydxphcY0w+zr/X94wx1xCi29tHRBJFJLnvNnAesIkj+bsd6J0Oh7CT4kJgB85xyrsCXc8QbtdLQDXQi/NT/AacY40rgRLX/xmutoJz1tBOYCNQFOj6D3GbT8H51XMD8KXr34WhvN3ADOAL1zZvAu5xrR8PfAaUAv8AYl3r41zLpa77xwd6Gw5j288A3gyH7XVt33rXv819WXUkf7f1CFWllApBwTYso5RSyg8a7kopFYI03JVSKgRpuCulVAjScFdKqRCk4a6UUiFIw10ppUKQhrtSSoWg/wfM8ryy+1uKJgAAAABJRU5ErkJggg==\n", 161 | "text/plain": [ 162 | "
" 163 | ] 164 | }, 165 | "metadata": {}, 166 | "output_type": "display_data" 167 | } 168 | ], 169 | "source": [ 170 | "# plotting MI using MINE\n", 171 | "plt.plot(plot_x, -plot_y_mine)\n", 172 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 173 | "plt.show()" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 8, 179 | "metadata": {}, 180 | "outputs": [ 181 | { 182 | "data": { 183 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmUVOWd//H3F1AWRZBFAVkFXIgLiT2CSVxHozjGZcYY/GXRqNHM6GTRJGNmMsYk5py4TGL0EBNHMyaahEQTIxqNGqNxAwRkEVCWBhSkhabZGlmb/v7++FZZ1d3V0E1X9+2u+3mdU+duT9373FtVn3rqubeqzN0REZF06ZR0BUREpO0p/EVEUkjhLyKSQgp/EZEUUviLiKSQwl9EJIUU/iIiKaTwFxFJIYW/iEgKdUm6Ao3p16+fDx8+POlqiIh0KLNmzVrn7v33Vq7dhv/w4cOZOXNm0tUQEelQzOztppRTt4+ISAop/EVEUkjhLyKSQgp/EZEUUviLiKSQwl9EJIUU/iIiKaTwF5HS4A6/+hVUVyddkw5B4S8ipWHWLLjsMrj22qRr0iEo/EVkz2pqYMaMaFm3J48/Du+9l5uurIzhW29FnbNqa1uvDhs3wl/+su/3370bduwoXn2aQeEv0p6tXg0VFRFgkybBH/7QsExlJcyfH+O33AKHHw7LlsHSpXXLbd8Ou3bF+Nq18MQTcPfdMT17dgRRIRddBCeeCA8/3PR6z58P778frfFJkwqXWbEC1q+P8QcfhKee2vt6N22CZ56J4fnnw0c/GvOnT4d582J8xgzYbz94/XX4+9+hc2eYM6fxdVZXQ//+cSwaOwYAq1ZFULvDbbfBCy/AeefBhAlxPLM2boQ33qh735UrYcmS3PTOnTGcOBG6dYt1Tp9e902rtbl7u7ydcMIJLh3c7Nnuq1Y1/37vvut++ukxfPFF91mzmr+O73/f/Y47mn8/d/c1a9xfeCE3XVvrfu+97vfc475unfumTTF/82b3q65yX7687v1/+cvYvrv73/8e93V3X7/e/Y9/dP/KV9wrK91/8Qv3116re99169wfeii26e4O7mbuDz8c4+D+/vvuO3fG8kmTcvOz5bM3s7rr7tfP/YwzYnz48Fy5p56K4Y9+5H7zze7/9m9RZscO97KyXLkLLih8vG6/3f2552K8vNz9iiui/H/8h/u4cTF+yilR16zKypg/ZEjdeu/YUXgb7u4vv5wrd/XVdfe10O2b33T/+Mdj/Oyz3T/9afdt22Jdu3e7//zn8VjOnJm7z1e/Go/RWWe5f+ELuW3Pnh3LTz01Hl+oewz/+tdc2QkTYt4dd8R2du9279o15s2d6z5wYIxn6wbuP/1pDG+4IZ4X2cd/HwAzvQkZa97ePspllJWV+b7+sNt3H1/AwtWbi1wjKWj79rj17t1wWbbV9fGPN37/nTvj6d+1a0xv2ACVa6HiPRgxApYvj/mnnhrD97fAxk1w2GExXVsLnQyw3DpXroyWb/799qaiIlqARxwBL78cLcCRI6MldvDBdVuOBx4AJ5TB229H63Xo0Khr/n4DnHwyvPRSjH/sozBnbrSGAXr2jO317QvHHBPzKith4cLcerp1i2MLMHhwtDyzDu4No0ZHKzfrQx+CBQvq7leP7nD4SOjVC155Jeb16ZNrcec76CDYvDm3vS5dYv+yOlm0kLdlHu9hQ6F6S+7YjBsXre9t22K6d+94fLduza1j3LjYz+zjA3Uf506doH8/6H0wHHpoHKOdO6POs1+HLe83rHdjunRp2JIeMyb2Yf36aJ2PGBHbLC8vvI6xx8c28z9F9egOW7fVLTdiBFRVxfPyzTdz84cMhve35o53587x3Bo0KD7VZZnV6VYbc8oJfOdLZzV9X/OY2Sx3L9tbuXb7q57SDLt3Aw6dW/nhrKqKF+kJHwHL9BhOnx7Dk0+OFxHEC66T5dWtEflhd+qpERrZj+5Qt495x44I+oUL44V3QA/oeRBMnRohMXJkBHh5ed37bamOEN20Gbp3hx49YM2a2Jcjj4wX44IFsG5dlO/XL1fnbCC8Xe9HEre8D/PmwoaNueMyYkS8mFcsz5XLBj/AK6/GsHu3OE7ZK1KqqmDLltj3/ODv3DkX/BDdHPlhtmFj1MsMRo+GxYsbBj/EsZo/P/Y7KxtEw4bl9s0sF/xQ940GYMCh8N4aWJPp3ti8Gd55J45tVva5cMQRsHEDrK1sWJ9smXzL845ZbW3cb9MmWLsmd4xHjYo3nYN6Qq/e8QYPuTepQYNg1kyo9VzXVvZYde2a61dfuDDe5Lp3b7jt/OOR9cb83PMh+6aZH/yHHRZdPtn1ZI/hscfGvJV5x/GQQ3LdQ6NHx/O0oiK3PP/NviLvjaG1NOXjQRI3dfs0UXW1e58+8ZHxttvct25tWGbnzlh27rnuP/tZw+V//nN0reza5b5sWcPlW7bEssMOi+0880zMnz4997H1gAOim+HLX47pn/0st2zevJi/a1fcb+pU9098ou5H9H/9V/d///e9f5TP3nr2dP/sZ3PTH/pQ42WPOio3/sgjufEvfcl9wwb37t1z84YObXw9P/2p+3HH1Z13ww1Nq+9++7lfdlns/+TJMS9/u/m3AQPca2rcL7oo10WQPUb1y55ySnQlZKc/85kY9upVeN0VFe4/+IF7jx7RXZWd/9pr0d1z993xfHj7bfdDDsktf/75GB56aHTHNbafBxwQXVdf/3pu3v/+r/utt8b44Ye7v/KK+zXXxPT117v/8z/nyp53XnQ/Zaevv77u+u+7L47hm2+6P/ts3edpTU10G0Ecu7PPjvHvfz93/2HDGq/7ihV1p08/PYYjRrgfeWS8RsaPzx3n3/wmXlv1H5cxY6Kr57nn6s5/770YDhoU9d2wIbqZBg6M12dlZTwORxzhfv75zcuBPDSx22evBZK6KfybYMsW906d6j7BJk5sWO7OO+uW+eEP3T/6UfeXXor+7ez8/BfsoEERAjt3Rt/mDTe4H310LoTco++6sRdSt26F5598svvgwY3fD9wvvLDw/IsuiuHXvhbhVX95ti92b7ehQ90vvbTuvAcfzL3YBw0qfL+sV191f+yx6HvfvTsXUN26RSBceGGE73nnxRve7t11+3ArKiIkJ092//Wv3X/1q3gsHnss1tO7d67srFm57f/ud9EPDXH+AeINacuWXJmamnjMXnwxpsvKciF07rm59WbfiP/7v6MvvZDqavebbor7bt4c50EqK2PZb37jfvnldY/PHXdE8LvH8P/+LwJ6587Y/z/8IdbjHtP5/fvZxsKVV+b64K+4Ipbl962vWLHHl4S7u1dVuW/fHg2h11+P8y4Q5zx27XI/8MCYvuaaus8D93g8stMrV8a68h+77JvY5Zfn5q1dGw2Xu+6KN/Rp03LLFiyI1+QDD8T088+7L1q05/r/539GY2kfKfzT4I47CofUtde6P/54nGSaNs39tNMKl+vUKddSLHQ77bR4wRZalj0R+MlPxhvIb3/rftBBTQtfiBbcunXxRvTyyxGYn/pUvPDd65b9/e/jhFptbYTajh0RPuPHx/R110WrurY2Po3Uf2MYONB91Kjcibh77okWVn6ZRYuiJXbLLfGml2313nJLDP/lX/b8WKxb5756ddMfu2z45tu6Nbb13e/WLZet49q1Mf3ee7Fs06ZcMD31VLypZG3b5n7xxRE+7nH8qqubXr+sPZ14nDo16vXFL8ZJ7UL71FTV1fHGlt23p5/OnZz9p3/KhfG+WLs27j9uXEy/804EdU1NTM+YEc+jrI0b4zlQyMaN8an19dcLL8+uM0EK/1JSWRkfJR96KDcve7XDoYfmwqF374Yhe+650UVSf/7DD0c3TaFgfvXVCMuePePjbv6yr341QiU7feutuTq9+mrMGzs2Wm2XX+7+l7/kyi5cGK3rXr3iDWNPPvKR3P3eead5x2vbtgiPBx+Mq09mzYqwr66Oll32SpkrrnAfOTKORX133RXbnjMnyu/e3bw67Kvt2xsG7ty5uSuG2ps//Wnf3lSao7LS/W9/a9k6fvvb6K5KAYV/RzR7doTU1KkRkP37R4suv3U+a1b0oeZPP/54jI8albvMrrHbeeflLmHL9pOfdVZu+be/HcuyH2/zb5dcEuHkHt0WEB9j8z36qPuSJbnpnTujTze/a6Epl7FVV8f5h+z5hdZQW7vnutS/hFOkA2hq+JfkpZ4djjt8/vPw0EMxPWFCwy+81L80rFu3+CbjsGHxRZqysrh6ZeHC+NZj9lLI+qqq4qoFgBdfjMv0rrkmrtwYODCuCunUKbZ1881xaVxtLdxwAzz5ZNQNYvzGG2HatLpXkohIopp6qWfiLfzGbiXZ8l+zpm6r+OWX4yTqn/7UsJX9jW9EN8r550crvKoq19Xzuc/lWuDu0deb32p3z62nqsr9/vvjBC/s25dHsn3pItLu0ZYtfzM7B/gJ0Bm4z91/WKDMJcDNgANz3f3/7WmdJdfyd89dB5895iefHF8oKuTRR+HCCxvO37AhvjxjVnf+6tVxvXvnzjE9bVpczzxmTExXV8c1xiNHtnxfRKTdarMveZlZZ2AScBawCphhZlPcfWFemdHAt4CPufsGMzukpdvtMHbsiC+Z/PWvuXl33w0PPBBdMI0ZN67w/IMPLjx/0KC60+PH153u2TNuIiIU5xu+JwJL3X0ZgJlNBi4A8r6uyBeBSe6+AcDd1zZYS6l56CG48sr4avqcOfDHP+aWffnLDcuXl8M3vhE/9LRqVfS/i4i0kmKE/2HAyrzpVUD9ZusRAGb2CtE1dLO7t+B3UNu5bdvgc5/LTZ9xRnwt/IQT4uQswOmnw/PPw4ABMGVK/BJjoV9sFBFpBcUIfyswr/6JhC7AaOA0YDDwkpkd4+4b66zI7GrgaoChQ4cWoWoJmTat7nT2t1S+850I/4oKuOsu+Pa34ZJL4B/+oe3rKCKpVozwXwUMyZseDNT/VaJVwDR33wUsN7NFxJvBjPxC7n4vcC/ECd8i1C0ZM2bUnf7CF+JyyQkT4JOfzM2//fa2rZeISEYxwn8GMNrMRgDvAhOB+lfy/Am4FHjAzPoR3UDLKDU1NfEHD/fdF904550XXT4XXJB0zURE6mhx+Lt7jZldBzxN9Of/wt0XmNn3iOtNp2SWfcLMFgK7gW+4+x4udemA5s6FsWNz07feCt/8ZnL1ERHZg6L8ALy7Pwk8WW/eTXnjDlyfuZWmO+7IjT/+eLT6RUTaKf2ZS0u5wz33xKWdPXrAF7+o4BeRdk/h31JvvQXXXhvjP/kJXHVVsvUREWmCTklXoMObMiU3ftJJydVDRKQZFP4t9dhj8Qfe99+f+x0dEZF2TuG/r3btgv/5n/gD8auugiuuaPhjayIi7ZTCf1/deSd8/evxg2oTJyZdGxGRZtEJ3331zDNw1FHx5ylq8YtIB6OW/77YtQtefRXOPFPBLyIdksJ/X8yeDVu3wimnJF0TEZF9ovBvrkWLcn+0cvLJydZFRGQfKfybK/+P1QcMSK4eIiItoPBvrjfeiOHy5cnWQ0SkBRT+zTVnDpx1FgwfnnRNRET2mcK/OVativDXzziISAen8G+OyZPjH7ny/59XRKQDUvg3x+uvw7BhMGpU0jUREWkRhX9zvPUWHH100rUQEWkxhX9T1dZG+B91VNI1ERFpMYV/Uz3zDGzbBscck3RNRERaTOHfVD/4QfT16xc8RaQEKPybwh3mzYOzz4YDDki6NiIiLabwb4p33oHNm+HYY5OuiYhIUSj8m2LevBgq/EWkRCj8m2L6dOjcGY4/PumaiIgUhcK/KaZOhbFj1d8vIiVD4b837jBjBowfn3RNRESKRuG/N+vXQ3U1jB6ddE1ERIpG4b8377wTwyFDkq2HiEgRKfz3Jhv+Q4cmWw8RkSJS+O+Nwl9ESpDCf09qa+GRR6B7d+jfP+naiIgUjcJ/T2bNghdfhK99DcySro2ISNEUJfzN7BwzW2RmS83sxj2Uu9jM3MzKirHdVldeHsNLL022HiIiRdbi8DezzsAkYAIwBrjUzMYUKNcT+DIwvaXbbDPLl8dQf9YuIiWmGC3/E4Gl7r7M3XcCk4ELCpT7PnAbsL0I22wby5dDv35w4IFJ10REpKiKEf6HASvzpldl5n3AzD4MDHH3J/a0IjO72sxmmtnMysrKIlSthZYvhxEjkq6FiEjRFSP8C50J9Q8WmnUCfgzcsLcVufu97l7m7mX9k766prY2/rBd/9wlIiWoGOG/Csj/+utgYHXedE/gGOAFM1sBjAemtPuTvvPmxU87nH560jURESm6YoT/DGC0mY0ws/2BicCU7EJ33+Tu/dx9uLsPB6YB57v7zCJsu/W8/HIMTz012XqIiLSCFoe/u9cA1wFPA28Cv3f3BWb2PTM7v6XrT8y8edCnj37TR0RKUpdirMTdnwSerDfvpkbKnlaMbba6N96A447Tl7tEpCTpG76F1NZG+OtvG0WkRCn8C6mogPffh6OOSromIiKtQuFfSPZnHUaNSrYeIiKtROFfyNKlMRw5Mtl6iIi0EoV/IeXl0KULDBuWdE1ERFqFwr+Q+fOj1d+lKBdDiYi0Owr/+txh6lQYPz7pmoiItBqFf33LlkFlJZx0UtI1ERFpNQr/+ubMieEJJyRbDxGRVqTwr2/+/PhW75gG/0cjIlIyFP71ZU/29uiRdE1ERFqNwr++hQvV6heRkqfwz+ce/96lL3eJSIlT+OerrIRt2/SH7SJS8hT++VasiKHCX0RKnMI/n8JfRFJC4Z9v0aIYjhiRbD1ERFqZwj/fSy/FH7j07Jl0TUREWpXCP6umBl59FU45JemaiIi0OoV/1uzZ8e9dCn8RSQGFf9aLL8ZQ4S8iKaDwz5o2Lb7cNWBA0jUREWl1Cv+sZcvgiCOSroWISJtQ+GetWKHr+0UkNRT+AJs3w/r1Cn8RSQ2FP8Dbb8dQ4S8iKaHwB1iyJIaHH55sPURE2ojCH2DePOjUSb/jLyKpofAHmDsXRo/Wv3eJSGoo/CFa/scdl3QtRETajMK/ujqu8T/++KRrIiLSZooS/mZ2jpktMrOlZnZjgeXXm9lCM5tnZs+Z2bBibLco3ngjhmr5i0iKtDj8zawzMAmYAIwBLjWz+mdOZwNl7n4c8AhwW0u3WzTz5sVQLX8RSZFitPxPBJa6+zJ33wlMBi7IL+Duz7v71szkNGBwEbZbHDNmQN++MGRI0jUREWkzxQj/w4CVedOrMvMacyXwVBG2WxzTpsH48WCWdE1ERNpMMcK/UGp6wYJmnwXKgNsbWX61mc00s5mVlZVFqNpebNoEb74J48a1/rZERNqRYoT/KiC/z2QwsLp+ITM7E/gv4Hx331FoRe5+r7uXuXtZ//79i1C1vXjrLXBXf7+IpE4xwn8GMNrMRpjZ/sBEYEp+ATP7MPBzIvjXFmGbxbF4cQyPPDLZeoiItLEWh7+71wDXAU8DbwK/d/cFZvY9Mzs/U+x24EDgYTObY2ZTGlld21q0CDp3hhEjkq6JiEib6lKMlbj7k8CT9ebdlDd+ZjG2U3SLF0fw779/0jUREWlT6f6G7+LF6vIRkVRKb/jX1kb4668bRSSF0hv+774L27Yp/EUkldIb/tkrfRT+IpJC6Q3/BQtiePTRydZDRCQB6Q3/efOgf38YMCDpmoiItLl0h/9xx+k3fUQkldIZ/rW10e1zzDFJ10REJBHpDP/Vq2HrVl3jLyKplc7wX7IkhrrSR0RSKp3hn73Mc/ToZOshIpKQdIb/kiXQrRsMbj9/KCYi0pbSGf6LF8OoUdApnbsvIpLO9FuyRP39IpJq6Qv/mhooL1f4i0iqpS/8334bdu3SyV4RSbX0hb8u8xQRSWH46zJPEZEUhv+SJXDQQXDIIUnXREQkMekL/+y/d+kH3UQkxdIZ/uryEZGUS1f479gRV/voZK+IpFy6wr+8HNzV8heR1EtX+OsyTxERIG3hv3RpDEeNSrYeIiIJS1f4L1sGBx8cNxGRFEtX+JeXw+GHJ10LEZHEpSv8ly1T+IuIkKbw370bVqxQ+IuIkKbwf/fd+DXPkSOTromISOLSE/7l5TFUy19EJEXhv2xZDBX+IiLFCX8zO8fMFpnZUjO7scDyrmb2u8zy6WY2vBjbbZZly6BLFxgypM03LSLS3rQ4/M2sMzAJmACMAS41szH1il0JbHD3UcCPgVtbut1mKy+HYcPiDUBEJOWK0fI/EVjq7svcfScwGbigXpkLgF9mxh8B/tGsjX9TWZd5ioh8oBjhfxiwMm96VWZewTLuXgNsAvoWYdtNp/AXEflAMcK/UAve96EMZna1mc00s5mVlZVFqFrGpk1QVaXLPEVEMooR/quA/LOog4HVjZUxsy5AL2B9/RW5+73uXubuZf379y9C1TKWL4/hiBHFW6eISAdWjPCfAYw2sxFmtj8wEZhSr8wU4LLM+MXA39y9Qcu/1bz7bgwHD26zTYqItGctvvTF3WvM7DrgaaAz8At3X2Bm3wNmuvsU4H7gQTNbSrT4J7Z0u81SURHDgQPbdLMiIu1VUa57dPcngSfrzbspb3w78KlibGufrM70Qg0YkFgVRETak3R8w7eiAvr2ha5dk66JiEi7kJ7wV5ePiMgHFP4iIimk8BcRSaHSD393eO89GDQo6ZqIiLQbpR/+VVXxJy5q+YuIfKD0wz97mafCX0TkA6Uf/vqCl4hIAwp/EZEUUviLiKRQOsK/Vy/o0SPpmoiItBvpCH+1+kVE6ij98F+9WuEvIlJP6Ye/Wv4iIg2Udvi7R/jr270iInWUdvhv3gzbt+t3/EVE6int8F+7NoaHHppsPURE2pnSDv81a2J4yCHJ1kNEpJ0p7fDPtvwV/iIidaQj/NXtIyJSRzrCv1+/ZOshItLOlHb4r1kDffrAfvslXRMRkXaltMN/7Vr194uIFFD64a/+fhGRBko7/NesUctfRKSA0g5/dfuIiBRUuuG/cyds2KBuHxGRAko3/CsrY6iWv4hIA6Ub/vp2r4hIo0o//Pv3T7YeIiLtUOmGf1VVDPv2TbYeIiLtkMJfRCSFWhT+ZtbHzJ41syWZ4cEFyow1s6lmtsDM5pnZp1uyzSbLhn+fPm2yORGRjqSlLf8bgefcfTTwXGa6vq3A5939Q8A5wJ1m1ruF2927qiro1Qu6dGn1TYmIdDQtDf8LgF9mxn8JXFi/gLsvdvclmfHVwFqg9c/CVlXp1zxFRBrR0vA/1N0rADLDPV5XaWYnAvsD5S3c7t5VVam/X0SkEXvtEzGzvwKF/gH9v5qzITMbCDwIXObutY2UuRq4GmDo0KHNWX1DVVW6xl9EpBF7DX93P7OxZWa2xswGuntFJtzXNlLuIODPwLfdfdoetnUvcC9AWVmZ761ue1RVBUcf3aJViIiUqpZ2+0wBLsuMXwY8Vr+Ame0PPAr8yt0fbuH2mk7dPiIijWpp+P8QOMvMlgBnZaYxszIzuy9T5hLgFOByM5uTuY1t4Xb3bOdOqK5W+IuINKJF10G6exXwjwXmzwSuyow/BDzUku002/r1MVT4i4gUVJrf8NW3e0VE9kjhLyKSQgp/EZEUUviLiKSQwl9EJIVKN/y7doUePZKuiYhIu1S64d+3L5glXRMRkXaptMNfREQKUviLiKSQwl9EJIUU/iIiKVR64e8ev+2j8BcRaVTphf/mzVBTo/AXEdmD0gv/mhr49Kfh2GOTromISLvVop90bpf69oXJk5OuhYhIu1Z6LX8REdkrhb+ISAop/EVEUkjhLyKSQgp/EZEUUviLiKSQwl9EJIUU/iIiKWTunnQdCjKzSuDtFqyiH7CuSNXpKLTP6aB9Tod93edh7t5/b4Xabfi3lJnNdPeypOvRlrTP6aB9TofW3md1+4iIpJDCX0QkhUo5/O9NugIJ0D6ng/Y5HVp1n0u2z19ERBpXyi1/ERFpRMmFv5mdY2aLzGypmd2YdH2Kxcx+YWZrzWx+3rw+ZvasmS3JDA/OzDczuytzDOaZ2UeSq/m+M7MhZva8mb1pZgvM7CuZ+SW732bWzcxeM7O5mX3+bmb+CDObntnn35nZ/pn5XTPTSzPLhydZ/5Yws85mNtvMnshMl/Q+m9kKM3vDzOaY2czMvDZ7bpdU+JtZZ2ASMAEYA1xqZmOSrVXRPACcU2/ejcBz7j4aeC4zDbH/ozO3q4F72qiOxVYD3ODuRwPjgWszj2cp7/cO4Ax3Px4YC5xjZuOBW4EfZ/Z5A3BlpvyVwAZ3HwX8OFOuo/oK8GbedBr2+XR3H5t3SWfbPbfdvWRuwEnA03nT3wK+lXS9irh/w4H5edOLgIGZ8YHAosz4z4FLC5XryDfgMeCstOw30AN4HRhHfNmnS2b+B89z4GngpMx4l0w5S7ru+7CvgzNhdwbwBGAp2OcVQL9689rsuV1SLX/gMGBl3vSqzLxSdai7VwBkhodk5pfccch8tP8wMJ0S3+9M98ccYC3wLFAObHT3mkyR/P36YJ8zyzcBfdu2xkVxJ/BNoDYz3ZfS32cHnjGzWWZ2dWZemz23S+0/fK3AvDRezlRSx8HMDgT+AHzV3TebFdq9KFpgXofbb3ffDYw1s97Ao8DRhYplhh1+n83sPGCtu88ys9OyswsULZl9zviYu682s0OAZ83srT2ULfo+l1rLfxUwJG96MLA6obq0hTVmNhAgM1ybmV8yx8HM9iOC/9fu/sfM7JLfbwB33wi8QJzv6G1m2cZa/n59sM+Z5b2A9W1b0xb7GHC+ma0AJhNdP3dS2vuMu6/ODNcSb/In0obP7VIL/xnA6MxVAvsDE4EpCdepNU0BLsuMX0b0iWfnfz5zhcB4YFP2o2RHYtHEvx94091/lLeoZPfbzPpnWvyYWXfgTOIk6PPAxZli9fc5eywuBv7mmU7hjsLdv+Xug919OPGa/Zu7f4YS3mczO8DMembHgU8A82nL53bSJz1a4STKucBiop/0v5KuTxH367dABbCLaAVcSfRzPgcsyQz7ZMoacdVTOfAGUJZ0/fdxnz9OfLSdB8zJ3M4t5f0GjgNmZ/Z5PnBTZv7hwGvAUuBhoGtmfrfM9NLM8sOT3ocW7v9pwBOlvs+3SUcdAAAARklEQVSZfZubuS3IZlVbPrf1DV8RkRQqtW4fERFpAoW/iEgKKfxFRFJI4S8ikkIKfxGRFFL4i4ikkMJfRCSFFP4iIin0/wFhp/zkX+rIAwAAAABJRU5ErkJggg==\n", 184 | "text/plain": [ 185 | "
" 186 | ] 187 | }, 188 | "metadata": {}, 189 | "output_type": "display_data" 190 | } 191 | ], 192 | "source": [ 193 | "# plotting MI using NWJ\n", 194 | "plt.plot(plot_x, -plot_y_nwj, c = \"red\")\n", 195 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 196 | "plt.show()" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 9, 202 | "metadata": {}, 203 | "outputs": [ 204 | { 205 | "data": { 206 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VPW5P/DPk5XsIQsQdhBQEAQkAm5obbVorba3rWK9rV1cfr16a6+tve5t6W2r9rba9mpbrl3VW9eqqFhF1KpAICEhCUnIvu/rZE9meX5/nMMkIRO2THLCnM/79ZrXzPnOd855vpPJZ86cOeeMqCqIiMhegqwugIiIJh/Dn4jIhhj+REQ2xPAnIrIhhj8RkQ0x/ImIbIjhT0RkQwx/IiIbYvgTEdlQiNUFjCUpKUkXLlxodRlERKeVAwcOtKhq8vH6TdnwX7hwITIyMqwug4jotCIilSfSj5t9iIhsiOFPRGRDDH8iIhti+BMR2RDDn4jIhhj+REQ2xPAnIrIhhj8RBQRVxb6fPoHu1g6rSzktMPyJKCCU/ONDbLj/DuR/4SarSzktMPyJ6Jhcg06U7PwY6vFYXcoI2U89j9byGu90b00dACC+vBge91CtJ1K3x6OnVENrbRP2bnv+lB4LAB6XG66BwVN+/Hgw/ImmsOaSSjSXVsHlcuPdu/4LH//iD6P61JbV4NB7+wEAH912D0pSFqMm+zAqMw6N6DfYNwDXoBMA0FZVj+w/voh99/wUAFD+z/1wO10+a8g9/wosueJiZD3+xxOuu3JPJvoc3SjduRv77n3EZ5+G/BI46psBAHt+8j/I+N/jh2hncxsOPv0qOhtbsPqWLejZcAEAoOTtD9GbmQMAWFZVgKCQYJTs/BiHX3oL7pBQlO3aM+Y8u1raUR8/Ex9+/2fHfKNoKqrAQF8/1OPB3n9/EHkvvInaSz6N82/bgubyam+/9voWFH2QPnKsBaWoPpDnnXa73ACAzEs+i5Bp4QCAon98OObfYEKo6pS8rFu3Tun0Vrhzj9YfLjvpxzUWlmvWig3aVFSuBS//Q0ve+fik5/Hu1+7SXf/2wEk/TlW1ubxWc154c0Tbvh/8Qvfe94i21zRqV2uHqqo6mtv0oyuu05rswhF90/7r1/rRN7+nqqr5L72l+x76haqqdtQ3a+Zvn9Hd//J1ba+q1/Qf/0qL3/pgxGPrK2p15w9/oy6X22gAtD84VPf99zZVQBXQ3s4eHewfUFXVvfc+7G0/0v/IpT84ZMS8G2KT9OCq81VVtSJ5nrdfzp9fVgV0750P6T+/9h396KobVVV1oLdfs5ee6+13YN2lPp+vD25/QLOeflVVVUv2H9IPP/kFVUB3b7lNDy9eqQpo1vL1uueen3kf01hZrwpoTULKiLqPjMuXglfe9vbbffWNQ48JCh4x7iOXPV+6WfOWrjFqX3WBfnT+ldrb2aOqqi6nSz+671HtaGzVoh0feB+z93M3adaTf9Ws1Rfp3k9f51127lsfqRui2Ss2aNqPf60KaHXi7KHn8C9/1wGnWz0ej2au2aQK6D/veFDdLre6nS7tCotQBbT0gzSti5+pbojmnLluaLn3/MwY13W3aNrWX6nH7R7zeTgeABl6AhkrRt+pJzU1VU/5xG5v3QM05Pq3IPJpoKcXrt4+RCUnjr7znx/AFRSMkIsvHvPxzv5+qCrCIiIAAD1NLXA3NCK2vRmOlHmIqzfXqC65FADQ19EJZ1s7YhcvAAB4PG6IBEFEvPN0lJQjrrZyxOOORaFwVNRAOjsRd87ZcH70MUI9LnTOXQB1uhCamIDI/KHXU294JCI3roejsARxDTVwzJiNuOXLRowbADwXXYygjz8CALjOvwCDmQcROdBrzGNaFCL6e9ETHYvodWsBAF219YgqKUIQjP/JvtBpiHD2G2NKmoW4lgbvIrqjYhG8bCkisg5427qWLkdMccGIsfWFTYNrwSJEJE1HyF5j7bcrJh4xXaO/FO2JiEZUX7exvOQUICRk6PkH4JEgdMclIHigH67YOEQvXYz+dgei8nLggcCZeh40OxvTnAPG/CKjEexyYdpgv3ceg6nr0VffOPT3AeCYPR9xdVXG8xQUjN7Y6UB8PKLnz0Z/mwPugQFEzUzGQHoGIgb6xvgrjuYMDkGoe+SadNcZZyJmbgq6G5oQXZiPjpR5kOBgxNVUAAAUAsFQJnYvXwl3dw/iqstHPKcRw8YEAB2z5iKovR2ulBQkVJR429uTZiFkoN/7fDuDQozXVnwSYjtahp5biPfvDgCt516NxG+f2uYkETmgqqnH6zdlz+pJJ87lckFVERoaOqHL6a5rRFB1Faadtw5BQcYWw9CMdIRD4bnoYkhwEAQCt8v4hwsGEOJxjzm/rtoGxJQcBgDopksw2N2DqIKhTRXDV0wGe/ugHg+kIB+xg/3oiY7CtIR4eNLS0BeXgKgVZ6K7sgbRNZWI06GP7n0dDoRGRqCnuQ0h0VGIiotBZ1Ut0NqKyJXLERIaiq6sHMR3thtjbGhCtMeoP7bGDKjG2hF1Rw70outAFuK6HQCAsI52KBRd5dWYVluFMLOffPyx9zEhe/cgBMBgcCg8QUGI7O8BAER3O9Dn6IS7uxcxJYXe/h4Rb/ADQGh3p/FGaj6f0T2d6CyrQDgEPfMWIKa6YlTwAzBCqrgAfZUR3n/2I0HUOWM2YpvqvH2PBD8AxDXXj5hPV1wiYhytQ4HV1w001iI41NhkEQRFeIax6alr/mKgox0x5nM6XFjGfu/z412WGfyA8XqJ6WjFYHcnepubEN3TadTa149I5yD6wiPgjI5FbGuj0Z4wAwgJQcSi+ZD0dO88AHiDfzAkFGEuY3NXTGkheupq4QmfBgCIH/bmdvTzAQDTCgu88+uJikVkT9eI4O9KmIFpHW2IbzC/ezCDv2fZcgRXVmD68DfsmOmI7jKek5jVZ6MrJw8x7UNvAMPrHKwe+ZqbECfy8cCKCzf7nBhHm0MbohONj7n//oD2dXaP6jPYP6B7v/2gZq67VPc98Oio+3P+9IKWvPOxugadWneoaNT9PY4udQ46tSEuWRXQ3L+8rKqqRW+85/3Y2hIVr92h03TPv3xdFdB99z/iva/0gzTd/fmvqXNgUFVVC17fpZmrLxrxEX33NV/R3Z+7yefHd18XR3iUpl16rXe6ZPaSMfuWzVrkvb3/V38a2ixw5RZta2jVntBwb1tNQsqY80n79oNaNWxziQKadvWXT7jmjAs2q6pq+k//R49szvHVrzk6QV0Dg5q75iLtiIgZWtanvzSq7+Elq7X83d3e6azzLlMFtC8kzOe8G3MLNe0rd6hTgrT2wCFve/Fr72j65V/QtNvv08xf/VHr9mdr57Qo7/25T/9dFdD2iBhtyCsac4y9IeHaXlGraZ8f+lum/+dPNPvWu4zlxyVr3rOvasamq43n5LNf1pxV5w/Vv/oi3Xv7fUN/r6Oe3/33Payqqo17MrT0ry+OeJ06+weM1z+gB1dfpHnLU1UB3XvTt72Pb4pNHLP22sy8EdMFS1Ybf4/YJG1MmKUVb72vhQtXqAJ64IJPa+nP/0c9/f26/4ovjnhc1cz56nG5tOL//j6ivaWkQhXQ1qh443+3tkHTP7NFOyJidN8t39WW0ko9/PJbWpU0VzPPveQUE0FPeLOP5SE/1oXhf3zdbY4RwaWA7r/oqlH90v7jhyP67Ln9fs1ddq4WvPqOtpZVe9t3b7lNFVCnBGlV4hzNfuo5Hezr1+rE2br7Szdr2ayFxj/TVTeoqurea7865j9Sd+g0n+05Z6VqXfzMMR+ngOasvtBne7b5hpF21Q3aGxI+6v79w95wjnVpjE3SAxsuH9GWvvVxzTvyzx493fdjTfnPva6Zv9imXU2t6na6vAHllCA98KPHNOecC7QvJExzlp+nzv4BdfYPjNiG21RYpr0h4Zrxk19r5o9+qfsfeFRzn35FM37+e1VAHdOivX2L3vzAu/wDj/xWMzd9xhjrdx4y2r63VXvbHN4+7kGnDnT1aP5zr6sCWjx3qeY/9TfjeV2x3jvfge5eVVVN33KrFr7whs/XV09Tix647mZVQLuaWvXQn17U1rIqVVXN+NFjeuDiq0Y8P3tv/q42F1eoqmp7eY2m3/tTzf7dMzrY26cet1uzH/2tdje2qKqqx+3WAUeXd1n77vqR8Xe47HNa/LqxUpGx6WpVVa2ZPvSGXJuZd9z/i7byGu1zdGlfu0PLd7ynh/7vNVVA2yLjjDcI840x45Of16z1nxzx903f+ph3uiW/WDtKK9Xjcg39L93yPVVA913y2aHllVZp+ubrdN8d96lLgrTklbe993Xu3a/Z512mGQ/+XFVVD//1Ja3ZnTGyYI9nxOTeL3xD0zZff9xxjoXhbwN77njAZ0h9dO1NmvnU89oWEauH33hfDw37Ymn4xSlBut9cA/N1yT1znR54/A8+7ytYvEqdEqQHV1+ozYWlmv6DX4y5tunrsv++h7WtvEbTvn6n5j37quas3KgH127Sotd2GoMb1vfgI09q0Z+eV4/brbl/eVn7u3s0fevjWjz/TM1/9lXdd8WXNOOCzepxu7Xi9Z3qFhnx+Obo6VqTkKK5K9Yby/7eVi18beeIPlW7M9RR06B7//V2zfrNn7UtMk4V0LSv3KEKaObaTcf8W7SXVmlz/uhPTWPx9Qmtr6PTCNEvf8vb5uzr99bYXl6jzr5+bTtcoqqqg41N3jeVvN8+ra1FQ1+u9zm6NGvtJi17b6+qqmZv+z/tNIP3pHg8qk6nz7vKXzeew5zLP69p391q1HqKelva9eAln9G2kgpVj0dzfveM8XyoavbKjaqANuQUHmcuvrWUVakCWjT/LFVVbco5rBn/fr+6zE+iRa++o/nPvOLt39fYrGX/+4zPeXXUNemhZedq8Rvv+7x/PM+BvzD8A0h7Vb2WzFmq6Q8/6W1Lu2qLKjBis0BneOSokM1ac7E6wqNGtWf87Ikxg7ngb9u1Pn6GtkfEaPHsM0aG9mduMILQnN5z693emvJfeFMV0LKURZp+0Wc0/cIr9eATf/X2Ldu1Ww+fsUq7wyK8a4hjKUtZ7H1c/Un+0/d3dmn+tmd134P/rVVpWVq4/V0tePkt7W5q1cwf/VKdvX2qqnpw01XaGD9Dcx55ctQ8sr/zoPGmsOtj7evsVucx9kLxp752h7qdrhFtNbt2a869P5mU5Z8Uj0erf/9ndbZ3TOhiWsuqNeePz49rHhlbH9fGvGI/VTS1MfxPQyXv7tGiHR/o4dd3aUtUvNbHJmnmU89r2qbPqsLYpa303d2674e/9AZj0eu79OCTRsBWJ87xbhYY65Jz9gZNv/BKVVWtmLlAFdBDy4Z259t33S2qqpp2292jHnsw9RM6YK6xHtmsk/v0KyPGkPnz32nVngPe6YGePs1ac7GxGUJV1eNR9wkEaWdDs1anZWn273yvgfmF263qcvm+z+PRrtzjb2IgmmoY/qcRj9utuy8Z+vIy66gvQxXQlqO2Qw8Eh2htVr6qqvERFNDK5HnqGRzUlvziMcPfUVXvXW7+M69o2r/dq/1d3Vr5Ubp2NTSre9D4iN9SUKJZl12r+/7f971vBAef/Kv3sQd/82etTZytPW0Tu9ZHRCeH4T8F1RRV6uG92d7p/X/bodmzlujeJ58dFdJ7P3eTlr70puavuUAzb7lLO8qrvXtIHNh4ufEln6m5qMx4zJduHlrYke3EFbW6/z9/oiULl6sCp3TwiNvp0oIXd4xr7EQ0OU40/P1ykJeIbAbwKxi7dj+lqg/76HMdgB8CUADZqvrlY81zXAd5TUEetwdBIcHGhPmcFyxZg+Wl2T77Zz3+B6y98xuj2h2VtYidlwIJGnlmjsbcIiSdtRjBocbe3CUvv4WwxATMv3QDAKCvpR3dlTVIXrfKX0Mioilo0g7yEpFgAE8AuBxADYB0EdmuqvnD+iwFcC+AC1W1XURmjHe5pwtHdz/ioqch5+m/Y43Z9t5//BgzXv4b4rvHPvXsvM2X+myPWzDHZ/vMVctGTC/5wpUjpiOSpiMiafoJ101Egc0fR/iuB1CiqmUAICLPAbgWQP6wPrcAeEJV2wFAVZv8sNwp7c37fomLfvkQ4gZ6ULRzD/qff8l732WPPzSqf/W+bLTedgc8118Pd2U1zjtz8WSWS0Q244/wnwOgeth0DYANR/VZBgAishvGpqEfquo//LDsKamj1YHP/Oy73un4z1+NxT0dKJ27FGfUFAMACpecgzNLctAeGYfmZ1/AsvXnYF7Wh1aVTEQ244/wFx9tR3+REAJgKYBLAcwF8JGIrFTVEds9RORWALcCwPz58/1QmjWKt+/CecOmZ3S3AQA6774X+9P2I7ixAStffRr7//VbiP/GV7Hs2sutKZSIbMsf4V8DYN6w6bkA6nz0SVNVJ4ByESmE8WYw4qTXqroNwDbA+MLXD7VZouvjkecOz7loM5weYPVtNyLk21/3tq9/7enJLo2ICIB/wj8dwFIRWQSgFsAWAEfvyfMqgBsA/FlEkmBsBirzw7KnlII6B/b+6e/Y9NrzaExMwcCnrkDoFZfjnG/caHVpREQjjDv8VdUlIncAeBvG9vw/qmqeiGyFsb/pdvO+K0QkH4AbwN2q2jreZU8l6a//E+ddcymWm9MN338Q8x/ZamlNRERj8cv5/FV1B4AdR7U9NOy2ArjLvAQk58OPem/3PPscZn35egurISI6Nv6G7zipKt67+2e4YM8OuIJDUPflryGKwU9EUxzDf5zKd2fhsv++DwBw6L6fYPazf7K4IiKi42P4j1PD0895b8/efJmFlRARnTj+hu84TX/3H2iJS4LznvuQcv46q8shIjohXPM/RX29A8i8eyuWl+Wi5NobkHLPfwDi63g3IqKph2v+pyjjrh/g4t8/graoeMz/1uizbxIRTWUM/1MU++F7qE2ei5T6SgQF8wMUEZ1emFqnoL9vAEtKclG//mIGPxGdlphcJ6nf6cZvfvEiopz9CL/sUqvLISI6JQz/k/TOqx/h7ge/AgBY8LnNFldDRHRqGP4nKXzn297bsYtP39NOE5G9MfxPUmh+HgDAVVhkcSVERKeO4X8S2noGMaOsAKWrNyJk2VKryyEiOmUM/5Pw+lsZWN5QhvjLNlldChHRuDD8T0L0Ky8iWD1I/NbNVpdCRDQuDP+TEFeQi9bEWcBSbvIhotMbw/8E7S9vw6y6CnQuXGJ1KURE48bwP0FbX8vFGa01CF91ttWlEBGNG8/tcwL6nW4k7vsYEa4BRFyYanU5RETjxjX/E5BX14l/+/hv6J27ANiyxepyiIjGjeF/AnYXN2N5czlk82YgOtrqcoiIxo2bfU7Awd05iB3oBdatsboUIiK/4Jr/cTjdHgQdyjUmVq2ythgiIj9h+B9HfUc/zqkthCc4GFi92upyiIj8guF/HDXtvTi37jB6zzqb2/uJKGAw/I9BVfHbD0qwur4IumGj1eUQEfkNv/A9htxaB3JyKxAz2Af32WdZXQ4Rkd9wzf8YSpu7MaezGQAQvHCBxdUQEfkPw/8Yylt6MafLCH/M5692EVHgYPgfQ3lLD852dRgTDH8iCiAM/zEMujzILGvB5sI9wLRpQHKy1SUREfkNw38Mb+TUIbHoEM46fAC4805AxOqSiIj8xi/hLyKbRaRQREpE5J5j9PuiiKiITPlTY35Y1Iyz+1uMiRtvtLYYIiI/G3f4i0gwgCcAXAlgBYAbRGSFj34xAL4NYN94lznRVBV7SltxflCX0bBokbUFERH5mT/W/NcDKFHVMlUdBPAcgGt99PsxgEcB9PthmROquXsATV0DOLO3BUhK4pG9RBRw/BH+cwBUD5uuMdu8RGQtgHmq+saxZiQit4pIhohkNDc3+6G0U1PS2A0AmNFaDyxcaFkdREQTxR/h7+ubUPXeKRIE4DEA3z3ejFR1m6qmqmpqsoV71xQ3dUPUg7j8HGDlSsvqICKaKP4I/xoA84ZNzwVQN2w6BsBKAB+ISAWAjQC2T+UvfQvqO7HOUYOg9jbgE5+wuhwiIr/zR/inA1gqIotEJAzAFgDbj9ypqg5VTVLVhaq6EEAagGtUNcMPy/Y7t0fxbkETvthfYTRccoml9RARTYRxh7+qugDcAeBtAAUAXlDVPBHZKiLXjHf+ky27pgMt3QM4v7sWSEjgkb1EFJD8clZPVd0BYMdRbQ+N0fdSfyxzomRXG6dzSKkqNn65iwd3EVEA4hG+R8mtdSA5OgxhBfn82UYiClgM/6Pk13XioshBoLsbWL7c6nKIiCYEw38YVUVFaw/WDLYaDWecYW1BREQThOE/THP3APqdHizpajQaliyxtiAiognC8B+mqrUXADCntRYIDuaePkQUsBj+w1S1GeGfXFlibPIJDbW4IiKiicHwHyanxoFpIYLIzHRg40aryyEimjAM/2HeL2zCtbGDkKYm4PzzrS6HiGjCMPxN+XWdqGztxZUe88ve1Cl76iEionFj+JteyapBaLBgQ3edcVTvilG/R0NEFDAY/qa8uk6snBOHiKICYPFiIDLS6pKIiCYMw99U29GHudMjgfx84OyzrS6HiGhCMfwBeDyKuo4+zImbBpSX88heIgp4DH8ATV0DcLoVi6UP6O3lTzcSUcBj+AOo7TAO7lrcbf5uMMOfiAIcwx9AaVMPAGBeZ5PRwPAnogDH8AeQX9+JiNBgJNeWGw0MfyIKcAx/GD/YfuasGATt3m38gEtsrNUlERFNKNuHv6ricEMXzp4RCezZA2zaZHVJREQTzvbhX+/oh6PPiQu6qo1f72L4E5EN2D78C+o7AQCrynKMhosvtrAaIqLJwfA3wz+l4KBxWoeUFIsrIiKaeLYP/+waBxYnRSG0sgI480yryyEimhS2Dn9VxcHqDqyZFw9UVHAXTyKyDVuHf1PXAJq7BnBuQjDQ2srwJyLbsHX4lzZ3AwDOHmw3Ghj+RGQTtg7/8hbjtA7z2+uMhkWLLKyGiGjy2Dr8K1p6EB4ShOklh41f7+J5/InIJmwd/uUtPViYGIWgnBxg6VL+ehcR2Ybtw39RUhSQkwOsXm11OUREk8a24e9ye1DV1oszowCUlgLnnGN1SUREk8a24V/b0QenW3FOe7XRwDV/IrIRv4S/iGwWkUIRKRGRe3zcf5eI5ItIjojsEpEF/ljueBzZ02dxfanRwDV/IrKRcYe/iAQDeALAlQBWALhBRFYc1S0LQKqqngPgJQCPjne541XY0AUAmF2SByQkAPPnW1wREdHk8cea/3oAJapapqqDAJ4DcO3wDqr6vqr2mpNpAOb6Ybnjkl/fiZS4aQjP2A9s3Gjs6klEZBP+CP85AKqHTdeYbWP5JoC3fN0hIreKSIaIZDQ3N/uhtLEV1Hfi3PggID8f2LBhQpdFRDTV+CP8fa0yq8+OIv8KIBXAz33dr6rbVDVVVVOTk5P9UJpvAy43Spt7sNHZDKgCa9ZM2LKIiKaiED/MowbAvGHTcwHUHd1JRD4F4H4Al6jqgB+We8pKmrrh9ihWdjUYDcuWWVkOEdGk88eafzqApSKySETCAGwBsH14BxFZC+D3AK5R1SY/LHNcDtcbX/YuaK0FgoONH3EhIrKRcYe/qroA3AHgbQAFAF5Q1TwR2Soi15jdfg4gGsCLInJQRLaPMbtJcbihE2EhQYivKTdO5hYWZmU5RESTzh+bfaCqOwDsOKrtoWG3P+WP5fjL4YYuLJsZjaA3irjJh4hsyZZH+B5u6MLyGdFAcTHDn4hsyXbh39Jt/HrX2pBeoLeXv9tLRLZku/A/cmTvOT2NRgPX/InIhmwX/gX1nQCARY0VRsPy5dYVQ0RkEduF/+GGLiTHhCOqMB9ISgJmzbK6JCKiSWe78M+v68RZs2KA3FzjTJ48pw8R2ZCtwr9v0I3Cxi6snhMH5OUBK1daXRIRkSVsFf55dQ64PYoN4f1ATw9w1llWl0REZAlbhX92jQMAsLLX3NNn6VILqyEiso6twj+v1oGZseGYXltpNHA3TyKyKVuFf26tAytnxwFFRcC0acBcy39ThojIErYJ/95BF0qbu7FyTpxxWoclS4Ag2wyfiGgE26RfQX0nPAoj/It4QjcisjfbhP+hWuPI3lUzo4DSUn7ZS0S2Zpvwz67pQFJ0GGZ2NAJOJ9f8icjWbBP+ByrbsW7BdEhxsdHANX8isjFbhH9TVz8qW3uRuiDB+LIX4Jo/EdmaLcL/QEU7AGDdwunGl72xscCMGRZXRURkHVuEf0ZlO8JDgob28V+6lCd0IyJbs034r54Xj7CQIP50IxERbBD+fYNu5NU6kLpgOjAwAFRW8steIrK9gA//g9UdcHkUqQunA2VlgMfDNX8isr2AD/8DlW0AgHXzE4zt/QDDn4hsL+DDP72iHctmRiMuMhQoKTEazzjD2qKIiCwW0OHv8Sgyq9qxbkGC0VBWBsTHAwkJ1hZGRGSxgA7/oqYudPW7jC97ASP8udZPRBTY4Z9uHtx13kJzTb+0FFi82MKKiIimhoAO/wMVbUiOCce8hAjA7QYqKhj+REQI8PDPqu7AufPjISJAba1xNk+GPxFR4IZ/a/cAKlt7sXa+ub2/tNS45jZ/IqLADf+D1R0AgLXz4o2GsjLjmmv+RET+CX8R2SwihSJSIiL3+Lg/XESeN+/fJyIL/bHcY8mq6kBwkGDV3DijoawMCAkB5s2b6EUTEU154w5/EQkG8ASAKwGsAHCDiKw4qts3AbSr6hIAjwF4ZLzLPZ6s6nacNSsGkWEhRkNpKbBggfEGQERkc/5Y818PoERVy1R1EMBzAK49qs+1AP5i3n4JwCdFJu6cym6PIrvagbXz44cay8q4yYeIyOSP8J8DoHrYdI3Z5rOPqroAOAAk+mHZPpU0daN7wIW186YPNTL8iYi8/BH+vtbg9RT6QERuFZEMEclobm4+5YKyqoyDu7xr/g4H0NrK8CciMvkj/GsADP8WdS6AurH6iEgIgDgAbUfPSFW3qWqqqqYmJyefckGZVe2IjwzFoqQoo6G83Lhm+BMRAfBP+KcDWCoii0QkDMAWANuP6rMdwE3m7S8CeE9VR635+0tWVQfWzjNJfMHtAAAINUlEQVQP7gKMA7wAYO7ciVokEdFpZdzhb27DvwPA2wAKALygqnkislVErjG7/QFAooiUALgLwKjdQf3F0edEcVP30MFdAFBfb1ynpEzUYomITit+2e9RVXcA2HFU20PDbvcD+JI/lnUCteDuT5+JS5YN22x0JPxnzZqMEoiIpryA2+k9PjIMt39iycjG+nogMREID7emKCKiKSZgT+8wQl0dN/kQEQ1jj/Cvr2f4ExENw/AnIrKhwA9/VaChgeFPRDRM4Id/a6vxIy6zZ1tdCRHRlBH44c99/ImIRgn88K8zzzTB8Cci8gr88OeaPxHRKAx/IiIbskf4x8UBkZFWV0JENGXYI/y51k9ENALDn4jIhgI//HleHyKiUQI7/FW55k9E5ENgh39nJ9Dfz/AnIjpKYId/U5NxPXOmtXUQEU0x9gj/GTOsrYOIaIoJ7PBvbDSuGf5ERCMEdvhzzZ+IyCd7hH9y8rH7ERHZTGCHf2MjkJAAhIZaXQkR0ZQS2OHf1MRNPkREPgR++HM3TyKiUQI7/BsbueZPRORDYIc/N/sQEfkUuOE/OAi0tzP8iYh8CNzwb242rrnNn4holMANfx7gRUQ0psAN/yNr/jzAi4holMAN/5YW4zox0do6iIimoMAN/9ZW45rhT0Q0yrjCX0QSRGSniBSb19N99FkjIntFJE9EckTk+vEs84QdCf+EhElZHBHR6WS8a/73ANilqksB7DKnj9YL4KuqejaAzQAeF5H4cS73+Fpbgbg4ICRkwhdFRHS6GW/4XwvgL+btvwD43NEdVLVIVYvN23UAmgBM/Lewra3c5ENENIbxhv9MVa0HAPP6mPtVish6AGEASse53ONj+BMRjem420RE5F0As3zcdf/JLEhEUgA8DeAmVfWM0edWALcCwPz5809m9qO1tnI3TyKiMRw3/FX1U2PdJyKNIpKiqvVmuDeN0S8WwJsAHlDVtGMsaxuAbQCQmpqqx6vtmFpbgeXLxzULIqJANd7NPtsB3GTevgnAa0d3EJEwAK8A+KuqvjjO5Z04bvYhIhrTeMP/YQCXi0gxgMvNaYhIqog8Zfa5DsAmAF8TkYPmZc04l3tsg4NAVxfDn4hoDOPaD1JVWwF80kd7BoCbzdvPAHhmPMs5aW1txjXDn4jIp8A8wpdH9xIRHRPDn4jIhhj+REQ2xPAnIrIhhj8RkQ0FbviHhwORkVZXQkQ0JQVu+CcmAiJWV0JENCUFdvgTEZFPDH8iIhti+BMR2RDDn4jIhgIv/FWNc/sw/ImIxhR44d/ZCbhcDH8iomMIvPB3uYDrrwdWrbK6EiKiKWtcp3SekhITgeees7oKIqIpLfDW/ImI6LgY/kRENsTwJyKyIYY/EZENMfyJiGyI4U9EZEMMfyIiG2L4ExHZkKiq1TX4JCLNACrHMYskAC1+Kud0wTHbA8dsD6c65gWqmny8TlM2/MdLRDJUNdXqOiYTx2wPHLM9TPSYudmHiMiGGP5ERDYUyOG/zeoCLMAx2wPHbA8TOuaA3eZPRERjC+Q1fyIiGkPAhb+IbBaRQhEpEZF7rK7HX0TkjyLSJCKHhrUliMhOESk2r6eb7SIivzafgxwROde6yk+diMwTkfdFpEBE8kTkTrM9YMctItNEZL+IZJtj/pHZvkhE9pljfl5Ewsz2cHO6xLx/oZX1j4eIBItIloi8YU4H9JhFpEJEckXkoIhkmG2T9toOqPAXkWAATwC4EsAKADeIyAprq/KbPwPYfFTbPQB2qepSALvMacAY/1LzciuA305Sjf7mAvBdVV0OYCOA282/ZyCPewDAZaq6GsAaAJtFZCOARwA8Zo65HcA3zf7fBNCuqksAPGb2O13dCaBg2LQdxvwJVV0zbJfOyXttq2rAXACcD+DtYdP3ArjX6rr8OL6FAA4Nmy4EkGLeTgFQaN7+PYAbfPU7nS8AXgNwuV3GDSASQCaADTAO9gkx272vcwBvAzjfvB1i9hOraz+Fsc41w+4yAG8AEBuMuQJA0lFtk/baDqg1fwBzAFQPm64x2wLVTFWtBwDzeobZHnDPg/nRfi2AfQjwcZubPw4CaAKwE0ApgA5VdZldho/LO2bzfgeAxMmt2C8eB/B9AB5zOhGBP2YF8I6IHBCRW822SXttB9pv+IqPNjvuzhRQz4OIRAN4GcB3VLVTxNfwjK4+2k67cauqG8AaEYkH8AqA5b66mden/ZhF5GoATap6QEQuPdLso2vAjNl0oarWicgMADtF5PAx+vp9zIG25l8DYN6w6bkA6iyqZTI0ikgKAJjXTWZ7wDwPIhIKI/ifVdW/m80BP24AUNUOAB/A+L4jXkSOrKwNH5d3zOb9cQDaJrfScbsQwDUiUgHgORibfh5HYI8ZqlpnXjfBeJNfj0l8bQda+KcDWGruJRAGYAuA7RbXNJG2A7jJvH0TjG3iR9q/au4hsBGA48hHydOJGKv4fwBQoKq/HHZXwI5bRJLNNX6ISASAT8H4EvR9AF80ux095iPPxRcBvKfmRuHThareq6pzVXUhjP/Z91T1RgTwmEUkSkRijtwGcAWAQ5jM17bVX3pMwJcoVwEogrGd9H6r6/HjuP4GoB6AE8ZawDdhbOfcBaDYvE4w+wqMvZ5KAeQCSLW6/lMc80UwPtrmADhoXq4K5HEDOAdAljnmQwAeMtsXA9gPoATAiwDCzfZp5nSJef9iq8cwzvFfCuCNQB+zObZs85J3JKsm87XNI3yJiGwo0Db7EBHRCWD4ExHZEMOfiMiGGP5ERDbE8CcisiGGPxGRDTH8iYhsiOFPRGRD/x948FA0G0RkHQAAAABJRU5ErkJggg==\n", 207 | "text/plain": [ 208 | "
" 209 | ] 210 | }, 211 | "metadata": {}, 212 | "output_type": "display_data" 213 | } 214 | ], 215 | "source": [ 216 | "# plotting both together\n", 217 | "plt.plot(plot_x, -plot_y_mine)\n", 218 | "plt.plot(plot_x, -plot_y_nwj, c = \"red\")\n", 219 | "plt.plot(plot_x, mi*np.ones(len(plot_loss_mine)))\n", 220 | "plt.show()" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "Again we see the consistency in the measurement of mutual information using these two methods." 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": null, 233 | "metadata": {}, 234 | "outputs": [], 235 | "source": [] 236 | } 237 | ], 238 | "metadata": { 239 | "kernelspec": { 240 | "display_name": "Python 3", 241 | "language": "python", 242 | "name": "python3" 243 | }, 244 | "language_info": { 245 | "codemirror_mode": { 246 | "name": "ipython", 247 | "version": 3 248 | }, 249 | "file_extension": ".py", 250 | "mimetype": "text/x-python", 251 | "name": "python", 252 | "nbconvert_exporter": "python", 253 | "pygments_lexer": "ipython3", 254 | "version": "3.6.5" 255 | } 256 | }, 257 | "nbformat": 4, 258 | "nbformat_minor": 2 259 | } 260 | --------------------------------------------------------------------------------