├── README.md ├── code_for_plots ├── Robustness_and_Privacy.ipynb └── print_privacy_leakage.ipynb ├── diagrams └── FbFTL_diagram.png ├── history.txt ├── main_GLOBECOM_CIFAR10_VGG16.py ├── main_TMLCN_CIFAR10_VGG16.py └── main_TMLCN_FLANT5_SAMSUM.py /README.md: -------------------------------------------------------------------------------- 1 | # FbFTL: Communication-Efficient Feature-based Federated Transfer Learning 2 | 3 | This is the offical implementation for Python simulation of Feature-based Federated Transfer Learning (FbFTL), from the following conference paper and journal paper: 4 | 5 | Communication-Efficient Feature-based Federated Transfer Learning.([Globecom2022](https://ieeexplore.ieee.org/abstract/document/10000612), [arXiv](https://arxiv.org/abs/2209.05395)) 6 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar 7 | Department of Electrical Engineering and Computer Science, Syracuse University 8 | 9 | Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy.([TMLCN](https://ieeexplore.ieee.org/abstract/document/10542971), [arXiv](https://arxiv.org/abs/2405.09014)) 10 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar 11 | Department of Electrical Engineering and Computer Science, Syracuse University 12 | 13 | --- 14 | 15 | 16 | 17 | We propose the FbFTL as an innovative federated learning approach that upload features and outputs instead of gradients to reduce the uplink payload by more than five orders of magnitude. Please refer to the journal paper for explicit explaination on learning structure, system design, robustness analysis, and privacy analysis. 18 | 19 | 20 | # Results on CIFAR-10 Dataset with VGG16 Model 21 | In the following table, we provide comparison between federated learning with [FedAvg](http://proceedings.mlr.press/v54/mcmahan17a.html) (FL), federated transfer learning with FedAvg that updating full model (FTLf), federated transfer learning with FedAvg that updating task-specific sub-model(FTLc), and FbFTL. All of them learn [VGG16](https://arxiv.org/abs/1409.1556) model on [CIFAR-10](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf) dataset. For transfer learning approaches, the source models are trained on [ImageNet](https://ieeexplore.ieee.org/abstract/document/5206848?casa_token=QncCRBM1tzAAAAAA:QuoJhjJAHRplmLJ4jcFw5JWdfASjmbIVlvpCrHgTPIFu63gpSUlBeACB78S0AH34qqQnsBOdoQ) dataset. Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude. 22 | 23 | | | FL | FTLf | FTLc | FbFTL | 24 | | ---- | ----- | ---- | ---- | ---- | 25 | | upload batches | 656250 | 193750 | 525000 | 50000 | 26 | | upload parameters per batch | 153144650 | 153144650 | 35665418 | 4096 | 27 | | uplink payload per batch | **4.9 Gb** | **4.9 Gb** | **1.1 Gb** | **131 Kb** | 28 | | total uplink payload | **3216 Tb** | **949 Tb** | **599 Tb** | **6.6 Gb** | 29 | | total downlink payload | 402 Tb | 253 Tb | 322 Tb | 3.8 Gb | 30 | | test accuracy | 89.42\% | 93.75\% | 86.51\% | 86.51\% | 31 | 32 | # Results on SAMSum summary task with FLAN-T5-small language model 33 | In the following table, we consider [FLAN-T5-small](https://www.jmlr.org/papers/volume25/23-0870/23-0870.pdf) as a pre-trained language model, and fine-tune on [SAMSum](https://www.aclweb.org/anthology/D19-5409) summary task. As a fine-tuning task, this experiment does not include an FL setting, and we provide comparison between federated transfer learning with FedAvg that updating full model (FTLf), federated transfer learning with FedAvg that updating task-specific sub-model(FTLc), and FbFTL. Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude. 34 | 35 | | | FTLf | FTLc | FbFTL | FTLc | FbFTL | FTLc | FbFTL | 36 | | ---- | ----- | ---- | ---- | ---- | ---- | ---- | ---- | 37 | | number of trained encoders | 8 | 8 | 8 | 4 | 4 | 2 | 2 | 38 | | number of upload batches | 132588 | 36830 | 7366 | 88392 | 7366 | 103124 | 7366 | 39 | | upload parameters per batch | 109860224 | 60511616 | 1024 | 51070144 | 1024 | 46349504 | 1024 | 40 | | uplink payload per batch | **3.5 Gb** | **1.9 Gb** | **32.7 Kb** | **1.6 Gb** | **32.7 Kb** | **1.5 Gb** | **32.7 Kb** | 41 | | total uplink payload | **466.1 Tb** | **71.3 Tb** | **241.4 Mb** | **144.5 Tb** | **241.4 Mb** | **152.9 Tb** | **241.4 Mb** | 42 | | total downlink payload | 116.0 Tb | 32.2 Tb | 1.58 Gb | 77.3 Tb | 1.88 Gb | 90.2 Tb | 2.03 Gb | 43 | | test ROUGE-1 | 45.9249 | 45.4680 | 45.4680 | 45.2827 | 45.2827 | 44.9862 | 44.9862 | 44 | 45 | # Required packages installation 46 | We use python==3.6.9, numpy==1.19.5, torch==1.4.0, torchvision==0.5.0, and CUDA version 11.6 for the experiments on CIFAR-10 with VGG16. The dataset and the source model will be automatically downloaded. 47 | 48 | Additionally, for the experiments on SAMSUM with FLAN-T5, we use transformers==4.30.2, torchinfo==1.8.0, datasets==2.13.2, nltk==3.8.1, evaluate==0.4.1, huggingface_hub==0.16.4 49 | 50 | # Citation 51 | If you find our work useful in your research, please consider citing: 52 | ``` 53 | @inproceedings{wang2022communication, 54 | title={Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learning}, 55 | author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem}, 56 | booktitle={GLOBECOM 2022-2022 IEEE Global Communications Conference}, 57 | pages={3875--3880}, 58 | year={2022}, 59 | organization={IEEE} 60 | } 61 | ``` 62 | 63 | ``` 64 | @article{wang2024feature, 65 | title={Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy}, 66 | author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem}, 67 | journal={IEEE Transactions on Machine Learning in Communications and Networking}, 68 | year={2024}, 69 | publisher={IEEE} 70 | } 71 | ``` 72 | -------------------------------------------------------------------------------- /code_for_plots/print_privacy_leakage.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "source": [ 20 | "# CIFAR-10" 21 | ], 22 | "metadata": { 23 | "id": "RnC64R4j-se4" 24 | } 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": { 30 | "id": "Ogn9Z9JF0EHG" 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "import numpy as np\n", 35 | "import matplotlib.pyplot as plt\n", 36 | "import math\n", 37 | "from math import factorial as f\n", 38 | "from time import sleep\n", 39 | "import itertools\n", 40 | "import time" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "source": [ 46 | "K = 8\n", 47 | "N = 10\n", 48 | "B = f(N+K-1) // f(K) // f(N-1)\n", 49 | "\n", 50 | "# calculate H(B|Y=y), y is uniform\n", 51 | "p_y = 0\n", 52 | "P_ys = []\n", 53 | "b_count = 0\n", 54 | "H_y = 0\n", 55 | "\n", 56 | "for c0 in range(0, N+K-1):\n", 57 | " for c1 in range(c0+1, N+K-1):\n", 58 | " for c2 in range(c1+1, N+K-1):\n", 59 | " for c3 in range(c2+1, N+K-1):\n", 60 | " for c4 in range(c3+1, N+K-1):\n", 61 | " for c5 in range(c4+1, N+K-1):\n", 62 | " for c6 in range(c5+1, N+K-1):\n", 63 | " for c7 in range(c6+1, N+K-1):\n", 64 | " b_count += 1\n", 65 | " Ns = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n", 66 | " Ns[c0-0] += 1\n", 67 | " Ns[c1-1] += 1\n", 68 | " Ns[c2-2] += 1\n", 69 | " Ns[c3-3] += 1\n", 70 | " Ns[c4-4] += 1\n", 71 | " Ns[c5-5] += 1\n", 72 | " Ns[c6-6] += 1\n", 73 | " Ns[c7-7] += 1\n", 74 | " p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5])*f(Ns[6])*f(Ns[7])*f(Ns[8])*f(Ns[9]))\n", 75 | " P_ys.append(p_y)\n", 76 | " H_y += -p_y * math.log2(p_y)\n", 77 | "\n", 78 | "print(len(P_ys))\n", 79 | "print(sum(P_ys))\n", 80 | "print(b_count)\n", 81 | "print('B', B)\n", 82 | "print(math.log2(B))\n", 83 | "print('H_y', H_y)\n", 84 | "print(2**H_y)" 85 | ], 86 | "metadata": { 87 | "colab": { 88 | "base_uri": "https://localhost:8080/" 89 | }, 90 | "id": "7qZfigBQGzVQ", 91 | "outputId": "1d0a0bbf-fdb0-4fc7-aee6-2248cb12cd81" 92 | }, 93 | "execution_count": null, 94 | "outputs": [ 95 | { 96 | "output_type": "stream", 97 | "name": "stdout", 98 | "text": [ 99 | "24310\n", 100 | "1.0000000000001252\n", 101 | "24310\n", 102 | "B 24310\n", 103 | "14.569262272916092\n", 104 | "H_y 13.860045564098357\n", 105 | "14869.263443691536\n" 106 | ] 107 | } 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "source": [ 113 | "# calculate H(B|S=s)\n", 114 | "U_range = np.arange(1, 6252, 1250)\n", 115 | "# U_range = np.arange(1, B, 3000)\n", 116 | "H_ss = np.zeros_like(U_range).astype(float)\n", 117 | "\n", 118 | "for i, U in enumerate(U_range):\n", 119 | " H_s = 0\n", 120 | " p_s = 0\n", 121 | " choices = np.random.choice(B, U, p=P_ys)\n", 122 | " # print(len(choices), choices)\n", 123 | " for u in range(U):\n", 124 | " p_s = sum([1 for c in choices if c == u]) / U\n", 125 | " # print(p_s)\n", 126 | " if p_s != 0:\n", 127 | " H_s += -p_s * math.log2(p_s)\n", 128 | " print('U:', U, 'H_s:', H_s)\n", 129 | " H_ss[i] = H_s\n", 130 | "\n", 131 | "print('H_ss', H_ss)" 132 | ], 133 | "metadata": { 134 | "colab": { 135 | "base_uri": "https://localhost:8080/" 136 | }, 137 | "id": "ePqcA5Ua7aZk", 138 | "outputId": "7f86a441-d4ff-4127-8e16-58b339d148b9" 139 | }, 140 | "execution_count": null, 141 | "outputs": [ 142 | { 143 | "output_type": "stream", 144 | "name": "stdout", 145 | "text": [ 146 | "U: 1 H_s: 0\n", 147 | "U: 1251 H_s: 0.19578959694642661\n", 148 | "U: 2501 H_s: 0.527494304228384\n", 149 | "U: 3751 H_s: 1.6176505965077759\n", 150 | "U: 5001 H_s: 2.240630027446816\n", 151 | "U: 6251 H_s: 3.09074958685801\n", 152 | "H_ss [0. 0.1957896 0.5274943 1.6176506 2.24063003 3.09074959]\n" 153 | ] 154 | } 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "source": [ 160 | "H_ss = np.array([ 0., 1.14445591, 2.92142532, 5.66463935, 7.32508148, 8.84656221, 10.8259317, 12.2532801, 13.15547667])\n", 161 | "H_y = 13.860045564098357\n", 162 | "plt.plot(np.arange(9)*6250/9, H_y-H_ss)\n", 163 | "# H_ss = np.array([ 0., 0.1891638, 0.78579461, 1.54249295, 2.22972449, 3.19595329])\n", 164 | "# H_y = 13.860045564098357\n", 165 | "# plt.plot(np.arange(6)*6250/6, H_y-H_ss)\n", 166 | "plt.xlabel('Number of clients')\n", 167 | "plt.ylabel('Privacy leakage (in bits)')\n", 168 | "plt.show()" 169 | ], 170 | "metadata": { 171 | "colab": { 172 | "base_uri": "https://localhost:8080/", 173 | "height": 279 174 | }, 175 | "id": "bbZ_N7hK87yL", 176 | "outputId": "b63f85ad-1ffd-4ae2-ab25-3659484863df" 177 | }, 178 | "execution_count": null, 179 | "outputs": [ 180 | { 181 | "output_type": "display_data", 182 | "data": { 183 | "text/plain": [ 184 | "
" 185 | ], 186 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gVVbb38e/qRM45Z1BEQIIiSQwoKgrGMaOiOGMA9DqjvuPMHZ2Z66hjGjOCiuFiFhVFxVGiiuSMCEhqsuRMw3r/ONXelrHh0N2n64Tf53nq6Tp16lStDYdF9a5da5u7IyIiqSMt7ABERKR4KfGLiKQYJX4RkRSjxC8ikmKU+EVEUkxG2AFEo2rVqt6wYcOwwxARSSjTpk3b6O7VDt2eEIm/YcOGTJ06NewwREQSipkt/7Xt6uoREUkxSvwiIilGiV9EJMXELPGb2Ytmtt7M5v7Ke/9lZm5mVWN1fhER+XWxvOJ/Geh16EYzqwecCayI4blFRCQfMUv87j4e2PQrbz0G/AFQdTgRkRAUax+/mfUBst19VhT7DjCzqWY2dcOGDcUQnYhIaii2xG9mpYH/B/w5mv3dfYi7d3D3DtWq/cfzB1EZv2gDL078kZ17cwr0eRGRZFScV/xNgEbALDNbBtQFpptZzVid8IsF67h/1HxOfuDfPPjpQtZt2xOrU4mIJAyL5UQsZtYQGOXurX7lvWVAB3ffeKTjdOjQwQv65O70FZsZOmEpn85dS3qacX6bOtzYvRHH1CxfoOOJiCQKM5vm7h0O3R6zkg1mNgLoAVQ1s1XAf7v7sFidLz/t6lfimSvbs+KnXbw46UfenLKSd6evonvzagzo1pguTatgZsUdlohIaGJ6xV9UCnPFf6gtu/bx+uQVvDRpGRt37OWYmuUY0L0xvVvXJitDz7OJSPLI74o/5RJ/rr05B/hg5mpeGL+UH9bvoEb5ElzXpRGXn1ifCqUyi/RcIiJhUOLPh7szbtEGXpiwlEmLf6JMVjqXnVif67o0pG6l0jE5p4hIcVDij8Lc7K0MnbCUj2avAeCc42sxoFtjjq9bIebnFhEpakr8R2H1lt28/PUy/nfyCnbszaFT48rc2K0xp7aoTlqabgSLSGJQ4i+A7Xv28+aUlbw48UdWb91Dk2pluLFbY/qeUIeSmenFHo+IyNFQ4i+E/QcO8smcNQwZv5R5q7dRtWwW15zckKs6NaBymazQ4hIRORwl/iLg7nyz9CdeGL+Ur77fQMnMNC5pX4/+XRvRsGqZsMMTEfmFYn+AKxmZGZ2bVKVzk6osWredoROW8uaUlbw2eTlntqzBgO6Nad+gcthhiogclq74C2n99j288vVyXv12OVt376dd/YoM6N6Yni1rkq4bwSISInX1xNiufTm8M20VQyf8yIpNu2hQpTT9uzbi4vZ1KZ2lX6xEpPgp8ReTAwedz+etZciEpcxYsYWKpTO56qQGXNO5AdXLlQw7PBFJIUr8IZi2fBNDxi/l8/nryExL44IT6nBDt0Y0q1Eu7NBEJAUo8Yfox407eXHij7w9bSV79h/k1BbV+F2PppzYSDeCRSR2lPjjwKad+3jt2+UM/3oZP+3cR7+TG3DPOcfqYTARiYn8Er/qEBejymWyGHh6MybdfRr9uzZi+DfL6f3kROZmbw07NBFJIUr8ISiZmc6ferfktf4nsX3Pfi54ZhLPjF3MgYPx/9uXiCQ+Jf4QdW1Wlc8Gd+fMljV56NPvuXzIt6zctCvssEQkySnxh6xi6SyeuuIEHr20DfPXbOOcJybw3vRVJMK9FxFJTEr8ccDMuLBdXUYP6sYxtcpxx1uzuHXEDLbs2hd2aCKShJT440i9yqV5Y8DJ/KFXCz6bu5Zej09g4g8bww5LRJKMEn+cSU8zbu7RlJG3dKFMiXSuGjaZv46az579B8IOTUSSRMwSv5m9aGbrzWxunm0Pm9lCM5ttZu+bWcVYnT/RtapTgVG3daPfyQ0YNvFH+jw1iQVrtoUdlogkgVhe8b8M9Dpk2xiglbu3BhYB98Tw/AmvVFY69/VpxUvXdWTTrn30eWoSL4xfykEN+xSRQohZ4nf38cCmQ7Z97u45wctvgbqxOn8yObVFdT4b3J0eLarx908WcOXQyazesjvssEQkQYXZx389MDq/N81sgJlNNbOpGzZsKMaw4lPlMlk8f3V7HrqoNbNXbaHX4+P5cNbqsMMSkQQUSuI3sz8COcDr+e3j7kPcvYO7d6hWrVrxBRfHzIxLO9bjk0HdaFq9LANHzGDQGzPYunt/2KGJSAIp9sRvZtcCvYErXU8pFUiDKmV466aTuaNnc0bNXsPZj4/nmyU/hR2WiCSIYk38ZtYL+ANwvrurNkEhZKSnMfD0Zrz7u86UyEzniqHf8sDoBezN0bBPETm8WA7nHAF8A7Qws1Vm1h94CigHjDGzmWb2XKzOnyra1qvIxwO7cvmJ9Xl+3FL6Pv01i9ZtDzssEYljqsefRL6Yv4673p3N9r053N3rGK7t3JA0TfgukrJUjz8FnNGyBp8O7k63plW5f9R8+r30HWu37gk7LBGJM0r8SaZauRIM7deB/7ngeKYu28xZj4/nkzlrwg5LROKIEn8SMjOuOKk+Hw/sSsMqpbn59en811uz2L5Hwz5FRIk/qTWuVpZ3fteZgac34/0Zqzj7iQlMWbbpyB8UkaSmxJ/kMtPTuKNnc97+bWfSzPjN89/w8GcL2ZdzMOzQRCQkSvwpon2DSnwyqBuXtK/H018t4cJnJ7F4/Y6wwxKRECjxp5CyJTJ48OLWPHdVe7I376b3kxN45ZtlmuZRJMVkHGkHM6sOdAFqA7uBucBUd1dfQYLq1aom7epX5PfvzObPH8zjy4Xreeji1lQvVzLs0ESkGOR7xW9mp5rZZ8DHwNlALaAlcC8wx8zuM7PyxROmFLXq5Uvy8nUdub/PcXyz5Cd6PT6Bz+atDTssESkGh7viPwe40d1XHPqGmWUQKbTWE3g3RrFJjJkZ15zckM5NqjD4zZnc9Oo0Lj+xPn/r24p0PfErkrTyTfzu/vvDvJcDjIxJRFLsmlYvx3u/68IjY77n+XGRGb7+cdHxmCn5iySjI97cNbNBZlbeIoaZ2XQzO7M4gpPik5WRxj1nH8vA05ry5tSVPPjp92GHJCIxEs2onuvdfRtwJlAJuBr4R0yjktDc3rM5V3Wqz3PjlvD8uCVhhyMiMXDEUT1A7u/75wCvuvs8Ux9A0jIz7ju/FVt27eeB0QupVDqLSzvWCzssESlC0ST+aWb2OdAIuMfMygEaypnE0tOMRy9ty7Y9Odz93mzKl8qkV6uaYYclIkUkmq6e/sDdQMdg1qws4LqYRiWhy8pI47mr2tGmXkUGjpjB10s2hh2SiBSRaBL/GHef7u5bANz9J+Cx2IYl8aB0VgYvXduRRlXLcOPwqcxetSXskESkCBzuAa6SZlYZqGpmlcyscrA0BOoUV4ASroqls3il/4lUKpPFtS9NUX0fkSRwuCv+m4BpwDHA9GB9GvABkblzJUXUKF+S1/qfRJrBNcMms3rL7rBDEpFCyDfxu/sT7t4IuNPdG+VZ2ri7En+KaVi1DMOvP5Hte3K4ethkNu3cF3ZIIlJAh+vqOS1YzTazCw9diik+iSPH1a7AsGs7smrzbq576Tt27M0JOyQRKYDDdfWcEvw871eW3kc6sJm9aGbrzWxunm2VzWyMmf0Q/KxUiNglBCc2qswzV7Zj7upt3PTqVPbmHAg7JBE5SharWuxm1h3YAbzi7q2CbQ8Bm9z9H2Z2N1DJ3e860rE6dOjgU6dOjUmcUjDvTV/FHW/NotdxNXn6ynYq6iYSh8xsmrt3OHR7NLV6qpjZv4IaPdPM7Akzq3Kkz7n7eODQCV77AMOD9eFA3yhilzh0Ybu6/Ll3Sz6dt5Y/vj9Hk7mIJJBoxvG/AWwALgIuDtbfLOD5arj7mmB9LVAjvx3NbICZTTWzqRs2bCjg6SSWru/aiIGnNeWNKSrqJpJIoinZUMvd/5rn9d/M7DeFPbG7u5nle5no7kOAIRDp6ins+SQ2bu/ZnE279vHcuCVUKp3JTac0CTskETmCaK74Pzezy8wsLVguBT4r4PnWmVktgODn+gIeR+JEblG33q1r8cDohbw1ZWXYIYnIEeR7xW9m2wEnUp1zMPBq8FY6kZu2dxbgfB8C/YiUde5H5GEwSXAq6iaSWA73AFc5dy8f/Exz98xgSXP3I861a2YjgG+AFma2ysz6E0n4Pc3sB+AMVNc/aaiom0jiiNlwzqKk4ZyJY8uufVz6/Ddkb97NiAGdaF23YtghiaSsAg/nFDkaFUtn8Wr/k34u6rZkg4q6icQbJX4pcnmLul09VEXdROJNVInfzNLNrLaZ1c9dYh2YJDYVdROJX9E8uXsbsA4YA3wcLKNiHJckARV1E4lP0VzxDwJauPtx7n58sLSOdWCSHFTUTST+RJP4VwJbYx2IJK/Tj63Bwxe3ZtLinxg0YiYHDsb/SDKRZBZNyYalwFgz+xjYm7vR3R+NWVSSdC5sV5ctu/Zz/6j5/PH9OTxw4fGYqaKnSBiiSfwrgiUrWEQK5Pqujdi8ax9PfrmYSmWyuKvXMWGHJJKSjpj43f2+4ghEUsMdPZuzedc+nh0bKeo2oLuKuokUt8PV6nnc3Qeb2UdEavb8grufH9PIJCnlFnXbsms///PJQiqWyuLSjvXCDkskpRzuij+3KNs/iyMQSR2HFnWrUDqTs45TUTeR4qJaPRKaXftyuHLoZOZlb+Pl6zvSuUnVsEMSSSpHXavHzD4ys/PMLPNX3mtsZveb2fVFHaikjtJZGbx0bUcaVi3NjcOnMnvVlrBDEkkJhxvHfyPQDVhoZlPM7BMz+9LMlgLPA9Pc/cViiVKSloq6iRS/qLp6zKwhUAvYDSxy912xDeuX1NWT/JZt3MnFz31NVnoa7/yuM7Urlgo7JJGEV6iyzO6+zN2/cfeZxZ30JTWoqJtI8VFZZokbx9WuwNB+HVTUTSTGlPglrpzUuApPX6GibiKxFG09/lJm1iLWwYgAnNHy/4q6DX5DRd1Eilo09fjPA2YCnwav25rZh7EOTFLbhe3q8ufeLRk9dy1/fH8OifC8iUiiiKZI21+AE4GxAO4+08waxTAmEeCXRd0qlMrkrl7HkJamip4ihRVNV89+dz+0Hn+hLr/M7HYzm2dmc81shJmVLMzxJHnd0bM5V3dqwPPjl3L5C9+ybOPOsEMSSXjRJP55ZnYFkG5mzczsSeDrgp7QzOoAA4EO7t4KSAcuK+jxJLmZGff3OY4HLzqe+au30euJ8QydsFT9/iKFEE3ivw04jsgkLCOAbcDgQp43AyhlZhlAaWB1IY8nSczM+E3H+oy54xS6NKnK3z5ewMXPfc3i9dvDDk0kIYVSpM3MBgF/J/Ik8OfufuWv7DMAGABQv3799suXLy/eICUuuTsfzlrNf384j117DzDojGYM6N6YzHSNTBY5VH5P7h4x8edTj38rMBV43t33HGUglYB3gd8AW4C3gXfc/bX8PqOSDXKoDdv38pcP5/HxnDW0qlOehy5qQ8va5cMOSySuFKZkw1JgB/BCsGwDtgPNg9dH6wzgR3ff4O77gfeAzgU4jqSwauVK8PSV7Xj2ynas3bqX85+ayKOff68HvkSiEM1wzs7u3jHP64/MbIq7dzSzeQU45wqgk5mVJtLVczqR3x5EjtrZx9eiU+Mq/HXUfP715WI+nbeWhy5uQ9t6FcMOTSRuRXPFX9bM6ue+CNbLBi+PupKWu08G3gGmA3OCGIYc7XFEclUqk8Wjv2nLS9d2ZPueHC58ZhIPfLKAPft19S/ya6Lp4z8HeA5YAhjQCLiZyANdN7r74zGOUX38ErVte/bzwCcLGfHdChpVLcNDF7emY8PKYYclEooC39wNPlwCOCZ4+f3R3tAtLCV+OVqTFm/krndnk71lN/1Obsjvz2pBmRLR9GyKJI9C1eMHmgEtgDbApWZ2TVEGJ1LUujStymeDu9Pv5IYM/2YZZz0+nok/bAw7LJG4EE2Rtv8GngyWU4GHgPNjHJdIoZUpkcFfzj+Ot246maz0NK4aNpm7353Ntj37ww5NJFTRXPFfTGTkzVp3v47IVX+FmEYlUoQ6NqzMJ4O6cdMpjXlr6krOfHQ8Xy5cF3ZYIqGJJvHvdveDQI6ZlQfWA/ViG5ZI0SqZmc49Zx/L+zd3oUKpTK5/eSq3vzmTzZriUVJQNIl/qplVJPKw1jQiwzC/iWlUIjHSpl5FPrqtK4NOb8ZHs1bT87FxjJ6zJuywRIpVNMM5zYOdzKwhUJ7IyJ69MY8uoFE9EgvzV2/jD+/OYm72Ns5uVZP7+7SiWrkSYYclUmQKM6pnWO6Kuy8jUsLhk6ILTSQcLWuXZ+TNXfhDrxb8e+F6ej42jvdnrNJsX5L0okn82Wb2DPxcYO1zIN+CaiKJJCM9jZt7NOWTgV1pXLUMt785i/7Dp7Jm6+6wQxOJmSMmfnf/E7DDzJ4jkvQfcfeXYh6ZSDFqWr0cb/+2M3/q3ZKvl2zkzEfH88Z3K3T1L0kp38RvZhfmLsBkoBMwA/Bgm0hSSU8z+ndtxGeDu3NcnfLc/d4crho2mZWbdoUdmkiRyvfmrpkd7qre3f362IT0n3RzV4rbwYPOiCkreOCThRx05w9nteCakxtqsndJKIWq1RM2JX4JS/aW3dzz3hzGL9pAx4aVePCi1jSuVvbIHxSJA4WZgask0J/IvLslc7fril9Shbvz7vRs7v9oHntzDnJHz+b079qIDE33KHGuMMM5XwVqAmcB44C6RGbgEkkJZsbF7evyxR2n0L15NR4YvZCLnv2a79fqn4EkpmgSf9NgZM9Odx8OnAucFNuwROJP9fIlGXJ1e568/ARWbt5N7ycn8MQXP2i6R0k40ST+3FKGW8ysFZECbdVjF5JI/DIzzmtTmzG3d6dXq1o89sUiej0+gXGLNoQdmkjUokn8Q4IHt/4EfAjMJ1KaWSRlVSlbgicvP4Hh158IQL8Xv+N3r00je4se/JL4p1E9IoW0N+cAQyf8yJNf/oBh3HpaU27o1ogSGelhhyYprsA3d82shpkNM7PRweuWZtY/FkGKJKISGenccmrT4OZvVR7+7HvOfnwC49X9I3Eqmq6el4HPgNrB60XA4FgFJJKo6lYqzfNXd+Cl6zpy0J1rXvyOm1+fxmp1/0iciSbxV3X3t4CDAO6eAxRqGIOZVTSzd8xsoZktMLOTC3M8kXhyaovqfDq4O3ee2ZwvF67n9EfG8czYxezLORh2aCJAdIl/p5lVAXJr8ncCthbyvE8An7r7MUSmclxQyOOJxJWSmencelozxtx+Ct2aVeWhT7+n1xOa8F3iQzSJ/w4io3mamNkk4BXgtoKe0MwqAN0J6vy7+z5331LQ44nEs3qVSzPkmg68dG1HDhx0rho2mVten66yzxKqqEb1mFkG0AIwIrNv7T/CRw53rLbAECLDQtsQmc5xkLvvPGS/AcAAgPr167dfvnx5QU8pEhf27D/AkPFLefqrxaSnGQNPb8b1XRqRlaHSDxIbR12r50ill939vQIG0gH4Fuji7pPN7AlgW/B08K/ScE5JJis37eL+UfMZM38dTaqV4f4+rejStGrYYUkSyi/xZxzmM+cd5j0HCpT4gVXAKnefHLx+B7i7gMcSSTj1KpfmhWs68OXCdfzlw/lcOXQy57auxb3nHkutCqXCDk9SQL6J392vi8UJ3X2tma00sxbu/j1wOpFuH5GUctoxNejcpCrPj1vKM2MX89XC9Qw6vRnXqftHYiyUJ3eDfv6hQBaRyduvc/fN+e2vrh5Jdis37eK+j+bzxYJ1NK1elvvPP47O6v6RQtJELCIJ4N8L1nHfR/NZsWkXvVvX4t5zW1KzQskjf1DkVxSmHr+IFJPTj63B57d35/YzmjNm/jpOf2QsQ8YvYf8BPfwlRSeaWj3TzOyWoEKniMRYycx0Bp0Refjr5CZV+J9PFnLOExP4eoke/pKiEc0V/2+I1OmZYmZvmNlZZqYZp0VirH6V0gzt15Gh13RgT84BrnhhMreNmMHarXvCDk0SXNR9/GaWBvQGniVSq+cl4Al33xS78CLUxy+pbs/+Azw7dgnPjltCZpox+IzmXNulIZma91cOo1B9/GbWGngEeBh4F7gE2AZ8WZRBisivK5mZzu09mzPm9u6c1LgKf/9kgbp/pMCi6uMHHgOmAK3dfaC7T3b3R4gMxRSRYtKgShlevDbS/bN7f6T7Z+CIGazbpu4fid4Ru3rMrLG7h5rg1dUj8p/27D/AM2OX8FzQ/XN7z+b066zuH/k/henqucHMKuY5UCUz+1uRRiciR61kZjp3BN0/JzaqzN8+XsC5/5rAt0t/Cjs0iXPRJP6z85ZNDp6wPSd2IYnI0cjt/nnhmg7s3HuAy4Z8y+A3ZrB1d4GL6EqSiybxp5tZidwXZlYKKHGY/UWkmJkZPVvW4Is7TmHgaU0ZNXsNfZ6ayMK128IOTeJQNIn/deDfZtY/mGR9DDA8tmGJSEGUykrnjjNb8OZNndi17wB9n57EBzOzww5L4swRE7+7Pwj8HTg2WP7q7g/FOjARKbj2DSozamBXWtetyKA3ZvKXD+dpzl/52eHq8f/M3UcDo2Mci4gUoerlSvL6DSfx4OiFDJ34I3Oyt/LMle2oUV5F31JdNOP4O5nZFDPbYWb7zOyAmanjUCQBZKancW/vljx1xQksWLONc/81kcka9ZPyounjfwq4HPgBKAXcADwdy6BEpGj1bl2bD27pQvlSGVwxdDJDJywlEUqyS2xE9aSHuy8G0t39gLu/BPSKbVgiUtSa1SjHB7d0oeexNfjbxwu4dcQMdu7NCTssCUE0iX+XmWUBM83sITO7PcrPiUicKVcyk2evasc9Zx/D6Dlr6PP0JJZs2BF2WFLMokngVwf73QrsBOoBF8UyKBGJHTPjplOa8Fr/k9i8cx99nprEp3PXhB2WFKNoEn97wN19m7vf5+53BF0/IpLAOjetyke3daVp9bL89rXpPDB6ATma6SslRJP4zwMWmdmrZtbbzKIaAioi8a92xVK8eVMnrupUn+fHLeXqYd+xccfesMOSGIvmAa7rgKbA20RG9ywxs6GxDkxEikeJjHT+1vd4/nlJG6av2Mx5T05kxorNYYclMRTtqJ79RB7gegOYBvQt7InNLN3MZpjZqMIeS0QK7+L2dXnv5s5kpBuXPv8Nr327XEM+k1Q0D3CdbWYvExnHfxEwFKhZBOceBCwoguOISBE5rnYFRt3aja5Nq3LvyLnc+fZs9uw/EHZYUsSiueK/BhgJtHD3a939E3cv1OBfM6sLnEvkPxERiSMVSmcyrF9HBp/RjPdmrOLCZ75mxU+7wg5LilA0ffyXu/tIdy/KOz6PA38A8h1CYGYDzGyqmU3dsGFDEZ5aRI4kLZjQ/cVrO5K9ZTe9n5zAVwvXhx2WFJF8E7+ZTQx+bjezbXmW7YWp1WNmvYH17j7tcPu5+xB37+DuHapVq1bQ04lIIZzaojof3dqVupVKc/3wKTw2ZhEHD6rfP9Hlm/jdvWvws5y7l8+zlHP38oU4ZxfgfDNbRuRm8Wlm9lohjiciMVS/Smneu7kzF55Qlyf+/QPXD5/Cll37wg5LCuGwXT3ByJuFRXlCd7/H3eu6e0PgMuBLd7+qKM8hIkWrZGY6/7ykNX+/oBWTFm/kvKcmMjd7a9hhSQEdNvG7+wHgezOrX0zxiEicMjOuPKkBb910MjkHnIue/Zp3pq0KOywpgGhG9VQC5pnZv83sw9ylKE7u7mPdvXdRHEtEiscJ9Ssx6rautG9QiTvfnsUf35/D3hwN+Uwk0ZRf+FPMoxCRhFKlbAleuf5EHhmziGfHLmHu6m08e2U7alcsFXZoEoXDjeopaWaDgUuAY4BJ7j4udym2CEUkLmWkp3FXr2N47qr2LFm/g95PTmTS4o1hhyVROFxXz3CgAzAHOBt4pFgiEpGE0qtVTT64tQtVymRx9bDJPDt2iUo9xLnDJf6W7n6Vuz8PXAx0K6aYRCTBNKlWlpG3dOGc42vx4KcL+e1r09i+Z3/YYUk+Dpf4f/5bK2yJBhFJfmVKZPDk5Sfwp94t+WLBevo8NYlF67aHHZb8isMl/jZ5n9YFWhfFk7sikrzMjP5dGzHixk5s25ND36cn8dGs1WGHJYc43JO76Yc8rZtRRE/uikiSO7FRZT4e2JWWtcpz24gZ/HXUfPZrdq+4oUnTRSQmapQvyYgBnbi2c0OGTfyRK1+YzPrte8IOS1DiF5EYykxP4y/nH8cTl7VlTvZWzv3XRD6Yma1RPyFT4heRmOvTtg7v39KZGuVLMOiNmVzwzNdMW74p7LBSlhK/iBSLY2qW58NbuvLPS9qwZutuLnr2G2753+ms3KRJXoqbEr+IFJu0NOPi9nX56s4eDD6jGV8uWM/pj4zjgdEL2KZx/8VGiV9Eil3prAwGn9Gcr+7swXltavP8uKWc+vBYXvt2OTka/RNzSvwiEpqaFUryyKVt+OjWrjSpXpZ7R87lnH9NYNwiTbcaS0r8IhK64+tW4M0BnXjuqvbszTlIvxe/o9+L3/GDnvyNCSV+EYkLZkavVjUZc/sp3HvusUxfsZleT0zg3pFz+GnH3rDDSypK/CISV7Iy0rihW2PG/f5Uru7UgBHfraTHw2N5ftwSTfhSRJT4RSQuVS6TxV/OP47PBnfnxEaVeWD0Qs54dBwfz16jB8AKSYlfROJa0+plGXZtR17rfxJlsjK45X+nc8lz3zBz5ZawQ0tYSvwikhC6NqvKxwO78eBFx7Psp130fXoSg9+YQfaW3WGHlnCU+EUkYaSnGb/pWJ+xv+/Brac2ZfTctZz2z7E88vn37NyraUOiVeyJ38zqmdlXZjbfzOaZ2aDijkFEElvZEhnceVYLvryzB71a1eTJLxfT459jeXPKCg4cVP//kYRxxZ8D/Je7twQ6AbeYWcsQ4hCRBFenYimeuOwE3r+5M/Url+aud+fQ+8mJfK1J3w+r2BO/u69x9+nB+nZgAVCnuOMQkeRxQv1KvPPbk3nqihPYvmc/V35ARjcAAAvHSURBVAydzA3Dp7Bkw46wQ4tLFuawKDNrCIwHWrn7tkPeGwAMAKhfv3775cuXF3t8IpJ49uw/wEuTlvH0V4vZs/8AV3VqwKDTm1GpTFbYoRU7M5vm7h3+Y3tYid/MygLjgL+7+3uH27dDhw4+derU4glMRJLCxh17eWzMIkZ8t4JyJTMZeHozru7UgKyM1BnTkl/iD+VPwMwygXeB14+U9EVECqJq2RL8/YLj+XRwd9rUq8hfR83nzMfG8dm8tSn/AFgYo3oMGAYscPdHi/v8IpJamtcoxyvXn8jL13UkMz2Nm16dxuUvfMvc7K1hhxaaMK74uwBXA6eZ2cxgOSeEOEQkhfRoUZ3Rg7rx176tWLRuB+c9NZE7357Fum2pNwF8qDd3o6U+fhEpStv27Ofprxbz0sRlpKcZN53SmCtPakC1ciXCDq1Ixd3N3aOhxC8isbDip108+OlCPp6zhvQ0o0vTqvRtW5szj6tJ2RIZYYdXaEr8IiL5WLx+OyNnrGbkzGxWbd5Nycw0erasSd+2tenevBqZ6Yk5EkiJX0TkCNydacs3M3JmNh/PXsPmXfupXCaLc4+vRd8TatOufiUi41MSgxK/iMhR2JdzkPGLNjByZjZj5q9jb85B6lUuRZ82deh7Qm2aVi8XdohHpMQvIlJAO/bm8NnctYycmc2kxRs56NCqTnn6tq3DeW1qU6N8ybBD/FVK/CIiRWD99j18NGsNH8zMZvaqraQZdG5SlT5ta9OrVU3KlcwMO8SfKfGLiBSxJRt28MGMbEbOXM2KTbsokZHGGcfWoE/b2vRoUT308hBK/CIiMeLuzFi5hQ9mZDNq9hp+2rmPiqUzOef4WvRtW4cODSqRllb8N4WV+EVEisH+AweZ+MNGRs7M5vN569i9/wB1Kpbi/La1ueCEOjSvUXw3hZX4RUSK2c69OYyZv46RM7OZ8MNGDhx0jq1Vnr5ta3N+29rUqlAqpudX4hcRCdHGHXsZNWs1I2euZubKLZjBSY0qc8EJdejVqhYVShX9TWElfhGROPHjxp18MDObD2au5seNO8nKSOO0FtXpe0JtTj2mOiUy0ovkPEr8IiJxxt2ZvWorI2dm89Gs1WzcsY9yJTM49/ha9Glbh5MaVS7UTWElfhGROJZz4CCTlvzEBzOy+WzeWnbuO0CtCiV55JI2dG5atUDHzC/xJ375ORGRJJCRnsYpzatxSvNq7N53gDEL1vHBjGzqVS5d9Ocq8iOKiEihlMpK5/w2tTm/Te2YHD8xa42KiEiBKfGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIilHiFxFJMUr8IiIpJiFKNpjZBmB5AT9eFdhYhOHEE7UtMSVr25K1XZC4bWvg7tUO3ZgQib8wzGzqr9WqSAZqW2JK1rYla7sg+dqmrh4RkRSjxC8ikmJSIfEPCTuAGFLbElOyti1Z2wVJ1rak7+MXEZFfSoUrfhERyUOJX0QkxSR14jezXmb2vZktNrO7w44nGmb2opmtN7O5ebZVNrMxZvZD8LNSsN3M7F9B+2abWbs8n+kX7P+DmfULoy15mVk9M/vKzOab2TwzGxRsT4a2lTSz78xsVtC2+4LtjcxsctCGN80sK9heIni9OHi/YZ5j3RNs/97MzgqnRb9kZulmNsPMRgWvk6Vdy8xsjpnNNLOpwbaE/z5Gxd2TcgHSgSVAYyALmAW0DDuuKOLuDrQD5ubZ9hBwd7B+N/BgsH4OMBowoBMwOdheGVga/KwUrFcKuV21gHbBejlgEdAySdpmQNlgPROYHMT8FnBZsP054HfB+s3Ac8H6ZcCbwXrL4HtaAmgUfH/T4+A7eQfwv8Co4HWytGsZUPWQbQn/fYxmSeYr/hOBxe6+1N33AW8AfUKO6YjcfTyw6ZDNfYDhwfpwoG+e7a94xLdARTOrBZwFjHH3Te6+GRgD9Ip99Plz9zXuPj1Y3w4sAOqQHG1zd98RvMwMFgdOA94Jth/attw2vwOcbmYWbH/D3fe6+4/AYiLf49CYWV3gXGBo8NpIgnYdRsJ/H6ORzIm/DrAyz+tVwbZEVMPd1wTra4EawXp+bYzrtgddACcQuTJOirYF3SEzgfVE/vEvAba4e06wS944f25D8P5WoArx2bbHgT8AB4PXVUiOdkHkP+fPzWyamQ0ItiXF9/FINNl6gnF3N7OEHYNrZmWBd4HB7r4tckEYkchtc/cDQFszqwi8DxwTckiFZma9gfXuPs3MeoQdTwx0dfdsM6sOjDGzhXnfTOTv45Ek8xV/NlAvz+u6wbZEtC74tZLg5/pge35tjMu2m1kmkaT/uru/F2xOirblcvctwFfAyUS6A3IvrvLG+XMbgvcrAD8Rf23rApxvZsuIdJWeBjxB4rcLAHfPDn6uJ/Kf9Ykk2fcxP8mc+KcAzYIRCFlEbjZ9GHJMBfUhkDtaoB/wQZ7t1wQjDjoBW4NfUz8DzjSzSsGohDODbaEJ+nqHAQvc/dE8byVD26oFV/qYWSmgJ5F7GF8BFwe7Hdq23DZfDHzpkTuFHwKXBaNjGgHNgO+KpxX/yd3vcfe67t6QyL+fL939ShK8XQBmVsbMyuWuE/kezSUJvo9RCfvuciwXInfiFxHpb/1j2PFEGfMIYA2wn0h/YX8i/aT/Bn4AvgAqB/sa8HTQvjlAhzzHuZ7ITbTFwHVx0K6uRPpUZwMzg+WcJGlba2BG0La5wJ+D7Y2JJLjFwNtAiWB7yeD14uD9xnmO9cegzd8DZ4fdtjxx9eD/RvUkfLuCNswKlnm5+SEZvo/RLCrZICKSYpK5q0dERH6FEr+ISIpR4hcRSTFK/CIiKUaJX0QkxSjxS9wxMzezR/K8vtPM/lJEx37ZzC4+8p6FPs8lZrbAzL462rjMbKiZtSzgeXuYWeeCfFZShxK/xKO9wIVmVjXsQPLK87RqNPoDN7r7qUd7Hne/wd3nH+3nAj0AJX45LCV+iUc5ROY4vf3QNw69YjezHcHPHmY2zsw+MLOlZvYPM7vSInXy55hZkzyHOcPMpprZoqAeTW6RtYfNbEpQb/2mPMedYGYfAv+RjM3s8uD4c83swWDbn4k8sDbMzB7+lc/cFXxmlpn941feH2tmHYL1M83sGzObbmZvB7WOcmvJ3xdsn2NmxwTF734L3G6RGvPdgt885gbnGh/dH78kOxVpk3j1NDDbzB46is+0AY4lUtZ6KTDU3U+0yKQvtwGDg/0aEqnL0gT4ysyaAtcQeQy/o5mVACaZ2efB/u2AVh4pKfwzM6sNPAi0BzYTqfTY193vN7PTgDvdfeohnzmbSInfk9x9l5lVzq8xwW889wJnuPtOM7uLSG38+4NdNrp7OzO7OTjXDWb2HLDD3f8ZHGMOcJZHipFVPIo/S0liuuKXuOTu24BXgIFH8bEpHqn7v5fIo/W5iXsOkWSf6y13P+juPxD5D+IYIjVWrrFIaeXJRB7dbxbs/92hST/QERjr7hs8Uob4dSIT6RzOGcBL7r4raOehcy/k1YnIJCaTgrj6AQ3yvJ9b6G7aIe3LaxLwspndSGRyIhFd8UtcexyYDryUZ1sOwQWLmaURmV0t19486wfzvD7IL7/rh9YpcSK1WG5z918U2LJIOeKdBQu/0IzIJB+X5/N+bvsOkM+/ZXf/rZmdRGQylWlm1t7dfyr6UCWR6Ipf4lZwNfwWkRuluZYR6VoBOJ/IbFdH6xIzSwv6/RsTKRz2GfA7i5SOxsyaB1UbD+c74BQzq2pm6cDlwLgjfGYMcJ2ZlQ7Ok29XD/At0CXoisqtKNn8CMffTmRqS4LPNHH3ye7+Z2ADvywhLClKiV/i3SNA3tE9LxBJtrOI1LwvyNX4CiJJezTwW3ffQ2RqwfnAdItMdP88R/iN2CNlee8mUqZ4FjDN3T84wmc+JVLid2rQfXPnYfbdAFwLjDCz2cA3HHmCl4+AC3Jv7gIP5958Br4O4pQUp+qcIiIpRlf8IiIpRolfRCTFKPGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIivn/qyWYxSu6saAAAAAASUVORK5CYII=\n" 187 | }, 188 | "metadata": { 189 | "needs_background": "light" 190 | } 191 | } 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "source": [ 197 | "# Fixed UK=50000" 198 | ], 199 | "metadata": { 200 | "id": "2eQs7BvRBM9Z" 201 | } 202 | }, 203 | { 204 | "cell_type": "code", 205 | "source": [ 206 | "UK = 50000 # 50000\n", 207 | "N = 10\n", 208 | "\n", 209 | "for K in [9]: # [8]\n", 210 | " U = int(UK / K)\n", 211 | " B = f(N+K-1) // f(K) // f(N-1)\n", 212 | " print(\"K, U, B:\", K, U, B)\n", 213 | "\n", 214 | " # calculate H(B|Y=y), y is uniform\n", 215 | " p_y = 0\n", 216 | " P_ys = []\n", 217 | " b_count = 0\n", 218 | " H_y = 0\n", 219 | " for comb in itertools.combinations(np.arange(N+K-1), N-1):\n", 220 | " b_count += 1\n", 221 | " p_y = f(K) / (N**K) / f(comb[0]) / f(N+K-2-comb[-1])\n", 222 | " for n in range(1, N-1):\n", 223 | " p_y = p_y / f(comb[n]-comb[n-1]-1)\n", 224 | " P_ys.append(p_y)\n", 225 | " H_y += -p_y * math.log2(p_y)\n", 226 | "\n", 227 | " # print('len(P_ys)', len(P_ys))\n", 228 | " # print('sum(P_ys)', sum(P_ys))\n", 229 | " # print('b_count', b_count)\n", 230 | " # print('B', B)\n", 231 | " # print('math.log2(B)', math.log2(B))\n", 232 | " print('H_y', H_y)\n", 233 | " # print('2**H_y', 2**H_y)\n", 234 | "\n", 235 | " # calculate H(B|S=s)\n", 236 | " H_s = 0\n", 237 | " p_s = 0\n", 238 | " choices = np.random.choice(B, U, p=P_ys)\n", 239 | " # print(len(choices), choices)\n", 240 | " start_time = time.time()\n", 241 | " for u in range(U):\n", 242 | " if u%10000==0:\n", 243 | " end_time = time.time()\n", 244 | " print('u:', u, '/', U, ', H_s:', H_s, ', time:', round((end_time-start_time)/60))\n", 245 | " p_s = sum([1 for c in choices if c == u]) / U\n", 246 | " # print(p_s)\n", 247 | " if p_s != 0:\n", 248 | " H_s += -p_s * math.log2(p_s)\n", 249 | " print('H_s', H_s)" 250 | ], 251 | "metadata": { 252 | "id": "I26xJAKI87vq", 253 | "colab": { 254 | "base_uri": "https://localhost:8080/" 255 | }, 256 | "outputId": "abc10e1d-2476-4108-f9ad-883f2e6fbaca" 257 | }, 258 | "execution_count": null, 259 | "outputs": [ 260 | { 261 | "output_type": "stream", 262 | "name": "stdout", 263 | "text": [ 264 | "K, U, B: 9 5555 48620\n", 265 | "H_y 14.7084540600815\n", 266 | "u: 0 / 5555 , H_s: 0 , time: 0\n", 267 | "H_s 0.5174473607079045\n" 268 | ] 269 | } 270 | ] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "source": [ 275 | "Ks = [1, 2, 4, 6, 7, 8, 9, 10, 16]\n", 276 | "Ks_H_y = [3.321928, 5.743856, 9.286393, 11.8602849, 12.91726, 13.860046, 14.708454, 15.4776598, 18.95812]\n", 277 | "K_s_H_s_UK100000 = [3.321874, 5.74309, 9.25941, 11.64429, 12.33949, 5.448149, 1.7305256, 0.505497, 0]\n", 278 | "K_s_H_s_UK50000 = [3.321928, 5.738995, 9.2478531, 11.399456, 6.4285309, 1.970087, 0.517447, 0.17900, 0]\n", 279 | "K_s_H_s_UK5000 = [3.318592, 5.73230, 8.81008, 1.091727, 0.146047, 0.056241, 0, 0, 0]\n", 280 | "\n", 281 | "# plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|Y=y_0)}$')\n", 282 | "# plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|S=s), UK=5000}$')\n", 283 | "# plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|S=s), UK=50000}$')\n", 284 | "# plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|S=s), UK=100000}$')\n", 285 | "plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|P_{y,uni})}$')\n", 286 | "plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=5000}$')\n", 287 | "plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=50000}$')\n", 288 | "plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=100000}$')\n", 289 | "plt.xlabel('Number of samples K from each client')\n", 290 | "plt.ylabel('Lable Privacy information (in bits)')\n", 291 | "plt.legend()\n", 292 | "plt.show()" 293 | ], 294 | "metadata": { 295 | "id": "befatQox87ql", 296 | "colab": { 297 | "base_uri": "https://localhost:8080/", 298 | "height": 279 299 | }, 300 | "outputId": "b202ec23-d599-42a4-a941-0ee1804dbff2" 301 | }, 302 | "execution_count": null, 303 | "outputs": [ 304 | { 305 | "output_type": "display_data", 306 | "data": { 307 | "text/plain": [ 308 | "
" 309 | ], 310 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1hUV/rA8e+hKyCiFAsqKgpYUYmJRlDBHrvRaDSJm2KKKcbkl76bstlssqkmpidqitEklo09xt4LJhqNYMOGBQSU3nl/f8zAolIGmGEYPJ/nuQ8zd+4993WXcObce877KhFB0zRN065lZ+0ANE3TtNpJdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappXKwdoBmJOXl5f4+/tbOwxN0zSbsW/fvkQR8S7tszrVQfj7+xMVFWXtMDRN02yGUup0WZ/pW0yapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmaZqNSMvNYduA8n20+YZH269RCOU3TtLpMRIhNzGBDdALrouOJOn2ZgkKhqYcL9/dpjYO9eb/z6w5C0zStFsvNLyTqVDLrohPYEBPPqaRMAIKauPNQ3zZEBPkS0qIh9nbK7NfWHYSmaVotk5yRy6YjCayPTmDL0Uuk5eTj5GBH77aNua9Pa/oH+eDnWd/icegOQtM0zcpEhKPx6ayPiWd9dAK/n7mMCHi7O3Nbl6ZEBPlwa4AXrs41+ydbdxCapmlWkJNfwK7YZDZEx7M+JoG4y1kAdG7uweMR7YgM9qFTMw/sLHDryFS6g9A0TashCWnZbIq5xPqYeLYeSyQztwAXRzv6BHgzvX8A/QN9aOLhYu0wi+kOQtM0zUJEhL/Op7IhJoH10fEciEsBoJmHC2O7NycyyJdebRvj4mhv5UhLpzsITdM0M8rKLWDHiUTWxySwITqBi6nZKAVd/Rry9KD2RAT5EtzUHaWsd+vIVLqD0DRNq6YLKVlsMHYI244nkpNfiKuTPeHtvYkI8qFfoA/e7s7WDrPSdAehaZpWSYWFwp/nUtgQHc+66AQOX0gFoEWjekzq2ZLIYB96tm6Es0PtvHVkKt1BaJqmmSAjJ5+txxLZEBPPhphLJKbnYKegRytPnhsaRGSQDwE+bjZx68hUuoPQNE0rw9nkTMMD5pgEdp1IIregEHcXB/oF+hAZ5EPf9t54ujpZO0yLsVgHoZSaAwwHEkSkk3Hfj0Cg8ZCGwBURCSnl3FNAGlAA5ItIqKXi1DRNK1JQKPxx5nLxA+Yj8WkAtPFy5Z7erYgI8iXU3xNHM+c8qq0sOYKYB8wGvi3aISJ3FL1WSr0LpJRzfn8RSbRYdJqmaUBqdh5bjl5iQ3QCG48kcDkzDwc7Rc/WjXgpNJiIIB/aeLtZO0yrsFgHISJblFL+pX2mDDfpJgARlrq+pmlaWU4mZrA+Op4NMQnsOZlMfqHgWd+R/oE+RAT7ENbOG496jtYO0+qs9QwiDIgXkWNlfC7AWqWUAJ+LyBdlNaSUmgZMA2jZsqXZA9U0zfblFRQSdeoyG2IMaS1iL2UA0N7XjQfC2xAZ5EO3lp4WyYhqy6zVQUwCFpTzeR8ROaeU8gF+U0rFiMiW0g40dh5fAISGhor5Q9U0zRZdzshl89FLrI9JYPORBFKz83Gyt+OWto25p5c/EUE+tGhk+YyotqzcDkIp5QdMxPCNvxmQBRwCVgKrRaSwshdUSjkAY4EeZR0jIueMPxOUUkuBnkCpHYSmaRoY0locT0gvfsAcdTqZQgEvNycGd2xCZLAvfdp54VbDGVFtWZn/Syml5gLNgRXAW0AC4AK0B4YALyqlnivrm305BgAxIhJXxnVdATsRSTO+HgS8VslraJp2A8jJL2DPyWTWRyewISaBM8mGYjodmjbg0f4BRAT70qW5dTOi2rLyutJ3ReRQKfsPAUuUUk5AmTf9lVILgH6Al1IqDnhZRL7GMCJZcM2xzYCvRGQY4AssNS42cQB+EJE1pv+TNE2ryxLTc9gYY+gQthy9REZuAc4Odtwa4MWDfdsQEeRDU4961g6zTlAipt+2V0p5Ai1E5E/LhVR1oaGhEhUVZe0wNE0zIxEh+kJa8QPm/WevIAK+DZyJCPJlQLAPvdt6Uc/JttNaWItSal9Za80qvBmnlNoEjDQeuw9IUErtEJEnzRqlpmmaUXZeATtPJLE+Jp4N0QmcT8kGoKufB08OaE9EkA8dmzWoU2ktaiNTntZ4iEiqUup+4FsReVkpVStHEJqm2a741Gxj3YQEth9PJCuvgPpO9vQJ8GLGgPb0C/LGx732FNO5EZjSQTgopZpiWNj2ooXj0TTtBlFYKBw6n1L8gPngOUNiheYN6zEh1I+IYF9ubt2o1hbTuRGY0kG8CvwKbBORvUqpNkBZC9w0TdPKlJmbz7ZjiYbaCTEJJKTloBR0b+nJM0MCiQzypb1v3cqIastM6SAuiEiXojciEquUes+CMWmaVoecu5LFhmjDA+YdJ5LIzS/E3dmB8PbeRAYbiuk0qsMZUW2ZKR3ER0B3E/ZpmqZRUCjsP3vFMOsoOoGYi4aMqP6N63PXLa2IDPIh1L8RTg43RkZUW1beQrleQG/AWyk1s8RHDQB9U1DTtGJp2XlsPZbI+ugENh1JICkjF3s7RWgrT14cFkxEsA9tvFz1rSMbU94IwglwMx7jXmJ/KnC7JYPSNK32O5OUyTpjRtTdJ5PIKxA86jnSL9CbyGBf+rbzxqO+zohqy8rsIERkM7BZKTVPRE7XYEyaptVC+QWF/H7mCuuNzxOOJ6QDEODjxr23tiYy2JfuLRvicIMU07kRlHeL6QMRmQHMNqbdvoqIjLRoZJqmWV1KZh6bjhpmHG06comUrDwc7RU3t27MnT1bEhnsQ6vGrtYOU7OQ8m4xfWf8+U5NBKJpmvWJCLHGYjrroxOIOn2ZgkKhkasTA4INaS36tPPC3UXfOroRlHeLaZ/x52ZjYr4gDIV8johIbg3Fp2maheXmF7L3VFFG1HhOJRkyogY1ceehvm2IDPalq19DXUznBmRKLqbbgM+AE4ACWiulHhSR1ZYOTtM0y0jOyL0qI2paTj5ODnb0btuY+8IMGVGbN9QZUW90pqyDeBfoLyLHAZRSbTEWDLJkYJqmmY+IcDQ+vXjW0e9nLiMC3u7O3NalKZHBvtwa0Jj6TrqYjvY/pvw2pBV1DkaxQJqF4tE0zUzyCgrZcSKp+HnCuStZAHRu7sETke2IDPKlY7MGupiOVqbyZjGNNb6MUkqtAn7C8AxiPLC3BmLTNK0KziRlsnDvGX7eF8eltBxcHO3oE+DNoxEBRAT54NtAZ0TVTFPeCGJEidfxQF/j60sYSo9qmlZL5OYXsvbwRRbuOcu244nYKegf6MOEm1rQt723zoiqVUl5s5j+Vp2GlVJzgOFAgoh0Mu57BXgAQycD8IKIrCrl3CHALAwpPb4SkTerE4um1VUnLqWzcM8ZFv9+juSMXJo3rMfMge0ZH+qny25q1WbJJ1LzgNnAt9fsf19EylxboZSyBz4GBgJxwF6l1DIROWypQDXNlmTnFbD60AUW7DnLnpPJONgpBgT7MunmlvQJ8NLTUTWzsVgHISJblFL+VTi1J3BcRGIBlFILgVGA7iC0G1rMxVQW7jnL0j/OkZKVh3/j+jw7JIhxPZrrSmuaRVhjTtujSqm7gSjgKRG5fM3nzYGzJd7HATfXVHCaVptk5uaz4sAFFuw9wx9nruBkb8fgTk2YdFMLbmnTWM9A0izKlIVyzsA4wL/k8SLyWhWu9ynwTwyzof6JYY3FvVVop2R804BpAC1btqxOU5pWaxw6l8KCPWf4Zf950nPyCfBx46Xbghnb3U8X19FqjCkjiF+AFGAfkFOdi4lIfNFrpdSXwIpSDjsHtCjx3s+4r6w2vwC+AAgNDb0uqaCm2Yr0nHz++8c5Fu49w6FzqTg72HFbl6ZM6tmS0FaeupaCVuNM6SD8RGSIOS6mlGoqIheMb8cAh0o5bC/QTinVGkPHMBG40xzX17Ta6ExSJvN2nOKnqLOk5+QT1MSd10Z1ZFRIczzq6aR4mvWY0kHsUEp1FpGDlWlYKbUA6Ad4KaXigJeBfkqpEAy3mE4BDxqPbYZhOuswEclXSj0K/IphmuscEfmrMtfWtNpORNgVm8yc7SdZFx2PvVIM79KUe3r7E9KioR4taLWCEin/roxS6jAQAJzEcItJASIiXSwfXuWEhoZKVFSUtcPQtDJl5xWw7MB55mw7SczFNBq5OjH55pZMuaWVXuGsWYVSap+IhJb2mSkjiKFmjkfTbjgJqdl8v+s083efISkjl6Am7rw1rjOjQprrVc5arVVeLqYGIpKKTsynaVV2MC6FOdtPsuLP8+QXCpFBvtx7qz+92jbWt5G0Wq+8EcQPGFJl7MPwzKDkb7MAbSwYl6bZrPyCQtYejmfOtpNEnb6Mq5M9U25pxT29/PH30uU5NdtRXi6m4cafrWsuHE2zXSmZeSzce4Zvd57m3JUsWjaqz9+Hd2B8qB8NdIlOzQaVd4vJX0ROlfO5ApqLSJwlAtM0W3E8IZ15O06yeN85svIKuKVNI14e0YHIYF+dF0mzaeXdYnpbKWWHYaHcPv6X5jsA6A9EYpi6qjsI7YYjImw9lsjX206y+eglnOztGBXSjL/d2poOzRpYOzxNM4vybjGNV0p1ACZjSIfRFMgEooFVwL9EJLtGotS0WqKwUFgXHc/sjcf5My4Fb3dnZg5sz503t8TLzdna4WmaWZU7zdWYYvvFGopF02qtgkJh1cELfLzxODEX02jZqD5vju3MmO7NcXbQ01S1uklXKNe0cuQVFPLL/vN8svE4sYkZtPV25f07ujKiSzMc7O2sHZ6mWZTuIDStFDn5BSzaF8enm04QdzmL4KYN+GRydwZ3bKIfPGs3DN1BaFoJWbkFLNhzhi+2xHIxNZuuLRryyoiORAb76IVt2g3HpA5CKdUcaMXV9SC2WCooTatp6Tn5fLfzNF9tjSUpI5eerRvxzviu3BqgVzxrNy5TCga9BdyBoeRngXG3ALqD0GxeSmYec3ecZO72U6Rk5RHe3ptH+wfQs3Ujk9vIy8sjLi6O7Gw9qU+rvVxcXPDz88PR0fRFm6aMIEYDgSJSrWJBmlabJKbn8PW2k3y38zTpOfkMCPblsYgAurZoWOm24uLicHd3x9/fX482tFpJREhKSiIuLo7WrU1PjmFKBxELOFLNanKaVhvEp2bz+eZYfthzmpz8QoZ1bsr0fgHVWtyWnZ2tOwetVlNK0bhxYy5dulSp80zpIDKB/Uqp9ZToJETk8cqFqGnWczElm9kbj/HT3jgKRBgV0oxH+gUQ4ONmlvZ156DVdlX5HTWlg1hm3DTN5qRm5/HZphPM2X6SgkLh9h4teLhvW1o2rm/t0DSt1quwgxCRb5RSTkB7464jIpJX0XlKqTkY0oUniEgn4763gRFALnAC+JuIXCnl3FMY6lAUAPllVTvStLLk5Bcwf9cZPtpwjMuZeYzs2oynBwXqjkHTKqHCpaBKqX7AMeBj4BPgqFIq3IS25wFDrtn3G9DJWK70KPB8Oef3F5EQ3TlolVFYKPyy/xwD3tvMaysO06FZA5Y/2ocPJ3Wr852Dm9vVt8vmzZvHo48+yl9//cULL7zAfffdx86dO4s/z8rKom/fvhQUGCYn2tvbExISQteuXenevTs7duwAoF+/fpw6dar4vKLjOnXqxPjx48nMzDTrvyM3N5fw8HDy8/PN2q5WeabkCngXGCQifUUkHBgMvF/RScZ1EsnX7FsrIkX/r+8C/CoZr6aVafvxREZ9vJ0nFu7HzdmRb+7tyff33UxnPw9rh2ZVTk5OZGdn4+vry3fffVe8f86cOYwdOxZ7e0MuqXr16rF//34OHDjAv//9b55/vvTvb0XHHTp0CCcnJz777DOzxxsZGcmPP/5o1na1yjOlg3AUkSNFb0TkKIZZTdV1L7C6jM8EWKuU2qeUmmaGa2l12OHzqdw9Zw+Tv9pNckYu703oysrH+tC3vbd+eAx8+OGHzJgxgwcffPCqb/vz589n1KhRpZ6TmpqKp6dnhW2HhYVx/Phxs8VaZPTo0cyfP9/s7WqVY8pD6iil1FfA98b3k4Go6lxUKfUikA+U9RvQR0TOKaV8gN+UUjFlrdw2diDTAFq2bFmdsDQbE3c5k/fWHmXp/nM0cHHkxWHB3NWrFS6O12dXzT5ylMx9UTQcNw47Z8ul5X51+V8cPp9q1jY7NGvAyyM6VnhcVlYWISEhxe+Tk5MZOXIk/fv356233sLX17f4s9zcXGJjY/H397/u/OzsbC5cuMCGDRvKvV5+fj6rV69myJBr7yRXX6dOndi7d6/Z29Uqx5QO4mFgOlA0rXUrhmcRVaKUmorh4XWkiEhpx4jIOePPBKXUUqAnZazcFpEvgC8AQkNDS21Pq1uuZOby8cbjfLPzNADTwtvwSN8APOpfPbDNu3CB1JUrSVm2nJyjRwGwc3Ki4e2313jMNaHo1k+RefPmERUVxdixYxk7duxVxyYmJtKwYcMyz9+5cyd33303hw4duu46JTuisLAw7rvvPnP/U7C3t8fJyYm0tDTc3d3N3r5mGlNmMeUA7xm3alFKDQGeAfqKSKlPtpRSroCdiKQZXw8CXqvutTXbl51XwLwdp/hk43HScvIZ192PmQPb06xhveJjClJTSf31V1KXryBz714QoV7Xrvi+9BJJX35J+tZtFu0gTPmmXxvUq1ev3NQgvXr1IjExsdSFVdd2ROaybNkyPD09CQsLAyAnJwcXFxezX0czXXk1qX8SkQlKqYMYnglcxTgTqUxKqQVAP8BLKRWHoTzp84AzhttGALtE5CGlVDPgKxEZBvgCS42fOwA/iMiaqvzjtLqhoFBY8nsc7/92lPMp2fQP9ObZoUEENTGsfi7MzSV982ZSly0nfdMmJC8PJ39/vB6djsfw4Ti1agVAdkw0ab+uRfLzUQ43diJjT09PCgoKyM7OLvWPcExMDAUFBTRu3Nik9mbOnMlTTz2Fl5cX999//1UPw8EwmvHy8mL48OFMnDiRIUOGsHnzZtq0aYOdnR0vvvgiycnJ2NkZHosmJSXh5eVVqbxBmvmV91/JE8afw6vSsIhMKmX312Ucex4YZnwdC3StyjW1ukVE2HT0Em+tjiHmYhpd/Tx4d0IIvdo2RgoLydizh9TlK0j99VcKU1Oxb9yYhpMm4jFiBC6dOl33gNqtTxgpixaTdeAA9Xv0sNK/qvYYNGgQ27ZtY8CAAcDVt45EhG+++aZ4hlNFHnnkEb744gvatGnDxIkTTTpnyJAh3HHHHUyadP2fio0bN3LbbbeZ+C/RLKW8mtQXjC8fEZFnS35mzPD67PVnaZp5RF9I5bXlh9kZm0SrxvWZfWc3buvclJxjx0h4dx4pK1aSf+ECqn593AdE4jFiJK69bil3ZODauxfY25O+dWud7CDS09Ovej916lSmTp1a5vHTp0/n/fffL+4gitZDVPY6AAEBAZw/f57o6GgWLlx43efOzs7F6xoyMjIAcHV1BQyd0bV++OEH3nzzTZPi0SzHlHH2QK7vDIaWsk/Tqi03v5BPNh1n9objNKjnyKsjOzK+pRNZa1Zx8u8ryDlyBOztce1zKz4zZ+IeGYFdfdMWwNk3aEC9kBAytmyFGTMs/C+p/bp3707//v0pKCgweaRQnsjISC5cuICdnR3x8fGsWLGi+AF23759eeaZZzh58iRXrlyXPOEqubm5jB49mvbt25d7nGZ5qoyJRCilHgYeAdpgSItRxB3YLiJTLB9e5YSGhkpUVLVm4GpWdOhcCk//fICYi2mM69CImfanyV+7hsw9e0AEl65d8BgxkgZDh+Bg4r3xayV+9hmXPphFu21bcfDyMkvc0dHRBAcHm6Wt2mjevHmMHj36ullPJW3fvp1Zs2bx9ddf4+7uzpo1a3ByciIiIqIGI9UqUtrvqlJqX1kZK8rrIDwAT+DfwHMlPkoTkeRST7Iy3UHYppz8AmZvOM4nm07QyNWJ95qn0OSbj8m/cAGnVq1oMGIEHiP+97C5OrL++otT426n2Vtv4lHGIrHKqusdhFZ3VLaDKO8ZRAqQAkwyNuIDuABuSik3ETljtqi1G9afcVd4+ucDHI1P557WTtyzbxE5323Gvl07ms2bR/2be5p1NbRLcDD2jRuTvnWb2ToITaurTCk5OgLDGohmQAKG2tTRgG1M+NZqpey8AmatP8YXW2JpUs+en9yO0ODT78i1s8Pr6afYeqsHcVm7eYyeZr2usrPDrc+tpG/eghQUoMxw713T6ipTHlK/DtwCrBORbkqp/kCte/6g2Y7fz1zmmUV/cjwhnScapTJ80/fkx8biOmAAZ+4dwPNn53F8jyG/T7hfOCE+IRW0WDmuYeGk/LKM7L/+ol6XcpfzaNoNzZRkfXkikgTYKaXsRGQjoFNwa5WWnVfAG6uiuf3THThcSWbplbUMmfMaKicH3n6RV0dm8dDBl8gtyOXfYf+mvkN9Fh9bbPY4XG/tDUqRvnWr2dvWtLrElBHEFaWUG4ZcSPOVUglAhmXD0uqaqFPJPLPoT05dSuMfHOWW1T8i2dm43DeFud1T+W/cf/Bw9uC5ns8xof0EHO0diboYxaqTq3j2pmdxczJPaVAAB09PXDp3JmPLVrynTzdbu5pW15gyghgFZAFPAmswTHkdYcmgtLojK7eA15YfZvznO2kSf4pfoudy8y9f4RQUyPY3xjG+6VJWnl/H1E5TWTl2JZODJ+Nob0ivMK7dOLLys1h9qqys8FXn1qcPWQcPkn/5stnb1rS6wpRkfRkASqkGwHKLR6TVGbtjk3hm8Z9cupjE+5d3ELjrV+w9GxL75Bj+7bGN5OT93NbmNh7v9jjN3Jpdd34nr06092zPkqNLGN9+vFljcwsPI/GTT8jcuZMGw4aZtW1NqytMmcX0IPAqkA0UAgpD8r42lg1Ns1UZOfn8Z00M3+w4xdiUaO478At2V5LJGB7GW93OEpO3nJs8b+KTHk/R0avsyXBKKca2G8ube97kSPIRAhsFmi1Gl86dsfPwIH3rNt1BaFoZTLnF9DSGOtL+ItJGRFqLiO4ctFLtOJ7IkFlbWP9bFHMPf8cDm+agfDyYNyOYv3XaQU59Bz6K+IivB31dbudQZHib4TjZOZn9YbWyt8ft1t6kb9uKFBaatW1rqama1LWBKXWrT506RadOna7a98orr/DOO+8Uvy/5v9mqVato3749p0+frnQ8/v7+dO7cmZCQEEJDr57Ds2bNGgIDAwkICLgqv1RZ+yv6rCaZ0kGcAMxblVyrc9Ky83hh6UGmfr6N0X+s5IvN79Hk/HF2TuzExNGn2dUwkb/f8neWjFxCvxb9TF785uHswYBWA1gRu4Ls/LLrF1SFa1g4BZcSDfmd6jBz16QGOHv2LK+88goPPPCAVf6Ambtu9fr163n88cdZvXo1raq4Yn/jxo3s37+fktkcCgoKmD59OqtXr+bw4cMsWLCAw4cPl7m/vHOswZQO4nlgh1Lqc6XUh0WbpQPTbMeWo5cY8sFWji5fy/wdsxj2xyrOh7bkkfsL+TTgFA+EPMiqsauYEDgBB7vK12EY124cablprDuzzqxxu97aG4D0rdvM2m5tY+6a1Lt37+axxx7j4YcfZty4caxatcpssX777bd06dKFrl27ctddd5V7rLnqVm/ZsoUHHniAFStW0LZt22q3V9KePXsICAigTZs2ODk5MXHiRH755Zcy95d3jjWY8l/r58AG4CCGZxCaBkBqdh7/WhHNuq2HmHl0Jd1P/k52s0a8c5c7e/3OMCpgFI+GPIqvq2/FjZXjpiY30dK9JYuPLmZ4myqVJymVo48PzsHBZGzZgte0B8zT6Orn4OJB87RVpElnGFrxt/SaqEmdlpbG5MmT2bZtG76+vvTr14+bbrqpwtjCwsJIS0u7bv8777xTnG78r7/+4vXXX2fHjh14eXmRnFx+yjdz1K3Oyclh9OjRbNq0iaCgoCrFDYbnZYMGDUIpxYMPPsi0adMAOHfuHC1atCg+zs/Pj927d5e5v7xzrMGUDsJRRGZaPBLNpmw8ksBLi/Zz8x/rmHvsN+wln1UDGvJ9txR6tryVn3vMNNtDZaUUY9qNYdbvsziVcgp/D3+ztAuG6a5Jc+dSkJaGvY3XPq6JmtQ///wz3bt3p0mTJgC4uLiYVBZ0qwmLEjds2MD48ePxMmbZbdSoUbnHV1S3uqzbmCX3Ozo60rt3b77++mtmzZpVpbgBtm3bRvPmzUlISGDgwIEEBQURHh5u0rm1mSkdxGql1DQMU1xzinaaktFVKTUHQ0W6BBHpZNzXCPgR8AdOARNE5LrJ6Eqpe4CXjG9fF5FvTIhVs7CUzDxeW3GYv9Zt49VDS2mefI5jwQ2Y1TcPjzZNmd3jHW5tfqvZrzuq7Shm/zGbpceX8mSPJ83Wrlt4GElffknGrl00GDiw+g2a8E2/NqhqTeojR46UehumZEnRmJgYFi5cyCuvvFL8uanfxEtTXtvl1a1u3Lgxl69Z55KcnEzr1q2L39vZ2fHTTz8RGRnJG2+8wQsvvHDV8abG3bx5cwB8fHwYM2YMe/bsITw8nObNm3P27Nni4+Li4mjevHmZ+4vaKuuzmmZKB1FUD7DkEytTp7nOA2YD35bY9xywXkTeVEo9Z3x/bcW6RhhqWIcar7VPKbWstI5EqznrDsfzr4W7GLVnKfef2k16QxfeGWvHqa6uPNr9OUa2HYm9nWWS33nX9ybcL5xfjv/Co90exdHOPLWK64WEYOfqSsbWbebpIGxEVWtSt23b9qr74YcOHSr+xl8eU76JR0REMGbMGGbOnEnjxo1JTk4udxRxbd3qyMhIvv322+I/pm5ubjRt2pQNGzYQERFBcnIya9as4Yknnriqnfr167Ny5UrCwsLw9fUtLnJkatwZGRkUFhbi7u5ORkYGa9eu5R//+AcAN910E8eOHePkyZM0b96chQsX8sMPPxAYGFjq/vLOsYZyOwillB3wnIhUaaqAiGxRSvlfs6pPDeUAACAASURBVHsU0M/4+htgE9dXpxsM/FY0SlFK/QYMARZUJQ6tei5n5PLqLwfJWL6M/0SvpH5uJitutmdZPwcmd3+QjzrcTX1H06q6Vce4duPYeHYjW+K2ENky0ixtKkdHXHv3In3rVkTErKnFa7uq1KT+29/+xo4dOxg8eDBt2rQhPDy81JrSVdGxY0defPFF+vbti729Pd26dWPevHllHl+ybnVhYSHHjx+/rkP59ttvmT59OjNnGu6Sv/zyy6WOgBo1asSaNWsIDw/H29ubkSNHmhx3fHw8Y8aMASA/P58777yTIUOGAODg4MDs2bMZPHgwBQUF3HvvvXTsaJjeXdb+8s6pcSJS7gZEVXRMBef7A4dKvL9S4rUq+b7E/qeBl0q8/zvwdEXX6tGjh2jmtfrgBRn11Dfyc+8hcjgwSH6J7CjD3+4kr+54VS5lXqrRWPIK8iTipwh5+LeHzdpu8o8/yuHAIMk+dqxK5x8+fNis8dSUffv2yZQpUyo8rm/fvnLy5Mlyj/nvf/8rc+bMERGRzZs3y6xZs8wRYrltjxkzRo4cOSIiIgcPHpQnn3zSbNesq0r7XS3vb7wpt5jWKaWexvDcoDhJn5ihqpyIiFKq9JJ2JjI+H5kG0LJly+qGpBklpefw+qJ9eC76ltdPbCGrnuLTYXYUDu3LBzfNpG1D804HNIWDnQOjA0bz1cGvuJhxkSauTczSrlufPoBhuqtzQIBZ2rQF5qxJPXjwYB588EEOHDhAYmIiH3zwgZmiLL3ta+tWd+rUiffee89s19QMyiw5WnyAUidL2S1i4mpq4y2mFfK/h9RHgH4ickEp1RTYJCKB15wzyXjMg8b3nxuPK/cWky45ah4rD5xn2cc/MDnqR7wz01nfVRE1JohH+j5Hz6bmLeBTWXFpcQxdMpTpIdN5qOtDZmv3xPDhOPr40HLOnEqfW9dLjppSk1qzDZUtOVrhQjkxpNa4dqtOqo1lwD3G1/cApa0A+RUYpJTyVEp5AoOM+zQLSkzP4bnZq0mc8QAztnxNpms6HzzgQ6s33mLOHYus3jkA+Ln7cUvTW1h6bCmFYr5lOW5h4WTujaIwUycNuNbUqVN153CDqrCDUEo5KqUeV0otMm6PKqVMmkKilFoA7AQClVJxSqn7gDeBgUqpY8AA43uUUqFKqa+g+PbVP4G9xu01c9zS0konIvwSdYqP7n+aiZ89TZfEWBYOdOH8h08y64lfGdF2BHbKlEX3NWNcu3GczzjPrgu7zNamW1gfJC+PjD17zNamptk6U55BfAo4Ap8Y399l3Hd/RSeKSFnTG66bgiIiUSXbFJE5QOXH+1qlJKRl88n78+m98lMmpmSwO9COyw+NY0b/p/B0KT3VgrVFtIygoXNDFh9dTO9mvc3SZr0ePVD16pGxZSvu/fqZpU1Ns3WmdBA3iUjXEu83KKUOWCogrWaICP/deIBL7z3PHcdPEd8QVk/vwbh7/kWrBlVLVlZTnOydGNF2BAtiFpCcnUwjl/JX3JrCztkZ1549Sd9Wt/MyaVplmHLfoEApVTxlRSnVBiiwXEiapaVk5fKfGTNo8eSd9Io9xaYIXzx+nsfMx76v9Z1DkbEBY8kvzGf5CfPVsHINDyPvzBlyq5DuWdPqIlM6iP8DNiqlNimlNmNI3PeUZcPSLOVicjo/3D2akb+u5UxTJy5+/jwPfbyRbi1utnZolRLgGUBX764sObaEimbimcotLAyo+9ldNc1UZXYQSqmiGo+xQDvgceAxIFBENtZAbJqZnYhL5Ne7htHv4El2RbTmthW7GRh2t82uHh7XbhyxKbHsv7S/4oNN4NSyJY6tWpK+dYtZ2tM0W1feCKIo99JiEckRkT+NW04552i11B/Rsey/Zyg9T1xi9+09uOfjFTg7OFs7rGoZ7D+Y+g71WXzUfNXm3MLCydy9h8Ic/WuuaeV1EElKqbVAa6XUsmu3mgpQq76NO/YR/8AY2l9I5/eHhjL19e9r1bTVqqrvWJ9hbYax9vRa0nKvz7hZFW5hfZDsbDJtcMGlLjl6tbpYcrSmS5GW91fiNuAfQCLwbimbZgOWrvgVuxn34JOay4kXpzJ5xnuQngArn4ZLtl9qc1y7cWTlZ7H65GqztFe/Z0+UkxMZW0yrA2ALdMnRitlCyVFrlCIts4MQkVwR2QX0FpHN124WjUozi3nzvqfZS0/ilF/AlXeeZ/TkZ+HsXvg8HPZ+CaufsXaI1daxcUfae7ZnybElZmnPrl496t90U52a7qpLjpbPVkqOWqMUqSnrIDyVUv/CkJW1+HgRibBUUFr1iAgfv/set37zFSmudjSY/T69egyCqDmw6hlo0AxC7zW8P70TWvWydshVppRibLuxvLnnTWKSYwhqdH3ZyMpyDetDwptvkXfuHI6VLNTy1p63iEmOqXYMJQU1CuLZntdmxL+eLjlaebZUctQapUhN6SB+Bj4DvkKvf6j1CgqFWS89y6Clyznr60C7r78hoGUHWPYY/PEdtI2EcV+BgwtEr4BNb8A95ltLYA3D2wznvaj3WHx0MS/e8mK123MLCyPhzbdI37YdzzsmmCHCmqFLjl5NlxytPlM6iHwR+dTikWjVlp2Xz6dPPMDQDbuIbl2PW79diq+TPcwdCud/h7Cnof8LUFT1rc8M+PUFOLUd/M1fJrSmeDh7MNB/ICtjV/JU6FO4OFT8B6s8Tm3a4NCsKelbt1S6gzDlm35toEuOGthSyVFrlCI1ZSrLcqXUI0qppkqpRkWbRaPSKi0lI5s5U8cxZMMu/ujkycCf1+Gbfho+7wuJx+CO+RD59/91DmC4zeTmC5v+bb3AzWRcu3Gk5aXx2+nfqt2WUsow3XXnLiQ31wzR1T4lS46WprySo3/++Wfx+0OHDnHx4sUKr7d161b2799/3Vbyj2xERAQ///wzSUlJABXeYiqt5Oi5c+eKPy9ZcrSovTVr1tDHWP+jSFHJ0fnz5/P1119XOu6MjIziTqSo5GjR7KmS5UNzc3NZuHAhI0eOLHN/Vc+xFFNGEEWpuf+vxD5Ta1JrNeBiUiqr7x1F/yMX2XWrH3d+ugznqK/ht39A4wCYOB+82l1/omM96PMkrHkOTm6F1mE1H7yZhPqG0tK9JUuOLWFE2xHVbs8trA9XfvyRzP37ce1p/TTnlqBLjv6PrZQcrfFSpGWVmrPF7UYsOXrs1Dn5cWBPORwYJF89PV7ys1JEfpoq8nIDkYWTRbJTy28gN1Pk7fYic4aKFBbWTNAW8uWfX0qneZ3k5JWT1W4rPy1NDnfsJPHvvFPhsbrkqC45aisqW3K0vFQbEcafY0vbLNttaaaIOnCYmLtuo8PZVPZMjeDeZ9/Afs5gOPxfiHwZJnwHztc/vLuKYz0Imwmnt8NJ204xMartKOyVPUuOV3/Kq72bG/W7d6/TeZlKlhytrsGDB7Np0yZmzJjBF198wZ133mmGCMtuW5ccrRnl3WLqiyExX2njdQHMM/Fcq5L1G7Zg9/x0/DLyiX56EveE3QxfRoCdHUxeBAHXldwoW/d7YNsHsPENaB0ONpqbybu+N339+vLL8V94rNtjONqZVNeqTK5hfbj07nvkxSfg6Otjpihrl3vvvbfCY0ypKOfi4sI333xjrrBMavvuu++2yPW0/ylvodzLxp9/K2Wr+LdKs5jFixZR/+mHcM3J5+K/nmR8W2f44Q7wbAXTNleucwBwdDGMIs7ugljbzsM4rv04krOT2XK2+qOhouyuGdu3V7stW6ZLjt64bD8hzw1m3uef4P/a38l3gIL3X2FIxnrDWoYud8B9aw2dRFV0vxsa+MHGf4OZ0mdbQ+9mvfGp78PiY9VP4OccGIiDt7fO7qrdsGq8g1BKBSql9pfYUpVSM645pp9SKqXEMf+o6ThrGxHh09f/QY9ZH5Hk4YDXrFfodfAtOL4Ohr4NYz4zPE+oKgdnwygibg+cWG++wGuYg50DowNGs/38di5mVDz9sjxKKVzDwsjYsRMpJymcptVVNd5BiMgREQkRkRCgB5AJLC3l0K1Fx4nIazUbZe1SWCjMfvoRwr//mRPNnen4r+kEb5gBuelwzwq4eZp5nht0uws8Wtj8KGJMwBgKpZClx0v7taoct7A+FKakkHXwoBki0zTbUmEHoZTap5SarpSyRAX7SOCEiOgaj2XIzsvns/snMGDlJg4GNqDfowNovuFZaNLJ8LzBnHmUHJwg7Ck4F2UYmdgoP3c/ejXtxdJjSykorN4MHddevcDOjgwTUy5oWl1iygjiDqAZsFcptVApNViZrwTZRGBBGZ/1UkodUEqtVkqVuRpEKTVNKRWllIq6Ni2ArUtJz+SHO4fSf8ch9vTwYfToxjT840sIvc8wcmjQ1PwXDZkMHi0NM5pseBQxtv1YLmRcYPeF6iUzs2/YkHpdu9bp6a6aVpYKOwgROS4iLwLtgR+AOcBppdSr1Um5oZRyAkZiSAZ4rd+BViLSFfgI+G858X0hIqEiEurt7V3VcGqdCwlJrJowgF4H49jVvyWTeyTiEhcFoz6G4e8Zvu1bgoMThD9tyN10bK1lrlEDIlpE0NC5IYuOLap2W65hfcg+dIj8ClI/aFpdY9IzCKVUFwxFgt4GFgPjgVQM6ySqaijwu4jEX/uBiKSKSLrx9SrAUSnlVY1r2ZQjx06w545BdIm9zO8j2jC12R84KAX3/Qrdplg+gJA7oWErQ44mGx1FONk7MaLtCDae3UhSVlK12nILCwMRMrbvMFN0mmYbTHoGAbwP7AW6iMjjIrJbRN4FYqtx7UmUcXtJKdWk6DaWUqqnMc7q/VduI/bs3cuZe8bgn5DJ8dubM9l1G6rlLTBtEzTrVjNB2DtC+P/B+T/g6JqauaYFjGs3jvzCfFbErqhWOy4dO2Lv6Vnrp7vqkqNXq8mSo/feey8+Pj7XXQ9qpuSopUqRmjKCGC8ikSLyg4hcVcldRKqUckMp5QoMpMRqbKXUQ0qph4xvbwcOKaUOAB8CE405Q+q0tatXk/vwVBpl5HFlrDujHPZC78dhyhJwreEBVNeJ4Olv088i2jZsS4h3CIuPLaY6vz7Kzg7XPn3I2LYdKSw0Y4Q1Q5ccrVh1S45OnTqVNWuu/zJVEyVHLVmK1JQO4n6lVPEySqWUp1Lq9epcVEQyRKSxiKSU2PeZiHxmfD1bRDqKSFcRuUVE6vzYftH3c/F87insENSIAvq6xMH4eTDon2BvStJdM7N3hL7PwsU/IWZlzV/fTMa2G8vJlJPsv7S/4oPL4RbWh4LkZLIPR5spspqjS46WzxwlR8PDw0stcFQTJUctWYrUlL88Q0WkuIqGiFxWSg0DXjJLBBrzPniTbl99Q7K7okX/ywT4NIWJv4BPsHUD6zwBtrwDm96EwGGGPE82ZrD/YN7a+xaLji6im0/Vb9G53mooqJSxdQv1Olk4xXIV6ZKjlWeukqNlqYmSo5YsRWpKB2GvlHIuur2klKoHOJvl6jc4EeGLv8+kz+I1nGliR9feF2nWcaBhVXS9WpD7xt4B+j4DSx+EmBXQwbLFSSyhvmN9hrYeyooTK3iu53O4O1WQ3bYMDo0b49KpE+lbt+H18MNlHnfxjTfIiTZvTWrn4CCaXFPprDS65OjVarLkaF1lylfC+cB6pdR9Sqn7gN8Ay6RtvIEUFBTy+fS7CF+0hhOtFLeEXaDZwGdg4g+1o3Mo0ul2Q9GhTW+CDd5/B8PD6uyCbFafXF2tdlzD+pC1fz8FKSkVH1zLWaLk6IoVhskAMTExV5UbBcM38ZCQkOu2desqXpBZXttVKTla1AHB/0qO7tmzhzfeeOO6NqoTN1StfGhlz7FoKdKyCkWU3DBMSX3HuA025RxrbLZSMCgrK0e+unOoHA4MkiWjAyXjdT+RI2usHVbZDvxoKEB0aKm1I6mSwsJCGfvLWJmwfEK12snYt08OBwZJyuqr/7+qLQWDXF1dr3o/d+5cmT59epnH+/n5SVZWVqnnR0dHS+PGjSU/P/+qgkGff/65DBs2rPi4gwcPyoULF2Tu3LmyfPny4nNffvnlSsd/6NAhadeunSQmJoqISFJSUvG/o7S2ExMTJTAwsPj8iIgIiYuLu6rNHj16yPr164vba9eunRw/fvy6f3NSUpJ06NBBvvrqq0rHXeTkyZPSsWPHq/bl5eVJ69atJTY2VnJycqRLly5y6NChMvdX5Zzy2rqW2QoGXdOJrBaRp43br+bpmm5MV65cYfGESHrvO0lMt3yGD/Cg/kOboP1ga4dWtk7jwKs9bH7LJkcRSinGtRvH4aTDHE6q+uyOel26YNegQa2f7mqqopKjRYqeQYSEhHDHHXeUWXLU29ubwYMH8/DDD3Pw4MHi203VVbLkaNeuXYvLhJbF1JKj//znPwkJCSEiIqLCkqOvv/46y5Ytq3TskyZNolevXhw5cgQ/P7/i2tYly4cGBwczYcIEOnbsWOb+qpxTXlvVVlbPUbQBt2BYA5EO5AIFQGpF51ljq+0jiLNxcbIksoccDgySZVNbS+FPU0Vy0q0dlmn+/Nkwiji42NqRVElKTorc9P1N8uLWF6vVztknZsjRsHApLFGetbaMICpLlxy98VhiBDEbw6K2Y0A94H7gY/N0TzeOvw79yV8ThxJwPoMz/bMY8cBM1O1zwMnV2qGZpuMY8Ao0jiKqX6KypjVwasDItiNZdXIViVmJVW7HLawP+QkJ5Bw9ZsborEOXHNUqVFbPUbRh7F2AP0vs+6Oi86yx1dYRxPYtG2TLTR3k905BsunJdiKxm60dUtUcXGQYRfz5s7UjqZLYK7HSaV4n+eSPT6rcRu7FeDkcGCSJX35ZvM9WRxCmmjt3rly+fNnaYWhmYIkRRKYxsd5+pdR/lFJPoivRmWz1kh9wePwRnPILyRvnTt+XNxjqPtuiDmPAO9hmRxGtPVoT1jyMH4/8SG5BbpXacPT1wTkw8IbK7qpLjt64TPlDf5fxuEeBDKAFMM6SQdUViz57k6Yv/5McJ3C5rzM9X9wCHn7WDqvq7Oyg37OQeBQOVb+kpzVM6TCFpOwk1pyqeo4pt7A+ZP7+OwXpGWaMTNNqH1M6iB6AiCHD6qsiMlNEjls6MFu34NUHCfzwGxIbCk1emEzX6T+BY8ULimq94FHg09EwiiiwvTKcvZr2oq1HW74//H3R7dJKc+0TBnl5ZO4xz2pVTautTOkgRgBHlVLfKaWGK6WskBjItiyYOY6QBVs42xw6fPg27UbVoawkdnbQ7zlIOg6Hql9roaYppZjcYTLRydHsi99XpTbqd++GXf36pG+pG9NdNa0sphQM+hsQgKGwzyTghFLqK0sHZquWzX6RLqsPc7SNHb2/+YVm3UZYOyTzCxoOvp1tdhQxvM1wPJw9mB9dtURvysmJ+r16kbF1W/EopKqjEU2rKVX5HTV1oVwesBpYCOwDRlf6SjeA37euoNlXS4hvDLd88iMNm7W3dkiWUTSKSI6Fgz9ZO5pKq+dQj/Htx7Ph7Abi0uKq1IZbWB/yzp0j9+QpXFxcSEpK0p2EVmuJCElJSSblzSqpwttFSqmhGOpS9wM2AV8BEyofYt12KT6OK88/gzvg/tIz+PpfXzikTgm6DZp0gc3/MWR9tUZK8mq4I/AO5h6ay4KYBfzfTf9X6fNd+4QBkL5lM36TJxMXF3dd3iJNq01cXFzw86vcJBlT/qu+G/gReFCuKRikGRQUFLD9/ttolyScuK8PI4f8zdohWZ5S0O95WDgJ/lxYM6VQzaiJaxMGtRrEkmNLeCTkEVwdK7dg0cmvOS4dO3J5/g80mjyZ1q1bWyhSTbMeU55BTBKR/+rOoWw/Tx9C4LFcDvb1YuTTX1o7nJoTOBSahhhGEQV51o6m0qZ0mEJ6Xjq/HK9acRWvxx4l7+xZrixeUvHBmmaDyuwglFLbjD/TlFKpJbY0pVRqdS+slDqllDqolNqvlIoq5XOllPpQKXVcKfWnUqp7da9pCf/94Ck6b4ojJtCe8R+tt3Y4NatoFHHlNBwotbx4rdbFuwtdvLswP3o+hVL5JIRufftSr1s3Ej/5hMJyUmdrmq0qs4MQkT7Gn+4i0qDE5i4iDcx0/f4iEiIioaV8NhRoZ9ymAZ+a6Zpms3fdYlrMXcUFbwj7ZCn2jk7WDqnmtR8MzbrDlrchv2qrk61pSvAUzqSdYWtc5QvDKKXwfnIG+QkJXP7B9jpITatIubeYlFL2Sinzlscy3SjgW2O6kF1AQ6VUUyvFcp34c6fI+MdLFNhB41f+jlfzdtYOyTqKRxFnYH/16wPXtAGtBuBT34fvor+r0vmuPXvieuutJH3xBQXp6WaOTtOsq9wOQkQKgCNKqZYWuLYAa5VS+5RS00r5vDlwtsT7OOO+qyilpimlopRSUTU1iyQ/N49d00bidRkS7omgW6T5MlfapHYDoXkobH3X5kYRjnaOTAqaxO4Luzl2uWoZWr1nzKDgyhWS5+lCi1rdYso6CE/gL6XUeqXUsqLNDNfuIyLdMdxKmq6UqlIGOxH5QkRCRSTU29vbDGFV7Ofpg2l/Io9DEU0YPkNnPkcp6P88pJyFP6r2Tdyabm93Oy72LlVeOFevcyfcBw4kee5c8q8pcalptsyUDuLvwHDgNeDdElu1iMg5488EYCnQ85pDzmFIDFjEz7jPqpa+/RghWy9wONiR8bPWWjuc2qNtJPj1NI4ibGvCW0OXhgxvO5zlJ5aTnJ1cpTa8n3icwsxMkr7USQa0uqO8WUwuSqkZwHggCNguIpuLtupcVCnlqpRyL3oNDAIOXXPYMuBu42ymW4AUEblQnetW165fF9Dqu3XE+ULEp8uwd3C0Zji1S9EoIvUc/P6ttaOptCnBU8gtzGXR0arll3IOCMBj5Eguz59PXny8maPTNOsobwTxDRAKHMRwG6jao4YSfIFtSqkDwB5gpYisUUo9pJR6yHjMKiAWOA58CTxixutX2vkzx8h55TXyHMDnn6/j2cTfmuHUTm36Q4tbYOt7kGdb0z7bNmxL72a9+THmR/KquKbD67FHkcJCEj+tdRPuNK1KyusgOojIFBH5HLgdCDPXRUUkVkS6GreOIvIv4/7PROQz42sRkeki0lZEOovIdWslakpebi5RD42lUQok3TeMruG6HEapikYRaedtchQxOXgyCVkJrD1dtVuHTn5+eI6/nSuLFpN75oyZo9O0mldeB1H8NUpEbC9lpxktfmgA7WLzOTzQj2HTzTmQqoNa94WWvWGb7Y0i+jTvg38Df747/F2VE+81fughlIMDl2bPNnN0mlbzyusgupZcPQ10MedKalux5I1pdN1xib86OnH7u6utHU7tVzyKuAD75lk7mkqxU3ZMDp7MX0l/ceDSgSq14ejjQ6O7ppC6fAXZR4+aOUJNq1nlraS2v2b1tIMFVlLXatuWz6X1gq2cbQIDP1+JvYNtZSy1mtbh0KqPcRSRZe1oKmVk25G4O7nzffT3VW6j8X33YefqyqUPPzRjZJpW80yqB3EjOnviEIX/+g85TuD373fw8LLhWtLW0P95SI+HqDnWjqRS6jvWZ1y7caw7vY4L6VWbNGffsCGN77uX9HXryfrzTzNHqGk1R3cQpcjJzmL/9Il4pkLKQ2Po0Os2a4dke/z7gH8YbPsAcjOtHU2lTAqahCAsOFL1/Eqed92NfaNGXPrgAzNGpmk1S3cQpVj6YCQBpwo4PLQ1gx94w9rh2K7+L0BGAkR9be1IKqWZWzMiW0ay6OgiMvOq1rnZu7ni9eA0MnbsJGPXLjNHqGk1w6QOQinVSik1wPi6XtEit7ro59em0nX3Zf7q7MKEt1dYOxzb1qo3tOlnHEVkWDuaSrmrw12k5aaxIrbqvwMNJ07EoUkTLr3/gS5HqtmkCjsIpdQDwCLgc+MuP+C/lgzKWjYt+ZR2P+/mdDPF4M/XoOz0AKva+r0AmYmw17ZSUIR4h9ChcQe+j/6+SrUiAOycnfGa/ghZBw6QvnGTeQPUtBpgyl/A6cCtQCqAiBwDfCwZlDWcjvkD+7c+JMsZ/P8zC/dGvtYOqW5oeTO0jYDtsyDHdtJhK6WYEjyFkykn2XF+R5XbaTh6NE6tWnHpgw+Qwqp1NJpmLaZ0EDkiUpzDWSnlgCFVd52RlZHGocen4JEOGY9OIih0oLVDqlv6vQCZSbDXtsqxDvEfglc9L74/XPUpr8rREa/HHyPn6FFSV+l1NJptMaWD2KyUegGop5QaCPwMLLdsWDVr2YMDaHOmkJjb2jNg6j+sHU7d0+ImCBgA2z+EnDRrR2MyR3tHJgZOZPv57cReia1yOw2GDsU5MJBLH32I5Nle7W7txmVKB/EccAlD0r4HMSTRe8mSQdWkMzFRtIpO5VBIPSa8VScfrdQO/V6ArGTY84W1I6mU8YHjcbJzqnKtCABlZ4f3jCfIO32GK0uXmjE6TbOsCjsIESkUkS9FZLyI3G58XWduMbUMCqXh5x8y7Iv1hjQRmmX49YB2g2DHR5BtO5laGrk04rY2t7HsxDJSclKq3I5bv37UCwkh8eNPKMyxrXoZ2o2rvHoQB5VSf5a11WSQlhYUOhDXBp7WDqPu6/ccZF2G3Z9XfGwtMjl4MtkF2VWuFQGGh97eTz5Jfnw8lxdUfQGeptWk8kYQw4ER5WyaVjnNe0D7IbDzI8iu+rfxmhbYKJCbm9zMgpgF5BVW/RmC6809ce3dm6TPv6Ag3bbWhWg3pvKS9Z0u2oAcoCvQBcOsptM1FaBWx/R7ztA57PrM2pFUyuTgycRnxrP+zPpqteP95AwKLl8m+dtvzBSZplmOKQvl7sdQ9W0shsJBu5RS91o6MK2OatYNAofBzo8h64q1ozFZuF84LdxbVGvKK0C9zp1xHziA5Dlzyb982UzRaZplmDKL6f+AbiIyVUTuAXoAz1b1gkqpFkqpjUqpw0qpv5RST5RyTD+lVIpS4FZ9QAAAF5ZJREFUar9x03NP65J+z0FOCuyyndKc9nb2TA6ezIFLBzh46WC12vJ+/HEKMzJI/tq2clRpNx5TOogkoOTk9TTjvqrKB54SkQ7ALcB0pVSHUo7bKiIhxu21alxPq22adoWg4bDrE8NDaxsxOmA0bo5u1aoVAeDcrh0eI0eQ/P188uITzBSdpplfebOYZiqlZgLHgd1KqVeUUi8Du4Aql8oSkQsi8rvxdRoQDTSvanuajer3POSkws5PrB2JyVwdXRkdMJq1p9YSnxFfrba8Hn0Uyc8n6XPbehaj3VjKG0G4G7cTGJLzFa19+AU4aY6LK6X8gW7A7lI+7qWUOqCUWq2U6lhOG9OUUlFKqahLly6ZIyytJjTpBMEjDbeZMpOtHY3J7gy+kwIp4McjP1arHacWLWg4/nYu//QzuWfPmik6TTMvZa01b0opN2Az8C8RWXLNZw2AQhFJV0oNA2aJSLuK2gwNDZWoqCjLBKyZX/xf8GlvCHsaIv9u7WhM9sSGJ/g94Xd+u/03XBxcqtxOXnwCJwYNosGQITR7600zRqhpplNK7ROR0NI+M2UWk7dS6m2l1Cql1IairZoBOQKLgfnXdg4AIpIqIunG16sAR6WUV3WuqdVCvh2hw2jY/ZlNjSKmdJjClZwrrIxdWa12HH198JwymZRly8g5dsxM0Wma+ZjykHo+EAO0Bl4FTgF7q3pBpZQCvgaiReS9Mo5pYjwOpVRPY5zVeTCu1Vb9njMUE9rxkbUjMVmobyiBnoF8H/19tQsBNb7/fuz+v70zD5OiOtf47+2eHZRNXAAVWYzbRaMxxiUumLjEPS7gFaMx9xpycxM1EgkxMZiQxLgkuFwDURGNxmiI+4a44kIENxTUCCoiCIoLOsw+3d/945yeqenp2XvoHub8nqeeOlV16tRbVd311jlV9Z2yMtZd1XP2P9B7aI9BDDKzG4A6M3vKzM4CxnZhm/sDpwNjI6+xfkvSREkTfZ6TgCWSFgNXAeM3pfhPgQhb7gy7nuDCb1R8nGs17UISE3aZwPL1y1nwwYIulVUwYAADz/ou5fPmUfVa116fDQSyTXsMIhVbYI2koyR9GRjY2Q2a2TNmJjMbE3mN9UEzm2FmM3yea8xsVzPb3cy+Zmad77ElkP8cNBnqKuG5q3KtpN0cucORbFm2JZOemsSj7z3apbIGnnEm8QEDWDf9yiypCwSyQ3sMYpqkfsD5wCTgeuDcblUV6F1suRPsdiIsvA429Iw30Yrjxdx0xE1sv/n2nPfkefzu+d9Rm6hte8UMxPv2YdDZZ1Px7LNUPL8wy0oDgc7TnnDf95vZ52a2xMwOMbO9gJEbQVugN3HQZKivhud6zl30sM2GcfORNzNh5wnc9uZtnP7Q6bz/RedeWR1w6ngKttqKtb/+NdVvdfozo0Agq7SnBpGJn2RVRSAweEf4j5Nh4fWwoed8XVwYL2TyVydz5SFX8n75+5xy/ynMXTG3w+XESkrYZto0Ep98wrvfPpGP/jSdZHV1NygOBNpPZw0i9KwTyD4HXgCJGni259QiUozdbixzjpnDiH4jmPTUJKb9axo1iY51DNT36wcw4qEH6XfUUXwycybvHHccFQu69hA8EOgKnTWI8EZRIPtsMQrGjINFN0B510JZ5IIhfYcw+4jZnLHLGdz+79uZ8OAE3vuiY5HxCwYMYMgfLmG7G2cBsPK7Z/HB5J+FyK+BnNBaLKZySV9kGMqBIRtRY6A3ceBPIVELz07PtZJOURgvZNLek7hm7DWsqVjDKfedwoPvPNjhcvrsuy8j7rmHQRO/z+cPPMA7R36L9Xfd3eXvLgKBjtBah0GbmdnmGYbNzKxgY4oM9CIGjYTdx8MLs6B8ba7VdJqDtj2IOcfMYccBOzL56clcvOBiqus79kwhVlLClueey4i77qRohx1YM2UKK797FrUrVnSP6EAgjc42MQUC3ceBkyBRB09n/NC+x7B1n62ZdcQsztrtLOa8NYfTHjyNdz/veJzL4tGj2f7WW9h66q+oXrqUd449jo9nzMBqO/dabSDQXoJBBPKPgSNgj/+EhTPhb+Nh1Yu5VtRpCmOFnLfXeVx76LWsq1zHuPvHcd/b93W4HMViDBg/nhEP3E/fsWNZN/1K3j3xRCpfeqkbVAcCjpxFc+0OQjTXTYjaCtct6YL/g+r1MHKsez6x/X65VtZpPqz4kAvmX8BLH73ECaNOYMo+UygtKO1UWeVPPMHa3/yG+g/W0H/cOLY8/yfEN988y4oDvYHWorkGgwjkNzXl7q2mBddAxTrY/gDXBDXiYFDPe9u6PlnPta9cy/WvXc/I/iO5/KDLGdm/c9+dJisqWHfV1Xz6178SHzSQrS+8kM0OPxz1wOMSyB3BIAI9n9pKeOkm941E+RoYtrerUYw+rEcaxXMfPMeUp6dQVV/Fz/f5OcePOr7TZVUtWcraiy6i+vXX6XvQQWx90S8pHBo6aQy0j2AQgU2H+hp4+RZ4Zjp8vhK2HuOMYqejIdazHqmtq1zH5Kcns2jtIo4deSwX7nMhZYVlnSrL6uv59JZbXNhwMwb/+McMPH0CKggvHAZaJxhEYNMjUQev3gFPXwGfvg2Dd3ZNT7ueALF4rtW1m0QywcxXZzJj8QyG9xvOxftdzG6DdqMwXtip8upWr2btb6ax4cknKdllF7b65S8oHjWKWGlpMItARoJBBDZdkglYehfMvxzWvQEDR8LXz4cxp0AnL7K54Pk1zzN5/mQ+qf6EAhUwvN9wRvUfxegBoxvGQ/sOJaa2a0lmRvncR1j722kk1jX2saHCQlRWRqy0tGFQWSmx0rKm02VljfPKSlGpz1Pm86RNx0pLUVFRdx6eQDcSDCKw6ZNMwpv3w/zLYO2r0H87OOA82OM0KCjOtbp2sb56Pc998BzL1i9j+WfLWbZ+Gas3rG5YXlpQyqj+o5oZxxalmXvjTXzxBeXz5pEoL8eqqkhWVpKsrCJZVUWyqhJrSDefto4GCiwsbNV4mhhNaSmxPmVNpxvMJ814ysqcsfXA50w9hWAQgd6DGSybB/MvhVWLYLMhsP85sOd3oKhz7fu5pKKuguXrlzcYxrLP3PBZTWNspoElA5uZxqj+o+hT2KfT27VEgmRVNVZV2WgilZXOaKqqvNFEpisqG42mYbk3m6rKtOmqjomJx72JlDWYRovG06TG03KtR748FRX1evPJO4OQdARwJRAHrjezS9KWFwM3A3vh+qIeZ2Yr2io3GESgATN49yl46jJ47xnoMxj2mQiDd3JGUdQXivr4oS8UlkFhaY95I+rjqo+bGEdqXFXfePEd2ncowzcfTlG8iIJYAXHFicfixBVvnI6m05f56VS6QAUZ80TnR9dpqdwYoqAmQaymllh1LbGaOlRVC9U1xKrdmKpqrNobTWVlu2o9yaoqrLKyYwcyFmta4ykra7EW1GS6rDRiVs1rPbHSUlRc3CPMJ68MQlIceAv4JrAKWAScamavR/L8DzDGzCZKGg+cYGbj2io7GEQgI+89555RvP1YGxmVZhzpQ2RZYWpcAvFi14xVUAwFJRAvapyOLmuSLsq6GSUtyQcbPmDZZ8tYvn45yz5bxsryldQl60gkEyQsQX2ynoQlGqZT6XqrbzIvH0g3npYMqomJWZyShCipE8X1UForiuuM4joorjOKalNDkqLaJIW1SQprEm5cm6CgJkFBTT0FtfXEayJDdR2xmlrUgculxQTFxVBaDCUlqNQPZaWopLFGFCsrJV7ah1hZGfGyMgrK+hBvGBpNK9r8ppKSrJlPvhnEvsBUMzvcT08BMLPfR/LM9XkWSCoA1gKDrQ2xwSACrfL5Kqj81H2lXVsBtRtcX9ipdMP8ihbyRPJ1snvRJmQyj1hBzmsxhpGAhqE+lRYkIsvqG+al8liz+W7aSAL1kXlN1lFkGw3rN9XQtFxrpiu6/USz7UfWyTQ/TVcqX/MDYxTVQ3EdlNTiTQdK6ozi2lQ6Mr/WGqdro8ssQxkQ68ClOAnUFUKtHyo3i3HYo0vbX0CE1gwiF++9DQWi/TKuAvZpKY+Z1Uv6HBgEfJyWD0lnA2cDbLfddt2hN7Cp0G+YG7JBos6ZRX1N45CIpOurnYm0uqwa6mubLkvWZUdfFxDuwtCbX4o1c6aWwKhvMJ5U2khYynTMz49MW9PpJutbY/710fXNSNYnodYNVmdQl0S1yYZxrNZQXZJYnRGrNeJ1ST82VNg9Z6vH/wbM7C/AX8DVIHIsJ9BbiBdC6YBcqwh0E8I9II0DvfkF3lx8eroa2DYyPczPy5jHNzH1wz2sDgQCgcBGIhcGsQgYLWkHSUXAeODetDz3Amf49EnA4209fwgEAoFAdtnoTUz+mcL/AnNxNbhZZrZU0q+BF8zsXuAG4K+SlgOf4kwkEAgEAhuRnDyDMLMHgQfT5l0USVcDJ29sXYFAIBBopGeFvwwEAoHARiMYRCAQCAQyEgwiEAgEAhkJBhEIBAKBjGxS0VwlrQPey7UOzxZk+PI7z8h3jfmuD/JfY77rg6AxG3RF3/ZmNjjTgk3KIPIJSS+0FN8kX8h3jfmuD/JfY77rg6AxG3SXvtDEFAgEAoGMBIMIBAKBQEaCQXQff8m1gHaQ7xrzXR/kv8Z81wdBYzboFn3hGUQgEAgEMhJqEIFAIBDISDCIQCAQCGQkGESWkbStpCckvS5pqaRzcq0pE5Likl6WdH+utWRCUn9JcyS9KekN31Vt3iDpPH9+l0i6TVJJHmiaJekjSUsi8wZKmidpmR/ntJejFjRe5s/zq5LuktQ/n/RFlp0vySRtkQttER0ZNUr6kT+OSyVdmo1tBYPIPvXA+Wa2C/A14IeSdsmxpkycA7yRaxGtcCXwsJntBOxOHmmVNBT4MfAVM9sNF7Y+H0LSzwaOSJv3M+AxMxsNPOanc8lsmmucB+xmZmOAt4ApG1tUhNk014ekbYHDgJUbW1AGZpOmUdIhwHHA7ma2K3B5NjYUDCLLmNkaM3vJp8txF7ahuVXVFEnDgKOA63OtJROS+gEH4voFwcxqzWx9blU1owAo9T0elgEf5FgPZjYf139KlOOAm3z6JuD4jSoqjUwazewRM6v3k//C9TKZE1o4hgB/Ai4Acv5WTwsafwBcYmY1Ps9H2dhWMIhuRNJw4MvA87lV0ozpuB97MtdCWmAHYB1wo28Gu15Sn1yLSmFmq3F3aCuBNcDnZvZIblW1yFZmtsan1wJb5VJMOzgLeCjXIqJIOg5YbWaLc62lFXYEvi7peUlPSdo7G4UGg+gmJPUF/gmca2Zf5FpPCklHAx+Z2Yu51tIKBcCewJ/N7MtABblvGmnAt+MfhzOyIUAfSRNyq6ptfLe9Ob8DbglJF+KaaG/NtZYUksqAnwMXtZU3xxQAA3HN2j8F7pCkrhYaDKIbkFSIM4dbzezOXOtJY3/gWEkrgL8DYyXdkltJzVgFrDKzVM1rDs4w8oVvAO+a2TozqwPuBPbLsaaW+FDSNgB+nJWmh2wj6UzgaOC0POt/fiTuRmCx/88MA16StHVOVTVnFXCnORbiWge6/DA9GESW8a59A/CGmf0x13rSMbMpZjbMzIbjHqw+bmZ5dfdrZmuB9yV9yc86FHg9h5LSWQl8TVKZP9+HkkcP0dO4FzjDp88A7smhloxIOgLX5HmsmVXmWk8UM3vNzLY0s+H+P7MK2NP/RvOJu4FDACTtCBSRheizwSCyz/7A6bg781f88K1ci+qB/Ai4VdKrwB7A73KspwFfs5kDvAS8hvsf5TwUg6TbgAXAlyStkvQ94BLgm5KW4Wo+l+ShxmuAzYB5/v8yI8/05RUtaJwFjPCvvv4dOCMbNbEQaiMQCAQCGQk1iEAgEAhkJBhEIBAIBDISDCIQCAQCGQkGEQgEAoGMBIMIBAKBQEaCQfQifCTKKyLTkyRNzVLZsyWdlI2y2tjOyT666xPdva02dKzoalRPSVMlTfLpEh9tdWqGfHmxz+1F0oYurNtwXCU914VyzpQ0pLPrBxzBIHoXNcC3cx2uOB0f8K69fA/4bzM7pLv0bGwkFeG+vH/RzKZmyJJxnzt43HocZtaVr9PPxIVBCXSBYBC9i3rcB13npS9IrwGk7gIlHeyDf90j6R1Jl0g6TdJCSa9JGhkp5huSXpD0lo/5lOp34jJJi3y8/+9Hyn1a0r1k+Epa0qm+/CWS/uDnXQQcANwg6bK0/NtImu8/tFoi6et+/p+9pqWSLo7kXyHp9z7/C5L2lDRX0tuSJkY0zpf0gKR/S5ohqdl/RtIEfzxekTTT73PcH9Mlfj+aHXNPAXA7sMzMmsWbSt9nf2d8r6THgcfk+nu42x/bf0ka49ebKukmf4zfk/RtSZd6LQ/LhYNJ39ZIv+xFv95Ofv4xckHgXpb0qKSt/Py+km70Zb4q6cRIWb+VtNhrahYgsLV1I3k2RNI/jfyGLvbzhvua1XX+/D4iqdT/jr+C+9DyFUmlLRz7QFuYWRh6yQBsADYHVgD9gEnAVL9sNnBSNK8fHwysB7YBioHVwMV+2TnA9Mj6D+NuOkbjQhKUAGcDv/B5ioEXcLFtDsYF4dshg84huHAWg3EX0MeB4/2yJ3H9MKSvcz5woU/Hgc18emBk3pPAGD+9AviBT/8JeBX3Ne9g4MPIvlcDI/z681LHyK+/BbAzcB9Q6OdfC3wH2AuYF9HXP4Pmqbiwzbe3cd4a9hl3Z7wqsl9XA7/y6bHAK5GynwEKcf1pVAJH+mV3pY5n2nYeA0b79D64MCwAA2j8qPa/gCt8+g+p85/K58cGHOPTl6bOf9q2Wlp3BbBF2m/wMNyNjXC/r/tx4eCH42569vD57gAmtPY7CUPHhk26ihpojpl9IelmXIc3Ve1cbZH5kNGS3gZSoa1fw8d/8dxhZklgmaR3gJ1wf+4xkdpJP5yB1AILzezdDNvbG3jSzNb5bd6KuyDc3ZpGYJa/M77bzF7x80+RdDbOaLYBdsGZAbg4Ran96Guu/45ySTVq7NVsoZm943XchrubnxPZ7qE4M1gkFzyzFBcQ7z5c6IOrgQcixyydZ4D9JO1oZm+1sn9R5plZqj+AA4ATAczscUmDJG3ulz1kZnWSXsMZ3MOR/R0eLVAu+vB+wD/UGAS02I+HAbfLBfsrAlLn7BtEOkoys898shZ3EQd4Efhmhn1oad1MHOaHl/10X9xvaCUuaGLqXL+Yvl+BrhEMoncyHRdH6MbIvHp8k6NvRimKLKuJpJOR6SRNf0PpcVsMd9f3IzObG10g6WBcDSIrmNl8SQfiOkKaLemPwNO4WtLeZvaZpNm4Wk2K6H6k72NqvzLtUxQBN5lZs17QJO0OHA5MBE7B9XWQznxcRz4PSTrAGvtuaI32HrdU5zFJSXXmb61pft7Anfv1ZrZHhnKuBv5oZvf68za1je1Gt5XIsK2OIuD3ZjazyUzX30r0vCVwBh3IEuEZRC/E333egXv4mWIF7k4Y4Fhc00RHOVlSTO65xAjg38Bc4AepNm9JO6rtzn8WAgdJ2kJSHDgVeKq1FSRtj2saug7XU96euOa0CuBz3w5+ZCf26auSdvCmOQ53xx/lMeAkSVt6HQMlbS/3IkDMzP4J/IJWwpX7PJcDD6vj/TE/DZzmt30w8LF1ov8Rv867kk72ZckbHLha32qfPiOy2jzgh6kJday/646sOxc4y9dykDQ0dbxboRzXZBjoAsEgei9X0DRe/HW4i/JiYF86d3e/EndxfwiYaGbVuIv167gY+kuAmbRxR+nvon8GPAEsxr3d01aY6oNxMftfxl3IrzTXA9jLwJvA34BnO7FPi3DRRt/ANa3clab1dZwBPCIXeXYerilrKPCkpFeAW2ijn2Uz+7Mv+15JJa3lTWMqsJff9iU0vYB3lNOA7/nfwFJcp0ipbfxD0os0DSE9DRgg9yB+MU2bG9ui3eua663vb8AC31w2h7Yv/rOBGeEhddcI0VwDgRbwd+STzOzoXGsJBHJBqEEEAoFAICOhBhEIBAKBjIQaRCAQCAQyEgwiEAgEAhkJBhEIBAKBjASDCAQCgUBGgkEEAoFAICP/D8KaMuJbqsqZAAAAAElFTkSuQmCC\n" 311 | }, 312 | "metadata": { 313 | "needs_background": "light" 314 | } 315 | } 316 | ] 317 | }, 318 | { 319 | "cell_type": "markdown", 320 | "source": [ 321 | "# Dry bean" 322 | ], 323 | "metadata": { 324 | "id": "DP0fdL00-wYq" 325 | } 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": null, 330 | "metadata": { 331 | "id": "xBPJ8vmAEiJQ" 332 | }, 333 | "outputs": [], 334 | "source": [ 335 | "K = 4\n", 336 | "N = 4\n", 337 | "B = f(N+K-1) // f(K) // f(N-1)" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "source": [ 343 | "# calculate H(B|Y=y), y is uniform\n", 344 | "p_y = 0\n", 345 | "P_ys = []\n", 346 | "b_count = 0\n", 347 | "H_y = 0\n", 348 | "\n", 349 | "for c0 in range(0, N+K-1):\n", 350 | " for c1 in range(c0+1, N+K-1):\n", 351 | " for c2 in range(c1+1, N+K-1):\n", 352 | " for c3 in range(c2+1, N+K-1):\n", 353 | " b_count += 1\n", 354 | " Ns = [0, 0, 0, 0, 0, 0]\n", 355 | " Ns[c0-0] += 1\n", 356 | " Ns[c1-1] += 1\n", 357 | " Ns[c2-2] += 1\n", 358 | " Ns[c3-3] += 1\n", 359 | " p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5]))\n", 360 | " P_ys.append(p_y)\n", 361 | " H_y += -p_y * math.log2(p_y)\n", 362 | "\n", 363 | "# print(sum(P_ys))\n", 364 | "# print(b_count)\n", 365 | "print('B', B)\n", 366 | "print(math.log2(B))\n", 367 | "print('H_y', H_y)\n", 368 | "print(2**H_y)" 369 | ], 370 | "metadata": { 371 | "colab": { 372 | "base_uri": "https://localhost:8080/" 373 | }, 374 | "outputId": "7c0c2d06-1666-4b67-8c66-130dfece1472", 375 | "id": "ObFHHIEsEiJa" 376 | }, 377 | "execution_count": null, 378 | "outputs": [ 379 | { 380 | "output_type": "stream", 381 | "name": "stdout", 382 | "text": [ 383 | "B 35\n", 384 | "5.129283016944966\n", 385 | "H_y 4.81510800723783\n", 386 | "28.150877863635262\n" 387 | ] 388 | } 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "source": [ 394 | "# calculate H(B|S=s)\n", 395 | "U_range = np.arange(1, 1176, 110)\n", 396 | "# U_range = np.arange(1, B, 3000)\n", 397 | "H_ss = np.zeros_like(U_range).astype(float)\n", 398 | "\n", 399 | "for i, U in enumerate(U_range):\n", 400 | " H_s = 0\n", 401 | " p_s = 0\n", 402 | " choices = np.random.choice(B, U, p=P_ys)\n", 403 | " # print(len(choices), choices)\n", 404 | " for u in range(U):\n", 405 | " p_s = sum([1 for c in choices if c == u]) / U\n", 406 | " # print(p_s)\n", 407 | " if p_s != 0:\n", 408 | " H_s += -p_s * math.log2(p_s)\n", 409 | " print(H_s)\n", 410 | " H_ss[i] = H_s\n", 411 | "\n", 412 | "print('H_ss', H_ss)" 413 | ], 414 | "metadata": { 415 | "colab": { 416 | "base_uri": "https://localhost:8080/" 417 | }, 418 | "outputId": "d4fe9fbc-70dd-4ccb-9947-13b5786c4743", 419 | "id": "z8SJ6CCVEiJb" 420 | }, 421 | "execution_count": null, 422 | "outputs": [ 423 | { 424 | "output_type": "stream", 425 | "name": "stdout", 426 | "text": [ 427 | "0\n", 428 | "4.488025774313082\n", 429 | "4.702617634802406\n", 430 | "4.802929804553897\n", 431 | "4.750954940629122\n", 432 | "4.744101917925756\n", 433 | "4.8117979838104485\n", 434 | "4.768437327163887\n", 435 | "4.815896365695055\n", 436 | "4.789852490143935\n", 437 | "4.782352207555449\n", 438 | "H_ss [0. 4.48802577 4.70261763 4.8029298 4.75095494 4.74410192\n", 439 | " 4.81179798 4.76843733 4.81589637 4.78985249 4.78235221]\n" 440 | ] 441 | } 442 | ] 443 | }, 444 | { 445 | "cell_type": "code", 446 | "source": [ 447 | "H_ss = np.array([0., 4.48802577, 4.70261763, 4.8029298, 4.75095494, 4.74410192, 4.81179798, 4.76843733, 4.81589637, 4.78985249, 4.78235221])\n", 448 | "H_y = 4.81510800723783\n", 449 | "plt.plot(np.arange(11)*1176/11, H_y-H_ss)\n", 450 | "plt.xlabel('Number of clients')\n", 451 | "plt.ylabel('Privacy leakage (in bits)')\n", 452 | "plt.show()" 453 | ], 454 | "metadata": { 455 | "colab": { 456 | "base_uri": "https://localhost:8080/", 457 | "height": 281 458 | }, 459 | "outputId": "c9ab6df1-93a5-481a-a404-dfc8767a4395", 460 | "id": "7YxzWs4OEiJb" 461 | }, 462 | "execution_count": null, 463 | "outputs": [ 464 | { 465 | "output_type": "display_data", 466 | "data": { 467 | "text/plain": [ 468 | "
" 469 | ], 470 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEICAYAAABVv+9nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xcdX3/8ddnLnudSTab3RkgJATYWSiiqI2K2lbwfq+1+vvJrz/1pyj6q7XS1p/VX2utbf+w9UpbW6Xgtdb7Da2K/ARErQWSCAISkhBCIAV2E5LsLXub/fz+OGcmk2WzO7vZM5cz7+fjMY89t5nzOZzwme+c8znfr7k7IiISP4l6ByAiItFQghcRiSkleBGRmFKCFxGJKSV4EZGYUoIXEYmpVJQfbmZ7gVGgCMy6+5Yo9yciIsdEmuBDF7v7gWo27Ovr882bN0ccjohIfGzbtu2Au/cvtK4WCb5qmzdvZuvWrfUOQ0SkaZjZ/SdaF/U1eAd+aGbbzOyyiPclIiIVom7B/4a77zezHHCdme1w95sqNwgT/2UAmzZtijgcEZHWEWkL3t33h3+HgG8CT11gmyvdfYu7b+nvX/AykoiIrEBkCd7Mus0sW5oGng/cGdX+RETkeFFeoskD3zSz0n7+zd1/EOH+RESkQmQJ3t33ABdE9fkiIrI4PckqIhJTTZ/gZ4pz/NONu7lp53C9QxERaShNn+BTCeOTP97DD+56uN6hiIg0lKZP8GbGYD7D7kfG6h2KiEhDafoEDzCQy7JzaBSNLysickwsEnwhl+HwxAwHxqbrHYqISMOIRYIfzGcB2DU0WudIREQaRywSfCGfAWD3kK7Di4iUxCLB57LtZDtS7HxELXgRkZJYJHgzo5DLsEuVNCIiZbFI8BBch9clGhGRY2KT4AdyGQ6OT3NwbKreoYiINITYJPhCuZJGrXgREYhRgh8MK2mU4EVEArFJ8Kes6SDTnmK3KmlERIAYJXgzYyCXYacqaUREgBgleAgu0+gSjYhIIFYJvpDLcmBsikPj6pNGRCRWCX5AN1pFRMpileALuVKC141WEZFYJfgNPZ10tyXVZYGICDFL8KVKGrXgRURiluAhGN1JLXgRkRgm+MF8hqHRKY5MzNQ7FBGRuopdgi8P/jGsyzQi0tril+BzQadjeqJVRFpd7BL8hp5OOtOqpBERiV2CTyRUSSMiAjFM8ICG7xMRIaYJfiCf4eGRSUYmVUkjIq0rlgl+MLzRqjFaRaSVxTLBl0sldZlGRFpY5AnezJJm9gsz+27U+yo5fV0X7akEOzW6k4i0sFq04N8B3F2D/ZQly5U0asGLSOuKNMGb2enAS4CrotzPQgq5jK7Bi0hLi7oF/zHgXcBcxPt5jEI+y/7DRxmbmq31rkVEGkJkCd7MXgoMufu2Jba7zMy2mtnW4eHhVdv/QDj4h1rxItKqomzBPxN4uZntBb4EPNvM/nX+Ru5+pbtvcfct/f39q7bzwXxQKrlLN1pFpEVFluDd/T3ufrq7bwZeA1zv7v8zqv3Nt3FdJ22phG60ikjLimUdPEAqmeCsvm614EWkZdUkwbv7je7+0lrsq9JgPqsWvIi0rNi24CEolXzw0FEmplVJIyKtJ94JPq9KGhFpXamlNjCzHEFFzGnAUeBOYKu717y2fbkK5UqaMZ5wek+doxERqa0TJngzuxh4N9AL/AIYAjqAVwBnm9nXgA+7+0gtAl2JM3q7SCdN1+FFpCUt1oJ/MfBmd983f4WZpYCXAs8Dvh5RbCctqKTJqJJGRFrSCRO8u/+fRdbNAt+KJKJVNpDPcMeDR+odhohIzS15k9XM3mFmayxwtZltN7Pn1yK41TCYy/LAoQmOThfrHYqISE1VU0XzxvA6+/OBdcBrgQ9EGtUqKuQzuMO9w7oOLyKtpZoEb+HfFwOfd/e7KpY1vELY6diuIV2HF5HWUk2C32ZmPyRI8NeaWZY6dP+7Upv7ukkljF0avk9EWsySdfDApcATgT3uPmFm64E3RBvW6kknE5zZ161SSRFpOdW04K9z9+3ufhjA3Q8CH402rNVVyKtUUkRazwkTvJl1mFkv0Gdm68ysN3xtBjbUKsDVMJDLsu/RCSZnVEkjIq1jsUs0bwEuJ+iiYHvF8hHgH6MMarUN5jPMOewZHue809bUOxwRkZpY7EGnK4ArzOzt7v4PNYxp1RVyYZ80Q6NK8CLSMhbri+bZ7n49sN/MXjl/vbt/I9LIVtHmvi6SqqQRkRaz2CWaZwHXAy9bYJ0DTZPg21NJNq/vUi28iLSUxS7RvC/82zQlkYsp5LLsVIIXkRZSTV80683s78M+aLaZ2RVhLXxTKeQz3H9wgqlZVdKISGuopg7+S8Aw8LvAq8LpL0cZVBQK+SzFOee+A+P1DkVEpCaqSfCnuvtfu/t94etvgHzUga22cp80utEqIi2imgT/QzN7jZklwtd/A66NOrDVdmZfNwlDT7SKSMtYrExylKBaxggeePp8uCoJjAHvjDy6VdSRTnLGevVJIyKtY7EqmmwtA6mFQi6jBC8iLaOaSzSxUchn2HtgnOnZpuntWERkxVorweeyzM45ew+qkkZE4q+1EnxelTQi0jqqGfADM0sSlEaWt3f3fVEFFZWz+zOYlYbvO7Xe4YiIRGrJBG9mbwfeBzzCsaH6HHhChHFFoiOdZFNvl1rwItISqmnBvwM4JxzJqekVcll1OiYiLaGaa/APAEeiDqRWCvkM9x0YZ6aoShoRibdqWvB7gBvN7N+BqdJCd/9IZFFFqJDLMFN07j84zkAudqX+IiJl1bTg9wHXAW1AtuK1qHBM11vM7HYzu8vM3n9yoa6O8uhOug4vIjG3ZAve3VeamKeAZ7v7mJmlgZ+a2ffd/T9X+HmrYiBXqqQZ40X1DEREJGKL9UXzMXe/3My+Q1A1cxx3f/liH+zuTtBnDUA6fD3mc2qtsy3J6es61WWBiMTeYi34UudiH1rph4f189uAAeDj7n7zSj9rNRVyWfUqKSKxt1hnY9vCvz9e6Ye7exF4opn1AN80s/Pd/c7KbczsMuAygE2bNq10V8tSyGf46a4DzBbnSCVb6mFeEWkhJ8xuZvYdM3tZeP18/rqzzOyvzOyN1ezE3Q8DNwAvXGDdle6+xd239Pf3Lyf2FSvkskwX59j36ERN9iciUg+LNV/fDPwmsMPMbjWz75nZ9Wa2B/gksM3dP3WiN5tZf9hyx8w6gecBO1Yx9hUrje60U5U0IhJji12ieRh4F/AuM9tM0HnLUWCnu1fT9D0V+Gx4HT4BfMXdv3vSEa+CgTDB7x4aBU6pbzAiIhGpqrMxd98L7F3OB7v7L4EnLT+k6HW3p9jQo0oaEYm3lr3DWMhndIlGRGKtdRN8LsO9w2MU5+pemi8iEomqEryZdZrZOVEHU0uFfJbp2TkeUCWNiMTUkgnezF4G3Ab8IJx/opldE3VgUStV0ug6vIjEVTUt+L8EngocBnD324AzI4ypJgbKpZJ6olVE4qmaBD/j7vP7g2/6C9fZjjSnre1gt1rwIhJT1ZRJ3mVm/wNImlkB+EPgP6INqzYG8hrdSUTiq5oW/NuBxxF0//tFYAS4PMqgaqWQy7B7aIw5VdKISAxV0x/8BPBn4StWCrkMkzNzPHjoKJvWd9U7HBGRVbVkgj9Bf/BHgK3AJ919MorAaqGQD0d3GhpVgheR2KnmEs0egoE7/iV8jQCjwGA437QGVCopIjFWzU3WZ7j7Uyrmv2Nmt7r7U8zsrqgCq4W1nWnya9pVKikisVRNCz5jZuWROMLpTDg7HUlUNTSYz6pUUkRiqZoW/J8QDJh9L2AEDzn9vpl1A5+NMrhaGMhl+PKtDzA35yQSVu9wRERWTTVVNN8L69/PDRfdU3Fj9WORRVYjhVyWieki+w8fZWOvbrSKSHxU1R88UADOATqAC8wMd/9cdGHVzmC+NPjHmBK8iMRKNZ2NvQ/4h/B1MfB3wMsjjqtmjlXS6EariMRLNTdZXwU8B3jY3d8AXACsjTSqGurpaqM/267BP0QkdqpJ8EfdfQ6YNbM1wBCwMdqwaquQy6gWXkRip5oEv9XMeggeatoGbAd+HmlUNTaYz7L7kVHc1SeNiMRHNTdZ3+ZB5vuEmf0AWAPcE21YtTWQyzA+XeShI5Oc1tNZ73BERFZFNS34q0sT7r6XoOuC70UVUD0UNPiHiMRQNQl+v5n9E4CZrQN+CPxrpFHV2GDY6ZieaBWROFkywbv7e4ExM/sEQXL/sLt/OvLIamhddxt9mTZ2qZJGRGLkhNfgzeyVFbM3A+8FbgHczF7p7t+IOrhaGshl2KlaeBGJkcVusr5s3vwvgHS43IFYJfhCLsu3frEfd8dMfdKISPM7YYIPH2pqGYP5DKNTszwyMsUpazvqHY6IyEmrZkSnDuBSgnFZy5nP3d8YYVw1N5ALbrTufGRUCV5EYqGaKprPA6cALwB+DJxOMKJTrBTyGt1JROKlmgQ/EFbSjLv7Z4GXAE+LNqza68u009vdxm7daBWRmKgmwc+Efw+b2fkEHY3logupfgZyGZVKikhsVJPgrwwfcHovcA3wK4Iug2OnkMuwU33SiEhMVDOi01Xh5I+Bs6r9YDPbCHwOyBOUVV7p7lesJMhaGcxnGZmcZXh0itwa3WgVkeZWzYAfeTO72sy+H86fZ2aXVvHZs8CfuPt5wIXA28zsvJMLN1qFnG60ikh8VHOJ5jPAtcBp4fxO4PKl3uTuD7n79nB6FLgb2LCyMGtjIK9Ox0QkPqpJ8H3u/hVgDsDdZ4HicnZiZpuBJxF0edCw+jPtrO1MqwUvIrFQTYIfN7P1BNfRMbMLgSPV7sDMMsDXgcvdfWSB9ZeZ2VYz2zo8PFztx0bCzBjMZ9itShoRiYFqEvwfE1TPnG1mPyO4cfr2aj7czNIEyf0LJ+qczN2vdPct7r6lv7+/yrCjM5DLsnNIlTQi0vyqqaLZbmbPAs4BDLjH3WeWeBsW9Nh1NXC3u3/kpCOtkUIuwxcnZjgwNk1/tr3e4YiIrFi13QVXGjQzqugu+JnAa4E7zOy2cNn/dfeGHg2qNPjHrqFRJXgRaWrL6S640pLdBbv7Twla/E2l1CfN7qExnnF2X52jERFZOXUXPE8u2062I6VSSRFpetXcZG0pQSVNVn3SiEjTU4JfQCGX0QDcItL0lOAXMJDLcHB8moNjU/UORURkxarpi2abmb0t7FGyJRTKlTRqxYtI86qmBf/fCfqhudXMvmRmL7CYj0o9qNGdRCQGlkzw7r7b3f8MGAT+DfgUcL+Zvd/MeqMOsB5OWdNBpj3FblXSiEgTq+oavJk9Afgw8EGCrgdeDYwA10cXWv2YGQO5DDtVSSMiTWzJrgrMbBtwmKDbgXe7e+nO481m9swog6unwXyG63fUt/MzEZGTUU0L/tXu/hx3/7eK5A6Au5+oO4OmV8hlOTA2xaHx6XqHIiKyItUk+DeZWU9pxszWmdnfRBhTQxjQjVYRaXLVJPgXufvh0oy7HwJeHF1IjeHY8H260SoizamaBJ80s3K3imbWCcS+m8UNPZ10tyXVZYGINK0lb7ICXwB+ZGafDuffAHw2upAaQ6mSRl0WiEizqmbAj781s18CzwkX/bW7XxttWI1hIJflJ7tUSSMizamaFjzu/n3g+xHH0nAG8xm+vv1BjkzMsLYrXe9wRESWpZq+aC40s1vNbMzMps2saGaPGTw7jsqDfwzrRquINJ9qbrL+I3AJsAvoBN4EfDzKoBpFIRd0OqYnWkWkGVXVVYG77waS7l50908DL4w2rMawoaeTzrQqaUSkOVVzDX7CzNqA28zs74CHaJF+5BOJoJJGtfAi0oyqSdSvDbf7A2Ac2Aj8bpRBNZJCLqMWvIg0pWoS/K8D7u4j7v5+d//j8JJNSxjIZ3h4ZJKRyZl6hyIisizVJPiXATvN7PNm9lIzq6q0Mi4GwxuteuBJRJpNNQN+vAEYAL5KUE1zr5ldFXVgjaJcKqnLNCLSZKp90GnGzL4POEGp5CsIyiVj7/R1XbSnEuzU6E4i0mSqedDpRWb2GYI6+N8FrgJOiTiuhpEsV9KoBS8izaWaFvzrgC8Db5k/4EerKOQy3Lr3UL3DEBFZlmquwV/i7t9q1eQOUMhn2X/4KGNTs/UORUSkaidM8Gb20/DvqJmNVLxGW6UvmpLS4B+qpBGRZnLCBO/uvxH+zbr7mopX1t3X1C7E+ivkg1LJXbrRKiJNZNFLNGaWNLMdtQqmUW1c10lbKqEbrSLSVBZN8O5eBO4xs001iqchpZIJzurrVgteRJpKNVU064C7zOwWgr5oAHD3ly/2JjP7FPBSYMjdzz+pKBvAYD7L9n2qpBGR5lFNgn/vCj/7MwR9yX9uhe9vKIVchmtu/y8mpmfpamup3hpEpEmdMFOZWQfwVoJuCu4Arnb3qusE3f0mM9t8sgE2inKXBUNjPOH0njpHIyKytMWuwX8W2EKQ3F8EfLgmETWoY5U0utEqIs1hsWsN57n74wHM7GrgligCMLPLgMsANm1q3Hu5Z/R2kU6aKmlEpGks1oIvd4C+nEszy+XuV7r7Fnff0t/fH9VuTlpQSZNRJY2INI3FWvAXVDyxakBnOG8EA4C01MNOEAz+cceDR+odhohIVRZ7kjU57+nV1HKeZDWzLwI/B84xswfN7NLVDLweBnNZHjg0wdHpYr1DERFZUmT1fu5+SVSfXS+FfAZ3uHd4jPM3rK13OCIii6pmyD4JlTod2zWk6/Ai0viU4Jdhc183qYSpVFJEmoIS/DKkkwnO7OtWqaSINAUl+GUq5FUqKSLNQQl+mQq5LPsenWByRpU0ItLYlOCXqZDPMOewZ3h86Y1FROpICX6ZCrmwTxpV0ohIg1OCX6bNfV0kVUkjIk1ACX6Z2lNJNq/vUgteRBqeEvwKFHJZlUqKSMNTgl+BQj7D/QcnmJpVJY2INC4l+BUo5LMU55z7DqiSRkQalxL8CpT7pNGNVhFpYErwK3BmXzcJQ0+0ikhDU4JfgY50ks3r1SeNiDQ2JfgVGshllOBFpKEpwa9QIZ9h74Fxpmfn6h2KiMiClOBXqJDLMjvn7D2oShoRaUxK8CtUyKuSRkQamxL8Cp3dn8FMnY6JSONSgl+hjnSSTb1dasGLSMNSgj8JhVyWux8eYaaoG60i0niU4E/C405bw57hcZ7819fxti9s56tbH2BodLLeYYmIAJCqdwDN7G0XD/Brp2a58Z5hbrhniH+/4yEAHr9hLRef089F5+a44PQekgmrc6Qi0orM3esdQ9mWLVt869at9Q5jRdydXz00EiT7HUNs33eIOYd1XWmeNdjPxefm+K1CP+u62+odqojEiJltc/ctC65Tgo/G4Ylpbtp1gBt3DHHjzmEeHZ8mYfDEjT08+9wcF52T43GnrcFMrXsRWTkl+Dorzjl37D/CDTuGuPGeIW5/8AgAuWw7F53Tz8Xn5HhmoY81Hek6RyoizUYJvsEMj05x087guv1NO4cZmZwllTC2bF7HxefkuPjcHIVcRq17EVmSEnwDmy3OsX3fYW64Z4gbdgyx4+HgwakNPZ3l1v0zBtbT1ab74SLyWErwTeShI0fLN2p/uvsAE9NF2lIJLjxrPReHCX9zX3e9wxSRBqEE36SmZots3XuIG3YMcf09Q+wZDjo2O7Ovm2cN9nPeaWvY1NvFGeu7yGc7SKgcU6TlKMHHxP0Hx8s19z+/9yBTFV0Vt6USbFzXyRnru9nU21VO/Jt6u9jY20VHOlnHyEUkKnVL8Gb2QuAKIAlc5e4fWGx7JfjqzRTn+K/DR7n/4AT7Hg1e9x8cZ9+jR9l3cJzx6eJx25+ypiNI/OuPT/6bervo7W5rmBu6s8U5jhyd4fDRGQ5PzHDk6HT4N3gV55w5d4pzwbMHpem5cDp4wdzcwtNFd9w9/ByOmy6//7jPC6qgutqS9GXa6c+205dppy/TVp7OZdvp7W4jldSD4VJ7iyX4yO7cmVkS+DjwPOBB4FYzu8bdfxXVPltJOpngjPXdnLH+sdfj3Z2D49NB4j84UfElMM5Pdg3zyMjUcdtn2lNs7O3ijDDxbwz/ntHbzak9HaRXkLgmZ4pBop6Y4fDEdDlpH5mY4XCYtOfPH5mYYXRqdtHPTRgkzIJXomLaIJGw4+aT4bzNny6/3459XsV0MtwunUyU3zMxXeT2Bw8zPDrFxLwvTwAzWNfVRn+mnb5sW/BlkGmnL3vsb+lLobertl8GxTlnbHKWkckZRidnGS39nSrNB+vGwunS+lTSWNuZZk1HmjWd6XA6xdqu+cvSrOlM0ZlONkxDIUrzGwWlRkepYVB0DxsVVEw/tiFRnDv2nmTCOH/D2lWPNcrSjKcCu919D4CZfQn4bUAJPmJmFrYy23nypnWPWX90usgDh8Lk/+gE+w6Os+/RCXYOjXL9jiGmKzpPSyaMDT2d5db/Gb1dZDpSQYt6IkzgFa3s0vzkzIk7YEsmjJ7ONGu70vR0psllOxjMZcP5Nnq60vR0BQmkpzNNT1cbPZ3BfCN0+zAxPcuB0WmGxyYZHp1meGyKA6NTHBibYjj8+4t9wZfB0ZmFvwx6u9oW/DVwbFkwvbYzzcT0sSQ8Oi8Jl/6OLLCsND3/19xC2pIJsh2p8JUm056iOOfsPTDByGRwbhf6YquUThprOoKkny19GXQ+9otgoS+NNZ3pEzYk3J2ZojM5W2RypsjUzByTM0UmZ+bKyybLy4pMzc4d9/e49RXLphZ4f3GBZFycczxM1sW5aK549GXa2frnz131z40ywW8AHqiYfxB4WoT7kyp1tiUZzGcZzGcfs6445zwyMsn9Byd44NEJ7n90vDz9vTse4vDETHnbtlSCdWFSXtuZZmNvF4/fkA4TdLCspyJpl+Yz7ammbul1taXYtD7FpvVdS247PjVbTvqlL4Dhsenjlu29f5wDY1OLfikupi2VYE2YmEtJOpftKCfr0rI15en0cck825Gq6h7NTHGO0clZjhydYSS8ZDYyOcPI0dnydGndSLjd/kNHy8tniosnx662JGs60nS2JZmal4xXmlfNoCOVpCOdoCOdpCOdpD1Vmk7Q291WXt+eSpJKWvnXXsKMZOL4X3jJ8nSwXekXYTBtJCt+SSYrfyGW5is+x8JlyQS0p6K5R1b34mozuwy4DGDTpk11jkaSCeO0nk5O6+nk6Wevf8z6oCU3y7quNt24rUJ3e4ru9tSSpa3uztjULAfGpo/7JXBkYobu9mPJuDKRZ8IkHVVymC+dDBJi7wr6U3J3JmfmKr4Ujn1BHJkIvhBKy47OFMsJOEi+xxJ0ezpJRypR/ltK2uUEXkrW4bK2ZKKpGxMnK8oEvx/YWDF/erjsOO5+JXAlBDdZI4xHVsHa8Ge1rC4zCxN3mjNj+JyDmdHZlqSzLckpazvqHU7LiPJOz61AwczONLM24DXANRHuT0REKkTWgnf3WTP7A+BagjLJT7n7XVHtT0REjhfpNXh3/x7wvSj3ISIiC9OTGSIiMaUELyISU0rwIiIxpQQvIhJTSvAiIjHVUN0Fm9kwcP8K394HHFjFcBqRjjEedIzNr5GO7wx3719oRUMl+JNhZltP1GVmXOgY40HH2Pya5fh0iUZEJKaU4EVEYipOCf7KegdQAzrGeNAxNr+mOL7YXIMXEZHjxakFLyIiFZo+wZvZC83sHjPbbWbvrnc8K2VmG83sBjP7lZndZWbvCJf3mtl1ZrYr/LsuXG5m9vfhcf/SzJ5c3yOonpklzewXZvbdcP5MM7s5PJYvh91LY2bt4fzucP3mesZdLTPrMbOvmdkOM7vbzJ4et/NoZn8U/ju908y+aGYdzX4ezexTZjZkZndWLFv2eTOz14fb7zKz19fjWEqaOsFXDOz9IuA84BIzO6++Ua3YLPAn7n4ecCHwtvBY3g38yN0LwI/CeQiOuRC+LgP+ufYhr9g7gLsr5v8W+Ki7DwCHgEvD5ZcCh8LlHw23awZXAD9w93OBCwiONTbn0cw2AH8IbHH38wm6A38NzX8ePwO8cN6yZZ03M+sF3kcwPOlTgfeVvhTqwt2b9gU8Hbi2Yv49wHvqHdcqHdu3gecB9wCnhstOBe4Jpz8JXFKxfXm7Rn4RjOz1I+DZwHcBI3hgJDX/nBKMJfD0cDoVbmf1PoYljm8tcN/8OON0Hjk23nJveF6+C7wgDucR2AzcudLzBlwCfLJi+XHb1frV1C14Fh7Ye0OdYlk14U/YJwE3A3l3fyhc9TCQD6eb9dg/BrwLKI0wvR447O6z4XzlcZSPMVx/JNy+kZ0JDAOfDi9DXWVm3cToPLr7fuBDwD7gIYLzso14nceS5Z63hjqfzZ7gY8fMMsDXgcvdfaRynQdNgqYtezKzlwJD7r6t3rFEKAU8Gfhnd38SMM6xn/VALM7jOuC3Cb7MTgO6eeyljdhpxvPW7Am+qoG9m4WZpQmS+xfc/Rvh4kfM7NRw/anAULi8GY/9mcDLzWwv8CWCyzRXAD1mVhpdrPI4yscYrl8LHKxlwCvwIPCgu98czn+NIOHH6Tw+F7jP3YfdfQb4BsG5jdN5LFnueWuo89nsCT42A3ubmQFXA3e7+0cqVl0DlO7Ev57g2nxp+evCu/kXAkcqfko2JHd/j7uf7u6bCc7V9e7+e8ANwKvCzeYfY+nYXxVu39AtKHd/GHjAzM4JFz0H+BUxOo8El2YuNLOu8N9t6Rhjcx4rLPe8XQs838zWhb90nh8uq49639RYhZsiLwZ2AvcCf1bveE7iOH6D4OffL4HbwteLCa5V/gjYBfw/oDfc3ggqiO4F7iCoaKj7cSzjeC8CvhtOnwXcAuwGvgq0h8s7wvnd4fqz6h13lcf2RGBreC6/BayL23kE3g/sAO4EPg+0N/t5BL5IcE9hhuCX2KUrOW/AG8Nj3Q28oZ7HpCdZRURiqtkv0YiIyAkowYuIxJQSvIhITCnBi4jElBK8iEhMKcFL3ZiZm9mHK+bfaWZ/uUqf/Rkze9XSW570fl4d9hh5w3LjCrsxWFHneGZ2kZk9YyXvldahBC/1NAW80sz66h1IpYqnMatxKfBmd794uftx9ze5+6+W+77QRYASvAjRhwoAAAM0SURBVCxKCV7qaZZg6LM/mr9ifgvczMbCvxeZ2Y/N7NtmtsfMPmBmv2dmt5jZHWZ2dsXHPNfMtprZzrAfnFJf9B80s1vDfrzfUvG5PzGzawieypwfzyXh599pZn8bLvsLggfUrjazDy7wnj8N33O7mX1ggfU3mtmWcPr5ZvZzM9tuZl8N+yTCzPaa2fvD5XeY2blhZ3RvBf7IzG4zs98Mf0ncGe7rpur+80vcLaelIhKFjwO/NLO/W8Z7LgB+DXgU2ANc5e5PtWCQlLcDl4fbbSbok/ts4AYzGwBeR/BY+VPMrB34mZn9MNz+ycD57n5f5c7M7DSCPsx/naCf8x+a2Svc/a/M7NnAO91967z3vIigQ66nuftE2E/4gsJfMH8OPNfdx83sT4E/Bv4q3OSAuz/ZzH4/3NebzOwTwJi7fyj8jDuAF7j7fjPrWcZ/S4kxteClrjzoMfNzBANIVOtWd3/I3acIHhUvJeg7CJJ6yVfcfc7ddxF8EZxL0DfI68zsNoLumNcTDNoAcMv85B56CnCjB51rzQJfAH5riRifC3za3SfC43x0kW0vJBiw5mdhXK8HzqhYX+p4btu846v0M+AzZvZmggE4RNSCl4bwMWA78OmKZbOEDRAzSwBtFeumKqbnKubnOP7f9Px+OJygD5G3u/txHUCZ2UUEXfvWgwHXufslJ1hfOr4iJ/h/1t3famZPA14CbDOzX3f3ZumxUSKiFrzUXdi6/QrHhngD2EtwSQTg5UB6BR/9ajNLhNflzyIYdeda4H9b0DUzZjZowYAci7kFeJaZ9VkwTOQlwI+XeM91wBvMrCvczwkv0QD/CTwzvISEmXWb2eASnz8KZEszZna2u9/s7n9BMODIxhO+U1qGErw0ig8DldU0/0KQVG8nGP5tJa3rfQTJ+fvAW919EriK4CbqdgsGV/4kS/yS9aAb2HcTdId7O7DN3b+9xHt+QNCl7Nbwsss7F9l2GPhfwBfN7JfAzwkuJy3mO8DvlG6yAh8s3QQG/iOMU1qcepMUEYkpteBFRGJKCV5EJKaU4EVEYkoJXkQkppTgRURiSgleRCSmlOBFRGJKCV5EJKb+P9NzPXLQKa35AAAAAElFTkSuQmCC\n" 471 | }, 472 | "metadata": { 473 | "needs_background": "light" 474 | } 475 | } 476 | ] 477 | } 478 | ] 479 | } -------------------------------------------------------------------------------- /diagrams/FbFTL_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wfwf10/Feature-based-Federated-Transfer-Learning/6e332076b157d3d7dfcfc46b3170c966747e5510/diagrams/FbFTL_diagram.png -------------------------------------------------------------------------------- /history.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /main_GLOBECOM_CIFAR10_VGG16.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import time 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | from torchvision import datasets 8 | from torchvision import transforms 9 | from torch.utils.data import DataLoader 10 | import datetime 11 | 12 | ####################################### 13 | ### PRE-TRAINED MODELS AVAILABLE HERE 14 | ## https://pytorch.org/docs/stable/torchvision/models.html 15 | from torchvision import models 16 | ####################################### 17 | 18 | now = datetime.datetime.now() 19 | 20 | # Device 21 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 22 | print('Device:', DEVICE) 23 | 24 | ########################## 25 | ### SETTINGS 26 | ########################## 27 | 28 | FL_type = 'FbFTL' 29 | train_set_denominator = 'full' # 'full', int <= 50000 # pick a subset with 50000/int training samples 30 | 31 | # Hyperparameters 32 | NUM_CLASSES = 10 33 | random_seed = 1 34 | learning_rate = 1e-2 35 | num_epochs = 200 36 | batch_size = 64 37 | momentum = 0.9 38 | lr_decay = 5e-4 39 | 40 | write_hist = True 41 | 42 | def adjust_learning_rate(optimizer, epoch): 43 | """Sets the learning rate to the initial LR decayed by 2 every 30 epochs""" 44 | lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs)) 45 | for param_group in optimizer.param_groups: 46 | param_group['lr'] = lr 47 | 48 | if torch.cuda.is_available(): 49 | torch.backends.cudnn.deterministic = True 50 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 51 | torch.manual_seed(random_seed) # sets the seed for generating random numbers. 52 | 53 | if write_hist: 54 | file1 = open('history.txt', 'a') 55 | file1.write('\n \n \n Time:') 56 | file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' ' 57 | + str(now.hour) + ' ' + str(now.minute) + ' ' + str(now.second) 58 | + ' ' + FL_type + ', train_deno:' + str(train_set_denominator)) 59 | file1.close() 60 | 61 | ########################## 62 | ### CIFAR10 DATASET 63 | ########################## 64 | 65 | custom_transform = transforms.Compose([ 66 | transforms.Resize((224, 224)), 67 | transforms.ToTensor(), 68 | transforms.Normalize(mean=[0.485, 0.456, 0.406], 69 | std=[0.229, 0.224, 0.225]) 70 | ]) 71 | 72 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True) 73 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform) 74 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False) 75 | 76 | if train_set_denominator == 'full': 77 | train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True) 78 | else: 79 | selected_list = list(range(0, len(train_dataset), train_set_denominator)) 80 | trainset_1 = torch.utils.data.Subset(train_dataset, selected_list) 81 | train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True) 82 | 83 | ########################## 84 | ### LOAD MODEL 85 | ########################## 86 | 87 | model = models.vgg16(pretrained=True) 88 | for param in model.parameters(): 89 | param.requires_grad = False 90 | 91 | model.classifier[3].requires_grad = True 92 | model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES)) 93 | 94 | ########################## 95 | ### TRAIN MODEL 96 | ########################## 97 | 98 | model = model.to(DEVICE) 99 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum) 100 | 101 | def compute_accuracy(model, data_loader): 102 | model.eval() 103 | correct_pred, num_examples = 0, 0 104 | with torch.no_grad(): 105 | for i, (features, targets) in enumerate(data_loader): 106 | features = features.to(DEVICE) 107 | targets = targets.to(DEVICE) 108 | 109 | logits = model(features) 110 | _, predicted_labels = torch.max(logits, 1) 111 | num_examples += targets.size(0) 112 | correct_pred += (predicted_labels == targets).sum() 113 | return correct_pred.float()/num_examples * 100 114 | 115 | 116 | def compute_epoch_loss(model, data_loader): 117 | model.eval() 118 | curr_loss, num_examples = 0., 0 119 | with torch.no_grad(): 120 | for batch_idx, (features, targets) in enumerate(data_loader): 121 | features = features.to(DEVICE) 122 | targets = targets.to(DEVICE) 123 | logits = model(features) 124 | loss = F.cross_entropy(logits, targets, reduction='sum') 125 | num_examples += targets.size(0) 126 | curr_loss += loss 127 | 128 | curr_loss = curr_loss / num_examples 129 | return curr_loss 130 | 131 | 132 | 133 | start_time = time.time() 134 | for epoch in range(num_epochs): 135 | model.train() 136 | for batch_idx, (features, targets) in enumerate(train_loader): 137 | 138 | features = features.to(DEVICE) 139 | targets = targets.to(DEVICE) 140 | 141 | ### FORWARD AND BACK PROP 142 | logits = model(features) 143 | cost = F.cross_entropy(logits, targets) 144 | optimizer.zero_grad() 145 | 146 | cost.backward() 147 | 148 | ### UPDATE MODEL PARAMETERS 149 | optimizer.step() 150 | 151 | ### LOGGING 152 | if not batch_idx % 50: 153 | print ('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f' 154 | %(epoch+1, num_epochs, batch_idx, 155 | len(train_loader), cost)) 156 | 157 | model.eval() 158 | accuracy = compute_accuracy(model, test_loader) 159 | loss = compute_epoch_loss(model, test_loader) 160 | with torch.set_grad_enabled(False): # save memory during inference 161 | print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss)) 162 | if write_hist: 163 | file1 = open('history.txt', 'a') 164 | file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss)) 165 | file1.close() 166 | 167 | 168 | print('Time elapsed: %.2f min' % ((time.time() - start_time)/60)) 169 | 170 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60)) 171 | 172 | with torch.set_grad_enabled(False): # save memory during inference 173 | accuracy = compute_accuracy(model, test_loader) 174 | print('Test accuracy: %.2f%%' % (accuracy)) 175 | if write_hist: 176 | file1 = open('history.txt', 'a') 177 | file1.write('\n Test accuracy: %.2f%%' % (accuracy)) 178 | file1.close() 179 | 180 | 181 | -------------------------------------------------------------------------------- /main_TMLCN_CIFAR10_VGG16.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import time 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | from torchvision import datasets 8 | from torchvision import transforms 9 | from torch.utils.data import DataLoader 10 | import matplotlib.pyplot as plt 11 | import datetime 12 | import math 13 | from collections import deque 14 | import copy 15 | 16 | ####################################### 17 | ### PRE-TRAINED MODELS AVAILABLE HERE 18 | ## https://pytorch.org/docs/stable/torchvision/models.html 19 | from torchvision import models 20 | ####################################### 21 | 22 | now = datetime.datetime.now() 23 | 24 | # Device 25 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 26 | print('Device:', DEVICE) 27 | 28 | ########################## 29 | ### SETTINGS 30 | ########################## 31 | 32 | FL_type = 'FbFTL' # 'FL', 'FTLf', 'FTLc', 'FbFTL' 33 | train_set_denominator = 'full' # 'full', int <= 50000 # pick a subset with 50000/int training samples 34 | 35 | # Hyperparameters 36 | NUM_CLASSES = 10 37 | U_clients = 6250 # number of clients, 50000/8 38 | random_seed = 1 # transfer = True / False 39 | learning_rate = 1e-2 # 1e-3, 0.05, 1e-2 40 | num_epochs = 200 # 10, 300, 200 41 | batch_size = 64 # 128, 128, (out of memory:64) 42 | momentum = 0.9 # None, 0.9 43 | lr_decay = 5e-4 # 1e-6, 5e-4 44 | if FL_type == 'FL': 45 | transfer, full = False, True # transfer or train model from scratch # train whole model or last few layers 46 | sigma = 0. # 0.5 relative std for addtive gaussian noise on gradients 47 | elif FL_type == 'FTLf': 48 | transfer, full = True, True 49 | sigma = 0. # 0.3305 relative std for addtive gaussian noise on gradients 50 | elif FL_type == 'FTLc': 51 | transfer, full = True, False 52 | sigma = 0. # 0.285 relative std for addtive gaussian noise on gradients 53 | elif FL_type == 'FbFTL': 54 | transfer, full = True, False 55 | sigma = 0 # 0.8? relative std for addtive gaussian noise on features 56 | saved_noise = True # save noise at beginning 57 | else: 58 | raise ValueError('Unknown FL_type: ' + FL_type) 59 | relative_noise_type = 'all_std' # 'individual', 'all_std' 60 | packet_loss_rate = 0. # 0, 0.05, 0.1, 0.15 61 | quan_digit = 32 # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int 62 | sparse_rate = 0.9 # ratio of uplink elements kept after sparsification: None or (0,1] 63 | class ErrorFeedback(object): 64 | queue = deque(maxlen=U_clients) 65 | temp = deque() 66 | if (quan_digit or sparse_rate) and FL_type != 'FbFTL': 67 | errfdbk = ErrorFeedback() 68 | # print(errfdbk.queue, errfdbk.temp) 69 | 70 | write_hist = True 71 | 72 | def adjust_learning_rate(optimizer, epoch): 73 | """Sets the learning rate to the initial LR decayed by 2 every 30 epochs""" 74 | lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs)) 75 | for param_group in optimizer.param_groups: 76 | param_group['lr'] = lr 77 | 78 | if torch.cuda.is_available(): 79 | torch.backends.cudnn.deterministic = True 80 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 81 | torch.manual_seed(random_seed) # sets the seed for generating random numbers. 82 | 83 | if write_hist: 84 | file1 = open('/data1/feng/LaTFL/history.txt', 'a') 85 | file1.write('\n \n \n Time:') 86 | file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' ' + str(now.hour) + ' ' 87 | + str(now.minute) + ' ' + str(now.second) + ' ' + FL_type 88 | # + ', train_deno:' + str(train_set_denominator) 89 | # + ', sigma:' + str(sigma) 90 | # + ', packet_loss_rate:' + str(packet_loss_rate) 91 | + ', quantization digits:' + str(quan_digit) 92 | + ', sparsification rate:' + str(sparse_rate) 93 | ) 94 | file1.close() 95 | 96 | ########################## 97 | ### CIFAR10 DATASET 98 | ########################## 99 | 100 | custom_transform = transforms.Compose([ 101 | transforms.Resize((224, 224)), 102 | transforms.ToTensor(), 103 | transforms.Normalize(mean=[0.485, 0.456, 0.406], 104 | std=[0.229, 0.224, 0.225]) 105 | ]) 106 | 107 | ## Note that this particular normalization scheme is necessary since it was used for pre-training the network on ImageNet. 108 | ## These are the channel-means and standard deviations for z-score normalization. 109 | 110 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True) 111 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform) 112 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False) 113 | 114 | if train_set_denominator == 'full': 115 | train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True) 116 | train_set_len = len(train_dataset) 117 | else: 118 | # print('len(train_dataset)', len(train_dataset)) # 50000 119 | # selected_list = list(range(0, len(train_dataset), 2)) 120 | selected_list = list(range(0, len(train_dataset), train_set_denominator)) 121 | trainset_1 = torch.utils.data.Subset(train_dataset, selected_list) 122 | train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True) 123 | train_set_len = len(selected_list) 124 | 125 | # # Checking the dataset 126 | # for images, labels in train_loader: 127 | # print('Image batch dimensions:', images.shape) 128 | # print('Image label dimensions:', labels.shape) 129 | # break 130 | 131 | # labels = torch.zeros(10, dtype=torch.long) 132 | # for batch_idx, (features, targets) in enumerate(train_loader): 133 | # for t in targets: 134 | # labels[t] += 1 135 | # print('labels', labels) 136 | 137 | ########################## 138 | ### LOAD MODEL 139 | ########################## 140 | 141 | class GaussianNoise(nn.Module): 142 | """Gaussian noise regularizer. 143 | Args: 144 | sigma (float, optional): relative standard deviation used to generate the 145 | noise. Relative means that it will be multiplied by the magnitude of 146 | the value your are adding the noise to. This means that sigma can be 147 | the same regardless of the scale of the vector. 148 | is_relative_detach (bool, optional): whether to detach the variable before 149 | computing the scale of the noise. If `False` then the scale of the noise 150 | won't be seen as a constant but something to optimize: this will bias the 151 | network to generate vectors with smaller values. 152 | """ 153 | def __init__(self, sigma=0, is_relative_detach=False): 154 | super().__init__() 155 | self.sigma = sigma 156 | self.is_relative_detach = is_relative_detach 157 | if saved_noise: 158 | self.register_buffer('noise', torch.empty(train_set_len*4096).normal_(mean=0,std=1)) 159 | self.i = 0 160 | else: 161 | self.register_buffer('noise', torch.tensor(0)) 162 | 163 | def forward(self, x): 164 | if self.training and quan_digit: 165 | # print(x) 166 | # print(x.dtype, torch.max(x), torch.min(x), torch.std(x)) 167 | x = torch.round((2**quan_digit-1) / torch.max(x) * x) * torch.max(x) / (2**quan_digit-1) 168 | # print(x.dtype, torch.max(x), torch.min(x), torch.std(x)) 169 | # print(x) 170 | # quit() 171 | if self.training and self.sigma != 0: 172 | if torch.cuda.is_available(): 173 | torch.backends.cudnn.deterministic = True 174 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 175 | torch.manual_seed(random_seed) # sets the seed for generating random numbers. 176 | if relative_noise_type == 'individual': 177 | scale = self.sigma * x.detach() if self.is_relative_detach else self.sigma * x 178 | sampled_noise = self.noise.expand(*x.size()).float().normal_() * scale 179 | elif relative_noise_type == 'all_std': 180 | x_std = torch.std(x.detach()) if self.is_relative_detach else torch.std(x) 181 | # print(*x.size()) # 64 4096 182 | if saved_noise: 183 | sampled_noise = torch.reshape(self.noise[self.i*batch_size*4096 : (self.i+1)*batch_size*4096],(-1, 4096) 184 | ).detach().float() * x_std * self.sigma 185 | self.i = self.i + 1 if (self.i+1)*batch_size*4096 256: 221 | m.reset_parameters() 222 | for i in range(21, 31): 223 | model.features[i].apply(weight_reset) 224 | for i in range(7): 225 | model.classifier[i].apply(weight_reset) 226 | 227 | for param in model.parameters(): 228 | param.requires_grad = True 229 | 230 | # model.classifier[3].requires_grad = True # (4096, 4096, relu, dropout(0.5)) 231 | # model.classifier[6] = nn.Sequential(nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES)) 232 | model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), 233 | nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES)) 234 | 235 | 236 | def Gaussian_noise_to_weights(m): 237 | if sigma!=0: 238 | with torch.no_grad(): 239 | for param in m.parameters(): 240 | if param.requires_grad: 241 | if relative_noise_type == 'individual': 242 | # print(param.grad.view(-1)) 243 | # print(param) 244 | # scale = sigma * param.detach() 245 | scale = sigma * param.grad.detach() # todo: * math.sqrt(batch_size/8) 246 | noise = torch.tensor(0).to(DEVICE) 247 | sampled_noise = noise.expand(*param.size()).float().normal_() * scale 248 | elif relative_noise_type == 'all_std': 249 | param_grad_std = torch.std(param.grad.detach()) 250 | noise = torch.tensor(0).to(DEVICE) 251 | sampled_noise = noise.expand(*param.size()).float().normal_(std=param_grad_std*sigma) # todo: * math.sqrt(batch_size/8) 252 | # param = param + sampled_noise 253 | param.add_(sampled_noise) 254 | 255 | def Errfdbk_to_weights(m): 256 | print("inner model.apply") 257 | with torch.no_grad(): 258 | for param in m.parameters(): 259 | print('len(param)', len(param)) 260 | if param.requires_grad: 261 | print('len(errfdbk.queue)', len(errfdbk.queue)) 262 | # print(errfdbk.temp) 263 | print('len(errfdbk.temp)', len(errfdbk.temp)) 264 | p_grad = param.grad.detach() 265 | if err_flag: 266 | p_grad += errfdbk.temp.popleft() 267 | p_grad_qs = copy.deepcopy(p_grad) 268 | if sparse_rate: 269 | pass 270 | if quan_digit: 271 | pass 272 | err = p_grad_qs - p_grad 273 | param.add_(err) 274 | errfdbk.temp.append(-err) 275 | print('seems good') 276 | 277 | if FL_type == 'FbFTL': 278 | received_batches_FbFTL = np.ones(len(train_loader)) 279 | received_batches_FbFTL[:int(len(train_loader)*packet_loss_rate)] = 0 280 | np.random.shuffle(received_batches_FbFTL) 281 | 282 | def Packet_Received(batch_idx): 283 | if FL_type == 'FbFTL': 284 | return received_batches_FbFTL[batch_idx] 285 | else: 286 | return np.random.choice(2, p=[packet_loss_rate, 1-packet_loss_rate]) 287 | 288 | # for a in range(3): 289 | # print('round', a) 290 | # for i in range(50): 291 | # print(Packet_Received(i)) 292 | 293 | # print(model) 294 | 295 | # for name, param in model.named_parameters(): 296 | # print(name, torch.numel(param), param.requires_grad) 297 | # quit() 298 | 299 | # model(torch.randn(1, 3, 224, 224)).mean().backward() 300 | # for name, param in model.named_parameters(): 301 | # print(name, param.grad) 302 | # print('value', param.data) 303 | # wait = input('next layer') 304 | 305 | ########################## 306 | ### TRAIN MODEL 307 | ########################## 308 | 309 | model = model.to(DEVICE) 310 | # if transfer: 311 | # optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) 312 | # else: 313 | # optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum) # , nesterov=True) 314 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum) 315 | 316 | 317 | def compute_accuracy(model, data_loader): 318 | model.eval() 319 | correct_pred, num_examples = 0, 0 320 | with torch.no_grad(): 321 | for i, (features, targets) in enumerate(data_loader): 322 | features = features.to(DEVICE) 323 | targets = targets.to(DEVICE) 324 | 325 | logits = model(features) 326 | _, predicted_labels = torch.max(logits, 1) 327 | num_examples += targets.size(0) 328 | correct_pred += (predicted_labels == targets).sum() 329 | return correct_pred.float()/num_examples * 100 330 | 331 | 332 | def compute_epoch_loss(model, data_loader): 333 | model.eval() 334 | curr_loss, num_examples = 0., 0 335 | with torch.no_grad(): 336 | for batch_idx, (features, targets) in enumerate(data_loader): 337 | features = features.to(DEVICE) 338 | targets = targets.to(DEVICE) 339 | logits = model(features) 340 | loss = F.cross_entropy(logits, targets, reduction='sum') 341 | num_examples += targets.size(0) 342 | curr_loss += loss 343 | 344 | curr_loss = curr_loss / num_examples 345 | return curr_loss 346 | 347 | 348 | 349 | start_time = time.time() 350 | for epoch in range(num_epochs): 351 | # adjust_learning_rate(optimizer, epoch) 352 | 353 | model.train() 354 | for batch_idx, (features, targets) in enumerate(train_loader): 355 | 356 | if Packet_Received(batch_idx): 357 | features = features.to(DEVICE) 358 | targets = targets.to(DEVICE) 359 | 360 | ### FORWARD AND BACK PROP 361 | logits = model(features) 362 | cost = F.cross_entropy(logits, targets) 363 | optimizer.zero_grad() 364 | 365 | cost.backward() 366 | 367 | ### UPDATE MODEL PARAMETERS 368 | optimizer.step() 369 | 370 | ### PRIVACY NOISE 371 | if FL_type != 'FbFTL': 372 | # Gaussian_noise_to_weights(model, sigma * math.sqrt(batch_size/8)) 373 | model.apply(Gaussian_noise_to_weights) 374 | 375 | ### Sparsification/Quantization with Error Feedback # errfdbk.queue = deque(maxlen=U_clients) 376 | if (quan_digit or sparse_rate) and FL_type != 'FbFTL': 377 | print('main loop: len(errfdbk.queue)', len(errfdbk.queue)) 378 | if len(errfdbk.queue) < U_clients: 379 | errfdbk.temp = deque() 380 | err_flag = False 381 | else: 382 | errfdbk.temp = errfdbk.queue.popleft() 383 | err_flag = True 384 | model.apply(Errfdbk_to_weights) 385 | print('completed one cycle!') 386 | errfdbk.queue.append(errfdbk.temp) 387 | 388 | ### LOGGING 389 | if not batch_idx % 50: 390 | print('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f' % (epoch+1, num_epochs, batch_idx, len(train_loader), cost)) 391 | # if FL_type == 'FbFTL': 392 | # print(model.classifier[6][0].sigma) 393 | # model.classifier[6][0].set_sigma(sigma=0.4) 394 | 395 | model.eval() 396 | accuracy = compute_accuracy(model, test_loader) 397 | loss = compute_epoch_loss(model, test_loader) 398 | with torch.set_grad_enabled(False): # save memory during inference 399 | print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss)) 400 | if write_hist: 401 | file1 = open('/data1/feng/LaTFL/history.txt', 'a') 402 | file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss)) 403 | file1.close() 404 | 405 | 406 | print('Time elapsed: %.2f min' % ((time.time() - start_time)/60)) 407 | 408 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60)) 409 | 410 | with torch.set_grad_enabled(False): # save memory during inference 411 | accuracy = compute_accuracy(model, test_loader) 412 | print('Test accuracy: %.2f%%' % (accuracy)) 413 | if write_hist: 414 | file1 = open('/data1/feng/LaTFL/history.txt', 'a') 415 | file1.write('\n Test accuracy: %.2f%%' % (accuracy)) 416 | file1.close() 417 | 418 | # model.save_weights('/data1/feng/LaTFL/cifar10vgg.h5') 419 | 420 | -------------------------------------------------------------------------------- /main_TMLCN_FLANT5_SAMSUM.py: -------------------------------------------------------------------------------- 1 | # Source: https://www.philschmid.de/fine-tune-flan-t5 2 | # tensorboard: tensorboard --logdir log 3 | # Run the following in terminal before python3 FbFTL_LLM/test.py: huggingface-cli login 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 4 | 5 | from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq 6 | from torchinfo import summary 7 | from datasets import load_dataset 8 | from random import randrange 9 | from datasets import concatenate_datasets 10 | import evaluate 11 | import nltk 12 | import numpy as np 13 | from nltk.tokenize import sent_tokenize 14 | nltk.download("punkt") 15 | from huggingface_hub import HfFolder 16 | from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments 17 | import sys 18 | from collections import deque 19 | 20 | HfFolder.save_token('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') 21 | 22 | # FL parameters 23 | FL_type = 'FbFTL' # 'FL', 'FTLf'(same as FL), 'FTLc', 'FbFTL' 24 | model_to_train = [False, 4, 0, True] # available for 'FTLc' and 'FbFTL': [False, 4, 0, True] 25 | # shared: (boolean), encoder: (int:0~8, index after which is trained, 10 to not train layer norm), decoder: (same as encoder), lm_head: (boolean) 26 | train_set_denominator = -1 # -1 to use full training dataset, or 0 < int <= 14732 : pick a subset with [int] training samples 27 | # Hyperparameters 28 | NUM_CLASSES = 32128 29 | learning_rate = 2e-4 # TODO: source code 5e-5, full model 2e-4 30 | num_train_epochs = 20 # TODO: source code 5, full model 20 31 | batch_size = 8 # source code 8 32 | # TODO: implement following params 33 | # if FL_type == 'FL': 34 | # transfer, full = False, True # transfer or train model from scratch # train whole model or last few layers 35 | # sigma = 0. # 0.5 relative std for addtive gaussian noise on gradients 36 | # elif FL_type == 'FTLf': 37 | # transfer, full = True, True 38 | # sigma = 0. # 0.3305 relative std for addtive gaussian noise on gradients 39 | # elif FL_type == 'FTLc': 40 | # transfer, full = True, False 41 | # sigma = 0. # 0.285 relative std for addtive gaussian noise on gradients 42 | # elif FL_type == 'FbFTL': 43 | # transfer, full = True, False 44 | # sigma = 0 # 0.8? relative std for addtive gaussian noise on features 45 | # saved_noise = True # save noise at beginning 46 | # else: 47 | # raise ValueError('Unknown FL_type: ' + FL_type) 48 | # relative_noise_type = 'all_std' # 'individual', 'all_std' 49 | # packet_loss_rate = 0. # 0, 0.05, 0.1, 0.15 50 | # quan_digit = 32 # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int 51 | # sparse_rate = 0.9 # ratio of uplink elements kept after sparsification: None or (0,1] 52 | 53 | # Load dataset from the hub 54 | dataset_id = "samsum" 55 | dataset = load_dataset(dataset_id) 56 | if train_set_denominator != -1: 57 | dataset['train'] = dataset['train'].select(range(train_set_denominator)) 58 | print(f"Train dataset size: {len(dataset['train'])}") # 14732 59 | print(f"Test dataset size: {len(dataset['test'])}") # 819 60 | sample = dataset['train'][randrange(len(dataset["train"]))] 61 | # print(f"dialogue: \n{sample['dialogue']}\n---------------") 62 | # print(f"summary: \n{sample['summary']}\n---------------") 63 | # sys.exit() 64 | 65 | # Loading the Model 66 | model_id="google/flan-t5-small" 67 | tokenizer = AutoTokenizer.from_pretrained(model_id) 68 | model = AutoModelForSeq2SeqLM.from_pretrained(model_id) 69 | # for parameter in model.parameters(): # model: shared, encoder, decoder, lm_head 70 | # parameter.requires_grad = False 71 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[0]: 72 | for parameter in model.shared.parameters(): # initial embedding layer, not trained 73 | parameter.requires_grad = False 74 | 75 | if FL_type in ['FTLc', 'FbFTL']: 76 | for i, m in enumerate(model.encoder.block): # whether train encoder blocks 77 | if i < model_to_train[1]: 78 | for parameter in m.parameters(): 79 | parameter.requires_grad = False 80 | if model_to_train[1] >= 10: 81 | for parameter in model.encoder.final_layer_norm.parameters(): # whether train encoder layer norm 82 | parameter.requires_grad = False 83 | 84 | if FL_type in ['FTLc', 'FbFTL']: 85 | for i, m in enumerate(model.decoder.block): # whether train decoder blocks 86 | if i < model_to_train[2]: 87 | for parameter in m.parameters(): 88 | parameter.requires_grad = False 89 | if model_to_train[2] >= 10: 90 | for parameter in model.decoder.final_layer_norm.parameters(): # whether train decoder layer norm 91 | parameter.requires_grad = False 92 | 93 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[3]: 94 | for parameter in model.lm_head.parameters(): # final output layer, always trained 95 | parameter.requires_grad = False 96 | summary(model) 97 | # print(model) 98 | # for param in model.state_dict(): 99 | # print(param) 100 | # sys.exit() 101 | 102 | # Preprocess data 103 | tokenized_inputs = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["dialogue"], truncation=True), 104 | batched=True, remove_columns=["dialogue", "summary"]) 105 | max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]]) 106 | print(f"Max source length: {max_source_length}") 107 | tokenized_targets = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["summary"], truncation=True), 108 | batched=True, remove_columns=["dialogue", "summary"]) 109 | max_target_length = max([len(x) for x in tokenized_targets["input_ids"]]) 110 | print(f"Max target length: {max_target_length}") 111 | 112 | def preprocess_function(sample,padding="max_length"): 113 | # add prefix to the input for t5 114 | inputs = ["summarize: " + item for item in sample["dialogue"]] 115 | # tokenize inputs 116 | model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True) 117 | # Tokenize targets with the `text_target` keyword argument 118 | labels = tokenizer(text_target=sample["summary"], max_length=max_target_length, padding=padding, truncation=True) 119 | # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore 120 | # padding in the loss. 121 | if padding == "max_length": 122 | labels["input_ids"] = [ 123 | [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"] 124 | ] 125 | model_inputs["labels"] = labels["input_ids"] 126 | return model_inputs 127 | 128 | tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["dialogue", "summary", "id"]) 129 | print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}") 130 | 131 | # Metric 132 | metric = evaluate.load("rouge") 133 | # helper function to postprocess text 134 | def postprocess_text(preds, labels): 135 | preds = [pred.strip() for pred in preds] 136 | labels = [label.strip() for label in labels] 137 | # rougeLSum expects newline after each sentence 138 | preds = ["\n".join(sent_tokenize(pred)) for pred in preds] 139 | labels = ["\n".join(sent_tokenize(label)) for label in labels] 140 | return preds, labels 141 | 142 | def compute_metrics(eval_preds): 143 | preds, labels = eval_preds 144 | if isinstance(preds, tuple): 145 | preds = preds[0] 146 | decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True) 147 | # Replace -100 in the labels as we can't decode them. 148 | labels = np.where(labels != -100, labels, tokenizer.pad_token_id) 149 | decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True) 150 | # Some simple post-processing 151 | decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels) 152 | result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True) 153 | result = {k: round(v * 100, 4) for k, v in result.items()} 154 | prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds] 155 | result["gen_len"] = np.mean(prediction_lens) 156 | return result 157 | 158 | # we want to ignore tokenizer pad token in the loss 159 | label_pad_token_id = -100 160 | # Data collator 161 | data_collator = DataCollatorForSeq2Seq( 162 | tokenizer, 163 | model=model, 164 | label_pad_token_id=label_pad_token_id, 165 | pad_to_multiple_of=8 166 | ) 167 | 168 | # Hugging Face repository id 169 | repository_id = f"{model_id.split('/')[1]}-{dataset_id}" 170 | 171 | # Define training args 172 | training_args = Seq2SeqTrainingArguments( 173 | output_dir=repository_id, 174 | per_device_train_batch_size=batch_size, 175 | per_device_eval_batch_size=8, 176 | predict_with_generate=True, 177 | fp16=False, # Overflows with fp16 178 | learning_rate=learning_rate, 179 | num_train_epochs=num_train_epochs, 180 | # logging & evaluation strategies 181 | logging_dir=f"{repository_id}/logs", 182 | logging_strategy="steps", 183 | logging_steps=500, 184 | evaluation_strategy="epoch", 185 | save_strategy="epoch", 186 | save_total_limit=2, 187 | load_best_model_at_end=True, 188 | # metric_for_best_model="overall_f1", 189 | # push to hub parameters 190 | report_to="tensorboard", 191 | push_to_hub=False, 192 | hub_strategy="every_save", 193 | hub_model_id=repository_id, 194 | hub_token=HfFolder.get_token(), 195 | ) 196 | 197 | # Create Trainer instance 198 | trainer = Seq2SeqTrainer( 199 | model=model, 200 | args=training_args, 201 | data_collator=data_collator, 202 | train_dataset=tokenized_dataset["train"], 203 | eval_dataset=tokenized_dataset["test"], 204 | compute_metrics=compute_metrics, 205 | ) 206 | 207 | # Start training 208 | trainer.train() 209 | 210 | trainer.evaluate() 211 | 212 | # Save our tokenizer and create model card 213 | tokenizer.save_pretrained(repository_id) 214 | trainer.create_model_card() 215 | # Push the results to the hub 216 | # trainer.push_to_hub() 217 | 218 | 219 | # Run Inference and summarize ChatGPT dialogues 220 | # from transformers import pipeline 221 | # from random import randrange 222 | # # load model and tokenizer from huggingface hub with pipeline 223 | # summarizer = pipeline("summarization", model="philschmid/flan-t5-base-samsum", device=0) 224 | # # select a random test sample 225 | # sample = dataset['test'][randrange(len(dataset["test"]))] 226 | # print(f"dialogue: \n{sample['dialogue']}\n---------------") 227 | # # summarize dialogue 228 | # res = summarizer(sample["dialogue"]) 229 | # print(f"flan-t5-base summary:\n{res[0]['summary_text']}") --------------------------------------------------------------------------------