├── README.md
├── code_for_plots
├── Robustness_and_Privacy.ipynb
└── print_privacy_leakage.ipynb
├── diagrams
└── FbFTL_diagram.png
├── history.txt
├── main_GLOBECOM_CIFAR10_VGG16.py
├── main_TMLCN_CIFAR10_VGG16.py
└── main_TMLCN_FLANT5_SAMSUM.py
/README.md:
--------------------------------------------------------------------------------
1 | # FbFTL: Communication-Efficient Feature-based Federated Transfer Learning
2 |
3 | This is the offical implementation for Python simulation of Feature-based Federated Transfer Learning (FbFTL), from the following conference paper and journal paper:
4 |
5 | Communication-Efficient Feature-based Federated Transfer Learning.([Globecom2022](https://ieeexplore.ieee.org/abstract/document/10000612), [arXiv](https://arxiv.org/abs/2209.05395))
6 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar
7 | Department of Electrical Engineering and Computer Science, Syracuse University
8 |
9 | Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy.([TMLCN](https://ieeexplore.ieee.org/abstract/document/10542971), [arXiv](https://arxiv.org/abs/2405.09014))
10 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar
11 | Department of Electrical Engineering and Computer Science, Syracuse University
12 |
13 | ---
14 |
15 |
16 |
17 | We propose the FbFTL as an innovative federated learning approach that upload features and outputs instead of gradients to reduce the uplink payload by more than five orders of magnitude. Please refer to the journal paper for explicit explaination on learning structure, system design, robustness analysis, and privacy analysis.
18 |
19 |
20 | # Results on CIFAR-10 Dataset with VGG16 Model
21 | In the following table, we provide comparison between federated learning with [FedAvg](http://proceedings.mlr.press/v54/mcmahan17a.html) (FL), federated transfer learning with FedAvg that updating full model (FTLf), federated transfer learning with FedAvg that updating task-specific sub-model(FTLc), and FbFTL. All of them learn [VGG16](https://arxiv.org/abs/1409.1556) model on [CIFAR-10](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf) dataset. For transfer learning approaches, the source models are trained on [ImageNet](https://ieeexplore.ieee.org/abstract/document/5206848?casa_token=QncCRBM1tzAAAAAA:QuoJhjJAHRplmLJ4jcFw5JWdfASjmbIVlvpCrHgTPIFu63gpSUlBeACB78S0AH34qqQnsBOdoQ) dataset. Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude.
22 |
23 | | | FL | FTLf | FTLc | FbFTL |
24 | | ---- | ----- | ---- | ---- | ---- |
25 | | upload batches | 656250 | 193750 | 525000 | 50000 |
26 | | upload parameters per batch | 153144650 | 153144650 | 35665418 | 4096 |
27 | | uplink payload per batch | **4.9 Gb** | **4.9 Gb** | **1.1 Gb** | **131 Kb** |
28 | | total uplink payload | **3216 Tb** | **949 Tb** | **599 Tb** | **6.6 Gb** |
29 | | total downlink payload | 402 Tb | 253 Tb | 322 Tb | 3.8 Gb |
30 | | test accuracy | 89.42\% | 93.75\% | 86.51\% | 86.51\% |
31 |
32 | # Results on SAMSum summary task with FLAN-T5-small language model
33 | In the following table, we consider [FLAN-T5-small](https://www.jmlr.org/papers/volume25/23-0870/23-0870.pdf) as a pre-trained language model, and fine-tune on [SAMSum](https://www.aclweb.org/anthology/D19-5409) summary task. As a fine-tuning task, this experiment does not include an FL setting, and we provide comparison between federated transfer learning with FedAvg that updating full model (FTLf), federated transfer learning with FedAvg that updating task-specific sub-model(FTLc), and FbFTL. Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude.
34 |
35 | | | FTLf | FTLc | FbFTL | FTLc | FbFTL | FTLc | FbFTL |
36 | | ---- | ----- | ---- | ---- | ---- | ---- | ---- | ---- |
37 | | number of trained encoders | 8 | 8 | 8 | 4 | 4 | 2 | 2 |
38 | | number of upload batches | 132588 | 36830 | 7366 | 88392 | 7366 | 103124 | 7366 |
39 | | upload parameters per batch | 109860224 | 60511616 | 1024 | 51070144 | 1024 | 46349504 | 1024 |
40 | | uplink payload per batch | **3.5 Gb** | **1.9 Gb** | **32.7 Kb** | **1.6 Gb** | **32.7 Kb** | **1.5 Gb** | **32.7 Kb** |
41 | | total uplink payload | **466.1 Tb** | **71.3 Tb** | **241.4 Mb** | **144.5 Tb** | **241.4 Mb** | **152.9 Tb** | **241.4 Mb** |
42 | | total downlink payload | 116.0 Tb | 32.2 Tb | 1.58 Gb | 77.3 Tb | 1.88 Gb | 90.2 Tb | 2.03 Gb |
43 | | test ROUGE-1 | 45.9249 | 45.4680 | 45.4680 | 45.2827 | 45.2827 | 44.9862 | 44.9862 |
44 |
45 | # Required packages installation
46 | We use python==3.6.9, numpy==1.19.5, torch==1.4.0, torchvision==0.5.0, and CUDA version 11.6 for the experiments on CIFAR-10 with VGG16. The dataset and the source model will be automatically downloaded.
47 |
48 | Additionally, for the experiments on SAMSUM with FLAN-T5, we use transformers==4.30.2, torchinfo==1.8.0, datasets==2.13.2, nltk==3.8.1, evaluate==0.4.1, huggingface_hub==0.16.4
49 |
50 | # Citation
51 | If you find our work useful in your research, please consider citing:
52 | ```
53 | @inproceedings{wang2022communication,
54 | title={Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learning},
55 | author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem},
56 | booktitle={GLOBECOM 2022-2022 IEEE Global Communications Conference},
57 | pages={3875--3880},
58 | year={2022},
59 | organization={IEEE}
60 | }
61 | ```
62 |
63 | ```
64 | @article{wang2024feature,
65 | title={Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy},
66 | author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem},
67 | journal={IEEE Transactions on Machine Learning in Communications and Networking},
68 | year={2024},
69 | publisher={IEEE}
70 | }
71 | ```
72 |
--------------------------------------------------------------------------------
/code_for_plots/print_privacy_leakage.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "provenance": []
7 | },
8 | "kernelspec": {
9 | "name": "python3",
10 | "display_name": "Python 3"
11 | },
12 | "language_info": {
13 | "name": "python"
14 | }
15 | },
16 | "cells": [
17 | {
18 | "cell_type": "markdown",
19 | "source": [
20 | "# CIFAR-10"
21 | ],
22 | "metadata": {
23 | "id": "RnC64R4j-se4"
24 | }
25 | },
26 | {
27 | "cell_type": "code",
28 | "execution_count": null,
29 | "metadata": {
30 | "id": "Ogn9Z9JF0EHG"
31 | },
32 | "outputs": [],
33 | "source": [
34 | "import numpy as np\n",
35 | "import matplotlib.pyplot as plt\n",
36 | "import math\n",
37 | "from math import factorial as f\n",
38 | "from time import sleep\n",
39 | "import itertools\n",
40 | "import time"
41 | ]
42 | },
43 | {
44 | "cell_type": "code",
45 | "source": [
46 | "K = 8\n",
47 | "N = 10\n",
48 | "B = f(N+K-1) // f(K) // f(N-1)\n",
49 | "\n",
50 | "# calculate H(B|Y=y), y is uniform\n",
51 | "p_y = 0\n",
52 | "P_ys = []\n",
53 | "b_count = 0\n",
54 | "H_y = 0\n",
55 | "\n",
56 | "for c0 in range(0, N+K-1):\n",
57 | " for c1 in range(c0+1, N+K-1):\n",
58 | " for c2 in range(c1+1, N+K-1):\n",
59 | " for c3 in range(c2+1, N+K-1):\n",
60 | " for c4 in range(c3+1, N+K-1):\n",
61 | " for c5 in range(c4+1, N+K-1):\n",
62 | " for c6 in range(c5+1, N+K-1):\n",
63 | " for c7 in range(c6+1, N+K-1):\n",
64 | " b_count += 1\n",
65 | " Ns = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
66 | " Ns[c0-0] += 1\n",
67 | " Ns[c1-1] += 1\n",
68 | " Ns[c2-2] += 1\n",
69 | " Ns[c3-3] += 1\n",
70 | " Ns[c4-4] += 1\n",
71 | " Ns[c5-5] += 1\n",
72 | " Ns[c6-6] += 1\n",
73 | " Ns[c7-7] += 1\n",
74 | " p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5])*f(Ns[6])*f(Ns[7])*f(Ns[8])*f(Ns[9]))\n",
75 | " P_ys.append(p_y)\n",
76 | " H_y += -p_y * math.log2(p_y)\n",
77 | "\n",
78 | "print(len(P_ys))\n",
79 | "print(sum(P_ys))\n",
80 | "print(b_count)\n",
81 | "print('B', B)\n",
82 | "print(math.log2(B))\n",
83 | "print('H_y', H_y)\n",
84 | "print(2**H_y)"
85 | ],
86 | "metadata": {
87 | "colab": {
88 | "base_uri": "https://localhost:8080/"
89 | },
90 | "id": "7qZfigBQGzVQ",
91 | "outputId": "1d0a0bbf-fdb0-4fc7-aee6-2248cb12cd81"
92 | },
93 | "execution_count": null,
94 | "outputs": [
95 | {
96 | "output_type": "stream",
97 | "name": "stdout",
98 | "text": [
99 | "24310\n",
100 | "1.0000000000001252\n",
101 | "24310\n",
102 | "B 24310\n",
103 | "14.569262272916092\n",
104 | "H_y 13.860045564098357\n",
105 | "14869.263443691536\n"
106 | ]
107 | }
108 | ]
109 | },
110 | {
111 | "cell_type": "code",
112 | "source": [
113 | "# calculate H(B|S=s)\n",
114 | "U_range = np.arange(1, 6252, 1250)\n",
115 | "# U_range = np.arange(1, B, 3000)\n",
116 | "H_ss = np.zeros_like(U_range).astype(float)\n",
117 | "\n",
118 | "for i, U in enumerate(U_range):\n",
119 | " H_s = 0\n",
120 | " p_s = 0\n",
121 | " choices = np.random.choice(B, U, p=P_ys)\n",
122 | " # print(len(choices), choices)\n",
123 | " for u in range(U):\n",
124 | " p_s = sum([1 for c in choices if c == u]) / U\n",
125 | " # print(p_s)\n",
126 | " if p_s != 0:\n",
127 | " H_s += -p_s * math.log2(p_s)\n",
128 | " print('U:', U, 'H_s:', H_s)\n",
129 | " H_ss[i] = H_s\n",
130 | "\n",
131 | "print('H_ss', H_ss)"
132 | ],
133 | "metadata": {
134 | "colab": {
135 | "base_uri": "https://localhost:8080/"
136 | },
137 | "id": "ePqcA5Ua7aZk",
138 | "outputId": "7f86a441-d4ff-4127-8e16-58b339d148b9"
139 | },
140 | "execution_count": null,
141 | "outputs": [
142 | {
143 | "output_type": "stream",
144 | "name": "stdout",
145 | "text": [
146 | "U: 1 H_s: 0\n",
147 | "U: 1251 H_s: 0.19578959694642661\n",
148 | "U: 2501 H_s: 0.527494304228384\n",
149 | "U: 3751 H_s: 1.6176505965077759\n",
150 | "U: 5001 H_s: 2.240630027446816\n",
151 | "U: 6251 H_s: 3.09074958685801\n",
152 | "H_ss [0. 0.1957896 0.5274943 1.6176506 2.24063003 3.09074959]\n"
153 | ]
154 | }
155 | ]
156 | },
157 | {
158 | "cell_type": "code",
159 | "source": [
160 | "H_ss = np.array([ 0., 1.14445591, 2.92142532, 5.66463935, 7.32508148, 8.84656221, 10.8259317, 12.2532801, 13.15547667])\n",
161 | "H_y = 13.860045564098357\n",
162 | "plt.plot(np.arange(9)*6250/9, H_y-H_ss)\n",
163 | "# H_ss = np.array([ 0., 0.1891638, 0.78579461, 1.54249295, 2.22972449, 3.19595329])\n",
164 | "# H_y = 13.860045564098357\n",
165 | "# plt.plot(np.arange(6)*6250/6, H_y-H_ss)\n",
166 | "plt.xlabel('Number of clients')\n",
167 | "plt.ylabel('Privacy leakage (in bits)')\n",
168 | "plt.show()"
169 | ],
170 | "metadata": {
171 | "colab": {
172 | "base_uri": "https://localhost:8080/",
173 | "height": 279
174 | },
175 | "id": "bbZ_N7hK87yL",
176 | "outputId": "b63f85ad-1ffd-4ae2-ab25-3659484863df"
177 | },
178 | "execution_count": null,
179 | "outputs": [
180 | {
181 | "output_type": "display_data",
182 | "data": {
183 | "text/plain": [
184 | ""
185 | ],
186 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gVVbb38e/qRM45Z1BEQIIiSQwoKgrGMaOiOGMA9DqjvuPMHZ2Z66hjGjOCiuFiFhVFxVGiiuSMCEhqsuRMw3r/ONXelrHh0N2n64Tf53nq6Tp16lStDYdF9a5da5u7IyIiqSMt7ABERKR4KfGLiKQYJX4RkRSjxC8ikmKU+EVEUkxG2AFEo2rVqt6wYcOwwxARSSjTpk3b6O7VDt2eEIm/YcOGTJ06NewwREQSipkt/7Xt6uoREUkxSvwiIilGiV9EJMXELPGb2Ytmtt7M5v7Ke/9lZm5mVWN1fhER+XWxvOJ/Geh16EYzqwecCayI4blFRCQfMUv87j4e2PQrbz0G/AFQdTgRkRAUax+/mfUBst19VhT7DjCzqWY2dcOGDcUQnYhIaii2xG9mpYH/B/w5mv3dfYi7d3D3DtWq/cfzB1EZv2gDL078kZ17cwr0eRGRZFScV/xNgEbALDNbBtQFpptZzVid8IsF67h/1HxOfuDfPPjpQtZt2xOrU4mIJAyL5UQsZtYQGOXurX7lvWVAB3ffeKTjdOjQwQv65O70FZsZOmEpn85dS3qacX6bOtzYvRHH1CxfoOOJiCQKM5vm7h0O3R6zkg1mNgLoAVQ1s1XAf7v7sFidLz/t6lfimSvbs+KnXbw46UfenLKSd6evonvzagzo1pguTatgZsUdlohIaGJ6xV9UCnPFf6gtu/bx+uQVvDRpGRt37OWYmuUY0L0xvVvXJitDz7OJSPLI74o/5RJ/rr05B/hg5mpeGL+UH9bvoEb5ElzXpRGXn1ifCqUyi/RcIiJhUOLPh7szbtEGXpiwlEmLf6JMVjqXnVif67o0pG6l0jE5p4hIcVDij8Lc7K0MnbCUj2avAeCc42sxoFtjjq9bIebnFhEpakr8R2H1lt28/PUy/nfyCnbszaFT48rc2K0xp7aoTlqabgSLSGJQ4i+A7Xv28+aUlbw48UdWb91Dk2pluLFbY/qeUIeSmenFHo+IyNFQ4i+E/QcO8smcNQwZv5R5q7dRtWwW15zckKs6NaBymazQ4hIRORwl/iLg7nyz9CdeGL+Ur77fQMnMNC5pX4/+XRvRsGqZsMMTEfmFYn+AKxmZGZ2bVKVzk6osWredoROW8uaUlbw2eTlntqzBgO6Nad+gcthhiogclq74C2n99j288vVyXv12OVt376dd/YoM6N6Yni1rkq4bwSISInX1xNiufTm8M20VQyf8yIpNu2hQpTT9uzbi4vZ1KZ2lX6xEpPgp8ReTAwedz+etZciEpcxYsYWKpTO56qQGXNO5AdXLlQw7PBFJIUr8IZi2fBNDxi/l8/nryExL44IT6nBDt0Y0q1Eu7NBEJAUo8Yfox407eXHij7w9bSV79h/k1BbV+F2PppzYSDeCRSR2lPjjwKad+3jt2+UM/3oZP+3cR7+TG3DPOcfqYTARiYn8Er/qEBejymWyGHh6MybdfRr9uzZi+DfL6f3kROZmbw07NBFJIUr8ISiZmc6ferfktf4nsX3Pfi54ZhLPjF3MgYPx/9uXiCQ+Jf4QdW1Wlc8Gd+fMljV56NPvuXzIt6zctCvssEQkySnxh6xi6SyeuuIEHr20DfPXbOOcJybw3vRVJMK9FxFJTEr8ccDMuLBdXUYP6sYxtcpxx1uzuHXEDLbs2hd2aCKShJT440i9yqV5Y8DJ/KFXCz6bu5Zej09g4g8bww5LRJKMEn+cSU8zbu7RlJG3dKFMiXSuGjaZv46az579B8IOTUSSRMwSv5m9aGbrzWxunm0Pm9lCM5ttZu+bWcVYnT/RtapTgVG3daPfyQ0YNvFH+jw1iQVrtoUdlogkgVhe8b8M9Dpk2xiglbu3BhYB98Tw/AmvVFY69/VpxUvXdWTTrn30eWoSL4xfykEN+xSRQohZ4nf38cCmQ7Z97u45wctvgbqxOn8yObVFdT4b3J0eLarx908WcOXQyazesjvssEQkQYXZx389MDq/N81sgJlNNbOpGzZsKMaw4lPlMlk8f3V7HrqoNbNXbaHX4+P5cNbqsMMSkQQUSuI3sz8COcDr+e3j7kPcvYO7d6hWrVrxBRfHzIxLO9bjk0HdaFq9LANHzGDQGzPYunt/2KGJSAIp9sRvZtcCvYErXU8pFUiDKmV466aTuaNnc0bNXsPZj4/nmyU/hR2WiCSIYk38ZtYL+ANwvrurNkEhZKSnMfD0Zrz7u86UyEzniqHf8sDoBezN0bBPETm8WA7nHAF8A7Qws1Vm1h94CigHjDGzmWb2XKzOnyra1qvIxwO7cvmJ9Xl+3FL6Pv01i9ZtDzssEYljqsefRL6Yv4673p3N9r053N3rGK7t3JA0TfgukrJUjz8FnNGyBp8O7k63plW5f9R8+r30HWu37gk7LBGJM0r8SaZauRIM7deB/7ngeKYu28xZj4/nkzlrwg5LROKIEn8SMjOuOKk+Hw/sSsMqpbn59en811uz2L5Hwz5FRIk/qTWuVpZ3fteZgac34/0Zqzj7iQlMWbbpyB8UkaSmxJ/kMtPTuKNnc97+bWfSzPjN89/w8GcL2ZdzMOzQRCQkSvwpon2DSnwyqBuXtK/H018t4cJnJ7F4/Y6wwxKRECjxp5CyJTJ48OLWPHdVe7I376b3kxN45ZtlmuZRJMVkHGkHM6sOdAFqA7uBucBUd1dfQYLq1aom7epX5PfvzObPH8zjy4Xreeji1lQvVzLs0ESkGOR7xW9mp5rZZ8DHwNlALaAlcC8wx8zuM7PyxROmFLXq5Uvy8nUdub/PcXyz5Cd6PT6Bz+atDTssESkGh7viPwe40d1XHPqGmWUQKbTWE3g3RrFJjJkZ15zckM5NqjD4zZnc9Oo0Lj+xPn/r24p0PfErkrTyTfzu/vvDvJcDjIxJRFLsmlYvx3u/68IjY77n+XGRGb7+cdHxmCn5iySjI97cNbNBZlbeIoaZ2XQzO7M4gpPik5WRxj1nH8vA05ry5tSVPPjp92GHJCIxEs2onuvdfRtwJlAJuBr4R0yjktDc3rM5V3Wqz3PjlvD8uCVhhyMiMXDEUT1A7u/75wCvuvs8Ux9A0jIz7ju/FVt27eeB0QupVDqLSzvWCzssESlC0ST+aWb2OdAIuMfMygEaypnE0tOMRy9ty7Y9Odz93mzKl8qkV6uaYYclIkUkmq6e/sDdQMdg1qws4LqYRiWhy8pI47mr2tGmXkUGjpjB10s2hh2SiBSRaBL/GHef7u5bANz9J+Cx2IYl8aB0VgYvXduRRlXLcOPwqcxetSXskESkCBzuAa6SZlYZqGpmlcyscrA0BOoUV4ASroqls3il/4lUKpPFtS9NUX0fkSRwuCv+m4BpwDHA9GB9GvABkblzJUXUKF+S1/qfRJrBNcMms3rL7rBDEpFCyDfxu/sT7t4IuNPdG+VZ2ri7En+KaVi1DMOvP5Hte3K4ethkNu3cF3ZIIlJAh+vqOS1YzTazCw9diik+iSPH1a7AsGs7smrzbq576Tt27M0JOyQRKYDDdfWcEvw871eW3kc6sJm9aGbrzWxunm2VzWyMmf0Q/KxUiNglBCc2qswzV7Zj7upt3PTqVPbmHAg7JBE5SharWuxm1h3YAbzi7q2CbQ8Bm9z9H2Z2N1DJ3e860rE6dOjgU6dOjUmcUjDvTV/FHW/NotdxNXn6ynYq6iYSh8xsmrt3OHR7NLV6qpjZv4IaPdPM7Akzq3Kkz7n7eODQCV77AMOD9eFA3yhilzh0Ybu6/Ll3Sz6dt5Y/vj9Hk7mIJJBoxvG/AWwALgIuDtbfLOD5arj7mmB9LVAjvx3NbICZTTWzqRs2bCjg6SSWru/aiIGnNeWNKSrqJpJIoinZUMvd/5rn9d/M7DeFPbG7u5nle5no7kOAIRDp6ins+SQ2bu/ZnE279vHcuCVUKp3JTac0CTskETmCaK74Pzezy8wsLVguBT4r4PnWmVktgODn+gIeR+JEblG33q1r8cDohbw1ZWXYIYnIEeR7xW9m2wEnUp1zMPBq8FY6kZu2dxbgfB8C/YiUde5H5GEwSXAq6iaSWA73AFc5dy8f/Exz98xgSXP3I861a2YjgG+AFma2ysz6E0n4Pc3sB+AMVNc/aaiom0jiiNlwzqKk4ZyJY8uufVz6/Ddkb97NiAGdaF23YtghiaSsAg/nFDkaFUtn8Wr/k34u6rZkg4q6icQbJX4pcnmLul09VEXdROJNVInfzNLNrLaZ1c9dYh2YJDYVdROJX9E8uXsbsA4YA3wcLKNiHJckARV1E4lP0VzxDwJauPtx7n58sLSOdWCSHFTUTST+RJP4VwJbYx2IJK/Tj63Bwxe3ZtLinxg0YiYHDsb/SDKRZBZNyYalwFgz+xjYm7vR3R+NWVSSdC5sV5ctu/Zz/6j5/PH9OTxw4fGYqaKnSBiiSfwrgiUrWEQK5Pqujdi8ax9PfrmYSmWyuKvXMWGHJJKSjpj43f2+4ghEUsMdPZuzedc+nh0bKeo2oLuKuokUt8PV6nnc3Qeb2UdEavb8grufH9PIJCnlFnXbsms///PJQiqWyuLSjvXCDkskpRzuij+3KNs/iyMQSR2HFnWrUDqTs45TUTeR4qJaPRKaXftyuHLoZOZlb+Pl6zvSuUnVsEMSSSpHXavHzD4ys/PMLPNX3mtsZveb2fVFHaikjtJZGbx0bUcaVi3NjcOnMnvVlrBDEkkJhxvHfyPQDVhoZlPM7BMz+9LMlgLPA9Pc/cViiVKSloq6iRS/qLp6zKwhUAvYDSxy912xDeuX1NWT/JZt3MnFz31NVnoa7/yuM7Urlgo7JJGEV6iyzO6+zN2/cfeZxZ30JTWoqJtI8VFZZokbx9WuwNB+HVTUTSTGlPglrpzUuApPX6GibiKxFG09/lJm1iLWwYgAnNHy/4q6DX5DRd1Eilo09fjPA2YCnwav25rZh7EOTFLbhe3q8ufeLRk9dy1/fH8OifC8iUiiiKZI21+AE4GxAO4+08waxTAmEeCXRd0qlMrkrl7HkJamip4ihRVNV89+dz+0Hn+hLr/M7HYzm2dmc81shJmVLMzxJHnd0bM5V3dqwPPjl3L5C9+ybOPOsEMSSXjRJP55ZnYFkG5mzczsSeDrgp7QzOoAA4EO7t4KSAcuK+jxJLmZGff3OY4HLzqe+au30euJ8QydsFT9/iKFEE3ivw04jsgkLCOAbcDgQp43AyhlZhlAaWB1IY8nSczM+E3H+oy54xS6NKnK3z5ewMXPfc3i9dvDDk0kIYVSpM3MBgF/J/Ik8OfufuWv7DMAGABQv3799suXLy/eICUuuTsfzlrNf384j117DzDojGYM6N6YzHSNTBY5VH5P7h4x8edTj38rMBV43t33HGUglYB3gd8AW4C3gXfc/bX8PqOSDXKoDdv38pcP5/HxnDW0qlOehy5qQ8va5cMOSySuFKZkw1JgB/BCsGwDtgPNg9dH6wzgR3ff4O77gfeAzgU4jqSwauVK8PSV7Xj2ynas3bqX85+ayKOff68HvkSiEM1wzs7u3jHP64/MbIq7dzSzeQU45wqgk5mVJtLVczqR3x5EjtrZx9eiU+Mq/HXUfP715WI+nbeWhy5uQ9t6FcMOTSRuRXPFX9bM6ue+CNbLBi+PupKWu08G3gGmA3OCGIYc7XFEclUqk8Wjv2nLS9d2ZPueHC58ZhIPfLKAPft19S/ya6Lp4z8HeA5YAhjQCLiZyANdN7r74zGOUX38ErVte/bzwCcLGfHdChpVLcNDF7emY8PKYYclEooC39wNPlwCOCZ4+f3R3tAtLCV+OVqTFm/krndnk71lN/1Obsjvz2pBmRLR9GyKJI9C1eMHmgEtgDbApWZ2TVEGJ1LUujStymeDu9Pv5IYM/2YZZz0+nok/bAw7LJG4EE2Rtv8GngyWU4GHgPNjHJdIoZUpkcFfzj+Ot246maz0NK4aNpm7353Ntj37ww5NJFTRXPFfTGTkzVp3v47IVX+FmEYlUoQ6NqzMJ4O6cdMpjXlr6krOfHQ8Xy5cF3ZYIqGJJvHvdveDQI6ZlQfWA/ViG5ZI0SqZmc49Zx/L+zd3oUKpTK5/eSq3vzmTzZriUVJQNIl/qplVJPKw1jQiwzC/iWlUIjHSpl5FPrqtK4NOb8ZHs1bT87FxjJ6zJuywRIpVNMM5zYOdzKwhUJ7IyJ69MY8uoFE9EgvzV2/jD+/OYm72Ns5uVZP7+7SiWrkSYYclUmQKM6pnWO6Kuy8jUsLhk6ILTSQcLWuXZ+TNXfhDrxb8e+F6ej42jvdnrNJsX5L0okn82Wb2DPxcYO1zIN+CaiKJJCM9jZt7NOWTgV1pXLUMt785i/7Dp7Jm6+6wQxOJmSMmfnf/E7DDzJ4jkvQfcfeXYh6ZSDFqWr0cb/+2M3/q3ZKvl2zkzEfH88Z3K3T1L0kp38RvZhfmLsBkoBMwA/Bgm0hSSU8z+ndtxGeDu3NcnfLc/d4crho2mZWbdoUdmkiRyvfmrpkd7qre3f362IT0n3RzV4rbwYPOiCkreOCThRx05w9nteCakxtqsndJKIWq1RM2JX4JS/aW3dzz3hzGL9pAx4aVePCi1jSuVvbIHxSJA4WZgask0J/IvLslc7fril9Shbvz7vRs7v9oHntzDnJHz+b079qIDE33KHGuMMM5XwVqAmcB44C6RGbgEkkJZsbF7evyxR2n0L15NR4YvZCLnv2a79fqn4EkpmgSf9NgZM9Odx8OnAucFNuwROJP9fIlGXJ1e568/ARWbt5N7ycn8MQXP2i6R0k40ST+3FKGW8ysFZECbdVjF5JI/DIzzmtTmzG3d6dXq1o89sUiej0+gXGLNoQdmkjUokn8Q4IHt/4EfAjMJ1KaWSRlVSlbgicvP4Hh158IQL8Xv+N3r00je4se/JL4p1E9IoW0N+cAQyf8yJNf/oBh3HpaU27o1ogSGelhhyYprsA3d82shpkNM7PRweuWZtY/FkGKJKISGenccmrT4OZvVR7+7HvOfnwC49X9I3Eqmq6el4HPgNrB60XA4FgFJJKo6lYqzfNXd+Cl6zpy0J1rXvyOm1+fxmp1/0iciSbxV3X3t4CDAO6eAxRqGIOZVTSzd8xsoZktMLOTC3M8kXhyaovqfDq4O3ee2ZwvF67n9EfG8czYxezLORh2aCJAdIl/p5lVAXJr8ncCthbyvE8An7r7MUSmclxQyOOJxJWSmencelozxtx+Ct2aVeWhT7+n1xOa8F3iQzSJ/w4io3mamNkk4BXgtoKe0MwqAN0J6vy7+z5331LQ44nEs3qVSzPkmg68dG1HDhx0rho2mVten66yzxKqqEb1mFkG0AIwIrNv7T/CRw53rLbAECLDQtsQmc5xkLvvPGS/AcAAgPr167dfvnx5QU8pEhf27D/AkPFLefqrxaSnGQNPb8b1XRqRlaHSDxIbR12r50ill939vQIG0gH4Fuji7pPN7AlgW/B08K/ScE5JJis37eL+UfMZM38dTaqV4f4+rejStGrYYUkSyi/xZxzmM+cd5j0HCpT4gVXAKnefHLx+B7i7gMcSSTj1KpfmhWs68OXCdfzlw/lcOXQy57auxb3nHkutCqXCDk9SQL6J392vi8UJ3X2tma00sxbu/j1wOpFuH5GUctoxNejcpCrPj1vKM2MX89XC9Qw6vRnXqftHYiyUJ3eDfv6hQBaRyduvc/fN+e2vrh5Jdis37eK+j+bzxYJ1NK1elvvPP47O6v6RQtJELCIJ4N8L1nHfR/NZsWkXvVvX4t5zW1KzQskjf1DkVxSmHr+IFJPTj63B57d35/YzmjNm/jpOf2QsQ8YvYf8BPfwlRSeaWj3TzOyWoEKniMRYycx0Bp0Refjr5CZV+J9PFnLOExP4eoke/pKiEc0V/2+I1OmZYmZvmNlZZqYZp0VirH6V0gzt15Gh13RgT84BrnhhMreNmMHarXvCDk0SXNR9/GaWBvQGniVSq+cl4Al33xS78CLUxy+pbs/+Azw7dgnPjltCZpox+IzmXNulIZma91cOo1B9/GbWGngEeBh4F7gE2AZ8WZRBisivK5mZzu09mzPm9u6c1LgKf/9kgbp/pMCi6uMHHgOmAK3dfaC7T3b3R4gMxRSRYtKgShlevDbS/bN7f6T7Z+CIGazbpu4fid4Ru3rMrLG7h5rg1dUj8p/27D/AM2OX8FzQ/XN7z+b066zuH/k/henqucHMKuY5UCUz+1uRRiciR61kZjp3BN0/JzaqzN8+XsC5/5rAt0t/Cjs0iXPRJP6z85ZNDp6wPSd2IYnI0cjt/nnhmg7s3HuAy4Z8y+A3ZrB1d4GL6EqSiybxp5tZidwXZlYKKHGY/UWkmJkZPVvW4Is7TmHgaU0ZNXsNfZ6ayMK128IOTeJQNIn/deDfZtY/mGR9DDA8tmGJSEGUykrnjjNb8OZNndi17wB9n57EBzOzww5L4swRE7+7Pwj8HTg2WP7q7g/FOjARKbj2DSozamBXWtetyKA3ZvKXD+dpzl/52eHq8f/M3UcDo2Mci4gUoerlSvL6DSfx4OiFDJ34I3Oyt/LMle2oUV5F31JdNOP4O5nZFDPbYWb7zOyAmanjUCQBZKancW/vljx1xQksWLONc/81kcka9ZPyounjfwq4HPgBKAXcADwdy6BEpGj1bl2bD27pQvlSGVwxdDJDJywlEUqyS2xE9aSHuy8G0t39gLu/BPSKbVgiUtSa1SjHB7d0oeexNfjbxwu4dcQMdu7NCTssCUE0iX+XmWUBM83sITO7PcrPiUicKVcyk2evasc9Zx/D6Dlr6PP0JJZs2BF2WFLMokngVwf73QrsBOoBF8UyKBGJHTPjplOa8Fr/k9i8cx99nprEp3PXhB2WFKNoEn97wN19m7vf5+53BF0/IpLAOjetyke3daVp9bL89rXpPDB6ATma6SslRJP4zwMWmdmrZtbbzKIaAioi8a92xVK8eVMnrupUn+fHLeXqYd+xccfesMOSGIvmAa7rgKbA20RG9ywxs6GxDkxEikeJjHT+1vd4/nlJG6av2Mx5T05kxorNYYclMRTtqJ79RB7gegOYBvQt7InNLN3MZpjZqMIeS0QK7+L2dXnv5s5kpBuXPv8Nr327XEM+k1Q0D3CdbWYvExnHfxEwFKhZBOceBCwoguOISBE5rnYFRt3aja5Nq3LvyLnc+fZs9uw/EHZYUsSiueK/BhgJtHD3a939E3cv1OBfM6sLnEvkPxERiSMVSmcyrF9HBp/RjPdmrOLCZ75mxU+7wg5LilA0ffyXu/tIdy/KOz6PA38A8h1CYGYDzGyqmU3dsGFDEZ5aRI4kLZjQ/cVrO5K9ZTe9n5zAVwvXhx2WFJF8E7+ZTQx+bjezbXmW7YWp1WNmvYH17j7tcPu5+xB37+DuHapVq1bQ04lIIZzaojof3dqVupVKc/3wKTw2ZhEHD6rfP9Hlm/jdvWvws5y7l8+zlHP38oU4ZxfgfDNbRuRm8Wlm9lohjiciMVS/Smneu7kzF55Qlyf+/QPXD5/Cll37wg5LCuGwXT3ByJuFRXlCd7/H3eu6e0PgMuBLd7+qKM8hIkWrZGY6/7ykNX+/oBWTFm/kvKcmMjd7a9hhSQEdNvG7+wHgezOrX0zxiEicMjOuPKkBb910MjkHnIue/Zp3pq0KOywpgGhG9VQC5pnZv83sw9ylKE7u7mPdvXdRHEtEiscJ9Ssx6rautG9QiTvfnsUf35/D3hwN+Uwk0ZRf+FPMoxCRhFKlbAleuf5EHhmziGfHLmHu6m08e2U7alcsFXZoEoXDjeopaWaDgUuAY4BJ7j4udym2CEUkLmWkp3FXr2N47qr2LFm/g95PTmTS4o1hhyVROFxXz3CgAzAHOBt4pFgiEpGE0qtVTT64tQtVymRx9bDJPDt2iUo9xLnDJf6W7n6Vuz8PXAx0K6aYRCTBNKlWlpG3dOGc42vx4KcL+e1r09i+Z3/YYUk+Dpf4f/5bK2yJBhFJfmVKZPDk5Sfwp94t+WLBevo8NYlF67aHHZb8isMl/jZ5n9YFWhfFk7sikrzMjP5dGzHixk5s25ND36cn8dGs1WGHJYc43JO76Yc8rZtRRE/uikiSO7FRZT4e2JWWtcpz24gZ/HXUfPZrdq+4oUnTRSQmapQvyYgBnbi2c0OGTfyRK1+YzPrte8IOS1DiF5EYykxP4y/nH8cTl7VlTvZWzv3XRD6Yma1RPyFT4heRmOvTtg7v39KZGuVLMOiNmVzwzNdMW74p7LBSlhK/iBSLY2qW58NbuvLPS9qwZutuLnr2G2753+ms3KRJXoqbEr+IFJu0NOPi9nX56s4eDD6jGV8uWM/pj4zjgdEL2KZx/8VGiV9Eil3prAwGn9Gcr+7swXltavP8uKWc+vBYXvt2OTka/RNzSvwiEpqaFUryyKVt+OjWrjSpXpZ7R87lnH9NYNwiTbcaS0r8IhK64+tW4M0BnXjuqvbszTlIvxe/o9+L3/GDnvyNCSV+EYkLZkavVjUZc/sp3HvusUxfsZleT0zg3pFz+GnH3rDDSypK/CISV7Iy0rihW2PG/f5Uru7UgBHfraTHw2N5ftwSTfhSRJT4RSQuVS6TxV/OP47PBnfnxEaVeWD0Qs54dBwfz16jB8AKSYlfROJa0+plGXZtR17rfxJlsjK45X+nc8lz3zBz5ZawQ0tYSvwikhC6NqvKxwO78eBFx7Psp130fXoSg9+YQfaW3WGHlnCU+EUkYaSnGb/pWJ+xv+/Brac2ZfTctZz2z7E88vn37NyraUOiVeyJ38zqmdlXZjbfzOaZ2aDijkFEElvZEhnceVYLvryzB71a1eTJLxfT459jeXPKCg4cVP//kYRxxZ8D/Je7twQ6AbeYWcsQ4hCRBFenYimeuOwE3r+5M/Url+aud+fQ+8mJfK1J3w+r2BO/u69x9+nB+nZgAVCnuOMQkeRxQv1KvPPbk3nqihPYvmc/V35ARjcAAAvHSURBVAydzA3Dp7Bkw46wQ4tLFuawKDNrCIwHWrn7tkPeGwAMAKhfv3775cuXF3t8IpJ49uw/wEuTlvH0V4vZs/8AV3VqwKDTm1GpTFbYoRU7M5vm7h3+Y3tYid/MygLjgL+7+3uH27dDhw4+derU4glMRJLCxh17eWzMIkZ8t4JyJTMZeHozru7UgKyM1BnTkl/iD+VPwMwygXeB14+U9EVECqJq2RL8/YLj+XRwd9rUq8hfR83nzMfG8dm8tSn/AFgYo3oMGAYscPdHi/v8IpJamtcoxyvXn8jL13UkMz2Nm16dxuUvfMvc7K1hhxaaMK74uwBXA6eZ2cxgOSeEOEQkhfRoUZ3Rg7rx176tWLRuB+c9NZE7357Fum2pNwF8qDd3o6U+fhEpStv27Ofprxbz0sRlpKcZN53SmCtPakC1ciXCDq1Ixd3N3aOhxC8isbDip108+OlCPp6zhvQ0o0vTqvRtW5szj6tJ2RIZYYdXaEr8IiL5WLx+OyNnrGbkzGxWbd5Nycw0erasSd+2tenevBqZ6Yk5EkiJX0TkCNydacs3M3JmNh/PXsPmXfupXCaLc4+vRd8TatOufiUi41MSgxK/iMhR2JdzkPGLNjByZjZj5q9jb85B6lUuRZ82deh7Qm2aVi8XdohHpMQvIlJAO/bm8NnctYycmc2kxRs56NCqTnn6tq3DeW1qU6N8ybBD/FVK/CIiRWD99j18NGsNH8zMZvaqraQZdG5SlT5ta9OrVU3KlcwMO8SfKfGLiBSxJRt28MGMbEbOXM2KTbsokZHGGcfWoE/b2vRoUT308hBK/CIiMeLuzFi5hQ9mZDNq9hp+2rmPiqUzOef4WvRtW4cODSqRllb8N4WV+EVEisH+AweZ+MNGRs7M5vN569i9/wB1Kpbi/La1ueCEOjSvUXw3hZX4RUSK2c69OYyZv46RM7OZ8MNGDhx0jq1Vnr5ta3N+29rUqlAqpudX4hcRCdHGHXsZNWs1I2euZubKLZjBSY0qc8EJdejVqhYVShX9TWElfhGROPHjxp18MDObD2au5seNO8nKSOO0FtXpe0JtTj2mOiUy0ovkPEr8IiJxxt2ZvWorI2dm89Gs1WzcsY9yJTM49/ha9Glbh5MaVS7UTWElfhGROJZz4CCTlvzEBzOy+WzeWnbuO0CtCiV55JI2dG5atUDHzC/xJ375ORGRJJCRnsYpzatxSvNq7N53gDEL1vHBjGzqVS5d9Ocq8iOKiEihlMpK5/w2tTm/Te2YHD8xa42KiEiBKfGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIilHiFxFJMUr8IiIpJiFKNpjZBmB5AT9eFdhYhOHEE7UtMSVr25K1XZC4bWvg7tUO3ZgQib8wzGzqr9WqSAZqW2JK1rYla7sg+dqmrh4RkRSjxC8ikmJSIfEPCTuAGFLbElOyti1Z2wVJ1rak7+MXEZFfSoUrfhERyUOJX0QkxSR14jezXmb2vZktNrO7w44nGmb2opmtN7O5ebZVNrMxZvZD8LNSsN3M7F9B+2abWbs8n+kX7P+DmfULoy15mVk9M/vKzOab2TwzGxRsT4a2lTSz78xsVtC2+4LtjcxsctCGN80sK9heIni9OHi/YZ5j3RNs/97MzgqnRb9kZulmNsPMRgWvk6Vdy8xsjpnNNLOpwbaE/z5Gxd2TcgHSgSVAYyALmAW0DDuuKOLuDrQD5ubZ9hBwd7B+N/BgsH4OMBowoBMwOdheGVga/KwUrFcKuV21gHbBejlgEdAySdpmQNlgPROYHMT8FnBZsP054HfB+s3Ac8H6ZcCbwXrL4HtaAmgUfH/T4+A7eQfwv8Co4HWytGsZUPWQbQn/fYxmSeYr/hOBxe6+1N33AW8AfUKO6YjcfTyw6ZDNfYDhwfpwoG+e7a94xLdARTOrBZwFjHH3Te6+GRgD9Ip99Plz9zXuPj1Y3w4sAOqQHG1zd98RvMwMFgdOA94Jth/attw2vwOcbmYWbH/D3fe6+4/AYiLf49CYWV3gXGBo8NpIgnYdRsJ/H6ORzIm/DrAyz+tVwbZEVMPd1wTra4EawXp+bYzrtgddACcQuTJOirYF3SEzgfVE/vEvAba4e06wS944f25D8P5WoArx2bbHgT8AB4PXVUiOdkHkP+fPzWyamQ0ItiXF9/FINNl6gnF3N7OEHYNrZmWBd4HB7r4tckEYkchtc/cDQFszqwi8DxwTckiFZma9gfXuPs3MeoQdTwx0dfdsM6sOjDGzhXnfTOTv45Ek8xV/NlAvz+u6wbZEtC74tZLg5/pge35tjMu2m1kmkaT/uru/F2xOirblcvctwFfAyUS6A3IvrvLG+XMbgvcrAD8Rf23rApxvZsuIdJWeBjxB4rcLAHfPDn6uJ/Kf9Ykk2fcxP8mc+KcAzYIRCFlEbjZ9GHJMBfUhkDtaoB/wQZ7t1wQjDjoBW4NfUz8DzjSzSsGohDODbaEJ+nqHAQvc/dE8byVD26oFV/qYWSmgJ5F7GF8BFwe7Hdq23DZfDHzpkTuFHwKXBaNjGgHNgO+KpxX/yd3vcfe67t6QyL+fL939ShK8XQBmVsbMyuWuE/kezSUJvo9RCfvuciwXInfiFxHpb/1j2PFEGfMIYA2wn0h/YX8i/aT/Bn4AvgAqB/sa8HTQvjlAhzzHuZ7ITbTFwHVx0K6uRPpUZwMzg+WcJGlba2BG0La5wJ+D7Y2JJLjFwNtAiWB7yeD14uD9xnmO9cegzd8DZ4fdtjxx9eD/RvUkfLuCNswKlnm5+SEZvo/RLCrZICKSYpK5q0dERH6FEr+ISIpR4hcRSTFK/CIiKUaJX0QkxSjxS9wxMzezR/K8vtPM/lJEx37ZzC4+8p6FPs8lZrbAzL462rjMbKiZtSzgeXuYWeeCfFZShxK/xKO9wIVmVjXsQPLK87RqNPoDN7r7qUd7Hne/wd3nH+3nAj0AJX45LCV+iUc5ROY4vf3QNw69YjezHcHPHmY2zsw+MLOlZvYPM7vSInXy55hZkzyHOcPMpprZoqAeTW6RtYfNbEpQb/2mPMedYGYfAv+RjM3s8uD4c83swWDbn4k8sDbMzB7+lc/cFXxmlpn941feH2tmHYL1M83sGzObbmZvB7WOcmvJ3xdsn2NmxwTF734L3G6RGvPdgt885gbnGh/dH78kOxVpk3j1NDDbzB46is+0AY4lUtZ6KTDU3U+0yKQvtwGDg/0aEqnL0gT4ysyaAtcQeQy/o5mVACaZ2efB/u2AVh4pKfwzM6sNPAi0BzYTqfTY193vN7PTgDvdfeohnzmbSInfk9x9l5lVzq8xwW889wJnuPtOM7uLSG38+4NdNrp7OzO7OTjXDWb2HLDD3f8ZHGMOcJZHipFVPIo/S0liuuKXuOTu24BXgIFH8bEpHqn7v5fIo/W5iXsOkWSf6y13P+juPxD5D+IYIjVWrrFIaeXJRB7dbxbs/92hST/QERjr7hs8Uob4dSIT6RzOGcBL7r4raOehcy/k1YnIJCaTgrj6AQ3yvJ9b6G7aIe3LaxLwspndSGRyIhFd8UtcexyYDryUZ1sOwQWLmaURmV0t19486wfzvD7IL7/rh9YpcSK1WG5z918U2LJIOeKdBQu/0IzIJB+X5/N+bvsOkM+/ZXf/rZmdRGQylWlm1t7dfyr6UCWR6Ipf4lZwNfwWkRuluZYR6VoBOJ/IbFdH6xIzSwv6/RsTKRz2GfA7i5SOxsyaB1UbD+c74BQzq2pm6cDlwLgjfGYMcJ2ZlQ7Ok29XD/At0CXoisqtKNn8CMffTmRqS4LPNHH3ye7+Z2ADvywhLClKiV/i3SNA3tE9LxBJtrOI1LwvyNX4CiJJezTwW3ffQ2RqwfnAdItMdP88R/iN2CNlee8mUqZ4FjDN3T84wmc+JVLid2rQfXPnYfbdAFwLjDCz2cA3HHmCl4+AC3Jv7gIP5958Br4O4pQUp+qcIiIpRlf8IiIpRolfRCTFKPGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIivn/qyWYxSu6saAAAAAASUVORK5CYII=\n"
187 | },
188 | "metadata": {
189 | "needs_background": "light"
190 | }
191 | }
192 | ]
193 | },
194 | {
195 | "cell_type": "markdown",
196 | "source": [
197 | "# Fixed UK=50000"
198 | ],
199 | "metadata": {
200 | "id": "2eQs7BvRBM9Z"
201 | }
202 | },
203 | {
204 | "cell_type": "code",
205 | "source": [
206 | "UK = 50000 # 50000\n",
207 | "N = 10\n",
208 | "\n",
209 | "for K in [9]: # [8]\n",
210 | " U = int(UK / K)\n",
211 | " B = f(N+K-1) // f(K) // f(N-1)\n",
212 | " print(\"K, U, B:\", K, U, B)\n",
213 | "\n",
214 | " # calculate H(B|Y=y), y is uniform\n",
215 | " p_y = 0\n",
216 | " P_ys = []\n",
217 | " b_count = 0\n",
218 | " H_y = 0\n",
219 | " for comb in itertools.combinations(np.arange(N+K-1), N-1):\n",
220 | " b_count += 1\n",
221 | " p_y = f(K) / (N**K) / f(comb[0]) / f(N+K-2-comb[-1])\n",
222 | " for n in range(1, N-1):\n",
223 | " p_y = p_y / f(comb[n]-comb[n-1]-1)\n",
224 | " P_ys.append(p_y)\n",
225 | " H_y += -p_y * math.log2(p_y)\n",
226 | "\n",
227 | " # print('len(P_ys)', len(P_ys))\n",
228 | " # print('sum(P_ys)', sum(P_ys))\n",
229 | " # print('b_count', b_count)\n",
230 | " # print('B', B)\n",
231 | " # print('math.log2(B)', math.log2(B))\n",
232 | " print('H_y', H_y)\n",
233 | " # print('2**H_y', 2**H_y)\n",
234 | "\n",
235 | " # calculate H(B|S=s)\n",
236 | " H_s = 0\n",
237 | " p_s = 0\n",
238 | " choices = np.random.choice(B, U, p=P_ys)\n",
239 | " # print(len(choices), choices)\n",
240 | " start_time = time.time()\n",
241 | " for u in range(U):\n",
242 | " if u%10000==0:\n",
243 | " end_time = time.time()\n",
244 | " print('u:', u, '/', U, ', H_s:', H_s, ', time:', round((end_time-start_time)/60))\n",
245 | " p_s = sum([1 for c in choices if c == u]) / U\n",
246 | " # print(p_s)\n",
247 | " if p_s != 0:\n",
248 | " H_s += -p_s * math.log2(p_s)\n",
249 | " print('H_s', H_s)"
250 | ],
251 | "metadata": {
252 | "id": "I26xJAKI87vq",
253 | "colab": {
254 | "base_uri": "https://localhost:8080/"
255 | },
256 | "outputId": "abc10e1d-2476-4108-f9ad-883f2e6fbaca"
257 | },
258 | "execution_count": null,
259 | "outputs": [
260 | {
261 | "output_type": "stream",
262 | "name": "stdout",
263 | "text": [
264 | "K, U, B: 9 5555 48620\n",
265 | "H_y 14.7084540600815\n",
266 | "u: 0 / 5555 , H_s: 0 , time: 0\n",
267 | "H_s 0.5174473607079045\n"
268 | ]
269 | }
270 | ]
271 | },
272 | {
273 | "cell_type": "code",
274 | "source": [
275 | "Ks = [1, 2, 4, 6, 7, 8, 9, 10, 16]\n",
276 | "Ks_H_y = [3.321928, 5.743856, 9.286393, 11.8602849, 12.91726, 13.860046, 14.708454, 15.4776598, 18.95812]\n",
277 | "K_s_H_s_UK100000 = [3.321874, 5.74309, 9.25941, 11.64429, 12.33949, 5.448149, 1.7305256, 0.505497, 0]\n",
278 | "K_s_H_s_UK50000 = [3.321928, 5.738995, 9.2478531, 11.399456, 6.4285309, 1.970087, 0.517447, 0.17900, 0]\n",
279 | "K_s_H_s_UK5000 = [3.318592, 5.73230, 8.81008, 1.091727, 0.146047, 0.056241, 0, 0, 0]\n",
280 | "\n",
281 | "# plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|Y=y_0)}$')\n",
282 | "# plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|S=s), UK=5000}$')\n",
283 | "# plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|S=s), UK=50000}$')\n",
284 | "# plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|S=s), UK=100000}$')\n",
285 | "plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|P_{y,uni})}$')\n",
286 | "plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=5000}$')\n",
287 | "plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=50000}$')\n",
288 | "plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=100000}$')\n",
289 | "plt.xlabel('Number of samples K from each client')\n",
290 | "plt.ylabel('Lable Privacy information (in bits)')\n",
291 | "plt.legend()\n",
292 | "plt.show()"
293 | ],
294 | "metadata": {
295 | "id": "befatQox87ql",
296 | "colab": {
297 | "base_uri": "https://localhost:8080/",
298 | "height": 279
299 | },
300 | "outputId": "b202ec23-d599-42a4-a941-0ee1804dbff2"
301 | },
302 | "execution_count": null,
303 | "outputs": [
304 | {
305 | "output_type": "display_data",
306 | "data": {
307 | "text/plain": [
308 | ""
309 | ],
310 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1hUV/rA8e+hKyCiFAsqKgpYUYmJRlDBHrvRaDSJm2KKKcbkl76bstlssqkmpidqitEklo09xt4LJhqNYMOGBQSU3nl/f8zAolIGmGEYPJ/nuQ8zd+4993WXcObce877KhFB0zRN065lZ+0ANE3TtNpJdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappXKwdoBmJOXl5f4+/tbOwxN0zSbsW/fvkQR8S7tszrVQfj7+xMVFWXtMDRN02yGUup0WZ/pW0yapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmaZqNSMvNYduA8n20+YZH269RCOU3TtLpMRIhNzGBDdALrouOJOn2ZgkKhqYcL9/dpjYO9eb/z6w5C0zStFsvNLyTqVDLrohPYEBPPqaRMAIKauPNQ3zZEBPkS0qIh9nbK7NfWHYSmaVotk5yRy6YjCayPTmDL0Uuk5eTj5GBH77aNua9Pa/oH+eDnWd/icegOQtM0zcpEhKPx6ayPiWd9dAK/n7mMCHi7O3Nbl6ZEBPlwa4AXrs41+ydbdxCapmlWkJNfwK7YZDZEx7M+JoG4y1kAdG7uweMR7YgM9qFTMw/sLHDryFS6g9A0TashCWnZbIq5xPqYeLYeSyQztwAXRzv6BHgzvX8A/QN9aOLhYu0wi+kOQtM0zUJEhL/Op7IhJoH10fEciEsBoJmHC2O7NycyyJdebRvj4mhv5UhLpzsITdM0M8rKLWDHiUTWxySwITqBi6nZKAVd/Rry9KD2RAT5EtzUHaWsd+vIVLqD0DRNq6YLKVlsMHYI244nkpNfiKuTPeHtvYkI8qFfoA/e7s7WDrPSdAehaZpWSYWFwp/nUtgQHc+66AQOX0gFoEWjekzq2ZLIYB96tm6Es0PtvHVkKt1BaJqmmSAjJ5+txxLZEBPPhphLJKbnYKegRytPnhsaRGSQDwE+bjZx68hUuoPQNE0rw9nkTMMD5pgEdp1IIregEHcXB/oF+hAZ5EPf9t54ujpZO0yLsVgHoZSaAwwHEkSkk3Hfj0Cg8ZCGwBURCSnl3FNAGlAA5ItIqKXi1DRNK1JQKPxx5nLxA+Yj8WkAtPFy5Z7erYgI8iXU3xNHM+c8qq0sOYKYB8wGvi3aISJ3FL1WSr0LpJRzfn8RSbRYdJqmaUBqdh5bjl5iQ3QCG48kcDkzDwc7Rc/WjXgpNJiIIB/aeLtZO0yrsFgHISJblFL+pX2mDDfpJgARlrq+pmlaWU4mZrA+Op4NMQnsOZlMfqHgWd+R/oE+RAT7ENbOG496jtYO0+qs9QwiDIgXkWNlfC7AWqWUAJ+LyBdlNaSUmgZMA2jZsqXZA9U0zfblFRQSdeoyG2IMaS1iL2UA0N7XjQfC2xAZ5EO3lp4WyYhqy6zVQUwCFpTzeR8ROaeU8gF+U0rFiMiW0g40dh5fAISGhor5Q9U0zRZdzshl89FLrI9JYPORBFKz83Gyt+OWto25p5c/EUE+tGhk+YyotqzcDkIp5QdMxPCNvxmQBRwCVgKrRaSwshdUSjkAY4EeZR0jIueMPxOUUkuBnkCpHYSmaRoY0locT0gvfsAcdTqZQgEvNycGd2xCZLAvfdp54VbDGVFtWZn/Syml5gLNgRXAW0AC4AK0B4YALyqlnivrm305BgAxIhJXxnVdATsRSTO+HgS8VslraJp2A8jJL2DPyWTWRyewISaBM8mGYjodmjbg0f4BRAT70qW5dTOi2rLyutJ3ReRQKfsPAUuUUk5AmTf9lVILgH6Al1IqDnhZRL7GMCJZcM2xzYCvRGQY4AssNS42cQB+EJE1pv+TNE2ryxLTc9gYY+gQthy9REZuAc4Odtwa4MWDfdsQEeRDU4961g6zTlAipt+2V0p5Ai1E5E/LhVR1oaGhEhUVZe0wNE0zIxEh+kJa8QPm/WevIAK+DZyJCPJlQLAPvdt6Uc/JttNaWItSal9Za80qvBmnlNoEjDQeuw9IUErtEJEnzRqlpmmaUXZeATtPJLE+Jp4N0QmcT8kGoKufB08OaE9EkA8dmzWoU2ktaiNTntZ4iEiqUup+4FsReVkpVStHEJqm2a741Gxj3YQEth9PJCuvgPpO9vQJ8GLGgPb0C/LGx732FNO5EZjSQTgopZpiWNj2ooXj0TTtBlFYKBw6n1L8gPngOUNiheYN6zEh1I+IYF9ubt2o1hbTuRGY0kG8CvwKbBORvUqpNkBZC9w0TdPKlJmbz7ZjiYbaCTEJJKTloBR0b+nJM0MCiQzypb1v3cqIastM6SAuiEiXojciEquUes+CMWmaVoecu5LFhmjDA+YdJ5LIzS/E3dmB8PbeRAYbiuk0qsMZUW2ZKR3ER0B3E/ZpmqZRUCjsP3vFMOsoOoGYi4aMqP6N63PXLa2IDPIh1L8RTg43RkZUW1beQrleQG/AWyk1s8RHDQB9U1DTtGJp2XlsPZbI+ugENh1JICkjF3s7RWgrT14cFkxEsA9tvFz1rSMbU94IwglwMx7jXmJ/KnC7JYPSNK32O5OUyTpjRtTdJ5PIKxA86jnSL9CbyGBf+rbzxqO+zohqy8rsIERkM7BZKTVPRE7XYEyaptVC+QWF/H7mCuuNzxOOJ6QDEODjxr23tiYy2JfuLRvicIMU07kRlHeL6QMRmQHMNqbdvoqIjLRoZJqmWV1KZh6bjhpmHG06comUrDwc7RU3t27MnT1bEhnsQ6vGrtYOU7OQ8m4xfWf8+U5NBKJpmvWJCLHGYjrroxOIOn2ZgkKhkasTA4INaS36tPPC3UXfOroRlHeLaZ/x52ZjYr4gDIV8johIbg3Fp2maheXmF7L3VFFG1HhOJRkyogY1ceehvm2IDPalq19DXUznBmRKLqbbgM+AE4ACWiulHhSR1ZYOTtM0y0jOyL0qI2paTj5ODnb0btuY+8IMGVGbN9QZUW90pqyDeBfoLyLHAZRSbTEWDLJkYJqmmY+IcDQ+vXjW0e9nLiMC3u7O3NalKZHBvtwa0Jj6TrqYjvY/pvw2pBV1DkaxQJqF4tE0zUzyCgrZcSKp+HnCuStZAHRu7sETke2IDPKlY7MGupiOVqbyZjGNNb6MUkqtAn7C8AxiPLC3BmLTNK0KziRlsnDvGX7eF8eltBxcHO3oE+DNoxEBRAT54NtAZ0TVTFPeCGJEidfxQF/j60sYSo9qmlZL5OYXsvbwRRbuOcu244nYKegf6MOEm1rQt723zoiqVUl5s5j+Vp2GlVJzgOFAgoh0Mu57BXgAQycD8IKIrCrl3CHALAwpPb4SkTerE4um1VUnLqWzcM8ZFv9+juSMXJo3rMfMge0ZH+qny25q1WbJJ1LzgNnAt9fsf19EylxboZSyBz4GBgJxwF6l1DIROWypQDXNlmTnFbD60AUW7DnLnpPJONgpBgT7MunmlvQJ8NLTUTWzsVgHISJblFL+VTi1J3BcRGIBlFILgVGA7iC0G1rMxVQW7jnL0j/OkZKVh3/j+jw7JIhxPZrrSmuaRVhjTtujSqm7gSjgKRG5fM3nzYGzJd7HATfXVHCaVptk5uaz4sAFFuw9wx9nruBkb8fgTk2YdFMLbmnTWM9A0izKlIVyzsA4wL/k8SLyWhWu9ynwTwyzof6JYY3FvVVop2R804BpAC1btqxOU5pWaxw6l8KCPWf4Zf950nPyCfBx46Xbghnb3U8X19FqjCkjiF+AFGAfkFOdi4lIfNFrpdSXwIpSDjsHtCjx3s+4r6w2vwC+AAgNDb0uqaCm2Yr0nHz++8c5Fu49w6FzqTg72HFbl6ZM6tmS0FaeupaCVuNM6SD8RGSIOS6mlGoqIheMb8cAh0o5bC/QTinVGkPHMBG40xzX17Ta6ExSJvN2nOKnqLOk5+QT1MSd10Z1ZFRIczzq6aR4mvWY0kHsUEp1FpGDlWlYKbUA6Ad4KaXigJeBfkqpEAy3mE4BDxqPbYZhOuswEclXSj0K/IphmuscEfmrMtfWtNpORNgVm8yc7SdZFx2PvVIM79KUe3r7E9KioR4taLWCEin/roxS6jAQAJzEcItJASIiXSwfXuWEhoZKVFSUtcPQtDJl5xWw7MB55mw7SczFNBq5OjH55pZMuaWVXuGsWYVSap+IhJb2mSkjiKFmjkfTbjgJqdl8v+s083efISkjl6Am7rw1rjOjQprrVc5arVVeLqYGIpKKTsynaVV2MC6FOdtPsuLP8+QXCpFBvtx7qz+92jbWt5G0Wq+8EcQPGFJl7MPwzKDkb7MAbSwYl6bZrPyCQtYejmfOtpNEnb6Mq5M9U25pxT29/PH30uU5NdtRXi6m4cafrWsuHE2zXSmZeSzce4Zvd57m3JUsWjaqz9+Hd2B8qB8NdIlOzQaVd4vJX0ROlfO5ApqLSJwlAtM0W3E8IZ15O06yeN85svIKuKVNI14e0YHIYF+dF0mzaeXdYnpbKWWHYaHcPv6X5jsA6A9EYpi6qjsI7YYjImw9lsjX206y+eglnOztGBXSjL/d2poOzRpYOzxNM4vybjGNV0p1ACZjSIfRFMgEooFVwL9EJLtGotS0WqKwUFgXHc/sjcf5My4Fb3dnZg5sz503t8TLzdna4WmaWZU7zdWYYvvFGopF02qtgkJh1cELfLzxODEX02jZqD5vju3MmO7NcXbQ01S1uklXKNe0cuQVFPLL/vN8svE4sYkZtPV25f07ujKiSzMc7O2sHZ6mWZTuIDStFDn5BSzaF8enm04QdzmL4KYN+GRydwZ3bKIfPGs3DN1BaFoJWbkFLNhzhi+2xHIxNZuuLRryyoiORAb76IVt2g3HpA5CKdUcaMXV9SC2WCooTatp6Tn5fLfzNF9tjSUpI5eerRvxzviu3BqgVzxrNy5TCga9BdyBoeRngXG3ALqD0GxeSmYec3ecZO72U6Rk5RHe3ptH+wfQs3Ujk9vIy8sjLi6O7Gw9qU+rvVxcXPDz88PR0fRFm6aMIEYDgSJSrWJBmlabJKbn8PW2k3y38zTpOfkMCPblsYgAurZoWOm24uLicHd3x9/fX482tFpJREhKSiIuLo7WrU1PjmFKBxELOFLNanKaVhvEp2bz+eZYfthzmpz8QoZ1bsr0fgHVWtyWnZ2tOwetVlNK0bhxYy5dulSp80zpIDKB/Uqp9ZToJETk8cqFqGnWczElm9kbj/HT3jgKRBgV0oxH+gUQ4ONmlvZ156DVdlX5HTWlg1hm3DTN5qRm5/HZphPM2X6SgkLh9h4teLhvW1o2rm/t0DSt1quwgxCRb5RSTkB7464jIpJX0XlKqTkY0oUniEgn4763gRFALnAC+JuIXCnl3FMY6lAUAPllVTvStLLk5Bcwf9cZPtpwjMuZeYzs2oynBwXqjkHTKqHCpaBKqX7AMeBj4BPgqFIq3IS25wFDrtn3G9DJWK70KPB8Oef3F5EQ3TlolVFYKPyy/xwD3tvMaysO06FZA5Y/2ocPJ3Wr852Dm9vVt8vmzZvHo48+yl9//cULL7zAfffdx86dO4s/z8rKom/fvhQUGCYn2tvbExISQteuXenevTs7duwAoF+/fpw6dar4vKLjOnXqxPjx48nMzDTrvyM3N5fw8HDy8/PN2q5WeabkCngXGCQifUUkHBgMvF/RScZ1EsnX7FsrIkX/r+8C/CoZr6aVafvxREZ9vJ0nFu7HzdmRb+7tyff33UxnPw9rh2ZVTk5OZGdn4+vry3fffVe8f86cOYwdOxZ7e0MuqXr16rF//34OHDjAv//9b55/vvTvb0XHHTp0CCcnJz777DOzxxsZGcmPP/5o1na1yjOlg3AUkSNFb0TkKIZZTdV1L7C6jM8EWKuU2qeUmmaGa2l12OHzqdw9Zw+Tv9pNckYu703oysrH+tC3vbd+eAx8+OGHzJgxgwcffPCqb/vz589n1KhRpZ6TmpqKp6dnhW2HhYVx/Phxs8VaZPTo0cyfP9/s7WqVY8pD6iil1FfA98b3k4Go6lxUKfUikA+U9RvQR0TOKaV8gN+UUjFlrdw2diDTAFq2bFmdsDQbE3c5k/fWHmXp/nM0cHHkxWHB3NWrFS6O12dXzT5ylMx9UTQcNw47Z8ul5X51+V8cPp9q1jY7NGvAyyM6VnhcVlYWISEhxe+Tk5MZOXIk/fv356233sLX17f4s9zcXGJjY/H397/u/OzsbC5cuMCGDRvKvV5+fj6rV69myJBr7yRXX6dOndi7d6/Z29Uqx5QO4mFgOlA0rXUrhmcRVaKUmorh4XWkiEhpx4jIOePPBKXUUqAnZazcFpEvgC8AQkNDS21Pq1uuZOby8cbjfLPzNADTwtvwSN8APOpfPbDNu3CB1JUrSVm2nJyjRwGwc3Ki4e2313jMNaHo1k+RefPmERUVxdixYxk7duxVxyYmJtKwYcMyz9+5cyd33303hw4duu46JTuisLAw7rvvPnP/U7C3t8fJyYm0tDTc3d3N3r5mGlNmMeUA7xm3alFKDQGeAfqKSKlPtpRSroCdiKQZXw8CXqvutTXbl51XwLwdp/hk43HScvIZ192PmQPb06xhveJjClJTSf31V1KXryBz714QoV7Xrvi+9BJJX35J+tZtFu0gTPmmXxvUq1ev3NQgvXr1IjExsdSFVdd2ROaybNkyPD09CQsLAyAnJwcXFxezX0czXXk1qX8SkQlKqYMYnglcxTgTqUxKqQVAP8BLKRWHoTzp84AzhttGALtE5CGlVDPgKxEZBvgCS42fOwA/iMiaqvzjtLqhoFBY8nsc7/92lPMp2fQP9ObZoUEENTGsfi7MzSV982ZSly0nfdMmJC8PJ39/vB6djsfw4Ti1agVAdkw0ab+uRfLzUQ43diJjT09PCgoKyM7OLvWPcExMDAUFBTRu3Nik9mbOnMlTTz2Fl5cX999//1UPw8EwmvHy8mL48OFMnDiRIUOGsHnzZtq0aYOdnR0vvvgiycnJ2NkZHosmJSXh5eVVqbxBmvmV91/JE8afw6vSsIhMKmX312Ucex4YZnwdC3StyjW1ukVE2HT0Em+tjiHmYhpd/Tx4d0IIvdo2RgoLydizh9TlK0j99VcKU1Oxb9yYhpMm4jFiBC6dOl33gNqtTxgpixaTdeAA9Xv0sNK/qvYYNGgQ27ZtY8CAAcDVt45EhG+++aZ4hlNFHnnkEb744gvatGnDxIkTTTpnyJAh3HHHHUyadP2fio0bN3LbbbeZ+C/RLKW8mtQXjC8fEZFnS35mzPD67PVnaZp5RF9I5bXlh9kZm0SrxvWZfWc3buvclJxjx0h4dx4pK1aSf+ECqn593AdE4jFiJK69bil3ZODauxfY25O+dWud7CDS09Ovej916lSmTp1a5vHTp0/n/fffL+4gitZDVPY6AAEBAZw/f57o6GgWLlx43efOzs7F6xoyMjIAcHV1BQyd0bV++OEH3nzzTZPi0SzHlHH2QK7vDIaWsk/Tqi03v5BPNh1n9objNKjnyKsjOzK+pRNZa1Zx8u8ryDlyBOztce1zKz4zZ+IeGYFdfdMWwNk3aEC9kBAytmyFGTMs/C+p/bp3707//v0pKCgweaRQnsjISC5cuICdnR3x8fGsWLGi+AF23759eeaZZzh58iRXrlyXPOEqubm5jB49mvbt25d7nGZ5qoyJRCilHgYeAdpgSItRxB3YLiJTLB9e5YSGhkpUVLVm4GpWdOhcCk//fICYi2mM69CImfanyV+7hsw9e0AEl65d8BgxkgZDh+Bg4r3xayV+9hmXPphFu21bcfDyMkvc0dHRBAcHm6Wt2mjevHmMHj36ullPJW3fvp1Zs2bx9ddf4+7uzpo1a3ByciIiIqIGI9UqUtrvqlJqX1kZK8rrIDwAT+DfwHMlPkoTkeRST7Iy3UHYppz8AmZvOM4nm07QyNWJ95qn0OSbj8m/cAGnVq1oMGIEHiP+97C5OrL++otT426n2Vtv4lHGIrHKqusdhFZ3VLaDKO8ZRAqQAkwyNuIDuABuSik3ETljtqi1G9afcVd4+ucDHI1P557WTtyzbxE5323Gvl07ms2bR/2be5p1NbRLcDD2jRuTvnWb2ToITaurTCk5OgLDGohmQAKG2tTRgG1M+NZqpey8AmatP8YXW2JpUs+en9yO0ODT78i1s8Pr6afYeqsHcVm7eYyeZr2usrPDrc+tpG/eghQUoMxw713T6ipTHlK/DtwCrBORbkqp/kCte/6g2Y7fz1zmmUV/cjwhnScapTJ80/fkx8biOmAAZ+4dwPNn53F8jyG/T7hfOCE+IRW0WDmuYeGk/LKM7L/+ol6XcpfzaNoNzZRkfXkikgTYKaXsRGQjoFNwa5WWnVfAG6uiuf3THThcSWbplbUMmfMaKicH3n6RV0dm8dDBl8gtyOXfYf+mvkN9Fh9bbPY4XG/tDUqRvnWr2dvWtLrElBHEFaWUG4ZcSPOVUglAhmXD0uqaqFPJPLPoT05dSuMfHOWW1T8i2dm43DeFud1T+W/cf/Bw9uC5ns8xof0EHO0diboYxaqTq3j2pmdxczJPaVAAB09PXDp3JmPLVrynTzdbu5pW15gyghgFZAFPAmswTHkdYcmgtLojK7eA15YfZvznO2kSf4pfoudy8y9f4RQUyPY3xjG+6VJWnl/H1E5TWTl2JZODJ+Nob0ivMK7dOLLys1h9qqys8FXn1qcPWQcPkn/5stnb1rS6wpRkfRkASqkGwHKLR6TVGbtjk3hm8Z9cupjE+5d3ELjrV+w9GxL75Bj+7bGN5OT93NbmNh7v9jjN3Jpdd34nr06092zPkqNLGN9+vFljcwsPI/GTT8jcuZMGw4aZtW1NqytMmcX0IPAqkA0UAgpD8r42lg1Ns1UZOfn8Z00M3+w4xdiUaO478At2V5LJGB7GW93OEpO3nJs8b+KTHk/R0avsyXBKKca2G8ube97kSPIRAhsFmi1Gl86dsfPwIH3rNt1BaFoZTLnF9DSGOtL+ItJGRFqLiO4ctFLtOJ7IkFlbWP9bFHMPf8cDm+agfDyYNyOYv3XaQU59Bz6K+IivB31dbudQZHib4TjZOZn9YbWyt8ft1t6kb9uKFBaatW1rqama1LWBKXWrT506RadOna7a98orr/DOO+8Uvy/5v9mqVato3749p0+frnQ8/v7+dO7cmZCQEEJDr57Ds2bNGgIDAwkICLgqv1RZ+yv6rCaZ0kGcAMxblVyrc9Ky83hh6UGmfr6N0X+s5IvN79Hk/HF2TuzExNGn2dUwkb/f8neWjFxCvxb9TF785uHswYBWA1gRu4Ls/LLrF1SFa1g4BZcSDfmd6jBz16QGOHv2LK+88goPPPCAVf6Ambtu9fr163n88cdZvXo1raq4Yn/jxo3s37+fktkcCgoKmD59OqtXr+bw4cMsWLCAw4cPl7m/vHOswZQO4nlgh1Lqc6XUh0WbpQPTbMeWo5cY8sFWji5fy/wdsxj2xyrOh7bkkfsL+TTgFA+EPMiqsauYEDgBB7vK12EY124cablprDuzzqxxu97aG4D0rdvM2m5tY+6a1Lt37+axxx7j4YcfZty4caxatcpssX777bd06dKFrl27ctddd5V7rLnqVm/ZsoUHHniAFStW0LZt22q3V9KePXsICAigTZs2ODk5MXHiRH755Zcy95d3jjWY8l/r58AG4CCGZxCaBkBqdh7/WhHNuq2HmHl0Jd1P/k52s0a8c5c7e/3OMCpgFI+GPIqvq2/FjZXjpiY30dK9JYuPLmZ4myqVJymVo48PzsHBZGzZgte0B8zT6Orn4OJB87RVpElnGFrxt/SaqEmdlpbG5MmT2bZtG76+vvTr14+bbrqpwtjCwsJIS0u7bv8777xTnG78r7/+4vXXX2fHjh14eXmRnFx+yjdz1K3Oyclh9OjRbNq0iaCgoCrFDYbnZYMGDUIpxYMPPsi0adMAOHfuHC1atCg+zs/Pj927d5e5v7xzrMGUDsJRRGZaPBLNpmw8ksBLi/Zz8x/rmHvsN+wln1UDGvJ9txR6tryVn3vMNNtDZaUUY9qNYdbvsziVcgp/D3+ztAuG6a5Jc+dSkJaGvY3XPq6JmtQ///wz3bt3p0mTJgC4uLiYVBZ0qwmLEjds2MD48ePxMmbZbdSoUbnHV1S3uqzbmCX3Ozo60rt3b77++mtmzZpVpbgBtm3bRvPmzUlISGDgwIEEBQURHh5u0rm1mSkdxGql1DQMU1xzinaaktFVKTUHQ0W6BBHpZNzXCPgR8AdOARNE5LrJ6Eqpe4CXjG9fF5FvTIhVs7CUzDxeW3GYv9Zt49VDS2mefI5jwQ2Y1TcPjzZNmd3jHW5tfqvZrzuq7Shm/zGbpceX8mSPJ83Wrlt4GElffknGrl00GDiw+g2a8E2/NqhqTeojR46UehumZEnRmJgYFi5cyCuvvFL8uanfxEtTXtvl1a1u3Lgxl69Z55KcnEzr1q2L39vZ2fHTTz8RGRnJG2+8wQsvvHDV8abG3bx5cwB8fHwYM2YMe/bsITw8nObNm3P27Nni4+Li4mjevHmZ+4vaKuuzmmZKB1FUD7DkEytTp7nOA2YD35bY9xywXkTeVEo9Z3x/bcW6RhhqWIcar7VPKbWstI5EqznrDsfzr4W7GLVnKfef2k16QxfeGWvHqa6uPNr9OUa2HYm9nWWS33nX9ybcL5xfjv/Co90exdHOPLWK64WEYOfqSsbWbebpIGxEVWtSt23b9qr74YcOHSr+xl8eU76JR0REMGbMGGbOnEnjxo1JTk4udxRxbd3qyMhIvv322+I/pm5ubjRt2pQNGzYQERFBcnIya9as4Yknnriqnfr167Ny5UrCwsLw9fUtLnJkatwZGRkUFhbi7u5ORkYGa9eu5R//+AcAN910E8eOHePkyZM0b96chQsX8sMPPxAYGFjq/vLOsYZyOwillB3wnIhUaaqAiGxRSvlfs6pPDeUAACAASURBVHsU0M/4+htgE9dXpxsM/FY0SlFK/QYMARZUJQ6tei5n5PLqLwfJWL6M/0SvpH5uJitutmdZPwcmd3+QjzrcTX1H06q6Vce4duPYeHYjW+K2ENky0ixtKkdHXHv3In3rVkTErKnFa7uq1KT+29/+xo4dOxg8eDBt2rQhPDy81JrSVdGxY0defPFF+vbti729Pd26dWPevHllHl+ybnVhYSHHjx+/rkP59ttvmT59OjNnGu6Sv/zyy6WOgBo1asSaNWsIDw/H29ubkSNHmhx3fHw8Y8aMASA/P58777yTIUOGAODg4MDs2bMZPHgwBQUF3HvvvXTsaJjeXdb+8s6pcSJS7gZEVXRMBef7A4dKvL9S4rUq+b7E/qeBl0q8/zvwdEXX6tGjh2jmtfrgBRn11Dfyc+8hcjgwSH6J7CjD3+4kr+54VS5lXqrRWPIK8iTipwh5+LeHzdpu8o8/yuHAIMk+dqxK5x8+fNis8dSUffv2yZQpUyo8rm/fvnLy5Mlyj/nvf/8rc+bMERGRzZs3y6xZs8wRYrltjxkzRo4cOSIiIgcPHpQnn3zSbNesq0r7XS3vb7wpt5jWKaWexvDcoDhJn5ihqpyIiFKq9JJ2JjI+H5kG0LJly+qGpBklpefw+qJ9eC76ltdPbCGrnuLTYXYUDu3LBzfNpG1D804HNIWDnQOjA0bz1cGvuJhxkSauTczSrlufPoBhuqtzQIBZ2rQF5qxJPXjwYB588EEOHDhAYmIiH3zwgZmiLL3ta+tWd+rUiffee89s19QMyiw5WnyAUidL2S1i4mpq4y2mFfK/h9RHgH4ickEp1RTYJCKB15wzyXjMg8b3nxuPK/cWky45ah4rD5xn2cc/MDnqR7wz01nfVRE1JohH+j5Hz6bmLeBTWXFpcQxdMpTpIdN5qOtDZmv3xPDhOPr40HLOnEqfW9dLjppSk1qzDZUtOVrhQjkxpNa4dqtOqo1lwD3G1/cApa0A+RUYpJTyVEp5AoOM+zQLSkzP4bnZq0mc8QAztnxNpms6HzzgQ6s33mLOHYus3jkA+Ln7cUvTW1h6bCmFYr5lOW5h4WTujaIwUycNuNbUqVN153CDqrCDUEo5KqUeV0otMm6PKqVMmkKilFoA7AQClVJxSqn7gDeBgUqpY8AA43uUUqFKqa+g+PbVP4G9xu01c9zS0konIvwSdYqP7n+aiZ89TZfEWBYOdOH8h08y64lfGdF2BHbKlEX3NWNcu3GczzjPrgu7zNamW1gfJC+PjD17zNamptk6U55BfAo4Ap8Y399l3Hd/RSeKSFnTG66bgiIiUSXbFJE5QOXH+1qlJKRl88n78+m98lMmpmSwO9COyw+NY0b/p/B0KT3VgrVFtIygoXNDFh9dTO9mvc3SZr0ePVD16pGxZSvu/fqZpU1Ns3WmdBA3iUjXEu83KKUOWCogrWaICP/deIBL7z3PHcdPEd8QVk/vwbh7/kWrBlVLVlZTnOydGNF2BAtiFpCcnUwjl/JX3JrCztkZ1549Sd9Wt/MyaVplmHLfoEApVTxlRSnVBiiwXEiapaVk5fKfGTNo8eSd9Io9xaYIXzx+nsfMx76v9Z1DkbEBY8kvzGf5CfPVsHINDyPvzBlyq5DuWdPqIlM6iP8DNiqlNimlNmNI3PeUZcPSLOVicjo/3D2akb+u5UxTJy5+/jwPfbyRbi1utnZolRLgGUBX764sObaEimbimcotLAyo+9ldNc1UZXYQSqmiGo+xQDvgceAxIFBENtZAbJqZnYhL5Ne7htHv4El2RbTmthW7GRh2t82uHh7XbhyxKbHsv7S/4oNN4NSyJY6tWpK+dYtZ2tM0W1feCKIo99JiEckRkT+NW04552i11B/Rsey/Zyg9T1xi9+09uOfjFTg7OFs7rGoZ7D+Y+g71WXzUfNXm3MLCydy9h8Ic/WuuaeV1EElKqbVAa6XUsmu3mgpQq76NO/YR/8AY2l9I5/eHhjL19e9r1bTVqqrvWJ9hbYax9vRa0nKvz7hZFW5hfZDsbDJtcMGlLjl6tbpYcrSmS5GW91fiNuAfQCLwbimbZgOWrvgVuxn34JOay4kXpzJ5xnuQngArn4ZLtl9qc1y7cWTlZ7H65GqztFe/Z0+UkxMZW0yrA2ALdMnRitlCyVFrlCIts4MQkVwR2QX0FpHN124WjUozi3nzvqfZS0/ilF/AlXeeZ/TkZ+HsXvg8HPZ+CaufsXaI1daxcUfae7ZnybElZmnPrl496t90U52a7qpLjpbPVkqOWqMUqSnrIDyVUv/CkJW1+HgRibBUUFr1iAgfv/set37zFSmudjSY/T69egyCqDmw6hlo0AxC7zW8P70TWvWydshVppRibLuxvLnnTWKSYwhqdH3ZyMpyDetDwptvkXfuHI6VLNTy1p63iEmOqXYMJQU1CuLZntdmxL+eLjlaebZUctQapUhN6SB+Bj4DvkKvf6j1CgqFWS89y6Clyznr60C7r78hoGUHWPYY/PEdtI2EcV+BgwtEr4BNb8A95ltLYA3D2wznvaj3WHx0MS/e8mK123MLCyPhzbdI37YdzzsmmCHCmqFLjl5NlxytPlM6iHwR+dTikWjVlp2Xz6dPPMDQDbuIbl2PW79diq+TPcwdCud/h7Cnof8LUFT1rc8M+PUFOLUd/M1fJrSmeDh7MNB/ICtjV/JU6FO4OFT8B6s8Tm3a4NCsKelbt1S6gzDlm35toEuOGthSyVFrlCI1ZSrLcqXUI0qppkqpRkWbRaPSKi0lI5s5U8cxZMMu/ujkycCf1+Gbfho+7wuJx+CO+RD59/91DmC4zeTmC5v+bb3AzWRcu3Gk5aXx2+nfqt2WUsow3XXnLiQ31wzR1T4lS46WprySo3/++Wfx+0OHDnHx4sUKr7d161b2799/3Vbyj2xERAQ///wzSUlJABXeYiqt5Oi5c+eKPy9ZcrSovTVr1tDHWP+jSFHJ0fnz5/P1119XOu6MjIziTqSo5GjR7KmS5UNzc3NZuHAhI0eOLHN/Vc+xFFNGEEWpuf+vxD5Ta1JrNeBiUiqr7x1F/yMX2XWrH3d+ugznqK/ht39A4wCYOB+82l1/omM96PMkrHkOTm6F1mE1H7yZhPqG0tK9JUuOLWFE2xHVbs8trA9XfvyRzP37ce1p/TTnlqBLjv6PrZQcrfFSpGWVmrPF7UYsOXrs1Dn5cWBPORwYJF89PV7ys1JEfpoq8nIDkYWTRbJTy28gN1Pk7fYic4aKFBbWTNAW8uWfX0qneZ3k5JWT1W4rPy1NDnfsJPHvvFPhsbrkqC45aisqW3K0vFQbEcafY0vbLNttaaaIOnCYmLtuo8PZVPZMjeDeZ9/Afs5gOPxfiHwZJnwHztc/vLuKYz0Imwmnt8NJ204xMartKOyVPUuOV3/Kq72bG/W7d6/TeZlKlhytrsGDB7Np0yZmzJjBF198wZ133mmGCMtuW5ccrRnl3WLqiyExX2njdQHMM/Fcq5L1G7Zg9/x0/DLyiX56EveE3QxfRoCdHUxeBAHXldwoW/d7YNsHsPENaB0ONpqbybu+N339+vLL8V94rNtjONqZVNeqTK5hfbj07nvkxSfg6Otjpihrl3vvvbfCY0ypKOfi4sI333xjrrBMavvuu++2yPW0/ylvodzLxp9/K2Wr+LdKs5jFixZR/+mHcM3J5+K/nmR8W2f44Q7wbAXTNleucwBwdDGMIs7ugljbzsM4rv04krOT2XK2+qOhouyuGdu3V7stW6ZLjt64bD8hzw1m3uef4P/a38l3gIL3X2FIxnrDWoYud8B9aw2dRFV0vxsa+MHGf4OZ0mdbQ+9mvfGp78PiY9VP4OccGIiDt7fO7qrdsGq8g1BKBSql9pfYUpVSM645pp9SKqXEMf+o6ThrGxHh09f/QY9ZH5Hk4YDXrFfodfAtOL4Ohr4NYz4zPE+oKgdnwygibg+cWG++wGuYg50DowNGs/38di5mVDz9sjxKKVzDwsjYsRMpJymcptVVNd5BiMgREQkRkRCgB5AJLC3l0K1Fx4nIazUbZe1SWCjMfvoRwr//mRPNnen4r+kEb5gBuelwzwq4eZp5nht0uws8Wtj8KGJMwBgKpZClx0v7taoct7A+FKakkHXwoBki0zTbUmEHoZTap5SarpSyRAX7SOCEiOgaj2XIzsvns/snMGDlJg4GNqDfowNovuFZaNLJ8LzBnHmUHJwg7Ck4F2UYmdgoP3c/ejXtxdJjSykorN4MHddevcDOjgwTUy5oWl1iygjiDqAZsFcptVApNViZrwTZRGBBGZ/1UkodUEqtVkqVuRpEKTVNKRWllIq6Ni2ArUtJz+SHO4fSf8ch9vTwYfToxjT840sIvc8wcmjQ1PwXDZkMHi0NM5pseBQxtv1YLmRcYPeF6iUzs2/YkHpdu9bp6a6aVpYKOwgROS4iLwLtgR+AOcBppdSr1Um5oZRyAkZiSAZ4rd+BViLSFfgI+G858X0hIqEiEurt7V3VcGqdCwlJrJowgF4H49jVvyWTeyTiEhcFoz6G4e8Zvu1bgoMThD9tyN10bK1lrlEDIlpE0NC5IYuOLap2W65hfcg+dIj8ClI/aFpdY9IzCKVUFwxFgt4GFgPjgVQM6ySqaijwu4jEX/uBiKSKSLrx9SrAUSnlVY1r2ZQjx06w545BdIm9zO8j2jC12R84KAX3/Qrdplg+gJA7oWErQ44mGx1FONk7MaLtCDae3UhSVlK12nILCwMRMrbvMFN0mmYbTHoGAbwP7AW6iMjjIrJbRN4FYqtx7UmUcXtJKdWk6DaWUqqnMc7q/VduI/bs3cuZe8bgn5DJ8dubM9l1G6rlLTBtEzTrVjNB2DtC+P/B+T/g6JqauaYFjGs3jvzCfFbErqhWOy4dO2Lv6Vnrp7vqkqNXq8mSo/feey8+Pj7XXQ9qpuSopUqRmjKCGC8ikSLyg4hcVcldRKqUckMp5QoMpMRqbKXUQ0qph4xvbwcOKaUOAB8CE405Q+q0tatXk/vwVBpl5HFlrDujHPZC78dhyhJwreEBVNeJ4Olv088i2jZsS4h3CIuPLaY6vz7Kzg7XPn3I2LYdKSw0Y4Q1Q5ccrVh1S45OnTqVNWuu/zJVEyVHLVmK1JQO4n6lVPEySqWUp1Lq9epcVEQyRKSxiKSU2PeZiHxmfD1bRDqKSFcRuUVE6vzYftH3c/F87insENSIAvq6xMH4eTDon2BvStJdM7N3hL7PwsU/IWZlzV/fTMa2G8vJlJPsv7S/4oPL4RbWh4LkZLIPR5spspqjS46WzxwlR8PDw0stcFQTJUctWYrUlL88Q0WkuIqGiFxWSg0DXjJLBBrzPniTbl99Q7K7okX/ywT4NIWJv4BPsHUD6zwBtrwDm96EwGGGPE82ZrD/YN7a+xaLji6im0/Vb9G53mooqJSxdQv1Olk4xXIV6ZKjlWeukqNlqYmSo5YsRWpKB2GvlHIuur2klKoHOJvl6jc4EeGLv8+kz+I1nGliR9feF2nWcaBhVXS9WpD7xt4B+j4DSx+EmBXQwbLFSSyhvmN9hrYeyooTK3iu53O4O1WQ3bYMDo0b49KpE+lbt+H18MNlHnfxjTfIiTZvTWrn4CCaXFPprDS65OjVarLkaF1lylfC+cB6pdR9Sqn7gN8Ay6RtvIEUFBTy+fS7CF+0hhOtFLeEXaDZwGdg4g+1o3Mo0ul2Q9GhTW+CDd5/B8PD6uyCbFafXF2tdlzD+pC1fz8FKSkVH1zLWaLk6IoVhskAMTExV5UbBcM38ZCQkOu2desqXpBZXttVKTla1AHB/0qO7tmzhzfeeOO6NqoTN1StfGhlz7FoKdKyCkWU3DBMSX3HuA025RxrbLZSMCgrK0e+unOoHA4MkiWjAyXjdT+RI2usHVbZDvxoKEB0aKm1I6mSwsJCGfvLWJmwfEK12snYt08OBwZJyuqr/7+qLQWDXF1dr3o/d+5cmT59epnH+/n5SVZWVqnnR0dHS+PGjSU/P/+qgkGff/65DBs2rPi4gwcPyoULF2Tu3LmyfPny4nNffvnlSsd/6NAhadeunSQmJoqISFJSUvG/o7S2ExMTJTAwsPj8iIgIiYuLu6rNHj16yPr164vba9eunRw/fvy6f3NSUpJ06NBBvvrqq0rHXeTkyZPSsWPHq/bl5eVJ69atJTY2VnJycqRLly5y6NChMvdX5Zzy2rqW2QoGXdOJrBaRp43br+bpmm5MV65cYfGESHrvO0lMt3yGD/Cg/kOboP1ga4dWtk7jwKs9bH7LJkcRSinGtRvH4aTDHE6q+uyOel26YNegQa2f7mqqopKjRYqeQYSEhHDHHXeUWXLU29ubwYMH8/DDD3Pw4MHi203VVbLkaNeuXYvLhJbF1JKj//znPwkJCSEiIqLCkqOvv/46y5Ytq3TskyZNolevXhw5cgQ/P7/i2tYly4cGBwczYcIEOnbsWOb+qpxTXlvVVlbPUbQBt2BYA5EO5AIFQGpF51ljq+0jiLNxcbIksoccDgySZVNbS+FPU0Vy0q0dlmn+/Nkwiji42NqRVElKTorc9P1N8uLWF6vVztknZsjRsHApLFGetbaMICpLlxy98VhiBDEbw6K2Y0A94H7gY/N0TzeOvw79yV8ThxJwPoMz/bMY8cBM1O1zwMnV2qGZpuMY8Ao0jiKqX6KypjVwasDItiNZdXIViVmJVW7HLawP+QkJ5Bw9ZsborEOXHNUqVFbPUbRh7F2AP0vs+6Oi86yx1dYRxPYtG2TLTR3k905BsunJdiKxm60dUtUcXGQYRfz5s7UjqZLYK7HSaV4n+eSPT6rcRu7FeDkcGCSJX35ZvM9WRxCmmjt3rly+fNnaYWhmYIkRRKYxsd5+pdR/lFJPoivRmWz1kh9wePwRnPILyRvnTt+XNxjqPtuiDmPAO9hmRxGtPVoT1jyMH4/8SG5BbpXacPT1wTkw8IbK7qpLjt64TPlDf5fxuEeBDKAFMM6SQdUViz57k6Yv/5McJ3C5rzM9X9wCHn7WDqvq7Oyg37OQeBQOVb+kpzVM6TCFpOwk1pyqeo4pt7A+ZP7+OwXpGWaMTNNqH1M6iB6AiCHD6qsiMlNEjls6MFu34NUHCfzwGxIbCk1emEzX6T+BY8ULimq94FHg09EwiiiwvTKcvZr2oq1HW74//H3R7dJKc+0TBnl5ZO4xz2pVTautTOkgRgBHlVLfKaWGK6WskBjItiyYOY6QBVs42xw6fPg27UbVoawkdnbQ7zlIOg6Hql9roaYppZjcYTLRydHsi99XpTbqd++GXf36pG+pG9NdNa0sphQM+hsQgKGwzyTghFLqK0sHZquWzX6RLqsPc7SNHb2/+YVm3UZYOyTzCxoOvp1tdhQxvM1wPJw9mB9dtURvysmJ+r16kbF1W/EopKqjEU2rKVX5HTV1oVwesBpYCOwDRlf6SjeA37euoNlXS4hvDLd88iMNm7W3dkiWUTSKSI6Fgz9ZO5pKq+dQj/Htx7Ph7Abi0uKq1IZbWB/yzp0j9+QpXFxcSEpK0p2EVmuJCElJSSblzSqpwttFSqmhGOpS9wM2AV8BEyofYt12KT6OK88/gzvg/tIz+PpfXzikTgm6DZp0gc3/MWR9tUZK8mq4I/AO5h6ay4KYBfzfTf9X6fNd+4QBkL5lM36TJxMXF3dd3iJNq01cXFzw86vcJBlT/qu+G/gReFCuKRikGRQUFLD9/ttolyScuK8PI4f8zdohWZ5S0O95WDgJ/lxYM6VQzaiJaxMGtRrEkmNLeCTkEVwdK7dg0cmvOS4dO3J5/g80mjyZ1q1bWyhSTbMeU55BTBKR/+rOoWw/Tx9C4LFcDvb1YuTTX1o7nJoTOBSahhhGEQV51o6m0qZ0mEJ6Xjq/HK9acRWvxx4l7+xZrixeUvHBmmaDyuwglFLbjD/TlFKpJbY0pVRqdS+slDqllDqolNqvlIoq5XOllPpQKXVcKfWnUqp7da9pCf/94Ck6b4ojJtCe8R+tt3Y4NatoFHHlNBwotbx4rdbFuwtdvLswP3o+hVL5JIRufftSr1s3Ej/5hMJyUmdrmq0qs4MQkT7Gn+4i0qDE5i4iDcx0/f4iEiIioaV8NhRoZ9ymAZ+a6Zpms3fdYlrMXcUFbwj7ZCn2jk7WDqnmtR8MzbrDlrchv2qrk61pSvAUzqSdYWtc5QvDKKXwfnIG+QkJXP7B9jpITatIubeYlFL2Sinzlscy3SjgW2O6kF1AQ6VUUyvFcp34c6fI+MdLFNhB41f+jlfzdtYOyTqKRxFnYH/16wPXtAGtBuBT34fvor+r0vmuPXvieuutJH3xBQXp6WaOTtOsq9wOQkQKgCNKqZYWuLYAa5VS+5RS00r5vDlwtsT7OOO+qyilpimlopRSUTU1iyQ/N49d00bidRkS7omgW6T5MlfapHYDoXkobH3X5kYRjnaOTAqaxO4Luzl2uWoZWr1nzKDgyhWS5+lCi1rdYso6CE/gL6XUeqXUsqLNDNfuIyLdMdxKmq6UqlIGOxH5QkRCRSTU29vbDGFV7Ofpg2l/Io9DEU0YPkNnPkcp6P88pJyFP6r2Tdyabm93Oy72LlVeOFevcyfcBw4kee5c8q8pcalptsyUDuLvwHDgNeDdElu1iMg5488EYCnQ85pDzmFIDFjEz7jPqpa+/RghWy9wONiR8bPWWjuc2qNtJPj1NI4ibGvCW0OXhgxvO5zlJ5aTnJ1cpTa8n3icwsxMkr7USQa0uqO8WUwuSqkZwHggCNguIpuLtupcVCnlqpRyL3oNDAIOXXPYMuBu42ymW4AUEblQnetW165fF9Dqu3XE+ULEp8uwd3C0Zji1S9EoIvUc/P6ttaOptCnBU8gtzGXR0arll3IOCMBj5Eguz59PXny8maPTNOsobwTxDRAKHMRwG6jao4YSfIFtSqkDwB5gpYisUUo9pJR6yHjMKiAWOA58CTxixutX2vkzx8h55TXyHMDnn6/j2cTfmuHUTm36Q4tbYOt7kGdb0z7bNmxL72a9+THmR/KquKbD67FHkcJCEj+tdRPuNK1KyusgOojIFBH5HLgdCDPXRUUkVkS6GreOIvIv4/7PROQz42sRkeki0lZEOovIdWslakpebi5RD42lUQok3TeMruG6HEapikYRaedtchQxOXgyCVkJrD1dtVuHTn5+eI6/nSuLFpN75oyZo9O0mldeB1H8NUpEbC9lpxktfmgA7WLzOTzQj2HTzTmQqoNa94WWvWGb7Y0i+jTvg38Df747/F2VE+81fughlIMDl2bPNnN0mlbzyusgupZcPQ10MedKalux5I1pdN1xib86OnH7u6utHU7tVzyKuAD75lk7mkqxU3ZMDp7MX0l/ceDSgSq14ejjQ6O7ppC6fAXZR4+aOUJNq1nlraS2v2b1tIMFVlLXatuWz6X1gq2cbQIDP1+JvYNtZSy1mtbh0KqPcRSRZe1oKmVk25G4O7nzffT3VW6j8X33YefqyqUPPzRjZJpW80yqB3EjOnviEIX/+g85TuD373fw8LLhWtLW0P95SI+HqDnWjqRS6jvWZ1y7caw7vY4L6VWbNGffsCGN77uX9HXryfrzTzNHqGk1R3cQpcjJzmL/9Il4pkLKQ2Po0Os2a4dke/z7gH8YbPsAcjOtHU2lTAqahCAsOFL1/Eqed92NfaNGXPrgAzNGpmk1S3cQpVj6YCQBpwo4PLQ1gx94w9rh2K7+L0BGAkR9be1IKqWZWzMiW0ay6OgiMvOq1rnZu7ni9eA0MnbsJGPXLjNHqGk1w6QOQinVSik1wPi6XtEit7ro59em0nX3Zf7q7MKEt1dYOxzb1qo3tOlnHEVkWDuaSrmrw12k5aaxIrbqvwMNJ07EoUkTLr3/gS5HqtmkCjsIpdQDwCLgc+MuP+C/lgzKWjYt+ZR2P+/mdDPF4M/XoOz0AKva+r0AmYmw17ZSUIR4h9ChcQe+j/6+SrUiAOycnfGa/ghZBw6QvnGTeQPUtBpgyl/A6cCtQCqAiBwDfCwZlDWcjvkD+7c+JMsZ/P8zC/dGvtYOqW5oeTO0jYDtsyDHdtJhK6WYEjyFkykn2XF+R5XbaTh6NE6tWnHpgw+Qwqp1NJpmLaZ0EDkiUpzDWSnlgCFVd52RlZHGocen4JEOGY9OIih0oLVDqlv6vQCZSbDXtsqxDvEfglc9L74/XPUpr8rREa/HHyPn6FFSV+l1NJptMaWD2KyUegGop5QaCPwMLLdsWDVr2YMDaHOmkJjb2jNg6j+sHU7d0+ImCBgA2z+EnDRrR2MyR3tHJgZOZPv57cReia1yOw2GDsU5MJBLH32I5Nle7W7txmVKB/EccAlD0r4HMSTRe8mSQdWkMzFRtIpO5VBIPSa8VScfrdQO/V6ArGTY84W1I6mU8YHjcbJzqnKtCABlZ4f3jCfIO32GK0uXmjE6TbOsCjsIESkUkS9FZLyI3G58XWduMbUMCqXh5x8y7Iv1hjQRmmX49YB2g2DHR5BtO5laGrk04rY2t7HsxDJSclKq3I5bv37UCwkh8eNPKMyxrXoZ2o2rvHoQB5VSf5a11WSQlhYUOhDXBp7WDqPu6/ccZF2G3Z9XfGwtMjl4MtkF2VWuFQGGh97eTz5Jfnw8lxdUfQGeptWk8kYQw4ER5WyaVjnNe0D7IbDzI8iu+rfxmhbYKJCbm9zMgpgF5BVW/RmC6809ce3dm6TPv6Ag3bbWhWg3pvKS9Z0u2oAcoCvQBcOsptM1FaBWx/R7ztA57PrM2pFUyuTgycRnxrP+zPpqteP95AwKLl8m+dtvzBSZplmOKQvl7sdQ9W0shsJBu5RS91o6MK2OatYNAofBzo8h64q1ozFZuF84LdxbVGvKK0C9zp1xHziA5Dlzyb982UzRaZplmDKL6f+AbiIyVUTuAXoAz1b1gkqpFkqpjUqpw0qpv5RST5RyTD+lVIpS4FZ9QAAAF5ZJREFUar9x03NP65J+z0FOCuyyndKc9nb2TA6ezIFLBzh46WC12vJ+/HEKMzJI/tq2clRpNx5TOogkoOTk9TTjvqrKB54SkQ7ALcB0pVSHUo7bKiIhxu21alxPq22adoWg4bDrE8NDaxsxOmA0bo5u1aoVAeDcrh0eI0eQ/P188uITzBSdpplfebOYZiqlZgLHgd1KqVeUUi8Du4Aql8oSkQsi8rvxdRoQDTSvanuajer3POSkws5PrB2JyVwdXRkdMJq1p9YSnxFfrba8Hn0Uyc8n6XPbehaj3VjKG0G4G7cTGJLzFa19+AU4aY6LK6X8gW7A7lI+7qWUOqCUWq2U6lhOG9OUUlFKqahLly6ZIyytJjTpBMEjDbeZMpOtHY3J7gy+kwIp4McjP1arHacWLWg4/nYu//QzuWfPmik6TTMvZa01b0opN2Az8C8RWXLNZw2AQhFJV0oNA2aJSLuK2gwNDZWoqCjLBKyZX/xf8GlvCHsaIv9u7WhM9sSGJ/g94Xd+u/03XBxcqtxOXnwCJwYNosGQITR7600zRqhpplNK7ROR0NI+M2UWk7dS6m2l1Cql1IairZoBOQKLgfnXdg4AIpIqIunG16sAR6WUV3WuqdVCvh2hw2jY/ZlNjSKmdJjClZwrrIxdWa12HH198JwymZRly8g5dsxM0Wma+ZjykHo+EAO0Bl4FTgF7q3pBpZQCvgaiReS9Mo5pYjwOpVRPY5zVeTCu1Vb9njMUE9rxkbUjMVmobyiBnoF8H/19tQsBNb7/fuz+v70zD5OiOtf47+2eHZRNXAAVWYzbRaMxxiUumLjEPS7gFaMx9xpycxM1EgkxMZiQxLgkuFwDURGNxmiI+4a44kIENxTUCCoiCIoLOsw+3d/945yeqenp2XvoHub8nqeeOlV16tRbVd311jlV9Z2yMtZd1XP2P9B7aI9BDDKzG4A6M3vKzM4CxnZhm/sDpwNjI6+xfkvSREkTfZ6TgCWSFgNXAeM3pfhPgQhb7gy7nuDCb1R8nGs17UISE3aZwPL1y1nwwYIulVUwYAADz/ou5fPmUfVa116fDQSyTXsMIhVbYI2koyR9GRjY2Q2a2TNmJjMbE3mN9UEzm2FmM3yea8xsVzPb3cy+Zmad77ElkP8cNBnqKuG5q3KtpN0cucORbFm2JZOemsSj7z3apbIGnnEm8QEDWDf9yiypCwSyQ3sMYpqkfsD5wCTgeuDcblUV6F1suRPsdiIsvA429Iw30Yrjxdx0xE1sv/n2nPfkefzu+d9Rm6hte8UMxPv2YdDZZ1Px7LNUPL8wy0oDgc7TnnDf95vZ52a2xMwOMbO9gJEbQVugN3HQZKivhud6zl30sM2GcfORNzNh5wnc9uZtnP7Q6bz/RedeWR1w6ngKttqKtb/+NdVvdfozo0Agq7SnBpGJn2RVRSAweEf4j5Nh4fWwoed8XVwYL2TyVydz5SFX8n75+5xy/ynMXTG3w+XESkrYZto0Ep98wrvfPpGP/jSdZHV1NygOBNpPZw0i9KwTyD4HXgCJGni259QiUozdbixzjpnDiH4jmPTUJKb9axo1iY51DNT36wcw4qEH6XfUUXwycybvHHccFQu69hA8EOgKnTWI8EZRIPtsMQrGjINFN0B510JZ5IIhfYcw+4jZnLHLGdz+79uZ8OAE3vuiY5HxCwYMYMgfLmG7G2cBsPK7Z/HB5J+FyK+BnNBaLKZySV9kGMqBIRtRY6A3ceBPIVELz07PtZJOURgvZNLek7hm7DWsqVjDKfedwoPvPNjhcvrsuy8j7rmHQRO/z+cPPMA7R36L9Xfd3eXvLgKBjtBah0GbmdnmGYbNzKxgY4oM9CIGjYTdx8MLs6B8ba7VdJqDtj2IOcfMYccBOzL56clcvOBiqus79kwhVlLClueey4i77qRohx1YM2UKK797FrUrVnSP6EAgjc42MQUC3ceBkyBRB09n/NC+x7B1n62ZdcQsztrtLOa8NYfTHjyNdz/veJzL4tGj2f7WW9h66q+oXrqUd449jo9nzMBqO/dabSDQXoJBBPKPgSNgj/+EhTPhb+Nh1Yu5VtRpCmOFnLfXeVx76LWsq1zHuPvHcd/b93W4HMViDBg/nhEP3E/fsWNZN/1K3j3xRCpfeqkbVAcCjpxFc+0OQjTXTYjaCtct6YL/g+r1MHKsez6x/X65VtZpPqz4kAvmX8BLH73ECaNOYMo+UygtKO1UWeVPPMHa3/yG+g/W0H/cOLY8/yfEN988y4oDvYHWorkGgwjkNzXl7q2mBddAxTrY/gDXBDXiYFDPe9u6PlnPta9cy/WvXc/I/iO5/KDLGdm/c9+dJisqWHfV1Xz6178SHzSQrS+8kM0OPxz1wOMSyB3BIAI9n9pKeOkm941E+RoYtrerUYw+rEcaxXMfPMeUp6dQVV/Fz/f5OcePOr7TZVUtWcraiy6i+vXX6XvQQWx90S8pHBo6aQy0j2AQgU2H+hp4+RZ4Zjp8vhK2HuOMYqejIdazHqmtq1zH5Kcns2jtIo4deSwX7nMhZYVlnSrL6uv59JZbXNhwMwb/+McMPH0CKggvHAZaJxhEYNMjUQev3gFPXwGfvg2Dd3ZNT7ueALF4rtW1m0QywcxXZzJj8QyG9xvOxftdzG6DdqMwXtip8upWr2btb6ax4cknKdllF7b65S8oHjWKWGlpMItARoJBBDZdkglYehfMvxzWvQEDR8LXz4cxp0AnL7K54Pk1zzN5/mQ+qf6EAhUwvN9wRvUfxegBoxvGQ/sOJaa2a0lmRvncR1j722kk1jX2saHCQlRWRqy0tGFQWSmx0rKm02VljfPKSlGpz1Pm86RNx0pLUVFRdx6eQDcSDCKw6ZNMwpv3w/zLYO2r0H87OOA82OM0KCjOtbp2sb56Pc998BzL1i9j+WfLWbZ+Gas3rG5YXlpQyqj+o5oZxxalmXvjTXzxBeXz5pEoL8eqqkhWVpKsrCJZVUWyqhJrSDefto4GCiwsbNV4mhhNaSmxPmVNpxvMJ814ysqcsfXA50w9hWAQgd6DGSybB/MvhVWLYLMhsP85sOd3oKhz7fu5pKKuguXrlzcYxrLP3PBZTWNspoElA5uZxqj+o+hT2KfT27VEgmRVNVZV2WgilZXOaKqqvNFEpisqG42mYbk3m6rKtOmqjomJx72JlDWYRovG06TG03KtR748FRX1evPJO4OQdARwJRAHrjezS9KWFwM3A3vh+qIeZ2Yr2io3GESgATN49yl46jJ47xnoMxj2mQiDd3JGUdQXivr4oS8UlkFhaY95I+rjqo+bGEdqXFXfePEd2ncowzcfTlG8iIJYAXHFicfixBVvnI6m05f56VS6QAUZ80TnR9dpqdwYoqAmQaymllh1LbGaOlRVC9U1xKrdmKpqrNobTWVlu2o9yaoqrLKyYwcyFmta4ykra7EW1GS6rDRiVs1rPbHSUlRc3CPMJ68MQlIceAv4JrAKWAScamavR/L8DzDGzCZKGg+cYGbj2io7GEQgI+89555RvP1YGxmVZhzpQ2RZYWpcAvFi14xVUAwFJRAvapyOLmuSLsq6GSUtyQcbPmDZZ8tYvn45yz5bxsryldQl60gkEyQsQX2ynoQlGqZT6XqrbzIvH0g3npYMqomJWZyShCipE8X1UForiuuM4joorjOKalNDkqLaJIW1SQprEm5cm6CgJkFBTT0FtfXEayJDdR2xmlrUgculxQTFxVBaDCUlqNQPZaWopLFGFCsrJV7ah1hZGfGyMgrK+hBvGBpNK9r8ppKSrJlPvhnEvsBUMzvcT08BMLPfR/LM9XkWSCoA1gKDrQ2xwSACrfL5Kqj81H2lXVsBtRtcX9ipdMP8ihbyRPJ1snvRJmQyj1hBzmsxhpGAhqE+lRYkIsvqG+al8liz+W7aSAL1kXlN1lFkGw3rN9XQtFxrpiu6/USz7UfWyTQ/TVcqX/MDYxTVQ3EdlNTiTQdK6ozi2lQ6Mr/WGqdro8ssQxkQ68ClOAnUFUKtHyo3i3HYo0vbX0CE1gwiF++9DQWi/TKuAvZpKY+Z1Uv6HBgEfJyWD0lnA2cDbLfddt2hN7Cp0G+YG7JBos6ZRX1N45CIpOurnYm0uqwa6mubLkvWZUdfFxDuwtCbX4o1c6aWwKhvMJ5U2khYynTMz49MW9PpJutbY/710fXNSNYnodYNVmdQl0S1yYZxrNZQXZJYnRGrNeJ1ST82VNg9Z6vH/wbM7C/AX8DVIHIsJ9BbiBdC6YBcqwh0E8I9II0DvfkF3lx8eroa2DYyPczPy5jHNzH1wz2sDgQCgcBGIhcGsQgYLWkHSUXAeODetDz3Amf49EnA4209fwgEAoFAdtnoTUz+mcL/AnNxNbhZZrZU0q+BF8zsXuAG4K+SlgOf4kwkEAgEAhuRnDyDMLMHgQfT5l0USVcDJ29sXYFAIBBopGeFvwwEAoHARiMYRCAQCAQyEgwiEAgEAhkJBhEIBAKBjGxS0VwlrQPey7UOzxZk+PI7z8h3jfmuD/JfY77rg6AxG3RF3/ZmNjjTgk3KIPIJSS+0FN8kX8h3jfmuD/JfY77rg6AxG3SXvtDEFAgEAoGMBIMIBAKBQEaCQXQff8m1gHaQ7xrzXR/kv8Z81wdBYzboFn3hGUQgEAgEMhJqEIFAIBDISDCIQCAQCGQkGESWkbStpCckvS5pqaRzcq0pE5Likl6WdH+utWRCUn9JcyS9KekN31Vt3iDpPH9+l0i6TVJJHmiaJekjSUsi8wZKmidpmR/ntJejFjRe5s/zq5LuktQ/n/RFlp0vySRtkQttER0ZNUr6kT+OSyVdmo1tBYPIPvXA+Wa2C/A14IeSdsmxpkycA7yRaxGtcCXwsJntBOxOHmmVNBT4MfAVM9sNF7Y+H0LSzwaOSJv3M+AxMxsNPOanc8lsmmucB+xmZmOAt4ApG1tUhNk014ekbYHDgJUbW1AGZpOmUdIhwHHA7ma2K3B5NjYUDCLLmNkaM3vJp8txF7ahuVXVFEnDgKOA63OtJROS+gEH4voFwcxqzWx9blU1owAo9T0elgEf5FgPZjYf139KlOOAm3z6JuD4jSoqjUwazewRM6v3k//C9TKZE1o4hgB/Ai4Acv5WTwsafwBcYmY1Ps9H2dhWMIhuRNJw4MvA87lV0ozpuB97MtdCWmAHYB1wo28Gu15Sn1yLSmFmq3F3aCuBNcDnZvZIblW1yFZmtsan1wJb5VJMOzgLeCjXIqJIOg5YbWaLc62lFXYEvi7peUlPSdo7G4UGg+gmJPUF/gmca2Zf5FpPCklHAx+Z2Yu51tIKBcCewJ/N7MtABblvGmnAt+MfhzOyIUAfSRNyq6ptfLe9Ob8DbglJF+KaaG/NtZYUksqAnwMXtZU3xxQAA3HN2j8F7pCkrhYaDKIbkFSIM4dbzezOXOtJY3/gWEkrgL8DYyXdkltJzVgFrDKzVM1rDs4w8oVvAO+a2TozqwPuBPbLsaaW+FDSNgB+nJWmh2wj6UzgaOC0POt/fiTuRmCx/88MA16StHVOVTVnFXCnORbiWge6/DA9GESW8a59A/CGmf0x13rSMbMpZjbMzIbjHqw+bmZ5dfdrZmuB9yV9yc86FHg9h5LSWQl8TVKZP9+HkkcP0dO4FzjDp88A7smhloxIOgLX5HmsmVXmWk8UM3vNzLY0s+H+P7MK2NP/RvOJu4FDACTtCBSRheizwSCyz/7A6bg781f88K1ci+qB/Ai4VdKrwB7A73KspwFfs5kDvAS8hvsf5TwUg6TbgAXAlyStkvQ94BLgm5KW4Wo+l+ShxmuAzYB5/v8yI8/05RUtaJwFjPCvvv4dOCMbNbEQaiMQCAQCGQk1iEAgEAhkJBhEIBAIBDISDCIQCAQCGQkGEQgEAoGMBIMIBAKBQEaCQfQifCTKKyLTkyRNzVLZsyWdlI2y2tjOyT666xPdva02dKzoalRPSVMlTfLpEh9tdWqGfHmxz+1F0oYurNtwXCU914VyzpQ0pLPrBxzBIHoXNcC3cx2uOB0f8K69fA/4bzM7pLv0bGwkFeG+vH/RzKZmyJJxnzt43HocZtaVr9PPxIVBCXSBYBC9i3rcB13npS9IrwGk7gIlHeyDf90j6R1Jl0g6TdJCSa9JGhkp5huSXpD0lo/5lOp34jJJi3y8/+9Hyn1a0r1k+Epa0qm+/CWS/uDnXQQcANwg6bK0/NtImu8/tFoi6et+/p+9pqWSLo7kXyHp9z7/C5L2lDRX0tuSJkY0zpf0gKR/S5ohqdl/RtIEfzxekTTT73PcH9Mlfj+aHXNPAXA7sMzMmsWbSt9nf2d8r6THgcfk+nu42x/bf0ka49ebKukmf4zfk/RtSZd6LQ/LhYNJ39ZIv+xFv95Ofv4xckHgXpb0qKSt/Py+km70Zb4q6cRIWb+VtNhrahYgsLV1I3k2RNI/jfyGLvbzhvua1XX+/D4iqdT/jr+C+9DyFUmlLRz7QFuYWRh6yQBsADYHVgD9gEnAVL9sNnBSNK8fHwysB7YBioHVwMV+2TnA9Mj6D+NuOkbjQhKUAGcDv/B5ioEXcLFtDsYF4dshg84huHAWg3EX0MeB4/2yJ3H9MKSvcz5woU/Hgc18emBk3pPAGD+9AviBT/8JeBX3Ne9g4MPIvlcDI/z681LHyK+/BbAzcB9Q6OdfC3wH2AuYF9HXP4Pmqbiwzbe3cd4a9hl3Z7wqsl9XA7/y6bHAK5GynwEKcf1pVAJH+mV3pY5n2nYeA0b79D64MCwAA2j8qPa/gCt8+g+p85/K58cGHOPTl6bOf9q2Wlp3BbBF2m/wMNyNjXC/r/tx4eCH42569vD57gAmtPY7CUPHhk26ihpojpl9IelmXIc3Ve1cbZH5kNGS3gZSoa1fw8d/8dxhZklgmaR3gJ1wf+4xkdpJP5yB1AILzezdDNvbG3jSzNb5bd6KuyDc3ZpGYJa/M77bzF7x80+RdDbOaLYBdsGZAbg4Ran96Guu/45ySTVq7NVsoZm943XchrubnxPZ7qE4M1gkFzyzFBcQ7z5c6IOrgQcixyydZ4D9JO1oZm+1sn9R5plZqj+AA4ATAczscUmDJG3ulz1kZnWSXsMZ3MOR/R0eLVAu+vB+wD/UGAS02I+HAbfLBfsrAlLn7BtEOkoys898shZ3EQd4Efhmhn1oad1MHOaHl/10X9xvaCUuaGLqXL+Yvl+BrhEMoncyHRdH6MbIvHp8k6NvRimKLKuJpJOR6SRNf0PpcVsMd9f3IzObG10g6WBcDSIrmNl8SQfiOkKaLemPwNO4WtLeZvaZpNm4Wk2K6H6k72NqvzLtUxQBN5lZs17QJO0OHA5MBE7B9XWQznxcRz4PSTrAGvtuaI32HrdU5zFJSXXmb61pft7Anfv1ZrZHhnKuBv5oZvf68za1je1Gt5XIsK2OIuD3ZjazyUzX30r0vCVwBh3IEuEZRC/E333egXv4mWIF7k4Y4Fhc00RHOVlSTO65xAjg38Bc4AepNm9JO6rtzn8WAgdJ2kJSHDgVeKq1FSRtj2saug7XU96euOa0CuBz3w5+ZCf26auSdvCmOQ53xx/lMeAkSVt6HQMlbS/3IkDMzP4J/IJWwpX7PJcDD6vj/TE/DZzmt30w8LF1ov8Rv867kk72ZckbHLha32qfPiOy2jzgh6kJday/646sOxc4y9dykDQ0dbxboRzXZBjoAsEgei9X0DRe/HW4i/JiYF86d3e/EndxfwiYaGbVuIv167gY+kuAmbRxR+nvon8GPAEsxr3d01aY6oNxMftfxl3IrzTXA9jLwJvA34BnO7FPi3DRRt/ANa3clab1dZwBPCIXeXYerilrKPCkpFeAW2ijn2Uz+7Mv+15JJa3lTWMqsJff9iU0vYB3lNOA7/nfwFJcp0ipbfxD0os0DSE9DRgg9yB+MU2bG9ui3eua663vb8AC31w2h7Yv/rOBGeEhddcI0VwDgRbwd+STzOzoXGsJBHJBqEEEAoFAICOhBhEIBAKBjIQaRCAQCAQyEgwiEAgEAhkJBhEIBAKBjASDCAQCgUBGgkEEAoFAICP/D8KaMuJbqsqZAAAAAElFTkSuQmCC\n"
311 | },
312 | "metadata": {
313 | "needs_background": "light"
314 | }
315 | }
316 | ]
317 | },
318 | {
319 | "cell_type": "markdown",
320 | "source": [
321 | "# Dry bean"
322 | ],
323 | "metadata": {
324 | "id": "DP0fdL00-wYq"
325 | }
326 | },
327 | {
328 | "cell_type": "code",
329 | "execution_count": null,
330 | "metadata": {
331 | "id": "xBPJ8vmAEiJQ"
332 | },
333 | "outputs": [],
334 | "source": [
335 | "K = 4\n",
336 | "N = 4\n",
337 | "B = f(N+K-1) // f(K) // f(N-1)"
338 | ]
339 | },
340 | {
341 | "cell_type": "code",
342 | "source": [
343 | "# calculate H(B|Y=y), y is uniform\n",
344 | "p_y = 0\n",
345 | "P_ys = []\n",
346 | "b_count = 0\n",
347 | "H_y = 0\n",
348 | "\n",
349 | "for c0 in range(0, N+K-1):\n",
350 | " for c1 in range(c0+1, N+K-1):\n",
351 | " for c2 in range(c1+1, N+K-1):\n",
352 | " for c3 in range(c2+1, N+K-1):\n",
353 | " b_count += 1\n",
354 | " Ns = [0, 0, 0, 0, 0, 0]\n",
355 | " Ns[c0-0] += 1\n",
356 | " Ns[c1-1] += 1\n",
357 | " Ns[c2-2] += 1\n",
358 | " Ns[c3-3] += 1\n",
359 | " p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5]))\n",
360 | " P_ys.append(p_y)\n",
361 | " H_y += -p_y * math.log2(p_y)\n",
362 | "\n",
363 | "# print(sum(P_ys))\n",
364 | "# print(b_count)\n",
365 | "print('B', B)\n",
366 | "print(math.log2(B))\n",
367 | "print('H_y', H_y)\n",
368 | "print(2**H_y)"
369 | ],
370 | "metadata": {
371 | "colab": {
372 | "base_uri": "https://localhost:8080/"
373 | },
374 | "outputId": "7c0c2d06-1666-4b67-8c66-130dfece1472",
375 | "id": "ObFHHIEsEiJa"
376 | },
377 | "execution_count": null,
378 | "outputs": [
379 | {
380 | "output_type": "stream",
381 | "name": "stdout",
382 | "text": [
383 | "B 35\n",
384 | "5.129283016944966\n",
385 | "H_y 4.81510800723783\n",
386 | "28.150877863635262\n"
387 | ]
388 | }
389 | ]
390 | },
391 | {
392 | "cell_type": "code",
393 | "source": [
394 | "# calculate H(B|S=s)\n",
395 | "U_range = np.arange(1, 1176, 110)\n",
396 | "# U_range = np.arange(1, B, 3000)\n",
397 | "H_ss = np.zeros_like(U_range).astype(float)\n",
398 | "\n",
399 | "for i, U in enumerate(U_range):\n",
400 | " H_s = 0\n",
401 | " p_s = 0\n",
402 | " choices = np.random.choice(B, U, p=P_ys)\n",
403 | " # print(len(choices), choices)\n",
404 | " for u in range(U):\n",
405 | " p_s = sum([1 for c in choices if c == u]) / U\n",
406 | " # print(p_s)\n",
407 | " if p_s != 0:\n",
408 | " H_s += -p_s * math.log2(p_s)\n",
409 | " print(H_s)\n",
410 | " H_ss[i] = H_s\n",
411 | "\n",
412 | "print('H_ss', H_ss)"
413 | ],
414 | "metadata": {
415 | "colab": {
416 | "base_uri": "https://localhost:8080/"
417 | },
418 | "outputId": "d4fe9fbc-70dd-4ccb-9947-13b5786c4743",
419 | "id": "z8SJ6CCVEiJb"
420 | },
421 | "execution_count": null,
422 | "outputs": [
423 | {
424 | "output_type": "stream",
425 | "name": "stdout",
426 | "text": [
427 | "0\n",
428 | "4.488025774313082\n",
429 | "4.702617634802406\n",
430 | "4.802929804553897\n",
431 | "4.750954940629122\n",
432 | "4.744101917925756\n",
433 | "4.8117979838104485\n",
434 | "4.768437327163887\n",
435 | "4.815896365695055\n",
436 | "4.789852490143935\n",
437 | "4.782352207555449\n",
438 | "H_ss [0. 4.48802577 4.70261763 4.8029298 4.75095494 4.74410192\n",
439 | " 4.81179798 4.76843733 4.81589637 4.78985249 4.78235221]\n"
440 | ]
441 | }
442 | ]
443 | },
444 | {
445 | "cell_type": "code",
446 | "source": [
447 | "H_ss = np.array([0., 4.48802577, 4.70261763, 4.8029298, 4.75095494, 4.74410192, 4.81179798, 4.76843733, 4.81589637, 4.78985249, 4.78235221])\n",
448 | "H_y = 4.81510800723783\n",
449 | "plt.plot(np.arange(11)*1176/11, H_y-H_ss)\n",
450 | "plt.xlabel('Number of clients')\n",
451 | "plt.ylabel('Privacy leakage (in bits)')\n",
452 | "plt.show()"
453 | ],
454 | "metadata": {
455 | "colab": {
456 | "base_uri": "https://localhost:8080/",
457 | "height": 281
458 | },
459 | "outputId": "c9ab6df1-93a5-481a-a404-dfc8767a4395",
460 | "id": "7YxzWs4OEiJb"
461 | },
462 | "execution_count": null,
463 | "outputs": [
464 | {
465 | "output_type": "display_data",
466 | "data": {
467 | "text/plain": [
468 | ""
469 | ],
470 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEICAYAAABVv+9nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xcdX3/8ddnLnudSTab3RkgJATYWSiiqI2K2lbwfq+1+vvJrz/1pyj6q7XS1p/VX2utbf+w9UpbW6Xgtdb7Da2K/ARErQWSCAISkhBCIAV2E5LsLXub/fz+OGcmk2WzO7vZM5cz7+fjMY89t5nzOZzwme+c8znfr7k7IiISP4l6ByAiItFQghcRiSkleBGRmFKCFxGJKSV4EZGYUoIXEYmpVJQfbmZ7gVGgCMy6+5Yo9yciIsdEmuBDF7v7gWo27Ovr882bN0ccjohIfGzbtu2Au/cvtK4WCb5qmzdvZuvWrfUOQ0SkaZjZ/SdaF/U1eAd+aGbbzOyyiPclIiIVom7B/4a77zezHHCdme1w95sqNwgT/2UAmzZtijgcEZHWEWkL3t33h3+HgG8CT11gmyvdfYu7b+nvX/AykoiIrEBkCd7Mus0sW5oGng/cGdX+RETkeFFeoskD3zSz0n7+zd1/EOH+RESkQmQJ3t33ABdE9fkiIrI4PckqIhJTTZ/gZ4pz/NONu7lp53C9QxERaShNn+BTCeOTP97DD+56uN6hiIg0lKZP8GbGYD7D7kfG6h2KiEhDafoEDzCQy7JzaBSNLysickwsEnwhl+HwxAwHxqbrHYqISMOIRYIfzGcB2DU0WudIREQaRywSfCGfAWD3kK7Di4iUxCLB57LtZDtS7HxELXgRkZJYJHgzo5DLsEuVNCIiZbFI8BBch9clGhGRY2KT4AdyGQ6OT3NwbKreoYiINITYJPhCuZJGrXgREYhRgh8MK2mU4EVEArFJ8Kes6SDTnmK3KmlERIAYJXgzYyCXYacqaUREgBgleAgu0+gSjYhIIFYJvpDLcmBsikPj6pNGRCRWCX5AN1pFRMpileALuVKC141WEZFYJfgNPZ10tyXVZYGICDFL8KVKGrXgRURiluAhGN1JLXgRkRgm+MF8hqHRKY5MzNQ7FBGRuopdgi8P/jGsyzQi0tril+BzQadjeqJVRFpd7BL8hp5OOtOqpBERiV2CTyRUSSMiAjFM8ICG7xMRIaYJfiCf4eGRSUYmVUkjIq0rlgl+MLzRqjFaRaSVxTLBl0sldZlGRFpY5AnezJJm9gsz+27U+yo5fV0X7akEOzW6k4i0sFq04N8B3F2D/ZQly5U0asGLSOuKNMGb2enAS4CrotzPQgq5jK7Bi0hLi7oF/zHgXcBcxPt5jEI+y/7DRxmbmq31rkVEGkJkCd7MXgoMufu2Jba7zMy2mtnW4eHhVdv/QDj4h1rxItKqomzBPxN4uZntBb4EPNvM/nX+Ru5+pbtvcfct/f39q7bzwXxQKrlLN1pFpEVFluDd/T3ufrq7bwZeA1zv7v8zqv3Nt3FdJ22phG60ikjLimUdPEAqmeCsvm614EWkZdUkwbv7je7+0lrsq9JgPqsWvIi0rNi24CEolXzw0FEmplVJIyKtJ94JPq9KGhFpXamlNjCzHEFFzGnAUeBOYKu717y2fbkK5UqaMZ5wek+doxERqa0TJngzuxh4N9AL/AIYAjqAVwBnm9nXgA+7+0gtAl2JM3q7SCdN1+FFpCUt1oJ/MfBmd983f4WZpYCXAs8Dvh5RbCctqKTJqJJGRFrSCRO8u/+fRdbNAt+KJKJVNpDPcMeDR+odhohIzS15k9XM3mFmayxwtZltN7Pn1yK41TCYy/LAoQmOThfrHYqISE1VU0XzxvA6+/OBdcBrgQ9EGtUqKuQzuMO9w7oOLyKtpZoEb+HfFwOfd/e7KpY1vELY6diuIV2HF5HWUk2C32ZmPyRI8NeaWZY6dP+7Upv7ukkljF0avk9EWsySdfDApcATgT3uPmFm64E3RBvW6kknE5zZ161SSRFpOdW04K9z9+3ufhjA3Q8CH402rNVVyKtUUkRazwkTvJl1mFkv0Gdm68ysN3xtBjbUKsDVMJDLsu/RCSZnVEkjIq1jsUs0bwEuJ+iiYHvF8hHgH6MMarUN5jPMOewZHue809bUOxwRkZpY7EGnK4ArzOzt7v4PNYxp1RVyYZ80Q6NK8CLSMhbri+bZ7n49sN/MXjl/vbt/I9LIVtHmvi6SqqQRkRaz2CWaZwHXAy9bYJ0DTZPg21NJNq/vUi28iLSUxS7RvC/82zQlkYsp5LLsVIIXkRZSTV80683s78M+aLaZ2RVhLXxTKeQz3H9wgqlZVdKISGuopg7+S8Aw8LvAq8LpL0cZVBQK+SzFOee+A+P1DkVEpCaqSfCnuvtfu/t94etvgHzUga22cp80utEqIi2imgT/QzN7jZklwtd/A66NOrDVdmZfNwlDT7SKSMtYrExylKBaxggeePp8uCoJjAHvjDy6VdSRTnLGevVJIyKtY7EqmmwtA6mFQi6jBC8iLaOaSzSxUchn2HtgnOnZpuntWERkxVorweeyzM45ew+qkkZE4q+1EnxelTQi0jqqGfADM0sSlEaWt3f3fVEFFZWz+zOYlYbvO7Xe4YiIRGrJBG9mbwfeBzzCsaH6HHhChHFFoiOdZFNvl1rwItISqmnBvwM4JxzJqekVcll1OiYiLaGaa/APAEeiDqRWCvkM9x0YZ6aoShoRibdqWvB7gBvN7N+BqdJCd/9IZFFFqJDLMFN07j84zkAudqX+IiJl1bTg9wHXAW1AtuK1qHBM11vM7HYzu8vM3n9yoa6O8uhOug4vIjG3ZAve3VeamKeAZ7v7mJmlgZ+a2ffd/T9X+HmrYiBXqqQZ40X1DEREJGKL9UXzMXe/3My+Q1A1cxx3f/liH+zuTtBnDUA6fD3mc2qtsy3J6es61WWBiMTeYi34UudiH1rph4f189uAAeDj7n7zSj9rNRVyWfUqKSKxt1hnY9vCvz9e6Ye7exF4opn1AN80s/Pd/c7KbczsMuAygE2bNq10V8tSyGf46a4DzBbnSCVb6mFeEWkhJ8xuZvYdM3tZeP18/rqzzOyvzOyN1ezE3Q8DNwAvXGDdle6+xd239Pf3Lyf2FSvkskwX59j36ERN9iciUg+LNV/fDPwmsMPMbjWz75nZ9Wa2B/gksM3dP3WiN5tZf9hyx8w6gecBO1Yx9hUrje60U5U0IhJji12ieRh4F/AuM9tM0HnLUWCnu1fT9D0V+Gx4HT4BfMXdv3vSEa+CgTDB7x4aBU6pbzAiIhGpqrMxd98L7F3OB7v7L4EnLT+k6HW3p9jQo0oaEYm3lr3DWMhndIlGRGKtdRN8LsO9w2MU5+pemi8iEomqEryZdZrZOVEHU0uFfJbp2TkeUCWNiMTUkgnezF4G3Ab8IJx/opldE3VgUStV0ug6vIjEVTUt+L8EngocBnD324AzI4ypJgbKpZJ6olVE4qmaBD/j7vP7g2/6C9fZjjSnre1gt1rwIhJT1ZRJ3mVm/wNImlkB+EPgP6INqzYG8hrdSUTiq5oW/NuBxxF0//tFYAS4PMqgaqWQy7B7aIw5VdKISAxV0x/8BPBn4StWCrkMkzNzPHjoKJvWd9U7HBGRVbVkgj9Bf/BHgK3AJ919MorAaqGQD0d3GhpVgheR2KnmEs0egoE7/iV8jQCjwGA437QGVCopIjFWzU3WZ7j7Uyrmv2Nmt7r7U8zsrqgCq4W1nWnya9pVKikisVRNCz5jZuWROMLpTDg7HUlUNTSYz6pUUkRiqZoW/J8QDJh9L2AEDzn9vpl1A5+NMrhaGMhl+PKtDzA35yQSVu9wRERWTTVVNN8L69/PDRfdU3Fj9WORRVYjhVyWieki+w8fZWOvbrSKSHxU1R88UADOATqAC8wMd/9cdGHVzmC+NPjHmBK8iMRKNZ2NvQ/4h/B1MfB3wMsjjqtmjlXS6EariMRLNTdZXwU8B3jY3d8AXACsjTSqGurpaqM/267BP0QkdqpJ8EfdfQ6YNbM1wBCwMdqwaquQy6gWXkRip5oEv9XMeggeatoGbAd+HmlUNTaYz7L7kVHc1SeNiMRHNTdZ3+ZB5vuEmf0AWAPcE21YtTWQyzA+XeShI5Oc1tNZ73BERFZFNS34q0sT7r6XoOuC70UVUD0UNPiHiMRQNQl+v5n9E4CZrQN+CPxrpFHV2GDY6ZieaBWROFkywbv7e4ExM/sEQXL/sLt/OvLIamhddxt9mTZ2qZJGRGLkhNfgzeyVFbM3A+8FbgHczF7p7t+IOrhaGshl2KlaeBGJkcVusr5s3vwvgHS43IFYJfhCLsu3frEfd8dMfdKISPM7YYIPH2pqGYP5DKNTszwyMsUpazvqHY6IyEmrZkSnDuBSgnFZy5nP3d8YYVw1N5ALbrTufGRUCV5EYqGaKprPA6cALwB+DJxOMKJTrBTyGt1JROKlmgQ/EFbSjLv7Z4GXAE+LNqza68u009vdxm7daBWRmKgmwc+Efw+b2fkEHY3logupfgZyGZVKikhsVJPgrwwfcHovcA3wK4Iug2OnkMuwU33SiEhMVDOi01Xh5I+Bs6r9YDPbCHwOyBOUVV7p7lesJMhaGcxnGZmcZXh0itwa3WgVkeZWzYAfeTO72sy+H86fZ2aXVvHZs8CfuPt5wIXA28zsvJMLN1qFnG60ikh8VHOJ5jPAtcBp4fxO4PKl3uTuD7n79nB6FLgb2LCyMGtjIK9Ox0QkPqpJ8H3u/hVgDsDdZ4HicnZiZpuBJxF0edCw+jPtrO1MqwUvIrFQTYIfN7P1BNfRMbMLgSPV7sDMMsDXgcvdfWSB9ZeZ2VYz2zo8PFztx0bCzBjMZ9itShoRiYFqEvwfE1TPnG1mPyO4cfr2aj7czNIEyf0LJ+qczN2vdPct7r6lv7+/yrCjM5DLsnNIlTQi0vyqqaLZbmbPAs4BDLjH3WeWeBsW9Nh1NXC3u3/kpCOtkUIuwxcnZjgwNk1/tr3e4YiIrFi13QVXGjQzqugu+JnAa4E7zOy2cNn/dfeGHg2qNPjHrqFRJXgRaWrL6S640pLdBbv7Twla/E2l1CfN7qExnnF2X52jERFZOXUXPE8u2062I6VSSRFpetXcZG0pQSVNVn3SiEjTU4JfQCGX0QDcItL0lOAXMJDLcHB8moNjU/UORURkxarpi2abmb0t7FGyJRTKlTRqxYtI86qmBf/fCfqhudXMvmRmL7CYj0o9qNGdRCQGlkzw7r7b3f8MGAT+DfgUcL+Zvd/MeqMOsB5OWdNBpj3FblXSiEgTq+oavJk9Afgw8EGCrgdeDYwA10cXWv2YGQO5DDtVSSMiTWzJrgrMbBtwmKDbgXe7e+nO481m9swog6unwXyG63fUt/MzEZGTUU0L/tXu/hx3/7eK5A6Au5+oO4OmV8hlOTA2xaHx6XqHIiKyItUk+DeZWU9pxszWmdnfRBhTQxjQjVYRaXLVJPgXufvh0oy7HwJeHF1IjeHY8H260SoizamaBJ80s3K3imbWCcS+m8UNPZ10tyXVZYGINK0lb7ICXwB+ZGafDuffAHw2upAaQ6mSRl0WiEizqmbAj781s18CzwkX/bW7XxttWI1hIJflJ7tUSSMizamaFjzu/n3g+xHH0nAG8xm+vv1BjkzMsLYrXe9wRESWpZq+aC40s1vNbMzMps2saGaPGTw7jsqDfwzrRquINJ9qbrL+I3AJsAvoBN4EfDzKoBpFIRd0OqYnWkWkGVXVVYG77waS7l50908DL4w2rMawoaeTzrQqaUSkOVVzDX7CzNqA28zs74CHaJF+5BOJoJJGtfAi0oyqSdSvDbf7A2Ac2Aj8bpRBNZJCLqMWvIg0pWoS/K8D7u4j7v5+d//j8JJNSxjIZ3h4ZJKRyZl6hyIisizVJPiXATvN7PNm9lIzq6q0Mi4GwxuteuBJRJpNNQN+vAEYAL5KUE1zr5ldFXVgjaJcKqnLNCLSZKp90GnGzL4POEGp5CsIyiVj7/R1XbSnEuzU6E4i0mSqedDpRWb2GYI6+N8FrgJOiTiuhpEsV9KoBS8izaWaFvzrgC8Db5k/4EerKOQy3Lr3UL3DEBFZlmquwV/i7t9q1eQOUMhn2X/4KGNTs/UORUSkaidM8Gb20/DvqJmNVLxGW6UvmpLS4B+qpBGRZnLCBO/uvxH+zbr7mopX1t3X1C7E+ivkg1LJXbrRKiJNZNFLNGaWNLMdtQqmUW1c10lbKqEbrSLSVBZN8O5eBO4xs001iqchpZIJzurrVgteRJpKNVU064C7zOwWgr5oAHD3ly/2JjP7FPBSYMjdzz+pKBvAYD7L9n2qpBGR5lFNgn/vCj/7MwR9yX9uhe9vKIVchmtu/y8mpmfpamup3hpEpEmdMFOZWQfwVoJuCu4Arnb3qusE3f0mM9t8sgE2inKXBUNjPOH0njpHIyKytMWuwX8W2EKQ3F8EfLgmETWoY5U0utEqIs1hsWsN57n74wHM7GrgligCMLPLgMsANm1q3Hu5Z/R2kU6aKmlEpGks1oIvd4C+nEszy+XuV7r7Fnff0t/fH9VuTlpQSZNRJY2INI3FWvAXVDyxakBnOG8EA4C01MNOEAz+cceDR+odhohIVRZ7kjU57+nV1HKeZDWzLwI/B84xswfN7NLVDLweBnNZHjg0wdHpYr1DERFZUmT1fu5+SVSfXS+FfAZ3uHd4jPM3rK13OCIii6pmyD4JlTod2zWk6/Ai0viU4Jdhc183qYSpVFJEmoIS/DKkkwnO7OtWqaSINAUl+GUq5FUqKSLNQQl+mQq5LPsenWByRpU0ItLYlOCXqZDPMOewZ3h86Y1FROpICX6ZCrmwTxpV0ohIg1OCX6bNfV0kVUkjIk1ACX6Z2lNJNq/vUgteRBqeEvwKFHJZlUqKSMNTgl+BQj7D/QcnmJpVJY2INC4l+BUo5LMU55z7DqiSRkQalxL8CpT7pNGNVhFpYErwK3BmXzcJQ0+0ikhDU4JfgY50ks3r1SeNiDQ2JfgVGshllOBFpKEpwa9QIZ9h74Fxpmfn6h2KiMiClOBXqJDLMjvn7D2oShoRaUxK8CtUyKuSRkQamxL8Cp3dn8FMnY6JSONSgl+hjnSSTb1dasGLSMNSgj8JhVyWux8eYaaoG60i0niU4E/C405bw57hcZ7819fxti9s56tbH2BodLLeYYmIAJCqdwDN7G0XD/Brp2a58Z5hbrhniH+/4yEAHr9hLRef089F5+a44PQekgmrc6Qi0orM3esdQ9mWLVt869at9Q5jRdydXz00EiT7HUNs33eIOYd1XWmeNdjPxefm+K1CP+u62+odqojEiJltc/ctC65Tgo/G4Ylpbtp1gBt3DHHjzmEeHZ8mYfDEjT08+9wcF52T43GnrcFMrXsRWTkl+Dorzjl37D/CDTuGuPGeIW5/8AgAuWw7F53Tz8Xn5HhmoY81Hek6RyoizUYJvsEMj05x087guv1NO4cZmZwllTC2bF7HxefkuPjcHIVcRq17EVmSEnwDmy3OsX3fYW64Z4gbdgyx4+HgwakNPZ3l1v0zBtbT1ab74SLyWErwTeShI0fLN2p/uvsAE9NF2lIJLjxrPReHCX9zX3e9wxSRBqEE36SmZots3XuIG3YMcf09Q+wZDjo2O7Ovm2cN9nPeaWvY1NvFGeu7yGc7SKgcU6TlKMHHxP0Hx8s19z+/9yBTFV0Vt6USbFzXyRnru9nU21VO/Jt6u9jY20VHOlnHyEUkKnVL8Gb2QuAKIAlc5e4fWGx7JfjqzRTn+K/DR7n/4AT7Hg1e9x8cZ9+jR9l3cJzx6eJx25+ypiNI/OuPT/6bervo7W5rmBu6s8U5jhyd4fDRGQ5PzHDk6HT4N3gV55w5d4pzwbMHpem5cDp4wdzcwtNFd9w9/ByOmy6//7jPC6qgutqS9GXa6c+205dppy/TVp7OZdvp7W4jldSD4VJ7iyX4yO7cmVkS+DjwPOBB4FYzu8bdfxXVPltJOpngjPXdnLH+sdfj3Z2D49NB4j84UfElMM5Pdg3zyMjUcdtn2lNs7O3ijDDxbwz/ntHbzak9HaRXkLgmZ4pBop6Y4fDEdDlpH5mY4XCYtOfPH5mYYXRqdtHPTRgkzIJXomLaIJGw4+aT4bzNny6/3459XsV0MtwunUyU3zMxXeT2Bw8zPDrFxLwvTwAzWNfVRn+mnb5sW/BlkGmnL3vsb+lLobertl8GxTlnbHKWkckZRidnGS39nSrNB+vGwunS+lTSWNuZZk1HmjWd6XA6xdqu+cvSrOlM0ZlONkxDIUrzGwWlRkepYVB0DxsVVEw/tiFRnDv2nmTCOH/D2lWPNcrSjKcCu919D4CZfQn4bUAJPmJmFrYy23nypnWPWX90usgDh8Lk/+gE+w6Os+/RCXYOjXL9jiGmKzpPSyaMDT2d5db/Gb1dZDpSQYt6IkzgFa3s0vzkzIk7YEsmjJ7ONGu70vR0psllOxjMZcP5Nnq60vR0BQmkpzNNT1cbPZ3BfCN0+zAxPcuB0WmGxyYZHp1meGyKA6NTHBibYjj8+4t9wZfB0ZmFvwx6u9oW/DVwbFkwvbYzzcT0sSQ8Oi8Jl/6OLLCsND3/19xC2pIJsh2p8JUm056iOOfsPTDByGRwbhf6YquUThprOoKkny19GXQ+9otgoS+NNZ3pEzYk3J2ZojM5W2RypsjUzByTM0UmZ+bKyybLy4pMzc4d9/e49RXLphZ4f3GBZFycczxM1sW5aK549GXa2frnz131z40ywW8AHqiYfxB4WoT7kyp1tiUZzGcZzGcfs6445zwyMsn9Byd44NEJ7n90vDz9vTse4vDETHnbtlSCdWFSXtuZZmNvF4/fkA4TdLCspyJpl+Yz7ammbul1taXYtD7FpvVdS247PjVbTvqlL4Dhsenjlu29f5wDY1OLfikupi2VYE2YmEtJOpftKCfr0rI15en0cck825Gq6h7NTHGO0clZjhydYSS8ZDYyOcPI0dnydGndSLjd/kNHy8tniosnx662JGs60nS2JZmal4xXmlfNoCOVpCOdoCOdpCOdpD1Vmk7Q291WXt+eSpJKWvnXXsKMZOL4X3jJ8nSwXekXYTBtJCt+SSYrfyGW5is+x8JlyQS0p6K5R1b34mozuwy4DGDTpk11jkaSCeO0nk5O6+nk6Wevf8z6oCU3y7quNt24rUJ3e4ru9tSSpa3uztjULAfGpo/7JXBkYobu9mPJuDKRZ8IkHVVymC+dDBJi7wr6U3J3JmfmKr4Ujn1BHJkIvhBKy47OFMsJOEi+xxJ0ezpJRypR/ltK2uUEXkrW4bK2ZKKpGxMnK8oEvx/YWDF/erjsOO5+JXAlBDdZI4xHVsHa8Ge1rC4zCxN3mjNj+JyDmdHZlqSzLckpazvqHU7LiPJOz61AwczONLM24DXANRHuT0REKkTWgnf3WTP7A+BagjLJT7n7XVHtT0REjhfpNXh3/x7wvSj3ISIiC9OTGSIiMaUELyISU0rwIiIxpQQvIhJTSvAiIjHVUN0Fm9kwcP8K394HHFjFcBqRjjEedIzNr5GO7wx3719oRUMl+JNhZltP1GVmXOgY40HH2Pya5fh0iUZEJKaU4EVEYipOCf7KegdQAzrGeNAxNr+mOL7YXIMXEZHjxakFLyIiFZo+wZvZC83sHjPbbWbvrnc8K2VmG83sBjP7lZndZWbvCJf3mtl1ZrYr/LsuXG5m9vfhcf/SzJ5c3yOonpklzewXZvbdcP5MM7s5PJYvh91LY2bt4fzucP3mesZdLTPrMbOvmdkOM7vbzJ4et/NoZn8U/ju908y+aGYdzX4ezexTZjZkZndWLFv2eTOz14fb7zKz19fjWEqaOsFXDOz9IuA84BIzO6++Ua3YLPAn7n4ecCHwtvBY3g38yN0LwI/CeQiOuRC+LgP+ufYhr9g7gLsr5v8W+Ki7DwCHgEvD5ZcCh8LlHw23awZXAD9w93OBCwiONTbn0cw2AH8IbHH38wm6A38NzX8ePwO8cN6yZZ03M+sF3kcwPOlTgfeVvhTqwt2b9gU8Hbi2Yv49wHvqHdcqHdu3gecB9wCnhstOBe4Jpz8JXFKxfXm7Rn4RjOz1I+DZwHcBI3hgJDX/nBKMJfD0cDoVbmf1PoYljm8tcN/8OON0Hjk23nJveF6+C7wgDucR2AzcudLzBlwCfLJi+XHb1frV1C14Fh7Ye0OdYlk14U/YJwE3A3l3fyhc9TCQD6eb9dg/BrwLKI0wvR447O6z4XzlcZSPMVx/JNy+kZ0JDAOfDi9DXWVm3cToPLr7fuBDwD7gIYLzso14nceS5Z63hjqfzZ7gY8fMMsDXgcvdfaRynQdNgqYtezKzlwJD7r6t3rFEKAU8Gfhnd38SMM6xn/VALM7jOuC3Cb7MTgO6eeyljdhpxvPW7Am+qoG9m4WZpQmS+xfc/Rvh4kfM7NRw/anAULi8GY/9mcDLzWwv8CWCyzRXAD1mVhpdrPI4yscYrl8LHKxlwCvwIPCgu98czn+NIOHH6Tw+F7jP3YfdfQb4BsG5jdN5LFnueWuo89nsCT42A3ubmQFXA3e7+0cqVl0DlO7Ev57g2nxp+evCu/kXAkcqfko2JHd/j7uf7u6bCc7V9e7+e8ANwKvCzeYfY+nYXxVu39AtKHd/GHjAzM4JFz0H+BUxOo8El2YuNLOu8N9t6Rhjcx4rLPe8XQs838zWhb90nh8uq49639RYhZsiLwZ2AvcCf1bveE7iOH6D4OffL4HbwteLCa5V/gjYBfw/oDfc3ggqiO4F7iCoaKj7cSzjeC8CvhtOnwXcAuwGvgq0h8s7wvnd4fqz6h13lcf2RGBreC6/BayL23kE3g/sAO4EPg+0N/t5BL5IcE9hhuCX2KUrOW/AG8Nj3Q28oZ7HpCdZRURiqtkv0YiIyAkowYuIxJQSvIhITCnBi4jElBK8iEhMKcFL3ZiZm9mHK+bfaWZ/uUqf/Rkze9XSW570fl4d9hh5w3LjCrsxWFHneGZ2kZk9YyXvldahBC/1NAW80sz66h1IpYqnMatxKfBmd794uftx9ze5+6+W+77QRYASvAjRhwoAAAM0SURBVCxKCV7qaZZg6LM/mr9ifgvczMbCvxeZ2Y/N7NtmtsfMPmBmv2dmt5jZHWZ2dsXHPNfMtprZzrAfnFJf9B80s1vDfrzfUvG5PzGzawieypwfzyXh599pZn8bLvsLggfUrjazDy7wnj8N33O7mX1ggfU3mtmWcPr5ZvZzM9tuZl8N+yTCzPaa2fvD5XeY2blhZ3RvBf7IzG4zs98Mf0ncGe7rpur+80vcLaelIhKFjwO/NLO/W8Z7LgB+DXgU2ANc5e5PtWCQlLcDl4fbbSbok/ts4AYzGwBeR/BY+VPMrB34mZn9MNz+ycD57n5f5c7M7DSCPsx/naCf8x+a2Svc/a/M7NnAO91967z3vIigQ66nuftE2E/4gsJfMH8OPNfdx83sT4E/Bv4q3OSAuz/ZzH4/3NebzOwTwJi7fyj8jDuAF7j7fjPrWcZ/S4kxteClrjzoMfNzBANIVOtWd3/I3acIHhUvJeg7CJJ6yVfcfc7ddxF8EZxL0DfI68zsNoLumNcTDNoAcMv85B56CnCjB51rzQJfAH5riRifC3za3SfC43x0kW0vJBiw5mdhXK8HzqhYX+p4btu846v0M+AzZvZmggE4RNSCl4bwMWA78OmKZbOEDRAzSwBtFeumKqbnKubnOP7f9Px+OJygD5G3u/txHUCZ2UUEXfvWgwHXufslJ1hfOr4iJ/h/1t3famZPA14CbDOzX3f3ZumxUSKiFrzUXdi6/QrHhngD2EtwSQTg5UB6BR/9ajNLhNflzyIYdeda4H9b0DUzZjZowYAci7kFeJaZ9VkwTOQlwI+XeM91wBvMrCvczwkv0QD/CTwzvISEmXWb2eASnz8KZEszZna2u9/s7n9BMODIxhO+U1qGErw0ig8DldU0/0KQVG8nGP5tJa3rfQTJ+fvAW919EriK4CbqdgsGV/4kS/yS9aAb2HcTdId7O7DN3b+9xHt+QNCl7Nbwsss7F9l2GPhfwBfN7JfAzwkuJy3mO8DvlG6yAh8s3QQG/iOMU1qcepMUEYkpteBFRGJKCV5EJKaU4EVEYkoJXkQkppTgRURiSgleRCSmlOBFRGJKCV5EJKb+P9NzPXLQKa35AAAAAElFTkSuQmCC\n"
471 | },
472 | "metadata": {
473 | "needs_background": "light"
474 | }
475 | }
476 | ]
477 | }
478 | ]
479 | }
--------------------------------------------------------------------------------
/diagrams/FbFTL_diagram.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/wfwf10/Feature-based-Federated-Transfer-Learning/6e332076b157d3d7dfcfc46b3170c966747e5510/diagrams/FbFTL_diagram.png
--------------------------------------------------------------------------------
/history.txt:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/main_GLOBECOM_CIFAR10_VGG16.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import time
3 | import numpy as np
4 | import torch
5 | import torch.nn as nn
6 | import torch.nn.functional as F
7 | from torchvision import datasets
8 | from torchvision import transforms
9 | from torch.utils.data import DataLoader
10 | import datetime
11 |
12 | #######################################
13 | ### PRE-TRAINED MODELS AVAILABLE HERE
14 | ## https://pytorch.org/docs/stable/torchvision/models.html
15 | from torchvision import models
16 | #######################################
17 |
18 | now = datetime.datetime.now()
19 |
20 | # Device
21 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
22 | print('Device:', DEVICE)
23 |
24 | ##########################
25 | ### SETTINGS
26 | ##########################
27 |
28 | FL_type = 'FbFTL'
29 | train_set_denominator = 'full' # 'full', int <= 50000 # pick a subset with 50000/int training samples
30 |
31 | # Hyperparameters
32 | NUM_CLASSES = 10
33 | random_seed = 1
34 | learning_rate = 1e-2
35 | num_epochs = 200
36 | batch_size = 64
37 | momentum = 0.9
38 | lr_decay = 5e-4
39 |
40 | write_hist = True
41 |
42 | def adjust_learning_rate(optimizer, epoch):
43 | """Sets the learning rate to the initial LR decayed by 2 every 30 epochs"""
44 | lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs))
45 | for param_group in optimizer.param_groups:
46 | param_group['lr'] = lr
47 |
48 | if torch.cuda.is_available():
49 | torch.backends.cudnn.deterministic = True
50 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU.
51 | torch.manual_seed(random_seed) # sets the seed for generating random numbers.
52 |
53 | if write_hist:
54 | file1 = open('history.txt', 'a')
55 | file1.write('\n \n \n Time:')
56 | file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' '
57 | + str(now.hour) + ' ' + str(now.minute) + ' ' + str(now.second)
58 | + ' ' + FL_type + ', train_deno:' + str(train_set_denominator))
59 | file1.close()
60 |
61 | ##########################
62 | ### CIFAR10 DATASET
63 | ##########################
64 |
65 | custom_transform = transforms.Compose([
66 | transforms.Resize((224, 224)),
67 | transforms.ToTensor(),
68 | transforms.Normalize(mean=[0.485, 0.456, 0.406],
69 | std=[0.229, 0.224, 0.225])
70 | ])
71 |
72 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True)
73 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform)
74 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False)
75 |
76 | if train_set_denominator == 'full':
77 | train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True)
78 | else:
79 | selected_list = list(range(0, len(train_dataset), train_set_denominator))
80 | trainset_1 = torch.utils.data.Subset(train_dataset, selected_list)
81 | train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True)
82 |
83 | ##########################
84 | ### LOAD MODEL
85 | ##########################
86 |
87 | model = models.vgg16(pretrained=True)
88 | for param in model.parameters():
89 | param.requires_grad = False
90 |
91 | model.classifier[3].requires_grad = True
92 | model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
93 |
94 | ##########################
95 | ### TRAIN MODEL
96 | ##########################
97 |
98 | model = model.to(DEVICE)
99 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum)
100 |
101 | def compute_accuracy(model, data_loader):
102 | model.eval()
103 | correct_pred, num_examples = 0, 0
104 | with torch.no_grad():
105 | for i, (features, targets) in enumerate(data_loader):
106 | features = features.to(DEVICE)
107 | targets = targets.to(DEVICE)
108 |
109 | logits = model(features)
110 | _, predicted_labels = torch.max(logits, 1)
111 | num_examples += targets.size(0)
112 | correct_pred += (predicted_labels == targets).sum()
113 | return correct_pred.float()/num_examples * 100
114 |
115 |
116 | def compute_epoch_loss(model, data_loader):
117 | model.eval()
118 | curr_loss, num_examples = 0., 0
119 | with torch.no_grad():
120 | for batch_idx, (features, targets) in enumerate(data_loader):
121 | features = features.to(DEVICE)
122 | targets = targets.to(DEVICE)
123 | logits = model(features)
124 | loss = F.cross_entropy(logits, targets, reduction='sum')
125 | num_examples += targets.size(0)
126 | curr_loss += loss
127 |
128 | curr_loss = curr_loss / num_examples
129 | return curr_loss
130 |
131 |
132 |
133 | start_time = time.time()
134 | for epoch in range(num_epochs):
135 | model.train()
136 | for batch_idx, (features, targets) in enumerate(train_loader):
137 |
138 | features = features.to(DEVICE)
139 | targets = targets.to(DEVICE)
140 |
141 | ### FORWARD AND BACK PROP
142 | logits = model(features)
143 | cost = F.cross_entropy(logits, targets)
144 | optimizer.zero_grad()
145 |
146 | cost.backward()
147 |
148 | ### UPDATE MODEL PARAMETERS
149 | optimizer.step()
150 |
151 | ### LOGGING
152 | if not batch_idx % 50:
153 | print ('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f'
154 | %(epoch+1, num_epochs, batch_idx,
155 | len(train_loader), cost))
156 |
157 | model.eval()
158 | accuracy = compute_accuracy(model, test_loader)
159 | loss = compute_epoch_loss(model, test_loader)
160 | with torch.set_grad_enabled(False): # save memory during inference
161 | print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
162 | if write_hist:
163 | file1 = open('history.txt', 'a')
164 | file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
165 | file1.close()
166 |
167 |
168 | print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))
169 |
170 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))
171 |
172 | with torch.set_grad_enabled(False): # save memory during inference
173 | accuracy = compute_accuracy(model, test_loader)
174 | print('Test accuracy: %.2f%%' % (accuracy))
175 | if write_hist:
176 | file1 = open('history.txt', 'a')
177 | file1.write('\n Test accuracy: %.2f%%' % (accuracy))
178 | file1.close()
179 |
180 |
181 |
--------------------------------------------------------------------------------
/main_TMLCN_CIFAR10_VGG16.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import time
3 | import numpy as np
4 | import torch
5 | import torch.nn as nn
6 | import torch.nn.functional as F
7 | from torchvision import datasets
8 | from torchvision import transforms
9 | from torch.utils.data import DataLoader
10 | import matplotlib.pyplot as plt
11 | import datetime
12 | import math
13 | from collections import deque
14 | import copy
15 |
16 | #######################################
17 | ### PRE-TRAINED MODELS AVAILABLE HERE
18 | ## https://pytorch.org/docs/stable/torchvision/models.html
19 | from torchvision import models
20 | #######################################
21 |
22 | now = datetime.datetime.now()
23 |
24 | # Device
25 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
26 | print('Device:', DEVICE)
27 |
28 | ##########################
29 | ### SETTINGS
30 | ##########################
31 |
32 | FL_type = 'FbFTL' # 'FL', 'FTLf', 'FTLc', 'FbFTL'
33 | train_set_denominator = 'full' # 'full', int <= 50000 # pick a subset with 50000/int training samples
34 |
35 | # Hyperparameters
36 | NUM_CLASSES = 10
37 | U_clients = 6250 # number of clients, 50000/8
38 | random_seed = 1 # transfer = True / False
39 | learning_rate = 1e-2 # 1e-3, 0.05, 1e-2
40 | num_epochs = 200 # 10, 300, 200
41 | batch_size = 64 # 128, 128, (out of memory:64)
42 | momentum = 0.9 # None, 0.9
43 | lr_decay = 5e-4 # 1e-6, 5e-4
44 | if FL_type == 'FL':
45 | transfer, full = False, True # transfer or train model from scratch # train whole model or last few layers
46 | sigma = 0. # 0.5 relative std for addtive gaussian noise on gradients
47 | elif FL_type == 'FTLf':
48 | transfer, full = True, True
49 | sigma = 0. # 0.3305 relative std for addtive gaussian noise on gradients
50 | elif FL_type == 'FTLc':
51 | transfer, full = True, False
52 | sigma = 0. # 0.285 relative std for addtive gaussian noise on gradients
53 | elif FL_type == 'FbFTL':
54 | transfer, full = True, False
55 | sigma = 0 # 0.8? relative std for addtive gaussian noise on features
56 | saved_noise = True # save noise at beginning
57 | else:
58 | raise ValueError('Unknown FL_type: ' + FL_type)
59 | relative_noise_type = 'all_std' # 'individual', 'all_std'
60 | packet_loss_rate = 0. # 0, 0.05, 0.1, 0.15
61 | quan_digit = 32 # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int
62 | sparse_rate = 0.9 # ratio of uplink elements kept after sparsification: None or (0,1]
63 | class ErrorFeedback(object):
64 | queue = deque(maxlen=U_clients)
65 | temp = deque()
66 | if (quan_digit or sparse_rate) and FL_type != 'FbFTL':
67 | errfdbk = ErrorFeedback()
68 | # print(errfdbk.queue, errfdbk.temp)
69 |
70 | write_hist = True
71 |
72 | def adjust_learning_rate(optimizer, epoch):
73 | """Sets the learning rate to the initial LR decayed by 2 every 30 epochs"""
74 | lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs))
75 | for param_group in optimizer.param_groups:
76 | param_group['lr'] = lr
77 |
78 | if torch.cuda.is_available():
79 | torch.backends.cudnn.deterministic = True
80 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU.
81 | torch.manual_seed(random_seed) # sets the seed for generating random numbers.
82 |
83 | if write_hist:
84 | file1 = open('/data1/feng/LaTFL/history.txt', 'a')
85 | file1.write('\n \n \n Time:')
86 | file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' ' + str(now.hour) + ' '
87 | + str(now.minute) + ' ' + str(now.second) + ' ' + FL_type
88 | # + ', train_deno:' + str(train_set_denominator)
89 | # + ', sigma:' + str(sigma)
90 | # + ', packet_loss_rate:' + str(packet_loss_rate)
91 | + ', quantization digits:' + str(quan_digit)
92 | + ', sparsification rate:' + str(sparse_rate)
93 | )
94 | file1.close()
95 |
96 | ##########################
97 | ### CIFAR10 DATASET
98 | ##########################
99 |
100 | custom_transform = transforms.Compose([
101 | transforms.Resize((224, 224)),
102 | transforms.ToTensor(),
103 | transforms.Normalize(mean=[0.485, 0.456, 0.406],
104 | std=[0.229, 0.224, 0.225])
105 | ])
106 |
107 | ## Note that this particular normalization scheme is necessary since it was used for pre-training the network on ImageNet.
108 | ## These are the channel-means and standard deviations for z-score normalization.
109 |
110 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True)
111 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform)
112 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False)
113 |
114 | if train_set_denominator == 'full':
115 | train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True)
116 | train_set_len = len(train_dataset)
117 | else:
118 | # print('len(train_dataset)', len(train_dataset)) # 50000
119 | # selected_list = list(range(0, len(train_dataset), 2))
120 | selected_list = list(range(0, len(train_dataset), train_set_denominator))
121 | trainset_1 = torch.utils.data.Subset(train_dataset, selected_list)
122 | train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True)
123 | train_set_len = len(selected_list)
124 |
125 | # # Checking the dataset
126 | # for images, labels in train_loader:
127 | # print('Image batch dimensions:', images.shape)
128 | # print('Image label dimensions:', labels.shape)
129 | # break
130 |
131 | # labels = torch.zeros(10, dtype=torch.long)
132 | # for batch_idx, (features, targets) in enumerate(train_loader):
133 | # for t in targets:
134 | # labels[t] += 1
135 | # print('labels', labels)
136 |
137 | ##########################
138 | ### LOAD MODEL
139 | ##########################
140 |
141 | class GaussianNoise(nn.Module):
142 | """Gaussian noise regularizer.
143 | Args:
144 | sigma (float, optional): relative standard deviation used to generate the
145 | noise. Relative means that it will be multiplied by the magnitude of
146 | the value your are adding the noise to. This means that sigma can be
147 | the same regardless of the scale of the vector.
148 | is_relative_detach (bool, optional): whether to detach the variable before
149 | computing the scale of the noise. If `False` then the scale of the noise
150 | won't be seen as a constant but something to optimize: this will bias the
151 | network to generate vectors with smaller values.
152 | """
153 | def __init__(self, sigma=0, is_relative_detach=False):
154 | super().__init__()
155 | self.sigma = sigma
156 | self.is_relative_detach = is_relative_detach
157 | if saved_noise:
158 | self.register_buffer('noise', torch.empty(train_set_len*4096).normal_(mean=0,std=1))
159 | self.i = 0
160 | else:
161 | self.register_buffer('noise', torch.tensor(0))
162 |
163 | def forward(self, x):
164 | if self.training and quan_digit:
165 | # print(x)
166 | # print(x.dtype, torch.max(x), torch.min(x), torch.std(x))
167 | x = torch.round((2**quan_digit-1) / torch.max(x) * x) * torch.max(x) / (2**quan_digit-1)
168 | # print(x.dtype, torch.max(x), torch.min(x), torch.std(x))
169 | # print(x)
170 | # quit()
171 | if self.training and self.sigma != 0:
172 | if torch.cuda.is_available():
173 | torch.backends.cudnn.deterministic = True
174 | torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU.
175 | torch.manual_seed(random_seed) # sets the seed for generating random numbers.
176 | if relative_noise_type == 'individual':
177 | scale = self.sigma * x.detach() if self.is_relative_detach else self.sigma * x
178 | sampled_noise = self.noise.expand(*x.size()).float().normal_() * scale
179 | elif relative_noise_type == 'all_std':
180 | x_std = torch.std(x.detach()) if self.is_relative_detach else torch.std(x)
181 | # print(*x.size()) # 64 4096
182 | if saved_noise:
183 | sampled_noise = torch.reshape(self.noise[self.i*batch_size*4096 : (self.i+1)*batch_size*4096],(-1, 4096)
184 | ).detach().float() * x_std * self.sigma
185 | self.i = self.i + 1 if (self.i+1)*batch_size*4096 256:
221 | m.reset_parameters()
222 | for i in range(21, 31):
223 | model.features[i].apply(weight_reset)
224 | for i in range(7):
225 | model.classifier[i].apply(weight_reset)
226 |
227 | for param in model.parameters():
228 | param.requires_grad = True
229 |
230 | # model.classifier[3].requires_grad = True # (4096, 4096, relu, dropout(0.5))
231 | # model.classifier[6] = nn.Sequential(nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
232 | model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512),
233 | nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
234 |
235 |
236 | def Gaussian_noise_to_weights(m):
237 | if sigma!=0:
238 | with torch.no_grad():
239 | for param in m.parameters():
240 | if param.requires_grad:
241 | if relative_noise_type == 'individual':
242 | # print(param.grad.view(-1))
243 | # print(param)
244 | # scale = sigma * param.detach()
245 | scale = sigma * param.grad.detach() # todo: * math.sqrt(batch_size/8)
246 | noise = torch.tensor(0).to(DEVICE)
247 | sampled_noise = noise.expand(*param.size()).float().normal_() * scale
248 | elif relative_noise_type == 'all_std':
249 | param_grad_std = torch.std(param.grad.detach())
250 | noise = torch.tensor(0).to(DEVICE)
251 | sampled_noise = noise.expand(*param.size()).float().normal_(std=param_grad_std*sigma) # todo: * math.sqrt(batch_size/8)
252 | # param = param + sampled_noise
253 | param.add_(sampled_noise)
254 |
255 | def Errfdbk_to_weights(m):
256 | print("inner model.apply")
257 | with torch.no_grad():
258 | for param in m.parameters():
259 | print('len(param)', len(param))
260 | if param.requires_grad:
261 | print('len(errfdbk.queue)', len(errfdbk.queue))
262 | # print(errfdbk.temp)
263 | print('len(errfdbk.temp)', len(errfdbk.temp))
264 | p_grad = param.grad.detach()
265 | if err_flag:
266 | p_grad += errfdbk.temp.popleft()
267 | p_grad_qs = copy.deepcopy(p_grad)
268 | if sparse_rate:
269 | pass
270 | if quan_digit:
271 | pass
272 | err = p_grad_qs - p_grad
273 | param.add_(err)
274 | errfdbk.temp.append(-err)
275 | print('seems good')
276 |
277 | if FL_type == 'FbFTL':
278 | received_batches_FbFTL = np.ones(len(train_loader))
279 | received_batches_FbFTL[:int(len(train_loader)*packet_loss_rate)] = 0
280 | np.random.shuffle(received_batches_FbFTL)
281 |
282 | def Packet_Received(batch_idx):
283 | if FL_type == 'FbFTL':
284 | return received_batches_FbFTL[batch_idx]
285 | else:
286 | return np.random.choice(2, p=[packet_loss_rate, 1-packet_loss_rate])
287 |
288 | # for a in range(3):
289 | # print('round', a)
290 | # for i in range(50):
291 | # print(Packet_Received(i))
292 |
293 | # print(model)
294 |
295 | # for name, param in model.named_parameters():
296 | # print(name, torch.numel(param), param.requires_grad)
297 | # quit()
298 |
299 | # model(torch.randn(1, 3, 224, 224)).mean().backward()
300 | # for name, param in model.named_parameters():
301 | # print(name, param.grad)
302 | # print('value', param.data)
303 | # wait = input('next layer')
304 |
305 | ##########################
306 | ### TRAIN MODEL
307 | ##########################
308 |
309 | model = model.to(DEVICE)
310 | # if transfer:
311 | # optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
312 | # else:
313 | # optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum) # , nesterov=True)
314 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum)
315 |
316 |
317 | def compute_accuracy(model, data_loader):
318 | model.eval()
319 | correct_pred, num_examples = 0, 0
320 | with torch.no_grad():
321 | for i, (features, targets) in enumerate(data_loader):
322 | features = features.to(DEVICE)
323 | targets = targets.to(DEVICE)
324 |
325 | logits = model(features)
326 | _, predicted_labels = torch.max(logits, 1)
327 | num_examples += targets.size(0)
328 | correct_pred += (predicted_labels == targets).sum()
329 | return correct_pred.float()/num_examples * 100
330 |
331 |
332 | def compute_epoch_loss(model, data_loader):
333 | model.eval()
334 | curr_loss, num_examples = 0., 0
335 | with torch.no_grad():
336 | for batch_idx, (features, targets) in enumerate(data_loader):
337 | features = features.to(DEVICE)
338 | targets = targets.to(DEVICE)
339 | logits = model(features)
340 | loss = F.cross_entropy(logits, targets, reduction='sum')
341 | num_examples += targets.size(0)
342 | curr_loss += loss
343 |
344 | curr_loss = curr_loss / num_examples
345 | return curr_loss
346 |
347 |
348 |
349 | start_time = time.time()
350 | for epoch in range(num_epochs):
351 | # adjust_learning_rate(optimizer, epoch)
352 |
353 | model.train()
354 | for batch_idx, (features, targets) in enumerate(train_loader):
355 |
356 | if Packet_Received(batch_idx):
357 | features = features.to(DEVICE)
358 | targets = targets.to(DEVICE)
359 |
360 | ### FORWARD AND BACK PROP
361 | logits = model(features)
362 | cost = F.cross_entropy(logits, targets)
363 | optimizer.zero_grad()
364 |
365 | cost.backward()
366 |
367 | ### UPDATE MODEL PARAMETERS
368 | optimizer.step()
369 |
370 | ### PRIVACY NOISE
371 | if FL_type != 'FbFTL':
372 | # Gaussian_noise_to_weights(model, sigma * math.sqrt(batch_size/8))
373 | model.apply(Gaussian_noise_to_weights)
374 |
375 | ### Sparsification/Quantization with Error Feedback # errfdbk.queue = deque(maxlen=U_clients)
376 | if (quan_digit or sparse_rate) and FL_type != 'FbFTL':
377 | print('main loop: len(errfdbk.queue)', len(errfdbk.queue))
378 | if len(errfdbk.queue) < U_clients:
379 | errfdbk.temp = deque()
380 | err_flag = False
381 | else:
382 | errfdbk.temp = errfdbk.queue.popleft()
383 | err_flag = True
384 | model.apply(Errfdbk_to_weights)
385 | print('completed one cycle!')
386 | errfdbk.queue.append(errfdbk.temp)
387 |
388 | ### LOGGING
389 | if not batch_idx % 50:
390 | print('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f' % (epoch+1, num_epochs, batch_idx, len(train_loader), cost))
391 | # if FL_type == 'FbFTL':
392 | # print(model.classifier[6][0].sigma)
393 | # model.classifier[6][0].set_sigma(sigma=0.4)
394 |
395 | model.eval()
396 | accuracy = compute_accuracy(model, test_loader)
397 | loss = compute_epoch_loss(model, test_loader)
398 | with torch.set_grad_enabled(False): # save memory during inference
399 | print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
400 | if write_hist:
401 | file1 = open('/data1/feng/LaTFL/history.txt', 'a')
402 | file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
403 | file1.close()
404 |
405 |
406 | print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))
407 |
408 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))
409 |
410 | with torch.set_grad_enabled(False): # save memory during inference
411 | accuracy = compute_accuracy(model, test_loader)
412 | print('Test accuracy: %.2f%%' % (accuracy))
413 | if write_hist:
414 | file1 = open('/data1/feng/LaTFL/history.txt', 'a')
415 | file1.write('\n Test accuracy: %.2f%%' % (accuracy))
416 | file1.close()
417 |
418 | # model.save_weights('/data1/feng/LaTFL/cifar10vgg.h5')
419 |
420 |
--------------------------------------------------------------------------------
/main_TMLCN_FLANT5_SAMSUM.py:
--------------------------------------------------------------------------------
1 | # Source: https://www.philschmid.de/fine-tune-flan-t5
2 | # tensorboard: tensorboard --logdir log
3 | # Run the following in terminal before python3 FbFTL_LLM/test.py: huggingface-cli login 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
4 |
5 | from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq
6 | from torchinfo import summary
7 | from datasets import load_dataset
8 | from random import randrange
9 | from datasets import concatenate_datasets
10 | import evaluate
11 | import nltk
12 | import numpy as np
13 | from nltk.tokenize import sent_tokenize
14 | nltk.download("punkt")
15 | from huggingface_hub import HfFolder
16 | from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
17 | import sys
18 | from collections import deque
19 |
20 | HfFolder.save_token('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')
21 |
22 | # FL parameters
23 | FL_type = 'FbFTL' # 'FL', 'FTLf'(same as FL), 'FTLc', 'FbFTL'
24 | model_to_train = [False, 4, 0, True] # available for 'FTLc' and 'FbFTL': [False, 4, 0, True]
25 | # shared: (boolean), encoder: (int:0~8, index after which is trained, 10 to not train layer norm), decoder: (same as encoder), lm_head: (boolean)
26 | train_set_denominator = -1 # -1 to use full training dataset, or 0 < int <= 14732 : pick a subset with [int] training samples
27 | # Hyperparameters
28 | NUM_CLASSES = 32128
29 | learning_rate = 2e-4 # TODO: source code 5e-5, full model 2e-4
30 | num_train_epochs = 20 # TODO: source code 5, full model 20
31 | batch_size = 8 # source code 8
32 | # TODO: implement following params
33 | # if FL_type == 'FL':
34 | # transfer, full = False, True # transfer or train model from scratch # train whole model or last few layers
35 | # sigma = 0. # 0.5 relative std for addtive gaussian noise on gradients
36 | # elif FL_type == 'FTLf':
37 | # transfer, full = True, True
38 | # sigma = 0. # 0.3305 relative std for addtive gaussian noise on gradients
39 | # elif FL_type == 'FTLc':
40 | # transfer, full = True, False
41 | # sigma = 0. # 0.285 relative std for addtive gaussian noise on gradients
42 | # elif FL_type == 'FbFTL':
43 | # transfer, full = True, False
44 | # sigma = 0 # 0.8? relative std for addtive gaussian noise on features
45 | # saved_noise = True # save noise at beginning
46 | # else:
47 | # raise ValueError('Unknown FL_type: ' + FL_type)
48 | # relative_noise_type = 'all_std' # 'individual', 'all_std'
49 | # packet_loss_rate = 0. # 0, 0.05, 0.1, 0.15
50 | # quan_digit = 32 # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int
51 | # sparse_rate = 0.9 # ratio of uplink elements kept after sparsification: None or (0,1]
52 |
53 | # Load dataset from the hub
54 | dataset_id = "samsum"
55 | dataset = load_dataset(dataset_id)
56 | if train_set_denominator != -1:
57 | dataset['train'] = dataset['train'].select(range(train_set_denominator))
58 | print(f"Train dataset size: {len(dataset['train'])}") # 14732
59 | print(f"Test dataset size: {len(dataset['test'])}") # 819
60 | sample = dataset['train'][randrange(len(dataset["train"]))]
61 | # print(f"dialogue: \n{sample['dialogue']}\n---------------")
62 | # print(f"summary: \n{sample['summary']}\n---------------")
63 | # sys.exit()
64 |
65 | # Loading the Model
66 | model_id="google/flan-t5-small"
67 | tokenizer = AutoTokenizer.from_pretrained(model_id)
68 | model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
69 | # for parameter in model.parameters(): # model: shared, encoder, decoder, lm_head
70 | # parameter.requires_grad = False
71 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[0]:
72 | for parameter in model.shared.parameters(): # initial embedding layer, not trained
73 | parameter.requires_grad = False
74 |
75 | if FL_type in ['FTLc', 'FbFTL']:
76 | for i, m in enumerate(model.encoder.block): # whether train encoder blocks
77 | if i < model_to_train[1]:
78 | for parameter in m.parameters():
79 | parameter.requires_grad = False
80 | if model_to_train[1] >= 10:
81 | for parameter in model.encoder.final_layer_norm.parameters(): # whether train encoder layer norm
82 | parameter.requires_grad = False
83 |
84 | if FL_type in ['FTLc', 'FbFTL']:
85 | for i, m in enumerate(model.decoder.block): # whether train decoder blocks
86 | if i < model_to_train[2]:
87 | for parameter in m.parameters():
88 | parameter.requires_grad = False
89 | if model_to_train[2] >= 10:
90 | for parameter in model.decoder.final_layer_norm.parameters(): # whether train decoder layer norm
91 | parameter.requires_grad = False
92 |
93 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[3]:
94 | for parameter in model.lm_head.parameters(): # final output layer, always trained
95 | parameter.requires_grad = False
96 | summary(model)
97 | # print(model)
98 | # for param in model.state_dict():
99 | # print(param)
100 | # sys.exit()
101 |
102 | # Preprocess data
103 | tokenized_inputs = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["dialogue"], truncation=True),
104 | batched=True, remove_columns=["dialogue", "summary"])
105 | max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
106 | print(f"Max source length: {max_source_length}")
107 | tokenized_targets = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["summary"], truncation=True),
108 | batched=True, remove_columns=["dialogue", "summary"])
109 | max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
110 | print(f"Max target length: {max_target_length}")
111 |
112 | def preprocess_function(sample,padding="max_length"):
113 | # add prefix to the input for t5
114 | inputs = ["summarize: " + item for item in sample["dialogue"]]
115 | # tokenize inputs
116 | model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)
117 | # Tokenize targets with the `text_target` keyword argument
118 | labels = tokenizer(text_target=sample["summary"], max_length=max_target_length, padding=padding, truncation=True)
119 | # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
120 | # padding in the loss.
121 | if padding == "max_length":
122 | labels["input_ids"] = [
123 | [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
124 | ]
125 | model_inputs["labels"] = labels["input_ids"]
126 | return model_inputs
127 |
128 | tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["dialogue", "summary", "id"])
129 | print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")
130 |
131 | # Metric
132 | metric = evaluate.load("rouge")
133 | # helper function to postprocess text
134 | def postprocess_text(preds, labels):
135 | preds = [pred.strip() for pred in preds]
136 | labels = [label.strip() for label in labels]
137 | # rougeLSum expects newline after each sentence
138 | preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
139 | labels = ["\n".join(sent_tokenize(label)) for label in labels]
140 | return preds, labels
141 |
142 | def compute_metrics(eval_preds):
143 | preds, labels = eval_preds
144 | if isinstance(preds, tuple):
145 | preds = preds[0]
146 | decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
147 | # Replace -100 in the labels as we can't decode them.
148 | labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
149 | decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
150 | # Some simple post-processing
151 | decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
152 | result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
153 | result = {k: round(v * 100, 4) for k, v in result.items()}
154 | prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
155 | result["gen_len"] = np.mean(prediction_lens)
156 | return result
157 |
158 | # we want to ignore tokenizer pad token in the loss
159 | label_pad_token_id = -100
160 | # Data collator
161 | data_collator = DataCollatorForSeq2Seq(
162 | tokenizer,
163 | model=model,
164 | label_pad_token_id=label_pad_token_id,
165 | pad_to_multiple_of=8
166 | )
167 |
168 | # Hugging Face repository id
169 | repository_id = f"{model_id.split('/')[1]}-{dataset_id}"
170 |
171 | # Define training args
172 | training_args = Seq2SeqTrainingArguments(
173 | output_dir=repository_id,
174 | per_device_train_batch_size=batch_size,
175 | per_device_eval_batch_size=8,
176 | predict_with_generate=True,
177 | fp16=False, # Overflows with fp16
178 | learning_rate=learning_rate,
179 | num_train_epochs=num_train_epochs,
180 | # logging & evaluation strategies
181 | logging_dir=f"{repository_id}/logs",
182 | logging_strategy="steps",
183 | logging_steps=500,
184 | evaluation_strategy="epoch",
185 | save_strategy="epoch",
186 | save_total_limit=2,
187 | load_best_model_at_end=True,
188 | # metric_for_best_model="overall_f1",
189 | # push to hub parameters
190 | report_to="tensorboard",
191 | push_to_hub=False,
192 | hub_strategy="every_save",
193 | hub_model_id=repository_id,
194 | hub_token=HfFolder.get_token(),
195 | )
196 |
197 | # Create Trainer instance
198 | trainer = Seq2SeqTrainer(
199 | model=model,
200 | args=training_args,
201 | data_collator=data_collator,
202 | train_dataset=tokenized_dataset["train"],
203 | eval_dataset=tokenized_dataset["test"],
204 | compute_metrics=compute_metrics,
205 | )
206 |
207 | # Start training
208 | trainer.train()
209 |
210 | trainer.evaluate()
211 |
212 | # Save our tokenizer and create model card
213 | tokenizer.save_pretrained(repository_id)
214 | trainer.create_model_card()
215 | # Push the results to the hub
216 | # trainer.push_to_hub()
217 |
218 |
219 | # Run Inference and summarize ChatGPT dialogues
220 | # from transformers import pipeline
221 | # from random import randrange
222 | # # load model and tokenizer from huggingface hub with pipeline
223 | # summarizer = pipeline("summarization", model="philschmid/flan-t5-base-samsum", device=0)
224 | # # select a random test sample
225 | # sample = dataset['test'][randrange(len(dataset["test"]))]
226 | # print(f"dialogue: \n{sample['dialogue']}\n---------------")
227 | # # summarize dialogue
228 | # res = summarizer(sample["dialogue"])
229 | # print(f"flan-t5-base summary:\n{res[0]['summary_text']}")
--------------------------------------------------------------------------------