├── README.md
├── code_for_plots
    ├── Robustness_and_Privacy.ipynb
    └── print_privacy_leakage.ipynb
├── diagrams
    └── FbFTL_diagram.png
├── history.txt
├── main_GLOBECOM_CIFAR10_VGG16.py
├── main_TMLCN_CIFAR10_VGG16.py
└── main_TMLCN_FLANT5_SAMSUM.py


/README.md:
--------------------------------------------------------------------------------
 1 | # FbFTL: Communication-Efficient Feature-based Federated Transfer Learning
 2 | 
 3 | This is the offical implementation for Python simulation of Feature-based Federated Transfer Learning (FbFTL), from the following conference paper and journal paper: 
 4 | 
 5 |   Communication-Efficient Feature-based Federated Transfer Learning.([Globecom2022](https://ieeexplore.ieee.org/abstract/document/10000612), [arXiv](https://arxiv.org/abs/2209.05395))  
 6 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar  
 7 | Department of Electrical Engineering and Computer Science, Syracuse University
 8 | 
 9 |   Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy.([TMLCN](https://ieeexplore.ieee.org/abstract/document/10542971), [arXiv](https://arxiv.org/abs/2405.09014))  
10 | Feng Wang, M. Cenk Gursoy and Senem Velipasalar  
11 | Department of Electrical Engineering and Computer Science, Syracuse University
12 | 
13 | ---
14 | 
15 | <img src="https://github.com/wfwf10/Feature-based-Federated-Transfer-Learning/blob/main/diagrams/FbFTL_diagram.png" width="644" height="501">
16 | 
17 | We propose the FbFTL as an innovative federated learning approach that upload features and outputs instead of gradients to reduce the uplink payload by more than five orders of magnitude. Please refer to the journal paper for explicit explaination on learning structure, system design, robustness analysis, and privacy analysis.
18 | 
19 | 
20 | # Results on CIFAR-10 Dataset with VGG16 Model
21 | In the following table, we provide comparison between federated learning with [FedAvg](http://proceedings.mlr.press/v54/mcmahan17a.html) (FL), federated transfer learning with FedAvg that updating full model (FTL<sub>f</sub>), federated transfer learning with FedAvg that updating task-specific sub-model(FTL<sub>c</sub>), and FbFTL. All of them learn [VGG16](https://arxiv.org/abs/1409.1556) model on [CIFAR-10](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf) dataset. For transfer learning approaches, the source models are trained on [ImageNet](https://ieeexplore.ieee.org/abstract/document/5206848?casa_token=QncCRBM1tzAAAAAA:QuoJhjJAHRplmLJ4jcFw5JWdfASjmbIVlvpCrHgTPIFu63gpSUlBeACB78S0AH34qqQnsBOdoQ) dataset. Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude. 
22 | 
23 | | | FL | FTL<sub>f</sub> | FTL<sub>c</sub> | FbFTL  |
24 | | ---- | ----- | ---- | ---- | ---- |
25 | | upload batches | 656250 | 193750 | 525000 | 50000 |
26 | | upload parameters per batch | 153144650 | 153144650 | 35665418 | 4096 |
27 | | uplink payload per batch | **4.9 Gb** | **4.9 Gb** | **1.1 Gb** | **131 Kb**  |
28 | | total uplink payload | **3216 Tb** | **949 Tb** | **599 Tb** | **6.6 Gb** |
29 | | total downlink payload | 402 Tb | 253 Tb | 322 Tb | 3.8 Gb |
30 | | test accuracy | 89.42\% | 93.75\% | 86.51\% | 86.51\% |
31 | 
32 | # Results on SAMSum summary task with FLAN-T5-small language model
33 | In the following table, we consider [FLAN-T5-small](https://www.jmlr.org/papers/volume25/23-0870/23-0870.pdf) as a pre-trained language model, and fine-tune on [SAMSum](https://www.aclweb.org/anthology/D19-5409) summary task. As a fine-tuning task, this experiment does not include an FL setting, and we provide comparison between federated transfer learning with FedAvg that updating full model (FTL<sub>f</sub>), federated transfer learning with FedAvg that updating task-specific sub-model(FTL<sub>c</sub>), and FbFTL.  Compared to all other methods, FbFTL reduces the uplink payload by up to five orders of magnitude. 
34 | 
35 | |  | FTL<sub>f</sub> | FTL<sub>c</sub> | FbFTL  | FTL<sub>c</sub> | FbFTL  | FTL<sub>c</sub> | FbFTL  |
36 | | ---- | ----- | ---- | ---- | ---- | ---- | ---- | ---- |
37 | | number of trained encoders | 8 | 8 | 8 | 4 | 4 | 2 | 2 |
38 | | number of upload batches | 132588 | 36830 | 7366 | 88392 | 7366 | 103124 | 7366 |
39 | | upload parameters per batch | 109860224 | 60511616 | 1024 | 51070144 | 1024 | 46349504 | 1024 |
40 | | uplink payload per batch  | **3.5 Gb** | **1.9 Gb** | **32.7 Kb** | **1.6 Gb** | **32.7 Kb** | **1.5 Gb** | **32.7 Kb** |
41 | | total uplink payload | **466.1 Tb** | **71.3 Tb** | **241.4 Mb** | **144.5 Tb** | **241.4 Mb** | **152.9 Tb** | **241.4 Mb** |
42 | | total downlink payload | 116.0 Tb | 32.2 Tb | 1.58 Gb | 77.3 Tb | 1.88 Gb | 90.2 Tb | 2.03 Gb |
43 | | test ROUGE-1 | 45.9249 | 45.4680 | 45.4680 | 45.2827 | 45.2827 | 44.9862 | 44.9862  |
44 | 
45 | # Required packages installation
46 | We use python==3.6.9, numpy==1.19.5, torch==1.4.0, torchvision==0.5.0, and CUDA version 11.6 for the experiments on CIFAR-10 with VGG16. The dataset and the source model will be automatically downloaded.
47 | 
48 | Additionally, for the experiments on SAMSUM with FLAN-T5, we use transformers==4.30.2, torchinfo==1.8.0, datasets==2.13.2, nltk==3.8.1, evaluate==0.4.1, huggingface_hub==0.16.4
49 | 
50 | # Citation
51 | If you find our work useful in your research, please consider citing:
52 | ```
53 | @inproceedings{wang2022communication,
54 |   title={Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learning},
55 |   author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem},
56 |   booktitle={GLOBECOM 2022-2022 IEEE Global Communications Conference},
57 |   pages={3875--3880},
58 |   year={2022},
59 |   organization={IEEE}
60 | }
61 | ```
62 | 
63 | ```
64 | @article{wang2024feature,
65 |   title={Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy},
66 |   author={Wang, Feng and Gursoy, M Cenk and Velipasalar, Senem},
67 |   journal={IEEE Transactions on Machine Learning in Communications and Networking},
68 |   year={2024},
69 |   publisher={IEEE}
70 | }
71 | ```
72 | 


--------------------------------------------------------------------------------
/code_for_plots/print_privacy_leakage.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |   "nbformat": 4,
  3 |   "nbformat_minor": 0,
  4 |   "metadata": {
  5 |     "colab": {
  6 |       "provenance": []
  7 |     },
  8 |     "kernelspec": {
  9 |       "name": "python3",
 10 |       "display_name": "Python 3"
 11 |     },
 12 |     "language_info": {
 13 |       "name": "python"
 14 |     }
 15 |   },
 16 |   "cells": [
 17 |     {
 18 |       "cell_type": "markdown",
 19 |       "source": [
 20 |         "# CIFAR-10"
 21 |       ],
 22 |       "metadata": {
 23 |         "id": "RnC64R4j-se4"
 24 |       }
 25 |     },
 26 |     {
 27 |       "cell_type": "code",
 28 |       "execution_count": null,
 29 |       "metadata": {
 30 |         "id": "Ogn9Z9JF0EHG"
 31 |       },
 32 |       "outputs": [],
 33 |       "source": [
 34 |         "import numpy as np\n",
 35 |         "import matplotlib.pyplot as plt\n",
 36 |         "import math\n",
 37 |         "from math import factorial as f\n",
 38 |         "from time import sleep\n",
 39 |         "import itertools\n",
 40 |         "import time"
 41 |       ]
 42 |     },
 43 |     {
 44 |       "cell_type": "code",
 45 |       "source": [
 46 |         "K = 8\n",
 47 |         "N = 10\n",
 48 |         "B = f(N+K-1) // f(K) // f(N-1)\n",
 49 |         "\n",
 50 |         "# calculate H(B|Y=y), y is uniform\n",
 51 |         "p_y = 0\n",
 52 |         "P_ys = []\n",
 53 |         "b_count = 0\n",
 54 |         "H_y = 0\n",
 55 |         "\n",
 56 |         "for c0 in range(0, N+K-1):\n",
 57 |         "  for c1 in range(c0+1, N+K-1):\n",
 58 |         "    for c2 in range(c1+1, N+K-1):\n",
 59 |         "      for c3 in range(c2+1, N+K-1):\n",
 60 |         "        for c4 in range(c3+1, N+K-1):\n",
 61 |         "          for c5 in range(c4+1, N+K-1):\n",
 62 |         "            for c6 in range(c5+1, N+K-1):\n",
 63 |         "              for c7 in range(c6+1, N+K-1):\n",
 64 |         "                b_count += 1\n",
 65 |         "                Ns = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
 66 |         "                Ns[c0-0] += 1\n",
 67 |         "                Ns[c1-1] += 1\n",
 68 |         "                Ns[c2-2] += 1\n",
 69 |         "                Ns[c3-3] += 1\n",
 70 |         "                Ns[c4-4] += 1\n",
 71 |         "                Ns[c5-5] += 1\n",
 72 |         "                Ns[c6-6] += 1\n",
 73 |         "                Ns[c7-7] += 1\n",
 74 |         "                p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5])*f(Ns[6])*f(Ns[7])*f(Ns[8])*f(Ns[9]))\n",
 75 |         "                P_ys.append(p_y)\n",
 76 |         "                H_y += -p_y * math.log2(p_y)\n",
 77 |         "\n",
 78 |         "print(len(P_ys))\n",
 79 |         "print(sum(P_ys))\n",
 80 |         "print(b_count)\n",
 81 |         "print('B', B)\n",
 82 |         "print(math.log2(B))\n",
 83 |         "print('H_y', H_y)\n",
 84 |         "print(2**H_y)"
 85 |       ],
 86 |       "metadata": {
 87 |         "colab": {
 88 |           "base_uri": "https://localhost:8080/"
 89 |         },
 90 |         "id": "7qZfigBQGzVQ",
 91 |         "outputId": "1d0a0bbf-fdb0-4fc7-aee6-2248cb12cd81"
 92 |       },
 93 |       "execution_count": null,
 94 |       "outputs": [
 95 |         {
 96 |           "output_type": "stream",
 97 |           "name": "stdout",
 98 |           "text": [
 99 |             "24310\n",
100 |             "1.0000000000001252\n",
101 |             "24310\n",
102 |             "B 24310\n",
103 |             "14.569262272916092\n",
104 |             "H_y 13.860045564098357\n",
105 |             "14869.263443691536\n"
106 |           ]
107 |         }
108 |       ]
109 |     },
110 |     {
111 |       "cell_type": "code",
112 |       "source": [
113 |         "# calculate H(B|S=s)\n",
114 |         "U_range = np.arange(1, 6252, 1250)\n",
115 |         "# U_range = np.arange(1, B, 3000)\n",
116 |         "H_ss = np.zeros_like(U_range).astype(float)\n",
117 |         "\n",
118 |         "for i, U in enumerate(U_range):\n",
119 |         "  H_s = 0\n",
120 |         "  p_s = 0\n",
121 |         "  choices = np.random.choice(B, U, p=P_ys)\n",
122 |         "  # print(len(choices), choices)\n",
123 |         "  for u in range(U):\n",
124 |         "    p_s = sum([1 for c in choices if c == u]) / U\n",
125 |         "    # print(p_s)\n",
126 |         "    if p_s != 0:\n",
127 |         "      H_s += -p_s * math.log2(p_s)\n",
128 |         "  print('U:', U, 'H_s:', H_s)\n",
129 |         "  H_ss[i] = H_s\n",
130 |         "\n",
131 |         "print('H_ss', H_ss)"
132 |       ],
133 |       "metadata": {
134 |         "colab": {
135 |           "base_uri": "https://localhost:8080/"
136 |         },
137 |         "id": "ePqcA5Ua7aZk",
138 |         "outputId": "7f86a441-d4ff-4127-8e16-58b339d148b9"
139 |       },
140 |       "execution_count": null,
141 |       "outputs": [
142 |         {
143 |           "output_type": "stream",
144 |           "name": "stdout",
145 |           "text": [
146 |             "U: 1 H_s: 0\n",
147 |             "U: 1251 H_s: 0.19578959694642661\n",
148 |             "U: 2501 H_s: 0.527494304228384\n",
149 |             "U: 3751 H_s: 1.6176505965077759\n",
150 |             "U: 5001 H_s: 2.240630027446816\n",
151 |             "U: 6251 H_s: 3.09074958685801\n",
152 |             "H_ss [0.         0.1957896  0.5274943  1.6176506  2.24063003 3.09074959]\n"
153 |           ]
154 |         }
155 |       ]
156 |     },
157 |     {
158 |       "cell_type": "code",
159 |       "source": [
160 |         "H_ss = np.array([ 0.,          1.14445591,  2.92142532,  5.66463935,  7.32508148,  8.84656221, 10.8259317,  12.2532801,  13.15547667])\n",
161 |         "H_y = 13.860045564098357\n",
162 |         "plt.plot(np.arange(9)*6250/9, H_y-H_ss)\n",
163 |         "# H_ss = np.array([ 0., 0.1891638,  0.78579461, 1.54249295, 2.22972449, 3.19595329])\n",
164 |         "# H_y = 13.860045564098357\n",
165 |         "# plt.plot(np.arange(6)*6250/6, H_y-H_ss)\n",
166 |         "plt.xlabel('Number of clients')\n",
167 |         "plt.ylabel('Privacy leakage (in bits)')\n",
168 |         "plt.show()"
169 |       ],
170 |       "metadata": {
171 |         "colab": {
172 |           "base_uri": "https://localhost:8080/",
173 |           "height": 279
174 |         },
175 |         "id": "bbZ_N7hK87yL",
176 |         "outputId": "b63f85ad-1ffd-4ae2-ab25-3659484863df"
177 |       },
178 |       "execution_count": null,
179 |       "outputs": [
180 |         {
181 |           "output_type": "display_data",
182 |           "data": {
183 |             "text/plain": [
184 |               "<Figure size 432x288 with 1 Axes>"
185 |             ],
186 |             "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gVVbb38e/qRM45Z1BEQIIiSQwoKgrGMaOiOGMA9DqjvuPMHZ2Z66hjGjOCiuFiFhVFxVGiiuSMCEhqsuRMw3r/ONXelrHh0N2n64Tf53nq6Tp16lStDYdF9a5da5u7IyIiqSMt7ABERKR4KfGLiKQYJX4RkRSjxC8ikmKU+EVEUkxG2AFEo2rVqt6wYcOwwxARSSjTpk3b6O7VDt2eEIm/YcOGTJ06NewwREQSipkt/7Xt6uoREUkxSvwiIilGiV9EJMXELPGb2Ytmtt7M5v7Ke/9lZm5mVWN1fhER+XWxvOJ/Geh16EYzqwecCayI4blFRCQfMUv87j4e2PQrbz0G/AFQdTgRkRAUax+/mfUBst19VhT7DjCzqWY2dcOGDcUQnYhIaii2xG9mpYH/B/w5mv3dfYi7d3D3DtWq/cfzB1EZv2gDL078kZ17cwr0eRGRZFScV/xNgEbALDNbBtQFpptZzVid8IsF67h/1HxOfuDfPPjpQtZt2xOrU4mIJAyL5UQsZtYQGOXurX7lvWVAB3ffeKTjdOjQwQv65O70FZsZOmEpn85dS3qacX6bOtzYvRHH1CxfoOOJiCQKM5vm7h0O3R6zkg1mNgLoAVQ1s1XAf7v7sFidLz/t6lfimSvbs+KnXbw46UfenLKSd6evonvzagzo1pguTatgZsUdlohIaGJ6xV9UCnPFf6gtu/bx+uQVvDRpGRt37OWYmuUY0L0xvVvXJitDz7OJSPLI74o/5RJ/rr05B/hg5mpeGL+UH9bvoEb5ElzXpRGXn1ifCqUyi/RcIiJhUOLPh7szbtEGXpiwlEmLf6JMVjqXnVif67o0pG6l0jE5p4hIcVDij8Lc7K0MnbCUj2avAeCc42sxoFtjjq9bIebnFhEpakr8R2H1lt28/PUy/nfyCnbszaFT48rc2K0xp7aoTlqabgSLSGJQ4i+A7Xv28+aUlbw48UdWb91Dk2pluLFbY/qeUIeSmenFHo+IyNFQ4i+E/QcO8smcNQwZv5R5q7dRtWwW15zckKs6NaBymazQ4hIRORwl/iLg7nyz9CdeGL+Ur77fQMnMNC5pX4/+XRvRsGqZsMMTEfmFYn+AKxmZGZ2bVKVzk6osWredoROW8uaUlbw2eTlntqzBgO6Nad+gcthhiogclq74C2n99j288vVyXv12OVt376dd/YoM6N6Yni1rkq4bwSISInX1xNiufTm8M20VQyf8yIpNu2hQpTT9uzbi4vZ1KZ2lX6xEpPgp8ReTAwedz+etZciEpcxYsYWKpTO56qQGXNO5AdXLlQw7PBFJIUr8IZi2fBNDxi/l8/nryExL44IT6nBDt0Y0q1Eu7NBEJAUo8Yfox407eXHij7w9bSV79h/k1BbV+F2PppzYSDeCRSR2lPjjwKad+3jt2+UM/3oZP+3cR7+TG3DPOcfqYTARiYn8Er/qEBejymWyGHh6MybdfRr9uzZi+DfL6f3kROZmbw07NBFJIUr8ISiZmc6ferfktf4nsX3Pfi54ZhLPjF3MgYPx/9uXiCQ+Jf4QdW1Wlc8Gd+fMljV56NPvuXzIt6zctCvssEQkySnxh6xi6SyeuuIEHr20DfPXbOOcJybw3vRVJMK9FxFJTEr8ccDMuLBdXUYP6sYxtcpxx1uzuHXEDLbs2hd2aCKShJT440i9yqV5Y8DJ/KFXCz6bu5Zej09g4g8bww5LRJKMEn+cSU8zbu7RlJG3dKFMiXSuGjaZv46az579B8IOTUSSRMwSv5m9aGbrzWxunm0Pm9lCM5ttZu+bWcVYnT/RtapTgVG3daPfyQ0YNvFH+jw1iQVrtoUdlogkgVhe8b8M9Dpk2xiglbu3BhYB98Tw/AmvVFY69/VpxUvXdWTTrn30eWoSL4xfykEN+xSRQohZ4nf38cCmQ7Z97u45wctvgbqxOn8yObVFdT4b3J0eLarx908WcOXQyazesjvssEQkQYXZx389MDq/N81sgJlNNbOpGzZsKMaw4lPlMlk8f3V7HrqoNbNXbaHX4+P5cNbqsMMSkQQUSuI3sz8COcDr+e3j7kPcvYO7d6hWrVrxBRfHzIxLO9bjk0HdaFq9LANHzGDQGzPYunt/2KGJSAIp9sRvZtcCvYErXU8pFUiDKmV466aTuaNnc0bNXsPZj4/nmyU/hR2WiCSIYk38ZtYL+ANwvrurNkEhZKSnMfD0Zrz7u86UyEzniqHf8sDoBezN0bBPETm8WA7nHAF8A7Qws1Vm1h94CigHjDGzmWb2XKzOnyra1qvIxwO7cvmJ9Xl+3FL6Pv01i9ZtDzssEYljqsefRL6Yv4673p3N9r053N3rGK7t3JA0TfgukrJUjz8FnNGyBp8O7k63plW5f9R8+r30HWu37gk7LBGJM0r8SaZauRIM7deB/7ngeKYu28xZj4/nkzlrwg5LROKIEn8SMjOuOKk+Hw/sSsMqpbn59en811uz2L5Hwz5FRIk/qTWuVpZ3fteZgac34/0Zqzj7iQlMWbbpyB8UkaSmxJ/kMtPTuKNnc97+bWfSzPjN89/w8GcL2ZdzMOzQRCQkSvwpon2DSnwyqBuXtK/H018t4cJnJ7F4/Y6wwxKRECjxp5CyJTJ48OLWPHdVe7I376b3kxN45ZtlmuZRJMVkHGkHM6sOdAFqA7uBucBUd1dfQYLq1aom7epX5PfvzObPH8zjy4Xreeji1lQvVzLs0ESkGOR7xW9mp5rZZ8DHwNlALaAlcC8wx8zuM7PyxROmFLXq5Uvy8nUdub/PcXyz5Cd6PT6Bz+atDTssESkGh7viPwe40d1XHPqGmWUQKbTWE3g3RrFJjJkZ15zckM5NqjD4zZnc9Oo0Lj+xPn/r24p0PfErkrTyTfzu/vvDvJcDjIxJRFLsmlYvx3u/68IjY77n+XGRGb7+cdHxmCn5iySjI97cNbNBZlbeIoaZ2XQzO7M4gpPik5WRxj1nH8vA05ry5tSVPPjp92GHJCIxEs2onuvdfRtwJlAJuBr4R0yjktDc3rM5V3Wqz3PjlvD8uCVhhyMiMXDEUT1A7u/75wCvuvs8Ux9A0jIz7ju/FVt27eeB0QupVDqLSzvWCzssESlC0ST+aWb2OdAIuMfMygEaypnE0tOMRy9ty7Y9Odz93mzKl8qkV6uaYYclIkUkmq6e/sDdQMdg1qws4LqYRiWhy8pI47mr2tGmXkUGjpjB10s2hh2SiBSRaBL/GHef7u5bANz9J+Cx2IYl8aB0VgYvXduRRlXLcOPwqcxetSXskESkCBzuAa6SZlYZqGpmlcyscrA0BOoUV4ASroqls3il/4lUKpPFtS9NUX0fkSRwuCv+m4BpwDHA9GB9GvABkblzJUXUKF+S1/qfRJrBNcMms3rL7rBDEpFCyDfxu/sT7t4IuNPdG+VZ2ri7En+KaVi1DMOvP5Hte3K4ethkNu3cF3ZIIlJAh+vqOS1YzTazCw9diik+iSPH1a7AsGs7smrzbq576Tt27M0JOyQRKYDDdfWcEvw871eW3kc6sJm9aGbrzWxunm2VzWyMmf0Q/KxUiNglBCc2qswzV7Zj7upt3PTqVPbmHAg7JBE5SharWuxm1h3YAbzi7q2CbQ8Bm9z9H2Z2N1DJ3e860rE6dOjgU6dOjUmcUjDvTV/FHW/NotdxNXn6ynYq6iYSh8xsmrt3OHR7NLV6qpjZv4IaPdPM7Akzq3Kkz7n7eODQCV77AMOD9eFA3yhilzh0Ybu6/Ll3Sz6dt5Y/vj9Hk7mIJJBoxvG/AWwALgIuDtbfLOD5arj7mmB9LVAjvx3NbICZTTWzqRs2bCjg6SSWru/aiIGnNeWNKSrqJpJIoinZUMvd/5rn9d/M7DeFPbG7u5nle5no7kOAIRDp6ins+SQ2bu/ZnE279vHcuCVUKp3JTac0CTskETmCaK74Pzezy8wsLVguBT4r4PnWmVktgODn+gIeR+JEblG33q1r8cDohbw1ZWXYIYnIEeR7xW9m2wEnUp1zMPBq8FY6kZu2dxbgfB8C/YiUde5H5GEwSXAq6iaSWA73AFc5dy8f/Exz98xgSXP3I861a2YjgG+AFma2ysz6E0n4Pc3sB+AMVNc/aaiom0jiiNlwzqKk4ZyJY8uufVz6/Ddkb97NiAGdaF23YtghiaSsAg/nFDkaFUtn8Wr/k34u6rZkg4q6icQbJX4pcnmLul09VEXdROJNVInfzNLNrLaZ1c9dYh2YJDYVdROJX9E8uXsbsA4YA3wcLKNiHJckARV1E4lP0VzxDwJauPtx7n58sLSOdWCSHFTUTST+RJP4VwJbYx2IJK/Tj63Bwxe3ZtLinxg0YiYHDsb/SDKRZBZNyYalwFgz+xjYm7vR3R+NWVSSdC5sV5ctu/Zz/6j5/PH9OTxw4fGYqaKnSBiiSfwrgiUrWEQK5Pqujdi8ax9PfrmYSmWyuKvXMWGHJJKSjpj43f2+4ghEUsMdPZuzedc+nh0bKeo2oLuKuokUt8PV6nnc3Qeb2UdEavb8grufH9PIJCnlFnXbsms///PJQiqWyuLSjvXCDkskpRzuij+3KNs/iyMQSR2HFnWrUDqTs45TUTeR4qJaPRKaXftyuHLoZOZlb+Pl6zvSuUnVsEMSSSpHXavHzD4ys/PMLPNX3mtsZveb2fVFHaikjtJZGbx0bUcaVi3NjcOnMnvVlrBDEkkJhxvHfyPQDVhoZlPM7BMz+9LMlgLPA9Pc/cViiVKSloq6iRS/qLp6zKwhUAvYDSxy912xDeuX1NWT/JZt3MnFz31NVnoa7/yuM7Urlgo7JJGEV6iyzO6+zN2/cfeZxZ30JTWoqJtI8VFZZokbx9WuwNB+HVTUTSTGlPglrpzUuApPX6GibiKxFG09/lJm1iLWwYgAnNHy/4q6DX5DRd1Eilo09fjPA2YCnwav25rZh7EOTFLbhe3q8ufeLRk9dy1/fH8OifC8iUiiiKZI21+AE4GxAO4+08waxTAmEeCXRd0qlMrkrl7HkJamip4ihRVNV89+dz+0Hn+hLr/M7HYzm2dmc81shJmVLMzxJHnd0bM5V3dqwPPjl3L5C9+ybOPOsEMSSXjRJP55ZnYFkG5mzczsSeDrgp7QzOoAA4EO7t4KSAcuK+jxJLmZGff3OY4HLzqe+au30euJ8QydsFT9/iKFEE3ivw04jsgkLCOAbcDgQp43AyhlZhlAaWB1IY8nSczM+E3H+oy54xS6NKnK3z5ewMXPfc3i9dvDDk0kIYVSpM3MBgF/J/Ik8OfufuWv7DMAGABQv3799suXLy/eICUuuTsfzlrNf384j117DzDojGYM6N6YzHSNTBY5VH5P7h4x8edTj38rMBV43t33HGUglYB3gd8AW4C3gXfc/bX8PqOSDXKoDdv38pcP5/HxnDW0qlOehy5qQ8va5cMOSySuFKZkw1JgB/BCsGwDtgPNg9dH6wzgR3ff4O77gfeAzgU4jqSwauVK8PSV7Xj2ynas3bqX85+ayKOff68HvkSiEM1wzs7u3jHP64/MbIq7dzSzeQU45wqgk5mVJtLVczqR3x5EjtrZx9eiU+Mq/HXUfP715WI+nbeWhy5uQ9t6FcMOTSRuRXPFX9bM6ue+CNbLBi+PupKWu08G3gGmA3OCGIYc7XFEclUqk8Wjv2nLS9d2ZPueHC58ZhIPfLKAPft19S/ya6Lp4z8HeA5YAhjQCLiZyANdN7r74zGOUX38ErVte/bzwCcLGfHdChpVLcNDF7emY8PKYYclEooC39wNPlwCOCZ4+f3R3tAtLCV+OVqTFm/krndnk71lN/1Obsjvz2pBmRLR9GyKJI9C1eMHmgEtgDbApWZ2TVEGJ1LUujStymeDu9Pv5IYM/2YZZz0+nok/bAw7LJG4EE2Rtv8GngyWU4GHgPNjHJdIoZUpkcFfzj+Ot246maz0NK4aNpm7353Ntj37ww5NJFTRXPFfTGTkzVp3v47IVX+FmEYlUoQ6NqzMJ4O6cdMpjXlr6krOfHQ8Xy5cF3ZYIqGJJvHvdveDQI6ZlQfWA/ViG5ZI0SqZmc49Zx/L+zd3oUKpTK5/eSq3vzmTzZriUVJQNIl/qplVJPKw1jQiwzC/iWlUIjHSpl5FPrqtK4NOb8ZHs1bT87FxjJ6zJuywRIpVNMM5zYOdzKwhUJ7IyJ69MY8uoFE9EgvzV2/jD+/OYm72Ns5uVZP7+7SiWrkSYYclUmQKM6pnWO6Kuy8jUsLhk6ILTSQcLWuXZ+TNXfhDrxb8e+F6ej42jvdnrNJsX5L0okn82Wb2DPxcYO1zIN+CaiKJJCM9jZt7NOWTgV1pXLUMt785i/7Dp7Jm6+6wQxOJmSMmfnf/E7DDzJ4jkvQfcfeXYh6ZSDFqWr0cb/+2M3/q3ZKvl2zkzEfH88Z3K3T1L0kp38RvZhfmLsBkoBMwA/Bgm0hSSU8z+ndtxGeDu3NcnfLc/d4crho2mZWbdoUdmkiRyvfmrpkd7qre3f362IT0n3RzV4rbwYPOiCkreOCThRx05w9nteCakxtqsndJKIWq1RM2JX4JS/aW3dzz3hzGL9pAx4aVePCi1jSuVvbIHxSJA4WZgask0J/IvLslc7fril9Shbvz7vRs7v9oHntzDnJHz+b079qIDE33KHGuMMM5XwVqAmcB44C6RGbgEkkJZsbF7evyxR2n0L15NR4YvZCLnv2a79fqn4EkpmgSf9NgZM9Odx8OnAucFNuwROJP9fIlGXJ1e568/ARWbt5N7ycn8MQXP2i6R0k40ST+3FKGW8ysFZECbdVjF5JI/DIzzmtTmzG3d6dXq1o89sUiej0+gXGLNoQdmkjUokn8Q4IHt/4EfAjMJ1KaWSRlVSlbgicvP4Hh158IQL8Xv+N3r00je4se/JL4p1E9IoW0N+cAQyf8yJNf/oBh3HpaU27o1ogSGelhhyYprsA3d82shpkNM7PRweuWZtY/FkGKJKISGenccmrT4OZvVR7+7HvOfnwC49X9I3Eqmq6el4HPgNrB60XA4FgFJJKo6lYqzfNXd+Cl6zpy0J1rXvyOm1+fxmp1/0iciSbxV3X3t4CDAO6eAxRqGIOZVTSzd8xsoZktMLOTC3M8kXhyaovqfDq4O3ee2ZwvF67n9EfG8czYxezLORh2aCJAdIl/p5lVAXJr8ncCthbyvE8An7r7MUSmclxQyOOJxJWSmencelozxtx+Ct2aVeWhT7+n1xOa8F3iQzSJ/w4io3mamNkk4BXgtoKe0MwqAN0J6vy7+z5331LQ44nEs3qVSzPkmg68dG1HDhx0rho2mVten66yzxKqqEb1mFkG0AIwIrNv7T/CRw53rLbAECLDQtsQmc5xkLvvPGS/AcAAgPr167dfvnx5QU8pEhf27D/AkPFLefqrxaSnGQNPb8b1XRqRlaHSDxIbR12r50ill939vQIG0gH4Fuji7pPN7AlgW/B08K/ScE5JJis37eL+UfMZM38dTaqV4f4+rejStGrYYUkSyi/xZxzmM+cd5j0HCpT4gVXAKnefHLx+B7i7gMcSSTj1KpfmhWs68OXCdfzlw/lcOXQy57auxb3nHkutCqXCDk9SQL6J392vi8UJ3X2tma00sxbu/j1wOpFuH5GUctoxNejcpCrPj1vKM2MX89XC9Qw6vRnXqftHYiyUJ3eDfv6hQBaRyduvc/fN+e2vrh5Jdis37eK+j+bzxYJ1NK1elvvPP47O6v6RQtJELCIJ4N8L1nHfR/NZsWkXvVvX4t5zW1KzQskjf1DkVxSmHr+IFJPTj63B57d35/YzmjNm/jpOf2QsQ8YvYf8BPfwlRSeaWj3TzOyWoEKniMRYycx0Bp0Refjr5CZV+J9PFnLOExP4eoke/pKiEc0V/2+I1OmZYmZvmNlZZqYZp0VirH6V0gzt15Gh13RgT84BrnhhMreNmMHarXvCDk0SXNR9/GaWBvQGniVSq+cl4Al33xS78CLUxy+pbs/+Azw7dgnPjltCZpox+IzmXNulIZma91cOo1B9/GbWGngEeBh4F7gE2AZ8WZRBisivK5mZzu09mzPm9u6c1LgKf/9kgbp/pMCi6uMHHgOmAK3dfaC7T3b3R4gMxRSRYtKgShlevDbS/bN7f6T7Z+CIGazbpu4fid4Ru3rMrLG7h5rg1dUj8p/27D/AM2OX8FzQ/XN7z+b066zuH/k/henqucHMKuY5UCUz+1uRRiciR61kZjp3BN0/JzaqzN8+XsC5/5rAt0t/Cjs0iXPRJP6z85ZNDp6wPSd2IYnI0cjt/nnhmg7s3HuAy4Z8y+A3ZrB1d4GL6EqSiybxp5tZidwXZlYKKHGY/UWkmJkZPVvW4Is7TmHgaU0ZNXsNfZ6ayMK128IOTeJQNIn/deDfZtY/mGR9DDA8tmGJSEGUykrnjjNb8OZNndi17wB9n57EBzOzww5L4swRE7+7Pwj8HTg2WP7q7g/FOjARKbj2DSozamBXWtetyKA3ZvKXD+dpzl/52eHq8f/M3UcDo2Mci4gUoerlSvL6DSfx4OiFDJ34I3Oyt/LMle2oUV5F31JdNOP4O5nZFDPbYWb7zOyAmanjUCQBZKancW/vljx1xQksWLONc/81kcka9ZPyounjfwq4HPgBKAXcADwdy6BEpGj1bl2bD27pQvlSGVwxdDJDJywlEUqyS2xE9aSHuy8G0t39gLu/BPSKbVgiUtSa1SjHB7d0oeexNfjbxwu4dcQMdu7NCTssCUE0iX+XmWUBM83sITO7PcrPiUicKVcyk2evasc9Zx/D6Dlr6PP0JJZs2BF2WFLMokngVwf73QrsBOoBF8UyKBGJHTPjplOa8Fr/k9i8cx99nprEp3PXhB2WFKNoEn97wN19m7vf5+53BF0/IpLAOjetyke3daVp9bL89rXpPDB6ATma6SslRJP4zwMWmdmrZtbbzKIaAioi8a92xVK8eVMnrupUn+fHLeXqYd+xccfesMOSGIvmAa7rgKbA20RG9ywxs6GxDkxEikeJjHT+1vd4/nlJG6av2Mx5T05kxorNYYclMRTtqJ79RB7gegOYBvQt7InNLN3MZpjZqMIeS0QK7+L2dXnv5s5kpBuXPv8Nr327XEM+k1Q0D3CdbWYvExnHfxEwFKhZBOceBCwoguOISBE5rnYFRt3aja5Nq3LvyLnc+fZs9uw/EHZYUsSiueK/BhgJtHD3a939E3cv1OBfM6sLnEvkPxERiSMVSmcyrF9HBp/RjPdmrOLCZ75mxU+7wg5LilA0ffyXu/tIdy/KOz6PA38A8h1CYGYDzGyqmU3dsGFDEZ5aRI4kLZjQ/cVrO5K9ZTe9n5zAVwvXhx2WFJF8E7+ZTQx+bjezbXmW7YWp1WNmvYH17j7tcPu5+xB37+DuHapVq1bQ04lIIZzaojof3dqVupVKc/3wKTw2ZhEHD6rfP9Hlm/jdvWvws5y7l8+zlHP38oU4ZxfgfDNbRuRm8Wlm9lohjiciMVS/Smneu7kzF55Qlyf+/QPXD5/Cll37wg5LCuGwXT3ByJuFRXlCd7/H3eu6e0PgMuBLd7+qKM8hIkWrZGY6/7ykNX+/oBWTFm/kvKcmMjd7a9hhSQEdNvG7+wHgezOrX0zxiEicMjOuPKkBb910MjkHnIue/Zp3pq0KOywpgGhG9VQC5pnZv83sw9ylKE7u7mPdvXdRHEtEiscJ9Ssx6rautG9QiTvfnsUf35/D3hwN+Uwk0ZRf+FPMoxCRhFKlbAleuf5EHhmziGfHLmHu6m08e2U7alcsFXZoEoXDjeopaWaDgUuAY4BJ7j4udym2CEUkLmWkp3FXr2N47qr2LFm/g95PTmTS4o1hhyVROFxXz3CgAzAHOBt4pFgiEpGE0qtVTT64tQtVymRx9bDJPDt2iUo9xLnDJf6W7n6Vuz8PXAx0K6aYRCTBNKlWlpG3dOGc42vx4KcL+e1r09i+Z3/YYUk+Dpf4f/5bK2yJBhFJfmVKZPDk5Sfwp94t+WLBevo8NYlF67aHHZb8isMl/jZ5n9YFWhfFk7sikrzMjP5dGzHixk5s25ND36cn8dGs1WGHJYc43JO76Yc8rZtRRE/uikiSO7FRZT4e2JWWtcpz24gZ/HXUfPZrdq+4oUnTRSQmapQvyYgBnbi2c0OGTfyRK1+YzPrte8IOS1DiF5EYykxP4y/nH8cTl7VlTvZWzv3XRD6Yma1RPyFT4heRmOvTtg7v39KZGuVLMOiNmVzwzNdMW74p7LBSlhK/iBSLY2qW58NbuvLPS9qwZutuLnr2G2753+ms3KRJXoqbEr+IFJu0NOPi9nX56s4eDD6jGV8uWM/pj4zjgdEL2KZx/8VGiV9Eil3prAwGn9Gcr+7swXltavP8uKWc+vBYXvt2OTka/RNzSvwiEpqaFUryyKVt+OjWrjSpXpZ7R87lnH9NYNwiTbcaS0r8IhK64+tW4M0BnXjuqvbszTlIvxe/o9+L3/GDnvyNCSV+EYkLZkavVjUZc/sp3HvusUxfsZleT0zg3pFz+GnH3rDDSypK/CISV7Iy0rihW2PG/f5Uru7UgBHfraTHw2N5ftwSTfhSRJT4RSQuVS6TxV/OP47PBnfnxEaVeWD0Qs54dBwfz16jB8AKSYlfROJa0+plGXZtR17rfxJlsjK45X+nc8lz3zBz5ZawQ0tYSvwikhC6NqvKxwO78eBFx7Psp130fXoSg9+YQfaW3WGHlnCU+EUkYaSnGb/pWJ+xv+/Brac2ZfTctZz2z7E88vn37NyraUOiVeyJ38zqmdlXZjbfzOaZ2aDijkFEElvZEhnceVYLvryzB71a1eTJLxfT459jeXPKCg4cVP//kYRxxZ8D/Je7twQ6AbeYWcsQ4hCRBFenYimeuOwE3r+5M/Url+aud+fQ+8mJfK1J3w+r2BO/u69x9+nB+nZgAVCnuOMQkeRxQv1KvPPbk3nqihPYvmc/V35ARjcAAAvHSURBVAydzA3Dp7Bkw46wQ4tLFuawKDNrCIwHWrn7tkPeGwAMAKhfv3775cuXF3t8IpJ49uw/wEuTlvH0V4vZs/8AV3VqwKDTm1GpTFbYoRU7M5vm7h3+Y3tYid/MygLjgL+7+3uH27dDhw4+derU4glMRJLCxh17eWzMIkZ8t4JyJTMZeHozru7UgKyM1BnTkl/iD+VPwMwygXeB14+U9EVECqJq2RL8/YLj+XRwd9rUq8hfR83nzMfG8dm8tSn/AFgYo3oMGAYscPdHi/v8IpJamtcoxyvXn8jL13UkMz2Nm16dxuUvfMvc7K1hhxaaMK74uwBXA6eZ2cxgOSeEOEQkhfRoUZ3Rg7rx176tWLRuB+c9NZE7357Fum2pNwF8qDd3o6U+fhEpStv27Ofprxbz0sRlpKcZN53SmCtPakC1ciXCDq1Ixd3N3aOhxC8isbDip108+OlCPp6zhvQ0o0vTqvRtW5szj6tJ2RIZYYdXaEr8IiL5WLx+OyNnrGbkzGxWbd5Nycw0erasSd+2tenevBqZ6Yk5EkiJX0TkCNydacs3M3JmNh/PXsPmXfupXCaLc4+vRd8TatOufiUi41MSgxK/iMhR2JdzkPGLNjByZjZj5q9jb85B6lUuRZ82deh7Qm2aVi8XdohHpMQvIlJAO/bm8NnctYycmc2kxRs56NCqTnn6tq3DeW1qU6N8ybBD/FVK/CIiRWD99j18NGsNH8zMZvaqraQZdG5SlT5ta9OrVU3KlcwMO8SfKfGLiBSxJRt28MGMbEbOXM2KTbsokZHGGcfWoE/b2vRoUT308hBK/CIiMeLuzFi5hQ9mZDNq9hp+2rmPiqUzOef4WvRtW4cODSqRllb8N4WV+EVEisH+AweZ+MNGRs7M5vN569i9/wB1Kpbi/La1ueCEOjSvUXw3hZX4RUSK2c69OYyZv46RM7OZ8MNGDhx0jq1Vnr5ta3N+29rUqlAqpudX4hcRCdHGHXsZNWs1I2euZubKLZjBSY0qc8EJdejVqhYVShX9TWElfhGROPHjxp18MDObD2au5seNO8nKSOO0FtXpe0JtTj2mOiUy0ovkPEr8IiJxxt2ZvWorI2dm89Gs1WzcsY9yJTM49/ha9Glbh5MaVS7UTWElfhGROJZz4CCTlvzEBzOy+WzeWnbuO0CtCiV55JI2dG5atUDHzC/xJ375ORGRJJCRnsYpzatxSvNq7N53gDEL1vHBjGzqVS5d9Ocq8iOKiEihlMpK5/w2tTm/Te2YHD8xa42KiEiBKfGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIilHiFxFJMUr8IiIpJiFKNpjZBmB5AT9eFdhYhOHEE7UtMSVr25K1XZC4bWvg7tUO3ZgQib8wzGzqr9WqSAZqW2JK1rYla7sg+dqmrh4RkRSjxC8ikmJSIfEPCTuAGFLbElOyti1Z2wVJ1rak7+MXEZFfSoUrfhERyUOJX0QkxSR14jezXmb2vZktNrO7w44nGmb2opmtN7O5ebZVNrMxZvZD8LNSsN3M7F9B+2abWbs8n+kX7P+DmfULoy15mVk9M/vKzOab2TwzGxRsT4a2lTSz78xsVtC2+4LtjcxsctCGN80sK9heIni9OHi/YZ5j3RNs/97MzgqnRb9kZulmNsPMRgWvk6Vdy8xsjpnNNLOpwbaE/z5Gxd2TcgHSgSVAYyALmAW0DDuuKOLuDrQD5ubZ9hBwd7B+N/BgsH4OMBowoBMwOdheGVga/KwUrFcKuV21gHbBejlgEdAySdpmQNlgPROYHMT8FnBZsP054HfB+s3Ac8H6ZcCbwXrL4HtaAmgUfH/T4+A7eQfwv8Co4HWytGsZUPWQbQn/fYxmSeYr/hOBxe6+1N33AW8AfUKO6YjcfTyw6ZDNfYDhwfpwoG+e7a94xLdARTOrBZwFjHH3Te6+GRgD9Ip99Plz9zXuPj1Y3w4sAOqQHG1zd98RvMwMFgdOA94Jth/attw2vwOcbmYWbH/D3fe6+4/AYiLf49CYWV3gXGBo8NpIgnYdRsJ/H6ORzIm/DrAyz+tVwbZEVMPd1wTra4EawXp+bYzrtgddACcQuTJOirYF3SEzgfVE/vEvAba4e06wS944f25D8P5WoArx2bbHgT8AB4PXVUiOdkHkP+fPzWyamQ0ItiXF9/FINNl6gnF3N7OEHYNrZmWBd4HB7r4tckEYkchtc/cDQFszqwi8DxwTckiFZma9gfXuPs3MeoQdTwx0dfdsM6sOjDGzhXnfTOTv45Ek8xV/NlAvz+u6wbZEtC74tZLg5/pge35tjMu2m1kmkaT/uru/F2xOirblcvctwFfAyUS6A3IvrvLG+XMbgvcrAD8Rf23rApxvZsuIdJWeBjxB4rcLAHfPDn6uJ/Kf9Ykk2fcxP8mc+KcAzYIRCFlEbjZ9GHJMBfUhkDtaoB/wQZ7t1wQjDjoBW4NfUz8DzjSzSsGohDODbaEJ+nqHAQvc/dE8byVD26oFV/qYWSmgJ5F7GF8BFwe7Hdq23DZfDHzpkTuFHwKXBaNjGgHNgO+KpxX/yd3vcfe67t6QyL+fL939ShK8XQBmVsbMyuWuE/kezSUJvo9RCfvuciwXInfiFxHpb/1j2PFEGfMIYA2wn0h/YX8i/aT/Bn4AvgAqB/sa8HTQvjlAhzzHuZ7ITbTFwHVx0K6uRPpUZwMzg+WcJGlba2BG0La5wJ+D7Y2JJLjFwNtAiWB7yeD14uD9xnmO9cegzd8DZ4fdtjxx9eD/RvUkfLuCNswKlnm5+SEZvo/RLCrZICKSYpK5q0dERH6FEr+ISIpR4hcRSTFK/CIiKUaJX0QkxSjxS9wxMzezR/K8vtPM/lJEx37ZzC4+8p6FPs8lZrbAzL462rjMbKiZtSzgeXuYWeeCfFZShxK/xKO9wIVmVjXsQPLK87RqNPoDN7r7qUd7Hne/wd3nH+3nAj0AJX45LCV+iUc5ROY4vf3QNw69YjezHcHPHmY2zsw+MLOlZvYPM7vSInXy55hZkzyHOcPMpprZoqAeTW6RtYfNbEpQb/2mPMedYGYfAv+RjM3s8uD4c83swWDbn4k8sDbMzB7+lc/cFXxmlpn941feH2tmHYL1M83sGzObbmZvB7WOcmvJ3xdsn2NmxwTF734L3G6RGvPdgt885gbnGh/dH78kOxVpk3j1NDDbzB46is+0AY4lUtZ6KTDU3U+0yKQvtwGDg/0aEqnL0gT4ysyaAtcQeQy/o5mVACaZ2efB/u2AVh4pKfwzM6sNPAi0BzYTqfTY193vN7PTgDvdfeohnzmbSInfk9x9l5lVzq8xwW889wJnuPtOM7uLSG38+4NdNrp7OzO7OTjXDWb2HLDD3f8ZHGMOcJZHipFVPIo/S0liuuKXuOTu24BXgIFH8bEpHqn7v5fIo/W5iXsOkWSf6y13P+juPxD5D+IYIjVWrrFIaeXJRB7dbxbs/92hST/QERjr7hs8Uob4dSIT6RzOGcBL7r4raOehcy/k1YnIJCaTgrj6AQ3yvJ9b6G7aIe3LaxLwspndSGRyIhFd8UtcexyYDryUZ1sOwQWLmaURmV0t19486wfzvD7IL7/rh9YpcSK1WG5z918U2LJIOeKdBQu/0IzIJB+X5/N+bvsOkM+/ZXf/rZmdRGQylWlm1t7dfyr6UCWR6Ipf4lZwNfwWkRuluZYR6VoBOJ/IbFdH6xIzSwv6/RsTKRz2GfA7i5SOxsyaB1UbD+c74BQzq2pm6cDlwLgjfGYMcJ2ZlQ7Ok29XD/At0CXoisqtKNn8CMffTmRqS4LPNHH3ye7+Z2ADvywhLClKiV/i3SNA3tE9LxBJtrOI1LwvyNX4CiJJezTwW3ffQ2RqwfnAdItMdP88R/iN2CNlee8mUqZ4FjDN3T84wmc+JVLid2rQfXPnYfbdAFwLjDCz2cA3HHmCl4+AC3Jv7gIP5958Br4O4pQUp+qcIiIpRlf8IiIpRolfRCTFKPGLiKQYJX4RkRSjxC8ikmKU+EVEUowSv4hIivn/qyWYxSu6saAAAAAASUVORK5CYII=\n"
187 |           },
188 |           "metadata": {
189 |             "needs_background": "light"
190 |           }
191 |         }
192 |       ]
193 |     },
194 |     {
195 |       "cell_type": "markdown",
196 |       "source": [
197 |         "# Fixed UK=50000"
198 |       ],
199 |       "metadata": {
200 |         "id": "2eQs7BvRBM9Z"
201 |       }
202 |     },
203 |     {
204 |       "cell_type": "code",
205 |       "source": [
206 |         "UK = 50000  # 50000\n",
207 |         "N = 10\n",
208 |         "\n",
209 |         "for K in [9]:  # [8]\n",
210 |         "  U = int(UK / K)\n",
211 |         "  B = f(N+K-1) // f(K) // f(N-1)\n",
212 |         "  print(\"K, U, B:\", K, U, B)\n",
213 |         "\n",
214 |         "  # calculate H(B|Y=y), y is uniform\n",
215 |         "  p_y = 0\n",
216 |         "  P_ys = []\n",
217 |         "  b_count = 0\n",
218 |         "  H_y = 0\n",
219 |         "  for comb in itertools.combinations(np.arange(N+K-1), N-1):\n",
220 |         "    b_count += 1\n",
221 |         "    p_y = f(K) / (N**K) / f(comb[0]) / f(N+K-2-comb[-1])\n",
222 |         "    for n in range(1, N-1):\n",
223 |         "      p_y = p_y / f(comb[n]-comb[n-1]-1)\n",
224 |         "    P_ys.append(p_y)\n",
225 |         "    H_y += -p_y * math.log2(p_y)\n",
226 |         "\n",
227 |         "  # print('len(P_ys)', len(P_ys))\n",
228 |         "  # print('sum(P_ys)', sum(P_ys))\n",
229 |         "  # print('b_count', b_count)\n",
230 |         "  # print('B', B)\n",
231 |         "  # print('math.log2(B)', math.log2(B))\n",
232 |         "  print('H_y', H_y)\n",
233 |         "  # print('2**H_y', 2**H_y)\n",
234 |         "\n",
235 |         "  # calculate H(B|S=s)\n",
236 |         "  H_s = 0\n",
237 |         "  p_s = 0\n",
238 |         "  choices = np.random.choice(B, U, p=P_ys)\n",
239 |         "  # print(len(choices), choices)\n",
240 |         "  start_time = time.time()\n",
241 |         "  for u in range(U):\n",
242 |         "    if u%10000==0:\n",
243 |         "      end_time = time.time()\n",
244 |         "      print('u:', u, '/', U, ', H_s:', H_s, ', time:', round((end_time-start_time)/60))\n",
245 |         "    p_s = sum([1 for c in choices if c == u]) / U\n",
246 |         "    # print(p_s)\n",
247 |         "    if p_s != 0:\n",
248 |         "      H_s += -p_s * math.log2(p_s)\n",
249 |         "  print('H_s', H_s)"
250 |       ],
251 |       "metadata": {
252 |         "id": "I26xJAKI87vq",
253 |         "colab": {
254 |           "base_uri": "https://localhost:8080/"
255 |         },
256 |         "outputId": "abc10e1d-2476-4108-f9ad-883f2e6fbaca"
257 |       },
258 |       "execution_count": null,
259 |       "outputs": [
260 |         {
261 |           "output_type": "stream",
262 |           "name": "stdout",
263 |           "text": [
264 |             "K, U, B: 9 5555 48620\n",
265 |             "H_y 14.7084540600815\n",
266 |             "u: 0 / 5555 , H_s: 0 , time: 0\n",
267 |             "H_s 0.5174473607079045\n"
268 |           ]
269 |         }
270 |       ]
271 |     },
272 |     {
273 |       "cell_type": "code",
274 |       "source": [
275 |         "Ks = [1, 2, 4, 6, 7, 8, 9, 10, 16]\n",
276 |         "Ks_H_y = [3.321928, 5.743856, 9.286393, 11.8602849, 12.91726, 13.860046, 14.708454, 15.4776598, 18.95812]\n",
277 |         "K_s_H_s_UK100000 = [3.321874, 5.74309, 9.25941, 11.64429, 12.33949, 5.448149, 1.7305256, 0.505497, 0]\n",
278 |         "K_s_H_s_UK50000 = [3.321928, 5.738995, 9.2478531, 11.399456, 6.4285309, 1.970087, 0.517447, 0.17900, 0]\n",
279 |         "K_s_H_s_UK5000 = [3.318592, 5.73230, 8.81008, 1.091727, 0.146047, 0.056241, 0, 0, 0]\n",
280 |         "\n",
281 |         "# plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|Y=y_0)}$')\n",
282 |         "# plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|S=s), UK=5000}$')\n",
283 |         "# plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|S=s), UK=50000}$')\n",
284 |         "# plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|S=s), UK=100000}$')\n",
285 |         "plt.plot(Ks, Ks_H_y, label='$\\mathregular{H^a(B|P_{y,uni})}$')\n",
286 |         "plt.plot(Ks, K_s_H_s_UK5000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=5000}$')\n",
287 |         "plt.plot(Ks, K_s_H_s_UK50000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=50000}$')\n",
288 |         "plt.plot(Ks, K_s_H_s_UK100000, label='$\\mathregular{H^a(B|\\mathcal{C}_U=c_U), UK=100000}$')\n",
289 |         "plt.xlabel('Number of samples K from each client')\n",
290 |         "plt.ylabel('Lable Privacy information (in bits)')\n",
291 |         "plt.legend()\n",
292 |         "plt.show()"
293 |       ],
294 |       "metadata": {
295 |         "id": "befatQox87ql",
296 |         "colab": {
297 |           "base_uri": "https://localhost:8080/",
298 |           "height": 279
299 |         },
300 |         "outputId": "b202ec23-d599-42a4-a941-0ee1804dbff2"
301 |       },
302 |       "execution_count": null,
303 |       "outputs": [
304 |         {
305 |           "output_type": "display_data",
306 |           "data": {
307 |             "text/plain": [
308 |               "<Figure size 432x288 with 1 Axes>"
309 |             ],
310 |             "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1hUV/rA8e+hKyCiFAsqKgpYUYmJRlDBHrvRaDSJm2KKKcbkl76bstlssqkmpidqitEklo09xt4LJhqNYMOGBQSU3nl/f8zAolIGmGEYPJ/nuQ8zd+4993WXcObce877KhFB0zRN065lZ+0ANE3TtNpJdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappXKwdoBmJOXl5f4+/tbOwxN0zSbsW/fvkQR8S7tszrVQfj7+xMVFWXtMDRN02yGUup0WZ/pW0yapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmappVKdxCapmlaqXQHoWmaZqNSMvNYduA8n20+YZH269RCOU3TtLpMRIhNzGBDdALrouOJOn2ZgkKhqYcL9/dpjYO9eb/z6w5C0zStFsvNLyTqVDLrohPYEBPPqaRMAIKauPNQ3zZEBPkS0qIh9nbK7NfWHYSmaVotk5yRy6YjCayPTmDL0Uuk5eTj5GBH77aNua9Pa/oH+eDnWd/icegOQtM0zcpEhKPx6ayPiWd9dAK/n7mMCHi7O3Nbl6ZEBPlwa4AXrs41+ydbdxCapmlWkJNfwK7YZDZEx7M+JoG4y1kAdG7uweMR7YgM9qFTMw/sLHDryFS6g9A0TashCWnZbIq5xPqYeLYeSyQztwAXRzv6BHgzvX8A/QN9aOLhYu0wi+kOQtM0zUJEhL/Op7IhJoH10fEciEsBoJmHC2O7NycyyJdebRvj4mhv5UhLpzsITdM0M8rKLWDHiUTWxySwITqBi6nZKAVd/Rry9KD2RAT5EtzUHaWsd+vIVLqD0DRNq6YLKVlsMHYI244nkpNfiKuTPeHtvYkI8qFfoA/e7s7WDrPSdAehaZpWSYWFwp/nUtgQHc+66AQOX0gFoEWjekzq2ZLIYB96tm6Es0PtvHVkKt1BaJqmmSAjJ5+txxLZEBPPhphLJKbnYKegRytPnhsaRGSQDwE+bjZx68hUuoPQNE0rw9nkTMMD5pgEdp1IIregEHcXB/oF+hAZ5EPf9t54ujpZO0yLsVgHoZSaAwwHEkSkk3Hfj0Cg8ZCGwBURCSnl3FNAGlAA5ItIqKXi1DRNK1JQKPxx5nLxA+Yj8WkAtPFy5Z7erYgI8iXU3xNHM+c8qq0sOYKYB8wGvi3aISJ3FL1WSr0LpJRzfn8RSbRYdJqmaUBqdh5bjl5iQ3QCG48kcDkzDwc7Rc/WjXgpNJiIIB/aeLtZO0yrsFgHISJblFL+pX2mDDfpJgARlrq+pmlaWU4mZrA+Op4NMQnsOZlMfqHgWd+R/oE+RAT7ENbOG496jtYO0+qs9QwiDIgXkWNlfC7AWqWUAJ+LyBdlNaSUmgZMA2jZsqXZA9U0zfblFRQSdeoyG2IMaS1iL2UA0N7XjQfC2xAZ5EO3lp4WyYhqy6zVQUwCFpTzeR8ROaeU8gF+U0rFiMiW0g40dh5fAISGhor5Q9U0zRZdzshl89FLrI9JYPORBFKz83Gyt+OWto25p5c/EUE+tGhk+YyotqzcDkIp5QdMxPCNvxmQBRwCVgKrRaSwshdUSjkAY4EeZR0jIueMPxOUUkuBnkCpHYSmaRoY0locT0gvfsAcdTqZQgEvNycGd2xCZLAvfdp54VbDGVFtWZn/Syml5gLNgRXAW0AC4AK0B4YALyqlnivrm305BgAxIhJXxnVdATsRSTO+HgS8VslraJp2A8jJL2DPyWTWRyewISaBM8mGYjodmjbg0f4BRAT70qW5dTOi2rLyutJ3ReRQKfsPAUuUUk5AmTf9lVILgH6Al1IqDnhZRL7GMCJZcM2xzYCvRGQY4AssNS42cQB+EJE1pv+TNE2ryxLTc9gYY+gQthy9REZuAc4Odtwa4MWDfdsQEeRDU4961g6zTlAipt+2V0p5Ai1E5E/LhVR1oaGhEhUVZe0wNE0zIxEh+kJa8QPm/WevIAK+DZyJCPJlQLAPvdt6Uc/JttNaWItSal9Za80qvBmnlNoEjDQeuw9IUErtEJEnzRqlpmmaUXZeATtPJLE+Jp4N0QmcT8kGoKufB08OaE9EkA8dmzWoU2ktaiNTntZ4iEiqUup+4FsReVkpVStHEJqm2a741Gxj3YQEth9PJCuvgPpO9vQJ8GLGgPb0C/LGx732FNO5EZjSQTgopZpiWNj2ooXj0TTtBlFYKBw6n1L8gPngOUNiheYN6zEh1I+IYF9ubt2o1hbTuRGY0kG8CvwKbBORvUqpNkBZC9w0TdPKlJmbz7ZjiYbaCTEJJKTloBR0b+nJM0MCiQzypb1v3cqIastM6SAuiEiXojciEquUes+CMWmaVoecu5LFhmjDA+YdJ5LIzS/E3dmB8PbeRAYbiuk0qsMZUW2ZKR3ER0B3E/ZpmqZRUCjsP3vFMOsoOoGYi4aMqP6N63PXLa2IDPIh1L8RTg43RkZUW1beQrleQG/AWyk1s8RHDQB9U1DTtGJp2XlsPZbI+ugENh1JICkjF3s7RWgrT14cFkxEsA9tvFz1rSMbU94IwglwMx7jXmJ/KnC7JYPSNK32O5OUyTpjRtTdJ5PIKxA86jnSL9CbyGBf+rbzxqO+zohqy8rsIERkM7BZKTVPRE7XYEyaptVC+QWF/H7mCuuNzxOOJ6QDEODjxr23tiYy2JfuLRvicIMU07kRlHeL6QMRmQHMNqbdvoqIjLRoZJqmWV1KZh6bjhpmHG06comUrDwc7RU3t27MnT1bEhnsQ6vGrtYOU7OQ8m4xfWf8+U5NBKJpmvWJCLHGYjrroxOIOn2ZgkKhkasTA4INaS36tPPC3UXfOroRlHeLaZ/x52ZjYr4gDIV8johIbg3Fp2maheXmF7L3VFFG1HhOJRkyogY1ceehvm2IDPalq19DXUznBmRKLqbbgM+AE4ACWiulHhSR1ZYOTtM0y0jOyL0qI2paTj5ODnb0btuY+8IMGVGbN9QZUW90pqyDeBfoLyLHAZRSbTEWDLJkYJqmmY+IcDQ+vXjW0e9nLiMC3u7O3NalKZHBvtwa0Jj6TrqYjvY/pvw2pBV1DkaxQJqF4tE0zUzyCgrZcSKp+HnCuStZAHRu7sETke2IDPKlY7MGupiOVqbyZjGNNb6MUkqtAn7C8AxiPLC3BmLTNK0KziRlsnDvGX7eF8eltBxcHO3oE+DNoxEBRAT54NtAZ0TVTFPeCGJEidfxQF/j60sYSo9qmlZL5OYXsvbwRRbuOcu244nYKegf6MOEm1rQt723zoiqVUl5s5j+Vp2GlVJzgOFAgoh0Mu57BXgAQycD8IKIrCrl3CHALAwpPb4SkTerE4um1VUnLqWzcM8ZFv9+juSMXJo3rMfMge0ZH+qny25q1WbJJ1LzgNnAt9fsf19EylxboZSyBz4GBgJxwF6l1DIROWypQDXNlmTnFbD60AUW7DnLnpPJONgpBgT7MunmlvQJ8NLTUTWzsVgHISJblFL+VTi1J3BcRGIBlFILgVGA7iC0G1rMxVQW7jnL0j/OkZKVh3/j+jw7JIhxPZrrSmuaRVhjTtujSqm7gSjgKRG5fM3nzYGzJd7HATfXVHCaVptk5uaz4sAFFuw9wx9nruBkb8fgTk2YdFMLbmnTWM9A0izKlIVyzsA4wL/k8SLyWhWu9ynwTwyzof6JYY3FvVVop2R804BpAC1btqxOU5pWaxw6l8KCPWf4Zf950nPyCfBx46Xbghnb3U8X19FqjCkjiF+AFGAfkFOdi4lIfNFrpdSXwIpSDjsHtCjx3s+4r6w2vwC+AAgNDb0uqaCm2Yr0nHz++8c5Fu49w6FzqTg72HFbl6ZM6tmS0FaeupaCVuNM6SD8RGSIOS6mlGoqIheMb8cAh0o5bC/QTinVGkPHMBG40xzX17Ta6ExSJvN2nOKnqLOk5+QT1MSd10Z1ZFRIczzq6aR4mvWY0kHsUEp1FpGDlWlYKbUA6Ad4KaXigJeBfkqpEAy3mE4BDxqPbYZhOuswEclXSj0K/IphmuscEfmrMtfWtNpORNgVm8yc7SdZFx2PvVIM79KUe3r7E9KioR4taLWCEin/roxS6jAQAJzEcItJASIiXSwfXuWEhoZKVFSUtcPQtDJl5xWw7MB55mw7SczFNBq5OjH55pZMuaWVXuGsWYVSap+IhJb2mSkjiKFmjkfTbjgJqdl8v+s083efISkjl6Am7rw1rjOjQprrVc5arVVeLqYGIpKKTsynaVV2MC6FOdtPsuLP8+QXCpFBvtx7qz+92jbWt5G0Wq+8EcQPGFJl7MPwzKDkb7MAbSwYl6bZrPyCQtYejmfOtpNEnb6Mq5M9U25pxT29/PH30uU5NdtRXi6m4cafrWsuHE2zXSmZeSzce4Zvd57m3JUsWjaqz9+Hd2B8qB8NdIlOzQaVd4vJX0ROlfO5ApqLSJwlAtM0W3E8IZ15O06yeN85svIKuKVNI14e0YHIYF+dF0mzaeXdYnpbKWWHYaHcPv6X5jsA6A9EYpi6qjsI7YYjImw9lsjX206y+eglnOztGBXSjL/d2poOzRpYOzxNM4vybjGNV0p1ACZjSIfRFMgEooFVwL9EJLtGotS0WqKwUFgXHc/sjcf5My4Fb3dnZg5sz503t8TLzdna4WmaWZU7zdWYYvvFGopF02qtgkJh1cELfLzxODEX02jZqD5vju3MmO7NcXbQ01S1uklXKNe0cuQVFPLL/vN8svE4sYkZtPV25f07ujKiSzMc7O2sHZ6mWZTuIDStFDn5BSzaF8enm04QdzmL4KYN+GRydwZ3bKIfPGs3DN1BaFoJWbkFLNhzhi+2xHIxNZuuLRryyoiORAb76IVt2g3HpA5CKdUcaMXV9SC2WCooTatp6Tn5fLfzNF9tjSUpI5eerRvxzviu3BqgVzxrNy5TCga9BdyBoeRngXG3ALqD0GxeSmYec3ecZO72U6Rk5RHe3ptH+wfQs3Ujk9vIy8sjLi6O7Gw9qU+rvVxcXPDz88PR0fRFm6aMIEYDgSJSrWJBmlabJKbn8PW2k3y38zTpOfkMCPblsYgAurZoWOm24uLicHd3x9/fX482tFpJREhKSiIuLo7WrU1PjmFKBxELOFLNanKaVhvEp2bz+eZYfthzmpz8QoZ1bsr0fgHVWtyWnZ2tOwetVlNK0bhxYy5dulSp80zpIDKB/Uqp9ZToJETk8cqFqGnWczElm9kbj/HT3jgKRBgV0oxH+gUQ4ONmlvZ156DVdlX5HTWlg1hm3DTN5qRm5/HZphPM2X6SgkLh9h4teLhvW1o2rm/t0DSt1quwgxCRb5RSTkB7464jIpJX0XlKqTkY0oUniEgn4763gRFALnAC+JuIXCnl3FMY6lAUAPllVTvStLLk5Bcwf9cZPtpwjMuZeYzs2oynBwXqjkHTKqHCpaBKqX7AMeBj4BPgqFIq3IS25wFDrtn3G9DJWK70KPB8Oef3F5EQ3TlolVFYKPyy/xwD3tvMaysO06FZA5Y/2ocPJ3Wr852Dm9vVt8vmzZvHo48+yl9//cULL7zAfffdx86dO4s/z8rKom/fvhQUGCYn2tvbExISQteuXenevTs7duwAoF+/fpw6dar4vKLjOnXqxPjx48nMzDTrvyM3N5fw8HDy8/PN2q5WeabkCngXGCQifUUkHBgMvF/RScZ1EsnX7FsrIkX/r+8C/CoZr6aVafvxREZ9vJ0nFu7HzdmRb+7tyff33UxnPw9rh2ZVTk5OZGdn4+vry3fffVe8f86cOYwdOxZ7e0MuqXr16rF//34OHDjAv//9b55/vvTvb0XHHTp0CCcnJz777DOzxxsZGcmPP/5o1na1yjOlg3AUkSNFb0TkKIZZTdV1L7C6jM8EWKuU2qeUmmaGa2l12OHzqdw9Zw+Tv9pNckYu703oysrH+tC3vbd+eAx8+OGHzJgxgwcffPCqb/vz589n1KhRpZ6TmpqKp6dnhW2HhYVx/Phxs8VaZPTo0cyfP9/s7WqVY8pD6iil1FfA98b3k4Go6lxUKfUikA+U9RvQR0TOKaV8gN+UUjFlrdw2diDTAFq2bFmdsDQbE3c5k/fWHmXp/nM0cHHkxWHB3NWrFS6O12dXzT5ylMx9UTQcNw47Z8ul5X51+V8cPp9q1jY7NGvAyyM6VnhcVlYWISEhxe+Tk5MZOXIk/fv356233sLX17f4s9zcXGJjY/H397/u/OzsbC5cuMCGDRvKvV5+fj6rV69myJBr7yRXX6dOndi7d6/Z29Uqx5QO4mFgOlA0rXUrhmcRVaKUmorh4XWkiEhpx4jIOePPBKXUUqAnZazcFpEvgC8AQkNDS21Pq1uuZOby8cbjfLPzNADTwtvwSN8APOpfPbDNu3CB1JUrSVm2nJyjRwGwc3Ki4e2313jMNaHo1k+RefPmERUVxdixYxk7duxVxyYmJtKwYcMyz9+5cyd33303hw4duu46JTuisLAw7rvvPnP/U7C3t8fJyYm0tDTc3d3N3r5mGlNmMeUA7xm3alFKDQGeAfqKSKlPtpRSroCdiKQZXw8CXqvutTXbl51XwLwdp/hk43HScvIZ192PmQPb06xhveJjClJTSf31V1KXryBz714QoV7Xrvi+9BJJX35J+tZtFu0gTPmmXxvUq1ev3NQgvXr1IjExsdSFVdd2ROaybNkyPD09CQsLAyAnJwcXFxezX0czXXk1qX8SkQlKqYMYnglcxTgTqUxKqQVAP8BLKRWHoTzp84AzhttGALtE5CGlVDPgKxEZBvgCS42fOwA/iMiaqvzjtLqhoFBY8nsc7/92lPMp2fQP9ObZoUEENTGsfi7MzSV982ZSly0nfdMmJC8PJ39/vB6djsfw4Ti1agVAdkw0ab+uRfLzUQ43diJjT09PCgoKyM7OLvWPcExMDAUFBTRu3Nik9mbOnMlTTz2Fl5cX999//1UPw8EwmvHy8mL48OFMnDiRIUOGsHnzZtq0aYOdnR0vvvgiycnJ2NkZHosmJSXh5eVVqbxBmvmV91/JE8afw6vSsIhMKmX312Ucex4YZnwdC3StyjW1ukVE2HT0Em+tjiHmYhpd/Tx4d0IIvdo2RgoLydizh9TlK0j99VcKU1Oxb9yYhpMm4jFiBC6dOl33gNqtTxgpixaTdeAA9Xv0sNK/qvYYNGgQ27ZtY8CAAcDVt45EhG+++aZ4hlNFHnnkEb744gvatGnDxIkTTTpnyJAh3HHHHUyadP2fio0bN3LbbbeZ+C/RLKW8mtQXjC8fEZFnS35mzPD67PVnaZp5RF9I5bXlh9kZm0SrxvWZfWc3buvclJxjx0h4dx4pK1aSf+ECqn593AdE4jFiJK69bil3ZODauxfY25O+dWud7CDS09Ovej916lSmTp1a5vHTp0/n/fffL+4gitZDVPY6AAEBAZw/f57o6GgWLlx43efOzs7F6xoyMjIAcHV1BQyd0bV++OEH3nzzTZPi0SzHlHH2QK7vDIaWsk/Tqi03v5BPNh1n9objNKjnyKsjOzK+pRNZa1Zx8u8ryDlyBOztce1zKz4zZ+IeGYFdfdMWwNk3aEC9kBAytmyFGTMs/C+p/bp3707//v0pKCgweaRQnsjISC5cuICdnR3x8fGsWLGi+AF23759eeaZZzh58iRXrlyXPOEqubm5jB49mvbt25d7nGZ5qoyJRCilHgYeAdpgSItRxB3YLiJTLB9e5YSGhkpUVLVm4GpWdOhcCk//fICYi2mM69CImfanyV+7hsw9e0AEl65d8BgxkgZDh+Bg4r3xayV+9hmXPphFu21bcfDyMkvc0dHRBAcHm6Wt2mjevHmMHj36ullPJW3fvp1Zs2bx9ddf4+7uzpo1a3ByciIiIqIGI9UqUtrvqlJqX1kZK8rrIDwAT+DfwHMlPkoTkeRST7Iy3UHYppz8AmZvOM4nm07QyNWJ95qn0OSbj8m/cAGnVq1oMGIEHiP+97C5OrL++otT426n2Vtv4lHGIrHKqusdhFZ3VLaDKO8ZRAqQAkwyNuIDuABuSik3ETljtqi1G9afcVd4+ucDHI1P557WTtyzbxE5323Gvl07ms2bR/2be5p1NbRLcDD2jRuTvnWb2ToITaurTCk5OgLDGohmQAKG2tTRgG1M+NZqpey8AmatP8YXW2JpUs+en9yO0ODT78i1s8Pr6afYeqsHcVm7eYyeZr2usrPDrc+tpG/eghQUoMxw713T6ipTHlK/DtwCrBORbkqp/kCte/6g2Y7fz1zmmUV/cjwhnScapTJ80/fkx8biOmAAZ+4dwPNn53F8jyG/T7hfOCE+IRW0WDmuYeGk/LKM7L/+ol6XcpfzaNoNzZRkfXkikgTYKaXsRGQjoFNwa5WWnVfAG6uiuf3THThcSWbplbUMmfMaKicH3n6RV0dm8dDBl8gtyOXfYf+mvkN9Fh9bbPY4XG/tDUqRvnWr2dvWtLrElBHEFaWUG4ZcSPOVUglAhmXD0uqaqFPJPLPoT05dSuMfHOWW1T8i2dm43DeFud1T+W/cf/Bw9uC5ns8xof0EHO0diboYxaqTq3j2pmdxczJPaVAAB09PXDp3JmPLVrynTzdbu5pW15gyghgFZAFPAmswTHkdYcmgtLojK7eA15YfZvznO2kSf4pfoudy8y9f4RQUyPY3xjG+6VJWnl/H1E5TWTl2JZODJ+Nob0ivMK7dOLLys1h9qqys8FXn1qcPWQcPkn/5stnb1rS6wpRkfRkASqkGwHKLR6TVGbtjk3hm8Z9cupjE+5d3ELjrV+w9GxL75Bj+7bGN5OT93NbmNh7v9jjN3Jpdd34nr06092zPkqNLGN9+vFljcwsPI/GTT8jcuZMGw4aZtW1NqytMmcX0IPAqkA0UAgpD8r42lg1Ns1UZOfn8Z00M3+w4xdiUaO478At2V5LJGB7GW93OEpO3nJs8b+KTHk/R0avsyXBKKca2G8ube97kSPIRAhsFmi1Gl86dsfPwIH3rNt1BaFoZTLnF9DSGOtL+ItJGRFqLiO4ctFLtOJ7IkFlbWP9bFHMPf8cDm+agfDyYNyOYv3XaQU59Bz6K+IivB31dbudQZHib4TjZOZn9YbWyt8ft1t6kb9uKFBaatW1rqama1LWBKXWrT506RadOna7a98orr/DOO+8Uvy/5v9mqVato3749p0+frnQ8/v7+dO7cmZCQEEJDr57Ds2bNGgIDAwkICLgqv1RZ+yv6rCaZ0kGcAMxblVyrc9Ky83hh6UGmfr6N0X+s5IvN79Hk/HF2TuzExNGn2dUwkb/f8neWjFxCvxb9TF785uHswYBWA1gRu4Ls/LLrF1SFa1g4BZcSDfmd6jBz16QGOHv2LK+88goPPPCAVf6Ambtu9fr163n88cdZvXo1raq4Yn/jxo3s37+fktkcCgoKmD59OqtXr+bw4cMsWLCAw4cPl7m/vHOswZQO4nlgh1Lqc6XUh0WbpQPTbMeWo5cY8sFWji5fy/wdsxj2xyrOh7bkkfsL+TTgFA+EPMiqsauYEDgBB7vK12EY124cablprDuzzqxxu97aG4D0rdvM2m5tY+6a1Lt37+axxx7j4YcfZty4caxatcpssX777bd06dKFrl27ctddd5V7rLnqVm/ZsoUHHniAFStW0LZt22q3V9KePXsICAigTZs2ODk5MXHiRH755Zcy95d3jjWY8l/r58AG4CCGZxCaBkBqdh7/WhHNuq2HmHl0Jd1P/k52s0a8c5c7e/3OMCpgFI+GPIqvq2/FjZXjpiY30dK9JYuPLmZ4myqVJymVo48PzsHBZGzZgte0B8zT6Orn4OJB87RVpElnGFrxt/SaqEmdlpbG5MmT2bZtG76+vvTr14+bbrqpwtjCwsJIS0u7bv8777xTnG78r7/+4vXXX2fHjh14eXmRnFx+yjdz1K3Oyclh9OjRbNq0iaCgoCrFDYbnZYMGDUIpxYMPPsi0adMAOHfuHC1atCg+zs/Pj927d5e5v7xzrMGUDsJRRGZaPBLNpmw8ksBLi/Zz8x/rmHvsN+wln1UDGvJ9txR6tryVn3vMNNtDZaUUY9qNYdbvsziVcgp/D3+ztAuG6a5Jc+dSkJaGvY3XPq6JmtQ///wz3bt3p0mTJgC4uLiYVBZ0qwmLEjds2MD48ePxMmbZbdSoUbnHV1S3uqzbmCX3Ozo60rt3b77++mtmzZpVpbgBtm3bRvPmzUlISGDgwIEEBQURHh5u0rm1mSkdxGql1DQMU1xzinaaktFVKTUHQ0W6BBHpZNzXCPgR8AdOARNE5LrJ6Eqpe4CXjG9fF5FvTIhVs7CUzDxeW3GYv9Zt49VDS2mefI5jwQ2Y1TcPjzZNmd3jHW5tfqvZrzuq7Shm/zGbpceX8mSPJ83Wrlt4GElffknGrl00GDiw+g2a8E2/NqhqTeojR46UehumZEnRmJgYFi5cyCuvvFL8uanfxEtTXtvl1a1u3Lgxl69Z55KcnEzr1q2L39vZ2fHTTz8RGRnJG2+8wQsvvHDV8abG3bx5cwB8fHwYM2YMe/bsITw8nObNm3P27Nni4+Li4mjevHmZ+4vaKuuzmmZKB1FUD7DkEytTp7nOA2YD35bY9xywXkTeVEo9Z3x/bcW6RhhqWIcar7VPKbWstI5EqznrDsfzr4W7GLVnKfef2k16QxfeGWvHqa6uPNr9OUa2HYm9nWWS33nX9ybcL5xfjv/Co90exdHOPLWK64WEYOfqSsbWbebpIGxEVWtSt23b9qr74YcOHSr+xl8eU76JR0REMGbMGGbOnEnjxo1JTk4udxRxbd3qyMhIvv322+I/pm5ubjRt2pQNGzYQERFBcnIya9as4Yknnriqnfr167Ny5UrCwsLw9fUtLnJkatwZGRkUFhbi7u5ORkYGa9eu5R//+AcAN910E8eOHePkyZM0b96chQsX8sMPPxAYGFjq/vLOsYZyOwillB3wnIhUaaqAiGxRSvlfs6pPDeUAACAASURBVHsU0M/4+htgE9dXpxsM/FY0SlFK/QYMARZUJQ6tei5n5PLqLwfJWL6M/0SvpH5uJitutmdZPwcmd3+QjzrcTX1H06q6Vce4duPYeHYjW+K2ENky0ixtKkdHXHv3In3rVkTErKnFa7uq1KT+29/+xo4dOxg8eDBt2rQhPDy81JrSVdGxY0defPFF+vbti729Pd26dWPevHllHl+ybnVhYSHHjx+/rkP59ttvmT59OjNnGu6Sv/zyy6WOgBo1asSaNWsIDw/H29ubkSNHmhx3fHw8Y8aMASA/P58777yTIUOGAODg4MDs2bMZPHgwBQUF3HvvvXTsaJjeXdb+8s6pcSJS7gZEVXRMBef7A4dKvL9S4rUq+b7E/qeBl0q8/zvwdEXX6tGjh2jmtfrgBRn11Dfyc+8hcjgwSH6J7CjD3+4kr+54VS5lXqrRWPIK8iTipwh5+LeHzdpu8o8/yuHAIMk+dqxK5x8+fNis8dSUffv2yZQpUyo8rm/fvnLy5Mlyj/nvf/8rc+bMERGRzZs3y6xZs8wRYrltjxkzRo4cOSIiIgcPHpQnn3zSbNesq0r7XS3vb7wpt5jWKaWexvDcoDhJn5ihqpyIiFKq9JJ2JjI+H5kG0LJly+qGpBklpefw+qJ9eC76ltdPbCGrnuLTYXYUDu3LBzfNpG1D804HNIWDnQOjA0bz1cGvuJhxkSauTczSrlufPoBhuqtzQIBZ2rQF5qxJPXjwYB588EEOHDhAYmIiH3zwgZmiLL3ta+tWd+rUiffee89s19QMyiw5WnyAUidL2S1i4mpq4y2mFfK/h9RHgH4ickEp1RTYJCKB15wzyXjMg8b3nxuPK/cWky45ah4rD5xn2cc/MDnqR7wz01nfVRE1JohH+j5Hz6bmLeBTWXFpcQxdMpTpIdN5qOtDZmv3xPDhOPr40HLOnEqfW9dLjppSk1qzDZUtOVrhQjkxpNa4dqtOqo1lwD3G1/cApa0A+RUYpJTyVEp5AoOM+zQLSkzP4bnZq0mc8QAztnxNpms6HzzgQ6s33mLOHYus3jkA+Ln7cUvTW1h6bCmFYr5lOW5h4WTujaIwUycNuNbUqVN153CDqrCDUEo5KqUeV0otMm6PKqVMmkKilFoA7AQClVJxSqn7gDeBgUqpY8AA43uUUqFKqa+g+PbVP4G9xu01c9zS0konIvwSdYqP7n+aiZ89TZfEWBYOdOH8h08y64lfGdF2BHbKlEX3NWNcu3GczzjPrgu7zNamW1gfJC+PjD17zNamptk6U55BfAo4Ap8Y399l3Hd/RSeKSFnTG66bgiIiUSXbFJE5QOXH+1qlJKRl88n78+m98lMmpmSwO9COyw+NY0b/p/B0KT3VgrVFtIygoXNDFh9dTO9mvc3SZr0ePVD16pGxZSvu/fqZpU1Ns3WmdBA3iUjXEu83KKUOWCogrWaICP/deIBL7z3PHcdPEd8QVk/vwbh7/kWrBlVLVlZTnOydGNF2BAtiFpCcnUwjl/JX3JrCztkZ1549Sd9Wt/MyaVplmHLfoEApVTxlRSnVBiiwXEiapaVk5fKfGTNo8eSd9Io9xaYIXzx+nsfMx76v9Z1DkbEBY8kvzGf5CfPVsHINDyPvzBlyq5DuWdPqIlM6iP8DNiqlNimlNmNI3PeUZcPSLOVicjo/3D2akb+u5UxTJy5+/jwPfbyRbi1utnZolRLgGUBX764sObaEimbimcotLAyo+9ldNc1UZXYQSqmiGo+xQDvgceAxIFBENtZAbJqZnYhL5Ne7htHv4El2RbTmthW7GRh2t82uHh7XbhyxKbHsv7S/4oNN4NSyJY6tWpK+dYtZ2tM0W1feCKIo99JiEckRkT+NW04552i11B/Rsey/Zyg9T1xi9+09uOfjFTg7OFs7rGoZ7D+Y+g71WXzUfNXm3MLCydy9h8Ic/WuuaeV1EElKqbVAa6XUsmu3mgpQq76NO/YR/8AY2l9I5/eHhjL19e9r1bTVqqrvWJ9hbYax9vRa0nKvz7hZFW5hfZDsbDJtcMGlLjl6tbpYcrSmS5GW91fiNuAfQCLwbimbZgOWrvgVuxn34JOay4kXpzJ5xnuQngArn4ZLtl9qc1y7cWTlZ7H65GqztFe/Z0+UkxMZW0yrA2ALdMnRitlCyVFrlCIts4MQkVwR2QX0FpHN124WjUozi3nzvqfZS0/ilF/AlXeeZ/TkZ+HsXvg8HPZ+CaufsXaI1daxcUfae7ZnybElZmnPrl496t90U52a7qpLjpbPVkqOWqMUqSnrIDyVUv/CkJW1+HgRibBUUFr1iAgfv/set37zFSmudjSY/T69egyCqDmw6hlo0AxC7zW8P70TWvWydshVppRibLuxvLnnTWKSYwhqdH3ZyMpyDetDwptvkXfuHI6VLNTy1p63iEmOqXYMJQU1CuLZntdmxL+eLjlaebZUctQapUhN6SB+Bj4DvkKvf6j1CgqFWS89y6Clyznr60C7r78hoGUHWPYY/PEdtI2EcV+BgwtEr4BNb8A95ltLYA3D2wznvaj3WHx0MS/e8mK123MLCyPhzbdI37YdzzsmmCHCmqFLjl5NlxytPlM6iHwR+dTikWjVlp2Xz6dPPMDQDbuIbl2PW79diq+TPcwdCud/h7Cnof8LUFT1rc8M+PUFOLUd/M1fJrSmeDh7MNB/ICtjV/JU6FO4OFT8B6s8Tm3a4NCsKelbt1S6gzDlm35toEuOGthSyVFrlCI1ZSrLcqXUI0qppkqpRkWbRaPSKi0lI5s5U8cxZMMu/ujkycCf1+Gbfho+7wuJx+CO+RD59/91DmC4zeTmC5v+bb3AzWRcu3Gk5aXx2+nfqt2WUsow3XXnLiQ31wzR1T4lS46WprySo3/++Wfx+0OHDnHx4sUKr7d161b2799/3Vbyj2xERAQ///wzSUlJABXeYiqt5Oi5c+eKPy9ZcrSovTVr1tDHWP+jSFHJ0fnz5/P1119XOu6MjIziTqSo5GjR7KmS5UNzc3NZuHAhI0eOLHN/Vc+xFFNGEEWpuf+vxD5Ta1JrNeBiUiqr7x1F/yMX2XWrH3d+ugznqK/ht39A4wCYOB+82l1/omM96PMkrHkOTm6F1mE1H7yZhPqG0tK9JUuOLWFE2xHVbs8trA9XfvyRzP37ce1p/TTnlqBLjv6PrZQcrfFSpGWVmrPF7UYsOXrs1Dn5cWBPORwYJF89PV7ys1JEfpoq8nIDkYWTRbJTy28gN1Pk7fYic4aKFBbWTNAW8uWfX0qneZ3k5JWT1W4rPy1NDnfsJPHvvFPhsbrkqC45aisqW3K0vFQbEcafY0vbLNttaaaIOnCYmLtuo8PZVPZMjeDeZ9/Afs5gOPxfiHwZJnwHztc/vLuKYz0Imwmnt8NJ204xMartKOyVPUuOV3/Kq72bG/W7d6/TeZlKlhytrsGDB7Np0yZmzJjBF198wZ133mmGCMtuW5ccrRnl3WLqiyExX2njdQHMM/Fcq5L1G7Zg9/x0/DLyiX56EveE3QxfRoCdHUxeBAHXldwoW/d7YNsHsPENaB0ONpqbybu+N339+vLL8V94rNtjONqZVNeqTK5hfbj07nvkxSfg6Otjpihrl3vvvbfCY0ypKOfi4sI333xjrrBMavvuu++2yPW0/ylvodzLxp9/K2Wr+LdKs5jFixZR/+mHcM3J5+K/nmR8W2f44Q7wbAXTNleucwBwdDGMIs7ugljbzsM4rv04krOT2XK2+qOhouyuGdu3V7stW6ZLjt64bD8hzw1m3uef4P/a38l3gIL3X2FIxnrDWoYud8B9aw2dRFV0vxsa+MHGf4OZ0mdbQ+9mvfGp78PiY9VP4OccGIiDt7fO7qrdsGq8g1BKBSql9pfYUpVSM645pp9SKqXEMf+o6ThrGxHh09f/QY9ZH5Hk4YDXrFfodfAtOL4Ohr4NYz4zPE+oKgdnwygibg+cWG++wGuYg50DowNGs/38di5mVDz9sjxKKVzDwsjYsRMpJymcptVVNd5BiMgREQkRkRCgB5AJLC3l0K1Fx4nIazUbZe1SWCjMfvoRwr//mRPNnen4r+kEb5gBuelwzwq4eZp5nht0uws8Wtj8KGJMwBgKpZClx0v7taoct7A+FKakkHXwoBki0zTbUmEHoZTap5SarpSyRAX7SOCEiOgaj2XIzsvns/snMGDlJg4GNqDfowNovuFZaNLJ8LzBnHmUHJwg7Ck4F2UYmdgoP3c/ejXtxdJjSykorN4MHddevcDOjgwTUy5oWl1iygjiDqAZsFcptVApNViZrwTZRGBBGZ/1UkodUEqtVkqVuRpEKTVNKRWllIq6Ni2ArUtJz+SHO4fSf8ch9vTwYfToxjT840sIvc8wcmjQ1PwXDZkMHi0NM5pseBQxtv1YLmRcYPeF6iUzs2/YkHpdu9bp6a6aVpYKOwgROS4iLwLtgR+AOcBppdSr1Um5oZRyAkZiSAZ4rd+BViLSFfgI+G858X0hIqEiEurt7V3VcGqdCwlJrJowgF4H49jVvyWTeyTiEhcFoz6G4e8Zvu1bgoMThD9tyN10bK1lrlEDIlpE0NC5IYuOLap2W65hfcg+dIj8ClI/aFpdY9IzCKVUFwxFgt4GFgPjgVQM6ySqaijwu4jEX/uBiKSKSLrx9SrAUSnlVY1r2ZQjx06w545BdIm9zO8j2jC12R84KAX3/Qrdplg+gJA7oWErQ44mGx1FONk7MaLtCDae3UhSVlK12nILCwMRMrbvMFN0mmYbTHoGAbwP7AW6iMjjIrJbRN4FYqtx7UmUcXtJKdWk6DaWUqqnMc7q/VduI/bs3cuZe8bgn5DJ8dubM9l1G6rlLTBtEzTrVjNB2DtC+P/B+T/g6JqauaYFjGs3jvzCfFbErqhWOy4dO2Lv6Vnrp7vqkqNXq8mSo/feey8+Pj7XXQ9qpuSopUqRmjKCGC8ikSLyg4hcVcldRKqUckMp5QoMpMRqbKXUQ0qph4xvbwcOKaUOAB8CE405Q+q0tatXk/vwVBpl5HFlrDujHPZC78dhyhJwreEBVNeJ4Olv088i2jZsS4h3CIuPLaY6vz7Kzg7XPn3I2LYdKSw0Y4Q1Q5ccrVh1S45OnTqVNWuu/zJVEyVHLVmK1JQO4n6lVPEySqWUp1Lq9epcVEQyRKSxiKSU2PeZiHxmfD1bRDqKSFcRuUVE6vzYftH3c/F87insENSIAvq6xMH4eTDon2BvStJdM7N3hL7PwsU/IWZlzV/fTMa2G8vJlJPsv7S/4oPL4RbWh4LkZLIPR5spspqjS46WzxwlR8PDw0stcFQTJUctWYrUlL88Q0WkuIqGiFxWSg0DXjJLBBrzPniTbl99Q7K7okX/ywT4NIWJv4BPsHUD6zwBtrwDm96EwGGGPE82ZrD/YN7a+xaLji6im0/Vb9G53mooqJSxdQv1Olk4xXIV6ZKjlWeukqNlqYmSo5YsRWpKB2GvlHIuur2klKoHOJvl6jc4EeGLv8+kz+I1nGliR9feF2nWcaBhVXS9WpD7xt4B+j4DSx+EmBXQwbLFSSyhvmN9hrYeyooTK3iu53O4O1WQ3bYMDo0b49KpE+lbt+H18MNlHnfxjTfIiTZvTWrn4CCaXFPprDS65OjVarLkaF1lylfC+cB6pdR9Sqn7gN8Ay6RtvIEUFBTy+fS7CF+0hhOtFLeEXaDZwGdg4g+1o3Mo0ul2Q9GhTW+CDd5/B8PD6uyCbFafXF2tdlzD+pC1fz8FKSkVH1zLWaLk6IoVhskAMTExV5UbBcM38ZCQkOu2desqXpBZXttVKTla1AHB/0qO7tmzhzfeeOO6NqoTN1StfGhlz7FoKdKyCkWU3DBMSX3HuA025RxrbLZSMCgrK0e+unOoHA4MkiWjAyXjdT+RI2usHVbZDvxoKEB0aKm1I6mSwsJCGfvLWJmwfEK12snYt08OBwZJyuqr/7+qLQWDXF1dr3o/d+5cmT59epnH+/n5SVZWVqnnR0dHS+PGjSU/P/+qgkGff/65DBs2rPi4gwcPyoULF2Tu3LmyfPny4nNffvnlSsd/6NAhadeunSQmJoqISFJSUvG/o7S2ExMTJTAwsPj8iIgIiYuLu6rNHj16yPr164vba9eunRw/fvy6f3NSUpJ06NBBvvrqq0rHXeTkyZPSsWPHq/bl5eVJ69atJTY2VnJycqRLly5y6NChMvdX5Zzy2rqW2QoGXdOJrBaRp43br+bpmm5MV65cYfGESHrvO0lMt3yGD/Cg/kOboP1ga4dWtk7jwKs9bH7LJkcRSinGtRvH4aTDHE6q+uyOel26YNegQa2f7mqqopKjRYqeQYSEhHDHHXeUWXLU29ubwYMH8/DDD3Pw4MHi203VVbLkaNeuXYvLhJbF1JKj//znPwkJCSEiIqLCkqOvv/46y5Ytq3TskyZNolevXhw5cgQ/P7/i2tYly4cGBwczYcIEOnbsWOb+qpxTXlvVVlbPUbQBt2BYA5EO5AIFQGpF51ljq+0jiLNxcbIksoccDgySZVNbS+FPU0Vy0q0dlmn+/Nkwiji42NqRVElKTorc9P1N8uLWF6vVztknZsjRsHApLFGetbaMICpLlxy98VhiBDEbw6K2Y0A94H7gY/N0TzeOvw79yV8ThxJwPoMz/bMY8cBM1O1zwMnV2qGZpuMY8Ao0jiKqX6KypjVwasDItiNZdXIViVmJVW7HLawP+QkJ5Bw9ZsborEOXHNUqVFbPUbRh7F2AP0vs+6Oi86yx1dYRxPYtG2TLTR3k905BsunJdiKxm60dUtUcXGQYRfz5s7UjqZLYK7HSaV4n+eSPT6rcRu7FeDkcGCSJX35ZvM9WRxCmmjt3rly+fNnaYWhmYIkRRKYxsd5+pdR/lFJPoivRmWz1kh9wePwRnPILyRvnTt+XNxjqPtuiDmPAO9hmRxGtPVoT1jyMH4/8SG5BbpXacPT1wTkw8IbK7qpLjt64TPlDf5fxuEeBDKAFMM6SQdUViz57k6Yv/5McJ3C5rzM9X9wCHn7WDqvq7Oyg37OQeBQOVb+kpzVM6TCFpOwk1pyqeo4pt7A+ZP7+OwXpGWaMTNNqH1M6iB6AiCHD6qsiMlNEjls6MFu34NUHCfzwGxIbCk1emEzX6T+BY8ULimq94FHg09EwiiiwvTKcvZr2oq1HW74//H3R7dJKc+0TBnl5ZO4xz2pVTautTOkgRgBHlVLfKaWGK6WskBjItiyYOY6QBVs42xw6fPg27UbVoawkdnbQ7zlIOg6Hql9roaYppZjcYTLRydHsi99XpTbqd++GXf36pG+pG9NdNa0sphQM+hsQgKGwzyTghFLqK0sHZquWzX6RLqsPc7SNHb2/+YVm3UZYOyTzCxoOvp1tdhQxvM1wPJw9mB9dtURvysmJ+r16kbF1W/EopKqjEU2rKVX5HTV1oVwesBpYCOwDRlf6SjeA37euoNlXS4hvDLd88iMNm7W3dkiWUTSKSI6Fgz9ZO5pKq+dQj/Htx7Ph7Abi0uKq1IZbWB/yzp0j9+QpXFxcSEpK0p2EVmuJCElJSSblzSqpwttFSqmhGOpS9wM2AV8BEyofYt12KT6OK88/gzvg/tIz+PpfXzikTgm6DZp0gc3/MWR9tUZK8mq4I/AO5h6ay4KYBfzfTf9X6fNd+4QBkL5lM36TJxMXF3dd3iJNq01cXFzw86vcJBlT/qu+G/gReFCuKRikGRQUFLD9/ttolyScuK8PI4f8zdohWZ5S0O95WDgJ/lxYM6VQzaiJaxMGtRrEkmNLeCTkEVwdK7dg0cmvOS4dO3J5/g80mjyZ1q1bWyhSTbMeU55BTBKR/+rOoWw/Tx9C4LFcDvb1YuTTX1o7nJoTOBSahhhGEQV51o6m0qZ0mEJ6Xjq/HK9acRWvxx4l7+xZrixeUvHBmmaDyuwglFLbjD/TlFKpJbY0pVRqdS+slDqllDqolNqvlIoq5XOllPpQKXVcKfWnUqp7da9pCf/94Ck6b4ojJtCe8R+tt3Y4NatoFHHlNBwotbx4rdbFuwtdvLswP3o+hVL5JIRufftSr1s3Ej/5hMJyUmdrmq0qs4MQkT7Gn+4i0qDE5i4iDcx0/f4iEiIioaV8NhRoZ9ymAZ+a6Zpms3fdYlrMXcUFbwj7ZCn2jk7WDqnmtR8MzbrDlrchv2qrk61pSvAUzqSdYWtc5QvDKKXwfnIG+QkJXP7B9jpITatIubeYlFL2Sinzlscy3SjgW2O6kF1AQ6VUUyvFcp34c6fI+MdLFNhB41f+jlfzdtYOyTqKRxFnYH/16wPXtAGtBuBT34fvor+r0vmuPXvieuutJH3xBQXp6WaOTtOsq9wOQkQKgCNKqZYWuLYAa5VS+5RS00r5vDlwtsT7OOO+qyilpimlopRSUTU1iyQ/N49d00bidRkS7omgW6T5MlfapHYDoXkobH3X5kYRjnaOTAqaxO4Luzl2uWoZWr1nzKDgyhWS5+lCi1rdYso6CE/gL6XUeqXUsqLNDNfuIyLdMdxKmq6UqlIGOxH5QkRCRSTU29vbDGFV7Ofpg2l/Io9DEU0YPkNnPkcp6P88pJyFP6r2Tdyabm93Oy72LlVeOFevcyfcBw4kee5c8q8pcalptsyUDuLvwHDgNeDdElu1iMg5488EYCnQ85pDzmFIDFjEz7jPqpa+/RghWy9wONiR8bPWWjuc2qNtJPj1NI4ibGvCW0OXhgxvO5zlJ5aTnJ1cpTa8n3icwsxMkr7USQa0uqO8WUwuSqkZwHggCNguIpuLtupcVCnlqpRyL3oNDAIOXXPYMuBu42ymW4AUEblQnetW165fF9Dqu3XE+ULEp8uwd3C0Zji1S9EoIvUc/P6ttaOptCnBU8gtzGXR0arll3IOCMBj5Eguz59PXny8maPTNOsobwTxDRAKHMRwG6jao4YSfIFtSqkDwB5gpYisUUo9pJR6yHjMKiAWOA58CTxixutX2vkzx8h55TXyHMDnn6/j2cTfmuHUTm36Q4tbYOt7kGdb0z7bNmxL72a9+THmR/KquKbD67FHkcJCEj+tdRPuNK1KyusgOojIFBH5HLgdCDPXRUUkVkS6GreOIvIv4/7PROQz42sRkeki0lZEOovIdWslakpebi5RD42lUQok3TeMruG6HEapikYRaedtchQxOXgyCVkJrD1dtVuHTn5+eI6/nSuLFpN75oyZo9O0mldeB1H8NUpEbC9lpxktfmgA7WLzOTzQj2HTzTmQqoNa94WWvWGb7Y0i+jTvg38Df747/F2VE+81fughlIMDl2bPNnN0mlbzyusgupZcPQ10MedKalux5I1pdN1xib86OnH7u6utHU7tVzyKuAD75lk7mkqxU3ZMDp7MX0l/ceDSgSq14ejjQ6O7ppC6fAXZR4+aOUJNq1nlraS2v2b1tIMFVlLXatuWz6X1gq2cbQIDP1+JvYNtZSy1mtbh0KqPcRSRZe1oKmVk25G4O7nzffT3VW6j8X33YefqyqUPPzRjZJpW80yqB3EjOnviEIX/+g85TuD373fw8LLhWtLW0P95SI+HqDnWjqRS6jvWZ1y7caw7vY4L6VWbNGffsCGN77uX9HXryfrzTzNHqGk1R3cQpcjJzmL/9Il4pkLKQ2Po0Os2a4dke/z7gH8YbPsAcjOtHU2lTAqahCAsOFL1/Eqed92NfaNGXPrgAzNGpmk1S3cQpVj6YCQBpwo4PLQ1gx94w9rh2K7+L0BGAkR9be1IKqWZWzMiW0ay6OgiMvOq1rnZu7ni9eA0MnbsJGPXLjNHqGk1w6QOQinVSik1wPi6XtEit7ro59em0nX3Zf7q7MKEt1dYOxzb1qo3tOlnHEVkWDuaSrmrw12k5aaxIrbqvwMNJ07EoUkTLr3/gS5HqtmkCjsIpdQDwCLgc+MuP+C/lgzKWjYt+ZR2P+/mdDPF4M/XoOz0AKva+r0AmYmw17ZSUIR4h9ChcQe+j/6+SrUiAOycnfGa/ghZBw6QvnGTeQPUtBpgyl/A6cCtQCqAiBwDfCwZlDWcjvkD+7c+JMsZ/P8zC/dGvtYOqW5oeTO0jYDtsyDHdtJhK6WYEjyFkykn2XF+R5XbaTh6NE6tWnHpgw+Qwqp1NJpmLaZ0EDkiUpzDWSnlgCFVd52RlZHGocen4JEOGY9OIih0oLVDqlv6vQCZSbDXtsqxDvEfglc9L74/XPUpr8rREa/HHyPn6FFSV+l1NJptMaWD2KyUegGop5QaCPwMLLdsWDVr2YMDaHOmkJjb2jNg6j+sHU7d0+ImCBgA2z+EnDRrR2MyR3tHJgZOZPv57cReia1yOw2GDsU5MJBLH32I5Nle7W7txmVKB/EccAlD0r4HMSTRe8mSQdWkMzFRtIpO5VBIPSa8VScfrdQO/V6ArGTY84W1I6mU8YHjcbJzqnKtCABlZ4f3jCfIO32GK0uXmjE6TbOsCjsIESkUkS9FZLyI3G58XWduMbUMCqXh5x8y7Iv1hjQRmmX49YB2g2DHR5BtO5laGrk04rY2t7HsxDJSclKq3I5bv37UCwkh8eNPKMyxrXoZ2o2rvHoQB5VSf5a11WSQlhYUOhDXBp7WDqPu6/ccZF2G3Z9XfGwtMjl4MtkF2VWuFQGGh97eTz5Jfnw8lxdUfQGeptWk8kYQw4ER5WyaVjnNe0D7IbDzI8iu+rfxmhbYKJCbm9zMgpgF5BVW/RmC6809ce3dm6TPv6Ag3bbWhWg3pvKS9Z0u2oAcoCvQBcOsptM1FaBWx/R7ztA57PrM2pFUyuTgycRnxrP+zPpqteP95AwKLl8m+dtvzBSZplmOKQvl7sdQ9W0shsJBu5RS91o6MK2OatYNAofBzo8h64q1ozFZuF84LdxbVGvKK0C9zp1xHziA5Dlzyb982UzRaZplmDKL6f+AbiIyVUTuAXoAz1b1gkqpFkqpjUqpw0qpv5RST5RyTD+lVIpS4FZ9QAAAF5ZJREFUar9x03NP65J+z0FOCuyyndKc9nb2TA6ezIFLBzh46WC12vJ+/HEKMzJI/tq2clRpNx5TOogkoOTk9TTjvqrKB54SkQ7ALcB0pVSHUo7bKiIhxu21alxPq22adoWg4bDrE8NDaxsxOmA0bo5u1aoVAeDcrh0eI0eQ/P188uITzBSdpplfebOYZiqlZgLHgd1KqVeUUi8Du4Aql8oSkQsi8rvxdRoQDTSvanuajer3POSkws5PrB2JyVwdXRkdMJq1p9YSnxFfrba8Hn0Uyc8n6XPbehaj3VjKG0G4G7cTGJLzFa19+AU4aY6LK6X8gW7A7lI+7qWUOqCUWq2U6lhOG9OUUlFKqahLly6ZIyytJjTpBMEjDbeZMpOtHY3J7gy+kwIp4McjP1arHacWLWg4/nYu//QzuWfPmik6TTMvZa01b0opN2Az8C8RWXLNZw2AQhFJV0oNA2aJSLuK2gwNDZWoqCjLBKyZX/xf8GlvCHsaIv9u7WhM9sSGJ/g94Xd+u/03XBxcqtxOXnwCJwYNosGQITR7600zRqhpplNK7ROR0NI+M2UWk7dS6m2l1Cql1IairZoBOQKLgfnXdg4AIpIqIunG16sAR6WUV3WuqdVCvh2hw2jY/ZlNjSKmdJjClZwrrIxdWa12HH198JwymZRly8g5dsxM0Wma+ZjykHo+EAO0Bl4FTgF7q3pBpZQCvgaiReS9Mo5pYjwOpVRPY5zVeTCu1Vb9njMUE9rxkbUjMVmobyiBnoF8H/19tQsBNb7/fuz+v70zD5OiOtf47+2eHZRNXAAVWYzbRaMxxiUumLjEPS7gFaMx9xpycxM1EgkxMZiQxLgkuFwDURGNxmiI+4a44kIENxTUCCoiCIoLOsw+3d/945yeqenp2XvoHub8nqeeOlV16tRbVd311jlV9Z2yMtZd1XP2P9B7aI9BDDKzG4A6M3vKzM4CxnZhm/sDpwNjI6+xfkvSREkTfZ6TgCWSFgNXAeM3pfhPgQhb7gy7nuDCb1R8nGs17UISE3aZwPL1y1nwwYIulVUwYAADz/ou5fPmUfVa116fDQSyTXsMIhVbYI2koyR9GRjY2Q2a2TNmJjMbE3mN9UEzm2FmM3yea8xsVzPb3cy+Zmad77ElkP8cNBnqKuG5q3KtpN0cucORbFm2JZOemsSj7z3apbIGnnEm8QEDWDf9yiypCwSyQ3sMYpqkfsD5wCTgeuDcblUV6F1suRPsdiIsvA429Iw30Yrjxdx0xE1sv/n2nPfkefzu+d9Rm6hte8UMxPv2YdDZZ1Px7LNUPL8wy0oDgc7TnnDf95vZ52a2xMwOMbO9gJEbQVugN3HQZKivhud6zl30sM2GcfORNzNh5wnc9uZtnP7Q6bz/RedeWR1w6ngKttqKtb/+NdVvdfozo0Agq7SnBpGJn2RVRSAweEf4j5Nh4fWwoed8XVwYL2TyVydz5SFX8n75+5xy/ynMXTG3w+XESkrYZto0Ep98wrvfPpGP/jSdZHV1NygOBNpPZw0i9KwTyD4HXgCJGni259QiUozdbixzjpnDiH4jmPTUJKb9axo1iY51DNT36wcw4qEH6XfUUXwycybvHHccFQu69hA8EOgKnTWI8EZRIPtsMQrGjINFN0B510JZ5IIhfYcw+4jZnLHLGdz+79uZ8OAE3vuiY5HxCwYMYMgfLmG7G2cBsPK7Z/HB5J+FyK+BnNBaLKZySV9kGMqBIRtRY6A3ceBPIVELz07PtZJOURgvZNLek7hm7DWsqVjDKfedwoPvPNjhcvrsuy8j7rmHQRO/z+cPPMA7R36L9Xfd3eXvLgKBjtBah0GbmdnmGYbNzKxgY4oM9CIGjYTdx8MLs6B8ba7VdJqDtj2IOcfMYccBOzL56clcvOBiqus79kwhVlLClueey4i77qRohx1YM2UKK797FrUrVnSP6EAgjc42MQUC3ceBkyBRB09n/NC+x7B1n62ZdcQsztrtLOa8NYfTHjyNdz/veJzL4tGj2f7WW9h66q+oXrqUd449jo9nzMBqO/dabSDQXoJBBPKPgSNgj/+EhTPhb+Nh1Yu5VtRpCmOFnLfXeVx76LWsq1zHuPvHcd/b93W4HMViDBg/nhEP3E/fsWNZN/1K3j3xRCpfeqkbVAcCjpxFc+0OQjTXTYjaCtct6YL/g+r1MHKsez6x/X65VtZpPqz4kAvmX8BLH73ECaNOYMo+UygtKO1UWeVPPMHa3/yG+g/W0H/cOLY8/yfEN988y4oDvYHWorkGgwjkNzXl7q2mBddAxTrY/gDXBDXiYFDPe9u6PlnPta9cy/WvXc/I/iO5/KDLGdm/c9+dJisqWHfV1Xz6178SHzSQrS+8kM0OPxz1wOMSyB3BIAI9n9pKeOkm941E+RoYtrerUYw+rEcaxXMfPMeUp6dQVV/Fz/f5OcePOr7TZVUtWcraiy6i+vXX6XvQQWx90S8pHBo6aQy0j2AQgU2H+hp4+RZ4Zjp8vhK2HuOMYqejIdazHqmtq1zH5Kcns2jtIo4deSwX7nMhZYVlnSrL6uv59JZbXNhwMwb/+McMPH0CKggvHAZaJxhEYNMjUQev3gFPXwGfvg2Dd3ZNT7ueALF4rtW1m0QywcxXZzJj8QyG9xvOxftdzG6DdqMwXtip8upWr2btb6ax4cknKdllF7b65S8oHjWKWGlpMItARoJBBDZdkglYehfMvxzWvQEDR8LXz4cxp0AnL7K54Pk1zzN5/mQ+qf6EAhUwvN9wRvUfxegBoxvGQ/sOJaa2a0lmRvncR1j722kk1jX2saHCQlRWRqy0tGFQWSmx0rKm02VljfPKSlGpz1Pm86RNx0pLUVFRdx6eQDcSDCKw6ZNMwpv3w/zLYO2r0H87OOA82OM0KCjOtbp2sb56Pc998BzL1i9j+WfLWbZ+Gas3rG5YXlpQyqj+o5oZxxalmXvjTXzxBeXz5pEoL8eqqkhWVpKsrCJZVUWyqhJrSDefto4GCiwsbNV4mhhNaSmxPmVNpxvMJ814ysqcsfXA50w9hWAQgd6DGSybB/MvhVWLYLMhsP85sOd3oKhz7fu5pKKuguXrlzcYxrLP3PBZTWNspoElA5uZxqj+o+hT2KfT27VEgmRVNVZV2WgilZXOaKqqvNFEpisqG42mYbk3m6rKtOmqjomJx72JlDWYRovG06TG03KtR748FRX1evPJO4OQdARwJRAHrjezS9KWFwM3A3vh+qIeZ2Yr2io3GESgATN49yl46jJ47xnoMxj2mQiDd3JGUdQXivr4oS8UlkFhaY95I+rjqo+bGEdqXFXfePEd2ncowzcfTlG8iIJYAXHFicfixBVvnI6m05f56VS6QAUZ80TnR9dpqdwYoqAmQaymllh1LbGaOlRVC9U1xKrdmKpqrNobTWVlu2o9yaoqrLKyYwcyFmta4ykra7EW1GS6rDRiVs1rPbHSUlRc3CPMJ68MQlIceAv4JrAKWAScamavR/L8DzDGzCZKGg+cYGbj2io7GEQgI+89555RvP1YGxmVZhzpQ2RZYWpcAvFi14xVUAwFJRAvapyOLmuSLsq6GSUtyQcbPmDZZ8tYvn45yz5bxsryldQl60gkEyQsQX2ynoQlGqZT6XqrbzIvH0g3npYMqomJWZyShCipE8X1UForiuuM4joorjOKalNDkqLaJIW1SQprEm5cm6CgJkFBTT0FtfXEayJDdR2xmlrUgculxQTFxVBaDCUlqNQPZaWopLFGFCsrJV7ah1hZGfGyMgrK+hBvGBpNK9r8ppKSrJlPvhnEvsBUMzvcT08BMLPfR/LM9XkWSCoA1gKDrQ2xwSACrfL5Kqj81H2lXVsBtRtcX9ipdMP8ihbyRPJ1snvRJmQyj1hBzmsxhpGAhqE+lRYkIsvqG+al8liz+W7aSAL1kXlN1lFkGw3rN9XQtFxrpiu6/USz7UfWyTQ/TVcqX/MDYxTVQ3EdlNTiTQdK6ozi2lQ6Mr/WGqdro8ssQxkQ68ClOAnUFUKtHyo3i3HYo0vbX0CE1gwiF++9DQWi/TKuAvZpKY+Z1Uv6HBgEfJyWD0lnA2cDbLfddt2hN7Cp0G+YG7JBos6ZRX1N45CIpOurnYm0uqwa6mubLkvWZUdfFxDuwtCbX4o1c6aWwKhvMJ5U2khYynTMz49MW9PpJutbY/710fXNSNYnodYNVmdQl0S1yYZxrNZQXZJYnRGrNeJ1ST82VNg9Z6vH/wbM7C/AX8DVIHIsJ9BbiBdC6YBcqwh0E8I9II0DvfkF3lx8eroa2DYyPczPy5jHNzH1wz2sDgQCgcBGIhcGsQgYLWkHSUXAeODetDz3Amf49EnA4209fwgEAoFAdtnoTUz+mcL/AnNxNbhZZrZU0q+BF8zsXuAG4K+SlgOf4kwkEAgEAhuRnDyDMLMHgQfT5l0USVcDJ29sXYFAIBBopGeFvwwEAoHARiMYRCAQCAQyEgwiEAgEAhkJBhEIBAKBjGxS0VwlrQPey7UOzxZk+PI7z8h3jfmuD/JfY77rg6AxG3RF3/ZmNjjTgk3KIPIJSS+0FN8kX8h3jfmuD/JfY77rg6AxG3SXvtDEFAgEAoGMBIMIBAKBQEaCQXQff8m1gHaQ7xrzXR/kv8Z81wdBYzboFn3hGUQgEAgEMhJqEIFAIBDISDCIQCAQCGQkGESWkbStpCckvS5pqaRzcq0pE5Likl6WdH+utWRCUn9JcyS9KekN31Vt3iDpPH9+l0i6TVJJHmiaJekjSUsi8wZKmidpmR/ntJejFjRe5s/zq5LuktQ/n/RFlp0vySRtkQttER0ZNUr6kT+OSyVdmo1tBYPIPvXA+Wa2C/A14IeSdsmxpkycA7yRaxGtcCXwsJntBOxOHmmVNBT4MfAVM9sNF7Y+H0LSzwaOSJv3M+AxMxsNPOanc8lsmmucB+xmZmOAt4ApG1tUhNk014ekbYHDgJUbW1AGZpOmUdIhwHHA7ma2K3B5NjYUDCLLmNkaM3vJp8txF7ahuVXVFEnDgKOA63OtJROS+gEH4voFwcxqzWx9blU1owAo9T0elgEf5FgPZjYf139KlOOAm3z6JuD4jSoqjUwazewRM6v3k//C9TKZE1o4hgB/Ai4Acv5WTwsafwBcYmY1Ps9H2dhWMIhuRNJw4MvA87lV0ozpuB97MtdCWmAHYB1wo28Gu15Sn1yLSmFmq3F3aCuBNcDnZvZIblW1yFZmtsan1wJb5VJMOzgLeCjXIqJIOg5YbWaLc62lFXYEvi7peUlPSdo7G4UGg+gmJPUF/gmca2Zf5FpPCklHAx+Z2Yu51tIKBcCewJ/N7MtABblvGmnAt+MfhzOyIUAfSRNyq6ptfLe9Ob8DbglJF+KaaG/NtZYUksqAnwMXtZU3xxQAA3HN2j8F7pCkrhYaDKIbkFSIM4dbzezOXOtJY3/gWEkrgL8DYyXdkltJzVgFrDKzVM1rDs4w8oVvAO+a2TozqwPuBPbLsaaW+FDSNgB+nJWmh2wj6UzgaOC0POt/fiTuRmCx/88MA16StHVOVTVnFXCnORbiWge6/DA9GESW8a59A/CGmf0x13rSMbMpZjbMzIbjHqw+bmZ5dfdrZmuB9yV9yc86FHg9h5LSWQl8TVKZP9+HkkcP0dO4FzjDp88A7smhloxIOgLX5HmsmVXmWk8UM3vNzLY0s+H+P7MK2NP/RvOJu4FDACTtCBSRheizwSCyz/7A6bg781f88K1ci+qB/Ai4VdKrwB7A73KspwFfs5kDvAS8hvsf5TwUg6TbgAXAlyStkvQ94BLgm5KW4Wo+l+ShxmuAzYB5/v8yI8/05RUtaJwFjPCvvv4dOCMbNbEQaiMQCAQCGQk1iEAgEAhkJBhEIBAIBDISDCIQCAQCGQkGEQgEAoGMBIMIBAKBQEaCQfQifCTKKyLTkyRNzVLZsyWdlI2y2tjOyT666xPdva02dKzoalRPSVMlTfLpEh9tdWqGfHmxz+1F0oYurNtwXCU914VyzpQ0pLPrBxzBIHoXNcC3cx2uOB0f8K69fA/4bzM7pLv0bGwkFeG+vH/RzKZmyJJxnzt43HocZtaVr9PPxIVBCXSBYBC9i3rcB13npS9IrwGk7gIlHeyDf90j6R1Jl0g6TdJCSa9JGhkp5huSXpD0lo/5lOp34jJJi3y8/+9Hyn1a0r1k+Epa0qm+/CWS/uDnXQQcANwg6bK0/NtImu8/tFoi6et+/p+9pqWSLo7kXyHp9z7/C5L2lDRX0tuSJkY0zpf0gKR/S5ohqdl/RtIEfzxekTTT73PcH9Mlfj+aHXNPAXA7sMzMmsWbSt9nf2d8r6THgcfk+nu42x/bf0ka49ebKukmf4zfk/RtSZd6LQ/LhYNJ39ZIv+xFv95Ofv4xckHgXpb0qKSt/Py+km70Zb4q6cRIWb+VtNhrahYgsLV1I3k2RNI/jfyGLvbzhvua1XX+/D4iqdT/jr+C+9DyFUmlLRz7QFuYWRh6yQBsADYHVgD9gEnAVL9sNnBSNK8fHwysB7YBioHVwMV+2TnA9Mj6D+NuOkbjQhKUAGcDv/B5ioEXcLFtDsYF4dshg84huHAWg3EX0MeB4/2yJ3H9MKSvcz5woU/Hgc18emBk3pPAGD+9AviBT/8JeBX3Ne9g4MPIvlcDI/z681LHyK+/BbAzcB9Q6OdfC3wH2AuYF9HXP4Pmqbiwzbe3cd4a9hl3Z7wqsl9XA7/y6bHAK5GynwEKcf1pVAJH+mV3pY5n2nYeA0b79D64MCwAA2j8qPa/gCt8+g+p85/K58cGHOPTl6bOf9q2Wlp3BbBF2m/wMNyNjXC/r/tx4eCH42569vD57gAmtPY7CUPHhk26ihpojpl9IelmXIc3Ve1cbZH5kNGS3gZSoa1fw8d/8dxhZklgmaR3gJ1wf+4xkdpJP5yB1AILzezdDNvbG3jSzNb5bd6KuyDc3ZpGYJa/M77bzF7x80+RdDbOaLYBdsGZAbg4Ran96Guu/45ySTVq7NVsoZm943XchrubnxPZ7qE4M1gkFzyzFBcQ7z5c6IOrgQcixyydZ4D9JO1oZm+1sn9R5plZqj+AA4ATAczscUmDJG3ulz1kZnWSXsMZ3MOR/R0eLVAu+vB+wD/UGAS02I+HAbfLBfsrAlLn7BtEOkoys898shZ3EQd4Efhmhn1oad1MHOaHl/10X9xvaCUuaGLqXL+Yvl+BrhEMoncyHRdH6MbIvHp8k6NvRimKLKuJpJOR6SRNf0PpcVsMd9f3IzObG10g6WBcDSIrmNl8SQfiOkKaLemPwNO4WtLeZvaZpNm4Wk2K6H6k72NqvzLtUxQBN5lZs17QJO0OHA5MBE7B9XWQznxcRz4PSTrAGvtuaI32HrdU5zFJSXXmb61pft7Anfv1ZrZHhnKuBv5oZvf68za1je1Gt5XIsK2OIuD3ZjazyUzX30r0vCVwBh3IEuEZRC/E333egXv4mWIF7k4Y4Fhc00RHOVlSTO65xAjg38Bc4AepNm9JO6rtzn8WAgdJ2kJSHDgVeKq1FSRtj2saug7XU96euOa0CuBz3w5+ZCf26auSdvCmOQ53xx/lMeAkSVt6HQMlbS/3IkDMzP4J/IJWwpX7PJcDD6vj/TE/DZzmt30w8LF1ov8Rv867kk72ZckbHLha32qfPiOy2jzgh6kJday/646sOxc4y9dykDQ0dbxboRzXZBjoAsEgei9X0DRe/HW4i/JiYF86d3e/EndxfwiYaGbVuIv167gY+kuAmbRxR+nvon8GPAEsxr3d01aY6oNxMftfxl3IrzTXA9jLwJvA34BnO7FPi3DRRt/ANa3clab1dZwBPCIXeXYerilrKPCkpFeAW2ijn2Uz+7Mv+15JJa3lTWMqsJff9iU0vYB3lNOA7/nfwFJcp0ipbfxD0os0DSE9DRgg9yB+MU2bG9ui3eua663vb8AC31w2h7Yv/rOBGeEhddcI0VwDgRbwd+STzOzoXGsJBHJBqEEEAoFAICOhBhEIBAKBjIQaRCAQCAQyEgwiEAgEAhkJBhEIBAKBjASDCAQCgUBGgkEEAoFAICP/D8KaMuJbqsqZAAAAAElFTkSuQmCC\n"
311 |           },
312 |           "metadata": {
313 |             "needs_background": "light"
314 |           }
315 |         }
316 |       ]
317 |     },
318 |     {
319 |       "cell_type": "markdown",
320 |       "source": [
321 |         "# Dry bean"
322 |       ],
323 |       "metadata": {
324 |         "id": "DP0fdL00-wYq"
325 |       }
326 |     },
327 |     {
328 |       "cell_type": "code",
329 |       "execution_count": null,
330 |       "metadata": {
331 |         "id": "xBPJ8vmAEiJQ"
332 |       },
333 |       "outputs": [],
334 |       "source": [
335 |         "K = 4\n",
336 |         "N = 4\n",
337 |         "B = f(N+K-1) // f(K) // f(N-1)"
338 |       ]
339 |     },
340 |     {
341 |       "cell_type": "code",
342 |       "source": [
343 |         "# calculate H(B|Y=y), y is uniform\n",
344 |         "p_y = 0\n",
345 |         "P_ys = []\n",
346 |         "b_count = 0\n",
347 |         "H_y = 0\n",
348 |         "\n",
349 |         "for c0 in range(0, N+K-1):\n",
350 |         "  for c1 in range(c0+1, N+K-1):\n",
351 |         "    for c2 in range(c1+1, N+K-1):\n",
352 |         "      for c3 in range(c2+1, N+K-1):\n",
353 |         "          b_count += 1\n",
354 |         "          Ns = [0, 0, 0, 0, 0, 0]\n",
355 |         "          Ns[c0-0] += 1\n",
356 |         "          Ns[c1-1] += 1\n",
357 |         "          Ns[c2-2] += 1\n",
358 |         "          Ns[c3-3] += 1\n",
359 |         "          p_y = f(K) / (N**K) / (f(Ns[0])*f(Ns[1])*f(Ns[2])*f(Ns[3])*f(Ns[4])*f(Ns[5]))\n",
360 |         "          P_ys.append(p_y)\n",
361 |         "          H_y += -p_y * math.log2(p_y)\n",
362 |         "\n",
363 |         "# print(sum(P_ys))\n",
364 |         "# print(b_count)\n",
365 |         "print('B', B)\n",
366 |         "print(math.log2(B))\n",
367 |         "print('H_y', H_y)\n",
368 |         "print(2**H_y)"
369 |       ],
370 |       "metadata": {
371 |         "colab": {
372 |           "base_uri": "https://localhost:8080/"
373 |         },
374 |         "outputId": "7c0c2d06-1666-4b67-8c66-130dfece1472",
375 |         "id": "ObFHHIEsEiJa"
376 |       },
377 |       "execution_count": null,
378 |       "outputs": [
379 |         {
380 |           "output_type": "stream",
381 |           "name": "stdout",
382 |           "text": [
383 |             "B 35\n",
384 |             "5.129283016944966\n",
385 |             "H_y 4.81510800723783\n",
386 |             "28.150877863635262\n"
387 |           ]
388 |         }
389 |       ]
390 |     },
391 |     {
392 |       "cell_type": "code",
393 |       "source": [
394 |         "# calculate H(B|S=s)\n",
395 |         "U_range = np.arange(1, 1176, 110)\n",
396 |         "# U_range = np.arange(1, B, 3000)\n",
397 |         "H_ss = np.zeros_like(U_range).astype(float)\n",
398 |         "\n",
399 |         "for i, U in enumerate(U_range):\n",
400 |         "  H_s = 0\n",
401 |         "  p_s = 0\n",
402 |         "  choices = np.random.choice(B, U, p=P_ys)\n",
403 |         "  # print(len(choices), choices)\n",
404 |         "  for u in range(U):\n",
405 |         "    p_s = sum([1 for c in choices if c == u]) / U\n",
406 |         "    # print(p_s)\n",
407 |         "    if p_s != 0:\n",
408 |         "      H_s += -p_s * math.log2(p_s)\n",
409 |         "  print(H_s)\n",
410 |         "  H_ss[i] = H_s\n",
411 |         "\n",
412 |         "print('H_ss', H_ss)"
413 |       ],
414 |       "metadata": {
415 |         "colab": {
416 |           "base_uri": "https://localhost:8080/"
417 |         },
418 |         "outputId": "d4fe9fbc-70dd-4ccb-9947-13b5786c4743",
419 |         "id": "z8SJ6CCVEiJb"
420 |       },
421 |       "execution_count": null,
422 |       "outputs": [
423 |         {
424 |           "output_type": "stream",
425 |           "name": "stdout",
426 |           "text": [
427 |             "0\n",
428 |             "4.488025774313082\n",
429 |             "4.702617634802406\n",
430 |             "4.802929804553897\n",
431 |             "4.750954940629122\n",
432 |             "4.744101917925756\n",
433 |             "4.8117979838104485\n",
434 |             "4.768437327163887\n",
435 |             "4.815896365695055\n",
436 |             "4.789852490143935\n",
437 |             "4.782352207555449\n",
438 |             "H_ss [0.         4.48802577 4.70261763 4.8029298  4.75095494 4.74410192\n",
439 |             " 4.81179798 4.76843733 4.81589637 4.78985249 4.78235221]\n"
440 |           ]
441 |         }
442 |       ]
443 |     },
444 |     {
445 |       "cell_type": "code",
446 |       "source": [
447 |         "H_ss = np.array([0., 4.48802577, 4.70261763, 4.8029298, 4.75095494, 4.74410192, 4.81179798, 4.76843733, 4.81589637, 4.78985249, 4.78235221])\n",
448 |         "H_y = 4.81510800723783\n",
449 |         "plt.plot(np.arange(11)*1176/11, H_y-H_ss)\n",
450 |         "plt.xlabel('Number of clients')\n",
451 |         "plt.ylabel('Privacy leakage (in bits)')\n",
452 |         "plt.show()"
453 |       ],
454 |       "metadata": {
455 |         "colab": {
456 |           "base_uri": "https://localhost:8080/",
457 |           "height": 281
458 |         },
459 |         "outputId": "c9ab6df1-93a5-481a-a404-dfc8767a4395",
460 |         "id": "7YxzWs4OEiJb"
461 |       },
462 |       "execution_count": null,
463 |       "outputs": [
464 |         {
465 |           "output_type": "display_data",
466 |           "data": {
467 |             "text/plain": [
468 |               "<Figure size 432x288 with 1 Axes>"
469 |             ],
470 |             "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEICAYAAABVv+9nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xcdX3/8ddnLnudSTab3RkgJATYWSiiqI2K2lbwfq+1+vvJrz/1pyj6q7XS1p/VX2utbf+w9UpbW6Xgtdb7Da2K/ARErQWSCAISkhBCIAV2E5LsLXub/fz+OGcmk2WzO7vZM5cz7+fjMY89t5nzOZzwme+c8znfr7k7IiISP4l6ByAiItFQghcRiSkleBGRmFKCFxGJKSV4EZGYUoIXEYmpVJQfbmZ7gVGgCMy6+5Yo9yciIsdEmuBDF7v7gWo27Ovr882bN0ccjohIfGzbtu2Au/cvtK4WCb5qmzdvZuvWrfUOQ0SkaZjZ/SdaF/U1eAd+aGbbzOyyiPclIiIVom7B/4a77zezHHCdme1w95sqNwgT/2UAmzZtijgcEZHWEWkL3t33h3+HgG8CT11gmyvdfYu7b+nvX/AykoiIrEBkCd7Mus0sW5oGng/cGdX+RETkeFFeoskD3zSz0n7+zd1/EOH+RESkQmQJ3t33ABdE9fkiIrI4PckqIhJTTZ/gZ4pz/NONu7lp53C9QxERaShNn+BTCeOTP97DD+56uN6hiIg0lKZP8GbGYD7D7kfG6h2KiEhDafoEDzCQy7JzaBSNLysickwsEnwhl+HwxAwHxqbrHYqISMOIRYIfzGcB2DU0WudIREQaRywSfCGfAWD3kK7Di4iUxCLB57LtZDtS7HxELXgRkZJYJHgzo5DLsEuVNCIiZbFI8BBch9clGhGRY2KT4AdyGQ6OT3NwbKreoYiINITYJPhCuZJGrXgREYhRgh8MK2mU4EVEArFJ8Kes6SDTnmK3KmlERIAYJXgzYyCXYacqaUREgBgleAgu0+gSjYhIIFYJvpDLcmBsikPj6pNGRCRWCX5AN1pFRMpileALuVKC141WEZFYJfgNPZ10tyXVZYGICDFL8KVKGrXgRURiluAhGN1JLXgRkRgm+MF8hqHRKY5MzNQ7FBGRuopdgi8P/jGsyzQi0tril+BzQadjeqJVRFpd7BL8hp5OOtOqpBERiV2CTyRUSSMiAjFM8ICG7xMRIaYJfiCf4eGRSUYmVUkjIq0rlgl+MLzRqjFaRaSVxTLBl0sldZlGRFpY5AnezJJm9gsz+27U+yo5fV0X7akEOzW6k4i0sFq04N8B3F2D/ZQly5U0asGLSOuKNMGb2enAS4CrotzPQgq5jK7Bi0hLi7oF/zHgXcBcxPt5jEI+y/7DRxmbmq31rkVEGkJkCd7MXgoMufu2Jba7zMy2mtnW4eHhVdv/QDj4h1rxItKqomzBPxN4uZntBb4EPNvM/nX+Ru5+pbtvcfct/f39q7bzwXxQKrlLN1pFpEVFluDd/T3ufrq7bwZeA1zv7v8zqv3Nt3FdJ22phG60ikjLimUdPEAqmeCsvm614EWkZdUkwbv7je7+0lrsq9JgPqsWvIi0rNi24CEolXzw0FEmplVJIyKtJ94JPq9KGhFpXamlNjCzHEFFzGnAUeBOYKu717y2fbkK5UqaMZ5wek+doxERqa0TJngzuxh4N9AL/AIYAjqAVwBnm9nXgA+7+0gtAl2JM3q7SCdN1+FFpCUt1oJ/MfBmd983f4WZpYCXAs8Dvh5RbCctqKTJqJJGRFrSCRO8u/+fRdbNAt+KJKJVNpDPcMeDR+odhohIzS15k9XM3mFmayxwtZltN7Pn1yK41TCYy/LAoQmOThfrHYqISE1VU0XzxvA6+/OBdcBrgQ9EGtUqKuQzuMO9w7oOLyKtpZoEb+HfFwOfd/e7KpY1vELY6diuIV2HF5HWUk2C32ZmPyRI8NeaWZY6dP+7Upv7ukkljF0avk9EWsySdfDApcATgT3uPmFm64E3RBvW6kknE5zZ161SSRFpOdW04K9z9+3ufhjA3Q8CH402rNVVyKtUUkRazwkTvJl1mFkv0Gdm68ysN3xtBjbUKsDVMJDLsu/RCSZnVEkjIq1jsUs0bwEuJ+iiYHvF8hHgH6MMarUN5jPMOewZHue809bUOxwRkZpY7EGnK4ArzOzt7v4PNYxp1RVyYZ80Q6NK8CLSMhbri+bZ7n49sN/MXjl/vbt/I9LIVtHmvi6SqqQRkRaz2CWaZwHXAy9bYJ0DTZPg21NJNq/vUi28iLSUxS7RvC/82zQlkYsp5LLsVIIXkRZSTV80683s78M+aLaZ2RVhLXxTKeQz3H9wgqlZVdKISGuopg7+S8Aw8LvAq8LpL0cZVBQK+SzFOee+A+P1DkVEpCaqSfCnuvtfu/t94etvgHzUga22cp80utEqIi2imgT/QzN7jZklwtd/A66NOrDVdmZfNwlDT7SKSMtYrExylKBaxggeePp8uCoJjAHvjDy6VdSRTnLGevVJIyKtY7EqmmwtA6mFQi6jBC8iLaOaSzSxUchn2HtgnOnZpuntWERkxVorweeyzM45ew+qkkZE4q+1EnxelTQi0jqqGfADM0sSlEaWt3f3fVEFFZWz+zOYlYbvO7Xe4YiIRGrJBG9mbwfeBzzCsaH6HHhChHFFoiOdZFNvl1rwItISqmnBvwM4JxzJqekVcll1OiYiLaGaa/APAEeiDqRWCvkM9x0YZ6aoShoRibdqWvB7gBvN7N+BqdJCd/9IZFFFqJDLMFN07j84zkAudqX+IiJl1bTg9wHXAW1AtuK1qHBM11vM7HYzu8vM3n9yoa6O8uhOug4vIjG3ZAve3VeamKeAZ7v7mJmlgZ+a2ffd/T9X+HmrYiBXqqQZ40X1DEREJGKL9UXzMXe/3My+Q1A1cxx3f/liH+zuTtBnDUA6fD3mc2qtsy3J6es61WWBiMTeYi34UudiH1rph4f189uAAeDj7n7zSj9rNRVyWfUqKSKxt1hnY9vCvz9e6Ye7exF4opn1AN80s/Pd/c7KbczsMuAygE2bNq10V8tSyGf46a4DzBbnSCVb6mFeEWkhJ8xuZvYdM3tZeP18/rqzzOyvzOyN1ezE3Q8DNwAvXGDdle6+xd239Pf3Lyf2FSvkskwX59j36ERN9iciUg+LNV/fDPwmsMPMbjWz75nZ9Wa2B/gksM3dP3WiN5tZf9hyx8w6gecBO1Yx9hUrje60U5U0IhJji12ieRh4F/AuM9tM0HnLUWCnu1fT9D0V+Gx4HT4BfMXdv3vSEa+CgTDB7x4aBU6pbzAiIhGpqrMxd98L7F3OB7v7L4EnLT+k6HW3p9jQo0oaEYm3lr3DWMhndIlGRGKtdRN8LsO9w2MU5+pemi8iEomqEryZdZrZOVEHU0uFfJbp2TkeUCWNiMTUkgnezF4G3Ab8IJx/opldE3VgUStV0ug6vIjEVTUt+L8EngocBnD324AzI4ypJgbKpZJ6olVE4qmaBD/j7vP7g2/6C9fZjjSnre1gt1rwIhJT1ZRJ3mVm/wNImlkB+EPgP6INqzYG8hrdSUTiq5oW/NuBxxF0//tFYAS4PMqgaqWQy7B7aIw5VdKISAxV0x/8BPBn4StWCrkMkzNzPHjoKJvWd9U7HBGRVbVkgj9Bf/BHgK3AJ919MorAaqGQD0d3GhpVgheR2KnmEs0egoE7/iV8jQCjwGA437QGVCopIjFWzU3WZ7j7Uyrmv2Nmt7r7U8zsrqgCq4W1nWnya9pVKikisVRNCz5jZuWROMLpTDg7HUlUNTSYz6pUUkRiqZoW/J8QDJh9L2AEDzn9vpl1A5+NMrhaGMhl+PKtDzA35yQSVu9wRERWTTVVNN8L69/PDRfdU3Fj9WORRVYjhVyWieki+w8fZWOvbrSKSHxU1R88UADOATqAC8wMd/9cdGHVzmC+NPjHmBK8iMRKNZ2NvQ/4h/B1MfB3wMsjjqtmjlXS6EariMRLNTdZXwU8B3jY3d8AXACsjTSqGurpaqM/267BP0QkdqpJ8EfdfQ6YNbM1wBCwMdqwaquQy6gWXkRip5oEv9XMeggeatoGbAd+HmlUNTaYz7L7kVHc1SeNiMRHNTdZ3+ZB5vuEmf0AWAPcE21YtTWQyzA+XeShI5Oc1tNZ73BERFZFNS34q0sT7r6XoOuC70UVUD0UNPiHiMRQNQl+v5n9E4CZrQN+CPxrpFHV2GDY6ZieaBWROFkywbv7e4ExM/sEQXL/sLt/OvLIamhddxt9mTZ2qZJGRGLkhNfgzeyVFbM3A+8FbgHczF7p7t+IOrhaGshl2KlaeBGJkcVusr5s3vwvgHS43IFYJfhCLsu3frEfd8dMfdKISPM7YYIPH2pqGYP5DKNTszwyMsUpazvqHY6IyEmrZkSnDuBSgnFZy5nP3d8YYVw1N5ALbrTufGRUCV5EYqGaKprPA6cALwB+DJxOMKJTrBTyGt1JROKlmgQ/EFbSjLv7Z4GXAE+LNqza68u009vdxm7daBWRmKgmwc+Efw+b2fkEHY3logupfgZyGZVKikhsVJPgrwwfcHovcA3wK4Iug2OnkMuwU33SiEhMVDOi01Xh5I+Bs6r9YDPbCHwOyBOUVV7p7lesJMhaGcxnGZmcZXh0itwa3WgVkeZWzYAfeTO72sy+H86fZ2aXVvHZs8CfuPt5wIXA28zsvJMLN1qFnG60ikh8VHOJ5jPAtcBp4fxO4PKl3uTuD7n79nB6FLgb2LCyMGtjIK9Ox0QkPqpJ8H3u/hVgDsDdZ4HicnZiZpuBJxF0edCw+jPtrO1MqwUvIrFQTYIfN7P1BNfRMbMLgSPV7sDMMsDXgcvdfWSB9ZeZ2VYz2zo8PFztx0bCzBjMZ9itShoRiYFqEvwfE1TPnG1mPyO4cfr2aj7czNIEyf0LJ+qczN2vdPct7r6lv7+/yrCjM5DLsnNIlTQi0vyqqaLZbmbPAs4BDLjH3WeWeBsW9Nh1NXC3u3/kpCOtkUIuwxcnZjgwNk1/tr3e4YiIrFi13QVXGjQzqugu+JnAa4E7zOy2cNn/dfeGHg2qNPjHrqFRJXgRaWrL6S640pLdBbv7Twla/E2l1CfN7qExnnF2X52jERFZOXUXPE8u2062I6VSSRFpetXcZG0pQSVNVn3SiEjTU4JfQCGX0QDcItL0lOAXMJDLcHB8moNjU/UORURkxarpi2abmb0t7FGyJRTKlTRqxYtI86qmBf/fCfqhudXMvmRmL7CYj0o9qNGdRCQGlkzw7r7b3f8MGAT+DfgUcL+Zvd/MeqMOsB5OWdNBpj3FblXSiEgTq+oavJk9Afgw8EGCrgdeDYwA10cXWv2YGQO5DDtVSSMiTWzJrgrMbBtwmKDbgXe7e+nO481m9swog6unwXyG63fUt/MzEZGTUU0L/tXu/hx3/7eK5A6Au5+oO4OmV8hlOTA2xaHx6XqHIiKyItUk+DeZWU9pxszWmdnfRBhTQxjQjVYRaXLVJPgXufvh0oy7HwJeHF1IjeHY8H260SoizamaBJ80s3K3imbWCcS+m8UNPZ10tyXVZYGINK0lb7ICXwB+ZGafDuffAHw2upAaQ6mSRl0WiEizqmbAj781s18CzwkX/bW7XxttWI1hIJflJ7tUSSMizamaFjzu/n3g+xHH0nAG8xm+vv1BjkzMsLYrXe9wRESWpZq+aC40s1vNbMzMps2saGaPGTw7jsqDfwzrRquINJ9qbrL+I3AJsAvoBN4EfDzKoBpFIRd0OqYnWkWkGVXVVYG77waS7l50908DL4w2rMawoaeTzrQqaUSkOVVzDX7CzNqA28zs74CHaJF+5BOJoJJGtfAi0oyqSdSvDbf7A2Ac2Aj8bpRBNZJCLqMWvIg0pWoS/K8D7u4j7v5+d//j8JJNSxjIZ3h4ZJKRyZl6hyIisizVJPiXATvN7PNm9lIzq6q0Mi4GwxuteuBJRJpNNQN+vAEYAL5KUE1zr5ldFXVgjaJcKqnLNCLSZKp90GnGzL4POEGp5CsIyiVj7/R1XbSnEuzU6E4i0mSqedDpRWb2GYI6+N8FrgJOiTiuhpEsV9KoBS8izaWaFvzrgC8Db5k/4EerKOQy3Lr3UL3DEBFZlmquwV/i7t9q1eQOUMhn2X/4KGNTs/UORUSkaidM8Gb20/DvqJmNVLxGW6UvmpLS4B+qpBGRZnLCBO/uvxH+zbr7mopX1t3X1C7E+ivkg1LJXbrRKiJNZNFLNGaWNLMdtQqmUW1c10lbKqEbrSLSVBZN8O5eBO4xs001iqchpZIJzurrVgteRJpKNVU064C7zOwWgr5oAHD3ly/2JjP7FPBSYMjdzz+pKBvAYD7L9n2qpBGR5lFNgn/vCj/7MwR9yX9uhe9vKIVchmtu/y8mpmfpamup3hpEpEmdMFOZWQfwVoJuCu4Arnb3qusE3f0mM9t8sgE2inKXBUNjPOH0njpHIyKytMWuwX8W2EKQ3F8EfLgmETWoY5U0utEqIs1hsWsN57n74wHM7GrgligCMLPLgMsANm1q3Hu5Z/R2kU6aKmlEpGks1oIvd4C+nEszy+XuV7r7Fnff0t/fH9VuTlpQSZNRJY2INI3FWvAXVDyxakBnOG8EA4C01MNOEAz+cceDR+odhohIVRZ7kjU57+nV1HKeZDWzLwI/B84xswfN7NLVDLweBnNZHjg0wdHpYr1DERFZUmT1fu5+SVSfXS+FfAZ3uHd4jPM3rK13OCIii6pmyD4JlTod2zWk6/Ai0viU4Jdhc183qYSpVFJEmoIS/DKkkwnO7OtWqaSINAUl+GUq5FUqKSLNQQl+mQq5LPsenWByRpU0ItLYlOCXqZDPMOewZ3h86Y1FROpICX6ZCrmwTxpV0ohIg1OCX6bNfV0kVUkjIk1ACX6Z2lNJNq/vUgteRBqeEvwKFHJZlUqKSMNTgl+BQj7D/QcnmJpVJY2INC4l+BUo5LMU55z7DqiSRkQalxL8CpT7pNGNVhFpYErwK3BmXzcJQ0+0ikhDU4JfgY50ks3r1SeNiDQ2JfgVGshllOBFpKEpwa9QIZ9h74Fxpmfn6h2KiMiClOBXqJDLMjvn7D2oShoRaUxK8CtUyKuSRkQamxL8Cp3dn8FMnY6JSONSgl+hjnSSTb1dasGLSMNSgj8JhVyWux8eYaaoG60i0niU4E/C405bw57hcZ7819fxti9s56tbH2BodLLeYYmIAJCqdwDN7G0XD/Brp2a58Z5hbrhniH+/4yEAHr9hLRef089F5+a44PQekgmrc6Qi0orM3esdQ9mWLVt869at9Q5jRdydXz00EiT7HUNs33eIOYd1XWmeNdjPxefm+K1CP+u62+odqojEiJltc/ctC65Tgo/G4Ylpbtp1gBt3DHHjzmEeHZ8mYfDEjT08+9wcF52T43GnrcFMrXsRWTkl+Dorzjl37D/CDTuGuPGeIW5/8AgAuWw7F53Tz8Xn5HhmoY81Hek6RyoizUYJvsEMj05x087guv1NO4cZmZwllTC2bF7HxefkuPjcHIVcRq17EVmSEnwDmy3OsX3fYW64Z4gbdgyx4+HgwakNPZ3l1v0zBtbT1ab74SLyWErwTeShI0fLN2p/uvsAE9NF2lIJLjxrPReHCX9zX3e9wxSRBqEE36SmZots3XuIG3YMcf09Q+wZDjo2O7Ovm2cN9nPeaWvY1NvFGeu7yGc7SKgcU6TlKMHHxP0Hx8s19z+/9yBTFV0Vt6USbFzXyRnru9nU21VO/Jt6u9jY20VHOlnHyEUkKnVL8Gb2QuAKIAlc5e4fWGx7JfjqzRTn+K/DR7n/4AT7Hg1e9x8cZ9+jR9l3cJzx6eJx25+ypiNI/OuPT/6bervo7W5rmBu6s8U5jhyd4fDRGQ5PzHDk6HT4N3gV55w5d4pzwbMHpem5cDp4wdzcwtNFd9w9/ByOmy6//7jPC6qgutqS9GXa6c+205dppy/TVp7OZdvp7W4jldSD4VJ7iyX4yO7cmVkS+DjwPOBB4FYzu8bdfxXVPltJOpngjPXdnLH+sdfj3Z2D49NB4j84UfElMM5Pdg3zyMjUcdtn2lNs7O3ijDDxbwz/ntHbzak9HaRXkLgmZ4pBop6Y4fDEdDlpH5mY4XCYtOfPH5mYYXRqdtHPTRgkzIJXomLaIJGw4+aT4bzNny6/3459XsV0MtwunUyU3zMxXeT2Bw8zPDrFxLwvTwAzWNfVRn+mnb5sW/BlkGmnL3vsb+lLobertl8GxTlnbHKWkckZRidnGS39nSrNB+vGwunS+lTSWNuZZk1HmjWd6XA6xdqu+cvSrOlM0ZlONkxDIUrzGwWlRkepYVB0DxsVVEw/tiFRnDv2nmTCOH/D2lWPNcrSjKcCu919D4CZfQn4bUAJPmJmFrYy23nypnWPWX90usgDh8Lk/+gE+w6Os+/RCXYOjXL9jiGmKzpPSyaMDT2d5db/Gb1dZDpSQYt6IkzgFa3s0vzkzIk7YEsmjJ7ONGu70vR0psllOxjMZcP5Nnq60vR0BQmkpzNNT1cbPZ3BfCN0+zAxPcuB0WmGxyYZHp1meGyKA6NTHBibYjj8+4t9wZfB0ZmFvwx6u9oW/DVwbFkwvbYzzcT0sSQ8Oi8Jl/6OLLCsND3/19xC2pIJsh2p8JUm056iOOfsPTDByGRwbhf6YquUThprOoKkny19GXQ+9otgoS+NNZ3pEzYk3J2ZojM5W2RypsjUzByTM0UmZ+bKyybLy4pMzc4d9/e49RXLphZ4f3GBZFycczxM1sW5aK549GXa2frnz131z40ywW8AHqiYfxB4WoT7kyp1tiUZzGcZzGcfs6445zwyMsn9Byd44NEJ7n90vDz9vTse4vDETHnbtlSCdWFSXtuZZmNvF4/fkA4TdLCspyJpl+Yz7ammbul1taXYtD7FpvVdS247PjVbTvqlL4Dhsenjlu29f5wDY1OLfikupi2VYE2YmEtJOpftKCfr0rI15en0cck825Gq6h7NTHGO0clZjhydYSS8ZDYyOcPI0dnydGndSLjd/kNHy8tniosnx662JGs60nS2JZmal4xXmlfNoCOVpCOdoCOdpCOdpD1Vmk7Q291WXt+eSpJKWvnXXsKMZOL4X3jJ8nSwXekXYTBtJCt+SSYrfyGW5is+x8JlyQS0p6K5R1b34mozuwy4DGDTpk11jkaSCeO0nk5O6+nk6Wevf8z6oCU3y7quNt24rUJ3e4ru9tSSpa3uztjULAfGpo/7JXBkYobu9mPJuDKRZ8IkHVVymC+dDBJi7wr6U3J3JmfmKr4Ujn1BHJkIvhBKy47OFMsJOEi+xxJ0ezpJRypR/ltK2uUEXkrW4bK2ZKKpGxMnK8oEvx/YWDF/erjsOO5+JXAlBDdZI4xHVsHa8Ge1rC4zCxN3mjNj+JyDmdHZlqSzLckpazvqHU7LiPJOz61AwczONLM24DXANRHuT0REKkTWgnf3WTP7A+BagjLJT7n7XVHtT0REjhfpNXh3/x7wvSj3ISIiC9OTGSIiMaUELyISU0rwIiIxpQQvIhJTSvAiIjHVUN0Fm9kwcP8K394HHFjFcBqRjjEedIzNr5GO7wx3719oRUMl+JNhZltP1GVmXOgY40HH2Pya5fh0iUZEJKaU4EVEYipOCf7KegdQAzrGeNAxNr+mOL7YXIMXEZHjxakFLyIiFZo+wZvZC83sHjPbbWbvrnc8K2VmG83sBjP7lZndZWbvCJf3mtl1ZrYr/LsuXG5m9vfhcf/SzJ5c3yOonpklzewXZvbdcP5MM7s5PJYvh91LY2bt4fzucP3mesZdLTPrMbOvmdkOM7vbzJ4et/NoZn8U/ju908y+aGYdzX4ezexTZjZkZndWLFv2eTOz14fb7zKz19fjWEqaOsFXDOz9IuA84BIzO6++Ua3YLPAn7n4ecCHwtvBY3g38yN0LwI/CeQiOuRC+LgP+ufYhr9g7gLsr5v8W+Ki7DwCHgEvD5ZcCh8LlHw23awZXAD9w93OBCwiONTbn0cw2AH8IbHH38wm6A38NzX8ePwO8cN6yZZ03M+sF3kcwPOlTgfeVvhTqwt2b9gU8Hbi2Yv49wHvqHdcqHdu3gecB9wCnhstOBe4Jpz8JXFKxfXm7Rn4RjOz1I+DZwHcBI3hgJDX/nBKMJfD0cDoVbmf1PoYljm8tcN/8OON0Hjk23nJveF6+C7wgDucR2AzcudLzBlwCfLJi+XHb1frV1C14Fh7Ye0OdYlk14U/YJwE3A3l3fyhc9TCQD6eb9dg/BrwLKI0wvR447O6z4XzlcZSPMVx/JNy+kZ0JDAOfDi9DXWVm3cToPLr7fuBDwD7gIYLzso14nceS5Z63hjqfzZ7gY8fMMsDXgcvdfaRynQdNgqYtezKzlwJD7r6t3rFEKAU8Gfhnd38SMM6xn/VALM7jOuC3Cb7MTgO6eeyljdhpxvPW7Am+qoG9m4WZpQmS+xfc/Rvh4kfM7NRw/anAULi8GY/9mcDLzWwv8CWCyzRXAD1mVhpdrPI4yscYrl8LHKxlwCvwIPCgu98czn+NIOHH6Tw+F7jP3YfdfQb4BsG5jdN5LFnueWuo89nsCT42A3ubmQFXA3e7+0cqVl0DlO7Ev57g2nxp+evCu/kXAkcqfko2JHd/j7uf7u6bCc7V9e7+e8ANwKvCzeYfY+nYXxVu39AtKHd/GHjAzM4JFz0H+BUxOo8El2YuNLOu8N9t6Rhjcx4rLPe8XQs838zWhb90nh8uq49639RYhZsiLwZ2AvcCf1bveE7iOH6D4OffL4HbwteLCa5V/gjYBfw/oDfc3ggqiO4F7iCoaKj7cSzjeC8CvhtOnwXcAuwGvgq0h8s7wvnd4fqz6h13lcf2RGBreC6/BayL23kE3g/sAO4EPg+0N/t5BL5IcE9hhuCX2KUrOW/AG8Nj3Q28oZ7HpCdZRURiqtkv0YiIyAkowYuIxJQSvIhITCnBi4jElBK8iEhMKcFL3ZiZm9mHK+bfaWZ/uUqf/Rkze9XSW570fl4d9hh5w3LjCrsxWFHneGZ2kZk9YyXvldahBC/1NAW80sz66h1IpYqnMatxKfBmd794uftx9ze5+6+W+77QRYASvAjRhwoAAAM0SURBVCxKCV7qaZZg6LM/mr9ifgvczMbCvxeZ2Y/N7NtmtsfMPmBmv2dmt5jZHWZ2dsXHPNfMtprZzrAfnFJf9B80s1vDfrzfUvG5PzGzawieypwfzyXh599pZn8bLvsLggfUrjazDy7wnj8N33O7mX1ggfU3mtmWcPr5ZvZzM9tuZl8N+yTCzPaa2fvD5XeY2blhZ3RvBf7IzG4zs98Mf0ncGe7rpur+80vcLaelIhKFjwO/NLO/W8Z7LgB+DXgU2ANc5e5PtWCQlLcDl4fbbSbok/ts4AYzGwBeR/BY+VPMrB34mZn9MNz+ycD57n5f5c7M7DSCPsx/naCf8x+a2Svc/a/M7NnAO91967z3vIigQ66nuftE2E/4gsJfMH8OPNfdx83sT4E/Bv4q3OSAuz/ZzH4/3NebzOwTwJi7fyj8jDuAF7j7fjPrWcZ/S4kxteClrjzoMfNzBANIVOtWd3/I3acIHhUvJeg7CJJ6yVfcfc7ddxF8EZxL0DfI68zsNoLumNcTDNoAcMv85B56CnCjB51rzQJfAH5riRifC3za3SfC43x0kW0vJBiw5mdhXK8HzqhYX+p4btu846v0M+AzZvZmggE4RNSCl4bwMWA78OmKZbOEDRAzSwBtFeumKqbnKubnOP7f9Px+OJygD5G3u/txHUCZ2UUEXfvWgwHXufslJ1hfOr4iJ/h/1t3famZPA14CbDOzX3f3ZumxUSKiFrzUXdi6/QrHhngD2EtwSQTg5UB6BR/9ajNLhNflzyIYdeda4H9b0DUzZjZowYAci7kFeJaZ9VkwTOQlwI+XeM91wBvMrCvczwkv0QD/CTwzvISEmXWb2eASnz8KZEszZna2u9/s7n9BMODIxhO+U1qGErw0ig8DldU0/0KQVG8nGP5tJa3rfQTJ+fvAW919EriK4CbqdgsGV/4kS/yS9aAb2HcTdId7O7DN3b+9xHt+QNCl7Nbwsss7F9l2GPhfwBfN7JfAzwkuJy3mO8DvlG6yAh8s3QQG/iOMU1qcepMUEYkpteBFRGJKCV5EJKaU4EVEYkoJXkQkppTgRURiSgleRCSmlOBFRGJKCV5EJKb+P9NzPXLQKa35AAAAAElFTkSuQmCC\n"
471 |           },
472 |           "metadata": {
473 |             "needs_background": "light"
474 |           }
475 |         }
476 |       ]
477 |     }
478 |   ]
479 | }


--------------------------------------------------------------------------------
/diagrams/FbFTL_diagram.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/wfwf10/Feature-based-Federated-Transfer-Learning/6e332076b157d3d7dfcfc46b3170c966747e5510/diagrams/FbFTL_diagram.png


--------------------------------------------------------------------------------
/history.txt:
--------------------------------------------------------------------------------
1 | 
2 | 


--------------------------------------------------------------------------------
/main_GLOBECOM_CIFAR10_VGG16.py:
--------------------------------------------------------------------------------
  1 | import torch
  2 | import time
  3 | import numpy as np
  4 | import torch
  5 | import torch.nn as nn
  6 | import torch.nn.functional as F
  7 | from torchvision import datasets
  8 | from torchvision import transforms
  9 | from torch.utils.data import DataLoader
 10 | import datetime
 11 | 
 12 | #######################################
 13 | ### PRE-TRAINED MODELS AVAILABLE HERE
 14 | ## https://pytorch.org/docs/stable/torchvision/models.html
 15 | from torchvision import models
 16 | #######################################
 17 | 
 18 | now = datetime.datetime.now()
 19 | 
 20 | # Device
 21 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
 22 | print('Device:', DEVICE)
 23 | 
 24 | ##########################
 25 | ### SETTINGS
 26 | ##########################
 27 | 
 28 | FL_type = 'FbFTL' 
 29 | train_set_denominator = 'full'  # 'full', int <= 50000  # pick a subset with 50000/int training samples
 30 | 
 31 | # Hyperparameters
 32 | NUM_CLASSES = 10
 33 | random_seed = 1 
 34 | learning_rate = 1e-2  
 35 | num_epochs = 200  
 36 | batch_size = 64  
 37 | momentum = 0.9  
 38 | lr_decay = 5e-4  
 39 | 
 40 | write_hist = True
 41 | 
 42 | def adjust_learning_rate(optimizer, epoch):
 43 |     """Sets the learning rate to the initial LR decayed by 2 every 30 epochs"""
 44 |     lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs))
 45 |     for param_group in optimizer.param_groups:
 46 |         param_group['lr'] = lr
 47 | 
 48 | if torch.cuda.is_available():
 49 |     torch.backends.cudnn.deterministic = True
 50 |     torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 
 51 | torch.manual_seed(random_seed) # sets the seed for generating random numbers.
 52 | 
 53 | if write_hist:
 54 |     file1 = open('history.txt', 'a')
 55 |     file1.write('\n \n \n Time:')
 56 |     file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' ' 
 57 |                 + str(now.hour) + ' ' + str(now.minute) + ' ' + str(now.second) 
 58 |                 + '     ' + FL_type + ',  train_deno:' + str(train_set_denominator))
 59 |     file1.close()
 60 | 
 61 | ##########################
 62 | ### CIFAR10 DATASET
 63 | ##########################
 64 | 
 65 | custom_transform = transforms.Compose([
 66 |     transforms.Resize((224, 224)),
 67 |     transforms.ToTensor(),
 68 |     transforms.Normalize(mean=[0.485, 0.456, 0.406],
 69 |                           std=[0.229, 0.224, 0.225])
 70 | ])
 71 | 
 72 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True)
 73 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform)
 74 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False)
 75 | 
 76 | if train_set_denominator == 'full':
 77 |     train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True)
 78 | else:
 79 |     selected_list = list(range(0, len(train_dataset), train_set_denominator))
 80 |     trainset_1 = torch.utils.data.Subset(train_dataset, selected_list)
 81 |     train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True)
 82 | 
 83 | ##########################
 84 | ### LOAD MODEL
 85 | ##########################
 86 | 
 87 | model = models.vgg16(pretrained=True)
 88 | for param in model.parameters():
 89 |     param.requires_grad = False
 90 | 
 91 | model.classifier[3].requires_grad = True
 92 | model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
 93 | 
 94 | ##########################
 95 | ### TRAIN MODEL
 96 | ##########################
 97 | 
 98 | model = model.to(DEVICE)
 99 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum)
100 | 
101 | def compute_accuracy(model, data_loader):
102 |     model.eval()
103 |     correct_pred, num_examples = 0, 0
104 |     with torch.no_grad():
105 |         for i, (features, targets) in enumerate(data_loader):
106 |             features = features.to(DEVICE)
107 |             targets = targets.to(DEVICE)
108 | 
109 |             logits = model(features)
110 |             _, predicted_labels = torch.max(logits, 1)
111 |             num_examples += targets.size(0)
112 |             correct_pred += (predicted_labels == targets).sum()
113 |         return correct_pred.float()/num_examples * 100
114 | 
115 | 
116 | def compute_epoch_loss(model, data_loader):
117 |     model.eval()
118 |     curr_loss, num_examples = 0., 0
119 |     with torch.no_grad():
120 |         for batch_idx, (features, targets) in enumerate(data_loader):
121 |             features = features.to(DEVICE)
122 |             targets = targets.to(DEVICE)
123 |             logits = model(features)
124 |             loss = F.cross_entropy(logits, targets, reduction='sum')
125 |             num_examples += targets.size(0)
126 |             curr_loss += loss
127 | 
128 |         curr_loss = curr_loss / num_examples
129 |         return curr_loss
130 |     
131 |     
132 | 
133 | start_time = time.time()
134 | for epoch in range(num_epochs):
135 |     model.train()
136 |     for batch_idx, (features, targets) in enumerate(train_loader):
137 |         
138 |         features = features.to(DEVICE)
139 |         targets = targets.to(DEVICE)
140 |             
141 |         ### FORWARD AND BACK PROP
142 |         logits = model(features)
143 |         cost = F.cross_entropy(logits, targets)
144 |         optimizer.zero_grad()
145 |         
146 |         cost.backward()
147 |         
148 |         ### UPDATE MODEL PARAMETERS
149 |         optimizer.step()
150 |         
151 |         ### LOGGING
152 |         if not batch_idx % 50:
153 |             print ('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f' 
154 |                    %(epoch+1, num_epochs, batch_idx, 
155 |                      len(train_loader), cost))
156 | 
157 |     model.eval()
158 |     accuracy = compute_accuracy(model, test_loader)
159 |     loss = compute_epoch_loss(model, test_loader)
160 |     with torch.set_grad_enabled(False): # save memory during inference
161 |         print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
162 |         if write_hist:
163 |             file1 = open('history.txt', 'a')
164 |             file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
165 |             file1.close()
166 | 
167 | 
168 |     print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))
169 |     
170 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))
171 | 
172 | with torch.set_grad_enabled(False): # save memory during inference
173 |     accuracy = compute_accuracy(model, test_loader)
174 |     print('Test accuracy: %.2f%%' % (accuracy))
175 |     if write_hist:
176 |         file1 = open('history.txt', 'a')
177 |         file1.write('\n Test accuracy: %.2f%%' % (accuracy))
178 |         file1.close()
179 | 
180 | 
181 | 


--------------------------------------------------------------------------------
/main_TMLCN_CIFAR10_VGG16.py:
--------------------------------------------------------------------------------
  1 | import torch
  2 | import time
  3 | import numpy as np
  4 | import torch
  5 | import torch.nn as nn
  6 | import torch.nn.functional as F
  7 | from torchvision import datasets
  8 | from torchvision import transforms
  9 | from torch.utils.data import DataLoader
 10 | import matplotlib.pyplot as plt
 11 | import datetime
 12 | import math
 13 | from collections import deque
 14 | import copy
 15 | 
 16 | #######################################
 17 | ### PRE-TRAINED MODELS AVAILABLE HERE
 18 | ## https://pytorch.org/docs/stable/torchvision/models.html
 19 | from torchvision import models
 20 | #######################################
 21 | 
 22 | now = datetime.datetime.now()
 23 | 
 24 | # Device
 25 | DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
 26 | print('Device:', DEVICE)
 27 | 
 28 | ##########################
 29 | ### SETTINGS
 30 | ##########################
 31 | 
 32 | FL_type = 'FbFTL'  # 'FL', 'FTLf', 'FTLc', 'FbFTL'
 33 | train_set_denominator = 'full'  # 'full', int <= 50000  # pick a subset with 50000/int training samples
 34 | 
 35 | # Hyperparameters
 36 | NUM_CLASSES = 10
 37 | U_clients = 6250  # number of clients, 50000/8
 38 | random_seed = 1  # transfer = True / False
 39 | learning_rate = 1e-2  # 1e-3, 0.05, 1e-2
 40 | num_epochs = 200  # 10, 300, 200
 41 | batch_size = 64  # 128, 128, (out of memory:64)
 42 | momentum = 0.9  # None, 0.9
 43 | lr_decay = 5e-4  # 1e-6, 5e-4
 44 | if FL_type == 'FL':
 45 |     transfer, full = False, True  # transfer or train model from scratch   # train whole model or last few layers
 46 |     sigma = 0.  # 0.5 relative std for addtive gaussian noise on gradients
 47 | elif FL_type == 'FTLf':
 48 |     transfer, full = True, True  
 49 |     sigma = 0.  # 0.3305 relative std for addtive gaussian noise on gradients
 50 | elif FL_type == 'FTLc':
 51 |     transfer, full = True, False 
 52 |     sigma = 0.  # 0.285 relative std for addtive gaussian noise on gradients
 53 | elif FL_type == 'FbFTL':
 54 |     transfer, full = True, False 
 55 |     sigma = 0  # 0.8? relative std for addtive gaussian noise on features
 56 |     saved_noise = True  # save noise at beginning
 57 | else:
 58 |     raise ValueError('Unknown FL_type: ' + FL_type)
 59 | relative_noise_type = 'all_std'  # 'individual', 'all_std'
 60 | packet_loss_rate = 0.  # 0, 0.05, 0.1, 0.15
 61 | quan_digit = 32  # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int
 62 | sparse_rate = 0.9  # ratio of uplink elements kept after sparsification: None or (0,1]
 63 | class ErrorFeedback(object):
 64 |     queue = deque(maxlen=U_clients)
 65 |     temp = deque()
 66 | if (quan_digit or sparse_rate) and FL_type != 'FbFTL':
 67 |     errfdbk = ErrorFeedback()
 68 |     # print(errfdbk.queue, errfdbk.temp)
 69 | 
 70 | write_hist = True
 71 | 
 72 | def adjust_learning_rate(optimizer, epoch):
 73 |     """Sets the learning rate to the initial LR decayed by 2 every 30 epochs"""
 74 |     lr = learning_rate * (0.5 ** ((epoch * 10) // num_epochs))
 75 |     for param_group in optimizer.param_groups:
 76 |         param_group['lr'] = lr
 77 | 
 78 | if torch.cuda.is_available():
 79 |     torch.backends.cudnn.deterministic = True
 80 |     torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 
 81 | torch.manual_seed(random_seed) # sets the seed for generating random numbers.
 82 | 
 83 | if write_hist:
 84 |     file1 = open('/data1/feng/LaTFL/history.txt', 'a')
 85 |     file1.write('\n \n \n Time:')
 86 |     file1.write(str(now.year) + ' ' + str(now.month) + ' ' + str(now.day) + ' ' + str(now.hour) + ' ' 
 87 |                 + str(now.minute) + ' ' + str(now.second) + '     ' + FL_type 
 88 |                 # + ',  train_deno:' + str(train_set_denominator)
 89 |                 # + ',  sigma:' + str(sigma)
 90 |                 # + ',  packet_loss_rate:' + str(packet_loss_rate)
 91 |                 + ',  quantization digits:' + str(quan_digit)
 92 |                 + ',  sparsification rate:' + str(sparse_rate)
 93 |                 )
 94 |     file1.close()
 95 | 
 96 | ##########################
 97 | ### CIFAR10 DATASET
 98 | ##########################
 99 | 
100 | custom_transform = transforms.Compose([
101 |     transforms.Resize((224, 224)),
102 |     transforms.ToTensor(),
103 |     transforms.Normalize(mean=[0.485, 0.456, 0.406],
104 |                           std=[0.229, 0.224, 0.225])
105 | ])
106 | 
107 | ## Note that this particular normalization scheme is necessary since it was used for pre-training the network on ImageNet.
108 | ## These are the channel-means and standard deviations for z-score normalization.
109 | 
110 | train_dataset = datasets.CIFAR10(root='data', train=True, transform=custom_transform,download=True)
111 | test_dataset = datasets.CIFAR10(root='data', train=False, transform=custom_transform)
112 | test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, num_workers=8, shuffle=False)
113 | 
114 | if train_set_denominator == 'full':
115 |     train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=8, shuffle=True)
116 |     train_set_len = len(train_dataset)
117 | else:
118 |     # print('len(train_dataset)', len(train_dataset))  # 50000
119 |     # selected_list = list(range(0, len(train_dataset), 2))
120 |     selected_list = list(range(0, len(train_dataset), train_set_denominator))
121 |     trainset_1 = torch.utils.data.Subset(train_dataset, selected_list)
122 |     train_loader = torch.utils.data.DataLoader(dataset=trainset_1, batch_size=batch_size, num_workers=8, shuffle=True)
123 |     train_set_len = len(selected_list)
124 | 
125 | # # Checking the dataset
126 | # for images, labels in train_loader:  
127 | #     print('Image batch dimensions:', images.shape)
128 | #     print('Image label dimensions:', labels.shape)
129 | #     break
130 | 
131 | # labels = torch.zeros(10, dtype=torch.long)
132 | # for batch_idx, (features, targets) in enumerate(train_loader):
133 | #     for t in targets:
134 | #         labels[t] += 1
135 | # print('labels', labels)
136 | 
137 | ##########################
138 | ### LOAD MODEL
139 | ##########################
140 | 
141 | class GaussianNoise(nn.Module):
142 |     """Gaussian noise regularizer.
143 |     Args:
144 |         sigma (float, optional): relative standard deviation used to generate the
145 |             noise. Relative means that it will be multiplied by the magnitude of
146 |             the value your are adding the noise to. This means that sigma can be
147 |             the same regardless of the scale of the vector.
148 |         is_relative_detach (bool, optional): whether to detach the variable before
149 |             computing the scale of the noise. If `False` then the scale of the noise
150 |             won't be seen as a constant but something to optimize: this will bias the
151 |             network to generate vectors with smaller values.
152 |     """
153 |     def __init__(self, sigma=0, is_relative_detach=False):
154 |         super().__init__()
155 |         self.sigma = sigma
156 |         self.is_relative_detach = is_relative_detach
157 |         if saved_noise:
158 |             self.register_buffer('noise', torch.empty(train_set_len*4096).normal_(mean=0,std=1))
159 |             self.i = 0
160 |         else:
161 |             self.register_buffer('noise', torch.tensor(0))
162 | 
163 |     def forward(self, x):
164 |         if self.training and quan_digit:
165 |             # print(x)
166 |             # print(x.dtype, torch.max(x), torch.min(x), torch.std(x))
167 |             x = torch.round((2**quan_digit-1) / torch.max(x) * x) * torch.max(x) / (2**quan_digit-1)
168 |             # print(x.dtype, torch.max(x), torch.min(x), torch.std(x))
169 |             # print(x)
170 |             # quit()
171 |         if self.training and self.sigma != 0:
172 |             if torch.cuda.is_available():
173 |                 torch.backends.cudnn.deterministic = True
174 |                 torch.cuda.manual_seed(random_seed) # Sets the seed for generating random numbers for the current GPU. 
175 |             torch.manual_seed(random_seed) # sets the seed for generating random numbers.
176 |             if relative_noise_type == 'individual':
177 |                 scale = self.sigma * x.detach() if self.is_relative_detach else self.sigma * x
178 |                 sampled_noise = self.noise.expand(*x.size()).float().normal_() * scale
179 |             elif relative_noise_type == 'all_std':
180 |                 x_std = torch.std(x.detach()) if self.is_relative_detach else torch.std(x)
181 |                 # print(*x.size())  # 64 4096
182 |                 if saved_noise:
183 |                     sampled_noise = torch.reshape(self.noise[self.i*batch_size*4096 : (self.i+1)*batch_size*4096],(-1, 4096)
184 |                                                   ).detach().float() * x_std * self.sigma
185 |                     self.i = self.i + 1 if (self.i+1)*batch_size*4096<train_set_len*4096 else 0
186 |                 else:
187 |                     sampled_noise = self.noise.expand(*x.size()).float().normal_(std=x_std*self.sigma)
188 |             x = x + sampled_noise
189 |         return x 
190 |         
191 |     def set_sigma(self, sigma):
192 |         self.sigma = sigma
193 | 
194 | if not full:
195 |     model = models.vgg16(pretrained=True)
196 |     for param in model.parameters():
197 |         param.requires_grad = False
198 | 
199 |     model.classifier[3].requires_grad = True
200 |     # model.classifier[6] = nn.Sequential(nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
201 |     if FL_type == 'FbFTL':
202 |         model.classifier[6] = nn.Sequential(GaussianNoise(sigma=sigma), nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 4096), 
203 |                                             nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
204 |     else:
205 |         model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), 
206 |                                             nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
207 | else:  # full
208 |     if transfer:
209 |         model = models.vgg16(pretrained=True)
210 |     if not transfer:
211 |         # model = models.vgg16(pretrained=False)
212 |         model = models.vgg16(pretrained=True)
213 |         def weight_reset(m):
214 |             if isinstance(m, nn.Linear):
215 |                 m.reset_parameters()
216 |             elif isinstance(m, nn.Conv2d):
217 |                 # print('m', m)
218 |                 # print(m.in_channels, m.out_channels)
219 |                 # print(m.__class__.__name__)
220 |                 # if m.out_channels > 256:
221 |                 m.reset_parameters()
222 |         for i in range(21, 31):
223 |             model.features[i].apply(weight_reset)
224 |         for i in range(7):
225 |             model.classifier[i].apply(weight_reset)
226 | 
227 |         for param in model.parameters():
228 |             param.requires_grad = True
229 |     
230 |     # model.classifier[3].requires_grad = True  # (4096, 4096, relu, dropout(0.5))
231 |     # model.classifier[6] = nn.Sequential(nn.Linear(4096, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
232 |     model.classifier[6] = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5), nn.Linear(4096, 512), 
233 |                                         nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, NUM_CLASSES))
234 | 
235 | 
236 | def Gaussian_noise_to_weights(m):
237 |     if sigma!=0:
238 |         with torch.no_grad():
239 |             for param in m.parameters():
240 |                 if param.requires_grad:
241 |                     if relative_noise_type == 'individual':
242 |                         # print(param.grad.view(-1))
243 |                         # print(param)
244 |                         # scale = sigma * param.detach()
245 |                         scale = sigma * param.grad.detach()  # todo: * math.sqrt(batch_size/8)
246 |                         noise = torch.tensor(0).to(DEVICE)
247 |                         sampled_noise = noise.expand(*param.size()).float().normal_() * scale
248 |                     elif relative_noise_type == 'all_std':
249 |                         param_grad_std = torch.std(param.grad.detach()) 
250 |                         noise = torch.tensor(0).to(DEVICE)
251 |                         sampled_noise = noise.expand(*param.size()).float().normal_(std=param_grad_std*sigma)  # todo: * math.sqrt(batch_size/8)
252 |                     # param = param + sampled_noise
253 |                     param.add_(sampled_noise)
254 | 
255 | def Errfdbk_to_weights(m):
256 |     print("inner model.apply")
257 |     with torch.no_grad():
258 |         for param in m.parameters():
259 |             print('len(param)', len(param))
260 |             if param.requires_grad:
261 |                 print('len(errfdbk.queue)', len(errfdbk.queue))
262 |                 # print(errfdbk.temp)
263 |                 print('len(errfdbk.temp)', len(errfdbk.temp))
264 |                 p_grad = param.grad.detach()
265 |                 if err_flag:
266 |                     p_grad += errfdbk.temp.popleft()
267 |                 p_grad_qs = copy.deepcopy(p_grad)
268 |                 if sparse_rate:
269 |                     pass
270 |                 if quan_digit:
271 |                     pass
272 |                 err = p_grad_qs - p_grad
273 |                 param.add_(err)
274 |                 errfdbk.temp.append(-err)
275 |     print('seems good')
276 | 
277 | if FL_type == 'FbFTL':
278 |     received_batches_FbFTL = np.ones(len(train_loader))
279 |     received_batches_FbFTL[:int(len(train_loader)*packet_loss_rate)] = 0
280 |     np.random.shuffle(received_batches_FbFTL)
281 |     
282 | def Packet_Received(batch_idx):
283 |     if FL_type == 'FbFTL':
284 |         return received_batches_FbFTL[batch_idx]
285 |     else:
286 |         return np.random.choice(2, p=[packet_loss_rate, 1-packet_loss_rate]) 
287 | 
288 | # for a in range(3):
289 | #     print('round', a)
290 | #     for i in range(50):
291 | #         print(Packet_Received(i))
292 | 
293 | # print(model)
294 | 
295 | # for name, param in model.named_parameters():
296 | #     print(name, torch.numel(param), param.requires_grad)
297 | # quit()
298 | 
299 | # model(torch.randn(1, 3, 224, 224)).mean().backward()
300 | # for name, param in model.named_parameters():
301 | #     print(name, param.grad)
302 | #     print('value', param.data)
303 | #     wait = input('next layer')
304 | 
305 | ##########################
306 | ### TRAIN MODEL
307 | ##########################
308 | 
309 | model = model.to(DEVICE)
310 | # if transfer:
311 | #     optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
312 | # else:
313 | #     optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum)  # , nesterov=True)
314 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=lr_decay, momentum=momentum)
315 | 
316 | 
317 | def compute_accuracy(model, data_loader):
318 |     model.eval()
319 |     correct_pred, num_examples = 0, 0
320 |     with torch.no_grad():
321 |         for i, (features, targets) in enumerate(data_loader):
322 |             features = features.to(DEVICE)
323 |             targets = targets.to(DEVICE)
324 | 
325 |             logits = model(features)
326 |             _, predicted_labels = torch.max(logits, 1)
327 |             num_examples += targets.size(0)
328 |             correct_pred += (predicted_labels == targets).sum()
329 |         return correct_pred.float()/num_examples * 100
330 | 
331 | 
332 | def compute_epoch_loss(model, data_loader):
333 |     model.eval()
334 |     curr_loss, num_examples = 0., 0
335 |     with torch.no_grad():
336 |         for batch_idx, (features, targets) in enumerate(data_loader):
337 |             features = features.to(DEVICE)
338 |             targets = targets.to(DEVICE)
339 |             logits = model(features)
340 |             loss = F.cross_entropy(logits, targets, reduction='sum')
341 |             num_examples += targets.size(0)
342 |             curr_loss += loss
343 | 
344 |         curr_loss = curr_loss / num_examples
345 |         return curr_loss
346 |     
347 |     
348 | 
349 | start_time = time.time()
350 | for epoch in range(num_epochs):
351 |     # adjust_learning_rate(optimizer, epoch)
352 |     
353 |     model.train()
354 |     for batch_idx, (features, targets) in enumerate(train_loader):
355 |         
356 |         if Packet_Received(batch_idx):
357 |             features = features.to(DEVICE)
358 |             targets = targets.to(DEVICE)
359 |                 
360 |             ### FORWARD AND BACK PROP
361 |             logits = model(features)
362 |             cost = F.cross_entropy(logits, targets)
363 |             optimizer.zero_grad()
364 |             
365 |             cost.backward()
366 |             
367 |             ### UPDATE MODEL PARAMETERS
368 |             optimizer.step()
369 | 
370 |             ### PRIVACY NOISE
371 |             if FL_type != 'FbFTL':
372 |                 # Gaussian_noise_to_weights(model, sigma * math.sqrt(batch_size/8))
373 |                 model.apply(Gaussian_noise_to_weights)
374 | 
375 |             ### Sparsification/Quantization with Error Feedback  # errfdbk.queue = deque(maxlen=U_clients)
376 |             if (quan_digit or sparse_rate) and FL_type != 'FbFTL':
377 |                 print('main loop: len(errfdbk.queue)', len(errfdbk.queue))
378 |                 if len(errfdbk.queue) < U_clients:
379 |                     errfdbk.temp = deque()
380 |                     err_flag = False
381 |                 else:
382 |                     errfdbk.temp = errfdbk.queue.popleft()
383 |                     err_flag = True
384 |                 model.apply(Errfdbk_to_weights)
385 |                 print('completed one cycle!')
386 |                 errfdbk.queue.append(errfdbk.temp)
387 |         
388 |             ### LOGGING
389 |             if not batch_idx % 50:
390 |                 print('Epoch: %03d/%03d | Batch %04d/%04d | Cost: %.4f' % (epoch+1, num_epochs, batch_idx, len(train_loader), cost))
391 |                 # if FL_type == 'FbFTL':
392 |                     # print(model.classifier[6][0].sigma)
393 |                     # model.classifier[6][0].set_sigma(sigma=0.4) 
394 | 
395 |     model.eval()
396 |     accuracy = compute_accuracy(model, test_loader)
397 |     loss = compute_epoch_loss(model, test_loader)
398 |     with torch.set_grad_enabled(False): # save memory during inference
399 |         print('Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
400 |         if write_hist:
401 |             file1 = open('/data1/feng/LaTFL/history.txt', 'a')
402 |             file1.write('\n Epoch: %03d/%03d | Test: %.3f%% | Loss: %.3f' % (epoch+1, num_epochs, accuracy, loss))
403 |             file1.close()
404 | 
405 | 
406 |     print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))
407 |     
408 | print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))
409 | 
410 | with torch.set_grad_enabled(False): # save memory during inference
411 |     accuracy = compute_accuracy(model, test_loader)
412 |     print('Test accuracy: %.2f%%' % (accuracy))
413 |     if write_hist:
414 |         file1 = open('/data1/feng/LaTFL/history.txt', 'a')
415 |         file1.write('\n Test accuracy: %.2f%%' % (accuracy))
416 |         file1.close()
417 | 
418 | # model.save_weights('/data1/feng/LaTFL/cifar10vgg.h5')
419 | 
420 | 


--------------------------------------------------------------------------------
/main_TMLCN_FLANT5_SAMSUM.py:
--------------------------------------------------------------------------------
  1 | # Source: https://www.philschmid.de/fine-tune-flan-t5
  2 | # tensorboard: tensorboard --logdir log
  3 | # Run the following in terminal before python3 FbFTL_LLM/test.py:  huggingface-cli login        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
  4 | 
  5 | from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq
  6 | from torchinfo import summary
  7 | from datasets import load_dataset
  8 | from random import randrange
  9 | from datasets import concatenate_datasets
 10 | import evaluate
 11 | import nltk
 12 | import numpy as np
 13 | from nltk.tokenize import sent_tokenize
 14 | nltk.download("punkt")
 15 | from huggingface_hub import HfFolder
 16 | from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
 17 | import sys
 18 | from collections import deque
 19 | 
 20 | HfFolder.save_token('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')
 21 | 
 22 | # FL parameters
 23 | FL_type = 'FbFTL'  # 'FL', 'FTLf'(same as FL), 'FTLc', 'FbFTL'
 24 | model_to_train = [False, 4, 0, True]  # available for 'FTLc' and 'FbFTL': [False, 4, 0, True]
 25 | # shared: (boolean), encoder: (int:0~8, index after which is trained, 10 to not train layer norm), decoder: (same as encoder), lm_head: (boolean)
 26 | train_set_denominator = -1  # -1 to use full training dataset, or 0 < int <= 14732 : pick a subset with [int] training samples
 27 | # Hyperparameters
 28 | NUM_CLASSES = 32128
 29 | learning_rate = 2e-4  # TODO: source code 5e-5, full model 2e-4
 30 | num_train_epochs = 20  # TODO: source code 5, full model 20
 31 | batch_size = 8  # source code 8
 32 | # TODO: implement following params
 33 | # if FL_type == 'FL':
 34 | #     transfer, full = False, True  # transfer or train model from scratch   # train whole model or last few layers
 35 | #     sigma = 0.  # 0.5 relative std for addtive gaussian noise on gradients
 36 | # elif FL_type == 'FTLf':
 37 | #     transfer, full = True, True  
 38 | #     sigma = 0.  # 0.3305 relative std for addtive gaussian noise on gradients
 39 | # elif FL_type == 'FTLc':
 40 | #     transfer, full = True, False 
 41 | #     sigma = 0.  # 0.285 relative std for addtive gaussian noise on gradients
 42 | # elif FL_type == 'FbFTL':
 43 | #     transfer, full = True, False 
 44 | #     sigma = 0  # 0.8? relative std for addtive gaussian noise on features
 45 | #     saved_noise = True  # save noise at beginning
 46 | # else:
 47 | #     raise ValueError('Unknown FL_type: ' + FL_type)
 48 | # relative_noise_type = 'all_std'  # 'individual', 'all_std'
 49 | # packet_loss_rate = 0.  # 0, 0.05, 0.1, 0.15
 50 | # quan_digit = 32  # digits kept after feature quantization: None (max:(12~18)(6~8), min=0, std~0.8) or int
 51 | # sparse_rate = 0.9  # ratio of uplink elements kept after sparsification: None or (0,1]
 52 | 
 53 | # Load dataset from the hub
 54 | dataset_id = "samsum"
 55 | dataset = load_dataset(dataset_id)
 56 | if train_set_denominator != -1:
 57 |     dataset['train'] = dataset['train'].select(range(train_set_denominator))
 58 | print(f"Train dataset size: {len(dataset['train'])}")  # 14732
 59 | print(f"Test dataset size: {len(dataset['test'])}")  # 819
 60 | sample = dataset['train'][randrange(len(dataset["train"]))]
 61 | # print(f"dialogue: \n{sample['dialogue']}\n---------------")
 62 | # print(f"summary: \n{sample['summary']}\n---------------")
 63 | # sys.exit()
 64 | 
 65 | # Loading the Model
 66 | model_id="google/flan-t5-small"
 67 | tokenizer = AutoTokenizer.from_pretrained(model_id)
 68 | model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
 69 | # for parameter in model.parameters():  # model: shared, encoder, decoder, lm_head
 70 | #     parameter.requires_grad = False
 71 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[0]:
 72 |     for parameter in model.shared.parameters():  # initial embedding layer, not trained
 73 |         parameter.requires_grad = False
 74 | 
 75 | if FL_type in ['FTLc', 'FbFTL']:
 76 |     for i, m in enumerate(model.encoder.block):  # whether train encoder blocks
 77 |         if i < model_to_train[1]:
 78 |             for parameter in m.parameters():
 79 |                 parameter.requires_grad = False
 80 |     if model_to_train[1] >= 10:
 81 |         for parameter in model.encoder.final_layer_norm.parameters():  # whether train encoder layer norm
 82 |             parameter.requires_grad = False
 83 | 
 84 | if FL_type in ['FTLc', 'FbFTL']:
 85 |     for i, m in enumerate(model.decoder.block):  # whether train decoder blocks
 86 |         if i < model_to_train[2]:
 87 |             for parameter in m.parameters():
 88 |                 parameter.requires_grad = False
 89 |     if model_to_train[2] >= 10:
 90 |         for parameter in model.decoder.final_layer_norm.parameters():  # whether train decoder layer norm
 91 |             parameter.requires_grad = False
 92 | 
 93 | if FL_type in ['FTLc', 'FbFTL'] and not model_to_train[3]:
 94 |     for parameter in model.lm_head.parameters():  # final output layer, always trained
 95 |         parameter.requires_grad = False
 96 | summary(model)
 97 | # print(model)
 98 | # for param in model.state_dict():
 99 | #     print(param)
100 | # sys.exit()
101 | 
102 | # Preprocess data
103 | tokenized_inputs = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["dialogue"], truncation=True), 
104 |                                                                                  batched=True, remove_columns=["dialogue", "summary"])
105 | max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
106 | print(f"Max source length: {max_source_length}")
107 | tokenized_targets = concatenate_datasets([dataset["train"], dataset["test"]]).map(lambda x: tokenizer(x["summary"], truncation=True), 
108 |                                                                                   batched=True, remove_columns=["dialogue", "summary"])
109 | max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
110 | print(f"Max target length: {max_target_length}")
111 | 
112 | def preprocess_function(sample,padding="max_length"):
113 |     # add prefix to the input for t5
114 |     inputs = ["summarize: " + item for item in sample["dialogue"]]
115 |     # tokenize inputs
116 |     model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)
117 |     # Tokenize targets with the `text_target` keyword argument
118 |     labels = tokenizer(text_target=sample["summary"], max_length=max_target_length, padding=padding, truncation=True)
119 |     # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
120 |     # padding in the loss.
121 |     if padding == "max_length":
122 |         labels["input_ids"] = [
123 |             [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
124 |         ]
125 |     model_inputs["labels"] = labels["input_ids"]
126 |     return model_inputs
127 | 
128 | tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["dialogue", "summary", "id"])
129 | print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")
130 | 
131 | # Metric
132 | metric = evaluate.load("rouge")
133 | # helper function to postprocess text
134 | def postprocess_text(preds, labels):
135 |     preds = [pred.strip() for pred in preds]
136 |     labels = [label.strip() for label in labels]
137 |     # rougeLSum expects newline after each sentence
138 |     preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
139 |     labels = ["\n".join(sent_tokenize(label)) for label in labels]
140 |     return preds, labels
141 | 
142 | def compute_metrics(eval_preds):
143 |     preds, labels = eval_preds
144 |     if isinstance(preds, tuple):
145 |         preds = preds[0]
146 |     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
147 |     # Replace -100 in the labels as we can't decode them.
148 |     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
149 |     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
150 |     # Some simple post-processing
151 |     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
152 |     result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
153 |     result = {k: round(v * 100, 4) for k, v in result.items()}
154 |     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
155 |     result["gen_len"] = np.mean(prediction_lens)
156 |     return result
157 | 
158 | # we want to ignore tokenizer pad token in the loss
159 | label_pad_token_id = -100
160 | # Data collator
161 | data_collator = DataCollatorForSeq2Seq(
162 |     tokenizer,
163 |     model=model,
164 |     label_pad_token_id=label_pad_token_id,
165 |     pad_to_multiple_of=8
166 | )
167 | 
168 | # Hugging Face repository id
169 | repository_id = f"{model_id.split('/')[1]}-{dataset_id}"
170 | 
171 | # Define training args
172 | training_args = Seq2SeqTrainingArguments(
173 |     output_dir=repository_id,
174 |     per_device_train_batch_size=batch_size,
175 |     per_device_eval_batch_size=8,
176 |     predict_with_generate=True,
177 |     fp16=False, # Overflows with fp16
178 |     learning_rate=learning_rate,
179 |     num_train_epochs=num_train_epochs,
180 |     # logging & evaluation strategies
181 |     logging_dir=f"{repository_id}/logs",
182 |     logging_strategy="steps",
183 |     logging_steps=500,
184 |     evaluation_strategy="epoch",
185 |     save_strategy="epoch",
186 |     save_total_limit=2,
187 |     load_best_model_at_end=True,
188 |     # metric_for_best_model="overall_f1",
189 |     # push to hub parameters
190 |     report_to="tensorboard",
191 |     push_to_hub=False,
192 |     hub_strategy="every_save",
193 |     hub_model_id=repository_id,
194 |     hub_token=HfFolder.get_token(),
195 | )
196 | 
197 | # Create Trainer instance
198 | trainer = Seq2SeqTrainer(
199 |     model=model,
200 |     args=training_args,
201 |     data_collator=data_collator,
202 |     train_dataset=tokenized_dataset["train"],
203 |     eval_dataset=tokenized_dataset["test"],
204 |     compute_metrics=compute_metrics,
205 | )
206 | 
207 | # Start training
208 | trainer.train()
209 | 
210 | trainer.evaluate()
211 | 
212 | # Save our tokenizer and create model card
213 | tokenizer.save_pretrained(repository_id)
214 | trainer.create_model_card()
215 | # Push the results to the hub
216 | # trainer.push_to_hub()
217 | 
218 | 
219 | # Run Inference and summarize ChatGPT dialogues
220 | # from transformers import pipeline
221 | # from random import randrange
222 | # # load model and tokenizer from huggingface hub with pipeline
223 | # summarizer = pipeline("summarization", model="philschmid/flan-t5-base-samsum", device=0)
224 | # # select a random test sample
225 | # sample = dataset['test'][randrange(len(dataset["test"]))]
226 | # print(f"dialogue: \n{sample['dialogue']}\n---------------")
227 | # # summarize dialogue
228 | # res = summarizer(sample["dialogue"])
229 | # print(f"flan-t5-base summary:\n{res[0]['summary_text']}")


--------------------------------------------------------------------------------