├── federated_learning_files ├── federated_learning_17_2.png ├── federated_learning_29_1.png ├── federated_learning_29_2.png ├── federated_learning_30_1.png ├── federated_learning_30_2.png ├── federated_learning_31_1.png └── federated_learning_31_2.png ├── create_readme_file.sh ├── .gitignore ├── README.md └── federated_learning.ipynb /federated_learning_files/federated_learning_17_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_17_2.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_29_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_29_1.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_29_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_29_2.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_30_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_30_1.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_30_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_30_2.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_31_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_31_1.png -------------------------------------------------------------------------------- /federated_learning_files/federated_learning_31_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mikemikezhu/federated-learning-facial-expression-recognition/HEAD/federated_learning_files/federated_learning_31_2.png -------------------------------------------------------------------------------- /create_readme_file.sh: -------------------------------------------------------------------------------- 1 | # Install nbconvert package 2 | pip install --upgrade nbconvert 3 | 4 | # Remove previously generated README.md file 5 | rm -rf federated_learning_files/ 6 | rm -rf README.md 7 | 8 | # Convert jupyter notebook to markdown 9 | jupyter nbconvert --to markdown federated_learning.ipynb 10 | 11 | # Rename README.md 12 | mv federated_learning.md README.md -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | */.ipynb_checkpoints/* 3 | 4 | # IPython 5 | profile_default/ 6 | ipython_config.py 7 | 8 | # Remove previous ipynb_checkpoints 9 | # git rm -r .ipynb_checkpoints/ 10 | 11 | ### Python ### 12 | # Byte-compiled / optimized / DLL files 13 | __pycache__/ 14 | *.py[cod] 15 | *$py.class 16 | 17 | # C extensions 18 | *.so 19 | 20 | # Distribution / packaging 21 | .Python 22 | build/ 23 | develop-eggs/ 24 | dist/ 25 | downloads/ 26 | eggs/ 27 | .eggs/ 28 | lib/ 29 | lib64/ 30 | parts/ 31 | sdist/ 32 | var/ 33 | wheels/ 34 | pip-wheel-metadata/ 35 | share/python-wheels/ 36 | *.egg-info/ 37 | .installed.cfg 38 | *.egg 39 | MANIFEST 40 | 41 | # PyInstaller 42 | # Usually these files are written by a python script from a template 43 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 44 | *.manifest 45 | *.spec 46 | 47 | # Installer logs 48 | pip-log.txt 49 | pip-delete-this-directory.txt 50 | 51 | # Unit test / coverage reports 52 | htmlcov/ 53 | .tox/ 54 | .nox/ 55 | .coverage 56 | .coverage.* 57 | .cache 58 | nosetests.xml 59 | coverage.xml 60 | *.cover 61 | .hypothesis/ 62 | .pytest_cache/ 63 | 64 | # Translations 65 | *.mo 66 | *.pot 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # pyenv 78 | .python-version 79 | 80 | # pipenv 81 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 82 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 83 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 84 | # install all needed dependencies. 85 | #Pipfile.lock 86 | 87 | # celery beat schedule file 88 | celerybeat-schedule 89 | 90 | # SageMath parsed files 91 | *.sage.py 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # Mr Developer 101 | .mr.developer.cfg 102 | .project 103 | .pydevproject 104 | 105 | # mkdocs documentation 106 | /site 107 | 108 | # mypy 109 | .mypy_cache/ 110 | .dmypy.json 111 | dmypy.json 112 | 113 | # Pyre type checker 114 | .pyre/ 115 | 116 | paper.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Federated Learning on Facial Expression Recognition 2 | 3 | Zhu He - Mike 4 | 5 | ## Abstract 6 | 7 | Recently, I have developed a mobile game, [Best Actor Game](https://apps.apple.com/cn/app/id1498575047), which is based on computer vision model to recognize human facial expression. I have designed the computer vision model using an architecture similar to VGGNet, and the details are described here in this [article](https://github.com/mikemikezhu/facial-expression-recognition). The computer vision model, using Tensorflow Lite, was deployed locally on the mobile device. However, if we allow the face data to be trained locally on user's device, we may improve our model constantly while protecting user's data privacy. Therefore, I try to use **federated learning** to re-design the computer vision model to make facial expression prediction. This document will discuss the basic concept of federated learning, as well as how the federated learning is implemented to create a facial expression recognition model with Tensorflow. 8 | 9 | ## Federated Learning 10 | 11 | Federated learning is a decentralized approach which allows the data to be trained locally on user devices (e.g. mobile devices). Specifically, within a large number of user devices, only a fraction of which may be available for training at a given time. Therefore, the available user devices will be used to train the data locally, and compute the update to the shared global model maintained by the server, which will aggregate all the updates from the distributed devices, and compute weight using the `FederatedAveraging` algorithm in the following math formula. 12 | 13 | 14 | 15 | Considering that modern smartphones have relatively fast processors (e.g. GPUs), the computation becomes essentially free for the federated learning. Therefore, the communication costs dominate during the federated optimization. Based on the research paper from Google, we may reduce the number of rounds of communication required to train a model in the following ways. 16 | 17 | - **Increase parallelism** by involving more clients to compute the update independently. 18 | - **Increase computation on each client** by allowing each client to perform a more complex calculation between each communication round. 19 | 20 | Federated learning has provided the following distinct advantages over the traditional approach to train the data in the data center. 21 | 22 | - It helps to **protect user's data privacy**, considering that the training data (e.g. user's photo, password, etc.) will only be retained locally on the edge devices. 23 | - It also helps to **reduce the network latency**. Since the data is trained locally, transporting large training data to the server is not necessary. Only the update of the training result will be communicated between the client and server, which greatly helps to reduce the network latency. 24 | - It also helps to **reduce the security risks**, since the privacy data is only stored locally, and therefore will not be easily leaked to attackers when there is attacks happening on the cloud, or the communication between client and server (e.g. Man-in-the-middle attack). 25 | 26 | Next, I will discuss how the federated learning is implemented to recognize human facial expression by using Tensorflow. 27 | 28 | ### Install Packages 29 | 30 | First and foremost, let's install Tensorflow federated learning package. 31 | 32 | 33 | ```python 34 | # Tensorflow federated package 35 | !pip install --upgrade tensorflow_federated > /dev/null 2>&1 36 | ``` 37 | 38 | ### Download Data 39 | 40 | Next, we need to download the dataset to train the computer vision model. Here we use FER2013 dataset in [Challenges in Representation Learning: Facial Expression Recognition Challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data) in Kaggle. Therefore, let's configure Kaggle API and download the training dataset. 41 | 42 | 43 | ```python 44 | import os 45 | 46 | # Configure kaggle 47 | os.chdir('/root/') 48 | !mkdir -p .kaggle 49 | os.chdir('/root/.kaggle') 50 | !wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Y-o0TVcjehM8SZB3Nt8U3xkyeQu-Nse-' -O kaggle.json > /dev/null 2>&1 51 | !ls /root/.kaggle 52 | 53 | # Set permissions 54 | !chmod 600 /root/.kaggle/kaggle.json 55 | 56 | # Create data folder 57 | os.chdir('/content/') 58 | !rm -rf data 59 | !mkdir data 60 | os.chdir('data') 61 | !pwd 62 | 63 | # Download data 64 | !pip install -q kaggle 65 | !kaggle competitions download -c challenges-in-representation-learning-facial-expression-recognition-challenge 66 | 67 | # Unzip data 68 | !unzip train.csv.zip train.csv 69 | ``` 70 | 71 | kaggle.json 72 | /content/data 73 | Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.6 / client 1.5.4) 74 | Downloading example_submission.csv to /content/data 75 | 0% 0.00/7.01k [00:00 168 | 169 | 170 | 171 | 172 | ![png](federated_learning_files/federated_learning_17_2.png) 173 | 174 | 175 | ### Preprocess Data 176 | 177 | Next, we need to preprocess and prepare the federated data. Since the training data will be distributed in each user's local device, let's assume we have 3 clients, and each client has 5 images to train, in order to simulate the scenario. 178 | 179 | 180 | ```python 181 | %tensorflow_version 2.x 182 | import tensorflow as tf 183 | 184 | print('Tensorflow version: {}'.format(tf.__version__)) 185 | 186 | from tensorflow import reshape 187 | from collections import OrderedDict 188 | 189 | """ 190 | Assume we have 3 clients, each client has 5 images to train 191 | """ 192 | 193 | # Assume each client has 5 images to train 194 | TRAIN_DATA_PER_CLIENT = 5 195 | 196 | # Train 3 times for each image 197 | TRAINING_EPOCHS = 3 198 | 199 | # Prepare training data for each client 200 | x_client_1 = x_train[0 : TRAIN_DATA_PER_CLIENT] 201 | y_client_1 = y_train[0 : TRAIN_DATA_PER_CLIENT] 202 | 203 | x_client_2 = x_train[TRAIN_DATA_PER_CLIENT : TRAIN_DATA_PER_CLIENT * 2] 204 | y_client_2 = y_train[TRAIN_DATA_PER_CLIENT : TRAIN_DATA_PER_CLIENT * 2] 205 | 206 | x_client_3 = x_train[TRAIN_DATA_PER_CLIENT * 2 : TRAIN_DATA_PER_CLIENT * 3] 207 | y_client_3 = y_train[TRAIN_DATA_PER_CLIENT * 2 : TRAIN_DATA_PER_CLIENT * 3] 208 | print(x_client_3.shape) 209 | 210 | # Prepare test data 211 | x_test = x_train[TRAIN_DATA_PER_CLIENT * 3 : TRAIN_DATA_PER_CLIENT * 4] 212 | y_test = y_train[TRAIN_DATA_PER_CLIENT * 3 : TRAIN_DATA_PER_CLIENT * 4] 213 | 214 | # Create federated data 215 | def create_federated_data(x_train, y_train): 216 | 217 | orderDict = OrderedDict() 218 | pixels_list = [] 219 | 220 | x_train = x_train / 255.0 221 | x_train = x_train.reshape(len(x_train), 48, 48, 1) 222 | 223 | orderDict['x'] = numpy.array(x_train) 224 | orderDict['y'] = numpy.array(y_train) 225 | dataset = tf.data.Dataset.from_tensor_slices(orderDict) 226 | batch = dataset.shuffle(10).batch(5) 227 | 228 | return batch 229 | 230 | federated_data_client_1 = [create_federated_data(x_client_1, y_client_1) for epoch in range(TRAINING_EPOCHS)] 231 | federated_data_client_2 = [create_federated_data(x_client_2, y_client_2) for epoch in range(TRAINING_EPOCHS)] 232 | federated_data_client_3 = [create_federated_data(x_client_3, y_client_3) for epoch in range(TRAINING_EPOCHS)] 233 | 234 | federated_data_test = [create_federated_data(x_test, y_test) for epoch in range(TRAINING_EPOCHS)] 235 | 236 | print('Total training epochs: {}'.format(len(federated_data_client_3))) 237 | ``` 238 | 239 | Tensorflow version: 2.1.0 240 | (5, 48, 48) 241 | Total training epochs: 3 242 | 243 | 244 | ### Create Model 245 | 246 | By referring to VGGNet architecture, we have designed the computer vision model with several stacks of layers. The model will have the following components: 247 | - Convolutional layers: These layers are the building blocks of our architecture, which learns the image feature by computing the dot product between the weights and the small region on the image. Similar to VGGNet architecture, all the convolutional layers are designed with 3 x 3 kernal size, and several filters. 248 | - Activation functions: The activation functions are those functions which are applied to the outputs of the layers in the network. Specifically, we use ReLU (Rectified Linear Unit) activation function to increase the non-linearity of the network. Besides, a Softmax function will be used to compute the probability of each category. 249 | - Pooling layers: These layers will down-sample the image to reduce the spatial data and extract features. In our model, we will use Max Pooling with A 3 x 3 pooling size and a 2 x 2 stride. 250 | - Dense layers: The dense layers are stacked as the fully connected layers of the network, which take in the feature data from the previous convolutional layers and perform decision making. 251 | - Dropout layers: The dropout layers are used to prevent over-fitting when training the model. 252 | - Batch normalization: This technique can be used to speed up learning by normalizing the output of the previous activation layer. 253 | 254 | The diagram of the model is displayed as follows. 255 | 256 | ![cnn](https://drive.google.com/uc?id=1jjORxRgvEDDMLZ-mX5JkCnUWUFn7IHpL) 257 | 258 | Our model contains 5 stacks of layers. In each of the first 4 stacks of layers, there are 2 convolutional layer followed by 1 pooling layer. Besides, we use batch normalization to speed up training and dropout to prevent over-fitting. Then we have one stack of 3 fully-connected layers, followed by a Softmax activation function, which generates the probability of the facial expression categories. Finally, we compile our model using Adam optimizer with a certain learning rate. Considering that we are dealing with classification issue, we will use `sparse_categorical_crossentropy` as the loss function. 259 | 260 | 261 | ```python 262 | from tensorflow.keras.models import Sequential 263 | from tensorflow.keras.layers import Conv2D, BatchNormalization, MaxPool2D, Dropout, Flatten, Dense 264 | 265 | from tensorflow.keras.optimizers import SGD 266 | from tensorflow.keras.losses import SparseCategoricalCrossentropy 267 | from tensorflow.keras.metrics import SparseCategoricalAccuracy 268 | 269 | def create_keras_model(): 270 | 271 | # Create keras model 272 | model = Sequential() 273 | 274 | # 1st convolution layer 275 | model.add(Conv2D(64, input_shape=(48, 48, 1), kernel_size=(3, 3), activation='relu')) 276 | model.add(BatchNormalization()) 277 | model.add(Conv2D(64, padding='same', kernel_size=(3, 3), activation='relu')) 278 | model.add(BatchNormalization()) 279 | model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2))) 280 | model.add(Dropout(0.3)) 281 | 282 | # 2nd convolution layer 283 | model.add(Conv2D(128, padding='same', kernel_size=(3, 3), activation='relu')) 284 | model.add(BatchNormalization()) 285 | model.add(Conv2D(128, padding='same', kernel_size=(3, 3), activation='relu')) 286 | model.add(BatchNormalization()) 287 | model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2))) 288 | model.add(Dropout(0.3)) 289 | 290 | # 3rd convolution layer 291 | model.add(Conv2D(256, padding='same', kernel_size=(3, 3), activation='relu')) 292 | model.add(BatchNormalization()) 293 | model.add(Conv2D(256, padding='same', kernel_size=(3, 3), activation='relu')) 294 | model.add(BatchNormalization()) 295 | model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2))) 296 | model.add(Dropout(0.3)) 297 | 298 | # 4th convolution layer 299 | model.add(Conv2D(512, padding='same', kernel_size=(3, 3), activation='relu')) 300 | model.add(BatchNormalization()) 301 | model.add(Conv2D(512, padding='same', kernel_size=(3, 3), activation='relu')) 302 | model.add(BatchNormalization()) 303 | model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2))) 304 | model.add(Dropout(0.3)) 305 | 306 | # Fully connected layer 307 | model.add(Flatten()) 308 | model.add(Dense(512, activation='relu')) 309 | model.add(Dropout(0.3)) 310 | model.add(Dense(256, activation='relu')) 311 | model.add(Dropout(0.3)) 312 | model.add(Dense(64, activation='relu')) 313 | model.add(Dropout(0.3)) 314 | 315 | model.add(Dense(7, activation='softmax')) 316 | 317 | # Compile the model 318 | model.compile(loss=SparseCategoricalCrossentropy(), 319 | optimizer=SGD(learning_rate=0.02), 320 | metrics=[SparseCategoricalAccuracy()]) 321 | 322 | return model 323 | 324 | # Summary model 325 | keras_model = create_keras_model() 326 | keras_model.summary() 327 | ``` 328 | 329 | Model: "sequential" 330 | _________________________________________________________________ 331 | Layer (type) Output Shape Param # 332 | ================================================================= 333 | conv2d (Conv2D) (None, 46, 46, 64) 640 334 | _________________________________________________________________ 335 | batch_normalization (BatchNo (None, 46, 46, 64) 256 336 | _________________________________________________________________ 337 | conv2d_1 (Conv2D) (None, 46, 46, 64) 36928 338 | _________________________________________________________________ 339 | batch_normalization_1 (Batch (None, 46, 46, 64) 256 340 | _________________________________________________________________ 341 | max_pooling2d (MaxPooling2D) (None, 22, 22, 64) 0 342 | _________________________________________________________________ 343 | dropout (Dropout) (None, 22, 22, 64) 0 344 | _________________________________________________________________ 345 | conv2d_2 (Conv2D) (None, 22, 22, 128) 73856 346 | _________________________________________________________________ 347 | batch_normalization_2 (Batch (None, 22, 22, 128) 512 348 | _________________________________________________________________ 349 | conv2d_3 (Conv2D) (None, 22, 22, 128) 147584 350 | _________________________________________________________________ 351 | batch_normalization_3 (Batch (None, 22, 22, 128) 512 352 | _________________________________________________________________ 353 | max_pooling2d_1 (MaxPooling2 (None, 10, 10, 128) 0 354 | _________________________________________________________________ 355 | dropout_1 (Dropout) (None, 10, 10, 128) 0 356 | _________________________________________________________________ 357 | conv2d_4 (Conv2D) (None, 10, 10, 256) 295168 358 | _________________________________________________________________ 359 | batch_normalization_4 (Batch (None, 10, 10, 256) 1024 360 | _________________________________________________________________ 361 | conv2d_5 (Conv2D) (None, 10, 10, 256) 590080 362 | _________________________________________________________________ 363 | batch_normalization_5 (Batch (None, 10, 10, 256) 1024 364 | _________________________________________________________________ 365 | max_pooling2d_2 (MaxPooling2 (None, 4, 4, 256) 0 366 | _________________________________________________________________ 367 | dropout_2 (Dropout) (None, 4, 4, 256) 0 368 | _________________________________________________________________ 369 | conv2d_6 (Conv2D) (None, 4, 4, 512) 1180160 370 | _________________________________________________________________ 371 | batch_normalization_6 (Batch (None, 4, 4, 512) 2048 372 | _________________________________________________________________ 373 | conv2d_7 (Conv2D) (None, 4, 4, 512) 2359808 374 | _________________________________________________________________ 375 | batch_normalization_7 (Batch (None, 4, 4, 512) 2048 376 | _________________________________________________________________ 377 | max_pooling2d_3 (MaxPooling2 (None, 1, 1, 512) 0 378 | _________________________________________________________________ 379 | dropout_3 (Dropout) (None, 1, 1, 512) 0 380 | _________________________________________________________________ 381 | flatten (Flatten) (None, 512) 0 382 | _________________________________________________________________ 383 | dense (Dense) (None, 512) 262656 384 | _________________________________________________________________ 385 | dropout_4 (Dropout) (None, 512) 0 386 | _________________________________________________________________ 387 | dense_1 (Dense) (None, 256) 131328 388 | _________________________________________________________________ 389 | dropout_5 (Dropout) (None, 256) 0 390 | _________________________________________________________________ 391 | dense_2 (Dense) (None, 64) 16448 392 | _________________________________________________________________ 393 | dropout_6 (Dropout) (None, 64) 0 394 | _________________________________________________________________ 395 | dense_3 (Dense) (None, 7) 455 396 | ================================================================= 397 | Total params: 5,102,791 398 | Trainable params: 5,098,951 399 | Non-trainable params: 3,840 400 | _________________________________________________________________ 401 | 402 | 403 | Next, wrap our compiled Keras model in an instance of the `tff.learning.Model` interface, in order to use the Tensorflow federated learning. 404 | 405 | 406 | ```python 407 | import tensorflow_federated as tff 408 | 409 | # Create dummy batch 410 | dummy_batch = tf.nest.map_structure(lambda x: x.numpy(), iter(federated_data_client_1[0]).next()) 411 | print(dummy_batch['x'].shape) 412 | 413 | def create_federated_model(): 414 | 415 | # Create keras model 416 | keras_model = create_keras_model() 417 | 418 | # Convert keras model to federated model 419 | return tff.learning.from_compiled_keras_model(keras_model, dummy_batch) 420 | ``` 421 | 422 | (5, 48, 48, 1) 423 | 424 | 425 | ### Train Model 426 | 427 | Then, we need to build federated averate process and start training. 428 | 429 | 430 | ```python 431 | # Build federated average process 432 | trainer = tff.learning.build_federated_averaging_process(create_federated_model) 433 | 434 | # Create initial state 435 | train_state = trainer.initialize() 436 | ``` 437 | 438 | WARNING:tensorflow:From /tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. 439 | Instructions for updating: 440 | If using Keras pass *_constraint arguments to layers. 441 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 442 | 443 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 444 | 445 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 446 | 447 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 448 | 449 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 450 | 451 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 452 | 453 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 454 | 455 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 456 | 457 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 458 | 459 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 460 | 461 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 462 | 463 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 464 | 465 | 466 | 467 | 468 | ```python 469 | # Train on federated model 470 | print('========== Train on Local Device of Client 1 ==========') 471 | 472 | history_accuracy_1 = [] 473 | history_loss_1 = [] 474 | 475 | for round_num in range(6): 476 | train_state, train_metrics = trainer.next(train_state, federated_data_client_1) 477 | history_accuracy_1.append(train_metrics.sparse_categorical_accuracy) 478 | history_loss_1.append(train_metrics.loss) 479 | print('round {:2d}, metrics={}'.format(round_num, train_metrics)) 480 | 481 | # Show accuracy diagram 482 | plt.title('Model Accuracy for Client 1') 483 | plt.plot(history_accuracy_1, label='accuracy') 484 | plt.xlabel('Epoch') 485 | plt.ylabel('Accuracy') 486 | plt.show() 487 | 488 | # Show loss diagram 489 | plt.title('Model Loss for Client 1') 490 | plt.plot(history_loss_1, label='loss') 491 | plt.xlabel('Epoch') 492 | plt.ylabel('Loss') 493 | plt.show() 494 | ``` 495 | 496 | ========== Train on Local Device of Client 1 ========== 497 | round 0, metrics= 498 | round 1, metrics= 499 | round 2, metrics= 500 | round 3, metrics= 501 | round 4, metrics= 502 | round 5, metrics= 503 | 504 | 505 | 506 | ![png](federated_learning_files/federated_learning_29_1.png) 507 | 508 | 509 | 510 | ![png](federated_learning_files/federated_learning_29_2.png) 511 | 512 | 513 | 514 | ```python 515 | # Train on federated model 516 | print('========== Train on Local Device of Client 2 ==========') 517 | 518 | history_accuracy_2 = [] 519 | history_loss_2 = [] 520 | 521 | for round_num in range(6): 522 | train_state, train_metrics = trainer.next(train_state, federated_data_client_2) 523 | history_accuracy_2.append(train_metrics.sparse_categorical_accuracy) 524 | history_loss_2.append(train_metrics.loss) 525 | print('round {:2d}, metrics={}'.format(round_num, train_metrics)) 526 | 527 | # Show accuracy diagram 528 | plt.title('Model Accuracy for Client 2') 529 | plt.plot(history_accuracy_2, label='accuracy') 530 | plt.xlabel('Epoch') 531 | plt.ylabel('Accuracy') 532 | plt.show() 533 | 534 | # Show loss diagram 535 | plt.title('Model Loss for Client 2') 536 | plt.plot(history_loss_2, label='loss') 537 | plt.xlabel('Epoch') 538 | plt.ylabel('Loss') 539 | plt.show() 540 | ``` 541 | 542 | ========== Train on Local Device of Client 2 ========== 543 | round 0, metrics= 544 | round 1, metrics= 545 | round 2, metrics= 546 | round 3, metrics= 547 | round 4, metrics= 548 | round 5, metrics= 549 | 550 | 551 | 552 | ![png](federated_learning_files/federated_learning_30_1.png) 553 | 554 | 555 | 556 | ![png](federated_learning_files/federated_learning_30_2.png) 557 | 558 | 559 | 560 | ```python 561 | # Train on federated model 562 | print('========== Train on Local Device of Client 3 ==========') 563 | 564 | history_accuracy_3 = [] 565 | history_loss_3 = [] 566 | 567 | for round_num in range(6): 568 | train_state, train_metrics = trainer.next(train_state, federated_data_client_3) 569 | history_accuracy_3.append(train_metrics.sparse_categorical_accuracy) 570 | history_loss_3.append(train_metrics.loss) 571 | print('round {:2d}, metrics={}'.format(round_num, train_metrics)) 572 | 573 | # Show accuracy diagram 574 | plt.title('Model Accuracy for Client 3') 575 | plt.plot(history_accuracy_3, label='accuracy') 576 | plt.xlabel('Epoch') 577 | plt.ylabel('Accuracy') 578 | plt.show() 579 | 580 | # Show loss diagram 581 | plt.title('Model Loss for Client 3') 582 | plt.plot(history_loss_3, label='loss') 583 | plt.xlabel('Epoch') 584 | plt.ylabel('Loss') 585 | plt.show() 586 | ``` 587 | 588 | ========== Train on Local Device of Client 3 ========== 589 | round 0, metrics= 590 | round 1, metrics= 591 | round 2, metrics= 592 | round 3, metrics= 593 | round 4, metrics= 594 | round 5, metrics= 595 | 596 | 597 | 598 | ![png](federated_learning_files/federated_learning_31_1.png) 599 | 600 | 601 | 602 | ![png](federated_learning_files/federated_learning_31_2.png) 603 | 604 | 605 | ### Evaluate Model 606 | 607 | Last but not least, let's evaluate the model after federated training process. 608 | 609 | 610 | ```python 611 | evaluator = tff.learning.build_federated_evaluation(create_federated_model) 612 | ``` 613 | 614 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 615 | 616 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 617 | 618 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 619 | 620 | WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. 621 | 622 | If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. 623 | 624 | To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor. 625 | 626 | 627 | 628 | 629 | ```python 630 | test_metrics = evaluator(train_state.model, federated_data_test) 631 | print(test_metrics) 632 | ``` 633 | 634 | 635 | 636 | 637 | ## Further Thinking 638 | 639 | During the federated training process, I have come up with the following thoughts which might consider to improve the federated training in the future. 640 | 641 | - Early stopping on federated training: 642 | 643 | **Early stopping** strategy is generally used to stop the training process when there is no obvious improvement in the validation process. Therefore, I would suggest that early stopping shall also be integrated in the federated training process. For example, the shared model in the centralized server may reject the client's update when the validation process reaches a certain plateau. 644 | 645 | - Centralized server handling upon receiving large amount of local update: 646 | 647 | There is a shared model in the centralized server constantly receiving the update from the clients. However, if there are multiple clients simultaneously update to the model of the centralized server, it might significantly increase the burder of the server load. Therefore, I would suggest to use a **Message Queue** (e.g. RabbitMQ, or Kafka) to reduce server load. Besides, instead of using one shared model in one single server, we could copy several instances of models in multiple paralleled servers, and compute the model update in parallel using **Load Balancing** techniques. 648 | 649 | - Future application of federate learning: 650 | 651 | We may also consider to perform federated learning in distributed devices other than user devices (e.g. mobile devices). For example, a **Content Delivery Network (CDN)** is generally used to provide fast internet delivery through a geographically distributed group of servers. Therefore, I would suggest that we may also conduct machine learning training in these servers in the CDN network, and inform the model update to other server nodes by using **Gossip Protocol** strategy. 652 | 653 | ## Conclusion 654 | 655 | To put it in a nutshell, this document has introduced the concept of federated learning, and the implementation of federated training to create a computer vision model for facial expression recognition. The code implementation can be referred to my [GitHub](https://github.com/mikemikezhu/federated-learning-facial-expression-recognition). 656 | 657 | ## Reference 658 | 659 | - [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/pdf/1602.05629.pdf) 660 | - [Federated Learning for Image Classification](https://www.tensorflow.org/federated/tutorials/federated_learning_for_image_classification#evaluation) 661 | - [TensorFlow Federated: Machine Learning on Decentralized Data](https://www.tensorflow.org/federated) 662 | - [Facial Expression Recognition Challenge using Convolutional Neural Network](https://mikemikezhu.github.io/dev/2020/02/11/facial_expression_recognition.html) 663 | - [Challenges in Representation Learning: Facial Expression Recognition Challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data) 664 | - [Best Actor Game](https://apps.apple.com/cn/app/id1498575047) 665 | - [Globally Distributed Content Delivery](https://people.cs.umass.edu/~ramesh/Site/PUBLICATIONS_files/DMPPSW02.pdf) 666 | - [Gossip Protocol](https://en.wikipedia.org/wiki/Gossip_protocol) 667 | - [Early Stopping](https://en.wikipedia.org/wiki/Early_stopping) 668 | - [Performance Tradeoffs in Static and Dynamic Load Balancing Strategies](https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19860014876.pdf) 669 | -------------------------------------------------------------------------------- /federated_learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "federated-learning.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [], 9 | "toc_visible": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "0yc0r4-itlGq", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Federated Learning on Facial Expression Recognition" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": { 30 | "id": "INJ_Zf1bZBR9", 31 | "colab_type": "text" 32 | }, 33 | "source": [ 34 | "Zhu He - Mike" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": { 40 | "id": "rkglnJyR1WXz", 41 | "colab_type": "text" 42 | }, 43 | "source": [ 44 | "## Abstract" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "id": "K2LLO6Mf1Ylr", 51 | "colab_type": "text" 52 | }, 53 | "source": [ 54 | "Recently, I have developed a mobile game, [Best Actor Game](https://apps.apple.com/cn/app/id1498575047), which is based on computer vision model to recognize human facial expression. I have designed the computer vision model using an architecture similar to VGGNet, and the details are described here in this [article](https://github.com/mikemikezhu/facial-expression-recognition). The computer vision model, using Tensorflow Lite, was deployed locally on the mobile device. However, if we allow the face data to be trained locally on user's device, we may improve our model constantly while protecting user's data privacy. Therefore, I try to use **federated learning** to re-design the computer vision model to make facial expression prediction. This document will discuss the basic concept of federated learning, as well as how the federated learning is implemented to create a facial expression recognition model with Tensorflow." 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": { 60 | "id": "8fKJNZdJ1d8I", 61 | "colab_type": "text" 62 | }, 63 | "source": [ 64 | "## Federated Learning" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": { 70 | "id": "oA1Z72CM1gV8", 71 | "colab_type": "text" 72 | }, 73 | "source": [ 74 | "Federated learning is a decentralized approach which allows the data to be trained locally on user devices (e.g. mobile devices). Specifically, within a large number of user devices, only a fraction of which may be available for training at a given time. Therefore, the available user devices will be used to train the data locally, and compute the update to the shared global model maintained by the server, which will aggregate all the updates from the distributed devices, and compute weight using the `FederatedAveraging` algorithm in the following math formula.\n", 75 | "\n", 76 | "$w_{t+1}=\\sum_{k=1}^K \\frac{n_{k}}{n}w^{k}_{t+1}$\n", 77 | "\n", 78 | "Considering that modern smartphones have relatively fast processors (e.g. GPUs), the computation becomes essentially free for the federated learning. Therefore, the communication costs dominate during the federated optimization. Based on the research paper from Google, we may reduce the number of rounds of communication required to train a model in the following ways.\n", 79 | "\n", 80 | "- **Increase parallelism** by involving more clients to compute the update independently.\n", 81 | "- **Increase computation on each client** by allowing each client to perform a more complex calculation between each communication round.\n", 82 | "\n", 83 | "Federated learning has provided the following distinct advantages over the traditional approach to train the data in the data center.\n", 84 | "\n", 85 | "- It helps to **protect user's data privacy**, considering that the training data (e.g. user's photo, password, etc.) will only be retained locally on the edge devices.\n", 86 | "- It also helps to **reduce the network latency**. Since the data is trained locally, transporting large training data to the server is not necessary. Only the update of the training result will be communicated between the client and server, which greatly helps to reduce the network latency.\n", 87 | "- It also helps to **reduce the security risks**, since the privacy data is only stored locally, and therefore will not be easily leaked to attackers when there is attacks happening on the cloud, or the communication between client and server (e.g. Man-in-the-middle attack)." 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": { 93 | "id": "eNZbq0khBP56", 94 | "colab_type": "text" 95 | }, 96 | "source": [ 97 | "Next, I will discuss how the federated learning is implemented to recognize human facial expression by using Tensorflow." 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": { 103 | "id": "05OI7sK_1iC0", 104 | "colab_type": "text" 105 | }, 106 | "source": [ 107 | "### Install Packages" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": { 113 | "id": "cWbWcLQJCi7F", 114 | "colab_type": "text" 115 | }, 116 | "source": [ 117 | "First and foremost, let's install Tensorflow federated learning package." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "metadata": { 123 | "id": "6jTINVamKDRD", 124 | "colab_type": "code", 125 | "colab": {} 126 | }, 127 | "source": [ 128 | "# Tensorflow federated package \n", 129 | "!pip install --upgrade tensorflow_federated > /dev/null 2>&1" 130 | ], 131 | "execution_count": 0, 132 | "outputs": [] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": { 137 | "id": "P7PZB48y1ldw", 138 | "colab_type": "text" 139 | }, 140 | "source": [ 141 | "### Download Data" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": { 147 | "id": "sHTJ4BsyCvu6", 148 | "colab_type": "text" 149 | }, 150 | "source": [ 151 | "Next, we need to download the dataset to train the computer vision model. Here we use FER2013 dataset in [Challenges in Representation Learning: Facial Expression Recognition Challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data) in Kaggle. Therefore, let's configure Kaggle API and download the training dataset." 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "metadata": { 157 | "id": "h4181vqWJ-mQ", 158 | "colab_type": "code", 159 | "outputId": "ff2d8782-0777-4623-cdd8-7adc4c39c6e9", 160 | "colab": { 161 | "base_uri": "https://localhost:8080/", 162 | "height": 377 163 | } 164 | }, 165 | "source": [ 166 | "import os\n", 167 | "\n", 168 | "# Configure kaggle\n", 169 | "os.chdir('/root/')\n", 170 | "!mkdir -p .kaggle\n", 171 | "os.chdir('/root/.kaggle')\n", 172 | "!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Y-o0TVcjehM8SZB3Nt8U3xkyeQu-Nse-' -O kaggle.json > /dev/null 2>&1\n", 173 | "!ls /root/.kaggle\n", 174 | "\n", 175 | "# Set permissions \n", 176 | "!chmod 600 /root/.kaggle/kaggle.json\n", 177 | "\n", 178 | "# Create data folder\n", 179 | "os.chdir('/content/')\n", 180 | "!rm -rf data\n", 181 | "!mkdir data\n", 182 | "os.chdir('data')\n", 183 | "!pwd\n", 184 | "\n", 185 | "# Download data\n", 186 | "!pip install -q kaggle\n", 187 | "!kaggle competitions download -c challenges-in-representation-learning-facial-expression-recognition-challenge\n", 188 | "\n", 189 | "# Unzip data\n", 190 | "!unzip train.csv.zip train.csv" 191 | ], 192 | "execution_count": 0, 193 | "outputs": [ 194 | { 195 | "output_type": "stream", 196 | "text": [ 197 | "kaggle.json\n", 198 | "/content/data\n", 199 | "Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.6 / client 1.5.4)\n", 200 | "Downloading example_submission.csv to /content/data\n", 201 | " 0% 0.00/7.01k [00:00" 354 | ] 355 | }, 356 | "metadata": { 357 | "tags": [] 358 | }, 359 | "execution_count": 4 360 | }, 361 | { 362 | "output_type": "display_data", 363 | "data": { 364 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2da4xfV3nun9fXJJjEcXz3GN9iyIUQ\nWziQgEmiBJRAS8IHQIVylCNF5APnSFRtKaFHOjqVWgFfSiq1tIoKwkDV0JtIFLUUk7qpKpCDE9sh\nYMVxiJ3YHttxfIsJBHu8zof5T+T9rMfzf+dv+z9j9vOTongtr7332mvv5T3vM+8lSikwxvzmM2m8\nJ2CM6Q/e7Ma0BG92Y1qCN7sxLcGb3ZiW4M1uTEs4q80eEXdGxLMRsSMi7j9XkzLGnHui19+zR8Rk\nANsBfADAbgA/BvCJUsrPznTMtGnTyiWXXNLomzSp+e9NRFTHDQ0Njdoew5wb7VOnTnUdo+Drv+lN\nb6rGTJs2rerje1Vrz31qDF9f3Yfq6zYfbgPA5MmTq74pU6Y02lOnTj1n5+4FXqPXXnutGnPixIlG\nWz1ntWa81idPnuw6Rj2zzPPIvJ/cx9c6ceIEhoaG5Es8RXUmeReAHaWUn3cm+hCAuwGccbNfcskl\nWLt2baPvzW9+c6OtXoBXX3210T506FDXyanz8AunXgp+cdV5jhw50mjfcMMN1Zhly5ZVfRdddFGj\nrf7R4pfy17/+ddfr//KXv6zGqL5u85kxY0Y15tJLL636Zs+e3WjPnTu3GsPPdfr06dWYmTNnNtrq\n5eZ/WNRm4zXbunVrNWbfvn2jnhcAfvWrX1V9R48ebbRffvnlaszhw4dHnQ9QPw/1XvG7d/z48WoM\n9/Ga7dq1qzpmhLP5MX4RgJdOa+/u9BljJiDnXaCLiPsiYlNEbFJfKWNMfzibzb4HwOLT2gOdvgal\nlAdLKWtKKWuUHWuM6Q9nY7P/GMDKiFiG4U3+OwA+OdoBp06dqmxQtqWUTcYihLJ1L7/88kab7VGg\ntn+V3cb6gLK/PvjBDzbaymZnIRLQ98Zk/kF8/fXXx3xeJZCxjajWTM2HbXs1hvsygqWCbVv17Fl7\nWb16dTVmw4YNjfaePdV3ST4znqO6Pq+bemcYJRCyXa/eT157vvfRBOaeN3sp5WRE/G8A/w5gMoCv\nl1J+2uv5jDHnl7P5sqOU8q8A/vUczcUYcx6xB50xLeGsvuxjZWhoCK+88kqjj22Oyy67rDqOf2er\n7Ca2b9TvTNmW4t9XA8CsWbMa7fe///3VmFWrVjXayh7t1WGE7Whlg/G5lVMLH5c5T8auV33KtuQ1\nUevBc1LOKDxG6RN8fX5fAGDlypWN9rPPPluNUffKdjT/3h2o32G1Hnxvyg+C9SKlIbA+wNcazWb3\nl92YluDNbkxL8GY3piV4sxvTEvoq0J08ebISxVhwUKIEO/+rAAEWNzKC0FVXXVWNueOOOxrt+fPn\nV2NYyMoKdDxOBX7wcZmACQXfqxLfWNzJrJkiE9GmRCu+fyW88nlUQA2fWwl9V199daP9/e9/vxqj\nBFt2YlHCGq+Rug8WFtU7nHEn77Zmo0Wx+stuTEvwZjemJXizG9MS+mqzA7XNwbaLSijBDjIqoQLb\njWrMdddd12jffPPN1Zh58+aNel6FstGUHc/3nrF1M845mUQdmeOUfZ7RB9T1eU2Uzc5zVGMy8PWV\n482cOXMa7YULF1Zjnnzyyapv0aJmigblsMPJK1RAUQY+jpN7ALVNzlrVaPjLbkxL8GY3piV4sxvT\nErzZjWkJfY96Y8cFdpJQwtIVV1zRaLPYAtSC3IoVK6ox119/faOtUkAzSnzqNf0231smVbBaDxa/\nlLDG6zrepbmVw0gm3XRGaMxw8cUXN9rLly+vxvzgBz+o+lgAW7JkSTXm2LFjjXYmwlA5B7EQffDg\nwWoMR4VmUlSP4C+7MS3Bm92YluDNbkxL6LtTDcO2S6a6iHI2eOtb39poL126tBrDdluv5ZfYJlPO\nIMpuywSnZM7DZOzxzLUyji9qTmqOY8mgMoKyPzPPoxdUxR6V3WhwcLDRVjoP60XK0SXjQMQZkjmr\nE1AH4oxlffxlN6YleLMb0xK82Y1pCd7sxrSEvgt07FzA6Z0zqYKVUw073iinhUwpIRZpMpFgGfEJ\nqO9NiSnnqx5eps57RrDMHpcpLdVLlJtycuJ1VUIfj1EZiNQcuUSzOnemHHNmjrxmSojmc7MY6FTS\nxhhvdmPagje7MS2hrzZ7KaWy0dlOUbYM23aqRBTbusoZhG3LTJCJcrTgMSqbS8YZRekT3KfsWrbb\nMs4oao48JhNQA9T3nynHpbKyMmrNMoFSPEaVTOZAHGWfqyw0e/fubbSVHc3vSEYvyjg5qTnyOmZ0\noDeu2fWKxpjfCLzZjWkJ3uzGtARvdmNaQt+dajLiFsNRRSryiPuUYwMLWUog6yV1dEZEU+OUIKbE\nJSaTqrhbmS2gvteMc4zqe/3116sxfG+qrFdGXMqUiGLU88isq3r2v/jFLxptJdhmngevWWY+Cr6+\nM9UYYyq82Y1pCV03e0R8PSIORMQzp/XNioj1EfFc5/+Xj3YOY8z4k7HZvwHgLwF887S++wE8Vkr5\nUkTc32l/vtuJIqKyb9gBYdasWdVx3McZPYBcUEUmq0fG0SSTJVbZZBn7iseorCdcIks5g/B9cIki\noF57ZXurNeISWer67MSSDRZiMuWw2I5XWky38wI6A26mjHLmemyzKy0kE6jVLWvwWQXClFL+C8Ah\n6r4bwLrOn9cB+Ei38xhjxpdebfZ5pZSR5Fz7AMwbbbAxZvw561+9lVJKRJzRITci7gNwH3DuEv4b\nY8ZOr7tvf0QsAIDO/w+caWAp5cFSyppSyppe7TZjzNnT65f9EQD3APhS5/8PZw6aNGlSJdBxhpmV\nK1dWxw0MDDTaykEjU8ecBY/MMUqgYUGKU1Sf6dx875mfdJRAxpFPSljjc2cEITWGxUCgduxQ93r8\n+PFGW2Xg4XVUa83XUmJYpmQWH6cEVHWvLPyqNeLnr87DqA9fJuKxW5ToWQl0EfH3AH4E4G0RsTsi\n7sXwJv9ARDwH4P2dtjFmAtP1y15K+cQZ/ur2czwXY8x5xIqZMS2hr4EwU6ZMqWx0LtOkSumyU02v\nmUrZJsoEwih7lO0i5fyg+ti+UrZ+pkSUytTDsE2qsqeo6zOqBNHRo0cbbaUZ8HFqPfg5KuccPk7Z\n2r3oNYcOsetIfV9qjsoe52fUa8kunqPSObplW3J2WWOMN7sxbcGb3ZiW4M1uTEvoq0A3derUquzO\nkiVLGm1V2omFikwdcRVhlkkTzQKIEkn279/faLMDCaBTJ7O4pMSU2bNnN9rKgYgFOs7kA9TzVufh\n47IpsTkS79ixY9UY7lNz3LNnT6PN4i0AzJgxo9HOODCp59qtbBKg14gdfdS5ed3U+8nvVSbjTkbE\nGwv+shvTErzZjWkJ3uzGtARvdmNawrh70HGKKeXplUk3zaKIEujY00t5Y7HYpoQ29g5TnldK2Mt4\n3h08eLDrmExNbl4PjhwEajFUebCp2mZ8PSUksSCmPM/Yiy2Tcko9Mxbx1Hx4zur5KIGO3xkVmdeL\n92ZmjhnGUuPeX3ZjWoI3uzEtwZvdmJbQV5t90qRJVZkmdpJQThxsy2Vsm0wklHKQYPtb2ZqZ6DVl\na7O9qexPLtvE5YeA2q7PlCTavXt31zHKqUWl7V68eHHXMawHqHtlxxvlnMQajrKZ2f5mGx7I6T6q\nrBin4FaRkpkMM6PVTR/LmG5RkY56M8Z4sxvTFrzZjWkJ3uzGtIS+CnQRUQklLChkamsphwgWRZTQ\nl4lOYtFKiV8skL3wwgvVGCWUZOqvMUogzNQDZ/FLCUscPafuVQmEHMGmnHFYxGThEQAGBwcbbZXe\nigXTl156qRrD66rmw05F6tmrvl7E4IyTk4JFXXUefvaZOnMj+MtuTEvwZjemJXizG9MS+mqzl1Iq\n54pMgECmvA/bTWoM23KZ8yjnGHYYUXaTctBgW185uvC9qsCg2267rdFWmkEmlTP3qaAXdR+smSin\nIj5u586d1Zjvfe97jbZKW82ahXpmbA+vXr2663zUmql3j8+tnIN4HdUcM84v/OwzTjaZ/fPGHLqe\nzRjzG4E3uzEtwZvdmJbgzW5MSxh3gY4FGBVBlRE3WEhRwhaLF5n615zaGagFqV27dlVjlHNQt9ra\nAPDiiy822jfddFM1Zu3atY22cjJiAUoJbfwsFi1aVI3hOnsAsGDBgkZ77ty51Rhet+9+97vVmC1b\ntjTaKisQi6G33nprNYY5cOBA1cdClqr1lsm4owQ6Pncmwi4TlanmMxYnGsZfdmNagje7MS3Bm92Y\nltBXm31oaKgK0ODgB2XHZmqmZxw92EZXjibseKPsSLalFi5cWI155plnqj6e08c+9rFqDF9PZY/h\neS9btqwaw4EfqqY7O/lknGPUnJStz3a80hU4440KlmFWrFhR9a1atarRfvzxx6sxbGtnsuIAtT6U\nsbV7tcd5jTKON2PJ4uQvuzEtwZvdmJbgzW5MS+i62SNicURsiIifRcRPI+Kznf5ZEbE+Ip7r/L/+\nBbkxZsKQEehOAviDUspTEfFmAE9GxHoA/xPAY6WUL0XE/QDuB/D50U40NDRUlS7KlPfJRAOxMKEc\nZliA6UUAAeqMLiozyrXXXlv1ceQXZ2oB6jTIKhKMnTaU+MbCmhKf2IFJras6jh1tMrXf77zzzmoM\nO94oZxh+jrfccks1hrnjjjuqvu3btzfaKiuPEiMzUZC9oN49fq6Z93wsJaO6ftlLKYOllKc6f34V\nwDYAiwDcDWBdZ9g6AB9JX9UY03fGZLNHxFIAqwFsBDCvlDLyadoHYN4ZjrkvIjZFxCb11TbG9If0\nZo+IGQD+GcDvlVIavywvwz9vyKj5UsqDpZQ1pZQ1GZ9hY8z5IeVUExFTMbzR/66U8i+d7v0RsaCU\nMhgRCwDUBhdx4sQJ7N27t9HH2UszZXuV4w3bN2oMn1v948NjlM3KDhnKjlMBPex8w/oFUAc6ZJyM\nVGYUtj+VhsEOPCq7q7p/dr7JlNm+5pprul5flZVmDUNl5OX5KCejzZs3N9oZbQjI2cS9ZI5V8JzU\n+8nzGUt2m4waHwC+BmBbKeXPT/urRwDc0/nzPQAe7nYuY8z4kfmyvxfA/wDwk4gYiUn8YwBfAvAP\nEXEvgF0APn5+pmiMORd03eyllP8GcKafZW4/t9Mxxpwv7EFnTEvoa9TbtGnTKhGGxRUlMLBwpKKz\neExGfFPCCotdymGEx6jsIer6LKYoZxgWaTLCjioHxddSc+RzK3FUnZuz8Kg58hqxkw0ALF++vNHe\ns2dPNYajJBXsnKPE0Uw5LOVo00sEW0bUy4jMmWu7/JMxpsKb3ZiW4M1uTEvoq80+ffr0KtMI29/K\nbmK7MZNlRNky7MShbCK2mzLlodW1lNMGnztTSihTWjhjI6o5ZoI8MnNUmXQz98EBRPPnz6/GLF26\ntNFW95rRYvj+1Xky9q/SlHoJjlHH8JzUmEyZszPhL7sxLcGb3ZiW4M1uTEvwZjemJfS9/BOLICzI\nqdTNGQGEj1MiTeY87OygIsHY0UZFfSmnCRa7MtlKMmQcZpTwmYkmVAId369aVyXaMSx+qow/HBWp\nBFMWedUa8hopcVatYyYzDB+n1iwTGcfrr47hdc3UcB/BX3ZjWoI3uzEtwZvdmJbgzW5MS+irQBcR\nlbiT8SpjlCCW8Y7LRNhlBA8WjVRknBKJWFxS98FkRCMFexmqFFjdUhyd6VosHKlnxs9DRSoyas14\nbZXHGK+rEgx5zkr8UvfPAm0m/bh6h8bi6TbaeXqpPffGHMY8A2PMBYk3uzEtwZvdmJbQd5u9m9OI\nsrfYJlJ2E9ttytZl+0vZbWx/KluLz62upUoJ8X1kyg0p+HqvvfZaNebo0aOjtoFcGSvVx+umstnw\nfShnmG7nPVMfw89IaQiZqDd1LeVUxShbvxuZiLbMGN5Prs9ujPFmN6YteLMb0xK82Y1pCX0V6BQs\nKGTSMquIKnbiUKIRXysjvijRhsUmda1e639latZ1ixwEgCNHjjTa7AgE1HXWM04tQO0go8Q37lNj\neN3UffCc1DPje1PvBzsZZcVAdnxSKdEyTj29pIlWY3pJNT6Cv+zGtARvdmNagje7MS2h70417ACR\ncUhgu0TZKWyzc11voLatVFYctoeV4wvb/pkyUmqcsi3ZTlPrw8epIBe22dk+V33qWure+P4zjkfq\nmfHaKnu4l7TdL7/8cjXm4MGDo573TLDN3ks5KEUm4426Fs/bmWqMMRXe7Ma0BG92Y1qCN7sxLaHv\nTjXdoqEyaZl7dTRhB41e65qzuJIR+oCco0nmPOxE8sorr1RjWLRSDjMs9Kn5ZCLI1FqzaKfENyWi\ndrt+pkbaCy+8UI3hyEB1r+q94nXLrGNmzTICYSYjUSbV9Qj+shvTErzZjWkJXTd7RFwUEU9ExNaI\n+GlE/Emnf1lEbIyIHRHxnYjo/jOpMWbcyNjsrwO4rZRyPCKmAvjviPg3AL8P4CullIci4m8A3Avg\nr0c7kXKqyTjZsPNFxkbO1HlXdhMHfmRsxLE4Noz1OOV4wza7shHZ0WbPnj3VGF7Hw4cPV2PUOnL2\nmquvvroaM3fu3EZbBeKwPrNgwYJqDNv1KhCG13Hbtm3VGH4/MnoJkNOLMjpPphwW0+t7dSa6ftnL\nMCPqytTOfwXAbQD+qdO/DsBHzunMjDHnlJTNHhGTI2ILgAMA1gN4HsCRUsrIP2G7ASw6P1M0xpwL\nUpu9lDJUSlkFYADAuwBclb1ARNwXEZsiYpNKjGiM6Q9jUuNLKUcAbABwE4CZETFi9A4AqI3C4WMe\nLKWsKaWsUYkQjDH9oatAFxFzAJwopRyJiIsBfADAlzG86T8K4CEA9wB4OHNBdqrJlCBiwUM5NrBI\npQQRFpsyNdwzEV0K5dzA58qU98nUVVcRbdu3b2+0f/jDH1Zjdu7c2Wjv27evGqPmODAw0Ghv3bq1\nGsP3qgSxW265ZdTzAjmnGo5ye+6556ox/M6o9yzzPDLRjOrZjyWjzGjHnE35p4wavwDAuoiYjOGf\nBP6hlPJoRPwMwEMR8acANgP4Wvqqxpi+03Wzl1KeBrBa9P8cw/a7MeYCwB50xrSEvgbClFIqG5Qd\nXZS9kynJlCFTtomvpQJz2LZTtp6yt7hPHZcp/8tOLcr+XLFiRaOtsrewrXnllVdWY5QzDGfBUU5O\nS5cubbTf9773VWPYRs88D7Wu7ETD8wPqe1VaiLJ/2dZXY3j9MyWjeykZBfT+7gP+shvTGrzZjWkJ\n3uzGtARvdmNaQt8FukyKYaZbpBwwNueCETLRSZl0vkpUzIh2GccOJRBy+SW1hosWNUMVbr311moM\ni3ZKaFNpqjkts3LqWbt2baOtIuM4Ek89D15/JRg+/fTTVR+TcYTKkHGYUWO4r9f67BbojDFd8WY3\npiV4sxvTEvpqs586daqyudgGUeWWMplSGWXrZrKFsD2ubKRMpppMeZ/M9ZVdz9lalG2X0UbY9r/0\n0kurMfPmzav6OAuNOo7nrbLp8PUz5bqfffbZasyOHTsa7UzW4OzzyTjV9FKOWcHPKFuiKou/7Ma0\nBG92Y1qCN7sxLcGb3ZiW0PfyT0wmEo0Fj17rmmfSVrMTSeY8SkhRIiILMCotMguLx44dq8aw84sS\nEXneKv8fz0etq8oww/er7p+jylSJKk4Tra7P96oy7mSETyUQMr2mbs6UluI+NeeMsxbDz97ln4wx\n3uzGtAVvdmNaQt+dathOZZtD2VZsx2bK7arz8LUyzijK9ub5KPtL2dE8J3Ucr8/Ro0erMWxb9hp4\nwTazWldlR/O9sXMMUNufhw4d6jpGpRrfuHFjo80ZcdX1lRbCa6/KRavjmGXLllV9HOSza9euasyL\nL77YaHO2IaB+rzJ60Vjwl92YluDNbkxL8GY3piV4sxvTEvqeqUZlQ+kGCxVKpOA+lSqYz6Mi4zLl\nlziqKpMSGqhFMuXoklmfTOYeFhaVYHn8+PFGW0WLLVy4sOrLiH8ZEZHHKMHyRz/6UaOt7oOFvkxk\nGt87oFNpv+Md72i058yZU4355Cc/Oeq1AGD9+vWN9le/+tVqzODgYKPdS9TbaCmq/WU3piV4sxvT\nErzZjWkJ3uzGtIS+CnRDQ0MyFfDpKJGIPZuUAJOpq84eYpl0Uko0YhEtUw9OzVGdm9M0qxptHJmn\nIuO43pkStvj6ShBauXJl1cdrqzzPbrrppkb7iiuuqMZw3+7du6sx7DGnvPX4PtS9zp8/v9G+/vrr\nqzGf/vSnqz5OU/2tb32rGsNps1WarhtvvLHRVum3H3jggUZbPVd+h1lAHi0llr/sxrQEb3ZjWoI3\nuzEtoe9Rb2xvZuwtjlBS9bfZmSCTLUTZ9RlHBp6jmrOyndi+Uo4mCxYsaLSV8wdHVR0+fLgaw8cp\nBx7uU1Fnyo6eOXNmo33ZZZdVY9j5RKUI5zXavHlzNYY1HnUefj8+/vGPV2NuvvnmUecH6DXi91W9\nH1/84hcb7X379lVjWGtQ11+8eHGjzeWxVN/ll1/eaI8WFecvuzEtwZvdmJaQ3uwRMTkiNkfEo532\nsojYGBE7IuI7EVH/3GyMmTCM5cv+WQDbTmt/GcBXSilXAjgM4N5zOTFjzLklJdBFxACA3wLwZwB+\nP4aVpdsAjIT7rAPw/wD89WjnUfXZWRRRAgiPydRfU+mceIyKEGIRT4loPEY5lWTqfanUROyQoSKx\nVK11htdIOTOx0KnWXjmIsCCXiXpT12cRcevWrdUYjt5Tz+z2229vtN/97ndXY/heH3nkkWqMuv6B\nAwcabZWmi0U8lTabRU31frL4qN4PXtft27c32ipKc4Tsl/0BAH8EYGSlrwBwpJQysnN3A1iUPJcx\nZhzoutkj4rcBHCilPNnLBSLivojYFBGbMon6jTHnh8yP8e8FcFdEfAjARQAuBfAXAGZGxJTO130A\nQP1LQQCllAcBPAgAM2bM6K3khjHmrOm62UspXwDwBQCIiFsB/GEp5Xcj4h8BfBTAQwDuAfBw4lyV\nAwrbO8r+ZQeETP1thXK0YdhuVcewPZ7JZgPUDhAqTTWvj8pcw3NUQTfsxMEOG0Ad5KICetS6dgtm\nAup1U44/bG+q2ut8r9ddd101Zvny5V2vtW3btkZbOQspW5ufrQpgYZTOw+95JgiL7XwAuOuuuxpt\nduD55je/ecZ5nc3v2T+PYbFuB4Zt+K+dxbmMMeeZMbnLllL+E8B/dv78cwDvOvdTMsacD+xBZ0xL\n8GY3piWMe312RtUEYwcRJVyw4KGENY6OyvwqUDmasNiixLhMrTc+jyJTx1s58HCWk71791ZjWPxT\n51FrPXv27EZbZY9hYUs9V04TPW/evK7Xv+GGG6ox7Hyi7kM5zDDXXHNN1bdjx45GW4mTnAUnU59P\nRW6yOK2e/bXXXttos9OVEllH8JfdmJbgzW5MS/BmN6Yl9D1TDds8bGMoW5ftPTWG7WbljMLHqSAP\ntvVV9pJMxhuVUYXnpAJIMg477DSinEg4YEMFcHDWE2VHquyy73nPexptZWvzmjzxxBPVmKeeeqrR\nfvvb316N+fCHP9xos30M1Gumnj3XR1d6DTs9AXX5K5XxlTMAq/VgvYgdioD6WatrPf744402OxSd\ni0AYY8wFjje7MS3Bm92YluDNbkxL6KtAN336dLztbW9r9LGzg0pnzAKUcjaYO3duo63Er0zaanYQ\nUQ4a7MShotcyKakzddXVfbDzhXLiyJQF4nvNlrFSqZIZXutHH320GsOi2Wc+85lqDEe5KYGO1+j5\n55+vxvCzVk5G+/fvr/r4eiyIAfUzU+dZunRpo63Sb3PUn3qH+Flv2bKl0Vbi5Aj+shvTErzZjWkJ\n3uzGtIS+2uxTp06tHA7YaYPtOKC2JVVQBTvrKJuInWGUHcvZQlSQC9t/ymFF2U7sxKPOzWPYGQOo\ng0NYBwFyWVn5PpSGkMmSq3SFDRs2NNovvfRSNeZzn/tco/2pT32q6xxVFhguc60Cc5YtW9Zocylm\nQGft5dLTXJ4LqJ2RVBkrvv8lS5ZUY9ipSGkPrF9lSnOP4C+7MS3Bm92YluDNbkxL8GY3piX0VaAb\nGhqqxDV2hlGperlMkEr5y2LG4OBgNYaFPhXRxsKaEvoyaauVgwpfT41hcUlFULH4phxf2PFHzZmF\nNSW0KcGHM+wosWvjxo2NtnIQYbFLOetw9KB69jt37uw6hrPQrFixohqjot7YqWdgYKAak0mtzeKn\nijBk4VVlzuFnxHtj1DmkRxpjLmi82Y1pCd7sxrSEcc8uy/afspHZruc2UDtNcPYQoM4UqsZwkInK\nFsLOMMrRQtnjbEcr5yB2GlHZdHiMsqvZRlbOKOwMpMaoclxsJ/7kJz+pxnB5JWVbfvvb32601bN/\ny1ve0mirdc2U2eZnpDQM9T7wM1IZiNgez5QCV6W4OXOQypD8zne+s9FmO18F+LwxzzP+jTHmNwpv\ndmNagje7MS3Bm92YlhDKIeO8XSziZQC7AMwGcLDL8InGhThn4MKct+fcO0tKKXPUX/R1s79x0YhN\npZQ1fb/wWXAhzhm4MOftOZ8f/GO8MS3Bm92YljBem/3Bcbru2XAhzhm4MOftOZ8HxsVmN8b0H/8Y\nb0xL6Ptmj4g7I+LZiNgREff3+/oZIuLrEXEgIp45rW9WRKyPiOc6/6+Dn8eRiFgcERsi4mcR8dOI\n+Gynf8LOOyIuiognImJrZ85/0ulfFhEbO+/IdyKidhIfZyJickRsjohHO+0JP+e+bvaImAzgrwB8\nEMA1AD4REXWE/vjzDQB3Ut/9AB4rpawE8FinPZE4CeAPSinXALgRwP/qrO1EnvfrAG4rpVwPYBWA\nOyPiRgBfBvCVUsqVAA4DuHcc53gmPgtg22ntCT/nfn/Z3wVgRynl56WUXwN4CMDdfZ5DV0op/wWA\nQ9LuBrCu8+d1AD7S10l1oZQyWEp5qvPnVzH8Ii7CBJ53GWYkHG5q578C4DYA/9Tpn1BzBoCIGADw\nWwD+ttMOTPA5A/3f7IsAnA8exsIAAAG+SURBVJ5Ae3en70JgXillJNfVPgB1vqgJQkQsBbAawEZM\n8Hl3fhzeAuAAgPUAngdwpJQyUixuIr4jDwD4IwAjsaxXYOLP2QJdL5ThX2FMyF9jRMQMAP8M4PdK\nKY1g/Ik471LKUCllFYABDP/kd9U4T2lUIuK3ARwopTw53nMZK/1OXrEHwOLT2gOdvguB/RGxoJQy\nGBELMPwlmlBExFQMb/S/K6X8S6d7ws8bAEopRyJiA4CbAMyMiCmdL+VEe0feC+CuiPgQgIsAXArg\nLzCx5wyg/1/2HwNY2VEupwH4HQCP9HkOvfIIgHs6f74HwMPjOJeKjt34NQDbSil/ftpfTdh5R8Sc\niJjZ+fPFAD6AYa1hA4CPdoZNqDmXUr5QShkopSzF8Pv7H6WU38UEnvMblFL6+h+ADwHYjmHb7P/0\n+/rJOf49gEEAJzBsf92LYbvsMQDPAfgBgFnjPU+a81oM/4j+NIAtnf8+NJHnDeAdADZ35vwMgP/b\n6V8O4AkAOwD8I4Dp4z3XM8z/VgCPXihztgedMS3BAp0xLcGb3ZiW4M1uTEvwZjemJXizG9MSvNmN\naQne7Ma0BG92Y1rC/weHKIQBzboPQQAAAABJRU5ErkJggg==\n", 365 | "text/plain": [ 366 | "
" 367 | ] 368 | }, 369 | "metadata": { 370 | "tags": [] 371 | } 372 | } 373 | ] 374 | }, 375 | { 376 | "cell_type": "markdown", 377 | "metadata": { 378 | "id": "dVJr-mxm2Pg5", 379 | "colab_type": "text" 380 | }, 381 | "source": [ 382 | "### Preprocess Data" 383 | ] 384 | }, 385 | { 386 | "cell_type": "markdown", 387 | "metadata": { 388 | "id": "pipJJEZmDRzo", 389 | "colab_type": "text" 390 | }, 391 | "source": [ 392 | "Next, we need to preprocess and prepare the federated data. Since the training data will be distributed in each user's local device, let's assume we have 3 clients, and each client has 5 images to train, in order to simulate the scenario." 393 | ] 394 | }, 395 | { 396 | "cell_type": "code", 397 | "metadata": { 398 | "id": "b4VaSktgKn3l", 399 | "colab_type": "code", 400 | "outputId": "8fba013c-e2be-4701-b98b-21069b7e64af", 401 | "colab": { 402 | "base_uri": "https://localhost:8080/", 403 | "height": 68 404 | } 405 | }, 406 | "source": [ 407 | "%tensorflow_version 2.x\n", 408 | "import tensorflow as tf\n", 409 | "\n", 410 | "print('Tensorflow version: {}'.format(tf.__version__))\n", 411 | "\n", 412 | "from tensorflow import reshape\n", 413 | "from collections import OrderedDict\n", 414 | "\n", 415 | "\"\"\"\n", 416 | "Assume we have 3 clients, each client has 5 images to train\n", 417 | "\"\"\"\n", 418 | "\n", 419 | "# Assume each client has 5 images to train\n", 420 | "TRAIN_DATA_PER_CLIENT = 5\n", 421 | "\n", 422 | "# Train 3 times for each image\n", 423 | "TRAINING_EPOCHS = 3\n", 424 | "\n", 425 | "# Prepare training data for each client\n", 426 | "x_client_1 = x_train[0 : TRAIN_DATA_PER_CLIENT]\n", 427 | "y_client_1 = y_train[0 : TRAIN_DATA_PER_CLIENT]\n", 428 | "\n", 429 | "x_client_2 = x_train[TRAIN_DATA_PER_CLIENT : TRAIN_DATA_PER_CLIENT * 2]\n", 430 | "y_client_2 = y_train[TRAIN_DATA_PER_CLIENT : TRAIN_DATA_PER_CLIENT * 2]\n", 431 | "\n", 432 | "x_client_3 = x_train[TRAIN_DATA_PER_CLIENT * 2 : TRAIN_DATA_PER_CLIENT * 3]\n", 433 | "y_client_3 = y_train[TRAIN_DATA_PER_CLIENT * 2 : TRAIN_DATA_PER_CLIENT * 3]\n", 434 | "print(x_client_3.shape)\n", 435 | "\n", 436 | "# Prepare test data\n", 437 | "x_test = x_train[TRAIN_DATA_PER_CLIENT * 3 : TRAIN_DATA_PER_CLIENT * 4]\n", 438 | "y_test = y_train[TRAIN_DATA_PER_CLIENT * 3 : TRAIN_DATA_PER_CLIENT * 4]\n", 439 | "\n", 440 | "# Create federated data\n", 441 | "def create_federated_data(x_train, y_train):\n", 442 | "\n", 443 | " orderDict = OrderedDict()\n", 444 | " pixels_list = []\n", 445 | " \n", 446 | " x_train = x_train / 255.0\n", 447 | " x_train = x_train.reshape(len(x_train), 48, 48, 1)\n", 448 | "\n", 449 | " orderDict['x'] = numpy.array(x_train)\n", 450 | " orderDict['y'] = numpy.array(y_train)\n", 451 | " dataset = tf.data.Dataset.from_tensor_slices(orderDict)\n", 452 | " batch = dataset.shuffle(10).batch(5)\n", 453 | "\n", 454 | " return batch\n", 455 | "\n", 456 | "federated_data_client_1 = [create_federated_data(x_client_1, y_client_1) for epoch in range(TRAINING_EPOCHS)]\n", 457 | "federated_data_client_2 = [create_federated_data(x_client_2, y_client_2) for epoch in range(TRAINING_EPOCHS)]\n", 458 | "federated_data_client_3 = [create_federated_data(x_client_3, y_client_3) for epoch in range(TRAINING_EPOCHS)]\n", 459 | "\n", 460 | "federated_data_test = [create_federated_data(x_test, y_test) for epoch in range(TRAINING_EPOCHS)]\n", 461 | "\n", 462 | "print('Total training epochs: {}'.format(len(federated_data_client_3)))" 463 | ], 464 | "execution_count": 0, 465 | "outputs": [ 466 | { 467 | "output_type": "stream", 468 | "text": [ 469 | "Tensorflow version: 2.1.0\n", 470 | "(5, 48, 48)\n", 471 | "Total training epochs: 3\n" 472 | ], 473 | "name": "stdout" 474 | } 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": { 480 | "id": "dO7RTarO2Tmd", 481 | "colab_type": "text" 482 | }, 483 | "source": [ 484 | "### Create Model" 485 | ] 486 | }, 487 | { 488 | "cell_type": "markdown", 489 | "metadata": { 490 | "id": "46n9fSVSFL2d", 491 | "colab_type": "text" 492 | }, 493 | "source": [ 494 | "By referring to VGGNet architecture, we have designed the computer vision model with several stacks of layers. The model will have the following components:\n", 495 | "- Convolutional layers: These layers are the building blocks of our architecture, which learns the image feature by computing the dot product between the weights and the small region on the image. Similar to VGGNet architecture, all the convolutional layers are designed with 3 x 3 kernal size, and several filters.\n", 496 | "- Activation functions: The activation functions are those functions which are applied to the outputs of the layers in the network. Specifically, we use ReLU (Rectified Linear Unit) activation function to increase the non-linearity of the network. Besides, a Softmax function will be used to compute the probability of each category.\n", 497 | "- Pooling layers: These layers will down-sample the image to reduce the spatial data and extract features. In our model, we will use Max Pooling with A 3 x 3 pooling size and a 2 x 2 stride.\n", 498 | "- Dense layers: The dense layers are stacked as the fully connected layers of the network, which take in the feature data from the previous convolutional layers and perform decision making.\n", 499 | "- Dropout layers: The dropout layers are used to prevent over-fitting when training the model.\n", 500 | "- Batch normalization: This technique can be used to speed up learning by normalizing the output of the previous activation layer. \n", 501 | "\n", 502 | "The diagram of the model is displayed as follows.\n", 503 | "\n", 504 | "![cnn](https://drive.google.com/uc?id=1jjORxRgvEDDMLZ-mX5JkCnUWUFn7IHpL)\n", 505 | "\n", 506 | "Our model contains 5 stacks of layers. In each of the first 4 stacks of layers, there are 2 convolutional layer followed by 1 pooling layer. Besides, we use batch normalization to speed up training and dropout to prevent over-fitting. Then we have one stack of 3 fully-connected layers, followed by a Softmax activation function, which generates the probability of the facial expression categories. Finally, we compile our model using Adam optimizer with a certain learning rate. Considering that we are dealing with classification issue, we will use `sparse_categorical_crossentropy` as the loss function." 507 | ] 508 | }, 509 | { 510 | "cell_type": "code", 511 | "metadata": { 512 | "id": "8F0LQtWLRqV3", 513 | "colab_type": "code", 514 | "outputId": "6d89e4a3-6cd2-4d34-aa4f-6664ff996cfd", 515 | "colab": { 516 | "base_uri": "https://localhost:8080/", 517 | "height": 1000 518 | } 519 | }, 520 | "source": [ 521 | "from tensorflow.keras.models import Sequential\n", 522 | "from tensorflow.keras.layers import Conv2D, BatchNormalization, MaxPool2D, Dropout, Flatten, Dense\n", 523 | "\n", 524 | "from tensorflow.keras.optimizers import SGD\n", 525 | "from tensorflow.keras.losses import SparseCategoricalCrossentropy\n", 526 | "from tensorflow.keras.metrics import SparseCategoricalAccuracy\n", 527 | "\n", 528 | "def create_keras_model():\n", 529 | "\n", 530 | " # Create keras model\n", 531 | " model = Sequential()\n", 532 | "\n", 533 | " # 1st convolution layer\n", 534 | " model.add(Conv2D(64, input_shape=(48, 48, 1), kernel_size=(3, 3), activation='relu'))\n", 535 | " model.add(BatchNormalization())\n", 536 | " model.add(Conv2D(64, padding='same', kernel_size=(3, 3), activation='relu'))\n", 537 | " model.add(BatchNormalization())\n", 538 | " model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2)))\n", 539 | " model.add(Dropout(0.3))\n", 540 | "\n", 541 | " # 2nd convolution layer\n", 542 | " model.add(Conv2D(128, padding='same', kernel_size=(3, 3), activation='relu'))\n", 543 | " model.add(BatchNormalization())\n", 544 | " model.add(Conv2D(128, padding='same', kernel_size=(3, 3), activation='relu'))\n", 545 | " model.add(BatchNormalization())\n", 546 | " model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2)))\n", 547 | " model.add(Dropout(0.3))\n", 548 | "\n", 549 | " # 3rd convolution layer\n", 550 | " model.add(Conv2D(256, padding='same', kernel_size=(3, 3), activation='relu'))\n", 551 | " model.add(BatchNormalization())\n", 552 | " model.add(Conv2D(256, padding='same', kernel_size=(3, 3), activation='relu'))\n", 553 | " model.add(BatchNormalization())\n", 554 | " model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2)))\n", 555 | " model.add(Dropout(0.3))\n", 556 | "\n", 557 | " # 4th convolution layer\n", 558 | " model.add(Conv2D(512, padding='same', kernel_size=(3, 3), activation='relu'))\n", 559 | " model.add(BatchNormalization())\n", 560 | " model.add(Conv2D(512, padding='same', kernel_size=(3, 3), activation='relu'))\n", 561 | " model.add(BatchNormalization())\n", 562 | " model.add(MaxPool2D(pool_size=(3, 3), strides=(2, 2)))\n", 563 | " model.add(Dropout(0.3))\n", 564 | "\n", 565 | " # Fully connected layer\n", 566 | " model.add(Flatten())\n", 567 | " model.add(Dense(512, activation='relu'))\n", 568 | " model.add(Dropout(0.3))\n", 569 | " model.add(Dense(256, activation='relu'))\n", 570 | " model.add(Dropout(0.3))\n", 571 | " model.add(Dense(64, activation='relu'))\n", 572 | " model.add(Dropout(0.3))\n", 573 | "\n", 574 | " model.add(Dense(7, activation='softmax'))\n", 575 | "\n", 576 | " # Compile the model\n", 577 | " model.compile(loss=SparseCategoricalCrossentropy(),\n", 578 | " optimizer=SGD(learning_rate=0.02),\n", 579 | " metrics=[SparseCategoricalAccuracy()])\n", 580 | " \n", 581 | " return model\n", 582 | "\n", 583 | "# Summary model\n", 584 | "keras_model = create_keras_model()\n", 585 | "keras_model.summary()" 586 | ], 587 | "execution_count": 0, 588 | "outputs": [ 589 | { 590 | "output_type": "stream", 591 | "text": [ 592 | "Model: \"sequential\"\n", 593 | "_________________________________________________________________\n", 594 | "Layer (type) Output Shape Param # \n", 595 | "=================================================================\n", 596 | "conv2d (Conv2D) (None, 46, 46, 64) 640 \n", 597 | "_________________________________________________________________\n", 598 | "batch_normalization (BatchNo (None, 46, 46, 64) 256 \n", 599 | "_________________________________________________________________\n", 600 | "conv2d_1 (Conv2D) (None, 46, 46, 64) 36928 \n", 601 | "_________________________________________________________________\n", 602 | "batch_normalization_1 (Batch (None, 46, 46, 64) 256 \n", 603 | "_________________________________________________________________\n", 604 | "max_pooling2d (MaxPooling2D) (None, 22, 22, 64) 0 \n", 605 | "_________________________________________________________________\n", 606 | "dropout (Dropout) (None, 22, 22, 64) 0 \n", 607 | "_________________________________________________________________\n", 608 | "conv2d_2 (Conv2D) (None, 22, 22, 128) 73856 \n", 609 | "_________________________________________________________________\n", 610 | "batch_normalization_2 (Batch (None, 22, 22, 128) 512 \n", 611 | "_________________________________________________________________\n", 612 | "conv2d_3 (Conv2D) (None, 22, 22, 128) 147584 \n", 613 | "_________________________________________________________________\n", 614 | "batch_normalization_3 (Batch (None, 22, 22, 128) 512 \n", 615 | "_________________________________________________________________\n", 616 | "max_pooling2d_1 (MaxPooling2 (None, 10, 10, 128) 0 \n", 617 | "_________________________________________________________________\n", 618 | "dropout_1 (Dropout) (None, 10, 10, 128) 0 \n", 619 | "_________________________________________________________________\n", 620 | "conv2d_4 (Conv2D) (None, 10, 10, 256) 295168 \n", 621 | "_________________________________________________________________\n", 622 | "batch_normalization_4 (Batch (None, 10, 10, 256) 1024 \n", 623 | "_________________________________________________________________\n", 624 | "conv2d_5 (Conv2D) (None, 10, 10, 256) 590080 \n", 625 | "_________________________________________________________________\n", 626 | "batch_normalization_5 (Batch (None, 10, 10, 256) 1024 \n", 627 | "_________________________________________________________________\n", 628 | "max_pooling2d_2 (MaxPooling2 (None, 4, 4, 256) 0 \n", 629 | "_________________________________________________________________\n", 630 | "dropout_2 (Dropout) (None, 4, 4, 256) 0 \n", 631 | "_________________________________________________________________\n", 632 | "conv2d_6 (Conv2D) (None, 4, 4, 512) 1180160 \n", 633 | "_________________________________________________________________\n", 634 | "batch_normalization_6 (Batch (None, 4, 4, 512) 2048 \n", 635 | "_________________________________________________________________\n", 636 | "conv2d_7 (Conv2D) (None, 4, 4, 512) 2359808 \n", 637 | "_________________________________________________________________\n", 638 | "batch_normalization_7 (Batch (None, 4, 4, 512) 2048 \n", 639 | "_________________________________________________________________\n", 640 | "max_pooling2d_3 (MaxPooling2 (None, 1, 1, 512) 0 \n", 641 | "_________________________________________________________________\n", 642 | "dropout_3 (Dropout) (None, 1, 1, 512) 0 \n", 643 | "_________________________________________________________________\n", 644 | "flatten (Flatten) (None, 512) 0 \n", 645 | "_________________________________________________________________\n", 646 | "dense (Dense) (None, 512) 262656 \n", 647 | "_________________________________________________________________\n", 648 | "dropout_4 (Dropout) (None, 512) 0 \n", 649 | "_________________________________________________________________\n", 650 | "dense_1 (Dense) (None, 256) 131328 \n", 651 | "_________________________________________________________________\n", 652 | "dropout_5 (Dropout) (None, 256) 0 \n", 653 | "_________________________________________________________________\n", 654 | "dense_2 (Dense) (None, 64) 16448 \n", 655 | "_________________________________________________________________\n", 656 | "dropout_6 (Dropout) (None, 64) 0 \n", 657 | "_________________________________________________________________\n", 658 | "dense_3 (Dense) (None, 7) 455 \n", 659 | "=================================================================\n", 660 | "Total params: 5,102,791\n", 661 | "Trainable params: 5,098,951\n", 662 | "Non-trainable params: 3,840\n", 663 | "_________________________________________________________________\n" 664 | ], 665 | "name": "stdout" 666 | } 667 | ] 668 | }, 669 | { 670 | "cell_type": "markdown", 671 | "metadata": { 672 | "id": "PCBQ_5XmFmS-", 673 | "colab_type": "text" 674 | }, 675 | "source": [ 676 | "Next, wrap our compiled Keras model in an instance of the `tff.learning.Model` interface, in order to use the Tensorflow federated learning." 677 | ] 678 | }, 679 | { 680 | "cell_type": "code", 681 | "metadata": { 682 | "id": "sE-xl-Ira8c4", 683 | "colab_type": "code", 684 | "outputId": "554425f6-cf37-4be0-ac4c-cff5ffb6c4dd", 685 | "colab": { 686 | "base_uri": "https://localhost:8080/", 687 | "height": 34 688 | } 689 | }, 690 | "source": [ 691 | "import tensorflow_federated as tff\n", 692 | "\n", 693 | "# Create dummy batch\n", 694 | "dummy_batch = tf.nest.map_structure(lambda x: x.numpy(), iter(federated_data_client_1[0]).next())\n", 695 | "print(dummy_batch['x'].shape)\n", 696 | "\n", 697 | "def create_federated_model():\n", 698 | "\n", 699 | " # Create keras model\n", 700 | " keras_model = create_keras_model()\n", 701 | " \n", 702 | " # Convert keras model to federated model\n", 703 | " return tff.learning.from_compiled_keras_model(keras_model, dummy_batch)" 704 | ], 705 | "execution_count": 0, 706 | "outputs": [ 707 | { 708 | "output_type": "stream", 709 | "text": [ 710 | "(5, 48, 48, 1)\n" 711 | ], 712 | "name": "stdout" 713 | } 714 | ] 715 | }, 716 | { 717 | "cell_type": "markdown", 718 | "metadata": { 719 | "id": "LtOjQitE2WF-", 720 | "colab_type": "text" 721 | }, 722 | "source": [ 723 | "### Train Model" 724 | ] 725 | }, 726 | { 727 | "cell_type": "markdown", 728 | "metadata": { 729 | "id": "B19X3LvXXq6f", 730 | "colab_type": "text" 731 | }, 732 | "source": [ 733 | "Then, we need to build federated averate process and start training." 734 | ] 735 | }, 736 | { 737 | "cell_type": "code", 738 | "metadata": { 739 | "id": "jmqJXOfzbfIL", 740 | "colab_type": "code", 741 | "outputId": "73b41ac7-ddce-476b-b963-aa889bff3a74", 742 | "colab": { 743 | "base_uri": "https://localhost:8080/", 744 | "height": 496 745 | } 746 | }, 747 | "source": [ 748 | "# Build federated average process\n", 749 | "trainer = tff.learning.build_federated_averaging_process(create_federated_model)\n", 750 | "\n", 751 | "# Create initial state\n", 752 | "train_state = trainer.initialize()" 753 | ], 754 | "execution_count": 0, 755 | "outputs": [ 756 | { 757 | "output_type": "stream", 758 | "text": [ 759 | "WARNING:tensorflow:From /tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n", 760 | "Instructions for updating:\n", 761 | "If using Keras pass *_constraint arguments to layers.\n", 762 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 763 | "\n", 764 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 765 | "\n", 766 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 767 | "\n", 768 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 769 | "\n", 770 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 771 | "\n", 772 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 773 | "\n", 774 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 775 | "\n", 776 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 777 | "\n", 778 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 779 | "\n", 780 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 781 | "\n", 782 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 783 | "\n", 784 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 785 | "\n" 786 | ], 787 | "name": "stdout" 788 | } 789 | ] 790 | }, 791 | { 792 | "cell_type": "code", 793 | "metadata": { 794 | "id": "Am2yZAkAGFs0", 795 | "colab_type": "code", 796 | "outputId": "2e4d4d88-82af-4544-e2e9-b3afef015b88", 797 | "colab": { 798 | "base_uri": "https://localhost:8080/", 799 | "height": 692 800 | } 801 | }, 802 | "source": [ 803 | "# Train on federated model\n", 804 | "print('========== Train on Local Device of Client 1 ==========')\n", 805 | "\n", 806 | "history_accuracy_1 = []\n", 807 | "history_loss_1 = []\n", 808 | "\n", 809 | "for round_num in range(6):\n", 810 | " train_state, train_metrics = trainer.next(train_state, federated_data_client_1)\n", 811 | " history_accuracy_1.append(train_metrics.sparse_categorical_accuracy)\n", 812 | " history_loss_1.append(train_metrics.loss)\n", 813 | " print('round {:2d}, metrics={}'.format(round_num, train_metrics))\n", 814 | "\n", 815 | "# Show accuracy diagram\n", 816 | "plt.title('Model Accuracy for Client 1')\n", 817 | "plt.plot(history_accuracy_1, label='accuracy')\n", 818 | "plt.xlabel('Epoch')\n", 819 | "plt.ylabel('Accuracy')\n", 820 | "plt.show()\n", 821 | "\n", 822 | "# Show loss diagram\n", 823 | "plt.title('Model Loss for Client 1')\n", 824 | "plt.plot(history_loss_1, label='loss')\n", 825 | "plt.xlabel('Epoch')\n", 826 | "plt.ylabel('Loss')\n", 827 | "plt.show()" 828 | ], 829 | "execution_count": 0, 830 | "outputs": [ 831 | { 832 | "output_type": "stream", 833 | "text": [ 834 | "========== Train on Local Device of Client 1 ==========\n", 835 | "round 0, metrics=\n", 836 | "round 1, metrics=\n", 837 | "round 2, metrics=\n", 838 | "round 3, metrics=\n", 839 | "round 4, metrics=\n", 840 | "round 5, metrics=\n" 841 | ], 842 | "name": "stdout" 843 | }, 844 | { 845 | "output_type": "display_data", 846 | "data": { 847 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUdb74/9c7jd4TAiSEJPQmoYgI\nSrEgNsC1YS8osFdd3ebq3r1+vd713l3v7np1LQj2ttYlYkVXJYAiEEzoNaFlkkBCDS31/ftjDv7G\nGMIkZHIyM+/n4zEPTp3znhky7zmfzznvj6gqxhhjwleE2wEYY4xxlyUCY4wJc5YIjDEmzFkiMMaY\nMGeJwBhjwpwlAmOMCXOWCEzAiEiyiKiIRPmx7a0isqQx4mrqROTnIrJbRA6LSKdGON7DIvK6M53k\nHDcy0Mc1TYclAgOAiGwXkTIRia22PMv5Mk92J7IfxdLa+ZL61O1YAkVEooG/ARNVtbWq7m2g571e\nRDKd969ARD4VkXOqb6eqO53jVjbAMReKyB2n2GaOiGwSkSoRufV0j2nqxxKB8bUNuO7EjIgMBlq6\nF85PXAmUAheKSJfGPLA/ZzUNJB5oDqyr647i9ZO/aRH5FfB/wH87z58EPANMOb1QG8Qq4N+A790O\nJJxZIjC+XgNu9pm/BXjVdwMRaScir4pIkYjsEJE/nPjyEZFIEfmLiBSLSC5waQ37vuD8IvWIyB/r\n2ARxCzAbWA3cWO25u4vIP5249orIUz7r7hSRDSJSIiLrRWSYs1xFpJfPdi+LyB+d6fEikicivxOR\nQuAlEekgIh85x9jvTCf67N9RRF4SkXxnfbqzfK2IXO6zXbTzHg2t9hr6AJuc2QMi8pWzfLSIrBCR\ng86/o332WSgij4rIN8BRILX6ew48Atylqv9U1SOqWq6qH6rqb6u/wdWb82r7zE405zmf+X4R2SYi\nFzvrHgXOBZ5yzkKeqn4sAFV9WlW/BI7XtN40DksExtd3QFsR6e/8sU8DXq+2zd+Bdni/cMbhTRy3\nOevuBC4DhgIjgKuq7fsyUAH0craZCNTadHCCiPQAxgNvOI+bfdZFAh8BO4BkIAF4y1l3NfCws31b\nYDLgb3NLF6Aj0AOYgffv5SVnPgk4Bvh+wb2G9wxqINAZeNxZ/io/TlyXAAWqmuV7MFXd7OwL0F5V\nzxORjsDHwJNAJ7zNRh9X6zu4yYmvjfMe+Dob7xnGPD9fc3UvU/tndhbe5BULPAa8ICKiqv8OLAbu\ndpqa7q7n8U1jUFV72ANgO3AB8Afgf4BJwBdAFKB4v2AjgTJggM9+M4GFzvRXwCyfdROdfaPwNkmU\nAi181l8HfO1M3wosqSW+PwDZznQCUAkMdebPBoqAqBr2WwDce5LnVKCXz/zLwB+d6fHOa21eS0xp\nwH5nuitQBXSoYbtuQAnQ1pl/D7j/JM+ZfOI9c+ZvApZX22YpcKszvRB4pJYYbwAKT/HZPwy8Xv34\nfn5mW33WtXT27eIT2x1+/v9bcuI12aPxH43V7mmCx2vAIiCFas1CeH/1RfPjX5078H4xg/cLb1e1\ndSf0cPYtEJETyyKqbV+bm4G5AKrqEZEMvE1FWUB3YIeqVtSwX3cgx89jVFekqj80WYhIS7y/8icB\nHZzFbZwzku7APlXdX/1JVDXfabq5UkTmARcD9/oZQzd++ivf9z2H2t/DvUCsiESd5P2pjT+fWeGJ\nCVU96mzXuo7HMS6zpiHzI6q6A2+n8SXAP6utLgbK8X5BnJAEeJzpArxfiL7rTtiF99dlrKq2dx5t\nVXUgp+C0ifcGHhSRQqfN/izgeqctexeQdJIO3V1Az5M89VF+3BlevQO6emneXwN9gbNUtS0w9kSI\nznE6ikj7kxzrFbzNQ1cDS1XVc5Ltqsvnx+83/Pg9rylOX0vxvu9T/Tyer3p/Zn7EZZoQSwSmJtOB\n81T1iO9C9V5S+A7wqIi0cdrtf8X/34/wDvALEUkUkQ7AAz77FgCfA38VkbYiEiEiPUVknB/x3IK3\nmWoA3uaYNGAQ0ALvr+vleJPQn0SklYg0F5Exzr7PA78RkeHOVTW9nLgBsvEmk0gRmYS3z6M2bfD2\nCxxw2u7/X7XX9ynwjNOpHC0iY332TQeG4T0TqH6mVZtPgD7ivfwzSkSudd6Hj/zZWVUPAg8BT4vI\nVBFp6cR2sYg8dop9T+czA9hNtc7r6kQkRkSa402m0c5nZ99LjczecPMTqpqjqpknWX0PcATIxduu\n+ybworNuLt42+VV4LwesfkZxMxADrAf2420r71pbLM6XxDXA31W10OexDW8z1i1Ogrocb4fmTiAP\nuNZ5Le8CjzpxluD9Qu7oPP29zn4H8Lalp9cWC95LMFvgPTP6Dvis2vqb8J4xbQT2APedWKGqx4D3\n8Ta5VX9fTkq99xFchvdsZC9wP3CZqhbX4Tn+ijdh/wFvX8ou4G5O/XqhHp+ZjyeAq5wrip48yTaf\n402uo4E5zvTYk2xrAkRU7ezNmMYgIg8BfVT1xlNubEwjss5iYxqB05Q0He9ZgzFNijUNGRNgInIn\n3uaYT1V1kdvxGFOdNQ0ZY0yYszMCY4wJc0HXRxAbG6vJycluh2GMMUFl5cqVxaoaV9O6oEsEycnJ\nZGae7MpGY4wxNRGR6neo/8CahowxJsxZIjDGmDBnicAYY8KcJQJjjAlzlgiMMSbMBTQRiMgk8Q5M\nvVVEHqhhfZKIfC3eAdJXi8glgYzHGGPMTwUsETiDdTyNt0zwAOA6ERlQbbM/AO+o6lC8wyI+E6h4\njDHG1CyQZwQj8Q5jl6uqZXjHkJ1SbRvFO44seMfBzQ9gPMYYE5TKK6t49OP15B84FpDnD2QiSODH\nQ9rl8ePh9cA7VuqNIpKHdwCOe2p6IhGZISKZIpJZVFQUiFiNMaZJOl5eyc9fX8ncxdv4etOegBzD\n7c7i64CXVTUR79CIr9U0OpGqzlHVEao6Ii6uxjukjTEm5Bw6Xs7NLy7ny417+K+pg7jhrOqjljaM\nQJaY8PDj8WsT+fE4q+Ctzz4JQFWXOqNRxeId3ckYY8JW8eFSbnlxOZsKS/i/a9OYkla9QaXhBPKM\nYAXQW0RSRCQGb2fw/Grb7ATOBxCR/kBzvEPpGWNM2PIcOMY1s5eSU3SYuTePCGgSgACeEahqhYjc\njXcM20jgRVVdJyKPAJmqOh/vOKxzReSXeDuOb1UbIMEYE8a27jnMTS8s43BpBa9NP4szkzueeqfT\nFNDqo6r6Cd5OYN9lD/lMrwfGBDIGY4wJFmvyDnLLS8uJEHhrxigGdmvXKMcNujLUxhgTipbm7OXO\nVzNp1yKa1+84i5TYVo12bEsExhjjsi/W7+auN7+nR8eWvDb9LLq0a96ox7dEYIwxLnp/ZR73v7+a\nQQntePnWM+nQKqbRY7BEYIwxLnlxyTYe+Wg9Y3p14rmbRtC6mTtfyZYIjDGmkakqj3+xmSe/2sqk\ngV144ro0mkVFuhaPJQJjjGlEVVXKwx+u49WlO7hmRCL/fcVgoiLdLfJgicAYYxpJeWUVv3l3FR9k\n5zNjbCoPXtwPEXE7LEsExhjTGI6VVXLXm9/z1cY93D+pLz8f17NJJAGwRGCMMQF38Fg5d7yygswd\n+3n0isAVj6svSwTGGBNARSXe4nFb9pTw5LShXD6km9sh/YQlAmOMCZBd+45y0wvL2H2olOdvOZNx\nfZpmGX1LBMYYEwBbdpdw0wvLOVpWwet3jGR4j8AXj6svSwTGGNPAVu06wK0vLScqMoK3Z55N/65t\nT72TiywRGGNMA/p2azF3vppJx9YxvD79LHp0arzicfVlicAYYxrIgnWF3PNmFimxrXh1+kji2zZu\n8bj6skRgjDEN4J3MXTzw/mqGdG/PS7eeSfuWjV88rr4sERhjzGl6fnEuf/x4A+f2jmX2jcNp5VLx\nuPoKrmiNMaYJUVX++vlmnvp6K5cM7sLj17pbPK6+LBEYY0w9VFYpD32wljeW7WTamd159IrBREY0\njZIRdWWJwBhj6qisoopfvZPNR6sLmDWuJ7+b1LfJ1A2qD0sExhhTB8fKKpn1+koyNhfxwMX9mDWu\np9shnTZLBCHscGkFRSWljToItjGh7ODRcm5/ZQVZO/fzp58NZtrIJLdDahCWCELYw/PX8d7KPM7t\nHcvMsT0Z06tTUJ++GuOmPSXHufmF5eQUHeap64dxyeCubofUYCwRhKgjpRV8vLqAwQnt2FhYwo0v\nLGNwQjtmjkvl4kFdg7ZTyxg37Np3lBtfWEZRSSkv3nom5/ZumsXj6ssSQYj6Yv1ujpVX8tDlAxic\n0I55WR7mLMrl7jezSOq4iTvHpnL18ESaRwffpW7GNKZNhSXc9MIySiuqeP2OsxiW1MHtkBqcuwNl\nmoCZl+UhsUMLhid1oHl0JNeNTOJfvxrH7BuH0aFVDP+RvpZz/vwVT321hYNHy90O15gmKWvnfq55\nbikA78w8OySTANgZQUgqKill8ZYi/m18LyJ8moAiI4RJg7py0cAuLNu2j9kZOfzl8808szCH60Ym\nMf2cFLq1b+Fi5MY0HUu2FDPjtUzi2jTj9eln0b1jS7dDChhLBCHow1X5VClMHVrzSEgiwqjUToxK\n7cSGgkM8l5HDy99u55VvtzMlLYFZ41LpHd+mkaM2pun4dE0B976VTWqct3hc5zbBUTyuvkRV3Y6h\nTkaMGKGZmZluh9GkTX5qCarw4T3n+L3Prn1HeWHJNt5esYtj5ZVc0L8zM8f15MzkpjuYhjGB8PaK\nnTz4zzUMTerAi7ecSbuW0W6H1CBEZKWqjqhpnfURhJitew6zOu8gU4cm1Gm/7h1b8vDkgXzzwHnc\nd0FvVu7Yz9Wzl3Lls9/yxfrdVFUF1w8GY+rjuYwcfvf+Gs7tHcdr00eGTBI4FWsaCjEfZHuIELh8\nSP2uce7YKob7LujDjLGpvJuZx9zFudz5aia9OrdmxthUpqYlEBNlvx9MaFFVHluwiWcX5nDZGV35\n2zVpYfX/PHxeaRhQVeZleTind9xpt2m2jIniltHJLPzNeJ6YlkZ0ZAT3v7easY99zdxFuRwurWig\nqI1xV2WV8vt5a3l2YQ7Xn5XEE9OGhlUSAEsEIWXljv3k7T/GFSfpJK6PqMgIpqQl8MkvzuGV20eS\nEtuKRz/ZwOj/+ZL/XbCRopLSBjuWMY2trKKKX/wji38s38ldE3ry6NRBYXmzZUCbhkRkEvAEEAk8\nr6p/qrb+cWCCM9sS6Kyq7QMZUyibl+WhRXQkEwd0afDnFhHG9YljXJ84Vu06wOyMHJ5ZmMPcxdu4\nangiM85NJdlqGpkgcrSsgpmvrWTxlmL+/ZL+3Dk21e2QXBOwRCAikcDTwIVAHrBCROar6voT26jq\nL322vwcYGqh4Ql1ZRRUfrS5g4sD4gI+ONKR7e569cTi5RYeZu3gb72Xm8Y/lO7l4UBdmjevJGYmW\ny03TduBoGbe/vILsXQd47MozuObM7m6H5KpANg2NBLaqaq6qlgFvAVNq2f464B8BjCekLdy0h4PH\nyut8tdDpSI1rzf/8bDBLHpjArHE9WbylmMlPfcP1c79j0eYigu3SZBMe9hw6zrXPfcdazyGeuWFY\n2CcBCGwiSAB2+cznOct+QkR6ACnAVwGMJ6SlZ3vo1CqGc3vFNvqxO7dpzu8m9ePbB87j95f0I6fo\nMDe/uJxLn1zCB9keKiqrGj0mY2qyY+8Rrpz9Lbv2H+Wl285k0qDQqSB6OppKZ/E04D1VraxppYjM\nEJFMEcksKipq5NCavoPHyvnXhj1cPqQbUZHufaRtmkczY2xPFt0/gceuPIPSikrufSubCX9dyKtL\nt3OsrMaP15hGsbHwEFfNXkrJ8QrevHMUY1z40dRUBfJbwwP4nnMlOstqMo1amoVUdY6qjlDVEXFx\noVX+tSF8traAsooqrmjEZqHaNIuK5Jozu/PFL8cx56bhxLVuxkMfrGPMn7/iiX9tYf+RMrdDNGFm\n5Y59XDN7KZEivDvzbNK6Wz+Wr0D2Kq4AeotICt4EMA24vvpGItIP6AAsDWAsIW1elofU2FackdjO\n7VB+JCJCmDiwCxcOiGfF9v3Mzsjh8X9tZnZGDtNGdueOc1NJsCJ3JsAyNhcx67WVxLdtxmshXjyu\nvgKWCFS1QkTuBhbgvXz0RVVdJyKPAJmqOt/ZdBrwllrPYr14Dhzju9x9/OrCPk129DERYWRKR0am\ndGRTYQnPLcrhtaU7eHXpDiYP6cbMcan069LW7TBNCPp4dQH3vZ1F785teOX2kcS1aeZ2SE2SFZ0L\ncs8uzOHPn21k0W8nkNQpeH7peA4c44XF23hrxU6OllUyoW8cs8b1ZGRKxyab0Exw+cfynfx+3hpG\n9OjA87ecSbsW4VE36GRqKzpniSCIqSoX/d8i2jSP5v2fj3Y7nHo5cLSM15bu4OVvt7P3SBlDk9oz\nc2xPJg6I/9FYCsbUxYkfSBP6xvHMDcNpEWMj8Vn10RC1oaCEzbsPN+q9Aw2tfcsY7jm/N988cB7/\nNWUgxYdLmfX6Si54PIO3V+yktMKuNDL+U1X+55MN/PmzjUxJ68acm0dYEvCDJYIglp7tISpCuGxw\n8F8L3Tw6kpvOTubrX4/n79cNpUV0pLcc8J+/ZnZGDoeO23CapnaVVcoD76/huUW53Hx2Dx6/xlss\n0ZyalaEOUpVVygfZHsb37UyHVjFuh9NgoiIjuHxINy47oytLthYzOyOHP326kae/2soNo3pw+5hk\nOrcN7dGiTN2VVlRy31vZfLq2kF+c14tfNuGLJ5oiSwRB6rvcvew+VMpDlwVvs1BtRIRze8dxbu84\n1uQdZPaiHOYsyuHFJdv42bAEZoxNJTWutdthmibgSKm3eNySrcX8x2UDmH5OitshBR1LBEFqXpaH\nNs2iOL9/Z7dDCbjBie14+vphbC8+wtzFuby7Mo+3M3dx0YAuzBrf024OCmP7j5Rx28srWOM5yF+u\nHsJVwxPdDikoWSIIQsfKKvlsbSGXDO5C8+jw6QhLjm3Fo1cM5r4L+vDKt9t5del2PltXyKjUjswc\n15PxfeKsOSCMFB48zk0vLGPHvqM8e8MwJg5s+PLr4cJ6UoLQvzbs5nBpRVBfLXQ64to04zcX9eXb\nB8/nD5f2Z8feo9z20goufmIx6Vkeyq3IXcjbXnyEq2Z/S/6BY7x825mWBE6TJYIglJ7loWu75oxK\n6eR2KK5q3SyKO85NJeO3E/jL1UOorFLuezub8f+7kJe+2cbRMhtOM9QUHy7lLws2cflTSzhSWsE/\nZoxidE8rHne6rGkoyOw9XErG5iKmn5tiN1w5YqIiuGp4Ij8bmsBXG/cwOyOH//xwPU9+uYWbz07m\nltHJdAyhK6vC0Y69R5izKJf3VuZRVlnFxAHx/G5SP7tgoIFYIggyH68poKJKm0yl0aYkIkK4YEA8\nFwyIJ3P7PmZn5PLEl1t4blEO147wFrmzgmPBZa3nIM9m5PDpmgKiIiK4YmgCM8al0tMSQIOyRBBk\n5mV56NeljRVpO4URyR15PrkjW3aXMGdRLm8u38nry3Zy2RldmTm2JwO62fvXVKkqS7YW81xGLku2\nFtOmWRR3jk3l9jEpxNs9JAFhiSCIbC8+QtbOAzx4cT+3QwkavePb8L9XD+FXE/vw4pJtvLlsJx9k\n5zO2TxyzxqVydmonu9KoiaiorOLTtYU8tyiHtZ5DxLVpxu8m9eOGUUm0bR7eBeMCzRJBEEnP9iAC\nk9O6uR1K0OnargX/fukA7p7Qm9eX7eClb7Zz/dxlDElsx6xxPZk4sAuR1ufiiuPllbybuYu5i7ex\nc99RUmNb8aefDeaKYQk0iwqfy6PdZIkgSKgq6Vkezk7tRNd2NphLfbVrGc1dE3ox/ZwU3v8+j7mL\ncvn5G9+TEtuKO89N5WfDEsLq3gw3Va88m9a9Pb+/pD8XDoi3pNzILBEEiexdB9i+9yj/NqGX26GE\nhObRkdxwVg+mnZnEgnWFzM7I4ffz1vC3LzZz25hkbhzVI+zr1wdK/oFjPO8zFsV4ZyyKs2wsCtdY\nIggS6VkemkVFMGmQ3TjTkCIjhEsGd+XiQV1YmrOXZzNy+N8Fm3h2YQ7Xn5XE7WNS6NLOOigbwubd\nJczOyGF+dj4KXH5GV2aO60n/rtZx7zZLBEGgvLKKD1cXcMGAeOs0CxARYXSvWEb3imWt5yBzFuXy\n/OJcXvpmG1PTEpg5LpVendu4HWbQUVVWbN/Pcxk5fLlxDy2iI7lxVA/uODeFxA52KW9TYYkgCCze\nUsS+I2VckWb3DjSGQQntePK6ofz2or7MXZzLO5m7eHdlHhcOiGfWuJ4M79HB7RCbvKoq5V8bdjM7\nI4fvdx6gQ8tofnlBH24+u0dIlU0PFZYIgkB6Vj4dWkYztk+c26GEle4dW/LIlEHce35vXlm6g1eX\nbueL9bsZmdyRmeNSmdC3s93dXU1pRSUfZOXz3KIccoqOkNihBf85eSDXjOhuI4U1YZYImrjDpRV8\nvr6Qq4d3JybKSkO5oVPrZvzqwj7MHJvK2yt28cKSbUx/JZM+8a2ZObYnk9O6hf1IWCXHy/nH8p28\nsGQbuw+V0r9rW56Ylsalg7sSFebvTTCwRNDELVhbyPHyqrCtNNqUtGoWxe3npHDT2T34aHU+z2Xk\n8ut3V/HXzzdx+zkpXDcyiVbNwutPak/JcV76Zjuvf7eDkuMVnJ3aiceuGsLY3rF2BVAQCa//tUEo\nPdtDUseWDEuywVeaiujICK4YmsjUtAQWbipidkYOf/x4A3//ais3n92DW0YnE9u6mdthBtS2Ym8R\nuPe/z6O8soqLB3Vh5tieDLFBgoKSJYImbPeh43yztZi7z+ttv66aIBFhQr/OTOjXmayd+5mdkcNT\nX29lzqJcrh6RyIxze5LUKbSujFm16wCzM3L4bF0h0ZHeqq93nptKSmwrt0Mzp8ESQRP24ap8qhSm\nWkmJJm9oUgeeu2kEOUWHmZORyzsr8nhz2U4uGdyVWeN6Miihndsh1puqsmhLMbMX5rA0dy9tmkfx\n83E9uXVMMp3b2D0WoUBU1e0Y6mTEiBGamZnpdhiN4tInFxMVGcEHd41xOxRTR7sPHefFb7bx5nc7\nKSmt4Jxescwa15MxvYKnyF1FZRUfrylgdkYuGwoOEd+2GdOdvpA2dj9L0BGRlao6oqZ1dkbQRG3e\nXcK6/EP8v8sHuB2KqYf4ts158OL+3DWhF298t5MXv9nGjS8sY3BCO2aOS+XiQV2bbD2dY2WVvJO5\ni7mLc8nbf4yeca147KozmJLWzYrAhShLBE1UepaHyAjhsjOsWSiYtW0ezc/H9+T2c5KZ972HOYty\nufvNLJI6buLOsalcPTyxyRS523+kjFeWbueVb7ez/2g5w5La89BlA7igf7zdLxHirGmoCaqqUs59\n7Gt6x7fm5dtGuh2OaUCVVcoX67133GbvOkBs6xhuHZ3MTaOSadfSneaWvP1HeX7xNt5esYtj5ZWc\n368zs8b35Mzkjq7EYwLDmoaCzIrt+/AcOMb9k/q6HYppYJERwqRBXbhoYDzLtu1jdkYOf/l8M88s\nzOG6kUlMPyeFbu0bp8z4hoJDPJeRw4erCxBgSloCM8am0reL1VQKN5YImqD0bA8tYyK5cEC826GY\nABERRqV2YlRqJzYUHGLOolxe/tbbLDPFKXLXJ77hv5BV9YcEtHBTES1jIrl1dHKjJiDT9JwyEYjI\nPcDrqrq/EeIJe8fLK/lodQGTBnahZYzl6XDQv2tbHr82jV9P7PNDE8373+c1aBONt0mqkGczclm1\n6wCdWsXwm4l9uHFUD9q3tCJw4c6fb5p4YIWIfA+8CCzQYOtYCCILN+2h5HiFlZQIQ4kdWvLw5IHc\ne35vXl26g5e/3cbVs5cyvEcHZo5NrVenbWlFJf/83sPcRbnkFh8hqWNL/mvqoCbVSW3c51dnsXgv\nfJ4I3AaMAN4BXlDVnMCG91Oh3lk887VMvt95gKUPnGfFusJc9cs4e3VuzYyxqUxNSzhlAcJDx8t/\nuGy1qKSUgd3aMmtcTy4e1MX+X4Wp0+4sVlUVkUKgEKgAOgDvicgXqnp/LQeeBDwBRALPq+qfatjm\nGuBhQIFVqnq9PzGFogNHy/h6YxE3nd3D/lgNLWIiuWV0MjeclfTDjV33v7eav32+mennpDBtZPef\n3Ni1+9BxXlyyjTeW7eSwcyPb49ekBdWNbKbx+dNHcC9wM1AMPA/8VlXLRSQC2ALUmAhEJBJ4GrgQ\nyMPbvDRfVdf7bNMbeBAYo6r7RaTz6b6gYPbJmkLKKqu4wpqFjI+oyAimpCUweUi3H0o9PPrJBp78\nags3jerBbWNSOHS8nDkZuczL8lBRVRUSpS1M4/HnjKAj8DNV3eG7UFWrROSyWvYbCWxV1VwAEXkL\nmAKs99nmTuDpEx3RqrqnLsGHmvQsD706t2ZgNxvD1fyUiDCuTxzj+sSxatcBnluUw7MZOTy/eBvl\nVVXEREZwzZneInA9OlkROOM/fxLBp8C+EzMi0hbor6rLVHVDLfslALt85vOAs6pt08d5zm/wNh89\nrKqfVX8iEZkBzABISkryI+Tgs2vfUZZv38dvL+prp/DmlIZ0b88zNwxnW/ERXlu6g9bNo7j57B4h\nX/7aBIY/ieBZYJjP/OEalp3O8XsD44FEYJGIDFbVA74bqeocYA54O4sb4LhNzvxV+QBMHmIlJYz/\nUmJb8ZDVozKnyZ8eSfG9XFRVq/AvgXiA7j7zic4yX3nAfFUtV9VtwGa8iSGsqCr//D6Pkckd6d4x\ntOrXG2OaPn8SQa6I/EJEop3HvUCuH/utAHqLSIqIxADTgPnVtknHezaAiMTibSry57lDyrr8Q+QU\nHbF7B4wxrvAnEcwCRuP9NX+inX/GqXZS1QrgbmABsAF4R1XXicgjIjLZ2WwBsFdE1gNf470iaW/d\nX0Zwm5flISYygksHd3U7FGNMGDplE49zJc+0+jy5qn4CfFJt2UM+0wr8ynmEpYrKKuavymdCvzjX\nqk8aY8KbP/cRNAemAwOBH8alU9XbAxhX2Pg2Zy9FJaV274AxxjX+NA29BnQBLgIy8Hb6lgQyqHCS\nnuWhbfMoxvcN63vpjDEu8vx8ipYAABAnSURBVCcR9FLV/wCOqOorwKX89H4AUw9Hyyr4bF0hl57R\n1QqAGWNc408iKHf+PSAig4B2gP18bQBfrN/N0bJKpqZZs5Axxj3+3A8wR0Q6AH/Ae/lna+A/AhpV\nmJiX5SGhfQsbEtAY46paE4FTWO6QUwtoEZDaKFGFgaKSUhZvKWbm2FQbGNwY46pam4acu4hPWmba\n1N9Hq/OprFK7WsgY4zp/+gj+JSK/EZHuItLxxCPgkYW49CwPA7u1pXcAxqU1xpi68KeP4Frn37t8\nlinWTFRvOUWHWZV3kD9c2t/tUIwxxq87i1MaI5Bw8kGWhwiBy63SqDGmCfDnzuKba1quqq82fDih\nT1WZl+1hTK9Y4ts2P/UOxhgTYP40DZ3pM90cOB/4HrBEUA/f79zPrn3HuO/8Pm6HYowxgH9NQ/f4\nzotIe+CtgEUU4uZleWgeHcFFg7q4HYoxxgD+XTVU3RHA+g3qoayiio9WFzBxQBdaN/PnZMwYYwLP\nnz6CD/FeJQTexDEAeCeQQYWqjM1FHDhabvcOGGOaFH9+lv7FZ7oC2KGqeQGKJ6SlZ3no1CqGc3rH\nuh2KMcb8wJ9EsBMoUNXjACLSQkSSVXV7QCMLMYeOl/PFht1cPzKJ6Mj6tMgZY0xg+PON9C5Q5TNf\n6SwzdfDZmkLKKqpsXGJjTJPjTyKIUtWyEzPOdEzgQgpN87I8pMS2YkhiO7dDMcaYH/EnERT5DDaP\niEwBigMXUujJP3CM77btZUpaN0Ss0qgxpmnxp49gFvCGiDzlzOcBNd5tbGo2f1U+qtgANMaYJsmf\nG8pygFEi0tqZPxzwqEJMepaHoUntSY5t5XYoxhjzE6dsGhKR/xaR9qp6WFUPi0gHEfljYwQXCjYU\nHGJjYYndO2CMabL86SO4WFUPnJhxRiu7JHAhhZb0bA9REcKlg7u6HYoxxtTIn0QQKSLNTsyISAug\nWS3bG0dVlfJBVj7j+sTRqbW9ZcaYpsmfRPAG8KWITBeRO4AvgFcCG1Zo+G7bXgoPHbd7B4wxTZo/\nncV/FpFVwAV4aw4tAHoEOrBQkJ7loXWzKC7oH+92KMYYc1L+1jrYjTcJXA2cB2wIWEQh4nh5JZ+u\nKWTSoC60iIl0OxxjjDmpk54RiEgf4DrnUQy8DYiqTmik2ILalxv2UFJaYVcLGWOavNqahjYCi4HL\nVHUrgIj8slGiCgHzsjzEt23GqNRObodijDG1qq1p6GdAAfC1iMwVkfMBq4/gh31Hyli4aQ9T0hKI\njLC3zBjTtJ00EahquqpOA/oBXwP3AZ1F5FkRmdhYAQajj9cUUFGlVlLCGBMUTtlZrKpHVPVNVb0c\nSASygN8FPLIglp7loW98G/p3beN2KMYYc0p1GiFFVfer6hxVPd+f7UVkkohsEpGtIvJADetvFZEi\nEcl2HnfUJZ6maOfeo6zcsZ+pQxOs0qgxJigEbAR1EYkEngYuxFuxdIWIzFfV9dU2fVtV7w5UHI0t\nPdsDwJS0bi5HYowx/gnkmIkjga2qmusMZvMWMCWAx3OdqpKe5WFUake6tW/hdjjGGOOXQCaCBGCX\nz3yes6y6K0VktYi8JyLda3oiEZkhIpkikllUVBSIWBvE6ryD5BYfsXsHjDFBxe1R1D8EklX1DGqp\nYeT0S4xQ1RFxcXGNGmBdzMvyEBMVwaRBVmnUGBM8ApkIPIDvL/xEZ9kPVHWvqpY6s88DwwMYT0CV\nV1bx4ap8LujfmXYtot0Oxxhj/BbIRLAC6C0iKSISA0wD5vtuICK+P50nE8Q1jJZsLWbvkTK7d8AY\nE3QCdtWQqlaIyN14q5VGAi+q6joReQTIVNX5wC9EZDJQAewDbg1UPIGWnuWhfctoxvft7HYoxhhT\nJwFLBACq+gnwSbVlD/lMPwg8GMgYGsPh0goWrCvkymGJxES53e1ijDF1Y99aDeDzdYUcL6+yq4WM\nMUHJEkEDmJflIbFDC4b36OB2KMYYU2eWCE7TnkPH+WZrMVdYSQljTJCyRHCa5q/Kp0phil0tZIwJ\nUpYITlN6toczEtvRq3Nrt0Mxxph6sURwGrbsLmGt55DdO2CMCWqWCE5DeraHyAjh8iFWadQYE7ws\nEdRTVZWSnpXPOb1iiWvTzO1wjDGm3iwR1FPmjv14DhyzeweMMUHPEkE9zcvy0DImkokD490OxRhj\nToslgnoorajk49X5XDSwCy1jAlqlwxhjAs4SQT18vbGIQ8crmGrNQsaYEGCJoB7SszzEto5hTM9O\nbodijDGnzRJBHR08Ws5XG/dw+ZBuREXa22eMCX72TVZHn6wtoKzSKo0aY0KHJYI6mpflITWuFYMT\n2rkdijHGNAhLBHWQt/8oy7ft44o0qzRqjAkdlgjq4IPsfMAqjRpjQoslAj+pKvOyPIzo0YGkTi3d\nDscYYxqMJQI/rcs/xNY9h+3eAWNMyLFE4Kf0LA/RkcKlg7u6HYoxxjQoSwR+qKxSPliVz/i+nenQ\nKsbtcIwxpkFZIvDDtznFFJWU2r0DxpiQZInAD/OyPLRpFsV5/Tq7HYoxxjQ4SwSncLSsggVrC7lk\ncFeaR0e6HY4xxjQ4SwSn8MX63Rwpq7SrhYwxIcsSwSmkZ3no2q45Z6V0dDsUY4wJCEsEtSg+XMqi\nLcVMSUsgIsJKShhjQpMlglp8tCqfyiq1q4WMMSHNEkEt0rPz6d+1LX27tHE7FGOMCRhLBCexrfgI\n2bsOcMXQbm6HYowxAWWJ4CTSszyIwOQh1ixkjAltlghqoKqkZ3sY3bMTXdo1dzscY4wJKEsENcja\ndYAde48y1cYdMMaEgYAmAhGZJCKbRGSriDxQy3ZXioiKyIhAxuOv9CwPzaIimDSoi9uhGGNMwAUs\nEYhIJPA0cDEwALhORAbUsF0b4F5gWaBiqYvyyio+XJXPhQPiadM82u1wjDEm4AJ5RjAS2Kqquapa\nBrwFTKlhu/8C/gwcD2Asflu0uYj9R8vt3gFjTNgIZCJIAHb5zOc5y34gIsOA7qr6cW1PJCIzRCRT\nRDKLiooaPlIf87I8dGgZzdg+cQE9jjHGNBWudRaLSATwN+DXp9pWVeeo6ghVHREXF7gv6JLj5Xyx\nfjeXD+lGdKT1oxtjwkMgv+08QHef+URn2QltgEHAQhHZDowC5rvZYfzZ2kJKK6qs0qgxJqwEMhGs\nAHqLSIqIxADTgPknVqrqQVWNVdVkVU0GvgMmq2pmAGOqVXq2hx6dWjK0e3u3QjDGmEYXsESgqhXA\n3cACYAPwjqquE5FHRGRyoI5bX4UHj/Ntzl6mpiUgYpVGjTHhIyqQT66qnwCfVFv20Em2HR/IWE5l\n/ioPqlizkDEm7FiPqGNeVj5p3duTEtvK7VCMMaZRWSIANhYeYkPBIbt3wBgTliwRAOlZ+URGCJed\n0dXtUIwxptGFfSKoqlI+yPYwrk8cnVo3czscY4xpdGGfCJZt20fBwePWSWyMCVthnwjSszy0ionk\nwv7xbodijDGuCOtEcLy8kk/WFHDRoC60iIl0OxxjjHFFWCeCrzbuoaS0wq4WMsaEtbBOBPOyPMS1\nacbonrFuh2KMMa4J20Sw/0gZCzftYcqQbkRGWEkJY0z4CttE8PGaAsor1a4WMsaEvbBNBOlZHnp3\nbs3Abm3dDsUYY1wVlolg596jZO7Yz9ShVmnUGGPCMhF8kO0dH2dKWjeXIzHGGPeFXSJQVeZlexiZ\n0pHEDi3dDscYY1wXdolgjecguUVH7N4BY4xxhF0imJflISYygksGWaVRY4yBMEsEFZVVfLgqn/P6\ndaZdy2i3wzHGmCYhrBLBkq3FFB8us3sHjDHGR1glgvQsD22bRzGhX5zboRhjTJMRNongSGkFC9bt\n5tIzutEsyiqNGmPMCWGTCD5fX8ix8kq7WsgYY6oJm0TQulk0Fw6IZ0SPDm6HYowxTUqU2wE0lgsH\nxHPhABuFzBhjqgubMwJjjDE1s0RgjDFhzhKBMcaEOUsExhgT5iwRGGNMmLNEYIwxYc4SgTHGhDlL\nBMYYE+ZEVd2OoU5EpAjYUc/dY4HiBgwnGNhrDg/2msPD6bzmHqpaY8XNoEsEp0NEMlV1hNtxNCZ7\nzeHBXnN4CNRrtqYhY4wJc5YIjDEmzIVbIpjjdgAusNccHuw1h4eAvOaw6iMwxhjzU+F2RmCMMaYa\nSwTGGBPmwiYRiMgkEdkkIltF5AG34wk0EXlRRPaIyFq3Y2ksItJdRL4WkfUisk5E7nU7pkATkeYi\nslxEVjmv+T/djqkxiEikiGSJyEdux9IYRGS7iKwRkWwRyWzw5w+HPgIRiQQ2AxcCecAK4DpVXe9q\nYAEkImOBw8CrqjrI7Xgag4h0Bbqq6vci0gZYCUwN8c9ZgFaqelhEooElwL2q+p3LoQWUiPwKGAG0\nVdXL3I4n0ERkOzBCVQNyA124nBGMBLaqaq6qlgFvAVNcjimgVHURsM/tOBqTqhao6vfOdAmwAUhw\nN6rAUq/Dzmy08wjpX3cikghcCjzvdiyhIlwSQQKwy2c+jxD/ggh3IpIMDAWWuRtJ4DnNJNnAHuAL\nVQ311/x/wP1AlduBNCIFPheRlSIyo6GfPFwSgQkjItIaeB+4T1UPuR1PoKlqpaqmAYnASBEJ2aZA\nEbkM2KOqK92OpZGdo6rDgIuBu5ym3wYTLonAA3T3mU90lpkQ47STvw+8oar/dDuexqSqB4CvgUlu\nxxJAY4DJTpv5W8B5IvK6uyEFnqp6nH/3APPwNnc3mHBJBCuA3iKSIiIxwDRgvssxmQbmdJy+AGxQ\n1b+5HU9jEJE4EWnvTLfAe0HERnejChxVfVBVE1U1Ge/f8VeqeqPLYQWUiLRyLn5ARFoBE4EGvRow\nLBKBqlYAdwML8HYgvqOq69yNKrBE5B/AUqCviOSJyHS3Y2oEY4Cb8P5KzHYel7gdVIB1Bb4WkdV4\nf/B8oaphcUllGIkHlojIKmA58LGqftaQBwiLy0eNMcacXFicERhjjDk5SwTGGBPmLBEYY0yYs0Rg\njDFhzhKBMcaEOUsExlQjIpU+l59mN2S1WhFJDqeKsCY4RLkdgDFN0DGnZIMxYcHOCIzxk1MT/jGn\nLvxyEenlLE8Wka9EZLWIfCkiSc7yeBGZ54wVsEpERjtPFSkic53xAz537gg2xjWWCIz5qRbVmoau\n9Vl3UFUHA0/hrYIJ8HfgFVU9A3gDeNJZ/iSQoapDgGHAibvZewNPq+pA4ABwZYBfjzG1sjuLjalG\nRA6rausalm8HzlPVXKe4XaGqdhKRYrwD4pQ7ywtUNVZEioBEVS31eY5kvGUgejvzvwOiVfWPgX9l\nxtTMzgiMqRs9yXRdlPpMV2J9dcZllgiMqZtrff5d6kx/i7cSJsANwGJn+kvg5/DD4DHtGitIY+rC\nfokY81MtnBG/TvhMVU9cQtrBqfRZClznLLsHeElEfgsUAbc5y+8F5jiVXyvxJoWCgEdvTB1ZH4Ex\nfgr0AOLGuMWahowxJszZGYExxoQ5OyMwxpgwZ4nAGGPCnCUCY4wJc5YIjDEmzFkiMMaYMPf/ATGU\nFmS7Z8FuAAAAAElFTkSuQmCC\n", 848 | "text/plain": [ 849 | "
" 850 | ] 851 | }, 852 | "metadata": { 853 | "tags": [] 854 | } 855 | }, 856 | { 857 | "output_type": "display_data", 858 | "data": { 859 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhcd33v8fdX+2qt40XjRbbjmMRx\nLAUlBEghDQRCSGPllrZwCxSa3jz00gIXbm+htw+FlG733lIeoLe9oQTKUlraEDsbWYCEJCxJZEd2\n4jiL19jyIsnyIsm21u/9Y47ksSLbY2uOziyf1/PMo5k5Z2a+M4nnM+f3O+d7zN0REZH8VRB1ASIi\nEi0FgYhInlMQiIjkOQWBiEieUxCIiOQ5BYGISJ5TEEjGMbNmM3MzK0ph3Q+Z2ZOzUdf5MLNyM7vX\nzI6a2b/P0mvuMrO3B9f/xMz+aTZeV7KfgkBmJPjyGTazxin3Pxt8mTdHU9n5BUoI3gPMAxrc/TfS\n8YRmNsfMvmRmr5rZgJltD243Tl3X3f/S3X8vDa95zs/QzC4zs4fMrNfMdGBSFlIQSDrsBN43ccPM\nVgMV0ZWTEZYAL7v76Pk+cLovXTMrAX4MrAJuAOYAbwQOAVfNrNQZGwG+D9wacR1ygRQEkg7fBj6Y\ndPt3gG8lr2BmNWb2LTPrMbPdZvanZlYQLCs0s/8T/KLcAbx7msd+3cz2m1mXmX3BzApnUrCZlQa/\npvcFly+ZWWmwrNHM7jOzI2bWZ2ZPJNX6x0EN/Wb2kpm9bZrn/jzwWeC3gl/ut5pZQfCed5tZd/BZ\n1ATrT/zqvtXMXgV+Mk3JHwQWA7e4+wvuPu7u3e7+5+7+wDQ1fM7MvpN0+2oz+3nwnjaZ2bVJyx4z\nsz83s58F7+vhpK2Mx4O/R4L38sapr+XuL7n714EtqXz2knkUBJIOvwTmmNklwRf0e4HvTFnnK0AN\nsAx4K4kvtg8Hy/4LcBPQCrSRGFZJ9k1gFLgoWOcdwEyHPf4ncDXQAqwh8av6T4NlnwL2AjESwzt/\nAriZrQT+ALjS3auBdwK7pj6xu/8Z8JfAv7l7VfAl+aHg8qskPoMq4KtTHvpW4JLgead6O/Cguw+c\n7xs1szhwP/AFoB7478BdZhZLWu0/k/jvMRcoCdYBeEvwtzZ4L78439eXzKcgkHSZ2Cq4HtgKdE0s\nSAqHz7h7v7vvAv4W+ECwym8CX3L3Pe7eB/xV0mPnATcCn3D3QXfvBv4ueL6Z+G3g9uBXdQ/w+aR6\nRoAFwBJ3H3H3JzzRlGsMKAUuNbNid9/l7tvP4/W+6O47gi/zzwDvnTIM9LngPZ6Y5vENwP7zf5sA\nvB94wN0fCLYkHgE6SHyuE77h7i8Hr/19EgEpeUJBIOnybRK/Kj/ElGEhoBEoBnYn3bcbiAfXm4A9\nU5ZNWBI8dn8wrHEE+H8kfrnORNM09TQF1/83sA142Mx2mNmnAdx9G/AJ4HNAt5n9q5k1kZrpXq+I\nxBbHhD2c2SES4XQhlgC/MfH5BZ/hNVOe70DS9eMktlgkTygIJC3cfTeJSeMbgR9MWdxL4lf2kqT7\nFnNqq2E/sGjKsgl7gCGg0d1rg8scd181w5L3TVPPvuC99Lv7p9x9GXAz8MmJuQB3/xd3vyZ4rAN/\nM4PXGwUOJt13tj1ufgS808wqU3y9ZHuAbyd9frXuXunuf53CY7UXUB5QEEg63Qpc5+6DyXe6+xiJ\n4Ya/MLNqM1sCfJJT8wjfBz5mZgvNrA74dNJj9wMPA38b7D5ZYGbLzeyt51FXqZmVJV0KgO8Bf2pm\nsWBi9LMT9ZjZTWZ2kZkZcJTEkNC4ma00s+uCSeWTwAlgPMUavgf8NzNbamZVnJpDSHWvom+T+EK/\ny8xeF3wODZY4XuDGczz2O8Cvmdk7g4n5MjO71swWpvC6PSTe47IzrWAJZSTmFgievzS1tyWZQEEg\naePu29294wyL/xAYBHYATwL/AtwZLPsa8BCwCdjIa7coPkjiS+YF4DDwH5zfMMkAiS/tict1JCZO\nO4DNwHPB634hWH8FiV/gA8AvgP/r7o+SmB/4axJbOAdIDE99JsUa7iTxZf44iS2nkyQ+k5S4+xCJ\nCeMXgUeAY8DTJIbdnjrHY/cAa0lMeveQCJQ/IoV//+5+HPgL4GfBsNLV06y2hMTnOrHX0AngpXO/\nK8kUphPTiIjkN20RiIjkOQWBiEieUxCIiOQ5BYGISJ6LoivjjDQ2Nnpzc3PUZYiIZJUNGzb0unts\numVZFwTNzc10dJxpD0UREZmOme0+0zINDYmI5DkFgYhInlMQiIjkOQWBiEieUxCIiOQ5BYGISJ5T\nEIiI5Lm8CYJt3f3cfu8LDI+m2j5eRCQ/5E0Q7Ok7wZ0/28lPX+6JuhQRkYySN0FwzYpG6itLWNfZ\nde6VRUTySN4EQXFhAb92+QJ+9MJB+k+ORF2OiEjGyJsgAFjbGmdodJwHnz8QdSkiIhkjr4KgdVEt\nSxoqWN+5L+pSREQyRl4FgZmxtiXOz7f3cvDYyajLERHJCHkVBADtLU2MO9y7SVsFIiKQh0GwLFbF\n5QtrtPeQiEgg74IAoL0lzvNdx9jW3R91KSIikcvLILhpzQIKDNY9q+EhEZG8DIK51WW8+aJG1m/q\nwt2jLkdEJFJ5GQQAt7TG2dN3go2vHo66FBGRSOVtELxj1XzKigu4+1lNGotIfgstCMyszMyeNrNN\nZrbFzD4/zTqlZvZvZrbNzJ4ys+aw6pmqqrSI6y+dz/2b9zMypo6kIpK/wtwiGAKuc/c1QAtwg5ld\nPWWdW4HD7n4R8HfA34RYz2u0tzRx+PgIj6sjqYjksdCCwBMGgpvFwWXqzOxa4J+D6/8BvM3MLKya\npnrLxTHqKoo1PCQieS3UOQIzKzSzTqAbeMTdn5qyShzYA+Duo8BRoGGa57nNzDrMrKOnJ32/3osL\nC7jp8iZ+tPUgA0OjaXteEZFsEmoQuPuYu7cAC4GrzOyyC3yeO9y9zd3bYrFYWmtsb23i5Mg4D6kj\nqYjkqVnZa8jdjwCPAjdMWdQFLAIwsyKgBjg0GzVNuGJxHYvqy9VyQkTyVph7DcXMrDa4Xg5cD7w4\nZbV7gN8Jrr8H+InP8hFeZkZ7S5yfbeulu18dSUUk/4S5RbAAeNTMNgPPkJgjuM/Mbjezm4N1vg40\nmNk24JPAp0Os54zWtsSDjqT7o3h5EZFIFYX1xO6+GWid5v7PJl0/CfxGWDWk6qK5VayO17C+s4tb\nr1kadTkiIrMqb48snmptSxOb9x5le8/AuVcWEckhCoLAzWuaKDBYr2MKRCTPKAgCc+ckOpKu69yn\njqQiklcUBEnWtsR5te84G189EnUpIiKzRkGQ5J2r5lFaVMB6HVMgInlEQZCkuqyY6y+dx33qSCoi\neURBMEV7S5y+wWGeeEUdSUUkPygIpnjLxTFqK4p1PmMRyRsKgilKigq46fIFPPzCAXUkFZG8oCCY\nRntLnJMj4zy8RR1JRST3KQim8foldSysK2ddp4aHRCT3KQimYWasbWniyVd66OkfirocEZFQKQjO\noH2yI6m2CkQktykIzmDFvGpWNc3RwWUikvMUBGfR3hJn096j7FBHUhHJYQqCs7i5pQkzNGksIjlN\nQXAW8+aU8ablDazv7FJHUhHJWQqCc1jbEmf3oeN07lFHUhHJTQqCc7jhsvlBR1IND4lIblIQnMOc\nsmLefsk87t20Tx1JRSQnKQhSsLaliUODwzy5rTfqUkRE0k5BkIJrV86ltqJY5zMWkZykIEhBSVEB\nN65ewENbDjKojqQikmMUBClqb4lzYmSMR144GHUpIiJppSBIUduSOuK15axTywkRyTEKghQVFCQ6\nkj7xSi+9A+pIKiK5Q0FwHtpb44yNO/epI6mI5BAFwXm4eF41ly6Yo95DIpJTQgsCM1tkZo+a2Qtm\ntsXMPj7NOtea2VEz6wwunw2rnnRpb22ic88RdvYORl2KiEhahLlFMAp8yt0vBa4GPmpml06z3hPu\n3hJcbg+xnrS4eU0cM3SeAhHJGaEFgbvvd/eNwfV+YCsQD+v1Zsv8mjKuXtrA+s596kgqIjlhVuYI\nzKwZaAWemmbxG81sk5n90MxWzUY9M3VLa5ydvYNs2ns06lJERGYs9CAwsyrgLuAT7n5syuKNwBJ3\nXwN8BVh3hue4zcw6zKyjp6cn3IJTcMPq+ZQUFbBOLSdEJAeEGgRmVkwiBL7r7j+Yutzdj7n7QHD9\nAaDYzBqnWe8Od29z97ZYLBZmySmZU1bM2143l/s272NUHUlFJMuFudeQAV8Htrr7F8+wzvxgPczs\nqqCeQ2HVlE7trXF6B9SRVESyX1GIz/1m4APAc2bWGdz3J8BiAHf/R+A9wO+b2ShwAnivZ8kM7LUr\nY8wpK2J95z6uXTk36nJERC5YaEHg7k8Cdo51vgp8NawawlRaVMi7L1/A+s59HB8epaIkzEwVEQmP\njiyegfaWOMeH1ZFURLKbgmAGrmyup6mmTHsPiUhWUxDMQEGBcXNLnMdf6eWQOpKKSJZSEMzQLUFH\n0vuf2x91KSIiF0RBMEMr51fzuvnV3K3hIRHJUgqCNGhvjfPsq0fYfUgdSUUk+ygI0uDmNU1BR1Kd\np0BEso+CIA2aast5w9J61j3bpY6kIpJ1FARp0t4SZ0fvIM91qSOpiGQXBUGavGv1AkoKC1j3rIaH\nRCS7KAjSpKa8mOteN5d7NqkjqYhkFwVBGrW3NtE7MMTPt2dFA1UREUBBkFbXrpxLdVkR63Q+YxHJ\nIgqCNCorLuTdqxfw0PMHODE8FnU5IiIpURCk2dqWOIPDYzyyVR1JRSQ7KAjS7A1L61lQU8Z6tZwQ\nkSyhIEizREfSJn76cg99g8NRlyMick4KghC0t8QZHXfu36xjCkQk8ykIQnDJgjmsnFfNOvUeEpEs\noCAISXtrnA27D/PqoeNRlyIiclYKgpDc3NIEwHodUyAiGU5BEJJ4bTlXLa1nXac6kopIZlMQhOiW\n1jjbewZ5vutY1KWIiJyRgiBEN14WdCTV8JCIZDAFQYhqKoq5dmWMezftY2xcw0MikpkUBCG7pTVO\nd/8Qv1BHUhHJUAqCkP3q6xIdSe9WywkRyVAKgpCVFRfyrsvm89CWA5wcUUdSEck8CoJZ0N4SZ2Bo\nlB+pI6mIZKDQgsDMFpnZo2b2gpltMbOPT7OOmdmXzWybmW02syvCqidKb1jWwPw5ZazT8JCIZKAw\ntwhGgU+5+6XA1cBHzezSKeu8C1gRXG4D/iHEeiJTGHQkfeylHg6rI6mIZJjQgsDd97v7xuB6P7AV\niE9ZbS3wLU/4JVBrZgvCqilKa1uaEh1Jn9sfdSkiIqdJKQjMbLmZlQbXrzWzj5lZbaovYmbNQCvw\n1JRFcWBP0u29vDYsMLPbzKzDzDp6enpSfdmMcumCOVw8r0rDQyKScVLdIrgLGDOzi4A7gEXAv6Ty\nQDOrCh7/CXe/oF4L7n6Hu7e5e1ssFruQp4icmbG2JU7H7sPs6VNHUhHJHKkGwbi7jwK3AF9x9z8C\nzjmEY2bFJELgu+7+g2lW6SIRKhMWBvflpLVBR9J7Nuk8BSKSOVINghEzex/wO8B9wX3FZ3uAmRnw\ndWCru3/xDKvdA3ww2HvoauCou+fsIPrCugquaq7n7mfVkVREMkeqQfBh4I3AX7j7TjNbCnz7HI95\nM/AB4Doz6wwuN5rZR8zsI8E6DwA7gG3A14D/ev5vIbusbW1iW/cAW/apI6mIZIaiVFZy9xeAjwGY\nWR1Q7e5/c47HPAnYOdZx4KOplZob3r16AZ+7ZwvrO7u4LF4TdTkiIinvNfSYmc0xs3pgI/A1MzvT\ncI+cRW1FCdeunMv6TnUkFZHMkOrQUE2wx89/IrHf/xuAt4dXVm5rb0l0JP3lDnUkFZHopRoERcGB\nXr/JqcliuUBvu2QuVaVFOqZARDJCqkFwO/AQsN3dnzGzZcAr4ZWV2yY6kv7weXUkFZHopRQE7v7v\n7n65u/9+cHuHu/96uKXltvbWREfSH2/tjroUEclzqU4WLzSzu82sO7jcZWYLwy4ul129rIG51aU6\nn7GIRC7VoaFvkDj4qym43BvcJxeosMC4eU0Tj73UzZHj6kgqItFJNQhi7v4Ndx8NLt8EsrPpTwZp\nb40zMqaOpCISrVSD4JCZvd/MCoPL+wHt+zhDq5rmcNHcKtY/q95DIhKdVIPgd0nsOnoA2A+8B/hQ\nSDXlDTOjvaWJp3f1sfewOpKKSDRS3Wtot7vf7O4xd5/r7u2A9hpKg7UtidMvrO/UVoGIRGMmZyj7\nZNqqyGOL6itoW1LH+k51JBWRaMwkCM7aUE5St7Y1zssHB9i6vz/qUkQkD80kCPTzNU1uWr2AogLT\nMQUiEomzBoGZ9ZvZsWku/SSOJ5A0qKss4dqVMe5RR1IRicBZg8Ddq919zjSXandP6VwGkpq1LXEO\nHDvJUzu1V66IzK6ZDA1JGr39knnqSCoikVAQZIjykkLeuWo+P3xOHUlFZHYpCDJIe2sT/UOjPPqi\nOpKKyOxREGSQNy1vJFZdyt0aHhKRWaQgyCCnOpL2cPT4SNTliEieUBBkmPaWOMNj4zzwvDqSisjs\nUBBkmMvic1geq9TwkIjMGgVBhkl0JI3z9M4+uo6ciLocEckDCoIMNNGR9B51JBWRWaAgyECLGyq4\nYnGtDi4TkVmhIMhQt7TGeelgP1v3H4u6FBHJcQqCDPXuy5vUkVREZkVoQWBmd5pZt5k9f4bl15rZ\nUTPrDC6fDauWbFRfWcJbLk50JB1XR1IRCVGYWwTfBG44xzpPuHtLcLk9xFqyUntrnP1HT/LUzr6o\nSxGRHBZaELj744C+wWbg+kvmUVlSyHoND4lIiKKeI3ijmW0ysx+a2aqIa8k4Ex1J739uvzqSikho\nogyCjcASd18DfAVYd6YVzew2M+sws46enp5ZKzATtLfG6T85ymMvqSOpiIQjsiBw92PuPhBcfwAo\nNrPGM6x7h7u3uXtbLBab1Tqj9qblDTRWlbLuWR1cJiLhiCwIzGy+mVlw/aqgFp2ncYqiwgJ+bc0C\nfvJiN0dPqCOpiKRfmLuPfg/4BbDSzPaa2a1m9hEz+0iwynuA581sE/Bl4L3urv0kp3FLa6Ij6Q+f\nU0dSEUm/0E5A7+7vO8fyrwJfDev1c8nqeA3LGitZ19nFe69aHHU5IpJjot5rSFJgZqxtifPUzj72\nqSOpiKSZgiBLtLc24Q73bNKksYikl4IgSyxpqKRVHUlFJAQKgizS3hLnxQP9vHhAHUlFJH0UBFnk\npssXUFhgOqZARNJKQZBFGqpKecuKRu7p7FJHUhFJGwVBlmlvjbPv6Eme2aV+fiKSHgqCLHP9pfOo\nKCnUCWtEJG0UBFmmoqQo0ZF0836GRtWRVERmTkGQhda2NHHs5CiPvZRfnVhFJBwKgix0zUWNNFaV\n6JgCEUkLBUEWKios4KbLm/jxi90cO6mOpCIyMwqCLNXeGmd4dJwHnzsQdSkikuUUBFlqzcIamhsq\nuFvDQyIyQwqCLGVmtLfG+eXOQxw4ejLqckQkiykIslh7SzzoSKqtAhG5cAqCLNbcWMmaRbXqPSQi\nM6IgyHK3tDTxwv5jvHywP+pSRCRLKQiy3E1rmoKOpBoeEpELoyDIco1VpVxzUSPrO/epI6mIXBAF\nQQ64pTVO15ETdOw+HHUpIpKFFAQ54PpL51FerI6kInJhFAQ5oLK0iHesmsf9m/czPDoedTkikmUU\nBDmivSXO0RMjPPZSd9SliEiWURDkiGtWNNJQWcL6Th1TICLnR0GQI4oLC7jp8gX8aOtBdSQVkfOi\nIMgha1vjDI2O8+Dz6kgqIqlTEOSQ1kW1LGmoYL32HhKR86AgyCFmxtqWOD/ffoiDx9SRVERSE1oQ\nmNmdZtZtZs+fYbmZ2ZfNbJuZbTazK8KqJZ+0tzQlOpJq0lhEUhTmFsE3gRvOsvxdwIrgchvwDyHW\nkjeWxapYs7BGB5eJSMpCCwJ3fxzoO8sqa4FvecIvgVozWxBWPflkbUucLfuO8Yo6kopICqKcI4gD\ne5Ju7w3uew0zu83MOsyso6enZ1aKy2Y3rVlAgaGtAhFJSVZMFrv7He7e5u5tsVgs6nIy3tzqMq5Z\nEWN95z7c1ZFURM4uyiDoAhYl3V4Y3Cdp0N7SxN7DJ9igjqQicg5RBsE9wAeDvYeuBo66+/4I68kp\n71g1n7LiAg0Picg5FYX1xGb2PeBaoNHM9gJ/BhQDuPs/Ag8ANwLbgOPAh8OqJR9VlRbxjkvnc9eG\nLnr7h1kWq2RZrIplsUqWN1ZRU1EcdYkikiFCCwJ3f985ljvw0bBeX+CT11/MyZExXj7YzyNbDzKW\ndAazxqoSljVWBQFRyfJYFctiVSyqK6eoMCumjkQkTUILAolec2Mld3ywDYCRsXFe7TvOjp5BtvcM\nsKNngB09gzz8wkH6BocnH1NcaCyur2BZrCoIh0qWxypZ1lhFXWVJVG9FREKkIMgTxYUFLA++3K9n\n3mnLDg8Os6N3gO09g+zoGUyERO8gj73UzcjYqa2I+soSljVWnhpmaqxk+dwqFtdXUKytiEiMjzt9\nx4cpKy6kqlT/nOXC6P8coa6yhNdX1vP6JfWn3T86Ns6ewycmtx529A6wvXuQn7zYzfc79k6uV1Qw\nsRVRGWxJnAqK+soSzGy231JWc3eOHB+hd2CInv4heoK/vQPDwd9T9/cNDjM27hQYvG7+HK5srqOt\nuZ4rm+uZX1MW9VuRLGHZtp95W1ubd3R0RF1G3jt6YmQyILYnBcWu3uMMj506XWZNeXHSHERiiGl5\nrJIlDZWUFOXPVoS70z80mvgCT/oy753mS753YOi0LbEJxYVGrKqUWHUpjUl/G6tKOHx8hI7dfWzc\nfYQTI2MALKov58ol9UEw1LE8VkVBgUI5X5nZBndvm3aZgkDSaWzc6Tp8gu09A4mA6B2cDIzu/qHJ\n9QoLjEV15ZNbDslbEo1V2bMVMRh8uU/9Yk/8ih+mZ2CI3uD2dOeTLiwwGqtKTvtij1WXEqsqpTH4\nG6suIVZVxpzyonN+LiNj42zdf4xndh2mY1cfz+zqo3cgMQdUW1FM25K6yWC4LF5DaVFhKJ+LZB4F\ngWSE/pMjk1sOO4L5iO09A+zsHWQo6UuyuqwoEQzBHMREUCxpqKCsOPwvrpMjY1OGZKZ8ySf9gp/4\n9Z3MDBoqS077Uj/tSz7pem15cai/0t2d3YeO80wQCh27DrOjdxCA0qIC1iyqnRxOumJxHTXl2q04\nVykIJKONjztdR06ctvUwMdx0IOm8CgUG8bryxDBT0q6vF8WqiFWXnvXX8tDoGL0Dw4lf51O/3AeG\n6A1+vff0DzEwNDrtc9RVFJ/xV/upvyXUV5Rk9C64vQNDdExsMew+zJauo4yOO2awcl41VzbX09Zc\nx5XN9TTVlkddrqSJgkCy1uDQKDt7B4OhplNBsbN38LRf41WlRcEcRCW1FSWTY+0Tv96Pnpj+PM5z\nyopeM+ae/Ct+4r6GqpKc3TPq+PAonXuO0LHrMM/s6mPj7sMMDic+23ht+WkT0Cvmap4hWykIJOeM\njzsHjp08NVE9OR8xyNETIzRWlZz913t1KQ2VJbMy1JRtRsfGefFA/+RQ0tO7+ugJ5nfmlBXRlrTF\nsDpeo88wSygIROSCuTt7+k4kgmF3H8/sOsy27gEASgoLuHxhDVcuTUxAv35xvdqXZCgFgYikVd/g\nMBt2H56chH6+6+jkLq8r51VPbjG0NdcRry3Pmr3AcpmCQERCdWJ4jE17jwS7rB5m4+7D9AeT7gtq\nymhrrueqYK7h4nnVFGqeYdadLQh0ZLGIzFh5SSFXL2vg6mUNQOJ4kpcO9E8OJT2zs497N+0DErsH\nv35JsMWwpI41i2o1zxAxbRGISOjcnb2HT0wGQ8euPl4+mJhnKC40VsdrgqGkRDiowWH6aWhIRDLO\nkeMT8wyJYNi89+hke5IVc6smj4C+srmehXWaZ5gpBYGIZLyTI2Ns3ns02G21j47dh+k/mZhnmDen\nNBEMQYuMSxbM0TzDedIcgYhkvLLiQq5aWs9VSxNdcMfHnZe7+ye3GDp2Heb+zYmz2VaVFnHJgmqW\nNlaytLGKpUF79MX1s9OGJNdoi0BEskbXkROTofDSwX529g5OHuwGiT5P8dryICBOXZY1VhGvK8/r\nrQhtEYhITojXlhNvibO2JT55X//JEXb1HmdHb6KB4c7eQXb1DnL3xq7JXVghcfDb4oaKIBgqaZ4M\nicpz9qrKdQoCEclq1WXFrF5Yw+qFNafd7+4cGhxOhEPPIDt6B9kZhMVPX+45rS14ZUkhS2OnhpmW\nNlZMXs+HjqwKAhHJSWYWnLinlCubTz/73ti4s//oicktiIlGhpv2HOH+zfsYTxoxb6gsOTXMFDu1\nNdHcUJkz8xEKAhHJO4UFxsK6ChbWVfArK2KnLRsaHWNP33F29h6f3ILY0ZPYivj3DadO0WoGTTVT\n5iOCoIjXlmd0K/KpFAQiIklKiwq5aG41F82tBuadtmxgaJRdvYlhpl0TWxO9g6zr7Jrc1RUSB8kt\nqq9g2WRInNqzaW4GzkcoCEREUlRVWsRl8Roui792PqIvmI/YkTRhvbN3kCde6T3tDHwVJYU0N5za\nekjesymqzq0KAhGRGTIzGqpKaahKHPiWbHzc2X/sJDt7EpPVE0GxpesoDz5/gLGkCYn6yhKaGxIT\n1ctip0KiuaGS8pLw5iMUBCIiISoosMRur7XlXLOi8bRlw6Pj7Dl8PAiJU3s2Pbmth7s27j1t3aaa\nMn73mqX83q8sS3uNCgIRkYiUFBWwPFbF8ljVa5YNDo2y69Dg5O6vO3sHiVWXhlKHgkBEJANVlhax\nqqmGVU015155hrJn/yYREQlFqEFgZjeY2Utmts3MPj3N8g+ZWY+ZdQaX3wuzHhERea3QhobMrBD4\ne+B6YC/wjJnd4+4vTFn139z9D8KqQ0REzi7MLYKrgG3uvsPdh4F/BdaG+HoiInIBwgyCOLAn6fbe\n4L6pft3MNpvZf5jZoumeyMxuM7MOM+vo6ekJo1YRkbwV9WTxvUCzu18OPAL883Qrufsd7t7m7m2x\nWGy6VURE5AKFGQRdQPIv/J0aeDgAAASKSURBVIXBfZPc/ZC7T5xV4p+A14dYj4iITCPMIHgGWGFm\nS82sBHgvcE/yCma2IOnmzcDWEOsREZFphLbXkLuPmtkfAA8BhcCd7r7FzG4HOtz9HuBjZnYzMAr0\nAR861/Nu2LCh18x2X2BZjUDvBT42W+k95we95/wwk/e85EwLsu6cxTNhZh1nOmdnrtJ7zg96z/kh\nrPcc9WSxiIhETEEgIpLn8i0I7oi6gAjoPecHvef8EMp7zqs5AhERea182yIQEZEpFAQiInkub4Lg\nXC2xc42Z3Wlm3Wb2fNS1zBYzW2Rmj5rZC2a2xcw+HnVNYTOzMjN72sw2Be/581HXNBvMrNDMnjWz\n+6KuZTaY2S4zey5o19+R9ufPhzmCoCX2yyS1xAbeN01L7JxhZm8BBoBvuftlUdczG4Ij1Re4+0Yz\nqwY2AO05/t/ZgEp3HzCzYuBJ4OPu/suISwuVmX0SaAPmuPtNUdcTNjPbBbS5eygH0OXLFkHetcR2\n98dJHK2dN9x9v7tvDK73k2hZMl3H25zhCQPBzeLgktO/7sxsIfBuEv3JJA3yJQhSbYktOcLMmoFW\n4KloKwlfMEzSCXQDj7h7rr/nLwH/AxiPupBZ5MDDZrbBzG5L95PnSxBIHjGzKuAu4BPufizqesLm\n7mPu3kKiw+9VZpazQ4FmdhPQ7e4boq5lll3j7lcA7wI+Ggz9pk2+BME5W2JLbgjGye8CvuvuP4i6\nntnk7keAR4Eboq4lRG8Gbg7GzP8VuM7MvhNtSeFz967gbzdwN4nh7rTJlyA4Z0tsyX7BxOnXga3u\n/sWo65kNZhYzs9rgejmJHSJejLaq8Lj7Z9x9obs3k/h3/BN3f3/EZYXKzCqDnR8ws0rgHUBa9wbM\niyBw91FgoiX2VuD77r4l2qrCZWbfA34BrDSzvWZ2a9Q1zYI3Ax8g8SuxM7jcGHVRIVsAPGpmm0n8\n4HnE3fNil8o8Mg940sw2AU8D97v7g+l8gbzYfVRERM4sL7YIRETkzBQEIiJ5TkEgIpLnFAQiInlO\nQSAikucUBCJTmNlY0u6nnensVmtmzfnUEVayQ1HUBYhkoBNBywaRvKAtApEUBT3h/1fQF/5pM7so\nuL/ZzH5iZpvN7Mdmtji4f56Z3R2cK2CTmb0peKpCM/tacP6Ah4MjgkUioyAQea3yKUNDv5W07Ki7\nrwa+SqILJsBXgH9298uB7wJfDu7/MvBTd18DXAFMHM2+Avh7d18FHAF+PeT3I3JWOrJYZAozG3D3\nqmnu3wVc5+47guZ2B9y9wcx6SZwQZyS4f7+7N5pZD7DQ3YeSnqOZRBuIFcHtPwaK3f0L4b8zkelp\ni0Dk/PgZrp+PoaTrY2iuTiKmIBA5P7+V9PcXwfWfk+iECfDbwBPB9R8Dvw+TJ4+pma0iRc6HfomI\nvFZ5cMavCQ+6+8QupHVBp88h4H3BfX8IfMPM/gjoAT4c3P9x4I6g8+sYiVDYH3r1IudJcwQiKQr7\nBOIiUdHQkIhIntMWgYhIntMWgYhInlMQiIjkOQWBiEieUxCIiOQ5BYGISJ77/witcPN/sJeLAAAA\nAElFTkSuQmCC\n", 860 | "text/plain": [ 861 | "
" 862 | ] 863 | }, 864 | "metadata": { 865 | "tags": [] 866 | } 867 | } 868 | ] 869 | }, 870 | { 871 | "cell_type": "code", 872 | "metadata": { 873 | "id": "U36mF5ffG0pr", 874 | "colab_type": "code", 875 | "outputId": "79b15000-bfb3-4c78-8f73-0284ab15d335", 876 | "colab": { 877 | "base_uri": "https://localhost:8080/", 878 | "height": 692 879 | } 880 | }, 881 | "source": [ 882 | "# Train on federated model\n", 883 | "print('========== Train on Local Device of Client 2 ==========')\n", 884 | "\n", 885 | "history_accuracy_2 = []\n", 886 | "history_loss_2 = []\n", 887 | "\n", 888 | "for round_num in range(6):\n", 889 | " train_state, train_metrics = trainer.next(train_state, federated_data_client_2)\n", 890 | " history_accuracy_2.append(train_metrics.sparse_categorical_accuracy)\n", 891 | " history_loss_2.append(train_metrics.loss)\n", 892 | " print('round {:2d}, metrics={}'.format(round_num, train_metrics))\n", 893 | "\n", 894 | "# Show accuracy diagram\n", 895 | "plt.title('Model Accuracy for Client 2')\n", 896 | "plt.plot(history_accuracy_2, label='accuracy')\n", 897 | "plt.xlabel('Epoch')\n", 898 | "plt.ylabel('Accuracy')\n", 899 | "plt.show()\n", 900 | "\n", 901 | "# Show loss diagram\n", 902 | "plt.title('Model Loss for Client 2')\n", 903 | "plt.plot(history_loss_2, label='loss')\n", 904 | "plt.xlabel('Epoch')\n", 905 | "plt.ylabel('Loss')\n", 906 | "plt.show()" 907 | ], 908 | "execution_count": 0, 909 | "outputs": [ 910 | { 911 | "output_type": "stream", 912 | "text": [ 913 | "========== Train on Local Device of Client 2 ==========\n", 914 | "round 0, metrics=\n", 915 | "round 1, metrics=\n", 916 | "round 2, metrics=\n", 917 | "round 3, metrics=\n", 918 | "round 4, metrics=\n", 919 | "round 5, metrics=\n" 920 | ], 921 | "name": "stdout" 922 | }, 923 | { 924 | "output_type": "display_data", 925 | "data": { 926 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xc1Zn/8c+jZrk3ybhIcpVtjDHY\nCNuhOIRqCDWk2GBKNok3u4GQsCnkt/lls2yyAULYFEh+awiEYIOXQCAOcYHQm9zAFHe5qNnGknuV\nVZ7fH3PNjoUsy6CrO6P5vl+vefmWM/c+M/LMM/ecc88xd0dERFJXWtQBiIhItJQIRERSnBKBiEiK\nUyIQEUlxSgQiIilOiUBEJMUpEUhozGyQmbmZZbSg7I1m9lpbxJXozOyfzOwDM9trZr3b4Hw/NrOZ\nwXJBcN70sM8riUOJQAAws41mdsjMchptfzv4Mh8UTWRHxNIl+JKaF3UsYTGzTOAe4EJ37+Lu21rp\nuNeY2ZLg/dtsZvPM7KzG5dy9LDhvfSuc8yUz+2oz+4eb2V/MrMrMtpvZAjMb8UnPK8dPiUDibQCm\nHl4xs5OBTtGF8xFXAzXABWbWty1P3JKrmlZyApANLD/eJ1rMRz7TZnYr8EvgP4PjFwC/Ba74ZKF+\nYj2AOcAIYnEtAv4SaUQpSolA4j0CXB+3fgPwx/gCZtbdzP4Y/IorNbMfHv7yMbN0M7vbzKrNbD3w\n2Sae+/vgF2mlmf3kOKsgbgD+H/AuMK3RsfPN7M9BXNvM7N64fV8zs5VmtsfMVpjZuGC7m9mwuHJ/\nMLOfBMvnmFmFmX3fzLYAD5lZTzN7JjjHjmA5L+75vczsITPbFOx/Otj+vpldFlcuM3iPxjZ6DcOB\n1cHqTjN7Idh+hpktNrNdwb9nxD3nJTP7qZm9DuwHhjR+z4HbgW+4+5/dfZ+717r7X939u43f4MbV\nec39zQ5X5wV/8x1mtsHMLg72/RQ4G7g3uAq5t/G53H2Ru//e3be7ey3wX8CItqgOkyMpEUi8YqCb\nmZ0YfNinADMblfkN0J3YF86niSWOLwf7vgZcCowFioDPN3ruH4A6YFhQ5kLgqFUH8cxsIHAOMCt4\nXB+3Lx14BigFBgEDgNnBvi8APw7KdwMuB1pa3dIX6AUMBKYT+7w8FKwXAAeA+C+4R4hdQZ0E9CH2\nxQaxZBqfuC4BNrv72/Enc/c1wXMBerj7uWbWC/gb8GugN7Fqo781+rK8Loiva/AexPsUsSuMp1r4\nmhv7A83/zSYQS145wF3A783M3P1fgVeBm4KqpptacK5JwJbWqg6T4+DueugBsBE4H/gh8DNgMvAc\nkAE4sS/YdOAQMCruef8IvBQsvwB8PW7fhcFzM4hd+tcAHeP2TwVeDJZvBF5rJr4fAsuC5QFAPTA2\nWP8UUAVkNPG8BcAtRzmmA8Pi1v8A/CRYPid4rdnNxHQqsCNY7gc0AD2bKNcf2AN0C9afAL53lGMO\nOvyeBevXAYsalXkTuDFYfgm4vZkYryX25drc3/7HwMzG52/h36wkbl+n4Ll942L7agv//+UBlcDU\nqD8Lqfhoq3pPSR6PAK8Ag2lULUTsV18mR/7qLCX2xQyxL7zyRvsOGxg8d7OZHd6W1qh8c64H7gdw\n90oze5lYVdHbQD5Q6u51TTwvH1jXwnM0VuXuBw+vmFknYr/yJwM9g81dgyuSfGC7u+9ofBB33xRU\n3VxtZk8BFwO3tDCG/nz0V378ew7Nv4fbgBwzyzjK+9OclvzNthxecPf9Qbkux3MSM8sFngV+6+6P\nHWeM0gpUNSRHcPdSYo3GlwB/brS7Gqgl9gVxWAGxX3IAm4l9IcbvO6yc2K/LHHfvETy6uftJHENQ\nJ14I/MDMtgR19hOAa4K67HKg4CgNuuXA0KMcej9HNoY3boBuPDTvvxBr2Jzg7t2IVWUAWHCeXmbW\n4yjnephY9dAXgDfdvfIo5RrbxJHvNxz5njcVZ7w3ib3vV7bwfPE+9t+sBXEBYGY9iSWBOe7+048R\no7QCJQJpyleAc919X/xGj3UpfBz4qZl1Dertb+V/2xEeB75pZnnBB/y2uOduJvaB/4WZdTOzNDMb\namafbkE8NxCrphpFrDrmVGA00JHYr+tFxJLQHWbW2cyyzezM4LkPAN8xs9OCXjXDgrgBlhFLJulm\nNplYm0dzuhJrF9gZ1N3/W6PXNw/4bdConGlmk+Ke+zQwjtiVQOMrrebMBYZbrPtnhpl9KXgfnmnJ\nk919F/Aj4D4zu9LMOgWxXWxmdx3juZ/kbwbwAY0ar+OZWTdiVXevu/ttRysn4VMikI9w93XuvuQo\nu28G9gHrgdeAR4EHg333E/tgvwO8xUevKK4HsoAVwA5ideX9movFzLKBLwK/cfctcY8NxKqxbggS\n1GXEGjTLgArgS8Fr+RPw0yDOPcS+kHsFh78leN5OYnXpTzcXC7EumB2JXRkVA/Mb7b+O2BXTKmAr\n8K3DO9z9APAksSq3xu/LUXms4fRSYlcj24DvAZe6e/VxHOMXxBL2D4m1pZQDN3Hs1wsf428W51fA\n54MeRb9uYv9VwOnAl4OeRYcfBU2UlRCZuyamEWkLZvYjYLi7TztmYZE2pMZikTYQVCV9hdhVg0hC\nUdWQSMjM7GvEqmPmufsrUccj0piqhkREUlyoVwRmNtnMVptZiZl9pFeAmQ00s+fN7N3gVvm8po4j\nIiLhCe2KILjJZg1wAbFeHIuJ3TW4Iq7Mn4Bn3P1hMzsX+LK7N1uHmpOT44MGDQolZhGR9mrp0qXV\n7p7b1L4wG4vHE7v9fD2Amc0mNtrhirgyo4h1awN4kRZ0Zxs0aBBLlhytZ6OIiDTFzBrfof6hMKuG\nBnDkregVHHlbPMT6m38uWL6K2O36Hxl50MymW2ws9SVVVVWhBCsikqqi7jX0HeDTZvY2sbs6K4kN\nJnYEd5/h7kXuXpSb2+SVjYiIfExhVg1VcuS4M4dHF/yQu28iuCIwsy7A1e6+M8SYRESkkTCvCBYD\nhWY22MyyiI1tPye+gJnl2P/OqPQD/neoAhERaSOhJYJgyNubiI09sxJ43N2Xm9ntZnZ5UOwcYLWZ\nrSE29rlGHxQRaWNJd0NZUVGRq9eQiMjxMbOl7l7U1L6oG4tFRCRiSgQiIglu14Fa7pq/irJt+0M5\nvkYfFRFJUDV19TzyZin3vljCrgO19OvRket6N56w7pNTIhARSTANDc7Tyyr5xbNrqNx5gEnDc/n+\n5BGc1L97KOdTIhARSRDuzstrqrhz/mpWbt7N6AHduOvzYzhzWE6o51UiEBFJAO9W7OSOeat4Y902\nCnp14tdTx3Lpyf1IS7PQz61EICISodJt+/j5gtU88+5menXO4seXjeKaCQPJymi7vjxKBCIiEaje\nW8Nvnl/LrIVlZKan8c1zh/G1SUPomp3Z5rEoEYiItKF9NXU88OoGZryyjoN1DUw5PZ9bziukT7fs\nyGJSIhARaQO19Q3MXlzOr/6+luq9NVw8ui/fuWgEQ3O7RB2aEoGISJjcnXnvb+HnC1azoXof4wf1\nYsb1pzGuoGfUoX1IiUBEJCQL12/jZ/NWsax8J8NP6MLvbyji3JF9MAu/J9DxUCIQEWllq7fs4c75\nq3hh1Vb6dc/mrs+P4epxeaS3QVfQj0OJQESklWzaeYB7nlvDk29V0KVDBrddPJIbzxhEdmZ61KE1\nS4lAROQT2rW/lt++VMJDb2wE4GtnD+GfzxlKj05Z0QbWQkoEIiIf08Haeh5+YyP3vVjCnpo6rho7\ngFsvGE5ez05Rh3ZcQk0EZjYZ+BWQDjzg7nc02l8APAz0CMrc5u5zw4xJROSTqm9wnnq7knueXc2m\nXQc5Z0Qu3588khP7dYs6tI8ltERgZunAfcAFQAWw2MzmuPuKuGI/JDaF5e/MbBQwFxgUVkzS/rk7\nNXUNUYfR5jpkpCVcT5T2yN15aXUVd85fxaotezglrzt3f/EUzhga7qBwYQvzimA8UOLu6wHMbDZw\nBRCfCBw4nEK7A5tCjEdSwK2Pv8NTb1dGHUab69O1A2cNy+GswhzOGpYT6V2q7dWy8p38bO5KFm7Y\nzqDenbjvmnFccnLfdpGAw0wEA4DyuPUKYEKjMj8GnjWzm4HOwPlNHcjMpgPTAQoKClo9UGkfNu08\nwF+WVXLeyD4UDeoVdThtpsGd1Vv28NKaKv4cJMGRfbt+mBgmDO5Nx6zE7rWSyDZU7+PnC1Yx970t\n5HTJ4j+uOIkp4wvITG8/EzxG3Vg8FfiDu//CzD4FPGJmo939iGt7d58BzIDY5PURxClJ4LFFZTjw\n48tPIr9XcjXWtYaGBmfF5t28VlLNq2ur+GNxKQ+8toGs9DSKBvXkrMIczh6Wy0n9u7XJ0MbJrmpP\nDb96fg2zF5WTlZHGLecV8rVJQ+jSIeqvzdYX5iuqBPLj1vOCbfG+AkwGcPc3zSwbyAG2hhiXtEOH\nx3E5Z3huSiYBgLQ0Y/SA7owe0J2vf3ooBw7Vs3jjdl5dW8Wra6u5a/5q7mI1vTpnccbQ3pxdmMNZ\nhbkM6NEx6tATyt6aOma8sp4HXl3PoboGpo4v4JvnFZLbtUPUoYUmzESwGCg0s8HEEsAU4JpGZcqA\n84A/mNmJQDZQFWJM0k49u/wDqvbUMG1i68/nmqw6ZqUzaXguk4bnArB1z0FeL6nm1bXVvLa2mmfe\n3QzAkNzOnD0slhQmDukVyTDIieBQXQOzF5fx6+fXUr33EJ89uR/fuWgEg3M6Rx1a6EJLBO5eZ2Y3\nAQuIdQ190N2Xm9ntwBJ3nwP8C3C/mX2bWMPxje6uqh85bjOLSxnQoyPnjOgTdSgJq0/XbK4am8dV\nY/Nwd9Zu3csra6p4raSax5dU8PCbpWSkGWMLenDWsFzOKszhlLzuZLSjuvCmNDQ4f3tvM3c/u5rS\nbfuZOKQXD9xwIqfm94g6tDZjyfa9W1RU5EuWLIk6DEkgJVv3cv49L/Pdi0bwjc8MizqcpFRTV8/S\n0h28traa10qqea9yF+7QNTuDM4b25qzCXCYV5jCwd/v6dfxGSTV3zF/FuxW7GNm3K9+/eCTnDM9t\nFz2BGjOzpe5e1NS+9tfqISln1sJSMtONLxblH7uwNKlDRjpnDM3hjKE5fA/Yse8Qr6+LVSG9uraa\nBcs/ACC/V0fOGpbL2YU5nDG0d9IModDYik27uXP+Kl5eU0X/7tn84guncOXYAQk7KFzYlAgkqR04\nVM+TSyuYPLpfu27Ma2s9O2dx6Zj+XDqmP+7Oxm37P2x0fuadTTy2qIw0g5PzegTtCzmMK+jZpvPs\nfhwVO/Zzz7NreGpZJd2yM/k/l4zk+k8l/qBwYVMikKT213c2sftgHdMm6P6SsJgZg3M6MzinM9d/\nahC19Q28U74z1uhcUs3vXl7HvS+W0CkrnYlDenPWsBwmDc9haG6XhKli2bHvEPe9WMIf3ywFg+mT\nhvDPnx5G906p2TDemBKBJLWZC0sZfkIXxg9OnRvIopaZnkbRoF4UDerFty8Yzu6Dtby5btuH7Qsv\nrIr1/u7bLTt270JhDmcOyyGnS9tfsR2srefB1zfwu5fWsa+mjqvH5fHtC4bTX11mj6BEIEnrnfKd\nvFuxi3+//KSE+eWZirplZ3LRSX256KS+AJRv389rJbH2hb+v/IAnllYAMKpft+DehRxOH9Qr1OqY\nuvoGnnyrgv96bi1bdh/kvJF9+N7kkYzo2zW0cyYzJQJJWjOLS+mYmc5V4wZEHYrEye/VianjC5g6\nvoD6Buf9yl0f3u384Osb+O9X1tMhI43xg3t9OAzGiX1b525nd+fvK7dy1/xVrN26l1Pze/CrKacy\nYUjvVnhl7ZcSgSSlXftr+eu7m7hq7AC6pegNUMkgPc04Jb8Hp+T34BufGca+mjoWbdjOq2tjieFn\n81bBPMjpksWZw2ID5p1dmEvf7sc/aN7S0h3cMW8lizfuYEhOZ3537Tgmj24fg8KFTYlAktITb1Vw\nsLaBayfoTuJk0rlDBp8Z2YfPjIzd+Ldl18GgGil2Y9tflsUGIC7s0+XD9oUJg3vTuZnxfdZV7eWu\n+atYsPwDcrp04CdXjuZLp+e3q0HhwqYbyiTpuDvn3fMy3bIzefobZ0YdjrSShgZn1ZY9vFYS66a6\naMN2auoayEw3xhX0/HBspJMHdCc9zdi6+yD/9fe1PL6knI6Z6fzjpCH8w1mDm00aqUw3lEm78ua6\nbayv2sfdXzgl6lCkFaWlGaP6d2NU/25MnzSUg7X1LNm4g1dLqnh1TTV3P7uGu59dQ/eOmYwr6EHx\n+u3UNTRw3cSB3HzuMHpH0CupvVAikKQzc2EpPTplcumYflGHIiHKzkyPTbRTmMMPLobqvTW8HvRG\nWrRxOxeMOoF/uXB4uxv2IgpKBJJUtu4+yLPLP+DLZ+pu0FST06UDV5w6gCtOVS+x1qbWFEkqsxeX\nU9fgXKNGYpFWo0QgSaOuvoFHF5ZxdmFOSowRL9JWlAgkaTy/aitbdh9Ul1GRVhZqIjCzyWa22sxK\nzOy2Jvb/l5ktCx5rzGxnmPFIcptZXErfbtmcf6ImnxFpTaE1FptZOnAfcAFQASw2sznuvuJwGXf/\ndlz5m4GxYcUjyW1j9T5eXVvNt84vbPczZom0tTA/UeOBEndf7+6HgNnAFc2Unwo8FmI8ksQeXVRG\nepox5XQNNy3S2sJMBAOA8rj1imDbR5jZQGAw8MJR9k83syVmtqSqSnPbp5qDtfX8aUk5F4464WON\nQSMizUuUa+wpwBPuXt/UTnef4e5F7l6Um5vbxqFJ1Oa+t5kd+2uZNlGNxCJhCDMRVALxk8jmBdua\nMgVVC8lRzCwuZUhOZ84YqqGERcIQZiJYDBSa2WAzyyL2ZT+ncSEzGwn0BN4MMRZJUss37eKtsp1c\nM6FAwwmLhCS0RODudcBNwAJgJfC4uy83s9vN7PK4olOA2Z5sw6BKm5hZXEaHjDQ+f1pe1KGItFuh\njjXk7nOBuY22/ajR+o/DjEGS156DtfxlWSWXndKfHp2yog5HpN1KlMZikY946u1K9h+qVyOxSMiU\nCCQhuTszi0sZPaAbp+R1jzockXZNiUAS0uKNO1jzwV6mTRioRmKRkCkRSEKaWVxK1+wMLj+1f9Sh\niLR7SgSScKr31jDv/c1cPS6PTlmaO0kkbEoEknAeX1JObb0zbaLGFRJpC0oEklDqG5xHF5YxcUgv\nhvXpGnU4IilBiUASyitrqqjYcUBdRkXakBKBJJRHikvJ6dKBC0f1jToUkZShRCAJo3z7fl5cvZUp\np+eTlaH/miJtRZ82SRiPLSrDgKkT1Egs0paUCCQhHKpr4PEl5Zw78gQG9OgYdTgiKUWJQBLC/OVb\nqN57SF1GRSKgRCAJYWZxKQW9OjGpUDPQibQ1JQKJ3JoP9rBow3aumVBAWprGFRJpa0oEErlZxaVk\npafxBU0+IxKJUBOBmU02s9VmVmJmtx2lzBfNbIWZLTezR8OMRxLPvpo6/vxWJZec3JfeXTpEHY5I\nSgptRC8zSwfuAy4AKoDFZjbH3VfElSkEfgCc6e47zKxPWPFIYvrLsk3sqanTncQiEQrzimA8UOLu\n6939EDAbuKJRma8B97n7DgB33xpiPJJgDk8+M7JvV04b2DPqcERSVpiJYABQHrdeEWyLNxwYbmav\nm1mxmU1u6kBmNt3MlpjZkqqqqpDClbb2dvlOVmzezbUTNfmMSJSibizOAAqBc4CpwP1m1qNxIXef\n4e5F7l6Um6vuhe3FzOJSOmelc9XYxr8PRKQthZkIKoH8uPW8YFu8CmCOu9e6+wZgDbHEIO3cjn2H\neObdzVw1bgBdOmjyGZEohZkIFgOFZjbYzLKAKcCcRmWeJnY1gJnlEKsqWh9iTJIgnlhawaG6BjUS\niySA0BKBu9cBNwELgJXA4+6+3MxuN7PLg2ILgG1mtgJ4Efiuu28LKyZJDA0NzqyFpRQN7MnIvt2i\nDkck5YV6Te7uc4G5jbb9KG7ZgVuDh6SI19dVs3Hbfr51/vCoQxERom8slhQ0s7iUXp2zuPhkTT4j\nkgiUCKRNbd51gOdWfMAXivLokJEedTgighKBtLHHFpXjwLXj1UgskiiUCKTN1NY3MHtRGZMKcyno\n3SnqcEQkoEQgbebvKz5g654arlOXUZGEokQgbWbmwlIG9OjIZ0ZqbEGRRKJEIG1ifdVeXi/ZxtTx\n+aRr8hmRhHLMRGBmN5uZhoaUT2TWwjIy0owvnp5/7MIi0qZackVwArG5BB4PJprRzzk5Lgdr63li\naQUXje5Ln67ZUYcjIo0cMxG4+w+JDQT3e+BGYK2Z/aeZDQ05Nmkn/vrOJnYdqGXaBDUSiySiFrUR\nBENBbAkedUBP4AkzuyvE2KSdmFlcyrA+XZg4pFfUoYhIE1rSRnCLmS0F7gJeB052938CTgOuDjk+\nSXLvVezinYpdXDuhQJPPiCSolgw61wv4nLuXxm909wYzuzScsKS9mFlcSsfMdD43Li/qUETkKFpS\nNTQP2H54xcy6mdkEAHdfGVZgkvx2HajlL+9UcsWp/eneMTPqcETkKFqSCH4H7I1b3xtsE2nWn9+q\n4GCtJp8RSXQtSQQWNBYDsSohQp7HQJKfuzNrYRmn5Pdg9IDuUYcjIs1oSSJYb2bfNLPM4HELLZxO\nMrjvYLWZlZjZbU3sv9HMqsxsWfD46vG+AElMxeu3U7J1L9MmFEQdiogcQ0sSwdeBM4hNPF8BTACm\nH+tJZpYO3AdcDIwCpprZqCaK/o+7nxo8Hmhx5JLQZi4spXvHTC47pX/UoYjIMRyzisfdtxKbeP54\njQdK3H09gJnNBq4AVnyMY0kS2brnIAve38INZwwiO1OTz4gkumMmAjPLBr4CnAR8OD6Au//DMZ46\nACiPWz98NdHY1WY2CVgDfNvdyxsXMLPpBFchBQWqakh0/7OonLoG51pVC4kkhZZUDT0C9AUuAl4G\n8oA9rXT+vwKD3H0M8BzwcFOF3H2Guxe5e1Fubm4rnVrCUN/gPLaojDOH9WZIbpeowxGRFmhJIhjm\n7v8X2OfuDwOfpelf9o1VAvFDTeYF2z7k7tvcvSZYfYDY3cqSxF5YtZVNuw5q8hmRJNKSRFAb/LvT\nzEYD3YGWzCyyGCg0s8FmlkWsnWFOfAEz6xe3ejmgG9SS3MziUk7o1oHzTzwh6lBEpIVacj/AjGA+\ngh8S+yLvAvzfYz3J3evM7CZgAZAOPOjuy83sdmCJu88BvmlmlxMbyG47sdFNJUmVbdvPK2ur+Oa5\nhWSka84jkWTRbCIwszRgt7vvAF4BhhzPwd19LjC30bYfxS3/APjB8RxTEtesRaWkmTF1vBqJRZJJ\nsz/bgruIv9dGsUgSq6mr509LKjj/xD707a7JZ0SSSUuu3/9uZt8xs3wz63X4EXpkklTmvbeF7fsO\naVwhkSTUkjaCLwX/fiNum3Oc1UTSvs0sLmVQ706cOTQn6lBE5Di15M7iwW0RiCSvlZt3s6R0B/96\nyYmkpWnyGZFk05I7i69varu7/7H1w5FkNLO4lA4ZaXz+NE0+I5KMWlI1dHrccjZwHvAWoEQg7K2p\n4+m3K7l0TH96ds6KOhwR+RhaUjV0c/y6mfUAZocWkSSVp96uZN+heqZNVJdRkWT1ce762Qeo3UBi\nk88Ul3JS/26cmt8j6nBE5GNqSRvBX4n1EoJY4hgFPB5mUJIclpbuYNWWPfzscydjpkZikWTVkjaC\nu+OW64BSd68IKR5JIjOLS+naIYMrTtXkMyLJrCWJoAzY7O4HAcyso5kNcveNoUYmCW3b3hrmvreF\nqePz6ZSlKaxFkllL2gj+BDTErdcH2ySF/WlpBYfqG7hWdxKLJL2WJIIMdz90eCVYVj/BFNbQ4Mxa\nWMr4wb0YfkLXqMMRkU+oJYmgKhgqGgAzuwKoDi8kSXQvr62ifPsBTT4j0k60pHL368AsM7s3WK8A\nmrzbWFLDrOJScrp04KKT+kYdioi0gmNeEbj7OnefSKzb6Ch3P8PdS1pycDObbGarzazEzG5rptzV\nZuZmVtTy0CUKlTsP8MKqrXzp9DyyMjT5jEh7cMxPspn9p5n1cPe97r7XzHqa2U9a8Lx04D7gYmJJ\nZKqZjWqiXFfgFmDh8Ycvbe2xhWU4aPIZkXakJT/pLnb3nYdXgtnKLmnB88YDJe6+Pmhgng1c0US5\n/wDuBA624JgSoUN1DcxeXM65I/qQ17NT1OGISCtpSSJIN7MOh1fMrCPQoZnyhw0AyuPWK4JtHzKz\ncUC+u/+tBceTiD27YgvVe2s0+YxIO9OSxuJZwPNm9hBgxCaYf/iTnjiYD/keWjBhvZlNB6YDFBSo\nSiIqM4tLyevZkUnDc6MORURaUUsai+8EfgKcCIwAFgAt+UlYCeTHrecF2w7rCowGXjKzjcBEYE5T\nDcbuPsPdi9y9KDdXX0JRKNm6h+L127lmQgHpmnxGpF1pabePD4gNPPcF4FxgZQuesxgoNLPBZpYF\nTAHmHN7p7rvcPcfdB7n7IKAYuNzdlxzPC5C2MbO4jMx044tF+ccuLCJJ5ahVQ2Y2HJgaPKqB/wHM\n3T/TkgO7e52Z3UTsCiIdeNDdl5vZ7cASd5/T/BEkUew/VMeTSyu45OR+5HRpSfOQiCST5toIVgGv\nApcevm/AzL59PAd397nA3EbbfnSUsuccz7Gl7cxZtok9NXVqJBZpp5qrGvocsBl40czuN7PziDUW\nSwpxd2YuLGXECV0pGtgz6nBEJARHTQTu/rS7TwFGAi8C3wL6mNnvzOzCtgpQovVOxS7er9zNtIkF\nmnxGpJ1qSa+hfe7+qLtfRqznz9vA90OPTBLCzOJSOmWlc+XYAccuLCJJ6bgGi3H3HUFXzvPCCkgS\nx879h/jrO5u4cuwAumZnRh2OiIREo4bJUT2xtIKaugamTVAjsUh7pkQgTXJ3Hl1YxriCHozq3y3q\ncEQkREoE0qQ31m1jffU+rvuUrgZE2jslAmnSI2+W0rNTJheP7hd1KCISMiUC+Ygtuw7y3MoP+GJR\nPtmZ6VGHIyIhUyKQj5i9uIz6BueaCRrpVSQVKBHIEerqG5i9qJxJw3MZ2Ltz1OGISBtQIpAj/H3l\nVrbsPsg0XQ2IpAwlAjnCrOhEvvYAAA35SURBVIWl9Ouezbkj+0Qdioi0ESUC+dCG6n28uraaqeML\nyEjXfw2RVKFPu3zo0YWlZKQZU07X5DMiqUSJQAA4WFvPn5ZWcNFJfenTLTvqcESkDYWaCMxsspmt\nNrMSM7utif1fN7P3zGyZmb1mZqPCjEeO7pl3N7Nzfy3XTlQjsUiqCS0RmFk6cB9wMTAKmNrEF/2j\n7n6yu58K3AXcE1Y80ryZxaUMze3Mp4b0jjoUEWljYV4RjAdK3H29ux8CZgNXxBdw991xq50BDzEe\nOYr3K3exrHwn104YqMlnRFJQc3MWf1IDgPK49QpgQuNCZvYN4FYgCzi3qQOZ2XRgOkBBgaouWtus\nhaVkZ6Zx9Wl5UYciIhGIvLHY3e9z96HEZj374VHKzHD3Incvys3NbdsA27ndB2t5+u1NXH5Kf7p3\n1OQzIqkozERQCcT3Q8wLth3NbODKEOORJjz1ViUHauuZNlHDTYukqjATwWKg0MwGm1kWMAWYE1/A\nzArjVj8LrA0xHmnE3ZlZXMqYvO6MyesRdTgiEpHQ2gjcvc7MbgIWAOnAg+6+3MxuB5a4+xzgJjM7\nH6gFdgA3hBWPfNSiDdtZu3Uvd109JupQRCRCYTYW4+5zgbmNtv0obvmWMM8vzZu5sIxu2Rlcdkr/\nqEMRkQhF3lgs0ajaU8P89zfz+dPy6ZilyWdEUpkSQYp6fEk5tfWuO4lFRIkgFdU3OI8uLOOMob0Z\nmtsl6nBEJGJKBCnopdVbqdx5QF1GRQRQIkhJM4tL6dO1AxeMOiHqUEQkASgRpJjy7ft5aU0VU07P\nJ1OTz4gISgQp59FFZRgwZbwaiUUkRokghdTU1fP44nLOP/EE+vfoGHU4IpIglAhSyPz3t7Bt3yE1\nEovIEZQIUsjM4lIG9u7EWcNyog5FRBKIEkGKWLVlN4s37uDaCQWkpWnyGRH5X0oEKWJWcRlZGWl8\n4bT8YxcWkZSiRJAC9tXU8dTblVx6cj96ds6KOhwRSTBKBCng6WWV7K2p41o1EotIE5QI2rnY5DNl\nnNivG+MKNPmMiHyUEkE791bZTlZu3s20iQWYqZFYRD4q1ERgZpPNbLWZlZjZbU3sv9XMVpjZu2b2\nvJmp7qKVzSoupUuHDK48dUDUoYhIggotEZhZOnAfcDEwCphqZqMaFXsbKHL3McATwF1hxZOKtu87\nxDPvbeZz4wbQuUOok9GJSBIL84pgPFDi7uvd/RAwG7givoC7v+ju+4PVYiAvxHhSyq4Dtfz0bys5\nVNegO4lFpFlh/kwcAJTHrVcAE5op/xVgXlM7zGw6MB2goECDpTWnpq6eR94s5d4XS9i5v5YbzxjE\n8BO6Rh2WiCSwhKgvMLNpQBHw6ab2u/sMYAZAUVGRt2FoSaOhwXl6WSW/eHYNlTsPcHZhDt+fPJLR\nA7pHHZqIJLgwE0ElEH8ba16w7Qhmdj7wr8Cn3b0mxHjaJXfn5TVV3Dl/NSs372b0gG7cefUYzirU\neEIi0jJhJoLFQKGZDSaWAKYA18QXMLOxwH8Dk919a4ixtEvvVuzkjnmreGPdNvJ7deRXU07lsjH9\nNZaQiByX0BKBu9eZ2U3AAiAdeNDdl5vZ7cASd58D/BzoAvwp6ONe5u6XhxVTe1G6bR8/X7CaZ97d\nTK/OWfzbZaO4dsJAsjJ0W4iIHL9Q2wjcfS4wt9G2H8Utnx/m+dub6r01/Ob5tcxaWEZmeho3nzuM\n6ZOG0DU7M+rQRCSJJURjsTRvX00dD7y6gRmvrONgXQNfOj2fb51XSJ9u2VGHJiLtgBJBAqutb2D2\n4nJ+9fe1VO+tYfJJffnu5BEMze0SdWgi0o4oESQgd2fe+1v4+YLVbKjex+mDevLf153GaQN7Rh2a\niLRDSgQJZuH6bfxs3iqWle+ksE8XHri+iPNO7KMB40QkNEoECWL1lj3cOX8VL6zaSt9u2dx19Riu\nPi2PdHUFFZGQKRFEbNPOA9zz3BqefKuCLh0y+P7kkXz5zEFkZ6ZHHZqIpAglgojs2l/Lb18q4aE3\nNoLDV88azDc+M4wenTSVpIi0LSWCNnawtp4/vrmR+15cx+6DtVw1dgC3XjCcvJ6dog5NRFKUEkEb\nqW9wnnq7knueXc2mXQc5Z0Qu37toJKP6d4s6NBFJcUoEIXN3XlpdxZ3zV7Fqyx7G5HXn7i+ewhlD\nNSiciCQGJYIQLSvfyR3zVlK8fjsDe3fi3mvG8tmT+6krqIgkFCWCEGyo3sfdC1bzt/c207tzFrdf\ncRJTTi/QoHAikpCUCFpR1Z4afv38Wh5bVEZWRhrfPK+Q6ZOG0EXzBYtIAtM3VCvYW1PH/a+s5/5X\n11NT18DU8fl887xC+nTVoHAikviUCD6B2voGHltUxq+fX0v13kNccnJfvnPhCIZoUDgRSSJKBB+D\nu/O39zZz94LVbNy2nwmDe3H/9SMZW6BB4UQk+YSaCMxsMvArYjOUPeDudzTaPwn4JTAGmOLuT4QZ\nT2t4Y101d85bxTsVuxhxQlceuvF0zhmRq55AIpK0QksEZpYO3AdcAFQAi81sjruviCtWBtwIfCes\nOFrLys27uXP+Kl5aXUW/7tn8/PNj+Nw4DQonIskvzCuC8UCJu68HMLPZwBXAh4nA3TcG+xpCjOMT\nqdx5gF88u5qn3q6ka4cMfnDxSG44Q4PCiUj7EWYiGACUx61XABM+zoHMbDowHaCgoOCTR9YCO/cf\n4r4XS3j4zVIApp89hH8+ZxjdO2l+YBFpX5KisdjdZwAzAIqKijzMcx2sreeh1zfy25dK2FtTx9Xj\n8rj1guH079ExzNOKiEQmzERQCeTHrecF2xJSfYPz5NIK7nluDVt2H+TckX343uQRjOyrQeFEpH0L\nMxEsBgrNbDCxBDAFuCbE830s7s7zK7dy14JVrPlgL6fk9+CXU05l4pDeUYcmItImQksE7l5nZjcB\nC4h1H33Q3Zeb2e3AEnefY2anA08BPYHLzOzf3f2ksGJq7K2yHdwxdxWLNm5ncE5nfnvtOC4e3Vdd\nQUUkpYTaRuDuc4G5jbb9KG55MbEqoza1rmovP5+/mvnLt5DTpQP/ceVoppyeT2a6BoUTkdSTFI3F\nrWXr7oP88vm1/M/icrIz0vj2+cP56tmD6axB4UQkhaXMN+Dji8v5tznLqa1vYNqEAm4+r5CcLh2i\nDktEJHIpkwgKenfivBP78J0LRzAop3PU4YiIJIyUSQQTh/RWTyARkSaodVREJMUpEYiIpDglAhGR\nFKdEICKS4pQIRERSnBKBiEiKUyIQEUlxSgQiIinO3EOd56XVmVkVUPoxn54DVLdiOMlArzk16DWn\nhk/ymge6e25TO5IuEXwSZrbE3YuijqMt6TWnBr3m1BDWa1bVkIhIilMiEBFJcamWCGZEHUAE9JpT\ng15zagjlNadUG4GIiHxUql0RiIhII0oEIiIpLmUSgZlNNrPVZlZiZrdFHU/YzOxBM9tqZu9HHUtb\nMbN8M3vRzFaY2XIzuyXqmMJmZtlmtsjM3gle879HHVNbMLN0M3vbzJ6JOpa2YGYbzew9M1tmZkta\n/fip0EZgZunAGuACoAJYDEx19xWRBhYiM5sE7AX+6O6jo46nLZhZP6Cfu79lZl2BpcCV7fzvbEBn\nd99rZpnAa8At7l4ccWihMrNbgSKgm7tfGnU8YTOzjUCRu4dyA12qXBGMB0rcfb27HwJmA1dEHFOo\n3P0VYHvUcbQld9/s7m8Fy3uAlcCAaKMKl8fsDVYzg0e7/nVnZnnAZ4EHoo6lvUiVRDAAKI9br6Cd\nf0GkOjMbBIwFFkYbSfiCapJlwFbgOXdv76/5l8D3gIaoA2lDDjxrZkvNbHprHzxVEoGkEDPrAjwJ\nfMvdd0cdT9jcvd7dTwXygPFm1m6rAs3sUmCruy+NOpY2dpa7jwMuBr4RVP22mlRJBJVAftx6XrBN\n2pmgnvxJYJa7/znqeNqSu+8EXgQmRx1LiM4ELg/qzGcD55rZzGhDCp+7Vwb/bgWeIlbd3WpSJREs\nBgrNbLCZZQFTgDkRxyStLGg4/T2w0t3viTqetmBmuWbWI1juSKxDxKpoowqPu//A3fPcfRCxz/EL\n7j4t4rBCZWadg84PmFln4EKgVXsDpkQicPc64CZgAbEGxMfdfXm0UYXLzB4D3gRGmFmFmX0l6pja\nwJnAdcR+JS4LHpdEHVTI+gEvmtm7xH7wPOfuKdGlMoWcALxmZu8Ai4C/ufv81jxBSnQfFRGRo0uJ\nKwIRETk6JQIRkRSnRCAikuKUCEREUpwSgYhIilMiEGnEzOrjup8ua83Ras1sUCqNCCvJISPqAEQS\n0IFgyAaRlKArApEWCsaEvysYF36RmQ0Ltg8ysxfM7F0ze97MCoLtJ5jZU8FcAe+Y2RnBodLN7P5g\n/oBngzuCRSKjRCDyUR0bVQ19KW7fLnc/GbiX2CiYAL8BHnb3McAs4NfB9l8DL7v7KcA44PDd7IXA\nfe5+ErATuDrk1yPSLN1ZLNKIme119y5NbN8InOvu64PB7ba4e28zqyY2IU5tsH2zu+eYWRWQ5+41\ncccYRGwYiMJg/ftAprv/JPxXJtI0XRGIHB8/yvLxqIlbrkdtdRIxJQKR4/OluH/fDJbfIDYSJsC1\nwKvB8vPAP8GHk8d0b6sgRY6HfomIfFTHYMavw+a7++EupD2DkT5rgKnBtpuBh8zsu0AV8OVg+y3A\njGDk13piSWFz6NGLHCe1EYi0UNgTiItERVVDIiIpTlcEIiIpTlcEIiIpTolARCTFKRGIiKQ4JQIR\nkRSnRCAikuL+Pwy3IJSqT0uCAAAAAElFTkSuQmCC\n", 927 | "text/plain": [ 928 | "
" 929 | ] 930 | }, 931 | "metadata": { 932 | "tags": [] 933 | } 934 | }, 935 | { 936 | "output_type": "display_data", 937 | "data": { 938 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3yW9X3/8dc7CQkh4UxEIFG0otaq\ngAbnaZ61aq3TWkDXbuvWjdXZ1lbtab+d2rXdulrX1c1uaA+rtZ51W9WftRM8VasGFE+oFQRBUQKK\nnAkJn/1xX4GYBrhDcuW6D+/n43E/cue6D9/PFeV9Xff3uu7PpYjAzMxKT0XWBZiZWToc8GZmJcoB\nb2ZWohzwZmYlygFvZlaiHPBmZiXKAW8DRtJESSGpKo/nfkLSIwNRV29IqpX0c0nvSrp1gMZcIum0\n5P5fSrpuIMa14ueAtx4lodImaUy35U8lIT0xm8p6t6FIwUeBscDoiJjeH28oaZik70p6TdJ6SYuS\n38d0f25EfDMi/rQfxtzt31DSH0maJ2mtpOWS/imjv7ntIQe87cqrwEWdv0g6DBiSXTkFYV/g5Yho\n7+0LewpHSdXA/cAHgDOBYcAxwGrgqL6V2mdDgM8BY4DfAU4Frsi0IusVB7ztyvXAH3b5/Y+An3R9\ngqThkn4iqVXSUkl/JakieaxS0pWSVklaDHyoh9f+QNIKSa9L+rqkyr4ULKkm2ft9I7l9V1JN8tgY\nSXdJWiPpbUkPd6n1S0kN6yS9JOnUHt77q8DfADOTPe1PSqpI1nmppJXJ32J48vzOveRPSnoNmNND\nyX8I7AOcHxEvRMS2iFgZEX8fEff0UMPfSfppl9+PlvRosk4LJJ3U5bEHJP29pF8l63Vfl08FDyU/\n1yTrckz3sSLi+xHxcES0RcTrwA3AcXn8Z7AC4YC3Xfk1MEzS+5PgvRD4abfnXA0MB/YHTiQXWH+c\nPPZnwDnAVKCZ3PRGVz8G2oEDkuecAfR1+uH/AUcDU4DJ5PaC/yp57HJgOdBAbprlL4GQdBDwaWBa\nRAwFPggs6f7GEfG3wDeBmyOiPiJ+AHwiuZ1M7m9QD/xrt5eeCLw/ed/uTgPujYj1vV1RSROAu4Gv\nA6PI7V3fLqmhy9N+n9x/j72AanbsgZ+Q/ByRrMtjeQx5AvB8b+u07DjgbXc69+JPBxYCr3c+0CX0\nvxIR6yJiCfAd4A+Sp8wAvhsRyyLibeAfurx2LHA28LmI2BARK4F/Tt6vLz4GfC3ZC24Fvtqlnq3A\nOGDfiNia7J0G0AHUAIdIGhQRSyJiUS/GuyoiFich/RXgwm7TMX+XrOOmHl4/GljR+9UE4OPAPRFx\nT7Ln/0ughdzftdOPIuLlZOxbyG34ek3Sn5DbSF+5h7VaBhzwtjvXk9sL/ATdpmfIzc0OApZ2WbYU\nmJDcHw8s6/ZYp32T165IphfWAP9Bbk+zL8b3UM/45P63gVeA+yQtlvRlgIh4hdxc898BKyXdJGk8\n+elpvCpynxA6LWPnVpPb6OyJfYHpnX+/5G94fLf3e7PL/Y3kPmH0iqTzyG2cz4qIVXtYq2XAAW+7\nFBFLyR1sPRu4o9vDq8jtFe/bZdk+7NjLXwE0dXus0zJgCzAmIkYkt2ER8YE+lvxGD/W8kazLuoi4\nPCL2B84FLuuca4+In0XE8clrA/hWH8ZrB97qsmxXLVv/F/igpLo8x+tqGXB9l7/fiIioi4h/zOO1\nebWRlXQmcC3w4Yh4dg9qtAw54C0fnwROiYgNXRdGRAe5j/3fkDRU0r7AZeyYp78F+KykRkkjgS93\nee0K4D7gO8lpghWS3ifpxF7UVSNpcJdbBXAj8FeSGpIDin/TWY+kcyQdIEnAu+SmZrZJOkjSKcnB\n2M3AJmBbnjXcCHxe0n6S6tkxR5/vWTbXkwvq2yUdnPwdRit3vvvZu3ntT4EPS/pgckB7sKSTJDXm\nMW4ruXXcf2dPkHQKuQOrF0TEE3mujxUQB7ztVkQsioiWnTz8GWADsBh4BPgZ8MPksWuBXwALgPn8\n9ieAPyR34O8F4B3gNno3XbGeXBh33k4hd8CxBXgGeDYZ9+vJ8yeR22NeDzwGXBMRc8nNv/8juU8k\nb5KbJvpKnjX8kFxIP0Tuk85mcn+TvETEFnIHWl8EfgmsBZ4gN/31+G5euwz4PXIHi1vJbSi+QB7/\nriNiI/AN4FfJ9M7RPTztr8kdQL8nOdNmvaT/n++6WfbkC36YmZUm78GbmZUoB7yZWYlywJuZlSgH\nvJlZiSqoznBjxoyJiRMnZl2GmVnRmDdv3qqIaOjpsYIK+IkTJ9LSsrOz8czMrDtJS3f2mKdozMxK\nlAPezKxEOeDNzEqUA97MrEQ54M3MSpQD3sysRDngzcxKVNEH/OatHcx+aBGPLvKFZszMuir6gK+q\nENc+/Co/fOTVrEsxMysoqQa8pM9Lel7Sc5JulDS4v8eoqqzggiMamftSKyvXbu7vtzczK1qpBbyk\nCcBngeaIOBSoBC5MY6wZzY10bAtun//67p9sZlYm0p6iqQJqJVUBQ0guftzf9m+oZ9rEkdzasgxf\nocrMLCe1gI+I14ErgdeAFcC7EXFf9+dJmiWpRVJLa2vrHo83o7mJxas20LL0nT1+DzOzUpLmFM1I\nchcE3g8YD9RJ+nj350XE7IhojojmhoYeO17m5ezDxlFXXcktTy7b4/cwMyslaU7RnAa8GhGtEbEV\nuAM4Nq3B6mqq+PDk8dz97ArWb2lPaxgzs6KRZsC/BhwtaYgkAacCC1Mcj+nNTWxs6+DuZ1KZ6jcz\nKyppzsE/DtwGzAeeTcaandZ4AEfsM4ID9qrnZk/TmJmlexZNRPxtRBwcEYdGxB9ExJY0x5PEjOZG\n5r+2hldWrktzKDOzglf032Tt7vypjVRViFtalmddiplZpkou4BuG1nDKwXtxx/zlbO3YlnU5ZmaZ\nKbmAB5g5rYlV69uY++LKrEsxM8tMSQb8iQc2sNfQGm5p8cFWMytfJRnwVZUVXHCkG5CZWXkryYAH\nmH6kG5CZWXkr2YDfv6GeoyaOcgMyMytbJRvwANObG92AzMzKVkkH/IcOdwMyMytfJR3wQ6rdgMzM\nyldJBzzAjGluQGZm5ankA35qkxuQmVl5KvmAl8TM5iY3IDOzslPyAQ9w/hET3IDMzMpOWQT8mPoa\nTn2/G5CZWXkpi4CH3EW5V61vY44bkJlZmUjzotsHSXq6y22tpM+lNd7udDYgu9UNyMysTKR5yb6X\nImJKREwBjgQ2AnemNd7uuAGZmZWbgZqiORVYFBFLB2i8Hs1obnIDMjMrGwMV8BcCN/b0gKRZklok\ntbS2tqZaxH5j6tyAzMzKRuoBL6kaOBe4tafHI2J2RDRHRHNDQ0Pa5TBjWpMbkJlZWRiIPfizgPkR\n8dYAjLVbZx+2N/U1Vf5mq5mVvIEI+IvYyfRMFnINyMZx9zNuQGZmpS3VgJdUB5wO3JHmOL01vbmJ\nTVs7uGuBG5CZWelKNeAjYkNEjI6Id9Mcp7emNo1g0l71vii3mZW0svkma1eSmOEGZGZW4soy4MEN\nyMys9JVtwLsBmZmVurINeICZ09yAzMxKV1kH/AmTcg3IfFFuMytFZR3wVZUVfPTIRua+tJK33IDM\nzEpMWQc85M6J3xZw+3wfbDWz0lL2Ab/fmDqO2m8Ut7YsdwMyMyspZR/wkGsj/OqqDTy5xA3IzKx0\nOODZ0YDM32w1s1LigOe9DcjWbd6adTlmZv3CAZ+YkTQgu/uZFVmXYmbWLxzwiSlJA7KbPU1jZiXC\nAZ+QxMxpTTz12hp+85YbkJlZ8XPAd3He1M4GZN6LN7Pi54DvYkx9Dae9fyx3zH/dDcjMrOg54LuZ\nMa2R1RvauH+hG5CZWXFL+5J9IyTdJulFSQslHZPmeP3hhEkNjB1Ww62epjGzIpf2Hvy/APdGxMHA\nZGBhyuP1WVVlBRcc4QZkZlb8Ugt4ScOBE4AfAEREW0SsSWu8/uQGZGZWCtLcg98PaAV+JOkpSddJ\nquv+JEmzJLVIamltbU2xnPy5AZmZlYI0A74KOAL4fkRMBTYAX+7+pIiYHRHNEdHc0NCQYjm94wZk\nZlbs0gz45cDyiHg8+f02coFfFDobkN3sqz2ZWZFKLeAj4k1gmaSDkkWnAi+kNV5/62xAds+zbkBm\nZsUp7bNoPgPcIOkZYArwzZTH61edDcjucgMyMytCqQZ8RDydzK8fHhHnRURRTWh3NiBz6wIzK0b+\nJusuuAGZmRUzB/xuuAGZmRUrB/xudG1A1tbuBmRmVjwc8HnobEA250U3IDOz4uGAz0NnAzJP05hZ\nMXHA56GzAdkDbkBmZkXEAZ+nGUkDstvmuQGZmRUHB3yeJm5vQLbMDcjMrCg44HthZnMTS1Zv5IlX\n3866FDOz3XLA98JZSQOyW1o8TWNmhc8B3wu5BmTj3YDMzIqCA76XZjQ3ugGZmRUFB3wvTWkawYFj\n690n3swKngO+lyQxo7mJp5et4WU3IDOzAuaA3wPndzYg8168mRUwB/weGJ00ILvzKTcgM7PClWrA\nS1oi6VlJT0tqSXOsgTZzWlPSgOytrEsxM+vRQOzBnxwRUyKieQDGGjC/O2lM0oDM58SbWWHyFM0e\nqqqs4KNH5hqQvfmuG5CZWeFJO+ADuE/SPEmzenqCpFmSWiS1tLa2plxO/5p+ZK4B2e3zvRdvZoUn\n7YA/PiKOAM4CLpF0QvcnRMTs5MLczQ0NDSmX078mjqnjd9yAzMwKVKoBHxGvJz9XAncCR6U5XhZm\nuAGZmRWo1AJeUp2koZ33gTOA59IaLytnHzaO+poqbvbVnsyswKS5Bz8WeETSAuAJ4O6IuDfF8TJR\nW13pBmRmVpBSC/iIWBwRk5PbByLiG2mNlbWZ05rYvHUbP1/gBmRmVjh8mmQ/mNw4nAPH1vui3GZW\nUBzw/cANyMysEDng+8n5UycwqNINyMyscDjg+0lnA7I73IDMzAqEA74fzWhu4m03IDOzApFXwEt6\nn6Sa5P5Jkj4raUS6pRWfEw5sYO9hg321JzMrCPnuwd8OdEg6AJgNNAE/S62qIlVZIS44cgIPvtzq\nBmRmlrl8A35bRLQD5wNXR8QXgHHplVW83IDMzApFvgG/VdJFwB8BdyXLBqVTUnHrbEB2ixuQmVnG\n8g34PwaOAb4REa9K2g+4Pr2yitvMaU0sXb2Rx92AzMwylFfAR8QLEfHZiLhR0khgaER8K+XaitZZ\nh45jaE2Vv9lqZpnK9yyaByQNkzQKmA9cK+mqdEsrXrXVlXx4Sq4B2Vo3IDOzjOQ7RTM8ItYCHwF+\nEhG/A5yWXlnFb0ZzrgHZXW5AZmYZyTfgqySNA2aw4yCr7cLkxuEcNHao+8SbWWbyDfivAb8AFkXE\nk5L2B36TXlnFTxLTmxtZsGwNL73pBmRmNvDyPch6a0QcHhEXJ78vjogL0i2t+G1vQOa9eDPLQL4H\nWRsl3SlpZXK7XVJj2sUVu84GZHe6AZmZZSDfKZofAf8DjE9uP0+W7ZakSklPSSrLufsZ03INyO5f\n6AZkZjaw8g34hoj4UUS0J7cfAw15vvZSYOEeVVcCTpiUa0DmaRozG2j5BvxqSR9P9sYrJX0cWL27\nFyXTOB8CrutLkcWsskJ89MhGNyAzswGXb8D/CblTJN8EVgAfBT6Rx+u+C3wR2OkEtKRZkloktbS2\ntuZZTnGZ3tzoBmRmNuDyPYtmaUScGxENEbFXRJwH7PIsGknnACsjYt5u3nt2RDRHRHNDQ76zPsVl\n39F1HL1/rgHZtm1uQGZmA6MvV3S6bDePHwecK2kJcBNwiqSf9mG8ojajOdeA7IklbkBmZgOjLwGv\nXT0YEV+JiMaImAhcCMyJiI/3Ybyitr0Bma/2ZGYDpC8B77mGXtjegOw5NyAzs4Gxy4CXtE7S2h5u\n68idD5+XiHggIs7pc7VFrrMB2c8XvJF1KWZWBnYZ8BExNCKG9XAbGhFVA1VkqehsQHZLi8+mMbP0\n9WWKxnrJDcjMbCA54AeYG5CZ2UBxwA8wNyAzs4HigM+AG5CZ2UBwwGfADcjMbCA44DPgBmRmNhAc\n8BlxAzIzS5sDPiNuQGZmaXPAZ8gNyMwsTQ74DLkBmZmlyQGfITcgM7M0OeAzNtMNyMwsJQ74jB3u\nBmRmlhIHfMYkMWNakxuQmVm/c8AXADcgM7M0pBbwkgZLekLSAknPS/pqWmMVu1F11Zx+iBuQmVn/\nSnMPfgtwSkRMBqYAZ0o6OsXxitr0ZjcgM7P+lVrAR8765NdByc1f2dyJzgZkN3uaxsz6Sapz8JIq\nJT0NrAR+GRGPpzleMetsQPbQy62seHdT1uWYWQlINeAjoiMipgCNwFGSDu3+HEmzJLVIamltbU2z\nnIK3vQHZPJ8yaWZ9NyBn0UTEGmAucGYPj82OiOaIaG5oaBiIcgrWjgZky92AzMz6LM2zaBokjUju\n1wKnAy+mNV6pmDmtidfe3sjjr7oBmZn1TZp78OOAuZKeAZ4kNwd/V4rjlYQzP5BrQHarD7aaWR+l\neRbNMxExNSIOj4hDI+JraY1VSmqrKznXDcjMrB/4m6wFaIYbkJlZP3DAF6DDG4dz8N5D3SfezPrE\nAV+AJDG9uYkFy9/lxTfXZl2OmRUpB3yB2t6A7EmfE29me8YBX6B2NCBb7gZkZrZHHPAFbEZzE+9s\n3Mr/ugGZme0BB3wB+91JDYwbPth94s1sjzjgC5gbkJlZXzjgC9z0I5vcgMzM9ogDvsDtM3oIx+w/\n2g3IzKzXHPBFYMa0RjcgM7Nec8AXgbMOHcfQwVU+2GpmveKALwKDB1Vy7uTx3POsG5CZWf4c8EVi\n5rQmtrRv43+edgMyM8uPA75IHDYh14DMfeLNLF8O+CIhiRluQGZmveCALyLnuQGZmfVCmtdkbZI0\nV9ILkp6XdGlaY5WLUXXVnHHI3tz51HK2tHdkXY6ZFbg09+Dbgcsj4hDgaOASSYekOF5ZmN7cyDsb\nt3L/wpVZl2JmBS7Na7KuiIj5yf11wEJgQlrjlYvOBmQ3+2pPZrYbAzIHL2kiMBV4vIfHZklqkdTS\n2to6EOUUte0NyH7Tyhtr3IDMzHYu9YCXVA/cDnwuIn7r9I+ImB0RzRHR3NDQkHY5JWH6kU1USnzk\nmke58yn3qDGznqUa8JIGkQv3GyLijjTHKif7jB7CTbOOpmFoDZ+/eQEf+f6jzH/tnazLMrMCk+ZZ\nNAJ+ACyMiKvSGqdcNU8cxX9fchxXTp/MG2s28ZFrHuXSm57ytI2ZbaeIdD7eSzoeeBh4Fui8qOhf\nRsQ9O3tNc3NztLS0pFJPKduwpZ1/f3ARsx9ajASzTngfnzpxf4ZUV2VdmpmlTNK8iGju8bG0An5P\nOOD7Zvk7G/nWvS/x8wVvMHZYDV8682DOmzKBigplXZqZpWRXAe9vspaQxpFDuPqiqdz2qWMYO2ww\nl92ygPOv+RXzlrqPvFk5csCXoOaJo/ivvziO70yfzJtrN3PB9x/jMzc+xfJ3NmZdmpkNIAd8iaqo\nEBcc2cicy0/is6ccwH3Pv8mp33mQ79z3Ehu2tGddnpkNAAd8iaurqeKyMw5izhUnceahe3P1nFc4\n+coHuG2ez583K3UO+DIxYUQt/3LhVG6/+FjGjajlilsXcN41v+LJJZ6fNytVDvgyc+S+I7nz4mP5\n55mTWbl2C9P//TEu+dl8lr3t+XmzUuOAL0MVFeL8qY3MueJELj11EvcvfItTr3qQb//iRdZ7ft6s\nZDjgy9iQ6io+f/qBzLn8JM4+dG/+be4iTr7yAW5pWeb5ebMS4IA3xo+o5bsXTuWOvziWCSNq+eJt\nz3Duvz3CE696ft6smDngbbsj9hnJnX9xLP9y4RRWr29jxn88xiU3eH7erFg54O09JPF7UyYw5/KT\n+PxpBzLnxZWcetWDfOtez8+bFRsHvPWotrqSS0+bxJwrTuScw8bx/QcWcdK3H+CWJ5fR4fl5s6Lg\ngLddGje8lqtmTuG/LjmOfUbV8sXbn+Hcf32EXy9enXVpZrYbDnjLy5SmEdx+8bF876KpvLOhjQtn\n/5qLfzqP11Z7ft6sULlhuOVNEudOHs/p7x/LdQ8v5poHFnH/wpX8yfH7ccnJ72Po4EFZl2hmXXgP\n3nqttrqSz5w6iblXnMSHJ4/n3x/MnT9/0xOveX7erICkecm+H0paKem5tMawbO09fDDfmTGZ/77k\nOCaOruPLdzzLOVc/wmOLPD9vVgjS3IP/MXBmiu9vBWJy0whu/dQxXH3RVNZu2spF1/6aP7++haWr\nN2RdmllZSy3gI+IhwF+FLBOS+PDk8dx/+YlcccaBPPybVZx+1UP8wz0LWbd5a9blmZUlz8Fbvxo8\nqJJPn5Kbnz93ynj+46HFnHzlA9zo+XmzAZd5wEuaJalFUktra2vW5Vg/GTtsMFdOn8zPP308+42p\n4yt3PMuHvvcwj76yKuvSzMpG5gEfEbMjojkimhsaGrIux/rZYY3DueXPj+Gajx3B+i3t/P51j/Nn\nP2nh1VWenzdLW+YBb6VPEmcfNo7/vexEvvDBg3j0lVWc8c8P8s17FrLW8/NmqUnzNMkbgceAgyQt\nl/TJtMay4jB4UCWXnHwAc684ifOnTuDahxdz8rcf4IbHl9LesS3r8sxKjiIK58BXc3NztLS0ZF2G\nDZDnXn+Xr931Ak+8+jYH7z2Uvz7nEI47YEzWZZkVFUnzIqK5p8c8RWOZOXTCcG6edTTf/9gRbGhr\n52PXPc6f/qfn5836iwPeMiWJsw4bxy8/fyJfOvNgHluUm5//+l0v8O4mz8+b9YUD3grC4EGVXHzS\n+5j7hZO44IhGfvCrVzn5yge4/teenzfbUw54Kyh7DR3MP15wOHd95ngm7VXPX//Xc5z9vYd5+Df+\njoRZb/kgqxWsiOAXz7/FN+9ZyGtvb+TEAxs4ZPwwBldVUltdweBBlQweVEltl5+11RXUVFVSW/3e\n5TVVFVRUKOtVKkkRQVvHNja3bWPj1nY2tnWwqa2DTVs7utzvsrytg41be7ifPGdbwMghgxhVV83o\numpG1dUkP6sZVZ9bNrKumqE1VUj+b7qrg6zuB28FSxJnHro3Jx/cwI9/tYTrHnmVxxatpm0Pp2xq\nqireE/y5W0Vuw9BlWW11RbIRqXzPRqRz+eDqyu2P13Z5j87lgypVUMHT3wHc9bWbk8d724ZiUKWS\nv2klQ6qrtt+vT0J79fo2fvPWelZv2MLmrT3/966urGBk3aD3bgA6NwqdG4Ih1Yyuz20kRtQOKruN\nvPfgrei0d2xjc/s2NifBs6W9g01t29i0tSO3LPm5eXswbdv+e+fjXZdtautgc3vyM1m+KbntyT+P\nygptD/7unzBqOjcoXTYS79nQJMsHJxuPQZVi89ZtBRPAQ6oru92vora6Io/nJPe3r1f+s8Ob2jpY\nvWELb29oY/WGNt5e37bjftflyWPrdnJx+ArByCFdNgT1nfdrtn8qGN1lIzGyrrpXdWbFe/BWUqoq\nK6ivrKC+Jt3/fbvu+XZuAH57I7Kt2wYiWdbDhmbz1m2s29xO67otXd4j99y29t59KtnVHnBDfc2A\nBnDaaqsraaweQuPIIXk9f0t7B2s2bmX19g1BbiPQfQPx0pvreHtDG2s2bd3phnzY4CpG19e859NB\n1w1BblnN9k8MgwdV9uOa950D3mwnJFFTVUlNVSXDSfdyhB3bIvkksiP4N2/tYEv7tveGcQEGcKGp\nqapk7LBKxg4bnNfzO7YFaza2veeTwI4NwZbty5a9vZGnl63hnQ1ttO/kE9GQ6sr3bAh2dRxhVF31\n9imptDjgzQpAZYUYUl3FkGr/kxxolRVidH0No+trmJTH8yOCtZvbk08FW7p8Umh7zyeFVeu38PKb\n61i9oY0tO/mE1nkcYZ9RQ7j1U8f274rhgDcz6xVJDK8dxPDaQew3pm63z48INm3t2L4h6Hr8oPOT\nQmVKB38d8GZmKZKST2ejqmgald9xhP7iiTwzsxLlgDczK1EOeDOzEuWANzMrUQ54M7MS5YA3MytR\nDngzsxLlgDczK1EF1U1SUiuwdA9fPgZY1Y/lFAOvc+krt/UFr3Nv7RsRDT09UFAB3xeSWnbWMrNU\neZ1LX7mtL3id+5OnaMzMSpQD3sysRJVSwM/OuoAMeJ1LX7mtL3id+03JzMGbmdl7ldIevJmZdeGA\nNzMrUUUf8JLOlPSSpFckfTnregaCpB9KWinpuaxrGQiSmiTNlfSCpOclXZp1TWmTNFjSE5IWJOv8\n1axrGiiSKiU9JemurGsZCJKWSHpW0tOSWvr1vYt5Dl5SJfAycDqwHHgSuCgiXsi0sJRJOgFYD/wk\nIg7Nup60SRoHjIuI+ZKGAvOA80r5v7NyV2Kui4j1kgYBjwCXRsSvMy4tdZIuA5qBYRFxTtb1pE3S\nEqA5Ivr9y13Fvgd/FPBKRCyOiDbgJuD3Mq4pdRHxEPB21nUMlIhYERHzk/vrgIXAhGyrSlfkrE9+\nHZTcindvLE+SGoEPAddlXUspKPaAnwAs6/L7ckr8H365kzQRmAo8nm0l6UumKp4GVgK/jIiSX2fg\nu8AXgW1ZFzKAArhP0jxJs/rzjYs94K2MSKoHbgc+FxFrs64nbRHRERFTgEbgKEklPR0n6RxgZUTM\ny7qWAXZ8RBwBnAVckkzB9otiD/jXgaYuvzcmy6zEJPPQtwM3RMQdWdczkCJiDTAXODPrWlJ2HHBu\nMid9E3CKpJ9mW1L6IuL15OdK4E5yU8/9otgD/klgkqT9JFUDFwL/k3FN1s+SA44/ABZGxFVZ1zMQ\nJDVIGpHcryV3IsGL2VaVroj4SkQ0RsREcv+W50TExzMuK1WS6pITB5BUB5wB9NvZcUUd8BHRDnwa\n+AW5A2+3RMTz2VaVPkk3Ao8BB0laLumTWdeUsuOAPyC3R/d0cjs766JSNg6YK+kZcjsyv4yIsjht\nsMyMBR6RtAB4Arg7Iu7trzcv6tMkzcxs54p6D97MzHbOAW9mVqIc8GZmJcoBb2ZWohzwZmYlygFv\nZUVSR5dTLZ/uzw6kkiaWS4dPKw5VWRdgNsA2JV//Nyt53oM3Y3tP7n9K+nI/IemAZPlESXMkPSPp\nfkn7JMvHSroz6de+QNKxyXG8BGgAAAEuSURBVFtVSro26eF+X/ItVLNMOOCt3NR2m6KZ2eWxdyPi\nMOBfyXU1BLga+M+IOBy4Afhesvx7wIMRMRk4Auj8BvUk4N8i4gPAGuCClNfHbKf8TVYrK5LWR0R9\nD8uXAKdExOKksdmbETFa0ipyFxvZmixfERFjJLUCjRGxpct7TCTXUmBS8vuXgEER8fX018zst3kP\n3myH2Mn93tjS5X4HPs5lGXLAm+0ws8vPx5L7j5LrbAjwMeDh5P79wMWw/cIcwweqSLN8ee/Cyk1t\ncpWkTvdGROepkiOT7o1bgIuSZZ8BfiTpC0Ar8MfJ8kuB2Uknzw5yYb8i9erNesFz8Gake+Fjs6x4\nisbMrER5D97MrER5D97MrEQ54M3MSpQD3sysRDngzcxKlAPezKxE/R/A1BkrI475LgAAAABJRU5E\nrkJggg==\n", 939 | "text/plain": [ 940 | "
" 941 | ] 942 | }, 943 | "metadata": { 944 | "tags": [] 945 | } 946 | } 947 | ] 948 | }, 949 | { 950 | "cell_type": "code", 951 | "metadata": { 952 | "id": "ZN-9LJJkG3Xf", 953 | "colab_type": "code", 954 | "outputId": "e1c83714-0373-4456-8cd7-134f240ac796", 955 | "colab": { 956 | "base_uri": "https://localhost:8080/", 957 | "height": 692 958 | } 959 | }, 960 | "source": [ 961 | "# Train on federated model\n", 962 | "print('========== Train on Local Device of Client 3 ==========')\n", 963 | "\n", 964 | "history_accuracy_3 = []\n", 965 | "history_loss_3 = []\n", 966 | "\n", 967 | "for round_num in range(6):\n", 968 | " train_state, train_metrics = trainer.next(train_state, federated_data_client_3)\n", 969 | " history_accuracy_3.append(train_metrics.sparse_categorical_accuracy)\n", 970 | " history_loss_3.append(train_metrics.loss)\n", 971 | " print('round {:2d}, metrics={}'.format(round_num, train_metrics))\n", 972 | "\n", 973 | "# Show accuracy diagram\n", 974 | "plt.title('Model Accuracy for Client 3')\n", 975 | "plt.plot(history_accuracy_3, label='accuracy')\n", 976 | "plt.xlabel('Epoch')\n", 977 | "plt.ylabel('Accuracy')\n", 978 | "plt.show()\n", 979 | "\n", 980 | "# Show loss diagram\n", 981 | "plt.title('Model Loss for Client 3')\n", 982 | "plt.plot(history_loss_3, label='loss')\n", 983 | "plt.xlabel('Epoch')\n", 984 | "plt.ylabel('Loss')\n", 985 | "plt.show()" 986 | ], 987 | "execution_count": 0, 988 | "outputs": [ 989 | { 990 | "output_type": "stream", 991 | "text": [ 992 | "========== Train on Local Device of Client 3 ==========\n", 993 | "round 0, metrics=\n", 994 | "round 1, metrics=\n", 995 | "round 2, metrics=\n", 996 | "round 3, metrics=\n", 997 | "round 4, metrics=\n", 998 | "round 5, metrics=\n" 999 | ], 1000 | "name": "stdout" 1001 | }, 1002 | { 1003 | "output_type": "display_data", 1004 | "data": { 1005 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9fX/8dchgEH2TZQ1KGBFkC2A\noFXrVlxx39kVtS60Vq221lqtfltba2tFKwphUxEULHXfN0BIwqaAYIAACVvY95Dl/P6YS39jDGEg\nmUySeT8fj3lw93vuDJkzn8+991xzd0REJH5Vi3UAIiISW0oEIiJxTolARCTOKRGIiMQ5JQIRkTin\nRCAiEueUCCRqzCzJzNzMqkew7GAz+6o84qrozOx2M9tgZrvMrHE57O8RM5sYDLcO9psQ7f1KxaFE\nIACYWaaZ7TezJkWmzwu+zJNiE9kPYqkTfEm9G+tYosXMagB/B8539zruvrmMtnuDmaUF7986M3vX\nzE4vupy7rw72W1AG+/zMzG4uYX4TM5thZpvNbJuZzTKz00q7Xzl8SgQSbiVw/YERM+sMHB27cH7k\nSiAXOM/Mji3PHUfSqikjzYBEYNHhrmghP/qbNrN7gH8ATwTbbw08B/QvXailtgsYCjQFGgJ/Af5b\nju+1BJQIJNwEYGDY+CBgfPgCZlbfzMabWY6ZrTKzhw58+ZhZgpn9zcw2mdkK4KJi1h0d/CLNNrM/\nHWYXxCDg38BC4KYi225lZlODuDab2bNh824xsyVmttPMFptZ92C6m1m7sOXGmtmfguGzzCzLzH5j\nZuuBFDNraGZvBfvYGgy3DFu/kZmlmNnaYP6bwfRvzeySsOVqBO9RtyLH0AFYGoxuM7NPgul9zSzV\nzLYH//YNW+czM3vczGYAe4Dji77nwKPAHe4+1d13u3ueu//X3e8r+gYX7c4r6TM70J0XfOZbzWyl\nmV0QzHsc+CnwbNAKebbovtx9n7svdfdCwIACQgmhUdFlJbqUCCTc10A9Mzsp+GO/DphYZJl/AfUJ\nfeGcSShxDAnm3QJcDHQDkoGriqw7FsgH2gXLnA8ctOsgnJm1Ac4CXg5eA8PmJQBvAauAJKAFMCmY\ndzXwSLB8PeBSINLulmMJfSm1AYYT+ntJCcZbA3uB8C+4CYRaUCcDxwBPB9PH88PEdSGwzt3nhe/M\n3ZcF6wI0cPezzawR8DbwDNCYULfR20XOHQwI4qsbvAfh+hBqYUyL8JiLGkvJn1lvQsmrCfAkMNrM\nzN1/B3wJ3Bl0Nd15sB2Y2UJgHzAdeMndNx5hrHKk3F0vvQAygXOBh4D/A/oBHwLVASf0BZsA7Ac6\nhq13K/BZMPwJcFvYvPODdasT6pLIBWqFzb8e+DQYHgx8VUJ8DwHzg+EWhH49dgvG+wA5QPVi1nsf\nGHGQbTrQLmx8LPCnYPis4FgTS4ipK7A1GD4OKAQaFrNcc2AnUC8Yfx24/yDbTDrwngXjA4A5RZaZ\nBQwOhj8DHi0hxhuB9Yf47B8BJhbdf4SfWUbYvKODdY8Ni+3mCP//JQbbHhTrv4V4fKkvToqaAHwB\ntKVItxChX301+OGvzlWEvpgh9IW3psi8A9oE664zswPTqhVZviQDgRcB3D3bzD4n1FU0D2gFrHL3\n/GLWawUsj3AfReW4+74DI2Z2NKFf+f0IdWEA1A1aJK2ALe6+tehG3H1t0HVzpZlNAy4ARkQYQ3N+\n/Cs//D2Hkt/DzUATM6t+kPenJJF8ZusPDLj7nmC5Ooe5H4L3+dWgC2++uy843G3IkVPXkPyAu68i\ndNL4QmBqkdmbgDxCXxAHtAayg+F1hL4Qw+cdsIbQr8sm7t4geNVz95M5hKBPvD3woJmtD/rsewM3\nBH3Za4DWBznJuAY44SCb3sMPT4YXPQFdtDTvr4ETgd7uXg8440CIwX4amVmDg+xrHKHuoauBWe6e\nfZDlilrLD99v+OF7Xlyc4WYRet8vi3B/4Y74M4sgroOpQZHzHBJ9SgRSnGHA2e6+O3yihy4pnAw8\nbmZ1g377e/j/5xEmA3ebWUszawg8ELbuOuAD4Ckzq2dm1czsBDM7M4J4BhHqpupIqDumK9AJqEXo\n1/UcQknoz2ZW28wSwy5DfAm418x6BFfVtAviBphPKJkkmFk/Quc8SlKX0HmBbUHf/R+KHN+7wHPB\nSeUaZnZG2LpvAt0JtQSKtrRK8g7QwUKXf1Y3s2uD9+GtSFZ29+3Aw8BIM7vMzI4OYrvAzJ48xLql\n+cwANlDCl7qZnWpmp5tZTTOrZWa/IdQdNTvC7UsZUSKQH3H35e6edpDZdwG7gRXAV8ArwJhg3ouE\n+uQXAHP5cYtiIFATWAxsJdRXflxJsZhZInAN8C93Xx/2WkmoG2tQkKAuIXRCczWQBVwbHMsU4PEg\nzp2EvpAPXJUyIlhvG6G+9DdLioXQJZi1CLWMvgbeKzJ/AKEW03fARuCXB2a4+17gDUJdbkXfl4Py\n0H0EFxNqjWwG7gcudvdNh7GNpwgl7IcInUtZA9zJoY8XjuAzC/NP4KrgiqJnipl/FDCS0HFlE2qF\nXuTuayPcvpQRc9eDaUTKg5k9DHRw95sOubBIOdLJYpFyEHQlDSPUahCpUNQ1JBJlZnYLoe6Yd939\ni1jHI1KUuoZEROKcWgQiInGu0p0jaNKkiSclJcU6DBGRSiU9PX2Tuzctbl6lSwRJSUmkpR3sykYR\nESmOmRW9Q/1/1DUkIhLnlAhEROKcEoGISJxTIhARiXNKBCIicS5qicDMxpjZRjP79iDzzcyeMbMM\nM1toweMDRUSkfEWzRTCW0AM8DuYCQjXm2xN6zN7zUYxFREQOImqJIKipsqWERfoD4z3ka6CBmUVa\n3lZEJG7kFRTyxDtLWLttb1S2H8tzBC344SPvsvjh4/f+x8yGm1mamaXl5OSUS3AiIhXBjn15DElJ\nZdQXK/jku41R2UelOFns7qPcPdndk5s2LfYOaRGRKid7216ufn4WX6/YzF+vOoWbTi361NKyEcsS\nE9n88Pm2Lfnhc1hFROLWN1nbGToulX15BYwb2ovT2jWJ2r5i2SKYDgwMrh46FdgePCNVRCSufbxk\nA9e8MIuaCdV44/a+UU0CEMUWgZm9CpwFNDGzLEIP+q4B4O7/JvRQ7guBDGAPMCRasYiIVBbjZ2Xy\nyPRFdGpRn5cGJXNM3cSo7zNqicDdrz/EfAfuiNb+RUQqk4JC54l3ljD6q5Wce1Iznrm+K0fXLJ/e\n+0pXhlpEpKrZu7+AX742j/cXbWBw3yR+f3FHEqpZue1fiUBEJIZyduZy8/g0FmZt4+GLOzL09Lbl\nHoMSgYhIjGRs3MnglFQ27crlhZt6cP7Jx8YkDiUCEZEYmLV8M7dOSKNm9QReG96HLq0axCwWJQIR\nkXI2bV4W97++kDaNa5MyuCetGh0d03iUCEREyom788zHGTz90TL6HN+Yfw/oQf1aNWIdlhKBiEh5\n2J9fyINTv+GNuVlc0b0Ff77iFGpWrxhVfpQIRESibPvePG6fmM7M5Zv51bkduPucdpiV3+Whh6JE\nICISRWu27GHo2FQyN+/m79d04YruLWMd0o8oEYiIRMnCrG0MHZvG/vwCxg/tTZ8TGsc6pGIpEYiI\nRMEHi9YzYtJ8GtepyaThvWl3TN1Yh3RQSgQiImVszFcreeztxZzSsgEvDUymad2jYh1SiZQIRETK\nSEGh89hbixk7M5Ofn9yMf1zbjVo1E2Id1iEpEYiIlIE9+/MZMWk+Hy7ewLDT2/LbC08q18JxpaFE\nICJSSht37uPmcWl8m72dP156MoP6JsU6pMOiRCAiUgrLNuxkSEoqW3bvZ9SAZM7t2CzWIR02JQIR\nkSM0M2MTt05MJ7FGApNv7UPnlvVjHdIRUSIQETkCr6dn8cAbCzm+aW1ShvSiRYNasQ7piCkRiIgc\nBnfn6Y++55mPv+f0dk147qbu1EuMfeG40lAiEBGJUG5+AQ++8Q1T52VzdY+WPHFFZ2okVIzCcaWh\nRCAiEoHte/IYPiGN2Su3cO/5HbjjZxWrcFxpKBGIiBzCmi17GJwyhzVb9vLP67rSv2uLWIdUppQI\nRERKMG/1Vm4Zn0ZegTNhWC96H18xC8eVhhKBiMhBvPftOkZMmk+zeomkDOnJCU3rxDqkqFAiEBEp\nwt0Z/dVKHn9nCV1bhQrHNa5TsQvHlYYSgYhImPyCQh59azHjZ63igk7H8vS1XUmsUfELx5WGEoGI\nSGB3bj53vzqPj7/byPAzjueBfj+hWiUpHFcaSgQiIsCGHfsYOjaVJet28NhlnRhwaptYh1RulAhE\nJO59t34HQ1NS2bY3j9GDevKznxwT65DKlRKBiMS1L7/P4RcT51KrZqhwXKcWlbNwXGkoEYhI3Jqc\nuobfTvuGdsfUYczgnjSvxIXjSiOqRTLMrJ+ZLTWzDDN7oJj5rc3sUzObZ2YLzezCaMYjIgKhy0P/\n9v5S7n9jIX1OaMyU2/rEbRKAKLYIzCwBGAmcB2QBqWY23d0Xhy32EDDZ3Z83s47AO0BStGISEcnN\nL+C+KQuZvmAt1/VsxWOXdaoSheNKI5pdQ72ADHdfAWBmk4D+QHgicKBeMFwfWBvFeEQkzm3dvZ9b\nJ6QzJ3ML9/c7kdvPPKHKFI4rjWgmghbAmrDxLKB3kWUeAT4ws7uA2sC5xW3IzIYDwwFat25d5oGK\nSNW3avNuhqSkkrV1L89c341LuzSPdUgVRqzbQ9cDY929JXAhMMHMfhSTu49y92R3T27atGm5Byki\nlVv6qq1c/txMtuzZz8u39FYSKCKaLYJsoFXYeMtgWrhhQD8Ad59lZolAE2BjFOMSkTjyzjfr+NVr\n8zm2fiJjh/SibZPasQ6pwolmiyAVaG9mbc2sJnAdML3IMquBcwDM7CQgEciJYkwiEifcnRc+X84v\nXp5Lpxb1mXp7XyWBg4hai8Dd883sTuB9IAEY4+6LzOxRIM3dpwO/Bl40s18ROnE82N09WjGJSHzI\nLyjkD9MX8fLs1Vx0ynE8dXWXKl84rjSiekOZu79D6JLQ8GkPhw0vBk6LZgwiEl925eZz1ytz+XRp\nDredeQL3//zEuCgcVxq6s1hEqoz120OF45Zu2MkTl3fmht66yjASSgQiUiUsWbeDISmp7NyXx+hB\nyZx1YnwVjisNJQIRqfQ+X5bDHS/Ppc5R1ZlyW186Nq936JXkf5QIRKRSe2X2an7/n2/p0KwuYwYn\nc1z9+K0ZdKSUCESkUiosdP76wVKe/2w5Z3Zoysgbu1PnKH2lHQm9ayJS6ezLK+DeKQt4a+E6bujd\nmkcvPZnqcV44rjSUCESk0igodD5asoFnP8ngm+ztPHjBTxh+xvEqHFdKSgQiUuFt35vHlLQ1jJuV\nyZote2leP5Hnb+zOBZ2Pi3VoVYISgYhUWMtzdjFuZiavp2exZ38BPZMa8uAFJ3F+x2bqCipDSgQi\nUqEUFjpffJ/D2JmZfLY0h5oJ1bi4y3EMPa1tXD5PuDwoEYhIhbA7N5+pc7NImZnJipzdNK17FL86\ntwM39G5N07pHxTq8Kk2JQERias2WPYyflcmk1DXs3JfPKS3r8/S1Xbioc3NqVlf3T3lQIhCRcufu\nfL1iCykzVvLRkg2YGRd0OpYhpyXRvXVDXQVUzpQIRKTc7MsrYPr8taTMzGTJuh00PLoGt515AgP6\ntNEdwTGkRCAiUbd++z4mfr2KV+asZsvu/ZzYrC5/vqIzl3VroecEVABKBCISNXNXbyVlRibvfrOO\nAnfOPakZQ05Los/xjdX9U4EoEYhImdqfX8i7365jzIxMFqzZRt2jqjOobxKD+iTRuvHRsQ5PiqFE\nICJlYtOuXF6dvZoJX69i485c2japzR8vPZkre7RUMbgKTp+OiJTKorXbSZmRyfQFa9mfX8gZHZry\nl6uSOLN9Uz0ispJQIhCRw5ZfUMhHSzYwZkYmc1ZuoVaNBK5Jbsngvkm0O6ZurMOTw6REICIR274n\nj0mpqxk/axXZ2/bSokEtfnvhT7g2uTX1j64R6/DkCCkRiMghZWzcScqMTKbOzWZvXgG92zbi9xd3\n5LyOzUhQ90+lp0QgIsUqLHQ+X5bDmBkr+fL7TdSsXo3+XZoz+LQkTm6u4m9ViRKBiPzArtx83kjP\nYtzMTFZs2k2zekdx7/kduL5XaxrXUfG3qkiJQEQAWL15D2NnZjIlbQ07c/Pp2qoB/7yuKxd0Ok7F\n36o4JQKROObuzFq+mTEzMvn4uw0kmHFh5+MYcloS3Vo3jHV4Uk6UCETi0L68AqbNy2bsjEyWbthJ\no9o1ufNn7bixdxuOrZ8Y6/CknCkRiMSRddv3Mn7WKl6ds5pte/I46bh6PHnVKVzapbmKv8UxJQKR\nKs7dmbt6K2NmZPLet+txd87r2Iwhp7Wld9tGKv4mSgQiVVVufgFvL1zH2JmZLMzaTr3E6gw7vS0D\nTm1Dq0Yq/ib/X1QTgZn1A/4JJAAvufufi1nmGuARwIEF7n5DNGMSqepyduby8uxVTPx6NZt25XJC\n09o8dlknrujWgtoq/ibFiNr/CjNLAEYC5wFZQKqZTXf3xWHLtAceBE5z961mdky04hGp6r7N3s6Y\nGSt5a8E69hcU8rMTmzL4tLb8tF0TFX+TEkXz50EvIMPdVwCY2SSgP7A4bJlbgJHuvhXA3TdGMR6J\nA99mb2f2yi2xDqNcFRQW8uHiDaRmbuXomglc36sVg/omcXzTOrEOTSqJQyYCM7sLmHjgy/owtADW\nhI1nAb2LLNMh2McMQt1Hj7j7e8XEMBwYDtC6devDDEPixZvzsrnv9QXkFXisQyl3rRrV4qGLTuKa\nnq2ol6jib3J4ImkRNCPUrTMXGAO87+5l9ZdWHWgPnAW0BL4ws87uvi18IXcfBYwCSE5Ojr+/cimR\nu/PsJxk89eEyTj2+Ef+4thu1asbXpZB1j6qu7h85YodMBO7+kJn9HjgfGAI8a2aTgdHuvryEVbOB\nVmHjLYNp4bKA2e6eB6w0s2WEEkPqYRyDxLG8gkJ+O/UbpqRncUW3Fvz5ylNUDkHkMEX0FxO0ANYH\nr3ygIfC6mT1ZwmqpQHsza2tmNYHrgOlFlnmTUGsAM2tCqKtoxeEcgMSv7XvzGJwyhynpWYw4pz1P\nXdNFSUDkCERyjmAEMBDYBLwE3OfueWZWDfgeuL+49dw938zuBN4n1P8/xt0XmdmjQJq7Tw/mnW9m\ni4GCYNuby+LApGrL2rqHoWNTWZGzm79d3YWrerSMdUgilVYk5wgaAVe4+6rwie5eaGYXl7Siu78D\nvFNk2sNhww7cE7xEIrIwaxvDxqWxL6+A8UN70bddk1iHJFKpRdKOfhf43/V4ZlbPzHoDuPuSaAUm\nUpyPFm/g2he+pmZCNabe3ldJQKQMRJIIngd2hY3vCqaJlKuxM1YyfEIa7ZvVYdodfWnfTA9JFykL\nkXQNWfjlokGXkO5Tl3JTUOg8/vYSxsxYyXkdm/HP67pydE39FxQpK5G0CFaY2d1mViN4jUBX9kg5\n2bu/gNsnpjNmxkqGntaWf9/UQ0lApIxFkghuA/oSugfgwN3Bw6MZlAiEiqddN2oWHy7ZwB8u6cjD\nl3QkQTdNiZS5SG4o20joHgCRcpOxcSeDU1LZvGs/owYkc17HZrEOSaTKiuQ+gkRgGHAy8L9n2Ln7\n0CjGJXFs5vJN3DYhnZrVE3jt1lM5pWWDWIckUqVF0jU0ATgW+DnwOaFSETujGZTErzfSsxg0Zg7N\n6iUy7Rd9lQREykEkiaCdu/8e2O3u44CL+HEVUZFScXf+8dEyfj1lAT2TGvH67X31FC2RchLJ5Rd5\nwb/bzKwToXpDeoCMlJn9+YU8MHUhU+dmc1WPljxxeWfVDBIpR5EkglFm1hB4iFDRuDrA76MalcSN\n7XvyuG1iOrNWbOae8zpw19nt9DB1kXJWYiIICsvtCB5K8wVwfLlEJXFhzZY9DBmbyqrNu3n62i5c\n3k2F40RiocT2t7sXcpDqoiKlMX/NNi5/bgYbd+xj/NDeSgIiMRRJR+xHZnavmbUys0YHXlGPTKqs\n9xet57pRs6hVM4Gpv+hLnxMaxzokkbgWyTmCa4N/7wib5qibSI7AmK9W8tjbizmlZQNGD0qmSZ2j\nYh2SSNyL5M7ituURiFRtBYXOY28tZuzMTH5+crO4fK6wSEUVyZ3FA4ub7u7jyz4cqYr27M/n7lfn\n89GSDdx8elsevPAk1QwSqUAi6RrqGTacCJwDzAWUCOSQNu7cx7CxaSxau51H+5/MwD5JsQ5JRIqI\npGvorvBxM2sATIpaRFJlLNuwkyEpqWzZvZ8XByZzzkkqHCdSER1JYffdgM4bSIlmZGzitonpJNZI\nYPKtfejcsn6sQxKRg4jkHMF/CV0lBKHLTTsCk6MZlFRuU9LW8ODUbzi+aW1ShvSiRYNasQ5JREoQ\nSYvgb2HD+cAqd8+KUjxSibk7T3+4jGc+yeD0dk147qbu1EusEeuwROQQIkkEq4F17r4PwMxqmVmS\nu2dGNTKpVHLzC3jgjW+YNi+ba5Jb8vjlnamRoMJxIpVBJH+pU4DCsPGCYJoIECocN3D0HKbNy+be\n8zvwlytPURIQqUQiaRFUd/f9B0bcfb+Z1YxiTFKJrN68h8Fj55C1ZS//vK4r/bu2iHVIInKYIvnZ\nlmNmlx4YMbP+wKbohSSVxbzVW7n8uRls3rWfiTf3VhIQqaQiaRHcBrxsZs8G41lAsXcbS/x479t1\njJg0n2b1EkkZ0pMTmtaJdUgicoQiuaFsOXCqmdUJxndFPSqpsNyd0V+t5PF3ltC1VQNeGphMYxWO\nE6nUDtk1ZGZPmFkDd9/l7rvMrKGZ/ak8gpOKJb+gkIf/s4g/vb2Eficfy6u3nKokIFIFRHKO4AJ3\n33ZgJHha2YXRC0kqot25+QyfkM6Er1dx6xnHM/KG7iTWUPVQkaogknMECWZ2lLvnQug+AkA/A+PI\nhh37GDo2lSXrdvDYZZ0YcGqbWIckImUokhbBy8DHZjbMzG4GPgTGRbJxM+tnZkvNLMPMHihhuSvN\nzM0sObKwpbx8t34Hl4+cwcpNuxk9qKeSgEgVFMnJ4r+Y2QLgXEI1h94HDvltYGYJwEjgPEJXGqWa\n2XR3X1xkubrACGD24Ycv0fTl9zncPnEutY8KFY7r1EKF40Sqokhv/9xAKAlcDZwNLIlgnV5Ahruv\nCG5ImwT0L2a5x4C/APsijEXKwWupqxmSkkrLhrWY9ovTlAREqrCDtgjMrANwffDaBLwGmLv/LMJt\ntwDWhI1nAb2L7KM70Mrd3zaz+0qIZTgwHKB169YR7l6OhLvz1AfLePbTDM7o0JSRN3SjrgrHiVRp\nJXUNfQd8CVzs7hkAZvarstqxmVUD/g4MPtSy7j4KGAWQnJzsh1hcjlBufgH3TVnI9AVrua5nKx67\nrJNqBonEgZISwRXAdcCnZvYeoa6dw3nQbDbQKmy8ZTDtgLpAJ+AzMwM4FphuZpe6e9ph7EfKwNbd\n+7l1QjpzMrdwf78Tuf3MEwg+FxGp4g6aCNz9TeBNM6tNqG//l8AxZvY8MM3dPzjEtlOB9mbWllAC\nuA64IWz724EmB8bN7DPgXiWB8rdq824Gp6SSvW0v/7q+G5d0aR7rkESkHB2y3e/uu939FXe/hNCv\n+nnAbyJYLx+4k9BVRkuAye6+yMweDS9iJ7GVvmorlz83k6179vPyzb2VBETikLlXri735ORkT0tT\no6EsvL1wHb+aPJ/m9RNJGdKLtk1qxzokEYkSM0t392Lv1TqSh9dLJefujPpiBf/37nf0aNOQFwcm\n06i2HjEhEq+UCOJMfkEhf5i+iJdnr+aiU47jqau7qGaQSJxTIogju3LzufOVuXy2NIfbzjyB+39+\nItWq6cogkXinRBAn1m8PFY5bumEnT1zemRt668Y8EQlRIogDi9fuYOjYVHbuy2P0oGTOOvGYWIck\nIhWIEkEV9/myHH4xMZ26iTWYcltfOjavF+uQRKSCUSKowqbOzeK+1xfSoVldxgxO5rj6tWIdkohU\nQEoEVdR7367n3ikLOPX4xowamEydo/RRi0jx9O1QBc1cvom7X51Hl1YNeHFgMrWVBESkBCotWcUs\nzNrGLePSSGpyNCmDeyoJiMghKRFUIRkbdzE4JZWGtWsyfmhvGhytu4VF5NCUCKqI7G17GTh6NtUM\nJgzrzbH1E2MdkohUEkoEVcDmXbkMGD2bnfvyGavicSJymNSBXMntys1nyNhUsrfuZfzQXnq2sIgc\nNiWCSmxfXgHDx6exaO0OXripB72PbxzrkESkElLXUCWVX1DIiEnzmLl8M3+96hTO7dgs1iGJSCWl\nRFAJuTu/m/Yt7y/awMMXd+SK7i1jHZKIVGJKBJXQn9/7jtfS1nD32e0YenrbWIcjIpWcEkEl8+/P\nl/PC5ysYcGobfnVeh1iHIyJVgBJBJTJpzmr+/O53XNKlOX+89GTM9FAZESk9JYJK4r1v1/Hbad9w\nZoemPHV1Fz1ZTETKjBJBJTAjYxN3vzqfrq0a8PxN3alZXR+biJQdfaNUcAvWbGP4+DTaNqnNmME9\nObqmbv0QkbKlRFCBZWzcyeCUOTSqU5Pxw3qpiJyIRIUSQQWVvW0vA0bPIaFaNSYM7U2zeioiJyLR\noURQAW3elcuAl2azKzef8UN7kaQiciISRUoEFczOfXkMSpnD2u17GTO4px42LyJRp0RQgezLK+CW\n8Wl8t24nz9/Yg55JjWIdkojEAV2CUkHkFxRy16vz+HrFFv5xbVd+9pNjYh2SiMQJtQgqAHfnganf\n8OHiDTxySUcu69Yi1iGJSByJaiIws35mttTMMszsgWLm32Nmi81soZl9bGZtohlPReTuPPHOEl5P\nz2LEOe0ZfJqKyIlI+YpaIjCzBGAkcAHQEbjezDoWWWwekOzupwCvA09GK56K6vnPl/PilysZ1KcN\nvzy3fazDEZE4FM0WQS8gw91XuPt+YBLQP3wBd//U3fcEo18DcVVY/5XZq3nyvaX079qcP1yiInIi\nEhvRTAQtgDVh41nBtIMZBrxb3AwzG25maWaWlpOTU4Yhxs4736zjd29+w1knNuVvKiInIjFUIU4W\nm9lNQDLw1+Lmu/sod09292FPgdMAAAnbSURBVOSmTZuWb3BR8OX3OYyYNI8erRvy/I09qJFQIT4G\nEYlT0bx8NBtoFTbeMpj2A2Z2LvA74Ex3z41iPBXCvNVbuXVCOic0rcPowT2pVTMh1iGJSJyL5k/R\nVKC9mbU1s5rAdcD08AXMrBvwAnCpu2+MYiwVwrINOxkyNpUmdY5i/NBe1K9VI9YhiYhELxG4ez5w\nJ/A+sASY7O6LzOxRM7s0WOyvQB1gipnNN7PpB9lcpbdmyx4GjJ5NjYRqTBzWm2NURE5EKoio3lns\n7u8A7xSZ9nDY8LnR3H9FkbMzl4Fj5rB3fwGTb+tD68ZHxzokEZH/UYmJKNuxL4/BKXNYt30vL9/c\nm58cqyJyIlKx6HKVKNqXV8DN49JYun4n/76pBz3aqIiciFQ8ahFESX5BIXe+MpfUzFARubNOVBE5\nEamY1CKIgsJC5/43FvLRko08eunJ9O+qInIiUnEpEZQxd+dPby9h6txs7jmvAwP6JMU6JBGREikR\nlLGRn2YwZsZKBvdN4q6z28U6HBGRQ1IiKEMTv17F3z5YxuXdWvDwxR1VRE5EKgUlgjLy3wVr+f1/\nvuWcnxzDk1edoiJyIlJpKBGUgc+X5XDP5Pn0bNOIkTd2VxE5EalU9I1VSumrtnLbhHTaHVOXFwcl\nk1hDReREpHJRIiiFpet3MnRsKs3qqYiciFReSgRH6EARucQa1ZgwrDdN6x4V65BERI6I7iw+Ajk7\nc7lp9Gxy8wuZfGsfWjVSETkRqbzUIjhM2/fmMXDMHDbuyGXM4J6ceGzdWIckIlIqSgSHYe/+Am4e\nl0rGxp38e0APerRpGOuQRERKTV1DEcorKOSOV+aStmorz1zXjTM7VP5nJ4uIgFoEESksdO5/fSGf\nfLeRx/p34pIuzWMdkohImVEiOAR359G3FjNtXjb3nt+Bm05tE+uQRETKlBLBIfzrkwzGzsxk2Olt\nueNnKiInIlWPEkEJJszK5O8fLuPK7i353YUnqYiciFRJSgQH8Z/52Tw8fRHnntSMv1zZWUXkRKTK\nUiIoxqdLN/LryQvomdSIZ2/oRnUVkRORKkzfcEWkr9rC7RPTOfHYurykInIiEgeUCMIsWbeDISmp\nHFe/FuOG9qJeoorIiUjVp0QQWL15DwPHzOHomtWZMKwXTeqoiJyIxAclAmDjjn3cNHo2eQWFTBjW\ni5YNVUROROJH3CeC7XtCReQ27colZXBP2jdTETkRiS9xnQj27i9g2LhUlufs4oUBPejWWkXkRCT+\nxG3RubyCQm5/OZ301VsZeUN3ftpeReREJD7FZYugsNC5d8oCPluawxOXd+bCzsfFOiQRkZiJu0Tg\n7vzxv4v4z/y13N/vRK7v1TrWIYmIxFRUE4GZ9TOzpWaWYWYPFDP/KDN7LZg/28ySohkPwD8++p5x\ns1Zxy0/bcvuZJ0R7dyIiFV7UEoGZJQAjgQuAjsD1ZtaxyGLDgK3u3g54GvhLtOIBGDtjJf/8+Huu\n7tGS36qInIgIEN0WQS8gw91XuPt+YBLQv8gy/YFxwfDrwDkWpW/n/8zP5pH/Lub8js34vys6KwmI\niASimQhaAGvCxrOCacUu4+75wHagcdENmdlwM0szs7ScnJwjCubYeomc17EZz1yvInIiIuEqxeWj\n7j4KGAWQnJzsR7KN3sc3pvfxP8oxIiJxL5o/jbOBVmHjLYNpxS5jZtWB+sDmKMYkIiJFRDMRpALt\nzaytmdUErgOmF1lmOjAoGL4K+MTdj+gXv4iIHJmodQ25e76Z3Qm8DyQAY9x9kZk9CqS5+3RgNDDB\nzDKALYSShYiIlKOoniNw93eAd4pMezhseB9wdTRjEBGRkunyGRGROKdEICIS55QIRETinBKBiEic\ns8p2taaZ5QCrjnD1JsCmMgynMtAxxwcdc3wozTG3cfdiH7xS6RJBaZhZmrsnxzqO8qRjjg865vgQ\nrWNW15CISJxTIhARiXPxlghGxTqAGNAxxwcdc3yIyjHH1TkCERH5sXhrEYiISBFKBCIicS5uEoGZ\n9TOzpWaWYWYPxDqeaDOzMWa20cy+jXUs5cXMWpnZp2a22MwWmdmIWMcUbWaWaGZzzGxBcMx/jHVM\n5cHMEsxsnpm9FetYyoOZZZrZN2Y238zSynz78XCOwMwSgGXAeYQemZkKXO/ui2MaWBSZ2RnALmC8\nu3eKdTzlwcyOA45z97lmVhdIBy6r4p+zAbXdfZeZ1QC+Aka4+9cxDi2qzOweIBmo5+4XxzqeaDOz\nTCDZ3aNyA128tAh6ARnuvsLd9wOTgP4xjimq3P0LQs94iBvuvs7d5wbDO4El/Pg52VWKh+wKRmsE\nryr9687MWgIXAS/FOpaqIl4SQQtgTdh4FlX8CyLemVkS0A2YHdtIoi/oJpkPbAQ+dPeqfsz/AO4H\nCmMdSDly4AMzSzez4WW98XhJBBJHzKwO8AbwS3ffEet4os3dC9y9K6HngvcysyrbFWhmFwMb3T09\n1rGUs9PdvTtwAXBH0PVbZuIlEWQDrcLGWwbTpIoJ+snfAF5296mxjqc8ufs24FOgX6xjiaLTgEuD\nPvNJwNlmNjG2IUWfu2cH/24EphHq7i4z8ZIIUoH2ZtbWzGoSejby9BjHJGUsOHE6Glji7n+PdTzl\nwcyamlmDYLgWoQsivottVNHj7g+6e0t3TyL0d/yJu98U47CiysxqBxc/YGa1gfOBMr0aMC4Sgbvn\nA3cC7xM6gTjZ3RfFNqroMrNXgVnAiWaWZWbDYh1TOTgNGEDoV+L84HVhrIOKsuOAT81sIaEfPB+6\ne1xcUhlHmgFfmdkCYA7wtru/V5Y7iIvLR0VE5ODiokUgIiIHp0QgIhLnlAhEROKcEoGISJxTIhAR\niXNKBCJFmFlB2OWn88uyWq2ZJcVTRVipHKrHOgCRCmhvULJBJC6oRSASoaAm/JNBXfg5ZtYumJ5k\nZp+Y2UIz+9jMWgfTm5nZtOBZAQvMrG+wqQQzezF4fsAHwR3BIjGjRCDyY7WKdA1dGzZvu7t3Bp4l\nVAUT4F/AOHc/BXgZeCaY/gzwubt3AboDB+5mbw+MdPeTgW3AlVE+HpES6c5ikSLMbJe71ylmeiZw\ntruvCIrbrXf3xma2idADcfKC6evcvYmZ5QAt3T03bBtJhMpAtA/GfwPUcPc/Rf/IRIqnFoHI4fGD\nDB+O3LDhAnSuTmJMiUDk8Fwb9u+sYHgmoUqYADcCXwbDHwO3w/8eHlO/vIIUORz6JSLyY7WCJ34d\n8J67H7iEtGFQ6TMXuD6YdheQYmb3ATnAkGD6CGBUUPm1gFBSWBf16EUOk84RiEQo2g8QF4kVdQ2J\niMQ5tQhEROKcWgQiInFOiUBEJM4pEYiIxDklAhGROKdEICIS5/4fOgmvGRJEpOEAAAAASUVORK5C\nYII=\n", 1006 | "text/plain": [ 1007 | "
" 1008 | ] 1009 | }, 1010 | "metadata": { 1011 | "tags": [] 1012 | } 1013 | }, 1014 | { 1015 | "output_type": "display_data", 1016 | "data": { 1017 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9b3/8dcnO5CwJoQ1CRBUFhUk\noiiyuWCtrd2rrba1trSK1lbv1t7+bm9v23tv723thlq12lbrctta22qtSGUTXIOCbIIBEtaQsBMg\nIcvn98ccMNIACeTkZGbez8djHkxmOec98eF7vvnOme8xd0dERBJPStQBREQkHCp4EZEEpYIXEUlQ\nKngRkQSlghcRSVAqeBGRBKWClw5jZkVm5maW1orHfs7MFnVErrYwsy5m9rSZ7TWz33XQPsvN7LLg\n+jfM7BcdsV+Jfyp4aVFQKofNLPeY298MSroommRte6MIwceAfKCPu3+8PTZoZt3N7MdmttHMasxs\nXfBz7rGPdff/dPcvtMM+T/o7NLNrzWxN8GZWZWa/NrPup7tv6TgqeDmRDcB1R34ws7OBrtHF6RQK\ngbXu3tDWJ7ZUpmaWAbwAjAKuBLoDE4CdwPjTi3raFgMXu3sPYCiQBnw32kjSFip4OZFHgM80+/mz\nwMPNH2BmPczsYTOrNrMKM/ummaUE96Wa2Q/MbIeZrQfe38JzHzSzbWa2xcy+a2appxPYzDKD0e/W\n4PJjM8sM7ss1s2fMbI+Z7TKzF5tl/ecgw/5g1HppC9v+NvBvwCeDkfZNZpYSvOaKYJT7sJn1CB5/\nZJR8k5ltBOa2EPkzQAHwYXdf5e5N7l7l7t9x92dbyPDvZvabZj9faGYvBa9pmZlNaXbffDP7jpkt\nDl7X883+KlgY/LsneC0Tjt2Xu29y9x3NbmoEik/w65dORgUvJ/IK0N3MRgTFey3wm2Me8zPgyAhv\nMrHCujG474vA1cBYoITY9EZzvwIaiJXGWOAK4HSnH/4VuBAYA5xLbBT8zeC+O4HNQB6xaZZvAG5m\nZwK3Aue7ew4wHSg/dsPu/i3gP4H/c/dsd38Q+FxwmUrsd5ANzDrmqZOBEcF2j3UZ8Jy717T1hZrZ\nQOAvxEbVvYF/AJ40s7xmD/sUsf8efYGM4DEAk4J/ewav5eXj7GOime0F9gMfBX7c1pwSHRW8nMyR\nUfzlwGpgy5E7mpX+1919v7uXAz8Ebgge8gngx8FIcBfwX82emw9cBXzV3Q+4exXwo2B7p+PTwH8E\no+Bq4NvN8tQD/YFCd6939xc9thhTI5AJjDSzdHcvd/d1bdjfXe6+PijprwPXHjMd8+/BazzUwvP7\nANva/jIBuB541t2fDUb+c4BSYr/XI37p7muDff+W2Btfq7n7omCKZhDwv7TwxiedlwpeTuYRYqPA\nz3HM9AyQC6QDFc1uqwAGBtcHAJuOue+IwuC524LphT3AfcRGmqdjQAt5BgTX/xcoA543s/Vm9i8A\n7l4GfBX4d6DKzJ4wswG0Tkv7SyP2F8IRmzi+ncTedE5FIfDxI7+/4Hc48ZjtVTa7fpDYXxht5u5b\ngOeAJ04xq0RABS8n5O4VxD5svQr4wzF37yA2Ki5sdlsB747ytwGDj7nviE1AHZDr7j2DS3d3H3Wa\nkbe2kGdr8Fr2u/ud7j4U+CBwx5G5dnd/zN0nBs914Punsb8GYHuz2060ZOvfgOlm1q2V+2tuE/BI\ns99fT3fv5u7/3YrnnsoysmnAsFN4nkREBS+tcRMwzd0PNL/R3RuJ/dn/PTPLMbNC4A7enaf/LfAV\nMxtkZr2Af2n23G3A88APg8MEU8xsmJlNbkOuTDPLanZJAR4HvmlmecEHiv92JI+ZXW1mxWZmwF5i\nUzNNZnammU0LPoytBQ4BTa3M8DjwNTMbYmbZvDtH39qjbB4hVtRPmtlZwe+hj8WOd7/qJM/9DfAB\nM5sefKCdZWZTzGxQK/ZbTew1Dj3eA8zs02ZWEFwvBL5H7IgfiRMqeDkpd1/n7qXHufs24ACwHlgE\nPAY8FNz3ADAbWAa8wd//BfAZYh/8rQJ2A7+nbdMVNcTK+MhlGrEPHEuBt4DlwX6PHNo3nNiIuQZ4\nGbjH3ecRm3//b2J/kVQSmyb6eiszPESspBcS+0unltjvpFXcvY7YB61vA3OAfcBrxKa/Xj3JczcB\n1xD7sLia2BvFP9KK/6/d/SCxwl4cTO9c2MLDRgIvmdkBYodMriH2wbnECdMJP0REEpNG8CIiCUoF\nLyKSoFTwIiIJSgUvIpKgoliN77hyc3O9qKgo6hgiInFjyZIlO9w9r6X7OlXBFxUVUVp6vKPxRETk\nWGZWcbz7NEUjIpKgVPAiIglKBS8ikqBU8CIiCUoFLyKSoFTwIiIJSgUvIpKg4r7ga+sbeWDhel5d\nvzPqKCIinUrcF7wZPLhoAz/629qoo4iIdCpxX/CZaanMmDSUV9bv4vXyXVHHERHpNOK+4AGuG19A\nn24ZzJpbFnUUEZFOIyEKvktGKjddMoQFa6tZvnlv1HFERDqFhCh4gBsuLKR7Vhqz5r0TdRQRkU4h\nYQo+JyudGy8ewuyV21lTuT/qOCIikUuYgge48eIiumWkcs98zcWLiCRUwffsmsH1Ewp5etlWNuw4\nEHUcEZFIJVTBA3xh4lDSU1O4V6N4EUlyCVfweTmZXDe+gD+8sYXNuw9GHUdEJDIJV/AAX5o8FDO4\nf+H6qKOIiEQm1II3s3IzW25mS82sw0622r9HFz42bhBPvL6Jqn21HbVbEZFOpSNG8FPdfYy7l3TA\nvo66eXIxjU3OAy9qFC8iySkhp2gACvp05YPnDuDRVzey68DhqOOIiHS4sAvegefNbImZzQh5X3/n\nlinDOFTfyC8Xb+joXYuIRC7sgp/o7ucB7wNmmtmkYx9gZjPMrNTMSqurq9t158Pzc3jf6H78anE5\new/Vt+u2RUQ6u1AL3t23BP9WAU8B41t4zP3uXuLuJXl5ee2e4ZYpxeyva+CRl8vbfdsiIp1ZaAVv\nZt3MLOfIdeAKYEVY+zue0QN7MO2svjy4aAMHDzd09O5FRCIT5gg+H1hkZsuA14C/uPtzIe7vuGZO\nLWb3wXoee3VjFLsXEYlEWlgbdvf1wLlhbb8txhX24uLiPty3cD3XX1hIVnpq1JFEREKXsIdJHmvm\n1GKq99fxuyWbo44iItIhkqbgJwztw7jCXvx8/jrqG5uijiMiErqkKXgz49ZpxWzZc4in3twSdRwR\nkdAlTcEDTDkjj9EDu3Pv/HU0NnnUcUREQpVUBW9m3Dq1mA07DvCX5duijiMiEqqkKniAK0b2Y3jf\nbO6eW0aTRvEiksCSruBTUoyZU4tZs30/c1ZvjzqOiEhokq7gAa4+pz+Ffbpy97wy3DWKF5HElJQF\nn5aawi1ThvHW5r0sfGdH1HFEREKRlAUP8OGxgxjQI4tZc9+JOoqISCiStuAz0lL40uRhvF6+m1fX\n74w6johIu0vaggf45PmDyc3OZNa8sqijiIi0u6Qu+Kz0VGZMGsKL7+zgzY27o44jItKukrrgAT59\nQSE9u6Zzt0bxIpJgkr7gu2Wm8fmLh/C31VWs2rov6jgiIu0m6Qse4LMXFZGTmcbd8zWKF5HEoYIH\nenRJ54YJhTy7fBtlVTVRxxERaRcq+MBNE4eQmZbCvfPXRR1FRKRdqOADfbIz+fQFhfxx6RY27ToY\ndRwRkdOmgm9mxqShpJpx7wKN4kUk/qngm8nvnsXHSwbx+9LNVO6tjTqOiMhpUcEf48uTh9Hozv0L\n10cdRUTktKjgjzG4d1c+PHYgj71WwY6auqjjiIicMhV8C26eMoy6hiYeXLQh6igiIqdMBd+CYXnZ\nvP/s/jzycgV7D9ZHHUdE5JSo4I9j5tRiauoa+NVL5VFHERE5JSr44xjRvzuXjcjnocUbqKlriDqO\niEibqeBP4NZpxew9VM+jr1REHUVEpM1U8CcwZnBPLhmeywMvrqe2vjHqOCIibaKCP4nbpg1nR81h\nnnhtY9RRRETaJPSCN7NUM3vTzJ4Je19hGD+kN+OLenPfwvUcbmiKOo6ISKt1xAj+dmB1B+wnNLdO\nK2bb3lr+8MbmqKOIiLRaqAVvZoOA9wO/CHM/YbtkeC7nDurBPfPX0dCoUbyIxIewR/A/Bv4JOG4r\nmtkMMys1s9Lq6uqQ45waM2Pm1GI27jrI029tjTqOiEirhFbwZnY1UOXuS070OHe/391L3L0kLy8v\nrDin7bIR+ZzVL4e7562jqcmjjiMiclJhjuAvBj5oZuXAE8A0M/tNiPsLVUpKbBRfVlXD7JWVUccR\nETmp0Are3b/u7oPcvQi4Fpjr7teHtb+OcNXZ/Rma241Z88pw1yheRDo3HQffBqkpxs1ThrFy6z7m\nr+mcnxeIiBzRIQXv7vPd/eqO2FfYPjR2IAN7duGnc9/RKF5EOjWN4NsoPTWFm6cM482Ne3h53c6o\n44iIHJcK/hR8bNwg+uZk8rO5ZVFHERE5LhX8KchKT2XGpKG8vH4nSyp2RR1HRKRFKvhT9KkLCujd\nLYNZGsWLSCelgj9FXTPSuGniEOatqWbFlr1RxxER+Tsq+NNww4RCcrLSuHueRvEi0vmo4E9D96x0\nbryoiL+uqGTt9v1RxxEReQ8V/Gm68eIhdM1I5R6N4kWkk1HBn6Ze3TK4/sJC/rxsK+U7DkQdR0Tk\nKBV8O/jCJUNIS03h5wvWRR1FROQoFXw76JuTxXXnD+bJNzazZc+hqOOIiAAq+HYzY/Iw3OF+jeJF\npJNQwbeTgT278NHzBvHE65uo2l8bdRwRERV8e7p5yjDqG5t48MUNUUcREVHBt6ei3G584NwBPPJK\nBbsPHI46jogkORV8O5s5tZiDhxv55UvlUUcRkSSngm9nZ+TncOWofvxq8Qb21dZHHUdEkpgKPgS3\nTitmX20Dj7xcEXUUEUliKvgQjB7Ygyln5vHgog0cPNwQdRwRSVIq+JDcNq2YXQcO8/hrm6KOIiJJ\nSgUfknGFvZkwtA/3L1xHbX1j1HFEJAmp4EN067Ritu+r4/dLNkcdRUSSkAo+RBcN68PYgp78fME6\n6huboo4jIklGBR8iM+O2acVs3n2IPy3dGnUcEUkyKviQTT2zLyP7d+eeeWU0NnnUcUQkiajgQ2Zm\n3DqtmPU7DvDs8m1RxxGRJKKC7wBXjupHcd9s7p5XRpNG8SLSQVTwHSAlxZg5dRhvV+7nhberoo4j\nIklCBd9BPnDOAAp6d2XW3Hdw1yheRMKngu8gaakp3DxlGMs272VR2Y6o44hIEgit4M0sy8xeM7Nl\nZrbSzL4d1r7ixUfOG0j/Hln8bG5Z1FFEJAmEOYKvA6a5+7nAGOBKM7swxP11eplpqcyYNJTXNuzi\ntQ27oo4jIgmuVQVvZsPMLDO4PsXMvmJmPU/0HI+pCX5MDy5JP/l87fkF5GZnMGueRvEiEq7WjuCf\nBBrNrBi4HxgMPHayJ5lZqpktBaqAOe7+aguPmWFmpWZWWl1d3Ybo8alLRipfuGQoC9dWs2zTnqjj\niEgCa23BN7l7A/Bh4Gfu/o9A/5M9yd0b3X0MMAgYb2ajW3jM/e5e4u4leXl5bcket66/sJAeXdI1\niheRULW24OvN7Drgs8AzwW3prd2Ju+8B5gFXti1eYsrOTOPGi4uYs2o7b1fuizqOiCSo1hb8jcAE\n4HvuvsHMhgCPnOgJZpZ3ZJ7ezLoAlwNvn07YRPK5i4rIzkzj7nnroo4iIgmqVQXv7qvc/Svu/riZ\n9QJy3P37J3laf2Cemb0FvE5sDv6ZkzwnafTsmsENEwp55q2trKuuOfkTRETaqLVH0cw3s+5m1ht4\nA3jAzO460XPc/S13H+vu57j7aHf/j/YInEhumjiEzLQU7p2vUbyItL/WTtH0cPd9wEeAh939AuCy\n8GIlh9zsTK4bX8Af39zCpl0Ho44jIgmmtQWfZmb9gU/w7oes0g5mTBpKihn3LdQoXkTaV2sL/j+A\n2cA6d3/dzIYC74QXK3n079GFj44bxG9f38z2fbVRxxGRBNLaD1l/F8yl3xz8vN7dPxputORx8+Rh\nNLpz/8L1UUcRkQTS2g9ZB5nZU2ZWFVyeNLNBYYdLFgV9unLNmAE89upGdtbURR1HRBJEa6dofgn8\nGRgQXJ4ObpN2csuUYmobGnlo8Yaoo4hIgmhtwee5+y/dvSG4/ApIjnUFOkhx32yuGt2fh1+qYO+h\n+qjjiEgCaG3B7zSz64PFw1LN7HpgZ5jBktHMqcXsr2vg4ZfKo44iIgmgtQX/eWKHSFYC24CPAZ8L\nKVPSGjmgO5eN6MuDizdwoK4h6jgiEudaexRNhbt/0N3z3L2vu38I0FE0IZg5tZg9B+t59NWKqKOI\nSJw7nTM63dFuKeSosQW9mFicywMvbqC2vjHqOCISx06n4K3dUsh73DqtmOr9dfy2dFPUUUQkjp1O\nwSf96ffCcsGQ3pQU9uLn89dxuKEp6jgiEqdOWPBmtt/M9rVw2U/seHgJgZlx67Ritu6t5ak3N0cd\nR0Ti1AkL3t1z3L17C5ccd0/rqJDJaPIZeZw9sAf3zl9HQ6NG8SLSdqczRSMhMjNmTi2mfOdB/rJ8\nW9RxRCQOqeA7sStG5nNGfjaz5pbR1KSPPESkbVTwnVhKSmwU/05VDc+v2h51HBGJMyr4Tu7qcwYw\nJLcbs+a9g7tG8SLSeir4Ti41xbh58jBWbNnH/LXVUccRkTiigo8DHxo7kIE9uzBrbplG8SLSair4\nOJCRlsKXJw9lScVuXlm/K+o4IhInVPBx4uMlg8nLyWTWPJ0KV0RaRwUfJ7LSU5lxyVAWl+3kjY27\no44jInFABR9HPnVBAb26pnP33LKoo4hIHFDBx5FumWl8/uIhvPB2FSu27I06joh0cir4OPOZi4rI\nyUzjnvkaxYvIiang40yPLul89qIi/rqikjc1Fy8iJ6CCj0OfnziEPt0y+fjPX+a/nl2t87eKSItC\nK3gzG2xm88xslZmtNLPbw9pXsundLYPnvzaJj543iPsWrufyuxbw3IpKfQlKRN4jzBF8A3Cnu48E\nLgRmmtnIEPeXVHp3y+D7HzuH3395At27pPPl3yzhpl+XsmnXwaijiUgnEVrBu/s2d38juL4fWA0M\nDGt/yaqkqDdP3zaRb75/BK+u38lldy1g1tx3qGvQCbtFkp11xJ/1ZlYELARGu/u+Y+6bAcwAKCgo\nGFdRURF6nkS1be8hvvPMKp5dXsnQvG5895rRXFScG3UsEQmRmS1x95IW7wu74M0sG1gAfM/d/3Ci\nx5aUlHhpaWmoeZLBvDVVfOtPK9m46yDXjBnAv75/BH1zsqKOJSIhOFHBh3oUjZmlA08Cj56s3KX9\nTD2zL89/bRJfuXQ4f11eyaU/WMCvXyqnUWeFEkkqYR5FY8CDwGp3vyus/UjLstJTuePyM3juq5dw\n7uCefOvPK7nm7kUs27Qn6mgi0kHCHMFfDNwATDOzpcHlqhD3Jy0YmpfNIzeN52fXjaVqXx0fumcx\n3/zjcvYerI86moiELC2sDbv7IsDC2r60npnxgXMHMOXMPO6as5Zfv1TOcysq+cZVI/jw2IHE/tgS\nkUSjb7ImkZysdL71gVH8+daJDOrVlTt+u4xr73+Fd7bvjzqaiIRABZ+ERg/swR9uvoj//PDZvF25\nn/f95EW+/9zbHDqsY+dFEokKPkmlpBifuqCAF+6czIfGDuTe+eu47K4FzFm1PepoItJOVPBJLjc7\nkx98/Fx++6UJdMtM5YsPl/KFX5eyebeWPBCJdyp4AWD8kN785SuX8PX3ncXish1cdtcC7plfxuGG\npqijicgpUsHLUempKXxp8jD+dudkJp+Rx/88t4arfvoiL6/bGXU0ETkFKnj5OwN7duG+G0p48LMl\n1NY3ct0Dr3DH/y2len9d1NFEpA1U8HJcl47IZ87XJnPr1GKefmsrl/5wPo+8UqElD0TihApeTqhL\nRir/MP1M/nr7JEYN6MH/++MKPnLPYpZv1km/RTo7Fby0SnHfbB774gX85NoxbNlTyzV3L+Jbf1rB\nvloteSDSWangpdXMjGvGDOSFOydzw4WFPPxKBdN+sIA/Ld2i0wWKdEIqeGmzHl3S+fY1o/nzzIkM\n6JnF7U8s5dO/eJWyqpqoo4lIMyp4OWVnD+rBU7dczHc+NJrlW/byvp8s5Aez11BbryUPRDoDFbyc\nltQU44YLC5l75xQ+cM4AZs0r4/IfLWDu21ryQCRqKnhpF3k5mdz1yTE8/sULyUxL5fO/KuVLj5Sy\ndc+hqKOJJC0VvLSrCcP68OxXLuGfrjyTBWurueyuBdy3YB31jVryQKSjqeCl3WWkpXDLlGLmfG0y\nFw3rw3/99W3e/9MXeW3DrqijiSQVFbyEZnDvrvzis+fzwGdKOFDXyCfue5l/+N0ydtZoyQORjqCC\nl9BdPjKfOXdM4uYpw/jjm1uY9sMFPPbqRpq05IFIqFTw0iG6ZqTxz1eexV9vv4Sz+uXwjaeW85F7\nX2LFFi15IBIWFbx0qOH5OTwx40Lu+sS5bNp1kA/OWsS3n17Jfi15INLuVPDS4cyMj5w3iLl3TuG6\n8QX86qVyLv3hAp5etlVLHoi0IxW8RKZH13S+9+GzeeqWi+nbPZPbHn+Tzzz0Ght2HIg6mkhCUMFL\n5MYM7smfZk7k2x8cxdKNe5j+o4XcNWetljwQOU0qeOkUUlOMz15UxAt3TuZ9Z/fjpy+8w/QfL2T+\nmqqoo4nELRW8dCp9u2fxk2vH8ugXLiDVjM/98nVueXQJlXtro44mEnesM32oVVJS4qWlpVHHkE6i\nrqGRBxau52dzy0hLMW67dDgj+3cnLdVIS0khLdVIP/Jvs9uOve/I9ZQUi/olibQ7M1vi7iUt3qeC\nl85u486DfOvPK5i3pvq0tpNiNHsTMNJT331DSE810lJTSEux99529A0khdSU976RpB9z37G3xbb1\n7jbffcNJIf3IfUGWYzMcyZadmUZ+96x2+k1KIjpRwad1dBiRtiro05WHPnc+a7fXsL+2nvpGp7HJ\nqW9qoqHRaWhsor4p9m9Do9PQ5DQ0NVEf3NfQ5NQ3NsWec8xtDY2x7TQ2eez6Mfc1NDVR2/De+xoa\ng203vXebRzK09xd0z8jP5spR/bhiVD9GDeiOmf4SkdZRwUtcMDPO7JcTdYxWaWp6902jpTeUhqYj\nbxQtv6EceZOqb2yien8df1u9nVnzyvjp3DIG9erCFSP7ceXofowr7EWqpp3kBEKbojGzh4CrgSp3\nH92a52iKRqRlO2vqeGF1FbNXVvJi2Q4ONzTRp1sGl4/MZ/qoflxU3IfMtNSoY0oEIpmDN7NJQA3w\nsApepP3U1DUwf00Vs1duZ97bVdTUNZCdmcbUs/oyfVQ+U87sS3am/jhPFpHMwbv7QjMrCmv7Iskq\nOzONq88ZwNXnDKCuoZGX1u1k9opK5qzaztPLtpKRlsLE4lymj8rnshH59MnOjDqyRCTUo2iCgn/m\nRCN4M5sBzAAoKCgYV1FREVoekUTW2OQsqdjN7JWVPLeiki17DpFicH5Rb6aP6sf00f0Y2LNL1DGl\nnUV2mGRrCr45TdGItA93Z+XWfTy/spLZK7ezZvt+AM4e2IPpo2Lz9sV9s3VETgJQwYskuQ07DjB7\nZSWzV1by5sY9AAzN7cb00f2YPqof5wzsoS+CxSkVvIgcVbm3ljmrYiP7V9bvpKHJ6dc9iyuCkf34\nIb1JT9UqJvEiqqNoHgemALnAduBb7v7giZ6jghfpWHsP1vPC29uZvbKSBWurqa1vomfXdC49K5/p\no/KZdEYeWek6/LIz01IFInJShw43smBtNc+vrORvq7ezr7aBLumpTD4jjytH92PqWX3p0SU96phy\nDC1VICIn1SUjlStHx74lW9/YxKvrd/Hcym08v3I7z62sJC3FmDCsD9NH9eOKkfn01Ro5nZ5G8CJy\nQk1NztLNe5i9IvYhbfnOg5jBeQW9jh6RU9inW9Qxk5amaESkXbg7a7fXHD0iZ+XWfQCc1S8ndqz9\nqH6M6J+jwy87kApeREKxaddBZq+s5PmV23m9YhfuUNC769GR/XkFvXT4ZchU8CISuiMrX85eWcni\nsh3UNzq52ZnBgmj5XDQsl4w0HX7Z3lTwItKh9tfWM29NNbNXVDJvTRUHDzeSk5XGtLP6Mn1UPyaf\nkUc3LYjWLlTwIhKZ2vpGFpftYPbKSv62uopdBw6TmZbCJcNzmT6qH5eNyKdXt4yoY8YtHSYpIpHJ\nSk/l0hH5XDoin4bGJl4v3x3M28cKPzXFGF/Um+mj8rliVD8GaEG0dqMRvIhEwt1ZvmVvcETOdsqq\nagAYPbA75xf1pqSwNyVFvXRO2pPQFI2IdHrrqmOHXy5YU82yzXuorW8CYFCvLowr7EVJYS/GFfbm\nzH45OlVhMyp4EYkrhxuaWLVtH6Xlu1hSsZvSit1U768DICczjTEFPYPS782Ygp5JfQYrFbyIxDV3\nZ/PuQ5RW7KK0fDdLKnazZvt+3CHFYET/7rERflFvxhX2SqoTm6jgRSTh7D1Uz9JNe1hSvovSit0s\n3bSHg4cbAejfI+votE5JUW/O6pdDWoIugayjaEQk4fToks7kM/KYfEYeAA2NTazetp8lFbHCX1Kx\nm2fe2gZA14xUxgzueXSUP7agJ92zEn9lTI3gRSRhbdlziNLyXbwRzOOv3raPJgczODM/JzbKL4rN\n5Q/q1SUu19DRFI2ICFBT18DSjXsorYh9ePvmxj3U1DUA0Dcnk3GFvYLS782oAd3j4sxWmqIREQGy\nM9OYODyXicNzAWhsctZUvjutU1q+m7+uqAQgKz2Fcwf1PDrKP6+gFz27xtc3bjWCFxFppnJvbXBo\nZmyUv3LrPhqbYj05vG/20bIvKepNUZ+ukU/raIpGROQUHTzcwLJNe9/z4e3+2ti0Tm52RlD2sS9h\njR7Yncy0jj2HraZoREROUdeMNCYM68OEYX2A2Bmu3qmqOTrCX1Kxm+dXbQcgIy2Fcwb2YFzwwe24\nwl70jnAhNY3gRUROU9X+2tiROuWxo3VWbt1LfWOsW4fmdjs6jz+usDfD8rq167SOpmhERDpQbX0j\nb23eGxvll+9mycbd7DlYD45s7wsAAAVYSURBVEDPrumMK+h1dJR/zqAeZKWf+rSOpmhERDpQVnoq\n44f0ZvyQ3kBsWmf9jgOxefxgqYUX3q4CID3VGDu4F0/MuLDdT2+oghcRCVlKilHcN5vivtl88vwC\nAHbW1PFGcEz+vkP1oZy7VgUvIhKBPsH5ai8fmR/aPjr/17REROSUqOBFRBKUCl5EJEGp4EVEElSo\nBW9mV5rZGjMrM7N/CXNfIiLyXqEVvJmlAncD7wNGAteZ2ciw9iciIu8V5gh+PFDm7uvd/TDwBHBN\niPsTEZFmwiz4gcCmZj9vDm57DzObYWalZlZaXV0dYhwRkeQS+Red3P1+4H4AM6s2s4pT3FQusKPd\ngsUHvebEl2yvF/Sa26rweHeEWfBbgMHNfh4U3HZc7p53qjszs9LjLbiTqPSaE1+yvV7Qa25PYU7R\nvA4MN7MhZpYBXAv8OcT9iYhIM6GN4N29wcxuBWYDqcBD7r4yrP2JiMh7hToH7+7PAs+GuY9m7u+g\n/XQmes2JL9leL+g1t5tOdcIPERFpP1qqQEQkQangRUQSVNwXfDKud2NmD5lZlZmtiDpLRzCzwWY2\nz8xWmdlKM7s96kxhM7MsM3vNzJYFr/nbUWfqKGaWamZvmtkzUWfpCGZWbmbLzWypmbXrSanjeg4+\nWO9mLXA5sW/Kvg5c5+6rIg0WMjObBNQAD7v76KjzhM3M+gP93f0NM8sBlgAfSuT/zmZmQDd3rzGz\ndGARcLu7vxJxtNCZ2R1ACdDd3a+OOk/YzKwcKHH3dv9yV7yP4JNyvRt3XwjsijpHR3H3be7+RnB9\nP7CaFpa9SCQeUxP8mB5c4nc01kpmNgh4P/CLqLMkgngv+FatdyOJw8yKgLHAq9EmCV8wVbEUqALm\nuHvCv2bgx8A/AU1RB+lADjxvZkvMbEZ7bjjeC16SiJllA08CX3X3fVHnCZu7N7r7GGLLfIw3s4Se\njjOzq4Eqd18SdZYONtHdzyO2tPrMYAq2XcR7wbd5vRuJT8E89JPAo+7+h6jzdCR33wPMA66MOkvI\nLgY+GMxJPwFMM7PfRBspfO6+Jfi3CniK2NRzu4j3gtd6N0kg+MDxQWC1u98VdZ6OYGZ5ZtYzuN6F\n2IEEb0ebKlzu/nV3H+TuRcT+X57r7tdHHCtUZtYtOHAAM+sGXAG029FxcV3w7t4AHFnvZjXw22RY\n78bMHgdeBs40s81mdlPUmUJ2MXADsRHd0uByVdShQtYfmGdmbxEbyMxx96Q4bDDJ5AOLzGwZ8Brw\nF3d/rr02HteHSYqIyPHF9QheRESOTwUvIpKgVPAiIglKBS8ikqBU8CIiCUoFL0nFzBqbHWq5tD1X\nIDWzomRZ4VPiQ6in7BPphA4FX/8XSXgawYtwdE3u/wnW5X7NzIqD24vMbK6ZvWVmL5hZQXB7vpk9\nFazXvszMLgo2lWpmDwRruD8ffAtVJBIqeEk2XY6Zovlks/v2uvvZwCxiqxoC/Az4tbufAzwK/DS4\n/afAAnc/FzgPOPIN6uHA3e4+CtgDfDTk1yNyXPomqyQVM6tx9+wWbi8Hprn7+mBhs0p372NmO4id\nbKQ+uH2bu+eaWTUwyN3rmm2jiNiSAsODn/8ZSHf374b/ykT+nkbwIu/y41xvi7pm1xvR51wSIRW8\nyLs+2ezfl4PrLxFb2RDg08CLwfUXgJvh6Ik5enRUSJHW0uhCkk2X4CxJRzzn7kcOlewVrN5YB1wX\n3HYb8Esz+0egGrgxuP124P5gJc9GYmW/LfT0Im2gOXgRwj3xsUhUNEUjIpKgNIIXEUlQGsGLiCQo\nFbyISIJSwYuIJCgVvIhIglLBi4gkqP8PnFyJH15lin8AAAAASUVORK5CYII=\n", 1018 | "text/plain": [ 1019 | "
" 1020 | ] 1021 | }, 1022 | "metadata": { 1023 | "tags": [] 1024 | } 1025 | } 1026 | ] 1027 | }, 1028 | { 1029 | "cell_type": "markdown", 1030 | "metadata": { 1031 | "id": "VoYTfdm8WKj-", 1032 | "colab_type": "text" 1033 | }, 1034 | "source": [ 1035 | "### Evaluate Model" 1036 | ] 1037 | }, 1038 | { 1039 | "cell_type": "markdown", 1040 | "metadata": { 1041 | "id": "d2wc31l8WM3P", 1042 | "colab_type": "text" 1043 | }, 1044 | "source": [ 1045 | "Last but not least, let's evaluate the model after federated training process." 1046 | ] 1047 | }, 1048 | { 1049 | "cell_type": "code", 1050 | "metadata": { 1051 | "id": "P-UijEryWWm0", 1052 | "colab_type": "code", 1053 | "outputId": "5f7912fc-f688-4243-b3b3-fb2fd9957d11", 1054 | "colab": { 1055 | "base_uri": "https://localhost:8080/", 1056 | "height": 241 1057 | } 1058 | }, 1059 | "source": [ 1060 | "evaluator = tff.learning.build_federated_evaluation(create_federated_model)" 1061 | ], 1062 | "execution_count": 0, 1063 | "outputs": [ 1064 | { 1065 | "output_type": "stream", 1066 | "text": [ 1067 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 1068 | "\n", 1069 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 1070 | "\n", 1071 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 1072 | "\n", 1073 | "WARNING:tensorflow:Layer conv2d is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n", 1074 | "\n", 1075 | "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n", 1076 | "\n", 1077 | "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n", 1078 | "\n" 1079 | ], 1080 | "name": "stdout" 1081 | } 1082 | ] 1083 | }, 1084 | { 1085 | "cell_type": "code", 1086 | "metadata": { 1087 | "id": "4cgOiNF7XWW8", 1088 | "colab_type": "code", 1089 | "outputId": "30f00ae6-17e4-49a2-de2b-2a5618a68cd6", 1090 | "colab": { 1091 | "base_uri": "https://localhost:8080/", 1092 | "height": 34 1093 | } 1094 | }, 1095 | "source": [ 1096 | "test_metrics = evaluator(train_state.model, federated_data_test)\n", 1097 | "print(test_metrics)" 1098 | ], 1099 | "execution_count": 0, 1100 | "outputs": [ 1101 | { 1102 | "output_type": "stream", 1103 | "text": [ 1104 | "\n" 1105 | ], 1106 | "name": "stdout" 1107 | } 1108 | ] 1109 | }, 1110 | { 1111 | "cell_type": "markdown", 1112 | "metadata": { 1113 | "id": "OuX_mvFVGRG6", 1114 | "colab_type": "text" 1115 | }, 1116 | "source": [ 1117 | "## Further Thinking" 1118 | ] 1119 | }, 1120 | { 1121 | "cell_type": "markdown", 1122 | "metadata": { 1123 | "id": "JhLGU0V1S8JA", 1124 | "colab_type": "text" 1125 | }, 1126 | "source": [ 1127 | "During the federated training process, I have come up with the following thoughts which might consider to improve the federated training in the future." 1128 | ] 1129 | }, 1130 | { 1131 | "cell_type": "markdown", 1132 | "metadata": { 1133 | "id": "DO5BBfnDGW1c", 1134 | "colab_type": "text" 1135 | }, 1136 | "source": [ 1137 | "- Early stopping on federated training: \n", 1138 | "\n", 1139 | "**Early stopping** strategy is generally used to stop the training process when there is no obvious improvement in the validation process. Therefore, I would suggest that early stopping shall also be integrated in the federated training process. For example, the shared model in the centralized server may reject the client's update when the validation process reaches a certain plateau.\n", 1140 | "\n", 1141 | "- Centralized server handling upon receiving large amount of local update: \n", 1142 | "\n", 1143 | "There is a shared model in the centralized server constantly receiving the update from the clients. However, if there are multiple clients simultaneously update to the model of the centralized server, it might significantly increase the burder of the server load. Therefore, I would suggest to use a **Message Queue** (e.g. RabbitMQ, or Kafka) to reduce server load. Besides, instead of using one shared model in one single server, we could copy several instances of models in multiple paralleled servers, and compute the model update in parallel using **Load Balancing** techniques.\n", 1144 | "\n", 1145 | "- Future application of federate learning: \n", 1146 | "\n", 1147 | "We may also consider to perform federated learning in distributed devices other than user devices (e.g. mobile devices). For example, a **Content Delivery Network (CDN)** is generally used to provide fast internet delivery through a geographically distributed group of servers. Therefore, I would suggest that we may also conduct machine learning training in these servers in the CDN network, and inform the model update to other server nodes by using **Gossip Protocol** strategy." 1148 | ] 1149 | }, 1150 | { 1151 | "cell_type": "markdown", 1152 | "metadata": { 1153 | "id": "DTgp0qi9GXSD", 1154 | "colab_type": "text" 1155 | }, 1156 | "source": [ 1157 | "## Conclusion" 1158 | ] 1159 | }, 1160 | { 1161 | "cell_type": "markdown", 1162 | "metadata": { 1163 | "id": "bgZk3ztPGZMu", 1164 | "colab_type": "text" 1165 | }, 1166 | "source": [ 1167 | "To put it in a nutshell, this document has introduced the concept of federated learning, and the implementation of federated training to create a computer vision model for facial expression recognition." 1168 | ] 1169 | }, 1170 | { 1171 | "cell_type": "markdown", 1172 | "metadata": { 1173 | "id": "D6S_bspNGZy3", 1174 | "colab_type": "text" 1175 | }, 1176 | "source": [ 1177 | "## Reference" 1178 | ] 1179 | }, 1180 | { 1181 | "cell_type": "markdown", 1182 | "metadata": { 1183 | "id": "kNYDP6-nGayG", 1184 | "colab_type": "text" 1185 | }, 1186 | "source": [ 1187 | "- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/pdf/1602.05629.pdf)\n", 1188 | "- [Federated Learning for Image Classification](https://www.tensorflow.org/federated/tutorials/federated_learning_for_image_classification#evaluation)\n", 1189 | "- [TensorFlow Federated: Machine Learning on Decentralized Data](https://www.tensorflow.org/federated)\n", 1190 | "- [Facial Expression Recognition Challenge using Convolutional Neural Network](https://mikemikezhu.github.io/dev/2020/02/11/facial_expression_recognition.html)\n", 1191 | "- [Challenges in Representation Learning: Facial Expression Recognition Challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)\n", 1192 | "- [Best Actor Game](https://apps.apple.com/cn/app/id1498575047)\n", 1193 | "- [Globally Distributed Content Delivery](https://people.cs.umass.edu/~ramesh/Site/PUBLICATIONS_files/DMPPSW02.pdf)\n", 1194 | "- [Gossip Protocol](https://en.wikipedia.org/wiki/Gossip_protocol)\n", 1195 | "- [Early Stopping](https://en.wikipedia.org/wiki/Early_stopping)\n", 1196 | "- [Performance Tradeoffs in Static and Dynamic Load Balancing Strategies](https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19860014876.pdf)" 1197 | ] 1198 | } 1199 | ] 1200 | } --------------------------------------------------------------------------------