├── README.md ├── configs ├── cifar10_cnn.yaml ├── cifar10_resnet.yaml ├── hello.yaml └── imagenet_resnet.yaml ├── data ├── __init__.py ├── cifar10.py ├── dummy.py └── imagenet.py ├── logs └── README.md ├── models ├── __init__.py ├── cnn.py └── resnet.py ├── notebooks └── Analysis.ipynb ├── scripts ├── cifar_cnn.sh └── cifar_resnet.sh ├── train_horovod.py └── utils ├── __init__.py ├── callbacks.py ├── device.py └── optimizers.py /README.md: -------------------------------------------------------------------------------- 1 | # Deep Learning for Science School Tutorial:
Deep Learning At Scale 2 | 3 | This repository contains the material for the DL4Sci tutorial: 4 | *Deep Learning at Scale*. 5 | 6 | It contains specifications for datasets, a couple of CNN models, and 7 | all the training code to enable training the models in a distributed fashion 8 | using Horovod. 9 | 10 | As part of the tutorial, you will train a ResNet model to classify images 11 | from the CIFAR10 dataset on multiple nodes with synchronous data parallelism. 12 | 13 | **Contents** 14 | * [Links](https://github.com/NERSC/dl4sci-scaling-tutorial#links) 15 | * [Installation](https://github.com/NERSC/dl4sci-scaling-tutorial#installation) 16 | * [Navigating the repository](https://github.com/NERSC/dl4sci-scaling-tutorial#navigating-the-repository) 17 | * [Hands-on walk-through](https://github.com/NERSC/dl4sci-scaling-tutorial#hands-on-multi-node-training-example) 18 | * [Code references](https://github.com/NERSC/dl4sci-scaling-tutorial#code-references) 19 | 20 | ## Links 21 | 22 | NERSC JupyterHub: https://jupyter-dl.nersc.gov 23 | 24 | Slides: https://drive.google.com/drive/folders/10NqOLaqPTZ0nobE7JNSaGwrXQi_ACD35?usp=sharing 25 | 26 | ## Installation 27 | 28 | 1. Start a terminal on Cori, either via ssh or from the Jupyter interface. 29 | * **IMPORTANT: if using jupyter, you need to use a SHARED CPU. Click the CPU button instead of the GPU button to run this example!** 30 | 2. Clone the repository using git:\ 31 | `git clone https://github.com/NERSC/dl4sci-scaling-tutorial.git` 32 | 33 | That's it! The rest of the software (Keras, TensorFlow) is pre-installed on Cori 34 | and loaded via the scripts used below. 35 | 36 | ## Navigating the repository 37 | 38 | **`train_horovod.py`** - the main training script which can be steered with YAML 39 | configuration files. 40 | 41 | **`data/`** - folder containing the specifications of the datasets. Each dataset 42 | has a corresponding name which is mapped to the specification in `data/__init__.py` 43 | 44 | **`models/`** - folder containing the Keras model definitions. Again, each model 45 | has a name which is interpreted in `models/__init__.py`. 46 | 47 | **`configs/`** - folder containing the configuration files. Each 48 | configuration specifies a dataset, a model, and all relevant configuration 49 | options (with some exceptions like the number of nodes, which is specified 50 | instead to SLURM via the command line). 51 | 52 | **`scripts/`** - contains an environment setup script and some SLURM scripts 53 | for easily submitting the example jobs to the Cori batch system. 54 | 55 | **`utils/`** - contains additional useful code for the training script, e.g. 56 | custom callbacks, device configuration, and optimizers logic. 57 | 58 | ## Hands-on multi-node training example 59 | 60 | We will use a customized ResNet model in this example to classify CIFAR10 61 | images and demonstrate distributed training with Horovod. 62 | 63 | 1. Check out the ResNet model code in [models/resnet.py](models/resnet.py). 64 | Note that the model code is broken into multiple functions for easy reuse. 65 | We provide here two versions of ResNet models: a standard ResNet50 (with 50 66 | layers) and a smaller ResNet consisting of 26 layers. 67 | * Identify the identy block and conv block functions. *How many convolutional 68 | layers do each of these have*? 69 | * Identify the functions that build the ResNet50 and the ResNetSmall. Given how 70 | many layers are in each block, *see if you can confirm how many layers (conv 71 | and dense) are in the models*. **Hint:** we don't normally count the 72 | convolution applied to the shortcuts. 73 | 74 | 2. Inspect the optimizer setup in [utils/optimizers.py](utils/optimizers.py). 75 | * Note how we scale the learning rate (`lr`) according to the number of 76 | processes (ranks). 77 | * Note how we construct our optimizer and then wrap it in the Horovod 78 | DistributedOptimizer. 79 | 80 | 3. Inspect the main [train_horovod.py](train_horovod.py) training script. 81 | * Identify the `init_workers` function where we initialize Horovod. 82 | Note where this is invoked in the main() function (right away). 83 | * Identify where we setup our distributed training callbacks. 84 | * *Which callback ensures we have consistent model weights at the start of training?* 85 | * Identify the callbacks responsible for the learning rate schedule (warmup and decay). 86 | 87 | 4. Finally, look at the configuration file: 88 | [configs/cifar10_resnet.yaml](configs/cifar10_resnet.yaml) 89 | * YAML allows to express configurations in rich, human-readable, hierarchical structure. 90 | * Identify where you would edit to modify the optimizer, learning-rate, batch-size, etc. 91 | 92 | That's mostly it for the code. Note that in general when training distributed 93 | you might want to use more complicated data handling, e.g. to ensure different 94 | workers are always processing different samples of your data within a training 95 | epoch. In this case we aren't worrying about that and are, for simplicity, 96 | relying on the independent random shuffling of the data by each worker as well 97 | as the random data augmentation. 98 | 99 | 5. To gain an appreciation for the speedup of training on 100 | multiple nodes, first train the ResNet model on a single node. 101 | Adjust the configuration in [configs/cifar10_resnet.yaml](configs/cifar10_resnet.yaml) 102 | to train for just 1 epoch and then submit the job to the Cori batch system with 103 | SLURM sbatch and our provided SLURM batch script:\ 104 | `sbatch -N 1 scripts/cifar_resnet.sh` 105 | * **Important:** the first time you run a CIFAR10 example, it will 106 | automatically download the dataset. If you have more than one job attempting 107 | this download simultaneously it will likely fail. 108 | 109 | 6. Now we are ready to train our ResNet model on multiple nodes using Horovod 110 | and MPI! If you changed the config to 1 epoch above, be sure to change it back 111 | to 32 epochs for this step. To launch the ResNet training on 8 nodes, do:\ 112 | `sbatch -N 8 scripts/cifar_resnet.sh` 113 | 114 | 7. Check on the status of your job by running `sqs`. 115 | Once the job starts running, you should see the output start to appear in the 116 | slurm log file `logs/cifar-cnn-*.out`. You'll see some printouts from every 117 | worker. Others are only printed from rank 0. 118 | 119 | 8. When the job is finished, check the log to identify how well your model learned 120 | to solve the CIFAR10 classification task. For every epoch you should see the 121 | loss and accuracy reported for both the training set and the validation set. 122 | Take note of the best validation accuracy achieved. 123 | 124 | Now that you've finished the main tutorial material, try to play with the code 125 | and/or configuration to see the effect on the training results. You can try changing 126 | things like 127 | * Change the optimizer (search for Keras optimizers on google). 128 | * Change the nominal learning rate, number of warmup epochs, decay schedule 129 | * Change the learning rate scaling (e.g. try "sqrt" scaling instead of linear) 130 | 131 | Most of these things can be changed entirely within the configuration. 132 | See [configs/imagenet_resnet.yaml](configs/imagenet_resnet.yaml) for examples. 133 | 134 | ## Code references 135 | 136 | Keras ResNet50 official model: 137 | https://github.com/keras-team/keras-applications/blob/master/keras_applications/resnet50.py 138 | 139 | Horovod ResNet + ImageNet example: 140 | https://github.com/uber/horovod/blob/master/examples/keras_imagenet_resnet50.py 141 | 142 | CIFAR10 CNN and ResNet examples: 143 | https://github.com/keras-team/keras/blob/master/examples/cifar10_cnn.py 144 | https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py 145 | -------------------------------------------------------------------------------- /configs/cifar10_cnn.yaml: -------------------------------------------------------------------------------- 1 | description: 'CNN CIFAR10' 2 | output_dir: $SCRATCH/isc19-dl-tutorial/cifar-cnn-N${SLURM_JOB_NUM_NODES}-${SLURM_JOB_ID} 3 | 4 | data: 5 | name: cifar10 6 | 7 | model: 8 | name: cnn 9 | input_shape: [32, 32, 3] 10 | n_classes: 10 11 | dropout: 0.1 12 | 13 | optimizer: 14 | name: Adam 15 | lr: 0.001 16 | 17 | training: 18 | batch_size: 64 19 | n_epochs: 24 20 | lr_warmup_epochs: 5 21 | loss: categorical_crossentropy 22 | metrics: [accuracy] 23 | -------------------------------------------------------------------------------- /configs/cifar10_resnet.yaml: -------------------------------------------------------------------------------- 1 | description: 'ResNet CIFAR10' 2 | output_dir: $SCRATCH/isc19-dl-tutorial/cifar10-resnet-N${SLURM_JOB_NUM_NODES}-${SLURM_JOB_ID} 3 | 4 | data: 5 | name: cifar10 6 | 7 | model: 8 | name: resnet_small 9 | input_shape: [32, 32, 3] 10 | n_classes: 10 11 | 12 | optimizer: 13 | name: Adam 14 | lr: 0.0001 15 | lr_scaling: linear 16 | 17 | training: 18 | batch_size: 64 19 | n_epochs: 32 20 | lr_warmup_epochs: 5 21 | loss: categorical_crossentropy 22 | metrics: [accuracy] 23 | 24 | device: 25 | intra_threads: 32 26 | -------------------------------------------------------------------------------- /configs/hello.yaml: -------------------------------------------------------------------------------- 1 | description: 'Hello world' 2 | output_dir: $SCRATCH/isc19-dl-tutorial/hello-world 3 | 4 | data: 5 | name: dummy 6 | input_shape: [64, 64, 3] 7 | n_train: 1024 8 | n_valid: 1024 9 | 10 | model: 11 | name: cnn 12 | input_shape: [64, 64, 3] 13 | n_classes: 1 14 | 15 | optimizer: 16 | name: Adam 17 | lr: 0.001 18 | 19 | training: 20 | batch_size: 32 21 | n_epochs: 1 22 | loss: binary_crossentropy 23 | metrics: [accuracy] 24 | -------------------------------------------------------------------------------- /configs/imagenet_resnet.yaml: -------------------------------------------------------------------------------- 1 | # This configuration should match what is implemented in the horovod example: 2 | # https://github.com/uber/horovod/blob/master/examples/keras_imagenet_resnet50.py 3 | 4 | description: 'ResNet ImageNet' 5 | output_dir: $SCRATCH/isc19-dl-tutorial/imagenet-resnet-N${SLURM_JOB_NUM_NODES}-${SLURM_JOB_ID} 6 | 7 | data: 8 | name: imagenet 9 | train_dir: /global/cscratch1/sd/sfarrell/ImageNet-100/train 10 | valid_dir: /global/cscratch1/sd/sfarrell/ImageNet-100/validation 11 | 12 | model: 13 | name: resnet50 14 | input_shape: [224, 224, 3] 15 | n_classes: 100 16 | 17 | optimizer: 18 | name: SGD 19 | lr: 0.0125 20 | momentum: 0.9 21 | 22 | training: 23 | batch_size: 32 24 | n_epochs: 100 25 | lr_warmup_epochs: 5 26 | loss: categorical_crossentropy 27 | metrics: [accuracy, top_k_categorical_accuracy] 28 | lr_schedule: 29 | - {start_epoch: 5, end_epoch: 30, multiplier: 1.} 30 | - {start_epoch: 30, end_epoch: 60, multiplier: 1.e-1} 31 | - {start_epoch: 60, end_epoch: 80, multiplier: 1.e-2} 32 | - {start_epoch: 80, multiplier: 1.e-3} 33 | -------------------------------------------------------------------------------- /data/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Keras dataset specifications. 3 | TODO: add MNIST. 4 | """ 5 | 6 | def get_datasets(name, **data_args): 7 | if name == 'dummy': 8 | from .dummy import get_datasets 9 | return get_datasets(**data_args) 10 | elif name == 'cifar10': 11 | from .cifar10 import get_datasets 12 | return get_datasets(**data_args) 13 | elif name == 'imagenet': 14 | from .imagenet import get_datasets 15 | return get_datasets(**data_args) 16 | else: 17 | raise ValueError('Dataset %s unknown' % name) 18 | -------------------------------------------------------------------------------- /data/cifar10.py: -------------------------------------------------------------------------------- 1 | """ 2 | CIFAR10 dataset specification. 3 | 4 | https://github.com/keras-team/keras/blob/master/examples/cifar10_cnn.py 5 | https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py 6 | """ 7 | 8 | # Externals 9 | import keras 10 | from keras.datasets import cifar10 11 | from keras.preprocessing.image import ImageDataGenerator 12 | 13 | def get_datasets(batch_size, n_train=None, n_valid=None): 14 | """ 15 | Load the CIFAR10 data and construct pipeline. 16 | """ 17 | (x_train, y_train), (x_valid, y_valid) = cifar10.load_data() 18 | 19 | # Normalize pixel values 20 | x_train = x_train.astype('float32') / 255 21 | x_valid = x_valid.astype('float32') / 255 22 | 23 | # Select subset of data if specified 24 | if n_train is not None: 25 | x_train, y_train = x_train[:n_train], y_train[:n_train] 26 | if n_valid is not None: 27 | x_valid, y_valid = x_valid[:n_valid], y_valid[:n_valid] 28 | 29 | # Convert labels to class vectors 30 | n_classes = 10 31 | y_train = keras.utils.to_categorical(y_train, n_classes) 32 | y_valid = keras.utils.to_categorical(y_valid, n_classes) 33 | 34 | # Prepare the generators with data augmentation 35 | train_gen = ImageDataGenerator(width_shift_range=0.1, 36 | height_shift_range=0.1, 37 | horizontal_flip=True) 38 | valid_gen = ImageDataGenerator() 39 | train_iter = train_gen.flow(x_train, y_train, batch_size=batch_size, shuffle=True) 40 | valid_iter = valid_gen.flow(x_valid, y_valid, batch_size=batch_size, shuffle=True) 41 | return train_iter, valid_iter 42 | -------------------------------------------------------------------------------- /data/dummy.py: -------------------------------------------------------------------------------- 1 | """ 2 | Random dummy dataset specification. 3 | """ 4 | 5 | # System 6 | import math 7 | 8 | # Externals 9 | import numpy as np 10 | from keras.utils import Sequence 11 | 12 | class DummyDataset(Sequence): 13 | 14 | def __init__(self, n_samples, batch_size, input_shape, target_shape): 15 | self.x = np.random.normal(size=(n_samples,) + tuple(input_shape)) 16 | self.y = np.random.normal(size=(n_samples,) + tuple(target_shape)) 17 | self.batch_size = batch_size 18 | 19 | def __len__(self): 20 | return math.ceil(len(self.x) / self.batch_size) 21 | 22 | def __getitem__(self, idx): 23 | start = idx * self.batch_size 24 | end = (idx + 1) * self.batch_size 25 | return self.x[start:end], self.y[start:end] 26 | 27 | def get_datasets(batch_size, n_train=1024, n_valid=1024, 28 | input_shape=(32, 32, 3), target_shape=()): 29 | train_data = DummyDataset(n_train, batch_size, input_shape, target_shape) 30 | valid_data = DummyDataset(n_valid, batch_size, input_shape, target_shape) 31 | return train_data, valid_data 32 | -------------------------------------------------------------------------------- /data/imagenet.py: -------------------------------------------------------------------------------- 1 | """ 2 | ImageNet dataset specification. 3 | 4 | Adapted from 5 | https://github.com/uber/horovod/blob/master/examples/keras_imagenet_resnet50.py 6 | """ 7 | 8 | # Externals 9 | import keras 10 | from keras.preprocessing.image import ImageDataGenerator 11 | 12 | def get_datasets(batch_size, train_dir, valid_dir): 13 | train_gen = ImageDataGenerator( 14 | preprocessing_function=keras.applications.resnet50.preprocess_input, 15 | width_shift_range=0.33, height_shift_range=0.33, zoom_range=0.5, 16 | horizontal_flip=True) 17 | test_gen = ImageDataGenerator( 18 | preprocessing_function=keras.applications.resnet50.preprocess_input, 19 | zoom_range=(0.875, 0.875)) 20 | train_iter = train_gen.flow_from_directory(train_dir, batch_size=batch_size, 21 | target_size=(224, 224), shuffle=True) 22 | test_iter = train_gen.flow_from_directory(valid_dir, batch_size=batch_size, 23 | target_size=(224, 224), shuffle=True) 24 | return train_iter, test_iter 25 | -------------------------------------------------------------------------------- /logs/README.md: -------------------------------------------------------------------------------- 1 | Slurm logs go in this directory 2 | -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Keras example model factory functions. 3 | """ 4 | 5 | def get_model(name, **model_args): 6 | if name == 'cnn': 7 | from .cnn import build_model 8 | return build_model(**model_args) 9 | elif name == 'resnet_small': 10 | from .resnet import ResNetSmall 11 | return ResNetSmall(**model_args) 12 | elif name == 'resnet50': 13 | from .resnet import ResNet50 14 | return ResNet50(**model_args) 15 | else: 16 | raise ValueError('Model %s unknown' % name) 17 | -------------------------------------------------------------------------------- /models/cnn.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple CNN classifier model. 3 | """ 4 | 5 | import keras 6 | from keras.models import Sequential 7 | from keras.layers import Dense, Dropout, Activation, Flatten 8 | from keras.layers import Conv2D, MaxPooling2D 9 | 10 | def build_model(input_shape=(32, 32, 3), n_classes=10, dropout=0): 11 | """Construct the simple CNN model""" 12 | conv_args = dict(kernel_size=3, padding='same', activation='relu') 13 | model = Sequential() 14 | model.add(Conv2D(16, input_shape=input_shape, **conv_args)) 15 | model.add(MaxPooling2D(pool_size=2)) 16 | model.add(Conv2D(32, **conv_args)) 17 | model.add(MaxPooling2D(pool_size=2)) 18 | model.add(Conv2D(64, **conv_args)) 19 | model.add(MaxPooling2D(pool_size=2)) 20 | model.add(Flatten()) 21 | model.add(Dense(128, activation='relu')) 22 | model.add(Dropout(dropout)) 23 | model.add(Dense(n_classes, activation='softmax')) 24 | return model 25 | -------------------------------------------------------------------------------- /models/resnet.py: -------------------------------------------------------------------------------- 1 | """ 2 | ResNet models for Keras. 3 | Implementations have been adapted from keras_applications/resnet50.py 4 | """ 5 | 6 | # Externals 7 | import keras 8 | from keras import backend, layers, models, regularizers 9 | 10 | def identity_block(input_tensor, kernel_size, filters, stage, block, 11 | l2_reg=5e-5, bn_mom=0.9): 12 | """The identity block is the block that has no conv layer at shortcut. 13 | 14 | # Arguments 15 | input_tensor: input tensor 16 | kernel_size: default 3, the kernel size of 17 | middle conv layer at main path 18 | filters: list of integers, the filters of 3 conv layer at main path 19 | stage: integer, current stage label, used for generating layer names 20 | block: 'a','b'..., current block label, used for generating layer names 21 | l2_reg: L2 weight regularization (weight decay) 22 | bn_mom: batch-norm momentum 23 | 24 | # Returns 25 | Output tensor for the block. 26 | """ 27 | filters1, filters2, filters3 = filters 28 | if backend.image_data_format() == 'channels_last': 29 | bn_axis = 3 30 | else: 31 | bn_axis = 1 32 | conv_name_base = 'res' + str(stage) + block + '_branch' 33 | bn_name_base = 'bn' + str(stage) + block + '_branch' 34 | 35 | x = layers.Conv2D(filters1, (1, 1), 36 | kernel_initializer='he_normal', 37 | kernel_regularizer=regularizers.l2(l2_reg), 38 | name=conv_name_base + '2a')(input_tensor) 39 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a', 40 | momentum=bn_mom, epsilon=1e-5)(x) 41 | x = layers.Activation('relu')(x) 42 | 43 | x = layers.Conv2D(filters2, kernel_size, 44 | padding='same', 45 | kernel_initializer='he_normal', 46 | kernel_regularizer=regularizers.l2(l2_reg), 47 | name=conv_name_base + '2b')(x) 48 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b', 49 | momentum=bn_mom, epsilon=1e-5)(x) 50 | x = layers.Activation('relu')(x) 51 | 52 | x = layers.Conv2D(filters3, (1, 1), 53 | kernel_initializer='he_normal', 54 | kernel_regularizer=regularizers.l2(l2_reg), 55 | name=conv_name_base + '2c')(x) 56 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c', 57 | momentum=bn_mom, epsilon=1e-5)(x) 58 | 59 | x = layers.add([x, input_tensor]) 60 | x = layers.Activation('relu')(x) 61 | return x 62 | 63 | def conv_block(input_tensor, kernel_size, filters, stage, block, 64 | strides=(2, 2), l2_reg=5e-5, bn_mom=0.9): 65 | """A block that has a conv layer at shortcut. 66 | 67 | # Arguments 68 | input_tensor: input tensor 69 | kernel_size: default 3, the kernel size of 70 | middle conv layer at main path 71 | filters: list of integers, the filters of 3 conv layer at main path 72 | stage: integer, current stage label, used for generating layer names 73 | block: 'a','b'..., current block label, used for generating layer names 74 | strides: Strides for the first conv layer in the block. 75 | l2_reg: L2 weight regularization (weight decay) 76 | bn_mom: batch-norm momentum 77 | 78 | # Returns 79 | Output tensor for the block. 80 | """ 81 | filters1, filters2, filters3 = filters 82 | if backend.image_data_format() == 'channels_last': 83 | bn_axis = 3 84 | else: 85 | bn_axis = 1 86 | conv_name_base = 'res' + str(stage) + block + '_branch' 87 | bn_name_base = 'bn' + str(stage) + block + '_branch' 88 | 89 | x = layers.Conv2D(filters1, (1, 1), strides=strides, 90 | kernel_initializer='he_normal', 91 | kernel_regularizer=regularizers.l2(l2_reg), 92 | name=conv_name_base + '2a')(input_tensor) 93 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 94 | x = layers.Activation('relu')(x) 95 | 96 | x = layers.Conv2D(filters2, kernel_size, padding='same', 97 | kernel_initializer='he_normal', 98 | kernel_regularizer=regularizers.l2(l2_reg), 99 | name=conv_name_base + '2b')(x) 100 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 101 | x = layers.Activation('relu')(x) 102 | 103 | x = layers.Conv2D(filters3, (1, 1), 104 | kernel_initializer='he_normal', 105 | kernel_regularizer=regularizers.l2(l2_reg), 106 | name=conv_name_base + '2c')(x) 107 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 108 | 109 | shortcut = layers.Conv2D(filters3, (1, 1), strides=strides, 110 | kernel_initializer='he_normal', 111 | kernel_regularizer=regularizers.l2(l2_reg), 112 | name=conv_name_base + '1')(input_tensor) 113 | shortcut = layers.BatchNormalization( 114 | axis=bn_axis, name=bn_name_base + '1')(shortcut) 115 | 116 | x = layers.add([x, shortcut]) 117 | x = layers.Activation('relu')(x) 118 | return x 119 | 120 | def ResNet50(input_shape=(224, 224, 3), n_classes=1000, 121 | l2_reg=5e-5, bn_mom=0.9): 122 | """Instantiates the ResNet50 architecture. 123 | 124 | # Arguments 125 | input_shape: input shape tuple. It should have 3 input channels. 126 | n_classes: number of classes to classify images. 127 | l2_reg: L2 weight regularization (weight decay) 128 | bn_mom: batch-norm momentum 129 | 130 | # Returns 131 | A Keras model instance. 132 | """ 133 | img_input = layers.Input(shape=input_shape) 134 | 135 | if backend.image_data_format() == 'channels_last': 136 | bn_axis = 3 137 | else: 138 | bn_axis = 1 139 | 140 | x = layers.ZeroPadding2D(padding=(3, 3), name='conv1_pad')(img_input) 141 | x = layers.Conv2D(64, (7, 7), strides=(2, 2), padding='valid', 142 | kernel_initializer='he_normal', 143 | kernel_regularizer=regularizers.l2(l2_reg), 144 | name='conv1')(x) 145 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 146 | x = layers.Activation('relu')(x) 147 | x = layers.ZeroPadding2D(padding=(1, 1), name='pool1_pad')(x) 148 | x = layers.MaxPooling2D((3, 3), strides=(2, 2))(x) 149 | 150 | x = conv_block(x, 3, [64, 64, 256], stage=2, block='a', strides=(1, 1), 151 | l2_reg=l2_reg, bn_mom=bn_mom) 152 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='b', 153 | l2_reg=l2_reg, bn_mom=bn_mom) 154 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='c', 155 | l2_reg=l2_reg, bn_mom=bn_mom) 156 | 157 | x = conv_block(x, 3, [128, 128, 512], stage=3, block='a', 158 | l2_reg=l2_reg, bn_mom=bn_mom) 159 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='b', 160 | l2_reg=l2_reg, bn_mom=bn_mom) 161 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='c', 162 | l2_reg=l2_reg, bn_mom=bn_mom) 163 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='d', 164 | l2_reg=l2_reg, bn_mom=bn_mom) 165 | 166 | x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a', 167 | l2_reg=l2_reg, bn_mom=bn_mom) 168 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b', 169 | l2_reg=l2_reg, bn_mom=bn_mom) 170 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c', 171 | l2_reg=l2_reg, bn_mom=bn_mom) 172 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d', 173 | l2_reg=l2_reg, bn_mom=bn_mom) 174 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e', 175 | l2_reg=l2_reg, bn_mom=bn_mom) 176 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f', 177 | l2_reg=l2_reg, bn_mom=bn_mom) 178 | 179 | x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a', 180 | l2_reg=l2_reg, bn_mom=bn_mom) 181 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b', 182 | l2_reg=l2_reg, bn_mom=bn_mom) 183 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c', 184 | l2_reg=l2_reg, bn_mom=bn_mom) 185 | 186 | x = layers.GlobalAveragePooling2D(name='avg_pool')(x) 187 | x = layers.Dense(n_classes, activation='softmax', 188 | kernel_regularizer=regularizers.l2(l2_reg), 189 | name='fc1000')(x) 190 | 191 | return models.Model(img_input, x, name='resnet50') 192 | 193 | 194 | def ResNetSmall(input_shape=(32, 32, 3), n_classes=10, 195 | l2_reg=5e-5, bn_mom=0.9): 196 | """Instantiates the small ResNet architecture. 197 | 198 | # Arguments 199 | input_shape: input shape tuple. It should have 3 input channels. 200 | n_classes: number of classes to classify images. 201 | l2_reg: L2 weight regularization (weight decay) 202 | bn_mom: batch-norm momentum 203 | 204 | # Returns 205 | A Keras model instance. 206 | """ 207 | img_input = layers.Input(shape=input_shape) 208 | 209 | if backend.image_data_format() == 'channels_last': 210 | bn_axis = 3 211 | else: 212 | bn_axis = 1 213 | 214 | x = img_input 215 | x = layers.Conv2D(64, (3, 3), strides=(1, 1), padding='same', 216 | kernel_initializer='he_normal', 217 | kernel_regularizer=regularizers.l2(l2_reg), 218 | name='conv1')(img_input) 219 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 220 | x = layers.Activation('relu')(x) 221 | 222 | x = conv_block(x, 3, [64, 64, 64], stage=2, block='a', 223 | l2_reg=l2_reg, bn_mom=bn_mom) 224 | x = identity_block(x, 3, [64, 64, 64], stage=2, block='b', 225 | l2_reg=l2_reg, bn_mom=bn_mom) 226 | 227 | x = conv_block(x, 3, [128, 128, 128], stage=3, block='a', 228 | l2_reg=l2_reg, bn_mom=bn_mom) 229 | x = identity_block(x, 3, [128, 128, 128], stage=3, block='b', 230 | l2_reg=l2_reg, bn_mom=bn_mom) 231 | 232 | x = conv_block(x, 3, [256, 256, 256], stage=4, block='a', 233 | l2_reg=l2_reg, bn_mom=bn_mom) 234 | x = identity_block(x, 3, [256, 256, 256], stage=4, block='b', 235 | l2_reg=l2_reg, bn_mom=bn_mom) 236 | 237 | x = conv_block(x, 3, [512, 512, 512], stage=5, block='a', 238 | l2_reg=l2_reg, bn_mom=bn_mom) 239 | x = identity_block(x, 3, [512, 512, 512], stage=5, block='b', 240 | l2_reg=l2_reg, bn_mom=bn_mom) 241 | 242 | x = layers.GlobalAveragePooling2D(name='avg_pool')(x) 243 | x = layers.Dense(n_classes, activation='softmax', 244 | kernel_regularizer=regularizers.l2(l2_reg), 245 | name='fc1000')(x) 246 | 247 | return models.Model(img_input, x, name='resnet50') 248 | -------------------------------------------------------------------------------- /notebooks/Analysis.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Results analysis\n", 8 | "\n", 9 | "You can use this notebook to analyze the results of your training runs on Cori.\n", 10 | "\n", 11 | "More documentation to come." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "# System\n", 21 | "import os\n", 22 | "\n", 23 | "# Externals\n", 24 | "import numpy as np\n", 25 | "import matplotlib.pyplot as plt" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# Magic\n", 35 | "%matplotlib inline" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Specify the results to load\n", 43 | "\n", 44 | "I'll start with plotting just one experiment. Later we may want to allow to plot multiple runs, like in Tensorboard." 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 22, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "results_dir = '/global/cscratch1/sd/sfarrell/sc18-dl-tutorial/cifar10-cnn-N1-16150683'" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 23, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "\u001b[0m\u001b[01;34mcheckpoints\u001b[0m/ history.npz out.log\n" 66 | ] 67 | } 68 | ], 69 | "source": [ 70 | "ls $results_dir" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "## Load the summary results and inspect them" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 24, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "history = np.load(os.path.join(results_dir, 'history.npz'))" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 25, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "n_ranks = int(history['n_ranks'])" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 26, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "data": { 105 | "text/plain": [ 106 | "['n_ranks', 'val_loss', 'val_acc', 'loss', 'acc', 'lr']" 107 | ] 108 | }, 109 | "execution_count": 26, 110 | "metadata": {}, 111 | "output_type": "execute_result" 112 | } 113 | ], 114 | "source": [ 115 | "history.files" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "## Plot loss and accuracy" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 27, 128 | "metadata": {}, 129 | "outputs": [ 130 | { 131 | "data": { 132 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAFgCAYAAACmKdhBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd8VfX9x/HXN3tCJishg41siICCgFWWKCIi4qraWqq1dVRb7dTa8VPbWmvdA7XWzXQg4mCIgAKCbAibsBJICNnrfn9/nKARw8y9OTfJ+/l4nMfNPfec7/kkHrn3c7/f7+drrLWIiIiIiIhI3QW4HYCIiIiIiEhjoQRLRERERETES5RgiYiIiIiIeIkSLBERERERES9RgiUiIiIiIuIlSrBERERERES8RAmWiIiIiIiIlyjBEhERERER8RIlWCIiIiIiIl4S5HYApyshIcGmpaW5HYaIiNTRihUrDlprE92Ow9f0viUi0jic6vtWg0uw0tLSWL58udthiIhIHRljdrodQ33Q+5aISONwqu9bGiIoIiIiIiLiJUqwREREREREvEQJloiIiIiIiJc0uDlYIiK+VlFRQVZWFqWlpW6H0iiEhYWRnJxMcHCw26H4Dd1j3qV7TET8iRIsEZFjZGVlER0dTVpaGsYYt8Np0Ky1HDp0iKysLNLT090Ox2/oHvMe3WMi4m80RFBE5BilpaXEx8frg68XGGOIj49XT80xdI95j+4xEfE3SrBERGqhD77eo79l7fR38R79LUXEnyjBEhERERER8RIlWCIifubw4cM8+eSTp33eRRddxOHDh30QkTQ2usdERHxHCZaIiJ853offqqqqE543e/ZsYmJifBVWo2GMGWWM2WSM2WKMubeW1/9ljFlVvW02xhyu8VpVjdfeqd/IvUf3mIiI7zS5KoLWWo6UVtI8XKVcReTk/vTuOtbvPeLVNs9q04z7Lul23Nfvvfdetm7dSu/evQkODiYqKorWrVuzatUq1q9fz7hx49i9ezelpaXcfvvtTJ48GYC0tDSWL19OYWEho0ePZvDgwSxevJikpCRmzZpFeHi4V3+PhsgYEwg8AQwHsoBlxph3rLXrjx5jrb2zxvG/APrUaKLEWtvbmzHpHhMROQPWOluA//UXNbkE67Y3VrFubz6f3jXM7VBERGr14IMPsnbtWlatWsX8+fMZM2YMa9eu/aYE9ZQpU4iLi6OkpISzzz6byy+/nPj4+O+0kZmZyeuvv85zzz3HxIkTmTZtGtdee60bv46/6Q9ssdZuAzDGvAFcCqw/zvFXAffVU2z1RveYiDQIVRWQvxtyt0Pejuqt+ufcHRAaBVe/Ba17uhvnMZpcgtUmJowP1+6nymMJDFDVIRE5sRP1AtSX/v37f2d9n8cee4wZM2YAsHv3bjIzM7/34Tc9PZ3evZ2Oln79+rFjx456i9fPJQG7azzPAgbUdqAxJhVIBz6tsTvMGLMcqAQetNbOPM65k4HJACkpKScMSPeYiEi1A+th2XNwaKuTROVnga0xdDkwFGJTITYNUs6BjbPhv5fC9e9Cq+5uRf09TS7BSo2LpLzKw/4jpSTFaCiDiPi/yMjIb36eP38+H3/8MUuWLCEiIoJhw4bVuv5PaGjoNz8HBgZSUlJSL7E2ALV9s2aPc+wkYKq1Nd/dSbHW7jXGtAM+NcassdZu/V6D1j4LPAuQkZFxvPb9hu4xEXGVxwNfPAUf/wkCgyGxCySfDT0nOslUbBrEpkN06+8OCRxwM7x0Mfx3LFz/HrQ8y63f4DuaXoIVHwHAzkNFSrBExC9FR0dTUFBQ62v5+fnExsYSERHBxo0bWbp0aT1H1+BlAW1rPE8G9h7n2EnArTV3WGv3Vj9uM8bMx5mf9b0Ey9/pHhMRv5GfBTNvge0LofNFcMljEJV4aufGt4cb3oMXL/o2yWrRxbfxnoIml2ClxDkJ1u7cYmjvcjAiIrWIj49n0KBBdO/enfDwcFq2bPnNa6NGjeLpp5+mZ8+edO7cmYEDB7oYaYO0DOhojEkH9uAkUVcfe5AxpjMQCyypsS8WKLbWlhljEoBBwMP1ErWX6R4TEb+wZiq8/0uoqnQSq74/hNNdOLxmkvXyJXDD+5DYyTfxniJjrd+PXPiOjIwMu3z58jM+v7LKQ5c/zGHykHb8epT7Ga6I+J8NGzbQtWtXt8NoVGr7mxpjVlhrM+o7FmPMRcCjQCAwxVr7V2PMA8Bya+071cfcD4RZa++tcd65wDOAB2eZk0ettS+c7Hq1vW/pHvM+/U1FGpCSPHj/blg7FZL7w/hnIK5d3drM2QQvjQETCDfOdhIvLzvV960m14MVFBhAcmw4O3OL3Q5FRERcYK2dDcw+Zt8fj3l+fy3nLQZ6+DQ4EZHGbtsCZ0hg4QE4//cw+E4I9EJKktjZKXbx0hhnXtYN7/kkyToV/lc4vh6kxEey65ASLBERERGRelFRCnN+68yVCo6AH38EQ3/lneTqqBZdnSSrstQZLpi73Xttn4YmmWClxkWw81CR22GIiIiIiDR++9fAc+fD0ifg7J/ATxdCUl/fXKtlN7j+HagodpKsvJ2+uc4JNM0EKz6CI6WVHC4udzsUEREREZHGqSQP5v4envsBFB+Ca6bBmH9ASIRvr9uqB1w3E8qOwMsXw+HdJz/Hi5pkgnW0kuBODRMUEREREfGuilL4/N/w716w+HHocQXcsgQ6Xlh/MbTp7SRZJflOkpW/p94u3SQTrNR4Z0FFFboQEREREcHpbXplPMz5DexZAWdSadxTBateh8cz4KM/QtsBcMvnMO5JiIz3fswnk9QXrpsBxblOklWaXy+XbZIJ1tEerF2ahyUijUBUVBQAe/fuZcKECbUeM2zYME62xMWjjz5KcfG3XzxddNFFHD582HuBSoOle0ykCfjiGdj6CSx73hnS95++8OlfnfLnJ2MtZH4MzwyBmTdDZIJTbOKat505UW5K7gfXToOekyC0Wb1cskkmWOEhgbSIDtUQQRFpVNq0acPUqVPP+PxjP/zOnj2bmJgYb4QmjYTuMZFGqqwAlj4FnS+CuzNh7OPQvC189g94oj88NRgWPVr7XKa9K+G/l8Krl0N5IUyYAjd9CulD6v/3OJ62/WHYPae/iPEZanLrYB2VEhfBLg0RFJGT+eBep/qRN7XqAaMfPO7L99xzD6mpqfzsZz8D4P7778cYw8KFC8nLy6OiooK//OUvXHrppd85b8eOHVx88cWsXbuWkpISbrzxRtavX0/Xrl0pKSn55rhbbrmFZcuWUVJSwoQJE/jTn/7EY489xt69ezn//PNJSEhg3rx5pKWlsXz5chISEnjkkUeYMmUKADfddBN33HEHO3bsYPTo0QwePJjFixeTlJTErFmzCA8P9+7fq7HTPaZ7TBoOa2HGT2HHImjZ3fl/rXVP5zEmDQIaaN/Fipeg9DAM/iWEx0Df65ytYD+smwFrpsLH9zlb24HQYwIkZzjzq9ZOhYh4GPUQZPwIgkLc/m1c13QTrPgIlmw95HYYIiLfM2nSJO64445vPvy+9dZbzJkzhzvvvJNmzZpx8OBBBg4cyNixYzHH+TbuqaeeIiIigtWrV7N69Wr69v22HO5f//pX4uLiqKqq4oILLmD16tXcdtttPPLII8ybN4+EhITvtLVixQpefPFFvvjiC6y1DBgwgKFDhxIbG0tmZiavv/46zz33HBMnTmTatGlce+21vvvjiFfoHhM5Q0seh9VvQrthkL8btnwMtsp5LSQaWlUnXa2qk64WXSEo1M2IT66i1EmU0odA27O/+1p0Kxh4i7PlboO105xka/bdzutB4XDeXTDodghrXv+x+6kmm2ClxkUyY+UeSiuqCAsOdDscEfFXJ+gF8JU+ffqQnZ3N3r17ycnJITY2ltatW3PnnXeycOFCAgIC2LNnDwcOHKBVq1a1trFw4UJuu+02AHr27EnPnj2/ee2tt97i2WefpbKykn379rF+/frvvH6sRYsWcdlllxEZ6RQIGj9+PJ999hljx44lPT2d3r17A9CvXz927Njhpb9CE6J7TPeYNAw7l8BH90HXsTDxv85ws4pSyNkA+1Y7PdH718Cq16D8WeecgCDoMBxG/Q3i2rkb//F8/RoU7ofxz5z4uLh2MORXcN7dcGAd7FoCXcZAszb1E2cD0nQTrPgIrIWsvGI6tIh2OxwRke+YMGECU6dOZf/+/UyaNIlXX32VnJwcVqxYQXBwMGlpaZSWlp6wjdp6HrZv384//vEPli1bRmxsLDfccMNJ27EnqCQVGvrtN7OBgYHfGSYm/k33mMhpKMyBqTdCbCpc+vi3c3mCw6BNH2c7yuOBvO2wf7VTjW/5i/DEQBh8Jwy+A4L9aIhrVaUztyqpH6QPPbVzjKnuqevu29gasAY6ULTuUuK1FpaI+K9JkybxxhtvMHXqVCZMmEB+fj4tWrQgODiYefPmsXPniVemHzJkCK+++ioAa9euZfXq1QAcOXKEyMhImjdvzoEDB/jggw++OSc6OpqCgoJa25o5cybFxcUUFRUxY8YMzjvvPC/+tuIG3WMip8hTBdN+7JQxn/jfkw+FCwiA+PbQ7TIY8Rf4+XLoejEseBCeHAib59ZP3Kdi7TQ4vNMZ5ldPBSCagqbbg6XFhkXEj3Xr1o2CggKSkpJo3bo111xzDZdccgkZGRn07t2bLl26nPD8W265hRtvvJGePXvSu3dv+vfvD0CvXr3o06cP3bp1o127dgwaNOibcyZPnszo0aNp3bo18+bN+2Z/3759ueGGG75p46abbqJPnz4aqtXA6R4TOUULHoLtC2Dsf5x5VaerWWunsl7fH8LsX8FrV0DnMc7w4JgU78d7qjweWPQItDgLOo12L45GyJyoW94fZWRk2JOts3EqrLX0uH8uE/olc/9Yl+vzi4hf2bBhA127dnU7jEaltr+pMWaFtTbDpZDqTW3vW7rHvE9/U/mefV/D9oXQf/KZF5rY8jH8bwL0vhoufaLuvTyV5bD0CVjwsFORcMjdcO4v3CmEseE9ePMaGP889Lyi/q/fAJ3q+1aTHSJojCElLoKdWmxYREREpHE5ss9JjOb+Hp6/EA5uOf028rNg2k+cHp6L/uGdIXRBIc5crFu/hI7D4dM/w1PnwtZP69726bAWPvsnxKY5QxnFq5psggVOoYudWgtLREREGhOPBz76Iyx9Gopz3Y6m/lWWw9vXQ3mRszZT/m54ZgisfNVJLE65jRuhqsKZdxUS4d0YY9rCla/ANdPAeuCVy+Ct6511p+rDtvmw9ysYdAcENtkZQz7TpBOslLgIsnJLqPI0rGGSIuJ7DW34tD/T37J2+rt4j/6Wx9j4Lnz+b5hzD/yzC0yfDDs+P/XkoqGb+zvY/QWMewIG3gy3LIakvjDrZ06xitL8k7fx8X2Q9SVc+h9I6OC7WDteCLcsgfN/B5vnwMuXQMlh313vqM/+CVGtnKGP4nVNO8GKj6C8ysOBIycuHysiTUtYWBiHDh3ShzYvsNZy6NAhwsLC3A7Fr+ge8x7dY8fweJz5PfEdYPIC6HsdbPoAXroIHj8bFv8Hig55/7rlRZC1HFa8BO/fDVNGw5PnwK4vvH+tE/n6TfjyWTjn598OfWvWBn44C37wB1g3E54eDLuXHb+N9bNg6ZMw4Ob6GT4XHAZDfw3XTofc7fD2DU75dF/Z9QXs+My9uV9NQJPuE0yNcxY03HmomDYxfrQmgYi4Kjk5maysLHJyctwOpVEICwsjOTnZ7TD8iu4x79I9VsOm9+HAWrjsGWjT29mGP+AkFl+97MxJ+uQB6HIx9Lse0oY4ZcVPlbVOWe8D65xt/xrnMXcbUP2FQUgUtOwGZQXw2kT40RxoUQ8FSPavgXdvh9TBcOGfvvtaQKBTUCJ9iNOLNWUk/OB3zhC5gMBvjzu0FWbeCkkZMPzPvo+5prRBcMmjMOtWp/dxzD99c51Fj0B4HPS7wTftSxNPsKrXwtqVW8Q57eNdjkZE/EVwcDDp6eluhyGNmO4x8QlrnZLice2g+4Rv94dEQp9rnC17A6x4Gb5+HdZNh9h0Z39EvJMQlRVWPxZA2ZFvfy6v3l+SBxVH568biEt3kqmeVzqPLbtBTKqTtOXtgBdGwCvj4ccf+rYkeUkevHkthMfAFS8ef15R2/5w8yJ49w4n0dw2Hy571imlXlECb/3QOfeKl5yCFPWtz7WQswkWPwYJnWHAZO+2v3+NMxTx/N9BaJR325ZvNOkEq3XzMIICjNbCEhERkYZv0wfOB+hLnzx+gtGiq7P+0oX3w4Z3nGTr07/UOMBAaDMIjXY+gIdGQ1gzaJ4EIdHOIrsJHaFld6etE31Ij01zhr29eJFTxOFHH0Jkgvd+36M8Hpj+U8jfAzfOhqgWJz4+rLmzLlWHC5x1qZ46F8Y9CRvfc3r/rpnqFKFwy4X3Oz1pc+5xkuWOF3qv7UX/cnoY+//Ee23K9/gswTLGTAEuBrKttd2Pc8ww4FEgGDhorR3qq3hqExQYQHJsuCoJioiISMN2tPcqNs3pTTqZ4DDoOdHZCg4A1vngHRLpnXLkR7XqDle/4SRYr06A6991kjZvWvh3yPzQKaXetv+pnWOM01vUdgBM/RG8PsnZf97dTvl0NwUEwvhnYcoomHoj/PgjaHHihb9PyaGtsG6GM/cqPLbu7clx+bLIxUvAqOO9aIyJAZ4ExlpruwGurHCWEh/JLvVgiYiISEOWORf2rXIShNMtux3dEqJbOb1R3kyujko91xlyt2+1M4yvssx7bWd+BPP/D3pdBWffdPrnJ3SEmz525mL1vgbO/633YquL0CgnMQ0Od+axFR2se5uL/gUBwTDw1rq3JSfkswTLWrsQONHiC1cD0621u6qPz/ZVLCeSqsWGRUREpCGzFuY/6Mxx6jXJ7Whq13k0jP2PM+dpxs3gqap7m7nbYdpNznDFMY+ceXIYFArD/+QME6xZ8MJtzZNh0utQeKDuiWl+Fnz9hlNVMrql92KUWrlZpr0TEGuMmW+MWWGM+aEbQaTERXCktJLDxeVuXF5ERESkbrZ84iwae95dEBjsdjTH1+ea6oqG0+GDe+q2LldFCbx1HWCdBXu9vRCwv0juB+Oegl1LnAqJZ/o3W/w4YGHQ7V4NT2rnZpGLIKAfcAEQDiwxxiy11m4+9kBjzGRgMkBKincr0KRUVxLceaiYmAgXqsWIiIiInClrYcGD0Lwt9GoAi8YOuh0Ks2HJ4xCZCMPuOf02rIX3fgn718LVbzmVDBuz7uPh0BaY91dI6ATn/fL0zi866KxP1mOibys5yjfc7MHKAuZYa4ustQeBhUCv2g601j5rrc2w1mYkJiZ6NYhvS7VrHpaIiIg0MFs/haxlMPhOd8qKn4nhf3bmTM3/Gyx74fTPX/4CfP0aDLsXOo3wfnz+aMivoMcV8MmfYP07p3fu0qegshQG3+Gb2OR73OzBmgU8bowJAkKAAcC/6juIlDglWCIiIuKS4lwICjuzIW5HKwc2S3Iq4jUUAQHOfKziXHj/LmcNrm7jaj/WWig+BAcz4eBmZ/viGeg4Aob8un7jdpMxMPZxZ22xGT91eqLa9P7uMWWFkLvV6e06VONx/2roegkkdnYl9KbIl2XaXweGAQnGmCzgPpxy7Fhrn7bWbjDGzAFWAx7geWvtWl/FczwRIUEkRoeq0IWISBNhjBkF/BsIxHnvefCY1/8FnF/9NAJoYa2NqX7teuD31a/9xVr7cv1ELY1ScS48ORCCI+DaaRDf/vTO374Adn/hlCcPCvVNjL4SGOxUFnxlHEz/iVM1r3kKHDqaSG1xHg9lOosIf3NeKKQMdMqYB7g5EMsFwWEw6TV47gdOWfkBN0Putm+TqcL93z2+WbJzT/W9/vSHFUqdGFuXCYYuyMjIsMuXL/dqmxOeWkxggOHNn57j1XZFROT4jDErrLUZ9XzNQGAzMBxnqPoy4Cpr7frjHP8LoI+19kfGmDhgOZABWGAF0M9am1fbuUf54n1LGomZt8LXrzsL+ZoAuOpNaHv2qZ//4kXOB+zbVjkfvhuikjzn98g+5n/BqJbOfKP4Ds5jQkdna97Wvyr9ueHAOpgyGsrynd6/+A7VW3vnMa69s0BxYy384aJTfd9yc4ig30iJj2DJ1kNuhyEiIr7XH9hird0GYIx5A7gUqDXBAq7CGYEBMBL4yFqbW33uRzjrPb7u04ilcdo6D1b9z5k71ec6+N/l8PIlMOEF6DLm5Odv/wx2fg6jH264yRU4C95eNxNWv+EkVfEdIaEDhDV3OzL/1bIb/HIdVFVARJzb0Ugtmljfau1S4yLZf6SU0govrMkgIiL+LAnYXeN5VvW+7zHGpALpwKdncO5kY8xyY8zynJycOgctjUx5Mbx3h9PTMPQep+fhxx9By7PgjWvgy+dO3saChyCqlTP8q6GLbulUF+w1ySlLruTq5EKjlVz5MSVYOJUErYWsPBW6EBFp5GpbifR4Y+UnAVOttUe/fTvlc31Z/VYagfl/c4oVjH0MgsOdfVGJcP17zoK8s++GuX8Aj6f283d8Djs+c5KShtx7JdJIKcEC2sZ9uxaWiIg0allA2xrPk4G9xzl2Et8d/nc654rUbu9KWPKE0/OUNvi7r4VEwJX/g7NvgsWPwfSboLLs+20seAgiW0DGjfUTs4icFiVYfLsWlhIsEZFGbxnQ0RiTbowJwUmivreojDGmMxALLKmx+0NghDEm1hgTC4yo3idyaqoq4J1fOMnR8AdqPyYg0KkKeOGfYO00eOWy71bR27XUqR446PZve79ExK8owQLiI0OIDAnUWlgiIo2ctbYS+DlOYrQBeMtau84Y84AxZmyNQ68C3rA1Su1WF7f4M06Stgx44GjBC5FTsvg/sH8NjPkHhMcc/zhjnEVhxz8Pu7+EF0bC4V3OawsegogE9V6J+DFVEQSMMaTERyrBEhFpAqy1s4HZx+z74zHP7z/OuVOAKT4LThqvQ1th/oPOgq9dLzm1c3peAdGtnMIXz1/oFMTY+qnT+xUS6dt4RRogay1F5VXkFpaTW1xOblEZhwrLyS0qp9JjufX8DvUShxKsaqlxEWRmF7gdhoiIiDQ2Hg+8cxsEhcHov5/euennwY8/hP9NgPd/6ax7lPFj38Qp4gc8HktReSWFZZUUlFZSUFrBkdJvfy4oraSw+uf8kgoOFTkJVG5ROYeKyimvrL04TExEsBKs+pYaH8Gnm7LxeCwBAbUVihIRERE5AytfgZ2L4JJ/Q7PWp39+i65w08cw61boMQFCo7wfo4gLPB7LtoOFfLXrMCt3HWblrjwyswup8hyvuKsjwEBUaBDNwoOJjwyhRXQoXVo1Iz4qhLhIZ4v/5jGUuChnOlB9UYJVLSU+gvJKD/uPlNImRpNGRURExAuO7HNKrqedV7c1q5q1huumey8uERccLi5n1e7D1QlVHqt2H6agtBKAZmFB9E6J5fwuLYiLCCEqLIjosCCiw4KdZKr65+iwICJCAjHGfztElGBVS41zxjLvPFSsBEtERES844NfQWWp03vlxx8IRbyhvNJDTmEZ2UdKyS4oI7ugjJwjpWTllbAq6zDbcooApweqU8toLu7Zhr4pMfRJiaVdQmSjGUWmBKva0VLtu3KLOKd9vMvRiIiISIO3/h3Y8C5ccB/Et3c7GpETKq/0sGl/AQWlFZRVeSivrLFVP6+o8lBWva+0soqcgjJyCsrIPlJGdkEpecUV32s3wEBidCg9kmK4vG8yfVJi6JkcQ1Ro401DGu9vdppaNw8jKMBoLSwRERGpu5LDMPtX0KoHnPsLt6MR+Z6yyipWZ+WzdOshvtiey/KduZRW1F4gojYhgQHER4XQolkYKfERZKTF0iI6jBbNQmkRHUqL6DBaNgslLjKEoMCmtTKUEqxqQYEBJMWGs1Ol2kVERKSuPvojFGXD1W9AYLDb0UgjU+WxGDitIXVllVWs2nWYL7bnsnTbIb7alfdNQtWlVTSTzk7h7LQ44iJDCAkKIDQogJCgAEICqx9rPg8MaDTD+XxBCVYNKXER7FIPloiIiNTF9s/gq5ednqs2fdyORho4a60zh2n3YVbtdopDrN17hPJKDxEhgUSEBBEZWv0YEkhEaPVj9f6QwADW7T3CV7vyKKv0YAx0adWMq/qnMCA9ngHpccRGhrj9azYqSrBqSI2PYHXWPrfDEBERkYYqZzNMuwli02DYb92ORhqggtIKVmflVydTh1m1O4+DheUAhAYF0DO5OT8cmEpkaBDF5ZUUlVdRXFb9WF7JkZIK9ueXUFRWRVF5JSXlVXRoEcU1A1IZ2C6O/ulxxEQoofIlJVg1pMZFkl9SQX5xBc0j1J0vIiIipyF7A7w81vn5qjcgJMLdeMR1BaUV7D1cSlF5JUVllU7SU1ZJcXklhWVOQnR0X2FZJZnZBWRmF2Krl4FqlxjJkE6J9EmJpU/bGDq3iia4ic1naoiUYNWQUl1JcGduET0jYlyORkRERBqM/Wvhv5dCQBBc/y4kdnI7InHJjoNFfLzhAJ9syGbZjlwqT7BorjEQFRJERGggkSFBpMRHMKZHG3qnxNA7OUZf+DdQSrBqOFqqfeehYnomK8ESERFxXWk+fP0mHNwMI/4CwWFuR/R9+1Y7yVVQGNzwnkqyNzFVHstXu/K+Saq2ZBcC0KllFD8Z0o7ubZoTGRpIZGgQkTXmS0WFBhEWHODXC+bKmVGCVUNK3NG1sFToQkRExDXWwp6vYMUUWDMNKkuc/UGhMPKv7sZ2rL0r4b/jIDQarn8H4tq5HZHUg4LSChZuPsgnGw4wb1M2ecUVBAcaBqTHc82AFC7s2pK2cRoi2lQpwaohIiSIxOhQdh4qcjsUERGRpqesAFa/BStehP1rIDgSek6EjBth5f9gyRPQaSSkD3E7UkfWcnhlPIQ3h+vfg9hUtyOSOiitqCKvuJzconLyiirILS4nr6icQ0XO49HnuUXlbM0ppKLKEhsRzPmdW3BB15YM6ZRAdJiG9IkSrO9JiYvQYsMiIiL1ae9KWP4irJkKFUXQsgeMeQR6XAFhzZxjEjrBtvkw4xb42WIIa+5qyOz6Av53OUQmOHOuYtq6G4+9thbmAAAgAElEQVSckewjpcxatZdpX2WxcX/BcY+LiQgmLjKEuIgQ2sZFMLRzIhd2bUnflFgCtR6UHEMJ1jFS4yJYsu2Q22GIiIg0bpVlsPpNWD7FSbCCwqH75U5vVVI/Z/Z/TSGRcNmz8MJwmP1rGP+MO3ED7FwMr14BUS2dOVfN2rgXi5y2kvIq5q7fz7Sv9rAoMwePhV5tY7jzwk4kRocSFxlMbEQI8VEhxEaE0Dw8mCBV7pPToATrGCnxEcxYtYfSiirCggPdDkdERKRxsRY2fQAf/hbytkNiVxj9d2coYPhJCkwl94Ohv4b5/wedR0G3y+on5pq2L4TXroTmyU7PVXSr+o9BTpvHY/liey7Tv8rig7X7KSyrJCkmnJ8N68BlfZNonxjldojSiCjBOkZqfATWQlZeMR1aRLsdjoiISOORvQHm/Aa2zYPELnDtNGh/wfd7q07kvLsgcy68dye0HQjNWvsu3mNtnQevX+UsInz9OxDVov6uLWdka04hM77aw4yVe9hzuITIkEAu6tGa8X2TGZAeR4CG94kPKME6RkpcJOBUElSCJSIi4gXFuU6v07IXIDQKRj8MGT+CwDMoCBAY7AwVfHowzLrVSdJ8Wea6vBiy18OupfDJA5DQEX44y5l7JX6ntKKKL7fnsnBzDgszc9h8oJAAA+d1TOTXozoz4qxWhIdohJL4lhKsY9RcC0tERETqoKrSqQg476/OelYZP4Jhv4XI+Lq1m9ABRv4F3r8Llj0P/X/inXhLjzjVC/d9/e12cBNYj/N6UgZc8zZExHnnelJn1lq2ZBeyYHMOCzMP8sW2Q5RVeggJDODs9FgmZrRlbK82tGjmh+unSaOlBOsY8ZEhRIYEKsESERGpi23z4YN7IWeDU1Z91IPQspv32s/4sTOXa+4foN0wp2fpdB3aChve+TaZyt327WtRraB1L+h6ifPYupcz70qLwrrucHE5i7YcZOHmHD7LPMi+/FIA2idGcvWAFIZ0SmRgerx6qsQ1SrCOYYwhJT5Siw2LiIicidxtTtKz8T1nrtKVr0KXMd5PTIyBS5+AJwfC9J/Ajz869SGHZQWw8O+w5EnwVEBMipNA9b4aWveGVj0huqV345UTstaSX1LBwcIysgvKyKneDhaWOz8XlnHw6GNhGdZCdFgQgzskcNsFiZzXMYHkWC3sK/5BCVYtUuLC2ZJd6HYYIiIiDUvWCnhxNAQEwQX3wcCfQbAPh2ZFt4JL/g1v/dBJmM7/7YmP93ic0vAf3weFB6D3NfCD36vMej2rqPKwaX8Ba/bkO1tWPpsOFFBe6fnescGBhsSoUBKiQ2ndPIyeyc1pExPOoA7x9EqOUfl08UtKsGqRGh/JvE05eDxW1WVEREROhbUw5x4Ij4XJ8+uvut9Zl0Kvq2DhP6DDcGh7du3H7VkBH9wDWcucdbYmvQbJGfUTYxNWWeVhS04hq7OcRGr1nnw27DvyTTIVHRZEz+TmXH9OKi2bhZEYHepsUc5j8/BgjIZlSgOjBKsWKXERlFd62H+klDYx4W6HIyIi4v/WTXeSl7GP12/pdIDRD8GORTBjMty8yFmU+KjCbPjkT7DyfxDZAi590knIAtTz4QvWWjYfKGTuuv0s2JzD2r35lFY4yVRUaBDdk5pxw7lpdE9qTs+k5qTGRyiBkkZHCVYtalYSVIIlIiJyEhWl8PH90LK7M4+pvoU1h8uehpcuhrm/h4v/BZXl8OWzsOAhqCiBc38BQ34NYc3qP75GrspjWb4jl4/WH2Du+gPfzGPv1TaGq/un0jO5OT2Sm5MeH6mRQdIkKMGqReo3a2EVcU77OpaSFRERaey+fAYO74LrZkKAS5Xb0gbDuT+Hxf9xKgCunQoHN0OHC50KhmdSZVCOq6S8ioWZOXy0/gCfbswmt6ickMAABnWI5+ah7bmwawuVRpcmSwlWLdrEhBEUYFSqXUSkETLGjAL+DQQCz1trH6zlmInA/YAFvrbWXl29vwpYU33YLmvt2HoJ2p8VHXTmP3UcCe3PdzeWH/wBtnwK8/8Gce3gqjeh00iVVveCkvIqtuYUsnZPPh9vyGbRlhxKKzxEhwVxQZcWDD+rFUM7JxIVqo+WIvq/oBZBgQEkxYarVLuISCNjjAkEngCGA1nAMmPMO9ba9TWO6Qj8Bhhkrc0zxrSo0USJtbZ3vQbt7+Y/COVFMOLPbkcCQaFw1euwfSH0nOg8l9NSWuEkUpkHCtl8oIDNBwrJzC5gV24x1jrHtGkexpUZbRnRrRX90+MIViU/ke9QgnUcKXERSrBERBqf/sAWa+02AGPMG8ClwPoax/wEeMJamwdgrc2u9ygbipzNsHwK9LsBEju7HY0jNhVir3M7igYhv6SCr3blsXJnHhv3F5CZXcjOQ0V4qhOpoABDWkIk3do0Y1zvJDq1jKZzqyjaJ0apMIXICSjBOo7U+Aje/Xqf22GIiIh3JQG7azzPAgYcc0wnAGPM5zjDCO+31s6pfi3MGLMcqAQetNbO9HG8/u2jP0JwBAz7jduRyElYa9mdW8Lynbks35nHih15bM4uwFoIDDCkxUfQpVU0l/RqQ6eWUXRqGU1afCQhQeqdEjldSrCOIyUugvySCvKLK2gecYorw4uIiL+r7Wt3e8zzIKAjMAxIBj4zxnS31h4GUqy1e40x7YBPjTFrrLVbv3cRYyYDkwFSUlK8Gb//2LYANn8AF94PUYluRyPHqKjysH7vESeZ2pnL8h15ZBeUARAdGkSf1FjG9GxNRmosvdrGEKm5UyJeo/+bjiOlupLgztwiekbEuByNiIh4SRbQtsbzZGBvLccstdZWANuNMZtwEq5l1tq9ANbabcaY+UAf4HsJlrX2WeBZgIyMjGMTuIbPUwVzfwfNU2DALW5HI9UOFZYxb1MOn2w4wMLNORSVVwGQFBPOOe3jyUiNpV9qHJ1bRROocukiPqME6zhqroXVM1kJlohII7EM6GiMSQf2AJOAYxdumglcBbxkjEnAGTK4zRgTCxRba8uq9w8CHq6/0P3I12/A/jVw+QsQrFLcbrHWkpldyMcbDvDJhmy+2pWHtdCyWShjeycxqEM8GalxtGqu/0Yi9UkJ1nGkxDkJlgpdiIg0HtbaSmPMz4EPceZXTbHWrjPGPAAst9a+U/3aCGPMeqAK+JW19pAx5lzgGWOMBwjAmYO1/jiXarzKi+DTP0NSP+h+udvRNDnllR6+3J7rJFUbD7A7twSA7knNuO0HHbmwa0u6JzVTEQoRFynBOo7I0CASokLZeajI7VBERMSLrLWzgdnH7PtjjZ8t8MvqreYxi4Ee9RGjX1v8OBTsgyte0vpS9Si/uIK/zd7A7DX7KCirJDQogEEdErh5aHsu6NJSvVQifkQJ1gmkxkewLUcJloiICABH9sHnj8JZl0LKQLejaTKW78jl9jdWceBIKeP7JjH8rFYM6hBPRIg+xon4I/2feQID28Xx1PytZBeU0iJa3wyJiEgTN+8vUFXhVA4Un6vyWJ6ct4VHP8kkKSacabecS6+2mhcu4u+0uMEJjOudhMei9bBERET2r4GVr8KAn0JcO7ejafT255dyzfNL+edHm7m4Z2vev22wkiuRBkIJ1gl0bBlN96RmzFq1x+1QRERE3GMtfPg7CI+BIXe7HU2j9/H6A4z+90JWZ+Xzjyt68eiVvYkO05qcIg2FzxIsY8wUY0y2MWbtSY472xhTZYyZ4KtY6mJc7yRWZ+WzJbvQ7VBERETckTkXti+AofdCeKzb0TRapRVV3P/OOm7673LaxITz7i8GM6FfsioCijQwvuzBegkYdaIDjDGBwEM4JXH90iW92hBgUC+WiIg0TVWVMPcPENceMn7kdjSN1tacQsY/uZiXFu/gxkFpTP/ZubRPjHI7LBE5Az5LsKy1C4Hckxz2C2AakO2rOOqqZbMwBnVIYMbKPTiVe0VERJqQ1W/CwU0w/E8QFOJ2NI2OtZa3lu/m4scWsf9IKS9cn8F9l3QjNCjQ7dBE5Ay5VkXQGJMEXAb8ADjbrThOxbjeSdz19tes2JlHRlqc2+GIiIjUj6pKWPh3aNUTulzsdjQNwpHSCqYs2s6GfUfwWCeB8ljwVD86zy0ej7OvqLyStXuOcE67eB6d1JuWzVS1WKShc7NM+6PAPdbaqpONLTbGTAYmA6SkpNRDaN81snsrfjdzDTNX7VGCJSIiTceatyFvO1z5qhYVPomyyipeXbqL/3yaSV5xBR1aRBEUYAgwhoAACDAGYwwBxvk50BiMgajQIH4zugs3ndeOwAD9jUUaAzcTrAzgjerkKgG4yBhTaa2deeyB1tpngWcBMjIy6n2cXlRoEMPPasV7q/fxx4u7ERKk4osiItLIHe29atkDuoxxOxq/5fFY3l29l3/M3cTu3BLObR/Pb0Z3pUdyc7dDExGXuJZgWWvTj/5sjHkJeK+25MpfXNanDe9+vZcFm3MYflZLt8MRERHxrbXTIHcrTHxFvVfH8fmWgzz4wUbW7MmnS6toXrrxbIZ2SlTVP5EmzmcJljHmdWAYkGCMyQLuA4IBrLVP++q6vnJex0TiIkOYuXKPEiwREWncPFXVvVfdNfeqFuv3HuHBORtZuDmHpJhwHpnYi3G9kwjQED8RwYcJlrX2qtM49gZfxeEtwYEBXNKzNa8v282R0gqaacE/ERFprNZOh0OZMPG/EKBh8Udl5RXzyNzNzFi1h2Zhwfzuoq5cd04qYcGq+Cci33JzDlaDM65PEi8v2cmctfuZmNHW7XBERES8z1MFCx+GFmdBl0vcjsYv7D1cwnOfbePVpbvAwOQh7fjZ0A40j9CXrSLyfUqwTkPvtjGkxkcwc+UeJVgiItI4rZsBBzfDFS81+d6rbTmFPL1gKzNW7sFjYXyfJO4c3ok2MeFuhyYifkwJ1mkwxjCudxKPfZrJvvwSWjfXP7AiItKIeDzO3KvErtD1Urejcc3aPfk8NX8rs9fuIyQwgKv7p/CTIe1Ijo1wOzQRaQCaXoJ1eBfk7YD0IWd0+rg+Sfz7k0zeWbWXnw5t793YRERE3LR+JuRshAlTmlzvlbWWL7fn8uT8rSzYnEN0aBC3DG3PjYPSSYwOdTs8EWlAml6C9dEfYecSuGvjGZWdTU+IpHfbGGas3KMES0REGg+PBxY8DAmd4axxbkdTb6y1zNuUzRPztrJiZx7xkSH8amRnrjsnVQWtROSMNL0Eq+NIZ3z5vq+hTe8zauKyPknc9846Nu4/QpdWzbwcoIiIiAs2vAM5G+DyFyCg8VfFK6/08P6avTyzYBsb9xeQFBPOA5d2Y2JGW1UFFJE6aYIJ1nDAQObcM06wxvRszQPvrWfmyr3cO1oJloiINHAeDyx4CBI6QbfL3I7Gpw4WlvHaF7t4ZelOcgrK6NAiin9e0YuxvdsQHNi0hkWKiG80vQQrMgGS+sHmD2Hor8+oiYSoUIZ0TGDWqj38emRnLSwoIiIN28Z3IXs9jH+u0fZerd97hBc/386sr/dSXulhaKdEfnRFOud1SND7uIh4VdNLsAA6jYR5f4Oig07CdQbG9Uni9jdW8cX2XM5pH+/lAEVEROrJ0blX8R2g++VuR+NVVR7LxxsOMGXRdr7Ynkt4cCBXZrTl+nPT6NAiyu3wRKSRapoJVscRMO+vkPkR9L7qjJoYcVYrIkMCmbVqjxIsERFpuDa9DwfWwmXPNpreqyOlFby1bDcvL9nB7twSkmLC+e1FXbgyI0WLA4uIzzXNBKt1L4hqBZkfnnGCFR4SyMjurXh/zT7uH9tNE2JFRKThsdaZexXXvlH0XlV5LM8u3Mbjn2ZSVF5F/7Q4fju6K8PPakmQ5leJSD1pmgmWMU6xi/WzoKoCAs/s26xxvZOY/tUe5m3MZnSP1l4OUkRExMc2zYb9a2Dc0xDYsD8S7Msv4c43V7F0Wy4jzmrJbRd0pHtSc7fDEpEmqOl+ndNpJJQdgV1Lz7iJc9vHkxgdyoyVe7wYmIiISD2wFuY/CHHtoMcVbkdTJx+s2ceoRz9jdVY+D0/oyTPX9VNyJSKuaboJVrthEBDsDBM8Q0GBAYzt1YZ5m7I5XFzutdBERER8bvMc2L8azru7wfZeFZdXcu+01dzy6lekxkfw/m3nMTGjLcaoKqCIuKfpJlih0ZA2CDbPrVMzl/VJoqLKMnvNfi8FJiIi4mNH517FpkHPK92O5oysycrn4scW8eby3fxsWHum3XIu6QmRboclItKEEyyAjiPh4CbI23HGTXRr04wOLaKYqWGCIiINgjFmlDFmkzFmizHm3uMcM9EYs94Ys84Y81qN/dcbYzKrt+vrL2ov2/sV7F0Jg25vcL1XHo/lmQVbGf/U5xSXV/HaTQP59aguWiRYRPxG0/7XqNNI57EOvVjGGC7rk8SXO3LZnVvspcBERMQXjDGBwBPAaOAs4CpjzFnHHNMR+A0wyFrbDbijen8ccB8wAOgP3GeMia3H8L1n7XRnmHy3y9yO5LTszy/luilf8H8fbOTCri2Zc8d5WipFRPxO006w4ts7pWnrMA8LYGyvNgDMWqVeLBERP9cf2GKt3WatLQfeAC495pifAE9Ya/MArLXZ1ftHAh9Za3OrX/sIGFVPcXuPtbBuJnS4AMIbTn44d91+Rv97IV/tPMxDl/fgyWv6EhMR4nZYIiLf07QTLHB6sbZ/BuVFZ9xE27gIzmkXz3+X7KSkvMqLwYmIiJclAbtrPM+q3ldTJ6CTMeZzY8xSY8yo0zgXAGPMZGPMcmPM8pycHC+F7iVZy+BIFnQb73YkpySvqJy73vqaya+sICk2nPduG8yVZ6eokIWI+C0lWB1HQFUZbF9Yp2buGtGJ7IIypny+3UuBiYiID9T2qdwe8zwI6AgMA64CnjfGxJziuc5Oa5+11mZYazMSExPrEK4PrJ0OgaHQebTbkZyQtZZZq/Zw4SMLmLVqD7ee357ptwyifWKU26GJiJyQEqzUQRASBZvrNkwwIy2OC7u24OkFW1WyXUTEf2UBbWs8Twb21nLMLGtthbV2O7AJJ+E6lXP9m8cD62dCx+EQ1sztaI5rd24xN7y4jNvfWEVyXATv/mIwvxrZhZAgfWwREf+nf6mCQpw1sTLnOuPS6+BXI7tQWFbJk/O3eiU0ERHxumVAR2NMujEmBJgEvHPMMTOB8wGMMQk4Qwa3AR8CI4wxsdXFLUZU72s4di+Fgn1+W9yiymN5/rNtjPjXQpbtyOW+S85i+i3n0rW1/yaDIiLHUoIFzjysI3vgwLo6NdO5VTTj+yTz0uId7D1c4qXgRETEW6y1lcDPcRKjDcBb1tp1xpgHjDFjqw/7EDhkjFkPzAN+Za09ZK3NBf6Mk6QtAx6o3tdwrJ0OQeHQyf9qc6zfe4TLnvycv7y/gXPax/PRL4dy46B0AgM010pEGpaGtfiFr3Qc4TxmfgitutepqTuHd+Tdr/fy6MebeXhCLy8EJyIi3mStnQ3MPmbfH2v8bIFfVm/HnjsFmOLrGH3CUwXrZ0GnERDqP/OYSiuqePTjTJ77bBuxEcH856o+XNyztYpYiEiDpR4sgOhW0LpXndbDOio5NoLrzkll6oosMg8UeCE4ERERL9j5ORRl+9XwwM+3HGTkowt5esFWLu+bxMe/HMolvdoouRKRBk0J1lEdR0LWl1Bc99Eet57fgciQIP7+4SYvBCYiIuIF62ZAcKTzfucyay2PzN3ENc9/gQFe+8kAHp7QS+taiUijoATrqE4jwXpgyyd1biouMoSfDm3H3PUHWLGzYQ3PFxFpKIwxP68uNiEnU1UJ69+BzqMgJMLVUCqqPPx66moe+3QLE/olM+eOIZzbPsHVmEREvEkJ1lFt+kJEgjMPywt+NDidxOhQHvpgE7aO1QlFRKRWrYBlxpi3jDGjjMaVHd+OhVB80PXhgUVlldz08nLeXpHFbRd05O8TehIWHOhqTCIi3qYE66iAAGddkC0fOxOB6ygiJIjbLujIlztymbcp2wsBiohITdba3+OsT/UCcAOQaYz5mzGmvauB+aN1M5w1HzsMdy2EnIIyJj27lM8yc/i/8T345fBOmmslIo2SEqyaOo6AkjzIWuaV5iad3Za0+AgenrOJKo96sUREvK264t/+6q0SiAWmGmMedjUwf1JVARvehc4XQXCYKyFsyylk/FOfsyW7kOd+mMFV/VNciUNEpD4owaqp/Q/ABMJm7wwTDA4M4K4Rndm4v4BZq/Z4pU0REXEYY24zxqwAHgY+B3pYa28B+gGXuxqcP9k23/nysPt4Vy7/1a48Ln9qMUVlVbw+eSAXdG3pShwiIvVFCVZN4TGQcg5k1r1c+1FjerSme1Iz/jl3M2WVdR96KCIi30gAxltrR1pr37bWVgBYaz3Axe6G5kfWzYDQ5s6XiPXso/UHuPq5pTQLD2b6LefSu21MvccgIlLflGAdq9MIOLAWDu/2SnMBAYZ7RnVhz+ESXl26yyttiogI4CwW/E2pVmNMtDFmAIC1doNrUfmTyjLY8B50GQNBofV66f8t3clPX1lO55bRTLvlXNISIuv1+iIiblGCdayj64N4sRfrvI6JDOoQz+PztlBQWuG1dkVEmringMIaz4uq98lRWz+Fsvx6HR5oreUfH27i9zPXMrRTIq9PHkhCVP0mdyIiblKCdazEzhCT4tUEC+CeUV3ILSrnuYXbvNquiEgTZmyNdTCqhwYGuRiP/1k3A8JioN2werlcZZWHu99ezePztjDp7LY898MMIkL0n0REmhYlWMcyxunF2rYAKkq81mzP5BjG9GjN84u2k1NQ5rV2RUSasG3VhS6Cq7fbAX2LdVRFKWycDV0vgcDgernk/32wkWlfZXHHhR35v/E9CArUxwwRaXr0L19tOo2EyhLYscirzd49sjNllR7+82mmV9sVEWmibgbOBfYAWcAAYLKrEfmTLR9DeUG9DQ+cuXIPLyzazg3npnHHhVrjSkSaLiVYtUkbDEHhXivXflR6QiSTzm7La1/sYuehIq+2LSLS1Fhrs621k6y1Lay1La21V1trtbL7UeumQ0Q8pA3x+aXW7snn3umr6Z8ex+/GdPX59URE/NkpJVjGmPbGmNDqn4dVD8lovLVWg8Oh3VDI/BCsdxcIvv2CjgQHBnD/O+uwXm5bRKQpMcaEGWNuNcY8aYyZcnRzOy6/UF4Mm+ZA17EQ6Ns5ULlF5fz0lRXERoTwxNV9CdawQBFp4k71X8FpQJUxpgPwApAOvOazqPxBxxFweBfkbPJqsy2ahXHv6C7M25TDC4u2e7VtEZEm5hWgFTASWAAkAwWuRuQvMudCRZHPhwdWVnn4xetfkVNYxtPX9iMxWtUCRURONcHyWGsrgcuAR621dwKtfReWH+g4wnnM9O4wQYAfnpPKyG4teWjORr7efdjr7YuINBEdrLV/AIqstS8DY4AeLsfkH9ZNh8gWkDrIp5f5+4eb+HzLIf4yrju9tIiwiAhw6glWhTHmKuB64L3qffVTksgtMW2hZXevz8MCMMbw8OW9aBEdxs9f/4ojWhtLRORMHP3H87AxpjvQHEhzLxw/UVYIm+fCWZdCQKDPLvPu13t5ZuE2rhuYysSMtj67johIQ3OqCdaNwDnAX621240x6cD/fBeWn+g0CnYtgeJcrzfdPCKYx67qzd7Dpfxm+hrNxxIROX3PGmNigd8D7wDrgYfcDckPbJ7jVMLtdpnPLrFh3xF+PXU1Gamx/OHis3x2HRGRhuiUEixr7Xpr7W3W2ter38yirbUP+jg293W5CKzHJ71YAP1S47hrRCfeX72P177c5ZNriIg0RsaYAOCItTbPWrvQWtuuuprgM27H5rp1MyC6NaSc45PmDxc7RS2ahQfx5LV9CQlSUQsRkZpOtYrgfGNMM2NMHPA18KIx5hHfhuYHWvdx3qQ2ve+zS9w8pD3ndUzggXfXs3H/EZ9dR0SkMbHWeoCfux2H3yk9ApkfwVnjIMD7iU+Vx3LbG6vYl1/CU9f2o0V0mNevISLS0J3qv77NrbVHgPHAi9bafsCFvgvLTwQEQOfRsOVTqCjx0SUMj0zsTbPwYG599SuKyyt9ch0RkUboI2PM3caYtsaYuKOb20G5atMHUFXms+GB/5y7iYWbc3jg0u70TYn1yTVERBq6U02wgowxrYGJfFvkomnoMsYpdbttgc8ukRgdyqNX9mbbwSLum7XOZ9cREWlkfgTcCiwEVlRvy12NyG3rpkOzZEg+2+tNz16zjyfnb+Wq/ilc1T/F6+2LiDQWp5pgPQB8CGy11i4zxrQDMk90QvWCj9nGmLXHef0aY8zq6m2xMabX6YVeT9LOg5Bonw4TBBjUIYGfn9+Bt1dkMXPlHp9eS0SkMbDWpteytXM7LteUFcKWT6Cb94cHbj5QwN1vf02flBjuH6uiFiIiJ3JKy7tba98G3q7xfBtw+UlOewl4HPjvcV7fDgy11uYZY0YDzwIDTiWeehUUCh0vhE1z/p+9+46Pqkr/OP45mfSQTgghhd57kaICKhaKggVWsVfUtW1xd3Wrv93VdV1Xd117XayIBUVFka5IV5o0DS0JHUJoCaSd3x93YCOGlszkzky+79drXnPnzi3PdXBunjnnPAcqK/3Sp/2wewa1Zv66Qn43YTldshJpkdbAb+cSEQl2xphrq1tvrT3WfSe0bfgSKsv+N4+jjxw4VM6YVxcRFxXOs1f3JCrcf6XfRURCwckWucgyxkzwtkhtM8a8Z4zJOt4+1tovgGPWN7fWzrHW7va+nAcc93iuajsMDmyHTf7teRLuCePfo7sRER7GnW8u5mBZhV/PJyIS5E6r8ugPPAAMdzMgV+VOg4hYyOnr08O+Nm8jG3YV88QV3UlPUFELEZETOdnmmFdw5hhpAmQCH3nX+cpNwKfHetMYM8YYs8gYs2jHjh0+PO1Jan0ehIXDav92EwTISIzh0ZFdWbllL7WsSOoAACAASURBVH+btMrv5xMRCVbW2ruqPG4BugORbsflmtypTrf28CifHbK4tJwXvljHgDZp9GuZ6rPjioiEspNNsNKsta9Ya8u9j/8Cab4IwBhzNk6C9ZtjbWOtfd5a28ta2ystzSenPTUxSdD0DFgzqU5Od26HdG48ozlj527ks2+31sk5RURCQDHQ2u0gXFG4Dnavh1aDfHrYN+fnsetAKfcMauXT44qIhLKTTbB2GmOuNsZ4vI+rgV21PbkxpgvwIjDCWlvr4/lVuwth53ewM7dOTvebIW3pnJnIr99dSn5hcZ2cU0QkmBhjPjLGTPQ+PgbWAB+exH6DjTFrjDG5xpj7qnn/emPMDmPMEu/j5irvVVRZP9G3V1QLudOc55a+S7AOllXw7Kx1nNEqlZ5N63f1exGRU3GyCdaNOCXatwJbgJHADbU5sTEmB3gfuMZa+11tjlUn2g5xnv1cTfCwqHAPT17ZHWvh6pfmU7BbSZaIyFEeBf7pffwNGGCt/VHCVJUxxgM8BQwBOgCjjTHVlcV721rbzft4scr6kirrA2e819rpkNQUUlv67JBvLchj5/5D3HVO/WwUFBGpqZNKsKy1edba4dbaNGttI2vtxTiTDh+TMeYtYC7Q1hhTYIy5yRhzmzHmNu8mfwRSgae9vwQG9twlSdnQuEudjMM6rGlqHGNv6s3uA6WMenYu63bsr7Nzi4gEgTxgvrV2lrX2K2CXMabZCfbpDeRaa9dZa0uBccAI/4bpZ+WlsP4Lp3ugMT45pNN6tZbezVPo20Jjr0RETkVtao7/4nhvWmtHW2szrLUR1tosa+1L1tpnrbXPet+/2VqbXOWXwF61iKVutBsG+Qtg//Y6O2WPnGTGjelHaXklP3luLqu27K2zc4uIBLh3gMoqryuoMqXIMWQC+VVeF3jXHe0y7zyN7xpjsqusj/YWXZpnjLn4WCep0+JMBQugdL9Puwe+syifbXsPcc8gtV6JiJyq2iRYvvmZLJi0HQpY+O6zOj1thyYJvH1rP8LDwrji+XksyS+q0/OLiASocG8rFADe5RNVEazu3mWPev0R0Mxa2wWYCoyt8l6O9wfBK4F/GWOq7ZNXp8WZcqc5lW6bD/DJ4Q6VV/D0zLX0bJrM6aocKCJyymqTYB19Qwp9jTtDYg6srptqglW1atSAd27rR2JMBFe9MI956wK7JoiISB3YYYw5Mg7KGDMC2HmCfQqAqi1SWcDmqhtYa3dZaw95X74A9Kzy3mbv8zpgJk5peHetnQZZvSE6wSeHe+/rTWzZc5C7B7XG+KjLoYhIfXLcBMsYs88Ys7eaxz6cObHqF2Og3VBYNwNKD9T56bNTYhl/az8aJ0Zz3csLmLmm7roqiogEoNuA3xpj8owxeTjTfdx6gn0WAq2NMc2NMZHAFTjzPB5hjMmo8nI4sMq7PtkYE+VdbgicAaz0yZXU1P4dsGUptDrHJ4crq6jk6Zm5dM1OYkDrhj45pohIfXPcBMtaG2+tTajmEW+tDa+rIANK26FQfhDWznDl9I0Toxl/az9apjXgllcX8enyLa7EISLiNmvtWmttX5xqgB2ttadba487l4a1thy4E5iMkziNt9auMMb8uUpr2N3GmBXGmKXA3cD13vXtgUXe9TOAh6217iZYa6c7zz4afzVh8SYKdpdwz6BWar0SEamh2nQRrJ+ang7RiXU26XB1UhtE8daYvnTOTOSON7/h/W8KXItFRMQtxpiHjDFJ1tr91tp93hamv55oP2vtJGttG2ttS2vtg951f7TWTvQu32+t7Wit7WqtPdtau9q7fo61trN3fWdr7Uv+vcKTsHYaxKZCRrdaH6q8opKnZuTSKTOBs9s28kFwIiL1kxKsU+WJgNYXwJpPoaLctTASYyJ47aY+9G2Ryi/GL+W1eRtdi0VExCVDrLVHqv5Ya3cDQ12Mp25VVjotWC3OhrDa384nLt3Mxl3F3HWOxl6JiNSGEqyaaDcUSgohf76rYcRFhfPy9acxqF0j/vDBtzw3a62r8YiI1DHP4TFRAMaYGCDqONuHlm3L4cAOaHVurQ9VUWl5cnou7RrHc177dB8EJyJSfynBqolW54In0tVugodFR3h49pqeDOuSwd8+Xc197y1j/yH3WtZEROrQ68A070T2NwFT+GFJ9dCWO815bln7AhcfL9vMup0HuHtQa8LC1HolIlIbSrBqIioemg+E1Z+Adb9afYQnjCeu6M5tA1syflE+Fzz+BXPWnqhSsYhIcLPWPgL8Faf4RAfgM6Cpq0HVpbXTIb0zxNeuxamy0vKf6bm0btSAwR0b+yg4EZH6SwlWTbUbCrvXw47VbkcCgCfMcN+Qdrxz2+lEhodx5Qvz+dOH31JcqtYsEQlpW4FK4DJgEN6S6iHv0H7Im+eT8uyffruV3O37uUutVyIiPqEEq6baDHGeV3/ibhxH6dk0mUl39+f605sxdu5Ghv77SxZtKHQ7LBERnzHGtDHG/NEYswp4EsgHjLfi35Muh1c3NnwJlWW1Ls/utF59T4u0OIZ1zjjxDiIickJKsGoqIQMyewbEOKyjxUR6eGB4R8aN6UuFtYx6bi4PTVrFwbIKt0MTEfGF1TitVRdZa8+01v4HqF9fcLnTICIWcvrW6jBTVm1j9dZ93HVOKzxqvRIR8QklWLXRdihs+hr2BuZkv31bpPLpPQMY3TuH579Yx7AnvmRJftGJdxQRCWyX4XQNnGGMecEYMwioX9lB7lRo1h/Ca1400VrLE9O+p2lqLBd1aeLD4ERE6jclWLXRbpjzHICtWIc1iArnoUs68+qNvSkureCyZ+bwj8mrOVRev37sFZHQYa2dYK29HGgHzAR+DqQbY54xxpzvanB1oXCdMwa4Ve26B05fvZ0Vm/dyx9mtCPfozwEREV/RN2ptpLWDlBYBnWAdNqBNGp/9bACXdM/kqRlrGfHkV3y7aY/bYYmI1Ji19oC19g1r7YVAFrAEuM/lsPzvSHn2midYh1uvspJjuKR7po8CExERUIJVO8Y43QTXfwGH9rkdzQklxkTw6KiuvHRdL3YdKOXip77isSnfUVpe6XZoIiK1Yq0ttNY+Z62tfVm9QLd2OiQ1hdSWNT7Ekvwilhbs4daBLYlQ65WIiE/pW7W22g2DilKnP3yQGNQ+nSk/H8Dwrk14Ytr3DH9ytlqzRESCQXmp86Neq0HOj3w1NGHxJqLCwxjRTWOvRER8TQlWbWX3gdhUWB343QSrSoqN5LHLu/Hitb0oPFDKiKe+4rHP16g1S0QkkBUsgNL9teoeWFpeyUdLN3Nuh3QSoiN8GJyIiIASrNoL80CbwfD9ZKgoczuaU3Zuh3Sm/HwgI7o14YnpuQx/cjbLC9SaJSISkHKnQVg4NB9Q40PM+m4Hu4vLuFRjr0RE/EIJli+0HQoH98DGOW5HUiOJsRE89pNuvHRdL3YXl3Lx01/x6OQ1qjQoIhJo1k6DrN4QnVDjQ0xYXEBqXCQD2qT5MDARETlMCZYvtDwHwmNg5YduR1Irg9qn8/nPBnJJ90yenJHL8P98xbICzZslIhIQ9u+ALUuhVc3reOwpLmPqyu1c1LWJiluIiPiJvl19ITIWOl8G34yFrd+6HU2tJMY6lQZfuf40ikpKueTpOTzy2WpKStWaJSLiqrXTnedajL/6ZPkWSisqubSHugeKiPiLEixfOe8vEJMCH/4UKsrdjqbWzm7XiM9/PpBLu2fy9My1DPzHDF6bt5GyChXBEBFxxdppTlGljG41PsSExQW0TIujc2aiDwMTEZGqlGD5SmwKDPun031jzr/djsYnEmMi+Meorrx7Wz+apsbyhw++ZdA/Z/HB4k1UVlq3wxMRqT8qK50WrBZnQ1jNbt35hcUs3LCbS3tkYWpR4l1ERI5PCZYvdRgOHS6GmQ/DjjVuR+MzvZqlMP7Wfrxy/WnERYXzs7eXMPSJL5m2ahvWKtESEfG7bcvhwA5n/qsamrB4E4DmvhIR8TMlWL429B8Q2QA+vAMqQ2fckjGGs9s14pO7zuSJ0d0pKavgprGLGPXsXOav2+V2eCIioS13mvPcsmYFLqy1TFi8iT7NU8hKjvVhYCIicjQlWL7WoBEMeQQKFsL8Z92OxufCwgzDuzZh6i8G8uAlncjfXczlz8/jupcX8O0mzZ8lIuIXa6dDemeIb1yj3ZfkF7F+5wEVtxARqQNKsPyh80hnbqxpf4Fda92Oxi8iPGFc1acps351NvcPaceS/CIu/M9s7njzG1ZsVqIlIuIzh/ZD3rxalWefsHgTUeFhDOmc4cPARESkOkqw/MEYGPYYeCJh4l3O4OQQFR3h4daBLfni12dz59mtmLl6O8OemM3VL85n1nc7NEZLRKS2NnwJlWU1Ls9eWl7JR0s3c16HdBKiI3wcnIiIHE0Jlr8kZMDgh2DjV7DoJbej8bvEmAjuvaAtc+4bxG8Gt+O7bfu47uUFDP7Xl7yzKJ9D5aEzHk1EpE7lToOIWMjpW6PdZ323g93FZeoeKCJSR5Rg+VO3q5wByVP+BLs3uh1NnUiMjeD2s1oy+zfn8OiorhgDv3p3Gf3/PoOnZ+ayp7jM7RBFRIJL7lRo1h/Co2q0+4TFBaTGRdK/dZqPAxMRkeoowfInY+CiJ5znj+6BetRdLjI8jJE9s/j0nv68emNv2jaO55HP1tDv4Wk8MHEF+YXFbocoIhL4Ksqh+1XQ45oa7b6nuIypK7dzUdcmRHh0yxcRqQv6tvW3pGw478+wbgYsfs3taOqcMYYBbdJ47aY+TLq7P4M7Nub1eRsZ+I8Z/PSNr5mTu1OTFotInTLGDDbGrDHG5Bpj7qvm/euNMTuMMUu8j5urvHedMeZ77+M6vwfrCYcBv4L2F9Vo90+Wb6G0olLdA0VE6lC42wHUCz1vgBUTYPLvoNW5kFA/J3ns0CSBxy7vxq8Gt+W/czbw1vw8Ji3fSrPUWK7oncPInlk0bFCzLjAiIifDGOMBngLOAwqAhcaYidbalUdt+ra19s6j9k0B/gT0AizwtXff3XUQeo1MWFxAy7Q4Omcmuh2KiEi9oRasuhAWBsOfgIoy+Ohn9aqrYHUyEmO4f0h7FvzuXB6/vCuN4qN5+NPV9H1oGj9942u+/H6HWrVExF96A7nW2nXW2lJgHDDiJPe9AJhirS30JlVTgMF+irPW8guLWbhhN5f2yMIY43Y4IiL1hlqw6kpKCxj0R5h8PywbD10vdzsi10VHeLikexaXdM8id/s+xi3I571vCpi0fCvZKTFccVoOo3pm0Sgh2u1QRSR0ZAL5VV4XAH2q2e4yY8wA4Dvg59ba/GPsG7B97yYs3gTAiG71s9eEiIhb1IJVl/rcClm94dNfw75tbkcTUFo1iuf3F3Zg7v2D+PcV3chKiuUfk9fQ7+HpjHl1EZNXbGX73oNuhykiwa+6ppyjm8w/AppZa7sAU4Gxp7Cvs6ExY4wxi4wxi3bs2FHjYGvKWsuExZvo2yKFrOTYOj+/iEh9phasuhTmgRFPwbNnwuTfwsjQnx/rVEVHeBjRLZMR3TJZv/MA4xbm8e6iAj5f6SSkafFRdGqSQMcmiXTKdJ6zkmPU/UVETlYBkF3ldRawueoG1tpdVV6+APy9yr5nHbXvzOpOYq19HngeoFevXnXe53lJfhHrdx7g9oEt6/rUIiL1nhKsupbWBs64G774B/S5DbJPczuigNW8YRz3D2nPL89ry5L8IlZs3sO3m/ayYvMevvh+JxXecVqJMRF0bJJAxyYJdMpMpFezFDKTYlyOXkQC1EKgtTGmObAJuAK4suoGxpgMa+0W78vhwCrv8mTgIWNMsvf1+cD9/g/51E1YvImo8DCGdG7sdigiIvWOEiw3nPEz+OZVpxXrps+debLkmCLDw+jdPIXezVOOrDtYVsGarfv4dvMeVmzey4pNexg7dyOl5ZV4wgxXnJbNPYNaa/yWiPyAtbbcGHMnTrLkAV621q4wxvwZWGStnQjcbYwZDpQDhcD13n0LjTF/wUnSAP5srS2s84s4gdLySj5aupnzOqQTHx3hdjgiIvWOEiw3RDWAc34PE+9yyrd3utTtiIJOdISHrtlJdM1OOrKurKKS3O37Gbcgjzfm5/H+N5u4pX9zbhnQQn9kiMgR1tpJwKSj1v2xyvL9HKNlylr7MvCyXwOspVnf7WB3cZnmvhIRcYmKXLil21WQ3hmm/gnKVLzBFyI8YbTPSOD/RnRi6i8GMqh9I56YnstZ/5jJf79aT2l5pdshioj43fvfFJAaF0n/1mluhyIiUi8pwXJLmAcu+CsU5cH8Z92OJuQ0axjHk1f24MM7zqBNejwPfLSS8x6fxUdLN2uOLREJWXuKy5i2ajsXdW1ChEe3eBERN+jb100tzoI2Q+DLf8L+ui/jWx90zU7izVv68N8bTiMmwsNdby1mxFNfMSd3p9uhiYj43CfLt1BaUanugSIiLlKC5bbz/wJlxTDzIbcjCVnGGM5q24hJd/fnsZ90pfBAKVe+OJ9rX17A0vwirFWLloiEhgmLC2jVqAGdMxPdDkVEpN5SguW2hq2h103w9X9h+6oTbi41FxZmuLRHFtN+OZDfDW3P0vwiRjz1Fec//gVPz8xlc1GJ2yGKiNRYSWkFCzfsZminxpobUETERUqwAsFZ90FUPHz+e7cjqReiIzzcMqAFX/7mbB68pBOJMRE88tkazvj7dEY/P4/xi/LZd7DM7TBFRE5Jwe5iAFo2auByJCIi9ZvfEixjzMvGmO3GmG+P8b4xxjxhjMk1xiwzxvTwVywBLzYFBvwacqfC91PdjqbeSIiO4Ko+TXn39tP54ldn87NBbdiyp4Rfv7uMXn+dyl1vLWbG6u2UV6j6oIgEvrxCJ8HKTol1ORIRkfrNn/Ng/Rd4Enj1GO8PAVp7H32AZ7zP9VPvW2Dhi04rVouzwKMpyupSTmos95zbmrsHtWJxfhETvtnER8s289HSzTRsEMlFXZswrHMGXbOTVJlLRALS4QQrRwmWiIir/PZXvLX2C2NMs+NsMgJ41ToVBuYZY5KMMRnW2i3+iimghUfBeX+G8dfA4leh141uR1QvGWPokZNMj5xk/nBhB2au2c6ExZt4Y14er3y1gbhID31apHJ6y1TOaNWQtunxhIVprIOIuC+/sITYSA+pcZFuhyIiUq+52UySCeRXeV3gXfejBMsYMwYYA5CTk1Mnwbmi/UXQ9AyY/iB0GgnRCW5HVK9FhodxfsfGnN+xMXtKypiTu5Ov1u5kTu4upq/eDkBqXCT9vMnWGS0bkpOqX45FxB15hcXkpMSqwIWIiMvcTLCquwNUWy/bWvs88DxAr169QremtjFwwYPw/Fkw+zE49wGXA5LDEmMiGNI5gyGdMwDYXFTCnLW7jiRdHy9zfhfISo7hjJYNOa15Cl2zEmmR1gCPWrhEpA7kFxZr/JWISABwM8EqALKrvM4CNrsUS+Bo0h26joa5T0PPGyC5qdsRSTWaJMUwsmcWI3tmYa1l7Y79fJW7i69yd/Lpt1t4e5HTOBsX6aFjZiJdsxLpkpVEl6xE/cIsIj5nrSWvsJgzWjV0OxQRkXrPzQRrInCnMWYcTnGLPfV2/NXRzvkDrPgApj4Ao15xOxo5AWMMrRrF06pRPNed3oyKSsu6HftZVrCHZQVFLNu0h7FzN1Javh6ApNgIOmcm0sWbdPVtnkpibITLVyEiwWzn/lJKyirISYlxOxQRkXrPbwmWMeYt4CygoTGmAPgTEAFgrX0WmAQMBXKBYuAGf8USdBIz4Yy7Ydbfoe/tkN3b7YjkFHjCDK3T42mdHs9lPbMAKKuoZM3WfSzf5CRdS/P38OysdVRUWiI8hjNbNWRYlyac1yGdxBglWyJyavK9c2BpHKiIiPv8WUVw9Anet8Ad/jp/0Dv9bvh6LHx2P9w81RmfJUErwhNGp8xEOmUmMrq3U6jlYFkFKzbv4fMV2/h42RZmvLOUSE8YA9o0ZFiXDM5tn058tJItETmx/MNzYCUrwRIRcZsmWwpUUQ1g0B/gwztgyRvQ/Wr3YilcB8nNleT5WHSEh55NU+jZNIX7hrRjacEePlm2mU+WbWHqqu1EhodxVps0hnXJYFD7dBpE6X9XEale3i4nwcpSgiUi4jr9xRbIuo6GxW/Axz+HxGxoMbDuY1jyFnxwG2SdBoP/Dlk96z6GesAYQ7fsJLplJ3H/kPYszi/ik2VbmLR8C5+v3EZUeBhnt21Ej6ZJNE2No1lqHDkpscREetwOXUQCQF5hMY3io/SdICISAJRgBbIwD4x+E14eAuOughs+gYyudXf+fVvhs99AWnvYvRFePAe6Xgnn/gniG9ddHPVMWJihZ9NkejZN5vfD2vN13m4+WbaFz77dymcrtv5g2/SEKG/CFUvT1DiapsbSzPus7oUi9Uf+bmcOLBERcZ8SrEAXkwzXvA8vnQ+vXwY3TobUlv4/r7Xw8S+g/BBc/jo0aARf/hPmPQ2rJkL/X0DfOyAi2v+x1GNhYYbTmqVwWrMUHhjekT0lZeTtKmbDrgNs3HWADbuK2bjrADPW7GDHvoIf7NuqUQNOb5lKvxap9G2RSnJcpEtXISL+ll9YQp/mKW6HISIiKMEKDglN4Or34eUL4PVL4cbPIT7dv+f89j1Y8wmc9xdo2MpZd97/QY9r4fM/wLQ/O0U4LngQ2l2o8Vl1JDEmgs5ZiXTOSvzRewcOlZNX6CRca3ccYMH6Qt79uoBX524EoH1GAv1apHJ6y1R6t0ghQS1cIiGhtLySzXtKyFILlohIQFCCFSzS2sBV78LYC52WrBs+gegf/5HtEwd2wqe/hsye0O+oQo+pLZ1ui2tnOBUO374amvWHwQ9D407+iUdOSlxUOO0zEmifkQDAHWc75eGXFRQxJ3cXc9ft4vX5G3n5q/WEGeiUmUg/bwtX95xklYcXCVKbikqwFnURFBEJEEqwgklWT7j8NXjzcmdM1lXv+qeL3qR74dA+GPGUMw6sOi3Phttmw9evwIwH4bn+0PMGOPt3EJfq+5ikRiI8YUcqFd41qDUHyypYnFfE3HW7mLt2Jy/PXs9zs9YBTpfCbtlJdM9xim20TY8n3BPm8hWIyInkeUu0K8ESEQkMSrCCTatz4eJn4P1b4P2bYdTYYydBNbFyIqyYAOf8Hhq1P/62nnDofQt0ugxmPgwLX4Rl451qh80HOs8N26j7YACJjvA4rVYtU+G8NhSXlvPNxiIW5+1mSX4R01dv592vnbFcMREeOmcl0j0nie7ZSXTPSSY9QWPuRAJNvhIsEZGAogQrGHX5idONb/L98Mkv4cLHfZPEFBc6x2vcBc742cnvF5sCQx+BXjfA3Kdg3SxY/bHzXoPG0HyAN+kaAEk5tY9TfCY2MpwzWzfkzNYNAbDWkl9YwuL83SzOK2JxfhEvz15PWYUFIDMphnPbN+LCrk3omZNMWJiSZxG35RcWExkeRqP4KLdDERERlGAFr34/hQPbYfbj0CAdzr6/9sf87H4oKXSqFnpqMB6nUXsY8aRTgXD3Blg/C9Z/AetmwPLxzjbJzf+XbDU/S90JA4wxhpzUWHJSYxnRLROAg2UVrNyylyV5Rcxbt4txC/MZO3cj6QlRDO2cwYVdMuierWRLxC15hcVkJcfo/0ERkQChBCuYDfoT7N8Bsx6GBmlw2s01P9Z3k2HZOBj4G2jcuXZxGQMpzZ1Hz+udhGv7qv8lXN++D1//FyJi4bqPNXlxgIuO8NAjJ5keOcnceGZz9h8qZ9qqbXy8bAtvzMvjla820CQxmqGdMxjWJYNu2UkYdQsVqTN5hZoDS0QkkCjBCmbGwEX/huJd8Mm9EJsKHS859eOUFMFH90CjDtD/Xv/Emd7BefS9HSrKYfNieO9GGH8NjJnlJIgSFBpEhTOiWyYjumWy92CZk2wt3cLYuRt4cfZ6MpNiuLBLBhd0akyHjASiI3w4RlBEfiS/sJieTZPdDkNERLyUYAU7TziMfBleuwTeHwN7t0DP6yAy7uSPMeUPsH8bXPEmhNfBZLSecMg+zZnA+KXz4d0b4JoPnPUSVBKiI7ikexaXdM9iT0kZU1Zu4+Nlm3lp9nqe+2IdYQaapsbRulED2qTH0zrdeW6RFkdUuBIvkdraU1zG3oPlasESEQkg+os2FETGwpXjYPx1TuGLLx6B3rdC7zEnHuO0djp886pT1CKzR93Ee1hGV6cFbsKtMOWPMPihuj2/+FRiTAQje2YxsmcWRcWlfJW7izXb9vH9tn18t20f01Zvp6LSKZbhCTM0TY2lTaN42qQ3oEVaA9Lio0iJiyS1QSQpsZEqES9+Y4wZDPwb8AAvWmsfPsZ2I4F3gNOstYuMMc2AVcAa7ybzrLW3+T/iYztcoj0rWQmWiEigUIIVKmKS4bqJkL8AZv/LGZc15wnoca0zWXB11fsO7YOJ90BqazjLB0UyaqLrFbDpG5j3FDTpDl1GuROH+FRSbCTDumQwjIwj6w6VV7B+5wG+27af77ftY83WfazZto/PV27Fm3cddYwIJ+GKiyQ1LoqUBpE0jIskLT6KrORYspJjyEqOJSZSLWFy8owxHuAp4DygAFhojJlorV151HbxwN3A/KMOsdZa261Ogj0JmgNLRCTwKMEKNdm9YfSbsH21k2AtfBEWvACdR8EZ9zjjoA6b+gDsyYcbJ/tnwuKTdcGDsHU5TLzLqUTYuJN7sYjfRIV7aNc4gXaNE36w/mBZBfmFxew6UMqu/aUUHjhUZbmUnfsPsXbHfhZuKKWwuBR7VDLWsEHkDxKu7BTvc3IMmckx6oooR+sN5Fpr1wEYY8YBI4CVR233F+ARwA8DU33ncIKVnRLjciQiInKYEqxQ1agdXPw0nP1bmPu0U7Vv2ThoM9jpDmgrnOSr708hp4+7sXoiYNR/4fmB8PZVcMsMZ24tqReiIzy0To+n9UlsW1Fpe5/ZPgAAHl9JREFU2bn/EAW7SyjYXUzB7hLyC53n5Zv2MHnF1iNzdoHTFbF5wzjaNo6nbXo8bRvH065xPNnJsSppXX9lAvlVXhcAP/gSNMZ0B7KttR8bY45OsJobYxYDe4HfW2u/rO4kxpgxwBiAnBz/zf+Xv7uYlLhI4qNrMLWGiIj4hRKsUJeY5YxtGnCvk1DNfxZeGQyeKGdOqnP+4HaEjvh0+Mlr8MoQeP8WuHI8hKnlQX7IE2ZIT4gmPSG62qppFZWWbXsPHknA1u04wOqt+1hWUMQny7Yc2S420knq2qXH08abdOWkxJIcF0lcpEdl5kNbdR/ukazcGBMGPA5cX812W4Aca+0uY0xP4ANjTEdr7d4fHdDa54HnAXr16lVNJ1jfyC8sJlvdA0VEAooSrPoiNgUG/hr63QmLX3cm/j3/r06BjECRfRoMfQQ+/jnMeAgGBUjyJ0HDE2ZokhRDk6QYejf/YSvogUPlfOcd+7V6q1N4Y+qqbby9KP8H20V4DIkxkSTFRpAUE0FSbNVl53XzhnF0z0kiNlJfoUGoAMiu8joL2FzldTzQCZjpTbQbAxONMcOttYuAQwDW2q+NMWuBNsCiugi8OnmFxXTOTHTr9CIiUg39dVDfRMZCnzHOIxD1vMEpevHlo07Ri/YXuh2RhIi4qHC65yTTPeeHLV879h1izdZ9bC4qoaiklKLiMnYXl7HHu7ypqISVm/dQVFJGcWnFkf3CwwxdshLp0yKVPs1T6Nk0Wd20gsNCoLUxpjmwCbgCuPLwm9baPUDDw6+NMTOBe71VBNOAQmtthTGmBdAaWFeXwVdVXlHJpt0lDOucceKNRUSkzijBksBiDAx9FLatgAm3QcPpkNbG7agkhKXFR5EWH3VS2x4sq2BPSRmrtuxl/vpC5q/bxQtfrOOZmWsJM9ApM5E+zVPo0zyV05qnkBijhCvQWGvLjTF3ApNxyrS/bK1dYYz5M7DIWjvxOLsPAP5sjCkHKoDbrLWF/o+6elv2HKS80qqCoIhIgFGCJYEnIhoufw2e8xa9uHkaRCeceD8RP4uO8BAd4SE9IZqz2jYCoLi0nG82FjF//S7mrytk7JyNvPDleoyB9o0TyEqOwRNmCAszeIxxlo3BE0aVZee50lrKKiopLXeeneVKSr3LZRXO+vIKS8cmCZzXIZ3+rdNUqv4UWWsnAZOOWvfHY2x7VpXl94D3/BrcKcjfrRLtIiKBSAmWBKbELPjJWBg7HD643SmAEaaJZyXwxEaGc2brhpzZ2ulVdrCsgiX5RcxfV8iCDbvIKyym0loqKi2V1inEceRhLZXe54pKS3iYIcITRoQnjMjwMCI8VV57woiJ8JAQHY4FPluxlXe+LiA6IowzW6Vxfod0zmnfiIYNTq41ToJf/pES7UqwREQCiRIsCVzNznQKcUy+H2Y/5lRCFAlw0REe+rZIpW+LVDip4vM1U1peyYL1hUxZuZUpK7cxddU2jIGeOcmc1yGd8zqk0yKtgd/OL+7LKyzGE2bISHRxHkMREfkRJVgS2PreDpu/gel/hcZdoM35bkckEhAiw8OOtJw9MLwjKzbvZcrKbUxZuY2/fbqav326mpZpcZzbIZ226fFOdcXEGBonRhMZrtbgUJBXWEJmUgzhHn2eIiKBRAmWBDZj4KInYOd38N5NzngsFb0Q+QFjDJ0yE+mUmcjPz2tDwe5ipq3azpSV23jpy/WUV9oq20LDBlHehCuaJkkxZCRGk5nkJF/RER5vt0RDuPc5IiyMiPCwI10YPZqkOSDkFxZr/JWISABSgiWBLzIWLn8DXjgbxo12kqyYJLejEglYWcmxXHd6M647vRkHyyrYXFTC5qKDbN5TwuaiErZ4l9ds28fMNTsoKas48UGrCDMQ7gnj3dv60SVL/y+6Jb+wmPM7NnY7DBEROYoSLAkOSdlOoYuxFzktWVeOhzBVThM5kegIDy3SGhxzPJa1lqLiMjbvKWHb3oMcKqukrNJSVl5JeWUlpRWWcm/lwlLvc1lFJWWVlSdd3l58b/+hcnYdKCU7JcbtUERE5ChKsCR4NO0Hwx6Fj+6BqQ/A+X9xOyKRoGeMITkukuS4SDo2SXQ7HDlJhysIqougiEjgUYIlwaXn9bD1W5jzBKR3gq6Xux2RiEidy1OCJSISsFR6SILP4L9Bs/4w8S7Y9I3b0YiI1Dm1YImIBC4lWBJ8PBEwaizEp8O4q2Df1lM/RlEefHCH093Q2hNvL6FP/w4kiOQXFhMfFU5iTITboYiIyFHURVCCU1wqXPEWvHQevH0NXP8xhJ/EgPviQvjyn7DgeagsB1sJGd2g1w3+j1kCU2UlvH0VbPzKaRltcRa0OBtSWzo1zUUCUF5hMdkpsRj9GxURCThqwZLg1bgTXPIsFCyAj39x/BaIshKY/Tj8uxvMfQo6j4J7ljp/TE/+HRSur6uoJdDMfgzWTILsPrBlKUy6F57sCY93clo5l70D+7e7HaXID+RpDiwRkYClFiwJbh1GwMDfwKy/Q+PO0Pe2H75fWQFL34IZD8HeTdD6fDj3AUjv6Lw/4il4uh988FOnFSxUSr8f3AtR8WqBOZH1X8KMB52E+9IXnHWF62DdTFg/C1Z/DEted9Y36ggtBjpJefOBEBHtUtBS31VWWgp2lzCofbrboYiISDWUYEnwG3gfbFsBk38Ljdo5fwBbC99/7pRz374SmvSAS56D5v1/uG9iFgz5O3xwO8x7Bk6/04UL8LHvp8C4K53ubiOehIQmbkcUmPZtc+ZUS20FF/7rf8loakvncdpNToK+dZmTcK2bCQtfgnlPO4n6Ve+4Gb3UYzv2H+JQeSXZasESEQlI6iIowS8szOkq2LANjL8OVn4I/70Q3vwJlB+EUf+FW6b/OLk6rOtoaDsMpv0Ztq+u09B9buNcZ0xaYjbkzYWn+zpd3FTA4YcqK5zk6uBep2BKVPWT8BLmgSbd4cyfw7Ufwn0b4Yx7nOR928q6jVnE63CJ9uxkTTIsIhKIlGBJaIiKh9FvOa0Q46+FnWtg6KNwxwLoeMnxu8oZAxf9y/kje8KtUFFWd3H70tbl8OblkJgJN06G22ZDw7bw/s3wznVwYJfbEQaOmQ/Dhi9h2D8hvcPJ7xcRA6ffA+HRsOA5/8Unchx5u1SiXUQkkCnBktCR0hyufg/OfxDuXgy9b3FKup+MBo3gwsdhyxL48jH/xukPu9bCa5c6SeI1H0CDNKeb242fwaA/wepJTmvWms/cjtR9uVPhi39At6uh+1Wnvn9cKnQeCcvGQ8lu38cncgJ5hcUYA5lqwRIRCUhKsCS0ZPZ0xlFFxZ/6vh1GQOefwBePwObFvo/NX/ZuhlcvBlvhJFdJ2f97L8wD/X8BY2Y4SeRbl8OHdzpd4+qjPZvg/THQqD0M/UfNj9P7VigrhsWv+y42kZOUv7uYjIRoosJDpCiPiEiIUYIlUtXQRyAuDSbcBmUH3Y7mxIoL4bVLnJaUq9+DtDbVb9e4szMO7cxfwJI34JkzYMPsuo3VbRVlzrir8kPwk1chshbdqzK6QM7p3vnUKnwXo8hJyC8sJkvdA0VEApYSLJGqYpJh+JOwY7VTvjuQHdoHb4x05vAa/ZZTjOF4wqPg3D8547M84U4hkM9+68wRVh9M/4tT+OOif0PD1rU/Xp8xUJQH302u/bFEToHmwBIRCWxKsESO1vpc6HkDzPmPU5UvEJUfgnFXweYlTpXEY1VIrE52b6cAxmk3w7yn4LmBsGON30INCGs+ha/+Db1ucsZP+UK7CyG+iYpdSJ06WFbBtr2HlGCJiAQwJVgi1Tn/L5CUAx/cBof2ux3ND1WUO13d1s+Ci5+GdkNP/RiRcTDsUbhmgtO98KXzIW++72MNBLs3Ol0+M7rCBQ/57rieCGeurHUzg7+8vwSNgt1Oi7MSLBGRwKUES6Q6UfHO3Fq7N8KUP57cPpWVztxIC19yKszt3+H7uKyFj+6BVR/B4Ieh6xW1O17Lc+CmzyE2FV4d4bT0hJLyUnjneue/26ixEBHt2+P3vB48Uc5YLJE6kH94DiwlWCIiASvc7QBEAlbT06HfHTD3SWg3DFoN+uH7FeWwdRlsnOM88ub8uGx3RldoOcjZN7vPyZeNr4618PnvYcnrMPA+6Ht7zY9VVUpzJ8l6Y5TT7fCif0GPa31zbLdN+QNs/gYuf925Tl+LawidLoOl45zxbdGJvj+HSBVHJhlOUYl2EZFApQRL5HjO+QN8P8UpbX7rLGe+qY1fOQlV/nwo9XYfTGnhJGFNz4CcvnBwjzPfUu50Z+zP7McgMh6aD4BW5zhJ14n+4C8/BAd2woEdULwT1s5wkr3et8JZ9/n2OuMawnUfOZM0T7wL9m+D/vcef4LmQPftezD/Wej7U2h/kf/O02cMLH0TFr8B/X7qv/OI4CRY0RFhpDWIcjsUERE5Br8mWMaYwcC/AQ/worX24aPezwHGAknebe6z1k7yZ0wipyQi2ukq+OK58GiVynONOjjd85qe7pTrTsj48b5NusOAXznJ1vovIHcarJ0Gaz5x3k9p4SRa0YlOAnU4mTqwAw7sgkN7fnzMrqOdroH+SHyiGsCVbzvJ5PS/wr5tMOTvzlxaNVF+yKlc6IbvPof3b4XsvnDu//n3XE26O62TC56HPrdBmHpei/8criBogvnHDxGREOe3BMsY4wGeAs4DCoCFxpiJ1tqVVTb7PTDeWvuMMaYDMAlo5q+YRGoks4dTTGLrcm9C1Q9iU05+/+hEpwWl/UVON79da51EK3eqMydV+UFnDFRcmtOS1KS7sxzb0Hkdl+Y8GqRBcnP/tip5IuDiZ5xJiec8AQe2wyXPn/zYpYN7nZajb151/ntd/Ax0GeW/eKuzdga8fTWkd3ASxvBI/5+z9xin8EjuFGhzgf/PJ/VWvkq0i4gEPH+2YPUGcq216wCMMeOAEUDVBMsCCd7lRGCzH+MRqbmuV9S+oAQ4yVHDVs6jz63OOC4TFlitHmFhThXF+MYw+bdOa9roN489vshap7vkN6/BivehrNhp4cvoAu/fDCWFzrXWhY1z4K3RkNoKrvkAYpLq5rwdRjjj4+Y/pwRL/MZaS35hMX1bpLodioiIHIc//6rLBPKrvC7wrqvqAeBqY0wBTuvVXdUdyBgzxhizyBizaMcOP1RmE3GLJzywkquq+t0Bl73kJE+vDIW9W374/oGdzlxhT/WGly+AlR9A51Fw83S4fQ5cP8mZK+rTX8OMh5xEzJ8KFjmFOpKy4doPTq2VsbY8EdDrRqdlcuf3dXdeqTFjzGBjzBpjTK4x5piDGo0xI40x1hjTq8q6+737rTHG1FlGXXiglAOlFWrBEhEJcP78y666fkxH/4U1GvivtTYLGAq8Zoz5UUzW2uettb2stb3S0tL8EKqIVKvzSLhqPOze4MyVtWON07Vx/LXwz3ZOq010Egx/En65BoY/AVk9nZa6iGinNHq3q2HW32HSvU4pe3/YshRev9TpUnnth04Xx7rW83rwRKpkexCo0oV9CNABGO3tpn70dvHA3cD8Kus6AFcAHYHBwNPe4/nd4QqCSrBERAKbP7sIFgDZVV5n8eMugDfh3KCw1s41xkQDDYHtfoxLRE5Fy3Pg+o+d1qGnejvrYlKccUc9roFG7Y+9ryccRjzptCbNecIpY3/xs74dF7VtJbx6MUQlOJUQE5r47tinokEj6HgpLHnTqT4ZnXDifcQtJ9OFHeAvwCPAvVXWjQDGWWsPAeuNMbne4831d9BHEqxUJVgiIoHMny1YC4HWxpjmxphInF/8Jh61TR4wCMAY0x6IBtQHUCTQNOnuzJXV41oY+Qr8cjUMfuj4ydVhxjhjus79P6cAxltXQOkB38S183t4dbhTrfC6iZCU45vj1lSfMU7p/iVvuhuHnMgJu7AbY7oD2dbaj091X+/+Pu/aXrC7BIDsZCVYIiKBzG8JlrW2HLgTmAyswqkWuMIY82djzHDvZr8EbjHGLAXeAq631t8DNUSkRlJawPD/QKdLa1Z+/cyfOfuvmwGvjoDiwtrFU7gexnq/Sq6d6MTntsyekNnL6Sbor+6Q4gvH7cLu7ar+OM496pT2PbLCD13b83YV07BBFDGRddIjUUREasiv82B557SadNS6P1ZZXgmc4c8YRCSA9LgWYpLh3RudwhnXvF+zLn1F+U5yVV4C138CaW18H2tN9bnNqZ64djq0PtftaKR6J+rCHg90AmZ655tqDEz0/jh4Mt3f/cKZAyumLk4lIiK1EKDly0QkZLW/CK5+D/YUwEsXOPOCnYq9W5xugQf3wDUTIL2jf+KsqQ4joEE6zH/W7Ujk2I7bhd1au8da29Ba28xa2wyYBwy31i7ybneFMSbKGNMcaA0sqIug8zQHlohIUPBrC5aISLWaD4DrP4LXRzrVCa982+niV1HqPMpLoeJQNcuHYOoDsH+7M89Vk+5uX8mPhUc6Jdtn/s1JHlNbuh2RHMVaW26MOdyF3QO8fLgLO7DIWnv0eOGq+64wxozHKYhRDtxhra3wd8xlFZVs2VNCTsqPhnuJiEiAUYIlIu5o0h1unAyvXQwvDjr5/cJjnBaw7NP8F1tt9bwBvngUFrwAQx52Oxqpxom6sB+1/qyjXj8IPOi34KqxuaiESgtZasESEQl4SrBExD0NW8HNU2HFBMA4E/aGR4EnqspyhPP68HJitjPfVSCLT4eOF8Pi1+Gc30FUvNsRSZDTHFgiIsFDCZaIuCu+MfS93e0ofK/PbbD8Hfj459BykNNVMKUFxKY6petFToESLBGR4KEES0TEH7J6QaeRztxfy9/53/qoREhp/r+EK8X7nNoyeJKvQ/thzn9gwL1Oq6L4XV5hMZGeMNITot0ORURETkAJloiIv4x8CS5+BoryoHAtFK5zCl8UroNNXztdI22V+bIiYiEhExIzITELErKc5YRMp2tkYiZExrl3PeAUGHljFGxdDs3OhOb93Y2nnigoLCErOQZPWBAk4CIi9ZwSLBERfwqPdMaaNWz14/fKS2FP/v+Srj353scmyJ0G+7byozlso5MgKRu6Xwun3QRhdTjp7M7v4fXL4MAOGP2Wkqs6lFdYrAIXIiJBQgmWiIhbwiOdroHHKuVeXgr7tsDeTc68YYcfW5fBp7+CZW/D8CfqZi6w/AXw5uVgwuD6jyGzp//PKUfkFRbTNTvR7TBEROQkKMESEQlU4ZGQ3NR5VGUtLH8XPrsPnhsAp98NA38NETH+iWPVx/DeTZDQxCmRn9LCP+eRau0pKWNPSZkKXIiIBIkwtwMQEZFTZAx0GQV3LoQul8Psx+DpfrBupu/PtfBFGH+N00p20xQlVy7IVwVBEZGgogRLRCRYxabAxU/DtROdpOvVETDhdigurP2xrYWpD8Anv4TW58N1HwX+/GMh6nCClZWsBEtEJBgowRIRCXYtBsLtc6D/L2H5eHiyFyx920mSaqK8FCbcCrMfh543wOVvuF+9sB47MgdWqhIsEZFgoARLRCQURMTAoD/CrV9AcnOYMAZevxQK15/acQ7uhTdHOQU0zvk9XPg4eDRc1015hcUkxUaQEK05x0REgoHumiIioSS9I9z0OSx6Gab+HzzdFxp1gPgMiG9c/XNsitPFcO8WeGMk7FjtzN/V7Uq3r0ZwEiyNvxIRCR5KsEREQk2YB3rfAm2Hwlf/cubZ2r0B8uZCSTXjszyR0KAxlO6HilK4cjy0GlTnYUv1CnaX0KFJgtthiIjISVKCJSISqhIzYeg/friu7CDs3+ZMYrxvyw+fyw5A/3uhSTd34pVqtWscT8+cZLfDEBGRk6QES0SkPomIrn5uLQlYz1ytSZ1FRIKJilyIiIiIiIj4iBIsERERERERH1GCJSIiIiIi4iNKsERERERERHxECZaIiIiIiIiPKMESERERERHxESVYIiIiIiIiPqIES0RERERExEeUYImIiIiIiPiIEiwREREREREfUYIlIiIiIiLiI0qwREREREREfMRYa92O4ZQYY3YAG2t5mIbATh+EE4hC9dpC9bogdK8tVK8LdG2+0tRam1ZH53KN7lvHFarXBbq2YBSq1wWhe211fV0ndd8KugTLF4wxi6y1vdyOwx9C9dpC9bogdK8tVK8LdG1S90L1cwnV6wJdWzAK1euC0L22QL0udREUERERERHxESVYIiIiIiIiPlJfE6zn3Q7Aj0L12kL1uiB0ry1Urwt0bVL3QvVzCdXrAl1bMArV64LQvbaAvK56OQZLRERERETEH+prC5aIiIiIiIjPKcESERERERHxkXqXYBljBhtj1hhjco0x97kdj68YYzYYY5YbY5YYYxa5HU9tGGNeNsZsN8Z8W2VdijFmijHme+9zspsx1tQxru0BY8wm72e3xBgz1M0Ya8IYk22MmWGMWWWMWWGMuce7Pqg/t+NcVyh8ZtHGmAXGmKXea/s/7/rmxpj53s/sbWNMpNux1mehes8C3beCQajes0D3rWD83ILpvlWvxmAZYzzAd8B5QAGwEBhtrV3pamA+YIzZAPSy1gb9JHLGmAHAfuBVa20n77pHgEJr7cPePzKSrbW/cTPOmjjGtT0A7LfWPupmbLVhjMkAMqy13xhj4oGvgYuB6wniz+041/UTgv8zM0CctXa/MSYCmA3cA/wCeN9aO84Y8yyw1Fr7jJux1lehfM8C3beCQajes0D3rWAUTPet+taC1RvItdaus9aWAuOAES7HJEex1n4BFB61egQw1rs8FufLIugc49qCnrV2i7X2G+/yPmAVkEmQf27Hua6gZx37vS8jvA8LnAO8610fdJ9ZiNE9K0iE6n0rVO9ZoPtWMAqm+1Z9S7AygfwqrwsIkX90OP/APjfGfG2MGeN2MH6Qbq3dAs6XB9DI5Xh87U5jzDJvd4yg6o5wNGNMM6A7MJ8Q+tyOui4Igc/MGOMxxiwBtgNTgLVAkbW23LtJKH1HBqNQvmeB7lvBLOi//6rSfSt4BMt9q74lWKaadaHSR/IMa20PYAhwh7dZX4LDM0BLoBuwBfinu+HUnDGmAfAe8DNr7V634/GVaq4rJD4za22FtbYbkIXTWtK+us3qNiqpIpTvWaD7VrAKie+/w3TfCi7Bct+qbwlWAZBd5XUWsNmlWHzKWrvZ+7wdmIDzjy6UbPP2Kz7cv3i7y/H4jLV2m/cLoxJ4gSD97Lz9od8D3rDWvu9dHfSfW3XXFSqf2WHW2iJgJtAXSDLGhHvfCpnvyCAVsvcs0H0rWIXS95/uW8Er0O9b9S3BWgi09lYbiQSuACa6HFOtGWPivAMZMcbEAecD3x5/r6AzEbjOu3wd8KGLsfjU4S9yr0sIws/OO/D0JWCVtfaxKm8F9ed2rOsKkc8szRiT5F2OAc7F6as/Axjp3SzoPrMQE5L3LNB9K5iFwvcf6L5FEH5uwXTfqldVBAG8ZSn/BXiAl621D7ocUq0ZY1rg/PoHEA68GczXZYx5CzgLaAhsA/4EfACMB3KAPGCUtTboBt4e49rOwmmyt8AG4NbD/b+DhTHmTOBLYDlQ6V39W5x+30H7uR3nukYT/J9ZF5zBwB6cH9vGW2v/7P0+GQekAIuBq621h9yLtH4LxXsW6L4VLEL1ngW6bxGEn1sw3bfqXYIlIiIiIiLiL/Wti6CIiIiIiIjfKMESERERERHxESVYIiIiIiIiPqIES0RERERExEeUYImIiIiIiPiIEiwRPzLGVBhjllR53OfDYzczxgTdPBYiIhK4dN8Sqb3wE28iIrVQYq3t5nYQIiIiJ0n3LZFaUguWiAuMMRuMMX83xizwPlp51zc1xkwzxizzPud416cbYyYYY5Z6H6d7D+UxxrxgjFlhjPncO7O5iIiIT+m+JXLylGCJ+FfMUV0tLq/y3l5rbW/gSeBf3nVPAq9aa7sAbwBPeNc/Acyy1nYFegArvOtbA09ZazsCRcBlfr4eEREJbbpvidSSsda6HYNIyDLG7LfWNqhm/QbgHGvtOmNMBLDVWptqjNkJZFhry7zrt1hrGxpjdgBZ1tpDVY7RDJhirW3tff0bIMJa+1f/X5mIiIQi3bdEak8tWCLuscdYPtY21TlUZbkCjasUERH/0X1L5CQowRJxz+VVnud6l+cAV3iXrwJme5enAbcDGGM8xpiEugpSRETES/ctkZOgXw3+v527tUEgCMIA+o0i9EMzSILCgKIZBHXQAo4uENDDIu5IcJCwFwTvuV01bjI/uzCteVVdXs6n1trzy9tZVZ0zNDqW4902ybGq9kluSVbj/S7JoarWGTp+myTXyaMH4N/IW/Alb7DgB8Zd9kVr7f7rWADgHXkLPmdFEAAAoBMTLAAAgE5MsAAAADpRYAEAAHSiwAIAAOhEgQUAANCJAgsAAKCTB3ApryzYeF+VAAAAAElFTkSuQmCC\n", 133 | "text/plain": [ 134 | "
" 135 | ] 136 | }, 137 | "metadata": { 138 | "needs_background": "light" 139 | }, 140 | "output_type": "display_data" 141 | } 142 | ], 143 | "source": [ 144 | "fig, (ax0, ax1) = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\n", 145 | "\n", 146 | "# Plot losses\n", 147 | "ax0.plot(history['loss'], label='train')\n", 148 | "ax0.plot(history['val_loss'], label='validation')\n", 149 | "ax0.set_xlabel('Epoch')\n", 150 | "ax0.set_ylabel('Loss')\n", 151 | "ax0.legend(loc=0)\n", 152 | "\n", 153 | "ax1.plot(history['acc'], label='train')\n", 154 | "ax1.plot(history['val_acc'], label='validation')\n", 155 | "ax1.set_xlabel('Epoch')\n", 156 | "ax1.set_ylabel('Accuracy')\n", 157 | "ax1.legend(loc=0)\n", 158 | "\n", 159 | "plt.tight_layout()" 160 | ] 161 | } 162 | ], 163 | "metadata": { 164 | "kernelspec": { 165 | "display_name": "tf-1.11.0-py36", 166 | "language": "python", 167 | "name": "tf-1.11.0-py36" 168 | }, 169 | "language_info": { 170 | "codemirror_mode": { 171 | "name": "ipython", 172 | "version": 3 173 | }, 174 | "file_extension": ".py", 175 | "mimetype": "text/x-python", 176 | "name": "python", 177 | "nbconvert_exporter": "python", 178 | "pygments_lexer": "ipython3", 179 | "version": "3.6.5" 180 | }, 181 | "toc-autonumbering": false, 182 | "toc-showmarkdowntxt": true, 183 | "toc-showtags": false 184 | }, 185 | "nbformat": 4, 186 | "nbformat_minor": 2 187 | } 188 | -------------------------------------------------------------------------------- /scripts/cifar_cnn.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH -J cifar-cnn 3 | #SBATCH -C knl 4 | #SBATCH -N 1 5 | #SBATCH -q regular 6 | #SBATCH --reservation dl4sci 7 | #SBATCH -t 1:00:00 8 | #SBATCH -o logs/%x-%j.out 9 | 10 | # Load the software 11 | module load tensorflow/intel-1.13.1-py36 12 | export KMP_BLOCKTIME=1 13 | export KMP_AFFINITY="granularity=fine,compact,1,0" 14 | 15 | # Ensure dataset is downloaded by single process 16 | python -c "import keras; keras.datasets.cifar10.load_data()" 17 | 18 | # Submit multi-node training 19 | srun -u python train_horovod.py configs/cifar10_cnn.yaml -d 20 | -------------------------------------------------------------------------------- /scripts/cifar_resnet.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH -J cifar-resnet 3 | #SBATCH -C knl 4 | #SBATCH -N 1 5 | #SBATCH -q regular 6 | #SBATCH --reservation dl4sci 7 | #SBATCH -t 1:00:00 8 | #SBATCH -o logs/%x-%j.out 9 | 10 | # Load the software 11 | module load tensorflow/intel-1.13.1-py36 12 | export KMP_BLOCKTIME=1 13 | export KMP_AFFINITY="granularity=fine,compact,1,0" 14 | 15 | # Ensure dataset is downloaded by single process 16 | python -c "import keras; keras.datasets.cifar10.load_data()" 17 | 18 | # Submit multi-node training 19 | srun -u python train_horovod.py configs/cifar10_resnet.yaml -d 20 | -------------------------------------------------------------------------------- /train_horovod.py: -------------------------------------------------------------------------------- 1 | """ 2 | Main training script for the Deep Learning at Scale Keras examples. 3 | """ 4 | 5 | # System 6 | import os 7 | import sys 8 | import argparse 9 | import logging 10 | 11 | # Externals 12 | import keras 13 | import horovod.keras as hvd 14 | import yaml 15 | import numpy as np 16 | 17 | # Locals 18 | from data import get_datasets 19 | from models import get_model 20 | from utils.device import configure_session 21 | from utils.optimizers import get_optimizer 22 | from utils.callbacks import TimingCallback 23 | 24 | # Suppress TF warnings 25 | import tensorflow as tf 26 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' 27 | tf.logging.set_verbosity(logging.ERROR) 28 | 29 | #load dictionary from argparse 30 | class StoreDictKeyPair(argparse.Action): 31 | def __call__(self, parser, namespace, values, option_string=None): 32 | my_dict = {} 33 | for kv in values.split(","): 34 | k,v = kv.split("=") 35 | my_dict[k] = v 36 | setattr(namespace, self.dest, my_dict) 37 | 38 | def parse_args(): 39 | """Parse command line arguments.""" 40 | parser = argparse.ArgumentParser('train.py') 41 | add_arg = parser.add_argument 42 | add_arg('config', nargs='?', default='configs/hello.yaml') 43 | add_arg('-d', '--distributed', action='store_true') 44 | add_arg('-v', '--verbose', action='store_true') 45 | add_arg('--show-config', action='store_true') 46 | add_arg('--interactive', action='store_true') 47 | #parameters which override the YAML file 48 | add_arg('--dropout', type=float, help='keep rate for dropout layers') 49 | add_arg("--optimizer", action=StoreDictKeyPair, help="optimizer parameters") 50 | add_arg('--batch_size', type=int, help='batch size for training') 51 | add_arg('--n_epochs', type=int, help='number of epochs to train') 52 | return parser.parse_args() 53 | 54 | def config_logging(verbose): 55 | log_format = '%(asctime)s %(levelname)s %(message)s' 56 | log_level = logging.DEBUG if verbose else logging.INFO 57 | logging.basicConfig(level=log_level, format=log_format) 58 | 59 | def init_workers(distributed=False): 60 | rank, n_ranks = 0, 1 61 | if distributed: 62 | hvd.init() 63 | rank, n_ranks = hvd.rank(), hvd.size() 64 | return rank, n_ranks 65 | 66 | def load_config(arguments): 67 | #read base config from yaml file 68 | config_file = arguments.config 69 | with open(config_file) as f: 70 | config = yaml.load(f, Loader=yaml.FullLoader) 71 | 72 | #override with CLA 73 | if arguments.dropout: 74 | config["model"]["dropout"] = arguments.dropout 75 | if arguments.batch_size: 76 | config["training"]["batch_size"] = arguments.batch_size 77 | if arguments.n_epochs: 78 | config["training"]["n_epochs"] = arguments.n_epochs 79 | if arguments.optimizer: 80 | if "name" in arguments.optimizer: 81 | config["optimizer"]["name"] = arguments.optimizer["name"] 82 | if "lr" in arguments.optimizer: 83 | config["optimizer"]["lr"] = float(arguments.optimizer["lr"] ) 84 | if "lr_scaling" in arguments.optimizer: 85 | config["optimizer"]["lr_scaling"] = arguments.optimizer["lr_scaling"] 86 | if "lr_warmup_epochs" in arguments.optimizer: 87 | config["training"]["lr_warmup_epochs"] = int(arguments.optimizer["lr_warmup_epochs"]) 88 | 89 | return config 90 | 91 | def get_basic_callbacks(distributed=False): 92 | cb = [] 93 | 94 | if distributed: 95 | #this is for broadcasting the initial model to all nodes 96 | cb.append(hvd.callbacks.BroadcastGlobalVariablesCallback(0)) 97 | 98 | #this is for averaging the reported metrics across all nodes 99 | cb.append(hvd.callbacks.MetricAverageCallback()) 100 | 101 | return cb 102 | 103 | def main(): 104 | """Main function""" 105 | 106 | # Initialization 107 | args = parse_args() 108 | rank, n_ranks = init_workers(args.distributed) 109 | 110 | # Load configuration 111 | config = load_config(args) 112 | train_config = config['training'] 113 | output_dir = os.path.expandvars(config['output_dir']) 114 | checkpoint_format = os.path.join(output_dir, 'checkpoints', 115 | 'checkpoint-{epoch}.h5') 116 | if rank==0: 117 | os.makedirs(output_dir, exist_ok=True) 118 | 119 | # Loggging 120 | config_logging(verbose=args.verbose) 121 | logging.info('Initialized rank %i out of %i', rank, n_ranks) 122 | if args.show_config: 123 | logging.info('Command line config: %s', args) 124 | if rank == 0: 125 | logging.info('Job configuration: %s', config) 126 | logging.info('Saving job outputs to %s', output_dir) 127 | 128 | # Configure session 129 | device_config = config.get('device', {}) 130 | configure_session(**device_config) 131 | 132 | # Load the data 133 | train_gen, valid_gen = get_datasets(batch_size=train_config['batch_size'], 134 | **config['data']) 135 | 136 | # Build the model 137 | model = get_model(**config['model']) 138 | # Configure optimizer 139 | opt = get_optimizer(n_ranks=n_ranks, **config['optimizer']) 140 | # Compile the model 141 | model.compile(loss=train_config['loss'], optimizer=opt, 142 | metrics=train_config['metrics']) 143 | if rank == 0: 144 | model.summary() 145 | 146 | # Prepare the training callbacks 147 | callbacks = get_basic_callbacks(args.distributed) 148 | 149 | # Learning rate warmup 150 | warmup_epochs = train_config.get('lr_warmup_epochs', 0) 151 | callbacks.append(hvd.callbacks.LearningRateWarmupCallback( 152 | warmup_epochs=warmup_epochs, verbose=1)) 153 | 154 | # Learning rate decay schedule 155 | for lr_schedule in train_config.get('lr_schedule', []): 156 | if rank == 0: 157 | logging.info('Adding LR schedule: %s', lr_schedule) 158 | callbacks.append(hvd.callbacks.LearningRateScheduleCallback(**lr_schedule)) 159 | 160 | # Checkpoint only from rank 0 161 | if rank == 0: 162 | os.makedirs(os.path.dirname(checkpoint_format), exist_ok=True) 163 | callbacks.append(keras.callbacks.ModelCheckpoint(checkpoint_format)) 164 | 165 | # Timing callback 166 | timing_callback = TimingCallback() 167 | callbacks.append(timing_callback) 168 | 169 | # Train the model 170 | train_steps_per_epoch = max([len(train_gen) // n_ranks, 1]) 171 | valid_steps_per_epoch = max([len(valid_gen) // n_ranks, 1]) 172 | history = model.fit_generator(train_gen, 173 | epochs=train_config['n_epochs'], 174 | steps_per_epoch=train_steps_per_epoch, 175 | validation_data=valid_gen, 176 | validation_steps=valid_steps_per_epoch, 177 | callbacks=callbacks, 178 | workers=4, verbose=2 if rank==0 else 0) 179 | 180 | # Save training history 181 | if rank == 0: 182 | # Print some best-found metrics 183 | if 'val_acc' in history.history.keys(): 184 | logging.info('Best validation accuracy: %.3f', 185 | max(history.history['val_acc'])) 186 | if 'val_top_k_categorical_accuracy' in history.history.keys(): 187 | logging.info('Best top-5 validation accuracy: %.3f', 188 | max(history.history['val_top_k_categorical_accuracy'])) 189 | logging.info('Average time per epoch: %.3f s', 190 | np.mean(timing_callback.times)) 191 | np.savez(os.path.join(output_dir, 'history'), 192 | n_ranks=n_ranks, **history.history) 193 | 194 | # Drop to IPython interactive shell 195 | if args.interactive and (rank == 0): 196 | logging.info('Starting IPython interactive session') 197 | import IPython 198 | IPython.embed() 199 | 200 | if rank == 0: 201 | logging.info('All done!') 202 | 203 | if __name__ == '__main__': 204 | main() 205 | -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | """ 3 | -------------------------------------------------------------------------------- /utils/callbacks.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module contains some utility callbacks for Keras training. 3 | """ 4 | 5 | # System 6 | from time import time 7 | 8 | # Externals 9 | import keras 10 | 11 | class TimingCallback(keras.callbacks.Callback): 12 | """A Keras Callback which records the time of each epoch""" 13 | def __init__(self): 14 | self.times = [] 15 | 16 | def on_epoch_begin(self, epoch, logs={}): 17 | self.starttime = time() 18 | 19 | def on_epoch_end(self, epoch, logs={}): 20 | epoch_time = time() - self.starttime 21 | self.times.append(epoch_time) 22 | -------------------------------------------------------------------------------- /utils/device.py: -------------------------------------------------------------------------------- 1 | """ 2 | Hardware/device configuration 3 | """ 4 | 5 | # System 6 | import os 7 | 8 | # Externals 9 | import keras 10 | import tensorflow as tf 11 | 12 | def configure_session(intra_threads=32, inter_threads=2, gpu=None): 13 | """Sets the thread knobs in the TF backend""" 14 | os.environ['OMP_NUM_THREADS'] = str(intra_threads) 15 | config = tf.ConfigProto( 16 | inter_op_parallelism_threads=inter_threads, 17 | intra_op_parallelism_threads=intra_threads 18 | ) 19 | if gpu is not None: 20 | config.gpu_options.visible_device_list = str(gpu) 21 | keras.backend.set_session(tf.Session(config=config)) 22 | -------------------------------------------------------------------------------- /utils/optimizers.py: -------------------------------------------------------------------------------- 1 | """ 2 | Utilty code for constructing optimizers and scheduling learning rates. 3 | """ 4 | 5 | # System 6 | import math 7 | 8 | # Externals 9 | import keras 10 | import horovod.keras as hvd 11 | 12 | def get_optimizer(name, lr, lr_scaling='linear', n_ranks=1, **opt_args): 13 | """ 14 | Configure the optimizer and scale the learning rate by n_ranks. 15 | """ 16 | # Scale the learning rate 17 | if lr_scaling == 'linear': 18 | lr = lr * n_ranks 19 | elif lr_scaling == 'sqrt': 20 | lr = lr * math.sqrt(n_ranks) 21 | 22 | # Construct the optimizer 23 | OptType = getattr(keras.optimizers, name) 24 | opt = OptType(lr=lr, **opt_args) 25 | 26 | # Distributed optimizer wrapper 27 | if n_ranks > 1: 28 | opt = hvd.DistributedOptimizer(opt) 29 | 30 | return opt 31 | --------------------------------------------------------------------------------